\input texinfo   @c -*-texinfo-*-
@c %**start of header
@setfilename python.info
@documentencoding UTF-8
@ifinfo
@*Generated by Sphinx 4.2.0.@*
@end ifinfo
@settitle Python
@defindex ge
@paragraphindent 0
@exampleindent 4
@finalout
@dircategory Miscellaneous
@direntry
* python: (python.info). One line description of project
@end direntry

@definfoenclose strong,`,'
@definfoenclose emph,`,'
@c %**end of header

@copying
@quotation
Python 3.11.0a5, March 07, 2022

unknown

Copyright @copyright{} 2001-2022, Python Software Foundation
@end quotation

@end copying

@titlepage
@title Python
@insertcopying
@end titlepage
@contents

@c %** start of user preamble

@c %** end of user preamble

@ifnottex
@node Top
@top Python
@insertcopying
@end ifnottex

@c %**start of body
@anchor{contents doc}@anchor{14d}
@menu
* What’s New in Python:: 
* The Python Tutorial:: 
* Python Setup and Usage:: 
* The Python Language Reference:: 
* The Python Standard Library:: 
* Extending and Embedding the Python Interpreter:: 
* Python/C API Reference Manual:: 
* Distributing Python Modules:: 
* Installing Python Modules:: 
* Python HOWTOs:: 
* Python Frequently Asked Questions:: 
* Glossary:: 
* About these documents:: 
* Dealing with Bugs:: 
* Copyright:: 
* History and License:: 
* Distributing Python Modules (Legacy version): Distributing Python Modules Legacy version. 
* Installing Python Modules (Legacy version): Installing Python Modules Legacy version. 
* Python Module Index:: 
* Index:: 

@detailmenu
 --- The Detailed Node Listing ---

What’s New in Python

* What’s New In Python 3.11: What’s New In Python 3 11. 
* What’s New In Python 3.10: What’s New In Python 3 10. 
* What’s New In Python 3.9: What’s New In Python 3 9. 
* What’s New In Python 3.8: What’s New In Python 3 8. 
* What’s New In Python 3.7: What’s New In Python 3 7. 
* What’s New In Python 3.6: What’s New In Python 3 6. 
* What’s New In Python 3.5: What’s New In Python 3 5. 
* What’s New In Python 3.4: What’s New In Python 3 4. 
* What’s New In Python 3.3: What’s New In Python 3 3. 
* What’s New In Python 3.2: What’s New In Python 3 2. 
* What’s New In Python 3.1: What’s New In Python 3 1. 
* What’s New In Python 3.0: What’s New In Python 3 0. 
* What’s New in Python 2.7: What’s New in Python 2 7. 
* What’s New in Python 2.6: What’s New in Python 2 6. 
* What’s New in Python 2.5: What’s New in Python 2 5. 
* What’s New in Python 2.4: What’s New in Python 2 4. 
* What’s New in Python 2.3: What’s New in Python 2 3. 
* What’s New in Python 2.2: What’s New in Python 2 2. 
* What’s New in Python 2.1: What’s New in Python 2 1. 
* What’s New in Python 2.0: What’s New in Python 2 0. 
* Changelog:: 

What’s New In Python 3.11

* Summary – Release highlights:: 
* New Features:: 
* Other Language Changes:: 
* Other CPython Implementation Changes:: 
* New Modules:: 
* Improved Modules:: 
* Optimizations:: 
* CPython bytecode changes:: 
* Deprecated:: 
* Removed:: 
* Porting to Python 3.11: Porting to Python 3 11. 
* Build Changes:: 
* C API Changes:: 

New Features

* Enhanced error locations in tracebacks:: 

Enhanced error locations in tracebacks

* Column information for code objects:: 
* Exceptions can be enriched with a string __note__:: 

Improved Modules

* fractions:: 
* IDLE and idlelib:: 
* inspect:: 
* math:: 
* operator:: 
* os:: 
* socket:: 
* sqlite3:: 
* sys:: 
* threading:: 
* time:: 
* unicodedata:: 
* fcntl:: 

Porting to Python 3.11

* Changes in the Python API:: 

C API Changes

* New Features: New Features<2>. 
* Porting to Python 3.11: Porting to Python 3 11<2>. 
* Deprecated: Deprecated<2>. 
* Removed: Removed<2>. 

What’s New In Python 3.10

* Summary – Release highlights: Summary – Release highlights<2>. 
* New Features: New Features<3>. 
* New Features Related to Type Hints:: 
* Other Language Changes: Other Language Changes<2>. 
* New Modules: New Modules<2>. 
* Improved Modules: Improved Modules<2>. 
* Optimizations: Optimizations<2>. 
* Deprecated: Deprecated<3>. 
* Removed: Removed<3>. 
* Porting to Python 3.10: Porting to Python 3 10. 
* CPython bytecode changes: CPython bytecode changes<2>. 
* Build Changes: Build Changes<2>. 
* C API Changes: C API Changes<2>. 

New Features

* Parenthesized context managers:: 
* Better error messages:: 
* PEP 626; Precise line numbers for debugging and other tools: PEP 626 Precise line numbers for debugging and other tools. 
* PEP 634; Structural Pattern Matching: PEP 634 Structural Pattern Matching. 
* Optional EncodingWarning and encoding="locale" option:: 

Better error messages

* SyntaxErrors:: 
* IndentationErrors:: 
* AttributeErrors:: 
* NameErrors:: 

PEP 634: Structural Pattern Matching

* Syntax and operations:: 
* Declarative approach:: 
* Simple pattern; match to a literal: Simple pattern match to a literal. 
* Patterns with a literal and variable:: 
* Patterns and classes:: 
* Nested patterns:: 
* Complex patterns and the wildcard:: 
* Guard:: 
* Other Key Features:: 

Simple pattern: match to a literal

* Behavior without the wildcard:: 

Patterns and classes

* Patterns with positional parameters:: 

New Features Related to Type Hints

* PEP 604; New Type Union Operator: PEP 604 New Type Union Operator. 
* PEP 612; Parameter Specification Variables: PEP 612 Parameter Specification Variables. 
* PEP 613; TypeAlias: PEP 613 TypeAlias. 
* PEP 647; User-Defined Type Guards: PEP 647 User-Defined Type Guards. 

Improved Modules

* asyncio:: 
* argparse:: 
* array:: 
* asynchat@comma{} asyncore@comma{} smtpd: asynchat asyncore smtpd. 
* base64:: 
* bdb:: 
* bisect:: 
* codecs:: 
* collections.abc: collections abc. 
* contextlib:: 
* curses:: 
* dataclasses:: 
* distutils:: 
* doctest:: 
* encodings:: 
* enum:: 
* fileinput:: 
* faulthandler:: 
* gc:: 
* glob:: 
* hashlib:: 
* hmac:: 
* IDLE and idlelib: IDLE and idlelib<2>. 
* importlib.metadata: importlib metadata. 
* inspect: inspect<2>. 
* linecache:: 
* os: os<2>. 
* os.path: os path. 
* pathlib:: 
* platform:: 
* pprint:: 
* py_compile:: 
* pyclbr:: 
* shelve:: 
* statistics:: 
* site:: 
* socket: socket<2>. 
* ssl:: 
* sqlite3: sqlite3<2>. 
* sys: sys<2>. 
* _thread:: 
* threading: threading<2>. 
* traceback:: 
* types:: 
* typing:: 
* unittest:: 
* urllib.parse: urllib parse. 
* xml:: 
* zipimport:: 

dataclasses

* __slots__:: 
* Keyword-only fields:: 

Porting to Python 3.10

* Changes in the Python syntax:: 
* Changes in the Python API: Changes in the Python API<2>. 
* Changes in the C API:: 

C API Changes

* PEP 652; Maintaining the Stable ABI: PEP 652 Maintaining the Stable ABI. 
* New Features: New Features<4>. 
* Porting to Python 3.10: Porting to Python 3 10<2>. 
* Deprecated: Deprecated<4>. 
* Removed: Removed<4>. 

What’s New In Python 3.9

* Summary – Release highlights: Summary – Release highlights<3>. 
* You should check for DeprecationWarning in your code:: 
* New Features: New Features<5>. 
* Other Language Changes: Other Language Changes<3>. 
* New Modules: New Modules<3>. 
* Improved Modules: Improved Modules<3>. 
* Optimizations: Optimizations<3>. 
* Deprecated: Deprecated<5>. 
* Removed: Removed<5>. 
* Porting to Python 3.9: Porting to Python 3 9. 
* Build Changes: Build Changes<3>. 
* C API Changes: C API Changes<3>. 
* Notable changes in Python 3.9.1: Notable changes in Python 3 9 1. 
* Notable changes in Python 3.9.2: Notable changes in Python 3 9 2. 

New Features

* Dictionary Merge & Update Operators:: 
* New String Methods to Remove Prefixes and Suffixes:: 
* Type Hinting Generics in Standard Collections:: 
* New Parser:: 

New Modules

* zoneinfo:: 
* graphlib:: 

Improved Modules

* ast:: 
* asyncio: asyncio<2>. 
* compileall:: 
* concurrent.futures: concurrent futures. 
* curses: curses<2>. 
* datetime:: 
* distutils: distutils<2>. 
* fcntl: fcntl<2>. 
* ftplib:: 
* gc: gc<2>. 
* hashlib: hashlib<2>. 
* http:: 
* IDLE and idlelib: IDLE and idlelib<3>. 
* imaplib:: 
* importlib:: 
* inspect: inspect<3>. 
* ipaddress:: 
* math: math<2>. 
* multiprocessing:: 
* nntplib:: 
* os: os<3>. 
* pathlib: pathlib<2>. 
* pdb:: 
* poplib:: 
* pprint: pprint<2>. 
* pydoc:: 
* random:: 
* signal:: 
* smtplib:: 
* socket: socket<3>. 
* time: time<2>. 
* sys: sys<3>. 
* tracemalloc:: 
* typing: typing<2>. 
* unicodedata: unicodedata<2>. 
* venv:: 
* xml: xml<2>. 

Porting to Python 3.9

* Changes in the Python API: Changes in the Python API<3>. 
* Changes in the C API: Changes in the C API<2>. 
* CPython bytecode changes: CPython bytecode changes<3>. 

C API Changes

* New Features: New Features<6>. 
* Porting to Python 3.9: Porting to Python 3 9<2>. 
* Removed: Removed<6>. 

Notable changes in Python 3.9.1

* typing: typing<3>. 
* macOS 11.0 (Big Sur) and Apple Silicon Mac support: macOS 11 0 Big Sur and Apple Silicon Mac support. 

Notable changes in Python 3.9.2

* collections.abc: collections abc<2>. 
* urllib.parse: urllib parse<2>. 

What’s New In Python 3.8

* Summary – Release highlights: Summary – Release highlights<4>. 
* New Features: New Features<7>. 
* Other Language Changes: Other Language Changes<4>. 
* New Modules: New Modules<4>. 
* Improved Modules: Improved Modules<4>. 
* Optimizations: Optimizations<4>. 
* Build and C API Changes:: 
* Deprecated: Deprecated<6>. 
* API and Feature Removals:: 
* Porting to Python 3.8: Porting to Python 3 8. 
* Notable changes in Python 3.8.1: Notable changes in Python 3 8 1. 
* Notable changes in Python 3.8.8: Notable changes in Python 3 8 8. 
* Notable changes in Python 3.8.12: Notable changes in Python 3 8 12. 

New Features

* Assignment expressions:: 
* Positional-only parameters:: 
* Parallel filesystem cache for compiled bytecode files:: 
* Debug build uses the same ABI as release build:: 
* f-strings support = for self-documenting expressions and debugging:: 
* PEP 578; Python Runtime Audit Hooks: PEP 578 Python Runtime Audit Hooks. 
* PEP 587; Python Initialization Configuration: PEP 587 Python Initialization Configuration. 
* PEP 590; Vectorcall; a fast calling protocol for CPython: PEP 590 Vectorcall a fast calling protocol for CPython. 
* Pickle protocol 5 with out-of-band data buffers:: 

Improved Modules

* ast: ast<2>. 
* asyncio: asyncio<3>. 
* builtins:: 
* collections:: 
* cProfile:: 
* csv:: 
* curses: curses<3>. 
* ctypes:: 
* datetime: datetime<2>. 
* functools:: 
* gc: gc<3>. 
* gettext:: 
* gzip:: 
* IDLE and idlelib: IDLE and idlelib<4>. 
* inspect: inspect<4>. 
* io:: 
* itertools:: 
* json.tool: json tool. 
* logging:: 
* math: math<3>. 
* mmap:: 
* multiprocessing: multiprocessing<2>. 
* os: os<4>. 
* os.path: os path<2>. 
* pathlib: pathlib<3>. 
* pickle:: 
* plistlib:: 
* pprint: pprint<3>. 
* py_compile: py_compile<2>. 
* shlex:: 
* shutil:: 
* socket: socket<4>. 
* ssl: ssl<2>. 
* statistics: statistics<2>. 
* sys: sys<4>. 
* tarfile:: 
* threading: threading<3>. 
* tokenize:: 
* tkinter:: 
* time: time<3>. 
* typing: typing<4>. 
* unicodedata: unicodedata<3>. 
* unittest: unittest<2>. 
* venv: venv<2>. 
* weakref:: 
* xml: xml<3>. 
* xmlrpc:: 

Porting to Python 3.8

* Changes in Python behavior:: 
* Changes in the Python API: Changes in the Python API<4>. 
* Changes in the C API: Changes in the C API<3>. 
* CPython bytecode changes: CPython bytecode changes<4>. 
* Demos and Tools:: 

What’s New In Python 3.7

* Summary – Release Highlights:: 
* New Features: New Features<8>. 
* Other Language Changes: Other Language Changes<5>. 
* New Modules: New Modules<5>. 
* Improved Modules: Improved Modules<5>. 
* C API Changes: C API Changes<4>. 
* Build Changes: Build Changes<4>. 
* Optimizations: Optimizations<5>. 
* Other CPython Implementation Changes: Other CPython Implementation Changes<2>. 
* Deprecated Python Behavior:: 
* Deprecated Python modules@comma{} functions and methods: Deprecated Python modules functions and methods. 
* Deprecated functions and types of the C API:: 
* Platform Support Removals:: 
* API and Feature Removals: API and Feature Removals<2>. 
* Module Removals:: 
* Windows-only Changes:: 
* Porting to Python 3.7: Porting to Python 3 7. 
* Notable changes in Python 3.7.1: Notable changes in Python 3 7 1. 
* Notable changes in Python 3.7.2: Notable changes in Python 3 7 2. 
* Notable changes in Python 3.7.6: Notable changes in Python 3 7 6. 
* Notable changes in Python 3.7.10: Notable changes in Python 3 7 10. 

New Features

* PEP 563; Postponed Evaluation of Annotations: PEP 563 Postponed Evaluation of Annotations. 
* PEP 538; Legacy C Locale Coercion: PEP 538 Legacy C Locale Coercion. 
* PEP 540; Forced UTF-8 Runtime Mode: PEP 540 Forced UTF-8 Runtime Mode. 
* PEP 553; Built-in breakpoint(): PEP 553 Built-in breakpoint. 
* PEP 539; New C API for Thread-Local Storage: PEP 539 New C API for Thread-Local Storage. 
* PEP 562; Customization of Access to Module Attributes: PEP 562 Customization of Access to Module Attributes. 
* PEP 564; New Time Functions With Nanosecond Resolution: PEP 564 New Time Functions With Nanosecond Resolution. 
* PEP 565; Show DeprecationWarning in __main__: PEP 565 Show DeprecationWarning in __main__. 
* PEP 560; Core Support for typing module and Generic Types: PEP 560 Core Support for typing module and Generic Types. 
* PEP 552; Hash-based .pyc Files: PEP 552 Hash-based pyc Files. 
* PEP 545; Python Documentation Translations: PEP 545 Python Documentation Translations. 
* Python Development Mode (-X dev): Python Development Mode -X dev. 

New Modules

* contextvars:: 
* dataclasses: dataclasses<2>. 
* importlib.resources: importlib resources. 

Improved Modules

* argparse: argparse<2>. 
* asyncio: asyncio<4>. 
* binascii:: 
* calendar:: 
* collections: collections<2>. 
* compileall: compileall<2>. 
* concurrent.futures: concurrent futures<2>. 
* contextlib: contextlib<2>. 
* cProfile: cProfile<2>. 
* crypt:: 
* datetime: datetime<3>. 
* dbm:: 
* decimal:: 
* dis:: 
* distutils: distutils<3>. 
* enum: enum<2>. 
* functools: functools<2>. 
* gc: gc<4>. 
* hmac: hmac<2>. 
* http.client: http client. 
* http.server: http server. 
* idlelib and IDLE:: 
* importlib: importlib<2>. 
* io: io<2>. 
* ipaddress: ipaddress<2>. 
* itertools: itertools<2>. 
* locale:: 
* logging: logging<2>. 
* math: math<4>. 
* mimetypes:: 
* msilib:: 
* multiprocessing: multiprocessing<3>. 
* os: os<5>. 
* pathlib: pathlib<4>. 
* pdb: pdb<2>. 
* py_compile: py_compile<3>. 
* pydoc: pydoc<2>. 
* queue:: 
* re:: 
* signal: signal<2>. 
* socket: socket<5>. 
* socketserver:: 
* sqlite3: sqlite3<3>. 
* ssl: ssl<3>. 
* string:: 
* subprocess:: 
* sys: sys<5>. 
* time: time<4>. 
* tkinter: tkinter<2>. 
* tracemalloc: tracemalloc<2>. 
* types: types<2>. 
* unicodedata: unicodedata<4>. 
* unittest: unittest<3>. 
* unittest.mock: unittest mock. 
* urllib.parse: urllib parse<3>. 
* uu:: 
* uuid:: 
* warnings:: 
* xml.etree: xml etree. 
* xmlrpc.server: xmlrpc server. 
* zipapp:: 
* zipfile:: 

Deprecated Python modules, functions and methods

* aifc:: 
* asyncio: asyncio<5>. 
* collections: collections<3>. 
* dbm: dbm<2>. 
* enum: enum<3>. 
* gettext: gettext<2>. 
* importlib: importlib<3>. 
* locale: locale<2>. 
* macpath:: 
* threading: threading<4>. 
* socket: socket<6>. 
* ssl: ssl<4>. 
* sunau:: 
* sys: sys<6>. 
* wave:: 

Porting to Python 3.7

* Changes in Python Behavior:: 
* Changes in the Python API: Changes in the Python API<5>. 
* Changes in the C API: Changes in the C API<4>. 
* CPython bytecode changes: CPython bytecode changes<5>. 
* Windows-only Changes: Windows-only Changes<2>. 
* Other CPython implementation changes:: 

What’s New In Python 3.6

* Summary – Release highlights: Summary – Release highlights<5>. 
* New Features: New Features<9>. 
* Other Language Changes: Other Language Changes<6>. 
* New Modules: New Modules<6>. 
* Improved Modules: Improved Modules<6>. 
* Optimizations: Optimizations<6>. 
* Build and C API Changes: Build and C API Changes<2>. 
* Other Improvements:: 
* Deprecated: Deprecated<7>. 
* Removed: Removed<7>. 
* Porting to Python 3.6: Porting to Python 3 6. 
* Notable changes in Python 3.6.2: Notable changes in Python 3 6 2. 
* Notable changes in Python 3.6.4: Notable changes in Python 3 6 4. 
* Notable changes in Python 3.6.5: Notable changes in Python 3 6 5. 
* Notable changes in Python 3.6.7: Notable changes in Python 3 6 7. 
* Notable changes in Python 3.6.10: Notable changes in Python 3 6 10. 
* Notable changes in Python 3.6.13: Notable changes in Python 3 6 13. 

New Features

* PEP 498; Formatted string literals: PEP 498 Formatted string literals. 
* PEP 526; Syntax for variable annotations: PEP 526 Syntax for variable annotations. 
* PEP 515; Underscores in Numeric Literals: PEP 515 Underscores in Numeric Literals. 
* PEP 525; Asynchronous Generators: PEP 525 Asynchronous Generators. 
* PEP 530; Asynchronous Comprehensions: PEP 530 Asynchronous Comprehensions. 
* PEP 487; Simpler customization of class creation: PEP 487 Simpler customization of class creation. 
* PEP 487; Descriptor Protocol Enhancements: PEP 487 Descriptor Protocol Enhancements. 
* PEP 519; Adding a file system path protocol: PEP 519 Adding a file system path protocol. 
* PEP 495; Local Time Disambiguation: PEP 495 Local Time Disambiguation. 
* PEP 529; Change Windows filesystem encoding to UTF-8: PEP 529 Change Windows filesystem encoding to UTF-8. 
* PEP 528; Change Windows console encoding to UTF-8: PEP 528 Change Windows console encoding to UTF-8. 
* PEP 520; Preserving Class Attribute Definition Order: PEP 520 Preserving Class Attribute Definition Order. 
* PEP 468; Preserving Keyword Argument Order: PEP 468 Preserving Keyword Argument Order. 
* New dict implementation:: 
* PEP 523; Adding a frame evaluation API to CPython: PEP 523 Adding a frame evaluation API to CPython. 
* PYTHONMALLOC environment variable:: 
* DTrace and SystemTap probing support:: 

New Modules

* secrets:: 

Improved Modules

* array: array<2>. 
* ast: ast<3>. 
* asyncio: asyncio<6>. 
* binascii: binascii<2>. 
* cmath:: 
* collections: collections<4>. 
* concurrent.futures: concurrent futures<3>. 
* contextlib: contextlib<3>. 
* datetime: datetime<4>. 
* decimal: decimal<2>. 
* distutils: distutils<4>. 
* email:: 
* encodings: encodings<2>. 
* enum: enum<4>. 
* faulthandler: faulthandler<2>. 
* fileinput: fileinput<2>. 
* hashlib: hashlib<3>. 
* http.client: http client<2>. 
* idlelib and IDLE: idlelib and IDLE<2>. 
* importlib: importlib<4>. 
* inspect: inspect<5>. 
* json:: 
* logging: logging<3>. 
* math: math<5>. 
* multiprocessing: multiprocessing<4>. 
* os: os<6>. 
* pathlib: pathlib<5>. 
* pdb: pdb<3>. 
* pickle: pickle<2>. 
* pickletools:: 
* pydoc: pydoc<3>. 
* random: random<2>. 
* re: re<2>. 
* readline:: 
* rlcompleter:: 
* shlex: shlex<2>. 
* site: site<2>. 
* sqlite3: sqlite3<4>. 
* socket: socket<7>. 
* socketserver: socketserver<2>. 
* ssl: ssl<5>. 
* statistics: statistics<3>. 
* struct:: 
* subprocess: subprocess<2>. 
* sys: sys<7>. 
* telnetlib:: 
* time: time<5>. 
* timeit:: 
* tkinter: tkinter<3>. 
* traceback: traceback<2>. 
* tracemalloc: tracemalloc<3>. 
* typing: typing<5>. 
* unicodedata: unicodedata<5>. 
* unittest.mock: unittest mock<2>. 
* urllib.request: urllib request. 
* urllib.robotparser: urllib robotparser. 
* venv: venv<3>. 
* warnings: warnings<2>. 
* winreg:: 
* winsound:: 
* xmlrpc.client: xmlrpc client. 
* zipfile: zipfile<2>. 
* zlib:: 

Deprecated

* New Keywords:: 
* Deprecated Python behavior:: 
* Deprecated Python modules@comma{} functions and methods: Deprecated Python modules functions and methods<2>. 
* Deprecated functions and types of the C API: Deprecated functions and types of the C API<2>. 
* Deprecated Build Options:: 

Deprecated Python modules, functions and methods

* asynchat:: 
* asyncore:: 
* dbm: dbm<3>. 
* distutils: distutils<5>. 
* grp:: 
* importlib: importlib<5>. 
* os: os<7>. 
* re: re<3>. 
* ssl: ssl<6>. 
* tkinter: tkinter<4>. 
* venv: venv<4>. 

Removed

* API and Feature Removals: API and Feature Removals<3>. 

Porting to Python 3.6

* Changes in ‘python’ Command Behavior:: 
* Changes in the Python API: Changes in the Python API<6>. 
* Changes in the C API: Changes in the C API<5>. 
* CPython bytecode changes: CPython bytecode changes<6>. 

Notable changes in Python 3.6.2

* New make regen-all build target:: 
* Removal of make touch build target:: 

What’s New In Python 3.5

* Summary – Release highlights: Summary – Release highlights<6>. 
* New Features: New Features<10>. 
* Other Language Changes: Other Language Changes<7>. 
* New Modules: New Modules<7>. 
* Improved Modules: Improved Modules<7>. 
* Other module-level changes:: 
* Optimizations: Optimizations<7>. 
* Build and C API Changes: Build and C API Changes<3>. 
* Deprecated: Deprecated<8>. 
* Removed: Removed<8>. 
* Porting to Python 3.5: Porting to Python 3 5. 
* Notable changes in Python 3.5.4: Notable changes in Python 3 5 4. 

New Features

* PEP 492 - Coroutines with async and await syntax:: 
* PEP 465 - A dedicated infix operator for matrix multiplication:: 
* PEP 448 - Additional Unpacking Generalizations:: 
* PEP 461 - percent formatting support for bytes and bytearray:: 
* PEP 484 - Type Hints:: 
* PEP 471 - os.scandir() function – a better and faster directory iterator: PEP 471 - os scandir function – a better and faster directory iterator. 
* PEP 475; Retry system calls failing with EINTR: PEP 475 Retry system calls failing with EINTR. 
* PEP 479; Change StopIteration handling inside generators: PEP 479 Change StopIteration handling inside generators. 
* PEP 485; A function for testing approximate equality: PEP 485 A function for testing approximate equality. 
* PEP 486; Make the Python Launcher aware of virtual environments: PEP 486 Make the Python Launcher aware of virtual environments. 
* PEP 488; Elimination of PYO files: PEP 488 Elimination of PYO files. 
* PEP 489; Multi-phase extension module initialization: PEP 489 Multi-phase extension module initialization. 

New Modules

* typing: typing<6>. 
* zipapp: zipapp<2>. 

Improved Modules

* argparse: argparse<3>. 
* asyncio: asyncio<7>. 
* bz2:: 
* cgi:: 
* cmath: cmath<2>. 
* code:: 
* collections: collections<5>. 
* collections.abc: collections abc<3>. 
* compileall: compileall<3>. 
* concurrent.futures: concurrent futures<4>. 
* configparser:: 
* contextlib: contextlib<4>. 
* csv: csv<2>. 
* curses: curses<4>. 
* dbm: dbm<4>. 
* difflib:: 
* distutils: distutils<6>. 
* doctest: doctest<2>. 
* email: email<2>. 
* enum: enum<5>. 
* faulthandler: faulthandler<3>. 
* functools: functools<3>. 
* glob: glob<2>. 
* gzip: gzip<2>. 
* heapq:: 
* http: http<2>. 
* http.client: http client<3>. 
* idlelib and IDLE: idlelib and IDLE<3>. 
* imaplib: imaplib<2>. 
* imghdr:: 
* importlib: importlib<6>. 
* inspect: inspect<6>. 
* io: io<3>. 
* ipaddress: ipaddress<3>. 
* json: json<2>. 
* linecache: linecache<2>. 
* locale: locale<3>. 
* logging: logging<4>. 
* lzma:: 
* math: math<6>. 
* multiprocessing: multiprocessing<5>. 
* operator: operator<2>. 
* os: os<8>. 
* pathlib: pathlib<6>. 
* pickle: pickle<3>. 
* poplib: poplib<2>. 
* re: re<4>. 
* readline: readline<2>. 
* selectors:: 
* shutil: shutil<2>. 
* signal: signal<3>. 
* smtpd:: 
* smtplib: smtplib<2>. 
* sndhdr:: 
* socket: socket<8>. 
* ssl: ssl<7>. 
* sqlite3: sqlite3<5>. 
* subprocess: subprocess<3>. 
* sys: sys<8>. 
* sysconfig:: 
* tarfile: tarfile<2>. 
* threading: threading<5>. 
* time: time<6>. 
* timeit: timeit<2>. 
* tkinter: tkinter<5>. 
* traceback: traceback<3>. 
* types: types<3>. 
* unicodedata: unicodedata<6>. 
* unittest: unittest<4>. 
* unittest.mock: unittest mock<3>. 
* urllib:: 
* wsgiref:: 
* xmlrpc: xmlrpc<2>. 
* xml.sax: xml sax. 
* zipfile: zipfile<3>. 

ssl

* Memory BIO Support:: 
* Application-Layer Protocol Negotiation Support:: 
* Other Changes:: 

Deprecated

* New Keywords: New Keywords<2>. 
* Deprecated Python Behavior: Deprecated Python Behavior<2>. 
* Unsupported Operating Systems:: 
* Deprecated Python modules@comma{} functions and methods: Deprecated Python modules functions and methods<3>. 

Removed

* API and Feature Removals: API and Feature Removals<4>. 

Porting to Python 3.5

* Changes in Python behavior: Changes in Python behavior<2>. 
* Changes in the Python API: Changes in the Python API<7>. 
* Changes in the C API: Changes in the C API<6>. 

Notable changes in Python 3.5.4

* New make regen-all build target: New make regen-all build target<2>. 
* Removal of make touch build target: Removal of make touch build target<2>. 

What’s New In Python 3.4

* Summary – Release Highlights: Summary – Release Highlights<2>. 
* New Features: New Features<11>. 
* New Modules: New Modules<8>. 
* Improved Modules: Improved Modules<8>. 
* CPython Implementation Changes:: 
* Deprecated: Deprecated<9>. 
* Removed: Removed<9>. 
* Porting to Python 3.4: Porting to Python 3 4. 
* Changed in 3.4.3: Changed in 3 4 3. 

New Features

* PEP 453; Explicit Bootstrapping of PIP in Python Installations: PEP 453 Explicit Bootstrapping of PIP in Python Installations. 
* PEP 446; Newly Created File Descriptors Are Non-Inheritable: PEP 446 Newly Created File Descriptors Are Non-Inheritable. 
* Improvements to Codec Handling:: 
* PEP 451; A ModuleSpec Type for the Import System: PEP 451 A ModuleSpec Type for the Import System. 
* Other Language Changes: Other Language Changes<8>. 

PEP 453: Explicit Bootstrapping of PIP in Python Installations

* Bootstrapping pip By Default:: 
* Documentation Changes:: 

New Modules

* asyncio: asyncio<8>. 
* ensurepip:: 
* enum: enum<6>. 
* pathlib: pathlib<7>. 
* selectors: selectors<2>. 
* statistics: statistics<4>. 
* tracemalloc: tracemalloc<4>. 

Improved Modules

* abc:: 
* aifc: aifc<2>. 
* argparse: argparse<4>. 
* audioop:: 
* base64: base64<2>. 
* collections: collections<6>. 
* colorsys:: 
* contextlib: contextlib<5>. 
* dbm: dbm<5>. 
* dis: dis<2>. 
* doctest: doctest<3>. 
* email: email<3>. 
* filecmp:: 
* functools: functools<4>. 
* gc: gc<5>. 
* glob: glob<3>. 
* hashlib: hashlib<4>. 
* hmac: hmac<3>. 
* html:: 
* http: http<3>. 
* idlelib and IDLE: idlelib and IDLE<4>. 
* importlib: importlib<7>. 
* inspect: inspect<7>. 
* ipaddress: ipaddress<4>. 
* logging: logging<5>. 
* marshal:: 
* mmap: mmap<2>. 
* multiprocessing: multiprocessing<6>. 
* operator: operator<3>. 
* os: os<9>. 
* pdb: pdb<4>. 
* pickle: pickle<4>. 
* plistlib: plistlib<2>. 
* poplib: poplib<3>. 
* pprint: pprint<4>. 
* pty:: 
* pydoc: pydoc<4>. 
* re: re<5>. 
* resource:: 
* select:: 
* shelve: shelve<2>. 
* shutil: shutil<3>. 
* smtpd: smtpd<2>. 
* smtplib: smtplib<3>. 
* socket: socket<9>. 
* sqlite3: sqlite3<6>. 
* ssl: ssl<8>. 
* stat:: 
* struct: struct<2>. 
* subprocess: subprocess<4>. 
* sunau: sunau<2>. 
* sys: sys<9>. 
* tarfile: tarfile<3>. 
* textwrap:: 
* threading: threading<6>. 
* traceback: traceback<4>. 
* types: types<4>. 
* urllib: urllib<2>. 
* unittest: unittest<5>. 
* venv: venv<5>. 
* wave: wave<2>. 
* weakref: weakref<2>. 
* xml.etree: xml etree<2>. 
* zipfile: zipfile<4>. 

CPython Implementation Changes

* PEP 445; Customization of CPython Memory Allocators: PEP 445 Customization of CPython Memory Allocators. 
* PEP 442; Safe Object Finalization: PEP 442 Safe Object Finalization. 
* PEP 456; Secure and Interchangeable Hash Algorithm: PEP 456 Secure and Interchangeable Hash Algorithm. 
* PEP 436; Argument Clinic: PEP 436 Argument Clinic. 
* Other Build and C API Changes:: 
* Other Improvements: Other Improvements<2>. 
* Significant Optimizations:: 

Deprecated

* Deprecations in the Python API:: 
* Deprecated Features:: 

Removed

* Operating Systems No Longer Supported:: 
* API and Feature Removals: API and Feature Removals<5>. 
* Code Cleanups:: 

Porting to Python 3.4

* Changes in ‘python’ Command Behavior: Changes in ‘python’ Command Behavior<2>. 
* Changes in the Python API: Changes in the Python API<8>. 
* Changes in the C API: Changes in the C API<7>. 

Changed in 3.4.3

* PEP 476; Enabling certificate verification by default for stdlib http clients: PEP 476 Enabling certificate verification by default for stdlib http clients. 

What’s New In Python 3.3

* Summary – Release highlights: Summary – Release highlights<7>. 
* PEP 405; Virtual Environments: PEP 405 Virtual Environments. 
* PEP 420; Implicit Namespace Packages: PEP 420 Implicit Namespace Packages. 
* PEP 3118; New memoryview implementation and buffer protocol documentation: PEP 3118 New memoryview implementation and buffer protocol documentation. 
* PEP 393; Flexible String Representation: PEP 393 Flexible String Representation. 
* PEP 397; Python Launcher for Windows: PEP 397 Python Launcher for Windows. 
* PEP 3151; Reworking the OS and IO exception hierarchy: PEP 3151 Reworking the OS and IO exception hierarchy. 
* PEP 380; Syntax for Delegating to a Subgenerator: PEP 380 Syntax for Delegating to a Subgenerator. 
* PEP 409; Suppressing exception context: PEP 409 Suppressing exception context. 
* PEP 414; Explicit Unicode literals: PEP 414 Explicit Unicode literals. 
* PEP 3155; Qualified name for classes and functions: PEP 3155 Qualified name for classes and functions. 
* PEP 412; Key-Sharing Dictionary: PEP 412 Key-Sharing Dictionary. 
* PEP 362; Function Signature Object: PEP 362 Function Signature Object. 
* PEP 421; Adding sys.implementation: PEP 421 Adding sys implementation. 
* Using importlib as the Implementation of Import:: 
* Other Language Changes: Other Language Changes<9>. 
* A Finer-Grained Import Lock:: 
* Builtin functions and types:: 
* New Modules: New Modules<9>. 
* Improved Modules: Improved Modules<9>. 
* Optimizations: Optimizations<8>. 
* Build and C API Changes: Build and C API Changes<4>. 
* Deprecated: Deprecated<10>. 
* Porting to Python 3.3: Porting to Python 3 3. 

PEP 3118: New memoryview implementation and buffer protocol documentation

* Features:: 
* API changes:: 

PEP 393: Flexible String Representation

* Functionality:: 
* Performance and resource usage:: 

PEP 421: Adding sys.implementation

* SimpleNamespace:: 

Using importlib as the Implementation of Import

* New APIs:: 
* Visible Changes:: 

New Modules

* faulthandler: faulthandler<4>. 
* ipaddress: ipaddress<5>. 
* lzma: lzma<2>. 

Improved Modules

* abc: abc<2>. 
* array: array<3>. 
* base64: base64<3>. 
* binascii: binascii<3>. 
* bz2: bz2<2>. 
* codecs: codecs<2>. 
* collections: collections<7>. 
* contextlib: contextlib<6>. 
* crypt: crypt<2>. 
* curses: curses<5>. 
* datetime: datetime<5>. 
* decimal: decimal<3>. 
* email: email<4>. 
* ftplib: ftplib<2>. 
* functools: functools<5>. 
* gc: gc<6>. 
* hmac: hmac<4>. 
* http: http<4>. 
* html: html<2>. 
* imaplib: imaplib<3>. 
* inspect: inspect<8>. 
* io: io<4>. 
* itertools: itertools<3>. 
* logging: logging<6>. 
* math: math<7>. 
* mmap: mmap<3>. 
* multiprocessing: multiprocessing<7>. 
* nntplib: nntplib<2>. 
* os: os<10>. 
* pdb: pdb<5>. 
* pickle: pickle<5>. 
* pydoc: pydoc<5>. 
* re: re<6>. 
* sched:: 
* select: select<2>. 
* shlex: shlex<3>. 
* shutil: shutil<4>. 
* signal: signal<4>. 
* smtpd: smtpd<3>. 
* smtplib: smtplib<4>. 
* socket: socket<10>. 
* socketserver: socketserver<3>. 
* sqlite3: sqlite3<7>. 
* ssl: ssl<9>. 
* stat: stat<2>. 
* struct: struct<3>. 
* subprocess: subprocess<5>. 
* sys: sys<10>. 
* tarfile: tarfile<4>. 
* tempfile:: 
* textwrap: textwrap<2>. 
* threading: threading<7>. 
* time: time<7>. 
* types: types<5>. 
* unittest: unittest<6>. 
* urllib: urllib<3>. 
* webbrowser:: 
* xml.etree.ElementTree: xml etree ElementTree. 
* zlib: zlib<2>. 

decimal

* Features: Features<2>. 
* API changes: API changes<2>. 

email

* Policy Framework:: 
* Provisional Policy with New Header API:: 
* Other API Changes:: 

Deprecated

* Unsupported Operating Systems: Unsupported Operating Systems<2>. 
* Deprecated Python modules@comma{} functions and methods: Deprecated Python modules functions and methods<4>. 
* Deprecated functions and types of the C API: Deprecated functions and types of the C API<3>. 
* Deprecated features:: 

Porting to Python 3.3

* Porting Python code:: 
* Porting C code:: 
* Building C extensions:: 
* Command Line Switch Changes:: 

What’s New In Python 3.2

* PEP 384; Defining a Stable ABI: PEP 384 Defining a Stable ABI. 
* PEP 389; Argparse Command Line Parsing Module: PEP 389 Argparse Command Line Parsing Module. 
* PEP 391; Dictionary Based Configuration for Logging: PEP 391 Dictionary Based Configuration for Logging. 
* PEP 3148; The concurrent.futures module: PEP 3148 The concurrent futures module. 
* PEP 3147; PYC Repository Directories: PEP 3147 PYC Repository Directories. 
* PEP 3149; ABI Version Tagged .so Files: PEP 3149 ABI Version Tagged so Files. 
* PEP 3333; Python Web Server Gateway Interface v1.0.1: PEP 3333 Python Web Server Gateway Interface v1 0 1. 
* Other Language Changes: Other Language Changes<10>. 
* New@comma{} Improved@comma{} and Deprecated Modules: New Improved and Deprecated Modules. 
* Multi-threading:: 
* Optimizations: Optimizations<9>. 
* Unicode:: 
* Codecs:: 
* Documentation:: 
* IDLE:: 
* Code Repository:: 
* Build and C API Changes: Build and C API Changes<5>. 
* Porting to Python 3.2: Porting to Python 3 2. 

New, Improved, and Deprecated Modules

* email: email<5>. 
* elementtree:: 
* functools: functools<6>. 
* itertools: itertools<4>. 
* collections: collections<8>. 
* threading: threading<8>. 
* datetime and time:: 
* math: math<8>. 
* abc: abc<3>. 
* io: io<5>. 
* reprlib:: 
* logging: logging<7>. 
* csv: csv<3>. 
* contextlib: contextlib<7>. 
* decimal and fractions:: 
* ftp:: 
* popen:: 
* select: select<3>. 
* gzip and zipfile:: 
* tarfile: tarfile<5>. 
* hashlib: hashlib<5>. 
* ast: ast<4>. 
* os: os<11>. 
* shutil: shutil<5>. 
* sqlite3: sqlite3<8>. 
* html: html<3>. 
* socket: socket<11>. 
* ssl: ssl<10>. 
* nntp:: 
* certificates:: 
* imaplib: imaplib<4>. 
* http.client: http client<4>. 
* unittest: unittest<7>. 
* random: random<3>. 
* poplib: poplib<4>. 
* asyncore: asyncore<2>. 
* tempfile: tempfile<2>. 
* inspect: inspect<9>. 
* pydoc: pydoc<6>. 
* dis: dis<3>. 
* dbm: dbm<6>. 
* ctypes: ctypes<2>. 
* site: site<3>. 
* sysconfig: sysconfig<2>. 
* pdb: pdb<6>. 
* configparser: configparser<2>. 
* urllib.parse: urllib parse<4>. 
* mailbox:: 
* turtledemo:: 

What’s New In Python 3.1

* PEP 372; Ordered Dictionaries: PEP 372 Ordered Dictionaries. 
* PEP 378; Format Specifier for Thousands Separator: PEP 378 Format Specifier for Thousands Separator. 
* Other Language Changes: Other Language Changes<11>. 
* New@comma{} Improved@comma{} and Deprecated Modules: New Improved and Deprecated Modules<2>. 
* Optimizations: Optimizations<10>. 
* IDLE: IDLE<2>. 
* Build and C API Changes: Build and C API Changes<6>. 
* Porting to Python 3.1: Porting to Python 3 1. 

What’s New In Python 3.0

* Common Stumbling Blocks:: 
* Overview Of Syntax Changes:: 
* Changes Already Present In Python 2.6: Changes Already Present In Python 2 6. 
* Library Changes:: 
* PEP 3101; A New Approach To String Formatting: PEP 3101 A New Approach To String Formatting. 
* Changes To Exceptions:: 
* Miscellaneous Other Changes:: 
* Build and C API Changes: Build and C API Changes<7>. 
* Performance:: 
* Porting To Python 3.0: Porting To Python 3 0. 

Common Stumbling Blocks

* Print Is A Function:: 
* Views And Iterators Instead Of Lists:: 
* Ordering Comparisons:: 
* Integers:: 
* Text Vs. Data Instead Of Unicode Vs. 8-bit: Text Vs Data Instead Of Unicode Vs 8-bit. 

Overview Of Syntax Changes

* New Syntax:: 
* Changed Syntax:: 
* Removed Syntax:: 

Miscellaneous Other Changes

* Operators And Special Methods:: 
* Builtins:: 

What’s New in Python 2.7

* The Future for Python 2.x: The Future for Python 2 x. 
* Changes to the Handling of Deprecation Warnings:: 
* Python 3.1 Features: Python 3 1 Features. 
* PEP 372; Adding an Ordered Dictionary to collections: PEP 372 Adding an Ordered Dictionary to collections. 
* PEP 378; Format Specifier for Thousands Separator: PEP 378 Format Specifier for Thousands Separator<2>. 
* PEP 389; The argparse Module for Parsing Command Lines: PEP 389 The argparse Module for Parsing Command Lines. 
* PEP 391; Dictionary-Based Configuration For Logging: PEP 391 Dictionary-Based Configuration For Logging. 
* PEP 3106; Dictionary Views: PEP 3106 Dictionary Views. 
* PEP 3137; The memoryview Object: PEP 3137 The memoryview Object. 
* Other Language Changes: Other Language Changes<12>. 
* New and Improved Modules:: 
* Build and C API Changes: Build and C API Changes<8>. 
* Other Changes and Fixes:: 
* Porting to Python 2.7: Porting to Python 2 7. 
* New Features Added to Python 2.7 Maintenance Releases: New Features Added to Python 2 7 Maintenance Releases. 
* Acknowledgements:: 

Other Language Changes

* Interpreter Changes:: 
* Optimizations: Optimizations<11>. 

New and Improved Modules

* New module; importlib: New module importlib. 
* New module; sysconfig: New module sysconfig. 
* ttk; Themed Widgets for Tk: ttk Themed Widgets for Tk. 
* Updated module; unittest: Updated module unittest. 
* Updated module; ElementTree 1.3: Updated module ElementTree 1 3. 

Build and C API Changes

* Capsules:: 
* Port-Specific Changes; Windows: Port-Specific Changes Windows. 
* Port-Specific Changes; Mac OS X: Port-Specific Changes Mac OS X. 
* Port-Specific Changes; FreeBSD: Port-Specific Changes FreeBSD. 

New Features Added to Python 2.7 Maintenance Releases

* Two new environment variables for debug mode:: 
* PEP 434; IDLE Enhancement Exception for All Branches: PEP 434 IDLE Enhancement Exception for All Branches. 
* PEP 466; Network Security Enhancements for Python 2.7: PEP 466 Network Security Enhancements for Python 2 7. 
* PEP 477; Backport ensurepip (PEP 453) to Python 2.7: PEP 477 Backport ensurepip PEP 453 to Python 2 7. 
* PEP 476; Enabling certificate verification by default for stdlib http clients: PEP 476 Enabling certificate verification by default for stdlib http clients<2>. 
* PEP 493; HTTPS verification migration tools for Python 2.7: PEP 493 HTTPS verification migration tools for Python 2 7. 
* New make regen-all build target: New make regen-all build target<3>. 
* Removal of make touch build target: Removal of make touch build target<3>. 

PEP 477: Backport ensurepip (PEP 453) to Python 2.7

* Bootstrapping pip By Default: Bootstrapping pip By Default<2>. 
* Documentation Changes: Documentation Changes<2>. 

What’s New in Python 2.6

* Python 3.0: Python 3 0. 
* Changes to the Development Process:: 
* PEP 343; The ‘with’ statement: PEP 343 The ‘with’ statement. 
* PEP 366; Explicit Relative Imports From a Main Module: PEP 366 Explicit Relative Imports From a Main Module. 
* PEP 370; Per-user site-packages Directory: PEP 370 Per-user site-packages Directory. 
* PEP 371; The multiprocessing Package: PEP 371 The multiprocessing Package. 
* PEP 3101; Advanced String Formatting: PEP 3101 Advanced String Formatting. 
* PEP 3105; print As a Function: PEP 3105 print As a Function. 
* PEP 3110; Exception-Handling Changes: PEP 3110 Exception-Handling Changes. 
* PEP 3112; Byte Literals: PEP 3112 Byte Literals. 
* PEP 3116; New I/O Library: PEP 3116 New I/O Library. 
* PEP 3118; Revised Buffer Protocol: PEP 3118 Revised Buffer Protocol. 
* PEP 3119; Abstract Base Classes: PEP 3119 Abstract Base Classes. 
* PEP 3127; Integer Literal Support and Syntax: PEP 3127 Integer Literal Support and Syntax. 
* PEP 3129; Class Decorators: PEP 3129 Class Decorators. 
* PEP 3141; A Type Hierarchy for Numbers: PEP 3141 A Type Hierarchy for Numbers. 
* Other Language Changes: Other Language Changes<13>. 
* New and Improved Modules: New and Improved Modules<2>. 
* Deprecations and Removals:: 
* Build and C API Changes: Build and C API Changes<9>. 
* Porting to Python 2.6: Porting to Python 2 6. 
* Acknowledgements: Acknowledgements<2>. 

Changes to the Development Process

* New Issue Tracker; Roundup: New Issue Tracker Roundup. 
* New Documentation Format; reStructuredText Using Sphinx: New Documentation Format reStructuredText Using Sphinx. 

PEP 343: The ‘with’ statement

* Writing Context Managers:: 
* The contextlib module:: 

PEP 3141: A Type Hierarchy for Numbers

* The fractions Module:: 

Other Language Changes

* Optimizations: Optimizations<12>. 
* Interpreter Changes: Interpreter Changes<2>. 

New and Improved Modules

* The ast module:: 
* The future_builtins module:: 
* The json module; JavaScript Object Notation: The json module JavaScript Object Notation. 
* The plistlib module; A Property-List Parser: The plistlib module A Property-List Parser. 
* ctypes Enhancements:: 
* Improved SSL Support:: 

Build and C API Changes

* Port-Specific Changes; Windows: Port-Specific Changes Windows<2>. 
* Port-Specific Changes; Mac OS X: Port-Specific Changes Mac OS X<2>. 
* Port-Specific Changes; IRIX: Port-Specific Changes IRIX. 

What’s New in Python 2.5

* PEP 308; Conditional Expressions: PEP 308 Conditional Expressions. 
* PEP 309; Partial Function Application: PEP 309 Partial Function Application. 
* PEP 314; Metadata for Python Software Packages v1.1: PEP 314 Metadata for Python Software Packages v1 1. 
* PEP 328; Absolute and Relative Imports: PEP 328 Absolute and Relative Imports. 
* PEP 338; Executing Modules as Scripts: PEP 338 Executing Modules as Scripts. 
* PEP 341; Unified try/except/finally: PEP 341 Unified try/except/finally. 
* PEP 342; New Generator Features: PEP 342 New Generator Features. 
* PEP 343; The ‘with’ statement: PEP 343 The ‘with’ statement<2>. 
* PEP 352; Exceptions as New-Style Classes: PEP 352 Exceptions as New-Style Classes. 
* PEP 353; Using ssize_t as the index type: PEP 353 Using ssize_t as the index type. 
* PEP 357; The ‘__index__’ method: PEP 357 The ‘__index__’ method. 
* Other Language Changes: Other Language Changes<14>. 
* New@comma{} Improved@comma{} and Removed Modules: New Improved and Removed Modules. 
* Build and C API Changes: Build and C API Changes<10>. 
* Porting to Python 2.5: Porting to Python 2 5. 
* Acknowledgements: Acknowledgements<3>. 

PEP 343: The ‘with’ statement

* Writing Context Managers: Writing Context Managers<2>. 
* The contextlib module: The contextlib module<2>. 

Other Language Changes

* Interactive Interpreter Changes:: 
* Optimizations: Optimizations<13>. 

New, Improved, and Removed Modules

* The ctypes package:: 
* The ElementTree package:: 
* The hashlib package:: 
* The sqlite3 package:: 
* The wsgiref package:: 

Build and C API Changes

* Port-Specific Changes:: 

What’s New in Python 2.4

* PEP 218; Built-In Set Objects: PEP 218 Built-In Set Objects. 
* PEP 237; Unifying Long Integers and Integers: PEP 237 Unifying Long Integers and Integers. 
* PEP 289; Generator Expressions: PEP 289 Generator Expressions. 
* PEP 292; Simpler String Substitutions: PEP 292 Simpler String Substitutions. 
* PEP 318; Decorators for Functions and Methods: PEP 318 Decorators for Functions and Methods. 
* PEP 322; Reverse Iteration: PEP 322 Reverse Iteration. 
* PEP 324; New subprocess Module: PEP 324 New subprocess Module. 
* PEP 327; Decimal Data Type: PEP 327 Decimal Data Type. 
* PEP 328; Multi-line Imports: PEP 328 Multi-line Imports. 
* PEP 331; Locale-Independent Float/String Conversions: PEP 331 Locale-Independent Float/String Conversions. 
* Other Language Changes: Other Language Changes<15>. 
* New@comma{} Improved@comma{} and Deprecated Modules: New Improved and Deprecated Modules<3>. 
* Build and C API Changes: Build and C API Changes<11>. 
* Porting to Python 2.4: Porting to Python 2 4. 
* Acknowledgements: Acknowledgements<4>. 

PEP 327: Decimal Data Type

* Why is Decimal needed?:: 
* The Decimal type:: 
* The Context type:: 

Other Language Changes

* Optimizations: Optimizations<14>. 

New, Improved, and Deprecated Modules

* cookielib:: 
* doctest: doctest<4>. 

Build and C API Changes

* Port-Specific Changes: Port-Specific Changes<2>. 

What’s New in Python 2.3

* PEP 218; A Standard Set Datatype: PEP 218 A Standard Set Datatype. 
* PEP 255; Simple Generators: PEP 255 Simple Generators. 
* PEP 263; Source Code Encodings: PEP 263 Source Code Encodings. 
* PEP 273; Importing Modules from ZIP Archives: PEP 273 Importing Modules from ZIP Archives. 
* PEP 277; Unicode file name support for Windows NT: PEP 277 Unicode file name support for Windows NT. 
* PEP 278; Universal Newline Support: PEP 278 Universal Newline Support. 
* PEP 279; enumerate(): PEP 279 enumerate. 
* PEP 282; The logging Package: PEP 282 The logging Package. 
* PEP 285; A Boolean Type: PEP 285 A Boolean Type. 
* PEP 293; Codec Error Handling Callbacks: PEP 293 Codec Error Handling Callbacks. 
* PEP 301; Package Index and Metadata for Distutils: PEP 301 Package Index and Metadata for Distutils. 
* PEP 302; New Import Hooks: PEP 302 New Import Hooks. 
* PEP 305; Comma-separated Files: PEP 305 Comma-separated Files. 
* PEP 307; Pickle Enhancements: PEP 307 Pickle Enhancements. 
* Extended Slices:: 
* Other Language Changes: Other Language Changes<16>. 
* New@comma{} Improved@comma{} and Deprecated Modules: New Improved and Deprecated Modules<4>. 
* Pymalloc; A Specialized Object Allocator: Pymalloc A Specialized Object Allocator. 
* Build and C API Changes: Build and C API Changes<12>. 
* Other Changes and Fixes: Other Changes and Fixes<2>. 
* Porting to Python 2.3: Porting to Python 2 3. 
* Acknowledgements: Acknowledgements<5>. 

Other Language Changes

* String Changes:: 
* Optimizations: Optimizations<15>. 

New, Improved, and Deprecated Modules

* Date/Time Type:: 
* The optparse Module:: 

Build and C API Changes

* Port-Specific Changes: Port-Specific Changes<3>. 

What’s New in Python 2.2

* Introduction:: 
* PEPs 252 and 253; Type and Class Changes: PEPs 252 and 253 Type and Class Changes. 
* PEP 234; Iterators: PEP 234 Iterators. 
* PEP 255; Simple Generators: PEP 255 Simple Generators<2>. 
* PEP 237; Unifying Long Integers and Integers: PEP 237 Unifying Long Integers and Integers<2>. 
* PEP 238; Changing the Division Operator: PEP 238 Changing the Division Operator. 
* Unicode Changes:: 
* PEP 227; Nested Scopes: PEP 227 Nested Scopes. 
* New and Improved Modules: New and Improved Modules<3>. 
* Interpreter Changes and Fixes:: 
* Other Changes and Fixes: Other Changes and Fixes<3>. 
* Acknowledgements: Acknowledgements<6>. 

PEPs 252 and 253: Type and Class Changes

* Old and New Classes:: 
* Descriptors:: 
* Multiple Inheritance; The Diamond Rule: Multiple Inheritance The Diamond Rule. 
* Attribute Access:: 
* Related Links:: 

What’s New in Python 2.1

* Introduction: Introduction<2>. 
* PEP 227; Nested Scopes: PEP 227 Nested Scopes<2>. 
* PEP 236; __future__ Directives: PEP 236 __future__ Directives. 
* PEP 207; Rich Comparisons: PEP 207 Rich Comparisons. 
* PEP 230; Warning Framework: PEP 230 Warning Framework. 
* PEP 229; New Build System: PEP 229 New Build System. 
* PEP 205; Weak References: PEP 205 Weak References. 
* PEP 232; Function Attributes: PEP 232 Function Attributes. 
* PEP 235; Importing Modules on Case-Insensitive Platforms: PEP 235 Importing Modules on Case-Insensitive Platforms. 
* PEP 217; Interactive Display Hook: PEP 217 Interactive Display Hook. 
* PEP 208; New Coercion Model: PEP 208 New Coercion Model. 
* PEP 241; Metadata in Python Packages: PEP 241 Metadata in Python Packages. 
* New and Improved Modules: New and Improved Modules<4>. 
* Other Changes and Fixes: Other Changes and Fixes<4>. 
* Acknowledgements: Acknowledgements<7>. 

What’s New in Python 2.0

* Introduction: Introduction<3>. 
* What About Python 1.6?: What About Python 1 6?. 
* New Development Process:: 
* Unicode: Unicode<2>. 
* List Comprehensions:: 
* Augmented Assignment:: 
* String Methods:: 
* Garbage Collection of Cycles:: 
* Other Core Changes:: 
* Porting to 2.0: Porting to 2 0. 
* Extending/Embedding Changes:: 
* Distutils; Making Modules Easy to Install: Distutils Making Modules Easy to Install. 
* XML Modules:: 
* Module changes:: 
* New modules:: 
* IDLE Improvements:: 
* Deleted and Deprecated Modules:: 
* Acknowledgements: Acknowledgements<8>. 

Other Core Changes

* Minor Language Changes:: 
* Changes to Built-in Functions:: 

XML Modules

* SAX2 Support:: 
* DOM Support:: 
* Relationship to PyXML:: 

Changelog

* Python next:: 
* Python 3.11.0 alpha 5: Python 3 11 0 alpha 5. 
* Python 3.11.0 alpha 4: Python 3 11 0 alpha 4. 
* Python 3.11.0 alpha 3: Python 3 11 0 alpha 3. 
* Python 3.11.0 alpha 2: Python 3 11 0 alpha 2. 
* Python 3.11.0 alpha 1: Python 3 11 0 alpha 1. 
* Python 3.10.0 beta 1: Python 3 10 0 beta 1. 
* Python 3.10.0 alpha 7: Python 3 10 0 alpha 7. 
* Python 3.10.0 alpha 6: Python 3 10 0 alpha 6. 
* Python 3.10.0 alpha 5: Python 3 10 0 alpha 5. 
* Python 3.10.0 alpha 4: Python 3 10 0 alpha 4. 
* Python 3.10.0 alpha 3: Python 3 10 0 alpha 3. 
* Python 3.10.0 alpha 2: Python 3 10 0 alpha 2. 
* Python 3.10.0 alpha 1: Python 3 10 0 alpha 1. 
* Python 3.9.0 beta 1: Python 3 9 0 beta 1. 
* Python 3.9.0 alpha 6: Python 3 9 0 alpha 6. 
* Python 3.9.0 alpha 5: Python 3 9 0 alpha 5. 
* Python 3.9.0 alpha 4: Python 3 9 0 alpha 4. 
* Python 3.9.0 alpha 3: Python 3 9 0 alpha 3. 
* Python 3.9.0 alpha 2: Python 3 9 0 alpha 2. 
* Python 3.9.0 alpha 1: Python 3 9 0 alpha 1. 
* Python 3.8.0 beta 1: Python 3 8 0 beta 1. 
* Python 3.8.0 alpha 4: Python 3 8 0 alpha 4. 
* Python 3.8.0 alpha 3: Python 3 8 0 alpha 3. 
* Python 3.8.0 alpha 2: Python 3 8 0 alpha 2. 
* Python 3.8.0 alpha 1: Python 3 8 0 alpha 1. 
* Python 3.7.0 final: Python 3 7 0 final. 
* Python 3.7.0 release candidate 1: Python 3 7 0 release candidate 1. 
* Python 3.7.0 beta 5: Python 3 7 0 beta 5. 
* Python 3.7.0 beta 4: Python 3 7 0 beta 4. 
* Python 3.7.0 beta 3: Python 3 7 0 beta 3. 
* Python 3.7.0 beta 2: Python 3 7 0 beta 2. 
* Python 3.7.0 beta 1: Python 3 7 0 beta 1. 
* Python 3.7.0 alpha 4: Python 3 7 0 alpha 4. 
* Python 3.7.0 alpha 3: Python 3 7 0 alpha 3. 
* Python 3.7.0 alpha 2: Python 3 7 0 alpha 2. 
* Python 3.7.0 alpha 1: Python 3 7 0 alpha 1. 
* Python 3.6.6 final: Python 3 6 6 final. 
* Python 3.6.6 release candidate 1: Python 3 6 6 release candidate 1. 
* Python 3.6.5 final: Python 3 6 5 final. 
* Python 3.6.5 release candidate 1: Python 3 6 5 release candidate 1. 
* Python 3.6.4 final: Python 3 6 4 final. 
* Python 3.6.4 release candidate 1: Python 3 6 4 release candidate 1. 
* Python 3.6.3 final: Python 3 6 3 final. 
* Python 3.6.3 release candidate 1: Python 3 6 3 release candidate 1. 
* Python 3.6.2 final: Python 3 6 2 final. 
* Python 3.6.2 release candidate 2: Python 3 6 2 release candidate 2. 
* Python 3.6.2 release candidate 1: Python 3 6 2 release candidate 1. 
* Python 3.6.1 final: Python 3 6 1 final. 
* Python 3.6.1 release candidate 1: Python 3 6 1 release candidate 1. 
* Python 3.6.0 final: Python 3 6 0 final. 
* Python 3.6.0 release candidate 2: Python 3 6 0 release candidate 2. 
* Python 3.6.0 release candidate 1: Python 3 6 0 release candidate 1. 
* Python 3.6.0 beta 4: Python 3 6 0 beta 4. 
* Python 3.6.0 beta 3: Python 3 6 0 beta 3. 
* Python 3.6.0 beta 2: Python 3 6 0 beta 2. 
* Python 3.6.0 beta 1: Python 3 6 0 beta 1. 
* Python 3.6.0 alpha 4: Python 3 6 0 alpha 4. 
* Python 3.6.0 alpha 3: Python 3 6 0 alpha 3. 
* Python 3.6.0 alpha 2: Python 3 6 0 alpha 2. 
* Python 3.6.0 alpha 1: Python 3 6 0 alpha 1. 
* Python 3.5.5 final: Python 3 5 5 final. 
* Python 3.5.5 release candidate 1: Python 3 5 5 release candidate 1. 
* Python 3.5.4 final: Python 3 5 4 final. 
* Python 3.5.4 release candidate 1: Python 3 5 4 release candidate 1. 
* Python 3.5.3 final: Python 3 5 3 final. 
* Python 3.5.3 release candidate 1: Python 3 5 3 release candidate 1. 
* Python 3.5.2 final: Python 3 5 2 final. 
* Python 3.5.2 release candidate 1: Python 3 5 2 release candidate 1. 
* Python 3.5.1 final: Python 3 5 1 final. 
* Python 3.5.1 release candidate 1: Python 3 5 1 release candidate 1. 
* Python 3.5.0 final: Python 3 5 0 final. 
* Python 3.5.0 release candidate 4: Python 3 5 0 release candidate 4. 
* Python 3.5.0 release candidate 3: Python 3 5 0 release candidate 3. 
* Python 3.5.0 release candidate 2: Python 3 5 0 release candidate 2. 
* Python 3.5.0 release candidate 1: Python 3 5 0 release candidate 1. 
* Python 3.5.0 beta 4: Python 3 5 0 beta 4. 
* Python 3.5.0 beta 3: Python 3 5 0 beta 3. 
* Python 3.5.0 beta 2: Python 3 5 0 beta 2. 
* Python 3.5.0 beta 1: Python 3 5 0 beta 1. 
* Python 3.5.0 alpha 4: Python 3 5 0 alpha 4. 
* Python 3.5.0 alpha 3: Python 3 5 0 alpha 3. 
* Python 3.5.0 alpha 2: Python 3 5 0 alpha 2. 
* Python 3.5.0 alpha 1: Python 3 5 0 alpha 1. 

Python next

* Core and Builtins:: 
* Library:: 
* Documentation: Documentation<2>. 
* Tests:: 
* Build:: 
* Windows:: 
* IDLE: IDLE<3>. 
* C API:: 

Python 3.11.0 alpha 5

* Core and Builtins: Core and Builtins<2>. 
* Library: Library<2>. 
* Documentation: Documentation<3>. 
* Tests: Tests<2>. 
* Build: Build<2>. 
* Windows: Windows<2>. 
* macOS:: 
* IDLE: IDLE<4>. 
* C API: C API<2>. 

Python 3.11.0 alpha 4

* Core and Builtins: Core and Builtins<3>. 
* Library: Library<3>. 
* Documentation: Documentation<4>. 
* Tests: Tests<3>. 
* Build: Build<3>. 
* Windows: Windows<3>. 
* macOS: macOS<2>. 
* C API: C API<3>. 

Python 3.11.0 alpha 3

* Core and Builtins: Core and Builtins<4>. 
* Library: Library<4>. 
* Documentation: Documentation<5>. 
* Tests: Tests<4>. 
* Build: Build<4>. 
* Windows: Windows<4>. 
* macOS: macOS<3>. 
* C API: C API<4>. 

Python 3.11.0 alpha 2

* Core and Builtins: Core and Builtins<5>. 
* Library: Library<5>. 
* Documentation: Documentation<6>. 
* Tests: Tests<5>. 
* Build: Build<5>. 
* Windows: Windows<5>. 
* macOS: macOS<4>. 
* IDLE: IDLE<5>. 
* C API: C API<5>. 

Python 3.11.0 alpha 1

* Security:: 
* Core and Builtins: Core and Builtins<6>. 
* Library: Library<6>. 
* Documentation: Documentation<7>. 
* Tests: Tests<6>. 
* Build: Build<6>. 
* Windows: Windows<6>. 
* macOS: macOS<5>. 
* IDLE: IDLE<6>. 
* Tools/Demos:: 
* C API: C API<6>. 

Python 3.10.0 beta 1

* Security: Security<2>. 
* Core and Builtins: Core and Builtins<7>. 
* Library: Library<7>. 
* Documentation: Documentation<8>. 
* Tests: Tests<7>. 
* Build: Build<7>. 
* Windows: Windows<7>. 
* macOS: macOS<6>. 
* IDLE: IDLE<7>. 
* C API: C API<7>. 

Python 3.10.0 alpha 7

* Security: Security<3>. 
* Core and Builtins: Core and Builtins<8>. 
* Library: Library<8>. 
* Documentation: Documentation<9>. 
* Tests: Tests<8>. 
* Build: Build<8>. 
* Windows: Windows<8>. 
* IDLE: IDLE<8>. 
* C API: C API<8>. 

Python 3.10.0 alpha 6

* Security: Security<4>. 
* Core and Builtins: Core and Builtins<9>. 
* Library: Library<9>. 
* Documentation: Documentation<10>. 
* Tests: Tests<9>. 
* Build: Build<9>. 
* Windows: Windows<9>. 
* macOS: macOS<7>. 
* IDLE: IDLE<9>. 
* C API: C API<9>. 

Python 3.10.0 alpha 5

* Security: Security<5>. 
* Core and Builtins: Core and Builtins<10>. 
* Library: Library<10>. 
* Documentation: Documentation<11>. 
* Tests: Tests<10>. 
* Build: Build<10>. 
* Windows: Windows<10>. 
* macOS: macOS<8>. 
* IDLE: IDLE<10>. 
* C API: C API<10>. 

Python 3.10.0 alpha 4

* Core and Builtins: Core and Builtins<11>. 
* Library: Library<11>. 
* Documentation: Documentation<12>. 
* Tests: Tests<11>. 
* Build: Build<11>. 
* macOS: macOS<9>. 
* Tools/Demos: Tools/Demos<2>. 
* C API: C API<11>. 

Python 3.10.0 alpha 3

* Security: Security<6>. 
* Core and Builtins: Core and Builtins<12>. 
* Library: Library<12>. 
* Documentation: Documentation<13>. 
* Tests: Tests<12>. 
* Build: Build<12>. 
* Windows: Windows<11>. 
* macOS: macOS<10>. 
* IDLE: IDLE<11>. 
* Tools/Demos: Tools/Demos<3>. 
* C API: C API<12>. 

Python 3.10.0 alpha 2

* Security: Security<7>. 
* Core and Builtins: Core and Builtins<13>. 
* Library: Library<13>. 
* Documentation: Documentation<14>. 
* Tests: Tests<13>. 
* Build: Build<13>. 
* Windows: Windows<12>. 
* macOS: macOS<11>. 
* IDLE: IDLE<12>. 
* C API: C API<13>. 

Python 3.10.0 alpha 1

* Security: Security<8>. 
* Core and Builtins: Core and Builtins<14>. 
* Library: Library<14>. 
* Documentation: Documentation<15>. 
* Tests: Tests<14>. 
* Build: Build<14>. 
* Windows: Windows<13>. 
* macOS: macOS<12>. 
* IDLE: IDLE<13>. 
* C API: C API<14>. 

Python 3.9.0 beta 1

* Security: Security<9>. 
* Core and Builtins: Core and Builtins<15>. 
* Library: Library<15>. 
* Documentation: Documentation<16>. 
* Tests: Tests<15>. 
* Build: Build<15>. 
* Windows: Windows<14>. 
* macOS: macOS<13>. 
* Tools/Demos: Tools/Demos<4>. 
* C API: C API<15>. 

Python 3.9.0 alpha 6

* Security: Security<10>. 
* Core and Builtins: Core and Builtins<16>. 
* Library: Library<16>. 
* Documentation: Documentation<17>. 
* Tests: Tests<16>. 
* Build: Build<16>. 
* Windows: Windows<15>. 
* macOS: macOS<14>. 
* IDLE: IDLE<14>. 
* Tools/Demos: Tools/Demos<5>. 
* C API: C API<16>. 

Python 3.9.0 alpha 5

* Security: Security<11>. 
* Core and Builtins: Core and Builtins<17>. 
* Library: Library<17>. 
* Documentation: Documentation<18>. 
* Tests: Tests<17>. 
* Build: Build<17>. 
* Windows: Windows<16>. 
* macOS: macOS<15>. 
* IDLE: IDLE<15>. 
* Tools/Demos: Tools/Demos<6>. 
* C API: C API<17>. 

Python 3.9.0 alpha 4

* Security: Security<12>. 
* Core and Builtins: Core and Builtins<18>. 
* Library: Library<18>. 
* Documentation: Documentation<19>. 
* Tests: Tests<18>. 
* Build: Build<18>. 
* Windows: Windows<17>. 
* IDLE: IDLE<16>. 
* C API: C API<18>. 

Python 3.9.0 alpha 3

* Core and Builtins: Core and Builtins<19>. 
* Library: Library<19>. 
* Documentation: Documentation<20>. 
* Build: Build<19>. 
* IDLE: IDLE<17>. 
* C API: C API<19>. 

Python 3.9.0 alpha 2

* Security: Security<13>. 
* Core and Builtins: Core and Builtins<20>. 
* Library: Library<20>. 
* Documentation: Documentation<21>. 
* Tests: Tests<19>. 
* Build: Build<20>. 
* Windows: Windows<18>. 
* macOS: macOS<16>. 
* IDLE: IDLE<18>. 
* C API: C API<20>. 

Python 3.9.0 alpha 1

* Security: Security<14>. 
* Core and Builtins: Core and Builtins<21>. 
* Library: Library<21>. 
* Documentation: Documentation<22>. 
* Tests: Tests<20>. 
* Build: Build<21>. 
* Windows: Windows<19>. 
* macOS: macOS<17>. 
* IDLE: IDLE<19>. 
* Tools/Demos: Tools/Demos<7>. 
* C API: C API<21>. 

Python 3.8.0 beta 1

* Security: Security<15>. 
* Core and Builtins: Core and Builtins<22>. 
* Library: Library<22>. 
* Documentation: Documentation<23>. 
* Tests: Tests<21>. 
* Build: Build<22>. 
* Windows: Windows<20>. 
* macOS: macOS<18>. 
* IDLE: IDLE<20>. 
* Tools/Demos: Tools/Demos<8>. 
* C API: C API<22>. 

Python 3.8.0 alpha 4

* Security: Security<16>. 
* Core and Builtins: Core and Builtins<23>. 
* Library: Library<23>. 
* Documentation: Documentation<24>. 
* Tests: Tests<22>. 
* Build: Build<23>. 
* Windows: Windows<21>. 
* macOS: macOS<19>. 
* IDLE: IDLE<21>. 
* Tools/Demos: Tools/Demos<9>. 
* C API: C API<23>. 

Python 3.8.0 alpha 3

* Security: Security<17>. 
* Core and Builtins: Core and Builtins<24>. 
* Library: Library<24>. 
* Documentation: Documentation<25>. 
* Tests: Tests<23>. 
* Build: Build<24>. 
* Windows: Windows<22>. 
* IDLE: IDLE<22>. 
* Tools/Demos: Tools/Demos<10>. 
* C API: C API<24>. 

Python 3.8.0 alpha 2

* Core and Builtins: Core and Builtins<25>. 
* Library: Library<25>. 
* Documentation: Documentation<26>. 
* Tests: Tests<24>. 
* Windows: Windows<23>. 
* IDLE: IDLE<23>. 

Python 3.8.0 alpha 1

* Security: Security<18>. 
* Core and Builtins: Core and Builtins<26>. 
* Library: Library<26>. 
* Documentation: Documentation<27>. 
* Tests: Tests<25>. 
* Build: Build<25>. 
* Windows: Windows<24>. 
* macOS: macOS<20>. 
* IDLE: IDLE<24>. 
* Tools/Demos: Tools/Demos<11>. 
* C API: C API<25>. 

Python 3.7.0 final

* Library: Library<27>. 
* C API: C API<26>. 

Python 3.7.0 release candidate 1

* Core and Builtins: Core and Builtins<27>. 
* Library: Library<28>. 
* Documentation: Documentation<28>. 
* Build: Build<26>. 
* Windows: Windows<25>. 
* IDLE: IDLE<25>. 

Python 3.7.0 beta 5

* Core and Builtins: Core and Builtins<28>. 
* Library: Library<29>. 
* Documentation: Documentation<29>. 
* Tests: Tests<26>. 
* Build: Build<27>. 
* macOS: macOS<21>. 
* IDLE: IDLE<26>. 

Python 3.7.0 beta 4

* Core and Builtins: Core and Builtins<29>. 
* Library: Library<30>. 
* Documentation: Documentation<30>. 
* Tests: Tests<27>. 
* Build: Build<28>. 
* Windows: Windows<26>. 
* macOS: macOS<22>. 
* IDLE: IDLE<27>. 
* Tools/Demos: Tools/Demos<12>. 

Python 3.7.0 beta 3

* Security: Security<19>. 
* Core and Builtins: Core and Builtins<30>. 
* Library: Library<31>. 
* Documentation: Documentation<31>. 
* Tests: Tests<28>. 
* Build: Build<29>. 
* Windows: Windows<27>. 
* macOS: macOS<23>. 
* IDLE: IDLE<28>. 
* Tools/Demos: Tools/Demos<13>. 
* C API: C API<27>. 

Python 3.7.0 beta 2

* Security: Security<20>. 
* Core and Builtins: Core and Builtins<31>. 
* Library: Library<32>. 
* Documentation: Documentation<32>. 
* Tests: Tests<29>. 
* Build: Build<30>. 
* Windows: Windows<28>. 
* macOS: macOS<24>. 
* IDLE: IDLE<29>. 
* Tools/Demos: Tools/Demos<14>. 

Python 3.7.0 beta 1

* Core and Builtins: Core and Builtins<32>. 
* Library: Library<33>. 
* Documentation: Documentation<33>. 
* Tests: Tests<30>. 
* Build: Build<31>. 
* Windows: Windows<29>. 
* macOS: macOS<25>. 
* C API: C API<28>. 

Python 3.7.0 alpha 4

* Core and Builtins: Core and Builtins<33>. 
* Library: Library<34>. 
* Documentation: Documentation<34>. 
* Tests: Tests<31>. 
* Windows: Windows<30>. 
* Tools/Demos: Tools/Demos<15>. 
* C API: C API<29>. 

Python 3.7.0 alpha 3

* Core and Builtins: Core and Builtins<34>. 
* Library: Library<35>. 
* Documentation: Documentation<35>. 
* Tests: Tests<32>. 
* Build: Build<32>. 
* Windows: Windows<31>. 
* macOS: macOS<26>. 
* IDLE: IDLE<30>. 
* Tools/Demos: Tools/Demos<16>. 
* C API: C API<30>. 

Python 3.7.0 alpha 2

* Core and Builtins: Core and Builtins<35>. 
* Library: Library<36>. 
* Documentation: Documentation<36>. 
* Build: Build<33>. 
* IDLE: IDLE<31>. 
* C API: C API<31>. 

Python 3.7.0 alpha 1

* Security: Security<21>. 
* Core and Builtins: Core and Builtins<36>. 
* Library: Library<37>. 
* Documentation: Documentation<37>. 
* Tests: Tests<33>. 
* Build: Build<34>. 
* Windows: Windows<32>. 
* IDLE: IDLE<32>. 
* Tools/Demos: Tools/Demos<17>. 
* C API: C API<32>. 

Python 3.6.6 release candidate 1

* Core and Builtins: Core and Builtins<37>. 
* Library: Library<38>. 
* Documentation: Documentation<38>. 
* Tests: Tests<34>. 
* Build: Build<35>. 
* Windows: Windows<33>. 
* macOS: macOS<27>. 
* IDLE: IDLE<33>. 
* Tools/Demos: Tools/Demos<18>. 
* C API: C API<33>. 

Python 3.6.5 final

* Tests: Tests<35>. 
* Build: Build<36>. 

Python 3.6.5 release candidate 1

* Security: Security<22>. 
* Core and Builtins: Core and Builtins<38>. 
* Library: Library<39>. 
* Documentation: Documentation<39>. 
* Tests: Tests<36>. 
* Build: Build<37>. 
* Windows: Windows<34>. 
* macOS: macOS<28>. 
* IDLE: IDLE<34>. 
* Tools/Demos: Tools/Demos<19>. 
* C API: C API<34>. 

Python 3.6.4 release candidate 1

* Core and Builtins: Core and Builtins<39>. 
* Library: Library<40>. 
* Documentation: Documentation<40>. 
* Tests: Tests<37>. 
* Build: Build<38>. 
* Windows: Windows<35>. 
* macOS: macOS<29>. 
* IDLE: IDLE<35>. 
* Tools/Demos: Tools/Demos<20>. 
* C API: C API<35>. 

Python 3.6.3 final

* Library: Library<41>. 
* Build: Build<39>. 

Python 3.6.3 release candidate 1

* Security: Security<23>. 
* Core and Builtins: Core and Builtins<40>. 
* Library: Library<42>. 
* Documentation: Documentation<41>. 
* Tests: Tests<38>. 
* Build: Build<40>. 
* Windows: Windows<36>. 
* IDLE: IDLE<36>. 
* Tools/Demos: Tools/Demos<21>. 

Python 3.6.2 release candidate 2

* Security: Security<24>. 

Python 3.6.2 release candidate 1

* Core and Builtins: Core and Builtins<41>. 
* Library: Library<43>. 
* Security: Security<25>. 
* Library: Library<44>. 
* IDLE: IDLE<37>. 
* C API: C API<36>. 
* Build: Build<41>. 
* Documentation: Documentation<42>. 
* Tools/Demos: Tools/Demos<22>. 
* Tests: Tests<39>. 
* Windows: Windows<37>. 

Python 3.6.1 final

* Core and Builtins: Core and Builtins<42>. 
* Build: Build<42>. 

Python 3.6.1 release candidate 1

* Core and Builtins: Core and Builtins<43>. 
* Library: Library<45>. 
* IDLE: IDLE<38>. 
* Windows: Windows<38>. 
* C API: C API<37>. 
* Documentation: Documentation<43>. 
* Tests: Tests<40>. 
* Build: Build<43>. 

Python 3.6.0 release candidate 2

* Core and Builtins: Core and Builtins<44>. 
* Tools/Demos: Tools/Demos<23>. 
* Windows: Windows<39>. 
* Build: Build<44>. 

Python 3.6.0 release candidate 1

* Core and Builtins: Core and Builtins<45>. 
* Library: Library<46>. 
* C API: C API<38>. 
* Documentation: Documentation<44>. 
* Tools/Demos: Tools/Demos<24>. 

Python 3.6.0 beta 4

* Core and Builtins: Core and Builtins<46>. 
* Library: Library<47>. 
* Documentation: Documentation<45>. 
* Tests: Tests<41>. 
* Build: Build<45>. 

Python 3.6.0 beta 3

* Core and Builtins: Core and Builtins<47>. 
* Library: Library<48>. 
* Windows: Windows<40>. 
* Build: Build<46>. 
* Tests: Tests<42>. 

Python 3.6.0 beta 2

* Core and Builtins: Core and Builtins<48>. 
* Library: Library<49>. 
* Windows: Windows<41>. 
* C API: C API<39>. 
* Build: Build<47>. 
* Tests: Tests<43>. 

Python 3.6.0 beta 1

* Core and Builtins: Core and Builtins<49>. 
* Library: Library<50>. 
* IDLE: IDLE<39>. 
* C API: C API<40>. 
* Tests: Tests<44>. 
* Build: Build<48>. 
* Tools/Demos: Tools/Demos<25>. 
* Windows: Windows<42>. 

Python 3.6.0 alpha 4

* Core and Builtins: Core and Builtins<50>. 
* Library: Library<51>. 
* IDLE: IDLE<40>. 
* Tests: Tests<45>. 
* Windows: Windows<43>. 
* Build: Build<49>. 

Python 3.6.0 alpha 3

* Core and Builtins: Core and Builtins<51>. 
* Library: Library<52>. 
* Security: Security<26>. 
* Library: Library<53>. 
* Security: Security<27>. 
* Library: Library<54>. 
* IDLE: IDLE<41>. 
* C API: C API<41>. 
* Build: Build<50>. 
* Tools/Demos: Tools/Demos<26>. 
* Documentation: Documentation<46>. 
* Tests: Tests<46>. 

Python 3.6.0 alpha 2

* Core and Builtins: Core and Builtins<52>. 
* Library: Library<55>. 
* Security: Security<28>. 
* Library: Library<56>. 
* Security: Security<29>. 
* Library: Library<57>. 
* IDLE: IDLE<42>. 
* Documentation: Documentation<47>. 
* Tests: Tests<47>. 
* Windows: Windows<44>. 
* Build: Build<51>. 
* Windows: Windows<45>. 
* C API: C API<42>. 
* Tools/Demos: Tools/Demos<27>. 

Python 3.6.0 alpha 1

* Core and Builtins: Core and Builtins<53>. 
* Library: Library<58>. 
* Security: Security<30>. 
* Library: Library<59>. 
* Security: Security<31>. 
* Library: Library<60>. 
* Security: Security<32>. 
* Library: Library<61>. 
* IDLE: IDLE<43>. 
* Documentation: Documentation<48>. 
* Tests: Tests<48>. 
* Build: Build<52>. 
* Windows: Windows<46>. 
* Tools/Demos: Tools/Demos<28>. 
* C API: C API<43>. 

Python 3.5.5 release candidate 1

* Security: Security<33>. 
* Core and Builtins: Core and Builtins<54>. 
* Library: Library<62>. 

Python 3.5.4 final

* Library: Library<63>. 

Python 3.5.4 release candidate 1

* Security: Security<34>. 
* Core and Builtins: Core and Builtins<55>. 
* Library: Library<64>. 
* Documentation: Documentation<49>. 
* Tests: Tests<49>. 
* Build: Build<53>. 
* Windows: Windows<47>. 
* C API: C API<44>. 

Python 3.5.3 release candidate 1

* Core and Builtins: Core and Builtins<56>. 
* Library: Library<65>. 
* Security: Security<35>. 
* Library: Library<66>. 
* Security: Security<36>. 
* Library: Library<67>. 
* IDLE: IDLE<44>. 
* C API: C API<45>. 
* Documentation: Documentation<50>. 
* Tests: Tests<50>. 
* Tools/Demos: Tools/Demos<29>. 
* Windows: Windows<48>. 
* Build: Build<54>. 

Python 3.5.2 final

* Core and Builtins: Core and Builtins<57>. 
* Tests: Tests<51>. 
* IDLE: IDLE<45>. 

Python 3.5.2 release candidate 1

* Core and Builtins: Core and Builtins<58>. 
* Security: Security<37>. 
* Library: Library<68>. 
* Security: Security<38>. 
* Library: Library<69>. 
* Security: Security<39>. 
* Library: Library<70>. 
* Security: Security<40>. 
* Library: Library<71>. 
* Security: Security<41>. 
* Library: Library<72>. 
* IDLE: IDLE<46>. 
* Documentation: Documentation<51>. 
* Tests: Tests<52>. 
* Build: Build<55>. 
* Windows: Windows<49>. 
* Tools/Demos: Tools/Demos<30>. 
* Windows: Windows<50>. 

Python 3.5.1 final

* Core and Builtins: Core and Builtins<59>. 
* Windows: Windows<51>. 

Python 3.5.1 release candidate 1

* Core and Builtins: Core and Builtins<60>. 
* Library: Library<73>. 
* IDLE: IDLE<47>. 
* Documentation: Documentation<52>. 
* Tests: Tests<53>. 
* Build: Build<56>. 
* Windows: Windows<52>. 
* Tools/Demos: Tools/Demos<31>. 

Python 3.5.0 final

* Build: Build<57>. 

Python 3.5.0 release candidate 4

* Library: Library<74>. 
* Build: Build<58>. 

Python 3.5.0 release candidate 3

* Core and Builtins: Core and Builtins<61>. 
* Library: Library<75>. 

Python 3.5.0 release candidate 2

* Core and Builtins: Core and Builtins<62>. 
* Library: Library<76>. 

Python 3.5.0 release candidate 1

* Core and Builtins: Core and Builtins<63>. 
* Library: Library<77>. 
* IDLE: IDLE<48>. 
* Documentation: Documentation<53>. 
* Tests: Tests<54>. 

Python 3.5.0 beta 4

* Core and Builtins: Core and Builtins<64>. 
* Library: Library<78>. 
* Build: Build<59>. 

Python 3.5.0 beta 3

* Core and Builtins: Core and Builtins<65>. 
* Library: Library<79>. 
* Tests: Tests<55>. 
* Documentation: Documentation<54>. 
* Build: Build<60>. 

Python 3.5.0 beta 2

* Core and Builtins: Core and Builtins<66>. 
* Library: Library<80>. 

Python 3.5.0 beta 1

* Core and Builtins: Core and Builtins<67>. 
* Library: Library<81>. 
* IDLE: IDLE<49>. 
* Tests: Tests<56>. 
* Documentation: Documentation<55>. 
* Tools/Demos: Tools/Demos<32>. 

Python 3.5.0 alpha 4

* Core and Builtins: Core and Builtins<68>. 
* Library: Library<82>. 
* Build: Build<61>. 
* Tests: Tests<57>. 
* Tools/Demos: Tools/Demos<33>. 
* C API: C API<46>. 

Python 3.5.0 alpha 3

* Core and Builtins: Core and Builtins<69>. 
* Library: Library<83>. 
* Build: Build<62>. 
* Tests: Tests<58>. 
* Tools/Demos: Tools/Demos<34>. 

Python 3.5.0 alpha 2

* Core and Builtins: Core and Builtins<70>. 
* Library: Library<84>. 
* Build: Build<63>. 
* C API: C API<47>. 
* Windows: Windows<53>. 

Python 3.5.0 alpha 1

* Core and Builtins: Core and Builtins<71>. 
* Library: Library<85>. 
* IDLE: IDLE<50>. 
* Build: Build<64>. 
* C API: C API<48>. 
* Documentation: Documentation<56>. 
* Tests: Tests<59>. 
* Tools/Demos: Tools/Demos<35>. 
* Windows: Windows<54>. 

The Python Tutorial

* Whetting Your Appetite:: 
* Using the Python Interpreter:: 
* An Informal Introduction to Python:: 
* More Control Flow Tools:: 
* Data Structures:: 
* Modules:: 
* Input and Output:: 
* Errors and Exceptions:: 
* Classes:: 
* Brief Tour of the Standard Library:: 
* Brief Tour of the Standard Library — Part II:: 
* Virtual Environments and Packages:: 
* What Now?:: 
* Interactive Input Editing and History Substitution:: 
* Floating Point Arithmetic; Issues and Limitations: Floating Point Arithmetic Issues and Limitations. 
* Appendix:: 

Using the Python Interpreter

* Invoking the Interpreter:: 
* The Interpreter and Its Environment:: 

Invoking the Interpreter

* Argument Passing:: 
* Interactive Mode:: 

The Interpreter and Its Environment

* Source Code Encoding:: 

An Informal Introduction to Python

* Using Python as a Calculator:: 
* First Steps Towards Programming:: 

Using Python as a Calculator

* Numbers:: 
* Strings:: 
* Lists:: 

More Control Flow Tools

* if Statements:: 
* for Statements:: 
* The range() Function: The range Function. 
* break and continue Statements@comma{} and else Clauses on Loops: break and continue Statements and else Clauses on Loops. 
* pass Statements:: 
* match Statements:: 
* Defining Functions:: 
* More on Defining Functions:: 
* Intermezzo; Coding Style: Intermezzo Coding Style. 

More on Defining Functions

* Default Argument Values:: 
* Keyword Arguments:: 
* Special parameters:: 
* Arbitrary Argument Lists:: 
* Unpacking Argument Lists:: 
* Lambda Expressions:: 
* Documentation Strings:: 
* Function Annotations:: 

Special parameters

* Positional-or-Keyword Arguments:: 
* Positional-Only Parameters:: 
* Keyword-Only Arguments:: 
* Function Examples:: 
* Recap:: 

Data Structures

* More on Lists:: 
* The del statement:: 
* Tuples and Sequences:: 
* Sets:: 
* Dictionaries:: 
* Looping Techniques:: 
* More on Conditions:: 
* Comparing Sequences and Other Types:: 

More on Lists

* Using Lists as Stacks:: 
* Using Lists as Queues:: 
* List Comprehensions: List Comprehensions<2>. 
* Nested List Comprehensions:: 

Modules

* More on Modules:: 
* Standard Modules:: 
* The dir() Function: The dir Function. 
* Packages:: 

More on Modules

* Executing modules as scripts:: 
* The Module Search Path:: 
* “Compiled” Python files:: 

Packages

* Importing * From a Package:: 
* Intra-package References:: 
* Packages in Multiple Directories:: 

Input and Output

* Fancier Output Formatting:: 
* Reading and Writing Files:: 

Fancier Output Formatting

* Formatted String Literals:: 
* The String format() Method: The String format Method. 
* Manual String Formatting:: 
* Old string formatting:: 

Reading and Writing Files

* Methods of File Objects:: 
* Saving structured data with json:: 

Errors and Exceptions

* Syntax Errors:: 
* Exceptions:: 
* Handling Exceptions:: 
* Raising Exceptions:: 
* Exception Chaining:: 
* User-defined Exceptions:: 
* Defining Clean-up Actions:: 
* Predefined Clean-up Actions:: 
* Raising and Handling Multiple Unrelated Exceptions:: 

Classes

* A Word About Names and Objects:: 
* Python Scopes and Namespaces:: 
* A First Look at Classes:: 
* Random Remarks:: 
* Inheritance:: 
* Private Variables:: 
* Odds and Ends:: 
* Iterators:: 
* Generators:: 
* Generator Expressions:: 

Python Scopes and Namespaces

* Scopes and Namespaces Example:: 

A First Look at Classes

* Class Definition Syntax:: 
* Class Objects:: 
* Instance Objects:: 
* Method Objects:: 
* Class and Instance Variables:: 

Inheritance

* Multiple Inheritance:: 

Brief Tour of the Standard Library

* Operating System Interface:: 
* File Wildcards:: 
* Command Line Arguments:: 
* Error Output Redirection and Program Termination:: 
* String Pattern Matching:: 
* Mathematics:: 
* Internet Access:: 
* Dates and Times:: 
* Data Compression:: 
* Performance Measurement:: 
* Quality Control:: 
* Batteries Included:: 

Brief Tour of the Standard Library — Part II

* Output Formatting:: 
* Templating:: 
* Working with Binary Data Record Layouts:: 
* Multi-threading: Multi-threading<2>. 
* Logging:: 
* Weak References:: 
* Tools for Working with Lists:: 
* Decimal Floating Point Arithmetic:: 

Virtual Environments and Packages

* Introduction: Introduction<4>. 
* Creating Virtual Environments:: 
* Managing Packages with pip:: 

Interactive Input Editing and History Substitution

* Tab Completion and History Editing:: 
* Alternatives to the Interactive Interpreter:: 

Floating Point Arithmetic:  Issues and Limitations

* Representation Error:: 

Appendix

* Interactive Mode: Interactive Mode<2>. 

Interactive Mode

* Error Handling:: 
* Executable Python Scripts:: 
* The Interactive Startup File:: 
* The Customization Modules:: 

Python Setup and Usage

* Command line and environment:: 
* Using Python on Unix platforms:: 
* Configure Python:: 
* Using Python on Windows:: 
* Using Python on a Mac:: 
* Editors and IDEs:: 

Command line and environment

* Command line:: 
* Environment variables:: 

Command line

* Interface options:: 
* Generic options:: 
* Miscellaneous options:: 
* Options you shouldn’t use:: 

Environment variables

* Debug-mode variables:: 

Using Python on Unix platforms

* Getting and installing the latest version of Python:: 
* Building Python:: 
* Python-related paths and files:: 
* Miscellaneous:: 
* Custom OpenSSL:: 

Getting and installing the latest version of Python

* On Linux:: 
* On FreeBSD and OpenBSD:: 
* On OpenSolaris:: 

Configure Python

* Configure Options:: 
* Python Build System:: 
* Compiler and linker flags:: 

Configure Options

* General Options:: 
* Install Options:: 
* Performance options:: 
* Python Debug Build:: 
* Debug options:: 
* Linker options:: 
* Libraries options:: 
* Security Options:: 
* macOS Options:: 
* Cross Compiling Options:: 

Python Build System

* Main files of the build system:: 
* Main build steps:: 
* Main Makefile targets:: 
* C extensions:: 

Compiler and linker flags

* Preprocessor flags:: 
* Compiler flags:: 
* Linker flags:: 

Using Python on Windows

* The full installer:: 
* The Microsoft Store package:: 
* The nuget.org packages: The nuget org packages. 
* The embeddable package:: 
* Alternative bundles:: 
* Configuring Python:: 
* UTF-8 mode:: 
* Python Launcher for Windows:: 
* Finding modules:: 
* Additional modules:: 
* Compiling Python on Windows:: 
* Other Platforms:: 

The full installer

* Installation steps:: 
* Removing the MAX_PATH Limitation:: 
* Installing Without UI:: 
* Installing Without Downloading:: 
* Modifying an install:: 

The Microsoft Store package

* Known Issues:: 

The embeddable package

* Python Application:: 
* Embedding Python:: 

Configuring Python

* Excursus; Setting environment variables: Excursus Setting environment variables. 
* Finding the Python executable:: 

Python Launcher for Windows

* Getting started:: 
* Shebang Lines:: 
* Arguments in shebang lines:: 
* Customization:: 
* Diagnostics:: 

Getting started

* From the command-line:: 
* Virtual environments:: 
* From a script:: 
* From file associations:: 

Customization

* Customization via INI files:: 
* Customizing default Python versions:: 

Additional modules

* PyWin32:: 
* cx_Freeze:: 
* WConio:: 

Using Python on a Mac

* Getting and Installing MacPython:: 
* The IDE:: 
* Installing Additional Python Packages:: 
* GUI Programming on the Mac:: 
* Distributing Python Applications on the Mac:: 
* Other Resources:: 

Getting and Installing MacPython

* How to run a Python script:: 
* Running scripts with a GUI:: 
* Configuration:: 

The Python Language Reference

* Introduction: Introduction<5>. 
* Lexical analysis:: 
* Data model:: 
* Execution model:: 
* The import system:: 
* Expressions:: 
* Simple statements:: 
* Compound statements:: 
* Top-level components:: 
* Full Grammar specification:: 

Introduction

* Alternate Implementations:: 
* Notation:: 

Lexical analysis

* Line structure:: 
* Other tokens:: 
* Identifiers and keywords:: 
* Literals:: 
* Operators:: 
* Delimiters:: 

Line structure

* Logical lines:: 
* Physical lines:: 
* Comments:: 
* Encoding declarations:: 
* Explicit line joining:: 
* Implicit line joining:: 
* Blank lines:: 
* Indentation:: 
* Whitespace between tokens:: 

Identifiers and keywords

* Keywords:: 
* Soft Keywords:: 
* Reserved classes of identifiers:: 

Literals

* String and Bytes literals:: 
* String literal concatenation:: 
* Formatted string literals:: 
* Numeric literals:: 
* Integer literals:: 
* Floating point literals:: 
* Imaginary literals:: 

Data model

* Objects@comma{} values and types: Objects values and types. 
* The standard type hierarchy:: 
* Special method names:: 
* Coroutines:: 

Special method names

* Basic customization:: 
* Customizing attribute access:: 
* Customizing class creation:: 
* Customizing instance and subclass checks:: 
* Emulating generic types:: 
* Emulating callable objects:: 
* Emulating container types:: 
* Emulating numeric types:: 
* With Statement Context Managers:: 
* Customizing positional arguments in class pattern matching:: 
* Special method lookup:: 

Customizing attribute access

* Customizing module attribute access:: 
* Implementing Descriptors:: 
* Invoking Descriptors:: 
* __slots__: __slots__<2>. 

__slots__

* Notes on using __slots__:: 

Customizing class creation

* Metaclasses:: 
* Resolving MRO entries:: 
* Determining the appropriate metaclass:: 
* Preparing the class namespace:: 
* Executing the class body:: 
* Creating the class object:: 
* Uses for metaclasses:: 

Emulating generic types

* The purpose of __class_getitem__:: 
* __class_getitem__ versus __getitem__:: 

Coroutines

* Awaitable Objects:: 
* Coroutine Objects:: 
* Asynchronous Iterators:: 
* Asynchronous Context Managers:: 

Execution model

* Structure of a program:: 
* Naming and binding:: 
* Exceptions: Exceptions<2>. 

Naming and binding

* Binding of names:: 
* Resolution of names:: 
* Builtins and restricted execution:: 
* Interaction with dynamic features:: 

The import system

* importlib: importlib<8>. 
* Packages: Packages<2>. 
* Searching:: 
* Loading:: 
* The Path Based Finder:: 
* Replacing the standard import system:: 
* Package Relative Imports:: 
* Special considerations for __main__:: 
* Open issues:: 
* References:: 

Packages

* Regular packages:: 
* Namespace packages:: 

Searching

* The module cache:: 
* Finders and loaders:: 
* Import hooks:: 
* The meta path:: 

Loading

* Loaders:: 
* Submodules:: 
* Module spec:: 
* Import-related module attributes:: 
* module.__path__: module __path__. 
* Module reprs:: 
* Cached bytecode invalidation:: 

The Path Based Finder

* Path entry finders:: 
* Path entry finder protocol:: 

Special considerations for __main__

* __main__.__spec__: __main__ __spec__. 

Expressions

* Arithmetic conversions:: 
* Atoms:: 
* Primaries:: 
* Await expression:: 
* The power operator:: 
* Unary arithmetic and bitwise operations:: 
* Binary arithmetic operations:: 
* Shifting operations:: 
* Binary bitwise operations:: 
* Comparisons:: 
* Boolean operations:: 
* Assignment expressions: Assignment expressions<2>. 
* Conditional expressions:: 
* Lambdas:: 
* Expression lists:: 
* Evaluation order:: 
* Operator precedence:: 

Atoms

* Identifiers (Names): Identifiers Names. 
* Literals: Literals<2>. 
* Parenthesized forms:: 
* Displays for lists@comma{} sets and dictionaries: Displays for lists sets and dictionaries. 
* List displays:: 
* Set displays:: 
* Dictionary displays:: 
* Generator expressions:: 
* Yield expressions:: 

Yield expressions

* Generator-iterator methods:: 
* Examples:: 
* Asynchronous generator functions:: 
* Asynchronous generator-iterator methods:: 

Primaries

* Attribute references:: 
* Subscriptions:: 
* Slicings:: 
* Calls:: 

Comparisons

* Value comparisons:: 
* Membership test operations:: 
* Identity comparisons:: 

Simple statements

* Expression statements:: 
* Assignment statements:: 
* The assert statement:: 
* The pass statement:: 
* The del statement: The del statement<2>. 
* The return statement:: 
* The yield statement:: 
* The raise statement:: 
* The break statement:: 
* The continue statement:: 
* The import statement:: 
* The global statement:: 
* The nonlocal statement:: 

Assignment statements

* Augmented assignment statements:: 
* Annotated assignment statements:: 

The import statement

* Future statements:: 

Compound statements

* The if statement:: 
* The while statement:: 
* The for statement:: 
* The try statement:: 
* The with statement:: 
* The match statement:: 
* Function definitions:: 
* Class definitions:: 
* Coroutines: Coroutines<2>. 

The match statement

* Overview:: 
* Guards:: 
* Irrefutable Case Blocks:: 
* Patterns:: 

Patterns

* OR Patterns:: 
* AS Patterns:: 
* Literal Patterns:: 
* Capture Patterns:: 
* Wildcard Patterns:: 
* Value Patterns:: 
* Group Patterns:: 
* Sequence Patterns:: 
* Mapping Patterns:: 
* Class Patterns:: 

Coroutines

* Coroutine function definition:: 
* The async for statement:: 
* The async with statement:: 

Top-level components

* Complete Python programs:: 
* File input:: 
* Interactive input:: 
* Expression input:: 

The Python Standard Library

* Introduction: Introduction<6>. 
* Built-in Functions:: 
* Built-in Constants:: 
* Built-in Types:: 
* Built-in Exceptions:: 
* Text Processing Services:: 
* Binary Data Services:: 
* Data Types:: 
* Numeric and Mathematical Modules:: 
* Functional Programming Modules:: 
* File and Directory Access:: 
* Data Persistence:: 
* Data Compression and Archiving:: 
* File Formats:: 
* Cryptographic Services:: 
* Generic Operating System Services:: 
* Concurrent Execution:: 
* Networking and Interprocess Communication:: 
* Internet Data Handling:: 
* Structured Markup Processing Tools:: 
* Internet Protocols and Support:: 
* Multimedia Services:: 
* Internationalization:: 
* Program Frameworks:: 
* Graphical User Interfaces with Tk:: 
* Development Tools:: 
* Debugging and Profiling:: 
* Software Packaging and Distribution:: 
* Python Runtime Services:: 
* Custom Python Interpreters:: 
* Importing Modules:: 
* Python Language Services:: 
* MS Windows Specific Services:: 
* Unix Specific Services:: 
* Superseded Modules:: 
* Undocumented Modules:: 
* Security Considerations: Security Considerations<3>. 

Introduction

* Notes on availability:: 

Built-in Constants

* Constants added by the site module:: 

Built-in Types

* Truth Value Testing:: 
* Boolean Operations — and@comma{} or@comma{} not: Boolean Operations — and or not. 
* Comparisons: Comparisons<2>. 
* Numeric Types — int@comma{} float@comma{} complex: Numeric Types — int float complex. 
* Iterator Types:: 
* Sequence Types — list@comma{} tuple@comma{} range: Sequence Types — list tuple range. 
* Text Sequence Type — str:: 
* Binary Sequence Types — bytes@comma{} bytearray@comma{} memoryview: Binary Sequence Types — bytes bytearray memoryview. 
* Set Types — set@comma{} frozenset: Set Types — set frozenset. 
* Mapping Types — dict:: 
* Context Manager Types:: 
* Type Annotation Types — Generic Alias@comma{} Union: Type Annotation Types — Generic Alias Union. 
* Other Built-in Types:: 
* Special Attributes:: 

Numeric Types — int, float, complex

* Bitwise Operations on Integer Types:: 
* Additional Methods on Integer Types:: 
* Additional Methods on Float:: 
* Hashing of numeric types:: 

Iterator Types

* Generator Types:: 

Sequence Types — list, tuple, range

* Common Sequence Operations:: 
* Immutable Sequence Types:: 
* Mutable Sequence Types:: 
* Lists: Lists<2>. 
* Tuples:: 
* Ranges:: 

Text Sequence Type — str

* String Methods: String Methods<2>. 
* printf-style String Formatting:: 

Binary Sequence Types — bytes, bytearray, memoryview

* Bytes Objects:: 
* Bytearray Objects:: 
* Bytes and Bytearray Operations:: 
* printf-style Bytes Formatting:: 
* Memory Views:: 

Mapping Types — dict

* Dictionary view objects:: 

Type Annotation Types — Generic Alias, Union

* Generic Alias Type:: 
* Union Type:: 

Generic Alias Type

* Standard Generic Classes:: 
* Special Attributes of GenericAlias objects:: 

Other Built-in Types

* Modules: Modules<2>. 
* Classes and Class Instances:: 
* Functions:: 
* Methods:: 
* Code Objects:: 
* Type Objects:: 
* The Null Object:: 
* The Ellipsis Object:: 
* The NotImplemented Object:: 
* Boolean Values:: 
* Internal Objects:: 

Built-in Exceptions

* Exception context:: 
* Inheriting from built-in exceptions:: 
* Base classes:: 
* Concrete exceptions:: 
* Warnings:: 
* Exception groups:: 
* Exception hierarchy:: 

Concrete exceptions

* OS exceptions:: 

Text Processing Services

* string — Common string operations:: 
* re — Regular expression operations:: 
* difflib — Helpers for computing deltas:: 
* textwrap — Text wrapping and filling:: 
* unicodedata — Unicode Database:: 
* stringprep — Internet String Preparation:: 
* readline — GNU readline interface:: 
* rlcompleter — Completion function for GNU readline:: 

string — Common string operations

* String constants:: 
* Custom String Formatting:: 
* Format String Syntax:: 
* Template strings:: 
* Helper functions:: 

Format String Syntax

* Format Specification Mini-Language:: 
* Format examples:: 

re — Regular expression operations

* Regular Expression Syntax:: 
* Module Contents:: 
* Regular Expression Objects:: 
* Match Objects:: 
* Regular Expression Examples:: 

Regular Expression Examples

* Checking for a Pair:: 
* Simulating scanf(): Simulating scanf. 
* search() vs. match(): search vs match. 
* Making a Phonebook:: 
* Text Munging:: 
* Finding all Adverbs:: 
* Finding all Adverbs and their Positions:: 
* Raw String Notation:: 
* Writing a Tokenizer:: 

difflib — Helpers for computing deltas

* SequenceMatcher Objects:: 
* SequenceMatcher Examples:: 
* Differ Objects:: 
* Differ Example:: 
* A command-line interface to difflib:: 

readline — GNU readline interface

* Init file:: 
* Line buffer:: 
* History file:: 
* History list:: 
* Startup hooks:: 
* Completion:: 
* Example:: 

rlcompleter — Completion function for GNU readline

* Completer Objects:: 

Binary Data Services

* struct — Interpret bytes as packed binary data:: 
* codecs — Codec registry and base classes:: 

struct — Interpret bytes as packed binary data

* Functions and Exceptions:: 
* Format Strings:: 
* Classes: Classes<2>. 

Format Strings

* Byte Order@comma{} Size@comma{} and Alignment: Byte Order Size and Alignment. 
* Format Characters:: 
* Examples: Examples<2>. 

codecs — Codec registry and base classes

* Codec Base Classes:: 
* Encodings and Unicode:: 
* Standard Encodings:: 
* Python Specific Encodings:: 
* encodings.idna — Internationalized Domain Names in Applications: encodings idna — Internationalized Domain Names in Applications. 
* encodings.mbcs — Windows ANSI codepage: encodings mbcs — Windows ANSI codepage. 
* encodings.utf_8_sig — UTF-8 codec with BOM signature: encodings utf_8_sig — UTF-8 codec with BOM signature. 

Codec Base Classes

* Error Handlers:: 
* Stateless Encoding and Decoding:: 
* Incremental Encoding and Decoding:: 
* Stream Encoding and Decoding:: 

Incremental Encoding and Decoding

* IncrementalEncoder Objects:: 
* IncrementalDecoder Objects:: 

Stream Encoding and Decoding

* StreamWriter Objects:: 
* StreamReader Objects:: 
* StreamReaderWriter Objects:: 
* StreamRecoder Objects:: 

Python Specific Encodings

* Text Encodings:: 
* Binary Transforms:: 
* Text Transforms:: 

Data Types

* datetime — Basic date and time types:: 
* zoneinfo — IANA time zone support:: 
* calendar — General calendar-related functions:: 
* collections — Container datatypes:: 
* collections.abc — Abstract Base Classes for Containers: collections abc — Abstract Base Classes for Containers. 
* heapq — Heap queue algorithm:: 
* bisect — Array bisection algorithm:: 
* array — Efficient arrays of numeric values:: 
* weakref — Weak references:: 
* types — Dynamic type creation and names for built-in types:: 
* copy — Shallow and deep copy operations:: 
* pprint — Data pretty printer:: 
* reprlib — Alternate repr() implementation: reprlib — Alternate repr implementation. 
* enum — Support for enumerations:: 
* graphlib — Functionality to operate with graph-like structures:: 

datetime — Basic date and time types

* Aware and Naive Objects:: 
* Constants:: 
* Available Types:: 
* timedelta Objects:: 
* date Objects:: 
* datetime Objects:: 
* time Objects:: 
* tzinfo Objects:: 
* timezone Objects:: 
* strftime() and strptime() Behavior: strftime and strptime Behavior. 

Available Types

* Common Properties:: 
* Determining if an Object is Aware or Naive:: 

timedelta Objects

* Examples of usage; timedelta: Examples of usage timedelta. 

date Objects

* Examples of Usage; date: Examples of Usage date. 

datetime Objects

* Examples of Usage; datetime: Examples of Usage datetime. 

time Objects

* Examples of Usage; time: Examples of Usage time. 

strftime() and strptime() Behavior

* strftime() and strptime() Format Codes: strftime and strptime Format Codes. 
* Technical Detail:: 

zoneinfo — IANA time zone support

* Using ZoneInfo:: 
* Data sources:: 
* The ZoneInfo class:: 
* Functions: Functions<2>. 
* Globals:: 
* Exceptions and warnings:: 

Data sources

* Configuring the data sources:: 

Configuring the data sources

* Compile-time configuration:: 
* Environment configuration:: 
* Runtime configuration:: 

The ZoneInfo class

* String representations:: 
* Pickle serialization:: 

collections — Container datatypes

* ChainMap objects:: 
* Counter objects:: 
* deque objects:: 
* defaultdict objects:: 
* namedtuple() Factory Function for Tuples with Named Fields: namedtuple Factory Function for Tuples with Named Fields. 
* OrderedDict objects:: 
* UserDict objects:: 
* UserList objects:: 
* UserString objects:: 

ChainMap objects

* ChainMap Examples and Recipes:: 

deque objects

* deque Recipes:: 

defaultdict objects

* defaultdict Examples:: 

OrderedDict objects

* OrderedDict Examples and Recipes:: 

collections.abc — Abstract Base Classes for Containers

* Collections Abstract Base Classes:: 
* Collections Abstract Base Classes – Detailed Descriptions:: 
* Examples and Recipes:: 

heapq — Heap queue algorithm

* Basic Examples:: 
* Priority Queue Implementation Notes:: 
* Theory:: 

bisect — Array bisection algorithm

* Performance Notes:: 
* Searching Sorted Lists:: 
* Examples: Examples<3>. 

weakref — Weak references

* Weak Reference Objects:: 
* Example: Example<2>. 
* Finalizer Objects:: 
* Comparing finalizers with __del__() methods: Comparing finalizers with __del__ methods. 

types — Dynamic type creation and names for built-in types

* Dynamic Type Creation:: 
* Standard Interpreter Types:: 
* Additional Utility Classes and Functions:: 
* Coroutine Utility Functions:: 

pprint — Data pretty printer

* PrettyPrinter Objects:: 
* Example: Example<3>. 

reprlib — Alternate repr() implementation

* Repr Objects:: 
* Subclassing Repr Objects:: 

enum — Support for enumerations

* Module Contents: Module Contents<2>. 
* Data Types: Data Types<2>. 
* Utilities and Decorators:: 
* Notes:: 

Data Types

* Supported __dunder__ names:: 
* Supported _sunder_ names:: 

graphlib — Functionality to operate with graph-like structures

* Exceptions: Exceptions<3>. 

Numeric and Mathematical Modules

* numbers — Numeric abstract base classes:: 
* math — Mathematical functions:: 
* cmath — Mathematical functions for complex numbers:: 
* decimal — Decimal fixed point and floating point arithmetic:: 
* fractions — Rational numbers:: 
* random — Generate pseudo-random numbers:: 
* statistics — Mathematical statistics functions:: 

numbers — Numeric abstract base classes

* The numeric tower:: 
* Notes for type implementors:: 

Notes for type implementors

* Adding More Numeric ABCs:: 
* Implementing the arithmetic operations:: 

math — Mathematical functions

* Number-theoretic and representation functions:: 
* Power and logarithmic functions:: 
* Trigonometric functions:: 
* Angular conversion:: 
* Hyperbolic functions:: 
* Special functions:: 
* Constants: Constants<2>. 

cmath — Mathematical functions for complex numbers

* Conversions to and from polar coordinates:: 
* Power and logarithmic functions: Power and logarithmic functions<2>. 
* Trigonometric functions: Trigonometric functions<2>. 
* Hyperbolic functions: Hyperbolic functions<2>. 
* Classification functions:: 
* Constants: Constants<3>. 

decimal — Decimal fixed point and floating point arithmetic

* Quick-start Tutorial:: 
* Decimal objects:: 
* Context objects:: 
* Constants: Constants<4>. 
* Rounding modes:: 
* Signals:: 
* Floating Point Notes:: 
* Working with threads:: 
* Recipes:: 
* Decimal FAQ:: 

Decimal objects

* Logical operands:: 

Floating Point Notes

* Mitigating round-off error with increased precision:: 
* Special values:: 

random — Generate pseudo-random numbers

* Bookkeeping functions:: 
* Functions for bytes:: 
* Functions for integers:: 
* Functions for sequences:: 
* Real-valued distributions:: 
* Alternative Generator:: 
* Notes on Reproducibility:: 
* Examples: Examples<4>. 
* Recipes: Recipes<2>. 

statistics — Mathematical statistics functions

* Averages and measures of central location:: 
* Measures of spread:: 
* Statistics for relations between two inputs:: 
* Function details:: 
* Exceptions: Exceptions<4>. 
* NormalDist objects:: 

NormalDist objects

* NormalDist Examples and Recipes:: 

Functional Programming Modules

* itertools — Functions creating iterators for efficient looping:: 
* functools — Higher-order functions and operations on callable objects:: 
* operator — Standard operators as functions:: 

itertools — Functions creating iterators for efficient looping

* Itertool functions:: 
* Itertools Recipes:: 

functools — Higher-order functions and operations on callable objects

* partial Objects:: 

operator — Standard operators as functions

* Mapping Operators to Functions:: 
* In-place Operators:: 

File and Directory Access

* pathlib — Object-oriented filesystem paths:: 
* os.path — Common pathname manipulations: os path — Common pathname manipulations. 
* fileinput — Iterate over lines from multiple input streams:: 
* stat — Interpreting stat() results: stat — Interpreting stat results. 
* filecmp — File and Directory Comparisons:: 
* tempfile — Generate temporary files and directories:: 
* glob — Unix style pathname pattern expansion:: 
* fnmatch — Unix filename pattern matching:: 
* linecache — Random access to text lines:: 
* shutil — High-level file operations:: 

pathlib — Object-oriented filesystem paths

* Basic use:: 
* Pure paths:: 
* Concrete paths:: 
* Correspondence to tools in the os module:: 

Pure paths

* General properties:: 
* Operators: Operators<2>. 
* Accessing individual parts:: 
* Methods and properties:: 

Concrete paths

* Methods: Methods<2>. 

filecmp — File and Directory Comparisons

* The dircmp class:: 

tempfile — Generate temporary files and directories

* Examples: Examples<5>. 
* Deprecated functions and variables:: 

shutil — High-level file operations

* Directory and files operations:: 
* Archiving operations:: 
* Querying the size of the output terminal:: 

Directory and files operations

* Platform-dependent efficient copy operations:: 
* copytree example:: 
* rmtree example:: 

Archiving operations

* Archiving example:: 
* Archiving example with base_dir:: 

Data Persistence

* pickle — Python object serialization:: 
* copyreg — Register pickle support functions:: 
* shelve — Python object persistence:: 
* marshal — Internal Python object serialization:: 
* dbm — Interfaces to Unix “databases”:: 
* sqlite3 — DB-API 2.0 interface for SQLite databases: sqlite3 — DB-API 2 0 interface for SQLite databases. 

pickle — Python object serialization

* Relationship to other Python modules:: 
* Data stream format:: 
* Module Interface:: 
* What can be pickled and unpickled?:: 
* Pickling Class Instances:: 
* Custom Reduction for Types@comma{} Functions@comma{} and Other Objects: Custom Reduction for Types Functions and Other Objects. 
* Out-of-band Buffers:: 
* Restricting Globals:: 
* Performance: Performance<2>. 
* Examples: Examples<6>. 

Relationship to other Python modules

* Comparison with marshal:: 
* Comparison with json:: 

Pickling Class Instances

* Persistence of External Objects:: 
* Dispatch Tables:: 
* Handling Stateful Objects:: 

Out-of-band Buffers

* Provider API:: 
* Consumer API:: 
* Example: Example<4>. 

copyreg — Register pickle support functions

* Example: Example<5>. 

shelve — Python object persistence

* Restrictions:: 
* Example: Example<6>. 

dbm — Interfaces to Unix “databases”

* dbm.gnu — GNU’s reinterpretation of dbm: dbm gnu — GNU’s reinterpretation of dbm. 
* dbm.ndbm — Interface based on ndbm: dbm ndbm — Interface based on ndbm. 
* dbm.dumb — Portable DBM implementation: dbm dumb — Portable DBM implementation. 

sqlite3 — DB-API 2.0 interface for SQLite databases

* Module functions and constants:: 
* Connection Objects:: 
* Cursor Objects:: 
* Row Objects:: 
* Exceptions: Exceptions<5>. 
* SQLite and Python types:: 
* Controlling Transactions:: 
* Using sqlite3 efficiently:: 

SQLite and Python types

* Introduction: Introduction<7>. 
* Using adapters to store additional Python types in SQLite databases:: 
* Converting SQLite values to custom Python types:: 
* Default adapters and converters:: 

Using adapters to store additional Python types in SQLite databases

* Letting your object adapt itself:: 
* Registering an adapter callable:: 

Using sqlite3 efficiently

* Using shortcut methods:: 
* Accessing columns by name instead of by index:: 
* Using the connection as a context manager:: 

Data Compression and Archiving

* zlib — Compression compatible with gzip:: 
* gzip — Support for gzip files:: 
* bz2 — Support for bzip2 compression:: 
* lzma — Compression using the LZMA algorithm:: 
* zipfile — Work with ZIP archives:: 
* tarfile — Read and write tar archive files:: 

gzip — Support for gzip files

* Examples of usage:: 
* Command Line Interface:: 

Command Line Interface

* Command line options:: 

bz2 — Support for bzip2 compression

* (De)compression of files: De compression of files. 
* Incremental (de)compression: Incremental de compression. 
* One-shot (de)compression: One-shot de compression. 
* Examples of usage: Examples of usage<2>. 

lzma — Compression using the LZMA algorithm

* Reading and writing compressed files:: 
* Compressing and decompressing data in memory:: 
* Miscellaneous: Miscellaneous<2>. 
* Specifying custom filter chains:: 
* Examples: Examples<7>. 

zipfile — Work with ZIP archives

* ZipFile Objects:: 
* Path Objects:: 
* PyZipFile Objects:: 
* ZipInfo Objects:: 
* Command-Line Interface:: 
* Decompression pitfalls:: 

Command-Line Interface

* Command-line options:: 

Decompression pitfalls

* From file itself:: 
* File System limitations:: 
* Resources limitations:: 
* Interruption:: 
* Default behaviors of extraction:: 

tarfile — Read and write tar archive files

* TarFile Objects:: 
* TarInfo Objects:: 
* Command-Line Interface: Command-Line Interface<2>. 
* Examples: Examples<8>. 
* Supported tar formats:: 
* Unicode issues:: 

Command-Line Interface

* Command-line options: Command-line options<2>. 

File Formats

* csv — CSV File Reading and Writing:: 
* configparser — Configuration file parser:: 
* netrc — netrc file processing:: 
* xdrlib — Encode and decode XDR data:: 
* plistlib — Generate and parse Apple .plist files: plistlib — Generate and parse Apple plist files. 

csv — CSV File Reading and Writing

* Module Contents: Module Contents<3>. 
* Dialects and Formatting Parameters:: 
* Reader Objects:: 
* Writer Objects:: 
* Examples: Examples<9>. 

configparser — Configuration file parser

* Quick Start:: 
* Supported Datatypes:: 
* Fallback Values:: 
* Supported INI File Structure:: 
* Interpolation of values:: 
* Mapping Protocol Access:: 
* Customizing Parser Behaviour:: 
* Legacy API Examples:: 
* ConfigParser Objects:: 
* RawConfigParser Objects:: 
* Exceptions: Exceptions<6>. 

netrc — netrc file processing

* netrc Objects:: 

xdrlib — Encode and decode XDR data

* Packer Objects:: 
* Unpacker Objects:: 
* Exceptions: Exceptions<7>. 

plistlib — Generate and parse Apple .plist files

* Examples: Examples<10>. 

Cryptographic Services

* hashlib — Secure hashes and message digests:: 
* hmac — Keyed-Hashing for Message Authentication:: 
* secrets — Generate secure random numbers for managing secrets:: 

hashlib — Secure hashes and message digests

* Hash algorithms:: 
* SHAKE variable length digests:: 
* Key derivation:: 
* BLAKE2:: 

BLAKE2

* Creating hash objects:: 
* Constants: Constants<5>. 
* Examples: Examples<11>. 
* Credits:: 

Examples

* Simple hashing:: 
* Using different digest sizes:: 
* Keyed hashing:: 
* Randomized hashing:: 
* Personalization:: 
* Tree mode:: 

secrets — Generate secure random numbers for managing secrets

* Random numbers:: 
* Generating tokens:: 
* Other functions:: 
* Recipes and best practices:: 

Generating tokens

* How many bytes should tokens use?:: 

Generic Operating System Services

* os — Miscellaneous operating system interfaces:: 
* io — Core tools for working with streams:: 
* time — Time access and conversions:: 
* argparse — Parser for command-line options@comma{} arguments and sub-commands: argparse — Parser for command-line options arguments and sub-commands. 
* getopt — C-style parser for command line options:: 
* logging — Logging facility for Python:: 
* logging.config — Logging configuration: logging config — Logging configuration. 
* logging.handlers — Logging handlers: logging handlers — Logging handlers. 
* getpass — Portable password input:: 
* curses — Terminal handling for character-cell displays:: 
* curses.textpad — Text input widget for curses programs: curses textpad — Text input widget for curses programs. 
* curses.ascii — Utilities for ASCII characters: curses ascii — Utilities for ASCII characters. 
* curses.panel — A panel stack extension for curses: curses panel — A panel stack extension for curses. 
* platform — Access to underlying platform’s identifying data:: 
* errno — Standard errno system symbols:: 
* ctypes — A foreign function library for Python:: 

os — Miscellaneous operating system interfaces

* File Names@comma{} Command Line Arguments@comma{} and Environment Variables: File Names Command Line Arguments and Environment Variables. 
* Python UTF-8 Mode:: 
* Process Parameters:: 
* File Object Creation:: 
* File Descriptor Operations:: 
* Files and Directories:: 
* Process Management:: 
* Interface to the scheduler:: 
* Miscellaneous System Information:: 
* Random numbers: Random numbers<2>. 

File Descriptor Operations

* Querying the size of a terminal:: 
* Inheritance of File Descriptors:: 

Files and Directories

* Linux extended attributes:: 

io — Core tools for working with streams

* Overview: Overview<2>. 
* Text Encoding:: 
* High-level Module Interface:: 
* Class hierarchy:: 
* Performance: Performance<3>. 

Overview

* Text I/O:: 
* Binary I/O:: 
* Raw I/O:: 

Text Encoding

* Opt-in EncodingWarning:: 

Class hierarchy

* I/O Base Classes:: 
* Raw File I/O:: 
* Buffered Streams:: 
* Text I/O: Text I/O<2>. 

Performance

* Binary I/O: Binary I/O<2>. 
* Text I/O: Text I/O<3>. 
* Multi-threading: Multi-threading<3>. 
* Reentrancy:: 

time — Time access and conversions

* Functions: Functions<3>. 
* Clock ID Constants:: 
* Timezone Constants:: 

argparse — Parser for command-line options, arguments and sub-commands

* Example: Example<7>. 
* ArgumentParser objects:: 
* The add_argument() method: The add_argument method. 
* The parse_args() method: The parse_args method. 
* Other utilities:: 
* Upgrading optparse code:: 

Example

* Creating a parser:: 
* Adding arguments:: 
* Parsing arguments:: 

ArgumentParser objects

* prog:: 
* usage:: 
* description:: 
* epilog:: 
* parents:: 
* formatter_class:: 
* prefix_chars:: 
* fromfile_prefix_chars:: 
* argument_default:: 
* allow_abbrev:: 
* conflict_handler:: 
* add_help:: 
* exit_on_error:: 

The add_argument() method

* name or flags:: 
* action:: 
* nargs:: 
* const:: 
* default:: 
* type:: 
* choices:: 
* required:: 
* help:: 
* metavar:: 
* dest:: 
* Action classes:: 

The parse_args() method

* Option value syntax:: 
* Invalid arguments:: 
* Arguments containing -:: 
* Argument abbreviations (prefix matching): Argument abbreviations prefix matching. 
* Beyond sys.argv: Beyond sys argv. 
* The Namespace object:: 

Other utilities

* Sub-commands:: 
* FileType objects:: 
* Argument groups:: 
* Mutual exclusion:: 
* Parser defaults:: 
* Printing help:: 
* Partial parsing:: 
* Customizing file parsing:: 
* Exiting methods:: 
* Intermixed parsing:: 

logging — Logging facility for Python

* Logger Objects:: 
* Logging Levels:: 
* Handler Objects:: 
* Formatter Objects:: 
* Filter Objects:: 
* LogRecord Objects:: 
* LogRecord attributes:: 
* LoggerAdapter Objects:: 
* Thread Safety:: 
* Module-Level Functions:: 
* Module-Level Attributes:: 
* Integration with the warnings module:: 

logging.config — Logging configuration

* Configuration functions:: 
* Security considerations:: 
* Configuration dictionary schema:: 
* Configuration file format:: 

Configuration dictionary schema

* Dictionary Schema Details:: 
* Incremental Configuration:: 
* Object connections:: 
* User-defined objects:: 
* Access to external objects:: 
* Access to internal objects:: 
* Import resolution and custom importers:: 

logging.handlers — Logging handlers

* StreamHandler:: 
* FileHandler:: 
* NullHandler:: 
* WatchedFileHandler:: 
* BaseRotatingHandler:: 
* RotatingFileHandler:: 
* TimedRotatingFileHandler:: 
* SocketHandler:: 
* DatagramHandler:: 
* SysLogHandler:: 
* NTEventLogHandler:: 
* SMTPHandler:: 
* MemoryHandler:: 
* HTTPHandler:: 
* QueueHandler:: 
* QueueListener:: 

curses — Terminal handling for character-cell displays

* Functions: Functions<4>. 
* Window Objects:: 
* Constants: Constants<6>. 

curses.textpad — Text input widget for curses programs

* Textbox objects:: 

curses.panel — A panel stack extension for curses

* Functions: Functions<5>. 
* Panel Objects:: 

platform —  Access to underlying platform’s identifying data

* Cross Platform:: 
* Java Platform:: 
* Windows Platform:: 
* macOS Platform:: 
* Unix Platforms:: 
* Linux Platforms:: 

ctypes — A foreign function library for Python

* ctypes tutorial:: 
* ctypes reference:: 

ctypes tutorial

* Loading dynamic link libraries:: 
* Accessing functions from loaded dlls:: 
* Calling functions:: 
* Fundamental data types:: 
* Calling functions@comma{} continued: Calling functions continued. 
* Calling functions with your own custom data types:: 
* Specifying the required argument types (function prototypes): Specifying the required argument types function prototypes. 
* Return types:: 
* Passing pointers (or; passing parameters by reference): Passing pointers or passing parameters by reference. 
* Structures and unions:: 
* Structure/union alignment and byte order:: 
* Bit fields in structures and unions:: 
* Arrays:: 
* Pointers:: 
* Type conversions:: 
* Incomplete Types:: 
* Callback functions:: 
* Accessing values exported from dlls:: 
* Surprises:: 
* Variable-sized data types:: 

ctypes reference

* Finding shared libraries:: 
* Loading shared libraries:: 
* Foreign functions:: 
* Function prototypes:: 
* Utility functions:: 
* Data types:: 
* Fundamental data types: Fundamental data types<2>. 
* Structured data types:: 
* Arrays and pointers:: 

Concurrent Execution

* threading — Thread-based parallelism:: 
* multiprocessing — Process-based parallelism:: 
* multiprocessing.shared_memory — Provides shared memory for direct access across processes: multiprocessing shared_memory — Provides shared memory for direct access across processes. 
* The concurrent package:: 
* concurrent.futures — Launching parallel tasks: concurrent futures — Launching parallel tasks. 
* subprocess — Subprocess management:: 
* sched — Event scheduler:: 
* queue — A synchronized queue class:: 
* contextvars — Context Variables:: 
* _thread — Low-level threading API:: 

threading — Thread-based parallelism

* Thread-Local Data:: 
* Thread Objects:: 
* Lock Objects:: 
* RLock Objects:: 
* Condition Objects:: 
* Semaphore Objects:: 
* Event Objects:: 
* Timer Objects:: 
* Barrier Objects:: 
* Using locks@comma{} conditions@comma{} and semaphores in the with statement: Using locks conditions and semaphores in the with statement. 

Semaphore Objects

* Semaphore Example:: 

multiprocessing — Process-based parallelism

* Introduction: Introduction<8>. 
* Reference:: 
* Programming guidelines:: 
* Examples: Examples<12>. 

Introduction

* The Process class:: 
* Contexts and start methods:: 
* Exchanging objects between processes:: 
* Synchronization between processes:: 
* Sharing state between processes:: 
* Using a pool of workers:: 

Reference

* Process and exceptions:: 
* Pipes and Queues:: 
* Miscellaneous: Miscellaneous<3>. 
* Connection Objects: Connection Objects<2>. 
* Synchronization primitives:: 
* Shared ctypes Objects:: 
* Managers:: 
* Proxy Objects:: 
* Process Pools:: 
* Listeners and Clients:: 
* Authentication keys:: 
* Logging: Logging<2>. 
* The multiprocessing.dummy module: The multiprocessing dummy module. 

Shared ctypes Objects

* The multiprocessing.sharedctypes module: The multiprocessing sharedctypes module. 

Managers

* Customized managers:: 
* Using a remote manager:: 

Proxy Objects

* Cleanup:: 

Listeners and Clients

* Address Formats:: 

Programming guidelines

* All start methods:: 
* The spawn and forkserver start methods:: 

concurrent.futures — Launching parallel tasks

* Executor Objects:: 
* ThreadPoolExecutor:: 
* ProcessPoolExecutor:: 
* Future Objects:: 
* Module Functions:: 
* Exception classes:: 

ThreadPoolExecutor

* ThreadPoolExecutor Example:: 

ProcessPoolExecutor

* ProcessPoolExecutor Example:: 

subprocess — Subprocess management

* Using the subprocess Module:: 
* Security Considerations:: 
* Popen Objects:: 
* Windows Popen Helpers:: 
* Older high-level API:: 
* Replacing Older Functions with the subprocess Module:: 
* Legacy Shell Invocation Functions:: 
* Notes: Notes<2>. 

Using the subprocess Module

* Frequently Used Arguments:: 
* Popen Constructor:: 
* Exceptions: Exceptions<8>. 

Windows Popen Helpers

* Windows Constants:: 

Replacing Older Functions with the subprocess Module

* Replacing /bin/sh shell command substitution:: 
* Replacing shell pipeline:: 
* Replacing os.system(): Replacing os system. 
* Replacing the os.spawn family: Replacing the os spawn family. 
* Replacing os.popen()@comma{} os.popen2()@comma{} os.popen3(): Replacing os popen os popen2 os popen3. 
* Replacing functions from the popen2 module:: 

Notes

* Converting an argument sequence to a string on Windows:: 

sched — Event scheduler

* Scheduler Objects:: 

queue — A synchronized queue class

* Queue Objects:: 
* SimpleQueue Objects:: 

contextvars — Context Variables

* Context Variables:: 
* Manual Context Management:: 
* asyncio support:: 

Networking and Interprocess Communication

* asyncio — Asynchronous I/O:: 
* socket — Low-level networking interface:: 
* ssl — TLS/SSL wrapper for socket objects:: 
* select — Waiting for I/O completion:: 
* selectors — High-level I/O multiplexing:: 
* asyncore — Asynchronous socket handler:: 
* asynchat — Asynchronous socket command/response handler:: 
* signal — Set handlers for asynchronous events:: 
* mmap — Memory-mapped file support:: 

asyncio — Asynchronous I/O

* Coroutines and Tasks:: 
* Streams:: 
* Synchronization Primitives:: 
* Subprocesses:: 
* Queues:: 
* Exceptions: Exceptions<10>. 
* Event Loop:: 
* Futures:: 
* Transports and Protocols:: 
* Policies:: 
* Platform Support:: 
* High-level API Index:: 
* Low-level API Index:: 
* Developing with asyncio:: 

Coroutines and Tasks

* Coroutines: Coroutines<3>. 
* Awaitables:: 
* Running an asyncio Program:: 
* Creating Tasks:: 
* Sleeping:: 
* Running Tasks Concurrently:: 
* Shielding From Cancellation:: 
* Timeouts:: 
* Waiting Primitives:: 
* Running in Threads:: 
* Scheduling From Other Threads:: 
* Introspection:: 
* Task Object:: 

Streams

* StreamReader:: 
* StreamWriter:: 
* Examples: Examples<13>. 

Examples

* TCP echo client using streams:: 
* TCP echo server using streams:: 
* Get HTTP headers:: 
* Register an open socket to wait for data using streams:: 

Synchronization Primitives

* Lock:: 
* Event:: 
* Condition:: 
* Semaphore:: 
* BoundedSemaphore:: 

Subprocesses

* Creating Subprocesses:: 
* Constants: Constants<7>. 
* Interacting with Subprocesses:: 

Interacting with Subprocesses

* Subprocess and Threads:: 
* Examples: Examples<14>. 

Queues

* Queue:: 
* Priority Queue:: 
* LIFO Queue:: 
* Exceptions: Exceptions<9>. 
* Examples: Examples<15>. 

Event Loop

* Event Loop Methods:: 
* Callback Handles:: 
* Server Objects:: 
* Event Loop Implementations:: 
* Examples: Examples<16>. 

Event Loop Methods

* Running and stopping the loop:: 
* Scheduling callbacks:: 
* Scheduling delayed callbacks:: 
* Creating Futures and Tasks:: 
* Opening network connections:: 
* Creating network servers:: 
* Transferring files:: 
* TLS Upgrade:: 
* Watching file descriptors:: 
* Working with socket objects directly:: 
* DNS:: 
* Working with pipes:: 
* Unix signals:: 
* Executing code in thread or process pools:: 
* Error Handling API:: 
* Enabling debug mode:: 
* Running Subprocesses:: 

Examples

* Hello World with call_soon(): Hello World with call_soon. 
* Display the current date with call_later(): Display the current date with call_later. 
* Watch a file descriptor for read events:: 
* Set signal handlers for SIGINT and SIGTERM:: 

Futures

* Future Functions:: 
* Future Object:: 

Transports and Protocols

* Transports:: 
* Protocols:: 
* Examples: Examples<17>. 

Transports

* Transports Hierarchy:: 
* Base Transport:: 
* Read-only Transports:: 
* Write-only Transports:: 
* Datagram Transports:: 
* Subprocess Transports:: 

Protocols

* Base Protocols:: 
* Base Protocol:: 
* Streaming Protocols:: 
* Buffered Streaming Protocols:: 
* Datagram Protocols:: 
* Subprocess Protocols:: 

Examples

* TCP Echo Server:: 
* TCP Echo Client:: 
* UDP Echo Server:: 
* UDP Echo Client:: 
* Connecting Existing Sockets:: 
* loop.subprocess_exec() and SubprocessProtocol: loop subprocess_exec and SubprocessProtocol. 

Policies

* Getting and Setting the Policy:: 
* Policy Objects:: 
* Process Watchers:: 
* Custom Policies:: 

Platform Support

* All Platforms:: 
* Windows: Windows<55>. 
* macOS: macOS<30>. 

Windows

* Subprocess Support on Windows:: 

High-level API Index

* Tasks:: 
* Queues: Queues<2>. 
* Subprocesses: Subprocesses<2>. 
* Streams: Streams<2>. 
* Synchronization:: 
* Exceptions: Exceptions<11>. 

Low-level API Index

* Obtaining the Event Loop:: 
* Event Loop Methods: Event Loop Methods<2>. 
* Transports: Transports<2>. 
* Protocols: Protocols<2>. 
* Event Loop Policies:: 

Developing with asyncio

* Debug Mode:: 
* Concurrency and Multithreading:: 
* Running Blocking Code:: 
* Logging: Logging<3>. 
* Detect never-awaited coroutines:: 
* Detect never-retrieved exceptions:: 

socket — Low-level networking interface

* Socket families:: 
* Module contents:: 
* Socket Objects:: 
* Notes on socket timeouts:: 
* Example: Example<8>. 

Module contents

* Exceptions: Exceptions<12>. 
* Constants: Constants<8>. 
* Functions: Functions<6>. 

Functions

* Creating sockets:: 
* Other functions: Other functions<2>. 

Notes on socket timeouts

* Timeouts and the connect method:: 
* Timeouts and the accept method:: 

ssl — TLS/SSL wrapper for socket objects

* Functions@comma{} Constants@comma{} and Exceptions: Functions Constants and Exceptions. 
* SSL Sockets:: 
* SSL Contexts:: 
* Certificates:: 
* Examples: Examples<18>. 
* Notes on non-blocking sockets:: 
* Memory BIO Support: Memory BIO Support<2>. 
* SSL session:: 
* Security considerations: Security considerations<2>. 
* TLS 1.3: TLS 1 3. 

Functions, Constants, and Exceptions

* Socket creation:: 
* Context creation:: 
* Exceptions: Exceptions<13>. 
* Random generation:: 
* Certificate handling:: 
* Constants: Constants<9>. 

Certificates

* Certificate chains:: 
* CA certificates:: 
* Combined key and certificate:: 
* Self-signed certificates:: 

Examples

* Testing for SSL support:: 
* Client-side operation:: 
* Server-side operation:: 

Security considerations

* Best defaults:: 
* Manual settings:: 
* Multi-processing:: 

Manual settings

* Verifying certificates:: 
* Protocol versions:: 
* Cipher selection:: 

select — Waiting for I/O completion

* /dev/poll Polling Objects:: 
* Edge and Level Trigger Polling (epoll) Objects: Edge and Level Trigger Polling epoll Objects. 
* Polling Objects:: 
* Kqueue Objects:: 
* Kevent Objects:: 

selectors — High-level I/O multiplexing

* Introduction: Introduction<9>. 
* Classes: Classes<3>. 
* Examples: Examples<19>. 

asyncore — Asynchronous socket handler

* asyncore Example basic HTTP client:: 
* asyncore Example basic echo server:: 

asynchat — Asynchronous socket command/response handler

* asynchat Example:: 

signal — Set handlers for asynchronous events

* General rules:: 
* Module contents: Module contents<2>. 
* Examples: Examples<20>. 
* Note on SIGPIPE:: 

General rules

* Execution of Python signal handlers:: 
* Signals and threads:: 

mmap — Memory-mapped file support

* MADV_* Constants:: 
* MAP_* Constants:: 

Internet Data Handling

* email — An email and MIME handling package:: 
* json — JSON encoder and decoder:: 
* mailcap — Mailcap file handling:: 
* mailbox — Manipulate mailboxes in various formats:: 
* mimetypes — Map filenames to MIME types:: 
* base64 — Base16@comma{} Base32@comma{} Base64@comma{} Base85 Data Encodings: base64 — Base16 Base32 Base64 Base85 Data Encodings. 
* binascii — Convert between binary and ASCII:: 
* quopri — Encode and decode MIME quoted-printable data:: 
* uu — Encode and decode uuencode files:: 

email — An email and MIME handling package

* email.message; Representing an email message: email message Representing an email message. 
* email.parser; Parsing email messages: email parser Parsing email messages. 
* email.generator; Generating MIME documents: email generator Generating MIME documents. 
* email.policy; Policy Objects: email policy Policy Objects. 
* email.errors; Exception and Defect classes: email errors Exception and Defect classes. 
* email.headerregistry; Custom Header Objects: email headerregistry Custom Header Objects. 
* email.contentmanager; Managing MIME Content: email contentmanager Managing MIME Content. 
* email; Examples: email Examples. 
* email.message.Message; Representing an email message using the compat32 API: email message Message Representing an email message using the compat32 API. 
* email.mime; Creating email and MIME objects from scratch: email mime Creating email and MIME objects from scratch. 
* email.header; Internationalized headers: email header Internationalized headers. 
* email.charset; Representing character sets: email charset Representing character sets. 
* email.encoders; Encoders: email encoders Encoders. 
* email.utils; Miscellaneous utilities: email utils Miscellaneous utilities. 
* email.iterators; Iterators: email iterators Iterators. 

email.parser: Parsing email messages

* FeedParser API:: 
* Parser API:: 
* Additional notes:: 

email.contentmanager: Managing MIME Content

* Content Manager Instances:: 

json — JSON encoder and decoder

* Basic Usage:: 
* Encoders and Decoders:: 
* Exceptions: Exceptions<14>. 
* Standard Compliance and Interoperability:: 
* Command Line Interface: Command Line Interface<2>. 

Standard Compliance and Interoperability

* Character Encodings:: 
* Infinite and NaN Number Values:: 
* Repeated Names Within an Object:: 
* Top-level Non-Object@comma{} Non-Array Values: Top-level Non-Object Non-Array Values. 
* Implementation Limitations:: 

Command Line Interface

* Command line options: Command line options<2>. 

mailbox — Manipulate mailboxes in various formats

* Mailbox objects:: 
* Message objects:: 
* Exceptions: Exceptions<15>. 
* Examples: Examples<21>. 

Mailbox objects

* Maildir:: 
* mbox:: 
* MH:: 
* Babyl:: 
* MMDF:: 

Message objects

* MaildirMessage:: 
* mboxMessage:: 
* MHMessage:: 
* BabylMessage:: 
* MMDFMessage:: 

mimetypes — Map filenames to MIME types

* MimeTypes Objects:: 

base64 — Base16, Base32, Base64, Base85 Data Encodings

* Security Considerations: Security Considerations<2>. 

Structured Markup Processing Tools

* html — HyperText Markup Language support:: 
* html.parser — Simple HTML and XHTML parser: html parser — Simple HTML and XHTML parser. 
* html.entities — Definitions of HTML general entities: html entities — Definitions of HTML general entities. 
* XML Processing Modules:: 
* xml.etree.ElementTree — The ElementTree XML API: xml etree ElementTree — The ElementTree XML API. 
* xml.dom — The Document Object Model API: xml dom — The Document Object Model API. 
* xml.dom.minidom — Minimal DOM implementation: xml dom minidom — Minimal DOM implementation. 
* xml.dom.pulldom — Support for building partial DOM trees: xml dom pulldom — Support for building partial DOM trees. 
* xml.sax — Support for SAX2 parsers: xml sax — Support for SAX2 parsers. 
* xml.sax.handler — Base classes for SAX handlers: xml sax handler — Base classes for SAX handlers. 
* xml.sax.saxutils — SAX Utilities: xml sax saxutils — SAX Utilities. 
* xml.sax.xmlreader — Interface for XML parsers: xml sax xmlreader — Interface for XML parsers. 
* xml.parsers.expat — Fast XML parsing using Expat: xml parsers expat — Fast XML parsing using Expat. 

html.parser — Simple HTML and XHTML parser

* Example HTML Parser Application:: 
* HTMLParser Methods:: 
* Examples: Examples<22>. 

XML Processing Modules

* XML vulnerabilities:: 
* The defusedxml Package:: 

xml.etree.ElementTree — The ElementTree XML API

* Tutorial:: 
* XPath support:: 
* Reference: Reference<2>. 
* XInclude support:: 
* Reference: Reference<3>. 

Tutorial

* XML tree and elements:: 
* Parsing XML:: 
* Pull API for non-blocking parsing:: 
* Finding interesting elements:: 
* Modifying an XML File:: 
* Building XML documents:: 
* Parsing XML with Namespaces:: 

XPath support

* Example: Example<9>. 
* Supported XPath syntax:: 

Reference

* Functions: Functions<7>. 

XInclude support

* Example: Example<10>. 

Reference

* Functions: Functions<8>. 
* Element Objects:: 
* ElementTree Objects:: 
* QName Objects:: 
* TreeBuilder Objects:: 
* XMLParser Objects:: 
* XMLPullParser Objects:: 
* Exceptions: Exceptions<16>. 

xml.dom — The Document Object Model API

* Module Contents: Module Contents<4>. 
* Objects in the DOM:: 
* Conformance:: 

Objects in the DOM

* DOMImplementation Objects:: 
* Node Objects:: 
* NodeList Objects:: 
* DocumentType Objects:: 
* Document Objects:: 
* Element Objects: Element Objects<2>. 
* Attr Objects:: 
* NamedNodeMap Objects:: 
* Comment Objects:: 
* Text and CDATASection Objects:: 
* ProcessingInstruction Objects:: 
* Exceptions: Exceptions<17>. 

Conformance

* Type Mapping:: 
* Accessor Methods:: 

xml.dom.minidom — Minimal DOM implementation

* DOM Objects:: 
* DOM Example:: 
* minidom and the DOM standard:: 

xml.dom.pulldom — Support for building partial DOM trees

* DOMEventStream Objects:: 

xml.sax — Support for SAX2 parsers

* SAXException Objects:: 

xml.sax.handler — Base classes for SAX handlers

* ContentHandler Objects:: 
* DTDHandler Objects:: 
* EntityResolver Objects:: 
* ErrorHandler Objects:: 
* LexicalHandler Objects:: 

xml.sax.xmlreader — Interface for XML parsers

* XMLReader Objects:: 
* IncrementalParser Objects:: 
* Locator Objects:: 
* InputSource Objects:: 
* The Attributes Interface:: 
* The AttributesNS Interface:: 

xml.parsers.expat — Fast XML parsing using Expat

* XMLParser Objects: XMLParser Objects<2>. 
* ExpatError Exceptions:: 
* Example: Example<11>. 
* Content Model Descriptions:: 
* Expat error constants:: 

Internet Protocols and Support

* webbrowser — Convenient web-browser controller:: 
* cgi — Common Gateway Interface support:: 
* cgitb — Traceback manager for CGI scripts:: 
* wsgiref — WSGI Utilities and Reference Implementation:: 
* urllib — URL handling modules:: 
* urllib.request — Extensible library for opening URLs: urllib request — Extensible library for opening URLs. 
* urllib.response — Response classes used by urllib: urllib response — Response classes used by urllib. 
* urllib.parse — Parse URLs into components: urllib parse — Parse URLs into components. 
* urllib.error — Exception classes raised by urllib.request: urllib error — Exception classes raised by urllib request. 
* urllib.robotparser — Parser for robots.txt: urllib robotparser — Parser for robots txt. 
* http — HTTP modules:: 
* http.client — HTTP protocol client: http client — HTTP protocol client. 
* ftplib — FTP protocol client:: 
* poplib — POP3 protocol client:: 
* imaplib — IMAP4 protocol client:: 
* nntplib — NNTP protocol client:: 
* smtplib — SMTP protocol client:: 
* smtpd — SMTP Server:: 
* telnetlib — Telnet client:: 
* uuid — UUID objects according to RFC 4122:: 
* socketserver — A framework for network servers:: 
* http.server — HTTP servers: http server — HTTP servers. 
* http.cookies — HTTP state management: http cookies — HTTP state management. 
* http.cookiejar — Cookie handling for HTTP clients: http cookiejar — Cookie handling for HTTP clients. 
* xmlrpc — XMLRPC server and client modules:: 
* xmlrpc.client — XML-RPC client access: xmlrpc client — XML-RPC client access. 
* xmlrpc.server — Basic XML-RPC servers: xmlrpc server — Basic XML-RPC servers. 
* ipaddress — IPv4/IPv6 manipulation library:: 

webbrowser — Convenient web-browser controller

* Browser Controller Objects:: 

cgi — Common Gateway Interface support

* Introduction: Introduction<10>. 
* Using the cgi module:: 
* Higher Level Interface:: 
* Functions: Functions<9>. 
* Caring about security:: 
* Installing your CGI script on a Unix system:: 
* Testing your CGI script:: 
* Debugging CGI scripts:: 
* Common problems and solutions:: 

wsgiref — WSGI Utilities and Reference Implementation

* wsgiref.util – WSGI environment utilities: wsgiref util – WSGI environment utilities. 
* wsgiref.headers – WSGI response header tools: wsgiref headers – WSGI response header tools. 
* wsgiref.simple_server – a simple WSGI HTTP server: wsgiref simple_server – a simple WSGI HTTP server. 
* wsgiref.validate — WSGI conformance checker: wsgiref validate — WSGI conformance checker. 
* wsgiref.handlers – server/gateway base classes: wsgiref handlers – server/gateway base classes. 
* Examples: Examples<23>. 

urllib.request — Extensible library for opening URLs

* Request Objects:: 
* OpenerDirector Objects:: 
* BaseHandler Objects:: 
* HTTPRedirectHandler Objects:: 
* HTTPCookieProcessor Objects:: 
* ProxyHandler Objects:: 
* HTTPPasswordMgr Objects:: 
* HTTPPasswordMgrWithPriorAuth Objects:: 
* AbstractBasicAuthHandler Objects:: 
* HTTPBasicAuthHandler Objects:: 
* ProxyBasicAuthHandler Objects:: 
* AbstractDigestAuthHandler Objects:: 
* HTTPDigestAuthHandler Objects:: 
* ProxyDigestAuthHandler Objects:: 
* HTTPHandler Objects:: 
* HTTPSHandler Objects:: 
* FileHandler Objects:: 
* DataHandler Objects:: 
* FTPHandler Objects:: 
* CacheFTPHandler Objects:: 
* UnknownHandler Objects:: 
* HTTPErrorProcessor Objects:: 
* Examples: Examples<24>. 
* Legacy interface:: 
* urllib.request Restrictions: urllib request Restrictions. 

urllib.parse — Parse URLs into components

* URL Parsing:: 
* Parsing ASCII Encoded Bytes:: 
* Structured Parse Results:: 
* URL Quoting:: 

http — HTTP modules

* HTTP status codes:: 

http.client — HTTP protocol client

* HTTPConnection Objects:: 
* HTTPResponse Objects:: 
* Examples: Examples<25>. 
* HTTPMessage Objects:: 

ftplib — FTP protocol client

* FTP Objects:: 
* FTP_TLS Objects:: 

poplib — POP3 protocol client

* POP3 Objects:: 
* POP3 Example:: 

imaplib — IMAP4 protocol client

* IMAP4 Objects:: 
* IMAP4 Example:: 

nntplib — NNTP protocol client

* NNTP Objects:: 
* Utility functions: Utility functions<2>. 

NNTP Objects

* Attributes:: 
* Methods: Methods<3>. 

smtplib — SMTP protocol client

* SMTP Objects:: 
* SMTP Example:: 

smtpd — SMTP Server

* SMTPServer Objects:: 
* DebuggingServer Objects:: 
* PureProxy Objects:: 
* SMTPChannel Objects:: 

telnetlib — Telnet client

* Telnet Objects:: 
* Telnet Example:: 

uuid — UUID objects according to RFC 4122

* Example: Example<12>. 

socketserver — A framework for network servers

* Server Creation Notes:: 
* Server Objects: Server Objects<2>. 
* Request Handler Objects:: 
* Examples: Examples<26>. 

Examples

* socketserver.TCPServer Example: socketserver TCPServer Example. 
* socketserver.UDPServer Example: socketserver UDPServer Example. 
* Asynchronous Mixins:: 

http.cookies — HTTP state management

* Cookie Objects:: 
* Morsel Objects:: 
* Example: Example<13>. 

http.cookiejar — Cookie handling for HTTP clients

* CookieJar and FileCookieJar Objects:: 
* FileCookieJar subclasses and co-operation with web browsers:: 
* CookiePolicy Objects:: 
* DefaultCookiePolicy Objects:: 
* Cookie Objects: Cookie Objects<2>. 
* Examples: Examples<27>. 

xmlrpc.client — XML-RPC client access

* ServerProxy Objects:: 
* DateTime Objects:: 
* Binary Objects:: 
* Fault Objects:: 
* ProtocolError Objects:: 
* MultiCall Objects:: 
* Convenience Functions:: 
* Example of Client Usage:: 
* Example of Client and Server Usage:: 

xmlrpc.server — Basic XML-RPC servers

* SimpleXMLRPCServer Objects:: 
* CGIXMLRPCRequestHandler:: 
* Documenting XMLRPC server:: 
* DocXMLRPCServer Objects:: 
* DocCGIXMLRPCRequestHandler:: 

SimpleXMLRPCServer Objects

* SimpleXMLRPCServer Example:: 

ipaddress — IPv4/IPv6 manipulation library

* Convenience factory functions:: 
* IP Addresses:: 
* IP Network definitions:: 
* Interface objects:: 
* Other Module Level Functions:: 
* Custom Exceptions:: 

IP Addresses

* Address objects:: 
* Conversion to Strings and Integers:: 
* Operators: Operators<3>. 

Operators

* Comparison operators:: 
* Arithmetic operators:: 

IP Network definitions

* Prefix@comma{} net mask and host mask: Prefix net mask and host mask. 
* Network objects:: 
* Operators: Operators<4>. 

Operators

* Logical operators:: 
* Iteration:: 
* Networks as containers of addresses:: 

Interface objects

* Operators: Operators<5>. 

Operators

* Logical operators: Logical operators<2>. 

Multimedia Services

* audioop — Manipulate raw audio data:: 
* aifc — Read and write AIFF and AIFC files:: 
* sunau — Read and write Sun AU files:: 
* wave — Read and write WAV files:: 
* chunk — Read IFF chunked data:: 
* colorsys — Conversions between color systems:: 
* imghdr — Determine the type of an image:: 
* sndhdr — Determine type of sound file:: 
* ossaudiodev — Access to OSS-compatible audio devices:: 

sunau — Read and write Sun AU files

* AU_read Objects:: 
* AU_write Objects:: 

wave — Read and write WAV files

* Wave_read Objects:: 
* Wave_write Objects:: 

ossaudiodev — Access to OSS-compatible audio devices

* Audio Device Objects:: 
* Mixer Device Objects:: 

Internationalization

* gettext — Multilingual internationalization services:: 
* locale — Internationalization services:: 

gettext — Multilingual internationalization services

* GNU gettext API:: 
* Class-based API:: 
* Internationalizing your programs and modules:: 
* Acknowledgements: Acknowledgements<9>. 

Class-based API

* The NullTranslations class:: 
* The GNUTranslations class:: 
* Solaris message catalog support:: 
* The Catalog constructor:: 

Internationalizing your programs and modules

* Localizing your module:: 
* Localizing your application:: 
* Changing languages on the fly:: 
* Deferred translations:: 

locale — Internationalization services

* Background@comma{} details@comma{} hints@comma{} tips and caveats: Background details hints tips and caveats. 
* For extension writers and programs that embed Python:: 
* Access to message catalogs:: 

Program Frameworks

* turtle — Turtle graphics:: 
* cmd — Support for line-oriented command interpreters:: 
* shlex — Simple lexical analysis:: 

turtle — Turtle graphics

* Introduction: Introduction<11>. 
* Overview of available Turtle and Screen methods:: 
* Methods of RawTurtle/Turtle and corresponding functions:: 
* Methods of TurtleScreen/Screen and corresponding functions:: 
* Public classes:: 
* Help and configuration:: 
* turtledemo — Demo scripts:: 
* Changes since Python 2.6: Changes since Python 2 6. 
* Changes since Python 3.0: Changes since Python 3 0. 

Overview of available Turtle and Screen methods

* Turtle methods:: 
* Methods of TurtleScreen/Screen:: 

Methods of RawTurtle/Turtle and corresponding functions

* Turtle motion:: 
* Tell Turtle’s state:: 
* Settings for measurement:: 
* Pen control:: 
* Turtle state:: 
* Using events:: 
* Special Turtle methods:: 
* Compound shapes:: 

Pen control

* Drawing state:: 
* Color control:: 
* Filling:: 
* More drawing control:: 

Turtle state

* Visibility:: 
* Appearance:: 

Methods of TurtleScreen/Screen and corresponding functions

* Window control:: 
* Animation control:: 
* Using screen events:: 
* Input methods:: 
* Settings and special methods:: 
* Methods specific to Screen@comma{} not inherited from TurtleScreen: Methods specific to Screen not inherited from TurtleScreen. 

Help and configuration

* How to use help:: 
* Translation of docstrings into different languages:: 
* How to configure Screen and Turtles:: 

cmd — Support for line-oriented command interpreters

* Cmd Objects:: 
* Cmd Example:: 

shlex — Simple lexical analysis

* shlex Objects:: 
* Parsing Rules:: 
* Improved Compatibility with Shells:: 

Graphical User Interfaces with Tk

* tkinter — Python interface to Tcl/Tk:: 
* tkinter.colorchooser — Color choosing dialog: tkinter colorchooser — Color choosing dialog. 
* tkinter.font — Tkinter font wrapper: tkinter font — Tkinter font wrapper. 
* Tkinter Dialogs:: 
* tkinter.messagebox — Tkinter message prompts: tkinter messagebox — Tkinter message prompts. 
* tkinter.scrolledtext — Scrolled Text Widget: tkinter scrolledtext — Scrolled Text Widget. 
* tkinter.dnd — Drag and drop support: tkinter dnd — Drag and drop support. 
* tkinter.ttk — Tk themed widgets: tkinter ttk — Tk themed widgets. 
* tkinter.tix — Extension widgets for Tk: tkinter tix — Extension widgets for Tk. 
* IDLE: IDLE<51>. 

tkinter — Python interface to Tcl/Tk

* Architecture:: 
* Tkinter Modules:: 
* Tkinter Life Preserver:: 
* Threading model:: 
* Handy Reference:: 
* File Handlers:: 

Tkinter Life Preserver

* A Hello World Program:: 
* Important Tk Concepts:: 
* Understanding How Tkinter Wraps Tcl/Tk:: 
* How do I…? What option does…?:: 
* Navigating the Tcl/Tk Reference Manual:: 

Handy Reference

* Setting Options:: 
* The Packer:: 
* Packer Options:: 
* Coupling Widget Variables:: 
* The Window Manager:: 
* Tk Option Data Types:: 
* Bindings and Events:: 
* The index Parameter:: 
* Images:: 

Tkinter Dialogs

* tkinter.simpledialog — Standard Tkinter input dialogs: tkinter simpledialog — Standard Tkinter input dialogs. 
* tkinter.filedialog — File selection dialogs: tkinter filedialog — File selection dialogs. 
* tkinter.commondialog — Dialog window templates: tkinter commondialog — Dialog window templates. 

tkinter.filedialog — File selection dialogs

* Native Load/Save Dialogs:: 

tkinter.ttk — Tk themed widgets

* Using Ttk:: 
* Ttk Widgets:: 
* Widget:: 
* Combobox:: 
* Spinbox:: 
* Notebook:: 
* Progressbar:: 
* Separator:: 
* Sizegrip:: 
* Treeview:: 
* Ttk Styling:: 

Widget

* Standard Options:: 
* Scrollable Widget Options:: 
* Label Options:: 
* Compatibility Options:: 
* Widget States:: 
* ttk.Widget: ttk Widget. 

Combobox

* Options:: 
* Virtual events:: 
* ttk.Combobox: ttk Combobox. 

Spinbox

* Options: Options<2>. 
* Virtual events: Virtual events<2>. 
* ttk.Spinbox: ttk Spinbox. 

Notebook

* Options: Options<3>. 
* Tab Options:: 
* Tab Identifiers:: 
* Virtual Events:: 
* ttk.Notebook: ttk Notebook. 

Progressbar

* Options: Options<4>. 
* ttk.Progressbar: ttk Progressbar. 

Separator

* Options: Options<5>. 

Sizegrip

* Platform-specific notes:: 
* Bugs:: 

Treeview

* Options: Options<6>. 
* Item Options:: 
* Tag Options:: 
* Column Identifiers:: 
* Virtual Events: Virtual Events<2>. 
* ttk.Treeview: ttk Treeview. 

Ttk Styling

* Layouts:: 

tkinter.tix — Extension widgets for Tk

* Using Tix:: 
* Tix Widgets:: 
* Tix Commands:: 

Tix Widgets

* Basic Widgets:: 
* File Selectors:: 
* Hierarchical ListBox:: 
* Tabular ListBox:: 
* Manager Widgets:: 
* Image Types:: 
* Miscellaneous Widgets:: 
* Form Geometry Manager:: 

IDLE

* Menus:: 
* Editing and navigation:: 
* Startup and code execution:: 
* Help and preferences:: 

Menus

* File menu (Shell and Editor): File menu Shell and Editor. 
* Edit menu (Shell and Editor): Edit menu Shell and Editor. 
* Format menu (Editor window only): Format menu Editor window only. 
* Run menu (Editor window only): Run menu Editor window only. 
* Shell menu (Shell window only): Shell menu Shell window only. 
* Debug menu (Shell window only): Debug menu Shell window only. 
* Options menu (Shell and Editor): Options menu Shell and Editor. 
* Window menu (Shell and Editor): Window menu Shell and Editor. 
* Help menu (Shell and Editor): Help menu Shell and Editor. 
* Context Menus:: 

Editing and navigation

* Editor windows:: 
* Key bindings:: 
* Automatic indentation:: 
* Completions:: 
* Calltips:: 
* Code Context:: 
* Python Shell window:: 
* Text colors:: 

Startup and code execution

* Command line usage:: 
* Startup failure:: 
* Running user code:: 
* User output in Shell:: 
* Developing tkinter applications:: 
* Running without a subprocess:: 

Help and preferences

* Help sources:: 
* Setting preferences:: 
* IDLE on macOS:: 
* Extensions:: 

Development Tools

* typing — Support for type hints:: 
* pydoc — Documentation generator and online help system:: 
* Python Development Mode:: 
* Effects of the Python Development Mode:: 
* ResourceWarning Example:: 
* Bad file descriptor error example:: 
* doctest — Test interactive Python examples:: 
* unittest — Unit testing framework:: 
* unittest.mock — mock object library: unittest mock — mock object library. 
* unittest.mock — getting started: unittest mock — getting started. 
* 2to3 - Automated Python 2 to 3 code translation:: 
* test — Regression tests package for Python:: 
* test.support — Utilities for the Python test suite: test support — Utilities for the Python test suite. 
* test.support.socket_helper — Utilities for socket tests: test support socket_helper — Utilities for socket tests. 
* test.support.script_helper — Utilities for the Python execution tests: test support script_helper — Utilities for the Python execution tests. 
* test.support.bytecode_helper — Support tools for testing correct bytecode generation: test support bytecode_helper — Support tools for testing correct bytecode generation. 
* test.support.threading_helper — Utilities for threading tests: test support threading_helper — Utilities for threading tests. 
* test.support.os_helper — Utilities for os tests: test support os_helper — Utilities for os tests. 
* test.support.import_helper — Utilities for import tests: test support import_helper — Utilities for import tests. 
* test.support.warnings_helper — Utilities for warnings tests: test support warnings_helper — Utilities for warnings tests. 

typing — Support for type hints

* Relevant PEPs:: 
* Type aliases:: 
* NewType:: 
* Callable:: 
* Generics:: 
* User-defined generic types:: 
* The Any type:: 
* Nominal vs structural subtyping:: 
* Module contents: Module contents<3>. 

Module contents

* Special typing primitives:: 
* Generic concrete collections:: 
* Abstract Base Classes:: 
* Protocols: Protocols<3>. 
* Functions and decorators:: 
* Introspection helpers:: 
* Constant:: 

Special typing primitives

* Special types:: 
* Special forms:: 
* Building generic types:: 
* Other special directives:: 

Generic concrete collections

* Corresponding to built-in types:: 
* Corresponding to types in collections:: 
* Other concrete types:: 

Abstract Base Classes

* Corresponding to collections in collections.abc: Corresponding to collections in collections abc. 
* Corresponding to other types in collections.abc: Corresponding to other types in collections abc. 
* Asynchronous programming:: 
* Context manager types:: 

doctest — Test interactive Python examples

* Simple Usage; Checking Examples in Docstrings: Simple Usage Checking Examples in Docstrings. 
* Simple Usage; Checking Examples in a Text File: Simple Usage Checking Examples in a Text File. 
* How It Works:: 
* Basic API:: 
* Unittest API:: 
* Advanced API:: 
* Debugging:: 
* Soapbox:: 

How It Works

* Which Docstrings Are Examined?:: 
* How are Docstring Examples Recognized?:: 
* What’s the Execution Context?:: 
* What About Exceptions?:: 
* Option Flags:: 
* Directives:: 
* Warnings: Warnings<2>. 

Advanced API

* DocTest Objects:: 
* Example Objects:: 
* DocTestFinder objects:: 
* DocTestParser objects:: 
* DocTestRunner objects:: 
* OutputChecker objects:: 

unittest — Unit testing framework

* Basic example:: 
* Command-Line Interface: Command-Line Interface<3>. 
* Test Discovery:: 
* Organizing test code:: 
* Re-using old test code:: 
* Skipping tests and expected failures:: 
* Distinguishing test iterations using subtests:: 
* Classes and functions:: 
* Class and Module Fixtures:: 
* Signal Handling:: 

Command-Line Interface

* Command-line options: Command-line options<3>. 

Classes and functions

* Test cases:: 
* Grouping tests:: 
* Loading and running tests:: 

Test cases

* Deprecated aliases:: 

Loading and running tests

* load_tests Protocol:: 

Class and Module Fixtures

* setUpClass and tearDownClass:: 
* setUpModule and tearDownModule:: 

unittest.mock — mock object library

* Quick Guide:: 
* The Mock Class:: 
* The patchers:: 
* MagicMock and magic method support:: 
* Helpers:: 

The Mock Class

* Calling:: 
* Deleting Attributes:: 
* Mock names and the name attribute:: 
* Attaching Mocks as Attributes:: 

The patchers

* patch:: 
* patch.object: patch object. 
* patch.dict: patch dict. 
* patch.multiple: patch multiple. 
* patch methods; start and stop: patch methods start and stop. 
* patch builtins:: 
* TEST_PREFIX:: 
* Nesting Patch Decorators:: 
* Where to patch:: 
* Patching Descriptors and Proxy Objects:: 

MagicMock and magic method support

* Mocking Magic Methods:: 
* Magic Mock:: 

Helpers

* sentinel:: 
* DEFAULT:: 
* call:: 
* create_autospec:: 
* ANY:: 
* FILTER_DIR:: 
* mock_open:: 
* Autospeccing:: 
* Sealing mocks:: 

unittest.mock — getting started

* Using Mock:: 
* Patch Decorators:: 
* Further Examples:: 

Using Mock

* Mock Patching Methods:: 
* Mock for Method Calls on an Object:: 
* Mocking Classes:: 
* Naming your mocks:: 
* Tracking all Calls:: 
* Setting Return Values and Attributes:: 
* Raising exceptions with mocks:: 
* Side effect functions and iterables:: 
* Mocking asynchronous iterators:: 
* Mocking asynchronous context manager:: 
* Creating a Mock from an Existing Object:: 

Further Examples

* Mocking chained calls:: 
* Partial mocking:: 
* Mocking a Generator Method:: 
* Applying the same patch to every test method:: 
* Mocking Unbound Methods:: 
* Checking multiple calls with mock:: 
* Coping with mutable arguments:: 
* Nesting Patches:: 
* Mocking a dictionary with MagicMock:: 
* Mock subclasses and their attributes:: 
* Mocking imports with patch.dict: Mocking imports with patch dict. 
* Tracking order of calls and less verbose call assertions:: 
* More complex argument matching:: 

2to3 - Automated Python 2 to 3 code translation

* Using 2to3:: 
* Fixers:: 
* lib2to3 - 2to3’s library:: 

test — Regression tests package for Python

* Writing Unit Tests for the test package:: 
* Running tests using the command-line interface:: 

Debugging and Profiling

* Audit events table:: 
* bdb — Debugger framework:: 
* faulthandler — Dump the Python traceback:: 
* pdb — The Python Debugger:: 
* The Python Profilers:: 
* timeit — Measure execution time of small code snippets:: 
* trace — Trace or track Python statement execution:: 
* tracemalloc — Trace memory allocations:: 

faulthandler — Dump the Python traceback

* Dumping the traceback:: 
* Fault handler state:: 
* Dumping the tracebacks after a timeout:: 
* Dumping the traceback on a user signal:: 
* Issue with file descriptors:: 
* Example: Example<14>. 

pdb — The Python Debugger

* Debugger Commands:: 

The Python Profilers

* Introduction to the profilers:: 
* Instant User’s Manual:: 
* profile and cProfile Module Reference:: 
* The Stats Class:: 
* What Is Deterministic Profiling?:: 
* Limitations:: 
* Calibration:: 
* Using a custom timer:: 

timeit — Measure execution time of small code snippets

* Basic Examples: Basic Examples<2>. 
* Python Interface:: 
* Command-Line Interface: Command-Line Interface<4>. 
* Examples: Examples<28>. 

trace — Trace or track Python statement execution

* Command-Line Usage:: 
* Programmatic Interface:: 

Command-Line Usage

* Main options:: 
* Modifiers:: 
* Filters:: 

tracemalloc — Trace memory allocations

* Examples: Examples<29>. 
* API:: 

Examples

* Display the top 10:: 
* Compute differences:: 
* Get the traceback of a memory block:: 
* Pretty top:: 

Pretty top

* Record the current and peak size of all traced memory blocks:: 

API

* Functions: Functions<10>. 
* DomainFilter:: 
* Filter:: 
* Frame:: 
* Snapshot:: 
* Statistic:: 
* StatisticDiff:: 
* Trace:: 
* Traceback:: 

Software Packaging and Distribution

* distutils — Building and installing Python modules:: 
* ensurepip — Bootstrapping the pip installer:: 
* venv — Creation of virtual environments:: 
* zipapp — Manage executable Python zip archives:: 

ensurepip — Bootstrapping the pip installer

* Command line interface:: 
* Module API:: 

venv — Creation of virtual environments

* Creating virtual environments:: 
* API: API<2>. 
* An example of extending EnvBuilder:: 

zipapp — Manage executable Python zip archives

* Basic Example:: 
* Command-Line Interface: Command-Line Interface<5>. 
* Python API:: 
* Examples: Examples<30>. 
* Specifying the Interpreter:: 
* Creating Standalone Applications with zipapp:: 
* The Python Zip Application Archive Format:: 

Creating Standalone Applications with zipapp

* Making a Windows executable:: 
* Caveats:: 

Python Runtime Services

* sys — System-specific parameters and functions:: 
* sysconfig — Provide access to Python’s configuration information:: 
* builtins — Built-in objects:: 
* __main__ — Top-level code environment:: 
* warnings — Warning control:: 
* dataclasses — Data Classes:: 
* contextlib — Utilities for with-statement contexts:: 
* abc — Abstract Base Classes:: 
* atexit — Exit handlers:: 
* traceback — Print or retrieve a stack traceback:: 
* __future__ — Future statement definitions:: 
* gc — Garbage Collector interface:: 
* inspect — Inspect live objects:: 
* site — Site-specific configuration hook:: 

sysconfig — Provide access to Python’s configuration information

* Configuration variables:: 
* Installation paths:: 
* Other functions: Other functions<3>. 
* Using sysconfig as a script:: 

__main__ — Top-level code environment

* __name__ == '__main__':: 
* __main__.py in Python Packages: __main__ py in Python Packages. 
* import __main__:: 

__name__ == '__main__'

* What is the “top-level code environment”?:: 
* Idiomatic Usage:: 
* Packaging Considerations:: 

__main__.py in Python Packages

* Idiomatic Usage: Idiomatic Usage<2>. 

warnings — Warning control

* Warning Categories:: 
* The Warnings Filter:: 
* Temporarily Suppressing Warnings:: 
* Testing Warnings:: 
* Updating Code For New Versions of Dependencies:: 
* Available Functions:: 
* Available Context Managers:: 

The Warnings Filter

* Describing Warning Filters:: 
* Default Warning Filter:: 
* Overriding the default filter:: 

dataclasses — Data Classes

* Module contents: Module contents<4>. 
* Post-init processing:: 
* Class variables:: 
* Init-only variables:: 
* Frozen instances:: 
* Inheritance: Inheritance<2>. 
* Re-ordering of keyword-only parameters in __init__(): Re-ordering of keyword-only parameters in __init__. 
* Default factory functions:: 
* Mutable default values:: 

contextlib — Utilities for with-statement contexts

* Utilities:: 
* Examples and Recipes: Examples and Recipes<2>. 
* Single use@comma{} reusable and reentrant context managers: Single use reusable and reentrant context managers. 

Examples and Recipes

* Supporting a variable number of context managers:: 
* Catching exceptions from __enter__ methods:: 
* Cleaning up in an __enter__ implementation:: 
* Replacing any use of try-finally and flag variables:: 
* Using a context manager as a function decorator:: 

Single use, reusable and reentrant context managers

* Reentrant context managers:: 
* Reusable context managers:: 

atexit — Exit handlers

* atexit Example:: 

traceback — Print or retrieve a stack traceback

* TracebackException Objects:: 
* StackSummary Objects:: 
* FrameSummary Objects:: 
* Traceback Examples:: 

inspect — Inspect live objects

* Types and members:: 
* Retrieving source code:: 
* Introspecting callables with the Signature object:: 
* Classes and functions: Classes and functions<2>. 
* The interpreter stack:: 
* Fetching attributes statically:: 
* Current State of Generators and Coroutines:: 
* Code Objects Bit Flags:: 
* Command Line Interface: Command Line Interface<3>. 

site — Site-specific configuration hook

* Readline configuration:: 
* Module contents: Module contents<5>. 
* Command Line Interface: Command Line Interface<4>. 

Custom Python Interpreters

* code — Interpreter base classes:: 
* codeop — Compile Python code:: 

code — Interpreter base classes

* Interactive Interpreter Objects:: 
* Interactive Console Objects:: 

Importing Modules

* zipimport — Import modules from Zip archives:: 
* pkgutil — Package extension utility:: 
* modulefinder — Find modules used by a script:: 
* runpy — Locating and executing Python modules:: 
* importlib — The implementation of import:: 
* importlib.resources – Resources: importlib resources – Resources. 
* Using importlib.metadata: Using importlib metadata. 

zipimport — Import modules from Zip archives

* zipimporter Objects:: 
* Examples: Examples<31>. 

modulefinder — Find modules used by a script

* Example usage of ModuleFinder:: 

importlib — The implementation of import

* Introduction: Introduction<12>. 
* Functions: Functions<11>. 
* importlib.abc – Abstract base classes related to import: importlib abc – Abstract base classes related to import. 
* importlib.machinery – Importers and path hooks: importlib machinery – Importers and path hooks. 
* importlib.util – Utility code for importers: importlib util – Utility code for importers. 
* Examples: Examples<32>. 

Examples

* Importing programmatically:: 
* Checking if a module can be imported:: 
* Importing a source file directly:: 
* Implementing lazy imports:: 
* Setting up an importer:: 
* Approximating importlib.import_module(): Approximating importlib import_module. 

Using importlib.metadata

* Overview: Overview<3>. 
* Functional API:: 
* Distributions:: 
* Extending the search algorithm:: 

Functional API

* Entry points:: 
* Distribution metadata:: 
* Distribution versions:: 
* Distribution files:: 
* Distribution requirements:: 
* Package distributions:: 

Python Language Services

* ast — Abstract Syntax Trees:: 
* symtable — Access to the compiler’s symbol tables:: 
* token — Constants used with Python parse trees:: 
* keyword — Testing for Python keywords:: 
* tokenize — Tokenizer for Python source:: 
* tabnanny — Detection of ambiguous indentation:: 
* pyclbr — Python module browser support:: 
* py_compile — Compile Python source files:: 
* compileall — Byte-compile Python libraries:: 
* dis — Disassembler for Python bytecode:: 
* pickletools — Tools for pickle developers:: 

ast — Abstract Syntax Trees

* Abstract Grammar:: 
* Node classes:: 
* ast Helpers:: 
* Compiler Flags:: 
* Command-Line Usage: Command-Line Usage<2>. 

Node classes

* Literals: Literals<3>. 
* Variables:: 
* Expressions: Expressions<2>. 
* Statements:: 
* Control flow:: 
* Pattern matching:: 
* Function and class definitions:: 
* Async and await:: 

Expressions

* Subscripting:: 
* Comprehensions:: 

Statements

* Imports:: 

symtable — Access to the compiler’s symbol tables

* Generating Symbol Tables:: 
* Examining Symbol Tables:: 

tokenize — Tokenizer for Python source

* Tokenizing Input:: 
* Command-Line Usage: Command-Line Usage<3>. 
* Examples: Examples<33>. 

pyclbr — Python module browser support

* Function Objects:: 
* Class Objects: Class Objects<2>. 

py_compile — Compile Python source files

* Command-Line Interface: Command-Line Interface<6>. 

compileall — Byte-compile Python libraries

* Command-line use:: 
* Public functions:: 

dis — Disassembler for Python bytecode

* Bytecode analysis:: 
* Analysis functions:: 
* Python Bytecode Instructions:: 
* Opcode collections:: 

pickletools — Tools for pickle developers

* Command line usage: Command line usage<2>. 
* Programmatic Interface: Programmatic Interface<2>. 

Command line usage

* Command line options: Command line options<3>. 

MS Windows Specific Services

* msilib — Read and write Microsoft Installer files:: 
* msvcrt — Useful routines from the MS VC++ runtime:: 
* winreg — Windows registry access:: 
* winsound — Sound-playing interface for Windows:: 

msilib — Read and write Microsoft Installer files

* Database Objects:: 
* View Objects:: 
* Summary Information Objects:: 
* Record Objects:: 
* Errors:: 
* CAB Objects:: 
* Directory Objects:: 
* Features: Features<3>. 
* GUI classes:: 
* Precomputed tables:: 

msvcrt — Useful routines from the MS VC++ runtime

* File Operations:: 
* Console I/O:: 
* Other Functions:: 

winreg — Windows registry access

* Functions: Functions<12>. 
* Constants: Constants<10>. 
* Registry Handle Objects:: 

Constants

* HKEY_* Constants:: 
* Access Rights:: 
* Value Types:: 

Access Rights

* 64-bit Specific:: 

Unix Specific Services

* posix — The most common POSIX system calls:: 
* pwd — The password database:: 
* spwd — The shadow password database:: 
* grp — The group database:: 
* crypt — Function to check Unix passwords:: 
* termios — POSIX style tty control:: 
* tty — Terminal control functions:: 
* pty — Pseudo-terminal utilities:: 
* fcntl — The fcntl and ioctl system calls:: 
* pipes — Interface to shell pipelines:: 
* resource — Resource usage information:: 
* nis — Interface to Sun’s NIS (Yellow Pages): nis — Interface to Sun’s NIS Yellow Pages. 
* syslog — Unix syslog library routines:: 

posix — The most common POSIX system calls

* Large File Support:: 
* Notable Module Contents:: 

crypt — Function to check Unix passwords

* Hashing Methods:: 
* Module Attributes:: 
* Module Functions: Module Functions<2>. 
* Examples: Examples<34>. 

termios — POSIX style tty control

* Example: Example<15>. 

pty — Pseudo-terminal utilities

* Example: Example<16>. 

pipes — Interface to shell pipelines

* Template Objects:: 

resource — Resource usage information

* Resource Limits:: 
* Resource Usage:: 

syslog — Unix syslog library routines

* Examples: Examples<35>. 

Examples

* Simple example:: 

Superseded Modules

* optparse — Parser for command line options:: 
* imp — Access the import internals:: 

optparse — Parser for command line options

* Background:: 
* Tutorial: Tutorial<2>. 
* Reference Guide:: 
* Option Callbacks:: 
* Extending optparse:: 

Background

* Terminology:: 
* What are options for?:: 
* What are positional arguments for?:: 

Tutorial

* Understanding option actions:: 
* The store action:: 
* Handling boolean (flag) options: Handling boolean flag options. 
* Other actions:: 
* Default values:: 
* Generating help:: 
* Printing a version string:: 
* How optparse handles errors:: 
* Putting it all together:: 

Generating help

* Grouping Options:: 

Reference Guide

* Creating the parser:: 
* Populating the parser:: 
* Defining options:: 
* Option attributes:: 
* Standard option actions:: 
* Standard option types:: 
* Parsing arguments: Parsing arguments<2>. 
* Querying and manipulating your option parser:: 
* Conflicts between options:: 
* Cleanup: Cleanup<2>. 
* Other methods:: 

Option Callbacks

* Defining a callback option:: 
* How callbacks are called:: 
* Raising errors in a callback:: 
* Callback example 1; trivial callback: Callback example 1 trivial callback. 
* Callback example 2; check option order: Callback example 2 check option order. 
* Callback example 3; check option order (generalized): Callback example 3 check option order generalized. 
* Callback example 4; check arbitrary condition: Callback example 4 check arbitrary condition. 
* Callback example 5; fixed arguments: Callback example 5 fixed arguments. 
* Callback example 6; variable arguments: Callback example 6 variable arguments. 

Extending optparse

* Adding new types:: 
* Adding new actions:: 

imp — Access the import internals

* Examples: Examples<36>. 

Undocumented Modules

* Platform specific modules:: 

Extending and Embedding the Python Interpreter

* Recommended third party tools:: 
* Creating extensions without third party tools:: 
* Embedding the CPython runtime in a larger application:: 

Creating extensions without third party tools

* Extending Python with C or C++:: 
* Defining Extension Types; Tutorial: Defining Extension Types Tutorial. 
* Defining Extension Types; Assorted Topics: Defining Extension Types Assorted Topics. 
* Building C and C++ Extensions:: 
* Building C and C++ Extensions on Windows:: 

Extending Python with C or C++

* A Simple Example:: 
* Intermezzo; Errors and Exceptions: Intermezzo Errors and Exceptions. 
* Back to the Example:: 
* The Module’s Method Table and Initialization Function:: 
* Compilation and Linkage:: 
* Calling Python Functions from C:: 
* Extracting Parameters in Extension Functions:: 
* Keyword Parameters for Extension Functions:: 
* Building Arbitrary Values:: 
* Reference Counts:: 
* Writing Extensions in C++:: 
* Providing a C API for an Extension Module:: 

Reference Counts

* Reference Counting in Python:: 
* Ownership Rules:: 
* Thin Ice:: 
* NULL Pointers:: 

Defining Extension Types: Tutorial

* The Basics:: 
* Adding data and methods to the Basic example:: 
* Providing finer control over data attributes:: 
* Supporting cyclic garbage collection:: 
* Subclassing other types:: 

Defining Extension Types: Assorted Topics

* Finalization and De-allocation:: 
* Object Presentation:: 
* Attribute Management:: 
* Object Comparison:: 
* Abstract Protocol Support:: 
* Weak Reference Support:: 
* More Suggestions:: 

Attribute Management

* Generic Attribute Management:: 
* Type-specific Attribute Management:: 

Building C and C++ Extensions

* Building C and C++ Extensions with distutils:: 
* Distributing your extension modules:: 

Building C and C++ Extensions on Windows

* A Cookbook Approach:: 
* Differences Between Unix and Windows:: 
* Using DLLs in Practice:: 

Embedding the CPython runtime in a larger application

* Embedding Python in Another Application:: 

Embedding Python in Another Application

* Very High Level Embedding:: 
* Beyond Very High Level Embedding; An overview: Beyond Very High Level Embedding An overview. 
* Pure Embedding:: 
* Extending Embedded Python:: 
* Embedding Python in C++:: 
* Compiling and Linking under Unix-like systems:: 

Python/C API Reference Manual

* Introduction: Introduction<13>. 
* C API Stability:: 
* The Very High Level Layer:: 
* Reference Counting:: 
* Exception Handling:: 
* Utilities: Utilities<2>. 
* Abstract Objects Layer:: 
* Concrete Objects Layer:: 
* Initialization@comma{} Finalization@comma{} and Threads: Initialization Finalization and Threads. 
* Python Initialization Configuration:: 
* Memory Management:: 
* Object Implementation Support:: 
* API and ABI Versioning:: 

Introduction

* Coding standards:: 
* Include Files:: 
* Useful macros:: 
* Objects@comma{} Types and Reference Counts: Objects Types and Reference Counts. 
* Exceptions: Exceptions<18>. 
* Embedding Python: Embedding Python<2>. 
* Debugging Builds:: 

Objects, Types and Reference Counts

* Reference Counts: Reference Counts<2>. 
* Types:: 

Reference Counts

* Reference Count Details:: 

C API Stability

* Stable Application Binary Interface:: 
* Platform Considerations:: 
* Contents of Limited API:: 

Stable Application Binary Interface

* Limited API Scope and Performance:: 
* Limited API Caveats:: 

Exception Handling

* Printing and clearing:: 
* Raising exceptions:: 
* Issuing warnings:: 
* Querying the error indicator:: 
* Signal Handling: Signal Handling<2>. 
* Exception Classes:: 
* Exception Objects:: 
* Unicode Exception Objects:: 
* Recursion Control:: 
* Standard Exceptions:: 
* Standard Warning Categories:: 

Utilities

* Operating System Utilities:: 
* System Functions:: 
* Process Control:: 
* Importing Modules: Importing Modules<2>. 
* Data marshalling support:: 
* Parsing arguments and building values:: 
* String conversion and formatting:: 
* Reflection:: 
* Codec registry and support functions:: 

Parsing arguments and building values

* Parsing arguments: Parsing arguments<3>. 
* Building values:: 

Parsing arguments

* Strings and buffers:: 
* Numbers: Numbers<2>. 
* Other objects:: 
* API Functions:: 

Codec registry and support functions

* Codec lookup API:: 
* Registry API for Unicode encoding error handlers:: 

Abstract Objects Layer

* Object Protocol:: 
* Call Protocol:: 
* Number Protocol:: 
* Sequence Protocol:: 
* Mapping Protocol:: 
* Iterator Protocol:: 
* Buffer Protocol:: 
* Old Buffer Protocol:: 

Call Protocol

* The tp_call Protocol:: 
* The Vectorcall Protocol:: 
* Object Calling API:: 
* Call Support API:: 

The Vectorcall Protocol

* Recursion Control: Recursion Control<2>. 
* Vectorcall Support API:: 

Buffer Protocol

* Buffer structure:: 
* Buffer request types:: 
* Complex arrays:: 
* Buffer-related functions:: 

Buffer request types

* request-independent fields:: 
* readonly@comma{} format: readonly format. 
* shape@comma{} strides@comma{} suboffsets: shape strides suboffsets. 
* contiguity requests:: 
* compound requests:: 

Complex arrays

* NumPy-style; shape and strides: NumPy-style shape and strides. 
* PIL-style; shape@comma{} strides and suboffsets: PIL-style shape strides and suboffsets. 

Concrete Objects Layer

* Fundamental Objects:: 
* Numeric Objects:: 
* Sequence Objects:: 
* Container Objects:: 
* Function Objects: Function Objects<2>. 
* Other Objects:: 

Fundamental Objects

* Type Objects: Type Objects<2>. 
* The None Object:: 

Type Objects

* Creating Heap-Allocated Types:: 

Numeric Objects

* Integer Objects:: 
* Boolean Objects:: 
* Floating Point Objects:: 
* Complex Number Objects:: 

Complex Number Objects

* Complex Numbers as C Structures:: 
* Complex Numbers as Python Objects:: 

Sequence Objects

* Bytes Objects: Bytes Objects<2>. 
* Byte Array Objects:: 
* Unicode Objects and Codecs:: 
* Tuple Objects:: 
* Struct Sequence Objects:: 
* List Objects:: 

Byte Array Objects

* Type check macros:: 
* Direct API functions:: 
* Macros:: 

Unicode Objects and Codecs

* Unicode Objects:: 
* Built-in Codecs:: 
* Methods and Slot Functions:: 

Unicode Objects

* Unicode Type:: 
* Unicode Character Properties:: 
* Creating and accessing Unicode strings:: 
* Deprecated Py_UNICODE APIs:: 
* Locale Encoding:: 
* File System Encoding:: 
* wchar_t Support:: 

Built-in Codecs

* Generic Codecs:: 
* UTF-8 Codecs:: 
* UTF-32 Codecs:: 
* UTF-16 Codecs:: 
* UTF-7 Codecs:: 
* Unicode-Escape Codecs:: 
* Raw-Unicode-Escape Codecs:: 
* Latin-1 Codecs:: 
* ASCII Codecs:: 
* Character Map Codecs:: 
* MBCS codecs for Windows:: 
* Methods & Slots:: 

Container Objects

* Dictionary Objects:: 
* Set Objects:: 

Function Objects

* Function Objects: Function Objects<3>. 
* Instance Method Objects:: 
* Method Objects: Method Objects<2>. 
* Cell Objects:: 
* Code Objects: Code Objects<2>. 

Other Objects

* File Objects:: 
* Module Objects:: 
* Iterator Objects:: 
* Descriptor Objects:: 
* Slice Objects:: 
* Ellipsis Object:: 
* MemoryView objects:: 
* Weak Reference Objects: Weak Reference Objects<2>. 
* Capsules: Capsules<2>. 
* Generator Objects:: 
* Coroutine Objects: Coroutine Objects<2>. 
* Context Variables Objects:: 
* DateTime Objects: DateTime Objects<2>. 
* Objects for Type Hinting:: 

Module Objects

* Initializing C modules:: 
* Module lookup:: 

Initializing C modules

* Single-phase initialization:: 
* Multi-phase initialization:: 
* Low-level module creation functions:: 
* Support functions:: 

Initialization, Finalization, and Threads

* Before Python Initialization:: 
* Global configuration variables:: 
* Initializing and finalizing the interpreter:: 
* Process-wide parameters:: 
* Thread State and the Global Interpreter Lock:: 
* Sub-interpreter support:: 
* Asynchronous Notifications:: 
* Profiling and Tracing:: 
* Advanced Debugger Support:: 
* Thread Local Storage Support:: 

Thread State and the Global Interpreter Lock

* Releasing the GIL from extension code:: 
* Non-Python created threads:: 
* Cautions about fork(): Cautions about fork. 
* High-level API:: 
* Low-level API:: 

Sub-interpreter support

* Bugs and caveats:: 

Thread Local Storage Support

* Thread Specific Storage (TSS) API: Thread Specific Storage TSS API. 
* Thread Local Storage (TLS) API: Thread Local Storage TLS API. 

Thread Specific Storage (TSS) API

* Dynamic Allocation:: 
* Methods: Methods<4>. 

Python Initialization Configuration

* Example: Example<17>. 
* PyWideStringList:: 
* PyStatus:: 
* PyPreConfig:: 
* Preinitialize Python with PyPreConfig:: 
* PyConfig:: 
* Initialization with PyConfig:: 
* Isolated Configuration:: 
* Python Configuration:: 
* Python Path Configuration:: 
* Py_RunMain(): Py_RunMain. 
* Py_GetArgcArgv(): Py_GetArgcArgv. 
* Multi-Phase Initialization Private Provisional API:: 

Memory Management

* Overview: Overview<4>. 
* Allocator Domains:: 
* Raw Memory Interface:: 
* Memory Interface:: 
* Object allocators:: 
* Default Memory Allocators:: 
* Customize Memory Allocators:: 
* Debug hooks on the Python memory allocators:: 
* The pymalloc allocator:: 
* tracemalloc C API:: 
* Examples: Examples<37>. 

The pymalloc allocator

* Customize pymalloc Arena Allocator:: 

Object Implementation Support

* Allocating Objects on the Heap:: 
* Common Object Structures:: 
* Type Objects: Type Objects<3>. 
* Number Object Structures:: 
* Mapping Object Structures:: 
* Sequence Object Structures:: 
* Buffer Object Structures:: 
* Async Object Structures:: 
* Slot Type typedefs:: 
* Examples: Examples<38>. 
* Supporting Cyclic Garbage Collection:: 

Common Object Structures

* Base object types and macros:: 
* Implementing functions and methods:: 
* Accessing attributes of extension types:: 

Type Objects

* Quick Reference:: 
* PyTypeObject Definition:: 
* PyObject Slots:: 
* PyVarObject Slots:: 
* PyTypeObject Slots:: 
* Static Types:: 
* Heap Types:: 

Quick Reference

* “tp slots”:: 
* sub-slots:: 
* slot typedefs:: 

Supporting Cyclic Garbage Collection

* Controlling the Garbage Collector State:: 

Distributing Python Modules

* Key terms:: 
* Open source licensing and collaboration:: 
* Installing the tools:: 
* Reading the Python Packaging User Guide:: 
* How do I…?:: 

How do I…?

* … choose a name for my project?:: 
* … create and distribute binary extensions?:: 

Installing Python Modules

* Key terms: Key terms<2>. 
* Basic usage:: 
* How do I …?:: 
* Common installation issues:: 

How do I …?

* … install pip in versions of Python prior to Python 3.4?: … install pip in versions of Python prior to Python 3 4?. 
* … install packages just for the current user?:: 
* … install scientific Python packages?:: 
* … work with multiple versions of Python installed in parallel?:: 

Common installation issues

* Installing into the system Python on Linux:: 
* Pip not installed:: 
* Installing binary extensions:: 

Python HOWTOs

* Porting Python 2 Code to Python 3:: 
* Porting Extension Modules to Python 3:: 
* Curses Programming with Python:: 
* Descriptor HowTo Guide:: 
* Enum HOWTO:: 
* Functional Programming HOWTO:: 
* Logging HOWTO:: 
* Logging Cookbook:: 
* Regular Expression HOWTO:: 
* Socket Programming HOWTO:: 
* Sorting HOW TO:: 
* Unicode HOWTO:: 
* HOWTO Fetch Internet Resources Using The urllib Package:: 
* Argparse Tutorial:: 
* An introduction to the ipaddress module:: 
* Argument Clinic How-To:: 
* Instrumenting CPython with DTrace and SystemTap:: 
* Annotations Best Practices:: 

Porting Python 2 Code to Python 3

* The Short Explanation:: 
* Details:: 

Details

* Drop support for Python 2.6 and older: Drop support for Python 2 6 and older. 
* Make sure you specify the proper version support in your setup.py file: Make sure you specify the proper version support in your setup py file. 
* Have good test coverage:: 
* Learn the differences between Python 2 & 3:: 
* Update your code:: 
* Prevent compatibility regressions:: 
* Check which dependencies block your transition:: 
* Update your setup.py file to denote Python 3 compatibility: Update your setup py file to denote Python 3 compatibility. 
* Use continuous integration to stay compatible:: 
* Consider using optional static type checking:: 

Update your code

* Division:: 
* Text versus binary data:: 
* Use feature detection instead of version detection:: 

Curses Programming with Python

* What is curses?:: 
* Starting and ending a curses application:: 
* Windows and Pads:: 
* Displaying Text:: 
* User Input:: 
* For More Information:: 

What is curses?

* The Python curses module:: 

Displaying Text

* Attributes and Color:: 

Descriptor HowTo Guide

* Primer:: 
* Complete Practical Example:: 
* Technical Tutorial:: 
* Pure Python Equivalents:: 

Primer

* Simple example; A descriptor that returns a constant: Simple example A descriptor that returns a constant. 
* Dynamic lookups:: 
* Managed attributes:: 
* Customized names:: 
* Closing thoughts:: 

Complete Practical Example

* Validator class:: 
* Custom validators:: 
* Practical application:: 

Technical Tutorial

* Abstract:: 
* Definition and introduction:: 
* Descriptor protocol:: 
* Overview of descriptor invocation:: 
* Invocation from an instance:: 
* Invocation from a class:: 
* Invocation from super:: 
* Summary of invocation logic:: 
* Automatic name notification:: 
* ORM example:: 

Pure Python Equivalents

* Properties:: 
* Functions and methods:: 
* Kinds of methods:: 
* Static methods:: 
* Class methods:: 
* Member objects and __slots__:: 

Enum HOWTO

* Programmatic access to enumeration members and their attributes:: 
* Duplicating enum members and values:: 
* Ensuring unique enumeration values:: 
* Using automatic values:: 
* Iteration: Iteration<2>. 
* Comparisons: Comparisons<3>. 
* Allowed members and attributes of enumerations:: 
* Restricted Enum subclassing:: 
* Pickling:: 
* Functional API: Functional API<2>. 
* Derived Enumerations:: 
* When to use __new__() vs. __init__(): When to use __new__ vs __init__. 
* How are Enums different?:: 
* Subclassing EnumType:: 

Derived Enumerations

* IntEnum:: 
* StrEnum:: 
* IntFlag:: 
* Flag:: 
* Others:: 

When to use __new__() vs. __init__()

* Finer Points:: 

Finer Points

* Supported __dunder__ names: Supported __dunder__ names<2>. 
* Supported _sunder_ names: Supported _sunder_ names<2>. 
* _Private__names:: 
* Enum member type:: 
* Creating members that are mixed with other data types:: 
* Boolean value of Enum classes and members:: 
* Enum classes with methods:: 
* Combining members of Flag:: 
* Flag and IntFlag minutia:: 

How are Enums different?

* Enum Classes:: 
* Enum Members (aka instances): Enum Members aka instances. 
* Omitting values:: 
* OrderedEnum:: 
* DuplicateFreeEnum:: 
* Planet:: 
* TimePeriod:: 

Omitting values

* Using auto:: 
* Using object:: 
* Using a descriptive string:: 
* Using a custom __new__(): Using a custom __new__. 

Functional Programming HOWTO

* Introduction: Introduction<14>. 
* Iterators: Iterators<2>. 
* Generator expressions and list comprehensions:: 
* Generators: Generators<2>. 
* Built-in functions:: 
* The itertools module:: 
* The functools module:: 
* Small functions and the lambda expression:: 
* Revision History and Acknowledgements:: 
* References: References<2>. 

Introduction

* Formal provability:: 
* Modularity:: 
* Ease of debugging and testing:: 
* Composability:: 

Iterators

* Data Types That Support Iterators:: 

Generators

* Passing values into a generator:: 

The itertools module

* Creating new iterators:: 
* Calling functions on elements:: 
* Selecting elements:: 
* Combinatoric functions:: 
* Grouping elements:: 

The functools module

* The operator module:: 

References

* General:: 
* Python-specific:: 
* Python documentation:: 

Logging HOWTO

* Basic Logging Tutorial:: 
* Advanced Logging Tutorial:: 
* Logging Levels: Logging Levels<2>. 
* Useful Handlers:: 
* Exceptions raised during logging:: 
* Using arbitrary objects as messages:: 
* Optimization:: 

Basic Logging Tutorial

* When to use logging:: 
* A simple example:: 
* Logging to a file:: 
* Logging from multiple modules:: 
* Logging variable data:: 
* Changing the format of displayed messages:: 
* Displaying the date/time in messages:: 
* Next Steps:: 

Advanced Logging Tutorial

* Logging Flow:: 
* Loggers:: 
* Handlers:: 
* Formatters:: 
* Configuring Logging:: 
* What happens if no configuration is provided:: 
* Configuring Logging for a Library:: 

Logging Levels

* Custom Levels:: 

Logging Cookbook

* Using logging in multiple modules:: 
* Logging from multiple threads:: 
* Multiple handlers and formatters:: 
* Logging to multiple destinations:: 
* Configuration server example:: 
* Dealing with handlers that block:: 
* Sending and receiving logging events across a network:: 
* Adding contextual information to your logging output:: 
* Logging to a single file from multiple processes:: 
* Using file rotation:: 
* Use of alternative formatting styles:: 
* Customizing LogRecord:: 
* Subclassing QueueHandler - a ZeroMQ example:: 
* Subclassing QueueListener - a ZeroMQ example:: 
* An example dictionary-based configuration:: 
* Using a rotator and namer to customize log rotation processing:: 
* A more elaborate multiprocessing example:: 
* Inserting a BOM into messages sent to a SysLogHandler:: 
* Implementing structured logging:: 
* Customizing handlers with dictConfig(): Customizing handlers with dictConfig. 
* Using particular formatting styles throughout your application:: 
* Configuring filters with dictConfig(): Configuring filters with dictConfig. 
* Customized exception formatting:: 
* Speaking logging messages:: 
* Buffering logging messages and outputting them conditionally:: 
* Formatting times using UTC (GMT) via configuration: Formatting times using UTC GMT via configuration. 
* Using a context manager for selective logging:: 
* A CLI application starter template:: 
* A Qt GUI for logging:: 
* Patterns to avoid:: 

Sending and receiving logging events across a network

* Running a logging socket listener in production:: 

Adding contextual information to your logging output

* Using LoggerAdapters to impart contextual information:: 
* Using Filters to impart contextual information:: 

Using LoggerAdapters to impart contextual information

* Using objects other than dicts to pass contextual information:: 

Logging to a single file from multiple processes

* Using concurrent.futures.ProcessPoolExecutor: Using concurrent futures ProcessPoolExecutor. 
* Deploying Web applications using Gunicorn and uWSGI:: 

Using particular formatting styles throughout your application

* Using LogRecord factories:: 
* Using custom message objects:: 

Patterns to avoid

* Opening the same log file multiple times:: 
* Using loggers as attributes in a class or passing them as parameters:: 
* Adding handlers other than NullHandler to a logger in a library:: 
* Creating a lot of loggers:: 

Regular Expression HOWTO

* Introduction: Introduction<15>. 
* Simple Patterns:: 
* Using Regular Expressions:: 
* More Pattern Power:: 
* Modifying Strings:: 
* Common Problems:: 
* Feedback:: 

Simple Patterns

* Matching Characters:: 
* Repeating Things:: 

Using Regular Expressions

* Compiling Regular Expressions:: 
* The Backslash Plague:: 
* Performing Matches:: 
* Module-Level Functions: Module-Level Functions<2>. 
* Compilation Flags:: 

More Pattern Power

* More Metacharacters:: 
* Grouping:: 
* Non-capturing and Named Groups:: 
* Lookahead Assertions:: 

Modifying Strings

* Splitting Strings:: 
* Search and Replace:: 

Common Problems

* Use String Methods:: 
* match() versus search(): match versus search. 
* Greedy versus Non-Greedy:: 
* Using re.VERBOSE: Using re VERBOSE. 

Socket Programming HOWTO

* Sockets:: 
* Creating a Socket:: 
* Using a Socket:: 
* Disconnecting:: 
* Non-blocking Sockets:: 

Sockets

* History:: 

Creating a Socket

* IPC:: 

Using a Socket

* Binary Data:: 

Disconnecting

* When Sockets Die:: 

Sorting HOW TO

* Sorting Basics:: 
* Key Functions:: 
* Operator Module Functions:: 
* Ascending and Descending:: 
* Sort Stability and Complex Sorts:: 
* The Old Way Using Decorate-Sort-Undecorate:: 
* The Old Way Using the cmp Parameter:: 
* Odd and Ends:: 

Unicode HOWTO

* Introduction to Unicode:: 
* Python’s Unicode Support:: 
* Reading and Writing Unicode Data:: 
* Acknowledgements: Acknowledgements<10>. 

Introduction to Unicode

* Definitions:: 
* Encodings:: 
* References: References<3>. 

Python’s Unicode Support

* The String Type:: 
* Converting to Bytes:: 
* Unicode Literals in Python Source Code:: 
* Unicode Properties:: 
* Comparing Strings:: 
* Unicode Regular Expressions:: 
* References: References<4>. 

Reading and Writing Unicode Data

* Unicode filenames:: 
* Tips for Writing Unicode-aware Programs:: 
* References: References<5>. 

Tips for Writing Unicode-aware Programs

* Converting Between File Encodings:: 
* Files in an Unknown Encoding:: 

HOWTO Fetch Internet Resources Using The urllib Package

* Introduction: Introduction<16>. 
* Fetching URLs:: 
* Handling Exceptions: Handling Exceptions<2>. 
* info and geturl:: 
* Openers and Handlers:: 
* Basic Authentication:: 
* Proxies:: 
* Sockets and Layers:: 
* Footnotes:: 

Fetching URLs

* Data:: 
* Headers:: 

Handling Exceptions

* URLError:: 
* HTTPError:: 
* Wrapping it Up:: 

HTTPError

* Error Codes:: 

Wrapping it Up

* Number 1:: 
* Number 2:: 

Argparse Tutorial

* Concepts:: 
* The basics:: 
* Introducing Positional arguments:: 
* Introducing Optional arguments:: 
* Combining Positional and Optional arguments:: 
* Getting a little more advanced:: 
* Conclusion:: 

Introducing Optional arguments

* Short options:: 

Getting a little more advanced

* Conflicting options:: 

An introduction to the ipaddress module

* Creating Address/Network/Interface objects:: 
* Inspecting Address/Network/Interface Objects:: 
* Networks as lists of Addresses:: 
* Comparisons: Comparisons<4>. 
* Using IP Addresses with other modules:: 
* Getting more detail when instance creation fails:: 

Creating Address/Network/Interface objects

* A Note on IP Versions:: 
* IP Host Addresses:: 
* Defining Networks:: 
* Host Interfaces:: 

Argument Clinic How-To

* The Goals Of Argument Clinic:: 
* Basic Concepts And Usage:: 
* Converting Your First Function:: 
* Advanced Topics:: 

Advanced Topics

* Symbolic default values:: 
* Renaming the C functions and variables generated by Argument Clinic:: 
* Converting functions using PyArg_UnpackTuple:: 
* Optional Groups:: 
* Using real Argument Clinic converters@comma{} instead of “legacy converters”: Using real Argument Clinic converters instead of “legacy converters”. 
* Py_buffer:: 
* Advanced converters:: 
* Parameter default values:: 
* The NULL default value:: 
* Expressions specified as default values:: 
* Using a return converter:: 
* Cloning existing functions:: 
* Calling Python code:: 
* Using a “self converter”:: 
* Using a “defining class” converter:: 
* Writing a custom converter:: 
* Writing a custom return converter:: 
* METH_O and METH_NOARGS:: 
* tp_new and tp_init functions:: 
* Changing and redirecting Clinic’s output:: 
* The #ifdef trick:: 
* Using Argument Clinic in Python files:: 

Instrumenting CPython with DTrace and SystemTap

* Enabling the static markers:: 
* Static DTrace probes:: 
* Static SystemTap markers:: 
* Available static markers:: 
* SystemTap Tapsets:: 
* Examples: Examples<39>. 

Annotations Best Practices

* Accessing The Annotations Dict Of An Object In Python 3.10 And Newer: Accessing The Annotations Dict Of An Object In Python 3 10 And Newer. 
* Accessing The Annotations Dict Of An Object In Python 3.9 And Older: Accessing The Annotations Dict Of An Object In Python 3 9 And Older. 
* Manually Un-Stringizing Stringized Annotations:: 
* Best Practices For __annotations__ In Any Python Version:: 
* __annotations__ Quirks:: 

Python Frequently Asked Questions

* General Python FAQ:: 
* Programming FAQ:: 
* Design and History FAQ:: 
* Library and Extension FAQ:: 
* Extending/Embedding FAQ:: 
* Python on Windows FAQ:: 
* Graphic User Interface FAQ:: 
* “Why is Python Installed on my Computer?” FAQ:: 

General Python FAQ

* General Information:: 
* Python in the real world:: 

General Information

* What is Python?:: 
* What is the Python Software Foundation?:: 
* Are there copyright restrictions on the use of Python?:: 
* Why was Python created in the first place?:: 
* What is Python good for?:: 
* How does the Python version numbering scheme work?:: 
* How do I obtain a copy of the Python source?:: 
* How do I get documentation on Python?:: 
* I’ve never programmed before. Is there a Python tutorial?: I’ve never programmed before Is there a Python tutorial?. 
* Is there a newsgroup or mailing list devoted to Python?:: 
* How do I get a beta test version of Python?:: 
* How do I submit bug reports and patches for Python?:: 
* Are there any published articles about Python that I can reference?:: 
* Are there any books on Python?:: 
* Where in the world is www.python.org located?: Where in the world is www python org located?. 
* Why is it called Python?:: 
* Do I have to like “Monty Python’s Flying Circus”?:: 

Python in the real world

* How stable is Python?:: 
* How many people are using Python?:: 
* Have any significant projects been done in Python?:: 
* What new developments are expected for Python in the future?:: 
* Is it reasonable to propose incompatible changes to Python?:: 
* Is Python a good language for beginning programmers?:: 

Programming FAQ

* General Questions:: 
* Core Language:: 
* Numbers and strings:: 
* Performance: Performance<4>. 
* Sequences (Tuples/Lists): Sequences Tuples/Lists. 
* Objects:: 
* Modules: Modules<3>. 

General Questions

* Is there a source code level debugger with breakpoints@comma{} single-stepping@comma{} etc.?: Is there a source code level debugger with breakpoints single-stepping etc ?. 
* Are there tools to help find bugs or perform static analysis?:: 
* How can I create a stand-alone binary from a Python script?:: 
* Are there coding standards or a style guide for Python programs?:: 

Core Language

* Why am I getting an UnboundLocalError when the variable has a value?:: 
* What are the rules for local and global variables in Python?:: 
* Why do lambdas defined in a loop with different values all return the same result?:: 
* How do I share global variables across modules?:: 
* What are the “best practices” for using import in a module?:: 
* Why are default values shared between objects?:: 
* How can I pass optional or keyword parameters from one function to another?:: 
* What is the difference between arguments and parameters?:: 
* Why did changing list ‘y’ also change list ‘x’?:: 
* How do I write a function with output parameters (call by reference)?: How do I write a function with output parameters call by reference ?. 
* How do you make a higher order function in Python?:: 
* How do I copy an object in Python?:: 
* How can I find the methods or attributes of an object?:: 
* How can my code discover the name of an object?:: 
* What’s up with the comma operator’s precedence?:: 
* Is there an equivalent of C’s “?;” ternary operator?: Is there an equivalent of C’s “? ” ternary operator?. 
* Is it possible to write obfuscated one-liners in Python?:: 
* What does the slash(/) in the parameter list of a function mean?: What does the slash / in the parameter list of a function mean?. 

Numbers and strings

* How do I specify hexadecimal and octal integers?:: 
* Why does -22 // 10 return -3?:: 
* How do I get int literal attribute instead of SyntaxError?:: 
* How do I convert a string to a number?:: 
* How do I convert a number to a string?:: 
* How do I modify a string in place?:: 
* How do I use strings to call functions/methods?:: 
* Is there an equivalent to Perl’s chomp() for removing trailing newlines from strings?: Is there an equivalent to Perl’s chomp for removing trailing newlines from strings?. 
* Is there a scanf() or sscanf() equivalent?: Is there a scanf or sscanf equivalent?. 
* What does ‘UnicodeDecodeError’ or ‘UnicodeEncodeError’ error mean?:: 

Performance

* My program is too slow. How do I speed it up?: My program is too slow How do I speed it up?. 
* What is the most efficient way to concatenate many strings together?:: 

Sequences (Tuples/Lists)

* How do I convert between tuples and lists?:: 
* What’s a negative index?:: 
* How do I iterate over a sequence in reverse order?:: 
* How do you remove duplicates from a list?:: 
* How do you remove multiple items from a list:: 
* How do you make an array in Python?:: 
* How do I create a multidimensional list?:: 
* How do I apply a method to a sequence of objects?:: 
* Why does a_tuple[i] += [‘item’] raise an exception when the addition works?:: 
* I want to do a complicated sort; can you do a Schwartzian Transform in Python?: I want to do a complicated sort can you do a Schwartzian Transform in Python?. 
* How can I sort one list by values from another list?:: 

Objects

* What is a class?:: 
* What is a method?:: 
* What is self?:: 
* How do I check if an object is an instance of a given class or of a subclass of it?:: 
* What is delegation?:: 
* How do I call a method defined in a base class from a derived class that extends it?:: 
* How can I organize my code to make it easier to change the base class?:: 
* How do I create static class data and static class methods?:: 
* How can I overload constructors (or methods) in Python?: How can I overload constructors or methods in Python?. 
* I try to use __spam and I get an error about _SomeClassName__spam.: I try to use __spam and I get an error about _SomeClassName__spam. 
* My class defines __del__ but it is not called when I delete the object.: My class defines __del__ but it is not called when I delete the object. 
* How do I get a list of all instances of a given class?:: 
* Why does the result of id() appear to be not unique?: Why does the result of id appear to be not unique?. 
* When can I rely on identity tests with the is operator?:: 
* How can a subclass control what data is stored in an immutable instance?:: 
* How do I cache method calls?:: 

Modules

* How do I create a .pyc file?: How do I create a pyc file?. 
* How do I find the current module name?:: 
* How can I have modules that mutually import each other?:: 
* __import__(‘x.y.z’) returns <module ‘x’>; how do I get z?: __import__ ‘x y z’ returns <module ‘x’>; how do I get z?. 
* When I edit an imported module and reimport it@comma{} the changes don’t show up. Why does this happen?: When I edit an imported module and reimport it the changes don’t show up Why does this happen?. 

Design and History FAQ

* Why does Python use indentation for grouping of statements?:: 
* Why am I getting strange results with simple arithmetic operations?:: 
* Why are floating-point calculations so inaccurate?:: 
* Why are Python strings immutable?:: 
* Why must ‘self’ be used explicitly in method definitions and calls?:: 
* Why can’t I use an assignment in an expression?:: 
* Why does Python use methods for some functionality (e.g. list.index()) but functions for other (e.g. len(list))?: Why does Python use methods for some functionality e g list index but functions for other e g len list ?. 
* Why is join() a string method instead of a list or tuple method?: Why is join a string method instead of a list or tuple method?. 
* How fast are exceptions?:: 
* Why isn’t there a switch or case statement in Python?:: 
* Can’t you emulate threads in the interpreter instead of relying on an OS-specific thread implementation?:: 
* Why can’t lambda expressions contain statements?:: 
* Can Python be compiled to machine code@comma{} C or some other language?: Can Python be compiled to machine code C or some other language?. 
* How does Python manage memory?:: 
* Why doesn’t CPython use a more traditional garbage collection scheme?:: 
* Why isn’t all memory freed when CPython exits?:: 
* Why are there separate tuple and list data types?:: 
* How are lists implemented in CPython?:: 
* How are dictionaries implemented in CPython?:: 
* Why must dictionary keys be immutable?:: 
* Why doesn’t list.sort() return the sorted list?: Why doesn’t list sort return the sorted list?. 
* How do you specify and enforce an interface spec in Python?:: 
* Why is there no goto?:: 
* Why can’t raw strings (r-strings) end with a backslash?: Why can’t raw strings r-strings end with a backslash?. 
* Why doesn’t Python have a “with” statement for attribute assignments?:: 
* Why don’t generators support the with statement?:: 
* Why are colons required for the if/while/def/class statements?:: 
* Why does Python allow commas at the end of lists and tuples?:: 

Library and Extension FAQ

* General Library Questions:: 
* Common tasks:: 
* Threads:: 
* Input and Output: Input and Output<2>. 
* Network/Internet Programming:: 
* Databases:: 
* Mathematics and Numerics:: 

General Library Questions

* How do I find a module or application to perform task X?:: 
* Where is the math.py (socket.py@comma{} regex.py@comma{} etc.) source file?: Where is the math py socket py regex py etc source file?. 
* How do I make a Python script executable on Unix?:: 
* Is there a curses/termcap package for Python?:: 
* Is there an equivalent to C’s onexit() in Python?: Is there an equivalent to C’s onexit in Python?. 
* Why don’t my signal handlers work?:: 

Common tasks

* How do I test a Python program or component?:: 
* How do I create documentation from doc strings?:: 
* How do I get a single keypress at a time?:: 

Threads

* How do I program using threads?:: 
* None of my threads seem to run; why?: None of my threads seem to run why?. 
* How do I parcel out work among a bunch of worker threads?:: 
* What kinds of global value mutation are thread-safe?:: 
* Can’t we get rid of the Global Interpreter Lock?:: 

Input and Output

* How do I delete a file? (And other file questions…): How do I delete a file? And other file questions…. 
* How do I copy a file?:: 
* How do I read (or write) binary data?: How do I read or write binary data?. 
* I can’t seem to use os.read() on a pipe created with os.popen(); why?: I can’t seem to use os read on a pipe created with os popen ; why?. 
* How do I access the serial (RS232) port?: How do I access the serial RS232 port?. 
* Why doesn’t closing sys.stdout (stdin@comma{} stderr) really close it?: Why doesn’t closing sys stdout stdin stderr really close it?. 

Network/Internet Programming

* What WWW tools are there for Python?:: 
* How can I mimic CGI form submission (METHOD=POST)?: How can I mimic CGI form submission METHOD=POST ?. 
* What module should I use to help with generating HTML?:: 
* How do I send mail from a Python script?:: 
* How do I avoid blocking in the connect() method of a socket?: How do I avoid blocking in the connect method of a socket?. 

Databases

* Are there any interfaces to database packages in Python?:: 
* How do you implement persistent objects in Python?:: 

Mathematics and Numerics

* How do I generate random numbers in Python?:: 

Extending/Embedding FAQ

* Can I create my own functions in C?:: 
* Can I create my own functions in C++?:: 
* Writing C is hard; are there any alternatives?:: 
* How can I execute arbitrary Python statements from C?:: 
* How can I evaluate an arbitrary Python expression from C?:: 
* How do I extract C values from a Python object?:: 
* How do I use Py_BuildValue() to create a tuple of arbitrary length?: How do I use Py_BuildValue to create a tuple of arbitrary length?. 
* How do I call an object’s method from C?:: 
* How do I catch the output from PyErr_Print() (or anything that prints to stdout/stderr)?: How do I catch the output from PyErr_Print or anything that prints to stdout/stderr ?. 
* How do I access a module written in Python from C?:: 
* How do I interface to C++ objects from Python?:: 
* I added a module using the Setup file and the make fails; why?:: 
* How do I debug an extension?:: 
* I want to compile a Python module on my Linux system@comma{} but some files are missing. Why?: I want to compile a Python module on my Linux system but some files are missing Why?. 
* How do I tell “incomplete input” from “invalid input”?:: 
* How do I find undefined g++ symbols __builtin_new or __pure_virtual?:: 
* Can I create an object class with some methods implemented in C and others in Python (e.g. through inheritance)?: Can I create an object class with some methods implemented in C and others in Python e g through inheritance ?. 

Python on Windows FAQ

* How do I run a Python program under Windows?:: 
* How do I make Python scripts executable?:: 
* Why does Python sometimes take so long to start?:: 
* How do I make an executable from a Python script?:: 
* Is a *.pyd file the same as a DLL?: Is a * pyd file the same as a DLL?. 
* How can I embed Python into a Windows application?:: 
* How do I keep editors from inserting tabs into my Python source?:: 
* How do I check for a keypress without blocking?:: 

Graphic User Interface FAQ

* General GUI Questions:: 
* What GUI toolkits exist for Python?:: 
* Tkinter questions:: 

Tkinter questions

* How do I freeze Tkinter applications?:: 
* Can I have Tk events handled while waiting for I/O?:: 
* I can’t get key bindings to work in Tkinter; why?: I can’t get key bindings to work in Tkinter why?. 

“Why is Python Installed on my Computer?” FAQ

* What is Python?: What is Python?<2>. 
* Why is Python installed on my machine?:: 
* Can I delete Python?:: 

About these documents

* Contributors to the Python Documentation:: 

Dealing with Bugs

* Documentation bugs:: 
* Using the Python issue tracker:: 
* Getting started contributing to Python yourself:: 

History and License

* History of the software:: 
* Terms and conditions for accessing or otherwise using Python:: 
* Licenses and Acknowledgements for Incorporated Software:: 

Terms and conditions for accessing or otherwise using Python

* PSF LICENSE AGREEMENT FOR PYTHON 3.11.0a5: PSF LICENSE AGREEMENT FOR PYTHON 3 11 0a5. 
* BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0: BEOPEN COM LICENSE AGREEMENT FOR PYTHON 2 0. 
* CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1: CNRI LICENSE AGREEMENT FOR PYTHON 1 6 1. 
* CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2: CWI LICENSE AGREEMENT FOR PYTHON 0 9 0 THROUGH 1 2. 
* ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON 3.11.0a5 DOCUMENTATION: ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON 3 11 0a5 DOCUMENTATION. 

Licenses and Acknowledgements for Incorporated Software

* Mersenne Twister:: 
* Sockets: Sockets<2>. 
* Asynchronous socket services:: 
* Cookie management:: 
* Execution tracing:: 
* UUencode and UUdecode functions:: 
* XML Remote Procedure Calls:: 
* test_epoll:: 
* Select kqueue:: 
* SipHash24:: 
* strtod and dtoa:: 
* OpenSSL:: 
* expat:: 
* libffi:: 
* zlib: zlib<3>. 
* cfuhash:: 
* libmpdec:: 
* W3C C14N test suite:: 

Distributing Python Modules (Legacy version)

* An Introduction to Distutils:: 
* Writing the Setup Script:: 
* Writing the Setup Configuration File:: 
* Creating a Source Distribution:: 
* Creating Built Distributions:: 
* Distutils Examples:: 
* Extending Distutils:: 
* Command Reference:: 
* API Reference:: 

An Introduction to Distutils

* Concepts & Terminology:: 
* A Simple Example: A Simple Example<2>. 
* General Python terminology:: 
* Distutils-specific terminology:: 

Writing the Setup Script

* Listing whole packages:: 
* Listing individual modules:: 
* Describing extension modules:: 
* Relationships between Distributions and Packages:: 
* Installing Scripts:: 
* Installing Package Data:: 
* Installing Additional Files:: 
* Additional meta-data:: 
* Debugging the setup script:: 

Describing extension modules

* Extension names and packages:: 
* Extension source files:: 
* Preprocessor options:: 
* Library options:: 
* Other options:: 

Creating a Source Distribution

* Specifying the files to distribute:: 
* Manifest-related options:: 

Creating Built Distributions

* Creating RPM packages:: 
* Cross-compiling on Windows:: 

Cross-compiling on Windows

* The Postinstallation script:: 

Distutils Examples

* Pure Python distribution (by module): Pure Python distribution by module. 
* Pure Python distribution (by package): Pure Python distribution by package. 
* Single extension module:: 
* Checking a package:: 
* Reading the metadata:: 

Extending Distutils

* Integrating new commands:: 
* Adding new distribution types:: 

Command Reference

* Installing modules; the install command family: Installing modules the install command family. 
* Creating a source distribution; the sdist command: Creating a source distribution the sdist command. 

Installing modules: the install command family

* install_data:: 
* install_scripts:: 

API Reference

* distutils.core — Core Distutils functionality: distutils core — Core Distutils functionality. 
* distutils.ccompiler — CCompiler base class: distutils ccompiler — CCompiler base class. 
* distutils.unixccompiler — Unix C Compiler: distutils unixccompiler — Unix C Compiler. 
* distutils.msvccompiler — Microsoft Compiler: distutils msvccompiler — Microsoft Compiler. 
* distutils.bcppcompiler — Borland Compiler: distutils bcppcompiler — Borland Compiler. 
* distutils.cygwincompiler — Cygwin Compiler: distutils cygwincompiler — Cygwin Compiler. 
* distutils.archive_util — Archiving utilities: distutils archive_util — Archiving utilities. 
* distutils.dep_util — Dependency checking: distutils dep_util — Dependency checking. 
* distutils.dir_util — Directory tree operations: distutils dir_util — Directory tree operations. 
* distutils.file_util — Single file operations: distutils file_util — Single file operations. 
* distutils.util — Miscellaneous other utility functions: distutils util — Miscellaneous other utility functions. 
* distutils.dist — The Distribution class: distutils dist — The Distribution class. 
* distutils.extension — The Extension class: distutils extension — The Extension class. 
* distutils.debug — Distutils debug mode: distutils debug — Distutils debug mode. 
* distutils.errors — Distutils exceptions: distutils errors — Distutils exceptions. 
* distutils.fancy_getopt — Wrapper around the standard getopt module: distutils fancy_getopt — Wrapper around the standard getopt module. 
* distutils.filelist — The FileList class: distutils filelist — The FileList class. 
* distutils.log — Simple PEP 282-style logging: distutils log — Simple PEP 282-style logging. 
* distutils.spawn — Spawn a sub-process: distutils spawn — Spawn a sub-process. 
* distutils.sysconfig — System configuration information: distutils sysconfig — System configuration information. 
* distutils.text_file — The TextFile class: distutils text_file — The TextFile class. 
* distutils.version — Version number classes: distutils version — Version number classes. 
* distutils.cmd — Abstract base class for Distutils commands: distutils cmd — Abstract base class for Distutils commands. 
* Creating a new Distutils command:: 
* distutils.command — Individual Distutils commands: distutils command — Individual Distutils commands. 
* distutils.command.bdist — Build a binary installer: distutils command bdist — Build a binary installer. 
* distutils.command.bdist_packager — Abstract base class for packagers: distutils command bdist_packager — Abstract base class for packagers. 
* distutils.command.bdist_dumb — Build a “dumb” installer: distutils command bdist_dumb — Build a “dumb” installer. 
* distutils.command.bdist_rpm — Build a binary distribution as a Redhat RPM and SRPM: distutils command bdist_rpm — Build a binary distribution as a Redhat RPM and SRPM. 
* distutils.command.sdist — Build a source distribution: distutils command sdist — Build a source distribution. 
* distutils.command.build — Build all files of a package: distutils command build — Build all files of a package. 
* distutils.command.build_clib — Build any C libraries in a package: distutils command build_clib — Build any C libraries in a package. 
* distutils.command.build_ext — Build any extensions in a package: distutils command build_ext — Build any extensions in a package. 
* distutils.command.build_py — Build the .py/.pyc files of a package: distutils command build_py — Build the py/ pyc files of a package. 
* distutils.command.build_scripts — Build the scripts of a package: distutils command build_scripts — Build the scripts of a package. 
* distutils.command.clean — Clean a package build area: distutils command clean — Clean a package build area. 
* distutils.command.config — Perform package configuration: distutils command config — Perform package configuration. 
* distutils.command.install — Install a package: distutils command install — Install a package. 
* distutils.command.install_data — Install data files from a package: distutils command install_data — Install data files from a package. 
* distutils.command.install_headers — Install C/C++ header files from a package: distutils command install_headers — Install C/C++ header files from a package. 
* distutils.command.install_lib — Install library files from a package: distutils command install_lib — Install library files from a package. 
* distutils.command.install_scripts — Install script files from a package: distutils command install_scripts — Install script files from a package. 
* distutils.command.register — Register a module with the Python Package Index: distutils command register — Register a module with the Python Package Index. 
* distutils.command.check — Check the meta-data of a package: distutils command check — Check the meta-data of a package. 

Installing Python Modules (Legacy version)

* Introduction: Introduction<17>. 
* Standard Build and Install:: 
* Alternate Installation:: 
* Custom Installation:: 
* Distutils Configuration Files:: 
* Building Extensions; Tips and Tricks: Building Extensions Tips and Tricks. 

Introduction

* Distutils based source distributions:: 

Standard Build and Install

* Platform variations:: 
* Splitting the job up:: 
* How building works:: 
* How installation works:: 

Alternate Installation

* Alternate installation; the user scheme: Alternate installation the user scheme. 
* Alternate installation; the home scheme: Alternate installation the home scheme. 
* Alternate installation; Unix (the prefix scheme): Alternate installation Unix the prefix scheme. 
* Alternate installation; Windows (the prefix scheme): Alternate installation Windows the prefix scheme. 

Custom Installation

* Modifying Python’s Search Path:: 

Distutils Configuration Files

* Location and names of config files:: 
* Syntax of config files:: 

Building Extensions: Tips and Tricks

* Tweaking compiler/linker flags:: 
* Using non-Microsoft compilers on Windows:: 

Using non-Microsoft compilers on Windows

* Borland/CodeGear C++:: 
* GNU C / Cygwin / MinGW:: 

GNU C / Cygwin / MinGW

* Older Versions of Python and MinGW:: 

@end detailmenu
@end menu

@node What’s New in Python,The Python Tutorial,Top,Top
@anchor{whatsnew/index doc}@anchor{14e}@anchor{whatsnew/index python-documentation-contents}@anchor{14f}@anchor{whatsnew/index what-s-new-in-python}@anchor{150}@anchor{whatsnew/index whatsnew-index}@anchor{151}
@chapter What’s New in Python


The “What’s New in Python” series of essays takes tours through the most
important changes between major Python versions.  They are a “must read” for
anyone wishing to stay up-to-date after a new release.

@menu
* What’s New In Python 3.11: What’s New In Python 3 11. 
* What’s New In Python 3.10: What’s New In Python 3 10. 
* What’s New In Python 3.9: What’s New In Python 3 9. 
* What’s New In Python 3.8: What’s New In Python 3 8. 
* What’s New In Python 3.7: What’s New In Python 3 7. 
* What’s New In Python 3.6: What’s New In Python 3 6. 
* What’s New In Python 3.5: What’s New In Python 3 5. 
* What’s New In Python 3.4: What’s New In Python 3 4. 
* What’s New In Python 3.3: What’s New In Python 3 3. 
* What’s New In Python 3.2: What’s New In Python 3 2. 
* What’s New In Python 3.1: What’s New In Python 3 1. 
* What’s New In Python 3.0: What’s New In Python 3 0. 
* What’s New in Python 2.7: What’s New in Python 2 7. 
* What’s New in Python 2.6: What’s New in Python 2 6. 
* What’s New in Python 2.5: What’s New in Python 2 5. 
* What’s New in Python 2.4: What’s New in Python 2 4. 
* What’s New in Python 2.3: What’s New in Python 2 3. 
* What’s New in Python 2.2: What’s New in Python 2 2. 
* What’s New in Python 2.1: What’s New in Python 2 1. 
* What’s New in Python 2.0: What’s New in Python 2 0. 
* Changelog:: 

@end menu

@node What’s New In Python 3 11,What’s New In Python 3 10,,What’s New in Python
@anchor{whatsnew/3 11 doc}@anchor{152}@anchor{whatsnew/3 11 what-s-new-in-python-3-11}@anchor{153}
@section What’s New In Python 3.11


@*Release: 
3.11.0a5


@*Date: 
March 07, 2022


@c Rules for maintenance:
@c 
@c * Anyone can add text to this document.  Do not spend very much time
@c on the wording of your changes, because your text will probably
@c get rewritten to some degree.
@c 
@c * The maintainer will go through Misc/NEWS periodically and add
@c changes; it's therefore more important to add your changes to
@c Misc/NEWS than to this file.
@c 
@c * This is not a complete list of every single change; completeness
@c is the purpose of Misc/NEWS.  Some changes I consider too small
@c or esoteric to include.  If such a change is added to the text,
@c I'll just remove it.  (This is another reason you shouldn't spend
@c too much time on writing your addition.)
@c 
@c * If you want to draw your new text to the attention of the
@c maintainer, add 'XXX' to the beginning of the paragraph or
@c section.
@c 
@c * It's OK to just add a fragmentary note about a change.  For
@c example: "XXX Describe the transmogrify() function added to the
@c socket module."  The maintainer will research the change and
@c write the necessary text.
@c 
@c * You can comment out your additions if you like, but it's not
@c necessary (especially when a final release is some months away).
@c 
@c * Credit the author of a patch or bugfix.   Just the name is
@c sufficient; the e-mail address isn't necessary.
@c 
@c * It's helpful to add the bug/patch number as a comment:
@c 
@c XXX Describe the transmogrify() function added to the socket
@c module.
@c (Contributed by P.Y. Developer in :issue:`12345`.)
@c 
@c This saves the maintainer the effort of going through the Mercurial log
@c when researching a change.

This article explains the new features in Python 3.11, compared to 3.10.

For full details, see the @ref{154,,changelog}.

@cartouche
@quotation Note 
Prerelease users should be aware that this document is currently in draft
form. It will be updated substantially as Python 3.11 moves towards release,
so it’s worth checking back even after reading earlier versions.
@end quotation
@end cartouche

@menu
* Summary – Release highlights:: 
* New Features:: 
* Other Language Changes:: 
* Other CPython Implementation Changes:: 
* New Modules:: 
* Improved Modules:: 
* Optimizations:: 
* CPython bytecode changes:: 
* Deprecated:: 
* Removed:: 
* Porting to Python 3.11: Porting to Python 3 11. 
* Build Changes:: 
* C API Changes:: 

@end menu

@node Summary – Release highlights,New Features,,What’s New In Python 3 11
@anchor{whatsnew/3 11 summary-release-highlights}@anchor{155}
@subsection Summary – Release highlights


@c This section singles out the most important changes in Python 3.11.
@c Brevity is key.

@c PEP-sized items next.

PEP-654: Exception Groups and @code{except*}.
(Contributed by Irit Katriel in bpo-45292@footnote{https://bugs.python.org/issue45292}.)
PEP-673: @code{Self} Type.
(Contributed by James Hilton-Balfe and Pradeep Kumar in bpo-30924@footnote{https://bugs.python.org/issue30924}.)

@node New Features,Other Language Changes,Summary – Release highlights,What’s New In Python 3 11
@anchor{whatsnew/3 11 new-features}@anchor{156}
@subsection New Features


@menu
* Enhanced error locations in tracebacks:: 

@end menu

@node Enhanced error locations in tracebacks,,,New Features
@anchor{whatsnew/3 11 enhanced-error-locations-in-tracebacks}@anchor{157}@anchor{whatsnew/3 11 whatsnew311-pep657}@anchor{158}
@subsubsection Enhanced error locations in tracebacks


When printing tracebacks, the interpreter will now point to the exact expression
that caused the error instead of just the line. For example:

@example
Traceback (most recent call last):
  File "distance.py", line 11, in <module>
    print(manhattan_distance(p1, p2))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "distance.py", line 6, in manhattan_distance
    return abs(point_1.x - point_2.x) + abs(point_1.y - point_2.y)
                           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'x'
@end example

Previous versions of the interpreter would point to just the line making it
ambiguous which object was @code{None}. These enhanced errors can also be helpful
when dealing with deeply nested dictionary objects and multiple function calls,

@example
Traceback (most recent call last):
  File "query.py", line 37, in <module>
    magic_arithmetic('foo')
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "query.py", line 18, in magic_arithmetic
    return add_counts(x) / 25
           ^^^^^^^^^^^^^
  File "query.py", line 24, in add_counts
    return 25 + query_user(user1) + query_user(user2)
                ^^^^^^^^^^^^^^^^^
  File "query.py", line 32, in query_user
    return 1 + query_count(db, response['a']['b']['c']['user'], retry=True)
                               ~~~~~~~~~~~~~~~~~~^^^^^
TypeError: 'NoneType' object is not subscriptable
@end example

as well as complex arithmetic expressions:

@example
Traceback (most recent call last):
  File "calculation.py", line 54, in <module>
    result = (x / y / z) * (a / b / c)
              ~~~~~~^~~
ZeroDivisionError: division by zero
@end example

See 
@geindex Python Enhancement Proposals; PEP 657
PEP 657@footnote{https://www.python.org/dev/peps/pep-0657} for more details. (Contributed by Pablo Galindo, Batuhan Taskaya
and Ammar Askar in bpo-43950@footnote{https://bugs.python.org/issue43950}.)

@cartouche
@quotation Note 
This feature requires storing column positions in code objects which may
result in a small increase of disk usage of compiled Python files or
interpreter memory usage. To avoid storing the extra information and/or
deactivate printing the extra traceback information, the
@ref{159,,-X} @code{no_debug_ranges} command line flag or the 
@geindex PYTHONNODEBUGRANGES
@geindex environment variable; PYTHONNODEBUGRANGES
@ref{15a,,PYTHONNODEBUGRANGES}
environment variable can be used.
@end quotation
@end cartouche

@menu
* Column information for code objects:: 
* Exceptions can be enriched with a string __note__:: 

@end menu

@node Column information for code objects,Exceptions can be enriched with a string __note__,,Enhanced error locations in tracebacks
@anchor{whatsnew/3 11 column-information-for-code-objects}@anchor{15b}
@subsubsection Column information for code objects


The information used by the enhanced traceback feature is made available as a
general API that can be used to correlate bytecode instructions with source
code. This information can be retrieved using:


@itemize -

@item 
The @ref{15c,,codeobject.co_positions()} method in Python.

@item 
The @ref{15d,,PyCode_Addr2Location()} function in the C-API.
@end itemize

The @ref{159,,-X} @code{no_debug_ranges} option and the environment variable
@geindex PYTHONNODEBUGRANGES
@geindex environment variable; PYTHONNODEBUGRANGES
@ref{15a,,PYTHONNODEBUGRANGES} can be used to disable this feature.

See 
@geindex Python Enhancement Proposals; PEP 657
PEP 657@footnote{https://www.python.org/dev/peps/pep-0657} for more details. (Contributed by Pablo Galindo, Batuhan Taskaya
and Ammar Askar in bpo-43950@footnote{https://bugs.python.org/issue43950}.)

@node Exceptions can be enriched with a string __note__,,Column information for code objects,Enhanced error locations in tracebacks
@anchor{whatsnew/3 11 exceptions-can-be-enriched-with-a-string-note}@anchor{15e}
@subsubsection Exceptions can be enriched with a string @code{__note__}


The @code{__note__} field was added to @ref{15f,,BaseException}. It is @code{None}
by default but can be set to a string which is added to the exception’s
traceback. (Contributed by Irit Katriel in bpo-45607@footnote{https://bugs.python.org/issue45607}.)

@node Other Language Changes,Other CPython Implementation Changes,New Features,What’s New In Python 3 11
@anchor{whatsnew/3 11 other-language-changes}@anchor{160}
@subsection Other Language Changes



@itemize *

@item 
Starred expressions can be used in @ref{161,,for statements}. (See
bpo-46725@footnote{https://bugs.python.org/issue46725} for more details.)

@item 
Asynchronous comprehensions are now allowed inside comprehensions in
asynchronous functions. Outer comprehensions implicitly become
asynchronous. (Contributed by Serhiy Storchaka in bpo-33346@footnote{https://bugs.python.org/issue33346}.)

@item 
A @ref{162,,TypeError} is now raised instead of an @ref{163,,AttributeError} in
@ref{164,,contextlib.ExitStack.enter_context()} and
@ref{165,,contextlib.AsyncExitStack.enter_async_context()} for objects which do not
support the @ref{166,,context manager} or @ref{167,,asynchronous context manager}
protocols correspondingly.
(Contributed by Serhiy Storchaka in bpo-44471@footnote{https://bugs.python.org/issue44471}.)

@item 
A @ref{162,,TypeError} is now raised instead of an @ref{163,,AttributeError} in
@ref{168,,with} and @ref{169,,async with} statements for objects which do not
support the @ref{166,,context manager} or @ref{167,,asynchronous context manager}
protocols correspondingly.
(Contributed by Serhiy Storchaka in bpo-12022@footnote{https://bugs.python.org/issue12022}.)
@end itemize

@node Other CPython Implementation Changes,New Modules,Other Language Changes,What’s New In Python 3 11
@anchor{whatsnew/3 11 other-cpython-implementation-changes}@anchor{16a}
@subsection Other CPython Implementation Changes



@itemize *

@item 
Special methods @code{complex.__complex__()} and @code{bytes.__bytes__()} are implemented to
support @ref{16b,,typing.SupportsComplex} and @ref{16c,,typing.SupportsBytes} protocols.
(Contributed by Mark Dickinson and Dong-hee Na in bpo-24234@footnote{https://bugs.python.org/issue24234}.)

@item 
@code{siphash13} is added as a new internal hashing algorithms. It’s has similar security
properties as @code{siphash24} but it is slightly faster for long inputs. @code{str}, @code{bytes},
and some other types now use it as default algorithm for @code{hash()}. 
@geindex Python Enhancement Proposals; PEP 552
PEP 552@footnote{https://www.python.org/dev/peps/pep-0552}
hash-based pyc files now use @code{siphash13}, too.
(Contributed by Inada Naoki in bpo-29410@footnote{https://bugs.python.org/issue29410}.)

@item 
When an active exception is re-raised by a @ref{16d,,raise} statement with no parameters,
the traceback attached to this exception is now always @code{sys.exc_info()[1].__traceback__}.
This means that changes made to the traceback in the current @ref{16e,,except} clause are
reflected in the re-raised exception.
(Contributed by Irit Katriel in bpo-45711@footnote{https://bugs.python.org/issue45711}.)

@item 
The interpreter state’s representation of handled exceptions (a.k.a exc_info, or
_PyErr_StackItem) now has only the @code{exc_value} field, @code{exc_type} and @code{exc_traceback}
have been removed as their values can be derived from @code{exc_value}.
(Contributed by Irit Katriel in bpo-45711@footnote{https://bugs.python.org/issue45711}.)

@item 
A new command line option for the Windows installer @code{AppendPath} has beend added.
It behaves similiar to @code{PrependPath} but appends the install and scripts directories
instead of prepending it.
(Contributed by Bastian Neuburger in bpo-44934@footnote{https://bugs.python.org/issue44934}.)
@end itemize

@node New Modules,Improved Modules,Other CPython Implementation Changes,What’s New In Python 3 11
@anchor{whatsnew/3 11 new-modules}@anchor{16f}
@subsection New Modules



@itemize *

@item 
None yet.
@end itemize

@node Improved Modules,Optimizations,New Modules,What’s New In Python 3 11
@anchor{whatsnew/3 11 improved-modules}@anchor{170}
@subsection Improved Modules


@menu
* fractions:: 
* IDLE and idlelib:: 
* inspect:: 
* math:: 
* operator:: 
* os:: 
* socket:: 
* sqlite3:: 
* sys:: 
* threading:: 
* time:: 
* unicodedata:: 
* fcntl:: 

@end menu

@node fractions,IDLE and idlelib,,Improved Modules
@anchor{whatsnew/3 11 fractions}@anchor{171}
@subsubsection fractions



@itemize *

@item 
Support 
@geindex Python Enhancement Proposals; PEP 515
PEP 515@footnote{https://www.python.org/dev/peps/pep-0515}-style initialization of @ref{172,,Fraction} from
string.  (Contributed by Sergey B Kirpichev in bpo-44258@footnote{https://bugs.python.org/issue44258}.)

@item 
@ref{172,,Fraction} now implements an @code{__int__} method, so
that an @code{isinstance(some_fraction, typing.SupportsInt)} check passes.
(Contributed by Mark Dickinson in bpo-44547@footnote{https://bugs.python.org/issue44547}.)
@end itemize

@node IDLE and idlelib,inspect,fractions,Improved Modules
@anchor{whatsnew/3 11 idle-and-idlelib}@anchor{173}
@subsubsection IDLE and idlelib



@itemize *

@item 
Apply syntax highlighting to @cite{.pyi} files. (Contributed by Alex
Waygood and Terry Jan Reedy in bpo-45447@footnote{https://bugs.python.org/issue45447}.)
@end itemize

@node inspect,math,IDLE and idlelib,Improved Modules
@anchor{whatsnew/3 11 inspect}@anchor{174}
@subsubsection inspect



@itemize *

@item 
Add @ref{175,,inspect.getmembers_static()}: return all members without
triggering dynamic lookup via the descriptor protocol. (Contributed by
Weipeng Hong in bpo-30533@footnote{https://bugs.python.org/issue30533}.)

@item 
Add @ref{176,,inspect.ismethodwrapper()} for checking if the type of an object is a
@ref{177,,MethodWrapperType}. (Contributed by Hakan Çelik in bpo-29418@footnote{https://bugs.python.org/issue29418}.)
@end itemize

@node math,operator,inspect,Improved Modules
@anchor{whatsnew/3 11 math}@anchor{178}
@subsubsection math



@itemize *

@item 
Add @ref{179,,math.exp2()}: return 2 raised to the power of x.
(Contributed by Gideon Mitchell in bpo-45917@footnote{https://bugs.python.org/issue45917}.)

@item 
Add @ref{17a,,math.cbrt()}: return the cube root of x.
(Contributed by Ajith Ramachandran in bpo-44357@footnote{https://bugs.python.org/issue44357}.)

@item 
The behaviour of two @ref{17b,,math.pow()} corner cases was changed, for
consistency with the IEEE 754 specification. The operations
@code{math.pow(0.0, -math.inf)} and @code{math.pow(-0.0, -math.inf)} now return
@code{inf}. Previously they raised @ref{17c,,ValueError}. (Contributed by Mark
Dickinson in bpo-44339@footnote{https://bugs.python.org/issue44339}.)
@end itemize

@node operator,os,math,Improved Modules
@anchor{whatsnew/3 11 operator}@anchor{17d}
@subsubsection operator



@itemize *

@item 
A new function @code{operator.call} has been added, such that
@code{operator.call(obj, *args, **kwargs) == obj(*args, **kwargs)}.
(Contributed by Antony Lee in bpo-44019@footnote{https://bugs.python.org/issue44019}.)
@end itemize

@node os,socket,operator,Improved Modules
@anchor{whatsnew/3 11 os}@anchor{17e}
@subsubsection os



@itemize *

@item 
On Windows, @ref{17f,,os.urandom()} now uses @code{BCryptGenRandom()},
instead of @code{CryptGenRandom()} which is deprecated.
(Contributed by Dong-hee Na in bpo-44611@footnote{https://bugs.python.org/issue44611}.)
@end itemize

@node socket,sqlite3,os,Improved Modules
@anchor{whatsnew/3 11 socket}@anchor{180}
@subsubsection socket



@itemize *

@item 
Add CAN Socket support for NetBSD.
(Contributed by Thomas Klausner in bpo-30512@footnote{https://bugs.python.org/issue30512}.)
@end itemize

@node sqlite3,sys,socket,Improved Modules
@anchor{whatsnew/3 11 sqlite3}@anchor{181}
@subsubsection sqlite3



@itemize *

@item 
You can now disable the authorizer by passing @ref{182,,None} to
@ref{183,,set_authorizer()}.
(Contributed by Erlend E. Aasland in bpo-44491@footnote{https://bugs.python.org/issue44491}.)

@item 
Collation name @ref{184,,create_collation()} can now
contain any Unicode character.  Collation names with invalid characters
now raise @ref{185,,UnicodeEncodeError} instead of @ref{186,,sqlite3.ProgrammingError}.
(Contributed by Erlend E. Aasland in bpo-44688@footnote{https://bugs.python.org/issue44688}.)

@item 
@ref{ed,,sqlite3} exceptions now include the SQLite extended error code as
@ref{187,,sqlite_errorcode} and the SQLite error name as
@ref{188,,sqlite_errorname}.
(Contributed by Aviv Palivoda, Daniel Shahaf, and Erlend E. Aasland in
bpo-16379@footnote{https://bugs.python.org/issue16379} and bpo-24139@footnote{https://bugs.python.org/issue24139}.)

@item 
Add @ref{189,,setlimit()} and
@ref{18a,,getlimit()} to @ref{18b,,sqlite3.Connection} for
setting and getting SQLite limits by connection basis.
(Contributed by Erlend E. Aasland in bpo-45243@footnote{https://bugs.python.org/issue45243}.)

@item 
@ref{ed,,sqlite3} now sets @ref{18c,,sqlite3.threadsafety} based on the default
threading mode the underlying SQLite library has been compiled with.
(Contributed by Erlend E. Aasland in bpo-45613@footnote{https://bugs.python.org/issue45613}.)

@item 
@ref{ed,,sqlite3} C callbacks now use unraisable exceptions if callback
tracebacks are enabled. Users can now register an
@ref{18d,,unraisable hook handler} to improve their debug
experience.
(Contributed by Erlend E. Aasland in bpo-45828@footnote{https://bugs.python.org/issue45828}.)

@item 
Fetch across rollback no longer raises @code{InterfaceError}.
Instead we leave it to the SQLite library to handle these cases.
(Contributed by Erlend E. Aasland in bpo-44092@footnote{https://bugs.python.org/issue44092}.)
@end itemize

@node sys,threading,sqlite3,Improved Modules
@anchor{whatsnew/3 11 sys}@anchor{18e}
@subsubsection sys



@itemize *

@item 
@ref{18f,,sys.exc_info()} now derives the @code{type} and @code{traceback} fields
from the @code{value} (the exception instance), so when an exception is
modified while it is being handled, the changes are reflected in
the results of subsequent calls to @code{exc_info()}.
(Contributed by Irit Katriel in bpo-45711@footnote{https://bugs.python.org/issue45711}.)

@item 
Add @ref{190,,sys.exception()} which returns the active exception instance
(equivalent to @code{sys.exc_info()[1]}).
(Contributed by Irit Katriel in bpo-46328@footnote{https://bugs.python.org/issue46328}.)
@end itemize

@node threading,time,sys,Improved Modules
@anchor{whatsnew/3 11 threading}@anchor{191}
@subsubsection threading



@itemize *

@item 
On Unix, if the @code{sem_clockwait()} function is available in the C library
(glibc 2.30 and newer), the @ref{192,,threading.Lock.acquire()} method now uses
the monotonic clock (@ref{193,,time.CLOCK_MONOTONIC}) for the timeout, rather
than using the system clock (@ref{194,,time.CLOCK_REALTIME}), to not be affected
by system clock changes.
(Contributed by Victor Stinner in bpo-41710@footnote{https://bugs.python.org/issue41710}.)
@end itemize

@node time,unicodedata,threading,Improved Modules
@anchor{whatsnew/3 11 time}@anchor{195}
@subsubsection time



@itemize *

@item 
On Unix, @ref{196,,time.sleep()} now uses the @code{clock_nanosleep()} or
@code{nanosleep()} function, if available, which has a resolution of 1 nanosecond
(10@w{^-9} seconds), rather than using @code{select()} which has a resolution
of 1 microsecond (10@w{^-6} seconds).
(Contributed by Benjamin Szőke and Victor Stinner in bpo-21302@footnote{https://bugs.python.org/issue21302}.)

@item 
On Windows 8.1 and newer, @ref{196,,time.sleep()} now uses a waitable timer based
on high-resolution timers@footnote{https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/high-resolution-timers}
which has a resolution of 100 nanoseconds (10@w{^-7} seconds). Previously,
it had a resolution of 1 millisecond (10@w{^-3} seconds).
(Contributed by Benjamin Szőke, Dong-hee Na, Eryk Sun and Victor Stinner in bpo-21302@footnote{https://bugs.python.org/issue21302} and bpo-45429@footnote{https://bugs.python.org/issue45429}.)
@end itemize

@node unicodedata,fcntl,time,Improved Modules
@anchor{whatsnew/3 11 unicodedata}@anchor{197}
@subsubsection unicodedata



@itemize *

@item 
The Unicode database has been updated to version 14.0.0. (bpo-45190@footnote{https://bugs.python.org/issue45190}).
@end itemize

@node fcntl,,unicodedata,Improved Modules
@anchor{whatsnew/3 11 fcntl}@anchor{198}
@subsubsection fcntl



@itemize *

@item 
On FreeBSD, the @code{F_DUP2FD} and @code{F_DUP2FD_CLOEXEC} flags respectively
are supported, the former equals to @code{dup2} usage while the latter set
the @code{FD_CLOEXEC} flag in addition.
@end itemize

@node Optimizations,CPython bytecode changes,Improved Modules,What’s New In Python 3 11
@anchor{whatsnew/3 11 optimizations}@anchor{199}
@subsection Optimizations



@itemize *

@item 
Compiler now optimizes simple C-style formatting with literal format
containing only format codes @code{%s}, @code{%r} and @code{%a} and makes it as
fast as corresponding f-string expression.
(Contributed by Serhiy Storchaka in bpo-28307@footnote{https://bugs.python.org/issue28307}.)

@item 
“Zero-cost” exceptions are implemented. The cost of @code{try} statements is
almost eliminated when no exception is raised.
(Contributed by Mark Shannon in bpo-40222@footnote{https://bugs.python.org/issue40222}.)

@item 
Method calls with keywords are now faster due to bytecode
changes which avoid creating bound method instances. Previously, this
optimization was applied only to method calls with purely positional
arguments.
(Contributed by Ken Jin and Mark Shannon in bpo-26110@footnote{https://bugs.python.org/issue26110}, based on ideas
implemented in PyPy.)

@item 
Pure ASCII strings are now normalized in constant time by @ref{19a,,unicodedata.normalize()}.
(Contributed by Dong-hee Na in bpo-44987@footnote{https://bugs.python.org/issue44987}.)

@item 
@ref{ad,,math} functions @ref{19b,,comb()} and @ref{19c,,perm()} are now up
to 10 times or more faster for large arguments (the speed up is larger for
larger @emph{k}).
(Contributed by Serhiy Storchaka in bpo-37295@footnote{https://bugs.python.org/issue37295}.)

@item 
Dict don’t store hash value when all inserted keys are Unicode objects.
This reduces dict size. For example, @code{sys.getsizeof(dict.fromkeys("abcdefg"))}
becomes 272 bytes from 352 bytes on 64bit platform.
(Contributed by Inada Naoki in bpo-46845@footnote{https://bugs.python.org/issue46845}.)
@end itemize

@node CPython bytecode changes,Deprecated,Optimizations,What’s New In Python 3 11
@anchor{whatsnew/3 11 cpython-bytecode-changes}@anchor{19d}
@subsection CPython bytecode changes



@itemize *

@item 
Replaced all numeric @code{BINARY_*} and @code{INPLACE_*} instructions with a single
@ref{19e,,BINARY_OP} implementation.

@item 
Replaced the three call instructions: @code{CALL_FUNCTION},
@code{CALL_FUNCTION_KW} and @code{CALL_METHOD} with
@ref{19f,,PUSH_NULL}, @ref{1a0,,PRECALL}, @ref{1a1,,CALL},
and @ref{1a2,,KW_NAMES}.
This decouples the argument shifting for methods from the handling of
keyword arguments and allows better specialization of calls.

@item 
Removed @code{COPY_DICT_WITHOUT_KEYS} and @code{GEN_START}.

@item 
@ref{1a3,,MATCH_CLASS} and @ref{1a4,,MATCH_KEYS} no longer push an additional
boolean value indicating whether the match succeeded or failed. Instead, they
indicate failure with @ref{182,,None} (where a tuple of extracted values would
otherwise be).

@item 
Replace several stack manipulation instructions (@code{DUP_TOP}, @code{DUP_TOP_TWO},
@code{ROT_TWO}, @code{ROT_THREE}, @code{ROT_FOUR}, and @code{ROT_N}) with new
@ref{1a5,,COPY} and @ref{1a6,,SWAP} instructions.

@item 
Add @ref{1a7,,POP_JUMP_IF_NOT_NONE} and @ref{1a8,,POP_JUMP_IF_NONE} opcodes to
speed up conditional jumps.

@item 
@ref{1a9,,JUMP_IF_NOT_EXC_MATCH} no longer pops the active exception.
@end itemize

@node Deprecated,Removed,CPython bytecode changes,What’s New In Python 3 11
@anchor{whatsnew/3 11 deprecated}@anchor{1aa}
@subsection Deprecated



@itemize *

@item 
The @ref{a3,,lib2to3} package and @code{2to3} tool are now deprecated and may not
be able to parse Python 3.10 or newer. See the 
@geindex Python Enhancement Proposals; PEP 617
PEP 617@footnote{https://www.python.org/dev/peps/pep-0617} (New PEG parser for
CPython).  (Contributed by Victor Stinner in bpo-40360@footnote{https://bugs.python.org/issue40360}.)

@item 
@code{webbrowser.MacOSX} is deprecated and will be removed in Python 3.13.
It is untested and undocumented and also not used by webbrowser itself.
(Contributed by Dong-hee Na in bpo-42255@footnote{https://bugs.python.org/issue42255}.)

@item 
The behavior of returning a value from a @ref{1ab,,TestCase} and
@ref{1ac,,IsolatedAsyncioTestCase} test methods (other than the
default @code{None} value), is now deprecated.

@item 
Deprecated the following @ref{122,,unittest} functions, scheduled for removal in
Python 3.13:


@itemize *

@item 
@code{unittest.findTestCases()}

@item 
@code{unittest.makeSuite()}

@item 
@code{unittest.getTestCaseNames()}
@end itemize

Use @ref{1ad,,TestLoader} method instead:


@itemize *

@item 
@ref{1ae,,unittest.TestLoader.loadTestsFromModule()}

@item 
@ref{1af,,unittest.TestLoader.loadTestsFromTestCase()}

@item 
@ref{1b0,,unittest.TestLoader.getTestCaseNames()}
@end itemize

(Contributed by Erlend E. Aasland in bpo-5846@footnote{https://bugs.python.org/issue5846}.)

@item 
The @code{turtle.RawTurtle.settiltangle()} is deprecated since Python 3.1,
it now emits a deprecation warning and will be removed in Python 3.13. Use
@code{turtle.RawTurtle.tiltangle()} instead (it was earlier incorrectly marked
as deprecated, its docstring is now corrected).
(Contributed by Hugo van Kemenade in bpo-45837@footnote{https://bugs.python.org/issue45837}.)

@item 
The delegation of @ref{1b1,,int()} to @code{__trunc__()} is now deprecated. Calling
@code{int(a)} when @code{type(a)} implements @code{__trunc__()} but not
@code{__int__()} or @code{__index__()} now raises a @ref{1b2,,DeprecationWarning}.
(Contributed by Zackery Spytz in bpo-44977@footnote{https://bugs.python.org/issue44977}.)

@item 
The following have been deprecated in @ref{21,,configparser} since Python 3.2.
Their deprecation warnings have now been updated to note they will removed in
Python 3.12:


@itemize *

@item 
the @code{configparser.SafeConfigParser} class

@item 
the @code{configparser.ParsingError.filename} property

@item 
the @code{configparser.ParsingError.readfp()} method
@end itemize

(Contributed by Hugo van Kemenade in bpo-45173@footnote{https://bugs.python.org/issue45173}.)

@item 
The @ref{1b3,,locale.getdefaultlocale()} function is deprecated and will be
removed in Python 3.13. Use @ref{1b4,,locale.setlocale()},
@ref{1b5,,locale.getpreferredencoding(False)} and
@ref{1b6,,locale.getlocale()} functions instead.
(Contributed by Victor Stinner in bpo-46659@footnote{https://bugs.python.org/issue46659}.)
@end itemize

@node Removed,Porting to Python 3 11,Deprecated,What’s New In Python 3 11
@anchor{whatsnew/3 11 removed}@anchor{1b7}
@subsection Removed



@itemize *

@item 
@code{smtpd.MailmanProxy} is now removed as it is unusable without
an external module, @code{mailman}. (Contributed by Dong-hee Na in bpo-35800@footnote{https://bugs.python.org/issue35800}.)

@item 
The @code{binhex} module, deprecated in Python 3.9, is now removed.
The following @ref{f,,binascii} functions, deprecated in Python 3.9, are now
also removed:


@itemize *

@item 
@code{a2b_hqx()}, @code{b2a_hqx()};

@item 
@code{rlecode_hqx()}, @code{rledecode_hqx()}.
@end itemize

The @ref{1b8,,binascii.crc_hqx()} function remains available.

(Contributed by Victor Stinner in bpo-45085@footnote{https://bugs.python.org/issue45085}.)

@item 
The distutils @code{bdist_msi} command, deprecated in Python 3.9, is now removed.
Use @code{bdist_wheel} (wheel packages) instead.
(Contributed by Hugo van Kemenade in bpo-45124@footnote{https://bugs.python.org/issue45124}.)

@item 
Due to significant security concerns, the @emph{reuse_address} parameter of
@ref{1b9,,asyncio.loop.create_datagram_endpoint()}, disabled in Python 3.9, is
now entirely removed. This is because of the behavior of the socket option
@code{SO_REUSEADDR} in UDP.
(Contributed by Hugo van Kemenade in bpo-45129@footnote{https://bugs.python.org/issue45129}.)

@item 
Removed @code{__getitem__()} methods of
@ref{1ba,,xml.dom.pulldom.DOMEventStream}, @ref{1bb,,wsgiref.util.FileWrapper}
and @ref{1bc,,fileinput.FileInput}, deprecated since Python 3.9.
(Contributed by Hugo van Kemenade in bpo-45132@footnote{https://bugs.python.org/issue45132}.)

@item 
The following deprecated functions and methods are removed in the @ref{83,,gettext}
module: @code{lgettext()}, @code{ldgettext()},
@code{lngettext()} and @code{ldngettext()}.

Function @code{bind_textdomain_codeset()}, methods
@code{output_charset()} and
@code{set_output_charset()}, and the @emph{codeset}
parameter of functions @ref{1bd,,translation()} and
@ref{1be,,install()} are also removed, since they are only used for
the @code{l*gettext()} functions.
(Contributed by Dong-hee Na and Serhiy Storchaka in bpo-44235@footnote{https://bugs.python.org/issue44235}.)

@item 
The @code{@@asyncio.coroutine} @ref{1bf,,decorator} enabling
legacy generator-based coroutines to be compatible with async/await code.
The function has been deprecated since Python 3.8 and the removal was
initially scheduled for Python 3.10. Use @ref{1c0,,async def} instead.
(Contributed by Illia Volochii in bpo-43216@footnote{https://bugs.python.org/issue43216}.)

@item 
@code{asyncio.coroutines.CoroWrapper} used for wrapping legacy
generator-based coroutine objects in the debug mode.
(Contributed by Illia Volochii in bpo-43216@footnote{https://bugs.python.org/issue43216}.)

@item 
Removed the deprecated @code{split()} method of @code{_tkinter.TkappType}.
(Contributed by Erlend E. Aasland in bpo-38371@footnote{https://bugs.python.org/issue38371}.)

@item 
Removed from the @ref{9c,,inspect} module:


@itemize *

@item 
the @code{getargspec} function, deprecated since Python 3.0;
use @ref{1c1,,inspect.signature()} or @ref{1c2,,inspect.getfullargspec()} instead.

@item 
the @code{formatargspec} function, deprecated since Python 3.5;
use the @ref{1c1,,inspect.signature()} function and @code{Signature} object
directly.

@item 
the undocumented @code{Signature.from_builtin} and @code{Signature.from_function}
functions, deprecated since Python 3.5; use the
@ref{1c3,,Signature.from_callable()} method
instead.
@end itemize

(Contributed by Hugo van Kemenade in bpo-45320@footnote{https://bugs.python.org/issue45320}.)

@item 
Remove namespace package support from unittest discovery. It was introduced in
Python 3.4 but has been broken since Python 3.7.
(Contributed by Inada Naoki in bpo-23882@footnote{https://bugs.python.org/issue23882}.)

@item 
Remove @code{__class_getitem__} method from @ref{1c4,,pathlib.PurePath},
because it was not used and added by mistake in previous versions.
(Contributed by Nikita Sobolev in bpo-46483@footnote{https://bugs.python.org/issue46483}.)

@item 
Remove the undocumented private @code{float.__set_format__()} method, previously
known as @code{float.__setformat__()} in Python 3.7. Its docstring said: “You
probably don’t want to use this function. It exists mainly to be used in
Python’s test suite.”
(Contributed by Victor Stinner in bpo-46852@footnote{https://bugs.python.org/issue46852}.)
@end itemize

@node Porting to Python 3 11,Build Changes,Removed,What’s New In Python 3 11
@anchor{whatsnew/3 11 porting-to-python-3-11}@anchor{1c5}
@subsection Porting to Python 3.11


This section lists previously described changes and other bugfixes
that may require changes to your code.

@menu
* Changes in the Python API:: 

@end menu

@node Changes in the Python API,,,Porting to Python 3 11
@anchor{whatsnew/3 11 changes-in-the-python-api}@anchor{1c6}
@subsubsection Changes in the Python API



@itemize *

@item 
Prohibited passing non-@ref{1c7,,concurrent.futures.ThreadPoolExecutor}
executors to @code{loop.set_default_executor()} following a deprecation in
Python 3.8.
(Contributed by Illia Volochii in bpo-43234@footnote{https://bugs.python.org/issue43234}.)

@item 
@ref{1c8,,open()}, @ref{1c9,,io.open()}, @ref{1ca,,codecs.open()} and
@ref{1bc,,fileinput.FileInput} no longer accept @code{'U'} (“universal newline”)
in the file mode. This flag was deprecated since Python 3.3. In Python 3, the
“universal newline” is used by default when a file is open in text mode.  The
@ref{1cb,,newline parameter} of @ref{1c8,,open()} controls
how universal newlines works.
(Contributed by Victor Stinner in bpo-37330@footnote{https://bugs.python.org/issue37330}.)

@item 
The @ref{c4,,pdb} module now reads the @code{.pdbrc} configuration file with
the @code{'utf-8'} encoding.
(Contributed by Srinivas Reddy Thatiparthy (శ్రీనివాస్  రెడ్డి తాటిపర్తి) in bpo-41137@footnote{https://bugs.python.org/issue41137}.)

@item 
When sorting using tuples as keys, the order of the result may differ
from earlier releases if the tuple elements don’t define a total
ordering (see @ref{1cc,,Value comparisons} for
information on total ordering).  It’s generally true that the result
of sorting simply isn’t well-defined in the absence of a total ordering
on list elements.

@item 
@ref{13,,calendar}: The @ref{1cd,,calendar.LocaleTextCalendar} and
@ref{1ce,,calendar.LocaleHTMLCalendar} classes now use
@ref{1b6,,locale.getlocale()}, instead of using @ref{1b3,,locale.getdefaultlocale()},
if no locale is specified.
(Contributed by Victor Stinner in bpo-46659@footnote{https://bugs.python.org/issue46659}.)
@end itemize

@node Build Changes,C API Changes,Porting to Python 3 11,What’s New In Python 3 11
@anchor{whatsnew/3 11 build-changes}@anchor{1cf}
@subsection Build Changes



@itemize *

@item 
Building Python now requires a C11 compiler without optional C11 features.
(Contributed by Victor Stinner in bpo-46656@footnote{https://bugs.python.org/issue46656}.)

@item 
CPython can now be built with the ThinLTO option via @code{--with-lto=thin}.
(Contributed by Dong-hee Na and Brett Holman in bpo-44340@footnote{https://bugs.python.org/issue44340}.)

@item 
libpython is no longer linked against libcrypt.
(Contributed by Mike Gilbert in bpo-45433@footnote{https://bugs.python.org/issue45433}.)

@item 
Building Python now requires a C99 @code{<math.h>} header file providing
the following functions: @code{copysign()}, @code{hypot()}, @code{isfinite()},
@code{isinf()}, @code{isnan()}, @code{round()}.
(Contributed by Victor Stinner in bpo-45440@footnote{https://bugs.python.org/issue45440}.)

@item 
Building Python now requires a C99 @code{<math.h>} header file providing
a @code{NAN} constant, or the @code{__builtin_nan()} built-in function.
(Contributed by Victor Stinner in bpo-46640@footnote{https://bugs.python.org/issue46640}.)

@item 
Building Python now requires support for floating point Not-a-Number (NaN):
remove the @code{Py_NO_NAN} macro.
(Contributed by Victor Stinner in bpo-46656@footnote{https://bugs.python.org/issue46656}.)

@item 
Freelists for object structs can now be disabled. A new @code{configure}
option @code{--without-freelists} can be used to disable all freelists
except empty tuple singleton.
(Contributed by Christian Heimes in bpo-45522@footnote{https://bugs.python.org/issue45522})

@item 
@code{Modules/Setup} and @code{Modules/makesetup} have been improved and tied up.
Extension modules can now be built through @code{makesetup}. All except some
test modules can be linked statically into main binary or library.
(Contributed by Brett Cannon and Christian Heimes in bpo-45548@footnote{https://bugs.python.org/issue45548},
bpo-45570@footnote{https://bugs.python.org/issue45570}, bpo-45571@footnote{https://bugs.python.org/issue45571}, and bpo-43974@footnote{https://bugs.python.org/issue43974}.)

@item 
Build dependencies, compiler flags, and linker flags for most stdlib
extension modules are now detected by @code{configure}. libffi, libnsl,
libsqlite3, zlib, bzip2, liblzma, libcrypt, and uuid flags are detected by
@code{pkg-config} (when available).
(Contributed by Christian Heimes and Erlend Egeberg Aasland in
bpo-bpo-45847@footnote{https://bugs.python.org/issuebpo-45847}, bpo-45747@footnote{https://bugs.python.org/issue45747}, and bpo-45763@footnote{https://bugs.python.org/issue45763}.)

@item 
CPython now has experimental support for cross compiling to WebAssembly
platform @code{wasm32-emscripten}. The effort is inspired by previous work
like Pyodide.
(Contributed by Christian Heimes and Ethan Smith in bpo-40280@footnote{https://bugs.python.org/issue40280}.)

@item 
CPython will now use 30-bit digits by default for the Python @ref{1b1,,int}
implementation. Previously, the default was to use 30-bit digits on platforms
with @code{SIZEOF_VOID_P >= 8}, and 15-bit digits otherwise. It’s still possible
to explicitly request use of 15-bit digits via either the
@code{--enable-big-digits} option to the configure script or (for Windows) the
@code{PYLONG_BITS_IN_DIGIT} variable in @code{PC/pyconfig.h}, but this option may
be removed at some point in the future. (Contributed by Mark Dickinson in
bpo-45569@footnote{https://bugs.python.org/issue45569}.)
@end itemize

@node C API Changes,,Build Changes,What’s New In Python 3 11
@anchor{whatsnew/3 11 c-api-changes}@anchor{1d0}
@subsection C API Changes



@itemize *

@item 
@ref{1d1,,PyErr_SetExcInfo()} no longer uses the @code{type} and @code{traceback}
arguments, the interpreter now derives those values from the exception
instance (the @code{value} argument). The function still steals references
of all three arguments.
(Contributed by Irit Katriel in bpo-45711@footnote{https://bugs.python.org/issue45711}.)

@item 
@ref{1d2,,PyErr_GetExcInfo()} now derives the @code{type} and @code{traceback}
fields of the result from the exception instance (the @code{value} field).
(Contributed by Irit Katriel in bpo-45711@footnote{https://bugs.python.org/issue45711}.)

@item 
@ref{1d3,,_frozen} has a new @code{is_package} field to indicate whether
or not the frozen module is a package.  Previously, a negative value
in the @code{size} field was the indicator.  Now only non-negative values
be used for @code{size}.
(Contributed by Kumar Aditya in bpo-46608@footnote{https://bugs.python.org/issue46608}.)
@end itemize

@menu
* New Features: New Features<2>. 
* Porting to Python 3.11: Porting to Python 3 11<2>. 
* Deprecated: Deprecated<2>. 
* Removed: Removed<2>. 

@end menu

@node New Features<2>,Porting to Python 3 11<2>,,C API Changes
@anchor{whatsnew/3 11 id1}@anchor{1d4}
@subsubsection New Features



@itemize *

@item 
Add a new @ref{1d5,,PyType_GetName()} function to get type’s short name.
(Contributed by Hai Shi in bpo-42035@footnote{https://bugs.python.org/issue42035}.)

@item 
Add a new @ref{1d6,,PyType_GetQualName()} function to get type’s qualified name.
(Contributed by Hai Shi in bpo-42035@footnote{https://bugs.python.org/issue42035}.)

@item 
Add new @ref{1d7,,PyThreadState_EnterTracing()} and
@ref{1d8,,PyThreadState_LeaveTracing()} functions to the limited C API to
suspend and resume tracing and profiling.
(Contributed by Victor Stinner in bpo-43760@footnote{https://bugs.python.org/issue43760}.)

@item 
Added the @ref{1d9,,Py_Version} constant which bears the same value as
@ref{1da,,PY_VERSION_HEX}.
(Contributed by  Gabriele N. Tornetta in bpo-43931@footnote{https://bugs.python.org/issue43931}.)

@item 
@ref{1db,,Py_buffer} and APIs are now part of the limited API and the stable
ABI:


@itemize *

@item 
@ref{1dc,,PyObject_CheckBuffer()}

@item 
@ref{1dd,,PyObject_GetBuffer()}

@item 
@ref{1de,,PyBuffer_GetPointer()}

@item 
@ref{1df,,PyBuffer_SizeFromFormat()}

@item 
@ref{1e0,,PyBuffer_ToContiguous()}

@item 
@ref{1e1,,PyBuffer_FromContiguous()}

@item 
@code{PyBuffer_CopyData()}

@item 
@ref{1e2,,PyBuffer_IsContiguous()}

@item 
@ref{1e3,,PyBuffer_FillContiguousStrides()}

@item 
@ref{1e4,,PyBuffer_FillInfo()}

@item 
@ref{1e5,,PyBuffer_Release()}

@item 
@ref{1e6,,PyMemoryView_FromBuffer()}

@item 
@ref{1e7,,bf_getbuffer} and
@ref{1e8,,bf_releasebuffer} type slots
@end itemize

(Contributed by Christian Heimes in bpo-45459@footnote{https://bugs.python.org/issue45459}.)

@item 
Added the @ref{1e9,,PyType_GetModuleByDef} function, used to get the module
in which a method was defined, in cases where this information is not
available directly (via @ref{1ea,,PyCMethod}).
(Contributed by Petr Viktorin in bpo-46613@footnote{https://bugs.python.org/issue46613}.)
@end itemize

@node Porting to Python 3 11<2>,Deprecated<2>,New Features<2>,C API Changes
@anchor{whatsnew/3 11 id2}@anchor{1eb}
@subsubsection Porting to Python 3.11



@itemize *

@item 
The old trashcan macros (@code{Py_TRASHCAN_SAFE_BEGIN}/@code{Py_TRASHCAN_SAFE_END})
are now deprecated. They should be replaced by the new macros
@code{Py_TRASHCAN_BEGIN} and @code{Py_TRASHCAN_END}.

A tp_dealloc function that has the old macros, such as:

@example
static void
mytype_dealloc(mytype *p)
@{
    PyObject_GC_UnTrack(p);
    Py_TRASHCAN_SAFE_BEGIN(p);
    ...
    Py_TRASHCAN_SAFE_END
@}
@end example

should migrate to the new macros as follows:

@example
static void
mytype_dealloc(mytype *p)
@{
    PyObject_GC_UnTrack(p);
    Py_TRASHCAN_BEGIN(p, mytype_dealloc)
    ...
    Py_TRASHCAN_END
@}
@end example

Note that @code{Py_TRASHCAN_BEGIN} has a second argument which
should be the deallocation function it is in.

To support older Python versions in the same codebase, you
can define the following macros and use them throughout
the code (credit: these were copied from the @code{mypy} codebase):

@example
#if PY_MAJOR_VERSION >= 3 && PY_MINOR_VERSION >= 8
#  define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_BEGIN(op, dealloc)
#  define CPy_TRASHCAN_END(op) Py_TRASHCAN_END
#else
#  define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_SAFE_BEGIN(op)
#  define CPy_TRASHCAN_END(op) Py_TRASHCAN_SAFE_END(op)
#endif
@end example

@item 
The @ref{1ec,,PyType_Ready()} function now raises an error if a type is defined
with the @ref{1ed,,Py_TPFLAGS_HAVE_GC} flag set but has no traverse function
(@ref{1ee,,PyTypeObject.tp_traverse}).
(Contributed by Victor Stinner in bpo-44263@footnote{https://bugs.python.org/issue44263}.)

@item 
Heap types with the @ref{1ef,,Py_TPFLAGS_IMMUTABLETYPE} flag can now inherit
the 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590} vectorcall protocol.  Previously, this was only possible for
@ref{1f0,,static types}.
(Contributed by Erlend E. Aasland in bpo-43908@footnote{https://bugs.python.org/issue43908})

@item 
Since @ref{1f1,,Py_TYPE()} is changed to a inline static function,
@code{Py_TYPE(obj) = new_type} must be replaced with
@code{Py_SET_TYPE(obj, new_type)}: see the @ref{1f2,,Py_SET_TYPE()} function
(available since Python 3.9). For backward compatibility, this macro can be
used:

@example
#if PY_VERSION_HEX < 0x030900A4 && !defined(Py_SET_TYPE)
static inline void _Py_SET_TYPE(PyObject *ob, PyTypeObject *type)
@{ ob->ob_type = type; @}
#define Py_SET_TYPE(ob, type) _Py_SET_TYPE((PyObject*)(ob), type)
#endif
@end example

(Contributed by Victor Stinner in bpo-39573@footnote{https://bugs.python.org/issue39573}.)

@item 
Since @ref{1f3,,Py_SIZE()} is changed to a inline static function,
@code{Py_SIZE(obj) = new_size} must be replaced with
@code{Py_SET_SIZE(obj, new_size)}: see the @ref{1f4,,Py_SET_SIZE()} function
(available since Python 3.9). For backward compatibility, this macro can be
used:

@example
#if PY_VERSION_HEX < 0x030900A4 && !defined(Py_SET_SIZE)
static inline void _Py_SET_SIZE(PyVarObject *ob, Py_ssize_t size)
@{ ob->ob_size = size; @}
#define Py_SET_SIZE(ob, size) _Py_SET_SIZE((PyVarObject*)(ob), size)
#endif
@end example

(Contributed by Victor Stinner in bpo-39573@footnote{https://bugs.python.org/issue39573}.)

@item 
@code{<Python.h>} no longer includes the header files @code{<stdlib.h>},
@code{<stdio.h>}, @code{<errno.h>} and @code{<string.h>} when the @code{Py_LIMITED_API}
macro is set to @code{0x030b0000} (Python 3.11) or higher. C extensions should
explicitly include the header files after @code{#include <Python.h>}.
(Contributed by Victor Stinner in bpo-45434@footnote{https://bugs.python.org/issue45434}.)

@item 
The non-limited API files @code{cellobject.h}, @code{classobject.h}, @code{context.h},
@code{funcobject.h}, @code{genobject.h} and @code{longintrepr.h} have been moved to
the @code{Include/cpython} directory. Moreover, the @code{eval.h} header file was
removed. These files must not be included directly, as they are already
included in @code{Python.h}: @ref{1f5,,Include Files}. If they have
been included directly, consider including @code{Python.h} instead.
(Contributed by Victor Stinner in bpo-35134@footnote{https://bugs.python.org/issue35134}.)

@item 
The @code{PyUnicode_CHECK_INTERNED()} macro has been excluded from the
limited C API. It was never usable there, because it used internal structures
which are not available in the limited C API.
(Contributed by Victor Stinner in bpo-46007@footnote{https://bugs.python.org/issue46007}.)

@item 
The @ref{1f6,,PyFrameObject} structure member has been moved to the internal C
API headers.

While the documentation notes that the @ref{1f6,,PyFrameObject} fields are
subject to change at any time, they have been stable for a long time and were
used in several popular extensions.

In Python 3.11, the frame struct was reorganized to allow performance
optimizations. Some fields were removed entirely, as they were details of the
old implementation.

@ref{1f6,,PyFrameObject} fields:


@itemize *

@item 
@code{f_back}: use @ref{1f7,,PyFrame_GetBack()}.

@item 
@code{f_blockstack}: removed.

@item 
@code{f_builtins}: use @code{PyObject_GetAttrString((PyObject*)frame, "f_builtins")}.

@item 
@code{f_code}: use @ref{1f8,,PyFrame_GetCode()}.

@item 
@code{f_gen}: removed.

@item 
@code{f_globals}: use @code{PyObject_GetAttrString((PyObject*)frame, "f_globals")}.

@item 
@code{f_iblock}: removed.

@item 
@code{f_lasti}: use @code{PyObject_GetAttrString((PyObject*)frame, "f_lasti")}.
Code using @code{f_lasti} with @code{PyCode_Addr2Line()} must use
@ref{1f9,,PyFrame_GetLineNumber()} instead.

@item 
@code{f_lineno}: use @ref{1f9,,PyFrame_GetLineNumber()}

@item 
@code{f_locals}: use @code{PyObject_GetAttrString((PyObject*)frame, "f_locals")}.

@item 
@code{f_stackdepth}: removed.

@item 
@code{f_state}: no public API (renamed to @code{f_frame.f_state}).

@item 
@code{f_trace}: no public API.

@item 
@code{f_trace_lines}: use @code{PyObject_GetAttrString((PyObject*)frame, "f_trace_lines")}
(it also be modified).

@item 
@code{f_trace_opcodes}: use @code{PyObject_GetAttrString((PyObject*)frame, "f_trace_opcodes")}
(it also be modified).

@item 
@code{f_localsplus}: no public API (renamed to @code{f_frame.localsplus}).

@item 
@code{f_valuestack}: removed.
@end itemize

The Python frame object is now created lazily. A side effect is that the
@code{f_back} member must not be accessed directly, since its value is now also
computed lazily. The @ref{1f7,,PyFrame_GetBack()} function must be called
instead.

Code defining @code{PyFrame_GetCode()} on Python 3.8 and older:

@example
#if PY_VERSION_HEX < 0x030900B1
static inline PyCodeObject* PyFrame_GetCode(PyFrameObject *frame)
@{
    Py_INCREF(frame->f_code);
    return frame->f_code;
@}
#endif
@end example

Code defining @code{PyFrame_GetBack()} on Python 3.8 and older:

@example
#if PY_VERSION_HEX < 0x030900B1
static inline PyFrameObject* PyFrame_GetBack(PyFrameObject *frame)
@{
    Py_XINCREF(frame->f_back);
    return frame->f_back;
@}
#endif
@end example

Or use the pythoncapi_compat project@footnote{https://github.com/python/pythoncapi_compat} to get these two
functions on older Python versions.

@item 
Changes of the @ref{1fa,,PyThreadState} structure members:


@itemize *

@item 
@code{frame}: removed, use @ref{1fb,,PyThreadState_GetFrame()} (function added
to Python 3.9 by bpo-40429@footnote{https://bugs.python.org/issue40429}).
Warning: the function returns a @ref{1fc,,strong reference}, need to call
@ref{1fd,,Py_XDECREF()}.

@item 
@code{tracing}: changed, use @ref{1d7,,PyThreadState_EnterTracing()}
and @ref{1d8,,PyThreadState_LeaveTracing()}
(functions added to Python 3.11 by bpo-43760@footnote{https://bugs.python.org/issue43760}).

@item 
@code{recursion_depth}: removed,
use @code{(tstate->recursion_limit - tstate->recursion_remaining)} instead.

@item 
@code{stackcheck_counter}: removed.
@end itemize

Code defining @code{PyThreadState_GetFrame()} on Python 3.8 and older:

@example
#if PY_VERSION_HEX < 0x030900B1
static inline PyFrameObject* PyThreadState_GetFrame(PyThreadState *tstate)
@{
    Py_XINCREF(tstate->frame);
    return tstate->frame;
@}
#endif
@end example

Code defining @code{PyThreadState_EnterTracing()} and
@code{PyThreadState_LeaveTracing()} on Python 3.10 and older:

@example
#if PY_VERSION_HEX < 0x030B00A2
static inline void PyThreadState_EnterTracing(PyThreadState *tstate)
@{
    tstate->tracing++;
#if PY_VERSION_HEX >= 0x030A00A1
    tstate->cframe->use_tracing = 0;
#else
    tstate->use_tracing = 0;
#endif
@}

static inline void PyThreadState_LeaveTracing(PyThreadState *tstate)
@{
    int use_tracing = (tstate->c_tracefunc != NULL || tstate->c_profilefunc != NULL);
    tstate->tracing--;
#if PY_VERSION_HEX >= 0x030A00A1
    tstate->cframe->use_tracing = use_tracing;
#else
    tstate->use_tracing = use_tracing;
#endif
@}
#endif
@end example

Or use the pythoncapi_compat project@footnote{https://github.com/python/pythoncapi_compat} to get these functions
on old Python functions.
@end itemize

@node Deprecated<2>,Removed<2>,Porting to Python 3 11<2>,C API Changes
@anchor{whatsnew/3 11 id3}@anchor{1fe}
@subsubsection Deprecated



@itemize *

@item 
Deprecate the following functions to configure the Python initialization:


@itemize *

@item 
@ref{1ff,,PySys_AddWarnOptionUnicode()}

@item 
@ref{200,,PySys_AddWarnOption()}

@item 
@ref{201,,PySys_AddXOption()}

@item 
@code{PySys_HasWarnOptions()}

@item 
@ref{202,,Py_SetPath()}

@item 
@ref{203,,Py_SetProgramName()}

@item 
@ref{204,,Py_SetPythonHome()}

@item 
@ref{205,,Py_SetStandardStreamEncoding()}

@item 
@code{_Py_SetProgramFullPath()}
@end itemize

Use the new @ref{206,,PyConfig} API of the @ref{207,,Python Initialization Configuration} instead (
@geindex Python Enhancement Proposals; PEP 587
PEP 587@footnote{https://www.python.org/dev/peps/pep-0587}).
(Contributed by Victor Stinner in bpo-44113@footnote{https://bugs.python.org/issue44113}.)

@item 
Deprecate the @code{ob_shash} member of the @ref{208,,PyBytesObject}. Use @ref{209,,PyObject_Hash()} instead.
(Contributed by Inada Naoki in bpo-46864@footnote{https://bugs.python.org/issue46864}.)
@end itemize

@node Removed<2>,,Deprecated<2>,C API Changes
@anchor{whatsnew/3 11 id4}@anchor{20a}
@subsubsection Removed



@itemize *

@item 
@code{PyFrame_BlockSetup()} and @code{PyFrame_BlockPop()} have been
removed.
(Contributed by Mark Shannon in bpo-40222@footnote{https://bugs.python.org/issue40222}.)

@item 
Remove the following math macros using the @code{errno} variable:


@itemize *

@item 
@code{Py_ADJUST_ERANGE1()}

@item 
@code{Py_ADJUST_ERANGE2()}

@item 
@code{Py_OVERFLOWED()}

@item 
@code{Py_SET_ERANGE_IF_OVERFLOW()}

@item 
@code{Py_SET_ERRNO_ON_MATH_ERROR()}
@end itemize

(Contributed by Victor Stinner in bpo-45412@footnote{https://bugs.python.org/issue45412}.)

@item 
Remove @code{Py_UNICODE_COPY()} and @code{Py_UNICODE_FILL()} macros, deprecated
since Python 3.3. Use @code{PyUnicode_CopyCharacters()} or @code{memcpy()}
(@code{wchar_t*} string), and @code{PyUnicode_Fill()} functions instead.
(Contributed by Victor Stinner in bpo-41123@footnote{https://bugs.python.org/issue41123}.)

@item 
Remove the @code{pystrhex.h} header file. It only contains private functions.
C extensions should only include the main @code{<Python.h>} header file.
(Contributed by Victor Stinner in bpo-45434@footnote{https://bugs.python.org/issue45434}.)

@item 
Remove the @code{Py_FORCE_DOUBLE()} macro. It was used by the
@code{Py_IS_INFINITY()} macro.
(Contributed by Victor Stinner in bpo-45440@footnote{https://bugs.python.org/issue45440}.)

@item 
The following items are no longer available when @ref{20b,,Py_LIMITED_API}
is defined:


@itemize *

@item 
@ref{20c,,PyMarshal_WriteLongToFile()}

@item 
@ref{20d,,PyMarshal_WriteObjectToFile()}

@item 
@ref{20e,,PyMarshal_ReadObjectFromString()}

@item 
@ref{20f,,PyMarshal_WriteObjectToString()}

@item 
the @code{Py_MARSHAL_VERSION} macro
@end itemize

These are not part of the @ref{210,,limited API}.

(Contributed by Victor Stinner in bpo-45474@footnote{https://bugs.python.org/issue45474}.)

@item 
Exclude @ref{211,,PyWeakref_GET_OBJECT()} from the limited C API. It never
worked since the @code{PyWeakReference} structure is opaque in the
limited C API.
(Contributed by Victor Stinner in bpo-35134@footnote{https://bugs.python.org/issue35134}.)

@item 
Remove the @code{PyHeapType_GET_MEMBERS()} macro. It was exposed in the
public C API by mistake, it must only be used by Python internally.
Use the @code{PyTypeObject.tp_members} member instead.
(Contributed by Victor Stinner in bpo-40170@footnote{https://bugs.python.org/issue40170}.)

@item 
Remove the @code{HAVE_PY_SET_53BIT_PRECISION} macro (moved to the internal C
API).
(Contributed by Victor Stinner in bpo-45412@footnote{https://bugs.python.org/issue45412}.)
@end itemize

@node What’s New In Python 3 10,What’s New In Python 3 9,What’s New In Python 3 11,What’s New in Python
@anchor{whatsnew/3 10 doc}@anchor{212}@anchor{whatsnew/3 10 what-s-new-in-python-3-10}@anchor{213}
@section What’s New In Python 3.10


@*Release: 
3.11.0a5


@*Date: 
March 07, 2022


@*Editor: 
Pablo Galindo Salgado


@c Rules for maintenance:
@c 
@c * Anyone can add text to this document.  Do not spend very much time
@c on the wording of your changes, because your text will probably
@c get rewritten to some degree.
@c 
@c * The maintainer will go through Misc/NEWS periodically and add
@c changes; it's therefore more important to add your changes to
@c Misc/NEWS than to this file.
@c 
@c * This is not a complete list of every single change; completeness
@c is the purpose of Misc/NEWS.  Some changes I consider too small
@c or esoteric to include.  If such a change is added to the text,
@c I'll just remove it.  (This is another reason you shouldn't spend
@c too much time on writing your addition.)
@c 
@c * If you want to draw your new text to the attention of the
@c maintainer, add 'XXX' to the beginning of the paragraph or
@c section.
@c 
@c * It's OK to just add a fragmentary note about a change.  For
@c example: "XXX Describe the transmogrify() function added to the
@c socket module."  The maintainer will research the change and
@c write the necessary text.
@c 
@c * You can comment out your additions if you like, but it's not
@c necessary (especially when a final release is some months away).
@c 
@c * Credit the author of a patch or bugfix.   Just the name is
@c sufficient; the e-mail address isn't necessary.
@c 
@c * It's helpful to add the bug/patch number as a comment:
@c 
@c XXX Describe the transmogrify() function added to the socket
@c module.
@c (Contributed by P.Y. Developer in :issue:`12345`.)
@c 
@c This saves the maintainer the effort of going through the git log
@c when researching a change.

This article explains the new features in Python 3.10, compared to 3.9.

For full details, see the @ref{154,,changelog}.

@menu
* Summary – Release highlights: Summary – Release highlights<2>. 
* New Features: New Features<3>. 
* New Features Related to Type Hints:: 
* Other Language Changes: Other Language Changes<2>. 
* New Modules: New Modules<2>. 
* Improved Modules: Improved Modules<2>. 
* Optimizations: Optimizations<2>. 
* Deprecated: Deprecated<3>. 
* Removed: Removed<3>. 
* Porting to Python 3.10: Porting to Python 3 10. 
* CPython bytecode changes: CPython bytecode changes<2>. 
* Build Changes: Build Changes<2>. 
* C API Changes: C API Changes<2>. 

@end menu

@node Summary – Release highlights<2>,New Features<3>,,What’s New In Python 3 10
@anchor{whatsnew/3 10 summary-release-highlights}@anchor{214}
@subsection Summary – Release highlights


@c This section singles out the most important changes in Python 3.10.
@c Brevity is key.

@c PEP-sized items next.

New syntax features:


@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 634
PEP 634@footnote{https://www.python.org/dev/peps/pep-0634}, Structural Pattern Matching: Specification

@item 
@geindex Python Enhancement Proposals; PEP 635
PEP 635@footnote{https://www.python.org/dev/peps/pep-0635}, Structural Pattern Matching: Motivation and Rationale

@item 
@geindex Python Enhancement Proposals; PEP 636
PEP 636@footnote{https://www.python.org/dev/peps/pep-0636}, Structural Pattern Matching: Tutorial

@item 
bpo-12782@footnote{https://bugs.python.org/issue12782}, Parenthesized context managers are now officially allowed.
@end itemize

New features in the standard library:


@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 618
PEP 618@footnote{https://www.python.org/dev/peps/pep-0618}, Add Optional Length-Checking To zip.
@end itemize

Interpreter improvements:


@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 626
PEP 626@footnote{https://www.python.org/dev/peps/pep-0626}, Precise line numbers for debugging and other tools.
@end itemize

New typing features:


@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 604
PEP 604@footnote{https://www.python.org/dev/peps/pep-0604}, Allow writing union types as X | Y

@item 
@geindex Python Enhancement Proposals; PEP 613
PEP 613@footnote{https://www.python.org/dev/peps/pep-0613}, Explicit Type Aliases

@item 
@geindex Python Enhancement Proposals; PEP 612
PEP 612@footnote{https://www.python.org/dev/peps/pep-0612}, Parameter Specification Variables
@end itemize

Important deprecations, removals or restrictions:


@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 644
PEP 644@footnote{https://www.python.org/dev/peps/pep-0644}, Require OpenSSL 1.1.1 or newer

@item 
@geindex Python Enhancement Proposals; PEP 632
PEP 632@footnote{https://www.python.org/dev/peps/pep-0632}, Deprecate distutils module.

@item 
@geindex Python Enhancement Proposals; PEP 623
PEP 623@footnote{https://www.python.org/dev/peps/pep-0623}, Deprecate and prepare for the removal of the wstr member in PyUnicodeObject.

@item 
@geindex Python Enhancement Proposals; PEP 624
PEP 624@footnote{https://www.python.org/dev/peps/pep-0624}, Remove Py_UNICODE encoder APIs

@item 
@geindex Python Enhancement Proposals; PEP 597
PEP 597@footnote{https://www.python.org/dev/peps/pep-0597}, Add optional EncodingWarning
@end itemize

@node New Features<3>,New Features Related to Type Hints,Summary – Release highlights<2>,What’s New In Python 3 10
@anchor{whatsnew/3 10 new-features}@anchor{215}
@subsection New Features


@menu
* Parenthesized context managers:: 
* Better error messages:: 
* PEP 626; Precise line numbers for debugging and other tools: PEP 626 Precise line numbers for debugging and other tools. 
* PEP 634; Structural Pattern Matching: PEP 634 Structural Pattern Matching. 
* Optional EncodingWarning and encoding="locale" option:: 

@end menu

@node Parenthesized context managers,Better error messages,,New Features<3>
@anchor{whatsnew/3 10 parenthesized-context-managers}@anchor{216}@anchor{whatsnew/3 10 whatsnew310-pep563}@anchor{217}
@subsubsection Parenthesized context managers


Using enclosing parentheses for continuation across multiple lines
in context managers is now supported. This allows formatting a long
collection of context managers in multiple lines in a similar way
as it was previously possible with import statements. For instance,
all these examples are now valid:

@example
with (CtxManager() as example):
    ...

with (
    CtxManager1(),
    CtxManager2()
):
    ...

with (CtxManager1() as example,
      CtxManager2()):
    ...

with (CtxManager1(),
      CtxManager2() as example):
    ...

with (
    CtxManager1() as example1,
    CtxManager2() as example2
):
    ...
@end example

it is also possible to use a trailing comma at the end of the
enclosed group:

@example
with (
    CtxManager1() as example1,
    CtxManager2() as example2,
    CtxManager3() as example3,
):
    ...
@end example

This new syntax uses the non LL(1) capacities of the new parser.
Check 
@geindex Python Enhancement Proposals; PEP 617
PEP 617@footnote{https://www.python.org/dev/peps/pep-0617} for more details.

(Contributed by Guido van Rossum, Pablo Galindo and Lysandros Nikolaou
in bpo-12782@footnote{https://bugs.python.org/issue12782} and bpo-40334@footnote{https://bugs.python.org/issue40334}.)

@node Better error messages,PEP 626 Precise line numbers for debugging and other tools,Parenthesized context managers,New Features<3>
@anchor{whatsnew/3 10 better-error-messages}@anchor{218}
@subsubsection Better error messages


@menu
* SyntaxErrors:: 
* IndentationErrors:: 
* AttributeErrors:: 
* NameErrors:: 

@end menu

@node SyntaxErrors,IndentationErrors,,Better error messages
@anchor{whatsnew/3 10 syntaxerrors}@anchor{219}
@subsubsection SyntaxErrors


When parsing code that contains unclosed parentheses or brackets the interpreter
now includes the location of the unclosed bracket of parentheses instead of displaying
@emph{SyntaxError: unexpected EOF while parsing} or pointing to some incorrect location.
For instance, consider the following code (notice the unclosed ‘@{‘):

@example
expected = @{9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4,
            38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6,
some_other_code = foo()
@end example

Previous versions of the interpreter reported confusing places as the location of
the syntax error:

@example
File "example.py", line 3
    some_other_code = foo()
                    ^
SyntaxError: invalid syntax
@end example

but in Python 3.10 a more informative error is emitted:

@example
File "example.py", line 1
    expected = @{9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4,
               ^
SyntaxError: '@{' was never closed
@end example

In a similar way, errors involving unclosed string literals (single and triple
quoted) now point to the start of the string instead of reporting EOF/EOL.

These improvements are inspired by previous work in the PyPy interpreter.

(Contributed by Pablo Galindo in bpo-42864@footnote{https://bugs.python.org/issue42864} and Batuhan Taskaya in
bpo-40176@footnote{https://bugs.python.org/issue40176}.)

@ref{21a,,SyntaxError} exceptions raised by the interpreter will now highlight the
full error range of the expression that constitutes the syntax error itself,
instead of just where the problem is detected. In this way, instead of displaying
(before Python 3.10):

@example
>>> foo(x, z for z in range(10), t, w)
  File "<stdin>", line 1
    foo(x, z for z in range(10), t, w)
           ^
SyntaxError: Generator expression must be parenthesized
@end example

now Python 3.10 will display the exception as:

@example
>>> foo(x, z for z in range(10), t, w)
  File "<stdin>", line 1
    foo(x, z for z in range(10), t, w)
           ^^^^^^^^^^^^^^^^^^^^
SyntaxError: Generator expression must be parenthesized
@end example

This improvement was contributed by Pablo Galindo in bpo-43914@footnote{https://bugs.python.org/issue43914}.

A considerable amount of new specialized messages for @ref{21a,,SyntaxError} exceptions
have been incorporated. Some of the most notable ones are as follows:


@itemize *

@item 
Missing @code{:} before blocks:

@quotation

@example
>>> if rocket.position > event_horizon
  File "<stdin>", line 1
    if rocket.position > event_horizon
                                      ^
SyntaxError: expected ':'
@end example

(Contributed by Pablo Galindo in bpo-42997@footnote{https://bugs.python.org/issue42997})
@end quotation

@item 
Unparenthesised tuples in comprehensions targets:

@quotation

@example
>>> @{x,y for x,y in zip('abcd', '1234')@}
  File "<stdin>", line 1
    @{x,y for x,y in zip('abcd', '1234')@}
     ^
SyntaxError: did you forget parentheses around the comprehension target?
@end example

(Contributed by Pablo Galindo in bpo-43017@footnote{https://bugs.python.org/issue43017})
@end quotation

@item 
Missing commas in collection literals and between expressions:

@quotation

@example
>>> items = @{
... x: 1,
... y: 2
... z: 3,
  File "<stdin>", line 3
    y: 2
       ^
SyntaxError: invalid syntax. Perhaps you forgot a comma?
@end example

(Contributed by Pablo Galindo in bpo-43822@footnote{https://bugs.python.org/issue43822})
@end quotation

@item 
Multiple Exception types without parentheses:

@quotation

@example
>>> try:
...     build_dyson_sphere()
... except NotEnoughScienceError, NotEnoughResourcesError:
  File "<stdin>", line 3
    except NotEnoughScienceError, NotEnoughResourcesError:
           ^
SyntaxError: multiple exception types must be parenthesized
@end example

(Contributed by Pablo Galindo in bpo-43149@footnote{https://bugs.python.org/issue43149})
@end quotation

@item 
Missing @code{:} and values in dictionary literals:

@quotation

@example
>>> values = @{
... x: 1,
... y: 2,
... z:
... @}
  File "<stdin>", line 4
    z:
     ^
SyntaxError: expression expected after dictionary key and ':'

>>> values = @{x:1, y:2, z w:3@}
  File "<stdin>", line 1
    values = @{x:1, y:2, z w:3@}
                        ^
SyntaxError: ':' expected after dictionary key
@end example

(Contributed by Pablo Galindo in bpo-43823@footnote{https://bugs.python.org/issue43823})
@end quotation

@item 
@code{try} blocks without @code{except} or @code{finally} blocks:

@quotation

@example
>>> try:
...     x = 2
... something = 3
  File "<stdin>", line 3
    something  = 3
    ^^^^^^^^^
SyntaxError: expected 'except' or 'finally' block
@end example

(Contributed by Pablo Galindo in bpo-44305@footnote{https://bugs.python.org/issue44305})
@end quotation

@item 
Usage of @code{=} instead of @code{==} in comparisons:

@quotation

@example
>>> if rocket.position = event_horizon:
  File "<stdin>", line 1
    if rocket.position = event_horizon:
                       ^
SyntaxError: cannot assign to attribute here. Maybe you meant '==' instead of '='?
@end example

(Contributed by Pablo Galindo in bpo-43797@footnote{https://bugs.python.org/issue43797})
@end quotation

@item 
Usage of @code{*} in f-strings:

@quotation

@example
>>> f"Black holes @{*all_black_holes@} and revelations"
  File "<stdin>", line 1
    (*all_black_holes)
     ^
SyntaxError: f-string: cannot use starred expression here
@end example

(Contributed by Pablo Galindo in bpo-41064@footnote{https://bugs.python.org/issue41064})
@end quotation
@end itemize

@node IndentationErrors,AttributeErrors,SyntaxErrors,Better error messages
@anchor{whatsnew/3 10 indentationerrors}@anchor{21b}
@subsubsection IndentationErrors


Many @ref{21c,,IndentationError} exceptions now have more context regarding what kind of block
was expecting an indentation, including the location of the statement:

@example
>>> def foo():
...    if lel:
...    x = 2
  File "<stdin>", line 3
    x = 2
    ^
IndentationError: expected an indented block after 'if' statement in line 2
@end example

@node AttributeErrors,NameErrors,IndentationErrors,Better error messages
@anchor{whatsnew/3 10 attributeerrors}@anchor{21d}
@subsubsection AttributeErrors


When printing @ref{163,,AttributeError}, @code{PyErr_Display()} will offer
suggestions of similar attribute names in the object that the exception was
raised from:

@example
>>> collections.namedtoplo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'collections' has no attribute 'namedtoplo'. Did you mean: namedtuple?
@end example

(Contributed by Pablo Galindo in bpo-38530@footnote{https://bugs.python.org/issue38530}.)

@quotation

@cartouche
@quotation Warning 
Notice this won’t work if @code{PyErr_Display()} is not called to display the error
which can happen if some other custom error display function is used. This is a common
scenario in some REPLs like IPython.
@end quotation
@end cartouche
@end quotation

@node NameErrors,,AttributeErrors,Better error messages
@anchor{whatsnew/3 10 nameerrors}@anchor{21e}
@subsubsection NameErrors


When printing @ref{21f,,NameError} raised by the interpreter, @code{PyErr_Display()}
will offer suggestions of similar variable names in the function that the exception
was raised from:

@example
>>> schwarzschild_black_hole = None
>>> schwarschild_black_hole
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'schwarschild_black_hole' is not defined. Did you mean: schwarzschild_black_hole?
@end example

(Contributed by Pablo Galindo in bpo-38530@footnote{https://bugs.python.org/issue38530}.)

@quotation

@cartouche
@quotation Warning 
Notice this won’t work if @code{PyErr_Display()} is not called to display the error,
which can happen if some other custom error display function is used. This is a common
scenario in some REPLs like IPython.
@end quotation
@end cartouche
@end quotation

@node PEP 626 Precise line numbers for debugging and other tools,PEP 634 Structural Pattern Matching,Better error messages,New Features<3>
@anchor{whatsnew/3 10 pep-626-precise-line-numbers-for-debugging-and-other-tools}@anchor{220}
@subsubsection PEP 626: Precise line numbers for debugging and other tools


PEP 626 brings more precise and reliable line numbers for debugging, profiling and coverage tools.
Tracing events, with the correct line number, are generated for all lines of code executed and only for lines of code that are executed.

The @code{f_lineno} attribute of frame objects will always contain the expected line number.

The @code{co_lnotab} attribute of code objects is deprecated and will be removed in 3.12.
Code that needs to convert from offset to line number should use the new @code{co_lines()} method instead.

@node PEP 634 Structural Pattern Matching,Optional EncodingWarning and encoding="locale" option,PEP 626 Precise line numbers for debugging and other tools,New Features<3>
@anchor{whatsnew/3 10 pep-634-structural-pattern-matching}@anchor{221}
@subsubsection PEP 634: Structural Pattern Matching


Structural pattern matching has been added in the form of a @emph{match statement}
and @emph{case statements} of patterns with associated actions. Patterns
consist of sequences, mappings, primitive data types as well as class instances.
Pattern matching enables programs to extract information from complex data types,
branch on the structure of data, and apply specific actions based on different
forms of data.

@menu
* Syntax and operations:: 
* Declarative approach:: 
* Simple pattern; match to a literal: Simple pattern match to a literal. 
* Patterns with a literal and variable:: 
* Patterns and classes:: 
* Nested patterns:: 
* Complex patterns and the wildcard:: 
* Guard:: 
* Other Key Features:: 

@end menu

@node Syntax and operations,Declarative approach,,PEP 634 Structural Pattern Matching
@anchor{whatsnew/3 10 syntax-and-operations}@anchor{222}
@subsubsection Syntax and operations


The generic syntax of pattern matching is:

@example
match subject:
    case <pattern_1>:
        <action_1>
    case <pattern_2>:
        <action_2>
    case <pattern_3>:
        <action_3>
    case _:
        <action_wildcard>
@end example

A match statement takes an expression and compares its value to successive
patterns given as one or more case blocks.  Specifically, pattern matching
operates by:

@quotation


@enumerate 

@item 
using data with type and shape (the @code{subject})

@item 
evaluating the @code{subject} in the @code{match} statement

@item 
comparing the subject with each pattern in a @code{case} statement
from top to bottom until a match is confirmed.

@item 
executing the action associated with the pattern of the confirmed
match

@item 
If an exact match is not confirmed, the last case, a wildcard @code{_},
if provided, will be used as the matching case. If an exact match is
not confirmed and a wildcard case does not exist, the entire match
block is a no-op.
@end enumerate
@end quotation

@node Declarative approach,Simple pattern match to a literal,Syntax and operations,PEP 634 Structural Pattern Matching
@anchor{whatsnew/3 10 declarative-approach}@anchor{223}
@subsubsection Declarative approach


Readers may be aware of pattern matching through the simple example of matching
a subject (data object) to a literal (pattern) with the switch statement found
in C, Java or JavaScript (and many other languages). Often the switch statement
is used for comparison of an object/expression with case statements containing
literals.

More powerful examples of pattern matching can be found in languages such as
Scala and Elixir. With structural pattern matching, the approach is “declarative” and
explicitly states the conditions (the patterns) for data to match.

While an “imperative” series of instructions using nested “if” statements
could be used to accomplish something similar to structural pattern matching,
it is less clear than the “declarative” approach. Instead the “declarative”
approach states the conditions to meet for a match and is more readable through
its explicit patterns. While structural pattern matching can be used in its
simplest form comparing a variable to a literal in a case statement, its
true value for Python lies in its handling of the subject’s type and shape.

@node Simple pattern match to a literal,Patterns with a literal and variable,Declarative approach,PEP 634 Structural Pattern Matching
@anchor{whatsnew/3 10 simple-pattern-match-to-a-literal}@anchor{224}
@subsubsection Simple pattern: match to a literal


Let’s look at this example as pattern matching in its simplest form: a value,
the subject, being matched to several literals, the patterns. In the example
below, @code{status} is the subject of the match statement. The patterns are
each of the case statements, where literals represent request status codes.
The associated action to the case is executed after a match:

@example
def http_error(status):
    match status:
        case 400:
            return "Bad request"
        case 404:
            return "Not found"
        case 418:
            return "I'm a teapot"
        case _:
            return "Something's wrong with the internet"
@end example

If the above function is passed a @code{status} of 418, “I’m a teapot” is returned.
If the above function is passed a @code{status} of 500, the case statement with
@code{_} will match as a wildcard, and “Something’s wrong with the internet” is
returned.
Note the last block: the variable name, @code{_}, acts as a @emph{wildcard} and insures
the subject will always match. The use of @code{_} is optional.

You can combine several literals in a single pattern using @code{|} (“or”):

@example
case 401 | 403 | 404:
    return "Not allowed"
@end example

@menu
* Behavior without the wildcard:: 

@end menu

@node Behavior without the wildcard,,,Simple pattern match to a literal
@anchor{whatsnew/3 10 behavior-without-the-wildcard}@anchor{225}
@subsubsection Behavior without the wildcard


If we modify the above example by removing the last case block, the example
becomes:

@example
def http_error(status):
    match status:
        case 400:
            return "Bad request"
        case 404:
            return "Not found"
        case 418:
            return "I'm a teapot"
@end example

Without the use of @code{_} in a case statement, a match may not exist. If no
match exists, the behavior is a no-op. For example, if @code{status} of 500 is
passed, a no-op occurs.

@node Patterns with a literal and variable,Patterns and classes,Simple pattern match to a literal,PEP 634 Structural Pattern Matching
@anchor{whatsnew/3 10 patterns-with-a-literal-and-variable}@anchor{226}
@subsubsection Patterns with a literal and variable


Patterns can look like unpacking assignments, and a pattern may be used to bind
variables. In this example, a data point can be unpacked to its x-coordinate
and y-coordinate:

@example
# point is an (x, y) tuple
match point:
    case (0, 0):
        print("Origin")
    case (0, y):
        print(f"Y=@{y@}")
    case (x, 0):
        print(f"X=@{x@}")
    case (x, y):
        print(f"X=@{x@}, Y=@{y@}")
    case _:
        raise ValueError("Not a point")
@end example

The first pattern has two literals, @code{(0, 0)}, and may be thought of as an
extension of the literal pattern shown above. The next two patterns combine a
literal and a variable, and the variable @emph{binds} a value from the subject
(@code{point}).  The fourth pattern captures two values, which makes it
conceptually similar to the unpacking assignment @code{(x, y) = point}.

@node Patterns and classes,Nested patterns,Patterns with a literal and variable,PEP 634 Structural Pattern Matching
@anchor{whatsnew/3 10 patterns-and-classes}@anchor{227}
@subsubsection Patterns and classes


If you are using classes to structure your data, you can use as a pattern
the class name followed by an argument list resembling a constructor. This
pattern has the ability to capture class attributes into variables:

@example
class Point:
    x: int
    y: int

def location(point):
    match point:
        case Point(x=0, y=0):
            print("Origin is the point's location.")
        case Point(x=0, y=y):
            print(f"Y=@{y@} and the point is on the y-axis.")
        case Point(x=x, y=0):
            print(f"X=@{x@} and the point is on the x-axis.")
        case Point():
            print("The point is located somewhere else on the plane.")
        case _:
            print("Not a point")
@end example

@menu
* Patterns with positional parameters:: 

@end menu

@node Patterns with positional parameters,,,Patterns and classes
@anchor{whatsnew/3 10 patterns-with-positional-parameters}@anchor{228}
@subsubsection Patterns with positional parameters


You can use positional parameters with some builtin classes that provide an
ordering for their attributes (e.g. dataclasses). You can also define a specific
position for attributes in patterns by setting the @code{__match_args__} special
attribute in your classes. If it’s set to (“x”, “y”), the following patterns
are all equivalent (and all bind the @code{y} attribute to the @code{var} variable):

@example
Point(1, var)
Point(1, y=var)
Point(x=1, y=var)
Point(y=var, x=1)
@end example

@node Nested patterns,Complex patterns and the wildcard,Patterns and classes,PEP 634 Structural Pattern Matching
@anchor{whatsnew/3 10 nested-patterns}@anchor{229}
@subsubsection Nested patterns


Patterns can be arbitrarily nested.  For example, if our data is a short
list of points, it could be matched like this:

@example
match points:
    case []:
        print("No points in the list.")
    case [Point(0, 0)]:
        print("The origin is the only point in the list.")
    case [Point(x, y)]:
        print(f"A single point @{x@}, @{y@} is in the list.")
    case [Point(0, y1), Point(0, y2)]:
        print(f"Two points on the Y axis at @{y1@}, @{y2@} are in the list.")
    case _:
        print("Something else is found in the list.")
@end example

@node Complex patterns and the wildcard,Guard,Nested patterns,PEP 634 Structural Pattern Matching
@anchor{whatsnew/3 10 complex-patterns-and-the-wildcard}@anchor{22a}
@subsubsection Complex patterns and the wildcard


To this point, the examples have used @code{_} alone in the last case statement.
A wildcard can be used in more complex patterns, such as @code{('error', code, _)}.
For example:

@example
match test_variable:
    case ('warning', code, 40):
        print("A warning has been received.")
    case ('error', code, _):
        print(f"An error @{code@} occurred.")
@end example

In the above case, @code{test_variable} will match for (‘error’, code, 100) and
(‘error’, code, 800).

@node Guard,Other Key Features,Complex patterns and the wildcard,PEP 634 Structural Pattern Matching
@anchor{whatsnew/3 10 guard}@anchor{22b}
@subsubsection Guard


We can add an @code{if} clause to a pattern, known as a “guard”.  If the
guard is false, @code{match} goes on to try the next case block.  Note
that value capture happens before the guard is evaluated:

@example
match point:
    case Point(x, y) if x == y:
        print(f"The point is located on the diagonal Y=X at @{x@}.")
    case Point(x, y):
        print(f"Point is not on the diagonal.")
@end example

@node Other Key Features,,Guard,PEP 634 Structural Pattern Matching
@anchor{whatsnew/3 10 other-key-features}@anchor{22c}
@subsubsection Other Key Features


Several other key features:


@itemize -

@item 
Like unpacking assignments, tuple and list patterns have exactly the
same meaning and actually match arbitrary sequences. Technically,
the subject must be a sequence.
Therefore, an important exception is that patterns don’t match iterators.
Also, to prevent a common mistake, sequence patterns don’t match strings.

@item 
Sequence patterns support wildcards: @code{[x, y, *rest]} and @code{(x, y,
*rest)} work similar to wildcards in unpacking assignments.  The
name after @code{*} may also be @code{_}, so @code{(x, y, *_)} matches a sequence
of at least two items without binding the remaining items.

@item 
Mapping patterns: @code{@{"bandwidth": b, "latency": l@}} captures the
@code{"bandwidth"} and @code{"latency"} values from a dict.  Unlike sequence
patterns, extra keys are ignored.  A wildcard @code{**rest} is also
supported.  (But @code{**_} would be redundant, so is not allowed.)

@item 
Subpatterns may be captured using the @code{as} keyword:

@example
case (Point(x1, y1), Point(x2, y2) as p2): ...
@end example

This binds x1, y1, x2, y2 like you would expect without the @code{as} clause,
and p2 to the entire second item of the subject.

@item 
Most literals are compared by equality. However, the singletons @code{True},
@code{False} and @code{None} are compared by identity.

@item 
Named constants may be used in patterns.  These named constants must be
dotted names to prevent the constant from being interpreted as a capture
variable:

@example
from enum import Enum
class Color(Enum):
    RED = 0
    GREEN = 1
    BLUE = 2

match color:
    case Color.RED:
        print("I see red!")
    case Color.GREEN:
        print("Grass is green")
    case Color.BLUE:
        print("I'm feeling the blues :(")
@end example
@end itemize

For the full specification see 
@geindex Python Enhancement Proposals; PEP 634
PEP 634@footnote{https://www.python.org/dev/peps/pep-0634}.  Motivation and rationale
are in 
@geindex Python Enhancement Proposals; PEP 635
PEP 635@footnote{https://www.python.org/dev/peps/pep-0635}, and a longer tutorial is in 
@geindex Python Enhancement Proposals; PEP 636
PEP 636@footnote{https://www.python.org/dev/peps/pep-0636}.

@node Optional EncodingWarning and encoding="locale" option,,PEP 634 Structural Pattern Matching,New Features<3>
@anchor{whatsnew/3 10 optional-encodingwarning-and-encoding-locale-option}@anchor{22d}@anchor{whatsnew/3 10 whatsnew310-pep597}@anchor{22e}
@subsubsection Optional @code{EncodingWarning} and @code{encoding="locale"} option


The default encoding of @code{TextIOWrapper} and @ref{1c8,,open()} is
platform and locale dependent. Since UTF-8 is used on most Unix
platforms, omitting @code{encoding} option when opening UTF-8 files
(e.g. JSON, YAML, TOML, Markdown) is a very common bug. For example:

@example
# BUG: "rb" mode or encoding="utf-8" should be used.
with open("data.json") as f:
    data = json.load(f)
@end example

To find this type of bug, an optional @code{EncodingWarning} is added.
It is emitted when @ref{22f,,sys.flags.warn_default_encoding}
is true and locale-specific default encoding is used.

@code{-X warn_default_encoding} option and 
@geindex PYTHONWARNDEFAULTENCODING
@geindex environment variable; PYTHONWARNDEFAULTENCODING
@ref{230,,PYTHONWARNDEFAULTENCODING}
are added to enable the warning.

See @ref{231,,Text Encoding} for more information.

@node New Features Related to Type Hints,Other Language Changes<2>,New Features<3>,What’s New In Python 3 10
@anchor{whatsnew/3 10 new-feat-related-type-hints}@anchor{232}@anchor{whatsnew/3 10 new-features-related-to-type-hints}@anchor{233}
@subsection New Features Related to Type Hints


This section covers major changes affecting 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} type hints and
the @ref{120,,typing} module.

@menu
* PEP 604; New Type Union Operator: PEP 604 New Type Union Operator. 
* PEP 612; Parameter Specification Variables: PEP 612 Parameter Specification Variables. 
* PEP 613; TypeAlias: PEP 613 TypeAlias. 
* PEP 647; User-Defined Type Guards: PEP 647 User-Defined Type Guards. 

@end menu

@node PEP 604 New Type Union Operator,PEP 612 Parameter Specification Variables,,New Features Related to Type Hints
@anchor{whatsnew/3 10 pep-604-new-type-union-operator}@anchor{234}
@subsubsection PEP 604: New Type Union Operator


A new type union operator was introduced which enables the syntax @code{X | Y}.
This provides a cleaner way of expressing ‘either type X or type Y’ instead of
using @ref{235,,typing.Union}, especially in type hints.

In previous versions of Python, to apply a type hint for functions accepting
arguments of multiple types, @ref{235,,typing.Union} was used:

@example
def square(number: Union[int, float]) -> Union[int, float]:
    return number ** 2
@end example

Type hints can now be written in a more succinct manner:

@example
def square(number: int | float) -> int | float:
    return number ** 2
@end example

This new syntax is also accepted as the second argument to @ref{236,,isinstance()}
and @ref{237,,issubclass()}:

@example
>>> isinstance(1, int | str)
True
@end example

See @ref{238,,Union Type} and 
@geindex Python Enhancement Proposals; PEP 604
PEP 604@footnote{https://www.python.org/dev/peps/pep-0604} for more details.

(Contributed by Maggie Moss and Philippe Prados in bpo-41428@footnote{https://bugs.python.org/issue41428},
with additions by Yurii Karabas and Serhiy Storchaka in bpo-44490@footnote{https://bugs.python.org/issue44490}.)

@node PEP 612 Parameter Specification Variables,PEP 613 TypeAlias,PEP 604 New Type Union Operator,New Features Related to Type Hints
@anchor{whatsnew/3 10 pep-612-parameter-specification-variables}@anchor{239}
@subsubsection PEP 612: Parameter Specification Variables


Two new options to improve the information provided to static type checkers for
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484}‘s @code{Callable} have been added to the @ref{120,,typing} module.

The first is the parameter specification variable.  They are used to forward the
parameter types of one callable to another callable – a pattern commonly
found in higher order functions and decorators.  Examples of usage can be found
in @ref{23a,,typing.ParamSpec}. Previously, there was no easy way to type annotate
dependency of parameter types in such a precise manner.

The second option is the new @code{Concatenate} operator. It’s used in conjunction
with parameter specification variables to type annotate a higher order callable
which adds or removes parameters of another callable.  Examples of usage can
be found in @ref{23b,,typing.Concatenate}.

See @ref{23c,,typing.Callable}, @ref{23a,,typing.ParamSpec},
@ref{23b,,typing.Concatenate}, @ref{23d,,typing.ParamSpecArgs},
@ref{23e,,typing.ParamSpecKwargs}, and 
@geindex Python Enhancement Proposals; PEP 612
PEP 612@footnote{https://www.python.org/dev/peps/pep-0612} for more details.

(Contributed by Ken Jin in bpo-41559@footnote{https://bugs.python.org/issue41559}, with minor enhancements by Jelle
Zijlstra in bpo-43783@footnote{https://bugs.python.org/issue43783}.  PEP written by Mark Mendoza.)

@node PEP 613 TypeAlias,PEP 647 User-Defined Type Guards,PEP 612 Parameter Specification Variables,New Features Related to Type Hints
@anchor{whatsnew/3 10 pep-613-typealias}@anchor{23f}
@subsubsection PEP 613: TypeAlias


@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} introduced the concept of type aliases, only requiring them to be
top-level unannotated assignments. This simplicity sometimes made it difficult
for type checkers to distinguish between type aliases and ordinary assignments,
especially when forward references or invalid types were involved. Compare:

@example
StrCache = 'Cache[str]'  # a type alias
LOG_PREFIX = 'LOG[DEBUG]'  # a module constant
@end example

Now the @ref{120,,typing} module has a special value @code{TypeAlias}
which lets you declare type aliases more explicitly:

@example
StrCache: TypeAlias = 'Cache[str]'  # a type alias
LOG_PREFIX = 'LOG[DEBUG]'  # a module constant
@end example

See 
@geindex Python Enhancement Proposals; PEP 613
PEP 613@footnote{https://www.python.org/dev/peps/pep-0613} for more details.

(Contributed by Mikhail Golubev in bpo-41923@footnote{https://bugs.python.org/issue41923}.)

@node PEP 647 User-Defined Type Guards,,PEP 613 TypeAlias,New Features Related to Type Hints
@anchor{whatsnew/3 10 pep-647-user-defined-type-guards}@anchor{240}
@subsubsection PEP 647: User-Defined Type Guards


@code{TypeGuard} has been added to the @ref{120,,typing} module to annotate
type guard functions and improve information provided to static type checkers
during type narrowing.  For more information, please see @code{TypeGuard}‘s
documentation, and 
@geindex Python Enhancement Proposals; PEP 647
PEP 647@footnote{https://www.python.org/dev/peps/pep-0647}.

(Contributed by Ken Jin and Guido van Rossum in bpo-43766@footnote{https://bugs.python.org/issue43766}.
PEP written by Eric Traut.)

@node Other Language Changes<2>,New Modules<2>,New Features Related to Type Hints,What’s New In Python 3 10
@anchor{whatsnew/3 10 other-language-changes}@anchor{241}
@subsection Other Language Changes



@itemize *

@item 
The @ref{1b1,,int} type has a new method @ref{242,,int.bit_count()}, returning the
number of ones in the binary expansion of a given integer, also known
as the population count. (Contributed by Niklas Fiekas in bpo-29882@footnote{https://bugs.python.org/issue29882}.)

@item 
The views returned by @ref{243,,dict.keys()}, @ref{244,,dict.values()} and
@ref{245,,dict.items()} now all have a @code{mapping} attribute that gives a
@ref{246,,types.MappingProxyType} object wrapping the original
dictionary. (Contributed by Dennis Sweeney in bpo-40890@footnote{https://bugs.python.org/issue40890}.)

@item 
@geindex Python Enhancement Proposals; PEP 618
PEP 618@footnote{https://www.python.org/dev/peps/pep-0618}: The @ref{247,,zip()} function now has an optional @code{strict} flag, used
to require that all the iterables have an equal length.

@item 
Builtin and extension functions that take integer arguments no longer accept
@ref{248,,Decimal}s, @ref{172,,Fraction}s and other
objects that can be converted to integers only with a loss (e.g. that have
the @ref{249,,__int__()} method but do not have the
@ref{24a,,__index__()} method).
(Contributed by Serhiy Storchaka in bpo-37999@footnote{https://bugs.python.org/issue37999}.)

@item 
If @ref{24b,,object.__ipow__()} returns @ref{24c,,NotImplemented}, the operator will
correctly fall back to @ref{24d,,object.__pow__()} and @ref{24e,,object.__rpow__()} as expected.
(Contributed by Alex Shkop in bpo-38302@footnote{https://bugs.python.org/issue38302}.)

@item 
Assignment expressions can now be used unparenthesized within set literals
and set comprehensions, as well as in sequence indexes (but not slices).

@item 
Functions have a new @code{__builtins__} attribute which is used to look for
builtin symbols when a function is executed, instead of looking into
@code{__globals__['__builtins__']}. The attribute is initialized from
@code{__globals__["__builtins__"]} if it exists, else from the current builtins.
(Contributed by Mark Shannon in bpo-42990@footnote{https://bugs.python.org/issue42990}.)

@item 
Two new builtin functions – @ref{24f,,aiter()} and @ref{250,,anext()} have been added
to provide asynchronous counterparts to @ref{251,,iter()} and @ref{252,,next()},
respectively.
(Contributed by Joshua Bronson, Daniel Pope, and Justin Wang in bpo-31861@footnote{https://bugs.python.org/issue31861}.)

@item 
Static methods (@ref{253,,@@staticmethod}) and class methods
(@ref{254,,@@classmethod}) now inherit the method attributes
(@code{__module__}, @code{__name__}, @code{__qualname__}, @code{__doc__},
@code{__annotations__}) and have a new @code{__wrapped__} attribute.
Moreover, static methods are now callable as regular functions.
(Contributed by Victor Stinner in bpo-43682@footnote{https://bugs.python.org/issue43682}.)

@item 
Annotations for complex targets (everything beside @code{simple name} targets
defined by 
@geindex Python Enhancement Proposals; PEP 526
PEP 526@footnote{https://www.python.org/dev/peps/pep-0526}) no longer cause any runtime effects with @code{from __future__ import annotations}.
(Contributed by Batuhan Taskaya in bpo-42737@footnote{https://bugs.python.org/issue42737}.)

@item 
Class and module objects now lazy-create empty annotations dicts on demand.
The annotations dicts are stored in the object’s @code{__dict__} for
backwards compatibility.  This improves the best practices for working
with @code{__annotations__}; for more information, please see
@ref{255,,Annotations Best Practices}.
(Contributed by Larry Hastings in bpo-43901@footnote{https://bugs.python.org/issue43901}.)

@item 
Annotations consist of @code{yield}, @code{yield from}, @code{await} or named expressions
are now forbidden under @code{from __future__ import annotations} due to their side
effects.
(Contributed by Batuhan Taskaya in bpo-42725@footnote{https://bugs.python.org/issue42725}.)

@item 
Usage of unbound variables, @code{super()} and other expressions that might
alter the processing of symbol table as annotations are now rendered
effectless under @code{from __future__ import annotations}.
(Contributed by Batuhan Taskaya in bpo-42725@footnote{https://bugs.python.org/issue42725}.)

@item 
Hashes of NaN values of both @ref{256,,float} type and
@ref{248,,decimal.Decimal} type now depend on object identity. Formerly, they
always hashed to @code{0} even though NaN values are not equal to one another.
This caused potentially quadratic runtime behavior due to excessive hash
collisions when creating dictionaries and sets containing multiple NaNs.
(Contributed by Raymond Hettinger in bpo-43475@footnote{https://bugs.python.org/issue43475}.)

@item 
A @ref{21a,,SyntaxError} (instead of a @ref{21f,,NameError}) will be raised when deleting
the @ref{257,,__debug__} constant.  (Contributed by Dong-hee Na in bpo-45000@footnote{https://bugs.python.org/issue45000}.)

@item 
@ref{21a,,SyntaxError} exceptions now have @code{end_lineno} and
@code{end_offset} attributes.  They will be @code{None} if not determined.
(Contributed by Pablo Galindo in bpo-43914@footnote{https://bugs.python.org/issue43914}.)
@end itemize

@node New Modules<2>,Improved Modules<2>,Other Language Changes<2>,What’s New In Python 3 10
@anchor{whatsnew/3 10 new-modules}@anchor{258}
@subsection New Modules



@itemize *

@item 
None yet.
@end itemize

@node Improved Modules<2>,Optimizations<2>,New Modules<2>,What’s New In Python 3 10
@anchor{whatsnew/3 10 improved-modules}@anchor{259}
@subsection Improved Modules


@menu
* asyncio:: 
* argparse:: 
* array:: 
* asynchat@comma{} asyncore@comma{} smtpd: asynchat asyncore smtpd. 
* base64:: 
* bdb:: 
* bisect:: 
* codecs:: 
* collections.abc: collections abc. 
* contextlib:: 
* curses:: 
* dataclasses:: 
* distutils:: 
* doctest:: 
* encodings:: 
* enum:: 
* fileinput:: 
* faulthandler:: 
* gc:: 
* glob:: 
* hashlib:: 
* hmac:: 
* IDLE and idlelib: IDLE and idlelib<2>. 
* importlib.metadata: importlib metadata. 
* inspect: inspect<2>. 
* linecache:: 
* os: os<2>. 
* os.path: os path. 
* pathlib:: 
* platform:: 
* pprint:: 
* py_compile:: 
* pyclbr:: 
* shelve:: 
* statistics:: 
* site:: 
* socket: socket<2>. 
* ssl:: 
* sqlite3: sqlite3<2>. 
* sys: sys<2>. 
* _thread:: 
* threading: threading<2>. 
* traceback:: 
* types:: 
* typing:: 
* unittest:: 
* urllib.parse: urllib parse. 
* xml:: 
* zipimport:: 

@end menu

@node asyncio,argparse,,Improved Modules<2>
@anchor{whatsnew/3 10 asyncio}@anchor{25a}
@subsubsection asyncio


Add missing @code{connect_accepted_socket()}
method.
(Contributed by Alex Grönholm in bpo-41332@footnote{https://bugs.python.org/issue41332}.)

@node argparse,array,asyncio,Improved Modules<2>
@anchor{whatsnew/3 10 argparse}@anchor{25b}
@subsubsection argparse


Misleading phrase “optional arguments” was replaced with “options” in argparse help. Some tests might require adaptation if they rely on exact output match.
(Contributed by Raymond Hettinger in bpo-9694@footnote{https://bugs.python.org/issue9694}.)

@node array,asynchat asyncore smtpd,argparse,Improved Modules<2>
@anchor{whatsnew/3 10 array}@anchor{25c}
@subsubsection array


The @ref{25d,,index()} method of @ref{25e,,array.array} now has
optional @emph{start} and @emph{stop} parameters.
(Contributed by Anders Lorentsen and Zackery Spytz in bpo-31956@footnote{https://bugs.python.org/issue31956}.)

@node asynchat asyncore smtpd,base64,array,Improved Modules<2>
@anchor{whatsnew/3 10 asynchat-asyncore-smtpd}@anchor{25f}
@subsubsection asynchat, asyncore, smtpd


These modules have been marked as deprecated in their module documentation
since Python 3.6.  An import-time @ref{1b2,,DeprecationWarning} has now been
added to all three of these modules.

@node base64,bdb,asynchat asyncore smtpd,Improved Modules<2>
@anchor{whatsnew/3 10 base64}@anchor{260}
@subsubsection base64


Add @ref{261,,base64.b32hexencode()} and @ref{262,,base64.b32hexdecode()} to support the
Base32 Encoding with Extended Hex Alphabet.

@node bdb,bisect,base64,Improved Modules<2>
@anchor{whatsnew/3 10 bdb}@anchor{263}
@subsubsection bdb


Add @code{clearBreakpoints()} to reset all set breakpoints.
(Contributed by Irit Katriel in bpo-24160@footnote{https://bugs.python.org/issue24160}.)

@node bisect,codecs,bdb,Improved Modules<2>
@anchor{whatsnew/3 10 bisect}@anchor{264}
@subsubsection bisect


Added the possibility of providing a @emph{key} function to the APIs in the @ref{10,,bisect}
module. (Contributed by Raymond Hettinger in bpo-4356@footnote{https://bugs.python.org/issue4356}.)

@node codecs,collections abc,bisect,Improved Modules<2>
@anchor{whatsnew/3 10 codecs}@anchor{265}
@subsubsection codecs


Add a @ref{266,,codecs.unregister()} function to unregister a codec search function.
(Contributed by Hai Shi in bpo-41842@footnote{https://bugs.python.org/issue41842}.)

@node collections abc,contextlib,codecs,Improved Modules<2>
@anchor{whatsnew/3 10 collections-abc}@anchor{267}
@subsubsection collections.abc


The @code{__args__} of the @ref{268,,parameterized generic} for
@ref{269,,collections.abc.Callable} are now consistent with @ref{23c,,typing.Callable}.
@ref{269,,collections.abc.Callable} generic now flattens type parameters, similar
to what @ref{23c,,typing.Callable} currently does.  This means that
@code{collections.abc.Callable[[int, str], str]} will have @code{__args__} of
@code{(int, str, str)}; previously this was @code{([int, str], str)}.  To allow this
change, @ref{26a,,types.GenericAlias} can now be subclassed, and a subclass will
be returned when subscripting the @ref{269,,collections.abc.Callable} type.  Note
that a @ref{162,,TypeError} may be raised for invalid forms of parameterizing
@ref{269,,collections.abc.Callable} which may have passed silently in Python 3.9.
(Contributed by Ken Jin in bpo-42195@footnote{https://bugs.python.org/issue42195}.)

@node contextlib,curses,collections abc,Improved Modules<2>
@anchor{whatsnew/3 10 contextlib}@anchor{26b}
@subsubsection contextlib


Add a @ref{26c,,contextlib.aclosing()} context manager to safely close async generators
and objects representing asynchronously released resources.
(Contributed by Joongi Kim and John Belmonte in bpo-41229@footnote{https://bugs.python.org/issue41229}.)

Add asynchronous context manager support to @ref{26d,,contextlib.nullcontext()}.
(Contributed by Tom Gringauz in bpo-41543@footnote{https://bugs.python.org/issue41543}.)

Add @code{AsyncContextDecorator}, for supporting usage of async context managers
as decorators.

@node curses,dataclasses,contextlib,Improved Modules<2>
@anchor{whatsnew/3 10 curses}@anchor{26e}
@subsubsection curses


The extended color functions added in ncurses 6.1 will be used transparently
by @ref{26f,,curses.color_content()}, @ref{270,,curses.init_color()},
@ref{271,,curses.init_pair()}, and @ref{272,,curses.pair_content()}. A new function,
@ref{273,,curses.has_extended_color_support()}, indicates whether extended color
support is provided by the underlying ncurses library.
(Contributed by Jeffrey Kintscher and Hans Petter Jansson in bpo-36982@footnote{https://bugs.python.org/issue36982}.)

The @code{BUTTON5_*} constants are now exposed in the @ref{2a,,curses} module if
they are provided by the underlying curses library.
(Contributed by Zackery Spytz in bpo-39273@footnote{https://bugs.python.org/issue39273}.)

@node dataclasses,distutils,curses,Improved Modules<2>
@anchor{whatsnew/3 10 dataclasses}@anchor{274}
@subsubsection dataclasses


@menu
* __slots__:: 
* Keyword-only fields:: 

@end menu

@node __slots__,Keyword-only fields,,dataclasses
@anchor{whatsnew/3 10 slots}@anchor{275}
@subsubsection __slots__


Added @code{slots} parameter in @ref{276,,dataclasses.dataclass()} decorator.
(Contributed by Yurii Karabas in bpo-42269@footnote{https://bugs.python.org/issue42269})

@node Keyword-only fields,,__slots__,dataclasses
@anchor{whatsnew/3 10 keyword-only-fields}@anchor{277}
@subsubsection Keyword-only fields


dataclasses now supports fields that are keyword-only in the
generated __init__ method.  There are a number of ways of specifying
keyword-only fields.

You can say that every field is keyword-only:

@example
from dataclasses import dataclass

@@dataclass(kw_only=True)
class Birthday:
    name: str
    birthday: datetime.date
@end example

Both @code{name} and @code{birthday} are keyword-only parameters to the
generated __init__ method.

You can specify keyword-only on a per-field basis:

@example
from dataclasses import dataclass, field

@@dataclass
class Birthday:
    name: str
    birthday: datetime.date = field(kw_only=True)
@end example

Here only @code{birthday} is keyword-only.  If you set @code{kw_only} on
individual fields, be aware that there are rules about re-ordering
fields due to keyword-only fields needing to follow non-keyword-only
fields.  See the full dataclasses documentation for details.

You can also specify that all fields following a KW_ONLY marker are
keyword-only.  This will probably be the most common usage:

@example
from dataclasses import dataclass, KW_ONLY

@@dataclass
class Point:
    x: float
    y: float
    _: KW_ONLY
    z: float = 0.0
    t: float = 0.0
@end example

Here, @code{z} and @code{t} are keyword-only parameters, while @code{x} and
@code{y} are not.
(Contributed by Eric V. Smith in bpo-43532@footnote{https://bugs.python.org/issue43532})

@node distutils,doctest,dataclasses,Improved Modules<2>
@anchor{whatsnew/3 10 distutils}@anchor{278}@anchor{whatsnew/3 10 distutils-deprecated}@anchor{279}
@subsubsection distutils


The entire @code{distutils} package is deprecated, to be removed in Python
3.12. Its functionality for specifying package builds has already been
completely replaced by third-party packages @code{setuptools} and
@code{packaging}, and most other commonly used APIs are available elsewhere
in the standard library (such as @ref{c9,,platform}, @ref{e4,,shutil},
@ref{f4,,subprocess} or @ref{f8,,sysconfig}). There are no plans to migrate
any other functionality from @code{distutils}, and applications that are
using other functions should plan to make private copies of the code.
Refer to 
@geindex Python Enhancement Proposals; PEP 632
PEP 632@footnote{https://www.python.org/dev/peps/pep-0632} for discussion.

The @code{bdist_wininst} command deprecated in Python 3.8 has been removed.
The @code{bdist_wheel} command is now recommended to distribute binary packages
on Windows.
(Contributed by Victor Stinner in bpo-42802@footnote{https://bugs.python.org/issue42802}.)

@node doctest,encodings,distutils,Improved Modules<2>
@anchor{whatsnew/3 10 doctest}@anchor{27a}
@subsubsection doctest


When a module does not define @code{__loader__}, fall back to @code{__spec__.loader}.
(Contributed by Brett Cannon in bpo-42133@footnote{https://bugs.python.org/issue42133}.)

@node encodings,enum,doctest,Improved Modules<2>
@anchor{whatsnew/3 10 encodings}@anchor{27b}
@subsubsection encodings


@code{encodings.normalize_encoding()} now ignores non-ASCII characters.
(Contributed by Hai Shi in bpo-39337@footnote{https://bugs.python.org/issue39337}.)

@node enum,fileinput,encodings,Improved Modules<2>
@anchor{whatsnew/3 10 enum}@anchor{27c}
@subsubsection enum


@code{Enum} @code{__repr__()} now returns @code{enum_name.member_name} and
@code{__str__()} now returns @code{member_name}.  Stdlib enums available as
module constants have a @ref{27d,,repr()} of @code{module_name.member_name}.
(Contributed by Ethan Furman in bpo-40066@footnote{https://bugs.python.org/issue40066}.)

Add @ref{27e,,enum.StrEnum} for enums where all members are strings.
(Contributed by Ethan Furman in bpo-41816@footnote{https://bugs.python.org/issue41816}.)

@node fileinput,faulthandler,enum,Improved Modules<2>
@anchor{whatsnew/3 10 fileinput}@anchor{27f}
@subsubsection fileinput


Add @emph{encoding} and @emph{errors} parameters in @ref{280,,fileinput.input()} and
@ref{1bc,,fileinput.FileInput}.
(Contributed by Inada Naoki in bpo-43712@footnote{https://bugs.python.org/issue43712}.)

@ref{281,,fileinput.hook_compressed()} now returns @code{TextIOWrapper} object
when @emph{mode} is “r” and file is compressed, like uncompressed files.
(Contributed by Inada Naoki in bpo-5758@footnote{https://bugs.python.org/issue5758}.)

@node faulthandler,gc,fileinput,Improved Modules<2>
@anchor{whatsnew/3 10 faulthandler}@anchor{282}
@subsubsection faulthandler


The @ref{78,,faulthandler} module now detects if a fatal error occurs during a
garbage collector collection.
(Contributed by Victor Stinner in bpo-44466@footnote{https://bugs.python.org/issue44466}.)

@node gc,glob,faulthandler,Improved Modules<2>
@anchor{whatsnew/3 10 gc}@anchor{283}
@subsubsection gc


Add audit hooks for @ref{284,,gc.get_objects()}, @ref{285,,gc.get_referrers()} and
@ref{286,,gc.get_referents()}. (Contributed by Pablo Galindo in bpo-43439@footnote{https://bugs.python.org/issue43439}.)

@node glob,hashlib,gc,Improved Modules<2>
@anchor{whatsnew/3 10 glob}@anchor{287}
@subsubsection glob


Add the @emph{root_dir} and @emph{dir_fd} parameters in @ref{288,,glob()} and
@ref{289,,iglob()} which allow to specify the root directory for searching.
(Contributed by Serhiy Storchaka in bpo-38144@footnote{https://bugs.python.org/issue38144}.)

@node hashlib,hmac,glob,Improved Modules<2>
@anchor{whatsnew/3 10 hashlib}@anchor{28a}
@subsubsection hashlib


The hashlib module requires OpenSSL 1.1.1 or newer.
(Contributed by Christian Heimes in 
@geindex Python Enhancement Proposals; PEP 644
PEP 644@footnote{https://www.python.org/dev/peps/pep-0644} and bpo-43669@footnote{https://bugs.python.org/issue43669}.)

The hashlib module has preliminary support for OpenSSL 3.0.0.
(Contributed by Christian Heimes in bpo-38820@footnote{https://bugs.python.org/issue38820} and other issues.)

The pure-Python fallback of @ref{28b,,pbkdf2_hmac()} is deprecated. In
the future PBKDF2-HMAC will only be available when Python has been built with
OpenSSL support.
(Contributed by Christian Heimes in bpo-43880@footnote{https://bugs.python.org/issue43880}.)

@node hmac,IDLE and idlelib<2>,hashlib,Improved Modules<2>
@anchor{whatsnew/3 10 hmac}@anchor{28c}
@subsubsection hmac


The hmac module now uses OpenSSL’s HMAC implementation internally.
(Contributed by Christian Heimes in bpo-40645@footnote{https://bugs.python.org/issue40645}.)

@node IDLE and idlelib<2>,importlib metadata,hmac,Improved Modules<2>
@anchor{whatsnew/3 10 idle-and-idlelib}@anchor{28d}
@subsubsection IDLE and idlelib


Make IDLE invoke @ref{28e,,sys.excepthook()} (when started without ‘-n’).
User hooks were previously ignored.  (Contributed by Ken Hilton in
bpo-43008@footnote{https://bugs.python.org/issue43008}.)

Rearrange the settings dialog.  Split the General tab into Windows
and Shell/Ed tabs.  Move help sources, which extend the Help menu, to the
Extensions tab.  Make space for new options and shorten the dialog. The
latter makes the dialog better fit small screens.  (Contributed by Terry Jan
Reedy in bpo-40468@footnote{https://bugs.python.org/issue40468}.)  Move the indent space setting from the Font tab to
the new Windows tab.  (Contributed by Mark Roseman and Terry Jan Reedy in
bpo-33962@footnote{https://bugs.python.org/issue33962}.)

The changes above were backported to a 3.9 maintenance release.

Add a Shell sidebar.  Move the primary prompt (‘>>>’) to the sidebar.
Add secondary prompts (’…’) to the sidebar.  Left click and optional
drag selects one or more lines of text, as with the editor
line number sidebar.  Right click after selecting text lines displays
a context menu with ‘copy with prompts’.  This zips together prompts
from the sidebar with lines from the selected text.  This option also
appears on the context menu for the text.  (Contributed by Tal Einat
in bpo-37903@footnote{https://bugs.python.org/issue37903}.)

Use spaces instead of tabs to indent interactive code.  This makes
interactive code entries ‘look right’.  Making this feasible was a
major motivation for adding the shell sidebar.  (Contributed by
Terry Jan Reedy in bpo-37892@footnote{https://bugs.python.org/issue37892}.)

Highlight the new @ref{28f,,soft keywords} @ref{290,,match},
@ref{290,,case}, and @ref{291,,_} in
pattern-matching statements. However, this highlighting is not perfect
and will be incorrect in some rare cases, including some @code{_}-s in
@code{case} patterns.  (Contributed by Tal Einat in bpo-44010@footnote{https://bugs.python.org/issue44010}.)

New in 3.10 maintenance releases.

Apply syntax highlighting to @cite{.pyi} files. (Contributed by Alex
Waygood and Terry Jan Reedy in bpo-45447@footnote{https://bugs.python.org/issue45447}.)

@node importlib metadata,inspect<2>,IDLE and idlelib<2>,Improved Modules<2>
@anchor{whatsnew/3 10 importlib-metadata}@anchor{292}
@subsubsection importlib.metadata


Feature parity with @code{importlib_metadata} 4.6
(history@footnote{https://importlib-metadata.readthedocs.io/en/latest/history.html}).

@ref{293,,importlib.metadata entry points}
now provide a nicer experience
for selecting entry points by group and name through a new
@code{importlib.metadata.EntryPoints} class. See the Compatibility
Note in the docs for more info on the deprecation and usage.

Added @code{importlib.metadata.packages_distributions()} for resolving
top-level Python modules and packages to their
@code{importlib.metadata.Distribution}.

@node inspect<2>,linecache,importlib metadata,Improved Modules<2>
@anchor{whatsnew/3 10 inspect}@anchor{294}
@subsubsection inspect


When a module does not define @code{__loader__}, fall back to @code{__spec__.loader}.
(Contributed by Brett Cannon in bpo-42133@footnote{https://bugs.python.org/issue42133}.)

Add @ref{295,,inspect.get_annotations()}, which safely computes the annotations
defined on an object. It works around the quirks of accessing the annotations
on various types of objects, and makes very few assumptions about the object
it examines. @ref{295,,inspect.get_annotations()} can also correctly un-stringize
stringized annotations.  @ref{295,,inspect.get_annotations()} is now considered
best practice for accessing the annotations dict defined on any Python object;
for more information on best practices for working with annotations, please see
@ref{255,,Annotations Best Practices}.
Relatedly, @ref{1c1,,inspect.signature()},
@ref{1c3,,inspect.Signature.from_callable()}, and @code{inspect.Signature.from_function()}
now call @ref{295,,inspect.get_annotations()} to retrieve annotations. This means
@ref{1c1,,inspect.signature()} and @ref{1c3,,inspect.Signature.from_callable()} can
also now un-stringize stringized annotations.
(Contributed by Larry Hastings in bpo-43817@footnote{https://bugs.python.org/issue43817}.)

@node linecache,os<2>,inspect<2>,Improved Modules<2>
@anchor{whatsnew/3 10 linecache}@anchor{296}
@subsubsection linecache


When a module does not define @code{__loader__}, fall back to @code{__spec__.loader}.
(Contributed by Brett Cannon in bpo-42133@footnote{https://bugs.python.org/issue42133}.)

@node os<2>,os path,linecache,Improved Modules<2>
@anchor{whatsnew/3 10 os}@anchor{297}
@subsubsection os


Add @ref{298,,os.cpu_count()} support for VxWorks RTOS.
(Contributed by Peixing Xin in bpo-41440@footnote{https://bugs.python.org/issue41440}.)

Add a new function @ref{299,,os.eventfd()} and related helpers to wrap the
@code{eventfd2} syscall on Linux.
(Contributed by Christian Heimes in bpo-41001@footnote{https://bugs.python.org/issue41001}.)

Add @ref{29a,,os.splice()} that allows to move data between two file
descriptors without copying between kernel address space and user
address space, where one of the file descriptors must refer to a
pipe. (Contributed by Pablo Galindo in bpo-41625@footnote{https://bugs.python.org/issue41625}.)

Add @ref{29b,,O_EVTONLY}, @ref{29c,,O_FSYNC}, @ref{29d,,O_SYMLINK}
and @ref{29e,,O_NOFOLLOW_ANY} for macOS.
(Contributed by Dong-hee Na in bpo-43106@footnote{https://bugs.python.org/issue43106}.)

@node os path,pathlib,os<2>,Improved Modules<2>
@anchor{whatsnew/3 10 os-path}@anchor{29f}
@subsubsection os.path


@ref{2a0,,os.path.realpath()} now accepts a @emph{strict} keyword-only argument. When set
to @code{True}, @ref{2a1,,OSError} is raised if a path doesn’t exist or a symlink loop
is encountered.
(Contributed by Barney Gale in bpo-43757@footnote{https://bugs.python.org/issue43757}.)

@node pathlib,platform,os path,Improved Modules<2>
@anchor{whatsnew/3 10 pathlib}@anchor{2a2}
@subsubsection pathlib


Add slice support to @ref{2a3,,PurePath.parents}.
(Contributed by Joshua Cannon in bpo-35498@footnote{https://bugs.python.org/issue35498})

Add negative indexing support to @ref{2a3,,PurePath.parents}.
(Contributed by Yaroslav Pankovych in bpo-21041@footnote{https://bugs.python.org/issue21041})

Add @ref{2a4,,Path.hardlink_to} method that
supersedes @ref{2a5,,link_to()}. The new method has the same argument
order as @ref{2a6,,symlink_to()}.
(Contributed by Barney Gale in bpo-39950@footnote{https://bugs.python.org/issue39950}.)

@ref{2a7,,pathlib.Path.stat()} and @ref{2a8,,chmod()} now accept a
@emph{follow_symlinks} keyword-only argument for consistency with corresponding
functions in the @ref{c0,,os} module.
(Contributed by Barney Gale in bpo-39906@footnote{https://bugs.python.org/issue39906}.)

@node platform,pprint,pathlib,Improved Modules<2>
@anchor{whatsnew/3 10 platform}@anchor{2a9}
@subsubsection platform


Add @ref{2aa,,platform.freedesktop_os_release()} to retrieve operation system
identification from freedesktop.org os-release@footnote{https://www.freedesktop.org/software/systemd/man/os-release.html} standard file.
(Contributed by Christian Heimes in bpo-28468@footnote{https://bugs.python.org/issue28468})

@node pprint,py_compile,platform,Improved Modules<2>
@anchor{whatsnew/3 10 pprint}@anchor{2ab}
@subsubsection pprint


@ref{2ac,,pprint.pprint()} now accepts a new @code{underscore_numbers} keyword argument.
(Contributed by sblondon in bpo-42914@footnote{https://bugs.python.org/issue42914}.)

@ref{cd,,pprint} can now pretty-print @ref{276,,dataclasses.dataclass} instances.
(Contributed by Lewis Gaul in bpo-43080@footnote{https://bugs.python.org/issue43080}.)

@node py_compile,pyclbr,pprint,Improved Modules<2>
@anchor{whatsnew/3 10 py-compile}@anchor{2ad}
@subsubsection py_compile


Add @code{--quiet} option to command-line interface of @ref{d2,,py_compile}.
(Contributed by Gregory Schevchenko in bpo-38731@footnote{https://bugs.python.org/issue38731}.)

@node pyclbr,shelve,py_compile,Improved Modules<2>
@anchor{whatsnew/3 10 pyclbr}@anchor{2ae}
@subsubsection pyclbr


Add an @code{end_lineno} attribute to the @code{Function} and @code{Class}
objects in the tree returned by @code{pyclbr.readline()} and
@code{pyclbr.readline_ex()}.  It matches the existing (start) @code{lineno}.
(Contributed by Aviral Srivastava in bpo-38307@footnote{https://bugs.python.org/issue38307}.)

@node shelve,statistics,pyclbr,Improved Modules<2>
@anchor{whatsnew/3 10 shelve}@anchor{2af}
@subsubsection shelve


The @ref{e2,,shelve} module now uses @ref{2b0,,pickle.DEFAULT_PROTOCOL} by default
instead of @ref{c5,,pickle} protocol @code{3} when creating shelves.
(Contributed by Zackery Spytz in bpo-34204@footnote{https://bugs.python.org/issue34204}.)

@node statistics,site,shelve,Improved Modules<2>
@anchor{whatsnew/3 10 statistics}@anchor{2b1}
@subsubsection statistics


Add @ref{2b2,,covariance()}, Pearson’s
@ref{2b3,,correlation()}, and simple
@ref{2b4,,linear_regression()} functions.
(Contributed by Tymoteusz Wołodźko in bpo-38490@footnote{https://bugs.python.org/issue38490}.)

@node site,socket<2>,statistics,Improved Modules<2>
@anchor{whatsnew/3 10 site}@anchor{2b5}
@subsubsection site


When a module does not define @code{__loader__}, fall back to @code{__spec__.loader}.
(Contributed by Brett Cannon in bpo-42133@footnote{https://bugs.python.org/issue42133}.)

@node socket<2>,ssl,site,Improved Modules<2>
@anchor{whatsnew/3 10 socket}@anchor{2b6}
@subsubsection socket


The exception @ref{2b7,,socket.timeout} is now an alias of @ref{2b8,,TimeoutError}.
(Contributed by Christian Heimes in bpo-42413@footnote{https://bugs.python.org/issue42413}.)

Add option to create MPTCP sockets with @code{IPPROTO_MPTCP}
(Contributed by Rui Cunha in bpo-43571@footnote{https://bugs.python.org/issue43571}.)

Add @code{IP_RECVTOS} option to receive the type of service (ToS) or DSCP/ECN fields
(Contributed by Georg Sauthoff in bpo-44077@footnote{https://bugs.python.org/issue44077}.)

@node ssl,sqlite3<2>,socket<2>,Improved Modules<2>
@anchor{whatsnew/3 10 ssl}@anchor{2b9}
@subsubsection ssl


The ssl module requires OpenSSL 1.1.1 or newer.
(Contributed by Christian Heimes in 
@geindex Python Enhancement Proposals; PEP 644
PEP 644@footnote{https://www.python.org/dev/peps/pep-0644} and bpo-43669@footnote{https://bugs.python.org/issue43669}.)

The ssl module has preliminary support for OpenSSL 3.0.0 and new option
@ref{2ba,,OP_IGNORE_UNEXPECTED_EOF}.
(Contributed by Christian Heimes in bpo-38820@footnote{https://bugs.python.org/issue38820}, bpo-43794@footnote{https://bugs.python.org/issue43794},
bpo-43788@footnote{https://bugs.python.org/issue43788}, bpo-43791@footnote{https://bugs.python.org/issue43791}, bpo-43799@footnote{https://bugs.python.org/issue43799}, bpo-43920@footnote{https://bugs.python.org/issue43920},
bpo-43789@footnote{https://bugs.python.org/issue43789}, and bpo-43811@footnote{https://bugs.python.org/issue43811}.)

Deprecated function and use of deprecated constants now result in
a @ref{1b2,,DeprecationWarning}. @ref{2bb,,ssl.SSLContext.options} has
@ref{2bc,,OP_NO_SSLv2} and @ref{2bd,,OP_NO_SSLv3} set by default and
therefore cannot warn about setting the flag again. The
@ref{2be,,deprecation section} has a list of deprecated
features.
(Contributed by Christian Heimes in bpo-43880@footnote{https://bugs.python.org/issue43880}.)

The ssl module now has more secure default settings. Ciphers without forward
secrecy or SHA-1 MAC are disabled by default. Security level 2 prohibits
weak RSA, DH, and ECC keys with less than 112 bits of security.
@ref{2bf,,SSLContext} defaults to minimum protocol version TLS 1.2.
Settings are based on Hynek Schlawack’s research.
(Contributed by Christian Heimes in bpo-43998@footnote{https://bugs.python.org/issue43998}.)

The deprecated protocols SSL 3.0, TLS 1.0, and TLS 1.1 are no longer
officially supported. Python does not block them actively. However
OpenSSL build options, distro configurations, vendor patches, and cipher
suites may prevent a successful handshake.

Add a @emph{timeout} parameter to the @ref{2c0,,ssl.get_server_certificate()} function.
(Contributed by Zackery Spytz in bpo-31870@footnote{https://bugs.python.org/issue31870}.)

The ssl module uses heap-types and multi-phase initialization.
(Contributed by Christian Heimes in bpo-42333@footnote{https://bugs.python.org/issue42333}.)

A new verify flag @ref{2c1,,VERIFY_X509_PARTIAL_CHAIN} has been added.
(Contributed by l0x in bpo-40849@footnote{https://bugs.python.org/issue40849}.)

@node sqlite3<2>,sys<2>,ssl,Improved Modules<2>
@anchor{whatsnew/3 10 sqlite3}@anchor{2c2}
@subsubsection sqlite3


Add audit events for @code{connect/handle()},
@ref{2c3,,enable_load_extension()}, and
@ref{2c4,,load_extension()}.
(Contributed by Erlend E. Aasland in bpo-43762@footnote{https://bugs.python.org/issue43762}.)

@node sys<2>,_thread,sqlite3<2>,Improved Modules<2>
@anchor{whatsnew/3 10 sys}@anchor{2c5}
@subsubsection sys


Add @ref{2c6,,sys.orig_argv} attribute: the list of the original command line
arguments passed to the Python executable.
(Contributed by Victor Stinner in bpo-23427@footnote{https://bugs.python.org/issue23427}.)

Add @ref{2c7,,sys.stdlib_module_names}, containing the list of the standard library
module names.
(Contributed by Victor Stinner in bpo-42955@footnote{https://bugs.python.org/issue42955}.)

@node _thread,threading<2>,sys<2>,Improved Modules<2>
@anchor{whatsnew/3 10 thread}@anchor{2c8}
@subsubsection _thread


@ref{2c9,,_thread.interrupt_main()} now takes an optional signal number to
simulate (the default is still @ref{2ca,,signal.SIGINT}).
(Contributed by Antoine Pitrou in bpo-43356@footnote{https://bugs.python.org/issue43356}.)

@node threading<2>,traceback,_thread,Improved Modules<2>
@anchor{whatsnew/3 10 threading}@anchor{2cb}
@subsubsection threading


Add @ref{2cc,,threading.gettrace()} and @ref{2cd,,threading.getprofile()} to
retrieve the functions set by @ref{2ce,,threading.settrace()} and
@ref{2cf,,threading.setprofile()} respectively.
(Contributed by Mario Corchero in bpo-42251@footnote{https://bugs.python.org/issue42251}.)

Add @ref{2d0,,threading.__excepthook__} to allow retrieving the original value
of @ref{2d1,,threading.excepthook()} in case it is set to a broken or a different
value.
(Contributed by Mario Corchero in bpo-42308@footnote{https://bugs.python.org/issue42308}.)

@node traceback,types,threading<2>,Improved Modules<2>
@anchor{whatsnew/3 10 traceback}@anchor{2d2}
@subsubsection traceback


The @ref{2d3,,format_exception()},
@ref{2d4,,format_exception_only()}, and
@ref{2d5,,print_exception()} functions can now take an exception object
as a positional-only argument.
(Contributed by Zackery Spytz and Matthias Bussonnier in bpo-26389@footnote{https://bugs.python.org/issue26389}.)

@node types,typing,traceback,Improved Modules<2>
@anchor{whatsnew/3 10 types}@anchor{2d6}
@subsubsection types


Reintroduce the @ref{2d7,,types.EllipsisType}, @ref{2d8,,types.NoneType}
and @ref{2d9,,types.NotImplementedType} classes, providing a new set
of types readily interpretable by type checkers.
(Contributed by Bas van Beek in bpo-41810@footnote{https://bugs.python.org/issue41810}.)

@node typing,unittest,types,Improved Modules<2>
@anchor{whatsnew/3 10 typing}@anchor{2da}
@subsubsection typing


For major changes, see @ref{232,,New Features Related to Type Hints}.

The behavior of @ref{2db,,typing.Literal} was changed to conform with 
@geindex Python Enhancement Proposals; PEP 586
PEP 586@footnote{https://www.python.org/dev/peps/pep-0586}
and to match the behavior of static type checkers specified in the PEP.


@enumerate 

@item 
@code{Literal} now de-duplicates parameters.

@item 
Equality comparisons between @code{Literal} objects are now order independent.

@item 
@code{Literal} comparisons now respect types.  For example,
@code{Literal[0] == Literal[False]} previously evaluated to @code{True}.  It is
now @code{False}.  To support this change, the internally used type cache now
supports differentiating types.

@item 
@code{Literal} objects will now raise a @ref{162,,TypeError} exception during
equality comparisons if any of their parameters are not @ref{2dc,,hashable}.
Note that declaring @code{Literal} with unhashable parameters will not throw
an error:

@example
>>> from typing import Literal
>>> Literal[@{0@}]
>>> Literal[@{0@}] == Literal[@{False@}]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'set'
@end example
@end enumerate

(Contributed by Yurii Karabas in bpo-42345@footnote{https://bugs.python.org/issue42345}.)

Add new function @ref{2dd,,typing.is_typeddict()} to introspect if an annotation
is a @ref{2de,,typing.TypedDict}.
(Contributed by Patrick Reader in bpo-41792@footnote{https://bugs.python.org/issue41792})

Subclasses of @code{typing.Protocol} which only have data variables declared
will now raise a @code{TypeError} when checked with @code{isinstance} unless they
are decorated with @code{runtime_checkable()}.  Previously, these checks
passed silently.  Users should decorate their
subclasses with the @code{runtime_checkable()} decorator
if they want runtime protocols.
(Contributed by Yurii Karabas in bpo-38908@footnote{https://bugs.python.org/issue38908})

Importing from the @code{typing.io} and @code{typing.re} submodules will now emit
@ref{1b2,,DeprecationWarning}.  These submodules have been deprecated since
Python 3.8 and will be removed in a future version of Python.  Anything
belonging to those submodules should be imported directly from
@ref{120,,typing} instead.
(Contributed by Sebastian Rittau in bpo-38291@footnote{https://bugs.python.org/issue38291})

@node unittest,urllib parse,typing,Improved Modules<2>
@anchor{whatsnew/3 10 unittest}@anchor{2df}
@subsubsection unittest


Add new method @ref{2e0,,assertNoLogs()} to complement the
existing @ref{2e1,,assertLogs()}. (Contributed by Kit Yan Choi
in bpo-39385@footnote{https://bugs.python.org/issue39385}.)

@node urllib parse,xml,unittest,Improved Modules<2>
@anchor{whatsnew/3 10 urllib-parse}@anchor{2e2}
@subsubsection urllib.parse


Python versions earlier than Python 3.10 allowed using both @code{;} and @code{&} as
query parameter separators in @ref{2e3,,urllib.parse.parse_qs()} and
@ref{2e4,,urllib.parse.parse_qsl()}.  Due to security concerns, and to conform with
newer W3C recommendations, this has been changed to allow only a single
separator key, with @code{&} as the default.  This change also affects
@ref{2e5,,cgi.parse()} and @ref{2e6,,cgi.parse_multipart()} as they use the affected
functions internally.  For more details, please see their respective
documentation.
(Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in bpo-42967@footnote{https://bugs.python.org/issue42967}.)

@node xml,zipimport,urllib parse,Improved Modules<2>
@anchor{whatsnew/3 10 xml}@anchor{2e7}
@subsubsection xml


Add a @ref{2e8,,LexicalHandler} class to the
@ref{143,,xml.sax.handler} module.
(Contributed by Jonathan Gossage and Zackery Spytz in bpo-35018@footnote{https://bugs.python.org/issue35018}.)

@node zipimport,,xml,Improved Modules<2>
@anchor{whatsnew/3 10 zipimport}@anchor{2e9}
@subsubsection zipimport


Add methods related to 
@geindex Python Enhancement Proposals; PEP 451
PEP 451@footnote{https://www.python.org/dev/peps/pep-0451}: @ref{2ea,,find_spec()},
@ref{2eb,,zipimport.zipimporter.create_module()}, and
@ref{2ec,,zipimport.zipimporter.exec_module()}.
(Contributed by Brett Cannon in bpo-42131@footnote{https://bugs.python.org/issue42131}.)

Add @ref{2ed,,invalidate_caches()} method.
(Contributed by Desmond Cheong in bpo-14678@footnote{https://bugs.python.org/issue14678}.)

@node Optimizations<2>,Deprecated<3>,Improved Modules<2>,What’s New In Python 3 10
@anchor{whatsnew/3 10 optimizations}@anchor{2ee}
@subsection Optimizations



@itemize *

@item 
Constructors @ref{2ef,,str()}, @ref{2f0,,bytes()} and @ref{2f1,,bytearray()} are now faster
(around 30–40% for small objects).
(Contributed by Serhiy Storchaka in bpo-41334@footnote{https://bugs.python.org/issue41334}.)

@item 
The @ref{dd,,runpy} module now imports fewer modules.
The @code{python3 -m module-name} command startup time is 1.4x faster in
average. On Linux, @code{python3 -I -m module-name} imports 69 modules on Python
3.9, whereas it only imports 51 modules (-18) on Python 3.10.
(Contributed by Victor Stinner in bpo-41006@footnote{https://bugs.python.org/issue41006} and bpo-41718@footnote{https://bugs.python.org/issue41718}.)

@item 
The @code{LOAD_ATTR} instruction now uses new “per opcode cache” mechanism.  It
is about 36% faster now for regular attributes and 44% faster for slots.
(Contributed by Pablo Galindo and Yury Selivanov in bpo-42093@footnote{https://bugs.python.org/issue42093} and Guido
van Rossum in bpo-42927@footnote{https://bugs.python.org/issue42927}, based on ideas implemented originally in PyPy
and MicroPython.)

@item 
When building Python with @ref{2f2,,--enable-optimizations} now
@code{-fno-semantic-interposition} is added to both the compile and link line.
This speeds builds of the Python interpreter created with @ref{2f3,,--enable-shared}
with @code{gcc} by up to 30%. See this article@footnote{https://developers.redhat.com/blog/2020/06/25/red-hat-enterprise-linux-8-2-brings-faster-python-3-8-run-speeds/}
for more details. (Contributed by Victor Stinner and Pablo Galindo in
bpo-38980@footnote{https://bugs.python.org/issue38980}.)

@item 
Use a new output buffer management code for @ref{12,,bz2} / @ref{a9,,lzma} /
@ref{14b,,zlib} modules, and add @code{.readall()} function to
@code{_compression.DecompressReader} class. bz2 decompression is now 1.09x ~ 1.17x
faster, lzma decompression 1.20x ~ 1.32x faster, @code{GzipFile.read(-1)} 1.11x
~ 1.18x faster. (Contributed by Ma Lin, reviewed by Gregory P. Smith, in bpo-41486@footnote{https://bugs.python.org/issue41486})

@item 
When using stringized annotations, annotations dicts for functions are no longer
created when the function is created.  Instead, they are stored as a tuple of
strings, and the function object lazily converts this into the annotations dict
on demand.  This optimization cuts the CPU time needed to define an annotated
function by half.
(Contributed by Yurii Karabas and Inada Naoki in bpo-42202@footnote{https://bugs.python.org/issue42202})

@item 
Substring search functions such as @code{str1 in str2} and @code{str2.find(str1)}
now sometimes use Crochemore & Perrin’s “Two-Way” string searching
algorithm to avoid quadratic behavior on long strings.  (Contributed
by Dennis Sweeney in bpo-41972@footnote{https://bugs.python.org/issue41972})

@item 
Add micro-optimizations to @code{_PyType_Lookup()} to improve type attribute cache lookup
performance in the common case of cache hits. This makes the interpreter 1.04 times faster
on average. (Contributed by Dino Viehland in bpo-43452@footnote{https://bugs.python.org/issue43452})

@item 
The following built-in functions now support the faster 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590} vectorcall calling convention:
@ref{2f4,,map()}, @ref{2f5,,filter()}, @ref{2f6,,reversed()}, @ref{2f7,,bool()} and @ref{256,,float()}.
(Contributed by Dong-hee Na and Jeroen Demeyer in bpo-43575@footnote{https://bugs.python.org/issue43575}, bpo-43287@footnote{https://bugs.python.org/issue43287}, bpo-41922@footnote{https://bugs.python.org/issue41922}, bpo-41873@footnote{https://bugs.python.org/issue41873} and bpo-41870@footnote{https://bugs.python.org/issue41870})

@item 
@code{BZ2File} performance is improved by removing internal @code{RLock}.
This makes @code{BZ2File} thread unsafe in the face of multiple simultaneous
readers or writers, just like its equivalent classes in @ref{87,,gzip} and
@ref{a9,,lzma} have always been.  (Contributed by Inada Naoki in bpo-43785@footnote{https://bugs.python.org/issue43785}).
@end itemize

@node Deprecated<3>,Removed<3>,Optimizations<2>,What’s New In Python 3 10
@anchor{whatsnew/3 10 deprecated}@anchor{2f8}@anchor{whatsnew/3 10 whatsnew310-deprecated}@anchor{2be}
@subsection Deprecated



@itemize *

@item 
Currently Python accepts numeric literals immediately followed by keywords,
for example @code{0in x}, @code{1or x}, @code{0if 1else 2}.  It allows confusing
and ambiguous expressions like @code{[0x1for x in y]} (which can be
interpreted as @code{[0x1 for x in y]} or @code{[0x1f or x in y]}).  Starting in
this release, a deprecation warning is raised if the numeric literal is
immediately followed by one of keywords @ref{2f9,,and}, @ref{2fa,,else},
@ref{161,,for}, @ref{2fb,,if}, @ref{2fc,,in}, @ref{2fd,,is} and @ref{2fe,,or}.
In future releases it will be changed to syntax warning, and finally to
syntax error.
(Contributed by Serhiy Storchaka in bpo-43833@footnote{https://bugs.python.org/issue43833}).

@item 
Starting in this release, there will be a concerted effort to begin
cleaning up old import semantics that were kept for Python 2.7
compatibility. Specifically,
@ref{2ff,,find_loader()}/@ref{300,,find_module()}
(superseded by @code{find_spec()}),
@ref{301,,load_module()}
(superseded by @ref{302,,exec_module()}),
@ref{303,,module_repr()} (which the import system
takes care of for you), the @code{__package__} attribute
(superseded by @code{__spec__.parent}), the @code{__loader__} attribute
(superseded by @code{__spec__.loader}), and the @code{__cached__} attribute
(superseded by @code{__spec__.cached}) will slowly be removed (as well
as other classes and methods in @ref{96,,importlib}).
@ref{304,,ImportWarning} and/or @ref{1b2,,DeprecationWarning} will be raised
as appropriate to help identify code which needs updating during
this transition.

@item 
The entire @code{distutils} namespace is deprecated, to be removed in
Python 3.12. Refer to the @ref{279,,module changes}
section for more information.

@item 
Non-integer arguments to @ref{305,,random.randrange()} are deprecated.
The @ref{17c,,ValueError} is deprecated in favor of a @ref{162,,TypeError}.
(Contributed by Serhiy Storchaka and Raymond Hettinger in bpo-37319@footnote{https://bugs.python.org/issue37319}.)

@item 
The various @code{load_module()} methods of @ref{96,,importlib} have been
documented as deprecated since Python 3.6, but will now also trigger
a @ref{1b2,,DeprecationWarning}. Use
@ref{302,,exec_module()} instead.
(Contributed by Brett Cannon in bpo-26131@footnote{https://bugs.python.org/issue26131}.)

@item 
@code{zimport.zipimporter.load_module()} has been deprecated in
preference for @ref{2ec,,exec_module()}.
(Contributed by Brett Cannon in bpo-26131@footnote{https://bugs.python.org/issue26131}.)

@item 
The use of @ref{301,,load_module()} by the import
system now triggers an @ref{304,,ImportWarning} as
@ref{302,,exec_module()} is preferred.
(Contributed by Brett Cannon in bpo-26131@footnote{https://bugs.python.org/issue26131}.)

@item 
The use of @ref{306,,importlib.abc.MetaPathFinder.find_module()} and
@ref{307,,importlib.abc.PathEntryFinder.find_module()} by the import system now
trigger an @ref{304,,ImportWarning} as
@ref{308,,importlib.abc.MetaPathFinder.find_spec()} and
@ref{309,,importlib.abc.PathEntryFinder.find_spec()}
are preferred, respectively. You can use
@ref{30a,,importlib.util.spec_from_loader()} to help in porting.
(Contributed by Brett Cannon in bpo-42134@footnote{https://bugs.python.org/issue42134}.)

@item 
The use of @ref{2ff,,importlib.abc.PathEntryFinder.find_loader()} by the import
system now triggers an @ref{304,,ImportWarning} as
@ref{309,,importlib.abc.PathEntryFinder.find_spec()} is preferred. You can use
@ref{30a,,importlib.util.spec_from_loader()} to help in porting.
(Contributed by Brett Cannon in bpo-43672@footnote{https://bugs.python.org/issue43672}.)

@item 
The various implementations of
@ref{306,,importlib.abc.MetaPathFinder.find_module()} (
@code{importlib.machinery.BuiltinImporter.find_module()},
@code{importlib.machinery.FrozenImporter.find_module()},
@code{importlib.machinery.WindowsRegistryFinder.find_module()},
@ref{30b,,importlib.machinery.PathFinder.find_module()},
@ref{306,,importlib.abc.MetaPathFinder.find_module()} ),
@ref{307,,importlib.abc.PathEntryFinder.find_module()} (
@code{importlib.machinery.FileFinder.find_module()} ), and
@ref{2ff,,importlib.abc.PathEntryFinder.find_loader()} (
@ref{30c,,importlib.machinery.FileFinder.find_loader()} )
now raise @ref{1b2,,DeprecationWarning} and are slated for removal in
Python 3.12 (previously they were documented as deprecated in Python 3.4).
(Contributed by Brett Cannon in bpo-42135@footnote{https://bugs.python.org/issue42135}.)

@item 
@ref{30d,,importlib.abc.Finder} is deprecated (including its sole method,
@ref{300,,find_module()}). Both
@ref{30e,,importlib.abc.MetaPathFinder} and @ref{30f,,importlib.abc.PathEntryFinder}
no longer inherit from the class. Users should inherit from one of these two
classes as appropriate instead.
(Contributed by Brett Cannon in bpo-42135@footnote{https://bugs.python.org/issue42135}.)

@item 
The deprecations of @ref{95,,imp}, @ref{310,,importlib.find_loader()},
@code{importlib.util.set_package_wrapper()},
@code{importlib.util.set_loader_wrapper()},
@ref{311,,importlib.util.module_for_loader()},
@ref{312,,pkgutil.ImpImporter}, and
@ref{313,,pkgutil.ImpLoader} have all been updated to list Python 3.12 as the
slated version of removal (they began raising @ref{1b2,,DeprecationWarning} in
previous versions of Python).
(Contributed by Brett Cannon in bpo-43720@footnote{https://bugs.python.org/issue43720}.)

@item 
The import system now uses the @code{__spec__} attribute on modules before
falling back on @ref{303,,module_repr()} for a module’s
@code{__repr__()} method. Removal of the use of @code{module_repr()} is scheduled
for Python 3.12.
(Contributed by Brett Cannon in bpo-42137@footnote{https://bugs.python.org/issue42137}.)

@item 
@ref{303,,importlib.abc.Loader.module_repr()},
@code{importlib.machinery.FrozenLoader.module_repr()}, and
@code{importlib.machinery.BuiltinLoader.module_repr()} are deprecated and
slated for removal in Python 3.12.
(Contributed by Brett Cannon in bpo-42136@footnote{https://bugs.python.org/issue42136}.)

@item 
@code{sqlite3.OptimizedUnicode} has been undocumented and obsolete since Python
3.3, when it was made an alias to @ref{2ef,,str}.  It is now deprecated,
scheduled for removal in Python 3.12.
(Contributed by Erlend E. Aasland in bpo-42264@footnote{https://bugs.python.org/issue42264}.)

@item 
@ref{314,,asyncio.get_event_loop()} now emits a deprecation warning if there is
no running event loop. In the future it will be an alias of
@ref{315,,get_running_loop()}.
@ref{9,,asyncio} functions which implicitly create @ref{316,,Future}
or @ref{317,,Task} objects now emit
a deprecation warning if there is no running event loop and no explicit
@emph{loop} argument is passed: @ref{318,,ensure_future()},
@ref{319,,wrap_future()}, @ref{31a,,gather()},
@ref{31b,,shield()}, @ref{31c,,as_completed()} and constructors of
@ref{316,,Future}, @ref{317,,Task},
@ref{31d,,StreamReader}, @code{StreamReaderProtocol}.
(Contributed by Serhiy Storchaka in bpo-39529@footnote{https://bugs.python.org/issue39529}.)

@item 
The undocumented built-in function @code{sqlite3.enable_shared_cache} is now
deprecated, scheduled for removal in Python 3.12.  Its use is strongly
discouraged by the SQLite3 documentation.  See the SQLite3 docs@footnote{https://sqlite.org/c3ref/enable_shared_cache.html} for more details.
If a shared cache must be used, open the database in URI mode using the
@code{cache=shared} query parameter.
(Contributed by Erlend E. Aasland in bpo-24464@footnote{https://bugs.python.org/issue24464}.)

@item 
The following @code{threading} methods are now deprecated:


@itemize *

@item 
@code{threading.currentThread} => @ref{31e,,threading.current_thread()}

@item 
@code{threading.activeCount} => @ref{31f,,threading.active_count()}

@item 
@code{threading.Condition.notifyAll} =>
@ref{320,,threading.Condition.notify_all()}

@item 
@code{threading.Event.isSet} => @ref{321,,threading.Event.is_set()}

@item 
@code{threading.Thread.setName} => @ref{322,,threading.Thread.name}

@item 
@code{threading.thread.getName} => @ref{322,,threading.Thread.name}

@item 
@code{threading.Thread.isDaemon} => @ref{323,,threading.Thread.daemon}

@item 
@code{threading.Thread.setDaemon} => @ref{323,,threading.Thread.daemon}
@end itemize

(Contributed by Jelle Zijlstra in bpo-21574@footnote{https://bugs.python.org/issue21574}.)

@item 
@ref{2a5,,pathlib.Path.link_to()} is deprecated and slated for removal in
Python 3.12. Use @ref{2a4,,pathlib.Path.hardlink_to()} instead.
(Contributed by Barney Gale in bpo-39950@footnote{https://bugs.python.org/issue39950}.)

@item 
@code{cgi.log()} is deprecated and slated for removal in Python 3.12.
(Contributed by Inada Naoki in bpo-41139@footnote{https://bugs.python.org/issue41139}.)

@item 
The following @ref{ee,,ssl} features have been deprecated since Python 3.6,
Python 3.7, or OpenSSL 1.1.0 and will be removed in 3.11:


@itemize *

@item 
@ref{2bc,,OP_NO_SSLv2}, @ref{2bd,,OP_NO_SSLv3}, @ref{324,,OP_NO_TLSv1},
@ref{325,,OP_NO_TLSv1_1}, @ref{326,,OP_NO_TLSv1_2}, and
@ref{327,,OP_NO_TLSv1_3} are replaced by
@code{sslSSLContext.minimum_version} and
@code{sslSSLContext.maximum_version}.

@item 
@ref{328,,PROTOCOL_SSLv2}, @ref{329,,PROTOCOL_SSLv3},
@ref{32a,,PROTOCOL_SSLv23}, @ref{32b,,PROTOCOL_TLSv1},
@ref{32c,,PROTOCOL_TLSv1_1}, @ref{32d,,PROTOCOL_TLSv1_2}, and
@ref{32e,,PROTOCOL_TLS} are deprecated in favor of
@ref{32f,,PROTOCOL_TLS_CLIENT} and @ref{330,,PROTOCOL_TLS_SERVER}

@item 
@ref{331,,wrap_socket()} is replaced by @ref{332,,ssl.SSLContext.wrap_socket()}

@item 
@ref{333,,match_hostname()}

@item 
@ref{334,,RAND_pseudo_bytes()}, @code{RAND_egd()}

@item 
NPN features like @ref{335,,ssl.SSLSocket.selected_npn_protocol()} and
@ref{336,,ssl.SSLContext.set_npn_protocols()} are replaced by ALPN.
@end itemize

@item 
The threading debug (
@geindex PYTHONTHREADDEBUG
@geindex environment variable; PYTHONTHREADDEBUG
@ref{337,,PYTHONTHREADDEBUG} environment variable) is
deprecated in Python 3.10 and will be removed in Python 3.12. This feature
requires a @ref{338,,debug build of Python}.
(Contributed by Victor Stinner in bpo-44584@footnote{https://bugs.python.org/issue44584}.)

@item 
Importing from the @code{typing.io} and @code{typing.re} submodules will now emit
@ref{1b2,,DeprecationWarning}.  These submodules will be removed in a future version
of Python.  Anything belonging to these submodules should be imported directly
from @ref{120,,typing} instead.
(Contributed by Sebastian Rittau in bpo-38291@footnote{https://bugs.python.org/issue38291})
@end itemize

@node Removed<3>,Porting to Python 3 10,Deprecated<3>,What’s New In Python 3 10
@anchor{whatsnew/3 10 removed}@anchor{339}@anchor{whatsnew/3 10 whatsnew310-removed}@anchor{33a}
@subsection Removed



@itemize *

@item 
Removed special methods @code{__int__}, @code{__float__}, @code{__floordiv__},
@code{__mod__}, @code{__divmod__}, @code{__rfloordiv__}, @code{__rmod__} and
@code{__rdivmod__} of the @ref{33b,,complex} class.  They always raised
a @ref{162,,TypeError}.
(Contributed by Serhiy Storchaka in bpo-41974@footnote{https://bugs.python.org/issue41974}.)

@item 
The @code{ParserBase.error()} method from the private and undocumented @code{_markupbase}
module has been removed.  @ref{33c,,html.parser.HTMLParser} is the only subclass of
@code{ParserBase} and its @code{error()} implementation was already removed in
Python 3.5.
(Contributed by Berker Peksag in bpo-31844@footnote{https://bugs.python.org/issue31844}.)

@item 
Removed the @code{unicodedata.ucnhash_CAPI} attribute which was an internal
PyCapsule object. The related private @code{_PyUnicode_Name_CAPI} structure was
moved to the internal C API.
(Contributed by Victor Stinner in bpo-42157@footnote{https://bugs.python.org/issue42157}.)

@item 
Removed the @code{parser} module, which was deprecated in 3.9 due to the
switch to the new PEG parser, as well as all the C source and header files
that were only being used by the old parser, including @code{node.h}, @code{parser.h},
@code{graminit.h} and @code{grammar.h}.

@item 
Removed the Public C API functions @code{PyParser_SimpleParseStringFlags},
@code{PyParser_SimpleParseStringFlagsFilename},
@code{PyParser_SimpleParseFileFlags} and @code{PyNode_Compile}
that were deprecated in 3.9 due to the switch to the new PEG parser.

@item 
Removed the @code{formatter} module, which was deprecated in Python 3.4.
It is somewhat obsolete, little used, and not tested. It was originally
scheduled to be removed in Python 3.6, but such removals were delayed until
after Python 2.7 EOL. Existing users should copy whatever classes they use
into their code.
(Contributed by Dong-hee Na and Terry J. Reedy in bpo-42299@footnote{https://bugs.python.org/issue42299}.)

@item 
Removed the @code{PyModule_GetWarningsModule()} function that was useless
now due to the _warnings module was converted to a builtin module in 2.6.
(Contributed by Hai Shi in bpo-42599@footnote{https://bugs.python.org/issue42599}.)

@item 
Remove deprecated aliases to @ref{33d,,Collections Abstract Base Classes} from
the @ref{1c,,collections} module.
(Contributed by Victor Stinner in bpo-37324@footnote{https://bugs.python.org/issue37324}.)

@item 
The @code{loop} parameter has been removed from most of @ref{9,,asyncio}‘s
@ref{33e,,high-level API} following deprecation
in Python 3.8.  The motivation behind this change is multifold:


@enumerate 

@item 
This simplifies the high-level API.

@item 
The functions in the high-level API have been implicitly getting the
current thread’s running event loop since Python 3.7.  There isn’t a need to
pass the event loop to the API in most normal use cases.

@item 
Event loop passing is error-prone especially when dealing with loops
running in different threads.
@end enumerate

Note that the low-level API will still accept @code{loop}.
See @ref{33f,,Changes in the Python API} for examples of how to replace existing code.

(Contributed by Yurii Karabas, Andrew Svetlov, Yury Selivanov and Kyle Stanley
in bpo-42392@footnote{https://bugs.python.org/issue42392}.)
@end itemize

@node Porting to Python 3 10,CPython bytecode changes<2>,Removed<3>,What’s New In Python 3 10
@anchor{whatsnew/3 10 porting-to-python-3-10}@anchor{340}
@subsection Porting to Python 3.10


This section lists previously described changes and other bugfixes
that may require changes to your code.

@menu
* Changes in the Python syntax:: 
* Changes in the Python API: Changes in the Python API<2>. 
* Changes in the C API:: 

@end menu

@node Changes in the Python syntax,Changes in the Python API<2>,,Porting to Python 3 10
@anchor{whatsnew/3 10 changes-in-the-python-syntax}@anchor{341}
@subsubsection Changes in the Python syntax



@itemize *

@item 
Deprecation warning is now emitted when compiling previously valid syntax
if the numeric literal is immediately followed by a keyword (like in @code{0in x}).
In future releases it will be changed to syntax warning, and finally to a
syntax error.  To get rid of the warning and make the code compatible with
future releases just add a space between the numeric literal and the
following keyword.
(Contributed by Serhiy Storchaka in bpo-43833@footnote{https://bugs.python.org/issue43833}).
@end itemize

@node Changes in the Python API<2>,Changes in the C API,Changes in the Python syntax,Porting to Python 3 10
@anchor{whatsnew/3 10 changes-in-the-python-api}@anchor{342}@anchor{whatsnew/3 10 changes-python-api}@anchor{33f}
@subsubsection Changes in the Python API



@itemize *

@item 
The @emph{etype} parameters of the @ref{2d3,,format_exception()},
@ref{2d4,,format_exception_only()}, and
@ref{2d5,,print_exception()} functions in the @ref{11a,,traceback} module
have been renamed to @emph{exc}.
(Contributed by Zackery Spytz and Matthias Bussonnier in bpo-26389@footnote{https://bugs.python.org/issue26389}.)

@item 
@ref{b,,atexit}: At Python exit, if a callback registered with
@ref{343,,atexit.register()} fails, its exception is now logged. Previously, only
some exceptions were logged, and the last exception was always silently
ignored.
(Contributed by Victor Stinner in bpo-42639@footnote{https://bugs.python.org/issue42639}.)

@item 
@ref{269,,collections.abc.Callable} generic now flattens type parameters, similar
to what @ref{23c,,typing.Callable} currently does.  This means that
@code{collections.abc.Callable[[int, str], str]} will have @code{__args__} of
@code{(int, str, str)}; previously this was @code{([int, str], str)}.  Code which
accesses the arguments via @ref{344,,typing.get_args()} or @code{__args__} need to account
for this change.  Furthermore, @ref{162,,TypeError} may be raised for invalid forms
of parameterizing @ref{269,,collections.abc.Callable} which may have passed
silently in Python 3.9.
(Contributed by Ken Jin in bpo-42195@footnote{https://bugs.python.org/issue42195}.)

@item 
@ref{345,,socket.htons()} and @ref{346,,socket.ntohs()} now raise @ref{347,,OverflowError}
instead of @ref{1b2,,DeprecationWarning} if the given parameter will not fit in
a 16-bit unsigned integer.
(Contributed by Erlend E. Aasland in bpo-42393@footnote{https://bugs.python.org/issue42393}.)

@item 
The @code{loop} parameter has been removed from most of @ref{9,,asyncio}‘s
@ref{33e,,high-level API} following deprecation
in Python 3.8.

A coroutine that currently looks like this:

@example
async def foo(loop):
    await asyncio.sleep(1, loop=loop)
@end example

Should be replaced with this:

@example
async def foo():
    await asyncio.sleep(1)
@end example

If @code{foo()} was specifically designed @emph{not} to run in the current thread’s
running event loop (e.g. running in another thread’s event loop), consider
using @ref{348,,asyncio.run_coroutine_threadsafe()} instead.

(Contributed by Yurii Karabas, Andrew Svetlov, Yury Selivanov and Kyle Stanley
in bpo-42392@footnote{https://bugs.python.org/issue42392}.)

@item 
The @ref{349,,types.FunctionType} constructor now inherits the current builtins
if the @emph{globals} dictionary has no @code{"__builtins__"} key, rather than using
@code{@{"None": None@}} as builtins: same behavior as @ref{34a,,eval()} and
@ref{34b,,exec()} functions.  Defining a function with @code{def function(...): ...}
in Python is not affected, globals cannot be overridden with this syntax: it
also inherits the current builtins.
(Contributed by Victor Stinner in bpo-42990@footnote{https://bugs.python.org/issue42990}.)
@end itemize

@node Changes in the C API,,Changes in the Python API<2>,Porting to Python 3 10
@anchor{whatsnew/3 10 changes-in-the-c-api}@anchor{34c}
@subsubsection Changes in the C API



@itemize *

@item 
The C API functions @code{PyParser_SimpleParseStringFlags},
@code{PyParser_SimpleParseStringFlagsFilename},
@code{PyParser_SimpleParseFileFlags}, @code{PyNode_Compile} and the type
used by these functions, @code{struct _node}, were removed due to the switch
to the new PEG parser.

Source should be now be compiled directly to a code object using, for
example, @ref{34d,,Py_CompileString()}. The resulting code object can then be
evaluated using, for example, @ref{34e,,PyEval_EvalCode()}.

Specifically:


@itemize *

@item 
A call to @code{PyParser_SimpleParseStringFlags} followed by
@code{PyNode_Compile} can be replaced by calling @ref{34d,,Py_CompileString()}.

@item 
There is no direct replacement for @code{PyParser_SimpleParseFileFlags}.
To compile code from a @code{FILE *} argument, you will need to read
the file in C and pass the resulting buffer to @ref{34d,,Py_CompileString()}.

@item 
To compile a file given a @code{char *} filename, explicitly open the file, read
it and compile the result. One way to do this is using the @ref{9d,,io}
module with @ref{34f,,PyImport_ImportModule()}, @ref{350,,PyObject_CallMethod()},
@ref{351,,PyBytes_AsString()} and @ref{34d,,Py_CompileString()},
as sketched below. (Declarations and error handling are omitted.)

@example
io_module = Import_ImportModule("io");
fileobject = PyObject_CallMethod(io_module, "open", "ss", filename, "rb");
source_bytes_object = PyObject_CallMethod(fileobject, "read", "");
result = PyObject_CallMethod(fileobject, "close", "");
source_buf = PyBytes_AsString(source_bytes_object);
code = Py_CompileString(source_buf, filename, Py_file_input);
@end example

@item 
For @code{FrameObject} objects, the @code{f_lasti} member now represents a wordcode
offset instead of a simple offset into the bytecode string. This means that this
number needs to be multiplied by 2 to be used with APIs that expect a byte offset
instead (like @ref{352,,PyCode_Addr2Line()} for example). Notice as well that the
@code{f_lasti} member of @code{FrameObject} objects is not considered stable: please
use @ref{1f9,,PyFrame_GetLineNumber()} instead.
@end itemize
@end itemize

@node CPython bytecode changes<2>,Build Changes<2>,Porting to Python 3 10,What’s New In Python 3 10
@anchor{whatsnew/3 10 cpython-bytecode-changes}@anchor{353}
@subsection CPython bytecode changes



@itemize *

@item 
The @code{MAKE_FUNCTION} instruction now accepts either a dict or a tuple of
strings as the function’s annotations.
(Contributed by Yurii Karabas and Inada Naoki in bpo-42202@footnote{https://bugs.python.org/issue42202})
@end itemize

@node Build Changes<2>,C API Changes<2>,CPython bytecode changes<2>,What’s New In Python 3 10
@anchor{whatsnew/3 10 build-changes}@anchor{354}
@subsection Build Changes



@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 644
PEP 644@footnote{https://www.python.org/dev/peps/pep-0644}: Python now requires OpenSSL 1.1.1 or newer. OpenSSL 1.0.2 is no
longer supported.
(Contributed by Christian Heimes in bpo-43669@footnote{https://bugs.python.org/issue43669}.)

@item 
The C99 functions @code{snprintf()} and @code{vsnprintf()} are now required
to build Python.
(Contributed by Victor Stinner in bpo-36020@footnote{https://bugs.python.org/issue36020}.)

@item 
@ref{ed,,sqlite3} requires SQLite 3.7.15 or higher. (Contributed by Sergey Fedoseev
and Erlend E. Aasland in bpo-40744@footnote{https://bugs.python.org/issue40744} and bpo-40810@footnote{https://bugs.python.org/issue40810}.)

@item 
The @ref{b,,atexit} module must now always be built as a built-in module.
(Contributed by Victor Stinner in bpo-42639@footnote{https://bugs.python.org/issue42639}.)

@item 
Add @ref{355,,--disable-test-modules} option to the @code{configure} script:
don’t build nor install test modules.
(Contributed by Xavier de Gaye, Thomas Petazzoni and Peixing Xin in bpo-27640@footnote{https://bugs.python.org/issue27640}.)

@item 
Add @ref{356,,--with-wheel-pkg-dir=PATH option}
to the @code{./configure} script. If
specified, the @ref{75,,ensurepip} module looks for @code{setuptools} and @code{pip}
wheel packages in this directory: if both are present, these wheel packages
are used instead of ensurepip bundled wheel packages.

Some Linux distribution packaging policies recommend against bundling
dependencies. For example, Fedora installs wheel packages in the
@code{/usr/share/python-wheels/} directory and don’t install the
@code{ensurepip._bundled} package.

(Contributed by Victor Stinner in bpo-42856@footnote{https://bugs.python.org/issue42856}.)

@item 
Add a new @ref{357,,configure --without-static-libpython option} to not build the @code{libpythonMAJOR.MINOR.a}
static library and not install the @code{python.o} object file.

(Contributed by Victor Stinner in bpo-43103@footnote{https://bugs.python.org/issue43103}.)

@item 
The @code{configure} script now uses the @code{pkg-config} utility, if available,
to detect the location of Tcl/Tk headers and libraries.  As before, those
locations can be explicitly specified with the @ref{358,,--with-tcltk-includes}
and @ref{359,,--with-tcltk-libs} configuration options.
(Contributed by Manolis Stamatogiannakis in bpo-42603@footnote{https://bugs.python.org/issue42603}.)

@item 
Add @ref{35a,,--with-openssl-rpath} option to @code{configure} script. The option
simplifies building Python with a custom OpenSSL installation, e.g.
@code{./configure --with-openssl=/path/to/openssl --with-openssl-rpath=auto}.
(Contributed by Christian Heimes in bpo-43466@footnote{https://bugs.python.org/issue43466}.)
@end itemize

@node C API Changes<2>,,Build Changes<2>,What’s New In Python 3 10
@anchor{whatsnew/3 10 c-api-changes}@anchor{35b}
@subsection C API Changes


@menu
* PEP 652; Maintaining the Stable ABI: PEP 652 Maintaining the Stable ABI. 
* New Features: New Features<4>. 
* Porting to Python 3.10: Porting to Python 3 10<2>. 
* Deprecated: Deprecated<4>. 
* Removed: Removed<4>. 

@end menu

@node PEP 652 Maintaining the Stable ABI,New Features<4>,,C API Changes<2>
@anchor{whatsnew/3 10 pep-652-maintaining-the-stable-abi}@anchor{35c}
@subsubsection PEP 652: Maintaining the Stable ABI


The Stable ABI (Application Binary Interface) for extension modules or
embedding Python is now explicitly defined.
@ref{35d,,C API Stability} describes C API and ABI stability guarantees along with best
practices for using the Stable ABI.

(Contributed by Petr Viktorin in 
@geindex Python Enhancement Proposals; PEP 652
PEP 652@footnote{https://www.python.org/dev/peps/pep-0652} and bpo-43795@footnote{https://bugs.python.org/issue43795}.)

@node New Features<4>,Porting to Python 3 10<2>,PEP 652 Maintaining the Stable ABI,C API Changes<2>
@anchor{whatsnew/3 10 id1}@anchor{35e}
@subsubsection New Features



@itemize *

@item 
The result of @ref{35f,,PyNumber_Index()} now always has exact type @ref{1b1,,int}.
Previously, the result could have been an instance of a subclass of @code{int}.
(Contributed by Serhiy Storchaka in bpo-40792@footnote{https://bugs.python.org/issue40792}.)

@item 
Add a new @ref{360,,orig_argv} member to the @ref{206,,PyConfig}
structure: the list of the original command line arguments passed to the
Python executable.
(Contributed by Victor Stinner in bpo-23427@footnote{https://bugs.python.org/issue23427}.)

@item 
The @ref{361,,PyDateTime_DATE_GET_TZINFO()} and
@ref{362,,PyDateTime_TIME_GET_TZINFO()} macros have been added for accessing
the @code{tzinfo} attributes of @ref{363,,datetime.datetime} and
@ref{364,,datetime.time} objects.
(Contributed by Zackery Spytz in bpo-30155@footnote{https://bugs.python.org/issue30155}.)

@item 
Add a @ref{365,,PyCodec_Unregister()} function to unregister a codec
search function.
(Contributed by Hai Shi in bpo-41842@footnote{https://bugs.python.org/issue41842}.)

@item 
The @ref{366,,PyIter_Send()} function was added to allow
sending value into iterator without raising @code{StopIteration} exception.
(Contributed by Vladimir Matveev in bpo-41756@footnote{https://bugs.python.org/issue41756}.)

@item 
Add @ref{367,,PyUnicode_AsUTF8AndSize()} to the limited C API.
(Contributed by Alex Gaynor in bpo-41784@footnote{https://bugs.python.org/issue41784}.)

@item 
Add @ref{368,,PyModule_AddObjectRef()} function: similar to
@ref{369,,PyModule_AddObject()} but don’t steal a reference to the value on
success.
(Contributed by Victor Stinner in bpo-1635741@footnote{https://bugs.python.org/issue1635741}.)

@item 
Add @ref{36a,,Py_NewRef()} and @ref{36b,,Py_XNewRef()} functions to increment the
reference count of an object and return the object.
(Contributed by Victor Stinner in bpo-42262@footnote{https://bugs.python.org/issue42262}.)

@item 
The @ref{36c,,PyType_FromSpecWithBases()} and @ref{36d,,PyType_FromModuleAndSpec()}
functions now accept a single class as the @emph{bases} argument.
(Contributed by Serhiy Storchaka in bpo-42423@footnote{https://bugs.python.org/issue42423}.)

@item 
The @ref{36d,,PyType_FromModuleAndSpec()} function now accepts NULL @code{tp_doc}
slot.
(Contributed by Hai Shi in bpo-41832@footnote{https://bugs.python.org/issue41832}.)

@item 
The @ref{36e,,PyType_GetSlot()} function can accept
@ref{1f0,,static types}.
(Contributed by Hai Shi and Petr Viktorin in bpo-41073@footnote{https://bugs.python.org/issue41073}.)

@item 
Add a new @ref{36f,,PySet_CheckExact()} function to the C-API to check if an
object is an instance of @ref{370,,set} but not an instance of a subtype.
(Contributed by Pablo Galindo in bpo-43277@footnote{https://bugs.python.org/issue43277}.)

@item 
Add @ref{371,,PyErr_SetInterruptEx()} which allows passing a signal number
to simulate.
(Contributed by Antoine Pitrou in bpo-43356@footnote{https://bugs.python.org/issue43356}.)

@item 
The limited C API is now supported if @ref{338,,Python is built in debug mode} (if the @code{Py_DEBUG} macro is defined). In the limited C API,
the @ref{372,,Py_INCREF()} and @ref{373,,Py_DECREF()} functions are now implemented
as opaque function
calls, rather than accessing directly the @ref{374,,PyObject.ob_refcnt}
member, if Python is built in debug mode and the @code{Py_LIMITED_API} macro
targets Python 3.10 or newer. It became possible to support the limited C API
in debug mode because the @ref{375,,PyObject} structure is the same in release
and debug mode since Python 3.8 (see bpo-36465@footnote{https://bugs.python.org/issue36465}).

The limited C API is still not supported in the @ref{376,,--with-trace-refs}
special build (@code{Py_TRACE_REFS} macro).
(Contributed by Victor Stinner in bpo-43688@footnote{https://bugs.python.org/issue43688}.)

@item 
Add the @ref{377,,Py_Is(x@comma{} y)} function to test if the @emph{x} object is
the @emph{y} object, the same as @code{x is y} in Python. Add also the
@ref{378,,Py_IsNone()}, @ref{379,,Py_IsTrue()}, @ref{37a,,Py_IsFalse()} functions to
test if an object is, respectively, the @code{None} singleton, the @code{True}
singleton or the @code{False} singleton.
(Contributed by Victor Stinner in bpo-43753@footnote{https://bugs.python.org/issue43753}.)

@item 
Add new functions to control the garbage collector from C code:
@ref{37b,,PyGC_Enable()},
@ref{37c,,PyGC_Disable()},
@ref{37d,,PyGC_IsEnabled()}.
These functions allow to activate, deactivate and query the state of the garbage collector from C code without
having to import the @ref{80,,gc} module.

@item 
Add a new @code{Py_TPFLAGS_DISALLOW_INSTANTIATION} type flag to disallow
creating type instances.
(Contributed by Victor Stinner in bpo-43916@footnote{https://bugs.python.org/issue43916}.)

@item 
Add a new @code{Py_TPFLAGS_IMMUTABLETYPE} type flag for creating immutable
type objects: type attributes cannot be set nor deleted.
(Contributed by Victor Stinner and Erlend E. Aasland in bpo-43908@footnote{https://bugs.python.org/issue43908}.)
@end itemize

@node Porting to Python 3 10<2>,Deprecated<4>,New Features<4>,C API Changes<2>
@anchor{whatsnew/3 10 id2}@anchor{37e}
@subsubsection Porting to Python 3.10



@itemize *

@item 
The @code{PY_SSIZE_T_CLEAN} macro must now be defined to use
@ref{37f,,PyArg_ParseTuple()} and @ref{380,,Py_BuildValue()} formats which use
@code{#}: @code{es#}, @code{et#}, @code{s#}, @code{u#}, @code{y#}, @code{z#}, @code{U#} and @code{Z#}.
See @ref{381,,Parsing arguments and building values} and the 
@geindex Python Enhancement Proposals; PEP 353
PEP 353@footnote{https://www.python.org/dev/peps/pep-0353}.
(Contributed by Victor Stinner in bpo-40943@footnote{https://bugs.python.org/issue40943}.)

@item 
Since @ref{382,,Py_REFCNT()} is changed to the inline static function,
@code{Py_REFCNT(obj) = new_refcnt} must be replaced with @code{Py_SET_REFCNT(obj, new_refcnt)}:
see @ref{383,,Py_SET_REFCNT()} (available since Python 3.9). For backward
compatibility, this macro can be used:

@example
#if PY_VERSION_HEX < 0x030900A4
#  define Py_SET_REFCNT(obj, refcnt) ((Py_REFCNT(obj) = (refcnt)), (void)0)
#endif
@end example

(Contributed by Victor Stinner in bpo-39573@footnote{https://bugs.python.org/issue39573}.)

@item 
Calling @ref{384,,PyDict_GetItem()} without @ref{385,,GIL} held had been allowed
for historical reason. It is no longer allowed.
(Contributed by Victor Stinner in bpo-40839@footnote{https://bugs.python.org/issue40839}.)

@item 
@code{PyUnicode_FromUnicode(NULL, size)} and @code{PyUnicode_FromStringAndSize(NULL, size)}
raise @code{DeprecationWarning} now.  Use @ref{386,,PyUnicode_New()} to allocate
Unicode object without initial data.
(Contributed by Inada Naoki in bpo-36346@footnote{https://bugs.python.org/issue36346}.)

@item 
The private @code{_PyUnicode_Name_CAPI} structure of the PyCapsule API
@code{unicodedata.ucnhash_CAPI} has been moved to the internal C API.
(Contributed by Victor Stinner in bpo-42157@footnote{https://bugs.python.org/issue42157}.)

@item 
@ref{387,,Py_GetPath()}, @ref{388,,Py_GetPrefix()}, @ref{389,,Py_GetExecPrefix()},
@ref{38a,,Py_GetProgramFullPath()}, @ref{38b,,Py_GetPythonHome()} and
@ref{38c,,Py_GetProgramName()} functions now return @code{NULL} if called before
@ref{38d,,Py_Initialize()} (before Python is initialized). Use the new
@ref{207,,Python Initialization Configuration API} to get the
@ref{38e,,Python Path Configuration.}.
(Contributed by Victor Stinner in bpo-42260@footnote{https://bugs.python.org/issue42260}.)

@item 
@ref{38f,,PyList_SET_ITEM()}, @ref{390,,PyTuple_SET_ITEM()} and
@ref{391,,PyCell_SET()} macros can no longer be used as l-value or r-value.
For example, @code{x = PyList_SET_ITEM(a, b, c)} and
@code{PyList_SET_ITEM(a, b, c) = x} now fail with a compiler error. It prevents
bugs like @code{if (PyList_SET_ITEM (a, b, c) < 0) ...} test.
(Contributed by Zackery Spytz and Victor Stinner in bpo-30459@footnote{https://bugs.python.org/issue30459}.)

@item 
The non-limited API files @code{odictobject.h}, @code{parser_interface.h},
@code{picklebufobject.h}, @code{pyarena.h}, @code{pyctype.h}, @code{pydebug.h},
@code{pyfpe.h}, and @code{pytime.h} have been moved to the @code{Include/cpython}
directory. These files must not be included directly, as they are already
included in @code{Python.h}: @ref{1f5,,Include Files}. If they have
been included directly, consider including @code{Python.h} instead.
(Contributed by Nicholas Sim in bpo-35134@footnote{https://bugs.python.org/issue35134})

@item 
Use the @code{Py_TPFLAGS_IMMUTABLETYPE} type flag to create immutable type
objects. Do not rely on @code{Py_TPFLAGS_HEAPTYPE} to decide if a type
object is mutable or not; check if @code{Py_TPFLAGS_IMMUTABLETYPE} is set
instead.
(Contributed by Victor Stinner and Erlend E. Aasland in bpo-43908@footnote{https://bugs.python.org/issue43908}.)

@item 
The undocumented function @code{Py_FrozenMain} has been removed from the
limited API. The function is mainly useful for custom builds of Python.
(Contributed by Petr Viktorin in bpo-26241@footnote{https://bugs.python.org/issue26241})
@end itemize

@node Deprecated<4>,Removed<4>,Porting to Python 3 10<2>,C API Changes<2>
@anchor{whatsnew/3 10 id3}@anchor{392}
@subsubsection Deprecated



@itemize *

@item 
The @code{PyUnicode_InternImmortal()} function is now deprecated
and will be removed in Python 3.12: use @ref{393,,PyUnicode_InternInPlace()}
instead.
(Contributed by Victor Stinner in bpo-41692@footnote{https://bugs.python.org/issue41692}.)
@end itemize

@node Removed<4>,,Deprecated<4>,C API Changes<2>
@anchor{whatsnew/3 10 id4}@anchor{394}
@subsubsection Removed



@itemize *

@item 
Removed @code{Py_UNICODE_str*} functions manipulating @code{Py_UNICODE*} strings.
(Contributed by Inada Naoki in bpo-41123@footnote{https://bugs.python.org/issue41123}.)

@quotation


@itemize *

@item 
@code{Py_UNICODE_strlen}: use @ref{395,,PyUnicode_GetLength()} or
@ref{396,,PyUnicode_GET_LENGTH}

@item 
@code{Py_UNICODE_strcat}: use @ref{397,,PyUnicode_CopyCharacters()} or
@ref{398,,PyUnicode_FromFormat()}

@item 
@code{Py_UNICODE_strcpy}, @code{Py_UNICODE_strncpy}: use
@ref{397,,PyUnicode_CopyCharacters()} or @ref{399,,PyUnicode_Substring()}

@item 
@code{Py_UNICODE_strcmp}: use @ref{39a,,PyUnicode_Compare()}

@item 
@code{Py_UNICODE_strncmp}: use @ref{39b,,PyUnicode_Tailmatch()}

@item 
@code{Py_UNICODE_strchr}, @code{Py_UNICODE_strrchr}: use
@ref{39c,,PyUnicode_FindChar()}
@end itemize
@end quotation

@item 
Removed @code{PyUnicode_GetMax()}. Please migrate to new (
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393}) APIs.
(Contributed by Inada Naoki in bpo-41103@footnote{https://bugs.python.org/issue41103}.)

@item 
Removed @code{PyLong_FromUnicode()}. Please migrate to @ref{39d,,PyLong_FromUnicodeObject()}.
(Contributed by Inada Naoki in bpo-41103@footnote{https://bugs.python.org/issue41103}.)

@item 
Removed @code{PyUnicode_AsUnicodeCopy()}. Please use @ref{39e,,PyUnicode_AsUCS4Copy()} or
@ref{39f,,PyUnicode_AsWideCharString()}
(Contributed by Inada Naoki in bpo-41103@footnote{https://bugs.python.org/issue41103}.)

@item 
Removed @code{_Py_CheckRecursionLimit} variable: it has been replaced by
@code{ceval.recursion_limit} of the @ref{3a0,,PyInterpreterState} structure.
(Contributed by Victor Stinner in bpo-41834@footnote{https://bugs.python.org/issue41834}.)

@item 
Removed undocumented macros @code{Py_ALLOW_RECURSION} and
@code{Py_END_ALLOW_RECURSION} and the @code{recursion_critical} field of the
@ref{3a0,,PyInterpreterState} structure.
(Contributed by Serhiy Storchaka in bpo-41936@footnote{https://bugs.python.org/issue41936}.)

@item 
Removed the undocumented @code{PyOS_InitInterrupts()} function. Initializing
Python already implicitly installs signal handlers: see
@ref{3a1,,PyConfig.install_signal_handlers}.
(Contributed by Victor Stinner in bpo-41713@footnote{https://bugs.python.org/issue41713}.)

@item 
Remove the @code{PyAST_Validate()} function. It is no longer possible to build a
AST object (@code{mod_ty} type) with the public C API. The function was already
excluded from the limited C API (
@geindex Python Enhancement Proposals; PEP 384
PEP 384@footnote{https://www.python.org/dev/peps/pep-0384}).
(Contributed by Victor Stinner in bpo-43244@footnote{https://bugs.python.org/issue43244}.)

@item 
Remove the @code{symtable.h} header file and the undocumented functions:


@itemize *

@item 
@code{PyST_GetScope()}

@item 
@code{PySymtable_Build()}

@item 
@code{PySymtable_BuildObject()}

@item 
@code{PySymtable_Free()}

@item 
@code{Py_SymtableString()}

@item 
@code{Py_SymtableStringObject()}
@end itemize

The @code{Py_SymtableString()} function was part the stable ABI by mistake but
it could not be used, because the @code{symtable.h} header file was excluded
from the limited C API.

Use Python @ref{f6,,symtable} module instead.
(Contributed by Victor Stinner in bpo-43244@footnote{https://bugs.python.org/issue43244}.)

@item 
Remove @ref{3a2,,PyOS_ReadlineFunctionPointer()} from the limited C API headers
and from @code{python3.dll}, the library that provides the stable ABI on
Windows. Since the function takes a @code{FILE*} argument, its ABI stability
cannot be guaranteed.
(Contributed by Petr Viktorin in bpo-43868@footnote{https://bugs.python.org/issue43868}.)

@item 
Remove @code{ast.h}, @code{asdl.h}, and @code{Python-ast.h} header files.
These functions were undocumented and excluded from the limited C API.
Most names defined by these header files were not prefixed by @code{Py} and so
could create names conflicts. For example, @code{Python-ast.h} defined a
@code{Yield} macro which was conflict with the @code{Yield} name used by the
Windows @code{<winbase.h>} header. Use the Python @ref{7,,ast} module instead.
(Contributed by Victor Stinner in bpo-43244@footnote{https://bugs.python.org/issue43244}.)

@item 
Remove the compiler and parser functions using @code{struct _mod} type, because
the public AST C API was removed:


@itemize *

@item 
@code{PyAST_Compile()}

@item 
@code{PyAST_CompileEx()}

@item 
@code{PyAST_CompileObject()}

@item 
@code{PyFuture_FromAST()}

@item 
@code{PyFuture_FromASTObject()}

@item 
@code{PyParser_ASTFromFile()}

@item 
@code{PyParser_ASTFromFileObject()}

@item 
@code{PyParser_ASTFromFilename()}

@item 
@code{PyParser_ASTFromString()}

@item 
@code{PyParser_ASTFromStringObject()}
@end itemize

These functions were undocumented and excluded from the limited C API.
(Contributed by Victor Stinner in bpo-43244@footnote{https://bugs.python.org/issue43244}.)

@item 
Remove the @code{pyarena.h} header file with functions:


@itemize *

@item 
@code{PyArena_New()}

@item 
@code{PyArena_Free()}

@item 
@code{PyArena_Malloc()}

@item 
@code{PyArena_AddPyObject()}
@end itemize

These functions were undocumented, excluded from the limited C API, and were
only used internally by the compiler.
(Contributed by Victor Stinner in bpo-43244@footnote{https://bugs.python.org/issue43244}.)

@item 
The @code{PyThreadState.use_tracing} member has been removed to optimize Python.
(Contributed by Mark Shannon in bpo-43760@footnote{https://bugs.python.org/issue43760}.)
@end itemize

@node What’s New In Python 3 9,What’s New In Python 3 8,What’s New In Python 3 10,What’s New in Python
@anchor{whatsnew/3 9 doc}@anchor{3a3}@anchor{whatsnew/3 9 what-s-new-in-python-3-9}@anchor{3a4}
@section What’s New In Python 3.9


@*Release: 
3.11.0a5


@*Date: 
March 07, 2022


@*Editor: 
Łukasz Langa


@c Rules for maintenance:
@c 
@c * Anyone can add text to this document.  Your text might get
@c rewritten to some degree.
@c 
@c * The maintainer will go through Misc/NEWS periodically and add
@c changes; it's therefore more important to add your changes to
@c Misc/NEWS than to this file.
@c 
@c * This is not a complete list of every single change; completeness
@c is the purpose of Misc/NEWS.  Some changes will be too small
@c or esoteric to include.  If such a change is added to the text,
@c it might get removed during final editing.
@c 
@c * If you want to draw your new text to the attention of the
@c maintainer, add 'XXX' to the beginning of the paragraph or
@c section.
@c 
@c * It's OK to just add a fragmentary note about a change.  For
@c example: "XXX Describe the transmogrify() function added to the
@c socket module."  The maintainer will research the change and
@c write the necessary text.
@c 
@c * You can comment out your additions if you like, but it's not
@c necessary (especially when a final release is some months away).
@c 
@c * Credit the author of a patch or bugfix.   Just the name is
@c sufficient; the e-mail address isn't necessary.
@c 
@c * It's helpful to add the bug/patch number as a comment:
@c 
@c XXX Describe the transmogrify() function added to the socket
@c module.
@c (Contributed by P.Y. Developer in :issue:`12345`.)
@c 
@c This saves the maintainer the effort of going through the Mercurial log
@c when researching a change.

This article explains the new features in Python 3.9, compared to 3.8.
Python 3.9 was released on October 5th, 2020.

For full details, see the @ref{154,,changelog}.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 596
PEP 596@footnote{https://www.python.org/dev/peps/pep-0596} - Python 3.9 Release Schedule


@menu
* Summary – Release highlights: Summary – Release highlights<3>. 
* You should check for DeprecationWarning in your code:: 
* New Features: New Features<5>. 
* Other Language Changes: Other Language Changes<3>. 
* New Modules: New Modules<3>. 
* Improved Modules: Improved Modules<3>. 
* Optimizations: Optimizations<3>. 
* Deprecated: Deprecated<5>. 
* Removed: Removed<5>. 
* Porting to Python 3.9: Porting to Python 3 9. 
* Build Changes: Build Changes<3>. 
* C API Changes: C API Changes<3>. 
* Notable changes in Python 3.9.1: Notable changes in Python 3 9 1. 
* Notable changes in Python 3.9.2: Notable changes in Python 3 9 2. 

@end menu

@node Summary – Release highlights<3>,You should check for DeprecationWarning in your code,,What’s New In Python 3 9
@anchor{whatsnew/3 9 summary-release-highlights}@anchor{3a5}
@subsection Summary – Release highlights


@c This section singles out the most important changes in Python 3.9.
@c Brevity is key.

New syntax features:


@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584}, union operators added to @code{dict};

@item 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}, type hinting generics in standard collections;

@item 
@geindex Python Enhancement Proposals; PEP 614
PEP 614@footnote{https://www.python.org/dev/peps/pep-0614}, relaxed grammar restrictions on decorators.
@end itemize

New built-in features:


@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 616
PEP 616@footnote{https://www.python.org/dev/peps/pep-0616}, string methods to remove prefixes and suffixes.
@end itemize

New features in the standard library:


@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 593
PEP 593@footnote{https://www.python.org/dev/peps/pep-0593}, flexible function and variable annotations;

@item 
@ref{3a6,,os.pidfd_open()} added that allows process management without races
and signals.
@end itemize

Interpreter improvements:


@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 573
PEP 573@footnote{https://www.python.org/dev/peps/pep-0573}, fast access to module state from methods of C extension
types;

@item 
@geindex Python Enhancement Proposals; PEP 617
PEP 617@footnote{https://www.python.org/dev/peps/pep-0617}, CPython now uses a new parser based on PEG;

@item 
a number of Python builtins (range, tuple, set, frozenset, list, dict) are
now sped up using 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590} vectorcall;

@item 
garbage collection does not block on resurrected objects;

@item 
a number of Python modules (@code{_abc}, @ref{c,,audioop}, @code{_bz2},
@code{_codecs}, @code{_contextvars}, @code{_crypt}, @code{_functools},
@code{_json}, @code{_locale}, @ref{ad,,math}, @ref{be,,operator}, @ref{db,,resource},
@ref{10a,,time}, @code{_weakref}) now use multiphase initialization as defined
by PEP 489;

@item 
a number of standard library modules (@ref{c,,audioop}, @ref{7,,ast}, @ref{86,,grp},
@code{_hashlib}, @ref{d1,,pwd}, @code{_posixsubprocess}, @ref{d7,,random},
@ref{e0,,select}, @ref{f3,,struct}, @ref{fe,,termios}, @ref{14b,,zlib}) are now using
the stable ABI defined by PEP 384.
@end itemize

New library modules:


@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 615
PEP 615@footnote{https://www.python.org/dev/peps/pep-0615}, the IANA Time Zone Database is now present in the standard
library in the @ref{14c,,zoneinfo} module;

@item 
an implementation of a topological sort of a graph is now provided in
the new @ref{85,,graphlib} module.
@end itemize

Release process changes:


@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 602
PEP 602@footnote{https://www.python.org/dev/peps/pep-0602}, CPython adopts an annual release cycle.
@end itemize

@node You should check for DeprecationWarning in your code,New Features<5>,Summary – Release highlights<3>,What’s New In Python 3 9
@anchor{whatsnew/3 9 you-should-check-for-deprecationwarning-in-your-code}@anchor{3a7}
@subsection You should check for DeprecationWarning in your code


When Python 2.7 was still supported, a lot of functionality in Python 3
was kept for backward compatibility with Python 2.7. With the end of Python
2 support, these backward compatibility layers have been removed, or will
be removed soon. Most of them emitted a @ref{1b2,,DeprecationWarning} warning for
several years. For example, using @code{collections.Mapping} instead of
@code{collections.abc.Mapping} emits a @ref{1b2,,DeprecationWarning} since Python
3.3, released in 2012.

Test your application with the @ref{3a8,,-W} @code{default} command-line option to see
@ref{1b2,,DeprecationWarning} and @ref{3a9,,PendingDeprecationWarning}, or even with
@ref{3a8,,-W} @code{error} to treat them as errors. @ref{3aa,,Warnings Filter} can be used to ignore warnings from third-party code.

Python 3.9 is the last version providing those Python 2 backward compatibility
layers, to give more time to Python projects maintainers to organize the
removal of the Python 2 support and add support for Python 3.9.

Aliases to @ref{33d,,Abstract Base Classes} in
the @ref{1c,,collections} module, like @code{collections.Mapping} alias to
@ref{3ab,,collections.abc.Mapping}, are kept for one last release for backward
compatibility. They will be removed from Python 3.10.

More generally, try to run your tests in the @ref{3ac,,Python Development Mode} which helps to prepare your code to make it compatible with the
next Python version.

Note: a number of pre-existing deprecations were removed in this version of
Python as well. Consult the @ref{3ad,,Removed} section.

@node New Features<5>,Other Language Changes<3>,You should check for DeprecationWarning in your code,What’s New In Python 3 9
@anchor{whatsnew/3 9 new-features}@anchor{3ae}
@subsection New Features


@menu
* Dictionary Merge & Update Operators:: 
* New String Methods to Remove Prefixes and Suffixes:: 
* Type Hinting Generics in Standard Collections:: 
* New Parser:: 

@end menu

@node Dictionary Merge & Update Operators,New String Methods to Remove Prefixes and Suffixes,,New Features<5>
@anchor{whatsnew/3 9 dictionary-merge-update-operators}@anchor{3af}
@subsubsection Dictionary Merge & Update Operators


Merge (@code{|}) and update (@code{|=}) operators have been added to the built-in
@ref{3b0,,dict} class. Those complement the existing @code{dict.update} and
@code{@{**d1, **d2@}} methods of merging dictionaries.

Example:

@example
>>> x = @{"key1": "value1 from x", "key2": "value2 from x"@}
>>> y = @{"key2": "value2 from y", "key3": "value3 from y"@}
>>> x | y
@{'key1': 'value1 from x', 'key2': 'value2 from y', 'key3': 'value3 from y'@}
>>> y | x
@{'key2': 'value2 from x', 'key3': 'value3 from y', 'key1': 'value1 from x'@}
@end example

See 
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584} for a full description.
(Contributed by Brandt Bucher in bpo-36144@footnote{https://bugs.python.org/issue36144}.)

@node New String Methods to Remove Prefixes and Suffixes,Type Hinting Generics in Standard Collections,Dictionary Merge & Update Operators,New Features<5>
@anchor{whatsnew/3 9 new-string-methods-to-remove-prefixes-and-suffixes}@anchor{3b1}
@subsubsection New String Methods to Remove Prefixes and Suffixes


@ref{3b2,,str.removeprefix(prefix)} and
@ref{3b3,,str.removesuffix(suffix)} have been added
to easily remove an unneeded prefix or a suffix from a string. Corresponding
@code{bytes}, @code{bytearray}, and @code{collections.UserString} methods have also been
added. See 
@geindex Python Enhancement Proposals; PEP 616
PEP 616@footnote{https://www.python.org/dev/peps/pep-0616} for a full description. (Contributed by Dennis Sweeney in
bpo-39939@footnote{https://bugs.python.org/issue39939}.)

@node Type Hinting Generics in Standard Collections,New Parser,New String Methods to Remove Prefixes and Suffixes,New Features<5>
@anchor{whatsnew/3 9 type-hinting-generics-in-standard-collections}@anchor{3b4}
@subsubsection Type Hinting Generics in Standard Collections


In type annotations you can now use built-in collection types such as
@code{list} and @code{dict} as generic types instead of importing the
corresponding capitalized types (e.g. @code{List} or @code{Dict}) from
@code{typing}.  Some other types in the standard library are also now generic,
for example @code{queue.Queue}.

Example:

@example
def greet_all(names: list[str]) -> None:
    for name in names:
        print("Hello", name)
@end example

See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} for more details.  (Contributed by Guido van Rossum,
Ethan Smith, and Batuhan Taşkaya in bpo-39481@footnote{https://bugs.python.org/issue39481}.)

@node New Parser,,Type Hinting Generics in Standard Collections,New Features<5>
@anchor{whatsnew/3 9 new-parser}@anchor{3b5}
@subsubsection New Parser


Python 3.9 uses a new parser, based on PEG@footnote{https://en.wikipedia.org/wiki/Parsing_expression_grammar} instead
of LL(1)@footnote{https://en.wikipedia.org/wiki/LL_parser}.  The new
parser’s performance is roughly comparable to that of the old parser,
but the PEG formalism is more flexible than LL(1) when it comes to
designing new language features.  We’ll start using this flexibility
in Python 3.10 and later.

The @ref{7,,ast} module uses the new parser and produces the same AST as
the old parser.

In Python 3.10, the old parser will be deleted and so will all
functionality that depends on it (primarily the @code{parser} module,
which has long been deprecated).  In Python 3.9 @emph{only}, you can switch
back to the LL(1) parser using a command line switch (@code{-X
oldparser}) or an environment variable (@code{PYTHONOLDPARSER=1}).

See 
@geindex Python Enhancement Proposals; PEP 617
PEP 617@footnote{https://www.python.org/dev/peps/pep-0617} for more details.  (Contributed by Guido van Rossum,
Pablo Galindo and Lysandros Nikolaou in bpo-40334@footnote{https://bugs.python.org/issue40334}.)

@node Other Language Changes<3>,New Modules<3>,New Features<5>,What’s New In Python 3 9
@anchor{whatsnew/3 9 other-language-changes}@anchor{3b6}
@subsection Other Language Changes



@itemize *

@item 
@ref{3b7,,__import__()} now raises @ref{3b8,,ImportError} instead of
@ref{17c,,ValueError}, which used to occur when a relative import went past
its top-level package.
(Contributed by Ngalim Siregar in bpo-37444@footnote{https://bugs.python.org/issue37444}.)

@item 
Python now gets the absolute path of the script filename specified on
the command line (ex: @code{python3 script.py}): the @code{__file__} attribute of
the @ref{1,,__main__} module became an absolute path, rather than a relative
path. These paths now remain valid after the current directory is changed
by @ref{3b9,,os.chdir()}. As a side effect, the traceback also displays the
absolute path for @ref{1,,__main__} module frames in this case.
(Contributed by Victor Stinner in bpo-20443@footnote{https://bugs.python.org/issue20443}.)

@item 
In the @ref{3ac,,Python Development Mode} and in @ref{338,,debug build}, the
@emph{encoding} and @emph{errors} arguments are now checked for string encoding and
decoding operations. Examples: @ref{1c8,,open()}, @ref{3ba,,str.encode()} and
@ref{3bb,,bytes.decode()}.

By default, for best performance, the @emph{errors} argument is only checked at
the first encoding/decoding error and the @emph{encoding} argument is sometimes
ignored for empty strings.
(Contributed by Victor Stinner in bpo-37388@footnote{https://bugs.python.org/issue37388}.)

@item 
@code{"".replace("", s, n)} now returns @code{s} instead of an empty string for
all non-zero @code{n}.  It is now consistent with @code{"".replace("", s)}.
There are similar changes for @ref{2f0,,bytes} and @ref{2f1,,bytearray} objects.
(Contributed by Serhiy Storchaka in bpo-28029@footnote{https://bugs.python.org/issue28029}.)

@item 
Any valid expression can now be used as a @ref{1bf,,decorator}.  Previously, the
grammar was much more restrictive.  See 
@geindex Python Enhancement Proposals; PEP 614
PEP 614@footnote{https://www.python.org/dev/peps/pep-0614} for details.
(Contributed by Brandt Bucher in bpo-39702@footnote{https://bugs.python.org/issue39702}.)

@item 
Improved help for the @ref{120,,typing} module. Docstrings are now shown for
all special forms and special generic aliases (like @code{Union} and @code{List}).
Using @ref{3bc,,help()} with generic alias like @code{List[int]} will show the help
for the correspondent concrete type (@code{list} in this case).
(Contributed by Serhiy Storchaka in bpo-40257@footnote{https://bugs.python.org/issue40257}.)

@item 
Parallel running of @ref{3bd,,aclose()} / @ref{3be,,asend()} /
@ref{3bf,,athrow()} is now prohibited, and @code{ag_running} now reflects
the actual running status of the async generator.
(Contributed by Yury Selivanov in bpo-30773@footnote{https://bugs.python.org/issue30773}.)

@item 
Unexpected errors in calling the @code{__iter__} method are no longer masked by
@code{TypeError} in the @ref{2fc,,in} operator and functions
@ref{3c0,,contains()}, @ref{3c1,,indexOf()} and
@ref{3c2,,countOf()} of the @ref{be,,operator} module.
(Contributed by Serhiy Storchaka in bpo-40824@footnote{https://bugs.python.org/issue40824}.)

@item 
Unparenthesized lambda expressions can no longer be the expression part in an
@code{if} clause in comprehensions and generator expressions. See bpo-41848@footnote{https://bugs.python.org/issue41848}
and bpo-43755@footnote{https://bugs.python.org/issue43755} for details.
@end itemize

@node New Modules<3>,Improved Modules<3>,Other Language Changes<3>,What’s New In Python 3 9
@anchor{whatsnew/3 9 new-modules}@anchor{3c3}
@subsection New Modules


@menu
* zoneinfo:: 
* graphlib:: 

@end menu

@node zoneinfo,graphlib,,New Modules<3>
@anchor{whatsnew/3 9 zoneinfo}@anchor{3c4}
@subsubsection zoneinfo


The @ref{14c,,zoneinfo} module brings support for the IANA time zone database to
the standard library. It adds @ref{3c5,,zoneinfo.ZoneInfo}, a concrete
@ref{3c6,,datetime.tzinfo} implementation backed by the system’s time zone data.

Example:

@example
>>> from zoneinfo import ZoneInfo
>>> from datetime import datetime, timedelta

>>> # Daylight saving time
>>> dt = datetime(2020, 10, 31, 12, tzinfo=ZoneInfo("America/Los_Angeles"))
>>> print(dt)
2020-10-31 12:00:00-07:00
>>> dt.tzname()
'PDT'

>>> # Standard time
>>> dt += timedelta(days=7)
>>> print(dt)
2020-11-07 12:00:00-08:00
>>> print(dt.tzname())
PST
@end example

As a fall-back source of data for platforms that don’t ship the IANA database,
the tzdata@footnote{https://pypi.org/project/tzdata/} module was released as a first-party package – distributed via
PyPI and maintained by the CPython core team.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 615

@item PEP 615@footnote{https://www.python.org/dev/peps/pep-0615} – Support for the IANA Time Zone Database in the Standard Library

PEP written and implemented by Paul Ganssle
@end table


@node graphlib,,zoneinfo,New Modules<3>
@anchor{whatsnew/3 9 graphlib}@anchor{3c7}
@subsubsection graphlib


A new module, @ref{85,,graphlib}, was added that contains the
@ref{3c8,,graphlib.TopologicalSorter} class to offer functionality to perform
topological sorting of graphs. (Contributed by Pablo Galindo, Tim Peters and
Larry Hastings in bpo-17005@footnote{https://bugs.python.org/issue17005}.)

@node Improved Modules<3>,Optimizations<3>,New Modules<3>,What’s New In Python 3 9
@anchor{whatsnew/3 9 improved-modules}@anchor{3c9}
@subsection Improved Modules


@menu
* ast:: 
* asyncio: asyncio<2>. 
* compileall:: 
* concurrent.futures: concurrent futures. 
* curses: curses<2>. 
* datetime:: 
* distutils: distutils<2>. 
* fcntl: fcntl<2>. 
* ftplib:: 
* gc: gc<2>. 
* hashlib: hashlib<2>. 
* http:: 
* IDLE and idlelib: IDLE and idlelib<3>. 
* imaplib:: 
* importlib:: 
* inspect: inspect<3>. 
* ipaddress:: 
* math: math<2>. 
* multiprocessing:: 
* nntplib:: 
* os: os<3>. 
* pathlib: pathlib<2>. 
* pdb:: 
* poplib:: 
* pprint: pprint<2>. 
* pydoc:: 
* random:: 
* signal:: 
* smtplib:: 
* socket: socket<3>. 
* time: time<2>. 
* sys: sys<3>. 
* tracemalloc:: 
* typing: typing<2>. 
* unicodedata: unicodedata<2>. 
* venv:: 
* xml: xml<2>. 

@end menu

@node ast,asyncio<2>,,Improved Modules<3>
@anchor{whatsnew/3 9 ast}@anchor{3ca}
@subsubsection ast


Added the @emph{indent} option to @ref{3cb,,dump()} which allows it to produce a
multiline indented output.
(Contributed by Serhiy Storchaka in bpo-37995@footnote{https://bugs.python.org/issue37995}.)

Added @ref{3cc,,ast.unparse()} as a function in the @ref{7,,ast} module that can
be used to unparse an @ref{3cd,,ast.AST} object and produce a string with code
that would produce an equivalent @ref{3cd,,ast.AST} object when parsed.
(Contributed by Pablo Galindo and Batuhan Taskaya in bpo-38870@footnote{https://bugs.python.org/issue38870}.)

Added docstrings to AST nodes that contains the ASDL signature used to
construct that node. (Contributed by Batuhan Taskaya in bpo-39638@footnote{https://bugs.python.org/issue39638}.)

@node asyncio<2>,compileall,ast,Improved Modules<3>
@anchor{whatsnew/3 9 asyncio}@anchor{3ce}
@subsubsection asyncio


Due to significant security concerns, the @emph{reuse_address} parameter of
@ref{1b9,,asyncio.loop.create_datagram_endpoint()} is no longer supported. This is
because of the behavior of the socket option @code{SO_REUSEADDR} in UDP. For more
details, see the documentation for @code{loop.create_datagram_endpoint()}.
(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in
bpo-37228@footnote{https://bugs.python.org/issue37228}.)

Added a new @ref{3cf,,coroutine} @ref{3d0,,shutdown_default_executor()}
that schedules a shutdown for the default executor that waits on the
@ref{1c7,,ThreadPoolExecutor} to finish closing. Also,
@ref{3d1,,asyncio.run()} has been updated to use the new @ref{3cf,,coroutine}.
(Contributed by Kyle Stanley in bpo-34037@footnote{https://bugs.python.org/issue34037}.)

Added @ref{3d2,,asyncio.PidfdChildWatcher}, a Linux-specific child watcher
implementation that polls process file descriptors. (bpo-38692@footnote{https://bugs.python.org/issue38692})

Added a new @ref{3cf,,coroutine} @ref{3d3,,asyncio.to_thread()}. It is mainly used for
running IO-bound functions in a separate thread to avoid blocking the event
loop, and essentially works as a high-level version of
@ref{3d4,,run_in_executor()} that can directly take keyword arguments.
(Contributed by Kyle Stanley and Yury Selivanov in bpo-32309@footnote{https://bugs.python.org/issue32309}.)

When cancelling the task due to a timeout, @ref{3d5,,asyncio.wait_for()} will now
wait until the cancellation is complete also in the case when @emph{timeout} is
<= 0, like it does with positive timeouts.
(Contributed by Elvis Pranskevichus in bpo-32751@footnote{https://bugs.python.org/issue32751}.)

@ref{9,,asyncio} now raises @code{TyperError} when calling incompatible
methods with an @ref{3d6,,ssl.SSLSocket} socket.
(Contributed by Ido Michael in bpo-37404@footnote{https://bugs.python.org/issue37404}.)

@node compileall,concurrent futures,asyncio<2>,Improved Modules<3>
@anchor{whatsnew/3 9 compileall}@anchor{3d7}
@subsubsection compileall


Added new possibility to use hardlinks for duplicated @code{.pyc} files: @emph{hardlink_dupes} parameter and –hardlink-dupes command line option.
(Contributed by  Lumír ‘Frenzy’ Balhar in bpo-40495@footnote{https://bugs.python.org/issue40495}.)

Added new options for path manipulation in resulting @code{.pyc} files: @emph{stripdir}, @emph{prependdir}, @emph{limit_sl_dest} parameters and -s, -p, -e command line options.
Added the possibility to specify the option for an optimization level multiple times.
(Contributed by Lumír ‘Frenzy’ Balhar in bpo-38112@footnote{https://bugs.python.org/issue38112}.)

@node concurrent futures,curses<2>,compileall,Improved Modules<3>
@anchor{whatsnew/3 9 concurrent-futures}@anchor{3d8}
@subsubsection concurrent.futures


Added a new @emph{cancel_futures} parameter to
@ref{3d9,,concurrent.futures.Executor.shutdown()} that cancels all pending futures
which have not started running, instead of waiting for them to complete before
shutting down the executor.
(Contributed by Kyle Stanley in bpo-39349@footnote{https://bugs.python.org/issue39349}.)

Removed daemon threads from @ref{1c7,,ThreadPoolExecutor}
and @ref{3da,,ProcessPoolExecutor}. This improves
compatibility with subinterpreters and predictability in their shutdown
processes. (Contributed by Kyle Stanley in bpo-39812@footnote{https://bugs.python.org/issue39812}.)

Workers in @ref{3da,,ProcessPoolExecutor} are now spawned on
demand, only when there are no available idle workers to reuse. This optimizes
startup overhead and reduces the amount of lost CPU time to idle workers.
(Contributed by Kyle Stanley in bpo-39207@footnote{https://bugs.python.org/issue39207}.)

@node curses<2>,datetime,concurrent futures,Improved Modules<3>
@anchor{whatsnew/3 9 curses}@anchor{3db}
@subsubsection curses


Added @ref{3dc,,curses.get_escdelay()}, @ref{3dd,,curses.set_escdelay()},
@ref{3de,,curses.get_tabsize()}, and @ref{3df,,curses.set_tabsize()} functions.
(Contributed by Anthony Sottile in bpo-38312@footnote{https://bugs.python.org/issue38312}.)

@node datetime,distutils<2>,curses<2>,Improved Modules<3>
@anchor{whatsnew/3 9 datetime}@anchor{3e0}
@subsubsection datetime


The @ref{3e1,,isocalendar()} of @ref{3e2,,datetime.date}
and @ref{3e3,,isocalendar()} of @ref{363,,datetime.datetime}
methods now returns a @ref{3e4,,namedtuple()} instead of a @ref{3e5,,tuple}.
(Contributed by Dong-hee Na in bpo-24416@footnote{https://bugs.python.org/issue24416}.)

@node distutils<2>,fcntl<2>,datetime,Improved Modules<3>
@anchor{whatsnew/3 9 distutils}@anchor{3e6}
@subsubsection distutils


The @code{upload} command now creates SHA2-256 and Blake2b-256 hash
digests. It skips MD5 on platforms that block MD5 digest.
(Contributed by Christian Heimes in bpo-40698@footnote{https://bugs.python.org/issue40698}.)

@node fcntl<2>,ftplib,distutils<2>,Improved Modules<3>
@anchor{whatsnew/3 9 fcntl}@anchor{3e7}
@subsubsection fcntl


Added constants @code{F_OFD_GETLK}, @code{F_OFD_SETLK}
and @code{F_OFD_SETLKW}.
(Contributed by Dong-hee Na in bpo-38602@footnote{https://bugs.python.org/issue38602}.)

@node ftplib,gc<2>,fcntl<2>,Improved Modules<3>
@anchor{whatsnew/3 9 ftplib}@anchor{3e8}
@subsubsection ftplib


@ref{3e9,,FTP} and @ref{3ea,,FTP_TLS} now raise a @ref{17c,,ValueError}
if the given timeout for their constructor is zero to prevent the creation of
a non-blocking socket. (Contributed by Dong-hee Na in bpo-39259@footnote{https://bugs.python.org/issue39259}.)

@node gc<2>,hashlib<2>,ftplib,Improved Modules<3>
@anchor{whatsnew/3 9 gc}@anchor{3eb}
@subsubsection gc


When the garbage collector makes a collection in which some objects resurrect
(they are reachable from outside the isolated cycles after the finalizers have
been executed), do not block the collection of all objects that are still
unreachable. (Contributed by Pablo Galindo and Tim Peters in bpo-38379@footnote{https://bugs.python.org/issue38379}.)

Added a new function @ref{3ec,,gc.is_finalized()} to check if an object has been
finalized by the garbage collector. (Contributed by Pablo Galindo in
bpo-39322@footnote{https://bugs.python.org/issue39322}.)

@node hashlib<2>,http,gc<2>,Improved Modules<3>
@anchor{whatsnew/3 9 hashlib}@anchor{3ed}
@subsubsection hashlib


The @ref{88,,hashlib} module can now use SHA3 hashes and SHAKE XOF from OpenSSL
when available.
(Contributed by Christian Heimes in bpo-37630@footnote{https://bugs.python.org/issue37630}.)

Builtin hash modules can now be disabled with
@code{./configure --without-builtin-hashlib-hashes} or selectively enabled with
e.g. @code{./configure --with-builtin-hashlib-hashes=sha3,blake2} to force use
of OpenSSL based implementation.
(Contributed by Christian Heimes in bpo-40479@footnote{https://bugs.python.org/issue40479})

@node http,IDLE and idlelib<3>,hashlib<2>,Improved Modules<3>
@anchor{whatsnew/3 9 http}@anchor{3ee}
@subsubsection http


HTTP status codes @code{103 EARLY_HINTS}, @code{418 IM_A_TEAPOT} and @code{425 TOO_EARLY} are added to
@ref{3ef,,http.HTTPStatus}. (Contributed by Dong-hee Na in bpo-39509@footnote{https://bugs.python.org/issue39509} and Ross Rhodes in bpo-39507@footnote{https://bugs.python.org/issue39507}.)

@node IDLE and idlelib<3>,imaplib,http,Improved Modules<3>
@anchor{whatsnew/3 9 idle-and-idlelib}@anchor{3f0}
@subsubsection IDLE and idlelib


Added option to toggle cursor blink off.  (Contributed by Zackery Spytz
in bpo-4603@footnote{https://bugs.python.org/issue4603}.)

Escape key now closes IDLE completion windows.  (Contributed by Johnny
Najera in bpo-38944@footnote{https://bugs.python.org/issue38944}.)

Added keywords to module name completion list.  (Contributed by Terry J.
Reedy in bpo-37765@footnote{https://bugs.python.org/issue37765}.)

New in 3.9 maintenance releases

Make IDLE invoke @ref{28e,,sys.excepthook()} (when started without ‘-n’).
User hooks were previously ignored.  (Contributed by Ken Hilton in
bpo-43008@footnote{https://bugs.python.org/issue43008}.)

The changes above have been backported to 3.8 maintenance releases.

Rearrange the settings dialog.  Split the General tab into Windows
and Shell/Ed tabs.  Move help sources, which extend the Help menu, to the
Extensions tab.  Make space for new options and shorten the dialog. The
latter makes the dialog better fit small screens.  (Contributed by Terry Jan
Reedy in bpo-40468@footnote{https://bugs.python.org/issue40468}.)  Move the indent space setting from the Font tab to
the new Windows tab.  (Contributed by Mark Roseman and Terry Jan Reedy in
bpo-33962@footnote{https://bugs.python.org/issue33962}.)

Apply syntax highlighting to @cite{.pyi} files. (Contributed by Alex
Waygood and Terry Jan Reedy in bpo-45447@footnote{https://bugs.python.org/issue45447}.)

@node imaplib,importlib,IDLE and idlelib<3>,Improved Modules<3>
@anchor{whatsnew/3 9 imaplib}@anchor{3f1}
@subsubsection imaplib


@ref{3f2,,IMAP4} and @ref{3f3,,IMAP4_SSL} now have
an optional @emph{timeout} parameter for their constructors.
Also, the @ref{3f4,,open()} method now has an optional @emph{timeout} parameter
with this change. The overridden methods of @ref{3f3,,IMAP4_SSL} and
@ref{3f5,,IMAP4_stream} were applied to this change.
(Contributed by Dong-hee Na in bpo-38615@footnote{https://bugs.python.org/issue38615}.)

@ref{3f6,,imaplib.IMAP4.unselect()} is added.
@ref{3f6,,imaplib.IMAP4.unselect()} frees server’s resources associated with the
selected mailbox and returns the server to the authenticated
state. This command performs the same actions as @ref{3f7,,imaplib.IMAP4.close()}, except
that no messages are permanently removed from the currently
selected mailbox. (Contributed by Dong-hee Na in bpo-40375@footnote{https://bugs.python.org/issue40375}.)

@node importlib,inspect<3>,imaplib,Improved Modules<3>
@anchor{whatsnew/3 9 importlib}@anchor{3f8}
@subsubsection importlib


To improve consistency with import statements, @ref{3f9,,importlib.util.resolve_name()}
now raises @ref{3b8,,ImportError} instead of @ref{17c,,ValueError} for invalid relative
import attempts.
(Contributed by Ngalim Siregar in bpo-37444@footnote{https://bugs.python.org/issue37444}.)

Import loaders which publish immutable module objects can now publish
immutable packages in addition to individual modules.
(Contributed by Dino Viehland in bpo-39336@footnote{https://bugs.python.org/issue39336}.)

Added @ref{3fa,,importlib.resources.files()} function with support for
subdirectories in package data, matching backport in @code{importlib_resources}
version 1.5.
(Contributed by Jason R. Coombs in bpo-39791@footnote{https://bugs.python.org/issue39791}.)

Refreshed @code{importlib.metadata} from @code{importlib_metadata} version 1.6.1.

@node inspect<3>,ipaddress,importlib,Improved Modules<3>
@anchor{whatsnew/3 9 inspect}@anchor{3fb}
@subsubsection inspect


@ref{3fc,,inspect.BoundArguments.arguments} is changed from @code{OrderedDict} to regular
dict.  (Contributed by Inada Naoki in bpo-36350@footnote{https://bugs.python.org/issue36350} and bpo-39775@footnote{https://bugs.python.org/issue39775}.)

@node ipaddress,math<2>,inspect<3>,Improved Modules<3>
@anchor{whatsnew/3 9 ipaddress}@anchor{3fd}
@subsubsection ipaddress


@ref{9e,,ipaddress} now supports IPv6 Scoped Addresses (IPv6 address with suffix @code{%<scope_id>}).

Scoped IPv6 addresses can be parsed using @ref{3fe,,ipaddress.IPv6Address}.
If present, scope zone ID is available through the @ref{3ff,,scope_id} attribute.
(Contributed by Oleksandr Pavliuk in bpo-34788@footnote{https://bugs.python.org/issue34788}.)

Starting with Python 3.9.5 the @ref{9e,,ipaddress} module no longer
accepts any leading zeros in IPv4 address strings.
(Contributed by Christian Heimes in bpo-36384@footnote{https://bugs.python.org/issue36384}).

@node math<2>,multiprocessing,ipaddress,Improved Modules<3>
@anchor{whatsnew/3 9 math}@anchor{400}
@subsubsection math


Expanded the @ref{401,,math.gcd()} function to handle multiple arguments.
Formerly, it only supported two arguments.
(Contributed by Serhiy Storchaka in bpo-39648@footnote{https://bugs.python.org/issue39648}.)

Added @ref{402,,math.lcm()}: return the least common multiple of specified arguments.
(Contributed by Mark Dickinson, Ananthakrishnan and Serhiy Storchaka in
bpo-39479@footnote{https://bugs.python.org/issue39479} and bpo-39648@footnote{https://bugs.python.org/issue39648}.)

Added @ref{403,,math.nextafter()}: return the next floating-point value after @emph{x}
towards @emph{y}.
(Contributed by Victor Stinner in bpo-39288@footnote{https://bugs.python.org/issue39288}.)

Added @ref{404,,math.ulp()}: return the value of the least significant bit
of a float.
(Contributed by Victor Stinner in bpo-39310@footnote{https://bugs.python.org/issue39310}.)

@node multiprocessing,nntplib,math<2>,Improved Modules<3>
@anchor{whatsnew/3 9 multiprocessing}@anchor{405}
@subsubsection multiprocessing


The @ref{406,,multiprocessing.SimpleQueue} class has a new
@ref{407,,close()} method to explicitly close the
queue.
(Contributed by Victor Stinner in bpo-30966@footnote{https://bugs.python.org/issue30966}.)

@node nntplib,os<3>,multiprocessing,Improved Modules<3>
@anchor{whatsnew/3 9 nntplib}@anchor{408}
@subsubsection nntplib


@ref{409,,NNTP} and @ref{40a,,NNTP_SSL} now raise a @ref{17c,,ValueError}
if the given timeout for their constructor is zero to prevent the creation of
a non-blocking socket. (Contributed by Dong-hee Na in bpo-39259@footnote{https://bugs.python.org/issue39259}.)

@node os<3>,pathlib<2>,nntplib,Improved Modules<3>
@anchor{whatsnew/3 9 os}@anchor{40b}
@subsubsection os


Added @ref{40c,,CLD_KILLED} and @ref{40d,,CLD_STOPPED} for @code{si_code}.
(Contributed by Dong-hee Na in bpo-38493@footnote{https://bugs.python.org/issue38493}.)

Exposed the Linux-specific @ref{3a6,,os.pidfd_open()} (bpo-38692@footnote{https://bugs.python.org/issue38692}) and
@ref{40e,,os.P_PIDFD} (bpo-38713@footnote{https://bugs.python.org/issue38713}) for process management with file
descriptors.

The @ref{40f,,os.unsetenv()} function is now also available on Windows.
(Contributed by Victor Stinner in bpo-39413@footnote{https://bugs.python.org/issue39413}.)

The @ref{410,,os.putenv()} and @ref{40f,,os.unsetenv()} functions are now always
available.
(Contributed by Victor Stinner in bpo-39395@footnote{https://bugs.python.org/issue39395}.)

Added @ref{411,,os.waitstatus_to_exitcode()} function:
convert a wait status to an exit code.
(Contributed by Victor Stinner in bpo-40094@footnote{https://bugs.python.org/issue40094}.)

@node pathlib<2>,pdb,os<3>,Improved Modules<3>
@anchor{whatsnew/3 9 pathlib}@anchor{412}
@subsubsection pathlib


Added @ref{413,,pathlib.Path.readlink()} which acts similarly to
@ref{414,,os.readlink()}.
(Contributed by Girts Folkmanis in bpo-30618@footnote{https://bugs.python.org/issue30618})

@node pdb,poplib,pathlib<2>,Improved Modules<3>
@anchor{whatsnew/3 9 pdb}@anchor{415}
@subsubsection pdb


On Windows now @ref{416,,Pdb} supports @code{~/.pdbrc}.
(Contributed by Tim Hopper and Dan Lidral-Porter in bpo-20523@footnote{https://bugs.python.org/issue20523}.)

@node poplib,pprint<2>,pdb,Improved Modules<3>
@anchor{whatsnew/3 9 poplib}@anchor{417}
@subsubsection poplib


@ref{418,,POP3} and @ref{419,,POP3_SSL} now raise a @ref{17c,,ValueError}
if the given timeout for their constructor is zero to prevent the creation of
a non-blocking socket. (Contributed by Dong-hee Na in bpo-39259@footnote{https://bugs.python.org/issue39259}.)

@node pprint<2>,pydoc,poplib,Improved Modules<3>
@anchor{whatsnew/3 9 pprint}@anchor{41a}
@subsubsection pprint


@ref{cd,,pprint} can now pretty-print @ref{41b,,types.SimpleNamespace}.
(Contributed by Carl Bordum Hansen in bpo-37376@footnote{https://bugs.python.org/issue37376}.)

@node pydoc,random,pprint<2>,Improved Modules<3>
@anchor{whatsnew/3 9 pydoc}@anchor{41c}
@subsubsection pydoc


The documentation string is now shown not only for class, function,
method etc, but for any object that has its own @code{__doc__} attribute.
(Contributed by Serhiy Storchaka in bpo-40257@footnote{https://bugs.python.org/issue40257}.)

@node random,signal,pydoc,Improved Modules<3>
@anchor{whatsnew/3 9 random}@anchor{41d}
@subsubsection random


Added a new @code{random.Random.randbytes} method: generate random bytes.
(Contributed by Victor Stinner in bpo-40286@footnote{https://bugs.python.org/issue40286}.)

@node signal,smtplib,random,Improved Modules<3>
@anchor{whatsnew/3 9 signal}@anchor{41e}
@subsubsection signal


Exposed the Linux-specific @ref{41f,,signal.pidfd_send_signal()} for sending to
signals to a process using a file descriptor instead of a pid. (bpo-38712@footnote{https://bugs.python.org/issue38712})

@node smtplib,socket<3>,signal,Improved Modules<3>
@anchor{whatsnew/3 9 smtplib}@anchor{420}
@subsubsection smtplib


@ref{421,,SMTP} and @ref{422,,SMTP_SSL} now raise a @ref{17c,,ValueError}
if the given timeout for their constructor is zero to prevent the creation of
a non-blocking socket. (Contributed by Dong-hee Na in bpo-39259@footnote{https://bugs.python.org/issue39259}.)

@ref{423,,LMTP} constructor  now has an optional @emph{timeout} parameter.
(Contributed by Dong-hee Na in bpo-39329@footnote{https://bugs.python.org/issue39329}.)

@node socket<3>,time<2>,smtplib,Improved Modules<3>
@anchor{whatsnew/3 9 socket}@anchor{424}
@subsubsection socket


The @ref{ea,,socket} module now exports the @ref{425,,CAN_RAW_JOIN_FILTERS}
constant on Linux 4.1 and greater.
(Contributed by Stefan Tatschner and Zackery Spytz in bpo-25780@footnote{https://bugs.python.org/issue25780}.)

The socket module now supports the @ref{426,,CAN_J1939} protocol on
platforms that support it.  (Contributed by Karl Ding in bpo-40291@footnote{https://bugs.python.org/issue40291}.)

The socket module now has the @ref{427,,socket.send_fds()} and
@ref{428,,socket.recv_fds()} functions. (Contributed by Joannah Nanjekye, Shinya
Okano and Victor Stinner in bpo-28724@footnote{https://bugs.python.org/issue28724}.)

@node time<2>,sys<3>,socket<3>,Improved Modules<3>
@anchor{whatsnew/3 9 time}@anchor{429}
@subsubsection time


On AIX, @ref{42a,,thread_time()} is now implemented with @code{thread_cputime()}
which has nanosecond resolution, rather than
@code{clock_gettime(CLOCK_THREAD_CPUTIME_ID)} which has a resolution of 10 milliseconds.
(Contributed by Batuhan Taskaya in bpo-40192@footnote{https://bugs.python.org/issue40192})

@node sys<3>,tracemalloc,time<2>,Improved Modules<3>
@anchor{whatsnew/3 9 sys}@anchor{42b}
@subsubsection sys


Added a new @ref{42c,,sys.platlibdir} attribute: name of the platform-specific
library directory. It is used to build the path of standard library and the
paths of installed extension modules. It is equal to @code{"lib"} on most
platforms.  On Fedora and SuSE, it is equal to @code{"lib64"} on 64-bit platforms.
(Contributed by Jan Matějek, Matěj Cepl, Charalampos Stratakis and Victor Stinner in bpo-1294959@footnote{https://bugs.python.org/issue1294959}.)

Previously, @ref{42d,,sys.stderr} was block-buffered when non-interactive. Now
@code{stderr} defaults to always being line-buffered.
(Contributed by Jendrik Seipp in bpo-13601@footnote{https://bugs.python.org/issue13601}.)

@node tracemalloc,typing<2>,sys<3>,Improved Modules<3>
@anchor{whatsnew/3 9 tracemalloc}@anchor{42e}
@subsubsection tracemalloc


Added @ref{42f,,tracemalloc.reset_peak()} to set the peak size of traced memory
blocks to the current size, to measure the peak of specific pieces of code.
(Contributed by Huon Wilson in bpo-40630@footnote{https://bugs.python.org/issue40630}.)

@node typing<2>,unicodedata<2>,tracemalloc,Improved Modules<3>
@anchor{whatsnew/3 9 typing}@anchor{430}
@subsubsection typing


@geindex Python Enhancement Proposals; PEP 593
PEP 593@footnote{https://www.python.org/dev/peps/pep-0593} introduced an @ref{431,,typing.Annotated} type to decorate existing
types with context-specific metadata and new @code{include_extras} parameter to
@ref{432,,typing.get_type_hints()} to access the metadata at runtime. (Contributed
by Till Varoquaux and Konstantin Kashin.)

@node unicodedata<2>,venv,typing<2>,Improved Modules<3>
@anchor{whatsnew/3 9 unicodedata}@anchor{433}
@subsubsection unicodedata


The Unicode database has been updated to version 13.0.0. (bpo-39926@footnote{https://bugs.python.org/issue39926}).

@node venv,xml<2>,unicodedata<2>,Improved Modules<3>
@anchor{whatsnew/3 9 venv}@anchor{434}
@subsubsection venv


The activation scripts provided by @ref{12c,,venv} now all specify their prompt
customization consistently by always using the value specified by
@code{__VENV_PROMPT__}. Previously some scripts unconditionally used
@code{__VENV_PROMPT__}, others only if it happened to be set (which was the default
case), and one used @code{__VENV_NAME__} instead.
(Contributed by Brett Cannon in bpo-37663@footnote{https://bugs.python.org/issue37663}.)

@node xml<2>,,venv,Improved Modules<3>
@anchor{whatsnew/3 9 xml}@anchor{435}
@subsubsection xml


White space characters within attributes are now preserved when serializing
@ref{13e,,xml.etree.ElementTree} to XML file. EOLNs are no longer normalized
to “n”. This is the result of discussion about how to interpret
section 2.11 of XML spec.
(Contributed by Mefistotelis in bpo-39011@footnote{https://bugs.python.org/issue39011}.)

@node Optimizations<3>,Deprecated<5>,Improved Modules<3>,What’s New In Python 3 9
@anchor{whatsnew/3 9 optimizations}@anchor{436}
@subsection Optimizations



@itemize *

@item 
Optimized the idiom for assignment a temporary variable in comprehensions.
Now @code{for y in [expr]} in comprehensions is as fast as a simple assignment
@code{y = expr}.  For example:

@quotation

sums = [s for s in [0] for x in data for s in [s + x]]
@end quotation

Unlike the @code{:=} operator this idiom does not leak a variable to the
outer scope.

(Contributed by Serhiy Storchaka in bpo-32856@footnote{https://bugs.python.org/issue32856}.)

@item 
Optimized signal handling in multithreaded applications. If a thread different
than the main thread gets a signal, the bytecode evaluation loop is no longer
interrupted at each bytecode instruction to check for pending signals which
cannot be handled. Only the main thread of the main interpreter can handle
signals.

Previously, the bytecode evaluation loop was interrupted at each instruction
until the main thread handles signals.
(Contributed by Victor Stinner in bpo-40010@footnote{https://bugs.python.org/issue40010}.)

@item 
Optimized the @ref{f4,,subprocess} module on FreeBSD using @code{closefrom()}.
(Contributed by Ed Maste, Conrad Meyer, Kyle Evans, Kubilay Kocak and Victor
Stinner in bpo-38061@footnote{https://bugs.python.org/issue38061}.)

@item 
@ref{437,,PyLong_FromDouble()} is now up to 1.87x faster for values that
fit into long.
(Contributed by Sergey Fedoseev in bpo-37986@footnote{https://bugs.python.org/issue37986}.)

@item 
A number of Python builtins (@ref{438,,range}, @ref{3e5,,tuple}, @ref{370,,set},
@ref{439,,frozenset}, @ref{43a,,list}, @ref{3b0,,dict}) are now sped up by using
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590} vectorcall protocol.
(Contributed by Dong-hee Na, Mark Shannon, Jeroen Demeyer and Petr Viktorin in bpo-37207@footnote{https://bugs.python.org/issue37207}.)

@item 
Optimized @code{difference_update()} for the case when the other set
is much larger than the base set.
(Suggested by Evgeny Kapun with code contributed by Michele Orrù in bpo-8425@footnote{https://bugs.python.org/issue8425}.)

@item 
Python’s small object allocator (@code{obmalloc.c}) now allows (no more than)
one empty arena to remain available for immediate reuse, without returning
it to the OS.  This prevents thrashing in simple loops where an arena could
be created and destroyed anew on each iteration.
(Contributed by Tim Peters in bpo-37257@footnote{https://bugs.python.org/issue37257}.)

@item 
@ref{43b,,floor division} of float operation now has a better performance. Also
the message of @ref{43c,,ZeroDivisionError} for this operation is updated.
(Contributed by Dong-hee Na in bpo-39434@footnote{https://bugs.python.org/issue39434}.)

@item 
Decoding short ASCII strings with UTF-8 and ascii codecs is now about
15% faster.  (Contributed by Inada Naoki in bpo-37348@footnote{https://bugs.python.org/issue37348}.)
@end itemize

Here’s a summary of performance improvements from Python 3.4 through Python 3.9:

@example
Python version                       3.4     3.5     3.6     3.7     3.8    3.9
--------------                       ---     ---     ---     ---     ---    ---

Variable and attribute read access:
    read_local                       7.1     7.1     5.4     5.1     3.9    3.9
    read_nonlocal                    7.1     8.1     5.8     5.4     4.4    4.5
    read_global                     15.5    19.0    14.3    13.6     7.6    7.8
    read_builtin                    21.1    21.6    18.5    19.0     7.5    7.8
    read_classvar_from_class        25.6    26.5    20.7    19.5    18.4   17.9
    read_classvar_from_instance     22.8    23.5    18.8    17.1    16.4   16.9
    read_instancevar                32.4    33.1    28.0    26.3    25.4   25.3
    read_instancevar_slots          27.8    31.3    20.8    20.8    20.2   20.5
    read_namedtuple                 73.8    57.5    45.0    46.8    18.4   18.7
    read_boundmethod                37.6    37.9    29.6    26.9    27.7   41.1

Variable and attribute write access:
    write_local                      8.7     9.3     5.5     5.3     4.3    4.3
    write_nonlocal                  10.5    11.1     5.6     5.5     4.7    4.8
    write_global                    19.7    21.2    18.0    18.0    15.8   16.7
    write_classvar                  92.9    96.0   104.6   102.1    39.2   39.8
    write_instancevar               44.6    45.8    40.0    38.9    35.5   37.4
    write_instancevar_slots         35.6    36.1    27.3    26.6    25.7   25.8

Data structure read access:
    read_list                       24.2    24.5    20.8    20.8    19.0   19.5
    read_deque                      24.7    25.5    20.2    20.6    19.8   20.2
    read_dict                       24.3    25.7    22.3    23.0    21.0   22.4
    read_strdict                    22.6    24.3    19.5    21.2    18.9   21.5

Data structure write access:
    write_list                      27.1    28.5    22.5    21.6    20.0   20.0
    write_deque                     28.7    30.1    22.7    21.8    23.5   21.7
    write_dict                      31.4    33.3    29.3    29.2    24.7   25.4
    write_strdict                   28.4    29.9    27.5    25.2    23.1   24.5

Stack (or queue) operations:
    list_append_pop                 93.4   112.7    75.4    74.2    50.8   50.6
    deque_append_pop                43.5    57.0    49.4    49.2    42.5   44.2
    deque_append_popleft            43.7    57.3    49.7    49.7    42.8   46.4

Timing loop:
    loop_overhead                    0.5     0.6     0.4     0.3     0.3    0.3
@end example

These results were generated from the variable access benchmark script at:
@code{Tools/scripts/var_access_benchmark.py}. The benchmark script displays timings
in nanoseconds.  The benchmarks were measured on an
Intel® Core™ i7-4960HQ processor@footnote{https://ark.intel.com/content/www/us/en/ark/products/76088/intel-core-i7-4960hq-processor-6m-cache-up-to-3-80-ghz.html}
running the macOS 64-bit builds found at
python.org@footnote{https://www.python.org/downloads/mac-osx/}.

@node Deprecated<5>,Removed<5>,Optimizations<3>,What’s New In Python 3 9
@anchor{whatsnew/3 9 deprecated}@anchor{43d}
@subsection Deprecated



@itemize *

@item 
The distutils @code{bdist_msi} command is now deprecated, use
@code{bdist_wheel} (wheel packages) instead.
(Contributed by Hugo van Kemenade in bpo-39586@footnote{https://bugs.python.org/issue39586}.)

@item 
Currently @ref{43e,,math.factorial()} accepts @ref{256,,float} instances with
non-negative integer values (like @code{5.0}).  It raises a @ref{17c,,ValueError}
for non-integral and negative floats.  It is now deprecated.  In future
Python versions it will raise a @ref{162,,TypeError} for all floats.
(Contributed by Serhiy Storchaka in bpo-37315@footnote{https://bugs.python.org/issue37315}.)

@item 
The @code{parser} and @code{symbol} modules are deprecated and will be
removed in future versions of Python. For the majority of use cases,
users can leverage the Abstract Syntax Tree (AST) generation and compilation
stage, using the @ref{7,,ast} module.

@item 
The Public C API functions @code{PyParser_SimpleParseStringFlags()},
@code{PyParser_SimpleParseStringFlagsFilename()},
@code{PyParser_SimpleParseFileFlags()} and @code{PyNode_Compile()}
are deprecated and will be removed in Python 3.10 together with the old parser.

@item 
Using @ref{24c,,NotImplemented} in a boolean context has been deprecated,
as it is almost exclusively the result of incorrect rich comparator
implementations. It will be made a @ref{162,,TypeError} in a future version
of Python.
(Contributed by Josh Rosenberg in bpo-35712@footnote{https://bugs.python.org/issue35712}.)

@item 
The @ref{d7,,random} module currently accepts any hashable type as a
possible seed value.  Unfortunately, some of those types are not
guaranteed to have a deterministic hash value.  After Python 3.9,
the module will restrict its seeds to @ref{182,,None}, @ref{1b1,,int},
@ref{256,,float}, @ref{2ef,,str}, @ref{2f0,,bytes}, and @ref{2f1,,bytearray}.

@item 
Opening the @ref{43f,,GzipFile} file for writing without specifying
the @emph{mode} argument is deprecated.  In future Python versions it will always
be opened for reading by default.  Specify the @emph{mode} argument for opening
it for writing and silencing a warning.
(Contributed by Serhiy Storchaka in bpo-28286@footnote{https://bugs.python.org/issue28286}.)

@item 
Deprecated the @code{split()} method of @code{_tkinter.TkappType} in
favour of the @code{splitlist()} method which has more consistent and
predicable behavior.
(Contributed by Serhiy Storchaka in bpo-38371@footnote{https://bugs.python.org/issue38371}.)

@item 
The explicit passing of coroutine objects to @ref{440,,asyncio.wait()} has been
deprecated and will be removed in version 3.11.
(Contributed by Yury Selivanov and Kyle Stanley in bpo-34790@footnote{https://bugs.python.org/issue34790}.)

@item 
binhex4 and hexbin4 standards are now deprecated. The @code{binhex} module
and the following @ref{f,,binascii} functions are now deprecated:


@itemize *

@item 
@code{b2a_hqx()}, @code{a2b_hqx()}

@item 
@code{rlecode_hqx()}, @code{rledecode_hqx()}
@end itemize

(Contributed by Victor Stinner in bpo-39353@footnote{https://bugs.python.org/issue39353}.)

@item 
@ref{7,,ast} classes @code{slice}, @code{Index} and @code{ExtSlice} are considered deprecated
and will be removed in future Python versions.  @code{value} itself should be
used instead of @code{Index(value)}.  @code{Tuple(slices, Load())} should be
used instead of @code{ExtSlice(slices)}.
(Contributed by Serhiy Storchaka in bpo-34822@footnote{https://bugs.python.org/issue34822}.)

@item 
@ref{7,,ast} classes @code{Suite}, @code{Param}, @code{AugLoad} and @code{AugStore}
are considered deprecated and will be removed in future Python versions.
They were not generated by the parser and not accepted by the code
generator in Python 3.
(Contributed by Batuhan Taskaya in bpo-39639@footnote{https://bugs.python.org/issue39639} and bpo-39969@footnote{https://bugs.python.org/issue39969}
and Serhiy Storchaka in bpo-39988@footnote{https://bugs.python.org/issue39988}.)

@item 
The @ref{441,,PyEval_InitThreads()} and @ref{442,,PyEval_ThreadsInitialized()}
functions are now deprecated and will be removed in Python 3.11. Calling
@ref{441,,PyEval_InitThreads()} now does nothing. The @ref{385,,GIL} is initialized
by @ref{38d,,Py_Initialize()} since Python 3.7.
(Contributed by Victor Stinner in bpo-39877@footnote{https://bugs.python.org/issue39877}.)

@item 
Passing @code{None} as the first argument to the @ref{443,,shlex.split()} function
has been deprecated.  (Contributed by Zackery Spytz in bpo-33262@footnote{https://bugs.python.org/issue33262}.)

@item 
@code{smtpd.MailmanProxy()} is now deprecated as it is unusable without
an external module, @code{mailman}.  (Contributed by Samuel Colvin in bpo-35800@footnote{https://bugs.python.org/issue35800}.)

@item 
The @ref{a3,,lib2to3} module now emits a @ref{3a9,,PendingDeprecationWarning}.
Python 3.9 switched to a PEG parser (see 
@geindex Python Enhancement Proposals; PEP 617
PEP 617@footnote{https://www.python.org/dev/peps/pep-0617}), and Python 3.10 may
include new language syntax that is not parsable by lib2to3’s LL(1) parser.
The @code{lib2to3} module may be removed from the standard library in a future
Python version. Consider third-party alternatives such as LibCST@footnote{https://libcst.readthedocs.io/} or
parso@footnote{https://parso.readthedocs.io/}.
(Contributed by Carl Meyer in bpo-40360@footnote{https://bugs.python.org/issue40360}.)

@item 
The @emph{random} parameter of @ref{444,,random.shuffle()} has been deprecated.
(Contributed by Raymond Hettinger in bpo-40465@footnote{https://bugs.python.org/issue40465})
@end itemize

@node Removed<5>,Porting to Python 3 9,Deprecated<5>,What’s New In Python 3 9
@anchor{whatsnew/3 9 removed}@anchor{445}@anchor{whatsnew/3 9 removed-in-python-39}@anchor{3ad}
@subsection Removed



@itemize *

@item 
The erroneous version at @code{unittest.mock.__version__} has been removed.

@item 
@ref{409,,nntplib.NNTP}: @code{xpath()} and @code{xgtitle()} methods have been removed.
These methods are deprecated since Python 3.3. Generally, these extensions
are not supported or not enabled by NNTP server administrators.
For @code{xgtitle()}, please use @ref{446,,nntplib.NNTP.descriptions()} or
@ref{447,,nntplib.NNTP.description()} instead.
(Contributed by Dong-hee Na in bpo-39366@footnote{https://bugs.python.org/issue39366}.)

@item 
@ref{25e,,array.array}: @code{tostring()} and @code{fromstring()} methods have been
removed. They were aliases to @code{tobytes()} and @code{frombytes()}, deprecated
since Python 3.2.
(Contributed by Victor Stinner in bpo-38916@footnote{https://bugs.python.org/issue38916}.)

@item 
The undocumented @code{sys.callstats()} function has been removed. Since Python
3.7, it was deprecated and always returned @ref{182,,None}. It required a special
build option @code{CALL_PROFILE} which was already removed in Python 3.7.
(Contributed by Victor Stinner in bpo-37414@footnote{https://bugs.python.org/issue37414}.)

@item 
The @code{sys.getcheckinterval()} and @code{sys.setcheckinterval()} functions have
been removed. They were deprecated since Python 3.2. Use
@ref{448,,sys.getswitchinterval()} and @ref{449,,sys.setswitchinterval()} instead.
(Contributed by Victor Stinner in bpo-37392@footnote{https://bugs.python.org/issue37392}.)

@item 
The C function @code{PyImport_Cleanup()} has been removed. It was documented as:
“Empty the module table.  For internal use only.”
(Contributed by Victor Stinner in bpo-36710@footnote{https://bugs.python.org/issue36710}.)

@item 
@code{_dummy_thread} and @code{dummy_threading} modules have been removed. These
modules were deprecated since Python 3.7 which requires threading support.
(Contributed by Victor Stinner in bpo-37312@footnote{https://bugs.python.org/issue37312}.)

@item 
@code{aifc.openfp()} alias to @code{aifc.open()}, @code{sunau.openfp()} alias to
@code{sunau.open()}, and @code{wave.openfp()} alias to @ref{44a,,wave.open()} have been
removed. They were deprecated since Python 3.7.
(Contributed by Victor Stinner in bpo-37320@footnote{https://bugs.python.org/issue37320}.)

@item 
The @code{isAlive()} method of @ref{44b,,threading.Thread}
has been removed. It was deprecated since Python 3.8.
Use @ref{44c,,is_alive()} instead.
(Contributed by Dong-hee Na in bpo-37804@footnote{https://bugs.python.org/issue37804}.)

@item 
Methods @code{getchildren()} and @code{getiterator()} of classes
@ref{44d,,ElementTree} and
@ref{44e,,Element} in the @ref{13e,,ElementTree}
module have been removed.  They were deprecated in Python 3.2.
Use @code{iter(x)} or @code{list(x)} instead of @code{x.getchildren()} and
@code{x.iter()} or @code{list(x.iter())} instead of @code{x.getiterator()}.
(Contributed by Serhiy Storchaka in bpo-36543@footnote{https://bugs.python.org/issue36543}.)

@item 
The old @ref{ca,,plistlib} API has been removed, it was deprecated since Python
3.4. Use the @ref{44f,,load()}, @ref{450,,loads()}, @ref{451,,dump()}, and
@ref{452,,dumps()} functions. Additionally, the @emph{use_builtin_types} parameter was
removed, standard @ref{2f0,,bytes} objects are always used instead.
(Contributed by Jon Janzen in bpo-36409@footnote{https://bugs.python.org/issue36409}.)

@item 
The C function @code{PyGen_NeedsFinalizing} has been removed. It was not
documented, tested, or used anywhere within CPython after the implementation
of 
@geindex Python Enhancement Proposals; PEP 442
PEP 442@footnote{https://www.python.org/dev/peps/pep-0442}. Patch by Joannah Nanjekye.
(Contributed by Joannah Nanjekye in bpo-15088@footnote{https://bugs.python.org/issue15088})

@item 
@code{base64.encodestring()} and @code{base64.decodestring()}, aliases deprecated
since Python 3.1, have been removed: use @ref{453,,base64.encodebytes()} and
@ref{454,,base64.decodebytes()} instead.
(Contributed by Victor Stinner in bpo-39351@footnote{https://bugs.python.org/issue39351}.)

@item 
@code{fractions.gcd()} function has been removed, it was deprecated since Python
3.5 (bpo-22486@footnote{https://bugs.python.org/issue22486}): use @ref{401,,math.gcd()} instead.
(Contributed by Victor Stinner in bpo-39350@footnote{https://bugs.python.org/issue39350}.)

@item 
The @emph{buffering} parameter of @ref{455,,bz2.BZ2File} has been removed. Since
Python 3.0, it was ignored and using it emitted a @ref{1b2,,DeprecationWarning}.
Pass an open file object to control how the file is opened.
(Contributed by Victor Stinner in bpo-39357@footnote{https://bugs.python.org/issue39357}.)

@item 
The @emph{encoding} parameter of @ref{456,,json.loads()} has been removed.
As of Python 3.1, it was deprecated and ignored; using it has emitted a
@ref{1b2,,DeprecationWarning} since Python 3.8.
(Contributed by Inada Naoki in bpo-39377@footnote{https://bugs.python.org/issue39377})

@item 
@code{with (await asyncio.lock):} and @code{with (yield from asyncio.lock):} statements are
not longer supported, use @code{async with lock} instead.  The same is correct for
@code{asyncio.Condition} and @code{asyncio.Semaphore}.
(Contributed by Andrew Svetlov in bpo-34793@footnote{https://bugs.python.org/issue34793}.)

@item 
The @code{sys.getcounts()} function, the @code{-X showalloccount} command line
option and the @code{show_alloc_count} field of the C structure
@ref{206,,PyConfig} have been removed. They required a special Python build by
defining @code{COUNT_ALLOCS} macro.
(Contributed by Victor Stinner in bpo-39489@footnote{https://bugs.python.org/issue39489}.)

@item 
The @code{_field_types} attribute of the @ref{457,,typing.NamedTuple} class
has been removed.  It was deprecated since Python 3.8.  Use
the @code{__annotations__} attribute instead.
(Contributed by Serhiy Storchaka in bpo-40182@footnote{https://bugs.python.org/issue40182}.)

@item 
The @code{symtable.SymbolTable.has_exec()} method has been removed. It was
deprecated since 2006, and only returning @code{False} when it’s called.
(Contributed by Batuhan Taskaya in bpo-40208@footnote{https://bugs.python.org/issue40208})

@item 
The @code{asyncio.Task.current_task()} and @code{asyncio.Task.all_tasks()}
have been removed. They were deprecated since Python 3.7 and you can use
@ref{458,,asyncio.current_task()} and @ref{459,,asyncio.all_tasks()} instead.
(Contributed by Rémi Lapeyre in bpo-40967@footnote{https://bugs.python.org/issue40967})

@item 
The @code{unescape()} method in the @ref{33c,,html.parser.HTMLParser} class
has been removed (it was deprecated since Python 3.4).  @ref{45a,,html.unescape()}
should be used for converting character references to the corresponding
unicode characters.
@end itemize

@node Porting to Python 3 9,Build Changes<3>,Removed<5>,What’s New In Python 3 9
@anchor{whatsnew/3 9 porting-to-python-3-9}@anchor{45b}
@subsection Porting to Python 3.9


This section lists previously described changes and other bugfixes
that may require changes to your code.

@menu
* Changes in the Python API: Changes in the Python API<3>. 
* Changes in the C API: Changes in the C API<2>. 
* CPython bytecode changes: CPython bytecode changes<3>. 

@end menu

@node Changes in the Python API<3>,Changes in the C API<2>,,Porting to Python 3 9
@anchor{whatsnew/3 9 changes-in-the-python-api}@anchor{45c}
@subsubsection Changes in the Python API



@itemize *

@item 
@ref{3b7,,__import__()} and @ref{3f9,,importlib.util.resolve_name()} now raise
@ref{3b8,,ImportError} where it previously raised @ref{17c,,ValueError}. Callers
catching the specific exception type and supporting both Python 3.9 and
earlier versions will need to catch both using @code{except (ImportError, ValueError):}.

@item 
The @ref{12c,,venv} activation scripts no longer special-case when
@code{__VENV_PROMPT__} is set to @code{""}.

@item 
The @ref{45d,,select.epoll.unregister()} method no longer ignores the
@ref{45e,,EBADF} error.
(Contributed by Victor Stinner in bpo-39239@footnote{https://bugs.python.org/issue39239}.)

@item 
The @emph{compresslevel} parameter of @ref{455,,bz2.BZ2File} became keyword-only,
since the @emph{buffering} parameter has been removed.
(Contributed by Victor Stinner in bpo-39357@footnote{https://bugs.python.org/issue39357}.)

@item 
Simplified AST for subscription. Simple indices will be represented by
their value, extended slices will be represented as tuples.
@code{Index(value)} will return a @code{value} itself, @code{ExtSlice(slices)}
will return @code{Tuple(slices, Load())}.
(Contributed by Serhiy Storchaka in bpo-34822@footnote{https://bugs.python.org/issue34822}.)

@item 
The @ref{96,,importlib} module now ignores the 
@geindex PYTHONCASEOK
@geindex environment variable; PYTHONCASEOK
@ref{45f,,PYTHONCASEOK}
environment variable when the @ref{460,,-E} or @ref{461,,-I} command line
options are being used.

@item 
The @emph{encoding} parameter has been added to the classes @ref{3e9,,ftplib.FTP} and
@ref{3ea,,ftplib.FTP_TLS} as a keyword-only parameter, and the default encoding
is changed from Latin-1 to UTF-8 to follow 
@geindex RFC; RFC 2640
RFC 2640@footnote{https://tools.ietf.org/html/rfc2640.html}.

@item 
@ref{3d0,,asyncio.loop.shutdown_default_executor()} has been added to
@ref{462,,AbstractEventLoop}, meaning alternative event loops that
inherit from it should have this method defined.
(Contributed by Kyle Stanley in bpo-34037@footnote{https://bugs.python.org/issue34037}.)

@item 
The constant values of future flags in the @ref{0,,__future__} module
is updated in order to prevent collision with compiler flags. Previously
@code{PyCF_ALLOW_TOP_LEVEL_AWAIT} was clashing with @code{CO_FUTURE_DIVISION}.
(Contributed by Batuhan Taskaya in bpo-39562@footnote{https://bugs.python.org/issue39562})

@item 
@code{array('u')} now uses @code{wchar_t} as C type instead of @code{Py_UNICODE}.
This change doesn’t affect to its behavior because @code{Py_UNICODE} is alias
of @code{wchar_t} since Python 3.3.
(Contributed by Inada Naoki in bpo-34538@footnote{https://bugs.python.org/issue34538}.)

@item 
The @ref{463,,logging.getLogger()} API now returns the root logger when passed
the name @code{'root'}, whereas previously it returned a non-root logger named
@code{'root'}. This could affect cases where user code explicitly wants a
non-root logger named @code{'root'}, or instantiates a logger using
@code{logging.getLogger(__name__)} in some top-level module called @code{'root.py'}.
(Contributed by Vinay Sajip in bpo-37742@footnote{https://bugs.python.org/issue37742}.)

@item 
Division handling of @ref{1c4,,PurePath} now returns @code{NotImplemented}
instead of raising a @ref{162,,TypeError} when passed something other than an
instance of @code{str} or @ref{1c4,,PurePath}.  This allows creating
compatible classes that don’t inherit from those mentioned types.
(Contributed by Roger Aiudi in bpo-34775@footnote{https://bugs.python.org/issue34775}).

@item 
Starting with Python 3.9.5 the @ref{9e,,ipaddress} module no longer
accepts any leading zeros in IPv4 address strings. Leading zeros are
ambiguous and interpreted as octal notation by some libraries. For example
the legacy function @ref{464,,socket.inet_aton()} treats leading zeros as octal
notatation. glibc implementation of modern @ref{465,,inet_pton()} does
not accept any leading zeros.
(Contributed by Christian Heimes in bpo-36384@footnote{https://bugs.python.org/issue36384}).

@item 
@ref{466,,codecs.lookup()} now normalizes the encoding name the same way as
@code{encodings.normalize_encoding()}, except that @ref{466,,codecs.lookup()} also
converts the name to lower case. For example, @code{"latex+latin1"} encoding
name is now normalized to @code{"latex_latin1"}.
(Contributed by Jordon Xu in bpo-37751@footnote{https://bugs.python.org/issue37751}.)
@end itemize

@node Changes in the C API<2>,CPython bytecode changes<3>,Changes in the Python API<3>,Porting to Python 3 9
@anchor{whatsnew/3 9 changes-in-the-c-api}@anchor{467}
@subsubsection Changes in the C API



@itemize *

@item 
Instances of @ref{468,,heap-allocated types} (such as those created with
@ref{469,,PyType_FromSpec()} and similar APIs) hold a reference to their type
object since Python 3.8. As indicated in the “Changes in the C API” of Python
3.8, for the vast majority of cases, there should be no side effect but for
types that have a custom @ref{1ee,,tp_traverse} function,
ensure that all custom @code{tp_traverse} functions of heap-allocated types
visit the object’s type.

@quotation

Example:

@example
int
foo_traverse(foo_struct *self, visitproc visit, void *arg) @{
// Rest of the traverse function
#if PY_VERSION_HEX >= 0x03090000
    // This was not needed before Python 3.9 (Python issue 35810 and 40217)
    Py_VISIT(Py_TYPE(self));
#endif
@}
@end example
@end quotation

If your traverse function delegates to @code{tp_traverse} of its base class
(or another type), ensure that @code{Py_TYPE(self)} is visited only once.
Note that only @ref{468,,heap type} are expected to visit the type
in @code{tp_traverse}.

@quotation

For example, if your @code{tp_traverse} function includes:

@example
base->tp_traverse(self, visit, arg)
@end example

then add:

@example
#if PY_VERSION_HEX >= 0x03090000
    // This was not needed before Python 3.9 (bpo-35810 and bpo-40217)
    if (base->tp_flags & Py_TPFLAGS_HEAPTYPE) @{
        // a heap type's tp_traverse already visited Py_TYPE(self)
    @} else @{
        Py_VISIT(Py_TYPE(self));
    @}
#else
@end example
@end quotation

(See bpo-35810@footnote{https://bugs.python.org/issue35810} and bpo-40217@footnote{https://bugs.python.org/issue40217} for more information.)

@item 
The functions @code{PyEval_CallObject}, @code{PyEval_CallFunction},
@code{PyEval_CallMethod} and @code{PyEval_CallObjectWithKeywords} are deprecated.
Use @ref{46a,,PyObject_Call()} and its variants instead.
(See more details in bpo-29548@footnote{https://bugs.python.org/issue29548}.)
@end itemize

@node CPython bytecode changes<3>,,Changes in the C API<2>,Porting to Python 3 9
@anchor{whatsnew/3 9 cpython-bytecode-changes}@anchor{46b}
@subsubsection CPython bytecode changes



@itemize *

@item 
The @ref{46c,,LOAD_ASSERTION_ERROR} opcode was added for handling the
@ref{46d,,assert} statement. Previously, the assert statement would not work
correctly if the @ref{46e,,AssertionError} exception was being shadowed.
(Contributed by Zackery Spytz in bpo-34880@footnote{https://bugs.python.org/issue34880}.)

@item 
The @ref{46f,,COMPARE_OP} opcode was split into four distinct instructions:


@itemize *

@item 
@code{COMPARE_OP} for rich comparisons

@item 
@code{IS_OP} for ‘is’ and ‘is not’ tests

@item 
@code{CONTAINS_OP} for ‘in’ and ‘not in’ tests

@item 
@code{JUMP_IF_NOT_EXC_MATCH} for checking exceptions in ‘try-except’
statements.
@end itemize

(Contributed by Mark Shannon in bpo-39156@footnote{https://bugs.python.org/issue39156}.)
@end itemize

@node Build Changes<3>,C API Changes<3>,Porting to Python 3 9,What’s New In Python 3 9
@anchor{whatsnew/3 9 build-changes}@anchor{470}
@subsection Build Changes



@itemize *

@item 
Added @code{--with-platlibdir} option to the @code{configure} script: name of the
platform-specific library directory, stored in the new @ref{42c,,sys.platlibdir}
attribute. See @ref{42c,,sys.platlibdir} attribute for more information.
(Contributed by Jan Matějek, Matěj Cepl, Charalampos Stratakis
and Victor Stinner in bpo-1294959@footnote{https://bugs.python.org/issue1294959}.)

@item 
The @code{COUNT_ALLOCS} special build macro has been removed.
(Contributed by Victor Stinner in bpo-39489@footnote{https://bugs.python.org/issue39489}.)

@item 
On non-Windows platforms, the @code{setenv()} and @code{unsetenv()}
functions are now required to build Python.
(Contributed by Victor Stinner in bpo-39395@footnote{https://bugs.python.org/issue39395}.)

@item 
On non-Windows platforms, creating @code{bdist_wininst} installers is now
officially unsupported.  (See bpo-10945@footnote{https://bugs.python.org/issue10945} for more details.)

@item 
When building Python on macOS from source, @code{_tkinter} now links with
non-system Tcl and Tk frameworks if they are installed in
@code{/Library/Frameworks}, as had been the case on older releases
of macOS. If a macOS SDK is explicitly configured, by using
@ref{471,,--enable-universalsdk} or @code{-isysroot}, only the SDK itself is
searched. The default behavior can still be overridden with
@ref{358,,--with-tcltk-includes} and @ref{359,,--with-tcltk-libs}.
(Contributed by Ned Deily in bpo-34956@footnote{https://bugs.python.org/issue34956}.)

@item 
Python can now be built for Windows 10 ARM64.
(Contributed by Steve Dower in bpo-33125@footnote{https://bugs.python.org/issue33125}.)

@item 
Some individual tests are now skipped when @code{--pgo} is used.  The tests
in question increased the PGO task time significantly and likely
didn’t help improve optimization of the final executable. This
speeds up the task by a factor of about 15x.  Running the full unit test
suite is slow.  This change may result in a slightly less optimized build
since not as many code branches will be executed.  If you are willing to
wait for the much slower build, the old behavior can be restored using
@code{./configure [..] PROFILE_TASK="-m test --pgo-extended"}.  We make no
guarantees as to which PGO task set produces a faster build.  Users who care
should run their own relevant benchmarks as results can depend on the
environment, workload, and compiler tool chain.
(See bpo-36044@footnote{https://bugs.python.org/issue36044} and bpo-37707@footnote{https://bugs.python.org/issue37707} for more details.)
@end itemize

@node C API Changes<3>,Notable changes in Python 3 9 1,Build Changes<3>,What’s New In Python 3 9
@anchor{whatsnew/3 9 c-api-changes}@anchor{472}
@subsection C API Changes


@menu
* New Features: New Features<6>. 
* Porting to Python 3.9: Porting to Python 3 9<2>. 
* Removed: Removed<6>. 

@end menu

@node New Features<6>,Porting to Python 3 9<2>,,C API Changes<3>
@anchor{whatsnew/3 9 id1}@anchor{473}
@subsubsection New Features



@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 573
PEP 573@footnote{https://www.python.org/dev/peps/pep-0573}: Added @ref{36d,,PyType_FromModuleAndSpec()} to associate
a module with a class; @ref{474,,PyType_GetModule()} and
@ref{475,,PyType_GetModuleState()} to retrieve the module and its state; and
@ref{1ea,,PyCMethod} and @code{METH_METHOD} to allow a method to
access the class it was defined in.
(Contributed by Marcel Plch and Petr Viktorin in bpo-38787@footnote{https://bugs.python.org/issue38787}.)

@item 
Added @ref{1f8,,PyFrame_GetCode()} function: get a frame code.
Added @ref{1f7,,PyFrame_GetBack()} function: get the frame next outer frame.
(Contributed by Victor Stinner in bpo-40421@footnote{https://bugs.python.org/issue40421}.)

@item 
Added @ref{1f9,,PyFrame_GetLineNumber()} to the limited C API.
(Contributed by Victor Stinner in bpo-40421@footnote{https://bugs.python.org/issue40421}.)

@item 
Added @ref{476,,PyThreadState_GetInterpreter()} and
@ref{477,,PyInterpreterState_Get()} functions to get the interpreter.
Added @ref{1fb,,PyThreadState_GetFrame()} function to get the current frame of a
Python thread state.
Added @ref{478,,PyThreadState_GetID()} function: get the unique identifier of a
Python thread state.
(Contributed by Victor Stinner in bpo-39947@footnote{https://bugs.python.org/issue39947}.)

@item 
Added a new public @ref{479,,PyObject_CallNoArgs()} function to the C API, which
calls a callable Python object without any arguments. It is the most efficient
way to call a callable Python object without any argument.
(Contributed by Victor Stinner in bpo-37194@footnote{https://bugs.python.org/issue37194}.)

@item 
Changes in the limited C API (if @code{Py_LIMITED_API} macro is defined):


@itemize *

@item 
Provide @ref{47a,,Py_EnterRecursiveCall()} and @ref{47b,,Py_LeaveRecursiveCall()}
as regular functions for the limited API. Previously, there were defined as
macros, but these macros didn’t compile with the limited C API which cannot
access @code{PyThreadState.recursion_depth} field (the structure is opaque in
the limited C API).

@item 
@code{PyObject_INIT()} and @code{PyObject_INIT_VAR()} become regular “opaque”
function to hide implementation details.
@end itemize

(Contributed by Victor Stinner in bpo-38644@footnote{https://bugs.python.org/issue38644} and bpo-39542@footnote{https://bugs.python.org/issue39542}.)

@item 
The @ref{47c,,PyModule_AddType()} function is added to help adding a type
to a module.
(Contributed by Dong-hee Na in bpo-40024@footnote{https://bugs.python.org/issue40024}.)

@item 
Added the functions @ref{47d,,PyObject_GC_IsTracked()} and
@ref{47e,,PyObject_GC_IsFinalized()} to the public API to allow to query if
Python objects are being currently tracked or have been already finalized by
the garbage collector respectively.
(Contributed by Pablo Galindo Salgado in bpo-40241@footnote{https://bugs.python.org/issue40241}.)

@item 
Added @code{_PyObject_FunctionStr()} to get a user-friendly string
representation of a function-like object.
(Patch by Jeroen Demeyer in bpo-37645@footnote{https://bugs.python.org/issue37645}.)

@item 
Added @ref{47f,,PyObject_CallOneArg()} for calling an object with one
positional argument
(Patch by Jeroen Demeyer in bpo-37483@footnote{https://bugs.python.org/issue37483}.)
@end itemize

@node Porting to Python 3 9<2>,Removed<6>,New Features<6>,C API Changes<3>
@anchor{whatsnew/3 9 id2}@anchor{480}
@subsubsection Porting to Python 3.9



@itemize *

@item 
@code{PyInterpreterState.eval_frame} (
@geindex Python Enhancement Proposals; PEP 523
PEP 523@footnote{https://www.python.org/dev/peps/pep-0523}) now requires a new mandatory
@emph{tstate} parameter (@code{PyThreadState*}).
(Contributed by Victor Stinner in bpo-38500@footnote{https://bugs.python.org/issue38500}.)

@item 
Extension modules: @ref{481,,m_traverse},
@ref{482,,m_clear} and @ref{483,,m_free}
functions of @ref{484,,PyModuleDef} are no longer called if the module state
was requested but is not allocated yet. This is the case immediately after
the module is created and before the module is executed
(@ref{485,,Py_mod_exec} function). More precisely, these functions are not called
if @ref{486,,m_size} is greater than 0 and the module state (as
returned by @ref{487,,PyModule_GetState()}) is @code{NULL}.

Extension modules without module state (@code{m_size <= 0}) are not affected.

@item 
If @ref{488,,Py_AddPendingCall()} is called in a subinterpreter, the function is
now scheduled to be called from the subinterpreter, rather than being called
from the main interpreter. Each subinterpreter now has its own list of
scheduled calls.
(Contributed by Victor Stinner in bpo-39984@footnote{https://bugs.python.org/issue39984}.)

@item 
The Windows registry is no longer used to initialize @ref{489,,sys.path} when
the @code{-E} option is used (if @ref{48a,,PyConfig.use_environment} is set to
@code{0}). This is significant when embedding Python on Windows.
(Contributed by Zackery Spytz in bpo-8901@footnote{https://bugs.python.org/issue8901}.)

@item 
The global variable @ref{48b,,PyStructSequence_UnnamedField} is now a constant
and refers to a constant string.
(Contributed by Serhiy Storchaka in bpo-38650@footnote{https://bugs.python.org/issue38650}.)

@item 
The @code{PyGC_Head} structure is now opaque. It is only defined in the
internal C API (@code{pycore_gc.h}).
(Contributed by Victor Stinner in bpo-40241@footnote{https://bugs.python.org/issue40241}.)

@item 
The @code{Py_UNICODE_COPY}, @code{Py_UNICODE_FILL}, @code{PyUnicode_WSTR_LENGTH},
@ref{48c,,PyUnicode_FromUnicode()}, @ref{48d,,PyUnicode_AsUnicode()},
@code{_PyUnicode_AsUnicode}, and @ref{48e,,PyUnicode_AsUnicodeAndSize()} are
marked as deprecated in C.  They have been deprecated by 
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393} since
Python 3.3.
(Contributed by Inada Naoki in bpo-36346@footnote{https://bugs.python.org/issue36346}.)

@item 
The @ref{48f,,Py_FatalError()} function is replaced with a macro which logs
automatically the name of the current function, unless the
@code{Py_LIMITED_API} macro is defined.
(Contributed by Victor Stinner in bpo-39882@footnote{https://bugs.python.org/issue39882}.)

@item 
The vectorcall protocol now requires that the caller passes only strings as
keyword names. (See bpo-37540@footnote{https://bugs.python.org/issue37540} for more information.)

@item 
Implementation details of a number of macros and functions are now hidden:


@itemize *

@item 
@ref{490,,PyObject_IS_GC()} macro was converted to a function.

@item 
The @code{PyObject_NEW()} macro becomes an alias to the
@ref{491,,PyObject_New()} macro, and the @code{PyObject_NEW_VAR()} macro
becomes an alias to the @ref{492,,PyObject_NewVar()} macro. They no longer
access directly the @ref{493,,PyTypeObject.tp_basicsize} member.

@item 
@code{PyObject_GET_WEAKREFS_LISTPTR()} macro was converted to a function:
the macro accessed directly the @ref{494,,PyTypeObject.tp_weaklistoffset}
member.

@item 
@ref{1dc,,PyObject_CheckBuffer()} macro was converted to a function: the macro
accessed directly the @ref{495,,PyTypeObject.tp_as_buffer} member.

@item 
@ref{496,,PyIndex_Check()} is now always declared as an opaque function to hide
implementation details: removed the @code{PyIndex_Check()} macro. The macro accessed
directly the @ref{497,,PyTypeObject.tp_as_number} member.
@end itemize

(See bpo-40170@footnote{https://bugs.python.org/issue40170} for more details.)
@end itemize

@node Removed<6>,,Porting to Python 3 9<2>,C API Changes<3>
@anchor{whatsnew/3 9 id3}@anchor{498}
@subsubsection Removed



@itemize *

@item 
Excluded @code{PyFPE_START_PROTECT()} and @code{PyFPE_END_PROTECT()} macros of
@code{pyfpe.h} from the limited C API.
(Contributed by Victor Stinner in bpo-38835@footnote{https://bugs.python.org/issue38835}.)

@item 
The @code{tp_print} slot of @ref{499,,PyTypeObject} has been removed.
It was used for printing objects to files in Python 2.7 and before. Since
Python 3.0, it has been ignored and unused.
(Contributed by Jeroen Demeyer in bpo-36974@footnote{https://bugs.python.org/issue36974}.)

@item 
Changes in the limited C API (if @code{Py_LIMITED_API} macro is defined):


@itemize *

@item 
Excluded the following functions from the limited C API:


@itemize *

@item 
@code{PyThreadState_DeleteCurrent()}
(Contributed by Joannah Nanjekye in bpo-37878@footnote{https://bugs.python.org/issue37878}.)

@item 
@code{_Py_CheckRecursionLimit}

@item 
@code{_Py_NewReference()}

@item 
@code{_Py_ForgetReference()}

@item 
@code{_PyTraceMalloc_NewReference()}

@item 
@code{_Py_GetRefTotal()}

@item 
The trashcan mechanism which never worked in the limited C API.

@item 
@code{PyTrash_UNWIND_LEVEL}

@item 
@code{Py_TRASHCAN_BEGIN_CONDITION}

@item 
@code{Py_TRASHCAN_BEGIN}

@item 
@code{Py_TRASHCAN_END}

@item 
@code{Py_TRASHCAN_SAFE_BEGIN}

@item 
@code{Py_TRASHCAN_SAFE_END}
@end itemize

@item 
Moved following functions and definitions to the internal C API:


@itemize *

@item 
@code{_PyDebug_PrintTotalRefs()}

@item 
@code{_Py_PrintReferences()}

@item 
@code{_Py_PrintReferenceAddresses()}

@item 
@code{_Py_tracemalloc_config}

@item 
@code{_Py_AddToAllObjects()} (specific to @code{Py_TRACE_REFS} build)
@end itemize
@end itemize

(Contributed by Victor Stinner in bpo-38644@footnote{https://bugs.python.org/issue38644} and bpo-39542@footnote{https://bugs.python.org/issue39542}.)

@item 
Removed @code{_PyRuntime.getframe} hook and removed @code{_PyThreadState_GetFrame}
macro which was an alias to @code{_PyRuntime.getframe}. They were only exposed
by the internal C API. Removed also @code{PyThreadFrameGetter} type.
(Contributed by Victor Stinner in bpo-39946@footnote{https://bugs.python.org/issue39946}.)

@item 
Removed the following functions from the C API. Call @ref{49a,,PyGC_Collect()}
explicitly to clear all free lists.
(Contributed by Inada Naoki and Victor Stinner in bpo-37340@footnote{https://bugs.python.org/issue37340},
bpo-38896@footnote{https://bugs.python.org/issue38896} and bpo-40428@footnote{https://bugs.python.org/issue40428}.)


@itemize *

@item 
@code{PyAsyncGen_ClearFreeLists()}

@item 
@code{PyContext_ClearFreeList()}

@item 
@code{PyDict_ClearFreeList()}

@item 
@code{PyFloat_ClearFreeList()}

@item 
@code{PyFrame_ClearFreeList()}

@item 
@code{PyList_ClearFreeList()}

@item 
@code{PyMethod_ClearFreeList()} and @code{PyCFunction_ClearFreeList()}:
the free lists of bound method objects have been removed.

@item 
@code{PySet_ClearFreeList()}: the set free list has been removed
in Python 3.4.

@item 
@code{PyTuple_ClearFreeList()}

@item 
@code{PyUnicode_ClearFreeList()}: the Unicode free list has been removed in
Python 3.3.
@end itemize

@item 
Removed @code{_PyUnicode_ClearStaticStrings()} function.
(Contributed by Victor Stinner in bpo-39465@footnote{https://bugs.python.org/issue39465}.)

@item 
Removed @code{Py_UNICODE_MATCH}. It has been deprecated by 
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393}, and
broken since Python 3.3. The @ref{39b,,PyUnicode_Tailmatch()} function can be
used instead.
(Contributed by Inada Naoki in bpo-36346@footnote{https://bugs.python.org/issue36346}.)

@item 
Cleaned header files of interfaces defined but with no implementation.
The public API symbols being removed are:
@code{_PyBytes_InsertThousandsGroupingLocale},
@code{_PyBytes_InsertThousandsGrouping}, @code{_Py_InitializeFromArgs},
@code{_Py_InitializeFromWideArgs}, @code{_PyFloat_Repr}, @code{_PyFloat_Digits},
@code{_PyFloat_DigitsInit}, @code{PyFrame_ExtendStack}, @code{_PyAIterWrapper_Type},
@code{PyNullImporter_Type}, @code{PyCmpWrapper_Type}, @code{PySortWrapper_Type},
@code{PyNoArgsFunction}.
(Contributed by Pablo Galindo Salgado in bpo-39372@footnote{https://bugs.python.org/issue39372}.)
@end itemize

@node Notable changes in Python 3 9 1,Notable changes in Python 3 9 2,C API Changes<3>,What’s New In Python 3 9
@anchor{whatsnew/3 9 notable-changes-in-python-3-9-1}@anchor{49b}
@subsection Notable changes in Python 3.9.1


@menu
* typing: typing<3>. 
* macOS 11.0 (Big Sur) and Apple Silicon Mac support: macOS 11 0 Big Sur and Apple Silicon Mac support. 

@end menu

@node typing<3>,macOS 11 0 Big Sur and Apple Silicon Mac support,,Notable changes in Python 3 9 1
@anchor{whatsnew/3 9 id4}@anchor{49c}
@subsubsection typing


The behavior of @ref{2db,,typing.Literal} was changed to conform with 
@geindex Python Enhancement Proposals; PEP 586
PEP 586@footnote{https://www.python.org/dev/peps/pep-0586}
and to match the behavior of static type checkers specified in the PEP.


@enumerate 

@item 
@code{Literal} now de-duplicates parameters.

@item 
Equality comparisons between @code{Literal} objects are now order independent.

@item 
@code{Literal} comparisons now respect types.  For example,
@code{Literal[0] == Literal[False]} previously evaluated to @code{True}.  It is
now @code{False}.  To support this change, the internally used type cache now
supports differentiating types.

@item 
@code{Literal} objects will now raise a @ref{162,,TypeError} exception during
equality comparisons if any of their parameters are not @ref{2dc,,hashable}.
Note that declaring @code{Literal} with mutable parameters will not throw
an error:

@example
>>> from typing import Literal
>>> Literal[@{0@}]
>>> Literal[@{0@}] == Literal[@{False@}]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'set'
@end example
@end enumerate

(Contributed by Yurii Karabas in bpo-42345@footnote{https://bugs.python.org/issue42345}.)

@node macOS 11 0 Big Sur and Apple Silicon Mac support,,typing<3>,Notable changes in Python 3 9 1
@anchor{whatsnew/3 9 macos-11-0-big-sur-and-apple-silicon-mac-support}@anchor{49d}
@subsubsection macOS 11.0 (Big Sur) and Apple Silicon Mac support


As of 3.9.1, Python now fully supports building and running on macOS 11.0
(Big Sur) and on Apple Silicon Macs (based on the @code{ARM64} architecture).
A new universal build variant, @code{universal2}, is now available to natively
support both @code{ARM64} and @code{Intel 64} in one set of executables. Binaries
can also now be built on current versions of macOS to be deployed on a range
of older macOS versions (tested to 10.9) while making some newer OS
functions and options conditionally available based on the operating system
version in use at runtime (“weaklinking”).

(Contributed by Ronald Oussoren and Lawrence D’Anna in bpo-41100@footnote{https://bugs.python.org/issue41100}.)

@node Notable changes in Python 3 9 2,,Notable changes in Python 3 9 1,What’s New In Python 3 9
@anchor{whatsnew/3 9 notable-changes-in-python-3-9-2}@anchor{49e}
@subsection Notable changes in Python 3.9.2


@menu
* collections.abc: collections abc<2>. 
* urllib.parse: urllib parse<2>. 

@end menu

@node collections abc<2>,urllib parse<2>,,Notable changes in Python 3 9 2
@anchor{whatsnew/3 9 collections-abc}@anchor{49f}
@subsubsection collections.abc


@ref{269,,collections.abc.Callable} generic now flattens type parameters, similar
to what @ref{23c,,typing.Callable} currently does.  This means that
@code{collections.abc.Callable[[int, str], str]} will have @code{__args__} of
@code{(int, str, str)}; previously this was @code{([int, str], str)}.  To allow this
change, @ref{26a,,types.GenericAlias} can now be subclassed, and a subclass will
be returned when subscripting the @ref{269,,collections.abc.Callable} type.
Code which accesses the arguments via @ref{344,,typing.get_args()} or @code{__args__}
need to account for this change.  A @ref{1b2,,DeprecationWarning} may be emitted for
invalid forms of parameterizing @ref{269,,collections.abc.Callable} which may have
passed silently in Python 3.9.1.  This @ref{1b2,,DeprecationWarning} will
become a @ref{162,,TypeError} in Python 3.10.
(Contributed by Ken Jin in bpo-42195@footnote{https://bugs.python.org/issue42195}.)

@node urllib parse<2>,,collections abc<2>,Notable changes in Python 3 9 2
@anchor{whatsnew/3 9 urllib-parse}@anchor{4a0}
@subsubsection urllib.parse


Earlier Python versions allowed using both @code{;} and @code{&} as
query parameter separators in @ref{2e3,,urllib.parse.parse_qs()} and
@ref{2e4,,urllib.parse.parse_qsl()}.  Due to security concerns, and to conform with
newer W3C recommendations, this has been changed to allow only a single
separator key, with @code{&} as the default.  This change also affects
@ref{2e5,,cgi.parse()} and @ref{2e6,,cgi.parse_multipart()} as they use the affected
functions internally. For more details, please see their respective
documentation.
(Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in bpo-42967@footnote{https://bugs.python.org/issue42967}.)

@node What’s New In Python 3 8,What’s New In Python 3 7,What’s New In Python 3 9,What’s New in Python
@anchor{whatsnew/3 8 doc}@anchor{4a1}@anchor{whatsnew/3 8 what-s-new-in-python-3-8}@anchor{4a2}
@section What’s New In Python 3.8


@c Rules for maintenance:
@c 
@c * Anyone can add text to this document.  Do not spend very much time
@c on the wording of your changes, because your text will probably
@c get rewritten to some degree.
@c 
@c * The maintainer will go through Misc/NEWS periodically and add
@c changes; it's therefore more important to add your changes to
@c Misc/NEWS than to this file.
@c 
@c * This is not a complete list of every single change; completeness
@c is the purpose of Misc/NEWS.  Some changes I consider too small
@c or esoteric to include.  If such a change is added to the text,
@c I'll just remove it.  (This is another reason you shouldn't spend
@c too much time on writing your addition.)
@c 
@c * If you want to draw your new text to the attention of the
@c maintainer, add 'XXX' to the beginning of the paragraph or
@c section.
@c 
@c * It's OK to just add a fragmentary note about a change.  For
@c example: "XXX Describe the transmogrify() function added to the
@c socket module."  The maintainer will research the change and
@c write the necessary text.
@c 
@c * You can comment out your additions if you like, but it's not
@c necessary (especially when a final release is some months away).
@c 
@c * Credit the author of a patch or bugfix.   Just the name is
@c sufficient; the e-mail address isn't necessary.
@c 
@c * It's helpful to add the bug/patch number as a comment:
@c 
@c XXX Describe the transmogrify() function added to the socket
@c module.
@c (Contributed by P.Y. Developer in :issue:`12345`.)
@c 
@c This saves the maintainer the effort of going through the Git log
@c when researching a change.

@*Editor: 
Raymond Hettinger


This article explains the new features in Python 3.8, compared to 3.7.
For full details, see the @ref{154,,changelog}.

@c from datetime import date
@c from math import cos, radians
@c from unicodedata import normalize
@c import re
@c import math

@menu
* Summary – Release highlights: Summary – Release highlights<4>. 
* New Features: New Features<7>. 
* Other Language Changes: Other Language Changes<4>. 
* New Modules: New Modules<4>. 
* Improved Modules: Improved Modules<4>. 
* Optimizations: Optimizations<4>. 
* Build and C API Changes:: 
* Deprecated: Deprecated<6>. 
* API and Feature Removals:: 
* Porting to Python 3.8: Porting to Python 3 8. 
* Notable changes in Python 3.8.1: Notable changes in Python 3 8 1. 
* Notable changes in Python 3.8.8: Notable changes in Python 3 8 8. 
* Notable changes in Python 3.8.12: Notable changes in Python 3 8 12. 

@end menu

@node Summary – Release highlights<4>,New Features<7>,,What’s New In Python 3 8
@anchor{whatsnew/3 8 summary-release-highlights}@anchor{4a3}
@subsection Summary – Release highlights


@c This section singles out the most important changes in Python 3.8.
@c Brevity is key.

@c PEP-sized items next.

@node New Features<7>,Other Language Changes<4>,Summary – Release highlights<4>,What’s New In Python 3 8
@anchor{whatsnew/3 8 new-features}@anchor{4a4}
@subsection New Features


@menu
* Assignment expressions:: 
* Positional-only parameters:: 
* Parallel filesystem cache for compiled bytecode files:: 
* Debug build uses the same ABI as release build:: 
* f-strings support = for self-documenting expressions and debugging:: 
* PEP 578; Python Runtime Audit Hooks: PEP 578 Python Runtime Audit Hooks. 
* PEP 587; Python Initialization Configuration: PEP 587 Python Initialization Configuration. 
* PEP 590; Vectorcall; a fast calling protocol for CPython: PEP 590 Vectorcall a fast calling protocol for CPython. 
* Pickle protocol 5 with out-of-band data buffers:: 

@end menu

@node Assignment expressions,Positional-only parameters,,New Features<7>
@anchor{whatsnew/3 8 assignment-expressions}@anchor{4a5}
@subsubsection Assignment expressions


There is new syntax @code{:=} that assigns values to variables as part of a larger
expression. It is affectionately known as “the walrus operator” due to
its resemblance to the eyes and tusks of a walrus@footnote{https://en.wikipedia.org/wiki/Walrus#/media/File:Pacific_Walrus_-_Bull_(8247646168).jpg}.

In this example, the assignment expression helps avoid calling
@ref{4a6,,len()} twice:

@example
if (n := len(a)) > 10:
    print(f"List is too long (@{n@} elements, expected <= 10)")
@end example

A similar benefit arises during regular expression matching where
match objects are needed twice, once to test whether a match
occurred and another to extract a subgroup:

@example
discount = 0.0
if (mo := re.search(r'(\d+)% discount', advertisement)):
    discount = float(mo.group(1)) / 100.0
@end example

The operator is also useful with while-loops that compute
a value to test loop termination and then need that same
value again in the body of the loop:

@example
# Loop over fixed length blocks
while (block := f.read(256)) != '@w{'}:
    process(block)
@end example

Another motivating use case arises in list comprehensions where
a value computed in a filtering condition is also needed in
the expression body:

@example
[clean_name.title() for name in names
 if (clean_name := normalize('NFC', name)) in allowed_names]
@end example

Try to limit use of the walrus operator to clean cases that reduce
complexity and improve readability.

See 
@geindex Python Enhancement Proposals; PEP 572
PEP 572@footnote{https://www.python.org/dev/peps/pep-0572} for a full description.

(Contributed by Emily Morehouse in bpo-35224@footnote{https://bugs.python.org/issue35224}.)

@node Positional-only parameters,Parallel filesystem cache for compiled bytecode files,Assignment expressions,New Features<7>
@anchor{whatsnew/3 8 positional-only-parameters}@anchor{4a7}
@subsubsection Positional-only parameters


There is a new function parameter syntax @code{/} to indicate that some
function parameters must be specified positionally and cannot be used as
keyword arguments.  This is the same notation shown by @code{help()} for C
functions annotated with Larry Hastings’ Argument Clinic@footnote{https://docs.python.org/3/howto/clinic.html} tool.

In the following example, parameters @emph{a} and @emph{b} are positional-only,
while @emph{c} or @emph{d} can be positional or keyword, and @emph{e} or @emph{f} are
required to be keywords:

@example
def f(a, b, /, c, d, *, e, f):
    print(a, b, c, d, e, f)
@end example

The following is a valid call:

@example
f(10, 20, 30, d=40, e=50, f=60)
@end example

However, these are invalid calls:

@example
f(10, b=20, c=30, d=40, e=50, f=60)   # b cannot be a keyword argument
f(10, 20, 30, 40, 50, f=60)           # e must be a keyword argument
@end example

One use case for this notation is that it allows pure Python functions
to fully emulate behaviors of existing C coded functions.  For example,
the built-in @ref{4a8,,divmod()} function does not accept keyword arguments:

@example
def divmod(a, b, /):
    "Emulate the built in divmod() function"
    return (a // b, a % b)
@end example

Another use case is to preclude keyword arguments when the parameter
name is not helpful.  For example, the builtin @ref{4a6,,len()} function has
the signature @code{len(obj, /)}.  This precludes awkward calls such as:

@example
len(obj='hello')  # The "obj" keyword argument impairs readability
@end example

A further benefit of marking a parameter as positional-only is that it
allows the parameter name to be changed in the future without risk of
breaking client code.  For example, in the @ref{f0,,statistics} module, the
parameter name @emph{dist} may be changed in the future.  This was made
possible with the following function specification:

@example
def quantiles(dist, /, *, n=4, method='exclusive')
    ...
@end example

Since the parameters to the left of @code{/} are not exposed as possible
keywords, the parameters names remain available for use in @code{**kwargs}:

@example
>>> def f(a, b, /, **kwargs):
...     print(a, b, kwargs)
...
>>> f(10, 20, a=1, b=2, c=3)         # a and b are used in two ways
10 20 @{'a': 1, 'b': 2, 'c': 3@}
@end example

This greatly simplifies the implementation of functions and methods
that need to accept arbitrary keyword arguments.  For example, here
is an excerpt from code in the @ref{1c,,collections} module:

@example
class Counter(dict):

    def __init__(self, iterable=None, /, **kwds):
        # Note "iterable" is a possible keyword argument
@end example

See 
@geindex Python Enhancement Proposals; PEP 570
PEP 570@footnote{https://www.python.org/dev/peps/pep-0570} for a full description.

(Contributed by Pablo Galindo in bpo-36540@footnote{https://bugs.python.org/issue36540}.)

@c TODO: Pablo will sprint on docs at PyCon US 2019.

@node Parallel filesystem cache for compiled bytecode files,Debug build uses the same ABI as release build,Positional-only parameters,New Features<7>
@anchor{whatsnew/3 8 parallel-filesystem-cache-for-compiled-bytecode-files}@anchor{4a9}
@subsubsection Parallel filesystem cache for compiled bytecode files


The new 
@geindex PYTHONPYCACHEPREFIX
@geindex environment variable; PYTHONPYCACHEPREFIX
@ref{4aa,,PYTHONPYCACHEPREFIX} setting (also available as
@ref{159,,-X} @code{pycache_prefix}) configures the implicit bytecode
cache to use a separate parallel filesystem tree, rather than
the default @code{__pycache__} subdirectories within each source
directory.

The location of the cache is reported in @ref{4ab,,sys.pycache_prefix}
(@ref{182,,None} indicates the default location in @code{__pycache__}
subdirectories).

(Contributed by Carl Meyer in bpo-33499@footnote{https://bugs.python.org/issue33499}.)

@node Debug build uses the same ABI as release build,f-strings support = for self-documenting expressions and debugging,Parallel filesystem cache for compiled bytecode files,New Features<7>
@anchor{whatsnew/3 8 debug-build-uses-the-same-abi-as-release-build}@anchor{4ac}
@subsubsection Debug build uses the same ABI as release build


Python now uses the same ABI whether it’s built in release or debug mode. On
Unix, when Python is built in debug mode, it is now possible to load C
extensions built in release mode and C extensions built using the stable ABI.

Release builds and @ref{338,,debug builds} are now ABI compatible: defining the
@code{Py_DEBUG} macro no longer implies the @code{Py_TRACE_REFS} macro, which
introduces the only ABI incompatibility. The @code{Py_TRACE_REFS} macro, which
adds the @code{sys.getobjects()} function and the 
@geindex PYTHONDUMPREFS
@geindex environment variable; PYTHONDUMPREFS
@ref{4ad,,PYTHONDUMPREFS}
environment variable, can be set using the new @ref{376,,./configure --with-trace-refs} build option.
(Contributed by Victor Stinner in bpo-36465@footnote{https://bugs.python.org/issue36465}.)

On Unix, C extensions are no longer linked to libpython except on Android
and Cygwin.
It is now possible
for a statically linked Python to load a C extension built using a shared
library Python.
(Contributed by Victor Stinner in bpo-21536@footnote{https://bugs.python.org/issue21536}.)

On Unix, when Python is built in debug mode, import now also looks for C
extensions compiled in release mode and for C extensions compiled with the
stable ABI.
(Contributed by Victor Stinner in bpo-36722@footnote{https://bugs.python.org/issue36722}.)

To embed Python into an application, a new @code{--embed} option must be passed to
@code{python3-config --libs --embed} to get @code{-lpython3.8} (link the application
to libpython). To support both 3.8 and older, try @code{python3-config --libs
--embed} first and fallback to @code{python3-config --libs} (without @code{--embed})
if the previous command fails.

Add a pkg-config @code{python-3.8-embed} module to embed Python into an
application: @code{pkg-config python-3.8-embed --libs} includes @code{-lpython3.8}.
To support both 3.8 and older, try @code{pkg-config python-X.Y-embed --libs} first
and fallback to @code{pkg-config python-X.Y --libs} (without @code{--embed}) if the
previous command fails (replace @code{X.Y} with the Python version).

On the other hand, @code{pkg-config python3.8 --libs} no longer contains
@code{-lpython3.8}. C extensions must not be linked to libpython (except on
Android and Cygwin, whose cases are handled by the script);
this change is backward incompatible on purpose.
(Contributed by Victor Stinner in bpo-36721@footnote{https://bugs.python.org/issue36721}.)

@node f-strings support = for self-documenting expressions and debugging,PEP 578 Python Runtime Audit Hooks,Debug build uses the same ABI as release build,New Features<7>
@anchor{whatsnew/3 8 f-strings-support-for-self-documenting-expressions-and-debugging}@anchor{4ae}
@subsubsection f-strings support @code{=} for self-documenting expressions and debugging


Added an @code{=} specifier to @ref{4af,,f-string}s. An f-string such as
@code{f'@{expr=@}'} will expand to the text of the expression, an equal sign,
then the representation of the evaluated expression.  For example:

@example
>>> user = 'eric_idle'
>>> member_since = date(1975, 7, 31)
>>> f'@{user=@} @{member_since=@}'
"user='eric_idle' member_since=datetime.date(1975, 7, 31)"
@end example

The usual @ref{4b0,,f-string format specifiers} allow more
control over how the result of the expression is displayed:

@example
>>> delta = date.today() - member_since
>>> f'@{user=!s@}  @{delta.days=:,d@}'
'user=eric_idle  delta.days=16,075'
@end example

The @code{=} specifier will display the whole expression so that
calculations can be shown:

@example
>>> print(f'@{theta=@}  @{cos(radians(theta))=:.3f@}')
theta=30  cos(radians(theta))=0.866
@end example

(Contributed by Eric V. Smith and Larry Hastings in bpo-36817@footnote{https://bugs.python.org/issue36817}.)

@node PEP 578 Python Runtime Audit Hooks,PEP 587 Python Initialization Configuration,f-strings support = for self-documenting expressions and debugging,New Features<7>
@anchor{whatsnew/3 8 pep-578-python-runtime-audit-hooks}@anchor{4b1}
@subsubsection PEP 578: Python Runtime Audit Hooks


The PEP adds an Audit Hook and Verified Open Hook. Both are available from
Python and native code, allowing applications and frameworks written in pure
Python code to take advantage of extra notifications, while also allowing
embedders or system administrators to deploy builds of Python where auditing is
always enabled.

See 
@geindex Python Enhancement Proposals; PEP 578
PEP 578@footnote{https://www.python.org/dev/peps/pep-0578} for full details.

@node PEP 587 Python Initialization Configuration,PEP 590 Vectorcall a fast calling protocol for CPython,PEP 578 Python Runtime Audit Hooks,New Features<7>
@anchor{whatsnew/3 8 pep-587-python-initialization-configuration}@anchor{4b2}
@subsubsection PEP 587: Python Initialization Configuration


The 
@geindex Python Enhancement Proposals; PEP 587
PEP 587@footnote{https://www.python.org/dev/peps/pep-0587} adds a new C API to configure the Python Initialization
providing finer control on the whole configuration and better error reporting.

New structures:


@itemize *

@item 
@ref{206,,PyConfig}

@item 
@ref{4b3,,PyPreConfig}

@item 
@ref{4b4,,PyStatus}

@item 
@ref{4b5,,PyWideStringList}
@end itemize

New functions:


@itemize *

@item 
@code{PyConfig_Clear()}

@item 
@code{PyConfig_InitIsolatedConfig()}

@item 
@code{PyConfig_InitPythonConfig()}

@item 
@code{PyConfig_Read()}

@item 
@code{PyConfig_SetArgv()}

@item 
@code{PyConfig_SetBytesArgv()}

@item 
@code{PyConfig_SetBytesString()}

@item 
@code{PyConfig_SetString()}

@item 
@code{PyPreConfig_InitIsolatedConfig()}

@item 
@code{PyPreConfig_InitPythonConfig()}

@item 
@code{PyStatus_Error()}

@item 
@code{PyStatus_Exception()}

@item 
@code{PyStatus_Exit()}

@item 
@code{PyStatus_IsError()}

@item 
@code{PyStatus_IsExit()}

@item 
@code{PyStatus_NoMemory()}

@item 
@code{PyStatus_Ok()}

@item 
@code{PyWideStringList_Append()}

@item 
@code{PyWideStringList_Insert()}

@item 
@ref{4b6,,Py_BytesMain()}

@item 
@code{Py_ExitStatusException()}

@item 
@ref{4b7,,Py_InitializeFromConfig()}

@item 
@ref{4b8,,Py_PreInitialize()}

@item 
@ref{4b9,,Py_PreInitializeFromArgs()}

@item 
@ref{4ba,,Py_PreInitializeFromBytesArgs()}

@item 
@ref{4bb,,Py_RunMain()}
@end itemize

This PEP also adds @code{_PyRuntimeState.preconfig} (@ref{4b3,,PyPreConfig} type)
and @code{PyInterpreterState.config} (@ref{206,,PyConfig} type) fields to these
internal structures. @code{PyInterpreterState.config} becomes the new
reference configuration, replacing global configuration variables and
other private variables.

See @ref{207,,Python Initialization Configuration} for the
documentation.

See 
@geindex Python Enhancement Proposals; PEP 587
PEP 587@footnote{https://www.python.org/dev/peps/pep-0587} for a full description.

(Contributed by Victor Stinner in bpo-36763@footnote{https://bugs.python.org/issue36763}.)

@node PEP 590 Vectorcall a fast calling protocol for CPython,Pickle protocol 5 with out-of-band data buffers,PEP 587 Python Initialization Configuration,New Features<7>
@anchor{whatsnew/3 8 pep-590-vectorcall-a-fast-calling-protocol-for-cpython}@anchor{4bc}
@subsubsection PEP 590: Vectorcall: a fast calling protocol for CPython


@ref{4bd,,The Vectorcall Protocol} is added to the Python/C API.
It is meant to formalize existing optimizations which were already done
for various classes.
Any @ref{1f0,,static type} implementing a callable can use this
protocol.

This is currently provisional.
The aim is to make it fully public in Python 3.9.

See 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590} for a full description.

(Contributed by Jeroen Demeyer, Mark Shannon and Petr Viktorin in bpo-36974@footnote{https://bugs.python.org/issue36974}.)

@node Pickle protocol 5 with out-of-band data buffers,,PEP 590 Vectorcall a fast calling protocol for CPython,New Features<7>
@anchor{whatsnew/3 8 pickle-protocol-5-with-out-of-band-data-buffers}@anchor{4be}
@subsubsection Pickle protocol 5 with out-of-band data buffers


When @ref{c5,,pickle} is used to transfer large data between Python processes
in order to take advantage of multi-core or multi-machine processing,
it is important to optimize the transfer by reducing memory copies, and
possibly by applying custom techniques such as data-dependent compression.

The @ref{c5,,pickle} protocol 5 introduces support for out-of-band buffers
where 
@geindex Python Enhancement Proposals; PEP 3118
PEP 3118@footnote{https://www.python.org/dev/peps/pep-3118}-compatible data can be transmitted separately from the
main pickle stream, at the discretion of the communication layer.

See 
@geindex Python Enhancement Proposals; PEP 574
PEP 574@footnote{https://www.python.org/dev/peps/pep-0574} for a full description.

(Contributed by Antoine Pitrou in bpo-36785@footnote{https://bugs.python.org/issue36785}.)

@node Other Language Changes<4>,New Modules<4>,New Features<7>,What’s New In Python 3 8
@anchor{whatsnew/3 8 other-language-changes}@anchor{4bf}
@subsection Other Language Changes



@itemize *

@item 
A @ref{4c0,,continue} statement was illegal in the @ref{4c1,,finally} clause
due to a problem with the implementation.  In Python 3.8 this restriction
was lifted.
(Contributed by Serhiy Storchaka in bpo-32489@footnote{https://bugs.python.org/issue32489}.)

@item 
The @ref{2f7,,bool}, @ref{1b1,,int}, and @ref{172,,fractions.Fraction} types
now have an @ref{4c2,,as_integer_ratio()} method like that found in
@ref{256,,float} and @ref{248,,decimal.Decimal}.  This minor API extension
makes it possible to write @code{numerator, denominator =
x.as_integer_ratio()} and have it work across multiple numeric types.
(Contributed by Lisa Roach in bpo-33073@footnote{https://bugs.python.org/issue33073} and Raymond Hettinger in
bpo-37819@footnote{https://bugs.python.org/issue37819}.)

@item 
Constructors of @ref{1b1,,int}, @ref{256,,float} and @ref{33b,,complex} will now
use the @ref{24a,,__index__()} special method, if available and the
corresponding method @ref{249,,__int__()}, @ref{4c3,,__float__()}
or @ref{4c4,,__complex__()} is not available.
(Contributed by Serhiy Storchaka in bpo-20092@footnote{https://bugs.python.org/issue20092}.)

@item 
Added support of @code{\N@{name@}} escapes in @ref{d8,,regular expressions}:

@example
>>> notice = 'Copyright © 2019'
>>> copyright_year_pattern = re.compile(r'\N@{copyright sign@}\s*(\d@{4@})')
>>> int(copyright_year_pattern.search(notice).group(1))
2019
@end example

(Contributed by Jonathan Eunice and Serhiy Storchaka in bpo-30688@footnote{https://bugs.python.org/issue30688}.)

@item 
Dict and dictviews are now iterable in reversed insertion order using
@ref{2f6,,reversed()}. (Contributed by Rémi Lapeyre in bpo-33462@footnote{https://bugs.python.org/issue33462}.)

@item 
The syntax allowed for keyword names in function calls was further
restricted. In particular, @code{f((keyword)=arg)} is no longer allowed. It was
never intended to permit more than a bare name on the left-hand side of a
keyword argument assignment term.
(Contributed by Benjamin Peterson in bpo-34641@footnote{https://bugs.python.org/issue34641}.)

@item 
Generalized iterable unpacking in @ref{4c5,,yield} and
@ref{4c6,,return} statements no longer requires enclosing parentheses.
This brings the @emph{yield} and @emph{return} syntax into better agreement with
normal assignment syntax:

@example
>>> def parse(family):
        lastname, *members = family.split()
        return lastname.upper(), *members

>>> parse('simpsons homer marge bart lisa maggie')
('SIMPSONS', 'homer', 'marge', 'bart', 'lisa', 'maggie')
@end example

(Contributed by David Cuthbert and Jordan Chapman in bpo-32117@footnote{https://bugs.python.org/issue32117}.)

@item 
When a comma is missed in code such as @code{[(10, 20) (30, 40)]}, the
compiler displays a @ref{4c7,,SyntaxWarning} with a helpful suggestion.
This improves on just having a @ref{162,,TypeError} indicating that the
first tuple was not callable.  (Contributed by Serhiy Storchaka in
bpo-15248@footnote{https://bugs.python.org/issue15248}.)

@item 
Arithmetic operations between subclasses of @ref{3e2,,datetime.date} or
@ref{363,,datetime.datetime} and @ref{4c8,,datetime.timedelta} objects now return
an instance of the subclass, rather than the base class. This also affects
the return type of operations whose implementation (directly or indirectly)
uses @ref{4c8,,datetime.timedelta} arithmetic, such as
@ref{4c9,,astimezone()}.
(Contributed by Paul Ganssle in bpo-32417@footnote{https://bugs.python.org/issue32417}.)

@item 
When the Python interpreter is interrupted by Ctrl-C (SIGINT) and the
resulting @ref{4ca,,KeyboardInterrupt} exception is not caught, the Python process
now exits via a SIGINT signal or with the correct exit code such that the
calling process can detect that it died due to a Ctrl-C.  Shells on POSIX
and Windows use this to properly terminate scripts in interactive sessions.
(Contributed by Google via Gregory P. Smith in bpo-1054041@footnote{https://bugs.python.org/issue1054041}.)

@item 
Some advanced styles of programming require updating the
@ref{4cb,,types.CodeType} object for an existing function.  Since code
objects are immutable, a new code object needs to be created, one
that is modeled on the existing code object.  With 19 parameters,
this was somewhat tedious.  Now, the new @code{replace()} method makes
it possible to create a clone with a few altered parameters.

Here’s an example that alters the @ref{4cc,,statistics.mean()} function to
prevent the @emph{data} parameter from being used as a keyword argument:

@example
>>> from statistics import mean
>>> mean(data=[10, 20, 90])
40
>>> mean.__code__ = mean.__code__.replace(co_posonlyargcount=1)
>>> mean(data=[10, 20, 90])
Traceback (most recent call last):
  ...
TypeError: mean() got some positional-only arguments passed as keyword arguments: 'data'
@end example

(Contributed by Victor Stinner in bpo-37032@footnote{https://bugs.python.org/issue37032}.)

@item 
For integers, the three-argument form of the @ref{4cd,,pow()} function now
permits the exponent to be negative in the case where the base is
relatively prime to the modulus. It then computes a modular inverse to
the base when the exponent is @code{-1}, and a suitable power of that
inverse for other negative exponents.  For example, to compute the
modular multiplicative inverse@footnote{https://en.wikipedia.org/wiki/Modular_multiplicative_inverse} of 38
modulo 137, write:

@example
>>> pow(38, -1, 137)
119
>>> 119 * 38 % 137
1
@end example

Modular inverses arise in the solution of linear Diophantine equations@footnote{https://en.wikipedia.org/wiki/Diophantine_equation}.
For example, to find integer solutions for @code{4258𝑥 + 147𝑦 = 369},
first rewrite as @code{4258𝑥 ≡ 369 (mod 147)} then solve:

@example
>>> x = 369 * pow(4258, -1, 147) % 147
>>> y = (4258 * x - 369) // -147
>>> 4258 * x + 147 * y
369
@end example

(Contributed by Mark Dickinson in bpo-36027@footnote{https://bugs.python.org/issue36027}.)

@item 
Dict comprehensions have been synced-up with dict literals so that the
key is computed first and the value second:

@example
>>> # Dict comprehension
>>> cast = @{input('role? '): input('actor? ') for i in range(2)@}
role? King Arthur
actor? Chapman
role? Black Knight
actor? Cleese

>>> # Dict literal
>>> cast = @{input('role? '): input('actor? ')@}
role? Sir Robin
actor? Eric Idle
@end example

The guaranteed execution order is helpful with assignment expressions
because variables assigned in the key expression will be available in
the value expression:

@example
>>> names = ['Martin von Löwis', 'Łukasz Langa', 'Walter Dörwald']
>>> @{(n := normalize('NFC', name)).casefold() : n for name in names@}
@{'martin von löwis': 'Martin von Löwis',
 'łukasz langa': 'Łukasz Langa',
 'walter dörwald': 'Walter Dörwald'@}
@end example

(Contributed by Jörn Heissler in bpo-35224@footnote{https://bugs.python.org/issue35224}.)

@item 
The @ref{4ce,,object.__reduce__()} method can now return a tuple from two to
six elements long. Formerly, five was the limit.  The new, optional sixth
element is a callable with a @code{(obj, state)} signature.  This allows the
direct control over the state-updating behavior of a specific object.  If
not @emph{None}, this callable will have priority over the object’s
@code{__setstate__()} method.
(Contributed by Pierre Glaser and Olivier Grisel in bpo-35900@footnote{https://bugs.python.org/issue35900}.)
@end itemize

@node New Modules<4>,Improved Modules<4>,Other Language Changes<4>,What’s New In Python 3 8
@anchor{whatsnew/3 8 new-modules}@anchor{4cf}
@subsection New Modules



@itemize *

@item 
The new @ref{99,,importlib.metadata} module provides (provisional) support for
reading metadata from third-party packages.  For example, it can extract an
installed package’s version number, list of entry points, and more:

@example
>>> # Note following example requires that the popular "requests"
>>> # package has been installed.
>>>
>>> from importlib.metadata import version, requires, files
>>> version('requests')
'2.22.0'
>>> list(requires('requests'))
['chardet (<3.1.0,>=3.0.2)']
>>> list(files('requests'))[:5]
[PackagePath('requests-2.22.0.dist-info/INSTALLER'),
 PackagePath('requests-2.22.0.dist-info/LICENSE'),
 PackagePath('requests-2.22.0.dist-info/METADATA'),
 PackagePath('requests-2.22.0.dist-info/RECORD'),
 PackagePath('requests-2.22.0.dist-info/WHEEL')]
@end example

(Contributed by Barry Warsaw and Jason R. Coombs in bpo-34632@footnote{https://bugs.python.org/issue34632}.)
@end itemize

@node Improved Modules<4>,Optimizations<4>,New Modules<4>,What’s New In Python 3 8
@anchor{whatsnew/3 8 improved-modules}@anchor{4d0}
@subsection Improved Modules


@menu
* ast: ast<2>. 
* asyncio: asyncio<3>. 
* builtins:: 
* collections:: 
* cProfile:: 
* csv:: 
* curses: curses<3>. 
* ctypes:: 
* datetime: datetime<2>. 
* functools:: 
* gc: gc<3>. 
* gettext:: 
* gzip:: 
* IDLE and idlelib: IDLE and idlelib<4>. 
* inspect: inspect<4>. 
* io:: 
* itertools:: 
* json.tool: json tool. 
* logging:: 
* math: math<3>. 
* mmap:: 
* multiprocessing: multiprocessing<2>. 
* os: os<4>. 
* os.path: os path<2>. 
* pathlib: pathlib<3>. 
* pickle:: 
* plistlib:: 
* pprint: pprint<3>. 
* py_compile: py_compile<2>. 
* shlex:: 
* shutil:: 
* socket: socket<4>. 
* ssl: ssl<2>. 
* statistics: statistics<2>. 
* sys: sys<4>. 
* tarfile:: 
* threading: threading<3>. 
* tokenize:: 
* tkinter:: 
* time: time<3>. 
* typing: typing<4>. 
* unicodedata: unicodedata<3>. 
* unittest: unittest<2>. 
* venv: venv<2>. 
* weakref:: 
* xml: xml<3>. 
* xmlrpc:: 

@end menu

@node ast<2>,asyncio<3>,,Improved Modules<4>
@anchor{whatsnew/3 8 ast}@anchor{4d1}
@subsubsection ast


AST nodes now have @code{end_lineno} and @code{end_col_offset} attributes,
which give the precise location of the end of the node.  (This only
applies to nodes that have @code{lineno} and @code{col_offset} attributes.)

New function @ref{4d2,,ast.get_source_segment()} returns the source code
for a specific AST node.

(Contributed by Ivan Levkivskyi in bpo-33416@footnote{https://bugs.python.org/issue33416}.)

The @ref{4d3,,ast.parse()} function has some new flags:


@itemize *

@item 
@code{type_comments=True} causes it to return the text of 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} and
@geindex Python Enhancement Proposals; PEP 526
PEP 526@footnote{https://www.python.org/dev/peps/pep-0526} type comments associated with certain AST nodes;

@item 
@code{mode='func_type'} can be used to parse 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} “signature type
comments” (returned for function definition AST nodes);

@item 
@code{feature_version=(3, N)} allows specifying an earlier Python 3
version.  For example, @code{feature_version=(3, 4)} will treat
@ref{4d4,,async} and @ref{4d5,,await} as non-reserved words.
@end itemize

(Contributed by Guido van Rossum in bpo-35766@footnote{https://bugs.python.org/issue35766}.)

@node asyncio<3>,builtins,ast<2>,Improved Modules<4>
@anchor{whatsnew/3 8 asyncio}@anchor{4d6}
@subsubsection asyncio


@ref{3d1,,asyncio.run()} has graduated from the provisional to stable API. This
function can be used to execute a @ref{3cf,,coroutine} and return the result while
automatically managing the event loop. For example:

@example
import asyncio

async def main():
    await asyncio.sleep(0)
    return 42

asyncio.run(main())
@end example

This is @emph{roughly} equivalent to:

@example
import asyncio

async def main():
    await asyncio.sleep(0)
    return 42

loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
    loop.run_until_complete(main())
finally:
    asyncio.set_event_loop(None)
    loop.close()
@end example

The actual implementation is significantly more complex. Thus,
@ref{3d1,,asyncio.run()} should be the preferred way of running asyncio programs.

(Contributed by Yury Selivanov in bpo-32314@footnote{https://bugs.python.org/issue32314}.)

Running @code{python -m asyncio} launches a natively async REPL.  This allows rapid
experimentation with code that has a top-level @ref{4d5,,await}.  There is no
longer a need to directly call @code{asyncio.run()} which would spawn a new event
loop on every invocation:

@example
$ python -m asyncio
asyncio REPL 3.8.0
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> await asyncio.sleep(10, result='hello')
hello
@end example

(Contributed by Yury Selivanov in bpo-37028@footnote{https://bugs.python.org/issue37028}.)

The exception @ref{4d7,,asyncio.CancelledError} now inherits from
@ref{15f,,BaseException} rather than @ref{4d8,,Exception} and no longer inherits
from @ref{4d9,,concurrent.futures.CancelledError}.
(Contributed by Yury Selivanov in bpo-32528@footnote{https://bugs.python.org/issue32528}.)

On Windows, the default event loop is now @ref{4da,,ProactorEventLoop}.
(Contributed by Victor Stinner in bpo-34687@footnote{https://bugs.python.org/issue34687}.)

@ref{4da,,ProactorEventLoop} now also supports UDP.
(Contributed by Adam Meily and Andrew Svetlov in bpo-29883@footnote{https://bugs.python.org/issue29883}.)

@ref{4da,,ProactorEventLoop} can now be interrupted by
@ref{4ca,,KeyboardInterrupt} (“CTRL+C”).
(Contributed by Vladimir Matveev in bpo-23057@footnote{https://bugs.python.org/issue23057}.)

Added @ref{4db,,asyncio.Task.get_coro()} for getting the wrapped coroutine
within an @ref{317,,asyncio.Task}.
(Contributed by Alex Grönholm in bpo-36999@footnote{https://bugs.python.org/issue36999}.)

Asyncio tasks can now be named, either by passing the @code{name} keyword
argument to @ref{4dc,,asyncio.create_task()} or
the @ref{4dd,,create_task()} event loop method, or by
calling the @ref{4de,,set_name()} method on the task object. The
task name is visible in the @code{repr()} output of @ref{317,,asyncio.Task} and
can also be retrieved using the @ref{4df,,get_name()} method.
(Contributed by Alex Grönholm in bpo-34270@footnote{https://bugs.python.org/issue34270}.)

Added support for
Happy Eyeballs@footnote{https://en.wikipedia.org/wiki/Happy_Eyeballs} to
@ref{4e0,,asyncio.loop.create_connection()}. To specify the behavior, two new
parameters have been added: @emph{happy_eyeballs_delay} and @emph{interleave}. The Happy
Eyeballs algorithm improves responsiveness in applications that support IPv4
and IPv6 by attempting to simultaneously connect using both.
(Contributed by twisteroid ambassador in bpo-33530@footnote{https://bugs.python.org/issue33530}.)

@node builtins,collections,asyncio<3>,Improved Modules<4>
@anchor{whatsnew/3 8 builtins}@anchor{4e1}
@subsubsection builtins


The @ref{4e2,,compile()} built-in has been improved to accept the
@code{ast.PyCF_ALLOW_TOP_LEVEL_AWAIT} flag. With this new flag passed,
@ref{4e2,,compile()} will allow top-level @code{await}, @code{async for} and @code{async with}
constructs that are usually considered invalid syntax. Asynchronous code object
marked with the @code{CO_COROUTINE} flag may then be returned.
(Contributed by Matthias Bussonnier in bpo-34616@footnote{https://bugs.python.org/issue34616})

@node collections,cProfile,builtins,Improved Modules<4>
@anchor{whatsnew/3 8 collections}@anchor{4e3}
@subsubsection collections


The @ref{4e4,,_asdict()} method for
@ref{3e4,,collections.namedtuple()} now returns a @ref{3b0,,dict} instead of a
@ref{4e5,,collections.OrderedDict}. This works because regular dicts have
guaranteed ordering since Python 3.7. If the extra features of
@code{OrderedDict} are required, the suggested remediation is to cast the
result to the desired type: @code{OrderedDict(nt._asdict())}.
(Contributed by Raymond Hettinger in bpo-35864@footnote{https://bugs.python.org/issue35864}.)

@node cProfile,csv,collections,Improved Modules<4>
@anchor{whatsnew/3 8 cprofile}@anchor{4e6}
@subsubsection cProfile


The @ref{4e7,,cProfile.Profile} class can now be used as a context manager.
Profile a block of code by running:

@example
import cProfile

with cProfile.Profile() as profiler:
      # code to be profiled
      ...
@end example

(Contributed by Scott Sanderson in bpo-29235@footnote{https://bugs.python.org/issue29235}.)

@node csv,curses<3>,cProfile,Improved Modules<4>
@anchor{whatsnew/3 8 csv}@anchor{4e8}
@subsubsection csv


The @ref{4e9,,csv.DictReader} now returns instances of @ref{3b0,,dict} instead of
a @ref{4e5,,collections.OrderedDict}.  The tool is now faster and uses less
memory while still preserving the field order.
(Contributed by Michael Selik in bpo-34003@footnote{https://bugs.python.org/issue34003}.)

@node curses<3>,ctypes,csv,Improved Modules<4>
@anchor{whatsnew/3 8 curses}@anchor{4ea}
@subsubsection curses


Added a new variable holding structured version information for the
underlying ncurses library: @ref{4eb,,ncurses_version}.
(Contributed by Serhiy Storchaka in bpo-31680@footnote{https://bugs.python.org/issue31680}.)

@node ctypes,datetime<2>,curses<3>,Improved Modules<4>
@anchor{whatsnew/3 8 ctypes}@anchor{4ec}
@subsubsection ctypes


On Windows, @ref{4ed,,CDLL} and subclasses now accept a @emph{winmode} parameter
to specify flags for the underlying @code{LoadLibraryEx} call. The default flags are
set to only load DLL dependencies from trusted locations, including the path
where the DLL is stored (if a full or partial path is used to load the initial
DLL) and paths added by @ref{4ee,,add_dll_directory()}.
(Contributed by Steve Dower in bpo-36085@footnote{https://bugs.python.org/issue36085}.)

@node datetime<2>,functools,ctypes,Improved Modules<4>
@anchor{whatsnew/3 8 datetime}@anchor{4ef}
@subsubsection datetime


Added new alternate constructors @ref{4f0,,datetime.date.fromisocalendar()} and
@ref{4f1,,datetime.datetime.fromisocalendar()}, which construct @code{date} and
@ref{2f,,datetime} objects respectively from ISO year, week number, and weekday;
these are the inverse of each class’s @code{isocalendar} method.
(Contributed by Paul Ganssle in bpo-36004@footnote{https://bugs.python.org/issue36004}.)

@node functools,gc<3>,datetime<2>,Improved Modules<4>
@anchor{whatsnew/3 8 functools}@anchor{4f2}
@subsubsection functools


@ref{4f3,,functools.lru_cache()} can now be used as a straight decorator rather
than as a function returning a decorator.  So both of these are now supported:

@example
@@lru_cache
def f(x):
    ...

@@lru_cache(maxsize=256)
def f(x):
    ...
@end example

(Contributed by Raymond Hettinger in bpo-36772@footnote{https://bugs.python.org/issue36772}.)

Added a new @ref{4f4,,functools.cached_property()} decorator, for computed properties
cached for the life of the instance.

@example
import functools
import statistics

class Dataset:
   def __init__(self, sequence_of_numbers):
      self.data = sequence_of_numbers

   @@functools.cached_property
   def variance(self):
      return statistics.variance(self.data)
@end example

(Contributed by Carl Meyer in bpo-21145@footnote{https://bugs.python.org/issue21145})

Added a new @ref{4f5,,functools.singledispatchmethod()} decorator that converts
methods into @ref{4f6,,generic functions} using
@ref{4f7,,single dispatch}:

@example
from functools import singledispatchmethod
from contextlib import suppress

class TaskManager:

    def __init__(self, tasks):
        self.tasks = list(tasks)

    @@singledispatchmethod
    def discard(self, value):
        with suppress(ValueError):
            self.tasks.remove(value)

    @@discard.register(list)
    def _(self, tasks):
        targets = set(tasks)
        self.tasks = [x for x in self.tasks if x not in targets]
@end example

(Contributed by Ethan Smith in bpo-32380@footnote{https://bugs.python.org/issue32380})

@node gc<3>,gettext,functools,Improved Modules<4>
@anchor{whatsnew/3 8 gc}@anchor{4f8}
@subsubsection gc


@ref{284,,get_objects()} can now receive an optional @emph{generation} parameter
indicating a generation to get objects from.
(Contributed by Pablo Galindo in bpo-36016@footnote{https://bugs.python.org/issue36016}.)

@node gettext,gzip,gc<3>,Improved Modules<4>
@anchor{whatsnew/3 8 gettext}@anchor{4f9}
@subsubsection gettext


Added @ref{4fa,,pgettext()} and its variants.
(Contributed by Franz Glasner, Éric Araujo, and Cheryl Sabella in bpo-2504@footnote{https://bugs.python.org/issue2504}.)

@node gzip,IDLE and idlelib<4>,gettext,Improved Modules<4>
@anchor{whatsnew/3 8 gzip}@anchor{4fb}
@subsubsection gzip


Added the @emph{mtime} parameter to @ref{4fc,,gzip.compress()} for reproducible output.
(Contributed by Guo Ci Teo in bpo-34898@footnote{https://bugs.python.org/issue34898}.)

A @ref{4fd,,BadGzipFile} exception is now raised instead of @ref{2a1,,OSError}
for certain types of invalid or corrupt gzip files.
(Contributed by Filip Gruszczyński, Michele Orrù, and Zackery Spytz in
bpo-6584@footnote{https://bugs.python.org/issue6584}.)

@node IDLE and idlelib<4>,inspect<4>,gzip,Improved Modules<4>
@anchor{whatsnew/3 8 idle-and-idlelib}@anchor{4fe}
@subsubsection IDLE and idlelib


Output over N lines (50 by default) is squeezed down to a button.
N can be changed in the PyShell section of the General page of the
Settings dialog.  Fewer, but possibly extra long, lines can be squeezed by
right clicking on the output.  Squeezed output can be expanded in place
by double-clicking the button or into the clipboard or a separate window
by right-clicking the button.  (Contributed by Tal Einat in bpo-1529353@footnote{https://bugs.python.org/issue1529353}.)

Add “Run Customized” to the Run menu to run a module with customized
settings. Any command line arguments entered are added to sys.argv.
They also re-appear in the box for the next customized run.  One can also
suppress the normal Shell main module restart.  (Contributed by Cheryl
Sabella, Terry Jan Reedy, and others in bpo-5680@footnote{https://bugs.python.org/issue5680} and bpo-37627@footnote{https://bugs.python.org/issue37627}.)

Added optional line numbers for IDLE editor windows. Windows
open without line numbers unless set otherwise in the General
tab of the configuration dialog.  Line numbers for an existing
window are shown and hidden in the Options menu.
(Contributed by Tal Einat and Saimadhav Heblikar in bpo-17535@footnote{https://bugs.python.org/issue17535}.)

OS native encoding is now used for converting between Python strings and Tcl
objects. This allows IDLE to work with emoji and other non-BMP characters.
These characters can be displayed or copied and pasted to or from the
clipboard.  Converting strings from Tcl to Python and back now never fails.
(Many people worked on this for eight years but the problem was finally
solved by Serhiy Storchaka in bpo-13153@footnote{https://bugs.python.org/issue13153}.)

New in 3.8.1:

Add option to toggle cursor blink off.  (Contributed by Zackery Spytz
in bpo-4603@footnote{https://bugs.python.org/issue4603}.)

Escape key now closes IDLE completion windows.  (Contributed by Johnny
Najera in bpo-38944@footnote{https://bugs.python.org/issue38944}.)

The changes above have been backported to 3.7 maintenance releases.

Add keywords to module name completion list.  (Contributed by Terry J.
Reedy in bpo-37765@footnote{https://bugs.python.org/issue37765}.)

@node inspect<4>,io,IDLE and idlelib<4>,Improved Modules<4>
@anchor{whatsnew/3 8 inspect}@anchor{4ff}
@subsubsection inspect


The @ref{500,,inspect.getdoc()} function can now find docstrings for @code{__slots__}
if that attribute is a @ref{3b0,,dict} where the values are docstrings.
This provides documentation options similar to what we already have
for @ref{501,,property()}, @ref{254,,classmethod()}, and @ref{253,,staticmethod()}:

@example
class AudioClip:
    __slots__ = @{'bit_rate': 'expressed in kilohertz to one decimal place',
                 'duration': 'in seconds, rounded up to an integer'@}
    def __init__(self, bit_rate, duration):
        self.bit_rate = round(bit_rate / 1000.0, 1)
        self.duration = ceil(duration)
@end example

(Contributed by Raymond Hettinger in bpo-36326@footnote{https://bugs.python.org/issue36326}.)

@node io,itertools,inspect<4>,Improved Modules<4>
@anchor{whatsnew/3 8 io}@anchor{502}
@subsubsection io


In development mode (@ref{159,,-X} @code{env}) and in @ref{338,,debug build}, the
@ref{503,,io.IOBase} finalizer now logs the exception if the @code{close()} method
fails. The exception is ignored silently by default in release build.
(Contributed by Victor Stinner in bpo-18748@footnote{https://bugs.python.org/issue18748}.)

@node itertools,json tool,io,Improved Modules<4>
@anchor{whatsnew/3 8 itertools}@anchor{504}
@subsubsection itertools


The @ref{505,,itertools.accumulate()} function added an option @emph{initial} keyword
argument to specify an initial value:

@example
>>> from itertools import accumulate
>>> list(accumulate([10, 5, 30, 15], initial=1000))
[1000, 1010, 1015, 1045, 1060]
@end example

(Contributed by Lisa Roach in bpo-34659@footnote{https://bugs.python.org/issue34659}.)

@node json tool,logging,itertools,Improved Modules<4>
@anchor{whatsnew/3 8 json-tool}@anchor{506}
@subsubsection json.tool


Add option @code{--json-lines} to parse every input line as a separate JSON object.
(Contributed by Weipeng Hong in bpo-31553@footnote{https://bugs.python.org/issue31553}.)

@node logging,math<3>,json tool,Improved Modules<4>
@anchor{whatsnew/3 8 logging}@anchor{507}
@subsubsection logging


Added a @emph{force} keyword argument to @ref{508,,logging.basicConfig()}
When set to true, any existing handlers attached
to the root logger are removed and closed before carrying out the
configuration specified by the other arguments.

This solves a long-standing problem.  Once a logger or @emph{basicConfig()} had
been called, subsequent calls to @emph{basicConfig()} were silently ignored.
This made it difficult to update, experiment with, or teach the various
logging configuration options using the interactive prompt or a Jupyter
notebook.

(Suggested by Raymond Hettinger, implemented by Dong-hee Na, and
reviewed by Vinay Sajip in bpo-33897@footnote{https://bugs.python.org/issue33897}.)

@node math<3>,mmap,logging,Improved Modules<4>
@anchor{whatsnew/3 8 math}@anchor{509}
@subsubsection math


Added new function @ref{50a,,math.dist()} for computing Euclidean distance
between two points.  (Contributed by Raymond Hettinger in bpo-33089@footnote{https://bugs.python.org/issue33089}.)

Expanded the @ref{50b,,math.hypot()} function to handle multiple dimensions.
Formerly, it only supported the 2-D case.
(Contributed by Raymond Hettinger in bpo-33089@footnote{https://bugs.python.org/issue33089}.)

Added new function, @ref{50c,,math.prod()}, as analogous function to @ref{50d,,sum()}
that returns the product of a ‘start’ value (default: 1) times an iterable of
numbers:

@example
>>> prior = 0.8
>>> likelihoods = [0.625, 0.84, 0.30]
>>> math.prod(likelihoods, start=prior)
0.126
@end example

(Contributed by Pablo Galindo in bpo-35606@footnote{https://bugs.python.org/issue35606}.)

Added two new combinatoric functions @ref{19c,,math.perm()} and @ref{19b,,math.comb()}:

@example
>>> math.perm(10, 3)    # Permutations of 10 things taken 3 at a time
720
>>> math.comb(10, 3)    # Combinations of 10 things taken 3 at a time
120
@end example

(Contributed by Yash Aggarwal, Keller Fuchs, Serhiy Storchaka, and Raymond
Hettinger in bpo-37128@footnote{https://bugs.python.org/issue37128}, bpo-37178@footnote{https://bugs.python.org/issue37178}, and bpo-35431@footnote{https://bugs.python.org/issue35431}.)

Added a new function @ref{50e,,math.isqrt()} for computing accurate integer square
roots without conversion to floating point.  The new function supports
arbitrarily large integers.  It is faster than @code{floor(sqrt(n))} but slower
than @ref{50f,,math.sqrt()}:

@example
>>> r = 650320427
>>> s = r ** 2
>>> isqrt(s - 1)         # correct
650320426
>>> floor(sqrt(s - 1))   # incorrect
650320427
@end example

(Contributed by Mark Dickinson in bpo-36887@footnote{https://bugs.python.org/issue36887}.)

The function @ref{43e,,math.factorial()} no longer accepts arguments that are not
int-like. (Contributed by Pablo Galindo in bpo-33083@footnote{https://bugs.python.org/issue33083}.)

@node mmap,multiprocessing<2>,math<3>,Improved Modules<4>
@anchor{whatsnew/3 8 mmap}@anchor{510}
@subsubsection mmap


The @ref{511,,mmap.mmap} class now has an @ref{512,,madvise()} method to
access the @code{madvise()} system call.
(Contributed by Zackery Spytz in bpo-32941@footnote{https://bugs.python.org/issue32941}.)

@node multiprocessing<2>,os<4>,mmap,Improved Modules<4>
@anchor{whatsnew/3 8 multiprocessing}@anchor{513}
@subsubsection multiprocessing


Added new @ref{b8,,multiprocessing.shared_memory} module.
(Contributed by Davin Potts in bpo-35813@footnote{https://bugs.python.org/issue35813}.)

On macOS, the @emph{spawn} start method is now used by default.
(Contributed by Victor Stinner in bpo-33725@footnote{https://bugs.python.org/issue33725}.)

@node os<4>,os path<2>,multiprocessing<2>,Improved Modules<4>
@anchor{whatsnew/3 8 os}@anchor{514}
@subsubsection os


Added new function @ref{4ee,,add_dll_directory()} on Windows for providing
additional search paths for native dependencies when importing extension
modules or loading DLLs using @ref{29,,ctypes}.
(Contributed by Steve Dower in bpo-36085@footnote{https://bugs.python.org/issue36085}.)

A new @ref{515,,os.memfd_create()} function was added to wrap the
@code{memfd_create()} syscall.
(Contributed by Zackery Spytz and Christian Heimes in bpo-26836@footnote{https://bugs.python.org/issue26836}.)

On Windows, much of the manual logic for handling reparse points (including
symlinks and directory junctions) has been delegated to the operating system.
Specifically, @ref{516,,os.stat()} will now traverse anything supported by the
operating system, while @ref{517,,os.lstat()} will only open reparse points that
identify as “name surrogates” while others are opened as for @ref{516,,os.stat()}.
In all cases, @code{stat_result.st_mode} will only have @code{S_IFLNK} set for
symbolic links and not other kinds of reparse points. To identify other kinds
of reparse point, check the new @code{stat_result.st_reparse_tag} attribute.

On Windows, @ref{414,,os.readlink()} is now able to read directory junctions. Note
that @ref{518,,islink()} will return @code{False} for directory junctions,
and so code that checks @code{islink} first will continue to treat junctions as
directories, while code that handles errors from @ref{414,,os.readlink()} may now
treat junctions as links.

(Contributed by Steve Dower in bpo-37834@footnote{https://bugs.python.org/issue37834}.)

@node os path<2>,pathlib<3>,os<4>,Improved Modules<4>
@anchor{whatsnew/3 8 os-path}@anchor{519}
@subsubsection os.path


@ref{c1,,os.path} functions that return a boolean result like
@ref{51a,,exists()}, @ref{51b,,lexists()}, @ref{51c,,isdir()},
@ref{51d,,isfile()}, @ref{518,,islink()}, and @ref{51e,,ismount()}
now return @code{False} instead of raising @ref{17c,,ValueError} or its subclasses
@ref{185,,UnicodeEncodeError} and @ref{51f,,UnicodeDecodeError} for paths that contain
characters or bytes unrepresentable at the OS level.
(Contributed by Serhiy Storchaka in bpo-33721@footnote{https://bugs.python.org/issue33721}.)

@ref{520,,expanduser()} on Windows now prefers the 
@geindex USERPROFILE
@geindex environment variable; USERPROFILE
@code{USERPROFILE}
environment variable and does not use 
@geindex HOME
@geindex environment variable; HOME
@code{HOME}, which is not normally set
for regular user accounts.
(Contributed by Anthony Sottile in bpo-36264@footnote{https://bugs.python.org/issue36264}.)

@ref{51c,,isdir()} on Windows no longer returns @code{True} for a link to a
non-existent directory.

@ref{2a0,,realpath()} on Windows now resolves reparse points, including
symlinks and directory junctions.

(Contributed by Steve Dower in bpo-37834@footnote{https://bugs.python.org/issue37834}.)

@node pathlib<3>,pickle,os path<2>,Improved Modules<4>
@anchor{whatsnew/3 8 pathlib}@anchor{521}
@subsubsection pathlib


@ref{522,,pathlib.Path} methods that return a boolean result like
@ref{523,,exists()}, @ref{524,,is_dir()},
@ref{525,,is_file()}, @ref{526,,is_mount()},
@ref{527,,is_symlink()}, @ref{528,,is_block_device()},
@ref{529,,is_char_device()}, @ref{52a,,is_fifo()},
@ref{52b,,is_socket()} now return @code{False} instead of raising
@ref{17c,,ValueError} or its subclass @ref{185,,UnicodeEncodeError} for paths that
contain characters unrepresentable at the OS level.
(Contributed by Serhiy Storchaka in bpo-33721@footnote{https://bugs.python.org/issue33721}.)

Added @ref{2a5,,pathlib.Path.link_to()} which creates a hard link pointing
to a path.
(Contributed by Joannah Nanjekye in bpo-26978@footnote{https://bugs.python.org/issue26978})

@node pickle,plistlib,pathlib<3>,Improved Modules<4>
@anchor{whatsnew/3 8 pickle}@anchor{52c}
@subsubsection pickle


@ref{c5,,pickle} extensions subclassing the C-optimized @ref{52d,,Pickler}
can now override the pickling logic of functions and classes by defining the
special @ref{52e,,reducer_override()} method.
(Contributed by Pierre Glaser and Olivier Grisel in bpo-35900@footnote{https://bugs.python.org/issue35900}.)

@node plistlib,pprint<3>,pickle,Improved Modules<4>
@anchor{whatsnew/3 8 plistlib}@anchor{52f}
@subsubsection plistlib


Added new @ref{530,,plistlib.UID} and enabled support for reading and writing
NSKeyedArchiver-encoded binary plists.
(Contributed by Jon Janzen in bpo-26707@footnote{https://bugs.python.org/issue26707}.)

@node pprint<3>,py_compile<2>,plistlib,Improved Modules<4>
@anchor{whatsnew/3 8 pprint}@anchor{531}
@subsubsection pprint


The @ref{cd,,pprint} module added a @emph{sort_dicts} parameter to several functions.
By default, those functions continue to sort dictionaries before rendering or
printing.  However, if @emph{sort_dicts} is set to false, the dictionaries retain
the order that keys were inserted.  This can be useful for comparison to JSON
inputs during debugging.

In addition, there is a convenience new function, @ref{532,,pprint.pp()} that is
like @ref{2ac,,pprint.pprint()} but with @emph{sort_dicts} defaulting to @code{False}:

@example
>>> from pprint import pprint, pp
>>> d = dict(source='input.txt', operation='filter', destination='output.txt')
>>> pp(d, width=40)                  # Original order
@{'source': 'input.txt',
 'operation': 'filter',
 'destination': 'output.txt'@}
>>> pprint(d, width=40)              # Keys sorted alphabetically
@{'destination': 'output.txt',
 'operation': 'filter',
 'source': 'input.txt'@}
@end example

(Contributed by Rémi Lapeyre in bpo-30670@footnote{https://bugs.python.org/issue30670}.)

@node py_compile<2>,shlex,pprint<3>,Improved Modules<4>
@anchor{whatsnew/3 8 py-compile}@anchor{533}
@subsubsection py_compile


@ref{534,,py_compile.compile()} now supports silent mode.
(Contributed by Joannah Nanjekye in bpo-22640@footnote{https://bugs.python.org/issue22640}.)

@node shlex,shutil,py_compile<2>,Improved Modules<4>
@anchor{whatsnew/3 8 shlex}@anchor{535}
@subsubsection shlex


The new @ref{536,,shlex.join()} function acts as the inverse of @ref{443,,shlex.split()}.
(Contributed by Bo Bayles in bpo-32102@footnote{https://bugs.python.org/issue32102}.)

@node shutil,socket<4>,shlex,Improved Modules<4>
@anchor{whatsnew/3 8 shutil}@anchor{537}
@subsubsection shutil


@ref{538,,shutil.copytree()} now accepts a new @code{dirs_exist_ok} keyword argument.
(Contributed by Josh Bronson in bpo-20849@footnote{https://bugs.python.org/issue20849}.)

@ref{539,,shutil.make_archive()} now defaults to the modern pax (POSIX.1-2001)
format for new archives to improve portability and standards conformance,
inherited from the corresponding change to the @ref{fb,,tarfile} module.
(Contributed by C.A.M. Gerlach in bpo-30661@footnote{https://bugs.python.org/issue30661}.)

@ref{53a,,shutil.rmtree()} on Windows now removes directory junctions without
recursively removing their contents first.
(Contributed by Steve Dower in bpo-37834@footnote{https://bugs.python.org/issue37834}.)

@node socket<4>,ssl<2>,shutil,Improved Modules<4>
@anchor{whatsnew/3 8 socket}@anchor{53b}
@subsubsection socket


Added @ref{53c,,create_server()} and @ref{53d,,has_dualstack_ipv6()}
convenience functions to automate the necessary tasks usually involved when
creating a server socket, including accepting both IPv4 and IPv6 connections
on the same socket.  (Contributed by Giampaolo Rodolà in bpo-17561@footnote{https://bugs.python.org/issue17561}.)

The @ref{53e,,socket.if_nameindex()}, @ref{53f,,socket.if_nametoindex()}, and
@ref{540,,socket.if_indextoname()} functions have been implemented on Windows.
(Contributed by Zackery Spytz in bpo-37007@footnote{https://bugs.python.org/issue37007}.)

@node ssl<2>,statistics<2>,socket<4>,Improved Modules<4>
@anchor{whatsnew/3 8 ssl}@anchor{541}
@subsubsection ssl


Added @ref{542,,post_handshake_auth} to enable and
@ref{543,,verify_client_post_handshake()} to initiate TLS 1.3
post-handshake authentication.
(Contributed by Christian Heimes in bpo-34670@footnote{https://bugs.python.org/issue34670}.)

@node statistics<2>,sys<4>,ssl<2>,Improved Modules<4>
@anchor{whatsnew/3 8 statistics}@anchor{544}
@subsubsection statistics


Added @ref{545,,statistics.fmean()} as a faster, floating point variant of
@ref{4cc,,statistics.mean()}.  (Contributed by Raymond Hettinger and
Steven D’Aprano in bpo-35904@footnote{https://bugs.python.org/issue35904}.)

Added @ref{546,,statistics.geometric_mean()}
(Contributed by Raymond Hettinger in bpo-27181@footnote{https://bugs.python.org/issue27181}.)

Added @ref{547,,statistics.multimode()} that returns a list of the most
common values. (Contributed by Raymond Hettinger in bpo-35892@footnote{https://bugs.python.org/issue35892}.)

Added @ref{548,,statistics.quantiles()} that divides data or a distribution
in to equiprobable intervals (e.g. quartiles, deciles, or percentiles).
(Contributed by Raymond Hettinger in bpo-36546@footnote{https://bugs.python.org/issue36546}.)

Added @ref{549,,statistics.NormalDist}, a tool for creating
and manipulating normal distributions of a random variable.
(Contributed by Raymond Hettinger in bpo-36018@footnote{https://bugs.python.org/issue36018}.)

@example
>>> temperature_feb = NormalDist.from_samples([4, 12, -3, 2, 7, 14])
>>> temperature_feb.mean
6.0
>>> temperature_feb.stdev
6.356099432828281

>>> temperature_feb.cdf(3)            # Chance of being under 3 degrees
0.3184678262814532
>>> # Relative chance of being 7 degrees versus 10 degrees
>>> temperature_feb.pdf(7) / temperature_feb.pdf(10)
1.2039930378537762

>>> el_niño = NormalDist(4, 2.5)
>>> temperature_feb += el_niño        # Add in a climate effect
>>> temperature_feb
NormalDist(mu=10.0, sigma=6.830080526611674)

>>> temperature_feb * (9/5) + 32      # Convert to Fahrenheit
NormalDist(mu=50.0, sigma=12.294144947901014)
>>> temperature_feb.samples(3)        # Generate random samples
[7.672102882379219, 12.000027119750287, 4.647488369766392]
@end example

@node sys<4>,tarfile,statistics<2>,Improved Modules<4>
@anchor{whatsnew/3 8 sys}@anchor{54a}
@subsubsection sys


Add new @ref{18d,,sys.unraisablehook()} function which can be overridden to control
how “unraisable exceptions” are handled. It is called when an exception has
occurred but there is no way for Python to handle it. For example, when a
destructor raises an exception or during garbage collection
(@ref{54b,,gc.collect()}).
(Contributed by Victor Stinner in bpo-36829@footnote{https://bugs.python.org/issue36829}.)

@node tarfile,threading<3>,sys<4>,Improved Modules<4>
@anchor{whatsnew/3 8 tarfile}@anchor{54c}
@subsubsection tarfile


The @ref{fb,,tarfile} module now defaults to the modern pax (POSIX.1-2001)
format for new archives, instead of the previous GNU-specific one.
This improves cross-platform portability with a consistent encoding (UTF-8)
in a standardized and extensible format, and offers several other benefits.
(Contributed by C.A.M. Gerlach in bpo-36268@footnote{https://bugs.python.org/issue36268}.)

@node threading<3>,tokenize,tarfile,Improved Modules<4>
@anchor{whatsnew/3 8 threading}@anchor{54d}
@subsubsection threading


Add a new @ref{2d1,,threading.excepthook()} function which handles uncaught
@ref{54e,,threading.Thread.run()} exception. It can be overridden to control how
uncaught @ref{54e,,threading.Thread.run()} exceptions are handled.
(Contributed by Victor Stinner in bpo-1230540@footnote{https://bugs.python.org/issue1230540}.)

Add a new @ref{54f,,threading.get_native_id()} function and
a @ref{550,,native_id}
attribute to the @ref{44b,,threading.Thread} class. These return the native
integral Thread ID of the current thread assigned by the kernel.
This feature is only available on certain platforms, see
@ref{54f,,get_native_id} for more information.
(Contributed by Jake Tesler in bpo-36084@footnote{https://bugs.python.org/issue36084}.)

@node tokenize,tkinter,threading<3>,Improved Modules<4>
@anchor{whatsnew/3 8 tokenize}@anchor{551}
@subsubsection tokenize


The @ref{118,,tokenize} module now implicitly emits a @code{NEWLINE} token when
provided with input that does not have a trailing new line.  This behavior
now matches what the C tokenizer does internally.
(Contributed by Ammar Askar in bpo-33899@footnote{https://bugs.python.org/issue33899}.)

@node tkinter,time<3>,tokenize,Improved Modules<4>
@anchor{whatsnew/3 8 tkinter}@anchor{552}
@subsubsection tkinter


Added methods @code{selection_from()},
@code{selection_present()},
@code{selection_range()} and
@code{selection_to()}
in the @code{tkinter.Spinbox} class.
(Contributed by Juliette Monsel in bpo-34829@footnote{https://bugs.python.org/issue34829}.)

Added method @code{moveto()}
in the @code{tkinter.Canvas} class.
(Contributed by Juliette Monsel in bpo-23831@footnote{https://bugs.python.org/issue23831}.)

The @code{tkinter.PhotoImage} class now has
@code{transparency_get()} and
@code{transparency_set()} methods.  (Contributed by
Zackery Spytz in bpo-25451@footnote{https://bugs.python.org/issue25451}.)

@node time<3>,typing<4>,tkinter,Improved Modules<4>
@anchor{whatsnew/3 8 time}@anchor{553}
@subsubsection time


Added new clock @ref{554,,CLOCK_UPTIME_RAW} for macOS 10.12.
(Contributed by Joannah Nanjekye in bpo-35702@footnote{https://bugs.python.org/issue35702}.)

@node typing<4>,unicodedata<3>,time<3>,Improved Modules<4>
@anchor{whatsnew/3 8 typing}@anchor{555}
@subsubsection typing


The @ref{120,,typing} module incorporates several new features:


@itemize *

@item 
A dictionary type with per-key types.  See 
@geindex Python Enhancement Proposals; PEP 589
PEP 589@footnote{https://www.python.org/dev/peps/pep-0589} and
@ref{2de,,typing.TypedDict}.
TypedDict uses only string keys.  By default, every key is required
to be present. Specify “total=False” to allow keys to be optional:

@example
class Location(TypedDict, total=False):
    lat_long: tuple
    grid_square: str
    xy_coordinate: tuple
@end example

@item 
Literal types.  See 
@geindex Python Enhancement Proposals; PEP 586
PEP 586@footnote{https://www.python.org/dev/peps/pep-0586} and @ref{2db,,typing.Literal}.
Literal types indicate that a parameter or return value
is constrained to one or more specific literal values:

@example
def get_status(port: int) -> Literal['connected', 'disconnected']:
    ...
@end example

@item 
“Final” variables, functions, methods and classes.  See 
@geindex Python Enhancement Proposals; PEP 591
PEP 591@footnote{https://www.python.org/dev/peps/pep-0591},
@ref{556,,typing.Final} and @ref{557,,typing.final()}.
The final qualifier instructs a static type checker to restrict
subclassing, overriding, or reassignment:

@example
pi: Final[float] = 3.1415926536
@end example

@item 
Protocol definitions.  See 
@geindex Python Enhancement Proposals; PEP 544
PEP 544@footnote{https://www.python.org/dev/peps/pep-0544}, @ref{558,,typing.Protocol} and
@ref{559,,typing.runtime_checkable()}.  Simple ABCs like
@ref{55a,,typing.SupportsInt} are now @code{Protocol} subclasses.

@item 
New protocol class @ref{55b,,typing.SupportsIndex}.

@item 
New functions @ref{55c,,typing.get_origin()} and @ref{344,,typing.get_args()}.
@end itemize

@node unicodedata<3>,unittest<2>,typing<4>,Improved Modules<4>
@anchor{whatsnew/3 8 unicodedata}@anchor{55d}
@subsubsection unicodedata


The @ref{121,,unicodedata} module has been upgraded to use the Unicode 12.1.0@footnote{http://blog.unicode.org/2019/05/unicode-12-1-en.html} release.

New function @ref{55e,,is_normalized()} can be used to verify a string
is in a specific normal form, often much faster than by actually normalizing
the string.  (Contributed by Max Belanger, David Euresti, and Greg Price in
bpo-32285@footnote{https://bugs.python.org/issue32285} and bpo-37966@footnote{https://bugs.python.org/issue37966}).

@node unittest<2>,venv<2>,unicodedata<3>,Improved Modules<4>
@anchor{whatsnew/3 8 unittest}@anchor{55f}
@subsubsection unittest


Added @ref{560,,AsyncMock} to support an asynchronous version of
@ref{561,,Mock}.  Appropriate new assert functions for testing
have been added as well.
(Contributed by Lisa Roach in bpo-26467@footnote{https://bugs.python.org/issue26467}).

Added @ref{562,,addModuleCleanup()} and
@ref{563,,addClassCleanup()} to unittest to support
cleanups for @code{setUpModule()} and
@ref{564,,setUpClass()}.
(Contributed by Lisa Roach in bpo-24412@footnote{https://bugs.python.org/issue24412}.)

Several mock assert functions now also print a list of actual calls upon
failure. (Contributed by Petter Strandmark in bpo-35047@footnote{https://bugs.python.org/issue35047}.)

@ref{122,,unittest} module gained support for coroutines to be used as test cases
with @ref{1ac,,unittest.IsolatedAsyncioTestCase}.
(Contributed by Andrew Svetlov in bpo-32972@footnote{https://bugs.python.org/issue32972}.)

Example:

@example
import unittest


class TestRequest(unittest.IsolatedAsyncioTestCase):

    async def asyncSetUp(self):
        self.connection = await AsyncConnection()

    async def test_get(self):
        response = await self.connection.get("https://example.com")
        self.assertEqual(response.status_code, 200)

    async def asyncTearDown(self):
        await self.connection.close()


if __name__ == "__main__":
    unittest.main()
@end example

@node venv<2>,weakref,unittest<2>,Improved Modules<4>
@anchor{whatsnew/3 8 venv}@anchor{565}
@subsubsection venv


@ref{12c,,venv} now includes an @code{Activate.ps1} script on all platforms for
activating virtual environments under PowerShell Core 6.1.
(Contributed by Brett Cannon in bpo-32718@footnote{https://bugs.python.org/issue32718}.)

@node weakref,xml<3>,venv<2>,Improved Modules<4>
@anchor{whatsnew/3 8 weakref}@anchor{566}
@subsubsection weakref


The proxy objects returned by @ref{567,,weakref.proxy()} now support the matrix
multiplication operators @code{@@} and @code{@@=} in addition to the other
numeric operators. (Contributed by Mark Dickinson in bpo-36669@footnote{https://bugs.python.org/issue36669}.)

@node xml<3>,xmlrpc,weakref,Improved Modules<4>
@anchor{whatsnew/3 8 xml}@anchor{568}
@subsubsection xml


As mitigation against DTD and external entity retrieval, the
@ref{13c,,xml.dom.minidom} and @ref{142,,xml.sax} modules no longer process
external entities by default.
(Contributed by Christian Heimes in bpo-17239@footnote{https://bugs.python.org/issue17239}.)

The @code{.find*()} methods in the @ref{13e,,xml.etree.ElementTree} module
support wildcard searches like @code{@{*@}tag} which ignores the namespace
and @code{@{namespace@}*} which returns all tags in the given namespace.
(Contributed by Stefan Behnel in bpo-28238@footnote{https://bugs.python.org/issue28238}.)

The @ref{13e,,xml.etree.ElementTree} module provides a new function
@code{–xml.etree.ElementTree.canonicalize()} that implements C14N 2.0.
(Contributed by Stefan Behnel in bpo-13611@footnote{https://bugs.python.org/issue13611}.)

The target object of @ref{569,,xml.etree.ElementTree.XMLParser} can
receive namespace declaration events through the new callback methods
@code{start_ns()} and @code{end_ns()}.  Additionally, the
@ref{56a,,xml.etree.ElementTree.TreeBuilder} target can be configured
to process events about comments and processing instructions to include
them in the generated tree.
(Contributed by Stefan Behnel in bpo-36676@footnote{https://bugs.python.org/issue36676} and bpo-36673@footnote{https://bugs.python.org/issue36673}.)

@node xmlrpc,,xml<3>,Improved Modules<4>
@anchor{whatsnew/3 8 xmlrpc}@anchor{56b}
@subsubsection xmlrpc


@ref{56c,,xmlrpc.client.ServerProxy} now supports an optional @emph{headers} keyword
argument for a sequence of HTTP headers to be sent with each request.  Among
other things, this makes it possible to upgrade from default basic
authentication to faster session authentication.
(Contributed by Cédric Krier in bpo-35153@footnote{https://bugs.python.org/issue35153}.)

@node Optimizations<4>,Build and C API Changes,Improved Modules<4>,What’s New In Python 3 8
@anchor{whatsnew/3 8 optimizations}@anchor{56d}
@subsection Optimizations



@itemize *

@item 
The @ref{f4,,subprocess} module can now use the @ref{56e,,os.posix_spawn()} function
in some cases for better performance. Currently, it is only used on macOS
and Linux (using glibc 2.24 or newer) if all these conditions are met:


@itemize *

@item 
@emph{close_fds} is false;

@item 
@emph{preexec_fn}, @emph{pass_fds}, @emph{cwd} and @emph{start_new_session} parameters
are not set;

@item 
the @emph{executable} path contains a directory.
@end itemize

(Contributed by Joannah Nanjekye and Victor Stinner in bpo-35537@footnote{https://bugs.python.org/issue35537}.)

@item 
@ref{56f,,shutil.copyfile()}, @ref{570,,shutil.copy()}, @ref{571,,shutil.copy2()},
@ref{538,,shutil.copytree()} and @ref{572,,shutil.move()} use platform-specific
“fast-copy” syscalls on Linux and macOS in order to copy the file
more efficiently.
“fast-copy” means that the copying operation occurs within the kernel,
avoiding the use of userspace buffers in Python as in
“@code{outfd.write(infd.read())}”.
On Windows @ref{56f,,shutil.copyfile()} uses a bigger default buffer size (1 MiB
instead of 16 KiB) and a @ref{573,,memoryview()}-based variant of
@ref{574,,shutil.copyfileobj()} is used.
The speedup for copying a 512 MiB file within the same partition is about
+26% on Linux, +50% on macOS and +40% on Windows. Also, much less CPU cycles
are consumed.
See @ref{575,,Platform-dependent efficient copy operations} section.
(Contributed by Giampaolo Rodolà in bpo-33671@footnote{https://bugs.python.org/issue33671}.)

@item 
@ref{538,,shutil.copytree()} uses @ref{576,,os.scandir()} function and all copy
functions depending from it use cached @ref{516,,os.stat()} values. The speedup
for copying a directory with 8000 files is around +9% on Linux, +20% on
Windows and +30% on a Windows SMB share. Also the number of @ref{516,,os.stat()}
syscalls is reduced by 38% making @ref{538,,shutil.copytree()} especially faster
on network filesystems. (Contributed by Giampaolo Rodolà in bpo-33695@footnote{https://bugs.python.org/issue33695}.)

@item 
The default protocol in the @ref{c5,,pickle} module is now Protocol 4,
first introduced in Python 3.4.  It offers better performance and smaller
size compared to Protocol 3 available since Python 3.0.

@item 
Removed one @code{Py_ssize_t} member from @code{PyGC_Head}.  All GC tracked
objects (e.g. tuple, list, dict) size is reduced 4 or 8 bytes.
(Contributed by Inada Naoki in bpo-33597@footnote{https://bugs.python.org/issue33597}.)

@item 
@ref{577,,uuid.UUID} now uses @code{__slots__} to reduce its memory footprint.
(Contributed by Wouter Bolsterlee and Tal Einat in bpo-30977@footnote{https://bugs.python.org/issue30977})

@item 
Improved performance of @ref{578,,operator.itemgetter()} by 33%.  Optimized
argument handling and added a fast path for the common case of a single
non-negative integer index into a tuple (which is the typical use case in
the standard library).  (Contributed by Raymond Hettinger in
bpo-35664@footnote{https://bugs.python.org/issue35664}.)

@item 
Sped-up field lookups in @ref{3e4,,collections.namedtuple()}.  They are now more
than two times faster, making them the fastest form of instance variable
lookup in Python. (Contributed by Raymond Hettinger, Pablo Galindo, and
Joe Jevnik, Serhiy Storchaka in bpo-32492@footnote{https://bugs.python.org/issue32492}.)

@item 
The @ref{43a,,list} constructor does not overallocate the internal item buffer
if the input iterable has a known length (the input implements @code{__len__}).
This makes the created list 12% smaller on average. (Contributed by
Raymond Hettinger and Pablo Galindo in bpo-33234@footnote{https://bugs.python.org/issue33234}.)

@item 
Doubled the speed of class variable writes.  When a non-dunder attribute
was updated, there was an unnecessary call to update slots.
(Contributed by Stefan Behnel, Pablo Galindo Salgado, Raymond Hettinger,
Neil Schemenauer, and Serhiy Storchaka in bpo-36012@footnote{https://bugs.python.org/issue36012}.)

@item 
Reduced an overhead of converting arguments passed to many builtin functions
and methods.  This sped up calling some simple builtin functions and
methods up to 20–50%.  (Contributed by Serhiy Storchaka in bpo-23867@footnote{https://bugs.python.org/issue23867},
bpo-35582@footnote{https://bugs.python.org/issue35582} and bpo-36127@footnote{https://bugs.python.org/issue36127}.)

@item 
@code{LOAD_GLOBAL} instruction now uses new “per opcode cache” mechanism.
It is about 40% faster now.  (Contributed by Yury Selivanov and Inada Naoki in
bpo-26219@footnote{https://bugs.python.org/issue26219}.)
@end itemize

@node Build and C API Changes,Deprecated<6>,Optimizations<4>,What’s New In Python 3 8
@anchor{whatsnew/3 8 build-and-c-api-changes}@anchor{579}
@subsection Build and C API Changes



@itemize *

@item 
Default @ref{57a,,sys.abiflags} became an empty string: the @code{m} flag for
pymalloc became useless (builds with and without pymalloc are ABI compatible)
and so has been removed. (Contributed by Victor Stinner in bpo-36707@footnote{https://bugs.python.org/issue36707}.)

Example of changes:


@itemize *

@item 
Only @code{python3.8} program is installed, @code{python3.8m} program is gone.

@item 
Only @code{python3.8-config} script is installed, @code{python3.8m-config} script
is gone.

@item 
The @code{m} flag has been removed from the suffix of dynamic library
filenames: extension modules in the standard library as well as those
produced and installed by third-party packages, like those downloaded from
PyPI. On Linux, for example, the Python 3.7 suffix
@code{.cpython-37m-x86_64-linux-gnu.so} became
@code{.cpython-38-x86_64-linux-gnu.so} in Python 3.8.
@end itemize

@item 
The header files have been reorganized to better separate the different kinds
of APIs:


@itemize *

@item 
@code{Include/*.h} should be the portable public stable C API.

@item 
@code{Include/cpython/*.h} should be the unstable C API specific to CPython;
public API, with some private API prefixed by @code{_Py} or @code{_PY}.

@item 
@code{Include/internal/*.h} is the private internal C API very specific to
CPython. This API comes with no backward compatibility warranty and should
not be used outside CPython. It is only exposed for very specific needs
like debuggers and profiles which has to access to CPython internals
without calling functions. This API is now installed by @code{make install}.
@end itemize

(Contributed by Victor Stinner in bpo-35134@footnote{https://bugs.python.org/issue35134} and bpo-35081@footnote{https://bugs.python.org/issue35081},
work initiated by Eric Snow in Python 3.7.)

@item 
Some macros have been converted to static inline functions: parameter types
and return type are well defined, they don’t have issues specific to macros,
variables have a local scopes. Examples:


@itemize *

@item 
@ref{372,,Py_INCREF()}, @ref{373,,Py_DECREF()}

@item 
@ref{57b,,Py_XINCREF()}, @ref{1fd,,Py_XDECREF()}

@item 
@code{PyObject_INIT()}, @code{PyObject_INIT_VAR()}

@item 
Private functions: @code{_PyObject_GC_TRACK()},
@code{_PyObject_GC_UNTRACK()}, @code{_Py_Dealloc()}
@end itemize

(Contributed by Victor Stinner in bpo-35059@footnote{https://bugs.python.org/issue35059}.)

@item 
The @code{PyByteArray_Init()} and @code{PyByteArray_Fini()} functions have
been removed. They did nothing since Python 2.7.4 and Python 3.2.0, were
excluded from the limited API (stable ABI), and were not documented.
(Contributed by Victor Stinner in bpo-35713@footnote{https://bugs.python.org/issue35713}.)

@item 
The result of @code{PyExceptionClass_Name()} is now of type
@code{const char *} rather of @code{char *}.
(Contributed by Serhiy Storchaka in bpo-33818@footnote{https://bugs.python.org/issue33818}.)

@item 
The duality of @code{Modules/Setup.dist} and @code{Modules/Setup} has been
removed.  Previously, when updating the CPython source tree, one had
to manually copy @code{Modules/Setup.dist} (inside the source tree) to
@code{Modules/Setup} (inside the build tree) in order to reflect any changes
upstream.  This was of a small benefit to packagers at the expense of
a frequent annoyance to developers following CPython development, as
forgetting to copy the file could produce build failures.

Now the build system always reads from @code{Modules/Setup} inside the source
tree.  People who want to customize that file are encouraged to maintain
their changes in a git fork of CPython or as patch files, as they would do
for any other change to the source tree.

(Contributed by Antoine Pitrou in bpo-32430@footnote{https://bugs.python.org/issue32430}.)

@item 
Functions that convert Python number to C integer like
@ref{57c,,PyLong_AsLong()} and argument parsing functions like
@ref{37f,,PyArg_ParseTuple()} with integer converting format units like @code{'i'}
will now use the @ref{24a,,__index__()} special method instead of
@ref{249,,__int__()}, if available.  The deprecation warning will be
emitted for objects with the @code{__int__()} method but without the
@code{__index__()} method (like @ref{248,,Decimal} and
@ref{172,,Fraction}).  @ref{57d,,PyNumber_Check()} will now return
@code{1} for objects implementing @code{__index__()}.
@ref{57e,,PyNumber_Long()}, @ref{57f,,PyNumber_Float()} and
@ref{580,,PyFloat_AsDouble()} also now use the @code{__index__()} method if
available.
(Contributed by Serhiy Storchaka in bpo-36048@footnote{https://bugs.python.org/issue36048} and bpo-20092@footnote{https://bugs.python.org/issue20092}.)

@item 
Heap-allocated type objects will now increase their reference count
in @ref{581,,PyObject_Init()} (and its parallel macro @code{PyObject_INIT})
instead of in @ref{582,,PyType_GenericAlloc()}. Types that modify instance
allocation or deallocation may need to be adjusted.
(Contributed by Eddie Elizondo in bpo-35810@footnote{https://bugs.python.org/issue35810}.)

@item 
The new function @ref{583,,PyCode_NewWithPosOnlyArgs()} allows to create
code objects like @ref{584,,PyCode_New()}, but with an extra @emph{posonlyargcount}
parameter for indicating the number of positional-only arguments.
(Contributed by Pablo Galindo in bpo-37221@footnote{https://bugs.python.org/issue37221}.)

@item 
@ref{202,,Py_SetPath()} now sets @ref{585,,sys.executable} to the program full
path (@ref{38a,,Py_GetProgramFullPath()}) rather than to the program name
(@ref{38c,,Py_GetProgramName()}).
(Contributed by Victor Stinner in bpo-38234@footnote{https://bugs.python.org/issue38234}.)
@end itemize

@node Deprecated<6>,API and Feature Removals,Build and C API Changes,What’s New In Python 3 8
@anchor{whatsnew/3 8 deprecated}@anchor{586}
@subsection Deprecated



@itemize *

@item 
The distutils @code{bdist_wininst} command is now deprecated, use
@code{bdist_wheel} (wheel packages) instead.
(Contributed by Victor Stinner in bpo-37481@footnote{https://bugs.python.org/issue37481}.)

@item 
Deprecated methods @code{getchildren()} and @code{getiterator()} in
the @ref{13e,,ElementTree} module now emit a
@ref{1b2,,DeprecationWarning} instead of @ref{3a9,,PendingDeprecationWarning}.
They will be removed in Python 3.9.
(Contributed by Serhiy Storchaka in bpo-29209@footnote{https://bugs.python.org/issue29209}.)

@item 
Passing an object that is not an instance of
@ref{1c7,,concurrent.futures.ThreadPoolExecutor} to
@ref{587,,loop.set_default_executor()} is
deprecated and will be prohibited in Python 3.9.
(Contributed by Elvis Pranskevichus in bpo-34075@footnote{https://bugs.python.org/issue34075}.)

@item 
The @code{__getitem__()} methods of @ref{1ba,,xml.dom.pulldom.DOMEventStream},
@ref{1bb,,wsgiref.util.FileWrapper} and @ref{1bc,,fileinput.FileInput} have been
deprecated.

Implementations of these methods have been ignoring their @emph{index} parameter,
and returning the next item instead.
(Contributed by Berker Peksag in bpo-9372@footnote{https://bugs.python.org/issue9372}.)

@item 
The @ref{457,,typing.NamedTuple} class has deprecated the @code{_field_types}
attribute in favor of the @code{__annotations__} attribute which has the same
information. (Contributed by Raymond Hettinger in bpo-36320@footnote{https://bugs.python.org/issue36320}.)

@item 
@ref{7,,ast} classes @code{Num}, @code{Str}, @code{Bytes}, @code{NameConstant} and
@code{Ellipsis} are considered deprecated and will be removed in future Python
versions. @ref{588,,Constant} should be used instead.
(Contributed by Serhiy Storchaka in bpo-32892@footnote{https://bugs.python.org/issue32892}.)

@item 
@ref{589,,ast.NodeVisitor} methods @code{visit_Num()}, @code{visit_Str()},
@code{visit_Bytes()}, @code{visit_NameConstant()} and @code{visit_Ellipsis()} are
deprecated now and will not be called in future Python versions.
Add the @code{visit_Constant()} method to handle all
constant nodes.
(Contributed by Serhiy Storchaka in bpo-36917@footnote{https://bugs.python.org/issue36917}.)

@item 
The @code{asyncio.coroutine()} @ref{1bf,,decorator} is deprecated and will be
removed in version 3.10.  Instead of @code{@@asyncio.coroutine}, use
@ref{1c0,,async def} instead.
(Contributed by Andrew Svetlov in bpo-36921@footnote{https://bugs.python.org/issue36921}.)

@item 
In @ref{9,,asyncio}, the explicit passing of a @emph{loop} argument has been
deprecated and will be removed in version 3.10 for the following:
@ref{58a,,asyncio.sleep()}, @ref{31a,,asyncio.gather()}, @ref{31b,,asyncio.shield()},
@ref{3d5,,asyncio.wait_for()}, @ref{440,,asyncio.wait()}, @ref{31c,,asyncio.as_completed()},
@ref{317,,asyncio.Task}, @ref{58b,,asyncio.Lock}, @ref{58c,,asyncio.Event},
@ref{58d,,asyncio.Condition}, @ref{58e,,asyncio.Semaphore},
@ref{58f,,asyncio.BoundedSemaphore}, @ref{590,,asyncio.Queue},
@ref{591,,asyncio.create_subprocess_exec()}, and
@ref{592,,asyncio.create_subprocess_shell()}.

@item 
The explicit passing of coroutine objects to @ref{440,,asyncio.wait()} has been
deprecated and will be removed in version 3.11.
(Contributed by Yury Selivanov in bpo-34790@footnote{https://bugs.python.org/issue34790}.)

@item 
The following functions and methods are deprecated in the @ref{83,,gettext}
module: @code{lgettext()}, @code{ldgettext()},
@code{lngettext()} and @code{ldngettext()}.
They return encoded bytes, and it’s possible that you will get unexpected
Unicode-related exceptions if there are encoding problems with the
translated strings. It’s much better to use alternatives which return
Unicode strings in Python 3. These functions have been broken for a long time.

Function @code{bind_textdomain_codeset()}, methods
@code{output_charset()} and
@code{set_output_charset()}, and the @emph{codeset}
parameter of functions @ref{1bd,,translation()} and
@ref{1be,,install()} are also deprecated, since they are only used for
the @code{l*gettext()} functions.
(Contributed by Serhiy Storchaka in bpo-33710@footnote{https://bugs.python.org/issue33710}.)

@item 
The @code{isAlive()} method of @ref{44b,,threading.Thread}
has been deprecated.
(Contributed by Dong-hee Na in bpo-35283@footnote{https://bugs.python.org/issue35283}.)

@item 
Many builtin and extension functions that take integer arguments will
now emit a deprecation warning for @ref{248,,Decimal}s,
@ref{172,,Fraction}s and any other objects that can be converted
to integers only with a loss (e.g. that have the @ref{249,,__int__()}
method but do not have the @ref{24a,,__index__()} method).  In future
version they will be errors.
(Contributed by Serhiy Storchaka in bpo-36048@footnote{https://bugs.python.org/issue36048}.)

@item 
Deprecated passing the following arguments as keyword arguments:


@itemize -

@item 
@emph{func} in @ref{593,,functools.partialmethod()}, @ref{594,,weakref.finalize()},
@ref{595,,profile.Profile.runcall()}, @code{cProfile.Profile.runcall()},
@ref{596,,bdb.Bdb.runcall()}, @ref{597,,trace.Trace.runfunc()} and
@ref{598,,curses.wrapper()}.

@item 
@emph{function} in @ref{599,,unittest.TestCase.addCleanup()}.

@item 
@emph{fn} in the @ref{59a,,submit()} method of
@ref{1c7,,concurrent.futures.ThreadPoolExecutor} and
@ref{3da,,concurrent.futures.ProcessPoolExecutor}.

@item 
@emph{callback} in @ref{59b,,contextlib.ExitStack.callback()},
@code{contextlib.AsyncExitStack.callback()} and
@ref{59c,,contextlib.AsyncExitStack.push_async_callback()}.

@item 
@emph{c} and @emph{typeid} in the @code{create()}
method of @code{multiprocessing.managers.Server} and
@code{multiprocessing.managers.SharedMemoryServer}.

@item 
@emph{obj} in @ref{594,,weakref.finalize()}.
@end itemize

In future releases of Python, they will be @ref{59d,,positional-only}.
(Contributed by Serhiy Storchaka in bpo-36492@footnote{https://bugs.python.org/issue36492}.)
@end itemize

@node API and Feature Removals,Porting to Python 3 8,Deprecated<6>,What’s New In Python 3 8
@anchor{whatsnew/3 8 api-and-feature-removals}@anchor{59e}
@subsection API and Feature Removals


The following features and APIs have been removed from Python 3.8:


@itemize *

@item 
Starting with Python 3.3, importing ABCs from @ref{1c,,collections} was
deprecated, and importing should be done from @ref{1d,,collections.abc}. Being
able to import from collections was marked for removal in 3.8, but has been
delayed to 3.9. (See bpo-36952@footnote{https://bugs.python.org/issue36952}.)

@item 
The @code{macpath} module, deprecated in Python 3.7, has been removed.
(Contributed by Victor Stinner in bpo-35471@footnote{https://bugs.python.org/issue35471}.)

@item 
The function @code{platform.popen()} has been removed, after having been
deprecated since Python 3.3: use @ref{59f,,os.popen()} instead.
(Contributed by Victor Stinner in bpo-35345@footnote{https://bugs.python.org/issue35345}.)

@item 
The function @code{time.clock()} has been removed, after having been
deprecated since Python 3.3: use @ref{5a0,,time.perf_counter()} or
@ref{5a1,,time.process_time()} instead, depending
on your requirements, to have well-defined behavior.
(Contributed by Matthias Bussonnier in bpo-36895@footnote{https://bugs.python.org/issue36895}.)

@item 
The @code{pyvenv} script has been removed in favor of @code{python3.8 -m venv}
to help eliminate confusion as to what Python interpreter the @code{pyvenv}
script is tied to. (Contributed by Brett Cannon in bpo-25427@footnote{https://bugs.python.org/issue25427}.)

@item 
@code{parse_qs}, @code{parse_qsl}, and @code{escape} are removed from the @ref{14,,cgi}
module.  They are deprecated in Python 3.2 or older. They should be imported
from the @code{urllib.parse} and @code{html} modules instead.

@item 
@code{filemode} function is removed from the @ref{fb,,tarfile} module.
It is not documented and deprecated since Python 3.3.

@item 
The @ref{569,,XMLParser} constructor no longer accepts
the @emph{html} argument.  It never had an effect and was deprecated in Python 3.4.
All other parameters are now @ref{5a2,,keyword-only}.
(Contributed by Serhiy Storchaka in bpo-29209@footnote{https://bugs.python.org/issue29209}.)

@item 
Removed the @code{doctype()} method of @ref{569,,XMLParser}.
(Contributed by Serhiy Storchaka in bpo-29209@footnote{https://bugs.python.org/issue29209}.)

@item 
“unicode_internal” codec is removed.
(Contributed by Inada Naoki in bpo-36297@footnote{https://bugs.python.org/issue36297}.)

@item 
The @code{Cache} and @code{Statement} objects of the @ref{ed,,sqlite3} module are not
exposed to the user.
(Contributed by Aviv Palivoda in bpo-30262@footnote{https://bugs.python.org/issue30262}.)

@item 
The @code{bufsize} keyword argument of @ref{280,,fileinput.input()} and
@ref{1bc,,fileinput.FileInput()} which was ignored and deprecated since Python 3.6
has been removed. bpo-36952@footnote{https://bugs.python.org/issue36952} (Contributed by Matthias Bussonnier.)

@item 
The functions @code{sys.set_coroutine_wrapper()} and
@code{sys.get_coroutine_wrapper()} deprecated in Python 3.7 have been removed;
bpo-36933@footnote{https://bugs.python.org/issue36933} (Contributed by Matthias Bussonnier.)
@end itemize

@node Porting to Python 3 8,Notable changes in Python 3 8 1,API and Feature Removals,What’s New In Python 3 8
@anchor{whatsnew/3 8 porting-to-python-3-8}@anchor{5a3}
@subsection Porting to Python 3.8


This section lists previously described changes and other bugfixes
that may require changes to your code.

@menu
* Changes in Python behavior:: 
* Changes in the Python API: Changes in the Python API<4>. 
* Changes in the C API: Changes in the C API<3>. 
* CPython bytecode changes: CPython bytecode changes<4>. 
* Demos and Tools:: 

@end menu

@node Changes in Python behavior,Changes in the Python API<4>,,Porting to Python 3 8
@anchor{whatsnew/3 8 changes-in-python-behavior}@anchor{5a4}
@subsubsection Changes in Python behavior



@itemize *

@item 
Yield expressions (both @code{yield} and @code{yield from} clauses) are now disallowed
in comprehensions and generator expressions (aside from the iterable expression
in the leftmost @code{for} clause).
(Contributed by Serhiy Storchaka in bpo-10544@footnote{https://bugs.python.org/issue10544}.)

@item 
The compiler now produces a @ref{4c7,,SyntaxWarning} when identity checks
(@code{is} and @code{is not}) are used with certain types of literals
(e.g. strings, numbers).  These can often work by accident in CPython,
but are not guaranteed by the language spec.  The warning advises users
to use equality tests (@code{==} and @code{!=}) instead.
(Contributed by Serhiy Storchaka in bpo-34850@footnote{https://bugs.python.org/issue34850}.)

@item 
The CPython interpreter can swallow exceptions in some circumstances.
In Python 3.8 this happens in fewer cases.  In particular, exceptions
raised when getting the attribute from the type dictionary are no longer
ignored. (Contributed by Serhiy Storchaka in bpo-35459@footnote{https://bugs.python.org/issue35459}.)

@item 
Removed @code{__str__} implementations from builtin types @ref{2f7,,bool},
@ref{1b1,,int}, @ref{256,,float}, @ref{33b,,complex} and few classes from
the standard library.  They now inherit @code{__str__()} from @ref{5a5,,object}.
As result, defining the @code{__repr__()} method in the subclass of these
classes will affect their string representation.
(Contributed by Serhiy Storchaka in bpo-36793@footnote{https://bugs.python.org/issue36793}.)

@item 
On AIX, @ref{5a6,,sys.platform} doesn’t contain the major version anymore.
It is always @code{'aix'}, instead of @code{'aix3'} .. @code{'aix7'}.  Since
older Python versions include the version number, so it is recommended to
always use @code{sys.platform.startswith('aix')}.
(Contributed by M. Felt in bpo-36588@footnote{https://bugs.python.org/issue36588}.)

@item 
@ref{5a7,,PyEval_AcquireLock()} and @ref{5a8,,PyEval_AcquireThread()} now
terminate the current thread if called while the interpreter is
finalizing, making them consistent with @ref{5a9,,PyEval_RestoreThread()},
@ref{5aa,,Py_END_ALLOW_THREADS()}, and @ref{5ab,,PyGILState_Ensure()}. If this
behavior is not desired, guard the call by checking @code{_Py_IsFinalizing()}
or @code{sys.is_finalizing()}.
(Contributed by Joannah Nanjekye in bpo-36475@footnote{https://bugs.python.org/issue36475}.)
@end itemize

@node Changes in the Python API<4>,Changes in the C API<3>,Changes in Python behavior,Porting to Python 3 8
@anchor{whatsnew/3 8 changes-in-the-python-api}@anchor{5ac}
@subsubsection Changes in the Python API



@itemize *

@item 
The @ref{5ad,,os.getcwdb()} function now uses the UTF-8 encoding on Windows,
rather than the ANSI code page: see 
@geindex Python Enhancement Proposals; PEP 529
PEP 529@footnote{https://www.python.org/dev/peps/pep-0529} for the rationale. The
function is no longer deprecated on Windows.
(Contributed by Victor Stinner in bpo-37412@footnote{https://bugs.python.org/issue37412}.)

@item 
@ref{5ae,,subprocess.Popen} can now use @ref{56e,,os.posix_spawn()} in some cases
for better performance. On Windows Subsystem for Linux and QEMU User
Emulation, the @code{Popen} constructor using @ref{56e,,os.posix_spawn()} no longer raises an
exception on errors like “missing program”.  Instead the child process fails with a
non-zero @code{returncode}.
(Contributed by Joannah Nanjekye and Victor Stinner in bpo-35537@footnote{https://bugs.python.org/issue35537}.)

@item 
The @emph{preexec_fn} argument of * @ref{5ae,,subprocess.Popen} is no longer
compatible with subinterpreters. The use of the parameter in a
subinterpreter now raises @ref{5af,,RuntimeError}.
(Contributed by Eric Snow in bpo-34651@footnote{https://bugs.python.org/issue34651}, modified by Christian Heimes
in bpo-37951@footnote{https://bugs.python.org/issue37951}.)

@item 
The @code{imap.IMAP4.logout()} method no longer silently ignores arbitrary
exceptions.
(Contributed by Victor Stinner in bpo-36348@footnote{https://bugs.python.org/issue36348}.)

@item 
The function @code{platform.popen()} has been removed, after having been deprecated since
Python 3.3: use @ref{59f,,os.popen()} instead.
(Contributed by Victor Stinner in bpo-35345@footnote{https://bugs.python.org/issue35345}.)

@item 
The @ref{5b0,,statistics.mode()} function no longer raises an exception
when given multimodal data.  Instead, it returns the first mode
encountered in the input data.  (Contributed by Raymond Hettinger
in bpo-35892@footnote{https://bugs.python.org/issue35892}.)

@item 
The @ref{5b1,,selection()} method of the
@ref{5b2,,tkinter.ttk.Treeview} class no longer takes arguments.  Using it with
arguments for changing the selection was deprecated in Python 3.6.  Use
specialized methods like @ref{5b3,,selection_set()} for
changing the selection.  (Contributed by Serhiy Storchaka in bpo-31508@footnote{https://bugs.python.org/issue31508}.)

@item 
The @code{writexml()}, @code{toxml()} and @code{toprettyxml()} methods of
@ref{13c,,xml.dom.minidom}, and the @code{write()} method of @code{xml.etree},
now preserve the attribute order specified by the user.
(Contributed by Diego Rojas and Raymond Hettinger in bpo-34160@footnote{https://bugs.python.org/issue34160}.)

@item 
A @ref{31,,dbm.dumb} database opened with flags @code{'r'} is now read-only.
@ref{5b4,,dbm.dumb.open()} with flags @code{'r'} and @code{'w'} no longer creates
a database if it does not exist.
(Contributed by Serhiy Storchaka in bpo-32749@footnote{https://bugs.python.org/issue32749}.)

@item 
The @code{doctype()} method defined in a subclass of
@ref{569,,XMLParser} will no longer be called and will
emit a @ref{5b5,,RuntimeWarning} instead of a @ref{1b2,,DeprecationWarning}.
Define the @ref{5b6,,doctype()}
method on a target for handling an XML doctype declaration.
(Contributed by Serhiy Storchaka in bpo-29209@footnote{https://bugs.python.org/issue29209}.)

@item 
A @ref{5af,,RuntimeError} is now raised when the custom metaclass doesn’t
provide the @code{__classcell__} entry in the namespace passed to
@code{type.__new__}.  A @ref{1b2,,DeprecationWarning} was emitted in Python
3.6–3.7.  (Contributed by Serhiy Storchaka in bpo-23722@footnote{https://bugs.python.org/issue23722}.)

@item 
The @code{cProfile.Profile} class can now be used as a context
manager. (Contributed by Scott Sanderson in bpo-29235@footnote{https://bugs.python.org/issue29235}.)

@item 
@ref{56f,,shutil.copyfile()}, @ref{570,,shutil.copy()}, @ref{571,,shutil.copy2()},
@ref{538,,shutil.copytree()} and @ref{572,,shutil.move()} use platform-specific
“fast-copy” syscalls (see
@ref{575,,Platform-dependent efficient copy operations} section).

@item 
@ref{56f,,shutil.copyfile()} default buffer size on Windows was changed from
16 KiB to 1 MiB.

@item 
The @code{PyGC_Head} struct has changed completely.  All code that touched the
struct member should be rewritten.  (See bpo-33597@footnote{https://bugs.python.org/issue33597}.)

@item 
The @ref{3a0,,PyInterpreterState} struct has been moved into the “internal”
header files (specifically Include/internal/pycore_pystate.h).  An
opaque @code{PyInterpreterState} is still available as part of the public
API (and stable ABI).  The docs indicate that none of the struct’s
fields are public, so we hope no one has been using them.  However,
if you do rely on one or more of those private fields and have no
alternative then please open a BPO issue.  We’ll work on helping
you adjust (possibly including adding accessor functions to the
public API).  (See bpo-35886@footnote{https://bugs.python.org/issue35886}.)

@item 
The @ref{5b7,,mmap.flush()} method now returns @code{None} on
success and raises an exception on error under all platforms.  Previously,
its behavior was platform-dependent: a nonzero value was returned on success;
zero was returned on error under Windows.  A zero value was returned on
success; an exception was raised on error under Unix.
(Contributed by Berker Peksag in bpo-2122@footnote{https://bugs.python.org/issue2122}.)

@item 
@ref{13c,,xml.dom.minidom} and @ref{142,,xml.sax} modules no longer process
external entities by default.
(Contributed by Christian Heimes in bpo-17239@footnote{https://bugs.python.org/issue17239}.)

@item 
Deleting a key from a read-only @ref{30,,dbm} database (@ref{31,,dbm.dumb},
@ref{32,,dbm.gnu} or @ref{33,,dbm.ndbm}) raises @code{error} (@ref{5b8,,dbm.dumb.error},
@ref{5b9,,dbm.gnu.error} or @ref{5ba,,dbm.ndbm.error}) instead of @ref{5bb,,KeyError}.
(Contributed by Xiang Zhang in bpo-33106@footnote{https://bugs.python.org/issue33106}.)

@item 
Simplified AST for literals.  All constants will be represented as
@ref{588,,ast.Constant} instances.  Instantiating old classes @code{Num},
@code{Str}, @code{Bytes}, @code{NameConstant} and @code{Ellipsis} will return
an instance of @code{Constant}.
(Contributed by Serhiy Storchaka in bpo-32892@footnote{https://bugs.python.org/issue32892}.)

@item 
@ref{520,,expanduser()} on Windows now prefers the 
@geindex USERPROFILE
@geindex environment variable; USERPROFILE
@code{USERPROFILE}
environment variable and does not use 
@geindex HOME
@geindex environment variable; HOME
@code{HOME}, which is not normally
set for regular user accounts.
(Contributed by Anthony Sottile in bpo-36264@footnote{https://bugs.python.org/issue36264}.)

@item 
The exception @ref{4d7,,asyncio.CancelledError} now inherits from
@ref{15f,,BaseException} rather than @ref{4d8,,Exception} and no longer inherits
from @ref{4d9,,concurrent.futures.CancelledError}.
(Contributed by Yury Selivanov in bpo-32528@footnote{https://bugs.python.org/issue32528}.)

@item 
The function @ref{3d5,,asyncio.wait_for()} now correctly waits for cancellation
when using an instance of @ref{317,,asyncio.Task}. Previously, upon reaching
@emph{timeout}, it was cancelled and immediately returned.
(Contributed by Elvis Pranskevichus in bpo-32751@footnote{https://bugs.python.org/issue32751}.)

@item 
The function @ref{5bc,,asyncio.BaseTransport.get_extra_info()} now returns a safe
to use socket object when ‘socket’ is passed to the @emph{name} parameter.
(Contributed by Yury Selivanov in bpo-37027@footnote{https://bugs.python.org/issue37027}.)

@item 
@ref{5bd,,asyncio.BufferedProtocol} has graduated to the stable API.
@end itemize
@anchor{whatsnew/3 8 bpo-36085-whatsnew}@anchor{5be}

@itemize *

@item 
DLL dependencies for extension modules and DLLs loaded with @ref{29,,ctypes} on
Windows are now resolved more securely. Only the system paths, the directory
containing the DLL or PYD file, and directories added with
@ref{4ee,,add_dll_directory()} are searched for load-time dependencies.
Specifically, 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} and the current working directory are no longer
used, and modifications to these will no longer have any effect on normal DLL
resolution. If your application relies on these mechanisms, you should check
for @ref{4ee,,add_dll_directory()} and if it exists, use it to add your DLLs
directory while loading your library. Note that Windows 7 users will need to
ensure that Windows Update KB2533623 has been installed (this is also verified
by the installer).
(Contributed by Steve Dower in bpo-36085@footnote{https://bugs.python.org/issue36085}.)

@item 
The header files and functions related to pgen have been removed after its
replacement by a pure Python implementation. (Contributed by Pablo Galindo
in bpo-36623@footnote{https://bugs.python.org/issue36623}.)

@item 
@ref{4cb,,types.CodeType} has a new parameter in the second position of the
constructor (@emph{posonlyargcount}) to support positional-only arguments defined
in 
@geindex Python Enhancement Proposals; PEP 570
PEP 570@footnote{https://www.python.org/dev/peps/pep-0570}. The first argument (@emph{argcount}) now represents the total
number of positional arguments (including positional-only arguments). The new
@code{replace()} method of @ref{4cb,,types.CodeType} can be used to make the code
future-proof.
@end itemize

@node Changes in the C API<3>,CPython bytecode changes<4>,Changes in the Python API<4>,Porting to Python 3 8
@anchor{whatsnew/3 8 changes-in-the-c-api}@anchor{5bf}
@subsubsection Changes in the C API



@itemize *

@item 
The @ref{5c0,,PyCompilerFlags} structure got a new @emph{cf_feature_version}
field. It should be initialized to @code{PY_MINOR_VERSION}. The field is ignored
by default, and is used if and only if @code{PyCF_ONLY_AST} flag is set in
@emph{cf_flags}.
(Contributed by Guido van Rossum in bpo-35766@footnote{https://bugs.python.org/issue35766}.)

@item 
The @code{PyEval_ReInitThreads()} function has been removed from the C API.
It should not be called explicitly: use @ref{5c1,,PyOS_AfterFork_Child()}
instead.
(Contributed by Victor Stinner in bpo-36728@footnote{https://bugs.python.org/issue36728}.)

@item 
On Unix, C extensions are no longer linked to libpython except on Android
and Cygwin. When Python is embedded, @code{libpython} must not be loaded with
@code{RTLD_LOCAL}, but @code{RTLD_GLOBAL} instead. Previously, using
@code{RTLD_LOCAL}, it was already not possible to load C extensions which
were not linked to @code{libpython}, like C extensions of the standard
library built by the @code{*shared*} section of @code{Modules/Setup}.
(Contributed by Victor Stinner in bpo-21536@footnote{https://bugs.python.org/issue21536}.)

@item 
Use of @code{#} variants of formats in parsing or building value (e.g.
@ref{37f,,PyArg_ParseTuple()}, @ref{380,,Py_BuildValue()}, @ref{5c2,,PyObject_CallFunction()},
etc.) without @code{PY_SSIZE_T_CLEAN} defined raises @code{DeprecationWarning} now.
It will be removed in 3.10 or 4.0.  Read @ref{381,,Parsing arguments and building values} for detail.
(Contributed by Inada Naoki in bpo-36381@footnote{https://bugs.python.org/issue36381}.)

@item 
Instances of heap-allocated types (such as those created with
@ref{469,,PyType_FromSpec()}) hold a reference to their type object.
Increasing the reference count of these type objects has been moved from
@ref{582,,PyType_GenericAlloc()} to the more low-level functions,
@ref{581,,PyObject_Init()} and @code{PyObject_INIT()}.
This makes types created through @ref{469,,PyType_FromSpec()} behave like
other classes in managed code.

@ref{1f0,,Statically allocated types} are not affected.

For the vast majority of cases, there should be no side effect.
However, types that manually increase the reference count after allocating
an instance (perhaps to work around the bug) may now become immortal.
To avoid this, these classes need to call Py_DECREF on the type object
during instance deallocation.

To correctly port these types into 3.8, please apply the following
changes:


@itemize *

@item 
Remove @ref{372,,Py_INCREF} on the type object after allocating an
instance - if any.
This may happen after calling @ref{491,,PyObject_New()},
@ref{492,,PyObject_NewVar()}, @ref{5c3,,PyObject_GC_New()},
@ref{5c4,,PyObject_GC_NewVar()}, or any other custom allocator that uses
@ref{581,,PyObject_Init()} or @code{PyObject_INIT()}.

Example:

@example
static foo_struct *
foo_new(PyObject *type) @{
    foo_struct *foo = PyObject_GC_New(foo_struct, (PyTypeObject *) type);
    if (foo == NULL)
        return NULL;
#if PY_VERSION_HEX < 0x03080000
    // Workaround for Python issue 35810; no longer necessary in Python 3.8
    PY_INCREF(type)
#endif
    return foo;
@}
@end example

@item 
Ensure that all custom @code{tp_dealloc} functions of heap-allocated types
decrease the type’s reference count.

Example:

@example
static void
foo_dealloc(foo_struct *instance) @{
    PyObject *type = Py_TYPE(instance);
    PyObject_GC_Del(instance);
#if PY_VERSION_HEX >= 0x03080000
    // This was not needed before Python 3.8 (Python issue 35810)
    Py_DECREF(type);
#endif
@}
@end example
@end itemize

(Contributed by Eddie Elizondo in bpo-35810@footnote{https://bugs.python.org/issue35810}.)

@item 
The @ref{5c5,,Py_DEPRECATED()} macro has been implemented for MSVC.
The macro now must be placed before the symbol name.

Example:

@example
Py_DEPRECATED(3.8) PyAPI_FUNC(int) Py_OldFunction(void);
@end example

(Contributed by Zackery Spytz in bpo-33407@footnote{https://bugs.python.org/issue33407}.)

@item 
The interpreter does not pretend to support binary compatibility of
extension types across feature releases, anymore.  A @ref{5c6,,PyTypeObject}
exported by a third-party extension module is supposed to have all the
slots expected in the current Python version, including
@ref{5c7,,tp_finalize} (@ref{5c8,,Py_TPFLAGS_HAVE_FINALIZE}
is not checked anymore before reading @ref{5c7,,tp_finalize}).

(Contributed by Antoine Pitrou in bpo-32388@footnote{https://bugs.python.org/issue32388}.)

@item 
The functions @code{PyNode_AddChild()} and @code{PyParser_AddToken()} now accept
two additional @code{int} arguments @emph{end_lineno} and @emph{end_col_offset}.

@item 
The @code{libpython38.a} file to allow MinGW tools to link directly against
@code{python38.dll} is no longer included in the regular Windows distribution.
If you require this file, it may be generated with the @code{gendef} and
@code{dlltool} tools, which are part of the MinGW binutils package:

@example
gendef - python38.dll > tmp.def
dlltool --dllname python38.dll --def tmp.def --output-lib libpython38.a
@end example

The location of an installed @code{pythonXY.dll} will depend on the
installation options and the version and language of Windows. See
@ref{5c9,,Using Python on Windows} for more information. The resulting library should be
placed in the same directory as @code{pythonXY.lib}, which is generally the
@code{libs} directory under your Python installation.

(Contributed by Steve Dower in bpo-37351@footnote{https://bugs.python.org/issue37351}.)
@end itemize

@node CPython bytecode changes<4>,Demos and Tools,Changes in the C API<3>,Porting to Python 3 8
@anchor{whatsnew/3 8 cpython-bytecode-changes}@anchor{5ca}
@subsubsection CPython bytecode changes



@itemize *

@item 
The interpreter loop  has been simplified by moving the logic of unrolling
the stack of blocks into the compiler.  The compiler emits now explicit
instructions for adjusting the stack of values and calling the
cleaning-up code for @ref{5cb,,break}, @ref{4c0,,continue} and
@ref{4c6,,return}.

Removed opcodes @code{BREAK_LOOP}, @code{CONTINUE_LOOP},
@code{SETUP_LOOP} and @code{SETUP_EXCEPT}.  Added new opcodes
@code{ROT_FOUR}, @code{BEGIN_FINALLY}, @code{CALL_FINALLY} and
@code{POP_FINALLY}.  Changed the behavior of @code{END_FINALLY}
and @code{WITH_CLEANUP_START}.

(Contributed by Mark Shannon, Antoine Pitrou and Serhiy Storchaka in
bpo-17611@footnote{https://bugs.python.org/issue17611}.)

@item 
Added new opcode @ref{5cc,,END_ASYNC_FOR} for handling exceptions raised
when awaiting a next item in an @ref{5cd,,async for} loop.
(Contributed by Serhiy Storchaka in bpo-33041@footnote{https://bugs.python.org/issue33041}.)

@item 
The @ref{5ce,,MAP_ADD} now expects the value as the first element in the
stack and the key as the second element. This change was made so the key
is always evaluated before the value in dictionary comprehensions, as
proposed by 
@geindex Python Enhancement Proposals; PEP 572
PEP 572@footnote{https://www.python.org/dev/peps/pep-0572}. (Contributed by Jörn Heissler in bpo-35224@footnote{https://bugs.python.org/issue35224}.)
@end itemize

@node Demos and Tools,,CPython bytecode changes<4>,Porting to Python 3 8
@anchor{whatsnew/3 8 demos-and-tools}@anchor{5cf}
@subsubsection Demos and Tools


Added a benchmark script for timing various ways to access variables:
@code{Tools/scripts/var_access_benchmark.py}.
(Contributed by Raymond Hettinger in bpo-35884@footnote{https://bugs.python.org/issue35884}.)

Here’s a summary of performance improvements since Python 3.3:

@example
Python version                       3.3     3.4     3.5     3.6     3.7     3.8
--------------                       ---     ---     ---     ---     ---     ---

Variable and attribute read access:
    read_local                       4.0     7.1     7.1     5.4     5.1     3.9
    read_nonlocal                    5.3     7.1     8.1     5.8     5.4     4.4
    read_global                     13.3    15.5    19.0    14.3    13.6     7.6
    read_builtin                    20.0    21.1    21.6    18.5    19.0     7.5
    read_classvar_from_class        20.5    25.6    26.5    20.7    19.5    18.4
    read_classvar_from_instance     18.5    22.8    23.5    18.8    17.1    16.4
    read_instancevar                26.8    32.4    33.1    28.0    26.3    25.4
    read_instancevar_slots          23.7    27.8    31.3    20.8    20.8    20.2
    read_namedtuple                 68.5    73.8    57.5    45.0    46.8    18.4
    read_boundmethod                29.8    37.6    37.9    29.6    26.9    27.7

Variable and attribute write access:
    write_local                      4.6     8.7     9.3     5.5     5.3     4.3
    write_nonlocal                   7.3    10.5    11.1     5.6     5.5     4.7
    write_global                    15.9    19.7    21.2    18.0    18.0    15.8
    write_classvar                  81.9    92.9    96.0   104.6   102.1    39.2
    write_instancevar               36.4    44.6    45.8    40.0    38.9    35.5
    write_instancevar_slots         28.7    35.6    36.1    27.3    26.6    25.7

Data structure read access:
    read_list                       19.2    24.2    24.5    20.8    20.8    19.0
    read_deque                      19.9    24.7    25.5    20.2    20.6    19.8
    read_dict                       19.7    24.3    25.7    22.3    23.0    21.0
    read_strdict                    17.9    22.6    24.3    19.5    21.2    18.9

Data structure write access:
    write_list                      21.2    27.1    28.5    22.5    21.6    20.0
    write_deque                     23.8    28.7    30.1    22.7    21.8    23.5
    write_dict                      25.9    31.4    33.3    29.3    29.2    24.7
    write_strdict                   22.9    28.4    29.9    27.5    25.2    23.1

Stack (or queue) operations:
    list_append_pop                144.2    93.4   112.7    75.4    74.2    50.8
    deque_append_pop                30.4    43.5    57.0    49.4    49.2    42.5
    deque_append_popleft            30.8    43.7    57.3    49.7    49.7    42.8

Timing loop:
    loop_overhead                    0.3     0.5     0.6     0.4     0.3     0.3
@end example

The benchmarks were measured on an
Intel® Core™ i7-4960HQ processor@footnote{https://ark.intel.com/content/www/us/en/ark/products/76088/intel-core-i7-4960hq-processor-6m-cache-up-to-3-80-ghz.html}
running the macOS 64-bit builds found at
python.org@footnote{https://www.python.org/downloads/mac-osx/}.
The benchmark script displays timings in nanoseconds.

@node Notable changes in Python 3 8 1,Notable changes in Python 3 8 8,Porting to Python 3 8,What’s New In Python 3 8
@anchor{whatsnew/3 8 notable-changes-in-python-3-8-1}@anchor{5d0}
@subsection Notable changes in Python 3.8.1


Due to significant security concerns, the @emph{reuse_address} parameter of
@ref{1b9,,asyncio.loop.create_datagram_endpoint()} is no longer supported. This is
because of the behavior of the socket option @code{SO_REUSEADDR} in UDP. For more
details, see the documentation for @code{loop.create_datagram_endpoint()}.
(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in
bpo-37228@footnote{https://bugs.python.org/issue37228}.)

@node Notable changes in Python 3 8 8,Notable changes in Python 3 8 12,Notable changes in Python 3 8 1,What’s New In Python 3 8
@anchor{whatsnew/3 8 notable-changes-in-python-3-8-8}@anchor{5d1}
@subsection Notable changes in Python 3.8.8


Earlier Python versions allowed using both @code{;} and @code{&} as
query parameter separators in @ref{2e3,,urllib.parse.parse_qs()} and
@ref{2e4,,urllib.parse.parse_qsl()}.  Due to security concerns, and to conform with
newer W3C recommendations, this has been changed to allow only a single
separator key, with @code{&} as the default.  This change also affects
@ref{2e5,,cgi.parse()} and @ref{2e6,,cgi.parse_multipart()} as they use the affected
functions internally. For more details, please see their respective
documentation.
(Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in bpo-42967@footnote{https://bugs.python.org/issue42967}.)

@node Notable changes in Python 3 8 12,,Notable changes in Python 3 8 8,What’s New In Python 3 8
@anchor{whatsnew/3 8 notable-changes-in-python-3-8-12}@anchor{5d2}
@subsection Notable changes in Python 3.8.12


Starting with Python 3.8.12 the @ref{9e,,ipaddress} module no longer accepts
any leading zeros in IPv4 address strings. Leading zeros are ambiguous and
interpreted as octal notation by some libraries. For example the legacy
function @ref{464,,socket.inet_aton()} treats leading zeros as octal notation.
glibc implementation of modern @ref{465,,inet_pton()} does not accept
any leading zeros.

(Originally contributed by Christian Heimes in bpo-36384@footnote{https://bugs.python.org/issue36384}, and backported
to 3.8 by Achraf Merzouki.)

@node What’s New In Python 3 7,What’s New In Python 3 6,What’s New In Python 3 8,What’s New in Python
@anchor{whatsnew/3 7 doc}@anchor{5d3}@anchor{whatsnew/3 7 what-s-new-in-python-3-7}@anchor{5d4}
@section What’s New In Python 3.7


@*Editor: 
Elvis Pranskevichus <@email{elvis@@magic.io}>


@c Rules for maintenance:
@c 
@c * Anyone can add text to this document.  Do not spend very much time
@c on the wording of your changes, because your text will probably
@c get rewritten to some degree.
@c 
@c * The maintainer will go through Misc/NEWS periodically and add
@c changes; it's therefore more important to add your changes to
@c Misc/NEWS than to this file.
@c 
@c * This is not a complete list of every single change; completeness
@c is the purpose of Misc/NEWS.  Some changes I consider too small
@c or esoteric to include.  If such a change is added to the text,
@c I'll just remove it.  (This is another reason you shouldn't spend
@c too much time on writing your addition.)
@c 
@c * If you want to draw your new text to the attention of the
@c maintainer, add 'XXX' to the beginning of the paragraph or
@c section.
@c 
@c * It's OK to just add a fragmentary note about a change.  For
@c example: "XXX Describe the transmogrify() function added to the
@c socket module."  The maintainer will research the change and
@c write the necessary text.
@c 
@c * You can comment out your additions if you like, but it's not
@c necessary (especially when a final release is some months away).
@c 
@c * Credit the author of a patch or bugfix.   Just the name is
@c sufficient; the e-mail address isn't necessary.
@c 
@c * It's helpful to add the bug/patch number as a comment:
@c 
@c XXX Describe the transmogrify() function added to the socket
@c module.
@c (Contributed by P.Y. Developer in :issue:`12345`.)
@c 
@c This saves the maintainer the effort of going through the Git log
@c when researching a change.

This article explains the new features in Python 3.7, compared to 3.6.
Python 3.7 was released on June 27, 2018.
For full details, see the @ref{154,,changelog}.

@menu
* Summary – Release Highlights:: 
* New Features: New Features<8>. 
* Other Language Changes: Other Language Changes<5>. 
* New Modules: New Modules<5>. 
* Improved Modules: Improved Modules<5>. 
* C API Changes: C API Changes<4>. 
* Build Changes: Build Changes<4>. 
* Optimizations: Optimizations<5>. 
* Other CPython Implementation Changes: Other CPython Implementation Changes<2>. 
* Deprecated Python Behavior:: 
* Deprecated Python modules@comma{} functions and methods: Deprecated Python modules functions and methods. 
* Deprecated functions and types of the C API:: 
* Platform Support Removals:: 
* API and Feature Removals: API and Feature Removals<2>. 
* Module Removals:: 
* Windows-only Changes:: 
* Porting to Python 3.7: Porting to Python 3 7. 
* Notable changes in Python 3.7.1: Notable changes in Python 3 7 1. 
* Notable changes in Python 3.7.2: Notable changes in Python 3 7 2. 
* Notable changes in Python 3.7.6: Notable changes in Python 3 7 6. 
* Notable changes in Python 3.7.10: Notable changes in Python 3 7 10. 

@end menu

@node Summary – Release Highlights,New Features<8>,,What’s New In Python 3 7
@anchor{whatsnew/3 7 summary-release-highlights}@anchor{5d5}
@subsection Summary – Release Highlights


@c This section singles out the most important changes in Python 3.7.
@c Brevity is key.

New syntax features:


@itemize *

@item 
@ref{5d6,,PEP 563}, postponed evaluation of type annotations.
@end itemize

Backwards incompatible syntax changes:


@itemize *

@item 
@ref{4d4,,async} and @ref{4d5,,await} are now reserved keywords.
@end itemize

New library modules:


@itemize *

@item 
@ref{23,,contextvars}: @ref{5d7,,PEP 567 – Context Variables}

@item 
@ref{2e,,dataclasses}: @ref{5d8,,PEP 557 – Data Classes}

@item 
@ref{5d9,,importlib.resources}
@end itemize

New built-in features:


@itemize *

@item 
@ref{5da,,PEP 553}, the new @ref{5db,,breakpoint()} function.
@end itemize

Python data model improvements:


@itemize *

@item 
@ref{5dc,,PEP 562}, customization of access to
module attributes.

@item 
@ref{5dd,,PEP 560}, core support for typing module and
generic types.

@item 
the insertion-order preservation nature of @ref{5de,,dict}
objects has been declared@footnote{https://mail.python.org/pipermail/python-dev/2017-December/151283.html} to be an official
part of the Python language spec.
@end itemize

Significant improvements in the standard library:


@itemize *

@item 
The @ref{9,,asyncio} module has received new features, significant
@ref{5df,,usability and performance improvements}.

@item 
The @ref{10a,,time} module gained support for
@ref{5e0,,functions with nanosecond resolution}.
@end itemize

CPython implementation improvements:


@itemize *

@item 
Avoiding the use of ASCII as a default text encoding:


@itemize *

@item 
@ref{5e1,,PEP 538}, legacy C locale coercion

@item 
@ref{5e2,,PEP 540}, forced UTF-8 runtime mode
@end itemize

@item 
@ref{5e3,,PEP 552}, deterministic .pycs

@item 
@ref{5e4,,New Python Development Mode}

@item 
@ref{5e5,,PEP 565}, improved @ref{1b2,,DeprecationWarning}
handling
@end itemize

C API improvements:


@itemize *

@item 
@ref{5e6,,PEP 539}, new C API for thread-local storage
@end itemize

Documentation improvements:


@itemize *

@item 
@ref{5e7,,PEP 545}, Python documentation translations

@item 
New documentation translations: Japanese@footnote{https://docs.python.org/ja/},
French@footnote{https://docs.python.org/fr/}, and
Korean@footnote{https://docs.python.org/ko/}.
@end itemize

This release features notable performance improvements in many areas.
The @ref{5e8,,Optimizations} section lists them in detail.

For a list of changes that may affect compatibility with previous Python
releases please refer to the @ref{5e9,,Porting to Python 3.7} section.

@node New Features<8>,Other Language Changes<5>,Summary – Release Highlights,What’s New In Python 3 7
@anchor{whatsnew/3 7 new-features}@anchor{5ea}
@subsection New Features


@menu
* PEP 563; Postponed Evaluation of Annotations: PEP 563 Postponed Evaluation of Annotations. 
* PEP 538; Legacy C Locale Coercion: PEP 538 Legacy C Locale Coercion. 
* PEP 540; Forced UTF-8 Runtime Mode: PEP 540 Forced UTF-8 Runtime Mode. 
* PEP 553; Built-in breakpoint(): PEP 553 Built-in breakpoint. 
* PEP 539; New C API for Thread-Local Storage: PEP 539 New C API for Thread-Local Storage. 
* PEP 562; Customization of Access to Module Attributes: PEP 562 Customization of Access to Module Attributes. 
* PEP 564; New Time Functions With Nanosecond Resolution: PEP 564 New Time Functions With Nanosecond Resolution. 
* PEP 565; Show DeprecationWarning in __main__: PEP 565 Show DeprecationWarning in __main__. 
* PEP 560; Core Support for typing module and Generic Types: PEP 560 Core Support for typing module and Generic Types. 
* PEP 552; Hash-based .pyc Files: PEP 552 Hash-based pyc Files. 
* PEP 545; Python Documentation Translations: PEP 545 Python Documentation Translations. 
* Python Development Mode (-X dev): Python Development Mode -X dev. 

@end menu

@node PEP 563 Postponed Evaluation of Annotations,PEP 538 Legacy C Locale Coercion,,New Features<8>
@anchor{whatsnew/3 7 pep-563-postponed-evaluation-of-annotations}@anchor{5eb}@anchor{whatsnew/3 7 whatsnew37-pep563}@anchor{5d6}
@subsubsection PEP 563: Postponed Evaluation of Annotations


The advent of type hints in Python uncovered two glaring usability issues
with the functionality of annotations added in 
@geindex Python Enhancement Proposals; PEP 3107
PEP 3107@footnote{https://www.python.org/dev/peps/pep-3107} and refined
further in 
@geindex Python Enhancement Proposals; PEP 526
PEP 526@footnote{https://www.python.org/dev/peps/pep-0526}:


@itemize *

@item 
annotations could only use names which were already available in the
current scope, in other words they didn’t support forward references
of any kind; and

@item 
annotating source code had adverse effects on startup time of Python
programs.
@end itemize

Both of these issues are fixed by postponing the evaluation of
annotations.  Instead of compiling code which executes expressions in
annotations at their definition time, the compiler stores the annotation
in a string form equivalent to the AST of the expression in question.
If needed, annotations can be resolved at runtime using
@ref{432,,typing.get_type_hints()}.  In the common case where this is not
required, the annotations are cheaper to store (since short strings
are interned by the interpreter) and make startup time faster.

Usability-wise, annotations now support forward references, making the
following syntax valid:

@example
class C:
    @@classmethod
    def from_string(cls, source: str) -> C:
        ...

    def validate_b(self, obj: B) -> bool:
        ...

class B:
    ...
@end example

Since this change breaks compatibility, the new behavior needs to be enabled
on a per-module basis in Python 3.7 using a @ref{0,,__future__} import:

@example
from __future__ import annotations
@end example

It will become the default in Python 3.10.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 563

@item PEP 563@footnote{https://www.python.org/dev/peps/pep-0563} – Postponed evaluation of annotations

PEP written and implemented by Łukasz Langa.
@end table


@node PEP 538 Legacy C Locale Coercion,PEP 540 Forced UTF-8 Runtime Mode,PEP 563 Postponed Evaluation of Annotations,New Features<8>
@anchor{whatsnew/3 7 pep-538-legacy-c-locale-coercion}@anchor{5ec}@anchor{whatsnew/3 7 whatsnew37-pep538}@anchor{5e1}
@subsubsection PEP 538: Legacy C Locale Coercion


An ongoing challenge within the Python 3 series has been determining a sensible
default strategy for handling the “7-bit ASCII” text encoding assumption
currently implied by the use of the default C or POSIX locale on non-Windows
platforms.

@geindex Python Enhancement Proposals; PEP 538
PEP 538@footnote{https://www.python.org/dev/peps/pep-0538} updates the default interpreter command line interface to
automatically coerce that locale to an available UTF-8 based locale as
described in the documentation of the new 
@geindex PYTHONCOERCECLOCALE
@geindex environment variable; PYTHONCOERCECLOCALE
@ref{5ed,,PYTHONCOERCECLOCALE}
environment variable. Automatically setting @code{LC_CTYPE} this way means that
both the core interpreter and locale-aware C extensions (such as
@ref{d9,,readline}) will assume the use of UTF-8 as the default text encoding,
rather than ASCII.

The platform support definition in 
@geindex Python Enhancement Proposals; PEP 11
PEP 11@footnote{https://www.python.org/dev/peps/pep-0011} has also been updated to limit
full text handling support to suitably configured non-ASCII based locales.

As part of this change, the default error handler for @ref{5ee,,stdin} and
@ref{5ef,,stdout} is now @code{surrogateescape} (rather than @code{strict}) when
using any of the defined coercion target locales (currently @code{C.UTF-8},
@code{C.utf8}, and @code{UTF-8}).  The default error handler for @ref{42d,,stderr}
continues to be @code{backslashreplace}, regardless of locale.

Locale coercion is silent by default, but to assist in debugging potentially
locale related integration problems, explicit warnings (emitted directly on
@ref{42d,,stderr}) can be requested by setting @code{PYTHONCOERCECLOCALE=warn}.
This setting will also cause the Python runtime to emit a warning if the
legacy C locale remains active when the core interpreter is initialized.

While 
@geindex Python Enhancement Proposals; PEP 538
PEP 538@footnote{https://www.python.org/dev/peps/pep-0538}’s locale coercion has the benefit of also affecting extension
modules (such as GNU @code{readline}), as well as child processes (including those
running non-Python applications and older versions of Python), it has the
downside of requiring that a suitable target locale be present on the running
system. To better handle the case where no suitable target locale is available
(as occurs on RHEL/CentOS 7, for example), Python 3.7 also implements
@ref{5e2,,PEP 540; Forced UTF-8 Runtime Mode}.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 538

@item PEP 538@footnote{https://www.python.org/dev/peps/pep-0538} – Coercing the legacy C locale to a UTF-8 based locale

PEP written and implemented by Nick Coghlan.
@end table


@node PEP 540 Forced UTF-8 Runtime Mode,PEP 553 Built-in breakpoint,PEP 538 Legacy C Locale Coercion,New Features<8>
@anchor{whatsnew/3 7 pep-540-forced-utf-8-runtime-mode}@anchor{5f0}@anchor{whatsnew/3 7 whatsnew37-pep540}@anchor{5e2}
@subsubsection PEP 540: Forced UTF-8 Runtime Mode


The new @ref{159,,-X} @code{utf8} command line option and 
@geindex PYTHONUTF8
@geindex environment variable; PYTHONUTF8
@ref{5f1,,PYTHONUTF8}
environment variable can be used to enable the @ref{5f2,,Python UTF-8 Mode}.

When in UTF-8 mode, CPython ignores the locale settings, and uses the
UTF-8 encoding by default.  The error handlers for @ref{5ee,,sys.stdin} and
@ref{5ef,,sys.stdout} streams are set to @code{surrogateescape}.

The forced UTF-8 mode can be used to change the text handling behavior in
an embedded Python interpreter without changing the locale settings of
an embedding application.

While 
@geindex Python Enhancement Proposals; PEP 540
PEP 540@footnote{https://www.python.org/dev/peps/pep-0540}’s UTF-8 mode has the benefit of working regardless of which
locales are available on the running system, it has the downside of having no
effect on extension modules (such as GNU @code{readline}), child processes running
non-Python applications, and child processes running older versions of Python.
To reduce the risk of corrupting text data when communicating with such
components, Python 3.7 also implements @ref{5e2,,PEP 540; Forced UTF-8 Runtime Mode}).

The UTF-8 mode is enabled by default when the locale is @code{C} or @code{POSIX}, and
the 
@geindex Python Enhancement Proposals; PEP 538
PEP 538@footnote{https://www.python.org/dev/peps/pep-0538} locale coercion feature fails to change it to a UTF-8 based
alternative (whether that failure is due to @code{PYTHONCOERCECLOCALE=0} being set,
@code{LC_ALL} being set, or the lack of a suitable target locale).


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 540

@item PEP 540@footnote{https://www.python.org/dev/peps/pep-0540} – Add a new UTF-8 mode

PEP written and implemented by Victor Stinner
@end table


@node PEP 553 Built-in breakpoint,PEP 539 New C API for Thread-Local Storage,PEP 540 Forced UTF-8 Runtime Mode,New Features<8>
@anchor{whatsnew/3 7 pep-553-built-in-breakpoint}@anchor{5f3}@anchor{whatsnew/3 7 whatsnew37-pep553}@anchor{5da}
@subsubsection PEP 553: Built-in @code{breakpoint()}


Python 3.7 includes the new built-in @ref{5db,,breakpoint()} function as
an easy and consistent way to enter the Python debugger.

Built-in @code{breakpoint()} calls @ref{5f4,,sys.breakpointhook()}.  By default, the
latter imports @ref{c4,,pdb} and then calls @code{pdb.set_trace()}, but by binding
@code{sys.breakpointhook()} to the function of your choosing, @code{breakpoint()} can
enter any debugger. Additionally, the environment variable
@geindex PYTHONBREAKPOINT
@geindex environment variable; PYTHONBREAKPOINT
@ref{5f5,,PYTHONBREAKPOINT} can be set to the callable of your debugger of
choice.  Set @code{PYTHONBREAKPOINT=0} to completely disable built-in
@code{breakpoint()}.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 553

@item PEP 553@footnote{https://www.python.org/dev/peps/pep-0553} – Built-in breakpoint()

PEP written and implemented by Barry Warsaw
@end table


@node PEP 539 New C API for Thread-Local Storage,PEP 562 Customization of Access to Module Attributes,PEP 553 Built-in breakpoint,New Features<8>
@anchor{whatsnew/3 7 pep-539-new-c-api-for-thread-local-storage}@anchor{5f6}@anchor{whatsnew/3 7 whatsnew37-pep539}@anchor{5e6}
@subsubsection PEP 539: New C API for Thread-Local Storage


While Python provides a C API for thread-local storage support; the existing
@ref{5f7,,Thread Local Storage (TLS) API} has used
int to represent TLS keys across all platforms.  This has not
generally been a problem for officially-support platforms, but that is neither
POSIX-compliant, nor portable in any practical sense.

@geindex Python Enhancement Proposals; PEP 539
PEP 539@footnote{https://www.python.org/dev/peps/pep-0539} changes this by providing a new @ref{5f8,,Thread Specific Storage (TSS) API} to CPython which supersedes use of the
existing TLS API within the CPython interpreter, while deprecating the existing
API.  The TSS API uses a new type @ref{5f9,,Py_tss_t} instead of int
to represent TSS keys–an opaque type the definition of which may depend on
the underlying TLS implementation.  Therefore, this will allow to build CPython
on platforms where the native TLS key is defined in a way that cannot be safely
cast to int.

Note that on platforms where the native TLS key is defined in a way that cannot
be safely cast to int, all functions of the existing TLS API will be
no-op and immediately return failure. This indicates clearly that the old API
is not supported on platforms where it cannot be used reliably, and that no
effort will be made to add such support.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 539

@item PEP 539@footnote{https://www.python.org/dev/peps/pep-0539} – A New C-API for Thread-Local Storage in CPython

PEP written by Erik M. Bray; implementation by Masayuki Yamamoto.
@end table


@node PEP 562 Customization of Access to Module Attributes,PEP 564 New Time Functions With Nanosecond Resolution,PEP 539 New C API for Thread-Local Storage,New Features<8>
@anchor{whatsnew/3 7 pep-562-customization-of-access-to-module-attributes}@anchor{5fa}@anchor{whatsnew/3 7 whatsnew37-pep562}@anchor{5dc}
@subsubsection PEP 562: Customization of Access to Module Attributes


Python 3.7 allows defining @code{__getattr__()} on modules and will call
it whenever a module attribute is otherwise not found.  Defining
@code{__dir__()} on modules is now also allowed.

A typical example of where this may be useful is module attribute deprecation
and lazy loading.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 562

@item PEP 562@footnote{https://www.python.org/dev/peps/pep-0562} – Module @code{__getattr__} and @code{__dir__}

PEP written and implemented by Ivan Levkivskyi
@end table


@node PEP 564 New Time Functions With Nanosecond Resolution,PEP 565 Show DeprecationWarning in __main__,PEP 562 Customization of Access to Module Attributes,New Features<8>
@anchor{whatsnew/3 7 pep-564-new-time-functions-with-nanosecond-resolution}@anchor{5fb}@anchor{whatsnew/3 7 whatsnew37-pep564}@anchor{5e0}
@subsubsection PEP 564: New Time Functions With Nanosecond Resolution


The resolution of clocks in modern systems can exceed the limited precision
of a floating point number returned by the @ref{5fc,,time.time()} function
and its variants.  To avoid loss of precision, 
@geindex Python Enhancement Proposals; PEP 564
PEP 564@footnote{https://www.python.org/dev/peps/pep-0564} adds six new
“nanosecond” variants of the existing timer functions to the @ref{10a,,time}
module:


@itemize *

@item 
@ref{5fd,,time.clock_gettime_ns()}

@item 
@ref{5fe,,time.clock_settime_ns()}

@item 
@ref{5ff,,time.monotonic_ns()}

@item 
@ref{600,,time.perf_counter_ns()}

@item 
@ref{601,,time.process_time_ns()}

@item 
@ref{602,,time.time_ns()}
@end itemize

The new functions return the number of nanoseconds as an integer value.

Measurements@footnote{https://www.python.org/dev/peps/pep-0564/#annex-clocks-resolution-in-python}
show that on Linux and Windows the resolution of @ref{602,,time.time_ns()} is
approximately 3 times better than that of @ref{5fc,,time.time()}.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 564

@item PEP 564@footnote{https://www.python.org/dev/peps/pep-0564} – Add new time functions with nanosecond resolution

PEP written and implemented by Victor Stinner
@end table


@node PEP 565 Show DeprecationWarning in __main__,PEP 560 Core Support for typing module and Generic Types,PEP 564 New Time Functions With Nanosecond Resolution,New Features<8>
@anchor{whatsnew/3 7 pep-565-show-deprecationwarning-in-main}@anchor{603}@anchor{whatsnew/3 7 whatsnew37-pep565}@anchor{5e5}
@subsubsection PEP 565: Show DeprecationWarning in @code{__main__}


The default handling of @ref{1b2,,DeprecationWarning} has been changed such that
these warnings are once more shown by default, but only when the code
triggering them is running directly in the @ref{1,,__main__} module.  As a result,
developers of single file scripts and those using Python interactively should
once again start seeing deprecation warnings for the APIs they use, but
deprecation warnings triggered by imported application, library and framework
modules will continue to be hidden by default.

As a result of this change, the standard library now allows developers to choose
between three different deprecation warning behaviours:


@itemize *

@item 
@ref{604,,FutureWarning}: always displayed by default, recommended for warnings
intended to be seen by application end users (e.g. for deprecated application
configuration settings).

@item 
@ref{1b2,,DeprecationWarning}: displayed by default only in @ref{1,,__main__} and when
running tests, recommended for warnings intended to be seen by other Python
developers where a version upgrade may result in changed behaviour or an
error.

@item 
@ref{3a9,,PendingDeprecationWarning}: displayed by default only when running
tests, intended for cases where a future version upgrade will change the
warning category to @ref{1b2,,DeprecationWarning} or @ref{604,,FutureWarning}.
@end itemize

Previously both @ref{1b2,,DeprecationWarning} and @ref{3a9,,PendingDeprecationWarning}
were only visible when running tests, which meant that developers primarily
writing single file scripts or using Python interactively could be surprised
by breaking changes in the APIs they used.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 565

@item PEP 565@footnote{https://www.python.org/dev/peps/pep-0565} – Show DeprecationWarning in @code{__main__}

PEP written and implemented by Nick Coghlan
@end table


@node PEP 560 Core Support for typing module and Generic Types,PEP 552 Hash-based pyc Files,PEP 565 Show DeprecationWarning in __main__,New Features<8>
@anchor{whatsnew/3 7 pep-560-core-support-for-typing-module-and-generic-types}@anchor{605}@anchor{whatsnew/3 7 whatsnew37-pep560}@anchor{5dd}
@subsubsection PEP 560: Core Support for @code{typing} module and Generic Types


Initially 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} was designed in such way that it would not introduce @emph{any}
changes to the core CPython interpreter. Now type hints and the @ref{120,,typing}
module are extensively used by the community, so this restriction is removed.
The PEP introduces two special methods @code{__class_getitem__()} and
@code{__mro_entries__}, these methods are now used by most classes and special
constructs in @ref{120,,typing}. As a result, the speed of various operations
with types increased up to 7 times, the generic types can be used without
metaclass conflicts, and several long standing bugs in @ref{120,,typing} module are
fixed.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 560

@item PEP 560@footnote{https://www.python.org/dev/peps/pep-0560} – Core support for typing module and generic types

PEP written and implemented by Ivan Levkivskyi
@end table


@node PEP 552 Hash-based pyc Files,PEP 545 Python Documentation Translations,PEP 560 Core Support for typing module and Generic Types,New Features<8>
@anchor{whatsnew/3 7 pep-552-hash-based-pyc-files}@anchor{606}@anchor{whatsnew/3 7 whatsnew37-pep552}@anchor{5e3}
@subsubsection PEP 552: Hash-based .pyc Files


Python has traditionally checked the up-to-dateness of bytecode cache files
(i.e., @code{.pyc} files) by comparing the source metadata (last-modified timestamp
and size) with source metadata saved in the cache file header when it was
generated. While effective, this invalidation method has its drawbacks.  When
filesystem timestamps are too coarse, Python can miss source updates, leading to
user confusion. Additionally, having a timestamp in the cache file is
problematic for build reproducibility@footnote{https://reproducible-builds.org/} and
content-based build systems.

@geindex Python Enhancement Proposals; PEP 552
PEP 552@footnote{https://www.python.org/dev/peps/pep-0552} extends the pyc format to allow the hash of the source file to be
used for invalidation instead of the source timestamp. Such @code{.pyc} files are
called “hash-based”. By default, Python still uses timestamp-based invalidation
and does not generate hash-based @code{.pyc} files at runtime. Hash-based @code{.pyc}
files may be generated with @ref{d2,,py_compile} or @ref{1f,,compileall}.

Hash-based @code{.pyc} files come in two variants: checked and unchecked. Python
validates checked hash-based @code{.pyc} files against the corresponding source
files at runtime but doesn’t do so for unchecked hash-based pycs. Unchecked
hash-based @code{.pyc} files are a useful performance optimization for environments
where a system external to Python (e.g., the build system) is responsible for
keeping @code{.pyc} files up-to-date.

See @ref{607,,Cached bytecode invalidation} for more information.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 552

@item PEP 552@footnote{https://www.python.org/dev/peps/pep-0552} – Deterministic pycs

PEP written and implemented by Benjamin Peterson
@end table


@node PEP 545 Python Documentation Translations,Python Development Mode -X dev,PEP 552 Hash-based pyc Files,New Features<8>
@anchor{whatsnew/3 7 pep-545-python-documentation-translations}@anchor{608}@anchor{whatsnew/3 7 whatsnew37-pep545}@anchor{5e7}
@subsubsection PEP 545: Python Documentation Translations


@geindex Python Enhancement Proposals; PEP 545
PEP 545@footnote{https://www.python.org/dev/peps/pep-0545} describes the process of creating and maintaining Python
documentation translations.

Three new translations have been added:


@itemize -

@item 
Japanese: @indicateurl{https://docs.python.org/ja/}

@item 
French: @indicateurl{https://docs.python.org/fr/}

@item 
Korean: @indicateurl{https://docs.python.org/ko/}
@end itemize


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 545

@item PEP 545@footnote{https://www.python.org/dev/peps/pep-0545} – Python Documentation Translations

PEP written and implemented by Julien Palard, Inada Naoki, and
Victor Stinner.
@end table


@node Python Development Mode -X dev,,PEP 545 Python Documentation Translations,New Features<8>
@anchor{whatsnew/3 7 python-development-mode-x-dev}@anchor{609}@anchor{whatsnew/3 7 whatsnew37-devmode}@anchor{5e4}
@subsubsection Python Development Mode (-X dev)


The new @ref{159,,-X} @code{dev} command line option or the new
@geindex PYTHONDEVMODE
@geindex environment variable; PYTHONDEVMODE
@ref{60a,,PYTHONDEVMODE} environment variable can be used to enable
@ref{3ac,,Python Development Mode}.  When in development mode, Python performs
additional runtime checks that are too expensive to be enabled by default.
See @ref{3ac,,Python Development Mode} documentation for the full
description.

@node Other Language Changes<5>,New Modules<5>,New Features<8>,What’s New In Python 3 7
@anchor{whatsnew/3 7 other-language-changes}@anchor{60b}
@subsection Other Language Changes



@itemize *

@item 
An @ref{4d5,,await} expression and comprehensions containing an
@ref{5cd,,async for} clause were illegal in the expressions in
@ref{4b0,,formatted string literals} due to a problem with the
implementation.  In Python 3.7 this restriction was lifted.

@item 
More than 255 arguments can now be passed to a function, and a function can
now have more than 255 parameters. (Contributed by Serhiy Storchaka in
bpo-12844@footnote{https://bugs.python.org/issue12844} and bpo-18896@footnote{https://bugs.python.org/issue18896}.)

@item 
@ref{60c,,bytes.fromhex()} and @ref{60d,,bytearray.fromhex()} now ignore all ASCII
whitespace, not only spaces. (Contributed by Robert Xiao in bpo-28927@footnote{https://bugs.python.org/issue28927}.)

@item 
@ref{2ef,,str}, @ref{2f0,,bytes}, and @ref{2f1,,bytearray} gained support for
the new @ref{60e,,isascii()} method, which can be used to
test if a string or bytes contain only the ASCII characters.
(Contributed by INADA Naoki in bpo-32677@footnote{https://bugs.python.org/issue32677}.)

@item 
@ref{3b8,,ImportError} now displays module name and module @code{__file__} path when
@code{from ... import ...} fails. (Contributed by Matthias Bussonnier in
bpo-29546@footnote{https://bugs.python.org/issue29546}.)

@item 
Circular imports involving absolute imports with binding a submodule to
a name are now supported.
(Contributed by Serhiy Storchaka in bpo-30024@footnote{https://bugs.python.org/issue30024}.)

@item 
@code{object.__format__(x, '@w{'})} is now equivalent to @code{str(x)} rather than
@code{format(str(self), '@w{'})}.
(Contributed by Serhiy Storchaka in bpo-28974@footnote{https://bugs.python.org/issue28974}.)

@item 
In order to better support dynamic creation of stack traces,
@ref{60f,,types.TracebackType} can now be instantiated from Python code, and
the @code{tb_next} attribute on @ref{610,,tracebacks} is now
writable.
(Contributed by Nathaniel J. Smith in bpo-30579@footnote{https://bugs.python.org/issue30579}.)

@item 
When using the @ref{611,,-m} switch, @code{sys.path[0]} is now eagerly expanded
to the full starting directory path, rather than being left as the empty
directory (which allows imports from the @emph{current} working directory at the
time when an import occurs)
(Contributed by Nick Coghlan in bpo-33053@footnote{https://bugs.python.org/issue33053}.)

@item 
The new @ref{159,,-X} @code{importtime} option or the
@geindex PYTHONPROFILEIMPORTTIME
@geindex environment variable; PYTHONPROFILEIMPORTTIME
@ref{612,,PYTHONPROFILEIMPORTTIME} environment variable can be used to show
the timing of each module import.
(Contributed by Victor Stinner in bpo-31415@footnote{https://bugs.python.org/issue31415}.)
@end itemize

@node New Modules<5>,Improved Modules<5>,Other Language Changes<5>,What’s New In Python 3 7
@anchor{whatsnew/3 7 new-modules}@anchor{613}
@subsection New Modules


@menu
* contextvars:: 
* dataclasses: dataclasses<2>. 
* importlib.resources: importlib resources. 

@end menu

@node contextvars,dataclasses<2>,,New Modules<5>
@anchor{whatsnew/3 7 contextvars}@anchor{614}@anchor{whatsnew/3 7 whatsnew37-pep567}@anchor{5d7}
@subsubsection contextvars


The new @ref{23,,contextvars} module and a set of
@ref{615,,new C APIs} introduce
support for @emph{context variables}.  Context variables are conceptually
similar to thread-local variables.  Unlike TLS, context variables
support asynchronous code correctly.

The @ref{9,,asyncio} and @ref{34,,decimal} modules have been updated to use
and support context variables out of the box.  Particularly the active
decimal context is now stored in a context variable, which allows
decimal operations to work with the correct context in asynchronous code.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 567

@item PEP 567@footnote{https://www.python.org/dev/peps/pep-0567} – Context Variables

PEP written and implemented by Yury Selivanov
@end table


@node dataclasses<2>,importlib resources,contextvars,New Modules<5>
@anchor{whatsnew/3 7 dataclasses}@anchor{616}@anchor{whatsnew/3 7 whatsnew37-pep557}@anchor{5d8}
@subsubsection dataclasses


The new @ref{276,,dataclass()} decorator provides a way to declare
@emph{data classes}.  A data class describes its attributes using class variable
annotations.  Its constructor and other magic methods, such as
@ref{617,,__repr__()}, @ref{618,,__eq__()}, and
@ref{619,,__hash__()} are generated automatically.

Example:

@example
@@dataclass
class Point:
    x: float
    y: float
    z: float = 0.0

p = Point(1.5, 2.5)
print(p)   # produces "Point(x=1.5, y=2.5, z=0.0)"
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 557

@item PEP 557@footnote{https://www.python.org/dev/peps/pep-0557} – Data Classes

PEP written and implemented by Eric V. Smith
@end table


@node importlib resources,,dataclasses<2>,New Modules<5>
@anchor{whatsnew/3 7 importlib-resources}@anchor{61a}@anchor{whatsnew/3 7 whatsnew37-importlib-resources}@anchor{5d9}
@subsubsection importlib.resources


The new @ref{9a,,importlib.resources} module provides several new APIs and one
new ABC for access to, opening, and reading @emph{resources} inside packages.
Resources are roughly similar to files inside packages, but they needn’t
be actual files on the physical file system.  Module loaders can provide a
@code{get_resource_reader()} function which returns
a @ref{61b,,importlib.abc.ResourceReader} instance to support this
new API.  Built-in file path loaders and zip file loaders both support this.

Contributed by Barry Warsaw and Brett Cannon in bpo-32248@footnote{https://bugs.python.org/issue32248}.


@subsubheading See also


importlib_resources@footnote{http://importlib-resources.readthedocs.io/en/latest/}
– a PyPI backport for earlier Python versions.


@node Improved Modules<5>,C API Changes<4>,New Modules<5>,What’s New In Python 3 7
@anchor{whatsnew/3 7 improved-modules}@anchor{61c}
@subsection Improved Modules


@menu
* argparse: argparse<2>. 
* asyncio: asyncio<4>. 
* binascii:: 
* calendar:: 
* collections: collections<2>. 
* compileall: compileall<2>. 
* concurrent.futures: concurrent futures<2>. 
* contextlib: contextlib<2>. 
* cProfile: cProfile<2>. 
* crypt:: 
* datetime: datetime<3>. 
* dbm:: 
* decimal:: 
* dis:: 
* distutils: distutils<3>. 
* enum: enum<2>. 
* functools: functools<2>. 
* gc: gc<4>. 
* hmac: hmac<2>. 
* http.client: http client. 
* http.server: http server. 
* idlelib and IDLE:: 
* importlib: importlib<2>. 
* io: io<2>. 
* ipaddress: ipaddress<2>. 
* itertools: itertools<2>. 
* locale:: 
* logging: logging<2>. 
* math: math<4>. 
* mimetypes:: 
* msilib:: 
* multiprocessing: multiprocessing<3>. 
* os: os<5>. 
* pathlib: pathlib<4>. 
* pdb: pdb<2>. 
* py_compile: py_compile<3>. 
* pydoc: pydoc<2>. 
* queue:: 
* re:: 
* signal: signal<2>. 
* socket: socket<5>. 
* socketserver:: 
* sqlite3: sqlite3<3>. 
* ssl: ssl<3>. 
* string:: 
* subprocess:: 
* sys: sys<5>. 
* time: time<4>. 
* tkinter: tkinter<2>. 
* tracemalloc: tracemalloc<2>. 
* types: types<2>. 
* unicodedata: unicodedata<4>. 
* unittest: unittest<3>. 
* unittest.mock: unittest mock. 
* urllib.parse: urllib parse<3>. 
* uu:: 
* uuid:: 
* warnings:: 
* xml.etree: xml etree. 
* xmlrpc.server: xmlrpc server. 
* zipapp:: 
* zipfile:: 

@end menu

@node argparse<2>,asyncio<4>,,Improved Modules<5>
@anchor{whatsnew/3 7 argparse}@anchor{61d}
@subsubsection argparse


The new @ref{61e,,ArgumentParser.parse_intermixed_args()}
method allows intermixing options and positional arguments.
(Contributed by paul.j3 in bpo-14191@footnote{https://bugs.python.org/issue14191}.)

@node asyncio<4>,binascii,argparse<2>,Improved Modules<5>
@anchor{whatsnew/3 7 asyncio}@anchor{61f}@anchor{whatsnew/3 7 whatsnew37-asyncio}@anchor{5df}
@subsubsection asyncio


The @ref{9,,asyncio} module has received many new features, usability and
@ref{620,,performance improvements}.  Notable changes
include:


@itemize *

@item 
The new @ref{621,,provisional} @ref{3d1,,asyncio.run()} function can
be used to run a coroutine from synchronous code by automatically creating and
destroying the event loop.
(Contributed by Yury Selivanov in bpo-32314@footnote{https://bugs.python.org/issue32314}.)

@item 
asyncio gained support for @ref{23,,contextvars}.
@ref{622,,loop.call_soon()},
@ref{623,,loop.call_soon_threadsafe()},
@ref{624,,loop.call_later()},
@ref{625,,loop.call_at()}, and
@ref{626,,Future.add_done_callback()}
have a new optional keyword-only @emph{context} parameter.
@ref{317,,Tasks} now track their context automatically.
See 
@geindex Python Enhancement Proposals; PEP 567
PEP 567@footnote{https://www.python.org/dev/peps/pep-0567} for more details.
(Contributed by Yury Selivanov in bpo-32436@footnote{https://bugs.python.org/issue32436}.)

@item 
The new @ref{4dc,,asyncio.create_task()} function has been added as a shortcut
to @code{asyncio.get_event_loop().create_task()}.
(Contributed by Andrew Svetlov in bpo-32311@footnote{https://bugs.python.org/issue32311}.)

@item 
The new @ref{627,,loop.start_tls()}
method can be used to upgrade an existing connection to TLS.
(Contributed by Yury Selivanov in bpo-23749@footnote{https://bugs.python.org/issue23749}.)

@item 
The new @ref{628,,loop.sock_recv_into()}
method allows reading data from a socket directly into a provided buffer making
it possible to reduce data copies.
(Contributed by Antoine Pitrou in bpo-31819@footnote{https://bugs.python.org/issue31819}.)

@item 
The new @ref{458,,asyncio.current_task()} function returns the currently running
@ref{317,,Task} instance, and the new @ref{459,,asyncio.all_tasks()}
function returns a set of all existing @code{Task} instances in a given loop.
The @code{Task.current_task()} and
@code{Task.all_tasks()} methods have been deprecated.
(Contributed by Andrew Svetlov in bpo-32250@footnote{https://bugs.python.org/issue32250}.)

@item 
The new @emph{provisional} @ref{5bd,,BufferedProtocol} class allows
implementing streaming protocols with manual control over the receive buffer.
(Contributed by Yury Selivanov in bpo-32251@footnote{https://bugs.python.org/issue32251}.)

@item 
The new @ref{315,,asyncio.get_running_loop()} function returns the currently
running loop, and raises a @ref{5af,,RuntimeError} if no loop is running.
This is in contrast with @ref{314,,asyncio.get_event_loop()}, which will @emph{create}
a new event loop if none is running.
(Contributed by Yury Selivanov in bpo-32269@footnote{https://bugs.python.org/issue32269}.)

@item 
The new @ref{629,,StreamWriter.wait_closed()}
coroutine method allows waiting until the stream writer is closed.  The new
@ref{62a,,StreamWriter.is_closing()} method
can be used to determine if the writer is closing.
(Contributed by Andrew Svetlov in bpo-32391@footnote{https://bugs.python.org/issue32391}.)

@item 
The new @ref{62b,,loop.sock_sendfile()}
coroutine method allows sending files using @ref{62c,,os.sendfile} when possible.
(Contributed by Andrew Svetlov in bpo-32410@footnote{https://bugs.python.org/issue32410}.)

@item 
The new @ref{62d,,Future.get_loop()} and
@code{Task.get_loop()} methods return the instance of the loop on which a task or
a future were created.
@ref{62e,,Server.get_loop()} allows doing the same for
@ref{62f,,asyncio.Server} objects.
(Contributed by Yury Selivanov in bpo-32415@footnote{https://bugs.python.org/issue32415} and
Srinivas Reddy Thatiparthy in bpo-32418@footnote{https://bugs.python.org/issue32418}.)

@item 
It is now possible to control how instances of @ref{62f,,asyncio.Server} begin
serving.  Previously, the server would start serving immediately when created.
The new @emph{start_serving} keyword argument to
@ref{630,,loop.create_server()} and
@ref{631,,loop.create_unix_server()},
as well as @ref{632,,Server.start_serving()}, and
@ref{633,,Server.serve_forever()}
can be used to decouple server instantiation and serving.  The new
@ref{634,,Server.is_serving()} method returns @code{True}
if the server is serving.  @ref{62f,,Server} objects are now
asynchronous context managers:

@example
srv = await loop.create_server(...)

async with srv:
    # some code

# At this point, srv is closed and no longer accepts new connections.
@end example

(Contributed by Yury Selivanov in bpo-32662@footnote{https://bugs.python.org/issue32662}.)

@item 
Callback objects returned by
@ref{624,,loop.call_later()}
gained the new @ref{635,,when()} method which
returns an absolute scheduled callback timestamp.
(Contributed by Andrew Svetlov in bpo-32741@footnote{https://bugs.python.org/issue32741}.)

@item 
The @ref{1b9,,loop.create_datagram_endpoint()} method
gained support for Unix sockets.
(Contributed by Quentin Dawans in bpo-31245@footnote{https://bugs.python.org/issue31245}.)

@item 
The @ref{636,,asyncio.open_connection()}, @ref{637,,asyncio.start_server()} functions,
@ref{4e0,,loop.create_connection()},
@ref{630,,loop.create_server()},
@ref{638,,loop.create_accepted_socket()}
methods and their corresponding UNIX socket variants now accept the
@emph{ssl_handshake_timeout} keyword argument.
(Contributed by Neil Aspinall in bpo-29970@footnote{https://bugs.python.org/issue29970}.)

@item 
The new @ref{639,,Handle.cancelled()} method returns
@code{True} if the callback was cancelled.
(Contributed by Marat Sharafutdinov in bpo-31943@footnote{https://bugs.python.org/issue31943}.)

@item 
The asyncio source has been converted to use the
@ref{4d4,,async}/@ref{4d5,,await} syntax.
(Contributed by Andrew Svetlov in bpo-32193@footnote{https://bugs.python.org/issue32193}.)

@item 
The new @ref{63a,,ReadTransport.is_reading()}
method can be used to determine the reading state of the transport.
Additionally, calls to
@ref{63b,,ReadTransport.resume_reading()}
and @ref{63c,,ReadTransport.pause_reading()}
are now idempotent.
(Contributed by Yury Selivanov in bpo-32356@footnote{https://bugs.python.org/issue32356}.)

@item 
Loop methods which accept socket paths now support passing
@ref{63d,,path-like objects}.
(Contributed by Yury Selivanov in bpo-32066@footnote{https://bugs.python.org/issue32066}.)

@item 
In @ref{9,,asyncio} TCP sockets on Linux are now created with @code{TCP_NODELAY}
flag set by default.
(Contributed by Yury Selivanov and Victor Stinner in bpo-27456@footnote{https://bugs.python.org/issue27456}.)

@item 
Exceptions occurring in cancelled tasks are no longer logged.
(Contributed by Yury Selivanov in bpo-30508@footnote{https://bugs.python.org/issue30508}.)

@item 
New @code{WindowsSelectorEventLoopPolicy} and
@code{WindowsProactorEventLoopPolicy} classes.
(Contributed by Yury Selivanov in bpo-33792@footnote{https://bugs.python.org/issue33792}.)
@end itemize

Several @code{asyncio} APIs have been
@ref{63e,,deprecated}.

@node binascii,calendar,asyncio<4>,Improved Modules<5>
@anchor{whatsnew/3 7 binascii}@anchor{63f}
@subsubsection binascii


The @ref{640,,b2a_uu()} function now accepts an optional @emph{backtick}
keyword argument.  When it’s true, zeros are represented by @code{'`'}
instead of spaces.  (Contributed by Xiang Zhang in bpo-30103@footnote{https://bugs.python.org/issue30103}.)

@node calendar,collections<2>,binascii,Improved Modules<5>
@anchor{whatsnew/3 7 calendar}@anchor{641}
@subsubsection calendar


The @ref{642,,HTMLCalendar} class has new class attributes which ease
the customization of CSS classes in the produced HTML calendar.
(Contributed by Oz Tiram in bpo-30095@footnote{https://bugs.python.org/issue30095}.)

@node collections<2>,compileall<2>,calendar,Improved Modules<5>
@anchor{whatsnew/3 7 collections}@anchor{643}
@subsubsection collections


@code{collections.namedtuple()} now supports default values.
(Contributed by Raymond Hettinger in bpo-32320@footnote{https://bugs.python.org/issue32320}.)

@node compileall<2>,concurrent futures<2>,collections<2>,Improved Modules<5>
@anchor{whatsnew/3 7 compileall}@anchor{644}
@subsubsection compileall


@ref{645,,compileall.compile_dir()} learned the new @emph{invalidation_mode} parameter,
which can be used to enable
@ref{5e3,,hash-based .pyc invalidation}.  The invalidation
mode can also be specified on the command line using the new
@code{--invalidation-mode} argument.
(Contributed by Benjamin Peterson in bpo-31650@footnote{https://bugs.python.org/issue31650}.)

@node concurrent futures<2>,contextlib<2>,compileall<2>,Improved Modules<5>
@anchor{whatsnew/3 7 concurrent-futures}@anchor{646}
@subsubsection concurrent.futures


@ref{3da,,ProcessPoolExecutor} and
@ref{1c7,,ThreadPoolExecutor} now
support the new @emph{initializer} and @emph{initargs} constructor arguments.
(Contributed by Antoine Pitrou in bpo-21423@footnote{https://bugs.python.org/issue21423}.)

The @ref{3da,,ProcessPoolExecutor}
can now take the multiprocessing context via the new @emph{mp_context} argument.
(Contributed by Thomas Moreau in bpo-31540@footnote{https://bugs.python.org/issue31540}.)

@node contextlib<2>,cProfile<2>,concurrent futures<2>,Improved Modules<5>
@anchor{whatsnew/3 7 contextlib}@anchor{647}
@subsubsection contextlib


The new @ref{26d,,nullcontext()} is a simpler and faster no-op
context manager than @ref{648,,ExitStack}.
(Contributed by Jesse-Bakker in bpo-10049@footnote{https://bugs.python.org/issue10049}.)

The new @ref{649,,asynccontextmanager()},
@ref{64a,,AbstractAsyncContextManager}, and
@ref{64b,,AsyncExitStack} have been added to
complement their synchronous counterparts.  (Contributed
by Jelle Zijlstra in bpo-29679@footnote{https://bugs.python.org/issue29679} and bpo-30241@footnote{https://bugs.python.org/issue30241},
and by Alexander Mohr and Ilya Kulakov in bpo-29302@footnote{https://bugs.python.org/issue29302}.)

@node cProfile<2>,crypt,contextlib<2>,Improved Modules<5>
@anchor{whatsnew/3 7 cprofile}@anchor{64c}
@subsubsection cProfile


The @ref{26,,cProfile} command line now accepts @code{-m module_name} as an
alternative to script path.  (Contributed by Sanyam Khurana in bpo-21862@footnote{https://bugs.python.org/issue21862}.)

@node crypt,datetime<3>,cProfile<2>,Improved Modules<5>
@anchor{whatsnew/3 7 crypt}@anchor{64d}
@subsubsection crypt


The @ref{27,,crypt} module now supports the Blowfish hashing method.
(Contributed by Serhiy Storchaka in bpo-31664@footnote{https://bugs.python.org/issue31664}.)

The @ref{64e,,mksalt()} function now allows specifying the number of rounds
for hashing.  (Contributed by Serhiy Storchaka in bpo-31702@footnote{https://bugs.python.org/issue31702}.)

@node datetime<3>,dbm,crypt,Improved Modules<5>
@anchor{whatsnew/3 7 datetime}@anchor{64f}
@subsubsection datetime


The new @ref{650,,datetime.fromisoformat()}
method constructs a @ref{363,,datetime} object from a string
in one of the formats output by
@ref{651,,datetime.isoformat()}.
(Contributed by Paul Ganssle in bpo-15873@footnote{https://bugs.python.org/issue15873}.)

The @ref{3c6,,tzinfo} class now supports sub-minute offsets.
(Contributed by Alexander Belopolsky in bpo-5288@footnote{https://bugs.python.org/issue5288}.)

@node dbm,decimal,datetime<3>,Improved Modules<5>
@anchor{whatsnew/3 7 dbm}@anchor{652}
@subsubsection dbm


@ref{31,,dbm.dumb} now supports reading read-only files and no longer writes the
index file when it is not changed.

@node decimal,dis,dbm,Improved Modules<5>
@anchor{whatsnew/3 7 decimal}@anchor{653}
@subsubsection decimal


The @ref{34,,decimal} module now uses @ref{5d7,,context variables}
to store the decimal context.
(Contributed by Yury Selivanov in bpo-32630@footnote{https://bugs.python.org/issue32630}.)

@node dis,distutils<3>,decimal,Improved Modules<5>
@anchor{whatsnew/3 7 dis}@anchor{654}
@subsubsection dis


The @ref{655,,dis()} function is now able to
disassemble nested code objects (the code of comprehensions, generator
expressions and nested functions, and the code used for building nested
classes).  The maximum depth of disassembly recursion is controlled by
the new @emph{depth} parameter.
(Contributed by Serhiy Storchaka in bpo-11822@footnote{https://bugs.python.org/issue11822}.)

@node distutils<3>,enum<2>,dis,Improved Modules<5>
@anchor{whatsnew/3 7 distutils}@anchor{656}
@subsubsection distutils


@code{README.rst} is now included in the list of distutils standard READMEs and
therefore included in source distributions.
(Contributed by Ryan Gonzalez in bpo-11913@footnote{https://bugs.python.org/issue11913}.)

@node enum<2>,functools<2>,distutils<3>,Improved Modules<5>
@anchor{whatsnew/3 7 enum}@anchor{657}
@subsubsection enum


The @ref{658,,Enum} learned the new @code{_ignore_} class property,
which allows listing the names of properties which should not become
enum members.
(Contributed by Ethan Furman in bpo-31801@footnote{https://bugs.python.org/issue31801}.)

In Python 3.8, attempting to check for non-Enum objects in @code{Enum}
classes will raise a @ref{162,,TypeError} (e.g. @code{1 in Color}); similarly,
attempting to check for non-Flag objects in a @code{Flag} member will
raise @ref{162,,TypeError} (e.g. @code{1 in Perm.RW}); currently, both operations
return @ref{659,,False} instead and are deprecated.
(Contributed by Ethan Furman in bpo-33217@footnote{https://bugs.python.org/issue33217}.)

@node functools<2>,gc<4>,enum<2>,Improved Modules<5>
@anchor{whatsnew/3 7 functools}@anchor{65a}
@subsubsection functools


@ref{65b,,functools.singledispatch()} now supports registering implementations
using type annotations.
(Contributed by Łukasz Langa in bpo-32227@footnote{https://bugs.python.org/issue32227}.)

@node gc<4>,hmac<2>,functools<2>,Improved Modules<5>
@anchor{whatsnew/3 7 gc}@anchor{65c}
@subsubsection gc


The new @ref{65d,,gc.freeze()} function allows freezing all objects tracked
by the garbage collector and excluding them from future collections.
This can be used before a POSIX @code{fork()} call to make the GC copy-on-write
friendly or to speed up collection.  The new @ref{65e,,gc.unfreeze()} functions
reverses this operation.  Additionally, @ref{65f,,gc.get_freeze_count()} can
be used to obtain the number of frozen objects.
(Contributed by Li Zekun in bpo-31558@footnote{https://bugs.python.org/issue31558}.)

@node hmac<2>,http client,gc<4>,Improved Modules<5>
@anchor{whatsnew/3 7 hmac}@anchor{660}
@subsubsection hmac


The @ref{8a,,hmac} module now has an optimized one-shot @ref{661,,digest()}
function, which is up to three times faster than @code{HMAC()}.
(Contributed by Christian Heimes in bpo-32433@footnote{https://bugs.python.org/issue32433}.)

@node http client,http server,hmac<2>,Improved Modules<5>
@anchor{whatsnew/3 7 http-client}@anchor{662}
@subsubsection http.client


@ref{663,,HTTPConnection} and @ref{664,,HTTPSConnection}
now support the new @emph{blocksize} argument for improved upload throughput.
(Contributed by Nir Soffer in bpo-31945@footnote{https://bugs.python.org/issue31945}.)

@node http server,idlelib and IDLE,http client,Improved Modules<5>
@anchor{whatsnew/3 7 http-server}@anchor{665}
@subsubsection http.server


@ref{666,,SimpleHTTPRequestHandler} now supports the HTTP
@code{If-Modified-Since} header.  The server returns the 304 response status if
the target file was not modified after the time specified in the header.
(Contributed by Pierre Quentel in bpo-29654@footnote{https://bugs.python.org/issue29654}.)

@ref{666,,SimpleHTTPRequestHandler} accepts the new @emph{directory}
argument, in addition to the new @code{--directory} command line argument.
With this parameter, the server serves the specified directory, by default it
uses the current working directory.
(Contributed by Stéphane Wirtel and Julien Palard in bpo-28707@footnote{https://bugs.python.org/issue28707}.)

The new @ref{667,,ThreadingHTTPServer} class
uses threads to handle requests using @code{ThreadingMixin}.
It is used when @code{http.server} is run with @code{-m}.
(Contributed by Julien Palard in bpo-31639@footnote{https://bugs.python.org/issue31639}.)

@node idlelib and IDLE,importlib<2>,http server,Improved Modules<5>
@anchor{whatsnew/3 7 idlelib-and-idle}@anchor{668}
@subsubsection idlelib and IDLE


Multiple fixes for autocompletion. (Contributed by Louie Lu in bpo-15786@footnote{https://bugs.python.org/issue15786}.)

Module Browser (on the File menu, formerly called Class Browser),
now displays nested functions and classes in addition to top-level
functions and classes.
(Contributed by Guilherme Polo, Cheryl Sabella, and Terry Jan Reedy
in bpo-1612262@footnote{https://bugs.python.org/issue1612262}.)

The Settings dialog (Options, Configure IDLE) has been partly rewritten
to improve both appearance and function.
(Contributed by Cheryl Sabella and Terry Jan Reedy in multiple issues.)

The font sample now includes a selection of non-Latin characters so that
users can better see the effect of selecting a particular font.
(Contributed by Terry Jan Reedy in bpo-13802@footnote{https://bugs.python.org/issue13802}.)
The sample can be edited to include other characters.
(Contributed by Serhiy Storchaka in bpo-31860@footnote{https://bugs.python.org/issue31860}.)

The IDLE features formerly implemented as extensions have been reimplemented
as normal features.  Their settings have been moved from the Extensions tab
to other dialog tabs.
(Contributed by Charles Wohlganger and Terry Jan Reedy in bpo-27099@footnote{https://bugs.python.org/issue27099}.)

Editor code context option revised.  Box displays all context lines up to
maxlines.  Clicking on a context line jumps the editor to that line.  Context
colors for custom themes is added to Highlights tab of Settings dialog.
(Contributed by Cheryl Sabella and Terry Jan Reedy in bpo-33642@footnote{https://bugs.python.org/issue33642},
bpo-33768@footnote{https://bugs.python.org/issue33768}, and bpo-33679@footnote{https://bugs.python.org/issue33679}.)

On Windows, a new API call tells Windows that tk scales for DPI. On Windows
8.1+ or 10, with DPI compatibility properties of the Python binary
unchanged, and a monitor resolution greater than 96 DPI, this should
make text and lines sharper.  It should otherwise have no effect.
(Contributed by Terry Jan Reedy in bpo-33656@footnote{https://bugs.python.org/issue33656}.)

New in 3.7.1:

Output over N lines (50 by default) is squeezed down to a button.
N can be changed in the PyShell section of the General page of the
Settings dialog.  Fewer, but possibly extra long, lines can be squeezed by
right clicking on the output.  Squeezed output can be expanded in place
by double-clicking the button or into the clipboard or a separate window
by right-clicking the button.  (Contributed by Tal Einat in bpo-1529353@footnote{https://bugs.python.org/issue1529353}.)

The changes above have been backported to 3.6 maintenance releases.

NEW in 3.7.4:

Add “Run Customized” to the Run menu to run a module with customized
settings. Any command line arguments entered are added to sys.argv.
They re-appear in the box for the next customized run.  One can also
suppress the normal Shell main module restart.  (Contributed by Cheryl
Sabella, Terry Jan Reedy, and others in bpo-5680@footnote{https://bugs.python.org/issue5680} and bpo-37627@footnote{https://bugs.python.org/issue37627}.)

New in 3.7.5:

Add optional line numbers for IDLE editor windows. Windows
open without line numbers unless set otherwise in the General
tab of the configuration dialog.  Line numbers for an existing
window are shown and hidden in the Options menu.
(Contributed by Tal Einat and Saimadhav Heblikar in bpo-17535@footnote{https://bugs.python.org/issue17535}.)

@node importlib<2>,io<2>,idlelib and IDLE,Improved Modules<5>
@anchor{whatsnew/3 7 importlib}@anchor{669}
@subsubsection importlib


The @ref{61b,,importlib.abc.ResourceReader} ABC was introduced to
support the loading of resources from packages.  See also
@ref{5d9,,importlib.resources}.
(Contributed by Barry Warsaw, Brett Cannon in bpo-32248@footnote{https://bugs.python.org/issue32248}.)

@ref{66a,,importlib.reload()} now raises @ref{66b,,ModuleNotFoundError} if the module
lacks a spec.
(Contributed by Garvit Khatri in bpo-29851@footnote{https://bugs.python.org/issue29851}.)

@code{importlib.find_spec()} now raises @ref{66b,,ModuleNotFoundError} instead of
@ref{163,,AttributeError} if the specified parent module is not a package (i.e.
lacks a @code{__path__} attribute).
(Contributed by Milan Oberkirch in bpo-30436@footnote{https://bugs.python.org/issue30436}.)

The new @code{importlib.source_hash()} can be used to compute the hash of
the passed source.  A @ref{5e3,,hash-based .pyc file}
embeds the value returned by this function.

@node io<2>,ipaddress<2>,importlib<2>,Improved Modules<5>
@anchor{whatsnew/3 7 io}@anchor{66c}
@subsubsection io


The new @ref{66d,,TextIOWrapper.reconfigure()}
method can be used to reconfigure the text stream with the new settings.
(Contributed by Antoine Pitrou in bpo-30526@footnote{https://bugs.python.org/issue30526} and
INADA Naoki in bpo-15216@footnote{https://bugs.python.org/issue15216}.)

@node ipaddress<2>,itertools<2>,io<2>,Improved Modules<5>
@anchor{whatsnew/3 7 ipaddress}@anchor{66e}
@subsubsection ipaddress


The new @code{subnet_of()} and @code{supernet_of()} methods of
@ref{66f,,ipaddress.IPv6Network} and @ref{670,,ipaddress.IPv4Network} can
be used for network containment tests.
(Contributed by Michel Albert and Cheryl Sabella in bpo-20825@footnote{https://bugs.python.org/issue20825}.)

@node itertools<2>,locale,ipaddress<2>,Improved Modules<5>
@anchor{whatsnew/3 7 itertools}@anchor{671}
@subsubsection itertools


@ref{672,,itertools.islice()} now accepts
@ref{24a,,integer-like objects} as start, stop,
and slice arguments.
(Contributed by Will Roberts in bpo-30537@footnote{https://bugs.python.org/issue30537}.)

@node locale,logging<2>,itertools<2>,Improved Modules<5>
@anchor{whatsnew/3 7 locale}@anchor{673}
@subsubsection locale


The new @emph{monetary} argument to @ref{674,,locale.format_string()} can be used
to make the conversion use monetary thousands separators and
grouping strings.  (Contributed by Garvit in bpo-10379@footnote{https://bugs.python.org/issue10379}.)

The @ref{1b5,,locale.getpreferredencoding()} function now always returns @code{'UTF-8'}
on Android or when in the @ref{5e2,,forced UTF-8 mode}.

@node logging<2>,math<4>,locale,Improved Modules<5>
@anchor{whatsnew/3 7 logging}@anchor{675}
@subsubsection logging


@ref{676,,Logger} instances can now be pickled.
(Contributed by Vinay Sajip in bpo-30520@footnote{https://bugs.python.org/issue30520}.)

The new @ref{677,,StreamHandler.setStream()}
method can be used to replace the logger stream after handler creation.
(Contributed by Vinay Sajip in bpo-30522@footnote{https://bugs.python.org/issue30522}.)

It is now possible to specify keyword arguments to handler constructors in
configuration passed to @ref{678,,logging.config.fileConfig()}.
(Contributed by Preston Landers in bpo-31080@footnote{https://bugs.python.org/issue31080}.)

@node math<4>,mimetypes,logging<2>,Improved Modules<5>
@anchor{whatsnew/3 7 math}@anchor{679}
@subsubsection math


The new @ref{67a,,math.remainder()} function implements the IEEE 754-style remainder
operation.  (Contributed by Mark Dickinson in bpo-29962@footnote{https://bugs.python.org/issue29962}.)

@node mimetypes,msilib,math<4>,Improved Modules<5>
@anchor{whatsnew/3 7 mimetypes}@anchor{67b}
@subsubsection mimetypes


The MIME type of .bmp has been changed from @code{'image/x-ms-bmp'} to
@code{'image/bmp'}.
(Contributed by Nitish Chandra in bpo-22589@footnote{https://bugs.python.org/issue22589}.)

@node msilib,multiprocessing<3>,mimetypes,Improved Modules<5>
@anchor{whatsnew/3 7 msilib}@anchor{67c}
@subsubsection msilib


The new @ref{67d,,Database.Close()} method can be used
to close the @abbr{MSI} database.
(Contributed by Berker Peksag in bpo-20486@footnote{https://bugs.python.org/issue20486}.)

@node multiprocessing<3>,os<5>,msilib,Improved Modules<5>
@anchor{whatsnew/3 7 multiprocessing}@anchor{67e}
@subsubsection multiprocessing


The new @ref{67f,,Process.close()} method
explicitly closes the process object and releases all resources associated
with it.  @ref{17c,,ValueError} is raised if the underlying process is still
running.
(Contributed by Antoine Pitrou in bpo-30596@footnote{https://bugs.python.org/issue30596}.)

The new @ref{680,,Process.kill()} method can
be used to terminate the process using the @code{SIGKILL} signal on Unix.
(Contributed by Vitor Pereira in bpo-30794@footnote{https://bugs.python.org/issue30794}.)

Non-daemonic threads created by @ref{681,,Process} are now
joined on process exit.
(Contributed by Antoine Pitrou in bpo-18966@footnote{https://bugs.python.org/issue18966}.)

@node os<5>,pathlib<4>,multiprocessing<3>,Improved Modules<5>
@anchor{whatsnew/3 7 os}@anchor{682}
@subsubsection os


@ref{683,,os.fwalk()} now accepts the @emph{path} argument as @ref{2f0,,bytes}.
(Contributed by Serhiy Storchaka in bpo-28682@footnote{https://bugs.python.org/issue28682}.)

@ref{576,,os.scandir()} gained support for @ref{684,,file descriptors}.
(Contributed by Serhiy Storchaka in bpo-25996@footnote{https://bugs.python.org/issue25996}.)

The new @ref{685,,register_at_fork()} function allows registering Python
callbacks to be executed at process fork.
(Contributed by Antoine Pitrou in bpo-16500@footnote{https://bugs.python.org/issue16500}.)

Added @ref{686,,os.preadv()} (combine the functionality of @ref{687,,os.readv()} and
@ref{688,,os.pread()}) and @ref{689,,os.pwritev()} functions (combine the functionality
of @ref{68a,,os.writev()} and @ref{68b,,os.pwrite()}). (Contributed by Pablo Galindo in
bpo-31368@footnote{https://bugs.python.org/issue31368}.)

The mode argument of @ref{68c,,os.makedirs()} no longer affects the file
permission bits of newly-created intermediate-level directories.
(Contributed by Serhiy Storchaka in bpo-19930@footnote{https://bugs.python.org/issue19930}.)

@ref{68d,,os.dup2()} now returns the new file descriptor.  Previously, @code{None}
was always returned.
(Contributed by Benjamin Peterson in bpo-32441@footnote{https://bugs.python.org/issue32441}.)

The structure returned by @ref{516,,os.stat()} now contains the
@ref{68e,,st_fstype} attribute on Solaris and its derivatives.
(Contributed by Jesús Cea Avión in bpo-32659@footnote{https://bugs.python.org/issue32659}.)

@node pathlib<4>,pdb<2>,os<5>,Improved Modules<5>
@anchor{whatsnew/3 7 pathlib}@anchor{68f}
@subsubsection pathlib


The new @ref{526,,Path.is_mount()} method is now available
on POSIX systems and can be used to determine whether a path is a mount point.
(Contributed by Cooper Ry Lees in bpo-30897@footnote{https://bugs.python.org/issue30897}.)

@node pdb<2>,py_compile<3>,pathlib<4>,Improved Modules<5>
@anchor{whatsnew/3 7 pdb}@anchor{690}
@subsubsection pdb


@ref{691,,pdb.set_trace()} now takes an optional @emph{header} keyword-only
argument.  If given, it is printed to the console just before debugging
begins.  (Contributed by Barry Warsaw in bpo-31389@footnote{https://bugs.python.org/issue31389}.)

@ref{c4,,pdb} command line now accepts @code{-m module_name} as an alternative to
script file.  (Contributed by Mario Corchero in bpo-32206@footnote{https://bugs.python.org/issue32206}.)

@node py_compile<3>,pydoc<2>,pdb<2>,Improved Modules<5>
@anchor{whatsnew/3 7 py-compile}@anchor{692}
@subsubsection py_compile


@ref{534,,py_compile.compile()} – and by extension, @ref{1f,,compileall} – now
respects the 
@geindex SOURCE_DATE_EPOCH
@geindex environment variable; SOURCE_DATE_EPOCH
@code{SOURCE_DATE_EPOCH} environment variable by
unconditionally creating @code{.pyc} files for hash-based validation.
This allows for guaranteeing
reproducible builds@footnote{https://reproducible-builds.org/} of @code{.pyc}
files when they are created eagerly. (Contributed by Bernhard M. Wiedemann
in bpo-29708@footnote{https://bugs.python.org/issue29708}.)

@node pydoc<2>,queue,py_compile<3>,Improved Modules<5>
@anchor{whatsnew/3 7 pydoc}@anchor{693}
@subsubsection pydoc


The pydoc server can now bind to an arbitrary hostname specified by the
new @code{-n} command-line argument.
(Contributed by Feanil Patel in bpo-31128@footnote{https://bugs.python.org/issue31128}.)

@node queue,re,pydoc<2>,Improved Modules<5>
@anchor{whatsnew/3 7 queue}@anchor{694}
@subsubsection queue


The new @ref{695,,SimpleQueue} class is an unbounded @abbr{FIFO} queue.
(Contributed by Antoine Pitrou in bpo-14976@footnote{https://bugs.python.org/issue14976}.)

@node re,signal<2>,queue,Improved Modules<5>
@anchor{whatsnew/3 7 re}@anchor{696}
@subsubsection re


The flags @ref{697,,re.ASCII}, @ref{698,,re.LOCALE} and @code{re.UNICODE}
can be set within the scope of a group.
(Contributed by Serhiy Storchaka in bpo-31690@footnote{https://bugs.python.org/issue31690}.)

@ref{699,,re.split()} now supports splitting on a pattern like @code{r'\b'},
@code{'^$'} or @code{(?=-)} that matches an empty string.
(Contributed by Serhiy Storchaka in bpo-25054@footnote{https://bugs.python.org/issue25054}.)

Regular expressions compiled with the @ref{698,,re.LOCALE} flag no longer
depend on the locale at compile time.  Locale settings are applied only
when the compiled regular expression is used.
(Contributed by Serhiy Storchaka in bpo-30215@footnote{https://bugs.python.org/issue30215}.)

@ref{604,,FutureWarning} is now emitted if a regular expression contains
character set constructs that will change semantically in the future,
such as nested sets and set operations.
(Contributed by Serhiy Storchaka in bpo-30349@footnote{https://bugs.python.org/issue30349}.)

Compiled regular expression and match objects can now be copied
using @ref{69a,,copy.copy()} and @ref{69b,,copy.deepcopy()}.
(Contributed by Serhiy Storchaka in bpo-10076@footnote{https://bugs.python.org/issue10076}.)

@node signal<2>,socket<5>,re,Improved Modules<5>
@anchor{whatsnew/3 7 signal}@anchor{69c}
@subsubsection signal


The new @emph{warn_on_full_buffer} argument to the @ref{69d,,signal.set_wakeup_fd()}
function makes it possible to specify whether Python prints a warning on
stderr when the wakeup buffer overflows.
(Contributed by Nathaniel J. Smith in bpo-30050@footnote{https://bugs.python.org/issue30050}.)

@node socket<5>,socketserver,signal<2>,Improved Modules<5>
@anchor{whatsnew/3 7 socket}@anchor{69e}
@subsubsection socket


The new @ref{69f,,socket.getblocking()} method
returns @code{True} if the socket is in blocking mode and @code{False} otherwise.
(Contributed by Yury Selivanov in bpo-32373@footnote{https://bugs.python.org/issue32373}.)

The new @ref{6a0,,socket.close()} function closes the passed socket file descriptor.
This function should be used instead of @ref{6a1,,os.close()} for better
compatibility across platforms.
(Contributed by Christian Heimes in bpo-32454@footnote{https://bugs.python.org/issue32454}.)

The @ref{ea,,socket} module now exposes the @code{socket.TCP_CONGESTION}
(Linux 2.6.13), @code{socket.TCP_USER_TIMEOUT} (Linux 2.6.37), and
@code{socket.TCP_NOTSENT_LOWAT} (Linux 3.12) constants.
(Contributed by Omar Sandoval in bpo-26273@footnote{https://bugs.python.org/issue26273} and
Nathaniel J. Smith in bpo-29728@footnote{https://bugs.python.org/issue29728}.)

Support for @ref{6a2,,socket.AF_VSOCK} sockets has been added to allow
communication between virtual machines and their hosts.
(Contributed by Cathy Avery in bpo-27584@footnote{https://bugs.python.org/issue27584}.)

Sockets now auto-detect family, type and protocol from file descriptor
by default.
(Contributed by Christian Heimes in bpo-28134@footnote{https://bugs.python.org/issue28134}.)

@node socketserver,sqlite3<3>,socket<5>,Improved Modules<5>
@anchor{whatsnew/3 7 socketserver}@anchor{6a3}
@subsubsection socketserver


@code{socketserver.ThreadingMixIn.server_close()} now waits until all non-daemon
threads complete. @code{socketserver.ForkingMixIn.server_close()} now waits
until all child processes complete.

Add a new @code{socketserver.ForkingMixIn.block_on_close} class attribute to
@ref{6a4,,socketserver.ForkingMixIn} and @ref{6a5,,socketserver.ThreadingMixIn}
classes. Set the class attribute to @code{False} to get the pre-3.7 behaviour.

@node sqlite3<3>,ssl<3>,socketserver,Improved Modules<5>
@anchor{whatsnew/3 7 sqlite3}@anchor{6a6}
@subsubsection sqlite3


@ref{18b,,sqlite3.Connection} now exposes the @ref{6a7,,backup()}
method when the underlying SQLite library is at version 3.6.11 or higher.
(Contributed by Lele Gaifax in bpo-27645@footnote{https://bugs.python.org/issue27645}.)

The @emph{database} argument of @ref{6a8,,sqlite3.connect()} now accepts any
@ref{63d,,path-like object}, instead of just a string.
(Contributed by Anders Lorentsen in bpo-31843@footnote{https://bugs.python.org/issue31843}.)

@node ssl<3>,string,sqlite3<3>,Improved Modules<5>
@anchor{whatsnew/3 7 ssl}@anchor{6a9}
@subsubsection ssl


The @ref{ee,,ssl} module now uses OpenSSL’s builtin API instead of
@ref{333,,match_hostname()} to check a host name or an IP address.  Values
are validated during TLS handshake.  Any certificate validation error
including failing the host name check now raises
@ref{6aa,,SSLCertVerificationError} and aborts the handshake with a proper
TLS Alert message.  The new exception contains additional information.
Host name validation can be customized with
@ref{6ab,,SSLContext.hostname_checks_common_name}.
(Contributed by Christian Heimes in bpo-31399@footnote{https://bugs.python.org/issue31399}.)

@cartouche
@quotation Note 
The improved host name check requires a @emph{libssl} implementation compatible
with OpenSSL 1.0.2 or 1.1.  Consequently, OpenSSL 0.9.8 and 1.0.1 are no
longer supported (see @ref{6ac,,Platform Support Removals} for more details).
The ssl module is mostly compatible with LibreSSL 2.7.2 and newer.
@end quotation
@end cartouche

The @code{ssl} module no longer sends IP addresses in SNI TLS extension.
(Contributed by Christian Heimes in bpo-32185@footnote{https://bugs.python.org/issue32185}.)

@ref{333,,match_hostname()} no longer supports partial wildcards like
@code{www*.example.org}.
(Contributed by Mandeep Singh in bpo-23033@footnote{https://bugs.python.org/issue23033} and Christian Heimes in
bpo-31399@footnote{https://bugs.python.org/issue31399}.)

The default cipher suite selection of the @code{ssl} module now uses a blacklist
approach rather than a hard-coded whitelist.  Python no longer re-enables
ciphers that have been blocked by OpenSSL security updates.  Default cipher
suite selection can be configured at compile time.
(Contributed by Christian Heimes in bpo-31429@footnote{https://bugs.python.org/issue31429}.)

Validation of server certificates containing internationalized domain names
(IDNs) is now supported.  As part of this change, the
@ref{6ad,,SSLSocket.server_hostname} attribute
now stores the expected hostname in A-label form (@code{"xn--pythn-mua.org"}),
rather than the U-label form (@code{"pythön.org"}).  (Contributed by
Nathaniel J. Smith and Christian Heimes in bpo-28414@footnote{https://bugs.python.org/issue28414}.)

The @code{ssl} module has preliminary and experimental support for TLS 1.3 and
OpenSSL 1.1.1.  At the time of Python 3.7.0 release, OpenSSL 1.1.1 is still
under development and TLS 1.3 hasn’t been finalized yet.  The TLS 1.3
handshake and protocol behaves slightly differently than TLS 1.2 and earlier,
see @ref{6ae,,TLS 1.3}.
(Contributed by Christian Heimes in bpo-32947@footnote{https://bugs.python.org/issue32947}, bpo-20995@footnote{https://bugs.python.org/issue20995},
bpo-29136@footnote{https://bugs.python.org/issue29136}, bpo-30622@footnote{https://bugs.python.org/issue30622} and bpo-33618@footnote{https://bugs.python.org/issue33618})

@ref{3d6,,SSLSocket} and @ref{6af,,SSLObject} no longer have a public
constructor.  Direct instantiation was never a documented and supported
feature.  Instances must be created with @ref{2bf,,SSLContext} methods
@ref{332,,wrap_socket()} and @ref{6b0,,wrap_bio()}.
(Contributed by Christian Heimes in bpo-32951@footnote{https://bugs.python.org/issue32951})

OpenSSL 1.1 APIs for setting the minimum and maximum TLS protocol version are
available as @ref{6b1,,SSLContext.minimum_version}
and @ref{6b2,,SSLContext.maximum_version}.
Supported protocols are indicated by several new flags, such as
@ref{6b3,,HAS_TLSv1_1}.
(Contributed by Christian Heimes in bpo-32609@footnote{https://bugs.python.org/issue32609}.)

@node string,subprocess,ssl<3>,Improved Modules<5>
@anchor{whatsnew/3 7 string}@anchor{6b4}
@subsubsection string


@ref{6b5,,string.Template} now lets you to optionally modify the regular
expression pattern for braced placeholders and non-braced placeholders
separately.  (Contributed by Barry Warsaw in bpo-1198569@footnote{https://bugs.python.org/issue1198569}.)

@node subprocess,sys<5>,string,Improved Modules<5>
@anchor{whatsnew/3 7 subprocess}@anchor{6b6}
@subsubsection subprocess


The @ref{6b7,,subprocess.run()} function accepts the new @emph{capture_output}
keyword argument.  When true, stdout and stderr will be captured.
This is equivalent to passing @ref{6b8,,subprocess.PIPE} as @emph{stdout} and
@emph{stderr} arguments.
(Contributed by Bo Bayles in bpo-32102@footnote{https://bugs.python.org/issue32102}.)

The @code{subprocess.run} function and the @ref{5ae,,subprocess.Popen} constructor
now accept the @emph{text} keyword argument as an alias
to @emph{universal_newlines}.
(Contributed by Andrew Clegg in bpo-31756@footnote{https://bugs.python.org/issue31756}.)

On Windows the default for @emph{close_fds} was changed from @code{False} to
@code{True} when redirecting the standard handles.  It’s now possible to set
@emph{close_fds} to true when redirecting the standard handles.  See
@ref{5ae,,subprocess.Popen}.  This means that @emph{close_fds} now defaults to
@code{True} on all supported platforms.
(Contributed by Segev Finer in bpo-19764@footnote{https://bugs.python.org/issue19764}.)

The subprocess module is now more graceful when handling
@ref{4ca,,KeyboardInterrupt} during @ref{6b9,,subprocess.call()},
@ref{6b7,,subprocess.run()}, or in a @ref{5ae,,Popen}
context manager.  It now waits a short amount of time for the child
to exit, before continuing the handling of the @code{KeyboardInterrupt}
exception.
(Contributed by Gregory P. Smith in bpo-25942@footnote{https://bugs.python.org/issue25942}.)

@node sys<5>,time<4>,subprocess,Improved Modules<5>
@anchor{whatsnew/3 7 sys}@anchor{6ba}
@subsubsection sys


The new @ref{5f4,,sys.breakpointhook()} hook function is called by the
built-in @ref{5db,,breakpoint()}.
(Contributed by Barry Warsaw in bpo-31353@footnote{https://bugs.python.org/issue31353}.)

On Android, the new @ref{6bb,,sys.getandroidapilevel()} returns the build-time
Android API version.
(Contributed by Victor Stinner in bpo-28740@footnote{https://bugs.python.org/issue28740}.)

The new @ref{6bc,,sys.get_coroutine_origin_tracking_depth()} function returns
the current coroutine origin tracking depth, as set by
the new @ref{6bd,,sys.set_coroutine_origin_tracking_depth()}.  @ref{9,,asyncio}
has been converted to use this new API instead of
the deprecated @code{sys.set_coroutine_wrapper()}.
(Contributed by Nathaniel J. Smith in bpo-32591@footnote{https://bugs.python.org/issue32591}.)

@node time<4>,tkinter<2>,sys<5>,Improved Modules<5>
@anchor{whatsnew/3 7 time}@anchor{6be}
@subsubsection time


@geindex Python Enhancement Proposals; PEP 564
PEP 564@footnote{https://www.python.org/dev/peps/pep-0564} adds six new functions with nanosecond resolution to the
@ref{10a,,time} module:


@itemize *

@item 
@ref{5fd,,time.clock_gettime_ns()}

@item 
@ref{5fe,,time.clock_settime_ns()}

@item 
@ref{5ff,,time.monotonic_ns()}

@item 
@ref{600,,time.perf_counter_ns()}

@item 
@ref{601,,time.process_time_ns()}

@item 
@ref{602,,time.time_ns()}
@end itemize

New clock identifiers have been added:


@itemize *

@item 
@ref{6bf,,time.CLOCK_BOOTTIME} (Linux): Identical to
@ref{193,,time.CLOCK_MONOTONIC}, except it also includes any time that the
system is suspended.

@item 
@ref{6c0,,time.CLOCK_PROF} (FreeBSD, NetBSD and OpenBSD): High-resolution
per-process CPU timer.

@item 
@ref{6c1,,time.CLOCK_UPTIME} (FreeBSD, OpenBSD): Time whose absolute value is
the time the system has been running and not suspended, providing accurate
uptime measurement.
@end itemize

The new @ref{42a,,time.thread_time()} and @ref{6c2,,time.thread_time_ns()} functions
can be used to get per-thread CPU time measurements.
(Contributed by Antoine Pitrou in bpo-32025@footnote{https://bugs.python.org/issue32025}.)

The new @ref{6c3,,time.pthread_getcpuclockid()} function returns the clock ID
of the thread-specific CPU-time clock.

@node tkinter<2>,tracemalloc<2>,time<4>,Improved Modules<5>
@anchor{whatsnew/3 7 tkinter}@anchor{6c4}
@subsubsection tkinter


The new @ref{6c5,,tkinter.ttk.Spinbox} class is now available.
(Contributed by Alan Moore in bpo-32585@footnote{https://bugs.python.org/issue32585}.)

@node tracemalloc<2>,types<2>,tkinter<2>,Improved Modules<5>
@anchor{whatsnew/3 7 tracemalloc}@anchor{6c6}
@subsubsection tracemalloc


@ref{6c7,,tracemalloc.Traceback} behaves more like regular tracebacks,
sorting the frames from oldest to most recent.
@ref{6c8,,Traceback.format()}
now accepts negative @emph{limit}, truncating the result to the
@code{abs(limit)} oldest frames.  To get the old behaviour, use
the new @emph{most_recent_first} argument to @code{Traceback.format()}.
(Contributed by Jesse Bakker in bpo-32121@footnote{https://bugs.python.org/issue32121}.)

@node types<2>,unicodedata<4>,tracemalloc<2>,Improved Modules<5>
@anchor{whatsnew/3 7 types}@anchor{6c9}
@subsubsection types


The new @ref{6ca,,WrapperDescriptorType},
@ref{177,,MethodWrapperType}, @ref{6cb,,MethodDescriptorType},
and @ref{6cc,,ClassMethodDescriptorType} classes are now available.
(Contributed by Manuel Krebber and Guido van Rossum in bpo-29377@footnote{https://bugs.python.org/issue29377},
and Serhiy Storchaka in bpo-32265@footnote{https://bugs.python.org/issue32265}.)

The new @ref{6cd,,types.resolve_bases()} function resolves MRO entries
dynamically as specified by 
@geindex Python Enhancement Proposals; PEP 560
PEP 560@footnote{https://www.python.org/dev/peps/pep-0560}.
(Contributed by Ivan Levkivskyi in bpo-32717@footnote{https://bugs.python.org/issue32717}.)

@node unicodedata<4>,unittest<3>,types<2>,Improved Modules<5>
@anchor{whatsnew/3 7 unicodedata}@anchor{6ce}
@subsubsection unicodedata


The internal @ref{121,,unicodedata} database has been upgraded to use Unicode 11@footnote{http://www.unicode.org/versions/Unicode11.0.0/}. (Contributed by Benjamin
Peterson.)

@node unittest<3>,unittest mock,unicodedata<4>,Improved Modules<5>
@anchor{whatsnew/3 7 unittest}@anchor{6cf}
@subsubsection unittest


The new @code{-k} command-line option allows filtering tests by a name
substring or a Unix shell-like pattern.
For example, @code{python -m unittest -k foo} runs
@code{foo_tests.SomeTest.test_something}, @code{bar_tests.SomeTest.test_foo},
but not @code{bar_tests.FooTest.test_something}.
(Contributed by Jonas Haag in bpo-32071@footnote{https://bugs.python.org/issue32071}.)

@node unittest mock,urllib parse<3>,unittest<3>,Improved Modules<5>
@anchor{whatsnew/3 7 unittest-mock}@anchor{6d0}
@subsubsection unittest.mock


The @ref{6d1,,sentinel} attributes now preserve their identity
when they are @ref{24,,copied} or @ref{c5,,pickled}. (Contributed by
Serhiy Storchaka in bpo-20804@footnote{https://bugs.python.org/issue20804}.)

The new @ref{6d2,,seal()} function allows sealing
@ref{561,,Mock} instances, which will disallow further creation
of attribute mocks.  The seal is applied recursively to all attributes that
are themselves mocks.
(Contributed by Mario Corchero in bpo-30541@footnote{https://bugs.python.org/issue30541}.)

@node urllib parse<3>,uu,unittest mock,Improved Modules<5>
@anchor{whatsnew/3 7 urllib-parse}@anchor{6d3}
@subsubsection urllib.parse


@ref{6d4,,urllib.parse.quote()} has been updated from 
@geindex RFC; RFC 2396
RFC 2396@footnote{https://tools.ietf.org/html/rfc2396.html} to 
@geindex RFC; RFC 3986
RFC 3986@footnote{https://tools.ietf.org/html/rfc3986.html},
adding @code{~} to the set of characters that are never quoted by default.
(Contributed by Christian Theune and Ratnadeep Debnath in bpo-16285@footnote{https://bugs.python.org/issue16285}.)

@node uu,uuid,urllib parse<3>,Improved Modules<5>
@anchor{whatsnew/3 7 uu}@anchor{6d5}
@subsubsection uu


The @ref{6d6,,uu.encode()} function now accepts an optional @emph{backtick}
keyword argument.  When it’s true, zeros are represented by @code{'`'}
instead of spaces.  (Contributed by Xiang Zhang in bpo-30103@footnote{https://bugs.python.org/issue30103}.)

@node uuid,warnings,uu,Improved Modules<5>
@anchor{whatsnew/3 7 uuid}@anchor{6d7}
@subsubsection uuid


The new @ref{6d8,,UUID.is_safe} attribute relays information
from the platform about whether generated UUIDs are generated with a
multiprocessing-safe method.
(Contributed by Barry Warsaw in bpo-22807@footnote{https://bugs.python.org/issue22807}.)

@ref{6d9,,uuid.getnode()} now prefers universally administered
MAC addresses over locally administered MAC addresses.
This makes a better guarantee for global uniqueness of UUIDs returned
from @ref{6da,,uuid.uuid1()}.  If only locally administered MAC addresses are
available, the first such one found is returned.
(Contributed by Barry Warsaw in bpo-32107@footnote{https://bugs.python.org/issue32107}.)

@node warnings,xml etree,uuid,Improved Modules<5>
@anchor{whatsnew/3 7 warnings}@anchor{6db}
@subsubsection warnings


The initialization of the default warnings filters has changed as follows:


@itemize *

@item 
warnings enabled via command line options (including those for @ref{6dc,,-b}
and the new CPython-specific @ref{159,,-X} @code{dev} option) are always passed
to the warnings machinery via the @ref{6dd,,sys.warnoptions} attribute.

@item 
warnings filters enabled via the command line or the environment now have the
following order of precedence:

@quotation


@itemize *

@item 
the @code{BytesWarning} filter for @ref{6dc,,-b} (or @code{-bb})

@item 
any filters specified with the @ref{3a8,,-W} option

@item 
any filters specified with the 
@geindex PYTHONWARNINGS
@geindex environment variable; PYTHONWARNINGS
@ref{6de,,PYTHONWARNINGS} environment
variable

@item 
any other CPython specific filters (e.g. the @code{default} filter added
for the new @code{-X dev} mode)

@item 
any implicit filters defined directly by the warnings machinery
@end itemize
@end quotation

@item 
in @ref{338,,CPython debug builds}, all warnings are now displayed
by default (the implicit filter list is empty)
@end itemize

(Contributed by Nick Coghlan and Victor Stinner in bpo-20361@footnote{https://bugs.python.org/issue20361},
bpo-32043@footnote{https://bugs.python.org/issue32043}, and bpo-32230@footnote{https://bugs.python.org/issue32230}.)

Deprecation warnings are once again shown by default in single-file scripts and
at the interactive prompt.  See @ref{5e5,,PEP 565; Show DeprecationWarning in __main__} for details.
(Contributed by Nick Coghlan in bpo-31975@footnote{https://bugs.python.org/issue31975}.)

@node xml etree,xmlrpc server,warnings,Improved Modules<5>
@anchor{whatsnew/3 7 xml-etree}@anchor{6df}
@subsubsection xml.etree


@ref{6e0,,ElementPath} predicates in the @code{find()}
methods can now compare text of the current node with @code{[. = "text"]},
not only text in children.  Predicates also allow adding spaces for
better readability.  (Contributed by Stefan Behnel in bpo-31648@footnote{https://bugs.python.org/issue31648}.)

@node xmlrpc server,zipapp,xml etree,Improved Modules<5>
@anchor{whatsnew/3 7 xmlrpc-server}@anchor{6e1}
@subsubsection xmlrpc.server


@code{SimpleXMLRPCDispatcher.register_function}
can now be used as a decorator.  (Contributed by Xiang Zhang in
bpo-7769@footnote{https://bugs.python.org/issue7769}.)

@node zipapp,zipfile,xmlrpc server,Improved Modules<5>
@anchor{whatsnew/3 7 zipapp}@anchor{6e2}
@subsubsection zipapp


Function @ref{6e3,,create_archive()} now accepts an optional @emph{filter}
argument to allow the user to select which files should be included in the
archive.  (Contributed by Irmen de Jong in bpo-31072@footnote{https://bugs.python.org/issue31072}.)

Function @ref{6e3,,create_archive()} now accepts an optional @emph{compressed}
argument to generate a compressed archive.  A command line option
@code{--compress} has also been added to support compression.
(Contributed by Zhiming Wang in bpo-31638@footnote{https://bugs.python.org/issue31638}.)

@node zipfile,,zipapp,Improved Modules<5>
@anchor{whatsnew/3 7 zipfile}@anchor{6e4}
@subsubsection zipfile


@ref{6e5,,ZipFile} now accepts the new @emph{compresslevel} parameter to
control the compression level.
(Contributed by Bo Bayles in bpo-21417@footnote{https://bugs.python.org/issue21417}.)

Subdirectories in archives created by @code{ZipFile} are now stored in
alphabetical order.
(Contributed by Bernhard M. Wiedemann in bpo-30693@footnote{https://bugs.python.org/issue30693}.)

@node C API Changes<4>,Build Changes<4>,Improved Modules<5>,What’s New In Python 3 7
@anchor{whatsnew/3 7 c-api-changes}@anchor{6e6}
@subsection C API Changes


A new API for thread-local storage has been implemented.  See
@ref{5e6,,PEP 539; New C API for Thread-Local Storage} for an overview and
@ref{5f8,,Thread Specific Storage (TSS) API} for a complete reference.
(Contributed by Masayuki Yamamoto in bpo-25658@footnote{https://bugs.python.org/issue25658}.)

The new @ref{5d7,,context variables} functionality
exposes a number of @ref{615,,new C APIs}.

The new @ref{6e7,,PyImport_GetModule()} function returns the previously
imported module with the given name.
(Contributed by Eric Snow in bpo-28411@footnote{https://bugs.python.org/issue28411}.)

The new @code{Py_RETURN_RICHCOMPARE} macro eases writing rich
comparison functions.
(Contributed by Petr Victorin in bpo-23699@footnote{https://bugs.python.org/issue23699}.)

The new @ref{6e8,,Py_UNREACHABLE} macro can be used to mark unreachable
code paths.
(Contributed by Barry Warsaw in bpo-31338@footnote{https://bugs.python.org/issue31338}.)

The @ref{11b,,tracemalloc} now exposes a C API through the new
@ref{6e9,,PyTraceMalloc_Track()} and @ref{6ea,,PyTraceMalloc_Untrack()}
functions.
(Contributed by Victor Stinner in bpo-30054@footnote{https://bugs.python.org/issue30054}.)

The new @code{import__find__load__start()} and
@code{import__find__load__done()} static markers can be used to trace
module imports.
(Contributed by Christian Heimes in bpo-31574@footnote{https://bugs.python.org/issue31574}.)

The fields @code{name} and @code{doc} of structures
@ref{6eb,,PyMemberDef}, @ref{6ec,,PyGetSetDef},
@ref{6ed,,PyStructSequence_Field}, @ref{6ee,,PyStructSequence_Desc},
and @code{wrapperbase} are now of type @code{const char *} rather of
@code{char *}.  (Contributed by Serhiy Storchaka in bpo-28761@footnote{https://bugs.python.org/issue28761}.)

The result of @ref{367,,PyUnicode_AsUTF8AndSize()} and @ref{6ef,,PyUnicode_AsUTF8()}
is now of type @code{const char *} rather of @code{char *}. (Contributed by Serhiy
Storchaka in bpo-28769@footnote{https://bugs.python.org/issue28769}.)

The result of @ref{6f0,,PyMapping_Keys()}, @ref{6f1,,PyMapping_Values()} and
@ref{6f2,,PyMapping_Items()} is now always a list, rather than a list or a
tuple. (Contributed by Oren Milman in bpo-28280@footnote{https://bugs.python.org/issue28280}.)

Added functions @ref{6f3,,PySlice_Unpack()} and @ref{6f4,,PySlice_AdjustIndices()}.
(Contributed by Serhiy Storchaka in bpo-27867@footnote{https://bugs.python.org/issue27867}.)

@ref{6f5,,PyOS_AfterFork()} is deprecated in favour of the new functions
@ref{6f6,,PyOS_BeforeFork()}, @ref{6f7,,PyOS_AfterFork_Parent()} and
@ref{5c1,,PyOS_AfterFork_Child()}.  (Contributed by Antoine Pitrou in
bpo-16500@footnote{https://bugs.python.org/issue16500}.)

The @code{PyExc_RecursionErrorInst} singleton that was part of the public API
has been removed as its members being never cleared may cause a segfault
during finalization of the interpreter. Contributed by Xavier de Gaye in
bpo-22898@footnote{https://bugs.python.org/issue22898} and bpo-30697@footnote{https://bugs.python.org/issue30697}.

Added C API support for timezones with timezone constructors
@ref{6f8,,PyTimeZone_FromOffset()} and @ref{6f9,,PyTimeZone_FromOffsetAndName()},
and access to the UTC singleton with @ref{6fa,,PyDateTime_TimeZone_UTC}.
Contributed by Paul Ganssle in bpo-10381@footnote{https://bugs.python.org/issue10381}.

The type of results of @code{PyThread_start_new_thread()} and
@code{PyThread_get_thread_ident()}, and the @emph{id} parameter of
@ref{6fb,,PyThreadState_SetAsyncExc()} changed from long to
unsigned  long.
(Contributed by Serhiy Storchaka in bpo-6532@footnote{https://bugs.python.org/issue6532}.)

@ref{39f,,PyUnicode_AsWideCharString()} now raises a @ref{17c,,ValueError} if the
second argument is @code{NULL} and the wchar_t* string contains null
characters.  (Contributed by Serhiy Storchaka in bpo-30708@footnote{https://bugs.python.org/issue30708}.)

Changes to the startup sequence and the management of dynamic memory
allocators mean that the long documented requirement to call
@ref{38d,,Py_Initialize()} before calling most C API functions is now
relied on more heavily, and failing to abide by it may lead to segfaults in
embedding applications. See the @ref{5e9,,Porting to Python 3.7} section in this
document and the @ref{6fc,,Before Python Initialization} section in the C API documentation
for more details.

The new @ref{6fd,,PyInterpreterState_GetID()} returns the unique ID for a
given interpreter.
(Contributed by Eric Snow in bpo-29102@footnote{https://bugs.python.org/issue29102}.)

@ref{6fe,,Py_DecodeLocale()}, @ref{6ff,,Py_EncodeLocale()} now use the UTF-8
encoding when the @ref{5e2,,UTF-8 mode} is enabled.
(Contributed by Victor Stinner in bpo-29240@footnote{https://bugs.python.org/issue29240}.)

@ref{700,,PyUnicode_DecodeLocaleAndSize()} and @ref{701,,PyUnicode_EncodeLocale()}
now use the current locale encoding for @code{surrogateescape} error handler.
(Contributed by Victor Stinner in bpo-29240@footnote{https://bugs.python.org/issue29240}.)

The @emph{start} and @emph{end} parameters of @ref{39c,,PyUnicode_FindChar()} are
now adjusted to behave like string slices.
(Contributed by Xiang Zhang in bpo-28822@footnote{https://bugs.python.org/issue28822}.)

@node Build Changes<4>,Optimizations<5>,C API Changes<4>,What’s New In Python 3 7
@anchor{whatsnew/3 7 build-changes}@anchor{702}
@subsection Build Changes


Support for building @code{--without-threads} has been removed.  The
@ref{109,,threading} module is now always available.
(Contributed by Antoine Pitrou in bpo-31370@footnote{https://bugs.python.org/issue31370}.).

A full copy of libffi is no longer bundled for use when building the
@ref{29,,_ctypes} module on non-OSX UNIX platforms.  An installed copy
of libffi is now required when building @code{_ctypes} on such platforms.
(Contributed by Zachary Ware in bpo-27979@footnote{https://bugs.python.org/issue27979}.)

The Windows build process no longer depends on Subversion to pull in external
sources, a Python script is used to download zipfiles from GitHub instead.
If Python 3.6 is not found on the system (via @code{py -3.6}), NuGet is used to
download a copy of 32-bit Python for this purpose.  (Contributed by Zachary
Ware in bpo-30450@footnote{https://bugs.python.org/issue30450}.)

The @ref{ee,,ssl} module requires OpenSSL 1.0.2 or 1.1 compatible libssl.
OpenSSL 1.0.1 has reached end of lifetime on 2016-12-31 and is no longer
supported. LibreSSL is temporarily not supported as well. LibreSSL releases
up to version 2.6.4 are missing required OpenSSL 1.0.2 APIs.

@node Optimizations<5>,Other CPython Implementation Changes<2>,Build Changes<4>,What’s New In Python 3 7
@anchor{whatsnew/3 7 optimizations}@anchor{703}@anchor{whatsnew/3 7 whatsnew37-perf}@anchor{5e8}
@subsection Optimizations


The overhead of calling many methods of various standard library classes
implemented in C has been significantly reduced by porting more code
to use the @code{METH_FASTCALL} convention.
(Contributed by Victor Stinner in bpo-29300@footnote{https://bugs.python.org/issue29300}, bpo-29507@footnote{https://bugs.python.org/issue29507},
bpo-29452@footnote{https://bugs.python.org/issue29452}, and bpo-29286@footnote{https://bugs.python.org/issue29286}.)

Various optimizations have reduced Python startup time by 10% on Linux and
up to 30% on macOS.
(Contributed by Victor Stinner, INADA Naoki in bpo-29585@footnote{https://bugs.python.org/issue29585}, and
Ivan Levkivskyi in bpo-31333@footnote{https://bugs.python.org/issue31333}.)

Method calls are now up to 20% faster due to the bytecode changes which
avoid creating bound method instances.
(Contributed by Yury Selivanov and INADA Naoki in bpo-26110@footnote{https://bugs.python.org/issue26110}.)
@anchor{whatsnew/3 7 whatsnew37-asyncio-perf}@anchor{620}
The @ref{9,,asyncio} module received a number of notable optimizations for
commonly used functions:


@itemize *

@item 
The @ref{314,,asyncio.get_event_loop()} function has been reimplemented in C to
make it up to 15 times faster.
(Contributed by Yury Selivanov in bpo-32296@footnote{https://bugs.python.org/issue32296}.)

@item 
@ref{316,,asyncio.Future} callback management has been optimized.
(Contributed by Yury Selivanov in bpo-32348@footnote{https://bugs.python.org/issue32348}.)

@item 
@ref{31a,,asyncio.gather()} is now up to 15% faster.
(Contributed by Yury Selivanov in bpo-32355@footnote{https://bugs.python.org/issue32355}.)

@item 
@ref{58a,,asyncio.sleep()} is now up to 2 times faster when the @emph{delay}
argument is zero or negative.
(Contributed by Andrew Svetlov in bpo-32351@footnote{https://bugs.python.org/issue32351}.)

@item 
The performance overhead of asyncio debug mode has been reduced.
(Contributed by Antoine Pitrou in bpo-31970@footnote{https://bugs.python.org/issue31970}.)
@end itemize

As a result of @ref{5dd,,PEP 560 work}, the import time
of @ref{120,,typing} has been reduced by a factor of 7, and many typing operations
are now faster.
(Contributed by Ivan Levkivskyi in bpo-32226@footnote{https://bugs.python.org/issue32226}.)

@ref{704,,sorted()} and @ref{705,,list.sort()} have been optimized for common cases
to be up to 40-75% faster.
(Contributed by Elliot Gorokhovsky in bpo-28685@footnote{https://bugs.python.org/issue28685}.)

@ref{706,,dict.copy()} is now up to 5.5 times faster.
(Contributed by Yury Selivanov in bpo-31179@footnote{https://bugs.python.org/issue31179}.)

@ref{707,,hasattr()} and @ref{708,,getattr()} are now about 4 times faster when
@emph{name} is not found and @emph{obj} does not override @ref{709,,object.__getattr__()}
or @ref{70a,,object.__getattribute__()}.
(Contributed by INADA Naoki in bpo-32544@footnote{https://bugs.python.org/issue32544}.)

Searching for certain Unicode characters (like Ukrainian capital “Є”)
in a string was up to 25 times slower than searching for other characters.
It is now only 3 times slower in the worst case.
(Contributed by Serhiy Storchaka in bpo-24821@footnote{https://bugs.python.org/issue24821}.)

The @ref{3e4,,collections.namedtuple()} factory has been reimplemented to
make the creation of named tuples 4 to 6 times faster.
(Contributed by Jelle Zijlstra with further improvements by INADA Naoki,
Serhiy Storchaka, and Raymond Hettinger in bpo-28638@footnote{https://bugs.python.org/issue28638}.)

@code{date.fromordinal()} and @code{date.fromtimestamp()} are now up to
30% faster in the common case.
(Contributed by Paul Ganssle in bpo-32403@footnote{https://bugs.python.org/issue32403}.)

The @ref{683,,os.fwalk()} function is now up to 2 times faster thanks to
the use of @ref{576,,os.scandir()}.
(Contributed by Serhiy Storchaka in bpo-25996@footnote{https://bugs.python.org/issue25996}.)

The speed of the @ref{53a,,shutil.rmtree()} function has been improved by
20–40% thanks to the use of the @ref{576,,os.scandir()} function.
(Contributed by Serhiy Storchaka in bpo-28564@footnote{https://bugs.python.org/issue28564}.)

Optimized case-insensitive matching and searching of @ref{d8,,regular expressions}.  Searching some patterns can now be up to 20 times faster.
(Contributed by Serhiy Storchaka in bpo-30285@footnote{https://bugs.python.org/issue30285}.)

@ref{70b,,re.compile()} now converts @code{flags} parameter to int object if
it is @code{RegexFlag}.  It is now as fast as Python 3.5, and faster than
Python 3.6 by about 10% depending on the pattern.
(Contributed by INADA Naoki in bpo-31671@footnote{https://bugs.python.org/issue31671}.)

The @ref{70c,,modify()} methods of classes
@ref{70d,,selectors.EpollSelector}, @ref{70e,,selectors.PollSelector}
and @ref{70f,,selectors.DevpollSelector} may be around 10% faster under
heavy loads.  (Contributed by Giampaolo Rodola’ in bpo-30014@footnote{https://bugs.python.org/issue30014})

Constant folding has been moved from the peephole optimizer to the new AST
optimizer, which is able perform optimizations more consistently.
(Contributed by Eugene Toder and INADA Naoki in bpo-29469@footnote{https://bugs.python.org/issue29469} and
bpo-11549@footnote{https://bugs.python.org/issue11549}.)

Most functions and methods in @ref{3,,abc} have been rewritten in C.
This makes creation of abstract base classes, and calling @ref{236,,isinstance()}
and @ref{237,,issubclass()} on them 1.5x faster.  This also reduces Python
start-up time by up to 10%. (Contributed by Ivan Levkivskyi and INADA Naoki
in bpo-31333@footnote{https://bugs.python.org/issue31333})

Significant speed improvements to alternate constructors for
@ref{3e2,,datetime.date} and @ref{363,,datetime.datetime} by using fast-path
constructors when not constructing subclasses. (Contributed by Paul Ganssle
in bpo-32403@footnote{https://bugs.python.org/issue32403})

The speed of comparison of @ref{25e,,array.array} instances has been
improved considerably in certain cases.  It is now from 10x to 70x faster
when comparing arrays holding values of the same integer type.
(Contributed by Adrian Wielgosik in bpo-24700@footnote{https://bugs.python.org/issue24700}.)

The @ref{710,,math.erf()} and @ref{711,,math.erfc()} functions now use the (faster)
C library implementation on most platforms.
(Contributed by Serhiy Storchaka in bpo-26121@footnote{https://bugs.python.org/issue26121}.)

@node Other CPython Implementation Changes<2>,Deprecated Python Behavior,Optimizations<5>,What’s New In Python 3 7
@anchor{whatsnew/3 7 other-cpython-implementation-changes}@anchor{712}
@subsection Other CPython Implementation Changes



@itemize *

@item 
Trace hooks may now opt out of receiving the @code{line} and opt into receiving
the @code{opcode} events from the interpreter by setting the corresponding new
@code{f_trace_lines} and @code{f_trace_opcodes} attributes on the
frame being traced. (Contributed by Nick Coghlan in bpo-31344@footnote{https://bugs.python.org/issue31344}.)

@item 
Fixed some consistency problems with namespace package module attributes.
Namespace module objects now have an @code{__file__} that is set to @code{None}
(previously unset), and their @code{__spec__.origin} is also set to @code{None}
(previously the string @code{"namespace"}).  See bpo-32305@footnote{https://bugs.python.org/issue32305}.  Also, the
namespace module object’s @code{__spec__.loader} is set to the same value as
@code{__loader__} (previously, the former was set to @code{None}).  See
bpo-32303@footnote{https://bugs.python.org/issue32303}.

@item 
The @ref{713,,locals()} dictionary now displays in the lexical order that
variables were defined.  Previously, the order was undefined.
(Contributed by Raymond Hettinger in bpo-32690@footnote{https://bugs.python.org/issue32690}.)

@item 
The @ref{37,,distutils} @code{upload} command no longer tries to change CR
end-of-line characters to CRLF.  This fixes a corruption issue with sdists
that ended with a byte equivalent to CR.
(Contributed by Bo Bayles in bpo-32304@footnote{https://bugs.python.org/issue32304}.)
@end itemize

@node Deprecated Python Behavior,Deprecated Python modules functions and methods,Other CPython Implementation Changes<2>,What’s New In Python 3 7
@anchor{whatsnew/3 7 deprecated-python-behavior}@anchor{714}
@subsection Deprecated Python Behavior


Yield expressions (both @code{yield} and @code{yield from} clauses) are now deprecated
in comprehensions and generator expressions (aside from the iterable expression
in the leftmost @code{for} clause). This ensures that comprehensions
always immediately return a container of the appropriate type (rather than
potentially returning a @ref{715,,generator iterator} object), while generator
expressions won’t attempt to interleave their implicit output with the output
from any explicit yield expressions.  In Python 3.7, such expressions emit
@ref{1b2,,DeprecationWarning} when compiled, in Python 3.8 this will be a
@ref{21a,,SyntaxError}.
(Contributed by Serhiy Storchaka in bpo-10544@footnote{https://bugs.python.org/issue10544}.)

Returning a subclass of @ref{33b,,complex} from @ref{4c4,,object.__complex__()} is
deprecated and will be an error in future Python versions.  This makes
@code{__complex__()} consistent with @ref{249,,object.__int__()} and
@ref{4c3,,object.__float__()}.
(Contributed by Serhiy Storchaka in bpo-28894@footnote{https://bugs.python.org/issue28894}.)

@node Deprecated Python modules functions and methods,Deprecated functions and types of the C API,Deprecated Python Behavior,What’s New In Python 3 7
@anchor{whatsnew/3 7 deprecated-python-modules-functions-and-methods}@anchor{716}
@subsection Deprecated Python modules, functions and methods


@menu
* aifc:: 
* asyncio: asyncio<5>. 
* collections: collections<3>. 
* dbm: dbm<2>. 
* enum: enum<3>. 
* gettext: gettext<2>. 
* importlib: importlib<3>. 
* locale: locale<2>. 
* macpath:: 
* threading: threading<4>. 
* socket: socket<6>. 
* ssl: ssl<4>. 
* sunau:: 
* sys: sys<6>. 
* wave:: 

@end menu

@node aifc,asyncio<5>,,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 aifc}@anchor{717}
@subsubsection aifc


@code{aifc.openfp()} has been deprecated and will be removed in Python 3.9.
Use @ref{718,,aifc.open()} instead.
(Contributed by Brian Curtin in bpo-31985@footnote{https://bugs.python.org/issue31985}.)

@node asyncio<5>,collections<3>,aifc,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 id2}@anchor{719}@anchor{whatsnew/3 7 whatsnew37-asyncio-deprecated}@anchor{63e}
@subsubsection asyncio


Support for directly @code{await}-ing instances of @ref{58b,,asyncio.Lock} and
other asyncio synchronization primitives has been deprecated.  An
asynchronous context manager must be used in order to acquire and release
the synchronization resource.
(Contributed by Andrew Svetlov in bpo-32253@footnote{https://bugs.python.org/issue32253}.)

The @code{asyncio.Task.current_task()} and @code{asyncio.Task.all_tasks()}
methods have been deprecated.
(Contributed by Andrew Svetlov in bpo-32250@footnote{https://bugs.python.org/issue32250}.)

@node collections<3>,dbm<2>,asyncio<5>,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 id3}@anchor{71a}
@subsubsection collections


In Python 3.8, the abstract base classes in @ref{1d,,collections.abc} will no
longer be exposed in the regular @ref{1c,,collections} module.  This will help
create a clearer distinction between the concrete classes and the abstract
base classes.
(Contributed by Serhiy Storchaka in bpo-25988@footnote{https://bugs.python.org/issue25988}.)

@node dbm<2>,enum<3>,collections<3>,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 id4}@anchor{71b}
@subsubsection dbm


@ref{31,,dbm.dumb} now supports reading read-only files and no longer writes the
index file when it is not changed.  A deprecation warning is now emitted
if the index file is missing and recreated in the @code{'r'} and @code{'w'}
modes (this will be an error in future Python releases).
(Contributed by Serhiy Storchaka in bpo-28847@footnote{https://bugs.python.org/issue28847}.)

@node enum<3>,gettext<2>,dbm<2>,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 id5}@anchor{71c}
@subsubsection enum


In Python 3.8, attempting to check for non-Enum objects in @code{Enum}
classes will raise a @ref{162,,TypeError} (e.g. @code{1 in Color}); similarly,
attempting to check for non-Flag objects in a @code{Flag} member will
raise @ref{162,,TypeError} (e.g. @code{1 in Perm.RW}); currently, both operations
return @ref{659,,False} instead.
(Contributed by Ethan Furman in bpo-33217@footnote{https://bugs.python.org/issue33217}.)

@node gettext<2>,importlib<3>,enum<3>,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 gettext}@anchor{71d}
@subsubsection gettext


Using non-integer value for selecting a plural form in @ref{83,,gettext} is
now deprecated.  It never correctly worked. (Contributed by Serhiy Storchaka
in bpo-28692@footnote{https://bugs.python.org/issue28692}.)

@node importlib<3>,locale<2>,gettext<2>,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 id6}@anchor{71e}
@subsubsection importlib


Methods
@ref{306,,MetaPathFinder.find_module()}
(replaced by
@ref{308,,MetaPathFinder.find_spec()})
and
@ref{2ff,,PathEntryFinder.find_loader()}
(replaced by
@ref{309,,PathEntryFinder.find_spec()})
both deprecated in Python 3.4 now emit @ref{1b2,,DeprecationWarning}.
(Contributed by Matthias Bussonnier in bpo-29576@footnote{https://bugs.python.org/issue29576})

The @ref{71f,,importlib.abc.ResourceLoader} ABC has been deprecated in
favour of @ref{61b,,importlib.abc.ResourceReader}.

@node locale<2>,macpath,importlib<3>,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 id7}@anchor{720}
@subsubsection locale


@ref{721,,locale.format()} has been deprecated, use @ref{674,,locale.format_string()}
instead.  (Contributed by Garvit in bpo-10379@footnote{https://bugs.python.org/issue10379}.)

@node macpath,threading<4>,locale<2>,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 macpath}@anchor{722}
@subsubsection macpath


The @code{macpath} is now deprecated and will be removed in Python 3.8.
(Contributed by Chi Hsuan Yen in bpo-9850@footnote{https://bugs.python.org/issue9850}.)

@node threading<4>,socket<6>,macpath,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 threading}@anchor{723}
@subsubsection threading


@code{dummy_threading} and @code{_dummy_thread} have been deprecated.  It is
no longer possible to build Python with threading disabled.
Use @ref{109,,threading} instead.
(Contributed by Antoine Pitrou in bpo-31370@footnote{https://bugs.python.org/issue31370}.)

@node socket<6>,ssl<4>,threading<4>,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 id8}@anchor{724}
@subsubsection socket


The silent argument value truncation in @ref{345,,socket.htons()} and
@ref{346,,socket.ntohs()} has been deprecated.  In future versions of Python,
if the passed argument is larger than 16 bits, an exception will be raised.
(Contributed by Oren Milman in bpo-28332@footnote{https://bugs.python.org/issue28332}.)

@node ssl<4>,sunau,socket<6>,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 id9}@anchor{725}
@subsubsection ssl


@ref{331,,ssl.wrap_socket()} is deprecated.  Use
@ref{332,,ssl.SSLContext.wrap_socket()} instead.
(Contributed by Christian Heimes in bpo-28124@footnote{https://bugs.python.org/issue28124}.)

@node sunau,sys<6>,ssl<4>,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 sunau}@anchor{726}
@subsubsection sunau


@code{sunau.openfp()} has been deprecated and will be removed in Python 3.9.
Use @ref{727,,sunau.open()} instead.
(Contributed by Brian Curtin in bpo-31985@footnote{https://bugs.python.org/issue31985}.)

@node sys<6>,wave,sunau,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 id10}@anchor{728}
@subsubsection sys


Deprecated @code{sys.set_coroutine_wrapper()} and
@code{sys.get_coroutine_wrapper()}.

The undocumented @code{sys.callstats()} function has been deprecated and
will be removed in a future Python version.
(Contributed by Victor Stinner in bpo-28799@footnote{https://bugs.python.org/issue28799}.)

@node wave,,sys<6>,Deprecated Python modules functions and methods
@anchor{whatsnew/3 7 wave}@anchor{729}
@subsubsection wave


@code{wave.openfp()} has been deprecated and will be removed in Python 3.9.
Use @ref{44a,,wave.open()} instead.
(Contributed by Brian Curtin in bpo-31985@footnote{https://bugs.python.org/issue31985}.)

@node Deprecated functions and types of the C API,Platform Support Removals,Deprecated Python modules functions and methods,What’s New In Python 3 7
@anchor{whatsnew/3 7 deprecated-functions-and-types-of-the-c-api}@anchor{72a}
@subsection Deprecated functions and types of the C API


Function @ref{72b,,PySlice_GetIndicesEx()} is deprecated and replaced with
a macro if @code{Py_LIMITED_API} is not set or set to a value in the range
between @code{0x03050400} and @code{0x03060000} (not inclusive), or is @code{0x03060100}
or higher.  (Contributed by Serhiy Storchaka in bpo-27867@footnote{https://bugs.python.org/issue27867}.)

@ref{6f5,,PyOS_AfterFork()} has been deprecated.  Use @ref{6f6,,PyOS_BeforeFork()},
@ref{6f7,,PyOS_AfterFork_Parent()} or @ref{5c1,,PyOS_AfterFork_Child()} instead.
(Contributed by Antoine Pitrou in bpo-16500@footnote{https://bugs.python.org/issue16500}.)

@node Platform Support Removals,API and Feature Removals<2>,Deprecated functions and types of the C API,What’s New In Python 3 7
@anchor{whatsnew/3 7 id11}@anchor{72c}@anchor{whatsnew/3 7 platform-support-removals}@anchor{6ac}
@subsection Platform Support Removals



@itemize *

@item 
FreeBSD 9 and older are no longer officially supported.

@item 
For full Unicode support, including within extension modules, *nix platforms
are now expected to provide at least one of @code{C.UTF-8} (full locale),
@code{C.utf8} (full locale) or @code{UTF-8} (@code{LC_CTYPE}-only locale) as an
alternative to the legacy @code{ASCII}-based @code{C} locale.

@item 
OpenSSL 0.9.8 and 1.0.1 are no longer supported, which means building CPython
3.7 with SSL/TLS support on older platforms still using these versions
requires custom build options that link to a more recent version of OpenSSL.

Notably, this issue affects the Debian 8 (aka “jessie”) and Ubuntu 14.04
(aka “Trusty”) LTS Linux distributions, as they still use OpenSSL 1.0.1 by
default.

Debian 9 (“stretch”) and Ubuntu 16.04 (“xenial”), as well as recent releases
of other LTS Linux releases (e.g. RHEL/CentOS 7.5, SLES 12-SP3), use OpenSSL
1.0.2 or later, and remain supported in the default build configuration.

CPython’s own CI configuration file@footnote{https://github.com/python/cpython/tree/main/.travis.yml} provides an
example of using the SSL
compatibility testing infrastructure@footnote{https://github.com/python/cpython/tree/main/Tools/ssl/multissltests.py} in
CPython’s test suite to build and link against OpenSSL 1.1.0 rather than an
outdated system provided OpenSSL.
@end itemize

@node API and Feature Removals<2>,Module Removals,Platform Support Removals,What’s New In Python 3 7
@anchor{whatsnew/3 7 api-and-feature-removals}@anchor{72d}
@subsection API and Feature Removals


The following features and APIs have been removed from Python 3.7:


@itemize *

@item 
The @code{os.stat_float_times()} function has been removed. It was introduced in
Python 2.3 for backward compatibility with Python 2.2, and was deprecated
since Python 3.1.

@item 
Unknown escapes consisting of @code{'\'} and an ASCII letter in replacement
templates for @ref{72e,,re.sub()} were deprecated in Python 3.5, and will now
cause an error.

@item 
Removed support of the @emph{exclude} argument in @ref{72f,,tarfile.TarFile.add()}.
It was deprecated in Python 2.7 and 3.2.  Use the @emph{filter} argument instead.

@item 
The @code{splitunc()} function in the @code{ntpath} module was deprecated in
Python 3.1, and has now been removed.  Use the @ref{730,,splitdrive()}
function instead.

@item 
@ref{3e4,,collections.namedtuple()} no longer supports the @emph{verbose} parameter
or @code{_source} attribute which showed the generated source code for the
named tuple class.  This was part of an optimization designed to speed-up
class creation.  (Contributed by Jelle Zijlstra with further improvements
by INADA Naoki, Serhiy Storchaka, and Raymond Hettinger in bpo-28638@footnote{https://bugs.python.org/issue28638}.)

@item 
Functions @ref{2f7,,bool()}, @ref{256,,float()}, @ref{43a,,list()} and @ref{3e5,,tuple()} no
longer take keyword arguments.  The first argument of @ref{1b1,,int()} can now
be passed only as positional argument.

@item 
Removed previously deprecated in Python 2.4 classes @code{Plist}, @code{Dict} and
@code{_InternalDict} in the @ref{ca,,plistlib} module.  Dict values in the result
of functions @code{readPlist()} and
@code{readPlistFromBytes()} are now normal dicts.  You no longer
can use attribute access to access items of these dictionaries.

@item 
The @code{asyncio.windows_utils.socketpair()} function has been
removed.  Use the @ref{731,,socket.socketpair()} function instead,
it is available on all platforms since Python 3.5.
@code{asyncio.windows_utils.socketpair} was just an alias to
@code{socket.socketpair} on Python 3.5 and newer.

@item 
@ref{9,,asyncio} no longer exports the @ref{e1,,selectors} and
@code{_overlapped} modules as @code{asyncio.selectors} and
@code{asyncio._overlapped}. Replace @code{from asyncio import selectors} with
@code{import selectors}.

@item 
Direct instantiation of @ref{3d6,,ssl.SSLSocket} and @ref{6af,,ssl.SSLObject}
objects is now prohibited. The constructors were never documented, tested,
or designed as public constructors.  Users were supposed to use
@ref{331,,ssl.wrap_socket()} or @ref{2bf,,ssl.SSLContext}.
(Contributed by Christian Heimes in bpo-32951@footnote{https://bugs.python.org/issue32951}.)

@item 
The unused @ref{37,,distutils} @code{install_misc} command has been removed.
(Contributed by Eric N. Vander Weele in bpo-29218@footnote{https://bugs.python.org/issue29218}.)
@end itemize

@node Module Removals,Windows-only Changes,API and Feature Removals<2>,What’s New In Python 3 7
@anchor{whatsnew/3 7 module-removals}@anchor{732}
@subsection Module Removals


The @code{fpectl} module has been removed.  It was never enabled by
default, never worked correctly on x86-64, and it changed the Python
ABI in ways that caused unexpected breakage of C extensions.
(Contributed by Nathaniel J. Smith in bpo-29137@footnote{https://bugs.python.org/issue29137}.)

@node Windows-only Changes,Porting to Python 3 7,Module Removals,What’s New In Python 3 7
@anchor{whatsnew/3 7 windows-only-changes}@anchor{733}
@subsection Windows-only Changes


The python launcher, (py.exe), can accept 32 & 64 bit specifiers @strong{without}
having to specify a minor version as well. So @code{py -3-32} and @code{py -3-64}
become valid as well as @code{py -3.7-32}, also the -@emph{m}-64 and -@emph{m.n}-64 forms
are now accepted to force 64 bit python even if 32 bit would have otherwise
been used. If the specified version is not available py.exe will error exit.
(Contributed by Steve Barnes in bpo-30291@footnote{https://bugs.python.org/issue30291}.)

The launcher can be run as @code{py -0} to produce a list of the installed pythons,
@emph{with default marked with an asterisk}. Running @code{py -0p} will include the paths.
If py is run with a version specifier that cannot be matched it will also print
the @emph{short form} list of available specifiers.
(Contributed by Steve Barnes in bpo-30362@footnote{https://bugs.python.org/issue30362}.)

@node Porting to Python 3 7,Notable changes in Python 3 7 1,Windows-only Changes,What’s New In Python 3 7
@anchor{whatsnew/3 7 porting-to-python-3-7}@anchor{734}@anchor{whatsnew/3 7 porting-to-python-37}@anchor{5e9}
@subsection Porting to Python 3.7


This section lists previously described changes and other bugfixes
that may require changes to your code.

@menu
* Changes in Python Behavior:: 
* Changes in the Python API: Changes in the Python API<5>. 
* Changes in the C API: Changes in the C API<4>. 
* CPython bytecode changes: CPython bytecode changes<5>. 
* Windows-only Changes: Windows-only Changes<2>. 
* Other CPython implementation changes:: 

@end menu

@node Changes in Python Behavior,Changes in the Python API<5>,,Porting to Python 3 7
@anchor{whatsnew/3 7 changes-in-python-behavior}@anchor{735}
@subsubsection Changes in Python Behavior



@itemize *

@item 
@ref{4d4,,async} and @ref{4d5,,await} names are now reserved keywords.
Code using these names as identifiers will now raise a @ref{21a,,SyntaxError}.
(Contributed by Jelle Zijlstra in bpo-30406@footnote{https://bugs.python.org/issue30406}.)

@item 
@geindex Python Enhancement Proposals; PEP 479
PEP 479@footnote{https://www.python.org/dev/peps/pep-0479} is enabled for all code in Python 3.7, meaning that
@ref{736,,StopIteration} exceptions raised directly or indirectly in
coroutines and generators are transformed into @ref{5af,,RuntimeError}
exceptions.
(Contributed by Yury Selivanov in bpo-32670@footnote{https://bugs.python.org/issue32670}.)

@item 
@ref{737,,object.__aiter__()} methods can no longer be declared as
asynchronous.  (Contributed by Yury Selivanov in bpo-31709@footnote{https://bugs.python.org/issue31709}.)

@item 
Due to an oversight, earlier Python versions erroneously accepted the
following syntax:

@example
f(1 for x in [1],)

class C(1 for x in [1]):
    pass
@end example

Python 3.7 now correctly raises a @ref{21a,,SyntaxError}, as a generator
expression always needs to be directly inside a set of parentheses
and cannot have a comma on either side, and the duplication of the
parentheses can be omitted only on calls.
(Contributed by Serhiy Storchaka in bpo-32012@footnote{https://bugs.python.org/issue32012} and bpo-32023@footnote{https://bugs.python.org/issue32023}.)

@item 
When using the @ref{611,,-m} switch, the initial working directory is now added
to @ref{489,,sys.path}, rather than an empty string (which dynamically denoted
the current working directory at the time of each import). Any programs that
are checking for the empty string, or otherwise relying on the previous
behaviour, will need to be updated accordingly (e.g. by also checking for
@code{os.getcwd()} or @code{os.path.dirname(__main__.__file__)}, depending on why
the code was checking for the empty string in the first place).
@end itemize

@node Changes in the Python API<5>,Changes in the C API<4>,Changes in Python Behavior,Porting to Python 3 7
@anchor{whatsnew/3 7 changes-in-the-python-api}@anchor{738}
@subsubsection Changes in the Python API



@itemize *

@item 
@code{socketserver.ThreadingMixIn.server_close()} now waits until all
non-daemon threads complete.  Set the new
@code{socketserver.ThreadingMixIn.block_on_close} class attribute to
@code{False} to get the pre-3.7 behaviour.
(Contributed by Victor Stinner in bpo-31233@footnote{https://bugs.python.org/issue31233} and bpo-33540@footnote{https://bugs.python.org/issue33540}.)

@item 
@code{socketserver.ForkingMixIn.server_close()} now waits until all
child processes complete. Set the new
@code{socketserver.ForkingMixIn.block_on_close} class attribute to @code{False}
to get the pre-3.7 behaviour.
(Contributed by Victor Stinner in bpo-31151@footnote{https://bugs.python.org/issue31151} and bpo-33540@footnote{https://bugs.python.org/issue33540}.)

@item 
The @ref{739,,locale.localeconv()} function now temporarily sets the @code{LC_CTYPE}
locale to the value of @code{LC_NUMERIC} in some cases.
(Contributed by Victor Stinner in bpo-31900@footnote{https://bugs.python.org/issue31900}.)

@item 
@ref{73a,,pkgutil.walk_packages()} now raises a @ref{17c,,ValueError} if @emph{path} is
a string.  Previously an empty list was returned.
(Contributed by Sanyam Khurana in bpo-24744@footnote{https://bugs.python.org/issue24744}.)

@item 
A format string argument for @ref{73b,,string.Formatter.format()}
is now @ref{59d,,positional-only}.
Passing it as a keyword argument was deprecated in Python 3.5. (Contributed
by Serhiy Storchaka in bpo-29193@footnote{https://bugs.python.org/issue29193}.)

@item 
Attributes @ref{73c,,key},
@ref{73d,,value} and
@ref{73e,,coded_value} of class
@ref{73f,,http.cookies.Morsel} are now read-only.
Assigning to them was deprecated in Python 3.5.
Use the @ref{740,,set()} method for setting them.
(Contributed by Serhiy Storchaka in bpo-29192@footnote{https://bugs.python.org/issue29192}.)

@item 
The @emph{mode} argument of @ref{68c,,os.makedirs()} no longer affects the file
permission bits of newly-created intermediate-level directories.
To set their file permission bits you can set the umask before invoking
@code{makedirs()}.
(Contributed by Serhiy Storchaka in bpo-19930@footnote{https://bugs.python.org/issue19930}.)

@item 
The @ref{741,,struct.Struct.format} type is now @ref{2ef,,str} instead of
@ref{2f0,,bytes}. (Contributed by Victor Stinner in bpo-21071@footnote{https://bugs.python.org/issue21071}.)

@item 
@ref{2e6,,parse_multipart()} now accepts the @emph{encoding} and @emph{errors}
arguments and returns the same results as
@code{FieldStorage}: for non-file fields, the value associated to a key
is a list of strings, not bytes.
(Contributed by Pierre Quentel in bpo-29979@footnote{https://bugs.python.org/issue29979}.)

@item 
Due to internal changes in @ref{ea,,socket}, calling @ref{742,,socket.fromshare()}
on a socket created by @ref{743,,socket.share} in older
Python versions is not supported.

@item 
@code{repr} for @ref{15f,,BaseException} has changed to not include the trailing
comma.  Most exceptions are affected by this change.
(Contributed by Serhiy Storchaka in bpo-30399@footnote{https://bugs.python.org/issue30399}.)

@item 
@code{repr} for @ref{4c8,,datetime.timedelta} has changed to include the keyword
arguments in the output. (Contributed by Utkarsh Upadhyay in bpo-30302@footnote{https://bugs.python.org/issue30302}.)

@item 
Because @ref{53a,,shutil.rmtree()} is now implemented using the @ref{576,,os.scandir()}
function, the user specified handler @emph{onerror} is now called with the first
argument @code{os.scandir} instead of @code{os.listdir} when listing the directory
is failed.

@item 
Support for nested sets and set operations in regular expressions as in
Unicode Technical Standard #18@footnote{https://unicode.org/reports/tr18/} might be added in the future.  This would
change the syntax.  To facilitate this future change a @ref{604,,FutureWarning}
will be raised in ambiguous cases for the time being.
That include sets starting with a literal @code{'['} or containing literal
character sequences @code{'--'}, @code{'&&'}, @code{'~~'}, and @code{'||'}.  To
avoid a warning, escape them with a backslash.
(Contributed by Serhiy Storchaka in bpo-30349@footnote{https://bugs.python.org/issue30349}.)

@item 
The result of splitting a string on a @ref{d8,,regular expression}
that could match an empty string has been changed.  For example
splitting on @code{r'\s*'} will now split not only on whitespaces as it
did previously, but also on empty strings before all non-whitespace
characters and just before the end of the string.
The previous behavior can be restored by changing the pattern
to @code{r'\s+'}.  A @ref{604,,FutureWarning} was emitted for such patterns since
Python 3.5.

For patterns that match both empty and non-empty strings, the result of
searching for all matches may also be changed in other cases.  For example
in the string @code{'a\n\n'}, the pattern @code{r'(?m)^\s*?$'} will not only
match empty strings at positions 2 and 3, but also the string @code{'\n'} at
positions 2–3.  To match only blank lines, the pattern should be rewritten
as @code{r'(?m)^[^\S\n]*$'}.

@ref{72e,,re.sub()} now replaces empty matches adjacent to a previous
non-empty match.  For example @code{re.sub('x*', '-', 'abxd')} returns now
@code{'-a-b--d-'} instead of @code{'-a-b-d-'} (the first minus between ‘b’ and
‘d’ replaces ‘x’, and the second minus replaces an empty string between
‘x’ and ‘d’).

(Contributed by Serhiy Storchaka in bpo-25054@footnote{https://bugs.python.org/issue25054} and bpo-32308@footnote{https://bugs.python.org/issue32308}.)

@item 
Change @ref{744,,re.escape()} to only escape regex special characters instead
of escaping all characters other than ASCII letters, numbers, and @code{'_'}.
(Contributed by Serhiy Storchaka in bpo-29995@footnote{https://bugs.python.org/issue29995}.)

@item 
@ref{6c7,,tracemalloc.Traceback} frames are now sorted from oldest to most
recent to be more consistent with @ref{11a,,traceback}.
(Contributed by Jesse Bakker in bpo-32121@footnote{https://bugs.python.org/issue32121}.)

@item 
On OSes that support @ref{745,,socket.SOCK_NONBLOCK} or
@ref{746,,socket.SOCK_CLOEXEC} bit flags, the
@ref{747,,socket.type} no longer has them applied.
Therefore, checks like @code{if sock.type == socket.SOCK_STREAM}
work as expected on all platforms.
(Contributed by Yury Selivanov in bpo-32331@footnote{https://bugs.python.org/issue32331}.)

@item 
On Windows the default for the @emph{close_fds} argument of
@ref{5ae,,subprocess.Popen} was changed from @ref{659,,False} to @ref{748,,True}
when redirecting the standard handles. If you previously depended on handles
being inherited when using @ref{5ae,,subprocess.Popen} with standard io
redirection, you will have to pass @code{close_fds=False} to preserve the
previous behaviour, or use
@ref{749,,STARTUPINFO.lpAttributeList}.

@item 
@ref{74a,,importlib.machinery.PathFinder.invalidate_caches()} – which implicitly
affects @ref{74b,,importlib.invalidate_caches()} – now deletes entries
in @ref{74c,,sys.path_importer_cache} which are set to @code{None}.
(Contributed by Brett Cannon in bpo-33169@footnote{https://bugs.python.org/issue33169}.)

@item 
In @ref{9,,asyncio},
@ref{74d,,loop.sock_recv()},
@ref{74e,,loop.sock_sendall()},
@ref{74f,,loop.sock_accept()},
@ref{750,,loop.getaddrinfo()},
@ref{751,,loop.getnameinfo()}
have been changed to be proper coroutine methods to match their
documentation.  Previously, these methods returned @ref{316,,asyncio.Future}
instances.
(Contributed by Yury Selivanov in bpo-32327@footnote{https://bugs.python.org/issue32327}.)

@item 
@ref{752,,asyncio.Server.sockets} now returns a copy of the internal list
of server sockets, instead of returning it directly.
(Contributed by Yury Selivanov in bpo-32662@footnote{https://bugs.python.org/issue32662}.)

@item 
@ref{741,,Struct.format} is now a @ref{2ef,,str} instance
instead of a @ref{2f0,,bytes} instance.
(Contributed by Victor Stinner in bpo-21071@footnote{https://bugs.python.org/issue21071}.)

@item 
@ref{5,,argparse} subparsers can now be made mandatory by passing @code{required=True}
to @ref{753,,ArgumentParser.add_subparsers()}.
(Contributed by Anthony Sottile in bpo-26510@footnote{https://bugs.python.org/issue26510}.)

@item 
@ref{754,,ast.literal_eval()} is now stricter.  Addition and subtraction of
arbitrary numbers are no longer allowed.
(Contributed by Serhiy Storchaka in bpo-31778@footnote{https://bugs.python.org/issue31778}.)

@item 
@ref{755,,Calendar.itermonthdates}
will now consistently raise an exception when a date falls outside of the
@code{0001-01-01} through @code{9999-12-31} range.  To support applications that
cannot tolerate such exceptions, the new
@ref{756,,Calendar.itermonthdays3} and
@ref{757,,Calendar.itermonthdays4} can be used.
The new methods return tuples and are not restricted by the range supported by
@ref{3e2,,datetime.date}.
(Contributed by Alexander Belopolsky in bpo-28292@footnote{https://bugs.python.org/issue28292}.)

@item 
@ref{758,,collections.ChainMap} now preserves the order of the underlying
mappings.  (Contributed by Raymond Hettinger in bpo-32792@footnote{https://bugs.python.org/issue32792}.)

@item 
The @code{submit()} method of @ref{1c7,,concurrent.futures.ThreadPoolExecutor}
and @ref{3da,,concurrent.futures.ProcessPoolExecutor} now raises
a @ref{5af,,RuntimeError} if called during interpreter shutdown.
(Contributed by Mark Nemec in bpo-33097@footnote{https://bugs.python.org/issue33097}.)

@item 
The @ref{759,,configparser.ConfigParser} constructor now uses @code{read_dict()}
to process the default values, making its behavior consistent with the
rest of the parser.  Non-string keys and values in the defaults
dictionary are now being implicitly converted to strings.
(Contributed by James Tocknell in bpo-23835@footnote{https://bugs.python.org/issue23835}.)

@item 
Several undocumented internal imports were removed.
One example is that @code{os.errno} is no longer available; use @code{import errno}
directly instead.
Note that such undocumented internal imports may be removed any time without
notice, even in micro version releases.
@end itemize

@node Changes in the C API<4>,CPython bytecode changes<5>,Changes in the Python API<5>,Porting to Python 3 7
@anchor{whatsnew/3 7 changes-in-the-c-api}@anchor{75a}
@subsubsection Changes in the C API


The function @ref{72b,,PySlice_GetIndicesEx()} is considered unsafe for
resizable sequences.  If the slice indices are not instances of @ref{1b1,,int},
but objects that implement the @code{__index__()} method, the sequence can be
resized after passing its length to !@ref{72b,,PySlice_GetIndicesEx}.  This
can lead to returning indices out of the length of the sequence.  For
avoiding possible problems use new functions @ref{6f3,,PySlice_Unpack()} and
@ref{6f4,,PySlice_AdjustIndices()}.
(Contributed by Serhiy Storchaka in bpo-27867@footnote{https://bugs.python.org/issue27867}.)

@node CPython bytecode changes<5>,Windows-only Changes<2>,Changes in the C API<4>,Porting to Python 3 7
@anchor{whatsnew/3 7 cpython-bytecode-changes}@anchor{75b}
@subsubsection CPython bytecode changes


There are two new opcodes: @ref{75c,,LOAD_METHOD} and @code{CALL_METHOD}.
(Contributed by Yury Selivanov and INADA Naoki in bpo-26110@footnote{https://bugs.python.org/issue26110}.)

The @code{STORE_ANNOTATION} opcode has been removed.
(Contributed by Mark Shannon in bpo-32550@footnote{https://bugs.python.org/issue32550}.)

@node Windows-only Changes<2>,Other CPython implementation changes,CPython bytecode changes<5>,Porting to Python 3 7
@anchor{whatsnew/3 7 id12}@anchor{75d}
@subsubsection Windows-only Changes


The file used to override @ref{489,,sys.path} is now called
@code{<python-executable>._pth} instead of @code{'sys.path'}.
See @ref{75e,,Finding modules} for more information.
(Contributed by Steve Dower in bpo-28137@footnote{https://bugs.python.org/issue28137}.)

@node Other CPython implementation changes,,Windows-only Changes<2>,Porting to Python 3 7
@anchor{whatsnew/3 7 id13}@anchor{75f}
@subsubsection Other CPython implementation changes


In preparation for potential future changes to the public CPython runtime
initialization API (see 
@geindex Python Enhancement Proposals; PEP 432
PEP 432@footnote{https://www.python.org/dev/peps/pep-0432} for an initial, but somewhat outdated,
draft), CPython’s internal startup
and configuration management logic has been significantly refactored. While
these updates are intended to be entirely transparent to both embedding
applications and users of the regular CPython CLI, they’re being mentioned
here as the refactoring changes the internal order of various operations
during interpreter startup, and hence may uncover previously latent defects,
either in embedding applications, or in CPython itself.
(Initially contributed by Nick Coghlan and Eric Snow as part of
bpo-22257@footnote{https://bugs.python.org/issue22257}, and further updated by Nick, Eric, and Victor Stinner in a
number of other issues). Some known details affected:


@itemize *

@item 
@ref{1ff,,PySys_AddWarnOptionUnicode()} is not currently usable by embedding
applications due to the requirement to create a Unicode object prior to
calling @cite{Py_Initialize}. Use @ref{200,,PySys_AddWarnOption()} instead.

@item 
warnings filters added by an embedding application with
@ref{200,,PySys_AddWarnOption()} should now more consistently take precedence
over the default filters set by the interpreter
@end itemize

Due to changes in the way the default warnings filters are configured,
setting @ref{760,,Py_BytesWarningFlag} to a value greater than one is no longer
sufficient to both emit @ref{761,,BytesWarning} messages and have them converted
to exceptions.  Instead, the flag must be set (to cause the warnings to be
emitted in the first place), and an explicit @code{error::BytesWarning}
warnings filter added to convert them to exceptions.

Due to a change in the way docstrings are handled by the compiler, the
implicit @code{return None} in a function body consisting solely of a docstring
is now marked as occurring on the same line as the docstring, not on the
function’s header line.

The current exception state has been moved from the frame object to the co-routine.
This simplified the interpreter and fixed a couple of obscure bugs caused by
having swap exception state when entering or exiting a generator.
(Contributed by Mark Shannon in bpo-25612@footnote{https://bugs.python.org/issue25612}.)

@node Notable changes in Python 3 7 1,Notable changes in Python 3 7 2,Porting to Python 3 7,What’s New In Python 3 7
@anchor{whatsnew/3 7 notable-changes-in-python-3-7-1}@anchor{762}
@subsection Notable changes in Python 3.7.1


Starting in 3.7.1, @ref{38d,,Py_Initialize()} now consistently reads and respects
all of the same environment settings as @ref{763,,Py_Main()} (in earlier Python
versions, it respected an ill-defined subset of those environment variables,
while in Python 3.7.0 it didn’t read any of them due to bpo-34247@footnote{https://bugs.python.org/issue34247}). If
this behavior is unwanted, set @ref{764,,Py_IgnoreEnvironmentFlag} to 1 before
calling @ref{38d,,Py_Initialize()}.

In 3.7.1 the C API for Context Variables
@ref{765,,was updated} to use
@ref{375,,PyObject} pointers.  See also bpo-34762@footnote{https://bugs.python.org/issue34762}.

In 3.7.1 the @ref{118,,tokenize} module now implicitly emits a @code{NEWLINE} token
when provided with input that does not have a trailing new line.  This behavior
now matches what the C tokenizer does internally.
(Contributed by Ammar Askar in bpo-33899@footnote{https://bugs.python.org/issue33899}.)

@node Notable changes in Python 3 7 2,Notable changes in Python 3 7 6,Notable changes in Python 3 7 1,What’s New In Python 3 7
@anchor{whatsnew/3 7 notable-changes-in-python-3-7-2}@anchor{766}
@subsection Notable changes in Python 3.7.2


In 3.7.2, @ref{12c,,venv} on Windows no longer copies the original binaries, but
creates redirector scripts named @code{python.exe} and @code{pythonw.exe} instead.
This resolves a long standing issue where all virtual environments would have
to be upgraded or recreated with each Python update. However, note that this
release will still require recreation of virtual environments in order to get
the new scripts.

@node Notable changes in Python 3 7 6,Notable changes in Python 3 7 10,Notable changes in Python 3 7 2,What’s New In Python 3 7
@anchor{whatsnew/3 7 notable-changes-in-python-3-7-6}@anchor{767}
@subsection Notable changes in Python 3.7.6


Due to significant security concerns, the @emph{reuse_address} parameter of
@ref{1b9,,asyncio.loop.create_datagram_endpoint()} is no longer supported. This is
because of the behavior of the socket option @code{SO_REUSEADDR} in UDP. For more
details, see the documentation for @code{loop.create_datagram_endpoint()}.
(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in
bpo-37228@footnote{https://bugs.python.org/issue37228}.)

@node Notable changes in Python 3 7 10,,Notable changes in Python 3 7 6,What’s New In Python 3 7
@anchor{whatsnew/3 7 notable-changes-in-python-3-7-10}@anchor{768}
@subsection Notable changes in Python 3.7.10


Earlier Python versions allowed using both @code{;} and @code{&} as
query parameter separators in @ref{2e3,,urllib.parse.parse_qs()} and
@ref{2e4,,urllib.parse.parse_qsl()}.  Due to security concerns, and to conform with
newer W3C recommendations, this has been changed to allow only a single
separator key, with @code{&} as the default.  This change also affects
@ref{2e5,,cgi.parse()} and @ref{2e6,,cgi.parse_multipart()} as they use the affected
functions internally. For more details, please see their respective
documentation.
(Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in bpo-42967@footnote{https://bugs.python.org/issue42967}.)

@node What’s New In Python 3 6,What’s New In Python 3 5,What’s New In Python 3 7,What’s New in Python
@anchor{whatsnew/3 6 doc}@anchor{769}@anchor{whatsnew/3 6 what-s-new-in-python-3-6}@anchor{76a}
@section What’s New In Python 3.6


@*Editors: 
Elvis Pranskevichus <@email{elvis@@magic.io}>, Yury Selivanov <@email{yury@@magic.io}>


@c Rules for maintenance:
@c 
@c * Anyone can add text to this document.  Do not spend very much time
@c on the wording of your changes, because your text will probably
@c get rewritten to some degree.
@c 
@c * The maintainer will go through Misc/NEWS periodically and add
@c changes; it's therefore more important to add your changes to
@c Misc/NEWS than to this file.
@c 
@c * This is not a complete list of every single change; completeness
@c is the purpose of Misc/NEWS.  Some changes I consider too small
@c or esoteric to include.  If such a change is added to the text,
@c I'll just remove it.  (This is another reason you shouldn't spend
@c too much time on writing your addition.)
@c 
@c * If you want to draw your new text to the attention of the
@c maintainer, add 'XXX' to the beginning of the paragraph or
@c section.
@c 
@c * It's OK to just add a fragmentary note about a change.  For
@c example: "XXX Describe the transmogrify() function added to the
@c socket module."  The maintainer will research the change and
@c write the necessary text.
@c 
@c * You can comment out your additions if you like, but it's not
@c necessary (especially when a final release is some months away).
@c 
@c * Credit the author of a patch or bugfix.   Just the name is
@c sufficient; the e-mail address isn't necessary.
@c 
@c * It's helpful to add the bug/patch number as a comment:
@c 
@c XXX Describe the transmogrify() function added to the socket
@c module.
@c (Contributed by P.Y. Developer in :issue:`12345`.)
@c 
@c This saves the maintainer the effort of going through the Mercurial log
@c when researching a change.

This article explains the new features in Python 3.6, compared to 3.5.
Python 3.6 was released on December 23, 2016.  See the
changelog@footnote{https://docs.python.org/3.6/whatsnew/changelog.html} for a full
list of changes.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 494
PEP 494@footnote{https://www.python.org/dev/peps/pep-0494} - Python 3.6 Release Schedule


@menu
* Summary – Release highlights: Summary – Release highlights<5>. 
* New Features: New Features<9>. 
* Other Language Changes: Other Language Changes<6>. 
* New Modules: New Modules<6>. 
* Improved Modules: Improved Modules<6>. 
* Optimizations: Optimizations<6>. 
* Build and C API Changes: Build and C API Changes<2>. 
* Other Improvements:: 
* Deprecated: Deprecated<7>. 
* Removed: Removed<7>. 
* Porting to Python 3.6: Porting to Python 3 6. 
* Notable changes in Python 3.6.2: Notable changes in Python 3 6 2. 
* Notable changes in Python 3.6.4: Notable changes in Python 3 6 4. 
* Notable changes in Python 3.6.5: Notable changes in Python 3 6 5. 
* Notable changes in Python 3.6.7: Notable changes in Python 3 6 7. 
* Notable changes in Python 3.6.10: Notable changes in Python 3 6 10. 
* Notable changes in Python 3.6.13: Notable changes in Python 3 6 13. 

@end menu

@node Summary – Release highlights<5>,New Features<9>,,What’s New In Python 3 6
@anchor{whatsnew/3 6 summary-release-highlights}@anchor{76b}
@subsection Summary – Release highlights


New syntax features:


@itemize *

@item 
@ref{76c,,PEP 498}, formatted string literals.

@item 
@ref{76d,,PEP 515}, underscores in numeric literals.

@item 
@ref{76e,,PEP 526}, syntax for variable annotations.

@item 
@ref{76f,,PEP 525}, asynchronous generators.

@item 
@ref{770,,PEP 530}: asynchronous comprehensions.
@end itemize

New library modules:


@itemize *

@item 
@ref{df,,secrets}: @ref{771,,PEP 506 – Adding A Secrets Module To The Standard Library}.
@end itemize

CPython implementation improvements:


@itemize *

@item 
The @ref{5de,,dict} type has been reimplemented to use
a @ref{772,,more compact representation}
based on a proposal by Raymond Hettinger@footnote{https://mail.python.org/pipermail/python-dev/2012-December/123028.html}
and similar to the PyPy dict implementation@footnote{https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html}.  This resulted in dictionaries
using 20% to 25% less memory when compared to Python 3.5.

@item 
Customization of class creation has been simplified with the
@ref{773,,new protocol}.

@item 
The class attribute definition order is
@ref{774,,now preserved}.

@item 
The order of elements in @code{**kwargs} now
@ref{775,,corresponds to the order} in which keyword
arguments were passed to the function.

@item 
DTrace and SystemTap @ref{776,,probing support} has
been added.

@item 
The new @ref{777,,PYTHONMALLOC} environment variable
can now be used to debug the interpreter memory allocation and access
errors.
@end itemize

Significant improvements in the standard library:


@itemize *

@item 
The @ref{9,,asyncio} module has received new features, significant
usability and performance improvements, and a fair amount of bug fixes.
Starting with Python 3.6 the @code{asyncio} module is no longer provisional
and its API is considered stable.

@item 
A new @ref{778,,file system path protocol} has been
implemented to support @ref{63d,,path-like objects}.
All standard library functions operating on paths have been updated to
work with the new protocol.

@item 
The @ref{2f,,datetime} module has gained support for
@ref{779,,Local Time Disambiguation}.

@item 
The @ref{120,,typing} module received a number of
@ref{77a,,improvements}.

@item 
The @ref{11b,,tracemalloc} module has been significantly reworked
and is now used to provide better output for @ref{77b,,ResourceWarning}
as well as provide better diagnostics for memory allocation errors.
See the @ref{777,,PYTHONMALLOC section} for more
information.
@end itemize

Security improvements:


@itemize *

@item 
The new @ref{df,,secrets} module has been added to simplify the generation of
cryptographically strong pseudo-random numbers suitable for
managing secrets such as account authentication, tokens, and similar.

@item 
On Linux, @ref{17f,,os.urandom()} now blocks until the system urandom entropy
pool is initialized to increase the security. See the 
@geindex Python Enhancement Proposals; PEP 524
PEP 524@footnote{https://www.python.org/dev/peps/pep-0524} for the
rationale.

@item 
The @ref{88,,hashlib} and @ref{ee,,ssl} modules now support OpenSSL 1.1.0.

@item 
The default settings and feature set of the @ref{ee,,ssl} module have been
improved.

@item 
The @ref{88,,hashlib} module received support for the BLAKE2, SHA-3 and SHAKE
hash algorithms and the @ref{77c,,scrypt()} key derivation function.
@end itemize

Windows improvements:


@itemize *

@item 
@ref{77d,,PEP 528} and @ref{77e,,PEP 529},
Windows filesystem and console encoding changed to UTF-8.

@item 
The @code{py.exe} launcher, when used interactively, no longer prefers
Python 2 over Python 3 when the user doesn’t specify a version (via
command line arguments or a config file).  Handling of shebang lines
remains unchanged - “python” refers to Python 2 in that case.

@item 
@code{python.exe} and @code{pythonw.exe} have been marked as long-path aware,
which means that the 260 character path limit may no longer apply.
See @ref{77f,,removing the MAX_PATH limitation} for details.

@item 
A @code{._pth} file can be added to force isolated mode and fully specify
all search paths to avoid registry and environment lookup. See
@ref{75e,,the documentation} for more information.

@item 
A @code{python36.zip} file now works as a landmark to infer
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME}. See @ref{75e,,the documentation} for
more information.
@end itemize

@node New Features<9>,Other Language Changes<6>,Summary – Release highlights<5>,What’s New In Python 3 6
@anchor{whatsnew/3 6 new-features}@anchor{781}@anchor{whatsnew/3 6 pypy-dict-implementation}@anchor{782}
@subsection New Features


@menu
* PEP 498; Formatted string literals: PEP 498 Formatted string literals. 
* PEP 526; Syntax for variable annotations: PEP 526 Syntax for variable annotations. 
* PEP 515; Underscores in Numeric Literals: PEP 515 Underscores in Numeric Literals. 
* PEP 525; Asynchronous Generators: PEP 525 Asynchronous Generators. 
* PEP 530; Asynchronous Comprehensions: PEP 530 Asynchronous Comprehensions. 
* PEP 487; Simpler customization of class creation: PEP 487 Simpler customization of class creation. 
* PEP 487; Descriptor Protocol Enhancements: PEP 487 Descriptor Protocol Enhancements. 
* PEP 519; Adding a file system path protocol: PEP 519 Adding a file system path protocol. 
* PEP 495; Local Time Disambiguation: PEP 495 Local Time Disambiguation. 
* PEP 529; Change Windows filesystem encoding to UTF-8: PEP 529 Change Windows filesystem encoding to UTF-8. 
* PEP 528; Change Windows console encoding to UTF-8: PEP 528 Change Windows console encoding to UTF-8. 
* PEP 520; Preserving Class Attribute Definition Order: PEP 520 Preserving Class Attribute Definition Order. 
* PEP 468; Preserving Keyword Argument Order: PEP 468 Preserving Keyword Argument Order. 
* New dict implementation:: 
* PEP 523; Adding a frame evaluation API to CPython: PEP 523 Adding a frame evaluation API to CPython. 
* PYTHONMALLOC environment variable:: 
* DTrace and SystemTap probing support:: 

@end menu

@node PEP 498 Formatted string literals,PEP 526 Syntax for variable annotations,,New Features<9>
@anchor{whatsnew/3 6 pep-498-formatted-string-literals}@anchor{783}@anchor{whatsnew/3 6 whatsnew36-pep498}@anchor{76c}
@subsubsection PEP 498: Formatted string literals


@geindex Python Enhancement Proposals; PEP 498
PEP 498@footnote{https://www.python.org/dev/peps/pep-0498} introduces a new kind of string literals: @emph{f-strings}, or
@ref{4b0,,formatted string literals}.

Formatted string literals are prefixed with @code{'f'} and are similar to
the format strings accepted by @ref{784,,str.format()}.  They contain replacement
fields surrounded by curly braces.  The replacement fields are expressions,
which are evaluated at run time, and then formatted using the
@ref{785,,format()} protocol:

@example
>>> name = "Fred"
>>> f"He said his name is @{name@}."
'He said his name is Fred.'
>>> width = 10
>>> precision = 4
>>> value = decimal.Decimal("12.34567")
>>> f"result: @{value:@{width@}.@{precision@}@}"  # nested fields
'result:      12.35'
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 498

@item PEP 498@footnote{https://www.python.org/dev/peps/pep-0498} – Literal String Interpolation.

PEP written and implemented by Eric V. Smith.
@end table

@ref{4b0,,Feature documentation}.


@node PEP 526 Syntax for variable annotations,PEP 515 Underscores in Numeric Literals,PEP 498 Formatted string literals,New Features<9>
@anchor{whatsnew/3 6 pep-526-syntax-for-variable-annotations}@anchor{786}@anchor{whatsnew/3 6 whatsnew36-pep526}@anchor{76e}
@subsubsection PEP 526: Syntax for variable annotations


@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} introduced the standard for type annotations of function
parameters, a.k.a. type hints. This PEP adds syntax to Python for annotating
the types of variables including class variables and instance variables:

@example
primes: List[int] = []

captain: str  # Note: no initial value!

class Starship:
    stats: Dict[str, int] = @{@}
@end example

Just as for function annotations, the Python interpreter does not attach any
particular meaning to variable annotations and only stores them in the
@code{__annotations__} attribute of a class or module.

In contrast to variable declarations in statically typed languages,
the goal of annotation syntax is to provide an easy way to specify structured
type metadata for third party tools and libraries via the abstract syntax tree
and the @code{__annotations__} attribute.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 526

@item PEP 526@footnote{https://www.python.org/dev/peps/pep-0526} – Syntax for variable annotations.

PEP written by Ryan Gonzalez, Philip House, Ivan Levkivskyi, Lisa Roach,
and Guido van Rossum. Implemented by Ivan Levkivskyi.
@end table

Tools that use or will use the new syntax:
mypy@footnote{http://www.mypy-lang.org/},
pytype@footnote{https://github.com/google/pytype}, PyCharm, etc.


@node PEP 515 Underscores in Numeric Literals,PEP 525 Asynchronous Generators,PEP 526 Syntax for variable annotations,New Features<9>
@anchor{whatsnew/3 6 pep-515-underscores-in-numeric-literals}@anchor{787}@anchor{whatsnew/3 6 whatsnew36-pep515}@anchor{76d}
@subsubsection PEP 515: Underscores in Numeric Literals


@geindex Python Enhancement Proposals; PEP 515
PEP 515@footnote{https://www.python.org/dev/peps/pep-0515} adds the ability to use underscores in numeric literals for
improved readability.  For example:

@example
>>> 1_000_000_000_000_000
1000000000000000
>>> 0x_FF_FF_FF_FF
4294967295
@end example

Single underscores are allowed between digits and after any base
specifier.  Leading, trailing, or multiple underscores in a row are not
allowed.

The @ref{788,,string formatting} language also now has support
for the @code{'_'} option to signal the use of an underscore for a thousands
separator for floating point presentation types and for integer
presentation type @code{'d'}.  For integer presentation types @code{'b'},
@code{'o'}, @code{'x'}, and @code{'X'}, underscores will be inserted every 4
digits:

@example
>>> '@{:_@}'.format(1000000)
'1_000_000'
>>> '@{:_x@}'.format(0xFFFFFFFF)
'ffff_ffff'
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 515

@item PEP 515@footnote{https://www.python.org/dev/peps/pep-0515} – Underscores in Numeric Literals

PEP written by Georg Brandl and Serhiy Storchaka.
@end table


@node PEP 525 Asynchronous Generators,PEP 530 Asynchronous Comprehensions,PEP 515 Underscores in Numeric Literals,New Features<9>
@anchor{whatsnew/3 6 pep-525-asynchronous-generators}@anchor{789}@anchor{whatsnew/3 6 whatsnew36-pep525}@anchor{76f}
@subsubsection PEP 525: Asynchronous Generators


@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} introduced support for native coroutines and @code{async} / @code{await}
syntax to Python 3.5.  A notable limitation of the Python 3.5 implementation
is that it was not possible to use @code{await} and @code{yield} in the same
function body.  In Python 3.6 this restriction has been lifted, making it
possible to define @emph{asynchronous generators}:

@example
async def ticker(delay, to):
    """Yield numbers from 0 to *to* every *delay* seconds."""
    for i in range(to):
        yield i
        await asyncio.sleep(delay)
@end example

The new syntax allows for faster and more concise code.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 525

@item PEP 525@footnote{https://www.python.org/dev/peps/pep-0525} – Asynchronous Generators

PEP written and implemented by Yury Selivanov.
@end table


@node PEP 530 Asynchronous Comprehensions,PEP 487 Simpler customization of class creation,PEP 525 Asynchronous Generators,New Features<9>
@anchor{whatsnew/3 6 pep-530-asynchronous-comprehensions}@anchor{78a}@anchor{whatsnew/3 6 whatsnew36-pep530}@anchor{770}
@subsubsection PEP 530: Asynchronous Comprehensions


@geindex Python Enhancement Proposals; PEP 530
PEP 530@footnote{https://www.python.org/dev/peps/pep-0530} adds support for using @code{async for} in list, set, dict
comprehensions and generator expressions:

@example
result = [i async for i in aiter() if i % 2]
@end example

Additionally, @code{await} expressions are supported in all kinds
of comprehensions:

@example
result = [await fun() for fun in funcs if await condition()]
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 530

@item PEP 530@footnote{https://www.python.org/dev/peps/pep-0530} – Asynchronous Comprehensions

PEP written and implemented by Yury Selivanov.
@end table


@node PEP 487 Simpler customization of class creation,PEP 487 Descriptor Protocol Enhancements,PEP 530 Asynchronous Comprehensions,New Features<9>
@anchor{whatsnew/3 6 pep-487-simpler-customization-of-class-creation}@anchor{78b}@anchor{whatsnew/3 6 whatsnew36-pep487}@anchor{773}
@subsubsection PEP 487: Simpler customization of class creation


It is now possible to customize subclass creation without using a metaclass.
The new @code{__init_subclass__} classmethod will be called on the base class
whenever a new subclass is created:

@example
class PluginBase:
    subclasses = []

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        cls.subclasses.append(cls)

class Plugin1(PluginBase):
    pass

class Plugin2(PluginBase):
    pass
@end example

In order to allow zero-argument @ref{78c,,super()} calls to work correctly from
@ref{78d,,__init_subclass__()} implementations, custom metaclasses must
ensure that the new @code{__classcell__} namespace entry is propagated to
@code{type.__new__} (as described in @ref{78e,,Creating the class object}).


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 487

@item PEP 487@footnote{https://www.python.org/dev/peps/pep-0487} – Simpler customization of class creation

PEP written and implemented by Martin Teichmann.
@end table

@ref{78f,,Feature documentation}


@node PEP 487 Descriptor Protocol Enhancements,PEP 519 Adding a file system path protocol,PEP 487 Simpler customization of class creation,New Features<9>
@anchor{whatsnew/3 6 pep-487-descriptor-protocol-enhancements}@anchor{790}@anchor{whatsnew/3 6 whatsnew36-pep487-descriptors}@anchor{791}
@subsubsection PEP 487: Descriptor Protocol Enhancements


@geindex Python Enhancement Proposals; PEP 487
PEP 487@footnote{https://www.python.org/dev/peps/pep-0487} extends the descriptor protocol to include the new optional
@ref{792,,__set_name__()} method.  Whenever a new class is defined, the new
method will be called on all descriptors included in the definition, providing
them with a reference to the class being defined and the name given to the
descriptor within the class namespace.  In other words, instances of
descriptors can now know the attribute name of the descriptor in the
owner class:

@example
class IntField:
    def __get__(self, instance, owner):
        return instance.__dict__[self.name]

    def __set__(self, instance, value):
        if not isinstance(value, int):
            raise ValueError(f'expecting integer in @{self.name@}')
        instance.__dict__[self.name] = value

    # this is the new initializer:
    def __set_name__(self, owner, name):
        self.name = name

class Model:
    int_field = IntField()
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 487

@item PEP 487@footnote{https://www.python.org/dev/peps/pep-0487} – Simpler customization of class creation

PEP written and implemented by Martin Teichmann.
@end table

@ref{793,,Feature documentation}


@node PEP 519 Adding a file system path protocol,PEP 495 Local Time Disambiguation,PEP 487 Descriptor Protocol Enhancements,New Features<9>
@anchor{whatsnew/3 6 pep-519-adding-a-file-system-path-protocol}@anchor{794}@anchor{whatsnew/3 6 whatsnew36-pep519}@anchor{778}
@subsubsection PEP 519: Adding a file system path protocol


File system paths have historically been represented as @ref{2ef,,str}
or @ref{2f0,,bytes} objects. This has led to people who write code which
operate on file system paths to assume that such objects are only one
of those two types (an @ref{1b1,,int} representing a file descriptor
does not count as that is not a file path). Unfortunately that
assumption prevents alternative object representations of file system
paths like @ref{c3,,pathlib} from working with pre-existing code,
including Python’s standard library.

To fix this situation, a new interface represented by
@ref{795,,os.PathLike} has been defined. By implementing the
@ref{796,,__fspath__()} method, an object signals that it
represents a path. An object can then provide a low-level
representation of a file system path as a @ref{2ef,,str} or
@ref{2f0,,bytes} object. This means an object is considered
@ref{63d,,path-like} if it implements
@ref{795,,os.PathLike} or is a @ref{2ef,,str} or @ref{2f0,,bytes} object
which represents a file system path. Code can use @ref{797,,os.fspath()},
@ref{798,,os.fsdecode()}, or @ref{799,,os.fsencode()} to explicitly get a
@ref{2ef,,str} and/or @ref{2f0,,bytes} representation of a path-like
object.

The built-in @ref{1c8,,open()} function has been updated to accept
@ref{795,,os.PathLike} objects, as have all relevant functions in the
@ref{c0,,os} and @ref{c1,,os.path} modules, and most other functions and
classes in the standard library.  The @ref{79a,,os.DirEntry} class
and relevant classes in @ref{c3,,pathlib} have also been updated to
implement @ref{795,,os.PathLike}.

The hope is that updating the fundamental functions for operating
on file system paths will lead to third-party code to implicitly
support all @ref{63d,,path-like objects} without any
code changes, or at least very minimal ones (e.g. calling
@ref{797,,os.fspath()} at the beginning of code before operating on a
path-like object).

Here are some examples of how the new interface allows for
@ref{522,,pathlib.Path} to be used more easily and transparently with
pre-existing code:

@example
>>> import pathlib
>>> with open(pathlib.Path("README")) as f:
...     contents = f.read()
...
>>> import os.path
>>> os.path.splitext(pathlib.Path("some_file.txt"))
('some_file', '.txt')
>>> os.path.join("/a/b", pathlib.Path("c"))
'/a/b/c'
>>> import os
>>> os.fspath(pathlib.Path("some_file.txt"))
'some_file.txt'
@end example

(Implemented by Brett Cannon, Ethan Furman, Dusty Phillips, and Jelle Zijlstra.)


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 519

@item PEP 519@footnote{https://www.python.org/dev/peps/pep-0519} – Adding a file system path protocol

PEP written by Brett Cannon and Koos Zevenhoven.
@end table


@node PEP 495 Local Time Disambiguation,PEP 529 Change Windows filesystem encoding to UTF-8,PEP 519 Adding a file system path protocol,New Features<9>
@anchor{whatsnew/3 6 pep-495-local-time-disambiguation}@anchor{79b}@anchor{whatsnew/3 6 whatsnew36-pep495}@anchor{779}
@subsubsection PEP 495: Local Time Disambiguation


In most world locations, there have been and will be times when local clocks
are moved back.  In those times, intervals are introduced in which local
clocks show the same time twice in the same day. In these situations, the
information displayed on a local clock (or stored in a Python datetime
instance) is insufficient to identify a particular moment in time.

@geindex Python Enhancement Proposals; PEP 495
PEP 495@footnote{https://www.python.org/dev/peps/pep-0495} adds the new @emph{fold} attribute to instances of
@ref{363,,datetime.datetime} and @ref{364,,datetime.time} classes to differentiate
between two moments in time for which local times are the same:

@example
>>> u0 = datetime(2016, 11, 6, 4, tzinfo=timezone.utc)
>>> for i in range(4):
...     u = u0 + i*HOUR
...     t = u.astimezone(Eastern)
...     print(u.time(), 'UTC =', t.time(), t.tzname(), t.fold)
...
04:00:00 UTC = 00:00:00 EDT 0
05:00:00 UTC = 01:00:00 EDT 0
06:00:00 UTC = 01:00:00 EST 1
07:00:00 UTC = 02:00:00 EST 0
@end example

The values of the @ref{79c,,fold} attribute have the
value @code{0} for all instances except those that represent the second
(chronologically) moment in time in an ambiguous case.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 495

@item PEP 495@footnote{https://www.python.org/dev/peps/pep-0495} – Local Time Disambiguation

PEP written by Alexander Belopolsky and Tim Peters, implementation
by Alexander Belopolsky.
@end table


@node PEP 529 Change Windows filesystem encoding to UTF-8,PEP 528 Change Windows console encoding to UTF-8,PEP 495 Local Time Disambiguation,New Features<9>
@anchor{whatsnew/3 6 pep-529-change-windows-filesystem-encoding-to-utf-8}@anchor{79d}@anchor{whatsnew/3 6 whatsnew36-pep529}@anchor{77e}
@subsubsection PEP 529: Change Windows filesystem encoding to UTF-8


Representing filesystem paths is best performed with str (Unicode) rather than
bytes. However, there are some situations where using bytes is sufficient and
correct.

Prior to Python 3.6, data loss could result when using bytes paths on Windows.
With this change, using bytes to represent paths is now supported on Windows,
provided those bytes are encoded with the encoding returned by
@ref{79e,,sys.getfilesystemencoding()}, which now defaults to @code{'utf-8'}.

Applications that do not use str to represent paths should use
@ref{799,,os.fsencode()} and @ref{798,,os.fsdecode()} to ensure their bytes are
correctly encoded. To revert to the previous behaviour, set
@geindex PYTHONLEGACYWINDOWSFSENCODING
@geindex environment variable; PYTHONLEGACYWINDOWSFSENCODING
@ref{79f,,PYTHONLEGACYWINDOWSFSENCODING} or call
@ref{7a0,,sys._enablelegacywindowsfsencoding()}.

See 
@geindex Python Enhancement Proposals; PEP 529
PEP 529@footnote{https://www.python.org/dev/peps/pep-0529} for more information and discussion of code modifications that
may be required.

@node PEP 528 Change Windows console encoding to UTF-8,PEP 520 Preserving Class Attribute Definition Order,PEP 529 Change Windows filesystem encoding to UTF-8,New Features<9>
@anchor{whatsnew/3 6 pep-528-change-windows-console-encoding-to-utf-8}@anchor{7a1}@anchor{whatsnew/3 6 whatsnew36-pep528}@anchor{77d}
@subsubsection PEP 528: Change Windows console encoding to UTF-8


The default console on Windows will now accept all Unicode characters and
provide correctly read str objects to Python code. @code{sys.stdin},
@code{sys.stdout} and @code{sys.stderr} now default to utf-8 encoding.

This change only applies when using an interactive console, and not when
redirecting files or pipes. To revert to the previous behaviour for interactive
console use, set 
@geindex PYTHONLEGACYWINDOWSSTDIO
@geindex environment variable; PYTHONLEGACYWINDOWSSTDIO
@ref{7a2,,PYTHONLEGACYWINDOWSSTDIO}.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 528

@item PEP 528@footnote{https://www.python.org/dev/peps/pep-0528} – Change Windows console encoding to UTF-8

PEP written and implemented by Steve Dower.
@end table


@node PEP 520 Preserving Class Attribute Definition Order,PEP 468 Preserving Keyword Argument Order,PEP 528 Change Windows console encoding to UTF-8,New Features<9>
@anchor{whatsnew/3 6 pep-520-preserving-class-attribute-definition-order}@anchor{7a3}@anchor{whatsnew/3 6 whatsnew36-pep520}@anchor{774}
@subsubsection PEP 520: Preserving Class Attribute Definition Order


Attributes in a class definition body have a natural ordering: the same
order in which the names appear in the source.  This order is now
preserved in the new class’s @ref{7a4,,__dict__} attribute.

Also, the effective default class @emph{execution} namespace (returned from
@ref{7a5,,type.__prepare__()}) is now an insertion-order-preserving
mapping.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 520

@item PEP 520@footnote{https://www.python.org/dev/peps/pep-0520} – Preserving Class Attribute Definition Order

PEP written and implemented by Eric Snow.
@end table


@node PEP 468 Preserving Keyword Argument Order,New dict implementation,PEP 520 Preserving Class Attribute Definition Order,New Features<9>
@anchor{whatsnew/3 6 pep-468-preserving-keyword-argument-order}@anchor{7a6}@anchor{whatsnew/3 6 whatsnew36-pep468}@anchor{775}
@subsubsection PEP 468: Preserving Keyword Argument Order


@code{**kwargs} in a function signature is now guaranteed to be an
insertion-order-preserving mapping.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 468

@item PEP 468@footnote{https://www.python.org/dev/peps/pep-0468} – Preserving Keyword Argument Order

PEP written and implemented by Eric Snow.
@end table


@node New dict implementation,PEP 523 Adding a frame evaluation API to CPython,PEP 468 Preserving Keyword Argument Order,New Features<9>
@anchor{whatsnew/3 6 new-dict-implementation}@anchor{7a7}@anchor{whatsnew/3 6 whatsnew36-compactdict}@anchor{772}
@subsubsection New dict implementation


The @ref{5de,,dict} type now uses a “compact” representation
based on a proposal by Raymond Hettinger@footnote{https://mail.python.org/pipermail/python-dev/2012-December/123028.html}
which was first implemented by PyPy@footnote{https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html}.
The memory usage of the new @ref{3b0,,dict()} is between 20% and 25% smaller
compared to Python 3.5.

The order-preserving aspect of this new implementation is considered an
implementation detail and should not be relied upon (this may change in
the future, but it is desired to have this new dict implementation in
the language for a few releases before changing the language spec to mandate
order-preserving semantics for all current and future Python
implementations; this also helps preserve backwards-compatibility
with older versions of the language where random iteration order is
still in effect, e.g. Python 3.5).

(Contributed by INADA Naoki in bpo-27350@footnote{https://bugs.python.org/issue27350}. Idea
originally suggested by Raymond Hettinger@footnote{https://mail.python.org/pipermail/python-dev/2012-December/123028.html}.)

@node PEP 523 Adding a frame evaluation API to CPython,PYTHONMALLOC environment variable,New dict implementation,New Features<9>
@anchor{whatsnew/3 6 pep-523-adding-a-frame-evaluation-api-to-cpython}@anchor{7a8}@anchor{whatsnew/3 6 whatsnew36-pep523}@anchor{7a9}
@subsubsection PEP 523: Adding a frame evaluation API to CPython


While Python provides extensive support to customize how code
executes, one place it has not done so is in the evaluation of frame
objects.  If you wanted some way to intercept frame evaluation in
Python there really wasn’t any way without directly manipulating
function pointers for defined functions.

@geindex Python Enhancement Proposals; PEP 523
PEP 523@footnote{https://www.python.org/dev/peps/pep-0523} changes this by providing an API to make frame
evaluation pluggable at the C level. This will allow for tools such
as debuggers and JITs to intercept frame evaluation before the
execution of Python code begins. This enables the use of alternative
evaluation implementations for Python code, tracking frame
evaluation, etc.

This API is not part of the limited C API and is marked as private to
signal that usage of this API is expected to be limited and only
applicable to very select, low-level use-cases. Semantics of the
API will change with Python as necessary.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 523

@item PEP 523@footnote{https://www.python.org/dev/peps/pep-0523} – Adding a frame evaluation API to CPython

PEP written by Brett Cannon and Dino Viehland.
@end table


@node PYTHONMALLOC environment variable,DTrace and SystemTap probing support,PEP 523 Adding a frame evaluation API to CPython,New Features<9>
@anchor{whatsnew/3 6 pythonmalloc-environment-variable}@anchor{7aa}@anchor{whatsnew/3 6 whatsnew36-pythonmalloc}@anchor{777}
@subsubsection PYTHONMALLOC environment variable


The new 
@geindex PYTHONMALLOC
@geindex environment variable; PYTHONMALLOC
@ref{7ab,,PYTHONMALLOC} environment variable allows setting the Python
memory allocators and installing debug hooks.

It is now possible to install debug hooks on Python memory allocators on Python
compiled in release mode using @code{PYTHONMALLOC=debug}. Effects of debug hooks:


@itemize *

@item 
Newly allocated memory is filled with the byte @code{0xCB}

@item 
Freed memory is filled with the byte @code{0xDB}

@item 
Detect violations of the Python memory allocator API. For example,
@ref{7ac,,PyObject_Free()} called on a memory block allocated by
@ref{7ad,,PyMem_Malloc()}.

@item 
Detect writes before the start of a buffer (buffer underflows)

@item 
Detect writes after the end of a buffer (buffer overflows)

@item 
Check that the @ref{7ae,,GIL} is held when allocator
functions of @code{PYMEM_DOMAIN_OBJ} (ex: @ref{7af,,PyObject_Malloc()}) and
@code{PYMEM_DOMAIN_MEM} (ex: @ref{7ad,,PyMem_Malloc()}) domains are called.
@end itemize

Checking if the GIL is held is also a new feature of Python 3.6.

See the @ref{7b0,,PyMem_SetupDebugHooks()} function for debug hooks on Python
memory allocators.

It is now also possible to force the usage of the @code{malloc()} allocator of
the C library for all Python memory allocations using @code{PYTHONMALLOC=malloc}.
This is helpful when using external memory debuggers like Valgrind on
a Python compiled in release mode.

On error, the debug hooks on Python memory allocators now use the
@ref{11b,,tracemalloc} module to get the traceback where a memory block was
allocated.

Example of fatal error on buffer overflow using
@code{python3.6 -X tracemalloc=5} (store 5 frames in traces):

@example
Debug memory block at address p=0x7fbcd41666f8: API 'o'
    4 bytes originally requested
    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
    The 8 pad bytes at tail=0x7fbcd41666fc are not all FORBIDDENBYTE (0xfb):
        at tail+0: 0x02 *** OUCH
        at tail+1: 0xfb
        at tail+2: 0xfb
        at tail+3: 0xfb
        at tail+4: 0xfb
        at tail+5: 0xfb
        at tail+6: 0xfb
        at tail+7: 0xfb
    The block was made by call #1233329 to debug malloc/realloc.
    Data at p: 1a 2b 30 00

Memory block allocated at (most recent call first):
  File "test/test_bytes.py", line 323
  File "unittest/case.py", line 600
  File "unittest/case.py", line 648
  File "unittest/suite.py", line 122
  File "unittest/suite.py", line 84

Fatal Python error: bad trailing pad byte

Current thread 0x00007fbcdbd32700 (most recent call first):
  File "test/test_bytes.py", line 323 in test_hex
  File "unittest/case.py", line 600 in run
  File "unittest/case.py", line 648 in __call__
  File "unittest/suite.py", line 122 in run
  File "unittest/suite.py", line 84 in __call__
  File "unittest/suite.py", line 122 in run
  File "unittest/suite.py", line 84 in __call__
  ...
@end example

(Contributed by Victor Stinner in bpo-26516@footnote{https://bugs.python.org/issue26516} and bpo-26564@footnote{https://bugs.python.org/issue26564}.)

@node DTrace and SystemTap probing support,,PYTHONMALLOC environment variable,New Features<9>
@anchor{whatsnew/3 6 dtrace-and-systemtap-probing-support}@anchor{7b1}@anchor{whatsnew/3 6 whatsnew36-tracing}@anchor{776}
@subsubsection DTrace and SystemTap probing support


Python can now be built @code{--with-dtrace} which enables static markers
for the following events in the interpreter:


@itemize *

@item 
function call/return

@item 
garbage collection started/finished

@item 
line of code executed.
@end itemize

This can be used to instrument running interpreters in production,
without the need to recompile specific @ref{338,,debug builds} or
providing application-specific profiling/debugging code.

More details in @ref{7b2,,Instrumenting CPython with DTrace and SystemTap}.

The current implementation is tested on Linux and macOS.  Additional
markers may be added in the future.

(Contributed by Łukasz Langa in bpo-21590@footnote{https://bugs.python.org/issue21590}, based on patches by
Jesús Cea Avión, David Malcolm, and Nikhil Benesch.)

@node Other Language Changes<6>,New Modules<6>,New Features<9>,What’s New In Python 3 6
@anchor{whatsnew/3 6 other-language-changes}@anchor{7b3}
@subsection Other Language Changes


Some smaller changes made to the core Python language are:


@itemize *

@item 
A @code{global} or @code{nonlocal} statement must now textually appear
before the first use of the affected name in the same scope.
Previously this was a @ref{4c7,,SyntaxWarning}.

@item 
It is now possible to set a @ref{7b4,,special method} to
@code{None} to indicate that the corresponding operation is not available.
For example, if a class sets @code{__iter__()} to @code{None}, the class
is not iterable.
(Contributed by Andrew Barnert and Ivan Levkivskyi in bpo-25958@footnote{https://bugs.python.org/issue25958}.)

@item 
Long sequences of repeated traceback lines are now abbreviated as
@code{"[Previous line repeated @{count@} more times]"} (see
@ref{7b5,,traceback} for an example).
(Contributed by Emanuel Barry in bpo-26823@footnote{https://bugs.python.org/issue26823}.)

@item 
Import now raises the new exception @ref{66b,,ModuleNotFoundError}
(subclass of @ref{3b8,,ImportError}) when it cannot find a module.  Code
that currently checks for ImportError (in try-except) will still work.
(Contributed by Eric Snow in bpo-15767@footnote{https://bugs.python.org/issue15767}.)

@item 
Class methods relying on zero-argument @code{super()} will now work correctly
when called from metaclass methods during class creation.
(Contributed by Martin Teichmann in bpo-23722@footnote{https://bugs.python.org/issue23722}.)
@end itemize

@node New Modules<6>,Improved Modules<6>,Other Language Changes<6>,What’s New In Python 3 6
@anchor{whatsnew/3 6 new-modules}@anchor{7b6}
@subsection New Modules


@menu
* secrets:: 

@end menu

@node secrets,,,New Modules<6>
@anchor{whatsnew/3 6 secrets}@anchor{7b7}@anchor{whatsnew/3 6 whatsnew36-pep506}@anchor{771}
@subsubsection secrets


The main purpose of the new @ref{df,,secrets} module is to provide an obvious way
to reliably generate cryptographically strong pseudo-random values suitable
for managing secrets, such as account authentication, tokens, and similar.

@cartouche
@quotation Warning 
Note that the pseudo-random generators in the @ref{d7,,random} module
should @emph{NOT} be used for security purposes.  Use @ref{df,,secrets}
on Python 3.6+ and @ref{17f,,os.urandom()} on Python 3.5 and earlier.
@end quotation
@end cartouche


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 506

@item PEP 506@footnote{https://www.python.org/dev/peps/pep-0506} – Adding A Secrets Module To The Standard Library

PEP written and implemented by Steven D’Aprano.
@end table


@node Improved Modules<6>,Optimizations<6>,New Modules<6>,What’s New In Python 3 6
@anchor{whatsnew/3 6 improved-modules}@anchor{7b8}
@subsection Improved Modules


@menu
* array: array<2>. 
* ast: ast<3>. 
* asyncio: asyncio<6>. 
* binascii: binascii<2>. 
* cmath:: 
* collections: collections<4>. 
* concurrent.futures: concurrent futures<3>. 
* contextlib: contextlib<3>. 
* datetime: datetime<4>. 
* decimal: decimal<2>. 
* distutils: distutils<4>. 
* email:: 
* encodings: encodings<2>. 
* enum: enum<4>. 
* faulthandler: faulthandler<2>. 
* fileinput: fileinput<2>. 
* hashlib: hashlib<3>. 
* http.client: http client<2>. 
* idlelib and IDLE: idlelib and IDLE<2>. 
* importlib: importlib<4>. 
* inspect: inspect<5>. 
* json:: 
* logging: logging<3>. 
* math: math<5>. 
* multiprocessing: multiprocessing<4>. 
* os: os<6>. 
* pathlib: pathlib<5>. 
* pdb: pdb<3>. 
* pickle: pickle<2>. 
* pickletools:: 
* pydoc: pydoc<3>. 
* random: random<2>. 
* re: re<2>. 
* readline:: 
* rlcompleter:: 
* shlex: shlex<2>. 
* site: site<2>. 
* sqlite3: sqlite3<4>. 
* socket: socket<7>. 
* socketserver: socketserver<2>. 
* ssl: ssl<5>. 
* statistics: statistics<3>. 
* struct:: 
* subprocess: subprocess<2>. 
* sys: sys<7>. 
* telnetlib:: 
* time: time<5>. 
* timeit:: 
* tkinter: tkinter<3>. 
* traceback: traceback<2>. 
* tracemalloc: tracemalloc<3>. 
* typing: typing<5>. 
* unicodedata: unicodedata<5>. 
* unittest.mock: unittest mock<2>. 
* urllib.request: urllib request. 
* urllib.robotparser: urllib robotparser. 
* venv: venv<3>. 
* warnings: warnings<2>. 
* winreg:: 
* winsound:: 
* xmlrpc.client: xmlrpc client. 
* zipfile: zipfile<2>. 
* zlib:: 

@end menu

@node array<2>,ast<3>,,Improved Modules<6>
@anchor{whatsnew/3 6 array}@anchor{7b9}
@subsubsection array


Exhausted iterators of @ref{25e,,array.array} will now stay exhausted even
if the iterated array is extended.  This is consistent with the behavior
of other mutable sequences.

Contributed by Serhiy Storchaka in bpo-26492@footnote{https://bugs.python.org/issue26492}.

@node ast<3>,asyncio<6>,array<2>,Improved Modules<6>
@anchor{whatsnew/3 6 ast}@anchor{7ba}
@subsubsection ast


The new @ref{588,,ast.Constant} AST node has been added.  It can be used
by external AST optimizers for the purposes of constant folding.

Contributed by Victor Stinner in bpo-26146@footnote{https://bugs.python.org/issue26146}.

@node asyncio<6>,binascii<2>,ast<3>,Improved Modules<6>
@anchor{whatsnew/3 6 asyncio}@anchor{7bb}
@subsubsection asyncio


Starting with Python 3.6 the @code{asyncio} module is no longer provisional and its
API is considered stable.

Notable changes in the @ref{9,,asyncio} module since Python 3.5.0
(all backported to 3.5.x due to the provisional status):


@itemize *

@item 
The @ref{314,,get_event_loop()} function has been changed to
always return the currently running loop when called from coroutines
and callbacks.
(Contributed by Yury Selivanov in bpo-28613@footnote{https://bugs.python.org/issue28613}.)

@item 
The @ref{318,,ensure_future()} function and all functions that
use it, such as @ref{7bc,,loop.run_until_complete()},
now accept all kinds of @ref{7bd,,awaitable objects}.
(Contributed by Yury Selivanov.)

@item 
New @ref{348,,run_coroutine_threadsafe()} function to submit
coroutines to event loops from other threads.
(Contributed by Vincent Michel.)

@item 
New @ref{7be,,Transport.is_closing()}
method to check if the transport is closing or closed.
(Contributed by Yury Selivanov.)

@item 
The @ref{630,,loop.create_server()}
method can now accept a list of hosts.
(Contributed by Yann Sionneau.)

@item 
New @ref{7bf,,loop.create_future()}
method to create Future objects.  This allows alternative event
loop implementations, such as
uvloop@footnote{https://github.com/MagicStack/uvloop}, to provide a faster
@ref{316,,asyncio.Future} implementation.
(Contributed by Yury Selivanov in bpo-27041@footnote{https://bugs.python.org/issue27041}.)

@item 
New @ref{7c0,,loop.get_exception_handler()}
method to get the current exception handler.
(Contributed by Yury Selivanov in bpo-27040@footnote{https://bugs.python.org/issue27040}.)

@item 
New @ref{7c1,,StreamReader.readuntil()}
method to read data from the stream until a separator bytes
sequence appears.
(Contributed by Mark Korenberg.)

@item 
The performance of @ref{7c2,,StreamReader.readexactly()}
has been improved.
(Contributed by Mark Korenberg in bpo-28370@footnote{https://bugs.python.org/issue28370}.)

@item 
The @ref{750,,loop.getaddrinfo()}
method is optimized to avoid calling the system @code{getaddrinfo}
function if the address is already resolved.
(Contributed by A. Jesse Jiryu Davis.)

@item 
The @ref{7c3,,loop.stop()}
method has been changed to stop the loop immediately after
the current iteration.  Any new callbacks scheduled as a result
of the last iteration will be discarded.
(Contributed by Guido van Rossum in bpo-25593@footnote{https://bugs.python.org/issue25593}.)

@item 
@code{Future.set_exception}
will now raise @ref{162,,TypeError} when passed an instance of
the @ref{736,,StopIteration} exception.
(Contributed by Chris Angelico in bpo-26221@footnote{https://bugs.python.org/issue26221}.)

@item 
New @ref{638,,loop.connect_accepted_socket()}
method to be used by servers that accept connections outside of asyncio,
but that use asyncio to handle them.
(Contributed by Jim Fulton in bpo-27392@footnote{https://bugs.python.org/issue27392}.)

@item 
@code{TCP_NODELAY} flag is now set for all TCP transports by default.
(Contributed by Yury Selivanov in bpo-27456@footnote{https://bugs.python.org/issue27456}.)

@item 
New @ref{7c4,,loop.shutdown_asyncgens()}
to properly close pending asynchronous generators before closing the
loop.
(Contributed by Yury Selivanov in bpo-28003@footnote{https://bugs.python.org/issue28003}.)

@item 
@ref{316,,Future} and @ref{317,,Task}
classes now have an optimized C implementation which makes asyncio
code up to 30% faster.
(Contributed by Yury Selivanov and INADA Naoki in bpo-26081@footnote{https://bugs.python.org/issue26081}
and bpo-28544@footnote{https://bugs.python.org/issue28544}.)
@end itemize

@node binascii<2>,cmath,asyncio<6>,Improved Modules<6>
@anchor{whatsnew/3 6 binascii}@anchor{7c5}
@subsubsection binascii


The @ref{7c6,,b2a_base64()} function now accepts an optional @emph{newline}
keyword argument to control whether the newline character is appended to the
return value.
(Contributed by Victor Stinner in bpo-25357@footnote{https://bugs.python.org/issue25357}.)

@node cmath,collections<4>,binascii<2>,Improved Modules<6>
@anchor{whatsnew/3 6 cmath}@anchor{7c7}
@subsubsection cmath


The new @ref{7c8,,cmath.tau} (@emph{τ}) constant has been added.
(Contributed by Lisa Roach in bpo-12345@footnote{https://bugs.python.org/issue12345}, see 
@geindex Python Enhancement Proposals; PEP 628
PEP 628@footnote{https://www.python.org/dev/peps/pep-0628} for details.)

New constants: @ref{7c9,,cmath.inf} and @ref{7ca,,cmath.nan} to
match @ref{7cb,,math.inf} and @ref{7cc,,math.nan}, and also @ref{7cd,,cmath.infj}
and @ref{7ce,,cmath.nanj} to match the format used by complex repr.
(Contributed by Mark Dickinson in bpo-23229@footnote{https://bugs.python.org/issue23229}.)

@node collections<4>,concurrent futures<3>,cmath,Improved Modules<6>
@anchor{whatsnew/3 6 collections}@anchor{7cf}
@subsubsection collections


The new @ref{7d0,,Collection} abstract base class has been
added to represent sized iterable container classes.
(Contributed by Ivan Levkivskyi, docs by Neil Girdhar in bpo-27598@footnote{https://bugs.python.org/issue27598}.)

The new @ref{7d1,,Reversible} abstract base class represents
iterable classes that also provide the @code{__reversed__()} method.
(Contributed by Ivan Levkivskyi in bpo-25987@footnote{https://bugs.python.org/issue25987}.)

The new @ref{7d2,,AsyncGenerator} abstract base class represents
asynchronous generators.
(Contributed by Yury Selivanov in bpo-28720@footnote{https://bugs.python.org/issue28720}.)

The @ref{3e4,,namedtuple()} function now accepts an optional
keyword argument @emph{module}, which, when specified, is used for
the @code{__module__} attribute of the returned named tuple class.
(Contributed by Raymond Hettinger in bpo-17941@footnote{https://bugs.python.org/issue17941}.)

The @emph{verbose} and @emph{rename} arguments for
@ref{3e4,,namedtuple()} are now keyword-only.
(Contributed by Raymond Hettinger in bpo-25628@footnote{https://bugs.python.org/issue25628}.)

Recursive @ref{7d3,,collections.deque} instances can now be pickled.
(Contributed by Serhiy Storchaka in bpo-26482@footnote{https://bugs.python.org/issue26482}.)

@node concurrent futures<3>,contextlib<3>,collections<4>,Improved Modules<6>
@anchor{whatsnew/3 6 concurrent-futures}@anchor{7d4}
@subsubsection concurrent.futures


The @ref{1c7,,ThreadPoolExecutor}
class constructor now accepts an optional @emph{thread_name_prefix} argument
to make it possible to customize the names of the threads created by the
pool.
(Contributed by Gregory P. Smith in bpo-27664@footnote{https://bugs.python.org/issue27664}.)

@node contextlib<3>,datetime<4>,concurrent futures<3>,Improved Modules<6>
@anchor{whatsnew/3 6 contextlib}@anchor{7d5}
@subsubsection contextlib


The @ref{7d6,,contextlib.AbstractContextManager} class has been added to
provide an abstract base class for context managers.  It provides a
sensible default implementation for @cite{__enter__()} which returns
@code{self} and leaves @cite{__exit__()} an abstract method.  A matching
class has been added to the @ref{120,,typing} module as
@ref{7d7,,typing.ContextManager}.
(Contributed by Brett Cannon in bpo-25609@footnote{https://bugs.python.org/issue25609}.)

@node datetime<4>,decimal<2>,contextlib<3>,Improved Modules<6>
@anchor{whatsnew/3 6 datetime}@anchor{7d8}
@subsubsection datetime


The @ref{363,,datetime} and @ref{364,,time} classes have
the new @code{fold} attribute used to disambiguate local time
when necessary.  Many functions in the @ref{2f,,datetime} have been
updated to support local time disambiguation.
See @ref{779,,Local Time Disambiguation} section for more
information.
(Contributed by Alexander Belopolsky in bpo-24773@footnote{https://bugs.python.org/issue24773}.)

The @ref{7d9,,datetime.strftime()} and
@ref{7da,,date.strftime()} methods now support
ISO 8601 date directives @code{%G}, @code{%u} and @code{%V}.
(Contributed by Ashley Anderson in bpo-12006@footnote{https://bugs.python.org/issue12006}.)

The @ref{651,,datetime.isoformat()} function
now accepts an optional @emph{timespec} argument that specifies the number
of additional components of the time value to include.
(Contributed by Alessandro Cucci and Alexander Belopolsky in bpo-19475@footnote{https://bugs.python.org/issue19475}.)

The @ref{7db,,datetime.combine()} now
accepts an optional @emph{tzinfo} argument.
(Contributed by Alexander Belopolsky in bpo-27661@footnote{https://bugs.python.org/issue27661}.)

@node decimal<2>,distutils<4>,datetime<4>,Improved Modules<6>
@anchor{whatsnew/3 6 decimal}@anchor{7dc}
@subsubsection decimal


New @ref{7dd,,Decimal.as_integer_ratio()}
method that returns a pair @code{(n, d)} of integers that represent the given
@ref{248,,Decimal} instance as a fraction, in lowest terms and
with a positive denominator:

@example
>>> Decimal('-3.14').as_integer_ratio()
(-157, 50)
@end example

(Contributed by Stefan Krah amd Mark Dickinson in bpo-25928@footnote{https://bugs.python.org/issue25928}.)

@node distutils<4>,email,decimal<2>,Improved Modules<6>
@anchor{whatsnew/3 6 distutils}@anchor{7de}
@subsubsection distutils


The @code{default_format} attribute has been removed from
@code{distutils.command.sdist.sdist} and the @code{formats}
attribute defaults to @code{['gztar']}. Although not anticipated,
any code relying on the presence of @code{default_format} may
need to be adapted. See bpo-27819@footnote{https://bugs.python.org/issue27819} for more details.

@node email,encodings<2>,distutils<4>,Improved Modules<6>
@anchor{whatsnew/3 6 email}@anchor{7df}
@subsubsection email


The new email API, enabled via the @emph{policy} keyword to various constructors, is
no longer provisional.  The @ref{64,,email} documentation has been reorganized and
rewritten to focus on the new API, while retaining the old documentation for
the legacy API.  (Contributed by R. David Murray in bpo-24277@footnote{https://bugs.python.org/issue24277}.)

The @ref{6e,,email.mime} classes now all accept an optional @emph{policy} keyword.
(Contributed by Berker Peksag in bpo-27331@footnote{https://bugs.python.org/issue27331}.)

The @ref{7e0,,DecodedGenerator} now supports the @emph{policy}
keyword.

There is a new @ref{70,,policy} attribute,
@ref{7e1,,message_factory}, that controls what class is used
by default when the parser creates new message objects.  For the
@ref{7e2,,email.policy.compat32} policy this is @ref{7e3,,Message},
for the new policies it is @ref{7e4,,EmailMessage}.
(Contributed by R. David Murray in bpo-20476@footnote{https://bugs.python.org/issue20476}.)

@node encodings<2>,enum<4>,email,Improved Modules<6>
@anchor{whatsnew/3 6 encodings}@anchor{7e5}
@subsubsection encodings


On Windows, added the @code{'oem'} encoding to use @code{CP_OEMCP}, and the @code{'ansi'}
alias for the existing @code{'mbcs'} encoding, which uses the @code{CP_ACP} code page.
(Contributed by Steve Dower in bpo-27959@footnote{https://bugs.python.org/issue27959}.)

@node enum<4>,faulthandler<2>,encodings<2>,Improved Modules<6>
@anchor{whatsnew/3 6 enum}@anchor{7e6}
@subsubsection enum


Two new enumeration base classes have been added to the @ref{76,,enum} module:
@ref{7e7,,Flag} and @code{IntFlags}.  Both are used to define
constants that can be combined using the bitwise operators.
(Contributed by Ethan Furman in bpo-23591@footnote{https://bugs.python.org/issue23591}.)

Many standard library modules have been updated to use the
@code{IntFlags} class for their constants.

The new @ref{7e8,,enum.auto} value can be used to assign values to enum
members automatically:

@example
>>> from enum import Enum, auto
>>> class Color(Enum):
...     red = auto()
...     blue = auto()
...     green = auto()
...
>>> list(Color)
[<Color.red: 1>, <Color.blue: 2>, <Color.green: 3>]
@end example

@node faulthandler<2>,fileinput<2>,enum<4>,Improved Modules<6>
@anchor{whatsnew/3 6 faulthandler}@anchor{7e9}
@subsubsection faulthandler


On Windows, the @ref{78,,faulthandler} module now installs a handler for Windows
exceptions: see @ref{7ea,,faulthandler.enable()}. (Contributed by Victor Stinner in
bpo-23848@footnote{https://bugs.python.org/issue23848}.)

@node fileinput<2>,hashlib<3>,faulthandler<2>,Improved Modules<6>
@anchor{whatsnew/3 6 fileinput}@anchor{7eb}
@subsubsection fileinput


@ref{7ec,,hook_encoded()} now supports the @emph{errors} argument.
(Contributed by Joseph Hackman in bpo-25788@footnote{https://bugs.python.org/issue25788}.)

@node hashlib<3>,http client<2>,fileinput<2>,Improved Modules<6>
@anchor{whatsnew/3 6 hashlib}@anchor{7ed}
@subsubsection hashlib


@ref{88,,hashlib} supports OpenSSL 1.1.0.  The minimum recommend version is 1.0.2.
(Contributed by Christian Heimes in bpo-26470@footnote{https://bugs.python.org/issue26470}.)

BLAKE2 hash functions were added to the module. @ref{7ee,,blake2b()}
and @ref{7ef,,blake2s()} are always available and support the full
feature set of BLAKE2.
(Contributed by Christian Heimes in bpo-26798@footnote{https://bugs.python.org/issue26798} based on code by
Dmitry Chestnykh and Samuel Neves. Documentation written by Dmitry Chestnykh.)

The SHA-3 hash functions @code{sha3_224()}, @code{sha3_256()},
@code{sha3_384()}, @code{sha3_512()}, and SHAKE hash functions
@code{shake_128()} and @code{shake_256()} were added.
(Contributed by Christian Heimes in bpo-16113@footnote{https://bugs.python.org/issue16113}. Keccak Code Package
by Guido Bertoni, Joan Daemen, Michaël Peeters, Gilles Van Assche, and
Ronny Van Keer.)

The password-based key derivation function @ref{77c,,scrypt()} is now
available with OpenSSL 1.1.0 and newer.
(Contributed by Christian Heimes in bpo-27928@footnote{https://bugs.python.org/issue27928}.)

@node http client<2>,idlelib and IDLE<2>,hashlib<3>,Improved Modules<6>
@anchor{whatsnew/3 6 http-client}@anchor{7f0}
@subsubsection http.client


@ref{7f1,,HTTPConnection.request()} and
@ref{7f2,,endheaders()} both now support
chunked encoding request bodies.
(Contributed by Demian Brecht and Rolf Krahl in bpo-12319@footnote{https://bugs.python.org/issue12319}.)

@node idlelib and IDLE<2>,importlib<4>,http client<2>,Improved Modules<6>
@anchor{whatsnew/3 6 idlelib-and-idle}@anchor{7f3}
@subsubsection idlelib and IDLE


The idlelib package is being modernized and refactored to make IDLE look and
work better and to make the code easier to understand, test, and improve. Part
of making IDLE look better, especially on Linux and Mac, is using ttk widgets,
mostly in the dialogs.  As a result, IDLE no longer runs with tcl/tk 8.4.  It
now requires tcl/tk 8.5 or 8.6.  We recommend running the latest release of
either.

‘Modernizing’ includes renaming and consolidation of idlelib modules. The
renaming of files with partial uppercase names is similar to the renaming of,
for instance, Tkinter and TkFont to tkinter and tkinter.font in 3.0.  As a
result, imports of idlelib files that worked in 3.5 will usually not work in
3.6.  At least a module name change will be needed (see idlelib/README.txt),
sometimes more.  (Name changes contributed by Al Swiegart and Terry Reedy in
bpo-24225@footnote{https://bugs.python.org/issue24225}.  Most idlelib patches since have been and will be part of the
process.)

In compensation, the eventual result with be that some idlelib classes will be
easier to use, with better APIs and docstrings explaining them.  Additional
useful information will be added to idlelib when available.

New in 3.6.2:

Multiple fixes for autocompletion. (Contributed by Louie Lu in bpo-15786@footnote{https://bugs.python.org/issue15786}.)

New in 3.6.3:

Module Browser (on the File menu, formerly called Class Browser),
now displays nested functions and classes in addition to top-level
functions and classes.
(Contributed by Guilherme Polo, Cheryl Sabella, and Terry Jan Reedy
in bpo-1612262@footnote{https://bugs.python.org/issue1612262}.)

The IDLE features formerly implemented as extensions have been reimplemented
as normal features.  Their settings have been moved from the Extensions tab
to other dialog tabs.
(Contributed by Charles Wohlganger and Terry Jan Reedy in bpo-27099@footnote{https://bugs.python.org/issue27099}.)

The Settings dialog (Options, Configure IDLE) has been partly rewritten
to improve both appearance and function.
(Contributed by Cheryl Sabella and Terry Jan Reedy in multiple issues.)

New in 3.6.4:

The font sample now includes a selection of non-Latin characters so that
users can better see the effect of selecting a particular font.
(Contributed by Terry Jan Reedy in bpo-13802@footnote{https://bugs.python.org/issue13802}.)
The sample can be edited to include other characters.
(Contributed by Serhiy Storchaka in bpo-31860@footnote{https://bugs.python.org/issue31860}.)

New in 3.6.6:

Editor code context option revised.  Box displays all context lines up to
maxlines.  Clicking on a context line jumps the editor to that line.  Context
colors for custom themes is added to Highlights tab of Settings dialog.
(Contributed by Cheryl Sabella and Terry Jan Reedy in bpo-33642@footnote{https://bugs.python.org/issue33642},
bpo-33768@footnote{https://bugs.python.org/issue33768}, and bpo-33679@footnote{https://bugs.python.org/issue33679}.)

On Windows, a new API call tells Windows that tk scales for DPI. On Windows
8.1+ or 10, with DPI compatibility properties of the Python binary
unchanged, and a monitor resolution greater than 96 DPI, this should
make text and lines sharper.  It should otherwise have no effect.
(Contributed by Terry Jan Reedy in bpo-33656@footnote{https://bugs.python.org/issue33656}.)

New in 3.6.7:

Output over N lines (50 by default) is squeezed down to a button.
N can be changed in the PyShell section of the General page of the
Settings dialog.  Fewer, but possibly extra long, lines can be squeezed by
right clicking on the output.  Squeezed output can be expanded in place
by double-clicking the button or into the clipboard or a separate window
by right-clicking the button.  (Contributed by Tal Einat in bpo-1529353@footnote{https://bugs.python.org/issue1529353}.)

@node importlib<4>,inspect<5>,idlelib and IDLE<2>,Improved Modules<6>
@anchor{whatsnew/3 6 importlib}@anchor{7f4}
@subsubsection importlib


Import now raises the new exception @ref{66b,,ModuleNotFoundError}
(subclass of @ref{3b8,,ImportError}) when it cannot find a module.  Code
that current checks for @code{ImportError} (in try-except) will still work.
(Contributed by Eric Snow in bpo-15767@footnote{https://bugs.python.org/issue15767}.)

@ref{7f5,,importlib.util.LazyLoader} now calls
@ref{7f6,,create_module()} on the wrapped loader, removing the
restriction that @ref{7f7,,importlib.machinery.BuiltinImporter} and
@ref{7f8,,importlib.machinery.ExtensionFileLoader} couldn’t be used with
@ref{7f5,,importlib.util.LazyLoader}.

@ref{7f9,,importlib.util.cache_from_source()},
@ref{7fa,,importlib.util.source_from_cache()}, and
@ref{7fb,,importlib.util.spec_from_file_location()} now accept a
@ref{63d,,path-like object}.

@node inspect<5>,json,importlib<4>,Improved Modules<6>
@anchor{whatsnew/3 6 inspect}@anchor{7fc}
@subsubsection inspect


The @ref{1c1,,inspect.signature()} function now reports the
implicit @code{.0} parameters generated by the compiler for comprehension and
generator expression scopes as if they were positional-only parameters called
@code{implicit0}. (Contributed by Jelle Zijlstra in bpo-19611@footnote{https://bugs.python.org/issue19611}.)

To reduce code churn when upgrading from Python 2.7 and the legacy
@code{inspect.getargspec()} API, the previously documented deprecation of
@ref{1c2,,inspect.getfullargspec()} has been reversed. While this function is
convenient for single/source Python 2/3 code bases, the richer
@ref{1c1,,inspect.signature()} interface remains the recommended approach for new
code. (Contributed by Nick Coghlan in bpo-27172@footnote{https://bugs.python.org/issue27172})

@node json,logging<3>,inspect<5>,Improved Modules<6>
@anchor{whatsnew/3 6 json}@anchor{7fd}
@subsubsection json


@ref{7fe,,json.load()} and @ref{456,,json.loads()} now support binary input.  Encoded
JSON should be represented using either UTF-8, UTF-16, or UTF-32.
(Contributed by Serhiy Storchaka in bpo-17909@footnote{https://bugs.python.org/issue17909}.)

@node logging<3>,math<5>,json,Improved Modules<6>
@anchor{whatsnew/3 6 logging}@anchor{7ff}
@subsubsection logging


The new @ref{800,,WatchedFileHandler.reopenIfNeeded()}
method has been added to add the ability to check if the log file needs to
be reopened.
(Contributed by Marian Horban in bpo-24884@footnote{https://bugs.python.org/issue24884}.)

@node math<5>,multiprocessing<4>,logging<3>,Improved Modules<6>
@anchor{whatsnew/3 6 math}@anchor{801}
@subsubsection math


The tau (@emph{τ}) constant has been added to the @ref{ad,,math} and @ref{17,,cmath}
modules.
(Contributed by Lisa Roach in bpo-12345@footnote{https://bugs.python.org/issue12345}, see 
@geindex Python Enhancement Proposals; PEP 628
PEP 628@footnote{https://www.python.org/dev/peps/pep-0628} for details.)

@node multiprocessing<4>,os<6>,math<5>,Improved Modules<6>
@anchor{whatsnew/3 6 multiprocessing}@anchor{802}
@subsubsection multiprocessing


@ref{803,,Proxy Objects} returned by
@code{multiprocessing.Manager()} can now be nested.
(Contributed by Davin Potts in bpo-6766@footnote{https://bugs.python.org/issue6766}.)

@node os<6>,pathlib<5>,multiprocessing<4>,Improved Modules<6>
@anchor{whatsnew/3 6 os}@anchor{804}
@subsubsection os


See the summary of @ref{778,,PEP 519} for details on how the
@ref{c0,,os} and @ref{c1,,os.path} modules now support
@ref{63d,,path-like objects}.

@ref{576,,scandir()} now supports @ref{2f0,,bytes} paths on Windows.

A new @ref{805,,close()} method allows explicitly closing a
@ref{576,,scandir()} iterator.  The @ref{576,,scandir()} iterator now
supports the @ref{166,,context manager} protocol.  If a @code{scandir()}
iterator is neither exhausted nor explicitly closed a @ref{77b,,ResourceWarning}
will be emitted in its destructor.
(Contributed by Serhiy Storchaka in bpo-25994@footnote{https://bugs.python.org/issue25994}.)

On Linux, @ref{17f,,os.urandom()} now blocks until the system urandom entropy pool
is initialized to increase the security. See the 
@geindex Python Enhancement Proposals; PEP 524
PEP 524@footnote{https://www.python.org/dev/peps/pep-0524} for the rationale.

The Linux @code{getrandom()} syscall (get random bytes) is now exposed as the new
@ref{806,,os.getrandom()} function.
(Contributed by Victor Stinner, part of the 
@geindex Python Enhancement Proposals; PEP 524
PEP 524@footnote{https://www.python.org/dev/peps/pep-0524})

@node pathlib<5>,pdb<3>,os<6>,Improved Modules<6>
@anchor{whatsnew/3 6 pathlib}@anchor{807}
@subsubsection pathlib


@ref{c3,,pathlib} now supports @ref{63d,,path-like objects}.
(Contributed by Brett Cannon in bpo-27186@footnote{https://bugs.python.org/issue27186}.)

See the summary of @ref{778,,PEP 519} for details.

@node pdb<3>,pickle<2>,pathlib<5>,Improved Modules<6>
@anchor{whatsnew/3 6 pdb}@anchor{808}
@subsubsection pdb


The @ref{416,,Pdb} class constructor has a new optional @emph{readrc} argument
to control whether @code{.pdbrc} files should be read.

@node pickle<2>,pickletools,pdb<3>,Improved Modules<6>
@anchor{whatsnew/3 6 pickle}@anchor{809}
@subsubsection pickle


Objects that need @code{__new__} called with keyword arguments can now be pickled
using @ref{80a,,pickle protocols} older than protocol version 4.
Protocol version 4 already supports this case.  (Contributed by Serhiy
Storchaka in bpo-24164@footnote{https://bugs.python.org/issue24164}.)

@node pickletools,pydoc<3>,pickle<2>,Improved Modules<6>
@anchor{whatsnew/3 6 pickletools}@anchor{80b}
@subsubsection pickletools


@ref{80c,,pickletools.dis()} now outputs the implicit memo index for the
@code{MEMOIZE} opcode.
(Contributed by Serhiy Storchaka in bpo-25382@footnote{https://bugs.python.org/issue25382}.)

@node pydoc<3>,random<2>,pickletools,Improved Modules<6>
@anchor{whatsnew/3 6 pydoc}@anchor{80d}
@subsubsection pydoc


The @ref{d4,,pydoc} module has learned to respect the @code{MANPAGER}
environment variable.
(Contributed by Matthias Klose in bpo-8637@footnote{https://bugs.python.org/issue8637}.)

@ref{3bc,,help()} and @ref{d4,,pydoc} can now list named tuple fields in the
order they were defined rather than alphabetically.
(Contributed by Raymond Hettinger in bpo-24879@footnote{https://bugs.python.org/issue24879}.)

@node random<2>,re<2>,pydoc<3>,Improved Modules<6>
@anchor{whatsnew/3 6 random}@anchor{80e}
@subsubsection random


The new @ref{80f,,choices()} function returns a list of elements of
specified size from the given population with optional weights.
(Contributed by Raymond Hettinger in bpo-18844@footnote{https://bugs.python.org/issue18844}.)

@node re<2>,readline,random<2>,Improved Modules<6>
@anchor{whatsnew/3 6 re}@anchor{810}
@subsubsection re


Added support of modifier spans in regular expressions.  Examples:
@code{'(?i:p)ython'} matches @code{'python'} and @code{'Python'}, but not @code{'PYTHON'};
@code{'(?i)g(?-i:v)r'} matches @code{'GvR'} and @code{'gvr'}, but not @code{'GVR'}.
(Contributed by Serhiy Storchaka in bpo-433028@footnote{https://bugs.python.org/issue433028}.)

Match object groups can be accessed by @code{__getitem__}, which is
equivalent to @code{group()}.  So @code{mo['name']} is now equivalent to
@code{mo.group('name')}.  (Contributed by Eric Smith in bpo-24454@footnote{https://bugs.python.org/issue24454}.)

@code{Match} objects now support
@ref{24a,,index-like objects} as group
indices.
(Contributed by Jeroen Demeyer and Xiang Zhang in bpo-27177@footnote{https://bugs.python.org/issue27177}.)

@node readline,rlcompleter,re<2>,Improved Modules<6>
@anchor{whatsnew/3 6 readline}@anchor{811}
@subsubsection readline


Added @ref{812,,set_auto_history()} to enable or disable
automatic addition of input to the history list.  (Contributed by
Tyler Crompton in bpo-26870@footnote{https://bugs.python.org/issue26870}.)

@node rlcompleter,shlex<2>,readline,Improved Modules<6>
@anchor{whatsnew/3 6 rlcompleter}@anchor{813}
@subsubsection rlcompleter


Private and special attribute names now are omitted unless the prefix starts
with underscores.  A space or a colon is added after some completed keywords.
(Contributed by Serhiy Storchaka in bpo-25011@footnote{https://bugs.python.org/issue25011} and bpo-25209@footnote{https://bugs.python.org/issue25209}.)

@node shlex<2>,site<2>,rlcompleter,Improved Modules<6>
@anchor{whatsnew/3 6 shlex}@anchor{814}
@subsubsection shlex


The @ref{815,,shlex} has much
@ref{816,,improved shell compatibility}
through the new @emph{punctuation_chars} argument to control which characters
are treated as punctuation.
(Contributed by Vinay Sajip in bpo-1521950@footnote{https://bugs.python.org/issue1521950}.)

@node site<2>,sqlite3<4>,shlex<2>,Improved Modules<6>
@anchor{whatsnew/3 6 site}@anchor{817}
@subsubsection site


When specifying paths to add to @ref{489,,sys.path} in a @cite{.pth} file,
you may now specify file paths on top of directories (e.g. zip files).
(Contributed by Wolfgang Langner in bpo-26587@footnote{https://bugs.python.org/issue26587}).

@node sqlite3<4>,socket<7>,site<2>,Improved Modules<6>
@anchor{whatsnew/3 6 sqlite3}@anchor{818}
@subsubsection sqlite3


@ref{819,,sqlite3.Cursor.lastrowid} now supports the @code{REPLACE} statement.
(Contributed by Alex LordThorsen in bpo-16864@footnote{https://bugs.python.org/issue16864}.)

@node socket<7>,socketserver<2>,sqlite3<4>,Improved Modules<6>
@anchor{whatsnew/3 6 socket}@anchor{81a}
@subsubsection socket


The @ref{81b,,ioctl()} function now supports the
@ref{81c,,SIO_LOOPBACK_FAST_PATH} control code.
(Contributed by Daniel Stokes in bpo-26536@footnote{https://bugs.python.org/issue26536}.)

The @ref{81d,,getsockopt()} constants @code{SO_DOMAIN},
@code{SO_PROTOCOL}, @code{SO_PEERSEC}, and @code{SO_PASSSEC} are now supported.
(Contributed by Christian Heimes in bpo-26907@footnote{https://bugs.python.org/issue26907}.)

The @ref{81e,,setsockopt()} now supports the
@code{setsockopt(level, optname, None, optlen: int)} form.
(Contributed by Christian Heimes in bpo-27744@footnote{https://bugs.python.org/issue27744}.)

The socket module now supports the address family
@ref{81f,,AF_ALG} to interface with Linux Kernel crypto API. @code{ALG_*},
@code{SOL_ALG} and @ref{820,,sendmsg_afalg()} were added.
(Contributed by Christian Heimes in bpo-27744@footnote{https://bugs.python.org/issue27744} with support from
Victor Stinner.)

New Linux constants @code{TCP_USER_TIMEOUT} and @code{TCP_CONGESTION} were added.
(Contributed by Omar Sandoval, bpo-26273@footnote{https://bugs.python.org/issue26273}).

@node socketserver<2>,ssl<5>,socket<7>,Improved Modules<6>
@anchor{whatsnew/3 6 socketserver}@anchor{821}
@subsubsection socketserver


Servers based on the @ref{eb,,socketserver} module, including those
defined in @ref{92,,http.server}, @ref{147,,xmlrpc.server} and
@ref{136,,wsgiref.simple_server}, now support the @ref{166,,context manager}
protocol.
(Contributed by Aviv Palivoda in bpo-26404@footnote{https://bugs.python.org/issue26404}.)

The @code{wfile} attribute of
@ref{822,,StreamRequestHandler} classes now implements
the @ref{823,,io.BufferedIOBase} writable interface.  In particular,
calling @ref{824,,write()} is now guaranteed to send the
data in full.  (Contributed by Martin Panter in bpo-26721@footnote{https://bugs.python.org/issue26721}.)

@node ssl<5>,statistics<3>,socketserver<2>,Improved Modules<6>
@anchor{whatsnew/3 6 ssl}@anchor{825}
@subsubsection ssl


@ref{ee,,ssl} supports OpenSSL 1.1.0.  The minimum recommend version is 1.0.2.
(Contributed by Christian Heimes in bpo-26470@footnote{https://bugs.python.org/issue26470}.)

3DES has been removed from the default cipher suites and ChaCha20 Poly1305
cipher suites have been added.
(Contributed by Christian Heimes in bpo-27850@footnote{https://bugs.python.org/issue27850} and bpo-27766@footnote{https://bugs.python.org/issue27766}.)

@ref{2bf,,SSLContext} has better default configuration for options
and ciphers.
(Contributed by Christian Heimes in bpo-28043@footnote{https://bugs.python.org/issue28043}.)

SSL session can be copied from one client-side connection to another
with the new @ref{826,,SSLSession} class.  TLS session resumption can
speed up the initial handshake, reduce latency and improve performance
(Contributed by Christian Heimes in bpo-19500@footnote{https://bugs.python.org/issue19500} based on a draft by
Alex Warhawk.)

The new @ref{827,,get_ciphers()} method can be used to
get a list of enabled ciphers in order of cipher priority.

All constants and flags have been converted to @ref{828,,IntEnum} and
@code{IntFlags}.
(Contributed by Christian Heimes in bpo-28025@footnote{https://bugs.python.org/issue28025}.)

Server and client-side specific TLS protocols for @ref{2bf,,SSLContext}
were added.
(Contributed by Christian Heimes in bpo-28085@footnote{https://bugs.python.org/issue28085}.)

@node statistics<3>,struct,ssl<5>,Improved Modules<6>
@anchor{whatsnew/3 6 statistics}@anchor{829}
@subsubsection statistics


A new @ref{82a,,harmonic_mean()} function has been added.
(Contributed by Steven D’Aprano in bpo-27181@footnote{https://bugs.python.org/issue27181}.)

@node struct,subprocess<2>,statistics<3>,Improved Modules<6>
@anchor{whatsnew/3 6 struct}@anchor{82b}
@subsubsection struct


@ref{f3,,struct} now supports IEEE 754 half-precision floats via the @code{'e'}
format specifier.
(Contributed by Eli Stevens, Mark Dickinson in bpo-11734@footnote{https://bugs.python.org/issue11734}.)

@node subprocess<2>,sys<7>,struct,Improved Modules<6>
@anchor{whatsnew/3 6 subprocess}@anchor{82c}
@subsubsection subprocess


@ref{5ae,,subprocess.Popen} destructor now emits a @ref{77b,,ResourceWarning} warning
if the child process is still running. Use the context manager protocol (@code{with
proc: ...}) or explicitly call the @ref{82d,,wait()} method to
read the exit status of the child process. (Contributed by Victor Stinner in
bpo-26741@footnote{https://bugs.python.org/issue26741}.)

The @ref{5ae,,subprocess.Popen} constructor and all functions that pass arguments
through to it now accept @emph{encoding} and @emph{errors} arguments. Specifying either
of these will enable text mode for the @emph{stdin}, @emph{stdout} and @emph{stderr} streams.
(Contributed by Steve Dower in bpo-6135@footnote{https://bugs.python.org/issue6135}.)

@node sys<7>,telnetlib,subprocess<2>,Improved Modules<6>
@anchor{whatsnew/3 6 sys}@anchor{82e}
@subsubsection sys


The new @ref{82f,,getfilesystemencodeerrors()} function returns the name of
the error mode used to convert between Unicode filenames and bytes filenames.
(Contributed by Steve Dower in bpo-27781@footnote{https://bugs.python.org/issue27781}.)

On Windows the return value of the @ref{830,,getwindowsversion()} function
now includes the @emph{platform_version} field which contains the accurate major
version, minor version and build number of the current operating system,
rather than the version that is being emulated for the process
(Contributed by Steve Dower in bpo-27932@footnote{https://bugs.python.org/issue27932}.)

@node telnetlib,time<5>,sys<7>,Improved Modules<6>
@anchor{whatsnew/3 6 telnetlib}@anchor{831}
@subsubsection telnetlib


@ref{832,,Telnet} is now a context manager (contributed by
Stéphane Wirtel in bpo-25485@footnote{https://bugs.python.org/issue25485}).

@node time<5>,timeit,telnetlib,Improved Modules<6>
@anchor{whatsnew/3 6 time}@anchor{833}
@subsubsection time


The @ref{834,,struct_time} attributes @code{tm_gmtoff} and
@code{tm_zone} are now available on all platforms.

@node timeit,tkinter<3>,time<5>,Improved Modules<6>
@anchor{whatsnew/3 6 timeit}@anchor{835}
@subsubsection timeit


The new @ref{836,,Timer.autorange()} convenience
method has been added to call @ref{837,,Timer.timeit()}
repeatedly so that the total run time is greater or equal to 200 milliseconds.
(Contributed by Steven D’Aprano in bpo-6422@footnote{https://bugs.python.org/issue6422}.)

@ref{10b,,timeit} now warns when there is substantial (4x) variance
between best and worst times.
(Contributed by Serhiy Storchaka in bpo-23552@footnote{https://bugs.python.org/issue23552}.)

@node tkinter<3>,traceback<2>,timeit,Improved Modules<6>
@anchor{whatsnew/3 6 tkinter}@anchor{838}
@subsubsection tkinter


Added methods @code{trace_add()},
@code{trace_remove()} and @code{trace_info()}
in the @code{tkinter.Variable} class.  They replace old methods
@code{trace_variable()}, @code{trace()},
@code{trace_vdelete()} and
@code{trace_vinfo()} that use obsolete Tcl commands and might
not work in future versions of Tcl.
(Contributed by Serhiy Storchaka in bpo-22115@footnote{https://bugs.python.org/issue22115}).

@node traceback<2>,tracemalloc<3>,tkinter<3>,Improved Modules<6>
@anchor{whatsnew/3 6 traceback}@anchor{839}@anchor{whatsnew/3 6 whatsnew36-traceback}@anchor{7b5}
@subsubsection traceback


Both the traceback module and the interpreter’s builtin exception display now
abbreviate long sequences of repeated lines in tracebacks as shown in the
following example:

@example
>>> def f(): f()
...
>>> f()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in f
  File "<stdin>", line 1, in f
  File "<stdin>", line 1, in f
  [Previous line repeated 995 more times]
RecursionError: maximum recursion depth exceeded
@end example

(Contributed by Emanuel Barry in bpo-26823@footnote{https://bugs.python.org/issue26823}.)

@node tracemalloc<3>,typing<5>,traceback<2>,Improved Modules<6>
@anchor{whatsnew/3 6 tracemalloc}@anchor{83a}
@subsubsection tracemalloc


The @ref{11b,,tracemalloc} module now supports tracing memory allocations in
multiple different address spaces.

The new @ref{83b,,DomainFilter} filter class has been added
to filter block traces by their address space (domain).

(Contributed by Victor Stinner in bpo-26588@footnote{https://bugs.python.org/issue26588}.)

@node typing<5>,unicodedata<5>,tracemalloc<3>,Improved Modules<6>
@anchor{whatsnew/3 6 typing}@anchor{83c}@anchor{whatsnew/3 6 whatsnew36-typing}@anchor{77a}
@subsubsection typing


Since the @ref{120,,typing} module is @ref{621,,provisional},
all changes introduced in Python 3.6 have also been
backported to Python 3.5.x.

The @ref{120,,typing} module has a much improved support for generic type
aliases.  For example @code{Dict[str, Tuple[S, T]]} is now a valid
type annotation.
(Contributed by Guido van Rossum in Github #195@footnote{https://github.com/python/typing/pull/195}.)

The @ref{7d7,,typing.ContextManager} class has been added for
representing @ref{7d6,,contextlib.AbstractContextManager}.
(Contributed by Brett Cannon in bpo-25609@footnote{https://bugs.python.org/issue25609}.)

The @ref{83d,,typing.Collection} class has been added for
representing @ref{7d0,,collections.abc.Collection}.
(Contributed by Ivan Levkivskyi in bpo-27598@footnote{https://bugs.python.org/issue27598}.)

The @ref{83e,,typing.ClassVar} type construct has been added to
mark class variables.  As introduced in 
@geindex Python Enhancement Proposals; PEP 526
PEP 526@footnote{https://www.python.org/dev/peps/pep-0526}, a variable annotation
wrapped in ClassVar indicates that a given attribute is intended to be used as
a class variable and should not be set on instances of that class.
(Contributed by Ivan Levkivskyi in Github #280@footnote{https://github.com/python/typing/pull/280}.)

A new @ref{83f,,TYPE_CHECKING} constant that is assumed to be
@code{True} by the static type checkers, but is @code{False} at runtime.
(Contributed by Guido van Rossum in Github #230@footnote{https://github.com/python/typing/issues/230}.)

A new @ref{840,,NewType()} helper function has been added to create
lightweight distinct types for annotations:

@example
from typing import NewType

UserId = NewType('UserId', int)
some_id = UserId(524313)
@end example

The static type checker will treat the new type as if it were a subclass
of the original type.  (Contributed by Ivan Levkivskyi in Github #189@footnote{https://github.com/python/typing/issues/189}.)

@node unicodedata<5>,unittest mock<2>,typing<5>,Improved Modules<6>
@anchor{whatsnew/3 6 unicodedata}@anchor{841}
@subsubsection unicodedata


The @ref{121,,unicodedata} module now uses data from Unicode 9.0.0@footnote{http://unicode.org/versions/Unicode9.0.0/}.
(Contributed by Benjamin Peterson.)

@node unittest mock<2>,urllib request,unicodedata<5>,Improved Modules<6>
@anchor{whatsnew/3 6 unittest-mock}@anchor{842}
@subsubsection unittest.mock


The @ref{561,,Mock} class has the following improvements:


@itemize *

@item 
Two new methods, @ref{843,,Mock.assert_called()} and @ref{844,,Mock.assert_called_once()} to check if the mock object
was called.
(Contributed by Amit Saha in bpo-26323@footnote{https://bugs.python.org/issue26323}.)

@item 
The @ref{845,,Mock.reset_mock()} method
now has two optional keyword only arguments: @emph{return_value} and
@emph{side_effect}.
(Contributed by Kushal Das in bpo-21271@footnote{https://bugs.python.org/issue21271}.)
@end itemize

@node urllib request,urllib robotparser,unittest mock<2>,Improved Modules<6>
@anchor{whatsnew/3 6 urllib-request}@anchor{846}
@subsubsection urllib.request


If a HTTP request has a file or iterable body (other than a
bytes object) but no @code{Content-Length} header, rather than
throwing an error, @code{AbstractHTTPHandler} now
falls back to use chunked transfer encoding.
(Contributed by Demian Brecht and Rolf Krahl in bpo-12319@footnote{https://bugs.python.org/issue12319}.)

@node urllib robotparser,venv<3>,urllib request,Improved Modules<6>
@anchor{whatsnew/3 6 urllib-robotparser}@anchor{847}
@subsubsection urllib.robotparser


@ref{848,,RobotFileParser} now supports the @code{Crawl-delay} and
@code{Request-rate} extensions.
(Contributed by Nikolay Bogoychev in bpo-16099@footnote{https://bugs.python.org/issue16099}.)

@node venv<3>,warnings<2>,urllib robotparser,Improved Modules<6>
@anchor{whatsnew/3 6 venv}@anchor{849}
@subsubsection venv


@ref{12c,,venv} accepts a new parameter @code{--prompt}. This parameter provides an
alternative prefix for the virtual environment. (Proposed by Łukasz Balcerzak
and ported to 3.6 by Stéphane Wirtel in bpo-22829@footnote{https://bugs.python.org/issue22829}.)

@node warnings<2>,winreg,venv<3>,Improved Modules<6>
@anchor{whatsnew/3 6 warnings}@anchor{84a}
@subsubsection warnings


A new optional @emph{source} parameter has been added to the
@ref{84b,,warnings.warn_explicit()} function: the destroyed object which emitted a
@ref{77b,,ResourceWarning}. A @emph{source} attribute has also been added to
@code{warnings.WarningMessage} (contributed by Victor Stinner in
bpo-26568@footnote{https://bugs.python.org/issue26568} and bpo-26567@footnote{https://bugs.python.org/issue26567}).

When a @ref{77b,,ResourceWarning} warning is logged, the @ref{11b,,tracemalloc} module is now
used to try to retrieve the traceback where the destroyed object was allocated.

Example with the script @code{example.py}:

@example
import warnings

def func():
    return open(__file__)

f = func()
f = None
@end example

Output of the command @code{python3.6 -Wd -X tracemalloc=5 example.py}:

@example
example.py:7: ResourceWarning: unclosed file <_io.TextIOWrapper name='example.py' mode='r' encoding='UTF-8'>
  f = None
Object allocated at (most recent call first):
  File "example.py", lineno 4
    return open(__file__)
  File "example.py", lineno 6
    f = func()
@end example

The “Object allocated at” traceback is new and is only displayed if
@ref{11b,,tracemalloc} is tracing Python memory allocations and if the
@ref{12d,,warnings} module was already imported.

@node winreg,winsound,warnings<2>,Improved Modules<6>
@anchor{whatsnew/3 6 winreg}@anchor{84c}
@subsubsection winreg


Added the 64-bit integer type @ref{84d,,REG_QWORD}.
(Contributed by Clement Rouault in bpo-23026@footnote{https://bugs.python.org/issue23026}.)

@node winsound,xmlrpc client,winreg,Improved Modules<6>
@anchor{whatsnew/3 6 winsound}@anchor{84e}
@subsubsection winsound


Allowed keyword arguments to be passed to @ref{84f,,Beep},
@ref{850,,MessageBeep}, and @ref{851,,PlaySound} (bpo-27982@footnote{https://bugs.python.org/issue27982}).

@node xmlrpc client,zipfile<2>,winsound,Improved Modules<6>
@anchor{whatsnew/3 6 xmlrpc-client}@anchor{852}
@subsubsection xmlrpc.client


The @ref{146,,xmlrpc.client} module now supports unmarshalling
additional data types used by the Apache XML-RPC implementation
for numerics and @code{None}.
(Contributed by Serhiy Storchaka in bpo-26885@footnote{https://bugs.python.org/issue26885}.)

@node zipfile<2>,zlib,xmlrpc client,Improved Modules<6>
@anchor{whatsnew/3 6 zipfile}@anchor{853}
@subsubsection zipfile


A new @ref{854,,ZipInfo.from_file()} class method
allows making a @ref{855,,ZipInfo} instance from a filesystem file.
A new @ref{856,,ZipInfo.is_dir()} method can be used
to check if the @ref{855,,ZipInfo} instance represents a directory.
(Contributed by Thomas Kluyver in bpo-26039@footnote{https://bugs.python.org/issue26039}.)

The @ref{857,,ZipFile.open()} method can now be used to
write data into a ZIP file, as well as for extracting data.
(Contributed by Thomas Kluyver in bpo-26039@footnote{https://bugs.python.org/issue26039}.)

@node zlib,,zipfile<2>,Improved Modules<6>
@anchor{whatsnew/3 6 zlib}@anchor{858}
@subsubsection zlib


The @ref{859,,compress()} and @ref{85a,,decompress()} functions now accept
keyword arguments.
(Contributed by Aviv Palivoda in bpo-26243@footnote{https://bugs.python.org/issue26243} and
Xiang Zhang in bpo-16764@footnote{https://bugs.python.org/issue16764} respectively.)

@node Optimizations<6>,Build and C API Changes<2>,Improved Modules<6>,What’s New In Python 3 6
@anchor{whatsnew/3 6 optimizations}@anchor{85b}
@subsection Optimizations



@itemize *

@item 
The Python interpreter now uses a 16-bit wordcode instead of bytecode which
made a number of opcode optimizations possible.
(Contributed by Demur Rumed with input and reviews from
Serhiy Storchaka and Victor Stinner in bpo-26647@footnote{https://bugs.python.org/issue26647} and bpo-28050@footnote{https://bugs.python.org/issue28050}.)

@item 
The @ref{316,,asyncio.Future} class now has an optimized C implementation.
(Contributed by Yury Selivanov and INADA Naoki in bpo-26081@footnote{https://bugs.python.org/issue26081}.)

@item 
The @ref{317,,asyncio.Task} class now has an optimized
C implementation. (Contributed by Yury Selivanov in bpo-28544@footnote{https://bugs.python.org/issue28544}.)

@item 
Various implementation improvements in the @ref{120,,typing} module
(such as caching of generic types) allow up to 30 times performance
improvements and reduced memory footprint.

@item 
The ASCII decoder is now up to 60 times as fast for error handlers
@code{surrogateescape}, @code{ignore} and @code{replace} (Contributed
by Victor Stinner in bpo-24870@footnote{https://bugs.python.org/issue24870}).

@item 
The ASCII and the Latin1 encoders are now up to 3 times as fast for the
error handler @code{surrogateescape}
(Contributed by Victor Stinner in bpo-25227@footnote{https://bugs.python.org/issue25227}).

@item 
The UTF-8 encoder is now up to 75 times as fast for error handlers
@code{ignore}, @code{replace}, @code{surrogateescape}, @code{surrogatepass} (Contributed
by Victor Stinner in bpo-25267@footnote{https://bugs.python.org/issue25267}).

@item 
The UTF-8 decoder is now up to 15 times as fast for error handlers
@code{ignore}, @code{replace} and @code{surrogateescape} (Contributed
by Victor Stinner in bpo-25301@footnote{https://bugs.python.org/issue25301}).

@item 
@code{bytes % args} is now up to 2 times faster. (Contributed by Victor Stinner
in bpo-25349@footnote{https://bugs.python.org/issue25349}).

@item 
@code{bytearray % args} is now between 2.5 and 5 times faster. (Contributed by
Victor Stinner in bpo-25399@footnote{https://bugs.python.org/issue25399}).

@item 
Optimize @ref{60c,,bytes.fromhex()} and @ref{60d,,bytearray.fromhex()}: they are now
between 2x and 3.5x faster. (Contributed by Victor Stinner in bpo-25401@footnote{https://bugs.python.org/issue25401}).

@item 
Optimize @code{bytes.replace(b'@w{'}, b'.')} and @code{bytearray.replace(b'@w{'}, b'.')}:
up to 80% faster. (Contributed by Josh Snider in bpo-26574@footnote{https://bugs.python.org/issue26574}).

@item 
Allocator functions of the @ref{7ad,,PyMem_Malloc()} domain
(@code{PYMEM_DOMAIN_MEM}) now use the @ref{85c,,pymalloc memory allocator} instead of @code{malloc()} function of the C library. The
pymalloc allocator is optimized for objects smaller or equal to 512 bytes
with a short lifetime, and use @code{malloc()} for larger memory blocks.
(Contributed by Victor Stinner in bpo-26249@footnote{https://bugs.python.org/issue26249}).

@item 
@ref{85d,,pickle.load()} and @ref{85e,,pickle.loads()} are now up to 10% faster when
deserializing many small objects (Contributed by Victor Stinner in
bpo-27056@footnote{https://bugs.python.org/issue27056}).

@item 
Passing @ref{85f,,keyword arguments} to a function has an
overhead in comparison with passing @ref{860,,positional arguments}.  Now in extension functions implemented with using
Argument Clinic this overhead is significantly decreased.
(Contributed by Serhiy Storchaka in bpo-27574@footnote{https://bugs.python.org/issue27574}).

@item 
Optimized @ref{288,,glob()} and @ref{289,,iglob()} functions in the
@ref{84,,glob} module; they are now about 3–6 times faster.
(Contributed by Serhiy Storchaka in bpo-25596@footnote{https://bugs.python.org/issue25596}).

@item 
Optimized globbing in @ref{c3,,pathlib} by using @ref{576,,os.scandir()};
it is now about 1.5–4 times faster.
(Contributed by Serhiy Storchaka in bpo-26032@footnote{https://bugs.python.org/issue26032}).

@item 
@ref{13e,,xml.etree.ElementTree} parsing, iteration and deepcopy performance
has been significantly improved.
(Contributed by Serhiy Storchaka in bpo-25638@footnote{https://bugs.python.org/issue25638}, bpo-25873@footnote{https://bugs.python.org/issue25873},
and bpo-25869@footnote{https://bugs.python.org/issue25869}.)

@item 
Creation of @ref{172,,fractions.Fraction} instances from floats and
decimals is now 2 to 3 times faster.
(Contributed by Serhiy Storchaka in bpo-25971@footnote{https://bugs.python.org/issue25971}.)
@end itemize

@node Build and C API Changes<2>,Other Improvements,Optimizations<6>,What’s New In Python 3 6
@anchor{whatsnew/3 6 build-and-c-api-changes}@anchor{861}
@subsection Build and C API Changes



@itemize *

@item 
Python now requires some C99 support in the toolchain to build.
Most notably, Python now uses standard integer types and macros in
place of custom macros like @code{PY_LONG_LONG}.
For more information, see 
@geindex Python Enhancement Proposals; PEP 7
PEP 7@footnote{https://www.python.org/dev/peps/pep-0007} and bpo-17884@footnote{https://bugs.python.org/issue17884}.

@item 
Cross-compiling CPython with the Android NDK and the Android API level set to
21 (Android 5.0 Lollipop) or greater runs successfully. While Android is not
yet a supported platform, the Python test suite runs on the Android emulator
with only about 16 tests failures. See the Android meta-issue bpo-26865@footnote{https://bugs.python.org/issue26865}.

@item 
The @code{--enable-optimizations} configure flag has been added. Turning it on
will activate expensive optimizations like PGO.
(Original patch by Alecsandru Patrascu of Intel in bpo-26359@footnote{https://bugs.python.org/issue26359}.)

@item 
The @ref{7ae,,GIL} must now be held when allocator
functions of @code{PYMEM_DOMAIN_OBJ} (ex: @ref{7af,,PyObject_Malloc()}) and
@code{PYMEM_DOMAIN_MEM} (ex: @ref{7ad,,PyMem_Malloc()}) domains are called.

@item 
New @ref{862,,Py_FinalizeEx()} API which indicates if flushing buffered data
failed.
(Contributed by Martin Panter in bpo-5319@footnote{https://bugs.python.org/issue5319}.)

@item 
@ref{863,,PyArg_ParseTupleAndKeywords()} now supports @ref{59d,,positional-only parameters}.  Positional-only parameters are
defined by empty names.
(Contributed by Serhiy Storchaka in bpo-26282@footnote{https://bugs.python.org/issue26282}).

@item 
@code{PyTraceback_Print} method now abbreviates long sequences of repeated lines
as @code{"[Previous line repeated @{count@} more times]"}.
(Contributed by Emanuel Barry in bpo-26823@footnote{https://bugs.python.org/issue26823}.)

@item 
The new @ref{864,,PyErr_SetImportErrorSubclass()} function allows for
specifying a subclass of @ref{3b8,,ImportError} to raise.
(Contributed by Eric Snow in bpo-15767@footnote{https://bugs.python.org/issue15767}.)

@item 
The new @ref{865,,PyErr_ResourceWarning()} function can be used to generate
a @ref{77b,,ResourceWarning} providing the source of the resource allocation.
(Contributed by Victor Stinner in bpo-26567@footnote{https://bugs.python.org/issue26567}.)

@item 
The new @ref{866,,PyOS_FSPath()} function returns the file system
representation of a @ref{63d,,path-like object}.
(Contributed by Brett Cannon in bpo-27186@footnote{https://bugs.python.org/issue27186}.)

@item 
The @ref{867,,PyUnicode_FSConverter()} and @ref{868,,PyUnicode_FSDecoder()}
functions will now accept @ref{63d,,path-like objects}.
@end itemize

@node Other Improvements,Deprecated<7>,Build and C API Changes<2>,What’s New In Python 3 6
@anchor{whatsnew/3 6 other-improvements}@anchor{869}
@subsection Other Improvements



@itemize *

@item 
When @ref{86a,,--version} (short form: @ref{86b,,-V}) is supplied twice,
Python prints @ref{86c,,sys.version} for detailed information.

@example
$ ./python -VV
Python 3.6.0b4+ (3.6:223967b49e49+, Nov 21 2016, 20:55:04)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
@end example
@end itemize

@node Deprecated<7>,Removed<7>,Other Improvements,What’s New In Python 3 6
@anchor{whatsnew/3 6 deprecated}@anchor{86d}
@subsection Deprecated


@menu
* New Keywords:: 
* Deprecated Python behavior:: 
* Deprecated Python modules@comma{} functions and methods: Deprecated Python modules functions and methods<2>. 
* Deprecated functions and types of the C API: Deprecated functions and types of the C API<2>. 
* Deprecated Build Options:: 

@end menu

@node New Keywords,Deprecated Python behavior,,Deprecated<7>
@anchor{whatsnew/3 6 new-keywords}@anchor{86e}
@subsubsection New Keywords


@code{async} and @code{await} are not recommended to be used as variable, class,
function or module names.  Introduced by 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} in Python 3.5, they will
become proper keywords in Python 3.7.  Starting in Python 3.6, the use of
@code{async} or @code{await} as names will generate a @ref{1b2,,DeprecationWarning}.

@node Deprecated Python behavior,Deprecated Python modules functions and methods<2>,New Keywords,Deprecated<7>
@anchor{whatsnew/3 6 deprecated-python-behavior}@anchor{86f}
@subsubsection Deprecated Python behavior


Raising the @ref{736,,StopIteration} exception inside a generator will now
generate a @ref{1b2,,DeprecationWarning}, and will trigger a @ref{5af,,RuntimeError}
in Python 3.7.  See @ref{870,,PEP 479; Change StopIteration handling inside generators} for details.

The @code{__aiter__()} method is now expected to return an asynchronous
iterator directly instead of returning an awaitable as previously.
Doing the former will trigger a @ref{1b2,,DeprecationWarning}.  Backward
compatibility will be removed in Python 3.7.
(Contributed by Yury Selivanov in bpo-27243@footnote{https://bugs.python.org/issue27243}.)

A backslash-character pair that is not a valid escape sequence now generates
a @ref{1b2,,DeprecationWarning}.  Although this will eventually become a
@ref{21a,,SyntaxError}, that will not be for several Python releases.
(Contributed by Emanuel Barry in bpo-27364@footnote{https://bugs.python.org/issue27364}.)

When performing a relative import, falling back on @code{__name__} and
@code{__path__} from the calling module when @code{__spec__} or
@code{__package__} are not defined now raises an @ref{304,,ImportWarning}.
(Contributed by Rose Ames in bpo-25791@footnote{https://bugs.python.org/issue25791}.)

@node Deprecated Python modules functions and methods<2>,Deprecated functions and types of the C API<2>,Deprecated Python behavior,Deprecated<7>
@anchor{whatsnew/3 6 deprecated-python-modules-functions-and-methods}@anchor{871}
@subsubsection Deprecated Python modules, functions and methods


@menu
* asynchat:: 
* asyncore:: 
* dbm: dbm<3>. 
* distutils: distutils<5>. 
* grp:: 
* importlib: importlib<5>. 
* os: os<7>. 
* re: re<3>. 
* ssl: ssl<6>. 
* tkinter: tkinter<4>. 
* venv: venv<4>. 

@end menu

@node asynchat,asyncore,,Deprecated Python modules functions and methods<2>
@anchor{whatsnew/3 6 asynchat}@anchor{872}
@subsubsection asynchat


The @ref{8,,asynchat} has been deprecated in favor of @ref{9,,asyncio}.
(Contributed by Mariatta in bpo-25002@footnote{https://bugs.python.org/issue25002}.)

@node asyncore,dbm<3>,asynchat,Deprecated Python modules functions and methods<2>
@anchor{whatsnew/3 6 asyncore}@anchor{873}
@subsubsection asyncore


The @ref{a,,asyncore} has been deprecated in favor of @ref{9,,asyncio}.
(Contributed by Mariatta in bpo-25002@footnote{https://bugs.python.org/issue25002}.)

@node dbm<3>,distutils<5>,asyncore,Deprecated Python modules functions and methods<2>
@anchor{whatsnew/3 6 dbm}@anchor{874}
@subsubsection dbm


Unlike other @ref{30,,dbm} implementations, the @ref{31,,dbm.dumb} module
creates databases with the @code{'rw'} mode and allows modifying the database
opened with the @code{'r'} mode.  This behavior is now deprecated and will
be removed in 3.8.
(Contributed by Serhiy Storchaka in bpo-21708@footnote{https://bugs.python.org/issue21708}.)

@node distutils<5>,grp,dbm<3>,Deprecated Python modules functions and methods<2>
@anchor{whatsnew/3 6 id2}@anchor{875}
@subsubsection distutils


The undocumented @code{extra_path} argument to the
@code{Distribution} constructor is now considered deprecated
and will raise a warning if set.   Support for this parameter will be
removed in a future Python release.  See bpo-27919@footnote{https://bugs.python.org/issue27919} for details.

@node grp,importlib<5>,distutils<5>,Deprecated Python modules functions and methods<2>
@anchor{whatsnew/3 6 grp}@anchor{876}
@subsubsection grp


The support of non-integer arguments in @ref{877,,getgrgid()} has been
deprecated.
(Contributed by Serhiy Storchaka in bpo-26129@footnote{https://bugs.python.org/issue26129}.)

@node importlib<5>,os<7>,grp,Deprecated Python modules functions and methods<2>
@anchor{whatsnew/3 6 id3}@anchor{878}
@subsubsection importlib


The @ref{879,,importlib.machinery.SourceFileLoader.load_module()} and
@ref{87a,,importlib.machinery.SourcelessFileLoader.load_module()} methods
are now deprecated. They were the only remaining implementations of
@ref{301,,importlib.abc.Loader.load_module()} in @ref{96,,importlib} that had not
been deprecated in previous versions of Python in favour of
@ref{302,,importlib.abc.Loader.exec_module()}.

The @ref{87b,,importlib.machinery.WindowsRegistryFinder} class is now
deprecated. As of 3.6.0, it is still added to @ref{87c,,sys.meta_path} by
default (on Windows), but this may change in future releases.

@node os<7>,re<3>,importlib<5>,Deprecated Python modules functions and methods<2>
@anchor{whatsnew/3 6 id4}@anchor{87d}
@subsubsection os


Undocumented support of general @ref{87e,,bytes-like objects}
as paths in @ref{c0,,os} functions, @ref{4e2,,compile()} and similar functions is
now deprecated.
(Contributed by Serhiy Storchaka in bpo-25791@footnote{https://bugs.python.org/issue25791} and bpo-26754@footnote{https://bugs.python.org/issue26754}.)

@node re<3>,ssl<6>,os<7>,Deprecated Python modules functions and methods<2>
@anchor{whatsnew/3 6 id5}@anchor{87f}
@subsubsection re


Support for inline flags @code{(?letters)} in the middle of the regular
expression has been deprecated and will be removed in a future Python
version.  Flags at the start of a regular expression are still allowed.
(Contributed by Serhiy Storchaka in bpo-22493@footnote{https://bugs.python.org/issue22493}.)

@node ssl<6>,tkinter<4>,re<3>,Deprecated Python modules functions and methods<2>
@anchor{whatsnew/3 6 id6}@anchor{880}
@subsubsection ssl


OpenSSL 0.9.8, 1.0.0 and 1.0.1 are deprecated and no longer supported.
In the future the @ref{ee,,ssl} module will require at least OpenSSL 1.0.2 or
1.1.0.

SSL-related arguments like @code{certfile}, @code{keyfile} and @code{check_hostname}
in @ref{7e,,ftplib}, @ref{8f,,http.client}, @ref{93,,imaplib}, @ref{cb,,poplib},
and @ref{e8,,smtplib} have been deprecated in favor of @code{context}.
(Contributed by Christian Heimes in bpo-28022@footnote{https://bugs.python.org/issue28022}.)

A couple of protocols and functions of the @ref{ee,,ssl} module are now
deprecated. Some features will no longer be available in future versions
of OpenSSL. Other features are deprecated in favor of a different API.
(Contributed by Christian Heimes in bpo-28022@footnote{https://bugs.python.org/issue28022} and bpo-26470@footnote{https://bugs.python.org/issue26470}.)

@node tkinter<4>,venv<4>,ssl<6>,Deprecated Python modules functions and methods<2>
@anchor{whatsnew/3 6 id7}@anchor{881}
@subsubsection tkinter


The @ref{115,,tkinter.tix} module is now deprecated.  @ref{10c,,tkinter} users
should use @ref{116,,tkinter.ttk} instead.

@node venv<4>,,tkinter<4>,Deprecated Python modules functions and methods<2>
@anchor{whatsnew/3 6 id8}@anchor{882}
@subsubsection venv


The @code{pyvenv} script has been deprecated in favour of @code{python3 -m venv}.
This prevents confusion as to what Python interpreter @code{pyvenv} is
connected to and thus what Python interpreter will be used by the virtual
environment.  (Contributed by Brett Cannon in bpo-25154@footnote{https://bugs.python.org/issue25154}.)

@node Deprecated functions and types of the C API<2>,Deprecated Build Options,Deprecated Python modules functions and methods<2>,Deprecated<7>
@anchor{whatsnew/3 6 deprecated-functions-and-types-of-the-c-api}@anchor{883}
@subsubsection Deprecated functions and types of the C API


Undocumented functions @code{PyUnicode_AsEncodedObject()},
@code{PyUnicode_AsDecodedObject()}, @code{PyUnicode_AsEncodedUnicode()}
and @code{PyUnicode_AsDecodedUnicode()} are deprecated now.
Use the @ref{884,,generic codec based API} instead.

@node Deprecated Build Options,,Deprecated functions and types of the C API<2>,Deprecated<7>
@anchor{whatsnew/3 6 deprecated-build-options}@anchor{885}
@subsubsection Deprecated Build Options


The @code{--with-system-ffi} configure flag is now on by default on non-macOS
UNIX platforms.  It may be disabled by using @code{--without-system-ffi}, but
using the flag is deprecated and will not be accepted in Python 3.7.
macOS is unaffected by this change.  Note that many OS distributors already
use the @code{--with-system-ffi} flag when building their system Python.

@node Removed<7>,Porting to Python 3 6,Deprecated<7>,What’s New In Python 3 6
@anchor{whatsnew/3 6 removed}@anchor{886}
@subsection Removed


@menu
* API and Feature Removals: API and Feature Removals<3>. 

@end menu

@node API and Feature Removals<3>,,,Removed<7>
@anchor{whatsnew/3 6 api-and-feature-removals}@anchor{887}
@subsubsection API and Feature Removals



@itemize *

@item 
Unknown escapes consisting of @code{'\'} and an ASCII letter in
regular expressions will now cause an error.  In replacement templates for
@ref{72e,,re.sub()} they are still allowed, but deprecated.
The @ref{698,,re.LOCALE} flag can now only be used with binary patterns.

@item 
@code{inspect.getmoduleinfo()} was removed (was deprecated since CPython 3.3).
@ref{888,,inspect.getmodulename()} should be used for obtaining the module
name for a given path.
(Contributed by Yury Selivanov in bpo-13248@footnote{https://bugs.python.org/issue13248}.)

@item 
@code{traceback.Ignore} class and @code{traceback.usage}, @code{traceback.modname},
@code{traceback.fullmodname}, @code{traceback.find_lines_from_code},
@code{traceback.find_lines}, @code{traceback.find_strings},
@code{traceback.find_executable_lines} methods were removed from the
@ref{11a,,traceback} module. They were undocumented methods deprecated since
Python 3.2 and equivalent functionality is available from private methods.

@item 
The @code{tk_menuBar()} and @code{tk_bindForTraversal()} dummy methods in
@ref{10c,,tkinter} widget classes were removed (corresponding Tk commands
were obsolete since Tk 4.0).

@item 
The @ref{857,,open()} method of the @ref{6e5,,zipfile.ZipFile}
class no longer supports the @code{'U'} mode (was deprecated since Python 3.4).
Use @ref{889,,io.TextIOWrapper} for reading compressed text files in
@ref{88a,,universal newlines} mode.

@item 
The undocumented @code{IN}, @code{CDROM}, @code{DLFCN}, @code{TYPES}, @code{CDIO}, and
@code{STROPTS} modules have been removed.  They had been available in the
platform specific @code{Lib/plat-*/} directories, but were chronically out of
date, inconsistently available across platforms, and unmaintained.  The
script that created these modules is still available in the source
distribution at Tools/scripts/h2py.py@footnote{https://github.com/python/cpython/tree/main/Tools/scripts/h2py.py}.

@item 
The deprecated @code{asynchat.fifo} class has been removed.
@end itemize

@node Porting to Python 3 6,Notable changes in Python 3 6 2,Removed<7>,What’s New In Python 3 6
@anchor{whatsnew/3 6 porting-to-python-3-6}@anchor{88b}
@subsection Porting to Python 3.6


This section lists previously described changes and other bugfixes
that may require changes to your code.

@menu
* Changes in ‘python’ Command Behavior:: 
* Changes in the Python API: Changes in the Python API<6>. 
* Changes in the C API: Changes in the C API<5>. 
* CPython bytecode changes: CPython bytecode changes<6>. 

@end menu

@node Changes in ‘python’ Command Behavior,Changes in the Python API<6>,,Porting to Python 3 6
@anchor{whatsnew/3 6 changes-in-python-command-behavior}@anchor{88c}
@subsubsection Changes in ‘python’ Command Behavior



@itemize *

@item 
The output of a special Python build with defined @code{COUNT_ALLOCS},
@code{SHOW_ALLOC_COUNT} or @code{SHOW_TRACK_COUNT} macros is now off by
default.  It can be re-enabled using the @code{-X showalloccount} option.
It now outputs to @code{stderr} instead of @code{stdout}.
(Contributed by Serhiy Storchaka in bpo-23034@footnote{https://bugs.python.org/issue23034}.)
@end itemize

@node Changes in the Python API<6>,Changes in the C API<5>,Changes in ‘python’ Command Behavior,Porting to Python 3 6
@anchor{whatsnew/3 6 changes-in-the-python-api}@anchor{88d}
@subsubsection Changes in the Python API



@itemize *

@item 
@ref{1c8,,open()} will no longer allow combining the @code{'U'} mode flag
with @code{'+'}.
(Contributed by Jeff Balogh and John O’Connor in bpo-2091@footnote{https://bugs.python.org/issue2091}.)

@item 
@ref{ed,,sqlite3} no longer implicitly commits an open transaction before DDL
statements.

@item 
On Linux, @ref{17f,,os.urandom()} now blocks until the system urandom entropy pool
is initialized to increase the security.

@item 
When @ref{302,,importlib.abc.Loader.exec_module()} is defined,
@ref{7f6,,importlib.abc.Loader.create_module()} must also be defined.

@item 
@ref{88e,,PyErr_SetImportError()} now sets @ref{162,,TypeError} when its @strong{msg}
argument is not set. Previously only @code{NULL} was returned.

@item 
The format of the @code{co_lnotab} attribute of code objects changed to support
a negative line number delta. By default, Python does not emit bytecode with
a negative line number delta. Functions using @code{frame.f_lineno},
@code{PyFrame_GetLineNumber()} or @code{PyCode_Addr2Line()} are not affected.
Functions directly decoding @code{co_lnotab} should be updated to use a signed
8-bit integer type for the line number delta, but this is only required to
support applications using a negative line number delta. See
@code{Objects/lnotab_notes.txt} for the @code{co_lnotab} format and how to decode
it, and see the 
@geindex Python Enhancement Proposals; PEP 511
PEP 511@footnote{https://www.python.org/dev/peps/pep-0511} for the rationale.

@item 
The functions in the @ref{1f,,compileall} module now return booleans instead
of @code{1} or @code{0} to represent success or failure, respectively. Thanks to
booleans being a subclass of integers, this should only be an issue if you
were doing identity checks for @code{1} or @code{0}. See bpo-25768@footnote{https://bugs.python.org/issue25768}.

@item 
Reading the @code{port} attribute of
@ref{88f,,urllib.parse.urlsplit()} and @ref{890,,urlparse()} results
now raises @ref{17c,,ValueError} for out-of-range values, rather than
returning @ref{182,,None}.  See bpo-20059@footnote{https://bugs.python.org/issue20059}.

@item 
The @ref{95,,imp} module now raises a @ref{1b2,,DeprecationWarning} instead of
@ref{3a9,,PendingDeprecationWarning}.

@item 
The following modules have had missing APIs added to their @code{__all__}
attributes to match the documented APIs:
@ref{13,,calendar}, @ref{14,,cgi}, @ref{28,,csv},
@ref{13e,,ElementTree}, @ref{76,,enum},
@ref{7b,,fileinput}, @ref{7e,,ftplib}, @ref{a6,,logging}, @ref{aa,,mailbox},
@ref{ae,,mimetypes}, @ref{bf,,optparse}, @ref{ca,,plistlib}, @ref{e7,,smtpd},
@ref{f4,,subprocess}, @ref{fb,,tarfile}, @ref{109,,threading} and
@ref{12e,,wave}.  This means they will export new symbols when @code{import *}
is used.
(Contributed by Joel Taddei and Jacek Kołodziej in bpo-23883@footnote{https://bugs.python.org/issue23883}.)

@item 
When performing a relative import, if @code{__package__} does not compare equal
to @code{__spec__.parent} then @ref{304,,ImportWarning} is raised.
(Contributed by Brett Cannon in bpo-25791@footnote{https://bugs.python.org/issue25791}.)

@item 
When a relative import is performed and no parent package is known, then
@ref{3b8,,ImportError} will be raised. Previously, @ref{891,,SystemError} could be
raised. (Contributed by Brett Cannon in bpo-18018@footnote{https://bugs.python.org/issue18018}.)

@item 
Servers based on the @ref{eb,,socketserver} module, including those
defined in @ref{92,,http.server}, @ref{147,,xmlrpc.server} and
@ref{136,,wsgiref.simple_server}, now only catch exceptions derived
from @ref{4d8,,Exception}. Therefore if a request handler raises
an exception like @ref{892,,SystemExit} or @ref{4ca,,KeyboardInterrupt},
@ref{893,,handle_error()} is no longer called, and
the exception will stop a single-threaded server. (Contributed by
Martin Panter in bpo-23430@footnote{https://bugs.python.org/issue23430}.)

@item 
@ref{894,,spwd.getspnam()} now raises a @ref{895,,PermissionError} instead of
@ref{5bb,,KeyError} if the user doesn’t have privileges.

@item 
The @ref{896,,socket.socket.close()} method now raises an exception if
an error (e.g. @code{EBADF}) was reported by the underlying system call.
(Contributed by Martin Panter in bpo-26685@footnote{https://bugs.python.org/issue26685}.)

@item 
The @emph{decode_data} argument for the @ref{897,,smtpd.SMTPChannel} and
@ref{898,,smtpd.SMTPServer} constructors is now @code{False} by default.
This means that the argument passed to
@ref{899,,process_message()} is now a bytes object by
default, and @code{process_message()} will be passed keyword arguments.
Code that has already been updated in accordance with the deprecation
warning generated by 3.5 will not be affected.

@item 
All optional arguments of the @ref{89a,,dump()}, @ref{89b,,dumps()},
@ref{7fe,,load()} and @ref{456,,loads()} functions and
@ref{89c,,JSONEncoder} and @ref{89d,,JSONDecoder} class
constructors in the @ref{a0,,json} module are now @ref{5a2,,keyword-only}.
(Contributed by Serhiy Storchaka in bpo-18726@footnote{https://bugs.python.org/issue18726}.)

@item 
Subclasses of @ref{89e,,type} which don’t override @code{type.__new__} may no
longer use the one-argument form to get the type of an object.

@item 
As part of 
@geindex Python Enhancement Proposals; PEP 487
PEP 487@footnote{https://www.python.org/dev/peps/pep-0487}, the handling of keyword arguments passed to
@ref{89e,,type} (other than the metaclass hint, @code{metaclass}) is now
consistently delegated to @ref{78d,,object.__init_subclass__()}. This means that
@code{type.__new__()} and @code{type.__init__()} both now accept arbitrary
keyword arguments, but @ref{78d,,object.__init_subclass__()} (which is called from
@code{type.__new__()}) will reject them by default. Custom metaclasses
accepting additional keyword arguments will need to adjust their calls to
@code{type.__new__()} (whether direct or via @ref{78c,,super}) accordingly.

@item 
In @code{distutils.command.sdist.sdist}, the @code{default_format}
attribute has been removed and is no longer honored. Instead, the
gzipped tarfile format is the default on all platforms and no
platform-specific selection is made.
In environments where distributions are
built on Windows and zip distributions are required, configure
the project with a @code{setup.cfg} file containing the following:

@example
[sdist]
formats=zip
@end example

This behavior has also been backported to earlier Python versions
by Setuptools 26.0.0.

@item 
In the @ref{127,,urllib.request} module and the
@ref{7f1,,http.client.HTTPConnection.request()} method, if no Content-Length
header field has been specified and the request body is a file object,
it is now sent with HTTP 1.1 chunked encoding. If a file object has to
be sent to a HTTP 1.0 server, the Content-Length value now has to be
specified by the caller.
(Contributed by Demian Brecht and Rolf Krahl with tweaks from
Martin Panter in bpo-12319@footnote{https://bugs.python.org/issue12319}.)

@item 
The @ref{4e9,,DictReader} now returns rows of type
@ref{4e5,,OrderedDict}.
(Contributed by Steve Holden in bpo-27842@footnote{https://bugs.python.org/issue27842}.)

@item 
The @ref{89f,,crypt.METHOD_CRYPT} will no longer be added to @code{crypt.methods}
if unsupported by the platform.
(Contributed by Victor Stinner in bpo-25287@footnote{https://bugs.python.org/issue25287}.)

@item 
The @emph{verbose} and @emph{rename} arguments for
@ref{3e4,,namedtuple()} are now keyword-only.
(Contributed by Raymond Hettinger in bpo-25628@footnote{https://bugs.python.org/issue25628}.)

@item 
On Linux, @ref{8a0,,ctypes.util.find_library()} now looks in
@code{LD_LIBRARY_PATH} for shared libraries.
(Contributed by Vinay Sajip in bpo-9998@footnote{https://bugs.python.org/issue9998}.)

@item 
The @ref{3f2,,imaplib.IMAP4} class now handles flags containing the
@code{']'} character in messages sent from the server to improve
real-world compatibility.
(Contributed by Lita Cho in bpo-21815@footnote{https://bugs.python.org/issue21815}.)

@item 
The @code{mmap.write()} function now returns the number
of bytes written like other write methods.
(Contributed by Jakub Stasiak in bpo-26335@footnote{https://bugs.python.org/issue26335}.)

@item 
The @ref{8a1,,pkgutil.iter_modules()} and @ref{73a,,pkgutil.walk_packages()}
functions now return @ref{8a2,,ModuleInfo} named tuples.
(Contributed by Ramchandra Apte in bpo-17211@footnote{https://bugs.python.org/issue17211}.)

@item 
@ref{72e,,re.sub()} now raises an error for invalid numerical group
references in replacement templates even if the pattern is not
found in the string.  The error message for invalid group references
now includes the group index and the position of the reference.
(Contributed by SilentGhost, Serhiy Storchaka in bpo-25953@footnote{https://bugs.python.org/issue25953}.)

@item 
@ref{6e5,,zipfile.ZipFile} will now raise @ref{8a3,,NotImplementedError} for
unrecognized compression values.  Previously a plain @ref{5af,,RuntimeError}
was raised.  Additionally, calling @ref{6e5,,ZipFile} methods
on a closed ZipFile or calling the @ref{8a4,,write()} method
on a ZipFile created with mode @code{'r'} will raise a @ref{17c,,ValueError}.
Previously, a @ref{5af,,RuntimeError} was raised in those scenarios.

@item 
when custom metaclasses are combined with zero-argument @ref{78c,,super()} or
direct references from methods to the implicit @code{__class__} closure
variable, the implicit @code{__classcell__} namespace entry must now be passed
up to @code{type.__new__} for initialisation. Failing to do so will result in
a @ref{1b2,,DeprecationWarning} in Python 3.6 and a @ref{5af,,RuntimeError} in
Python 3.8.

@item 
With the introduction of @ref{66b,,ModuleNotFoundError}, import system consumers
may start expecting import system replacements to raise that more specific
exception when appropriate, rather than the less-specific @ref{3b8,,ImportError}.
To provide future compatibility with such consumers, implementors of
alternative import systems that completely replace @ref{3b7,,__import__()} will
need to update their implementations to raise the new subclass when a module
can’t be found at all. Implementors of compliant plugins to the default
import system shouldn’t need to make any changes, as the default import
system will raise the new subclass when appropriate.
@end itemize

@node Changes in the C API<5>,CPython bytecode changes<6>,Changes in the Python API<6>,Porting to Python 3 6
@anchor{whatsnew/3 6 changes-in-the-c-api}@anchor{8a5}
@subsubsection Changes in the C API



@itemize *

@item 
The @ref{7ad,,PyMem_Malloc()} allocator family now uses the @ref{85c,,pymalloc allocator} rather than the system @code{malloc()}. Applications calling
@ref{7ad,,PyMem_Malloc()} without holding the GIL can now crash. Set the
@geindex PYTHONMALLOC
@geindex environment variable; PYTHONMALLOC
@ref{7ab,,PYTHONMALLOC} environment variable to @code{debug} to validate the
usage of memory allocators in your application. See bpo-26249@footnote{https://bugs.python.org/issue26249}.

@item 
@ref{8a6,,Py_Exit()} (and the main interpreter) now override the exit status
with 120 if flushing buffered data failed.  See bpo-5319@footnote{https://bugs.python.org/issue5319}.
@end itemize

@node CPython bytecode changes<6>,,Changes in the C API<5>,Porting to Python 3 6
@anchor{whatsnew/3 6 cpython-bytecode-changes}@anchor{8a7}
@subsubsection CPython bytecode changes


There have been several major changes to the @ref{8a8,,bytecode} in Python 3.6.


@itemize *

@item 
The Python interpreter now uses a 16-bit wordcode instead of bytecode.
(Contributed by Demur Rumed with input and reviews from
Serhiy Storchaka and Victor Stinner in bpo-26647@footnote{https://bugs.python.org/issue26647} and bpo-28050@footnote{https://bugs.python.org/issue28050}.)

@item 
The new @ref{8a9,,FORMAT_VALUE} and @ref{8aa,,BUILD_STRING} opcodes as part
of the  @ref{76c,,formatted string literal} implementation.
(Contributed by Eric Smith in bpo-25483@footnote{https://bugs.python.org/issue25483} and
Serhiy Storchaka in bpo-27078@footnote{https://bugs.python.org/issue27078}.)

@item 
The new @ref{8ab,,BUILD_CONST_KEY_MAP} opcode to optimize the creation
of dictionaries with constant keys.
(Contributed by Serhiy Storchaka in bpo-27140@footnote{https://bugs.python.org/issue27140}.)

@item 
The function call opcodes have been heavily reworked for better performance
and simpler implementation.
The @ref{8ac,,MAKE_FUNCTION}, @code{CALL_FUNCTION},
@code{CALL_FUNCTION_KW} and @code{BUILD_MAP_UNPACK_WITH_CALL} opcodes
have been modified, the new @ref{8ad,,CALL_FUNCTION_EX} and
@code{BUILD_TUPLE_UNPACK_WITH_CALL} have been added, and
@code{CALL_FUNCTION_VAR}, @code{CALL_FUNCTION_VAR_KW} and @code{MAKE_CLOSURE} opcodes
have been removed.
(Contributed by Demur Rumed in bpo-27095@footnote{https://bugs.python.org/issue27095}, and Serhiy Storchaka in
bpo-27213@footnote{https://bugs.python.org/issue27213}, bpo-28257@footnote{https://bugs.python.org/issue28257}.)

@item 
The new @ref{8ae,,SETUP_ANNOTATIONS} and @code{STORE_ANNOTATION} opcodes
have been added to support the new @ref{8af,,variable annotation} syntax.
(Contributed by Ivan Levkivskyi in bpo-27985@footnote{https://bugs.python.org/issue27985}.)
@end itemize

@node Notable changes in Python 3 6 2,Notable changes in Python 3 6 4,Porting to Python 3 6,What’s New In Python 3 6
@anchor{whatsnew/3 6 notable-changes-in-python-3-6-2}@anchor{8b0}
@subsection Notable changes in Python 3.6.2


@menu
* New make regen-all build target:: 
* Removal of make touch build target:: 

@end menu

@node New make regen-all build target,Removal of make touch build target,,Notable changes in Python 3 6 2
@anchor{whatsnew/3 6 new-make-regen-all-build-target}@anchor{8b1}
@subsubsection New @code{make regen-all} build target


To simplify cross-compilation, and to ensure that CPython can reliably be
compiled without requiring an existing version of Python to already be
available, the autotools-based build system no longer attempts to implicitly
recompile generated files based on file modification times.

Instead, a new @code{make regen-all} command has been added to force regeneration
of these files when desired (e.g. after an initial version of Python has
already been built based on the pregenerated versions).

More selective regeneration targets are also defined - see
Makefile.pre.in@footnote{https://github.com/python/cpython/tree/main/Makefile.pre.in} for details.

(Contributed by Victor Stinner in bpo-23404@footnote{https://bugs.python.org/issue23404}.)


New in version 3.6.2.


@node Removal of make touch build target,,New make regen-all build target,Notable changes in Python 3 6 2
@anchor{whatsnew/3 6 removal-of-make-touch-build-target}@anchor{8b2}
@subsubsection Removal of @code{make touch} build target


The @code{make touch} build target previously used to request implicit regeneration
of generated files by updating their modification times has been removed.

It has been replaced by the new @code{make regen-all} target.

(Contributed by Victor Stinner in bpo-23404@footnote{https://bugs.python.org/issue23404}.)


Changed in version 3.6.2.


@node Notable changes in Python 3 6 4,Notable changes in Python 3 6 5,Notable changes in Python 3 6 2,What’s New In Python 3 6
@anchor{whatsnew/3 6 notable-changes-in-python-3-6-4}@anchor{8b3}
@subsection Notable changes in Python 3.6.4


The @code{PyExc_RecursionErrorInst} singleton that was part of the public API
has been removed as its members being never cleared may cause a segfault
during finalization of the interpreter.
(Contributed by Xavier de Gaye in bpo-22898@footnote{https://bugs.python.org/issue22898} and bpo-30697@footnote{https://bugs.python.org/issue30697}.)

@node Notable changes in Python 3 6 5,Notable changes in Python 3 6 7,Notable changes in Python 3 6 4,What’s New In Python 3 6
@anchor{whatsnew/3 6 notable-changes-in-python-3-6-5}@anchor{8b4}
@subsection Notable changes in Python 3.6.5


The @ref{739,,locale.localeconv()} function now sets temporarily the @code{LC_CTYPE}
locale to the @code{LC_NUMERIC} locale in some cases.
(Contributed by Victor Stinner in bpo-31900@footnote{https://bugs.python.org/issue31900}.)

@node Notable changes in Python 3 6 7,Notable changes in Python 3 6 10,Notable changes in Python 3 6 5,What’s New In Python 3 6
@anchor{whatsnew/3 6 notable-changes-in-python-3-6-7}@anchor{8b5}
@subsection Notable changes in Python 3.6.7


In 3.6.7 the @ref{118,,tokenize} module now implicitly emits a @code{NEWLINE} token
when provided with input that does not have a trailing new line.  This behavior
now matches what the C tokenizer does internally.
(Contributed by Ammar Askar in bpo-33899@footnote{https://bugs.python.org/issue33899}.)

@node Notable changes in Python 3 6 10,Notable changes in Python 3 6 13,Notable changes in Python 3 6 7,What’s New In Python 3 6
@anchor{whatsnew/3 6 notable-changes-in-python-3-6-10}@anchor{8b6}
@subsection Notable changes in Python 3.6.10


Due to significant security concerns, the @emph{reuse_address} parameter of
@ref{1b9,,asyncio.loop.create_datagram_endpoint()} is no longer supported. This is
because of the behavior of the socket option @code{SO_REUSEADDR} in UDP. For more
details, see the documentation for @code{loop.create_datagram_endpoint()}.
(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in
bpo-37228@footnote{https://bugs.python.org/issue37228}.)

@node Notable changes in Python 3 6 13,,Notable changes in Python 3 6 10,What’s New In Python 3 6
@anchor{whatsnew/3 6 notable-changes-in-python-3-6-13}@anchor{8b7}
@subsection Notable changes in Python 3.6.13


Earlier Python versions allowed using both @code{;} and @code{&} as
query parameter separators in @ref{2e3,,urllib.parse.parse_qs()} and
@ref{2e4,,urllib.parse.parse_qsl()}.  Due to security concerns, and to conform with
newer W3C recommendations, this has been changed to allow only a single
separator key, with @code{&} as the default.  This change also affects
@ref{2e5,,cgi.parse()} and @ref{2e6,,cgi.parse_multipart()} as they use the affected
functions internally. For more details, please see their respective
documentation.
(Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in bpo-42967@footnote{https://bugs.python.org/issue42967}.)

@node What’s New In Python 3 5,What’s New In Python 3 4,What’s New In Python 3 6,What’s New in Python
@anchor{whatsnew/3 5 doc}@anchor{8b8}@anchor{whatsnew/3 5 what-s-new-in-python-3-5}@anchor{8b9}
@section What’s New In Python 3.5


@*Editors: 
Elvis Pranskevichus <@email{elvis@@magic.io}>, Yury Selivanov <@email{yury@@magic.io}>


@c Rules for maintenance:
@c 
@c * Anyone can add text to this document.  Do not spend very much time
@c on the wording of your changes, because your text will probably
@c get rewritten to some degree.
@c 
@c * The maintainer will go through Misc/NEWS periodically and add
@c changes; it's therefore more important to add your changes to
@c Misc/NEWS than to this file.
@c 
@c * This is not a complete list of every single change; completeness
@c is the purpose of Misc/NEWS.  Some changes I consider too small
@c or esoteric to include.  If such a change is added to the text,
@c I'll just remove it.  (This is another reason you shouldn't spend
@c too much time on writing your addition.)
@c 
@c * If you want to draw your new text to the attention of the
@c maintainer, add 'XXX' to the beginning of the paragraph or
@c section.
@c 
@c * It's OK to just add a fragmentary note about a change.  For
@c example: "XXX Describe the transmogrify() function added to the
@c socket module."  The maintainer will research the change and
@c write the necessary text.
@c 
@c * You can comment out your additions if you like, but it's not
@c necessary (especially when a final release is some months away).
@c 
@c * Credit the author of a patch or bugfix.   Just the name is
@c sufficient; the e-mail address isn't necessary.
@c 
@c * It's helpful to add the bug/patch number as a comment:
@c 
@c XXX Describe the transmogrify() function added to the socket
@c module.
@c (Contributed by P.Y. Developer in :issue:`12345`.)
@c 
@c This saves the maintainer the effort of going through the Mercurial log
@c when researching a change.

This article explains the new features in Python 3.5, compared to 3.4.
Python 3.5 was released on September 13, 2015.  See the
changelog@footnote{https://docs.python.org/3.5/whatsnew/changelog.html} for a full
list of changes.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 478
PEP 478@footnote{https://www.python.org/dev/peps/pep-0478} - Python 3.5 Release Schedule


@menu
* Summary – Release highlights: Summary – Release highlights<6>. 
* New Features: New Features<10>. 
* Other Language Changes: Other Language Changes<7>. 
* New Modules: New Modules<7>. 
* Improved Modules: Improved Modules<7>. 
* Other module-level changes:: 
* Optimizations: Optimizations<7>. 
* Build and C API Changes: Build and C API Changes<3>. 
* Deprecated: Deprecated<8>. 
* Removed: Removed<8>. 
* Porting to Python 3.5: Porting to Python 3 5. 
* Notable changes in Python 3.5.4: Notable changes in Python 3 5 4. 

@end menu

@node Summary – Release highlights<6>,New Features<10>,,What’s New In Python 3 5
@anchor{whatsnew/3 5 summary-release-highlights}@anchor{8ba}
@subsection Summary – Release highlights


New syntax features:


@itemize *

@item 
@ref{8bb,,PEP 492}, coroutines with async and await syntax.

@item 
@ref{8bc,,PEP 465}, a new matrix multiplication operator: @code{a @@ b}.

@item 
@ref{8bd,,PEP 448}, additional unpacking generalizations.
@end itemize

New library modules:


@itemize *

@item 
@ref{120,,typing}: @ref{8be,,PEP 484 – Type Hints}.

@item 
@ref{148,,zipapp}: @ref{8bf,,PEP 441 Improving Python ZIP Application Support}.
@end itemize

New built-in features:


@itemize *

@item 
@code{bytes % args}, @code{bytearray % args}: @ref{8c0,,PEP 461} –
Adding @code{%} formatting to bytes and bytearray.

@item 
New @ref{8c1,,bytes.hex()}, @ref{8c2,,bytearray.hex()} and @ref{8c3,,memoryview.hex()}
methods. (Contributed by Arnon Yaari in bpo-9951@footnote{https://bugs.python.org/issue9951}.)

@item 
@ref{573,,memoryview} now supports tuple indexing (including multi-dimensional).
(Contributed by Antoine Pitrou in bpo-23632@footnote{https://bugs.python.org/issue23632}.)

@item 
Generators have a new @code{gi_yieldfrom} attribute, which returns the
object being iterated by @code{yield from} expressions. (Contributed
by Benno Leslie and Yury Selivanov in bpo-24450@footnote{https://bugs.python.org/issue24450}.)

@item 
A new @ref{8c4,,RecursionError} exception is now raised when maximum
recursion depth is reached.  (Contributed by Georg Brandl
in bpo-19235@footnote{https://bugs.python.org/issue19235}.)
@end itemize

CPython implementation improvements:


@itemize *

@item 
When the @code{LC_TYPE} locale is the POSIX locale (@code{C} locale),
@ref{5ee,,sys.stdin} and @ref{5ef,,sys.stdout} now use the
@code{surrogateescape} error handler, instead of the @code{strict} error handler.
(Contributed by Victor Stinner in bpo-19977@footnote{https://bugs.python.org/issue19977}.)

@item 
@code{.pyo} files are no longer used and have been replaced by a more flexible
scheme that includes the optimization level explicitly in @code{.pyc} name.
(See @ref{8c5,,PEP 488 overview}.)

@item 
Builtin and extension modules are now initialized in a multi-phase process,
which is similar to how Python modules are loaded.
(See @ref{8c6,,PEP 489 overview}.)
@end itemize

Significant improvements in the standard library:


@itemize *

@item 
@ref{4e5,,collections.OrderedDict} is now
@ref{8c7,,implemented in C}, which makes it
4 to 100 times faster.

@item 
The @ref{ee,,ssl} module gained
@ref{8c8,,support for Memory BIO}, which decouples SSL
protocol handling from network IO.

@item 
The new @ref{576,,os.scandir()} function provides a
@ref{8c9,,better and significantly faster way}
of directory traversal.

@item 
@ref{4f3,,functools.lru_cache()} has been mostly
@ref{8ca,,reimplemented in C}, yielding much better
performance.

@item 
The new @ref{6b7,,subprocess.run()} function provides a
@ref{8cb,,streamlined way to run subprocesses}.

@item 
The @ref{11a,,traceback} module has been significantly
@ref{8cc,,enhanced} for improved
performance and developer convenience.
@end itemize

Security improvements:


@itemize *

@item 
SSLv3 is now disabled throughout the standard library.
It can still be enabled by instantiating a @ref{2bf,,ssl.SSLContext}
manually.  (See bpo-22638@footnote{https://bugs.python.org/issue22638} for more details; this change was
backported to CPython 3.4 and 2.7.)

@item 
HTTP cookie parsing is now stricter, in order to protect
against potential injection attacks. (Contributed by Antoine Pitrou
in bpo-22796@footnote{https://bugs.python.org/issue22796}.)
@end itemize

Windows improvements:


@itemize *

@item 
A new installer for Windows has replaced the old MSI.
See @ref{5c9,,Using Python on Windows} for more information.

@item 
Windows builds now use Microsoft Visual C++ 14.0, and extension modules
should use the same.
@end itemize

Please read on for a comprehensive list of user-facing changes, including many
other smaller improvements, CPython optimizations, deprecations, and potential
porting issues.

@node New Features<10>,Other Language Changes<7>,Summary – Release highlights<6>,What’s New In Python 3 5
@anchor{whatsnew/3 5 new-features}@anchor{8cd}
@subsection New Features


@menu
* PEP 492 - Coroutines with async and await syntax:: 
* PEP 465 - A dedicated infix operator for matrix multiplication:: 
* PEP 448 - Additional Unpacking Generalizations:: 
* PEP 461 - percent formatting support for bytes and bytearray:: 
* PEP 484 - Type Hints:: 
* PEP 471 - os.scandir() function – a better and faster directory iterator: PEP 471 - os scandir function – a better and faster directory iterator. 
* PEP 475; Retry system calls failing with EINTR: PEP 475 Retry system calls failing with EINTR. 
* PEP 479; Change StopIteration handling inside generators: PEP 479 Change StopIteration handling inside generators. 
* PEP 485; A function for testing approximate equality: PEP 485 A function for testing approximate equality. 
* PEP 486; Make the Python Launcher aware of virtual environments: PEP 486 Make the Python Launcher aware of virtual environments. 
* PEP 488; Elimination of PYO files: PEP 488 Elimination of PYO files. 
* PEP 489; Multi-phase extension module initialization: PEP 489 Multi-phase extension module initialization. 

@end menu

@node PEP 492 - Coroutines with async and await syntax,PEP 465 - A dedicated infix operator for matrix multiplication,,New Features<10>
@anchor{whatsnew/3 5 pep-492-coroutines-with-async-and-await-syntax}@anchor{8ce}@anchor{whatsnew/3 5 whatsnew-pep-492}@anchor{8bb}
@subsubsection PEP 492 - Coroutines with async and await syntax


@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} greatly improves support for asynchronous programming in Python
by adding @ref{7bd,,awaitable objects},
@ref{8cf,,coroutine functions},
@ref{8d0,,asynchronous iteration},
and @ref{167,,asynchronous context managers}.

Coroutine functions are declared using the new @ref{1c0,,async def} syntax:

@example
>>> async def coro():
...     return 'spam'
@end example

Inside a coroutine function, the new @ref{4d5,,await} expression can be used
to suspend coroutine execution until the result is available.  Any object
can be @emph{awaited}, as long as it implements the @ref{7bd,,awaitable} protocol by
defining the @code{__await__()} method.

PEP 492 also adds @ref{5cd,,async for} statement for convenient iteration
over asynchronous iterables.

An example of a rudimentary HTTP client written using the new syntax:

@example
import asyncio

async def http_get(domain):
    reader, writer = await asyncio.open_connection(domain, 80)

    writer.write(b'\r\n'.join([
        b'GET / HTTP/1.1',
        b'Host: %b' % domain.encode('latin-1'),
        b'Connection: close',
        b'@w{'}, b'@w{'}
    ]))

    async for line in reader:
        print('>>>', line)

    writer.close()

loop = asyncio.get_event_loop()
try:
    loop.run_until_complete(http_get('example.com'))
finally:
    loop.close()
@end example

Similarly to asynchronous iteration, there is a new syntax for asynchronous
context managers.  The following script:

@example
import asyncio

async def coro(name, lock):
    print('coro @{@}: waiting for lock'.format(name))
    async with lock:
        print('coro @{@}: holding the lock'.format(name))
        await asyncio.sleep(1)
        print('coro @{@}: releasing the lock'.format(name))

loop = asyncio.get_event_loop()
lock = asyncio.Lock()
coros = asyncio.gather(coro(1, lock), coro(2, lock))
try:
    loop.run_until_complete(coros)
finally:
    loop.close()
@end example

will output:

@example
coro 2: waiting for lock
coro 2: holding the lock
coro 1: waiting for lock
coro 2: releasing the lock
coro 1: holding the lock
coro 1: releasing the lock
@end example

Note that both @ref{5cd,,async for} and @ref{169,,async with} can only
be used inside a coroutine function declared with @ref{1c0,,async def}.

Coroutine functions are intended to be run inside a compatible event loop,
such as the @ref{8d1,,asyncio loop}.

@cartouche
@quotation Note 

Changed in version 3.5.2: Starting with CPython 3.5.2, @code{__aiter__} can directly return
@ref{8d2,,asynchronous iterators}.  Returning
an @ref{7bd,,awaitable} object will result in a
@ref{3a9,,PendingDeprecationWarning}.

See more details in the @ref{8d3,,Asynchronous Iterators} documentation
section.

@end quotation
@end cartouche


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 492

@item PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} – Coroutines with async and await syntax

PEP written and implemented by Yury Selivanov.
@end table


@node PEP 465 - A dedicated infix operator for matrix multiplication,PEP 448 - Additional Unpacking Generalizations,PEP 492 - Coroutines with async and await syntax,New Features<10>
@anchor{whatsnew/3 5 pep-465-a-dedicated-infix-operator-for-matrix-multiplication}@anchor{8d4}@anchor{whatsnew/3 5 whatsnew-pep-465}@anchor{8bc}
@subsubsection PEP 465 - A dedicated infix operator for matrix multiplication


@geindex Python Enhancement Proposals; PEP 465
PEP 465@footnote{https://www.python.org/dev/peps/pep-0465} adds the @code{@@} infix operator for matrix multiplication.
Currently, no builtin Python types implement the new operator, however, it
can be implemented by defining @code{__matmul__()}, @code{__rmatmul__()},
and @code{__imatmul__()} for regular, reflected, and in-place matrix
multiplication.  The semantics of these methods is similar to that of
methods defining other infix arithmetic operators.

Matrix multiplication is a notably common operation in many fields of
mathematics, science, engineering, and the addition of @code{@@} allows writing
cleaner code:

@example
S = (H @@ beta - r).T @@ inv(H @@ V @@ H.T) @@ (H @@ beta - r)
@end example

instead of:

@example
S = dot((dot(H, beta) - r).T,
        dot(inv(dot(dot(H, V), H.T)), dot(H, beta) - r))
@end example

NumPy 1.10 has support for the new operator:

@example
>>> import numpy

>>> x = numpy.ones(3)
>>> x
array([ 1., 1., 1.])

>>> m = numpy.eye(3)
>>> m
array([[ 1., 0., 0.],
       [ 0., 1., 0.],
       [ 0., 0., 1.]])

>>> x @@ m
array([ 1., 1., 1.])
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 465

@item PEP 465@footnote{https://www.python.org/dev/peps/pep-0465} – A dedicated infix operator for matrix multiplication

PEP written by Nathaniel J. Smith; implemented by Benjamin Peterson.
@end table


@node PEP 448 - Additional Unpacking Generalizations,PEP 461 - percent formatting support for bytes and bytearray,PEP 465 - A dedicated infix operator for matrix multiplication,New Features<10>
@anchor{whatsnew/3 5 pep-448-additional-unpacking-generalizations}@anchor{8d5}@anchor{whatsnew/3 5 whatsnew-pep-448}@anchor{8bd}
@subsubsection PEP 448 - Additional Unpacking Generalizations


@geindex Python Enhancement Proposals; PEP 448
PEP 448@footnote{https://www.python.org/dev/peps/pep-0448} extends the allowed uses of the @code{*} iterable unpacking
operator and @code{**} dictionary unpacking operator.  It is now possible
to use an arbitrary number of unpackings in @ref{8d6,,function calls}:

@example
>>> print(*[1], *[2], 3, *[4, 5])
1 2 3 4 5

>>> def fn(a, b, c, d):
...     print(a, b, c, d)
...

>>> fn(**@{'a': 1, 'c': 3@}, **@{'b': 2, 'd': 4@})
1 2 3 4
@end example

Similarly, tuple, list, set, and dictionary displays allow multiple
unpackings (see @ref{8d7,,Expression lists} and @ref{8d8,,Dictionary displays}):

@example
>>> *range(4), 4
(0, 1, 2, 3, 4)

>>> [*range(4), 4]
[0, 1, 2, 3, 4]

>>> @{*range(4), 4, *(5, 6, 7)@}
@{0, 1, 2, 3, 4, 5, 6, 7@}

>>> @{'x': 1, **@{'y': 2@}@}
@{'x': 1, 'y': 2@}
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 448

@item PEP 448@footnote{https://www.python.org/dev/peps/pep-0448} – Additional Unpacking Generalizations

PEP written by Joshua Landau; implemented by Neil Girdhar,
Thomas Wouters, and Joshua Landau.
@end table


@node PEP 461 - percent formatting support for bytes and bytearray,PEP 484 - Type Hints,PEP 448 - Additional Unpacking Generalizations,New Features<10>
@anchor{whatsnew/3 5 pep-461-percent-formatting-support-for-bytes-and-bytearray}@anchor{8d9}@anchor{whatsnew/3 5 whatsnew-pep-461}@anchor{8c0}
@subsubsection PEP 461 - percent formatting support for bytes and bytearray


@geindex Python Enhancement Proposals; PEP 461
PEP 461@footnote{https://www.python.org/dev/peps/pep-0461} adds support for the @code{%}
@ref{8da,,interpolation operator} to @ref{2f0,,bytes}
and @ref{2f1,,bytearray}.

While interpolation is usually thought of as a string operation, there are
cases where interpolation on @code{bytes} or @code{bytearrays} makes sense, and the
work needed to make up for this missing functionality detracts from the
overall readability of the code.  This issue is particularly important when
dealing with wire format protocols, which are often a mixture of binary
and ASCII compatible text.

Examples:

@example
>>> b'Hello %b!' % b'World'
b'Hello World!'

>>> b'x=%i y=%f' % (1, 2.5)
b'x=1 y=2.500000'
@end example

Unicode is not allowed for @code{%b}, but it is accepted by @code{%a} (equivalent of
@code{repr(obj).encode('ascii', 'backslashreplace')}):

@example
>>> b'Hello %b!' % 'World'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: %b requires bytes, or an object that implements __bytes__, not 'str'

>>> b'price: %a' % '10€'
b"price: '10\\u20ac'"
@end example

Note that @code{%s} and @code{%r} conversion types, although supported, should
only be used in codebases that need compatibility with Python 2.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 461

@item PEP 461@footnote{https://www.python.org/dev/peps/pep-0461} – Adding % formatting to bytes and bytearray

PEP written by Ethan Furman; implemented by Neil Schemenauer and
Ethan Furman.
@end table


@node PEP 484 - Type Hints,PEP 471 - os scandir function – a better and faster directory iterator,PEP 461 - percent formatting support for bytes and bytearray,New Features<10>
@anchor{whatsnew/3 5 pep-484-type-hints}@anchor{8db}@anchor{whatsnew/3 5 whatsnew-pep-484}@anchor{8be}
@subsubsection PEP 484 - Type Hints


Function annotation syntax has been a Python feature since version 3.0
(
@geindex Python Enhancement Proposals; PEP 3107
PEP 3107@footnote{https://www.python.org/dev/peps/pep-3107}), however the semantics of annotations has been left undefined.

Experience has shown that the majority of function annotation
uses were to provide type hints to function parameters and return values.  It
became evident that it would be beneficial for Python users, if the
standard library included the base definitions and tools for type annotations.

@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} introduces a @ref{621,,provisional module} to
provide these standard definitions and tools, along with some conventions
for situations where annotations are not available.

For example, here is a simple function whose argument and return type
are declared in the annotations:

@example
def greeting(name: str) -> str:
    return 'Hello ' + name
@end example

While these annotations are available at runtime through the usual
@code{__annotations__} attribute, @emph{no automatic type checking happens at
runtime}.  Instead, it is assumed that a separate off-line type checker
(e.g. mypy@footnote{http://mypy-lang.org}) will be used for on-demand
source code analysis.

The type system supports unions, generic types, and a special type
named @ref{8dc,,Any} which is consistent with (i.e. assignable to
and from) all types.


@subsubheading See also



@itemize *

@item 
@ref{120,,typing} module documentation

@item 

@table @asis

@geindex Python Enhancement Proposals; PEP 484

@item PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} – Type Hints

PEP written by Guido van Rossum, Jukka Lehtosalo, and Łukasz Langa;
implemented by Guido van Rossum.
@end table

@item 

@table @asis

@geindex Python Enhancement Proposals; PEP 483

@item PEP 483@footnote{https://www.python.org/dev/peps/pep-0483} – The Theory of Type Hints

PEP written by Guido van Rossum
@end table
@end itemize


@node PEP 471 - os scandir function – a better and faster directory iterator,PEP 475 Retry system calls failing with EINTR,PEP 484 - Type Hints,New Features<10>
@anchor{whatsnew/3 5 pep-471-os-scandir-function-a-better-and-faster-directory-iterator}@anchor{8dd}@anchor{whatsnew/3 5 whatsnew-pep-471}@anchor{8c9}
@subsubsection PEP 471 - os.scandir() function – a better and faster directory iterator


@geindex Python Enhancement Proposals; PEP 471
PEP 471@footnote{https://www.python.org/dev/peps/pep-0471} adds a new directory iteration function, @ref{576,,os.scandir()},
to the standard library.  Additionally, @ref{8de,,os.walk()} is now
implemented using @code{scandir}, which makes it 3 to 5 times faster
on POSIX systems and 7 to 20 times faster on Windows systems.  This is
largely achieved by greatly reducing the number of calls to @ref{516,,os.stat()}
required to walk a directory tree.

Additionally, @code{scandir} returns an iterator, as opposed to returning
a list of file names, which improves memory efficiency when iterating
over very large directories.

The following example shows a simple use of @ref{576,,os.scandir()} to display all
the files (excluding directories) in the given @emph{path} that don’t start with
@code{'.'}. The @ref{8df,,entry.is_file()} call will generally
not make an additional system call:

@example
for entry in os.scandir(path):
    if not entry.name.startswith('.') and entry.is_file():
        print(entry.name)
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 471

@item PEP 471@footnote{https://www.python.org/dev/peps/pep-0471} – os.scandir() function – a better and faster directory iterator

PEP written and implemented by Ben Hoyt with the help of Victor Stinner.
@end table


@node PEP 475 Retry system calls failing with EINTR,PEP 479 Change StopIteration handling inside generators,PEP 471 - os scandir function – a better and faster directory iterator,New Features<10>
@anchor{whatsnew/3 5 pep-475-retry-system-calls-failing-with-eintr}@anchor{8e0}@anchor{whatsnew/3 5 whatsnew-pep-475}@anchor{8e1}
@subsubsection PEP 475: Retry system calls failing with EINTR


An @ref{8e2,,errno.EINTR} error code is returned whenever a system call, that
is waiting for I/O, is interrupted by a signal.  Previously, Python would
raise @ref{8e3,,InterruptedError} in such cases.  This meant that, when writing a
Python application, the developer had two choices:


@enumerate 

@item 
Ignore the @code{InterruptedError}.

@item 
Handle the @code{InterruptedError} and attempt to restart the interrupted
system call at every call site.
@end enumerate

The first option makes an application fail intermittently.
The second option adds a large amount of boilerplate that makes the
code nearly unreadable.  Compare:

@example
print("Hello World")
@end example

and:

@example
while True:
    try:
        print("Hello World")
        break
    except InterruptedError:
        continue
@end example

@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} implements automatic retry of system calls on
@code{EINTR}.  This removes the burden of dealing with @code{EINTR}
or @ref{8e3,,InterruptedError} in user code in most situations and makes
Python programs, including the standard library, more robust.  Note that
the system call is only retried if the signal handler does not raise an
exception.

Below is a list of functions which are now retried when interrupted
by a signal:


@itemize *

@item 
@ref{1c8,,open()} and @ref{1c9,,io.open()};

@item 
functions of the @ref{78,,faulthandler} module;

@item 
@ref{c0,,os} functions: @ref{8e4,,fchdir()}, @ref{8e5,,fchmod()},
@ref{8e6,,fchown()}, @ref{8e7,,fdatasync()}, @ref{8e8,,fstat()},
@ref{8e9,,fstatvfs()}, @ref{8ea,,fsync()}, @ref{8eb,,ftruncate()},
@ref{8ec,,mkfifo()}, @ref{8ed,,mknod()}, @ref{8ee,,open()},
@ref{8ef,,posix_fadvise()}, @ref{8f0,,posix_fallocate()}, @ref{688,,pread()},
@ref{68b,,pwrite()}, @ref{8f1,,read()}, @ref{687,,readv()}, @ref{62c,,sendfile()},
@ref{8f2,,wait3()}, @ref{8f3,,wait4()}, @ref{8f4,,wait()},
@ref{8f5,,waitid()}, @ref{8f6,,waitpid()}, @ref{8f7,,write()},
@ref{68a,,writev()};

@item 
special cases: @ref{6a1,,os.close()} and @ref{68d,,os.dup2()} now ignore
@ref{8e2,,EINTR} errors; the syscall is not retried (see the PEP
for the rationale);

@item 
@ref{e0,,select} functions: @ref{8f8,,devpoll.poll()},
@ref{8f9,,epoll.poll()},
@ref{8fa,,kqueue.control()},
@ref{8fb,,poll.poll()}, @ref{8fc,,select()};

@item 
methods of the @ref{8fd,,socket} class: @ref{8fe,,accept()},
@ref{8ff,,connect()} (except for non-blocking sockets),
@ref{900,,recv()}, @ref{901,,recvfrom()},
@ref{902,,recvmsg()}, @ref{903,,send()},
@ref{904,,sendall()}, @ref{905,,sendmsg()},
@ref{906,,sendto()};

@item 
@ref{907,,signal.sigtimedwait()} and @ref{908,,signal.sigwaitinfo()};

@item 
@ref{196,,time.sleep()}.
@end itemize


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 475

@item PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} – Retry system calls failing with EINTR

PEP and implementation written by Charles-François Natali and
Victor Stinner, with the help of Antoine Pitrou (the French connection).
@end table


@node PEP 479 Change StopIteration handling inside generators,PEP 485 A function for testing approximate equality,PEP 475 Retry system calls failing with EINTR,New Features<10>
@anchor{whatsnew/3 5 pep-479-change-stopiteration-handling-inside-generators}@anchor{909}@anchor{whatsnew/3 5 whatsnew-pep-479}@anchor{870}
@subsubsection PEP 479: Change StopIteration handling inside generators


The interaction of generators and @ref{736,,StopIteration} in Python 3.4 and
earlier was sometimes surprising, and could conceal obscure bugs.  Previously,
@code{StopIteration} raised accidentally inside a generator function was
interpreted as the end of the iteration by the loop construct driving the
generator.

@geindex Python Enhancement Proposals; PEP 479
PEP 479@footnote{https://www.python.org/dev/peps/pep-0479} changes the behavior of generators: when a @code{StopIteration}
exception is raised inside a generator, it is replaced with a
@ref{5af,,RuntimeError} before it exits the generator frame.  The main goal of
this change is to ease debugging in the situation where an unguarded
@ref{252,,next()} call raises @code{StopIteration} and causes the iteration controlled
by the generator to terminate silently. This is particularly pernicious in
combination with the @code{yield from} construct.

This is a backwards incompatible change, so to enable the new behavior,
a @ref{90a,,__future__} import is necessary:

@example
>>> from __future__ import generator_stop

>>> def gen():
...     next(iter([]))
...     yield
...
>>> next(gen())
Traceback (most recent call last):
  File "<stdin>", line 2, in gen
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: generator raised StopIteration
@end example

Without a @code{__future__} import, a @ref{3a9,,PendingDeprecationWarning} will be
raised whenever a @ref{736,,StopIteration} exception is raised inside a generator.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 479

@item PEP 479@footnote{https://www.python.org/dev/peps/pep-0479} – Change StopIteration handling inside generators

PEP written by Chris Angelico and Guido van Rossum. Implemented by
Chris Angelico, Yury Selivanov and Nick Coghlan.
@end table


@node PEP 485 A function for testing approximate equality,PEP 486 Make the Python Launcher aware of virtual environments,PEP 479 Change StopIteration handling inside generators,New Features<10>
@anchor{whatsnew/3 5 pep-485-a-function-for-testing-approximate-equality}@anchor{90b}@anchor{whatsnew/3 5 whatsnew-pep-485}@anchor{90c}
@subsubsection PEP 485: A function for testing approximate equality


@geindex Python Enhancement Proposals; PEP 485
PEP 485@footnote{https://www.python.org/dev/peps/pep-0485} adds the @ref{90d,,math.isclose()} and @ref{90e,,cmath.isclose()}
functions which tell whether two values are approximately equal or
“close” to each other.  Whether or not two values are considered
close is determined according to given absolute and relative tolerances.
Relative tolerance is the maximum allowed difference between @code{isclose}
arguments, relative to the larger absolute value:

@example
>>> import math
>>> a = 5.0
>>> b = 4.99998
>>> math.isclose(a, b, rel_tol=1e-5)
True
>>> math.isclose(a, b, rel_tol=1e-6)
False
@end example

It is also possible to compare two values using absolute tolerance, which
must be a non-negative value:

@example
>>> import math
>>> a = 5.0
>>> b = 4.99998
>>> math.isclose(a, b, abs_tol=0.00003)
True
>>> math.isclose(a, b, abs_tol=0.00001)
False
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 485

@item PEP 485@footnote{https://www.python.org/dev/peps/pep-0485} – A function for testing approximate equality

PEP written by Christopher Barker; implemented by Chris Barker and
Tal Einat.
@end table


@node PEP 486 Make the Python Launcher aware of virtual environments,PEP 488 Elimination of PYO files,PEP 485 A function for testing approximate equality,New Features<10>
@anchor{whatsnew/3 5 pep-486-make-the-python-launcher-aware-of-virtual-environments}@anchor{90f}@anchor{whatsnew/3 5 whatsnew-pep-486}@anchor{910}
@subsubsection PEP 486: Make the Python Launcher aware of virtual environments


@geindex Python Enhancement Proposals; PEP 486
PEP 486@footnote{https://www.python.org/dev/peps/pep-0486} makes the Windows launcher (see 
@geindex Python Enhancement Proposals; PEP 397
PEP 397@footnote{https://www.python.org/dev/peps/pep-0397}) aware of an active
virtual environment. When the default interpreter would be used and the
@code{VIRTUAL_ENV} environment variable is set, the interpreter in the virtual
environment will be used.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 486

@item PEP 486@footnote{https://www.python.org/dev/peps/pep-0486} – Make the Python Launcher aware of virtual environments

PEP written and implemented by Paul Moore.
@end table


@node PEP 488 Elimination of PYO files,PEP 489 Multi-phase extension module initialization,PEP 486 Make the Python Launcher aware of virtual environments,New Features<10>
@anchor{whatsnew/3 5 pep-488-elimination-of-pyo-files}@anchor{911}@anchor{whatsnew/3 5 whatsnew-pep-488}@anchor{8c5}
@subsubsection PEP 488: Elimination of PYO files


@geindex Python Enhancement Proposals; PEP 488
PEP 488@footnote{https://www.python.org/dev/peps/pep-0488} does away with the concept of @code{.pyo} files. This means that
@code{.pyc} files represent both unoptimized and optimized bytecode. To prevent the
need to constantly regenerate bytecode files, @code{.pyc} files now have an
optional @code{opt-} tag in their name when the bytecode is optimized. This has the
side-effect of no more bytecode file name clashes when running under either
@ref{912,,-O} or @ref{913,,-OO}. Consequently, bytecode files generated from
@ref{912,,-O}, and @ref{913,,-OO} may now exist simultaneously.
@ref{7f9,,importlib.util.cache_from_source()} has an updated API to help with
this change.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 488

@item PEP 488@footnote{https://www.python.org/dev/peps/pep-0488} – Elimination of PYO files

PEP written and implemented by Brett Cannon.
@end table


@node PEP 489 Multi-phase extension module initialization,,PEP 488 Elimination of PYO files,New Features<10>
@anchor{whatsnew/3 5 pep-489-multi-phase-extension-module-initialization}@anchor{914}@anchor{whatsnew/3 5 whatsnew-pep-489}@anchor{8c6}
@subsubsection PEP 489: Multi-phase extension module initialization


@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489} updates extension module initialization to take advantage of the
two step module loading mechanism introduced by 
@geindex Python Enhancement Proposals; PEP 451
PEP 451@footnote{https://www.python.org/dev/peps/pep-0451} in Python 3.4.

This change brings the import semantics of extension modules that opt-in to
using the new mechanism much closer to those of Python source and bytecode
modules, including the ability to use any valid identifier as a module name,
rather than being restricted to ASCII.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 489

@item PEP 489@footnote{https://www.python.org/dev/peps/pep-0489} – Multi-phase extension module initialization

PEP written by Petr Viktorin, Stefan Behnel, and Nick Coghlan;
implemented by Petr Viktorin.
@end table


@node Other Language Changes<7>,New Modules<7>,New Features<10>,What’s New In Python 3 5
@anchor{whatsnew/3 5 other-language-changes}@anchor{915}
@subsection Other Language Changes


Some smaller changes made to the core Python language are:


@itemize *

@item 
Added the @code{"namereplace"} error handlers.  The @code{"backslashreplace"}
error handlers now work with decoding and translating.
(Contributed by Serhiy Storchaka in bpo-19676@footnote{https://bugs.python.org/issue19676} and bpo-22286@footnote{https://bugs.python.org/issue22286}.)

@item 
The @ref{6dc,,-b} option now affects comparisons of @ref{2f0,,bytes} with
@ref{1b1,,int}.  (Contributed by Serhiy Storchaka in bpo-23681@footnote{https://bugs.python.org/issue23681}.)

@item 
New Kazakh @code{kz1048} and Tajik @code{koi8_t} @ref{916,,codecs}.
(Contributed by Serhiy Storchaka in bpo-22682@footnote{https://bugs.python.org/issue22682} and bpo-22681@footnote{https://bugs.python.org/issue22681}.)

@item 
Property docstrings are now writable. This is especially useful for
@ref{3e4,,collections.namedtuple()} docstrings.
(Contributed by Berker Peksag in bpo-24064@footnote{https://bugs.python.org/issue24064}.)

@item 
Circular imports involving relative imports are now supported.
(Contributed by Brett Cannon and Antoine Pitrou in bpo-17636@footnote{https://bugs.python.org/issue17636}.)
@end itemize

@node New Modules<7>,Improved Modules<7>,Other Language Changes<7>,What’s New In Python 3 5
@anchor{whatsnew/3 5 new-modules}@anchor{917}
@subsection New Modules


@menu
* typing: typing<6>. 
* zipapp: zipapp<2>. 

@end menu

@node typing<6>,zipapp<2>,,New Modules<7>
@anchor{whatsnew/3 5 typing}@anchor{918}
@subsubsection typing


The new @ref{120,,typing} @ref{621,,provisional} module
provides standard definitions and tools for function type annotations.
See @ref{8be,,Type Hints} for more information.

@node zipapp<2>,,typing<6>,New Modules<7>
@anchor{whatsnew/3 5 whatsnew-zipapp}@anchor{8bf}@anchor{whatsnew/3 5 zipapp}@anchor{919}
@subsubsection zipapp


The new @ref{148,,zipapp} module (specified in 
@geindex Python Enhancement Proposals; PEP 441
PEP 441@footnote{https://www.python.org/dev/peps/pep-0441}) provides an API and
command line tool for creating executable Python Zip Applications, which
were introduced in Python 2.6 in bpo-1739468@footnote{https://bugs.python.org/issue1739468}, but which were not well
publicized, either at the time or since.

With the new module, bundling your application is as simple as putting all
the files, including a @code{__main__.py} file, into a directory @code{myapp}
and running:

@example
$ python -m zipapp myapp
$ python myapp.pyz
@end example

The module implementation has been contributed by Paul Moore in
bpo-23491@footnote{https://bugs.python.org/issue23491}.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 441
PEP 441@footnote{https://www.python.org/dev/peps/pep-0441} – Improving Python ZIP Application Support


@node Improved Modules<7>,Other module-level changes,New Modules<7>,What’s New In Python 3 5
@anchor{whatsnew/3 5 improved-modules}@anchor{91a}
@subsection Improved Modules


@menu
* argparse: argparse<3>. 
* asyncio: asyncio<7>. 
* bz2:: 
* cgi:: 
* cmath: cmath<2>. 
* code:: 
* collections: collections<5>. 
* collections.abc: collections abc<3>. 
* compileall: compileall<3>. 
* concurrent.futures: concurrent futures<4>. 
* configparser:: 
* contextlib: contextlib<4>. 
* csv: csv<2>. 
* curses: curses<4>. 
* dbm: dbm<4>. 
* difflib:: 
* distutils: distutils<6>. 
* doctest: doctest<2>. 
* email: email<2>. 
* enum: enum<5>. 
* faulthandler: faulthandler<3>. 
* functools: functools<3>. 
* glob: glob<2>. 
* gzip: gzip<2>. 
* heapq:: 
* http: http<2>. 
* http.client: http client<3>. 
* idlelib and IDLE: idlelib and IDLE<3>. 
* imaplib: imaplib<2>. 
* imghdr:: 
* importlib: importlib<6>. 
* inspect: inspect<6>. 
* io: io<3>. 
* ipaddress: ipaddress<3>. 
* json: json<2>. 
* linecache: linecache<2>. 
* locale: locale<3>. 
* logging: logging<4>. 
* lzma:: 
* math: math<6>. 
* multiprocessing: multiprocessing<5>. 
* operator: operator<2>. 
* os: os<8>. 
* pathlib: pathlib<6>. 
* pickle: pickle<3>. 
* poplib: poplib<2>. 
* re: re<4>. 
* readline: readline<2>. 
* selectors:: 
* shutil: shutil<2>. 
* signal: signal<3>. 
* smtpd:: 
* smtplib: smtplib<2>. 
* sndhdr:: 
* socket: socket<8>. 
* ssl: ssl<7>. 
* sqlite3: sqlite3<5>. 
* subprocess: subprocess<3>. 
* sys: sys<8>. 
* sysconfig:: 
* tarfile: tarfile<2>. 
* threading: threading<5>. 
* time: time<6>. 
* timeit: timeit<2>. 
* tkinter: tkinter<5>. 
* traceback: traceback<3>. 
* types: types<3>. 
* unicodedata: unicodedata<6>. 
* unittest: unittest<4>. 
* unittest.mock: unittest mock<3>. 
* urllib:: 
* wsgiref:: 
* xmlrpc: xmlrpc<2>. 
* xml.sax: xml sax. 
* zipfile: zipfile<3>. 

@end menu

@node argparse<3>,asyncio<7>,,Improved Modules<7>
@anchor{whatsnew/3 5 argparse}@anchor{91b}
@subsubsection argparse


The @ref{91c,,ArgumentParser} class now allows disabling
@ref{91d,,abbreviated usage} of long options by setting
@ref{91e,,allow_abbrev} to @code{False}.  (Contributed by Jonathan Paugh,
Steven Bethard, paul j3 and Daniel Eriksson in bpo-14910@footnote{https://bugs.python.org/issue14910}.)

@node asyncio<7>,bz2,argparse<3>,Improved Modules<7>
@anchor{whatsnew/3 5 asyncio}@anchor{91f}
@subsubsection asyncio


Since the @ref{9,,asyncio} module is @ref{621,,provisional},
all changes introduced in Python 3.5 have also been backported to Python 3.4.x.

Notable changes in the @ref{9,,asyncio} module since Python 3.4.0:


@itemize *

@item 
New debugging APIs: @ref{920,,loop.set_debug()}
and @ref{921,,loop.get_debug()} methods.
(Contributed by Victor Stinner.)

@item 
The proactor event loop now supports SSL.
(Contributed by Antoine Pitrou and Victor Stinner in bpo-22560@footnote{https://bugs.python.org/issue22560}.)

@item 
A new @ref{922,,loop.is_closed()} method to
check if the event loop is closed.
(Contributed by Victor Stinner in bpo-21326@footnote{https://bugs.python.org/issue21326}.)

@item 
A new @ref{4dd,,loop.create_task()}
to conveniently create and schedule a new @ref{317,,Task}
for a coroutine.  The @code{create_task} method is also used by all
asyncio functions that wrap coroutines into tasks, such as
@ref{440,,asyncio.wait()}, @ref{31a,,asyncio.gather()}, etc.
(Contributed by Victor Stinner.)

@item 
A new @ref{923,,transport.get_write_buffer_limits()}
method to inquire for @emph{high-} and @emph{low-} water limits of the flow
control.
(Contributed by Victor Stinner.)

@item 
The @code{async()} function is deprecated in favor of
@ref{318,,ensure_future()}.
(Contributed by Yury Selivanov.)

@item 
New @ref{924,,loop.set_task_factory()} and
@ref{925,,loop.get_task_factory()}
methods to customize the task factory that @ref{4dd,,loop.create_task()} method uses.  (Contributed by Yury
Selivanov.)

@item 
New @ref{926,,Queue.join()} and
@ref{927,,Queue.task_done()} queue methods.
(Contributed by Victor Stinner.)

@item 
The @code{JoinableQueue} class was removed, in favor of the
@ref{590,,asyncio.Queue} class.
(Contributed by Victor Stinner.)
@end itemize

Updates in 3.5.1:


@itemize *

@item 
The @ref{318,,ensure_future()} function and all functions that
use it, such as @ref{7bc,,loop.run_until_complete()},
now accept all kinds of @ref{7bd,,awaitable objects}.
(Contributed by Yury Selivanov.)

@item 
New @ref{348,,run_coroutine_threadsafe()} function to submit
coroutines to event loops from other threads.
(Contributed by Vincent Michel.)

@item 
New @ref{7be,,Transport.is_closing()}
method to check if the transport is closing or closed.
(Contributed by Yury Selivanov.)

@item 
The @ref{630,,loop.create_server()}
method can now accept a list of hosts.
(Contributed by Yann Sionneau.)
@end itemize

Updates in 3.5.2:


@itemize *

@item 
New @ref{7bf,,loop.create_future()}
method to create Future objects.  This allows alternative event
loop implementations, such as
uvloop@footnote{https://github.com/MagicStack/uvloop}, to provide a faster
@ref{316,,asyncio.Future} implementation.
(Contributed by Yury Selivanov.)

@item 
New @ref{7c0,,loop.get_exception_handler()}
method to get the current exception handler.
(Contributed by Yury Selivanov.)

@item 
New @ref{7c1,,StreamReader.readuntil()}
method to read data from the stream until a separator bytes
sequence appears.
(Contributed by Mark Korenberg.)

@item 
The @ref{4e0,,loop.create_connection()}
and @ref{630,,loop.create_server()}
methods are optimized to avoid calling the system @code{getaddrinfo}
function if the address is already resolved.
(Contributed by A. Jesse Jiryu Davis.)

@item 
The @ref{928,,loop.sock_connect(sock@comma{} address)}
no longer requires the @emph{address} to be resolved prior to the call.
(Contributed by A. Jesse Jiryu Davis.)
@end itemize

@node bz2,cgi,asyncio<7>,Improved Modules<7>
@anchor{whatsnew/3 5 bz2}@anchor{929}
@subsubsection bz2


The @ref{92a,,BZ2Decompressor.decompress}
method now accepts an optional @emph{max_length} argument to limit the maximum
size of decompressed data. (Contributed by Nikolaus Rath in bpo-15955@footnote{https://bugs.python.org/issue15955}.)

@node cgi,cmath<2>,bz2,Improved Modules<7>
@anchor{whatsnew/3 5 cgi}@anchor{92b}
@subsubsection cgi


The @code{FieldStorage} class now supports the @ref{166,,context manager}
protocol.  (Contributed by Berker Peksag in bpo-20289@footnote{https://bugs.python.org/issue20289}.)

@node cmath<2>,code,cgi,Improved Modules<7>
@anchor{whatsnew/3 5 cmath}@anchor{92c}
@subsubsection cmath


A new function @ref{90e,,isclose()} provides a way to test for approximate
equality.  (Contributed by Chris Barker and Tal Einat in bpo-24270@footnote{https://bugs.python.org/issue24270}.)

@node code,collections<5>,cmath<2>,Improved Modules<7>
@anchor{whatsnew/3 5 code}@anchor{92d}
@subsubsection code


The @ref{92e,,InteractiveInterpreter.showtraceback()}
method now prints the full chained traceback, just like the interactive
interpreter.  (Contributed by Claudiu Popa in bpo-17442@footnote{https://bugs.python.org/issue17442}.)

@node collections<5>,collections abc<3>,code,Improved Modules<7>
@anchor{whatsnew/3 5 collections}@anchor{92f}
@subsubsection collections

@anchor{whatsnew/3 5 whatsnew-ordereddict}@anchor{8c7}
The @ref{4e5,,OrderedDict} class is now implemented in C, which
makes it 4 to 100 times faster.  (Contributed by Eric Snow in bpo-16991@footnote{https://bugs.python.org/issue16991}.)

@code{OrderedDict.items()},
@code{OrderedDict.keys()},
@code{OrderedDict.values()} views now support
@ref{2f6,,reversed()} iteration.
(Contributed by Serhiy Storchaka in bpo-19505@footnote{https://bugs.python.org/issue19505}.)

The @ref{7d3,,deque} class now defines
@ref{930,,index()}, @ref{931,,insert()}, and
@ref{932,,copy()}, and supports the @code{+} and @code{*} operators.
This allows deques to be recognized as a @ref{933,,MutableSequence}
and improves their substitutability for lists.
(Contributed by Raymond Hettinger in bpo-23704@footnote{https://bugs.python.org/issue23704}.)

Docstrings produced by @ref{3e4,,namedtuple()} can now be updated:

@example
Point = namedtuple('Point', ['x', 'y'])
Point.__doc__ += ': Cartesian coodinate'
Point.x.__doc__ = 'abscissa'
Point.y.__doc__ = 'ordinate'
@end example

(Contributed by Berker Peksag in bpo-24064@footnote{https://bugs.python.org/issue24064}.)

The @ref{934,,UserString} class now implements the
@code{__getnewargs__()}, @code{__rmod__()}, @ref{935,,casefold()},
@ref{936,,format_map()}, @ref{937,,isprintable()}, and @ref{938,,maketrans()}
methods to match the corresponding methods of @ref{2ef,,str}.
(Contributed by Joe Jevnik in bpo-22189@footnote{https://bugs.python.org/issue22189}.)

@node collections abc<3>,compileall<3>,collections<5>,Improved Modules<7>
@anchor{whatsnew/3 5 collections-abc}@anchor{939}
@subsubsection collections.abc


The @code{Sequence.index()} method now
accepts @emph{start} and @emph{stop} arguments to match the corresponding methods
of @ref{3e5,,tuple}, @ref{43a,,list}, etc.
(Contributed by Devin Jeanpierre in bpo-23086@footnote{https://bugs.python.org/issue23086}.)

A new @ref{93a,,Generator} abstract base class. (Contributed
by Stefan Behnel in bpo-24018@footnote{https://bugs.python.org/issue24018}.)

New @ref{93b,,Awaitable}, @ref{93c,,Coroutine},
@ref{93d,,AsyncIterator}, and
@ref{93e,,AsyncIterable} abstract base classes.
(Contributed by Yury Selivanov in bpo-24184@footnote{https://bugs.python.org/issue24184}.)

For earlier Python versions, a backport of the new ABCs is available in an
external PyPI package@footnote{https://pypi.org/project/backports_abc}.

@node compileall<3>,concurrent futures<4>,collections abc<3>,Improved Modules<7>
@anchor{whatsnew/3 5 compileall}@anchor{93f}
@subsubsection compileall


A new @ref{1f,,compileall} option, @code{-j @emph{N}}, allows running @emph{N} workers
simultaneously to perform parallel bytecode compilation.
The @ref{645,,compile_dir()} function has a corresponding @code{workers}
parameter.  (Contributed by Claudiu Popa in bpo-16104@footnote{https://bugs.python.org/issue16104}.)

Another new option, @code{-r}, allows controlling the maximum recursion
level for subdirectories.  (Contributed by Claudiu Popa in bpo-19628@footnote{https://bugs.python.org/issue19628}.)

The @code{-q} command line option can now be specified more than once, in
which case all output, including errors, will be suppressed.  The corresponding
@code{quiet} parameter in @ref{645,,compile_dir()},
@ref{940,,compile_file()}, and @ref{941,,compile_path()} can now
accept an integer value indicating the level of output suppression.
(Contributed by Thomas Kluyver in bpo-21338@footnote{https://bugs.python.org/issue21338}.)

@node concurrent futures<4>,configparser,compileall<3>,Improved Modules<7>
@anchor{whatsnew/3 5 concurrent-futures}@anchor{942}
@subsubsection concurrent.futures


The @ref{943,,Executor.map()} method now accepts a
@emph{chunksize} argument to allow batching of tasks to improve performance when
@ref{3da,,ProcessPoolExecutor()} is used.
(Contributed by Dan O’Reilly in bpo-11271@footnote{https://bugs.python.org/issue11271}.)

The number of workers in the @ref{1c7,,ThreadPoolExecutor}
constructor is optional now.  The default value is 5 times the number of CPUs.
(Contributed by Claudiu Popa in bpo-21527@footnote{https://bugs.python.org/issue21527}.)

@node configparser,contextlib<4>,concurrent futures<4>,Improved Modules<7>
@anchor{whatsnew/3 5 configparser}@anchor{944}
@subsubsection configparser


@ref{21,,configparser} now provides a way to customize the conversion
of values by specifying a dictionary of converters in the
@ref{759,,ConfigParser} constructor, or by defining them
as methods in @code{ConfigParser} subclasses.  Converters defined in
a parser instance are inherited by its section proxies.

Example:

@example
>>> import configparser
>>> conv = @{@}
>>> conv['list'] = lambda v: [e.strip() for e in v.split() if e.strip()]
>>> cfg = configparser.ConfigParser(converters=conv)
>>> cfg.read_string("""
... [s]
... list = a b c d e f g
... """)
>>> cfg.get('s', 'list')
'a b c d e f g'
>>> cfg.getlist('s', 'list')
['a', 'b', 'c', 'd', 'e', 'f', 'g']
>>> section = cfg['s']
>>> section.getlist('list')
['a', 'b', 'c', 'd', 'e', 'f', 'g']
@end example

(Contributed by Łukasz Langa in bpo-18159@footnote{https://bugs.python.org/issue18159}.)

@node contextlib<4>,csv<2>,configparser,Improved Modules<7>
@anchor{whatsnew/3 5 contextlib}@anchor{945}
@subsubsection contextlib


The new @ref{946,,redirect_stderr()} @ref{166,,context manager} (similar to
@ref{947,,redirect_stdout()}) makes it easier for utility scripts to
handle inflexible APIs that write their output to @ref{42d,,sys.stderr} and
don’t provide any options to redirect it:

@example
>>> import contextlib, io, logging
>>> f = io.StringIO()
>>> with contextlib.redirect_stderr(f):
...     logging.warning('warning')
...
>>> f.getvalue()
'WARNING:root:warning\n'
@end example

(Contributed by Berker Peksag in bpo-22389@footnote{https://bugs.python.org/issue22389}.)

@node csv<2>,curses<4>,contextlib<4>,Improved Modules<7>
@anchor{whatsnew/3 5 csv}@anchor{948}
@subsubsection csv


The @ref{949,,writerow()} method now supports arbitrary iterables,
not just sequences.  (Contributed by Serhiy Storchaka in bpo-23171@footnote{https://bugs.python.org/issue23171}.)

@node curses<4>,dbm<4>,csv<2>,Improved Modules<7>
@anchor{whatsnew/3 5 curses}@anchor{94a}
@subsubsection curses


The new @ref{94b,,update_lines_cols()} function updates the 
@geindex LINES
@geindex environment variable; LINES
@code{LINES}
and 
@geindex COLS
@geindex environment variable; COLS
@code{COLS} environment variables.  This is useful for detecting
manual screen resizing.  (Contributed by Arnon Yaari in bpo-4254@footnote{https://bugs.python.org/issue4254}.)

@node dbm<4>,difflib,curses<4>,Improved Modules<7>
@anchor{whatsnew/3 5 dbm}@anchor{94c}
@subsubsection dbm


@ref{5b4,,dumb.open} always creates a new database when the flag
has the value @code{"n"}.  (Contributed by Claudiu Popa in bpo-18039@footnote{https://bugs.python.org/issue18039}.)

@node difflib,distutils<6>,dbm<4>,Improved Modules<7>
@anchor{whatsnew/3 5 difflib}@anchor{94d}
@subsubsection difflib


The charset of HTML documents generated by
@ref{94e,,HtmlDiff.make_file()}
can now be customized by using a new @emph{charset} keyword-only argument.
The default charset of HTML document changed from @code{"ISO-8859-1"}
to @code{"utf-8"}.
(Contributed by Berker Peksag in bpo-2052@footnote{https://bugs.python.org/issue2052}.)

The @ref{94f,,diff_bytes()} function can now compare lists of byte
strings.  This fixes a regression from Python 2.
(Contributed by Terry J. Reedy and Greg Ward in bpo-17445@footnote{https://bugs.python.org/issue17445}.)

@node distutils<6>,doctest<2>,difflib,Improved Modules<7>
@anchor{whatsnew/3 5 distutils}@anchor{950}
@subsubsection distutils


Both the @code{build} and @code{build_ext} commands now accept a @code{-j} option to
enable parallel building of extension modules.
(Contributed by Antoine Pitrou in bpo-5309@footnote{https://bugs.python.org/issue5309}.)

The @ref{37,,distutils} module now supports @code{xz} compression, and can be
enabled by passing @code{xztar} as an argument to @code{bdist --format}.
(Contributed by Serhiy Storchaka in bpo-16314@footnote{https://bugs.python.org/issue16314}.)

@node doctest<2>,email<2>,distutils<6>,Improved Modules<7>
@anchor{whatsnew/3 5 doctest}@anchor{951}
@subsubsection doctest


The @ref{952,,DocTestSuite()} function returns an empty
@ref{953,,unittest.TestSuite} if @emph{module} contains no docstrings, instead of
raising @ref{17c,,ValueError}.  (Contributed by Glenn Jones in bpo-15916@footnote{https://bugs.python.org/issue15916}.)

@node email<2>,enum<5>,doctest<2>,Improved Modules<7>
@anchor{whatsnew/3 5 email}@anchor{954}
@subsubsection email


A new policy option @ref{955,,Policy.mangle_from_}
controls whether or not lines that start with @code{"From "} in email bodies are
prefixed with a @code{">"} character by generators.  The default is @code{True} for
@ref{7e2,,compat32} and @code{False} for all other policies.
(Contributed by Milan Oberkirch in bpo-20098@footnote{https://bugs.python.org/issue20098}.)

A new
@ref{956,,Message.get_content_disposition()}
method provides easy access to a canonical value for the
@code{Content-Disposition} header.
(Contributed by Abhilash Raj in bpo-21083@footnote{https://bugs.python.org/issue21083}.)

A new policy option @ref{957,,EmailPolicy.utf8}
can be set to @code{True} to encode email headers using the UTF-8 charset instead
of using encoded words.  This allows @code{Messages} to be formatted according to
@geindex RFC; RFC 6532
RFC 6532@footnote{https://tools.ietf.org/html/rfc6532.html} and used with an SMTP server that supports the 
@geindex RFC; RFC 6531
RFC 6531@footnote{https://tools.ietf.org/html/rfc6531.html}
@code{SMTPUTF8} extension.  (Contributed by R. David Murray in
bpo-24211@footnote{https://bugs.python.org/issue24211}.)

The @ref{958,,mime.text.MIMEText} constructor now
accepts a @ref{959,,charset.Charset} instance.
(Contributed by Claude Paroz and Berker Peksag in bpo-16324@footnote{https://bugs.python.org/issue16324}.)

@node enum<5>,faulthandler<3>,email<2>,Improved Modules<7>
@anchor{whatsnew/3 5 enum}@anchor{95a}
@subsubsection enum


The @ref{658,,Enum} callable has a new parameter @emph{start} to
specify the initial number of enum values if only @emph{names} are provided:

@example
>>> Animal = enum.Enum('Animal', 'cat dog', start=10)
>>> Animal.cat
<Animal.cat: 10>
>>> Animal.dog
<Animal.dog: 11>
@end example

(Contributed by Ethan Furman in bpo-21706@footnote{https://bugs.python.org/issue21706}.)

@node faulthandler<3>,functools<3>,enum<5>,Improved Modules<7>
@anchor{whatsnew/3 5 faulthandler}@anchor{95b}
@subsubsection faulthandler


The @ref{7ea,,enable()}, @ref{95c,,register()},
@ref{95d,,dump_traceback()} and
@ref{95e,,dump_traceback_later()} functions now accept file
descriptors in addition to file-like objects.
(Contributed by Wei Wu in bpo-23566@footnote{https://bugs.python.org/issue23566}.)

@node functools<3>,glob<2>,faulthandler<3>,Improved Modules<7>
@anchor{whatsnew/3 5 functools}@anchor{95f}
@subsubsection functools

@anchor{whatsnew/3 5 whatsnew-lrucache}@anchor{8ca}
Most of the @ref{4f3,,lru_cache()} machinery is now implemented in C, making
it significantly faster.  (Contributed by Matt Joiner, Alexey Kachayev, and
Serhiy Storchaka in bpo-14373@footnote{https://bugs.python.org/issue14373}.)

@node glob<2>,gzip<2>,functools<3>,Improved Modules<7>
@anchor{whatsnew/3 5 glob}@anchor{960}
@subsubsection glob


The @ref{289,,iglob()} and @ref{288,,glob()} functions now support recursive
search in subdirectories, using the @code{"**"} pattern.
(Contributed by Serhiy Storchaka in bpo-13968@footnote{https://bugs.python.org/issue13968}.)

@node gzip<2>,heapq,glob<2>,Improved Modules<7>
@anchor{whatsnew/3 5 gzip}@anchor{961}
@subsubsection gzip


The @emph{mode} argument of the @ref{43f,,GzipFile} constructor now
accepts @code{"x"} to request exclusive creation.
(Contributed by Tim Heaney in bpo-19222@footnote{https://bugs.python.org/issue19222}.)

@node heapq,http<2>,gzip<2>,Improved Modules<7>
@anchor{whatsnew/3 5 heapq}@anchor{962}
@subsubsection heapq


Element comparison in @ref{963,,merge()} can now be customized by
passing a @ref{964,,key function} in a new optional @emph{key} keyword argument,
and a new optional @emph{reverse} keyword argument can be used to reverse element
comparison:

@example
>>> import heapq
>>> a = ['9', '777', '55555']
>>> b = ['88', '6666']
>>> list(heapq.merge(a, b, key=len))
['9', '88', '777', '6666', '55555']
>>> list(heapq.merge(reversed(a), reversed(b), key=len, reverse=True))
['55555', '6666', '777', '88', '9']
@end example

(Contributed by Raymond Hettinger in bpo-13742@footnote{https://bugs.python.org/issue13742}.)

@node http<2>,http client<3>,heapq,Improved Modules<7>
@anchor{whatsnew/3 5 http}@anchor{965}
@subsubsection http


A new @ref{3ef,,HTTPStatus} enum that defines a set of
HTTP status codes, reason phrases and long descriptions written in English.
(Contributed by Demian Brecht in bpo-21793@footnote{https://bugs.python.org/issue21793}.)

@node http client<3>,idlelib and IDLE<3>,http<2>,Improved Modules<7>
@anchor{whatsnew/3 5 http-client}@anchor{966}
@subsubsection http.client


@ref{967,,HTTPConnection.getresponse()}
now raises a @ref{968,,RemoteDisconnected} exception when a
remote server connection is closed unexpectedly.  Additionally, if a
@ref{969,,ConnectionError} (of which @code{RemoteDisconnected}
is a subclass) is raised, the client socket is now closed automatically,
and will reconnect on the next request:

@example
import http.client
conn = http.client.HTTPConnection('www.python.org')
for retries in range(3):
    try:
        conn.request('GET', '/')
        resp = conn.getresponse()
    except http.client.RemoteDisconnected:
        pass
@end example

(Contributed by Martin Panter in bpo-3566@footnote{https://bugs.python.org/issue3566}.)

@node idlelib and IDLE<3>,imaplib<2>,http client<3>,Improved Modules<7>
@anchor{whatsnew/3 5 idlelib-and-idle}@anchor{96a}
@subsubsection idlelib and IDLE


Since idlelib implements the IDLE shell and editor and is not intended for
import by other programs, it gets improvements with every release.  See
@code{Lib/idlelib/NEWS.txt} for a cumulative list of changes since 3.4.0,
as well as changes made in future 3.5.x releases. This file is also available
from the IDLE Help ‣ About IDLE dialog.

@node imaplib<2>,imghdr,idlelib and IDLE<3>,Improved Modules<7>
@anchor{whatsnew/3 5 imaplib}@anchor{96b}
@subsubsection imaplib


The @ref{3f2,,IMAP4} class now supports the @ref{166,,context manager} protocol.
When used in a @ref{168,,with} statement, the IMAP4 @code{LOGOUT}
command will be called automatically at the end of the block.
(Contributed by Tarek Ziadé and Serhiy Storchaka in bpo-4972@footnote{https://bugs.python.org/issue4972}.)

The @ref{93,,imaplib} module now supports 
@geindex RFC; RFC 5161
RFC 5161@footnote{https://tools.ietf.org/html/rfc5161.html} (ENABLE Extension)
and 
@geindex RFC; RFC 6855
RFC 6855@footnote{https://tools.ietf.org/html/rfc6855.html} (UTF-8 Support) via the @ref{96c,,IMAP4.enable()}
method.  A new @ref{96d,,IMAP4.utf8_enabled}
attribute tracks whether or not 
@geindex RFC; RFC 6855
RFC 6855@footnote{https://tools.ietf.org/html/rfc6855.html} support is enabled.
(Contributed by Milan Oberkirch, R. David Murray, and Maciej Szulik in
bpo-21800@footnote{https://bugs.python.org/issue21800}.)

The @ref{93,,imaplib} module now automatically encodes non-ASCII string usernames
and passwords using UTF-8, as recommended by the RFCs.  (Contributed by Milan
Oberkirch in bpo-21800@footnote{https://bugs.python.org/issue21800}.)

@node imghdr,importlib<6>,imaplib<2>,Improved Modules<7>
@anchor{whatsnew/3 5 imghdr}@anchor{96e}
@subsubsection imghdr


The @ref{96f,,what()} function now recognizes the
OpenEXR@footnote{http://www.openexr.com} format
(contributed by Martin Vignali and Claudiu Popa in bpo-20295@footnote{https://bugs.python.org/issue20295}),
and the WebP@footnote{https://en.wikipedia.org/wiki/WebP} format
(contributed by Fabrice Aneche and Claudiu Popa in bpo-20197@footnote{https://bugs.python.org/issue20197}.)

@node importlib<6>,inspect<6>,imghdr,Improved Modules<7>
@anchor{whatsnew/3 5 importlib}@anchor{970}
@subsubsection importlib


The @ref{7f5,,util.LazyLoader} class allows for
lazy loading of modules in applications where startup time is important.
(Contributed by Brett Cannon in bpo-17621@footnote{https://bugs.python.org/issue17621}.)

The @ref{971,,abc.InspectLoader.source_to_code()}
method is now a static method.  This makes it easier to initialize a module
object with code compiled from a string by running
@code{exec(code, module.__dict__)}.
(Contributed by Brett Cannon in bpo-21156@footnote{https://bugs.python.org/issue21156}.)

The new @ref{972,,util.module_from_spec()}
function is now the preferred way to create a new module.  As opposed to
creating a @ref{973,,types.ModuleType} instance directly, this new function
will set the various import-controlled attributes based on the passed-in
spec object.  (Contributed by Brett Cannon in bpo-20383@footnote{https://bugs.python.org/issue20383}.)

@node inspect<6>,io<3>,importlib<6>,Improved Modules<7>
@anchor{whatsnew/3 5 inspect}@anchor{974}
@subsubsection inspect


Both the @ref{975,,Signature} and @ref{976,,Parameter} classes are
now picklable and hashable.  (Contributed by Yury Selivanov in bpo-20726@footnote{https://bugs.python.org/issue20726}
and bpo-20334@footnote{https://bugs.python.org/issue20334}.)

A new
@ref{977,,BoundArguments.apply_defaults()}
method provides a way to set default values for missing arguments:

@example
>>> def foo(a, b='ham', *args): pass
>>> ba = inspect.signature(foo).bind('spam')
>>> ba.apply_defaults()
>>> ba.arguments
OrderedDict([('a', 'spam'), ('b', 'ham'), ('args', ())])
@end example

(Contributed by Yury Selivanov in bpo-24190@footnote{https://bugs.python.org/issue24190}.)

A new class method
@ref{1c3,,Signature.from_callable()} makes
subclassing of @ref{975,,Signature} easier.  (Contributed
by Yury Selivanov and Eric Snow in bpo-17373@footnote{https://bugs.python.org/issue17373}.)

The @ref{1c1,,signature()} function now accepts a @emph{follow_wrapped}
optional keyword argument, which, when set to @code{False}, disables automatic
following of @code{__wrapped__} links.
(Contributed by Yury Selivanov in bpo-20691@footnote{https://bugs.python.org/issue20691}.)

A set of new functions to inspect
@ref{8cf,,coroutine functions} and
@ref{3cf,,coroutine objects} has been added:
@ref{978,,iscoroutine()}, @ref{979,,iscoroutinefunction()},
@ref{97a,,isawaitable()}, @ref{97b,,getcoroutinelocals()},
and @ref{97c,,getcoroutinestate()}.
(Contributed by Yury Selivanov in bpo-24017@footnote{https://bugs.python.org/issue24017} and bpo-24400@footnote{https://bugs.python.org/issue24400}.)

The @ref{97d,,stack()}, @ref{97e,,trace()},
@ref{97f,,getouterframes()}, and @ref{980,,getinnerframes()}
functions now return a list of named tuples.
(Contributed by Daniel Shahaf in bpo-16808@footnote{https://bugs.python.org/issue16808}.)

@node io<3>,ipaddress<3>,inspect<6>,Improved Modules<7>
@anchor{whatsnew/3 5 io}@anchor{981}
@subsubsection io


A new @ref{982,,BufferedIOBase.readinto1()}
method, that uses at most one call to the underlying raw stream’s
@ref{983,,RawIOBase.read()} or
@ref{984,,RawIOBase.readinto()} methods.
(Contributed by Nikolaus Rath in bpo-20578@footnote{https://bugs.python.org/issue20578}.)

@node ipaddress<3>,json<2>,io<3>,Improved Modules<7>
@anchor{whatsnew/3 5 ipaddress}@anchor{985}
@subsubsection ipaddress


Both the @ref{670,,IPv4Network} and @ref{66f,,IPv6Network} classes
now accept an @code{(address, netmask)} tuple argument, so as to easily construct
network objects from existing addresses:

@example
>>> import ipaddress
>>> ipaddress.IPv4Network(('127.0.0.0', 8))
IPv4Network('127.0.0.0/8')
>>> ipaddress.IPv4Network(('127.0.0.0', '255.0.0.0'))
IPv4Network('127.0.0.0/8')
@end example

(Contributed by Peter Moody and Antoine Pitrou in bpo-16531@footnote{https://bugs.python.org/issue16531}.)

A new @code{reverse_pointer} attribute for the
@ref{670,,IPv4Network} and @ref{66f,,IPv6Network} classes
returns the name of the reverse DNS PTR record:

@example
>>> import ipaddress
>>> addr = ipaddress.IPv4Address('127.0.0.1')
>>> addr.reverse_pointer
'1.0.0.127.in-addr.arpa'
>>> addr6 = ipaddress.IPv6Address('::1')
>>> addr6.reverse_pointer
'1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa'
@end example

(Contributed by Leon Weber in bpo-20480@footnote{https://bugs.python.org/issue20480}.)

@node json<2>,linecache<2>,ipaddress<3>,Improved Modules<7>
@anchor{whatsnew/3 5 json}@anchor{986}
@subsubsection json


The @ref{a1,,json.tool} command line interface now preserves the order of keys in
JSON objects passed in input.  The new @code{--sort-keys} option can be used
to sort the keys alphabetically. (Contributed by Berker Peksag
in bpo-21650@footnote{https://bugs.python.org/issue21650}.)

JSON decoder now raises @ref{987,,JSONDecodeError} instead of
@ref{17c,,ValueError} to provide better context information about the error.
(Contributed by Serhiy Storchaka in bpo-19361@footnote{https://bugs.python.org/issue19361}.)

@node linecache<2>,locale<3>,json<2>,Improved Modules<7>
@anchor{whatsnew/3 5 linecache}@anchor{988}
@subsubsection linecache


A new @ref{989,,lazycache()} function can be used to capture information
about a non-file-based module to permit getting its lines later via
@ref{98a,,getline()}. This avoids doing I/O until a line is actually
needed, without having to carry the module globals around indefinitely.
(Contributed by Robert Collins in bpo-17911@footnote{https://bugs.python.org/issue17911}.)

@node locale<3>,logging<4>,linecache<2>,Improved Modules<7>
@anchor{whatsnew/3 5 locale}@anchor{98b}
@subsubsection locale


A new @ref{98c,,delocalize()} function can be used to convert a string into
a normalized number string, taking the @code{LC_NUMERIC} settings into account:

@example
>>> import locale
>>> locale.setlocale(locale.LC_NUMERIC, 'de_DE.UTF-8')
'de_DE.UTF-8'
>>> locale.delocalize('1.234,56')
'1234.56'
>>> locale.setlocale(locale.LC_NUMERIC, 'en_US.UTF-8')
'en_US.UTF-8'
>>> locale.delocalize('1,234.56')
'1234.56'
@end example

(Contributed by Cédric Krier in bpo-13918@footnote{https://bugs.python.org/issue13918}.)

@node logging<4>,lzma,locale<3>,Improved Modules<7>
@anchor{whatsnew/3 5 logging}@anchor{98d}
@subsubsection logging


All logging methods (@ref{676,,Logger} @ref{98e,,log()},
@ref{98f,,exception()}, @ref{990,,critical()},
@ref{991,,debug()}, etc.), now accept exception instances
as an @emph{exc_info} argument, in addition to boolean values and exception
tuples:

@example
>>> import logging
>>> try:
...     1/0
... except ZeroDivisionError as ex:
...     logging.error('exception', exc_info=ex)
ERROR:root:exception
@end example

(Contributed by Yury Selivanov in bpo-20537@footnote{https://bugs.python.org/issue20537}.)

The @ref{992,,handlers.HTTPHandler} class now
accepts an optional @ref{2bf,,ssl.SSLContext} instance to configure SSL
settings used in an HTTP connection.
(Contributed by Alex Gaynor in bpo-22788@footnote{https://bugs.python.org/issue22788}.)

The @ref{993,,handlers.QueueListener} class now
takes a @emph{respect_handler_level} keyword argument which, if set to @code{True},
will pass messages to handlers taking handler levels into account.
(Contributed by Vinay Sajip.)

@node lzma,math<6>,logging<4>,Improved Modules<7>
@anchor{whatsnew/3 5 lzma}@anchor{994}
@subsubsection lzma


The @ref{995,,LZMADecompressor.decompress()}
method now accepts an optional @emph{max_length} argument to limit the maximum
size of decompressed data.
(Contributed by Martin Panter in bpo-15955@footnote{https://bugs.python.org/issue15955}.)

@node math<6>,multiprocessing<5>,lzma,Improved Modules<7>
@anchor{whatsnew/3 5 math}@anchor{996}
@subsubsection math


Two new constants have been added to the @ref{ad,,math} module: @ref{7cb,,inf}
and @ref{7cc,,nan}.  (Contributed by Mark Dickinson in bpo-23185@footnote{https://bugs.python.org/issue23185}.)

A new function @ref{90d,,isclose()} provides a way to test for approximate
equality. (Contributed by Chris Barker and Tal Einat in bpo-24270@footnote{https://bugs.python.org/issue24270}.)

A new @ref{401,,gcd()} function has been added.  The @code{fractions.gcd()}
function is now deprecated. (Contributed by Mark Dickinson and Serhiy
Storchaka in bpo-22486@footnote{https://bugs.python.org/issue22486}.)

@node multiprocessing<5>,operator<2>,math<6>,Improved Modules<7>
@anchor{whatsnew/3 5 multiprocessing}@anchor{997}
@subsubsection multiprocessing


@ref{998,,sharedctypes.synchronized()}
objects now support the @ref{166,,context manager} protocol.
(Contributed by Charles-François Natali in bpo-21565@footnote{https://bugs.python.org/issue21565}.)

@node operator<2>,os<8>,multiprocessing<5>,Improved Modules<7>
@anchor{whatsnew/3 5 operator}@anchor{999}
@subsubsection operator


@ref{99a,,attrgetter()}, @ref{578,,itemgetter()},
and @ref{99b,,methodcaller()} objects now support pickling.
(Contributed by Josh Rosenberg and Serhiy Storchaka in bpo-22955@footnote{https://bugs.python.org/issue22955}.)

New @ref{99c,,matmul()} and @ref{99d,,imatmul()} functions
to perform matrix multiplication.
(Contributed by Benjamin Peterson in bpo-21176@footnote{https://bugs.python.org/issue21176}.)

@node os<8>,pathlib<6>,operator<2>,Improved Modules<7>
@anchor{whatsnew/3 5 os}@anchor{99e}
@subsubsection os


The new @ref{576,,scandir()} function returning an iterator of
@ref{79a,,DirEntry} objects has been added.  If possible, @ref{576,,scandir()}
extracts file attributes while scanning a directory, removing the need to
perform subsequent system calls to determine file type or attributes, which may
significantly improve performance.  (Contributed by Ben Hoyt with the help
of Victor Stinner in bpo-22524@footnote{https://bugs.python.org/issue22524}.)

On Windows, a new
@ref{99f,,stat_result.st_file_attributes}
attribute is now available.  It corresponds to the @code{dwFileAttributes} member
of the @code{BY_HANDLE_FILE_INFORMATION} structure returned by
@code{GetFileInformationByHandle()}.  (Contributed by Ben Hoyt in bpo-21719@footnote{https://bugs.python.org/issue21719}.)

The @ref{17f,,urandom()} function now uses the @code{getrandom()} syscall on Linux 3.17
or newer, and @code{getentropy()} on OpenBSD 5.6 and newer, removing the need to
use @code{/dev/urandom} and avoiding failures due to potential file descriptor
exhaustion.  (Contributed by Victor Stinner in bpo-22181@footnote{https://bugs.python.org/issue22181}.)

New @ref{9a0,,get_blocking()} and @ref{9a1,,set_blocking()} functions allow
getting and setting a file descriptor’s blocking mode (@ref{9a2,,O_NONBLOCK}.)
(Contributed by Victor Stinner in bpo-22054@footnote{https://bugs.python.org/issue22054}.)

The @ref{9a3,,truncate()} and @ref{8eb,,ftruncate()} functions are now supported
on Windows.  (Contributed by Steve Dower in bpo-23668@footnote{https://bugs.python.org/issue23668}.)

There is a new @ref{9a4,,os.path.commonpath()} function returning the longest
common sub-path of each passed pathname.  Unlike the
@ref{9a5,,os.path.commonprefix()} function, it always returns a valid
path:

@example
>>> os.path.commonprefix(['/usr/lib', '/usr/local/lib'])
'/usr/l'

>>> os.path.commonpath(['/usr/lib', '/usr/local/lib'])
'/usr'
@end example

(Contributed by Rafik Draoui and Serhiy Storchaka in bpo-10395@footnote{https://bugs.python.org/issue10395}.)

@node pathlib<6>,pickle<3>,os<8>,Improved Modules<7>
@anchor{whatsnew/3 5 pathlib}@anchor{9a6}
@subsubsection pathlib


The new @ref{9a7,,Path.samefile()} method can be used
to check whether the path points to the same file as another path, which can
be either another @ref{522,,Path} object, or a string:

@example
>>> import pathlib
>>> p1 = pathlib.Path('/etc/hosts')
>>> p2 = pathlib.Path('/etc/../etc/hosts')
>>> p1.samefile(p2)
True
@end example

(Contributed by Vajrasky Kok and Antoine Pitrou in bpo-19775@footnote{https://bugs.python.org/issue19775}.)

The @ref{9a8,,Path.mkdir()} method now accepts a new optional
@emph{exist_ok} argument to match @code{mkdir -p} and @ref{68c,,os.makedirs()}
functionality.  (Contributed by Berker Peksag in bpo-21539@footnote{https://bugs.python.org/issue21539}.)

There is a new @ref{9a9,,Path.expanduser()} method to
expand @code{~} and @code{~user} prefixes.  (Contributed by Serhiy Storchaka and
Claudiu Popa in bpo-19776@footnote{https://bugs.python.org/issue19776}.)

A new @ref{9aa,,Path.home()} class method can be used to get
a @ref{522,,Path} instance representing the user’s home
directory.
(Contributed by Victor Salgado and Mayank Tripathi in bpo-19777@footnote{https://bugs.python.org/issue19777}.)

New @ref{9ab,,Path.write_text()},
@ref{9ac,,Path.read_text()},
@ref{9ad,,Path.write_bytes()},
@ref{9ae,,Path.read_bytes()} methods to simplify
read/write operations on files.

The following code snippet will create or rewrite existing file
@code{~/spam42}:

@example
>>> import pathlib
>>> p = pathlib.Path('~/spam42')
>>> p.expanduser().write_text('ham')
3
@end example

(Contributed by Christopher Welborn in bpo-20218@footnote{https://bugs.python.org/issue20218}.)

@node pickle<3>,poplib<2>,pathlib<6>,Improved Modules<7>
@anchor{whatsnew/3 5 pickle}@anchor{9af}
@subsubsection pickle


Nested objects, such as unbound methods or nested classes, can now be pickled
using @ref{80a,,pickle protocols} older than protocol version 4.
Protocol version 4 already supports these cases.  (Contributed by Serhiy
Storchaka in bpo-23611@footnote{https://bugs.python.org/issue23611}.)

@node poplib<2>,re<4>,pickle<3>,Improved Modules<7>
@anchor{whatsnew/3 5 poplib}@anchor{9b0}
@subsubsection poplib


A new @ref{9b1,,POP3.utf8()} command enables 
@geindex RFC; RFC 6856
RFC 6856@footnote{https://tools.ietf.org/html/rfc6856.html}
(Internationalized Email) support, if a POP server supports it.
(Contributed by Milan OberKirch in bpo-21804@footnote{https://bugs.python.org/issue21804}.)

@node re<4>,readline<2>,poplib<2>,Improved Modules<7>
@anchor{whatsnew/3 5 re}@anchor{9b2}
@subsubsection re


References and conditional references to groups with fixed length are now
allowed in lookbehind assertions:

@example
>>> import re
>>> pat = re.compile(r'(a|b).(?<=\1)c')
>>> pat.match('aac')
<_sre.SRE_Match object; span=(0, 3), match='aac'>
>>> pat.match('bbc')
<_sre.SRE_Match object; span=(0, 3), match='bbc'>
@end example

(Contributed by Serhiy Storchaka in bpo-9179@footnote{https://bugs.python.org/issue9179}.)

The number of capturing groups in regular expressions is no longer limited to
100.  (Contributed by Serhiy Storchaka in bpo-22437@footnote{https://bugs.python.org/issue22437}.)

The @ref{72e,,sub()} and @ref{9b3,,subn()} functions now replace unmatched
groups with empty strings instead of raising an exception.
(Contributed by Serhiy Storchaka in bpo-1519638@footnote{https://bugs.python.org/issue1519638}.)

The @ref{9b4,,re.error} exceptions have new attributes,
@ref{9b5,,msg}, @ref{9b6,,pattern},
@ref{9b7,,pos}, @ref{9b8,,lineno},
and @ref{9b9,,colno}, that provide better context
information about the error:

@example
>>> re.compile("""
...     (?x)
...     .++
... """)
Traceback (most recent call last):
   ...
sre_constants.error: multiple repeat at position 16 (line 3, column 7)
@end example

(Contributed by Serhiy Storchaka in bpo-22578@footnote{https://bugs.python.org/issue22578}.)

@node readline<2>,selectors,re<4>,Improved Modules<7>
@anchor{whatsnew/3 5 readline}@anchor{9ba}
@subsubsection readline


A new @ref{9bb,,append_history_file()} function can be used to append
the specified number of trailing elements in history to the given file.
(Contributed by Bruno Cauet in bpo-22940@footnote{https://bugs.python.org/issue22940}.)

@node selectors,shutil<2>,readline<2>,Improved Modules<7>
@anchor{whatsnew/3 5 selectors}@anchor{9bc}
@subsubsection selectors


The new @ref{70f,,DevpollSelector} supports efficient
@code{/dev/poll} polling on Solaris.
(Contributed by Giampaolo Rodola’ in bpo-18931@footnote{https://bugs.python.org/issue18931}.)

@node shutil<2>,signal<3>,selectors,Improved Modules<7>
@anchor{whatsnew/3 5 shutil}@anchor{9bd}
@subsubsection shutil


The @ref{572,,move()} function now accepts a @emph{copy_function} argument,
allowing, for example, the @ref{570,,copy()} function to be used instead of
the default @ref{571,,copy2()} if there is a need to ignore file metadata
when moving.
(Contributed by Claudiu Popa in bpo-19840@footnote{https://bugs.python.org/issue19840}.)

The @ref{539,,make_archive()} function now supports the @emph{xztar} format.
(Contributed by Serhiy Storchaka in bpo-5411@footnote{https://bugs.python.org/issue5411}.)

@node signal<3>,smtpd,shutil<2>,Improved Modules<7>
@anchor{whatsnew/3 5 signal}@anchor{9be}
@subsubsection signal


On Windows, the @ref{69d,,set_wakeup_fd()} function now also supports
socket handles.  (Contributed by Victor Stinner in bpo-22018@footnote{https://bugs.python.org/issue22018}.)

Various @code{SIG*} constants in the @ref{e5,,signal} module have been converted into
@ref{76,,Enums}.  This allows meaningful names to be printed
during debugging, instead of integer “magic numbers”.
(Contributed by Giampaolo Rodola’ in bpo-21076@footnote{https://bugs.python.org/issue21076}.)

@node smtpd,smtplib<2>,signal<3>,Improved Modules<7>
@anchor{whatsnew/3 5 smtpd}@anchor{9bf}
@subsubsection smtpd


Both the @ref{898,,SMTPServer} and @ref{897,,SMTPChannel} classes now
accept a @emph{decode_data} keyword argument to determine if the @code{DATA} portion of
the SMTP transaction is decoded using the @code{"utf-8"} codec or is instead
provided to the
@ref{899,,SMTPServer.process_message()}
method as a byte string.  The default is @code{True} for backward compatibility
reasons, but will change to @code{False} in Python 3.6.  If @emph{decode_data} is set
to @code{False}, the @code{process_message} method must be prepared to accept keyword
arguments.
(Contributed by Maciej Szulik in bpo-19662@footnote{https://bugs.python.org/issue19662}.)

The @ref{898,,SMTPServer} class now advertises the @code{8BITMIME} extension
(
@geindex RFC; RFC 6152
RFC 6152@footnote{https://tools.ietf.org/html/rfc6152.html}) if @emph{decode_data} has been set @code{True}.  If the client
specifies @code{BODY=8BITMIME} on the @code{MAIL} command, it is passed to
@ref{899,,SMTPServer.process_message()}
via the @emph{mail_options} keyword.
(Contributed by Milan Oberkirch and R.  David Murray in bpo-21795@footnote{https://bugs.python.org/issue21795}.)

The @ref{898,,SMTPServer} class now also supports the @code{SMTPUTF8}
extension (
@geindex RFC; RFC 6531
RFC 6531@footnote{https://tools.ietf.org/html/rfc6531.html}: Internationalized Email).  If the client specified
@code{SMTPUTF8 BODY=8BITMIME} on the @code{MAIL} command, they are passed to
@ref{899,,SMTPServer.process_message()}
via the @emph{mail_options} keyword.  It is the responsibility of the
@code{process_message} method to correctly handle the @code{SMTPUTF8} data.
(Contributed by Milan Oberkirch in bpo-21725@footnote{https://bugs.python.org/issue21725}.)

It is now possible to provide, directly or via name resolution, IPv6
addresses in the @ref{898,,SMTPServer} constructor, and have it
successfully connect.  (Contributed by Milan Oberkirch in bpo-14758@footnote{https://bugs.python.org/issue14758}.)

@node smtplib<2>,sndhdr,smtpd,Improved Modules<7>
@anchor{whatsnew/3 5 smtplib}@anchor{9c0}
@subsubsection smtplib


A new @ref{9c1,,SMTP.auth()} method provides a convenient way to
implement custom authentication mechanisms. (Contributed by Milan
Oberkirch in bpo-15014@footnote{https://bugs.python.org/issue15014}.)

The @ref{9c2,,SMTP.set_debuglevel()} method now
accepts an additional debuglevel (2), which enables timestamps in debug
messages. (Contributed by Gavin Chappell and Maciej Szulik in bpo-16914@footnote{https://bugs.python.org/issue16914}.)

Both the @ref{9c3,,SMTP.sendmail()} and
@ref{9c4,,SMTP.send_message()} methods now
support 
@geindex RFC; RFC 6531
RFC 6531@footnote{https://tools.ietf.org/html/rfc6531.html} (SMTPUTF8).
(Contributed by Milan Oberkirch and R. David Murray in bpo-22027@footnote{https://bugs.python.org/issue22027}.)

@node sndhdr,socket<8>,smtplib<2>,Improved Modules<7>
@anchor{whatsnew/3 5 sndhdr}@anchor{9c5}
@subsubsection sndhdr


The @ref{9c6,,what()} and @ref{9c7,,whathdr()} functions  now return
a @ref{3e4,,namedtuple()}.  (Contributed by Claudiu Popa in
bpo-18615@footnote{https://bugs.python.org/issue18615}.)

@node socket<8>,ssl<7>,sndhdr,Improved Modules<7>
@anchor{whatsnew/3 5 socket}@anchor{9c8}
@subsubsection socket


Functions with timeouts now use a monotonic clock, instead of a system clock.
(Contributed by Victor Stinner in bpo-22043@footnote{https://bugs.python.org/issue22043}.)

A new @ref{9c9,,socket.sendfile()} method allows
sending a file over a socket by using the high-performance @ref{62c,,os.sendfile()}
function on UNIX, resulting in uploads being from 2 to 3 times faster than when
using plain @ref{903,,socket.send()}.
(Contributed by Giampaolo Rodola’ in bpo-17552@footnote{https://bugs.python.org/issue17552}.)

The @ref{904,,socket.sendall()} method no longer resets the
socket timeout every time bytes are received or sent.  The socket timeout is
now the maximum total duration to send all data.
(Contributed by Victor Stinner in bpo-23853@footnote{https://bugs.python.org/issue23853}.)

The @emph{backlog} argument of the @ref{9ca,,socket.listen()}
method is now optional.  By default it is set to
@ref{9cb,,SOMAXCONN} or to @code{128}, whichever is less.
(Contributed by Charles-François Natali in bpo-21455@footnote{https://bugs.python.org/issue21455}.)

@node ssl<7>,sqlite3<5>,socket<8>,Improved Modules<7>
@anchor{whatsnew/3 5 ssl}@anchor{9cc}
@subsubsection ssl


@menu
* Memory BIO Support:: 
* Application-Layer Protocol Negotiation Support:: 
* Other Changes:: 

@end menu

@node Memory BIO Support,Application-Layer Protocol Negotiation Support,,ssl<7>
@anchor{whatsnew/3 5 memory-bio-support}@anchor{9cd}@anchor{whatsnew/3 5 whatsnew-sslmemorybio}@anchor{8c8}
@subsubsection Memory BIO Support


(Contributed by Geert Jansen in bpo-21965@footnote{https://bugs.python.org/issue21965}.)

The new @ref{6af,,SSLObject} class has been added to provide SSL protocol
support for cases when the network I/O capabilities of @ref{3d6,,SSLSocket}
are not necessary or are suboptimal.  @code{SSLObject} represents
an SSL protocol instance, but does not implement any network I/O methods, and
instead provides a memory buffer interface.  The new @ref{9ce,,MemoryBIO}
class can be used to pass data between Python and an SSL protocol instance.

The memory BIO SSL support is primarily intended to be used in frameworks
implementing asynchronous I/O for which @ref{3d6,,SSLSocket}’s readiness
model (“select/poll”) is inefficient.

A new @ref{6b0,,SSLContext.wrap_bio()} method can be used
to create a new @code{SSLObject} instance.

@node Application-Layer Protocol Negotiation Support,Other Changes,Memory BIO Support,ssl<7>
@anchor{whatsnew/3 5 application-layer-protocol-negotiation-support}@anchor{9cf}
@subsubsection Application-Layer Protocol Negotiation Support


(Contributed by Benjamin Peterson in bpo-20188@footnote{https://bugs.python.org/issue20188}.)

Where OpenSSL support is present, the @ref{ee,,ssl} module now implements
the @emph{Application-Layer Protocol Negotiation} TLS extension as described
in 
@geindex RFC; RFC 7301
RFC 7301@footnote{https://tools.ietf.org/html/rfc7301.html}.

The new @ref{9d0,,SSLContext.set_alpn_protocols()}
can be used to specify which protocols a socket should advertise during
the TLS handshake.

The new
@ref{9d1,,SSLSocket.selected_alpn_protocol()}
returns the protocol that was selected during the TLS handshake.
The @ref{9d2,,HAS_ALPN} flag indicates whether ALPN support is present.

@node Other Changes,,Application-Layer Protocol Negotiation Support,ssl<7>
@anchor{whatsnew/3 5 other-changes}@anchor{9d3}
@subsubsection Other Changes


There is a new @ref{9d4,,SSLSocket.version()} method to
query the actual protocol version in use.
(Contributed by Antoine Pitrou in bpo-20421@footnote{https://bugs.python.org/issue20421}.)

The @ref{3d6,,SSLSocket} class now implements
a @code{SSLSocket.sendfile()} method.
(Contributed by Giampaolo Rodola’ in bpo-17552@footnote{https://bugs.python.org/issue17552}.)

The @code{SSLSocket.send()} method now raises either
the @ref{9d5,,ssl.SSLWantReadError} or @ref{9d6,,ssl.SSLWantWriteError} exception on a
non-blocking socket if the operation would block. Previously, it would return
@code{0}.  (Contributed by Nikolaus Rath in bpo-20951@footnote{https://bugs.python.org/issue20951}.)

The @ref{9d7,,cert_time_to_seconds()} function now interprets the input time
as UTC and not as local time, per 
@geindex RFC; RFC 5280
RFC 5280@footnote{https://tools.ietf.org/html/rfc5280.html}.  Additionally, the return
value is always an @ref{1b1,,int}. (Contributed by Akira Li in bpo-19940@footnote{https://bugs.python.org/issue19940}.)

New @code{SSLObject.shared_ciphers()} and
@ref{9d8,,SSLSocket.shared_ciphers()} methods return
the list of ciphers sent by the client during the handshake.
(Contributed by Benjamin Peterson in bpo-23186@footnote{https://bugs.python.org/issue23186}.)

The @ref{9d9,,SSLSocket.do_handshake()},
@ref{9da,,SSLSocket.read()},
@code{SSLSocket.shutdown()}, and
@ref{9db,,SSLSocket.write()} methods of the @ref{3d6,,SSLSocket}
class no longer reset the socket timeout every time bytes are received or sent.
The socket timeout is now the maximum total duration of the method.
(Contributed by Victor Stinner in bpo-23853@footnote{https://bugs.python.org/issue23853}.)

The @ref{333,,match_hostname()} function now supports matching of IP addresses.
(Contributed by Antoine Pitrou in bpo-23239@footnote{https://bugs.python.org/issue23239}.)

@node sqlite3<5>,subprocess<3>,ssl<7>,Improved Modules<7>
@anchor{whatsnew/3 5 sqlite3}@anchor{9dc}
@subsubsection sqlite3


The @ref{9dd,,Row} class now fully supports the sequence protocol,
in particular @ref{2f6,,reversed()} iteration and slice indexing.
(Contributed by Claudiu Popa in bpo-10203@footnote{https://bugs.python.org/issue10203}; by Lucas Sinclair,
Jessica McKellar, and  Serhiy Storchaka in bpo-13583@footnote{https://bugs.python.org/issue13583}.)

@node subprocess<3>,sys<8>,sqlite3<5>,Improved Modules<7>
@anchor{whatsnew/3 5 subprocess}@anchor{9de}@anchor{whatsnew/3 5 whatsnew-subprocess}@anchor{8cb}
@subsubsection subprocess


The new @ref{6b7,,run()} function has been added.
It runs the specified command and returns a
@ref{9df,,CompletedProcess} object, which describes a finished
process.  The new API is more consistent and is the recommended approach
to invoking subprocesses in Python code that does not need to maintain
compatibility with earlier Python versions.
(Contributed by Thomas Kluyver in bpo-23342@footnote{https://bugs.python.org/issue23342}.)

Examples:

@example
>>> subprocess.run(["ls", "-l"])  # doesn't capture output
CompletedProcess(args=['ls', '-l'], returncode=0)

>>> subprocess.run("exit 1", shell=True, check=True)
Traceback (most recent call last):
  ...
subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1

>>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE)
CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0,
stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n')
@end example

@node sys<8>,sysconfig,subprocess<3>,Improved Modules<7>
@anchor{whatsnew/3 5 sys}@anchor{9e0}
@subsubsection sys


A new @code{set_coroutine_wrapper()} function allows setting a global
hook that will be called whenever a @ref{3cf,,coroutine object}
is created by an @ref{1c0,,async def} function.  A corresponding
@code{get_coroutine_wrapper()} can be used to obtain a currently set
wrapper.  Both functions are @ref{621,,provisional},
and are intended for debugging purposes only.  (Contributed by Yury Selivanov
in bpo-24017@footnote{https://bugs.python.org/issue24017}.)

A new @ref{9e1,,is_finalizing()} function can be used to check if the Python
interpreter is @ref{9e2,,shutting down}.
(Contributed by Antoine Pitrou in bpo-22696@footnote{https://bugs.python.org/issue22696}.)

@node sysconfig,tarfile<2>,sys<8>,Improved Modules<7>
@anchor{whatsnew/3 5 sysconfig}@anchor{9e3}
@subsubsection sysconfig


The name of the user scripts directory on Windows now includes the first
two components of the Python version. (Contributed by Paul Moore
in bpo-23437@footnote{https://bugs.python.org/issue23437}.)

@node tarfile<2>,threading<5>,sysconfig,Improved Modules<7>
@anchor{whatsnew/3 5 tarfile}@anchor{9e4}
@subsubsection tarfile


The @emph{mode} argument of the @ref{9e5,,open()} function now accepts @code{"x"}
to request exclusive creation.  (Contributed by Berker Peksag in bpo-21717@footnote{https://bugs.python.org/issue21717}.)

The @ref{9e6,,TarFile.extractall()} and
@ref{9e7,,TarFile.extract()} methods now take a keyword
argument @emph{numeric_owner}.  If set to @code{True}, the extracted files and
directories will be owned by the numeric @code{uid} and @code{gid} from the tarfile.
If set to @code{False} (the default, and the behavior in versions prior to 3.5),
they will be owned by the named user and group in the tarfile.
(Contributed by Michael Vogt and Eric Smith in bpo-23193@footnote{https://bugs.python.org/issue23193}.)

The @ref{9e8,,TarFile.list()} now accepts an optional
@emph{members} keyword argument that can be set to a subset of the list returned
by @ref{9e9,,TarFile.getmembers()}.
(Contributed by Serhiy Storchaka in bpo-21549@footnote{https://bugs.python.org/issue21549}.)

@node threading<5>,time<6>,tarfile<2>,Improved Modules<7>
@anchor{whatsnew/3 5 threading}@anchor{9ea}
@subsubsection threading


Both the @ref{192,,Lock.acquire()} and
@ref{9eb,,RLock.acquire()} methods
now use a monotonic clock for timeout management.
(Contributed by Victor Stinner in bpo-22043@footnote{https://bugs.python.org/issue22043}.)

@node time<6>,timeit<2>,threading<5>,Improved Modules<7>
@anchor{whatsnew/3 5 time}@anchor{9ec}
@subsubsection time


The @ref{9ed,,monotonic()} function is now always available.
(Contributed by Victor Stinner in bpo-22043@footnote{https://bugs.python.org/issue22043}.)

@node timeit<2>,tkinter<5>,time<6>,Improved Modules<7>
@anchor{whatsnew/3 5 timeit}@anchor{9ee}
@subsubsection timeit


A new command line option @code{-u} or @code{--unit=@emph{U}} can be used to specify the time
unit for the timer output.  Supported options are @code{usec}, @code{msec},
or @code{sec}.  (Contributed by Julian Gindi in bpo-18983@footnote{https://bugs.python.org/issue18983}.)

The @ref{9ef,,timeit()} function has a new @emph{globals} parameter for
specifying the namespace in which the code will be running.
(Contributed by Ben Roberts in bpo-2527@footnote{https://bugs.python.org/issue2527}.)

@node tkinter<5>,traceback<3>,timeit<2>,Improved Modules<7>
@anchor{whatsnew/3 5 tkinter}@anchor{9f0}
@subsubsection tkinter


The @code{tkinter._fix} module used for setting up the Tcl/Tk environment
on Windows has been replaced by a private function in the @code{_tkinter}
module which makes no permanent changes to environment variables.
(Contributed by Zachary Ware in bpo-20035@footnote{https://bugs.python.org/issue20035}.)

@node traceback<3>,types<3>,tkinter<5>,Improved Modules<7>
@anchor{whatsnew/3 5 traceback}@anchor{9f1}@anchor{whatsnew/3 5 whatsnew-traceback}@anchor{8cc}
@subsubsection traceback


New @ref{9f2,,walk_stack()} and @ref{9f3,,walk_tb()}
functions to conveniently traverse frame and traceback objects.
(Contributed by Robert Collins in bpo-17911@footnote{https://bugs.python.org/issue17911}.)

New lightweight classes: @ref{9f4,,TracebackException},
@ref{9f5,,StackSummary}, and @ref{9f6,,FrameSummary}.
(Contributed by Robert Collins in bpo-17911@footnote{https://bugs.python.org/issue17911}.)

Both the @ref{9f7,,print_tb()} and @ref{9f8,,print_stack()} functions
now support negative values for the @emph{limit} argument.
(Contributed by Dmitry Kazakov in bpo-22619@footnote{https://bugs.python.org/issue22619}.)

@node types<3>,unicodedata<6>,traceback<3>,Improved Modules<7>
@anchor{whatsnew/3 5 types}@anchor{9f9}
@subsubsection types


A new @ref{9fa,,coroutine()} function to transform
@ref{715,,generator} and
@ref{93a,,generator-like} objects into
@ref{7bd,,awaitables}.
(Contributed by Yury Selivanov in bpo-24017@footnote{https://bugs.python.org/issue24017}.)

A new type called @ref{9fb,,CoroutineType}, which is used for
@ref{3cf,,coroutine} objects created by @ref{1c0,,async def} functions.
(Contributed by Yury Selivanov in bpo-24400@footnote{https://bugs.python.org/issue24400}.)

@node unicodedata<6>,unittest<4>,types<3>,Improved Modules<7>
@anchor{whatsnew/3 5 unicodedata}@anchor{9fc}
@subsubsection unicodedata


The @ref{121,,unicodedata} module now uses data from Unicode 8.0.0@footnote{http://unicode.org/versions/Unicode8.0.0/}.

@node unittest<4>,unittest mock<3>,unicodedata<6>,Improved Modules<7>
@anchor{whatsnew/3 5 unittest}@anchor{9fd}
@subsubsection unittest


The @ref{1ae,,TestLoader.loadTestsFromModule()}
method now accepts a keyword-only argument @emph{pattern} which is passed to
@code{load_tests} as the third argument.  Found packages are now checked for
@code{load_tests} regardless of whether their path matches @emph{pattern}, because it
is impossible for a package name to match the default pattern.
(Contributed by Robert Collins and Barry A. Warsaw in bpo-16662@footnote{https://bugs.python.org/issue16662}.)

Unittest discovery errors now are exposed in the
@ref{9fe,,TestLoader.errors} attribute of the
@ref{1ad,,TestLoader} instance.
(Contributed by Robert Collins in bpo-19746@footnote{https://bugs.python.org/issue19746}.)

A new command line option @code{--locals} to show local variables in
tracebacks.  (Contributed by Robert Collins in bpo-22936@footnote{https://bugs.python.org/issue22936}.)

@node unittest mock<3>,urllib,unittest<4>,Improved Modules<7>
@anchor{whatsnew/3 5 unittest-mock}@anchor{9ff}
@subsubsection unittest.mock


The @ref{561,,Mock} class has the following improvements:


@itemize *

@item 
The class constructor has a new @emph{unsafe} parameter, which causes mock
objects to raise @ref{163,,AttributeError} on attribute names starting
with @code{"assert"}.
(Contributed by Kushal Das in bpo-21238@footnote{https://bugs.python.org/issue21238}.)

@item 
A new @ref{a00,,Mock.assert_not_called()}
method to check if the mock object was called.
(Contributed by Kushal Das in bpo-21262@footnote{https://bugs.python.org/issue21262}.)
@end itemize

The @ref{a01,,MagicMock} class now supports @code{__truediv__()},
@code{__divmod__()} and @code{__matmul__()} operators.
(Contributed by Johannes Baiter in bpo-20968@footnote{https://bugs.python.org/issue20968}, and Håkan Lövdahl
in bpo-23581@footnote{https://bugs.python.org/issue23581} and bpo-23568@footnote{https://bugs.python.org/issue23568}.)

It is no longer necessary to explicitly pass @code{create=True} to the
@ref{a02,,patch()} function when patching builtin names.
(Contributed by Kushal Das in bpo-17660@footnote{https://bugs.python.org/issue17660}.)

@node urllib,wsgiref,unittest mock<3>,Improved Modules<7>
@anchor{whatsnew/3 5 urllib}@anchor{a03}
@subsubsection urllib


A new
@ref{a04,,request.HTTPPasswordMgrWithPriorAuth}
class allows HTTP Basic Authentication credentials to be managed so as to
eliminate unnecessary @code{401} response handling, or to unconditionally send
credentials on the first request in order to communicate with servers that
return a @code{404} response instead of a @code{401} if the @code{Authorization} header
is not sent. (Contributed by Matej Cepl in bpo-19494@footnote{https://bugs.python.org/issue19494} and Akshit Khurana in
bpo-7159@footnote{https://bugs.python.org/issue7159}.)

A new @emph{quote_via} argument for the
@ref{a05,,parse.urlencode()}
function provides a way to control the encoding of query parts if needed.
(Contributed by Samwyse and Arnon Yaari in bpo-13866@footnote{https://bugs.python.org/issue13866}.)

The @ref{a06,,request.urlopen()} function accepts an
@ref{2bf,,ssl.SSLContext} object as a @emph{context} argument, which will be used for
the HTTPS connection.  (Contributed by Alex Gaynor in bpo-22366@footnote{https://bugs.python.org/issue22366}.)

The @ref{a07,,parse.urljoin()} was updated to use the
@geindex RFC; RFC 3986
RFC 3986@footnote{https://tools.ietf.org/html/rfc3986.html} semantics for the resolution of relative URLs, rather than
@geindex RFC; RFC 1808
RFC 1808@footnote{https://tools.ietf.org/html/rfc1808.html} and 
@geindex RFC; RFC 2396
RFC 2396@footnote{https://tools.ietf.org/html/rfc2396.html}.
(Contributed by Demian Brecht and Senthil Kumaran in bpo-22118@footnote{https://bugs.python.org/issue22118}.)

@node wsgiref,xmlrpc<2>,urllib,Improved Modules<7>
@anchor{whatsnew/3 5 wsgiref}@anchor{a08}
@subsubsection wsgiref


The @emph{headers} argument of the @ref{a09,,headers.Headers}
class constructor is now optional.
(Contributed by Pablo Torres Navarrete and SilentGhost in bpo-5800@footnote{https://bugs.python.org/issue5800}.)

@node xmlrpc<2>,xml sax,wsgiref,Improved Modules<7>
@anchor{whatsnew/3 5 xmlrpc}@anchor{a0a}
@subsubsection xmlrpc


The @ref{56c,,client.ServerProxy} class now supports
the @ref{166,,context manager} protocol.
(Contributed by Claudiu Popa in bpo-20627@footnote{https://bugs.python.org/issue20627}.)

The @ref{56c,,client.ServerProxy} constructor now accepts
an optional @ref{2bf,,ssl.SSLContext} instance.
(Contributed by Alex Gaynor in bpo-22960@footnote{https://bugs.python.org/issue22960}.)

@node xml sax,zipfile<3>,xmlrpc<2>,Improved Modules<7>
@anchor{whatsnew/3 5 xml-sax}@anchor{a0b}
@subsubsection xml.sax


SAX parsers now support a character stream of the
@ref{a0c,,xmlreader.InputSource} object.
(Contributed by Serhiy Storchaka in bpo-2175@footnote{https://bugs.python.org/issue2175}.)

@ref{a0d,,parseString()} now accepts a @ref{2ef,,str} instance.
(Contributed by Serhiy Storchaka in bpo-10590@footnote{https://bugs.python.org/issue10590}.)

@node zipfile<3>,,xml sax,Improved Modules<7>
@anchor{whatsnew/3 5 zipfile}@anchor{a0e}
@subsubsection zipfile


ZIP output can now be written to unseekable streams.
(Contributed by Serhiy Storchaka in bpo-23252@footnote{https://bugs.python.org/issue23252}.)

The @emph{mode} argument of @ref{857,,ZipFile.open()} method now
accepts @code{"x"} to request exclusive creation.
(Contributed by Serhiy Storchaka in bpo-21717@footnote{https://bugs.python.org/issue21717}.)

@node Other module-level changes,Optimizations<7>,Improved Modules<7>,What’s New In Python 3 5
@anchor{whatsnew/3 5 other-module-level-changes}@anchor{a0f}
@subsection Other module-level changes


Many functions in the @ref{af,,mmap}, @ref{c2,,ossaudiodev}, @ref{ea,,socket},
@ref{ee,,ssl}, and @ref{1a,,codecs} modules now accept writable
@ref{87e,,bytes-like objects}.
(Contributed by Serhiy Storchaka in bpo-23001@footnote{https://bugs.python.org/issue23001}.)

@node Optimizations<7>,Build and C API Changes<3>,Other module-level changes,What’s New In Python 3 5
@anchor{whatsnew/3 5 optimizations}@anchor{a10}
@subsection Optimizations


The @ref{8de,,os.walk()} function has been sped up by 3 to 5 times on POSIX systems,
and by 7 to 20 times on Windows.  This was done using the new @ref{576,,os.scandir()}
function, which exposes file information from the underlying @code{readdir} or
@code{FindFirstFile}/@code{FindNextFile} system calls.  (Contributed by
Ben Hoyt with help from Victor Stinner in bpo-23605@footnote{https://bugs.python.org/issue23605}.)

Construction of @code{bytes(int)} (filled by zero bytes) is faster and uses less
memory for large objects. @code{calloc()} is used instead of @code{malloc()} to
allocate memory for these objects.
(Contributed by Victor Stinner in bpo-21233@footnote{https://bugs.python.org/issue21233}.)

Some operations on @ref{9e,,ipaddress} @ref{670,,IPv4Network} and
@ref{66f,,IPv6Network} have been massively sped up, such as
@ref{a11,,subnets()}, @ref{a12,,supernet()},
@ref{a13,,summarize_address_range()}, @ref{a14,,collapse_addresses()}.
The speed up can range from 3 to 15 times.
(Contributed by Antoine Pitrou, Michel Albert, and Markus in
bpo-21486@footnote{https://bugs.python.org/issue21486}, bpo-21487@footnote{https://bugs.python.org/issue21487}, bpo-20826@footnote{https://bugs.python.org/issue20826}, bpo-23266@footnote{https://bugs.python.org/issue23266}.)

Pickling of @ref{9e,,ipaddress} objects was optimized to produce significantly
smaller output.  (Contributed by Serhiy Storchaka in bpo-23133@footnote{https://bugs.python.org/issue23133}.)

Many operations on @ref{a15,,io.BytesIO} are now 50% to 100% faster.
(Contributed by Serhiy Storchaka in bpo-15381@footnote{https://bugs.python.org/issue15381} and David Wilson in
bpo-22003@footnote{https://bugs.python.org/issue22003}.)

The @ref{a16,,marshal.dumps()} function is now faster: 65–85% with versions 3
and 4, 20–25% with versions 0 to 2 on typical data, and up to 5 times in
best cases.
(Contributed by Serhiy Storchaka in bpo-20416@footnote{https://bugs.python.org/issue20416} and bpo-23344@footnote{https://bugs.python.org/issue23344}.)

The UTF-32 encoder is now 3 to 7 times faster.
(Contributed by Serhiy Storchaka in bpo-15027@footnote{https://bugs.python.org/issue15027}.)

Regular expressions are now parsed up to 10% faster.
(Contributed by Serhiy Storchaka in bpo-19380@footnote{https://bugs.python.org/issue19380}.)

The @ref{89b,,json.dumps()} function was optimized to run with
@code{ensure_ascii=False} as fast as with @code{ensure_ascii=True}.
(Contributed by Naoki Inada in bpo-23206@footnote{https://bugs.python.org/issue23206}.)

The @ref{a17,,PyObject_IsInstance()} and @ref{a18,,PyObject_IsSubclass()}
functions have been sped up in the common case that the second argument
has @ref{89e,,type} as its metaclass.
(Contributed Georg Brandl by in bpo-22540@footnote{https://bugs.python.org/issue22540}.)

Method caching was slightly improved, yielding up to 5% performance
improvement in some benchmarks.
(Contributed by Antoine Pitrou in bpo-22847@footnote{https://bugs.python.org/issue22847}.)

Objects from the @ref{d7,,random} module now use 50% less memory on 64-bit
builds.  (Contributed by Serhiy Storchaka in bpo-23488@footnote{https://bugs.python.org/issue23488}.)

The @ref{501,,property()} getter calls are up to 25% faster.
(Contributed by Joe Jevnik in bpo-23910@footnote{https://bugs.python.org/issue23910}.)

Instantiation of @ref{172,,fractions.Fraction} is now up to 30% faster.
(Contributed by Stefan Behnel in bpo-22464@footnote{https://bugs.python.org/issue22464}.)

String methods @ref{a19,,find()}, @ref{a1a,,rfind()}, @ref{a1b,,split()},
@ref{a1c,,partition()} and the @ref{2fc,,in} string operator are now significantly
faster for searching 1-character substrings.
(Contributed by Serhiy Storchaka in bpo-23573@footnote{https://bugs.python.org/issue23573}.)

@node Build and C API Changes<3>,Deprecated<8>,Optimizations<7>,What’s New In Python 3 5
@anchor{whatsnew/3 5 build-and-c-api-changes}@anchor{a1d}
@subsection Build and C API Changes


New @code{calloc} functions were added:


@itemize *

@item 
@ref{a1e,,PyMem_RawCalloc()},

@item 
@ref{a1f,,PyMem_Calloc()},

@item 
@ref{a20,,PyObject_Calloc()}.
@end itemize

(Contributed by Victor Stinner in bpo-21233@footnote{https://bugs.python.org/issue21233}.)

New encoding/decoding helper functions:


@itemize *

@item 
@ref{6fe,,Py_DecodeLocale()} (replaced @code{_Py_char2wchar()}),

@item 
@ref{6ff,,Py_EncodeLocale()} (replaced @code{_Py_wchar2char()}).
@end itemize

(Contributed by Victor Stinner in bpo-18395@footnote{https://bugs.python.org/issue18395}.)

A new @ref{a21,,PyCodec_NameReplaceErrors()} function to replace the unicode
encode error with @code{\N@{...@}} escapes.
(Contributed by Serhiy Storchaka in bpo-19676@footnote{https://bugs.python.org/issue19676}.)

A new @ref{a22,,PyErr_FormatV()} function similar to @ref{a23,,PyErr_Format()},
but accepts a @code{va_list} argument.
(Contributed by Antoine Pitrou in bpo-18711@footnote{https://bugs.python.org/issue18711}.)

A new @code{PyExc_RecursionError} exception.
(Contributed by Georg Brandl in bpo-19235@footnote{https://bugs.python.org/issue19235}.)

New @ref{a24,,PyModule_FromDefAndSpec()}, @ref{a25,,PyModule_FromDefAndSpec2()},
and @ref{a26,,PyModule_ExecDef()} functions introduced by 
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489} –
multi-phase extension module initialization.
(Contributed by Petr Viktorin in bpo-24268@footnote{https://bugs.python.org/issue24268}.)

New @ref{a27,,PyNumber_MatrixMultiply()} and
@ref{a28,,PyNumber_InPlaceMatrixMultiply()} functions to perform matrix
multiplication.
(Contributed by Benjamin Peterson in bpo-21176@footnote{https://bugs.python.org/issue21176}.  See also 
@geindex Python Enhancement Proposals; PEP 465
PEP 465@footnote{https://www.python.org/dev/peps/pep-0465}
for details.)

The @ref{5c7,,PyTypeObject.tp_finalize} slot is now part of the stable ABI.

Windows builds now require Microsoft Visual C++ 14.0, which
is available as part of Visual Studio 2015@footnote{https://www.visualstudio.com/}.

Extension modules now include a platform information tag in their filename on
some platforms (the tag is optional, and CPython will import extensions without
it, although if the tag is present and mismatched, the extension won’t be
loaded):


@itemize *

@item 
On Linux, extension module filenames end with
@code{.cpython-<major><minor>m-<architecture>-<os>.pyd}:


@itemize *

@item 
@code{<major>} is the major number of the Python version;
for Python 3.5 this is @code{3}.

@item 
@code{<minor>} is the minor number of the Python version;
for Python 3.5 this is @code{5}.

@item 
@code{<architecture>} is the hardware architecture the extension module
was built to run on. It’s most commonly either @code{i386} for 32-bit Intel
platforms or @code{x86_64} for 64-bit Intel (and AMD) platforms.

@item 
@code{<os>} is always @code{linux-gnu}, except for extensions built to
talk to the 32-bit ABI on 64-bit platforms, in which case it is
@code{linux-gnu32} (and @code{<architecture>} will be @code{x86_64}).
@end itemize

@item 
On Windows, extension module filenames end with
@code{<debug>.cp<major><minor>-<platform>.pyd}:


@itemize *

@item 
@code{<major>} is the major number of the Python version;
for Python 3.5 this is @code{3}.

@item 
@code{<minor>} is the minor number of the Python version;
for Python 3.5 this is @code{5}.

@item 
@code{<platform>} is the platform the extension module was built for,
either @code{win32} for Win32, @code{win_amd64} for Win64, @code{win_ia64} for
Windows Itanium 64, and @code{win_arm} for Windows on ARM.

@item 
If built in debug mode, @code{<debug>} will be @code{_d},
otherwise it will be blank.
@end itemize

@item 
On OS X platforms, extension module filenames now end with @code{-darwin.so}.

@item 
On all other platforms, extension module filenames are the same as they were
with Python 3.4.
@end itemize

@node Deprecated<8>,Removed<8>,Build and C API Changes<3>,What’s New In Python 3 5
@anchor{whatsnew/3 5 deprecated}@anchor{a29}
@subsection Deprecated


@menu
* New Keywords: New Keywords<2>. 
* Deprecated Python Behavior: Deprecated Python Behavior<2>. 
* Unsupported Operating Systems:: 
* Deprecated Python modules@comma{} functions and methods: Deprecated Python modules functions and methods<3>. 

@end menu

@node New Keywords<2>,Deprecated Python Behavior<2>,,Deprecated<8>
@anchor{whatsnew/3 5 new-keywords}@anchor{a2a}
@subsubsection New Keywords


@code{async} and @code{await} are not recommended to be used as variable, class,
function or module names.  Introduced by 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} in Python 3.5, they will
become proper keywords in Python 3.7.

@node Deprecated Python Behavior<2>,Unsupported Operating Systems,New Keywords<2>,Deprecated<8>
@anchor{whatsnew/3 5 deprecated-python-behavior}@anchor{a2b}
@subsubsection Deprecated Python Behavior


Raising the @ref{736,,StopIteration} exception inside a generator will now generate a silent
@ref{3a9,,PendingDeprecationWarning}, which will become a non-silent deprecation
warning in Python 3.6 and will trigger a @ref{5af,,RuntimeError} in Python 3.7.
See @ref{870,,PEP 479; Change StopIteration handling inside generators}
for details.

@node Unsupported Operating Systems,Deprecated Python modules functions and methods<3>,Deprecated Python Behavior<2>,Deprecated<8>
@anchor{whatsnew/3 5 unsupported-operating-systems}@anchor{a2c}
@subsubsection Unsupported Operating Systems


Windows XP is no longer supported by Microsoft, thus, per 
@geindex Python Enhancement Proposals; PEP 11
PEP 11@footnote{https://www.python.org/dev/peps/pep-0011}, CPython
3.5 is no longer officially supported on this OS.

@node Deprecated Python modules functions and methods<3>,,Unsupported Operating Systems,Deprecated<8>
@anchor{whatsnew/3 5 deprecated-python-modules-functions-and-methods}@anchor{a2d}
@subsubsection Deprecated Python modules, functions and methods


The @code{formatter} module has now graduated to full deprecation and is still
slated for removal in Python 3.6.

The @code{asyncio.async()} function is deprecated in favor of
@ref{318,,ensure_future()}.

The @ref{e7,,smtpd} module has in the past always decoded the DATA portion of
email messages using the @code{utf-8} codec.  This can now be controlled by the
new @emph{decode_data} keyword to @ref{898,,SMTPServer}.  The default value is
@code{True}, but this default is deprecated.  Specify the @emph{decode_data} keyword
with an appropriate value to avoid the deprecation warning.

Directly assigning values to the @ref{73c,,key},
@ref{73d,,value} and
@ref{73e,,coded_value} of @ref{73f,,http.cookies.Morsel}
objects is deprecated.  Use the @ref{740,,set()} method
instead.  In addition, the undocumented @emph{LegalChars} parameter of
@ref{740,,set()} is deprecated, and is now ignored.

Passing a format string as keyword argument @emph{format_string} to the
@ref{73b,,format()} method of the @ref{a2e,,string.Formatter}
class has been deprecated.
(Contributed by Serhiy Storchaka in bpo-23671@footnote{https://bugs.python.org/issue23671}.)

The @code{platform.dist()} and @code{platform.linux_distribution()} functions
are now deprecated.  Linux distributions use too many different ways of
describing themselves, so the functionality is left to a package.
(Contributed by Vajrasky Kok and Berker Peksag in bpo-1322@footnote{https://bugs.python.org/issue1322}.)

The previously undocumented @code{from_function} and @code{from_builtin} methods of
@ref{975,,inspect.Signature} are deprecated.  Use the new
@ref{1c3,,Signature.from_callable()}
method instead. (Contributed by Yury Selivanov in bpo-24248@footnote{https://bugs.python.org/issue24248}.)

The @code{inspect.getargspec()} function is deprecated and scheduled to be
removed in Python 3.6.  (See bpo-20438@footnote{https://bugs.python.org/issue20438} for details.)

The @ref{9c,,inspect} @ref{1c2,,getfullargspec()},
@ref{a2f,,getcallargs()}, and @code{formatargspec()} functions are
deprecated in favor of the @ref{1c1,,inspect.signature()} API. (Contributed by Yury
Selivanov in bpo-20438@footnote{https://bugs.python.org/issue20438}.)

@ref{a30,,getargvalues()} and @ref{a31,,formatargvalues()} functions
were inadvertently marked as deprecated with the release of Python 3.5.0.

Use of @ref{698,,re.LOCALE} flag with str patterns or @ref{697,,re.ASCII} is now
deprecated.  (Contributed by Serhiy Storchaka in bpo-22407@footnote{https://bugs.python.org/issue22407}.)

Use of unrecognized special sequences consisting of @code{'\'} and an ASCII letter
in regular expression patterns and replacement patterns now raises a
deprecation warning and will be forbidden in Python 3.6.
(Contributed by Serhiy Storchaka in bpo-23622@footnote{https://bugs.python.org/issue23622}.)

The undocumented and unofficial @emph{use_load_tests} default argument of the
@ref{1ae,,unittest.TestLoader.loadTestsFromModule()} method now is
deprecated and ignored.
(Contributed by Robert Collins and Barry A. Warsaw in bpo-16662@footnote{https://bugs.python.org/issue16662}.)

@node Removed<8>,Porting to Python 3 5,Deprecated<8>,What’s New In Python 3 5
@anchor{whatsnew/3 5 removed}@anchor{a32}
@subsection Removed


@menu
* API and Feature Removals: API and Feature Removals<4>. 

@end menu

@node API and Feature Removals<4>,,,Removed<8>
@anchor{whatsnew/3 5 api-and-feature-removals}@anchor{a33}
@subsubsection API and Feature Removals


The following obsolete and previously deprecated APIs and features have been
removed:


@itemize *

@item 
The @code{__version__} attribute has been dropped from the email package.  The
email code hasn’t been shipped separately from the stdlib for a long time,
and the @code{__version__} string was not updated in the last few releases.

@item 
The internal @code{Netrc} class in the @ref{7e,,ftplib} module was deprecated in
3.4, and has now been removed.
(Contributed by Matt Chaput in bpo-6623@footnote{https://bugs.python.org/issue6623}.)

@item 
The concept of @code{.pyo} files has been removed.

@item 
The JoinableQueue class in the provisional @ref{9,,asyncio} module was
deprecated in 3.4.4 and is now removed.
(Contributed by A. Jesse Jiryu Davis in bpo-23464@footnote{https://bugs.python.org/issue23464}.)
@end itemize

@node Porting to Python 3 5,Notable changes in Python 3 5 4,Removed<8>,What’s New In Python 3 5
@anchor{whatsnew/3 5 porting-to-python-3-5}@anchor{a34}
@subsection Porting to Python 3.5


This section lists previously described changes and other bugfixes
that may require changes to your code.

@menu
* Changes in Python behavior: Changes in Python behavior<2>. 
* Changes in the Python API: Changes in the Python API<7>. 
* Changes in the C API: Changes in the C API<6>. 

@end menu

@node Changes in Python behavior<2>,Changes in the Python API<7>,,Porting to Python 3 5
@anchor{whatsnew/3 5 changes-in-python-behavior}@anchor{a35}
@subsubsection Changes in Python behavior



@itemize *

@item 
Due to an oversight, earlier Python versions erroneously accepted the
following syntax:

@example
f(1 for x in [1], *args)
f(1 for x in [1], **kwargs)
@end example

Python 3.5 now correctly raises a @ref{21a,,SyntaxError}, as generator
expressions must be put in parentheses if not a sole argument to a function.
@end itemize

@node Changes in the Python API<7>,Changes in the C API<6>,Changes in Python behavior<2>,Porting to Python 3 5
@anchor{whatsnew/3 5 changes-in-the-python-api}@anchor{a36}
@subsubsection Changes in the Python API



@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475}: System calls are now retried when interrupted by a signal instead
of raising @ref{8e3,,InterruptedError} if the Python signal handler does not
raise an exception.

@item 
Before Python 3.5, a @ref{364,,datetime.time} object was considered to be false
if it represented midnight in UTC.  This behavior was considered obscure and
error-prone and has been removed in Python 3.5.  See bpo-13936@footnote{https://bugs.python.org/issue13936} for full
details.

@item 
The @code{ssl.SSLSocket.send()} method now raises either
@ref{9d5,,ssl.SSLWantReadError} or @ref{9d6,,ssl.SSLWantWriteError}
on a non-blocking socket if the operation would block.  Previously,
it would return @code{0}.  (Contributed by Nikolaus Rath in bpo-20951@footnote{https://bugs.python.org/issue20951}.)

@item 
The @code{__name__} attribute of generators is now set from the function name,
instead of being set from the code name. Use @code{gen.gi_code.co_name} to
retrieve the code name. Generators also have a new @code{__qualname__}
attribute, the qualified name, which is now used for the representation
of a generator (@code{repr(gen)}).
(Contributed by Victor Stinner in bpo-21205@footnote{https://bugs.python.org/issue21205}.)

@item 
The deprecated “strict” mode and argument of @ref{33c,,HTMLParser},
@code{HTMLParser.error()}, and the @code{HTMLParserError} exception have been
removed.  (Contributed by Ezio Melotti in bpo-15114@footnote{https://bugs.python.org/issue15114}.)
The @emph{convert_charrefs} argument of @ref{33c,,HTMLParser} is
now @code{True} by default.  (Contributed by Berker Peksag in bpo-21047@footnote{https://bugs.python.org/issue21047}.)

@item 
Although it is not formally part of the API, it is worth noting for porting
purposes (ie: fixing tests) that error messages that were previously of the
form “‘sometype’ does not support the buffer protocol” are now of the form “a
@ref{87e,,bytes-like object} is required, not ‘sometype’”.
(Contributed by Ezio Melotti in bpo-16518@footnote{https://bugs.python.org/issue16518}.)

@item 
If the current directory is set to a directory that no longer exists then
@ref{a37,,FileNotFoundError} will no longer be raised and instead
@ref{a38,,find_spec()} will return @code{None}
@strong{without} caching @code{None} in @ref{74c,,sys.path_importer_cache}, which is
different than the typical case (bpo-22834@footnote{https://bugs.python.org/issue22834}).

@item 
HTTP status code and messages from @ref{8f,,http.client} and @ref{92,,http.server}
were refactored into a common @ref{3ef,,HTTPStatus} enum.  The values in
@ref{8f,,http.client} and @ref{92,,http.server} remain available for backwards
compatibility.  (Contributed by Demian Brecht in bpo-21793@footnote{https://bugs.python.org/issue21793}.)

@item 
When an import loader defines @code{importlib.machinery.Loader.exec_module()}
it is now expected to also define
@code{create_module()} (raises a
@ref{1b2,,DeprecationWarning} now, will be an error in Python 3.6). If the loader
inherits from @ref{a39,,importlib.abc.Loader} then there is nothing to do, else
simply define @code{create_module()} to return
@code{None}.  (Contributed by Brett Cannon in bpo-23014@footnote{https://bugs.python.org/issue23014}.)

@item 
The @ref{699,,re.split()} function always ignored empty pattern matches, so the
@code{"x*"} pattern worked the same as @code{"x+"}, and the @code{"\b"} pattern never
worked.  Now @ref{699,,re.split()} raises a warning if the pattern could match
an empty string.  For compatibility, use patterns that never match an empty
string (e.g. @code{"x+"} instead of @code{"x*"}).  Patterns that could only match
an empty string (such as @code{"\b"}) now raise an error.
(Contributed by Serhiy Storchaka in bpo-22818@footnote{https://bugs.python.org/issue22818}.)

@item 
The @ref{73f,,http.cookies.Morsel} dict-like interface has been made self
consistent:  morsel comparison now takes the @ref{73c,,key}
and @ref{73d,,value} into account,
@ref{a3a,,copy()} now results in a
@ref{73f,,Morsel} instance rather than a @ref{3b0,,dict}, and
@ref{a3b,,update()} will now raise an exception if any of the
keys in the update dictionary are invalid.  In addition, the undocumented
@emph{LegalChars} parameter of @ref{740,,set()} is deprecated and
is now ignored.  (Contributed by Demian Brecht in bpo-2211@footnote{https://bugs.python.org/issue2211}.)

@item 
@geindex Python Enhancement Proposals; PEP 488
PEP 488@footnote{https://www.python.org/dev/peps/pep-0488} has removed @code{.pyo} files from Python and introduced the optional
@code{opt-} tag in @code{.pyc} file names. The
@ref{7f9,,importlib.util.cache_from_source()} has gained an @emph{optimization}
parameter to help control the @code{opt-} tag. Because of this, the
@emph{debug_override} parameter of the function is now deprecated. @cite{.pyo} files
are also no longer supported as a file argument to the Python interpreter and
thus serve no purpose when distributed on their own (i.e. sourceless code
distribution). Due to the fact that the magic number for bytecode has changed
in Python 3.5, all old @cite{.pyo} files from previous versions of Python are
invalid regardless of this PEP.

@item 
The @ref{ea,,socket} module now exports the @ref{a3c,,CAN_RAW_FD_FRAMES}
constant on linux 3.6 and greater.

@item 
The @ref{9d7,,ssl.cert_time_to_seconds()} function now interprets the input time
as UTC and not as local time, per 
@geindex RFC; RFC 5280
RFC 5280@footnote{https://tools.ietf.org/html/rfc5280.html}.  Additionally, the return
value is always an @ref{1b1,,int}. (Contributed by Akira Li in bpo-19940@footnote{https://bugs.python.org/issue19940}.)

@item 
The @code{pygettext.py} Tool now uses the standard +NNNN format for timezones in
the POT-Creation-Date header.

@item 
The @ref{e8,,smtplib} module now uses @ref{42d,,sys.stderr} instead of the previous
module-level @code{stderr} variable for debug output.  If your (test)
program depends on patching the module-level variable to capture the debug
output, you will need to update it to capture sys.stderr instead.

@item 
The @ref{a3d,,str.startswith()} and @ref{a3e,,str.endswith()} methods no longer return
@code{True} when finding the empty string and the indexes are completely out of
range.  (Contributed by Serhiy Storchaka in bpo-24284@footnote{https://bugs.python.org/issue24284}.)

@item 
The @ref{500,,inspect.getdoc()} function now returns documentation strings
inherited from base classes.  Documentation strings no longer need to be
duplicated if the inherited documentation is appropriate.  To suppress an
inherited string, an empty string must be specified (or the documentation
may be filled in).  This change affects the output of the @ref{d4,,pydoc}
module and the @ref{3bc,,help()} function.
(Contributed by Serhiy Storchaka in bpo-15582@footnote{https://bugs.python.org/issue15582}.)

@item 
Nested @ref{a3f,,functools.partial()} calls are now flattened.  If you were
relying on the previous behavior, you can now either add an attribute to a
@ref{a3f,,functools.partial()} object or you can create a subclass of
@ref{a3f,,functools.partial()}.
(Contributed by Alexander Belopolsky in bpo-7830@footnote{https://bugs.python.org/issue7830}.)
@end itemize

@node Changes in the C API<6>,,Changes in the Python API<7>,Porting to Python 3 5
@anchor{whatsnew/3 5 changes-in-the-c-api}@anchor{a40}
@subsubsection Changes in the C API



@itemize *

@item 
The undocumented @code{format} member of the
(non-public) @code{PyMemoryViewObject} structure has been removed.
All extensions relying on the relevant parts in @code{memoryobject.h}
must be rebuilt.

@item 
The @code{PyMemAllocator} structure was renamed to
@ref{a41,,PyMemAllocatorEx} and a new @code{calloc} field was added.

@item 
Removed non-documented macro @code{PyObject_REPR} which leaked references.
Use format character @code{%R} in @ref{398,,PyUnicode_FromFormat()}-like functions
to format the @ref{27d,,repr()} of the object.
(Contributed by Serhiy Storchaka in bpo-22453@footnote{https://bugs.python.org/issue22453}.)

@item 
Because the lack of the @code{__module__} attribute breaks pickling and
introspection, a deprecation warning is now raised for builtin types without
the @code{__module__} attribute.  This would be an AttributeError in
the future.
(Contributed by Serhiy Storchaka in bpo-20204@footnote{https://bugs.python.org/issue20204}.)

@item 
As part of the 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} implementation, the @code{tp_reserved} slot of
@ref{5c6,,PyTypeObject} was replaced with a
@code{tp_as_async} slot.  Refer to @ref{a42,,Coroutine Objects} for
new types, structures and functions.
@end itemize

@node Notable changes in Python 3 5 4,,Porting to Python 3 5,What’s New In Python 3 5
@anchor{whatsnew/3 5 notable-changes-in-python-3-5-4}@anchor{a43}
@subsection Notable changes in Python 3.5.4


@menu
* New make regen-all build target: New make regen-all build target<2>. 
* Removal of make touch build target: Removal of make touch build target<2>. 

@end menu

@node New make regen-all build target<2>,Removal of make touch build target<2>,,Notable changes in Python 3 5 4
@anchor{whatsnew/3 5 new-make-regen-all-build-target}@anchor{a44}
@subsubsection New @code{make regen-all} build target


To simplify cross-compilation, and to ensure that CPython can reliably be
compiled without requiring an existing version of Python to already be
available, the autotools-based build system no longer attempts to implicitly
recompile generated files based on file modification times.

Instead, a new @code{make regen-all} command has been added to force regeneration
of these files when desired (e.g. after an initial version of Python has
already been built based on the pregenerated versions).

More selective regeneration targets are also defined - see
Makefile.pre.in@footnote{https://github.com/python/cpython/tree/main/Makefile.pre.in} for details.

(Contributed by Victor Stinner in bpo-23404@footnote{https://bugs.python.org/issue23404}.)


New in version 3.5.4.


@node Removal of make touch build target<2>,,New make regen-all build target<2>,Notable changes in Python 3 5 4
@anchor{whatsnew/3 5 removal-of-make-touch-build-target}@anchor{a45}
@subsubsection Removal of @code{make touch} build target


The @code{make touch} build target previously used to request implicit regeneration
of generated files by updating their modification times has been removed.

It has been replaced by the new @code{make regen-all} target.

(Contributed by Victor Stinner in bpo-23404@footnote{https://bugs.python.org/issue23404}.)


Changed in version 3.5.4.


@node What’s New In Python 3 4,What’s New In Python 3 3,What’s New In Python 3 5,What’s New in Python
@anchor{whatsnew/3 4 doc}@anchor{a46}@anchor{whatsnew/3 4 what-s-new-in-python-3-4}@anchor{a47}
@section What’s New In Python 3.4


@*Author: 
R. David Murray <@email{rdmurray@@bitdance.com}> (Editor)


@c Rules for maintenance:
@c 
@c * Anyone can add text to this document, but the maintainer reserves the
@c right to rewrite any additions. In particular, for obscure or esoteric
@c features, the maintainer may reduce any addition to a simple reference to
@c the new documentation rather than explaining the feature inline.
@c 
@c * While the maintainer will periodically go through Misc/NEWS
@c and add changes, it's best not to rely on this. We know from experience
@c that any changes that aren't in the What's New documentation around the
@c time of the original release will remain largely unknown to the community
@c for years, even if they're added later. We also know from experience that
@c other priorities can arise, and the maintainer will run out of time to do
@c updates -- in such cases, end users will be much better served by partial
@c notifications that at least give a hint about new features to
@c investigate.
@c 
@c * This is not a complete list of every single change; completeness
@c is the purpose of Misc/NEWS. The What's New should focus on changes that
@c are visible to Python *users* and that *require* a feature release (i.e.
@c most bug fixes should only be recorded in Misc/NEWS)
@c 
@c * PEPs should not be marked Final until they have an entry in What's New.
@c A placeholder entry that is just a section header and a link to the PEP
@c (e.g ":pep:`397` has been implemented") is acceptable. If a PEP has been
@c implemented and noted in What's New, don't forget to mark it as Final!
@c 
@c * If you want to draw your new text to the attention of the
@c maintainer, add 'XXX' to the beginning of the paragraph or
@c section.
@c 
@c * It's OK to add just a very brief note about a change.  For
@c example: "The :ref:`~socket.transmogrify()` function was added to the
@c :mod:`socket` module."  The maintainer will research the change and
@c write the necessary text (if appropriate). The advantage of doing this
@c is that even if no more descriptive text is ever added, readers will at
@c least have a notification that the new feature exists and a link to the
@c relevant documentation.
@c 
@c * You can comment out your additions if you like, but it's not
@c necessary (especially when a final release is some months away).
@c 
@c * Credit the author of a patch or bugfix.   Just the name is
@c sufficient; the e-mail address isn't necessary.
@c 
@c * It's helpful to add the bug/patch number as a comment:
@c 
@c The :ref:`~socket.transmogrify()` function was added to the
@c :mod:`socket` module.  (Contributed by P.Y. Developer in :issue:`12345`.)
@c 
@c This saves the maintainer the effort of going through the Mercurial log
@c when researching a change.
@c 
@c * Cross referencing tip: :ref:`mod.attr` will display as ``mod.attr``,
@c while :ref:`~mod.attr` will display as ``attr``.

This article explains the new features in Python 3.4, compared to 3.3.
Python 3.4 was released on March 16, 2014.  For full details, see the
changelog@footnote{https://docs.python.org/3.4/whatsnew/changelog.html}.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 429
PEP 429@footnote{https://www.python.org/dev/peps/pep-0429} – Python 3.4 Release Schedule


@menu
* Summary – Release Highlights: Summary – Release Highlights<2>. 
* New Features: New Features<11>. 
* New Modules: New Modules<8>. 
* Improved Modules: Improved Modules<8>. 
* CPython Implementation Changes:: 
* Deprecated: Deprecated<9>. 
* Removed: Removed<9>. 
* Porting to Python 3.4: Porting to Python 3 4. 
* Changed in 3.4.3: Changed in 3 4 3. 

@end menu

@node Summary – Release Highlights<2>,New Features<11>,,What’s New In Python 3 4
@anchor{whatsnew/3 4 summary-release-highlights}@anchor{a48}
@subsection Summary – Release Highlights


@c This section singles out the most important changes in Python 3.4.
@c Brevity is key.

New syntax features:


@itemize *

@item 
No new syntax features were added in Python 3.4.
@end itemize

Other new features:


@itemize *

@item 
@ref{a49,,pip should always be available} (
@geindex Python Enhancement Proposals; PEP 453
PEP 453@footnote{https://www.python.org/dev/peps/pep-0453}).

@item 
@ref{a4a,,Newly created file descriptors are non-inheritable}
(
@geindex Python Enhancement Proposals; PEP 446
PEP 446@footnote{https://www.python.org/dev/peps/pep-0446}).

@item 
command line option for @ref{a4b,,isolated mode}
(bpo-16499@footnote{https://bugs.python.org/issue16499}).

@item 
@ref{a4c,,improvements in the handling of codecs}
that are not text encodings (multiple issues).

@item 
@ref{a4d,,A ModuleSpec Type} for the Import System
(
@geindex Python Enhancement Proposals; PEP 451
PEP 451@footnote{https://www.python.org/dev/peps/pep-0451}).  (Affects importer authors.)

@item 
The @ref{ac,,marshal} format has been made @ref{a4e,,more compact and efficient} (bpo-16475@footnote{https://bugs.python.org/issue16475}).
@end itemize

New library modules:


@itemize *

@item 
@ref{9,,asyncio}: @ref{a4f,,New provisional API for asynchronous IO} (
@geindex Python Enhancement Proposals; PEP 3156
PEP 3156@footnote{https://www.python.org/dev/peps/pep-3156}).

@item 
@ref{75,,ensurepip}: @ref{a50,,Bootstrapping the pip installer}
(
@geindex Python Enhancement Proposals; PEP 453
PEP 453@footnote{https://www.python.org/dev/peps/pep-0453}).

@item 
@ref{76,,enum}: @ref{a51,,Support for enumeration types}
(
@geindex Python Enhancement Proposals; PEP 435
PEP 435@footnote{https://www.python.org/dev/peps/pep-0435}).

@item 
@ref{c3,,pathlib}: @ref{a52,,Object-oriented filesystem paths}
(
@geindex Python Enhancement Proposals; PEP 428
PEP 428@footnote{https://www.python.org/dev/peps/pep-0428}).

@item 
@ref{e1,,selectors}: @ref{a53,,High-level and efficient I/O multiplexing}, built upon the @ref{e0,,select} module primitives (part
of 
@geindex Python Enhancement Proposals; PEP 3156
PEP 3156@footnote{https://www.python.org/dev/peps/pep-3156}).

@item 
@ref{f0,,statistics}: A basic @ref{a54,,numerically stable statistics library} (
@geindex Python Enhancement Proposals; PEP 450
PEP 450@footnote{https://www.python.org/dev/peps/pep-0450}).

@item 
@ref{11b,,tracemalloc}: @ref{a55,,Trace Python memory allocations} (
@geindex Python Enhancement Proposals; PEP 454
PEP 454@footnote{https://www.python.org/dev/peps/pep-0454}).
@end itemize

Significantly improved library modules:


@itemize *

@item 
@ref{a56,,Single-dispatch generic functions} in
@ref{7f,,functools} (
@geindex Python Enhancement Proposals; PEP 443
PEP 443@footnote{https://www.python.org/dev/peps/pep-0443}).

@item 
New @ref{c5,,pickle} @ref{a57,,protocol 4} (
@geindex Python Enhancement Proposals; PEP 3154
PEP 3154@footnote{https://www.python.org/dev/peps/pep-3154}).

@item 
@ref{b3,,multiprocessing} now has @ref{a58,,an option to avoid using os.fork on Unix} (bpo-8713@footnote{https://bugs.python.org/issue8713}).

@item 
@ref{64,,email} has a new submodule, @ref{66,,contentmanager}, and
a new @ref{7e3,,Message} subclass
(@code{EmailMessage}) that @ref{a59,,simplify MIME handling} (bpo-18891@footnote{https://bugs.python.org/issue18891}).

@item 
The @ref{9c,,inspect} and @ref{d4,,pydoc} modules are now capable of
correct introspection of a much wider variety of callable objects,
which improves the output of the Python @ref{3bc,,help()} system.

@item 
The @ref{9e,,ipaddress} module API has been declared stable
@end itemize

Security improvements:


@itemize *

@item 
@ref{a5a,,Secure and interchangeable hash algorithm}
(
@geindex Python Enhancement Proposals; PEP 456
PEP 456@footnote{https://www.python.org/dev/peps/pep-0456}).

@item 
@ref{a4a,,Make newly created file descriptors non-inheritable}
(
@geindex Python Enhancement Proposals; PEP 446
PEP 446@footnote{https://www.python.org/dev/peps/pep-0446}) to avoid leaking file descriptors to child processes.

@item 
New command line option for @ref{a4b,,isolated mode},
(bpo-16499@footnote{https://bugs.python.org/issue16499}).

@item 
@ref{b3,,multiprocessing} now has @ref{a58,,an option to avoid using os.fork on Unix}.  @emph{spawn} and @emph{forkserver} are
more secure because they avoid sharing data with child processes.

@item 
@ref{b3,,multiprocessing} child processes on Windows no longer inherit
all of the parent’s inheritable handles, only the necessary ones.

@item 
A new @ref{28b,,hashlib.pbkdf2_hmac()} function provides
the PKCS#5 password-based key derivation function 2@footnote{https://en.wikipedia.org/wiki/PBKDF2}.

@item 
@ref{a5b,,TLSv1.1 and TLSv1.2 support} for @ref{ee,,ssl}.

@item 
@ref{a5c,,Retrieving certificates from the Windows system cert store support} for @ref{ee,,ssl}.

@item 
@ref{a5d,,Server-side SNI (Server Name Indication) support} for @ref{ee,,ssl}.

@item 
The @ref{2bf,,ssl.SSLContext} class has a @ref{a5e,,lot of improvements}.

@item 
All modules in the standard library that support SSL now support server
certificate verification, including hostname matching
(@ref{333,,ssl.match_hostname()}) and CRLs (Certificate Revocation lists, see
@ref{a5f,,ssl.SSLContext.load_verify_locations()}).
@end itemize

CPython implementation improvements:


@itemize *

@item 
@ref{a60,,Safe object finalization} (
@geindex Python Enhancement Proposals; PEP 442
PEP 442@footnote{https://www.python.org/dev/peps/pep-0442}).

@item 
Leveraging 
@geindex Python Enhancement Proposals; PEP 442
PEP 442@footnote{https://www.python.org/dev/peps/pep-0442}, in most cases @ref{a60,,module globals are no longer set to None during finalization} (bpo-18214@footnote{https://bugs.python.org/issue18214}).

@item 
@ref{a61,,Configurable memory allocators} (
@geindex Python Enhancement Proposals; PEP 445
PEP 445@footnote{https://www.python.org/dev/peps/pep-0445}).

@item 
@ref{a62,,Argument Clinic} (
@geindex Python Enhancement Proposals; PEP 436
PEP 436@footnote{https://www.python.org/dev/peps/pep-0436}).
@end itemize

Please read on for a comprehensive list of user-facing changes, including many
other smaller improvements, CPython optimizations, deprecations, and potential
porting issues.

@node New Features<11>,New Modules<8>,Summary – Release Highlights<2>,What’s New In Python 3 4
@anchor{whatsnew/3 4 new-features}@anchor{a63}
@subsection New Features


@menu
* PEP 453; Explicit Bootstrapping of PIP in Python Installations: PEP 453 Explicit Bootstrapping of PIP in Python Installations. 
* PEP 446; Newly Created File Descriptors Are Non-Inheritable: PEP 446 Newly Created File Descriptors Are Non-Inheritable. 
* Improvements to Codec Handling:: 
* PEP 451; A ModuleSpec Type for the Import System: PEP 451 A ModuleSpec Type for the Import System. 
* Other Language Changes: Other Language Changes<8>. 

@end menu

@node PEP 453 Explicit Bootstrapping of PIP in Python Installations,PEP 446 Newly Created File Descriptors Are Non-Inheritable,,New Features<11>
@anchor{whatsnew/3 4 pep-453-explicit-bootstrapping-of-pip-in-python-installations}@anchor{a64}@anchor{whatsnew/3 4 whatsnew-pep-453}@anchor{a49}
@subsubsection PEP 453: Explicit Bootstrapping of PIP in Python Installations


@menu
* Bootstrapping pip By Default:: 
* Documentation Changes:: 

@end menu

@node Bootstrapping pip By Default,Documentation Changes,,PEP 453 Explicit Bootstrapping of PIP in Python Installations
@anchor{whatsnew/3 4 bootstrapping-pip-by-default}@anchor{a65}
@subsubsection Bootstrapping pip By Default


The new @ref{75,,ensurepip} module (defined in 
@geindex Python Enhancement Proposals; PEP 453
PEP 453@footnote{https://www.python.org/dev/peps/pep-0453}) provides a standard
cross-platform mechanism to bootstrap the pip installer into Python
installations and virtual environments. The version of @code{pip} included
with Python 3.4.0 is @code{pip} 1.5.4, and future 3.4.x maintenance releases
will update the bundled version to the latest version of @code{pip} that is
available at the time of creating the release candidate.

By default, the commands @code{pipX} and @code{pipX.Y} will be installed on all
platforms (where X.Y stands for the version of the Python installation),
along with the @code{pip} Python package and its dependencies. On Windows and
in virtual environments on all platforms, the unversioned @code{pip} command
will also be installed. On other platforms, the system wide unversioned
@code{pip} command typically refers to the separately installed Python 2
version.

The @code{pyvenv} command line utility and the @ref{12c,,venv}
module make use of the @ref{75,,ensurepip} module to make @code{pip} readily
available in virtual environments. When using the command line utility,
@code{pip} is installed by default, while when using the @ref{12c,,venv} module
@ref{a66,,API} installation of @code{pip} must be requested explicitly.

For CPython @ref{a67,,source builds on POSIX systems},
the @code{make install} and @code{make altinstall} commands bootstrap @code{pip} by
default.  This behaviour can be controlled through configure options, and
overridden through Makefile options.

On Windows and Mac OS X, the CPython installers now default to installing
@code{pip} along with CPython itself (users may opt out of installing it
during the installation process). Window users will need to opt in to the
automatic @code{PATH} modifications to have @code{pip} available from the command
line by default, otherwise it can still be accessed through the Python
launcher for Windows as @code{py -m pip}.

As discussed in the PEP@footnote{https://www.python.org/dev/peps/pep-0453/#recommendations-for-downstream-distributors}, platform packagers may choose not to install
these commands by default, as long as, when invoked, they provide clear and
simple directions on how to install them on that platform (usually using
the system package manager).

@cartouche
@quotation Note 
To avoid conflicts between parallel Python 2 and Python 3 installations,
only the versioned @code{pip3} and @code{pip3.4} commands are bootstrapped by
default when @code{ensurepip} is invoked directly - the @code{--default-pip}
option is needed to also request the unversioned @code{pip} command.
@code{pyvenv} and the Windows installer ensure that the unqualified @code{pip}
command is made available in those environments, and @code{pip} can always be
invoked via the @code{-m} switch rather than directly to avoid ambiguity on
systems with multiple Python installations.
@end quotation
@end cartouche

@node Documentation Changes,,Bootstrapping pip By Default,PEP 453 Explicit Bootstrapping of PIP in Python Installations
@anchor{whatsnew/3 4 documentation-changes}@anchor{a68}
@subsubsection Documentation Changes


As part of this change, the @ref{a69,,Installing Python Modules} and
@ref{a6a,,Distributing Python Modules} sections of the documentation have been
completely redesigned as short getting started and FAQ documents. Most
packaging documentation has now been moved out to the Python Packaging
Authority maintained Python Packaging User Guide@footnote{https://packaging.python.org} and the documentation of the individual
projects.

However, as this migration is currently still incomplete, the legacy
versions of those guides remaining available as @ref{a6b,,Installing Python Modules (Legacy version)}
and @ref{a6c,,Distributing Python Modules (Legacy version)}.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 453

@item PEP 453@footnote{https://www.python.org/dev/peps/pep-0453} – Explicit bootstrapping of pip in Python installations

PEP written by Donald Stufft and Nick Coghlan, implemented by
Donald Stufft, Nick Coghlan, Martin von Löwis and Ned Deily.
@end table


@node PEP 446 Newly Created File Descriptors Are Non-Inheritable,Improvements to Codec Handling,PEP 453 Explicit Bootstrapping of PIP in Python Installations,New Features<11>
@anchor{whatsnew/3 4 pep-446-newly-created-file-descriptors-are-non-inheritable}@anchor{a6d}@anchor{whatsnew/3 4 whatsnew-pep-446}@anchor{a4a}
@subsubsection PEP 446: Newly Created File Descriptors Are Non-Inheritable


@geindex Python Enhancement Proposals; PEP 446
PEP 446@footnote{https://www.python.org/dev/peps/pep-0446} makes newly created file descriptors @ref{a6e,,non-inheritable}.  In general, this is the behavior an application will
want: when launching a new process, having currently open files also
open in the new process can lead to all sorts of hard to find bugs,
and potentially to security issues.

However, there are occasions when inheritance is desired.  To support
these cases, the following new functions and methods are available:


@itemize *

@item 
@ref{a6f,,os.get_inheritable()}, @ref{a70,,os.set_inheritable()}

@item 
@ref{a71,,os.get_handle_inheritable()}, @ref{a72,,os.set_handle_inheritable()}

@item 
@ref{a73,,socket.socket.get_inheritable()}, @ref{a74,,socket.socket.set_inheritable()}
@end itemize


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 446

@item PEP 446@footnote{https://www.python.org/dev/peps/pep-0446} – Make newly created file descriptors non-inheritable

PEP written and implemented by Victor Stinner.
@end table


@node Improvements to Codec Handling,PEP 451 A ModuleSpec Type for the Import System,PEP 446 Newly Created File Descriptors Are Non-Inheritable,New Features<11>
@anchor{whatsnew/3 4 codec-handling-improvements}@anchor{a4c}@anchor{whatsnew/3 4 improvements-to-codec-handling}@anchor{a75}
@subsubsection Improvements to Codec Handling


Since it was first introduced, the @ref{1a,,codecs} module has always been
intended to operate as a type-neutral dynamic encoding and decoding
system. However, its close coupling with the Python text model, especially
the type restricted convenience methods on the builtin @ref{2ef,,str},
@ref{2f0,,bytes} and @ref{2f1,,bytearray} types, has historically obscured that
fact.

As a key step in clarifying the situation, the @ref{a76,,codecs.encode()} and
@ref{a77,,codecs.decode()} convenience functions are now properly documented in
Python 2.7, 3.3 and 3.4. These functions have existed in the @ref{1a,,codecs}
module (and have been covered by the regression test suite) since Python 2.4,
but were previously only discoverable through runtime introspection.

Unlike the convenience methods on @ref{2ef,,str}, @ref{2f0,,bytes} and
@ref{2f1,,bytearray}, the @ref{1a,,codecs} convenience functions support arbitrary
codecs in both Python 2 and Python 3, rather than being limited to Unicode text
encodings (in Python 3) or @code{basestring} <-> @code{basestring} conversions (in
Python 2).

In Python 3.4, the interpreter is able to identify the known non-text
encodings provided in the standard library and direct users towards these
general purpose convenience functions when appropriate:

@example
>>> b"abcdef".decode("hex")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: 'hex' is not a text encoding; use codecs.decode() to handle arbitrary codecs

>>> "hello".encode("rot13")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: 'rot13' is not a text encoding; use codecs.encode() to handle arbitrary codecs

>>> open("foo.txt", encoding="hex")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: 'hex' is not a text encoding; use codecs.open() to handle arbitrary codecs
@end example

In a related change, whenever it is feasible without breaking backwards
compatibility, exceptions raised during encoding and decoding operations
are wrapped in a chained exception of the same type that mentions the
name of the codec responsible for producing the error:

@example
>>> import codecs

>>> codecs.decode(b"abcdefgh", "hex")
Traceback (most recent call last):
  File "/usr/lib/python3.4/encodings/hex_codec.py", line 20, in hex_decode
    return (binascii.a2b_hex(input), len(input))
binascii.Error: Non-hexadecimal digit found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
binascii.Error: decoding with 'hex' codec failed (Error: Non-hexadecimal digit found)

>>> codecs.encode("hello", "bz2")
Traceback (most recent call last):
  File "/usr/lib/python3.4/encodings/bz2_codec.py", line 17, in bz2_encode
    return (bz2.compress(input), len(input))
  File "/usr/lib/python3.4/bz2.py", line 498, in compress
    return comp.compress(data) + comp.flush()
TypeError: 'str' does not support the buffer interface

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: encoding with 'bz2' codec failed (TypeError: 'str' does not support the buffer interface)
@end example

Finally, as the examples above show, these improvements have permitted
the restoration of the convenience aliases for the non-Unicode codecs that
were themselves restored in Python 3.2. This means that encoding binary data
to and from its hexadecimal representation (for example) can now be written
as:

@example
>>> from codecs import encode, decode
>>> encode(b"hello", "hex")
b'68656c6c6f'
>>> decode(b"68656c6c6f", "hex")
b'hello'
@end example

The binary and text transforms provided in the standard library are detailed
in @ref{a78,,Binary Transforms} and @ref{a79,,Text Transforms}.

(Contributed by Nick Coghlan in bpo-7475@footnote{https://bugs.python.org/issue7475}, bpo-17827@footnote{https://bugs.python.org/issue17827},
bpo-17828@footnote{https://bugs.python.org/issue17828} and bpo-19619@footnote{https://bugs.python.org/issue19619}.)

@node PEP 451 A ModuleSpec Type for the Import System,Other Language Changes<8>,Improvements to Codec Handling,New Features<11>
@anchor{whatsnew/3 4 pep-451-a-modulespec-type-for-the-import-system}@anchor{a7a}@anchor{whatsnew/3 4 whatsnew-pep-451}@anchor{a4d}
@subsubsection PEP 451: A ModuleSpec Type for the Import System


@geindex Python Enhancement Proposals; PEP 451
PEP 451@footnote{https://www.python.org/dev/peps/pep-0451} provides an encapsulation of the information about a module that the
import machinery will use to load it (that is, a module specification).  This
helps simplify both the import implementation and several import-related APIs.
The change is also a stepping stone for several future import-related improvements@footnote{https://mail.python.org/pipermail/python-dev/2013-November/130111.html}.

The public-facing changes from the PEP are entirely backward-compatible.
Furthermore, they should be transparent to everyone but importer authors.  Key
finder and loader methods have been deprecated, but they will continue working.
New importers should use the new methods described in the PEP.  Existing
importers should be updated to implement the new methods.  See the
@ref{a7b,,Deprecated} section for a list of methods that should be replaced and
their replacements.

@node Other Language Changes<8>,,PEP 451 A ModuleSpec Type for the Import System,New Features<11>
@anchor{whatsnew/3 4 other-language-changes}@anchor{a7c}
@subsubsection Other Language Changes


Some smaller changes made to the core Python language are:


@itemize *

@item 
Unicode database updated to UCD version 6.3.

@item 
@ref{a7d,,min()} and @ref{a7e,,max()} now accept a @emph{default} keyword-only argument that
can be used to specify the value they return if the iterable they are
evaluating has no elements.  (Contributed by Julian Berman in
bpo-18111@footnote{https://bugs.python.org/issue18111}.)

@item 
Module objects are now @ref{12f,,weakref}’able.

@item 
Module @code{__file__} attributes (and related values) should now always
contain absolute paths by default, with the sole exception of
@code{__main__.__file__} when a script has been executed directly using
a relative path.  (Contributed by Brett Cannon in bpo-18416@footnote{https://bugs.python.org/issue18416}.)

@item 
All the UTF-* codecs (except UTF-7) now reject surrogates during both
encoding and decoding unless the @code{surrogatepass} error handler is used,
with the exception of the UTF-16 decoder (which accepts valid surrogate pairs)
and the UTF-16 encoder (which produces them while encoding non-BMP characters).
(Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in
bpo-12892@footnote{https://bugs.python.org/issue12892}.)

@item 
New German EBCDIC @ref{916,,codec} @code{cp273}.  (Contributed
by Michael Bierenfeld and Andrew Kuchling in bpo-1097797@footnote{https://bugs.python.org/issue1097797}.)

@item 
New Ukrainian @ref{916,,codec} @code{cp1125}.  (Contributed by
Serhiy Storchaka in bpo-19668@footnote{https://bugs.python.org/issue19668}.)

@item 
@ref{2f0,,bytes}.join() and @ref{2f1,,bytearray}.join() now accept arbitrary
buffer objects as arguments.  (Contributed by Antoine Pitrou in
bpo-15958@footnote{https://bugs.python.org/issue15958}.)

@item 
The @ref{1b1,,int} constructor now accepts any object that has an @code{__index__}
method for its @emph{base} argument.  (Contributed by Mark Dickinson in
bpo-16772@footnote{https://bugs.python.org/issue16772}.)

@item 
Frame objects now have a @ref{a7f,,clear()} method that clears all
references to local variables from the frame.  (Contributed by Antoine Pitrou
in bpo-17934@footnote{https://bugs.python.org/issue17934}.)

@item 
@ref{573,,memoryview} is now registered as a @ref{1d,,Sequence},
and supports the @ref{2f6,,reversed()} builtin.  (Contributed by Nick Coghlan
and Claudiu Popa in bpo-18690@footnote{https://bugs.python.org/issue18690} and bpo-19078@footnote{https://bugs.python.org/issue19078}.)

@item 
Signatures reported by @ref{3bc,,help()} have been modified and improved in
several cases as a result of the introduction of Argument Clinic and other
changes to the @ref{9c,,inspect} and @ref{d4,,pydoc} modules.

@item 
@ref{a80,,__length_hint__()} is now part of the formal language
specification (see 
@geindex Python Enhancement Proposals; PEP 424
PEP 424@footnote{https://www.python.org/dev/peps/pep-0424}).  (Contributed by Armin Ronacher in
bpo-16148@footnote{https://bugs.python.org/issue16148}.)
@end itemize

@node New Modules<8>,Improved Modules<8>,New Features<11>,What’s New In Python 3 4
@anchor{whatsnew/3 4 new-modules}@anchor{a81}
@subsection New Modules


@menu
* asyncio: asyncio<8>. 
* ensurepip:: 
* enum: enum<6>. 
* pathlib: pathlib<7>. 
* selectors: selectors<2>. 
* statistics: statistics<4>. 
* tracemalloc: tracemalloc<4>. 

@end menu

@node asyncio<8>,ensurepip,,New Modules<8>
@anchor{whatsnew/3 4 asyncio}@anchor{a82}@anchor{whatsnew/3 4 whatsnew-asyncio}@anchor{a4f}
@subsubsection asyncio


The new @ref{9,,asyncio} module (defined in 
@geindex Python Enhancement Proposals; PEP 3156
PEP 3156@footnote{https://www.python.org/dev/peps/pep-3156}) provides a standard
pluggable event loop model for Python, providing solid asynchronous IO
support in the standard library, and making it easier for other event loop
implementations to interoperate with the standard library and each other.

For Python 3.4, this module is considered a @ref{621,,provisional API}.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3156

@item PEP 3156@footnote{https://www.python.org/dev/peps/pep-3156} – Asynchronous IO Support Rebooted: the “asyncio” Module

PEP written and implementation led by Guido van Rossum.
@end table


@node ensurepip,enum<6>,asyncio<8>,New Modules<8>
@anchor{whatsnew/3 4 ensurepip}@anchor{a83}@anchor{whatsnew/3 4 whatsnew-ensurepip}@anchor{a50}
@subsubsection ensurepip


The new @ref{75,,ensurepip} module is the primary infrastructure for the
@geindex Python Enhancement Proposals; PEP 453
PEP 453@footnote{https://www.python.org/dev/peps/pep-0453} implementation.  In the normal course of events end users will not
need to interact with this module, but it can be used to manually bootstrap
@code{pip} if the automated bootstrapping into an installation or virtual
environment was declined.

@ref{75,,ensurepip} includes a bundled copy of @code{pip}, up-to-date as of the first
release candidate of the release of CPython with which it ships (this applies
to both maintenance releases and feature releases).  @code{ensurepip} does not
access the internet.  If the installation has internet access, after
@code{ensurepip} is run the bundled @code{pip} can be used to upgrade @code{pip} to a
more recent release than the bundled one.  (Note that such an upgraded version
of @code{pip} is considered to be a separately installed package and will not be
removed if Python is uninstalled.)

The module is named @emph{ensure}pip because if called when @code{pip} is already
installed, it does nothing.  It also has an @code{--upgrade} option that will
cause it to install the bundled copy of @code{pip} if the existing installed
version of @code{pip} is older than the bundled copy.

@node enum<6>,pathlib<7>,ensurepip,New Modules<8>
@anchor{whatsnew/3 4 enum}@anchor{a84}@anchor{whatsnew/3 4 whatsnew-enum}@anchor{a51}
@subsubsection enum


The new @ref{76,,enum} module (defined in 
@geindex Python Enhancement Proposals; PEP 435
PEP 435@footnote{https://www.python.org/dev/peps/pep-0435}) provides a standard
implementation of enumeration types, allowing other modules (such as
@ref{ea,,socket}) to provide more informative error messages and better
debugging support by replacing opaque integer constants with backwards
compatible enumeration values.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 435

@item PEP 435@footnote{https://www.python.org/dev/peps/pep-0435} – Adding an Enum type to the Python standard library

PEP written by Barry Warsaw, Eli Bendersky and Ethan Furman,
implemented by Ethan Furman.
@end table


@node pathlib<7>,selectors<2>,enum<6>,New Modules<8>
@anchor{whatsnew/3 4 pathlib}@anchor{a85}@anchor{whatsnew/3 4 whatsnew-pathlib}@anchor{a52}
@subsubsection pathlib


The new @ref{c3,,pathlib} module offers classes representing filesystem paths
with semantics appropriate for different operating systems.  Path classes are
divided between @emph{pure paths}, which provide purely computational operations
without I/O, and @emph{concrete paths}, which inherit from pure paths but also
provide I/O operations.

For Python 3.4, this module is considered a @ref{621,,provisional API}.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 428

@item PEP 428@footnote{https://www.python.org/dev/peps/pep-0428} – The pathlib module – object-oriented filesystem paths

PEP written and implemented by Antoine Pitrou.
@end table


@node selectors<2>,statistics<4>,pathlib<7>,New Modules<8>
@anchor{whatsnew/3 4 selectors}@anchor{a86}@anchor{whatsnew/3 4 whatsnew-selectors}@anchor{a53}
@subsubsection selectors


The new @ref{e1,,selectors} module (created as part of implementing 
@geindex Python Enhancement Proposals; PEP 3156
PEP 3156@footnote{https://www.python.org/dev/peps/pep-3156})
allows high-level and efficient I/O multiplexing, built upon the
@ref{e0,,select} module primitives.

@node statistics<4>,tracemalloc<4>,selectors<2>,New Modules<8>
@anchor{whatsnew/3 4 statistics}@anchor{a87}@anchor{whatsnew/3 4 whatsnew-statistics}@anchor{a54}
@subsubsection statistics


The new @ref{f0,,statistics} module (defined in 
@geindex Python Enhancement Proposals; PEP 450
PEP 450@footnote{https://www.python.org/dev/peps/pep-0450}) offers some core
statistics functionality directly in the standard library. This module
supports calculation of the mean, median, mode, variance and standard
deviation of a data series.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 450

@item PEP 450@footnote{https://www.python.org/dev/peps/pep-0450} – Adding A Statistics Module To The Standard Library

PEP written and implemented by Steven D’Aprano
@end table


@node tracemalloc<4>,,statistics<4>,New Modules<8>
@anchor{whatsnew/3 4 tracemalloc}@anchor{a88}@anchor{whatsnew/3 4 whatsnew-tracemalloc}@anchor{a55}
@subsubsection tracemalloc


The new @ref{11b,,tracemalloc} module (defined in 
@geindex Python Enhancement Proposals; PEP 454
PEP 454@footnote{https://www.python.org/dev/peps/pep-0454}) is a debug tool to
trace memory blocks allocated by Python. It provides the following information:


@itemize *

@item 
Trace where an object was allocated

@item 
Statistics on allocated memory blocks per filename and per line number:
total size, number and average size of allocated memory blocks

@item 
Compute the differences between two snapshots to detect memory leaks
@end itemize


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 454

@item PEP 454@footnote{https://www.python.org/dev/peps/pep-0454} – Add a new tracemalloc module to trace Python memory allocations

PEP written and implemented by Victor Stinner
@end table


@node Improved Modules<8>,CPython Implementation Changes,New Modules<8>,What’s New In Python 3 4
@anchor{whatsnew/3 4 improved-modules}@anchor{a89}
@subsection Improved Modules


@menu
* abc:: 
* aifc: aifc<2>. 
* argparse: argparse<4>. 
* audioop:: 
* base64: base64<2>. 
* collections: collections<6>. 
* colorsys:: 
* contextlib: contextlib<5>. 
* dbm: dbm<5>. 
* dis: dis<2>. 
* doctest: doctest<3>. 
* email: email<3>. 
* filecmp:: 
* functools: functools<4>. 
* gc: gc<5>. 
* glob: glob<3>. 
* hashlib: hashlib<4>. 
* hmac: hmac<3>. 
* html:: 
* http: http<3>. 
* idlelib and IDLE: idlelib and IDLE<4>. 
* importlib: importlib<7>. 
* inspect: inspect<7>. 
* ipaddress: ipaddress<4>. 
* logging: logging<5>. 
* marshal:: 
* mmap: mmap<2>. 
* multiprocessing: multiprocessing<6>. 
* operator: operator<3>. 
* os: os<9>. 
* pdb: pdb<4>. 
* pickle: pickle<4>. 
* plistlib: plistlib<2>. 
* poplib: poplib<3>. 
* pprint: pprint<4>. 
* pty:: 
* pydoc: pydoc<4>. 
* re: re<5>. 
* resource:: 
* select:: 
* shelve: shelve<2>. 
* shutil: shutil<3>. 
* smtpd: smtpd<2>. 
* smtplib: smtplib<3>. 
* socket: socket<9>. 
* sqlite3: sqlite3<6>. 
* ssl: ssl<8>. 
* stat:: 
* struct: struct<2>. 
* subprocess: subprocess<4>. 
* sunau: sunau<2>. 
* sys: sys<9>. 
* tarfile: tarfile<3>. 
* textwrap:: 
* threading: threading<6>. 
* traceback: traceback<4>. 
* types: types<4>. 
* urllib: urllib<2>. 
* unittest: unittest<5>. 
* venv: venv<5>. 
* wave: wave<2>. 
* weakref: weakref<2>. 
* xml.etree: xml etree<2>. 
* zipfile: zipfile<4>. 

@end menu

@node abc,aifc<2>,,Improved Modules<8>
@anchor{whatsnew/3 4 abc}@anchor{a8a}
@subsubsection abc


New function @ref{a8b,,abc.get_cache_token()} can be used to know when to invalidate
caches that are affected by changes in the object graph.  (Contributed
by Łukasz Langa in bpo-16832@footnote{https://bugs.python.org/issue16832}.)

New class @ref{a8c,,ABC} has @ref{a8d,,ABCMeta} as its meta class.
Using @code{ABC} as a base class has essentially the same effect as specifying
@code{metaclass=abc.ABCMeta}, but is simpler to type and easier to read.
(Contributed by Bruno Dupuis in bpo-16049@footnote{https://bugs.python.org/issue16049}.)

@node aifc<2>,argparse<4>,abc,Improved Modules<8>
@anchor{whatsnew/3 4 aifc}@anchor{a8e}
@subsubsection aifc


The @ref{a8f,,getparams()} method now returns a namedtuple rather than a
plain tuple.  (Contributed by Claudiu Popa in bpo-17818@footnote{https://bugs.python.org/issue17818}.)

@ref{718,,aifc.open()} now supports the context management protocol: when used in a
@ref{168,,with} block, the @ref{a90,,close()} method of the returned
object will be called automatically at the end of the block.  (Contributed by
Serhiy Storchacha in bpo-16486@footnote{https://bugs.python.org/issue16486}.)

The @ref{a91,,writeframesraw()} and @ref{a92,,writeframes()}
methods now accept any @ref{87e,,bytes-like object}.  (Contributed by Serhiy
Storchaka in bpo-8311@footnote{https://bugs.python.org/issue8311}.)

@node argparse<4>,audioop,aifc<2>,Improved Modules<8>
@anchor{whatsnew/3 4 argparse}@anchor{a93}
@subsubsection argparse


The @ref{a94,,FileType} class now accepts @emph{encoding} and
@emph{errors} arguments, which are passed through to @ref{1c8,,open()}.  (Contributed
by Lucas Maystre in bpo-11175@footnote{https://bugs.python.org/issue11175}.)

@node audioop,base64<2>,argparse<4>,Improved Modules<8>
@anchor{whatsnew/3 4 audioop}@anchor{a95}
@subsubsection audioop


@ref{c,,audioop} now supports 24-bit samples.  (Contributed by Serhiy Storchaka
in bpo-12866@footnote{https://bugs.python.org/issue12866}.)

New @ref{a96,,byteswap()} function converts big-endian samples to
little-endian and vice versa.  (Contributed by Serhiy Storchaka in
bpo-19641@footnote{https://bugs.python.org/issue19641}.)

All @ref{c,,audioop} functions now accept any @ref{87e,,bytes-like object}.  Strings
are not accepted: they didn’t work before, now they raise an error right away.
(Contributed by Serhiy Storchaka in bpo-16685@footnote{https://bugs.python.org/issue16685}.)

@node base64<2>,collections<6>,audioop,Improved Modules<8>
@anchor{whatsnew/3 4 base64}@anchor{a97}
@subsubsection base64


The encoding and decoding functions in @ref{d,,base64} now accept any
@ref{87e,,bytes-like object} in cases where it previously required a
@ref{2f0,,bytes} or @ref{2f1,,bytearray} instance.  (Contributed by Nick Coghlan in
bpo-17839@footnote{https://bugs.python.org/issue17839}.)

New functions @ref{a98,,a85encode()}, @ref{a99,,a85decode()},
@ref{a9a,,b85encode()}, and @ref{a9b,,b85decode()} provide the ability to
encode and decode binary data from and to @code{Ascii85} and the git/mercurial
@code{Base85} formats, respectively.  The @code{a85} functions have options that can
be used to make them compatible with the variants of the @code{Ascii85} encoding,
including the Adobe variant.  (Contributed by Martin Morrison, the Mercurial
project, Serhiy Storchaka, and Antoine Pitrou in bpo-17618@footnote{https://bugs.python.org/issue17618}.)

@node collections<6>,colorsys,base64<2>,Improved Modules<8>
@anchor{whatsnew/3 4 collections}@anchor{a9c}
@subsubsection collections


The @ref{a9d,,ChainMap.new_child()} method now accepts an @emph{m} argument specifying
the child map to add to the chain.  This allows an existing mapping and/or a
custom mapping type to be used for the child.  (Contributed by Vinay Sajip in
bpo-16613@footnote{https://bugs.python.org/issue16613}.)

@node colorsys,contextlib<5>,collections<6>,Improved Modules<8>
@anchor{whatsnew/3 4 colorsys}@anchor{a9e}
@subsubsection colorsys


The number of digits in the coefficients for the RGB — YIQ conversions have
been expanded so that they match the FCC NTSC versions.  The change in
results should be less than 1% and may better match results found elsewhere.
(Contributed by Brian Landers and Serhiy Storchaka in bpo-14323@footnote{https://bugs.python.org/issue14323}.)

@node contextlib<5>,dbm<5>,colorsys,Improved Modules<8>
@anchor{whatsnew/3 4 contextlib}@anchor{a9f}
@subsubsection contextlib


The new @ref{aa0,,contextlib.suppress} context manager helps to clarify the
intent of code that deliberately suppresses exceptions from a single
statement.  (Contributed by Raymond Hettinger in bpo-15806@footnote{https://bugs.python.org/issue15806} and
Zero Piraeus in bpo-19266@footnote{https://bugs.python.org/issue19266}.)

The new @ref{947,,contextlib.redirect_stdout()} context manager makes it easier
for utility scripts to handle inflexible APIs that write their output to
@ref{5ef,,sys.stdout} and don’t provide any options to redirect it.  Using the
context manager, the @ref{5ef,,sys.stdout} output can be redirected to any
other stream or, in conjunction with @ref{aa1,,io.StringIO}, to a string.
The latter can be especially useful, for example, to capture output
from a function that was written to implement a command line interface.
It is recommended only for utility scripts because it affects the
global state of @ref{5ef,,sys.stdout}.  (Contributed by Raymond Hettinger
in bpo-15805@footnote{https://bugs.python.org/issue15805}.)

The @ref{22,,contextlib} documentation has also been updated to include a
@ref{aa2,,discussion} of the
differences between single use, reusable and reentrant context managers.

@node dbm<5>,dis<2>,contextlib<5>,Improved Modules<8>
@anchor{whatsnew/3 4 dbm}@anchor{aa3}
@subsubsection dbm


@ref{aa4,,dbm.open()} objects now support the context management protocol.  When
used in a @ref{168,,with} statement, the @code{close} method of the database
object will be called automatically at the end of the block.  (Contributed by
Claudiu Popa and Nick Coghlan in bpo-19282@footnote{https://bugs.python.org/issue19282}.)

@node dis<2>,doctest<3>,dbm<5>,Improved Modules<8>
@anchor{whatsnew/3 4 dis}@anchor{aa5}
@subsubsection dis


Functions @ref{aa6,,show_code()}, @ref{655,,dis()}, @ref{aa7,,distb()}, and
@ref{aa8,,disassemble()} now accept a keyword-only @emph{file} argument that
controls where they write their output.

The @ref{36,,dis} module is now built around an @ref{aa9,,Instruction} class
that provides object oriented access to the details of each individual bytecode
operation.

A new method, @ref{aaa,,get_instructions()}, provides an iterator that emits
the Instruction stream for a given piece of Python code.  Thus it is now
possible to write a program that inspects and manipulates a bytecode
object in ways different from those provided by the @ref{36,,dis} module
itself.  For example:

@example
>>> import dis
>>> for instr in dis.get_instructions(lambda x: x + 1):
...     print(instr.opname)
LOAD_FAST
LOAD_CONST
BINARY_ADD
RETURN_VALUE
@end example

The various display tools in the @ref{36,,dis} module have been rewritten to use
these new components.

In addition, a new application-friendly class @ref{aab,,Bytecode} provides
an object-oriented API for inspecting bytecode in both in human-readable form
and for iterating over instructions.  The @ref{aab,,Bytecode} constructor
takes the same arguments that @code{get_instruction()} does (plus an
optional @emph{current_offset}), and the resulting object can be iterated to produce
@ref{aa9,,Instruction} objects.  But it also has a @ref{aac,,dis}
method, equivalent to calling @ref{655,,dis} on the constructor argument, but
returned as a multi-line string:

@example
>>> bytecode = dis.Bytecode(lambda x: x + 1, current_offset=3)
>>> for instr in bytecode:
...     print('@{@} (@{@})'.format(instr.opname, instr.opcode))
LOAD_FAST (124)
LOAD_CONST (100)
BINARY_ADD (23)
RETURN_VALUE (83)
>>> bytecode.dis().splitlines()       
['  1           0 LOAD_FAST                0 (x)',
 '      -->     3 LOAD_CONST               1 (1)',
 '              6 BINARY_ADD',
 '              7 RETURN_VALUE']
@end example

@ref{aab,,Bytecode} also has a class method,
@ref{aad,,from_traceback()}, that provides the ability to manipulate a
traceback (that is, @code{print(Bytecode.from_traceback(tb).dis())} is equivalent
to @code{distb(tb)}).

(Contributed by Nick Coghlan, Ryan Kelly and Thomas Kluyver in bpo-11816@footnote{https://bugs.python.org/issue11816}
and Claudiu Popa in bpo-17916@footnote{https://bugs.python.org/issue17916}.)

New function @ref{aae,,stack_effect()} computes the effect on the Python stack
of a given opcode and argument, information that is not otherwise available.
(Contributed by Larry Hastings in bpo-19722@footnote{https://bugs.python.org/issue19722}.)

@node doctest<3>,email<3>,dis<2>,Improved Modules<8>
@anchor{whatsnew/3 4 doctest}@anchor{aaf}
@subsubsection doctest


A new @ref{ab0,,option flag}, @ref{ab1,,FAIL_FAST}, halts
test running as soon as the first failure is detected.  (Contributed by R.
David Murray and Daniel Urban in bpo-16522@footnote{https://bugs.python.org/issue16522}.)

The @ref{63,,doctest} command line interface now uses @ref{5,,argparse}, and has two
new options, @code{-o} and @code{-f}.  @code{-o} allows @ref{ab0,,doctest options} to be specified on the command line, and @code{-f} is a
shorthand for @code{-o FAIL_FAST} (to parallel the similar option supported by the
@ref{122,,unittest} CLI).  (Contributed by R. David Murray in bpo-11390@footnote{https://bugs.python.org/issue11390}.)

@ref{63,,doctest} will now find doctests in extension module @code{__doc__} strings.
(Contributed by Zachary Ware in bpo-3158@footnote{https://bugs.python.org/issue3158}.)

@node email<3>,filecmp,doctest<3>,Improved Modules<8>
@anchor{whatsnew/3 4 email}@anchor{ab2}
@subsubsection email


@ref{ab3,,as_string()} now accepts a @emph{policy} argument to
override the default policy of the message when generating a string
representation of it.  This means that @code{as_string} can now be used in more
circumstances, instead of having to create and use a @ref{69,,generator} in
order to pass formatting parameters to its @code{flatten} method.  (Contributed by
R. David Murray in bpo-18600@footnote{https://bugs.python.org/issue18600}.)

New method @ref{ab4,,as_bytes()} added to produce a bytes
representation of the message in a fashion similar to how @code{as_string}
produces a string representation.  It does not accept the @emph{maxheaderlen}
argument, but does accept the @emph{unixfrom} and @emph{policy} arguments. The
@ref{7e3,,Message} @ref{ab5,,__bytes__()} method
calls it, meaning that @code{bytes(mymsg)} will now produce the intuitive
result:  a bytes object containing the fully formatted message.  (Contributed
by R. David Murray in bpo-18600@footnote{https://bugs.python.org/issue18600}.)

The @ref{ab6,,Message.set_param()} message now accepts a @emph{replace} keyword argument.
When specified, the associated header will be updated without changing
its location in the list of headers.  For backward compatibility, the default
is @code{False}.  (Contributed by R. David Murray in bpo-18891@footnote{https://bugs.python.org/issue18891}.)
@anchor{whatsnew/3 4 whatsnew-email-contentmanager}@anchor{a59}
A pair of new subclasses of @ref{7e3,,Message} have been added
(@ref{7e4,,EmailMessage} and @ref{ab7,,MIMEPart}), along with a new sub-module,
@ref{66,,contentmanager} and a new @ref{70,,policy} attribute
@ref{ab8,,content_manager}.  All documentation is
currently in the new module, which is being added as part of email’s new
@ref{621,,provisional API}.  These classes provide a number of new methods that
make extracting content from and inserting content into email messages much
easier.  For details, see the @ref{66,,contentmanager} documentation and
the @ref{ab9,,email; Examples}.  These API additions complete the
bulk of the work that was planned as part of the email6 project.  The currently
provisional API is scheduled to become final in Python 3.5 (possibly with a few
minor additions in the area of error handling).  (Contributed by R. David
Murray in bpo-18891@footnote{https://bugs.python.org/issue18891}.)

@node filecmp,functools<4>,email<3>,Improved Modules<8>
@anchor{whatsnew/3 4 filecmp}@anchor{aba}
@subsubsection filecmp


A new @ref{abb,,clear_cache()} function provides the ability to clear the
@ref{7a,,filecmp} comparison cache, which uses @ref{516,,os.stat()} information to
determine if the file has changed since the last compare.  This can be used,
for example, if the file might have been changed and re-checked in less time
than the resolution of a particular filesystem’s file modification time field.
(Contributed by Mark Levitt in bpo-18149@footnote{https://bugs.python.org/issue18149}.)

New module attribute @ref{abc,,DEFAULT_IGNORES} provides the list of
directories that are used as the default value for the @emph{ignore} parameter of
the @ref{abd,,dircmp()} function.  (Contributed by Eli Bendersky in
bpo-15442@footnote{https://bugs.python.org/issue15442}.)

@node functools<4>,gc<5>,filecmp,Improved Modules<8>
@anchor{whatsnew/3 4 functools}@anchor{abe}
@subsubsection functools


The new @ref{593,,partialmethod()} descriptor brings partial argument
application to descriptors, just as @ref{a3f,,partial()} provides
for normal callables. The new descriptor also makes it easier to get
arbitrary callables (including @ref{a3f,,partial()} instances)
to behave like normal instance methods when included in a class definition.
(Contributed by Alon Horev and Nick Coghlan in bpo-4331@footnote{https://bugs.python.org/issue4331}.)
@anchor{whatsnew/3 4 whatsnew-singledispatch}@anchor{a56}
The new @ref{65b,,singledispatch()} decorator brings support for
single-dispatch generic functions to the Python standard library. Where
object oriented programming focuses on grouping multiple operations on a
common set of data into a class, a generic function focuses on grouping
multiple implementations of an operation that allows it to work with
@emph{different} kinds of data.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 443

@item PEP 443@footnote{https://www.python.org/dev/peps/pep-0443} – Single-dispatch generic functions

PEP written and implemented by Łukasz Langa.
@end table


@ref{abf,,total_ordering()} now supports a return value of
@ref{24c,,NotImplemented} from the underlying comparison function.  (Contributed
by Katie Miller in bpo-10042@footnote{https://bugs.python.org/issue10042}.)

A pure-python version of the @ref{a3f,,partial()} function is now in the
stdlib; in CPython it is overridden by the C accelerated version, but it is
available for other implementations to use.  (Contributed by Brian Thorne in
bpo-12428@footnote{https://bugs.python.org/issue12428}.)

@node gc<5>,glob<3>,functools<4>,Improved Modules<8>
@anchor{whatsnew/3 4 gc}@anchor{ac0}
@subsubsection gc


New function @ref{ac1,,get_stats()} returns a list of three per-generation
dictionaries containing the collections statistics since interpreter startup.
(Contributed by Antoine Pitrou in bpo-16351@footnote{https://bugs.python.org/issue16351}.)

@node glob<3>,hashlib<4>,gc<5>,Improved Modules<8>
@anchor{whatsnew/3 4 glob}@anchor{ac2}
@subsubsection glob


A new function @ref{ac3,,escape()} provides a way to escape special characters
in a filename so that they do not become part of the globbing expansion but are
instead matched literally.  (Contributed by Serhiy Storchaka in bpo-8402@footnote{https://bugs.python.org/issue8402}.)

@node hashlib<4>,hmac<3>,glob<3>,Improved Modules<8>
@anchor{whatsnew/3 4 hashlib}@anchor{ac4}
@subsubsection hashlib


A new @ref{28b,,hashlib.pbkdf2_hmac()} function provides
the PKCS#5 password-based key derivation function 2@footnote{https://en.wikipedia.org/wiki/PBKDF2}.  (Contributed by Christian
Heimes in bpo-18582@footnote{https://bugs.python.org/issue18582}.)

The @ref{ac5,,name} attribute of @ref{88,,hashlib} hash objects is now
a formally supported interface.  It has always existed in CPython’s
@ref{88,,hashlib} (although it did not return lower case names for all supported
hashes), but it was not a public interface and so some other Python
implementations have not previously supported it.  (Contributed by Jason R.
Coombs in bpo-18532@footnote{https://bugs.python.org/issue18532}.)

@node hmac<3>,html,hashlib<4>,Improved Modules<8>
@anchor{whatsnew/3 4 hmac}@anchor{ac6}
@subsubsection hmac


@ref{8a,,hmac} now accepts @code{bytearray} as well as @code{bytes} for the @emph{key}
argument to the @ref{ac7,,new()} function, and the @emph{msg} parameter to both the
@ref{ac7,,new()} function and the @ref{ac8,,update()} method now
accepts any type supported by the @ref{88,,hashlib} module.  (Contributed
by Jonas Borgström in bpo-18240@footnote{https://bugs.python.org/issue18240}.)

The @emph{digestmod} argument to the @ref{ac7,,hmac.new()} function may now be any hash
digest name recognized by @ref{88,,hashlib}.  In addition, the current behavior in
which the value of @emph{digestmod} defaults to @code{MD5} is deprecated: in a
future version of Python there will be no default value.  (Contributed by
Christian Heimes in bpo-17276@footnote{https://bugs.python.org/issue17276}.)

With the addition of @ref{ac9,,block_size} and @ref{aca,,name}
attributes (and the formal documentation of the @ref{acb,,digest_size}
attribute), the @ref{8a,,hmac} module now conforms fully to the 
@geindex Python Enhancement Proposals; PEP 247
PEP 247@footnote{https://www.python.org/dev/peps/pep-0247} API.
(Contributed by Christian Heimes in bpo-18775@footnote{https://bugs.python.org/issue18775}.)

@node html,http<3>,hmac<3>,Improved Modules<8>
@anchor{whatsnew/3 4 html}@anchor{acc}
@subsubsection html


New function @ref{45a,,unescape()} function converts HTML5 character references to
the corresponding Unicode characters.  (Contributed by Ezio Melotti in
bpo-2927@footnote{https://bugs.python.org/issue2927}.)

@ref{33c,,HTMLParser} accepts a new keyword argument
@emph{convert_charrefs} that, when @code{True}, automatically converts all character
references.  For backward-compatibility, its value defaults to @code{False}, but
it will change to @code{True} in a future version of Python, so you are invited to
set it explicitly and update your code to use this new feature.  (Contributed
by Ezio Melotti in bpo-13633@footnote{https://bugs.python.org/issue13633}.)

The @emph{strict} argument of @ref{33c,,HTMLParser} is now deprecated.
(Contributed by Ezio Melotti in bpo-15114@footnote{https://bugs.python.org/issue15114}.)

@node http<3>,idlelib and IDLE<4>,html,Improved Modules<8>
@anchor{whatsnew/3 4 http}@anchor{acd}
@subsubsection http


@ref{ace,,send_error()} now accepts an
optional additional @emph{explain} parameter which can be used to provide an
extended error description, overriding the hardcoded default if there is one.
This extended error description will be formatted using the
@code{error_message_format} attribute and sent as the body
of the error response.  (Contributed by Karl Cow in bpo-12921@footnote{https://bugs.python.org/issue12921}.)

The @ref{92,,http.server} @ref{acf,,command line interface} now has
a @code{-b/--bind} option that causes the server to listen on a specific address.
(Contributed by Malte Swart in bpo-17764@footnote{https://bugs.python.org/issue17764}.)

@node idlelib and IDLE<4>,importlib<7>,http<3>,Improved Modules<8>
@anchor{whatsnew/3 4 idlelib-and-idle}@anchor{ad0}
@subsubsection idlelib and IDLE


Since idlelib implements the IDLE shell and editor and is not intended for
import by other programs, it gets improvements with every release. See
@code{Lib/idlelib/NEWS.txt} for a cumulative list of changes since 3.3.0,
as well as changes made in future 3.4.x releases. This file is also available
from the IDLE Help ‣ About IDLE dialog.

@node importlib<7>,inspect<7>,idlelib and IDLE<4>,Improved Modules<8>
@anchor{whatsnew/3 4 importlib}@anchor{ad1}
@subsubsection importlib


The @ref{ad2,,InspectLoader} ABC defines a new method,
@ref{971,,source_to_code()} that accepts source
data and a path and returns a code object.  The default implementation
is equivalent to @code{compile(data, path, 'exec', dont_inherit=True)}.
(Contributed by Eric Snow and Brett Cannon in bpo-15627@footnote{https://bugs.python.org/issue15627}.)

@ref{ad2,,InspectLoader} also now has a default implementation
for the @ref{ad3,,get_code()} method.  However,
it will normally be desirable to override the default implementation
for performance reasons.  (Contributed by Brett Cannon in bpo-18072@footnote{https://bugs.python.org/issue18072}.)

The @ref{66a,,reload()} function has been moved from @ref{95,,imp} to
@ref{96,,importlib} as part of the @ref{95,,imp} module deprecation.  (Contributed by
Berker Peksag in bpo-18193@footnote{https://bugs.python.org/issue18193}.)

@ref{9b,,importlib.util} now has a @ref{ad4,,MAGIC_NUMBER} attribute
providing access to the bytecode version number.  This replaces the
@ref{ad5,,get_magic()} function in the deprecated @ref{95,,imp} module.
(Contributed by Brett Cannon in bpo-18192@footnote{https://bugs.python.org/issue18192}.)

New @ref{9b,,importlib.util} functions @ref{7f9,,cache_from_source()}
and @ref{7fa,,source_from_cache()} replace the same-named functions
in the deprecated @ref{95,,imp} module.  (Contributed by Brett Cannon in
bpo-18194@footnote{https://bugs.python.org/issue18194}.)

The @ref{96,,importlib} bootstrap @code{NamespaceLoader} now conforms to
the @code{InspectLoader} ABC, which means that @code{runpy} and
@code{python -m} can now be used with namespace packages.  (Contributed
by Brett Cannon in bpo-18058@footnote{https://bugs.python.org/issue18058}.)

@ref{9b,,importlib.util} has a new function @ref{ad6,,decode_source()}
that decodes source from bytes using universal newline processing.  This is
useful for implementing @code{InspectLoader.get_source()} methods.

@ref{7f8,,importlib.machinery.ExtensionFileLoader} now has a
@ref{ad7,,get_filename()} method.  This was
inadvertently omitted in the original implementation.  (Contributed by Eric
Snow in bpo-19152@footnote{https://bugs.python.org/issue19152}.)

@node inspect<7>,ipaddress<4>,importlib<7>,Improved Modules<8>
@anchor{whatsnew/3 4 inspect}@anchor{ad8}
@subsubsection inspect


The @ref{9c,,inspect} module now offers a basic @ref{ad9,,command line interface} to quickly display source code and other
information for modules, classes and functions.  (Contributed by Claudiu Popa
and Nick Coghlan in bpo-18626@footnote{https://bugs.python.org/issue18626}.)

@ref{ada,,unwrap()} makes it easy to unravel wrapper function chains
created by @ref{adb,,functools.wraps()} (and any other API that sets the
@code{__wrapped__} attribute on a wrapper function).  (Contributed by
Daniel Urban, Aaron Iles and Nick Coghlan in bpo-13266@footnote{https://bugs.python.org/issue13266}.)

As part of the implementation of the new @ref{76,,enum} module, the
@ref{9c,,inspect} module now has substantially better support for custom
@code{__dir__} methods and dynamic class attributes provided through
metaclasses.  (Contributed by Ethan Furman in bpo-18929@footnote{https://bugs.python.org/issue18929} and
bpo-19030@footnote{https://bugs.python.org/issue19030}.)

@ref{1c2,,getfullargspec()} and @code{getargspec()}
now use the @ref{1c1,,signature()} API. This allows them to
support a much broader range of callables, including those with
@code{__signature__} attributes, those with metadata provided by argument
clinic, @ref{a3f,,functools.partial()} objects and more. Note that, unlike
@ref{1c1,,signature()}, these functions still ignore @code{__wrapped__}
attributes, and report the already bound first argument for bound methods,
so it is still necessary to update your code to use
@ref{1c1,,signature()} directly if those features are desired.
(Contributed by Yury Selivanov in bpo-17481@footnote{https://bugs.python.org/issue17481}.)

@ref{1c1,,signature()} now supports duck types of CPython functions,
which adds support for functions compiled with Cython.  (Contributed
by Stefan Behnel and Yury Selivanov in bpo-17159@footnote{https://bugs.python.org/issue17159}.)

@node ipaddress<4>,logging<5>,inspect<7>,Improved Modules<8>
@anchor{whatsnew/3 4 ipaddress}@anchor{adc}
@subsubsection ipaddress


@ref{9e,,ipaddress} was added to the standard library in Python 3.3 as a
@ref{621,,provisional API}. With the release of Python 3.4, this qualification
has been removed: @ref{9e,,ipaddress} is now considered a stable API, covered
by the normal standard library requirements to maintain backwards
compatibility.

A new @ref{add,,is_global} property is @code{True} if
an address is globally routeable.  (Contributed by Peter Moody in
bpo-17400@footnote{https://bugs.python.org/issue17400}.)

@node logging<5>,marshal,ipaddress<4>,Improved Modules<8>
@anchor{whatsnew/3 4 logging}@anchor{ade}
@subsubsection logging


The @ref{adf,,TimedRotatingFileHandler} has a new @emph{atTime}
parameter that can be used to specify the time of day when rollover should
happen.  (Contributed by Ronald Oussoren in bpo-9556@footnote{https://bugs.python.org/issue9556}.)

@ref{ae0,,SocketHandler} and
@ref{ae1,,DatagramHandler} now support Unix domain sockets (by
setting @emph{port} to @code{None}).  (Contributed by Vinay Sajip in commit
ce46195b56a9.)

@ref{678,,fileConfig()} now accepts a
@ref{ae2,,configparser.RawConfigParser} subclass instance for the @emph{fname}
parameter.  This facilitates using a configuration file when logging
configuration is just a part of the overall application configuration, or where
the application modifies the configuration before passing it to
@ref{678,,fileConfig()}.  (Contributed by Vinay Sajip in
bpo-16110@footnote{https://bugs.python.org/issue16110}.)

Logging configuration data received from a socket via the
@ref{ae3,,logging.config.listen()} function can now be validated before being
processed by supplying a verification function as the argument to the new
@emph{verify} keyword argument.  (Contributed by Vinay Sajip in bpo-15452@footnote{https://bugs.python.org/issue15452}.)

@node marshal,mmap<2>,logging<5>,Improved Modules<8>
@anchor{whatsnew/3 4 marshal}@anchor{ae4}@anchor{whatsnew/3 4 whatsnew-marshal-3}@anchor{a4e}
@subsubsection marshal


The default @ref{ac,,marshal} version has been bumped to 3.  The code implementing
the new version restores the Python2 behavior of recording only one copy of
interned strings and preserving the interning on deserialization, and extends
this “one copy” ability to any object type (including handling recursive
references).  This reduces both the size of @code{.pyc} files and the amount of
memory a module occupies in memory when it is loaded from a @code{.pyc} (or
@code{.pyo}) file.  (Contributed by Kristján Valur Jónsson in bpo-16475@footnote{https://bugs.python.org/issue16475},
with additional speedups by Antoine Pitrou in bpo-19219@footnote{https://bugs.python.org/issue19219}.)

@node mmap<2>,multiprocessing<6>,marshal,Improved Modules<8>
@anchor{whatsnew/3 4 mmap}@anchor{ae5}
@subsubsection mmap


mmap objects can now be @ref{12f,,weakref}ed.  (Contributed by Valerie Lambert in
bpo-4885@footnote{https://bugs.python.org/issue4885}.)

@node multiprocessing<6>,operator<3>,mmap<2>,Improved Modules<8>
@anchor{whatsnew/3 4 multiprocessing}@anchor{ae6}
@subsubsection multiprocessing

@anchor{whatsnew/3 4 whatsnew-multiprocessing-no-fork}@anchor{a58}
On Unix two new @ref{ae7,,start methods},
@code{spawn} and @code{forkserver}, have been added for starting processes using
@ref{b3,,multiprocessing}.  These make the mixing of processes with threads more
robust, and the @code{spawn} method matches the semantics that multiprocessing has
always used on Windows.  New function
@ref{ae8,,get_all_start_methods()} reports all start methods
available on the platform, @ref{ae9,,get_start_method()} reports
the current start method, and @ref{aea,,set_start_method()} sets
the start method.  (Contributed by Richard Oudkerk in bpo-8713@footnote{https://bugs.python.org/issue8713}.)

@ref{b3,,multiprocessing} also now has the concept of a @code{context}, which
determines how child processes are created.  New function
@ref{aeb,,get_context()} returns a context that uses a specified
start method.  It has the same API as the @ref{b3,,multiprocessing} module itself,
so you can use it to create @ref{aec,,Pool}s and other
objects that will operate within that context.  This allows a framework and an
application or different parts of the same application to use multiprocessing
without interfering with each other.  (Contributed by Richard Oudkerk in
bpo-18999@footnote{https://bugs.python.org/issue18999}.)

Except when using the old @emph{fork} start method, child processes no longer
inherit unneeded handles/file descriptors from their parents (part of
bpo-8713@footnote{https://bugs.python.org/issue8713}).

@ref{b3,,multiprocessing} now relies on @ref{dd,,runpy} (which implements the
@code{-m} switch) to initialise @code{__main__} appropriately in child processes
when using the @code{spawn} or @code{forkserver} start methods. This resolves some
edge cases where combining multiprocessing, the @code{-m} command line switch,
and explicit relative imports could cause obscure failures in child
processes.  (Contributed by Nick Coghlan in bpo-19946@footnote{https://bugs.python.org/issue19946}.)

@node operator<3>,os<9>,multiprocessing<6>,Improved Modules<8>
@anchor{whatsnew/3 4 operator}@anchor{aed}
@subsubsection operator


New function @ref{aee,,length_hint()} provides an implementation of the
specification for how the @ref{a80,,__length_hint__()} special method should
be used, as part of the 
@geindex Python Enhancement Proposals; PEP 424
PEP 424@footnote{https://www.python.org/dev/peps/pep-0424} formal specification of this language
feature.  (Contributed by Armin Ronacher in bpo-16148@footnote{https://bugs.python.org/issue16148}.)

There is now a pure-python version of the @ref{be,,operator} module available for
reference and for use by alternate implementations of Python.  (Contributed by
Zachary Ware in bpo-16694@footnote{https://bugs.python.org/issue16694}.)

@node os<9>,pdb<4>,operator<3>,Improved Modules<8>
@anchor{whatsnew/3 4 os}@anchor{aef}
@subsubsection os


There are new functions to get and set the @ref{a6e,,inheritable flag} of a file descriptor (@ref{a6f,,os.get_inheritable()},
@ref{a70,,os.set_inheritable()}) or a Windows handle
(@ref{a71,,os.get_handle_inheritable()}, @ref{a72,,os.set_handle_inheritable()}).

New function @ref{298,,cpu_count()} reports the number of CPUs available on the
platform on which Python is running (or @code{None} if the count can’t be
determined).  The @ref{af0,,multiprocessing.cpu_count()} function is now implemented
in terms of this function).  (Contributed by Trent Nelson, Yogesh Chaudhari,
Victor Stinner, and Charles-François Natali in bpo-17914@footnote{https://bugs.python.org/issue17914}.)

@ref{af1,,os.path.samestat()} is now available on the Windows platform (and the
@ref{af2,,os.path.samefile()} implementation is now shared between Unix and
Windows).  (Contributed by Brian Curtin in bpo-11939@footnote{https://bugs.python.org/issue11939}.)

@ref{51e,,os.path.ismount()} now recognizes volumes mounted below a drive
root on Windows.  (Contributed by Tim Golden in bpo-9035@footnote{https://bugs.python.org/issue9035}.)

@ref{8ee,,os.open()} supports two new flags on platforms that provide them,
@ref{af3,,O_PATH} (un-opened file descriptor), and @ref{af4,,O_TMPFILE}
(unnamed temporary file; as of 3.4.0 release available only on Linux systems
with a kernel version of 3.11 or newer that have uapi headers).  (Contributed
by Christian Heimes in bpo-18673@footnote{https://bugs.python.org/issue18673} and Benjamin Peterson, respectively.)

@node pdb<4>,pickle<4>,os<9>,Improved Modules<8>
@anchor{whatsnew/3 4 pdb}@anchor{af5}
@subsubsection pdb


@ref{c4,,pdb} has been enhanced to handle generators, @ref{4c5,,yield}, and
@code{yield from} in a more useful fashion.  This is especially helpful when
debugging @ref{9,,asyncio} based programs.  (Contributed by Andrew Svetlov and
Xavier de Gaye in bpo-16596@footnote{https://bugs.python.org/issue16596}.)

The @code{print} command has been removed from @ref{c4,,pdb}, restoring access to the
Python @ref{af6,,print()} function from the pdb command line.  Python2’s @code{pdb} did
not have a @code{print} command; instead, entering @code{print} executed the
@code{print} statement.  In Python3 @code{print} was mistakenly made an alias for the
pdb @ref{af7,,p} command.  @code{p}, however, prints the @code{repr} of its argument,
not the @code{str} like the Python2 @code{print} command did.  Worse, the Python3
@code{pdb print} command shadowed the Python3 @code{print} function, making it
inaccessible at the @code{pdb} prompt.  (Contributed by Connor Osborn in
bpo-18764@footnote{https://bugs.python.org/issue18764}.)

@node pickle<4>,plistlib<2>,pdb<4>,Improved Modules<8>
@anchor{whatsnew/3 4 pickle}@anchor{af8}@anchor{whatsnew/3 4 whatsnew-protocol-4}@anchor{a57}
@subsubsection pickle


@ref{c5,,pickle} now supports (but does not use by default) a new pickle protocol,
protocol 4.  This new protocol addresses a number of issues that were present
in previous protocols, such as the serialization of nested classes, very large
strings and containers, and classes whose @code{__new__()} method takes
keyword-only arguments.  It also provides some efficiency improvements.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3154

@item PEP 3154@footnote{https://www.python.org/dev/peps/pep-3154} – Pickle protocol 4

PEP written by Antoine Pitrou and implemented by Alexandre Vassalotti.
@end table


@node plistlib<2>,poplib<3>,pickle<4>,Improved Modules<8>
@anchor{whatsnew/3 4 plistlib}@anchor{af9}
@subsubsection plistlib


@ref{ca,,plistlib} now has an API that is similar to the standard pattern for
stdlib serialization protocols, with new @ref{44f,,load()},
@ref{451,,dump()}, @ref{450,,loads()}, and @ref{452,,dumps()}
functions.  (The older API is now deprecated.)  In addition to the already
supported XML plist format (@ref{afa,,FMT_XML}), it also now supports
the binary plist format (@ref{afb,,FMT_BINARY}).  (Contributed by Ronald
Oussoren and others in bpo-14455@footnote{https://bugs.python.org/issue14455}.)

@node poplib<3>,pprint<4>,plistlib<2>,Improved Modules<8>
@anchor{whatsnew/3 4 poplib}@anchor{afc}
@subsubsection poplib


Two new methods have been added to @ref{cb,,poplib}: @ref{afd,,capa()},
which returns the list of capabilities advertised by the POP server, and
@ref{afe,,stls()}, which switches a clear-text POP3 session into an
encrypted POP3 session if the POP server supports it.  (Contributed by Lorenzo
Catucci in bpo-4473@footnote{https://bugs.python.org/issue4473}.)

@node pprint<4>,pty,poplib<3>,Improved Modules<8>
@anchor{whatsnew/3 4 pprint}@anchor{aff}
@subsubsection pprint


The @ref{cd,,pprint} module’s @ref{b00,,PrettyPrinter} class and its
@ref{b01,,pformat()}, and @ref{2ac,,pprint()} functions have a new
option, @emph{compact}, that controls how the output is formatted.  Currently
setting @emph{compact} to @code{True} means that sequences will be printed with as many
sequence elements as will fit within @emph{width} on each (indented) line.
(Contributed by Serhiy Storchaka in bpo-19132@footnote{https://bugs.python.org/issue19132}.)

Long strings are now wrapped using Python’s normal line continuation
syntax.  (Contributed by Antoine Pitrou in bpo-17150@footnote{https://bugs.python.org/issue17150}.)

@node pty,pydoc<4>,pprint<4>,Improved Modules<8>
@anchor{whatsnew/3 4 pty}@anchor{b02}
@subsubsection pty


@ref{b03,,pty.spawn()} now returns the status value from @ref{8f6,,os.waitpid()} on
the child process, instead of @code{None}.  (Contributed by Gregory P. Smith.)

@node pydoc<4>,re<5>,pty,Improved Modules<8>
@anchor{whatsnew/3 4 pydoc}@anchor{b04}
@subsubsection pydoc


The @ref{d4,,pydoc} module is now based directly on the @ref{1c1,,inspect.signature()}
introspection API, allowing it to provide signature information for a wider
variety of callable objects. This change also means that @code{__wrapped__}
attributes are now taken into account when displaying help information.
(Contributed by Larry Hastings in bpo-19674@footnote{https://bugs.python.org/issue19674}.)

The @ref{d4,,pydoc} module no longer displays the @code{self} parameter for
already bound methods. Instead, it aims to always display the exact current
signature of the supplied callable.  (Contributed by Larry Hastings in
bpo-20710@footnote{https://bugs.python.org/issue20710}.)

In addition to the changes that have been made to @ref{d4,,pydoc} directly,
its handling of custom @code{__dir__} methods and various descriptor
behaviours has also been improved substantially by the underlying changes in
the @ref{9c,,inspect} module.

As the @ref{3bc,,help()} builtin is based on @ref{d4,,pydoc}, the above changes also
affect the behaviour of @ref{3bc,,help()}.

@node re<5>,resource,pydoc<4>,Improved Modules<8>
@anchor{whatsnew/3 4 re}@anchor{b05}
@subsubsection re


New @ref{b06,,fullmatch()} function and @code{regex.fullmatch()} method anchor
the pattern at both ends of the string to match.  This provides a way to be
explicit about the goal of the match, which avoids a class of subtle bugs where
@code{$} characters get lost during code changes or the addition of alternatives
to an existing regular expression.  (Contributed by Matthew Barnett in
bpo-16203@footnote{https://bugs.python.org/issue16203}.)

The repr of @ref{b07,,regex objects} now includes the pattern
and the flags; the repr of @ref{b08,,match objects} now
includes the start, end, and the part of the string that matched.  (Contributed
by Hugo Lopes Tavares and Serhiy Storchaka in bpo-13592@footnote{https://bugs.python.org/issue13592} and
bpo-17087@footnote{https://bugs.python.org/issue17087}.)

@node resource,select,re<5>,Improved Modules<8>
@anchor{whatsnew/3 4 resource}@anchor{b09}
@subsubsection resource


New @ref{b0a,,prlimit()} function, available on Linux platforms with a
kernel version of 2.6.36 or later and glibc of 2.13 or later, provides the
ability to query or set the resource limits for processes other than the one
making the call.  (Contributed by Christian Heimes in bpo-16595@footnote{https://bugs.python.org/issue16595}.)

On Linux kernel version 2.6.36 or later, there are also some new
Linux specific constants: @ref{b0b,,RLIMIT_MSGQUEUE},
@ref{b0c,,RLIMIT_NICE}, @ref{b0d,,RLIMIT_RTPRIO},
@ref{b0e,,RLIMIT_RTTIME}, and @ref{b0f,,RLIMIT_SIGPENDING}.
(Contributed by Christian Heimes in bpo-19324@footnote{https://bugs.python.org/issue19324}.)

On FreeBSD version 9 and later, there some new FreeBSD specific constants:
@ref{b10,,RLIMIT_SBSIZE}, @ref{b11,,RLIMIT_SWAP}, and
@ref{b12,,RLIMIT_NPTS}.  (Contributed by Claudiu Popa in
bpo-19343@footnote{https://bugs.python.org/issue19343}.)

@node select,shelve<2>,resource,Improved Modules<8>
@anchor{whatsnew/3 4 select}@anchor{b13}
@subsubsection select


@ref{b14,,epoll} objects now support the context management protocol.
When used in a @ref{168,,with} statement, the @ref{b15,,close()}
method will be called automatically at the end of the block.  (Contributed
by Serhiy Storchaka in bpo-16488@footnote{https://bugs.python.org/issue16488}.)

@ref{b16,,devpoll} objects now have @ref{b17,,fileno()} and
@ref{b18,,close()} methods, as well as a new attribute
@ref{b19,,closed}.  (Contributed by Victor Stinner in
bpo-18794@footnote{https://bugs.python.org/issue18794}.)

@node shelve<2>,shutil<3>,select,Improved Modules<8>
@anchor{whatsnew/3 4 shelve}@anchor{b1a}
@subsubsection shelve


@ref{b1b,,Shelf} instances may now be used in @ref{168,,with} statements,
and will be automatically closed at the end of the @code{with} block.
(Contributed by Filip Gruszczyński in bpo-13896@footnote{https://bugs.python.org/issue13896}.)

@node shutil<3>,smtpd<2>,shelve<2>,Improved Modules<8>
@anchor{whatsnew/3 4 shutil}@anchor{b1c}
@subsubsection shutil


@ref{56f,,copyfile()} now raises a specific @ref{b1d,,Error} subclass,
@ref{b1e,,SameFileError}, when the source and destination are the same
file, which allows an application to take appropriate action on this specific
error.  (Contributed by Atsuo Ishimoto and Hynek Schlawack in
bpo-1492704@footnote{https://bugs.python.org/issue1492704}.)

@node smtpd<2>,smtplib<3>,shutil<3>,Improved Modules<8>
@anchor{whatsnew/3 4 smtpd}@anchor{b1f}
@subsubsection smtpd


The @ref{898,,SMTPServer} and @ref{897,,SMTPChannel} classes now
accept a @emph{map} keyword argument which, if specified, is passed in to
@ref{b20,,asynchat.async_chat} as its @emph{map} argument.  This allows an application
to avoid affecting the global socket map.  (Contributed by Vinay Sajip in
bpo-11959@footnote{https://bugs.python.org/issue11959}.)

@node smtplib<3>,socket<9>,smtpd<2>,Improved Modules<8>
@anchor{whatsnew/3 4 smtplib}@anchor{b21}
@subsubsection smtplib


@ref{b22,,SMTPException} is now a subclass of @ref{2a1,,OSError}, which allows
both socket level errors and SMTP protocol level errors to be caught in one
try/except statement by code that only cares whether or not an error occurred.
(Contributed by Ned Jackson Lovely in bpo-2118@footnote{https://bugs.python.org/issue2118}.)

@node socket<9>,sqlite3<6>,smtplib<3>,Improved Modules<8>
@anchor{whatsnew/3 4 socket}@anchor{b23}
@subsubsection socket


The socket module now supports the @ref{b24,,CAN_BCM} protocol on
platforms that support it.  (Contributed by Brian Thorne in bpo-15359@footnote{https://bugs.python.org/issue15359}.)

Socket objects have new methods to get or set their @ref{a6e,,inheritable flag}, @ref{a73,,get_inheritable()} and
@ref{a74,,set_inheritable()}.

The @code{socket.AF_*} and @code{socket.SOCK_*} constants are now enumeration values
using the new @ref{76,,enum} module.  This allows meaningful names to be printed
during debugging, instead of integer “magic numbers”.

The @ref{b25,,AF_LINK} constant is now available on BSD and OSX.

@ref{465,,inet_pton()} and @ref{b26,,inet_ntop()} are now supported
on Windows.  (Contributed by Atsuo Ishimoto in bpo-7171@footnote{https://bugs.python.org/issue7171}.)

@node sqlite3<6>,ssl<8>,socket<9>,Improved Modules<8>
@anchor{whatsnew/3 4 sqlite3}@anchor{b27}
@subsubsection sqlite3


A new boolean parameter to the @ref{6a8,,connect()} function, @emph{uri}, can be
used to indicate that the @emph{database} parameter is a @code{uri} (see the SQLite URI documentation@footnote{https://www.sqlite.org/uri.html}).  (Contributed by poq in
bpo-13773@footnote{https://bugs.python.org/issue13773}.)

@node ssl<8>,stat,sqlite3<6>,Improved Modules<8>
@anchor{whatsnew/3 4 ssl}@anchor{b28}
@subsubsection ssl

@anchor{whatsnew/3 4 whatsnew-tls-11-12}@anchor{a5b}
@ref{32c,,PROTOCOL_TLSv1_1} and @ref{32d,,PROTOCOL_TLSv1_2} (TLSv1.1 and
TLSv1.2 support) have been added; support for these protocols is only available if
Python is linked with OpenSSL 1.0.1 or later.  (Contributed by Michele Orrù and
Antoine Pitrou in bpo-16692@footnote{https://bugs.python.org/issue16692}.)
@anchor{whatsnew/3 4 whatsnew34-sslcontext}@anchor{a5e}
New function @ref{b29,,create_default_context()} provides a standard way to
obtain an @ref{2bf,,SSLContext} whose settings are intended to be a
reasonable balance between compatibility and security.  These settings are
more stringent than the defaults provided by the @ref{2bf,,SSLContext}
constructor, and may be adjusted in the future, without prior deprecation, if
best-practice security requirements change.  The new recommended best
practice for using stdlib libraries that support SSL is to use
@ref{b29,,create_default_context()} to obtain an @ref{2bf,,SSLContext}
object, modify it if needed, and then pass it as the @emph{context} argument
of the appropriate stdlib API.  (Contributed by Christian Heimes
in bpo-19689@footnote{https://bugs.python.org/issue19689}.)

@ref{2bf,,SSLContext} method @ref{a5f,,load_verify_locations()}
accepts a new optional argument @emph{cadata}, which can be used to provide PEM or
DER encoded certificates directly via strings or bytes, respectively.
(Contributed by Christian Heimes in bpo-18138@footnote{https://bugs.python.org/issue18138}.)

New function @ref{b2a,,get_default_verify_paths()} returns
a named tuple of the paths and environment variables that the
@ref{b2b,,set_default_verify_paths()} method uses to set
OpenSSL’s default @code{cafile} and @code{capath}.  This can be an aid in
debugging default verification issues.  (Contributed by Christian Heimes
in bpo-18143@footnote{https://bugs.python.org/issue18143}.)

@ref{2bf,,SSLContext} has a new method,
@ref{b2c,,cert_store_stats()}, that reports the number of loaded
@code{X.509} certs, @code{X.509 CA} certs, and certificate revocation lists
(@code{crl}s), as well as a @ref{b2d,,get_ca_certs()} method that
returns a list of the loaded @code{CA} certificates.  (Contributed by Christian
Heimes in bpo-18147@footnote{https://bugs.python.org/issue18147}.)

If OpenSSL 0.9.8 or later is available, @ref{2bf,,SSLContext} has a new
attribute @ref{b2e,,verify_flags} that can be used to control the
certificate verification process by setting it to some combination of the new
constants @ref{b2f,,VERIFY_DEFAULT}, @ref{b30,,VERIFY_CRL_CHECK_LEAF},
@ref{b31,,VERIFY_CRL_CHECK_CHAIN}, or @ref{b32,,VERIFY_X509_STRICT}.
OpenSSL does not do any CRL verification by default.  (Contributed by
Christien Heimes in bpo-8813@footnote{https://bugs.python.org/issue8813}.)

New @ref{2bf,,SSLContext} method @ref{b33,,load_default_certs()}
loads a set of default “certificate authority” (CA) certificates from default
locations, which vary according to the platform.  It can be used to load both
TLS web server authentication certificates
(@code{purpose=}@ref{b34,,SERVER_AUTH}) for a client to use to verify a
server, and certificates for a server to use in verifying client certificates
(@code{purpose=}@ref{b35,,CLIENT_AUTH}).  (Contributed by Christian
Heimes in bpo-19292@footnote{https://bugs.python.org/issue19292}.)
@anchor{whatsnew/3 4 whatsnew34-win-cert-store}@anchor{a5c}
Two new windows-only functions, @ref{b36,,enum_certificates()} and
@ref{b37,,enum_crls()} provide the ability to retrieve certificates,
certificate information, and CRLs from the Windows cert store.  (Contributed
by Christian Heimes in bpo-17134@footnote{https://bugs.python.org/issue17134}.)
@anchor{whatsnew/3 4 whatsnew34-sni}@anchor{a5d}
Support for server-side SNI (Server Name Indication) using the new
@ref{b38,,ssl.SSLContext.set_servername_callback()} method.
(Contributed by Daniel Black in bpo-8109@footnote{https://bugs.python.org/issue8109}.)

The dictionary returned by @ref{b39,,SSLSocket.getpeercert()} contains additional
@code{X509v3} extension items: @code{crlDistributionPoints}, @code{calIssuers}, and
@code{OCSP} URIs.  (Contributed by Christian Heimes in bpo-18379@footnote{https://bugs.python.org/issue18379}.)

@node stat,struct<2>,ssl<8>,Improved Modules<8>
@anchor{whatsnew/3 4 stat}@anchor{b3a}
@subsubsection stat


The @ref{ef,,stat} module is now backed by a C implementation in @code{_stat}. A C
implementation is required as most of the values aren’t standardized and
are platform-dependent.  (Contributed by Christian Heimes in bpo-11016@footnote{https://bugs.python.org/issue11016}.)

The module supports new @ref{b3b,,ST_MODE} flags, @ref{b3c,,S_IFDOOR},
@ref{b3d,,S_IFPORT}, and @ref{b3e,,S_IFWHT}.  (Contributed by
Christian Hiemes in bpo-11016@footnote{https://bugs.python.org/issue11016}.)

@node struct<2>,subprocess<4>,stat,Improved Modules<8>
@anchor{whatsnew/3 4 struct}@anchor{b3f}
@subsubsection struct


New function @ref{b40,,iter_unpack} and a new
@ref{b41,,struct.Struct.iter_unpack()} method on compiled formats provide streamed
unpacking of a buffer containing repeated instances of a given format of data.
(Contributed by Antoine Pitrou in bpo-17804@footnote{https://bugs.python.org/issue17804}.)

@node subprocess<4>,sunau<2>,struct<2>,Improved Modules<8>
@anchor{whatsnew/3 4 subprocess}@anchor{b42}
@subsubsection subprocess


@ref{b43,,check_output()} now accepts an @emph{input} argument that can
be used to provide the contents of @code{stdin} for the command that is run.
(Contributed by Zack Weinberg in bpo-16624@footnote{https://bugs.python.org/issue16624}.)

@code{getstatus()} and @ref{b44,,getstatusoutput()} now
work on Windows.  This change was actually inadvertently made in 3.3.4.
(Contributed by Tim Golden in bpo-10197@footnote{https://bugs.python.org/issue10197}.)

@node sunau<2>,sys<9>,subprocess<4>,Improved Modules<8>
@anchor{whatsnew/3 4 sunau}@anchor{b45}
@subsubsection sunau


The @code{getparams()} method now returns a namedtuple rather than a
plain tuple.  (Contributed by Claudiu Popa in bpo-18901@footnote{https://bugs.python.org/issue18901}.)

@ref{727,,sunau.open()} now supports the context management protocol: when used in a
@ref{168,,with} block, the @code{close} method of the returned object will be
called automatically at the end of the block.  (Contributed by Serhiy Storchaka
in bpo-18878@footnote{https://bugs.python.org/issue18878}.)

@ref{b46,,AU_write.setsampwidth()} now supports 24 bit samples, thus adding
support for writing 24 sample using the module.  (Contributed by
Serhiy Storchaka in bpo-19261@footnote{https://bugs.python.org/issue19261}.)

The @ref{b47,,writeframesraw()} and
@ref{b48,,writeframes()} methods now accept any @ref{87e,,bytes-like object}.  (Contributed by Serhiy Storchaka in bpo-8311@footnote{https://bugs.python.org/issue8311}.)

@node sys<9>,tarfile<3>,sunau<2>,Improved Modules<8>
@anchor{whatsnew/3 4 sys}@anchor{b49}
@subsubsection sys


New function @ref{b4a,,sys.getallocatedblocks()} returns the current number of
blocks allocated by the interpreter.  (In CPython with the default
@code{--with-pymalloc} setting, this is allocations made through the
@ref{7af,,PyObject_Malloc()} API.)  This can be useful for tracking memory leaks,
especially if automated via a test suite.  (Contributed by Antoine Pitrou
in bpo-13390@footnote{https://bugs.python.org/issue13390}.)

When the Python interpreter starts in @ref{b4b,,interactive mode}, it checks for an @ref{b4c,,__interactivehook__} attribute
on the @ref{f7,,sys} module.  If the attribute exists, its value is called with no
arguments just before interactive mode is started.  The check is made after the
@geindex PYTHONSTARTUP
@geindex environment variable; PYTHONSTARTUP
@ref{b4d,,PYTHONSTARTUP} file is read, so it can be set there.  The @ref{e6,,site}
module @ref{b4e,,sets it} to a function that enables tab
completion and history saving (in @code{~/.python-history}) if the platform
supports @ref{d9,,readline}.  If you do not want this (new) behavior, you can
override it in 
@geindex PYTHONSTARTUP
@geindex environment variable; PYTHONSTARTUP
@ref{b4d,,PYTHONSTARTUP}, @code{sitecustomize}, or
@code{usercustomize} by deleting this attribute from @ref{f7,,sys} (or setting it
to some other callable).  (Contributed by Éric Araujo and Antoine Pitrou in
bpo-5845@footnote{https://bugs.python.org/issue5845}.)

@node tarfile<3>,textwrap,sys<9>,Improved Modules<8>
@anchor{whatsnew/3 4 tarfile}@anchor{b4f}
@subsubsection tarfile


The @ref{fb,,tarfile} module now supports a simple @ref{b50,,Command-Line Interface} when
called as a script directly or via @code{-m}.  This can be used to create and
extract tarfile archives.  (Contributed by Berker Peksag in bpo-13477@footnote{https://bugs.python.org/issue13477}.)

@node textwrap,threading<6>,tarfile<3>,Improved Modules<8>
@anchor{whatsnew/3 4 textwrap}@anchor{b51}
@subsubsection textwrap


The @ref{b52,,TextWrapper} class has two new attributes/constructor
arguments: @ref{b53,,max_lines}, which limits the number of
lines in the output, and @ref{b54,,placeholder}, which is a
string that will appear at the end of the output if it has been truncated
because of @emph{max_lines}.  Building on these capabilities, a new convenience
function @ref{b55,,shorten()} collapses all of the whitespace in the input
to single spaces and produces a single line of a given @emph{width} that ends with
the @emph{placeholder} (by default, @code{[...]}).  (Contributed by Antoine Pitrou and
Serhiy Storchaka in bpo-18585@footnote{https://bugs.python.org/issue18585} and bpo-18725@footnote{https://bugs.python.org/issue18725}.)

@node threading<6>,traceback<4>,textwrap,Improved Modules<8>
@anchor{whatsnew/3 4 threading}@anchor{b56}
@subsubsection threading


The @ref{44b,,Thread} object representing the main thread can be
obtained from the new @ref{b57,,main_thread()} function.  In normal
conditions this will be the thread from which the Python interpreter was
started.  (Contributed by Andrew Svetlov in bpo-18882@footnote{https://bugs.python.org/issue18882}.)

@node traceback<4>,types<4>,threading<6>,Improved Modules<8>
@anchor{whatsnew/3 4 traceback}@anchor{b58}
@subsubsection traceback


A new @ref{b59,,traceback.clear_frames()} function takes a traceback object
and clears the local variables in all of the frames it references,
reducing the amount of memory consumed.  (Contributed by Andrew Kuchling in
bpo-1565525@footnote{https://bugs.python.org/issue1565525}.)

@node types<4>,urllib<2>,traceback<4>,Improved Modules<8>
@anchor{whatsnew/3 4 types}@anchor{b5a}
@subsubsection types


A new @ref{b5b,,DynamicClassAttribute()} descriptor provides a way to define
an attribute that acts normally when looked up through an instance object, but
which is routed to the @emph{class} @code{__getattr__} when looked up through the
class.  This allows one to have properties active on a class, and have virtual
attributes on the class with the same name (see @code{Enum} for an example).
(Contributed by Ethan Furman in bpo-19030@footnote{https://bugs.python.org/issue19030}.)

@node urllib<2>,unittest<5>,types<4>,Improved Modules<8>
@anchor{whatsnew/3 4 urllib}@anchor{b5c}
@subsubsection urllib


@ref{127,,urllib.request} now supports @code{data:} URLs via the
@ref{b5d,,DataHandler} class.  (Contributed by Mathias Panzenböck
in bpo-16423@footnote{https://bugs.python.org/issue16423}.)

The http method that will be used by a @ref{b5e,,Request} class
can now be specified by setting a @ref{b5f,,method}
class attribute on the subclass.  (Contributed by Jason R Coombs in
bpo-18978@footnote{https://bugs.python.org/issue18978}.)

@ref{b5e,,Request} objects are now reusable: if the
@ref{b60,,full_url} or @ref{b61,,data}
attributes are modified, all relevant internal properties are updated.  This
means, for example, that it is now possible to use the same
@ref{b5e,,Request} object in more than one
@ref{b62,,OpenerDirector.open()} call with different @emph{data} arguments, or to
modify a @ref{b5e,,Request}‘s @code{url} rather than recomputing it
from scratch.  There is also a new
@ref{b63,,remove_header()} method that can be used to remove
headers from a @ref{b5e,,Request}.  (Contributed by Alexey
Kachayev in bpo-16464@footnote{https://bugs.python.org/issue16464}, Daniel Wozniak in bpo-17485@footnote{https://bugs.python.org/issue17485}, and Damien Brecht
and Senthil Kumaran in bpo-17272@footnote{https://bugs.python.org/issue17272}.)

@ref{b64,,HTTPError} objects now have a
@ref{b65,,headers} attribute that provides access to the
HTTP response headers associated with the error.  (Contributed by
Berker Peksag in bpo-15701@footnote{https://bugs.python.org/issue15701}.)

@node unittest<5>,venv<5>,urllib<2>,Improved Modules<8>
@anchor{whatsnew/3 4 unittest}@anchor{b66}
@subsubsection unittest


The @ref{1ab,,TestCase} class has a new method,
@ref{b67,,subTest()}, that produces a context manager whose
@ref{168,,with} block becomes a “sub-test”.  This context manager allows a test
method to dynamically generate subtests  by, say, calling the @code{subTest}
context manager inside a loop.  A single test method can thereby produce an
indefinite number of separately-identified and separately-counted tests, all of
which will run even if one or more of them fail.  For example:

@example
class NumbersTest(unittest.TestCase):
    def test_even(self):
        for i in range(6):
            with self.subTest(i=i):
                self.assertEqual(i % 2, 0)
@end example

will result in six subtests, each identified in the unittest verbose output
with a label consisting of the variable name @code{i} and a particular value for
that variable (@code{i=0}, @code{i=1}, etc).  See @ref{b68,,Distinguishing test iterations using subtests} for the full
version of this example.  (Contributed by Antoine Pitrou in bpo-16997@footnote{https://bugs.python.org/issue16997}.)

@ref{b69,,unittest.main()} now accepts an iterable of test names for
@emph{defaultTest}, where previously it only accepted a single test name as a
string.  (Contributed by Jyrki Pulliainen in bpo-15132@footnote{https://bugs.python.org/issue15132}.)

If @ref{b6a,,SkipTest} is raised during test discovery (that is, at the
module level in the test file), it is now reported as a skip instead of an
error.  (Contributed by Zach Ware in bpo-16935@footnote{https://bugs.python.org/issue16935}.)

@ref{b6b,,discover()} now sorts the discovered files to provide
consistent test ordering.  (Contributed by Martin Melin and Jeff Ramnani in
bpo-16709@footnote{https://bugs.python.org/issue16709}.)

@ref{953,,TestSuite} now drops references to tests as soon as the test
has been run, if the test is successful.  On Python interpreters that do
garbage collection, this allows the tests to be garbage collected if nothing
else is holding a reference to the test.  It is possible to override this
behavior by creating a @ref{953,,TestSuite} subclass that defines a
custom @code{_removeTestAtIndex} method.  (Contributed by Tom Wardill, Matt
McClure, and Andrew Svetlov in bpo-11798@footnote{https://bugs.python.org/issue11798}.)

A new test assertion context-manager, @ref{2e1,,assertLogs()},
will ensure that a given block of code emits a log message using the
@ref{a6,,logging} module.  By default the message can come from any logger and
have a priority of @code{INFO} or higher, but both the logger name and an
alternative minimum logging level may be specified.  The object returned by the
context manager can be queried for the @ref{b6c,,LogRecord}s and/or
formatted messages that were logged.  (Contributed by Antoine Pitrou in
bpo-18937@footnote{https://bugs.python.org/issue18937}.)

Test discovery now works with namespace packages (Contributed by Claudiu Popa
in bpo-17457@footnote{https://bugs.python.org/issue17457}.)

@ref{123,,unittest.mock} objects now inspect their specification signatures when
matching calls, which means an argument can now be matched by either position
or name, instead of only by position.  (Contributed by Antoine Pitrou in
bpo-17015@footnote{https://bugs.python.org/issue17015}.)

@code{mock_open()} objects now have @code{readline} and @code{readlines}
methods.  (Contributed by Toshio Kuratomi in bpo-17467@footnote{https://bugs.python.org/issue17467}.)

@node venv<5>,wave<2>,unittest<5>,Improved Modules<8>
@anchor{whatsnew/3 4 venv}@anchor{b6d}
@subsubsection venv


@ref{12c,,venv} now includes activation scripts for the @code{csh} and @code{fish}
shells.  (Contributed by Andrew Svetlov in bpo-15417@footnote{https://bugs.python.org/issue15417}.)

@ref{b6e,,EnvBuilder} and the @ref{b6f,,create()} convenience function
take a new keyword argument @emph{with_pip}, which defaults to @code{False}, that
controls whether or not @ref{b6e,,EnvBuilder} ensures that @code{pip} is
installed in the virtual environment.  (Contributed by Nick Coghlan in
bpo-19552@footnote{https://bugs.python.org/issue19552} as part of the 
@geindex Python Enhancement Proposals; PEP 453
PEP 453@footnote{https://www.python.org/dev/peps/pep-0453} implementation.)

@node wave<2>,weakref<2>,venv<5>,Improved Modules<8>
@anchor{whatsnew/3 4 wave}@anchor{b70}
@subsubsection wave


The @code{getparams()} method now returns a namedtuple rather than a
plain tuple.  (Contributed by Claudiu Popa in bpo-17487@footnote{https://bugs.python.org/issue17487}.)

@ref{44a,,wave.open()} now supports the context management protocol.  (Contributed
by Claudiu Popa in bpo-17616@footnote{https://bugs.python.org/issue17616}.)

@ref{12e,,wave} can now @ref{b71,,write output to unseekable files}.  (Contributed by David Jones, Guilherme Polo, and Serhiy
Storchaka in bpo-5202@footnote{https://bugs.python.org/issue5202}.)

The @ref{b72,,writeframesraw()} and
@ref{b73,,writeframes()} methods now accept any @ref{87e,,bytes-like object}.  (Contributed by Serhiy Storchaka in bpo-8311@footnote{https://bugs.python.org/issue8311}.)

@node weakref<2>,xml etree<2>,wave<2>,Improved Modules<8>
@anchor{whatsnew/3 4 weakref}@anchor{b74}
@subsubsection weakref


New @ref{b75,,WeakMethod} class simulates weak references to bound
methods.  (Contributed by Antoine Pitrou in bpo-14631@footnote{https://bugs.python.org/issue14631}.)

New @ref{594,,finalize} class makes it possible to register a callback
to be invoked when an object is garbage collected, without needing to
carefully manage the lifecycle of the weak reference itself.  (Contributed by
Richard Oudkerk in bpo-15528@footnote{https://bugs.python.org/issue15528}.)

The callback, if any, associated with a @ref{b76,,ref} is now
exposed via the @ref{b77,,__callback__} attribute.  (Contributed
by Mark Dickinson in bpo-17643@footnote{https://bugs.python.org/issue17643}.)

@node xml etree<2>,zipfile<4>,weakref<2>,Improved Modules<8>
@anchor{whatsnew/3 4 xml-etree}@anchor{b78}
@subsubsection xml.etree


A new parser, @ref{b79,,XMLPullParser}, allows a
non-blocking applications to parse XML documents.  An example can be
seen at @ref{b7a,,Pull API for non-blocking parsing}.  (Contributed by Antoine
Pitrou in bpo-17741@footnote{https://bugs.python.org/issue17741}.)

The @ref{13e,,xml.etree.ElementTree} @ref{b7b,,tostring()} and
@ref{b7c,,tostringlist()} functions, and the
@ref{44d,,ElementTree}
@ref{b7d,,write()} method, now have a
@emph{short_empty_elements} @ref{5a2,,keyword-only parameter}
providing control over whether elements with no content are written in
abbreviated (@code{<tag />}) or expanded (@code{<tag></tag>}) form.  (Contributed by
Ariel Poliak and Serhiy Storchaka in bpo-14377@footnote{https://bugs.python.org/issue14377}.)

@node zipfile<4>,,xml etree<2>,Improved Modules<8>
@anchor{whatsnew/3 4 zipfile}@anchor{b7e}
@subsubsection zipfile


The @ref{b7f,,writepy()} method of the
@ref{b80,,PyZipFile} class has a new @emph{filterfunc} option that can be
used to control which directories and files are added to the archive.  For
example, this could be used to exclude test files from the archive.
(Contributed by Christian Tismer in bpo-19274@footnote{https://bugs.python.org/issue19274}.)

The @emph{allowZip64} parameter to @ref{6e5,,ZipFile} and
@code{PyZipfile} is now @code{True} by default.  (Contributed by
William Mallard in bpo-17201@footnote{https://bugs.python.org/issue17201}.)

@node CPython Implementation Changes,Deprecated<9>,Improved Modules<8>,What’s New In Python 3 4
@anchor{whatsnew/3 4 cpython-implementation-changes}@anchor{b81}
@subsection CPython Implementation Changes


@menu
* PEP 445; Customization of CPython Memory Allocators: PEP 445 Customization of CPython Memory Allocators. 
* PEP 442; Safe Object Finalization: PEP 442 Safe Object Finalization. 
* PEP 456; Secure and Interchangeable Hash Algorithm: PEP 456 Secure and Interchangeable Hash Algorithm. 
* PEP 436; Argument Clinic: PEP 436 Argument Clinic. 
* Other Build and C API Changes:: 
* Other Improvements: Other Improvements<2>. 
* Significant Optimizations:: 

@end menu

@node PEP 445 Customization of CPython Memory Allocators,PEP 442 Safe Object Finalization,,CPython Implementation Changes
@anchor{whatsnew/3 4 pep-445-customization-of-cpython-memory-allocators}@anchor{b82}@anchor{whatsnew/3 4 whatsnew-pep-445}@anchor{a61}
@subsubsection PEP 445: Customization of CPython Memory Allocators


@geindex Python Enhancement Proposals; PEP 445
PEP 445@footnote{https://www.python.org/dev/peps/pep-0445} adds new C level interfaces to customize memory allocation in
the CPython interpreter.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 445

@item PEP 445@footnote{https://www.python.org/dev/peps/pep-0445} – Add new APIs to customize Python memory allocators

PEP written and implemented by Victor Stinner.
@end table


@node PEP 442 Safe Object Finalization,PEP 456 Secure and Interchangeable Hash Algorithm,PEP 445 Customization of CPython Memory Allocators,CPython Implementation Changes
@anchor{whatsnew/3 4 pep-442-safe-object-finalization}@anchor{b83}@anchor{whatsnew/3 4 whatsnew-pep-442}@anchor{a60}
@subsubsection PEP 442: Safe Object Finalization


@geindex Python Enhancement Proposals; PEP 442
PEP 442@footnote{https://www.python.org/dev/peps/pep-0442} removes the current limitations and quirks of object finalization
in CPython. With it, objects with @code{__del__()} methods, as well as
generators with @ref{4c1,,finally} clauses, can be finalized when they are
part of a reference cycle.

As part of this change, module globals are no longer forcibly set to
@ref{182,,None} during interpreter shutdown in most cases, instead relying
on the normal operation of the cyclic garbage collector.  This avoids a
whole class of interpreter-shutdown-time errors, usually involving
@code{__del__} methods, that have plagued Python since the cyclic GC
was first introduced.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 442

@item PEP 442@footnote{https://www.python.org/dev/peps/pep-0442} – Safe object finalization

PEP written and implemented by Antoine Pitrou.
@end table


@node PEP 456 Secure and Interchangeable Hash Algorithm,PEP 436 Argument Clinic,PEP 442 Safe Object Finalization,CPython Implementation Changes
@anchor{whatsnew/3 4 pep-456-secure-and-interchangeable-hash-algorithm}@anchor{b84}@anchor{whatsnew/3 4 whatsnew-pep-456}@anchor{a5a}
@subsubsection PEP 456: Secure and Interchangeable Hash Algorithm


@geindex Python Enhancement Proposals; PEP 456
PEP 456@footnote{https://www.python.org/dev/peps/pep-0456} follows up on earlier security fix work done on Python’s hash
algorithm to address certain DOS attacks to which public facing APIs backed by
dictionary lookups may be subject.  (See bpo-14621@footnote{https://bugs.python.org/issue14621} for the start of the
current round of improvements.)  The PEP unifies CPython’s hash code to make it
easier for a packager to substitute a different hash algorithm, and switches
Python’s default implementation to a SipHash implementation on platforms that
have a 64 bit data type.  Any performance differences in comparison with the
older FNV algorithm are trivial.

The PEP adds additional fields to the @ref{b85,,sys.hash_info} named tuple to
describe the hash algorithm in use by the currently executing binary.  Otherwise,
the PEP does not alter any existing CPython APIs.

@node PEP 436 Argument Clinic,Other Build and C API Changes,PEP 456 Secure and Interchangeable Hash Algorithm,CPython Implementation Changes
@anchor{whatsnew/3 4 pep-436-argument-clinic}@anchor{b86}@anchor{whatsnew/3 4 whatsnew-pep-436}@anchor{a62}
@subsubsection PEP 436: Argument Clinic


“Argument Clinic” (
@geindex Python Enhancement Proposals; PEP 436
PEP 436@footnote{https://www.python.org/dev/peps/pep-0436}) is now part of the CPython build process
and can be used to simplify the process of defining and maintaining
accurate signatures for builtins and standard library extension modules
implemented in C.

Some standard library extension modules have been converted to use Argument
Clinic in Python 3.4, and @ref{d4,,pydoc} and @ref{9c,,inspect} have been updated
accordingly.

It is expected that signature metadata for programmatic introspection will
be added to additional callables implemented in C as part of Python 3.4
maintenance releases.

@cartouche
@quotation Note 
The Argument Clinic PEP is not fully up to date with the state of the
implementation. This has been deemed acceptable by the release manager
and core development team in this case, as Argument Clinic will not
be made available as a public API for third party use in Python 3.4.
@end quotation
@end cartouche


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 436

@item PEP 436@footnote{https://www.python.org/dev/peps/pep-0436} – The Argument Clinic DSL

PEP written and implemented by Larry Hastings.
@end table


@node Other Build and C API Changes,Other Improvements<2>,PEP 436 Argument Clinic,CPython Implementation Changes
@anchor{whatsnew/3 4 other-build-and-c-api-changes}@anchor{b87}
@subsubsection Other Build and C API Changes



@itemize *

@item 
The new @ref{36e,,PyType_GetSlot()} function has been added to the stable ABI,
allowing retrieval of function pointers from named type slots when using
the limited API.  (Contributed by Martin von Löwis in bpo-17162@footnote{https://bugs.python.org/issue17162}.)

@item 
The new @ref{205,,Py_SetStandardStreamEncoding()} pre-initialization API
allows applications embedding the CPython interpreter to reliably force
a particular encoding and error handler for the standard streams.
(Contributed by Bastien Montagne and Nick Coghlan in bpo-16129@footnote{https://bugs.python.org/issue16129}.)

@item 
Most Python C APIs that don’t mutate string arguments are now correctly
marked as accepting @code{const char *} rather than @code{char *}.  (Contributed
by Serhiy Storchaka in bpo-1772673@footnote{https://bugs.python.org/issue1772673}.)

@item 
A new shell version of @code{python-config} can be used even when a python
interpreter is not available (for example, in cross compilation scenarios).

@item 
@ref{398,,PyUnicode_FromFormat()} now supports width and precision
specifications for @code{%s}, @code{%A}, @code{%U}, @code{%V}, @code{%S}, and @code{%R}.
(Contributed by Ysj Ray and Victor Stinner in bpo-7330@footnote{https://bugs.python.org/issue7330}.)

@item 
New function @ref{b88,,PyStructSequence_InitType2()} supplements the
existing @ref{b89,,PyStructSequence_InitType()} function.  The difference
is that it returns @code{0} on success and @code{-1} on failure.

@item 
The CPython source can now be compiled using the address sanity checking
features of recent versions of GCC and clang:  the false alarms in the small
object allocator have been silenced.  (Contributed by Dhiru Kholia in
bpo-18596@footnote{https://bugs.python.org/issue18596}.)

@item 
The Windows build now uses Address Space Layout Randomization@footnote{https://en.wikipedia.org/wiki/Address_space_layout_randomization} and Data Execution Prevention@footnote{https://en.wikipedia.org/wiki/Data_Execution_Prevention}.  (Contributed by
Christian Heimes in bpo-16632@footnote{https://bugs.python.org/issue16632}.)

@item 
New function @ref{b8a,,PyObject_LengthHint()} is the C API equivalent
of @ref{aee,,operator.length_hint()}.  (Contributed by Armin Ronacher in
bpo-16148@footnote{https://bugs.python.org/issue16148}.)
@end itemize

@node Other Improvements<2>,Significant Optimizations,Other Build and C API Changes,CPython Implementation Changes
@anchor{whatsnew/3 4 other-improvements}@anchor{b8b}@anchor{whatsnew/3 4 other-improvements-3-4}@anchor{b8c}
@subsubsection Other Improvements

@anchor{whatsnew/3 4 whatsnew-isolated-mode}@anchor{a4b}

@itemize *

@item 
The @ref{b8d,,python} command has a new @ref{b8e,,option}, @code{-I}, which causes it to run in “isolated mode”,
which means that @ref{489,,sys.path} contains neither the script’s directory nor
the user’s @code{site-packages} directory, and all 
@geindex PYTHON*
@geindex environment variable; PYTHON*
@code{PYTHON*} environment
variables are ignored (it implies both @code{-s} and @code{-E}).  Other
restrictions may also be applied in the future, with the goal being to
isolate the execution of a script from the user’s environment.  This is
appropriate, for example, when Python is used to run a system script.  On
most POSIX systems it can and should be used in the @code{#!} line of system
scripts.  (Contributed by Christian Heimes in bpo-16499@footnote{https://bugs.python.org/issue16499}.)

@item 
Tab-completion is now enabled by default in the interactive interpreter
on systems that support @ref{d9,,readline}.  History is also enabled by default,
and is written to (and read from) the file @code{~/.python-history}.
(Contributed by Antoine Pitrou and Éric Araujo in bpo-5845@footnote{https://bugs.python.org/issue5845}.)

@item 
Invoking the Python interpreter with @code{--version} now outputs the version to
standard output instead of standard error (bpo-18338@footnote{https://bugs.python.org/issue18338}). Similar changes
were made to @ref{5,,argparse} (bpo-18920@footnote{https://bugs.python.org/issue18920}) and other modules that have
script-like invocation capabilities (bpo-18922@footnote{https://bugs.python.org/issue18922}).

@item 
The CPython Windows installer now adds @code{.py} to the 
@geindex PATHEXT
@geindex environment variable; PATHEXT
@code{PATHEXT}
variable when extensions are registered, allowing users to run a python
script at the windows command prompt by just typing its name without the
@code{.py} extension.  (Contributed by Paul Moore in bpo-18569@footnote{https://bugs.python.org/issue18569}.)

@item 
A new @code{make} target coverage-report@footnote{https://devguide.python.org/coverage/#measuring-coverage-of-c-code-with-gcov-and-lcov}
will build python, run the test suite, and generate an HTML coverage report
for the C codebase using @code{gcov} and lcov@footnote{http://ltp.sourceforge.net/coverage/lcov.php}.

@item 
The @code{-R} option to the @ref{b8f,,python regression test suite} now
also checks for memory allocation leaks, using
@ref{b4a,,sys.getallocatedblocks()}.  (Contributed by Antoine Pitrou in
bpo-13390@footnote{https://bugs.python.org/issue13390}.)

@item 
@code{python -m} now works with namespace packages.

@item 
The @ref{ef,,stat} module is now implemented in C, which means it gets the
values for its constants from the C header files, instead of having the
values hard-coded in the python module as was previously the case.

@item 
Loading multiple python modules from a single OS module (@code{.so}, @code{.dll})
now works correctly (previously it silently returned the first python
module in the file).  (Contributed by Václav Šmilauer in bpo-16421@footnote{https://bugs.python.org/issue16421}.)

@item 
A new opcode, @ref{b90,,LOAD_CLASSDEREF}, has been added to fix a bug in the
loading of free variables in class bodies that could be triggered by certain
uses of @ref{7a5,,__prepare__}.  (Contributed by Benjamin Peterson in
bpo-17853@footnote{https://bugs.python.org/issue17853}.)

@item 
A number of MemoryError-related crashes were identified and fixed by Victor
Stinner using his 
@geindex Python Enhancement Proposals; PEP 445
PEP 445@footnote{https://www.python.org/dev/peps/pep-0445}-based @code{pyfailmalloc} tool (bpo-18408@footnote{https://bugs.python.org/issue18408},
bpo-18520@footnote{https://bugs.python.org/issue18520}).

@item 
The @code{pyvenv} command now accepts a @code{--copies} option
to use copies rather than symlinks even on systems where symlinks are the
default.  (Contributed by Vinay Sajip in bpo-18807@footnote{https://bugs.python.org/issue18807}.)

@item 
The @code{pyvenv} command also accepts a @code{--without-pip}
option to suppress the otherwise-automatic bootstrapping of pip into
the virtual environment.  (Contributed by Nick Coghlan in bpo-19552@footnote{https://bugs.python.org/issue19552}
as part of the 
@geindex Python Enhancement Proposals; PEP 453
PEP 453@footnote{https://www.python.org/dev/peps/pep-0453} implementation.)

@item 
The encoding name is now optional in the value set for the
@geindex PYTHONIOENCODING
@geindex environment variable; PYTHONIOENCODING
@ref{b91,,PYTHONIOENCODING} environment variable.  This makes it possible to
set just the error handler, without changing the default encoding.
(Contributed by Serhiy Storchaka in bpo-18818@footnote{https://bugs.python.org/issue18818}.)

@item 
The @ref{12,,bz2}, @ref{a9,,lzma}, and @ref{87,,gzip} module @code{open} functions now
support @code{x} (exclusive creation) mode.  (Contributed by Tim Heaney and
Vajrasky Kok in bpo-19201@footnote{https://bugs.python.org/issue19201}, bpo-19222@footnote{https://bugs.python.org/issue19222}, and bpo-19223@footnote{https://bugs.python.org/issue19223}.)
@end itemize

@node Significant Optimizations,,Other Improvements<2>,CPython Implementation Changes
@anchor{whatsnew/3 4 significant-optimizations}@anchor{b92}
@subsubsection Significant Optimizations



@itemize *

@item 
The UTF-32 decoder is now 3x to 4x faster.  (Contributed by Serhiy Storchaka
in bpo-14625@footnote{https://bugs.python.org/issue14625}.)

@item 
The cost of hash collisions for sets is now reduced.  Each hash table
probe now checks a series of consecutive, adjacent key/hash pairs before
continuing to make random probes through the hash table.  This exploits
cache locality to make collision resolution less expensive.
The collision resolution scheme can be described as a hybrid of linear
probing and open addressing.  The number of additional linear probes
defaults to nine.  This can be changed at compile-time by defining
LINEAR_PROBES to be any value.  Set LINEAR_PROBES=0 to turn-off
linear probing entirely.  (Contributed by Raymond Hettinger in
bpo-18771@footnote{https://bugs.python.org/issue18771}.)

@item 
The interpreter starts about 30% faster. A couple of measures lead to the
speedup. The interpreter loads fewer modules on startup, e.g. the @ref{d8,,re},
@ref{1c,,collections} and @ref{a5,,locale} modules and their dependencies are no
longer imported by default. The marshal module has been improved to load
compiled Python code faster.  (Contributed by Antoine Pitrou, Christian
Heimes and Victor Stinner in bpo-19219@footnote{https://bugs.python.org/issue19219}, bpo-19218@footnote{https://bugs.python.org/issue19218}, bpo-19209@footnote{https://bugs.python.org/issue19209},
bpo-19205@footnote{https://bugs.python.org/issue19205} and bpo-9548@footnote{https://bugs.python.org/issue9548}.)

@item 
@ref{455,,bz2.BZ2File} is now as fast or faster than the Python2 version for
most cases.  @ref{b93,,lzma.LZMAFile} has also been optimized.  (Contributed by
Serhiy Storchaka and Nadeem Vawda in bpo-16034@footnote{https://bugs.python.org/issue16034}.)

@item 
@ref{b94,,random.getrandbits()} is 20%-40% faster for small integers (the most
common use case).  (Contributed by Serhiy Storchaka in bpo-16674@footnote{https://bugs.python.org/issue16674}.)

@item 
By taking advantage of the new storage format for strings, pickling of
strings is now significantly faster.  (Contributed by Victor Stinner and
Antoine Pitrou in bpo-15596@footnote{https://bugs.python.org/issue15596}.)

@item 
A performance issue in @code{io.FileIO.readall()} has been solved.  This
particularly affects Windows, and significantly speeds up the case of piping
significant amounts of data through @ref{f4,,subprocess}.  (Contributed
by Richard Oudkerk in bpo-15758@footnote{https://bugs.python.org/issue15758}.)

@item 
@ref{b95,,html.escape()} is now 10x faster.  (Contributed by Matt Bryant in
bpo-18020@footnote{https://bugs.python.org/issue18020}.)

@item 
On Windows, the native @code{VirtualAlloc} is now used instead of the CRT
@code{malloc} in @code{obmalloc}.  Artificial benchmarks show about a 3% memory
savings.

@item 
@ref{17f,,os.urandom()} now uses a lazily-opened persistent file descriptor
so as to avoid using many file descriptors when run in parallel from
multiple threads.  (Contributed by Antoine Pitrou in bpo-18756@footnote{https://bugs.python.org/issue18756}.)
@end itemize

@node Deprecated<9>,Removed<9>,CPython Implementation Changes,What’s New In Python 3 4
@anchor{whatsnew/3 4 deprecated}@anchor{b96}@anchor{whatsnew/3 4 deprecated-3-4}@anchor{a7b}
@subsection Deprecated


This section covers various APIs and other features that have been deprecated
in Python 3.4, and will be removed in Python 3.5 or later.  In most (but not
all) cases, using the deprecated APIs will produce a @ref{1b2,,DeprecationWarning}
when the interpreter is run with deprecation warnings enabled (for example, by
using @code{-Wd}).

@menu
* Deprecations in the Python API:: 
* Deprecated Features:: 

@end menu

@node Deprecations in the Python API,Deprecated Features,,Deprecated<9>
@anchor{whatsnew/3 4 deprecations-in-the-python-api}@anchor{b97}
@subsubsection Deprecations in the Python API



@itemize *

@item 
As mentioned in @ref{a4d,,PEP 451; A ModuleSpec Type for the Import System}, a number of @ref{96,,importlib}
methods and functions are deprecated: @ref{310,,importlib.find_loader()} is
replaced by @ref{b98,,importlib.util.find_spec()};
@ref{30b,,importlib.machinery.PathFinder.find_module()} is replaced by
@ref{b99,,importlib.machinery.PathFinder.find_spec()};
@ref{306,,importlib.abc.MetaPathFinder.find_module()} is replaced by
@ref{308,,importlib.abc.MetaPathFinder.find_spec()};
@ref{2ff,,importlib.abc.PathEntryFinder.find_loader()} and
@ref{307,,find_module()} are replaced by
@ref{309,,importlib.abc.PathEntryFinder.find_spec()}; all of the @code{xxxLoader} ABC
@code{load_module} methods (@ref{301,,importlib.abc.Loader.load_module()},
@ref{b9a,,importlib.abc.InspectLoader.load_module()},
@ref{b9b,,importlib.abc.FileLoader.load_module()},
@ref{b9c,,importlib.abc.SourceLoader.load_module()}) should no longer be
implemented, instead loaders should implement an
@code{exec_module} method
(@ref{302,,importlib.abc.Loader.exec_module()},
@ref{b9d,,importlib.abc.InspectLoader.exec_module()}
@ref{b9e,,importlib.abc.SourceLoader.exec_module()}) and let the import system
take care of the rest; and
@ref{303,,importlib.abc.Loader.module_repr()},
@ref{311,,importlib.util.module_for_loader()}, @ref{b9f,,importlib.util.set_loader()},
and @ref{ba0,,importlib.util.set_package()} are no longer needed because their
functions are now handled automatically by the import system.

@item 
The @ref{95,,imp} module is pending deprecation. To keep compatibility with
Python 2/3 code bases, the module’s removal is currently not scheduled.

@item 
The @code{formatter} module is pending deprecation and is slated for removal
in Python 3.6.

@item 
@code{MD5} as the default @emph{digestmod} for the @ref{ac7,,hmac.new()} function is
deprecated.  Python 3.6 will require an explicit digest name or constructor as
@emph{digestmod} argument.

@item 
The internal @code{Netrc} class in the @ref{7e,,ftplib} module has been documented
as deprecated in its docstring for quite some time.  It now emits a
@ref{1b2,,DeprecationWarning} and will be removed completely in Python 3.5.

@item 
The undocumented @emph{endtime} argument to @ref{82d,,subprocess.Popen.wait()} should
not have been exposed and is hopefully not in use; it is deprecated and
will mostly likely be removed in Python 3.5.

@item 
The @emph{strict} argument of @ref{33c,,HTMLParser} is deprecated.

@item 
The @ref{ca,,plistlib} @code{readPlist()},
@code{writePlist()}, @code{readPlistFromBytes()}, and
@code{writePlistToBytes()} functions are deprecated in favor of the
corresponding new functions @ref{44f,,load()}, @ref{451,,dump()},
@ref{450,,loads()}, and @ref{452,,dumps()}.  @code{Data()}
is deprecated in favor of just using the @ref{2f0,,bytes} constructor.

@item 
The @ref{f8,,sysconfig} key @code{SO} is deprecated, it has been replaced by
@code{EXT_SUFFIX}.

@item 
The @code{U} mode accepted by various @code{open} functions is deprecated.
In Python3 it does not do anything useful, and should be replaced by
appropriate uses of @ref{889,,io.TextIOWrapper} (if needed) and its @emph{newline}
argument.

@item 
The @emph{parser} argument of @ref{ba1,,xml.etree.ElementTree.iterparse()} has
been deprecated, as has the @emph{html} argument of
@ref{569,,XMLParser()}.  To prepare for the removal of the
latter, all arguments to @code{XMLParser} should be passed by keyword.
@end itemize

@node Deprecated Features,,Deprecations in the Python API,Deprecated<9>
@anchor{whatsnew/3 4 deprecated-features}@anchor{ba2}
@subsubsection Deprecated Features



@itemize *

@item 
Running @ref{ba3,,IDLE} with the @code{-n} flag (no subprocess) is deprecated.
However, the feature will not be removed until bpo-18823@footnote{https://bugs.python.org/issue18823} is resolved.

@item 
The site module adding a “site-python” directory to sys.path, if it
exists, is deprecated (bpo-19375@footnote{https://bugs.python.org/issue19375}).
@end itemize

@node Removed<9>,Porting to Python 3 4,Deprecated<9>,What’s New In Python 3 4
@anchor{whatsnew/3 4 removed}@anchor{ba4}
@subsection Removed


@menu
* Operating Systems No Longer Supported:: 
* API and Feature Removals: API and Feature Removals<5>. 
* Code Cleanups:: 

@end menu

@node Operating Systems No Longer Supported,API and Feature Removals<5>,,Removed<9>
@anchor{whatsnew/3 4 operating-systems-no-longer-supported}@anchor{ba5}
@subsubsection Operating Systems No Longer Supported


Support for the following operating systems has been removed from the source
and build tools:


@itemize *

@item 
OS/2 (bpo-16135@footnote{https://bugs.python.org/issue16135}).

@item 
Windows 2000 (changeset e52df05b496a).

@item 
Windows systems where @code{COMSPEC} points to @code{command.com} (bpo-14470@footnote{https://bugs.python.org/issue14470}).

@item 
VMS (bpo-16136@footnote{https://bugs.python.org/issue16136}).
@end itemize

@node API and Feature Removals<5>,Code Cleanups,Operating Systems No Longer Supported,Removed<9>
@anchor{whatsnew/3 4 api-and-feature-removals}@anchor{ba6}
@subsubsection API and Feature Removals


The following obsolete and previously deprecated APIs and features have been
removed:


@itemize *

@item 
The unmaintained @code{Misc/TextMate} and @code{Misc/vim} directories have been
removed (see the devguide@footnote{https://devguide.python.org}
for suggestions on what to use instead).

@item 
The @code{SO} makefile macro is removed (it was replaced by the
@code{SHLIB_SUFFIX} and @code{EXT_SUFFIX} macros) (bpo-16754@footnote{https://bugs.python.org/issue16754}).

@item 
The @code{PyThreadState.tick_counter} field has been removed; its value has
been meaningless since Python 3.2, when the “new GIL” was introduced
(bpo-19199@footnote{https://bugs.python.org/issue19199}).

@item 
@code{PyLoader} and @code{PyPycLoader} have been removed from @ref{96,,importlib}.
(Contributed by Taras Lyapun in bpo-15641@footnote{https://bugs.python.org/issue15641}.)

@item 
The @emph{strict} argument to @ref{663,,HTTPConnection} and
@ref{664,,HTTPSConnection} has been removed.  HTTP 0.9-style
“Simple Responses” are no longer supported.

@item 
The deprecated @ref{b5e,,urllib.request.Request} getter and setter methods
@code{add_data}, @code{has_data}, @code{get_data}, @code{get_type}, @code{get_host},
@code{get_selector}, @code{set_proxy}, @code{get_origin_req_host}, and
@code{is_unverifiable} have been removed (use direct attribute access instead).

@item 
Support for loading the deprecated @code{TYPE_INT64} has been removed from
@ref{ac,,marshal}.  (Contributed by Dan Riti in bpo-15480@footnote{https://bugs.python.org/issue15480}.)

@item 
@ref{975,,inspect.Signature}: positional-only parameters are now required
to have a valid name.

@item 
@ref{ba7,,object.__format__()} no longer accepts non-empty format strings, it now
raises a @ref{162,,TypeError} instead.  Using a non-empty string has been
deprecated since Python 3.2.  This change has been made to prevent a
situation where previously working (but incorrect) code would start failing
if an object gained a __format__ method, which means that your code may now
raise a @ref{162,,TypeError} if you are using an @code{'s'} format code with objects
that do not have a __format__ method that handles it.  See bpo-7994@footnote{https://bugs.python.org/issue7994} for
background.

@item 
@code{difflib.SequenceMatcher.isbjunk()} and
@code{difflib.SequenceMatcher.isbpopular()} were deprecated in 3.2, and have
now been removed: use @code{x in sm.bjunk} and
@code{x in sm.bpopular}, where @emph{sm} is a @ref{ba8,,SequenceMatcher} object
(bpo-13248@footnote{https://bugs.python.org/issue13248}).
@end itemize

@node Code Cleanups,,API and Feature Removals<5>,Removed<9>
@anchor{whatsnew/3 4 code-cleanups}@anchor{ba9}
@subsubsection Code Cleanups



@itemize *

@item 
The unused and undocumented internal @code{Scanner} class has been removed from
the @ref{d4,,pydoc} module.

@item 
The private and effectively unused @code{_gestalt} module has been removed,
along with the private @ref{c9,,platform} functions @code{_mac_ver_lookup},
@code{_mac_ver_gstalt}, and @code{_bcd2str}, which would only have ever been called
on badly broken OSX systems (see bpo-18393@footnote{https://bugs.python.org/issue18393}).

@item 
The hardcoded copies of certain @ref{ef,,stat} constants that were included in
the @ref{fb,,tarfile} module namespace have been removed.
@end itemize

@node Porting to Python 3 4,Changed in 3 4 3,Removed<9>,What’s New In Python 3 4
@anchor{whatsnew/3 4 porting-to-python-3-4}@anchor{baa}
@subsection Porting to Python 3.4


This section lists previously described changes and other bugfixes
that may require changes to your code.

@menu
* Changes in ‘python’ Command Behavior: Changes in ‘python’ Command Behavior<2>. 
* Changes in the Python API: Changes in the Python API<8>. 
* Changes in the C API: Changes in the C API<7>. 

@end menu

@node Changes in ‘python’ Command Behavior<2>,Changes in the Python API<8>,,Porting to Python 3 4
@anchor{whatsnew/3 4 changes-in-python-command-behavior}@anchor{bab}
@subsubsection Changes in ‘python’ Command Behavior



@itemize *

@item 
In a posix shell, setting the 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} environment variable to
an empty value is equivalent to not setting it at all.  However, setting
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} to an empty value was @emph{not} equivalent to not setting it
at all: setting 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} to an empty value was equivalent to
setting it to @code{.}, which leads to confusion when reasoning by analogy to
how 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} works.  The behavior now conforms to the posix convention
for 
@geindex PATH
@geindex environment variable; PATH
@code{PATH}.

@item 
The [X refs, Y blocks] output of a debug (@code{--with-pydebug}) build of the
CPython interpreter is now off by default.  It can be re-enabled using the
@code{-X showrefcount} option.  (Contributed by Ezio Melotti in bpo-17323@footnote{https://bugs.python.org/issue17323}.)

@item 
The python command and most stdlib scripts (as well as @ref{5,,argparse}) now
output @code{--version} information to @code{stdout} instead of @code{stderr} (for
issue list see @ref{b8c,,Other Improvements} above).
@end itemize

@node Changes in the Python API<8>,Changes in the C API<7>,Changes in ‘python’ Command Behavior<2>,Porting to Python 3 4
@anchor{whatsnew/3 4 changes-in-the-python-api}@anchor{bad}
@subsubsection Changes in the Python API



@itemize *

@item 
The ABCs defined in @ref{97,,importlib.abc} now either raise the appropriate
exception or return a default value instead of raising
@ref{8a3,,NotImplementedError} blindly. This will only affect code calling
@ref{78c,,super()} and falling through all the way to the ABCs. For compatibility,
catch both @ref{8a3,,NotImplementedError} or the appropriate exception as needed.

@item 
The module type now initializes the @ref{bae,,__package__} and @ref{baf,,__loader__}
attributes to @code{None} by default. To determine if these attributes were set
in a backwards-compatible fashion, use e.g.
@code{getattr(module, '__loader__', None) is not None}.  (bpo-17115@footnote{https://bugs.python.org/issue17115}.)

@item 
@ref{311,,importlib.util.module_for_loader()} now sets @code{__loader__} and
@code{__package__} unconditionally to properly support reloading. If this is not
desired then you will need to set these attributes manually. You can use
@code{importlib.util.module_to_load()} for module management.

@item 
Import now resets relevant attributes (e.g. @code{__name__}, @code{__loader__},
@code{__package__}, @code{__file__}, @code{__cached__}) unconditionally when reloading.
Note that this restores a pre-3.3 behavior in that it means a module is
re-found when re-loaded (bpo-19413@footnote{https://bugs.python.org/issue19413}).

@item 
Frozen packages no longer set @code{__path__} to a list containing the package
name, they now set it to an empty list.  The previous behavior could cause
the import system to do the wrong thing on submodule imports if there was
also a directory with the same name as the frozen package.  The correct way
to determine if a module is a package or not is to use @code{hasattr(module,
'__path__')} (bpo-18065@footnote{https://bugs.python.org/issue18065}).

@item 
Frozen modules no longer define a @code{__file__} attribute. It’s semantically
incorrect for frozen modules to set the attribute as they are not loaded from
any explicit location. If you must know that a module comes from frozen code
then you can see if the module’s @code{__spec__.location} is set to @code{'frozen'},
check if the loader is a subclass of
@ref{bb0,,importlib.machinery.FrozenImporter},
or if Python 2 compatibility is necessary you can use @code{imp.is_frozen()}.

@item 
@ref{534,,py_compile.compile()} now raises @ref{bb1,,FileExistsError} if the file path
it would write to is a symlink or a non-regular file. This is to act as a
warning that import will overwrite those files with a regular file regardless
of what type of file path they were originally.

@item 
@ref{bb2,,importlib.abc.SourceLoader.get_source()} no longer raises
@ref{3b8,,ImportError} when the source code being loaded triggers a
@ref{21a,,SyntaxError} or @ref{51f,,UnicodeDecodeError}. As @ref{3b8,,ImportError} is
meant to be raised only when source code cannot be found but it should, it was
felt to be over-reaching/overloading of that meaning when the source code is
found but improperly structured. If you were catching ImportError before and
wish to continue to ignore syntax or decoding issues, catch all three
exceptions now.

@item 
@ref{bb3,,functools.update_wrapper()} and @ref{adb,,functools.wraps()} now correctly
set the @code{__wrapped__} attribute to the function being wrapped, even if
that function also had its @code{__wrapped__} attribute set. This means
@code{__wrapped__} attributes now correctly link a stack of decorated
functions rather than every @code{__wrapped__} attribute in the chain
referring to the innermost function. Introspection libraries that
assumed the previous behaviour was intentional can use
@ref{ada,,inspect.unwrap()} to access the first function in the chain that has
no @code{__wrapped__} attribute.

@item 
@ref{1c2,,inspect.getfullargspec()} has been reimplemented on top of
@ref{1c1,,inspect.signature()} and hence handles a much wider variety of callable
objects than it did in the past. It is expected that additional builtin and
extension module callables will gain signature metadata over the course of
the Python 3.4 series. Code that assumes that
@ref{1c2,,inspect.getfullargspec()} will fail on non-Python callables may need
to be adjusted accordingly.

@item 
@ref{bb4,,importlib.machinery.PathFinder} now passes on the current working
directory to objects in @ref{bb5,,sys.path_hooks} for the empty string. This
results in @ref{74c,,sys.path_importer_cache} never containing @code{'@w{'}}, thus
iterating through @ref{74c,,sys.path_importer_cache} based on @ref{489,,sys.path}
will not find all keys. A module’s @code{__file__} when imported in the current
working directory will also now have an absolute path, including when using
@code{-m} with the interpreter (except for @code{__main__.__file__} when a script
has been executed directly using a relative path) (Contributed by Brett
Cannon in bpo-18416@footnote{https://bugs.python.org/issue18416}).  is specified on the command-line)
(bpo-18416@footnote{https://bugs.python.org/issue18416}).

@item 
The removal of the @emph{strict} argument to @ref{663,,HTTPConnection}
and @ref{664,,HTTPSConnection} changes the meaning of the
remaining arguments if you are specifying them positionally rather than by
keyword.  If you’ve been paying attention to deprecation warnings your code
should already be specifying any additional arguments via keywords.

@item 
Strings between @code{from __future__ import ...} statements now @emph{always} raise
a @ref{21a,,SyntaxError}.  Previously if there was no leading docstring, an
interstitial string would sometimes be ignored.  This brings CPython into
compliance with the language spec; Jython and PyPy already were.
(bpo-17434@footnote{https://bugs.python.org/issue17434}).

@item 
@ref{b39,,ssl.SSLSocket.getpeercert()} and @ref{9d9,,ssl.SSLSocket.do_handshake()}
now raise an @ref{2a1,,OSError} with @code{ENOTCONN} when the @code{SSLSocket} is not
connected, instead of the previous behavior of raising an
@ref{163,,AttributeError}.  In addition, @ref{b39,,getpeercert()}
will raise a @ref{17c,,ValueError} if the handshake has not yet been done.

@item 
@ref{bb6,,base64.b32decode()} now raises a @ref{bb7,,binascii.Error} when the
input string contains non-b32-alphabet characters, instead of a
@ref{162,,TypeError}.  This particular @ref{162,,TypeError} was missed when the other
@ref{162,,TypeError}s were converted.  (Contributed by Serhiy Storchaka in
bpo-18011@footnote{https://bugs.python.org/issue18011}.)  Note: this change was also inadvertently applied in Python
3.3.3.

@item 
The @code{file} attribute is now automatically closed when
the creating @code{cgi.FieldStorage} instance is garbage collected. If you
were pulling the file object out separately from the @code{cgi.FieldStorage}
instance and not keeping the instance alive, then you should either store the
entire @code{cgi.FieldStorage} instance or read the contents of the file
before the @code{cgi.FieldStorage} instance is garbage collected.

@item 
Calling @code{read} or @code{write} on a closed SSL socket now raises an
informative @ref{17c,,ValueError} rather than the previous more mysterious
@ref{163,,AttributeError} (bpo-9177@footnote{https://bugs.python.org/issue9177}).

@item 
@ref{bb8,,slice.indices()} no longer produces an @ref{347,,OverflowError} for huge
values.  As a consequence of this fix, @ref{bb8,,slice.indices()} now raises a
@ref{17c,,ValueError} if given a negative length; previously it returned nonsense
values (bpo-14794@footnote{https://bugs.python.org/issue14794}).

@item 
The @ref{33b,,complex} constructor, unlike the @ref{17,,cmath} functions, was
incorrectly accepting @ref{256,,float} values if an object’s @code{__complex__}
special method returned one.  This now raises a @ref{162,,TypeError}.
(bpo-16290@footnote{https://bugs.python.org/issue16290}.)

@item 
The @ref{1b1,,int} constructor in 3.2 and 3.3 erroneously accepts @ref{256,,float}
values for the @emph{base} parameter.  It is unlikely anyone was doing this, but
if so, it will now raise a @ref{162,,TypeError} (bpo-16772@footnote{https://bugs.python.org/issue16772}).

@item 
Defaults for keyword-only arguments are now evaluated @emph{after} defaults for
regular keyword arguments, instead of before.  Hopefully no one wrote any
code that depends on the previous buggy behavior (bpo-16967@footnote{https://bugs.python.org/issue16967}).

@item 
Stale thread states are now cleared after @ref{bb9,,fork()}.  This may cause
some system resources to be released that previously were incorrectly kept
perpetually alive (for example, database connections kept in thread-local
storage).  (bpo-17094@footnote{https://bugs.python.org/issue17094}.)

@item 
Parameter names in @code{__annotations__} dicts are now mangled properly,
similarly to @code{__kwdefaults__}.  (Contributed by Yury Selivanov in
bpo-20625@footnote{https://bugs.python.org/issue20625}.)

@item 
@ref{ac5,,hashlib.hash.name} now always returns the identifier in lower case.
Previously some builtin hashes had uppercase names, but now that it is a
formal public interface the naming has been made consistent (bpo-18532@footnote{https://bugs.python.org/issue18532}).

@item 
Because @ref{953,,unittest.TestSuite} now drops references to tests after they
are run, test harnesses that re-use a @ref{953,,TestSuite} to re-run
a set of tests may fail.   Test suites should not be re-used in this fashion
since it means state is retained between test runs, breaking the test
isolation that @ref{122,,unittest} is designed to provide.  However, if the lack
of isolation is considered acceptable, the old behavior can be restored by
creating a @ref{953,,TestSuite} subclass that defines a
@code{_removeTestAtIndex} method that does nothing (see
@ref{bba,,TestSuite.__iter__()}) (bpo-11798@footnote{https://bugs.python.org/issue11798}).

@item 
@ref{122,,unittest} now uses @ref{5,,argparse} for command line parsing.  There are
certain invalid command forms that used to work that are no longer allowed;
in theory this should not cause backward compatibility issues since the
disallowed command forms didn’t make any sense and are unlikely to be in use.

@item 
The @ref{699,,re.split()}, @ref{bbb,,re.findall()}, and @ref{72e,,re.sub()} functions, and
the @code{group()} and @code{groups()} methods of
@code{match} objects now always return a @emph{bytes} object when the string
to be matched is a @ref{87e,,bytes-like object}.  Previously the return type
matched the input type, so if your code was depending on the return value
being, say, a @code{bytearray}, you will need to change your code.

@item 
@ref{c,,audioop} functions now raise an error immediately if passed string
input, instead of failing randomly later on (bpo-16685@footnote{https://bugs.python.org/issue16685}).

@item 
The new @emph{convert_charrefs} argument to @ref{33c,,HTMLParser}
currently defaults to @code{False} for backward compatibility, but will
eventually be changed to default to @code{True}.  It is recommended that you add
this keyword, with the appropriate value, to any
@ref{33c,,HTMLParser} calls in your code (bpo-13633@footnote{https://bugs.python.org/issue13633}).

@item 
Since the @emph{digestmod} argument to the @ref{ac7,,hmac.new()} function will in the
future have no default, all calls to @ref{ac7,,hmac.new()} should be changed to
explicitly specify a @emph{digestmod} (bpo-17276@footnote{https://bugs.python.org/issue17276}).

@item 
Calling @ref{bbc,,sysconfig.get_config_var()} with the @code{SO} key, or looking
@code{SO} up in the results of a call to @ref{bbd,,sysconfig.get_config_vars()}
is deprecated.  This key should be replaced by @code{EXT_SUFFIX} or
@code{SHLIB_SUFFIX}, depending on the context (bpo-19555@footnote{https://bugs.python.org/issue19555}).

@item 
Any calls to @code{open} functions that specify @code{U} should be modified.
@code{U} is ineffective in Python3 and will eventually raise an error if used.
Depending on the function, the equivalent of its old Python2 behavior can be
achieved using either a @emph{newline} argument, or if necessary by wrapping the
stream in @ref{889,,TextIOWrapper} to use its @emph{newline} argument
(bpo-15204@footnote{https://bugs.python.org/issue15204}).

@item 
If you use @code{pyvenv} in a script and desire that pip
@emph{not} be installed, you must add @code{--without-pip} to your command
invocation.

@item 
The default behavior of @ref{89a,,json.dump()} and @ref{89b,,json.dumps()} when
an indent is specified has changed: it no longer produces trailing
spaces after the item separating commas at the ends of lines.  This
will matter only if you have tests that are doing white-space-sensitive
comparisons of such output (bpo-16333@footnote{https://bugs.python.org/issue16333}).

@item 
@ref{63,,doctest} now looks for doctests in extension module @code{__doc__}
strings, so if your doctest test discovery includes extension modules that
have things that look like doctests in them you may see test failures you’ve
never seen before when running your tests (bpo-3158@footnote{https://bugs.python.org/issue3158}).

@item 
The @ref{1d,,collections.abc} module has been slightly refactored as
part of the Python startup improvements.  As a consequence of this, it is no
longer the case that importing @ref{1c,,collections} automatically imports
@ref{1d,,collections.abc}.  If your program depended on the (undocumented)
implicit import, you will need to add an explicit @code{import collections.abc}
(bpo-20784@footnote{https://bugs.python.org/issue20784}).
@end itemize

@node Changes in the C API<7>,,Changes in the Python API<8>,Porting to Python 3 4
@anchor{whatsnew/3 4 changes-in-the-c-api}@anchor{bbe}
@subsubsection Changes in the C API



@itemize *

@item 
@ref{bbf,,PyEval_EvalFrameEx()}, @ref{bc0,,PyObject_Repr()}, and
@ref{bc1,,PyObject_Str()}, along with some other internal C APIs, now include
a debugging assertion that ensures they are not used in situations where
they may silently discard a currently active exception. In cases where
discarding the active exception is expected and desired (for example,
because it has already been saved locally with @ref{bc2,,PyErr_Fetch()} or
is being deliberately replaced with a different exception), an explicit
@ref{bc3,,PyErr_Clear()} call will be needed to avoid triggering the
assertion when invoking these operations (directly or indirectly) and
running against a version of Python that is compiled with assertions
enabled.

@item 
@ref{88e,,PyErr_SetImportError()} now sets @ref{162,,TypeError} when its @strong{msg}
argument is not set. Previously only @code{NULL} was returned with no exception
set.

@item 
The result of the @ref{3a2,,PyOS_ReadlineFunctionPointer} callback must
now be a string allocated by @ref{bc4,,PyMem_RawMalloc()} or
@ref{bc5,,PyMem_RawRealloc()}, or @code{NULL} if an error occurred, instead of a
string allocated by @ref{7ad,,PyMem_Malloc()} or @ref{bc6,,PyMem_Realloc()}
(bpo-16742@footnote{https://bugs.python.org/issue16742})

@item 
@ref{bc7,,PyThread_set_key_value()} now always set the value. In Python
3.3, the function did nothing if the key already exists (if the current
value is a non-@code{NULL} pointer).

@item 
The @code{f_tstate} (thread state) field of the @ref{1f6,,PyFrameObject}
structure has been removed to fix a bug: see bpo-14432@footnote{https://bugs.python.org/issue14432} for the
rationale.
@end itemize

@node Changed in 3 4 3,,Porting to Python 3 4,What’s New In Python 3 4
@anchor{whatsnew/3 4 changed-in-3-4-3}@anchor{bc8}
@subsection Changed in 3.4.3


@menu
* PEP 476; Enabling certificate verification by default for stdlib http clients: PEP 476 Enabling certificate verification by default for stdlib http clients. 

@end menu

@node PEP 476 Enabling certificate verification by default for stdlib http clients,,,Changed in 3 4 3
@anchor{whatsnew/3 4 pep-476}@anchor{bc9}@anchor{whatsnew/3 4 pep-476-enabling-certificate-verification-by-default-for-stdlib-http-clients}@anchor{bca}
@subsubsection PEP 476: Enabling certificate verification by default for stdlib http clients


@ref{8f,,http.client} and modules which use it, such as @ref{127,,urllib.request} and
@ref{146,,xmlrpc.client}, will now verify that the server presents a certificate
which is signed by a CA in the platform trust store and whose hostname matches
the hostname being requested by default, significantly improving security for
many applications.

For applications which require the old previous behavior, they can pass an
alternate context:

@example
import urllib.request
import ssl

# This disables all verification
context = ssl._create_unverified_context()

# This allows using a specific certificate for the host, which doesn't need
# to be in the trust store
context = ssl.create_default_context(cafile="/path/to/file.crt")

urllib.request.urlopen("https://invalid-cert", context=context)
@end example

@node What’s New In Python 3 3,What’s New In Python 3 2,What’s New In Python 3 4,What’s New in Python
@anchor{whatsnew/3 3 doc}@anchor{bcb}@anchor{whatsnew/3 3 what-s-new-in-python-3-3}@anchor{bcc}
@section What’s New In Python 3.3


@c Rules for maintenance:
@c 
@c * Anyone can add text to this document.  Do not spend very much time
@c on the wording of your changes, because your text will probably
@c get rewritten to some degree.
@c 
@c * The maintainer will go through Misc/NEWS periodically and add
@c changes; it's therefore more important to add your changes to
@c Misc/NEWS than to this file.
@c 
@c * This is not a complete list of every single change; completeness
@c is the purpose of Misc/NEWS.  Some changes I consider too small
@c or esoteric to include.  If such a change is added to the text,
@c I'll just remove it.  (This is another reason you shouldn't spend
@c too much time on writing your addition.)
@c 
@c * If you want to draw your new text to the attention of the
@c maintainer, add 'XXX' to the beginning of the paragraph or
@c section.
@c 
@c * It's OK to just add a fragmentary note about a change.  For
@c example: "XXX Describe the transmogrify() function added to the
@c socket module."  The maintainer will research the change and
@c write the necessary text.
@c 
@c * You can comment out your additions if you like, but it's not
@c necessary (especially when a final release is some months away).
@c 
@c * Credit the author of a patch or bugfix.   Just the name is
@c sufficient; the e-mail address isn't necessary.
@c 
@c * It's helpful to add the bug/patch number as a comment:
@c 
@c XXX Describe the transmogrify() function added to the socket
@c module.
@c (Contributed by P.Y. Developer in :issue:`12345`.)
@c 
@c This saves the maintainer the effort of going through the Mercurial log
@c when researching a change.

This article explains the new features in Python 3.3, compared to 3.2.
Python 3.3 was released on September 29, 2012.  For full details,
see the changelog@footnote{https://docs.python.org/3.3/whatsnew/changelog.html}.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 398
PEP 398@footnote{https://www.python.org/dev/peps/pep-0398} - Python 3.3 Release Schedule


@menu
* Summary – Release highlights: Summary – Release highlights<7>. 
* PEP 405; Virtual Environments: PEP 405 Virtual Environments. 
* PEP 420; Implicit Namespace Packages: PEP 420 Implicit Namespace Packages. 
* PEP 3118; New memoryview implementation and buffer protocol documentation: PEP 3118 New memoryview implementation and buffer protocol documentation. 
* PEP 393; Flexible String Representation: PEP 393 Flexible String Representation. 
* PEP 397; Python Launcher for Windows: PEP 397 Python Launcher for Windows. 
* PEP 3151; Reworking the OS and IO exception hierarchy: PEP 3151 Reworking the OS and IO exception hierarchy. 
* PEP 380; Syntax for Delegating to a Subgenerator: PEP 380 Syntax for Delegating to a Subgenerator. 
* PEP 409; Suppressing exception context: PEP 409 Suppressing exception context. 
* PEP 414; Explicit Unicode literals: PEP 414 Explicit Unicode literals. 
* PEP 3155; Qualified name for classes and functions: PEP 3155 Qualified name for classes and functions. 
* PEP 412; Key-Sharing Dictionary: PEP 412 Key-Sharing Dictionary. 
* PEP 362; Function Signature Object: PEP 362 Function Signature Object. 
* PEP 421; Adding sys.implementation: PEP 421 Adding sys implementation. 
* Using importlib as the Implementation of Import:: 
* Other Language Changes: Other Language Changes<9>. 
* A Finer-Grained Import Lock:: 
* Builtin functions and types:: 
* New Modules: New Modules<9>. 
* Improved Modules: Improved Modules<9>. 
* Optimizations: Optimizations<8>. 
* Build and C API Changes: Build and C API Changes<4>. 
* Deprecated: Deprecated<10>. 
* Porting to Python 3.3: Porting to Python 3 3. 

@end menu

@node Summary – Release highlights<7>,PEP 405 Virtual Environments,,What’s New In Python 3 3
@anchor{whatsnew/3 3 summary-release-highlights}@anchor{bcd}
@subsection Summary – Release highlights


@c This section singles out the most important changes in Python 3.3.
@c Brevity is key.

New syntax features:


@itemize *

@item 
New @code{yield from} expression for @ref{bce,,generator delegation}.

@item 
The @code{u'unicode'} syntax is accepted again for @ref{2ef,,str} objects.
@end itemize

New library modules:


@itemize *

@item 
@ref{78,,faulthandler} (helps debugging low-level crashes)

@item 
@ref{9e,,ipaddress} (high-level objects representing IP addresses and masks)

@item 
@ref{a9,,lzma} (compress data using the XZ / LZMA algorithm)

@item 
@ref{123,,unittest.mock} (replace parts of your system under test with mock objects)

@item 
@ref{12c,,venv} (Python @ref{bcf,,virtual environments}, as in the
popular @code{virtualenv} package)
@end itemize

New built-in features:


@itemize *

@item 
Reworked @ref{bd0,,I/O exception hierarchy}.
@end itemize

Implementation improvements:


@itemize *

@item 
Rewritten @ref{bd1,,import machinery} based on @ref{96,,importlib}.

@item 
More compact @ref{bd2,,unicode strings}.

@item 
More compact @ref{bd3,,attribute dictionaries}.
@end itemize

Significantly Improved Library Modules:


@itemize *

@item 
C Accelerator for the @ref{bd4,,decimal} module.

@item 
Better unicode handling in the @ref{bd5,,email} module
(@ref{bd6,,provisional}).
@end itemize

Security improvements:


@itemize *

@item 
Hash randomization is switched on by default.
@end itemize

Please read on for a comprehensive list of user-facing changes.

@node PEP 405 Virtual Environments,PEP 420 Implicit Namespace Packages,Summary – Release highlights<7>,What’s New In Python 3 3
@anchor{whatsnew/3 3 pep-405}@anchor{bcf}@anchor{whatsnew/3 3 pep-405-virtual-environments}@anchor{bd7}
@subsection PEP 405: Virtual Environments


Virtual environments help create separate Python setups while sharing a
system-wide base install, for ease of maintenance.  Virtual environments
have their own set of private site packages (i.e. locally-installed
libraries), and are optionally segregated from the system-wide site
packages.  Their concept and implementation are inspired by the popular
@code{virtualenv} third-party package, but benefit from tighter integration
with the interpreter core.

This PEP adds the @ref{12c,,venv} module for programmatic access, and the
@code{pyvenv} script for command-line access and
administration.  The Python interpreter checks for a @code{pyvenv.cfg},
file whose existence signals the base of a virtual environment’s directory
tree.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 405

@item PEP 405@footnote{https://www.python.org/dev/peps/pep-0405} - Python Virtual Environments

PEP written by Carl Meyer; implementation by Carl Meyer and Vinay Sajip
@end table


@node PEP 420 Implicit Namespace Packages,PEP 3118 New memoryview implementation and buffer protocol documentation,PEP 405 Virtual Environments,What’s New In Python 3 3
@anchor{whatsnew/3 3 pep-420-implicit-namespace-packages}@anchor{bd8}
@subsection PEP 420: Implicit Namespace Packages


Native support for package directories that don’t require @code{__init__.py}
marker files and can automatically span multiple path segments (inspired by
various third party approaches to namespace packages, as described in
@geindex Python Enhancement Proposals; PEP 420
PEP 420@footnote{https://www.python.org/dev/peps/pep-0420})


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 420

@item PEP 420@footnote{https://www.python.org/dev/peps/pep-0420} - Implicit Namespace Packages

PEP written by Eric V. Smith; implementation by Eric V. Smith
and Barry Warsaw
@end table


@node PEP 3118 New memoryview implementation and buffer protocol documentation,PEP 393 Flexible String Representation,PEP 420 Implicit Namespace Packages,What’s New In Python 3 3
@anchor{whatsnew/3 3 pep-3118-new-memoryview-implementation-and-buffer-protocol-documentation}@anchor{bd9}@anchor{whatsnew/3 3 pep-3118-update}@anchor{bda}
@subsection PEP 3118: New memoryview implementation and buffer protocol documentation


The implementation of 
@geindex Python Enhancement Proposals; PEP 3118
PEP 3118@footnote{https://www.python.org/dev/peps/pep-3118} has been significantly improved.

The new memoryview implementation comprehensively fixes all ownership and
lifetime issues of dynamically allocated fields in the Py_buffer struct
that led to multiple crash reports. Additionally, several functions that
crashed or returned incorrect results for non-contiguous or multi-dimensional
input have been fixed.

The memoryview object now has a PEP-3118 compliant getbufferproc()
that checks the consumer’s request type. Many new features have been
added, most of them work in full generality for non-contiguous arrays
and arrays with suboffsets.

The documentation has been updated, clearly spelling out responsibilities
for both exporters and consumers. Buffer request flags are grouped into
basic and compound flags. The memory layout of non-contiguous and
multi-dimensional NumPy-style arrays is explained.

@menu
* Features:: 
* API changes:: 

@end menu

@node Features,API changes,,PEP 3118 New memoryview implementation and buffer protocol documentation
@anchor{whatsnew/3 3 features}
@subsubsection Features



@itemize *

@item 
All native single character format specifiers in struct module syntax
(optionally prefixed with ‘@@’) are now supported.

@item 
With some restrictions, the cast() method allows changing of format and
shape of C-contiguous arrays.

@item 
Multi-dimensional list representations are supported for any array type.

@item 
Multi-dimensional comparisons are supported for any array type.

@item 
One-dimensional memoryviews of hashable (read-only) types with formats B,
b or c are now hashable.  (Contributed by Antoine Pitrou in bpo-13411@footnote{https://bugs.python.org/issue13411}.)

@item 
Arbitrary slicing of any 1-D arrays type is supported. For example, it
is now possible to reverse a memoryview in O(1) by using a negative step.
@end itemize

@node API changes,,Features,PEP 3118 New memoryview implementation and buffer protocol documentation
@anchor{whatsnew/3 3 api-changes}@anchor{bdc}
@subsubsection API changes



@itemize *

@item 
The maximum number of dimensions is officially limited to 64.

@item 
The representation of empty shape, strides and suboffsets is now
an empty tuple instead of @code{None}.

@item 
Accessing a memoryview element with format ‘B’ (unsigned bytes)
now returns an integer (in accordance with the struct module syntax).
For returning a bytes object the view must be cast to ‘c’ first.

@item 
memoryview comparisons now use the logical structure of the operands
and compare all array elements by value. All format strings in struct
module syntax are supported. Views with unrecognised format strings
are still permitted, but will always compare as unequal, regardless
of view contents.

@item 
For further changes see @ref{bdd,,Build and C API Changes} and @ref{bde,,Porting C code}.
@end itemize

(Contributed by Stefan Krah in bpo-10181@footnote{https://bugs.python.org/issue10181}.)


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 3118
PEP 3118@footnote{https://www.python.org/dev/peps/pep-3118} - Revising the Buffer Protocol


@node PEP 393 Flexible String Representation,PEP 397 Python Launcher for Windows,PEP 3118 New memoryview implementation and buffer protocol documentation,What’s New In Python 3 3
@anchor{whatsnew/3 3 pep-393}@anchor{bd2}@anchor{whatsnew/3 3 pep-393-flexible-string-representation}@anchor{bdf}
@subsection PEP 393: Flexible String Representation


The Unicode string type is changed to support multiple internal
representations, depending on the character with the largest Unicode ordinal
(1, 2, or 4 bytes) in the represented string.  This allows a space-efficient
representation in common cases, but gives access to full UCS-4 on all
systems.  For compatibility with existing APIs, several representations may
exist in parallel; over time, this compatibility should be phased out.

On the Python side, there should be no downside to this change.

On the C API side, 
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393} is fully backward compatible.  The legacy API
should remain available at least five years.  Applications using the legacy
API will not fully benefit of the memory reduction, or - worse - may use
a bit more memory, because Python may have to maintain two versions of each
string (in the legacy format and in the new efficient storage).

@menu
* Functionality:: 
* Performance and resource usage:: 

@end menu

@node Functionality,Performance and resource usage,,PEP 393 Flexible String Representation
@anchor{whatsnew/3 3 functionality}@anchor{be0}
@subsubsection Functionality


Changes introduced by 
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393} are the following:


@itemize *

@item 
Python now always supports the full range of Unicode code points, including
non-BMP ones (i.e. from @code{U+0000} to @code{U+10FFFF}).  The distinction between
narrow and wide builds no longer exists and Python now behaves like a wide
build, even under Windows.

@item 
With the death of narrow builds, the problems specific to narrow builds have
also been fixed, for example:


@itemize *

@item 
@ref{4a6,,len()} now always returns 1 for non-BMP characters,
so @code{len('\U0010FFFF') == 1};

@item 
surrogate pairs are not recombined in string literals,
so @code{'\uDBFF\uDFFF' != '\U0010FFFF'};

@item 
indexing or slicing non-BMP characters returns the expected value,
so @code{'\U0010FFFF'[0]} now returns @code{'\U0010FFFF'} and not @code{'\uDBFF'};

@item 
all other functions in the standard library now correctly handle
non-BMP code points.
@end itemize

@item 
The value of @ref{be1,,sys.maxunicode} is now always @code{1114111} (@code{0x10FFFF}
in hexadecimal).  The @code{PyUnicode_GetMax()} function still returns
either @code{0xFFFF} or @code{0x10FFFF} for backward compatibility, and it should
not be used with the new Unicode API (see bpo-13054@footnote{https://bugs.python.org/issue13054}).

@item 
The @code{./configure} flag @code{--with-wide-unicode} has been removed.
@end itemize

@node Performance and resource usage,,Functionality,PEP 393 Flexible String Representation
@anchor{whatsnew/3 3 performance-and-resource-usage}@anchor{be2}
@subsubsection Performance and resource usage


The storage of Unicode strings now depends on the highest code point in the string:


@itemize *

@item 
pure ASCII and Latin1 strings (@code{U+0000-U+00FF}) use 1 byte per code point;

@item 
BMP strings (@code{U+0000-U+FFFF}) use 2 bytes per code point;

@item 
non-BMP strings (@code{U+10000-U+10FFFF}) use 4 bytes per code point.
@end itemize

The net effect is that for most applications, memory usage of string
storage should decrease significantly - especially compared to former
wide unicode builds - as, in many cases, strings will be pure ASCII
even in international contexts (because many strings store non-human
language data, such as XML fragments, HTTP headers, JSON-encoded data,
etc.).  We also hope that it will, for the same reasons, increase CPU
cache efficiency on non-trivial applications. The memory usage of
Python 3.3 is two to three times smaller than Python 3.2, and a little
bit better than Python 2.7, on a Django benchmark (see the PEP for
details).


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 393

@item PEP 393@footnote{https://www.python.org/dev/peps/pep-0393} - Flexible String Representation

PEP written by Martin von Löwis; implementation by Torsten Becker
and Martin von Löwis.
@end table


@node PEP 397 Python Launcher for Windows,PEP 3151 Reworking the OS and IO exception hierarchy,PEP 393 Flexible String Representation,What’s New In Python 3 3
@anchor{whatsnew/3 3 pep-397}@anchor{be3}@anchor{whatsnew/3 3 pep-397-python-launcher-for-windows}@anchor{be4}
@subsection PEP 397: Python Launcher for Windows


The Python 3.3 Windows installer now includes a @code{py} launcher application
that can be used to launch Python applications in a version independent
fashion.

This launcher is invoked implicitly when double-clicking @code{*.py} files.
If only a single Python version is installed on the system, that version
will be used to run the file. If multiple versions are installed, the most
recent version is used by default, but this can be overridden by including
a Unix-style “shebang line” in the Python script.

The launcher can also be used explicitly from the command line as the @code{py}
application. Running @code{py} follows the same version selection rules as
implicitly launching scripts, but a more specific version can be selected
by passing appropriate arguments (such as @code{-3} to request Python 3 when
Python 2 is also installed, or @code{-2.6} to specifically request an earlier
Python version when a more recent version is installed).

In addition to the launcher, the Windows installer now includes an
option to add the newly installed Python to the system PATH.  (Contributed
by Brian Curtin in bpo-3561@footnote{https://bugs.python.org/issue3561}.)


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 397

@item PEP 397@footnote{https://www.python.org/dev/peps/pep-0397} - Python Launcher for Windows

PEP written by Mark Hammond and Martin v. Löwis; implementation by
Vinay Sajip.
@end table

Launcher documentation: @ref{be5,,Python Launcher for Windows}

Installer PATH modification: @ref{be6,,Finding the Python executable}


@node PEP 3151 Reworking the OS and IO exception hierarchy,PEP 380 Syntax for Delegating to a Subgenerator,PEP 397 Python Launcher for Windows,What’s New In Python 3 3
@anchor{whatsnew/3 3 pep-3151}@anchor{bd0}@anchor{whatsnew/3 3 pep-3151-reworking-the-os-and-io-exception-hierarchy}@anchor{be7}
@subsection PEP 3151: Reworking the OS and IO exception hierarchy


The hierarchy of exceptions raised by operating system errors is now both
simplified and finer-grained.

You don’t have to worry anymore about choosing the appropriate exception
type between @ref{2a1,,OSError}, @ref{be8,,IOError}, @ref{be9,,EnvironmentError},
@ref{bea,,WindowsError}, @code{mmap.error}, @ref{beb,,socket.error} or
@ref{bec,,select.error}.  All these exception types are now only one:
@ref{2a1,,OSError}.  The other names are kept as aliases for compatibility
reasons.

Also, it is now easier to catch a specific error condition.  Instead of
inspecting the @code{errno} attribute (or @code{args[0]}) for a particular
constant from the @ref{77,,errno} module, you can catch the adequate
@ref{2a1,,OSError} subclass.  The available subclasses are the following:


@itemize *

@item 
@ref{bed,,BlockingIOError}

@item 
@ref{bee,,ChildProcessError}

@item 
@ref{969,,ConnectionError}

@item 
@ref{bb1,,FileExistsError}

@item 
@ref{a37,,FileNotFoundError}

@item 
@ref{8e3,,InterruptedError}

@item 
@ref{bef,,IsADirectoryError}

@item 
@ref{bf0,,NotADirectoryError}

@item 
@ref{895,,PermissionError}

@item 
@ref{bf1,,ProcessLookupError}

@item 
@ref{2b8,,TimeoutError}
@end itemize

And the @ref{969,,ConnectionError} itself has finer-grained subclasses:


@itemize *

@item 
@ref{bf2,,BrokenPipeError}

@item 
@ref{bf3,,ConnectionAbortedError}

@item 
@ref{bf4,,ConnectionRefusedError}

@item 
@ref{bf5,,ConnectionResetError}
@end itemize

Thanks to the new exceptions, common usages of the @ref{77,,errno} can now be
avoided.  For example, the following code written for Python 3.2:

@example
from errno import ENOENT, EACCES, EPERM

try:
    with open("document.txt") as f:
        content = f.read()
except IOError as err:
    if err.errno == ENOENT:
        print("document.txt file is missing")
    elif err.errno in (EACCES, EPERM):
        print("You are not allowed to read document.txt")
    else:
        raise
@end example

can now be written without the @ref{77,,errno} import and without manual
inspection of exception attributes:

@example
try:
    with open("document.txt") as f:
        content = f.read()
except FileNotFoundError:
    print("document.txt file is missing")
except PermissionError:
    print("You are not allowed to read document.txt")
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3151

@item PEP 3151@footnote{https://www.python.org/dev/peps/pep-3151} - Reworking the OS and IO Exception Hierarchy

PEP written and implemented by Antoine Pitrou
@end table


@geindex yield; yield from (in What's New)

@node PEP 380 Syntax for Delegating to a Subgenerator,PEP 409 Suppressing exception context,PEP 3151 Reworking the OS and IO exception hierarchy,What’s New In Python 3 3
@anchor{whatsnew/3 3 pep-380}@anchor{bce}@anchor{whatsnew/3 3 pep-380-syntax-for-delegating-to-a-subgenerator}@anchor{bf6}
@subsection PEP 380: Syntax for Delegating to a Subgenerator


PEP 380 adds the @code{yield from} expression, allowing a @ref{bf7,,generator} to
delegate
part of its operations to another generator. This allows a section of code
containing @ref{4c5,,yield} to be factored out and placed in another generator.
Additionally, the subgenerator is allowed to return with a value, and the
value is made available to the delegating generator.

While designed primarily for use in delegating to a subgenerator, the @code{yield
from} expression actually allows delegation to arbitrary subiterators.

For simple iterators, @code{yield from iterable} is essentially just a shortened
form of @code{for item in iterable: yield item}:

@example
>>> def g(x):
...     yield from range(x, 0, -1)
...     yield from range(x)
...
>>> list(g(5))
[5, 4, 3, 2, 1, 0, 1, 2, 3, 4]
@end example

However, unlike an ordinary loop, @code{yield from} allows subgenerators to
receive sent and thrown values directly from the calling scope, and
return a final value to the outer generator:

@example
>>> def accumulate():
...     tally = 0
...     while 1:
...         next = yield
...         if next is None:
...             return tally
...         tally += next
...
>>> def gather_tallies(tallies):
...     while 1:
...         tally = yield from accumulate()
...         tallies.append(tally)
...
>>> tallies = []
>>> acc = gather_tallies(tallies)
>>> next(acc)  # Ensure the accumulator is ready to accept values
>>> for i in range(4):
...     acc.send(i)
...
>>> acc.send(None)  # Finish the first tally
>>> for i in range(5):
...     acc.send(i)
...
>>> acc.send(None)  # Finish the second tally
>>> tallies
[6, 10]
@end example

The main principle driving this change is to allow even generators that are
designed to be used with the @code{send} and @code{throw} methods to be split into
multiple subgenerators as easily as a single large function can be split into
multiple subfunctions.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 380

@item PEP 380@footnote{https://www.python.org/dev/peps/pep-0380} - Syntax for Delegating to a Subgenerator

PEP written by Greg Ewing; implementation by Greg Ewing, integrated into
3.3 by Renaud Blanch, Ryan Kelly and Nick Coghlan; documentation by
Zbigniew Jędrzejewski-Szmek and Nick Coghlan
@end table


@node PEP 409 Suppressing exception context,PEP 414 Explicit Unicode literals,PEP 380 Syntax for Delegating to a Subgenerator,What’s New In Python 3 3
@anchor{whatsnew/3 3 pep-409-suppressing-exception-context}@anchor{bf8}
@subsection PEP 409: Suppressing exception context


PEP 409 introduces new syntax that allows the display of the chained
exception context to be disabled. This allows cleaner error messages in
applications that convert between exception types:

@example
>>> class D:
...     def __init__(self, extra):
...         self._extra_attributes = extra
...     def __getattr__(self, attr):
...         try:
...             return self._extra_attributes[attr]
...         except KeyError:
...             raise AttributeError(attr) from None
...
>>> D(@{@}).x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 8, in __getattr__
AttributeError: x
@end example

Without the @code{from None} suffix to suppress the cause, the original
exception would be displayed by default:

@example
>>> class C:
...     def __init__(self, extra):
...         self._extra_attributes = extra
...     def __getattr__(self, attr):
...         try:
...             return self._extra_attributes[attr]
...         except KeyError:
...             raise AttributeError(attr)
...
>>> C(@{@}).x
Traceback (most recent call last):
  File "<stdin>", line 6, in __getattr__
KeyError: 'x'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 8, in __getattr__
AttributeError: x
@end example

No debugging capability is lost, as the original exception context remains
available if needed (for example, if an intervening library has incorrectly
suppressed valuable underlying details):

@example
>>> try:
...     D(@{@}).x
... except AttributeError as exc:
...     print(repr(exc.__context__))
...
KeyError('x',)
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 409

@item PEP 409@footnote{https://www.python.org/dev/peps/pep-0409} - Suppressing exception context

PEP written by Ethan Furman; implemented by Ethan Furman and Nick
Coghlan.
@end table


@node PEP 414 Explicit Unicode literals,PEP 3155 Qualified name for classes and functions,PEP 409 Suppressing exception context,What’s New In Python 3 3
@anchor{whatsnew/3 3 pep-414-explicit-unicode-literals}@anchor{bf9}
@subsection PEP 414: Explicit Unicode literals


To ease the transition from Python 2 for Unicode aware Python applications
that make heavy use of Unicode literals, Python 3.3 once again supports the
“@code{u}” prefix for string literals. This prefix has no semantic significance
in Python 3, it is provided solely to reduce the number of purely mechanical
changes in migrating to Python 3, making it easier for developers to focus on
the more significant semantic changes (such as the stricter default
separation of binary and text data).


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 414

@item PEP 414@footnote{https://www.python.org/dev/peps/pep-0414} - Explicit Unicode literals

PEP written by Armin Ronacher.
@end table


@node PEP 3155 Qualified name for classes and functions,PEP 412 Key-Sharing Dictionary,PEP 414 Explicit Unicode literals,What’s New In Python 3 3
@anchor{whatsnew/3 3 pep-3155-qualified-name-for-classes-and-functions}@anchor{bfa}
@subsection PEP 3155: Qualified name for classes and functions


Functions and class objects have a new @code{__qualname__} attribute representing
the “path” from the module top-level to their definition.  For global functions
and classes, this is the same as @code{__name__}.  For other functions and classes,
it provides better information about where they were actually defined, and
how they might be accessible from the global scope.

Example with (non-bound) methods:

@example
>>> class C:
...     def meth(self):
...         pass
>>> C.meth.__name__
'meth'
>>> C.meth.__qualname__
'C.meth'
@end example

Example with nested classes:

@example
>>> class C:
...     class D:
...         def meth(self):
...             pass
...
>>> C.D.__name__
'D'
>>> C.D.__qualname__
'C.D'
>>> C.D.meth.__name__
'meth'
>>> C.D.meth.__qualname__
'C.D.meth'
@end example

Example with nested functions:

@example
>>> def outer():
...     def inner():
...         pass
...     return inner
...
>>> outer().__name__
'inner'
>>> outer().__qualname__
'outer.<locals>.inner'
@end example

The string representation of those objects is also changed to include the
new, more precise information:

@example
>>> str(C.D)
"<class '__main__.C.D'>"
>>> str(C.D.meth)
'<function C.D.meth at 0x7f46b9fe31e0>'
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3155

@item PEP 3155@footnote{https://www.python.org/dev/peps/pep-3155} - Qualified name for classes and functions

PEP written and implemented by Antoine Pitrou.
@end table


@node PEP 412 Key-Sharing Dictionary,PEP 362 Function Signature Object,PEP 3155 Qualified name for classes and functions,What’s New In Python 3 3
@anchor{whatsnew/3 3 pep-412}@anchor{bd3}@anchor{whatsnew/3 3 pep-412-key-sharing-dictionary}@anchor{bfb}
@subsection PEP 412: Key-Sharing Dictionary


Dictionaries used for the storage of objects’ attributes are now able to
share part of their internal storage between each other (namely, the part
which stores the keys and their respective hashes).  This reduces the memory
consumption of programs creating many instances of non-builtin types.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 412

@item PEP 412@footnote{https://www.python.org/dev/peps/pep-0412} - Key-Sharing Dictionary

PEP written and implemented by Mark Shannon.
@end table


@node PEP 362 Function Signature Object,PEP 421 Adding sys implementation,PEP 412 Key-Sharing Dictionary,What’s New In Python 3 3
@anchor{whatsnew/3 3 pep-362-function-signature-object}@anchor{bfc}
@subsection PEP 362: Function Signature Object


A new function @ref{1c1,,inspect.signature()} makes introspection of python
callables easy and straightforward.  A broad range of callables is supported:
python functions, decorated or not, classes, and @ref{a3f,,functools.partial()}
objects.  New classes @ref{975,,inspect.Signature}, @ref{976,,inspect.Parameter}
and @ref{bfd,,inspect.BoundArguments} hold information about the call signatures,
such as, annotations, default values, parameters kinds, and bound arguments,
which considerably simplifies writing decorators and any code that validates
or amends calling signatures or arguments.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 362

@item PEP 362@footnote{https://www.python.org/dev/peps/pep-0362}: -  Function Signature Object

PEP written by Brett Cannon, Yury Selivanov, Larry Hastings, Jiwon Seo;
implemented by Yury Selivanov.
@end table


@node PEP 421 Adding sys implementation,Using importlib as the Implementation of Import,PEP 362 Function Signature Object,What’s New In Python 3 3
@anchor{whatsnew/3 3 pep-421-adding-sys-implementation}@anchor{bfe}
@subsection PEP 421: Adding sys.implementation


A new attribute on the @ref{f7,,sys} module exposes details specific to the
implementation of the currently running interpreter.  The initial set of
attributes on @ref{bff,,sys.implementation} are @code{name}, @code{version},
@code{hexversion}, and @code{cache_tag}.

The intention of @code{sys.implementation} is to consolidate into one namespace
the implementation-specific data used by the standard library.  This allows
different Python implementations to share a single standard library code base
much more easily.  In its initial state, @code{sys.implementation} holds only a
small portion of the implementation-specific data.  Over time that ratio will
shift in order to make the standard library more portable.

One example of improved standard library portability is @code{cache_tag}.  As of
Python 3.3, @code{sys.implementation.cache_tag} is used by @ref{96,,importlib} to
support 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} compliance.  Any Python implementation that uses
@code{importlib} for its built-in import system may use @code{cache_tag} to control
the caching behavior for modules.

@menu
* SimpleNamespace:: 

@end menu

@node SimpleNamespace,,,PEP 421 Adding sys implementation
@anchor{whatsnew/3 3 simplenamespace}@anchor{c00}
@subsubsection SimpleNamespace


The implementation of @code{sys.implementation} also introduces a new type to
Python: @ref{41b,,types.SimpleNamespace}.  In contrast to a mapping-based
namespace, like @ref{3b0,,dict}, @code{SimpleNamespace} is attribute-based, like
@ref{5a5,,object}.  However, unlike @code{object}, @code{SimpleNamespace} instances
are writable.  This means that you can add, remove, and modify the namespace
through normal attribute access.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 421

@item PEP 421@footnote{https://www.python.org/dev/peps/pep-0421} - Adding sys.implementation

PEP written and implemented by Eric Snow.
@end table


@node Using importlib as the Implementation of Import,Other Language Changes<9>,PEP 421 Adding sys implementation,What’s New In Python 3 3
@anchor{whatsnew/3 3 importlib}@anchor{bd1}@anchor{whatsnew/3 3 using-importlib-as-the-implementation-of-import}@anchor{c01}
@subsection Using importlib as the Implementation of Import


bpo-2377@footnote{https://bugs.python.org/issue2377} - Replace __import__ w/ importlib.__import__
bpo-13959@footnote{https://bugs.python.org/issue13959} - Re-implement parts of @ref{95,,imp} in pure Python
bpo-14605@footnote{https://bugs.python.org/issue14605} - Make import machinery explicit
bpo-14646@footnote{https://bugs.python.org/issue14646} - Require loaders set __loader__ and __package__

The @ref{3b7,,__import__()} function is now powered by @ref{c02,,importlib.__import__()}.
This work leads to the completion of “phase 2” of 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302}. There are
multiple benefits to this change. First, it has allowed for more of the
machinery powering import to be exposed instead of being implicit and hidden
within the C code. It also provides a single implementation for all Python VMs
supporting Python 3.3 to use, helping to end any VM-specific deviations in
import semantics. And finally it eases the maintenance of import, allowing for
future growth to occur.

For the common user, there should be no visible change in semantics.  For
those whose code currently manipulates import or calls import
programmatically, the code changes that might possibly be required are covered
in the @ref{c03,,Porting Python code} section of this document.

@menu
* New APIs:: 
* Visible Changes:: 

@end menu

@node New APIs,Visible Changes,,Using importlib as the Implementation of Import
@anchor{whatsnew/3 3 new-apis}@anchor{c04}
@subsubsection New APIs


One of the large benefits of this work is the exposure of what goes into
making the import statement work. That means the various importers that were
once implicit are now fully exposed as part of the @ref{96,,importlib} package.

The abstract base classes defined in @ref{97,,importlib.abc} have been expanded
to properly delineate between @ref{c05,,meta path finders}
and @ref{c06,,path entry finders} by introducing
@ref{30e,,importlib.abc.MetaPathFinder} and
@ref{30f,,importlib.abc.PathEntryFinder}, respectively. The old ABC of
@ref{30d,,importlib.abc.Finder} is now only provided for backwards-compatibility
and does not enforce any method requirements.

In terms of finders, @ref{c07,,importlib.machinery.FileFinder} exposes the
mechanism used to search for source and bytecode files of a module. Previously
this class was an implicit member of @ref{bb5,,sys.path_hooks}.

For loaders, the new abstract base class @ref{c08,,importlib.abc.FileLoader} helps
write a loader that uses the file system as the storage mechanism for a module’s
code. The loader for source files
(@ref{c09,,importlib.machinery.SourceFileLoader}), sourceless bytecode files
(@ref{c0a,,importlib.machinery.SourcelessFileLoader}), and extension modules
(@ref{7f8,,importlib.machinery.ExtensionFileLoader}) are now available for
direct use.

@ref{3b8,,ImportError} now has @code{name} and @code{path} attributes which are set when
there is relevant data to provide. The message for failed imports will also
provide the full name of the module now instead of just the tail end of the
module’s name.

The @ref{74b,,importlib.invalidate_caches()} function will now call the method with
the same name on all finders cached in @ref{74c,,sys.path_importer_cache} to help
clean up any stored state as necessary.

@node Visible Changes,,New APIs,Using importlib as the Implementation of Import
@anchor{whatsnew/3 3 visible-changes}@anchor{c0b}
@subsubsection Visible Changes


For potential required changes to code, see the @ref{c03,,Porting Python code}
section.

Beyond the expanse of what @ref{96,,importlib} now exposes, there are other
visible changes to import. The biggest is that @ref{87c,,sys.meta_path} and
@ref{bb5,,sys.path_hooks} now store all of the meta path finders and path entry
hooks used by import.  Previously the finders were implicit and hidden within
the C code of import instead of being directly exposed. This means that one can
now easily remove or change the order of the various finders to fit one’s needs.

Another change is that all modules have a @code{__loader__} attribute, storing the
loader used to create the module. 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} has been updated to make this
attribute mandatory for loaders to implement, so in the future once 3rd-party
loaders have been updated people will be able to rely on the existence of the
attribute. Until such time, though, import is setting the module post-load.

Loaders are also now expected to set the @code{__package__} attribute from
@geindex Python Enhancement Proposals; PEP 366
PEP 366@footnote{https://www.python.org/dev/peps/pep-0366}. Once again, import itself is already setting this on all loaders
from @ref{96,,importlib} and import itself is setting the attribute post-load.

@code{None} is now inserted into @ref{74c,,sys.path_importer_cache} when no finder
can be found on @ref{bb5,,sys.path_hooks}. Since @ref{c0c,,imp.NullImporter} is not
directly exposed on @ref{bb5,,sys.path_hooks} it could no longer be relied upon to
always be available to use as a value representing no finder found.

All other changes relate to semantic changes which should be taken into
consideration when updating code for Python 3.3, and thus should be read about
in the @ref{c03,,Porting Python code} section of this document.

(Implementation by Brett Cannon)

@node Other Language Changes<9>,A Finer-Grained Import Lock,Using importlib as the Implementation of Import,What’s New In Python 3 3
@anchor{whatsnew/3 3 other-language-changes}@anchor{c0d}
@subsection Other Language Changes


Some smaller changes made to the core Python language are:


@itemize *

@item 
Added support for Unicode name aliases and named sequences.
Both @ref{c0e,,unicodedata.lookup()} and @code{'\N@{...@}'} now resolve name aliases,
and @ref{c0e,,unicodedata.lookup()} resolves named sequences too.

(Contributed by Ezio Melotti in bpo-12753@footnote{https://bugs.python.org/issue12753}.)

@item 
Unicode database updated to UCD version 6.1.0

@item 
Equality comparisons on @ref{438,,range()} objects now return a result reflecting
the equality of the underlying sequences generated by those range objects.
(bpo-13201@footnote{https://bugs.python.org/issue13201})

@item 
The @code{count()}, @code{find()}, @code{rfind()}, @code{index()} and @code{rindex()}
methods of @ref{2f0,,bytes} and @ref{2f1,,bytearray} objects now accept an
integer between 0 and 255 as their first argument.

(Contributed by Petri Lehtinen in bpo-12170@footnote{https://bugs.python.org/issue12170}.)

@item 
The @code{rjust()}, @code{ljust()}, and @code{center()} methods of @ref{2f0,,bytes}
and @ref{2f1,,bytearray} now accept a @ref{2f1,,bytearray} for the @code{fill}
argument.  (Contributed by Petri Lehtinen in bpo-12380@footnote{https://bugs.python.org/issue12380}.)

@item 
New methods have been added to @ref{43a,,list} and @ref{2f1,,bytearray}:
@code{copy()} and @code{clear()} (bpo-10516@footnote{https://bugs.python.org/issue10516}).  Consequently,
@ref{933,,MutableSequence} now also defines a
@code{clear()} method (bpo-11388@footnote{https://bugs.python.org/issue11388}).

@item 
Raw bytes literals can now be written @code{rb"..."} as well as @code{br"..."}.

(Contributed by Antoine Pitrou in bpo-13748@footnote{https://bugs.python.org/issue13748}.)

@item 
@ref{c0f,,dict.setdefault()} now does only one lookup for the given key, making
it atomic when used with built-in types.

(Contributed by Filip Gruszczyński in bpo-13521@footnote{https://bugs.python.org/issue13521}.)

@item 
The error messages produced when a function call does not match the function
signature have been significantly improved.

(Contributed by Benjamin Peterson.)
@end itemize

@node A Finer-Grained Import Lock,Builtin functions and types,Other Language Changes<9>,What’s New In Python 3 3
@anchor{whatsnew/3 3 a-finer-grained-import-lock}@anchor{c10}
@subsection A Finer-Grained Import Lock


Previous versions of CPython have always relied on a global import lock.
This led to unexpected annoyances, such as deadlocks when importing a module
would trigger code execution in a different thread as a side-effect.
Clumsy workarounds were sometimes employed, such as the
@ref{c11,,PyImport_ImportModuleNoBlock()} C API function.

In Python 3.3, importing a module takes a per-module lock.  This correctly
serializes importation of a given module from multiple threads (preventing
the exposure of incompletely initialized modules), while eliminating the
aforementioned annoyances.

(Contributed by Antoine Pitrou in bpo-9260@footnote{https://bugs.python.org/issue9260}.)

@node Builtin functions and types,New Modules<9>,A Finer-Grained Import Lock,What’s New In Python 3 3
@anchor{whatsnew/3 3 builtin-functions-and-types}@anchor{c12}
@subsection Builtin functions and types



@itemize *

@item 
@ref{1c8,,open()} gets a new @emph{opener} parameter: the underlying file descriptor
for the file object is then obtained by calling @emph{opener} with (@emph{file},
@emph{flags}). It can be used to use custom flags like @ref{c13,,os.O_CLOEXEC} for
example. The @code{'x'} mode was added: open for exclusive creation, failing if
the file already exists.

@item 
@ref{af6,,print()}: added the @emph{flush} keyword argument. If the @emph{flush} keyword
argument is true, the stream is forcibly flushed.

@item 
@ref{c14,,hash()}: hash randomization is enabled by default, see
@ref{619,,object.__hash__()} and 
@geindex PYTHONHASHSEED
@geindex environment variable; PYTHONHASHSEED
@ref{c15,,PYTHONHASHSEED}.

@item 
The @ref{2ef,,str} type gets a new @ref{935,,casefold()} method: return a
casefolded copy of the string, casefolded strings may be used for caseless
matching. For example, @code{'ß'.casefold()} returns @code{'ss'}.

@item 
The sequence documentation has been substantially rewritten to better
explain the binary/text sequence distinction and to provide specific
documentation sections for the individual builtin sequence types
(bpo-4966@footnote{https://bugs.python.org/issue4966}).
@end itemize

@node New Modules<9>,Improved Modules<9>,Builtin functions and types,What’s New In Python 3 3
@anchor{whatsnew/3 3 new-modules}@anchor{c16}
@subsection New Modules


@menu
* faulthandler: faulthandler<4>. 
* ipaddress: ipaddress<5>. 
* lzma: lzma<2>. 

@end menu

@node faulthandler<4>,ipaddress<5>,,New Modules<9>
@anchor{whatsnew/3 3 faulthandler}@anchor{c17}
@subsubsection faulthandler


This new debug module @ref{78,,faulthandler} contains functions to dump Python tracebacks explicitly,
on a fault (a crash like a segmentation fault), after a timeout, or on a user
signal. Call @ref{7ea,,faulthandler.enable()} to install fault handlers for the
@code{SIGSEGV}, @code{SIGFPE}, @code{SIGABRT}, @code{SIGBUS}, and
@code{SIGILL} signals. You can also enable them at startup by setting the
@geindex PYTHONFAULTHANDLER
@geindex environment variable; PYTHONFAULTHANDLER
@ref{c18,,PYTHONFAULTHANDLER} environment variable or by using @ref{159,,-X}
@code{faulthandler} command line option.

Example of a segmentation fault on Linux:

@example
$ python -q -X faulthandler
>>> import ctypes
>>> ctypes.string_at(0)
Fatal Python error: Segmentation fault

Current thread 0x00007fb899f39700:
  File "/home/python/cpython/Lib/ctypes/__init__.py", line 486 in string_at
  File "<stdin>", line 1 in <module>
Segmentation fault
@end example

@node ipaddress<5>,lzma<2>,faulthandler<4>,New Modules<9>
@anchor{whatsnew/3 3 ipaddress}@anchor{c19}
@subsubsection ipaddress


The new @ref{9e,,ipaddress} module provides tools for creating and manipulating
objects representing IPv4 and IPv6 addresses, networks and interfaces (i.e.
an IP address associated with a specific IP subnet).

(Contributed by Google and Peter Moody in 
@geindex Python Enhancement Proposals; PEP 3144
PEP 3144@footnote{https://www.python.org/dev/peps/pep-3144}.)

@node lzma<2>,,ipaddress<5>,New Modules<9>
@anchor{whatsnew/3 3 lzma}@anchor{c1a}
@subsubsection lzma


The newly-added @ref{a9,,lzma} module provides data compression and decompression
using the LZMA algorithm, including support for the @code{.xz} and @code{.lzma}
file formats.

(Contributed by Nadeem Vawda and Per Øyvind Karlsen in bpo-6715@footnote{https://bugs.python.org/issue6715}.)

@node Improved Modules<9>,Optimizations<8>,New Modules<9>,What’s New In Python 3 3
@anchor{whatsnew/3 3 improved-modules}@anchor{c1b}
@subsection Improved Modules


@menu
* abc: abc<2>. 
* array: array<3>. 
* base64: base64<3>. 
* binascii: binascii<3>. 
* bz2: bz2<2>. 
* codecs: codecs<2>. 
* collections: collections<7>. 
* contextlib: contextlib<6>. 
* crypt: crypt<2>. 
* curses: curses<5>. 
* datetime: datetime<5>. 
* decimal: decimal<3>. 
* email: email<4>. 
* ftplib: ftplib<2>. 
* functools: functools<5>. 
* gc: gc<6>. 
* hmac: hmac<4>. 
* http: http<4>. 
* html: html<2>. 
* imaplib: imaplib<3>. 
* inspect: inspect<8>. 
* io: io<4>. 
* itertools: itertools<3>. 
* logging: logging<6>. 
* math: math<7>. 
* mmap: mmap<3>. 
* multiprocessing: multiprocessing<7>. 
* nntplib: nntplib<2>. 
* os: os<10>. 
* pdb: pdb<5>. 
* pickle: pickle<5>. 
* pydoc: pydoc<5>. 
* re: re<6>. 
* sched:: 
* select: select<2>. 
* shlex: shlex<3>. 
* shutil: shutil<4>. 
* signal: signal<4>. 
* smtpd: smtpd<3>. 
* smtplib: smtplib<4>. 
* socket: socket<10>. 
* socketserver: socketserver<3>. 
* sqlite3: sqlite3<7>. 
* ssl: ssl<9>. 
* stat: stat<2>. 
* struct: struct<3>. 
* subprocess: subprocess<5>. 
* sys: sys<10>. 
* tarfile: tarfile<4>. 
* tempfile:: 
* textwrap: textwrap<2>. 
* threading: threading<7>. 
* time: time<7>. 
* types: types<5>. 
* unittest: unittest<6>. 
* urllib: urllib<3>. 
* webbrowser:: 
* xml.etree.ElementTree: xml etree ElementTree. 
* zlib: zlib<2>. 

@end menu

@node abc<2>,array<3>,,Improved Modules<9>
@anchor{whatsnew/3 3 abc}@anchor{c1c}
@subsubsection abc


Improved support for abstract base classes containing descriptors composed with
abstract methods. The recommended approach to declaring abstract descriptors is
now to provide @code{__isabstractmethod__} as a dynamically updated
property. The built-in descriptors have been updated accordingly.

@quotation


@itemize *

@item 
@ref{c1d,,abc.abstractproperty} has been deprecated, use @ref{501,,property}
with @ref{c1e,,abc.abstractmethod()} instead.

@item 
@ref{c1f,,abc.abstractclassmethod} has been deprecated, use
@ref{254,,classmethod} with @ref{c1e,,abc.abstractmethod()} instead.

@item 
@ref{c20,,abc.abstractstaticmethod} has been deprecated, use
@ref{253,,staticmethod} with @ref{c1e,,abc.abstractmethod()} instead.
@end itemize
@end quotation

(Contributed by Darren Dale in bpo-11610@footnote{https://bugs.python.org/issue11610}.)

@ref{c21,,abc.ABCMeta.register()} now returns the registered subclass, which means
it can now be used as a class decorator (bpo-10868@footnote{https://bugs.python.org/issue10868}).

@node array<3>,base64<3>,abc<2>,Improved Modules<9>
@anchor{whatsnew/3 3 array}@anchor{c22}
@subsubsection array


The @ref{6,,array} module supports the long  long type using @code{q} and
@code{Q} type codes.

(Contributed by Oren Tirosh and Hirokazu Yamamoto in bpo-1172711@footnote{https://bugs.python.org/issue1172711}.)

@node base64<3>,binascii<3>,array<3>,Improved Modules<9>
@anchor{whatsnew/3 3 base64}@anchor{c23}
@subsubsection base64


ASCII-only Unicode strings are now accepted by the decoding functions of the
@ref{d,,base64} modern interface. For example, @code{base64.b64decode('YWJj')}
returns @code{b'abc'}.  (Contributed by Catalin Iacob in bpo-13641@footnote{https://bugs.python.org/issue13641}.)

@node binascii<3>,bz2<2>,base64<3>,Improved Modules<9>
@anchor{whatsnew/3 3 binascii}@anchor{c24}
@subsubsection binascii


In addition to the binary objects they normally accept, the @code{a2b_} functions
now all also accept ASCII-only strings as input.  (Contributed by Antoine
Pitrou in bpo-13637@footnote{https://bugs.python.org/issue13637}.)

@node bz2<2>,codecs<2>,binascii<3>,Improved Modules<9>
@anchor{whatsnew/3 3 bz2}@anchor{c25}
@subsubsection bz2


The @ref{12,,bz2} module has been rewritten from scratch. In the process, several
new features have been added:


@itemize *

@item 
New @ref{c26,,bz2.open()} function: open a bzip2-compressed file in binary or
text mode.

@item 
@ref{455,,bz2.BZ2File} can now read from and write to arbitrary file-like
objects, by means of its constructor’s @emph{fileobj} argument.

(Contributed by Nadeem Vawda in bpo-5863@footnote{https://bugs.python.org/issue5863}.)

@item 
@ref{455,,bz2.BZ2File} and @ref{c27,,bz2.decompress()} can now decompress
multi-stream inputs (such as those produced by the @code{pbzip2} tool).
@ref{455,,bz2.BZ2File} can now also be used to create this type of file, using
the @code{'a'} (append) mode.

(Contributed by Nir Aides in bpo-1625@footnote{https://bugs.python.org/issue1625}.)

@item 
@ref{455,,bz2.BZ2File} now implements all of the @ref{823,,io.BufferedIOBase} API,
except for the @code{detach()} and @code{truncate()} methods.
@end itemize

@node codecs<2>,collections<7>,bz2<2>,Improved Modules<9>
@anchor{whatsnew/3 3 codecs}@anchor{c28}
@subsubsection codecs


The @ref{73,,mbcs} codec has been rewritten to handle correctly
@code{replace} and @code{ignore} error handlers on all Windows versions.  The
@ref{73,,mbcs} codec now supports all error handlers, instead of only
@code{replace} to encode and @code{ignore} to decode.

A new Windows-only codec has been added: @code{cp65001} (bpo-13216@footnote{https://bugs.python.org/issue13216}). It is the
Windows code page 65001 (Windows UTF-8, @code{CP_UTF8}).  For example, it is used
by @code{sys.stdout} if the console output code page is set to cp65001 (e.g., using
@code{chcp 65001} command).

Multibyte CJK decoders now resynchronize faster.  They only ignore the first
byte of an invalid byte sequence. For example, @code{b'\xff\n'.decode('gb2312',
'replace')} now returns a @code{\n} after the replacement character.

(bpo-12016@footnote{https://bugs.python.org/issue12016})

Incremental CJK codec encoders are no longer reset at each call to their
encode() methods. For example:

@example
>>> import codecs
>>> encoder = codecs.getincrementalencoder('hz')('strict')
>>> b'@w{'}.join(encoder.encode(x) for x in '\u52ff\u65bd\u65bc\u4eba\u3002 Bye.')
b'~@{NpJ)l6HK!#~@} Bye.'
@end example

This example gives @code{b'~@{Np~@}~@{J)~@}~@{l6~@}~@{HK~@}~@{!#~@} Bye.'} with older Python
versions.

(bpo-12100@footnote{https://bugs.python.org/issue12100})

The @code{unicode_internal} codec has been deprecated.

@node collections<7>,contextlib<6>,codecs<2>,Improved Modules<9>
@anchor{whatsnew/3 3 collections}@anchor{c29}
@subsubsection collections


Addition of a new @ref{758,,ChainMap} class to allow treating a
number of mappings as a single unit.  (Written by Raymond Hettinger for
bpo-11089@footnote{https://bugs.python.org/issue11089}, made public in bpo-11297@footnote{https://bugs.python.org/issue11297}.)

The abstract base classes have been moved in a new @ref{1d,,collections.abc}
module, to better differentiate between the abstract and the concrete
collections classes.  Aliases for ABCs are still present in the
@ref{1c,,collections} module to preserve existing imports.  (bpo-11085@footnote{https://bugs.python.org/issue11085})

@c XXX addition of __slots__ to ABCs not recorded here: internal detail

The @ref{c2a,,Counter} class now supports the unary @code{+} and @code{-}
operators, as well as the in-place operators @code{+=}, @code{-=}, @code{|=}, and
@code{&=}.  (Contributed by Raymond Hettinger in bpo-13121@footnote{https://bugs.python.org/issue13121}.)

@node contextlib<6>,crypt<2>,collections<7>,Improved Modules<9>
@anchor{whatsnew/3 3 contextlib}@anchor{c2b}
@subsubsection contextlib


@ref{648,,ExitStack} now provides a solid foundation for
programmatic manipulation of context managers and similar cleanup
functionality. Unlike the previous @code{contextlib.nested} API (which was
deprecated and removed), the new API is designed to work correctly
regardless of whether context managers acquire their resources in
their @code{__init__} method (for example, file objects) or in their
@code{__enter__} method (for example, synchronisation objects from the
@ref{109,,threading} module).

(bpo-13585@footnote{https://bugs.python.org/issue13585})

@node crypt<2>,curses<5>,contextlib<6>,Improved Modules<9>
@anchor{whatsnew/3 3 crypt}@anchor{c2c}
@subsubsection crypt


Addition of salt and modular crypt format (hashing method) and the @ref{64e,,mksalt()}
function to the @ref{27,,crypt} module.

(bpo-10924@footnote{https://bugs.python.org/issue10924})

@node curses<5>,datetime<5>,crypt<2>,Improved Modules<9>
@anchor{whatsnew/3 3 curses}@anchor{c2d}
@subsubsection curses


@quotation


@itemize *

@item 
If the @ref{2a,,curses} module is linked to the ncursesw library, use Unicode
functions when Unicode strings or characters are passed (e.g.
@code{waddwstr()}), and bytes functions otherwise (e.g. @code{waddstr()}).

@item 
Use the locale encoding instead of @code{utf-8} to encode Unicode strings.

@item 
@code{curses.window} has a new @ref{c2e,,curses.window.encoding} attribute.

@item 
The @code{curses.window} class has a new @ref{c2f,,get_wch()}
method to get a wide character

@item 
The @ref{2a,,curses} module has a new @ref{c30,,unget_wch()} function to
push a wide character so the next @ref{c2f,,get_wch()} will return
it
@end itemize
@end quotation

(Contributed by Iñigo Serna in bpo-6755@footnote{https://bugs.python.org/issue6755}.)

@node datetime<5>,decimal<3>,curses<5>,Improved Modules<9>
@anchor{whatsnew/3 3 datetime}@anchor{c31}
@subsubsection datetime


@quotation


@itemize *

@item 
Equality comparisons between naive and aware @ref{363,,datetime}
instances now return @ref{659,,False} instead of raising @ref{162,,TypeError}
(bpo-15006@footnote{https://bugs.python.org/issue15006}).

@item 
New @ref{c32,,datetime.datetime.timestamp()} method: Return POSIX timestamp
corresponding to the @ref{363,,datetime} instance.

@item 
The @ref{7d9,,datetime.datetime.strftime()} method supports formatting years
older than 1000.

@item 
The @ref{4c9,,datetime.datetime.astimezone()} method can now be
called without arguments to convert datetime instance to the system
timezone.
@end itemize
@end quotation

@node decimal<3>,email<4>,datetime<5>,Improved Modules<9>
@anchor{whatsnew/3 3 decimal}@anchor{c33}@anchor{whatsnew/3 3 new-decimal}@anchor{bd4}
@subsubsection decimal



@table @asis

@item bpo-7652@footnote{https://bugs.python.org/issue7652} - integrate fast native decimal arithmetic.

C-module and libmpdec written by Stefan Krah.
@end table

The new C version of the decimal module integrates the high speed libmpdec
library for arbitrary precision correctly-rounded decimal floating point
arithmetic. libmpdec conforms to IBM’s General Decimal Arithmetic Specification.

Performance gains range from 10x for database applications to 100x for
numerically intensive applications. These numbers are expected gains
for standard precisions used in decimal floating point arithmetic. Since
the precision is user configurable, the exact figures may vary. For example,
in integer bignum arithmetic the differences can be significantly higher.

The following table is meant as an illustration. Benchmarks are available
at @indicateurl{http://www.bytereef.org/mpdecimal/quickstart.html}.

@quotation


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxx} 
@headitem

@tab

decimal.py

@tab

_decimal

@tab

speedup

@item

pi

@tab

42.02s

@tab

0.345s

@tab

120x

@item

telco

@tab

172.19s

@tab

5.68s

@tab

30x

@item

psycopg

@tab

3.57s

@tab

0.29s

@tab

12x

@end multitable

@end quotation

@menu
* Features: Features<2>. 
* API changes: API changes<2>. 

@end menu

@node Features<2>,API changes<2>,,decimal<3>
@anchor{whatsnew/3 3 id1}@anchor{c34}
@subsubsection Features



@itemize *

@item 
The @ref{c35,,FloatOperation} signal optionally enables stricter
semantics for mixing floats and Decimals.

@item 
If Python is compiled without threads, the C version automatically
disables the expensive thread local context machinery. In this case,
the variable @ref{c36,,HAVE_THREADS} is set to @code{False}.
@end itemize

@node API changes<2>,,Features<2>,decimal<3>
@anchor{whatsnew/3 3 id2}@anchor{c37}
@subsubsection API changes



@itemize *

@item 
The C module has the following context limits, depending on the machine
architecture:

@quotation


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

@tab

32-bit

@tab

64-bit

@item

@code{MAX_PREC}

@tab

@code{425000000}

@tab

@code{999999999999999999}

@item

@code{MAX_EMAX}

@tab

@code{425000000}

@tab

@code{999999999999999999}

@item

@code{MIN_EMIN}

@tab

@code{-425000000}

@tab

@code{-999999999999999999}

@end multitable

@end quotation

@item 
In the context templates (@ref{c38,,DefaultContext},
@ref{c39,,BasicContext} and @ref{c3a,,ExtendedContext})
the magnitude of @code{Emax} and
@code{Emin} has changed to @code{999999}.

@item 
The @ref{248,,Decimal} constructor in decimal.py does not observe
the context limits and converts values with arbitrary exponents or precision
exactly. Since the C version has internal limits, the following scheme is
used: If possible, values are converted exactly, otherwise
@ref{c3b,,InvalidOperation} is raised and the result is NaN. In the
latter case it is always possible to use @ref{c3c,,create_decimal()}
in order to obtain a rounded or inexact value.

@item 
The power function in decimal.py is always correctly-rounded. In the
C version, it is defined in terms of the correctly-rounded
@ref{c3d,,exp()} and @ref{c3e,,ln()} functions,
but the final result is only “almost always correctly rounded”.

@item 
In the C version, the context dictionary containing the signals is a
@ref{c3f,,MutableMapping}.  For speed reasons,
@code{flags} and @code{traps} always
refer to the same @ref{c3f,,MutableMapping} that the context
was initialized with. If a new signal dictionary is assigned,
@code{flags} and @code{traps}
are updated with the new values, but they do not reference the RHS
dictionary.

@item 
Pickling a @ref{c40,,Context} produces a different output in order
to have a common interchange format for the Python and C versions.

@item 
The order of arguments in the @ref{c40,,Context} constructor has been
changed to match the order displayed by @ref{27d,,repr()}.

@item 
The @code{watchexp} parameter in the @ref{c41,,quantize()} method
is deprecated.
@end itemize

@node email<4>,ftplib<2>,decimal<3>,Improved Modules<9>
@anchor{whatsnew/3 3 email}@anchor{c42}@anchor{whatsnew/3 3 new-email}@anchor{bd5}
@subsubsection email


@menu
* Policy Framework:: 
* Provisional Policy with New Header API:: 
* Other API Changes:: 

@end menu

@node Policy Framework,Provisional Policy with New Header API,,email<4>
@anchor{whatsnew/3 3 policy-framework}@anchor{c43}
@subsubsection Policy Framework


The email package now has a @ref{70,,policy} framework.  A
@ref{c44,,Policy} is an object with several methods and properties
that control how the email package behaves.  The primary policy for Python 3.3
is the @ref{c45,,Compat32} policy, which provides backward
compatibility with the email package in Python 3.2.  A @code{policy} can be
specified when an email message is parsed by a @ref{6f,,parser}, or when a
@ref{7e3,,Message} object is created, or when an email is
serialized using a @ref{69,,generator}.  Unless overridden, a policy passed
to a @code{parser} is inherited by all the @code{Message} object and sub-objects
created by the @code{parser}.  By default a @code{generator} will use the policy of
the @code{Message} object it is serializing.  The default policy is
@ref{7e2,,compat32}.

The minimum set of controls implemented by all @code{policy} objects are:

@quotation


@multitable {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

max_line_length

@tab

The maximum length, excluding the linesep character(s),
individual lines may have when a @code{Message} is
serialized.  Defaults to 78.

@item

linesep

@tab

The character used to separate individual lines when a
@code{Message} is serialized.  Defaults to @code{\n}.

@item

cte_type

@tab

@code{7bit} or @code{8bit}.  @code{8bit} applies only to a
@code{Bytes} @code{generator}, and means that non-ASCII may
be used where allowed by the protocol (or where it
exists in the original input).

@item

raise_on_defect

@tab

Causes a @code{parser} to raise error when defects are
encountered instead of adding them to the @code{Message}
object’s @code{defects} list.

@end multitable

@end quotation

A new policy instance, with new settings, is created using the
@ref{c46,,clone()} method of policy objects.  @code{clone} takes
any of the above controls as keyword arguments.  Any control not specified in
the call retains its default value.  Thus you can create a policy that uses
@code{\r\n} linesep characters like this:

@example
mypolicy = compat32.clone(linesep='\r\n')
@end example

Policies can be used to make the generation of messages in the format needed by
your application simpler.  Instead of having to remember to specify
@code{linesep='\r\n'} in all the places you call a @code{generator}, you can specify
it once, when you set the policy used by the @code{parser} or the @code{Message},
whichever your program uses to create @code{Message} objects.  On the other hand,
if you need to generate messages in multiple forms, you can still specify the
parameters in the appropriate @code{generator} call.  Or you can have custom
policy instances for your different cases, and pass those in when you create
the @code{generator}.

@node Provisional Policy with New Header API,Other API Changes,Policy Framework,email<4>
@anchor{whatsnew/3 3 provisional-policy-with-new-header-api}@anchor{c47}
@subsubsection Provisional Policy with New Header API


While the policy framework is worthwhile all by itself, the main motivation for
introducing it is to allow the creation of new policies that implement new
features for the email package in a way that maintains backward compatibility
for those who do not use the new policies.  Because the new policies introduce a
new API, we are releasing them in Python 3.3 as a @ref{bd6,,provisional policy}.  Backwards incompatible changes (up to and including
removal of the code) may occur if deemed necessary by the core developers.

The new policies are instances of @ref{c48,,EmailPolicy},
and add the following additional controls:

@quotation


@multitable {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

refold_source

@tab

Controls whether or not headers parsed by a
@ref{6f,,parser} are refolded by the
@ref{69,,generator}.  It can be @code{none}, @code{long},
or @code{all}.  The default is @code{long}, which means that
source headers with a line longer than
@code{max_line_length} get refolded.  @code{none} means no
line get refolded, and @code{all} means that all lines
get refolded.

@item

header_factory

@tab

A callable that take a @code{name} and @code{value} and
produces a custom header object.

@end multitable

@end quotation

The @code{header_factory} is the key to the new features provided by the new
policies.  When one of the new policies is used, any header retrieved from
a @code{Message} object is an object produced by the @code{header_factory}, and any
time you set a header on a @code{Message} it becomes an object produced by
@code{header_factory}.  All such header objects have a @code{name} attribute equal
to the header name.  Address and Date headers have additional attributes
that give you access to the parsed data of the header.  This means you can now
do things like this:

@example
>>> m = Message(policy=SMTP)
>>> m['To'] = 'Éric <foo@@example.com>'
>>> m['to']
'Éric <foo@@example.com>'
>>> m['to'].addresses
(Address(display_name='Éric', username='foo', domain='example.com'),)
>>> m['to'].addresses[0].username
'foo'
>>> m['to'].addresses[0].display_name
'Éric'
>>> m['Date'] = email.utils.localtime()
>>> m['Date'].datetime
datetime.datetime(2012, 5, 25, 21, 39, 24, 465484, tzinfo=datetime.timezone(datetime.timedelta(-1, 72000), 'EDT'))
>>> m['Date']
'Fri, 25 May 2012 21:44:27 -0400'
>>> print(m)
To: =?utf-8?q?=C3=89ric?= <foo@@example.com>
Date: Fri, 25 May 2012 21:44:27 -0400
@end example

You will note that the unicode display name is automatically encoded as
@code{utf-8} when the message is serialized, but that when the header is accessed
directly, you get the unicode version.  This eliminates any need to deal with
the @ref{6a,,email.header} @ref{c49,,decode_header()} or
@ref{c4a,,make_header()} functions.

You can also create addresses from parts:

@example
>>> m['cc'] = [Group('pals', [Address('Bob', 'bob', 'example.com'),
...                           Address('Sally', 'sally', 'example.com')]),
...            Address('Bonzo', addr_spec='bonz@@laugh.com')]
>>> print(m)
To: =?utf-8?q?=C3=89ric?= <foo@@example.com>
Date: Fri, 25 May 2012 21:44:27 -0400
cc: pals: Bob <bob@@example.com>, Sally <sally@@example.com>;, Bonzo <bonz@@laugh.com>
@end example

Decoding to unicode is done automatically:

@example
>>> m2 = message_from_string(str(m))
>>> m2['to']
'Éric <foo@@example.com>'
@end example

When you parse a message, you can use the @code{addresses} and @code{groups}
attributes of the header objects to access the groups and individual
addresses:

@example
>>> m2['cc'].addresses
(Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com'), Address(display_name='Bonzo', username='bonz', domain='laugh.com'))
>>> m2['cc'].groups
(Group(display_name='pals', addresses=(Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com')), Group(display_name=None, addresses=(Address(display_name='Bonzo', username='bonz', domain='laugh.com'),))
@end example

In summary, if you use one of the new policies, header manipulation works the
way it ought to:  your application works with unicode strings, and the email
package transparently encodes and decodes the unicode to and from the RFC
standard Content Transfer Encodings.

@node Other API Changes,,Provisional Policy with New Header API,email<4>
@anchor{whatsnew/3 3 other-api-changes}@anchor{c4b}
@subsubsection Other API Changes


New @ref{c4c,,BytesHeaderParser}, added to the @ref{6f,,parser}
module to complement @ref{c4d,,HeaderParser} and complete the Bytes
API.

New utility functions:

@quotation


@itemize *

@item 
@ref{c4e,,format_datetime()}: given a @ref{363,,datetime},
produce a string formatted for use in an email header.

@item 
@ref{c4f,,parsedate_to_datetime()}: given a date string from
an email header, convert it into an aware @ref{363,,datetime},
or a naive @ref{363,,datetime} if the offset is @code{-0000}.

@item 
@ref{c50,,localtime()}: With no argument, returns the
current local time as an aware @ref{363,,datetime} using the local
@ref{c51,,timezone}.  Given an aware @ref{363,,datetime},
converts it into an aware @ref{363,,datetime} using the
local @ref{c51,,timezone}.
@end itemize
@end quotation

@node ftplib<2>,functools<5>,email<4>,Improved Modules<9>
@anchor{whatsnew/3 3 ftplib}@anchor{c52}
@subsubsection ftplib



@itemize *

@item 
@ref{3e9,,ftplib.FTP} now accepts a @code{source_address} keyword argument to
specify the @code{(host, port)} to use as the source address in the bind call
when creating the outgoing socket.  (Contributed by Giampaolo Rodolà
in bpo-8594@footnote{https://bugs.python.org/issue8594}.)

@item 
The @ref{3ea,,FTP_TLS} class now provides a new
@ref{c53,,ccc()} function to revert control channel back to
plaintext.  This can be useful to take advantage of firewalls that know how
to handle NAT with non-secure FTP without opening fixed ports.  (Contributed
by Giampaolo Rodolà in bpo-12139@footnote{https://bugs.python.org/issue12139}.)

@item 
Added @ref{c54,,ftplib.FTP.mlsd()} method which provides a parsable directory
listing format and deprecates @ref{c55,,ftplib.FTP.nlst()} and
@ref{c56,,ftplib.FTP.dir()}.  (Contributed by Giampaolo Rodolà in bpo-11072@footnote{https://bugs.python.org/issue11072}.)
@end itemize

@node functools<5>,gc<6>,ftplib<2>,Improved Modules<9>
@anchor{whatsnew/3 3 functools}@anchor{c57}
@subsubsection functools


The @ref{4f3,,functools.lru_cache()} decorator now accepts a @code{typed} keyword
argument (that defaults to @code{False} to ensure that it caches values of
different types that compare equal in separate cache slots.  (Contributed
by Raymond Hettinger in bpo-13227@footnote{https://bugs.python.org/issue13227}.)

@node gc<6>,hmac<4>,functools<5>,Improved Modules<9>
@anchor{whatsnew/3 3 gc}@anchor{c58}
@subsubsection gc


It is now possible to register callbacks invoked by the garbage collector
before and after collection using the new @ref{c59,,callbacks} list.

@node hmac<4>,http<4>,gc<6>,Improved Modules<9>
@anchor{whatsnew/3 3 hmac}@anchor{c5a}
@subsubsection hmac


A new @ref{c5b,,compare_digest()} function has been added to prevent side
channel attacks on digests through timing analysis.  (Contributed by Nick
Coghlan and Christian Heimes in bpo-15061@footnote{https://bugs.python.org/issue15061}.)

@node http<4>,html<2>,hmac<4>,Improved Modules<9>
@anchor{whatsnew/3 3 http}@anchor{c5c}
@subsubsection http


@ref{c5d,,http.server.BaseHTTPRequestHandler} now buffers the headers and writes
them all at once when @ref{c5e,,end_headers()} is
called.  A new method @ref{c5f,,flush_headers()}
can be used to directly manage when the accumulated headers are sent.
(Contributed by Andrew Schaaf in bpo-3709@footnote{https://bugs.python.org/issue3709}.)

@ref{92,,http.server} now produces valid @code{HTML 4.01 strict} output.
(Contributed by Ezio Melotti in bpo-13295@footnote{https://bugs.python.org/issue13295}.)

@ref{c60,,http.client.HTTPResponse} now has a
@ref{c61,,readinto()} method, which means it can be used
as an @ref{c62,,io.RawIOBase} class.  (Contributed by John Kuhn in
bpo-13464@footnote{https://bugs.python.org/issue13464}.)

@node html<2>,imaplib<3>,http<4>,Improved Modules<9>
@anchor{whatsnew/3 3 html}@anchor{c63}
@subsubsection html


@ref{33c,,html.parser.HTMLParser} is now able to parse broken markup without
raising errors, therefore the @emph{strict} argument of the constructor and the
@code{HTMLParseError} exception are now deprecated.
The ability to parse broken markup is the result of a number of bug fixes that
are also available on the latest bug fix releases of Python 2.7/3.2.
(Contributed by Ezio Melotti in bpo-15114@footnote{https://bugs.python.org/issue15114}, and bpo-14538@footnote{https://bugs.python.org/issue14538},
bpo-13993@footnote{https://bugs.python.org/issue13993}, bpo-13960@footnote{https://bugs.python.org/issue13960}, bpo-13358@footnote{https://bugs.python.org/issue13358}, bpo-1745761@footnote{https://bugs.python.org/issue1745761},
bpo-755670@footnote{https://bugs.python.org/issue755670}, bpo-13357@footnote{https://bugs.python.org/issue13357}, bpo-12629@footnote{https://bugs.python.org/issue12629}, bpo-1200313@footnote{https://bugs.python.org/issue1200313},
bpo-670664@footnote{https://bugs.python.org/issue670664}, bpo-13273@footnote{https://bugs.python.org/issue13273}, bpo-12888@footnote{https://bugs.python.org/issue12888}, bpo-7311@footnote{https://bugs.python.org/issue7311}.)

A new @ref{c64,,html5} dictionary that maps HTML5 named character
references to the equivalent Unicode character(s) (e.g. @code{html5['gt;'] ==
'>'}) has been added to the @ref{8c,,html.entities} module.  The dictionary is
now also used by @ref{33c,,HTMLParser}.  (Contributed by Ezio
Melotti in bpo-11113@footnote{https://bugs.python.org/issue11113} and bpo-15156@footnote{https://bugs.python.org/issue15156}.)

@node imaplib<3>,inspect<8>,html<2>,Improved Modules<9>
@anchor{whatsnew/3 3 imaplib}@anchor{c65}
@subsubsection imaplib


The @ref{3f3,,IMAP4_SSL} constructor now accepts an SSLContext
parameter to control parameters of the secure channel.

(Contributed by Sijin Joseph in bpo-8808@footnote{https://bugs.python.org/issue8808}.)

@node inspect<8>,io<4>,imaplib<3>,Improved Modules<9>
@anchor{whatsnew/3 3 inspect}@anchor{c66}
@subsubsection inspect


A new @ref{c67,,getclosurevars()} function has been added. This function
reports the current binding of all names referenced from the function body and
where those names were resolved, making it easier to verify correct internal
state when testing code that relies on stateful closures.

(Contributed by Meador Inge and Nick Coghlan in bpo-13062@footnote{https://bugs.python.org/issue13062}.)

A new @ref{c68,,getgeneratorlocals()} function has been added. This
function reports the current binding of local variables in the generator’s
stack frame, making it easier to verify correct internal state when testing
generators.

(Contributed by Meador Inge in bpo-15153@footnote{https://bugs.python.org/issue15153}.)

@node io<4>,itertools<3>,inspect<8>,Improved Modules<9>
@anchor{whatsnew/3 3 io}@anchor{c69}
@subsubsection io


The @ref{1c9,,open()} function has a new @code{'x'} mode that can be used to
exclusively create a new file, and raise a @ref{bb1,,FileExistsError} if the file
already exists. It is based on the C11 ‘x’ mode to fopen().

(Contributed by David Townshend in bpo-12760@footnote{https://bugs.python.org/issue12760}.)

The constructor of the @ref{889,,TextIOWrapper} class has a new
@emph{write_through} optional argument. If @emph{write_through} is @code{True}, calls to
@code{write()} are guaranteed not to be buffered: any data
written on the @ref{889,,TextIOWrapper} object is immediately handled to its
underlying binary buffer.

@node itertools<3>,logging<6>,io<4>,Improved Modules<9>
@anchor{whatsnew/3 3 itertools}@anchor{c6a}
@subsubsection itertools


@ref{505,,accumulate()} now takes an optional @code{func} argument for
providing a user-supplied binary function.

@node logging<6>,math<7>,itertools<3>,Improved Modules<9>
@anchor{whatsnew/3 3 logging}@anchor{c6b}
@subsubsection logging


The @ref{508,,basicConfig()} function now supports an optional @code{handlers}
argument taking an iterable of handlers to be added to the root logger.

A class level attribute @code{append_nul} has
been added to @ref{c6c,,SysLogHandler} to allow control of the
appending of the @code{NUL} (@code{\000}) byte to syslog records, since for some
daemons it is required while for others it is passed through to the log.

@node math<7>,mmap<3>,logging<6>,Improved Modules<9>
@anchor{whatsnew/3 3 math}@anchor{c6d}
@subsubsection math


The @ref{ad,,math} module has a new function, @ref{c6e,,log2()},  which returns
the base-2 logarithm of @emph{x}.

(Written by Mark Dickinson in bpo-11888@footnote{https://bugs.python.org/issue11888}.)

@node mmap<3>,multiprocessing<7>,math<7>,Improved Modules<9>
@anchor{whatsnew/3 3 mmap}@anchor{c6f}
@subsubsection mmap


The @ref{c70,,read()} method is now more compatible with other file-like
objects: if the argument is omitted or specified as @code{None}, it returns the
bytes from the current file position to the end of the mapping.  (Contributed
by Petri Lehtinen in bpo-12021@footnote{https://bugs.python.org/issue12021}.)

@node multiprocessing<7>,nntplib<2>,mmap<3>,Improved Modules<9>
@anchor{whatsnew/3 3 multiprocessing}@anchor{c71}
@subsubsection multiprocessing


The new @ref{c72,,multiprocessing.connection.wait()} function allows polling
multiple objects (such as connections, sockets and pipes) with a timeout.
(Contributed by Richard Oudkerk in bpo-12328@footnote{https://bugs.python.org/issue12328}.)

@code{multiprocessing.Connection} objects can now be transferred over
multiprocessing connections.
(Contributed by Richard Oudkerk in bpo-4892@footnote{https://bugs.python.org/issue4892}.)

@ref{681,,multiprocessing.Process} now accepts a @code{daemon} keyword argument
to override the default behavior of inheriting the @code{daemon} flag from
the parent process (bpo-6064@footnote{https://bugs.python.org/issue6064}).

New attribute @ref{c73,,multiprocessing.Process.sentinel} allows a
program to wait on multiple @ref{681,,Process} objects at one
time using the appropriate OS primitives (for example, @ref{e0,,select} on
posix systems).

New methods @ref{c74,,multiprocessing.pool.Pool.starmap()} and
@ref{c75,,starmap_async()} provide
@ref{c76,,itertools.starmap()} equivalents to the existing
@ref{c77,,multiprocessing.pool.Pool.map()} and
@ref{c78,,map_async()} functions.  (Contributed by Hynek
Schlawack in bpo-12708@footnote{https://bugs.python.org/issue12708}.)

@node nntplib<2>,os<10>,multiprocessing<7>,Improved Modules<9>
@anchor{whatsnew/3 3 nntplib}@anchor{c79}
@subsubsection nntplib


The @ref{409,,nntplib.NNTP} class now supports the context management protocol to
unconditionally consume @ref{beb,,socket.error} exceptions and to close the NNTP
connection when done:

@example
>>> from nntplib import NNTP
>>> with NNTP('news.gmane.org') as n:
...     n.group('gmane.comp.python.committers')
...
('211 1755 1 1755 gmane.comp.python.committers', 1755, 1, 1755, 'gmane.comp.python.committers')
>>>
@end example

(Contributed by Giampaolo Rodolà in bpo-9795@footnote{https://bugs.python.org/issue9795}.)

@node os<10>,pdb<5>,nntplib<2>,Improved Modules<9>
@anchor{whatsnew/3 3 os}@anchor{c7a}
@subsubsection os



@itemize *

@item 
The @ref{c0,,os} module has a new @ref{c7b,,pipe2()} function that makes it
possible to create a pipe with @ref{c13,,O_CLOEXEC} or
@ref{9a2,,O_NONBLOCK} flags set atomically. This is especially useful to
avoid race conditions in multi-threaded programs.

@item 
The @ref{c0,,os} module has a new @ref{62c,,sendfile()} function which provides
an efficient “zero-copy” way for copying data from one file (or socket)
descriptor to another. The phrase “zero-copy” refers to the fact that all of
the copying of data between the two descriptors is done entirely by the
kernel, with no copying of data into userspace buffers. @ref{62c,,sendfile()}
can be used to efficiently copy data from a file on disk to a network socket,
e.g. for downloading a file.

(Patch submitted by Ross Lagerwall and Giampaolo Rodolà in bpo-10882@footnote{https://bugs.python.org/issue10882}.)

@item 
To avoid race conditions like symlink attacks and issues with temporary
files and directories, it is more reliable (and also faster) to manipulate
file descriptors instead of file names. Python 3.3 enhances existing functions
and introduces new functions to work on file descriptors (bpo-4761@footnote{https://bugs.python.org/issue4761},
bpo-10755@footnote{https://bugs.python.org/issue10755} and bpo-14626@footnote{https://bugs.python.org/issue14626}).


@itemize -

@item 
The @ref{c0,,os} module has a new @ref{683,,fwalk()} function similar to
@ref{8de,,walk()} except that it also yields file descriptors referring to the
directories visited. This is especially useful to avoid symlink races.

@item 
The following functions get new optional @emph{dir_fd} (@ref{c7c,,paths relative to directory descriptors}) and/or @emph{follow_symlinks} (@ref{c7d,,not following symlinks}):
@ref{c7e,,access()}, @ref{c7f,,chflags()}, @ref{c80,,chmod()}, @ref{c81,,chown()},
@ref{c82,,link()}, @ref{517,,lstat()}, @ref{c83,,mkdir()}, @ref{8ec,,mkfifo()},
@ref{8ed,,mknod()}, @ref{8ee,,open()}, @ref{414,,readlink()}, @ref{c84,,remove()},
@ref{c85,,rename()}, @ref{c86,,replace()}, @ref{c87,,rmdir()}, @ref{516,,stat()},
@ref{c88,,symlink()}, @ref{c89,,unlink()}, @ref{c8a,,utime()}.  Platform
support for using these parameters can be checked via the sets
@ref{c8b,,os.supports_dir_fd} and @code{os.supports_follows_symlinks}.

@item 
The following functions now support a file descriptor for their path argument:
@ref{3b9,,chdir()}, @ref{c80,,chmod()}, @ref{c81,,chown()},
@ref{c8c,,execve()}, @ref{c8d,,listdir()}, @ref{c8e,,pathconf()}, @ref{51a,,exists()},
@ref{516,,stat()}, @ref{c8f,,statvfs()}, @ref{c8a,,utime()}.  Platform support
for this can be checked via the @ref{c90,,os.supports_fd} set.
@end itemize

@item 
@ref{c7e,,access()} accepts an @code{effective_ids} keyword argument to turn on
using the effective uid/gid rather than the real uid/gid in the access check.
Platform support for this can be checked via the
@ref{c91,,supports_effective_ids} set.

@item 
The @ref{c0,,os} module has two new functions: @ref{c92,,getpriority()} and
@ref{c93,,setpriority()}. They can be used to get or set process
niceness/priority in a fashion similar to @ref{c94,,os.nice()} but extended to all
processes instead of just the current one.

(Patch submitted by Giampaolo Rodolà in bpo-10784@footnote{https://bugs.python.org/issue10784}.)

@item 
The new @ref{c86,,os.replace()} function allows cross-platform renaming of a
file with overwriting the destination.  With @ref{c85,,os.rename()}, an existing
destination file is overwritten under POSIX, but raises an error under
Windows.
(Contributed by Antoine Pitrou in bpo-8828@footnote{https://bugs.python.org/issue8828}.)

@item 
The stat family of functions (@ref{516,,stat()}, @ref{8e8,,fstat()},
and @ref{517,,lstat()}) now support reading a file’s timestamps
with nanosecond precision.  Symmetrically, @ref{c8a,,utime()}
can now write file timestamps with nanosecond precision.  (Contributed by
Larry Hastings in bpo-14127@footnote{https://bugs.python.org/issue14127}.)

@item 
The new @ref{c95,,os.get_terminal_size()} function queries the size of the
terminal attached to a file descriptor. See also
@ref{c96,,shutil.get_terminal_size()}.
(Contributed by Zbigniew Jędrzejewski-Szmek in bpo-13609@footnote{https://bugs.python.org/issue13609}.)
@end itemize

@c XXX sort out this mess after beta1


@itemize *

@item 
New functions to support Linux extended attributes (bpo-12720@footnote{https://bugs.python.org/issue12720}):
@ref{c97,,getxattr()}, @ref{c98,,listxattr()}, @ref{c99,,removexattr()},
@ref{c9a,,setxattr()}.

@item 
New interface to the scheduler. These functions
control how a process is allocated CPU time by the operating system. New
functions:
@ref{c9b,,sched_get_priority_max()}, @ref{c9c,,sched_get_priority_min()},
@ref{c9d,,sched_getaffinity()}, @ref{c9e,,sched_getparam()},
@ref{c9f,,sched_getscheduler()}, @ref{ca0,,sched_rr_get_interval()},
@ref{ca1,,sched_setaffinity()}, @ref{ca2,,sched_setparam()},
@ref{ca3,,sched_setscheduler()}, @ref{ca4,,sched_yield()},

@item 
New functions to control the file system:


@itemize *

@item 
@ref{8ef,,posix_fadvise()}: Announces an intention to access data in a
specific pattern thus allowing the kernel to make optimizations.

@item 
@ref{8f0,,posix_fallocate()}: Ensures that enough disk space is allocated
for a file.

@item 
@ref{ca5,,sync()}: Force write of everything to disk.
@end itemize

@item 
Additional new  posix functions:


@itemize *

@item 
@ref{ca6,,lockf()}: Apply, test or remove a POSIX lock on an open file descriptor.

@item 
@ref{688,,pread()}: Read from a file descriptor at an offset, the file
offset remains unchanged.

@item 
@ref{68b,,pwrite()}: Write to a file descriptor from an offset, leaving
the file offset unchanged.

@item 
@ref{687,,readv()}: Read from a file descriptor into a number of writable buffers.

@item 
@ref{9a3,,truncate()}: Truncate the file corresponding to @emph{path}, so that
it is at most @emph{length} bytes in size.

@item 
@ref{8f5,,waitid()}: Wait for the completion of one or more child processes.

@item 
@ref{68a,,writev()}: Write the contents of @emph{buffers} to a file descriptor,
where @emph{buffers} is an arbitrary sequence of buffers.

@item 
@ref{ca7,,getgrouplist()} (bpo-9344@footnote{https://bugs.python.org/issue9344}): Return list of group ids that
specified user belongs to.
@end itemize

@item 
@ref{ca8,,times()} and @ref{ca9,,uname()}: Return type changed from a tuple to
a tuple-like object with named attributes.

@item 
Some platforms now support additional constants for the @ref{caa,,lseek()}
function, such as @code{os.SEEK_HOLE} and @code{os.SEEK_DATA}.

@item 
New constants @ref{cab,,RTLD_LAZY}, @ref{cac,,RTLD_NOW},
@ref{cad,,RTLD_GLOBAL}, @ref{cae,,RTLD_LOCAL}, @ref{caf,,RTLD_NODELETE},
@ref{cb0,,RTLD_NOLOAD}, and @ref{cb1,,RTLD_DEEPBIND} are available on
platforms that support them.   These are for use with the
@ref{cb2,,sys.setdlopenflags()} function, and supersede the similar constants
defined in @ref{29,,ctypes} and @code{DLFCN}.  (Contributed by Victor Stinner
in bpo-13226@footnote{https://bugs.python.org/issue13226}.)

@item 
@ref{c88,,os.symlink()} now accepts (and ignores) the @code{target_is_directory}
keyword argument on non-Windows platforms, to ease cross-platform support.
@end itemize

@node pdb<5>,pickle<5>,os<10>,Improved Modules<9>
@anchor{whatsnew/3 3 pdb}@anchor{cb3}
@subsubsection pdb


Tab-completion is now available not only for command names, but also their
arguments.  For example, for the @code{break} command, function and file names
are completed.

(Contributed by Georg Brandl in bpo-14210@footnote{https://bugs.python.org/issue14210})

@node pickle<5>,pydoc<5>,pdb<5>,Improved Modules<9>
@anchor{whatsnew/3 3 pickle}@anchor{cb4}
@subsubsection pickle


@ref{52d,,pickle.Pickler} objects now have an optional
@ref{cb5,,dispatch_table} attribute allowing per-pickler
reduction functions to be set.

(Contributed by Richard Oudkerk in bpo-14166@footnote{https://bugs.python.org/issue14166}.)

@node pydoc<5>,re<6>,pickle<5>,Improved Modules<9>
@anchor{whatsnew/3 3 pydoc}@anchor{cb6}
@subsubsection pydoc


The Tk GUI and the @code{serve()} function have been removed from the
@ref{d4,,pydoc} module: @code{pydoc -g} and @code{serve()} have been deprecated
in Python 3.2.

@node re<6>,sched,pydoc<5>,Improved Modules<9>
@anchor{whatsnew/3 3 re}@anchor{cb7}
@subsubsection re


@ref{2ef,,str} regular expressions now support @code{\u} and @code{\U} escapes.

(Contributed by Serhiy Storchaka in bpo-3665@footnote{https://bugs.python.org/issue3665}.)

@node sched,select<2>,re<6>,Improved Modules<9>
@anchor{whatsnew/3 3 sched}@anchor{cb8}
@subsubsection sched



@itemize *

@item 
@ref{cb9,,run()} now accepts a @emph{blocking} parameter which when
set to false makes the method execute the scheduled events due to expire
soonest (if any) and then return immediately.
This is useful in case you want to use the @ref{cba,,scheduler} in
non-blocking applications.  (Contributed by Giampaolo Rodolà in bpo-13449@footnote{https://bugs.python.org/issue13449}.)

@item 
@ref{cba,,scheduler} class can now be safely used in multi-threaded
environments.  (Contributed by Josiah Carlson and Giampaolo Rodolà in
bpo-8684@footnote{https://bugs.python.org/issue8684}.)

@item 
@emph{timefunc} and @emph{delayfunct} parameters of @ref{cba,,scheduler} class
constructor are now optional and defaults to @ref{5fc,,time.time()} and
@ref{196,,time.sleep()} respectively.  (Contributed by Chris Clark in
bpo-13245@footnote{https://bugs.python.org/issue13245}.)

@item 
@ref{cbb,,enter()} and @ref{cbc,,enterabs()}
@emph{argument} parameter is now optional.  (Contributed by Chris Clark in
bpo-13245@footnote{https://bugs.python.org/issue13245}.)

@item 
@ref{cbb,,enter()} and @ref{cbc,,enterabs()}
now accept a @emph{kwargs} parameter.  (Contributed by Chris Clark in
bpo-13245@footnote{https://bugs.python.org/issue13245}.)
@end itemize

@node select<2>,shlex<3>,sched,Improved Modules<9>
@anchor{whatsnew/3 3 select}@anchor{cbd}
@subsubsection select


Solaris and derivative platforms have a new class @ref{b16,,select.devpoll}
for high performance asynchronous sockets via @code{/dev/poll}.
(Contributed by Jesús Cea Avión in bpo-6397@footnote{https://bugs.python.org/issue6397}.)

@node shlex<3>,shutil<4>,select<2>,Improved Modules<9>
@anchor{whatsnew/3 3 shlex}@anchor{cbe}
@subsubsection shlex


The previously undocumented helper function @code{quote} from the
@ref{c7,,pipes} modules has been moved to the @ref{e3,,shlex} module and
documented.  @ref{cbf,,quote()} properly escapes all characters in a string
that might be otherwise given special meaning by the shell.

@node shutil<4>,signal<4>,shlex<3>,Improved Modules<9>
@anchor{whatsnew/3 3 shutil}@anchor{cc0}
@subsubsection shutil



@itemize *

@item 
New functions:


@itemize *

@item 
@ref{cc1,,disk_usage()}: provides total, used and free disk space
statistics.  (Contributed by Giampaolo Rodolà in bpo-12442@footnote{https://bugs.python.org/issue12442}.)

@item 
@ref{cc2,,chown()}: allows one to change user and/or group of the given
path also specifying the user/group names and not only their numeric
ids.  (Contributed by Sandro Tosi in bpo-12191@footnote{https://bugs.python.org/issue12191}.)

@item 
@ref{c96,,shutil.get_terminal_size()}: returns the size of the terminal window
to which the interpreter is attached.  (Contributed by Zbigniew
Jędrzejewski-Szmek in bpo-13609@footnote{https://bugs.python.org/issue13609}.)
@end itemize

@item 
@ref{571,,copy2()} and @ref{cc3,,copystat()} now preserve file
timestamps with nanosecond precision on platforms that support it.
They also preserve file “extended attributes” on Linux.  (Contributed
by Larry Hastings in bpo-14127@footnote{https://bugs.python.org/issue14127} and  bpo-15238@footnote{https://bugs.python.org/issue15238}.)

@item 
Several functions now take an optional @code{symlinks} argument: when that
parameter is true, symlinks aren’t dereferenced and the operation instead
acts on the symlink itself (or creates one, if relevant).
(Contributed by Hynek Schlawack in bpo-12715@footnote{https://bugs.python.org/issue12715}.)

@item 
When copying files to a different file system, @ref{572,,move()} now
handles symlinks the way the posix @code{mv} command does, recreating the
symlink rather than copying the target file contents.  (Contributed by
Jonathan Niehof in bpo-9993@footnote{https://bugs.python.org/issue9993}.)  @ref{572,,move()} now also returns
the @code{dst} argument as its result.

@item 
@ref{53a,,rmtree()} is now resistant to symlink attacks on platforms
which support the new @code{dir_fd} parameter in @ref{8ee,,os.open()} and
@ref{c89,,os.unlink()}.  (Contributed by Martin von Löwis and Hynek Schlawack
in bpo-4489@footnote{https://bugs.python.org/issue4489}.)
@end itemize

@node signal<4>,smtpd<3>,shutil<4>,Improved Modules<9>
@anchor{whatsnew/3 3 signal}@anchor{cc4}
@subsubsection signal



@itemize *

@item 
The @ref{e5,,signal} module has new functions:


@itemize *

@item 
@ref{cc5,,pthread_sigmask()}: fetch and/or change the signal mask of the
calling thread (Contributed by Jean-Paul Calderone in bpo-8407@footnote{https://bugs.python.org/issue8407});

@item 
@ref{cc6,,pthread_kill()}: send a signal to a thread;

@item 
@ref{cc7,,sigpending()}: examine pending functions;

@item 
@ref{cc8,,sigwait()}: wait a signal;

@item 
@ref{908,,sigwaitinfo()}: wait for a signal, returning detailed
information about it;

@item 
@ref{907,,sigtimedwait()}: like @ref{908,,sigwaitinfo()} but with a
timeout.
@end itemize

@item 
The signal handler writes the signal number as a single byte instead of
a nul byte into the wakeup file descriptor. So it is possible to wait more
than one signal and know which signals were raised.

@item 
@ref{cc9,,signal.signal()} and @ref{cca,,signal.siginterrupt()} raise an OSError,
instead of a RuntimeError: OSError has an errno attribute.
@end itemize

@node smtpd<3>,smtplib<4>,signal<4>,Improved Modules<9>
@anchor{whatsnew/3 3 smtpd}@anchor{ccb}
@subsubsection smtpd


The @ref{e7,,smtpd} module now supports 
@geindex RFC; RFC 5321
RFC 5321@footnote{https://tools.ietf.org/html/rfc5321.html} (extended SMTP) and 
@geindex RFC; RFC 1870
RFC 1870@footnote{https://tools.ietf.org/html/rfc1870.html}
(size extension).  Per the standard, these extensions are enabled if and only
if the client initiates the session with an @code{EHLO} command.

(Initial @code{ELHO} support by Alberto Trevino.  Size extension by Juhana
Jauhiainen.  Substantial additional work on the patch contributed by Michele
Orrù and Dan Boswell.  bpo-8739@footnote{https://bugs.python.org/issue8739})

@node smtplib<4>,socket<10>,smtpd<3>,Improved Modules<9>
@anchor{whatsnew/3 3 smtplib}@anchor{ccc}
@subsubsection smtplib


The @ref{421,,SMTP}, @ref{422,,SMTP_SSL}, and
@ref{423,,LMTP} classes now accept a @code{source_address} keyword argument
to specify the @code{(host, port)} to use as the source address in the bind call
when creating the outgoing socket.  (Contributed by Paulo Scardine in
bpo-11281@footnote{https://bugs.python.org/issue11281}.)

@ref{421,,SMTP} now supports the context management protocol, allowing an
@code{SMTP} instance to be used in a @code{with} statement.  (Contributed
by Giampaolo Rodolà in bpo-11289@footnote{https://bugs.python.org/issue11289}.)

The @ref{422,,SMTP_SSL} constructor and the @ref{ccd,,starttls()}
method now accept an SSLContext parameter to control parameters of the secure
channel.  (Contributed by Kasun Herath in bpo-8809@footnote{https://bugs.python.org/issue8809}.)

@node socket<10>,socketserver<3>,smtplib<4>,Improved Modules<9>
@anchor{whatsnew/3 3 socket}@anchor{cce}
@subsubsection socket



@itemize *

@item 
The @ref{8fd,,socket} class now exposes additional methods to process
ancillary data when supported by the underlying platform:


@itemize *

@item 
@ref{905,,sendmsg()}

@item 
@ref{902,,recvmsg()}

@item 
@ref{ccf,,recvmsg_into()}
@end itemize

(Contributed by David Watson in bpo-6560@footnote{https://bugs.python.org/issue6560}, based on an earlier patch by
Heiko Wundram)

@item 
The @ref{8fd,,socket} class now supports the PF_CAN protocol family
(@indicateurl{https://en.wikipedia.org/wiki/Socketcan}), on Linux
(@indicateurl{https://lwn.net/Articles/253425}).

(Contributed by Matthias Fuchs, updated by Tiago Gonçalves in bpo-10141@footnote{https://bugs.python.org/issue10141}.)

@item 
The @ref{8fd,,socket} class now supports the PF_RDS protocol family
(@indicateurl{https://en.wikipedia.org/wiki/Reliable_Datagram_Sockets} and
@indicateurl{https://oss.oracle.com/projects/rds/}).

@item 
The @ref{8fd,,socket} class now supports the @code{PF_SYSTEM} protocol
family on OS X.  (Contributed by Michael Goderbauer in bpo-13777@footnote{https://bugs.python.org/issue13777}.)

@item 
New function @ref{cd0,,sethostname()} allows the hostname to be set
on unix systems if the calling process has sufficient privileges.
(Contributed by Ross Lagerwall in bpo-10866@footnote{https://bugs.python.org/issue10866}.)
@end itemize

@node socketserver<3>,sqlite3<7>,socket<10>,Improved Modules<9>
@anchor{whatsnew/3 3 socketserver}@anchor{cd1}
@subsubsection socketserver


@ref{cd2,,BaseServer} now has an overridable method
@ref{cd3,,service_actions()} that is called by the
@ref{cd4,,serve_forever()} method in the service loop.
@ref{6a4,,ForkingMixIn} now uses this to clean up zombie
child processes.  (Contributed by Justin Warkentin in bpo-11109@footnote{https://bugs.python.org/issue11109}.)

@node sqlite3<7>,ssl<9>,socketserver<3>,Improved Modules<9>
@anchor{whatsnew/3 3 sqlite3}@anchor{cd5}
@subsubsection sqlite3


New @ref{18b,,sqlite3.Connection} method
@ref{cd6,,set_trace_callback()} can be used to capture a trace of
all sql commands processed by sqlite.  (Contributed by Torsten Landschoff
in bpo-11688@footnote{https://bugs.python.org/issue11688}.)

@node ssl<9>,stat<2>,sqlite3<7>,Improved Modules<9>
@anchor{whatsnew/3 3 ssl}@anchor{cd7}
@subsubsection ssl



@itemize *

@item 
The @ref{ee,,ssl} module has two new random generation functions:


@itemize *

@item 
@ref{cd8,,RAND_bytes()}: generate cryptographically strong
pseudo-random bytes.

@item 
@ref{334,,RAND_pseudo_bytes()}: generate pseudo-random bytes.
@end itemize

(Contributed by Victor Stinner in bpo-12049@footnote{https://bugs.python.org/issue12049}.)

@item 
The @ref{ee,,ssl} module now exposes a finer-grained exception hierarchy
in order to make it easier to inspect the various kinds of errors.
(Contributed by Antoine Pitrou in bpo-11183@footnote{https://bugs.python.org/issue11183}.)

@item 
@ref{cd9,,load_cert_chain()} now accepts a @emph{password} argument
to be used if the private key is encrypted.
(Contributed by Adam Simpkins in bpo-12803@footnote{https://bugs.python.org/issue12803}.)

@item 
Diffie-Hellman key exchange, both regular and Elliptic Curve-based, is
now supported through the @ref{cda,,load_dh_params()} and
@ref{cdb,,set_ecdh_curve()} methods.
(Contributed by Antoine Pitrou in bpo-13626@footnote{https://bugs.python.org/issue13626} and bpo-13627@footnote{https://bugs.python.org/issue13627}.)

@item 
SSL sockets have a new @ref{cdc,,get_channel_binding()} method
allowing the implementation of certain authentication mechanisms such as
SCRAM-SHA-1-PLUS.  (Contributed by Jacek Konieczny in bpo-12551@footnote{https://bugs.python.org/issue12551}.)

@item 
You can query the SSL compression algorithm used by an SSL socket, thanks
to its new @ref{cdd,,compression()} method.  The new attribute
@ref{cde,,OP_NO_COMPRESSION} can be used to disable compression.
(Contributed by Antoine Pitrou in bpo-13634@footnote{https://bugs.python.org/issue13634}.)

@item 
Support has been added for the Next Protocol Negotiation extension using
the @ref{336,,ssl.SSLContext.set_npn_protocols()} method.
(Contributed by Colin Marc in bpo-14204@footnote{https://bugs.python.org/issue14204}.)

@item 
SSL errors can now be introspected more easily thanks to
@ref{cdf,,library} and @ref{ce0,,reason} attributes.
(Contributed by Antoine Pitrou in bpo-14837@footnote{https://bugs.python.org/issue14837}.)

@item 
The @ref{2c0,,get_server_certificate()} function now supports IPv6.
(Contributed by Charles-François Natali in bpo-11811@footnote{https://bugs.python.org/issue11811}.)

@item 
New attribute @ref{ce1,,OP_CIPHER_SERVER_PREFERENCE} allows setting
SSLv3 server sockets to use the server’s cipher ordering preference rather
than the client’s (bpo-13635@footnote{https://bugs.python.org/issue13635}).
@end itemize

@node stat<2>,struct<3>,ssl<9>,Improved Modules<9>
@anchor{whatsnew/3 3 stat}@anchor{ce2}
@subsubsection stat


The undocumented tarfile.filemode function has been moved to
@ref{ce3,,stat.filemode()}. It can be used to convert a file’s mode to a string of
the form ‘-rwxrwxrwx’.

(Contributed by Giampaolo Rodolà in bpo-14807@footnote{https://bugs.python.org/issue14807}.)

@node struct<3>,subprocess<5>,stat<2>,Improved Modules<9>
@anchor{whatsnew/3 3 struct}@anchor{ce4}
@subsubsection struct


The @ref{f3,,struct} module now supports @code{ssize_t} and @code{size_t} via the
new codes @code{n} and @code{N}, respectively.  (Contributed by Antoine Pitrou
in bpo-3163@footnote{https://bugs.python.org/issue3163}.)

@node subprocess<5>,sys<10>,struct<3>,Improved Modules<9>
@anchor{whatsnew/3 3 subprocess}@anchor{ce5}
@subsubsection subprocess


Command strings can now be bytes objects on posix platforms.  (Contributed by
Victor Stinner in bpo-8513@footnote{https://bugs.python.org/issue8513}.)

A new constant @ref{ce6,,DEVNULL} allows suppressing output in a
platform-independent fashion.  (Contributed by Ross Lagerwall in
bpo-5870@footnote{https://bugs.python.org/issue5870}.)

@node sys<10>,tarfile<4>,subprocess<5>,Improved Modules<9>
@anchor{whatsnew/3 3 sys}@anchor{ce7}
@subsubsection sys


The @ref{f7,,sys} module has a new @ref{ce8,,thread_info} @ref{ce9,,named tuple} holding information about the thread implementation
(bpo-11223@footnote{https://bugs.python.org/issue11223}).

@node tarfile<4>,tempfile,sys<10>,Improved Modules<9>
@anchor{whatsnew/3 3 tarfile}@anchor{cea}
@subsubsection tarfile


@ref{fb,,tarfile} now supports @code{lzma} encoding via the @ref{a9,,lzma} module.
(Contributed by Lars Gustäbel in bpo-5689@footnote{https://bugs.python.org/issue5689}.)

@node tempfile,textwrap<2>,tarfile<4>,Improved Modules<9>
@anchor{whatsnew/3 3 tempfile}@anchor{ceb}
@subsubsection tempfile


@ref{cec,,tempfile.SpooledTemporaryFile}'s
@code{truncate()} method now accepts
a @code{size} parameter.  (Contributed by Ryan Kelly in bpo-9957@footnote{https://bugs.python.org/issue9957}.)

@node textwrap<2>,threading<7>,tempfile,Improved Modules<9>
@anchor{whatsnew/3 3 textwrap}@anchor{ced}
@subsubsection textwrap


The @ref{108,,textwrap} module has a new @ref{cee,,indent()} that makes
it straightforward to add a common prefix to selected lines in a block
of text  (bpo-13857@footnote{https://bugs.python.org/issue13857}).

@node threading<7>,time<7>,textwrap<2>,Improved Modules<9>
@anchor{whatsnew/3 3 threading}@anchor{cef}
@subsubsection threading


@ref{cf0,,threading.Condition}, @ref{cf1,,threading.Semaphore},
@ref{cf2,,threading.BoundedSemaphore}, @ref{cf3,,threading.Event}, and
@ref{cf4,,threading.Timer}, all of which used to be factory functions returning a
class instance, are now classes and may be subclassed.  (Contributed by Éric
Araujo in bpo-10968@footnote{https://bugs.python.org/issue10968}.)

The @ref{44b,,threading.Thread} constructor now accepts a @code{daemon} keyword
argument to override the default behavior of inheriting the @code{daemon} flag
value from the parent thread (bpo-6064@footnote{https://bugs.python.org/issue6064}).

The formerly private function @code{_thread.get_ident} is now available as the
public function @ref{cf5,,threading.get_ident()}.  This eliminates several cases of
direct access to the @code{_thread} module in the stdlib.  Third party code that
used @code{_thread.get_ident} should likewise be changed to use the new public
interface.

@node time<7>,types<5>,threading<7>,Improved Modules<9>
@anchor{whatsnew/3 3 time}@anchor{cf6}
@subsubsection time


The 
@geindex Python Enhancement Proposals; PEP 418
PEP 418@footnote{https://www.python.org/dev/peps/pep-0418} added new functions to the @ref{10a,,time} module:


@itemize *

@item 
@ref{cf7,,get_clock_info()}: Get information on a clock.

@item 
@ref{9ed,,monotonic()}: Monotonic clock (cannot go backward), not affected
by system clock updates.

@item 
@ref{5a0,,perf_counter()}: Performance counter with the highest available
resolution to measure a short duration.

@item 
@ref{5a1,,process_time()}: Sum of the system and user CPU time of the
current process.
@end itemize

Other new functions:


@itemize *

@item 
@ref{cf8,,clock_getres()}, @ref{cf9,,clock_gettime()} and
@ref{cfa,,clock_settime()} functions with @code{CLOCK_xxx} constants.
(Contributed by Victor Stinner in bpo-10278@footnote{https://bugs.python.org/issue10278}.)
@end itemize

To improve cross platform consistency, @ref{196,,sleep()} now raises a
@ref{17c,,ValueError} when passed a negative sleep value.  Previously this was an
error on posix, but produced an infinite sleep on Windows.

@node types<5>,unittest<6>,time<7>,Improved Modules<9>
@anchor{whatsnew/3 3 types}@anchor{cfb}
@subsubsection types


Add a new @ref{246,,types.MappingProxyType} class: Read-only proxy of a mapping.
(bpo-14386@footnote{https://bugs.python.org/issue14386})

The new functions @ref{cfc,,types.new_class()} and @ref{cfd,,types.prepare_class()} provide support
for 
@geindex Python Enhancement Proposals; PEP 3115
PEP 3115@footnote{https://www.python.org/dev/peps/pep-3115} compliant dynamic type creation. (bpo-14588@footnote{https://bugs.python.org/issue14588})

@node unittest<6>,urllib<3>,types<5>,Improved Modules<9>
@anchor{whatsnew/3 3 unittest}@anchor{cfe}
@subsubsection unittest


@ref{cff,,assertRaises()}, @ref{d00,,assertRaisesRegex()}, @ref{d01,,assertWarns()}, and
@ref{d02,,assertWarnsRegex()} now accept a keyword argument @emph{msg} when used as
context managers.  (Contributed by Ezio Melotti and Winston Ewert in
bpo-10775@footnote{https://bugs.python.org/issue10775}.)

@ref{d03,,unittest.TestCase.run()} now returns the @ref{d04,,TestResult}
object.

@node urllib<3>,webbrowser,unittest<6>,Improved Modules<9>
@anchor{whatsnew/3 3 urllib}@anchor{d05}
@subsubsection urllib


The @ref{b5e,,Request} class, now accepts a @emph{method} argument
used by @ref{d06,,get_method()} to determine what HTTP method
should be used.  For example, this will send a @code{'HEAD'} request:

@example
>>> urlopen(Request('https://www.python.org', method='HEAD'))
@end example

(bpo-1673007@footnote{https://bugs.python.org/issue1673007})

@node webbrowser,xml etree ElementTree,urllib<3>,Improved Modules<9>
@anchor{whatsnew/3 3 webbrowser}@anchor{d07}
@subsubsection webbrowser


The @ref{130,,webbrowser} module supports more “browsers”: Google Chrome (named
@code{chrome}, @code{chromium}, @code{chrome-browser} or
@code{chromium-browser} depending on the version and operating system),
and the generic launchers @code{xdg-open}, from the FreeDesktop.org
project, and @code{gvfs-open}, which is the default URI handler for GNOME
3.  (The former contributed by Arnaud Calmettes in bpo-13620@footnote{https://bugs.python.org/issue13620}, the latter
by Matthias Klose in bpo-14493@footnote{https://bugs.python.org/issue14493}.)

@node xml etree ElementTree,zlib<2>,webbrowser,Improved Modules<9>
@anchor{whatsnew/3 3 xml-etree-elementtree}@anchor{d08}
@subsubsection xml.etree.ElementTree


The @ref{13e,,xml.etree.ElementTree} module now imports its C accelerator by
default; there is no longer a need to explicitly import
@code{xml.etree.cElementTree} (this module stays for backwards compatibility,
but is now deprecated).  In addition,  the @code{iter} family of methods of
@ref{44e,,Element} has been optimized (rewritten in C).
The module’s documentation has also been greatly improved with added examples
and a more detailed reference.

@node zlib<2>,,xml etree ElementTree,Improved Modules<9>
@anchor{whatsnew/3 3 zlib}@anchor{d09}
@subsubsection zlib


New attribute @ref{d0a,,zlib.Decompress.eof} makes it possible to distinguish
between a properly-formed compressed stream and an incomplete or truncated one.
(Contributed by Nadeem Vawda in bpo-12646@footnote{https://bugs.python.org/issue12646}.)

New attribute @ref{d0b,,zlib.ZLIB_RUNTIME_VERSION} reports the version string of
the underlying @code{zlib} library that is loaded at runtime.  (Contributed by
Torsten Landschoff in bpo-12306@footnote{https://bugs.python.org/issue12306}.)

@node Optimizations<8>,Build and C API Changes<4>,Improved Modules<9>,What’s New In Python 3 3
@anchor{whatsnew/3 3 optimizations}@anchor{d0c}
@subsection Optimizations


Major performance enhancements have been added:


@itemize *

@item 
Thanks to 
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393}, some operations on Unicode strings have been optimized:


@itemize *

@item 
the memory footprint is divided by 2 to 4 depending on the text

@item 
encode an ASCII string to UTF-8 doesn’t need to encode characters anymore,
the UTF-8 representation is shared with the ASCII representation

@item 
the UTF-8 encoder has been optimized

@item 
repeating a single ASCII letter and getting a substring of an ASCII string
is 4 times faster
@end itemize

@item 
UTF-8 is now 2x to 4x faster.  UTF-16 encoding is now up to 10x faster.

(Contributed by Serhiy Storchaka, bpo-14624@footnote{https://bugs.python.org/issue14624}, bpo-14738@footnote{https://bugs.python.org/issue14738} and
bpo-15026@footnote{https://bugs.python.org/issue15026}.)
@end itemize

@node Build and C API Changes<4>,Deprecated<10>,Optimizations<8>,What’s New In Python 3 3
@anchor{whatsnew/3 3 build-and-c-api-changes}@anchor{bdd}
@subsection Build and C API Changes


Changes to Python’s build process and to the C API include:


@itemize *

@item 
New 
@geindex Python Enhancement Proposals; PEP 3118
PEP 3118@footnote{https://www.python.org/dev/peps/pep-3118} related function:


@itemize *

@item 
@ref{d0d,,PyMemoryView_FromMemory()}
@end itemize

@item 
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393} added new Unicode types, macros and functions:


@itemize *

@item 
High-level API:


@itemize *

@item 
@ref{397,,PyUnicode_CopyCharacters()}

@item 
@ref{39c,,PyUnicode_FindChar()}

@item 
@ref{395,,PyUnicode_GetLength()}, @ref{396,,PyUnicode_GET_LENGTH}

@item 
@ref{386,,PyUnicode_New()}

@item 
@ref{399,,PyUnicode_Substring()}

@item 
@ref{d0e,,PyUnicode_ReadChar()}, @ref{d0f,,PyUnicode_WriteChar()}
@end itemize

@item 
Low-level API:


@itemize *

@item 
@ref{d10,,Py_UCS1}, @ref{d11,,Py_UCS2}, @ref{d12,,Py_UCS4} types

@item 
@ref{d13,,PyASCIIObject} and @ref{d14,,PyCompactUnicodeObject} structures

@item 
@ref{d15,,PyUnicode_READY}

@item 
@ref{d16,,PyUnicode_FromKindAndData()}

@item 
@ref{d17,,PyUnicode_AsUCS4()}, @ref{39e,,PyUnicode_AsUCS4Copy()}

@item 
@ref{d18,,PyUnicode_DATA}, @ref{d19,,PyUnicode_1BYTE_DATA},
@ref{d1a,,PyUnicode_2BYTE_DATA}, @ref{d1b,,PyUnicode_4BYTE_DATA}

@item 
@ref{d1c,,PyUnicode_KIND} with @code{PyUnicode_Kind} enum:
@ref{d1d,,PyUnicode_WCHAR_KIND}, @ref{d1e,,PyUnicode_1BYTE_KIND},
@ref{d1f,,PyUnicode_2BYTE_KIND}, @ref{d20,,PyUnicode_4BYTE_KIND}

@item 
@ref{d21,,PyUnicode_READ}, @ref{d22,,PyUnicode_READ_CHAR}, @ref{d23,,PyUnicode_WRITE}

@item 
@ref{d24,,PyUnicode_MAX_CHAR_VALUE}
@end itemize
@end itemize

@item 
@ref{37f,,PyArg_ParseTuple} now accepts a @ref{2f1,,bytearray} for the @code{c}
format (bpo-12380@footnote{https://bugs.python.org/issue12380}).
@end itemize

@node Deprecated<10>,Porting to Python 3 3,Build and C API Changes<4>,What’s New In Python 3 3
@anchor{whatsnew/3 3 deprecated}@anchor{d25}
@subsection Deprecated


@menu
* Unsupported Operating Systems: Unsupported Operating Systems<2>. 
* Deprecated Python modules@comma{} functions and methods: Deprecated Python modules functions and methods<4>. 
* Deprecated functions and types of the C API: Deprecated functions and types of the C API<3>. 
* Deprecated features:: 

@end menu

@node Unsupported Operating Systems<2>,Deprecated Python modules functions and methods<4>,,Deprecated<10>
@anchor{whatsnew/3 3 unsupported-operating-systems}@anchor{d26}
@subsubsection Unsupported Operating Systems


OS/2 and VMS are no longer supported due to the lack of a maintainer.

Windows 2000 and Windows platforms which set @code{COMSPEC} to @code{command.com}
are no longer supported due to maintenance burden.

OSF support, which was deprecated in 3.2, has been completely removed.

@node Deprecated Python modules functions and methods<4>,Deprecated functions and types of the C API<3>,Unsupported Operating Systems<2>,Deprecated<10>
@anchor{whatsnew/3 3 deprecated-python-modules-functions-and-methods}@anchor{d27}
@subsubsection Deprecated Python modules, functions and methods



@itemize *

@item 
Passing a non-empty string to @code{object.__format__()} is deprecated, and
will produce a @ref{162,,TypeError} in Python 3.4 (bpo-9856@footnote{https://bugs.python.org/issue9856}).

@item 
The @code{unicode_internal} codec has been deprecated because of the
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393}, use UTF-8, UTF-16 (@code{utf-16-le} or @code{utf-16-be}), or UTF-32
(@code{utf-32-le} or @code{utf-32-be})

@item 
@ref{c55,,ftplib.FTP.nlst()} and @ref{c56,,ftplib.FTP.dir()}: use
@ref{c54,,ftplib.FTP.mlsd()}

@item 
@code{platform.popen()}: use the @ref{f4,,subprocess} module. Check especially
the @ref{d28,,Replacing Older Functions with the subprocess Module} section (bpo-11377@footnote{https://bugs.python.org/issue11377}).

@item 
bpo-13374@footnote{https://bugs.python.org/issue13374}: The Windows bytes API has been deprecated in the @ref{c0,,os}
module. Use Unicode filenames, instead of bytes filenames, to not depend on
the ANSI code page anymore and to support any filename.

@item 
bpo-13988@footnote{https://bugs.python.org/issue13988}: The @code{xml.etree.cElementTree} module is deprecated.  The
accelerator is used automatically whenever available.

@item 
The behaviour of @code{time.clock()} depends on the platform: use the new
@ref{5a0,,time.perf_counter()} or @ref{5a1,,time.process_time()} function instead,
depending on your requirements, to have a well defined behaviour.

@item 
The @code{os.stat_float_times()} function is deprecated.

@item 
@ref{3,,abc} module:


@itemize *

@item 
@ref{c1d,,abc.abstractproperty} has been deprecated, use @ref{501,,property}
with @ref{c1e,,abc.abstractmethod()} instead.

@item 
@ref{c1f,,abc.abstractclassmethod} has been deprecated, use
@ref{254,,classmethod} with @ref{c1e,,abc.abstractmethod()} instead.

@item 
@ref{c20,,abc.abstractstaticmethod} has been deprecated, use
@ref{253,,staticmethod} with @ref{c1e,,abc.abstractmethod()} instead.
@end itemize

@item 
@ref{96,,importlib} package:


@itemize *

@item 
@ref{d29,,importlib.abc.SourceLoader.path_mtime()} is now deprecated in favour of
@ref{d2a,,importlib.abc.SourceLoader.path_stats()} as bytecode files now store
both the modification time and size of the source file the bytecode file was
compiled from.
@end itemize
@end itemize

@node Deprecated functions and types of the C API<3>,Deprecated features,Deprecated Python modules functions and methods<4>,Deprecated<10>
@anchor{whatsnew/3 3 deprecated-functions-and-types-of-the-c-api}@anchor{d2b}
@subsubsection Deprecated functions and types of the C API


The @ref{d2c,,Py_UNICODE} has been deprecated by 
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393} and will be
removed in Python 4. All functions using this type are deprecated:

Unicode functions and methods using @ref{d2c,,Py_UNICODE} and
@ref{d2c,,Py_UNICODE}* types:


@itemize *

@item 
@ref{48c,,PyUnicode_FromUnicode}: use @ref{d2d,,PyUnicode_FromWideChar()} or
@ref{d16,,PyUnicode_FromKindAndData()}

@item 
@ref{d2e,,PyUnicode_AS_UNICODE}, @ref{48d,,PyUnicode_AsUnicode()},
@ref{48e,,PyUnicode_AsUnicodeAndSize()}: use @ref{39f,,PyUnicode_AsWideCharString()}

@item 
@ref{d2f,,PyUnicode_AS_DATA}: use @ref{d18,,PyUnicode_DATA} with
@ref{d21,,PyUnicode_READ} and @ref{d23,,PyUnicode_WRITE}

@item 
@ref{d30,,PyUnicode_GET_SIZE}, @ref{d31,,PyUnicode_GetSize()}: use
@ref{396,,PyUnicode_GET_LENGTH} or @ref{395,,PyUnicode_GetLength()}

@item 
@ref{d32,,PyUnicode_GET_DATA_SIZE}: use
@code{PyUnicode_GET_LENGTH(str) * PyUnicode_KIND(str)} (only work on ready
strings)

@item 
@code{PyUnicode_AsUnicodeCopy()}: use @ref{39e,,PyUnicode_AsUCS4Copy()} or
@ref{39f,,PyUnicode_AsWideCharString()}

@item 
@code{PyUnicode_GetMax()}
@end itemize

Functions and macros manipulating Py_UNICODE* strings:


@itemize *

@item 
@code{Py_UNICODE_strlen}: use @ref{395,,PyUnicode_GetLength()} or
@ref{396,,PyUnicode_GET_LENGTH}

@item 
@code{Py_UNICODE_strcat}: use @ref{397,,PyUnicode_CopyCharacters()} or
@ref{398,,PyUnicode_FromFormat()}

@item 
@code{Py_UNICODE_strcpy}, @code{Py_UNICODE_strncpy},
@code{Py_UNICODE_COPY}: use @ref{397,,PyUnicode_CopyCharacters()} or
@ref{399,,PyUnicode_Substring()}

@item 
@code{Py_UNICODE_strcmp}: use @ref{39a,,PyUnicode_Compare()}

@item 
@code{Py_UNICODE_strncmp}: use @ref{39b,,PyUnicode_Tailmatch()}

@item 
@code{Py_UNICODE_strchr}, @code{Py_UNICODE_strrchr}: use
@ref{39c,,PyUnicode_FindChar()}

@item 
@code{Py_UNICODE_FILL}: use @ref{d33,,PyUnicode_Fill()}

@item 
@code{Py_UNICODE_MATCH}
@end itemize

Encoders:


@itemize *

@item 
@code{PyUnicode_Encode()}: use @code{PyUnicode_AsEncodedObject()}

@item 
@code{PyUnicode_EncodeUTF7()}

@item 
@code{PyUnicode_EncodeUTF8()}: use @ref{6ef,,PyUnicode_AsUTF8()} or
@ref{d34,,PyUnicode_AsUTF8String()}

@item 
@code{PyUnicode_EncodeUTF32()}

@item 
@code{PyUnicode_EncodeUTF16()}

@item 
@code{PyUnicode_EncodeUnicodeEscape()} use
@ref{d35,,PyUnicode_AsUnicodeEscapeString()}

@item 
@code{PyUnicode_EncodeRawUnicodeEscape()} use
@ref{d36,,PyUnicode_AsRawUnicodeEscapeString()}

@item 
@code{PyUnicode_EncodeLatin1()}: use @ref{d37,,PyUnicode_AsLatin1String()}

@item 
@code{PyUnicode_EncodeASCII()}: use @ref{d38,,PyUnicode_AsASCIIString()}

@item 
@code{PyUnicode_EncodeCharmap()}

@item 
@code{PyUnicode_TranslateCharmap()}

@item 
@code{PyUnicode_EncodeMBCS()}: use @ref{d39,,PyUnicode_AsMBCSString()} or
@ref{d3a,,PyUnicode_EncodeCodePage()} (with @code{CP_ACP} code_page)

@item 
@code{PyUnicode_EncodeDecimal()},
@code{PyUnicode_TransformDecimalToASCII()}
@end itemize

@node Deprecated features,,Deprecated functions and types of the C API<3>,Deprecated<10>
@anchor{whatsnew/3 3 deprecated-features}@anchor{d3b}
@subsubsection Deprecated features


The @ref{6,,array} module’s @code{'u'} format code is now deprecated and will be
removed in Python 4 together with the rest of the (@ref{d2c,,Py_UNICODE}) API.

@node Porting to Python 3 3,,Deprecated<10>,What’s New In Python 3 3
@anchor{whatsnew/3 3 porting-to-python-3-3}@anchor{d3c}
@subsection Porting to Python 3.3


This section lists previously described changes and other bugfixes
that may require changes to your code.

@menu
* Porting Python code:: 
* Porting C code:: 
* Building C extensions:: 
* Command Line Switch Changes:: 

@end menu

@node Porting Python code,Porting C code,,Porting to Python 3 3
@anchor{whatsnew/3 3 porting-python-code}@anchor{c03}@anchor{whatsnew/3 3 portingpythoncode}@anchor{d3d}
@subsubsection Porting Python code



@itemize *

@item 
Hash randomization is enabled by default. Set the 
@geindex PYTHONHASHSEED
@geindex environment variable; PYTHONHASHSEED
@ref{c15,,PYTHONHASHSEED}
environment variable to @code{0} to disable hash randomization. See also the
@ref{619,,object.__hash__()} method.

@item 
bpo-12326@footnote{https://bugs.python.org/issue12326}: On Linux, sys.platform doesn’t contain the major version
anymore. It is now always ‘linux’, instead of ‘linux2’ or ‘linux3’ depending
on the Linux version used to build Python. Replace sys.platform == ‘linux2’
with sys.platform.startswith(‘linux’), or directly sys.platform == ‘linux’ if
you don’t need to support older Python versions.

@item 
bpo-13847@footnote{https://bugs.python.org/issue13847}, bpo-14180@footnote{https://bugs.python.org/issue14180}: @ref{10a,,time} and @ref{2f,,datetime}:
@ref{347,,OverflowError} is now raised instead of @ref{17c,,ValueError} if a
timestamp is out of range. @ref{2a1,,OSError} is now raised if C functions
@code{gmtime()} or @code{localtime()} failed.

@item 
The default finders used by import now utilize a cache of what is contained
within a specific directory. If you create a Python source file or sourceless
bytecode file, make sure to call @ref{74b,,importlib.invalidate_caches()} to clear
out the cache for the finders to notice the new file.

@item 
@ref{3b8,,ImportError} now uses the full name of the module that was attempted to
be imported. Doctests that check ImportErrors’ message will need to be
updated to use the full name of the module instead of just the tail of the
name.

@item 
The @emph{index} argument to @ref{3b7,,__import__()} now defaults to 0 instead of -1
and no longer support negative values. It was an oversight when 
@geindex Python Enhancement Proposals; PEP 328
PEP 328@footnote{https://www.python.org/dev/peps/pep-0328} was
implemented that the default value remained -1. If you need to continue to
perform a relative import followed by an absolute import, then perform the
relative import using an index of 1, followed by another import using an
index of 0. It is preferred, though, that you use
@ref{d3e,,importlib.import_module()} rather than call @ref{3b7,,__import__()} directly.

@item 
@ref{3b7,,__import__()} no longer allows one to use an index value other than 0
for top-level modules. E.g. @code{__import__('sys', level=1)} is now an error.

@item 
Because @ref{87c,,sys.meta_path} and @ref{bb5,,sys.path_hooks} now have finders on
them by default, you will most likely want to use @code{list.insert()} instead
of @code{list.append()} to add to those lists.

@item 
Because @code{None} is now inserted into @ref{74c,,sys.path_importer_cache}, if you
are clearing out entries in the dictionary of paths that do not have a
finder, you will need to remove keys paired with values of @code{None} @strong{and}
@ref{c0c,,imp.NullImporter} to be backwards-compatible. This will lead to extra
overhead on older versions of Python that re-insert @code{None} into
@ref{74c,,sys.path_importer_cache} where it represents the use of implicit
finders, but semantically it should not change anything.

@item 
@ref{30d,,importlib.abc.Finder} no longer specifies a @cite{find_module()} abstract
method that must be implemented. If you were relying on subclasses to
implement that method, make sure to check for the method’s existence first.
You will probably want to check for @cite{find_loader()} first, though, in the
case of working with @ref{c06,,path entry finders}.

@item 
@ref{c8,,pkgutil} has been converted to use @ref{96,,importlib} internally. This
eliminates many edge cases where the old behaviour of the 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} import
emulation failed to match the behaviour of the real import system. The
import emulation itself is still present, but is now deprecated. The
@ref{d3f,,pkgutil.iter_importers()} and @ref{73a,,pkgutil.walk_packages()} functions
special case the standard import hooks so they are still supported even
though they do not provide the non-standard @code{iter_modules()} method.

@item 
A longstanding RFC-compliance bug (bpo-1079@footnote{https://bugs.python.org/issue1079}) in the parsing done by
@ref{c49,,email.header.decode_header()} has been fixed.  Code that uses the
standard idiom to convert encoded headers into unicode
(@code{str(make_header(decode_header(h))}) will see no change, but code that
looks at the individual tuples returned by decode_header will see that
whitespace that precedes or follows @code{ASCII} sections is now included in the
@code{ASCII} section.  Code that builds headers using @code{make_header} should
also continue to work without change, since @code{make_header} continues to add
whitespace between @code{ASCII} and non-@code{ASCII} sections if it is not already
present in the input strings.

@item 
@ref{d40,,email.utils.formataddr()} now does the correct content transfer
encoding when passed non-@code{ASCII} display names.  Any code that depended on
the previous buggy behavior that preserved the non-@code{ASCII} unicode in the
formatted output string will need to be changed (bpo-1690608@footnote{https://bugs.python.org/issue1690608}).

@item 
@ref{d41,,poplib.POP3.quit()} may now raise protocol errors like all other
@code{poplib} methods.  Code that assumes @code{quit} does not raise
@ref{d42,,poplib.error_proto} errors may need to be changed if errors on @code{quit}
are encountered by a particular application (bpo-11291@footnote{https://bugs.python.org/issue11291}).

@item 
The @code{strict} argument to @ref{d43,,email.parser.Parser}, deprecated since
Python 2.4, has finally been removed.

@item 
The deprecated method @code{unittest.TestCase.assertSameElements} has been
removed.

@item 
The deprecated variable @code{time.accept2dyear} has been removed.

@item 
The deprecated @code{Context._clamp} attribute has been removed from the
@ref{34,,decimal} module.  It was previously replaced by the public attribute
@code{clamp}.  (See bpo-8540@footnote{https://bugs.python.org/issue8540}.)

@item 
The undocumented internal helper class @code{SSLFakeFile} has been removed
from @ref{e8,,smtplib}, since its functionality has long been provided directly
by @ref{d44,,socket.socket.makefile()}.

@item 
Passing a negative value to @ref{196,,time.sleep()} on Windows now raises an
error instead of sleeping forever.  It has always raised an error on posix.

@item 
The @code{ast.__version__} constant has been removed.  If you need to
make decisions affected by the AST version, use @ref{d45,,sys.version_info}
to make the decision.

@item 
Code that used to work around the fact that the @ref{109,,threading} module used
factory functions by subclassing the private classes will need to change to
subclass the now-public classes.

@item 
The undocumented debugging machinery in the threading module has been
removed, simplifying the code.  This should have no effect on production
code, but is mentioned here in case any application debug frameworks were
interacting with it (bpo-13550@footnote{https://bugs.python.org/issue13550}).
@end itemize

@node Porting C code,Building C extensions,Porting Python code,Porting to Python 3 3
@anchor{whatsnew/3 3 porting-c-code}@anchor{bde}
@subsubsection Porting C code



@itemize *

@item 
In the course of changes to the buffer API the undocumented
@code{smalltable} member of the
@ref{1db,,Py_buffer} structure has been removed and the
layout of the @code{PyMemoryViewObject} has changed.

All extensions relying on the relevant parts in @code{memoryobject.h}
or @code{object.h} must be rebuilt.

@item 
Due to @ref{bd2,,PEP 393}, the @ref{d2c,,Py_UNICODE} type and all
functions using this type are deprecated (but will stay available for
at least five years).  If you were using low-level Unicode APIs to
construct and access unicode objects and you want to benefit of the
memory footprint reduction provided by 
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393}, you have to convert
your code to the new @ref{d46,,Unicode API}.

However, if you only have been using high-level functions such as
@ref{d47,,PyUnicode_Concat()}, @ref{d48,,PyUnicode_Join()} or
@ref{398,,PyUnicode_FromFormat()}, your code will automatically take
advantage of the new unicode representations.

@item 
@ref{d49,,PyImport_GetMagicNumber()} now returns @code{-1} upon failure.

@item 
As a negative value for the @emph{level} argument to @ref{3b7,,__import__()} is no
longer valid, the same now holds for @ref{d4a,,PyImport_ImportModuleLevel()}.
This also means that the value of @emph{level} used by
@ref{d4b,,PyImport_ImportModuleEx()} is now @code{0} instead of @code{-1}.
@end itemize

@node Building C extensions,Command Line Switch Changes,Porting C code,Porting to Python 3 3
@anchor{whatsnew/3 3 building-c-extensions}@anchor{d4c}
@subsubsection Building C extensions



@itemize *

@item 
The range of possible file names for C extensions has been narrowed.
Very rarely used spellings have been suppressed: under POSIX, files
named @code{xxxmodule.so}, @code{xxxmodule.abi3.so} and
@code{xxxmodule.cpython-*.so} are no longer recognized as implementing
the @code{xxx} module.  If you had been generating such files, you have
to switch to the other spellings (i.e., remove the @code{module} string
from the file names).

(implemented in bpo-14040@footnote{https://bugs.python.org/issue14040}.)
@end itemize

@node Command Line Switch Changes,,Building C extensions,Porting to Python 3 3
@anchor{whatsnew/3 3 command-line-switch-changes}@anchor{d4d}
@subsubsection Command Line Switch Changes



@itemize *

@item 
The -Q command-line flag and related artifacts have been removed.  Code
checking sys.flags.division_warning will need updating.

(bpo-10998@footnote{https://bugs.python.org/issue10998}, contributed by Éric Araujo.)

@item 
When @code{python} is started with @ref{d4e,,-S}, @code{import site}
will no longer add site-specific paths to the module search paths.  In
previous versions, it did.

(bpo-11591@footnote{https://bugs.python.org/issue11591}, contributed by Carl Meyer with editions by Éric Araujo.)
@end itemize

@node What’s New In Python 3 2,What’s New In Python 3 1,What’s New In Python 3 3,What’s New in Python
@anchor{whatsnew/3 2 doc}@anchor{d4f}@anchor{whatsnew/3 2 what-s-new-in-python-3-2}@anchor{d50}
@section What’s New In Python 3.2


@*Author: 
Raymond Hettinger


@c $Id$
@c Rules for maintenance:
@c 
@c * Anyone can add text to this document.  Do not spend very much time
@c on the wording of your changes, because your text will probably
@c get rewritten.  (Note, during release candidate phase or just before
@c a beta release, please use the tracker instead -- this helps avoid
@c merge conflicts.   If you must add a suggested entry directly,
@c please put it in an XXX comment and the maintainer will take notice).
@c 
@c * The maintainer will go through Misc/NEWS periodically and add
@c changes; it's therefore more important to add your changes to
@c Misc/NEWS than to this file.
@c 
@c * This is not a complete list of every single change; completeness
@c is the purpose of Misc/NEWS.  Some changes I consider too small
@c or esoteric to include.  If such a change is added to the text,
@c I'll just remove it.  (This is another reason you shouldn't spend
@c too much time on writing your addition.)
@c 
@c * If you want to draw your new text to the attention of the
@c maintainer, add 'XXX' to the beginning of the paragraph or
@c section.
@c 
@c * It's OK to just add a fragmentary note about a change.  For
@c example: "XXX Describe the transmogrify() function added to the
@c socket module."  The maintainer will research the change and
@c write the necessary text.
@c 
@c * You can comment out your additions if you like, but it's not
@c necessary (especially when a final release is some months away).
@c 
@c * Credit the author of a patch or bugfix.   Just the name is
@c sufficient; the e-mail address isn't necessary.  It's helpful to
@c add the issue number:
@c 
@c   XXX Describe the transmogrify() function added to the socket
@c   module.
@c 
@c   (Contributed by P.Y. Developer; :issue:`12345`.)
@c 
@c This saves the maintainer the effort of going through the SVN log
@c when researching a change.

This article explains the new features in Python 3.2 as compared to 3.1.  It
focuses on a few highlights and gives a few examples.  For full details, see the
Misc/NEWS@footnote{https://github.com/python/cpython/blob/076ca6c3c8df3030307e548d9be792ce3c1c6eea/Misc/NEWS}
file.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 392
PEP 392@footnote{https://www.python.org/dev/peps/pep-0392} - Python 3.2 Release Schedule


@menu
* PEP 384; Defining a Stable ABI: PEP 384 Defining a Stable ABI. 
* PEP 389; Argparse Command Line Parsing Module: PEP 389 Argparse Command Line Parsing Module. 
* PEP 391; Dictionary Based Configuration for Logging: PEP 391 Dictionary Based Configuration for Logging. 
* PEP 3148; The concurrent.futures module: PEP 3148 The concurrent futures module. 
* PEP 3147; PYC Repository Directories: PEP 3147 PYC Repository Directories. 
* PEP 3149; ABI Version Tagged .so Files: PEP 3149 ABI Version Tagged so Files. 
* PEP 3333; Python Web Server Gateway Interface v1.0.1: PEP 3333 Python Web Server Gateway Interface v1 0 1. 
* Other Language Changes: Other Language Changes<10>. 
* New@comma{} Improved@comma{} and Deprecated Modules: New Improved and Deprecated Modules. 
* Multi-threading:: 
* Optimizations: Optimizations<9>. 
* Unicode:: 
* Codecs:: 
* Documentation:: 
* IDLE:: 
* Code Repository:: 
* Build and C API Changes: Build and C API Changes<5>. 
* Porting to Python 3.2: Porting to Python 3 2. 

@end menu

@node PEP 384 Defining a Stable ABI,PEP 389 Argparse Command Line Parsing Module,,What’s New In Python 3 2
@anchor{whatsnew/3 2 pep-384-defining-a-stable-abi}@anchor{d51}
@subsection PEP 384: Defining a Stable ABI


In the past, extension modules built for one Python version were often
not usable with other Python versions. Particularly on Windows, every
feature release of Python required rebuilding all extension modules that
one wanted to use. This requirement was the result of the free access to
Python interpreter internals that extension modules could use.

With Python 3.2, an alternative approach becomes available: extension
modules which restrict themselves to a limited API (by defining
Py_LIMITED_API) cannot use many of the internals, but are constrained
to a set of API functions that are promised to be stable for several
releases. As a consequence, extension modules built for 3.2 in that
mode will also work with 3.3, 3.4, and so on. Extension modules that
make use of details of memory structures can still be built, but will
need to be recompiled for every feature release.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 384

@item PEP 384@footnote{https://www.python.org/dev/peps/pep-0384} - Defining a Stable ABI

PEP written by Martin von Löwis.
@end table


@node PEP 389 Argparse Command Line Parsing Module,PEP 391 Dictionary Based Configuration for Logging,PEP 384 Defining a Stable ABI,What’s New In Python 3 2
@anchor{whatsnew/3 2 pep-389-argparse-command-line-parsing-module}@anchor{d52}
@subsection PEP 389: Argparse Command Line Parsing Module


A new module for command line parsing, @ref{5,,argparse}, was introduced to
overcome the limitations of @ref{bf,,optparse} which did not provide support for
positional arguments (not just options), subcommands, required options and other
common patterns of specifying and validating options.

This module has already had widespread success in the community as a
third-party module.  Being more fully featured than its predecessor, the
@ref{5,,argparse} module is now the preferred module for command-line processing.
The older module is still being kept available because of the substantial amount
of legacy code that depends on it.

Here’s an annotated example parser showing features like limiting results to a
set of choices, specifying a @emph{metavar} in the help screen, validating that one
or more positional arguments is present, and making a required option:

@example
import argparse
parser = argparse.ArgumentParser(
            description = 'Manage servers',         # main description for help
            epilog = 'Tested on Solaris and Linux') # displayed after help
parser.add_argument('action',                       # argument name
            choices = ['deploy', 'start', 'stop'],  # three allowed values
            help = 'action on each target')         # help msg
parser.add_argument('targets',
            metavar = 'HOSTNAME',                   # var name used in help msg
            nargs = '+',                            # require one or more targets
            help = 'url for target machines')       # help msg explanation
parser.add_argument('-u', '--user',                 # -u or --user option
            required = True,                        # make it a required argument
            help = 'login as user')
@end example

Example of calling the parser on a command string:

@example
>>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain'
>>> result = parser.parse_args(cmd.split())
>>> result.action
'deploy'
>>> result.targets
['sneezy.example.com', 'sleepy.example.com']
>>> result.user
'skycaptain'
@end example

Example of the parser’s automatically generated help:

@example
>>> parser.parse_args('-h'.split())

usage: manage_cloud.py [-h] -u USER
                       @{deploy,start,stop@} HOSTNAME [HOSTNAME ...]

Manage servers

positional arguments:
  @{deploy,start,stop@}   action on each target
  HOSTNAME              url for target machines

optional arguments:
  -h, --help            show this help message and exit
  -u USER, --user USER  login as user

Tested on Solaris and Linux
@end example

An especially nice @ref{5,,argparse} feature is the ability to define subparsers,
each with their own argument patterns and help displays:

@example
import argparse
parser = argparse.ArgumentParser(prog='HELM')
subparsers = parser.add_subparsers()

parser_l = subparsers.add_parser('launch', help='Launch Control')   # first subgroup
parser_l.add_argument('-m', '--missiles', action='store_true')
parser_l.add_argument('-t', '--torpedos', action='store_true')

parser_m = subparsers.add_parser('move', help='Move Vessel',        # second subgroup
                                 aliases=('steer', 'turn'))         # equivalent names
parser_m.add_argument('-c', '--course', type=int, required=True)
parser_m.add_argument('-s', '--speed', type=int, default=0)
@end example

@example
$ ./helm.py --help                         # top level help (launch and move)
$ ./helm.py launch --help                  # help for launch options
$ ./helm.py launch --missiles              # set missiles=True and torpedos=False
$ ./helm.py steer --course 180 --speed 5   # set movement parameters
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 389

@item PEP 389@footnote{https://www.python.org/dev/peps/pep-0389} - New Command Line Parsing Module

PEP written by Steven Bethard.
@end table

@ref{d53,,Upgrading optparse code} for details on the differences from @ref{bf,,optparse}.


@node PEP 391 Dictionary Based Configuration for Logging,PEP 3148 The concurrent futures module,PEP 389 Argparse Command Line Parsing Module,What’s New In Python 3 2
@anchor{whatsnew/3 2 pep-391-dictionary-based-configuration-for-logging}@anchor{d54}
@subsection PEP 391:  Dictionary Based Configuration for Logging


The @ref{a6,,logging} module provided two kinds of configuration, one style with
function calls for each option or another style driven by an external file saved
in a @code{ConfigParser} format.  Those options did not provide the flexibility
to create configurations from JSON or YAML files, nor did they support
incremental configuration, which is needed for specifying logger options from a
command line.

To support a more flexible style, the module now offers
@ref{d55,,logging.config.dictConfig()} for specifying logging configuration with
plain Python dictionaries.  The configuration options include formatters,
handlers, filters, and loggers.  Here’s a working example of a configuration
dictionary:

@example
@{"version": 1,
 "formatters": @{"brief": @{"format": "%(levelname)-8s: %(name)-15s: %(message)s"@},
                "full": @{"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"@}
                @},
 "handlers": @{"console": @{
                   "class": "logging.StreamHandler",
                   "formatter": "brief",
                   "level": "INFO",
                   "stream": "ext://sys.stdout"@},
              "console_priority": @{
                   "class": "logging.StreamHandler",
                   "formatter": "full",
                   "level": "ERROR",
                   "stream": "ext://sys.stderr"@}
              @},
 "root": @{"level": "DEBUG", "handlers": ["console", "console_priority"]@}@}
@end example

If that dictionary is stored in a file called @code{conf.json}, it can be
loaded and called with code like this:

@example
>>> import json, logging.config
>>> with open('conf.json') as f:
...     conf = json.load(f)
...
>>> logging.config.dictConfig(conf)
>>> logging.info("Transaction completed normally")
INFO    : root           : Transaction completed normally
>>> logging.critical("Abnormal termination")
2011-02-17 11:14:36,694 root            CRITICAL Abnormal termination
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 391

@item PEP 391@footnote{https://www.python.org/dev/peps/pep-0391} - Dictionary Based Configuration for Logging

PEP written by Vinay Sajip.
@end table


@node PEP 3148 The concurrent futures module,PEP 3147 PYC Repository Directories,PEP 391 Dictionary Based Configuration for Logging,What’s New In Python 3 2
@anchor{whatsnew/3 2 pep-3148-the-concurrent-futures-module}@anchor{d56}
@subsection PEP 3148:  The @code{concurrent.futures} module


Code for creating and managing concurrency is being collected in a new top-level
namespace, @emph{concurrent}.  Its first member is a @emph{futures} package which provides
a uniform high-level interface for managing threads and processes.

The design for @ref{20,,concurrent.futures} was inspired by the
@emph{java.util.concurrent} package.  In that model, a running call and its result
are represented by a @ref{d57,,Future} object that abstracts
features common to threads, processes, and remote procedure calls.  That object
supports status checks (running or done), timeouts, cancellations, adding
callbacks, and access to results or exceptions.

The primary offering of the new module is a pair of executor classes for
launching and managing calls.  The goal of the executors is to make it easier to
use existing tools for making parallel calls. They save the effort needed to
setup a pool of resources, launch the calls, create a results queue, add
time-out handling, and limit the total number of threads, processes, or remote
procedure calls.

Ideally, each application should share a single executor across multiple
components so that process and thread limits can be centrally managed.  This
solves the design challenge that arises when each component has its own
competing strategy for resource management.

Both classes share a common interface with three methods:
@ref{59a,,submit()} for scheduling a callable and
returning a @ref{d57,,Future} object;
@ref{943,,map()} for scheduling many asynchronous calls
at a time, and @ref{3d9,,shutdown()} for freeing
resources.  The class is a @ref{166,,context manager} and can be used in a
@ref{168,,with} statement to assure that resources are automatically released
when currently pending futures are done executing.

A simple of example of @ref{1c7,,ThreadPoolExecutor} is a
launch of four parallel threads for copying files:

@example
import concurrent.futures, shutil
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e:
    e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
    e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest4.txt')
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3148

@item PEP 3148@footnote{https://www.python.org/dev/peps/pep-3148} - Futures – Execute Computations Asynchronously

PEP written by Brian Quinlan.
@end table

@ref{d58,,Code for Threaded Parallel URL reads}, an
example using threads to fetch multiple web pages in parallel.

@ref{d59,,Code for computing prime numbers in parallel}, an example demonstrating
@ref{3da,,ProcessPoolExecutor}.


@node PEP 3147 PYC Repository Directories,PEP 3149 ABI Version Tagged so Files,PEP 3148 The concurrent futures module,What’s New In Python 3 2
@anchor{whatsnew/3 2 pep-3147-pyc-repository-directories}@anchor{d5a}
@subsection PEP 3147:  PYC Repository Directories


Python’s scheme for caching bytecode in @emph{.pyc} files did not work well in
environments with multiple Python interpreters.  If one interpreter encountered
a cached file created by another interpreter, it would recompile the source and
overwrite the cached file, thus losing the benefits of caching.

The issue of “pyc fights” has become more pronounced as it has become
commonplace for Linux distributions to ship with multiple versions of Python.
These conflicts also arise with CPython alternatives such as Unladen Swallow.

To solve this problem, Python’s import machinery has been extended to use
distinct filenames for each interpreter.  Instead of Python 3.2 and Python 3.3 and
Unladen Swallow each competing for a file called “mymodule.pyc”, they will now
look for “mymodule.cpython-32.pyc”, “mymodule.cpython-33.pyc”, and
“mymodule.unladen10.pyc”.  And to prevent all of these new files from
cluttering source directories, the @emph{pyc} files are now collected in a
“__pycache__” directory stored under the package directory.

Aside from the filenames and target directories, the new scheme has a few
aspects that are visible to the programmer:


@itemize *

@item 
Imported modules now have a @ref{d5b,,__cached__} attribute which stores the name
of the actual file that was imported:

@example
>>> import collections
>>> collections.__cached__ 
'c:/py32/lib/__pycache__/collections.cpython-32.pyc'
@end example

@item 
The tag that is unique to each interpreter is accessible from the @ref{95,,imp}
module:

@example
>>> import imp
>>> imp.get_tag() 
'cpython-32'
@end example

@item 
Scripts that try to deduce source filename from the imported file now need to
be smarter.  It is no longer sufficient to simply strip the “c” from a “.pyc”
filename.  Instead, use the new functions in the @ref{95,,imp} module:

@example
>>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc')
'c:/py32/lib/collections.py'
>>> imp.cache_from_source('c:/py32/lib/collections.py') 
'c:/py32/lib/__pycache__/collections.cpython-32.pyc'
@end example

@item 
The @ref{d2,,py_compile} and @ref{1f,,compileall} modules have been updated to
reflect the new naming convention and target directory.  The command-line
invocation of @emph{compileall} has new options: @code{-i} for
specifying a list of files and directories to compile and @code{-b} which causes
bytecode files to be written to their legacy location rather than
@emph{__pycache__}.

@item 
The @ref{97,,importlib.abc} module has been updated with new @ref{d5c,,abstract base classes} for loading bytecode files.  The obsolete
ABCs, @code{PyLoader} and
@code{PyPycLoader}, have been deprecated (instructions on how
to stay Python 3.1 compatible are included with the documentation).
@end itemize


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3147

@item PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} - PYC Repository Directories

PEP written by Barry Warsaw.
@end table


@node PEP 3149 ABI Version Tagged so Files,PEP 3333 Python Web Server Gateway Interface v1 0 1,PEP 3147 PYC Repository Directories,What’s New In Python 3 2
@anchor{whatsnew/3 2 pep-3149-abi-version-tagged-so-files}@anchor{d5d}
@subsection PEP 3149: ABI Version Tagged .so Files


The PYC repository directory allows multiple bytecode cache files to be
co-located.  This PEP implements a similar mechanism for shared object files by
giving them a common directory and distinct names for each version.

The common directory is “pyshared” and the file names are made distinct by
identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the
major and minor version numbers, and optional build flags (such as “d” for
debug, “m” for pymalloc, “u” for wide-unicode).  For an arbitrary package “foo”,
you may see these files when the distribution package is installed:

@example
/usr/share/pyshared/foo.cpython-32m.so
/usr/share/pyshared/foo.cpython-33md.so
@end example

In Python itself, the tags are accessible from functions in the @ref{f8,,sysconfig}
module:

@example
>>> import sysconfig
>>> sysconfig.get_config_var('SOABI')       # find the version tag
'cpython-32mu'
>>> sysconfig.get_config_var('EXT_SUFFIX')  # find the full filename extension
'.cpython-32mu.so'
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3149

@item PEP 3149@footnote{https://www.python.org/dev/peps/pep-3149} - ABI Version Tagged .so Files

PEP written by Barry Warsaw.
@end table


@node PEP 3333 Python Web Server Gateway Interface v1 0 1,Other Language Changes<10>,PEP 3149 ABI Version Tagged so Files,What’s New In Python 3 2
@anchor{whatsnew/3 2 pep-3333-python-web-server-gateway-interface-v1-0-1}@anchor{d5e}
@subsection PEP 3333: Python Web Server Gateway Interface v1.0.1


This informational PEP clarifies how bytes/text issues are to be handled by the
WSGI protocol.  The challenge is that string handling in Python 3 is most
conveniently handled with the @ref{2ef,,str} type even though the HTTP protocol
is itself bytes oriented.

The PEP differentiates so-called @emph{native strings} that are used for
request/response headers and metadata versus @emph{byte strings} which are used for
the bodies of requests and responses.

The @emph{native strings} are always of type @ref{2ef,,str} but are restricted to code
points between @emph{U+0000} through @emph{U+00FF} which are translatable to bytes using
@emph{Latin-1} encoding.  These strings are used for the keys and values in the
environment dictionary and for response headers and statuses in the
@code{start_response()} function.  They must follow 
@geindex RFC; RFC 2616
RFC 2616@footnote{https://tools.ietf.org/html/rfc2616.html} with respect to
encoding. That is, they must either be @emph{ISO-8859-1} characters or use
@geindex RFC; RFC 2047
RFC 2047@footnote{https://tools.ietf.org/html/rfc2047.html} MIME encoding.

For developers porting WSGI applications from Python 2, here are the salient
points:


@itemize *

@item 
If the app already used strings for headers in Python 2, no change is needed.

@item 
If instead, the app encoded output headers or decoded input headers, then the
headers will need to be re-encoded to Latin-1.  For example, an output header
encoded in utf-8 was using @code{h.encode('utf-8')} now needs to convert from
bytes to native strings using @code{h.encode('utf-8').decode('latin-1')}.

@item 
Values yielded by an application or sent using the @code{write()} method
must be byte strings.  The @code{start_response()} function and environ
must use native strings.  The two cannot be mixed.
@end itemize

For server implementers writing CGI-to-WSGI pathways or other CGI-style
protocols, the users must to be able access the environment using native strings
even though the underlying platform may have a different convention.  To bridge
this gap, the @ref{133,,wsgiref} module has a new function,
@ref{d5f,,wsgiref.handlers.read_environ()} for transcoding CGI variables from
@ref{d60,,os.environ} into native strings and returning a new dictionary.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3333

@item PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333} - Python Web Server Gateway Interface v1.0.1

PEP written by Phillip Eby.
@end table


@node Other Language Changes<10>,New Improved and Deprecated Modules,PEP 3333 Python Web Server Gateway Interface v1 0 1,What’s New In Python 3 2
@anchor{whatsnew/3 2 other-language-changes}@anchor{d61}
@subsection Other Language Changes


Some smaller changes made to the core Python language are:


@itemize *

@item 
String formatting for @ref{785,,format()} and @ref{784,,str.format()} gained new
capabilities for the format character @strong{#}.  Previously, for integers in
binary, octal, or hexadecimal, it caused the output to be prefixed with ‘0b’,
‘0o’, or ‘0x’ respectively.  Now it can also handle floats, complex, and
Decimal, causing the output to always have a decimal point even when no digits
follow it.

@example
>>> format(20, '#o')
'0o24'
>>> format(12.34, '#5.0f')
'  12.'
@end example

(Suggested by Mark Dickinson and implemented by Eric Smith in bpo-7094@footnote{https://bugs.python.org/issue7094}.)

@item 
There is also a new @ref{936,,str.format_map()} method that extends the
capabilities of the existing @ref{784,,str.format()} method by accepting arbitrary
@ref{d62,,mapping} objects.  This new method makes it possible to use string
formatting with any of Python’s many dictionary-like objects such as
@ref{d63,,defaultdict}, @ref{b1b,,Shelf},
@ref{759,,ConfigParser}, or @ref{30,,dbm}.  It is also useful with
custom @ref{3b0,,dict} subclasses that normalize keys before look-up or that
supply a @code{__missing__()} method for unknown keys:

@example
>>> import shelve
>>> d = shelve.open('tmp.shl')
>>> 'The @{project_name@} status is @{status@} as of @{date@}'.format_map(d)
'The testing project status is green as of February 15, 2011'

>>> class LowerCasedDict(dict):
...     def __getitem__(self, key):
...         return dict.__getitem__(self, key.lower())
>>> lcd = LowerCasedDict(part='widgets', quantity=10)
>>> 'There are @{QUANTITY@} @{Part@} in stock'.format_map(lcd)
'There are 10 widgets in stock'

>>> class PlaceholderDict(dict):
...     def __missing__(self, key):
...         return '<@{@}>'.format(key)
>>> 'Hello @{name@}, welcome to @{location@}'.format_map(PlaceholderDict())
'Hello <name>, welcome to <location>'
@end example
@end itemize

@quotation

(Suggested by Raymond Hettinger and implemented by Eric Smith in
bpo-6081@footnote{https://bugs.python.org/issue6081}.)
@end quotation


@itemize *

@item 
The interpreter can now be started with a quiet option, @code{-q}, to prevent
the copyright and version information from being displayed in the interactive
mode.  The option can be introspected using the @ref{22f,,sys.flags} attribute:

@example
$ python -q
>>> sys.flags
sys.flags(debug=0, division_warning=0, inspect=0, interactive=0,
optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0,
ignore_environment=0, verbose=0, bytes_warning=0, quiet=1)
@end example

(Contributed by Marcin Wojdyr in bpo-1772833@footnote{https://bugs.python.org/issue1772833}).

@item 
The @ref{707,,hasattr()} function works by calling @ref{708,,getattr()} and detecting
whether an exception is raised.  This technique allows it to detect methods
created dynamically by @code{__getattr__()} or @code{__getattribute__()} which
would otherwise be absent from the class dictionary.  Formerly, @emph{hasattr}
would catch any exception, possibly masking genuine errors.  Now, @emph{hasattr}
has been tightened to only catch @ref{163,,AttributeError} and let other
exceptions pass through:

@example
>>> class A:
...     @@property
...     def f(self):
...         return 1 // 0
...
>>> a = A()
>>> hasattr(a, 'f')
Traceback (most recent call last):
  ...
ZeroDivisionError: integer division or modulo by zero
@end example

(Discovered by Yury Selivanov and fixed by Benjamin Peterson; bpo-9666@footnote{https://bugs.python.org/issue9666}.)

@item 
The @ref{2ef,,str()} of a float or complex number is now the same as its
@ref{27d,,repr()}. Previously, the @ref{2ef,,str()} form was shorter but that just
caused confusion and is no longer needed now that the shortest possible
@ref{27d,,repr()} is displayed by default:

@example
>>> import math
>>> repr(math.pi)
'3.141592653589793'
>>> str(math.pi)
'3.141592653589793'
@end example

(Proposed and implemented by Mark Dickinson; bpo-9337@footnote{https://bugs.python.org/issue9337}.)

@item 
@ref{573,,memoryview} objects now have a @ref{d64,,release()} method
and they also now support the context management protocol.  This allows timely
release of any resources that were acquired when requesting a buffer from the
original object.

@example
>>> with memoryview(b'abcdefgh') as v:
...     print(v.tolist())
[97, 98, 99, 100, 101, 102, 103, 104]
@end example

(Added by Antoine Pitrou; bpo-9757@footnote{https://bugs.python.org/issue9757}.)

@item 
Previously it was illegal to delete a name from the local namespace if it
occurs as a free variable in a nested block:

@example
def outer(x):
    def inner():
        return x
    inner()
    del x
@end example

This is now allowed.  Remember that the target of an @ref{16e,,except} clause
is cleared, so this code which used to work with Python 2.6, raised a
@ref{21a,,SyntaxError} with Python 3.1 and now works again:

@example
def f():
    def print_error():
        print(e)
    try:
        something
    except Exception as e:
        print_error()
        # implicit "del e" here
@end example

(See bpo-4617@footnote{https://bugs.python.org/issue4617}.)

@item 
The internal @code{structsequence} tool now creates subclasses of tuple.
This means that C structures like those returned by @ref{516,,os.stat()},
@ref{d65,,time.gmtime()}, and @ref{d45,,sys.version_info} now work like a
@ref{ce9,,named tuple} and now work with functions and methods that
expect a tuple as an argument.  This is a big step forward in making the C
structures as flexible as their pure Python counterparts:

@example
>>> import sys
>>> isinstance(sys.version_info, tuple)
True
>>> 'Version %d.%d.%d %s(%d)' % sys.version_info 
'Version 3.2.0 final(0)'
@end example

(Suggested by Arfrever Frehtes Taifersar Arahesis and implemented
by Benjamin Peterson in bpo-8413@footnote{https://bugs.python.org/issue8413}.)

@item 
Warnings are now easier to control using the 
@geindex PYTHONWARNINGS
@geindex environment variable; PYTHONWARNINGS
@ref{6de,,PYTHONWARNINGS}
environment variable as an alternative to using @code{-W} at the command line:

@example
$ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::'
@end example

(Suggested by Barry Warsaw and implemented by Philip Jenvey in bpo-7301@footnote{https://bugs.python.org/issue7301}.)

@item 
A new warning category, @ref{77b,,ResourceWarning}, has been added.  It is
emitted when potential issues with resource consumption or cleanup
are detected.  It is silenced by default in normal release builds but
can be enabled through the means provided by the @ref{12d,,warnings}
module, or on the command line.

A @ref{77b,,ResourceWarning} is issued at interpreter shutdown if the
@ref{d66,,gc.garbage} list isn’t empty, and if @ref{d67,,gc.DEBUG_UNCOLLECTABLE} is
set, all uncollectable objects are printed.  This is meant to make the
programmer aware that their code contains object finalization issues.

A @ref{77b,,ResourceWarning} is also issued when a @ref{d68,,file object} is destroyed
without having been explicitly closed.  While the deallocator for such
object ensures it closes the underlying operating system resource
(usually, a file descriptor), the delay in deallocating the object could
produce various issues, especially under Windows.  Here is an example
of enabling the warning from the command line:

@example
$ python -q -Wdefault
>>> f = open("foo", "wb")
>>> del f
__main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'>
@end example

(Added by Antoine Pitrou and Georg Brandl in bpo-10093@footnote{https://bugs.python.org/issue10093} and bpo-477863@footnote{https://bugs.python.org/issue477863}.)

@item 
@ref{438,,range} objects now support @emph{index} and @emph{count} methods. This is part
of an effort to make more objects fully implement the
@code{collections.Sequence} @ref{d5c,,abstract base class}.  As a result, the
language will have a more uniform API.  In addition, @ref{438,,range} objects
now support slicing and negative indices, even with values larger than
@ref{d69,,sys.maxsize}.  This makes @emph{range} more interoperable with lists:

@example
>>> range(0, 100, 2).count(10)
1
>>> range(0, 100, 2).index(10)
5
>>> range(0, 100, 2)[5]
10
>>> range(0, 100, 2)[0:5]
range(0, 10, 2)
@end example

(Contributed by Daniel Stutzbach in bpo-9213@footnote{https://bugs.python.org/issue9213}, by Alexander Belopolsky
in bpo-2690@footnote{https://bugs.python.org/issue2690}, and by Nick Coghlan in bpo-10889@footnote{https://bugs.python.org/issue10889}.)

@item 
The @ref{d6a,,callable()} builtin function from Py2.x was resurrected.  It provides
a concise, readable alternative to using an @ref{d5c,,abstract base class} in an
expression like @code{isinstance(x, collections.Callable)}:

@example
>>> callable(max)
True
>>> callable(20)
False
@end example

(See bpo-10518@footnote{https://bugs.python.org/issue10518}.)

@item 
Python’s import mechanism can now load modules installed in directories with
non-ASCII characters in the path name.  This solved an aggravating problem
with home directories for users with non-ASCII characters in their usernames.
@end itemize

@quotation

(Required extensive work by Victor Stinner in bpo-9425@footnote{https://bugs.python.org/issue9425}.)
@end quotation

@node New Improved and Deprecated Modules,Multi-threading,Other Language Changes<10>,What’s New In Python 3 2
@anchor{whatsnew/3 2 new-improved-and-deprecated-modules}@anchor{d6b}
@subsection New, Improved, and Deprecated Modules


Python’s standard library has undergone significant maintenance efforts and
quality improvements.

The biggest news for Python 3.2 is that the @ref{64,,email} package, @ref{aa,,mailbox}
module, and @ref{bc,,nntplib} modules now work correctly with the bytes/text model
in Python 3.  For the first time, there is correct handling of messages with
mixed encodings.

Throughout the standard library, there has been more careful attention to
encodings and text versus bytes issues.  In particular, interactions with the
operating system are now better able to exchange non-ASCII data using the
Windows MBCS encoding, locale-aware encodings, or UTF-8.

Another significant win is the addition of substantially better support for
@emph{SSL} connections and security certificates.

In addition, more classes now implement a @ref{166,,context manager} to support
convenient and reliable resource clean-up using a @ref{168,,with} statement.

@menu
* email: email<5>. 
* elementtree:: 
* functools: functools<6>. 
* itertools: itertools<4>. 
* collections: collections<8>. 
* threading: threading<8>. 
* datetime and time:: 
* math: math<8>. 
* abc: abc<3>. 
* io: io<5>. 
* reprlib:: 
* logging: logging<7>. 
* csv: csv<3>. 
* contextlib: contextlib<7>. 
* decimal and fractions:: 
* ftp:: 
* popen:: 
* select: select<3>. 
* gzip and zipfile:: 
* tarfile: tarfile<5>. 
* hashlib: hashlib<5>. 
* ast: ast<4>. 
* os: os<11>. 
* shutil: shutil<5>. 
* sqlite3: sqlite3<8>. 
* html: html<3>. 
* socket: socket<11>. 
* ssl: ssl<10>. 
* nntp:: 
* certificates:: 
* imaplib: imaplib<4>. 
* http.client: http client<4>. 
* unittest: unittest<7>. 
* random: random<3>. 
* poplib: poplib<4>. 
* asyncore: asyncore<2>. 
* tempfile: tempfile<2>. 
* inspect: inspect<9>. 
* pydoc: pydoc<6>. 
* dis: dis<3>. 
* dbm: dbm<6>. 
* ctypes: ctypes<2>. 
* site: site<3>. 
* sysconfig: sysconfig<2>. 
* pdb: pdb<6>. 
* configparser: configparser<2>. 
* urllib.parse: urllib parse<4>. 
* mailbox:: 
* turtledemo:: 

@end menu

@node email<5>,elementtree,,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 email}@anchor{d6c}
@subsubsection email


The usability of the @ref{64,,email} package in Python 3 has been mostly fixed by
the extensive efforts of R. David Murray.  The problem was that emails are
typically read and stored in the form of @ref{2f0,,bytes} rather than @ref{2ef,,str}
text, and they may contain multiple encodings within a single email.  So, the
email package had to be extended to parse and generate email messages in bytes
format.


@itemize *

@item 
New functions @ref{d6d,,message_from_bytes()} and
@ref{d6e,,message_from_binary_file()}, and new classes
@ref{d6f,,BytesFeedParser} and @ref{d70,,BytesParser}
allow binary message data to be parsed into model objects.

@item 
Given bytes input to the model, @ref{d71,,get_payload()}
will by default decode a message body that has a
@code{Content-Transfer-Encoding} of @emph{8bit} using the charset
specified in the MIME headers and return the resulting string.

@item 
Given bytes input to the model, @ref{d72,,Generator} will
convert message bodies that have a @code{Content-Transfer-Encoding} of
@emph{8bit} to instead have a @emph{7bit} @code{Content-Transfer-Encoding}.

Headers with unencoded non-ASCII bytes are deemed to be 
@geindex RFC; RFC 2047
RFC 2047@footnote{https://tools.ietf.org/html/rfc2047.html}-encoded
using the @emph{unknown-8bit} character set.

@item 
A new class @ref{d73,,BytesGenerator} produces bytes as output,
preserving any unchanged non-ASCII data that was present in the input used to
build the model, including message bodies with a
@code{Content-Transfer-Encoding} of @emph{8bit}.

@item 
The @ref{e8,,smtplib} @ref{421,,SMTP} class now accepts a byte string
for the @emph{msg} argument to the @ref{9c3,,sendmail()} method,
and a new method, @ref{9c4,,send_message()} accepts a
@ref{7e3,,Message} object and can optionally obtain the
@emph{from_addr} and @emph{to_addrs} addresses directly from the object.
@end itemize

(Proposed and implemented by R. David Murray, bpo-4661@footnote{https://bugs.python.org/issue4661} and bpo-10321@footnote{https://bugs.python.org/issue10321}.)

@node elementtree,functools<6>,email<5>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 elementtree}@anchor{d74}
@subsubsection elementtree


The @ref{13e,,xml.etree.ElementTree} package and its @code{xml.etree.cElementTree}
counterpart have been updated to version 1.3.

Several new and useful functions and methods have been added:


@itemize *

@item 
@ref{d75,,xml.etree.ElementTree.fromstringlist()} which builds an XML document
from a sequence of fragments

@item 
@ref{d76,,xml.etree.ElementTree.register_namespace()} for registering a global
namespace prefix

@item 
@ref{b7c,,xml.etree.ElementTree.tostringlist()} for string representation
including all sublists

@item 
@ref{d77,,xml.etree.ElementTree.Element.extend()} for appending a sequence of zero
or more elements

@item 
@ref{d78,,xml.etree.ElementTree.Element.iterfind()} searches an element and
subelements

@item 
@ref{d79,,xml.etree.ElementTree.Element.itertext()} creates a text iterator over
an element and its subelements

@item 
@ref{d7a,,xml.etree.ElementTree.TreeBuilder.end()} closes the current element

@item 
@ref{5b6,,xml.etree.ElementTree.TreeBuilder.doctype()} handles a doctype
declaration
@end itemize

Two methods have been deprecated:


@itemize *

@item 
@code{xml.etree.ElementTree.getchildren()} use @code{list(elem)} instead.

@item 
@code{xml.etree.ElementTree.getiterator()} use @code{Element.iter} instead.
@end itemize

For details of the update, see Introducing ElementTree@footnote{http://effbot.org/zone/elementtree-13-intro.htm} on Fredrik Lundh’s website.

(Contributed by Florent Xicluna and Fredrik Lundh, bpo-6472@footnote{https://bugs.python.org/issue6472}.)

@node functools<6>,itertools<4>,elementtree,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 functools}@anchor{d7b}
@subsubsection functools



@itemize *

@item 
The @ref{7f,,functools} module includes a new decorator for caching function
calls.  @ref{4f3,,functools.lru_cache()} can save repeated queries to an external
resource whenever the results are expected to be the same.

For example, adding a caching decorator to a database query function can save
database accesses for popular searches:

@example
>>> import functools
>>> @@functools.lru_cache(maxsize=300)
... def get_phone_number(name):
...     c = conn.cursor()
...     c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,))
...     return c.fetchone()[0]
@end example

@example
>>> for name in user_requests:        
...     get_phone_number(name)        # cached lookup
@end example

To help with choosing an effective cache size, the wrapped function is
instrumented for tracking cache statistics:

@example
>>> get_phone_number.cache_info()     
CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300)
@end example

If the phonelist table gets updated, the outdated contents of the cache can be
cleared with:

@example
>>> get_phone_number.cache_clear()
@end example

(Contributed by Raymond Hettinger and incorporating design ideas from Jim
Baker, Miki Tebeka, and Nick Coghlan; see recipe 498245@footnote{https://code.activestate.com/recipes/498245}, recipe 577479@footnote{https://code.activestate.com/recipes/577479}, bpo-10586@footnote{https://bugs.python.org/issue10586}, and
bpo-10593@footnote{https://bugs.python.org/issue10593}.)

@item 
The @ref{adb,,functools.wraps()} decorator now adds a @code{__wrapped__} attribute
pointing to the original callable function.  This allows wrapped functions to
be introspected.  It also copies @code{__annotations__} if defined.  And now
it also gracefully skips over missing attributes such as @code{__doc__} which
might not be defined for the wrapped callable.

In the above example, the cache can be removed by recovering the original
function:

@example
>>> get_phone_number = get_phone_number.__wrapped__    # uncached function
@end example

(By Nick Coghlan and Terrence Cole; bpo-9567@footnote{https://bugs.python.org/issue9567}, bpo-3445@footnote{https://bugs.python.org/issue3445}, and
bpo-8814@footnote{https://bugs.python.org/issue8814}.)

@item 
To help write classes with rich comparison methods, a new decorator
@ref{abf,,functools.total_ordering()} will use existing equality and inequality
methods to fill in the remaining methods.

For example, supplying @emph{__eq__} and @emph{__lt__} will enable
@ref{abf,,total_ordering()} to fill-in @emph{__le__}, @emph{__gt__} and @emph{__ge__}:

@example
@@total_ordering
class Student:
    def __eq__(self, other):
        return ((self.lastname.lower(), self.firstname.lower()) ==
                (other.lastname.lower(), other.firstname.lower()))

    def __lt__(self, other):
        return ((self.lastname.lower(), self.firstname.lower()) <
                (other.lastname.lower(), other.firstname.lower()))
@end example

With the @emph{total_ordering} decorator, the remaining comparison methods
are filled in automatically.

(Contributed by Raymond Hettinger.)

@item 
To aid in porting programs from Python 2, the @ref{d7c,,functools.cmp_to_key()}
function converts an old-style comparison function to
modern @ref{964,,key function}:

@example
>>> # locale-aware sort order
>>> sorted(iterable, key=cmp_to_key(locale.strcoll)) 
@end example

For sorting examples and a brief sorting tutorial, see the Sorting HowTo@footnote{https://wiki.python.org/moin/HowTo/Sorting/} tutorial.

(Contributed by Raymond Hettinger.)
@end itemize

@node itertools<4>,collections<8>,functools<6>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 itertools}@anchor{d7d}
@subsubsection itertools



@itemize *

@item 
The @ref{9f,,itertools} module has a new @ref{505,,accumulate()} function
modeled on APL’s @emph{scan} operator and Numpy’s @emph{accumulate} function:

@example
>>> from itertools import accumulate
>>> list(accumulate([8, 2, 50]))
[8, 10, 60]
@end example

@example
>>> prob_dist = [0.1, 0.4, 0.2, 0.3]
>>> list(accumulate(prob_dist))      # cumulative probability distribution
[0.1, 0.5, 0.7, 1.0]
@end example

For an example using @ref{505,,accumulate()}, see the @ref{d7e,,examples for the random module}.

(Contributed by Raymond Hettinger and incorporating design suggestions
from Mark Dickinson.)
@end itemize

@node collections<8>,threading<8>,itertools<4>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 collections}@anchor{d7f}
@subsubsection collections



@itemize *

@item 
The @ref{c2a,,collections.Counter} class now has two forms of in-place
subtraction, the existing @emph{-=} operator for saturating subtraction@footnote{https://en.wikipedia.org/wiki/Saturation_arithmetic} and the new
@ref{d80,,subtract()} method for regular subtraction.  The
former is suitable for multisets@footnote{https://en.wikipedia.org/wiki/Multiset}
which only have positive counts, and the latter is more suitable for use cases
that allow negative counts:

@example
>>> from collections import Counter
>>> tally = Counter(dogs=5, cats=3)
>>> tally -= Counter(dogs=2, cats=8)    # saturating subtraction
>>> tally
Counter(@{'dogs': 3@})
@end example

@example
>>> tally = Counter(dogs=5, cats=3)
>>> tally.subtract(dogs=2, cats=8)      # regular subtraction
>>> tally
Counter(@{'dogs': 3, 'cats': -5@})
@end example

(Contributed by Raymond Hettinger.)

@item 
The @ref{4e5,,collections.OrderedDict} class has a new method
@ref{d81,,move_to_end()} which takes an existing key and
moves it to either the first or last position in the ordered sequence.

The default is to move an item to the last position.  This is equivalent of
renewing an entry with @code{od[k] = od.pop(k)}.

A fast move-to-end operation is useful for resequencing entries.  For example,
an ordered dictionary can be used to track order of access by aging entries
from the oldest to the most recently accessed.

@example
>>> from collections import OrderedDict
>>> d = OrderedDict.fromkeys(['a', 'b', 'X', 'd', 'e'])
>>> list(d)
['a', 'b', 'X', 'd', 'e']
>>> d.move_to_end('X')
>>> list(d)
['a', 'b', 'd', 'e', 'X']
@end example

(Contributed by Raymond Hettinger.)

@item 
The @ref{7d3,,collections.deque} class grew two new methods
@ref{d82,,count()} and @ref{d83,,reverse()} that
make them more substitutable for @ref{43a,,list} objects:

@example
>>> from collections import deque
>>> d = deque('simsalabim')
>>> d.count('s')
2
>>> d.reverse()
>>> d
deque(['m', 'i', 'b', 'a', 'l', 'a', 's', 'm', 'i', 's'])
@end example

(Contributed by Raymond Hettinger.)
@end itemize

@node threading<8>,datetime and time,collections<8>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 threading}@anchor{d84}
@subsubsection threading


The @ref{109,,threading} module has a new @ref{d85,,Barrier}
synchronization class for making multiple threads wait until all of them have
reached a common barrier point.  Barriers are useful for making sure that a task
with multiple preconditions does not run until all of the predecessor tasks are
complete.

Barriers can work with an arbitrary number of threads.  This is a generalization
of a Rendezvous@footnote{https://en.wikipedia.org/wiki/Synchronous_rendezvous} which
is defined for only two threads.

Implemented as a two-phase cyclic barrier, @ref{d85,,Barrier} objects
are suitable for use in loops.  The separate @emph{filling} and @emph{draining} phases
assure that all threads get released (drained) before any one of them can loop
back and re-enter the barrier.  The barrier fully resets after each cycle.

Example of using barriers:

@example
from threading import Barrier, Thread

def get_votes(site):
    ballots = conduct_election(site)
    all_polls_closed.wait()        # do not count until all polls are closed
    totals = summarize(ballots)
    publish(site, totals)

all_polls_closed = Barrier(len(sites))
for site in sites:
    Thread(target=get_votes, args=(site,)).start()
@end example

In this example, the barrier enforces a rule that votes cannot be counted at any
polling site until all polls are closed.  Notice how a solution with a barrier
is similar to one with @ref{d86,,threading.Thread.join()}, but the threads stay alive
and continue to do work (summarizing ballots) after the barrier point is
crossed.

If any of the predecessor tasks can hang or be delayed, a barrier can be created
with an optional @emph{timeout} parameter.  Then if the timeout period elapses before
all the predecessor tasks reach the barrier point, all waiting threads are
released and a @ref{d87,,BrokenBarrierError} exception is raised:

@example
def get_votes(site):
    ballots = conduct_election(site)
    try:
        all_polls_closed.wait(timeout=midnight - time.now())
    except BrokenBarrierError:
        lockbox = seal_ballots(ballots)
        queue.put(lockbox)
    else:
        totals = summarize(ballots)
        publish(site, totals)
@end example

In this example, the barrier enforces a more robust rule.  If some election
sites do not finish before midnight, the barrier times-out and the ballots are
sealed and deposited in a queue for later handling.

See Barrier Synchronization Patterns@footnote{http://osl.cs.illinois.edu/media/papers/karmani-2009-barrier_synchronization_pattern.pdf}
for more examples of how barriers can be used in parallel computing.  Also, there is
a simple but thorough explanation of barriers in The Little Book of Semaphores@footnote{https://greenteapress.com/semaphores/LittleBookOfSemaphores.pdf}, @emph{section 3.6}.

(Contributed by Kristján Valur Jónsson with an API review by Jeffrey Yasskin in
bpo-8777@footnote{https://bugs.python.org/issue8777}.)

@node datetime and time,math<8>,threading<8>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 datetime-and-time}@anchor{d88}
@subsubsection datetime and time



@itemize *

@item 
The @ref{2f,,datetime} module has a new type @ref{c51,,timezone} that
implements the @ref{3c6,,tzinfo} interface by returning a fixed UTC
offset and timezone name. This makes it easier to create timezone-aware
datetime objects:

@example
>>> from datetime import datetime, timezone

>>> datetime.now(timezone.utc)
datetime.datetime(2010, 12, 8, 21, 4, 2, 923754, tzinfo=datetime.timezone.utc)

>>> datetime.strptime("01/01/2000 12:00 +0000", "%m/%d/%Y %H:%M %z")
datetime.datetime(2000, 1, 1, 12, 0, tzinfo=datetime.timezone.utc)
@end example

@item 
Also, @ref{4c8,,timedelta} objects can now be multiplied by
@ref{256,,float} and divided by @ref{256,,float} and @ref{1b1,,int} objects.
And @ref{4c8,,timedelta} objects can now divide one another.

@item 
The @ref{7da,,datetime.date.strftime()} method is no longer restricted to years
after 1900.  The new supported year range is from 1000 to 9999 inclusive.

@item 
Whenever a two-digit year is used in a time tuple, the interpretation has been
governed by @code{time.accept2dyear}.  The default is @code{True} which means that
for a two-digit year, the century is guessed according to the POSIX rules
governing the @code{%y} strptime format.

Starting with Py3.2, use of the century guessing heuristic will emit a
@ref{1b2,,DeprecationWarning}.  Instead, it is recommended that
@code{time.accept2dyear} be set to @code{False} so that large date ranges
can be used without guesswork:

@example
>>> import time, warnings
>>> warnings.resetwarnings()      # remove the default warning filters

>>> time.accept2dyear = True      # guess whether 11 means 11 or 2011
>>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0))
Warning (from warnings module):
  ...
DeprecationWarning: Century info guessed for a 2-digit year.
'Fri Jan  1 12:34:56 2011'

>>> time.accept2dyear = False     # use the full range of allowable dates
>>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0))
'Fri Jan  1 12:34:56 11'
@end example

Several functions now have significantly expanded date ranges.  When
@code{time.accept2dyear} is false, the @ref{d89,,time.asctime()} function will
accept any year that fits in a C int, while the @ref{d8a,,time.mktime()} and
@ref{d8b,,time.strftime()} functions will accept the full range supported by the
corresponding operating system functions.
@end itemize

(Contributed by Alexander Belopolsky and Victor Stinner in bpo-1289118@footnote{https://bugs.python.org/issue1289118},
bpo-5094@footnote{https://bugs.python.org/issue5094}, bpo-6641@footnote{https://bugs.python.org/issue6641}, bpo-2706@footnote{https://bugs.python.org/issue2706}, bpo-1777412@footnote{https://bugs.python.org/issue1777412}, bpo-8013@footnote{https://bugs.python.org/issue8013},
and bpo-10827@footnote{https://bugs.python.org/issue10827}.)

@c XXX https://bugs.python.org/issue?%40search_text=datetime&%40sort=-activity

@node math<8>,abc<3>,datetime and time,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 math}@anchor{d8c}
@subsubsection math


The @ref{ad,,math} module has been updated with six new functions inspired by the
C99 standard.

The @ref{d8d,,isfinite()} function provides a reliable and fast way to detect
special values.  It returns @code{True} for regular numbers and @code{False} for @emph{Nan} or
@emph{Infinity}:

@example
>>> from math import isfinite
>>> [isfinite(x) for x in (123, 4.56, float('Nan'), float('Inf'))]
[True, True, False, False]
@end example

The @ref{d8e,,expm1()} function computes @code{e**x-1} for small values of @emph{x}
without incurring the loss of precision that usually accompanies the subtraction
of nearly equal quantities:

@example
>>> from math import expm1
>>> expm1(0.013671875)   # more accurate way to compute e**x-1 for a small x
0.013765762467652909
@end example

The @ref{710,,erf()} function computes a probability integral or Gaussian error function@footnote{https://en.wikipedia.org/wiki/Error_function}.  The
complementary error function, @ref{711,,erfc()}, is @code{1 - erf(x)}:

@example
>>> from math import erf, erfc, sqrt
>>> erf(1.0/sqrt(2.0))   # portion of normal distribution within 1 standard deviation
0.682689492137086
>>> erfc(1.0/sqrt(2.0))  # portion of normal distribution outside 1 standard deviation
0.31731050786291404
>>> erf(1.0/sqrt(2.0)) + erfc(1.0/sqrt(2.0))
1.0
@end example

The @ref{d8f,,gamma()} function is a continuous extension of the factorial
function.  See @indicateurl{https://en.wikipedia.org/wiki/Gamma_function} for details.  Because
the function is related to factorials, it grows large even for small values of
@emph{x}, so there is also a @ref{d90,,lgamma()} function for computing the natural
logarithm of the gamma function:

@example
>>> from math import gamma, lgamma
>>> gamma(7.0)           # six factorial
720.0
>>> lgamma(801.0)        # log(800 factorial)
4551.950730698041
@end example

(Contributed by Mark Dickinson.)

@node abc<3>,io<5>,math<8>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 abc}@anchor{d91}
@subsubsection abc


The @ref{3,,abc} module now supports @ref{c1f,,abstractclassmethod()} and
@ref{c20,,abstractstaticmethod()}.

These tools make it possible to define an @ref{d5c,,abstract base class} that
requires a particular @ref{254,,classmethod()} or @ref{253,,staticmethod()} to be
implemented:

@example
class Temperature(metaclass=abc.ABCMeta):
    @@abc.abstractclassmethod
    def from_fahrenheit(cls, t):
        ...
    @@abc.abstractclassmethod
    def from_celsius(cls, t):
        ...
@end example

(Patch submitted by Daniel Urban; bpo-5867@footnote{https://bugs.python.org/issue5867}.)

@node io<5>,reprlib,abc<3>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 io}@anchor{d92}
@subsubsection io


The @ref{a15,,io.BytesIO} has a new method, @ref{d93,,getbuffer()}, which
provides functionality similar to @ref{573,,memoryview()}.  It creates an editable
view of the data without making a copy.  The buffer’s random access and support
for slice notation are well-suited to in-place editing:

@example
>>> REC_LEN, LOC_START, LOC_LEN = 34, 7, 11

>>> def change_location(buffer, record_number, location):
...     start = record_number * REC_LEN + LOC_START
...     buffer[start: start+LOC_LEN] = location

>>> import io

>>> byte_stream = io.BytesIO(
...     b'G3805  storeroom  Main chassis    '
...     b'X7899  shipping   Reserve cog     '
...     b'L6988  receiving  Primary sprocket'
... )
>>> buffer = byte_stream.getbuffer()
>>> change_location(buffer, 1, b'warehouse  ')
>>> change_location(buffer, 0, b'showroom   ')
>>> print(byte_stream.getvalue())
b'G3805  showroom   Main chassis    '
b'X7899  warehouse  Reserve cog     '
b'L6988  receiving  Primary sprocket'
@end example

(Contributed by Antoine Pitrou in bpo-5506@footnote{https://bugs.python.org/issue5506}.)

@node reprlib,logging<7>,io<5>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 reprlib}@anchor{d94}
@subsubsection reprlib


When writing a @code{__repr__()} method for a custom container, it is easy to
forget to handle the case where a member refers back to the container itself.
Python’s builtin objects such as @ref{43a,,list} and @ref{370,,set} handle
self-reference by displaying “…” in the recursive part of the representation
string.

To help write such @code{__repr__()} methods, the @ref{da,,reprlib} module has a new
decorator, @ref{d95,,recursive_repr()}, for detecting recursive calls to
@code{__repr__()} and substituting a placeholder string instead:

@example
>>> class MyList(list):
...     @@recursive_repr()
...     def __repr__(self):
...         return '<' + '|'.join(map(repr, self)) + '>'
...
>>> m = MyList('abc')
>>> m.append(m)
>>> m.append('x')
>>> print(m)
<'a'|'b'|'c'|...|'x'>
@end example

(Contributed by Raymond Hettinger in bpo-9826@footnote{https://bugs.python.org/issue9826} and bpo-9840@footnote{https://bugs.python.org/issue9840}.)

@node logging<7>,csv<3>,reprlib,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 logging}@anchor{d96}
@subsubsection logging


In addition to dictionary-based configuration described above, the
@ref{a6,,logging} package has many other improvements.

The logging documentation has been augmented by a @ref{d97,,basic tutorial}, an @ref{d98,,advanced tutorial}, and a @ref{d99,,cookbook} of
logging recipes.  These documents are the fastest way to learn about logging.

The @ref{508,,logging.basicConfig()} set-up function gained a @emph{style} argument to
support three different types of string formatting.  It defaults to “%” for
traditional %-formatting, can be set to “@{” for the new @ref{784,,str.format()} style, or
can be set to “$” for the shell-style formatting provided by
@ref{6b5,,string.Template}.  The following three configurations are equivalent:

@example
>>> from logging import basicConfig
>>> basicConfig(style='%', format="%(name)s -> %(levelname)s: %(message)s")
>>> basicConfig(style='@{', format="@{name@} -> @{levelname@} @{message@}")
>>> basicConfig(style='$', format="$name -> $levelname: $message")
@end example

If no configuration is set-up before a logging event occurs, there is now a
default configuration using a @ref{d9a,,StreamHandler} directed to
@ref{42d,,sys.stderr} for events of @code{WARNING} level or higher.  Formerly, an
event occurring before a configuration was set-up would either raise an
exception or silently drop the event depending on the value of
@code{logging.raiseExceptions}.  The new default handler is stored in
@ref{d9b,,logging.lastResort}.

The use of filters has been simplified.  Instead of creating a
@ref{d9c,,Filter} object, the predicate can be any Python callable that
returns @code{True} or @code{False}.

There were a number of other improvements that add flexibility and simplify
configuration.  See the module documentation for a full listing of changes in
Python 3.2.

@node csv<3>,contextlib<7>,logging<7>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 csv}@anchor{d9d}
@subsubsection csv


The @ref{28,,csv} module now supports a new dialect, @ref{d9e,,unix_dialect},
which applies quoting for all fields and a traditional Unix style with @code{'\n'} as
the line terminator.  The registered dialect name is @code{unix}.

The @ref{d9f,,csv.DictWriter} has a new method,
@ref{da0,,writeheader()} for writing-out an initial row to document
the field names:

@example
>>> import csv, sys
>>> w = csv.DictWriter(sys.stdout, ['name', 'dept'], dialect='unix')
>>> w.writeheader()
"name","dept"
>>> w.writerows([
...     @{'name': 'tom', 'dept': 'accounting'@},
...     @{'name': 'susan', 'dept': 'Salesl'@}])
"tom","accounting"
"susan","sales"
@end example

(New dialect suggested by Jay Talbot in bpo-5975@footnote{https://bugs.python.org/issue5975}, and the new method
suggested by Ed Abraham in bpo-1537721@footnote{https://bugs.python.org/issue1537721}.)

@node contextlib<7>,decimal and fractions,csv<3>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 contextlib}@anchor{da1}
@subsubsection contextlib


There is a new and slightly mind-blowing tool
@ref{da2,,ContextDecorator} that is helpful for creating a
@ref{166,,context manager} that does double duty as a function decorator.

As a convenience, this new functionality is used by
@ref{da3,,contextmanager()} so that no extra effort is needed to support
both roles.

The basic idea is that both context managers and function decorators can be used
for pre-action and post-action wrappers.  Context managers wrap a group of
statements using a @ref{168,,with} statement, and function decorators wrap a
group of statements enclosed in a function.  So, occasionally there is a need to
write a pre-action or post-action wrapper that can be used in either role.

For example, it is sometimes useful to wrap functions or groups of statements
with a logger that can track the time of entry and time of exit.  Rather than
writing both a function decorator and a context manager for the task, the
@ref{da3,,contextmanager()} provides both capabilities in a single
definition:

@example
from contextlib import contextmanager
import logging

logging.basicConfig(level=logging.INFO)

@@contextmanager
def track_entry_and_exit(name):
    logging.info('Entering: %s', name)
    yield
    logging.info('Exiting: %s', name)
@end example

Formerly, this would have only been usable as a context manager:

@example
with track_entry_and_exit('widget loader'):
    print('Some time consuming activity goes here')
    load_widget()
@end example

Now, it can be used as a decorator as well:

@example
@@track_entry_and_exit('widget loader')
def activity():
    print('Some time consuming activity goes here')
    load_widget()
@end example

Trying to fulfill two roles at once places some limitations on the technique.
Context managers normally have the flexibility to return an argument usable by
a @ref{168,,with} statement, but there is no parallel for function decorators.

In the above example, there is not a clean way for the @emph{track_entry_and_exit}
context manager to return a logging instance for use in the body of enclosed
statements.

(Contributed by Michael Foord in bpo-9110@footnote{https://bugs.python.org/issue9110}.)

@node decimal and fractions,ftp,contextlib<7>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 decimal-and-fractions}@anchor{da4}
@subsubsection decimal and fractions


Mark Dickinson crafted an elegant and efficient scheme for assuring that
different numeric datatypes will have the same hash value whenever their actual
values are equal (bpo-8188@footnote{https://bugs.python.org/issue8188}):

@example
assert hash(Fraction(3, 2)) == hash(1.5) == \
       hash(Decimal("1.5")) == hash(complex(1.5, 0))
@end example

Some of the hashing details are exposed through a new attribute,
@ref{b85,,sys.hash_info}, which describes the bit width of the hash value, the
prime modulus, the hash values for @emph{infinity} and @emph{nan}, and the multiplier
used for the imaginary part of a number:

@example
>>> sys.hash_info 
sys.hash_info(width=64, modulus=2305843009213693951, inf=314159, nan=0, imag=1000003)
@end example

An early decision to limit the inter-operability of various numeric types has
been relaxed.  It is still unsupported (and ill-advised) to have implicit
mixing in arithmetic expressions such as @code{Decimal('1.1') + float('1.1')}
because the latter loses information in the process of constructing the binary
float.  However, since existing floating point value can be converted losslessly
to either a decimal or rational representation, it makes sense to add them to
the constructor and to support mixed-type comparisons.


@itemize *

@item 
The @ref{248,,decimal.Decimal} constructor now accepts @ref{256,,float} objects
directly so there in no longer a need to use the @ref{da5,,from_float()}
method (bpo-8257@footnote{https://bugs.python.org/issue8257}).

@item 
Mixed type comparisons are now fully supported so that
@ref{248,,Decimal} objects can be directly compared with @ref{256,,float}
and @ref{172,,fractions.Fraction} (bpo-2531@footnote{https://bugs.python.org/issue2531} and bpo-8188@footnote{https://bugs.python.org/issue8188}).
@end itemize

Similar changes were made to @ref{172,,fractions.Fraction} so that the
@ref{da6,,from_float()} and @ref{da7,,from_decimal()}
methods are no longer needed (bpo-8294@footnote{https://bugs.python.org/issue8294}):

@example
>>> from decimal import Decimal
>>> from fractions import Fraction
>>> Decimal(1.1)
Decimal('1.100000000000000088817841970012523233890533447265625')
>>> Fraction(1.1)
Fraction(2476979795053773, 2251799813685248)
@end example

Another useful change for the @ref{34,,decimal} module is that the
@code{Context.clamp} attribute is now public.  This is useful in creating
contexts that correspond to the decimal interchange formats specified in IEEE
754 (see bpo-8540@footnote{https://bugs.python.org/issue8540}).

(Contributed by Mark Dickinson and Raymond Hettinger.)

@node ftp,popen,decimal and fractions,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 ftp}@anchor{da8}
@subsubsection ftp


The @ref{3e9,,ftplib.FTP} class now supports the context management protocol to
unconditionally consume @ref{beb,,socket.error} exceptions and to close the FTP
connection when done:

@example
>>> from ftplib import FTP
>>> with FTP("ftp1.at.proftpd.org") as ftp:
        ftp.login()
        ftp.dir()

'230 Anonymous login ok, restrictions apply.'
dr-xr-xr-x   9 ftp      ftp           154 May  6 10:43 .
dr-xr-xr-x   9 ftp      ftp           154 May  6 10:43 ..
dr-xr-xr-x   5 ftp      ftp          4096 May  6 10:43 CentOS
dr-xr-xr-x   3 ftp      ftp            18 Jul 10  2008 Fedora
@end example

Other file-like objects such as @ref{511,,mmap.mmap} and @ref{280,,fileinput.input()}
also grew auto-closing context managers:

@example
with fileinput.input(files=('log1.txt', 'log2.txt')) as f:
    for line in f:
        process(line)
@end example

(Contributed by Tarek Ziadé and Giampaolo Rodolà in bpo-4972@footnote{https://bugs.python.org/issue4972}, and
by Georg Brandl in bpo-8046@footnote{https://bugs.python.org/issue8046} and bpo-1286@footnote{https://bugs.python.org/issue1286}.)

The @ref{3ea,,FTP_TLS} class now accepts a @emph{context} parameter, which is a
@ref{2bf,,ssl.SSLContext} object allowing bundling SSL configuration options,
certificates and private keys into a single (potentially long-lived) structure.

(Contributed by Giampaolo Rodolà; bpo-8806@footnote{https://bugs.python.org/issue8806}.)

@node popen,select<3>,ftp,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 popen}@anchor{da9}
@subsubsection popen


The @ref{59f,,os.popen()} and @ref{5ae,,subprocess.Popen()} functions now support
@ref{168,,with} statements for auto-closing of the file descriptors.

(Contributed by Antoine Pitrou and Brian Curtin in bpo-7461@footnote{https://bugs.python.org/issue7461} and
bpo-10554@footnote{https://bugs.python.org/issue10554}.)

@node select<3>,gzip and zipfile,popen,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 select}@anchor{daa}
@subsubsection select


The @ref{e0,,select} module now exposes a new, constant attribute,
@ref{dab,,PIPE_BUF}, which gives the minimum number of bytes which are
guaranteed not to block when @ref{8fc,,select.select()} says a pipe is ready
for writing.

@example
>>> import select
>>> select.PIPE_BUF  
512
@end example

(Available on Unix systems. Patch by Sébastien Sablé in bpo-9862@footnote{https://bugs.python.org/issue9862})

@node gzip and zipfile,tarfile<5>,select<3>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 gzip-and-zipfile}@anchor{dac}
@subsubsection gzip and zipfile


@ref{43f,,gzip.GzipFile} now implements the @ref{823,,io.BufferedIOBase}
@ref{d5c,,abstract base class} (except for @code{truncate()}).  It also has a
@ref{dad,,peek()} method and supports unseekable as well as
zero-padded file objects.

The @ref{87,,gzip} module also gains the @ref{4fc,,compress()} and
@ref{dae,,decompress()} functions for easier in-memory compression and
decompression.  Keep in mind that text needs to be encoded as @ref{2f0,,bytes}
before compressing and decompressing:

@example
>>> import gzip
>>> s = 'Three shall be the number thou shalt count, '
>>> s += 'and the number of the counting shall be three'
>>> b = s.encode()                        # convert to utf-8
>>> len(b)
89
>>> c = gzip.compress(b)
>>> len(c)
77
>>> gzip.decompress(c).decode()[:42]      # decompress and convert to text
'Three shall be the number thou shalt count'
@end example

(Contributed by Anand B. Pillai in bpo-3488@footnote{https://bugs.python.org/issue3488}; and by Antoine Pitrou, Nir
Aides and Brian Curtin in bpo-9962@footnote{https://bugs.python.org/issue9962}, bpo-1675951@footnote{https://bugs.python.org/issue1675951}, bpo-7471@footnote{https://bugs.python.org/issue7471} and
bpo-2846@footnote{https://bugs.python.org/issue2846}.)

Also, the @code{zipfile.ZipExtFile} class was reworked internally to represent
files stored inside an archive.  The new implementation is significantly faster
and can be wrapped in an @ref{daf,,io.BufferedReader} object for more speedups.  It
also solves an issue where interleaved calls to @emph{read} and @emph{readline} gave the
wrong results.

(Patch submitted by Nir Aides in bpo-7610@footnote{https://bugs.python.org/issue7610}.)

@node tarfile<5>,hashlib<5>,gzip and zipfile,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 tarfile}@anchor{db0}
@subsubsection tarfile


The @ref{db1,,TarFile} class can now be used as a context manager.  In
addition, its @ref{72f,,add()} method has a new option, @emph{filter},
that controls which files are added to the archive and allows the file metadata
to be edited.

The new @emph{filter} option replaces the older, less flexible @emph{exclude} parameter
which is now deprecated.  If specified, the optional @emph{filter} parameter needs to
be a @ref{85f,,keyword argument}.  The user-supplied filter function accepts a
@ref{db2,,TarInfo} object and returns an updated
@ref{db2,,TarInfo} object, or if it wants the file to be excluded, the
function can return @code{None}:

@example
>>> import tarfile, glob

>>> def myfilter(tarinfo):
...     if tarinfo.isfile():             # only save real files
...         tarinfo.uname = 'monty'      # redact the user name
...         return tarinfo

>>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:
...     for filename in glob.glob('*.txt'):
...         tf.add(filename, filter=myfilter)
...     tf.list()
-rw-r--r-- monty/501        902 2011-01-26 17:59:11 annotations.txt
-rw-r--r-- monty/501        123 2011-01-26 17:59:11 general_questions.txt
-rw-r--r-- monty/501       3514 2011-01-26 17:59:11 prion.txt
-rw-r--r-- monty/501        124 2011-01-26 17:59:11 py_todo.txt
-rw-r--r-- monty/501       1399 2011-01-26 17:59:11 semaphore_notes.txt
@end example

(Proposed by Tarek Ziadé and implemented by Lars Gustäbel in bpo-6856@footnote{https://bugs.python.org/issue6856}.)

@node hashlib<5>,ast<4>,tarfile<5>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 hashlib}@anchor{db3}
@subsubsection hashlib


The @ref{88,,hashlib} module has two new constant attributes listing the hashing
algorithms guaranteed to be present in all implementations and those available
on the current implementation:

@example
>>> import hashlib

>>> hashlib.algorithms_guaranteed
@{'sha1', 'sha224', 'sha384', 'sha256', 'sha512', 'md5'@}

>>> hashlib.algorithms_available
@{'md2', 'SHA256', 'SHA512', 'dsaWithSHA', 'mdc2', 'SHA224', 'MD4', 'sha256',
'sha512', 'ripemd160', 'SHA1', 'MDC2', 'SHA', 'SHA384', 'MD2',
'ecdsa-with-SHA1','md4', 'md5', 'sha1', 'DSA-SHA', 'sha224',
'dsaEncryption', 'DSA', 'RIPEMD160', 'sha', 'MD5', 'sha384'@}
@end example

(Suggested by Carl Chenet in bpo-7418@footnote{https://bugs.python.org/issue7418}.)

@node ast<4>,os<11>,hashlib<5>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 ast}@anchor{db4}
@subsubsection ast


The @ref{7,,ast} module has a wonderful a general-purpose tool for safely
evaluating expression strings using the Python literal
syntax.  The @ref{754,,ast.literal_eval()} function serves as a secure alternative to
the builtin @ref{34a,,eval()} function which is easily abused.  Python 3.2 adds
@ref{2f0,,bytes} and @ref{370,,set} literals to the list of supported types:
strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and @code{None}.

@example
>>> from ast import literal_eval

>>> request = "@{'req': 3, 'func': 'pow', 'args': (2, 0.5)@}"
>>> literal_eval(request)
@{'args': (2, 0.5), 'req': 3, 'func': 'pow'@}

>>> request = "os.system('do something harmful')"
>>> literal_eval(request)
Traceback (most recent call last):
  ...
ValueError: malformed node or string: <_ast.Call object at 0x101739a10>
@end example

(Implemented by Benjamin Peterson and Georg Brandl.)

@node os<11>,shutil<5>,ast<4>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 os}@anchor{db5}
@subsubsection os


Different operating systems use various encodings for filenames and environment
variables.  The @ref{c0,,os} module provides two new functions,
@ref{799,,fsencode()} and @ref{798,,fsdecode()}, for encoding and decoding
filenames:

@example
>>> import os
>>> filename = 'Sehenswürdigkeiten'
>>> os.fsencode(filename)
b'Sehensw\xc3\xbcrdigkeiten'
@end example

Some operating systems allow direct access to encoded bytes in the
environment.  If so, the @ref{db6,,os.supports_bytes_environ} constant will be
true.

For direct access to encoded environment variables (if available),
use the new @ref{db7,,os.getenvb()} function or use @ref{db8,,os.environb}
which is a bytes version of @ref{d60,,os.environ}.

(Contributed by Victor Stinner.)

@node shutil<5>,sqlite3<8>,os<11>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 shutil}@anchor{db9}
@subsubsection shutil


The @ref{538,,shutil.copytree()} function has two new options:


@itemize *

@item 
@emph{ignore_dangling_symlinks}: when @code{symlinks=False} so that the function
copies a file pointed to by a symlink, not the symlink itself. This option
will silence the error raised if the file doesn’t exist.

@item 
@emph{copy_function}: is a callable that will be used to copy files.
@ref{571,,shutil.copy2()} is used by default.
@end itemize

(Contributed by Tarek Ziadé.)

In addition, the @ref{e4,,shutil} module now supports @ref{dba,,archiving operations} for zipfiles, uncompressed tarfiles, gzipped tarfiles,
and bzipped tarfiles.  And there are functions for registering additional
archiving file formats (such as xz compressed tarfiles or custom formats).

The principal functions are @ref{539,,make_archive()} and
@ref{dbb,,unpack_archive()}.  By default, both operate on the current
directory (which can be set by @ref{3b9,,os.chdir()}) and on any sub-directories.
The archive filename needs to be specified with a full pathname.  The archiving
step is non-destructive (the original files are left unchanged).

@example
>>> import shutil, pprint

>>> os.chdir('mydata')  # change to the source directory
>>> f = shutil.make_archive('/var/backup/mydata',
...                         'zip')      # archive the current directory
>>> f                                   # show the name of archive
'/var/backup/mydata.zip'
>>> os.chdir('tmp')                     # change to an unpacking
>>> shutil.unpack_archive('/var/backup/mydata.zip')  # recover the data

>>> pprint.pprint(shutil.get_archive_formats())  # display known formats
[('bztar', "bzip2'ed tar-file"),
 ('gztar', "gzip'ed tar-file"),
 ('tar', 'uncompressed tar file'),
 ('zip', 'ZIP file')]

>>> shutil.register_archive_format(     # register a new archive format
...     name='xz',
...     function=xz.compress,           # callable archiving function
...     extra_args=[('level', 8)],      # arguments to the function
...     description='xz compression'
... )
@end example

(Contributed by Tarek Ziadé.)

@node sqlite3<8>,html<3>,shutil<5>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 sqlite3}@anchor{dbc}
@subsubsection sqlite3


The @ref{ed,,sqlite3} module was updated to pysqlite version 2.6.0.  It has two new capabilities.


@itemize *

@item 
The @code{sqlite3.Connection.in_transit} attribute is true if there is an
active transaction for uncommitted changes.

@item 
The @ref{2c3,,sqlite3.Connection.enable_load_extension()} and
@ref{2c4,,sqlite3.Connection.load_extension()} methods allows you to load SQLite
extensions from “.so” files.  One well-known extension is the fulltext-search
extension distributed with SQLite.
@end itemize

(Contributed by R. David Murray and Shashwat Anand; bpo-8845@footnote{https://bugs.python.org/issue8845}.)

@node html<3>,socket<11>,sqlite3<8>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 html}@anchor{dbd}
@subsubsection html


A new @ref{8b,,html} module was introduced with only a single function,
@ref{b95,,escape()}, which is used for escaping reserved characters from HTML
markup:

@example
>>> import html
>>> html.escape('x > 2 && x < 7')
'x &gt; 2 &amp;&amp; x &lt; 7'
@end example

@node socket<11>,ssl<10>,html<3>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 socket}@anchor{dbe}
@subsubsection socket


The @ref{ea,,socket} module has two new improvements.


@itemize *

@item 
Socket objects now have a @ref{dbf,,detach()} method which puts
the socket into closed state without actually closing the underlying file
descriptor.  The latter can then be reused for other purposes.
(Added by Antoine Pitrou; bpo-8524@footnote{https://bugs.python.org/issue8524}.)

@item 
@ref{dc0,,socket.create_connection()} now supports the context management protocol
to unconditionally consume @ref{beb,,socket.error} exceptions and to close the
socket when done.
(Contributed by Giampaolo Rodolà; bpo-9794@footnote{https://bugs.python.org/issue9794}.)
@end itemize

@node ssl<10>,nntp,socket<11>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 ssl}@anchor{dc1}
@subsubsection ssl


The @ref{ee,,ssl} module added a number of features to satisfy common requirements
for secure (encrypted, authenticated) internet connections:


@itemize *

@item 
A new class, @ref{2bf,,SSLContext}, serves as a container for persistent
SSL data, such as protocol settings, certificates, private keys, and various
other options. It includes a @ref{332,,wrap_socket()} for creating
an SSL socket from an SSL context.

@item 
A new function, @ref{333,,ssl.match_hostname()}, supports server identity
verification for higher-level protocols by implementing the rules of HTTPS
(from 
@geindex RFC; RFC 2818
RFC 2818@footnote{https://tools.ietf.org/html/rfc2818.html}) which are also suitable for other protocols.

@item 
The @ref{331,,ssl.wrap_socket()} constructor function now takes a @emph{ciphers}
argument.  The @emph{ciphers} string lists the allowed encryption algorithms using
the format described in the OpenSSL documentation@footnote{https://www.openssl.org/docs/manmaster/man1/ciphers.html#CIPHER-LIST-FORMAT}.

@item 
When linked against recent versions of OpenSSL, the @ref{ee,,ssl} module now
supports the Server Name Indication extension to the TLS protocol, allowing
multiple “virtual hosts” using different certificates on a single IP port.
This extension is only supported in client mode, and is activated by passing
the @emph{server_hostname} argument to @ref{332,,ssl.SSLContext.wrap_socket()}.

@item 
Various options have been added to the @ref{ee,,ssl} module, such as
@ref{2bc,,OP_NO_SSLv2} which disables the insecure and obsolete SSLv2
protocol.

@item 
The extension now loads all the OpenSSL ciphers and digest algorithms.  If
some SSL certificates cannot be verified, they are reported as an “unknown
algorithm” error.

@item 
The version of OpenSSL being used is now accessible using the module
attributes @ref{dc2,,ssl.OPENSSL_VERSION} (a string),
@ref{dc3,,ssl.OPENSSL_VERSION_INFO} (a 5-tuple), and
@ref{dc4,,ssl.OPENSSL_VERSION_NUMBER} (an integer).
@end itemize

(Contributed by Antoine Pitrou in bpo-8850@footnote{https://bugs.python.org/issue8850}, bpo-1589@footnote{https://bugs.python.org/issue1589}, bpo-8322@footnote{https://bugs.python.org/issue8322},
bpo-5639@footnote{https://bugs.python.org/issue5639}, bpo-4870@footnote{https://bugs.python.org/issue4870}, bpo-8484@footnote{https://bugs.python.org/issue8484}, and bpo-8321@footnote{https://bugs.python.org/issue8321}.)

@node nntp,certificates,ssl<10>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 nntp}@anchor{dc5}
@subsubsection nntp


The @ref{bc,,nntplib} module has a revamped implementation with better bytes and
text semantics as well as more practical APIs.  These improvements break
compatibility with the nntplib version in Python 3.1, which was partly
dysfunctional in itself.

Support for secure connections through both implicit (using
@ref{40a,,nntplib.NNTP_SSL}) and explicit (using @ref{dc6,,nntplib.NNTP.starttls()})
TLS has also been added.

(Contributed by Antoine Pitrou in bpo-9360@footnote{https://bugs.python.org/issue9360} and Andrew Vant in bpo-1926@footnote{https://bugs.python.org/issue1926}.)

@node certificates,imaplib<4>,nntp,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 certificates}@anchor{dc7}
@subsubsection certificates


@ref{664,,http.client.HTTPSConnection}, @ref{dc8,,urllib.request.HTTPSHandler}
and @ref{a06,,urllib.request.urlopen()} now take optional arguments to allow for
server certificate checking against a set of Certificate Authorities,
as recommended in public uses of HTTPS.

(Added by Antoine Pitrou, bpo-9003@footnote{https://bugs.python.org/issue9003}.)

@node imaplib<4>,http client<4>,certificates,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 imaplib}@anchor{dc9}
@subsubsection imaplib


Support for explicit TLS on standard IMAP4 connections has been added through
the new @ref{dca,,imaplib.IMAP4.starttls} method.

(Contributed by Lorenzo M. Catucci and Antoine Pitrou, bpo-4471@footnote{https://bugs.python.org/issue4471}.)

@node http client<4>,unittest<7>,imaplib<4>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 http-client}@anchor{dcb}
@subsubsection http.client


There were a number of small API improvements in the @ref{8f,,http.client} module.
The old-style HTTP 0.9 simple responses are no longer supported and the @emph{strict}
parameter is deprecated in all classes.

The @ref{663,,HTTPConnection} and
@ref{664,,HTTPSConnection} classes now have a @emph{source_address}
parameter for a (host, port) tuple indicating where the HTTP connection is made
from.

Support for certificate checking and HTTPS virtual hosts were added to
@ref{664,,HTTPSConnection}.

The @ref{7f1,,request()} method on connection objects
allowed an optional @emph{body} argument so that a @ref{d68,,file object} could be used
to supply the content of the request.  Conveniently, the @emph{body} argument now
also accepts an @ref{dcc,,iterable} object so long as it includes an explicit
@code{Content-Length} header.  This extended interface is much more flexible than
before.

To establish an HTTPS connection through a proxy server, there is a new
@ref{dcd,,set_tunnel()} method that sets the host and
port for HTTP Connect tunneling.

To match the behavior of @ref{92,,http.server}, the HTTP client library now also
encodes headers with ISO-8859-1 (Latin-1) encoding.  It was already doing that
for incoming headers, so now the behavior is consistent for both incoming and
outgoing traffic. (See work by Armin Ronacher in bpo-10980@footnote{https://bugs.python.org/issue10980}.)

@node unittest<7>,random<3>,http client<4>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 unittest}@anchor{dce}
@subsubsection unittest


The unittest module has a number of improvements supporting test discovery for
packages, easier experimentation at the interactive prompt, new testcase
methods, improved diagnostic messages for test failures, and better method
names.


@itemize *

@item 
The command-line call @code{python -m unittest} can now accept file paths
instead of module names for running specific tests (bpo-10620@footnote{https://bugs.python.org/issue10620}).  The new
test discovery can find tests within packages, locating any test importable
from the top-level directory.  The top-level directory can be specified with
the @cite{-t} option, a pattern for matching files with @code{-p}, and a directory to
start discovery with @code{-s}:

@example
$ python -m unittest discover -s my_proj_dir -p _test.py
@end example

(Contributed by Michael Foord.)

@item 
Experimentation at the interactive prompt is now easier because the
@code{unittest.case.TestCase} class can now be instantiated without
arguments:

@example
>>> from unittest import TestCase
>>> TestCase().assertEqual(pow(2, 3), 8)
@end example

(Contributed by Michael Foord.)

@item 
The @ref{122,,unittest} module has two new methods,
@ref{d01,,assertWarns()} and
@ref{d02,,assertWarnsRegex()} to verify that a given warning type
is triggered by the code under test:

@example
with self.assertWarns(DeprecationWarning):
    legacy_function('XYZ')
@end example

(Contributed by Antoine Pitrou, bpo-9754@footnote{https://bugs.python.org/issue9754}.)

Another new method, @ref{dcf,,assertCountEqual()} is used to
compare two iterables to determine if their element counts are equal (whether
the same elements are present with the same number of occurrences regardless
of order):

@example
def test_anagram(self):
    self.assertCountEqual('algorithm', 'logarithm')
@end example

(Contributed by Raymond Hettinger.)

@item 
A principal feature of the unittest module is an effort to produce meaningful
diagnostics when a test fails.  When possible, the failure is recorded along
with a diff of the output.  This is especially helpful for analyzing log files
of failed test runs. However, since diffs can sometime be voluminous, there is
a new @ref{dd0,,maxDiff} attribute that sets maximum length of
diffs displayed.

@item 
In addition, the method names in the module have undergone a number of clean-ups.

For example, @ref{dd1,,assertRegex()} is the new name for
@code{assertRegexpMatches()} which was misnamed because the
test uses @ref{dd2,,re.search()}, not @ref{dd3,,re.match()}.  Other methods using
regular expressions are now named using short form “Regex” in preference to
“Regexp” – this matches the names used in other unittest implementations,
matches Python’s old name for the @ref{d8,,re} module, and it has unambiguous
camel-casing.

(Contributed by Raymond Hettinger and implemented by Ezio Melotti.)

@item 
To improve consistency, some long-standing method aliases are being
deprecated in favor of the preferred names:

@quotation


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Old Name

@tab

Preferred Name

@item

@code{assert_()}

@tab

@ref{dd4,,assertTrue()}

@item

@code{assertEquals()}

@tab

@ref{dd5,,assertEqual()}

@item

@code{assertNotEquals()}

@tab

@ref{dd6,,assertNotEqual()}

@item

@code{assertAlmostEquals()}

@tab

@ref{dd7,,assertAlmostEqual()}

@item

@code{assertNotAlmostEquals()}

@tab

@ref{dd8,,assertNotAlmostEqual()}

@end multitable

@end quotation

Likewise, the @code{TestCase.fail*} methods deprecated in Python 3.1 are expected
to be removed in Python 3.3.  Also see the @ref{dd9,,Deprecated aliases} section in
the @ref{122,,unittest} documentation.

(Contributed by Ezio Melotti; bpo-9424@footnote{https://bugs.python.org/issue9424}.)

@item 
The @code{assertDictContainsSubset()} method was deprecated
because it was misimplemented with the arguments in the wrong order.  This
created hard-to-debug optical illusions where tests like
@code{TestCase().assertDictContainsSubset(@{'a':1, 'b':2@}, @{'a':1@})} would fail.

(Contributed by Raymond Hettinger.)
@end itemize

@node random<3>,poplib<4>,unittest<7>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 random}@anchor{dda}
@subsubsection random


The integer methods in the @ref{d7,,random} module now do a better job of producing
uniform distributions.  Previously, they computed selections with
@code{int(n*random())} which had a slight bias whenever @emph{n} was not a power of two.
Now, multiple selections are made from a range up to the next power of two and a
selection is kept only when it falls within the range @code{0 <= x < n}.  The
functions and methods affected are @ref{305,,randrange()},
@ref{ddb,,randint()}, @ref{ddc,,choice()}, @ref{444,,shuffle()} and
@ref{ddd,,sample()}.

(Contributed by Raymond Hettinger; bpo-9025@footnote{https://bugs.python.org/issue9025}.)

@node poplib<4>,asyncore<2>,random<3>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 poplib}@anchor{dde}
@subsubsection poplib


@ref{419,,POP3_SSL} class now accepts a @emph{context} parameter, which is a
@ref{2bf,,ssl.SSLContext} object allowing bundling SSL configuration options,
certificates and private keys into a single (potentially long-lived)
structure.

(Contributed by Giampaolo Rodolà; bpo-8807@footnote{https://bugs.python.org/issue8807}.)

@node asyncore<2>,tempfile<2>,poplib<4>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 asyncore}@anchor{ddf}
@subsubsection asyncore


@ref{de0,,asyncore.dispatcher} now provides a
@ref{de1,,handle_accepted()} method
returning a @cite{(sock@comma{} addr)} pair which is called when a connection has actually
been established with a new remote endpoint. This is supposed to be used as a
replacement for old @ref{de2,,handle_accept()} and avoids
the user  to call @ref{de3,,accept()} directly.

(Contributed by Giampaolo Rodolà; bpo-6706@footnote{https://bugs.python.org/issue6706}.)

@node tempfile<2>,inspect<9>,asyncore<2>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 tempfile}@anchor{de4}
@subsubsection tempfile


The @ref{fd,,tempfile} module has a new context manager,
@ref{de5,,TemporaryDirectory} which provides easy deterministic
cleanup of temporary directories:

@example
with tempfile.TemporaryDirectory() as tmpdirname:
    print('created temporary dir:', tmpdirname)
@end example

(Contributed by Neil Schemenauer and Nick Coghlan; bpo-5178@footnote{https://bugs.python.org/issue5178}.)

@node inspect<9>,pydoc<6>,tempfile<2>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 inspect}@anchor{de6}
@subsubsection inspect



@itemize *

@item 
The @ref{9c,,inspect} module has a new function
@ref{de7,,getgeneratorstate()} to easily identify the current state of a
generator-iterator:

@example
>>> from inspect import getgeneratorstate
>>> def gen():
...     yield 'demo'
>>> g = gen()
>>> getgeneratorstate(g)
'GEN_CREATED'
>>> next(g)
'demo'
>>> getgeneratorstate(g)
'GEN_SUSPENDED'
>>> next(g, None)
>>> getgeneratorstate(g)
'GEN_CLOSED'
@end example

(Contributed by Rodolpho Eckhardt and Nick Coghlan, bpo-10220@footnote{https://bugs.python.org/issue10220}.)

@item 
To support lookups without the possibility of activating a dynamic attribute,
the @ref{9c,,inspect} module has a new function, @ref{de8,,getattr_static()}.
Unlike @ref{707,,hasattr()}, this is a true read-only search, guaranteed not to
change state while it is searching:

@example
>>> class A:
...     @@property
...     def f(self):
...         print('Running')
...         return 10
...
>>> a = A()
>>> getattr(a, 'f')
Running
10
>>> inspect.getattr_static(a, 'f')
<property object at 0x1022bd788>
@end example
@end itemize

@quotation

(Contributed by Michael Foord.)
@end quotation

@node pydoc<6>,dis<3>,inspect<9>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 pydoc}@anchor{de9}
@subsubsection pydoc


The @ref{d4,,pydoc} module now provides a much-improved web server interface, as
well as a new command-line option @code{-b} to automatically open a browser window
to display that server:

@example
$ pydoc3.2 -b
@end example

(Contributed by Ron Adam; bpo-2001@footnote{https://bugs.python.org/issue2001}.)

@node dis<3>,dbm<6>,pydoc<6>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 dis}@anchor{dea}
@subsubsection dis


The @ref{36,,dis} module gained two new functions for inspecting code,
@ref{deb,,code_info()} and @ref{aa6,,show_code()}.  Both provide detailed code
object information for the supplied function, method, source code string or code
object.  The former returns a string and the latter prints it:

@example
>>> import dis, random
>>> dis.show_code(random.choice)
Name:              choice
Filename:          /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/random.py
Argument count:    2
Kw-only arguments: 0
Number of locals:  3
Stack size:        11
Flags:             OPTIMIZED, NEWLOCALS, NOFREE
Constants:
   0: 'Choose a random element from a non-empty sequence.'
   1: 'Cannot choose from an empty sequence'
Names:
   0: _randbelow
   1: len
   2: ValueError
   3: IndexError
Variable names:
   0: self
   1: seq
   2: i
@end example

In addition, the @ref{655,,dis()} function now accepts string arguments
so that the common idiom @code{dis(compile(s, '@w{'}, 'eval'))} can be shortened
to @code{dis(s)}:

@example
>>> dis('3*x+1 if x%2==1 else x//2')
  1           0 LOAD_NAME                0 (x)
              3 LOAD_CONST               0 (2)
              6 BINARY_MODULO
              7 LOAD_CONST               1 (1)
             10 COMPARE_OP               2 (==)
             13 POP_JUMP_IF_FALSE       28
             16 LOAD_CONST               2 (3)
             19 LOAD_NAME                0 (x)
             22 BINARY_MULTIPLY
             23 LOAD_CONST               1 (1)
             26 BINARY_ADD
             27 RETURN_VALUE
        >>   28 LOAD_NAME                0 (x)
             31 LOAD_CONST               0 (2)
             34 BINARY_FLOOR_DIVIDE
             35 RETURN_VALUE
@end example

Taken together, these improvements make it easier to explore how CPython is
implemented and to see for yourself what the language syntax does
under-the-hood.

(Contributed by Nick Coghlan in bpo-9147@footnote{https://bugs.python.org/issue9147}.)

@node dbm<6>,ctypes<2>,dis<3>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 dbm}@anchor{dec}
@subsubsection dbm


All database modules now support the @code{get()} and @code{setdefault()} methods.

(Suggested by Ray Allen in bpo-9523@footnote{https://bugs.python.org/issue9523}.)

@node ctypes<2>,site<3>,dbm<6>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 ctypes}@anchor{ded}
@subsubsection ctypes


A new type, @ref{dee,,ctypes.c_ssize_t} represents the C @code{ssize_t} datatype.

@node site<3>,sysconfig<2>,ctypes<2>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 site}@anchor{def}
@subsubsection site


The @ref{e6,,site} module has three new functions useful for reporting on the
details of a given Python installation.


@itemize *

@item 
@ref{df0,,getsitepackages()} lists all global site-packages directories.

@item 
@ref{df1,,getuserbase()} reports on the user’s base directory where data can
be stored.

@item 
@ref{df2,,getusersitepackages()} reveals the user-specific site-packages
directory path.
@end itemize

@example
>>> import site
>>> site.getsitepackages()
['/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages',
 '/Library/Frameworks/Python.framework/Versions/3.2/lib/site-python',
 '/Library/Python/3.2/site-packages']
>>> site.getuserbase()
'/Users/raymondhettinger/Library/Python/3.2'
>>> site.getusersitepackages()
'/Users/raymondhettinger/Library/Python/3.2/lib/python/site-packages'
@end example

Conveniently, some of site’s functionality is accessible directly from the
command-line:

@example
$ python -m site --user-base
/Users/raymondhettinger/.local
$ python -m site --user-site
/Users/raymondhettinger/.local/lib/python3.2/site-packages
@end example

(Contributed by Tarek Ziadé in bpo-6693@footnote{https://bugs.python.org/issue6693}.)

@node sysconfig<2>,pdb<6>,site<3>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 sysconfig}@anchor{df3}
@subsubsection sysconfig


The new @ref{f8,,sysconfig} module makes it straightforward to discover
installation paths and configuration variables that vary across platforms and
installations.

The module offers access simple access functions for platform and version
information:


@itemize *

@item 
@ref{df4,,get_platform()} returning values like @emph{linux-i586} or
@emph{macosx-10.6-ppc}.

@item 
@ref{df5,,get_python_version()} returns a Python version string
such as “3.2”.
@end itemize

It also provides access to the paths and variables corresponding to one of
seven named schemes used by @ref{37,,distutils}.  Those include @emph{posix_prefix},
@emph{posix_home}, @emph{posix_user}, @emph{nt}, @emph{nt_user}, @emph{os2}, @emph{os2_home}:


@itemize *

@item 
@ref{df6,,get_paths()} makes a dictionary containing installation paths
for the current installation scheme.

@item 
@ref{bbd,,get_config_vars()} returns a dictionary of platform specific
variables.
@end itemize

There is also a convenient command-line interface:

@example
C:\Python32>python -m sysconfig
Platform: "win32"
Python version: "3.2"
Current installation scheme: "nt"

Paths:
        data = "C:\Python32"
        include = "C:\Python32\Include"
        platinclude = "C:\Python32\Include"
        platlib = "C:\Python32\Lib\site-packages"
        platstdlib = "C:\Python32\Lib"
        purelib = "C:\Python32\Lib\site-packages"
        scripts = "C:\Python32\Scripts"
        stdlib = "C:\Python32\Lib"

Variables:
        BINDIR = "C:\Python32"
        BINLIBDEST = "C:\Python32\Lib"
        EXE = ".exe"
        INCLUDEPY = "C:\Python32\Include"
        LIBDEST = "C:\Python32\Lib"
        SO = ".pyd"
        VERSION = "32"
        abiflags = ""
        base = "C:\Python32"
        exec_prefix = "C:\Python32"
        platbase = "C:\Python32"
        prefix = "C:\Python32"
        projectbase = "C:\Python32"
        py_version = "3.2"
        py_version_nodot = "32"
        py_version_short = "3.2"
        srcdir = "C:\Python32"
        userbase = "C:\Documents and Settings\Raymond\Application Data\Python"
@end example

(Moved out of Distutils by Tarek Ziadé.)

@node pdb<6>,configparser<2>,sysconfig<2>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 pdb}@anchor{df7}
@subsubsection pdb


The @ref{c4,,pdb} debugger module gained a number of usability improvements:


@itemize *

@item 
@code{pdb.py} now has a @code{-c} option that executes commands as given in a
@code{.pdbrc} script file.

@item 
A @code{.pdbrc} script file can contain @code{continue} and @code{next} commands
that continue debugging.

@item 
The @code{Pdb} class constructor now accepts a @emph{nosigint} argument.

@item 
New commands: @code{l(list)}, @code{ll(long list)} and @code{source} for
listing source code.

@item 
New commands: @code{display} and @code{undisplay} for showing or hiding
the value of an expression if it has changed.

@item 
New command: @code{interact} for starting an interactive interpreter containing
the global and local  names found in the current scope.

@item 
Breakpoints can be cleared by breakpoint number.
@end itemize

(Contributed by Georg Brandl, Antonio Cuni and Ilya Sandler.)

@node configparser<2>,urllib parse<4>,pdb<6>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 configparser}@anchor{df8}
@subsubsection configparser


The @ref{21,,configparser} module was modified to improve usability and
predictability of the default parser and its supported INI syntax.  The old
@code{ConfigParser} class was removed in favor of @code{SafeConfigParser}
which has in turn been renamed to @ref{759,,ConfigParser}. Support
for inline comments is now turned off by default and section or option
duplicates are not allowed in a single configuration source.

Config parsers gained a new API based on the mapping protocol:

@example
>>> parser = ConfigParser()
>>> parser.read_string("""
... [DEFAULT]
... location = upper left
... visible = yes
... editable = no
... color = blue
...
... [main]
... title = Main Menu
... color = green
...
... [options]
... title = Options
... """)
>>> parser['main']['color']
'green'
>>> parser['main']['editable']
'no'
>>> section = parser['options']
>>> section['title']
'Options'
>>> section['title'] = 'Options (editable: %(editable)s)'
>>> section['title']
'Options (editable: no)'
@end example

The new API is implemented on top of the classical API, so custom parser
subclasses should be able to use it without modifications.

The INI file structure accepted by config parsers can now be customized. Users
can specify alternative option/value delimiters and comment prefixes, change the
name of the @emph{DEFAULT} section or switch the interpolation syntax.

There is support for pluggable interpolation including an additional interpolation
handler @ref{df9,,ExtendedInterpolation}:

@example
>>> parser = ConfigParser(interpolation=ExtendedInterpolation())
>>> parser.read_dict(@{'buildout': @{'directory': '/home/ambv/zope9'@},
...                   'custom': @{'prefix': '/usr/local'@}@})
>>> parser.read_string("""
... [buildout]
... parts =
...   zope9
...   instance
... find-links =
...   $@{buildout:directory@}/downloads/dist
...
... [zope9]
... recipe = plone.recipe.zope9install
... location = /opt/zope
...
... [instance]
... recipe = plone.recipe.zope9instance
... zope9-location = $@{zope9:location@}
... zope-conf = $@{custom:prefix@}/etc/zope.conf
... """)
>>> parser['buildout']['find-links']
'\n/home/ambv/zope9/downloads/dist'
>>> parser['instance']['zope-conf']
'/usr/local/etc/zope.conf'
>>> instance = parser['instance']
>>> instance['zope-conf']
'/usr/local/etc/zope.conf'
>>> instance['zope9-location']
'/opt/zope'
@end example

A number of smaller features were also introduced, like support for specifying
encoding in read operations, specifying fallback values for get-functions, or
reading directly from dictionaries and strings.

(All changes contributed by Łukasz Langa.)

@c XXX consider showing a difflib example

@node urllib parse<4>,mailbox,configparser<2>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 urllib-parse}@anchor{dfa}
@subsubsection urllib.parse


A number of usability improvements were made for the @ref{126,,urllib.parse} module.

The @ref{890,,urlparse()} function now supports IPv6@footnote{https://en.wikipedia.org/wiki/IPv6} addresses as described in 
@geindex RFC; RFC 2732
RFC 2732@footnote{https://tools.ietf.org/html/rfc2732.html}:

@example
>>> import urllib.parse
>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') 
ParseResult(scheme='http',
            netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]',
            path='/foo/',
            params='@w{'},
            query='@w{'},
            fragment='@w{'})
@end example

The @ref{dfb,,urldefrag()} function now returns a @ref{ce9,,named tuple}:

@example
>>> r = urllib.parse.urldefrag('http://python.org/about/#target')
>>> r
DefragResult(url='http://python.org/about/', fragment='target')
>>> r[0]
'http://python.org/about/'
>>> r.fragment
'target'
@end example

And, the @ref{a05,,urlencode()} function is now much more flexible,
accepting either a string or bytes type for the @emph{query} argument.  If it is a
string, then the @emph{safe}, @emph{encoding}, and @emph{error} parameters are sent to
@ref{dfc,,quote_plus()} for encoding:

@example
>>> urllib.parse.urlencode([
...      ('type', 'telenovela'),
...      ('name', '¿Dónde Está Elisa?')],
...      encoding='latin-1')
'type=telenovela&name=%BFD%F3nde+Est%E1+Elisa%3F'
@end example

As detailed in @ref{dfd,,Parsing ASCII Encoded Bytes}, all the @ref{126,,urllib.parse}
functions now accept ASCII-encoded byte strings as input, so long as they are
not mixed with regular strings.  If ASCII-encoded byte strings are given as
parameters, the return types will also be an ASCII-encoded byte strings:

@example
>>> urllib.parse.urlparse(b'http://www.python.org:80/about/') 
ParseResultBytes(scheme=b'http', netloc=b'www.python.org:80',
                 path=b'/about/', params=b'@w{'}, query=b'@w{'}, fragment=b'@w{'})
@end example

(Work by Nick Coghlan, Dan Mahn, and Senthil Kumaran in bpo-2987@footnote{https://bugs.python.org/issue2987},
bpo-5468@footnote{https://bugs.python.org/issue5468}, and bpo-9873@footnote{https://bugs.python.org/issue9873}.)

@node mailbox,turtledemo,urllib parse<4>,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 mailbox}@anchor{dfe}
@subsubsection mailbox


Thanks to a concerted effort by R. David Murray, the @ref{aa,,mailbox} module has
been fixed for Python 3.2.  The challenge was that mailbox had been originally
designed with a text interface, but email messages are best represented with
@ref{2f0,,bytes} because various parts of a message may have different encodings.

The solution harnessed the @ref{64,,email} package’s binary support for parsing
arbitrary email messages.  In addition, the solution required a number of API
changes.

As expected, the @ref{dff,,add()} method for
@ref{e00,,mailbox.Mailbox} objects now accepts binary input.

@ref{aa1,,StringIO} and text file input are deprecated.  Also, string input
will fail early if non-ASCII characters are used.  Previously it would fail when
the email was processed in a later step.

There is also support for binary output.  The @ref{e01,,get_file()}
method now returns a file in the binary mode (where it used to incorrectly set
the file to text-mode).  There is also a new @ref{e02,,get_bytes()}
method that returns a @ref{2f0,,bytes} representation of a message corresponding
to a given @emph{key}.

It is still possible to get non-binary output using the old API’s
@ref{e03,,get_string()} method, but that approach
is not very useful.  Instead, it is best to extract messages from
a @ref{e04,,Message} object or to load them from binary input.

(Contributed by R. David Murray, with efforts from Steffen Daode Nurpmeso and an
initial patch by Victor Stinner in bpo-9124@footnote{https://bugs.python.org/issue9124}.)

@node turtledemo,,mailbox,New Improved and Deprecated Modules
@anchor{whatsnew/3 2 turtledemo}@anchor{e05}
@subsubsection turtledemo


The demonstration code for the @ref{11d,,turtle} module was moved from the @emph{Demo}
directory to main library.  It includes over a dozen sample scripts with
lively displays.  Being on @ref{489,,sys.path}, it can now be run directly
from the command-line:

@example
$ python -m turtledemo
@end example

(Moved from the Demo directory by Alexander Belopolsky in bpo-10199@footnote{https://bugs.python.org/issue10199}.)

@node Multi-threading,Optimizations<9>,New Improved and Deprecated Modules,What’s New In Python 3 2
@anchor{whatsnew/3 2 multi-threading}@anchor{e06}
@subsection Multi-threading



@itemize *

@item 
The mechanism for serializing execution of concurrently running Python threads
(generally known as the @ref{385,,GIL} or Global Interpreter Lock) has
been rewritten.  Among the objectives were more predictable switching
intervals and reduced overhead due to lock contention and the number of
ensuing system calls.  The notion of a “check interval” to allow thread
switches has been abandoned and replaced by an absolute duration expressed in
seconds.  This parameter is tunable through @ref{449,,sys.setswitchinterval()}.
It currently defaults to 5 milliseconds.

Additional details about the implementation can be read from a python-dev mailing-list message@footnote{https://mail.python.org/pipermail/python-dev/2009-October/093321.html}
(however, “priority requests” as exposed in this message have not been kept
for inclusion).

(Contributed by Antoine Pitrou.)

@item 
Regular and recursive locks now accept an optional @emph{timeout} argument to their
@ref{192,,acquire()} method.  (Contributed by Antoine Pitrou;
bpo-7316@footnote{https://bugs.python.org/issue7316}.)

@item 
Similarly, @ref{e07,,threading.Semaphore.acquire()} also gained a @emph{timeout}
argument.  (Contributed by Torsten Landschoff; bpo-850728@footnote{https://bugs.python.org/issue850728}.)

@item 
Regular and recursive lock acquisitions can now be interrupted by signals on
platforms using Pthreads.  This means that Python programs that deadlock while
acquiring locks can be successfully killed by repeatedly sending SIGINT to the
process (by pressing @code{Ctrl+C} in most shells).
(Contributed by Reid Kleckner; bpo-8844@footnote{https://bugs.python.org/issue8844}.)
@end itemize

@node Optimizations<9>,Unicode,Multi-threading,What’s New In Python 3 2
@anchor{whatsnew/3 2 optimizations}@anchor{e08}
@subsection Optimizations


A number of small performance enhancements have been added:


@itemize *

@item 
Python’s peephole optimizer now recognizes patterns such @code{x in @{1, 2, 3@}} as
being a test for membership in a set of constants.  The optimizer recasts the
@ref{370,,set} as a @ref{439,,frozenset} and stores the pre-built constant.

Now that the speed penalty is gone, it is practical to start writing
membership tests using set-notation.  This style is both semantically clear
and operationally fast:

@example
extension = name.rpartition('.')[2]
if extension in @{'xml', 'html', 'xhtml', 'css'@}:
    handle(name)
@end example

(Patch and additional tests contributed by Dave Malcolm; bpo-6690@footnote{https://bugs.python.org/issue6690}).

@item 
Serializing and unserializing data using the @ref{c5,,pickle} module is now
several times faster.

(Contributed by Alexandre Vassalotti, Antoine Pitrou
and the Unladen Swallow team in bpo-9410@footnote{https://bugs.python.org/issue9410} and bpo-3873@footnote{https://bugs.python.org/issue3873}.)

@item 
The Timsort algorithm@footnote{https://en.wikipedia.org/wiki/Timsort} used in
@ref{705,,list.sort()} and @ref{704,,sorted()} now runs faster and uses less memory
when called with a @ref{964,,key function}.  Previously, every element of
a list was wrapped with a temporary object that remembered the key value
associated with each element.  Now, two arrays of keys and values are
sorted in parallel.  This saves the memory consumed by the sort wrappers,
and it saves time lost to delegating comparisons.

(Patch by Daniel Stutzbach in bpo-9915@footnote{https://bugs.python.org/issue9915}.)

@item 
JSON decoding performance is improved and memory consumption is reduced
whenever the same string is repeated for multiple keys.  Also, JSON encoding
now uses the C speedups when the @code{sort_keys} argument is true.

(Contributed by Antoine Pitrou in bpo-7451@footnote{https://bugs.python.org/issue7451} and by Raymond Hettinger and
Antoine Pitrou in bpo-10314@footnote{https://bugs.python.org/issue10314}.)

@item 
Recursive locks (created with the @ref{e09,,threading.RLock()} API) now benefit
from a C implementation which makes them as fast as regular locks, and between
10x and 15x faster than their previous pure Python implementation.

(Contributed by Antoine Pitrou; bpo-3001@footnote{https://bugs.python.org/issue3001}.)

@item 
The fast-search algorithm in stringlib is now used by the @code{split()},
@code{rsplit()}, @code{splitlines()} and @code{replace()} methods on
@ref{2f0,,bytes}, @ref{2f1,,bytearray} and @ref{2ef,,str} objects. Likewise, the
algorithm is also used by @code{rfind()}, @code{rindex()}, @code{rsplit()} and
@code{rpartition()}.

(Patch by Florent Xicluna in bpo-7622@footnote{https://bugs.python.org/issue7622} and bpo-7462@footnote{https://bugs.python.org/issue7462}.)

@item 
Integer to string conversions now work two “digits” at a time, reducing the
number of division and modulo operations.

(bpo-6713@footnote{https://bugs.python.org/issue6713} by Gawain Bolton, Mark Dickinson, and Victor Stinner.)
@end itemize

There were several other minor optimizations. Set differencing now runs faster
when one operand is much larger than the other (patch by Andress Bennetts in
bpo-8685@footnote{https://bugs.python.org/issue8685}).  The @code{array.repeat()} method has a faster implementation
(bpo-1569291@footnote{https://bugs.python.org/issue1569291} by Alexander Belopolsky). The @code{BaseHTTPRequestHandler}
has more efficient buffering (bpo-3709@footnote{https://bugs.python.org/issue3709} by Andrew Schaaf).  The
@ref{99a,,operator.attrgetter()} function has been sped-up (bpo-10160@footnote{https://bugs.python.org/issue10160} by
Christos Georgiou).  And @code{ConfigParser} loads multi-line arguments a bit
faster (bpo-7113@footnote{https://bugs.python.org/issue7113} by Łukasz Langa).

@node Unicode,Codecs,Optimizations<9>,What’s New In Python 3 2
@anchor{whatsnew/3 2 unicode}@anchor{e0a}
@subsection Unicode


Python has been updated to Unicode 6.0.0@footnote{http://unicode.org/versions/Unicode6.0.0/}.  The update to the standard adds
over 2,000 new characters including emoji@footnote{https://en.wikipedia.org/wiki/Emoji}
symbols which are important for mobile phones.

In addition, the updated standard has altered the character properties for two
Kannada characters (U+0CF1, U+0CF2) and one New Tai Lue numeric character
(U+19DA), making the former eligible for use in identifiers while disqualifying
the latter.  For more information, see Unicode Character Database Changes@footnote{http://www.unicode.org/versions/Unicode6.0.0/#Database_Changes}.

@node Codecs,Documentation,Unicode,What’s New In Python 3 2
@anchor{whatsnew/3 2 codecs}@anchor{e0b}
@subsection Codecs


Support was added for @emph{cp720} Arabic DOS encoding (bpo-1616979@footnote{https://bugs.python.org/issue1616979}).

MBCS encoding no longer ignores the error handler argument. In the default
strict mode, it raises an @ref{51f,,UnicodeDecodeError} when it encounters an
undecodable byte sequence and an @ref{185,,UnicodeEncodeError} for an unencodable
character.

The MBCS codec supports @code{'strict'} and @code{'ignore'} error handlers for
decoding, and @code{'strict'} and @code{'replace'} for encoding.

To emulate Python3.1 MBCS encoding, select the @code{'ignore'} handler for decoding
and the @code{'replace'} handler for encoding.

On Mac OS X, Python decodes command line arguments with @code{'utf-8'} rather than
the locale encoding.

By default, @ref{fb,,tarfile} uses @code{'utf-8'} encoding on Windows (instead of
@code{'mbcs'}) and the @code{'surrogateescape'} error handler on all operating
systems.

@node Documentation,IDLE,Codecs,What’s New In Python 3 2
@anchor{whatsnew/3 2 documentation}@anchor{e0c}
@subsection Documentation


The documentation continues to be improved.


@itemize *

@item 
A table of quick links has been added to the top of lengthy sections such as
@ref{e0d,,Built-in Functions}.  In the case of @ref{9f,,itertools}, the links are
accompanied by tables of cheatsheet-style summaries to provide an overview and
memory jog without having to read all of the docs.

@item 
In some cases, the pure Python source code can be a helpful adjunct to the
documentation, so now many modules now feature quick links to the latest
version of the source code.  For example, the @ref{7f,,functools} module
documentation has a quick link at the top labeled:

@quotation

@strong{Source code} Lib/functools.py@footnote{https://github.com/python/cpython/tree/main/Lib/functools.py}.
@end quotation

(Contributed by Raymond Hettinger; see
rationale@footnote{https://rhettinger.wordpress.com/2011/01/28/open-your-source-more/}.)

@item 
The docs now contain more examples and recipes.  In particular, @ref{d8,,re}
module has an extensive section, @ref{e0e,,Regular Expression Examples}.  Likewise, the
@ref{9f,,itertools} module continues to be updated with new
@ref{e0f,,Itertools Recipes}.

@item 
The @ref{2f,,datetime} module now has an auxiliary implementation in pure Python.
No functionality was changed.  This just provides an easier-to-read alternate
implementation.

(Contributed by Alexander Belopolsky in bpo-9528@footnote{https://bugs.python.org/issue9528}.)

@item 
The unmaintained @code{Demo} directory has been removed.  Some demos were
integrated into the documentation, some were moved to the @code{Tools/demo}
directory, and others were removed altogether.

(Contributed by Georg Brandl in bpo-7962@footnote{https://bugs.python.org/issue7962}.)
@end itemize

@node IDLE,Code Repository,Documentation,What’s New In Python 3 2
@anchor{whatsnew/3 2 idle}@anchor{e10}
@subsection IDLE



@itemize *

@item 
The format menu now has an option to clean source files by stripping
trailing whitespace.

(Contributed by Raymond Hettinger; bpo-5150@footnote{https://bugs.python.org/issue5150}.)

@item 
IDLE on Mac OS X now works with both Carbon AquaTk and Cocoa AquaTk.

(Contributed by Kevin Walzer, Ned Deily, and Ronald Oussoren; bpo-6075@footnote{https://bugs.python.org/issue6075}.)
@end itemize

@node Code Repository,Build and C API Changes<5>,IDLE,What’s New In Python 3 2
@anchor{whatsnew/3 2 code-repository}@anchor{e11}
@subsection Code Repository


In addition to the existing Subversion code repository at @indicateurl{http://svn.python.org}
there is now a Mercurial@footnote{https://www.mercurial-scm.org/} repository at
@indicateurl{https://hg.python.org/}.

After the 3.2 release, there are plans to switch to Mercurial as the primary
repository.  This distributed version control system should make it easier for
members of the community to create and share external changesets.  See
@geindex Python Enhancement Proposals; PEP 385
PEP 385@footnote{https://www.python.org/dev/peps/pep-0385} for details.

To learn to use the new version control system, see the Quick Start@footnote{https://www.mercurial-scm.org/wiki/QuickStart} or the Guide to Mercurial Workflows@footnote{https://www.mercurial-scm.org/guide}.

@node Build and C API Changes<5>,Porting to Python 3 2,Code Repository,What’s New In Python 3 2
@anchor{whatsnew/3 2 build-and-c-api-changes}@anchor{e12}
@subsection Build and C API Changes


Changes to Python’s build process and to the C API include:


@itemize *

@item 
The @emph{idle}, @emph{pydoc} and @emph{2to3} scripts are now installed with a
version-specific suffix on @code{make altinstall} (bpo-10679@footnote{https://bugs.python.org/issue10679}).

@item 
The C functions that access the Unicode Database now accept and return
characters from the full Unicode range, even on narrow unicode builds
(Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others).  A visible difference
in Python is that @ref{e13,,unicodedata.numeric()} now returns the correct value
for large code points, and @ref{27d,,repr()} may consider more characters as
printable.

(Reported by Bupjoe Lee and fixed by Amaury Forgeot D’Arc; bpo-5127@footnote{https://bugs.python.org/issue5127}.)

@item 
Computed gotos are now enabled by default on supported compilers (which are
detected by the configure script).  They can still be disabled selectively by
specifying @code{--without-computed-gotos}.

(Contributed by Antoine Pitrou; bpo-9203@footnote{https://bugs.python.org/issue9203}.)

@item 
The option @code{--with-wctype-functions} was removed.  The built-in unicode
database is now used for all functions.

(Contributed by Amaury Forgeot D’Arc; bpo-9210@footnote{https://bugs.python.org/issue9210}.)

@item 
Hash values are now values of a new type, @code{Py_hash_t}, which is
defined to be the same size as a pointer.  Previously they were of type long,
which on some 64-bit operating systems is still only 32 bits long.  As a
result of this fix, @ref{370,,set} and @ref{3b0,,dict} can now hold more than
@code{2**32} entries on builds with 64-bit pointers (previously, they could grow
to that size but their performance degraded catastrophically).

(Suggested by Raymond Hettinger and implemented by Benjamin Peterson;
bpo-9778@footnote{https://bugs.python.org/issue9778}.)

@item 
A new macro @code{Py_VA_COPY} copies the state of the variable argument
list.  It is equivalent to C99 @emph{va_copy} but available on all Python platforms
(bpo-2443@footnote{https://bugs.python.org/issue2443}).

@item 
A new C API function @ref{e14,,PySys_SetArgvEx()} allows an embedded interpreter
to set @ref{e15,,sys.argv} without also modifying @ref{489,,sys.path}
(bpo-5753@footnote{https://bugs.python.org/issue5753}).

@item 
@code{PyEval_CallObject} is now only available in macro form.  The
function declaration, which was kept for backwards compatibility reasons, is
now removed – the macro was introduced in 1997 (bpo-8276@footnote{https://bugs.python.org/issue8276}).

@item 
There is a new function @ref{e16,,PyLong_AsLongLongAndOverflow()} which
is analogous to @ref{e17,,PyLong_AsLongAndOverflow()}.  They both serve to
convert Python @ref{1b1,,int} into a native fixed-width type while providing
detection of cases where the conversion won’t fit (bpo-7767@footnote{https://bugs.python.org/issue7767}).

@item 
The @ref{e18,,PyUnicode_CompareWithASCIIString()} function now returns @emph{not
equal} if the Python string is @emph{NUL} terminated.

@item 
There is a new function @ref{e19,,PyErr_NewExceptionWithDoc()} that is
like @ref{e1a,,PyErr_NewException()} but allows a docstring to be specified.
This lets C exceptions have the same self-documenting capabilities as
their pure Python counterparts (bpo-7033@footnote{https://bugs.python.org/issue7033}).

@item 
When compiled with the @code{--with-valgrind} option, the pymalloc
allocator will be automatically disabled when running under Valgrind.  This
gives improved memory leak detection when running under Valgrind, while taking
advantage of pymalloc at other times (bpo-2422@footnote{https://bugs.python.org/issue2422}).

@item 
Removed the @code{O?} format from the @emph{PyArg_Parse} functions.  The format is no
longer used and it had never been documented (bpo-8837@footnote{https://bugs.python.org/issue8837}).
@end itemize

There were a number of other small changes to the C-API.  See the
Misc/NEWS@footnote{https://github.com/python/cpython/tree/main/Misc/NEWS} file for a complete list.

Also, there were a number of updates to the Mac OS X build, see
Mac/BuildScript/README.txt@footnote{https://github.com/python/cpython/tree/main/Mac/BuildScript/README.txt} for details.  For users running a 32/64-bit
build, there is a known problem with the default Tcl/Tk on Mac OS X 10.6.
Accordingly, we recommend installing an updated alternative such as
ActiveState Tcl/Tk 8.5.9@footnote{https://www.activestate.com/activetcl/downloads}.
See @indicateurl{https://www.python.org/download/mac/tcltk/} for additional details.

@node Porting to Python 3 2,,Build and C API Changes<5>,What’s New In Python 3 2
@anchor{whatsnew/3 2 porting-to-python-3-2}@anchor{e1b}
@subsection Porting to Python 3.2


This section lists previously described changes and other bugfixes that may
require changes to your code:


@itemize *

@item 
The @ref{21,,configparser} module has a number of clean-ups.  The major change is
to replace the old @code{ConfigParser} class with long-standing preferred
alternative @code{SafeConfigParser}.  In addition there are a number of
smaller incompatibilities:


@itemize *

@item 
The interpolation syntax is now validated on
@ref{e1c,,get()} and
@ref{e1d,,set()} operations. In the default
interpolation scheme, only two tokens with percent signs are valid: @code{%(name)s}
and @code{%%}, the latter being an escaped percent sign.

@item 
The @ref{e1d,,set()} and
@ref{e1e,,add_section()} methods now verify that
values are actual strings.  Formerly, unsupported types could be introduced
unintentionally.

@item 
Duplicate sections or options from a single source now raise either
@ref{e1f,,DuplicateSectionError} or
@ref{e20,,DuplicateOptionError}.  Formerly, duplicates would
silently overwrite a previous entry.

@item 
Inline comments are now disabled by default so now the @strong{;} character
can be safely used in values.

@item 
Comments now can be indented.  Consequently, for @strong{;} or @strong{#} to appear at
the start of a line in multiline values, it has to be interpolated.  This
keeps comment prefix characters in values from being mistaken as comments.

@item 
@code{""} is now a valid value and is no longer automatically converted to an
empty string. For empty strings, use @code{"option ="} in a line.
@end itemize

@item 
The @ref{bc,,nntplib} module was reworked extensively, meaning that its APIs
are often incompatible with the 3.1 APIs.

@item 
@ref{2f1,,bytearray} objects can no longer be used as filenames; instead,
they should be converted to @ref{2f0,,bytes}.

@item 
The @code{array.tostring()} and @code{array.fromstring()} have been renamed to
@code{array.tobytes()} and @code{array.frombytes()} for clarity.  The old names
have been deprecated. (See bpo-8990@footnote{https://bugs.python.org/issue8990}.)

@item 
@code{PyArg_Parse*()} functions:


@itemize *

@item 
“t#” format has been removed: use “s#” or “s*” instead

@item 
“w” and “w#” formats has been removed: use “w*” instead
@end itemize

@item 
The @code{PyCObject} type, deprecated in 3.1, has been removed.  To wrap
opaque C pointers in Python objects, the @ref{e21,,PyCapsule} API should be used
instead; the new type has a well-defined interface for passing typing safety
information and a less complicated signature for calling a destructor.

@item 
The @code{sys.setfilesystemencoding()} function was removed because
it had a flawed design.

@item 
The @ref{e22,,random.seed()} function and method now salt string seeds with an
sha512 hash function.  To access the previous version of @emph{seed} in order to
reproduce Python 3.1 sequences, set the @emph{version} argument to @emph{1},
@code{random.seed(s, version=1)}.

@item 
The previously deprecated @code{string.maketrans()} function has been removed
in favor of the static methods @ref{e23,,bytes.maketrans()} and
@ref{e24,,bytearray.maketrans()}.  This change solves the confusion around which
types were supported by the @ref{f1,,string} module.  Now, @ref{2ef,,str},
@ref{2f0,,bytes}, and @ref{2f1,,bytearray} each have their own @strong{maketrans} and
@strong{translate} methods with intermediate translation tables of the appropriate
type.

(Contributed by Georg Brandl; bpo-5675@footnote{https://bugs.python.org/issue5675}.)

@item 
The previously deprecated @code{contextlib.nested()} function has been removed
in favor of a plain @ref{168,,with} statement which can accept multiple
context managers.  The latter technique is faster (because it is built-in),
and it does a better job finalizing multiple context managers when one of them
raises an exception:

@example
with open('mylog.txt') as infile, open('a.out', 'w') as outfile:
    for line in infile:
        if '<critical>' in line:
            outfile.write(line)
@end example

(Contributed by Georg Brandl and Mattias Brändström;
appspot issue 53094@footnote{https://codereview.appspot.com/53094}.)

@item 
@ref{e25,,struct.pack()} now only allows bytes for the @code{s} string pack code.
Formerly, it would accept text arguments and implicitly encode them to bytes
using UTF-8.  This was problematic because it made assumptions about the
correct encoding and because a variable-length encoding can fail when writing
to fixed length segment of a structure.

Code such as @code{struct.pack('<6sHHBBB', 'GIF87a', x, y)} should be rewritten
with to use bytes instead of text, @code{struct.pack('<6sHHBBB', b'GIF87a', x, y)}.

(Discovered by David Beazley and fixed by Victor Stinner; bpo-10783@footnote{https://bugs.python.org/issue10783}.)

@item 
The @ref{13e,,xml.etree.ElementTree} class now raises an
@ref{e26,,xml.etree.ElementTree.ParseError} when a parse fails. Previously it
raised an @ref{e27,,xml.parsers.expat.ExpatError}.

@item 
The new, longer @ref{2ef,,str()} value on floats may break doctests which rely on
the old output format.

@item 
In @ref{5ae,,subprocess.Popen}, the default value for @emph{close_fds} is now
@code{True} under Unix; under Windows, it is @code{True} if the three standard
streams are set to @code{None}, @code{False} otherwise.  Previously, @emph{close_fds}
was always @code{False} by default, which produced difficult to solve bugs
or race conditions when open file descriptors would leak into the child
process.

@item 
Support for legacy HTTP 0.9 has been removed from @ref{127,,urllib.request}
and @ref{8f,,http.client}.  Such support is still present on the server side
(in @ref{92,,http.server}).

(Contributed by Antoine Pitrou, bpo-10711@footnote{https://bugs.python.org/issue10711}.)

@item 
SSL sockets in timeout mode now raise @ref{2b7,,socket.timeout} when a timeout
occurs, rather than a generic @ref{e28,,SSLError}.

(Contributed by Antoine Pitrou, bpo-10272@footnote{https://bugs.python.org/issue10272}.)

@item 
The misleading functions @ref{5a7,,PyEval_AcquireLock()} and
@ref{e29,,PyEval_ReleaseLock()} have been officially deprecated.  The
thread-state aware APIs (such as @ref{e2a,,PyEval_SaveThread()}
and @ref{5a9,,PyEval_RestoreThread()}) should be used instead.

@item 
Due to security risks, @code{asyncore.handle_accept()} has been deprecated, and
a new function, @code{asyncore.handle_accepted()}, was added to replace it.

(Contributed by Giampaolo Rodola in bpo-6706@footnote{https://bugs.python.org/issue6706}.)

@item 
Due to the new @ref{385,,GIL} implementation, @ref{441,,PyEval_InitThreads()}
cannot be called before @ref{38d,,Py_Initialize()} anymore.
@end itemize

@node What’s New In Python 3 1,What’s New In Python 3 0,What’s New In Python 3 2,What’s New in Python
@anchor{whatsnew/3 1 doc}@anchor{e2b}@anchor{whatsnew/3 1 what-s-new-in-python-3-1}@anchor{e2c}
@section What’s New In Python 3.1


@*Author: 
Raymond Hettinger


@c $Id$
@c Rules for maintenance:
@c 
@c * Anyone can add text to this document.  Do not spend very much time
@c on the wording of your changes, because your text will probably
@c get rewritten to some degree.
@c 
@c * The maintainer will go through Misc/NEWS periodically and add
@c changes; it's therefore more important to add your changes to
@c Misc/NEWS than to this file.
@c 
@c * This is not a complete list of every single change; completeness
@c is the purpose of Misc/NEWS.  Some changes I consider too small
@c or esoteric to include.  If such a change is added to the text,
@c I'll just remove it.  (This is another reason you shouldn't spend
@c too much time on writing your addition.)
@c 
@c * If you want to draw your new text to the attention of the
@c maintainer, add 'XXX' to the beginning of the paragraph or
@c section.
@c 
@c * It's OK to just add a fragmentary note about a change.  For
@c example: "XXX Describe the transmogrify() function added to the
@c socket module."  The maintainer will research the change and
@c write the necessary text.
@c 
@c * You can comment out your additions if you like, but it's not
@c necessary (especially when a final release is some months away).
@c 
@c * Credit the author of a patch or bugfix.   Just the name is
@c sufficient; the e-mail address isn't necessary.
@c 
@c * It's helpful to add the bug/patch number as a comment:
@c 
@c % Patch 12345
@c XXX Describe the transmogrify() function added to the socket
@c module.
@c (Contributed by P.Y. Developer.)
@c 
@c This saves the maintainer the effort of going through the SVN log
@c when researching a change.

This article explains the new features in Python 3.1, compared to 3.0.

@menu
* PEP 372; Ordered Dictionaries: PEP 372 Ordered Dictionaries. 
* PEP 378; Format Specifier for Thousands Separator: PEP 378 Format Specifier for Thousands Separator. 
* Other Language Changes: Other Language Changes<11>. 
* New@comma{} Improved@comma{} and Deprecated Modules: New Improved and Deprecated Modules<2>. 
* Optimizations: Optimizations<10>. 
* IDLE: IDLE<2>. 
* Build and C API Changes: Build and C API Changes<6>. 
* Porting to Python 3.1: Porting to Python 3 1. 

@end menu

@node PEP 372 Ordered Dictionaries,PEP 378 Format Specifier for Thousands Separator,,What’s New In Python 3 1
@anchor{whatsnew/3 1 pep-372-ordered-dictionaries}@anchor{e2d}
@subsection PEP 372: Ordered Dictionaries


Regular Python dictionaries iterate over key/value pairs in arbitrary order.
Over the years, a number of authors have written alternative implementations
that remember the order that the keys were originally inserted.  Based on
the experiences from those implementations, a new
@ref{4e5,,collections.OrderedDict} class has been introduced.

The OrderedDict API is substantially the same as regular dictionaries
but will iterate over keys and values in a guaranteed order depending on
when a key was first inserted.  If a new entry overwrites an existing entry,
the original insertion position is left unchanged.  Deleting an entry and
reinserting it will move it to the end.

The standard library now supports use of ordered dictionaries in several
modules.  The @ref{21,,configparser} module uses them by default.  This lets
configuration files be read, modified, and then written back in their original
order.  The @emph{_asdict()} method for @ref{3e4,,collections.namedtuple()} now
returns an ordered dictionary with the values appearing in the same order as
the underlying tuple indices.  The @ref{a0,,json} module is being built-out with
an @emph{object_pairs_hook} to allow OrderedDicts to be built by the decoder.
Support was also added for third-party tools like PyYAML@footnote{http://pyyaml.org/}.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 372

@item PEP 372@footnote{https://www.python.org/dev/peps/pep-0372} - Ordered Dictionaries

PEP written by Armin Ronacher and Raymond Hettinger.  Implementation
written by Raymond Hettinger.
@end table


@node PEP 378 Format Specifier for Thousands Separator,Other Language Changes<11>,PEP 372 Ordered Dictionaries,What’s New In Python 3 1
@anchor{whatsnew/3 1 pep-378-format-specifier-for-thousands-separator}@anchor{e2e}
@subsection PEP 378: Format Specifier for Thousands Separator


The built-in @ref{785,,format()} function and the @ref{784,,str.format()} method use
a mini-language that now includes a simple, non-locale aware way to format
a number with a thousands separator.  That provides a way to humanize a
program’s output, improving its professional appearance and readability:

@example
>>> format(1234567, ',d')
'1,234,567'
>>> format(1234567.89, ',.2f')
'1,234,567.89'
>>> format(12345.6 + 8901234.12j, ',f')
'12,345.600000+8,901,234.120000j'
>>> format(Decimal('1234567.89'), ',f')
'1,234,567.89'
@end example

The supported types are @ref{1b1,,int}, @ref{256,,float}, @ref{33b,,complex}
and @ref{248,,decimal.Decimal}.

Discussions are underway about how to specify alternative separators
like dots, spaces, apostrophes, or underscores.  Locale-aware applications
should use the existing @emph{n} format specifier which already has some support
for thousands separators.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 378

@item PEP 378@footnote{https://www.python.org/dev/peps/pep-0378} - Format Specifier for Thousands Separator

PEP written by Raymond Hettinger and implemented by Eric Smith and
Mark Dickinson.
@end table


@node Other Language Changes<11>,New Improved and Deprecated Modules<2>,PEP 378 Format Specifier for Thousands Separator,What’s New In Python 3 1
@anchor{whatsnew/3 1 other-language-changes}@anchor{e2f}
@subsection Other Language Changes


Some smaller changes made to the core Python language are:


@itemize *

@item 
Directories and zip archives containing a @code{__main__.py}
file can now be executed directly by passing their name to the
interpreter. The directory/zipfile is automatically inserted as the
first entry in sys.path.  (Suggestion and initial patch by Andy Chu;
revised patch by Phillip J. Eby and Nick Coghlan; bpo-1739468@footnote{https://bugs.python.org/issue1739468}.)

@item 
The @ref{1b1,,int()} type gained a @code{bit_length} method that returns the
number of bits necessary to represent its argument in binary:

@example
>>> n = 37
>>> bin(37)
'0b100101'
>>> n.bit_length()
6
>>> n = 2**123-1
>>> n.bit_length()
123
>>> (n+1).bit_length()
124
@end example

(Contributed by Fredrik Johansson, Victor Stinner, Raymond Hettinger,
and Mark Dickinson; bpo-3439@footnote{https://bugs.python.org/issue3439}.)

@item 
The fields in @ref{785,,format()} strings can now be automatically
numbered:

@example
>>> 'Sir @{@} of @{@}'.format('Gallahad', 'Camelot')
'Sir Gallahad of Camelot'
@end example

Formerly, the string would have required numbered fields such as:
@code{'Sir @{0@} of @{1@}'}.

(Contributed by Eric Smith; bpo-5237@footnote{https://bugs.python.org/issue5237}.)

@item 
The @code{string.maketrans()} function is deprecated and is replaced by new
static methods, @ref{e23,,bytes.maketrans()} and @ref{e24,,bytearray.maketrans()}.
This change solves the confusion around which types were supported by the
@ref{f1,,string} module. Now, @ref{2ef,,str}, @ref{2f0,,bytes}, and
@ref{2f1,,bytearray} each have their own @strong{maketrans} and @strong{translate}
methods with intermediate translation tables of the appropriate type.

(Contributed by Georg Brandl; bpo-5675@footnote{https://bugs.python.org/issue5675}.)

@item 
The syntax of the @ref{168,,with} statement now allows multiple context
managers in a single statement:

@example
>>> with open('mylog.txt') as infile, open('a.out', 'w') as outfile:
...     for line in infile:
...         if '<critical>' in line:
...             outfile.write(line)
@end example

With the new syntax, the @code{contextlib.nested()} function is no longer
needed and is now deprecated.

(Contributed by Georg Brandl and Mattias Brändström;
appspot issue 53094@footnote{https://codereview.appspot.com/53094}.)

@item 
@code{round(x, n)} now returns an integer if @emph{x} is an integer.
Previously it returned a float:

@example
>>> round(1123, -2)
1100
@end example

(Contributed by Mark Dickinson; bpo-4707@footnote{https://bugs.python.org/issue4707}.)

@item 
Python now uses David Gay’s algorithm for finding the shortest floating
point representation that doesn’t change its value.  This should help
mitigate some of the confusion surrounding binary floating point
numbers.

The significance is easily seen with a number like @code{1.1} which does not
have an exact equivalent in binary floating point.  Since there is no exact
equivalent, an expression like @code{float('1.1')} evaluates to the nearest
representable value which is @code{0x1.199999999999ap+0} in hex or
@code{1.100000000000000088817841970012523233890533447265625} in decimal. That
nearest value was and still is used in subsequent floating point
calculations.

What is new is how the number gets displayed.  Formerly, Python used a
simple approach.  The value of @code{repr(1.1)} was computed as @code{format(1.1,
'.17g')} which evaluated to @code{'1.1000000000000001'}. The advantage of
using 17 digits was that it relied on IEEE-754 guarantees to assure that
@code{eval(repr(1.1))} would round-trip exactly to its original value.  The
disadvantage is that many people found the output to be confusing (mistaking
intrinsic limitations of binary floating point representation as being a
problem with Python itself).

The new algorithm for @code{repr(1.1)} is smarter and returns @code{'1.1'}.
Effectively, it searches all equivalent string representations (ones that
get stored with the same underlying float value) and returns the shortest
representation.

The new algorithm tends to emit cleaner representations when possible, but
it does not change the underlying values.  So, it is still the case that
@code{1.1 + 2.2 != 3.3} even though the representations may suggest otherwise.

The new algorithm depends on certain features in the underlying floating
point implementation.  If the required features are not found, the old
algorithm will continue to be used.  Also, the text pickle protocols
assure cross-platform portability by using the old algorithm.

(Contributed by Eric Smith and Mark Dickinson; bpo-1580@footnote{https://bugs.python.org/issue1580})
@end itemize

@node New Improved and Deprecated Modules<2>,Optimizations<10>,Other Language Changes<11>,What’s New In Python 3 1
@anchor{whatsnew/3 1 new-improved-and-deprecated-modules}@anchor{e30}
@subsection New, Improved, and Deprecated Modules



@itemize *

@item 
Added a @ref{c2a,,collections.Counter} class to support convenient
counting of unique items in a sequence or iterable:

@example
>>> Counter(['red', 'blue', 'red', 'green', 'blue', 'blue'])
Counter(@{'blue': 3, 'red': 2, 'green': 1@})
@end example

(Contributed by Raymond Hettinger; bpo-1696199@footnote{https://bugs.python.org/issue1696199}.)

@item 
Added a new module, @ref{116,,tkinter.ttk} for access to the Tk themed widget set.
The basic idea of ttk is to separate, to the extent possible, the code
implementing a widget’s behavior from the code implementing its appearance.

(Contributed by Guilherme Polo; bpo-2983@footnote{https://bugs.python.org/issue2983}.)

@item 
The @ref{43f,,gzip.GzipFile} and @ref{455,,bz2.BZ2File} classes now support
the context management protocol:

@example
>>> # Automatically close file after writing
>>> with gzip.GzipFile(filename, "wb") as f:
...     f.write(b"xxx")
@end example

(Contributed by Antoine Pitrou.)

@item 
The @ref{34,,decimal} module now supports methods for creating a
decimal object from a binary @ref{256,,float}.  The conversion is
exact but can sometimes be surprising:

@example
>>> Decimal.from_float(1.1)
Decimal('1.100000000000000088817841970012523233890533447265625')
@end example

The long decimal result shows the actual binary fraction being
stored for @emph{1.1}.  The fraction has many digits because @emph{1.1} cannot
be exactly represented in binary.

(Contributed by Raymond Hettinger and Mark Dickinson.)

@item 
The @ref{9f,,itertools} module grew two new functions.  The
@ref{e31,,itertools.combinations_with_replacement()} function is one of
four for generating combinatorics including permutations and Cartesian
products.  The @ref{e32,,itertools.compress()} function mimics its namesake
from APL.  Also, the existing @ref{e33,,itertools.count()} function now has
an optional @emph{step} argument and can accept any type of counting
sequence including @ref{172,,fractions.Fraction} and
@ref{248,,decimal.Decimal}:

@example
>>> [p+q for p,q in combinations_with_replacement('LOVE', 2)]
['LL', 'LO', 'LV', 'LE', 'OO', 'OV', 'OE', 'VV', 'VE', 'EE']

>>> list(compress(data=range(10), selectors=[0,0,1,1,0,1,0,1,0,0]))
[2, 3, 5, 7]

>>> c = count(start=Fraction(1,2), step=Fraction(1,6))
>>> [next(c), next(c), next(c), next(c)]
[Fraction(1, 2), Fraction(2, 3), Fraction(5, 6), Fraction(1, 1)]
@end example

(Contributed by Raymond Hettinger.)

@item 
@ref{3e4,,collections.namedtuple()} now supports a keyword argument
@emph{rename} which lets invalid fieldnames be automatically converted to
positional names in the form _0, _1, etc.  This is useful when
the field names are being created by an external source such as a
CSV header, SQL field list, or user input:

@example
>>> query = input()
SELECT region, dept, count(*) FROM main GROUPBY region, dept

>>> cursor.execute(query)
>>> query_fields = [desc[0] for desc in cursor.description]
>>> UserQuery = namedtuple('UserQuery', query_fields, rename=True)
>>> pprint.pprint([UserQuery(*row) for row in cursor])
[UserQuery(region='South', dept='Shipping', _2=185),
 UserQuery(region='North', dept='Accounting', _2=37),
 UserQuery(region='West', dept='Sales', _2=419)]
@end example

(Contributed by Raymond Hettinger; bpo-1818@footnote{https://bugs.python.org/issue1818}.)

@item 
The @ref{72e,,re.sub()}, @ref{9b3,,re.subn()} and @ref{699,,re.split()} functions now
accept a flags parameter.

(Contributed by Gregory Smith.)

@item 
The @ref{a6,,logging} module now implements a simple @ref{e34,,logging.NullHandler}
class for applications that are not using logging but are calling
library code that does.  Setting-up a null handler will suppress
spurious warnings such as “No handlers could be found for logger foo”:

@example
>>> h = logging.NullHandler()
>>> logging.getLogger("foo").addHandler(h)
@end example

(Contributed by Vinay Sajip; bpo-4384@footnote{https://bugs.python.org/issue4384}).

@item 
The @ref{dd,,runpy} module which supports the @code{-m} command line switch
now supports the execution of packages by looking for and executing
a @code{__main__} submodule when a package name is supplied.

(Contributed by Andi Vajda; bpo-4195@footnote{https://bugs.python.org/issue4195}.)

@item 
The @ref{c4,,pdb} module can now access and display source code loaded via
@ref{14a,,zipimport} (or any other conformant 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} loader).

(Contributed by Alexander Belopolsky; bpo-4201@footnote{https://bugs.python.org/issue4201}.)

@item 
@ref{a3f,,functools.partial} objects can now be pickled.
@end itemize

@quotation

(Suggested by Antoine Pitrou and Jesse Noller.  Implemented by
Jack Diederich; bpo-5228@footnote{https://bugs.python.org/issue5228}.)
@end quotation


@itemize *

@item 
Add @ref{d4,,pydoc} help topics for symbols so that @code{help('@@')}
works as expected in the interactive environment.

(Contributed by David Laban; bpo-4739@footnote{https://bugs.python.org/issue4739}.)

@item 
The @ref{122,,unittest} module now supports skipping individual tests or classes
of tests. And it supports marking a test as an expected failure, a test that
is known to be broken, but shouldn’t be counted as a failure on a
TestResult:

@example
class TestGizmo(unittest.TestCase):

    @@unittest.skipUnless(sys.platform.startswith("win"), "requires Windows")
    def test_gizmo_on_windows(self):
        ...

    @@unittest.expectedFailure
    def test_gimzo_without_required_library(self):
        ...
@end example

Also, tests for exceptions have been builtout to work with context managers
using the @ref{168,,with} statement:

@example
def test_division_by_zero(self):
    with self.assertRaises(ZeroDivisionError):
        x / 0
@end example

In addition, several new assertion methods were added including
@code{assertSetEqual()}, @code{assertDictEqual()},
@code{assertDictContainsSubset()}, @code{assertListEqual()},
@code{assertTupleEqual()}, @code{assertSequenceEqual()},
@code{assertRaisesRegexp()}, @code{assertIsNone()},
and @code{assertIsNotNone()}.

(Contributed by Benjamin Peterson and Antoine Pitrou.)

@item 
The @ref{9d,,io} module has three new constants for the @code{seek()}
method @code{SEEK_SET}, @code{SEEK_CUR}, and @code{SEEK_END}.

@item 
The @ref{d45,,sys.version_info} tuple is now a named tuple:

@example
>>> sys.version_info
sys.version_info(major=3, minor=1, micro=0, releaselevel='alpha', serial=2)
@end example

(Contributed by Ross Light; bpo-4285@footnote{https://bugs.python.org/issue4285}.)

@item 
The @ref{bc,,nntplib} and @ref{93,,imaplib} modules now support IPv6.

(Contributed by Derek Morr; bpo-1655@footnote{https://bugs.python.org/issue1655} and bpo-1664@footnote{https://bugs.python.org/issue1664}.)

@item 
The @ref{c5,,pickle} module has been adapted for better interoperability with
Python 2.x when used with protocol 2 or lower.  The reorganization of the
standard library changed the formal reference for many objects.  For
example, @code{__builtin__.set} in Python 2 is called @code{builtins.set} in Python
3. This change confounded efforts to share data between different versions of
Python.  But now when protocol 2 or lower is selected, the pickler will
automatically use the old Python 2 names for both loading and dumping. This
remapping is turned-on by default but can be disabled with the @emph{fix_imports}
option:

@example
>>> s = @{1, 2, 3@}
>>> pickle.dumps(s, protocol=0)
b'c__builtin__\nset\np0\n((lp1\nL1L\naL2L\naL3L\natp2\nRp3\n.'
>>> pickle.dumps(s, protocol=0, fix_imports=False)
b'cbuiltins\nset\np0\n((lp1\nL1L\naL2L\naL3L\natp2\nRp3\n.'
@end example

An unfortunate but unavoidable side-effect of this change is that protocol 2
pickles produced by Python 3.1 won’t be readable with Python 3.0. The latest
pickle protocol, protocol 3, should be used when migrating data between
Python 3.x implementations, as it doesn’t attempt to remain compatible with
Python 2.x.

(Contributed by Alexandre Vassalotti and Antoine Pitrou, bpo-6137@footnote{https://bugs.python.org/issue6137}.)

@item 
A new module, @ref{96,,importlib} was added.  It provides a complete, portable,
pure Python reference implementation of the @ref{e35,,import} statement and its
counterpart, the @ref{3b7,,__import__()} function.  It represents a substantial
step forward in documenting and defining the actions that take place during
imports.

(Contributed by Brett Cannon.)
@end itemize

@node Optimizations<10>,IDLE<2>,New Improved and Deprecated Modules<2>,What’s New In Python 3 1
@anchor{whatsnew/3 1 optimizations}@anchor{e36}
@subsection Optimizations


Major performance enhancements have been added:


@itemize *

@item 
The new I/O library (as defined in 
@geindex Python Enhancement Proposals; PEP 3116
PEP 3116@footnote{https://www.python.org/dev/peps/pep-3116}) was mostly written in
Python and quickly proved to be a problematic bottleneck in Python 3.0.
In Python 3.1, the I/O library has been entirely rewritten in C and is
2 to 20 times faster depending on the task at hand. The pure Python
version is still available for experimentation purposes through
the @code{_pyio} module.

(Contributed by Amaury Forgeot d’Arc and Antoine Pitrou.)

@item 
Added a heuristic so that tuples and dicts containing only untrackable objects
are not tracked by the garbage collector. This can reduce the size of
collections and therefore the garbage collection overhead on long-running
programs, depending on their particular use of datatypes.

(Contributed by Antoine Pitrou, bpo-4688@footnote{https://bugs.python.org/issue4688}.)

@item 
Enabling a configure option named @code{--with-computed-gotos}
on compilers that support it (notably: gcc, SunPro, icc), the bytecode
evaluation loop is compiled with a new dispatch mechanism which gives
speedups of up to 20%, depending on the system, the compiler, and
the benchmark.

(Contributed by Antoine Pitrou along with a number of other participants,
bpo-4753@footnote{https://bugs.python.org/issue4753}).

@item 
The decoding of UTF-8, UTF-16 and LATIN-1 is now two to four times
faster.

(Contributed by Antoine Pitrou and Amaury Forgeot d’Arc, bpo-4868@footnote{https://bugs.python.org/issue4868}.)

@item 
The @ref{a0,,json} module now has a C extension to substantially improve
its performance.  In addition, the API was modified so that json works
only with @ref{2ef,,str}, not with @ref{2f0,,bytes}.  That change makes the
module closely match the JSON specification@footnote{http://json.org/}
which is defined in terms of Unicode.

(Contributed by Bob Ippolito and converted to Py3.1 by Antoine Pitrou
and Benjamin Peterson; bpo-4136@footnote{https://bugs.python.org/issue4136}.)

@item 
Unpickling now interns the attribute names of pickled objects.  This saves
memory and allows pickles to be smaller.

(Contributed by Jake McGuire and Antoine Pitrou; bpo-5084@footnote{https://bugs.python.org/issue5084}.)
@end itemize

@node IDLE<2>,Build and C API Changes<6>,Optimizations<10>,What’s New In Python 3 1
@anchor{whatsnew/3 1 idle}@anchor{e37}
@subsection IDLE



@itemize *

@item 
IDLE’s format menu now provides an option to strip trailing whitespace
from a source file.

(Contributed by Roger D. Serwy; bpo-5150@footnote{https://bugs.python.org/issue5150}.)
@end itemize

@node Build and C API Changes<6>,Porting to Python 3 1,IDLE<2>,What’s New In Python 3 1
@anchor{whatsnew/3 1 build-and-c-api-changes}@anchor{e38}
@subsection Build and C API Changes


Changes to Python’s build process and to the C API include:


@itemize *

@item 
Integers are now stored internally either in base @code{2**15} or in base
@code{2**30}, the base being determined at build time.  Previously, they
were always stored in base @code{2**15}.  Using base @code{2**30} gives
significant performance improvements on 64-bit machines, but
benchmark results on 32-bit machines have been mixed.  Therefore,
the default is to use base @code{2**30} on 64-bit machines and base @code{2**15}
on 32-bit machines; on Unix, there’s a new configure option
@code{--enable-big-digits} that can be used to override this default.

Apart from the performance improvements this change should be invisible to
end users, with one exception: for testing and debugging purposes there’s a
new @ref{e39,,sys.int_info} that provides information about the
internal format, giving the number of bits per digit and the size in bytes
of the C type used to store each digit:

@example
>>> import sys
>>> sys.int_info
sys.int_info(bits_per_digit=30, sizeof_digit=4)
@end example

(Contributed by Mark Dickinson; bpo-4258@footnote{https://bugs.python.org/issue4258}.)

@item 
The @ref{e3a,,PyLong_AsUnsignedLongLong()} function now handles a negative
@emph{pylong} by raising @ref{347,,OverflowError} instead of @ref{162,,TypeError}.

(Contributed by Mark Dickinson and Lisandro Dalcrin; bpo-5175@footnote{https://bugs.python.org/issue5175}.)

@item 
Deprecated @code{PyNumber_Int()}.  Use @ref{57e,,PyNumber_Long()} instead.

(Contributed by Mark Dickinson; bpo-4910@footnote{https://bugs.python.org/issue4910}.)

@item 
Added a new @ref{e3b,,PyOS_string_to_double()} function to replace the
deprecated functions @code{PyOS_ascii_strtod()} and @code{PyOS_ascii_atof()}.

(Contributed by Mark Dickinson; bpo-5914@footnote{https://bugs.python.org/issue5914}.)

@item 
Added @ref{e21,,PyCapsule} as a replacement for the @code{PyCObject} API.
The principal difference is that the new type has a well defined interface
for passing typing safety information and a less complicated signature
for calling a destructor.  The old type had a problematic API and is now
deprecated.

(Contributed by Larry Hastings; bpo-5630@footnote{https://bugs.python.org/issue5630}.)
@end itemize

@node Porting to Python 3 1,,Build and C API Changes<6>,What’s New In Python 3 1
@anchor{whatsnew/3 1 porting-to-python-3-1}@anchor{e3c}
@subsection Porting to Python 3.1


This section lists previously described changes and other bugfixes
that may require changes to your code:


@itemize *

@item 
The new floating point string representations can break existing doctests.
For example:

@example
def e():
    '@w{'}'Compute the base of natural logarithms.

    >>> e()
    2.7182818284590451

    '@w{'}'
    return sum(1/math.factorial(x) for x in reversed(range(30)))

doctest.testmod()

**********************************************************************
Failed example:
    e()
Expected:
    2.7182818284590451
Got:
    2.718281828459045
**********************************************************************
@end example

@item 
The automatic name remapping in the pickle module for protocol 2 or lower can
make Python 3.1 pickles unreadable in Python 3.0.  One solution is to use
protocol 3.  Another solution is to set the @emph{fix_imports} option to @code{False}.
See the discussion above for more details.
@end itemize

@node What’s New In Python 3 0,What’s New in Python 2 7,What’s New In Python 3 1,What’s New in Python
@anchor{whatsnew/3 0 doc}@anchor{e3d}@anchor{whatsnew/3 0 what-s-new-in-python-3-0}@anchor{e3e}
@section What’s New In Python 3.0


@c XXX Add trademark info for Apple, Microsoft.

@*Author: 
Guido van Rossum


@c $Id$
@c Rules for maintenance:
@c 
@c * Anyone can add text to this document.  Do not spend very much time
@c on the wording of your changes, because your text will probably
@c get rewritten to some degree.
@c 
@c * The maintainer will go through Misc/NEWS periodically and add
@c changes; it's therefore more important to add your changes to
@c Misc/NEWS than to this file.  (Note: I didn't get to this for 3.0.
@c GvR.)
@c 
@c * This is not a complete list of every single change; completeness
@c is the purpose of Misc/NEWS.  Some changes I consider too small
@c or esoteric to include.  If such a change is added to the text,
@c I'll just remove it.  (This is another reason you shouldn't spend
@c too much time on writing your addition.)
@c 
@c * If you want to draw your new text to the attention of the
@c maintainer, add 'XXX' to the beginning of the paragraph or
@c section.
@c 
@c * It's OK to just add a fragmentary note about a change.  For
@c example: "XXX Describe the transmogrify() function added to the
@c socket module."  The maintainer will research the change and
@c write the necessary text.
@c 
@c * You can comment out your additions if you like, but it's not
@c necessary (especially when a final release is some months away).
@c 
@c * Credit the author of a patch or bugfix.   Just the name is
@c sufficient; the e-mail address isn't necessary.  (Due to time
@c constraints I haven't managed to do this for 3.0.  GvR.)
@c 
@c * It's helpful to add the bug/patch number as a comment:
@c 
@c % Patch 12345
@c XXX Describe the transmogrify() function added to the socket
@c module.
@c (Contributed by P.Y. Developer.)
@c 
@c This saves the maintainer the effort of going through the SVN log
@c when researching a change.  (Again, I didn't get to this for 3.0.
@c GvR.)

This article explains the new features in Python 3.0, compared to 2.6.
Python 3.0, also known as “Python 3000” or “Py3K”, is the first ever
@emph{intentionally backwards incompatible} Python release.  There are more
changes than in a typical release, and more that are important for all
Python users.  Nevertheless, after digesting the changes, you’ll find
that Python really hasn’t changed all that much – by and large, we’re
mostly fixing well-known annoyances and warts, and removing a lot of
old cruft.

This article doesn’t attempt to provide a complete specification of
all new features, but instead tries to give a convenient overview.
For full details, you should refer to the documentation for Python
3.0, and/or the many PEPs referenced in the text. If you want to
understand the complete implementation and design rationale for a
particular feature, PEPs usually have more details than the regular
documentation; but note that PEPs usually are not kept up-to-date once
a feature has been fully implemented.

Due to time constraints this document is not as complete as it should
have been.  As always for a new release, the @code{Misc/NEWS} file in the
source distribution contains a wealth of detailed information about
every small thing that was changed.

@c Compare with previous release in 2 - 3 sentences here.

@c add hyperlink when the documentation becomes available online.

@c ======================================================================

@c Large, PEP-level features and changes should be described here.

@c Should there be a new section here for 3k migration?

@c Or perhaps a more general section describing module changes/deprecation?

@c sets module deprecated

@c ======================================================================

@menu
* Common Stumbling Blocks:: 
* Overview Of Syntax Changes:: 
* Changes Already Present In Python 2.6: Changes Already Present In Python 2 6. 
* Library Changes:: 
* PEP 3101; A New Approach To String Formatting: PEP 3101 A New Approach To String Formatting. 
* Changes To Exceptions:: 
* Miscellaneous Other Changes:: 
* Build and C API Changes: Build and C API Changes<7>. 
* Performance:: 
* Porting To Python 3.0: Porting To Python 3 0. 

@end menu

@node Common Stumbling Blocks,Overview Of Syntax Changes,,What’s New In Python 3 0
@anchor{whatsnew/3 0 common-stumbling-blocks}@anchor{e3f}
@subsection Common Stumbling Blocks


This section lists those few changes that are most likely to trip you
up if you’re used to Python 2.5.

@menu
* Print Is A Function:: 
* Views And Iterators Instead Of Lists:: 
* Ordering Comparisons:: 
* Integers:: 
* Text Vs. Data Instead Of Unicode Vs. 8-bit: Text Vs Data Instead Of Unicode Vs 8-bit. 

@end menu

@node Print Is A Function,Views And Iterators Instead Of Lists,,Common Stumbling Blocks
@anchor{whatsnew/3 0 print-is-a-function}@anchor{e40}
@subsubsection Print Is A Function


The @code{print} statement has been replaced with a @ref{af6,,print()}
function, with keyword arguments to replace most of the special syntax
of the old @code{print} statement (
@geindex Python Enhancement Proposals; PEP 3105
PEP 3105@footnote{https://www.python.org/dev/peps/pep-3105}).  Examples:

@example
Old: print "The answer is", 2*2
New: print("The answer is", 2*2)

Old: print x,           # Trailing comma suppresses newline
New: print(x, end=" ")  # Appends a space instead of a newline

Old: print              # Prints a newline
New: print()            # You must call the function!

Old: print >>sys.stderr, "fatal error"
New: print("fatal error", file=sys.stderr)

Old: print (x, y)       # prints repr((x, y))
New: print((x, y))      # Not the same as print(x, y)!
@end example

You can also customize the separator between items, e.g.:

@example
print("There are <", 2**32, "> possibilities!", sep="")
@end example

which produces:

@example
There are <4294967296> possibilities!
@end example

Note:


@itemize *

@item 
The @ref{af6,,print()} function doesn’t support the “softspace” feature of
the old @code{print} statement.  For example, in Python 2.x,
@code{print "A\n", "B"} would write @code{"A\nB\n"}; but in Python 3.0,
@code{print("A\n", "B")} writes @code{"A\n B\n"}.

@item 
Initially, you’ll be finding yourself typing the old @code{print x}
a lot in interactive mode.  Time to retrain your fingers to type
@code{print(x)} instead!

@item 
When using the @code{2to3} source-to-source conversion tool, all
@code{print} statements are automatically converted to
@ref{af6,,print()} function calls, so this is mostly a non-issue for
larger projects.
@end itemize

@node Views And Iterators Instead Of Lists,Ordering Comparisons,Print Is A Function,Common Stumbling Blocks
@anchor{whatsnew/3 0 views-and-iterators-instead-of-lists}@anchor{e41}
@subsubsection Views And Iterators Instead Of Lists


Some well-known APIs no longer return lists:


@itemize *

@item 
@ref{3b0,,dict} methods @ref{243,,dict.keys()}, @ref{245,,dict.items()} and
@ref{244,,dict.values()} return “views” instead of lists.  For example,
this no longer works: @code{k = d.keys(); k.sort()}.  Use @code{k =
sorted(d)} instead (this works in Python 2.5 too and is just
as efficient).

@item 
Also, the @code{dict.iterkeys()}, @code{dict.iteritems()} and
@code{dict.itervalues()} methods are no longer supported.

@item 
@ref{2f4,,map()} and @ref{2f5,,filter()} return iterators.  If you really need
a list and the input sequences are all of equal length, a quick
fix is to wrap @ref{2f4,,map()} in @ref{43a,,list()}, e.g. @code{list(map(...))},
but a better fix is
often to use a list comprehension (especially when the original code
uses @ref{e42,,lambda}), or rewriting the code so it doesn’t need a
list at all.  Particularly tricky is @ref{2f4,,map()} invoked for the
side effects of the function; the correct transformation is to use a
regular @ref{161,,for} loop (since creating a list would just be
wasteful).

If the input sequences are not of equal length, @ref{2f4,,map()} will
stop at the termination of the shortest of the sequences. For full
compatibility with @ref{2f4,,map()} from Python 2.x, also wrap the sequences in
@ref{e43,,itertools.zip_longest()}, e.g. @code{map(func, *sequences)} becomes
@code{list(map(func, itertools.zip_longest(*sequences)))}.

@item 
@ref{438,,range()} now behaves like @code{xrange()} used to behave, except
it works with values of arbitrary size.  The latter no longer
exists.

@item 
@ref{247,,zip()} now returns an iterator.
@end itemize

@node Ordering Comparisons,Integers,Views And Iterators Instead Of Lists,Common Stumbling Blocks
@anchor{whatsnew/3 0 ordering-comparisons}@anchor{e44}
@subsubsection Ordering Comparisons


Python 3.0 has simplified the rules for ordering comparisons:


@itemize *

@item 
The ordering comparison operators (@code{<}, @code{<=}, @code{>=}, @code{>})
raise a TypeError exception when the operands don’t have a
meaningful natural ordering.  Thus, expressions like @code{1 < '@w{'}}, @code{0
> None} or @code{len <= len} are no longer valid, and e.g. @code{None <
None} raises @ref{162,,TypeError} instead of returning
@code{False}.  A corollary is that sorting a heterogeneous list
no longer makes sense – all the elements must be comparable to each
other.  Note that this does not apply to the @code{==} and @code{!=}
operators: objects of different incomparable types always compare
unequal to each other.

@item 
@code{builtin.sorted()} and @ref{705,,list.sort()} no longer accept the
@emph{cmp} argument providing a comparison function.  Use the @emph{key}
argument instead. N.B. the @emph{key} and @emph{reverse} arguments are now
“keyword-only”.

@item 
The @code{cmp()} function should be treated as gone, and the @code{__cmp__()}
special method is no longer supported.  Use @code{__lt__()} for sorting,
@code{__eq__()} with @code{__hash__()}, and other rich comparisons as needed.
(If you really need the @code{cmp()} functionality, you could use the
expression @code{(a > b) - (a < b)} as the equivalent for @code{cmp(a, b)}.)
@end itemize

@node Integers,Text Vs Data Instead Of Unicode Vs 8-bit,Ordering Comparisons,Common Stumbling Blocks
@anchor{whatsnew/3 0 integers}@anchor{e45}
@subsubsection Integers



@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 237
PEP 237@footnote{https://www.python.org/dev/peps/pep-0237}: Essentially, @code{long} renamed to @ref{1b1,,int}.
That is, there is only one built-in integral type, named
@ref{1b1,,int}; but it behaves mostly like the old @code{long} type.

@item 
@geindex Python Enhancement Proposals; PEP 238
PEP 238@footnote{https://www.python.org/dev/peps/pep-0238}: An expression like @code{1/2} returns a float.  Use
@code{1//2} to get the truncating behavior.  (The latter syntax has
existed for years, at least since Python 2.2.)

@item 
The @code{sys.maxint} constant was removed, since there is no
longer a limit to the value of integers.  However, @ref{d69,,sys.maxsize}
can be used as an integer larger than any practical list or string
index.  It conforms to the implementation’s “natural” integer size
and is typically the same as @code{sys.maxint} in previous releases
on the same platform (assuming the same build options).

@item 
The @ref{27d,,repr()} of a long integer doesn’t include the trailing @code{L}
anymore, so code that unconditionally strips that character will
chop off the last digit instead.  (Use @ref{2ef,,str()} instead.)

@item 
Octal literals are no longer of the form @code{0720}; use @code{0o720}
instead.
@end itemize

@node Text Vs Data Instead Of Unicode Vs 8-bit,,Integers,Common Stumbling Blocks
@anchor{whatsnew/3 0 text-vs-data-instead-of-unicode-vs-8-bit}@anchor{e46}
@subsubsection Text Vs. Data Instead Of Unicode Vs. 8-bit


Everything you thought you knew about binary data and Unicode has
changed.


@itemize *

@item 
Python 3.0 uses the concepts of @emph{text} and (binary) @emph{data} instead
of Unicode strings and 8-bit strings.  All text is Unicode; however
@emph{encoded} Unicode is represented as binary data.  The type used to
hold text is @ref{2ef,,str}, the type used to hold data is
@ref{2f0,,bytes}.  The biggest difference with the 2.x situation is
that any attempt to mix text and data in Python 3.0 raises
@ref{162,,TypeError}, whereas if you were to mix Unicode and 8-bit
strings in Python 2.x, it would work if the 8-bit string happened to
contain only 7-bit (ASCII) bytes, but you would get
@ref{51f,,UnicodeDecodeError} if it contained non-ASCII values.  This
value-specific behavior has caused numerous sad faces over the
years.

@item 
As a consequence of this change in philosophy, pretty much all code
that uses Unicode, encodings or binary data most likely has to
change.  The change is for the better, as in the 2.x world there
were numerous bugs having to do with mixing encoded and unencoded
text.  To be prepared in Python 2.x, start using @code{unicode}
for all unencoded text, and @ref{2ef,,str} for binary or encoded data
only.  Then the @code{2to3} tool will do most of the work for you.

@item 
You can no longer use @code{u"..."} literals for Unicode text.
However, you must use @code{b"..."} literals for binary data.

@item 
As the @ref{2ef,,str} and @ref{2f0,,bytes} types cannot be mixed, you
must always explicitly convert between them.  Use @ref{3ba,,str.encode()}
to go from @ref{2ef,,str} to @ref{2f0,,bytes}, and @ref{3bb,,bytes.decode()}
to go from @ref{2f0,,bytes} to @ref{2ef,,str}.  You can also use
@code{bytes(s, encoding=...)} and @code{str(b, encoding=...)},
respectively.

@item 
Like @ref{2ef,,str}, the @ref{2f0,,bytes} type is immutable.  There is a
separate @emph{mutable} type to hold buffered binary data,
@ref{2f1,,bytearray}.  Nearly all APIs that accept @ref{2f0,,bytes} also
accept @ref{2f1,,bytearray}.  The mutable API is based on
@code{collections.MutableSequence}.

@item 
All backslashes in raw string literals are interpreted literally.
This means that @code{'\U'} and @code{'\u'} escapes in raw strings are not
treated specially.  For example, @code{r'\u20ac'} is a string of 6
characters in Python 3.0, whereas in 2.6, @code{ur'\u20ac'} was the
single “euro” character.  (Of course, this change only affects raw
string literals; the euro character is @code{'\u20ac'} in Python 3.0.)

@item 
The built-in @code{basestring} abstract type was removed.  Use
@ref{2ef,,str} instead.  The @ref{2ef,,str} and @ref{2f0,,bytes} types
don’t have functionality enough in common to warrant a shared base
class.  The @code{2to3} tool (see below) replaces every occurrence of
@code{basestring} with @ref{2ef,,str}.

@item 
Files opened as text files (still the default mode for @ref{1c8,,open()})
always use an encoding to map between strings (in memory) and bytes
(on disk).  Binary files (opened with a @code{b} in the mode argument)
always use bytes in memory.  This means that if a file is opened
using an incorrect mode or encoding, I/O will likely fail loudly,
instead of silently producing incorrect data.  It also means that
even Unix users will have to specify the correct mode (text or
binary) when opening a file.  There is a platform-dependent default
encoding, which on Unixy platforms can be set with the @code{LANG}
environment variable (and sometimes also with some other
platform-specific locale-related environment variables).  In many
cases, but not all, the system default is UTF-8; you should never
count on this default.  Any application reading or writing more than
pure ASCII text should probably have a way to override the encoding.
There is no longer any need for using the encoding-aware streams
in the @ref{1a,,codecs} module.

@item 
The initial values of @ref{5ee,,sys.stdin}, @ref{5ef,,sys.stdout} and
@ref{42d,,sys.stderr} are now unicode-only text files (i.e., they are
instances of @ref{e47,,io.TextIOBase}).  To read and write bytes data
with these streams, you need to use their @ref{e48,,io.TextIOBase.buffer}
attribute.

@item 
Filenames are passed to and returned from APIs as (Unicode) strings.
This can present platform-specific problems because on some
platforms filenames are arbitrary byte strings.  (On the other hand,
on Windows filenames are natively stored as Unicode.)  As a
work-around, most APIs (e.g. @ref{1c8,,open()} and many functions in the
@ref{c0,,os} module) that take filenames accept @ref{2f0,,bytes} objects
as well as strings, and a few APIs have a way to ask for a
@ref{2f0,,bytes} return value.  Thus, @ref{c8d,,os.listdir()} returns a
list of @ref{2f0,,bytes} instances if the argument is a @ref{2f0,,bytes}
instance, and @ref{5ad,,os.getcwdb()} returns the current working
directory as a @ref{2f0,,bytes} instance.  Note that when
@ref{c8d,,os.listdir()} returns a list of strings, filenames that
cannot be decoded properly are omitted rather than raising
@ref{e49,,UnicodeError}.

@item 
Some system APIs like @ref{d60,,os.environ} and @ref{e15,,sys.argv} can
also present problems when the bytes made available by the system is
not interpretable using the default encoding.  Setting the @code{LANG}
variable and rerunning the program is probably the best approach.

@item 
@geindex Python Enhancement Proposals; PEP 3138
PEP 3138@footnote{https://www.python.org/dev/peps/pep-3138}: The @ref{27d,,repr()} of a string no longer escapes
non-ASCII characters.  It still escapes control characters and code
points with non-printable status in the Unicode standard, however.

@item 
@geindex Python Enhancement Proposals; PEP 3120
PEP 3120@footnote{https://www.python.org/dev/peps/pep-3120}: The default source encoding is now UTF-8.

@item 
@geindex Python Enhancement Proposals; PEP 3131
PEP 3131@footnote{https://www.python.org/dev/peps/pep-3131}: Non-ASCII letters are now allowed in identifiers.
(However, the standard library remains ASCII-only with the exception
of contributor names in comments.)

@item 
The @code{StringIO} and @code{cStringIO} modules are gone.  Instead,
import the @ref{9d,,io} module and use @ref{aa1,,io.StringIO} or
@ref{a15,,io.BytesIO} for text and data respectively.

@item 
See also the @ref{e4a,,Unicode HOWTO}, which was updated for Python 3.0.
@end itemize

@node Overview Of Syntax Changes,Changes Already Present In Python 2 6,Common Stumbling Blocks,What’s New In Python 3 0
@anchor{whatsnew/3 0 overview-of-syntax-changes}@anchor{e4b}
@subsection Overview Of Syntax Changes


This section gives a brief overview of every @emph{syntactic} change in
Python 3.0.

@menu
* New Syntax:: 
* Changed Syntax:: 
* Removed Syntax:: 

@end menu

@node New Syntax,Changed Syntax,,Overview Of Syntax Changes
@anchor{whatsnew/3 0 new-syntax}@anchor{e4c}
@subsubsection New Syntax



@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 3107
PEP 3107@footnote{https://www.python.org/dev/peps/pep-3107}: Function argument and return value annotations.  This
provides a standardized way of annotating a function’s parameters
and return value.  There are no semantics attached to such
annotations except that they can be introspected at runtime using
the @code{__annotations__} attribute.  The intent is to encourage
experimentation through metaclasses, decorators or frameworks.

@item 
@geindex Python Enhancement Proposals; PEP 3102
PEP 3102@footnote{https://www.python.org/dev/peps/pep-3102}: Keyword-only arguments.  Named parameters occurring
after @code{*args} in the parameter list @emph{must} be specified using
keyword syntax in the call.  You can also use a bare @code{*} in the
parameter list to indicate that you don’t accept a variable-length
argument list, but you do have keyword-only arguments.

@item 
Keyword arguments are allowed after the list of base classes in a
class definition.  This is used by the new convention for specifying
a metaclass (see next section), but can be used for other purposes
as well, as long as the metaclass supports it.

@item 
@geindex Python Enhancement Proposals; PEP 3104
PEP 3104@footnote{https://www.python.org/dev/peps/pep-3104}: @ref{e4d,,nonlocal} statement.  Using @code{nonlocal x}
you can now assign directly to a variable in an outer (but
non-global) scope.  @code{nonlocal} is a new reserved word.

@item 
@geindex Python Enhancement Proposals; PEP 3132
PEP 3132@footnote{https://www.python.org/dev/peps/pep-3132}: Extended Iterable Unpacking.  You can now write things
like @code{a, b, *rest = some_sequence}.  And even @code{*rest, a =
stuff}.  The @code{rest} object is always a (possibly empty) list; the
right-hand side may be any iterable.  Example:

@example
(a, *rest, b) = range(5)
@end example

This sets @emph{a} to @code{0}, @emph{b} to @code{4}, and @emph{rest} to @code{[1, 2, 3]}.

@item 
Dictionary comprehensions: @code{@{k: v for k, v in stuff@}} means the
same thing as @code{dict(stuff)} but is more flexible.  (This is
@geindex Python Enhancement Proposals; PEP 274
PEP 274@footnote{https://www.python.org/dev/peps/pep-0274} vindicated. :-)

@item 
Set literals, e.g. @code{@{1, 2@}}.  Note that @code{@{@}} is an empty
dictionary; use @code{set()} for an empty set.  Set comprehensions are
also supported; e.g., @code{@{x for x in stuff@}} means the same thing as
@code{set(stuff)} but is more flexible.

@item 
New octal literals, e.g. @code{0o720} (already in 2.6).  The old octal
literals (@code{0720}) are gone.

@item 
New binary literals, e.g. @code{0b1010} (already in 2.6), and
there is a new corresponding built-in function, @ref{e4e,,bin()}.

@item 
Bytes literals are introduced with a leading @code{b} or @code{B}, and
there is a new corresponding built-in function, @ref{2f0,,bytes()}.
@end itemize

@node Changed Syntax,Removed Syntax,New Syntax,Overview Of Syntax Changes
@anchor{whatsnew/3 0 changed-syntax}@anchor{e4f}
@subsubsection Changed Syntax



@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 3109
PEP 3109@footnote{https://www.python.org/dev/peps/pep-3109} and 
@geindex Python Enhancement Proposals; PEP 3134
PEP 3134@footnote{https://www.python.org/dev/peps/pep-3134}: new @ref{16d,,raise} statement syntax:
@code{raise [@emph{expr} [from @emph{expr}]]}.  See below.

@item 
@code{as} and @ref{168,,with} are now reserved words.  (Since
2.6, actually.)

@item 
@code{True}, @code{False}, and @code{None} are reserved words.  (2.6 partially enforced
the restrictions on @code{None} already.)

@item 
Change from @ref{16e,,except} @emph{exc}, @emph{var} to
@code{except} @emph{exc} @code{as} @emph{var}.  See 
@geindex Python Enhancement Proposals; PEP 3110
PEP 3110@footnote{https://www.python.org/dev/peps/pep-3110}.

@item 
@geindex Python Enhancement Proposals; PEP 3115
PEP 3115@footnote{https://www.python.org/dev/peps/pep-3115}: New Metaclass Syntax.  Instead of:

@example
class C:
    __metaclass__ = M
    ...
@end example

you must now use:

@example
class C(metaclass=M):
    ...
@end example

The module-global @code{__metaclass__} variable is no longer
supported.  (It was a crutch to make it easier to default to
new-style classes without deriving every class from
@ref{5a5,,object}.)

@item 
List comprehensions no longer support the syntactic form
@code{[... for @emph{var} in @emph{item1}, @emph{item2}, ...]}.  Use
@code{[... for @emph{var} in (@emph{item1}, @emph{item2}, ...)]} instead.
Also note that list comprehensions have different semantics: they
are closer to syntactic sugar for a generator expression inside a
@ref{43a,,list()} constructor, and in particular the loop control
variables are no longer leaked into the surrounding scope.

@item 
The @emph{ellipsis} (@code{...}) can be used as an atomic expression
anywhere.  (Previously it was only allowed in slices.)  Also, it
@emph{must} now be spelled as @code{...}.  (Previously it could also be
spelled as @code{. . .}, by a mere accident of the grammar.)
@end itemize

@node Removed Syntax,,Changed Syntax,Overview Of Syntax Changes
@anchor{whatsnew/3 0 removed-syntax}@anchor{e50}
@subsubsection Removed Syntax



@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 3113
PEP 3113@footnote{https://www.python.org/dev/peps/pep-3113}: Tuple parameter unpacking removed.  You can no longer
write @code{def foo(a, (b, c)): ...}.
Use @code{def foo(a, b_c): b, c = b_c} instead.

@item 
Removed backticks (use @ref{27d,,repr()} instead).

@item 
Removed @code{<>} (use @code{!=} instead).

@item 
Removed keyword: @ref{34b,,exec()} is no longer a keyword; it remains as
a function.  (Fortunately the function syntax was also accepted in
2.x.)  Also note that @ref{34b,,exec()} no longer takes a stream argument;
instead of @code{exec(f)} you can use @code{exec(f.read())}.

@item 
Integer literals no longer support a trailing @code{l} or @code{L}.

@item 
String literals no longer support a leading @code{u} or @code{U}.

@item 
The @ref{e51,,from} @emph{module} @ref{e35,,import} @code{*} syntax is only
allowed at the module level, no longer inside functions.

@item 
The only acceptable syntax for relative imports is @code{from .[@emph{module}]
import @emph{name}}.  All @ref{e35,,import} forms not starting with @code{.} are
interpreted as absolute imports.  (
@geindex Python Enhancement Proposals; PEP 328
PEP 328@footnote{https://www.python.org/dev/peps/pep-0328})

@item 
Classic classes are gone.
@end itemize

@node Changes Already Present In Python 2 6,Library Changes,Overview Of Syntax Changes,What’s New In Python 3 0
@anchor{whatsnew/3 0 changes-already-present-in-python-2-6}@anchor{e52}
@subsection Changes Already Present In Python 2.6


Since many users presumably make the jump straight from Python 2.5 to
Python 3.0, this section reminds the reader of new features that were
originally designed for Python 3.0 but that were back-ported to Python
2.6.  The corresponding sections in @ref{e53,,What’s New in Python 2.6} should be
consulted for longer descriptions.


@itemize *

@item 
@ref{e54,,PEP 343; The ‘with’ statement}.  The @ref{168,,with} statement is now a standard
feature and no longer needs to be imported from the @ref{0,,__future__}.
Also check out @ref{e55,,Writing Context Managers} and
@ref{e56,,The contextlib module}.

@item 
@ref{e57,,PEP 366; Explicit Relative Imports From a Main Module}.  This enhances the usefulness of the @ref{611,,-m}
option when the referenced module lives in a package.

@item 
@ref{e58,,PEP 370; Per-user site-packages Directory}.

@item 
@ref{e59,,PEP 371; The multiprocessing Package}.

@item 
@ref{e5a,,PEP 3101; Advanced String Formatting}.  Note: the 2.6 description mentions the
@ref{785,,format()} method for both 8-bit and Unicode strings.  In 3.0,
only the @ref{2ef,,str} type (text strings with Unicode support)
supports this method; the @ref{2f0,,bytes} type does not.  The plan is
to eventually make this the only API for string formatting, and to
start deprecating the @code{%} operator in Python 3.1.

@item 
@ref{e5b,,PEP 3105; print As a Function}.  This is now a standard feature and no longer needs
to be imported from @ref{0,,__future__}.  More details were given above.

@item 
@ref{e5c,,PEP 3110; Exception-Handling Changes}.  The @ref{16e,,except} @emph{exc} @code{as} @emph{var}
syntax is now standard and @code{except} @emph{exc}, @emph{var} is no
longer supported.  (Of course, the @code{as} @emph{var} part is still
optional.)

@item 
@ref{e5d,,PEP 3112; Byte Literals}.  The @code{b"..."} string literal notation (and its
variants like @code{b'...'}, @code{b"""..."""}, and @code{br"..."}) now
produces a literal of type @ref{2f0,,bytes}.

@item 
@ref{e5e,,PEP 3116; New I/O Library}.  The @ref{9d,,io} module is now the standard way of
doing file I/O.  The built-in @ref{1c8,,open()} function is now an
alias for @ref{1c9,,io.open()} and has additional keyword arguments
@emph{encoding}, @emph{errors}, @emph{newline} and @emph{closefd}.  Also note that an
invalid @emph{mode} argument now raises @ref{17c,,ValueError}, not
@ref{be8,,IOError}.  The binary file object underlying a text file
object can be accessed as @code{f.buffer} (but beware that the
text object maintains a buffer of itself in order to speed up
the encoding and decoding operations).

@item 
@ref{e5f,,PEP 3118; Revised Buffer Protocol}.  The old builtin @code{buffer()} is now really gone;
the new builtin @ref{573,,memoryview()} provides (mostly) similar
functionality.

@item 
@ref{e60,,PEP 3119; Abstract Base Classes}.  The @ref{3,,abc} module and the ABCs defined in the
@ref{1c,,collections} module plays a somewhat more prominent role in
the language now, and built-in collection types like @ref{3b0,,dict}
and @ref{43a,,list} conform to the @code{collections.MutableMapping}
and @code{collections.MutableSequence} ABCs, respectively.

@item 
@ref{e61,,PEP 3127; Integer Literal Support and Syntax}.  As mentioned above, the new octal literal
notation is the only one supported, and binary literals have been
added.

@item 
@ref{e62,,PEP 3129; Class Decorators}.

@item 
@ref{e63,,PEP 3141; A Type Hierarchy for Numbers}.  The @ref{bd,,numbers} module is another new use of
ABCs, defining Python’s “numeric tower”.  Also note the new
@ref{7d,,fractions} module which implements @ref{e64,,numbers.Rational}.
@end itemize

@node Library Changes,PEP 3101 A New Approach To String Formatting,Changes Already Present In Python 2 6,What’s New In Python 3 0
@anchor{whatsnew/3 0 library-changes}@anchor{e65}
@subsection Library Changes


Due to time constraints, this document does not exhaustively cover the
very extensive changes to the standard library.  
@geindex Python Enhancement Proposals; PEP 3108
PEP 3108@footnote{https://www.python.org/dev/peps/pep-3108} is the
reference for the major changes to the library.  Here’s a capsule
review:


@itemize *

@item 
Many old modules were removed.  Some, like @code{gopherlib} (no
longer used) and @code{md5} (replaced by @ref{88,,hashlib}), were
already deprecated by 
@geindex Python Enhancement Proposals; PEP 4
PEP 4@footnote{https://www.python.org/dev/peps/pep-0004}.  Others were removed as a result
of the removal of support for various platforms such as Irix, BeOS
and Mac OS 9 (see 
@geindex Python Enhancement Proposals; PEP 11
PEP 11@footnote{https://www.python.org/dev/peps/pep-0011}).  Some modules were also selected for
removal in Python 3.0 due to lack of use or because a better
replacement exists.  See 
@geindex Python Enhancement Proposals; PEP 3108
PEP 3108@footnote{https://www.python.org/dev/peps/pep-3108} for an exhaustive list.

@item 
The @code{bsddb3} package was removed because its presence in the
core standard library has proved over time to be a particular burden
for the core developers due to testing instability and Berkeley DB’s
release schedule.  However, the package is alive and well,
externally maintained at @indicateurl{https://www.jcea.es/programacion/pybsddb.htm}.

@item 
Some modules were renamed because their old name disobeyed
@geindex Python Enhancement Proposals; PEP 8
PEP 8@footnote{https://www.python.org/dev/peps/pep-0008}, or for various other reasons.  Here’s the list:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Old Name

@tab

New Name

@item

_winreg

@tab

winreg

@item

ConfigParser

@tab

configparser

@item

copy_reg

@tab

copyreg

@item

Queue

@tab

queue

@item

SocketServer

@tab

socketserver

@item

markupbase

@tab

_markupbase

@item

repr

@tab

reprlib

@item

test.test_support

@tab

test.support

@end multitable


@item 
A common pattern in Python 2.x is to have one version of a module
implemented in pure Python, with an optional accelerated version
implemented as a C extension; for example, @ref{c5,,pickle} and
@code{cPickle}.  This places the burden of importing the accelerated
version and falling back on the pure Python version on each user of
these modules.  In Python 3.0, the accelerated versions are
considered implementation details of the pure Python versions.
Users should always import the standard version, which attempts to
import the accelerated version and falls back to the pure Python
version.  The @ref{c5,,pickle} / @code{cPickle} pair received this
treatment.  The @ref{ce,,profile} module is on the list for 3.1.  The
@code{StringIO} module has been turned into a class in the @ref{9d,,io}
module.

@item 
Some related modules have been grouped into packages, and usually
the submodule names have been simplified.  The resulting new
packages are:


@itemize *

@item 
@ref{30,,dbm} (@code{anydbm}, @code{dbhash}, @ref{30,,dbm},
@code{dumbdbm}, @code{gdbm}, @code{whichdb}).

@item 
@ref{8b,,html} (@code{HTMLParser}, @code{htmlentitydefs}).

@item 
@ref{8e,,http} (@code{httplib}, @code{BaseHTTPServer},
@code{CGIHTTPServer}, @code{SimpleHTTPServer}, @code{Cookie},
@code{cookielib}).

@item 
@ref{10c,,tkinter} (all @code{Tkinter}-related modules except
@ref{11d,,turtle}).  The target audience of @ref{11d,,turtle} doesn’t
really care about @ref{10c,,tkinter}.  Also note that as of Python
2.6, the functionality of @ref{11d,,turtle} has been greatly enhanced.

@item 
@ref{124,,urllib} (@ref{124,,urllib}, @code{urllib2}, @code{urlparse},
@code{robotparse}).

@item 
@code{xmlrpc} (@code{xmlrpclib}, @code{DocXMLRPCServer},
@code{SimpleXMLRPCServer}).
@end itemize
@end itemize

Some other changes to standard library modules, not covered by
@geindex Python Enhancement Proposals; PEP 3108
PEP 3108@footnote{https://www.python.org/dev/peps/pep-3108}:


@itemize *

@item 
Killed @code{sets}.  Use the built-in @ref{370,,set()} class.

@item 
Cleanup of the @ref{f7,,sys} module: removed @code{sys.exitfunc()},
@code{sys.exc_clear()}, @code{sys.exc_type}, @code{sys.exc_value},
@code{sys.exc_traceback}.  (Note that @ref{e66,,sys.last_type}
etc. remain.)

@item 
Cleanup of the @ref{25e,,array.array} type: the @code{read()} and
@code{write()} methods are gone; use @code{fromfile()} and
@code{tofile()} instead.  Also, the @code{'c'} typecode for array is
gone – use either @code{'b'} for bytes or @code{'u'} for Unicode
characters.

@item 
Cleanup of the @ref{be,,operator} module: removed
@code{sequenceIncludes()} and @code{isCallable()}.

@item 
Cleanup of the @code{thread} module: @code{acquire_lock()} and
@code{release_lock()} are gone; use @code{acquire()} and
@code{release()} instead.

@item 
Cleanup of the @ref{d7,,random} module: removed the @code{jumpahead()} API.

@item 
The @code{new} module is gone.

@item 
The functions @code{os.tmpnam()}, @code{os.tempnam()} and
@code{os.tmpfile()} have been removed in favor of the @ref{fd,,tempfile}
module.

@item 
The @ref{118,,tokenize} module has been changed to work with bytes.  The
main entry point is now @ref{e67,,tokenize.tokenize()}, instead of
generate_tokens.

@item 
@code{string.letters} and its friends (@code{string.lowercase} and
@code{string.uppercase}) are gone.  Use
@ref{e68,,string.ascii_letters} etc. instead.  (The reason for the
removal is that @code{string.letters} and friends had
locale-specific behavior, which is a bad idea for such
attractively-named global “constants”.)

@item 
Renamed module @code{__builtin__} to @ref{11,,builtins} (removing the
underscores, adding an ‘s’).  The @code{__builtins__} variable
found in most global namespaces is unchanged.  To modify a builtin,
you should use @ref{11,,builtins}, not @code{__builtins__}!
@end itemize

@node PEP 3101 A New Approach To String Formatting,Changes To Exceptions,Library Changes,What’s New In Python 3 0
@anchor{whatsnew/3 0 pep-3101-a-new-approach-to-string-formatting}@anchor{e69}
@subsection 
@geindex Python Enhancement Proposals; PEP 3101
@strong{PEP 3101}: A New Approach To String Formatting



@itemize *

@item 
A new system for  built-in string formatting operations replaces the
@code{%} string  formatting operator.   (However, the @code{%}  operator is
still supported;  it will  be deprecated in  Python 3.1  and removed
from the language at some later time.)  Read 
@geindex Python Enhancement Proposals; PEP 3101
PEP 3101@footnote{https://www.python.org/dev/peps/pep-3101} for the full
scoop.
@end itemize

@node Changes To Exceptions,Miscellaneous Other Changes,PEP 3101 A New Approach To String Formatting,What’s New In Python 3 0
@anchor{whatsnew/3 0 changes-to-exceptions}@anchor{e6a}
@subsection Changes To Exceptions


The APIs for raising and catching exception have been cleaned up and
new powerful features added:


@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 352
PEP 352@footnote{https://www.python.org/dev/peps/pep-0352}: All exceptions must be derived (directly or indirectly)
from @ref{15f,,BaseException}.  This is the root of the exception
hierarchy.  This is not new as a recommendation, but the
@emph{requirement} to inherit from @ref{15f,,BaseException} is new.  (Python
2.6 still allowed classic classes to be raised, and placed no
restriction on what you can catch.)  As a consequence, string
exceptions are finally truly and utterly dead.

@item 
Almost all exceptions should actually derive from @ref{4d8,,Exception};
@ref{15f,,BaseException} should only be used as a base class for
exceptions that should only be handled at the top level, such as
@ref{892,,SystemExit} or @ref{4ca,,KeyboardInterrupt}.  The recommended
idiom for handling all exceptions except for this latter category is
to use @ref{16e,,except} @ref{4d8,,Exception}.

@item 
@code{StandardError} was removed.

@item 
Exceptions no longer behave as sequences.  Use the @code{args}
attribute instead.

@item 
@geindex Python Enhancement Proposals; PEP 3109
PEP 3109@footnote{https://www.python.org/dev/peps/pep-3109}: Raising exceptions.  You must now use @code{raise
@emph{Exception}(@emph{args})} instead of @code{raise @emph{Exception}, @emph{args}}.
Additionally, you can no longer explicitly specify a traceback;
instead, if you @emph{have} to do this, you can assign directly to the
@code{__traceback__} attribute (see below).

@item 
@geindex Python Enhancement Proposals; PEP 3110
PEP 3110@footnote{https://www.python.org/dev/peps/pep-3110}: Catching exceptions.  You must now use
@code{except @emph{SomeException} as @emph{variable}} instead
of @code{except @emph{SomeException}, @emph{variable}}.  Moreover, the
@emph{variable} is explicitly deleted when the @ref{16e,,except} block
is left.

@item 
@geindex Python Enhancement Proposals; PEP 3134
PEP 3134@footnote{https://www.python.org/dev/peps/pep-3134}: Exception chaining.  There are two cases: implicit
chaining and explicit chaining.  Implicit chaining happens when an
exception is raised in an @ref{16e,,except} or @ref{4c1,,finally}
handler block.  This usually happens due to a bug in the handler
block; we call this a @emph{secondary} exception.  In this case, the
original exception (that was being handled) is saved as the
@code{__context__} attribute of the secondary exception.
Explicit chaining is invoked with this syntax:

@example
raise SecondaryException() from primary_exception
@end example

(where @emph{primary_exception} is any expression that produces an
exception object, probably an exception that was previously caught).
In this case, the primary exception is stored on the
@code{__cause__} attribute of the secondary exception.  The
traceback printed when an unhandled exception occurs walks the chain
of @code{__cause__} and @code{__context__} attributes and prints a
separate traceback for each component of the chain, with the primary
exception at the top.  (Java users may recognize this behavior.)

@item 
@geindex Python Enhancement Proposals; PEP 3134
PEP 3134@footnote{https://www.python.org/dev/peps/pep-3134}: Exception objects now store their traceback as the
@code{__traceback__} attribute.  This means that an exception
object now contains all the information pertaining to an exception,
and there are fewer reasons to use @ref{18f,,sys.exc_info()} (though the
latter is not removed).

@item 
A few exception messages are improved when Windows fails to load an
extension module.  For example, @code{error code 193} is now @code{%1 is
not a valid Win32 application}.  Strings now deal with non-English
locales.
@end itemize

@node Miscellaneous Other Changes,Build and C API Changes<7>,Changes To Exceptions,What’s New In Python 3 0
@anchor{whatsnew/3 0 miscellaneous-other-changes}@anchor{e6b}
@subsection Miscellaneous Other Changes


@menu
* Operators And Special Methods:: 
* Builtins:: 

@end menu

@node Operators And Special Methods,Builtins,,Miscellaneous Other Changes
@anchor{whatsnew/3 0 operators-and-special-methods}@anchor{e6c}
@subsubsection Operators And Special Methods



@itemize *

@item 
@code{!=} now returns the opposite of @code{==}, unless @code{==} returns
@ref{24c,,NotImplemented}.

@item 
The concept of “unbound methods” has been removed from the language.
When referencing a method as a class attribute, you now get a plain
function object.

@item 
@code{__getslice__()}, @code{__setslice__()} and @code{__delslice__()}
were killed.  The syntax @code{a[i:j]} now translates to
@code{a.__getitem__(slice(i, j))} (or @code{__setitem__()} or
@code{__delitem__()}, when used as an assignment or deletion target,
respectively).

@item 
@geindex Python Enhancement Proposals; PEP 3114
PEP 3114@footnote{https://www.python.org/dev/peps/pep-3114}: the standard @ref{252,,next()} method has been renamed to
@ref{e6d,,__next__()}.

@item 
The @code{__oct__()} and @code{__hex__()} special methods are removed
– @ref{e6e,,oct()} and @ref{e6f,,hex()} use @code{__index__()} now to convert
the argument to an integer.

@item 
Removed support for @code{__members__} and @code{__methods__}.

@item 
The function attributes named @code{func_X} have been renamed to
use the @code{__X__} form, freeing up these names in the function
attribute namespace for user-defined attributes.  To wit,
@code{func_closure}, @code{func_code}, @code{func_defaults},
@code{func_dict}, @code{func_doc}, @code{func_globals},
@code{func_name} were renamed to @code{__closure__},
@code{__code__}, @code{__defaults__}, @ref{7a4,,__dict__},
@code{__doc__}, @code{__globals__}, @ref{e70,,__name__},
respectively.

@item 
@code{__nonzero__()} is now @code{__bool__()}.
@end itemize

@node Builtins,,Operators And Special Methods,Miscellaneous Other Changes
@anchor{whatsnew/3 0 builtins}@anchor{e71}
@subsubsection Builtins



@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 3135
PEP 3135@footnote{https://www.python.org/dev/peps/pep-3135}: New @ref{78c,,super()}.  You can now invoke @ref{78c,,super()}
without arguments and (assuming this is in a regular instance method
defined inside a @ref{e72,,class} statement) the right class and
instance will automatically be chosen.  With arguments, the behavior
of @ref{78c,,super()} is unchanged.

@item 
@geindex Python Enhancement Proposals; PEP 3111
PEP 3111@footnote{https://www.python.org/dev/peps/pep-3111}: @code{raw_input()} was renamed to @ref{e73,,input()}.  That
is, the new @ref{e73,,input()} function reads a line from
@ref{5ee,,sys.stdin} and returns it with the trailing newline stripped.
It raises @ref{e74,,EOFError} if the input is terminated prematurely.
To get the old behavior of @ref{e73,,input()}, use @code{eval(input())}.

@item 
A new built-in function @ref{252,,next()} was added to call the
@ref{e6d,,__next__()} method on an object.

@item 
The @ref{e75,,round()} function rounding strategy and return type have
changed.  Exact halfway cases are now rounded to the nearest even
result instead of away from zero.  (For example, @code{round(2.5)} now
returns @code{2} rather than @code{3}.)  @code{round(x[, n])} now
delegates to @code{x.__round__([n])} instead of always returning a
float.  It generally returns an integer when called with a single
argument and a value of the same type as @code{x} when called with two
arguments.

@item 
Moved @code{intern()} to @ref{e76,,sys.intern()}.

@item 
Removed: @code{apply()}.  Instead of @code{apply(f, args)} use
@code{f(*args)}.

@item 
Removed @ref{d6a,,callable()}.  Instead of @code{callable(f)} you can use
@code{isinstance(f, collections.Callable)}.  The @code{operator.isCallable()}
function is also gone.

@item 
Removed @code{coerce()}.  This function no longer serves a purpose
now that classic classes are gone.

@item 
Removed @code{execfile()}.  Instead of @code{execfile(fn)} use
@code{exec(open(fn).read())}.

@item 
Removed the @code{file} type.  Use @ref{1c8,,open()}.  There are now several
different kinds of streams that open can return in the @ref{9d,,io} module.

@item 
Removed @code{reduce()}.  Use @ref{e77,,functools.reduce()} if you really
need it; however, 99 percent of the time an explicit @ref{161,,for}
loop is more readable.

@item 
Removed @code{reload()}.  Use @ref{e78,,imp.reload()}.

@item 
Removed. @code{dict.has_key()} – use the @ref{2fc,,in} operator
instead.
@end itemize

@c ======================================================================

@node Build and C API Changes<7>,Performance,Miscellaneous Other Changes,What’s New In Python 3 0
@anchor{whatsnew/3 0 build-and-c-api-changes}@anchor{e79}
@subsection Build and C API Changes


Due to time constraints, here is a @emph{very} incomplete list of changes
to the C API.


@itemize *

@item 
Support for several platforms was dropped, including but not limited
to Mac OS 9, BeOS, RISCOS, Irix, and Tru64.

@item 
@geindex Python Enhancement Proposals; PEP 3118
PEP 3118@footnote{https://www.python.org/dev/peps/pep-3118}: New Buffer API.

@item 
@geindex Python Enhancement Proposals; PEP 3121
PEP 3121@footnote{https://www.python.org/dev/peps/pep-3121}: Extension Module Initialization & Finalization.

@item 
@geindex Python Enhancement Proposals; PEP 3123
PEP 3123@footnote{https://www.python.org/dev/peps/pep-3123}: Making @ref{e7a,,PyObject_HEAD} conform to standard C.

@item 
No more C API support for restricted execution.

@item 
@code{PyNumber_Coerce()}, @code{PyNumber_CoerceEx()},
@code{PyMember_Get()}, and @code{PyMember_Set()} C APIs are removed.

@item 
New C API @ref{c11,,PyImport_ImportModuleNoBlock()}, works like
@ref{34f,,PyImport_ImportModule()} but won’t block on the import lock
(returning an error instead).

@item 
Renamed the boolean conversion C-level slot and method:
@code{nb_nonzero} is now @code{nb_bool}.

@item 
Removed @code{METH_OLDARGS} and @code{WITH_CYCLE_GC} from the C API.
@end itemize

@c ======================================================================

@node Performance,Porting To Python 3 0,Build and C API Changes<7>,What’s New In Python 3 0
@anchor{whatsnew/3 0 performance}@anchor{e7b}
@subsection Performance


The net result of the 3.0 generalizations is that Python 3.0 runs the
pystone benchmark around 10% slower than Python 2.5.  Most likely the
biggest cause is the removal of special-casing for small integers.
There’s room for improvement, but it will happen after 3.0 is
released!

@c ======================================================================

@node Porting To Python 3 0,,Performance,What’s New In Python 3 0
@anchor{whatsnew/3 0 porting-to-python-3-0}@anchor{e7c}
@subsection Porting To Python 3.0


For porting existing Python 2.5 or 2.6 source code to Python 3.0, the
best strategy is the following:


@enumerate 0

@item 
(Prerequisite:) Start with excellent test coverage.

@item 
Port to Python 2.6.  This should be no more work than the average
port from Python 2.x to Python 2.(x+1).  Make sure all your tests
pass.

@item 
(Still using 2.6:) Turn on the @code{-3} command line switch.
This enables warnings about features that will be removed (or
change) in 3.0.  Run your test suite again, and fix code that you
get warnings about until there are no warnings left, and all your
tests still pass.

@item 
Run the @code{2to3} source-to-source translator over your source code
tree.  (See @ref{e7d,,2to3 - Automated Python 2 to 3 code translation} for more on this tool.)  Run the
result of the translation under Python 3.0.  Manually fix up any
remaining issues, fixing problems until all tests pass again.
@end enumerate

It is not recommended to try to write source code that runs unchanged
under both Python 2.6 and 3.0; you’d have to use a very contorted
coding style, e.g. avoiding @code{print} statements, metaclasses,
and much more.  If you are maintaining a library that needs to support
both Python 2.6 and Python 3.0, the best approach is to modify step 3
above by editing the 2.6 version of the source code and running the
@code{2to3} translator again, rather than editing the 3.0 version of the
source code.

For porting C extensions to Python 3.0, please see @ref{e7e,,Porting Extension Modules to Python 3}.

@c ======================================================================

@node What’s New in Python 2 7,What’s New in Python 2 6,What’s New In Python 3 0,What’s New in Python
@anchor{whatsnew/2 7 doc}@anchor{e7f}@anchor{whatsnew/2 7 what-s-new-in-python-2-7}@anchor{e80}
@section What’s New in Python 2.7


@*Author: 
A.M. Kuchling (amk at amk.ca)


@c hyperlink all the methods & functions.

@c T_STRING_INPLACE not described in main docs

@c $Id$
@c Rules for maintenance:
@c 
@c * Anyone can add text to this document.  Do not spend very much time
@c on the wording of your changes, because your text will probably
@c get rewritten to some degree.
@c 
@c * The maintainer will go through Misc/NEWS periodically and add
@c changes; it's therefore more important to add your changes to
@c Misc/NEWS than to this file.
@c 
@c * This is not a complete list of every single change; completeness
@c is the purpose of Misc/NEWS.  Some changes I consider too small
@c or esoteric to include.  If such a change is added to the text,
@c I'll just remove it.  (This is another reason you shouldn't spend
@c too much time on writing your addition.)
@c 
@c * If you want to draw your new text to the attention of the
@c maintainer, add 'XXX' to the beginning of the paragraph or
@c section.
@c 
@c * It's OK to just add a fragmentary note about a change.  For
@c example: "XXX Describe the transmogrify() function added to the
@c socket module."  The maintainer will research the change and
@c write the necessary text.
@c 
@c * You can comment out your additions if you like, but it's not
@c necessary (especially when a final release is some months away).
@c 
@c * Credit the author of a patch or bugfix.  Just the name is
@c sufficient; the e-mail address isn't necessary.
@c 
@c * It's helpful to add the bug/patch number in a parenthetical comment.
@c 
@c XXX Describe the transmogrify() function added to the socket
@c module.
@c (Contributed by P.Y. Developer; :issue:`12345`.)
@c 
@c This saves the maintainer some effort going through the SVN logs
@c when researching a change.

This article explains the new features in Python 2.7.  Python 2.7 was released
on July 3, 2010.

Numeric handling has been improved in many ways, for both
floating-point numbers and for the @ref{248,,Decimal} class.
There are some useful additions to the standard library, such as a
greatly enhanced @ref{122,,unittest} module, the @ref{5,,argparse} module
for parsing command-line options, convenient @ref{4e5,,OrderedDict}
and @ref{c2a,,Counter} classes in the @ref{1c,,collections} module,
and many other improvements.

Python 2.7 is planned to be the last of the 2.x releases, so we worked
on making it a good release for the long term.  To help with porting
to Python 3, several new features from the Python 3.x series have been
included in 2.7.

This article doesn’t attempt to provide a complete specification of
the new features, but instead provides a convenient overview.  For
full details, you should refer to the documentation for Python 2.7 at
@indicateurl{https://docs.python.org}. If you want to understand the rationale for
the design and implementation, refer to the PEP for a particular new
feature or the issue on @indicateurl{https://bugs.python.org} in which a change was
discussed.  Whenever possible, “What’s New in Python” links to the
bug/patch item for each change.

@menu
* The Future for Python 2.x: The Future for Python 2 x. 
* Changes to the Handling of Deprecation Warnings:: 
* Python 3.1 Features: Python 3 1 Features. 
* PEP 372; Adding an Ordered Dictionary to collections: PEP 372 Adding an Ordered Dictionary to collections. 
* PEP 378; Format Specifier for Thousands Separator: PEP 378 Format Specifier for Thousands Separator<2>. 
* PEP 389; The argparse Module for Parsing Command Lines: PEP 389 The argparse Module for Parsing Command Lines. 
* PEP 391; Dictionary-Based Configuration For Logging: PEP 391 Dictionary-Based Configuration For Logging. 
* PEP 3106; Dictionary Views: PEP 3106 Dictionary Views. 
* PEP 3137; The memoryview Object: PEP 3137 The memoryview Object. 
* Other Language Changes: Other Language Changes<12>. 
* New and Improved Modules:: 
* Build and C API Changes: Build and C API Changes<8>. 
* Other Changes and Fixes:: 
* Porting to Python 2.7: Porting to Python 2 7. 
* New Features Added to Python 2.7 Maintenance Releases: New Features Added to Python 2 7 Maintenance Releases. 
* Acknowledgements:: 

@end menu

@node The Future for Python 2 x,Changes to the Handling of Deprecation Warnings,,What’s New in Python 2 7
@anchor{whatsnew/2 7 the-future-for-python-2-x}@anchor{e81}@anchor{whatsnew/2 7 whatsnew27-python31}@anchor{e82}
@subsection The Future for Python 2.x


Python 2.7 is the last major release in the 2.x series, as the Python
maintainers have shifted the focus of their new feature development efforts
to the Python 3.x series. This means that while Python 2 continues to
receive bug fixes, and to be updated to build correctly on new hardware and
versions of supported operated systems, there will be no new full feature
releases for the language or standard library.

However, while there is a large common subset between Python 2.7 and Python
3, and many of the changes involved in migrating to that common subset, or
directly to Python 3, can be safely automated, some other changes (notably
those associated with Unicode handling) may require careful consideration,
and preferably robust automated regression test suites, to migrate
effectively.

This means that Python 2.7 will remain in place for a long time, providing a
stable and supported base platform for production systems that have not yet
been ported to Python 3. The full expected lifecycle of the Python 2.7
series is detailed in 
@geindex Python Enhancement Proposals; PEP 373
PEP 373@footnote{https://www.python.org/dev/peps/pep-0373}.

Some key consequences of the long-term significance of 2.7 are:


@itemize *

@item 
As noted above, the 2.7 release has a much longer period of maintenance
when compared to earlier 2.x versions. Python 2.7 is currently expected to
remain supported by the core development team (receiving security updates
and other bug fixes) until at least 2020 (10 years after its initial
release, compared to the more typical support period of 18–24 months).

@item 
As the Python 2.7 standard library ages, making effective use of the
Python Package Index (either directly or via a redistributor) becomes
more important for Python 2 users. In addition to a wide variety of third
party packages for various tasks, the available packages include backports
of new modules and features from the Python 3 standard library that are
compatible with Python 2, as well as various tools and libraries that can
make it easier to migrate to Python 3. The Python Packaging User Guide@footnote{https://packaging.python.org} provides guidance on downloading and
installing software from the Python Package Index.

@item 
While the preferred approach to enhancing Python 2 is now the publication
of new packages on the Python Package Index, this approach doesn’t
necessarily work in all cases, especially those related to network
security. In exceptional cases that cannot be handled adequately by
publishing new or updated packages on PyPI, the Python Enhancement
Proposal process may be used to make the case for adding new features
directly to the Python 2 standard library. Any such additions, and the
maintenance releases where they were added, will be noted in the
@ref{e83,,New Features Added to Python 2.7 Maintenance Releases} section below.
@end itemize

For projects wishing to migrate from Python 2 to Python 3, or for library
and framework developers wishing to support users on both Python 2 and
Python 3, there are a variety of tools and guides available to help decide
on a suitable approach and manage some of the technical details involved.
The recommended starting point is the @ref{e84,,Porting Python 2 Code to Python 3} HOWTO guide.

@node Changes to the Handling of Deprecation Warnings,Python 3 1 Features,The Future for Python 2 x,What’s New in Python 2 7
@anchor{whatsnew/2 7 changes-to-the-handling-of-deprecation-warnings}@anchor{e85}
@subsection Changes to the Handling of Deprecation Warnings


For Python 2.7, a policy decision was made to silence warnings only of
interest to developers by default.  @ref{1b2,,DeprecationWarning} and its
descendants are now ignored unless otherwise requested, preventing
users from seeing warnings triggered by an application.  This change
was also made in the branch that became Python 3.2. (Discussed
on stdlib-sig and carried out in bpo-7319@footnote{https://bugs.python.org/issue7319}.)

In previous releases, @ref{1b2,,DeprecationWarning} messages were
enabled by default, providing Python developers with a clear
indication of where their code may break in a future major version
of Python.

However, there are increasingly many users of Python-based
applications who are not directly involved in the development of
those applications.  @ref{1b2,,DeprecationWarning} messages are
irrelevant to such users, making them worry about an application
that’s actually working correctly and burdening application developers
with responding to these concerns.

You can re-enable display of @ref{1b2,,DeprecationWarning} messages by
running Python with the @ref{3a8,,-Wdefault} (short form:
@ref{3a8,,-Wd}) switch, or by setting the 
@geindex PYTHONWARNINGS
@geindex environment variable; PYTHONWARNINGS
@ref{6de,,PYTHONWARNINGS}
environment variable to @code{"default"} (or @code{"d"}) before running
Python.  Python code can also re-enable them
by calling @code{warnings.simplefilter('default')}.

The @code{unittest} module also automatically reenables deprecation warnings
when running tests.

@node Python 3 1 Features,PEP 372 Adding an Ordered Dictionary to collections,Changes to the Handling of Deprecation Warnings,What’s New in Python 2 7
@anchor{whatsnew/2 7 python-3-1-features}@anchor{e86}
@subsection Python 3.1 Features


Much as Python 2.6 incorporated features from Python 3.0,
version 2.7 incorporates some of the new features
in Python 3.1.  The 2.x series continues to provide tools
for migrating to the 3.x series.

A partial list of 3.1 features that were backported to 2.7:


@itemize *

@item 
The syntax for set literals (@code{@{1,2,3@}} is a mutable set).

@item 
Dictionary and set comprehensions (@code{@{i: i*2 for i in range(3)@}}).

@item 
Multiple context managers in a single @ref{168,,with} statement.

@item 
A new version of the @ref{9d,,io} library, rewritten in C for performance.

@item 
The ordered-dictionary type described in @ref{e87,,PEP 372; Adding an Ordered Dictionary to collections}.

@item 
The new @code{","} format specifier described in @ref{e88,,PEP 378; Format Specifier for Thousands Separator}.

@item 
The @ref{573,,memoryview} object.

@item 
A small subset of the @ref{96,,importlib} module,
@ref{e89,,described below}.

@item 
The @ref{27d,,repr()} of a float @code{x} is shorter in many cases: it’s now
based on the shortest decimal string that’s guaranteed to round back
to @code{x}.  As in previous versions of Python, it’s guaranteed that
@code{float(repr(x))} recovers @code{x}.

@item 
Float-to-string and string-to-float conversions are correctly rounded.
The @ref{e75,,round()} function is also now correctly rounded.

@item 
The @ref{e21,,PyCapsule} type, used to provide a C API for extension modules.

@item 
The @ref{e17,,PyLong_AsLongAndOverflow()} C API function.
@end itemize

Other new Python3-mode warnings include:


@itemize *

@item 
@code{operator.isCallable()} and @code{operator.sequenceIncludes()},
which are not supported in 3.x, now trigger warnings.

@item 
The @code{-3} switch now automatically
enables the @code{-Qwarn} switch that causes warnings
about using classic division with integers and long integers.
@end itemize

@c ========================================================================

@c Large, PEP-level features and changes should be described here.

@c ========================================================================

@node PEP 372 Adding an Ordered Dictionary to collections,PEP 378 Format Specifier for Thousands Separator<2>,Python 3 1 Features,What’s New in Python 2 7
@anchor{whatsnew/2 7 pep-0372}@anchor{e87}@anchor{whatsnew/2 7 pep-372-adding-an-ordered-dictionary-to-collections}@anchor{e8a}
@subsection PEP 372: Adding an Ordered Dictionary to collections


Regular Python dictionaries iterate over key/value pairs in arbitrary order.
Over the years, a number of authors have written alternative implementations
that remember the order that the keys were originally inserted.  Based on
the experiences from those implementations, 2.7 introduces a new
@ref{4e5,,OrderedDict} class in the @ref{1c,,collections} module.

The @ref{4e5,,OrderedDict} API provides the same interface as regular
dictionaries but iterates over keys and values in a guaranteed order
depending on when a key was first inserted:

@example
>>> from collections import OrderedDict
>>> d = OrderedDict([('first', 1),
...                  ('second', 2),
...                  ('third', 3)])
>>> d.items()
[('first', 1), ('second', 2), ('third', 3)]
@end example

If a new entry overwrites an existing entry, the original insertion
position is left unchanged:

@example
>>> d['second'] = 4
>>> d.items()
[('first', 1), ('second', 4), ('third', 3)]
@end example

Deleting an entry and reinserting it will move it to the end:

@example
>>> del d['second']
>>> d['second'] = 5
>>> d.items()
[('first', 1), ('third', 3), ('second', 5)]
@end example

The @ref{e8b,,popitem()} method has an optional @emph{last}
argument that defaults to @code{True}.  If @emph{last} is true, the most recently
added key is returned and removed; if it’s false, the
oldest key is selected:

@example
>>> od = OrderedDict([(x,0) for x in range(20)])
>>> od.popitem()
(19, 0)
>>> od.popitem()
(18, 0)
>>> od.popitem(last=False)
(0, 0)
>>> od.popitem(last=False)
(1, 0)
@end example

Comparing two ordered dictionaries checks both the keys and values,
and requires that the insertion order was the same:

@example
>>> od1 = OrderedDict([('first', 1),
...                    ('second', 2),
...                    ('third', 3)])
>>> od2 = OrderedDict([('third', 3),
...                    ('first', 1),
...                    ('second', 2)])
>>> od1 == od2
False
>>> # Move 'third' key to the end
>>> del od2['third']; od2['third'] = 3
>>> od1 == od2
True
@end example

Comparing an @ref{4e5,,OrderedDict} with a regular dictionary
ignores the insertion order and just compares the keys and values.

How does the @ref{4e5,,OrderedDict} work?  It maintains a
doubly-linked list of keys, appending new keys to the list as they’re inserted.
A secondary dictionary maps keys to their corresponding list node, so
deletion doesn’t have to traverse the entire linked list and therefore
remains O(1).

The standard library now supports use of ordered dictionaries in several
modules.


@itemize *

@item 
The @code{ConfigParser} module uses them by default, meaning that
configuration files can now be read, modified, and then written back
in their original order.

@item 
The @ref{4e4,,_asdict()} method for
@ref{3e4,,collections.namedtuple()} now returns an ordered dictionary with the
values appearing in the same order as the underlying tuple indices.

@item 
The @ref{a0,,json} module’s @ref{89d,,JSONDecoder} class
constructor was extended with an @emph{object_pairs_hook} parameter to
allow @code{OrderedDict} instances to be built by the decoder.
Support was also added for third-party tools like
PyYAML@footnote{http://pyyaml.org/}.
@end itemize


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 372

@item PEP 372@footnote{https://www.python.org/dev/peps/pep-0372} - Adding an ordered dictionary to collections

PEP written by Armin Ronacher and Raymond Hettinger;
implemented by Raymond Hettinger.
@end table


@node PEP 378 Format Specifier for Thousands Separator<2>,PEP 389 The argparse Module for Parsing Command Lines,PEP 372 Adding an Ordered Dictionary to collections,What’s New in Python 2 7
@anchor{whatsnew/2 7 pep-0378}@anchor{e88}@anchor{whatsnew/2 7 pep-378-format-specifier-for-thousands-separator}@anchor{e8c}
@subsection PEP 378: Format Specifier for Thousands Separator


To make program output more readable, it can be useful to add
separators to large numbers, rendering them as
18,446,744,073,709,551,616 instead of 18446744073709551616.

The fully general solution for doing this is the @ref{a5,,locale} module,
which can use different separators (“,” in North America, “.” in
Europe) and different grouping sizes, but @ref{a5,,locale} is complicated
to use and unsuitable for multi-threaded applications where different
threads are producing output for different locales.

Therefore, a simple comma-grouping mechanism has been added to the
mini-language used by the @ref{784,,str.format()} method.  When
formatting a floating-point number, simply include a comma between the
width and the precision:

@example
>>> '@{:20,.2f@}'.format(18446744073709551616.0)
'18,446,744,073,709,551,616.00'
@end example

When formatting an integer, include the comma after the width:

@example
>>> '@{:20,d@}'.format(18446744073709551616)
'18,446,744,073,709,551,616'
@end example

This mechanism is not adaptable at all; commas are always used as the
separator and the grouping is always into three-digit groups.  The
comma-formatting mechanism isn’t as general as the @ref{a5,,locale}
module, but it’s easier to use.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 378

@item PEP 378@footnote{https://www.python.org/dev/peps/pep-0378} - Format Specifier for Thousands Separator

PEP written by Raymond Hettinger; implemented by Eric Smith.
@end table


@node PEP 389 The argparse Module for Parsing Command Lines,PEP 391 Dictionary-Based Configuration For Logging,PEP 378 Format Specifier for Thousands Separator<2>,What’s New in Python 2 7
@anchor{whatsnew/2 7 pep-389-the-argparse-module-for-parsing-command-lines}@anchor{e8d}
@subsection PEP 389: The argparse Module for Parsing Command Lines


The @ref{5,,argparse} module for parsing command-line arguments was
added as a more powerful replacement for the
@ref{bf,,optparse} module.

This means Python now supports three different modules for parsing
command-line arguments: @ref{81,,getopt}, @ref{bf,,optparse}, and
@ref{5,,argparse}.  The @ref{81,,getopt} module closely resembles the C
library’s @code{getopt()} function, so it remains useful if you’re writing a
Python prototype that will eventually be rewritten in C.
@ref{bf,,optparse} becomes redundant, but there are no plans to remove it
because there are many scripts still using it, and there’s no
automated way to update these scripts.  (Making the @ref{5,,argparse}
API consistent with @ref{bf,,optparse}’s interface was discussed but
rejected as too messy and difficult.)

In short, if you’re writing a new script and don’t need to worry
about compatibility with earlier versions of Python, use
@ref{5,,argparse} instead of @ref{bf,,optparse}.

Here’s an example:

@example
import argparse

parser = argparse.ArgumentParser(description='Command-line example.')

# Add optional switches
parser.add_argument('-v', action='store_true', dest='is_verbose',
                    help='produce verbose output')
parser.add_argument('-o', action='store', dest='output',
                    metavar='FILE',
                    help='direct output to FILE instead of stdout')
parser.add_argument('-C', action='store', type=int, dest='context',
                    metavar='NUM', default=0,
                    help='display NUM lines of added context')

# Allow any number of additional arguments.
parser.add_argument(nargs='*', action='store', dest='inputs',
                    help='input filenames (default is stdin)')

args = parser.parse_args()
print args.__dict__
@end example

Unless you override it, @code{-h} and @code{--help} switches
are automatically added, and produce neatly formatted output:

@example
-> ./python.exe argparse-example.py --help
usage: argparse-example.py [-h] [-v] [-o FILE] [-C NUM] [inputs [inputs ...]]

Command-line example.

positional arguments:
  inputs      input filenames (default is stdin)

optional arguments:
  -h, --help  show this help message and exit
  -v          produce verbose output
  -o FILE     direct output to FILE instead of stdout
  -C NUM      display NUM lines of added context
@end example

As with @ref{bf,,optparse}, the command-line switches and arguments
are returned as an object with attributes named by the @emph{dest} parameters:

@example
-> ./python.exe argparse-example.py -v
@{'output': None,
 'is_verbose': True,
 'context': 0,
 'inputs': []@}

-> ./python.exe argparse-example.py -v -o /tmp/output -C 4 file1 file2
@{'output': '/tmp/output',
 'is_verbose': True,
 'context': 4,
 'inputs': ['file1', 'file2']@}
@end example

@ref{5,,argparse} has much fancier validation than @ref{bf,,optparse}; you
can specify an exact number of arguments as an integer, 0 or more
arguments by passing @code{'*'}, 1 or more by passing @code{'+'}, or an
optional argument with @code{'?'}.  A top-level parser can contain
sub-parsers to define subcommands that have different sets of
switches, as in @code{svn commit}, @code{svn checkout}, etc.  You can
specify an argument’s type as @ref{a94,,FileType}, which will
automatically open files for you and understands that @code{'-'} means
standard input or output.


@subsubheading See also



@table @asis

@item @ref{5,,argparse} documentation

The documentation page of the argparse module.

@item @ref{d53,,Upgrading optparse code}

Part of the Python documentation, describing how to convert
code that uses @ref{bf,,optparse}.

@geindex Python Enhancement Proposals; PEP 389

@item PEP 389@footnote{https://www.python.org/dev/peps/pep-0389} - argparse - New Command Line Parsing Module

PEP written and implemented by Steven Bethard.
@end table


@node PEP 391 Dictionary-Based Configuration For Logging,PEP 3106 Dictionary Views,PEP 389 The argparse Module for Parsing Command Lines,What’s New in Python 2 7
@anchor{whatsnew/2 7 pep-391-dictionary-based-configuration-for-logging}@anchor{e8e}
@subsection PEP 391: Dictionary-Based Configuration For Logging


The @ref{a6,,logging} module is very flexible; applications can define
a tree of logging subsystems, and each logger in this tree can filter
out certain messages, format them differently, and direct messages to
a varying number of handlers.

All this flexibility can require a lot of configuration.  You can
write Python statements to create objects and set their properties,
but a complex set-up requires verbose but boring code.
@ref{a6,,logging} also supports a @code{fileConfig()}
function that parses a file, but the file format doesn’t support
configuring filters, and it’s messier to generate programmatically.

Python 2.7 adds a @code{dictConfig()} function that
uses a dictionary to configure logging.  There are many ways to
produce a dictionary from different sources: construct one with code;
parse a file containing JSON; or use a YAML parsing library if one is
installed.  For more information see @ref{e8f,,Configuration functions}.

The following example configures two loggers, the root logger and a
logger named “network”.  Messages sent to the root logger will be
sent to the system log using the syslog protocol, and messages
to the “network” logger will be written to a @code{network.log} file
that will be rotated once the log reaches 1MB.

@example
import logging
import logging.config

configdict = @{
 'version': 1,    # Configuration schema in use; must be 1 for now
 'formatters': @{
     'standard': @{
         'format': ('%(asctime)s %(name)-15s '
                    '%(levelname)-8s %(message)s')@}@},

 'handlers': @{'netlog': @{'backupCount': 10,
                     'class': 'logging.handlers.RotatingFileHandler',
                     'filename': '/logs/network.log',
                     'formatter': 'standard',
                     'level': 'INFO',
                     'maxBytes': 1000000@},
              'syslog': @{'class': 'logging.handlers.SysLogHandler',
                         'formatter': 'standard',
                         'level': 'ERROR'@}@},

 # Specify all the subordinate loggers
 'loggers': @{
             'network': @{
                         'handlers': ['netlog']
             @}
 @},
 # Specify properties of the root logger
 'root': @{
          'handlers': ['syslog']
 @},
@}

# Set up configuration
logging.config.dictConfig(configdict)

# As an example, log two error messages
logger = logging.getLogger('/')
logger.error('Database not found')

netlogger = logging.getLogger('network')
netlogger.error('Connection failed')
@end example

Three smaller enhancements to the @ref{a6,,logging} module, all
implemented by Vinay Sajip, are:

@c rev79293


@itemize *

@item 
The @ref{c6c,,SysLogHandler} class now supports
syslogging over TCP.  The constructor has a @emph{socktype} parameter
giving the type of socket to use, either @ref{e90,,socket.SOCK_DGRAM}
for UDP or @ref{e91,,socket.SOCK_STREAM} for TCP.  The default
protocol remains UDP.

@item 
@ref{676,,Logger} instances gained a @ref{e92,,getChild()}
method that retrieves a descendant logger using a relative path.
For example, once you retrieve a logger by doing @code{log = getLogger('app')},
calling @code{log.getChild('network.listen')} is equivalent to
@code{getLogger('app.network.listen')}.

@item 
The @ref{e93,,LoggerAdapter} class gained an
@code{isEnabledFor()} method that takes a
@emph{level} and returns whether the underlying logger would
process a message of that level of importance.
@end itemize

@c XXX: Logger objects don't have a class declaration so the link don't work


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 391

@item PEP 391@footnote{https://www.python.org/dev/peps/pep-0391} - Dictionary-Based Configuration For Logging

PEP written and implemented by Vinay Sajip.
@end table


@node PEP 3106 Dictionary Views,PEP 3137 The memoryview Object,PEP 391 Dictionary-Based Configuration For Logging,What’s New in Python 2 7
@anchor{whatsnew/2 7 pep-3106-dictionary-views}@anchor{e94}
@subsection PEP 3106: Dictionary Views


The dictionary methods @ref{243,,keys()}, @ref{244,,values()}, and
@ref{245,,items()} are different in Python 3.x.  They return an object
called a @emph{view} instead of a fully materialized list.

It’s not possible to change the return values of @ref{243,,keys()},
@ref{244,,values()}, and @ref{245,,items()} in Python 2.7 because
too much code would break.  Instead the 3.x versions were added
under the new names @code{viewkeys()}, @code{viewvalues()},
and @code{viewitems()}.

@example
>>> d = dict((i*10, chr(65+i)) for i in range(26))
>>> d
@{0: 'A', 130: 'N', 10: 'B', 140: 'O', 20: ..., 250: 'Z'@}
>>> d.viewkeys()
dict_keys([0, 130, 10, 140, 20, 150, 30, ..., 250])
@end example

Views can be iterated over, but the key and item views also behave
like sets.  The @code{&} operator performs intersection, and @code{|}
performs a union:

@example
>>> d1 = dict((i*10, chr(65+i)) for i in range(26))
>>> d2 = dict((i**.5, i) for i in range(1000))
>>> d1.viewkeys() & d2.viewkeys()
set([0.0, 10.0, 20.0, 30.0])
>>> d1.viewkeys() | range(0, 30)
set([0, 1, 130, 3, 4, 5, 6, ..., 120, 250])
@end example

The view keeps track of the dictionary and its contents change as the
dictionary is modified:

@example
>>> vk = d.viewkeys()
>>> vk
dict_keys([0, 130, 10, ..., 250])
>>> d[260] = '&'
>>> vk
dict_keys([0, 130, 260, 10, ..., 250])
@end example

However, note that you can’t add or remove keys while you’re iterating
over the view:

@example
>>> for k in vk:
...     d[k*2] = k
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: dictionary changed size during iteration
@end example

You can use the view methods in Python 2.x code, and the 2to3
converter will change them to the standard @ref{243,,keys()},
@ref{244,,values()}, and @ref{245,,items()} methods.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3106

@item PEP 3106@footnote{https://www.python.org/dev/peps/pep-3106} - Revamping dict.keys(), .values() and .items()

PEP written by Guido van Rossum.
Backported to 2.7 by Alexandre Vassalotti; bpo-1967@footnote{https://bugs.python.org/issue1967}.
@end table


@node PEP 3137 The memoryview Object,Other Language Changes<12>,PEP 3106 Dictionary Views,What’s New in Python 2 7
@anchor{whatsnew/2 7 pep-3137-the-memoryview-object}@anchor{e95}
@subsection PEP 3137: The memoryview Object


The @ref{573,,memoryview} object provides a view of another object’s
memory content that matches the @ref{2f0,,bytes} type’s interface.

@example
>>> import string
>>> m = memoryview(string.letters)
>>> m
<memory at 0x37f850>
>>> len(m)           # Returns length of underlying object
52
>>> m[0], m[25], m[26]   # Indexing returns one byte
('a', 'z', 'A')
>>> m2 = m[0:26]         # Slicing returns another memoryview
>>> m2
<memory at 0x37f080>
@end example

The content of the view can be converted to a string of bytes or
a list of integers:

@example
>>> m2.tobytes()
'abcdefghijklmnopqrstuvwxyz'
>>> m2.tolist()
[97, 98, 99, 100, 101, 102, 103, ... 121, 122]
>>>
@end example

@ref{573,,memoryview} objects allow modifying the underlying object if
it’s a mutable object.

@example
>>> m2[0] = 75
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot modify read-only memory
>>> b = bytearray(string.letters)  # Creating a mutable object
>>> b
bytearray(b'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ')
>>> mb = memoryview(b)
>>> mb[0] = '*'         # Assign to view, changing the bytearray.
>>> b[0:5]              # The bytearray has been changed.
bytearray(b'*bcde')
>>>
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3137

@item PEP 3137@footnote{https://www.python.org/dev/peps/pep-3137} - Immutable Bytes and Mutable Buffer

PEP written by Guido van Rossum.
Implemented by Travis Oliphant, Antoine Pitrou and others.
Backported to 2.7 by Antoine Pitrou; bpo-2396@footnote{https://bugs.python.org/issue2396}.
@end table


@node Other Language Changes<12>,New and Improved Modules,PEP 3137 The memoryview Object,What’s New in Python 2 7
@anchor{whatsnew/2 7 other-language-changes}@anchor{e96}
@subsection Other Language Changes


Some smaller changes made to the core Python language are:


@itemize *

@item 
The syntax for set literals has been backported from Python 3.x.
Curly brackets are used to surround the contents of the resulting
mutable set; set literals are
distinguished from dictionaries by not containing colons and values.
@code{@{@}} continues to represent an empty dictionary; use
@code{set()} for an empty set.

@example
>>> @{1, 2, 3, 4, 5@}
set([1, 2, 3, 4, 5])
>>> set() # empty set
set([])
>>> @{@}    # empty dict
@{@}
@end example

Backported by Alexandre Vassalotti; bpo-2335@footnote{https://bugs.python.org/issue2335}.

@item 
Dictionary and set comprehensions are another feature backported from
3.x, generalizing list/generator comprehensions to use
the literal syntax for sets and dictionaries.

@example
>>> @{x: x*x for x in range(6)@}
@{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25@}
>>> @{('a'*x) for x in range(6)@}
set(['@w{'}, 'a', 'aa', 'aaa', 'aaaa', 'aaaaa'])
@end example

Backported by Alexandre Vassalotti; bpo-2333@footnote{https://bugs.python.org/issue2333}.

@item 
The @ref{168,,with} statement can now use multiple context managers
in one statement.  Context managers are processed from left to right
and each one is treated as beginning a new @code{with} statement.
This means that:

@example
with A() as a, B() as b:
    ... suite of statements ...
@end example

is equivalent to:

@example
with A() as a:
    with B() as b:
        ... suite of statements ...
@end example

The @code{contextlib.nested()} function provides a very similar
function, so it’s no longer necessary and has been deprecated.

(Proposed in @indicateurl{https://codereview.appspot.com/53094}; implemented by
Georg Brandl.)

@item 
Conversions between floating-point numbers and strings are
now correctly rounded on most platforms.  These conversions occur
in many different places: @ref{2ef,,str()} on
floats and complex numbers; the @ref{256,,float} and @ref{33b,,complex}
constructors;
numeric formatting; serializing and
deserializing floats and complex numbers using the
@ref{ac,,marshal}, @ref{c5,,pickle}
and @ref{a0,,json} modules;
parsing of float and imaginary literals in Python code;
and @ref{248,,Decimal}-to-float conversion.

Related to this, the @ref{27d,,repr()} of a floating-point number @emph{x}
now returns a result based on the shortest decimal string that’s
guaranteed to round back to @emph{x} under correct rounding (with
round-half-to-even rounding mode).  Previously it gave a string
based on rounding x to 17 decimal digits.

@c maybe add an example?

The rounding library responsible for this improvement works on
Windows and on Unix platforms using the gcc, icc, or suncc
compilers.  There may be a small number of platforms where correct
operation of this code cannot be guaranteed, so the code is not
used on such systems.  You can find out which code is being used
by checking @ref{e97,,sys.float_repr_style},  which will be @code{short}
if the new code is in use and @code{legacy} if it isn’t.

Implemented by Eric Smith and Mark Dickinson, using David Gay’s
@code{dtoa.c} library; bpo-7117@footnote{https://bugs.python.org/issue7117}.

@item 
Conversions from long integers and regular integers to floating
point now round differently, returning the floating-point number
closest to the number.  This doesn’t matter for small integers that
can be converted exactly, but for large numbers that will
unavoidably lose precision, Python 2.7 now approximates more
closely.  For example, Python 2.6 computed the following:

@example
>>> n = 295147905179352891391
>>> float(n)
2.9514790517935283e+20
>>> n - long(float(n))
65535L
@end example

Python 2.7’s floating-point result is larger, but much closer to the
true value:

@example
>>> n = 295147905179352891391
>>> float(n)
2.9514790517935289e+20
>>> n - long(float(n))
-1L
@end example

(Implemented by Mark Dickinson; bpo-3166@footnote{https://bugs.python.org/issue3166}.)

Integer division is also more accurate in its rounding behaviours.  (Also
implemented by Mark Dickinson; bpo-1811@footnote{https://bugs.python.org/issue1811}.)

@item 
Implicit coercion for complex numbers has been removed; the interpreter
will no longer ever attempt to call a @code{__coerce__()} method on complex
objects.  (Removed by Meador Inge and Mark Dickinson; bpo-5211@footnote{https://bugs.python.org/issue5211}.)

@item 
The @ref{784,,str.format()} method now supports automatic numbering of the replacement
fields.  This makes using @ref{784,,str.format()} more closely resemble using
@code{%s} formatting:

@example
>>> '@{@}:@{@}:@{@}'.format(2009, 04, 'Sunday')
'2009:4:Sunday'
>>> '@{@}:@{@}:@{day@}'.format(2009, 4, day='Sunday')
'2009:4:Sunday'
@end example

The auto-numbering takes the fields from left to right, so the first @code{@{...@}}
specifier will use the first argument to @ref{784,,str.format()}, the next
specifier will use the next argument, and so on.  You can’t mix auto-numbering
and explicit numbering – either number all of your specifier fields or none
of them – but you can mix auto-numbering and named fields, as in the second
example above.  (Contributed by Eric Smith; bpo-5237@footnote{https://bugs.python.org/issue5237}.)

Complex numbers now correctly support usage with @ref{785,,format()},
and default to being right-aligned.
Specifying a precision or comma-separation applies to both the real
and imaginary parts of the number, but a specified field width and
alignment is applied to the whole of the resulting @code{1.5+3j}
output.  (Contributed by Eric Smith; bpo-1588@footnote{https://bugs.python.org/issue1588} and bpo-7988@footnote{https://bugs.python.org/issue7988}.)

The ‘F’ format code now always formats its output using uppercase characters,
so it will now produce ‘INF’ and ‘NAN’.
(Contributed by Eric Smith; bpo-3382@footnote{https://bugs.python.org/issue3382}.)

A low-level change: the @ref{ba7,,object.__format__()} method now triggers
a @ref{3a9,,PendingDeprecationWarning} if it’s passed a format string,
because the @code{__format__()} method for @ref{5a5,,object} converts
the object to a string representation and formats that.  Previously
the method silently applied the format string to the string
representation, but that could hide mistakes in Python code.  If
you’re supplying formatting information such as an alignment or
precision, presumably you’re expecting the formatting to be applied
in some object-specific way.  (Fixed by Eric Smith; bpo-7994@footnote{https://bugs.python.org/issue7994}.)

@item 
The @ref{1b1,,int()} and @code{long()} types gained a @code{bit_length}
method that returns the number of bits necessary to represent
its argument in binary:

@example
>>> n = 37
>>> bin(n)
'0b100101'
>>> n.bit_length()
6
>>> n = 2**123-1
>>> n.bit_length()
123
>>> (n+1).bit_length()
124
@end example

(Contributed by Fredrik Johansson and Victor Stinner; bpo-3439@footnote{https://bugs.python.org/issue3439}.)

@item 
The @ref{e35,,import} statement will no longer try an absolute import
if a relative import (e.g. @code{from .os import sep}) fails.  This
fixes a bug, but could possibly break certain @code{import}
statements that were only working by accident.  (Fixed by Meador Inge;
bpo-7902@footnote{https://bugs.python.org/issue7902}.)

@item 
It’s now possible for a subclass of the built-in @code{unicode} type
to override the @code{__unicode__()} method.  (Implemented by
Victor Stinner; bpo-1583863@footnote{https://bugs.python.org/issue1583863}.)

@item 
The @ref{2f1,,bytearray} type’s @ref{e98,,translate()} method now accepts
@code{None} as its first argument.  (Fixed by Georg Brandl;
bpo-4759@footnote{https://bugs.python.org/issue4759}.)

@c XXX bytearray doesn't seem to be documented

@item 
When using @code{@@classmethod} and @code{@@staticmethod} to wrap
methods as class or static methods, the wrapper object now
exposes the wrapped function as their @code{__func__} attribute.
(Contributed by Amaury Forgeot d’Arc, after a suggestion by
George Sakkis; bpo-5982@footnote{https://bugs.python.org/issue5982}.)

@item 
When a restricted set of attributes were set using @code{__slots__},
deleting an unset attribute would not raise @ref{163,,AttributeError}
as you would expect.  Fixed by Benjamin Peterson; bpo-7604@footnote{https://bugs.python.org/issue7604}.)

@item 
Two new encodings are now supported: “cp720”, used primarily for
Arabic text; and “cp858”, a variant of CP 850 that adds the euro
symbol.  (CP720 contributed by Alexander Belchenko and Amaury
Forgeot d’Arc in bpo-1616979@footnote{https://bugs.python.org/issue1616979}; CP858 contributed by Tim Hatch in
bpo-8016@footnote{https://bugs.python.org/issue8016}.)

@item 
The @code{file} object will now set the @code{filename} attribute
on the @ref{be8,,IOError} exception when trying to open a directory
on POSIX platforms (noted by Jan Kaliszewski; bpo-4764@footnote{https://bugs.python.org/issue4764}), and
now explicitly checks for and forbids writing to read-only file objects
instead of trusting the C library to catch and report the error
(fixed by Stefan Krah; bpo-5677@footnote{https://bugs.python.org/issue5677}).

@item 
The Python tokenizer now translates line endings itself, so the
@ref{4e2,,compile()} built-in function now accepts code using any
line-ending convention.  Additionally, it no longer requires that the
code end in a newline.

@item 
Extra parentheses in function definitions are illegal in Python 3.x,
meaning that you get a syntax error from @code{def f((x)): pass}.  In
Python3-warning mode, Python 2.7 will now warn about this odd usage.
(Noted by James Lingard; bpo-7362@footnote{https://bugs.python.org/issue7362}.)

@item 
It’s now possible to create weak references to old-style class
objects.  New-style classes were always weak-referenceable.  (Fixed
by Antoine Pitrou; bpo-8268@footnote{https://bugs.python.org/issue8268}.)

@item 
When a module object is garbage-collected, the module’s dictionary is
now only cleared if no one else is holding a reference to the
dictionary (bpo-7140@footnote{https://bugs.python.org/issue7140}).
@end itemize

@c ======================================================================

@menu
* Interpreter Changes:: 
* Optimizations: Optimizations<11>. 

@end menu

@node Interpreter Changes,Optimizations<11>,,Other Language Changes<12>
@anchor{whatsnew/2 7 interpreter-changes}@anchor{e99}@anchor{whatsnew/2 7 new-27-interpreter}@anchor{e9a}
@subsubsection Interpreter Changes


A new environment variable, 
@geindex PYTHONWARNINGS
@geindex environment variable; PYTHONWARNINGS
@ref{6de,,PYTHONWARNINGS},
allows controlling warnings.  It should be set to a string
containing warning settings, equivalent to those
used with the @ref{3a8,,-W} switch, separated by commas.
(Contributed by Brian Curtin; bpo-7301@footnote{https://bugs.python.org/issue7301}.)

For example, the following setting will print warnings every time
they occur, but turn warnings from the @code{Cookie} module into an
error.  (The exact syntax for setting an environment variable varies
across operating systems and shells.)

@example
export PYTHONWARNINGS=all,error:::Cookie:0
@end example

@c ======================================================================

@node Optimizations<11>,,Interpreter Changes,Other Language Changes<12>
@anchor{whatsnew/2 7 optimizations}@anchor{e9b}
@subsubsection Optimizations


Several performance enhancements have been added:


@itemize *

@item 
A new opcode was added to perform the initial setup for
@ref{168,,with} statements, looking up the @code{__enter__()} and
@code{__exit__()} methods.  (Contributed by Benjamin Peterson.)

@item 
The garbage collector now performs better for one common usage
pattern: when many objects are being allocated without deallocating
any of them.  This would previously take quadratic
time for garbage collection, but now the number of full garbage collections
is reduced as the number of objects on the heap grows.
The new logic only performs a full garbage collection pass when
the middle generation has been collected 10 times and when the
number of survivor objects from the middle generation exceeds 10% of
the number of objects in the oldest generation.  (Suggested by Martin
von Löwis and implemented by Antoine Pitrou; bpo-4074@footnote{https://bugs.python.org/issue4074}.)

@item 
The garbage collector tries to avoid tracking simple containers
which can’t be part of a cycle. In Python 2.7, this is now true for
tuples and dicts containing atomic types (such as ints, strings,
etc.). Transitively, a dict containing tuples of atomic types won’t
be tracked either. This helps reduce the cost of each
garbage collection by decreasing the number of objects to be
considered and traversed by the collector.
(Contributed by Antoine Pitrou; bpo-4688@footnote{https://bugs.python.org/issue4688}.)

@item 
Long integers are now stored internally either in base @code{2**15} or in base
@code{2**30}, the base being determined at build time.  Previously, they
were always stored in base @code{2**15}.  Using base @code{2**30} gives
significant performance improvements on 64-bit machines, but
benchmark results on 32-bit machines have been mixed.  Therefore,
the default is to use base @code{2**30} on 64-bit machines and base @code{2**15}
on 32-bit machines; on Unix, there’s a new configure option
@code{--enable-big-digits} that can be used to override this default.

Apart from the performance improvements this change should be
invisible to end users, with one exception: for testing and
debugging purposes there’s a new structseq @code{sys.long_info} that
provides information about the internal format, giving the number of
bits per digit and the size in bytes of the C type used to store
each digit:

@example
>>> import sys
>>> sys.long_info
sys.long_info(bits_per_digit=30, sizeof_digit=4)
@end example

(Contributed by Mark Dickinson; bpo-4258@footnote{https://bugs.python.org/issue4258}.)

Another set of changes made long objects a few bytes smaller: 2 bytes
smaller on 32-bit systems and 6 bytes on 64-bit.
(Contributed by Mark Dickinson; bpo-5260@footnote{https://bugs.python.org/issue5260}.)

@item 
The division algorithm for long integers has been made faster
by tightening the inner loop, doing shifts instead of multiplications,
and fixing an unnecessary extra iteration.
Various benchmarks show speedups of between 50% and 150% for long
integer divisions and modulo operations.
(Contributed by Mark Dickinson; bpo-5512@footnote{https://bugs.python.org/issue5512}.)
Bitwise operations are also significantly faster (initial patch by
Gregory Smith; bpo-1087418@footnote{https://bugs.python.org/issue1087418}).

@item 
The implementation of @code{%} checks for the left-side operand being
a Python string and special-cases it; this results in a 1–3%
performance increase for applications that frequently use @code{%}
with strings, such as templating libraries.
(Implemented by Collin Winter; bpo-5176@footnote{https://bugs.python.org/issue5176}.)

@item 
List comprehensions with an @code{if} condition are compiled into
faster bytecode.  (Patch by Antoine Pitrou, back-ported to 2.7
by Jeffrey Yasskin; bpo-4715@footnote{https://bugs.python.org/issue4715}.)

@item 
Converting an integer or long integer to a decimal string was made
faster by special-casing base 10 instead of using a generalized
conversion function that supports arbitrary bases.
(Patch by Gawain Bolton; bpo-6713@footnote{https://bugs.python.org/issue6713}.)

@item 
The @code{split()}, @code{replace()}, @code{rindex()},
@code{rpartition()}, and @code{rsplit()} methods of string-like types
(strings, Unicode strings, and @ref{2f1,,bytearray} objects) now use a
fast reverse-search algorithm instead of a character-by-character
scan.  This is sometimes faster by a factor of 10.  (Added by
Florent Xicluna; bpo-7462@footnote{https://bugs.python.org/issue7462} and bpo-7622@footnote{https://bugs.python.org/issue7622}.)

@item 
The @ref{c5,,pickle} and @code{cPickle} modules now automatically
intern the strings used for attribute names, reducing memory usage
of the objects resulting from unpickling.  (Contributed by Jake
McGuire; bpo-5084@footnote{https://bugs.python.org/issue5084}.)

@item 
The @code{cPickle} module now special-cases dictionaries,
nearly halving the time required to pickle them.
(Contributed by Collin Winter; bpo-5670@footnote{https://bugs.python.org/issue5670}.)
@end itemize

@c ======================================================================

@node New and Improved Modules,Build and C API Changes<8>,Other Language Changes<12>,What’s New in Python 2 7
@anchor{whatsnew/2 7 new-and-improved-modules}@anchor{e9c}
@subsection New and Improved Modules


As in every release, Python’s standard library received a number of
enhancements and bug fixes.  Here’s a partial list of the most notable
changes, sorted alphabetically by module name. Consult the
@code{Misc/NEWS} file in the source tree for a more complete list of
changes, or look through the Subversion logs for all the details.


@itemize *

@item 
The @ref{e,,bdb} module’s base debugging class @ref{e9d,,Bdb}
gained a feature for skipping modules.  The constructor
now takes an iterable containing glob-style patterns such as
@code{django.*}; the debugger will not step into stack frames
from a module that matches one of these patterns.
(Contributed by Maru Newby after a suggestion by
Senthil Kumaran; bpo-5142@footnote{https://bugs.python.org/issue5142}.)

@item 
The @ref{f,,binascii} module now supports the buffer API, so it can be
used with @ref{573,,memoryview} instances and other similar buffer objects.
(Backported from 3.x by Florent Xicluna; bpo-7703@footnote{https://bugs.python.org/issue7703}.)

@item 
Updated module: the @code{bsddb} module has been updated from 4.7.2devel9
to version 4.8.4 of
the pybsddb package@footnote{https://www.jcea.es/programacion/pybsddb.htm}.
The new version features better Python 3.x compatibility, various bug fixes,
and adds several new BerkeleyDB flags and methods.
(Updated by Jesús Cea Avión; bpo-8156@footnote{https://bugs.python.org/issue8156}.  The pybsddb
changelog can be read at @indicateurl{http://hg.jcea.es/pybsddb/file/tip/ChangeLog}.)

@item 
The @ref{12,,bz2} module’s @ref{455,,BZ2File} now supports the context
management protocol, so you can write @code{with bz2.BZ2File(...) as f:}.
(Contributed by Hagen Fürstenau; bpo-3860@footnote{https://bugs.python.org/issue3860}.)

@item 
New class: the @ref{c2a,,Counter} class in the @ref{1c,,collections}
module is useful for tallying data.  @ref{c2a,,Counter} instances
behave mostly like dictionaries but return zero for missing keys instead of
raising a @ref{5bb,,KeyError}:

@example
>>> from collections import Counter
>>> c = Counter()
>>> for letter in 'here is a sample of english text':
...   c[letter] += 1
...
>>> c 
Counter(@{' ': 6, 'e': 5, 's': 3, 'a': 2, 'i': 2, 'h': 2,
'l': 2, 't': 2, 'g': 1, 'f': 1, 'm': 1, 'o': 1, 'n': 1,
'p': 1, 'r': 1, 'x': 1@})
>>> c['e']
5
>>> c['z']
0
@end example

There are three additional @ref{c2a,,Counter} methods.
@ref{e9e,,most_common()} returns the N most common
elements and their counts.  @ref{e9f,,elements()}
returns an iterator over the contained elements, repeating each
element as many times as its count.
@ref{d80,,subtract()} takes an iterable and
subtracts one for each element instead of adding; if the argument is
a dictionary or another @code{Counter}, the counts are
subtracted.

@example
>>> c.most_common(5)
[(' ', 6), ('e', 5), ('s', 3), ('a', 2), ('i', 2)]
>>> c.elements() ->
   'a', 'a', ' ', ' ', ' ', ' ', ' ', ' ',
   'e', 'e', 'e', 'e', 'e', 'g', 'f', 'i', 'i',
   'h', 'h', 'm', 'l', 'l', 'o', 'n', 'p', 's',
   's', 's', 'r', 't', 't', 'x'
>>> c['e']
5
>>> c.subtract('very heavy on the letter e')
>>> c['e']    # Count is now lower
-1
@end example

Contributed by Raymond Hettinger; bpo-1696199@footnote{https://bugs.python.org/issue1696199}.

@c revision 79660

New class: @ref{4e5,,OrderedDict} is described in the earlier
section @ref{e87,,PEP 372; Adding an Ordered Dictionary to collections}.

New method: The @ref{7d3,,deque} data type now has a
@ref{d82,,count()} method that returns the number of
contained elements equal to the supplied argument @emph{x}, and a
@ref{d83,,reverse()} method that reverses the elements
of the deque in-place.  @ref{7d3,,deque} also exposes its maximum
length as the read-only @ref{ea0,,maxlen} attribute.
(Both features added by Raymond Hettinger.)

The @ref{3e4,,namedtuple} class now has an optional @emph{rename} parameter.
If @emph{rename} is true, field names that are invalid because they’ve
been repeated or aren’t legal Python identifiers will be
renamed to legal names that are derived from the field’s
position within the list of fields:

@example
>>> from collections import namedtuple
>>> T = namedtuple('T', ['field1', '$illegal', 'for', 'field2'], rename=True)
>>> T._fields
('field1', '_1', '_2', 'field2')
@end example

(Added by Raymond Hettinger; bpo-1818@footnote{https://bugs.python.org/issue1818}.)

Finally, the @code{Mapping} abstract base class now
returns @ref{24c,,NotImplemented} if a mapping is compared to
another type that isn’t a @code{Mapping}.
(Fixed by Daniel Stutzbach; bpo-8729@footnote{https://bugs.python.org/issue8729}.)

@item 
Constructors for the parsing classes in the @code{ConfigParser} module now
take an @emph{allow_no_value} parameter, defaulting to false; if true,
options without values will be allowed.  For example:

@example
>>> import ConfigParser, StringIO
>>> sample_config = """
... [mysqld]
... user = mysql
... pid-file = /var/run/mysqld/mysqld.pid
... skip-bdb
... """
>>> config = ConfigParser.RawConfigParser(allow_no_value=True)
>>> config.readfp(StringIO.StringIO(sample_config))
>>> config.get('mysqld', 'user')
'mysql'
>>> print config.get('mysqld', 'skip-bdb')
None
>>> print config.get('mysqld', 'unknown')
Traceback (most recent call last):
  ...
NoOptionError: No option 'unknown' in section: 'mysqld'
@end example

(Contributed by Mats Kindahl; bpo-7005@footnote{https://bugs.python.org/issue7005}.)

@item 
Deprecated function: @code{contextlib.nested()}, which allows
handling more than one context manager with a single @ref{168,,with}
statement, has been deprecated, because the @code{with} statement
now supports multiple context managers.

@item 
The @code{cookielib} module now ignores cookies that have an invalid
version field, one that doesn’t contain an integer value.  (Fixed by
John J. Lee; bpo-3924@footnote{https://bugs.python.org/issue3924}.)

@item 
The @ref{24,,copy} module’s @ref{69b,,deepcopy()} function will now
correctly copy bound instance methods.  (Implemented by
Robert Collins; bpo-1515@footnote{https://bugs.python.org/issue1515}.)

@item 
The @ref{29,,ctypes} module now always converts @code{None} to a C @code{NULL}
pointer for arguments declared as pointers.  (Changed by Thomas
Heller; bpo-4606@footnote{https://bugs.python.org/issue4606}.)  The underlying libffi library@footnote{https://sourceware.org/libffi/} has been updated to version
3.0.9, containing various fixes for different platforms.  (Updated
by Matthias Klose; bpo-8142@footnote{https://bugs.python.org/issue8142}.)

@item 
New method: the @ref{2f,,datetime} module’s @ref{4c8,,timedelta} class
gained a @ref{ea1,,total_seconds()} method that returns the
number of seconds in the duration.  (Contributed by Brian Quinlan; bpo-5788@footnote{https://bugs.python.org/issue5788}.)

@item 
New method: the @ref{248,,Decimal} class gained a
@ref{da5,,from_float()} class method that performs an exact
conversion of a floating-point number to a @ref{248,,Decimal}.
This exact conversion strives for the
closest decimal approximation to the floating-point representation’s value;
the resulting decimal value will therefore still include the inaccuracy,
if any.
For example, @code{Decimal.from_float(0.1)} returns
@code{Decimal('0.1000000000000000055511151231257827021181583404541015625')}.
(Implemented by Raymond Hettinger; bpo-4796@footnote{https://bugs.python.org/issue4796}.)

Comparing instances of @ref{248,,Decimal} with floating-point
numbers now produces sensible results based on the numeric values
of the operands.  Previously such comparisons would fall back to
Python’s default rules for comparing objects, which produced arbitrary
results based on their type.  Note that you still cannot combine
@code{Decimal} and floating-point in other operations such as addition,
since you should be explicitly choosing how to convert between float and
@ref{248,,Decimal}.  (Fixed by Mark Dickinson; bpo-2531@footnote{https://bugs.python.org/issue2531}.)

The constructor for @ref{248,,Decimal} now accepts
floating-point numbers (added by Raymond Hettinger; bpo-8257@footnote{https://bugs.python.org/issue8257})
and non-European Unicode characters such as Arabic-Indic digits
(contributed by Mark Dickinson; bpo-6595@footnote{https://bugs.python.org/issue6595}).

Most of the methods of the @ref{c40,,Context} class now accept integers
as well as @ref{248,,Decimal} instances; the only exceptions are the
@ref{ea2,,canonical()} and @ref{ea3,,is_canonical()}
methods.  (Patch by Juan José Conti; bpo-7633@footnote{https://bugs.python.org/issue7633}.)

When using @ref{248,,Decimal} instances with a string’s
@ref{784,,format()} method, the default alignment was previously
left-alignment.  This has been changed to right-alignment, which is
more sensible for numeric types.  (Changed by Mark Dickinson; bpo-6857@footnote{https://bugs.python.org/issue6857}.)

Comparisons involving a signaling NaN value (or @code{sNAN}) now signal
@code{InvalidOperation} instead of silently returning a true or
false value depending on the comparison operator.  Quiet NaN values
(or @code{NaN}) are now hashable.  (Fixed by Mark Dickinson;
bpo-7279@footnote{https://bugs.python.org/issue7279}.)

@item 
The @ref{35,,difflib} module now produces output that is more
compatible with modern @code{diff}/@code{patch} tools
through one small change, using a tab character instead of spaces as
a separator in the header giving the filename.  (Fixed by Anatoly
Techtonik; bpo-7585@footnote{https://bugs.python.org/issue7585}.)

@item 
The Distutils @code{sdist} command now always regenerates the
@code{MANIFEST} file, since even if the @code{MANIFEST.in} or
@code{setup.py} files haven’t been modified, the user might have
created some new files that should be included.
(Fixed by Tarek Ziadé; bpo-8688@footnote{https://bugs.python.org/issue8688}.)

@item 
The @ref{63,,doctest} module’s @code{IGNORE_EXCEPTION_DETAIL} flag
will now ignore the name of the module containing the exception
being tested.  (Patch by Lennart Regebro; bpo-7490@footnote{https://bugs.python.org/issue7490}.)

@item 
The @ref{64,,email} module’s @ref{7e3,,Message} class will
now accept a Unicode-valued payload, automatically converting the
payload to the encoding specified by @code{output_charset}.
(Added by R. David Murray; bpo-1368247@footnote{https://bugs.python.org/issue1368247}.)

@item 
The @ref{172,,Fraction} class now accepts a single float or
@ref{248,,Decimal} instance, or two rational numbers, as
arguments to its constructor.  (Implemented by Mark Dickinson;
rationals added in bpo-5812@footnote{https://bugs.python.org/issue5812}, and float/decimal in
bpo-8294@footnote{https://bugs.python.org/issue8294}.)

Ordering comparisons (@code{<}, @code{<=}, @code{>}, @code{>=}) between
fractions and complex numbers now raise a @ref{162,,TypeError}.
This fixes an oversight, making the @ref{172,,Fraction}
match the other numeric types.

@c revision 79455

@item 
New class: @ref{3ea,,FTP_TLS} in
the @ref{7e,,ftplib} module provides secure FTP
connections using TLS encapsulation of authentication as well as
subsequent control and data transfers.
(Contributed by Giampaolo Rodola; bpo-2054@footnote{https://bugs.python.org/issue2054}.)

The @ref{ea4,,storbinary()} method for binary uploads can now restart
uploads thanks to an added @emph{rest} parameter (patch by Pablo Mouzo;
bpo-6845@footnote{https://bugs.python.org/issue6845}.)

@item 
New class decorator: @ref{abf,,total_ordering()} in the @ref{7f,,functools}
module takes a class that defines an @code{__eq__()} method and one of
@code{__lt__()}, @code{__le__()}, @code{__gt__()}, or @code{__ge__()},
and generates the missing comparison methods.  Since the
@code{__cmp__()} method is being deprecated in Python 3.x,
this decorator makes it easier to define ordered classes.
(Added by Raymond Hettinger; bpo-5479@footnote{https://bugs.python.org/issue5479}.)

New function: @ref{d7c,,cmp_to_key()} will take an old-style comparison
function that expects two arguments and return a new callable that
can be used as the @emph{key} parameter to functions such as
@ref{704,,sorted()}, @ref{a7d,,min()} and @ref{a7e,,max()}, etc.  The primary
intended use is to help with making code compatible with Python 3.x.
(Added by Raymond Hettinger.)

@item 
New function: the @ref{80,,gc} module’s @ref{ea5,,is_tracked()} returns
true if a given instance is tracked by the garbage collector, false
otherwise. (Contributed by Antoine Pitrou; bpo-4688@footnote{https://bugs.python.org/issue4688}.)

@item 
The @ref{87,,gzip} module’s @ref{43f,,GzipFile} now supports the context
management protocol, so you can write @code{with gzip.GzipFile(...) as f:}
(contributed by Hagen Fürstenau; bpo-3860@footnote{https://bugs.python.org/issue3860}), and it now implements
the @ref{823,,io.BufferedIOBase} ABC, so you can wrap it with
@ref{daf,,io.BufferedReader} for faster processing
(contributed by Nir Aides; bpo-7471@footnote{https://bugs.python.org/issue7471}).
It’s also now possible to override the modification time
recorded in a gzipped file by providing an optional timestamp to
the constructor.  (Contributed by Jacques Frechet; bpo-4272@footnote{https://bugs.python.org/issue4272}.)

Files in gzip format can be padded with trailing zero bytes; the
@ref{87,,gzip} module will now consume these trailing bytes.  (Fixed by
Tadek Pietraszek and Brian Curtin; bpo-2846@footnote{https://bugs.python.org/issue2846}.)

@item 
New attribute: the @ref{88,,hashlib} module now has an @code{algorithms}
attribute containing a tuple naming the supported algorithms.
In Python 2.7, @code{hashlib.algorithms} contains
@code{('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512')}.
(Contributed by Carl Chenet; bpo-7418@footnote{https://bugs.python.org/issue7418}.)

@item 
The default @code{HTTPResponse} class used by the @code{httplib} module now
supports buffering, resulting in much faster reading of HTTP responses.
(Contributed by Kristján Valur Jónsson; bpo-4879@footnote{https://bugs.python.org/issue4879}.)

The @code{HTTPConnection} and @code{HTTPSConnection} classes
now support a @emph{source_address} parameter, a @code{(host, port)} 2-tuple
giving the source address that will be used for the connection.
(Contributed by Eldon Ziegler; bpo-3972@footnote{https://bugs.python.org/issue3972}.)

@item 
The @code{ihooks} module now supports relative imports.  Note that
@code{ihooks} is an older module for customizing imports,
superseded by the @code{imputil} module added in Python 2.0.
(Relative import support added by Neil Schemenauer.)

@c revision 75423

@item 
The @ref{93,,imaplib} module now supports IPv6 addresses.
(Contributed by Derek Morr; bpo-1655@footnote{https://bugs.python.org/issue1655}.)

@item 
New function: the @ref{9c,,inspect} module’s @ref{a2f,,getcallargs()}
takes a callable and its positional and keyword arguments,
and figures out which of the callable’s parameters will receive each argument,
returning a dictionary mapping argument names to their values.  For example:

@example
>>> from inspect import getcallargs
>>> def f(a, b=1, *pos, **named):
...     pass
>>> getcallargs(f, 1, 2, 3)
@{'a': 1, 'b': 2, 'pos': (3,), 'named': @{@}@}
>>> getcallargs(f, a=2, x=4)
@{'a': 2, 'b': 1, 'pos': (), 'named': @{'x': 4@}@}
>>> getcallargs(f)
Traceback (most recent call last):
...
TypeError: f() takes at least 1 argument (0 given)
@end example

Contributed by George Sakkis; bpo-3135@footnote{https://bugs.python.org/issue3135}.

@item 
Updated module: The @ref{9d,,io} library has been upgraded to the version shipped with
Python 3.1.  For 3.1, the I/O library was entirely rewritten in C
and is 2 to 20 times faster depending on the task being performed.  The
original Python version was renamed to the @code{_pyio} module.

One minor resulting change: the @ref{e47,,io.TextIOBase} class now
has an @code{errors} attribute giving the error setting
used for encoding and decoding errors (one of @code{'strict'}, @code{'replace'},
@code{'ignore'}).

The @ref{ea6,,io.FileIO} class now raises an @ref{2a1,,OSError} when passed
an invalid file descriptor.  (Implemented by Benjamin Peterson;
bpo-4991@footnote{https://bugs.python.org/issue4991}.)  The @ref{ea7,,truncate()} method now preserves the
file position; previously it would change the file position to the
end of the new file.  (Fixed by Pascal Chambon; bpo-6939@footnote{https://bugs.python.org/issue6939}.)

@item 
New function: @code{itertools.compress(data, selectors)} takes two
iterators.  Elements of @emph{data} are returned if the corresponding
value in @emph{selectors} is true:

@example
itertools.compress('ABCDEF', [1,0,1,0,1,1]) =>
  A, C, E, F
@end example

@c maybe here is better to use >>> list(itertools.compress(...)) instead

New function: @code{itertools.combinations_with_replacement(iter, r)}
returns all the possible @emph{r}-length combinations of elements from the
iterable @emph{iter}.  Unlike @ref{ea8,,combinations()}, individual elements
can be repeated in the generated combinations:

@example
itertools.combinations_with_replacement('abc', 2) =>
  ('a', 'a'), ('a', 'b'), ('a', 'c'),
  ('b', 'b'), ('b', 'c'), ('c', 'c')
@end example

Note that elements are treated as unique depending on their position
in the input, not their actual values.

The @ref{e33,,itertools.count()} function now has a @emph{step} argument that
allows incrementing by values other than 1.  @ref{e33,,count()} also
now allows keyword arguments, and using non-integer values such as
floats or @ref{248,,Decimal} instances.  (Implemented by Raymond
Hettinger; bpo-5032@footnote{https://bugs.python.org/issue5032}.)

@ref{ea8,,itertools.combinations()} and @ref{ea9,,itertools.product()}
previously raised @ref{17c,,ValueError} for values of @emph{r} larger than
the input iterable.  This was deemed a specification error, so they
now return an empty iterator.  (Fixed by Raymond Hettinger; bpo-4816@footnote{https://bugs.python.org/issue4816}.)

@item 
Updated module: The @ref{a0,,json} module was upgraded to version 2.0.9 of the
simplejson package, which includes a C extension that makes
encoding and decoding faster.
(Contributed by Bob Ippolito; bpo-4136@footnote{https://bugs.python.org/issue4136}.)

To support the new @ref{4e5,,collections.OrderedDict} type, @ref{7fe,,json.load()}
now has an optional @emph{object_pairs_hook} parameter that will be called
with any object literal that decodes to a list of pairs.
(Contributed by Raymond Hettinger; bpo-5381@footnote{https://bugs.python.org/issue5381}.)

@item 
The @ref{aa,,mailbox} module’s @ref{eaa,,Maildir} class now records the
timestamp on the directories it reads, and only re-reads them if the
modification time has subsequently changed.  This improves
performance by avoiding unneeded directory scans.  (Fixed by
A.M. Kuchling and Antoine Pitrou; bpo-1607951@footnote{https://bugs.python.org/issue1607951}, bpo-6896@footnote{https://bugs.python.org/issue6896}.)

@item 
New functions: the @ref{ad,,math} module gained
@ref{710,,erf()} and @ref{711,,erfc()} for the error function and the complementary error function,
@ref{d8e,,expm1()} which computes @code{e**x - 1} with more precision than
using @ref{eab,,exp()} and subtracting 1,
@ref{d8f,,gamma()} for the Gamma function, and
@ref{d90,,lgamma()} for the natural log of the Gamma function.
(Contributed by Mark Dickinson and nirinA raseliarison; bpo-3366@footnote{https://bugs.python.org/issue3366}.)

@item 
The @ref{b3,,multiprocessing} module’s @code{Manager*} classes
can now be passed a callable that will be called whenever
a subprocess is started, along with a set of arguments that will be
passed to the callable.
(Contributed by lekma; bpo-5585@footnote{https://bugs.python.org/issue5585}.)

The @code{Pool} class, which controls a pool of worker processes,
now has an optional @emph{maxtasksperchild} parameter.  Worker processes
will perform the specified number of tasks and then exit, causing the
@code{Pool} to start a new worker.  This is useful if tasks may leak
memory or other resources, or if some tasks will cause the worker to
become very large.
(Contributed by Charles Cazabon; bpo-6963@footnote{https://bugs.python.org/issue6963}.)

@item 
The @ref{bc,,nntplib} module now supports IPv6 addresses.
(Contributed by Derek Morr; bpo-1664@footnote{https://bugs.python.org/issue1664}.)

@item 
New functions: the @ref{c0,,os} module wraps the following POSIX system
calls: @ref{eac,,getresgid()} and @ref{ead,,getresuid()}, which return the
real, effective, and saved GIDs and UIDs;
@ref{eae,,setresgid()} and @ref{eaf,,setresuid()}, which set
real, effective, and saved GIDs and UIDs to new values;
@ref{eb0,,initgroups()}, which initialize the group access list
for the current process.  (GID/UID functions
contributed by Travis H.; bpo-6508@footnote{https://bugs.python.org/issue6508}.  Support for initgroups added
by Jean-Paul Calderone; bpo-7333@footnote{https://bugs.python.org/issue7333}.)

The @ref{bb9,,os.fork()} function now re-initializes the import lock in
the child process; this fixes problems on Solaris when @ref{bb9,,fork()}
is called from a thread.  (Fixed by Zsolt Cserna; bpo-7242@footnote{https://bugs.python.org/issue7242}.)

@item 
In the @ref{c1,,os.path} module, the @ref{eb1,,normpath()} and
@ref{eb2,,abspath()} functions now preserve Unicode; if their input path
is a Unicode string, the return value is also a Unicode string.
(@ref{eb1,,normpath()} fixed by Matt Giuca in bpo-5827@footnote{https://bugs.python.org/issue5827};
@ref{eb2,,abspath()} fixed by Ezio Melotti in bpo-3426@footnote{https://bugs.python.org/issue3426}.)

@item 
The @ref{d4,,pydoc} module now has help for the various symbols that Python
uses.  You can now do @code{help('<<')} or @code{help('@@')}, for example.
(Contributed by David Laban; bpo-4739@footnote{https://bugs.python.org/issue4739}.)

@item 
The @ref{d8,,re} module’s @ref{699,,split()}, @ref{72e,,sub()}, and @ref{9b3,,subn()}
now accept an optional @emph{flags} argument, for consistency with the
other functions in the module.  (Added by Gregory P. Smith.)

@item 
New function: @ref{eb3,,run_path()} in the @ref{dd,,runpy} module
will execute the code at a provided @emph{path} argument.  @emph{path} can be
the path of a Python source file (@code{example.py}), a compiled
bytecode file (@code{example.pyc}), a directory
(@code{./package/}), or a zip archive (@code{example.zip}).  If a
directory or zip path is provided, it will be added to the front of
@code{sys.path} and the module @ref{1,,__main__} will be imported.  It’s
expected that the directory or zip contains a @code{__main__.py};
if it doesn’t, some other @code{__main__.py} might be imported from
a location later in @code{sys.path}.  This makes more of the machinery
of @ref{dd,,runpy} available to scripts that want to mimic the way
Python’s command line processes an explicit path name.
(Added by Nick Coghlan; bpo-6816@footnote{https://bugs.python.org/issue6816}.)

@item 
New function: in the @ref{e4,,shutil} module, @ref{539,,make_archive()}
takes a filename, archive type (zip or tar-format), and a directory
path, and creates an archive containing the directory’s contents.
(Added by Tarek Ziadé.)

@ref{e4,,shutil}’s @ref{56f,,copyfile()} and @ref{538,,copytree()}
functions now raise a @code{SpecialFileError} exception when
asked to copy a named pipe.  Previously the code would treat
named pipes like a regular file by opening them for reading, and
this would block indefinitely.  (Fixed by Antoine Pitrou; bpo-3002@footnote{https://bugs.python.org/issue3002}.)

@item 
The @ref{e5,,signal} module no longer re-installs the signal handler
unless this is truly necessary, which fixes a bug that could make it
impossible to catch the EINTR signal robustly.  (Fixed by
Charles-Francois Natali; bpo-8354@footnote{https://bugs.python.org/issue8354}.)

@item 
New functions: in the @ref{e6,,site} module, three new functions
return various site- and user-specific paths.
@ref{df0,,getsitepackages()} returns a list containing all
global site-packages directories,
@ref{df2,,getusersitepackages()} returns the path of the user’s
site-packages directory, and
@ref{df1,,getuserbase()} returns the value of the 
@geindex USER_BASE
@geindex environment variable; USER_BASE
@code{USER_BASE}
environment variable, giving the path to a directory that can be used
to store data.
(Contributed by Tarek Ziadé; bpo-6693@footnote{https://bugs.python.org/issue6693}.)

The @ref{e6,,site} module now reports exceptions occurring
when the @code{sitecustomize} module is imported, and will no longer
catch and swallow the @ref{4ca,,KeyboardInterrupt} exception.  (Fixed by
Victor Stinner; bpo-3137@footnote{https://bugs.python.org/issue3137}.)

@item 
The @ref{dc0,,create_connection()} function
gained a @emph{source_address} parameter, a @code{(host, port)} 2-tuple
giving the source address that will be used for the connection.
(Contributed by Eldon Ziegler; bpo-3972@footnote{https://bugs.python.org/issue3972}.)

The @ref{eb4,,recv_into()} and @ref{eb5,,recvfrom_into()}
methods will now write into objects that support the buffer API, most usefully
the @ref{2f1,,bytearray} and @ref{573,,memoryview} objects.  (Implemented by
Antoine Pitrou; bpo-8104@footnote{https://bugs.python.org/issue8104}.)

@item 
The @code{SocketServer} module’s @code{TCPServer} class now
supports socket timeouts and disabling the Nagle algorithm.
The @code{disable_nagle_algorithm} class attribute
defaults to @code{False}; if overridden to be true,
new request connections will have the TCP_NODELAY option set to
prevent buffering many small sends into a single TCP packet.
The @code{timeout} class attribute can hold
a timeout in seconds that will be applied to the request socket; if
no request is received within that time, @code{handle_timeout()}
will be called and @code{handle_request()} will return.
(Contributed by Kristján Valur Jónsson; bpo-6192@footnote{https://bugs.python.org/issue6192} and bpo-6267@footnote{https://bugs.python.org/issue6267}.)

@item 
Updated module: the @ref{ed,,sqlite3} module has been updated to
version 2.6.0 of the pysqlite package@footnote{https://github.com/ghaering/pysqlite}. Version 2.6.0 includes a number of bugfixes, and adds
the ability to load SQLite extensions from shared libraries.
Call the @code{enable_load_extension(True)} method to enable extensions,
and then call @ref{2c4,,load_extension()} to load a particular shared library.
(Updated by Gerhard Häring.)

@item 
The @ref{ee,,ssl} module’s @ref{3d6,,SSLSocket} objects now support the
buffer API, which fixed a test suite failure (fix by Antoine Pitrou;
bpo-7133@footnote{https://bugs.python.org/issue7133}) and automatically set
OpenSSL’s @code{SSL_MODE_AUTO_RETRY}, which will prevent an error
code being returned from @code{recv()} operations that trigger an SSL
renegotiation (fix by Antoine Pitrou; bpo-8222@footnote{https://bugs.python.org/issue8222}).

The @ref{331,,ssl.wrap_socket()} constructor function now takes a
@emph{ciphers} argument that’s a string listing the encryption algorithms
to be allowed; the format of the string is described
in the OpenSSL documentation@footnote{https://www.openssl.org/docs/manmaster/man1/ciphers.html#CIPHER-LIST-FORMAT}.
(Added by Antoine Pitrou; bpo-8322@footnote{https://bugs.python.org/issue8322}.)

Another change makes the extension load all of OpenSSL’s ciphers and
digest algorithms so that they’re all available.  Some SSL
certificates couldn’t be verified, reporting an “unknown algorithm”
error.  (Reported by Beda Kosata, and fixed by Antoine Pitrou;
bpo-8484@footnote{https://bugs.python.org/issue8484}.)

The version of OpenSSL being used is now available as the module
attributes @ref{dc2,,ssl.OPENSSL_VERSION} (a string),
@ref{dc3,,ssl.OPENSSL_VERSION_INFO} (a 5-tuple), and
@ref{dc4,,ssl.OPENSSL_VERSION_NUMBER} (an integer).  (Added by Antoine
Pitrou; bpo-8321@footnote{https://bugs.python.org/issue8321}.)

@item 
The @ref{f3,,struct} module will no longer silently ignore overflow
errors when a value is too large for a particular integer format
code (one of @code{bBhHiIlLqQ}); it now always raises a
@ref{eb6,,struct.error} exception.  (Changed by Mark Dickinson;
bpo-1523@footnote{https://bugs.python.org/issue1523}.)  The @ref{e25,,pack()} function will also
attempt to use @code{__index__()} to convert and pack non-integers
before trying the @code{__int__()} method or reporting an error.
(Changed by Mark Dickinson; bpo-8300@footnote{https://bugs.python.org/issue8300}.)

@item 
New function: the @ref{f4,,subprocess} module’s
@ref{b43,,check_output()} runs a command with a specified set of arguments
and returns the command’s output as a string when the command runs without
error, or raises a @ref{eb7,,CalledProcessError} exception otherwise.

@example
>>> subprocess.check_output(['df', '-h', '.'])
'Filesystem     Size   Used  Avail Capacity  Mounted on\n
/dev/disk0s2    52G    49G   3.0G    94%    /\n'

>>> subprocess.check_output(['df', '-h', '/bogus'])
  ...
subprocess.CalledProcessError: Command '['df', '-h', '/bogus']' returned non-zero exit status 1
@end example

(Contributed by Gregory P. Smith.)

The @ref{f4,,subprocess} module will now retry its internal system calls
on receiving an @code{EINTR} signal.  (Reported by several people; final
patch by Gregory P. Smith in bpo-1068268@footnote{https://bugs.python.org/issue1068268}.)

@item 
New function: @ref{eb8,,is_declared_global()} in the @ref{f6,,symtable} module
returns true for variables that are explicitly declared to be global,
false for ones that are implicitly global.
(Contributed by Jeremy Hylton.)

@item 
The @ref{f9,,syslog} module will now use the value of @code{sys.argv[0]} as the
identifier instead of the previous default value of @code{'python'}.
(Changed by Sean Reifschneider; bpo-8451@footnote{https://bugs.python.org/issue8451}.)

@item 
The @code{sys.version_info} value is now a named tuple, with attributes
named @code{major}, @code{minor}, @code{micro},
@code{releaselevel}, and @code{serial}.  (Contributed by Ross
Light; bpo-4285@footnote{https://bugs.python.org/issue4285}.)

@ref{830,,sys.getwindowsversion()} also returns a named tuple,
with attributes named @code{major}, @code{minor}, @code{build},
@ref{c9,,platform}, @code{service_pack}, @code{service_pack_major},
@code{service_pack_minor}, @code{suite_mask}, and
@code{product_type}.  (Contributed by Brian Curtin; bpo-7766@footnote{https://bugs.python.org/issue7766}.)

@item 
The @ref{fb,,tarfile} module’s default error handling has changed, to
no longer suppress fatal errors.  The default error level was previously 0,
which meant that errors would only result in a message being written to the
debug log, but because the debug log is not activated by default,
these errors go unnoticed.  The default error level is now 1,
which raises an exception if there’s an error.
(Changed by Lars Gustäbel; bpo-7357@footnote{https://bugs.python.org/issue7357}.)

@ref{fb,,tarfile} now supports filtering the @ref{db2,,TarInfo}
objects being added to a tar file.  When you call @ref{72f,,add()},
you may supply an optional @emph{filter} argument
that’s a callable.  The @emph{filter} callable will be passed the
@ref{db2,,TarInfo} for every file being added, and can modify and return it.
If the callable returns @code{None}, the file will be excluded from the
resulting archive.  This is more powerful than the existing
@emph{exclude} argument, which has therefore been deprecated.
(Added by Lars Gustäbel; bpo-6856@footnote{https://bugs.python.org/issue6856}.)
The @ref{db1,,TarFile} class also now supports the context management protocol.
(Added by Lars Gustäbel; bpo-7232@footnote{https://bugs.python.org/issue7232}.)

@item 
The @ref{eb9,,wait()} method of the @ref{cf3,,threading.Event} class
now returns the internal flag on exit.  This means the method will usually
return true because @ref{eb9,,wait()} is supposed to block until the
internal flag becomes true.  The return value will only be false if
a timeout was provided and the operation timed out.
(Contributed by Tim Lesher; bpo-1674032@footnote{https://bugs.python.org/issue1674032}.)

@item 
The Unicode database provided by the @ref{121,,unicodedata} module is
now used internally to determine which characters are numeric,
whitespace, or represent line breaks.  The database also
includes information from the @code{Unihan.txt} data file (patch
by Anders Chrigström and Amaury Forgeot d’Arc; bpo-1571184@footnote{https://bugs.python.org/issue1571184})
and has been updated to version 5.2.0 (updated by
Florent Xicluna; bpo-8024@footnote{https://bugs.python.org/issue8024}).

@item 
The @code{urlparse} module’s @code{urlsplit()} now handles
unknown URL schemes in a fashion compliant with 
@geindex RFC; RFC 3986
RFC 3986@footnote{https://tools.ietf.org/html/rfc3986.html}: if the
URL is of the form @code{"<something>://..."}, the text before the
@code{://} is treated as the scheme, even if it’s a made-up scheme that
the module doesn’t know about.  This change may break code that
worked around the old behaviour.  For example, Python 2.6.4 or 2.5
will return the following:

@example
>>> import urlparse
>>> urlparse.urlsplit('invented://host/filename?query')
('invented', '@w{'}, '//host/filename?query', '@w{'}, '@w{'})
@end example

Python 2.7 (and Python 2.6.5) will return:

@example
>>> import urlparse
>>> urlparse.urlsplit('invented://host/filename?query')
('invented', 'host', '/filename?query', '@w{'}, '@w{'})
@end example

(Python 2.7 actually produces slightly different output, since it
returns a named tuple instead of a standard tuple.)

The @code{urlparse} module also supports IPv6 literal addresses as defined by
@geindex RFC; RFC 2732
RFC 2732@footnote{https://tools.ietf.org/html/rfc2732.html} (contributed by Senthil Kumaran; bpo-2987@footnote{https://bugs.python.org/issue2987}).

@example
>>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo')
ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]',
            path='/foo', params='@w{'}, query='@w{'}, fragment='@w{'})
@end example

@item 
New class: the @ref{eba,,WeakSet} class in the @ref{12f,,weakref}
module is a set that only holds weak references to its elements; elements
will be removed once there are no references pointing to them.
(Originally implemented in Python 3.x by Raymond Hettinger, and backported
to 2.7 by Michael Foord.)

@item 
The ElementTree library, @code{xml.etree}, no longer escapes
ampersands and angle brackets when outputting an XML processing
instruction (which looks like @code{<?xml-stylesheet href="#style1"?>})
or comment (which looks like @code{<!-- comment -->}).
(Patch by Neil Muller; bpo-2746@footnote{https://bugs.python.org/issue2746}.)

@item 
The XML-RPC client and server, provided by the @code{xmlrpclib} and
@code{SimpleXMLRPCServer} modules, have improved performance by
supporting HTTP/1.1 keep-alive and by optionally using gzip encoding
to compress the XML being exchanged.  The gzip compression is
controlled by the @code{encode_threshold} attribute of
@code{SimpleXMLRPCRequestHandler}, which contains a size in bytes;
responses larger than this will be compressed.
(Contributed by Kristján Valur Jónsson; bpo-6267@footnote{https://bugs.python.org/issue6267}.)

@item 
The @ref{149,,zipfile} module’s @ref{6e5,,ZipFile} now supports the context
management protocol, so you can write @code{with zipfile.ZipFile(...) as f:}.
(Contributed by Brian Curtin; bpo-5511@footnote{https://bugs.python.org/issue5511}.)

@ref{149,,zipfile} now also supports archiving empty directories and
extracts them correctly.  (Fixed by Kuba Wieczorek; bpo-4710@footnote{https://bugs.python.org/issue4710}.)
Reading files out of an archive is faster, and interleaving
@ref{ebb,,read()} and @code{readline()} now works correctly.
(Contributed by Nir Aides; bpo-7610@footnote{https://bugs.python.org/issue7610}.)

The @ref{ebc,,is_zipfile()} function now
accepts a file object, in addition to the path names accepted in earlier
versions.  (Contributed by Gabriel Genellina; bpo-4756@footnote{https://bugs.python.org/issue4756}.)

The @ref{ebd,,writestr()} method now has an optional @emph{compress_type} parameter
that lets you override the default compression method specified in the
@ref{6e5,,ZipFile} constructor.  (Contributed by Ronald Oussoren;
bpo-6003@footnote{https://bugs.python.org/issue6003}.)
@end itemize

@c ======================================================================

@c whole new modules get described in subsections here

@menu
* New module; importlib: New module importlib. 
* New module; sysconfig: New module sysconfig. 
* ttk; Themed Widgets for Tk: ttk Themed Widgets for Tk. 
* Updated module; unittest: Updated module unittest. 
* Updated module; ElementTree 1.3: Updated module ElementTree 1 3. 

@end menu

@node New module importlib,New module sysconfig,,New and Improved Modules
@anchor{whatsnew/2 7 importlib-section}@anchor{e89}@anchor{whatsnew/2 7 new-module-importlib}@anchor{ebe}
@subsubsection New module: importlib


Python 3.1 includes the @ref{96,,importlib} package, a re-implementation
of the logic underlying Python’s @ref{e35,,import} statement.
@ref{96,,importlib} is useful for implementors of Python interpreters and
to users who wish to write new importers that can participate in the
import process.  Python 2.7 doesn’t contain the complete
@ref{96,,importlib} package, but instead has a tiny subset that contains
a single function, @ref{d3e,,import_module()}.

@code{import_module(name, package=None)} imports a module.  @emph{name} is
a string containing the module or package’s name.  It’s possible to do
relative imports by providing a string that begins with a @code{.}
character, such as @code{..utils.errors}.  For relative imports, the
@emph{package} argument must be provided and is the name of the package that
will be used as the anchor for
the relative import.  @ref{d3e,,import_module()} both inserts the imported
module into @code{sys.modules} and returns the module object.

Here are some examples:

@example
>>> from importlib import import_module
>>> anydbm = import_module('anydbm')  # Standard absolute import
>>> anydbm
<module 'anydbm' from '/p/python/Lib/anydbm.py'>
>>> # Relative import
>>> file_util = import_module('..file_util', 'distutils.command')
>>> file_util
<module 'distutils.file_util' from '/python/Lib/distutils/file_util.pyc'>
@end example

@ref{96,,importlib} was implemented by Brett Cannon and introduced in
Python 3.1.

@node New module sysconfig,ttk Themed Widgets for Tk,New module importlib,New and Improved Modules
@anchor{whatsnew/2 7 new-module-sysconfig}@anchor{ebf}
@subsubsection New module: sysconfig


The @ref{f8,,sysconfig} module has been pulled out of the Distutils
package, becoming a new top-level module in its own right.
@ref{f8,,sysconfig} provides functions for getting information about
Python’s build process: compiler switches, installation paths, the
platform name, and whether Python is running from its source
directory.

Some of the functions in the module are:


@itemize *

@item 
@ref{bbc,,get_config_var()} returns variables from Python’s
Makefile and the @code{pyconfig.h} file.

@item 
@ref{bbd,,get_config_vars()} returns a dictionary containing
all of the configuration variables.

@item 
@ref{ec0,,get_path()} returns the configured path for
a particular type of module: the standard library,
site-specific modules, platform-specific modules, etc.

@item 
@ref{ec1,,is_python_build()} returns true if you’re running a
binary from a Python source tree, and false otherwise.
@end itemize

Consult the @ref{f8,,sysconfig} documentation for more details and for
a complete list of functions.

The Distutils package and @ref{f8,,sysconfig} are now maintained by Tarek
Ziadé, who has also started a Distutils2 package (source repository at
@indicateurl{https://hg.python.org/distutils2/}) for developing a next-generation
version of Distutils.

@node ttk Themed Widgets for Tk,Updated module unittest,New module sysconfig,New and Improved Modules
@anchor{whatsnew/2 7 ttk-themed-widgets-for-tk}@anchor{ec2}
@subsubsection ttk: Themed Widgets for Tk


Tcl/Tk 8.5 includes a set of themed widgets that re-implement basic Tk
widgets but have a more customizable appearance and can therefore more
closely resemble the native platform’s widgets.  This widget
set was originally called Tile, but was renamed to Ttk (for “themed Tk”)
on being added to Tcl/Tck release 8.5.

To learn more, read the @code{ttk} module documentation.  You may also
wish to read the Tcl/Tk manual page describing the
Ttk theme engine, available at
@indicateurl{https://www.tcl.tk/man/tcl8.5/TkCmd/ttk_intro.htm}. Some
screenshots of the Python/Ttk code in use are at
@indicateurl{https://code.google.com/archive/p/python-ttk/wikis/Screenshots.wiki}.

The @code{ttk} module was written by Guilherme Polo and added in
bpo-2983@footnote{https://bugs.python.org/issue2983}.  An alternate version called @code{Tile.py}, written by
Martin Franklin and maintained by Kevin Walzer, was proposed for
inclusion in bpo-2618@footnote{https://bugs.python.org/issue2618}, but the authors argued that Guilherme
Polo’s work was more comprehensive.

@node Updated module unittest,Updated module ElementTree 1 3,ttk Themed Widgets for Tk,New and Improved Modules
@anchor{whatsnew/2 7 unittest-section}@anchor{ec3}@anchor{whatsnew/2 7 updated-module-unittest}@anchor{ec4}
@subsubsection Updated module: unittest


The @ref{122,,unittest} module was greatly enhanced; many
new features were added.  Most of these features were implemented
by Michael Foord, unless otherwise noted.  The enhanced version of
the module is downloadable separately for use with Python versions 2.4 to 2.6,
packaged as the @code{unittest2} package, from
@indicateurl{https://pypi.org/project/unittest2}.

When used from the command line, the module can automatically discover
tests.  It’s not as fancy as py.test@footnote{http://pytest.org} or
nose@footnote{https://nose.readthedocs.io/}, but provides a
simple way to run tests kept within a set of package directories.  For example,
the following command will search the @code{test/} subdirectory for
any importable test files named @code{test*.py}:

@example
python -m unittest discover -s test
@end example

Consult the @ref{122,,unittest} module documentation for more details.
(Developed in bpo-6001@footnote{https://bugs.python.org/issue6001}.)

The @ref{b69,,main()} function supports some other new options:


@itemize *

@item 
@ref{ec5,,-b} or @code{--buffer} will buffer the standard output
and standard error streams during each test.  If the test passes,
any resulting output will be discarded; on failure, the buffered
output will be displayed.

@item 
@ref{ec6,,-c} or @code{--catch} will cause the control-C interrupt
to be handled more gracefully.  Instead of interrupting the test
process immediately, the currently running test will be completed
and then the partial results up to the interruption will be reported.
If you’re impatient, a second press of control-C will cause an immediate
interruption.

This control-C handler tries to avoid causing problems when the code
being tested or the tests being run have defined a signal handler of
their own, by noticing that a signal handler was already set and
calling it.  If this doesn’t work for you, there’s a
@ref{ec7,,removeHandler()} decorator that can be used to mark tests that
should have the control-C handling disabled.

@item 
@ref{ec8,,-f} or @code{--failfast} makes
test execution stop immediately when a test fails instead of
continuing to execute further tests.  (Suggested by Cliff Dyer and
implemented by Michael Foord; bpo-8074@footnote{https://bugs.python.org/issue8074}.)
@end itemize

The progress messages now show ‘x’ for expected failures
and ‘u’ for unexpected successes when run in verbose mode.
(Contributed by Benjamin Peterson.)

Test cases can raise the @ref{b6a,,SkipTest} exception to skip a
test (bpo-1034053@footnote{https://bugs.python.org/issue1034053}).

The error messages for @ref{dd5,,assertEqual()},
@ref{dd4,,assertTrue()}, and @ref{ec9,,assertFalse()}
failures now provide more information.  If you set the
@ref{eca,,longMessage} attribute of your @ref{1ab,,TestCase} classes to
true, both the standard error message and any additional message you
provide will be printed for failures.  (Added by Michael Foord; bpo-5663@footnote{https://bugs.python.org/issue5663}.)

The @ref{cff,,assertRaises()} method now
returns a context handler when called without providing a callable
object to run.  For example, you can write this:

@example
with self.assertRaises(KeyError):
    @{@}['foo']
@end example

(Implemented by Antoine Pitrou; bpo-4444@footnote{https://bugs.python.org/issue4444}.)

@c rev 78774

Module- and class-level setup and teardown fixtures are now supported.
Modules can contain @code{setUpModule()} and @code{tearDownModule()}
functions.  Classes can have @ref{564,,setUpClass()} and
@ref{ecb,,tearDownClass()} methods that must be defined as class methods
(using @code{@@classmethod} or equivalent).  These functions and
methods are invoked when the test runner switches to a test case in a
different module or class.

The methods @ref{599,,addCleanup()} and
@ref{ecc,,doCleanups()} were added.
@ref{599,,addCleanup()} lets you add cleanup functions that
will be called unconditionally (after @ref{ecd,,setUp()} if
@ref{ecd,,setUp()} fails, otherwise after @ref{ece,,tearDown()}). This allows
for much simpler resource allocation and deallocation during tests
(bpo-5679@footnote{https://bugs.python.org/issue5679}).

A number of new methods were added that provide more specialized
tests.  Many of these methods were written by Google engineers
for use in their test suites; Gregory P. Smith, Michael Foord, and
GvR worked on merging them into Python’s version of @ref{122,,unittest}.


@itemize *

@item 
@ref{ecf,,assertIsNone()} and @ref{ed0,,assertIsNotNone()} take one
expression and verify that the result is or is not @code{None}.

@item 
@ref{ed1,,assertIs()} and @ref{ed2,,assertIsNot()}
take two values and check whether the two values evaluate to the same object or not.
(Added by Michael Foord; bpo-2578@footnote{https://bugs.python.org/issue2578}.)

@item 
@ref{ed3,,assertIsInstance()} and
@ref{ed4,,assertNotIsInstance()} check whether
the resulting object is an instance of a particular class, or of
one of a tuple of classes.  (Added by Georg Brandl; bpo-7031@footnote{https://bugs.python.org/issue7031}.)

@item 
@ref{ed5,,assertGreater()}, @ref{ed6,,assertGreaterEqual()},
@ref{ed7,,assertLess()}, and @ref{ed8,,assertLessEqual()} compare
two quantities.

@item 
@ref{ed9,,assertMultiLineEqual()} compares two strings, and if they’re
not equal, displays a helpful comparison that highlights the
differences in the two strings.  This comparison is now used by
default when Unicode strings are compared with @ref{dd5,,assertEqual()}.

@item 
@code{assertRegexpMatches()} and
@code{assertNotRegexpMatches()} checks whether the
first argument is a string matching or not matching the regular
expression provided as the second argument (bpo-8038@footnote{https://bugs.python.org/issue8038}).

@item 
@code{assertRaisesRegexp()} checks whether a particular exception
is raised, and then also checks that the string representation of
the exception matches the provided regular expression.

@item 
@ref{eda,,assertIn()} and @ref{edb,,assertNotIn()}
tests whether @emph{first} is or is not in  @emph{second}.

@item 
@code{assertItemsEqual()} tests whether two provided sequences
contain the same elements.

@item 
@ref{edc,,assertSetEqual()} compares whether two sets are equal, and
only reports the differences between the sets in case of error.

@item 
Similarly, @ref{edd,,assertListEqual()} and @ref{ede,,assertTupleEqual()}
compare the specified types and explain any differences without necessarily
printing their full values; these methods are now used by default
when comparing lists and tuples using @ref{dd5,,assertEqual()}.
More generally, @ref{edf,,assertSequenceEqual()} compares two sequences
and can optionally check whether both sequences are of a
particular type.

@item 
@ref{ee0,,assertDictEqual()} compares two dictionaries and reports the
differences; it’s now used by default when you compare two dictionaries
using @ref{dd5,,assertEqual()}.  @code{assertDictContainsSubset()} checks whether
all of the key/value pairs in @emph{first} are found in @emph{second}.

@item 
@ref{dd7,,assertAlmostEqual()} and @ref{dd8,,assertNotAlmostEqual()} test
whether @emph{first} and @emph{second} are approximately equal.  This method
can either round their difference to an optionally-specified number
of @emph{places} (the default is 7) and compare it to zero, or require
the difference to be smaller than a supplied @emph{delta} value.

@item 
@ref{ee1,,loadTestsFromName()} properly honors the
@ref{ee2,,suiteClass} attribute of
the @ref{1ad,,TestLoader}. (Fixed by Mark Roddy; bpo-6866@footnote{https://bugs.python.org/issue6866}.)

@item 
A new hook lets you extend the @ref{dd5,,assertEqual()} method to handle
new data types.  The @ref{ee3,,addTypeEqualityFunc()} method takes a type
object and a function. The function will be used when both of the
objects being compared are of the specified type.  This function
should compare the two objects and raise an exception if they don’t
match; it’s a good idea for the function to provide additional
information about why the two objects aren’t matching, much as the new
sequence comparison methods do.
@end itemize

@ref{b69,,unittest.main()} now takes an optional @code{exit} argument.  If
false, @ref{b69,,main()} doesn’t call @ref{ee4,,sys.exit()}, allowing
@ref{b69,,main()} to be used from the interactive interpreter.
(Contributed by J. Pablo Fernández; bpo-3379@footnote{https://bugs.python.org/issue3379}.)

@ref{d04,,TestResult} has new @ref{ee5,,startTestRun()} and
@ref{ee6,,stopTestRun()} methods that are called immediately before
and after a test run.  (Contributed by Robert Collins; bpo-5728@footnote{https://bugs.python.org/issue5728}.)

With all these changes, the @code{unittest.py} was becoming awkwardly
large, so the module was turned into a package and the code split into
several files (by Benjamin Peterson).  This doesn’t affect how the
module is imported or used.


@subsubheading See also



@table @asis

@item @indicateurl{http://www.voidspace.org.uk/python/articles/unittest2.shtml}

Describes the new features, how to use them, and the
rationale for various design decisions.  (By Michael Foord.)
@end table


@node Updated module ElementTree 1 3,,Updated module unittest,New and Improved Modules
@anchor{whatsnew/2 7 elementtree-section}@anchor{ee7}@anchor{whatsnew/2 7 updated-module-elementtree-1-3}@anchor{ee8}
@subsubsection Updated module: ElementTree 1.3


The version of the ElementTree library included with Python was updated to
version 1.3.  Some of the new features are:


@itemize *

@item 
The various parsing functions now take a @emph{parser} keyword argument
giving an @ref{569,,XMLParser} instance that will
be used.  This makes it possible to override the file’s internal encoding:

@example
p = ET.XMLParser(encoding='utf-8')
t = ET.XML("""<root/>""", parser=p)
@end example

Errors in parsing XML now raise a @code{ParseError} exception, whose
instances have a @code{position} attribute
containing a (@emph{line}, @emph{column}) tuple giving the location of the problem.

@item 
ElementTree’s code for converting trees to a string has been
significantly reworked, making it roughly twice as fast in many
cases.  The @ref{b7d,,ElementTree.write()}
and @code{Element.write()} methods now have a @emph{method} parameter that can be
“xml” (the default), “html”, or “text”.  HTML mode will output empty
elements as @code{<empty></empty>} instead of @code{<empty/>}, and text
mode will skip over elements and only output the text chunks.  If
you set the @code{tag} attribute of an element to @code{None} but
leave its children in place, the element will be omitted when the
tree is written out, so you don’t need to do more extensive rearrangement
to remove a single element.

Namespace handling has also been improved.  All @code{xmlns:<whatever>}
declarations are now output on the root element, not scattered throughout
the resulting XML.  You can set the default namespace for a tree
by setting the @code{default_namespace} attribute and can
register new prefixes with @ref{d76,,register_namespace()}.  In XML mode,
you can use the true/false @emph{xml_declaration} parameter to suppress the
XML declaration.

@item 
New @ref{44e,,Element} method:
@ref{d77,,extend()} appends the items from a
sequence to the element’s children.  Elements themselves behave like
sequences, so it’s easy to move children from one element to
another:

@example
from xml.etree import ElementTree as ET

t = ET.XML("""<list>
  <item>1</item> <item>2</item>  <item>3</item>
</list>""")
new = ET.XML('<root/>')
new.extend(t)

# Outputs <root><item>1</item>...</root>
print ET.tostring(new)
@end example

@item 
New @code{Element} method:
@ref{ee9,,iter()} yields the children of the
element as a generator.  It’s also possible to write @code{for child in
elem:} to loop over an element’s children.  The existing method
@code{getiterator()} is now deprecated, as is @code{getchildren()}
which constructs and returns a list of children.

@item 
New @code{Element} method:
@ref{d79,,itertext()} yields all chunks of
text that are descendants of the element.  For example:

@example
t = ET.XML("""<list>
  <item>1</item> <item>2</item>  <item>3</item>
</list>""")

# Outputs ['\n  ', '1', ' ', '2', '  ', '3', '\n']
print list(t.itertext())
@end example

@item 
Deprecated: using an element as a Boolean (i.e., @code{if elem:}) would
return true if the element had any children, or false if there were
no children.  This behaviour is confusing – @code{None} is false, but
so is a childless element? – so it will now trigger a
@ref{604,,FutureWarning}.  In your code, you should be explicit: write
@code{len(elem) != 0} if you’re interested in the number of children,
or @code{elem is not None}.
@end itemize

Fredrik Lundh develops ElementTree and produced the 1.3 version;
you can read his article describing 1.3 at
@indicateurl{http://effbot.org/zone/elementtree-13-intro.htm}.
Florent Xicluna updated the version included with
Python, after discussions on python-dev and in bpo-6472@footnote{https://bugs.python.org/issue6472}.)

@c ======================================================================

@node Build and C API Changes<8>,Other Changes and Fixes,New and Improved Modules,What’s New in Python 2 7
@anchor{whatsnew/2 7 build-and-c-api-changes}@anchor{eea}
@subsection Build and C API Changes


Changes to Python’s build process and to the C API include:


@itemize *

@item 
The latest release of the GNU Debugger, GDB 7, can be scripted using Python@footnote{https://sourceware.org/gdb/current/onlinedocs/gdb/Python.html}.
When you begin debugging an executable program P, GDB will look for
a file named @code{P-gdb.py} and automatically read it.  Dave Malcolm
contributed a @code{python-gdb.py} that adds a number of
commands useful when debugging Python itself.  For example,
@code{py-up} and @code{py-down} go up or down one Python stack frame,
which usually corresponds to several C stack frames.  @code{py-print}
prints the value of a Python variable, and @code{py-bt} prints the
Python stack trace.  (Added as a result of bpo-8032@footnote{https://bugs.python.org/issue8032}.)

@item 
If you use the @code{.gdbinit} file provided with Python,
the “pyo” macro in the 2.7 version now works correctly when the thread being
debugged doesn’t hold the GIL; the macro now acquires it before printing.
(Contributed by Victor Stinner; bpo-3632@footnote{https://bugs.python.org/issue3632}.)

@item 
@ref{488,,Py_AddPendingCall()} is now thread-safe, letting any
worker thread submit notifications to the main Python thread.  This
is particularly useful for asynchronous IO operations.
(Contributed by Kristján Valur Jónsson; bpo-4293@footnote{https://bugs.python.org/issue4293}.)

@item 
New function: @ref{eeb,,PyCode_NewEmpty()} creates an empty code object;
only the filename, function name, and first line number are required.
This is useful for extension modules that are attempting to
construct a more useful traceback stack.  Previously such
extensions needed to call @ref{584,,PyCode_New()}, which had many
more arguments.  (Added by Jeffrey Yasskin.)

@item 
New function: @ref{e19,,PyErr_NewExceptionWithDoc()} creates a new
exception class, just as the existing @ref{e1a,,PyErr_NewException()} does,
but takes an extra @code{char *} argument containing the docstring for the
new exception class.  (Added by ‘lekma’ on the Python bug tracker;
bpo-7033@footnote{https://bugs.python.org/issue7033}.)

@item 
New function: @ref{1f9,,PyFrame_GetLineNumber()} takes a frame object
and returns the line number that the frame is currently executing.
Previously code would need to get the index of the bytecode
instruction currently executing, and then look up the line number
corresponding to that address.  (Added by Jeffrey Yasskin.)

@item 
New functions: @ref{e17,,PyLong_AsLongAndOverflow()} and
@ref{e16,,PyLong_AsLongLongAndOverflow()}  approximates a Python long
integer as a C long or long  long.
If the number is too large to fit into
the output type, an @emph{overflow} flag is set and returned to the caller.
(Contributed by Case Van Horsen; bpo-7528@footnote{https://bugs.python.org/issue7528} and bpo-7767@footnote{https://bugs.python.org/issue7767}.)

@item 
New function: stemming from the rewrite of string-to-float conversion,
a new @ref{e3b,,PyOS_string_to_double()} function was added.  The old
@code{PyOS_ascii_strtod()} and @code{PyOS_ascii_atof()} functions
are now deprecated.

@item 
New function: @ref{e14,,PySys_SetArgvEx()} sets the value of
@code{sys.argv} and can optionally update @code{sys.path} to include the
directory containing the script named by @code{sys.argv[0]} depending
on the value of an @emph{updatepath} parameter.

This function was added to close a security hole for applications
that embed Python.  The old function, @ref{eec,,PySys_SetArgv()}, would
always update @code{sys.path}, and sometimes it would add the current
directory.  This meant that, if you ran an application embedding
Python in a directory controlled by someone else, attackers could
put a Trojan-horse module in the directory (say, a file named
@code{os.py}) that your application would then import and run.

If you maintain a C/C++ application that embeds Python, check
whether you’re calling @ref{eec,,PySys_SetArgv()} and carefully consider
whether the application should be using @ref{e14,,PySys_SetArgvEx()}
with @emph{updatepath} set to false.

Security issue reported as CVE-2008-5983@footnote{https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5983};
discussed in bpo-5753@footnote{https://bugs.python.org/issue5753}, and fixed by Antoine Pitrou.

@item 
New macros: the Python header files now define the following macros:
@code{Py_ISALNUM},
@code{Py_ISALPHA},
@code{Py_ISDIGIT},
@code{Py_ISLOWER},
@code{Py_ISSPACE},
@code{Py_ISUPPER},
@code{Py_ISXDIGIT},
@code{Py_TOLOWER}, and @code{Py_TOUPPER}.
All of these functions are analogous to the C
standard macros for classifying characters, but ignore the current
locale setting, because in
several places Python needs to analyze characters in a
locale-independent way.  (Added by Eric Smith;
bpo-5793@footnote{https://bugs.python.org/issue5793}.)

@c XXX these macros don't seem to be described in the c-api docs.

@item 
Removed function: @code{PyEval_CallObject} is now only available
as a macro.  A function version was being kept around to preserve
ABI linking compatibility, but that was in 1997; it can certainly be
deleted by now.  (Removed by Antoine Pitrou; bpo-8276@footnote{https://bugs.python.org/issue8276}.)

@item 
New format codes: the @code{PyFormat_FromString()},
@code{PyFormat_FromStringV()}, and @ref{a23,,PyErr_Format()} functions now
accept @code{%lld} and @code{%llu} format codes for displaying
C’s long  long types.
(Contributed by Mark Dickinson; bpo-7228@footnote{https://bugs.python.org/issue7228}.)

@item 
The complicated interaction between threads and process forking has
been changed.  Previously, the child process created by
@ref{bb9,,os.fork()} might fail because the child is created with only a
single thread running, the thread performing the @ref{bb9,,os.fork()}.
If other threads were holding a lock, such as Python’s import lock,
when the fork was performed, the lock would still be marked as
“held” in the new process.  But in the child process nothing would
ever release the lock, since the other threads weren’t replicated,
and the child process would no longer be able to perform imports.

Python 2.7 acquires the import lock before performing an
@ref{bb9,,os.fork()}, and will also clean up any locks created using the
@ref{109,,threading} module.  C extension modules that have internal
locks, or that call @code{fork()} themselves, will not benefit
from this clean-up.

(Fixed by Thomas Wouters; bpo-1590864@footnote{https://bugs.python.org/issue1590864}.)

@item 
The @ref{eed,,Py_Finalize()} function now calls the internal
@code{threading._shutdown()} function; this prevents some exceptions from
being raised when an interpreter shuts down.
(Patch by Adam Olsen; bpo-1722344@footnote{https://bugs.python.org/issue1722344}.)

@item 
When using the @ref{6eb,,PyMemberDef} structure to define attributes
of a type, Python will no longer let you try to delete or set a
@code{T_STRING_INPLACE} attribute.

@c rev 79644

@item 
Global symbols defined by the @ref{29,,ctypes} module are now prefixed
with @code{Py}, or with @code{_ctypes}.  (Implemented by Thomas
Heller; bpo-3102@footnote{https://bugs.python.org/issue3102}.)

@item 
New configure option: the @code{--with-system-expat} switch allows
building the @code{pyexpat} module to use the system Expat library.
(Contributed by Arfrever Frehtes Taifersar Arahesis; bpo-7609@footnote{https://bugs.python.org/issue7609}.)

@item 
New configure option: the
@code{--with-valgrind} option will now disable the pymalloc
allocator, which is difficult for the Valgrind memory-error detector
to analyze correctly.
Valgrind will therefore be better at detecting memory leaks and
overruns. (Contributed by James Henstridge; bpo-2422@footnote{https://bugs.python.org/issue2422}.)

@item 
New configure option: you can now supply an empty string to
@code{--with-dbmliborder=} in order to disable all of the various
DBM modules.  (Added by Arfrever Frehtes Taifersar Arahesis;
bpo-6491@footnote{https://bugs.python.org/issue6491}.)

@item 
The @code{configure} script now checks for floating-point rounding bugs
on certain 32-bit Intel chips and defines a @code{X87_DOUBLE_ROUNDING}
preprocessor definition.  No code currently uses this definition,
but it’s available if anyone wishes to use it.
(Added by Mark Dickinson; bpo-2937@footnote{https://bugs.python.org/issue2937}.)

@code{configure} also now sets a 
@geindex LDCXXSHARED
@geindex environment variable; LDCXXSHARED
@code{LDCXXSHARED} Makefile
variable for supporting C++ linking.  (Contributed by Arfrever
Frehtes Taifersar Arahesis; bpo-1222585@footnote{https://bugs.python.org/issue1222585}.)

@item 
The build process now creates the necessary files for pkg-config
support.  (Contributed by Clinton Roy; bpo-3585@footnote{https://bugs.python.org/issue3585}.)

@item 
The build process now supports Subversion 1.7.  (Contributed by
Arfrever Frehtes Taifersar Arahesis; bpo-6094@footnote{https://bugs.python.org/issue6094}.)
@end itemize

@menu
* Capsules:: 
* Port-Specific Changes; Windows: Port-Specific Changes Windows. 
* Port-Specific Changes; Mac OS X: Port-Specific Changes Mac OS X. 
* Port-Specific Changes; FreeBSD: Port-Specific Changes FreeBSD. 

@end menu

@node Capsules,Port-Specific Changes Windows,,Build and C API Changes<8>
@anchor{whatsnew/2 7 capsules}@anchor{eee}@anchor{whatsnew/2 7 whatsnew27-capsules}@anchor{eef}
@subsubsection Capsules


Python 3.1 adds a new C datatype, @ref{e21,,PyCapsule}, for providing a
C API to an extension module.  A capsule is essentially the holder of
a C @code{void *} pointer, and is made available as a module attribute; for
example, the @ref{ea,,socket} module’s API is exposed as @code{socket.CAPI},
and @ref{121,,unicodedata} exposes @code{ucnhash_CAPI}.  Other extensions
can import the module, access its dictionary to get the capsule
object, and then get the @code{void *} pointer, which will usually point
to an array of pointers to the module’s various API functions.

There is an existing data type already used for this,
@code{PyCObject}, but it doesn’t provide type safety.  Evil code
written in pure Python could cause a segmentation fault by taking a
@code{PyCObject} from module A and somehow substituting it for the
@code{PyCObject} in module B.   Capsules know their own name,
and getting the pointer requires providing the name:

@example
void *vtable;

if (!PyCapsule_IsValid(capsule, "mymodule.CAPI") @{
        PyErr_SetString(PyExc_ValueError, "argument type invalid");
        return NULL;
@}

vtable = PyCapsule_GetPointer(capsule, "mymodule.CAPI");
@end example

You are assured that @code{vtable} points to whatever you’re expecting.
If a different capsule was passed in, @ref{ef0,,PyCapsule_IsValid()} would
detect the mismatched name and return false.  Refer to
@ref{ef1,,Providing a C API for an Extension Module} for more information on using these objects.

Python 2.7 now uses capsules internally to provide various
extension-module APIs, but the @code{PyCObject_AsVoidPtr()} was
modified to handle capsules, preserving compile-time compatibility
with the @code{CObject} interface.  Use of
@code{PyCObject_AsVoidPtr()} will signal a
@ref{3a9,,PendingDeprecationWarning}, which is silent by default.

Implemented in Python 3.1 and backported to 2.7 by Larry Hastings;
discussed in bpo-5630@footnote{https://bugs.python.org/issue5630}.

@c ======================================================================

@node Port-Specific Changes Windows,Port-Specific Changes Mac OS X,Capsules,Build and C API Changes<8>
@anchor{whatsnew/2 7 port-specific-changes-windows}@anchor{ef2}
@subsubsection Port-Specific Changes: Windows



@itemize *

@item 
The @ref{b2,,msvcrt} module now contains some constants from
the @code{crtassem.h} header file:
@code{CRT_ASSEMBLY_VERSION},
@code{VC_ASSEMBLY_PUBLICKEYTOKEN},
and @code{LIBRARIES_ASSEMBLY_NAME_PREFIX}.
(Contributed by David Cournapeau; bpo-4365@footnote{https://bugs.python.org/issue4365}.)

@item 
The @code{_winreg} module for accessing the registry now implements
the @code{CreateKeyEx()} and @code{DeleteKeyEx()}
functions, extended versions of previously-supported functions that
take several extra arguments.  The @code{DisableReflectionKey()},
@code{EnableReflectionKey()}, and @code{QueryReflectionKey()}
were also tested and documented.
(Implemented by Brian Curtin: bpo-7347@footnote{https://bugs.python.org/issue7347}.)

@item 
The new @code{_beginthreadex()} API is used to start threads, and
the native thread-local storage functions are now used.
(Contributed by Kristján Valur Jónsson; bpo-3582@footnote{https://bugs.python.org/issue3582}.)

@item 
The @ref{ef3,,os.kill()} function now works on Windows.  The signal value
can be the constants @code{CTRL_C_EVENT},
@code{CTRL_BREAK_EVENT}, or any integer.  The first two constants
will send @code{Control-C} and @code{Control-Break} keystroke events to
subprocesses; any other value will use the @code{TerminateProcess()}
API.  (Contributed by Miki Tebeka; bpo-1220212@footnote{https://bugs.python.org/issue1220212}.)

@item 
The @ref{c8d,,os.listdir()} function now correctly fails
for an empty path.  (Fixed by Hirokazu Yamamoto; bpo-5913@footnote{https://bugs.python.org/issue5913}.)

@item 
The @code{mimelib} module will now read the MIME database from
the Windows registry when initializing.
(Patch by Gabriel Genellina; bpo-4969@footnote{https://bugs.python.org/issue4969}.)
@end itemize

@c ======================================================================

@node Port-Specific Changes Mac OS X,Port-Specific Changes FreeBSD,Port-Specific Changes Windows,Build and C API Changes<8>
@anchor{whatsnew/2 7 port-specific-changes-mac-os-x}@anchor{ef4}
@subsubsection Port-Specific Changes: Mac OS X



@itemize *

@item 
The path @code{/Library/Python/2.7/site-packages} is now appended to
@code{sys.path}, in order to share added packages between the system
installation and a user-installed copy of the same version.
(Changed by Ronald Oussoren; bpo-4865@footnote{https://bugs.python.org/issue4865}.)

@quotation


Changed in version 2.7.13: As of 2.7.13, this change was removed.
@code{/Library/Python/2.7/site-packages}, the site-packages directory
used by the Apple-supplied system Python 2.7 is no longer appended to
@code{sys.path} for user-installed Pythons such as from the python.org
installers.  As of macOS 10.12, Apple changed how the system
site-packages directory is configured, which could cause installation
of pip components, like setuptools, to fail.  Packages installed for
the system Python will no longer be shared with user-installed
Pythons. (bpo-28440@footnote{https://bugs.python.org/issue28440})

@end quotation
@end itemize

@node Port-Specific Changes FreeBSD,,Port-Specific Changes Mac OS X,Build and C API Changes<8>
@anchor{whatsnew/2 7 port-specific-changes-freebsd}@anchor{ef5}
@subsubsection Port-Specific Changes: FreeBSD



@itemize *

@item 
FreeBSD 7.1’s @code{SO_SETFIB} constant, used with
@code{getsockopt()}/@code{setsockopt()} to select an
alternate routing table, is now available in the @ref{ea,,socket}
module.  (Added by Kyle VanderBeek; bpo-8235@footnote{https://bugs.python.org/issue8235}.)
@end itemize

@node Other Changes and Fixes,Porting to Python 2 7,Build and C API Changes<8>,What’s New in Python 2 7
@anchor{whatsnew/2 7 other-changes-and-fixes}@anchor{ef6}
@subsection Other Changes and Fixes



@itemize *

@item 
Two benchmark scripts, @code{iobench} and @code{ccbench}, were
added to the @code{Tools} directory.  @code{iobench} measures the
speed of the built-in file I/O objects returned by @ref{1c8,,open()}
while performing various operations, and @code{ccbench} is a
concurrency benchmark that tries to measure computing throughput,
thread switching latency, and IO processing bandwidth when
performing several tasks using a varying number of threads.

@item 
The @code{Tools/i18n/msgfmt.py} script now understands plural
forms in @code{.po} files.  (Fixed by Martin von Löwis;
bpo-5464@footnote{https://bugs.python.org/issue5464}.)

@item 
When importing a module from a @code{.pyc} or @code{.pyo} file
with an existing @code{.py} counterpart, the @code{co_filename}
attributes of the resulting code objects are overwritten when the
original filename is obsolete.  This can happen if the file has been
renamed, moved, or is accessed through different paths.  (Patch by
Ziga Seilnacht and Jean-Paul Calderone; bpo-1180193@footnote{https://bugs.python.org/issue1180193}.)

@item 
The @code{regrtest.py} script now takes a @code{--randseed=}
switch that takes an integer that will be used as the random seed
for the @code{-r} option that executes tests in random order.
The @code{-r} option also reports the seed that was used
(Added by Collin Winter.)

@item 
Another @code{regrtest.py} switch is @code{-j}, which
takes an integer specifying how many tests run in parallel. This
allows reducing the total runtime on multi-core machines.
This option is compatible with several other options, including the
@code{-R} switch which is known to produce long runtimes.
(Added by Antoine Pitrou, bpo-6152@footnote{https://bugs.python.org/issue6152}.)  This can also be used
with a new @code{-F} switch that runs selected tests in a loop
until they fail.  (Added by Antoine Pitrou; bpo-7312@footnote{https://bugs.python.org/issue7312}.)

@item 
When executed as a script, the @code{py_compile.py} module now
accepts @code{'-'} as an argument, which will read standard input for
the list of filenames to be compiled.  (Contributed by Piotr
Ożarowski; bpo-8233@footnote{https://bugs.python.org/issue8233}.)
@end itemize

@c ======================================================================

@node Porting to Python 2 7,New Features Added to Python 2 7 Maintenance Releases,Other Changes and Fixes,What’s New in Python 2 7
@anchor{whatsnew/2 7 porting-to-python-2-7}@anchor{ef7}
@subsection Porting to Python 2.7


This section lists previously described changes and other bugfixes
that may require changes to your code:


@itemize *

@item 
The @ref{438,,range()} function processes its arguments more
consistently; it will now call @code{__int__()} on non-float,
non-integer arguments that are supplied to it.  (Fixed by Alexander
Belopolsky; bpo-1533@footnote{https://bugs.python.org/issue1533}.)

@item 
The string @ref{785,,format()} method changed the default precision used
for floating-point and complex numbers from 6 decimal
places to 12, which matches the precision used by @ref{2ef,,str()}.
(Changed by Eric Smith; bpo-5920@footnote{https://bugs.python.org/issue5920}.)

@item 
Because of an optimization for the @ref{168,,with} statement, the special
methods @code{__enter__()} and @code{__exit__()} must belong to the object’s
type, and cannot be directly attached to the object’s instance.  This
affects new-style classes (derived from @ref{5a5,,object}) and C extension
types.  (bpo-6101@footnote{https://bugs.python.org/issue6101}.)

@item 
Due to a bug in Python 2.6, the @emph{exc_value} parameter to
@code{__exit__()} methods was often the string representation of the
exception, not an instance.  This was fixed in 2.7, so @emph{exc_value}
will be an instance as expected.  (Fixed by Florent Xicluna;
bpo-7853@footnote{https://bugs.python.org/issue7853}.)

@item 
When a restricted set of attributes were set using @code{__slots__},
deleting an unset attribute would not raise @ref{163,,AttributeError}
as you would expect.  Fixed by Benjamin Peterson; bpo-7604@footnote{https://bugs.python.org/issue7604}.)
@end itemize

In the standard library:


@itemize *

@item 
Operations with @ref{363,,datetime} instances that resulted in a year
falling outside the supported range didn’t always raise
@ref{347,,OverflowError}.  Such errors are now checked more carefully
and will now raise the exception. (Reported by Mark Leander, patch
by Anand B. Pillai and Alexander Belopolsky; bpo-7150@footnote{https://bugs.python.org/issue7150}.)

@item 
When using @ref{248,,Decimal} instances with a string’s
@ref{785,,format()} method, the default alignment was previously
left-alignment.  This has been changed to right-alignment, which might
change the output of your programs.
(Changed by Mark Dickinson; bpo-6857@footnote{https://bugs.python.org/issue6857}.)

Comparisons involving a signaling NaN value (or @code{sNAN}) now signal
@ref{c3b,,InvalidOperation} instead of silently returning a true or
false value depending on the comparison operator.  Quiet NaN values
(or @code{NaN}) are now hashable.  (Fixed by Mark Dickinson;
bpo-7279@footnote{https://bugs.python.org/issue7279}.)

@item 
The ElementTree library, @code{xml.etree}, no longer escapes
ampersands and angle brackets when outputting an XML processing
instruction (which looks like @cite{<?xml-stylesheet href=”#style1”?>})
or comment (which looks like @cite{<!– comment –>}).
(Patch by Neil Muller; bpo-2746@footnote{https://bugs.python.org/issue2746}.)

@item 
The @code{readline()} method of @code{StringIO} objects now does
nothing when a negative length is requested, as other file-like
objects do.  (bpo-7348@footnote{https://bugs.python.org/issue7348}).

@item 
The @ref{f9,,syslog} module will now use the value of @code{sys.argv[0]} as the
identifier instead of the previous default value of @code{'python'}.
(Changed by Sean Reifschneider; bpo-8451@footnote{https://bugs.python.org/issue8451}.)

@item 
The @ref{fb,,tarfile} module’s default error handling has changed, to
no longer suppress fatal errors.  The default error level was previously 0,
which meant that errors would only result in a message being written to the
debug log, but because the debug log is not activated by default,
these errors go unnoticed.  The default error level is now 1,
which raises an exception if there’s an error.
(Changed by Lars Gustäbel; bpo-7357@footnote{https://bugs.python.org/issue7357}.)

@item 
The @code{urlparse} module’s @code{urlsplit()} now handles
unknown URL schemes in a fashion compliant with 
@geindex RFC; RFC 3986
RFC 3986@footnote{https://tools.ietf.org/html/rfc3986.html}: if the
URL is of the form @code{"<something>://..."}, the text before the
@code{://} is treated as the scheme, even if it’s a made-up scheme that
the module doesn’t know about.  This change may break code that
worked around the old behaviour.  For example, Python 2.6.4 or 2.5
will return the following:

@example
>>> import urlparse
>>> urlparse.urlsplit('invented://host/filename?query')
('invented', '@w{'}, '//host/filename?query', '@w{'}, '@w{'})
@end example

Python 2.7 (and Python 2.6.5) will return:

@example
>>> import urlparse
>>> urlparse.urlsplit('invented://host/filename?query')
('invented', 'host', '/filename?query', '@w{'}, '@w{'})
@end example

(Python 2.7 actually produces slightly different output, since it
returns a named tuple instead of a standard tuple.)
@end itemize

For C extensions:


@itemize *

@item 
C extensions that use integer format codes with the @code{PyArg_Parse*}
family of functions will now raise a @ref{162,,TypeError} exception
instead of triggering a @ref{1b2,,DeprecationWarning} (bpo-5080@footnote{https://bugs.python.org/issue5080}).

@item 
Use the new @ref{e3b,,PyOS_string_to_double()} function instead of the old
@code{PyOS_ascii_strtod()} and @code{PyOS_ascii_atof()} functions,
which are now deprecated.
@end itemize

For applications that embed Python:


@itemize *

@item 
The @ref{e14,,PySys_SetArgvEx()} function was added, letting
applications close a security hole when the existing
@ref{eec,,PySys_SetArgv()} function was used.  Check whether you’re
calling @ref{eec,,PySys_SetArgv()} and carefully consider whether the
application should be using @ref{e14,,PySys_SetArgvEx()} with
@emph{updatepath} set to false.
@end itemize

@c ======================================================================

@node New Features Added to Python 2 7 Maintenance Releases,Acknowledgements,Porting to Python 2 7,What’s New in Python 2 7
@anchor{whatsnew/2 7 new-features-added-to-python-2-7-maintenance-releases}@anchor{ef8}@anchor{whatsnew/2 7 py27-maintenance-enhancements}@anchor{e83}
@subsection New Features Added to Python 2.7 Maintenance Releases


New features may be added to Python 2.7 maintenance releases when the
situation genuinely calls for it. Any such additions must go through
the Python Enhancement Proposal process, and make a compelling case for why
they can’t be adequately addressed by either adding the new feature solely to
Python 3, or else by publishing it on the Python Package Index.

In addition to the specific proposals listed below, there is a general
exemption allowing new @code{-3} warnings to be added in any Python 2.7
maintenance release.

@menu
* Two new environment variables for debug mode:: 
* PEP 434; IDLE Enhancement Exception for All Branches: PEP 434 IDLE Enhancement Exception for All Branches. 
* PEP 466; Network Security Enhancements for Python 2.7: PEP 466 Network Security Enhancements for Python 2 7. 
* PEP 477; Backport ensurepip (PEP 453) to Python 2.7: PEP 477 Backport ensurepip PEP 453 to Python 2 7. 
* PEP 476; Enabling certificate verification by default for stdlib http clients: PEP 476 Enabling certificate verification by default for stdlib http clients<2>. 
* PEP 493; HTTPS verification migration tools for Python 2.7: PEP 493 HTTPS verification migration tools for Python 2 7. 
* New make regen-all build target: New make regen-all build target<3>. 
* Removal of make touch build target: Removal of make touch build target<3>. 

@end menu

@node Two new environment variables for debug mode,PEP 434 IDLE Enhancement Exception for All Branches,,New Features Added to Python 2 7 Maintenance Releases
@anchor{whatsnew/2 7 two-new-environment-variables-for-debug-mode}@anchor{ef9}
@subsubsection Two new environment variables for debug mode


In debug mode, the @code{[xxx refs]} statistic is not written by default, the
@geindex PYTHONSHOWREFCOUNT
@geindex environment variable; PYTHONSHOWREFCOUNT
@code{PYTHONSHOWREFCOUNT} environment variable now must also be set.
(Contributed by Victor Stinner; bpo-31733@footnote{https://bugs.python.org/issue31733}.)

When Python is compiled with @code{COUNT_ALLOC} defined, allocation counts are no
longer dumped by default anymore: the 
@geindex PYTHONSHOWALLOCCOUNT
@geindex environment variable; PYTHONSHOWALLOCCOUNT
@code{PYTHONSHOWALLOCCOUNT} environment
variable must now also be set. Moreover, allocation counts are now dumped into
stderr, rather than stdout. (Contributed by Victor Stinner; bpo-31692@footnote{https://bugs.python.org/issue31692}.)


New in version 2.7.15.


@node PEP 434 IDLE Enhancement Exception for All Branches,PEP 466 Network Security Enhancements for Python 2 7,Two new environment variables for debug mode,New Features Added to Python 2 7 Maintenance Releases
@anchor{whatsnew/2 7 pep-434-idle-enhancement-exception-for-all-branches}@anchor{efa}
@subsubsection PEP 434: IDLE Enhancement Exception for All Branches


@geindex Python Enhancement Proposals; PEP 434
PEP 434@footnote{https://www.python.org/dev/peps/pep-0434} describes a general exemption for changes made to the IDLE
development environment shipped along with Python. This exemption makes it
possible for the IDLE developers to provide a more consistent user
experience across all supported versions of Python 2 and 3.

For details of any IDLE changes, refer to the NEWS file for the specific
release.

@node PEP 466 Network Security Enhancements for Python 2 7,PEP 477 Backport ensurepip PEP 453 to Python 2 7,PEP 434 IDLE Enhancement Exception for All Branches,New Features Added to Python 2 7 Maintenance Releases
@anchor{whatsnew/2 7 pep-466-network-security-enhancements-for-python-2-7}@anchor{efb}
@subsubsection PEP 466: Network Security Enhancements for Python 2.7


@geindex Python Enhancement Proposals; PEP 466
PEP 466@footnote{https://www.python.org/dev/peps/pep-0466} describes a number of network security enhancement proposals
that have been approved for inclusion in Python 2.7 maintenance releases,
with the first of those changes appearing in the Python 2.7.7 release.

@geindex Python Enhancement Proposals; PEP 466
PEP 466@footnote{https://www.python.org/dev/peps/pep-0466} related features added in Python 2.7.7:


@itemize *

@item 
@ref{c5b,,hmac.compare_digest()} was backported from Python 3 to make a timing
attack resistant comparison operation available to Python 2 applications.
(Contributed by Alex Gaynor; bpo-21306@footnote{https://bugs.python.org/issue21306}.)

@item 
OpenSSL 1.0.1g was upgraded in the official Windows installers published on
python.org. (Contributed by Zachary Ware; bpo-21462@footnote{https://bugs.python.org/issue21462}.)
@end itemize

@geindex Python Enhancement Proposals; PEP 466
PEP 466@footnote{https://www.python.org/dev/peps/pep-0466} related features added in Python 2.7.8:


@itemize *

@item 
@ref{28b,,hashlib.pbkdf2_hmac()} was backported from Python 3 to make a hashing
algorithm suitable for secure password storage broadly available to Python
2 applications. (Contributed by Alex Gaynor; bpo-21304@footnote{https://bugs.python.org/issue21304}.)

@item 
OpenSSL 1.0.1h was upgraded for the official Windows installers published on
python.org. (contributed by Zachary Ware in bpo-21671@footnote{https://bugs.python.org/issue21671} for CVE-2014-0224)
@end itemize

@geindex Python Enhancement Proposals; PEP 466
PEP 466@footnote{https://www.python.org/dev/peps/pep-0466} related features added in Python 2.7.9:


@itemize *

@item 
Most of Python 3.4’s @ref{ee,,ssl} module was backported. This means @ref{ee,,ssl}
now supports Server Name Indication, TLS1.x settings, access to the platform
certificate store, the @ref{2bf,,SSLContext} class, and other
features. (Contributed by Alex Gaynor and David Reid; bpo-21308@footnote{https://bugs.python.org/issue21308}.)

Refer to the “Version added: 2.7.9” notes in the module documentation for
specific details.

@item 
@ref{17f,,os.urandom()} was changed to cache a file descriptor to @code{/dev/urandom}
instead of reopening @code{/dev/urandom} on every call. (Contributed by Alex
Gaynor; bpo-21305@footnote{https://bugs.python.org/issue21305}.)

@item 
@ref{efc,,hashlib.algorithms_guaranteed} and
@ref{efd,,hashlib.algorithms_available} were backported from Python 3 to make
it easier for Python 2 applications to select the strongest available hash
algorithm. (Contributed by Alex Gaynor in bpo-21307@footnote{https://bugs.python.org/issue21307})
@end itemize

@node PEP 477 Backport ensurepip PEP 453 to Python 2 7,PEP 476 Enabling certificate verification by default for stdlib http clients<2>,PEP 466 Network Security Enhancements for Python 2 7,New Features Added to Python 2 7 Maintenance Releases
@anchor{whatsnew/2 7 pep-477-backport-ensurepip-pep-453-to-python-2-7}@anchor{efe}
@subsubsection PEP 477: Backport ensurepip (PEP 453) to Python 2.7


@geindex Python Enhancement Proposals; PEP 477
PEP 477@footnote{https://www.python.org/dev/peps/pep-0477} approves the inclusion of the 
@geindex Python Enhancement Proposals; PEP 453
PEP 453@footnote{https://www.python.org/dev/peps/pep-0453} ensurepip module and the
improved documentation that was enabled by it in the Python 2.7 maintenance
releases, appearing first in the Python 2.7.9 release.

@menu
* Bootstrapping pip By Default: Bootstrapping pip By Default<2>. 
* Documentation Changes: Documentation Changes<2>. 

@end menu

@node Bootstrapping pip By Default<2>,Documentation Changes<2>,,PEP 477 Backport ensurepip PEP 453 to Python 2 7
@anchor{whatsnew/2 7 bootstrapping-pip-by-default}@anchor{eff}
@subsubsection Bootstrapping pip By Default


The new @ref{75,,ensurepip} module (defined in 
@geindex Python Enhancement Proposals; PEP 453
PEP 453@footnote{https://www.python.org/dev/peps/pep-0453}) provides a standard
cross-platform mechanism to bootstrap the pip installer into Python
installations. The version of @code{pip} included with Python 2.7.9 is @code{pip}
1.5.6, and future 2.7.x maintenance releases will update the bundled version to
the latest version of @code{pip} that is available at the time of creating the
release candidate.

By default, the commands @code{pip}, @code{pipX} and @code{pipX.Y} will be installed on
all platforms (where X.Y stands for the version of the Python installation),
along with the @code{pip} Python package and its dependencies.

For CPython @ref{a67,,source builds on POSIX systems},
the @code{make install} and @code{make altinstall} commands do not bootstrap @code{pip}
by default.  This behaviour can be controlled through configure options, and
overridden through Makefile options.

On Windows and Mac OS X, the CPython installers now default to installing
@code{pip} along with CPython itself (users may opt out of installing it
during the installation process). Window users will need to opt in to the
automatic @code{PATH} modifications to have @code{pip} available from the command
line by default, otherwise it can still be accessed through the Python
launcher for Windows as @code{py -m pip}.

As discussed in the PEP@footnote{https://www.python.org/dev/peps/pep-0477/#disabling-ensurepip-by-downstream-distributors}, platform packagers may choose not to install
these commands by default, as long as, when invoked, they provide clear and
simple directions on how to install them on that platform (usually using
the system package manager).

@node Documentation Changes<2>,,Bootstrapping pip By Default<2>,PEP 477 Backport ensurepip PEP 453 to Python 2 7
@anchor{whatsnew/2 7 documentation-changes}@anchor{f00}@anchor{whatsnew/2 7 id1}@anchor{f01}
@subsubsection Documentation Changes


As part of this change, the @ref{a69,,Installing Python Modules} and
@ref{a6a,,Distributing Python Modules} sections of the documentation have been
completely redesigned as short getting started and FAQ documents. Most
packaging documentation has now been moved out to the Python Packaging
Authority maintained Python Packaging User Guide@footnote{http://packaging.python.org} and the documentation of the individual
projects.

However, as this migration is currently still incomplete, the legacy
versions of those guides remaining available as @ref{a6b,,Installing Python Modules (Legacy version)}
and @ref{a6c,,Distributing Python Modules (Legacy version)}.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 453

@item PEP 453@footnote{https://www.python.org/dev/peps/pep-0453} – Explicit bootstrapping of pip in Python installations

PEP written by Donald Stufft and Nick Coghlan, implemented by
Donald Stufft, Nick Coghlan, Martin von Löwis and Ned Deily.
@end table


@node PEP 476 Enabling certificate verification by default for stdlib http clients<2>,PEP 493 HTTPS verification migration tools for Python 2 7,PEP 477 Backport ensurepip PEP 453 to Python 2 7,New Features Added to Python 2 7 Maintenance Releases
@anchor{whatsnew/2 7 pep-476-enabling-certificate-verification-by-default-for-stdlib-http-clients}@anchor{f02}
@subsubsection PEP 476: Enabling certificate verification by default for stdlib http clients


@geindex Python Enhancement Proposals; PEP 476
PEP 476@footnote{https://www.python.org/dev/peps/pep-0476} updated @code{httplib} and modules which use it, such as
@code{urllib2} and @code{xmlrpclib}, to now verify that the server
presents a certificate which is signed by a Certificate Authority in the
platform trust store and whose hostname matches the hostname being requested
by default, significantly improving security for many applications. This
change was made in the Python 2.7.9 release.

For applications which require the old previous behavior, they can pass an
alternate context:

@example
import urllib2
import ssl

# This disables all verification
context = ssl._create_unverified_context()

# This allows using a specific certificate for the host, which doesn't need
# to be in the trust store
context = ssl.create_default_context(cafile="/path/to/file.crt")

urllib2.urlopen("https://invalid-cert", context=context)
@end example

@node PEP 493 HTTPS verification migration tools for Python 2 7,New make regen-all build target<3>,PEP 476 Enabling certificate verification by default for stdlib http clients<2>,New Features Added to Python 2 7 Maintenance Releases
@anchor{whatsnew/2 7 pep-493-https-verification-migration-tools-for-python-2-7}@anchor{f03}
@subsubsection PEP 493: HTTPS verification migration tools for Python 2.7


@geindex Python Enhancement Proposals; PEP 493
PEP 493@footnote{https://www.python.org/dev/peps/pep-0493} provides additional migration tools to support a more incremental
infrastructure upgrade process for environments containing applications and
services relying on the historically permissive processing of server
certificates when establishing client HTTPS connections.  These additions were
made in the Python 2.7.12 release.

These tools are intended for use in cases where affected applications and
services can’t be modified to explicitly pass a more permissive SSL context
when establishing the connection.

For applications and services which can’t be modified at all, the new
@code{PYTHONHTTPSVERIFY} environment variable may be set to @code{0} to revert an
entire Python process back to the default permissive behaviour of Python 2.7.8
and earlier.

For cases where the connection establishment code can’t be modified, but the
overall application can be, the new @code{ssl._https_verify_certificates()}
function can be used to adjust the default behaviour at runtime.

@node New make regen-all build target<3>,Removal of make touch build target<3>,PEP 493 HTTPS verification migration tools for Python 2 7,New Features Added to Python 2 7 Maintenance Releases
@anchor{whatsnew/2 7 new-make-regen-all-build-target}@anchor{f04}
@subsubsection New @code{make regen-all} build target


To simplify cross-compilation, and to ensure that CPython can reliably be
compiled without requiring an existing version of Python to already be
available, the autotools-based build system no longer attempts to implicitly
recompile generated files based on file modification times.

Instead, a new @code{make regen-all} command has been added to force regeneration
of these files when desired (e.g. after an initial version of Python has
already been built based on the pregenerated versions).

More selective regeneration targets are also defined - see
Makefile.pre.in@footnote{https://github.com/python/cpython/tree/main/Makefile.pre.in} for details.

(Contributed by Victor Stinner in bpo-23404@footnote{https://bugs.python.org/issue23404}.)


New in version 2.7.14.


@node Removal of make touch build target<3>,,New make regen-all build target<3>,New Features Added to Python 2 7 Maintenance Releases
@anchor{whatsnew/2 7 removal-of-make-touch-build-target}@anchor{f05}
@subsubsection Removal of @code{make touch} build target


The @code{make touch} build target previously used to request implicit regeneration
of generated files by updating their modification times has been removed.

It has been replaced by the new @code{make regen-all} target.

(Contributed by Victor Stinner in bpo-23404@footnote{https://bugs.python.org/issue23404}.)


Changed in version 2.7.14.


@c ======================================================================

@node Acknowledgements,,New Features Added to Python 2 7 Maintenance Releases,What’s New in Python 2 7
@anchor{whatsnew/2 7 acknowledgements}@anchor{f06}@anchor{whatsnew/2 7 acks27}@anchor{f07}
@subsection Acknowledgements


The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Nick Coghlan, Philip Jenvey, Ryan Lovett, R. David Murray,
Hugh Secker-Walker.

@node What’s New in Python 2 6,What’s New in Python 2 5,What’s New in Python 2 7,What’s New in Python
@anchor{whatsnew/2 6 doc}@anchor{f08}@anchor{whatsnew/2 6 what-s-new-in-python-2-6}@anchor{f09}@anchor{whatsnew/2 6 whats-new-in-2-6}@anchor{e53}
@section What’s New in Python 2.6


@c XXX add trademark info for Apple, Microsoft, SourceForge.

@*Author: 
A.M. Kuchling (amk at amk.ca)


@c $Id$
@c Rules for maintenance:
@c 
@c * Anyone can add text to this document.  Do not spend very much time
@c on the wording of your changes, because your text will probably
@c get rewritten to some degree.
@c 
@c * The maintainer will go through Misc/NEWS periodically and add
@c changes; it's therefore more important to add your changes to
@c Misc/NEWS than to this file.
@c 
@c * This is not a complete list of every single change; completeness
@c is the purpose of Misc/NEWS.  Some changes I consider too small
@c or esoteric to include.  If such a change is added to the text,
@c I'll just remove it.  (This is another reason you shouldn't spend
@c too much time on writing your addition.)
@c 
@c * If you want to draw your new text to the attention of the
@c maintainer, add 'XXX' to the beginning of the paragraph or
@c section.
@c 
@c * It's OK to just add a fragmentary note about a change.  For
@c example: "XXX Describe the transmogrify() function added to the
@c socket module."  The maintainer will research the change and
@c write the necessary text.
@c 
@c * You can comment out your additions if you like, but it's not
@c necessary (especially when a final release is some months away).
@c 
@c * Credit the author of a patch or bugfix.   Just the name is
@c sufficient; the e-mail address isn't necessary.
@c 
@c * It's helpful to add the bug/patch number in a parenthetical comment.
@c 
@c XXX Describe the transmogrify() function added to the socket
@c module.
@c (Contributed by P.Y. Developer; :issue:`12345`.)
@c 
@c This saves the maintainer some effort going through the SVN logs
@c when researching a change.

This article explains the new features in Python 2.6, released on October 1
2008.  The release schedule is described in 
@geindex Python Enhancement Proposals; PEP 361
PEP 361@footnote{https://www.python.org/dev/peps/pep-0361}.

The major theme of Python 2.6 is preparing the migration path to
Python 3.0, a major redesign of the language.  Whenever possible,
Python 2.6 incorporates new features and syntax from 3.0 while
remaining compatible with existing code by not removing older features
or syntax.  When it’s not possible to do that, Python 2.6 tries to do
what it can, adding compatibility functions in a
@code{future_builtins} module and a @code{-3} switch to warn about
usages that will become unsupported in 3.0.

Some significant new packages have been added to the standard library,
such as the @ref{b3,,multiprocessing} and @ref{a0,,json} modules, but
there aren’t many new features that aren’t related to Python 3.0 in
some way.

Python 2.6 also sees a number of improvements and bugfixes throughout
the source.  A search through the change logs finds there were 259
patches applied and 612 bugs fixed between Python 2.5 and 2.6.  Both
figures are likely to be underestimates.

This article doesn’t attempt to provide a complete specification of
the new features, but instead provides a convenient overview.  For
full details, you should refer to the documentation for Python 2.6. If
you want to understand the rationale for the design and
implementation, refer to the PEP for a particular new feature.
Whenever possible, “What’s New in Python” links to the bug/patch item
for each change.

@c Compare with previous release in 2 - 3 sentences here.
@c add hyperlink when the documentation becomes available online.

@c ========================================================================

@c Large, PEP-level features and changes should be described here.

@c ========================================================================

@menu
* Python 3.0: Python 3 0. 
* Changes to the Development Process:: 
* PEP 343; The ‘with’ statement: PEP 343 The ‘with’ statement. 
* PEP 366; Explicit Relative Imports From a Main Module: PEP 366 Explicit Relative Imports From a Main Module. 
* PEP 370; Per-user site-packages Directory: PEP 370 Per-user site-packages Directory. 
* PEP 371; The multiprocessing Package: PEP 371 The multiprocessing Package. 
* PEP 3101; Advanced String Formatting: PEP 3101 Advanced String Formatting. 
* PEP 3105; print As a Function: PEP 3105 print As a Function. 
* PEP 3110; Exception-Handling Changes: PEP 3110 Exception-Handling Changes. 
* PEP 3112; Byte Literals: PEP 3112 Byte Literals. 
* PEP 3116; New I/O Library: PEP 3116 New I/O Library. 
* PEP 3118; Revised Buffer Protocol: PEP 3118 Revised Buffer Protocol. 
* PEP 3119; Abstract Base Classes: PEP 3119 Abstract Base Classes. 
* PEP 3127; Integer Literal Support and Syntax: PEP 3127 Integer Literal Support and Syntax. 
* PEP 3129; Class Decorators: PEP 3129 Class Decorators. 
* PEP 3141; A Type Hierarchy for Numbers: PEP 3141 A Type Hierarchy for Numbers. 
* Other Language Changes: Other Language Changes<13>. 
* New and Improved Modules: New and Improved Modules<2>. 
* Deprecations and Removals:: 
* Build and C API Changes: Build and C API Changes<9>. 
* Porting to Python 2.6: Porting to Python 2 6. 
* Acknowledgements: Acknowledgements<2>. 

@end menu

@node Python 3 0,Changes to the Development Process,,What’s New in Python 2 6
@anchor{whatsnew/2 6 python-3-0}@anchor{f0a}
@subsection Python 3.0


The development cycle for Python versions 2.6 and 3.0 was
synchronized, with the alpha and beta releases for both versions being
made on the same days.  The development of 3.0 has influenced many
features in 2.6.

Python 3.0 is a far-ranging redesign of Python that breaks
compatibility with the 2.x series.  This means that existing Python
code will need some conversion in order to run on
Python 3.0.  However, not all the changes in 3.0 necessarily break
compatibility.  In cases where new features won’t cause existing code
to break, they’ve been backported to 2.6 and are described in this
document in the appropriate place.  Some of the 3.0-derived features
are:


@itemize *

@item 
A @code{__complex__()} method for converting objects to a complex number.

@item 
Alternate syntax for catching exceptions: @code{except TypeError as exc}.

@item 
The addition of @ref{e77,,functools.reduce()} as a synonym for the built-in
@code{reduce()} function.
@end itemize

Python 3.0 adds several new built-in functions and changes the
semantics of some existing builtins.  Functions that are new in 3.0
such as @ref{e4e,,bin()} have simply been added to Python 2.6, but existing
builtins haven’t been changed; instead, the @code{future_builtins}
module has versions with the new 3.0 semantics.  Code written to be
compatible with 3.0 can do @code{from future_builtins import hex, map} as
necessary.

A new command-line switch, @code{-3}, enables warnings
about features that will be removed in Python 3.0.  You can run code
with this switch to see how much work will be necessary to port
code to 3.0.  The value of this switch is available
to Python code as the boolean variable @code{sys.py3kwarning},
and to C extension code as @code{Py_Py3kWarningFlag}.


@subsubheading See also


The 3xxx series of PEPs, which contains proposals for Python 3.0.
@geindex Python Enhancement Proposals; PEP 3000
PEP 3000@footnote{https://www.python.org/dev/peps/pep-3000} describes the development process for Python 3.0.
Start with 
@geindex Python Enhancement Proposals; PEP 3100
PEP 3100@footnote{https://www.python.org/dev/peps/pep-3100} that describes the general goals for Python
3.0, and then explore the higher-numbered PEPS that propose
specific features.


@node Changes to the Development Process,PEP 343 The ‘with’ statement,Python 3 0,What’s New in Python 2 6
@anchor{whatsnew/2 6 changes-to-the-development-process}@anchor{f0b}
@subsection Changes to the Development Process


While 2.6 was being developed, the Python development process
underwent two significant changes: we switched from SourceForge’s
issue tracker to a customized Roundup installation, and the
documentation was converted from LaTeX to reStructuredText.

@menu
* New Issue Tracker; Roundup: New Issue Tracker Roundup. 
* New Documentation Format; reStructuredText Using Sphinx: New Documentation Format reStructuredText Using Sphinx. 

@end menu

@node New Issue Tracker Roundup,New Documentation Format reStructuredText Using Sphinx,,Changes to the Development Process
@anchor{whatsnew/2 6 new-issue-tracker-roundup}@anchor{f0c}
@subsubsection New Issue Tracker: Roundup


For a long time, the Python developers had been growing increasingly
annoyed by SourceForge’s bug tracker.  SourceForge’s hosted solution
doesn’t permit much customization; for example, it wasn’t possible to
customize the life cycle of issues.

The infrastructure committee of the Python Software Foundation
therefore posted a call for issue trackers, asking volunteers to set
up different products and import some of the bugs and patches from
SourceForge.  Four different trackers were examined: Jira@footnote{https://www.atlassian.com/software/jira/},
Launchpad@footnote{https://launchpad.net/},
Roundup@footnote{http://roundup.sourceforge.net/}, and
Trac@footnote{https://trac.edgewall.org/}.
The committee eventually settled on Jira
and Roundup as the two candidates.  Jira is a commercial product that
offers no-cost hosted instances to free-software projects; Roundup
is an open-source project that requires volunteers
to administer it and a server to host it.

After posting a call for volunteers, a new Roundup installation was
set up at @indicateurl{https://bugs.python.org}.  One installation of Roundup can
host multiple trackers, and this server now also hosts issue trackers
for Jython and for the Python web site.  It will surely find
other uses in the future.  Where possible,
this edition of “What’s New in Python” links to the bug/patch
item for each change.

Hosting of the Python bug tracker is kindly provided by
Upfront Systems@footnote{http://www.upfrontsoftware.co.za}
of Stellenbosch, South Africa.  Martin von Löwis put a
lot of effort into importing existing bugs and patches from
SourceForge; his scripts for this import operation are at
@indicateurl{http://svn.python.org/view/tracker/importer/} and may be useful to
other projects wishing to move from SourceForge to Roundup.


@subsubheading See also



@table @asis

@item @indicateurl{https://bugs.python.org}

The Python bug tracker.

@item @indicateurl{http://bugs.jython.org}:

The Jython bug tracker.

@item @indicateurl{http://roundup.sourceforge.net/}

Roundup downloads and documentation.

@item @indicateurl{http://svn.python.org/view/tracker/importer/}

Martin von Löwis’s conversion scripts.
@end table


@node New Documentation Format reStructuredText Using Sphinx,,New Issue Tracker Roundup,Changes to the Development Process
@anchor{whatsnew/2 6 new-documentation-format-restructuredtext-using-sphinx}@anchor{f0d}
@subsubsection New Documentation Format: reStructuredText Using Sphinx


The Python documentation was written using LaTeX since the project
started around 1989.  In the 1980s and early 1990s, most documentation
was printed out for later study, not viewed online. LaTeX was widely
used because it provided attractive printed output while remaining
straightforward to write once the basic rules of the markup were
learned.

Today LaTeX is still used for writing publications destined for
printing, but the landscape for programming tools has shifted.  We no
longer print out reams of documentation; instead, we browse through it
online and HTML has become the most important format to support.
Unfortunately, converting LaTeX to HTML is fairly complicated and Fred
L. Drake Jr., the long-time Python documentation editor, spent a lot
of time maintaining the conversion process.  Occasionally people would
suggest converting the documentation into SGML and later XML, but
performing a good conversion is a major task and no one ever committed
the time required to finish the job.

During the 2.6 development cycle, Georg Brandl put a lot of effort
into building a new toolchain for processing the documentation.  The
resulting package is called Sphinx, and is available from
@indicateurl{http://sphinx-doc.org/}.

Sphinx concentrates on HTML output, producing attractively styled and
modern HTML; printed output is still supported through conversion to
LaTeX.  The input format is reStructuredText, a markup syntax
supporting custom extensions and directives that is commonly used in
the Python community.

Sphinx is a standalone package that can be used for writing, and
almost two dozen other projects
(listed on the Sphinx web site@footnote{https://www.sphinx-doc.org/en/master/examples.html})
have adopted Sphinx as their documentation tool.


@subsubheading See also



@table @asis

@item Documenting Python@footnote{https://devguide.python.org/documenting/}

Describes how to write for Python’s documentation.

@item Sphinx@footnote{http://sphinx-doc.org/}

Documentation and code for the Sphinx toolchain.

@item Docutils@footnote{http://docutils.sourceforge.net}

The underlying reStructuredText parser and toolset.
@end table


@node PEP 343 The ‘with’ statement,PEP 366 Explicit Relative Imports From a Main Module,Changes to the Development Process,What’s New in Python 2 6
@anchor{whatsnew/2 6 pep-0343}@anchor{e54}@anchor{whatsnew/2 6 pep-343-the-with-statement}@anchor{f0e}
@subsection PEP 343: The ‘with’ statement


The previous version, Python 2.5, added the ‘@ref{168,,with}’
statement as an optional feature, to be enabled by a @code{from __future__
import with_statement} directive.  In 2.6 the statement no longer needs to
be specially enabled; this means that @code{with} is now always a
keyword.  The rest of this section is a copy of the corresponding
section from the “What’s New in Python 2.5” document; if you’re
familiar with the ‘@code{with}’ statement
from Python 2.5, you can skip this section.

The ‘@ref{168,,with}’ statement clarifies code that previously would use
@code{try...finally} blocks to ensure that clean-up code is executed.  In this
section, I’ll discuss the statement as it will commonly be used.  In the next
section, I’ll examine the implementation details and show how to write objects
for use with this statement.

The ‘@ref{168,,with}’ statement is a control-flow structure whose basic
structure is:

@example
with expression [as variable]:
    with-block
@end example

The expression is evaluated, and it should result in an object that supports the
context management protocol (that is, has @code{__enter__()} and @code{__exit__()}
methods).

The object’s @code{__enter__()} is called before @emph{with-block} is executed and
therefore can run set-up code. It also may return a value that is bound to the
name @emph{variable}, if given.  (Note carefully that @emph{variable} is @emph{not} assigned
the result of @emph{expression}.)

After execution of the @emph{with-block} is finished, the object’s @code{__exit__()}
method is called, even if the block raised an exception, and can therefore run
clean-up code.

Some standard Python objects now support the context management protocol and can
be used with the ‘@ref{168,,with}’ statement. File objects are one example:

@example
with open('/etc/passwd', 'r') as f:
    for line in f:
        print line
        ... more processing code ...
@end example

After this statement has executed, the file object in @emph{f} will have been
automatically closed, even if the @ref{161,,for} loop raised an exception
part-way through the block.

@cartouche
@quotation Note 
In this case, @emph{f} is the same object created by @ref{1c8,,open()}, because
@code{file.__enter__()} returns @emph{self}.
@end quotation
@end cartouche

The @ref{109,,threading} module’s locks and condition variables  also support the
‘@ref{168,,with}’ statement:

@example
lock = threading.Lock()
with lock:
    # Critical section of code
    ...
@end example

The lock is acquired before the block is executed and always released once  the
block is complete.

The @code{localcontext()} function in the @ref{34,,decimal} module makes it easy
to save and restore the current decimal context, which encapsulates the desired
precision and rounding characteristics for computations:

@example
from decimal import Decimal, Context, localcontext

# Displays with default precision of 28 digits
v = Decimal('578')
print v.sqrt()

with localcontext(Context(prec=16)):
    # All code in this block uses a precision of 16 digits.
    # The original context is restored on exiting the block.
    print v.sqrt()
@end example

@menu
* Writing Context Managers:: 
* The contextlib module:: 

@end menu

@node Writing Context Managers,The contextlib module,,PEP 343 The ‘with’ statement
@anchor{whatsnew/2 6 new-26-context-managers}@anchor{e55}@anchor{whatsnew/2 6 writing-context-managers}@anchor{f0f}
@subsubsection Writing Context Managers


Under the hood, the ‘@ref{168,,with}’ statement is fairly complicated. Most
people will only use ‘@code{with}’ in company with existing objects and
don’t need to know these details, so you can skip the rest of this section if
you like.  Authors of new objects will need to understand the details of the
underlying implementation and should keep reading.

A high-level explanation of the context management protocol is:


@itemize *

@item 
The expression is evaluated and should result in an object called a “context
manager”.  The context manager must have @code{__enter__()} and @code{__exit__()}
methods.

@item 
The context manager’s @code{__enter__()} method is called.  The value returned
is assigned to @emph{VAR}.  If no @code{as VAR} clause is present, the value is simply
discarded.

@item 
The code in @emph{BLOCK} is executed.

@item 
If @emph{BLOCK} raises an exception, the context manager’s @code{__exit__()} method
is called with three arguments, the exception details (@code{type, value, traceback},
the same values returned by @ref{18f,,sys.exc_info()}, which can also be @code{None}
if no exception occurred).  The method’s return value controls whether an exception
is re-raised: any false value re-raises the exception, and @code{True} will result
in suppressing it.  You’ll only rarely want to suppress the exception, because
if you do the author of the code containing the ‘@ref{168,,with}’ statement will
never realize anything went wrong.

@item 
If @emph{BLOCK} didn’t raise an exception,  the @code{__exit__()} method is still
called, but @emph{type}, @emph{value}, and @emph{traceback} are all @code{None}.
@end itemize

Let’s think through an example.  I won’t present detailed code but will only
sketch the methods necessary for a database that supports transactions.

(For people unfamiliar with database terminology: a set of changes to the
database are grouped into a transaction.  Transactions can be either committed,
meaning that all the changes are written into the database, or rolled back,
meaning that the changes are all discarded and the database is unchanged.  See
any database textbook for more information.)

Let’s assume there’s an object representing a database connection. Our goal will
be to let the user write code like this:

@example
db_connection = DatabaseConnection()
with db_connection as cursor:
    cursor.execute('insert into ...')
    cursor.execute('delete from ...')
    # ... more operations ...
@end example

The transaction should be committed if the code in the block runs flawlessly or
rolled back if there’s an exception. Here’s the basic interface for
@code{DatabaseConnection} that I’ll assume:

@example
class DatabaseConnection:
    # Database interface
    def cursor(self):
        "Returns a cursor object and starts a new transaction"
    def commit(self):
        "Commits current transaction"
    def rollback(self):
        "Rolls back current transaction"
@end example

The @code{__enter__()} method is pretty easy, having only to start a new
transaction.  For this application the resulting cursor object would be a useful
result, so the method will return it.  The user can then add @code{as cursor} to
their ‘@ref{168,,with}’ statement to bind the cursor to a variable name.

@example
class DatabaseConnection:
    ...
    def __enter__(self):
        # Code to start a new transaction
        cursor = self.cursor()
        return cursor
@end example

The @code{__exit__()} method is the most complicated because it’s where most of
the work has to be done.  The method has to check if an exception occurred.  If
there was no exception, the transaction is committed.  The transaction is rolled
back if there was an exception.

In the code below, execution will just fall off the end of the function,
returning the default value of @code{None}.  @code{None} is false, so the exception
will be re-raised automatically.  If you wished, you could be more explicit and
add a @ref{4c6,,return} statement at the marked location.

@example
class DatabaseConnection:
    ...
    def __exit__(self, type, value, tb):
        if tb is None:
            # No exception, so commit
            self.commit()
        else:
            # Exception occurred, so rollback.
            self.rollback()
            # return False
@end example

@node The contextlib module,,Writing Context Managers,PEP 343 The ‘with’ statement
@anchor{whatsnew/2 6 new-module-contextlib}@anchor{e56}@anchor{whatsnew/2 6 the-contextlib-module}@anchor{f10}
@subsubsection The contextlib module


The @ref{22,,contextlib} module provides some functions and a decorator that
are useful when writing objects for use with the ‘@ref{168,,with}’ statement.

The decorator is called @code{contextmanager()}, and lets you write a single
generator function instead of defining a new class.  The generator should yield
exactly one value.  The code up to the @ref{4c5,,yield} will be executed as the
@code{__enter__()} method, and the value yielded will be the method’s return
value that will get bound to the variable in the ‘@ref{168,,with}’ statement’s
@code{as} clause, if any.  The code after the @code{yield} will be
executed in the @code{__exit__()} method.  Any exception raised in the block will
be raised by the @code{yield} statement.

Using this decorator, our database example from the previous section
could be written as:

@example
from contextlib import contextmanager

@@contextmanager
def db_transaction(connection):
    cursor = connection.cursor()
    try:
        yield cursor
    except:
        connection.rollback()
        raise
    else:
        connection.commit()

db = DatabaseConnection()
with db_transaction(db) as cursor:
    ...
@end example

The @ref{22,,contextlib} module also has a @code{nested(mgr1, mgr2, ...)} function
that combines a number of context managers so you don’t need to write nested
‘@ref{168,,with}’ statements.  In this example, the single ‘@code{with}’
statement both starts a database transaction and acquires a thread lock:

@example
lock = threading.Lock()
with nested (db_transaction(db), lock) as (cursor, locked):
    ...
@end example

Finally, the @code{closing()} function returns its argument so that it can be
bound to a variable, and calls the argument’s @code{.close()} method at the end
of the block.

@example
import urllib, sys
from contextlib import closing

with closing(urllib.urlopen('http://www.yahoo.com')) as f:
    for line in f:
        sys.stdout.write(line)
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 343

@item PEP 343@footnote{https://www.python.org/dev/peps/pep-0343} - The “with” statement

PEP written by Guido van Rossum and Nick Coghlan; implemented by Mike Bland,
Guido van Rossum, and Neal Norwitz.  The PEP shows the code generated for a
‘@ref{168,,with}’ statement, which can be helpful in learning how the statement
works.
@end table

The documentation  for the @ref{22,,contextlib} module.


@c ======================================================================

@node PEP 366 Explicit Relative Imports From a Main Module,PEP 370 Per-user site-packages Directory,PEP 343 The ‘with’ statement,What’s New in Python 2 6
@anchor{whatsnew/2 6 pep-0366}@anchor{e57}@anchor{whatsnew/2 6 pep-366-explicit-relative-imports-from-a-main-module}@anchor{f11}
@subsection PEP 366: Explicit Relative Imports From a Main Module


Python’s @ref{611,,-m} switch allows running a module as a script.
When you ran a module that was located inside a package, relative
imports didn’t work correctly.

The fix for Python 2.6 adds a @ref{bae,,__package__} attribute to
modules.  When this attribute is present, relative imports will be
relative to the value of this attribute instead of the
@ref{f12,,__name__} attribute.

PEP 302-style importers can then set @ref{bae,,__package__} as necessary.
The @ref{dd,,runpy} module that implements the @ref{611,,-m} switch now
does this, so relative imports will now work correctly in scripts
running from inside a package.

@c ======================================================================

@node PEP 370 Per-user site-packages Directory,PEP 371 The multiprocessing Package,PEP 366 Explicit Relative Imports From a Main Module,What’s New in Python 2 6
@anchor{whatsnew/2 6 pep-0370}@anchor{e58}@anchor{whatsnew/2 6 pep-370-per-user-site-packages-directory}@anchor{f13}
@subsection PEP 370: Per-user @code{site-packages} Directory


When you run Python, the module search path @code{sys.path} usually
includes a directory whose path ends in @code{"site-packages"}.  This
directory is intended to hold locally-installed packages available to
all users using a machine or a particular site installation.

Python 2.6 introduces a convention for user-specific site directories.
The directory varies depending on the platform:


@itemize *

@item 
Unix and Mac OS X: @code{~/.local/}

@item 
Windows: @code{%APPDATA%/Python}
@end itemize

Within this directory, there will be version-specific subdirectories,
such as @code{lib/python2.6/site-packages} on Unix/Mac OS and
@code{Python26/site-packages} on Windows.

If you don’t like the default directory, it can be overridden by an
environment variable.  
@geindex PYTHONUSERBASE
@geindex environment variable; PYTHONUSERBASE
@ref{f14,,PYTHONUSERBASE} sets the root
directory used for all Python versions supporting this feature.  On
Windows, the directory for application-specific data can be changed by
setting the 
@geindex APPDATA
@geindex environment variable; APPDATA
@code{APPDATA} environment variable.  You can also
modify the @code{site.py} file for your Python installation.

The feature can be disabled entirely by running Python with the
@ref{f15,,-s} option or setting the 
@geindex PYTHONNOUSERSITE
@geindex environment variable; PYTHONNOUSERSITE
@ref{f16,,PYTHONNOUSERSITE}
environment variable.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 370

@item PEP 370@footnote{https://www.python.org/dev/peps/pep-0370} - Per-user @code{site-packages} Directory

PEP written and implemented by Christian Heimes.
@end table


@c ======================================================================

@node PEP 371 The multiprocessing Package,PEP 3101 Advanced String Formatting,PEP 370 Per-user site-packages Directory,What’s New in Python 2 6
@anchor{whatsnew/2 6 pep-0371}@anchor{e59}@anchor{whatsnew/2 6 pep-371-the-multiprocessing-package}@anchor{f17}
@subsection PEP 371: The @code{multiprocessing} Package


The new @ref{b3,,multiprocessing} package lets Python programs create new
processes that will perform a computation and return a result to the
parent.  The parent and child processes can communicate using queues
and pipes, synchronize their operations using locks and semaphores,
and can share simple arrays of data.

The @ref{b3,,multiprocessing} module started out as an exact emulation of
the @ref{109,,threading} module using processes instead of threads.  That
goal was discarded along the path to Python 2.6, but the general
approach of the module is still similar.  The fundamental class
is the @code{Process}, which is passed a callable object and
a collection of arguments.  The @code{start()} method
sets the callable running in a subprocess, after which you can call
the @code{is_alive()} method to check whether the subprocess is still running
and the @code{join()} method to wait for the process to exit.

Here’s a simple example where the subprocess will calculate a
factorial.  The function doing the calculation is written strangely so
that it takes significantly longer when the input argument is a
multiple of 4.

@example
import time
from multiprocessing import Process, Queue


def factorial(queue, N):
    "Compute a factorial."
    # If N is a multiple of 4, this function will take much longer.
    if (N % 4) == 0:
        time.sleep(.05 * N/4)

    # Calculate the result
    fact = 1L
    for i in range(1, N+1):
        fact = fact * i

    # Put the result on the queue
    queue.put(fact)

if __name__ == '__main__':
    queue = Queue()

    N = 5

    p = Process(target=factorial, args=(queue, N))
    p.start()
    p.join()

    result = queue.get()
    print 'Factorial', N, '=', result
@end example

A @ref{f18,,Queue} is used to communicate the result of the factorial.
The @ref{f18,,Queue} object is stored in a global variable.
The child process will use the value of the variable when the child
was created; because it’s a @ref{f18,,Queue}, parent and child can use
the object to communicate.  (If the parent were to change the value of
the global variable, the child’s value would be unaffected, and vice
versa.)

Two other classes, @code{Pool} and @code{Manager}, provide
higher-level interfaces.  @code{Pool} will create a fixed number of
worker processes, and requests can then be distributed to the workers
by calling @code{apply()} or @code{apply_async()} to add a single request,
and @ref{2f4,,map()} or @code{map_async()} to add a number of
requests.  The following code uses a @code{Pool} to spread requests
across 5 worker processes and retrieve a list of results:

@example
from multiprocessing import Pool

def factorial(N, dictionary):
    "Compute a factorial."
    ...
p = Pool(5)
result = p.map(factorial, range(1, 1000, 10))
for v in result:
    print v
@end example

This produces the following output:

@example
1
39916800
51090942171709440000
8222838654177922817725562880000000
33452526613163807108170062053440751665152000000000
...
@end example

The other high-level interface, the @code{Manager} class, creates a
separate server process that can hold master copies of Python data
structures.  Other processes can then access and modify these data
structures using proxy objects.  The following example creates a
shared dictionary by calling the @ref{3b0,,dict()} method; the worker
processes then insert values into the dictionary.  (Locking is not
done for you automatically, which doesn’t matter in this example.
@code{Manager}’s methods also include @code{Lock()}, @code{RLock()},
and @code{Semaphore()} to create shared locks.)

@example
import time
from multiprocessing import Pool, Manager

def factorial(N, dictionary):
    "Compute a factorial."
    # Calculate the result
    fact = 1L
    for i in range(1, N+1):
        fact = fact * i

    # Store result in dictionary
    dictionary[N] = fact

if __name__ == '__main__':
    p = Pool(5)
    mgr = Manager()
    d = mgr.dict()         # Create shared dictionary

    # Run tasks using the pool
    for N in range(1, 1000, 10):
        p.apply_async(factorial, (N, d))

    # Mark pool as closed -- no more tasks can be added.
    p.close()

    # Wait for tasks to exit
    p.join()

    # Output results
    for k, v in sorted(d.items()):
        print k, v
@end example

This will produce the output:

@example
1 1
11 39916800
21 51090942171709440000
31 8222838654177922817725562880000000
41 33452526613163807108170062053440751665152000000000
51 15511187532873822802242430164693032110632597200169861120000...
@end example


@subsubheading See also


The documentation for the @ref{b3,,multiprocessing} module.


@table @asis

@geindex Python Enhancement Proposals; PEP 371

@item PEP 371@footnote{https://www.python.org/dev/peps/pep-0371} - Addition of the multiprocessing package

PEP written by Jesse Noller and Richard Oudkerk;
implemented by Richard Oudkerk and Jesse Noller.
@end table


@c ======================================================================

@node PEP 3101 Advanced String Formatting,PEP 3105 print As a Function,PEP 371 The multiprocessing Package,What’s New in Python 2 6
@anchor{whatsnew/2 6 pep-3101}@anchor{e5a}@anchor{whatsnew/2 6 pep-3101-advanced-string-formatting}@anchor{f19}
@subsection PEP 3101: Advanced String Formatting


In Python 3.0, the @cite{%} operator is supplemented by a more powerful string
formatting method, @ref{785,,format()}.  Support for the @ref{784,,str.format()} method
has been backported to Python 2.6.

In 2.6, both 8-bit and Unicode strings have a @cite{.format()} method that
treats the string as a template and takes the arguments to be formatted.
The formatting template uses curly brackets (@cite{@{}, @cite{@}}) as special characters:

@example
>>> # Substitute positional argument 0 into the string.
>>> "User ID: @{0@}".format("root")
'User ID: root'
>>> # Use the named keyword arguments
>>> "User ID: @{uid@}   Last seen: @{last_login@}".format(
...    uid="root",
...    last_login = "5 Mar 2008 07:20")
'User ID: root   Last seen: 5 Mar 2008 07:20'
@end example

Curly brackets can be escaped by doubling them:

@example
>>> "Empty dict: @{@{@}@}".format()
"Empty dict: @{@}"
@end example

Field names can be integers indicating positional arguments, such as
@code{@{0@}}, @code{@{1@}}, etc. or names of keyword arguments.  You can also
supply compound field names that read attributes or access dictionary keys:

@example
>>> import sys
>>> print 'Platform: @{0.platform@}\nPython version: @{0.version@}'.format(sys)
Platform: darwin
Python version: 2.6a1+ (trunk:61261M, Mar  5 2008, 20:29:41)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)]'

>>> import mimetypes
>>> 'Content-type: @{0[.mp4]@}'.format(mimetypes.types_map)
'Content-type: video/mp4'
@end example

Note that when using dictionary-style notation such as @code{[.mp4]}, you
don’t need to put any quotation marks around the string; it will look
up the value using @code{.mp4} as the key.  Strings beginning with a
number will be converted to an integer.  You can’t write more
complicated expressions inside a format string.

So far we’ve shown how to specify which field to substitute into the
resulting string.  The precise formatting used is also controllable by
adding a colon followed by a format specifier.  For example:

@example
>>> # Field 0: left justify, pad to 15 characters
>>> # Field 1: right justify, pad to 6 characters
>>> fmt = '@{0:15@} $@{1:>6@}'
>>> fmt.format('Registration', 35)
'Registration    $    35'
>>> fmt.format('Tutorial', 50)
'Tutorial        $    50'
>>> fmt.format('Banquet', 125)
'Banquet         $   125'
@end example

Format specifiers can reference other fields through nesting:

@example
>>> fmt = '@{0:@{1@}@}'
>>> width = 15
>>> fmt.format('Invoice #1234', width)
'Invoice #1234  '
>>> width = 35
>>> fmt.format('Invoice #1234', width)
'Invoice #1234                      '
@end example

The alignment of a field within the desired width can be specified:


@multitable {xxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Character

@tab

Effect

@item

< (default)

@tab

Left-align

@item

>

@tab

Right-align

@item

^

@tab

Center

@item

=

@tab

(For numeric types only) Pad after the sign.

@end multitable


Format specifiers can also include a presentation type, which
controls how the value is formatted.  For example, floating-point numbers
can be formatted as a general number or in exponential notation:

@example
>>> '@{0:g@}'.format(3.75)
'3.75'
>>> '@{0:e@}'.format(3.75)
'3.750000e+00'
@end example

A variety of presentation types are available.  Consult the 2.6
documentation for a @ref{f1a,,complete list}; here’s a sample:


@multitable {xxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{b}

@tab

Binary. Outputs the number in base 2.

@item

@code{c}

@tab

Character. Converts the integer to the corresponding Unicode character
before printing.

@item

@code{d}

@tab

Decimal Integer. Outputs the number in base 10.

@item

@code{o}

@tab

Octal format. Outputs the number in base 8.

@item

@code{x}

@tab

Hex format. Outputs the number in base 16, using lower-case letters for
the digits above 9.

@item

@code{e}

@tab

Exponent notation. Prints the number in scientific notation using the
letter ‘e’ to indicate the exponent.

@item

@code{g}

@tab

General format. This prints the number as a fixed-point number, unless
the number is too large, in which case it switches to ‘e’ exponent
notation.

@item

@code{n}

@tab

Number. This is the same as ‘g’ (for floats) or ‘d’ (for integers),
except that it uses the current locale setting to insert the appropriate
number separator characters.

@item

@code{%}

@tab

Percentage. Multiplies the number by 100 and displays in fixed (‘f’)
format, followed by a percent sign.

@end multitable


Classes and types can define a @code{__format__()} method to control how they’re
formatted.  It receives a single argument, the format specifier:

@example
def __format__(self, format_spec):
    if isinstance(format_spec, unicode):
        return unicode(str(self))
    else:
        return str(self)
@end example

There’s also a @ref{785,,format()} builtin that will format a single
value.  It calls the type’s @code{__format__()} method with the
provided specifier:

@example
>>> format(75.6564, '.2f')
'75.66'
@end example


@subsubheading See also



@table @asis

@item @ref{f1a,,Format String Syntax}

The reference documentation for format fields.

@geindex Python Enhancement Proposals; PEP 3101

@item PEP 3101@footnote{https://www.python.org/dev/peps/pep-3101} - Advanced String Formatting

PEP written by Talin. Implemented by Eric Smith.
@end table


@c ======================================================================

@node PEP 3105 print As a Function,PEP 3110 Exception-Handling Changes,PEP 3101 Advanced String Formatting,What’s New in Python 2 6
@anchor{whatsnew/2 6 pep-3105}@anchor{e5b}@anchor{whatsnew/2 6 pep-3105-print-as-a-function}@anchor{f1b}
@subsection PEP 3105: @code{print} As a Function


The @code{print} statement becomes the @ref{af6,,print()} function in Python 3.0.
Making @ref{af6,,print()} a function makes it possible to replace the function
by doing @code{def print(...)} or importing a new function from somewhere else.

Python 2.6 has a @code{__future__} import that removes @code{print} as language
syntax, letting you use the functional form instead.  For example:

@example
>>> from __future__ import print_function
>>> print('# of entries', len(dictionary), file=sys.stderr)
@end example

The signature of the new function is:

@example
def print(*args, sep=' ', end='\n', file=None)
@end example

The parameters are:

@quotation


@itemize *

@item 
@emph{args}: positional arguments whose values will be printed out.

@item 
@emph{sep}: the separator, which will be printed between arguments.

@item 
@emph{end}: the ending text, which will be printed after all of the
arguments have been output.

@item 
@emph{file}: the file object to which the output will be sent.
@end itemize
@end quotation


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3105

@item PEP 3105@footnote{https://www.python.org/dev/peps/pep-3105} - Make print a function

PEP written by Georg Brandl.
@end table


@c ======================================================================

@node PEP 3110 Exception-Handling Changes,PEP 3112 Byte Literals,PEP 3105 print As a Function,What’s New in Python 2 6
@anchor{whatsnew/2 6 pep-3110}@anchor{e5c}@anchor{whatsnew/2 6 pep-3110-exception-handling-changes}@anchor{f1c}
@subsection PEP 3110: Exception-Handling Changes


One error that Python programmers occasionally make
is writing the following code:

@example
try:
    ...
except TypeError, ValueError:  # Wrong!
    ...
@end example

The author is probably trying to catch both @ref{162,,TypeError} and
@ref{17c,,ValueError} exceptions, but this code actually does something
different: it will catch @ref{162,,TypeError} and bind the resulting
exception object to the local name @code{"ValueError"}.  The
@ref{17c,,ValueError} exception will not be caught at all.  The correct
code specifies a tuple of exceptions:

@example
try:
    ...
except (TypeError, ValueError):
    ...
@end example

This error happens because the use of the comma here is ambiguous:
does it indicate two different nodes in the parse tree, or a single
node that’s a tuple?

Python 3.0 makes this unambiguous by replacing the comma with the word
“as”.  To catch an exception and store the exception object in the
variable @code{exc}, you must write:

@example
try:
    ...
except TypeError as exc:
    ...
@end example

Python 3.0 will only support the use of “as”, and therefore interprets
the first example as catching two different exceptions.  Python 2.6
supports both the comma and “as”, so existing code will continue to
work.  We therefore suggest using “as” when writing new Python code
that will only be executed with 2.6.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3110

@item PEP 3110@footnote{https://www.python.org/dev/peps/pep-3110} - Catching Exceptions in Python 3000

PEP written and implemented by Collin Winter.
@end table


@c ======================================================================

@node PEP 3112 Byte Literals,PEP 3116 New I/O Library,PEP 3110 Exception-Handling Changes,What’s New in Python 2 6
@anchor{whatsnew/2 6 pep-3112}@anchor{e5d}@anchor{whatsnew/2 6 pep-3112-byte-literals}@anchor{f1d}
@subsection PEP 3112: Byte Literals


Python 3.0 adopts Unicode as the language’s fundamental string type and
denotes 8-bit literals differently, either as @code{b'string'}
or using a @ref{2f0,,bytes} constructor.  For future compatibility,
Python 2.6 adds @ref{2f0,,bytes} as a synonym for the @ref{2ef,,str} type,
and it also supports the @code{b'@w{'}} notation.

The 2.6 @ref{2ef,,str} differs from 3.0’s @ref{2f0,,bytes} type in various
ways; most notably, the constructor is completely different.  In 3.0,
@code{bytes([65, 66, 67])} is 3 elements long, containing the bytes
representing @code{ABC}; in 2.6, @code{bytes([65, 66, 67])} returns the
12-byte string representing the @ref{2ef,,str()} of the list.

The primary use of @ref{2f0,,bytes} in 2.6 will be to write tests of
object type such as @code{isinstance(x, bytes)}.  This will help the 2to3
converter, which can’t tell whether 2.x code intends strings to
contain either characters or 8-bit bytes; you can now
use either @ref{2f0,,bytes} or @ref{2ef,,str} to represent your intention
exactly, and the resulting code will also be correct in Python 3.0.

There’s also a @code{__future__} import that causes all string literals
to become Unicode strings.  This means that @code{\u} escape sequences
can be used to include Unicode characters:

@example
from __future__ import unicode_literals

s = ('\u751f\u3080\u304e\u3000\u751f\u3054'
     '\u3081\u3000\u751f\u305f\u307e\u3054')

print len(s)               # 12 Unicode characters
@end example

At the C level, Python 3.0 will rename the existing 8-bit
string type, called @code{PyStringObject} in Python 2.x,
to @ref{208,,PyBytesObject}.  Python 2.6 uses @code{#define}
to support using the names @ref{208,,PyBytesObject()},
@ref{f1e,,PyBytes_Check()}, @ref{f1f,,PyBytes_FromStringAndSize()},
and all the other functions and macros used with strings.

Instances of the @ref{2f0,,bytes} type are immutable just
as strings are.  A new @ref{2f1,,bytearray} type stores a mutable
sequence of bytes:

@example
>>> bytearray([65, 66, 67])
bytearray(b'ABC')
>>> b = bytearray(u'\u21ef\u3244', 'utf-8')
>>> b
bytearray(b'\xe2\x87\xaf\xe3\x89\x84')
>>> b[0] = '\xe3'
>>> b
bytearray(b'\xe3\x87\xaf\xe3\x89\x84')
>>> unicode(str(b), 'utf-8')
u'\u31ef \u3244'
@end example

Byte arrays support most of the methods of string types, such as
@code{startswith()}/@code{endswith()}, @code{find()}/@code{rfind()},
and some of the methods of lists, such as @code{append()},
@code{pop()},  and @code{reverse()}.

@example
>>> b = bytearray('ABC')
>>> b.append('d')
>>> b.append(ord('e'))
>>> b
bytearray(b'ABCde')
@end example

There’s also a corresponding C API, with
@ref{f20,,PyByteArray_FromObject()},
@ref{f21,,PyByteArray_FromStringAndSize()},
and various other functions.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3112

@item PEP 3112@footnote{https://www.python.org/dev/peps/pep-3112} - Bytes literals in Python 3000

PEP written by Jason Orendorff; backported to 2.6 by Christian Heimes.
@end table


@c ======================================================================

@node PEP 3116 New I/O Library,PEP 3118 Revised Buffer Protocol,PEP 3112 Byte Literals,What’s New in Python 2 6
@anchor{whatsnew/2 6 pep-3116}@anchor{e5e}@anchor{whatsnew/2 6 pep-3116-new-i-o-library}@anchor{f22}
@subsection PEP 3116: New I/O Library


Python’s built-in file objects support a number of methods, but
file-like objects don’t necessarily support all of them.  Objects that
imitate files usually support @code{read()} and @code{write()}, but they
may not support @ref{d9,,readline()}, for example.  Python 3.0 introduces
a layered I/O library in the @ref{9d,,io} module that separates buffering
and text-handling features from the fundamental read and write
operations.

There are three levels of abstract base classes provided by
the @ref{9d,,io} module:


@itemize *

@item 
@code{RawIOBase} defines raw I/O operations: @code{read()},
@code{readinto()},
@code{write()}, @code{seek()}, @code{tell()}, @code{truncate()},
and @code{close()}.
Most of the methods of this class will often map to a single system call.
There are also @code{readable()}, @code{writable()}, and @code{seekable()}
methods for determining what operations a given object will allow.

Python 3.0 has concrete implementations of this class for files and
sockets, but Python 2.6 hasn’t restructured its file and socket objects
in this way.

@c XXX should 2.6 register them in io.py?

@item 
@code{BufferedIOBase} is an abstract base class that
buffers data in memory to reduce the number of
system calls used, making I/O processing more efficient.
It supports all of the methods of @code{RawIOBase},
and adds a @code{raw} attribute holding the underlying raw object.

There are five concrete classes implementing this ABC.
@code{BufferedWriter} and @code{BufferedReader} are for objects
that support write-only or read-only usage that have a @code{seek()}
method for random access.  @code{BufferedRandom} objects support
read and write access upon the same underlying stream, and
@code{BufferedRWPair} is for objects such as TTYs that have both
read and write operations acting upon unconnected streams of data.
The @code{BytesIO} class supports reading, writing, and seeking
over an in-memory buffer.

@geindex universal newlines; What's new

@item 
@code{TextIOBase}: Provides functions for reading and writing
strings (remember, strings will be Unicode in Python 3.0),
and supporting @ref{88a,,universal newlines}.  @code{TextIOBase} defines
the @ref{d9,,readline()} method and supports iteration upon
objects.

There are two concrete implementations.  @code{TextIOWrapper}
wraps a buffered I/O object, supporting all of the methods for
text I/O and adding a @code{buffer} attribute for access
to the underlying object.  @code{StringIO} simply buffers
everything in memory without ever writing anything to disk.

(In Python 2.6, @ref{aa1,,io.StringIO} is implemented in
pure Python, so it’s pretty slow.   You should therefore stick with the
existing @code{StringIO} module or @code{cStringIO} for now.  At some
point Python 3.0’s @ref{9d,,io} module will be rewritten into C for speed,
and perhaps the C implementation will be  backported to the 2.x releases.)
@end itemize

In Python 2.6, the underlying implementations haven’t been
restructured to build on top of the @ref{9d,,io} module’s classes.  The
module is being provided to make it easier to write code that’s
forward-compatible with 3.0, and to save developers the effort of writing
their own implementations of buffering and text I/O.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3116

@item PEP 3116@footnote{https://www.python.org/dev/peps/pep-3116} - New I/O

PEP written by Daniel Stutzbach, Mike Verdone, and Guido van Rossum.
Code by Guido van Rossum, Georg Brandl, Walter Doerwald,
Jeremy Hylton, Martin von Löwis, Tony Lownds, and others.
@end table


@c ======================================================================

@node PEP 3118 Revised Buffer Protocol,PEP 3119 Abstract Base Classes,PEP 3116 New I/O Library,What’s New in Python 2 6
@anchor{whatsnew/2 6 pep-3118}@anchor{e5f}@anchor{whatsnew/2 6 pep-3118-revised-buffer-protocol}@anchor{f23}
@subsection PEP 3118: Revised Buffer Protocol


The buffer protocol is a C-level API that lets Python types
exchange pointers into their internal representations.  A
memory-mapped file can be viewed as a buffer of characters, for
example, and this lets another module such as @ref{d8,,re}
treat memory-mapped files as a string of characters to be searched.

The primary users of the buffer protocol are numeric-processing
packages such as NumPy, which expose the internal representation
of arrays so that callers can write data directly into an array instead
of going through a slower API.  This PEP updates the buffer protocol in light of experience
from NumPy development, adding a number of new features
such as indicating the shape of an array or locking a memory region.

The most important new C API function is
@code{PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags)}, which
takes an object and a set of flags, and fills in the
@code{Py_buffer} structure with information
about the object’s memory representation.  Objects
can use this operation to lock memory in place
while an external caller could be modifying the contents,
so there’s a corresponding @code{PyBuffer_Release(Py_buffer *view)} to
indicate that the external caller is done.

@c XXX PyObject_GetBuffer not documented in c-api

The @emph{flags} argument to @ref{1dd,,PyObject_GetBuffer()} specifies
constraints upon the memory returned.  Some examples are:

@quotation


@itemize *

@item 
@code{PyBUF_WRITABLE} indicates that the memory must be writable.

@item 
@code{PyBUF_LOCK} requests a read-only or exclusive lock on the memory.

@item 
@code{PyBUF_C_CONTIGUOUS} and @code{PyBUF_F_CONTIGUOUS}
requests a C-contiguous (last dimension varies the fastest) or
Fortran-contiguous (first dimension varies the fastest) array layout.
@end itemize
@end quotation

Two new argument codes for @ref{37f,,PyArg_ParseTuple()},
@code{s*} and @code{z*}, return locked buffer objects for a parameter.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3118

@item PEP 3118@footnote{https://www.python.org/dev/peps/pep-3118} - Revising the buffer protocol

PEP written by Travis Oliphant and Carl Banks; implemented by
Travis Oliphant.
@end table


@c ======================================================================

@node PEP 3119 Abstract Base Classes,PEP 3127 Integer Literal Support and Syntax,PEP 3118 Revised Buffer Protocol,What’s New in Python 2 6
@anchor{whatsnew/2 6 pep-3119}@anchor{e60}@anchor{whatsnew/2 6 pep-3119-abstract-base-classes}@anchor{f24}
@subsection PEP 3119: Abstract Base Classes


Some object-oriented languages such as Java support interfaces,
declaring that a class has a given set of methods or supports a given
access protocol.  Abstract Base Classes (or ABCs) are an equivalent
feature for Python. The ABC support consists of an @ref{3,,abc} module
containing a metaclass called @code{ABCMeta}, special handling of
this metaclass by the @ref{236,,isinstance()} and @ref{237,,issubclass()}
builtins, and a collection of basic ABCs that the Python developers
think will be widely useful.  Future versions of Python will probably
add more ABCs.

Let’s say you have a particular class and wish to know whether it supports
dictionary-style access.  The phrase “dictionary-style” is vague, however.
It probably means that accessing items with @code{obj[1]} works.
Does it imply that setting items with @code{obj[2] = value} works?
Or that the object will have @code{keys()}, @code{values()}, and @code{items()}
methods?  What about the iterative variants  such as @code{iterkeys()}?  @ref{24,,copy()}
and @code{update()}?  Iterating over the object with @ref{251,,iter()}?

The Python 2.6 @ref{1c,,collections} module includes a number of
different ABCs that represent these distinctions.  @code{Iterable}
indicates that a class defines @code{__iter__()}, and
@code{Container} means the class defines a @code{__contains__()}
method and therefore supports @code{x in y} expressions.  The basic
dictionary interface of getting items, setting items, and
@code{keys()}, @code{values()}, and @code{items()}, is defined by the
@code{MutableMapping} ABC.

You can derive your own classes from a particular ABC
to indicate they support that ABC’s interface:

@example
import collections

class Storage(collections.MutableMapping):
    ...
@end example

Alternatively, you could write the class without deriving from
the desired ABC and instead register the class by
calling the ABC’s @code{register()} method:

@example
import collections

class Storage:
    ...

collections.MutableMapping.register(Storage)
@end example

For classes that you write, deriving from the ABC is probably clearer.
The @code{register()}  method is useful when you’ve written a new
ABC that can describe an existing type or class, or if you want
to declare that some third-party class implements an ABC.
For example, if you defined a @code{PrintableType} ABC,
it’s legal to do:

@example
# Register Python's types
PrintableType.register(int)
PrintableType.register(float)
PrintableType.register(str)
@end example

Classes should obey the semantics specified by an ABC, but
Python can’t check this; it’s up to the class author to
understand the ABC’s requirements and to implement the code accordingly.

To check whether an object supports a particular interface, you can
now write:

@example
def func(d):
    if not isinstance(d, collections.MutableMapping):
        raise ValueError("Mapping object expected, not %r" % d)
@end example

Don’t feel that you must now begin writing lots of checks as in the
above example.  Python has a strong tradition of duck-typing, where
explicit type-checking is never done and code simply calls methods on
an object, trusting that those methods will be there and raising an
exception if they aren’t.  Be judicious in checking for ABCs and only
do it where it’s absolutely necessary.

You can write your own ABCs by using @code{abc.ABCMeta} as the
metaclass in a class definition:

@example
from abc import ABCMeta, abstractmethod

class Drawable():
    __metaclass__ = ABCMeta

    @@abstractmethod
    def draw(self, x, y, scale=1.0):
        pass

    def draw_doubled(self, x, y):
        self.draw(x, y, scale=2.0)


class Square(Drawable):
    def draw(self, x, y, scale):
        ...
@end example

In the @code{Drawable} ABC above, the @code{draw_doubled()} method
renders the object at twice its size and can be implemented in terms
of other methods described in @code{Drawable}.  Classes implementing
this ABC therefore don’t need to provide their own implementation
of @code{draw_doubled()}, though they can do so.  An implementation
of @code{draw()} is necessary, though; the ABC can’t provide
a useful generic implementation.

You can apply the @code{@@abstractmethod} decorator to methods such as
@code{draw()} that must be implemented; Python will then raise an
exception for classes that don’t define the method.
Note that the exception is only raised when you actually
try to create an instance of a subclass lacking the method:

@example
>>> class Circle(Drawable):
...     pass
...
>>> c = Circle()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class Circle with abstract methods draw
>>>
@end example

Abstract data attributes can be declared using the
@code{@@abstractproperty} decorator:

@example
from abc import abstractproperty
...

@@abstractproperty
def readonly(self):
   return self._x
@end example

Subclasses must then define a @code{readonly()} property.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3119

@item PEP 3119@footnote{https://www.python.org/dev/peps/pep-3119} - Introducing Abstract Base Classes

PEP written by Guido van Rossum and Talin.
Implemented by Guido van Rossum.
Backported to 2.6 by Benjamin Aranguren, with Alex Martelli.
@end table


@c ======================================================================

@node PEP 3127 Integer Literal Support and Syntax,PEP 3129 Class Decorators,PEP 3119 Abstract Base Classes,What’s New in Python 2 6
@anchor{whatsnew/2 6 pep-3127}@anchor{e61}@anchor{whatsnew/2 6 pep-3127-integer-literal-support-and-syntax}@anchor{f25}
@subsection PEP 3127: Integer Literal Support and Syntax


Python 3.0 changes the syntax for octal (base-8) integer literals,
prefixing them with “0o” or “0O” instead of a leading zero, and adds
support for binary (base-2) integer literals, signalled by a “0b” or
“0B” prefix.

Python 2.6 doesn’t drop support for a leading 0 signalling
an octal number, but it does add support for “0o” and “0b”:

@example
>>> 0o21, 2*8 + 1
(17, 17)
>>> 0b101111
47
@end example

The @ref{e6e,,oct()} builtin still returns numbers
prefixed with a leading zero, and a new @ref{e4e,,bin()}
builtin returns the binary representation for a number:

@example
>>> oct(42)
'052'
>>> future_builtins.oct(42)
'0o52'
>>> bin(173)
'0b10101101'
@end example

The @ref{1b1,,int()} and @code{long()} builtins will now accept the “0o”
and “0b” prefixes when base-8 or base-2 are requested, or when the
@emph{base} argument is zero (signalling that the base used should be
determined from the string):

@example
>>> int ('0o52', 0)
42
>>> int('1101', 2)
13
>>> int('0b1101', 2)
13
>>> int('0b1101', 0)
13
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3127

@item PEP 3127@footnote{https://www.python.org/dev/peps/pep-3127} - Integer Literal Support and Syntax

PEP written by Patrick Maupin; backported to 2.6 by
Eric Smith.
@end table


@c ======================================================================

@node PEP 3129 Class Decorators,PEP 3141 A Type Hierarchy for Numbers,PEP 3127 Integer Literal Support and Syntax,What’s New in Python 2 6
@anchor{whatsnew/2 6 pep-3129}@anchor{e62}@anchor{whatsnew/2 6 pep-3129-class-decorators}@anchor{f26}
@subsection PEP 3129: Class Decorators


Decorators have been extended from functions to classes.  It’s now legal to
write:

@example
@@foo
@@bar
class A:
  pass
@end example

This is equivalent to:

@example
class A:
  pass

A = foo(bar(A))
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3129

@item PEP 3129@footnote{https://www.python.org/dev/peps/pep-3129} - Class Decorators

PEP written by Collin Winter.
@end table


@c ======================================================================

@node PEP 3141 A Type Hierarchy for Numbers,Other Language Changes<13>,PEP 3129 Class Decorators,What’s New in Python 2 6
@anchor{whatsnew/2 6 pep-3141}@anchor{e63}@anchor{whatsnew/2 6 pep-3141-a-type-hierarchy-for-numbers}@anchor{f27}
@subsection PEP 3141: A Type Hierarchy for Numbers


Python 3.0 adds several abstract base classes for numeric types
inspired by Scheme’s numeric tower.  These classes were backported to
2.6 as the @ref{bd,,numbers} module.

The most general ABC is @code{Number}.  It defines no operations at
all, and only exists to allow checking if an object is a number by
doing @code{isinstance(obj, Number)}.

@code{Complex} is a subclass of @code{Number}.  Complex numbers
can undergo the basic operations of addition, subtraction,
multiplication, division, and exponentiation, and you can retrieve the
real and imaginary parts and obtain a number’s conjugate.  Python’s built-in
complex type is an implementation of @code{Complex}.

@code{Real} further derives from @code{Complex}, and adds
operations that only work on real numbers: @code{floor()}, @code{trunc()},
rounding, taking the remainder mod N, floor division,
and comparisons.

@code{Rational} numbers derive from @code{Real}, have
@code{numerator} and @code{denominator} properties, and can be
converted to floats.  Python 2.6 adds a simple rational-number class,
@code{Fraction}, in the @ref{7d,,fractions} module.  (It’s called
@code{Fraction} instead of @code{Rational} to avoid
a name clash with @ref{e64,,numbers.Rational}.)

@code{Integral} numbers derive from @code{Rational}, and
can be shifted left and right with @code{<<} and @code{>>},
combined using bitwise operations such as @code{&} and @code{|},
and can be used as array indexes and slice boundaries.

In Python 3.0, the PEP slightly redefines the existing builtins
@ref{e75,,round()}, @ref{f28,,math.floor()}, @ref{f29,,math.ceil()}, and adds a new
one, @ref{f2a,,math.trunc()}, that’s been backported to Python 2.6.
@ref{f2a,,math.trunc()} rounds toward zero, returning the closest
@code{Integral} that’s between the function’s argument and zero.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3141

@item PEP 3141@footnote{https://www.python.org/dev/peps/pep-3141} - A Type Hierarchy for Numbers

PEP written by Jeffrey Yasskin.
@end table

Scheme's numerical tower@footnote{https://www.gnu.org/software/guile/manual/html_node/Numerical-Tower.html#Numerical-Tower}, from the Guile manual.

Scheme's number datatypes@footnote{http://schemers.org/Documents/Standards/R5RS/HTML/r5rs-Z-H-9.html#%_sec_6.2} from the R5RS Scheme specification.


@menu
* The fractions Module:: 

@end menu

@node The fractions Module,,,PEP 3141 A Type Hierarchy for Numbers
@anchor{whatsnew/2 6 the-fractions-module}@anchor{f2b}
@subsubsection The @code{fractions} Module


To fill out the hierarchy of numeric types, the @ref{7d,,fractions}
module provides a rational-number class.  Rational numbers store their
values as a numerator and denominator forming a fraction, and can
exactly represent numbers such as @code{2/3} that floating-point numbers
can only approximate.

The @code{Fraction} constructor takes two @code{Integral} values
that will be the numerator and denominator of the resulting fraction.

@example
>>> from fractions import Fraction
>>> a = Fraction(2, 3)
>>> b = Fraction(2, 5)
>>> float(a), float(b)
(0.66666666666666663, 0.40000000000000002)
>>> a+b
Fraction(16, 15)
>>> a/b
Fraction(5, 3)
@end example

For converting floating-point numbers to rationals,
the float type now has an @code{as_integer_ratio()} method that returns
the numerator and denominator for a fraction that evaluates to the same
floating-point value:

@example
>>> (2.5) .as_integer_ratio()
(5, 2)
>>> (3.1415) .as_integer_ratio()
(7074029114692207L, 2251799813685248L)
>>> (1./3) .as_integer_ratio()
(6004799503160661L, 18014398509481984L)
@end example

Note that values that can only be approximated by floating-point
numbers, such as 1./3, are not simplified to the number being
approximated; the fraction attempts to match the floating-point value
@strong{exactly}.

The @ref{7d,,fractions} module is based upon an implementation by Sjoerd
Mullender that was in Python’s @code{Demo/classes/} directory for a
long time.  This implementation was significantly updated by Jeffrey
Yasskin.

@node Other Language Changes<13>,New and Improved Modules<2>,PEP 3141 A Type Hierarchy for Numbers,What’s New in Python 2 6
@anchor{whatsnew/2 6 other-language-changes}@anchor{f2c}
@subsection Other Language Changes


Some smaller changes made to the core Python language are:


@itemize *

@item 
Directories and zip archives containing a @code{__main__.py} file
can now be executed directly by passing their name to the
interpreter. The directory or zip archive is automatically inserted
as the first entry in sys.path.  (Suggestion and initial patch by
Andy Chu, subsequently revised by Phillip J. Eby and Nick Coghlan;
bpo-1739468@footnote{https://bugs.python.org/issue1739468}.)

@item 
The @ref{707,,hasattr()} function was catching and ignoring all errors,
under the assumption that they meant a @code{__getattr__()} method
was failing somehow and the return value of @ref{707,,hasattr()} would
therefore be @code{False}.  This logic shouldn’t be applied to
@ref{4ca,,KeyboardInterrupt} and @ref{892,,SystemExit}, however; Python 2.6
will no longer discard such exceptions when @ref{707,,hasattr()}
encounters them.  (Fixed by Benjamin Peterson; bpo-2196@footnote{https://bugs.python.org/issue2196}.)

@item 
When calling a function using the @code{**} syntax to provide keyword
arguments, you are no longer required to use a Python dictionary;
any mapping will now work:

@example
>>> def f(**kw):
...    print sorted(kw)
...
>>> ud=UserDict.UserDict()
>>> ud['a'] = 1
>>> ud['b'] = 'string'
>>> f(**ud)
['a', 'b']
@end example

(Contributed by Alexander Belopolsky; bpo-1686487@footnote{https://bugs.python.org/issue1686487}.)

It’s also become legal to provide keyword arguments after a @code{*args} argument
to a function call.

@example
>>> def f(*args, **kw):
...     print args, kw
...
>>> f(1,2,3, *(4,5,6), keyword=13)
(1, 2, 3, 4, 5, 6) @{'keyword': 13@}
@end example

Previously this would have been a syntax error.
(Contributed by Amaury Forgeot d’Arc; bpo-3473@footnote{https://bugs.python.org/issue3473}.)

@item 
A new builtin, @code{next(iterator, [default])} returns the next item
from the specified iterator.  If the @emph{default} argument is supplied,
it will be returned if @emph{iterator} has been exhausted; otherwise,
the @ref{736,,StopIteration} exception will be raised.  (Backported
in bpo-2719@footnote{https://bugs.python.org/issue2719}.)

@item 
Tuples now have @code{index()} and @code{count()} methods matching the
list type’s @code{index()} and @code{count()} methods:

@example
>>> t = (0,1,2,3,4,0,1,2)
>>> t.index(3)
3
>>> t.count(0)
2
@end example

(Contributed by Raymond Hettinger)

@item 
The built-in types now have improved support for extended slicing syntax,
accepting various combinations of @code{(start, stop, step)}.
Previously, the support was partial and certain corner cases wouldn’t work.
(Implemented by Thomas Wouters.)

@c Revision 57619

@item 
Properties now have three attributes, @code{getter}, @code{setter}
and @code{deleter}, that are decorators providing useful shortcuts
for adding a getter, setter or deleter function to an existing
property. You would use them like this:

@example
class C(object):
    @@property
    def x(self):
        return self._x

    @@x.setter
    def x(self, value):
        self._x = value

    @@x.deleter
    def x(self):
        del self._x

class D(C):
    @@C.x.getter
    def x(self):
        return self._x * 2

    @@x.setter
    def x(self, value):
        self._x = value / 2
@end example

@item 
Several methods of the built-in set types now accept multiple iterables:
@code{intersection()},
@code{intersection_update()},
@code{union()}, @code{update()},
@code{difference()} and @code{difference_update()}.

@example
>>> s=set('1234567890')
>>> s.intersection('abc123', 'cdf246')  # Intersection between all inputs
set(['2'])
>>> s.difference('246', '789')
set(['1', '0', '3', '5'])
@end example

(Contributed by Raymond Hettinger.)

@item 
Many floating-point features were added.  The @ref{256,,float()} function
will now turn the string @code{nan} into an
IEEE 754 Not A Number value, and @code{+inf} and @code{-inf} into
positive or negative infinity.  This works on any platform with
IEEE 754 semantics.  (Contributed by Christian Heimes; bpo-1635@footnote{https://bugs.python.org/issue1635}.)

Other functions in the @ref{ad,,math} module, @code{isinf()} and
@code{isnan()}, return true if their floating-point argument is
infinite or Not A Number.  (bpo-1640@footnote{https://bugs.python.org/issue1640})

Conversion functions were added to convert floating-point numbers
into hexadecimal strings (bpo-3008@footnote{https://bugs.python.org/issue3008}).  These functions
convert floats to and from a string representation without
introducing rounding errors from the conversion between decimal and
binary.  Floats have a @ref{e6f,,hex()} method that returns a string
representation, and the @code{float.fromhex()} method converts a string
back into a number:

@example
>>> a = 3.75
>>> a.hex()
'0x1.e000000000000p+1'
>>> float.fromhex('0x1.e000000000000p+1')
3.75
>>> b=1./3
>>> b.hex()
'0x1.5555555555555p-2'
@end example

@item 
A numerical nicety: when creating a complex number from two floats
on systems that support signed zeros (-0 and +0), the
@ref{33b,,complex()} constructor will now preserve the sign
of the zero.  (Fixed by Mark T. Dickinson; bpo-1507@footnote{https://bugs.python.org/issue1507}.)

@item 
Classes that inherit a @code{__hash__()} method from a parent class
can set @code{__hash__ = None} to indicate that the class isn’t
hashable.  This will make @code{hash(obj)} raise a @ref{162,,TypeError}
and the class will not be indicated as implementing the
@code{Hashable} ABC.

You should do this when you’ve defined a @code{__cmp__()} or
@code{__eq__()} method that compares objects by their value rather
than by identity.  All objects have a default hash method that uses
@code{id(obj)} as the hash value.  There’s no tidy way to remove the
@code{__hash__()} method inherited from a parent class, so
assigning @code{None} was implemented as an override.  At the
C level, extensions can set @code{tp_hash} to
@ref{f2d,,PyObject_HashNotImplemented()}.
(Fixed by Nick Coghlan and Amaury Forgeot d’Arc; bpo-2235@footnote{https://bugs.python.org/issue2235}.)

@item 
The @ref{f2e,,GeneratorExit} exception now subclasses
@ref{15f,,BaseException} instead of @ref{4d8,,Exception}.  This means
that an exception handler that does @code{except Exception:}
will not inadvertently catch @ref{f2e,,GeneratorExit}.
(Contributed by Chad Austin; bpo-1537@footnote{https://bugs.python.org/issue1537}.)

@item 
Generator objects now have a @code{gi_code} attribute that refers to
the original code object backing the generator.
(Contributed by Collin Winter; bpo-1473257@footnote{https://bugs.python.org/issue1473257}.)

@item 
The @ref{4e2,,compile()} built-in function now accepts keyword arguments
as well as positional parameters.  (Contributed by Thomas Wouters;
bpo-1444529@footnote{https://bugs.python.org/issue1444529}.)

@item 
The @ref{33b,,complex()} constructor now accepts strings containing
parenthesized complex numbers, meaning that @code{complex(repr(cplx))}
will now round-trip values.  For example, @code{complex('(3+4j)')}
now returns the value (3+4j).  (bpo-1491866@footnote{https://bugs.python.org/issue1491866})

@item 
The string @code{translate()} method now accepts @code{None} as the
translation table parameter, which is treated as the identity
transformation.   This makes it easier to carry out operations
that only delete characters.  (Contributed by Bengt Richter and
implemented by Raymond Hettinger; bpo-1193128@footnote{https://bugs.python.org/issue1193128}.)

@item 
The built-in @ref{f2f,,dir()} function now checks for a @code{__dir__()}
method on the objects it receives.  This method must return a list
of strings containing the names of valid attributes for the object,
and lets the object control the value that @ref{f2f,,dir()} produces.
Objects that have @code{__getattr__()} or @code{__getattribute__()}
methods can use this to advertise pseudo-attributes they will honor.
(bpo-1591665@footnote{https://bugs.python.org/issue1591665})

@item 
Instance method objects have new attributes for the object and function
comprising the method; the new synonym for @code{im_self} is
@code{__self__}, and @code{im_func} is also available as @code{__func__}.
The old names are still supported in Python 2.6, but are gone in 3.0.

@item 
An obscure change: when you use the @ref{713,,locals()} function inside a
@ref{e72,,class} statement, the resulting dictionary no longer returns free
variables.  (Free variables, in this case, are variables referenced in the
@code{class} statement  that aren’t attributes of the class.)
@end itemize

@c ======================================================================

@menu
* Optimizations: Optimizations<12>. 
* Interpreter Changes: Interpreter Changes<2>. 

@end menu

@node Optimizations<12>,Interpreter Changes<2>,,Other Language Changes<13>
@anchor{whatsnew/2 6 optimizations}@anchor{f30}
@subsubsection Optimizations



@itemize *

@item 
The @ref{12d,,warnings} module has been rewritten in C.  This makes
it possible to invoke warnings from the parser, and may also
make the interpreter’s startup faster.
(Contributed by Neal Norwitz and Brett Cannon; bpo-1631171@footnote{https://bugs.python.org/issue1631171}.)

@item 
Type objects now have a cache of methods that can reduce
the work required to find the correct method implementation
for a particular class; once cached, the interpreter doesn’t need to
traverse base classes to figure out the right method to call.
The cache is cleared if a base class or the class itself is modified,
so the cache should remain correct even in the face of Python’s dynamic
nature.
(Original optimization implemented by Armin Rigo, updated for
Python 2.6 by Kevin Jacobs; bpo-1700288@footnote{https://bugs.python.org/issue1700288}.)

By default, this change is only applied to types that are included with
the Python core.  Extension modules may not necessarily be compatible with
this cache,
so they must explicitly add @code{Py_TPFLAGS_HAVE_VERSION_TAG}
to the module’s @code{tp_flags} field to enable the method cache.
(To be compatible with the method cache, the extension module’s code
must not directly access and modify the @code{tp_dict} member of
any of the types it implements.  Most modules don’t do this,
but it’s impossible for the Python interpreter to determine that.
See bpo-1878@footnote{https://bugs.python.org/issue1878} for some discussion.)

@item 
Function calls that use keyword arguments are significantly faster
by doing a quick pointer comparison, usually saving the time of a
full string comparison.  (Contributed by Raymond Hettinger, after an
initial implementation by Antoine Pitrou; bpo-1819@footnote{https://bugs.python.org/issue1819}.)

@item 
All of the functions in the @ref{f3,,struct} module have been rewritten in
C, thanks to work at the Need For Speed sprint.
(Contributed by Raymond Hettinger.)

@item 
Some of the standard built-in types now set a bit in their type
objects.  This speeds up checking whether an object is a subclass of
one of these types.  (Contributed by Neal Norwitz.)

@item 
Unicode strings now use faster code for detecting
whitespace and line breaks; this speeds up the @code{split()} method
by about 25% and @code{splitlines()} by 35%.
(Contributed by Antoine Pitrou.)  Memory usage is reduced
by using pymalloc for the Unicode string’s data.

@item 
The @code{with} statement now stores the @code{__exit__()} method on the stack,
producing a small speedup.  (Implemented by Jeffrey Yasskin.)

@item 
To reduce memory usage, the garbage collector will now clear internal
free lists when garbage-collecting the highest generation of objects.
This may return memory to the operating system sooner.
@end itemize

@c ======================================================================

@node Interpreter Changes<2>,,Optimizations<12>,Other Language Changes<13>
@anchor{whatsnew/2 6 interpreter-changes}@anchor{f31}@anchor{whatsnew/2 6 new-26-interpreter}@anchor{f32}
@subsubsection Interpreter Changes


Two command-line options have been reserved for use by other Python
implementations.  The @ref{f33,,-J} switch has been reserved for use by
Jython for Jython-specific options, such as switches that are passed to
the underlying JVM.  @ref{159,,-X} has been reserved for options
specific to a particular implementation of Python such as CPython,
Jython, or IronPython.  If either option is used with Python 2.6, the
interpreter will report that the option isn’t currently used.

Python can now be prevented from writing @code{.pyc} or @code{.pyo}
files by supplying the @ref{f34,,-B} switch to the Python interpreter,
or by setting the 
@geindex PYTHONDONTWRITEBYTECODE
@geindex environment variable; PYTHONDONTWRITEBYTECODE
@ref{f35,,PYTHONDONTWRITEBYTECODE} environment
variable before running the interpreter.  This setting is available to
Python programs as the @code{sys.dont_write_bytecode} variable, and
Python code can change the value to modify the interpreter’s
behaviour.  (Contributed by Neal Norwitz and Georg Brandl.)

The encoding used for standard input, output, and standard error can
be specified by setting the 
@geindex PYTHONIOENCODING
@geindex environment variable; PYTHONIOENCODING
@ref{b91,,PYTHONIOENCODING} environment
variable before running the interpreter.  The value should be a string
in the form @code{<encoding>} or @code{<encoding>:<errorhandler>}.
The @emph{encoding} part specifies the encoding’s name, e.g. @code{utf-8} or
@code{latin-1}; the optional @emph{errorhandler} part specifies
what to do with characters that can’t be handled by the encoding,
and  should be one of “error”, “ignore”, or “replace”.   (Contributed
by Martin von Löwis.)

@c ======================================================================

@node New and Improved Modules<2>,Deprecations and Removals,Other Language Changes<13>,What’s New in Python 2 6
@anchor{whatsnew/2 6 new-and-improved-modules}@anchor{f36}
@subsection New and Improved Modules


As in every release, Python’s standard library received a number of
enhancements and bug fixes.  Here’s a partial list of the most notable
changes, sorted alphabetically by module name. Consult the
@code{Misc/NEWS} file in the source tree for a more complete list of
changes, or look through the Subversion logs for all the details.


@itemize *

@item 
The @ref{a,,asyncore} and @ref{8,,asynchat} modules are
being actively maintained again, and a number of patches and bugfixes
were applied.  (Maintained by Josiah Carlson; see bpo-1736190@footnote{https://bugs.python.org/issue1736190} for
one patch.)

@item 
The @code{bsddb} module also has a new maintainer, Jesús Cea Avión, and the package
is now available as a standalone package.  The web page for the package is
www.jcea.es/programacion/pybsddb.htm@footnote{https://www.jcea.es/programacion/pybsddb.htm}.
The plan is to remove the package from the standard library
in Python 3.0, because its pace of releases is much more frequent than
Python’s.

The @code{bsddb.dbshelve} module now uses the highest pickling protocol
available, instead of restricting itself to protocol 1.
(Contributed by W. Barnes.)

@item 
The @ref{14,,cgi} module will now read variables from the query string
of an HTTP POST request.  This makes it possible to use form actions
with URLs that include query strings such as
“/cgi-bin/add.py?category=1”.  (Contributed by Alexandre Fiori and
Nubis; bpo-1817@footnote{https://bugs.python.org/issue1817}.)

The @code{parse_qs()} and @code{parse_qsl()} functions have been
relocated from the @ref{14,,cgi} module to the @code{urlparse} module.
The versions still available in the @ref{14,,cgi} module will
trigger @ref{3a9,,PendingDeprecationWarning} messages in 2.6
(bpo-600362@footnote{https://bugs.python.org/issue600362}).

@item 
The @ref{17,,cmath} module underwent extensive revision,
contributed by Mark Dickinson and Christian Heimes.
Five new functions were added:


@itemize *

@item 
@code{polar()} converts a complex number to polar form, returning
the modulus and argument of the complex number.

@item 
@code{rect()} does the opposite, turning a modulus, argument pair
back into the corresponding complex number.

@item 
@code{phase()} returns the argument (also called the angle) of a complex
number.

@item 
@code{isnan()} returns True if either
the real or imaginary part of its argument is a NaN.

@item 
@code{isinf()} returns True if either the real or imaginary part of
its argument is infinite.
@end itemize

The revisions also improved the numerical soundness of the
@ref{17,,cmath} module.  For all functions, the real and imaginary
parts of the results are accurate to within a few units of least
precision (ulps) whenever possible.  See bpo-1381@footnote{https://bugs.python.org/issue1381} for the
details.  The branch cuts for @code{asinh()}, @code{atanh()}: and
@code{atan()} have also been corrected.

The tests for the module have been greatly expanded; nearly 2000 new
test cases exercise the algebraic functions.

On IEEE 754 platforms, the @ref{17,,cmath} module now handles IEEE 754
special values and floating-point exceptions in a manner consistent
with Annex ‘G’ of the C99 standard.

@item 
A new data type in the @ref{1c,,collections} module: @code{namedtuple(typename,
fieldnames)} is a factory function that creates subclasses of the standard tuple
whose fields are accessible by name as well as index.  For example:

@example
>>> var_type = collections.namedtuple('variable',
...             'id name type size')
>>> # Names are separated by spaces or commas.
>>> # 'id, name, type, size' would also work.
>>> var_type._fields
('id', 'name', 'type', 'size')

>>> var = var_type(1, 'frequency', 'int', 4)
>>> print var[0], var.id    # Equivalent
1 1
>>> print var[2], var.type  # Equivalent
int int
>>> var._asdict()
@{'size': 4, 'type': 'int', 'id': 1, 'name': 'frequency'@}
>>> v2 = var._replace(name='amplitude')
>>> v2
variable(id=1, name='amplitude', type='int', size=4)
@end example

Several places in the standard library that returned tuples have
been modified to return @code{namedtuple} instances.  For example,
the @code{Decimal.as_tuple()} method now returns a named tuple with
@code{sign}, @code{digits}, and @code{exponent} fields.

(Contributed by Raymond Hettinger.)

@item 
Another change to the @ref{1c,,collections} module is that the
@code{deque} type now supports an optional @emph{maxlen} parameter;
if supplied, the deque’s size will be restricted to no more
than @emph{maxlen} items.  Adding more items to a full deque causes
old items to be discarded.

@example
>>> from collections import deque
>>> dq=deque(maxlen=3)
>>> dq
deque([], maxlen=3)
>>> dq.append(1); dq.append(2); dq.append(3)
>>> dq
deque([1, 2, 3], maxlen=3)
>>> dq.append(4)
>>> dq
deque([2, 3, 4], maxlen=3)
@end example

(Contributed by Raymond Hettinger.)

@item 
The @code{Cookie} module’s @code{Morsel} objects now support an
@code{httponly} attribute.  In some browsers. cookies with this attribute
set cannot be accessed or manipulated by JavaScript code.
(Contributed by Arvin Schnell; bpo-1638033@footnote{https://bugs.python.org/issue1638033}.)

@item 
A new window method in the @ref{2a,,curses} module,
@code{chgat()}, changes the display attributes for a certain number of
characters on a single line.  (Contributed by Fabian Kreutz.)

@example
# Boldface text starting at y=0,x=21
# and affecting the rest of the line.
stdscr.chgat(0, 21, curses.A_BOLD)
@end example

The @code{Textbox} class in the @ref{2d,,curses.textpad} module
now supports editing in insert mode as well as overwrite mode.
Insert mode is enabled by supplying a true value for the @emph{insert_mode}
parameter when creating the @code{Textbox} instance.

@item 
The @ref{2f,,datetime} module’s @code{strftime()} methods now support a
@code{%f} format code that expands to the number of microseconds in the
object, zero-padded on
the left to six places.  (Contributed by Skip Montanaro; bpo-1158@footnote{https://bugs.python.org/issue1158}.)

@item 
The @ref{34,,decimal} module was updated to version 1.66 of
the General Decimal Specification@footnote{http://speleotrove.com/decimal/decarith.html}.  New features
include some methods for some basic mathematical functions such as
@code{exp()} and @code{log10()}:

@example
>>> Decimal(1).exp()
Decimal("2.718281828459045235360287471")
>>> Decimal("2.7182818").ln()
Decimal("0.9999999895305022877376682436")
>>> Decimal(1000).log10()
Decimal("3")
@end example

The @code{as_tuple()} method of @code{Decimal} objects now returns a
named tuple with @code{sign}, @code{digits}, and @code{exponent} fields.

(Implemented by Facundo Batista and Mark Dickinson.  Named tuple
support added by Raymond Hettinger.)

@item 
The @ref{35,,difflib} module’s @code{SequenceMatcher} class
now returns named tuples representing matches,
with @code{a}, @code{b}, and @code{size} attributes.
(Contributed by Raymond Hettinger.)

@item 
An optional @code{timeout} parameter, specifying a timeout measured in
seconds, was added to the @ref{3e9,,ftplib.FTP} class constructor as
well as the @code{connect()} method.  (Added by Facundo Batista.)
Also, the @code{FTP} class’s @code{storbinary()} and
@code{storlines()} now take an optional @emph{callback} parameter that
will be called with each block of data after the data has been sent.
(Contributed by Phil Schwartz; bpo-1221598@footnote{https://bugs.python.org/issue1221598}.)

@item 
The @code{reduce()} built-in function is also available in the
@ref{7f,,functools} module.  In Python 3.0, the builtin has been
dropped and @code{reduce()} is only available from @ref{7f,,functools};
currently there are no plans to drop the builtin in the 2.x series.
(Patched by Christian Heimes; bpo-1739906@footnote{https://bugs.python.org/issue1739906}.)

@item 
When possible, the @ref{82,,getpass} module will now use
@code{/dev/tty} to print a prompt message and read the password,
falling back to standard error and standard input.  If the
password may be echoed to the terminal, a warning is printed before
the prompt is displayed.  (Contributed by Gregory P. Smith.)

@item 
The @ref{288,,glob.glob()} function can now return Unicode filenames if
a Unicode path was used and Unicode filenames are matched within the
directory.  (bpo-1001604@footnote{https://bugs.python.org/issue1001604})

@item 
A new function in the @ref{89,,heapq} module, @code{merge(iter1, iter2, ...)},
takes any number of iterables returning data in sorted
order, and returns a new generator that returns the contents of all
the iterators, also in sorted order.  For example:

@example
>>> list(heapq.merge([1, 3, 5, 9], [2, 8, 16]))
[1, 2, 3, 5, 8, 9, 16]
@end example

Another new function, @code{heappushpop(heap, item)},
pushes @emph{item} onto @emph{heap}, then pops off and returns the smallest item.
This is more efficient than making a call to @code{heappush()} and then
@code{heappop()}.

@ref{89,,heapq} is now implemented to only use less-than comparison,
instead of the less-than-or-equal comparison it previously used.
This makes @ref{89,,heapq}’s usage of a type match the
@ref{705,,list.sort()} method.
(Contributed by Raymond Hettinger.)

@item 
An optional @code{timeout} parameter, specifying a timeout measured in
seconds, was added to the @code{httplib.HTTPConnection} and
@code{HTTPSConnection} class constructors.  (Added by Facundo
Batista.)

@item 
Most of the @ref{9c,,inspect} module’s functions, such as
@code{getmoduleinfo()} and @code{getargs()}, now return named tuples.
In addition to behaving like tuples, the elements of the  return value
can also be accessed as attributes.
(Contributed by Raymond Hettinger.)

Some new functions in the module include
@code{isgenerator()}, @code{isgeneratorfunction()},
and @code{isabstract()}.

@item 
The @ref{9f,,itertools} module gained several new functions.

@code{izip_longest(iter1, iter2, ...[, fillvalue])} makes tuples from
each of the elements; if some of the iterables are shorter than
others, the missing values are set to @emph{fillvalue}.  For example:

@example
>>> tuple(itertools.izip_longest([1,2,3], [1,2,3,4,5]))
((1, 1), (2, 2), (3, 3), (None, 4), (None, 5))
@end example

@code{product(iter1, iter2, ..., [repeat=N])} returns the Cartesian product
of the supplied iterables, a set of tuples containing
every possible combination of the elements returned from each iterable.

@example
>>> list(itertools.product([1,2,3], [4,5,6]))
[(1, 4), (1, 5), (1, 6),
 (2, 4), (2, 5), (2, 6),
 (3, 4), (3, 5), (3, 6)]
@end example

The optional @emph{repeat} keyword argument is used for taking the
product of an iterable or a set of iterables with themselves,
repeated @emph{N} times.  With a single iterable argument, @emph{N}-tuples
are returned:

@example
>>> list(itertools.product([1,2], repeat=3))
[(1, 1, 1), (1, 1, 2), (1, 2, 1), (1, 2, 2),
 (2, 1, 1), (2, 1, 2), (2, 2, 1), (2, 2, 2)]
@end example

With two iterables, @emph{2N}-tuples are returned.

@example
>>> list(itertools.product([1,2], [3,4], repeat=2))
[(1, 3, 1, 3), (1, 3, 1, 4), (1, 3, 2, 3), (1, 3, 2, 4),
 (1, 4, 1, 3), (1, 4, 1, 4), (1, 4, 2, 3), (1, 4, 2, 4),
 (2, 3, 1, 3), (2, 3, 1, 4), (2, 3, 2, 3), (2, 3, 2, 4),
 (2, 4, 1, 3), (2, 4, 1, 4), (2, 4, 2, 3), (2, 4, 2, 4)]
@end example

@code{combinations(iterable, r)} returns sub-sequences of length @emph{r} from
the elements of @emph{iterable}.

@example
>>> list(itertools.combinations('123', 2))
[('1', '2'), ('1', '3'), ('2', '3')]
>>> list(itertools.combinations('123', 3))
[('1', '2', '3')]
>>> list(itertools.combinations('1234', 3))
[('1', '2', '3'), ('1', '2', '4'),
 ('1', '3', '4'), ('2', '3', '4')]
@end example

@code{permutations(iter[, r])} returns all the permutations of length @emph{r} of
the iterable’s elements.  If @emph{r} is not specified, it will default to the
number of elements produced by the iterable.

@example
>>> list(itertools.permutations([1,2,3,4], 2))
[(1, 2), (1, 3), (1, 4),
 (2, 1), (2, 3), (2, 4),
 (3, 1), (3, 2), (3, 4),
 (4, 1), (4, 2), (4, 3)]
@end example

@code{itertools.chain(*iterables)} is an existing function in
@ref{9f,,itertools} that gained a new constructor in Python 2.6.
@code{itertools.chain.from_iterable(iterable)} takes a single
iterable that should return other iterables.  @code{chain()} will
then return all the elements of the first iterable, then
all the elements of the second, and so on.

@example
>>> list(itertools.chain.from_iterable([[1,2,3], [4,5,6]]))
[1, 2, 3, 4, 5, 6]
@end example

(All contributed by Raymond Hettinger.)

@item 
The @ref{a6,,logging} module’s @code{FileHandler} class
and its subclasses @code{WatchedFileHandler}, @code{RotatingFileHandler},
and @code{TimedRotatingFileHandler} now
have an optional @emph{delay} parameter to their constructors.  If @emph{delay}
is true, opening of the log file is deferred until the first
@code{emit()} call is made.  (Contributed by Vinay Sajip.)

@code{TimedRotatingFileHandler} also has a @emph{utc} constructor
parameter.  If the argument is true, UTC time will be used
in determining when midnight occurs and in generating filenames;
otherwise local time will be used.

@item 
Several new functions were added to the @ref{ad,,math} module:


@itemize *

@item 
@ref{f37,,isinf()} and @ref{f38,,isnan()} determine whether a given float
is a (positive or negative) infinity or a NaN (Not a Number), respectively.

@item 
@ref{f39,,copysign()} copies the sign bit of an IEEE 754 number,
returning the absolute value of @emph{x} combined with the sign bit of
@emph{y}.  For example, @code{math.copysign(1, -0.0)} returns -1.0.
(Contributed by Christian Heimes.)

@item 
@ref{43e,,factorial()} computes the factorial of a number.
(Contributed by Raymond Hettinger; bpo-2138@footnote{https://bugs.python.org/issue2138}.)

@item 
@ref{f3a,,fsum()} adds up the stream of numbers from an iterable,
and is careful to avoid loss of precision through using partial sums.
(Contributed by Jean Brouwers, Raymond Hettinger, and Mark Dickinson;
bpo-2819@footnote{https://bugs.python.org/issue2819}.)

@item 
@ref{f3b,,acosh()}, @ref{f3c,,asinh()}
and @ref{f3d,,atanh()} compute the inverse hyperbolic functions.

@item 
@ref{f3e,,log1p()} returns the natural logarithm of @emph{1+x}
(base @emph{e}).

@item 
@code{trunc()} rounds a number toward zero, returning the closest
@code{Integral} that’s between the function’s argument and zero.
Added as part of the backport of
@ref{e63,,PEP 3141's type hierarchy for numbers}.
@end itemize

@item 
The @ref{ad,,math} module has been improved to give more consistent
behaviour across platforms, especially with respect to handling of
floating-point exceptions and IEEE 754 special values.

Whenever possible, the module follows the recommendations of the C99
standard about 754’s special values.  For example, @code{sqrt(-1.)}
should now give a @ref{17c,,ValueError} across almost all platforms,
while @code{sqrt(float('NaN'))} should return a NaN on all IEEE 754
platforms.  Where Annex ‘F’ of the C99 standard recommends signaling
‘divide-by-zero’ or ‘invalid’, Python will raise @ref{17c,,ValueError}.
Where Annex ‘F’ of the C99 standard recommends signaling ‘overflow’,
Python will raise @ref{347,,OverflowError}.  (See bpo-711019@footnote{https://bugs.python.org/issue711019} and
bpo-1640@footnote{https://bugs.python.org/issue1640}.)

(Contributed by Christian Heimes and Mark Dickinson.)

@item 
@ref{511,,mmap} objects now have a @code{rfind()} method that searches for a
substring beginning at the end of the string and searching
backwards.  The @code{find()} method also gained an @emph{end} parameter
giving an index at which to stop searching.
(Contributed by John Lenton.)

@item 
The @ref{be,,operator} module gained a
@code{methodcaller()} function that takes a name and an optional
set of arguments, returning a callable that will call
the named function on any arguments passed to it.  For example:

@example
>>> # Equivalent to lambda s: s.replace('old', 'new')
>>> replacer = operator.methodcaller('replace', 'old', 'new')
>>> replacer('old wine in old bottles')
'new wine in new bottles'
@end example

(Contributed by Georg Brandl, after a suggestion by Gregory Petrosyan.)

The @code{attrgetter()} function now accepts dotted names and performs
the corresponding attribute lookups:

@example
>>> inst_name = operator.attrgetter(
...        '__class__.__name__')
>>> inst_name('@w{'})
'str'
>>> inst_name(help)
'_Helper'
@end example

(Contributed by Georg Brandl, after a suggestion by Barry Warsaw.)

@item 
The @ref{c0,,os} module now wraps several new system calls.
@code{fchmod(fd, mode)} and @code{fchown(fd, uid, gid)} change the mode
and ownership of an opened file, and @code{lchmod(path, mode)} changes
the mode of a symlink.  (Contributed by Georg Brandl and Christian
Heimes.)

@code{chflags()} and @code{lchflags()} are wrappers for the
corresponding system calls (where they’re available), changing the
flags set on a file.  Constants for the flag values are defined in
the @ref{ef,,stat} module; some possible values include
@code{UF_IMMUTABLE} to signal the file may not be changed and
@code{UF_APPEND} to indicate that data can only be appended to the
file.  (Contributed by M. Levinson.)

@code{os.closerange(low, high)} efficiently closes all file descriptors
from @emph{low} to @emph{high}, ignoring any errors and not including @emph{high} itself.
This function is now used by the @ref{f4,,subprocess} module to make starting
processes faster.  (Contributed by Georg Brandl; bpo-1663329@footnote{https://bugs.python.org/issue1663329}.)

@item 
The @code{os.environ} object’s @code{clear()} method will now unset the
environment variables using @ref{40f,,os.unsetenv()} in addition to clearing
the object’s keys.  (Contributed by Martin Horcicka; bpo-1181@footnote{https://bugs.python.org/issue1181}.)

@item 
The @ref{8de,,os.walk()} function now has a @code{followlinks} parameter. If
set to True, it will follow symlinks pointing to directories and
visit the directory’s contents.  For backward compatibility, the
parameter’s default value is false.  Note that the function can fall
into an infinite recursion if there’s a symlink that points to a
parent directory.  (bpo-1273829@footnote{https://bugs.python.org/issue1273829})

@item 
In the @ref{c1,,os.path} module, the @code{splitext()} function
has been changed to not split on leading period characters.
This produces better results when operating on Unix’s dot-files.
For example, @code{os.path.splitext('.ipython')}
now returns @code{('.ipython', '@w{'})} instead of @code{('@w{'}, '.ipython')}.
(bpo-1115886@footnote{https://bugs.python.org/issue1115886})

A new function, @code{os.path.relpath(path, start='.')}, returns a relative path
from the @code{start} path, if it’s supplied, or from the current
working directory to the destination @code{path}.  (Contributed by
Richard Barran; bpo-1339796@footnote{https://bugs.python.org/issue1339796}.)

On Windows, @ref{f3f,,os.path.expandvars()} will now expand environment variables
given in the form “%var%”, and “~user” will be expanded into the
user’s home directory path.  (Contributed by Josiah Carlson;
bpo-957650@footnote{https://bugs.python.org/issue957650}.)

@item 
The Python debugger provided by the @ref{c4,,pdb} module
gained a new command: “run” restarts the Python program being debugged
and can optionally take new command-line arguments for the program.
(Contributed by Rocky Bernstein; bpo-1393667@footnote{https://bugs.python.org/issue1393667}.)

@item 
The @ref{f40,,pdb.post_mortem()} function, used to begin debugging a
traceback, will now use the traceback returned by @ref{18f,,sys.exc_info()}
if no traceback is supplied.   (Contributed by Facundo Batista;
bpo-1106316@footnote{https://bugs.python.org/issue1106316}.)

@item 
The @ref{c6,,pickletools} module now has an @code{optimize()} function
that takes a string containing a pickle and removes some unused
opcodes, returning a shorter pickle that contains the same data structure.
(Contributed by Raymond Hettinger.)

@item 
A @code{get_data()} function was added to the @ref{c8,,pkgutil}
module that returns the contents of resource files included
with an installed Python package.  For example:

@example
>>> import pkgutil
>>> print pkgutil.get_data('test', 'exception_hierarchy.txt')
BaseException
 +-- SystemExit
 +-- KeyboardInterrupt
 +-- GeneratorExit
 +-- Exception
      +-- StopIteration
      +-- StandardError
 ...
@end example

(Contributed by Paul Moore; bpo-2439@footnote{https://bugs.python.org/issue2439}.)

@item 
The @code{pyexpat} module’s @code{Parser} objects now allow setting
their @code{buffer_size} attribute to change the size of the buffer
used to hold character data.
(Contributed by Achim Gaedke; bpo-1137@footnote{https://bugs.python.org/issue1137}.)

@item 
The @code{Queue} module now provides queue variants that retrieve entries
in different orders.  The @code{PriorityQueue} class stores
queued items in a heap and retrieves them in priority order,
and @code{LifoQueue} retrieves the most recently added entries first,
meaning that it behaves like a stack.
(Contributed by Raymond Hettinger.)

@item 
The @ref{d7,,random} module’s @code{Random} objects can
now be pickled on a 32-bit system and unpickled on a 64-bit
system, and vice versa.  Unfortunately, this change also means
that Python 2.6’s @code{Random} objects can’t be unpickled correctly
on earlier versions of Python.
(Contributed by Shawn Ligocki; bpo-1727780@footnote{https://bugs.python.org/issue1727780}.)

The new @code{triangular(low, high, mode)} function returns random
numbers following a triangular distribution.   The returned values
are between @emph{low} and @emph{high}, not including @emph{high} itself, and
with @emph{mode} as the most frequently occurring value
in the distribution.  (Contributed by Wladmir van der Laan and
Raymond Hettinger; bpo-1681432@footnote{https://bugs.python.org/issue1681432}.)

@item 
Long regular expression searches carried out by the  @ref{d8,,re}
module will check for signals being delivered, so
time-consuming searches can now be interrupted.
(Contributed by Josh Hoyt and Ralf Schmitt; bpo-846388@footnote{https://bugs.python.org/issue846388}.)

The regular expression module is implemented by compiling bytecodes
for a tiny regex-specific virtual machine.  Untrusted code
could create malicious strings of bytecode directly and cause crashes,
so Python 2.6 includes a verifier for the regex bytecode.
(Contributed by Guido van Rossum from work for Google App Engine;
bpo-3487@footnote{https://bugs.python.org/issue3487}.)

@item 
The @ref{dc,,rlcompleter} module’s @code{Completer.complete()} method
will now ignore exceptions triggered while evaluating a name.
(Fixed by Lorenz Quack; bpo-2250@footnote{https://bugs.python.org/issue2250}.)

@item 
The @ref{de,,sched} module’s @code{scheduler} instances now
have a read-only @ref{d5,,queue} attribute that returns the
contents of the scheduler’s queue, represented as a list of
named tuples with the fields @code{(time, priority, action, argument)}.
(Contributed by Raymond Hettinger; bpo-1861@footnote{https://bugs.python.org/issue1861}.)

@item 
The @ref{e0,,select} module now has wrapper functions
for the Linux @code{epoll()} and BSD @code{kqueue()} system calls.
@code{modify()} method was added to the existing @code{poll}
objects; @code{pollobj.modify(fd, eventmask)} takes a file descriptor
or file object and an event mask, modifying the recorded event mask
for that file.
(Contributed by Christian Heimes; bpo-1657@footnote{https://bugs.python.org/issue1657}.)

@item 
The @ref{538,,shutil.copytree()} function now has an optional @emph{ignore} argument
that takes a callable object.  This callable will receive each directory path
and a list of the directory’s contents, and returns a list of names that
will be ignored, not copied.

The @ref{e4,,shutil} module also provides an @code{ignore_patterns()}
function for use with this new parameter.  @code{ignore_patterns()}
takes an arbitrary number of glob-style patterns and returns a
callable that will ignore any files and directories that match any
of these patterns.  The following example copies a directory tree,
but skips both @code{.svn} directories and Emacs backup files,
which have names ending with ‘~’:

@example
shutil.copytree('Doc/library', '/tmp/library',
                ignore=shutil.ignore_patterns('*~', '.svn'))
@end example

(Contributed by Tarek Ziadé; bpo-2663@footnote{https://bugs.python.org/issue2663}.)

@item 
Integrating signal handling with GUI handling event loops
like those used by Tkinter or GTk+ has long been a problem; most
software ends up polling, waking up every fraction of a second to check
if any GUI events have occurred.
The @ref{e5,,signal} module can now make this more efficient.
Calling @code{signal.set_wakeup_fd(fd)} sets a file descriptor
to be used; when a signal is received, a byte is written to that
file descriptor.  There’s also a C-level function,
@ref{f41,,PySignal_SetWakeupFd()}, for setting the descriptor.

Event loops will use this by opening a pipe to create two descriptors,
one for reading and one for writing.  The writable descriptor
will be passed to @code{set_wakeup_fd()}, and the readable descriptor
will be added to the list of descriptors monitored by the event loop via
@code{select()} or @code{poll()}.
On receiving a signal, a byte will be written and the main event loop
will be woken up, avoiding the need to poll.

(Contributed by Adam Olsen; bpo-1583@footnote{https://bugs.python.org/issue1583}.)

The @code{siginterrupt()} function is now available from Python code,
and allows changing whether signals can interrupt system calls or not.
(Contributed by Ralf Schmitt.)

The @code{setitimer()} and @code{getitimer()} functions have also been
added (where they’re available).  @code{setitimer()}
allows setting interval timers that will cause a signal to be
delivered to the process after a specified time, measured in
wall-clock time, consumed process time, or combined process+system
time.  (Contributed by Guilherme Polo; bpo-2240@footnote{https://bugs.python.org/issue2240}.)

@item 
The @ref{e8,,smtplib} module now supports SMTP over SSL thanks to the
addition of the @code{SMTP_SSL} class. This class supports an
interface identical to the existing @code{SMTP} class.
(Contributed by Monty Taylor.)  Both class constructors also have an
optional @code{timeout} parameter that specifies a timeout for the
initial connection attempt, measured in seconds.  (Contributed by
Facundo Batista.)

An implementation of the LMTP protocol (
@geindex RFC; RFC 2033
RFC 2033@footnote{https://tools.ietf.org/html/rfc2033.html}) was also added
to the module.  LMTP is used in place of SMTP when transferring
e-mail between agents that don’t manage a mail queue.  (LMTP
implemented by Leif Hedstrom; bpo-957003@footnote{https://bugs.python.org/issue957003}.)

@code{SMTP.starttls()} now complies with 
@geindex RFC; RFC 3207
RFC 3207@footnote{https://tools.ietf.org/html/rfc3207.html} and forgets any
knowledge obtained from the server not obtained from the TLS
negotiation itself.  (Patch contributed by Bill Fenner;
bpo-829951@footnote{https://bugs.python.org/issue829951}.)

@item 
The @ref{ea,,socket} module now supports TIPC (@indicateurl{http://tipc.sourceforge.net/}),
a high-performance non-IP-based protocol designed for use in clustered
environments.  TIPC addresses are 4- or 5-tuples.
(Contributed by Alberto Bertogli; bpo-1646@footnote{https://bugs.python.org/issue1646}.)

A new function, @code{create_connection()}, takes an address and
connects to it using an optional timeout value, returning the
connected socket object.  This function also looks up the address’s
type and connects to it using IPv4 or IPv6 as appropriate.  Changing
your code to use @code{create_connection()} instead of
@code{socket(socket.AF_INET, ...)} may be all that’s required to make
your code work with IPv6.

@item 
The base classes in the @code{SocketServer} module now support
calling a @code{handle_timeout()} method after a span of inactivity
specified by the server’s @code{timeout} attribute.  (Contributed
by Michael Pomraning.)  The @code{serve_forever()} method
now takes an optional poll interval measured in seconds,
controlling how often the server will check for a shutdown request.
(Contributed by Pedro Werneck and Jeffrey Yasskin;
bpo-742598@footnote{https://bugs.python.org/issue742598}, bpo-1193577@footnote{https://bugs.python.org/issue1193577}.)

@item 
The @ref{ed,,sqlite3} module, maintained by Gerhard Häring,
has been updated from version 2.3.2 in Python 2.5 to
version 2.4.1.

@item 
The @ref{f3,,struct} module now supports the C99 _Bool type,
using the format character @code{'?'}.
(Contributed by David Remahl.)

@item 
The @code{Popen} objects provided by the @ref{f4,,subprocess} module
now have @code{terminate()}, @code{kill()}, and @code{send_signal()} methods.
On Windows, @code{send_signal()} only supports the @code{SIGTERM}
signal, and all these methods are aliases for the Win32 API function
@code{TerminateProcess()}.
(Contributed by Christian Heimes.)

@item 
A new variable in the @ref{f7,,sys} module, @code{float_info}, is an
object containing information derived from the @code{float.h} file
about the platform’s floating-point support.  Attributes of this
object include @code{mant_dig} (number of digits in the mantissa),
@code{epsilon} (smallest difference between 1.0 and the next
largest value representable), and several others.  (Contributed by
Christian Heimes; bpo-1534@footnote{https://bugs.python.org/issue1534}.)

Another new variable, @code{dont_write_bytecode}, controls whether Python
writes any @code{.pyc} or @code{.pyo} files on importing a module.
If this variable is true, the compiled files are not written.  The
variable is initially set on start-up by supplying the @ref{f34,,-B}
switch to the Python interpreter, or by setting the
@geindex PYTHONDONTWRITEBYTECODE
@geindex environment variable; PYTHONDONTWRITEBYTECODE
@ref{f35,,PYTHONDONTWRITEBYTECODE} environment variable before
running the interpreter.  Python code can subsequently
change the value of this variable to control whether bytecode files
are written or not.
(Contributed by Neal Norwitz and Georg Brandl.)

Information about the command-line arguments supplied to the Python
interpreter is available by reading attributes of a named
tuple available as @code{sys.flags}.  For example, the @code{verbose}
attribute is true if Python
was executed in verbose mode, @code{debug} is true in debugging mode, etc.
These attributes are all read-only.
(Contributed by Christian Heimes.)

A new function, @code{getsizeof()}, takes a Python object and returns
the amount of memory used by the object, measured in bytes.  Built-in
objects return correct results; third-party extensions may not,
but can define a @code{__sizeof__()} method to return the
object’s size.
(Contributed by Robert Schuppenies; bpo-2898@footnote{https://bugs.python.org/issue2898}.)

It’s now possible to determine the current profiler and tracer functions
by calling @ref{f42,,sys.getprofile()} and @ref{f43,,sys.gettrace()}.
(Contributed by Georg Brandl; bpo-1648@footnote{https://bugs.python.org/issue1648}.)

@item 
The @ref{fb,,tarfile} module now supports POSIX.1-2001 (pax) tarfiles in
addition to the POSIX.1-1988 (ustar) and GNU tar formats that were
already supported.  The default format is GNU tar; specify the
@code{format} parameter to open a file using a different format:

@example
tar = tarfile.open("output.tar", "w",
                   format=tarfile.PAX_FORMAT)
@end example

The new @code{encoding} and @code{errors} parameters specify an encoding and
an error handling scheme for character conversions.  @code{'strict'},
@code{'ignore'}, and @code{'replace'} are the three standard ways Python can
handle errors,;
@code{'utf-8'} is a special value that replaces bad characters with
their UTF-8 representation.  (Character conversions occur because the
PAX format supports Unicode filenames, defaulting to UTF-8 encoding.)

The @code{TarFile.add()} method now accepts an @code{exclude} argument that’s
a function that can be used to exclude certain filenames from
an archive.
The function must take a filename and return true if the file
should be excluded or false if it should be archived.
The function is applied to both the name initially passed to @code{add()}
and to the names of files in recursively-added directories.

(All changes contributed by Lars Gustäbel).

@item 
An optional @code{timeout} parameter was added to the
@ref{832,,telnetlib.Telnet} class constructor, specifying a timeout
measured in seconds.  (Added by Facundo Batista.)

@item 
The @ref{f44,,tempfile.NamedTemporaryFile} class usually deletes
the temporary file it created when the file is closed.  This
behaviour can now be changed by passing @code{delete=False} to the
constructor.  (Contributed by Damien Miller; bpo-1537850@footnote{https://bugs.python.org/issue1537850}.)

A new class, @code{SpooledTemporaryFile}, behaves like
a temporary file but stores its data in memory until a maximum size is
exceeded.  On reaching that limit, the contents will be written to
an on-disk temporary file.  (Contributed by Dustin J. Mitchell.)

The @code{NamedTemporaryFile} and @code{SpooledTemporaryFile} classes
both work as context managers, so you can write
@code{with tempfile.NamedTemporaryFile() as tmp: ...}.
(Contributed by Alexander Belopolsky; bpo-2021@footnote{https://bugs.python.org/issue2021}.)

@item 
The @code{test.test_support} module gained a number
of context managers useful for writing tests.
@code{EnvironmentVarGuard()} is a
context manager that temporarily changes environment variables and
automatically restores them to their old values.

Another context manager, @code{TransientResource}, can surround calls
to resources that may or may not be available; it will catch and
ignore a specified list of exceptions.  For example,
a network test may ignore certain failures when connecting to an
external web site:

@example
with test_support.TransientResource(IOError,
                                errno=errno.ETIMEDOUT):
    f = urllib.urlopen('https://sf.net')
    ...
@end example

Finally, @code{check_warnings()} resets the @code{warning} module’s
warning filters and returns an object that will record all warning
messages triggered (bpo-3781@footnote{https://bugs.python.org/issue3781}):

@example
with test_support.check_warnings() as wrec:
    warnings.simplefilter("always")
    # ... code that triggers a warning ...
    assert str(wrec.message) == "function is outdated"
    assert len(wrec.warnings) == 1, "Multiple warnings raised"
@end example

(Contributed by Brett Cannon.)

@item 
The @ref{108,,textwrap} module can now preserve existing whitespace
at the beginnings and ends of the newly-created lines
by specifying @code{drop_whitespace=False}
as an argument:

@example
>>> S = """This  sentence  has a bunch   of
...   extra   whitespace."""
>>> print textwrap.fill(S, width=15)
This  sentence
has a bunch
of    extra
whitespace.
>>> print textwrap.fill(S, drop_whitespace=False, width=15)
This  sentence
  has a bunch
   of    extra
   whitespace.
>>>
@end example

(Contributed by Dwayne Bailey; bpo-1581073@footnote{https://bugs.python.org/issue1581073}.)

@item 
The @ref{109,,threading} module API is being changed to use properties
such as @code{daemon} instead of @code{setDaemon()} and
@code{isDaemon()} methods, and some methods have been renamed to use
underscores instead of camel-case; for example, the
@code{activeCount()} method is renamed to @code{active_count()}.  Both
the 2.6 and 3.0 versions of the module support the same properties
and renamed methods, but don’t remove the old methods.  No date has been set
for the deprecation of the old APIs in Python 3.x; the old APIs won’t
be removed in any 2.x version.
(Carried out by several people, most notably Benjamin Peterson.)

The @ref{109,,threading} module’s @code{Thread} objects
gained an @code{ident} property that returns the thread’s
identifier, a nonzero integer.  (Contributed by Gregory P. Smith;
bpo-2871@footnote{https://bugs.python.org/issue2871}.)

@item 
The @ref{10b,,timeit} module now accepts callables as well as strings
for the statement being timed and for the setup code.
Two convenience functions were added for creating
@code{Timer} instances:
@code{repeat(stmt, setup, time, repeat, number)} and
@code{timeit(stmt, setup, time, number)} create an instance and call
the corresponding method. (Contributed by Erik Demaine;
bpo-1533909@footnote{https://bugs.python.org/issue1533909}.)

@item 
The @code{Tkinter} module now accepts lists and tuples for options,
separating the elements by spaces before passing the resulting value to
Tcl/Tk.
(Contributed by Guilherme Polo; bpo-2906@footnote{https://bugs.python.org/issue2906}.)

@item 
The @ref{11d,,turtle} module for turtle graphics was greatly enhanced by
Gregor Lingl.  New features in the module include:


@itemize *

@item 
Better animation of turtle movement and rotation.

@item 
Control over turtle movement using the new @code{delay()},
@code{tracer()}, and @code{speed()} methods.

@item 
The ability to set new shapes for the turtle, and to
define a new coordinate system.

@item 
Turtles now have an @code{undo()} method that can roll back actions.

@item 
Simple support for reacting to input events such as mouse and keyboard
activity, making it possible to write simple games.

@item 
A @code{turtle.cfg} file can be used to customize the starting appearance
of the turtle’s screen.

@item 
The module’s docstrings can be replaced by new docstrings that have been
translated into another language.
@end itemize

(bpo-1513695@footnote{https://bugs.python.org/issue1513695})

@item 
An optional @code{timeout} parameter was added to the
@code{urllib.urlopen()} function and the
@code{urllib.ftpwrapper} class constructor, as well as the
@code{urllib2.urlopen()} function.  The parameter specifies a timeout
measured in seconds.   For example:

@example
>>> u = urllib2.urlopen("http://slow.example.com",
                        timeout=3)
Traceback (most recent call last):
  ...
urllib2.URLError: <urlopen error timed out>
>>>
@end example

(Added by Facundo Batista.)

@item 
The Unicode database provided by the @ref{121,,unicodedata} module
has been updated to version 5.1.0.  (Updated by
Martin von Löwis; bpo-3811@footnote{https://bugs.python.org/issue3811}.)

@item 
The @ref{12d,,warnings} module’s @code{formatwarning()} and @code{showwarning()}
gained an optional @emph{line} argument that can be used to supply the
line of source code.  (Added as part of bpo-1631171@footnote{https://bugs.python.org/issue1631171}, which re-implemented
part of the @ref{12d,,warnings} module in C code.)

A new function, @code{catch_warnings()}, is a context manager
intended for testing purposes that lets you temporarily modify the
warning filters and then restore their original values (bpo-3781@footnote{https://bugs.python.org/issue3781}).

@item 
The XML-RPC @code{SimpleXMLRPCServer} and @code{DocXMLRPCServer}
classes can now be prevented from immediately opening and binding to
their socket by passing @code{False} as the @emph{bind_and_activate}
constructor parameter.  This can be used to modify the instance’s
@code{allow_reuse_address} attribute before calling the
@code{server_bind()} and @code{server_activate()} methods to
open the socket and begin listening for connections.
(Contributed by Peter Parente; bpo-1599845@footnote{https://bugs.python.org/issue1599845}.)

@code{SimpleXMLRPCServer} also has a @code{_send_traceback_header}
attribute; if true, the exception and formatted traceback are returned
as HTTP headers “X-Exception” and “X-Traceback”.  This feature is
for debugging purposes only and should not be used on production servers
because the tracebacks might reveal passwords or other sensitive
information.  (Contributed by Alan McIntyre as part of his
project for Google’s Summer of Code 2007.)

@item 
The @code{xmlrpclib} module no longer automatically converts
@ref{3e2,,datetime.date} and @ref{364,,datetime.time} to the
@code{xmlrpclib.DateTime} type; the conversion semantics were
not necessarily correct for all applications.  Code using
@code{xmlrpclib} should convert @code{date} and @ref{364,,time}
instances. (bpo-1330538@footnote{https://bugs.python.org/issue1330538})  The code can also handle
dates before 1900 (contributed by Ralf Schmitt; bpo-2014@footnote{https://bugs.python.org/issue2014})
and 64-bit integers represented by using @code{<i8>} in XML-RPC responses
(contributed by Riku Lindblad; bpo-2985@footnote{https://bugs.python.org/issue2985}).

@item 
The @ref{149,,zipfile} module’s @code{ZipFile} class now has
@code{extract()} and @code{extractall()} methods that will unpack
a single file or all the files in the archive to the current directory, or
to a specified directory:

@example
z = zipfile.ZipFile('python-251.zip')

# Unpack a single file, writing it relative
# to the /tmp directory.
z.extract('Python/sysmodule.c', '/tmp')

# Unpack all the files in the archive.
z.extractall()
@end example

(Contributed by Alan McIntyre; bpo-467924@footnote{https://bugs.python.org/issue467924}.)

The @ref{1c8,,open()}, @code{read()} and @code{extract()} methods can now
take either a filename or a @code{ZipInfo} object.  This is useful when an
archive accidentally contains a duplicated filename.
(Contributed by Graham Horler; bpo-1775025@footnote{https://bugs.python.org/issue1775025}.)

Finally, @ref{149,,zipfile} now supports using Unicode filenames
for archived files.  (Contributed by Alexey Borzenkov; bpo-1734346@footnote{https://bugs.python.org/issue1734346}.)
@end itemize

@c ======================================================================

@c whole new modules get described in subsections here

@menu
* The ast module:: 
* The future_builtins module:: 
* The json module; JavaScript Object Notation: The json module JavaScript Object Notation. 
* The plistlib module; A Property-List Parser: The plistlib module A Property-List Parser. 
* ctypes Enhancements:: 
* Improved SSL Support:: 

@end menu

@node The ast module,The future_builtins module,,New and Improved Modules<2>
@anchor{whatsnew/2 6 the-ast-module}@anchor{f45}
@subsubsection The @code{ast} module


The @ref{7,,ast} module provides an Abstract Syntax Tree
representation of Python code, and Armin Ronacher
contributed a set of helper functions that perform a variety of
common tasks.  These will be useful for HTML templating
packages, code analyzers, and similar tools that process
Python code.

The @code{parse()} function takes an expression and returns an AST.
The @code{dump()} function outputs a representation of a tree, suitable
for debugging:

@example
import ast

t = ast.parse("""
d = @{@}
for i in 'abcdefghijklm':
    d[i + i] = ord(i) - ord('a') + 1
print d
""")
print ast.dump(t)
@end example

This outputs a deeply nested tree:

@example
Module(body=[
  Assign(targets=[
    Name(id='d', ctx=Store())
   ], value=Dict(keys=[], values=[]))
  For(target=Name(id='i', ctx=Store()),
      iter=Str(s='abcdefghijklm'), body=[
    Assign(targets=[
      Subscript(value=
        Name(id='d', ctx=Load()),
          slice=
          Index(value=
            BinOp(left=Name(id='i', ctx=Load()), op=Add(),
             right=Name(id='i', ctx=Load()))), ctx=Store())
     ], value=
     BinOp(left=
      BinOp(left=
       Call(func=
        Name(id='ord', ctx=Load()), args=[
          Name(id='i', ctx=Load())
         ], keywords=[], starargs=None, kwargs=None),
       op=Sub(), right=Call(func=
        Name(id='ord', ctx=Load()), args=[
          Str(s='a')
         ], keywords=[], starargs=None, kwargs=None)),
       op=Add(), right=Num(n=1)))
    ], orelse=[])
   Print(dest=None, values=[
     Name(id='d', ctx=Load())
   ], nl=True)
 ])
@end example

The @code{literal_eval()} method takes a string or an AST
representing a literal expression, parses and evaluates it, and
returns the resulting value.  A literal expression is a Python
expression containing only strings, numbers, dictionaries,
etc. but no statements or function calls.  If you need to
evaluate an expression but cannot accept the security risk of using an
@ref{34a,,eval()} call, @code{literal_eval()} will handle it safely:

@example
>>> literal = '("a", "b", @{2:4, 3:8, 1:2@})'
>>> print ast.literal_eval(literal)
('a', 'b', @{1: 2, 2: 4, 3: 8@})
>>> print ast.literal_eval('"a" + "b"')
Traceback (most recent call last):
  ...
ValueError: malformed string
@end example

The module also includes @code{NodeVisitor} and
@code{NodeTransformer} classes for traversing and modifying an AST,
and functions for common transformations such as changing line
numbers.

@c ======================================================================

@node The future_builtins module,The json module JavaScript Object Notation,The ast module,New and Improved Modules<2>
@anchor{whatsnew/2 6 the-future-builtins-module}@anchor{f46}
@subsubsection The @code{future_builtins} module


Python 3.0 makes many changes to the repertoire of built-in
functions, and most of the changes can’t be introduced in the Python
2.x series because they would break compatibility.
The @code{future_builtins} module provides versions
of these built-in functions that can be imported when writing
3.0-compatible code.

The functions in this module currently include:


@itemize *

@item 
@code{ascii(obj)}: equivalent to @ref{27d,,repr()}.  In Python 3.0,
@ref{27d,,repr()} will return a Unicode string, while @ref{f47,,ascii()} will
return a pure ASCII bytestring.

@item 
@code{filter(predicate, iterable)},
@code{map(func, iterable1, ...)}: the 3.0 versions
return iterators, unlike the 2.x builtins which return lists.

@item 
@code{hex(value)}, @code{oct(value)}: instead of calling the
@code{__hex__()} or @code{__oct__()} methods, these versions will
call the @code{__index__()} method and convert the result to hexadecimal
or octal.  @ref{e6e,,oct()} will use the new @code{0o} notation for its
result.
@end itemize

@c ======================================================================

@node The json module JavaScript Object Notation,The plistlib module A Property-List Parser,The future_builtins module,New and Improved Modules<2>
@anchor{whatsnew/2 6 the-json-module-javascript-object-notation}@anchor{f48}
@subsubsection The @code{json} module: JavaScript Object Notation


The new @ref{a0,,json} module supports the encoding and decoding of Python types in
JSON (Javascript Object Notation). JSON is a lightweight interchange format
often used in web applications. For more information about JSON, see
@indicateurl{http://www.json.org}.

@ref{a0,,json} comes with support for decoding and encoding most built-in Python
types. The following example encodes and decodes a dictionary:

@example
>>> import json
>>> data = @{"spam": "foo", "parrot": 42@}
>>> in_json = json.dumps(data) # Encode the data
>>> in_json
'@{"parrot": 42, "spam": "foo"@}'
>>> json.loads(in_json) # Decode into a Python object
@{"spam": "foo", "parrot": 42@}
@end example

It’s also possible to write your own decoders and encoders to support
more types. Pretty-printing of the JSON strings is also supported.

@ref{a0,,json} (originally called simplejson) was written by Bob
Ippolito.

@c ======================================================================

@node The plistlib module A Property-List Parser,ctypes Enhancements,The json module JavaScript Object Notation,New and Improved Modules<2>
@anchor{whatsnew/2 6 the-plistlib-module-a-property-list-parser}@anchor{f49}
@subsubsection The @code{plistlib} module: A Property-List Parser


The @code{.plist} format is commonly used on Mac OS X to
store basic data types (numbers, strings, lists,
and dictionaries) by serializing them into an XML-based format.
It resembles the XML-RPC serialization of data types.

Despite being primarily used on Mac OS X, the format
has nothing Mac-specific about it and the Python implementation works
on any platform that Python supports, so the @ref{ca,,plistlib} module
has been promoted to the standard library.

Using the module is simple:

@example
import sys
import plistlib
import datetime

# Create data structure
data_struct = dict(lastAccessed=datetime.datetime.now(),
                   version=1,
                   categories=('Personal','Shared','Private'))

# Create string containing XML.
plist_str = plistlib.writePlistToString(data_struct)
new_struct = plistlib.readPlistFromString(plist_str)
print data_struct
print new_struct

# Write data structure to a file and read it back.
plistlib.writePlist(data_struct, '/tmp/customizations.plist')
new_struct = plistlib.readPlist('/tmp/customizations.plist')

# read/writePlist accepts file-like objects as well as paths.
plistlib.writePlist(data_struct, sys.stdout)
@end example

@c ======================================================================

@node ctypes Enhancements,Improved SSL Support,The plistlib module A Property-List Parser,New and Improved Modules<2>
@anchor{whatsnew/2 6 ctypes-enhancements}@anchor{f4a}
@subsubsection ctypes Enhancements


Thomas Heller continued to maintain and enhance the
@ref{29,,ctypes} module.

@ref{29,,ctypes} now supports a @code{c_bool} datatype
that represents the C99 @code{bool} type.  (Contributed by David Remahl;
bpo-1649190@footnote{https://bugs.python.org/issue1649190}.)

The @ref{29,,ctypes} string, buffer and array types have improved
support for extended slicing syntax,
where various combinations of @code{(start, stop, step)} are supplied.
(Implemented by Thomas Wouters.)

@c Revision 57769

All @ref{29,,ctypes} data types now support
@code{from_buffer()} and @code{from_buffer_copy()}
methods that create a ctypes instance based on a
provided buffer object.  @code{from_buffer_copy()} copies
the contents of the object,
while @code{from_buffer()} will share the same memory area.

A new calling convention tells @ref{29,,ctypes} to clear the @code{errno} or
Win32 LastError variables at the outset of each wrapped call.
(Implemented by Thomas Heller; bpo-1798@footnote{https://bugs.python.org/issue1798}.)

You can now retrieve the Unix @code{errno} variable after a function
call.  When creating a wrapped function, you can supply
@code{use_errno=True} as a keyword parameter to the @code{DLL()} function
and then call the module-level methods @code{set_errno()} and
@code{get_errno()} to set and retrieve the error value.

The Win32 LastError variable is similarly supported by
the @code{DLL()}, @code{OleDLL()}, and @code{WinDLL()} functions.
You supply @code{use_last_error=True} as a keyword parameter
and then call the module-level methods @code{set_last_error()}
and @code{get_last_error()}.

The @code{byref()} function, used to retrieve a pointer to a ctypes
instance, now has an optional @emph{offset} parameter that is a byte
count that will be added to the returned pointer.

@c ======================================================================

@node Improved SSL Support,,ctypes Enhancements,New and Improved Modules<2>
@anchor{whatsnew/2 6 improved-ssl-support}@anchor{f4b}
@subsubsection Improved SSL Support


Bill Janssen made extensive improvements to Python 2.6’s support for
the Secure Sockets Layer by adding a new module, @ref{ee,,ssl}, that’s
built atop the OpenSSL@footnote{https://www.openssl.org/} library.
This new module provides more control over the protocol negotiated,
the X.509 certificates used, and has better support for writing SSL
servers (as opposed to clients) in Python.  The existing SSL support
in the @ref{ea,,socket} module hasn’t been removed and continues to work,
though it will be removed in Python 3.0.

To use the new module, you must first create a TCP connection in the
usual way and then pass it to the @ref{331,,ssl.wrap_socket()} function.
It’s possible to specify whether a certificate is required, and to
obtain certificate info by calling the @code{getpeercert()} method.


@subsubheading See also


The documentation for the @ref{ee,,ssl} module.


@c ======================================================================

@node Deprecations and Removals,Build and C API Changes<9>,New and Improved Modules<2>,What’s New in Python 2 6
@anchor{whatsnew/2 6 deprecations-and-removals}@anchor{f4c}
@subsection Deprecations and Removals



@itemize *

@item 
String exceptions have been removed.  Attempting to use them raises a
@ref{162,,TypeError}.

@item 
Changes to the @ref{4d8,,Exception} interface
as dictated by 
@geindex Python Enhancement Proposals; PEP 352
PEP 352@footnote{https://www.python.org/dev/peps/pep-0352} continue to be made.  For 2.6,
the @code{message} attribute is being deprecated in favor of the
@code{args} attribute.

@item 
(3.0-warning mode) Python 3.0 will feature a reorganized standard
library that will drop many outdated modules and rename others.
Python 2.6 running in 3.0-warning mode will warn about these modules
when they are imported.

The list of deprecated modules is:
@code{audiodev},
@code{bgenlocations},
@code{buildtools},
@code{bundlebuilder},
@code{Canvas},
@code{compiler},
@code{dircache},
@code{dl},
@code{fpformat},
@code{gensuitemodule},
@code{ihooks},
@code{imageop},
@code{imgfile},
@code{linuxaudiodev},
@code{mhlib},
@code{mimetools},
@code{multifile},
@code{new},
@code{pure},
@code{statvfs},
@code{sunaudiodev},
@code{test.testall}, and
@code{toaiff}.

@item 
The @code{gopherlib} module has been removed.

@item 
The @code{MimeWriter} module and @code{mimify} module
have been deprecated; use the @ref{64,,email}
package instead.

@item 
The @code{md5} module has been deprecated; use the @ref{88,,hashlib} module
instead.

@item 
The @code{posixfile} module has been deprecated; @ref{f4d,,fcntl.lockf()}
provides better locking.

@item 
The @code{popen2} module has been deprecated; use the @ref{f4,,subprocess}
module.

@item 
The @code{rgbimg} module has been removed.

@item 
The @code{sets} module has been deprecated; it’s better to
use the built-in @ref{370,,set} and @ref{439,,frozenset} types.

@item 
The @code{sha} module has been deprecated; use the @ref{88,,hashlib} module
instead.
@end itemize

@c ======================================================================

@node Build and C API Changes<9>,Porting to Python 2 6,Deprecations and Removals,What’s New in Python 2 6
@anchor{whatsnew/2 6 build-and-c-api-changes}@anchor{f4e}
@subsection Build and C API Changes


Changes to Python’s build process and to the C API include:


@itemize *

@item 
Python now must be compiled with C89 compilers (after 19
years!).  This means that the Python source tree has dropped its
own implementations of @code{memmove()} and @code{strerror()}, which
are in the C89 standard library.

@item 
Python 2.6 can be built with Microsoft Visual Studio 2008 (version
9.0), and this is the new default compiler.  See the
@code{PCbuild} directory for the build files.  (Implemented by
Christian Heimes.)

@item 
On Mac OS X, Python 2.6 can be compiled as a 4-way universal build.
The @code{configure} script
can take a @code{--with-universal-archs=[32-bit|64-bit|all]}
switch, controlling whether the binaries are built for 32-bit
architectures (x86, PowerPC), 64-bit (x86-64 and PPC-64), or both.
(Contributed by Ronald Oussoren.)

@item 
The BerkeleyDB module now has a C API object, available as
@code{bsddb.db.api}.   This object can be used by other C extensions
that wish to use the @code{bsddb} module for their own purposes.
(Contributed by Duncan Grisby.)

@item 
The new buffer interface, previously described in
@ref{f23,,the PEP 3118 section},
adds @ref{1dd,,PyObject_GetBuffer()} and @ref{1e5,,PyBuffer_Release()},
as well as a few other functions.

@item 
Python’s use of the C stdio library is now thread-safe, or at least
as thread-safe as the underlying library is.  A long-standing potential
bug occurred if one thread closed a file object while another thread
was reading from or writing to the object.  In 2.6 file objects
have a reference count, manipulated by the
@code{PyFile_IncUseCount()} and @code{PyFile_DecUseCount()}
functions.  File objects can’t be closed unless the reference count
is zero.  @code{PyFile_IncUseCount()} should be called while the GIL
is still held, before carrying out an I/O operation using the
@code{FILE *} pointer, and @code{PyFile_DecUseCount()} should be called
immediately after the GIL is re-acquired.
(Contributed by Antoine Pitrou and Gregory P. Smith.)

@item 
Importing modules simultaneously in two different threads no longer
deadlocks; it will now raise an @ref{3b8,,ImportError}.  A new API
function, @ref{c11,,PyImport_ImportModuleNoBlock()}, will look for a
module in @code{sys.modules} first, then try to import it after
acquiring an import lock.  If the import lock is held by another
thread, an @ref{3b8,,ImportError} is raised.
(Contributed by Christian Heimes.)

@item 
Several functions return information about the platform’s
floating-point support.  @ref{f4f,,PyFloat_GetMax()} returns
the maximum representable floating point value,
and @ref{f50,,PyFloat_GetMin()} returns the minimum
positive value.  @ref{f51,,PyFloat_GetInfo()} returns an object
containing more information from the @code{float.h} file, such as
@code{"mant_dig"} (number of digits in the mantissa), @code{"epsilon"}
(smallest difference between 1.0 and the next largest value
representable), and several others.
(Contributed by Christian Heimes; bpo-1534@footnote{https://bugs.python.org/issue1534}.)

@item 
C functions and methods that use
@ref{f52,,PyComplex_AsCComplex()} will now accept arguments that
have a @code{__complex__()} method.  In particular, the functions in the
@ref{17,,cmath} module will now accept objects with this method.
This is a backport of a Python 3.0 change.
(Contributed by Mark Dickinson; bpo-1675423@footnote{https://bugs.python.org/issue1675423}.)

@item 
Python’s C API now includes two functions for case-insensitive string
comparisons, @code{PyOS_stricmp(char*, char*)}
and @code{PyOS_strnicmp(char*, char*, Py_ssize_t)}.
(Contributed by Christian Heimes; bpo-1635@footnote{https://bugs.python.org/issue1635}.)

@item 
Many C extensions define their own little macro for adding
integers and strings to the module’s dictionary in the
@code{init*} function.  Python 2.6 finally defines standard macros
for adding values to a module, @ref{f53,,PyModule_AddStringMacro}
and @ref{f54,,PyModule_AddIntMacro()}.  (Contributed by
Christian Heimes.)

@item 
Some macros were renamed in both 3.0 and 2.6 to make it clearer that
they are macros,
not functions.  @code{Py_Size()} became @ref{1f3,,Py_SIZE()},
@code{Py_Type()} became @ref{1f1,,Py_TYPE()}, and
@code{Py_Refcnt()} became @ref{382,,Py_REFCNT()}.
The mixed-case macros are still available
in Python 2.6 for backward compatibility.
(bpo-1629@footnote{https://bugs.python.org/issue1629})

@item 
Distutils now places C extensions it builds in a
different directory when running on a debug version of Python.
(Contributed by Collin Winter; bpo-1530959@footnote{https://bugs.python.org/issue1530959}.)

@item 
Several basic data types, such as integers and strings, maintain
internal free lists of objects that can be re-used.  The data
structures for these free lists now follow a naming convention: the
variable is always named @code{free_list}, the counter is always named
@code{numfree}, and a macro @code{Py<typename>_MAXFREELIST} is
always defined.

@item 
A new Makefile target, “make patchcheck”, prepares the Python source tree
for making a patch: it fixes trailing whitespace in all modified
@code{.py} files, checks whether the documentation has been changed,
and reports whether the @code{Misc/ACKS} and @code{Misc/NEWS} files
have been updated.
(Contributed by Brett Cannon.)

Another new target, “make profile-opt”, compiles a Python binary
using GCC’s profile-guided optimization.  It compiles Python with
profiling enabled, runs the test suite to obtain a set of profiling
results, and then compiles using these results for optimization.
(Contributed by Gregory P. Smith.)
@end itemize

@c ======================================================================

@menu
* Port-Specific Changes; Windows: Port-Specific Changes Windows<2>. 
* Port-Specific Changes; Mac OS X: Port-Specific Changes Mac OS X<2>. 
* Port-Specific Changes; IRIX: Port-Specific Changes IRIX. 

@end menu

@node Port-Specific Changes Windows<2>,Port-Specific Changes Mac OS X<2>,,Build and C API Changes<9>
@anchor{whatsnew/2 6 port-specific-changes-windows}@anchor{f55}
@subsubsection Port-Specific Changes: Windows



@itemize *

@item 
The support for Windows 95, 98, ME and NT4 has been dropped.
Python 2.6 requires at least Windows 2000 SP4.

@item 
The new default compiler on Windows is Visual Studio 2008 (version
9.0). The build directories for Visual Studio 2003 (version 7.1) and
2005 (version 8.0) were moved into the PC/ directory. The new
@code{PCbuild} directory supports cross compilation for X64, debug
builds and Profile Guided Optimization (PGO). PGO builds are roughly
10% faster than normal builds.  (Contributed by Christian Heimes
with help from Amaury Forgeot d’Arc and Martin von Löwis.)

@item 
The @ref{b2,,msvcrt} module now supports
both the normal and wide char variants of the console I/O
API.  The @code{getwch()} function reads a keypress and returns a Unicode
value, as does the @code{getwche()} function.  The @code{putwch()} function
takes a Unicode character and writes it to the console.
(Contributed by Christian Heimes.)

@item 
@ref{f3f,,os.path.expandvars()} will now expand environment variables in
the form “%var%”, and “~user” will be expanded into the user’s home
directory path.  (Contributed by Josiah Carlson; bpo-957650@footnote{https://bugs.python.org/issue957650}.)

@item 
The @ref{ea,,socket} module’s socket objects now have an
@code{ioctl()} method that provides a limited interface to the
@code{WSAIoctl()} system interface.

@item 
The @code{_winreg} module now has a function,
@code{ExpandEnvironmentStrings()},
that expands environment variable references such as @code{%NAME%}
in an input string.  The handle objects provided by this
module now support the context protocol, so they can be used
in @ref{168,,with} statements. (Contributed by Christian Heimes.)

@code{_winreg} also has better support for x64 systems,
exposing the @code{DisableReflectionKey()}, @code{EnableReflectionKey()},
and @code{QueryReflectionKey()} functions, which enable and disable
registry reflection for 32-bit processes running on 64-bit systems.
(bpo-1753245@footnote{https://bugs.python.org/issue1753245})

@item 
The @ref{b1,,msilib} module’s @code{Record} object
gained @code{GetInteger()} and @code{GetString()} methods that
return field values as an integer or a string.
(Contributed by Floris Bruynooghe; bpo-2125@footnote{https://bugs.python.org/issue2125}.)
@end itemize

@c ======================================================================

@node Port-Specific Changes Mac OS X<2>,Port-Specific Changes IRIX,Port-Specific Changes Windows<2>,Build and C API Changes<9>
@anchor{whatsnew/2 6 port-specific-changes-mac-os-x}@anchor{f56}
@subsubsection Port-Specific Changes: Mac OS X



@itemize *

@item 
When compiling a framework build of Python, you can now specify the
framework name to be used by providing the
@code{--with-framework-name=} option to the
@code{configure} script.

@item 
The @code{macfs} module has been removed.  This in turn required the
@code{macostools.touched()} function to be removed because it depended on the
@code{macfs} module.  (bpo-1490190@footnote{https://bugs.python.org/issue1490190})

@item 
Many other Mac OS modules have been deprecated and will be removed in
Python 3.0:
@code{_builtinSuites},
@code{aepack},
@code{aetools},
@code{aetypes},
@code{applesingle},
@code{appletrawmain},
@code{appletrunner},
@code{argvemulator},
@code{Audio_mac},
@code{autoGIL},
@code{Carbon},
@code{cfmfile},
@code{CodeWarrior},
@code{ColorPicker},
@code{EasyDialogs},
@code{Explorer},
@code{Finder},
@code{FrameWork},
@code{findertools},
@code{ic},
@code{icglue},
@code{icopen},
@code{macerrors},
@code{MacOS},
@code{macfs},
@code{macostools},
@code{macresource},
@code{MiniAEFrame},
@code{Nav},
@code{Netscape},
@code{OSATerminology},
@code{pimp},
@code{PixMapWrapper},
@code{StdSuites},
@code{SystemEvents},
@code{Terminal}, and
@code{terminalcommand}.
@end itemize

@c ======================================================================

@node Port-Specific Changes IRIX,,Port-Specific Changes Mac OS X<2>,Build and C API Changes<9>
@anchor{whatsnew/2 6 port-specific-changes-irix}@anchor{f57}
@subsubsection Port-Specific Changes: IRIX


A number of old IRIX-specific modules were deprecated and will
be removed in Python 3.0:
@code{al} and @code{AL},
@code{cd},
@code{cddb},
@code{cdplayer},
@code{CL} and @code{cl},
@code{DEVICE},
@code{ERRNO},
@code{FILE},
@code{FL} and @code{fl},
@code{flp},
@code{fm},
@code{GET},
@code{GLWS},
@code{GL} and @code{gl},
@code{IN},
@code{IOCTL},
@code{jpeg},
@code{panelparser},
@code{readcd},
@code{SV} and @code{sv},
@code{torgb},
@code{videoreader}, and
@code{WAIT}.

@c ======================================================================

@node Porting to Python 2 6,Acknowledgements<2>,Build and C API Changes<9>,What’s New in Python 2 6
@anchor{whatsnew/2 6 porting-to-python-2-6}@anchor{f58}
@subsection Porting to Python 2.6


This section lists previously described changes and other bugfixes
that may require changes to your code:


@itemize *

@item 
Classes that aren’t supposed to be hashable should
set @code{__hash__ = None} in their definitions to indicate
the fact.

@item 
String exceptions have been removed.  Attempting to use them raises a
@ref{162,,TypeError}.

@item 
The @code{__init__()} method of @ref{7d3,,collections.deque}
now clears any existing contents of the deque
before adding elements from the iterable.  This change makes the
behavior match @code{list.__init__()}.

@item 
@ref{f59,,object.__init__()} previously accepted arbitrary arguments and
keyword arguments, ignoring them.  In Python 2.6, this is no longer
allowed and will result in a @ref{162,,TypeError}.  This will affect
@code{__init__()} methods that end up calling the corresponding
method on @ref{5a5,,object} (perhaps through using @ref{78c,,super()}).
See bpo-1683368@footnote{https://bugs.python.org/issue1683368} for discussion.

@item 
The @code{Decimal} constructor now accepts leading and trailing
whitespace when passed a string.  Previously it would raise an
@code{InvalidOperation} exception.  On the other hand, the
@code{create_decimal()} method of @code{Context} objects now
explicitly disallows extra whitespace, raising a
@code{ConversionSyntax} exception.

@item 
Due to an implementation accident, if you passed a file path to
the built-in  @ref{3b7,,__import__()} function, it would actually import
the specified file.  This was never intended to work, however, and
the implementation now explicitly checks for this case and raises
an @ref{3b8,,ImportError}.

@item 
C API: the @ref{f5a,,PyImport_Import()} and @ref{34f,,PyImport_ImportModule()}
functions now default to absolute imports, not relative imports.
This will affect C extensions that import other modules.

@item 
C API: extension data types that shouldn’t be hashable
should define their @code{tp_hash} slot to
@ref{f2d,,PyObject_HashNotImplemented()}.

@item 
The @ref{ea,,socket} module exception @ref{beb,,socket.error} now inherits
from @ref{be8,,IOError}.  Previously it wasn’t a subclass of
@code{StandardError} but now it is, through @ref{be8,,IOError}.
(Implemented by Gregory P. Smith; bpo-1706815@footnote{https://bugs.python.org/issue1706815}.)

@item 
The @code{xmlrpclib} module no longer automatically converts
@ref{3e2,,datetime.date} and @ref{364,,datetime.time} to the
@code{xmlrpclib.DateTime} type; the conversion semantics were
not necessarily correct for all applications.  Code using
@code{xmlrpclib} should convert @code{date} and @ref{364,,time}
instances. (bpo-1330538@footnote{https://bugs.python.org/issue1330538})

@item 
(3.0-warning mode) The @ref{4d8,,Exception} class now warns
when accessed using slicing or index access; having
@ref{4d8,,Exception} behave like a tuple is being phased out.

@item 
(3.0-warning mode) inequality comparisons between two dictionaries
or two objects that don’t implement comparison methods are reported
as warnings.  @code{dict1 == dict2} still works, but @code{dict1 < dict2}
is being phased out.

Comparisons between cells, which are an implementation detail of Python’s
scoping rules, also cause warnings because such comparisons are forbidden
entirely in 3.0.
@end itemize

@c ======================================================================

@node Acknowledgements<2>,,Porting to Python 2 6,What’s New in Python 2 6
@anchor{whatsnew/2 6 acknowledgements}@anchor{f5b}@anchor{whatsnew/2 6 acks}@anchor{f5c}
@subsection Acknowledgements


The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Georg Brandl, Steve Brown, Nick Coghlan, Ralph Corderoy,
Jim Jewett, Kent Johnson, Chris Lambacher,  Martin Michlmayr,
Antoine Pitrou, Brian Warner.

@node What’s New in Python 2 5,What’s New in Python 2 4,What’s New in Python 2 6,What’s New in Python
@anchor{whatsnew/2 5 doc}@anchor{f5d}@anchor{whatsnew/2 5 what-s-new-in-python-2-5}@anchor{f5e}
@section What’s New in Python 2.5


@*Author: 
A.M. Kuchling


@c $Id: whatsnew25.tex 56611 2007-07-29 08:26:10Z georg.brandl $

@c Fix XXX comments

This article explains the new features in Python 2.5.  The final release of
Python 2.5 is scheduled for August 2006; 
@geindex Python Enhancement Proposals; PEP 356
PEP 356@footnote{https://www.python.org/dev/peps/pep-0356} describes the planned
release schedule.

The changes in Python 2.5 are an interesting mix of language and library
improvements. The library enhancements will be more important to Python’s user
community, I think, because several widely-useful packages were added.  New
modules include ElementTree for XML processing (@code{xml.etree}),
the SQLite database module (@code{sqlite}), and the @ref{29,,ctypes}
module for calling C functions.

The language changes are of middling significance.  Some pleasant new features
were added, but most of them aren’t features that you’ll use every day.
Conditional expressions were finally added to the language using a novel syntax;
see section @ref{f5f,,PEP 308; Conditional Expressions}.  The new ‘@ref{168,,with}’ statement will make
writing cleanup code easier (section @ref{f60,,PEP 343; The ‘with’ statement}).  Values can now be passed
into generators (section @ref{f61,,PEP 342; New Generator Features}).  Imports are now visible as either
absolute or relative (section @ref{f62,,PEP 328; Absolute and Relative Imports}).  Some corner cases of exception
handling are handled better (section @ref{f63,,PEP 341; Unified try/except/finally}).  All these improvements
are worthwhile, but they’re improvements to one specific language feature or
another; none of them are broad modifications to Python’s semantics.

As well as the language and library additions, other improvements and bugfixes
were made throughout the source tree.  A search through the SVN change logs
finds there were 353 patches applied and 458 bugs fixed between Python 2.4 and
2.5.  (Both figures are likely to be underestimates.)

This article doesn’t try to be a complete specification of the new features;
instead changes are briefly introduced using helpful examples.  For full
details, you should always refer to the documentation for Python 2.5 at
@indicateurl{https://docs.python.org}. If you want to understand the complete implementation
and design rationale, refer to the PEP for a particular new feature.

Comments, suggestions, and error reports for this document are welcome; please
e-mail them to the author or open a bug in the Python bug tracker.

@c ======================================================================

@menu
* PEP 308; Conditional Expressions: PEP 308 Conditional Expressions. 
* PEP 309; Partial Function Application: PEP 309 Partial Function Application. 
* PEP 314; Metadata for Python Software Packages v1.1: PEP 314 Metadata for Python Software Packages v1 1. 
* PEP 328; Absolute and Relative Imports: PEP 328 Absolute and Relative Imports. 
* PEP 338; Executing Modules as Scripts: PEP 338 Executing Modules as Scripts. 
* PEP 341; Unified try/except/finally: PEP 341 Unified try/except/finally. 
* PEP 342; New Generator Features: PEP 342 New Generator Features. 
* PEP 343; The ‘with’ statement: PEP 343 The ‘with’ statement<2>. 
* PEP 352; Exceptions as New-Style Classes: PEP 352 Exceptions as New-Style Classes. 
* PEP 353; Using ssize_t as the index type: PEP 353 Using ssize_t as the index type. 
* PEP 357; The ‘__index__’ method: PEP 357 The ‘__index__’ method. 
* Other Language Changes: Other Language Changes<14>. 
* New@comma{} Improved@comma{} and Removed Modules: New Improved and Removed Modules. 
* Build and C API Changes: Build and C API Changes<10>. 
* Porting to Python 2.5: Porting to Python 2 5. 
* Acknowledgements: Acknowledgements<3>. 

@end menu

@node PEP 308 Conditional Expressions,PEP 309 Partial Function Application,,What’s New in Python 2 5
@anchor{whatsnew/2 5 pep-308}@anchor{f5f}@anchor{whatsnew/2 5 pep-308-conditional-expressions}@anchor{f64}
@subsection PEP 308: Conditional Expressions


For a long time, people have been requesting a way to write conditional
expressions, which are expressions that return value A or value B depending on
whether a Boolean value is true or false.  A conditional expression lets you
write a single assignment statement that has the same effect as the following:

@example
if condition:
    x = true_value
else:
    x = false_value
@end example

There have been endless tedious discussions of syntax on both python-dev and
comp.lang.python.  A vote was even held that found the majority of voters wanted
conditional expressions in some form, but there was no syntax that was preferred
by a clear majority. Candidates included C’s @code{cond ? true_v : false_v}, @code{if
cond then true_v else false_v}, and 16 other variations.

Guido van Rossum eventually chose a surprising syntax:

@example
x = true_value if condition else false_value
@end example

Evaluation is still lazy as in existing Boolean expressions, so the order of
evaluation jumps around a bit.  The @emph{condition} expression in the middle is
evaluated first, and the @emph{true_value} expression is evaluated only if the
condition was true.  Similarly, the @emph{false_value} expression is only evaluated
when the condition is false.

This syntax may seem strange and backwards; why does the condition go in the
@emph{middle} of the expression, and not in the front as in C’s @code{c ? x : y}?  The
decision was checked by applying the new syntax to the modules in the standard
library and seeing how the resulting code read.  In many cases where a
conditional expression is used, one value seems to be the ‘common case’ and one
value is an ‘exceptional case’, used only on rarer occasions when the condition
isn’t met.  The conditional syntax makes this pattern a bit more obvious:

@example
contents = ((doc + '\n') if doc else '@w{'})
@end example

I read the above statement as meaning “here @emph{contents} is  usually assigned a
value of @code{doc+'\n'}; sometimes  @emph{doc} is empty, in which special case an empty
string is returned.”   I doubt I will use conditional expressions very often
where there  isn’t a clear common and uncommon case.

There was some discussion of whether the language should require surrounding
conditional expressions with parentheses.  The decision was made to @emph{not}
require parentheses in the Python language’s grammar, but as a matter of style I
think you should always use them. Consider these two statements:

@example
# First version -- no parens
level = 1 if logging else 0

# Second version -- with parens
level = (1 if logging else 0)
@end example

In the first version, I think a reader’s eye might group the statement into
‘level = 1’, ‘if logging’, ‘else 0’, and think that the condition decides
whether the assignment to @emph{level} is performed.  The second version reads
better, in my opinion, because it makes it clear that the assignment is always
performed and the choice is being made between two values.

Another reason for including the brackets: a few odd combinations of list
comprehensions and lambdas could look like incorrect conditional expressions.
See 
@geindex Python Enhancement Proposals; PEP 308
PEP 308@footnote{https://www.python.org/dev/peps/pep-0308} for some examples.  If you put parentheses around your
conditional expressions, you won’t run into this case.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 308

@item PEP 308@footnote{https://www.python.org/dev/peps/pep-0308} - Conditional Expressions

PEP written by Guido van Rossum and Raymond D. Hettinger; implemented by Thomas
Wouters.
@end table


@c ======================================================================

@node PEP 309 Partial Function Application,PEP 314 Metadata for Python Software Packages v1 1,PEP 308 Conditional Expressions,What’s New in Python 2 5
@anchor{whatsnew/2 5 pep-309}@anchor{f65}@anchor{whatsnew/2 5 pep-309-partial-function-application}@anchor{f66}
@subsection PEP 309: Partial Function Application


The @ref{7f,,functools} module is intended to contain tools for functional-style
programming.

One useful tool in this module is the @code{partial()} function. For programs
written in a functional style, you’ll sometimes want to construct variants of
existing functions that have some of the parameters filled in.  Consider a
Python function @code{f(a, b, c)}; you could create a new function @code{g(b, c)} that
was equivalent to @code{f(1, b, c)}.  This is called “partial function
application”.

@code{partial()} takes the arguments @code{(function, arg1, arg2, ... kwarg1=value1,
kwarg2=value2)}.  The resulting object is callable, so you can just call it to
invoke @emph{function} with the filled-in arguments.

Here’s a small but realistic example:

@example
import functools

def log (message, subsystem):
    "Write the contents of 'message' to the specified subsystem."
    print '%s: %s' % (subsystem, message)
    ...

server_log = functools.partial(log, subsystem='server')
server_log('Unable to open socket')
@end example

Here’s another example, from a program that uses PyGTK.  Here a context-sensitive
pop-up menu is being constructed dynamically.  The callback provided
for the menu option is a partially applied version of the @code{open_item()}
method, where the first argument has been provided.

@example
...
class Application:
    def open_item(self, path):
       ...
    def init (self):
        open_func = functools.partial(self.open_item, item_path)
        popup_menu.append( ("Open", open_func, 1) )
@end example

Another function in the @ref{7f,,functools} module is the
@code{update_wrapper(wrapper, wrapped)} function that helps you write
well-behaved decorators.  @code{update_wrapper()} copies the name, module, and
docstring attribute to a wrapper function so that tracebacks inside the wrapped
function are easier to understand.  For example, you might write:

@example
def my_decorator(f):
    def wrapper(*args, **kwds):
        print 'Calling decorated function'
        return f(*args, **kwds)
    functools.update_wrapper(wrapper, f)
    return wrapper
@end example

@code{wraps()} is a decorator that can be used inside your own decorators to copy
the wrapped function’s information.  An alternate  version of the previous
example would be:

@example
def my_decorator(f):
    @@functools.wraps(f)
    def wrapper(*args, **kwds):
        print 'Calling decorated function'
        return f(*args, **kwds)
    return wrapper
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 309

@item PEP 309@footnote{https://www.python.org/dev/peps/pep-0309} - Partial Function Application

PEP proposed and written by Peter Harris; implemented by Hye-Shik Chang and Nick
Coghlan, with adaptations by Raymond Hettinger.
@end table


@c ======================================================================

@node PEP 314 Metadata for Python Software Packages v1 1,PEP 328 Absolute and Relative Imports,PEP 309 Partial Function Application,What’s New in Python 2 5
@anchor{whatsnew/2 5 pep-314}@anchor{f67}@anchor{whatsnew/2 5 pep-314-metadata-for-python-software-packages-v1-1}@anchor{f68}
@subsection PEP 314: Metadata for Python Software Packages v1.1


Some simple dependency support was added to Distutils.  The @code{setup()}
function now has @code{requires}, @code{provides}, and @code{obsoletes} keyword
parameters.  When you build a source distribution using the @code{sdist} command,
the dependency information will be recorded in the @code{PKG-INFO} file.

Another new keyword parameter is @code{download_url}, which should be set to a URL
for the package’s source code.  This means it’s now possible to look up an entry
in the package index, determine the dependencies for a package, and download the
required packages.

@example
VERSION = '1.0'
setup(name='PyPackage',
      version=VERSION,
      requires=['numarray', 'zlib (>=1.1.4)'],
      obsoletes=['OldPackage']
      download_url=('http://www.example.com/pypackage/dist/pkg-%s.tar.gz'
                    % VERSION),
     )
@end example

Another new enhancement to the Python package index at
@indicateurl{https://pypi.org} is storing source and binary archives for a
package.  The new @code{upload} Distutils command will upload a package to
the repository.

Before a package can be uploaded, you must be able to build a distribution using
the @code{sdist} Distutils command.  Once that works, you can run @code{python
setup.py upload} to add your package to the PyPI archive.  Optionally you can
GPG-sign the package by supplying the @code{--sign} and @code{--identity}
options.

Package uploading was implemented by Martin von Löwis and Richard Jones.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 314

@item PEP 314@footnote{https://www.python.org/dev/peps/pep-0314} - Metadata for Python Software Packages v1.1

PEP proposed and written by A.M. Kuchling, Richard Jones, and Fred Drake;
implemented by Richard Jones and Fred Drake.
@end table


@c ======================================================================

@node PEP 328 Absolute and Relative Imports,PEP 338 Executing Modules as Scripts,PEP 314 Metadata for Python Software Packages v1 1,What’s New in Python 2 5
@anchor{whatsnew/2 5 pep-328}@anchor{f62}@anchor{whatsnew/2 5 pep-328-absolute-and-relative-imports}@anchor{f69}
@subsection PEP 328: Absolute and Relative Imports


The simpler part of 
@geindex Python Enhancement Proposals; PEP 328
PEP 328@footnote{https://www.python.org/dev/peps/pep-0328} was implemented in Python 2.4: parentheses could now
be used to enclose the names imported from a module using the @code{from ... import
...} statement, making it easier to import many different names.

The more complicated part has been implemented in Python 2.5: importing a module
can be specified to use absolute or package-relative imports.  The plan is to
move toward making absolute imports the default in future versions of Python.

Let’s say you have a package directory like this:

@example
pkg/
pkg/__init__.py
pkg/main.py
pkg/string.py
@end example

This defines a package named @code{pkg} containing the @code{pkg.main} and
@code{pkg.string} submodules.

Consider the code in the @code{main.py} module.  What happens if it executes
the statement @code{import string}?  In Python 2.4 and earlier, it will first look
in the package’s directory to perform a relative import, finds
@code{pkg/string.py}, imports the contents of that file as the
@code{pkg.string} module, and that module is bound to the name @code{string} in the
@code{pkg.main} module’s namespace.

That’s fine if @code{pkg.string} was what you wanted.  But what if you wanted
Python’s standard @ref{f1,,string} module?  There’s no clean way to ignore
@code{pkg.string} and look for the standard module; generally you had to look at
the contents of @code{sys.modules}, which is slightly unclean.    Holger Krekel’s
@code{py.std} package provides a tidier way to perform imports from the standard
library, @code{import py; py.std.string.join()}, but that package isn’t available
on all Python installations.

Reading code which relies on relative imports is also less clear, because a
reader may be confused about which module, @ref{f1,,string} or @code{pkg.string},
is intended to be used.  Python users soon learned not to duplicate the names of
standard library modules in the names of their packages’ submodules, but you
can’t protect against having your submodule’s name being used for a new module
added in a future version of Python.

In Python 2.5, you can switch @ref{e35,,import}’s behaviour to  absolute imports
using a @code{from __future__ import absolute_import} directive.  This absolute-import
behaviour will become the default in a future version (probably Python
2.7).  Once absolute imports  are the default, @code{import string} will always
find the standard library’s version. It’s suggested that users should begin
using absolute imports as much as possible, so it’s preferable to begin writing
@code{from pkg import string} in your code.

Relative imports are still possible by adding a leading period  to the module
name when using the @code{from ... import} form:

@example
# Import names from pkg.string
from .string import name1, name2
# Import pkg.string
from . import string
@end example

This imports the @ref{f1,,string} module relative to the current package, so in
@code{pkg.main} this will import @emph{name1} and @emph{name2} from @code{pkg.string}.
Additional leading periods perform the relative import starting from the parent
of the current package.  For example, code in the @code{A.B.C} module can do:

@example
from . import D                 # Imports A.B.D
from .. import E                # Imports A.E
from ..F import G               # Imports A.F.G
@end example

Leading periods cannot be used with the @code{import modname}  form of the import
statement, only the @code{from ... import} form.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 328

@item PEP 328@footnote{https://www.python.org/dev/peps/pep-0328} - Imports: Multi-Line and Absolute/Relative

PEP written by Aahz; implemented by Thomas Wouters.

@item @indicateurl{https://pylib.readthedocs.io/}

The py library by Holger Krekel, which contains the @code{py.std} package.
@end table


@c ======================================================================

@node PEP 338 Executing Modules as Scripts,PEP 341 Unified try/except/finally,PEP 328 Absolute and Relative Imports,What’s New in Python 2 5
@anchor{whatsnew/2 5 pep-338}@anchor{f6a}@anchor{whatsnew/2 5 pep-338-executing-modules-as-scripts}@anchor{f6b}
@subsection PEP 338: Executing Modules as Scripts


The @ref{611,,-m} switch added in Python 2.4 to execute a module as a script
gained a few more abilities.  Instead of being implemented in C code inside the
Python interpreter, the switch now uses an implementation in a new module,
@ref{dd,,runpy}.

The @ref{dd,,runpy} module implements a more sophisticated import mechanism so that
it’s now possible to run modules in a package such as @code{pychecker.checker}.
The module also supports alternative import mechanisms such as the
@ref{14a,,zipimport} module.  This means you can add a .zip archive’s path to
@code{sys.path} and then use the @ref{611,,-m} switch to execute code from the
archive.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 338

@item PEP 338@footnote{https://www.python.org/dev/peps/pep-0338} - Executing modules as scripts

PEP written and  implemented by Nick Coghlan.
@end table


@c ======================================================================

@node PEP 341 Unified try/except/finally,PEP 342 New Generator Features,PEP 338 Executing Modules as Scripts,What’s New in Python 2 5
@anchor{whatsnew/2 5 pep-341}@anchor{f63}@anchor{whatsnew/2 5 pep-341-unified-try-except-finally}@anchor{f6c}
@subsection PEP 341: Unified try/except/finally


Until Python 2.5, the @ref{f6d,,try} statement came in two flavours. You could
use a @ref{4c1,,finally} block to ensure that code is always executed, or one or
more @ref{16e,,except} blocks to catch  specific exceptions.  You couldn’t
combine both @code{except} blocks and a @code{finally} block, because
generating the right bytecode for the combined version was complicated and it
wasn’t clear what the semantics of the combined statement should be.

Guido van Rossum spent some time working with Java, which does support the
equivalent of combining @ref{16e,,except} blocks and a @ref{4c1,,finally} block,
and this clarified what the statement should mean.  In Python 2.5, you can now
write:

@example
try:
    block-1 ...
except Exception1:
    handler-1 ...
except Exception2:
    handler-2 ...
else:
    else-block
finally:
    final-block
@end example

The code in @emph{block-1} is executed.  If the code raises an exception, the various
@ref{16e,,except} blocks are tested: if the exception is of class
@code{Exception1}, @emph{handler-1} is executed; otherwise if it’s of class
@code{Exception2}, @emph{handler-2} is executed, and so forth.  If no exception is
raised, the @emph{else-block} is executed.

No matter what happened previously, the @emph{final-block} is executed once the code
block is complete and any raised exceptions handled. Even if there’s an error in
an exception handler or the @emph{else-block} and a new exception is raised, the code
in the @emph{final-block} is still run.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 341

@item PEP 341@footnote{https://www.python.org/dev/peps/pep-0341} - Unifying try-except and try-finally

PEP written by Georg Brandl;  implementation by Thomas Lee.
@end table


@c ======================================================================

@node PEP 342 New Generator Features,PEP 343 The ‘with’ statement<2>,PEP 341 Unified try/except/finally,What’s New in Python 2 5
@anchor{whatsnew/2 5 pep-342}@anchor{f61}@anchor{whatsnew/2 5 pep-342-new-generator-features}@anchor{f6e}
@subsection PEP 342: New Generator Features


Python 2.5 adds a simple way to pass values @emph{into} a generator. As introduced in
Python 2.3, generators only produce output; once a generator’s code was invoked
to create an iterator, there was no way to pass any new information into the
function when its execution is resumed.  Sometimes the ability to pass in some
information would be useful.  Hackish solutions to this include making the
generator’s code look at a global variable and then changing the global
variable’s value, or passing in some mutable object that callers then modify.

To refresh your memory of basic generators, here’s a simple example:

@example
def counter (maximum):
    i = 0
    while i < maximum:
        yield i
        i += 1
@end example

When you call @code{counter(10)}, the result is an iterator that returns the values
from 0 up to 9.  On encountering the @ref{4c5,,yield} statement, the iterator
returns the provided value and suspends the function’s execution, preserving the
local variables. Execution resumes on the following call to the iterator’s
@ref{252,,next()} method, picking up after the @code{yield} statement.

In Python 2.3, @ref{4c5,,yield} was a statement; it didn’t return any value.  In
2.5, @code{yield} is now an expression, returning a value that can be
assigned to a variable or otherwise operated on:

@example
val = (yield i)
@end example

I recommend that you always put parentheses around a @ref{4c5,,yield} expression
when you’re doing something with the returned value, as in the above example.
The parentheses aren’t always necessary, but it’s easier to always add them
instead of having to remember when they’re needed.

(
@geindex Python Enhancement Proposals; PEP 342
PEP 342@footnote{https://www.python.org/dev/peps/pep-0342} explains the exact rules, which are that a
@ref{4c5,,yield}-expression must always be parenthesized except when it
occurs at the top-level
expression on the right-hand side of an assignment.  This means you can write
@code{val = yield i} but have to use parentheses when there’s an operation, as in
@code{val = (yield i) + 12}.)

Values are sent into a generator by calling its @code{send(value)} method.  The
generator’s code is then resumed and the @ref{4c5,,yield} expression returns the
specified @emph{value}.  If the regular @ref{252,,next()} method is called, the
@code{yield} returns @ref{182,,None}.

Here’s the previous example, modified to allow changing the value of the
internal counter.

@example
def counter (maximum):
    i = 0
    while i < maximum:
        val = (yield i)
        # If value provided, change counter
        if val is not None:
            i = val
        else:
            i += 1
@end example

And here’s an example of changing the counter:

@example
>>> it = counter(10)
>>> print it.next()
0
>>> print it.next()
1
>>> print it.send(8)
8
>>> print it.next()
9
>>> print it.next()
Traceback (most recent call last):
  File "t.py", line 15, in ?
    print it.next()
StopIteration
@end example

@ref{4c5,,yield} will usually return @ref{182,,None}, so you should always check
for this case.  Don’t just use its value in expressions unless you’re sure that
the @code{send()} method will be the only method used to resume your generator
function.

In addition to @code{send()}, there are two other new methods on generators:


@itemize *

@item 
@code{throw(type, value=None, traceback=None)} is used to raise an exception
inside the generator; the exception is raised by the @ref{4c5,,yield} expression
where the generator’s execution is paused.

@item 
@code{close()} raises a new @ref{f2e,,GeneratorExit} exception inside the generator
to terminate the iteration.  On receiving this exception, the generator’s code
must either raise @ref{f2e,,GeneratorExit} or @ref{736,,StopIteration}.  Catching the
@ref{f2e,,GeneratorExit} exception and returning a value is illegal and will trigger
a @ref{5af,,RuntimeError}; if the function raises some other exception, that
exception is propagated to the caller.  @code{close()} will also be called by
Python’s garbage collector when the generator is garbage-collected.

If you need to run cleanup code when a @ref{f2e,,GeneratorExit} occurs, I suggest
using a @code{try: ... finally:} suite instead of  catching @ref{f2e,,GeneratorExit}.
@end itemize

The cumulative effect of these changes is to turn generators from one-way
producers of information into both producers and consumers.

Generators also become @emph{coroutines}, a more generalized form of subroutines.
Subroutines are entered at one point and exited at another point (the top of the
function, and a @ref{4c6,,return} statement), but coroutines can be entered,
exited, and resumed at many different points (the @ref{4c5,,yield} statements).
We’ll have to figure out patterns for using coroutines effectively in Python.

The addition of the @code{close()} method has one side effect that isn’t obvious.
@code{close()} is called when a generator is garbage-collected, so this means the
generator’s code gets one last chance to run before the generator is destroyed.
This last chance means that @code{try...finally} statements in generators can now
be guaranteed to work; the @ref{4c1,,finally} clause will now always get a
chance to run.  The syntactic restriction that you couldn’t mix @ref{4c5,,yield}
statements with a @code{try...finally} suite has therefore been removed.  This
seems like a minor bit of language trivia, but using generators and
@code{try...finally} is actually necessary in order to implement the
@ref{168,,with} statement described by 
@geindex Python Enhancement Proposals; PEP 343
PEP 343@footnote{https://www.python.org/dev/peps/pep-0343}.  I’ll look at this new statement
in the following  section.

Another even more esoteric effect of this change: previously, the
@code{gi_frame} attribute of a generator was always a frame object. It’s now
possible for @code{gi_frame} to be @code{None} once the generator has been
exhausted.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 342

@item PEP 342@footnote{https://www.python.org/dev/peps/pep-0342} - Coroutines via Enhanced Generators

PEP written by  Guido van Rossum and Phillip J. Eby; implemented by Phillip J.
Eby.  Includes examples of  some fancier uses of generators as coroutines.

Earlier versions of these features were proposed in  
@geindex Python Enhancement Proposals; PEP 288
PEP 288@footnote{https://www.python.org/dev/peps/pep-0288} by Raymond
Hettinger and 
@geindex Python Enhancement Proposals; PEP 325
PEP 325@footnote{https://www.python.org/dev/peps/pep-0325} by Samuele Pedroni.

@item @indicateurl{https://en.wikipedia.org/wiki/Coroutine}

The Wikipedia entry for  coroutines.

@item @indicateurl{http://www.sidhe.org/~dan/blog/archives/000178.html}

An explanation of coroutines from a Perl point of view, written by Dan Sugalski.
@end table


@c ======================================================================

@node PEP 343 The ‘with’ statement<2>,PEP 352 Exceptions as New-Style Classes,PEP 342 New Generator Features,What’s New in Python 2 5
@anchor{whatsnew/2 5 pep-343}@anchor{f60}@anchor{whatsnew/2 5 pep-343-the-with-statement}@anchor{f6f}
@subsection PEP 343: The ‘with’ statement


The ‘@ref{168,,with}’ statement clarifies code that previously would use
@code{try...finally} blocks to ensure that clean-up code is executed.  In this
section, I’ll discuss the statement as it will commonly be used.  In the next
section, I’ll examine the implementation details and show how to write objects
for use with this statement.

The ‘@ref{168,,with}’ statement is a new control-flow structure whose basic
structure is:

@example
with expression [as variable]:
    with-block
@end example

The expression is evaluated, and it should result in an object that supports the
context management protocol (that is, has @code{__enter__()} and @code{__exit__()}
methods.

The object’s @code{__enter__()} is called before @emph{with-block} is executed and
therefore can run set-up code. It also may return a value that is bound to the
name @emph{variable}, if given.  (Note carefully that @emph{variable} is @emph{not} assigned
the result of @emph{expression}.)

After execution of the @emph{with-block} is finished, the object’s @code{__exit__()}
method is called, even if the block raised an exception, and can therefore run
clean-up code.

To enable the statement in Python 2.5, you need to add the following directive
to your module:

@example
from __future__ import with_statement
@end example

The statement will always be enabled in Python 2.6.

Some standard Python objects now support the context management protocol and can
be used with the ‘@ref{168,,with}’ statement. File objects are one example:

@example
with open('/etc/passwd', 'r') as f:
    for line in f:
        print line
        ... more processing code ...
@end example

After this statement has executed, the file object in @emph{f} will have been
automatically closed, even if the @ref{161,,for} loop raised an exception
part-way through the block.

@cartouche
@quotation Note 
In this case, @emph{f} is the same object created by @ref{1c8,,open()}, because
@code{file.__enter__()} returns @emph{self}.
@end quotation
@end cartouche

The @ref{109,,threading} module’s locks and condition variables  also support the
‘@ref{168,,with}’ statement:

@example
lock = threading.Lock()
with lock:
    # Critical section of code
    ...
@end example

The lock is acquired before the block is executed and always released once  the
block is complete.

The new @code{localcontext()} function in the @ref{34,,decimal} module makes it easy
to save and restore the current decimal context, which encapsulates the desired
precision and rounding characteristics for computations:

@example
from decimal import Decimal, Context, localcontext

# Displays with default precision of 28 digits
v = Decimal('578')
print v.sqrt()

with localcontext(Context(prec=16)):
    # All code in this block uses a precision of 16 digits.
    # The original context is restored on exiting the block.
    print v.sqrt()
@end example

@menu
* Writing Context Managers: Writing Context Managers<2>. 
* The contextlib module: The contextlib module<2>. 

@end menu

@node Writing Context Managers<2>,The contextlib module<2>,,PEP 343 The ‘with’ statement<2>
@anchor{whatsnew/2 5 new-25-context-managers}@anchor{f70}@anchor{whatsnew/2 5 writing-context-managers}@anchor{f71}
@subsubsection Writing Context Managers


Under the hood, the ‘@ref{168,,with}’ statement is fairly complicated. Most
people will only use ‘@code{with}’ in company with existing objects and
don’t need to know these details, so you can skip the rest of this section if
you like.  Authors of new objects will need to understand the details of the
underlying implementation and should keep reading.

A high-level explanation of the context management protocol is:


@itemize *

@item 
The expression is evaluated and should result in an object called a “context
manager”.  The context manager must have @code{__enter__()} and @code{__exit__()}
methods.

@item 
The context manager’s @code{__enter__()} method is called.  The value returned
is assigned to @emph{VAR}.  If no @code{'as VAR'} clause is present, the value is simply
discarded.

@item 
The code in @emph{BLOCK} is executed.

@item 
If @emph{BLOCK} raises an exception, the @code{__exit__(type, value, traceback)}
is called with the exception details, the same values returned by
@ref{18f,,sys.exc_info()}.  The method’s return value controls whether the exception
is re-raised: any false value re-raises the exception, and @code{True} will result
in suppressing it.  You’ll only rarely want to suppress the exception, because
if you do the author of the code containing the ‘@ref{168,,with}’ statement will
never realize anything went wrong.

@item 
If @emph{BLOCK} didn’t raise an exception,  the @code{__exit__()} method is still
called, but @emph{type}, @emph{value}, and @emph{traceback} are all @code{None}.
@end itemize

Let’s think through an example.  I won’t present detailed code but will only
sketch the methods necessary for a database that supports transactions.

(For people unfamiliar with database terminology: a set of changes to the
database are grouped into a transaction.  Transactions can be either committed,
meaning that all the changes are written into the database, or rolled back,
meaning that the changes are all discarded and the database is unchanged.  See
any database textbook for more information.)

Let’s assume there’s an object representing a database connection. Our goal will
be to let the user write code like this:

@example
db_connection = DatabaseConnection()
with db_connection as cursor:
    cursor.execute('insert into ...')
    cursor.execute('delete from ...')
    # ... more operations ...
@end example

The transaction should be committed if the code in the block runs flawlessly or
rolled back if there’s an exception. Here’s the basic interface for
@code{DatabaseConnection} that I’ll assume:

@example
class DatabaseConnection:
    # Database interface
    def cursor (self):
        "Returns a cursor object and starts a new transaction"
    def commit (self):
        "Commits current transaction"
    def rollback (self):
        "Rolls back current transaction"
@end example

The @code{__enter__()} method is pretty easy, having only to start a new
transaction.  For this application the resulting cursor object would be a useful
result, so the method will return it.  The user can then add @code{as cursor} to
their ‘@ref{168,,with}’ statement to bind the cursor to a variable name.

@example
class DatabaseConnection:
    ...
    def __enter__ (self):
        # Code to start a new transaction
        cursor = self.cursor()
        return cursor
@end example

The @code{__exit__()} method is the most complicated because it’s where most of
the work has to be done.  The method has to check if an exception occurred.  If
there was no exception, the transaction is committed.  The transaction is rolled
back if there was an exception.

In the code below, execution will just fall off the end of the function,
returning the default value of @code{None}.  @code{None} is false, so the exception
will be re-raised automatically.  If you wished, you could be more explicit and
add a @ref{4c6,,return} statement at the marked location.

@example
class DatabaseConnection:
    ...
    def __exit__ (self, type, value, tb):
        if tb is None:
            # No exception, so commit
            self.commit()
        else:
            # Exception occurred, so rollback.
            self.rollback()
            # return False
@end example

@node The contextlib module<2>,,Writing Context Managers<2>,PEP 343 The ‘with’ statement<2>
@anchor{whatsnew/2 5 contextlibmod}@anchor{f72}@anchor{whatsnew/2 5 the-contextlib-module}@anchor{f73}
@subsubsection The contextlib module


The new @ref{22,,contextlib} module provides some functions and a decorator that
are useful for writing objects for use with the ‘@ref{168,,with}’ statement.

The decorator is called @code{contextmanager()}, and lets you write a single
generator function instead of defining a new class.  The generator should yield
exactly one value.  The code up to the @ref{4c5,,yield} will be executed as the
@code{__enter__()} method, and the value yielded will be the method’s return
value that will get bound to the variable in the ‘@ref{168,,with}’ statement’s
@code{as} clause, if any.  The code after the @ref{4c5,,yield} will be
executed in the @code{__exit__()} method.  Any exception raised in the block will
be raised by the @code{yield} statement.

Our database example from the previous section could be written  using this
decorator as:

@example
from contextlib import contextmanager

@@contextmanager
def db_transaction (connection):
    cursor = connection.cursor()
    try:
        yield cursor
    except:
        connection.rollback()
        raise
    else:
        connection.commit()

db = DatabaseConnection()
with db_transaction(db) as cursor:
    ...
@end example

The @ref{22,,contextlib} module also has a @code{nested(mgr1, mgr2, ...)} function
that combines a number of context managers so you don’t need to write nested
‘@ref{168,,with}’ statements.  In this example, the single ‘@code{with}’
statement both starts a database transaction and acquires a thread lock:

@example
lock = threading.Lock()
with nested (db_transaction(db), lock) as (cursor, locked):
    ...
@end example

Finally, the @code{closing(object)} function returns @emph{object} so that it can be
bound to a variable, and calls @code{object.close} at the end of the block.

@example
import urllib, sys
from contextlib import closing

with closing(urllib.urlopen('http://www.yahoo.com')) as f:
    for line in f:
        sys.stdout.write(line)
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 343

@item PEP 343@footnote{https://www.python.org/dev/peps/pep-0343} - The “with” statement

PEP written by Guido van Rossum and Nick Coghlan; implemented by Mike Bland,
Guido van Rossum, and Neal Norwitz.  The PEP shows the code generated for a
‘@ref{168,,with}’ statement, which can be helpful in learning how the statement
works.
@end table

The documentation  for the @ref{22,,contextlib} module.


@c ======================================================================

@node PEP 352 Exceptions as New-Style Classes,PEP 353 Using ssize_t as the index type,PEP 343 The ‘with’ statement<2>,What’s New in Python 2 5
@anchor{whatsnew/2 5 pep-352}@anchor{f74}@anchor{whatsnew/2 5 pep-352-exceptions-as-new-style-classes}@anchor{f75}
@subsection PEP 352: Exceptions as New-Style Classes


Exception classes can now be new-style classes, not just classic classes, and
the built-in @ref{4d8,,Exception} class and all the standard built-in exceptions
(@ref{21f,,NameError}, @ref{17c,,ValueError}, etc.) are now new-style classes.

The inheritance hierarchy for exceptions has been rearranged a bit. In 2.5, the
inheritance relationships are:

@example
BaseException       # New in Python 2.5
|- KeyboardInterrupt
|- SystemExit
|- Exception
   |- (all other current built-in exceptions)
@end example

This rearrangement was done because people often want to catch all exceptions
that indicate program errors.  @ref{4ca,,KeyboardInterrupt} and @ref{892,,SystemExit}
aren’t errors, though, and usually represent an explicit action such as the user
hitting @code{Control-C} or code calling @ref{ee4,,sys.exit()}.  A bare @code{except:} will
catch all exceptions, so you commonly need to list @ref{4ca,,KeyboardInterrupt} and
@ref{892,,SystemExit} in order to re-raise them.  The usual pattern is:

@example
try:
    ...
except (KeyboardInterrupt, SystemExit):
    raise
except:
    # Log error...
    # Continue running program...
@end example

In Python 2.5, you can now write @code{except Exception} to achieve the same
result, catching all the exceptions that usually indicate errors  but leaving
@ref{4ca,,KeyboardInterrupt} and @ref{892,,SystemExit} alone.  As in previous versions,
a bare @code{except:} still catches all exceptions.

The goal for Python 3.0 is to require any class raised as an exception to derive
from @ref{15f,,BaseException} or some descendant of @ref{15f,,BaseException}, and future
releases in the Python 2.x series may begin to enforce this constraint.
Therefore, I suggest you begin making all your exception classes derive from
@ref{4d8,,Exception} now.  It’s been suggested that the bare @code{except:} form should
be removed in Python 3.0, but Guido van Rossum hasn’t decided whether to do this
or not.

Raising of strings as exceptions, as in the statement @code{raise "Error
occurred"}, is deprecated in Python 2.5 and will trigger a warning.  The aim is
to be able to remove the string-exception feature in a few releases.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 352

@item PEP 352@footnote{https://www.python.org/dev/peps/pep-0352} - Required Superclass for Exceptions

PEP written by  Brett Cannon and Guido van Rossum; implemented by Brett Cannon.
@end table


@c ======================================================================

@node PEP 353 Using ssize_t as the index type,PEP 357 The ‘__index__’ method,PEP 352 Exceptions as New-Style Classes,What’s New in Python 2 5
@anchor{whatsnew/2 5 pep-353}@anchor{f76}@anchor{whatsnew/2 5 pep-353-using-ssize-t-as-the-index-type}@anchor{f77}
@subsection PEP 353: Using ssize_t as the index type


A wide-ranging change to Python’s C API, using a new  @code{Py_ssize_t} type
definition instead of int,  will permit the interpreter to handle more
data on 64-bit platforms. This change doesn’t affect Python’s capacity on 32-bit
platforms.

Various pieces of the Python interpreter used C’s int type to store
sizes or counts; for example, the number of items in a list or tuple were stored
in an int.  The C compilers for most 64-bit platforms still define
int as a 32-bit type, so that meant that lists could only hold up to
@code{2**31 - 1} = 2147483647 items. (There are actually a few different
programming models that 64-bit C compilers can use – see
@indicateurl{http://www.unix.org/version2/whatsnew/lp64_wp.html} for a discussion – but the
most commonly available model leaves int as 32 bits.)

A limit of 2147483647 items doesn’t really matter on a 32-bit platform because
you’ll run out of memory before hitting the length limit. Each list item
requires space for a pointer, which is 4 bytes, plus space for a
@ref{375,,PyObject} representing the item.  2147483647*4 is already more bytes
than a 32-bit address space can contain.

It’s possible to address that much memory on a 64-bit platform, however.  The
pointers for a list that size would only require 16 GiB of space, so it’s not
unreasonable that Python programmers might construct lists that large.
Therefore, the Python interpreter had to be changed to use some type other than
int, and this will be a 64-bit type on 64-bit platforms.  The change
will cause incompatibilities on 64-bit machines, so it was deemed worth making
the transition now, while the number of 64-bit users is still relatively small.
(In 5 or 10 years, we may @emph{all} be on 64-bit machines, and the transition would
be more painful then.)

This change most strongly affects authors of C extension modules.   Python
strings and container types such as lists and tuples  now use
@code{Py_ssize_t} to store their size.   Functions such as
@ref{f78,,PyList_Size()}  now return @code{Py_ssize_t}.  Code in extension modules
may therefore need to have some variables changed to @code{Py_ssize_t}.

The @ref{37f,,PyArg_ParseTuple()} and @ref{380,,Py_BuildValue()} functions have a new
conversion code, @code{n}, for @code{Py_ssize_t}.   @ref{37f,,PyArg_ParseTuple()}’s
@code{s#} and @code{t#} still output int by default, but you can define the
macro  @code{PY_SSIZE_T_CLEAN} before including @code{Python.h}  to make
them return @code{Py_ssize_t}.

@geindex Python Enhancement Proposals; PEP 353
PEP 353@footnote{https://www.python.org/dev/peps/pep-0353} has a section on conversion guidelines that  extension authors should
read to learn about supporting 64-bit platforms.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 353

@item PEP 353@footnote{https://www.python.org/dev/peps/pep-0353} - Using ssize_t as the index type

PEP written and implemented by Martin von Löwis.
@end table


@c ======================================================================

@node PEP 357 The ‘__index__’ method,Other Language Changes<14>,PEP 353 Using ssize_t as the index type,What’s New in Python 2 5
@anchor{whatsnew/2 5 pep-357}@anchor{f79}@anchor{whatsnew/2 5 pep-357-the-index-method}@anchor{f7a}
@subsection PEP 357: The ‘__index__’ method


The NumPy developers had a problem that could only be solved by adding a new
special method, @code{__index__()}.  When using slice notation, as in
@code{[start:stop:step]}, the values of the @emph{start}, @emph{stop}, and @emph{step} indexes
must all be either integers or long integers.  NumPy defines a variety of
specialized integer types corresponding to unsigned and signed integers of 8,
16, 32, and 64 bits, but there was no way to signal that these types could be
used as slice indexes.

Slicing can’t just use the existing @code{__int__()} method because that method
is also used to implement coercion to integers.  If slicing used
@code{__int__()}, floating-point numbers would also become legal slice indexes
and that’s clearly an undesirable behaviour.

Instead, a new special method called @code{__index__()} was added.  It takes no
arguments and returns an integer giving the slice index to use.  For example:

@example
class C:
    def __index__ (self):
        return self.value
@end example

The return value must be either a Python integer or long integer. The
interpreter will check that the type returned is correct, and raises a
@ref{162,,TypeError} if this requirement isn’t met.

A corresponding @code{nb_index} slot was added to the C-level
@ref{f7b,,PyNumberMethods} structure to let C extensions implement this protocol.
@code{PyNumber_Index(obj)} can be used in extension code to call the
@code{__index__()} function and retrieve its result.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 357

@item PEP 357@footnote{https://www.python.org/dev/peps/pep-0357} - Allowing Any Object to be Used for Slicing

PEP written  and implemented by Travis Oliphant.
@end table


@c ======================================================================

@node Other Language Changes<14>,New Improved and Removed Modules,PEP 357 The ‘__index__’ method,What’s New in Python 2 5
@anchor{whatsnew/2 5 other-lang}@anchor{f7c}@anchor{whatsnew/2 5 other-language-changes}@anchor{f7d}
@subsection Other Language Changes


Here are all of the changes that Python 2.5 makes to the core Python language.


@itemize *

@item 
The @ref{3b0,,dict} type has a new hook for letting subclasses provide a default
value when a key isn’t contained in the dictionary. When a key isn’t found, the
dictionary’s @code{__missing__(key)} method will be called.  This hook is used
to implement the new @code{defaultdict} class in the @ref{1c,,collections}
module.  The following example defines a dictionary  that returns zero for any
missing key:

@example
class zerodict (dict):
    def __missing__ (self, key):
        return 0

d = zerodict(@{1:1, 2:2@})
print d[1], d[2]   # Prints 1, 2
print d[3], d[4]   # Prints 0, 0
@end example

@item 
Both 8-bit and Unicode strings have new @code{partition(sep)}  and
@code{rpartition(sep)} methods that simplify a common use case.

The @code{find(S)} method is often used to get an index which is then used to
slice the string and obtain the pieces that are before and after the separator.
@code{partition(sep)} condenses this pattern into a single method call that
returns a 3-tuple containing the substring before the separator, the separator
itself, and the substring after the separator.  If the separator isn’t found,
the first element of the tuple is the entire string and the other two elements
are empty.  @code{rpartition(sep)} also returns a 3-tuple but starts searching
from the end of the string; the @code{r} stands for ‘reverse’.

Some examples:

@example
>>> ('http://www.python.org').partition('://')
('http', '://', 'www.python.org')
>>> ('file:/usr/share/doc/index.html').partition('://')
('file:/usr/share/doc/index.html', '@w{'}, '@w{'})
>>> (u'Subject: a quick question').partition(':')
(u'Subject', u':', u' a quick question')
>>> 'www.python.org'.rpartition('.')
('www.python', '.', 'org')
>>> 'www.python.org'.rpartition(':')
('@w{'}, '@w{'}, 'www.python.org')
@end example

(Implemented by Fredrik Lundh following a suggestion by Raymond Hettinger.)

@item 
The @code{startswith()} and @code{endswith()} methods of string types now accept
tuples of strings to check for.

@example
def is_image_file (filename):
    return filename.endswith(('.gif', '.jpg', '.tiff'))
@end example

(Implemented by Georg Brandl following a suggestion by Tom Lynn.)

@c RFE #1491485

@item 
The @ref{a7d,,min()} and @ref{a7e,,max()} built-in functions gained a @code{key} keyword
parameter analogous to the @code{key} argument for @code{sort()}.  This parameter
supplies a function that takes a single argument and is called for every value
in the list; @ref{a7d,,min()}/@ref{a7e,,max()} will return the element with the
smallest/largest return value from this function. For example, to find the
longest string in a list, you can do:

@example
L = ['medium', 'longest', 'short']
# Prints 'longest'
print max(L, key=len)
# Prints 'short', because lexicographically 'short' has the largest value
print max(L)
@end example

(Contributed by Steven Bethard and Raymond Hettinger.)

@item 
Two new built-in functions, @ref{f7e,,any()} and @ref{f7f,,all()}, evaluate whether an
iterator contains any true or false values.  @ref{f7e,,any()} returns @ref{748,,True}
if any value returned by the iterator is true; otherwise it will return
@ref{659,,False}.  @ref{f7f,,all()} returns @ref{748,,True} only if all of the values
returned by the iterator evaluate as true. (Suggested by Guido van Rossum, and
implemented by Raymond Hettinger.)

@item 
The result of a class’s @code{__hash__()} method can now be either a long
integer or a regular integer.  If a long integer is returned, the hash of that
value is taken.  In earlier versions the hash value was required to be a
regular integer, but in 2.5 the @ref{f80,,id()} built-in was changed to always
return non-negative numbers, and users often seem to use @code{id(self)} in
@code{__hash__()} methods (though this is discouraged).

@c Bug #1536021

@item 
ASCII is now the default encoding for modules.  It’s now  a syntax error if a
module contains string literals with 8-bit characters but doesn’t have an
encoding declaration.  In Python 2.4 this triggered a warning, not a syntax
error.  See 
@geindex Python Enhancement Proposals; PEP 263
PEP 263@footnote{https://www.python.org/dev/peps/pep-0263}  for how to declare a module’s encoding; for example, you
might add  a line like this near the top of the source file:

@example
# -*- coding: latin1 -*-
@end example

@item 
A new warning, @ref{f81,,UnicodeWarning}, is triggered when  you attempt to
compare a Unicode string and an 8-bit string  that can’t be converted to Unicode
using the default ASCII encoding.   The result of the comparison is false:

@example
>>> chr(128) == unichr(128)   # Can't convert chr(128) to Unicode
__main__:1: UnicodeWarning: Unicode equal comparison failed
  to convert both arguments to Unicode - interpreting them
  as being unequal
False
>>> chr(127) == unichr(127)   # chr(127) can be converted
True
@end example

Previously this would raise a @ref{51f,,UnicodeDecodeError} exception, but in 2.5
this could result in puzzling problems when accessing a dictionary.  If you
looked up @code{unichr(128)} and @code{chr(128)} was being used as a key, you’d get a
@ref{51f,,UnicodeDecodeError} exception.  Other changes in 2.5 resulted in this
exception being raised instead of suppressed by the code in @code{dictobject.c}
that implements dictionaries.

Raising an exception for such a comparison is strictly correct, but the change
might have broken code, so instead  @ref{f81,,UnicodeWarning} was introduced.

(Implemented by Marc-André Lemburg.)

@item 
One error that Python programmers sometimes make is forgetting to include an
@code{__init__.py} module in a package directory. Debugging this mistake can be
confusing, and usually requires running Python with the @ref{f82,,-v} switch to
log all the paths searched. In Python 2.5, a new @ref{304,,ImportWarning} warning is
triggered when an import would have picked up a directory as a package but no
@code{__init__.py} was found.  This warning is silently ignored by default;
provide the @ref{3a8,,-Wd} option when running the Python executable to display
the warning message. (Implemented by Thomas Wouters.)

@item 
The list of base classes in a class definition can now be empty.   As an
example, this is now legal:

@example
class C():
    pass
@end example

(Implemented by Brett Cannon.)
@end itemize

@c ======================================================================

@menu
* Interactive Interpreter Changes:: 
* Optimizations: Optimizations<13>. 

@end menu

@node Interactive Interpreter Changes,Optimizations<13>,,Other Language Changes<14>
@anchor{whatsnew/2 5 interactive}@anchor{f83}@anchor{whatsnew/2 5 interactive-interpreter-changes}@anchor{f84}
@subsubsection Interactive Interpreter Changes


In the interactive interpreter, @code{quit} and @code{exit}  have long been strings so
that new users get a somewhat helpful message when they try to quit:

@example
>>> quit
'Use Ctrl-D (i.e. EOF) to exit.'
@end example

In Python 2.5, @code{quit} and @code{exit} are now objects that still produce string
representations of themselves, but are also callable. Newbies who try @code{quit()}
or @code{exit()} will now exit the interpreter as they expect.  (Implemented by
Georg Brandl.)

The Python executable now accepts the standard long options  @ref{f85,,--help}
and @ref{86a,,--version}; on Windows,  it also accepts the @ref{f86,,/?} option
for displaying a help message. (Implemented by Georg Brandl.)

@c ======================================================================

@node Optimizations<13>,,Interactive Interpreter Changes,Other Language Changes<14>
@anchor{whatsnew/2 5 optimizations}@anchor{f87}@anchor{whatsnew/2 5 opts}@anchor{f88}
@subsubsection Optimizations


Several of the optimizations were developed at the NeedForSpeed sprint, an event
held in Reykjavik, Iceland, from May 21–28 2006. The sprint focused on speed
enhancements to the CPython implementation and was funded by EWT LLC with local
support from CCP Games.  Those optimizations added at this sprint are specially
marked in the following list.


@itemize *

@item 
When they were introduced  in Python 2.4, the built-in @ref{370,,set} and
@ref{439,,frozenset} types were built on top of Python’s dictionary type.   In 2.5
the internal data structure has been customized for implementing sets, and as a
result sets will use a third less memory and are somewhat faster. (Implemented
by Raymond Hettinger.)

@item 
The speed of some Unicode operations, such as finding substrings, string
splitting, and character map encoding and decoding, has been improved.
(Substring search and splitting improvements were added by Fredrik Lundh and
Andrew Dalke at the NeedForSpeed sprint. Character maps were improved by Walter
Dörwald and Martin von Löwis.)

@c Patch 1313939, 1359618

@item 
The @code{long(str, base)} function is now faster on long digit strings
because fewer intermediate results are calculated.  The peak is for strings of
around 800–1000 digits where  the function is 6 times faster. (Contributed by
Alan McIntyre and committed at the NeedForSpeed sprint.)

@c Patch 1442927

@item 
It’s now illegal to mix iterating over a file  with @code{for line in file} and
calling  the file object’s @code{read()}/@ref{d9,,readline()}/@code{readlines()}
methods.  Iteration uses an internal buffer and the  @code{read*()} methods
don’t use that buffer.   Instead they would return the data following the
buffer, causing the data to appear out of order.  Mixing iteration and these
methods will now trigger a @ref{17c,,ValueError} from the @code{read*()} method.
(Implemented by Thomas Wouters.)

@c Patch 1397960

@item 
The @ref{f3,,struct} module now compiles structure format  strings into an
internal representation and caches this representation, yielding a 20% speedup.
(Contributed by Bob Ippolito at the NeedForSpeed sprint.)

@item 
The @ref{d8,,re} module got a 1 or 2% speedup by switching to  Python’s allocator
functions instead of the system’s  @code{malloc()} and @code{free()}.
(Contributed by Jack Diederich at the NeedForSpeed sprint.)

@item 
The code generator’s peephole optimizer now performs simple constant folding
in expressions.  If you write something like @code{a = 2+3}, the code generator
will do the arithmetic and produce code corresponding to @code{a = 5}.  (Proposed
and implemented  by Raymond Hettinger.)

@item 
Function calls are now faster because code objects now keep  the most recently
finished frame (a “zombie frame”) in an internal field of the code object,
reusing it the next time the code object is invoked.  (Original patch by Michael
Hudson, modified by Armin Rigo and Richard Jones; committed at the NeedForSpeed
sprint.)  Frame objects are also slightly smaller, which may improve cache
locality and reduce memory usage a bit.  (Contributed by Neal Norwitz.)

@c Patch 876206

@c Patch 1337051

@item 
Python’s built-in exceptions are now new-style classes, a change that speeds
up instantiation considerably.  Exception handling in Python 2.5 is therefore
about 30% faster than in 2.4. (Contributed by Richard Jones, Georg Brandl and
Sean Reifschneider at the NeedForSpeed sprint.)

@item 
Importing now caches the paths tried, recording whether  they exist or not so
that the interpreter makes fewer  @code{open()} and @code{stat()} calls on
startup. (Contributed by Martin von Löwis and Georg Brandl.)

@c Patch 921466
@end itemize

@c ======================================================================

@node New Improved and Removed Modules,Build and C API Changes<10>,Other Language Changes<14>,What’s New in Python 2 5
@anchor{whatsnew/2 5 modules}@anchor{f89}@anchor{whatsnew/2 5 new-improved-and-removed-modules}@anchor{f8a}
@subsection New, Improved, and Removed Modules


The standard library received many enhancements and bug fixes in Python 2.5.
Here’s a partial list of the most notable changes, sorted alphabetically by
module name. Consult the @code{Misc/NEWS} file in the source tree for a more
complete list of changes, or look through the SVN logs for all the details.


@itemize *

@item 
The @ref{c,,audioop} module now supports the a-LAW encoding, and the code for
u-LAW encoding has been improved.  (Contributed by Lars Immisch.)

@item 
The @ref{1a,,codecs} module gained support for incremental codecs.  The
@code{codec.lookup()} function now returns a @code{CodecInfo} instance instead
of a tuple. @code{CodecInfo} instances behave like a 4-tuple to preserve
backward compatibility but also have the attributes @code{encode},
@code{decode}, @code{incrementalencoder}, @code{incrementaldecoder},
@code{streamwriter}, and @code{streamreader}.  Incremental codecs  can receive
input and produce output in multiple chunks; the output is the same as if the
entire input was fed to the non-incremental codec. See the @ref{1a,,codecs} module
documentation for details. (Designed and implemented by Walter Dörwald.)

@c Patch  1436130

@item 
The @ref{1c,,collections} module gained a new type, @code{defaultdict}, that
subclasses the standard @ref{3b0,,dict} type.  The new type mostly behaves like a
dictionary but constructs a default value when a key isn’t present,
automatically adding it to the dictionary for the requested key value.

The first argument to @code{defaultdict}’s constructor is a factory function
that gets called whenever a key is requested but not found. This factory
function receives no arguments, so you can use built-in type constructors such
as @ref{43a,,list()} or @ref{1b1,,int()}.  For example,  you can make an index of words
based on their initial letter like this:

@example
words = """Nel mezzo del cammin di nostra vita
mi ritrovai per una selva oscura
che la diritta via era smarrita""".lower().split()

index = defaultdict(list)

for w in words:
    init_letter = w[0]
    index[init_letter].append(w)
@end example

Printing @code{index} results in the following output:

@example
defaultdict(<type 'list'>, @{'c': ['cammin', 'che'], 'e': ['era'],
        'd': ['del', 'di', 'diritta'], 'm': ['mezzo', 'mi'],
        'l': ['la'], 'o': ['oscura'], 'n': ['nel', 'nostra'],
        'p': ['per'], 's': ['selva', 'smarrita'],
        'r': ['ritrovai'], 'u': ['una'], 'v': ['vita', 'via']@}
@end example

(Contributed by Guido van Rossum.)

@item 
The @code{deque} double-ended queue type supplied by the @ref{1c,,collections}
module now has a @code{remove(value)} method that removes the first occurrence
of @emph{value} in the queue, raising @ref{17c,,ValueError} if the value isn’t found.
(Contributed by Raymond Hettinger.)

@item 
New module: The @ref{22,,contextlib} module contains helper functions for use
with the new ‘@ref{168,,with}’ statement.  See section @ref{f72,,The contextlib module}
for more about this module.

@item 
New module: The @ref{26,,cProfile} module is a C implementation of  the existing
@ref{ce,,profile} module that has much lower overhead. The module’s interface is
the same as @ref{ce,,profile}: you run @code{cProfile.run('main()')} to profile a
function, can save profile data to a file, etc.  It’s not yet known if the
Hotshot profiler, which is also written in C but doesn’t match the
@ref{ce,,profile} module’s interface, will continue to be maintained in future
versions of Python.  (Contributed by Armin Rigo.)

Also, the @ref{cf,,pstats} module for analyzing the data measured by the profiler
now supports directing the output to any file object by supplying a @emph{stream}
argument to the @code{Stats} constructor. (Contributed by Skip Montanaro.)

@item 
The @ref{28,,csv} module, which parses files in comma-separated value format,
received several enhancements and a number of bugfixes.  You can now set the
maximum size in bytes of a field by calling the
@code{csv.field_size_limit(new_limit)} function; omitting the @emph{new_limit}
argument will return the currently-set limit.  The @code{reader} class now has
a @code{line_num} attribute that counts the number of physical lines read from
the source; records can span multiple physical lines, so @code{line_num} is not
the same as the number of records read.

The CSV parser is now stricter about multi-line quoted fields. Previously, if a
line ended within a quoted field without a terminating newline character, a
newline would be inserted into the returned field. This behavior caused problems
when reading files that contained carriage return characters within fields, so
the code was changed to return the field without inserting newlines. As a
consequence, if newlines embedded within fields are important, the input should
be split into lines in a manner that preserves the newline characters.

(Contributed by Skip Montanaro and Andrew McNamara.)

@item 
The @ref{363,,datetime} class in the @ref{2f,,datetime}  module now has a
@code{strptime(string, format)}  method for parsing date strings, contributed
by Josh Spoerri. It uses the same format characters as @ref{f8b,,time.strptime()} and
@ref{d8b,,time.strftime()}:

@example
from datetime import datetime

ts = datetime.strptime('10:13:15 2006-03-07',
                       '%H:%M:%S %Y-%m-%d')
@end example

@item 
The @code{SequenceMatcher.get_matching_blocks()} method in the @ref{35,,difflib}
module now guarantees to return a minimal list of blocks describing matching
subsequences.  Previously, the algorithm would occasionally break a block of
matching elements into two list entries. (Enhancement by Tim Peters.)

@item 
The @ref{63,,doctest} module gained a @code{SKIP} option that keeps an example from
being executed at all.  This is intended for code snippets that are usage
examples intended for the reader and aren’t actually test cases.

An @emph{encoding} parameter was added to the @code{testfile()} function and the
@code{DocFileSuite} class to specify the file’s encoding.  This makes it
easier to use non-ASCII characters in  tests contained within a docstring.
(Contributed by Bjorn Tillenius.)

@c Patch 1080727

@item 
The @ref{64,,email} package has been updated to version 4.0. (Contributed by
Barry Warsaw.)

@c XXX need to provide some more detail here

@geindex universal newlines; What's new

@item 
The @ref{7b,,fileinput} module was made more flexible. Unicode filenames are now
supported, and a @emph{mode} parameter that defaults to @code{"r"} was added to the
@ref{e73,,input()} function to allow opening files in binary or @ref{88a,,universal newlines} mode.  Another new parameter, @emph{openhook}, lets you use a function
other than @ref{1c8,,open()}  to open the input files.  Once you’re iterating over
the set of files, the @code{FileInput} object’s new @code{fileno()} returns
the file descriptor for the currently opened file. (Contributed by Georg
Brandl.)

@item 
In the @ref{80,,gc} module, the new @code{get_count()} function returns a 3-tuple
containing the current collection counts for the three GC generations.  This is
accounting information for the garbage collector; when these counts reach a
specified threshold, a garbage collection sweep will be made.  The existing
@ref{54b,,gc.collect()} function now takes an optional @emph{generation} argument of 0, 1,
or 2 to specify which generation to collect. (Contributed by Barry Warsaw.)

@item 
The @code{nsmallest()} and  @code{nlargest()} functions in the @ref{89,,heapq}
module  now support a @code{key} keyword parameter similar to the one provided by
the @ref{a7d,,min()}/@ref{a7e,,max()} functions and the @code{sort()} methods.  For
example:

@example
>>> import heapq
>>> L = ["short", 'medium', 'longest', 'longer still']
>>> heapq.nsmallest(2, L)  # Return two lowest elements, lexicographically
['longer still', 'longest']
>>> heapq.nsmallest(2, L, key=len)   # Return two shortest elements
['short', 'medium']
@end example

(Contributed by Raymond Hettinger.)

@item 
The @ref{672,,itertools.islice()} function now accepts @code{None} for the start and
step arguments.  This makes it more compatible with the attributes of slice
objects, so that you can now write the following:

@example
s = slice(5)     # Create slice object
itertools.islice(iterable, s.start, s.stop, s.step)
@end example

(Contributed by Raymond Hettinger.)

@item 
The @ref{785,,format()} function in the @ref{a5,,locale} module has been modified and
two new functions were added, @code{format_string()} and @code{currency()}.

The @ref{785,,format()} function’s @emph{val} parameter could previously be a string as
long as no more than one %char specifier appeared; now the parameter must be
exactly one %char specifier with no surrounding text.  An optional @emph{monetary}
parameter was also added which, if @code{True}, will use the locale’s rules for
formatting currency in placing a separator between groups of three digits.

To format strings with multiple %char specifiers, use the new
@code{format_string()} function that works like @ref{785,,format()} but also supports
mixing %char specifiers with arbitrary text.

A new @code{currency()} function was also added that formats a number according
to the current locale’s settings.

(Contributed by Georg Brandl.)

@c Patch 1180296

@item 
The @ref{aa,,mailbox} module underwent a massive rewrite to add the capability to
modify mailboxes in addition to reading them.  A new set of classes that include
@code{mbox}, @code{MH}, and @code{Maildir} are used to read mailboxes, and
have an @code{add(message)} method to add messages, @code{remove(key)} to
remove messages, and @code{lock()}/@code{unlock()} to lock/unlock the mailbox.
The following example converts a maildir-format mailbox into an mbox-format
one:

@example
import mailbox

# 'factory=None' uses email.Message.Message as the class representing
# individual messages.
src = mailbox.Maildir('maildir', factory=None)
dest = mailbox.mbox('/tmp/mbox')

for msg in src:
    dest.add(msg)
@end example

(Contributed by Gregory K. Johnson.  Funding was provided by Google’s 2005
Summer of Code.)

@item 
New module: the @ref{b1,,msilib} module allows creating Microsoft Installer
@code{.msi} files and CAB files.  Some support for reading the @code{.msi}
database is also included. (Contributed by Martin von Löwis.)

@item 
The @ref{bb,,nis} module now supports accessing domains other than the system
default domain by supplying a @emph{domain} argument to the @ref{f8c,,nis.match()} and
@ref{f8d,,nis.maps()} functions. (Contributed by Ben Bell.)

@item 
The @ref{be,,operator} module’s @code{itemgetter()}  and @code{attrgetter()}
functions now support multiple fields.   A call such as
@code{operator.attrgetter('a', 'b')} will return a function  that retrieves the
@code{a} and @code{b} attributes.  Combining  this new feature with the
@code{sort()} method’s @code{key} parameter  lets you easily sort lists using
multiple fields. (Contributed by Raymond Hettinger.)

@item 
The @ref{bf,,optparse} module was updated to version 1.5.1 of the Optik library.
The @code{OptionParser} class gained an @code{epilog} attribute, a string
that will be printed after the help message, and a @code{destroy()} method to
break reference cycles created by the object. (Contributed by Greg Ward.)

@item 
The @ref{c0,,os} module underwent several changes.  The @code{stat_float_times}
variable now defaults to true, meaning that @ref{516,,os.stat()} will now return time
values as floats.  (This doesn’t necessarily mean that @ref{516,,os.stat()} will
return times that are precise to fractions of a second; not all systems support
such precision.)

Constants named @ref{f8e,,os.SEEK_SET}, @ref{f8f,,os.SEEK_CUR}, and
@ref{f90,,os.SEEK_END} have been added; these are the parameters to the
@ref{caa,,os.lseek()} function.  Two new constants for locking are
@ref{f91,,os.O_SHLOCK} and @ref{f92,,os.O_EXLOCK}.

Two new functions, @code{wait3()} and @code{wait4()}, were added.  They’re similar
the @code{waitpid()} function which waits for a child process to exit and returns
a tuple of the process ID and its exit status, but @code{wait3()} and
@code{wait4()} return additional information.  @code{wait3()} doesn’t take a
process ID as input, so it waits for any child process to exit and returns a
3-tuple of @emph{process-id}, @emph{exit-status}, @emph{resource-usage} as returned from the
@ref{f93,,resource.getrusage()} function. @code{wait4(pid)} does take a process ID.
(Contributed by Chad J. Schroeder.)

On FreeBSD, the @ref{516,,os.stat()} function now returns  times with nanosecond
resolution, and the returned object now has @code{st_gen} and
@code{st_birthtime}. The @code{st_flags} attribute is also available, if the
platform supports it. (Contributed by Antti Louko and  Diego Pettenò.)

@c (Patch 1180695, 1212117)

@item 
The Python debugger provided by the @ref{c4,,pdb} module can now store lists of
commands to execute when a breakpoint is reached and execution stops.  Once
breakpoint #1 has been created, enter @code{commands 1} and enter a series of
commands to be executed, finishing the list with @code{end}.  The command list can
include commands that resume execution, such as @code{continue} or @code{next}.
(Contributed by Grégoire Dooms.)

@c Patch 790710

@item 
The @ref{c5,,pickle} and @code{cPickle} modules no longer accept a return value
of @code{None} from the @code{__reduce__()} method; the method must return a tuple
of arguments instead.  The ability to return @code{None} was deprecated in Python
2.4, so this completes the removal of the feature.

@item 
The @ref{c8,,pkgutil} module, containing various utility functions for finding
packages, was enhanced to support 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302}’s import hooks and now also works for
packages stored in ZIP-format archives. (Contributed by Phillip J. Eby.)

@item 
The pybench benchmark suite by Marc-André Lemburg is now included in the
@code{Tools/pybench} directory.  The pybench suite is an improvement on the
commonly used @code{pystone.py} program because pybench provides a more
detailed measurement of the interpreter’s speed.  It times particular operations
such as function calls, tuple slicing, method lookups, and numeric operations,
instead of performing many different operations and reducing the result to a
single number as @code{pystone.py} does.

@item 
The @code{pyexpat} module now uses version 2.0 of the Expat parser.
(Contributed by Trent Mick.)

@item 
The @ref{f18,,Queue} class provided by the @code{Queue} module gained two new
methods.  @code{join()} blocks until all items in the queue have been retrieved
and all processing work on the items  have been completed.  Worker threads call
the other new method,  @code{task_done()}, to signal that processing for an item
has been completed.  (Contributed by Raymond Hettinger.)

@item 
The old @code{regex} and @code{regsub} modules, which have been  deprecated
ever since Python 2.0, have finally been deleted.   Other deleted modules:
@code{statcache}, @code{tzparse}, @code{whrandom}.

@item 
Also deleted: the @code{lib-old} directory, which includes ancient modules
such as @code{dircmp} and @code{ni}, was removed.  @code{lib-old} wasn’t on the
default @code{sys.path}, so unless your programs explicitly added the directory to
@code{sys.path}, this removal shouldn’t affect your code.

@item 
The @ref{dc,,rlcompleter} module is no longer  dependent on importing the
@ref{d9,,readline} module and therefore now works on non-Unix platforms. (Patch
from Robert Kiendl.)

@c Patch #1472854

@item 
The @code{SimpleXMLRPCServer} and @code{DocXMLRPCServer}  classes now have a
@code{rpc_paths} attribute that constrains XML-RPC operations to a limited set
of URL paths; the default is to allow only @code{'/'} and @code{'/RPC2'}.  Setting
@code{rpc_paths} to @code{None} or an empty tuple disables  this path checking.

@c Bug #1473048

@item 
The @ref{ea,,socket} module now supports @code{AF_NETLINK} sockets on Linux,
thanks to a patch from Philippe Biondi.   Netlink sockets are a Linux-specific
mechanism for communications between a user-space process and kernel code; an
introductory  article about them is at @indicateurl{https://www.linuxjournal.com/article/7356}.
In Python code, netlink addresses are represented as a tuple of 2 integers,
@code{(pid, group_mask)}.

Two new methods on socket objects, @code{recv_into(buffer)} and
@code{recvfrom_into(buffer)}, store the received data in an object  that
supports the buffer protocol instead of returning the data as a string.  This
means you can put the data directly into an array or a memory-mapped file.

Socket objects also gained @code{getfamily()}, @code{gettype()}, and
@code{getproto()} accessor methods to retrieve the family, type, and protocol
values for the socket.

@item 
New module: the @ref{ec,,spwd} module provides functions for accessing the shadow
password database on systems that support  shadow passwords.

@item 
The @ref{f3,,struct} is now faster because it  compiles format strings into
@code{Struct} objects with @code{pack()} and @code{unpack()} methods.  This is
similar to how the @ref{d8,,re} module lets you create compiled regular expression
objects.  You can still use the module-level  @code{pack()} and @code{unpack()}
functions; they’ll create  @code{Struct} objects and cache them.  Or you can
use  @code{Struct} instances directly:

@example
s = struct.Struct('ih3s')

data = s.pack(1972, 187, 'abc')
year, number, name = s.unpack(data)
@end example

You can also pack and unpack data to and from buffer objects directly using the
@code{pack_into(buffer, offset, v1, v2, ...)} and @code{unpack_from(buffer,
offset)} methods.  This lets you store data directly into an array or a
memory-mapped file.

(@code{Struct} objects were implemented by Bob Ippolito at the NeedForSpeed
sprint.  Support for buffer objects was added by Martin Blais, also at the
NeedForSpeed sprint.)

@item 
The Python developers switched from CVS to Subversion during the 2.5
development process.  Information about the exact build version is available as
the @code{sys.subversion} variable, a 3-tuple of @code{(interpreter-name, branch-name,
revision-range)}.  For example, at the time of writing my copy of 2.5 was
reporting @code{('CPython', 'trunk', '45313:45315')}.

This information is also available to C extensions via the
@ref{f94,,Py_GetBuildInfo()} function that returns a  string of build information
like this: @code{"trunk:45355:45356M, Apr 13 2006, 07:42:19"}.   (Contributed by
Barry Warsaw.)

@item 
Another new function, @ref{f95,,sys._current_frames()}, returns the current stack
frames for all running threads as a dictionary mapping thread identifiers to the
topmost stack frame currently active in that thread at the time the function is
called.  (Contributed by Tim Peters.)

@item 
The @code{TarFile} class in the @ref{fb,,tarfile} module now has an
@code{extractall()} method that extracts all members from the archive into the
current working directory.  It’s also possible to set a different directory as
the extraction target, and to unpack only a subset of the archive’s members.

The compression used for a tarfile opened in stream mode can now be autodetected
using the mode @code{'r|*'}. (Contributed by Lars Gustäbel.)

@c patch 918101

@item 
The @ref{109,,threading} module now lets you set the stack size used when new
threads are created. The @code{stack_size([*size*])} function returns the
currently configured stack size, and supplying the optional @emph{size} parameter
sets a new value.  Not all platforms support changing the stack size, but
Windows, POSIX threading, and OS/2 all do. (Contributed by Andrew MacIntyre.)

@c Patch 1454481

@item 
The @ref{121,,unicodedata} module has been updated to use version 4.1.0 of the
Unicode character database.  Version 3.2.0 is required  by some specifications,
so it’s still available as  @ref{f96,,unicodedata.ucd_3_2_0}.

@item 
New module: the  @ref{12b,,uuid} module generates  universally unique identifiers
(UUIDs) according to 
@geindex RFC; RFC 4122
RFC 4122@footnote{https://tools.ietf.org/html/rfc4122.html}.  The RFC defines several different UUID
versions that are generated from a starting string, from system properties, or
purely randomly.  This module contains a @code{UUID} class and  functions
named @code{uuid1()}, @code{uuid3()}, @code{uuid4()},  and  @code{uuid5()} to
generate different versions of UUID.  (Version 2 UUIDs  are not specified in
@geindex RFC; RFC 4122
RFC 4122@footnote{https://tools.ietf.org/html/rfc4122.html} and are not supported by this module.)

@example
>>> import uuid
>>> # make a UUID based on the host ID and current time
>>> uuid.uuid1()
UUID('a8098c1a-f86e-11da-bd1a-00112444be1e')

>>> # make a UUID using an MD5 hash of a namespace UUID and a name
>>> uuid.uuid3(uuid.NAMESPACE_DNS, 'python.org')
UUID('6fa459ea-ee8a-3ca4-894e-db77e160355e')

>>> # make a random UUID
>>> uuid.uuid4()
UUID('16fd2706-8baf-433b-82eb-8c7fada847da')

>>> # make a UUID using a SHA-1 hash of a namespace UUID and a name
>>> uuid.uuid5(uuid.NAMESPACE_DNS, 'python.org')
UUID('886313e1-3b8a-5372-9b90-0c9aee199e5d')
@end example

(Contributed by Ka-Ping Yee.)

@item 
The @ref{12f,,weakref} module’s @code{WeakKeyDictionary} and
@code{WeakValueDictionary} types gained new methods for iterating over the
weak references contained in the dictionary.  @code{iterkeyrefs()} and
@code{keyrefs()} methods were added to @code{WeakKeyDictionary}, and
@code{itervaluerefs()} and @code{valuerefs()} were added to
@code{WeakValueDictionary}.  (Contributed by Fred L. Drake, Jr.)

@item 
The @ref{130,,webbrowser} module received a number of enhancements. It’s now
usable as a script with @code{python -m webbrowser}, taking a URL as the argument;
there are a number of switches  to control the behaviour (@code{-n} for a new
browser window,  @code{-t} for a new tab).  New module-level functions,
@code{open_new()} and @code{open_new_tab()}, were added  to support this.  The
module’s @ref{1c8,,open()} function supports an additional feature, an @emph{autoraise}
parameter that signals whether to raise the open window when possible. A number
of additional browsers were added to the supported list such as Firefox, Opera,
Konqueror, and elinks.  (Contributed by Oleg Broytmann and Georg Brandl.)

@c Patch #754022

@item 
The @code{xmlrpclib} module now supports returning  @ref{363,,datetime} objects
for the XML-RPC date type.  Supply  @code{use_datetime=True} to the @code{loads()}
function or the @code{Unmarshaller} class to enable this feature. (Contributed
by Skip Montanaro.)

@c Patch 1120353

@item 
The @ref{149,,zipfile} module now supports the ZIP64 version of the  format,
meaning that a .zip archive can now be larger than 4 GiB and can contain
individual files larger than 4 GiB.  (Contributed by Ronald Oussoren.)

@c Patch 1446489

@item 
The @ref{14b,,zlib} module’s @code{Compress} and @code{Decompress} objects now
support a @ref{24,,copy()} method that makes a copy of the  object’s internal state
and returns a new  @code{Compress} or @code{Decompress} object.
(Contributed by Chris AtLee.)

@c Patch 1435422
@end itemize

@c ======================================================================

@menu
* The ctypes package:: 
* The ElementTree package:: 
* The hashlib package:: 
* The sqlite3 package:: 
* The wsgiref package:: 

@end menu

@node The ctypes package,The ElementTree package,,New Improved and Removed Modules
@anchor{whatsnew/2 5 module-ctypes}@anchor{f97}@anchor{whatsnew/2 5 the-ctypes-package}@anchor{f98}
@subsubsection The ctypes package


The @ref{29,,ctypes} package, written by Thomas Heller, has been added  to the
standard library.  @ref{29,,ctypes} lets you call arbitrary functions  in shared
libraries or DLLs.  Long-time users may remember the @code{dl} module, which
provides functions for loading shared libraries and calling functions in them.
The @ref{29,,ctypes} package is much fancier.

To load a shared library or DLL, you must create an instance of the
@code{CDLL} class and provide the name or path of the shared library or DLL.
Once that’s done, you can call arbitrary functions by accessing them as
attributes of the @code{CDLL} object.

@example
import ctypes

libc = ctypes.CDLL('libc.so.6')
result = libc.printf("Line of output\n")
@end example

Type constructors for the various C types are provided: @code{c_int()},
@code{c_float()}, @code{c_double()}, @code{c_char_p()} (equivalent to char*), and so forth.  Unlike Python’s types, the C versions are all mutable; you
can assign to their @code{value} attribute to change the wrapped value.  Python
integers and strings will be automatically converted to the corresponding C
types, but for other types you  must call the correct type constructor.  (And I
mean @emph{must};  getting it wrong will often result in the interpreter crashing
with a segmentation fault.)

You shouldn’t use @code{c_char_p()} with a Python string when the C function will
be modifying the memory area, because Python strings are  supposed to be
immutable; breaking this rule will cause puzzling bugs.  When you need a
modifiable memory area, use @code{create_string_buffer()}:

@example
s = "this is a string"
buf = ctypes.create_string_buffer(s)
libc.strfry(buf)
@end example

C functions are assumed to return integers, but you can set the @code{restype}
attribute of the function object to  change this:

@example
>>> libc.atof('2.71828')
-1783957616
>>> libc.atof.restype = ctypes.c_double
>>> libc.atof('2.71828')
2.71828
@end example

@ref{29,,ctypes} also provides a wrapper for Python’s C API  as the
@code{ctypes.pythonapi} object.  This object does @emph{not}  release the global
interpreter lock before calling a function, because the lock must be held when
calling into the interpreter’s code.   There’s a @code{py_object()} type
constructor that will create a  @ref{375,,PyObject}* pointer.  A simple usage:

@example
import ctypes

d = @{@}
ctypes.pythonapi.PyObject_SetItem(ctypes.py_object(d),
          ctypes.py_object("abc"),  ctypes.py_object(1))
# d is now @{'abc', 1@}.
@end example

Don’t forget to use @code{py_object()}; if it’s omitted you end  up with a
segmentation fault.

@ref{29,,ctypes} has been around for a while, but people still write  and
distribution hand-coded extension modules because you can’t rely on
@ref{29,,ctypes} being present. Perhaps developers will begin to write  Python
wrappers atop a library accessed through @ref{29,,ctypes} instead of extension
modules, now that @ref{29,,ctypes} is included with core Python.


@subsubheading See also



@table @asis

@item @indicateurl{http://starship.python.net/crew/theller/ctypes/}

The ctypes web page, with a tutorial, reference, and FAQ.
@end table

The documentation  for the @ref{29,,ctypes} module.


@c ======================================================================

@node The ElementTree package,The hashlib package,The ctypes package,New Improved and Removed Modules
@anchor{whatsnew/2 5 module-etree}@anchor{f99}@anchor{whatsnew/2 5 the-elementtree-package}@anchor{f9a}
@subsubsection The ElementTree package


A subset of Fredrik Lundh’s ElementTree library for processing XML has been
added to the standard library as @code{xml.etree}.  The available modules are
@code{ElementTree}, @code{ElementPath}, and @code{ElementInclude} from
ElementTree 1.2.6.    The @code{cElementTree} accelerator module is also
included.

The rest of this section will provide a brief overview of using ElementTree.
Full documentation for ElementTree is available at
@indicateurl{http://effbot.org/zone/element-index.htm}.

ElementTree represents an XML document as a tree of element nodes. The text
content of the document is stored as the @code{text} and @code{tail}
attributes of  (This is one of the major differences between ElementTree and
the Document Object Model; in the DOM there are many different types of node,
including @code{TextNode}.)

The most commonly used parsing function is @code{parse()}, that takes either a
string (assumed to contain a filename) or a file-like object and returns an
@code{ElementTree} instance:

@example
from xml.etree import ElementTree as ET

tree = ET.parse('ex-1.xml')

feed = urllib.urlopen(
          'http://planet.python.org/rss10.xml')
tree = ET.parse(feed)
@end example

Once you have an @code{ElementTree} instance, you can call its @code{getroot()}
method to get the root @code{Element} node.

There’s also an @code{XML()} function that takes a string literal and returns an
@code{Element} node (not an @code{ElementTree}).   This function provides a
tidy way to incorporate XML fragments, approaching the convenience of an XML
literal:

@example
svg = ET.XML("""<svg width="10px" version="1.0">
             </svg>""")
svg.set('height', '320px')
svg.append(elem1)
@end example

Each XML element supports some dictionary-like and some list-like access
methods.  Dictionary-like operations are used to access attribute values, and
list-like operations are used to access child nodes.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Operation

@tab

Result

@item

@code{elem[n]}

@tab

Returns n’th child element.

@item

@code{elem[m:n]}

@tab

Returns list of m’th through n’th child
elements.

@item

@code{len(elem)}

@tab

Returns number of child elements.

@item

@code{list(elem)}

@tab

Returns list of child elements.

@item

@code{elem.append(elem2)}

@tab

Adds @emph{elem2} as a child.

@item

@code{elem.insert(index, elem2)}

@tab

Inserts @emph{elem2} at the specified location.

@item

@code{del elem[n]}

@tab

Deletes n’th child element.

@item

@code{elem.keys()}

@tab

Returns list of attribute names.

@item

@code{elem.get(name)}

@tab

Returns value of attribute @emph{name}.

@item

@code{elem.set(name, value)}

@tab

Sets new value for attribute @emph{name}.

@item

@code{elem.attrib}

@tab

Retrieves the dictionary containing
attributes.

@item

@code{del elem.attrib[name]}

@tab

Deletes attribute @emph{name}.

@end multitable


Comments and processing instructions are also represented as @code{Element}
nodes.  To check if a node is a comment or processing instructions:

@example
if elem.tag is ET.Comment:
    ...
elif elem.tag is ET.ProcessingInstruction:
    ...
@end example

To generate XML output, you should call the @code{ElementTree.write()} method.
Like @code{parse()}, it can take either a string or a file-like object:

@example
# Encoding is US-ASCII
tree.write('output.xml')

# Encoding is UTF-8
f = open('output.xml', 'w')
tree.write(f, encoding='utf-8')
@end example

(Caution: the default encoding used for output is ASCII.  For general XML work,
where an element’s name may contain arbitrary Unicode characters, ASCII isn’t a
very useful encoding because it will raise an exception if an element’s name
contains any characters with values greater than 127.  Therefore, it’s best to
specify a different encoding such as UTF-8 that can handle any Unicode
character.)

This section is only a partial description of the ElementTree interfaces. Please
read the package’s official documentation for more details.


@subsubheading See also



@table @asis

@item @indicateurl{http://effbot.org/zone/element-index.htm}

Official documentation for ElementTree.
@end table


@c ======================================================================

@node The hashlib package,The sqlite3 package,The ElementTree package,New Improved and Removed Modules
@anchor{whatsnew/2 5 module-hashlib}@anchor{f9b}@anchor{whatsnew/2 5 the-hashlib-package}@anchor{f9c}
@subsubsection The hashlib package


A new @ref{88,,hashlib} module, written by Gregory P. Smith,  has been added to
replace the @code{md5} and @code{sha} modules.  @ref{88,,hashlib} adds support for
additional secure hashes (SHA-224, SHA-256, SHA-384, and SHA-512). When
available, the module uses OpenSSL for fast platform optimized implementations
of algorithms.

The old @code{md5} and @code{sha} modules still exist as wrappers around hashlib
to preserve backwards compatibility.  The new module’s interface is very close
to that of the old modules, but not identical. The most significant difference
is that the constructor functions for creating new hashing objects are named
differently.

@example
# Old versions
h = md5.md5()
h = md5.new()

# New version
h = hashlib.md5()

# Old versions
h = sha.sha()
h = sha.new()

# New version
h = hashlib.sha1()

# Hash that weren't previously available
h = hashlib.sha224()
h = hashlib.sha256()
h = hashlib.sha384()
h = hashlib.sha512()

# Alternative form
h = hashlib.new('md5')          # Provide algorithm as a string
@end example

Once a hash object has been created, its methods are the same as before:
@code{update(string)} hashes the specified string into the  current digest
state, @code{digest()} and @code{hexdigest()} return the digest value as a binary
string or a string of hex digits, and @ref{24,,copy()} returns a new hashing object
with the same digest state.


@subsubheading See also


The documentation  for the @ref{88,,hashlib} module.


@c ======================================================================

@node The sqlite3 package,The wsgiref package,The hashlib package,New Improved and Removed Modules
@anchor{whatsnew/2 5 module-sqlite}@anchor{f9d}@anchor{whatsnew/2 5 the-sqlite3-package}@anchor{f9e}
@subsubsection The sqlite3 package


The pysqlite module (@indicateurl{http://www.pysqlite.org}), a wrapper for the SQLite embedded
database, has been added to the standard library under the package name
@ref{ed,,sqlite3}.

SQLite is a C library that provides a lightweight disk-based database that
doesn’t require a separate server process and allows accessing the database
using a nonstandard variant of the SQL query language. Some applications can use
SQLite for internal data storage.  It’s also possible to prototype an
application using SQLite and then port the code to a larger database such as
PostgreSQL or Oracle.

pysqlite was written by Gerhard Häring and provides a SQL interface compliant
with the DB-API 2.0 specification described by 
@geindex Python Enhancement Proposals; PEP 249
PEP 249@footnote{https://www.python.org/dev/peps/pep-0249}.

If you’re compiling the Python source yourself, note that the source tree
doesn’t include the SQLite code, only the wrapper module. You’ll need to have
the SQLite libraries and headers installed before compiling Python, and the
build process will compile the module when the necessary headers are available.

To use the module, you must first create a @code{Connection} object that
represents the database.  Here the data will be stored in the
@code{/tmp/example} file:

@example
conn = sqlite3.connect('/tmp/example')
@end example

You can also supply the special name @code{:memory:} to create a database in RAM.

Once you have a @code{Connection}, you can create a @code{Cursor}  object
and call its @code{execute()} method to perform SQL commands:

@example
c = conn.cursor()

# Create table
c.execute('@w{'}'create table stocks
(date text, trans text, symbol text,
 qty real, price real)'@w{'}')

# Insert a row of data
c.execute("""insert into stocks
          values ('2006-01-05','BUY','RHAT',100,35.14)""")
@end example

Usually your SQL operations will need to use values from Python variables.  You
shouldn’t assemble your query using Python’s string operations because doing so
is insecure; it makes your program vulnerable to an SQL injection attack.

Instead, use the DB-API’s parameter substitution.  Put @code{?} as a placeholder
wherever you want to use a value, and then provide a tuple of values as the
second argument to the cursor’s @code{execute()} method.  (Other database modules
may use a different placeholder, such as @code{%s} or @code{:1}.) For example:

@example
# Never do this -- insecure!
symbol = 'IBM'
c.execute("... where symbol = '%s'" % symbol)

# Do this instead
t = (symbol,)
c.execute('select * from stocks where symbol=?', t)

# Larger example
for t in (('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
          ('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00),
          ('2006-04-06', 'SELL', 'IBM', 500, 53.00),
         ):
    c.execute('insert into stocks values (?,?,?,?,?)', t)
@end example

To retrieve data after executing a SELECT statement, you can either  treat the
cursor as an iterator, call the cursor’s @code{fetchone()} method to retrieve a
single matching row,  or call @code{fetchall()} to get a list of the matching
rows.

This example uses the iterator form:

@example
>>> c = conn.cursor()
>>> c.execute('select * from stocks order by price')
>>> for row in c:
...    print row
...
(u'2006-01-05', u'BUY', u'RHAT', 100, 35.140000000000001)
(u'2006-03-28', u'BUY', u'IBM', 1000, 45.0)
(u'2006-04-06', u'SELL', u'IBM', 500, 53.0)
(u'2006-04-05', u'BUY', u'MSOFT', 1000, 72.0)
>>>
@end example

For more information about the SQL dialect supported by SQLite, see
@indicateurl{https://www.sqlite.org}.


@subsubheading See also



@table @asis

@item @indicateurl{http://www.pysqlite.org}

The pysqlite web page.

@item @indicateurl{https://www.sqlite.org}

The SQLite web page; the documentation describes the syntax and the available
data types for the supported SQL dialect.
@end table

The documentation  for the @ref{ed,,sqlite3} module.


@table @asis

@geindex Python Enhancement Proposals; PEP 249

@item PEP 249@footnote{https://www.python.org/dev/peps/pep-0249} - Database API Specification 2.0

PEP written by Marc-André Lemburg.
@end table


@c ======================================================================

@node The wsgiref package,,The sqlite3 package,New Improved and Removed Modules
@anchor{whatsnew/2 5 module-wsgiref}@anchor{f9f}@anchor{whatsnew/2 5 the-wsgiref-package}@anchor{fa0}
@subsubsection The wsgiref package


The Web Server Gateway Interface (WSGI) v1.0 defines a standard interface
between web servers and Python web applications and is described in 
@geindex Python Enhancement Proposals; PEP 333
PEP 333@footnote{https://www.python.org/dev/peps/pep-0333}.
The @ref{133,,wsgiref} package is a reference implementation of the WSGI
specification.

@c XXX should this be in a :pep:`333` section instead?

The package includes a basic HTTP server that will run a WSGI application; this
server is useful for debugging but isn’t intended for  production use.  Setting
up a server takes only a few lines of code:

@example
from wsgiref import simple_server

wsgi_app = ...

host = '@w{'}
port = 8000
httpd = simple_server.make_server(host, port, wsgi_app)
httpd.serve_forever()
@end example

@c XXX discuss structure of WSGI applications?

@c XXX provide an example using Django or some other framework?


@subsubheading See also



@table @asis

@item @indicateurl{http://www.wsgi.org}

A central web site for WSGI-related resources.

@geindex Python Enhancement Proposals; PEP 333

@item PEP 333@footnote{https://www.python.org/dev/peps/pep-0333} - Python Web Server Gateway Interface v1.0

PEP written by Phillip J. Eby.
@end table


@c ======================================================================

@node Build and C API Changes<10>,Porting to Python 2 5,New Improved and Removed Modules,What’s New in Python 2 5
@anchor{whatsnew/2 5 build-and-c-api-changes}@anchor{fa1}@anchor{whatsnew/2 5 build-api}@anchor{fa2}
@subsection Build and C API Changes


Changes to Python’s build process and to the C API include:


@itemize *

@item 
The Python source tree was converted from CVS to Subversion,  in a complex
migration procedure that was supervised and flawlessly carried out by Martin von
Löwis.  The procedure was developed as 
@geindex Python Enhancement Proposals; PEP 347
PEP 347@footnote{https://www.python.org/dev/peps/pep-0347}.

@item 
Coverity, a company that markets a source code analysis tool called Prevent,
provided the results of their examination of the Python source code.  The
analysis found about 60 bugs that  were quickly fixed.  Many of the bugs were
refcounting problems, often occurring in error-handling code.  See
@indicateurl{https://scan.coverity.com} for the statistics.

@item 
The largest change to the C API came from 
@geindex Python Enhancement Proposals; PEP 353
PEP 353@footnote{https://www.python.org/dev/peps/pep-0353}, which modifies the
interpreter to use a @code{Py_ssize_t} type definition instead of
int.  See the earlier section @ref{f76,,PEP 353; Using ssize_t as the index type} for a discussion of this
change.

@item 
The design of the bytecode compiler has changed a great deal,  no longer
generating bytecode by traversing the parse tree.  Instead the parse tree is
converted to an abstract syntax tree (or AST), and it is  the abstract syntax
tree that’s traversed to produce the bytecode.

It’s possible for Python code to obtain AST objects by using the
@ref{4e2,,compile()} built-in and specifying @code{_ast.PyCF_ONLY_AST} as the value of
the  @emph{flags} parameter:

@example
from _ast import PyCF_ONLY_AST
ast = compile("""a=0
for i in range(10):
    a += i
""", "<string>", 'exec', PyCF_ONLY_AST)

assignment = ast.body[0]
for_loop = ast.body[1]
@end example

No official documentation has been written for the AST code yet, but 
@geindex Python Enhancement Proposals; PEP 339
PEP 339@footnote{https://www.python.org/dev/peps/pep-0339}
discusses the design.  To start learning about the code, read the definition of
the various AST nodes in @code{Parser/Python.asdl}.  A Python script reads this
file and generates a set of C structure definitions in
@code{Include/Python-ast.h}.  The @code{PyParser_ASTFromString()} and
@code{PyParser_ASTFromFile()}, defined in @code{Include/pythonrun.h}, take
Python source as input and return the root of an AST representing the contents.
This AST can then be turned into a code object by @code{PyAST_Compile()}.  For
more information, read the source code, and then ask questions on python-dev.

The AST code was developed under Jeremy Hylton’s management, and implemented by
(in alphabetical order) Brett Cannon, Nick Coghlan, Grant Edwards, John
Ehresman, Kurt Kaiser, Neal Norwitz, Tim Peters, Armin Rigo, and Neil
Schemenauer, plus the participants in a number of AST sprints at conferences
such as PyCon.

@c List of names taken from Jeremy's python-dev post at

@c https://mail.python.org/pipermail/python-dev/2005-October/057500.html

@item 
Evan Jones’s patch to obmalloc, first described in a talk at PyCon DC 2005,
was applied.  Python 2.4 allocated small objects in 256K-sized arenas, but never
freed arenas.  With this patch, Python will free arenas when they’re empty.  The
net effect is that on some platforms, when you allocate many objects, Python’s
memory usage may actually drop when you delete them and the memory may be
returned to the operating system.  (Implemented by Evan Jones, and reworked by
Tim Peters.)

Note that this change means extension modules must be more careful when
allocating memory.  Python’s API has many different functions for allocating
memory that are grouped into families.  For example, @ref{7ad,,PyMem_Malloc()},
@ref{bc6,,PyMem_Realloc()}, and @ref{fa3,,PyMem_Free()} are one family that allocates
raw memory, while @ref{7af,,PyObject_Malloc()}, @ref{fa4,,PyObject_Realloc()}, and
@ref{7ac,,PyObject_Free()} are another family that’s supposed to be used for
creating Python objects.

Previously these different families all reduced to the platform’s
@code{malloc()} and @code{free()} functions.  This meant  it didn’t matter if
you got things wrong and allocated memory with the @code{PyMem()} function but
freed it with the @ref{375,,PyObject()} function.  With 2.5’s changes to obmalloc,
these families now do different things and mismatches will probably result in a
segfault.  You should carefully test your C extension modules with Python 2.5.

@item 
The built-in set types now have an official C API.  Call @ref{fa5,,PySet_New()}
and @ref{fa6,,PyFrozenSet_New()} to create a new set, @ref{fa7,,PySet_Add()} and
@ref{fa8,,PySet_Discard()} to add and remove elements, and @ref{fa9,,PySet_Contains()}
and @ref{faa,,PySet_Size()} to examine the set’s state. (Contributed by Raymond
Hettinger.)

@item 
C code can now obtain information about the exact revision of the Python
interpreter by calling the  @ref{f94,,Py_GetBuildInfo()} function that returns a
string of build information like this: @code{"trunk:45355:45356M, Apr 13 2006,
07:42:19"}.   (Contributed by Barry Warsaw.)

@item 
Two new macros can be used to indicate C functions that are local to the
current file so that a faster calling convention can be used.
@code{Py_LOCAL(type)} declares the function as returning a value of the
specified @emph{type} and uses a fast-calling qualifier.
@code{Py_LOCAL_INLINE(type)} does the same thing and also requests the
function be inlined.  If @code{PY_LOCAL_AGGRESSIVE()} is defined before
@code{python.h} is included, a set of more aggressive optimizations are enabled
for the module; you should benchmark the results to find out if these
optimizations actually make the code faster.  (Contributed by Fredrik Lundh at
the NeedForSpeed sprint.)

@item 
@code{PyErr_NewException(name, base, dict)} can now accept a tuple of base
classes as its @emph{base} argument.  (Contributed by Georg Brandl.)

@item 
The @code{PyErr_Warn()} function for issuing warnings is now deprecated in
favour of @code{PyErr_WarnEx(category, message, stacklevel)} which lets you
specify the number of stack frames separating this function and the caller.  A
@emph{stacklevel} of 1 is the function calling @ref{fab,,PyErr_WarnEx()}, 2 is the
function above that, and so forth.  (Added by Neal Norwitz.)

@item 
The CPython interpreter is still written in C, but  the code can now be
compiled with a C++ compiler without errors.   (Implemented by Anthony Baxter,
Martin von Löwis, Skip Montanaro.)

@item 
The @code{PyRange_New()} function was removed.  It was never documented, never
used in the core code, and had dangerously lax error checking.  In the unlikely
case that your extensions were using it, you can replace it by something like
the following:

@example
range = PyObject_CallFunction((PyObject*) &PyRange_Type, "lll",
                              start, stop, step);
@end example
@end itemize

@c ======================================================================

@menu
* Port-Specific Changes:: 

@end menu

@node Port-Specific Changes,,,Build and C API Changes<10>
@anchor{whatsnew/2 5 port-specific-changes}@anchor{fac}@anchor{whatsnew/2 5 ports}@anchor{fad}
@subsubsection Port-Specific Changes



@itemize *

@item 
MacOS X (10.3 and higher): dynamic loading of modules now uses the
@code{dlopen()} function instead of MacOS-specific functions.

@item 
MacOS X: an @code{--enable-universalsdk} switch was added to the
@code{configure} script that compiles the interpreter as a universal binary
able to run on both PowerPC and Intel processors. (Contributed by Ronald
Oussoren; bpo-2573@footnote{https://bugs.python.org/issue2573}.)

@item 
Windows: @code{.dll} is no longer supported as a filename extension for
extension modules.  @code{.pyd} is now the only filename extension that will be
searched for.
@end itemize

@c ======================================================================

@node Porting to Python 2 5,Acknowledgements<3>,Build and C API Changes<10>,What’s New in Python 2 5
@anchor{whatsnew/2 5 porting}@anchor{fae}@anchor{whatsnew/2 5 porting-to-python-2-5}@anchor{faf}
@subsection Porting to Python 2.5


This section lists previously described changes that may require changes to your
code:


@itemize *

@item 
ASCII is now the default encoding for modules.  It’s now  a syntax error if a
module contains string literals with 8-bit characters but doesn’t have an
encoding declaration.  In Python 2.4 this triggered a warning, not a syntax
error.

@item 
Previously, the @code{gi_frame} attribute of a generator was always a frame
object.  Because of the 
@geindex Python Enhancement Proposals; PEP 342
PEP 342@footnote{https://www.python.org/dev/peps/pep-0342} changes described in section @ref{f61,,PEP 342; New Generator Features},
it’s now possible for @code{gi_frame} to be @code{None}.

@item 
A new warning, @ref{f81,,UnicodeWarning}, is triggered when  you attempt to
compare a Unicode string and an 8-bit string that can’t be converted to Unicode
using the default ASCII encoding.  Previously such comparisons would raise a
@ref{51f,,UnicodeDecodeError} exception.

@item 
Library: the @ref{28,,csv} module is now stricter about multi-line quoted fields.
If your files contain newlines embedded within fields, the input should be split
into lines in a manner which preserves the newline characters.

@item 
Library: the @ref{a5,,locale} module’s  @ref{785,,format()} function’s would
previously  accept any string as long as no more than one %char specifier
appeared.  In Python 2.5, the argument must be exactly one %char specifier with
no surrounding text.

@item 
Library: The @ref{c5,,pickle} and @code{cPickle} modules no longer accept a
return value of @code{None} from the @code{__reduce__()} method; the method must
return a tuple of arguments instead.  The modules also no longer accept the
deprecated @emph{bin} keyword parameter.

@item 
Library: The @code{SimpleXMLRPCServer} and @code{DocXMLRPCServer}  classes now
have a @code{rpc_paths} attribute that constrains XML-RPC operations to a
limited set of URL paths; the default is to allow only @code{'/'} and @code{'/RPC2'}.
Setting  @code{rpc_paths} to @code{None} or an empty tuple disables  this path
checking.

@item 
C API: Many functions now use @code{Py_ssize_t}  instead of int to
allow processing more data on 64-bit machines.  Extension code may need to make
the same change to avoid warnings and to support 64-bit machines.  See the
earlier section @ref{f76,,PEP 353; Using ssize_t as the index type} for a discussion of this change.

@item 
C API:  The obmalloc changes mean that  you must be careful to not mix usage
of the PyMem_* and PyObject_* families of functions. Memory
allocated with  one family’s *_Malloc must be  freed with the
corresponding family’s *_Free function.
@end itemize

@c ======================================================================

@node Acknowledgements<3>,,Porting to Python 2 5,What’s New in Python 2 5
@anchor{whatsnew/2 5 acknowledgements}@anchor{fb0}
@subsection Acknowledgements


The author would like to thank the following people for offering suggestions,
corrections and assistance with various drafts of this article: Georg Brandl,
Nick Coghlan, Phillip J. Eby, Lars Gustäbel, Raymond Hettinger, Ralf W.
Grosse-Kunstleve, Kent Johnson, Iain Lowe, Martin von Löwis, Fredrik Lundh, Andrew
McNamara, Skip Montanaro, Gustavo Niemeyer, Paul Prescod, James Pryor, Mike
Rovner, Scott Weikart, Barry Warsaw, Thomas Wouters.

@node What’s New in Python 2 4,What’s New in Python 2 3,What’s New in Python 2 5,What’s New in Python
@anchor{whatsnew/2 4 doc}@anchor{fb1}@anchor{whatsnew/2 4 what-s-new-in-python-2-4}@anchor{fb2}
@section What’s New in Python 2.4


@*Author: 
A.M. Kuchling


@c $Id: whatsnew24.tex 54632 2007-03-31 11:59:54Z georg.brandl $

@c Don't write extensive text for new sections; I'll do that.

@c Feel free to add commented-out reminders of things that need

@c to be covered.  --amk

This article explains the new features in Python 2.4.1, released on March 30,
2005.

Python 2.4 is a medium-sized release.  It doesn’t introduce as many changes as
the radical Python 2.2, but introduces more features than the conservative 2.3
release.  The most significant new language features are function decorators and
generator expressions; most other changes are to the standard library.

According to the CVS change logs, there were 481 patches applied and 502 bugs
fixed between Python 2.3 and 2.4.  Both figures are likely to be underestimates.

This article doesn’t attempt to provide a complete specification of every single
new feature, but instead provides a brief introduction to each feature.  For
full details, you should refer to the documentation for Python 2.4, such as the
Python Library Reference and the Python Reference Manual.  Often you will be
referred to the PEP for a particular new feature for explanations of the
implementation and design rationale.

@c ======================================================================

@menu
* PEP 218; Built-In Set Objects: PEP 218 Built-In Set Objects. 
* PEP 237; Unifying Long Integers and Integers: PEP 237 Unifying Long Integers and Integers. 
* PEP 289; Generator Expressions: PEP 289 Generator Expressions. 
* PEP 292; Simpler String Substitutions: PEP 292 Simpler String Substitutions. 
* PEP 318; Decorators for Functions and Methods: PEP 318 Decorators for Functions and Methods. 
* PEP 322; Reverse Iteration: PEP 322 Reverse Iteration. 
* PEP 324; New subprocess Module: PEP 324 New subprocess Module. 
* PEP 327; Decimal Data Type: PEP 327 Decimal Data Type. 
* PEP 328; Multi-line Imports: PEP 328 Multi-line Imports. 
* PEP 331; Locale-Independent Float/String Conversions: PEP 331 Locale-Independent Float/String Conversions. 
* Other Language Changes: Other Language Changes<15>. 
* New@comma{} Improved@comma{} and Deprecated Modules: New Improved and Deprecated Modules<3>. 
* Build and C API Changes: Build and C API Changes<11>. 
* Porting to Python 2.4: Porting to Python 2 4. 
* Acknowledgements: Acknowledgements<4>. 

@end menu

@node PEP 218 Built-In Set Objects,PEP 237 Unifying Long Integers and Integers,,What’s New in Python 2 4
@anchor{whatsnew/2 4 pep-218-built-in-set-objects}@anchor{fb3}
@subsection PEP 218: Built-In Set Objects


Python 2.3 introduced the @code{sets} module.  C implementations of set data
types have now been added to the Python core as two new built-in types,
@code{set(iterable)} and @code{frozenset(iterable)}.  They provide high speed
operations for membership testing, for eliminating duplicates from sequences,
and for mathematical operations like unions, intersections, differences, and
symmetric differences.

@example
>>> a = set('abracadabra')              # form a set from a string
>>> 'z' in a                            # fast membership testing
False
>>> a                                   # unique letters in a
set(['a', 'r', 'b', 'c', 'd'])
>>> '@w{'}.join(a)                          # convert back into a string
'arbcd'

>>> b = set('alacazam')                 # form a second set
>>> a - b                               # letters in a but not in b
set(['r', 'd', 'b'])
>>> a | b                               # letters in either a or b
set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'])
>>> a & b                               # letters in both a and b
set(['a', 'c'])
>>> a ^ b                               # letters in a or b but not both
set(['r', 'd', 'b', 'm', 'z', 'l'])

>>> a.add('z')                          # add a new element
>>> a.update('wxy')                     # add multiple new elements
>>> a
set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'x', 'z'])
>>> a.remove('x')                       # take one element out
>>> a
set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'z'])
@end example

The @ref{439,,frozenset()} type is an immutable version of @ref{370,,set()}. Since it is
immutable and hashable, it may be used as a dictionary key or as a member of
another set.

The @code{sets} module remains in the standard library, and may be useful if you
wish to subclass the @code{Set} or @code{ImmutableSet} classes.  There are
currently no plans to deprecate the module.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 218

@item PEP 218@footnote{https://www.python.org/dev/peps/pep-0218} - Adding a Built-In Set Object Type

Originally proposed by Greg Wilson and ultimately implemented by Raymond
Hettinger.
@end table


@c ======================================================================

@node PEP 237 Unifying Long Integers and Integers,PEP 289 Generator Expressions,PEP 218 Built-In Set Objects,What’s New in Python 2 4
@anchor{whatsnew/2 4 pep-237-unifying-long-integers-and-integers}@anchor{fb4}
@subsection PEP 237: Unifying Long Integers and Integers


The lengthy transition process for this PEP, begun in Python 2.2, takes another
step forward in Python 2.4.  In 2.3, certain integer operations that would
behave differently after int/long unification triggered @ref{604,,FutureWarning}
warnings and returned values limited to 32 or 64 bits (depending on your
platform).  In 2.4, these expressions no longer produce a warning and instead
produce a different result that’s usually a long integer.

The problematic expressions are primarily left shifts and lengthy hexadecimal
and octal constants.  For example, @code{2 << 32} results in a warning in 2.3,
evaluating to 0 on 32-bit platforms.  In Python 2.4, this expression now returns
the correct answer, 8589934592.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 237

@item PEP 237@footnote{https://www.python.org/dev/peps/pep-0237} - Unifying Long Integers and Integers

Original PEP written by Moshe Zadka and GvR.  The changes for 2.4 were
implemented by  Kalle Svensson.
@end table


@c ======================================================================

@node PEP 289 Generator Expressions,PEP 292 Simpler String Substitutions,PEP 237 Unifying Long Integers and Integers,What’s New in Python 2 4
@anchor{whatsnew/2 4 pep-289-generator-expressions}@anchor{fb5}
@subsection PEP 289: Generator Expressions


The iterator feature introduced in Python 2.2 and the @ref{9f,,itertools} module
make it easier to write programs that loop through large data sets without
having the entire data set in memory at one time.  List comprehensions don’t fit
into this picture very well because they produce a Python list object containing
all of the items.  This unavoidably pulls all of the objects into memory, which
can be a problem if your data set is very large.  When trying to write a
functionally-styled program, it would be natural to write something like:

@example
links = [link for link in get_all_links() if not link.followed]
for link in links:
    ...
@end example

instead of

@example
for link in get_all_links():
    if link.followed:
        continue
    ...
@end example

The first form is more concise and perhaps more readable, but if you’re dealing
with a large number of link objects you’d have to write the second form to avoid
having all link objects in memory at the same time.

Generator expressions work similarly to list comprehensions but don’t
materialize the entire list; instead they create a generator that will return
elements one by one.  The above example could be written as:

@example
links = (link for link in get_all_links() if not link.followed)
for link in links:
    ...
@end example

Generator expressions always have to be written inside parentheses, as in the
above example.  The parentheses signalling a function call also count, so if you
want to create an iterator that will be immediately passed to a function you
could write:

@example
print sum(obj.count for obj in list_all_objects())
@end example

Generator expressions differ from list comprehensions in various small ways.
Most notably, the loop variable (@emph{obj} in the above example) is not accessible
outside of the generator expression.  List comprehensions leave the variable
assigned to its last value; future versions of Python will change this, making
list comprehensions match generator expressions in this respect.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 289

@item PEP 289@footnote{https://www.python.org/dev/peps/pep-0289} - Generator Expressions

Proposed by Raymond Hettinger and implemented by Jiwon Seo with early efforts
steered by Hye-Shik Chang.
@end table


@c ======================================================================

@node PEP 292 Simpler String Substitutions,PEP 318 Decorators for Functions and Methods,PEP 289 Generator Expressions,What’s New in Python 2 4
@anchor{whatsnew/2 4 pep-292-simpler-string-substitutions}@anchor{fb6}
@subsection PEP 292: Simpler String Substitutions


Some new classes in the standard library provide an alternative mechanism for
substituting variables into strings; this style of substitution may be better
for applications where untrained users need to edit templates.

The usual way of substituting variables by name is the @code{%} operator:

@example
>>> '%(page)i: %(title)s' % @{'page':2, 'title': 'The Best of Times'@}
'2: The Best of Times'
@end example

When writing the template string, it can be easy to forget the @code{i} or @code{s}
after the closing parenthesis.  This isn’t a big problem if the template is in a
Python module, because you run the code, get an “Unsupported format character”
@ref{17c,,ValueError}, and fix the problem.  However, consider an application such
as Mailman where template strings or translations are being edited by users who
aren’t aware of the Python language.  The format string’s syntax is complicated
to explain to such users, and if they make a mistake, it’s difficult to provide
helpful feedback to them.

PEP 292 adds a @code{Template} class to the @ref{f1,,string} module that uses
@code{$} to indicate a substitution:

@example
>>> import string
>>> t = string.Template('$page: $title')
>>> t.substitute(@{'page':2, 'title': 'The Best of Times'@})
'2: The Best of Times'
@end example

If a key is missing from the dictionary, the @code{substitute()} method will
raise a @ref{5bb,,KeyError}.  There’s also a @code{safe_substitute()} method that
ignores missing keys:

@example
>>> t = string.Template('$page: $title')
>>> t.safe_substitute(@{'page':3@})
'3: $title'
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 292

@item PEP 292@footnote{https://www.python.org/dev/peps/pep-0292} - Simpler String Substitutions

Written and implemented  by Barry Warsaw.
@end table


@c ======================================================================

@node PEP 318 Decorators for Functions and Methods,PEP 322 Reverse Iteration,PEP 292 Simpler String Substitutions,What’s New in Python 2 4
@anchor{whatsnew/2 4 pep-318-decorators-for-functions-and-methods}@anchor{fb7}
@subsection PEP 318: Decorators for Functions and Methods


Python 2.2 extended Python’s object model by adding static methods and class
methods, but it didn’t extend Python’s syntax to provide any new way of defining
static or class methods.  Instead, you had to write a @ref{fb8,,def} statement
in the usual way, and pass the resulting method to a @ref{253,,staticmethod()} or
@ref{254,,classmethod()} function that would wrap up the function as a method of the
new type. Your code would look like this:

@example
class C:
   def meth (cls):
       ...

   meth = classmethod(meth)   # Rebind name to wrapped-up class method
@end example

If the method was very long, it would be easy to miss or forget the
@ref{254,,classmethod()} invocation after the function body.

The intention was always to add some syntax to make such definitions more
readable, but at the time of 2.2’s release a good syntax was not obvious.  Today
a good syntax @emph{still} isn’t obvious but users are asking for easier access to
the feature; a new syntactic feature has been added to meet this need.

The new feature is called “function decorators”.  The name comes from the idea
that @ref{254,,classmethod()}, @ref{253,,staticmethod()}, and friends are storing
additional information on a function object; they’re @emph{decorating} functions with
more details.

The notation borrows from Java and uses the @code{'@@'} character as an indicator.
Using the new syntax, the example above would be written:

@example
class C:

   @@classmethod
   def meth (cls):
       ...
@end example

The @code{@@classmethod} is shorthand for the @code{meth=classmethod(meth)} assignment.
More generally, if you have the following:

@example
@@A
@@B
@@C
def f ():
    ...
@end example

It’s equivalent to the following pre-decorator code:

@example
def f(): ...
f = A(B(C(f)))
@end example

Decorators must come on the line before a function definition, one decorator per
line, and can’t be on the same line as the def statement, meaning that @code{@@A def
f(): ...} is illegal.  You can only decorate function definitions, either at
the module level or inside a class; you can’t decorate class definitions.

A decorator is just a function that takes the function to be decorated as an
argument and returns either the same function or some new object.  The return
value of the decorator need not be callable (though it typically is), unless
further decorators will be applied to the result.  It’s easy to write your own
decorators.  The following simple example just sets an attribute on the function
object:

@example
>>> def deco(func):
...    func.attr = 'decorated'
...    return func
...
>>> @@deco
... def f(): pass
...
>>> f
<function f at 0x402ef0d4>
>>> f.attr
'decorated'
>>>
@end example

As a slightly more realistic example, the following decorator checks that the
supplied argument is an integer:

@example
def require_int (func):
    def wrapper (arg):
        assert isinstance(arg, int)
        return func(arg)

    return wrapper

@@require_int
def p1 (arg):
    print arg

@@require_int
def p2(arg):
    print arg*2
@end example

An example in 
@geindex Python Enhancement Proposals; PEP 318
PEP 318@footnote{https://www.python.org/dev/peps/pep-0318} contains a fancier version of this idea that lets you
both specify the required type and check the returned type.

Decorator functions can take arguments.  If arguments are supplied, your
decorator function is called with only those arguments and must return a new
decorator function; this function must take a single function and return a
function, as previously described.  In other words, @code{@@A @@B @@C(args)} becomes:

@example
def f(): ...
_deco = C(args)
f = A(B(_deco(f)))
@end example

Getting this right can be slightly brain-bending, but it’s not too difficult.

A small related change makes the @code{func_name} attribute of functions
writable.  This attribute is used to display function names in tracebacks, so
decorators should change the name of any new function that’s constructed and
returned.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 318

@item PEP 318@footnote{https://www.python.org/dev/peps/pep-0318} - Decorators for Functions, Methods and Classes

Written  by Kevin D. Smith, Jim Jewett, and Skip Montanaro.  Several people
wrote patches implementing function decorators, but the one that was actually
checked in was patch #979728, written by Mark Russell.

@item @indicateurl{https://wiki.python.org/moin/PythonDecoratorLibrary}

This Wiki page contains several examples of decorators.
@end table


@c ======================================================================

@node PEP 322 Reverse Iteration,PEP 324 New subprocess Module,PEP 318 Decorators for Functions and Methods,What’s New in Python 2 4
@anchor{whatsnew/2 4 pep-322-reverse-iteration}@anchor{fb9}
@subsection PEP 322: Reverse Iteration


A new built-in function, @code{reversed(seq)}, takes a sequence and returns an
iterator that loops over the elements of the sequence  in reverse order.

@example
>>> for i in reversed(xrange(1,4)):
...    print i
...
3
2
1
@end example

Compared to extended slicing, such as @code{range(1,4)[::-1]}, @ref{2f6,,reversed()} is
easier to read, runs faster, and uses substantially less memory.

Note that @ref{2f6,,reversed()} only accepts sequences, not arbitrary iterators.  If
you want to reverse an iterator, first convert it to  a list with @ref{43a,,list()}.

@example
>>> input = open('/etc/passwd', 'r')
>>> for line in reversed(list(input)):
...   print line
...
root:*:0:0:System Administrator:/var/root:/bin/tcsh
  ...
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 322

@item PEP 322@footnote{https://www.python.org/dev/peps/pep-0322} - Reverse Iteration

Written and implemented by Raymond Hettinger.
@end table


@c ======================================================================

@node PEP 324 New subprocess Module,PEP 327 Decimal Data Type,PEP 322 Reverse Iteration,What’s New in Python 2 4
@anchor{whatsnew/2 4 pep-324-new-subprocess-module}@anchor{fba}
@subsection PEP 324: New subprocess Module


The standard library provides a number of ways to execute a subprocess, offering
different features and different levels of complexity.
@code{os.system(command)} is easy to use, but slow (it runs a shell process
which executes the command) and dangerous (you have to be careful about escaping
the shell’s metacharacters).  The @code{popen2} module offers classes that can
capture standard output and standard error from the subprocess, but the naming
is confusing.  The @ref{f4,,subprocess} module cleans  this up, providing a unified
interface that offers all the features you might need.

Instead of @code{popen2}’s collection of classes, @ref{f4,,subprocess} contains a
single class called @code{Popen}  whose constructor supports a number of
different keyword arguments.

@example
class Popen(args, bufsize=0, executable=None,
            stdin=None, stdout=None, stderr=None,
            preexec_fn=None, close_fds=False, shell=False,
            cwd=None, env=None, universal_newlines=False,
            startupinfo=None, creationflags=0):
@end example

@emph{args} is commonly a sequence of strings that will be the arguments to the
program executed as the subprocess.  (If the @emph{shell} argument is true, @emph{args}
can be a string which will then be passed on to the shell for interpretation,
just as @ref{fbb,,os.system()} does.)

@emph{stdin}, @emph{stdout}, and @emph{stderr} specify what the subprocess’s input, output, and
error streams will be.  You can provide a file object or a file descriptor, or
you can use the constant @code{subprocess.PIPE} to create a pipe between the
subprocess and the parent.

@geindex universal newlines; What's new

The constructor has a number of handy options:


@itemize *

@item 
@emph{close_fds} requests that all file descriptors be closed before running the
subprocess.

@item 
@emph{cwd} specifies the working directory in which the subprocess will be executed
(defaulting to whatever the parent’s working directory is).

@item 
@emph{env} is a dictionary specifying environment variables.

@item 
@emph{preexec_fn} is a function that gets called before the child is started.

@item 
@emph{universal_newlines} opens the child’s input and output using Python’s
@ref{88a,,universal newlines} feature.
@end itemize

Once you’ve created the @code{Popen} instance,  you can call its @code{wait()}
method to pause until the subprocess has exited, @code{poll()} to check if it’s
exited without pausing,  or @code{communicate(data)} to send the string @emph{data}
to the subprocess’s standard input.   @code{communicate(data)}  then reads any
data that the subprocess has sent to its standard output  or standard error,
returning a tuple @code{(stdout_data, stderr_data)}.

@code{call()} is a shortcut that passes its arguments along to the @code{Popen}
constructor, waits for the command to complete, and returns the status code of
the subprocess.  It can serve as a safer analog to @ref{fbb,,os.system()}:

@example
sts = subprocess.call(['dpkg', '-i', '/tmp/new-package.deb'])
if sts == 0:
    # Success
    ...
else:
    # dpkg returned an error
    ...
@end example

The command is invoked without use of the shell.  If you really do want to  use
the shell, you can add @code{shell=True} as a keyword argument and provide a string
instead of a sequence:

@example
sts = subprocess.call('dpkg -i /tmp/new-package.deb', shell=True)
@end example

The PEP takes various examples of shell and Python code and shows how they’d be
translated into Python code that uses @ref{f4,,subprocess}.  Reading this section
of the PEP is highly recommended.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 324

@item PEP 324@footnote{https://www.python.org/dev/peps/pep-0324} - subprocess - New process module

Written and implemented by Peter Åstrand, with assistance from Fredrik Lundh and
others.
@end table


@c ======================================================================

@node PEP 327 Decimal Data Type,PEP 328 Multi-line Imports,PEP 324 New subprocess Module,What’s New in Python 2 4
@anchor{whatsnew/2 4 pep-327-decimal-data-type}@anchor{fbc}
@subsection PEP 327: Decimal Data Type


Python has always supported floating-point (FP) numbers, based on the underlying
C double type, as a data type.  However, while most programming
languages provide a floating-point type, many people (even programmers) are
unaware that floating-point numbers don’t represent certain decimal fractions
accurately.  The new @code{Decimal} type can represent these fractions
accurately, up to a user-specified precision limit.

@menu
* Why is Decimal needed?:: 
* The Decimal type:: 
* The Context type:: 

@end menu

@node Why is Decimal needed?,The Decimal type,,PEP 327 Decimal Data Type
@anchor{whatsnew/2 4 why-is-decimal-needed}@anchor{fbd}
@subsubsection Why is Decimal needed?


The limitations arise from the representation used for floating-point numbers.
FP numbers are made up of three components:


@itemize *

@item 
The sign, which is positive or negative.

@item 
The mantissa, which is a single-digit binary number   followed by a fractional
part.  For example, @code{1.01} in base-2 notation is @code{1 + 0/2 + 1/4}, or 1.25 in
decimal notation.

@item 
The exponent, which tells where the decimal point is located in the number
represented.
@end itemize

For example, the number 1.25 has positive sign, a mantissa value of 1.01 (in
binary), and an exponent of 0 (the decimal point doesn’t need to be shifted).
The number 5 has the same sign and mantissa, but the exponent is 2 because the
mantissa is multiplied by 4 (2 to the power of the exponent 2); 1.25 * 4 equals
5.

Modern systems usually provide floating-point support that conforms to a
standard called IEEE 754.  C’s double type is usually implemented as a
64-bit IEEE 754 number, which uses 52 bits of space for the mantissa.  This
means that numbers can only be specified to 52 bits of precision.  If you’re
trying to represent numbers whose expansion repeats endlessly, the expansion is
cut off after 52 bits. Unfortunately, most software needs to produce output in
base 10, and common fractions in base 10 are often repeating decimals in binary.
For example, 1.1 decimal is binary @code{1.0001100110011 ...}; .1 = 1/16 + 1/32 +
1/256 plus an infinite number of additional terms.  IEEE 754 has to chop off
that infinitely repeated decimal after 52 digits, so the representation is
slightly inaccurate.

Sometimes you can see this inaccuracy when the number is printed:

@example
>>> 1.1
1.1000000000000001
@end example

The inaccuracy isn’t always visible when you print the number because the
FP-to-decimal-string conversion is provided by the C library, and most C libraries try
to produce sensible output.  Even if it’s not displayed, however, the inaccuracy
is still there and subsequent operations can magnify the error.

For many applications this doesn’t matter.  If I’m plotting points and
displaying them on my monitor, the difference between 1.1 and 1.1000000000000001
is too small to be visible.  Reports often limit output to a certain number of
decimal places, and if you round the number to two or three or even eight
decimal places, the error is never apparent.  However, for applications where it
does matter,  it’s a lot of work to implement your own custom arithmetic
routines.

Hence, the @code{Decimal} type was created.

@node The Decimal type,The Context type,Why is Decimal needed?,PEP 327 Decimal Data Type
@anchor{whatsnew/2 4 the-decimal-type}@anchor{fbe}
@subsubsection The @code{Decimal} type


A new module, @ref{34,,decimal}, was added to Python’s standard library.  It
contains two classes, @code{Decimal} and @code{Context}.  @code{Decimal}
instances represent numbers, and @code{Context} instances are used to wrap up
various settings such as the precision and default rounding mode.

@code{Decimal} instances are immutable, like regular Python integers and FP
numbers; once it’s been created, you can’t change the value an instance
represents.  @code{Decimal} instances can be created from integers or
strings:

@example
>>> import decimal
>>> decimal.Decimal(1972)
Decimal("1972")
>>> decimal.Decimal("1.1")
Decimal("1.1")
@end example

You can also provide tuples containing the sign, the mantissa represented  as a
tuple of decimal digits, and the exponent:

@example
>>> decimal.Decimal((1, (1, 4, 7, 5), -2))
Decimal("-14.75")
@end example

Cautionary note: the sign bit is a Boolean value, so 0 is positive and 1 is
negative.

Converting from floating-point numbers poses a bit of a problem: should the FP
number representing 1.1 turn into the decimal number for exactly 1.1, or for 1.1
plus whatever inaccuracies are introduced? The decision was to dodge the issue
and leave such a conversion out of the API.  Instead, you should convert the
floating-point number into a string using the desired precision and pass the
string to the @code{Decimal} constructor:

@example
>>> f = 1.1
>>> decimal.Decimal(str(f))
Decimal("1.1")
>>> decimal.Decimal('%.12f' % f)
Decimal("1.100000000000")
@end example

Once you have @code{Decimal} instances, you can perform the usual mathematical
operations on them.  One limitation: exponentiation requires an integer
exponent:

@example
>>> a = decimal.Decimal('35.72')
>>> b = decimal.Decimal('1.73')
>>> a+b
Decimal("37.45")
>>> a-b
Decimal("33.99")
>>> a*b
Decimal("61.7956")
>>> a/b
Decimal("20.64739884393063583815028902")
>>> a ** 2
Decimal("1275.9184")
>>> a**b
Traceback (most recent call last):
  ...
decimal.InvalidOperation: x ** (non-integer)
@end example

You can combine @code{Decimal} instances with integers, but not with
floating-point numbers:

@example
>>> a + 4
Decimal("39.72")
>>> a + 4.5
Traceback (most recent call last):
  ...
TypeError: You can interact Decimal only with int, long or Decimal data types.
>>>
@end example

@code{Decimal} numbers can be used with the @ref{ad,,math} and @ref{17,,cmath}
modules, but note that they’ll be immediately converted to  floating-point
numbers before the operation is performed, resulting in a possible loss of
precision and accuracy.  You’ll also get back a regular floating-point number
and not a @code{Decimal}.

@example
>>> import math, cmath
>>> d = decimal.Decimal('123456789012.345')
>>> math.sqrt(d)
351364.18288201344
>>> cmath.sqrt(-d)
351364.18288201344j
@end example

@code{Decimal} instances have a @code{sqrt()} method that returns a
@code{Decimal}, but if you need other things such as trigonometric functions
you’ll have to implement them.

@example
>>> d.sqrt()
Decimal("351364.1828820134592177245001")
@end example

@node The Context type,,The Decimal type,PEP 327 Decimal Data Type
@anchor{whatsnew/2 4 the-context-type}@anchor{fbf}
@subsubsection The @code{Context} type


Instances of the @code{Context} class encapsulate several settings for
decimal operations:


@itemize *

@item 
@code{prec} is the precision, the number of decimal places.

@item 
@code{rounding} specifies the rounding mode.  The @ref{34,,decimal} module has
constants for the various possibilities: @code{ROUND_DOWN},
@code{ROUND_CEILING},  @code{ROUND_HALF_EVEN}, and various others.

@item 
@code{traps} is a dictionary specifying what happens on encountering certain
error conditions: either  an exception is raised or  a value is returned.  Some
examples of error conditions are division by zero, loss of precision, and
overflow.
@end itemize

There’s a thread-local default context available by calling @code{getcontext()};
you can change the properties of this context to alter the default precision,
rounding, or trap handling.  The following example shows the effect of changing
the precision of the default context:

@example
>>> decimal.getcontext().prec
28
>>> decimal.Decimal(1) / decimal.Decimal(7)
Decimal("0.1428571428571428571428571429")
>>> decimal.getcontext().prec = 9
>>> decimal.Decimal(1) / decimal.Decimal(7)
Decimal("0.142857143")
@end example

The default action for error conditions is selectable; the module can either
return a special value such as infinity or not-a-number, or exceptions can be
raised:

@example
>>> decimal.Decimal(1) / decimal.Decimal(0)
Traceback (most recent call last):
  ...
decimal.DivisionByZero: x / 0
>>> decimal.getcontext().traps[decimal.DivisionByZero] = False
>>> decimal.Decimal(1) / decimal.Decimal(0)
Decimal("Infinity")
>>>
@end example

The @code{Context} instance also has various methods for formatting  numbers
such as @code{to_eng_string()} and @code{to_sci_string()}.

For more information, see the documentation for the @ref{34,,decimal} module, which
includes a quick-start tutorial and a reference.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 327

@item PEP 327@footnote{https://www.python.org/dev/peps/pep-0327} - Decimal Data Type

Written by Facundo Batista and implemented by Facundo Batista, Eric Price,
Raymond Hettinger, Aahz, and Tim Peters.

@item @indicateurl{http://www.lahey.com/float.htm}

The article uses Fortran code to illustrate many of the problems that
floating-point inaccuracy can cause.

@item @indicateurl{http://speleotrove.com/decimal/}

A description of a decimal-based representation.  This representation is being
proposed as a standard, and underlies the new Python decimal type.  Much of this
material was written by Mike Cowlishaw, designer of the Rexx language.
@end table


@c ======================================================================

@node PEP 328 Multi-line Imports,PEP 331 Locale-Independent Float/String Conversions,PEP 327 Decimal Data Type,What’s New in Python 2 4
@anchor{whatsnew/2 4 pep-328-multi-line-imports}@anchor{fc0}
@subsection PEP 328: Multi-line Imports


One language change is a small syntactic tweak aimed at making it easier to
import many names from a module.  In a @code{from module import names} statement,
@emph{names} is a sequence of names separated by commas.  If the sequence is  very
long, you can either write multiple imports from the same module, or you can use
backslashes to escape the line endings like this:

@example
from SimpleXMLRPCServer import SimpleXMLRPCServer,\
            SimpleXMLRPCRequestHandler,\
            CGIXMLRPCRequestHandler,\
            resolve_dotted_attribute
@end example

The syntactic change in Python 2.4 simply allows putting the names within
parentheses.  Python ignores newlines within a parenthesized expression, so the
backslashes are no longer needed:

@example
from SimpleXMLRPCServer import (SimpleXMLRPCServer,
                                SimpleXMLRPCRequestHandler,
                                CGIXMLRPCRequestHandler,
                                resolve_dotted_attribute)
@end example

The PEP also proposes that all @ref{e35,,import} statements be absolute imports,
with a leading @code{.} character to indicate a relative import.  This part of the
PEP was not implemented for Python 2.4, but was completed for Python 2.5.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 328

@item PEP 328@footnote{https://www.python.org/dev/peps/pep-0328} - Imports: Multi-Line and Absolute/Relative

Written by Aahz.  Multi-line imports were implemented by Dima Dorfman.
@end table


@c ======================================================================

@node PEP 331 Locale-Independent Float/String Conversions,Other Language Changes<15>,PEP 328 Multi-line Imports,What’s New in Python 2 4
@anchor{whatsnew/2 4 pep-331-locale-independent-float-string-conversions}@anchor{fc1}
@subsection PEP 331: Locale-Independent Float/String Conversions


The @ref{a5,,locale} modules lets Python software select various conversions and
display conventions that are localized to a particular country or language.
However, the module was careful to not change the numeric locale because various
functions in Python’s implementation required that the numeric locale remain set
to the @code{'C'} locale.  Often this was because the code was using the C
library’s @code{atof()} function.

Not setting the numeric locale caused trouble for extensions that used third-party
C libraries, however, because they wouldn’t have the correct locale set.
The motivating example was GTK+, whose user interface widgets weren’t displaying
numbers in the current locale.

The solution described in the PEP is to add three new functions to the Python
API that perform ASCII-only conversions, ignoring the locale setting:


@itemize *

@item 
@code{PyOS_ascii_strtod(str, ptr)}  and @code{PyOS_ascii_atof(str, ptr)}
both convert a string to a C double.

@item 
@code{PyOS_ascii_formatd(buffer, buf_len, format, d)} converts a
double to an ASCII string.
@end itemize

The code for these functions came from the GLib library
(@indicateurl{https://developer.gnome.org/glib/stable/}), whose developers kindly
relicensed the relevant functions and donated them to the Python Software
Foundation.  The @ref{a5,,locale} module  can now change the numeric locale,
letting extensions such as GTK+  produce the correct results.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 331

@item PEP 331@footnote{https://www.python.org/dev/peps/pep-0331} - Locale-Independent Float/String Conversions

Written by Christian R. Reis, and implemented by Gustavo Carneiro.
@end table


@c ======================================================================

@node Other Language Changes<15>,New Improved and Deprecated Modules<3>,PEP 331 Locale-Independent Float/String Conversions,What’s New in Python 2 4
@anchor{whatsnew/2 4 other-language-changes}@anchor{fc2}
@subsection Other Language Changes


Here are all of the changes that Python 2.4 makes to the core Python language.


@itemize *

@item 
Decorators for functions and methods were added (
@geindex Python Enhancement Proposals; PEP 318
PEP 318@footnote{https://www.python.org/dev/peps/pep-0318}).

@item 
Built-in @ref{370,,set()} and @ref{439,,frozenset()} types were  added (
@geindex Python Enhancement Proposals; PEP 218
PEP 218@footnote{https://www.python.org/dev/peps/pep-0218}).
Other new built-ins include the @code{reversed(seq)} function (
@geindex Python Enhancement Proposals; PEP 322
PEP 322@footnote{https://www.python.org/dev/peps/pep-0322}).

@item 
Generator expressions were added (
@geindex Python Enhancement Proposals; PEP 289
PEP 289@footnote{https://www.python.org/dev/peps/pep-0289}).

@item 
Certain numeric expressions no longer return values restricted to 32 or 64
bits (
@geindex Python Enhancement Proposals; PEP 237
PEP 237@footnote{https://www.python.org/dev/peps/pep-0237}).

@item 
You can now put parentheses around the list of names in a @code{from module import
names} statement (
@geindex Python Enhancement Proposals; PEP 328
PEP 328@footnote{https://www.python.org/dev/peps/pep-0328}).

@item 
The @ref{fc3,,dict.update()} method now accepts the same argument forms as the
@ref{3b0,,dict} constructor.  This includes any mapping, any iterable of key/value
pairs, and keyword arguments. (Contributed by Raymond Hettinger.)

@item 
The string methods @code{ljust()}, @code{rjust()}, and @code{center()} now take
an optional argument for specifying a fill character other than a space.
(Contributed by Raymond Hettinger.)

@item 
Strings also gained an @code{rsplit()} method that works like the @code{split()}
method but splits from the end of the string.   (Contributed by Sean
Reifschneider.)

@example
>>> 'www.python.org'.split('.', 1)
['www', 'python.org']
'www.python.org'.rsplit('.', 1)
['www.python', 'org']
@end example

@item 
Three keyword parameters, @emph{cmp}, @emph{key}, and @emph{reverse}, were added to the
@code{sort()} method of lists. These parameters make some common usages of
@code{sort()} simpler. All of these parameters are optional.

For the @emph{cmp} parameter, the value should be a comparison function that takes
two parameters and returns -1, 0, or +1 depending on how the parameters compare.
This function will then be used to sort the list.  Previously this was the only
parameter that could be provided to @code{sort()}.

@emph{key} should be a single-parameter function that takes a list element and
returns a comparison key for the element.  The list is then sorted using the
comparison keys.  The following example sorts a list case-insensitively:

@example
>>> L = ['A', 'b', 'c', 'D']
>>> L.sort()                 # Case-sensitive sort
>>> L
['A', 'D', 'b', 'c']
>>> # Using 'key' parameter to sort list
>>> L.sort(key=lambda x: x.lower())
>>> L
['A', 'b', 'c', 'D']
>>> # Old-fashioned way
>>> L.sort(cmp=lambda x,y: cmp(x.lower(), y.lower()))
>>> L
['A', 'b', 'c', 'D']
@end example

The last example, which uses the @emph{cmp} parameter, is the old way to perform a
case-insensitive sort.  It works but is slower than using a @emph{key} parameter.
Using @emph{key} calls @code{lower()} method once for each element in the list while
using @emph{cmp} will call it twice for each comparison, so using @emph{key} saves on
invocations of the @code{lower()} method.

For simple key functions and comparison functions, it is often possible to avoid
a @ref{e42,,lambda} expression by using an unbound method instead.  For example,
the above case-insensitive sort is best written as:

@example
>>> L.sort(key=str.lower)
>>> L
['A', 'b', 'c', 'D']
@end example

Finally, the @emph{reverse} parameter takes a Boolean value.  If the value is true,
the list will be sorted into reverse order. Instead of @code{L.sort();
L.reverse()}, you can now write @code{L.sort(reverse=True)}.

The results of sorting are now guaranteed to be stable.  This means that two
entries with equal keys will be returned in the same order as they were input.
For example, you can sort a list of people by name, and then sort the list by
age, resulting in a list sorted by age where people with the same age are in
name-sorted order.

(All changes to @code{sort()} contributed by Raymond Hettinger.)

@item 
There is a new built-in function @code{sorted(iterable)} that works like the
in-place @ref{705,,list.sort()} method but can be used in expressions.  The
differences are:

@item 
the input may be any iterable;

@item 
a newly formed copy is sorted, leaving the original intact; and

@item 
the expression returns the new sorted copy

@example
>>> L = [9,7,8,3,2,4,1,6,5]
>>> [10+i for i in sorted(L)]       # usable in a list comprehension
[11, 12, 13, 14, 15, 16, 17, 18, 19]
>>> L                               # original is left unchanged
[9,7,8,3,2,4,1,6,5]
>>> sorted('Monty Python')          # any iterable may be an input
[' ', 'M', 'P', 'h', 'n', 'n', 'o', 'o', 't', 't', 'y', 'y']

>>> # List the contents of a dict sorted by key values
>>> colormap = dict(red=1, blue=2, green=3, black=4, yellow=5)
>>> for k, v in sorted(colormap.iteritems()):
...     print k, v
...
black 4
blue 2
green 3
red 1
yellow 5
@end example

(Contributed by Raymond Hettinger.)

@item 
Integer operations will no longer trigger an @code{OverflowWarning}. The
@code{OverflowWarning} warning will disappear in Python 2.5.

@item 
The interpreter gained a new switch, @ref{611,,-m}, that takes a name, searches
for the corresponding  module on @code{sys.path}, and runs the module as a script.
For example,  you can now run the Python profiler with @code{python -m profile}.
(Contributed by Nick Coghlan.)

@item 
The @code{eval(expr, globals, locals)} and @code{execfile(filename, globals,
locals)} functions and the @code{exec} statement now accept any mapping type
for the @emph{locals} parameter.  Previously this had to be a regular Python
dictionary.  (Contributed by Raymond Hettinger.)

@item 
The @ref{247,,zip()} built-in function and @code{itertools.izip()} now return an
empty list if called with no arguments. Previously they raised a
@ref{162,,TypeError} exception.  This makes them more suitable for use with variable
length argument lists:

@example
>>> def transpose(array):
...    return zip(*array)
...
>>> transpose([(1,2,3), (4,5,6)])
[(1, 4), (2, 5), (3, 6)]
>>> transpose([])
[]
@end example

(Contributed by Raymond Hettinger.)

@item 
Encountering a failure while importing a module no longer leaves a partially-initialized
module object in @code{sys.modules}.  The incomplete module object left
behind would fool further imports of the same module into succeeding, leading to
confusing errors.   (Fixed by Tim Peters.)

@item 
@ref{182,,None} is now a constant; code that binds a new value to  the name
@code{None} is now a syntax error. (Contributed by Raymond Hettinger.)
@end itemize

@c ======================================================================

@menu
* Optimizations: Optimizations<14>. 

@end menu

@node Optimizations<14>,,,Other Language Changes<15>
@anchor{whatsnew/2 4 optimizations}@anchor{fc4}
@subsubsection Optimizations



@itemize *

@item 
The inner loops for list and tuple slicing were optimized and now run about
one-third faster.  The inner loops for dictionaries were also optimized,
resulting in performance boosts for @code{keys()}, @code{values()}, @code{items()},
@code{iterkeys()}, @code{itervalues()}, and @code{iteritems()}. (Contributed by
Raymond Hettinger.)

@item 
The machinery for growing and shrinking lists was optimized for speed and for
space efficiency.  Appending and popping from lists now runs faster due to more
efficient code paths and less frequent use of the underlying system
@code{realloc()}.  List comprehensions also benefit.   @code{list.extend()} was
also optimized and no longer converts its argument into a temporary list before
extending the base list.  (Contributed by Raymond Hettinger.)

@item 
@ref{43a,,list()}, @ref{3e5,,tuple()}, @ref{2f4,,map()}, @ref{2f5,,filter()}, and @ref{247,,zip()} now
run several times faster with non-sequence arguments that supply a
@code{__len__()} method.  (Contributed by Raymond Hettinger.)

@item 
The methods @code{list.__getitem__()}, @code{dict.__getitem__()}, and
@code{dict.__contains__()} are now implemented as @code{method_descriptor}
objects rather than @code{wrapper_descriptor} objects.  This form of  access
doubles their performance and makes them more suitable for use as arguments to
functionals: @code{map(mydict.__getitem__, keylist)}. (Contributed by Raymond
Hettinger.)

@item 
Added a new opcode, @code{LIST_APPEND}, that simplifies the generated bytecode
for list comprehensions and speeds them up by about a third.  (Contributed by
Raymond Hettinger.)

@item 
The peephole bytecode optimizer has been improved to  produce shorter, faster
bytecode; remarkably, the resulting bytecode is  more readable.  (Enhanced by
Raymond Hettinger.)

@item 
String concatenations in statements of the form @code{s = s + "abc"} and @code{s +=
"abc"} are now performed more efficiently in certain circumstances.  This
optimization won’t be present in other Python implementations such as Jython, so
you shouldn’t rely on it; using the @code{join()} method of strings is still
recommended when you want to efficiently glue a large number of strings
together. (Contributed by Armin Rigo.)
@end itemize

The net result of the 2.4 optimizations is that Python 2.4 runs the pystone
benchmark around 5% faster than Python 2.3 and 35% faster than Python 2.2.
(pystone is not a particularly good benchmark, but it’s the most commonly used
measurement of Python’s performance.  Your own applications may show greater or
smaller benefits from Python 2.4.)

@c pystone is almost useless for comparing different versions of Python;
@c instead, it excels at predicting relative Python performance on different
@c machines.  So, this section would be more informative if it used other tools
@c such as pybench and parrotbench.  For a more application oriented benchmark,
@c try comparing the timings of test_decimal.py under 2.3 and 2.4.

@c ======================================================================

@node New Improved and Deprecated Modules<3>,Build and C API Changes<11>,Other Language Changes<15>,What’s New in Python 2 4
@anchor{whatsnew/2 4 new-improved-and-deprecated-modules}@anchor{fc5}
@subsection New, Improved, and Deprecated Modules


As usual, Python’s standard library received a number of enhancements and bug
fixes.  Here’s a partial list of the most notable changes, sorted alphabetically
by module name. Consult the @code{Misc/NEWS} file in the source tree for a more
complete list of changes, or look through the CVS logs for all the details.


@itemize *

@item 
The @ref{a,,asyncore} module’s @code{loop()} function now has a @emph{count} parameter
that lets you perform a limited number of passes through the polling loop.  The
default is still to loop forever.

@item 
The @ref{d,,base64} module now has more complete 
@geindex RFC; RFC 3548
RFC 3548@footnote{https://tools.ietf.org/html/rfc3548.html} support for Base64,
Base32, and Base16 encoding and decoding, including optional case folding and
optional alternative alphabets. (Contributed by Barry Warsaw.)

@item 
The @ref{10,,bisect} module now has an underlying C implementation for improved
performance. (Contributed by Dmitry Vasiliev.)

@item 
The CJKCodecs collections of East Asian codecs, maintained by Hye-Shik Chang,
was integrated into 2.4.   The new encodings are:

@item 
Chinese (PRC): gb2312, gbk, gb18030, big5hkscs, hz

@item 
Chinese (ROC): big5, cp950

@item 

@table @asis

@item Japanese: cp932, euc-jis-2004, euc-jp, euc-jisx0213, iso-2022-jp,

iso-2022-jp-1, iso-2022-jp-2, iso-2022-jp-3, iso-2022-jp-ext, iso-2022-jp-2004,
shift-jis, shift-jisx0213, shift-jis-2004
@end table

@item 
Korean: cp949, euc-kr, johab, iso-2022-kr

@item 
Some other new encodings were added: HP Roman8,  ISO_8859-11, ISO_8859-16,
PCTP-154, and TIS-620.

@item 
The UTF-8 and UTF-16 codecs now cope better with receiving partial input.
Previously the @code{StreamReader} class would try to read more data, making
it impossible to resume decoding from the stream.  The @code{read()} method will
now return as much data as it can and future calls will resume decoding where
previous ones left off.  (Implemented by Walter Dörwald.)

@item 
There is a new @ref{1c,,collections} module for  various specialized collection
datatypes.  Currently it contains just one type, @code{deque}, a double-ended
queue that supports efficiently adding and removing elements from either
end:

@example
>>> from collections import deque
>>> d = deque('ghi')        # make a new deque with three items
>>> d.append('j')           # add a new entry to the right side
>>> d.appendleft('f')       # add a new entry to the left side
>>> d                       # show the representation of the deque
deque(['f', 'g', 'h', 'i', 'j'])
>>> d.pop()                 # return and remove the rightmost item
'j'
>>> d.popleft()             # return and remove the leftmost item
'f'
>>> list(d)                 # list the contents of the deque
['g', 'h', 'i']
>>> 'h' in d                # search the deque
True
@end example

Several modules, such as the @code{Queue} and @ref{109,,threading} modules, now take
advantage of @ref{7d3,,collections.deque} for improved performance.  (Contributed
by Raymond Hettinger.)

@item 
The @code{ConfigParser} classes have been enhanced slightly. The @code{read()}
method now returns a list of the files that were successfully parsed, and the
@ref{370,,set()} method raises @ref{162,,TypeError} if passed a @emph{value} argument that
isn’t a string.   (Contributed by John Belmonte and David Goodger.)

@item 
The @ref{2a,,curses} module now supports the ncurses extension
@code{use_default_colors()}.  On platforms where the terminal supports
transparency, this makes it possible to use a transparent background.
(Contributed by Jörg Lehmann.)

@item 
The @ref{35,,difflib} module now includes an @code{HtmlDiff} class that creates
an HTML table showing a side by side comparison of two versions of a text.
(Contributed by Dan Gass.)

@item 
The @ref{64,,email} package was updated to version 3.0,  which dropped various
deprecated APIs and removes support for Python versions earlier than 2.3.  The
3.0 version of the package uses a new incremental parser for MIME messages,
available in the @code{email.FeedParser} module.  The new parser doesn’t require
reading the entire message into memory, and doesn’t raise exceptions if a
message is malformed; instead it records any problems in the  @code{defect}
attribute of the message.  (Developed by Anthony Baxter, Barry Warsaw, Thomas
Wouters, and others.)

@item 
The @ref{89,,heapq} module has been converted to C.  The resulting tenfold
improvement in speed makes the module suitable for handling high volumes of
data.  In addition, the module has two new functions @code{nlargest()} and
@code{nsmallest()} that use heaps to find the N largest or smallest values in a
dataset without the expense of a full sort.  (Contributed by Raymond Hettinger.)

@item 
The @code{httplib} module now contains constants for HTTP status codes defined
in various HTTP-related RFC documents.  Constants have names such as
@code{OK}, @code{CREATED}, @code{CONTINUE}, and
@code{MOVED_PERMANENTLY}; use pydoc to get a full list.  (Contributed by
Andrew Eland.)

@item 
The @ref{93,,imaplib} module now supports IMAP’s THREAD command (contributed by
Yves Dionne) and new @code{deleteacl()} and @code{myrights()} methods (contributed
by Arnaud Mazin).

@item 
The @ref{9f,,itertools} module gained a @code{groupby(iterable[, *func*])}
function. @emph{iterable} is something that can be iterated over to return a stream
of elements, and the optional @emph{func} parameter is a function that takes an
element and returns a key value; if omitted, the key is simply the element
itself.  @code{groupby()} then groups the elements into subsequences which have
matching values of the key, and returns a series of 2-tuples containing the key
value and an iterator over the subsequence.

Here’s an example to make this clearer.  The @emph{key} function simply returns
whether a number is even or odd, so the result of @code{groupby()} is to return
consecutive runs of odd or even numbers.

@example
>>> import itertools
>>> L = [2, 4, 6, 7, 8, 9, 11, 12, 14]
>>> for key_val, it in itertools.groupby(L, lambda x: x % 2):
...    print key_val, list(it)
...
0 [2, 4, 6]
1 [7]
0 [8]
1 [9, 11]
0 [12, 14]
>>>
@end example

@code{groupby()} is typically used with sorted input.  The logic for
@code{groupby()} is similar to the Unix @code{uniq} filter which makes it handy for
eliminating, counting, or identifying duplicate elements:

@example
>>> word = 'abracadabra'
>>> letters = sorted(word)   # Turn string into a sorted list of letters
>>> letters
['a', 'a', 'a', 'a', 'a', 'b', 'b', 'c', 'd', 'r', 'r']
>>> for k, g in itertools.groupby(letters):
...    print k, list(g)
...
a ['a', 'a', 'a', 'a', 'a']
b ['b', 'b']
c ['c']
d ['d']
r ['r', 'r']
>>> # List unique letters
>>> [k for k, g in groupby(letters)]
['a', 'b', 'c', 'd', 'r']
>>> # Count letter occurrences
>>> [(k, len(list(g))) for k, g in groupby(letters)]
[('a', 5), ('b', 2), ('c', 1), ('d', 1), ('r', 2)]
@end example

(Contributed by Hye-Shik Chang.)

@item 
@ref{9f,,itertools} also gained a function named @code{tee(iterator, N)} that
returns @emph{N} independent iterators that replicate @emph{iterator}.  If @emph{N} is omitted,
the default is 2.

@example
>>> L = [1,2,3]
>>> i1, i2 = itertools.tee(L)
>>> i1,i2
(<itertools.tee object at 0x402c2080>, <itertools.tee object at 0x402c2090>)
>>> list(i1)               # Run the first iterator to exhaustion
[1, 2, 3]
>>> list(i2)               # Run the second iterator to exhaustion
[1, 2, 3]
@end example

Note that @code{tee()} has to keep copies of the values returned  by the
iterator; in the worst case, it may need to keep all of them.   This should
therefore be used carefully if the leading iterator can run far ahead of the
trailing iterator in a long stream of inputs. If the separation is large, then
you might as well use  @ref{43a,,list()} instead.  When the iterators track closely
with one another, @code{tee()} is ideal.  Possible applications include
bookmarking, windowing, or lookahead iterators. (Contributed by Raymond
Hettinger.)

@item 
A number of functions were added to the @ref{a5,,locale}  module, such as
@code{bind_textdomain_codeset()} to specify a particular encoding and a family of
@code{l*gettext()} functions that return messages in the chosen encoding.
(Contributed by Gustavo Niemeyer.)

@item 
Some keyword arguments were added to the @ref{a6,,logging} package’s
@code{basicConfig()} function to simplify log configuration.  The default
behavior is to log messages to standard error, but various keyword arguments can
be specified to log to a particular file, change the logging format, or set the
logging level. For example:

@example
import logging
logging.basicConfig(filename='/var/log/application.log',
    level=0,  # Log all messages
    format='%(levelname):%(process):%(thread):%(message)')
@end example

Other additions to the @ref{a6,,logging} package include a @code{log(level, msg)}
convenience method, as well as a @code{TimedRotatingFileHandler} class that
rotates its log files at a timed interval.  The module already had
@code{RotatingFileHandler}, which rotated logs once the file exceeded a
certain size.  Both classes derive from a new @code{BaseRotatingHandler} class
that can be used to implement other rotating handlers.

(Changes implemented by Vinay Sajip.)

@item 
The @ref{ac,,marshal} module now shares interned strings on unpacking a  data
structure.  This may shrink the size of certain pickle strings, but the primary
effect is to make @code{.pyc} files significantly smaller. (Contributed by
Martin von Löwis.)

@item 
The @ref{bc,,nntplib} module’s @code{NNTP} class gained @code{description()} and
@code{descriptions()} methods to retrieve  newsgroup descriptions for a single
group or for a range of groups. (Contributed by Jürgen A. Erhard.)

@item 
Two new functions were added to the @ref{be,,operator} module,
@code{attrgetter(attr)} and @code{itemgetter(index)}. Both functions return
callables that take a single argument and return the corresponding attribute or
item; these callables make excellent data extractors when used with @ref{2f4,,map()}
or @ref{704,,sorted()}.  For example:

@example
>>> L = [('c', 2), ('d', 1), ('a', 4), ('b', 3)]
>>> map(operator.itemgetter(0), L)
['c', 'd', 'a', 'b']
>>> map(operator.itemgetter(1), L)
[2, 1, 4, 3]
>>> sorted(L, key=operator.itemgetter(1)) # Sort list by second tuple item
[('d', 1), ('c', 2), ('b', 3), ('a', 4)]
@end example

(Contributed by Raymond Hettinger.)

@item 
The @ref{bf,,optparse} module was updated in various ways.  The module now passes
its messages through @ref{fc6,,gettext.gettext()}, making it possible to
internationalize Optik’s help and error messages.  Help messages for options can
now include the string @code{'%default'}, which will be replaced by the option’s
default value.  (Contributed by Greg Ward.)

@item 
The long-term plan is to deprecate the @code{rfc822} module in some future
Python release in favor of the @ref{64,,email} package. To this end, the
@code{email.Utils.formatdate()} function has been changed to make it usable as a
replacement for @code{rfc822.formatdate()}.  You may want to write new e-mail
processing code with this in mind.  (Change implemented by Anthony Baxter.)

@item 
A new @code{urandom(n)} function was added to the @ref{c0,,os} module, returning
a string containing @emph{n} bytes of random data.  This function provides access to
platform-specific sources of randomness such as @code{/dev/urandom} on Linux or
the Windows CryptoAPI.  (Contributed by Trevor Perrin.)

@item 
Another new function: @code{os.path.lexists(path)}  returns true if the file
specified by @emph{path} exists, whether or not it’s a symbolic link.  This differs
from the existing @code{os.path.exists(path)} function, which returns false if
@emph{path} is a symlink that points to a destination that doesn’t exist.
(Contributed by Beni Cherniavsky.)

@item 
A new @code{getsid()} function was added to the @ref{cc,,posix} module that
underlies the @ref{c0,,os} module. (Contributed by J. Raynor.)

@item 
The @ref{cb,,poplib} module now supports POP over SSL.  (Contributed by Hector
Urtubia.)

@item 
The @ref{ce,,profile} module can now profile C extension functions. (Contributed
by Nick Bastin.)

@item 
The @ref{d7,,random} module has a new method called @code{getrandbits(N)} that
returns a long integer @emph{N} bits in length.  The existing @code{randrange()}
method now uses @code{getrandbits()} where appropriate, making generation of
arbitrarily large random numbers more efficient.  (Contributed by Raymond
Hettinger.)

@item 
The regular expression language accepted by the @ref{d8,,re} module was extended
with simple conditional expressions, written as @code{(?(group)A|B)}.  @emph{group} is
either a numeric group ID or a group name defined with @code{(?P<group>...)}
earlier in the expression.  If the specified group matched, the regular
expression pattern @emph{A} will be tested against the string; if the group didn’t
match, the pattern @emph{B} will be used instead. (Contributed by Gustavo Niemeyer.)

@item 
The @ref{d8,,re} module is also no longer recursive, thanks to a massive amount
of work by Gustavo Niemeyer.  In a recursive regular expression engine, certain
patterns result in a large amount of C stack space being consumed, and it was
possible to overflow the stack. For example, if you matched a 30000-byte string
of @code{a} characters against the expression @code{(a|b)+}, one stack frame was
consumed per character.  Python 2.3 tried to check for stack overflow and raise
a @ref{5af,,RuntimeError} exception, but certain patterns could sidestep the
checking and if you were unlucky Python could segfault. Python 2.4’s regular
expression engine can match this pattern without problems.

@item 
The @ref{e5,,signal} module now performs tighter error-checking on the parameters
to the @ref{cc9,,signal.signal()} function.  For example, you can’t set a handler on
the @code{SIGKILL} signal; previous versions of Python would quietly accept
this, but 2.4 will raise a @ref{5af,,RuntimeError} exception.

@item 
Two new functions were added to the @ref{ea,,socket} module. @code{socketpair()}
returns a pair of connected sockets and @code{getservbyport(port)} looks up the
service name for a given port number. (Contributed by Dave Cole and Barry
Warsaw.)

@item 
The @code{sys.exitfunc()} function has been deprecated.  Code should be using
the existing @ref{b,,atexit} module, which correctly handles calling multiple exit
functions.  Eventually @code{sys.exitfunc()} will become a purely internal
interface, accessed only by @ref{b,,atexit}.

@item 
The @ref{fb,,tarfile} module now generates GNU-format tar files by default.
(Contributed by Lars Gustäbel.)

@item 
The @ref{109,,threading} module now has an elegantly simple way to support
thread-local data.  The module contains a @code{local} class whose attribute
values are local to different threads.

@example
import threading

data = threading.local()
data.number = 42
data.url = ('www.python.org', 80)
@end example

Other threads can assign and retrieve their own values for the @code{number}
and @code{url} attributes.  You can subclass @code{local} to initialize
attributes or to add methods. (Contributed by Jim Fulton.)

@item 
The @ref{10b,,timeit} module now automatically disables periodic garbage
collection during the timing loop.  This change makes consecutive timings more
comparable.  (Contributed by Raymond Hettinger.)

@item 
The @ref{12f,,weakref} module now supports a wider variety of objects including
Python functions, class instances, sets, frozensets, deques, arrays, files,
sockets, and regular expression pattern objects. (Contributed by Raymond
Hettinger.)

@item 
The @code{xmlrpclib} module now supports a multi-call extension for
transmitting multiple XML-RPC calls in a single HTTP operation. (Contributed by
Brian Quinlan.)

@item 
The @code{mpz}, @code{rotor}, and @code{xreadlines} modules have  been
removed.
@end itemize

@c ======================================================================

@c whole new modules get described in subsections here

@c =====================

@menu
* cookielib:: 
* doctest: doctest<4>. 

@end menu

@node cookielib,doctest<4>,,New Improved and Deprecated Modules<3>
@anchor{whatsnew/2 4 cookielib}@anchor{fc7}
@subsubsection cookielib


The @code{cookielib} library supports client-side handling for HTTP cookies,
mirroring the @code{Cookie} module’s server-side cookie support. Cookies are
stored in cookie jars; the library transparently stores cookies offered by the
web server in the cookie jar, and fetches the cookie from the jar when
connecting to the server. As in web browsers, policy objects control whether
cookies are accepted or not.

In order to store cookies across sessions, two implementations of cookie jars
are provided: one that stores cookies in the Netscape format so applications can
use the Mozilla or Lynx cookie files, and one that stores cookies in the same
format as the Perl libwww library.

@code{urllib2} has been changed to interact with @code{cookielib}:
@code{HTTPCookieProcessor} manages a cookie jar that is used when accessing
URLs.

This module was contributed by John J. Lee.

@c ==================

@node doctest<4>,,cookielib,New Improved and Deprecated Modules<3>
@anchor{whatsnew/2 4 doctest}@anchor{fc8}
@subsubsection doctest


The @ref{63,,doctest} module underwent considerable refactoring thanks to Edward
Loper and Tim Peters.  Testing can still be as simple as running
@ref{fc9,,doctest.testmod()}, but the refactorings allow customizing the module’s
operation in various ways

The new @code{DocTestFinder} class extracts the tests from a given  object’s
docstrings:

@example
def f (x, y):
    """>>> f(2,2)
4
>>> f(3,2)
6
    """
    return x*y

finder = doctest.DocTestFinder()

# Get list of DocTest instances
tests = finder.find(f)
@end example

The new @code{DocTestRunner} class then runs individual tests and can produce
a summary of the results:

@example
runner = doctest.DocTestRunner()
for t in tests:
    tried, failed = runner.run(t)

runner.summarize(verbose=1)
@end example

The above example produces the following output:

@example
1 items passed all tests:
   2 tests in f
2 tests in 1 items.
2 passed and 0 failed.
Test passed.
@end example

@code{DocTestRunner} uses an instance of the @code{OutputChecker} class to
compare the expected output with the actual output.  This class takes a number
of different flags that customize its behaviour; ambitious users can also write
a completely new subclass of @code{OutputChecker}.

The default output checker provides a number of handy features. For example,
with the @ref{fca,,doctest.ELLIPSIS} option flag, an ellipsis (@code{...}) in the
expected output matches any substring,  making it easier to accommodate outputs
that vary in minor ways:

@example
def o (n):
    """>>> o(1)
<__main__.C instance at 0x...>
>>>
"""
@end example

Another special string, @code{<BLANKLINE>}, matches a blank line:

@example
def p (n):
    """>>> p(1)
<BLANKLINE>
>>>
"""
@end example

Another new capability is producing a diff-style display of the output by
specifying the @ref{fcb,,doctest.REPORT_UDIFF} (unified diffs),
@ref{fcc,,doctest.REPORT_CDIFF} (context diffs), or @ref{fcd,,doctest.REPORT_NDIFF}
(delta-style) option flags.  For example:

@example
def g (n):
    """>>> g(4)
here
is
a
lengthy
>>>"""
    L = 'here is a rather lengthy list of words'.split()
    for word in L[:n]:
        print word
@end example

Running the above function’s tests with @ref{fcb,,doctest.REPORT_UDIFF} specified,
you get the following output:

@example
**********************************************************************
File "t.py", line 15, in g
Failed example:
    g(4)
Differences (unified diff with -expected +actual):
    @@@@ -2,3 +2,3 @@@@
     is
     a
    -lengthy
    +rather
**********************************************************************
@end example

@c ======================================================================

@node Build and C API Changes<11>,Porting to Python 2 4,New Improved and Deprecated Modules<3>,What’s New in Python 2 4
@anchor{whatsnew/2 4 build-and-c-api-changes}@anchor{fce}
@subsection Build and C API Changes


Some of the changes to Python’s build process and to the C API are:


@itemize *

@item 
Three new convenience macros were added for common return values from
extension functions: @ref{fcf,,Py_RETURN_NONE}, @ref{fd0,,Py_RETURN_TRUE}, and
@ref{fd1,,Py_RETURN_FALSE}. (Contributed by Brett Cannon.)

@item 
Another new macro, @ref{fd2,,Py_CLEAR}(obj),  decreases the reference count of
@emph{obj} and sets @emph{obj} to the null pointer.  (Contributed by Jim Fulton.)

@item 
A new function, @code{PyTuple_Pack(N, obj1, obj2, ..., objN)}, constructs
tuples from a variable length argument list of Python objects.  (Contributed by
Raymond Hettinger.)

@item 
A new function, @code{PyDict_Contains(d, k)}, implements fast dictionary
lookups without masking exceptions raised during the look-up process.
(Contributed by Raymond Hettinger.)

@item 
The Py_IS_NAN(X) macro returns 1 if  its float or double argument
@emph{X} is a NaN.   (Contributed by Tim Peters.)

@item 
C code can avoid unnecessary locking by using the new
@ref{442,,PyEval_ThreadsInitialized()} function to tell  if any thread operations
have been performed.  If this function  returns false, no lock operations are
needed. (Contributed by Nick Coghlan.)

@item 
A new function, @ref{fd3,,PyArg_VaParseTupleAndKeywords()}, is the same as
@ref{863,,PyArg_ParseTupleAndKeywords()} but takes a  @code{va_list} instead of a
number of arguments. (Contributed by Greg Chapman.)

@item 
A new method flag, @code{METH_COEXISTS}, allows a function defined in slots
to co-exist with a @ref{fd4,,PyCFunction} having the same name.  This can halve
the access time for a method such as @code{set.__contains__()}.  (Contributed by
Raymond Hettinger.)

@item 
Python can now be built with additional profiling for the interpreter itself,
intended as an aid to people developing the Python core.  Providing
@code{--enable-profiling} to the @code{configure} script will let you
profile the interpreter with @code{gprof}, and providing the
@code{--with-tsc} switch enables profiling using the Pentium’s
Time-Stamp-Counter register.  Note that the @code{--with-tsc} switch is slightly
misnamed, because the profiling feature also works on the PowerPC platform,
though that processor architecture doesn’t call that register “the TSC
register”.  (Contributed by Jeremy Hylton.)

@item 
The @code{tracebackobject} type has been renamed to
@code{PyTracebackObject}.
@end itemize

@c ======================================================================

@menu
* Port-Specific Changes: Port-Specific Changes<2>. 

@end menu

@node Port-Specific Changes<2>,,,Build and C API Changes<11>
@anchor{whatsnew/2 4 port-specific-changes}@anchor{fd5}
@subsubsection Port-Specific Changes



@itemize *

@item 
The Windows port now builds under MSVC++ 7.1 as well as version 6.
(Contributed by Martin von Löwis.)
@end itemize

@c ======================================================================

@node Porting to Python 2 4,Acknowledgements<4>,Build and C API Changes<11>,What’s New in Python 2 4
@anchor{whatsnew/2 4 porting-to-python-2-4}@anchor{fd6}
@subsection Porting to Python 2.4


This section lists previously described changes that may require changes to your
code:


@itemize *

@item 
Left shifts and hexadecimal/octal constants that are too  large no longer
trigger a @ref{604,,FutureWarning} and return  a value limited to 32 or 64 bits;
instead they return a long integer.

@item 
Integer operations will no longer trigger an @code{OverflowWarning}. The
@code{OverflowWarning} warning will disappear in Python 2.5.

@item 
The @ref{247,,zip()} built-in function and @code{itertools.izip()} now return  an
empty list instead of raising a @ref{162,,TypeError} exception if called with no
arguments.

@item 
You can no longer compare the @code{date} and @ref{363,,datetime} instances
provided by the @ref{2f,,datetime} module.  Two  instances of different classes
will now always be unequal, and  relative comparisons (@code{<}, @code{>}) will raise
a @ref{162,,TypeError}.

@item 
@code{dircache.listdir()} now passes exceptions to the caller instead of
returning empty lists.

@item 
@code{LexicalHandler.startDTD()} used to receive the public and system IDs in
the wrong order.  This has been corrected; applications relying on the wrong
order need to be fixed.

@item 
@ref{fd7,,fcntl.ioctl()} now warns if the @emph{mutate}  argument is omitted and
relevant.

@item 
The @ref{fb,,tarfile} module now generates GNU-format tar files by default.

@item 
Encountering a failure while importing a module no longer leaves a
partially-initialized module object in @code{sys.modules}.

@item 
@ref{182,,None} is now a constant; code that binds a new value to  the name
@code{None} is now a syntax error.

@item 
The @code{signals.signal()} function now raises a @ref{5af,,RuntimeError} exception
for certain illegal values; previously these errors would pass silently.  For
example, you can no longer set a handler on the @code{SIGKILL} signal.
@end itemize

@c ======================================================================

@node Acknowledgements<4>,,Porting to Python 2 4,What’s New in Python 2 4
@anchor{whatsnew/2 4 acknowledgements}@anchor{fd8}@anchor{whatsnew/2 4 acks}@anchor{fd9}
@subsection Acknowledgements


The author would like to thank the following people for offering suggestions,
corrections and assistance with various drafts of this article: Koray Can,
Hye-Shik Chang, Michael Dyck, Raymond Hettinger, Brian Hurt, Hamish Lawson,
Fredrik Lundh, Sean Reifschneider, Sadruddin Rejeb.

@node What’s New in Python 2 3,What’s New in Python 2 2,What’s New in Python 2 4,What’s New in Python
@anchor{whatsnew/2 3 doc}@anchor{fda}@anchor{whatsnew/2 3 what-s-new-in-python-2-3}@anchor{fdb}
@section What’s New in Python 2.3


@*Author: 
A.M. Kuchling


@c $Id: whatsnew23.tex 54631 2007-03-31 11:58:36Z georg.brandl $

This article explains the new features in Python 2.3.  Python 2.3 was released
on July 29, 2003.

The main themes for Python 2.3 are polishing some of the features added in 2.2,
adding various small but useful enhancements to the core language, and expanding
the standard library.  The new object model introduced in the previous version
has benefited from 18 months of bugfixes and from optimization efforts that have
improved the performance of new-style classes.  A few new built-in functions
have been added such as @ref{50d,,sum()} and @ref{fdc,,enumerate()}.  The @ref{2fc,,in}
operator can now be used for substring searches (e.g. @code{"ab" in "abc"} returns
@ref{748,,True}).

Some of the many new library features include Boolean, set, heap, and date/time
data types, the ability to import modules from ZIP-format archives, metadata
support for the long-awaited Python catalog, an updated version of IDLE, and
modules for logging messages, wrapping text, parsing CSV files, processing
command-line options, using BerkeleyDB databases…  the list of new and
enhanced modules is lengthy.

This article doesn’t attempt to provide a complete specification of the new
features, but instead provides a convenient overview.  For full details, you
should refer to the documentation for Python 2.3, such as the Python Library
Reference and the Python Reference Manual.  If you want to understand the
complete implementation and design rationale, refer to the PEP for a particular
new feature.

@c ======================================================================

@menu
* PEP 218; A Standard Set Datatype: PEP 218 A Standard Set Datatype. 
* PEP 255; Simple Generators: PEP 255 Simple Generators. 
* PEP 263; Source Code Encodings: PEP 263 Source Code Encodings. 
* PEP 273; Importing Modules from ZIP Archives: PEP 273 Importing Modules from ZIP Archives. 
* PEP 277; Unicode file name support for Windows NT: PEP 277 Unicode file name support for Windows NT. 
* PEP 278; Universal Newline Support: PEP 278 Universal Newline Support. 
* PEP 279; enumerate(): PEP 279 enumerate. 
* PEP 282; The logging Package: PEP 282 The logging Package. 
* PEP 285; A Boolean Type: PEP 285 A Boolean Type. 
* PEP 293; Codec Error Handling Callbacks: PEP 293 Codec Error Handling Callbacks. 
* PEP 301; Package Index and Metadata for Distutils: PEP 301 Package Index and Metadata for Distutils. 
* PEP 302; New Import Hooks: PEP 302 New Import Hooks. 
* PEP 305; Comma-separated Files: PEP 305 Comma-separated Files. 
* PEP 307; Pickle Enhancements: PEP 307 Pickle Enhancements. 
* Extended Slices:: 
* Other Language Changes: Other Language Changes<16>. 
* New@comma{} Improved@comma{} and Deprecated Modules: New Improved and Deprecated Modules<4>. 
* Pymalloc; A Specialized Object Allocator: Pymalloc A Specialized Object Allocator. 
* Build and C API Changes: Build and C API Changes<12>. 
* Other Changes and Fixes: Other Changes and Fixes<2>. 
* Porting to Python 2.3: Porting to Python 2 3. 
* Acknowledgements: Acknowledgements<5>. 

@end menu

@node PEP 218 A Standard Set Datatype,PEP 255 Simple Generators,,What’s New in Python 2 3
@anchor{whatsnew/2 3 pep-218-a-standard-set-datatype}@anchor{fdd}
@subsection PEP 218: A Standard Set Datatype


The new @code{sets} module contains an implementation of a set datatype.  The
@code{Set} class is for mutable sets, sets that can have members added and
removed.  The @code{ImmutableSet} class is for sets that can’t be modified,
and instances of @code{ImmutableSet} can therefore be used as dictionary keys.
Sets are built on top of dictionaries, so the elements within a set must be
hashable.

Here’s a simple example:

@example
>>> import sets
>>> S = sets.Set([1,2,3])
>>> S
Set([1, 2, 3])
>>> 1 in S
True
>>> 0 in S
False
>>> S.add(5)
>>> S.remove(3)
>>> S
Set([1, 2, 5])
>>>
@end example

The union and intersection of sets can be computed with the @code{union()} and
@code{intersection()} methods; an alternative notation uses the bitwise operators
@code{&} and @code{|}. Mutable sets also have in-place versions of these methods,
@code{union_update()} and @code{intersection_update()}.

@example
>>> S1 = sets.Set([1,2,3])
>>> S2 = sets.Set([4,5,6])
>>> S1.union(S2)
Set([1, 2, 3, 4, 5, 6])
>>> S1 | S2                  # Alternative notation
Set([1, 2, 3, 4, 5, 6])
>>> S1.intersection(S2)
Set([])
>>> S1 & S2                  # Alternative notation
Set([])
>>> S1.union_update(S2)
>>> S1
Set([1, 2, 3, 4, 5, 6])
>>>
@end example

It’s also possible to take the symmetric difference of two sets.  This is the
set of all elements in the union that aren’t in the intersection.  Another way
of putting it is that the symmetric difference contains all elements that are in
exactly one set.  Again, there’s an alternative notation (@code{^}), and an
in-place version with the ungainly name @code{symmetric_difference_update()}.

@example
>>> S1 = sets.Set([1,2,3,4])
>>> S2 = sets.Set([3,4,5,6])
>>> S1.symmetric_difference(S2)
Set([1, 2, 5, 6])
>>> S1 ^ S2
Set([1, 2, 5, 6])
>>>
@end example

There are also @code{issubset()} and @code{issuperset()} methods for checking
whether one set is a subset or superset of another:

@example
>>> S1 = sets.Set([1,2,3])
>>> S2 = sets.Set([2,3])
>>> S2.issubset(S1)
True
>>> S1.issubset(S2)
False
>>> S1.issuperset(S2)
True
>>>
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 218

@item PEP 218@footnote{https://www.python.org/dev/peps/pep-0218} - Adding a Built-In Set Object Type

PEP written by Greg V. Wilson. Implemented by Greg V. Wilson, Alex Martelli, and
GvR.
@end table


@c ======================================================================

@node PEP 255 Simple Generators,PEP 263 Source Code Encodings,PEP 218 A Standard Set Datatype,What’s New in Python 2 3
@anchor{whatsnew/2 3 pep-255-simple-generators}@anchor{fde}@anchor{whatsnew/2 3 section-generators}@anchor{fdf}
@subsection PEP 255: Simple Generators


In Python 2.2, generators were added as an optional feature, to be enabled by a
@code{from __future__ import generators} directive.  In 2.3 generators no longer
need to be specially enabled, and are now always present; this means that
@ref{4c5,,yield} is now always a keyword.  The rest of this section is a copy of
the description of generators from the “What’s New in Python 2.2” document; if
you read it back when Python 2.2 came out, you can skip the rest of this
section.

You’re doubtless familiar with how function calls work in Python or C. When you
call a function, it gets a private namespace where its local variables are
created.  When the function reaches a @ref{4c6,,return} statement, the local
variables are destroyed and the resulting value is returned to the caller.  A
later call to the same function will get a fresh new set of local variables.
But, what if the local variables weren’t thrown away on exiting a function?
What if you could later resume the function where it left off?  This is what
generators provide; they can be thought of as resumable functions.

Here’s the simplest example of a generator function:

@example
def generate_ints(N):
    for i in range(N):
        yield i
@end example

A new keyword, @ref{4c5,,yield}, was introduced for generators.  Any function
containing a @code{yield} statement is a generator function; this is
detected by Python’s bytecode compiler which compiles the function specially as
a result.

When you call a generator function, it doesn’t return a single value; instead it
returns a generator object that supports the iterator protocol.  On executing
the @ref{4c5,,yield} statement, the generator outputs the value of @code{i},
similar to a @ref{4c6,,return} statement.  The big difference between
@code{yield} and a @code{return} statement is that on reaching a
@code{yield} the generator’s state of execution is suspended and local
variables are preserved.  On the next call to the generator’s @code{.next()}
method, the function will resume executing immediately after the
@code{yield} statement.  (For complicated reasons, the @code{yield}
statement isn’t allowed inside the @ref{f6d,,try} block of a
@code{try}…@code{finally} statement; read 
@geindex Python Enhancement Proposals; PEP 255
PEP 255@footnote{https://www.python.org/dev/peps/pep-0255} for a full
explanation of the interaction between @code{yield} and exceptions.)

Here’s a sample usage of the @code{generate_ints()} generator:

@example
>>> gen = generate_ints(3)
>>> gen
<generator object at 0x8117f90>
>>> gen.next()
0
>>> gen.next()
1
>>> gen.next()
2
>>> gen.next()
Traceback (most recent call last):
  File "stdin", line 1, in ?
  File "stdin", line 2, in generate_ints
StopIteration
@end example

You could equally write @code{for i in generate_ints(5)}, or @code{a,b,c =
generate_ints(3)}.

Inside a generator function, the @ref{4c6,,return} statement can only be used
without a value, and signals the end of the procession of values; afterwards the
generator cannot return any further values. @code{return} with a value, such
as @code{return 5}, is a syntax error inside a generator function.  The end of the
generator’s results can also be indicated by raising @ref{736,,StopIteration}
manually, or by just letting the flow of execution fall off the bottom of the
function.

You could achieve the effect of generators manually by writing your own class
and storing all the local variables of the generator as instance variables.  For
example, returning a list of integers could be done by setting @code{self.count} to
0, and having the @ref{252,,next()} method increment @code{self.count} and return it.
However, for a moderately complicated generator, writing a corresponding class
would be much messier. @code{Lib/test/test_generators.py} contains a number of
more interesting examples.  The simplest one implements an in-order traversal of
a tree using generators recursively.

@example
# A recursive generator that generates Tree leaves in in-order.
def inorder(t):
    if t:
        for x in inorder(t.left):
            yield x
        yield t.label
        for x in inorder(t.right):
            yield x
@end example

Two other examples in @code{Lib/test/test_generators.py} produce solutions for
the N-Queens problem (placing $N$ queens on an $NxN$ chess board so that no
queen threatens another) and the Knight’s Tour (a route that takes a knight to
every square of an $NxN$ chessboard without visiting any square twice).

The idea of generators comes from other programming languages, especially Icon
(@indicateurl{https://www.cs.arizona.edu/icon/}), where the idea of generators is central.  In
Icon, every expression and function call behaves like a generator.  One example
from “An Overview of the Icon Programming Language” at
@indicateurl{https://www.cs.arizona.edu/icon/docs/ipd266.htm} gives an idea of what this looks
like:

@example
sentence := "Store it in the neighboring harbor"
if (i := find("or", sentence)) > 5 then write(i)
@end example

In Icon the @code{find()} function returns the indexes at which the substring
“or” is found: 3, 23, 33.  In the @ref{2fb,,if} statement, @code{i} is first
assigned a value of 3, but 3 is less than 5, so the comparison fails, and Icon
retries it with the second value of 23.  23 is greater than 5, so the comparison
now succeeds, and the code prints the value 23 to the screen.

Python doesn’t go nearly as far as Icon in adopting generators as a central
concept.  Generators are considered part of the core Python language, but
learning or using them isn’t compulsory; if they don’t solve any problems that
you have, feel free to ignore them. One novel feature of Python’s interface as
compared to Icon’s is that a generator’s state is represented as a concrete
object (the iterator) that can be passed around to other functions or stored in
a data structure.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 255

@item PEP 255@footnote{https://www.python.org/dev/peps/pep-0255} - Simple Generators

Written by Neil Schemenauer, Tim Peters, Magnus Lie Hetland.  Implemented mostly
by Neil Schemenauer and Tim Peters, with other fixes from the Python Labs crew.
@end table


@c ======================================================================

@node PEP 263 Source Code Encodings,PEP 273 Importing Modules from ZIP Archives,PEP 255 Simple Generators,What’s New in Python 2 3
@anchor{whatsnew/2 3 pep-263-source-code-encodings}@anchor{fe0}@anchor{whatsnew/2 3 section-encodings}@anchor{fe1}
@subsection PEP 263: Source Code Encodings


Python source files can now be declared as being in different character set
encodings.  Encodings are declared by including a specially formatted comment in
the first or second line of the source file.  For example, a UTF-8 file can be
declared with:

@example
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
@end example

Without such an encoding declaration, the default encoding used is 7-bit ASCII.
Executing or importing modules that contain string literals with 8-bit
characters and have no encoding declaration will result in a
@ref{1b2,,DeprecationWarning} being signalled by Python 2.3; in 2.4 this will be a
syntax error.

The encoding declaration only affects Unicode string literals, which will be
converted to Unicode using the specified encoding.  Note that Python identifiers
are still restricted to ASCII characters, so you can’t have variable names that
use characters outside of the usual alphanumerics.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 263

@item PEP 263@footnote{https://www.python.org/dev/peps/pep-0263} - Defining Python Source Code Encodings

Written by Marc-André Lemburg and Martin von Löwis; implemented by Suzuki Hisao
and Martin von Löwis.
@end table


@c ======================================================================

@node PEP 273 Importing Modules from ZIP Archives,PEP 277 Unicode file name support for Windows NT,PEP 263 Source Code Encodings,What’s New in Python 2 3
@anchor{whatsnew/2 3 pep-273-importing-modules-from-zip-archives}@anchor{fe2}
@subsection PEP 273: Importing Modules from ZIP Archives


The new @ref{14a,,zipimport} module adds support for importing modules from a
ZIP-format archive.  You don’t need to import the module explicitly; it will be
automatically imported if a ZIP archive’s filename is added to @code{sys.path}.
For example:

@example
amk@@nyman:~/src/python$ unzip -l /tmp/example.zip
Archive:  /tmp/example.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
     8467  11-26-02 22:30   jwzthreading.py
 --------                   -------
     8467                   1 file
amk@@nyman:~/src/python$ ./python
Python 2.3 (#1, Aug 1 2003, 19:54:32)
>>> import sys
>>> sys.path.insert(0, '/tmp/example.zip')  # Add .zip file to front of path
>>> import jwzthreading
>>> jwzthreading.__file__
'/tmp/example.zip/jwzthreading.py'
>>>
@end example

An entry in @code{sys.path} can now be the filename of a ZIP archive. The ZIP
archive can contain any kind of files, but only files named @code{*.py},
@code{*.pyc}, or @code{*.pyo} can be imported.  If an archive only contains
@code{*.py} files, Python will not attempt to modify the archive by adding the
corresponding @code{*.pyc} file, meaning that if a ZIP archive doesn’t contain
@code{*.pyc} files, importing may be rather slow.

A path within the archive can also be specified to only import from a
subdirectory; for example, the path @code{/tmp/example.zip/lib/} would only
import from the @code{lib/} subdirectory within the archive.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 273

@item PEP 273@footnote{https://www.python.org/dev/peps/pep-0273} - Import Modules from Zip Archives

Written by James C. Ahlstrom,  who also provided an implementation. Python 2.3
follows the specification in 
@geindex Python Enhancement Proposals; PEP 273
PEP 273@footnote{https://www.python.org/dev/peps/pep-0273},  but uses an implementation written by
Just van Rossum  that uses the import hooks described in 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302}. See section
@ref{fe3,,PEP 302; New Import Hooks} for a description of the new import hooks.
@end table


@c ======================================================================

@node PEP 277 Unicode file name support for Windows NT,PEP 278 Universal Newline Support,PEP 273 Importing Modules from ZIP Archives,What’s New in Python 2 3
@anchor{whatsnew/2 3 pep-277-unicode-file-name-support-for-windows-nt}@anchor{fe4}
@subsection PEP 277: Unicode file name support for Windows NT


On Windows NT, 2000, and XP, the system stores file names as Unicode strings.
Traditionally, Python has represented file names as byte strings, which is
inadequate because it renders some file names inaccessible.

Python now allows using arbitrary Unicode strings (within the limitations of the
file system) for all functions that expect file names, most notably the
@ref{1c8,,open()} built-in function. If a Unicode string is passed to
@ref{c8d,,os.listdir()}, Python now returns a list of Unicode strings.  A new
function, @code{os.getcwdu()}, returns the current directory as a Unicode string.

Byte strings still work as file names, and on Windows Python will transparently
convert them to Unicode using the @code{mbcs} encoding.

Other systems also allow Unicode strings as file names but convert them to byte
strings before passing them to the system, which can cause a @ref{e49,,UnicodeError}
to be raised. Applications can test whether arbitrary Unicode strings are
supported as file names by checking @ref{fe5,,os.path.supports_unicode_filenames},
a Boolean value.

Under MacOS, @ref{c8d,,os.listdir()} may now return Unicode filenames.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 277

@item PEP 277@footnote{https://www.python.org/dev/peps/pep-0277} - Unicode file name support for Windows NT

Written by Neil Hodgson; implemented by Neil Hodgson, Martin von Löwis, and Mark
Hammond.
@end table


@c ======================================================================

@geindex universal newlines; What's new

@node PEP 278 Universal Newline Support,PEP 279 enumerate,PEP 277 Unicode file name support for Windows NT,What’s New in Python 2 3
@anchor{whatsnew/2 3 pep-278-universal-newline-support}@anchor{fe6}
@subsection PEP 278: Universal Newline Support


The three major operating systems used today are Microsoft Windows, Apple’s
Macintosh OS, and the various Unix derivatives.  A minor irritation of
cross-platform work  is that these three platforms all use different characters to
mark the ends of lines in text files.  Unix uses the linefeed (ASCII character
10), MacOS uses the carriage return (ASCII character 13), and Windows uses a
two-character sequence of a carriage return plus a newline.

Python’s file objects can now support end of line conventions other than the
one followed by the platform on which Python is running. Opening a file with
the mode @code{'U'} or @code{'rU'} will open a file for reading in @ref{88a,,universal newlines} mode.  All three line ending conventions will be translated to a
@code{'\n'} in the strings returned by the various file methods such as
@code{read()} and @ref{d9,,readline()}.

Universal newline support is also used when importing modules and when executing
a file with the @code{execfile()} function.  This means that Python modules can
be shared between all three operating systems without needing to convert the
line-endings.

This feature can be disabled when compiling Python by specifying the
@code{--without-universal-newlines} switch when running Python’s
@code{configure} script.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 278

@item PEP 278@footnote{https://www.python.org/dev/peps/pep-0278} - Universal Newline Support

Written and implemented by Jack Jansen.
@end table


@c ======================================================================

@node PEP 279 enumerate,PEP 282 The logging Package,PEP 278 Universal Newline Support,What’s New in Python 2 3
@anchor{whatsnew/2 3 pep-279-enumerate}@anchor{fe7}@anchor{whatsnew/2 3 section-enumerate}@anchor{fe8}
@subsection PEP 279: enumerate()


A new built-in function, @ref{fdc,,enumerate()}, will make certain loops a bit
clearer.  @code{enumerate(thing)}, where @emph{thing} is either an iterator or a
sequence, returns an iterator that will return @code{(0, thing[0])}, @code{(1,
thing[1])}, @code{(2, thing[2])}, and so forth.

A common idiom to change every element of a list looks like this:

@example
for i in range(len(L)):
    item = L[i]
    # ... compute some result based on item ...
    L[i] = result
@end example

This can be rewritten using @ref{fdc,,enumerate()} as:

@example
for i, item in enumerate(L):
    # ... compute some result based on item ...
    L[i] = result
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 279

@item PEP 279@footnote{https://www.python.org/dev/peps/pep-0279} - The enumerate() built-in function

Written and implemented by Raymond D. Hettinger.
@end table


@c ======================================================================

@node PEP 282 The logging Package,PEP 285 A Boolean Type,PEP 279 enumerate,What’s New in Python 2 3
@anchor{whatsnew/2 3 pep-282-the-logging-package}@anchor{fe9}
@subsection PEP 282: The logging Package


A standard package for writing logs, @ref{a6,,logging}, has been added to Python
2.3.  It provides a powerful and flexible mechanism for generating logging
output which can then be filtered and processed in various ways.  A
configuration file written in a standard format can be used to control the
logging behavior of a program.  Python includes handlers that will write log
records to standard error or to a file or socket, send them to the system log,
or even e-mail them to a particular address; of course, it’s also possible to
write your own handler classes.

The @code{Logger} class is the primary class. Most application code will deal
with one or more @code{Logger} objects, each one used by a particular
subsystem of the application. Each @code{Logger} is identified by a name, and
names are organized into a hierarchy using @code{.}  as the component separator.
For example, you might have @code{Logger} instances named @code{server},
@code{server.auth} and @code{server.network}.  The latter two instances are below
@code{server} in the hierarchy.  This means that if you turn up the verbosity for
@code{server} or direct @code{server} messages to a different handler, the changes
will also apply to records logged to @code{server.auth} and @code{server.network}.
There’s also a root @code{Logger} that’s the parent of all other loggers.

For simple uses, the @ref{a6,,logging} package contains some convenience functions
that always use the root log:

@example
import logging

logging.debug('Debugging information')
logging.info('Informational message')
logging.warning('Warning:config file %s not found', 'server.conf')
logging.error('Error occurred')
logging.critical('Critical error -- shutting down')
@end example

This produces the following output:

@example
WARNING:root:Warning:config file server.conf not found
ERROR:root:Error occurred
CRITICAL:root:Critical error -- shutting down
@end example

In the default configuration, informational and debugging messages are
suppressed and the output is sent to standard error.  You can enable the display
of informational and debugging messages by calling the @code{setLevel()} method
on the root logger.

Notice the @code{warning()} call’s use of string formatting operators; all of the
functions for logging messages take the arguments @code{(msg, arg1, arg2, ...)} and
log the string resulting from @code{msg % (arg1, arg2, ...)}.

There’s also an @code{exception()} function that records the most recent
traceback.  Any of the other functions will also record the traceback if you
specify a true value for the keyword argument @emph{exc_info}.

@example
def f():
    try:    1/0
    except: logging.exception('Problem recorded')

f()
@end example

This produces the following output:

@example
ERROR:root:Problem recorded
Traceback (most recent call last):
  File "t.py", line 6, in f
    1/0
ZeroDivisionError: integer division or modulo by zero
@end example

Slightly more advanced programs will use a logger other than the root logger.
The @code{getLogger(name)} function is used to get a particular log, creating
it if it doesn’t exist yet. @code{getLogger(None)} returns the root logger.

@example
log = logging.getLogger('server')
 ...
log.info('Listening on port %i', port)
 ...
log.critical('Disk full')
 ...
@end example

Log records are usually propagated up the hierarchy, so a message logged to
@code{server.auth} is also seen by @code{server} and @code{root}, but a @code{Logger}
can prevent this by setting its @code{propagate} attribute to @ref{659,,False}.

There are more classes provided by the @ref{a6,,logging} package that can be
customized.  When a @code{Logger} instance is told to log a message, it
creates a @code{LogRecord} instance that is sent to any number of different
@code{Handler} instances.  Loggers and handlers can also have an attached list
of filters, and each filter can cause the @code{LogRecord} to be ignored or
can modify the record before passing it along.  When they’re finally output,
@code{LogRecord} instances are converted to text by a @code{Formatter}
class.  All of these classes can be replaced by your own specially-written
classes.

With all of these features the @ref{a6,,logging} package should provide enough
flexibility for even the most complicated applications.  This is only an
incomplete overview of its features, so please see the package’s reference
documentation for all of the details.  Reading 
@geindex Python Enhancement Proposals; PEP 282
PEP 282@footnote{https://www.python.org/dev/peps/pep-0282} will also be helpful.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 282

@item PEP 282@footnote{https://www.python.org/dev/peps/pep-0282} - A Logging System

Written by Vinay Sajip and Trent Mick; implemented by Vinay Sajip.
@end table


@c ======================================================================

@node PEP 285 A Boolean Type,PEP 293 Codec Error Handling Callbacks,PEP 282 The logging Package,What’s New in Python 2 3
@anchor{whatsnew/2 3 pep-285-a-boolean-type}@anchor{fea}@anchor{whatsnew/2 3 section-bool}@anchor{feb}
@subsection PEP 285: A Boolean Type


A Boolean type was added to Python 2.3.  Two new constants were added to the
@code{__builtin__} module, @ref{748,,True} and @ref{659,,False}.  (@ref{748,,True} and
@ref{659,,False} constants were added to the built-ins in Python 2.2.1, but the
2.2.1 versions are simply set to integer values of 1 and 0 and aren’t a
different type.)

The type object for this new type is named @ref{2f7,,bool}; the constructor for it
takes any Python value and converts it to @ref{748,,True} or @ref{659,,False}.

@example
>>> bool(1)
True
>>> bool(0)
False
>>> bool([])
False
>>> bool( (1,) )
True
@end example

Most of the standard library modules and built-in functions have been changed to
return Booleans.

@example
>>> obj = []
>>> hasattr(obj, 'append')
True
>>> isinstance(obj, list)
True
>>> isinstance(obj, tuple)
False
@end example

Python’s Booleans were added with the primary goal of making code clearer.  For
example, if you’re reading a function and encounter the statement @code{return 1},
you might wonder whether the @code{1} represents a Boolean truth value, an index,
or a coefficient that multiplies some other quantity.  If the statement is
@code{return True}, however, the meaning of the return value is quite clear.

Python’s Booleans were @emph{not} added for the sake of strict type-checking.  A very
strict language such as Pascal would also prevent you performing arithmetic with
Booleans, and would require that the expression in an @ref{2fb,,if} statement
always evaluate to a Boolean result.  Python is not this strict and never will
be, as 
@geindex Python Enhancement Proposals; PEP 285
PEP 285@footnote{https://www.python.org/dev/peps/pep-0285} explicitly says.  This means you can still use any expression
in an @code{if} statement, even ones that evaluate to a list or tuple or
some random object.  The Boolean type is a subclass of the @ref{1b1,,int} class so
that arithmetic using a Boolean still works.

@example
>>> True + 1
2
>>> False + 1
1
>>> False * 75
0
>>> True * 75
75
@end example

To sum up @ref{748,,True} and @ref{659,,False} in a sentence: they’re alternative
ways to spell the integer values 1 and 0, with the single difference that
@ref{2ef,,str()} and @ref{27d,,repr()} return the strings @code{'True'} and @code{'False'}
instead of @code{'1'} and @code{'0'}.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 285

@item PEP 285@footnote{https://www.python.org/dev/peps/pep-0285} - Adding a bool type

Written and implemented by GvR.
@end table


@c ======================================================================

@node PEP 293 Codec Error Handling Callbacks,PEP 301 Package Index and Metadata for Distutils,PEP 285 A Boolean Type,What’s New in Python 2 3
@anchor{whatsnew/2 3 pep-293-codec-error-handling-callbacks}@anchor{fec}
@subsection PEP 293: Codec Error Handling Callbacks


When encoding a Unicode string into a byte string, unencodable characters may be
encountered.  So far, Python has allowed specifying the error processing as
either “strict” (raising @ref{e49,,UnicodeError}), “ignore” (skipping the
character), or “replace” (using a question mark in the output string), with
“strict” being the default behavior. It may be desirable to specify alternative
processing of such errors, such as inserting an XML character reference or HTML
entity reference into the converted string.

Python now has a flexible framework to add different processing strategies.  New
error handlers can be added with @ref{fed,,codecs.register_error()}, and codecs then
can access the error handler with @ref{fee,,codecs.lookup_error()}. An equivalent C
API has been added for codecs written in C. The error handler gets the necessary
state information such as the string being converted, the position in the string
where the error was detected, and the target encoding.  The handler can then
either raise an exception or return a replacement string.

Two additional error handlers have been implemented using this framework:
“backslashreplace” uses Python backslash quoting to represent unencodable
characters and “xmlcharrefreplace” emits XML character references.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 293

@item PEP 293@footnote{https://www.python.org/dev/peps/pep-0293} - Codec Error Handling Callbacks

Written and implemented by Walter Dörwald.
@end table


@c ======================================================================

@node PEP 301 Package Index and Metadata for Distutils,PEP 302 New Import Hooks,PEP 293 Codec Error Handling Callbacks,What’s New in Python 2 3
@anchor{whatsnew/2 3 pep-301-package-index-and-metadata-for-distutils}@anchor{fef}@anchor{whatsnew/2 3 section-pep301}@anchor{ff0}
@subsection PEP 301: Package Index and Metadata for Distutils


Support for the long-requested Python catalog makes its first appearance in 2.3.

The heart of the catalog is the new Distutils @code{register} command.
Running @code{python setup.py register} will collect the metadata describing a
package, such as its name, version, maintainer, description, &c., and send it to
a central catalog server.  The resulting catalog is available from
@indicateurl{https://pypi.org}.

To make the catalog a bit more useful, a new optional @emph{classifiers} keyword
argument has been added to the Distutils @code{setup()} function.  A list of
Trove@footnote{http://catb.org/~esr/trove/}-style strings can be supplied to help
classify the software.

Here’s an example @code{setup.py} with classifiers, written to be compatible
with older versions of the Distutils:

@example
from distutils import core
kw = @{'name': "Quixote",
      'version': "0.5.1",
      'description': "A highly Pythonic Web application framework",
      # ...
      @}

if (hasattr(core, 'setup_keywords') and
    'classifiers' in core.setup_keywords):
    kw['classifiers'] = \
        ['Topic :: Internet :: WWW/HTTP :: Dynamic Content',
         'Environment :: No Input/Output (Daemon)',
         'Intended Audience :: Developers'],

core.setup(**kw)
@end example

The full list of classifiers can be obtained by running  @code{python setup.py
register --list-classifiers}.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 301

@item PEP 301@footnote{https://www.python.org/dev/peps/pep-0301} - Package Index and Metadata for Distutils

Written and implemented by Richard Jones.
@end table


@c ======================================================================

@node PEP 302 New Import Hooks,PEP 305 Comma-separated Files,PEP 301 Package Index and Metadata for Distutils,What’s New in Python 2 3
@anchor{whatsnew/2 3 pep-302-new-import-hooks}@anchor{ff1}@anchor{whatsnew/2 3 section-pep302}@anchor{fe3}
@subsection PEP 302: New Import Hooks


While it’s been possible to write custom import hooks ever since the
@code{ihooks} module was introduced in Python 1.3, no one has ever been really
happy with it because writing new import hooks is difficult and messy.  There
have been various proposed alternatives such as the @code{imputil} and @code{iu}
modules, but none of them has ever gained much acceptance, and none of them were
easily usable from C code.

@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} borrows ideas from its predecessors, especially from Gordon
McMillan’s @code{iu} module.  Three new items  are added to the @ref{f7,,sys}
module:


@itemize *

@item 
@code{sys.path_hooks} is a list of callable objects; most  often they’ll be
classes.  Each callable takes a string containing a path and either returns an
importer object that will handle imports from this path or raises an
@ref{3b8,,ImportError} exception if it can’t handle this path.

@item 
@code{sys.path_importer_cache} caches importer objects for each path, so
@code{sys.path_hooks} will only need to be traversed once for each path.

@item 
@code{sys.meta_path} is a list of importer objects that will be traversed before
@code{sys.path} is checked.  This list is initially empty, but user code can add
objects to it.  Additional built-in and frozen modules can be imported by an
object added to this list.
@end itemize

Importer objects must have a single method, @code{find_module(fullname,
path=None)}.  @emph{fullname} will be a module or package name, e.g. @code{string} or
@code{distutils.core}.  @code{find_module()} must return a loader object that has a
single method, @code{load_module(fullname)}, that creates and returns the
corresponding module object.

Pseudo-code for Python’s new import logic, therefore, looks something like this
(simplified a bit; see 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} for the full details):

@example
for mp in sys.meta_path:
    loader = mp(fullname)
    if loader is not None:
        <module> = loader.load_module(fullname)

for path in sys.path:
    for hook in sys.path_hooks:
        try:
            importer = hook(path)
        except ImportError:
            # ImportError, so try the other path hooks
            pass
        else:
            loader = importer.find_module(fullname)
            <module> = loader.load_module(fullname)

# Not found!
raise ImportError
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 302

@item PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} - New Import Hooks

Written by Just van Rossum and Paul Moore. Implemented by Just van Rossum.
@end table


@c ======================================================================

@node PEP 305 Comma-separated Files,PEP 307 Pickle Enhancements,PEP 302 New Import Hooks,What’s New in Python 2 3
@anchor{whatsnew/2 3 pep-305-comma-separated-files}@anchor{ff2}@anchor{whatsnew/2 3 section-pep305}@anchor{ff3}
@subsection PEP 305: Comma-separated Files


Comma-separated files are a format frequently used for exporting data from
databases and spreadsheets.  Python 2.3 adds a parser for comma-separated files.

Comma-separated format is deceptively simple at first glance:

@example
Costs,150,200,3.95
@end example

Read a line and call @code{line.split(',')}: what could be simpler? But toss in
string data that can contain commas, and things get more complicated:

@example
"Costs",150,200,3.95,"Includes taxes, shipping, and sundry items"
@end example

A big ugly regular expression can parse this, but using the new  @ref{28,,csv}
package is much simpler:

@example
import csv

input = open('datafile', 'rb')
reader = csv.reader(input)
for line in reader:
    print line
@end example

The @code{reader()} function takes a number of different options. The field
separator isn’t limited to the comma and can be changed to any character, and so
can the quoting and line-ending characters.

Different dialects of comma-separated files can be defined and registered;
currently there are two dialects, both used by Microsoft Excel. A separate
@ref{ff4,,csv.writer} class will generate comma-separated files from a succession
of tuples or lists, quoting strings that contain the delimiter.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 305

@item PEP 305@footnote{https://www.python.org/dev/peps/pep-0305} - CSV File API

Written and implemented  by Kevin Altis, Dave Cole, Andrew McNamara, Skip
Montanaro, Cliff Wells.
@end table


@c ======================================================================

@node PEP 307 Pickle Enhancements,Extended Slices,PEP 305 Comma-separated Files,What’s New in Python 2 3
@anchor{whatsnew/2 3 pep-307-pickle-enhancements}@anchor{ff5}@anchor{whatsnew/2 3 section-pep307}@anchor{ff6}
@subsection PEP 307: Pickle Enhancements


The @ref{c5,,pickle} and @code{cPickle} modules received some attention during the
2.3 development cycle.  In 2.2, new-style classes could be pickled without
difficulty, but they weren’t pickled very compactly; 
@geindex Python Enhancement Proposals; PEP 307
PEP 307@footnote{https://www.python.org/dev/peps/pep-0307} quotes a trivial
example where a new-style class results in a pickled string three times longer
than that for a classic class.

The solution was to invent a new pickle protocol.  The @ref{ff7,,pickle.dumps()}
function has supported a text-or-binary flag  for a long time.  In 2.3, this
flag is redefined from a Boolean to an integer: 0 is the old text-mode pickle
format, 1 is the old binary format, and now 2 is a new 2.3-specific format.  A
new constant, @ref{ff8,,pickle.HIGHEST_PROTOCOL}, can be used to select the
fanciest protocol available.

Unpickling is no longer considered a safe operation.  2.2’s @ref{c5,,pickle}
provided hooks for trying to prevent unsafe classes from being unpickled
(specifically, a @code{__safe_for_unpickling__} attribute), but none of this
code was ever audited and therefore it’s all been ripped out in 2.3.  You should
not unpickle untrusted data in any version of Python.

To reduce the pickling overhead for new-style classes, a new interface for
customizing pickling was added using three special methods:
@code{__getstate__()}, @code{__setstate__()}, and @code{__getnewargs__()}.  Consult
@geindex Python Enhancement Proposals; PEP 307
PEP 307@footnote{https://www.python.org/dev/peps/pep-0307} for the full semantics  of these methods.

As a way to compress pickles yet further, it’s now possible to use integer codes
instead of long strings to identify pickled classes. The Python Software
Foundation will maintain a list of standardized codes; there’s also a range of
codes for private use.  Currently no codes have been specified.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 307

@item PEP 307@footnote{https://www.python.org/dev/peps/pep-0307} - Extensions to the pickle protocol

Written and implemented  by Guido van Rossum and Tim Peters.
@end table


@c ======================================================================

@node Extended Slices,Other Language Changes<16>,PEP 307 Pickle Enhancements,What’s New in Python 2 3
@anchor{whatsnew/2 3 extended-slices}@anchor{ff9}@anchor{whatsnew/2 3 section-slices}@anchor{ffa}
@subsection Extended Slices


Ever since Python 1.4, the slicing syntax has supported an optional third “step”
or “stride” argument.  For example, these are all legal Python syntax:
@code{L[1:10:2]}, @code{L[:-1:1]}, @code{L[::-1]}.  This was added to Python at the
request of the developers of Numerical Python, which uses the third argument
extensively.  However, Python’s built-in list, tuple, and string sequence types
have never supported this feature, raising a @ref{162,,TypeError} if you tried it.
Michael Hudson contributed a patch to fix this shortcoming.

For example, you can now easily extract the elements of a list that have even
indexes:

@example
>>> L = range(10)
>>> L[::2]
[0, 2, 4, 6, 8]
@end example

Negative values also work to make a copy of the same list in reverse order:

@example
>>> L[::-1]
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
@end example

This also works for tuples, arrays, and strings:

@example
>>> s='abcd'
>>> s[::2]
'ac'
>>> s[::-1]
'dcba'
@end example

If you have a mutable sequence such as a list or an array you can assign to or
delete an extended slice, but there are some differences between assignment to
extended and regular slices.  Assignment to a regular slice can be used to
change the length of the sequence:

@example
>>> a = range(3)
>>> a
[0, 1, 2]
>>> a[1:3] = [4, 5, 6]
>>> a
[0, 4, 5, 6]
@end example

Extended slices aren’t this flexible.  When assigning to an extended slice, the
list on the right hand side of the statement must contain the same number of
items as the slice it is replacing:

@example
>>> a = range(4)
>>> a
[0, 1, 2, 3]
>>> a[::2]
[0, 2]
>>> a[::2] = [0, -1]
>>> a
[0, 1, -1, 3]
>>> a[::2] = [0,1,2]
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
ValueError: attempt to assign sequence of size 3 to extended slice of size 2
@end example

Deletion is more straightforward:

@example
>>> a = range(4)
>>> a
[0, 1, 2, 3]
>>> a[::2]
[0, 2]
>>> del a[::2]
>>> a
[1, 3]
@end example

One can also now pass slice objects to the @code{__getitem__()} methods of the
built-in sequences:

@example
>>> range(10).__getitem__(slice(0, 5, 2))
[0, 2, 4]
@end example

Or use slice objects directly in subscripts:

@example
>>> range(10)[slice(0, 5, 2)]
[0, 2, 4]
@end example

To simplify implementing sequences that support extended slicing, slice objects
now have a method @code{indices(length)} which, given the length of a sequence,
returns a @code{(start, stop, step)} tuple that can be passed directly to
@ref{438,,range()}. @code{indices()} handles omitted and out-of-bounds indices in a
manner consistent with regular slices (and this innocuous phrase hides a welter
of confusing details!).  The method is intended to be used like this:

@example
class FakeSeq:
    ...
    def calc_item(self, i):
        ...
    def __getitem__(self, item):
        if isinstance(item, slice):
            indices = item.indices(len(self))
            return FakeSeq([self.calc_item(i) for i in range(*indices)])
        else:
            return self.calc_item(i)
@end example

From this example you can also see that the built-in @ref{ffb,,slice} object is
now the type object for the slice type, and is no longer a function.  This is
consistent with Python 2.2, where @ref{1b1,,int}, @ref{2ef,,str}, etc., underwent
the same change.

@c ======================================================================

@node Other Language Changes<16>,New Improved and Deprecated Modules<4>,Extended Slices,What’s New in Python 2 3
@anchor{whatsnew/2 3 other-language-changes}@anchor{ffc}
@subsection Other Language Changes


Here are all of the changes that Python 2.3 makes to the core Python language.


@itemize *

@item 
The @ref{4c5,,yield} statement is now always a keyword, as described in
section @ref{fdf,,PEP 255; Simple Generators} of this document.

@item 
A new built-in function @ref{fdc,,enumerate()} was added, as described in section
@ref{fe8,,PEP 279; enumerate()} of this document.

@item 
Two new constants, @ref{748,,True} and @ref{659,,False} were added along with the
built-in @ref{2f7,,bool} type, as described in section @ref{feb,,PEP 285; A Boolean Type} of this
document.

@item 
The @ref{1b1,,int()} type constructor will now return a long integer instead of
raising an @ref{347,,OverflowError} when a string or floating-point number is too
large to fit into an integer.  This can lead to the paradoxical result that
@code{isinstance(int(expression), int)} is false, but that seems unlikely to cause
problems in practice.

@item 
Built-in types now support the extended slicing syntax, as described in
section @ref{ffa,,Extended Slices} of this document.

@item 
A new built-in function, @code{sum(iterable, start=0)},  adds up the numeric
items in the iterable object and returns their sum.  @ref{50d,,sum()} only accepts
numbers, meaning that you can’t use it to concatenate a bunch of strings.
(Contributed by Alex Martelli.)

@item 
@code{list.insert(pos, value)} used to  insert @emph{value} at the front of the list
when @emph{pos} was negative.  The behaviour has now been changed to be consistent
with slice indexing, so when @emph{pos} is -1 the value will be inserted before the
last element, and so forth.

@item 
@code{list.index(value)}, which searches for @emph{value}  within the list and returns
its index, now takes optional  @emph{start} and @emph{stop} arguments to limit the search
to  only part of the list.

@item 
Dictionaries have a new method, @code{pop(key[, *default*])}, that returns
the value corresponding to @emph{key} and removes that key/value pair from the
dictionary.  If the requested key isn’t present in the dictionary, @emph{default} is
returned if it’s specified and @ref{5bb,,KeyError} raised if it isn’t.

@example
>>> d = @{1:2@}
>>> d
@{1: 2@}
>>> d.pop(4)
Traceback (most recent call last):
  File "stdin", line 1, in ?
KeyError: 4
>>> d.pop(1)
2
>>> d.pop(1)
Traceback (most recent call last):
  File "stdin", line 1, in ?
KeyError: 'pop(): dictionary is empty'
>>> d
@{@}
>>>
@end example

There’s also a new class method,  @code{dict.fromkeys(iterable, value)}, that
creates a dictionary with keys taken from the supplied iterator @emph{iterable} and
all values set to @emph{value}, defaulting to @code{None}.

(Patches contributed by Raymond Hettinger.)

Also, the @ref{3b0,,dict()} constructor now accepts keyword arguments to simplify
creating small dictionaries:

@example
>>> dict(red=1, blue=2, green=3, black=4)
@{'blue': 2, 'black': 4, 'green': 3, 'red': 1@}
@end example

(Contributed by Just van Rossum.)

@item 
The @ref{46d,,assert} statement no longer checks the @code{__debug__} flag, so
you can no longer disable assertions by assigning to @code{__debug__}. Running
Python with the @ref{912,,-O} switch will still generate code that doesn’t
execute any assertions.

@item 
Most type objects are now callable, so you can use them to create new objects
such as functions, classes, and modules.  (This means that the @code{new} module
can be deprecated in a future Python version, because you can now use the type
objects available in the @ref{11f,,types} module.) For example, you can create a new
module object with the following code:

@example
>>> import types
>>> m = types.ModuleType('abc','docstring')
>>> m
<module 'abc' (built-in)>
>>> m.__doc__
'docstring'
@end example

@item 
A new warning, @ref{3a9,,PendingDeprecationWarning} was added to indicate features
which are in the process of being deprecated.  The warning will @emph{not} be printed
by default.  To check for use of features that will be deprecated in the future,
supply @ref{3a8,,-Walways;;PendingDeprecationWarning;;} on the command line or
use @ref{ffd,,warnings.filterwarnings()}.

@item 
The process of deprecating string-based exceptions, as in @code{raise "Error
occurred"}, has begun.  Raising a string will now trigger
@ref{3a9,,PendingDeprecationWarning}.

@item 
Using @code{None} as a variable name will now result in a @ref{4c7,,SyntaxWarning}
warning.  In a future version of Python, @code{None} may finally become a keyword.

@item 
The @code{xreadlines()} method of file objects, introduced in Python 2.1, is no
longer necessary because files now behave as their own iterator.
@code{xreadlines()} was originally introduced as a faster way to loop over all
the lines in a file, but now you can simply write @code{for line in file_obj}.
File objects also have a new read-only @code{encoding} attribute that gives the
encoding used by the file; Unicode strings written to the file will be
automatically  converted to bytes using the given encoding.

@item 
The method resolution order used by new-style classes has changed, though
you’ll only notice the difference if you have a really complicated inheritance
hierarchy.  Classic classes are unaffected by this change.  Python 2.2
originally used a topological sort of a class’s ancestors, but 2.3 now uses the
C3 algorithm as described in the paper "A Monotonic Superclass Linearization for Dylan"@footnote{http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.19.3910}. To
understand the motivation for this change,  read Michele Simionato’s article
"Python 2.3 Method Resolution Order"@footnote{http://www.phyast.pitt.edu/~micheles/mro.html}, or
read the thread on python-dev starting with the message at
@indicateurl{https://mail.python.org/pipermail/python-dev/2002-October/029035.html}. Samuele
Pedroni first pointed out the problem and also implemented the fix by coding the
C3 algorithm.

@item 
Python runs multithreaded programs by switching between threads after
executing N bytecodes.  The default value for N has been increased from 10 to
100 bytecodes, speeding up single-threaded applications by reducing the
switching overhead.  Some multithreaded applications may suffer slower response
time, but that’s easily fixed by setting the limit back to a lower number using
@code{sys.setcheckinterval(N)}. The limit can be retrieved with the new
@code{sys.getcheckinterval()} function.

@item 
One minor but far-reaching change is that the names of extension types defined
by the modules included with Python now contain the module and a @code{'.'} in
front of the type name.  For example, in Python 2.2, if you created a socket and
printed its @code{__class__}, you’d get this output:

@example
>>> s = socket.socket()
>>> s.__class__
<type 'socket'>
@end example

In 2.3, you get this:

@example
>>> s.__class__
<type '_socket.socket'>
@end example

@item 
One of the noted incompatibilities between old- and new-style classes has been
removed: you can now assign to the @ref{e70,,__name__} and @ref{ffe,,__bases__}
attributes of new-style classes.  There are some restrictions on what can be
assigned to @ref{ffe,,__bases__} along the lines of those relating to assigning to
an instance’s @ref{fff,,__class__} attribute.
@end itemize

@c ======================================================================

@menu
* String Changes:: 
* Optimizations: Optimizations<15>. 

@end menu

@node String Changes,Optimizations<15>,,Other Language Changes<16>
@anchor{whatsnew/2 3 string-changes}@anchor{1000}
@subsubsection String Changes



@itemize *

@item 
The @ref{2fc,,in} operator now works differently for strings. Previously, when
evaluating @code{X in Y} where @emph{X} and @emph{Y} are strings, @emph{X} could only be a single
character. That’s now changed; @emph{X} can be a string of any length, and @code{X in Y}
will return @ref{748,,True} if @emph{X} is a substring of @emph{Y}.  If @emph{X} is the empty
string, the result is always @ref{748,,True}.

@example
>>> 'ab' in 'abcd'
True
>>> 'ad' in 'abcd'
False
>>> '@w{'} in 'abcd'
True
@end example

Note that this doesn’t tell you where the substring starts; if you need that
information, use the @code{find()} string method.

@item 
The @code{strip()}, @code{lstrip()}, and @code{rstrip()} string methods now have
an optional argument for specifying the characters to strip.  The default is
still to remove all whitespace characters:

@example
>>> '   abc '.strip()
'abc'
>>> '><><abc<><><>'.strip('<>')
'abc'
>>> '><><abc<><><>\n'.strip('<>')
'abc<><><>\n'
>>> u'\u4000\u4001abc\u4000'.strip(u'\u4000')
u'\u4001abc'
>>>
@end example

(Suggested by Simon Brunning and implemented by Walter Dörwald.)

@item 
The @code{startswith()} and @code{endswith()} string methods now accept negative
numbers for the @emph{start} and @emph{end} parameters.

@item 
Another new string method is @code{zfill()}, originally a function in the
@ref{f1,,string} module.  @code{zfill()} pads a numeric string with zeros on the
left until it’s the specified width. Note that the @code{%} operator is still more
flexible and powerful than @code{zfill()}.

@example
>>> '45'.zfill(4)
'0045'
>>> '12345'.zfill(4)
'12345'
>>> 'goofy'.zfill(6)
'0goofy'
@end example

(Contributed by Walter Dörwald.)

@item 
A new type object, @code{basestring}, has been added. Both 8-bit strings and
Unicode strings inherit from this type, so @code{isinstance(obj, basestring)} will
return @ref{748,,True} for either kind of string.  It’s a completely abstract
type, so you can’t create @code{basestring} instances.

@item 
Interned strings are no longer immortal and will now be garbage-collected in
the usual way when the only reference to them is from the internal dictionary of
interned strings.  (Implemented by Oren Tirosh.)
@end itemize

@c ======================================================================

@node Optimizations<15>,,String Changes,Other Language Changes<16>
@anchor{whatsnew/2 3 optimizations}@anchor{1001}
@subsubsection Optimizations



@itemize *

@item 
The creation of new-style class instances has been made much faster; they’re
now faster than classic classes!

@item 
The @code{sort()} method of list objects has been extensively rewritten by Tim
Peters, and the implementation is significantly faster.

@item 
Multiplication of large long integers is now much faster thanks to an
implementation of Karatsuba multiplication, an algorithm that scales better than
the O(n*n) required for the grade-school multiplication algorithm.  (Original
patch by Christopher A. Craig, and significantly reworked by Tim Peters.)

@item 
The @code{SET_LINENO} opcode is now gone.  This may provide a small speed
increase, depending on your compiler’s idiosyncrasies. See section
@ref{1002,,Other Changes and Fixes} for a longer explanation. (Removed by Michael Hudson.)

@item 
@code{xrange()} objects now have their own iterator, making @code{for i in
xrange(n)} slightly faster than @code{for i in range(n)}.  (Patch by Raymond
Hettinger.)

@item 
A number of small rearrangements have been made in various hotspots to improve
performance, such as inlining a function or removing some code.  (Implemented
mostly by GvR, but lots of people have contributed single changes.)
@end itemize

The net result of the 2.3 optimizations is that Python 2.3 runs the  pystone
benchmark around 25% faster than Python 2.2.

@c ======================================================================

@node New Improved and Deprecated Modules<4>,Pymalloc A Specialized Object Allocator,Other Language Changes<16>,What’s New in Python 2 3
@anchor{whatsnew/2 3 new-improved-and-deprecated-modules}@anchor{1003}
@subsection New, Improved, and Deprecated Modules


As usual, Python’s standard library received a number of enhancements and bug
fixes.  Here’s a partial list of the most notable changes, sorted alphabetically
by module name. Consult the @code{Misc/NEWS} file in the source tree for a more
complete list of changes, or look through the CVS logs for all the details.


@itemize *

@item 
The @ref{6,,array} module now supports arrays of Unicode characters using the
@code{'u'} format character.  Arrays also now support using the @code{+=} assignment
operator to add another array’s contents, and the @code{*=} assignment operator to
repeat an array. (Contributed by Jason Orendorff.)

@item 
The @code{bsddb} module has been replaced by version 4.1.6 of the PyBSDDB@footnote{http://pybsddb.sourceforge.net} package, providing a more complete interface
to the transactional features of the BerkeleyDB library.

The old version of the module has been renamed to  @code{bsddb185} and is no
longer built automatically; you’ll  have to edit @code{Modules/Setup} to enable
it.  Note that the new @code{bsddb} package is intended to be compatible with
the  old module, so be sure to file bugs if you discover any incompatibilities.
When upgrading to Python 2.3, if the new interpreter is compiled with a new
version of  the underlying BerkeleyDB library, you will almost certainly have to
convert your database files to the new version.  You can do this fairly easily
with the new scripts @code{db2pickle.py} and @code{pickle2db.py} which you
will find in the distribution’s @code{Tools/scripts} directory.  If you’ve
already been using the PyBSDDB package and importing it as @code{bsddb3}, you
will have to change your @code{import} statements to import it as @code{bsddb}.

@item 
The new @ref{12,,bz2} module is an interface to the bz2 data compression library.
bz2-compressed data is usually smaller than  corresponding
@ref{14b,,zlib}-compressed data. (Contributed by Gustavo Niemeyer.)

@item 
A set of standard date/time types has been added in the new @ref{2f,,datetime}
module.  See the following section for more details.

@item 
The Distutils @code{Extension} class now supports an extra constructor
argument named @emph{depends} for listing additional source files that an extension
depends on.  This lets Distutils recompile the module if any of the dependency
files are modified.  For example, if @code{sampmodule.c} includes the header
file @code{sample.h}, you would create the @code{Extension} object like
this:

@example
ext = Extension("samp",
                sources=["sampmodule.c"],
                depends=["sample.h"])
@end example

Modifying @code{sample.h} would then cause the module to be recompiled.
(Contributed by Jeremy Hylton.)

@item 
Other minor changes to Distutils: it now checks for the 
@geindex CC
@geindex environment variable; CC
@ref{1004,,CC},
@geindex CFLAGS
@geindex environment variable; CFLAGS
@ref{1005,,CFLAGS}, 
@geindex CPP
@geindex environment variable; CPP
@code{CPP}, 
@geindex LDFLAGS
@geindex environment variable; LDFLAGS
@ref{1006,,LDFLAGS}, and 
@geindex CPPFLAGS
@geindex environment variable; CPPFLAGS
@ref{1007,,CPPFLAGS}
environment variables, using them to override the settings in Python’s
configuration (contributed by Robert Weber).

@item 
Previously the @ref{63,,doctest} module would only search the docstrings of
public methods and functions for test cases, but it now also examines private
ones as well.  The @code{DocTestSuite()} function creates a
@ref{953,,unittest.TestSuite} object from a set of @ref{63,,doctest} tests.

@item 
The new @code{gc.get_referents(object)} function returns a list of all the
objects referenced by @emph{object}.

@item 
The @ref{81,,getopt} module gained a new function, @code{gnu_getopt()}, that
supports the same arguments as the existing @ref{81,,getopt()} function but uses
GNU-style scanning mode. The existing @ref{81,,getopt()} stops processing options as
soon as a non-option argument is encountered, but in GNU-style mode processing
continues, meaning that options and arguments can be mixed.  For example:

@example
>>> getopt.getopt(['-f', 'filename', 'output', '-v'], 'f:v')
([('-f', 'filename')], ['output', '-v'])
>>> getopt.gnu_getopt(['-f', 'filename', 'output', '-v'], 'f:v')
([('-f', 'filename'), ('-v', '@w{'})], ['output'])
@end example

(Contributed by Peter Åstrand.)

@item 
The @ref{86,,grp}, @ref{d1,,pwd}, and @ref{db,,resource} modules now return enhanced
tuples:

@example
>>> import grp
>>> g = grp.getgrnam('amk')
>>> g.gr_name, g.gr_gid
('amk', 500)
@end example

@item 
The @ref{87,,gzip} module can now handle files exceeding 2 GiB.

@item 
The new @ref{89,,heapq} module contains an implementation of a heap queue
algorithm.  A heap is an array-like data structure that keeps items in a
partially sorted order such that, for every index @emph{k}, @code{heap[k] <=
heap[2*k+1]} and @code{heap[k] <= heap[2*k+2]}.  This makes it quick to remove the
smallest item, and inserting a new item while maintaining the heap property is
O(lg n).  (See @indicateurl{https://xlinux.nist.gov/dads//HTML/priorityque.html} for more
information about the priority queue data structure.)

The @ref{89,,heapq} module provides @code{heappush()} and @code{heappop()} functions
for adding and removing items while maintaining the heap property on top of some
other mutable Python sequence type.  Here’s an example that uses a Python list:

@example
>>> import heapq
>>> heap = []
>>> for item in [3, 7, 5, 11, 1]:
...    heapq.heappush(heap, item)
...
>>> heap
[1, 3, 5, 11, 7]
>>> heapq.heappop(heap)
1
>>> heapq.heappop(heap)
3
>>> heap
[5, 7, 11]
@end example

(Contributed by Kevin O’Connor.)

@item 
The IDLE integrated development environment has been updated using the code
from the IDLEfork project (@indicateurl{http://idlefork.sourceforge.net}).  The most notable feature is
that the code being developed is now executed in a subprocess, meaning that
there’s no longer any need for manual @code{reload()} operations. IDLE’s core code
has been incorporated into the standard library as the @code{idlelib} package.

@item 
The @ref{93,,imaplib} module now supports IMAP over SSL. (Contributed by Piers
Lauder and Tino Lange.)

@item 
The @ref{9f,,itertools} contains a number of useful functions for use with
iterators, inspired by various functions provided by the ML and Haskell
languages.  For example, @code{itertools.ifilter(predicate, iterator)} returns all
elements in the iterator for which the function @code{predicate()} returns
@ref{748,,True}, and @code{itertools.repeat(obj, N)} returns @code{obj} @emph{N} times.
There are a number of other functions in the module; see the package’s reference
documentation for details.
(Contributed by Raymond Hettinger.)

@item 
Two new functions in the @ref{ad,,math} module, @code{degrees(rads)} and
@code{radians(degs)}, convert between radians and degrees.  Other functions in
the @ref{ad,,math} module such as @ref{1008,,math.sin()} and @ref{1009,,math.cos()} have always
required input values measured in radians.  Also, an optional @emph{base} argument
was added to @ref{100a,,math.log()} to make it easier to compute logarithms for bases
other than @code{e} and @code{10}.  (Contributed by Raymond Hettinger.)

@item 
Several new POSIX functions (@code{getpgid()}, @code{killpg()}, @code{lchown()},
@code{loadavg()}, @code{major()}, @code{makedev()}, @code{minor()}, and
@code{mknod()}) were added to the @ref{cc,,posix} module that underlies the
@ref{c0,,os} module. (Contributed by Gustavo Niemeyer, Geert Jansen, and Denis S.
Otkidach.)

@item 
In the @ref{c0,,os} module, the @code{*stat()} family of functions can now report
fractions of a second in a timestamp.  Such time stamps are represented as
floats, similar to the value returned by @ref{5fc,,time.time()}.

During testing, it was found that some applications will break if time stamps
are floats.  For compatibility, when using the tuple interface of the
@code{stat_result} time stamps will be represented as integers. When using
named fields (a feature first introduced in Python 2.2), time stamps are still
represented as integers, unless @code{os.stat_float_times()} is invoked to enable
float return values:

@example
>>> os.stat("/tmp").st_mtime
1034791200
>>> os.stat_float_times(True)
>>> os.stat("/tmp").st_mtime
1034791200.6335014
@end example

In Python 2.4, the default will change to always returning floats.

Application developers should enable this feature only if all their libraries
work properly when confronted with floating point time stamps, or if they use
the tuple API. If used, the feature should be activated on an application level
instead of trying to enable it on a per-use basis.

@item 
The @ref{bf,,optparse} module contains a new parser for command-line arguments
that can convert option values to a particular Python type  and will
automatically generate a usage message.  See the following section for  more
details.

@item 
The old and never-documented @code{linuxaudiodev} module has been deprecated,
and a new version named @ref{c2,,ossaudiodev} has been added.  The module was
renamed because the OSS sound drivers can be used on platforms other than Linux,
and the interface has also been tidied and brought up to date in various ways.
(Contributed by Greg Ward and Nicholas FitzRoy-Dale.)

@item 
The new @ref{c9,,platform} module contains a number of functions that try to
determine various properties of the platform you’re running on.  There are
functions for getting the architecture, CPU type, the Windows OS version, and
even the Linux distribution version. (Contributed by Marc-André Lemburg.)

@item 
The parser objects provided by the @code{pyexpat} module can now optionally
buffer character data, resulting in fewer calls to your character data handler
and therefore faster performance.  Setting the parser object’s
@code{buffer_text} attribute to @ref{748,,True} will enable buffering.

@item 
The @code{sample(population, k)} function was added to the @ref{d7,,random}
module.  @emph{population} is a sequence or @code{xrange} object containing the
elements of a population, and @code{sample()} chooses @emph{k} elements from the
population without replacing chosen elements.  @emph{k} can be any value up to
@code{len(population)}. For example:

@example
>>> days = ['Mo', 'Tu', 'We', 'Th', 'Fr', 'St', 'Sn']
>>> random.sample(days, 3)      # Choose 3 elements
['St', 'Sn', 'Th']
>>> random.sample(days, 7)      # Choose 7 elements
['Tu', 'Th', 'Mo', 'We', 'St', 'Fr', 'Sn']
>>> random.sample(days, 7)      # Choose 7 again
['We', 'Mo', 'Sn', 'Fr', 'Tu', 'St', 'Th']
>>> random.sample(days, 8)      # Can't choose eight
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "random.py", line 414, in sample
      raise ValueError, "sample larger than population"
ValueError: sample larger than population
>>> random.sample(xrange(1,10000,2), 10)   # Choose ten odd nos. under 10000
[3407, 3805, 1505, 7023, 2401, 2267, 9733, 3151, 8083, 9195]
@end example

The @ref{d7,,random} module now uses a new algorithm, the Mersenne Twister,
implemented in C.  It’s faster and more extensively studied than the previous
algorithm.

(All changes contributed by Raymond Hettinger.)

@item 
The @ref{d9,,readline} module also gained a number of new functions:
@code{get_history_item()}, @code{get_current_history_length()}, and
@code{redisplay()}.

@item 
The @code{rexec} and @code{Bastion} modules have been declared dead, and
attempts to import them will fail with a @ref{5af,,RuntimeError}.  New-style classes
provide new ways to break out of the restricted execution environment provided
by @code{rexec}, and no one has interest in fixing them or time to do so.  If
you have applications using @code{rexec}, rewrite them to use something else.

(Sticking with Python 2.2 or 2.1 will not make your applications any safer
because there are known bugs in the @code{rexec} module in those versions.  To
repeat: if you’re using @code{rexec}, stop using it immediately.)

@item 
The @code{rotor} module has been deprecated because the  algorithm it uses for
encryption is not believed to be secure.  If you need encryption, use one of the
several AES Python modules that are available separately.

@item 
The @ref{e4,,shutil} module gained a @code{move(src, dest)} function that
recursively moves a file or directory to a new location.

@item 
Support for more advanced POSIX signal handling was added to the @ref{e5,,signal}
but then removed again as it proved impossible to make it work reliably across
platforms.

@item 
The @ref{ea,,socket} module now supports timeouts.  You can call the
@code{settimeout(t)} method on a socket object to set a timeout of @emph{t} seconds.
Subsequent socket operations that take longer than @emph{t} seconds to complete will
abort and raise a @ref{2b7,,socket.timeout} exception.

The original timeout implementation was by Tim O’Malley.  Michael Gilfix
integrated it into the Python @ref{ea,,socket} module and shepherded it through a
lengthy review.  After the code was checked in, Guido van Rossum rewrote parts
of it.  (This is a good example of a collaborative development process in
action.)

@item 
On Windows, the @ref{ea,,socket} module now ships with Secure  Sockets Layer
(SSL) support.

@item 
The value of the C @code{PYTHON_API_VERSION} macro is now exposed at the
Python level as @code{sys.api_version}.  The current exception can be cleared by
calling the new @code{sys.exc_clear()} function.

@item 
The new @ref{fb,,tarfile} module  allows reading from and writing to
@code{tar}-format archive files. (Contributed by Lars Gustäbel.)

@item 
The new @ref{108,,textwrap} module contains functions for wrapping strings
containing paragraphs of text.  The @code{wrap(text, width)} function takes a
string and returns a list containing the text split into lines of no more than
the chosen width.  The @code{fill(text, width)} function returns a single
string, reformatted to fit into lines no longer than the chosen width. (As you
can guess, @code{fill()} is built on top of @code{wrap()}.  For example:

@example
>>> import textwrap
>>> paragraph = "Not a whit, we defy augury: ... more text ..."
>>> textwrap.wrap(paragraph, 60)
["Not a whit, we defy augury: there's a special providence in",
 "the fall of a sparrow. If it be now, 'tis not to come; if it",
 ...]
>>> print textwrap.fill(paragraph, 35)
Not a whit, we defy augury: there's
a special providence in the fall of
a sparrow. If it be now, 'tis not
to come; if it be not to come, it
will be now; if it be not now, yet
it will come: the readiness is all.
>>>
@end example

The module also contains a @code{TextWrapper} class that actually implements
the text wrapping strategy.   Both the @code{TextWrapper} class and the
@code{wrap()} and @code{fill()} functions support a number of additional keyword
arguments for fine-tuning the formatting; consult the module’s documentation
for details. (Contributed by Greg Ward.)

@item 
The @code{thread} and @ref{109,,threading} modules now have companion modules,
@code{dummy_thread} and @code{dummy_threading}, that provide a do-nothing
implementation of the @code{thread} module’s interface for platforms where
threads are not supported.  The intention is to simplify thread-aware modules
(ones that @emph{don’t} rely on threads to run) by putting the following code at the
top:

@example
try:
    import threading as _threading
except ImportError:
    import dummy_threading as _threading
@end example

In this example, @code{_threading} is used as the module name to make it clear
that the module being used is not necessarily the actual @ref{109,,threading}
module. Code can call functions and use classes in @code{_threading} whether or
not threads are supported, avoiding an @ref{2fb,,if} statement and making the
code slightly clearer.  This module will not magically make multithreaded code
run without threads; code that waits for another thread to return or to do
something will simply hang forever.

@item 
The @ref{10a,,time} module’s @code{strptime()} function has long been an annoyance
because it uses the platform C library’s @code{strptime()} implementation, and
different platforms sometimes have odd bugs.  Brett Cannon contributed a
portable implementation that’s written in pure Python and should behave
identically on all platforms.

@item 
The new @ref{10b,,timeit} module helps measure how long snippets of Python code
take to execute.  The @code{timeit.py} file can be run directly from the
command line, or the module’s @code{Timer} class can be imported and used
directly.  Here’s a short example that figures out whether it’s faster to
convert an 8-bit string to Unicode by appending an empty Unicode string to it or
by using the @code{unicode()} function:

@example
import timeit

timer1 = timeit.Timer('unicode("abc")')
timer2 = timeit.Timer('"abc" + u""')

# Run three trials
print timer1.repeat(repeat=3, number=100000)
print timer2.repeat(repeat=3, number=100000)

# On my laptop this outputs:
# [0.36831796169281006, 0.37441694736480713, 0.35304892063140869]
# [0.17574405670166016, 0.18193507194519043, 0.17565798759460449]
@end example

@item 
The @code{Tix} module has received various bug fixes and updates for the
current version of the Tix package.

@item 
The @code{Tkinter} module now works with a thread-enabled  version of Tcl.
Tcl’s threading model requires that widgets only be accessed from the thread in
which they’re created; accesses from another thread can cause Tcl to panic.  For
certain Tcl interfaces, @code{Tkinter} will now automatically avoid this  when a
widget is accessed from a different thread by marshalling a command, passing it
to the correct thread, and waiting for the results.  Other interfaces can’t be
handled automatically but @code{Tkinter} will now raise an exception on such an
access so that you can at least find out about the problem.  See
@indicateurl{https://mail.python.org/pipermail/python-dev/2002-December/031107.html} for a more
detailed explanation of this change.  (Implemented by Martin von Löwis.)

@item 
Calling Tcl methods through @code{_tkinter} no longer  returns only strings.
Instead, if Tcl returns other objects those objects are converted to their
Python equivalent, if one exists, or wrapped with a @code{_tkinter.Tcl_Obj}
object if no Python equivalent exists. This behavior can be controlled through
the @code{wantobjects()} method of @code{tkapp} objects.

When using @code{_tkinter} through the @code{Tkinter} module (as most Tkinter
applications will), this feature is always activated. It should not cause
compatibility problems, since Tkinter would always convert string results to
Python types where possible.

If any incompatibilities are found, the old behavior can be restored by setting
the @code{wantobjects} variable in the @code{Tkinter} module to false before
creating the first @code{tkapp} object.

@example
import Tkinter
Tkinter.wantobjects = 0
@end example

Any breakage caused by this change should be reported as a bug.

@item 
The @code{UserDict} module has a new @code{DictMixin} class which defines
all dictionary methods for classes that already have a minimum mapping
interface.  This greatly simplifies writing classes that need to be
substitutable for dictionaries, such as the classes in  the @ref{e2,,shelve}
module.

Adding the mix-in as a superclass provides the full dictionary interface
whenever the class defines @code{__getitem__()}, @code{__setitem__()},
@code{__delitem__()}, and @code{keys()}. For example:

@example
>>> import UserDict
>>> class SeqDict(UserDict.DictMixin):
...     """Dictionary lookalike implemented with lists."""
...     def __init__(self):
...         self.keylist = []
...         self.valuelist = []
...     def __getitem__(self, key):
...         try:
...             i = self.keylist.index(key)
...         except ValueError:
...             raise KeyError
...         return self.valuelist[i]
...     def __setitem__(self, key, value):
...         try:
...             i = self.keylist.index(key)
...             self.valuelist[i] = value
...         except ValueError:
...             self.keylist.append(key)
...             self.valuelist.append(value)
...     def __delitem__(self, key):
...         try:
...             i = self.keylist.index(key)
...         except ValueError:
...             raise KeyError
...         self.keylist.pop(i)
...         self.valuelist.pop(i)
...     def keys(self):
...         return list(self.keylist)
...
>>> s = SeqDict()
>>> dir(s)      # See that other dictionary methods are implemented
['__cmp__', '__contains__', '__delitem__', '__doc__', '__getitem__',
 '__init__', '__iter__', '__len__', '__module__', '__repr__',
 '__setitem__', 'clear', 'get', 'has_key', 'items', 'iteritems',
 'iterkeys', 'itervalues', 'keylist', 'keys', 'pop', 'popitem',
 'setdefault', 'update', 'valuelist', 'values']
@end example

(Contributed by Raymond Hettinger.)

@item 
The DOM implementation in @ref{13c,,xml.dom.minidom} can now generate XML output
in a particular encoding by providing an optional encoding argument to the
@code{toxml()} and @code{toprettyxml()} methods of DOM nodes.

@item 
The @code{xmlrpclib} module now supports an XML-RPC extension for handling nil
data values such as Python’s @code{None}.  Nil values are always supported on
unmarshalling an XML-RPC response.  To generate requests containing @code{None},
you must supply a true value for the @emph{allow_none} parameter when creating a
@code{Marshaller} instance.

@item 
The new @code{DocXMLRPCServer} module allows writing self-documenting XML-RPC
servers. Run it in demo mode (as a program) to see it in action.   Pointing the
web browser to the RPC server produces pydoc-style documentation; pointing
xmlrpclib to the server allows invoking the actual methods. (Contributed by
Brian Quinlan.)

@item 
Support for internationalized domain names (RFCs 3454, 3490, 3491, and 3492)
has been added. The “idna” encoding can be used to convert between a Unicode
domain name and the ASCII-compatible encoding (ACE) of that name.

@example
>@{@}>@{@}> u"www.Alliancefrançaise.nu".encode("idna")
'www.xn--alliancefranaise-npb.nu'
@end example

The @ref{ea,,socket} module has also been extended to transparently convert
Unicode hostnames to the ACE version before passing them to the C library.
Modules that deal with hostnames such as @code{httplib} and @ref{7e,,ftplib})
also support Unicode host names; @code{httplib} also sends HTTP @code{Host}
headers using the ACE version of the domain name.  @ref{124,,urllib} supports
Unicode URLs with non-ASCII host names as long as the @code{path} part of the URL
is ASCII only.

To implement this change, the @ref{f2,,stringprep} module, the  @code{mkstringprep}
tool and the @code{punycode} encoding have been added.
@end itemize

@c ======================================================================

@menu
* Date/Time Type:: 
* The optparse Module:: 

@end menu

@node Date/Time Type,The optparse Module,,New Improved and Deprecated Modules<4>
@anchor{whatsnew/2 3 date-time-type}@anchor{100b}
@subsubsection Date/Time Type


Date and time types suitable for expressing timestamps were added as the
@ref{2f,,datetime} module.  The types don’t support different calendars or many
fancy features, and just stick to the basics of representing time.

The three primary types are: @code{date}, representing a day, month, and year;
@ref{364,,time}, consisting of hour, minute, and second; and @ref{363,,datetime},
which contains all the attributes of both @code{date} and @ref{364,,time}.
There’s also a @code{timedelta} class representing differences between two
points in time, and time zone logic is implemented by classes inheriting from
the abstract @code{tzinfo} class.

You can create instances of @code{date} and @ref{364,,time} by either supplying
keyword arguments to the appropriate constructor, e.g.
@code{datetime.date(year=1972, month=10, day=15)}, or by using one of a number of
class methods.  For example, the @code{date.today()} class method returns the
current local date.

Once created, instances of the date/time classes are all immutable. There are a
number of methods for producing formatted strings from objects:

@example
>>> import datetime
>>> now = datetime.datetime.now()
>>> now.isoformat()
'2002-12-30T21:27:03.994956'
>>> now.ctime()  # Only available on date, datetime
'Mon Dec 30 21:27:03 2002'
>>> now.strftime('%Y %d %b')
'2002 30 Dec'
@end example

The @code{replace()} method allows modifying one or more fields  of a
@code{date} or @ref{363,,datetime} instance, returning a new instance:

@example
>>> d = datetime.datetime.now()
>>> d
datetime.datetime(2002, 12, 30, 22, 15, 38, 827738)
>>> d.replace(year=2001, hour = 12)
datetime.datetime(2001, 12, 30, 12, 15, 38, 827738)
>>>
@end example

Instances can be compared, hashed, and converted to strings (the result is the
same as that of @code{isoformat()}).  @code{date} and @ref{363,,datetime}
instances can be subtracted from each other, and added to @code{timedelta}
instances.  The largest missing feature is that there’s no standard library
support for parsing strings and getting back a @code{date} or
@ref{363,,datetime}.

For more information, refer to the module’s reference documentation.
(Contributed by Tim Peters.)

@c ======================================================================

@node The optparse Module,,Date/Time Type,New Improved and Deprecated Modules<4>
@anchor{whatsnew/2 3 the-optparse-module}@anchor{100c}
@subsubsection The optparse Module


The @ref{81,,getopt} module provides simple parsing of command-line arguments.  The
new @ref{bf,,optparse} module (originally named Optik) provides more elaborate
command-line parsing that follows the Unix conventions, automatically creates
the output for @code{--help}, and can perform different actions for different
options.

You start by creating an instance of @code{OptionParser} and telling it what
your program’s options are.

@example
import sys
from optparse import OptionParser

op = OptionParser()
op.add_option('-i', '--input',
              action='store', type='string', dest='input',
              help='set input filename')
op.add_option('-l', '--length',
              action='store', type='int', dest='length',
              help='set maximum length of output')
@end example

Parsing a command line is then done by calling the @code{parse_args()} method.

@example
options, args = op.parse_args(sys.argv[1:])
print options
print args
@end example

This returns an object containing all of the option values, and a list of
strings containing the remaining arguments.

Invoking the script with the various arguments now works as you’d expect it to.
Note that the length argument is automatically converted to an integer.

@example
$ ./python opt.py -i data arg1
<Values at 0x400cad4c: @{'input': 'data', 'length': None@}>
['arg1']
$ ./python opt.py --input=data --length=4
<Values at 0x400cad2c: @{'input': 'data', 'length': 4@}>
[]
$
@end example

The help message is automatically generated for you:

@example
$ ./python opt.py --help
usage: opt.py [options]

options:
  -h, --help            show this help message and exit
  -iINPUT, --input=INPUT
                        set input filename
  -lLENGTH, --length=LENGTH
                        set maximum length of output
$
@end example

See the module’s documentation for more details.

Optik was written by Greg Ward, with suggestions from the readers of the Getopt
SIG.

@c ======================================================================

@node Pymalloc A Specialized Object Allocator,Build and C API Changes<12>,New Improved and Deprecated Modules<4>,What’s New in Python 2 3
@anchor{whatsnew/2 3 pymalloc-a-specialized-object-allocator}@anchor{100d}@anchor{whatsnew/2 3 section-pymalloc}@anchor{100e}
@subsection Pymalloc: A Specialized Object Allocator


Pymalloc, a specialized object allocator written by Vladimir Marangozov, was a
feature added to Python 2.1.  Pymalloc is intended to be faster than the system
@code{malloc()} and to have less memory overhead for allocation patterns typical
of Python programs. The allocator uses C’s @code{malloc()} function to get large
pools of memory and then fulfills smaller memory requests from these pools.

In 2.1 and 2.2, pymalloc was an experimental feature and wasn’t enabled by
default; you had to explicitly enable it when compiling Python by providing the
@code{--with-pymalloc} option to the @code{configure} script.  In 2.3,
pymalloc has had further enhancements and is now enabled by default; you’ll have
to supply @code{--without-pymalloc} to disable it.

This change is transparent to code written in Python; however, pymalloc may
expose bugs in C extensions.  Authors of C extension modules should test their
code with pymalloc enabled, because some incorrect code may cause core dumps at
runtime.

There’s one particularly common error that causes problems.  There are a number
of memory allocation functions in Python’s C API that have previously just been
aliases for the C library’s @code{malloc()} and @code{free()}, meaning that if
you accidentally called mismatched functions the error wouldn’t be noticeable.
When the object allocator is enabled, these functions aren’t aliases of
@code{malloc()} and @code{free()} any more, and calling the wrong function to
free memory may get you a core dump.  For example, if memory was allocated using
@ref{7af,,PyObject_Malloc()}, it has to be freed using @ref{7ac,,PyObject_Free()}, not
@code{free()}.  A few modules included with Python fell afoul of this and had to
be fixed; doubtless there are more third-party modules that will have the same
problem.

As part of this change, the confusing multiple interfaces for allocating memory
have been consolidated down into two API families. Memory allocated with one
family must not be manipulated with functions from the other family.  There is
one family for allocating chunks of memory and another family of functions
specifically for allocating Python objects.


@itemize *

@item 
To allocate and free an undistinguished chunk of memory use the “raw memory”
family: @ref{7ad,,PyMem_Malloc()}, @ref{bc6,,PyMem_Realloc()}, and @ref{fa3,,PyMem_Free()}.

@item 
The “object memory” family is the interface to the pymalloc facility described
above and is biased towards a large number of “small” allocations:
@ref{7af,,PyObject_Malloc()}, @ref{fa4,,PyObject_Realloc()}, and @ref{7ac,,PyObject_Free()}.

@item 
To allocate and free Python objects, use the “object” family
@ref{491,,PyObject_New()}, @ref{492,,PyObject_NewVar()}, and @ref{100f,,PyObject_Del()}.
@end itemize

Thanks to lots of work by Tim Peters, pymalloc in 2.3 also provides debugging
features to catch memory overwrites and doubled frees in both extension modules
and in the interpreter itself.  To enable this support, compile a debugging
version of the Python interpreter by running @code{configure} with
@code{--with-pydebug}.

To aid extension writers, a header file @code{Misc/pymemcompat.h} is
distributed with the source to Python 2.3 that allows Python extensions to use
the 2.3 interfaces to memory allocation while compiling against any version of
Python since 1.5.2.  You would copy the file from Python’s source distribution
and bundle it with the source of your extension.


@subsubheading See also



@table @asis

@item @indicateurl{https://hg.python.org/cpython/file/default/Objects/obmalloc.c}

For the full details of the pymalloc implementation, see the comments at
the top of the file @code{Objects/obmalloc.c} in the Python source code.
The above link points to the file within the python.org SVN browser.
@end table


@c ======================================================================

@node Build and C API Changes<12>,Other Changes and Fixes<2>,Pymalloc A Specialized Object Allocator,What’s New in Python 2 3
@anchor{whatsnew/2 3 build-and-c-api-changes}@anchor{1010}
@subsection Build and C API Changes


Changes to Python’s build process and to the C API include:


@itemize *

@item 
The cycle detection implementation used by the garbage collection has proven
to be stable, so it’s now been made mandatory.  You can no longer compile Python
without it, and the @code{--with-cycle-gc} switch to @code{configure} has
been removed.

@item 
Python can now optionally be built as a shared library
(@code{libpython2.3.so}) by supplying @code{--enable-shared} when running
Python’s @code{configure} script.  (Contributed by Ondrej Palkovsky.)

@item 
The @code{DL_EXPORT} and @code{DL_IMPORT} macros are now deprecated.
Initialization functions for Python extension modules should now be declared
using the new macro @code{PyMODINIT_FUNC}, while the Python core will
generally use the @code{PyAPI_FUNC} and @code{PyAPI_DATA} macros.

@item 
The interpreter can be compiled without any docstrings for the built-in
functions and modules by supplying @code{--without-doc-strings} to the
@code{configure} script. This makes the Python executable about 10% smaller,
but will also mean that you can’t get help for Python’s built-ins.  (Contributed
by Gustavo Niemeyer.)

@item 
The @code{PyArg_NoArgs()} macro is now deprecated, and code that uses it
should be changed.  For Python 2.2 and later, the method definition table can
specify the @ref{1011,,METH_NOARGS} flag, signalling that there are no arguments,
and the argument checking can then be removed.  If compatibility with pre-2.2
versions of Python is important, the code could use @code{PyArg_ParseTuple(args,
"")} instead, but this will be slower than using @ref{1011,,METH_NOARGS}.

@item 
@ref{37f,,PyArg_ParseTuple()} accepts new format characters for various sizes of
unsigned integers: @code{B} for unsigned  char, @code{H} for unsigned  short  int,  @code{I} for unsigned  int,  and @code{K} for unsigned  long  long.

@item 
A new function, @code{PyObject_DelItemString(mapping, char *key)} was added
as shorthand for @code{PyObject_DelItem(mapping, PyString_New(key))}.

@item 
File objects now manage their internal string buffer differently, increasing
it exponentially when needed.  This results in the benchmark tests in
@code{Lib/test/test_bufio.py} speeding up considerably (from 57 seconds to 1.7
seconds, according to one measurement).

@item 
It’s now possible to define class and static methods for a C extension type by
setting either the @ref{1012,,METH_CLASS} or @ref{1013,,METH_STATIC} flags in a
method’s @ref{1014,,PyMethodDef} structure.

@item 
Python now includes a copy of the Expat XML parser’s source code, removing any
dependence on a system version or local installation of Expat.

@item 
If you dynamically allocate type objects in your extension, you should be
aware of a change in the rules relating to the @code{__module__} and
@ref{e70,,__name__} attributes.  In summary, you will want to ensure the type’s
dictionary contains a @code{'__module__'} key; making the module name the part of
the type name leading up to the final period will no longer have the desired
effect.  For more detail, read the API reference documentation or the  source.
@end itemize

@c ======================================================================

@menu
* Port-Specific Changes: Port-Specific Changes<3>. 

@end menu

@node Port-Specific Changes<3>,,,Build and C API Changes<12>
@anchor{whatsnew/2 3 port-specific-changes}@anchor{1015}
@subsubsection Port-Specific Changes


Support for a port to IBM’s OS/2 using the EMX runtime environment was merged
into the main Python source tree.  EMX is a POSIX emulation layer over the OS/2
system APIs.  The Python port for EMX tries to support all the POSIX-like
capability exposed by the EMX runtime, and mostly succeeds; @code{fork()} and
@ref{79,,fcntl()} are restricted by the limitations of the underlying emulation
layer.  The standard OS/2 port, which uses IBM’s Visual Age compiler, also
gained support for case-sensitive import semantics as part of the integration of
the EMX port into CVS.  (Contributed by Andrew MacIntyre.)

On MacOS, most toolbox modules have been weaklinked to improve backward
compatibility.  This means that modules will no longer fail to load if a single
routine is missing on the current OS version. Instead calling the missing
routine will raise an exception. (Contributed by Jack Jansen.)

The RPM spec files, found in the @code{Misc/RPM/} directory in the Python
source distribution, were updated for 2.3.  (Contributed by Sean Reifschneider.)

Other new platforms now supported by Python include AtheOS
(@indicateurl{http://www.atheos.cx/}), GNU/Hurd, and OpenVMS.

@c ======================================================================

@node Other Changes and Fixes<2>,Porting to Python 2 3,Build and C API Changes<12>,What’s New in Python 2 3
@anchor{whatsnew/2 3 other-changes-and-fixes}@anchor{1016}@anchor{whatsnew/2 3 section-other}@anchor{1002}
@subsection Other Changes and Fixes


As usual, there were a bunch of other improvements and bugfixes scattered
throughout the source tree.  A search through the CVS change logs finds there
were 523 patches applied and 514 bugs fixed between Python 2.2 and 2.3.  Both
figures are likely to be underestimates.

Some of the more notable changes are:


@itemize *

@item 
If the 
@geindex PYTHONINSPECT
@geindex environment variable; PYTHONINSPECT
@ref{1017,,PYTHONINSPECT} environment variable is set, the Python
interpreter will enter the interactive prompt after running a Python program, as
if Python had been invoked with the @ref{1018,,-i} option. The environment
variable can be set before running the Python interpreter, or it can be set by
the Python program as part of its execution.

@item 
The @code{regrtest.py} script now provides a way to allow “all resources
except @emph{foo}.”  A resource name passed to the @code{-u} option can now be
prefixed with a hyphen (@code{'-'}) to mean “remove this resource.”  For example,
the option ‘@code{-uall,-bsddb}’ could be used to enable the use of all resources
except @code{bsddb}.

@item 
The tools used to build the documentation now work under Cygwin as well as
Unix.

@item 
The @code{SET_LINENO} opcode has been removed.  Back in the mists of time, this
opcode was needed to produce line numbers in tracebacks and support trace
functions (for, e.g., @ref{c4,,pdb}). Since Python 1.5, the line numbers in
tracebacks have been computed using a different mechanism that works with
“python -O”.  For Python 2.3 Michael Hudson implemented a similar scheme to
determine when to call the trace function, removing the need for @code{SET_LINENO}
entirely.

It would be difficult to detect any resulting difference from Python code, apart
from a slight speed up when Python is run without @ref{912,,-O}.

C extensions that access the @code{f_lineno} field of frame objects should
instead call @code{PyCode_Addr2Line(f->f_code, f->f_lasti)}. This will have the
added effect of making the code work as desired under “python -O” in earlier
versions of Python.

A nifty new feature is that trace functions can now assign to the
@code{f_lineno} attribute of frame objects, changing the line that will be
executed next.  A @code{jump} command has been added to the @ref{c4,,pdb} debugger
taking advantage of this new feature. (Implemented by Richie Hindle.)
@end itemize

@c ======================================================================

@node Porting to Python 2 3,Acknowledgements<5>,Other Changes and Fixes<2>,What’s New in Python 2 3
@anchor{whatsnew/2 3 porting-to-python-2-3}@anchor{1019}
@subsection Porting to Python 2.3


This section lists previously described changes that may require changes to your
code:


@itemize *

@item 
@ref{4c5,,yield} is now always a keyword; if it’s used as a variable name in
your code, a different name must be chosen.

@item 
For strings @emph{X} and @emph{Y}, @code{X in Y} now works if @emph{X} is more than one
character long.

@item 
The @ref{1b1,,int()} type constructor will now return a long integer instead of
raising an @ref{347,,OverflowError} when a string or floating-point number is too
large to fit into an integer.

@item 
If you have Unicode strings that contain 8-bit characters, you must declare
the file’s encoding (UTF-8, Latin-1, or whatever) by adding a comment to the top
of the file.  See section @ref{fe1,,PEP 263; Source Code Encodings} for more information.

@item 
Calling Tcl methods through @code{_tkinter} no longer  returns only strings.
Instead, if Tcl returns other objects those objects are converted to their
Python equivalent, if one exists, or wrapped with a @code{_tkinter.Tcl_Obj}
object if no Python equivalent exists.

@item 
Large octal and hex literals such as @code{0xffffffff} now trigger a
@ref{604,,FutureWarning}. Currently they’re stored as 32-bit numbers and result in a
negative value, but in Python 2.4 they’ll become positive long integers.

There are a few ways to fix this warning.  If you really need a positive number,
just add an @code{L} to the end of the literal.  If you’re trying to get a 32-bit
integer with low bits set and have previously used an expression such as @code{~(1
<< 31)}, it’s probably clearest to start with all bits set and clear the
desired upper bits. For example, to clear just the top bit (bit 31), you could
write @code{0xffffffffL &~(1L<<31)}.

@item 
You can no longer disable assertions by assigning to @code{__debug__}.

@item 
The Distutils @code{setup()} function has gained various new keyword arguments
such as @emph{depends}.  Old versions of the Distutils will abort if passed unknown
keywords.  A solution is to check for the presence of the new
@code{get_distutil_options()} function in your @code{setup.py} and only uses the
new keywords with a version of the Distutils that supports them:

@example
from distutils import core

kw = @{'sources': 'foo.c', ...@}
if hasattr(core, 'get_distutil_options'):
    kw['depends'] = ['foo.h']
ext = Extension(**kw)
@end example

@item 
Using @code{None} as a variable name will now result in a @ref{4c7,,SyntaxWarning}
warning.

@item 
Names of extension types defined by the modules included with Python now
contain the module and a @code{'.'} in front of the type name.
@end itemize

@c ======================================================================

@node Acknowledgements<5>,,Porting to Python 2 3,What’s New in Python 2 3
@anchor{whatsnew/2 3 acknowledgements}@anchor{101a}@anchor{whatsnew/2 3 acks}@anchor{101b}
@subsection Acknowledgements


The author would like to thank the following people for offering suggestions,
corrections and assistance with various drafts of this article: Jeff Bauer,
Simon Brunning, Brett Cannon, Michael Chermside, Andrew Dalke, Scott David
Daniels, Fred L. Drake, Jr., David Fraser,  Kelly Gerber, Raymond Hettinger,
Michael Hudson, Chris Lambert, Detlef Lannert, Martin von Löwis, Andrew
MacIntyre, Lalo Martins, Chad Netzer, Gustavo Niemeyer, Neal Norwitz, Hans
Nowak, Chris Reedy, Francesco Ricciardi, Vinay Sajip, Neil Schemenauer, Roman
Suzi, Jason Tishler, Just van Rossum.

@node What’s New in Python 2 2,What’s New in Python 2 1,What’s New in Python 2 3,What’s New in Python
@anchor{whatsnew/2 2 doc}@anchor{101c}@anchor{whatsnew/2 2 what-s-new-in-python-2-2}@anchor{101d}
@section What’s New in Python 2.2


@*Author: 
A.M. Kuchling


@c $Id: whatsnew22.tex 37315 2004-09-10 19:33:00Z akuchling $

@menu
* Introduction:: 
* PEPs 252 and 253; Type and Class Changes: PEPs 252 and 253 Type and Class Changes. 
* PEP 234; Iterators: PEP 234 Iterators. 
* PEP 255; Simple Generators: PEP 255 Simple Generators<2>. 
* PEP 237; Unifying Long Integers and Integers: PEP 237 Unifying Long Integers and Integers<2>. 
* PEP 238; Changing the Division Operator: PEP 238 Changing the Division Operator. 
* Unicode Changes:: 
* PEP 227; Nested Scopes: PEP 227 Nested Scopes. 
* New and Improved Modules: New and Improved Modules<3>. 
* Interpreter Changes and Fixes:: 
* Other Changes and Fixes: Other Changes and Fixes<3>. 
* Acknowledgements: Acknowledgements<6>. 

@end menu

@node Introduction,PEPs 252 and 253 Type and Class Changes,,What’s New in Python 2 2
@anchor{whatsnew/2 2 introduction}@anchor{101e}
@subsection Introduction


This article explains the new features in Python 2.2.2, released on October 14,
2002.  Python 2.2.2 is a bugfix release of Python 2.2, originally released on
December 21, 2001.

Python 2.2 can be thought of as the “cleanup release”.  There are some features
such as generators and iterators that are completely new, but most of the
changes, significant and far-reaching though they may be, are aimed at cleaning
up irregularities and dark corners of the language design.

This article doesn’t attempt to provide a complete specification of the new
features, but instead provides a convenient overview.  For full details, you
should refer to the documentation for Python 2.2, such as the Python Library Reference@footnote{https://docs.python.org/2.2/lib/lib.html} and the Python Reference Manual@footnote{https://docs.python.org/2.2/ref/ref.html}.  If you want to
understand the complete implementation and design rationale for a change, refer
to the PEP for a particular new feature.

@c see also, now defunct
@c 
@c http://www.unixreview.com/documents/s=1356/urm0109h/0109h.htm
@c    "What's So Special About Python 2.2?" is also about the new 2.2 features, and
@c    was written by Cameron Laird and Kathryn Soraiz.

@c ======================================================================

@node PEPs 252 and 253 Type and Class Changes,PEP 234 Iterators,Introduction,What’s New in Python 2 2
@anchor{whatsnew/2 2 peps-252-and-253-type-and-class-changes}@anchor{101f}
@subsection PEPs 252 and 253: Type and Class Changes


The largest and most far-reaching changes in Python 2.2 are to Python’s model of
objects and classes.  The changes should be backward compatible, so it’s likely
that your code will continue to run unchanged, but the changes provide some
amazing new capabilities. Before beginning this, the longest and most
complicated section of this article, I’ll provide an overview of the changes and
offer some comments.

A long time ago I wrote a web page listing flaws in Python’s design.  One of the
most significant flaws was that it’s impossible to subclass Python types
implemented in C.  In particular, it’s not possible to subclass built-in types,
so you can’t just subclass, say, lists in order to add a single useful method to
them. The @code{UserList} module provides a class that supports all of the
methods of lists and that can be subclassed further, but there’s lots of C code
that expects a regular Python list and won’t accept a @code{UserList}
instance.

Python 2.2 fixes this, and in the process adds some exciting new capabilities.
A brief summary:


@itemize *

@item 
You can subclass built-in types such as lists and even integers, and your
subclasses should work in every place that requires the original type.

@item 
It’s now possible to define static and class methods, in addition to the
instance methods available in previous versions of Python.

@item 
It’s also possible to automatically call methods on accessing or setting an
instance attribute by using a new mechanism called @emph{properties}.  Many uses
of @code{__getattr__()} can be rewritten to use properties instead, making the
resulting code simpler and faster.  As a small side benefit, attributes can now
have docstrings, too.

@item 
The list of legal attributes for an instance can be limited to a particular
set using @emph{slots}, making it possible to safeguard against typos and
perhaps make more optimizations possible in future versions of Python.
@end itemize

Some users have voiced concern about all these changes.  Sure, they say, the new
features are neat and lend themselves to all sorts of tricks that weren’t
possible in previous versions of Python, but they also make the language more
complicated.  Some people have said that they’ve always recommended Python for
its simplicity, and feel that its simplicity is being lost.

Personally, I think there’s no need to worry.  Many of the new features are
quite esoteric, and you can write a lot of Python code without ever needed to be
aware of them.  Writing a simple class is no more difficult than it ever was, so
you don’t need to bother learning or teaching them unless they’re actually
needed.  Some very complicated tasks that were previously only possible from C
will now be possible in pure Python, and to my mind that’s all for the better.

I’m not going to attempt to cover every single corner case and small change that
were required to make the new features work.  Instead this section will paint
only the broad strokes.  See section @ref{1020,,Related Links}, “Related Links”, for
further sources of information about Python 2.2’s new object model.

@menu
* Old and New Classes:: 
* Descriptors:: 
* Multiple Inheritance; The Diamond Rule: Multiple Inheritance The Diamond Rule. 
* Attribute Access:: 
* Related Links:: 

@end menu

@node Old and New Classes,Descriptors,,PEPs 252 and 253 Type and Class Changes
@anchor{whatsnew/2 2 old-and-new-classes}@anchor{1021}
@subsubsection Old and New Classes


First, you should know that Python 2.2 really has two kinds of classes: classic
or old-style classes, and new-style classes.  The old-style class model is
exactly the same as the class model in earlier versions of Python.  All the new
features described in this section apply only to new-style classes. This
divergence isn’t intended to last forever; eventually old-style classes will be
dropped, possibly in Python 3.0.

So how do you define a new-style class?  You do it by subclassing an existing
new-style class.  Most of Python’s built-in types, such as integers, lists,
dictionaries, and even files, are new-style classes now.  A new-style class
named @ref{5a5,,object}, the base class for all built-in types, has also been
added so if no built-in type is suitable, you can just subclass
@ref{5a5,,object}:

@example
class C(object):
    def __init__ (self):
        ...
    ...
@end example

This means that @ref{e72,,class} statements that don’t have any base classes are
always classic classes in Python 2.2.  (Actually you can also change this by
setting a module-level variable named @code{__metaclass__} — see 
@geindex Python Enhancement Proposals; PEP 253
PEP 253@footnote{https://www.python.org/dev/peps/pep-0253}
for the details — but it’s easier to just subclass @ref{5a5,,object}.)

The type objects for the built-in types are available as built-ins, named using
a clever trick.  Python has always had built-in functions named @ref{1b1,,int()},
@ref{256,,float()}, and @ref{2ef,,str()}.  In 2.2, they aren’t functions any more, but
type objects that behave as factories when called.

@example
>>> int
<type 'int'>
>>> int('123')
123
@end example

To make the set of types complete, new type objects such as @ref{3b0,,dict()} and
@code{file()} have been added.  Here’s a more interesting example, adding a
@code{lock()} method to file objects:

@example
class LockableFile(file):
    def lock (self, operation, length=0, start=0, whence=0):
        import fcntl
        return fcntl.lockf(self.fileno(), operation,
                           length, start, whence)
@end example

The now-obsolete @code{posixfile} module contained a class that emulated all of
a file object’s methods and also added a @code{lock()} method, but this class
couldn’t be passed to internal functions that expected a built-in file,
something which is possible with our new @code{LockableFile}.

@node Descriptors,Multiple Inheritance The Diamond Rule,Old and New Classes,PEPs 252 and 253 Type and Class Changes
@anchor{whatsnew/2 2 descriptors}@anchor{1022}
@subsubsection Descriptors


In previous versions of Python, there was no consistent way to discover what
attributes and methods were supported by an object. There were some informal
conventions, such as defining @code{__members__} and @code{__methods__}
attributes that were lists of names, but often the author of an extension type
or a class wouldn’t bother to define them.  You could fall back on inspecting
the @ref{7a4,,__dict__} of an object, but when class inheritance or an arbitrary
@code{__getattr__()} hook were in use this could still be inaccurate.

The one big idea underlying the new class model is that an API for describing
the attributes of an object using @emph{descriptors} has been formalized.
Descriptors specify the value of an attribute, stating whether it’s a method or
a field.  With the descriptor API, static methods and class methods become
possible, as well as more exotic constructs.

Attribute descriptors are objects that live inside class objects, and have a few
attributes of their own:


@itemize *

@item 
@ref{e70,,__name__} is the attribute’s name.

@item 
@code{__doc__} is the attribute’s docstring.

@item 
@code{__get__(object)} is a method that retrieves the attribute value from
@emph{object}.

@item 
@code{__set__(object, value)} sets the attribute on @emph{object} to @emph{value}.

@item 
@code{__delete__(object, value)} deletes the @emph{value}  attribute of @emph{object}.
@end itemize

For example, when you write @code{obj.x}, the steps that Python actually performs
are:

@example
descriptor = obj.__class__.x
descriptor.__get__(obj)
@end example

For methods, @code{descriptor.__get__()} returns a temporary object that’s
callable, and wraps up the instance and the method to be called on it. This is
also why static methods and class methods are now possible; they have
descriptors that wrap up just the method, or the method and the class.  As a
brief explanation of these new kinds of methods, static methods aren’t passed
the instance, and therefore resemble regular functions.  Class methods are
passed the class of the object, but not the object itself.  Static and class
methods are defined like this:

@example
class C(object):
    def f(arg1, arg2):
        ...
    f = staticmethod(f)

    def g(cls, arg1, arg2):
        ...
    g = classmethod(g)
@end example

The @ref{253,,staticmethod()} function takes the function @code{f()}, and returns it
wrapped up in a descriptor so it can be stored in the class object.  You might
expect there to be special syntax for creating such methods (@code{def static f},
@code{defstatic f()}, or something like that) but no such syntax has been defined
yet; that’s been left for future versions of Python.

More new features, such as slots and properties, are also implemented as new
kinds of descriptors, and it’s not difficult to write a descriptor class that
does something novel.  For example, it would be possible to write a descriptor
class that made it possible to write Eiffel-style preconditions and
postconditions for a method.  A class that used this feature might be defined
like this:

@example
from eiffel import eiffelmethod

class C(object):
    def f(self, arg1, arg2):
        # The actual function
        ...
    def pre_f(self):
        # Check preconditions
        ...
    def post_f(self):
        # Check postconditions
        ...

    f = eiffelmethod(f, pre_f, post_f)
@end example

Note that a person using the new @code{eiffelmethod()} doesn’t have to understand
anything about descriptors.  This is why I think the new features don’t increase
the basic complexity of the language. There will be a few wizards who need to
know about it in order to write @code{eiffelmethod()} or the ZODB or whatever,
but most users will just write code on top of the resulting libraries and ignore
the implementation details.

@node Multiple Inheritance The Diamond Rule,Attribute Access,Descriptors,PEPs 252 and 253 Type and Class Changes
@anchor{whatsnew/2 2 multiple-inheritance-the-diamond-rule}@anchor{1023}
@subsubsection Multiple Inheritance: The Diamond Rule


Multiple inheritance has also been made more useful through changing the rules
under which names are resolved.  Consider this set of classes (diagram taken
from 
@geindex Python Enhancement Proposals; PEP 253
PEP 253@footnote{https://www.python.org/dev/peps/pep-0253} by Guido van Rossum):

@example
      class A:
        ^ ^  def save(self): ...
       /   \
      /     \
     /       \
    /         \
class B     class C:
    ^         ^  def save(self): ...
     \       /
      \     /
       \   /
        \ /
      class D
@end example

The lookup rule for classic classes is simple but not very smart; the base
classes are searched depth-first, going from left to right.  A reference to
@code{D.save()} will search the classes @code{D}, @code{B}, and then
@code{A}, where @code{save()} would be found and returned.  @code{C.save()}
would never be found at all.  This is bad, because if @code{C}’s @code{save()}
method is saving some internal state specific to @code{C}, not calling it will
result in that state never getting saved.

New-style classes follow a different algorithm that’s a bit more complicated to
explain, but does the right thing in this situation. (Note that Python 2.3
changes this algorithm to one that produces the same results in most cases, but
produces more useful results for really complicated inheritance graphs.)


@enumerate 

@item 
List all the base classes, following the classic lookup rule and include a
class multiple times if it’s visited repeatedly.  In the above example, the list
of visited classes is [@code{D}, @code{B}, @code{A}, @code{C},
@code{A}].

@item 
Scan the list for duplicated classes.  If any are found, remove all but one
occurrence, leaving the @emph{last} one in the list.  In the above example, the list
becomes [@code{D}, @code{B}, @code{C}, @code{A}] after dropping
duplicates.
@end enumerate

Following this rule, referring to @code{D.save()} will return @code{C.save()},
which is the behaviour we’re after.  This lookup rule is the same as the one
followed by Common Lisp.  A new built-in function, @ref{78c,,super()}, provides a way
to get at a class’s superclasses without having to reimplement Python’s
algorithm. The most commonly used form will be  @code{super(class, obj)}, which
returns  a bound superclass object (not the actual class object).  This form
will be used in methods to call a method in the superclass; for example,
@code{D}’s @code{save()} method would look like this:

@example
class D (B,C):
    def save (self):
        # Call superclass .save()
        super(D, self).save()
        # Save D's private information here
        ...
@end example

@ref{78c,,super()} can also return unbound superclass objects when called as
@code{super(class)} or @code{super(class1, class2)}, but this probably won’t
often be useful.

@node Attribute Access,Related Links,Multiple Inheritance The Diamond Rule,PEPs 252 and 253 Type and Class Changes
@anchor{whatsnew/2 2 attribute-access}@anchor{1024}
@subsubsection Attribute Access


A fair number of sophisticated Python classes define hooks for attribute access
using @code{__getattr__()}; most commonly this is done for convenience, to make
code more readable by automatically mapping an attribute access such as
@code{obj.parent} into a method call such as @code{obj.get_parent}.  Python 2.2 adds
some new ways of controlling attribute access.

First, @code{__getattr__(attr_name)} is still supported by new-style classes,
and nothing about it has changed.  As before, it will be called when an attempt
is made to access @code{obj.foo} and no attribute named @code{foo} is found in the
instance’s dictionary.

New-style classes also support a new method,
@code{__getattribute__(attr_name)}.  The difference between the two methods is
that @code{__getattribute__()} is @emph{always} called whenever any attribute is
accessed, while the old @code{__getattr__()} is only called if @code{foo} isn’t
found in the instance’s dictionary.

However, Python 2.2’s support for @emph{properties} will often be a simpler way
to trap attribute references.  Writing a @code{__getattr__()} method is
complicated because to avoid recursion you can’t use regular attribute accesses
inside them, and instead have to mess around with the contents of
@ref{7a4,,__dict__}. @code{__getattr__()} methods also end up being called by Python
when it checks for other methods such as @code{__repr__()} or @code{__coerce__()},
and so have to be written with this in mind. Finally, calling a function on
every attribute access results in a sizable performance loss.

@ref{501,,property} is a new built-in type that packages up three functions that
get, set, or delete an attribute, and a docstring.  For example, if you want to
define a @code{size} attribute that’s computed, but also settable, you could
write:

@example
class C(object):
    def get_size (self):
        result = ... computation ...
        return result
    def set_size (self, size):
        ... compute something based on the size
        and set internal state appropriately ...

    # Define a property.  The 'delete this attribute'
    # method is defined as None, so the attribute
    # can't be deleted.
    size = property(get_size, set_size,
                    None,
                    "Storage size of this instance")
@end example

That is certainly clearer and easier to write than a pair of
@code{__getattr__()}/@code{__setattr__()} methods that check for the @code{size}
attribute and handle it specially while retrieving all other attributes from the
instance’s @ref{7a4,,__dict__}.  Accesses to @code{size} are also the only ones
which have to perform the work of calling a function, so references to other
attributes run at their usual speed.

Finally, it’s possible to constrain the list of attributes that can be
referenced on an object using the new @ref{1025,,__slots__} class attribute. Python
objects are usually very dynamic; at any time it’s possible to define a new
attribute on an instance by just doing @code{obj.new_attr=1}.   A new-style class
can define a class attribute named @ref{1025,,__slots__} to limit the legal
attributes  to a particular set of names.  An example will make this clear:

@example
>>> class C(object):
...     __slots__ = ('template', 'name')
...
>>> obj = C()
>>> print obj.template
None
>>> obj.template = 'Test'
>>> print obj.template
Test
>>> obj.newattr = None
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
AttributeError: 'C' object has no attribute 'newattr'
@end example

Note how you get an @ref{163,,AttributeError} on the attempt to assign to an
attribute not listed in @ref{1025,,__slots__}.

@node Related Links,,Attribute Access,PEPs 252 and 253 Type and Class Changes
@anchor{whatsnew/2 2 related-links}@anchor{1026}@anchor{whatsnew/2 2 sect-rellinks}@anchor{1020}
@subsubsection Related Links


This section has just been a quick overview of the new features, giving enough
of an explanation to start you programming, but many details have been
simplified or ignored.  Where should you go to get a more complete picture?

@indicateurl{https://docs.python.org/dev/howto/descriptor.html} is a lengthy tutorial introduction to
the descriptor features, written by Guido van Rossum. If my description has
whetted your appetite, go read this tutorial next, because it goes into much
more detail about the new features while still remaining quite easy to read.

Next, there are two relevant PEPs, 
@geindex Python Enhancement Proposals; PEP 252
PEP 252@footnote{https://www.python.org/dev/peps/pep-0252} and 
@geindex Python Enhancement Proposals; PEP 253
PEP 253@footnote{https://www.python.org/dev/peps/pep-0253}.  
@geindex Python Enhancement Proposals; PEP 252
PEP 252@footnote{https://www.python.org/dev/peps/pep-0252} is
titled “Making Types Look More Like Classes”, and covers the descriptor API.
@geindex Python Enhancement Proposals; PEP 253
PEP 253@footnote{https://www.python.org/dev/peps/pep-0253} is titled “Subtyping Built-in Types”, and describes the changes to
type objects that make it possible to subtype built-in objects.  
@geindex Python Enhancement Proposals; PEP 253
PEP 253@footnote{https://www.python.org/dev/peps/pep-0253} is
the more complicated PEP of the two, and at a few points the necessary
explanations of types and meta-types may cause your head to explode.  Both PEPs
were written and implemented by Guido van Rossum, with substantial assistance
from the rest of the Zope Corp. team.

Finally, there’s the ultimate authority: the source code.  Most of the machinery
for the type handling is in @code{Objects/typeobject.c}, but you should only
resort to it after all other avenues have been exhausted, including posting a
question to python-list or python-dev.

@c ======================================================================

@node PEP 234 Iterators,PEP 255 Simple Generators<2>,PEPs 252 and 253 Type and Class Changes,What’s New in Python 2 2
@anchor{whatsnew/2 2 pep-234-iterators}@anchor{1027}
@subsection PEP 234: Iterators


Another significant addition to 2.2 is an iteration interface at both the C and
Python levels.  Objects can define how they can be looped over by callers.

In Python versions up to 2.1, the usual way to make @code{for item in obj} work is
to define a @code{__getitem__()} method that looks something like this:

@example
def __getitem__(self, index):
    return <next item>
@end example

@code{__getitem__()} is more properly used to define an indexing operation on an
object so that you can write @code{obj[5]} to retrieve the sixth element.  It’s a
bit misleading when you’re using this only to support @ref{161,,for} loops.
Consider some file-like object that wants to be looped over; the @emph{index}
parameter is essentially meaningless, as the class probably assumes that a
series of @code{__getitem__()} calls will be made with @emph{index} incrementing by
one each time.  In other words, the presence of the @code{__getitem__()} method
doesn’t mean that using @code{file[5]}  to randomly access the sixth element will
work, though it really should.

In Python 2.2, iteration can be implemented separately, and @code{__getitem__()}
methods can be limited to classes that really do support random access.  The
basic idea of iterators is  simple.  A new built-in function, @code{iter(obj)}
or @code{iter(C, sentinel)}, is used to get an iterator. @code{iter(obj)} returns
an iterator for the object @emph{obj}, while @code{iter(C, sentinel)} returns an
iterator that will invoke the callable object @emph{C} until it returns @emph{sentinel} to
signal that the iterator is done.

Python classes can define an @code{__iter__()} method, which should create and
return a new iterator for the object; if the object is its own iterator, this
method can just return @code{self}.  In particular, iterators will usually be their
own iterators.  Extension types implemented in C can implement a @ref{1028,,tp_iter}
function in order to return an iterator, and extension types that want to behave
as iterators can define a @ref{1029,,tp_iternext} function.

So, after all this, what do iterators actually do?  They have one required
method, @ref{252,,next()}, which takes no arguments and returns the next value.  When
there are no more values to be returned, calling @ref{252,,next()} should raise the
@ref{736,,StopIteration} exception.

@example
>>> L = [1,2,3]
>>> i = iter(L)
>>> print i
<iterator object at 0x8116870>
>>> i.next()
1
>>> i.next()
2
>>> i.next()
3
>>> i.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
StopIteration
>>>
@end example

In 2.2, Python’s @ref{161,,for} statement no longer expects a sequence; it
expects something for which @ref{251,,iter()} will return an iterator. For backward
compatibility and convenience, an iterator is automatically constructed for
sequences that don’t implement @code{__iter__()} or a @ref{1028,,tp_iter} slot, so
@code{for i in [1,2,3]} will still work.  Wherever the Python interpreter loops
over a sequence, it’s been changed to use the iterator protocol.  This means you
can do things like this:

@example
>>> L = [1,2,3]
>>> i = iter(L)
>>> a,b,c = i
>>> a,b,c
(1, 2, 3)
@end example

Iterator support has been added to some of Python’s basic types.   Calling
@ref{251,,iter()} on a dictionary will return an iterator which loops over its keys:

@example
>>> m = @{'Jan': 1, 'Feb': 2, 'Mar': 3, 'Apr': 4, 'May': 5, 'Jun': 6,
...      'Jul': 7, 'Aug': 8, 'Sep': 9, 'Oct': 10, 'Nov': 11, 'Dec': 12@}
>>> for key in m: print key, m[key]
...
Mar 3
Feb 2
Aug 8
Sep 9
May 5
Jun 6
Jul 7
Jan 1
Apr 4
Nov 11
Dec 12
Oct 10
@end example

That’s just the default behaviour.  If you want to iterate over keys, values, or
key/value pairs, you can explicitly call the @code{iterkeys()},
@code{itervalues()}, or @code{iteritems()} methods to get an appropriate iterator.
In a minor related change, the @ref{2fc,,in} operator now works on dictionaries,
so @code{key in dict} is now equivalent to @code{dict.has_key(key)}.

Files also provide an iterator, which calls the @ref{d9,,readline()} method until
there are no more lines in the file.  This means you can now read each line of a
file using code like this:

@example
for line in file:
    # do something for each line
    ...
@end example

Note that you can only go forward in an iterator; there’s no way to get the
previous element, reset the iterator, or make a copy of it. An iterator object
could provide such additional capabilities, but the iterator protocol only
requires a @ref{252,,next()} method.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 234

@item PEP 234@footnote{https://www.python.org/dev/peps/pep-0234} - Iterators

Written by Ka-Ping Yee and GvR; implemented  by the Python Labs crew, mostly by
GvR and Tim Peters.
@end table


@c ======================================================================

@node PEP 255 Simple Generators<2>,PEP 237 Unifying Long Integers and Integers<2>,PEP 234 Iterators,What’s New in Python 2 2
@anchor{whatsnew/2 2 pep-255-simple-generators}@anchor{102a}
@subsection PEP 255: Simple Generators


Generators are another new feature, one that interacts with the introduction of
iterators.

You’re doubtless familiar with how function calls work in Python or C.  When you
call a function, it gets a private namespace where its local variables are
created.  When the function reaches a @ref{4c6,,return} statement, the local
variables are destroyed and the resulting value is returned to the caller.  A
later call to the same function will get a fresh new set of local variables.
But, what if the local variables weren’t thrown away on exiting a function?
What if you could later resume the function where it left off?  This is what
generators provide; they can be thought of as resumable functions.

Here’s the simplest example of a generator function:

@example
def generate_ints(N):
    for i in range(N):
        yield i
@end example

A new keyword, @ref{4c5,,yield}, was introduced for generators.  Any function
containing a @code{yield} statement is a generator function; this is
detected by Python’s bytecode compiler which compiles the function specially as
a result.  Because a new keyword was introduced, generators must be explicitly
enabled in a module by including a @code{from __future__ import generators}
statement near the top of the module’s source code.  In Python 2.3 this
statement will become unnecessary.

When you call a generator function, it doesn’t return a single value; instead it
returns a generator object that supports the iterator protocol.  On executing
the @ref{4c5,,yield} statement, the generator outputs the value of @code{i},
similar to a @ref{4c6,,return} statement.  The big difference between
@code{yield} and a @code{return} statement is that on reaching a
@code{yield} the generator’s state of execution is suspended and local
variables are preserved.  On the next call to the generator’s @code{next()} method,
the function will resume executing immediately after the @code{yield}
statement.  (For complicated reasons, the @code{yield} statement isn’t
allowed inside the @code{try} block of a
@ref{f6d,,try}…@ref{4c1,,finally} statement; read 
@geindex Python Enhancement Proposals; PEP 255
PEP 255@footnote{https://www.python.org/dev/peps/pep-0255} for a full
explanation of the interaction between @code{yield} and exceptions.)

Here’s a sample usage of the @code{generate_ints()} generator:

@example
>>> gen = generate_ints(3)
>>> gen
<generator object at 0x8117f90>
>>> gen.next()
0
>>> gen.next()
1
>>> gen.next()
2
>>> gen.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 2, in generate_ints
StopIteration
@end example

You could equally write @code{for i in generate_ints(5)}, or @code{a,b,c =
generate_ints(3)}.

Inside a generator function, the @ref{4c6,,return} statement can only be used
without a value, and signals the end of the procession of values; afterwards the
generator cannot return any further values. @code{return} with a value, such
as @code{return 5}, is a syntax error inside a generator function.  The end of the
generator’s results can also be indicated by raising @ref{736,,StopIteration}
manually, or by just letting the flow of execution fall off the bottom of the
function.

You could achieve the effect of generators manually by writing your own class
and storing all the local variables of the generator as instance variables.  For
example, returning a list of integers could be done by setting @code{self.count} to
0, and having the @ref{252,,next()} method increment @code{self.count} and return it.
However, for a moderately complicated generator, writing a corresponding class
would be much messier. @code{Lib/test/test_generators.py} contains a number of
more interesting examples.  The simplest one implements an in-order traversal of
a tree using generators recursively.

@example
# A recursive generator that generates Tree leaves in in-order.
def inorder(t):
    if t:
        for x in inorder(t.left):
            yield x
        yield t.label
        for x in inorder(t.right):
            yield x
@end example

Two other examples in @code{Lib/test/test_generators.py} produce solutions for
the N-Queens problem (placing $N$ queens on an $NxN$ chess board so that no
queen threatens another) and the Knight’s Tour (a route that takes a knight to
every square of an $NxN$ chessboard without visiting any square twice).

The idea of generators comes from other programming languages, especially Icon
(@indicateurl{https://www.cs.arizona.edu/icon/}), where the idea of generators is central.  In
Icon, every expression and function call behaves like a generator.  One example
from “An Overview of the Icon Programming Language” at
@indicateurl{https://www.cs.arizona.edu/icon/docs/ipd266.htm} gives an idea of what this looks
like:

@example
sentence := "Store it in the neighboring harbor"
if (i := find("or", sentence)) > 5 then write(i)
@end example

In Icon the @code{find()} function returns the indexes at which the substring
“or” is found: 3, 23, 33.  In the @ref{2fb,,if} statement, @code{i} is first
assigned a value of 3, but 3 is less than 5, so the comparison fails, and Icon
retries it with the second value of 23.  23 is greater than 5, so the comparison
now succeeds, and the code prints the value 23 to the screen.

Python doesn’t go nearly as far as Icon in adopting generators as a central
concept.  Generators are considered a new part of the core Python language, but
learning or using them isn’t compulsory; if they don’t solve any problems that
you have, feel free to ignore them. One novel feature of Python’s interface as
compared to Icon’s is that a generator’s state is represented as a concrete
object (the iterator) that can be passed around to other functions or stored in
a data structure.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 255

@item PEP 255@footnote{https://www.python.org/dev/peps/pep-0255} - Simple Generators

Written by Neil Schemenauer, Tim Peters, Magnus Lie Hetland.  Implemented mostly
by Neil Schemenauer and Tim Peters, with other fixes from the Python Labs crew.
@end table


@c ======================================================================

@node PEP 237 Unifying Long Integers and Integers<2>,PEP 238 Changing the Division Operator,PEP 255 Simple Generators<2>,What’s New in Python 2 2
@anchor{whatsnew/2 2 pep-237-unifying-long-integers-and-integers}@anchor{102b}
@subsection PEP 237: Unifying Long Integers and Integers


In recent versions, the distinction between regular integers, which are 32-bit
values on most machines, and long integers, which can be of arbitrary size, was
becoming an annoyance.  For example, on platforms that support files larger than
@code{2**32} bytes, the @code{tell()} method of file objects has to return a long
integer. However, there were various bits of Python that expected plain integers
and would raise an error if a long integer was provided instead.  For example,
in Python 1.5, only regular integers could be used as a slice index, and
@code{'abc'[1L:]} would raise a @ref{162,,TypeError} exception with the message ‘slice
index must be int’.

Python 2.2 will shift values from short to long integers as required. The ‘L’
suffix is no longer needed to indicate a long integer literal, as now the
compiler will choose the appropriate type.  (Using the ‘L’ suffix will be
discouraged in future 2.x versions of Python, triggering a warning in Python
2.4, and probably dropped in Python 3.0.)  Many operations that used to raise an
@ref{347,,OverflowError} will now return a long integer as their result.  For
example:

@example
>>> 1234567890123
1234567890123L
>>> 2 ** 64
18446744073709551616L
@end example

In most cases, integers and long integers will now be treated identically.  You
can still distinguish them with the @ref{89e,,type()} built-in function, but that’s
rarely needed.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 237

@item PEP 237@footnote{https://www.python.org/dev/peps/pep-0237} - Unifying Long Integers and Integers

Written by Moshe Zadka and Guido van Rossum.  Implemented mostly by Guido van
Rossum.
@end table


@c ======================================================================

@node PEP 238 Changing the Division Operator,Unicode Changes,PEP 237 Unifying Long Integers and Integers<2>,What’s New in Python 2 2
@anchor{whatsnew/2 2 pep-238-changing-the-division-operator}@anchor{102c}
@subsection PEP 238: Changing the Division Operator


The most controversial change in Python 2.2 heralds the start of an effort to
fix an old design flaw that’s been in Python from the beginning. Currently
Python’s division operator, @code{/}, behaves like C’s division operator when
presented with two integer arguments: it returns an integer result that’s
truncated down when there would be a fractional part.  For example, @code{3/2} is
1, not 1.5, and @code{(-1)/2} is -1, not -0.5.  This means that the results of
division can vary unexpectedly depending on the type of the two operands and
because Python is dynamically typed, it can be difficult to determine the
possible types of the operands.

(The controversy is over whether this is @emph{really} a design flaw, and whether
it’s worth breaking existing code to fix this.  It’s caused endless discussions
on python-dev, and in July 2001 erupted into a storm of acidly sarcastic
postings on @code{comp.lang.python}. I won’t argue for either side here
and will stick to describing what’s  implemented in 2.2.  Read 
@geindex Python Enhancement Proposals; PEP 238
PEP 238@footnote{https://www.python.org/dev/peps/pep-0238} for a
summary of arguments and counter-arguments.)

Because this change might break code, it’s being introduced very gradually.
Python 2.2 begins the transition, but the switch won’t be complete until Python
3.0.

First, I’ll borrow some terminology from 
@geindex Python Enhancement Proposals; PEP 238
PEP 238@footnote{https://www.python.org/dev/peps/pep-0238}.  “True division” is the
division that most non-programmers are familiar with: 3/2 is 1.5, 1/4 is 0.25,
and so forth.  “Floor division” is what Python’s @code{/} operator currently does
when given integer operands; the result is the floor of the value returned by
true division.  “Classic division” is the current mixed behaviour of @code{/}; it
returns the result of floor division when the operands are integers, and returns
the result of true division when one of the operands is a floating-point number.

Here are the changes 2.2 introduces:


@itemize *

@item 
A new operator, @code{//}, is the floor division operator. (Yes, we know it looks
like C++’s comment symbol.)  @code{//} @emph{always} performs floor division no matter
what the types of its operands are, so @code{1 // 2} is 0 and @code{1.0 // 2.0} is
also 0.0.

@code{//} is always available in Python 2.2; you don’t need to enable it using a
@code{__future__} statement.

@item 
By including a @code{from __future__ import division} in a module, the @code{/}
operator will be changed to return the result of true division, so @code{1/2} is
0.5.  Without the @code{__future__} statement, @code{/} still means classic division.
The default meaning of @code{/} will not change until Python 3.0.

@item 
Classes can define methods called @code{__truediv__()} and @code{__floordiv__()}
to overload the two division operators.  At the C level, there are also slots in
the @ref{f7b,,PyNumberMethods} structure so extension types can define the two
operators.

@item 
Python 2.2 supports some command-line arguments for testing whether code will
work with the changed division semantics.  Running python with @code{-Q
warn} will cause a warning to be issued whenever division is applied to two
integers.  You can use this to find code that’s affected by the change and fix
it.  By default, Python 2.2 will simply perform classic division without a
warning; the warning will be turned on by default in Python 2.3.
@end itemize


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 238

@item PEP 238@footnote{https://www.python.org/dev/peps/pep-0238} - Changing the Division Operator

Written by Moshe Zadka and  Guido van Rossum.  Implemented by Guido van Rossum..
@end table


@c ======================================================================

@node Unicode Changes,PEP 227 Nested Scopes,PEP 238 Changing the Division Operator,What’s New in Python 2 2
@anchor{whatsnew/2 2 unicode-changes}@anchor{102d}
@subsection Unicode Changes


Python’s Unicode support has been enhanced a bit in 2.2.  Unicode strings are
usually stored as UCS-2, as 16-bit unsigned integers. Python 2.2 can also be
compiled to use UCS-4, 32-bit unsigned integers, as its internal encoding by
supplying @code{--enable-unicode=ucs4} to the configure script.   (It’s also
possible to specify @code{--disable-unicode} to completely disable Unicode
support.)

When built to use UCS-4 (a “wide Python”), the interpreter can natively handle
Unicode characters from U+000000 to U+110000, so the range of legal values for
the @code{unichr()} function is expanded accordingly.  Using an interpreter
compiled to use UCS-2 (a “narrow Python”), values greater than 65535 will still
cause @code{unichr()} to raise a @ref{17c,,ValueError} exception. This is all
described in 
@geindex Python Enhancement Proposals; PEP 261
PEP 261@footnote{https://www.python.org/dev/peps/pep-0261}, “Support for ‘wide’ Unicode characters”; consult it for
further details.

Another change is simpler to explain. Since their introduction, Unicode strings
have supported an @code{encode()} method to convert the string to a selected
encoding such as UTF-8 or Latin-1.  A symmetric @code{decode([*encoding*])}
method has been added to 8-bit strings (though not to Unicode strings) in 2.2.
@code{decode()} assumes that the string is in the specified encoding and decodes
it, returning whatever is returned by the codec.

Using this new feature, codecs have been added for tasks not directly related to
Unicode.  For example, codecs have been added for uu-encoding, MIME’s base64
encoding, and compression with the @ref{14b,,zlib} module:

@example
>>> s = """Here is a lengthy piece of redundant, overly verbose,
... and repetitive text.
... """
>>> data = s.encode('zlib')
>>> data
'x\x9c\r\xc9\xc1\r\x80 \x10\x04\xc0?Ul...'
>>> data.decode('zlib')
'Here is a lengthy piece of redundant, overly verbose,\nand repetitive text.\n'
>>> print s.encode('uu')
begin 666 <data>
M2&5R92!I<R!A(&QE;F=T:'D@@<&EE8V4@@;V8@@<F5D=6YD86YT+"!O=F5R;'D@@
>=F5R8F]S92P*86YD(')E<&5T:71I=F4@@=&5X="X*

end
>>> "sheesh".encode('rot-13')
'furrfu'
@end example

To convert a class instance to Unicode, a @code{__unicode__()} method can be
defined by a class, analogous to @code{__str__()}.

@code{encode()}, @code{decode()}, and @code{__unicode__()} were implemented by
Marc-André Lemburg.  The changes to support using UCS-4 internally were
implemented by Fredrik Lundh and Martin von Löwis.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 261

@item PEP 261@footnote{https://www.python.org/dev/peps/pep-0261} - Support for ‘wide’ Unicode characters

Written by Paul Prescod.
@end table


@c ======================================================================

@node PEP 227 Nested Scopes,New and Improved Modules<3>,Unicode Changes,What’s New in Python 2 2
@anchor{whatsnew/2 2 pep-227-nested-scopes}@anchor{102e}
@subsection PEP 227: Nested Scopes


In Python 2.1, statically nested scopes were added as an optional feature, to be
enabled by a @code{from __future__ import nested_scopes} directive.  In 2.2 nested
scopes no longer need to be specially enabled, and are now always present.  The
rest of this section is a copy of the description of nested scopes from my
“What’s New in Python 2.1” document; if you read it when 2.1 came out, you can
skip the rest of this section.

The largest change introduced in Python 2.1, and made complete in 2.2, is to
Python’s scoping rules.  In Python 2.0, at any given time there are at most
three namespaces used to look up variable names: local, module-level, and the
built-in namespace.  This often surprised people because it didn’t match their
intuitive expectations.  For example, a nested recursive function definition
doesn’t work:

@example
def f():
    ...
    def g(value):
        ...
        return g(value-1) + 1
    ...
@end example

The function @code{g()} will always raise a @ref{21f,,NameError} exception, because
the binding of the name @code{g} isn’t in either its local namespace or in the
module-level namespace.  This isn’t much of a problem in practice (how often do
you recursively define interior functions like this?), but this also made using
the @ref{e42,,lambda} expression clumsier, and this was a problem in practice.
In code which uses @code{lambda} you can often find local variables being
copied by passing them as the default values of arguments.

@example
def find(self, name):
    "Return list of any entries equal to 'name'"
    L = filter(lambda x, name=name: x == name,
               self.list_attribute)
    return L
@end example

The readability of Python code written in a strongly functional style suffers
greatly as a result.

The most significant change to Python 2.2 is that static scoping has been added
to the language to fix this problem.  As a first effect, the @code{name=name}
default argument is now unnecessary in the above example.  Put simply, when a
given variable name is not assigned a value within a function (by an assignment,
or the @ref{fb8,,def}, @ref{e72,,class}, or @ref{e35,,import} statements),
references to the variable will be looked up in the local namespace of the
enclosing scope.  A more detailed explanation of the rules, and a dissection of
the implementation, can be found in the PEP.

This change may cause some compatibility problems for code where the same
variable name is used both at the module level and as a local variable within a
function that contains further function definitions. This seems rather unlikely
though, since such code would have been pretty confusing to read in the first
place.

One side effect of the change is that the @code{from module import *} and
@code{exec} statements have been made illegal inside a function scope under
certain conditions.  The Python reference manual has said all along that @code{from
module import *} is only legal at the top level of a module, but the CPython
interpreter has never enforced this before.  As part of the implementation of
nested scopes, the compiler which turns Python source into bytecodes has to
generate different code to access variables in a containing scope.  @code{from
module import *} and @code{exec} make it impossible for the compiler to
figure this out, because they add names to the local namespace that are
unknowable at compile time. Therefore, if a function contains function
definitions or @ref{e42,,lambda} expressions with free variables, the compiler
will flag this by raising a @ref{21a,,SyntaxError} exception.

To make the preceding explanation a bit clearer, here’s an example:

@example
x = 1
def f():
    # The next line is a syntax error
    exec 'x=2'
    def g():
        return x
@end example

Line 4 containing the @code{exec} statement is a syntax error, since
@code{exec} would define a new local variable named @code{x} whose value should
be accessed by @code{g()}.

This shouldn’t be much of a limitation, since @code{exec} is rarely used in
most Python code (and when it is used, it’s often a sign of a poor design
anyway).


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 227

@item PEP 227@footnote{https://www.python.org/dev/peps/pep-0227} - Statically Nested Scopes

Written and implemented by Jeremy Hylton.
@end table


@c ======================================================================

@node New and Improved Modules<3>,Interpreter Changes and Fixes,PEP 227 Nested Scopes,What’s New in Python 2 2
@anchor{whatsnew/2 2 new-and-improved-modules}@anchor{102f}
@subsection New and Improved Modules



@itemize *

@item 
The @code{xmlrpclib} module was contributed to the standard library by Fredrik
Lundh, providing support for writing XML-RPC clients.  XML-RPC is a simple
remote procedure call protocol built on top of HTTP and XML. For example, the
following snippet retrieves a list of RSS channels from the O’Reilly Network,
and then  lists the recent headlines for one channel:

@example
import xmlrpclib
s = xmlrpclib.Server(
      'http://www.oreillynet.com/meerkat/xml-rpc/server.php')
channels = s.meerkat.getChannels()
# channels is a list of dictionaries, like this:
# [@{'id': 4, 'title': 'Freshmeat Daily News'@}
#  @{'id': 190, 'title': '32Bits Online'@},
#  @{'id': 4549, 'title': '3DGamers'@}, ... ]

# Get the items for one channel
items = s.meerkat.getItems( @{'channel': 4@} )

# 'items' is another list of dictionaries, like this:
# [@{'link': 'http://freshmeat.net/releases/52719/',
#   'description': 'A utility which converts HTML to XSL FO.',
#   'title': 'html2fo 0.3 (Default)'@}, ... ]
@end example

The @code{SimpleXMLRPCServer} module makes it easy to create straightforward
XML-RPC servers.  See @indicateurl{http://xmlrpc.scripting.com/} for more information about XML-RPC.

@item 
The new @ref{8a,,hmac} module implements the HMAC algorithm described by
@geindex RFC; RFC 2104
RFC 2104@footnote{https://tools.ietf.org/html/rfc2104.html}. (Contributed by Gerhard Häring.)

@item 
Several functions that originally returned lengthy tuples now return
pseudo-sequences that still behave like tuples but also have mnemonic attributes such
as memberst_mtime or @code{tm_year}. The enhanced functions include
@ref{ef,,stat()}, @code{fstat()}, @code{statvfs()}, and @code{fstatvfs()} in the
@ref{c0,,os} module, and @code{localtime()}, @code{gmtime()}, and @code{strptime()} in
the @ref{10a,,time} module.

For example, to obtain a file’s size using the old tuples, you’d end up writing
something like @code{file_size = os.stat(filename)[stat.ST_SIZE]}, but now this can
be written more clearly as @code{file_size = os.stat(filename).st_size}.

The original patch for this feature was contributed by Nick Mathewson.

@item 
The Python profiler has been extensively reworked and various errors in its
output have been corrected.  (Contributed by Fred L. Drake, Jr. and Tim Peters.)

@item 
The @ref{ea,,socket} module can be compiled to support IPv6; specify the
@code{--enable-ipv6} option to Python’s configure script.  (Contributed by
Jun-ichiro “itojun” Hagino.)

@item 
Two new format characters were added to the @ref{f3,,struct} module for 64-bit
integers on platforms that support the C long  long type.  @code{q} is for
a signed 64-bit integer, and @code{Q} is for an unsigned one.  The value is
returned in Python’s long integer type.  (Contributed by Tim Peters.)

@item 
In the interpreter’s interactive mode, there’s a new built-in function
@ref{3bc,,help()} that uses the @ref{d4,,pydoc} module introduced in Python 2.1 to
provide interactive help. @code{help(object)} displays any available help text
about @emph{object}.  @ref{3bc,,help()} with no argument puts you in an online help
utility, where you can enter the names of functions, classes, or modules to read
their help text. (Contributed by Guido van Rossum, using Ka-Ping Yee’s
@ref{d4,,pydoc} module.)

@item 
Various bugfixes and performance improvements have been made to the SRE engine
underlying the @ref{d8,,re} module.  For example, the @ref{72e,,re.sub()} and
@ref{699,,re.split()} functions have been rewritten in C.  Another contributed patch
speeds up certain Unicode character ranges by a factor of two, and a new
@code{finditer()}  method that returns an iterator over all the non-overlapping
matches in  a given string.  (SRE is maintained by Fredrik Lundh.  The
BIGCHARSET patch was contributed by Martin von Löwis.)

@item 
The @ref{e8,,smtplib} module now supports 
@geindex RFC; RFC 2487
RFC 2487@footnote{https://tools.ietf.org/html/rfc2487.html}, “Secure SMTP over TLS”, so
it’s now possible to encrypt the SMTP traffic between a Python program and the
mail transport agent being handed a message.  @ref{e8,,smtplib} also supports SMTP
authentication.  (Contributed by Gerhard Häring.)

@item 
The @ref{93,,imaplib} module, maintained by Piers Lauder, has support for several
new extensions: the NAMESPACE extension defined in 
@geindex RFC; RFC 2342
RFC 2342@footnote{https://tools.ietf.org/html/rfc2342.html}, SORT, GETACL and
SETACL.  (Contributed by Anthony Baxter and Michel Pelletier.)

@item 
The @code{rfc822} module’s parsing of email addresses is now compliant with
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html}, an update to 
@geindex RFC; RFC 822
RFC 822@footnote{https://tools.ietf.org/html/rfc822.html}.  (The module’s name is @emph{not} going to be
changed to @code{rfc2822}.)  A new package, @ref{64,,email}, has also been added for
parsing and generating e-mail messages.  (Contributed by Barry Warsaw, and
arising out of his work on Mailman.)

@item 
The @ref{35,,difflib} module now contains a new @code{Differ} class for
producing human-readable lists of changes (a “delta”) between two sequences of
lines of text.  There are also two generator functions, @code{ndiff()} and
@code{restore()}, which respectively return a delta from two sequences, or one of
the original sequences from a delta. (Grunt work contributed by David Goodger,
from ndiff.py code by Tim Peters who then did the generatorization.)

@item 
New constants @code{ascii_letters}, @code{ascii_lowercase}, and
@code{ascii_uppercase} were added to the @ref{f1,,string} module.  There were
several modules in the standard library that used @code{string.letters} to
mean the ranges A-Za-z, but that assumption is incorrect when locales are in
use, because @code{string.letters} varies depending on the set of legal
characters defined by the current locale.  The buggy modules have all been fixed
to use @code{ascii_letters} instead. (Reported by an unknown person; fixed by
Fred L. Drake, Jr.)

@item 
The @ref{ae,,mimetypes} module now makes it easier to use alternative MIME-type
databases by the addition of a @code{MimeTypes} class, which takes a list of
filenames to be parsed.  (Contributed by Fred L. Drake, Jr.)

@item 
A @code{Timer} class was added to the @ref{109,,threading} module that allows
scheduling an activity to happen at some future time.  (Contributed by Itamar
Shtull-Trauring.)
@end itemize

@c ======================================================================

@node Interpreter Changes and Fixes,Other Changes and Fixes<3>,New and Improved Modules<3>,What’s New in Python 2 2
@anchor{whatsnew/2 2 interpreter-changes-and-fixes}@anchor{1030}
@subsection Interpreter Changes and Fixes


Some of the changes only affect people who deal with the Python interpreter at
the C level because they’re writing Python extension modules, embedding the
interpreter, or just hacking on the interpreter itself. If you only write Python
code, none of the changes described here will affect you very much.


@itemize *

@item 
Profiling and tracing functions can now be implemented in C, which can operate
at much higher speeds than Python-based functions and should reduce the overhead
of profiling and tracing.  This  will be of interest to authors of development
environments for Python.  Two new C functions were added to Python’s API,
@ref{1031,,PyEval_SetProfile()} and @ref{1032,,PyEval_SetTrace()}. The existing
@ref{1033,,sys.setprofile()} and @ref{1034,,sys.settrace()} functions still exist, and have
simply been changed to use the new C-level interface.  (Contributed by Fred L.
Drake, Jr.)

@item 
Another low-level API, primarily of interest to implementors of Python
debuggers and development tools, was added. @ref{1035,,PyInterpreterState_Head()} and
@ref{1036,,PyInterpreterState_Next()} let a caller walk through all the existing
interpreter objects; @ref{1037,,PyInterpreterState_ThreadHead()} and
@ref{1038,,PyThreadState_Next()} allow looping over all the thread states for a given
interpreter.  (Contributed by David Beazley.)

@item 
The C-level interface to the garbage collector has been changed to make it
easier to write extension types that support garbage collection and to debug
misuses of the functions. Various functions have slightly different semantics,
so a bunch of functions had to be renamed.  Extensions that use the old API will
still compile but will @emph{not} participate in garbage collection, so updating them
for 2.2 should be considered fairly high priority.

To upgrade an extension module to the new API, perform the following steps:

@item 
Rename @code{Py_TPFLAGS_GC()} to @code{PyTPFLAGS_HAVE_GC()}.

@item 

@table @asis

@item Use @ref{5c3,,PyObject_GC_New()} or @ref{5c4,,PyObject_GC_NewVar()} to allocate

objects, and @ref{1039,,PyObject_GC_Del()} to deallocate them.
@end table

@item 

@table @asis

@item Rename @code{PyObject_GC_Init()} to @ref{103a,,PyObject_GC_Track()} and

@code{PyObject_GC_Fini()} to @ref{103b,,PyObject_GC_UnTrack()}.
@end table

@item 
Remove @code{PyGC_HEAD_SIZE()} from object size calculations.

@item 
Remove calls to @code{PyObject_AS_GC()} and @code{PyObject_FROM_GC()}.

@item 
A new @code{et} format sequence was added to @ref{37f,,PyArg_ParseTuple()}; @code{et}
takes both a parameter and an encoding name, and converts the parameter to the
given encoding if the parameter turns out to be a Unicode string, or leaves it
alone if it’s an 8-bit string, assuming it to already be in the desired
encoding.  This differs from the @code{es} format character, which assumes that
8-bit strings are in Python’s default ASCII encoding and converts them to the
specified new encoding. (Contributed by M.-A. Lemburg, and used for the MBCS
support on Windows described in the following section.)

@item 
A different argument parsing function, @ref{103c,,PyArg_UnpackTuple()}, has been
added that’s simpler and presumably faster.  Instead of specifying a format
string, the caller simply gives the minimum and maximum number of arguments
expected, and a set of pointers to @ref{375,,PyObject}* variables that will be
filled in with argument values.

@item 
Two new flags @ref{1011,,METH_NOARGS} and @ref{103d,,METH_O} are available in method
definition tables to simplify implementation of methods with no arguments or a
single untyped argument. Calling such methods is more efficient than calling a
corresponding method that uses @ref{103e,,METH_VARARGS}.  Also, the old
@code{METH_OLDARGS} style of writing C methods is  now officially deprecated.

@item 
Two new wrapper functions, @ref{103f,,PyOS_snprintf()} and @ref{1040,,PyOS_vsnprintf()}
were added to provide  cross-platform implementations for the relatively new
@code{snprintf()} and @code{vsnprintf()} C lib APIs. In contrast to the standard
@code{sprintf()} and @code{vsprintf()} functions, the Python versions check the
bounds of the buffer used to protect against buffer overruns. (Contributed by
M.-A. Lemburg.)

@item 
The @ref{1041,,_PyTuple_Resize()} function has lost an unused parameter, so now it
takes 2 parameters instead of 3.  The third argument was never used, and can
simply be discarded when porting code from earlier versions to Python 2.2.
@end itemize

@c ======================================================================

@node Other Changes and Fixes<3>,Acknowledgements<6>,Interpreter Changes and Fixes,What’s New in Python 2 2
@anchor{whatsnew/2 2 other-changes-and-fixes}@anchor{1042}
@subsection Other Changes and Fixes


As usual there were a bunch of other improvements and bugfixes scattered
throughout the source tree.  A search through the CVS change logs finds there
were 527 patches applied and 683 bugs fixed between Python 2.1 and 2.2; 2.2.1
applied 139 patches and fixed 143 bugs; 2.2.2 applied 106 patches and fixed 82
bugs.  These figures are likely to be underestimates.

Some of the more notable changes are:


@itemize *

@item 
The code for the MacOS port for Python, maintained by Jack Jansen, is now kept
in the main Python CVS tree, and many changes have been made to support MacOS X.

The most significant change is the ability to build Python as a framework,
enabled by supplying the @code{--enable-framework} option to the configure
script when compiling Python.  According to Jack Jansen, “This installs a
self-contained Python installation plus the OS X framework “glue” into
@code{/Library/Frameworks/Python.framework} (or another location of choice).
For now there is little immediate added benefit to this (actually, there is the
disadvantage that you have to change your PATH to be able to find Python), but
it is the basis for creating a full-blown Python application, porting the
MacPython IDE, possibly using Python as a standard OSA scripting language and
much more.”

Most of the MacPython toolbox modules, which interface to MacOS APIs such as
windowing, QuickTime, scripting, etc. have been ported to OS X, but they’ve been
left commented out in @code{setup.py}.  People who want to experiment with
these modules can uncomment them manually.

@c Jack's original comments:
@c The main change is the possibility to build Python as a
@c framework. This installs a self-contained Python installation plus the
@c OSX framework "glue" into /Library/Frameworks/Python.framework (or
@c another location of choice). For now there is little immediate added
@c benefit to this (actually, there is the disadvantage that you have to
@c change your PATH to be able to find Python), but it is the basis for
@c creating a fullblown Python application, porting the MacPython IDE,
@c possibly using Python as a standard OSA scripting language and much
@c more. You enable this with "configure --enable-framework".
@c The other change is that most MacPython toolbox modules, which
@c interface to all the MacOS APIs such as windowing, quicktime,
@c scripting, etc. have been ported. Again, most of these are not of
@c immediate use, as they need a full application to be really useful, so
@c they have been commented out in setup.py. People wanting to experiment
@c can uncomment them. Gestalt and Internet Config modules are enabled by
@c default.

@item 
Keyword arguments passed to built-in functions that don’t take them now cause a
@ref{162,,TypeError} exception to be raised, with the message “@emph{function} takes no
keyword arguments”.

@item 
Weak references, added in Python 2.1 as an extension module, are now part of
the core because they’re used in the implementation of new-style classes.  The
@ref{1043,,ReferenceError} exception has therefore moved from the @ref{12f,,weakref}
module to become a built-in exception.

@item 
A new script, @code{Tools/scripts/cleanfuture.py} by Tim Peters,
automatically removes obsolete @code{__future__} statements from Python source
code.

@item 
An additional @emph{flags} argument has been added to the built-in function
@ref{4e2,,compile()}, so the behaviour of @code{__future__} statements can now be
correctly observed in simulated shells, such as those presented by IDLE and
other development environments.  This is described in 
@geindex Python Enhancement Proposals; PEP 264
PEP 264@footnote{https://www.python.org/dev/peps/pep-0264}. (Contributed
by Michael Hudson.)

@item 
The new license introduced with Python 1.6 wasn’t GPL-compatible.  This is
fixed by some minor textual changes to the 2.2 license, so it’s now legal to
embed Python inside a GPLed program again.  Note that Python itself is not
GPLed, but instead is under a license that’s essentially equivalent to the BSD
license, same as it always was.  The license changes were also applied to the
Python 2.0.1 and 2.1.1 releases.

@item 
When presented with a Unicode filename on Windows, Python will now convert it
to an MBCS encoded string, as used by the Microsoft file APIs.  As MBCS is
explicitly used by the file APIs, Python’s choice of ASCII as the default
encoding turns out to be an annoyance.  On Unix, the locale’s character set is
used if @code{locale.nl_langinfo(CODESET)} is available.  (Windows support was
contributed by Mark Hammond with assistance from Marc-André Lemburg. Unix
support was added by Martin von Löwis.)

@item 
Large file support is now enabled on Windows.  (Contributed by Tim Peters.)

@item 
The @code{Tools/scripts/ftpmirror.py} script now parses a @code{.netrc}
file, if you have one. (Contributed by Mike Romberg.)

@item 
Some features of the object returned by the @code{xrange()} function are now
deprecated, and trigger warnings when they’re accessed; they’ll disappear in
Python 2.3. @code{xrange} objects tried to pretend they were full sequence
types by supporting slicing, sequence multiplication, and the @ref{2fc,,in}
operator, but these features were rarely used and therefore buggy.  The
@code{tolist()} method and the @code{start}, @code{stop}, and @code{step}
attributes are also being deprecated.  At the C level, the fourth argument to
the @code{PyRange_New()} function, @code{repeat}, has also been deprecated.

@item 
There were a bunch of patches to the dictionary implementation, mostly to fix
potential core dumps if a dictionary contains objects that sneakily changed
their hash value, or mutated the dictionary they were contained in. For a while
python-dev fell into a gentle rhythm of Michael Hudson finding a case that
dumped core, Tim Peters fixing the bug, Michael finding another case, and round
and round it went.

@item 
On Windows, Python can now be compiled with Borland C thanks to a number of
patches contributed by Stephen Hansen, though the result isn’t fully functional
yet.  (But this @emph{is} progress…)

@item 
Another Windows enhancement: Wise Solutions generously offered PythonLabs use
of their InstallerMaster 8.1 system.  Earlier PythonLabs Windows installers used
Wise 5.0a, which was beginning to show its age.  (Packaged up by Tim Peters.)

@item 
Files ending in @code{.pyw} can now be imported on Windows. @code{.pyw} is a
Windows-only thing, used to indicate that a script needs to be run using
PYTHONW.EXE instead of PYTHON.EXE in order to prevent a DOS console from popping
up to display the output.  This patch makes it possible to import such scripts,
in case they’re also usable as modules.  (Implemented by David Bolen.)

@item 
On platforms where Python uses the C @code{dlopen()} function  to load
extension modules, it’s now possible to set the flags used  by @code{dlopen()}
using the @ref{1044,,sys.getdlopenflags()} and @ref{cb2,,sys.setdlopenflags()} functions.
(Contributed by Bram Stolk.)

@item 
The @ref{4cd,,pow()} built-in function no longer supports 3 arguments when
floating-point numbers are supplied. @code{pow(x, y, z)} returns @code{(x**y) % z},
but this is never useful for floating point numbers, and the final result varies
unpredictably depending on the platform.  A call such as @code{pow(2.0, 8.0, 7.0)}
will now raise a @ref{162,,TypeError} exception.
@end itemize

@c ======================================================================

@node Acknowledgements<6>,,Other Changes and Fixes<3>,What’s New in Python 2 2
@anchor{whatsnew/2 2 acknowledgements}@anchor{1045}
@subsection Acknowledgements


The author would like to thank the following people for offering suggestions,
corrections and assistance with various drafts of this article: Fred Bremmer,
Keith Briggs, Andrew Dalke, Fred L. Drake, Jr., Carel Fellinger, David Goodger,
Mark Hammond, Stephen Hansen, Michael Hudson, Jack Jansen, Marc-André Lemburg,
Martin von Löwis, Fredrik Lundh, Michael McLay, Nick Mathewson, Paul Moore,
Gustavo Niemeyer, Don O’Donnell, Joonas Paalasma, Tim Peters, Jens Quade, Tom
Reinhardt, Neil Schemenauer, Guido van Rossum, Greg Ward, Edward Welbourne.

@node What’s New in Python 2 1,What’s New in Python 2 0,What’s New in Python 2 2,What’s New in Python
@anchor{whatsnew/2 1 doc}@anchor{1046}@anchor{whatsnew/2 1 what-s-new-in-python-2-1}@anchor{1047}
@section What’s New in Python 2.1


@*Author: 
A.M. Kuchling


@c $Id: whatsnew21.tex 50964 2006-07-30 03:03:43Z fred.drake $

@menu
* Introduction: Introduction<2>. 
* PEP 227; Nested Scopes: PEP 227 Nested Scopes<2>. 
* PEP 236; __future__ Directives: PEP 236 __future__ Directives. 
* PEP 207; Rich Comparisons: PEP 207 Rich Comparisons. 
* PEP 230; Warning Framework: PEP 230 Warning Framework. 
* PEP 229; New Build System: PEP 229 New Build System. 
* PEP 205; Weak References: PEP 205 Weak References. 
* PEP 232; Function Attributes: PEP 232 Function Attributes. 
* PEP 235; Importing Modules on Case-Insensitive Platforms: PEP 235 Importing Modules on Case-Insensitive Platforms. 
* PEP 217; Interactive Display Hook: PEP 217 Interactive Display Hook. 
* PEP 208; New Coercion Model: PEP 208 New Coercion Model. 
* PEP 241; Metadata in Python Packages: PEP 241 Metadata in Python Packages. 
* New and Improved Modules: New and Improved Modules<4>. 
* Other Changes and Fixes: Other Changes and Fixes<4>. 
* Acknowledgements: Acknowledgements<7>. 

@end menu

@node Introduction<2>,PEP 227 Nested Scopes<2>,,What’s New in Python 2 1
@anchor{whatsnew/2 1 introduction}@anchor{1048}
@subsection Introduction


This article explains the new features in Python 2.1.  While there aren’t as
many changes in 2.1 as there were in Python 2.0, there are still some pleasant
surprises in store.  2.1 is the first release to be steered through the use of
Python Enhancement Proposals, or PEPs, so most of the sizable changes have
accompanying PEPs that provide more complete documentation and a design
rationale for the change.  This article doesn’t attempt to document the new
features completely, but simply provides an overview of the new features for
Python programmers. Refer to the Python 2.1 documentation, or to the specific
PEP, for more details about any new feature that particularly interests you.

One recent goal of the Python development team has been to accelerate the pace
of new releases, with a new release coming every 6 to 9 months. 2.1 is the first
release to come out at this faster pace, with the first alpha appearing in
January, 3 months after the final version of 2.0 was released.

The final release of Python 2.1 was made on April 17, 2001.

@c ======================================================================

@node PEP 227 Nested Scopes<2>,PEP 236 __future__ Directives,Introduction<2>,What’s New in Python 2 1
@anchor{whatsnew/2 1 pep-227-nested-scopes}@anchor{1049}
@subsection PEP 227: Nested Scopes


The largest change in Python 2.1 is to Python’s scoping rules.  In Python 2.0,
at any given time there are at most three namespaces used to look up variable
names: local, module-level, and the built-in namespace.  This often surprised
people because it didn’t match their intuitive expectations.  For example, a
nested recursive function definition doesn’t work:

@example
def f():
    ...
    def g(value):
        ...
        return g(value-1) + 1
    ...
@end example

The function @code{g()} will always raise a @ref{21f,,NameError} exception, because
the binding of the name @code{g} isn’t in either its local namespace or in the
module-level namespace.  This isn’t much of a problem in practice (how often do
you recursively define interior functions like this?), but this also made using
the @ref{e42,,lambda} expression clumsier, and this was a problem in practice.
In code which uses @ref{e42,,lambda} you can often find local variables being
copied by passing them as the default values of arguments.

@example
def find(self, name):
    "Return list of any entries equal to 'name'"
    L = filter(lambda x, name=name: x == name,
               self.list_attribute)
    return L
@end example

The readability of Python code written in a strongly functional style suffers
greatly as a result.

The most significant change to Python 2.1 is that static scoping has been added
to the language to fix this problem.  As a first effect, the @code{name=name}
default argument is now unnecessary in the above example.  Put simply, when a
given variable name is not assigned a value within a function (by an assignment,
or the @ref{fb8,,def}, @ref{e72,,class}, or @ref{e35,,import} statements),
references to the variable will be looked up in the local namespace of the
enclosing scope.  A more detailed explanation of the rules, and a dissection of
the implementation, can be found in the PEP.

This change may cause some compatibility problems for code where the same
variable name is used both at the module level and as a local variable within a
function that contains further function definitions. This seems rather unlikely
though, since such code would have been pretty confusing to read in the first
place.

One side effect of the change is that the @code{from module import *} and
@code{exec} statements have been made illegal inside a function scope under
certain conditions.  The Python reference manual has said all along that @code{from
module import *} is only legal at the top level of a module, but the CPython
interpreter has never enforced this before.  As part of the implementation of
nested scopes, the compiler which turns Python source into bytecodes has to
generate different code to access variables in a containing scope.  @code{from
module import *} and @code{exec} make it impossible for the compiler to
figure this out, because they add names to the local namespace that are
unknowable at compile time. Therefore, if a function contains function
definitions or @ref{e42,,lambda} expressions with free variables, the compiler
will flag this by raising a @ref{21a,,SyntaxError} exception.

To make the preceding explanation a bit clearer, here’s an example:

@example
x = 1
def f():
    # The next line is a syntax error
    exec 'x=2'
    def g():
        return x
@end example

Line 4 containing the @code{exec} statement is a syntax error, since
@code{exec} would define a new local variable named @code{x} whose value should
be accessed by @code{g()}.

This shouldn’t be much of a limitation, since @code{exec} is rarely used in
most Python code (and when it is used, it’s often a sign of a poor design
anyway).

Compatibility concerns have led to nested scopes being introduced gradually; in
Python 2.1, they aren’t enabled by default, but can be turned on within a module
by using a future statement as described in 
@geindex Python Enhancement Proposals; PEP 236
PEP 236@footnote{https://www.python.org/dev/peps/pep-0236}.  (See the following section
for further discussion of 
@geindex Python Enhancement Proposals; PEP 236
PEP 236@footnote{https://www.python.org/dev/peps/pep-0236}.)  In Python 2.2, nested scopes will become
the default and there will be no way to turn them off, but users will have had
all of 2.1’s lifetime to fix any breakage resulting from their introduction.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 227

@item PEP 227@footnote{https://www.python.org/dev/peps/pep-0227} - Statically Nested Scopes

Written and implemented by Jeremy Hylton.
@end table


@c ======================================================================

@node PEP 236 __future__ Directives,PEP 207 Rich Comparisons,PEP 227 Nested Scopes<2>,What’s New in Python 2 1
@anchor{whatsnew/2 1 pep-236-future-directives}@anchor{104a}
@subsection PEP 236: __future__ Directives


The reaction to nested scopes was widespread concern about the dangers of
breaking code with the 2.1 release, and it was strong enough to make the
Pythoneers take a more conservative approach.  This approach consists of
introducing a convention for enabling optional functionality in release N that
will become compulsory in release N+1.

The syntax uses a @code{from...import} statement using the reserved module name
@ref{0,,__future__}.  Nested scopes can be enabled by the following statement:

@example
from __future__ import nested_scopes
@end example

While it looks like a normal @ref{e35,,import} statement, it’s not; there are
strict rules on where such a future statement can be put. They can only be at
the top of a module, and must precede any Python code or regular
@code{import} statements.  This is because such statements can affect how
the Python bytecode compiler parses code and generates bytecode, so they must
precede any statement that will result in bytecodes being produced.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 236

@item PEP 236@footnote{https://www.python.org/dev/peps/pep-0236} - Back to the @ref{0,,__future__}

Written by Tim Peters, and primarily implemented by Jeremy Hylton.
@end table


@c ======================================================================

@node PEP 207 Rich Comparisons,PEP 230 Warning Framework,PEP 236 __future__ Directives,What’s New in Python 2 1
@anchor{whatsnew/2 1 pep-207-rich-comparisons}@anchor{104b}
@subsection PEP 207: Rich Comparisons


In earlier versions, Python’s support for implementing comparisons on user-defined
classes and extension types was quite simple. Classes could implement a
@code{__cmp__()} method that was given two instances of a class, and could only
return 0 if they were equal or +1 or -1 if they weren’t; the method couldn’t
raise an exception or return anything other than a Boolean value.  Users of
Numeric Python often found this model too weak and restrictive, because in the
number-crunching programs that numeric Python is used for, it would be more
useful to be able to perform elementwise comparisons of two matrices, returning
a matrix containing the results of a given comparison for each element.  If the
two matrices are of different sizes, then the compare has to be able to raise an
exception to signal the error.

In Python 2.1, rich comparisons were added in order to support this need.
Python classes can now individually overload each of the @code{<}, @code{<=}, @code{>},
@code{>=}, @code{==}, and @code{!=} operations.  The new magic method names are:


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxx} 
@headitem

Operation

@tab

Method name

@item

@code{<}

@tab

@code{__lt__()}

@item

@code{<=}

@tab

@code{__le__()}

@item

@code{>}

@tab

@code{__gt__()}

@item

@code{>=}

@tab

@code{__ge__()}

@item

@code{==}

@tab

@code{__eq__()}

@item

@code{!=}

@tab

@code{__ne__()}

@end multitable


(The magic methods are named after the corresponding Fortran operators @code{.LT.}.
@code{.LE.}, &c.  Numeric programmers are almost certainly quite familiar with
these names and will find them easy to remember.)

Each of these magic methods is of the form @code{method(self, other)}, where
@code{self} will be the object on the left-hand side of the operator, while
@code{other} will be the object on the right-hand side.  For example, the
expression @code{A < B} will cause @code{A.__lt__(B)} to be called.

Each of these magic methods can return anything at all: a Boolean, a matrix, a
list, or any other Python object.  Alternatively they can raise an exception if
the comparison is impossible, inconsistent, or otherwise meaningless.

The built-in @code{cmp(A,B)} function can use the rich comparison machinery,
and now accepts an optional argument specifying which comparison operation to
use; this is given as one of the strings @code{"<"}, @code{"<="}, @code{">"}, @code{">="},
@code{"=="}, or @code{"!="}.  If called without the optional third argument,
@code{cmp()} will only return -1, 0, or +1 as in previous versions of Python;
otherwise it will call the appropriate method and can return any Python object.

There are also corresponding changes of interest to C programmers; there’s a new
slot @code{tp_richcmp} in type objects and an API for performing a given rich
comparison.  I won’t cover the C API here, but will refer you to 
@geindex Python Enhancement Proposals; PEP 207
PEP 207@footnote{https://www.python.org/dev/peps/pep-0207}, or to
2.1’s C API documentation, for the full list of related functions.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 207

@item PEP 207@footnote{https://www.python.org/dev/peps/pep-0207} - Rich Comparisons

Written by Guido van Rossum, heavily based on earlier work by David Ascher, and
implemented by Guido van Rossum.
@end table


@c ======================================================================

@node PEP 230 Warning Framework,PEP 229 New Build System,PEP 207 Rich Comparisons,What’s New in Python 2 1
@anchor{whatsnew/2 1 pep-230-warning-framework}@anchor{104c}
@subsection PEP 230: Warning Framework


Over its 10 years of existence, Python has accumulated a certain number of
obsolete modules and features along the way.  It’s difficult to know when a
feature is safe to remove, since there’s no way of knowing how much code uses it
— perhaps no programs depend on the feature, or perhaps many do.  To enable
removing old features in a more structured way, a warning framework was added.
When the Python developers want to get rid of a feature, it will first trigger a
warning in the next version of Python.  The following Python version can then
drop the feature, and users will have had a full release cycle to remove uses of
the old feature.

Python 2.1 adds the warning framework to be used in this scheme.  It adds a
@ref{12d,,warnings} module that provide functions to issue warnings, and to filter
out warnings that you don’t want to be displayed. Third-party modules can also
use this framework to deprecate old features that they no longer wish to
support.

For example, in Python 2.1 the @code{regex} module is deprecated, so importing
it causes a warning to be printed:

@example
>>> import regex
__main__:1: DeprecationWarning: the regex module
         is deprecated; please use the re module
>>>
@end example

Warnings can be issued by calling the @ref{104d,,warnings.warn()} function:

@example
warnings.warn("feature X no longer supported")
@end example

The first parameter is the warning message; an additional optional parameters
can be used to specify a particular warning category.

Filters can be added to disable certain warnings; a regular expression pattern
can be applied to the message or to the module name in order to suppress a
warning.  For example, you may have a program that uses the @code{regex} module
and not want to spare the time to convert it to use the @ref{d8,,re} module right
now.  The warning can be suppressed by calling

@example
import warnings
warnings.filterwarnings(action = 'ignore',
                        message='.*regex module is deprecated',
                        category=DeprecationWarning,
                        module = '__main__')
@end example

This adds a filter that will apply only to warnings of the class
@ref{1b2,,DeprecationWarning} triggered in the @ref{1,,__main__} module, and applies
a regular expression to only match the message about the @code{regex} module
being deprecated, and will cause such warnings to be ignored.  Warnings can also
be printed only once, printed every time the offending code is executed, or
turned into exceptions that will cause the program to stop (unless the
exceptions are caught in the usual way, of course).

Functions were also added to Python’s C API for issuing warnings; refer to PEP
230 or to Python’s API documentation for the details.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 5

@item PEP 5@footnote{https://www.python.org/dev/peps/pep-0005} - Guidelines for Language Evolution

Written by Paul Prescod, to specify procedures to be followed when removing old
features from Python.  The policy described in this PEP hasn’t been officially
adopted, but the eventual policy probably won’t be too different from Prescod’s
proposal.

@geindex Python Enhancement Proposals; PEP 230

@item PEP 230@footnote{https://www.python.org/dev/peps/pep-0230} - Warning Framework

Written and implemented by Guido van Rossum.
@end table


@c ======================================================================

@node PEP 229 New Build System,PEP 205 Weak References,PEP 230 Warning Framework,What’s New in Python 2 1
@anchor{whatsnew/2 1 pep-229-new-build-system}@anchor{104e}
@subsection PEP 229: New Build System


When compiling Python, the user had to go in and edit the @code{Modules/Setup}
file in order to enable various additional modules; the default set is
relatively small and limited to modules that compile on most Unix platforms.
This means that on Unix platforms with many more features, most notably Linux,
Python installations often don’t contain all useful modules they could.

Python 2.0 added the Distutils, a set of modules for distributing and installing
extensions.  In Python 2.1, the Distutils are used to compile much of the
standard library of extension modules, autodetecting which ones are supported on
the current machine.  It’s hoped that this will make Python installations easier
and more featureful.

Instead of having to edit the @code{Modules/Setup} file in order to enable
modules, a @code{setup.py} script in the top directory of the Python source
distribution is run at build time, and attempts to discover which modules can be
enabled by examining the modules and header files on the system.  If a module is
configured in @code{Modules/Setup}, the @code{setup.py} script won’t attempt
to compile that module and will defer to the @code{Modules/Setup} file’s
contents.  This provides a way to specific any strange command-line flags or
libraries that are required for a specific platform.

In another far-reaching change to the build mechanism, Neil Schemenauer
restructured things so Python now uses a single makefile that isn’t recursive,
instead of makefiles in the top directory and in each of the @code{Python/},
@code{Parser/}, @code{Objects/}, and @code{Modules/} subdirectories.  This
makes building Python faster and also makes hacking the Makefiles clearer and
simpler.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 229

@item PEP 229@footnote{https://www.python.org/dev/peps/pep-0229} - Using Distutils to Build Python

Written and implemented by A.M. Kuchling.
@end table


@c ======================================================================

@node PEP 205 Weak References,PEP 232 Function Attributes,PEP 229 New Build System,What’s New in Python 2 1
@anchor{whatsnew/2 1 pep-205-weak-references}@anchor{104f}
@subsection PEP 205: Weak References


Weak references, available through the @ref{12f,,weakref} module, are a minor but
useful new data type in the Python programmer’s toolbox.

Storing a reference to an object (say, in a dictionary or a list) has the side
effect of keeping that object alive forever.  There are a few specific cases
where this behaviour is undesirable, object caches being the most common one,
and another being circular references in data structures such as trees.

For example, consider a memoizing function that caches the results of another
function @code{f(x)} by storing the function’s argument and its result in a
dictionary:

@example
_cache = @{@}
def memoize(x):
    if _cache.has_key(x):
        return _cache[x]

    retval = f(x)

    # Cache the returned object
    _cache[x] = retval

    return retval
@end example

This version works for simple things such as integers, but it has a side effect;
the @code{_cache} dictionary holds a reference to the return values, so they’ll
never be deallocated until the Python process exits and cleans up. This isn’t
very noticeable for integers, but if @code{f()} returns an object, or a data
structure that takes up a lot of memory, this can be a problem.

Weak references provide a way to implement a cache that won’t keep objects alive
beyond their time.  If an object is only accessible through weak references, the
object will be deallocated and the weak references will now indicate that the
object it referred to no longer exists.  A weak reference to an object @emph{obj} is
created by calling @code{wr = weakref.ref(obj)}.  The object being referred to is
returned by calling the weak reference as if it were a function: @code{wr()}.  It
will return the referenced object, or @code{None} if the object no longer exists.

This makes it possible to write a @code{memoize()} function whose cache doesn’t
keep objects alive, by storing weak references in the cache.

@example
_cache = @{@}
def memoize(x):
    if _cache.has_key(x):
        obj = _cache[x]()
        # If weak reference object still exists,
        # return it
        if obj is not None: return obj

    retval = f(x)

    # Cache a weak reference
    _cache[x] = weakref.ref(retval)

    return retval
@end example

The @ref{12f,,weakref} module also allows creating proxy objects which behave like
weak references — an object referenced only by proxy objects is deallocated –
but instead of requiring an explicit call to retrieve the object, the proxy
transparently forwards all operations to the object as long as the object still
exists.  If the object is deallocated, attempting to use a proxy will cause a
@code{weakref.ReferenceError} exception to be raised.

@example
proxy = weakref.proxy(obj)
proxy.attr   # Equivalent to obj.attr
proxy.meth() # Equivalent to obj.meth()
del obj
proxy.attr   # raises weakref.ReferenceError
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 205

@item PEP 205@footnote{https://www.python.org/dev/peps/pep-0205} - Weak References

Written and implemented by Fred L. Drake, Jr.
@end table


@c ======================================================================

@node PEP 232 Function Attributes,PEP 235 Importing Modules on Case-Insensitive Platforms,PEP 205 Weak References,What’s New in Python 2 1
@anchor{whatsnew/2 1 pep-232-function-attributes}@anchor{1050}
@subsection PEP 232: Function Attributes


In Python 2.1, functions can now have arbitrary information attached to them.
People were often using docstrings to hold information about functions and
methods, because the @code{__doc__} attribute was the only way of attaching any
information to a function.  For example, in the Zope web application server,
functions are marked as safe for public access by having a docstring, and in
John Aycock’s SPARK parsing framework, docstrings hold parts of the BNF grammar
to be parsed.  This overloading is unfortunate, since docstrings are really
intended to hold a function’s documentation; for example, it means you can’t
properly document functions intended for private use in Zope.

Arbitrary attributes can now be set and retrieved on functions using the regular
Python syntax:

@example
def f(): pass

f.publish = 1
f.secure = 1
f.grammar = "A ::= B (C D)*"
@end example

The dictionary containing attributes can be accessed as the function’s
@ref{7a4,,__dict__}. Unlike the @ref{7a4,,__dict__} attribute of class instances, in
functions you can actually assign a new dictionary to @ref{7a4,,__dict__}, though
the new value is restricted to a regular Python dictionary; you @emph{can’t} be
tricky and set it to a @code{UserDict} instance, or any other random object
that behaves like a mapping.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 232

@item PEP 232@footnote{https://www.python.org/dev/peps/pep-0232} - Function Attributes

Written and implemented by Barry Warsaw.
@end table


@c ======================================================================

@node PEP 235 Importing Modules on Case-Insensitive Platforms,PEP 217 Interactive Display Hook,PEP 232 Function Attributes,What’s New in Python 2 1
@anchor{whatsnew/2 1 pep-235-importing-modules-on-case-insensitive-platforms}@anchor{1051}
@subsection PEP 235: Importing Modules on Case-Insensitive Platforms


Some operating systems have filesystems that are case-insensitive, MacOS and
Windows being the primary examples; on these systems, it’s impossible to
distinguish the filenames @code{FILE.PY} and @code{file.py}, even though they do store
the file’s name  in its original case (they’re case-preserving, too).

In Python 2.1, the @ref{e35,,import} statement will work to simulate case-sensitivity
on case-insensitive platforms.  Python will now search for the first
case-sensitive match by default, raising an @ref{3b8,,ImportError} if no such file
is found, so @code{import file} will not import a module named @code{FILE.PY}.
Case-insensitive matching can be requested by setting the 
@geindex PYTHONCASEOK
@geindex environment variable; PYTHONCASEOK
@ref{45f,,PYTHONCASEOK}
environment variable before starting the Python interpreter.

@c ======================================================================

@node PEP 217 Interactive Display Hook,PEP 208 New Coercion Model,PEP 235 Importing Modules on Case-Insensitive Platforms,What’s New in Python 2 1
@anchor{whatsnew/2 1 pep-217-interactive-display-hook}@anchor{1052}
@subsection PEP 217: Interactive Display Hook


When using the Python interpreter interactively, the output of commands is
displayed using the built-in @ref{27d,,repr()} function. In Python 2.1, the variable
@ref{1053,,sys.displayhook()} can be set to a callable object which will be called
instead of @ref{27d,,repr()}. For example, you can set it to a special
pretty-printing function:

@example
>>> # Create a recursive data structure
... L = [1,2,3]
>>> L.append(L)
>>> L # Show Python's default output
[1, 2, 3, [...]]
>>> # Use pprint.pprint() as the display function
... import sys, pprint
>>> sys.displayhook = pprint.pprint
>>> L
[1, 2, 3,  <Recursion on list with id=135143996>]
>>>
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 217

@item PEP 217@footnote{https://www.python.org/dev/peps/pep-0217} - Display Hook for Interactive Use

Written and implemented by Moshe Zadka.
@end table


@c ======================================================================

@node PEP 208 New Coercion Model,PEP 241 Metadata in Python Packages,PEP 217 Interactive Display Hook,What’s New in Python 2 1
@anchor{whatsnew/2 1 pep-208-new-coercion-model}@anchor{1054}
@subsection PEP 208: New Coercion Model


How numeric coercion is done at the C level was significantly modified.  This
will only affect the authors of C extensions to Python, allowing them more
flexibility in writing extension types that support numeric operations.

Extension types can now set the type flag @code{Py_TPFLAGS_CHECKTYPES} in their
@code{PyTypeObject} structure to indicate that they support the new coercion model.
In such extension types, the numeric slot functions can no longer assume that
they’ll be passed two arguments of the same type; instead they may be passed two
arguments of differing types, and can then perform their own internal coercion.
If the slot function is passed a type it can’t handle, it can indicate the
failure by returning a reference to the @code{Py_NotImplemented} singleton value.
The numeric functions of the other type will then be tried, and perhaps they can
handle the operation; if the other type also returns @code{Py_NotImplemented}, then
a @ref{162,,TypeError} will be raised.  Numeric methods written in Python can also
return @code{Py_NotImplemented}, causing the interpreter to act as if the method
did not exist (perhaps raising a @ref{162,,TypeError}, perhaps trying another
object’s numeric methods).


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 208

@item PEP 208@footnote{https://www.python.org/dev/peps/pep-0208} - Reworking the Coercion Model

Written and implemented by Neil Schemenauer, heavily based upon earlier work by
Marc-André Lemburg.  Read this to understand the fine points of how numeric
operations will now be processed at the C level.
@end table


@c ======================================================================

@node PEP 241 Metadata in Python Packages,New and Improved Modules<4>,PEP 208 New Coercion Model,What’s New in Python 2 1
@anchor{whatsnew/2 1 pep-241-metadata-in-python-packages}@anchor{1055}
@subsection PEP 241: Metadata in Python Packages


A common complaint from Python users is that there’s no single catalog of all
the Python modules in existence.  T. Middleton’s Vaults of Parnassus at
@indicateurl{http://www.vex.net/parnassus/} are the largest catalog of Python modules, but
registering software at the Vaults is optional, and many people don’t bother.

As a first small step toward fixing the problem, Python software packaged using
the Distutils @code{sdist} command will include a file named
@code{PKG-INFO} containing information about the package such as its name,
version, and author (metadata, in cataloguing terminology).  
@geindex Python Enhancement Proposals; PEP 241
PEP 241@footnote{https://www.python.org/dev/peps/pep-0241} contains
the full list of fields that can be present in the @code{PKG-INFO} file.  As
people began to package their software using Python 2.1, more and more packages
will include metadata, making it possible to build automated cataloguing systems
and experiment with them.  With the result experience, perhaps it’ll be possible
to design a really good catalog and then build support for it into Python 2.2.
For example, the Distutils @code{sdist} and @code{bdist_*} commands
could support an @code{upload} option that would automatically upload your
package to a catalog server.

You can start creating packages containing @code{PKG-INFO} even if you’re not
using Python 2.1, since a new release of the Distutils will be made for users of
earlier Python versions.  Version 1.0.2 of the Distutils includes the changes
described in 
@geindex Python Enhancement Proposals; PEP 241
PEP 241@footnote{https://www.python.org/dev/peps/pep-0241}, as well as various bugfixes and enhancements.  It will be
available from the Distutils SIG at @indicateurl{https://www.python.org/community/sigs/current/distutils-sig/}.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 241

@item PEP 241@footnote{https://www.python.org/dev/peps/pep-0241} - Metadata for Python Software Packages

Written and implemented by A.M. Kuchling.

@geindex Python Enhancement Proposals; PEP 243

@item PEP 243@footnote{https://www.python.org/dev/peps/pep-0243} - Module Repository Upload Mechanism

Written by Sean Reifschneider, this draft PEP describes a proposed mechanism for
uploading  Python packages to a central server.
@end table


@c ======================================================================

@node New and Improved Modules<4>,Other Changes and Fixes<4>,PEP 241 Metadata in Python Packages,What’s New in Python 2 1
@anchor{whatsnew/2 1 new-and-improved-modules}@anchor{1056}
@subsection New and Improved Modules



@itemize *

@item 
Ka-Ping Yee contributed two new modules: @code{inspect.py}, a module for
getting information about live Python code, and @code{pydoc.py}, a module for
interactively converting docstrings to HTML or text.  As a bonus,
@code{Tools/scripts/pydoc}, which is now automatically installed, uses
@code{pydoc.py} to display documentation given a Python module, package, or
class name.  For example, @code{pydoc xml.dom} displays the following:

@example
Python Library Documentation: package xml.dom in xml

NAME
    xml.dom - W3C Document Object Model implementation for Python.

FILE
    /usr/local/lib/python2.1/xml/dom/__init__.pyc

DESCRIPTION
    The Python mapping of the Document Object Model is documented in the
    Python Library Reference in the section on the xml.dom package.

    This package contains the following modules:
      ...
@end example

@code{pydoc} also includes a Tk-based interactive help browser.   @code{pydoc}
quickly becomes addictive; try it out!

@item 
Two different modules for unit testing were added to the standard library.
The @ref{63,,doctest} module, contributed by Tim Peters, provides a testing
framework based on running embedded examples in docstrings and comparing the
results against the expected output.  PyUnit, contributed by Steve Purcell, is a
unit testing framework inspired by JUnit, which was in turn an adaptation of
Kent Beck’s Smalltalk testing framework.  See @indicateurl{http://pyunit.sourceforge.net/} for
more information about PyUnit.

@item 
The @ref{35,,difflib} module contains a class, @code{SequenceMatcher}, which
compares two sequences and computes the changes required to transform one
sequence into the other.  For example, this module can be used to write a tool
similar to the Unix @code{diff} program, and in fact the sample program
@code{Tools/scripts/ndiff.py} demonstrates how to write such a script.

@item 
@ref{2c,,curses.panel}, a wrapper for the panel library, part of ncurses and of
SYSV curses, was contributed by Thomas Gellekum.  The panel library provides
windows with the additional feature of depth. Windows can be moved higher or
lower in the depth ordering, and the panel library figures out where panels
overlap and which sections are visible.

@item 
The PyXML package has gone through a few releases since Python 2.0, and Python
2.1 includes an updated version of the @ref{13a,,xml} package.  Some of the
noteworthy changes include support for Expat 1.2 and later versions, the ability
for Expat parsers to handle files in any encoding supported by Python, and
various bugfixes for SAX, DOM, and the @code{minidom} module.

@item 
Ping also contributed another hook for handling uncaught exceptions.
@ref{28e,,sys.excepthook()} can be set to a callable object.  When an exception isn’t
caught by any @ref{f6d,,try}…@ref{16e,,except} blocks, the exception will be
passed to @ref{28e,,sys.excepthook()}, which can then do whatever it likes.  At the
Ninth Python Conference, Ping demonstrated an application for this hook:
printing an extended traceback that not only lists the stack frames, but also
lists the function arguments and the local variables for each frame.

@item 
Various functions in the @ref{10a,,time} module, such as @code{asctime()} and
@code{localtime()}, require a floating point argument containing the time in
seconds since the epoch.  The most common use of these functions is to work with
the current time, so the floating point argument has been made optional; when a
value isn’t provided, the current time will be used.  For example, log file
entries usually need a string containing the current time; in Python 2.1,
@code{time.asctime()} can be used, instead of the lengthier
@code{time.asctime(time.localtime(time.time()))} that was previously required.

This change was proposed and implemented by Thomas Wouters.

@item 
The @ref{7e,,ftplib} module now defaults to retrieving files in passive mode,
because passive mode is more likely to work from behind a firewall.  This
request came from the Debian bug tracking system, since other Debian packages
use @ref{7e,,ftplib} to retrieve files and then don’t work from behind a firewall.
It’s deemed unlikely that this will cause problems for anyone, because Netscape
defaults to passive mode and few people complain, but if passive mode is
unsuitable for your application or network setup, call @code{set_pasv(0)} on
FTP objects to disable passive mode.

@item 
Support for raw socket access has been added to the @ref{ea,,socket} module,
contributed by Grant Edwards.

@item 
The @ref{cf,,pstats} module now contains a simple interactive statistics browser
for displaying timing profiles for Python programs, invoked when the module is
run as a script.  Contributed by  Eric S. Raymond.

@item 
A new implementation-dependent function, @code{sys._getframe([depth])}, has
been added to return a given frame object from the current call stack.
@ref{1057,,sys._getframe()} returns the frame at the top of the call stack;  if the
optional integer argument @emph{depth} is supplied, the function returns the frame
that is @emph{depth} calls below the top of the stack.  For example,
@code{sys._getframe(1)} returns the caller’s frame object.

This function is only present in CPython, not in Jython or the .NET
implementation.  Use it for debugging, and resist the temptation to put it into
production code.
@end itemize

@c ======================================================================

@node Other Changes and Fixes<4>,Acknowledgements<7>,New and Improved Modules<4>,What’s New in Python 2 1
@anchor{whatsnew/2 1 other-changes-and-fixes}@anchor{1058}
@subsection Other Changes and Fixes


There were relatively few smaller changes made in Python 2.1 due to the shorter
release cycle.  A search through the CVS change logs turns up 117 patches
applied, and 136 bugs fixed; both figures are likely to be underestimates.  Some
of the more notable changes are:


@itemize *

@item 
A specialized object allocator is now optionally available, that should be
faster than the system @code{malloc()} and have less memory overhead.  The
allocator uses C’s @code{malloc()} function to get large pools of memory, and
then fulfills smaller memory requests from these pools.  It can be enabled by
providing the @code{--with-pymalloc} option to the @code{configure}
script; see @code{Objects/obmalloc.c} for the implementation details.

Authors of C extension modules should test their code with the object allocator
enabled, because some incorrect code may break, causing core dumps at runtime.
There are a bunch of memory allocation functions in Python’s C API that have
previously been just aliases for the C library’s @code{malloc()} and
@code{free()}, meaning that if you accidentally called mismatched functions, the
error wouldn’t be noticeable.  When the object allocator is enabled, these
functions aren’t aliases of @code{malloc()} and @code{free()} any more, and
calling the wrong function to free memory will get you a core dump.  For
example, if memory was allocated using @code{PyMem_New()}, it has to be freed
using @code{PyMem_Del()}, not @code{free()}.  A few modules included with Python
fell afoul of this and had to be fixed; doubtless there are more third-party
modules that will have the same problem.

The object allocator was contributed by Vladimir Marangozov.

@item 
The speed of line-oriented file I/O has been improved because people often
complain about its lack of speed, and because it’s often been used as a naïve
benchmark.  The @ref{d9,,readline()} method of file objects has therefore been
rewritten to be much faster.  The exact amount of the speedup will vary from
platform to platform depending on how slow the C library’s @code{getc()} was, but
is around 66%, and potentially much faster on some particular operating systems.
Tim Peters did much of the benchmarking and coding for this change, motivated by
a discussion in comp.lang.python.

A new module and method for file objects was also added, contributed by Jeff
Epler. The new method, @code{xreadlines()}, is similar to the existing
@code{xrange()} built-in.  @code{xreadlines()} returns an opaque sequence object
that only supports being iterated over, reading a line on every iteration but
not reading the entire file into memory as the existing @code{readlines()} method
does. You’d use it like this:

@example
for line in sys.stdin.xreadlines():
    # ... do something for each line ...
    ...
@end example

For a fuller discussion of the line I/O changes, see the python-dev summary for
January 1–15, 2001 at @indicateurl{https://mail.python.org/pipermail/python-dev/2001-January/}.

@item 
A new method, @code{popitem()}, was added to dictionaries to enable
destructively iterating through the contents of a dictionary; this can be faster
for large dictionaries because there’s no need to construct a list containing
all the keys or values. @code{D.popitem()} removes a random @code{(key, value)} pair
from the dictionary @code{D} and returns it as a 2-tuple.  This was implemented
mostly by Tim Peters and Guido van Rossum, after a suggestion and preliminary
patch by Moshe Zadka.

@item 
Modules can now control which names are imported when @code{from module import *}
is used, by defining an @code{__all__} attribute containing a list of names that
will be imported.  One common complaint is that if the module imports other
modules such as @ref{f7,,sys} or @ref{f1,,string}, @code{from module import *} will add
them to the importing module’s namespace.  To fix this, simply list the public
names in @code{__all__}:

@example
# List public names
__all__ = ['Database', 'open']
@end example

A stricter version of this patch was first suggested and implemented by Ben
Wolfson, but after some python-dev discussion, a weaker final version was
checked in.

@item 
Applying @ref{27d,,repr()} to strings previously used octal escapes for
non-printable characters; for example, a newline was @code{'\012'}.  This was a
vestigial trace of Python’s C ancestry, but today octal is of very little
practical use.  Ka-Ping Yee suggested using hex escapes instead of octal ones,
and using the @code{\n}, @code{\t}, @code{\r} escapes for the appropriate characters,
and implemented this new formatting.

@item 
Syntax errors detected at compile-time can now raise exceptions containing the
filename and line number of the error, a pleasant side effect of the compiler
reorganization done by Jeremy Hylton.

@item 
C extensions which import other modules have been changed to use
@code{PyImport_ImportModule()}, which means that they will use any import hooks
that have been installed.  This is also encouraged for third-party extensions
that need to import some other module from C code.

@item 
The size of the Unicode character database was shrunk by another 340K thanks
to Fredrik Lundh.

@item 
Some new ports were contributed: MacOS X (by Steven Majewski), Cygwin (by
Jason Tishler); RISCOS (by Dietmar Schwertberger); Unixware 7  (by Billy G.
Allie).
@end itemize

And there’s the usual list of minor bugfixes, minor memory leaks, docstring
edits, and other tweaks, too lengthy to be worth itemizing; see the CVS logs for
the full details if you want them.

@c ======================================================================

@node Acknowledgements<7>,,Other Changes and Fixes<4>,What’s New in Python 2 1
@anchor{whatsnew/2 1 acknowledgements}@anchor{1059}
@subsection Acknowledgements


The author would like to thank the following people for offering suggestions on
various drafts of this article: Graeme Cross, David Goodger, Jay Graves, Michael
Hudson, Marc-André Lemburg, Fredrik Lundh, Neil Schemenauer, Thomas Wouters.

@node What’s New in Python 2 0,Changelog,What’s New in Python 2 1,What’s New in Python
@anchor{whatsnew/2 0 doc}@anchor{105a}@anchor{whatsnew/2 0 what-s-new-in-python-2-0}@anchor{105b}
@section What’s New in Python 2.0


@*Author: 
A.M. Kuchling and Moshe Zadka


@c $Id: whatsnew20.tex 50964 2006-07-30 03:03:43Z fred.drake $

@menu
* Introduction: Introduction<3>. 
* What About Python 1.6?: What About Python 1 6?. 
* New Development Process:: 
* Unicode: Unicode<2>. 
* List Comprehensions:: 
* Augmented Assignment:: 
* String Methods:: 
* Garbage Collection of Cycles:: 
* Other Core Changes:: 
* Porting to 2.0: Porting to 2 0. 
* Extending/Embedding Changes:: 
* Distutils; Making Modules Easy to Install: Distutils Making Modules Easy to Install. 
* XML Modules:: 
* Module changes:: 
* New modules:: 
* IDLE Improvements:: 
* Deleted and Deprecated Modules:: 
* Acknowledgements: Acknowledgements<8>. 

@end menu

@node Introduction<3>,What About Python 1 6?,,What’s New in Python 2 0
@anchor{whatsnew/2 0 introduction}@anchor{105c}
@subsection Introduction


A new release of Python, version 2.0, was released on October 16, 2000. This
article covers the exciting new features in 2.0, highlights some other useful
changes, and points out a few incompatible changes that may require rewriting
code.

Python’s development never completely stops between releases, and a steady flow
of bug fixes and improvements are always being submitted. A host of minor fixes,
a few optimizations, additional docstrings, and better error messages went into
2.0; to list them all would be impossible, but they’re certainly significant.
Consult the publicly-available CVS logs if you want to see the full list.  This
progress is due to the five developers working for  PythonLabs are now getting
paid to spend their days fixing bugs, and also due to the improved communication
resulting  from moving to SourceForge.

@c ======================================================================

@node What About Python 1 6?,New Development Process,Introduction<3>,What’s New in Python 2 0
@anchor{whatsnew/2 0 what-about-python-1-6}@anchor{105d}
@subsection What About Python 1.6?


Python 1.6 can be thought of as the Contractual Obligations Python release.
After the core development team left CNRI in May 2000, CNRI requested that a 1.6
release be created, containing all the work on Python that had been performed at
CNRI.  Python 1.6 therefore represents the state of the CVS tree as of May 2000,
with the most significant new feature being Unicode support.  Development
continued after May, of course, so the 1.6 tree received a few fixes to ensure
that it’s forward-compatible with Python 2.0.  1.6 is therefore part of Python’s
evolution, and not a side branch.

So, should you take much interest in Python 1.6?  Probably not.  The 1.6final
and 2.0beta1 releases were made on the same day (September 5, 2000), the plan
being to finalize Python 2.0 within a month or so.  If you have applications to
maintain, there seems little point in breaking things by moving to 1.6, fixing
them, and then having another round of breakage within a month by moving to 2.0;
you’re better off just going straight to 2.0.  Most of the really interesting
features described in this document are only in 2.0, because a lot of work was
done between May and September.

@c ======================================================================

@node New Development Process,Unicode<2>,What About Python 1 6?,What’s New in Python 2 0
@anchor{whatsnew/2 0 new-development-process}@anchor{105e}
@subsection New Development Process


The most important change in Python 2.0 may not be to the code at all, but to
how Python is developed: in May 2000 the Python developers began using the tools
made available by SourceForge for storing  source code, tracking bug reports,
and managing the queue of patch submissions.  To report bugs or submit patches
for Python 2.0, use the bug tracking and patch manager tools available from
Python’s project page, located at @indicateurl{https://sourceforge.net/projects/python/}.

The most important of the services now hosted at SourceForge is the Python CVS
tree, the version-controlled repository containing the source code for Python.
Previously, there were roughly 7 or so people who had write access to the CVS
tree, and all patches had to be inspected and checked in by one of the people on
this short list. Obviously, this wasn’t very scalable.  By moving the CVS tree
to SourceForge, it became possible to grant write access to more people; as of
September 2000 there were 27 people able to check in changes, a fourfold
increase.  This makes possible large-scale changes that wouldn’t be attempted if
they’d have to be filtered through the small group of core developers.  For
example, one day Peter Schneider-Kamp took it into his head to drop K&R C
compatibility and convert the C source for Python to ANSI C. After getting
approval on the python-dev mailing list, he launched into a flurry of checkins
that lasted about a week, other developers joined in to help, and the job was
done.  If there were only 5 people with write access, probably that task would
have been viewed as “nice, but not worth the time and effort needed” and it
would never have gotten done.

The shift to using SourceForge’s services has resulted in a remarkable increase
in the speed of development.  Patches now get submitted, commented on, revised
by people other than the original submitter, and bounced back and forth between
people until the patch is deemed worth checking in.  Bugs are tracked in one
central location and can be assigned to a specific person for fixing, and we can
count the number of open bugs to measure progress.  This didn’t come without a
cost: developers now have more e-mail to deal with, more mailing lists to
follow, and special tools had to be written for the new environment. For
example, SourceForge sends default patch and bug notification e-mail messages
that are completely unhelpful, so Ka-Ping Yee wrote an HTML screen-scraper that
sends more useful messages.

The ease of adding code caused a few initial growing pains, such as code was
checked in before it was ready or without getting clear agreement from the
developer group.  The approval process that has emerged is somewhat similar to
that used by the Apache group. Developers can vote +1, +0, -0, or -1 on a patch;
+1 and -1 denote acceptance or rejection, while +0 and -0 mean the developer is
mostly indifferent to the change, though with a slight positive or negative
slant.  The most significant change from the Apache model is that the voting is
essentially advisory, letting Guido van Rossum, who has Benevolent Dictator For
Life status, know what the general opinion is. He can still ignore the result of
a vote, and approve or reject a change even if the community disagrees with him.

Producing an actual patch is the last step in adding a new feature, and is
usually easy compared to the earlier task of coming up with a good design.
Discussions of new features can often explode into lengthy mailing list threads,
making the discussion hard to follow, and no one can read every posting to
python-dev.  Therefore, a relatively formal process has been set up to write
Python Enhancement Proposals (PEPs), modelled on the internet RFC process.  PEPs
are draft documents that describe a proposed new feature, and are continually
revised until the community reaches a consensus, either accepting or rejecting
the proposal.  Quoting from the introduction to 
@geindex Python Enhancement Proposals; PEP 1
PEP 1@footnote{https://www.python.org/dev/peps/pep-0001}, “PEP Purpose and
Guidelines”:

@quotation

PEP stands for Python Enhancement Proposal.  A PEP is a design document
providing information to the Python community, or describing a new feature for
Python.  The PEP should provide a concise technical specification of the feature
and a rationale for the feature.

We intend PEPs to be the primary mechanisms for proposing new features, for
collecting community input on an issue, and for documenting the design decisions
that have gone into Python.  The PEP author is responsible for building
consensus within the community and documenting dissenting opinions.
@end quotation

Read the rest of 
@geindex Python Enhancement Proposals; PEP 1
PEP 1@footnote{https://www.python.org/dev/peps/pep-0001} for the details of the PEP editorial process, style, and
format.  PEPs are kept in the Python CVS tree on SourceForge, though they’re not
part of the Python 2.0 distribution, and are also available in HTML form from
@indicateurl{https://www.python.org/dev/peps/}.  As of September 2000, there are 25 PEPS, ranging
from 
@geindex Python Enhancement Proposals; PEP 201
PEP 201@footnote{https://www.python.org/dev/peps/pep-0201}, “Lockstep Iteration”, to PEP 225, “Elementwise/Objectwise
Operators”.

@c ======================================================================

@node Unicode<2>,List Comprehensions,New Development Process,What’s New in Python 2 0
@anchor{whatsnew/2 0 unicode}@anchor{105f}
@subsection Unicode


The largest new feature in Python 2.0 is a new fundamental data type: Unicode
strings.  Unicode uses 16-bit numbers to represent characters instead of the
8-bit number used by ASCII, meaning that 65,536 distinct characters can be
supported.

The final interface for Unicode support was arrived at through countless
often-stormy discussions on the python-dev mailing list, and mostly implemented by
Marc-André Lemburg, based on a Unicode string type implementation by Fredrik
Lundh.  A detailed explanation of the interface was written up as 
@geindex Python Enhancement Proposals; PEP 100
PEP 100@footnote{https://www.python.org/dev/peps/pep-0100},
“Python Unicode Integration”. This article will simply cover the most
significant points about the Unicode interfaces.

In Python source code, Unicode strings are written as @code{u"string"}.  Arbitrary
Unicode characters can be written using a new escape sequence, @code{\uHHHH}, where
@emph{HHHH} is a 4-digit hexadecimal number from 0000 to FFFF.  The existing
@code{\xHHHH} escape sequence can also be used, and octal escapes can be used for
characters up to U+01FF, which is represented by @code{\777}.

Unicode strings, just like regular strings, are an immutable sequence type.
They can be indexed and sliced, but not modified in place. Unicode strings have
an @code{encode( [encoding] )} method that returns an 8-bit string in the desired
encoding.  Encodings are named by strings, such as @code{'ascii'}, @code{'utf-8'},
@code{'iso-8859-1'}, or whatever.  A codec API is defined for implementing and
registering new encodings that are then available throughout a Python program.
If an encoding isn’t specified, the default encoding is usually 7-bit ASCII,
though it can be changed for your Python installation by calling the
@code{sys.setdefaultencoding(encoding)} function in a customized version of
@code{site.py}.

Combining 8-bit and Unicode strings always coerces to Unicode, using the default
ASCII encoding; the result of @code{'a' + u'bc'} is @code{u'abc'}.

New built-in functions have been added, and existing built-ins modified to
support Unicode:


@itemize *

@item 
@code{unichr(ch)} returns a Unicode string 1 character long, containing the
character @emph{ch}.

@item 
@code{ord(u)}, where @emph{u} is a 1-character regular or Unicode string, returns the
number of the character as an integer.

@item 
@code{unicode(string [, encoding]  [, errors] )} creates a Unicode string
from an 8-bit string.  @code{encoding} is a string naming the encoding to use. The
@code{errors} parameter specifies the treatment of characters that are invalid for
the current encoding; passing @code{'strict'} as the value causes an exception to
be raised on any encoding error, while @code{'ignore'} causes errors to be silently
ignored and @code{'replace'} uses U+FFFD, the official replacement character, in
case of any problems.

@item 
The @code{exec} statement, and various built-ins such as @code{eval()},
@code{getattr()}, and @code{setattr()} will also accept Unicode strings as well as
regular strings.  (It’s possible that the process of fixing this missed some
built-ins; if you find a built-in function that accepts strings but doesn’t
accept Unicode strings at all, please report it as a bug.)
@end itemize

A new module, @ref{121,,unicodedata}, provides an interface to Unicode character
properties.  For example, @code{unicodedata.category(u'A')} returns the 2-character
string ‘Lu’, the ‘L’ denoting it’s a letter, and ‘u’ meaning that it’s
uppercase. @code{unicodedata.bidirectional(u'\u0660')} returns ‘AN’, meaning that
U+0660 is an Arabic number.

The @ref{1a,,codecs} module contains functions to look up existing encodings and
register new ones.  Unless you want to implement a new encoding, you’ll most
often use the @code{codecs.lookup(encoding)} function, which returns a
4-element tuple: @code{(encode_func, decode_func, stream_reader, stream_writer)}.


@itemize *

@item 
@emph{encode_func} is a function that takes a Unicode string, and returns a 2-tuple
@code{(string, length)}.  @emph{string} is an 8-bit string containing a portion (perhaps
all) of the Unicode string converted into the given encoding, and @emph{length} tells
you how much of the Unicode string was converted.

@item 
@emph{decode_func} is the opposite of @emph{encode_func}, taking an 8-bit string and
returning a 2-tuple @code{(ustring, length)}, consisting of the resulting Unicode
string @emph{ustring} and the integer @emph{length} telling how much of the 8-bit string
was consumed.

@item 
@emph{stream_reader} is a class that supports decoding input from a stream.
@emph{stream_reader(file_obj)} returns an object that supports the @code{read()},
@ref{d9,,readline()}, and @code{readlines()} methods.  These methods will all
translate from the given encoding and return Unicode strings.

@item 
@emph{stream_writer}, similarly, is a class that supports encoding output to a
stream.  @emph{stream_writer(file_obj)} returns an object that supports the
@code{write()} and @code{writelines()} methods.  These methods expect Unicode
strings, translating them to the given encoding on output.
@end itemize

For example, the following code writes a Unicode string into a file,  encoding
it as UTF-8:

@example
import codecs

unistr = u'\u0660\u2000ab ...'

(UTF8_encode, UTF8_decode,
 UTF8_streamreader, UTF8_streamwriter) = codecs.lookup('UTF-8')

output = UTF8_streamwriter( open( '/tmp/output', 'wb') )
output.write( unistr )
output.close()
@end example

The following code would then read UTF-8 input from the file:

@example
input = UTF8_streamreader( open( '/tmp/output', 'rb') )
print repr(input.read())
input.close()
@end example

Unicode-aware regular expressions are available through the @ref{d8,,re} module,
which has a new underlying implementation called SRE written by Fredrik Lundh of
Secret Labs AB.

A @code{-U} command line option was added which causes the Python compiler to
interpret all string literals as Unicode string literals. This is intended to be
used in testing and future-proofing your Python code, since some future version
of Python may drop support for 8-bit strings and provide only Unicode strings.

@c ======================================================================

@node List Comprehensions,Augmented Assignment,Unicode<2>,What’s New in Python 2 0
@anchor{whatsnew/2 0 list-comprehensions}@anchor{1060}
@subsection List Comprehensions


Lists are a workhorse data type in Python, and many programs manipulate a list
at some point.  Two common operations on lists are to loop over them, and either
pick out the elements that meet a certain criterion, or apply some function to
each element.  For example, given a list of strings, you might want to pull out
all the strings containing a given substring, or strip off trailing whitespace
from each line.

The existing @ref{2f4,,map()} and @ref{2f5,,filter()} functions can be used for this
purpose, but they require a function as one of their arguments.  This is fine if
there’s an existing built-in function that can be passed directly, but if there
isn’t, you have to create a little function to do the required work, and
Python’s scoping rules make the result ugly if the little function needs
additional information.  Take the first example in the previous paragraph,
finding all the strings in the list containing a given substring.  You could
write the following to do it:

@example
# Given the list L, make a list of all strings
# containing the substring S.
sublist = filter( lambda s, substring=S:
                     string.find(s, substring) != -1,
                  L)
@end example

Because of Python’s scoping rules, a default argument is used so that the
anonymous function created by the @ref{e42,,lambda} expression knows what
substring is being searched for.  List comprehensions make this cleaner:

@example
sublist = [ s for s in L if string.find(s, S) != -1 ]
@end example

List comprehensions have the form:

@example
[ expression for expr in sequence1
             for expr2 in sequence2 ...
             for exprN in sequenceN
             if condition ]
@end example

The @code{for}…@code{in} clauses contain the sequences to be
iterated over.  The sequences do not have to be the same length, because they
are @emph{not} iterated over in parallel, but from left to right; this is explained
more clearly in the following paragraphs.  The elements of the generated list
will be the successive values of @emph{expression}.  The final @code{if} clause
is optional; if present, @emph{expression} is only evaluated and added to the result
if @emph{condition} is true.

To make the semantics very clear, a list comprehension is equivalent to the
following Python code:

@example
for expr1 in sequence1:
    for expr2 in sequence2:
    ...
        for exprN in sequenceN:
             if (condition):
                  # Append the value of
                  # the expression to the
                  # resulting list.
@end example

This means that when there are multiple @code{for}…@code{in}
clauses, the resulting list will be equal to the product of the lengths of all
the sequences.  If you have two lists of length 3, the output list is 9 elements
long:

@example
seq1 = 'abc'
seq2 = (1,2,3)
>>> [ (x,y) for x in seq1 for y in seq2]
[('a', 1), ('a', 2), ('a', 3), ('b', 1), ('b', 2), ('b', 3), ('c', 1),
('c', 2), ('c', 3)]
@end example

To avoid introducing an ambiguity into Python’s grammar, if @emph{expression} is
creating a tuple, it must be surrounded with parentheses.  The first list
comprehension below is a syntax error, while the second one is correct:

@example
# Syntax error
[ x,y for x in seq1 for y in seq2]
# Correct
[ (x,y) for x in seq1 for y in seq2]
@end example

The idea of list comprehensions originally comes from the functional programming
language Haskell (@indicateurl{https://www.haskell.org}).  Greg Ewing argued most effectively
for adding them to Python and wrote the initial list comprehension patch, which
was then discussed for a seemingly endless time on the python-dev mailing list
and kept up-to-date by Skip Montanaro.

@c ======================================================================

@node Augmented Assignment,String Methods,List Comprehensions,What’s New in Python 2 0
@anchor{whatsnew/2 0 augmented-assignment}@anchor{1061}
@subsection Augmented Assignment


Augmented assignment operators, another long-requested feature, have been added
to Python 2.0.  Augmented assignment operators include @code{+=}, @code{-=}, @code{*=},
and so forth.  For example, the statement @code{a += 2} increments the value of the
variable  @code{a} by 2, equivalent to the slightly lengthier @code{a = a + 2}.

The full list of supported assignment operators is @code{+=}, @code{-=}, @code{*=},
@code{/=}, @code{%=}, @code{**=}, @code{&=}, @code{|=}, @code{^=}, @code{>>=}, and @code{<<=}.  Python
classes can override the augmented assignment operators by defining methods
named @code{__iadd__()}, @code{__isub__()}, etc.  For example, the following
@code{Number} class stores a number and supports using += to create a new
instance with an incremented value.

@c The empty groups below prevent conversion to guillemets.

@example
class Number:
    def __init__(self, value):
        self.value = value
    def __iadd__(self, increment):
        return Number( self.value + increment)

n = Number(5)
n += 3
print n.value
@end example

The @code{__iadd__()} special method is called with the value of the increment,
and should return a new instance with an appropriately modified value; this
return value is bound as the new value of the variable on the left-hand side.

Augmented assignment operators were first introduced in the C programming
language, and most C-derived languages, such as @code{awk}, C++, Java, Perl,
and PHP also support them.  The augmented assignment patch was implemented by
Thomas Wouters.

@c ======================================================================

@node String Methods,Garbage Collection of Cycles,Augmented Assignment,What’s New in Python 2 0
@anchor{whatsnew/2 0 string-methods}@anchor{1062}
@subsection String Methods


Until now string-manipulation functionality was in the @ref{f1,,string} module,
which was usually a front-end for the @code{strop} module written in C.  The
addition of Unicode posed a difficulty for the @code{strop} module, because the
functions would all need to be rewritten in order to accept either 8-bit or
Unicode strings.  For functions such as @code{string.replace()}, which takes 3
string arguments, that means eight possible permutations, and correspondingly
complicated code.

Instead, Python 2.0 pushes the problem onto the string type, making string
manipulation functionality available through methods on both 8-bit strings and
Unicode strings.

@example
>>> 'andrew'.capitalize()
'Andrew'
>>> 'hostname'.replace('os', 'linux')
'hlinuxtname'
>>> 'moshe'.find('sh')
2
@end example

One thing that hasn’t changed, a noteworthy April Fools’ joke notwithstanding,
is that Python strings are immutable. Thus, the string methods return new
strings, and do not modify the string on which they operate.

The old @ref{f1,,string} module is still around for backwards compatibility, but it
mostly acts as a front-end to the new string methods.

Two methods which have no parallel in pre-2.0 versions, although they did exist
in JPython for quite some time, are @code{startswith()} and @code{endswith()}.
@code{s.startswith(t)} is equivalent to @code{s[:len(t)] == t}, while
@code{s.endswith(t)} is equivalent to @code{s[-len(t):] == t}.

One other method which deserves special mention is @code{join()}.  The
@code{join()} method of a string receives one parameter, a sequence of strings,
and is equivalent to the @code{string.join()} function from the old @ref{f1,,string}
module, with the arguments reversed. In other words, @code{s.join(seq)} is
equivalent to the old @code{string.join(seq, s)}.

@c ======================================================================

@node Garbage Collection of Cycles,Other Core Changes,String Methods,What’s New in Python 2 0
@anchor{whatsnew/2 0 garbage-collection-of-cycles}@anchor{1063}
@subsection Garbage Collection of Cycles


The C implementation of Python uses reference counting to implement garbage
collection.  Every Python object maintains a count of the number of references
pointing to itself, and adjusts the count as references are created or
destroyed.  Once the reference count reaches zero, the object is no longer
accessible, since you need to have a reference to an object to access it, and if
the count is zero, no references exist any longer.

Reference counting has some pleasant properties: it’s easy to understand and
implement, and the resulting implementation is portable, fairly fast, and reacts
well with other libraries that implement their own memory handling schemes.  The
major problem with reference counting is that it sometimes doesn’t realise that
objects are no longer accessible, resulting in a memory leak.  This happens when
there are cycles of references.

Consider the simplest possible cycle,  a class instance which has a reference to
itself:

@example
instance = SomeClass()
instance.myself = instance
@end example

After the above two lines of code have been executed, the reference count of
@code{instance} is 2; one reference is from the variable named @code{'instance'}, and
the other is from the @code{myself} attribute of the instance.

If the next line of code is @code{del instance}, what happens?  The reference count
of @code{instance} is decreased by 1, so it has a reference count of 1; the
reference in the @code{myself} attribute still exists.  Yet the instance is no
longer accessible through Python code, and it could be deleted.  Several objects
can participate in a cycle if they have references to each other, causing all of
the objects to be leaked.

Python 2.0 fixes this problem by periodically executing a cycle detection
algorithm which looks for inaccessible cycles and deletes the objects involved.
A new @ref{80,,gc} module provides functions to perform a garbage collection,
obtain debugging statistics, and tuning the collector’s parameters.

Running the cycle detection algorithm takes some time, and therefore will result
in some additional overhead.  It is hoped that after we’ve gotten experience
with the cycle collection from using 2.0, Python 2.1 will be able to minimize
the overhead with careful tuning.  It’s not yet obvious how much performance is
lost, because benchmarking this is tricky and depends crucially on how often the
program creates and destroys objects.  The detection of cycles can be disabled
when Python is compiled, if you can’t afford even a tiny speed penalty or
suspect that the cycle collection is buggy, by specifying the
@code{--without-cycle-gc} switch when running the @code{configure}
script.

Several people tackled this problem and contributed to a solution.  An early
implementation of the cycle detection approach was written by Toby Kelsey.  The
current algorithm was suggested by Eric Tiedemann during a visit to CNRI, and
Guido van Rossum and Neil Schemenauer wrote two different implementations, which
were later integrated by Neil.  Lots of other people offered suggestions along
the way; the March 2000 archives of the python-dev mailing list contain most of
the relevant discussion, especially in the threads titled “Reference cycle
collection for Python” and “Finalization again”.

@c ======================================================================

@node Other Core Changes,Porting to 2 0,Garbage Collection of Cycles,What’s New in Python 2 0
@anchor{whatsnew/2 0 other-core-changes}@anchor{1064}
@subsection Other Core Changes


Various minor changes have been made to Python’s syntax and built-in functions.
None of the changes are very far-reaching, but they’re handy conveniences.

@menu
* Minor Language Changes:: 
* Changes to Built-in Functions:: 

@end menu

@node Minor Language Changes,Changes to Built-in Functions,,Other Core Changes
@anchor{whatsnew/2 0 minor-language-changes}@anchor{1065}
@subsubsection Minor Language Changes


A new syntax makes it more convenient to call a given function with a tuple of
arguments and/or a dictionary of keyword arguments. In Python 1.5 and earlier,
you’d use the @code{apply()} built-in function: @code{apply(f, args, kw)} calls the
function @code{f()} with the argument tuple @emph{args} and the keyword arguments in
the dictionary @emph{kw}.  @code{apply()}  is the same in 2.0, but thanks to a patch
from Greg Ewing, @code{f(*args, **kw)} is a shorter and clearer way to achieve the
same effect.  This syntax is symmetrical with the syntax for defining
functions:

@example
def f(*args, **kw):
    # args is a tuple of positional args,
    # kw is a dictionary of keyword args
    ...
@end example

The @code{print} statement can now have its output directed to a file-like
object by following the @code{print} with  @code{>> file}, similar to the
redirection operator in Unix shells. Previously you’d either have to use the
@code{write()} method of the file-like object, which lacks the convenience and
simplicity of @code{print}, or you could assign a new value to
@code{sys.stdout} and then restore the old value.  For sending output to standard
error, it’s much easier to write this:

@example
print >> sys.stderr, "Warning: action field not supplied"
@end example

Modules can now be renamed on importing them, using the syntax @code{import module
as name} or @code{from module import name as othername}.  The patch was submitted
by Thomas Wouters.

A new format style is available when using the @code{%} operator; ‘%r’ will insert
the @ref{27d,,repr()} of its argument.  This was also added from symmetry
considerations, this time for symmetry with the existing ‘%s’ format style,
which inserts the @ref{2ef,,str()} of its argument.  For example, @code{'%r %s' % ('abc',
'abc')} returns a string containing @code{'abc' abc}.

Previously there was no way to implement a class that overrode Python’s built-in
@ref{2fc,,in} operator and implemented a custom version.  @code{obj in seq} returns
true if @emph{obj} is present in the sequence @emph{seq}; Python computes this by simply
trying every index of the sequence until either @emph{obj} is found or an
@ref{1066,,IndexError} is encountered.  Moshe Zadka contributed a patch which adds a
@code{__contains__()} magic method for providing a custom implementation for
@code{in}. Additionally, new built-in objects written in C can define what
@code{in} means for them via a new slot in the sequence protocol.

Earlier versions of Python used a recursive algorithm for deleting objects.
Deeply nested data structures could cause the interpreter to fill up the C stack
and crash; Christian Tismer rewrote the deletion logic to fix this problem.  On
a related note, comparing recursive objects recursed infinitely and crashed;
Jeremy Hylton rewrote the code to no longer crash, producing a useful result
instead.  For example, after this code:

@example
a = []
b = []
a.append(a)
b.append(b)
@end example

The comparison @code{a==b} returns true, because the two recursive data structures
are isomorphic. See the thread “trashcan and PR#7” in the April 2000 archives of
the python-dev mailing list for the discussion leading up to this
implementation, and some useful relevant links.    Note that comparisons can now
also raise exceptions. In earlier versions of Python, a comparison operation
such as @code{cmp(a,b)} would always produce an answer, even if a user-defined
@code{__cmp__()} method encountered an error, since the resulting exception would
simply be silently swallowed.

@c Starting URL:

@c https://www.python.org/pipermail/python-dev/2000-April/004834.html

Work has been done on porting Python to 64-bit Windows on the Itanium processor,
mostly by Trent Mick of ActiveState.  (Confusingly, @code{sys.platform} is still
@code{'win32'} on Win64 because it seems that for ease of porting, MS Visual C++
treats code as 32 bit on Itanium.) PythonWin also supports Windows CE; see the
Python CE page at @indicateurl{http://pythonce.sourceforge.net/} for more information.

Another new platform is Darwin/MacOS X; initial support for it is in Python 2.0.
Dynamic loading works, if you specify “configure –with-dyld –with-suffix=.x”.
Consult the README in the Python source distribution for more instructions.

An attempt has been made to alleviate one of Python’s warts, the often-confusing
@ref{21f,,NameError} exception when code refers to a local variable before the
variable has been assigned a value.  For example, the following code raises an
exception on the @code{print} statement in both 1.5.2 and 2.0; in 1.5.2 a
@ref{21f,,NameError} exception is raised, while 2.0 raises a new
@ref{1067,,UnboundLocalError} exception. @ref{1067,,UnboundLocalError} is a subclass of
@ref{21f,,NameError}, so any existing code that expects @ref{21f,,NameError} to be
raised should still work.

@example
def f():
    print "i=",i
    i = i + 1
f()
@end example

Two new exceptions, @ref{1068,,TabError} and @ref{21c,,IndentationError}, have been
introduced.  They’re both subclasses of @ref{21a,,SyntaxError}, and are raised when
Python code is found to be improperly indented.

@node Changes to Built-in Functions,,Minor Language Changes,Other Core Changes
@anchor{whatsnew/2 0 changes-to-built-in-functions}@anchor{1069}
@subsubsection Changes to Built-in Functions


A new built-in, @code{zip(seq1, seq2, ...)}, has been added.  @ref{247,,zip()}
returns a list of tuples where each tuple contains the i-th element from each of
the argument sequences.  The difference between @ref{247,,zip()} and @code{map(None,
seq1, seq2)} is that @ref{2f4,,map()} pads the sequences with @code{None} if the
sequences aren’t all of the same length, while @ref{247,,zip()} truncates the
returned list to the length of the shortest argument sequence.

The @ref{1b1,,int()} and @code{long()} functions now accept an optional “base”
parameter when the first argument is a string. @code{int('123', 10)} returns 123,
while @code{int('123', 16)} returns 291.  @code{int(123, 16)} raises a
@ref{162,,TypeError} exception with the message “can’t convert non-string with
explicit base”.

A new variable holding more detailed version information has been added to the
@ref{f7,,sys} module.  @code{sys.version_info} is a tuple @code{(major, minor, micro,
level, serial)} For example, in a hypothetical 2.0.1beta1, @code{sys.version_info}
would be @code{(2, 0, 1, 'beta', 1)}. @emph{level} is a string such as @code{"alpha"},
@code{"beta"}, or @code{"final"} for a final release.

Dictionaries have an odd new method, @code{setdefault(key, default)}, which
behaves similarly to the existing @code{get()} method.  However, if the key is
missing, @code{setdefault()} both returns the value of @emph{default} as @code{get()}
would do, and also inserts it into the dictionary as the value for @emph{key}.  Thus,
the following lines of code:

@example
if dict.has_key( key ): return dict[key]
else:
    dict[key] = []
    return dict[key]
@end example

can be reduced to a single @code{return dict.setdefault(key, [])} statement.

The interpreter sets a maximum recursion depth in order to catch runaway
recursion before filling the C stack and causing a core dump or GPF..
Previously this limit was fixed when you compiled Python, but in 2.0 the maximum
recursion depth can be read and modified using @ref{106a,,sys.getrecursionlimit()} and
@ref{106b,,sys.setrecursionlimit()}. The default value is 1000, and a rough maximum
value for a given platform can be found by running a new script,
@code{Misc/find_recursionlimit.py}.

@c ======================================================================

@node Porting to 2 0,Extending/Embedding Changes,Other Core Changes,What’s New in Python 2 0
@anchor{whatsnew/2 0 porting-to-2-0}@anchor{106c}
@subsection Porting to 2.0


New Python releases try hard to be compatible with previous releases, and the
record has been pretty good.  However, some changes are considered useful
enough, usually because they fix initial design decisions that turned out to be
actively mistaken, that breaking backward compatibility can’t always be avoided.
This section lists the changes in Python 2.0 that may cause old Python code to
break.

The change which will probably break the most code is tightening up the
arguments accepted by some methods.  Some methods would take multiple arguments
and treat them as a tuple, particularly various list methods such as
@code{append()} and @code{insert()}. In earlier versions of Python, if @code{L} is
a list, @code{L.append( 1,2 )} appends the tuple @code{(1,2)} to the list.  In Python
2.0 this causes a @ref{162,,TypeError} exception to be raised, with the message:
‘append requires exactly 1 argument; 2 given’.  The fix is to simply add an
extra set of parentheses to pass both values as a tuple:  @code{L.append( (1,2) )}.

The earlier versions of these methods were more forgiving because they used an
old function in Python’s C interface to parse their arguments; 2.0 modernizes
them to use @code{PyArg_ParseTuple()}, the current argument parsing function,
which provides more helpful error messages and treats multi-argument calls as
errors.  If you absolutely must use 2.0 but can’t fix your code, you can edit
@code{Objects/listobject.c} and define the preprocessor symbol
@code{NO_STRICT_LIST_APPEND} to preserve the old behaviour; this isn’t recommended.

Some of the functions in the @ref{ea,,socket} module are still forgiving in this
way.  For example, @code{socket.connect( ('hostname', 25) )()} is the correct
form, passing a tuple representing an IP address, but @code{socket.connect(
'hostname', 25 )()} also works. @code{socket.connect_ex()} and @code{socket.bind()}
are similarly easy-going.  2.0alpha1 tightened these functions up, but because
the documentation actually used the erroneous multiple argument form, many
people wrote code which would break with the stricter checking.  GvR backed out
the changes in the face of public reaction, so for the @ref{ea,,socket} module, the
documentation was fixed and the multiple argument form is simply marked as
deprecated; it @emph{will} be tightened up again in a future Python version.

The @code{\x} escape in string literals now takes exactly 2 hex digits.  Previously
it would consume all the hex digits following the ‘x’ and take the lowest 8 bits
of the result, so @code{\x123456} was equivalent to @code{\x56}.

The @ref{163,,AttributeError} and @ref{21f,,NameError} exceptions have a more friendly
error message, whose text will be something like @code{'Spam' instance has no
attribute 'eggs'} or @code{name 'eggs' is not defined}.  Previously the error
message was just the missing attribute name @code{eggs}, and code written to take
advantage of this fact will break in 2.0.

Some work has been done to make integers and long integers a bit more
interchangeable.  In 1.5.2, large-file support was added for Solaris, to allow
reading files larger than 2 GiB; this made the @code{tell()} method of file
objects return a long integer instead of a regular integer.  Some code would
subtract two file offsets and attempt to use the result to multiply a sequence
or slice a string, but this raised a @ref{162,,TypeError}.  In 2.0, long integers
can be used to multiply or slice a sequence, and it’ll behave as you’d
intuitively expect it to; @code{3L * 'abc'} produces ‘abcabcabc’, and
@code{(0,1,2,3)[2L:4L]} produces (2,3). Long integers can also be used in various
contexts where previously only integers were accepted, such as in the
@code{seek()} method of file objects, and in the formats supported by the @code{%}
operator (@code{%d}, @code{%i}, @code{%x}, etc.).  For example, @code{"%d" % 2L**64} will
produce the string @code{18446744073709551616}.

The subtlest long integer change of all is that the @ref{2ef,,str()} of a long
integer no longer has a trailing ‘L’ character, though @ref{27d,,repr()} still
includes it.  The ‘L’ annoyed many people who wanted to print long integers that
looked just like regular integers, since they had to go out of their way to chop
off the character.  This is no longer a problem in 2.0, but code which does
@code{str(longval)[:-1]} and assumes the ‘L’ is there, will now lose the final
digit.

Taking the @ref{27d,,repr()} of a float now uses a different formatting precision
than @ref{2ef,,str()}.  @ref{27d,,repr()} uses @code{%.17g} format string for C’s
@code{sprintf()}, while @ref{2ef,,str()} uses @code{%.12g} as before.  The effect is that
@ref{27d,,repr()} may occasionally show more decimal places than  @ref{2ef,,str()}, for
certain numbers.  For example, the number 8.1 can’t be represented exactly in
binary, so @code{repr(8.1)} is @code{'8.0999999999999996'}, while str(8.1) is
@code{'8.1'}.

The @code{-X} command-line option, which turned all standard exceptions into
strings instead of classes, has been removed; the standard exceptions will now
always be classes.  The @code{exceptions} module containing the standard
exceptions was translated from Python to a built-in C module, written by Barry
Warsaw and Fredrik Lundh.

@c Commented out for now -- I don't think anyone will care.
@c The pattern and match objects provided by SRE are C types, not Python
@c class instances as in 1.5.  This means you can no longer inherit from
@c \class{RegexObject} or \class{MatchObject}, but that shouldn't be much
@c of a problem since no one should have been doing that in the first
@c place.

@c ======================================================================

@node Extending/Embedding Changes,Distutils Making Modules Easy to Install,Porting to 2 0,What’s New in Python 2 0
@anchor{whatsnew/2 0 extending-embedding-changes}@anchor{106d}
@subsection Extending/Embedding Changes


Some of the changes are under the covers, and will only be apparent to people
writing C extension modules or embedding a Python interpreter in a larger
application.  If you aren’t dealing with Python’s C API, you can safely skip
this section.

The version number of the Python C API was incremented, so C extensions compiled
for 1.5.2 must be recompiled in order to work with 2.0.  On Windows, it’s not
possible for Python 2.0 to import a third party extension built for Python 1.5.x
due to how Windows DLLs work, so Python will raise an exception and the import
will fail.

Users of Jim Fulton’s ExtensionClass module will be pleased to find out that
hooks have been added so that ExtensionClasses are now supported by
@ref{236,,isinstance()} and @ref{237,,issubclass()}. This means you no longer have to
remember to write code such as @code{if type(obj) == myExtensionClass}, but can use
the more natural @code{if isinstance(obj, myExtensionClass)}.

The @code{Python/importdl.c} file, which was a mass of #ifdefs to support
dynamic loading on many different platforms, was cleaned up and reorganised by
Greg Stein.  @code{importdl.c} is now quite small, and platform-specific code
has been moved into a bunch of @code{Python/dynload_*.c} files.  Another
cleanup: there were also a number of @code{my*.h} files in the Include/
directory that held various portability hacks; they’ve been merged into a single
file, @code{Include/pyport.h}.

Vladimir Marangozov’s long-awaited malloc restructuring was completed, to make
it easy to have the Python interpreter use a custom allocator instead of C’s
standard @code{malloc()}.  For documentation, read the comments in
@code{Include/pymem.h} and @code{Include/objimpl.h}.  For the lengthy
discussions during which the interface was hammered out, see the web archives of
the ‘patches’ and ‘python-dev’ lists at python.org.

Recent versions of the GUSI development environment for MacOS support POSIX
threads.  Therefore, Python’s POSIX threading support now works on the
Macintosh.  Threading support using the user-space GNU @code{pth} library was also
contributed.

Threading support on Windows was enhanced, too.  Windows supports thread locks
that use kernel objects only in case of contention; in the common case when
there’s no contention, they use simpler functions which are an order of
magnitude faster.  A threaded version of Python 1.5.2 on NT is twice as slow as
an unthreaded version; with the 2.0 changes, the difference is only 10%.  These
improvements were contributed by Yakov Markovitch.

Python 2.0’s source now uses only ANSI C prototypes, so compiling Python now
requires an ANSI C compiler, and can no longer be done using a compiler that
only supports K&R C.

Previously the Python virtual machine used 16-bit numbers in its bytecode,
limiting the size of source files.  In particular, this affected the maximum
size of literal lists and dictionaries in Python source; occasionally people who
are generating Python code would run into this limit.  A patch by Charles G.
Waldman raises the limit from @code{2**16} to @code{2**32}.

Three new convenience functions intended for adding constants to a module’s
dictionary at module initialization time were added: @code{PyModule_AddObject()},
@code{PyModule_AddIntConstant()}, and @code{PyModule_AddStringConstant()}.  Each
of these functions takes a module object, a null-terminated C string containing
the name to be added, and a third argument for the value to be assigned to the
name.  This third argument is, respectively, a Python object, a C long, or a C
string.

A wrapper API was added for Unix-style signal handlers. @code{PyOS_getsig()} gets
a signal handler and @code{PyOS_setsig()} will set a new handler.

@c ======================================================================

@node Distutils Making Modules Easy to Install,XML Modules,Extending/Embedding Changes,What’s New in Python 2 0
@anchor{whatsnew/2 0 distutils-making-modules-easy-to-install}@anchor{106e}
@subsection Distutils: Making Modules Easy to Install


Before Python 2.0, installing modules was a tedious affair – there was no way
to figure out automatically where Python is installed, or what compiler options
to use for extension modules.  Software authors had to go through an arduous
ritual of editing Makefiles and configuration files, which only really work on
Unix and leave Windows and MacOS unsupported.  Python users faced wildly
differing installation instructions which varied between different extension
packages, which made administering a Python installation something of  a chore.

The SIG for distribution utilities, shepherded by Greg Ward, has created the
Distutils, a system to make package installation much easier.  They form the
@ref{37,,distutils} package, a new part of Python’s standard library. In the best
case, installing a Python module from source will require the same steps: first
you simply mean unpack the tarball or zip archive, and the run “@code{python
setup.py install}”.  The platform will be automatically detected, the compiler
will be recognized, C extension modules will be compiled, and the distribution
installed into the proper directory.  Optional command-line arguments provide
more control over the installation process, the distutils package offers many
places to override defaults – separating the build from the install, building
or installing in non-default directories, and more.

In order to use the Distutils, you need to write a @code{setup.py} script.  For
the simple case, when the software contains only .py files, a minimal
@code{setup.py} can be just a few lines long:

@example
from distutils.core import setup
setup (name = "foo", version = "1.0",
       py_modules = ["module1", "module2"])
@end example

The @code{setup.py} file isn’t much more complicated if the software consists
of a few packages:

@example
from distutils.core import setup
setup (name = "foo", version = "1.0",
       packages = ["package", "package.subpackage"])
@end example

A C extension can be the most complicated case; here’s an example taken from
the PyXML package:

@example
from distutils.core import setup, Extension

expat_extension = Extension('xml.parsers.pyexpat',
     define_macros = [('XML_NS', None)],
     include_dirs = [ 'extensions/expat/xmltok',
                      'extensions/expat/xmlparse' ],
     sources = [ 'extensions/pyexpat.c',
                 'extensions/expat/xmltok/xmltok.c',
                 'extensions/expat/xmltok/xmlrole.c', ]
       )
setup (name = "PyXML", version = "0.5.4",
       ext_modules =[ expat_extension ] )
@end example

The Distutils can also take care of creating source and binary distributions.
The “sdist” command, run by “@code{python setup.py sdist}’, builds a source
distribution such as @code{foo-1.0.tar.gz}. Adding new commands isn’t
difficult, “bdist_rpm” and “bdist_wininst” commands have already been
contributed to create an RPM distribution and a Windows installer for the
software, respectively.  Commands to create other distribution formats such as
Debian packages and Solaris @code{.pkg} files are in various stages of
development.

All this is documented in a new manual, @emph{Distributing Python Modules}, that
joins the basic set of Python documentation.

@c ======================================================================

@node XML Modules,Module changes,Distutils Making Modules Easy to Install,What’s New in Python 2 0
@anchor{whatsnew/2 0 xml-modules}@anchor{106f}
@subsection XML Modules


Python 1.5.2 included a simple XML parser in the form of the @code{xmllib}
module, contributed by Sjoerd Mullender.  Since 1.5.2’s release, two different
interfaces for processing XML have become common: SAX2 (version 2 of the Simple
API for XML) provides an event-driven interface with some similarities to
@code{xmllib}, and the DOM (Document Object Model) provides a tree-based
interface, transforming an XML document into a tree of nodes that can be
traversed and modified.  Python 2.0 includes a SAX2 interface and a stripped-down
DOM interface as part of the @ref{13a,,xml} package. Here we will give a brief
overview of these new interfaces; consult the Python documentation or the source
code for complete details. The Python XML SIG is also working on improved
documentation.

@menu
* SAX2 Support:: 
* DOM Support:: 
* Relationship to PyXML:: 

@end menu

@node SAX2 Support,DOM Support,,XML Modules
@anchor{whatsnew/2 0 sax2-support}@anchor{1070}
@subsubsection SAX2 Support


SAX defines an event-driven interface for parsing XML.  To use SAX, you must
write a SAX handler class.  Handler classes inherit from various classes
provided by SAX, and override various methods that will then be called by the
XML parser.  For example, the @code{startElement()} and @code{endElement()}
methods are called for every starting and end tag encountered by the parser, the
@code{characters()} method is called for every chunk of character data, and so
forth.

The advantage of the event-driven approach is that the whole document doesn’t
have to be resident in memory at any one time, which matters if you are
processing really huge documents.  However, writing the SAX handler class can
get very complicated if you’re trying to modify the document structure in some
elaborate way.

For example, this little example program defines a handler that prints a message
for every starting and ending tag, and then parses the file @code{hamlet.xml}
using it:

@example
from xml import sax

class SimpleHandler(sax.ContentHandler):
    def startElement(self, name, attrs):
        print 'Start of element:', name, attrs.keys()

    def endElement(self, name):
        print 'End of element:', name

# Create a parser object
parser = sax.make_parser()

# Tell it what handler to use
handler = SimpleHandler()
parser.setContentHandler( handler )

# Parse a file!
parser.parse( 'hamlet.xml' )
@end example

For more information, consult the Python documentation, or the XML HOWTO at
@indicateurl{http://pyxml.sourceforge.net/topics/howto/xml-howto.html}.

@node DOM Support,Relationship to PyXML,SAX2 Support,XML Modules
@anchor{whatsnew/2 0 dom-support}@anchor{1071}
@subsubsection DOM Support


The Document Object Model is a tree-based representation for an XML document.  A
top-level @code{Document} instance is the root of the tree, and has a single
child which is the top-level @code{Element} instance. This @code{Element}
has children nodes representing character data and any sub-elements, which may
have further children of their own, and so forth.  Using the DOM you can
traverse the resulting tree any way you like, access element and attribute
values, insert and delete nodes, and convert the tree back into XML.

The DOM is useful for modifying XML documents, because you can create a DOM
tree, modify it by adding new nodes or rearranging subtrees, and then produce a
new XML document as output.  You can also construct a DOM tree manually and
convert it to XML, which can be a more flexible way of producing XML output than
simply writing @code{<tag1>}…@code{</tag1>} to a file.

The DOM implementation included with Python lives in the @ref{13c,,xml.dom.minidom}
module.  It’s a lightweight implementation of the Level 1 DOM with support for
XML namespaces.  The  @code{parse()} and @code{parseString()} convenience
functions are provided for generating a DOM tree:

@example
from xml.dom import minidom
doc = minidom.parse('hamlet.xml')
@end example

@code{doc} is a @code{Document} instance.  @code{Document}, like all the other
DOM classes such as @code{Element} and @code{Text}, is a subclass of the
@code{Node} base class.  All the nodes in a DOM tree therefore support certain
common methods, such as @code{toxml()} which returns a string containing the XML
representation of the node and its children.  Each class also has special
methods of its own; for example, @code{Element} and @code{Document}
instances have a method to find all child elements with a given tag name.
Continuing from the previous 2-line example:

@example
perslist = doc.getElementsByTagName( 'PERSONA' )
print perslist[0].toxml()
print perslist[1].toxml()
@end example

For the @emph{Hamlet} XML file, the above few lines output:

@example
<PERSONA>CLAUDIUS, king of Denmark. </PERSONA>
<PERSONA>HAMLET, son to the late, and nephew to the present king.</PERSONA>
@end example

The root element of the document is available as @code{doc.documentElement}, and
its children can be easily modified by deleting, adding, or removing nodes:

@example
root = doc.documentElement

# Remove the first child
root.removeChild( root.childNodes[0] )

# Move the new first child to the end
root.appendChild( root.childNodes[0] )

# Insert the new first child (originally,
# the third child) before the 20th child.
root.insertBefore( root.childNodes[0], root.childNodes[20] )
@end example

Again, I will refer you to the Python documentation for a complete listing of
the different @code{Node} classes and their various methods.

@node Relationship to PyXML,,DOM Support,XML Modules
@anchor{whatsnew/2 0 relationship-to-pyxml}@anchor{1072}
@subsubsection Relationship to PyXML


The XML Special Interest Group has been working on XML-related Python code for a
while.  Its code distribution, called PyXML, is available from the SIG’s web
pages at @indicateurl{https://www.python.org/community/sigs/current/xml-sig}. The PyXML distribution also used
the package name @code{xml}.  If you’ve written programs that used PyXML, you’re
probably wondering about its compatibility with the 2.0 @ref{13a,,xml} package.

The answer is that Python 2.0’s @ref{13a,,xml} package isn’t compatible with PyXML,
but can be made compatible by installing a recent version PyXML.  Many
applications can get by with the XML support that is included with Python 2.0,
but more complicated applications will require that the full PyXML package will
be installed.  When installed, PyXML versions 0.6.0 or greater will replace the
@ref{13a,,xml} package shipped with Python, and will be a strict superset of the
standard package, adding a bunch of additional features.  Some of the additional
features in PyXML include:


@itemize *

@item 
4DOM, a full DOM implementation from FourThought, Inc.

@item 
The xmlproc validating parser, written by Lars Marius Garshol.

@item 
The @code{sgmlop} parser accelerator module, written by Fredrik Lundh.
@end itemize

@c ======================================================================

@node Module changes,New modules,XML Modules,What’s New in Python 2 0
@anchor{whatsnew/2 0 module-changes}@anchor{1073}
@subsection Module changes


Lots of improvements and bugfixes were made to Python’s extensive standard
library; some of the affected modules include @ref{d9,,readline},
@code{ConfigParser}, @ref{14,,cgi}, @ref{13,,calendar}, @ref{cc,,posix}, @ref{d9,,readline},
@code{xmllib}, @ref{4,,aifc}, @code{chunk, wave}, @ref{d7,,random}, @ref{e2,,shelve},
and @ref{bc,,nntplib}.  Consult the CVS logs for the exact patch-by-patch details.

Brian Gallew contributed OpenSSL support for the @ref{ea,,socket} module.  OpenSSL
is an implementation of the Secure Socket Layer, which encrypts the data being
sent over a socket.  When compiling Python, you can edit @code{Modules/Setup}
to include SSL support, which adds an additional function to the @ref{ea,,socket}
module: @code{socket.ssl(socket, keyfile, certfile)}, which takes a socket
object and returns an SSL socket.  The @code{httplib} and @ref{124,,urllib} modules
were also changed to support @code{https://} URLs, though no one has implemented
FTP or SMTP over SSL.

The @code{httplib} module has been rewritten by Greg Stein to support HTTP/1.1.
Backward compatibility with the 1.5 version of @code{httplib} is provided,
though using HTTP/1.1 features such as pipelining will require rewriting code to
use a different set of interfaces.

The @code{Tkinter} module now supports Tcl/Tk version 8.1, 8.2, or 8.3, and
support for the older 7.x versions has been dropped.  The Tkinter module now
supports displaying Unicode strings in Tk widgets. Also, Fredrik Lundh
contributed an optimization which makes operations like @code{create_line} and
@code{create_polygon} much faster, especially when using lots of coordinates.

The @ref{2a,,curses} module has been greatly extended, starting from Oliver
Andrich’s enhanced version, to provide many additional functions from ncurses
and SYSV curses, such as colour, alternative character set support, pads, and
mouse support.  This means the module is no longer compatible with operating
systems that only have BSD curses, but there don’t seem to be any currently
maintained OSes that fall into this category.

As mentioned in the earlier discussion of 2.0’s Unicode support, the underlying
implementation of the regular expressions provided by the @ref{d8,,re} module has
been changed.  SRE, a new regular expression engine written by Fredrik Lundh and
partially funded by Hewlett Packard, supports matching against both 8-bit
strings and Unicode strings.

@c ======================================================================

@node New modules,IDLE Improvements,Module changes,What’s New in Python 2 0
@anchor{whatsnew/2 0 new-modules}@anchor{1074}
@subsection New modules


A number of new modules were added.  We’ll simply list them with brief
descriptions; consult the 2.0 documentation for the details of a particular
module.


@itemize *

@item 
@ref{b,,atexit}:  For registering functions to be called before the Python
interpreter exits. Code that currently sets @code{sys.exitfunc} directly should be
changed to  use the @ref{b,,atexit} module instead, importing @ref{b,,atexit} and
calling @ref{343,,atexit.register()} with  the function to be called on exit.
(Contributed by Skip Montanaro.)

@item 
@ref{1a,,codecs}, @code{encodings}, @ref{121,,unicodedata}:  Added as part of the new
Unicode support.

@item 
@ref{7a,,filecmp}: Supersedes the old @code{cmp}, @code{cmpcache} and
@code{dircmp} modules, which have now become deprecated. (Contributed by Gordon
MacMillan and Moshe Zadka.)

@item 
@ref{83,,gettext}: This module provides internationalization (I18N) and
localization (L10N) support for Python programs by providing an interface to the
GNU gettext message catalog library. (Integrated by Barry Warsaw, from separate
contributions by Martin  von Löwis, Peter Funk, and James Henstridge.)

@item 
@code{linuxaudiodev}: Support for the @code{/dev/audio} device on Linux, a
twin to the existing @code{sunaudiodev} module. (Contributed by Peter Bosch,
with fixes by Jeremy Hylton.)

@item 
@ref{af,,mmap}: An interface to memory-mapped files on both Windows and Unix.  A
file’s contents can be mapped directly into memory, at which point it behaves
like a mutable string, so its contents can be read and modified.  They can even
be passed to functions that expect ordinary strings, such as the @ref{d8,,re}
module. (Contributed by Sam Rushing, with some extensions by A.M. Kuchling.)

@item 
@code{pyexpat}: An interface to the Expat XML parser. (Contributed by Paul
Prescod.)

@item 
@code{robotparser}: Parse a @code{robots.txt} file, which is used for writing
web spiders that politely avoid certain areas of a web site.  The parser accepts
the contents of a @code{robots.txt} file, builds a set of rules from it, and
can then answer questions about the fetchability of a given URL.  (Contributed
by Skip Montanaro.)

@item 
@ref{fa,,tabnanny}: A module/script to  check Python source code for ambiguous
indentation. (Contributed by Tim Peters.)

@item 
@code{UserString}: A base class useful for deriving objects that behave like
strings.

@item 
@ref{130,,webbrowser}: A module that provides a platform independent way to launch
a web browser on a specific URL. For each platform, various browsers are tried
in a specific order. The user can alter which browser is launched by setting the
@emph{BROWSER} environment variable.  (Originally inspired by Eric S. Raymond’s patch
to @ref{124,,urllib} which added similar functionality, but the final module comes
from code originally  implemented by Fred Drake as
@code{Tools/idle/BrowserControl.py}, and adapted for the standard library by
Fred.)

@item 
@code{_winreg}: An interface to the Windows registry.  @code{_winreg} is an
adaptation of functions that have been part of PythonWin since 1995, but has now
been added to the core  distribution, and enhanced to support Unicode.
@code{_winreg} was written by Bill Tutt and Mark Hammond.

@item 
@ref{149,,zipfile}: A module for reading and writing ZIP-format archives.  These
are archives produced by @code{PKZIP} on DOS/Windows or @code{zip} on
Unix, not to be confused with @code{gzip}-format files (which are
supported by the @ref{87,,gzip} module) (Contributed by James C. Ahlstrom.)

@item 
@code{imputil}: A module that provides a simpler way for writing customized
import hooks, in comparison to the existing @code{ihooks} module.  (Implemented
by Greg Stein, with much discussion on python-dev along the way.)
@end itemize

@c ======================================================================

@node IDLE Improvements,Deleted and Deprecated Modules,New modules,What’s New in Python 2 0
@anchor{whatsnew/2 0 idle-improvements}@anchor{1075}
@subsection IDLE Improvements


IDLE is the official Python cross-platform IDE, written using Tkinter. Python
2.0 includes IDLE 0.6, which adds a number of new features and improvements.  A
partial list:


@itemize *

@item 
UI improvements and optimizations, especially in the area of syntax
highlighting and auto-indentation.

@item 
The class browser now shows more information, such as the top level functions
in a module.

@item 
Tab width is now a user settable option. When opening an existing Python file,
IDLE automatically detects the indentation conventions, and adapts.

@item 
There is now support for calling browsers on various platforms, used to open
the Python documentation in a browser.

@item 
IDLE now has a command line, which is largely similar to  the vanilla Python
interpreter.

@item 
Call tips were added in many places.

@item 
IDLE can now be installed as a package.

@item 
In the editor window, there is now a line/column bar at the bottom.

@item 
Three new keystroke commands: Check module (@code{Alt-F5}), Import module (@code{F5}) and
Run script (@code{Ctrl-F5}).
@end itemize

@c ======================================================================

@node Deleted and Deprecated Modules,Acknowledgements<8>,IDLE Improvements,What’s New in Python 2 0
@anchor{whatsnew/2 0 deleted-and-deprecated-modules}@anchor{1076}
@subsection Deleted and Deprecated Modules


A few modules have been dropped because they’re obsolete, or because there are
now better ways to do the same thing.  The @code{stdwin} module is gone; it was
for a platform-independent windowing toolkit that’s no longer developed.

A number of modules have been moved to the @code{lib-old} subdirectory:
@code{cmp}, @code{cmpcache}, @code{dircmp}, @code{dump},  @code{find},
@code{grep}, @code{packmail},  @code{poly}, @code{util}, @code{whatsound},
@code{zmod}.  If you have code which relies on a module  that’s been moved to
@code{lib-old}, you can simply add that directory to @code{sys.path}   to get them
back, but you’re encouraged to update any code that uses these modules.

@node Acknowledgements<8>,,Deleted and Deprecated Modules,What’s New in Python 2 0
@anchor{whatsnew/2 0 acknowledgements}@anchor{1077}
@subsection Acknowledgements


The authors would like to thank the following people for offering suggestions on
various drafts of this article: David Bolen, Mark Hammond, Gregg Hauser, Jeremy
Hylton, Fredrik Lundh, Detlef Lannert, Aahz Maruch, Skip Montanaro, Vladimir
Marangozov, Tobias Polzin, Guido van Rossum, Neil Schemenauer, and Russ Schmidt.

The “Changelog” is an HTML version of the file built@footnote{https://pypi.org/project/blurb} from the contents of the
Misc/NEWS.d@footnote{https://github.com/python/cpython/tree/main/Misc/NEWS.d} directory tree, which contains @emph{all} nontrivial changes
to Python for the current version.

@node Changelog,,What’s New in Python 2 0,What’s New in Python
@anchor{whatsnew/changelog doc}@anchor{1078}@anchor{whatsnew/changelog changelog}@anchor{154}@anchor{whatsnew/changelog id1}@anchor{1079}
@section Changelog


@menu
* Python next:: 
* Python 3.11.0 alpha 5: Python 3 11 0 alpha 5. 
* Python 3.11.0 alpha 4: Python 3 11 0 alpha 4. 
* Python 3.11.0 alpha 3: Python 3 11 0 alpha 3. 
* Python 3.11.0 alpha 2: Python 3 11 0 alpha 2. 
* Python 3.11.0 alpha 1: Python 3 11 0 alpha 1. 
* Python 3.10.0 beta 1: Python 3 10 0 beta 1. 
* Python 3.10.0 alpha 7: Python 3 10 0 alpha 7. 
* Python 3.10.0 alpha 6: Python 3 10 0 alpha 6. 
* Python 3.10.0 alpha 5: Python 3 10 0 alpha 5. 
* Python 3.10.0 alpha 4: Python 3 10 0 alpha 4. 
* Python 3.10.0 alpha 3: Python 3 10 0 alpha 3. 
* Python 3.10.0 alpha 2: Python 3 10 0 alpha 2. 
* Python 3.10.0 alpha 1: Python 3 10 0 alpha 1. 
* Python 3.9.0 beta 1: Python 3 9 0 beta 1. 
* Python 3.9.0 alpha 6: Python 3 9 0 alpha 6. 
* Python 3.9.0 alpha 5: Python 3 9 0 alpha 5. 
* Python 3.9.0 alpha 4: Python 3 9 0 alpha 4. 
* Python 3.9.0 alpha 3: Python 3 9 0 alpha 3. 
* Python 3.9.0 alpha 2: Python 3 9 0 alpha 2. 
* Python 3.9.0 alpha 1: Python 3 9 0 alpha 1. 
* Python 3.8.0 beta 1: Python 3 8 0 beta 1. 
* Python 3.8.0 alpha 4: Python 3 8 0 alpha 4. 
* Python 3.8.0 alpha 3: Python 3 8 0 alpha 3. 
* Python 3.8.0 alpha 2: Python 3 8 0 alpha 2. 
* Python 3.8.0 alpha 1: Python 3 8 0 alpha 1. 
* Python 3.7.0 final: Python 3 7 0 final. 
* Python 3.7.0 release candidate 1: Python 3 7 0 release candidate 1. 
* Python 3.7.0 beta 5: Python 3 7 0 beta 5. 
* Python 3.7.0 beta 4: Python 3 7 0 beta 4. 
* Python 3.7.0 beta 3: Python 3 7 0 beta 3. 
* Python 3.7.0 beta 2: Python 3 7 0 beta 2. 
* Python 3.7.0 beta 1: Python 3 7 0 beta 1. 
* Python 3.7.0 alpha 4: Python 3 7 0 alpha 4. 
* Python 3.7.0 alpha 3: Python 3 7 0 alpha 3. 
* Python 3.7.0 alpha 2: Python 3 7 0 alpha 2. 
* Python 3.7.0 alpha 1: Python 3 7 0 alpha 1. 
* Python 3.6.6 final: Python 3 6 6 final. 
* Python 3.6.6 release candidate 1: Python 3 6 6 release candidate 1. 
* Python 3.6.5 final: Python 3 6 5 final. 
* Python 3.6.5 release candidate 1: Python 3 6 5 release candidate 1. 
* Python 3.6.4 final: Python 3 6 4 final. 
* Python 3.6.4 release candidate 1: Python 3 6 4 release candidate 1. 
* Python 3.6.3 final: Python 3 6 3 final. 
* Python 3.6.3 release candidate 1: Python 3 6 3 release candidate 1. 
* Python 3.6.2 final: Python 3 6 2 final. 
* Python 3.6.2 release candidate 2: Python 3 6 2 release candidate 2. 
* Python 3.6.2 release candidate 1: Python 3 6 2 release candidate 1. 
* Python 3.6.1 final: Python 3 6 1 final. 
* Python 3.6.1 release candidate 1: Python 3 6 1 release candidate 1. 
* Python 3.6.0 final: Python 3 6 0 final. 
* Python 3.6.0 release candidate 2: Python 3 6 0 release candidate 2. 
* Python 3.6.0 release candidate 1: Python 3 6 0 release candidate 1. 
* Python 3.6.0 beta 4: Python 3 6 0 beta 4. 
* Python 3.6.0 beta 3: Python 3 6 0 beta 3. 
* Python 3.6.0 beta 2: Python 3 6 0 beta 2. 
* Python 3.6.0 beta 1: Python 3 6 0 beta 1. 
* Python 3.6.0 alpha 4: Python 3 6 0 alpha 4. 
* Python 3.6.0 alpha 3: Python 3 6 0 alpha 3. 
* Python 3.6.0 alpha 2: Python 3 6 0 alpha 2. 
* Python 3.6.0 alpha 1: Python 3 6 0 alpha 1. 
* Python 3.5.5 final: Python 3 5 5 final. 
* Python 3.5.5 release candidate 1: Python 3 5 5 release candidate 1. 
* Python 3.5.4 final: Python 3 5 4 final. 
* Python 3.5.4 release candidate 1: Python 3 5 4 release candidate 1. 
* Python 3.5.3 final: Python 3 5 3 final. 
* Python 3.5.3 release candidate 1: Python 3 5 3 release candidate 1. 
* Python 3.5.2 final: Python 3 5 2 final. 
* Python 3.5.2 release candidate 1: Python 3 5 2 release candidate 1. 
* Python 3.5.1 final: Python 3 5 1 final. 
* Python 3.5.1 release candidate 1: Python 3 5 1 release candidate 1. 
* Python 3.5.0 final: Python 3 5 0 final. 
* Python 3.5.0 release candidate 4: Python 3 5 0 release candidate 4. 
* Python 3.5.0 release candidate 3: Python 3 5 0 release candidate 3. 
* Python 3.5.0 release candidate 2: Python 3 5 0 release candidate 2. 
* Python 3.5.0 release candidate 1: Python 3 5 0 release candidate 1. 
* Python 3.5.0 beta 4: Python 3 5 0 beta 4. 
* Python 3.5.0 beta 3: Python 3 5 0 beta 3. 
* Python 3.5.0 beta 2: Python 3 5 0 beta 2. 
* Python 3.5.0 beta 1: Python 3 5 0 beta 1. 
* Python 3.5.0 alpha 4: Python 3 5 0 alpha 4. 
* Python 3.5.0 alpha 3: Python 3 5 0 alpha 3. 
* Python 3.5.0 alpha 2: Python 3 5 0 alpha 2. 
* Python 3.5.0 alpha 1: Python 3 5 0 alpha 1. 

@end menu

@node Python next,Python 3 11 0 alpha 5,,Changelog
@anchor{whatsnew/changelog python-next}@anchor{107a}
@subsection Python next


@emph{Release date: XXXX-XX-XX}

@menu
* Core and Builtins:: 
* Library:: 
* Documentation: Documentation<2>. 
* Tests:: 
* Build:: 
* Windows:: 
* IDLE: IDLE<3>. 
* C API:: 

@end menu

@node Core and Builtins,Library,,Python next
@anchor{whatsnew/changelog core-and-builtins}@anchor{107b}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-46927@footnote{https://bugs.python.org/issue46927}: Include the type’s name in the error message for subscripting
non-generic types.

@item 
bpo-46921@footnote{https://bugs.python.org/issue46921}: Support vectorcall for @code{super()}. Patch by Ken Jin.

@item 
bpo-46841@footnote{https://bugs.python.org/issue46841}: Fix incorrect handling of inline cache entries when
specializing @ref{19e,,BINARY_OP}.

@item 
bpo-46841@footnote{https://bugs.python.org/issue46841}: Use an oparg to simplify the construction of helpful error
messages in @ref{107c,,GET_AWAITABLE}.

@item 
bpo-46903@footnote{https://bugs.python.org/issue46903}: Make sure that str subclasses can be used as attribute names
for instances with virtual dictionaries. Fixes regression in 3.11alpha

@item 
bpo-46841@footnote{https://bugs.python.org/issue46841}: Add more detailed specialization failure stats for
@ref{46f,,COMPARE_OP} followed by @ref{107d,,EXTENDED_ARG}.

@item 
bpo-46891@footnote{https://bugs.python.org/issue46891}: Fix bug introduced during 3.11alpha where subclasses of
@code{types.ModuleType} with @code{__slots__} were not initialized correctly,
resulting in an interpreter crash.

@item 
bpo-46841@footnote{https://bugs.python.org/issue46841}: Use inline caching for @ref{107e,,LOAD_ATTR},
@ref{75c,,LOAD_METHOD}, and @ref{107f,,STORE_ATTR}.

@item 
bpo-46841@footnote{https://bugs.python.org/issue46841}: Use inline cache for @ref{1080,,BINARY_SUBSCR}.

@item 
bpo-46841@footnote{https://bugs.python.org/issue46841}: Use inline caching for @ref{46f,,COMPARE_OP}.

@item 
bpo-46864@footnote{https://bugs.python.org/issue46864}: Deprecate @code{PyBytesObject.ob_shash}. It will be removed in
Python 3.13.

@item 
bpo-46841@footnote{https://bugs.python.org/issue46841}: Use inline caching for @ref{1081,,UNPACK_SEQUENCE}.

@item 
bpo-46845@footnote{https://bugs.python.org/issue46845}: Reduces dict size by removing hash value from hash table when
all inserted keys are Unicode. For example,
@code{sys.getsizeof(dict.fromkeys("abcdefg"))} becomes 272 bytes from 352
bytes on 64bit platform.

@item 
bpo-46841@footnote{https://bugs.python.org/issue46841}: Use inline cache for @ref{1082,,LOAD_GLOBAL}.

@item 
bpo-46852@footnote{https://bugs.python.org/issue46852}: Rename the private undocumented @code{float.__set_format__()}
method to @code{float.__setformat__()} to fix a typo introduced in Python
3.7. The method is only used by test_float. Patch by Victor Stinner.

@item 
bpo-46852@footnote{https://bugs.python.org/issue46852}: Remove the undocumented private @code{float.__set_format__()}
method, previously known as @code{float.__setformat__()} in Python 3.7. Its
docstring said: “You probably don’t want to use this function. It exists
mainly to be used in Python’s test suite.” Patch by Victor Stinner.

@item 
bpo-40116@footnote{https://bugs.python.org/issue40116}: Fix regression that dict.update(other) may don’t respect
iterate order of other when other is key sharing dict.

@item 
bpo-46712@footnote{https://bugs.python.org/issue46712}: Share global string identifiers in deep-frozen modules.

@item 
bpo-46430@footnote{https://bugs.python.org/issue46430}: Fix memory leak in interned strings of deep-frozen modules.

@item 
bpo-46841@footnote{https://bugs.python.org/issue46841}: Store @ref{19e,,BINARY_OP} caches inline using a new
@code{CACHE} instruction.

@item 
bpo-45107@footnote{https://bugs.python.org/issue45107}: Specialize @code{LOAD_METHOD} for instances with a dict.

@item 
bpo-44337@footnote{https://bugs.python.org/issue44337}: Reduce the memory usage of specialized @ref{107e,,LOAD_ATTR} and
@ref{107f,,STORE_ATTR} instructions.

@item 
bpo-46729@footnote{https://bugs.python.org/issue46729}: Add number of sub-exceptions to @code{BaseException.__str__()}.

@item 
bpo-45885@footnote{https://bugs.python.org/issue45885}: Don’t un-adapt @ref{46f,,COMPARE_OP} when collecting
specialization stats.

@item 
bpo-46329@footnote{https://bugs.python.org/issue46329}: Fix specialization stats gathering for @ref{1a0,,PRECALL}
instructions.

@item 
bpo-46794@footnote{https://bugs.python.org/issue46794}: Bump up the libexpat version into 2.4.6

@item 
bpo-46823@footnote{https://bugs.python.org/issue46823}: Implement a specialized combined opcode
@code{LOAD_FAST__LOAD_ATTR_INSTANCE_VALUE}.  Patch by Dennis Sweeney.

@item 
bpo-46820@footnote{https://bugs.python.org/issue46820}: Fix parsing a numeric literal immediately (without spaces)
followed by “not in” keywords, like in @code{1not in x}. Now the parser only
emits a warning, not a syntax error.

@item 
bpo-46329@footnote{https://bugs.python.org/issue46329}: Move @code{KW_NAMES} before @code{PRECALL} instruction in call
sequence. Change @code{operand} of @code{CALL} to match @code{PRECALL} for easier
specialization.

@item 
bpo-46808@footnote{https://bugs.python.org/issue46808}: Remove the @code{NEXT_BLOCK} macro from compile.c, and make the
compiler automatically generate implicit blocks when they are needed.

@item 
bpo-46329@footnote{https://bugs.python.org/issue46329}: Add @code{PUSH_NULL} instruction. This is used as a prefix when
evaluating a callable, so that the stack has the same shape for methods
and other calls. @code{PRECALL_FUNCTION} and @code{PRECALL_METHOD} are merged
into a single @code{PRECALL} instruction.

There is no change in semantics.

@item 
bpo-46762@footnote{https://bugs.python.org/issue46762}: Fix an assert failure in debug builds when a ‘<’, ‘>’, or ‘=’
is the last character in an f-string that’s missing a closing right brace.

@item 
bpo-46730@footnote{https://bugs.python.org/issue46730}: Message of AttributeError caused by getting, setting or
deleting a property without the corresponding function now mentions that
the attribute is in fact a property and also specifies type of the class
that it belongs to.

@item 
bpo-46724@footnote{https://bugs.python.org/issue46724}: Make sure that all backwards jumps use the @code{JUMP_ABSOLUTE}
instruction, rather than @code{JUMP_FORWARD} with an argument of
@code{(2**32)+offset}.

@item 
bpo-46732@footnote{https://bugs.python.org/issue46732}: Correct the docstring for the @code{__bool__()} method. Patch by
Jelle Zijlstra.

@item 
bpo-46072@footnote{https://bugs.python.org/issue46072}: Add more detailed specialization failure statistics for
@ref{19e,,BINARY_OP}.

@item 
bpo-46707@footnote{https://bugs.python.org/issue46707}: Avoid potential exponential backtracking when producing some
syntax errors involving lots of brackets. Patch by Pablo Galindo.

@item 
bpo-46323@footnote{https://bugs.python.org/issue46323}: @ref{29,,ctypes} now allocates memory on the stack instead of on
the heap to pass arguments while calling a Python callback function. Patch
by Dong-hee Na.

@item 
bpo-45923@footnote{https://bugs.python.org/issue45923}: Add a quickened form of @ref{1083,,RESUME} that skips quickening
checks.

@item 
bpo-46702@footnote{https://bugs.python.org/issue46702}: Specialize @ref{1081,,UNPACK_SEQUENCE} for @ref{3e5,,tuple} and
@ref{43a,,list} unpackings.

@item 
bpo-46072@footnote{https://bugs.python.org/issue46072}: Opcode pair stats are now gathered with @code{--enable-pystats}.
Defining @code{DYNAMIC_EXECUTION_PROFILE} or  @code{DXPAIRS} no longer has any
effect.

@item 
bpo-46675@footnote{https://bugs.python.org/issue46675}: Allow more than 16 items in a split dict before it is combined.
The limit is now 254.

@item 
bpo-40479@footnote{https://bugs.python.org/issue40479}: Add a missing call to @code{va_end()} in
@code{Modules/_hashopenssl.c}.

@item 
bpo-46323@footnote{https://bugs.python.org/issue46323}: Use @ref{1084,,PyObject_Vectorcall()} while calling ctypes callback
function. Patch by Dong-hee Na.

@item 
bpo-46615@footnote{https://bugs.python.org/issue46615}: When iterating over sets internally in @code{setobject.c}, acquire
strong references to the resulting items from the set.  This prevents
crashes in corner-cases of various set operations where the set gets
mutated.

@item 
bpo-45828@footnote{https://bugs.python.org/issue45828}: The bytecode compiler now attempts to apply runtime stack
manipulations at compile-time (whenever it is feasible to do so).

@item 
bpo-30496@footnote{https://bugs.python.org/issue30496}: Fixed a minor portability issue in the implementation of
@ref{1085,,PyLong_FromLong()}, and added a fast path for single-digit integers
to @ref{1086,,PyLong_FromLongLong()}.
@end itemize

@node Library,Documentation<2>,Core and Builtins,Python next
@anchor{whatsnew/changelog library}@anchor{1087}
@subsubsection Library



@itemize -

@item 
bpo-46877@footnote{https://bugs.python.org/issue46877}: Export @ref{1088,,unittest.doModuleCleanups()} in @ref{122,,unittest}.
Patch by Kumar Aditya.

@item 
bpo-46848@footnote{https://bugs.python.org/issue46848}: For performance, use the optimized string-searching
implementations from @ref{1089,,find()} and @ref{108a,,rfind()} for
@code{find()} and @code{rfind()}.

@item 
bpo-46736@footnote{https://bugs.python.org/issue46736}: @ref{666,,SimpleHTTPRequestHandler} now uses HTML5
grammar. Patch by Dong-hee Na.

@item 
bpo-44886@footnote{https://bugs.python.org/issue44886}: Inherit asyncio proactor datagram transport from
@ref{108b,,asyncio.DatagramTransport}.

@item 
bpo-46827@footnote{https://bugs.python.org/issue46827}: Support UDP sockets in  @ref{928,,asyncio.loop.sock_connect()} for
selector-based event loops.  Patch by Thomas Grainger.

@item 
bpo-46811@footnote{https://bugs.python.org/issue46811}: Make test suite support Expat >=2.4.5

@item 
bpo-46252@footnote{https://bugs.python.org/issue46252}: Raise @ref{162,,TypeError} if @ref{3d6,,ssl.SSLSocket} is passed to
transport-based APIs.

@item 
bpo-46784@footnote{https://bugs.python.org/issue46784}: Fix libexpat symbols collisions with user dynamically loaded or
statically linked libexpat in embedded Python.

@item 
bpo-46786@footnote{https://bugs.python.org/issue46786}: The HTML serialisation in xml.etree.ElementTree now writes
@code{embed}, @code{source}, @code{track} and @code{wbr} as empty tags, as defined in
HTML 5.

@item 
bpo-39327@footnote{https://bugs.python.org/issue39327}: @ref{53a,,shutil.rmtree()} can now work with VirtualBox shared
folders when running from the guest operating-system.

@item 
bpo-45390@footnote{https://bugs.python.org/issue45390}: Propagate @ref{4d7,,asyncio.CancelledError} message from inner task
to outer awaiter.

@item 
bpo-46756@footnote{https://bugs.python.org/issue46756}: Fix a bug in
@ref{108c,,urllib.request.HTTPPasswordMgr.find_user_password()} and
@ref{108d,,urllib.request.HTTPPasswordMgrWithPriorAuth.is_authenticated()} which
allowed to bypass authorization. For example, access to URI
@code{example.org/foobar} was allowed if the user was authorized for URI
@code{example.org/foo}.

@item 
bpo-46737@footnote{https://bugs.python.org/issue46737}: @ref{108e,,random.gauss()} and @ref{108f,,random.normalvariate()} now have
default arguments.

@item 
bpo-46752@footnote{https://bugs.python.org/issue46752}: Add task groups to asyncio (structured concurrency, inspired by
Trio’s nurseries). This also introduces a change to task cancellation,
where a cancelled task can’t be cancelled again until it calls
.uncancel().

@item 
bpo-46724@footnote{https://bugs.python.org/issue46724}: Fix @ref{36,,dis} behavior on negative jump offsets.

@item 
bpo-46333@footnote{https://bugs.python.org/issue46333}: The @code{__repr__()} method of @ref{1090,,typing.ForwardRef} now
includes the @code{module} parameter of @ref{1090,,typing.ForwardRef} when it is
set.

@item 
bpo-46643@footnote{https://bugs.python.org/issue46643}: In @ref{432,,typing.get_type_hints()}, support evaluating
stringified @code{ParamSpecArgs} and @code{ParamSpecKwargs} annotations. Patch
by Gregory Beauregard.

@item 
bpo-45863@footnote{https://bugs.python.org/issue45863}: When the @ref{fb,,tarfile} module creates a pax format archive, it
will put an integer representation of timestamps in the ustar header (if
possible) for the benefit of older unarchivers, in addition to the
existing full-precision timestamps in the pax extended header.

@item 
bpo-46066@footnote{https://bugs.python.org/issue46066}: Deprecate kwargs-based syntax for @ref{2de,,typing.TypedDict}
definitions. It had confusing semantics when specifying totality, and was
largely unused. Patch by Jingchen Ye.

@item 
bpo-46676@footnote{https://bugs.python.org/issue46676}: Make @ref{23a,,typing.ParamSpec} args and kwargs equal to
themselves. Patch by Gregory Beauregard.

@item 
bpo-46323@footnote{https://bugs.python.org/issue46323}: @code{ctypes.CFUNCTYPE()} and @code{ctypes.WINFUNCTYPE()} now fail to
create the type if its @code{_argtypes_} member contains too many arguments.
Previously, the error was only raised when calling a function. Patch by
Victor Stinner.

@item 
bpo-46672@footnote{https://bugs.python.org/issue46672}: Fix @code{NameError} in @ref{31a,,asyncio.gather()} when initial type
check fails.

@item 
bpo-46659@footnote{https://bugs.python.org/issue46659}: The @ref{1cd,,calendar.LocaleTextCalendar} and
@ref{1ce,,calendar.LocaleHTMLCalendar} classes now use
@ref{1b6,,locale.getlocale()}, instead of using
@ref{1b3,,locale.getdefaultlocale()}, if no locale is specified. Patch by
Victor Stinner.

@item 
bpo-46659@footnote{https://bugs.python.org/issue46659}: The @ref{1b3,,locale.getdefaultlocale()} function is deprecated and
will be removed in Python 3.13. Use @ref{1b4,,locale.setlocale()},
@ref{1b5,,locale.getpreferredencoding(False)}
and @ref{1b6,,locale.getlocale()} functions instead.  Patch by Victor Stinner.

@item 
bpo-46655@footnote{https://bugs.python.org/issue46655}: In @ref{432,,typing.get_type_hints()}, support evaluating bare
stringified @code{TypeAlias} annotations. Patch by Gregory Beauregard.

@item 
bpo-45948@footnote{https://bugs.python.org/issue45948}: Fixed a discrepancy in the C implementation of the
@ref{13e,,xml.etree.ElementTree} module. Now, instantiating an
@ref{569,,xml.etree.ElementTree.XMLParser} with a @code{target=None} keyword
provides a default @ref{56a,,xml.etree.ElementTree.TreeBuilder} target as
the Python implementation does.

@item 
bpo-46626@footnote{https://bugs.python.org/issue46626}: Expose Linux’s @code{IP_BIND_ADDRESS_NO_PORT} option in
@ref{ea,,socket}.

@item 
bpo-46521@footnote{https://bugs.python.org/issue46521}: Fix a bug in the @ref{1b,,codeop} module that was incorrectly
identifying invalid code involving string quotes as valid code.

@item 
bpo-46571@footnote{https://bugs.python.org/issue46571}: Improve @ref{1091,,typing.no_type_check()}.

Now it does not modify external classes and functions. We also now
correctly mark classmethods as not to be type checked.

@item 
bpo-46400@footnote{https://bugs.python.org/issue46400}: expat: Update libexpat from 2.4.1 to 2.4.4

@item 
bpo-46556@footnote{https://bugs.python.org/issue46556}: Deprecate undocumented support for using a
@ref{522,,pathlib.Path} object as a context manager.

@item 
bpo-46534@footnote{https://bugs.python.org/issue46534}: Implement 
@geindex Python Enhancement Proposals; PEP 673
PEP 673@footnote{https://www.python.org/dev/peps/pep-0673} @ref{1092,,typing.Self}. Patch by James
Hilton-Balfe.

@item 
bpo-46522@footnote{https://bugs.python.org/issue46522}: Make various module @code{__getattr__} AttributeErrors more
closely match a typical AttributeError

@item 
bpo-46475@footnote{https://bugs.python.org/issue46475}: Add @ref{1093,,typing.Never} and @ref{1094,,typing.assert_never()}. Patch
by Jelle Zijlstra.

@item 
bpo-46333@footnote{https://bugs.python.org/issue46333}: The @code{__eq__()} and @code{__hash__()} methods of
@ref{1090,,typing.ForwardRef} now honor the @code{module} parameter of
@ref{1090,,typing.ForwardRef}. Forward references from different modules are
now differentiated.

@item 
bpo-46246@footnote{https://bugs.python.org/issue46246}: Add missing @code{__slots__} to
@code{importlib.metadata.DeprecatedList}. Patch by Arie Bovenberg.

@item 
bpo-46232@footnote{https://bugs.python.org/issue46232}: The @ref{ee,,ssl} module now handles certificates with bit strings
in DN correctly.

@item 
bpo-46195@footnote{https://bugs.python.org/issue46195}: @ref{432,,typing.get_type_hints()} no longer adds @code{Optional} to
parameters with @code{None} as a default. This aligns to changes to PEP 484
in @indicateurl{https://github.com/python/peps/pull/689}

@item 
bpo-31369@footnote{https://bugs.python.org/issue31369}: Add @ref{1095,,RegexFlag} to @code{re.__all__} and documented it.
Add @code{NOFLAG} to indicate no flags being set.

@item 
bpo-45898@footnote{https://bugs.python.org/issue45898}: @ref{29,,ctypes} no longer defines @code{ffi_type_*} symbols in
@code{cfield.c}. The symbols have been provided by libffi for over a decade.

@item 
bpo-44953@footnote{https://bugs.python.org/issue44953}: Calling @code{operator.itemgetter} objects and
@code{operator.attrgetter} objects is now faster due to use of the vectorcall
calling convention.

@item 
bpo-44289@footnote{https://bugs.python.org/issue44289}: Fix an issue with @ref{1096,,is_tarfile()} method when using
@emph{fileobj} argument: position in the @emph{fileobj} was advanced forward which
made it unreadable with @ref{1097,,tarfile.TarFile.open()}.

@item 
bpo-44011@footnote{https://bugs.python.org/issue44011}: Reimplement SSL/TLS support in asyncio, borrow the
implementation from uvloop library.

@item 
bpo-41086@footnote{https://bugs.python.org/issue41086}: Make the @ref{759,,configparser.ConfigParser} constructor raise
@ref{162,,TypeError} if the @code{interpolation} parameter is not of type
@code{configparser.Interpolation}

@item 
bpo-29418@footnote{https://bugs.python.org/issue29418}: Implement @ref{176,,inspect.ismethodwrapper()} and fix
@ref{1098,,inspect.isroutine()} for cases where methodwrapper is given. Patch by
Hakan Çelik.

@item 
bpo-14156@footnote{https://bugs.python.org/issue14156}: argparse.FileType now supports an argument of ‘-’ in binary
mode, returning the .buffer attribute of sys.stdin/sys.stdout as
appropriate. Modes including ‘x’ and ‘a’ are treated equivalently to ‘w’
when argument is ‘-’. Patch contributed by Josh Rosenberg
@end itemize

@node Documentation<2>,Tests,Library,Python next
@anchor{whatsnew/changelog documentation}@anchor{1099}
@subsubsection Documentation



@itemize -

@item 
bpo-42238@footnote{https://bugs.python.org/issue42238}: @code{Doc/tools/rstlint.py} has moved to its own repository and is
now packaged on PyPI as @code{sphinx-lint}.
@end itemize

@node Tests,Build,Documentation<2>,Python next
@anchor{whatsnew/changelog tests}@anchor{109a}
@subsubsection Tests



@itemize -

@item 
bpo-46913@footnote{https://bugs.python.org/issue46913}: Fix test_faulthandler.test_sigfpe() if Python is built with
undefined behavior sanitizer (UBSAN): disable UBSAN on the
faulthandler_sigfpe() function. Patch by Victor Stinner.

@item 
bpo-46760@footnote{https://bugs.python.org/issue46760}: Remove bytecode offsets from expected values in test.test_dis
module. Reduces the obstacles to modifying the VM or compiler.

@item 
bpo-46708@footnote{https://bugs.python.org/issue46708}: Prevent default asyncio event loop policy modification warning
after @code{test_asyncio} execution.

@item 
bpo-46678@footnote{https://bugs.python.org/issue46678}: The function @code{make_legacy_pyc} in
@code{Lib/test/support/import_helper.py} no longer fails when
@code{PYTHONPYCACHEPREFIX} is set to a directory on a different device from
where tempfiles are stored.

@item 
bpo-46623@footnote{https://bugs.python.org/issue46623}: Skip test_pair() and test_speech128() of test_zlib on s390x
since they fail if zlib uses the s390x hardware accelerator. Patch by
Victor Stinner.
@end itemize

@node Build,Windows,Tests,Python next
@anchor{whatsnew/changelog build}@anchor{109b}
@subsubsection Build



@itemize -

@item 
bpo-46860@footnote{https://bugs.python.org/issue46860}: Respect @code{--with-suffix} when building on case-insensitive file
systems.

@item 
bpo-46656@footnote{https://bugs.python.org/issue46656}: Building Python now requires a C11 compiler without optional
C11 features. Patch by Victor Stinner.

@item 
bpo-46656@footnote{https://bugs.python.org/issue46656}: Building Python now requires support for floating point
Not-a-Number (NaN): remove the @code{Py_NO_NAN} macro. Patch by by Victor
Stinner.

@item 
bpo-46640@footnote{https://bugs.python.org/issue46640}: Building Python now requires a C99 @code{<math.h>} header file
providing a @code{NAN} constant, or the @code{__builtin_nan()} built-in
function. Patch by Victor Stinner.

@item 
bpo-46608@footnote{https://bugs.python.org/issue46608}: Exclude marshalled-frozen data if deep-freezing to save 300 KB
disk space. This includes adding a new @code{is_package} field to
@ref{1d3,,_frozen}. Patch by Kumar Aditya.

@item 
bpo-40280@footnote{https://bugs.python.org/issue40280}: Fix wasm32-emscripten test failures and platform issues. -
Disable syscalls that are not supported or don’t work, e.g.   wait,
getrusage, prlimit, mkfifo, mknod, setres[gu]id, setgroups. - Use fd_count
to cound open fds. - Add more checks for subprocess and fork. - Add
workarounds for missing _multiprocessing and failing socket.accept(). -
Enable bzip2. - Disable large file support. - Disable signal.alarm.

@item 
bpo-46430@footnote{https://bugs.python.org/issue46430}: Intern strings in deep-frozen modules. Patch by Kumar Aditya.
@end itemize

@node Windows,IDLE<3>,Build,Python next
@anchor{whatsnew/changelog windows}@anchor{109c}
@subsubsection Windows



@itemize -

@item 
bpo-46744@footnote{https://bugs.python.org/issue46744}: The default all users install directory for ARM64 is now under
the native @code{Program Files} folder, rather than @code{Program Files (Arm)}
which is intended for ARM (32-bit) files.

@item 
bpo-46567@footnote{https://bugs.python.org/issue46567}: Adds Tcl and Tk support for Windows ARM64. This also adds IDLE
to the installation.

@item 
bpo-46638@footnote{https://bugs.python.org/issue46638}: Ensures registry virtualization is consistently disabled. For
3.10 and earlier, it remains enabled (some registry writes are protected),
while for 3.11 and later it is disabled (registry modifications affect all
applications).
@end itemize

@node IDLE<3>,C API,Windows,Python next
@anchor{whatsnew/changelog idle}@anchor{109d}
@subsubsection IDLE



@itemize -

@item 
bpo-46630@footnote{https://bugs.python.org/issue46630}: Make query dialogs on Windows start with a cursor in the entry
box.

@item 
bpo-45447@footnote{https://bugs.python.org/issue45447}: Apply IDLE syntax highlighting to @code{pyi} files. Patch by Alex
Waygood and Terry Jan Reedy.
@end itemize

@node C API,,IDLE<3>,Python next
@anchor{whatsnew/changelog c-api}@anchor{109e}
@subsubsection C API



@itemize -

@item 
bpo-46748@footnote{https://bugs.python.org/issue46748}: Python’s public headers no longer import @code{<stdbool.h>},
leaving code that embedd/extends Python free to define @code{bool}, @code{true}
and @code{false}.

@item 
bpo-46836@footnote{https://bugs.python.org/issue46836}: Move the @ref{1f6,,PyFrameObject} type definition (@code{struct
_frame}) to the internal C API @code{pycore_frame.h} header file. Patch by
Victor Stinner.

@item 
bpo-45459@footnote{https://bugs.python.org/issue45459}: Rename @code{Include/buffer.h} header file to
@code{Include/pybuffer.h} to avoid conflits with projects having an existing
@code{buffer.h} header file. Patch by Victor Stinner.

@item 
bpo-45412@footnote{https://bugs.python.org/issue45412}: Remove the @code{HAVE_PY_SET_53BIT_PRECISION} macro (moved to the
internal C API). Patch by Victor Stinner.

@item 
bpo-46613@footnote{https://bugs.python.org/issue46613}: Added function @ref{1e9,,PyType_GetModuleByDef()}, which allows
accesss to module state when a method’s defining class is not available.
@end itemize

@node Python 3 11 0 alpha 5,Python 3 11 0 alpha 4,Python next,Changelog
@anchor{whatsnew/changelog python-3-11-0-alpha-5}@anchor{109f}
@subsection Python 3.11.0 alpha 5


@emph{Release date: 2022-02-03}

@menu
* Core and Builtins: Core and Builtins<2>. 
* Library: Library<2>. 
* Documentation: Documentation<3>. 
* Tests: Tests<2>. 
* Build: Build<2>. 
* Windows: Windows<2>. 
* macOS:: 
* IDLE: IDLE<4>. 
* C API: C API<2>. 

@end menu

@node Core and Builtins<2>,Library<2>,,Python 3 11 0 alpha 5
@anchor{whatsnew/changelog id2}@anchor{10a0}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-45773@footnote{https://bugs.python.org/issue45773}: Remove two invalid “peephole” optimizations from the bytecode
compiler.

@item 
bpo-46564@footnote{https://bugs.python.org/issue46564}: Do not create frame objects when creating @ref{78c,,super}
object. Patch by Kumar Aditya.

@item 
bpo-45885@footnote{https://bugs.python.org/issue45885}: Added more fined-grained specialization failure stats regarding
the @code{COMPARE_OP} bytecode.

@item 
bpo-44977@footnote{https://bugs.python.org/issue44977}: The delegation of @ref{1b1,,int()} to @code{__trunc__()} is now
deprecated. Calling @code{int(a)} when @code{type(a)} implements
@code{__trunc__()} but not @code{__int__()} or @code{__index__()} now raises
a @ref{1b2,,DeprecationWarning}.

@item 
bpo-46458@footnote{https://bugs.python.org/issue46458}: Reorder code emitted by the compiler for a
@ref{f6d,,try}-@ref{16e,,except} block so that the @ref{2fa,,else} block’s
code immediately follows the @ref{f6d,,try} body (without a jump). This is
more optimal for the happy path.

@item 
bpo-46527@footnote{https://bugs.python.org/issue46527}: Allow passing @code{iterable} as a keyword argument to
@ref{fdc,,enumerate()} again. Patch by Jelle Zijlstra.

@item 
bpo-46528@footnote{https://bugs.python.org/issue46528}: Replace several stack manipulation instructions (@code{DUP_TOP},
@code{DUP_TOP_TWO}, @code{ROT_TWO}, @code{ROT_THREE}, @code{ROT_FOUR}, and @code{ROT_N})
with new @ref{1a5,,COPY} and @ref{1a6,,SWAP} instructions.

@item 
bpo-46329@footnote{https://bugs.python.org/issue46329}: Use two or three bytecodes to implement most calls.

Calls without named arguments are implemented as a sequence of two
instructions: @code{PRECALL; CALL}. Calls with named arguments are
implemented as a sequence of three instructions: @code{PRECALL; KW_NAMES;
CALL}. There are two different @code{PRECALL} instructions:
@code{PRECALL_FUNTION} and @code{PRECALL_METHOD}. The latter pairs with
@code{LOAD_METHOD}.

This partition into pre-call and call allows better specialization, and
thus better performance ultimately.

There is no change in semantics.

@item 
bpo-46503@footnote{https://bugs.python.org/issue46503}: Fix an assert when parsing some invalid N escape sequences in
f-strings.

@item 
bpo-46431@footnote{https://bugs.python.org/issue46431}: Improve error message on invalid calls to
@code{BaseExceptionGroup.__new__()}.

@item 
bpo-46476@footnote{https://bugs.python.org/issue46476}: Fix memory leak in code objects generated by deepfreeze. Patch
by Kumar Aditya.

@item 
bpo-46481@footnote{https://bugs.python.org/issue46481}: Speed up calls to @code{weakref.ref.__call__()} by using the
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590} @code{vectorcall} calling convention. Patch by Dong-hee Na.

@item 
bpo-46417@footnote{https://bugs.python.org/issue46417}: Fix a race condition on setting a type @code{__bases__} attribute:
the internal function @code{add_subclass()} now gets the
@code{PyTypeObject.tp_subclasses} member after calling
@ref{10a1,,PyWeakref_NewRef()} which can trigger a garbage collection which
can indirectly modify @code{PyTypeObject.tp_subclasses}. Patch by Victor
Stinner.

@item 
bpo-46417@footnote{https://bugs.python.org/issue46417}: @code{python -X showrefcount} now shows the total reference count
after clearing and destroyed the main Python interpreter. Previously, it
was shown before. Patch by Victor Stinner.

@item 
bpo-43683@footnote{https://bugs.python.org/issue43683}: Add ASYNC_GEN_WRAP opcode to wrap the value to be yielded in
async generators. Removes the need to special case async generators in the
@code{YIELD_VALUE} instruction.

@item 
bpo-46407@footnote{https://bugs.python.org/issue46407}: Optimize some modulo operations in @code{Objects/longobject.c}.
Patch by Jeremiah Vivian.

@item 
bpo-46409@footnote{https://bugs.python.org/issue46409}: Add new @code{RETURN_GENERATOR} bytecode to make generators.
Simplifies calling Python functions in the VM, as they no longer any need
to special case generator functions.

Also add @code{JUMP_NO_INTERRUPT} bytecode that acts like @code{JUMP_ABSOLUTE},
but does not check for interrupts.

@item 
bpo-46406@footnote{https://bugs.python.org/issue46406}: The integer division @code{//} implementation has been optimized
to better let the compiler understand its constraints. It can be 20%
faster on the amd64 platform when dividing an int by a value smaller than
@code{2**30}.

@item 
bpo-46383@footnote{https://bugs.python.org/issue46383}: Fix invalid signature of @code{_zoneinfo}’s @code{module_free}
function to resolve a crash on wasm32-emscripten platform.

@item 
bpo-46361@footnote{https://bugs.python.org/issue46361}: Ensure that “small” integers created by @ref{10a2,,int.from_bytes()}
and @ref{248,,decimal.Decimal} are properly cached.

@item 
bpo-46161@footnote{https://bugs.python.org/issue46161}: Fix the class building error when the arguments are constants
and CALL_FUNCTION_EX is used.

@item 
bpo-46028@footnote{https://bugs.python.org/issue46028}: Fixes calculation of @code{sys._base_executable} when inside a
virtual environment that uses symlinks with different binary names than
the base environment provides.

@item 
bpo-46091@footnote{https://bugs.python.org/issue46091}: Correctly calculate indentation levels for lines with
whitespace character that are ended by line continuation characters. Patch
by Pablo Galindo

@item 
bpo-30512@footnote{https://bugs.python.org/issue30512}: Add CAN Socket support for NetBSD.

@item 
bpo-46045@footnote{https://bugs.python.org/issue46045}: Do not use POSIX semaphores on NetBSD

@item 
bpo-44024@footnote{https://bugs.python.org/issue44024}: Improve the exc:@ref{162,,TypeError} message for non-string second
arguments passed to the built-in functions @ref{708,,getattr()} and
@ref{707,,hasattr()}. Patch by Géry Ogam.
@end itemize

@node Library<2>,Documentation<3>,Core and Builtins<2>,Python 3 11 0 alpha 5
@anchor{whatsnew/changelog id3}@anchor{10a3}
@subsubsection Library



@itemize -

@item 
bpo-46624@footnote{https://bugs.python.org/issue46624}: Restore support for non-integer arguments of
@ref{305,,random.randrange()} and @ref{ddb,,random.randint()}.

@item 
bpo-46591@footnote{https://bugs.python.org/issue46591}: Make the IDLE doc URL on the About IDLE dialog clickable.

@item 
bpo-46565@footnote{https://bugs.python.org/issue46565}: Remove loop variables that are leaking into modules’
namespaces.

@item 
bpo-46553@footnote{https://bugs.python.org/issue46553}: In @ref{432,,typing.get_type_hints()}, support evaluating bare
stringified @code{ClassVar} annotations. Patch by Gregory Beauregard.

@item 
bpo-46544@footnote{https://bugs.python.org/issue46544}: Don’t leak @code{x} & @code{uspace} intermediate vars in
@ref{b52,,textwrap.TextWrapper}.

@item 
bpo-46487@footnote{https://bugs.python.org/issue46487}: Add the @code{get_write_buffer_limits} method to
@code{asyncio.transports.WriteTransport} and to the SSL transport.

@item 
bpo-45173@footnote{https://bugs.python.org/issue45173}: Note the configparser deprecations will be removed in Python
3.12.

@item 
bpo-45162@footnote{https://bugs.python.org/issue45162}: The deprecated @ref{122,,unittest} APIs removed in 3.11a1 have been
temporarily restored to be removed in 3.12 while cleanups in external
projects go in.

@item 
bpo-46539@footnote{https://bugs.python.org/issue46539}: In @ref{432,,typing.get_type_hints()}, support evaluating
stringified @code{ClassVar} and @code{Final} annotations inside @code{Annotated}.
Patch by Gregory Beauregard.

@item 
bpo-46510@footnote{https://bugs.python.org/issue46510}: Add missing test for @ref{60f,,types.TracebackType} and
@ref{10a4,,types.FrameType}. Calculate them directly from the caught
exception without calling @ref{18f,,sys.exc_info()}.

@item 
bpo-46491@footnote{https://bugs.python.org/issue46491}: Allow @ref{431,,typing.Annotated} to wrap @ref{556,,typing.Final} and
@ref{83e,,typing.ClassVar}. Patch by Gregory Beauregard.

@item 
bpo-46483@footnote{https://bugs.python.org/issue46483}: Remove @ref{10a5,,__class_getitem__()} from
@ref{1c4,,pathlib.PurePath} as this class was not supposed to be generic.

@item 
bpo-46436@footnote{https://bugs.python.org/issue46436}: Fix command-line option @code{-d}/@code{--directory} in module
@ref{92,,http.server} which is ignored when combined with command-line option
@code{--cgi}. Patch by Géry Ogam.

@item 
bpo-41403@footnote{https://bugs.python.org/issue41403}: Make @code{mock.patch()} raise a @ref{162,,TypeError} with a
relevant error message on invalid arg. Previously it allowed a cryptic
@ref{163,,AttributeError} to escape.

@item 
bpo-46474@footnote{https://bugs.python.org/issue46474}: In @code{importlib.metadata.EntryPoint.pattern}, avoid potential
REDoS by limiting ambiguity in consecutive whitespace.

@item 
bpo-46474@footnote{https://bugs.python.org/issue46474}: Removed private method from @code{importlib.metadata.Path}. Sync
with importlib_metadata 4.10.0.

@item 
bpo-46470@footnote{https://bugs.python.org/issue46470}: Remove unused branch from @code{typing._remove_dups_flatten}

@item 
bpo-46469@footnote{https://bugs.python.org/issue46469}: @ref{9,,asyncio} generic classes now return
@ref{26a,,types.GenericAlias} in @code{__class_getitem__} instead of the same
class.

@item 
bpo-41906@footnote{https://bugs.python.org/issue41906}: Support passing filter instances in the @code{filters} values of
@code{handlers} and @code{loggers} in the dictionary passed to
@ref{d55,,logging.config.dictConfig()}.

@item 
bpo-46422@footnote{https://bugs.python.org/issue46422}: Use @code{dis.Positions} in @code{dis.Instruction} instead of a
regular @code{tuple}.

@item 
bpo-46434@footnote{https://bugs.python.org/issue46434}: @ref{c4,,pdb} now gracefully handles @code{help} when @code{__doc__}
is missing, for example when run with pregenerated optimized @code{.pyc}
files.

@item 
bpo-43869@footnote{https://bugs.python.org/issue43869}: Python uses the same time Epoch on all platforms. Add an
explicit unit test to ensure that it’s the case. Patch by Victor Stinner.

@item 
bpo-46414@footnote{https://bugs.python.org/issue46414}: Add @ref{10a6,,typing.reveal_type()}. Patch by Jelle Zijlstra.

@item 
bpo-40280@footnote{https://bugs.python.org/issue40280}: @ref{f4,,subprocess} now imports Windows-specific imports when
@code{msvcrt} module is available, and POSIX-specific imports on all other
platforms. This gives a clean exception when @code{_posixsubprocess} is not
available (e.g. Emscripten browser target).

@item 
bpo-40066@footnote{https://bugs.python.org/issue40066}: @code{IntEnum}, @code{IntFlag}, and @code{StrEnum} use the mixed-in type
for their @code{str()} and @code{format()} output.

@item 
bpo-46316@footnote{https://bugs.python.org/issue46316}: Optimize @ref{10a7,,pathlib.Path.iterdir()} by removing an
unnecessary check for special entries.

@item 
bpo-29688@footnote{https://bugs.python.org/issue29688}: Document @ref{10a8,,pathlib.Path.absolute()} (which has always
existed).

@item 
bpo-43012@footnote{https://bugs.python.org/issue43012}: The pathlib module’s obsolete and internal @code{_Accessor} class
has been removed to prepare the terrain for upcoming enhancements to the
module.

@item 
bpo-46258@footnote{https://bugs.python.org/issue46258}: Speed up @ref{50e,,math.isqrt()} for small positive integers by
replacing two division steps with a lookup table.

@item 
bpo-46242@footnote{https://bugs.python.org/issue46242}: Improve error message when creating a new @ref{658,,enum.Enum}
type subclassing an existing @code{Enum} with @code{_member_names_} using
@ref{10a9,,enum.Enum.__call__()}.

@item 
bpo-43118@footnote{https://bugs.python.org/issue43118}: Fix a bug in @ref{1c1,,inspect.signature()} that was causing it to
fail on some subclasses of classes with a @code{__text_signature__}
referencing module globals. Patch by Weipeng Hong.

@item 
bpo-26552@footnote{https://bugs.python.org/issue26552}: Fixed case where failing @ref{318,,asyncio.ensure_future()} did not
close the coroutine. Patch by Kumar Aditya.

@item 
bpo-21987@footnote{https://bugs.python.org/issue21987}: Fix an issue with @ref{10aa,,tarfile.TarFile.getmember()} getting a
directory name with a trailing slash.

@item 
bpo-46124@footnote{https://bugs.python.org/issue46124}: Update @ref{14c,,zoneinfo} to rely on importlib.resources
traversable API.

@item 
bpo-46103@footnote{https://bugs.python.org/issue46103}: Now @ref{10ab,,inspect.getmembers()} only gets @code{__bases__}
attribute from class type. Patch by Weipeng Hong.

@item 
bpo-46080@footnote{https://bugs.python.org/issue46080}: Fix exception in argparse help text generation if a
@code{argparse.BooleanOptionalAction} argument’s default is
@code{argparse.SUPPRESS} and it has @code{help} specified.  Patch by Felix
Fontein.

@item 
bpo-44791@footnote{https://bugs.python.org/issue44791}: Fix substitution of @ref{23a,,ParamSpec} in
@ref{23b,,Concatenate} with different parameter expressions.
Substitution with a list of types returns now a tuple of types.
Substitution with @code{Concatenate} returns now a @code{Concatenate} with
concatenated lists of arguments.
@end itemize

@node Documentation<3>,Tests<2>,Library<2>,Python 3 11 0 alpha 5
@anchor{whatsnew/changelog id4}@anchor{10ac}
@subsubsection Documentation



@itemize -

@item 
bpo-46463@footnote{https://bugs.python.org/issue46463}: Fixes @code{escape4chm.py} script used when building the CHM
documentation file
@end itemize

@node Tests<2>,Build<2>,Documentation<3>,Python 3 11 0 alpha 5
@anchor{whatsnew/changelog id5}@anchor{10ad}
@subsubsection Tests



@itemize -

@item 
bpo-43478@footnote{https://bugs.python.org/issue43478}: Mocks can no longer be provided as the specs for other Mocks.
As a result, an already-mocked object cannot be passed to @code{mock.Mock()}.
This can uncover bugs in tests since these Mock-derived Mocks will always
pass certain tests (e.g. isinstance) and builtin assert functions (e.g.
assert_called_once_with) will unconditionally pass.

@item 
bpo-46616@footnote{https://bugs.python.org/issue46616}: Ensures @code{test_importlib.test_windows} cleans up registry keys
after completion.

@item 
bpo-44359@footnote{https://bugs.python.org/issue44359}: test_ftplib now silently ignores socket errors to prevent
logging unhandled threading exceptions. Patch by Victor Stinner.

@item 
bpo-46600@footnote{https://bugs.python.org/issue46600}: Fix test_gdb.test_pycfunction() for Python built with @code{clang
-Og}. Tolerate inlined functions in the gdb traceback. Patch by Victor
Stinner.

@item 
bpo-46542@footnote{https://bugs.python.org/issue46542}: Fix a Python crash in test_lib2to3 when using Python built in
debug mode: limit the recursion limit. Patch by Victor Stinner.

@item 
bpo-46576@footnote{https://bugs.python.org/issue46576}: test_peg_generator now disables compiler optimization when
testing compilation of its own C extensions to significantly speed up the
testing on non-debug builds of CPython.

@item 
bpo-46542@footnote{https://bugs.python.org/issue46542}: Fix @code{test_json} tests checking for @ref{8c4,,RecursionError}:
modify these tests to use @code{support.infinite_recursion()}. Patch by
Victor Stinner.

@item 
bpo-13886@footnote{https://bugs.python.org/issue13886}: Skip test_builtin PTY tests on non-ASCII characters if the
readline module is loaded. The readline module changes input() behavior,
but test_builtin is not intented to test the readline module. Patch by
Victor Stinner.

@item 
bpo-40280@footnote{https://bugs.python.org/issue40280}: Add @code{test.support.requires_fork()} decorators to mark tests
that require a working @ref{bb9,,os.fork()}.

@item 
bpo-40280@footnote{https://bugs.python.org/issue40280}: Add @code{test.support.requires_subprocess()} decorator to mark
tests which require working @ref{f4,,subprocess} module or @code{os.spawn*}. The
wasm32-emscripten platform has no support for processes.

@item 
bpo-46126@footnote{https://bugs.python.org/issue46126}: Disable ‘descriptions’ when running tests internally.
@end itemize

@node Build<2>,Windows<2>,Tests<2>,Python 3 11 0 alpha 5
@anchor{whatsnew/changelog id6}@anchor{10ae}
@subsubsection Build



@itemize -

@item 
bpo-46602@footnote{https://bugs.python.org/issue46602}: Tidied up configure.ac so that conftest.c is truncated rather
than appended. This assists in the case where the ‘rm’ of conftest.c fails
to happen between tests.  Downstream issues such as a clobbered SOABI can
result.

@item 
bpo-46600@footnote{https://bugs.python.org/issue46600}: Fix the test checking if the C compiler supports @code{-Og} option
in the @code{./configure} script to also use @code{-Og} on clang which supports
it. Patch by Victor Stinner.

@item 
bpo-38472@footnote{https://bugs.python.org/issue38472}: Fix GCC detection in setup.py when cross-compiling. The C
compiler is now run with LC_ALL=C. Previously, the detection failed with a
German locale.

@item 
bpo-46513@footnote{https://bugs.python.org/issue46513}: @code{configure} no longer uses @code{AC_C_CHAR_UNSIGNED}
macro and @code{pyconfig.h} no longer defines reserved symbol
@code{__CHAR_UNSIGNED__}.

@item 
bpo-46471@footnote{https://bugs.python.org/issue46471}: Use global singletons for single byte bytes objects in
deepfreeze.

@item 
bpo-46443@footnote{https://bugs.python.org/issue46443}: Deepfreeze now uses cached small integers as it saves some
space for common small integers.

@item 
bpo-46429@footnote{https://bugs.python.org/issue46429}: Merge all deep-frozen files into one for space savings. Patch
by Kumar Aditya.

@item 
bpo-45569@footnote{https://bugs.python.org/issue45569}: The build now defaults to using 30-bit digits for Python
integers. Previously either 15-bit or 30-bit digits would be selected,
depending on the platform. 15-bit digits may still be selected using the
@code{--enable-big-digits=15} option to the @code{configure} script, or by
defining @code{PYLONG_BITS_IN_DIGIT} in @code{pyconfig.h}.

@item 
bpo-45925@footnote{https://bugs.python.org/issue45925}: Update Windows installer to use SQLite 3.37.2.

@item 
bpo-43112@footnote{https://bugs.python.org/issue43112}: Detect musl libc as a separate SOABI (tagged as
@code{linux-musl}).
@end itemize

@node Windows<2>,macOS,Build<2>,Python 3 11 0 alpha 5
@anchor{whatsnew/changelog id7}@anchor{10af}
@subsubsection Windows



@itemize -

@item 
bpo-33125@footnote{https://bugs.python.org/issue33125}: The traditional EXE/MSI based installer for Windows is now
available for ARM64

@item 
bpo-46362@footnote{https://bugs.python.org/issue46362}: os.path.abspath(“C:CON”) is now fixed to return “\.CON”, not
the same path. The regression was true of all legacy DOS devices such as
COM1, LPT1, or NUL.

@item 
bpo-44934@footnote{https://bugs.python.org/issue44934}: The installer now offers a command-line only option to add the
installation directory to the end of 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} instead of at the
start.
@end itemize

@node macOS,IDLE<4>,Windows<2>,Python 3 11 0 alpha 5
@anchor{whatsnew/changelog macos}@anchor{10b0}
@subsubsection macOS



@itemize -

@item 
bpo-45925@footnote{https://bugs.python.org/issue45925}: Update macOS installer to SQLite 3.37.2.
@end itemize

@node IDLE<4>,C API<2>,macOS,Python 3 11 0 alpha 5
@anchor{whatsnew/changelog id8}@anchor{10b1}
@subsubsection IDLE



@itemize -

@item 
bpo-45296@footnote{https://bugs.python.org/issue45296}: Clarify close, quit, and exit in IDLE.  In the File menu,
‘Close’ and ‘Exit’ are now ‘Close Window’ (the current one) and ‘Exit’ is
now ‘Exit IDLE’ (by closing all windows).  In Shell, ‘quit()’ and ‘exit()’
mean ‘close Shell’. If there are no other windows, this also exits IDLE.
@end itemize

@node C API<2>,,IDLE<4>,Python 3 11 0 alpha 5
@anchor{whatsnew/changelog id9}@anchor{10b2}
@subsubsection C API



@itemize -

@item 
bpo-40170@footnote{https://bugs.python.org/issue40170}: Remove the @code{PyHeapType_GET_MEMBERS()} macro. It was exposed
in the public C API by mistake, it must only be used by Python internally.
Use the @code{PyTypeObject.tp_members} member instead. Patch by Victor
Stinner.

@item 
bpo-40170@footnote{https://bugs.python.org/issue40170}: Move _Py_GetAllocatedBlocks() and _PyObject_DebugMallocStats()
private functions to the internal C API. Patch by Victor Stinner.

@item 
bpo-46433@footnote{https://bugs.python.org/issue46433}: The internal function _PyType_GetModuleByDef now correctly
handles inheritance patterns involving static types.

@item 
bpo-45459@footnote{https://bugs.python.org/issue45459}: @ref{1db,,Py_buffer} and various @code{Py_buffer} related functions
are now part of the limited API and stable ABI.

@item 
bpo-14916@footnote{https://bugs.python.org/issue14916}: Fixed bug in the tokenizer that prevented
@code{PyRun_InteractiveOne} from parsing from the provided FD.
@end itemize

@node Python 3 11 0 alpha 4,Python 3 11 0 alpha 3,Python 3 11 0 alpha 5,Changelog
@anchor{whatsnew/changelog python-3-11-0-alpha-4}@anchor{10b3}
@subsection Python 3.11.0 alpha 4


@emph{Release date: 2022-01-13}

@menu
* Core and Builtins: Core and Builtins<3>. 
* Library: Library<3>. 
* Documentation: Documentation<4>. 
* Tests: Tests<3>. 
* Build: Build<3>. 
* Windows: Windows<3>. 
* macOS: macOS<2>. 
* C API: C API<3>. 

@end menu

@node Core and Builtins<3>,Library<3>,,Python 3 11 0 alpha 4
@anchor{whatsnew/changelog id10}@anchor{10b4}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-46070@footnote{https://bugs.python.org/issue46070}: @ref{10b5,,Py_EndInterpreter()} now explicitly untracks all objects
currently tracked by the GC. Previously, if an object was used later by
another interpreter, calling @ref{103b,,PyObject_GC_UnTrack()} on the object
crashed if the previous or the next object of the @code{PyGC_Head}
structure became a dangling pointer. Patch by Victor Stinner.

@item 
bpo-46347@footnote{https://bugs.python.org/issue46347}: Fix memory leak in PyEval_EvalCodeEx.

@item 
bpo-46339@footnote{https://bugs.python.org/issue46339}: Fix a crash in the parser when retrieving the error text for
multi-line f-strings expressions that do not start in the first line of
the string. Patch by Pablo Galindo

@item 
bpo-46331@footnote{https://bugs.python.org/issue46331}: Do not set line number of instruction storing doc-string. Fixes
regression introduced in 3.11 alpha.

@item 
bpo-46314@footnote{https://bugs.python.org/issue46314}: Remove spurious “call” event when creating a lambda function
that was accidentally introduced in 3.11a4.

@item 
bpo-46289@footnote{https://bugs.python.org/issue46289}: ASDL declaration of @code{FormattedValue} has changed to reflect
@code{conversion} field is not optional.

@item 
bpo-46297@footnote{https://bugs.python.org/issue46297}: Fixed an interpreter crash on bootup with multiple PythonPaths
set in the Windows registry. Patch by Derzsi Dániel.

@item 
bpo-46237@footnote{https://bugs.python.org/issue46237}: Fix the line number of tokenizer errors inside f-strings. Patch
by Pablo Galindo.

@item 
bpo-46263@footnote{https://bugs.python.org/issue46263}: We always expect the “use_frozen_modules” config to be set, now
that getpath.c was rewritten in pure Python and the logic improved.

@item 
bpo-46006@footnote{https://bugs.python.org/issue46006}: Fix a regression when a type method like @code{__init__()} is
modified in a subinterpreter. Fix a regression in
@code{_PyUnicode_EqualToASCIIId()} and type @code{update_slot()}. Revert the
change which made the Unicode dictionary of interned strings compatible
with subinterpreters: the internal interned dictionary is shared again by
all interpreters. Patch by Victor Stinner.

@item 
bpo-45923@footnote{https://bugs.python.org/issue45923}: Add RESUME opcode. This is a logical no-op. It is emitted by
the compiler anywhere a Python function can be entered. It is used by the
interpreter to perform tracing and optimizer checks.

@item 
bpo-46208@footnote{https://bugs.python.org/issue46208}: Fix the regression of os.path.normpath(“A/../../B”) not
returning expected “../B” but “B”.

@item 
bpo-46240@footnote{https://bugs.python.org/issue46240}: Correct the error message for unclosed parentheses when the
tokenizer doesn’t reach the end of the source when the error is reported.
Patch by Pablo Galindo

@item 
bpo-46009@footnote{https://bugs.python.org/issue46009}: Remove the @code{GEN_START} opcode.

@item 
bpo-46235@footnote{https://bugs.python.org/issue46235}: Certain sequence multiplication operations like @code{[0] * 1_000}
are now faster due to reference-counting optimizations. Patch by Dennis
Sweeney.

@item 
bpo-46221@footnote{https://bugs.python.org/issue46221}: @ref{10b6,,PREP_RERAISE_STAR} no longer pushes @code{lasti} to the
stack.

@item 
bpo-46202@footnote{https://bugs.python.org/issue46202}: Remove @code{POP_EXCEPT_AND_RERAISE} and replace it by an
equivalent sequence of other opcodes.

@item 
bpo-46085@footnote{https://bugs.python.org/issue46085}: Fix iterator cache mechanism of @code{OrderedDict}.

@item 
bpo-46055@footnote{https://bugs.python.org/issue46055}: Speed up shifting operation involving integers less than
@code{PyLong_BASE}. Patch by Xinhang Xu.

@item 
bpo-46110@footnote{https://bugs.python.org/issue46110}: Add a maximum recursion check to the PEG parser to avoid stack
overflow. Patch by Pablo Galindo

@item 
bpo-46107@footnote{https://bugs.python.org/issue46107}: Fix bug where @code{ExceptionGroup.split()} and
@code{ExceptionGroup.subgroup()} did not copy the exception group’s
@code{__note__} field to the parts.

@item 
bpo-45711@footnote{https://bugs.python.org/issue45711}: The interpreter state’s representation of handled exceptions
(a.k.a exc_info, or _PyErr_StackItem) now has only the @code{exc_value}
field, @code{exc_type} and @code{exc_traceback} have been removed as their
values can be derived from @code{exc_value}.

@item 
bpo-44525@footnote{https://bugs.python.org/issue44525}: Replace the four call bytecode instructions which one pre-call
instruction and two call instructions.

Removes @code{CALL_FUNCTION}, @code{CALL_FUNCTION_KW}, @code{CALL_METHOD} and
@code{CALL_METHOD_KW}.

Adds @code{CALL_NO_KW} and @code{CALL_KW} call instructions, and
@code{PRECALL_METHOD} prefix for pairing with @code{LOAD_METHOD}.

@item 
bpo-46039@footnote{https://bugs.python.org/issue46039}: Remove the @code{YIELD_FROM} instruction and replace it with the
@code{SEND} instruction which performs the same operation, but without the
loop.

@item 
bpo-45635@footnote{https://bugs.python.org/issue45635}: The code called from @code{_PyErr_Display()} was refactored to
improve error handling. It now exits immediately upon an unrecoverable
error.

@item 
bpo-46054@footnote{https://bugs.python.org/issue46054}: Fix parser error when parsing non-utf8 characters in source
files. Patch by Pablo Galindo.

@item 
bpo-46042@footnote{https://bugs.python.org/issue46042}: Improve the location of the caret in @ref{21a,,SyntaxError}
exceptions emitted by the symbol table. Patch by Pablo Galindo.

@item 
bpo-46049@footnote{https://bugs.python.org/issue46049}: Ensure @code{._pth} files work as intended on platforms other
than Windows.

@item 
bpo-46048@footnote{https://bugs.python.org/issue46048}: Fixes parsing of @code{._pth} files on startup so that
single-character paths are correctly read.

@item 
bpo-37971@footnote{https://bugs.python.org/issue37971}: Fix a bug where the line numbers given in a traceback when a
decorator application raised an exception were wrong.

@item 
bpo-46031@footnote{https://bugs.python.org/issue46031}: Add @ref{1a7,,POP_JUMP_IF_NOT_NONE} and
@ref{1a8,,POP_JUMP_IF_NONE} opcodes to speed up conditional jumps.

@item 
bpo-45654@footnote{https://bugs.python.org/issue45654}: Deepfreeze @ref{dd,,runpy}, patch by Kumar Aditya.

@item 
bpo-46025@footnote{https://bugs.python.org/issue46025}: Fix a crash in the @ref{b,,atexit} module involving functions
that unregister themselves before raising exceptions. Patch by Pablo
Galindo.

@item 
bpo-46000@footnote{https://bugs.python.org/issue46000}: Improve compatibility of the @ref{2a,,curses} module with NetBSD
curses.

@item 
bpo-44525@footnote{https://bugs.python.org/issue44525}: Specialize the CALL_FUNCTION instruction for calls to builtin
types with a single argument. Speeds up @code{range(x)}, @code{list(x)}, and
specifically @code{type(obj)}.

@item 
bpo-42918@footnote{https://bugs.python.org/issue42918}: Fix bug where the built-in @ref{4e2,,compile()} function did not
always raise a @ref{21a,,SyntaxError} when passed multiple statements in
‘single’ mode. Patch by Weipeng Hong.

@item 
bpo-45953@footnote{https://bugs.python.org/issue45953}: The main interpreter in _PyRuntimeState.interpreters is now
statically allocated (as part of _PyRuntime).  Likewise for the initial
thread state of each interpreter.  This means less allocation during
runtime init, as well as better memory locality for these key state
objects.

@item 
bpo-45292@footnote{https://bugs.python.org/issue45292}: Complete the 
@geindex Python Enhancement Proposals; PEP 654
PEP 654@footnote{https://www.python.org/dev/peps/pep-0654} implementation: add @code{except*}.

@item 
bpo-43413@footnote{https://bugs.python.org/issue43413}: Revert changes in @code{set.__init__}. Subclass of @ref{370,,set}
needs to define a @code{__init__()} method if it defines a @code{__new__()}
method with additional keyword parameters.

@item 
bpo-43931@footnote{https://bugs.python.org/issue43931}: Added the @ref{1d9,,Py_Version} constant which bears the same
value as @ref{1da,,PY_VERSION_HEX}. Patch by Gabriele N. Tornetta.
@end itemize

@node Library<3>,Documentation<4>,Core and Builtins<3>,Python 3 11 0 alpha 4
@anchor{whatsnew/changelog id11}@anchor{10b7}
@subsubsection Library



@itemize -

@item 
bpo-46342@footnote{https://bugs.python.org/issue46342}: The @code{@@typing.final} decorator now sets the @code{__final__}
attribute on the decorated object to allow runtime introspection. Patch by
Jelle Zijlstra.

@item 
bpo-46328@footnote{https://bugs.python.org/issue46328}: Added the @ref{190,,sys.exception()} method which returns the active
exception instance.

@item 
bpo-46307@footnote{https://bugs.python.org/issue46307}: Add @ref{10b8,,string.Template.is_valid()} and
@ref{10b9,,string.Template.get_identifiers()} methods.

@item 
bpo-46306@footnote{https://bugs.python.org/issue46306}: Assume that @ref{4cb,,types.CodeType} always has
@code{types.CodeType.co_firstlineno} in @ref{63,,doctest}.

@item 
bpo-40479@footnote{https://bugs.python.org/issue40479}: Fix @ref{88,,hashlib} @emph{usedforsecurity} option to work correctly
with OpenSSL 3.0.0 in FIPS mode.

@item 
bpo-46070@footnote{https://bugs.python.org/issue46070}: Fix possible segfault when importing the @ref{9,,asyncio} module
from different sub-interpreters in parallel. Patch by Erlend E. Aasland.

@item 
bpo-46244@footnote{https://bugs.python.org/issue46244}: Removed @code{__slots__} from @ref{23a,,typing.ParamSpec} and
@ref{10ba,,typing.TypeVar}. They served no purpose. Patch by Arie Bovenberg.

@item 
bpo-46278@footnote{https://bugs.python.org/issue46278}: Reflect @code{context} argument in @code{AbstractEventLoop.call_*()}
methods. Loop implementations already support it.

@item 
bpo-46269@footnote{https://bugs.python.org/issue46269}: Remove special-casing of @code{__new__} in
@ref{10bb,,enum.Enum.__dir__()}.

@item 
bpo-46266@footnote{https://bugs.python.org/issue46266}: Improve day constants in @ref{13,,calendar}.

Now all constants (@code{MONDAY} … @code{SUNDAY}) are documented, tested, and
added to @code{__all__}.

@item 
bpo-46257@footnote{https://bugs.python.org/issue46257}: Optimized the mean, variance, and stdev functions in the
statistics module. If the input is an iterator, it is consumed in a single
pass rather than eating memory by conversion to a list.  The single pass
algorithm is about twice as fast as the previous two pass code.

@item 
bpo-41011@footnote{https://bugs.python.org/issue41011}: Added two new variables to @emph{pyvenv.cfg} which is generated by
@ref{12c,,venv} module: @emph{executable} for the executable and @emph{command} for the
command line used to create the environment.

@item 
bpo-46239@footnote{https://bugs.python.org/issue46239}: Improve error message when importing
@code{asyncio.windows_events} on non-Windows.

@item 
bpo-46238@footnote{https://bugs.python.org/issue46238}: Reuse @code{_winapi} constants in @code{asyncio.windows_events}.

@item 
bpo-46222@footnote{https://bugs.python.org/issue46222}: Adding @code{SF_NOCACHE} sendfile constant for FreeBSD for the
posixmodule.

@item 
bpo-37295@footnote{https://bugs.python.org/issue37295}: Add fast path for @code{0 <= k <= n <= 67} for @ref{19b,,math.comb()}.

@item 
bpo-46176@footnote{https://bugs.python.org/issue46176}: Adding the @code{MAP_STACK} constant for the mmap module.

@item 
bpo-43424@footnote{https://bugs.python.org/issue43424}: Deprecate @code{webbrowser.MacOSXOSAScript._name} and use
@code{name} instead.

@item 
bpo-45321@footnote{https://bugs.python.org/issue45321}: Added missing error codes to module
@code{xml.parsers.expat.errors}.

@item 
bpo-46125@footnote{https://bugs.python.org/issue46125}: Refactor tests to test traversable API directly. Includes
changes from importlib 5.4.0.

@item 
bpo-46118@footnote{https://bugs.python.org/issue46118}: Moved importlib.resources and its related functionality to a
package.

@item 
bpo-37578@footnote{https://bugs.python.org/issue37578}: Add @emph{include_hidden} parameter to @ref{288,,glob()} and
@ref{289,,iglob()} to match hidden files and directories when using
special characters like @code{*}, @code{**}, @code{?} and @code{[]}.

@item 
bpo-20369@footnote{https://bugs.python.org/issue20369}: @ref{10bc,,concurrent.futures.wait()} no longer blocks forever when
given duplicate Futures. Patch by Kumar Aditya.

@item 
bpo-46105@footnote{https://bugs.python.org/issue46105}: Honor spec when generating requirement specs with urls and
extras (importlib_metadata 4.8.3).

@item 
bpo-44893@footnote{https://bugs.python.org/issue44893}: EntryPoint objects are no longer tuples. Recommended means to
access is by attribute (‘.name’, ‘.group’) or accessor (‘.load()’). Access
by index is deprecated and will raise deprecation warning.

@item 
bpo-22815@footnote{https://bugs.python.org/issue22815}: Print unexpected successes together with failures and errors in
summary in @ref{10bd,,unittest.TextTestResult}.

@item 
bpo-22047@footnote{https://bugs.python.org/issue22047}: Calling @code{add_argument_group()} on an argument group is
deprecated. Calling @code{add_argument_group()} or
@code{add_mutually_exclusive_group()} on a mutually exclusive group is
deprecated.

These features were never supported and do not always work correctly. The
functions exist on the API by accident through inheritance and will be
removed in the future.

@item 
bpo-26952@footnote{https://bugs.python.org/issue26952}: @ref{5,,argparse} raises @ref{17c,,ValueError} with clear message
when trying to render usage for an empty mutually-exclusive group.
Previously it raised a cryptic @ref{1066,,IndexError}.

@item 
bpo-45615@footnote{https://bugs.python.org/issue45615}: Functions in the @ref{11a,,traceback} module raise @ref{162,,TypeError}
rather than @ref{163,,AttributeError} when an exception argument is not of
type @ref{15f,,BaseException}.

@item 
bpo-16594@footnote{https://bugs.python.org/issue16594}: Add allow allow_reuse_port flag in socketserver.

@item 
bpo-27718@footnote{https://bugs.python.org/issue27718}: Fix help for the @ref{e5,,signal} module. Some functions (e.g.
@code{signal()} and @code{getsignal()}) were omitted.

@item 
bpo-46032@footnote{https://bugs.python.org/issue46032}: The @code{registry()} method of @ref{65b,,functools.singledispatch()}
functions checks now the first argument or the first parameter annotation
and raises a TypeError if it is not supported. Previously unsupported
“types” were ignored (e.g. @code{typing.List[int]}) or caused an error at
calling time (e.g. @code{list[int]}).

@item 
bpo-46014@footnote{https://bugs.python.org/issue46014}: Add ability to use @code{typing.Union} and @code{types.UnionType} as
dispatch argument to @code{functools.singledispatch}. Patch provided by Yurii
Karabas.

@item 
bpo-27062@footnote{https://bugs.python.org/issue27062}: Add @code{__all__} to @ref{9c,,inspect}, patch by Kumar Aditya.

@item 
bpo-46018@footnote{https://bugs.python.org/issue46018}: Ensure that @ref{d8e,,math.expm1()} does not raise on underflow.

@item 
bpo-46016@footnote{https://bugs.python.org/issue46016}: Adding @code{F_DUP2FD} and @code{F_DUP2FD_CLOEXEC} constants
from FreeBSD into the fcntl module.

@item 
bpo-45755@footnote{https://bugs.python.org/issue45755}: @ref{120,,typing} generic aliases now reveal the class attributes
of the original generic class when passed to @code{dir()}. This was the
behavior up to Python 3.6, but was changed in 3.7-3.9.

@item 
bpo-45874@footnote{https://bugs.python.org/issue45874}: The empty query string, consisting of no query arguments, is
now handled correctly in @code{urllib.parse.parse_qsl}. This caused problems
before when strict parsing was enabled.

@item 
bpo-44674@footnote{https://bugs.python.org/issue44674}: Change how dataclasses disallows mutable default values.  It
used to use a list of known types (list, dict, set).  Now it disallows
unhashable objects to be defaults.  It’s using unhashability as a proxy
for mutability.  Patch by Eric V. Smith, idea by Raymond Hettinger.

@item 
bpo-23882@footnote{https://bugs.python.org/issue23882}: Remove namespace package (PEP 420) support from unittest
discovery. It was introduced in Python 3.4 but has been broken since
Python 3.7.

@item 
bpo-25066@footnote{https://bugs.python.org/issue25066}: Added a @code{__repr__()} method to
@ref{10be,,multiprocessing.Event} objects, patch by Kumar Aditya.

@item 
bpo-45643@footnote{https://bugs.python.org/issue45643}: Added @ref{10bf,,signal.SIGSTKFLT} on platforms where this signal
is defined.

@item 
bpo-44092@footnote{https://bugs.python.org/issue44092}: Fetch across rollback no longer raises
@code{InterfaceError}. Instead we leave it to the SQLite library
to handle these cases. Patch by Erlend E. Aasland.

@item 
bpo-42413@footnote{https://bugs.python.org/issue42413}: Replace @code{concurrent.futures.TimeoutError} and
@code{asyncio.TimeoutError} with builtin @ref{2b8,,TimeoutError}, keep these
names as deprecated aliases.
@end itemize

@node Documentation<4>,Tests<3>,Library<3>,Python 3 11 0 alpha 4
@anchor{whatsnew/changelog id12}@anchor{10c0}
@subsubsection Documentation



@itemize -

@item 
bpo-46196@footnote{https://bugs.python.org/issue46196}: Document method @ref{10c1,,cmd.Cmd.columnize()}.

@item 
bpo-46120@footnote{https://bugs.python.org/issue46120}: State that @code{|} is preferred for readability over @code{Union} in
the @ref{120,,typing} docs.

@item 
bpo-46109@footnote{https://bugs.python.org/issue46109}: Extracted @code{importlib.resources} and
@code{importlib.resources.abc} documentation into separate files.

@item 
bpo-19737@footnote{https://bugs.python.org/issue19737}: Update the documentation for the @ref{10c2,,globals()} function.
@end itemize

@node Tests<3>,Build<3>,Documentation<4>,Python 3 11 0 alpha 4
@anchor{whatsnew/changelog id13}@anchor{10c3}
@subsubsection Tests



@itemize -

@item 
bpo-46296@footnote{https://bugs.python.org/issue46296}: Add a test case for @ref{76,,enum} with @code{_use_args_ == True} and
@code{_member_type_ == object}.

@item 
bpo-46205@footnote{https://bugs.python.org/issue46205}: Fix hang in runtest_mp due to race condition

@item 
bpo-46263@footnote{https://bugs.python.org/issue46263}: Fix test_capi on FreeBSD 14-dev: instruct jemalloc to not fill
freed memory with junk byte.

@item 
bpo-46262@footnote{https://bugs.python.org/issue46262}: Cover @code{ValueError} path in tests for
@code{enum.Flag._missing_()}.

@item 
bpo-46150@footnote{https://bugs.python.org/issue46150}: Now @code{fakename} in
@code{test_pathlib.PosixPathTest.test_expanduser} is checked to be
non-existent.

@item 
bpo-46129@footnote{https://bugs.python.org/issue46129}: Rewrite @code{asyncio.locks} tests with
@ref{1ac,,unittest.IsolatedAsyncioTestCase} usage.

@item 
bpo-23819@footnote{https://bugs.python.org/issue23819}: Fixed @ref{9,,asyncio} tests in python optimized mode. Patch by
Kumar Aditya.

@item 
bpo-46114@footnote{https://bugs.python.org/issue46114}: Fix test case for OpenSSL 3.0.1 version. OpenSSL 3.0 uses
@code{0xMNN00PP0L}.
@end itemize

@node Build<3>,Windows<3>,Tests<3>,Python 3 11 0 alpha 4
@anchor{whatsnew/changelog id14}@anchor{10c4}
@subsubsection Build



@itemize -

@item 
bpo-44133@footnote{https://bugs.python.org/issue44133}: When Python is configured with
@ref{357,,--without-static-libpython}, the Python static library
(libpython.a) is no longer built. Patch by Victor Stinner.

@item 
bpo-44133@footnote{https://bugs.python.org/issue44133}: When Python is built without @ref{2f3,,--enable-shared}, the
@code{python} program is now linked to object files, rather than being linked
to the Python static library (libpython.a), to make sure that all symbols
are exported. Previously, the linker omitted some symbols like the
@code{Py_FrozenMain()} function. Patch by Victor Stinner.

@item 
bpo-40280@footnote{https://bugs.python.org/issue40280}: The @code{configure} script has a new option
@code{--with-emscripten-target} to select browser or node as Emscripten build
target.

@item 
bpo-46315@footnote{https://bugs.python.org/issue46315}: Added and fixed @code{#ifdef HAVE_FEATURE} checks for
functionality that is not available on WASI platform.

@item 
bpo-45723@footnote{https://bugs.python.org/issue45723}: Fixed a regression in @code{configure} check for
@ref{b14,,select.epoll()}.

@item 
bpo-46263@footnote{https://bugs.python.org/issue46263}: @code{configure} no longer sets @code{MULTIARCH} on FreeBSD
platforms.

@item 
bpo-46106@footnote{https://bugs.python.org/issue46106}: Updated OpenSSL to 1.1.1m in Windows builds, macOS installer
builds, and CI. Patch by Kumar Aditya.

@item 
bpo-46088@footnote{https://bugs.python.org/issue46088}: Automatically detect or install bootstrap Python runtime when
building from Visual Studio.

@item 
bpo-46072@footnote{https://bugs.python.org/issue46072}: Add a –with-pystats configure option to turn on internal
statistics gathering.

@item 
bpo-40280@footnote{https://bugs.python.org/issue40280}: A new directory @code{Tools/wasm} contains WebAssembly-related
helpers like @code{config.site} override for wasm32-emscripten, wasm assets
generator to bundle the stdlib, and a README.

@item 
bpo-46023@footnote{https://bugs.python.org/issue46023}: @code{makesetup} no longer builds extensions that have been
marked as @emph{disabled}. This allows users to disable modules in
@code{Modules/Setup.local}.

@item 
bpo-45949@footnote{https://bugs.python.org/issue45949}: Use pure Python @code{freeze_module} for all but importlib
bootstrap files. @code{--with-freeze-module} @code{configure} option is
no longer needed for cross builds.
@end itemize

@node Windows<3>,macOS<2>,Build<3>,Python 3 11 0 alpha 4
@anchor{whatsnew/changelog id15}@anchor{10c5}
@subsubsection Windows



@itemize -

@item 
bpo-46217@footnote{https://bugs.python.org/issue46217}: Removed parameter that is unsupported on Windows 8.1 and early
Windows 10 and may have caused build or runtime failures.
@end itemize

@node macOS<2>,C API<3>,Windows<3>,Python 3 11 0 alpha 4
@anchor{whatsnew/changelog id16}@anchor{10c6}
@subsubsection macOS



@itemize -

@item 
bpo-40477@footnote{https://bugs.python.org/issue40477}: The Python Launcher app for macOS now properly launches scripts
and, if necessary, the Terminal app when running on recent macOS releases.
@end itemize

@node C API<3>,,macOS<2>,Python 3 11 0 alpha 4
@anchor{whatsnew/changelog id17}@anchor{10c7}
@subsubsection C API



@itemize -

@item 
bpo-46236@footnote{https://bugs.python.org/issue46236}: Fix a bug in @ref{10c8,,PyFunction_GetAnnotations()} that caused it
to return a @code{tuple} instead of a @code{dict}.

@item 
bpo-46140@footnote{https://bugs.python.org/issue46140}: @ref{1de,,PyBuffer_GetPointer()},
@ref{1e1,,PyBuffer_FromContiguous()}, @ref{1e0,,PyBuffer_ToContiguous()} and
@ref{1e6,,PyMemoryView_FromBuffer()} now take buffer info by @code{const
Py_buffer *} instead of @code{Py_buffer *}, as they do not need mutability.
@ref{1e1,,PyBuffer_FromContiguous()} also now takes the source buffer as
@code{const void *}, and similarly @ref{1de,,PyBuffer_GetPointer()} takes the
strides as @code{const Py_ssize_t *}.

@item 
bpo-45855@footnote{https://bugs.python.org/issue45855}: Document that the @emph{no_block} argument to
@ref{10c9,,PyCapsule_Import()} is a no-op now.

@item 
bpo-45855@footnote{https://bugs.python.org/issue45855}: Replaced deprecated usage of
@ref{c11,,PyImport_ImportModuleNoBlock()} with
@ref{34f,,PyImport_ImportModule()} in stdlib modules. Patch by Kumar Aditya.

@item 
bpo-46007@footnote{https://bugs.python.org/issue46007}: The @code{PyUnicode_CHECK_INTERNED()} macro has been excluded
from the limited C API. It was never usable there, because it used
internal structures which are not available in the limited C API. Patch by
Victor Stinner.
@end itemize

@node Python 3 11 0 alpha 3,Python 3 11 0 alpha 2,Python 3 11 0 alpha 4,Changelog
@anchor{whatsnew/changelog python-3-11-0-alpha-3}@anchor{10ca}
@subsection Python 3.11.0 alpha 3


@emph{Release date: 2021-12-08}

@menu
* Core and Builtins: Core and Builtins<4>. 
* Library: Library<4>. 
* Documentation: Documentation<5>. 
* Tests: Tests<4>. 
* Build: Build<4>. 
* Windows: Windows<4>. 
* macOS: macOS<3>. 
* C API: C API<4>. 

@end menu

@node Core and Builtins<4>,Library<4>,,Python 3 11 0 alpha 3
@anchor{whatsnew/changelog id18}@anchor{10cb}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-46009@footnote{https://bugs.python.org/issue46009}: Restore behavior from 3.9 and earlier when sending non-None to
newly started generator. In 3.9 this did not affect the state of the
generator. In 3.10.0 and 3.10.1 @code{gen_func().send(0)} is equivalent to
@code{gen_func().throw(TypeError(...)} which exhausts the generator. In
3.10.2 onward, the behavior has been reverted to that of 3.9.

@item 
bpo-46004@footnote{https://bugs.python.org/issue46004}: Fix the @ref{21a,,SyntaxError} location for errors involving for
loops with invalid targets. Patch by Pablo Galindo

@item 
bpo-45711@footnote{https://bugs.python.org/issue45711}: @code{_PyErr_ChainStackItem()} no longer normalizes
@code{exc_info} (including setting the traceback on the exception instance)
because @code{exc_info} is always normalized.

@item 
bpo-45607@footnote{https://bugs.python.org/issue45607}: The @code{__note__} field was added to @ref{15f,,BaseException}. It is
@code{None} by default but can be set to a string which is added to the
exception’s traceback.

@item 
bpo-45947@footnote{https://bugs.python.org/issue45947}: Place pointers to dict and values immediately before GC header.
This reduces number of dependent memory loads to access either dict or
values from 3 to 1.

@item 
bpo-45915@footnote{https://bugs.python.org/issue45915}: @code{is_valid_fd} now uses faster @code{fcntl(fd, F_GETFD)} on
Linux, macOS, and Windows.

@item 
bpo-44530@footnote{https://bugs.python.org/issue44530}: Reverts a change to the @code{code.__new__} @ref{10cc,,audit event} from an earlier prerelease.

@item 
bpo-42268@footnote{https://bugs.python.org/issue42268}: Fail the configure step if the selected compiler doesn’t
support memory sanitizer. Patch by Pablo Galindo

@item 
bpo-45711@footnote{https://bugs.python.org/issue45711}: The three values of @code{exc_info} are now always consistent with
each other. In particular, the @code{type} and @code{traceback} fields are now
derived from the exception instance. This impacts the return values of
@ref{18f,,sys.exc_info()} and @ref{1d2,,PyErr_GetExcInfo()} if the exception
instance is modified while the exception is handled, as well as
@ref{1d1,,PyErr_SetExcInfo()}, which now ignores the @code{type} and
@code{traceback} arguments provided to it.

@item 
bpo-45727@footnote{https://bugs.python.org/issue45727}: Refine the custom syntax error that suggests that a comma may
be missing to trigger only when the expressions are detected between
parentheses or brackets. Patch by Pablo Galindo

@item 
bpo-45885@footnote{https://bugs.python.org/issue45885}: Specialized the @code{COMPARE_OP} opcode using the PEP 659
machinery.

@item 
bpo-45786@footnote{https://bugs.python.org/issue45786}: Allocate space for the interpreter frame in the frame object,
to avoid an additional allocation when the frame object outlives the frame
activation.

@item 
bpo-45614@footnote{https://bugs.python.org/issue45614}: Fix @ref{11a,,traceback} display for exceptions with invalid module
name.

@item 
bpo-45813@footnote{https://bugs.python.org/issue45813}: Fix crash when calling coro.cr_frame.clear() after coroutine
has been freed.

@item 
bpo-45811@footnote{https://bugs.python.org/issue45811}: Improve the tokenizer errors when encountering invisible
control characters in the parser. Patch by Pablo Galindo

@item 
bpo-45848@footnote{https://bugs.python.org/issue45848}: Allow the parser to obtain error lines directly from encoded
files. Patch by Pablo Galindo

@item 
bpo-45709@footnote{https://bugs.python.org/issue45709}: Restore behavior from 3.10 when tracing an exception raised
within a with statement.

@item 
bpo-44525@footnote{https://bugs.python.org/issue44525}: Adds new @ref{10cd,,COPY_FREE_VARS} opcode, to make copying of
free variables from function to frame explicit. Helps optimization of
calls to Python function.

@item 
bpo-45829@footnote{https://bugs.python.org/issue45829}: Specialize @ref{1080,,BINARY_SUBSCR} for classes with a
@code{__getitem__} method implemented in Python

@item 
bpo-45826@footnote{https://bugs.python.org/issue45826}: Fixed a crash when calling @code{.with_traceback(None)} on
@code{NameError}. This occurs internally in
@code{unittest.TestCase.assertRaises()}.

@item 
bpo-45822@footnote{https://bugs.python.org/issue45822}: Fixed a bug in the parser that was causing it to not respect
@geindex Python Enhancement Proposals; PEP 263
PEP 263@footnote{https://www.python.org/dev/peps/pep-0263} coding cookies when no flags are provided. Patch by Pablo
Galindo

@item 
bpo-45820@footnote{https://bugs.python.org/issue45820}: Fix a segfault when the parser fails without reading any input.
Patch by Pablo Galindo

@item 
bpo-45636@footnote{https://bugs.python.org/issue45636}: Simplify the implementation of @ref{19e,,BINARY_OP} by indexing
into an array of function pointers (rather than switching on the oparg).

@item 
bpo-42540@footnote{https://bugs.python.org/issue42540}: Fix crash when @ref{bb9,,os.fork()} is called with an active
non-default memory allocator.

@item 
bpo-45738@footnote{https://bugs.python.org/issue45738}: Fix computation of error location for invalid continuation
characters in the parser. Patch by Pablo Galindo.

@item 
bpo-45636@footnote{https://bugs.python.org/issue45636}: Remove an existing “fast path” for old-style string formatting,
since it no longer appears to have any measurable impact.

@item 
bpo-45753@footnote{https://bugs.python.org/issue45753}: Make recursion checks a bit more efficient by tracking amount
of calls left before overflow.

@item 
bpo-45773@footnote{https://bugs.python.org/issue45773}: Fix a compiler hang when attempting to optimize certain jump
patterns.

@item 
bpo-45764@footnote{https://bugs.python.org/issue45764}: The parser now gives a better error message when leaving out
the opening parenthesis @code{(} after a @code{def}-statement:

@example
>>> def f:
  File "<stdin>", line 1
    def f:
         ^
SyntaxError: expected '('
@end example

@item 
bpo-45609@footnote{https://bugs.python.org/issue45609}: Specialized the @code{STORE_SUBSCR} opcode using the PEP 659
machinery.

@item 
bpo-45636@footnote{https://bugs.python.org/issue45636}: Replace all numeric @code{BINARY_*} and @code{INPLACE_*} instructions
with a single @ref{19e,,BINARY_OP} implementation.

@item 
bpo-45582@footnote{https://bugs.python.org/issue45582}: Path calculation (known as @code{getpath}) has been reimplemented
as a frozen Python module. This should have no visible impact, but may
affect calculation of all paths referenced in @ref{f7,,sys} and
@ref{f8,,sysconfig}.

@item 
bpo-45450@footnote{https://bugs.python.org/issue45450}: Improve the syntax error message for parenthesized arguments.
Patch by Pablo Galindo.
@end itemize

@node Library<4>,Documentation<5>,Core and Builtins<4>,Python 3 11 0 alpha 3
@anchor{whatsnew/changelog id19}@anchor{10ce}
@subsubsection Library



@itemize -

@item 
bpo-27946@footnote{https://bugs.python.org/issue27946}: Fix possible crash when getting an attribute of
class:@ref{44e,,xml.etree.ElementTree.Element} simultaneously with replacing the
@code{attrib} dict.

@item 
bpo-45711@footnote{https://bugs.python.org/issue45711}: Make @ref{9,,asyncio} normalize exceptions as soon as they are
captured with @ref{bc2,,PyErr_Fetch()}, and before they are stored as an
exc_info triplet. This brings @ref{9,,asyncio} in line with the rest of the
codebase, where an exc_info triplet is always normalized.

@item 
bpo-23819@footnote{https://bugs.python.org/issue23819}: Replaced asserts with exceptions in asyncio, patch by Kumar
Aditya.

@item 
bpo-13236@footnote{https://bugs.python.org/issue13236}: @ref{10bd,,unittest.TextTestResult} and
@ref{10cf,,unittest.TextTestRunner} flush now the output stream more often.

@item 
bpo-45917@footnote{https://bugs.python.org/issue45917}: Added @ref{179,,math.exp2()}:, which returns 2 raised to the power
of x.

@item 
bpo-37658@footnote{https://bugs.python.org/issue37658}: Fix issue when on certain conditions @code{asyncio.wait_for()} may
allow a coroutine to complete successfully, but fail to return the result,
potentially causing memory leaks or other issues.

@item 
bpo-45876@footnote{https://bugs.python.org/issue45876}: Improve the accuracy of stdev() and pstdev() in the statistics
module.  When the inputs are floats or fractions, the output is a
correctly rounded float

@item 
bpo-44649@footnote{https://bugs.python.org/issue44649}: Handle dataclass(slots=True) with a field that has default a
default value, but for which init=False.

@item 
bpo-45803@footnote{https://bugs.python.org/issue45803}: Added missing kw_only parameter to
dataclasses.make_dataclass().

@item 
bpo-45837@footnote{https://bugs.python.org/issue45837}: The @code{turtle.RawTurtle.settiltangle()} is deprecated since
Python 3.1, it now emits a deprecation warning and will be removed in
Python 3.13.

Use @code{turtle.RawTurtle.tiltangle()} instead.

@code{turtle.RawTurtle.tiltangle()} was earlier incorrectly marked as
deprecated, its docstring has been corrected.

Patch by Hugo van Kemenade.

@item 
bpo-45831@footnote{https://bugs.python.org/issue45831}: @ref{78,,faulthandler} can now write ASCII-only strings (like
filenames and function names) with a single write() syscall when dumping a
traceback. It reduces the risk of getting an unreadable dump when two
threads or two processes dump a traceback to the same file (like stderr)
at the same time. Patch by Victor Stinner.

@item 
bpo-45828@footnote{https://bugs.python.org/issue45828}: @code{sqlite} C callbacks now use unraisable exceptions if
callback tracebacks are enabled. Patch by Erlend E. Aasland.

@item 
bpo-41735@footnote{https://bugs.python.org/issue41735}: Fix thread lock in @code{zlib.Decompress.flush()} method before
@code{PyObject_GetBuffer}.

@item 
bpo-45235@footnote{https://bugs.python.org/issue45235}: Reverted an argparse bugfix that caused regression in the
handling of default arguments for subparsers.  This prevented leaf level
arguments from taking precedence over root level arguments.

@item 
bpo-45754@footnote{https://bugs.python.org/issue45754}: Fix a regression in Python 3.11a1 and 3.11a2 where
@ref{ed,,sqlite3} incorrectly would use @code{SQLITE_LIMIT_LENGTH} when checking
SQL statement lengths. Now, @code{SQLITE_LIMIT_SQL_LENGTH} is used. Patch by
Erlend E. Aasland.

@item 
bpo-45766@footnote{https://bugs.python.org/issue45766}: Added @emph{proportional} option to
@ref{2b4,,statistics.linear_regression()}.

@item 
bpo-45765@footnote{https://bugs.python.org/issue45765}: In importlib.metadata, fix distribution discovery for an empty
path.

@item 
bpo-45757@footnote{https://bugs.python.org/issue45757}: Fix bug where @ref{36,,dis} produced an incorrect oparg when
@ref{107d,,EXTENDED_ARG} is followed by an opcode that does not use its
argument.

@item 
bpo-45644@footnote{https://bugs.python.org/issue45644}: In-place JSON file formatting using @code{python3 -m json.tool
infile infile} now works correctly, previously it left the file empty.
Patch by Chris Wesseling.

@item 
bpo-45703@footnote{https://bugs.python.org/issue45703}: When a namespace package is imported before another module from
the same namespace is created/installed in a different @ref{489,,sys.path}
location while the program is running, calling the
@ref{74b,,importlib.invalidate_caches()} function will now also guarantee the
new module is noticed.

@item 
bpo-45535@footnote{https://bugs.python.org/issue45535}: Improve output of @code{dir()} with Enums.

@item 
bpo-45664@footnote{https://bugs.python.org/issue45664}: Fix @ref{6cd,,types.resolve_bases()} and @ref{cfc,,types.new_class()} for
@ref{26a,,types.GenericAlias} instance as a base.

@item 
bpo-45663@footnote{https://bugs.python.org/issue45663}: Fix @ref{10d0,,dataclasses.is_dataclass()} for dataclasses which are
subclasses of @ref{26a,,types.GenericAlias}.

@item 
bpo-45662@footnote{https://bugs.python.org/issue45662}: Fix the repr of @code{dataclasses.InitVar} with a type alias
to the built-in class, e.g. @code{InitVar[list[int]]}.

@item 
bpo-43137@footnote{https://bugs.python.org/issue43137}: Launch GNOME web browsers via gio tool instead of obsolete
gvfs-open

@item 
bpo-45429@footnote{https://bugs.python.org/issue45429}: On Windows, @ref{196,,time.sleep()} now uses a waitable timer which
supports high-resolution timers. Patch by Dong-hee Na and Eryk Sun.

@item 
bpo-37295@footnote{https://bugs.python.org/issue37295}: Optimize @ref{19b,,math.comb()} and @ref{19c,,math.perm()}.

@item 
bpo-45514@footnote{https://bugs.python.org/issue45514}: Deprecated legacy functions in @ref{9a,,importlib.resources}.

@item 
bpo-45507@footnote{https://bugs.python.org/issue45507}: Add tests for truncated/missing trailers in gzip.decompress
implementation.

@item 
bpo-45359@footnote{https://bugs.python.org/issue45359}: Implement 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} for @ref{3c8,,graphlib.TopologicalSorter}.

@item 
bpo-44733@footnote{https://bugs.python.org/issue44733}: Add @code{max_tasks_per_child} to
@ref{3da,,concurrent.futures.ProcessPoolExecutor}. This allows users to
specify the maximum number of tasks a single process should execute before
the process needs to be restarted.

@item 
bpo-28806@footnote{https://bugs.python.org/issue28806}: Improve netrc library. netrc file no longer needs to contain
all tokens. And if the login name is anonymous, security check is no
longer need.

@item 
bpo-43498@footnote{https://bugs.python.org/issue43498}: Avoid a possible @emph{“RuntimeError: dictionary changed size during
iteration”} when adjusting the process count of
@code{ProcessPoolExecutor}.

@item 
bpo-42158@footnote{https://bugs.python.org/issue42158}: Add MIME types for N-quads, N-triples, Notation3 and TriG to
@code{mimetypes}.

@item 
bpo-30533@footnote{https://bugs.python.org/issue30533}: Add @ref{175,,inspect.getmembers_static()} , it return all members
without triggering dynamic lookup via the descriptor protocol. Patch by
Weipeng Hong.
@end itemize

@node Documentation<5>,Tests<4>,Library<4>,Python 3 11 0 alpha 3
@anchor{whatsnew/changelog id20}@anchor{10d1}
@subsubsection Documentation



@itemize -

@item 
bpo-42238@footnote{https://bugs.python.org/issue42238}: @code{make -C Doc suspicious} will be removed soon in favor of
@code{make -C Doc check}, mark it as deprecated.

@item 
bpo-45840@footnote{https://bugs.python.org/issue45840}: Improve cross-references in the documentation for the data
model.

@item 
bpo-45640@footnote{https://bugs.python.org/issue45640}: Properly marked-up grammar tokens in the documentation are now
clickable and take you to the definition of a given piece of grammar.
Patch by Arthur Milchior.

@item 
bpo-45788@footnote{https://bugs.python.org/issue45788}: Link doc for sys.prefix to sysconfig doc on installation paths.

@item 
bpo-45772@footnote{https://bugs.python.org/issue45772}: @code{socket.socket} documentation is corrected to a class from a
function.

@item 
bpo-45392@footnote{https://bugs.python.org/issue45392}: Update the docstring of the @ref{89e,,type} built-in to remove a
redundant line and to mention keyword arguments for the constructor.

@item 
bpo-45250@footnote{https://bugs.python.org/issue45250}: Update the documentation to note that CPython does not
consistently require iterators to define @code{__iter__}.

@item 
bpo-25381@footnote{https://bugs.python.org/issue25381}: In the extending chapter of the extending doc, update a
paragraph about the global variables containing exception information.

@item 
bpo-43905@footnote{https://bugs.python.org/issue43905}: Expanded @ref{10d2,,astuple()} and
@ref{10d3,,asdict()} docs, warning about deepcopy being applied and
providing a workaround.
@end itemize

@node Tests<4>,Build<4>,Documentation<5>,Python 3 11 0 alpha 3
@anchor{whatsnew/changelog id21}@anchor{10d4}
@subsubsection Tests



@itemize -

@item 
bpo-45695@footnote{https://bugs.python.org/issue45695}: Out-of-tree builds with a read-only source directory are now
tested by CI.

@item 
bpo-19460@footnote{https://bugs.python.org/issue19460}: Add new Test for
@code{Lib/email/mime/nonmultipart.py::MIMENonMultipart}.

@item 
bpo-45835@footnote{https://bugs.python.org/issue45835}: Fix race condition in test_queue tests with multiple “feeder”
threads.

@item 
bpo-45783@footnote{https://bugs.python.org/issue45783}: The test for the freeze tool now handles file moves and
deletions.

@item 
bpo-45745@footnote{https://bugs.python.org/issue45745}: Remove the @code{--findleaks} command line option of regrtest: use
the @code{--fail-env-changed} option instead. Since Python 3.7, it was a
deprecated alias to the @code{--fail-env-changed} option.

@item 
bpo-45701@footnote{https://bugs.python.org/issue45701}: Add tests with @code{tuple} type with @ref{4f3,,functools.lru_cache()}
to @code{test_functools}.
@end itemize

@node Build<4>,Windows<4>,Tests<4>,Python 3 11 0 alpha 3
@anchor{whatsnew/changelog id22}@anchor{10d5}
@subsubsection Build



@itemize -

@item 
bpo-44035@footnote{https://bugs.python.org/issue44035}: CI now verifies that autoconf files have been regenerated with
a current and unpatched autoconf package.

@item 
bpo-45950@footnote{https://bugs.python.org/issue45950}: The build system now uses a @code{_bootstrap_python}
interpreter for freezing and deepfreezing again. To speed up build process
the build tools @code{_bootstrap_python} and @code{_freeze_module}
are no longer build with LTO.

@item 
bpo-45881@footnote{https://bugs.python.org/issue45881}: The @code{configure} script now accepts
@code{--with-build-python} and @code{--with-freeze-module} options to make cross
compiling easier.

@item 
bpo-40280@footnote{https://bugs.python.org/issue40280}: Emscripten platform now uses @code{.wasm} suffix by default.

@item 
bpo-40280@footnote{https://bugs.python.org/issue40280}: Disable unusable core extension modules on WASM/Emscripten
targets.

@item 
bpo-40280@footnote{https://bugs.python.org/issue40280}: @code{configure} now checks for socket @code{shutdown} function. The
check makes it possible to disable @code{SYS_shutdown} with
@code{ac_cv_func_shutdown=no} in CONFIG_SITE.

@item 
bpo-40280@footnote{https://bugs.python.org/issue40280}: @code{configure} now checks for functions @code{fork1, getegid,
geteuid, getgid, getppid, getuid, opendir, pipe, system, wait, ttyname}.

@item 
bpo-33393@footnote{https://bugs.python.org/issue33393}: Update @code{config.guess} to 2021-06-03 and @code{config.sub} to
2021-08-14. @code{Makefile} now has an @code{update-config} target to make
updating more convenient.

@item 
bpo-45866@footnote{https://bugs.python.org/issue45866}: @code{make regen-all} now produces the same output when run from a
directory other than the source tree: when building Python out of the
source tree. pegen now strips directory of the “generated by pygen from
<FILENAME>” header Patch by Victor Stinner.

@item 
bpo-40280@footnote{https://bugs.python.org/issue40280}: @code{configure} now accepts machine @code{wasm32} or @code{wasm64} and
OS @code{wasi} or @code{emscripten} for cross building, e.g.
@code{wasm32-unknown-emscripten}, @code{wasm32-wasi}, or
@code{wasm32-unknown-wasi}.

@item 
bpo-41498@footnote{https://bugs.python.org/issue41498}: Python now compiles on platforms without @code{sigset_t}. Several
functions in @ref{e5,,signal} are not available when @code{sigset_t} is missing.

Based on patch by Roman Yurchak for pyodide.

@item 
bpo-45881@footnote{https://bugs.python.org/issue45881}: @code{setup.py} now uses @code{CC} from environment first to discover
multiarch and cross compile paths.

@item 
bpo-45886@footnote{https://bugs.python.org/issue45886}: The @code{_freeze_module} program path can now be overridden on
the command line, e.g. @code{make
FREEZE_MODULE=../x86_64/Program/_freeze_module}.

@item 
bpo-45873@footnote{https://bugs.python.org/issue45873}: Get rid of the @code{_bootstrap_python} build step. The
deepfreeze.py script is now run using @code{$(PYTHON_FOR_REGEN)} which can be
Python 3.7 or newer (on Windows, 3.8 or newer).

@item 
bpo-45847@footnote{https://bugs.python.org/issue45847}: Port builtin hashlib extensions to @code{PY_STDLIB_MOD} macro and
@code{addext()}.

@item 
bpo-45723@footnote{https://bugs.python.org/issue45723}: Add @code{autoconf} helpers for saving and restoring environment
variables:


@itemize *

@item 
@code{SAVE_ENV}: Save @code{$CFLAGS}, @code{$LDFLAGS}, @code{$LIBS}, and
@code{$CPPFLAGS}.

@item 
@code{RESTORE_ENV}: Restore @code{$CFLAGS}, @code{$LDFLAGS}, @code{$LIBS}, and
@code{$CPPFLAGS}.

@item 
@code{WITH_SAVE_ENV([SCRIPT])}: Run @code{SCRIPT} wrapped with @code{SAVE_ENV} and
@code{RESTORE_ENV}.
@end itemize

Patch by Erlend E. Aasland.

@item 
bpo-45573@footnote{https://bugs.python.org/issue45573}: Mandatory core modules, that are required to bootstrap Python,
are now in @code{Modules/Setup.bootstrap}.

@item 
bpo-45573@footnote{https://bugs.python.org/issue45573}: @code{configure} now creates @code{Modules/Setup.stdlib} with
conditionally enabled/disabled extension module lines. The file is not
used, yet.

@item 
bpo-45573@footnote{https://bugs.python.org/issue45573}: @code{configure} now uses a unified format to set state, compiler
flags, and linker flags in Makefile. The new macro @code{PY_STDLIB_MOD} sets
three variables that are consumed by @code{Modules/Setup} and @code{setup.py}.

@item 
bpo-45816@footnote{https://bugs.python.org/issue45816}: Python now supports building with Visual Studio 2022 (MSVC
v143, VS Version 17.0). Patch by Jeremiah Vivian.

@item 
bpo-45800@footnote{https://bugs.python.org/issue45800}: Settings for @code{pyexpat} C extension are now detected by
@code{configure}. The bundled @code{expat} library is built in @code{Makefile}.

@item 
bpo-45798@footnote{https://bugs.python.org/issue45798}: Settings for @ref{34,,decimal} internal C extension are now
detected by @code{configure}. The bundled @code{libmpdec} library is built in
@code{Makefile}.

@item 
bpo-45723@footnote{https://bugs.python.org/issue45723}: @code{configure} has a new option @code{--with-pkg-config} to
disable or require pkg-config.

@item 
bpo-45774@footnote{https://bugs.python.org/issue45774}: The build dependencies for @ref{ed,,sqlite3} are now detected by
@code{configure} and @code{pkg-config}. Patch by Erlend E. Aasland.

@item 
bpo-45763@footnote{https://bugs.python.org/issue45763}: The build dependencies for @ref{14b,,zlib}, @ref{12,,bz2}, and
@ref{a9,,lzma} are now detected by @code{configure}.

@item 
bpo-45747@footnote{https://bugs.python.org/issue45747}: gdbm and dbm build dependencies are now detected by
@code{configure}.

@item 
bpo-45743@footnote{https://bugs.python.org/issue45743}: On macOS, the build system no longer passes
@code{search_paths_first} to the linker. The flag has been the default since
Xcode 4 / macOS 10.6.

@item 
bpo-45723@footnote{https://bugs.python.org/issue45723}: @code{configure.ac} is now compatible with autoconf 2.71.
Deprecated checks @code{STDC_HEADERS} and @code{AC_HEADER_TIME} have been
removed.

@item 
bpo-45723@footnote{https://bugs.python.org/issue45723}: @code{configure} now prints a warning when pkg-config is missing.

@item 
bpo-45731@footnote{https://bugs.python.org/issue45731}: @code{configure --enable-loadable-sqlite-extensions} is now
handled by new @code{PY_SQLITE_ENABLE_LOAD_EXTENSION} macro instead of logic
in setup.py.

@item 
bpo-45723@footnote{https://bugs.python.org/issue45723}: configure.ac now uses custom helper macros and
@code{AC_CACHE_CHECK} to simplify and speed up configure runs.

@item 
bpo-45696@footnote{https://bugs.python.org/issue45696}: Skip the marshal step for frozen modules by generating C code
that produces a set of ready-to-use code objects. This speeds up startup
time by another 10% or more.

@item 
bpo-45561@footnote{https://bugs.python.org/issue45561}: Run smelly.py tool from $(srcdir).
@end itemize

@node Windows<4>,macOS<3>,Build<4>,Python 3 11 0 alpha 3
@anchor{whatsnew/changelog id23}@anchor{10d6}
@subsubsection Windows



@itemize -

@item 
bpo-46105@footnote{https://bugs.python.org/issue46105}: Fixed calculation of @ref{489,,sys.path} in a venv on Windows.

@item 
bpo-45901@footnote{https://bugs.python.org/issue45901}: When installed through the Microsoft Store and set as the
default app for @code{*.py} files, command line arguments will now be
passed to Python when invoking a script without explicitly launching
Python (that is, @code{script.py args} rather than @code{python script.py
args}).

@item 
bpo-45616@footnote{https://bugs.python.org/issue45616}: Fix Python Launcher’s ability to distinguish between versions
3.1 and 3.10 when either one is explicitly requested.  Previously, 3.1
would be used if 3.10 was requested but not installed, and 3.10 would be
used if 3.1 was requested but 3.10 was installed.

@item 
bpo-45850@footnote{https://bugs.python.org/issue45850}: Implement changes to build with deep-frozen modules on Windows.
Note that we now require Python 3.10 as the “bootstrap” or “host” Python.

@item 
bpo-45732@footnote{https://bugs.python.org/issue45732}: Updates bundled Tcl/Tk to 8.6.12.

@item 
bpo-45720@footnote{https://bugs.python.org/issue45720}: Internal reference to @code{shlwapi.dll} was dropped to help
improve startup time. This DLL will no longer be loaded at the start of
every Python process.
@end itemize

@node macOS<3>,C API<4>,Windows<4>,Python 3 11 0 alpha 3
@anchor{whatsnew/changelog id24}@anchor{10d7}
@subsubsection macOS



@itemize -

@item 
bpo-45732@footnote{https://bugs.python.org/issue45732}: Update python.org macOS installer to use Tcl/Tk 8.6.12.
@end itemize

@node C API<4>,,macOS<3>,Python 3 11 0 alpha 3
@anchor{whatsnew/changelog id25}@anchor{10d8}
@subsubsection C API



@itemize -

@item 
bpo-39026@footnote{https://bugs.python.org/issue39026}: Fix Python.h to build C extensions with Xcode: remove a
relative include from @code{Include/cpython/pystate.h}.
@end itemize

@node Python 3 11 0 alpha 2,Python 3 11 0 alpha 1,Python 3 11 0 alpha 3,Changelog
@anchor{whatsnew/changelog python-3-11-0-alpha-2}@anchor{10d9}
@subsection Python 3.11.0 alpha 2


@emph{Release date: 2021-11-05}

@menu
* Core and Builtins: Core and Builtins<5>. 
* Library: Library<5>. 
* Documentation: Documentation<6>. 
* Tests: Tests<5>. 
* Build: Build<5>. 
* Windows: Windows<5>. 
* macOS: macOS<4>. 
* IDLE: IDLE<5>. 
* C API: C API<5>. 

@end menu

@node Core and Builtins<5>,Library<5>,,Python 3 11 0 alpha 2
@anchor{whatsnew/changelog id26}@anchor{10da}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-45716@footnote{https://bugs.python.org/issue45716}: Improve the @ref{21a,,SyntaxError} message when using @code{True},
@code{None} or @code{False} as keywords in a function call. Patch by Pablo
Galindo.

@item 
bpo-45688@footnote{https://bugs.python.org/issue45688}: @ref{2c7,,sys.stdlib_module_names} now contains the macOS-specific
module @code{_scproxy}.

@item 
bpo-45379@footnote{https://bugs.python.org/issue45379}: Clarify @ref{3b8,,ImportError} message when we try to explicitly
import a frozen module but frozen modules are disabled.

@item 
bpo-44525@footnote{https://bugs.python.org/issue44525}: Specialize simple calls to Python functions (no starargs,
keyowrd dict, or closure)

@item 
bpo-45530@footnote{https://bugs.python.org/issue45530}: Cases of sorting using tuples as keys may now be significantly
faster in some cases. Patch by Tim Peters.

The order of the result may differ from earlier releases if the tuple
elements don’t define a total ordering (see
@ref{1cc,,Value comparisons} for information on total ordering).
It’s generally true that the result of sorting simply isn’t well-defined
in the absence of a total ordering on list elements.

@item 
bpo-45526@footnote{https://bugs.python.org/issue45526}: In obmalloc, set ADDRESS_BITS to not ignore any bits (ignored
16 before). That is safer in the case that the kernel gives user-space
virtual addresses that span a range greater than 48 bits.

@item 
bpo-30570@footnote{https://bugs.python.org/issue30570}: Fixed a crash in @code{issubclass()} from infinite recursion when
searching pathological @code{__bases__} tuples.

@item 
bpo-45521@footnote{https://bugs.python.org/issue45521}: Fix a bug in the obmalloc radix tree code.  On 64-bit machines,
the bug causes the tree to hold 46-bits of virtual addresses, rather than
the intended 48-bits.

@item 
bpo-45494@footnote{https://bugs.python.org/issue45494}: Fix parser crash when reporting errors involving invalid
continuation characters. Patch by Pablo Galindo.

@item 
bpo-45445@footnote{https://bugs.python.org/issue45445}: Python now fails to initialize if it finds an invalid
@ref{159,,-X} option in the command line. Patch by Pablo Galindo.

@item 
bpo-45340@footnote{https://bugs.python.org/issue45340}: Object attributes are held in an array instead of a dictionary.
An object’s dictionary are created lazily, only when needed. Reduces the
memory consumption of a typical Python object by about 30%. Patch by Mark
Shannon.

@item 
bpo-45408@footnote{https://bugs.python.org/issue45408}: Fix a crash in the parser when reporting tokenizer errors that
occur at the same time unclosed parentheses are detected. Patch by Pablo
Galindo.

@item 
bpo-29410@footnote{https://bugs.python.org/issue29410}: Add SipHash13 for string hash algorithm and use it by default.

@item 
bpo-45385@footnote{https://bugs.python.org/issue45385}: Fix reference leak from descr_check. Patch by Dong-hee Na.

@item 
bpo-45367@footnote{https://bugs.python.org/issue45367}: Specialized the @code{BINARY_MULTIPLY} opcode to
@code{BINARY_MULTIPLY_INT} and @code{BINARY_MULTIPLY_FLOAT} using the PEP 659
machinery.

@item 
bpo-21736@footnote{https://bugs.python.org/issue21736}: Frozen stdlib modules now have @code{__file__} to the .py file
they would otherwise be loaded from, if possible.  For packages,
@code{__path__} now has the correct entry instead of being an empty list,
which allows unfrozen submodules to be imported.  These are set only if
the stdlib directory is known when the runtime is initialized.  Note that
the file at @code{__file__} is not guaranteed to exist.  None of this affects
non-stdlib frozen modules nor, for now, frozen modules imported using
@code{PyImport_ImportFrozenModule()}.  Also, at the moment @code{co_filename} is
not updated for the module.

@item 
bpo-45020@footnote{https://bugs.python.org/issue45020}: For frozen stdlib modules, record the original module name as
@code{module.__spec__.loader_state.origname}.  If the value is different than
@code{module.__spec__.name} then the module was defined as an alias in
Tools/scripts/freeze_modules.py.  If it is @code{None} then the module comes
from a source file outside the stdlib.

@item 
bpo-45324@footnote{https://bugs.python.org/issue45324}: In FrozenImporter.find_spec(), we now preserve the information
needed in exec_module() to load the module.  This change mostly impacts
internal details, rather than changing the importer’s behavior.

@item 
bpo-45292@footnote{https://bugs.python.org/issue45292}: Implement 
@geindex Python Enhancement Proposals; PEP 654
PEP 654@footnote{https://www.python.org/dev/peps/pep-0654}. Add @ref{10db,,ExceptionGroup} and
@ref{10dc,,BaseExceptionGroup}. Update traceback display code.

@item 
bpo-40116@footnote{https://bugs.python.org/issue40116}: Change to the implementation of split dictionaries. Classes
where the instances differ either in the exact set of attributes, or in
the order in which those attributes are set, can still share keys. This
should have no observable effect on users of Python or the C-API. Patch by
Mark Shannon.

@item 
bpo-44050@footnote{https://bugs.python.org/issue44050}: Extensions that indicate they use global state (by setting
@code{m_size} to -1) can again be used in multiple interpreters. This reverts
to behavior of Python 3.8.

@item 
bpo-44525@footnote{https://bugs.python.org/issue44525}: Setup initial specialization infrastructure for the
@code{CALL_FUNCTION} opcode. Implemented initial specializations for C
function calls:


@itemize *

@item 
@code{CALL_FUNCTION_BUILTIN_O} for @code{METH_O} flag.

@item 
@code{CALL_FUNCTION_BUILTIN_FAST} for @code{METH_FASTCALL} flag without keywords.

@item 
@code{CALL_FUNCTION_LEN} for @code{len(o)}.

@item 
@code{CALL_FUNCTION_ISINSTANCE} for @code{isinstance(o, t)}.
@end itemize

@item 
bpo-44511@footnote{https://bugs.python.org/issue44511}: Improve the generated bytecode for class and mapping patterns.

@item 
bpo-43706@footnote{https://bugs.python.org/issue43706}: Speed up calls to @code{enumerate()} by using the 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590}
@code{vectorcall} calling convention. Patch by Dong-hee Na.
@end itemize

@node Library<5>,Documentation<6>,Core and Builtins<5>,Python 3 11 0 alpha 2
@anchor{whatsnew/changelog id27}@anchor{10dd}
@subsubsection Library



@itemize -

@item 
bpo-45679@footnote{https://bugs.python.org/issue45679}: Fix caching of multi-value @ref{2db,,typing.Literal}.
@code{Literal[True, 2]} is no longer equal to @code{Literal[1, 2]}.

@item 
bpo-42064@footnote{https://bugs.python.org/issue42064}: Convert @ref{ed,,sqlite3} to multi-phase initialisation (PEP 489).
Patches by Erlend E. Aasland.

@item 
bpo-45438@footnote{https://bugs.python.org/issue45438}: Fix typing.Signature string representation for generic builtin
types.

@item 
bpo-45613@footnote{https://bugs.python.org/issue45613}: @ref{ed,,sqlite3} now sets @ref{18c,,sqlite3.threadsafety} based on
the default threading mode the underlying SQLite library has been compiled
with. Patch by Erlend E. Aasland.

@item 
bpo-45574@footnote{https://bugs.python.org/issue45574}: Fix warning about @code{print_escape} being unused.

@item 
bpo-45581@footnote{https://bugs.python.org/issue45581}: @ref{6a8,,sqlite3.connect()} now correctly raises @ref{10de,,MemoryError}
if the underlying SQLite API signals memory error. Patch by Erlend E.
Aasland.

@item 
bpo-45557@footnote{https://bugs.python.org/issue45557}: pprint.pprint() now handles underscore_numbers correctly.
Previously it was always setting it to False.

@item 
bpo-44019@footnote{https://bugs.python.org/issue44019}: Add @ref{10df,,operator.call()} to @code{operator.__all__}. Patch by
Kreusada.

@item 
bpo-42174@footnote{https://bugs.python.org/issue42174}: @ref{c96,,shutil.get_terminal_size()} now falls back to sane values
if the column or line count are 0.

@item 
bpo-35673@footnote{https://bugs.python.org/issue35673}: Improve the introspectability of the @code{__loader__} attribute
for namespace packages.  @code{importlib.machinery.NamespaceLoader} is
now public, and implements the @ref{ad2,,importlib.abc.InspectLoader}
interface. @code{_NamespaceLoader} is kept for backward compatibility.

@item 
bpo-45515@footnote{https://bugs.python.org/issue45515}: Add references to @ref{14c,,zoneinfo} in the @ref{2f,,datetime}
documentation, mostly replacing outdated references to @code{dateutil.tz}.
Change by Paul Ganssle.

@item 
bpo-45475@footnote{https://bugs.python.org/issue45475}: Reverted optimization of iterating @ref{43f,,gzip.GzipFile},
@ref{455,,bz2.BZ2File}, and @ref{b93,,lzma.LZMAFile} (see bpo-43787@footnote{https://bugs.python.org/issue43787}) because
it caused regression when user iterate them without having reference of
them. Patch by Inada Naoki.

@item 
bpo-45489@footnote{https://bugs.python.org/issue45489}: Update @ref{1090,,ForwardRef} to support @code{|} operator.
Patch by Dong-hee Na.

@item 
bpo-42222@footnote{https://bugs.python.org/issue42222}: Removed deprecated support for float arguments in
@emph{randrange()}.

@item 
bpo-45428@footnote{https://bugs.python.org/issue45428}: Fix a regression in py_compile when reading filenames from
standard input.

@item 
bpo-45467@footnote{https://bugs.python.org/issue45467}: Fix incremental decoder and stream reader in the
“raw-unicode-escape” codec. Previously they failed if the escape sequence
was split.

@item 
bpo-45461@footnote{https://bugs.python.org/issue45461}: Fix incremental decoder and stream reader in the
“unicode-escape” codec. Previously they failed if the escape sequence was
split.

@item 
bpo-45239@footnote{https://bugs.python.org/issue45239}: Fixed @ref{10e0,,email.utils.parsedate_tz()} crashing with
@ref{1067,,UnboundLocalError} on certain invalid input instead of returning
@code{None}. Patch by Ben Hoyt.

@item 
bpo-45417@footnote{https://bugs.python.org/issue45417}: Fix quadratic behaviour in the enum module: Creation of enum
classes with a lot of entries was quadratic.

@item 
bpo-45249@footnote{https://bugs.python.org/issue45249}: Fix the behaviour of @ref{10e1,,traceback.print_exc()} when
displaying the caret when the @code{end_offset} in the exception is set to 0.
Patch by Pablo Galindo

@item 
bpo-45416@footnote{https://bugs.python.org/issue45416}: Fix use of @ref{58d,,asyncio.Condition} with explicit
@ref{58b,,asyncio.Lock} objects, which was a regression due to removal of
explicit loop arguments. Patch by Joongi Kim.

@item 
bpo-20028@footnote{https://bugs.python.org/issue20028}: Empty escapechar/quotechar is not allowed when initializing
@ref{10e2,,csv.Dialect}. Patch by Vajrasky Kok and Dong-hee Na.

@item 
bpo-44904@footnote{https://bugs.python.org/issue44904}: Fix bug in the @ref{63,,doctest} module that caused it to fail if
a docstring included an example with a @code{classmethod} @code{property}. Patch
by Alex Waygood.

@item 
bpo-45406@footnote{https://bugs.python.org/issue45406}: Make @ref{10e3,,inspect.getmodule()} catch @code{FileNotFoundError}
raised by :’func:@code{inspect.getabsfile}, and return @code{None} to indicate
that the module could not be determined.

@item 
bpo-45411@footnote{https://bugs.python.org/issue45411}: Add extensions for files containing subtitles - .srt & .vtt -
to the mimetypes.py module.

@item 
bpo-10716@footnote{https://bugs.python.org/issue10716}: Migrated pydoc to HTML5 (without changing the look of it). Side
effect is to update xmlrpc’s @code{ServerHTMLDoc} which now uses the CSS too.
cgitb now relies less on pydoc (as it can’t use the CSS file).

@item 
bpo-27580@footnote{https://bugs.python.org/issue27580}: Add support of null characters in @ref{28,,csv}.

@item 
bpo-45262@footnote{https://bugs.python.org/issue45262}: Prevent use-after-free in asyncio. Make sure the cached running
loop holder gets cleared on dealloc to prevent use-after-free in
get_running_loop

@item 
bpo-45386@footnote{https://bugs.python.org/issue45386}: Make @ref{146,,xmlrpc.client} more robust to C runtimes where the
underlying C @code{strftime} function results in a @code{ValueError} when
testing for year formatting options.

@item 
bpo-20028@footnote{https://bugs.python.org/issue20028}: Improve error message of @ref{10e2,,csv.Dialect} when
initializing. Patch by Vajrasky Kok and Dong-hee Na.

@item 
bpo-45343@footnote{https://bugs.python.org/issue45343}: Update bundled pip to 21.2.4 and setuptools to 58.1.0

@item 
bpo-45328@footnote{https://bugs.python.org/issue45328}: Fixed @ref{663,,http.client.HTTPConnection} to work properly in
OSs that don’t support the @code{TCP_NODELAY} socket option.

@item 
bpo-45243@footnote{https://bugs.python.org/issue45243}: Add @ref{189,,setlimit()} and
@ref{18a,,getlimit()} to @ref{18b,,sqlite3.Connection} for
setting and getting SQLite limits by connection basis. Patch by Erlend E.
Aasland.

@item 
bpo-45320@footnote{https://bugs.python.org/issue45320}: Removed from the @ref{9c,,inspect} module:


@itemize *

@item 

@table @asis

@item the @code{getargspec} function, deprecated since Python 3.0;

use @ref{1c1,,inspect.signature()} or @ref{1c2,,inspect.getfullargspec()} instead.
@end table

@item 
the @code{formatargspec} function, deprecated since Python 3.5;
use the @ref{1c1,,inspect.signature()} function and @code{Signature} object
directly.

@item 
the undocumented @code{Signature.from_callable} and @code{Signature.from_function}
functions, deprecated since Python 3.5; use the
@ref{1c3,,Signature.from_callable()} method
instead.
@end itemize

Patch by Hugo van Kemenade.

@item 
bpo-45192@footnote{https://bugs.python.org/issue45192}: Fix the @code{tempfile._infer_return_type} function so that the
@code{dir} argument of the @ref{fd,,tempfile} functions accepts an object
implementing the @code{os.PathLike} protocol.

Patch by Kyungmin Lee.

@item 
bpo-45160@footnote{https://bugs.python.org/issue45160}: When tracing a tkinter variable used by a ttk OptionMenu,
callbacks are no longer made twice.

@item 
bpo-25625@footnote{https://bugs.python.org/issue25625}: Added non parallel-safe @ref{10e4,,chdir()} context
manager to change the current working directory and then restore it on
exit. Simple wrapper around @ref{3b9,,chdir()}.

@item 
bpo-24139@footnote{https://bugs.python.org/issue24139}: Add support for SQLite extended result codes in
@ref{10e5,,sqlite3.Error}. Patch by Erlend E. Aasland.

@item 
bpo-24444@footnote{https://bugs.python.org/issue24444}: Fixed an error raised in @ref{5,,argparse} help display when help
for an option is set to 1+ blank spaces or when @emph{choices} arg is an empty
container.

@item 
bpo-44547@footnote{https://bugs.python.org/issue44547}: Implement @code{Fraction.__int__}, so that a
@ref{172,,fractions.Fraction} instance @code{f} passes an @code{isinstance(f,
typing.SupportsInt)} check.

@item 
bpo-40321@footnote{https://bugs.python.org/issue40321}: Adds support for HTTP 308 redirects to @ref{124,,urllib}. See
@geindex RFC; RFC 7538
RFC 7538@footnote{https://tools.ietf.org/html/rfc7538.html} for details. Patch by Jochem Schulenklopper.

@item 
bpo-10572@footnote{https://bugs.python.org/issue10572}: Move @ref{ed,,sqlite3} tests to @code{/Lib/test/test_sqlite3}. Patch
by Erlend E. Aasland.

@item 
bpo-41374@footnote{https://bugs.python.org/issue41374}: Ensure that @code{socket.TCP_*} constants are exposed on Cygwin
3.1.6 and greater.

@item 
bpo-35970@footnote{https://bugs.python.org/issue35970}: Add help flag to the base64 module’s command line interface.
Patch contributed by Robert Kuska.
@end itemize

@node Documentation<6>,Tests<5>,Library<5>,Python 3 11 0 alpha 2
@anchor{whatsnew/changelog id28}@anchor{10e6}
@subsubsection Documentation



@itemize -

@item 
bpo-45726@footnote{https://bugs.python.org/issue45726}: Improve documentation for @ref{65b,,functools.singledispatch()} and
@ref{4f5,,functools.singledispatchmethod}.

@item 
bpo-45680@footnote{https://bugs.python.org/issue45680}: Amend the docs on @code{GenericAlias} objects to clarify that
non-container classes can also implement @code{__class_getitem__}. Patch
contributed by Alex Waygood.

@item 
bpo-45618@footnote{https://bugs.python.org/issue45618}: Update Sphinx version used to build the documentation to 4.2.0.
Patch by Maciej Olko.

@item 
bpo-45655@footnote{https://bugs.python.org/issue45655}: Add a new “relevant PEPs” section to the top of the
documentation for the @code{typing} module. Patch by Alex Waygood.

@item 
bpo-45604@footnote{https://bugs.python.org/issue45604}: Add @code{level} argument to @code{multiprocessing.log_to_stderr}
function docs.

@item 
bpo-45516@footnote{https://bugs.python.org/issue45516}: Add protocol description to the
@ref{10e7,,importlib.abc.TraversableResources} documentation.

@item 
bpo-45464@footnote{https://bugs.python.org/issue45464}: Mention in the documentation of @ref{10e8,,Built-in Exceptions} that inheriting from multiple exception types in a
single subclass is not recommended due to possible memory layout
incompatibility.

@item 
bpo-45449@footnote{https://bugs.python.org/issue45449}: Add note about 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} in @ref{1d,,collections.abc}.

@item 
bpo-45516@footnote{https://bugs.python.org/issue45516}: Add protocol description to the
@ref{10e9,,importlib.abc.Traversable} documentation.

@item 
bpo-20692@footnote{https://bugs.python.org/issue20692}: Add Programming FAQ entry explaining that int literal attribute
access requires either a space after or parentheses around the literal.
@end itemize

@node Tests<5>,Build<5>,Documentation<6>,Python 3 11 0 alpha 2
@anchor{whatsnew/changelog id29}@anchor{10ea}
@subsubsection Tests



@itemize -

@item 
bpo-45678@footnote{https://bugs.python.org/issue45678}: Add tests for scenarios in which
@ref{4f5,,functools.singledispatchmethod} is stacked on top of a method that
has already been wrapped by two other decorators. Patch by Alex Waygood.

@item 
bpo-45578@footnote{https://bugs.python.org/issue45578}: Add tests for @ref{aa7,,dis.distb()}

@item 
bpo-45678@footnote{https://bugs.python.org/issue45678}: Add tests to ensure that @code{functools.singledispatchmethod}
correctly wraps the attributes of the target function.

@item 
bpo-45668@footnote{https://bugs.python.org/issue45668}: PGO tests now pass when Python is built without test extension
modules.

@item 
bpo-45577@footnote{https://bugs.python.org/issue45577}: Add subtests for all @code{pickle} protocols in @code{test_zoneinfo}.

@item 
bpo-45566@footnote{https://bugs.python.org/issue45566}: Fix @code{test_frozen_pickle} in @code{test_dataclasses} to check all
@code{pickle} versions.

@item 
bpo-43592@footnote{https://bugs.python.org/issue43592}: @code{test.libregrtest} now raises the soft resource limit for
the maximum number of file descriptors when the default is too low for our
test suite as was often the case on macOS.

@item 
bpo-39679@footnote{https://bugs.python.org/issue39679}: Add more test cases for @code{@@functools.singledispatchmethod} when
combined with @code{@@classmethod} or @code{@@staticmethod}.

@item 
bpo-45410@footnote{https://bugs.python.org/issue45410}: When libregrtest spawns a worker process, stderr is now written
into stdout to keep messages order. Use a single pipe for stdout and
stderr, rather than two pipes. Previously, messages were out of order
which made analysis of buildbot logs harder Patch by Victor Stinner.

@item 
bpo-45402@footnote{https://bugs.python.org/issue45402}: Fix test_tools.test_sundry() when Python is built out of tree:
fix how the freeze_modules.py tool locates the _freeze_module program.
Patch by Victor Stinner.

@item 
bpo-45403@footnote{https://bugs.python.org/issue45403}: Fix test_sys.test_stdlib_dir() when Python is built outside the
source tree: compare normalized paths. Patch by Victor Stinner.

@item 
bpo-45400@footnote{https://bugs.python.org/issue45400}: Fix
test_name_error_suggestions_do_not_trigger_for_too_many_locals() of
test_exceptions if a directory name contains “a1” (like
“Python-3.11.0a1”): use a stricter regular expression. Patch by Victor
Stinner.
@end itemize

@node Build<5>,Windows<5>,Tests<5>,Python 3 11 0 alpha 2
@anchor{whatsnew/changelog id30}@anchor{10eb}
@subsubsection Build



@itemize -

@item 
bpo-43158@footnote{https://bugs.python.org/issue43158}: @code{setup.py} now uses values from configure script to build the
@code{_uuid} extension module. Configure now detects util-linux’s
@code{libuuid}, too.

@item 
bpo-45666@footnote{https://bugs.python.org/issue45666}: Fix warning of @code{swprintf} and @code{%s} usage in
@code{_testembed.c}

@item 
bpo-45548@footnote{https://bugs.python.org/issue45548}: @code{Modules/Setup} and @code{Modules/makesetup} have been improved.
The @code{Setup} file now contains working rules for all extensions. Outdated
comments have been removed. Rules defined by @code{makesetup} track
dependencies correctly.

@item 
bpo-45548@footnote{https://bugs.python.org/issue45548}: The @ref{ad,,math} and @ref{17,,cmath} implementation now require a
C99 compatible @code{libm} and no longer ship with workarounds for missing
acosh, asinh, atanh, expm1, and log1p functions.

@item 
bpo-45595@footnote{https://bugs.python.org/issue45595}: @code{setup.py} and @code{makesetup} now track build dependencies on
all Python header files and module specific header files.

@item 
bpo-45571@footnote{https://bugs.python.org/issue45571}: @code{Modules/Setup} now use @code{PY_CFLAGS_NODIST} instead of
@code{PY_CFLAGS} to compile shared modules.

@item 
bpo-45570@footnote{https://bugs.python.org/issue45570}: @code{pyexpat} and @code{_elementtree} no longer define
obsolete macros @code{HAVE_EXPAT_CONFIG_H} and @code{USE_PYEXPAT_CAPI}.
@code{XML_POOR_ENTROPY} is now defined in @code{expat_config.h}.

@item 
bpo-43974@footnote{https://bugs.python.org/issue43974}: @code{setup.py} no longer defines @code{Py_BUILD_CORE_MODULE}.
Instead every module, that uses the internal API, defines the macro.

@item 
bpo-45548@footnote{https://bugs.python.org/issue45548}: Fill in missing entries in Modules/Setup.

@item 
bpo-45532@footnote{https://bugs.python.org/issue45532}: Update @ref{86c,,sys.version} to use @code{main} as fallback
information. Patch by Jeong YunWon.

@item 
bpo-45536@footnote{https://bugs.python.org/issue45536}: The @code{configure} script now checks whether OpenSSL headers and
libraries provide required APIs. Most common APIs are verified. The check
detects outdated or missing OpenSSL. Failures do not stop configure.

@item 
bpo-45221@footnote{https://bugs.python.org/issue45221}: Fixed regression in handling of @code{LDFLAGS} and @code{CPPFLAGS}
options where @code{argparse.parse_known_args()} could interpret an option
as one of the built-in command line argument, for example @code{-h} for help.

@item 
bpo-45440@footnote{https://bugs.python.org/issue45440}: Building Python now requires a C99 @code{<math.h>} header file
providing the following functions: @code{copysign()}, @code{hypot()},
@code{isfinite()}, @code{isinf()}, @code{isnan()}, @code{round()}. Patch by Victor
Stinner.

@item 
bpo-45405@footnote{https://bugs.python.org/issue45405}: Prevent @code{internal configure error} when running @code{configure}
with recent versions of non-Apple clang.  Patch by David Bohman.

@item 
bpo-45433@footnote{https://bugs.python.org/issue45433}: Avoid linking libpython with libcrypt.
@end itemize

@node Windows<5>,macOS<4>,Build<5>,Python 3 11 0 alpha 2
@anchor{whatsnew/changelog id31}@anchor{10ec}
@subsubsection Windows



@itemize -

@item 
bpo-43652@footnote{https://bugs.python.org/issue43652}: Update Tcl/Tk to 8.6.11, actually this time. The previous
update incorrectly included 8.6.10.

@item 
bpo-45337@footnote{https://bugs.python.org/issue45337}: venv now warns when the created environment may need to be
accessed at a different path, due to redirections, links or junctions. It
also now correctly installs or upgrades components when the alternate path
is required.

@item 
bpo-43851@footnote{https://bugs.python.org/issue43851}: Build SQLite @code{SQLITE_OMIT_AUTOINIT} on Windows. Patch by
Erlend E. Aasland.
@end itemize

@node macOS<4>,IDLE<5>,Windows<5>,Python 3 11 0 alpha 2
@anchor{whatsnew/changelog id32}@anchor{10ed}
@subsubsection macOS



@itemize -

@item 
bpo-44828@footnote{https://bugs.python.org/issue44828}: Avoid tkinter file dialog failure on macOS 12 Monterey when
using the Tk 8.6.11 provided by python.org macOS installers. Patch by Marc
Culler of the Tk project.
@end itemize

@node IDLE<5>,C API<5>,macOS<4>,Python 3 11 0 alpha 2
@anchor{whatsnew/changelog id33}@anchor{10ee}
@subsubsection IDLE



@itemize -

@item 
bpo-45495@footnote{https://bugs.python.org/issue45495}: Add context keywords ‘case’ and ‘match’ to completions list.
@end itemize

@node C API<5>,,IDLE<5>,Python 3 11 0 alpha 2
@anchor{whatsnew/changelog id34}@anchor{10ef}
@subsubsection C API



@itemize -

@item 
bpo-29103@footnote{https://bugs.python.org/issue29103}: @ref{36d,,PyType_FromSpec*} now
copies the class name from the spec to a buffer owned by the class, so the
original can be safely deallocated. Patch by Petr Viktorin.

@item 
bpo-45522@footnote{https://bugs.python.org/issue45522}: The internal freelists for frame, float, list, dict, async
generators, and context objects can now be disabled.

@item 
bpo-35134@footnote{https://bugs.python.org/issue35134}: Exclude @ref{211,,PyWeakref_GET_OBJECT()} from the limited C API.
It never worked since the @code{PyWeakReference} structure is opaque in
the limited C API.

@item 
bpo-35081@footnote{https://bugs.python.org/issue35081}: Move the @code{interpreteridobject.h} header file from
@code{Include/} to @code{Include/internal/}. It only provides private functions.
Patch by Victor Stinner.

@item 
bpo-35134@footnote{https://bugs.python.org/issue35134}: The non-limited API files @code{cellobject.h}, @code{classobject.h},
@code{context.h}, @code{funcobject.h}, @code{genobject.h} and @code{longintrepr.h}
have been moved to the @code{Include/cpython} directory. Moreover, the
@code{eval.h} header file was removed. These files must not be included
directly, as they are already included in @code{Python.h}: @ref{1f5,,Include Files}. If they have been included directly, consider
including @code{Python.h} instead. Patch by Victor Stinner.

@item 
bpo-45474@footnote{https://bugs.python.org/issue45474}: The following items are no longer available when
@code{Py_LIMITED_API} is defined:


@itemize *

@item 
@ref{20c,,PyMarshal_WriteLongToFile()}

@item 
@ref{20d,,PyMarshal_WriteObjectToFile()}

@item 
@ref{20e,,PyMarshal_ReadObjectFromString()}

@item 
@ref{20f,,PyMarshal_WriteObjectToString()}

@item 
the @code{Py_MARSHAL_VERSION} macro
@end itemize

These are not part of the @ref{210,,limited API}.

Patch by Victor Stinner.

@item 
bpo-45434@footnote{https://bugs.python.org/issue45434}: Remove the @code{pystrhex.h} header file. It only contains private
functions. C extensions should only include the main @code{<Python.h>} header
file. Patch by Victor Stinner.

@item 
bpo-45440@footnote{https://bugs.python.org/issue45440}: Remove the @code{Py_FORCE_DOUBLE()} macro. It was used by the
@code{Py_IS_INFINITY()} macro. Patch by Victor Stinner.

@item 
bpo-45434@footnote{https://bugs.python.org/issue45434}: @code{<Python.h>} no longer includes the header files
@code{<stdlib.h>}, @code{<stdio.h>}, @code{<errno.h>} and @code{<string.h>} when the
@code{Py_LIMITED_API} macro is set to @code{0x030b0000} (Python 3.11) or higher.
C extensions should explicitly include the header files after @code{#include
<Python.h>}. Patch by Victor Stinner.

@item 
bpo-41123@footnote{https://bugs.python.org/issue41123}: Remove @code{Py_UNICODE_COPY()} and @code{Py_UNICODE_FILL()} macros,
deprecated since Python 3.3. Use @code{PyUnicode_CopyCharacters()} or
@code{memcpy()} (@code{wchar_t*} string), and @code{PyUnicode_Fill()} functions
instead. Patch by Victor Stinner.

@item 
bpo-45412@footnote{https://bugs.python.org/issue45412}: Remove the following math macros using the @code{errno} variable:


@itemize *

@item 
@code{Py_ADJUST_ERANGE1()}

@item 
@code{Py_ADJUST_ERANGE2()}

@item 
@code{Py_OVERFLOWED()}

@item 
@code{Py_SET_ERANGE_IF_OVERFLOW()}

@item 
@code{Py_SET_ERRNO_ON_MATH_ERROR()}
@end itemize

Patch by Victor Stinner.

@item 
bpo-45395@footnote{https://bugs.python.org/issue45395}: Custom frozen modules (the array set to
@code{PyImport_FrozenModules}) are now treated as additions, rather than
replacing all the default frozen modules. Frozen stdlib modules can still
be disabled by setting the “code” field of the custom array entry to NULL.

@item 
bpo-43760@footnote{https://bugs.python.org/issue43760}: Add new @ref{1d7,,PyThreadState_EnterTracing()}, and
@ref{1d8,,PyThreadState_LeaveTracing()} functions to the limited C API to
suspend and resume tracing and profiling. Patch by Victor Stinner.

@item 
bpo-44220@footnote{https://bugs.python.org/issue44220}: @ref{48b,,PyStructSequence_UnnamedField} is added to the Stable
ABI.
@end itemize

@node Python 3 11 0 alpha 1,Python 3 10 0 beta 1,Python 3 11 0 alpha 2,Changelog
@anchor{whatsnew/changelog python-3-11-0-alpha-1}@anchor{10f0}
@subsection Python 3.11.0 alpha 1


@emph{Release date: 2021-10-05}

@menu
* Security:: 
* Core and Builtins: Core and Builtins<6>. 
* Library: Library<6>. 
* Documentation: Documentation<7>. 
* Tests: Tests<6>. 
* Build: Build<6>. 
* Windows: Windows<6>. 
* macOS: macOS<5>. 
* IDLE: IDLE<6>. 
* Tools/Demos:: 
* C API: C API<6>. 

@end menu

@node Security,Core and Builtins<6>,,Python 3 11 0 alpha 1
@anchor{whatsnew/changelog security}@anchor{10f1}
@subsubsection Security



@itemize -

@item 
bpo-42278@footnote{https://bugs.python.org/issue42278}: Replaced usage of @ref{10f2,,tempfile.mktemp()} with
@ref{de5,,TemporaryDirectory} to avoid a potential race condition.

@item 
bpo-44600@footnote{https://bugs.python.org/issue44600}: Fix incorrect line numbers while tracing some failed patterns
in @ref{290,,match} statements. Patch by Charles Burkland.

@item 
bpo-41180@footnote{https://bugs.python.org/issue41180}: Add auditing events to the @ref{ac,,marshal} module, and stop
raising @code{code.__init__} events for every unmarshalled code object.
Directly instantiated code objects will continue to raise an event, and
audit event handlers should inspect or collect the raw marshal data. This
reduces a significant performance overhead when loading from @code{.pyc}
files.

@item 
bpo-44394@footnote{https://bugs.python.org/issue44394}: Update the vendored copy of libexpat to 2.4.1 (from 2.2.8) to
get the fix for the CVE-2013-0340 “Billion Laughs” vulnerability. This
copy is most used on Windows and macOS.

@item 
bpo-43124@footnote{https://bugs.python.org/issue43124}: Made the internal @code{putcmd} function in @ref{e8,,smtplib}
sanitize input for presence of @code{\r} and @code{\n} characters to avoid
(unlikely) command injection.

@item 
bpo-44022@footnote{https://bugs.python.org/issue44022}: @ref{8f,,http.client} now avoids infinitely reading potential HTTP
headers after a @code{100 Continue} status response from the server.
@end itemize

@node Core and Builtins<6>,Library<6>,Security,Python 3 11 0 alpha 1
@anchor{whatsnew/changelog id35}@anchor{10f3}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-43760@footnote{https://bugs.python.org/issue43760}: The number of hardware branches per instruction dispatch is
reduced from two to one by adding a special instruction for tracing. Patch
by Mark Shannon.

@item 
bpo-45061@footnote{https://bugs.python.org/issue45061}: Add a deallocator to the bool type to detect refcount bugs in C
extensions which call Py_DECREF(Py_True) or Py_DECREF(Py_False) by
mistake. Detect also refcount bugs when the empty tuple singleton or the
Unicode empty string singleton is destroyed by mistake. Patch by Victor
Stinner.

@item 
bpo-24076@footnote{https://bugs.python.org/issue24076}: sum() was further optimised for summing up single digit
integers.

@item 
bpo-45190@footnote{https://bugs.python.org/issue45190}: Update Unicode databases to Unicode 14.0.0.

@item 
bpo-45167@footnote{https://bugs.python.org/issue45167}: Fix deepcopying of @ref{26a,,types.GenericAlias} objects.

@item 
bpo-45155@footnote{https://bugs.python.org/issue45155}: @ref{10f4,,int.to_bytes()} and @ref{10a2,,int.from_bytes()} now take a
default value of @code{"big"} for the @code{byteorder} argument.
@ref{10f4,,int.to_bytes()} also takes a default value of @code{1} for the
@code{length} argument.

@item 
bpo-44219@footnote{https://bugs.python.org/issue44219}: Release the GIL while performing @code{isatty} system calls on
arbitrary file descriptors. In particular, this affects @ref{10f5,,os.isatty()},
@ref{10f6,,os.device_encoding()} and @ref{889,,io.TextIOWrapper}. By extension,
@ref{1c9,,io.open()} in text mode is also affected. This change solves a
deadlock in @ref{10f5,,os.isatty()}. Patch by Vincent Michel in bpo-44219@footnote{https://bugs.python.org/issue44219}.

@item 
bpo-44959@footnote{https://bugs.python.org/issue44959}: Added fallback to extension modules with ‘.sl’ suffix on HP-UX

@item 
bpo-45121@footnote{https://bugs.python.org/issue45121}: Fix issue where @code{Protocol.__init__} raises @code{RecursionError}
when it’s called directly or via @code{super()}. Patch provided by Yurii
Karabas.

@item 
bpo-44348@footnote{https://bugs.python.org/issue44348}: The deallocator function of the @ref{15f,,BaseException} type now
uses the trashcan mechanism to prevent stack overflow. For example, when a
@ref{8c4,,RecursionError} instance is raised, it can be linked to another
RecursionError through the @code{__context__} attribute or the
@code{__traceback__} attribute, and then a chain of exceptions is created.
When the chain is destroyed, nested deallocator function calls can crash
with a stack overflow if the chain is too long compared to the available
stack memory. Patch by Victor Stinner.

@item 
bpo-45123@footnote{https://bugs.python.org/issue45123}: Fix PyAiter_Check to only check for the __anext__ presence (not
for __aiter__). Rename PyAiter_Check to PyAIter_Check, PyObject_GetAiter
-> PyObject_GetAIter.

@item 
bpo-1514420@footnote{https://bugs.python.org/issue1514420}: Interpreter no longer attempts to open files with names in
angle brackets (like “<string>” or “<stdin>”) when formatting an
exception.

@item 
bpo-41031@footnote{https://bugs.python.org/issue41031}: Match C and Python code formatting of unprintable exceptions
and exceptions in the @ref{1,,__main__} module.

@item 
bpo-37330@footnote{https://bugs.python.org/issue37330}: @ref{1c8,,open()}, @ref{1c9,,io.open()}, @ref{1ca,,codecs.open()} and
@ref{1bc,,fileinput.FileInput} no longer accept @code{'U'} (“universal
newline”) in the file mode. This flag was deprecated since Python 3.3.
Patch by Victor Stinner.

@item 
bpo-45083@footnote{https://bugs.python.org/issue45083}: When the interpreter renders an exception, its name now has a
complete qualname. Previously only the class name was concatenated to the
module name, which sometimes resulted in an incorrect full name being
displayed.

(This issue impacted only the C code exception rendering, the
@ref{11a,,traceback} module was using qualname already).

@item 
bpo-34561@footnote{https://bugs.python.org/issue34561}: List sorting now uses the merge-ordering strategy from Munro
and Wild’s @code{powersort()}. Unlike the former strategy, this is provably
near-optimal in the entropy of the distribution of run lengths. Most uses
of @code{list.sort()} probably won’t see a significant time difference, but
may see significant improvements in cases where the former strategy was
exceptionally poor. However, as these are all fast linear-time
approximations to a problem that’s inherently at best quadratic-time to
solve truly optimally, it’s also possible to contrive cases where the
former strategy did better.

@item 
bpo-45056@footnote{https://bugs.python.org/issue45056}: Compiler now removes trailing unused constants from co_consts.

@item 
bpo-45020@footnote{https://bugs.python.org/issue45020}: Add a new command line option, “-X frozen_modules=[on|off]” to
opt out of (or into) using optional frozen modules.  This defaults to “on”
(or “off” if it’s running out of the source tree).

@item 
bpo-45012@footnote{https://bugs.python.org/issue45012}: In @ref{cc,,posix}, release GIL during @code{stat()}, @code{lstat()},
and @code{fstatat()} syscalls made by @ref{10f7,,os.DirEntry.stat()}. Patch by
Stanisław Skonieczny.

@item 
bpo-45018@footnote{https://bugs.python.org/issue45018}: Fixed pickling of range iterators that iterated for over
@code{2**32} times.

@item 
bpo-45000@footnote{https://bugs.python.org/issue45000}: A @ref{21a,,SyntaxError} is now raised when trying to delete
@ref{257,,__debug__}. Patch by Dong-hee Na.

@item 
bpo-44963@footnote{https://bugs.python.org/issue44963}: Implement @code{send()} and @code{throw()} methods for
@code{anext_awaitable} objects. Patch by Pablo Galindo.

@item 
bpo-44962@footnote{https://bugs.python.org/issue44962}: Fix a race in WeakKeyDictionary, WeakValueDictionary and
WeakSet when two threads attempt to commit the last pending removal. This
fixes asyncio.create_task and fixes a data loss in asyncio.run where
shutdown_asyncgens is not run

@item 
bpo-24234@footnote{https://bugs.python.org/issue24234}: Implement the @code{__bytes__()} special method on the
@ref{2f0,,bytes} type, so a bytes object @code{b}  passes an @code{isinstance(b,
typing.SupportsBytes)} check.

@item 
bpo-24234@footnote{https://bugs.python.org/issue24234}: Implement the @code{__complex__()} special method on the
@ref{33b,,complex} type, so a complex number @code{z} passes an @code{isinstance(z,
typing.SupportsComplex)} check.

@item 
bpo-44954@footnote{https://bugs.python.org/issue44954}: Fixed a corner case bug where the result of
@code{float.fromhex('0x.8p-1074')} was rounded the wrong way.

@item 
bpo-44947@footnote{https://bugs.python.org/issue44947}: Refine the syntax error for trailing commas in import
statements. Patch by Pablo Galindo.

@item 
bpo-44945@footnote{https://bugs.python.org/issue44945}: Specialize the BINARY_ADD instruction using the PEP 659
machinery. Adds five new instructions:


@itemize *

@item 
BINARY_ADD_ADAPTIVE

@item 
BINARY_ADD_FLOAT

@item 
BINARY_ADD_INT

@item 
BINARY_ADD_UNICODE

@item 
BINARY_ADD_UNICODE_INPLACE_FAST
@end itemize

@item 
bpo-44929@footnote{https://bugs.python.org/issue44929}: Fix some edge cases of @code{enum.Flag} string representation in
the REPL. Patch by Pablo Galindo.

@item 
bpo-44914@footnote{https://bugs.python.org/issue44914}: Class version tags are no longer recycled.

This means that a version tag serves as a unique identifier for the state
of a class. We rely on this for effective specialization of the LOAD_ATTR
and other instructions.

@item 
bpo-44698@footnote{https://bugs.python.org/issue44698}: Restore behaviour of complex exponentiation with integer-valued
exponent of type @ref{256,,float} or @ref{33b,,complex}.

@item 
bpo-44895@footnote{https://bugs.python.org/issue44895}: A debug variable 
@geindex PYTHONDUMPREFSFILE
@geindex environment variable; PYTHONDUMPREFSFILE
@code{PYTHONDUMPREFSFILE} is added for
creating a dump file which is generated by @ref{376,,--with-trace-refs}.
Patch by Dong-hee Na.

@item 
bpo-44900@footnote{https://bugs.python.org/issue44900}: Add five superinstructions for PEP 659 quickening:


@itemize *

@item 
LOAD_FAST LOAD_FAST

@item 
STORE_FAST LOAD_FAST

@item 
LOAD_FAST LOAD_CONST

@item 
LOAD_CONST LOAD_FAST

@item 
STORE_FAST STORE_FAST
@end itemize

@item 
bpo-44889@footnote{https://bugs.python.org/issue44889}: Initial implementation of adaptive specialization of
@code{LOAD_METHOD}. The following specialized forms were added:


@itemize *

@item 
@code{LOAD_METHOD_CACHED}

@item 
@code{LOAD_METHOD_MODULE}

@item 
@code{LOAD_METHOD_CLASS}
@end itemize

@item 
bpo-44890@footnote{https://bugs.python.org/issue44890}: Specialization stats are always collected in debug builds.

@item 
bpo-44885@footnote{https://bugs.python.org/issue44885}: Correct the ast locations of f-strings with format specs and
repeated expressions. Patch by Pablo Galindo

@item 
bpo-44878@footnote{https://bugs.python.org/issue44878}: Remove the loop from the bytecode interpreter. All instructions
end with a DISPATCH macro, so the loop is now redundant.

@item 
bpo-44878@footnote{https://bugs.python.org/issue44878}: Remove switch statement for interpreter loop when using
computed gotos. This makes sure that we only have one dispatch table in
the interpreter.

@item 
bpo-44874@footnote{https://bugs.python.org/issue44874}: Deprecate the old trashcan macros
(@code{Py_TRASHCAN_SAFE_BEGIN}/@code{Py_TRASHCAN_SAFE_END}). They should be
replaced by the new macros @code{Py_TRASHCAN_BEGIN} and @code{Py_TRASHCAN_END}.

@item 
bpo-44872@footnote{https://bugs.python.org/issue44872}: Use new trashcan macros (Py_TRASHCAN_BEGIN/END) in
frameobject.c instead of the old ones (Py_TRASHCAN_SAFE_BEGIN/END).

@item 
bpo-33930@footnote{https://bugs.python.org/issue33930}: Fix segmentation fault with deep recursion when cleaning method
objects. Patch by Augusto Goulart and Pablo Galindo.

@item 
bpo-25782@footnote{https://bugs.python.org/issue25782}: Fix bug where @code{PyErr_SetObject} hangs when the current
exception has a cycle in its context chain.

@item 
bpo-44856@footnote{https://bugs.python.org/issue44856}: Fix reference leaks in the error paths of @code{update_bases()}
and @code{__build_class__}. Patch by Pablo Galindo.

@item 
bpo-44826@footnote{https://bugs.python.org/issue44826}: Initial implementation of adaptive specialization of STORE_ATTR

Three specialized forms of STORE_ATTR are added:


@itemize *

@item 
STORE_ATTR_SLOT

@item 
STORE_ATTR_SPLIT_KEYS

@item 
STORE_ATTR_WITH_HINT
@end itemize

@item 
bpo-44838@footnote{https://bugs.python.org/issue44838}: Fixed a bug that was causing the parser to raise an incorrect
custom @ref{21a,,SyntaxError} for invalid ‘if’ expressions. Patch by Pablo
Galindo.

@item 
bpo-44821@footnote{https://bugs.python.org/issue44821}: Create instance dictionaries (__dict__) eagerly, to improve
regularity of object layout and assist specialization.

@item 
bpo-44792@footnote{https://bugs.python.org/issue44792}: Improve syntax errors for if expressions. Patch by Miguel Brito

@item 
bpo-34013@footnote{https://bugs.python.org/issue34013}: Generalize the invalid legacy statement custom error message
(like the one generated when “print” is called without parentheses) to
include more generic expressions. Patch by Pablo Galindo

@item 
bpo-44732@footnote{https://bugs.python.org/issue44732}: Rename @code{types.Union} to @code{types.UnionType}.

@item 
bpo-44725@footnote{https://bugs.python.org/issue44725}: Expose specialization stats in python via
@code{_opcode.get_specialization_stats()}.

@item 
bpo-44717@footnote{https://bugs.python.org/issue44717}: Improve AttributeError on circular imports of submodules.

@item 
bpo-44698@footnote{https://bugs.python.org/issue44698}: Fix undefined behaviour in complex object exponentiation.

@item 
bpo-44653@footnote{https://bugs.python.org/issue44653}: Support @ref{120,,typing} types in parameter substitution in the
union type.

@item 
bpo-44676@footnote{https://bugs.python.org/issue44676}: Add ability to serialise @code{types.Union} objects. Patch
provided by Yurii Karabas.

@item 
bpo-44633@footnote{https://bugs.python.org/issue44633}: Parameter substitution of the union type with wrong types now
raises @code{TypeError} instead of returning @code{NotImplemented}.

@item 
bpo-44661@footnote{https://bugs.python.org/issue44661}: Update @code{property_descr_set} to use vectorcall if possible.
Patch by Dong-hee Na.

@item 
bpo-44662@footnote{https://bugs.python.org/issue44662}: Add @code{__module__} to @code{types.Union}. This also fixes
@code{types.Union} issues with @code{typing.Annotated}. Patch provided by Yurii
Karabas.

@item 
bpo-44655@footnote{https://bugs.python.org/issue44655}: Include the name of the type in unset __slots__ attribute
errors. Patch by Pablo Galindo

@item 
bpo-44655@footnote{https://bugs.python.org/issue44655}: Don’t include a missing attribute with the same name as the
failing one when offering suggestions for missing attributes. Patch by
Pablo Galindo

@item 
bpo-44646@footnote{https://bugs.python.org/issue44646}: Fix the hash of the union type: it no longer depends on the
order of arguments.

@item 
bpo-44636@footnote{https://bugs.python.org/issue44636}: Collapse union of equal types. E.g. the result of @code{int | int}
is now @code{int}. Fix comparison of the union type with non-hashable
objects. E.g. @code{int | str == @{@}} no longer raises a TypeError.

@item 
bpo-44611@footnote{https://bugs.python.org/issue44611}: On Windows, @ref{17f,,os.urandom()}: uses BCryptGenRandom API
instead of CryptGenRandom API which is deprecated from Microsoft Windows
API. Patch by Dong-hee Na.

@item 
bpo-44635@footnote{https://bugs.python.org/issue44635}: Convert @code{None} to @code{type(None)} in the union type
constructor.

@item 
bpo-26280@footnote{https://bugs.python.org/issue26280}: Implement adaptive specialization for BINARY_SUBSCR

Three specialized forms of BINARY_SUBSCR are added:


@itemize *

@item 
BINARY_SUBSCR_LIST_INT

@item 
BINARY_SUBSCR_TUPLE_INT

@item 
BINARY_SUBSCR_DICT
@end itemize

@item 
bpo-44589@footnote{https://bugs.python.org/issue44589}: Mapping patterns in @code{match} statements with two or more equal
literal keys will now raise a @ref{21a,,SyntaxError} at compile-time.

@item 
bpo-44606@footnote{https://bugs.python.org/issue44606}: Fix @code{__instancecheck__} and @code{__subclasscheck__} for the
union type.

@item 
bpo-42073@footnote{https://bugs.python.org/issue42073}: The @code{@@classmethod} decorator can now wrap other
classmethod-like descriptors.

@item 
bpo-41972@footnote{https://bugs.python.org/issue41972}: Tuned the string-searching algorithm of fastsearch.h to have a
shorter inner loop for most cases.

@item 
bpo-44590@footnote{https://bugs.python.org/issue44590}: All necessary data for executing a Python function (local
variables, stack, etc) is now kept in a per-thread stack. Frame objects
are lazily allocated on demand. This increases performance by about 7% on
the standard benchmark suite. Introspection and debugging are unaffected
as frame objects are always available when needed. Patch by Mark Shannon.

@item 
bpo-44584@footnote{https://bugs.python.org/issue44584}: The threading debug (
@geindex PYTHONTHREADDEBUG
@geindex environment variable; PYTHONTHREADDEBUG
@ref{337,,PYTHONTHREADDEBUG} environment
variable) is deprecated in Python 3.10 and will be removed in Python 3.12.
This feature requires a debug build of Python. Patch by Victor Stinner.

@item 
bpo-43895@footnote{https://bugs.python.org/issue43895}: An obsolete internal cache of shared object file handles added
in 1995 that attempted, but did not guarantee, that a .so would not be
dlopen’ed twice to work around flaws in mid-1990s posix-ish operating
systems has been removed from dynload_shlib.c.

@item 
bpo-44490@footnote{https://bugs.python.org/issue44490}: @ref{120,,typing} now searches for type parameters in
@code{types.Union} objects. @code{get_type_hints} will also properly resolve
annotations with nested @code{types.Union} objects. Patch provided by Yurii
Karabas.

@item 
bpo-43950@footnote{https://bugs.python.org/issue43950}: Code objects can now provide the column information for
instructions when available. This is levaraged during traceback printing
to show the expressions responsible for errors.

Contributed by Pablo Galindo, Batuhan Taskaya and Ammar Askar as part of
@geindex Python Enhancement Proposals; PEP 657
PEP 657@footnote{https://www.python.org/dev/peps/pep-0657}.

@item 
bpo-44562@footnote{https://bugs.python.org/issue44562}: Remove uses of @ref{1039,,PyObject_GC_Del()} in error path when
initializing @ref{26a,,types.GenericAlias}.

@item 
bpo-41486@footnote{https://bugs.python.org/issue41486}: Fix a memory consumption and copying performance regression in
earlier 3.10 beta releases if someone used an output buffer larger than
4GiB with zlib.decompress on input data that expands that large.

@item 
bpo-43908@footnote{https://bugs.python.org/issue43908}: Heap types with the @ref{1ef,,Py_TPFLAGS_IMMUTABLETYPE} flag can
now inherit the 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590} vectorcall protocol.  Previously, this was only
possible for @ref{1f0,,static types}.  Patch by Erlend E.
Aasland.

@item 
bpo-44553@footnote{https://bugs.python.org/issue44553}: Implement GC methods for @code{types.Union} to break reference
cycles and prevent memory leaks.

@item 
bpo-44490@footnote{https://bugs.python.org/issue44490}: Add @code{__parameters__} attribute and @code{__getitem__} operator
to @code{types.Union}. Patch provided by Yurii Karabas.

@item 
bpo-44523@footnote{https://bugs.python.org/issue44523}: Remove the pass-through for @ref{c14,,hash()} of
@ref{567,,weakref.proxy} objects to prevent unintended consequences when the
original referred object dies while the proxy is part of a hashable
object. Patch by Pablo Galindo.

@item 
bpo-44483@footnote{https://bugs.python.org/issue44483}: Fix a crash in @code{types.Union} objects when creating a union of
an object with bad @code{__module__} field.

@item 
bpo-44486@footnote{https://bugs.python.org/issue44486}: Modules will always have a dictionary, even when created by
@code{types.ModuleType.__new__()}

@item 
bpo-44472@footnote{https://bugs.python.org/issue44472}: Fix ltrace functionality when exceptions are raised. Patch by
Pablo Galindo

@item 
bpo-12022@footnote{https://bugs.python.org/issue12022}: A @ref{162,,TypeError} is now raised instead of an
@ref{163,,AttributeError} in @ref{168,,with} and @ref{169,,async with}
statements for objects which do not support the @ref{166,,context manager} or
@ref{167,,asynchronous context manager} protocols correspondingly.

@item 
bpo-44297@footnote{https://bugs.python.org/issue44297}: Make sure that the line number is set when entering a
comprehension scope. Ensures that backtraces inclusing generator
expressions show the correct line number.

@item 
bpo-44456@footnote{https://bugs.python.org/issue44456}: Improve the syntax error when mixing positional and keyword
patterns. Patch by Pablo Galindo.

@item 
bpo-44409@footnote{https://bugs.python.org/issue44409}: Fix error location information for tokenizer errors raised on
initialization of the tokenizer. Patch by Pablo Galindo.

@item 
bpo-44396@footnote{https://bugs.python.org/issue44396}: Fix a possible crash in the tokenizer when raising syntax
errors for unclosed strings. Patch by Pablo Galindo.

@item 
bpo-44376@footnote{https://bugs.python.org/issue44376}: Exact integer exponentiation (like @code{i**2} or @code{pow(i, 2)})
with a small exponent is much faster, due to reducing overhead in such
cases.

@item 
bpo-44313@footnote{https://bugs.python.org/issue44313}: Directly imported objects and modules (through import and from
import statements) don’t generate @code{LOAD_METHOD}/@code{CALL_METHOD} for
directly accessed objects on their namespace. They now use the regular
@code{LOAD_ATTR}/@code{CALL_FUNCTION}.

@item 
bpo-44338@footnote{https://bugs.python.org/issue44338}: Implement adaptive specialization for LOAD_GLOBAL

Two specialized forms of LOAD_GLOBAL are added:


@itemize *

@item 
LOAD_GLOBAL_MODULE

@item 
LOAD_GLOBAL_BUILTIN
@end itemize

@item 
bpo-44368@footnote{https://bugs.python.org/issue44368}: Improve syntax errors for invalid “as” targets. Patch by Pablo
Galindo

@item 
bpo-44349@footnote{https://bugs.python.org/issue44349}: Fix an edge case when displaying text from files with encoding
in syntax errors. Patch by Pablo Galindo.

@item 
bpo-44337@footnote{https://bugs.python.org/issue44337}: Initial implementation of adaptive specialization of LOAD_ATTR

Four specialized forms of LOAD_ATTR are added:


@itemize *

@item 
LOAD_ATTR_SLOT

@item 
LOAD_ATTR_SPLIT_KEYS

@item 
LOAD_ATTR_WITH_HINT

@item 
LOAD_ATTR_MODULE
@end itemize

@item 
bpo-44335@footnote{https://bugs.python.org/issue44335}: Fix a regression when identifying incorrect characters in
syntax errors. Patch by Pablo Galindo

@item 
bpo-43693@footnote{https://bugs.python.org/issue43693}: Computation of the offsets of cell variables is done in the
compiler instead of at runtime. This reduces the overhead of handling cell
and free variables, especially in the case where a variable is both an
argument and cell variable.

@item 
bpo-44317@footnote{https://bugs.python.org/issue44317}: Improve tokenizer error with improved locations. Patch by Pablo
Galindo.

@item 
bpo-44304@footnote{https://bugs.python.org/issue44304}: Fix a crash in the @ref{ed,,sqlite3} module that happened when the
garbage collector clears @code{sqlite.Statement} objects. Patch by Pablo
Galindo

@item 
bpo-44305@footnote{https://bugs.python.org/issue44305}: Improve error message for @code{try} blocks without @code{except} or
@code{finally} blocks. Patch by Pablo Galindo.

@item 
bpo-43413@footnote{https://bugs.python.org/issue43413}: Constructors of subclasses of some builtin classes (e.g.
@ref{3e5,,tuple}, @ref{43a,,list}, @ref{439,,frozenset}) no longer accept
arbitrary keyword arguments. [reverted in 3.11a4] Subclass of @ref{370,,set}
can now define a @code{__new__()} method with additional keyword parameters
without overriding also @code{__init__()}.

@item 
bpo-43667@footnote{https://bugs.python.org/issue43667}: Improve Unicode support in non-UTF locales on Oracle Solaris.
This issue does not affect other Solaris systems.

@item 
bpo-43693@footnote{https://bugs.python.org/issue43693}: A new opcode MAKE_CELL has been added that effectively moves
some of the work done on function entry into the compiler and into the
eval loop.  In addition to creating the required cell objects, the new
opcode converts relevant arguments (and other locals) to cell variables on
function entry.

@item 
bpo-44232@footnote{https://bugs.python.org/issue44232}: Fix a regression in @ref{89e,,type()} when a metaclass raises an
exception. The C function @code{type_new()} must properly report the
exception when a metaclass constructor raises an exception and the winner
class is not the metaclass. Patch by Victor Stinner.

@item 
bpo-44201@footnote{https://bugs.python.org/issue44201}: Avoid side effects of checking for specialized syntax errors in
the REPL that was causing it to ask for extra tokens after a syntax error
had been detected. Patch by Pablo Galindo

@item 
bpo-43693@footnote{https://bugs.python.org/issue43693}: @code{PyCodeObject} gained @code{co_fastlocalnames} and
@code{co_fastlocalkinds} as the authoritative source of fast locals info.
Marshaled code objects have changed accordingly.

@item 
bpo-44184@footnote{https://bugs.python.org/issue44184}: Fix a crash at Python exit when a deallocator function removes
the last strong reference to a heap type. Patch by Victor Stinner.

@item 
bpo-44187@footnote{https://bugs.python.org/issue44187}: Implement quickening in the interpreter. This offers no
advantages as yet, but is an enabler of future optimizations. See PEP 659
for full explanation.

@item 
bpo-44180@footnote{https://bugs.python.org/issue44180}: The parser doesn’t report generic syntax errors that happen in
a position further away that the one it reached in the first pass. Patch
by Pablo Galindo

@item 
bpo-44168@footnote{https://bugs.python.org/issue44168}: Fix error message in the parser involving keyword arguments
with invalid expressions. Patch by Pablo Galindo

@item 
bpo-44156@footnote{https://bugs.python.org/issue44156}: String caches in @code{compile.c} are now subinterpreter
compatible.

@item 
bpo-44143@footnote{https://bugs.python.org/issue44143}: Fixed a crash in the parser that manifest when raising
tokenizer errors when an existing exception was present. Patch by Pablo
Galindo.

@item 
bpo-44032@footnote{https://bugs.python.org/issue44032}: Move ‘fast’ locals and other variables from the frame object to
a per-thread datastack.

@item 
bpo-44114@footnote{https://bugs.python.org/issue44114}: Fix incorrect dictkeys_reversed and dictitems_reversed function
signatures in C code, which broke webassembly builds.

@item 
bpo-44110@footnote{https://bugs.python.org/issue44110}: Improve @code{str.__getitem__()} error message

@item 
bpo-26110@footnote{https://bugs.python.org/issue26110}: Add @code{CALL_METHOD_KW} opcode to speed up method calls with
keyword arguments.  Idea originated from PyPy.  A side effect is executing
@code{CALL_METHOD} is now branchless in the evaluation loop.

@item 
bpo-28307@footnote{https://bugs.python.org/issue28307}: Compiler now optimizes simple C-style formatting with literal
format containing only format codes %s, %r and %a by converting them to
f-string expressions.

@item 
bpo-43149@footnote{https://bugs.python.org/issue43149}: Correct the syntax error message regarding multiple exception
types to not refer to “exception groups”. Patch by Pablo Galindo

@item 
bpo-43822@footnote{https://bugs.python.org/issue43822}: The parser will prioritize tokenizer errors over custom syntax
errors when raising exceptions. Patch by Pablo Galindo.

@item 
bpo-40222@footnote{https://bugs.python.org/issue40222}: “Zero cost” exception handling.


@itemize *

@item 
Uses a lookup table to determine how to handle exceptions.

@item 
Removes SETUP_FINALLY and POP_TOP block instructions, eliminating the runtime overhead of try statements.

@item 
Reduces the size of the frame object by about 60%.
@end itemize

Patch by Mark Shannon

@item 
bpo-43918@footnote{https://bugs.python.org/issue43918}: Document the signature and @code{default} argument in the
docstring of the new @code{anext} builtin.

@item 
bpo-43833@footnote{https://bugs.python.org/issue43833}: Emit a deprecation warning if the numeric literal is
immediately followed by one of keywords: and, else, for, if, in, is, or.
Raise a syntax error with more informative message if it is immediately
followed by other keyword or identifier.

@item 
bpo-43879@footnote{https://bugs.python.org/issue43879}: Add native_thread_id to PyThreadState. Patch by Gabriele N.
Tornetta.

@item 
bpo-43693@footnote{https://bugs.python.org/issue43693}: Compute cell offsets relative to locals in compiler. Allows the
interpreter to treats locals and cells a single array, which is slightly
more efficient. Also make the LOAD_CLOSURE opcode an alias for LOAD_FAST.
Preserving LOAD_CLOSURE helps keep bytecode a bit more readable.

@item 
bpo-17792@footnote{https://bugs.python.org/issue17792}: More accurate error messages for access of unbound locals or
free vars.

@item 
bpo-28146@footnote{https://bugs.python.org/issue28146}: Fix a confusing error message in @ref{784,,str.format()}.

@item 
bpo-11105@footnote{https://bugs.python.org/issue11105}: When compiling @ref{3cd,,ast.AST} objects with recursive
references through @ref{4e2,,compile()}, the interpreter doesn’t crash anymore
instead it raises a @ref{8c4,,RecursionError}.

@item 
bpo-39091@footnote{https://bugs.python.org/issue39091}: Fix crash when using passing a non-exception to a generator’s
@code{throw()} method. Patch by Noah Oxer

@item 
bpo-33346@footnote{https://bugs.python.org/issue33346}: Asynchronous comprehensions are now allowed inside
comprehensions in asynchronous functions.  Outer comprehensions implicitly
become asynchronous.
@end itemize

@node Library<6>,Documentation<7>,Core and Builtins<6>,Python 3 11 0 alpha 1
@anchor{whatsnew/changelog id36}@anchor{10f8}
@subsubsection Library



@itemize -

@item 
bpo-45371@footnote{https://bugs.python.org/issue45371}: Fix clang rpath issue in @ref{37,,distutils}. The UnixCCompiler
now uses correct clang option to add a runtime library directory (rpath)
to a shared library.

@item 
bpo-45329@footnote{https://bugs.python.org/issue45329}: Fix freed memory access in @code{pyexpat.xmlparser} when
building it with an installed expat library <= 2.2.0.

@item 
bpo-41710@footnote{https://bugs.python.org/issue41710}: On Unix, if the @code{sem_clockwait()} function is available in
the C library (glibc 2.30 and newer), the @ref{192,,threading.Lock.acquire()}
method now uses the monotonic clock (@ref{193,,time.CLOCK_MONOTONIC}) for the
timeout, rather than using the system clock (@ref{194,,time.CLOCK_REALTIME}),
to not be affected by system clock changes. Patch by Victor Stinner.

@item 
bpo-1596321@footnote{https://bugs.python.org/issue1596321}: Fix the @code{threading._shutdown()} function when the
@ref{109,,threading} module was imported first from a thread different than
the main thread: no longer log an error at Python exit.

@item 
bpo-45274@footnote{https://bugs.python.org/issue45274}: Fix a race condition in the @ref{d86,,Thread.join()} method of the @ref{109,,threading} module. If the
function is interrupted by a signal and the signal handler raises an
exception, make sure that the thread remains in a consistent state to
prevent a deadlock. Patch by Victor Stinner.

@item 
bpo-21302@footnote{https://bugs.python.org/issue21302}: In Unix operating systems, @ref{196,,time.sleep()} now uses the
@code{nanosleep()} function, if @code{clock_nanosleep()} is not available but
@code{nanosleep()} is available. @code{nanosleep()} allows to sleep with
nanosecond precision.

@item 
bpo-21302@footnote{https://bugs.python.org/issue21302}: On Windows, @ref{196,,time.sleep()} now uses a waitable timer which
has a resolution of 100 nanoseconds (10@w{^-7} seconds). Previously,
it had a resolution of 1 millisecond (10@w{^-3} seconds). Patch by
Benjamin Szőke and Victor Stinner.

@item 
bpo-45238@footnote{https://bugs.python.org/issue45238}: Fix @code{unittest.IsolatedAsyncioTestCase.debug()}: it runs now
asynchronous methods and callbacks.

@item 
bpo-36674@footnote{https://bugs.python.org/issue36674}: @ref{10f9,,unittest.TestCase.debug()} raises now a
@ref{b6a,,unittest.SkipTest} if the class or the test method are decorated
with the skipping decorator.

@item 
bpo-45235@footnote{https://bugs.python.org/issue45235}: Fix an issue where argparse would not preserve values in a
provided namespace when using a subparser with defaults.

@item 
bpo-45183@footnote{https://bugs.python.org/issue45183}: Have zipimport.zipimporter.find_spec() not raise an exception
when the underlying zip file has been deleted and the internal cache has
been reset via invalidate_cache().

@item 
bpo-45234@footnote{https://bugs.python.org/issue45234}: Fixed a regression in @ref{56f,,copyfile()},
@ref{570,,copy()}, @ref{571,,copy2()} raising
@ref{a37,,FileNotFoundError} when source is a directory, which should raise
@ref{bef,,IsADirectoryError}

@item 
bpo-45228@footnote{https://bugs.python.org/issue45228}: Fix stack buffer overflow in parsing J1939 network address.

@item 
bpo-45225@footnote{https://bugs.python.org/issue45225}: use map function instead of genexpr in capwords.

@item 
bpo-42135@footnote{https://bugs.python.org/issue42135}: Fix typo: @code{importlib.find_loader} is really slated for
removal in Python 3.12 not 3.10, like the others in GH-25169.

Patch by Hugo van Kemenade.

@item 
bpo-20524@footnote{https://bugs.python.org/issue20524}: Improves error messages on @code{.format()} operation for @code{str},
@code{float}, @code{int}, and @code{complex}. New format now shows the problematic
pattern and the object type.

@item 
bpo-45168@footnote{https://bugs.python.org/issue45168}: Change @ref{655,,dis.dis()} output to omit op arg values that cannot
be resolved due to @code{co_consts}, @code{co_names} etc not being provided.
Previously the oparg itself was repeated in the value field, which is not
useful and can be confusing.

@item 
bpo-21302@footnote{https://bugs.python.org/issue21302}: In Unix operating systems, @ref{196,,time.sleep()} now uses the
@code{clock_nanosleep()} function, if available, which allows to sleep for an
interval specified with nanosecond precision.

@item 
bpo-45173@footnote{https://bugs.python.org/issue45173}: Remove from the @ref{21,,configparser} module: the
@code{SafeConfigParser} class, the @code{filename} property of the
@code{ParsingError} class, the @code{readfp()} method of the
@code{ConfigParser} class, deprecated since Python 3.2.

Patch by Hugo van Kemenade.

@item 
bpo-44987@footnote{https://bugs.python.org/issue44987}: Pure ASCII strings are now normalized in constant time by
@ref{19a,,unicodedata.normalize()}. Patch by Dong-hee Na.

@item 
bpo-35474@footnote{https://bugs.python.org/issue35474}: Calling @ref{10fa,,mimetypes.guess_all_extensions()} with
@code{strict=False} no longer affects the result of the following call with
@code{strict=True}. Also, mutating the returned list no longer affects the
global state.

@item 
bpo-45166@footnote{https://bugs.python.org/issue45166}: @ref{432,,typing.get_type_hints()} now works with
@ref{556,,Final} wrapped in @ref{1090,,ForwardRef}.

@item 
bpo-45162@footnote{https://bugs.python.org/issue45162}: Remove many old deprecated @ref{122,,unittest} features:


@itemize *

@item 
“@code{fail*}” and “@code{assert*}” aliases of @ref{1ab,,TestCase} methods.

@item 
Broken from start @ref{1ab,,TestCase} method @code{assertDictContainsSubset()}.

@item 
Ignored @code{<unittest.TestLoader.loadTestsFromModule> TestLoader.loadTestsFromModule()} parameter @emph{use_load_tests}.

@item 
Old alias @code{_TextTestResult} of @ref{10bd,,TextTestResult}.
@end itemize

@item 
bpo-38371@footnote{https://bugs.python.org/issue38371}: Remove the deprecated @code{split()} method of
@code{_tkinter.TkappType}. Patch by Erlend E. Aasland.

@item 
bpo-20499@footnote{https://bugs.python.org/issue20499}: Improve the speed and accuracy of statistics.pvariance().

@item 
bpo-45132@footnote{https://bugs.python.org/issue45132}: Remove @code{__getitem__()} methods of
@ref{1ba,,xml.dom.pulldom.DOMEventStream}, @ref{1bb,,wsgiref.util.FileWrapper}
and @ref{1bc,,fileinput.FileInput}, deprecated since Python 3.9.

Patch by Hugo van Kemenade.

@item 
bpo-45129@footnote{https://bugs.python.org/issue45129}: Due to significant security concerns, the @emph{reuse_address}
parameter of @ref{1b9,,asyncio.loop.create_datagram_endpoint()}, disabled in
Python 3.9, is now entirely removed. This is because of the behavior of
the socket option @code{SO_REUSEADDR} in UDP.

Patch by Hugo van Kemenade.

@item 
bpo-45124@footnote{https://bugs.python.org/issue45124}: The @code{bdist_msi} command, deprecated in Python 3.9, is now
removed.

Use @code{bdist_wheel} (wheel packages) instead.

Patch by Hugo van Kemenade.

@item 
bpo-30856@footnote{https://bugs.python.org/issue30856}: @ref{d04,,unittest.TestResult} methods
@ref{10fb,,addFailure()},
@ref{10fc,,addError()},
@ref{10fd,,addSkip()} and
@ref{10fe,,addSubTest()} are now called immediately after
raising an exception in test or finishing a subtest. Previously they were
called only after finishing the test clean up.

@item 
bpo-45034@footnote{https://bugs.python.org/issue45034}: Changes how error is formatted for @code{struct.pack} with @code{'H'}
and @code{'h'} modes and too large / small numbers. Now it shows the actual
numeric limits, while previously it was showing arithmetic expressions.

@item 
bpo-25894@footnote{https://bugs.python.org/issue25894}: @ref{122,,unittest} now always reports skipped and failed subtests
separately: separate characters in default mode and separate lines in
verbose mode. Also the test description is now output for errors in test
method, class and module cleanups.

@item 
bpo-45081@footnote{https://bugs.python.org/issue45081}: Fix issue when dataclasses that inherit from
@code{typing.Protocol} subclasses have wrong @code{__init__}. Patch provided by
Yurii Karabas.

@item 
bpo-45085@footnote{https://bugs.python.org/issue45085}: The @code{binhex} module, deprecated in Python 3.9, is now
removed. The following @ref{f,,binascii} functions, deprecated in Python
3.9, are now also removed:


@itemize *

@item 
@code{a2b_hqx()}, @code{b2a_hqx()};

@item 
@code{rlecode_hqx()}, @code{rledecode_hqx()}.
@end itemize

The @ref{1b8,,binascii.crc_hqx()} function remains available.

Patch by Victor Stinner.

@item 
bpo-40360@footnote{https://bugs.python.org/issue40360}: The @ref{a3,,lib2to3} package is now deprecated and may not be
able to parse Python 3.10 or newer. See the 
@geindex Python Enhancement Proposals; PEP 617
PEP 617@footnote{https://www.python.org/dev/peps/pep-0617} (New PEG parser for
CPython). Patch by Victor Stinner.

@item 
bpo-45075@footnote{https://bugs.python.org/issue45075}: Rename @code{traceback.StackSummary.format_frame()} to
@ref{10ff,,traceback.StackSummary.format_frame_summary()}. This method was added
for 3.11 so it was not released yet.

Updated code and docs to better distinguish frame and FrameSummary.

@item 
bpo-31299@footnote{https://bugs.python.org/issue31299}: Add option to completely drop frames from a traceback by
returning @code{None} from a @code{format_frame()}
override.

@item 
bpo-41620@footnote{https://bugs.python.org/issue41620}: @ref{d03,,run()} now always return a
@ref{d04,,TestResult} instance. Previously it returned @code{None} if
the test class or method was decorated with a skipping decorator.

@item 
bpo-45021@footnote{https://bugs.python.org/issue45021}: Fix a potential deadlock at shutdown of forked children when
using @ref{20,,concurrent.futures} module

@item 
bpo-43913@footnote{https://bugs.python.org/issue43913}: Fix bugs in cleaning up classes and modules in @ref{122,,unittest}:


@itemize *

@item 
Functions registered with @ref{562,,addModuleCleanup()} were not called unless the user defines @code{tearDownModule()} in their test module.

@item 
Functions registered with @ref{563,,addClassCleanup()} were not called if @code{tearDownClass} is set to @code{None}.

@item 
Buffering in @ref{d04,,TestResult} did not work with functions registered with @code{addClassCleanup()} and @code{addModuleCleanup()}.

@item 
Errors in functions registered with @code{addClassCleanup()} and @code{addModuleCleanup()} were not handled correctly in buffered and debug modes.

@item 
Errors in @code{setUpModule()} and functions registered with @code{addModuleCleanup()} were reported in wrong order.

@item 
And several lesser bugs.
@end itemize

@item 
bpo-45030@footnote{https://bugs.python.org/issue45030}: Fix integer overflow in pickling and copying the range
iterator.

@item 
bpo-45001@footnote{https://bugs.python.org/issue45001}: Made email date parsing more robust against malformed input,
namely a whitespace-only @code{Date:} header. Patch by Wouter Bolsterlee.

@item 
bpo-45010@footnote{https://bugs.python.org/issue45010}: Remove support of special method @code{__div__} in
@ref{123,,unittest.mock}. It is not used in Python 3.

@item 
bpo-39218@footnote{https://bugs.python.org/issue39218}: Improve accuracy of variance calculations by using @code{x*x}
instead of @code{x**2}.

@item 
bpo-43613@footnote{https://bugs.python.org/issue43613}: Improve the speed of @ref{4fc,,gzip.compress()} and
@ref{dae,,gzip.decompress()} by compressing and decompressing at once in memory
instead of in a streamed fashion.

@item 
bpo-37596@footnote{https://bugs.python.org/issue37596}: Ensure that @ref{370,,set} and @ref{439,,frozenset} objects are
always @ref{ac,,marshalled} reproducibly.

@item 
bpo-44019@footnote{https://bugs.python.org/issue44019}: A new function @code{operator.call} has been added, such that
@code{operator.call(obj, *args, **kwargs) == obj(*args, **kwargs)}.

@item 
bpo-42255@footnote{https://bugs.python.org/issue42255}: @code{webbrowser.MacOSX} is deprecated and will be removed in
Python 3.13. It is untested and undocumented and also not used by
webbrowser itself. Patch by Dong-hee Na.

@item 
bpo-44955@footnote{https://bugs.python.org/issue44955}: Method @ref{ee6,,stopTestRun()} is now always
called in pair with method @ref{ee5,,startTestRun()} for
@ref{d04,,TestResult} objects implicitly created in
@ref{d03,,run()}. Previously it was not called for test
methods and classes decorated with a skipping decorator.

@item 
bpo-39039@footnote{https://bugs.python.org/issue39039}: tarfile.open raises @ref{1100,,ReadError} when a zlib error
occurs during file extraction.

@item 
bpo-44935@footnote{https://bugs.python.org/issue44935}: @ref{f4,,subprocess} on Solaris now also uses
@ref{56e,,os.posix_spawn()} for better performance.

@item 
bpo-44911@footnote{https://bugs.python.org/issue44911}: @ref{1ac,,IsolatedAsyncioTestCase} will no longer throw
an exception while cancelling leaked tasks. Patch by Bar Harel.

@item 
bpo-41322@footnote{https://bugs.python.org/issue41322}: Added @code{DeprecationWarning} for tests and async tests that
return a value!=None (as this may indicate an improperly written test, for
example a test written as a generator function).

@item 
bpo-44524@footnote{https://bugs.python.org/issue44524}: Make exception message more useful when subclass from typing
special form alias. Patch provided by Yurii Karabas.

@item 
bpo-38956@footnote{https://bugs.python.org/issue38956}: @code{argparse.BooleanOptionalAction}’s default value is no
longer printed twice when used with
@ref{1101,,argparse.ArgumentDefaultsHelpFormatter}.

@item 
bpo-44860@footnote{https://bugs.python.org/issue44860}: Fix the @code{posix_user} scheme in @ref{f8,,sysconfig} to not depend
on @ref{42c,,sys.platlibdir}.

@item 
bpo-44859@footnote{https://bugs.python.org/issue44859}: Improve error handling in @ref{ed,,sqlite3} and raise more
accurate exceptions.


@itemize *

@item 
@ref{10de,,MemoryError} is now raised instead of @ref{1102,,sqlite3.Warning} when memory is not enough for encoding a statement to UTF-8 in @code{Connection.__call__()} and @code{Cursor.execute()}.

@item 
@code{UnicodEncodeError} is now raised instead of @ref{1102,,sqlite3.Warning} when the statement contains surrogate characters in @code{Connection.__call__()} and @code{Cursor.execute()}.

@item 
@ref{162,,TypeError} is now raised instead of @ref{17c,,ValueError} for non-string script argument in @code{Cursor.executescript()}.

@item 
@ref{17c,,ValueError} is now raised for script containing the null character instead of truncating it in @code{Cursor.executescript()}.

@item 
Correctly handle exceptions raised when getting boolean value of the result of the progress handler.

@item 
Add many tests covering different corner cases.
@end itemize

@item 
bpo-44581@footnote{https://bugs.python.org/issue44581}: Upgrade bundled pip to 21.2.3 and setuptools to 57.4.0

@item 
bpo-44849@footnote{https://bugs.python.org/issue44849}: Fix the @ref{a70,,os.set_inheritable()} function on FreeBSD 14 for
file descriptor opened with the @ref{af3,,O_PATH} flag: ignore the
@ref{45e,,EBADF} error on @code{ioctl()}, fallback on the @code{fcntl()}
implementation. Patch by Victor Stinner.

@item 
bpo-44605@footnote{https://bugs.python.org/issue44605}: The @@functools.total_ordering() decorator now works with
metaclasses.

@item 
bpo-44524@footnote{https://bugs.python.org/issue44524}: Fixed an issue wherein the @code{__name__} and @code{__qualname__}
attributes of subscribed specialforms could be @code{None}.

@item 
bpo-44839@footnote{https://bugs.python.org/issue44839}: @ref{10de,,MemoryError} raised in user-defined functions will now
produce a @code{MemoryError} in @ref{ed,,sqlite3}. @ref{347,,OverflowError} will
now be converted to @code{DataError}. Previously
@ref{1103,,OperationalError} was produced in these cases.

@item 
bpo-44822@footnote{https://bugs.python.org/issue44822}: @ref{ed,,sqlite3} user-defined functions and aggregators returning
@ref{2ef,,strings} with embedded NUL characters are no longer
truncated. Patch by Erlend E. Aasland.

@item 
bpo-44801@footnote{https://bugs.python.org/issue44801}: Ensure that the @ref{23a,,ParamSpec} variable in Callable
can only be substituted with a parameters expression (a list of types, an
ellipsis, ParamSpec or Concatenate).

@item 
bpo-44806@footnote{https://bugs.python.org/issue44806}: Non-protocol subclasses of @ref{558,,typing.Protocol} ignore now
the @code{__init__} method inherited from protocol base classes.

@item 
bpo-27275@footnote{https://bugs.python.org/issue27275}: @ref{e8b,,collections.OrderedDict.popitem()} and
@code{collections.OrderedDict.pop()} no longer call @code{__getitem__} and
@code{__delitem__} methods of the OrderedDict subclasses.

@item 
bpo-44793@footnote{https://bugs.python.org/issue44793}: Fix checking the number of arguments when subscribe a generic
type with @code{ParamSpec} parameter.

@item 
bpo-44784@footnote{https://bugs.python.org/issue44784}: In importlib.metadata tests, override warnings behavior under
expected DeprecationWarnings (importlib_metadata 4.6.3).

@item 
bpo-44667@footnote{https://bugs.python.org/issue44667}: The @ref{e67,,tokenize.tokenize()} doesn’t incorrectly generate a
@code{NEWLINE} token if the source doesn’t end with a new line character but
the last line is a comment, as the function is already generating a @code{NL}
token. Patch by Pablo Galindo

@item 
bpo-44771@footnote{https://bugs.python.org/issue44771}: Added @code{importlib.simple} module implementing adapters from a
low-level resources reader interface to a @code{TraversableResources}
interface. Legacy API (@code{path}, @code{contents}, …) is now supported
entirely by the @code{.files()} API with a compatibility shim supplied for
resource loaders without that functionality. Feature parity with
@code{importlib_resources} 5.2.

@item 
bpo-44752@footnote{https://bugs.python.org/issue44752}: @code{rcompleter} does not call @ref{708,,getattr()} on
@ref{501,,property} objects to avoid the side-effect of  evaluating the
corresponding method.

@item 
bpo-44747@footnote{https://bugs.python.org/issue44747}: Refactor usage of @code{sys._getframe} in @code{typing} module. Patch
provided by Yurii Karabas.

@item 
bpo-42378@footnote{https://bugs.python.org/issue42378}: Fixes the issue with log file being overwritten when
@ref{1104,,logging.FileHandler} is used in @ref{b,,atexit} with @emph{filemode} set
to @code{'w'}. Note this will cause the message in @emph{atexit} not being logged
if the log stream is already closed due to shutdown of logging.

@item 
bpo-44720@footnote{https://bugs.python.org/issue44720}: @code{weakref.proxy} objects referencing non-iterators now raise
@code{TypeError} rather than dereferencing the null @code{tp_iternext} slot and
crashing.

@item 
bpo-44704@footnote{https://bugs.python.org/issue44704}: The implementation of @code{collections.abc.Set._hash()} now
matches that of @code{frozenset.__hash__()}.

@item 
bpo-44666@footnote{https://bugs.python.org/issue44666}: Fixed issue in @ref{940,,compileall.compile_file()} when
@code{sys.stdout} is redirected. Patch by Stefan Hölzl.

@item 
bpo-44688@footnote{https://bugs.python.org/issue44688}: @ref{184,,sqlite3.Connection.create_collation()} now accepts
non-ASCII collation names. Patch by Erlend E. Aasland.

@item 
bpo-44690@footnote{https://bugs.python.org/issue44690}: Adopt @emph{binacii.a2b_base64}’s strict mode in @emph{base64.b64decode}.

@item 
bpo-42854@footnote{https://bugs.python.org/issue42854}: Fixed a bug in the @code{_ssl} module that was throwing
@ref{347,,OverflowError} when using @code{_ssl._SSLSocket.write()} and
@code{_ssl._SSLSocket.read()} for a big value of the @code{len} parameter.
Patch by Pablo Galindo

@item 
bpo-44686@footnote{https://bugs.python.org/issue44686}: Replace @code{unittest.mock._importer} with
@code{pkgutil.resolve_name}.

@item 
bpo-44353@footnote{https://bugs.python.org/issue44353}: Make @code{NewType.__call__} faster by implementing it in C. Patch
provided by Yurii Karabas.

@item 
bpo-44682@footnote{https://bugs.python.org/issue44682}: Change the @ref{c4,,pdb} @emph{commands} directive to disallow setting
commands for an invalid breakpoint and to display an appropriate error.

@item 
bpo-44353@footnote{https://bugs.python.org/issue44353}: Refactor @code{typing.NewType} from function into callable class.
Patch provided by Yurii Karabas.

@item 
bpo-44678@footnote{https://bugs.python.org/issue44678}: Added a separate error message for discontinuous padding in
@emph{binascii.a2b_base64} strict mode.

@item 
bpo-44524@footnote{https://bugs.python.org/issue44524}: Add missing @code{__name__} and @code{__qualname__} attributes to
@code{typing} module classes. Patch provided by Yurii Karabas.

@item 
bpo-40897@footnote{https://bugs.python.org/issue40897}: Give priority to using the current class constructor in
@ref{1c1,,inspect.signature()}. Patch by Weipeng Hong.

@item 
bpo-44638@footnote{https://bugs.python.org/issue44638}: Add a reference to the zipp project and hint as to how to use
it.

@item 
bpo-44648@footnote{https://bugs.python.org/issue44648}: Fixed wrong error being thrown by @ref{1105,,inspect.getsource()}
when examining a class in the interactive session. Instead of
@ref{162,,TypeError}, it should be @ref{2a1,,OSError} with appropriate error
message.

@item 
bpo-44608@footnote{https://bugs.python.org/issue44608}: Fix memory leak in @code{_tkinter._flatten()} if it is called
with a sequence or set, but not list or tuple.

@item 
bpo-44594@footnote{https://bugs.python.org/issue44594}: Fix an edge case of @code{ExitStack} and
@code{AsyncExitStack} exception chaining.  They will now match @code{with}
block behavior when @code{__context__} is explicitly set to @code{None} when the
exception is in flight.

@item 
bpo-42799@footnote{https://bugs.python.org/issue42799}: In @ref{7c,,fnmatch}, the cache size for compiled regex patterns
(@ref{4f3,,functools.lru_cache()}) was bumped up from 256 to 32768, affecting
functions: @ref{1106,,fnmatch.fnmatch()}, @ref{1107,,fnmatch.fnmatchcase()},
@ref{1108,,fnmatch.filter()}.

@item 
bpo-41928@footnote{https://bugs.python.org/issue41928}: Update @ref{56f,,shutil.copyfile()} to raise
@ref{a37,,FileNotFoundError} instead of confusing @ref{bef,,IsADirectoryError}
when a path ending with a @code{os.path.sep} does not exist;
@ref{570,,shutil.copy()} and @ref{571,,shutil.copy2()} are also affected.

@item 
bpo-44569@footnote{https://bugs.python.org/issue44569}: Added the @code{StackSummary.format_frame()} function in
@ref{11a,,traceback}. This allows users to customize the way individual lines
are formatted in tracebacks without re-implementing logic to handle
recursive tracebacks.

@item 
bpo-44566@footnote{https://bugs.python.org/issue44566}: handle StopIteration subclass raised from
@@contextlib.contextmanager generator

@item 
bpo-44558@footnote{https://bugs.python.org/issue44558}: Make the implementation consistency of
@ref{3c1,,indexOf()} between C and Python versions. Patch by Dong-hee
Na.

@item 
bpo-41249@footnote{https://bugs.python.org/issue41249}: Fixes @code{TypedDict} to work with @code{typing.get_type_hints()}
and postponed evaluation of annotations across modules.

@item 
bpo-44554@footnote{https://bugs.python.org/issue44554}: Refactor argument processing in @code{pdb.main()} to simplify
detection of errors in input loading and clarify behavior around module or
script invocation.

@item 
bpo-34798@footnote{https://bugs.python.org/issue34798}: Break up paragraph about @ref{b00,,pprint.PrettyPrinter}
construction parameters to make it easier to read.

@item 
bpo-44539@footnote{https://bugs.python.org/issue44539}: Added support for recognizing JPEG files without JFIF or Exif
markers.

@item 
bpo-44461@footnote{https://bugs.python.org/issue44461}: Fix bug with @ref{c4,,pdb}’s handling of import error due to a
package which does not have a @code{__main__} module

@item 
bpo-43625@footnote{https://bugs.python.org/issue43625}: Fix a bug in the detection of CSV file headers by
@ref{1109,,csv.Sniffer.has_header()} and improve documentation of same.

@item 
bpo-44516@footnote{https://bugs.python.org/issue44516}: Update vendored pip to 21.1.3

@item 
bpo-42892@footnote{https://bugs.python.org/issue42892}: Fixed an exception thrown while parsing a malformed multipart
email by @ref{7e4,,email.message.EmailMessage}.

@item 
bpo-44468@footnote{https://bugs.python.org/issue44468}: @ref{432,,typing.get_type_hints()} now finds annotations in classes
and base classes with unexpected @code{__module__}. Previously, it skipped
those MRO elements.

@item 
bpo-44491@footnote{https://bugs.python.org/issue44491}: Allow clearing the @ref{ed,,sqlite3} authorizer callback by
passing @ref{182,,None} to @ref{183,,set_authorizer()}. Patch
by Erlend E. Aasland.

@item 
bpo-43977@footnote{https://bugs.python.org/issue43977}: Set the proper @ref{110a,,Py_TPFLAGS_MAPPING} and
@ref{110b,,Py_TPFLAGS_SEQUENCE} flags for subclasses created before a parent
has been registered as a @ref{3ab,,collections.abc.Mapping} or
@ref{110c,,collections.abc.Sequence}.

@item 
bpo-44482@footnote{https://bugs.python.org/issue44482}: Fix very unlikely resource leak in @ref{84,,glob} in alternate
Python implementations.

@item 
bpo-44466@footnote{https://bugs.python.org/issue44466}: The @ref{78,,faulthandler} module now detects if a fatal error
occurs during a garbage collector collection. Patch by Victor Stinner.

@item 
bpo-44471@footnote{https://bugs.python.org/issue44471}: A @ref{162,,TypeError} is now raised instead of an
@ref{163,,AttributeError} in @ref{164,,contextlib.ExitStack.enter_context()} and
@ref{165,,contextlib.AsyncExitStack.enter_async_context()} for objects which do
not support the @ref{166,,context manager} or @ref{167,,asynchronous context manager} protocols correspondingly.

@item 
bpo-44404@footnote{https://bugs.python.org/issue44404}: @ref{10c,,tkinter}’s @code{after()} method now supports callables
without the @code{__name__} attribute.

@item 
bpo-41546@footnote{https://bugs.python.org/issue41546}: Make @ref{cd,,pprint} (like the builtin @code{print}) not attempt to
write to @code{stdout} when it is @code{None}.

@item 
bpo-44458@footnote{https://bugs.python.org/issue44458}: @code{BUFFER_BLOCK_SIZE} is now declared static, to avoid linking
collisions when bz2, lmza or zlib are statically linked.

@item 
bpo-44464@footnote{https://bugs.python.org/issue44464}: Remove exception for flake8 in deprecated importlib.metadata
interfaces. Sync with importlib_metadata 4.6.

@item 
bpo-44446@footnote{https://bugs.python.org/issue44446}: Take into account that @code{lineno} might be @code{None} in
@ref{9f6,,traceback.FrameSummary}.

@item 
bpo-44439@footnote{https://bugs.python.org/issue44439}: Fix in @code{bz2.BZ2File.write()} / @code{lzma.LZMAFile.write()}
methods, when the input data is an object that supports the buffer
protocol, the file length may be wrong.

@item 
bpo-44434@footnote{https://bugs.python.org/issue44434}: _thread.start_new_thread() no longer calls
PyThread_exit_thread() explicitly at the thread exit, the call was
redundant. On Linux with the glibc, pthread_exit() aborts the whole
process if dlopen() fails to open libgcc_s.so file (ex: EMFILE error).
Patch by Victor Stinner.

@item 
bpo-42972@footnote{https://bugs.python.org/issue42972}: The _thread.RLock type now fully implement the GC protocol: add
a traverse function and the @ref{1ed,,Py_TPFLAGS_HAVE_GC} flag. Patch by
Victor Stinner.

@item 
bpo-44422@footnote{https://bugs.python.org/issue44422}: The @ref{110d,,threading.enumerate()} function now uses a reentrant
lock to prevent a hang on reentrant call. Patch by Victor Stinner.

@item 
bpo-38291@footnote{https://bugs.python.org/issue38291}: Importing typing.io or typing.re now prints a
@code{DeprecationWarning}.

@item 
bpo-37880@footnote{https://bugs.python.org/issue37880}: argparse actions store_const and append_const each receive a
default value of None when the @code{const} kwarg is not provided.
Previously, this raised a @ref{162,,TypeError}.

@item 
bpo-44389@footnote{https://bugs.python.org/issue44389}: Fix deprecation of @ref{327,,ssl.OP_NO_TLSv1_3}

@item 
bpo-27827@footnote{https://bugs.python.org/issue27827}: @code{pathlib.PureWindowsPath.is_reserved()} now identifies a
greater range of reserved filenames, including those with trailing spaces
or colons.

@item 
bpo-44395@footnote{https://bugs.python.org/issue44395}: Fix @code{as_string()} to pass unixfrom
properly. Patch by Dong-hee Na.

@item 
bpo-34266@footnote{https://bugs.python.org/issue34266}: Handle exceptions from parsing the arg of @ref{c4,,pdb}’s
run/restart command.

@item 
bpo-44362@footnote{https://bugs.python.org/issue44362}: Improve @ref{ee,,ssl} module’s deprecation messages, error
reporting, and documentation for deprecations.

@item 
bpo-44342@footnote{https://bugs.python.org/issue44342}: [Enum] Change pickling from by-value to by-name.

@item 
bpo-44356@footnote{https://bugs.python.org/issue44356}: [Enum] Allow multiple data-type mixins if they are all the
same.

@item 
bpo-44351@footnote{https://bugs.python.org/issue44351}: Restore back @code{parse_makefile()} in
@ref{5e,,distutils.sysconfig} because it behaves differently than the similar
implementation in @ref{f8,,sysconfig}.

@item 
bpo-35800@footnote{https://bugs.python.org/issue35800}: @code{smtpd.MailmanProxy} is now removed as it is unusable
without an external module, @code{mailman}. Patch by Dong-hee Na.

@item 
bpo-44357@footnote{https://bugs.python.org/issue44357}: Added a function that returns cube root of the given number
@ref{17a,,math.cbrt()}

@item 
bpo-44339@footnote{https://bugs.python.org/issue44339}: Change @code{math.pow(±0.0, -math.inf)} to return @code{inf} instead
of raising @code{ValueError}. This brings the special-case handling of
@code{math.pow} into compliance with the IEEE 754 standard.

@item 
bpo-44242@footnote{https://bugs.python.org/issue44242}: Remove missing flag check from Enum creation and move into a
@code{verify} decorator.

@item 
bpo-44246@footnote{https://bugs.python.org/issue44246}: In @code{importlib.metadata}, restore compatibility in the result
from @code{Distribution.entry_points} (@code{EntryPoints}) to honor expectations
in older implementations and issuing deprecation warnings for these cases:
A. @code{EntryPoints} objects are once again mutable, allowing   for
@code{sort()} and other list-based mutation operations.   Avoid deprecation
warnings by casting to a   mutable sequence (e.g.
@code{list(dist.entry_points).sort()}). B. @code{EntryPoints} results once again
allow   for access by index. To avoid deprecation warnings,   cast the
result to a Sequence first   (e.g. @code{tuple(dist.entry_points)[0]}).

@item 
bpo-44246@footnote{https://bugs.python.org/issue44246}: In importlib.metadata.entry_points, de-duplication of
distributions no longer requires loading the full metadata for
PathDistribution objects, improving entry point loading performance by
~10x.

@item 
bpo-43858@footnote{https://bugs.python.org/issue43858}: Added a function that returns a copy of a dict of logging
levels: @ref{110e,,logging.getLevelNamesMapping()}

@item 
bpo-44260@footnote{https://bugs.python.org/issue44260}: The @ref{110f,,random.Random} constructor no longer reads system
entropy without need.

@item 
bpo-44254@footnote{https://bugs.python.org/issue44254}: On Mac, give turtledemo button text a color that works on both
light or dark background.  Programmers cannot control the latter.

@item 
bpo-44258@footnote{https://bugs.python.org/issue44258}: Support PEP 515 for Fraction’s initialization from string.

@item 
bpo-44235@footnote{https://bugs.python.org/issue44235}: Remove deprecated functions in the @ref{83,,gettext}. Patch by
Dong-hee Na.

@item 
bpo-38693@footnote{https://bugs.python.org/issue38693}: Prefer f-strings to @code{.format} in importlib.resources.

@item 
bpo-33693@footnote{https://bugs.python.org/issue33693}: Importlib.metadata now prefers f-strings to .format.

@item 
bpo-44241@footnote{https://bugs.python.org/issue44241}: Incorporate minor tweaks from importlib_metadata 4.1:
SimplePath protocol, support for Metadata 2.2.

@item 
bpo-43216@footnote{https://bugs.python.org/issue43216}: Remove the @code{@@asyncio.coroutine}
@ref{1bf,,decorator} enabling legacy generator-based coroutines to be
compatible with async/await code; remove
@code{asyncio.coroutines.CoroWrapper} used for wrapping legacy coroutine
objects in the debug mode. The decorator has been deprecated since Python
3.8 and the removal was initially scheduled for Python 3.10. Patch by
Illia Volochii.

@item 
bpo-44210@footnote{https://bugs.python.org/issue44210}: Make importlib.metadata._meta.PackageMetadata public.

@item 
bpo-43643@footnote{https://bugs.python.org/issue43643}: Declare readers.MultiplexedPath.name as a property per the
spec.

@item 
bpo-27334@footnote{https://bugs.python.org/issue27334}: The @ref{ed,,sqlite3} context manager now performs a rollback
(thus releasing the database lock) if commit failed.  Patch by Luca Citi
and Erlend E. Aasland.

@item 
bpo-4928@footnote{https://bugs.python.org/issue4928}: Documented existing behavior on POSIX: NamedTemporaryFiles are
not deleted when creating process is killed with SIGKILL

@item 
bpo-44154@footnote{https://bugs.python.org/issue44154}: Optimize @ref{172,,fractions.Fraction} pickling for large
components.

@item 
bpo-33433@footnote{https://bugs.python.org/issue33433}: For IPv4 mapped IPv6 addresses (
@geindex RFC; RFC 4291
RFC 4291@footnote{https://tools.ietf.org/html/rfc4291.html} Section 2.5.5.2),
the @ref{1110,,ipaddress.IPv6Address.is_private} check is deferred to the
mapped IPv4 address. This solves a bug where public mapped IPv4 addresses
were considered private by the IPv6 check.

@item 
bpo-44150@footnote{https://bugs.python.org/issue44150}: Add optional @emph{weights} argument to statistics.fmean().

@item 
bpo-44142@footnote{https://bugs.python.org/issue44142}: @ref{3cc,,ast.unparse()} will now drop the redundant parentheses
when tuples used as assignment targets (e.g in for loops).

@item 
bpo-44145@footnote{https://bugs.python.org/issue44145}: @ref{8a,,hmac} computations were not releasing the GIL while
calling the OpenSSL @code{HMAC_Update} C API (a new feature in 3.9).  This
unintentionally prevented parallel computation as other @ref{88,,hashlib}
algorithms support.

@item 
bpo-44095@footnote{https://bugs.python.org/issue44095}: @ref{1111,,zipfile.Path} now supports @ref{1112,,zipfile.Path.stem},
@ref{1113,,zipfile.Path.suffixes}, and @ref{1114,,zipfile.Path.suffix} attributes.

@item 
bpo-44077@footnote{https://bugs.python.org/issue44077}: It’s now possible to receive the type of service (ToS), a.k.a.
differentiated services (DS), a.k.a. differentiated services code point
(DSCP) and explicit congestion notification (ECN) IP header fields with
@code{socket.IP_RECVTOS}.

@item 
bpo-37788@footnote{https://bugs.python.org/issue37788}: Fix a reference leak when a Thread object is never joined.

@item 
bpo-38908@footnote{https://bugs.python.org/issue38908}: Subclasses of @code{typing.Protocol} which only have data
variables declared will now raise a @code{TypeError} when checked with
@code{isinstance} unless they are decorated with @code{runtime_checkable()}.
Previously, these checks passed silently. Patch provided by Yurii Karabas.

@item 
bpo-44098@footnote{https://bugs.python.org/issue44098}: @code{typing.ParamSpec} will no longer be found in the
@code{__parameters__} of most @ref{120,,typing} generics except in valid use
locations specified by 
@geindex Python Enhancement Proposals; PEP 612
PEP 612@footnote{https://www.python.org/dev/peps/pep-0612}. This prevents incorrect usage like
@code{typing.List[P][int]}. This change means incorrect usage which may have
passed silently in 3.10 beta 1 and earlier will now error.

@item 
bpo-44089@footnote{https://bugs.python.org/issue44089}: Allow subclassing @code{csv.Error} in 3.10 (it was allowed in 3.9
and earlier but was disallowed in early versions of 3.10).

@item 
bpo-44081@footnote{https://bugs.python.org/issue44081}: @ref{3cc,,ast.unparse()} now doesn’t use redundant spaces to
separate @code{lambda} and the @code{:} if there are no parameters.

@item 
bpo-44061@footnote{https://bugs.python.org/issue44061}: Fix regression in previous release when calling
@ref{8a1,,pkgutil.iter_modules()} with a list of @ref{522,,pathlib.Path} objects

@item 
bpo-44059@footnote{https://bugs.python.org/issue44059}: Register the SerenityOS Browser in the @ref{130,,webbrowser}
module.

@item 
bpo-36515@footnote{https://bugs.python.org/issue36515}: The @ref{88,,hashlib} module no longer does unaligned memory
accesses when compiled for ARM platforms.

@item 
bpo-40465@footnote{https://bugs.python.org/issue40465}: Remove random module features deprecated in Python 3.9.

@item 
bpo-44018@footnote{https://bugs.python.org/issue44018}: random.seed() no longer mutates bytearray inputs.

@item 
bpo-38352@footnote{https://bugs.python.org/issue38352}: Add @code{IO}, @code{BinaryIO}, @code{TextIO}, @code{Match}, and
@code{Pattern} to @code{typing.__all__}. Patch by Jelle Zijlstra.

@item 
bpo-44002@footnote{https://bugs.python.org/issue44002}: @ref{126,,urllib.parse} now uses @code{functool.lru_cache()} for its
internal URL splitting and quoting caches instead of rolling its own like
its the ’90s.

The undocumented internal @ref{126,,urllib.parse} @code{Quoted} class API is now
deprecated, for removal in 3.14.

@item 
bpo-43972@footnote{https://bugs.python.org/issue43972}: When @ref{666,,http.server.SimpleHTTPRequestHandler} sends a
@code{301 (Moved Permanently)} for a directory path not ending with @code{/}, add
a @code{Content-Length: 0} header. This improves the behavior for certain
clients.

@item 
bpo-28528@footnote{https://bugs.python.org/issue28528}: Fix a bug in @ref{c4,,pdb} where @code{checkline()} raises
@ref{163,,AttributeError} if it is called after @code{reset()}.

@item 
bpo-43853@footnote{https://bugs.python.org/issue43853}: Improved string handling for @ref{ed,,sqlite3} user-defined
functions and aggregates:


@itemize *

@item 
It is now possible to pass strings with embedded null characters to UDFs

@item 
Conversion failures now correctly raise @ref{10de,,MemoryError}
@end itemize

Patch by Erlend E. Aasland.

@item 
bpo-43666@footnote{https://bugs.python.org/issue43666}: AIX: @code{Lib/_aix_support.get_platform()} may fail in an AIX WPAR.
The fileset bos.rte appears to have a builddate in both LPAR and WPAR so
this fileset is queried rather than bos.mp64. To prevent a similar
situation (no builddate in ODM) a value (9988) sufficient for completing a
build is provided. Patch by M Felt.

@item 
bpo-43650@footnote{https://bugs.python.org/issue43650}: Fix @ref{10de,,MemoryError} in @ref{dbb,,shutil.unpack_archive()} which
fails inside @code{shutil._unpack_zipfile()} on large files. Patch by Igor
Bolshakov.

@item 
bpo-43612@footnote{https://bugs.python.org/issue43612}: @ref{859,,zlib.compress()} now accepts a wbits parameter which
allows users to compress data as a raw deflate block without zlib headers
and trailers in one go. Previously this required instantiating a
@code{zlib.compressobj}. It also provides a faster alternative to
@code{gzip.compress} when wbits=31 is used.

@item 
bpo-43392@footnote{https://bugs.python.org/issue43392}: @code{importlib._bootstrap._find_and_load()} now implements a
two-step check to avoid locking when modules have been already imported
and are ready. This improves performance of repeated calls to
@ref{d3e,,importlib.import_module()} and @ref{c02,,importlib.__import__()}.

@item 
bpo-43318@footnote{https://bugs.python.org/issue43318}: Fix a bug where @ref{c4,,pdb} does not always echo cleared
breakpoints.

@item 
bpo-43234@footnote{https://bugs.python.org/issue43234}: Prohibit passing
non-@ref{1c7,,concurrent.futures.ThreadPoolExecutor} executors to
@code{loop.set_default_executor()} following a deprecation in Python 3.8.
Patch by Illia Volochii.

@item 
bpo-43232@footnote{https://bugs.python.org/issue43232}: Prohibit previously deprecated potentially disruptive
operations on @code{asyncio.trsock.TransportSocket}. Patch by Illia
Volochii.

@item 
bpo-30077@footnote{https://bugs.python.org/issue30077}: Added support for Apple’s aifc/sowt pseudo-compression

@item 
bpo-42971@footnote{https://bugs.python.org/issue42971}: Add definition of @code{errno.EQFULL} for platforms that define
this constant (such as macOS).

@item 
bpo-43086@footnote{https://bugs.python.org/issue43086}: Added a new optional @code{strict_mode} parameter to
@emph{binascii.a2b_base64}. When @code{scrict_mode} is set to @code{True},
the @emph{a2b_base64} function will accept only valid base64 content. More
details about what “valid base64 content” is, can be found in the
function’s documentation.

@item 
bpo-43024@footnote{https://bugs.python.org/issue43024}: Improve the help signature of
@ref{2d5,,traceback.print_exception()}, @ref{2d3,,traceback.format_exception()} and
@ref{2d4,,traceback.format_exception_only()}.

@item 
bpo-33809@footnote{https://bugs.python.org/issue33809}: Add the @ref{1115,,traceback.TracebackException.print()} method which
prints the formatted exception information.

@item 
bpo-42862@footnote{https://bugs.python.org/issue42862}: @ref{ed,,sqlite3} now utilizes @ref{4f3,,functools.lru_cache()} to
implement the connection statement cache. As a small optimisation, the
default statement cache size has been increased from 100 to 128. Patch by
Erlend E. Aasland.

@item 
bpo-41818@footnote{https://bugs.python.org/issue41818}: Soumendra Ganguly: add termios.tcgetwinsize(),
termios.tcsetwinsize().

@item 
bpo-40497@footnote{https://bugs.python.org/issue40497}: @ref{b43,,subprocess.check_output()} now raises @ref{17c,,ValueError}
when the invalid keyword argument @emph{check} is passed by user code.
Previously such use would fail later with a @ref{162,,TypeError}. Patch by
Rémi Lapeyre.

@item 
bpo-37449@footnote{https://bugs.python.org/issue37449}: @code{ensurepip} now uses @code{importlib.resources.files()}
traversable APIs

@item 
bpo-40956@footnote{https://bugs.python.org/issue40956}: Use Argument Clinic in @ref{ed,,sqlite3}.  Patches by Erlend E.
Aasland.

@item 
bpo-41730@footnote{https://bugs.python.org/issue41730}: @code{DeprecationWarning} is now raised when importing
@ref{115,,tkinter.tix}, which has been deprecated in documentation since
Python 3.6.

@item 
bpo-20684@footnote{https://bugs.python.org/issue20684}: Remove unused @code{_signature_get_bound_param} function from
@ref{9c,,inspect} - by Anthony Sottile.

@item 
bpo-41402@footnote{https://bugs.python.org/issue41402}: Fix @ref{1116,,email.message.EmailMessage.set_content()} when called
with binary data and @code{7bit} content transfer encoding.

@item 
bpo-32695@footnote{https://bugs.python.org/issue32695}: The @emph{compresslevel} and @emph{preset} keyword arguments of
@ref{9e5,,tarfile.open()} are now both documented and tested.

@item 
bpo-41137@footnote{https://bugs.python.org/issue41137}: Use utf-8 encoding while reading .pdbrc files. Patch by
Srinivas Reddy Thatiparthy

@item 
bpo-24391@footnote{https://bugs.python.org/issue24391}: Improved reprs of @ref{109,,threading} synchronization objects:
@ref{cf1,,Semaphore}, @ref{cf2,,BoundedSemaphore},
@ref{cf3,,Event} and @ref{d85,,Barrier}.

@item 
bpo-5846@footnote{https://bugs.python.org/issue5846}: Deprecated the following @ref{122,,unittest} functions, scheduled
for removal in Python 3.13:


@itemize *

@item 
@code{findTestCases()}

@item 
@code{makeSuite()}

@item 
@code{getTestCaseNames()}
@end itemize

Use @ref{1ad,,TestLoader} methods instead:


@itemize *

@item 
@ref{1ae,,unittest.TestLoader.loadTestsFromModule()}

@item 
@ref{1af,,unittest.TestLoader.loadTestsFromTestCase()}

@item 
@ref{1b0,,unittest.TestLoader.getTestCaseNames()}
@end itemize

Patch by Erlend E. Aasland.

@item 
bpo-40563@footnote{https://bugs.python.org/issue40563}: Support pathlike objects on dbm/shelve. Patch by Hakan Çelik
and Henry-Joseph Audéoud.

@item 
bpo-34990@footnote{https://bugs.python.org/issue34990}: Fixed a Y2k38 bug in the compileall module where it would fail
to compile files with a modification time after the year 2038.

@item 
bpo-39549@footnote{https://bugs.python.org/issue39549}: Whereas the code for reprlib.Repr had previously used a
hardcoded string value of ‘…’, this PR updates it to use of a
“fillvalue” attribute, whose value defaults to ‘…’ and can be reset in
either individual reprlib.Repr instances or in subclasses thereof.

@item 
bpo-37022@footnote{https://bugs.python.org/issue37022}: @ref{c4,,pdb} now displays exceptions from @code{repr()} with its
@code{p} and @code{pp} commands.

@item 
bpo-38840@footnote{https://bugs.python.org/issue38840}: Fix @code{test___all__} on platforms lacking a shared memory
implementation.

@item 
bpo-39359@footnote{https://bugs.python.org/issue39359}: Add one missing check that the password is a bytes object for
an encrypted zipfile.

@item 
bpo-38741@footnote{https://bugs.python.org/issue38741}: @ref{21,,configparser}: using ‘]’ inside a section header will no
longer cut the section name short at the ‘]’

@item 
bpo-38415@footnote{https://bugs.python.org/issue38415}: Added missing behavior to
@ref{649,,contextlib.asynccontextmanager()} to match
@ref{da3,,contextlib.contextmanager()} so decorated functions can themselves be
decorators.

@item 
bpo-30256@footnote{https://bugs.python.org/issue30256}: Pass multiprocessing BaseProxy argument @code{manager_owned}
through AutoProxy.

@item 
bpo-27513@footnote{https://bugs.python.org/issue27513}: @ref{1117,,email.utils.getaddresses()} now accepts
@ref{1118,,email.header.Header} objects along with string values. Patch by
Zackery Spytz.

@item 
bpo-16379@footnote{https://bugs.python.org/issue16379}: Add SQLite error code and name to @ref{ed,,sqlite3} exceptions.
Patch by Aviv Palivoda, Daniel Shahaf, and Erlend E. Aasland.

@item 
bpo-26228@footnote{https://bugs.python.org/issue26228}: pty.spawn no longer hangs on FreeBSD, macOS, and Solaris.

@item 
bpo-33349@footnote{https://bugs.python.org/issue33349}: lib2to3 now recognizes async generators everywhere.

@item 
bpo-29298@footnote{https://bugs.python.org/issue29298}: Fix @code{TypeError} when required subparsers without @code{dest} do
not receive arguments. Patch by Anthony Sottile.
@end itemize

@node Documentation<7>,Tests<6>,Library<6>,Python 3 11 0 alpha 1
@anchor{whatsnew/changelog id37}@anchor{1119}
@subsubsection Documentation



@itemize -

@item 
bpo-45216@footnote{https://bugs.python.org/issue45216}: Remove extra documentation listing methods in @code{difflib}. It
was rendering twice in pydoc and was outdated in some places.

@item 
bpo-45024@footnote{https://bugs.python.org/issue45024}: @ref{1d,,collections.abc} documentation has been expanded to
explicitly cover how instance and subclass checks work, with additional
doctest examples and an exhaustive list of ABCs which test membership
purely by presence of the right @ref{111a,,special method}s. Patch by Raymond
Hettinger.

@item 
bpo-44957@footnote{https://bugs.python.org/issue44957}: Promote PEP 604 union syntax by using it where possible. Also,
mention @code{X | Y} more prominently in section about @code{Union} and mention
@code{X | None} at all in section about @code{Optional}.

@item 
bpo-16580@footnote{https://bugs.python.org/issue16580}: Added code equivalents for the @ref{10f4,,int.to_bytes()} and
@ref{10a2,,int.from_bytes()} methods, as well as tests ensuring that these code
equivalents are valid.

@item 
bpo-44903@footnote{https://bugs.python.org/issue44903}: Removed the othergui.rst file, any references to it, and the
list of GUI frameworks in the FAQ. In their place I’ve added links to the
Python Wiki @code{page on GUI frameworks}.

@item 
bpo-33479@footnote{https://bugs.python.org/issue33479}: Tkinter documentation has been greatly expanded with new
“Architecture” and “Threading model” sections.

@item 
bpo-36700@footnote{https://bugs.python.org/issue36700}: @ref{d,,base64} RFC references were updated to point to
@geindex RFC; RFC 4648
RFC 4648@footnote{https://tools.ietf.org/html/rfc4648.html}; a section was added to point users to the new “security
considerations” section of the RFC.

@item 
bpo-44740@footnote{https://bugs.python.org/issue44740}: Replaced occurrences of uppercase “Web” and “Internet” with
lowercase versions per the 2016 revised Associated Press Style Book.

@item 
bpo-44693@footnote{https://bugs.python.org/issue44693}: Update the definition of __future__ in the glossary by
replacing the confusing word “pseudo-module” with a more accurate
description.

@item 
bpo-35183@footnote{https://bugs.python.org/issue35183}: Add typical examples to os.path.splitext docs

@item 
bpo-30511@footnote{https://bugs.python.org/issue30511}: Clarify that @ref{539,,shutil.make_archive()} is not thread-safe due
to reliance on changing the current working directory.

@item 
bpo-44561@footnote{https://bugs.python.org/issue44561}: Update of three expired hyperlinks in
Doc/distributing/index.rst: “Project structure”, “Building and packaging
the project”, and “Uploading the project to the Python Packaging Index”.

@item 
bpo-44651@footnote{https://bugs.python.org/issue44651}: Delete entry “coercion” in Doc/glossary.rst for its outdated
definition.

@item 
bpo-42958@footnote{https://bugs.python.org/issue42958}: Updated the docstring and docs of @ref{111b,,filecmp.cmp()} to be
more accurate and less confusing especially in respect to @emph{shallow} arg.

@item 
bpo-44631@footnote{https://bugs.python.org/issue44631}: Refactored the @code{repr()} code of the @code{_Environ} (os module).

@item 
bpo-44613@footnote{https://bugs.python.org/issue44613}: importlib.metadata is no longer provisional.

@item 
bpo-44558@footnote{https://bugs.python.org/issue44558}: Match the docstring and python implementation of
@ref{3c2,,countOf()} to the behavior of its c implementation.

@item 
bpo-44544@footnote{https://bugs.python.org/issue44544}: List all kwargs for @ref{111c,,textwrap.wrap()},
@ref{111d,,textwrap.fill()}, and @ref{b55,,textwrap.shorten()}. Now, there are nav
links to attributes of @code{TextWrap}, which makes navigation much
easier while minimizing duplication in the documentation.

@item 
bpo-38062@footnote{https://bugs.python.org/issue38062}: Clarify that atexit uses equality comparisons internally.

@item 
bpo-40620@footnote{https://bugs.python.org/issue40620}: Convert examples in tutorial controlflow.rst section 4.3 to be
interpreter-demo style.

@item 
bpo-43066@footnote{https://bugs.python.org/issue43066}: Added a warning to @ref{149,,zipfile} docs: filename arg with a
leading slash may cause archive to be un-openable on Windows systems.

@item 
bpo-39452@footnote{https://bugs.python.org/issue39452}: Rewrote @code{Doc/library/__main__.rst}. Broadened scope of the
document to explicitly discuss and differentiate between @code{__main__.py}
in packages versus the @code{__name__ == '__main__'} expression (and the
idioms that surround it).

@item 
bpo-13814@footnote{https://bugs.python.org/issue13814}: In the Design FAQ, answer “Why don’t generators support the
with statement?”

@item 
bpo-27752@footnote{https://bugs.python.org/issue27752}: Documentation of csv.Dialect is more descriptive.

@item 
bpo-44453@footnote{https://bugs.python.org/issue44453}: Fix documentation for the return type of
@ref{ec0,,sysconfig.get_path()}.

@item 
bpo-44392@footnote{https://bugs.python.org/issue44392}: Added a new section in the C API documentation for types used
in type hinting.  Documented @code{Py_GenericAlias} and
@code{Py_GenericAliasType}.

@item 
bpo-38291@footnote{https://bugs.python.org/issue38291}: Mark @code{typing.io} and @code{typing.re} as deprecated since Python
3.8 in the documentation. They were never properly supported by type
checkers.

@item 
bpo-44322@footnote{https://bugs.python.org/issue44322}: Document that SyntaxError args have a details tuple and that
details are adjusted for errors in f-string field replacement expressions.

@item 
bpo-42392@footnote{https://bugs.python.org/issue42392}: Document the deprecation and removal of the @code{loop} parameter
for many functions and classes in @ref{9,,asyncio}.

@item 
bpo-44195@footnote{https://bugs.python.org/issue44195}: Corrected references to @code{TraversableResources} in docs. There
is no @code{TraversableReader}.

@item 
bpo-41963@footnote{https://bugs.python.org/issue41963}: Document that @code{ConfigParser} strips off comments when reading
configuration files.

@item 
bpo-44072@footnote{https://bugs.python.org/issue44072}: Correct where in the numeric ABC hierarchy @code{**} support is
added, i.e., in numbers.Complex, not numbers.Integral.

@item 
bpo-43558@footnote{https://bugs.python.org/issue43558}: Add the remark to @ref{2e,,dataclasses} documentation that the
@code{__init__()} of any base class has to be called in
@code{__post_init__()}, along with a code example.

@item 
bpo-44025@footnote{https://bugs.python.org/issue44025}: Clarify when ‘_’ in match statements is a keyword, and when
not.

@item 
bpo-41706@footnote{https://bugs.python.org/issue41706}: Fix docs about how methods like @code{__add__} are invoked when
evaluating operator expressions.

@item 
bpo-41621@footnote{https://bugs.python.org/issue41621}: Document that @ref{d63,,collections.defaultdict} parameter
@code{default_factory} defaults to None and is positional-only.

@item 
bpo-41576@footnote{https://bugs.python.org/issue41576}: document BaseException in favor of bare except

@item 
bpo-21760@footnote{https://bugs.python.org/issue21760}: The description for __file__ fixed. Patch by Furkan Onder

@item 
bpo-39498@footnote{https://bugs.python.org/issue39498}: Add a “Security Considerations” index which links to standard
library modules that have explicitly documented security considerations.

@item 
bpo-33479@footnote{https://bugs.python.org/issue33479}: Remove the unqualified claim that tkinter is threadsafe. It has
not been true for several years and likely never was. An explanation of
what is true may be added later, after more discussion, and possibly after
patching _tkinter.c,
@end itemize

@node Tests<6>,Build<6>,Documentation<7>,Python 3 11 0 alpha 1
@anchor{whatsnew/changelog id38}@anchor{111e}
@subsubsection Tests



@itemize -

@item 
bpo-40173@footnote{https://bugs.python.org/issue40173}: Fix @ref{111f,,test.support.import_helper.import_fresh_module()}.

@item 
bpo-45280@footnote{https://bugs.python.org/issue45280}: Add a test case for empty @ref{457,,typing.NamedTuple}.

@item 
bpo-45269@footnote{https://bugs.python.org/issue45269}: Cover case when invalid @code{markers} type is supplied to
@code{c_make_encoder}.

@item 
bpo-45128@footnote{https://bugs.python.org/issue45128}: Fix @code{test_multiprocessing_fork} failure due to
@code{test_logging} and @code{sys.modules} manipulation.

@item 
bpo-45209@footnote{https://bugs.python.org/issue45209}: Fix @code{UserWarning: resource_tracker} warning in
@code{_test_multiprocessing._TestSharedMemory.test_shared_memory_cleaned_after_process_termination}

@item 
bpo-45185@footnote{https://bugs.python.org/issue45185}: Enables @code{TestEnumerations} test cases in @code{test_ssl} suite.

@item 
bpo-45195@footnote{https://bugs.python.org/issue45195}: Fix test_readline.test_nonascii(): sometimes, the newline
character is not written at the end, so don’t expect it in the output.
Patch by Victor Stinner.

@item 
bpo-45156@footnote{https://bugs.python.org/issue45156}: Fixes infinite loop on @ref{6d2,,unittest.mock.seal()} of mocks
created by @code{create_autospec()}.

@item 
bpo-45125@footnote{https://bugs.python.org/issue45125}: Improves pickling tests and docs of @code{SharedMemory} and
@code{SharableList} objects.

@item 
bpo-44860@footnote{https://bugs.python.org/issue44860}: Update @code{test_sysconfig.test_user_similar()} for the
posix_user scheme: @code{platlib} doesn’t use @ref{42c,,sys.platlibdir}. Patch
by Victor Stinner.

@item 
bpo-45052@footnote{https://bugs.python.org/issue45052}: @code{WithProcessesTestSharedMemory.test_shared_memory_basics}
test was ignored, because @code{self.assertEqual(sms.size, sms2.size)} line
was failing. It is now removed and test is unskipped.

The main motivation for this line to be removed from the test is that the
@code{size} of @code{SharedMemory} is not ever guaranteed to be the same. It is
decided by the platform.

@item 
bpo-44895@footnote{https://bugs.python.org/issue44895}: libregrtest now clears the type cache later to reduce the risk
of false alarm when checking for reference leaks. Previously, the type
cache was cleared too early and libregrtest raised a false alarm about
reference leaks under very specific conditions. Patch by Irit Katriel and
Victor Stinner.

@item 
bpo-45042@footnote{https://bugs.python.org/issue45042}: Fixes that test classes decorated with
@code{@@hashlib_helper.requires_hashdigest} were skipped all the time.

@item 
bpo-25130@footnote{https://bugs.python.org/issue25130}: Add calls of @ref{54b,,gc.collect()} in tests to support PyPy.

@item 
bpo-45011@footnote{https://bugs.python.org/issue45011}: Made tests relying on the @code{_asyncio} C extension module
optional to allow running on alternative Python implementations. Patch by
Serhiy Storchaka.

@item 
bpo-44949@footnote{https://bugs.python.org/issue44949}: Fix auto history tests of test_readline: sometimes, the newline
character is not written at the end, so don’t expect it in the output.

@item 
bpo-44891@footnote{https://bugs.python.org/issue44891}: Tests were added to clarify @ref{f80,,id()} is preserved when @code{obj
* 1} is used on @ref{2ef,,str} and @ref{2f0,,bytes} objects. Patch by Nikita
Sobolev.

@item 
bpo-44852@footnote{https://bugs.python.org/issue44852}: Add ability to wholesale silence DeprecationWarnings while
running the regression test suite.

@item 
bpo-40928@footnote{https://bugs.python.org/issue40928}: Notify users running test_decimal regression tests on macOS of
potential harmless “malloc can’t allocate region” messages spewed by
test_decimal.

@item 
bpo-44734@footnote{https://bugs.python.org/issue44734}: Fixed floating point precision issue in turtle tests.

@item 
bpo-44708@footnote{https://bugs.python.org/issue44708}: Regression tests, when run with -w, are now re-running only the
affected test methods instead of re-running the entire test file.

@item 
bpo-42095@footnote{https://bugs.python.org/issue42095}: Added interop tests for Apple plists: generate plist files with
Python plistlib and parse with Apple plutil; and the other way round.

@item 
bpo-44647@footnote{https://bugs.python.org/issue44647}: Added a permanent Unicode-valued environment variable to
regression tests to ensure they handle this use case in the future. If
your test environment breaks because of that, report a bug to us, and
temporarily set PYTHONREGRTEST_UNICODE_GUARD=0 in your test environment.

@item 
bpo-44515@footnote{https://bugs.python.org/issue44515}: Adjust recently added contextlib tests to avoid assuming the
use of a refcounted GC

@item 
bpo-44287@footnote{https://bugs.python.org/issue44287}: Fix asyncio test_popen() of test_windows_utils by using a
longer timeout. Use military grade battle-tested
@ref{1120,,test.support.SHORT_TIMEOUT} timeout rather than a hardcoded timeout
of 10 seconds: it’s 30 seconds by default, but it is made longer on slow
buildbots. Patch by Victor Stinner.

@item 
bpo-44451@footnote{https://bugs.python.org/issue44451}: Reset @code{DeprecationWarning} filters in
@code{test.test_importlib.test_metadata_api.APITests.test_entry_points_by_index}
to avoid @code{StopIteration} error if @code{DeprecationWarnings} are ignored.

@item 
bpo-44363@footnote{https://bugs.python.org/issue44363}: Account for address sanitizer in test_capi. test_capi now
passes when run GCC address sanitizer.

@item 
bpo-44364@footnote{https://bugs.python.org/issue44364}: Add non integral tests for @ref{50f,,math.sqrt()} function.

@item 
bpo-43921@footnote{https://bugs.python.org/issue43921}: Fix test_ssl.test_wrong_cert_tls13(): use
@code{suppress_ragged_eofs=False}, since @code{read()} can raise
@ref{1121,,ssl.SSLEOFError} on Windows. Patch by Victor Stinner.

@item 
bpo-43921@footnote{https://bugs.python.org/issue43921}: Fix test_pha_required_nocert() of test_ssl: catch two more EOF
cases (when the @code{recv()} method returns an empty string). Patch by
Victor Stinner.

@item 
bpo-44131@footnote{https://bugs.python.org/issue44131}: Add test_frozenmain to test_embed to test the
@code{Py_FrozenMain()} C function. Patch by Victor Stinner.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Ignore error string case in test_file_not_exists().

@item 
bpo-42083@footnote{https://bugs.python.org/issue42083}: Add test to check that @code{PyStructSequence_NewType} accepts a
@code{PyStructSequence_Desc} with @code{doc} field set to @code{NULL}.

@item 
bpo-35753@footnote{https://bugs.python.org/issue35753}: Fix crash in doctest when doctest parses modules that include
unwrappable functions by skipping those functions.

@item 
bpo-30256@footnote{https://bugs.python.org/issue30256}: Add test for nested queues when using @code{multiprocessing}
shared objects @code{AutoProxy[Queue]} inside @code{ListProxy} and @code{DictProxy}
@end itemize

@node Build<6>,Windows<6>,Tests<6>,Python 3 11 0 alpha 1
@anchor{whatsnew/changelog id39}@anchor{1122}
@subsubsection Build



@itemize -

@item 
bpo-45220@footnote{https://bugs.python.org/issue45220}: Avoid building with the Windows 11 SDK previews automatically.
This may be overridden by setting the @code{DefaultWindowsSDKVersion}
environment variable before building.

@item 
bpo-45020@footnote{https://bugs.python.org/issue45020}: Freeze stdlib modules that are imported during startup.  This
provides significant performance improvements to startup.  If necessary,
use the previously added “-X frozen_modules=off” commandline option to
force importing the source modules.

@item 
bpo-45188@footnote{https://bugs.python.org/issue45188}: Windows builds now regenerate frozen modules as the first part
of the build. Previously the regeneration was later in the build, which
would require it to be restarted if any modules had changed.

@item 
bpo-45163@footnote{https://bugs.python.org/issue45163}: Fixes Haiku platform build.

@item 
bpo-45067@footnote{https://bugs.python.org/issue45067}: The ncurses function extended_color_content was introduced in
2017

(@indicateurl{https://invisible-island.net/ncurses/NEWS.html#index-t20170401}).  The

ncurses-devel package in CentOS 7 had a older version ncurses resulted in
compilation error.  For compiling ncurses with extended color support, we
verify the version of the ncurses library >= 20170401.

@item 
bpo-45019@footnote{https://bugs.python.org/issue45019}: Generate lines in relevant files for frozen modules.  Up until
now each of the files had to be edited manually.  This change makes it
easier to add to and modify the frozen modules.

@item 
bpo-44340@footnote{https://bugs.python.org/issue44340}: Add support for building with clang thin lto via
–with-lto=thin/full. Patch by Dong-hee Na and Brett Holman.

@item 
bpo-44535@footnote{https://bugs.python.org/issue44535}: Enable building using a Visual Studio 2022 install on Windows.

@item 
bpo-43298@footnote{https://bugs.python.org/issue43298}: Improved error message when building without a Windows SDK
installed.

@item 
bpo-44381@footnote{https://bugs.python.org/issue44381}: The Windows build now accepts 
@geindex EnableControlFlowGuard
@geindex environment variable; EnableControlFlowGuard
@code{EnableControlFlowGuard}
set to @code{guard} to enable CFG.

@item 
bpo-41282@footnote{https://bugs.python.org/issue41282}: Fix broken @code{make install} that caused standard library
extension modules to be unnecessarily and incorrectly rebuilt during the
install phase of cpython.
@end itemize

@node Windows<6>,macOS<5>,Build<6>,Python 3 11 0 alpha 1
@anchor{whatsnew/changelog id40}@anchor{1123}
@subsubsection Windows



@itemize -

@item 
bpo-45375@footnote{https://bugs.python.org/issue45375}: Fixes an assertion failure due to searching for the standard
library in unnormalised paths.

@item 
bpo-45022@footnote{https://bugs.python.org/issue45022}: Update Windows release to include libffi 3.4.2

@item 
bpo-45007@footnote{https://bugs.python.org/issue45007}: Update to OpenSSL 1.1.1l in Windows build

@item 
bpo-44848@footnote{https://bugs.python.org/issue44848}: Upgrade Windows installer to use SQLite 3.36.0.

@item 
bpo-44572@footnote{https://bugs.python.org/issue44572}: Avoid consuming standard input in the @ref{c9,,platform} module

@item 
bpo-44582@footnote{https://bugs.python.org/issue44582}: Accelerate speed of @ref{ae,,mimetypes} initialization using a
native implementation of the registry scan.

@item 
bpo-41299@footnote{https://bugs.python.org/issue41299}: Fix 16 milliseconds jitter when using timeouts in
@ref{109,,threading}, such as with @ref{192,,threading.Lock.acquire()} or
@ref{1124,,threading.Condition.wait()}.

@item 
bpo-42686@footnote{https://bugs.python.org/issue42686}: Build @ref{ed,,sqlite3} with math functions enabled. Patch by
Erlend E. Aasland.

@item 
bpo-40263@footnote{https://bugs.python.org/issue40263}: This is a follow-on bug from
@indicateurl{https://bugs.python.org/issue26903}. Once that is applied we run into an
off-by-one assertion problem. The assert was not correct.
@end itemize

@node macOS<5>,IDLE<6>,Windows<6>,Python 3 11 0 alpha 1
@anchor{whatsnew/changelog id41}@anchor{1125}
@subsubsection macOS



@itemize -

@item 
bpo-45007@footnote{https://bugs.python.org/issue45007}: Update macOS installer builds to use OpenSSL 1.1.1l.

@item 
bpo-34602@footnote{https://bugs.python.org/issue34602}: When building CPython on macOS with @code{./configure
--with-undefined-behavior-sanitizer --with-pydebug}, the stack size is
now quadrupled to allow for the entire test suite to pass.

@item 
bpo-44848@footnote{https://bugs.python.org/issue44848}: Update macOS installer to use SQLite 3.36.0.

@item 
bpo-44689@footnote{https://bugs.python.org/issue44689}: @ref{8a0,,ctypes.util.find_library()} now works correctly on macOS
11 Big Sur even if Python is built on an older version of macOS.
Previously, when built on older macOS systems, @code{find_library} was not
able to find  macOS system libraries when running on Big Sur due to
changes in  how system libraries are stored.

@item 
bpo-41972@footnote{https://bugs.python.org/issue41972}: The framework build’s user header path in sysconfig is changed
to add a ‘pythonX.Y’ component to match distutils’s behavior.

@item 
bpo-43109@footnote{https://bugs.python.org/issue43109}: Allow –with-lto configure option to work with Apple-supplied
Xcode or Command Line Tools.

@item 
bpo-34932@footnote{https://bugs.python.org/issue34932}: Add socket.TCP_KEEPALIVE support for macOS. Patch by Shane
Harvey.
@end itemize

@node IDLE<6>,Tools/Demos,macOS<5>,Python 3 11 0 alpha 1
@anchor{whatsnew/changelog id42}@anchor{1126}
@subsubsection IDLE



@itemize -

@item 
bpo-45296@footnote{https://bugs.python.org/issue45296}: On Windows, change exit/quit message to suggest Ctrl-D, which
works, instead of <Ctrl-Z Return>, which does not work in IDLE.

@item 
bpo-45193@footnote{https://bugs.python.org/issue45193}: Make completion boxes appear on Ubuntu again.

@item 
bpo-40128@footnote{https://bugs.python.org/issue40128}: Mostly fix completions on macOS when not using tcl/tk 8.6.11
(as with 3.9). The added update_idletask call should be harmless and
possibly helpful otherwise.

@item 
bpo-33962@footnote{https://bugs.python.org/issue33962}: Move the indent space setting from the Font tab to the new
Windows tab. Patch by Mark Roseman and Terry Jan Reedy.

@item 
bpo-40468@footnote{https://bugs.python.org/issue40468}: Split the settings dialog General tab into Windows and Shell/ED
tabs. Move help sources, which extend the Help menu, to the Extensions
tab. Make space for new options and shorten the dialog. The latter makes
the dialog better fit small screens.

@item 
bpo-41611@footnote{https://bugs.python.org/issue41611}: Avoid uncaught exceptions in
@code{AutoCompleteWindow.winconfig_event()}.

@item 
bpo-41611@footnote{https://bugs.python.org/issue41611}: Fix IDLE sometimes freezing upon tab-completion on macOS.

@item 
bpo-44010@footnote{https://bugs.python.org/issue44010}: Highlight the new @ref{290,,match} statement’s @ref{28f,,soft keywords}: @ref{290,,match}, @ref{290,,case}, and
@ref{291,,_}. However, this highlighting is not
perfect and will be incorrect in some rare cases, including some @code{_}-s
in @code{case} patterns.

@item 
bpo-44026@footnote{https://bugs.python.org/issue44026}: Include interpreter’s typo fix suggestions in message line for
NameErrors and AttributeErrors.  Patch by E. Paine.
@end itemize

@node Tools/Demos,C API<6>,IDLE<6>,Python 3 11 0 alpha 1
@anchor{whatsnew/changelog tools-demos}@anchor{1127}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-44786@footnote{https://bugs.python.org/issue44786}: Fix a warning in regular expression in the c-analyzer script.

@item 
bpo-44967@footnote{https://bugs.python.org/issue44967}: pydoc now returns a non-zero status code when a module cannot
be found.

@item 
bpo-44978@footnote{https://bugs.python.org/issue44978}: Allow the Argument Clinic tool to handle @code{__complex__}
special methods.

@item 
bpo-43425@footnote{https://bugs.python.org/issue43425}: Removed the ‘test2to3’ demo project that demonstrated using
lib2to3 to support Python 2.x and Python 3.x from a single source in a
distutils package. Patch by Dong-hee Na

@item 
bpo-44074@footnote{https://bugs.python.org/issue44074}: Make patchcheck automatically detect the correct base branch
name (previously it was hardcoded to ‘master’)

@item 
bpo-20291@footnote{https://bugs.python.org/issue20291}: Added support for variadic positional parameters in Argument
Clinic.
@end itemize

@node C API<6>,,Tools/Demos,Python 3 11 0 alpha 1
@anchor{whatsnew/changelog id43}@anchor{1128}
@subsubsection C API



@itemize -

@item 
bpo-41710@footnote{https://bugs.python.org/issue41710}: The PyThread_acquire_lock_timed() function now clamps the
timeout if it is too large, rather than aborting the process. Patch by
Victor Stinner.

@item 
bpo-44687@footnote{https://bugs.python.org/issue44687}: @code{BufferedReader.peek()} no longer raises @ref{17c,,ValueError}
when the entire file has already been buffered.

@item 
bpo-45116@footnote{https://bugs.python.org/issue45116}: Add the @ref{1129,,Py_ALWAYS_INLINE} macro to ask the compiler
to always inline a static inline function. The compiler can ignore it and
decides to not inline the function. Patch by Victor Stinner.

@item 
bpo-45094@footnote{https://bugs.python.org/issue45094}: Add the @ref{112a,,Py_NO_INLINE} macro to disable inlining on a
function. Patch by Victor Stinner.

@item 
bpo-45061@footnote{https://bugs.python.org/issue45061}: Add a deallocator to the @ref{2f7,,bool} type to detect refcount
bugs in C extensions which call @code{Py_DECREF(Py_True);} or
@code{Py_DECREF(Py_False);} by mistake. Patch by Victor Stinner.

@item 
bpo-42035@footnote{https://bugs.python.org/issue42035}: Add a new @ref{1d6,,PyType_GetQualName()} function to get type’s
qualified name.

@item 
bpo-41103@footnote{https://bugs.python.org/issue41103}: Reverts removal of the old buffer protocol because they are
part of stable ABI.

@item 
bpo-44751@footnote{https://bugs.python.org/issue44751}: Remove @code{crypt.h} include from the public @code{Python.h} header.

@item 
bpo-42747@footnote{https://bugs.python.org/issue42747}: The @code{Py_TPFLAGS_HAVE_VERSION_TAG} type flag now does nothing.
The @code{Py_TPFLAGS_HAVE_AM_SEND} flag (which was added in 3.10) is removed.
Both were unnecessary because it is not possible to have type objects with
the relevant fields missing.

@item 
bpo-44530@footnote{https://bugs.python.org/issue44530}: Added the @code{co_qualname} to the @code{PyCodeObject} structure to
propagate the qualified name from the compiler to code objects.

Patch by Gabriele N. Tornetta

@item 
bpo-44441@footnote{https://bugs.python.org/issue44441}: @ref{4bb,,Py_RunMain()} now resets @code{PyImport_Inittab} to
its initial value at exit. It must be possible to call
@ref{112b,,PyImport_AppendInittab()} or @ref{112c,,PyImport_ExtendInittab()} at
each Python initialization. Patch by Victor Stinner.

@item 
bpo-39947@footnote{https://bugs.python.org/issue39947}: Remove 4 private trashcan C API functions which were only kept
for the backward compatibility of the stable ABI with Python 3.8 and
older, since the trashcan API was not usable with the limited C API on
Python 3.8 and older. The trashcan API was excluded from the limited C API
in Python 3.9.

Removed functions:


@itemize *

@item 
_PyTrash_deposit_object()

@item 
_PyTrash_destroy_chain()

@item 
_PyTrash_thread_deposit_object()

@item 
_PyTrash_thread_destroy_chain()
@end itemize

The trashcan C API was never usable with the limited C API, since old
trashcan macros accessed directly @ref{1fa,,PyThreadState} members like
@code{_tstate->trash_delete_nesting}, whereas the @ref{1fa,,PyThreadState}
structure is opaque in the limited C API.

Exclude also the @code{PyTrash_UNWIND_LEVEL} constant from the C API.

Patch by Victor Stinner.

@item 
bpo-40939@footnote{https://bugs.python.org/issue40939}: Removed documentation for the removed @code{PyParser_*} C API.

@item 
bpo-43795@footnote{https://bugs.python.org/issue43795}: The list in @ref{210,,Contents of Limited API} now shows the public name
@ref{1f6,,PyFrameObject} rather than @code{_frame}. The non-existing entry
@code{_node} no longer appears in the list.

@item 
bpo-44378@footnote{https://bugs.python.org/issue44378}: @ref{112d,,Py_IS_TYPE()} no longer uses @ref{1f1,,Py_TYPE()} to avoid
a compiler warning: no longer cast @code{const PyObject*} to @code{PyObject*}.
Patch by Victor Stinner.

@item 
bpo-39573@footnote{https://bugs.python.org/issue39573}: Convert the @ref{1f1,,Py_TYPE()} and @ref{1f3,,Py_SIZE()} macros to
static inline functions. The @ref{1f2,,Py_SET_TYPE()} and
@ref{1f4,,Py_SET_SIZE()} functions must now be used to set an object type and
size. Patch by Victor Stinner.

@item 
bpo-44263@footnote{https://bugs.python.org/issue44263}: The @ref{1ec,,PyType_Ready()} function now raises an error if a
type is defined with the @ref{1ed,,Py_TPFLAGS_HAVE_GC} flag set but has no
traverse function (@ref{1ee,,PyTypeObject.tp_traverse}). Patch by Victor
Stinner.

@item 
bpo-43795@footnote{https://bugs.python.org/issue43795}: The undocumented function @code{Py_FrozenMain()} is removed
from the Limited API.

@item 
bpo-44113@footnote{https://bugs.python.org/issue44113}: Deprecate the following functions to configure the Python
initialization:


@itemize *

@item 
@ref{1ff,,PySys_AddWarnOptionUnicode()}

@item 
@ref{200,,PySys_AddWarnOption()}

@item 
@ref{201,,PySys_AddXOption()}

@item 
@code{PySys_HasWarnOptions()}

@item 
@ref{202,,Py_SetPath()}

@item 
@ref{203,,Py_SetProgramName()}

@item 
@ref{204,,Py_SetPythonHome()}

@item 
@ref{205,,Py_SetStandardStreamEncoding()}

@item 
@code{_Py_SetProgramFullPath()}
@end itemize

Use the new @ref{206,,PyConfig} API of the @ref{207,,Python Initialization Configuration} instead (
@geindex Python Enhancement Proposals; PEP 587
PEP 587@footnote{https://www.python.org/dev/peps/pep-0587}).

@item 
bpo-44094@footnote{https://bugs.python.org/issue44094}: Remove @code{PyErr_SetFromErrnoWithUnicodeFilename()},
@code{PyErr_SetFromWindowsErrWithUnicodeFilename()}, and
@code{PyErr_SetExcFromWindowsErrWithUnicodeFilename()}. They are not
documented and have been deprecated since Python 3.3.

@item 
bpo-43795@footnote{https://bugs.python.org/issue43795}: @ref{365,,PyCodec_Unregister()} is now properly exported as a
function in the Windows Stable ABI DLL.

@item 
bpo-44029@footnote{https://bugs.python.org/issue44029}: Remove deprecated @code{Py_UNICODE} APIs: @code{PyUnicode_Encode},
@code{PyUnicode_EncodeUTF7}, @code{PyUnicode_EncodeUTF8},
@code{PyUnicode_EncodeUTF16}, @code{PyUnicode_EncodeUTF32},
@code{PyUnicode_EncodeLatin1}, @code{PyUnicode_EncodeMBCS},
@code{PyUnicode_EncodeDecimal}, @code{PyUnicode_EncodeRawUnicodeEscape},
@code{PyUnicode_EncodeCharmap}, @code{PyUnicode_EncodeUnicodeEscape},
@code{PyUnicode_TransformDecimalToASCII}, @code{PyUnicode_TranslateCharmap},
@code{PyUnicodeEncodeError_Create}, @code{PyUnicodeTranslateError_Create}. See
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393} and 
@geindex Python Enhancement Proposals; PEP 624
PEP 624@footnote{https://www.python.org/dev/peps/pep-0624} for reference.

@item 
bpo-42035@footnote{https://bugs.python.org/issue42035}: Add a new @ref{1d5,,PyType_GetName()} function to get type’s short
name.
@end itemize

@node Python 3 10 0 beta 1,Python 3 10 0 alpha 7,Python 3 11 0 alpha 1,Changelog
@anchor{whatsnew/changelog python-3-10-0-beta-1}@anchor{112e}
@subsection Python 3.10.0 beta 1


@emph{Release date: 2021-05-03}

@menu
* Security: Security<2>. 
* Core and Builtins: Core and Builtins<7>. 
* Library: Library<7>. 
* Documentation: Documentation<8>. 
* Tests: Tests<7>. 
* Build: Build<7>. 
* Windows: Windows<7>. 
* macOS: macOS<6>. 
* IDLE: IDLE<7>. 
* C API: C API<7>. 

@end menu

@node Security<2>,Core and Builtins<7>,,Python 3 10 0 beta 1
@anchor{whatsnew/changelog id44}@anchor{112f}
@subsubsection Security



@itemize -

@item 
bpo-43434@footnote{https://bugs.python.org/issue43434}: Creating @ref{18b,,sqlite3.Connection} objects now also produces
@code{sqlite3.connect} and @code{sqlite3.connect/handle} @ref{1130,,auditing events}. Previously these events were only produced by
@ref{6a8,,sqlite3.connect()} calls. Patch by Erlend E. Aasland.

@item 
bpo-43998@footnote{https://bugs.python.org/issue43998}: The @ref{ee,,ssl} module sets more secure cipher suites defaults.
Ciphers without forward secrecy and with SHA-1 MAC are disabled by
default. Security level 2 prohibits weak RSA, DH, and ECC keys with less
than 112 bits of security. @ref{2bf,,SSLContext} defaults to minimum
protocol version TLS 1.2. Settings are based on Hynek Schlawack’s
research.

@item 
bpo-43882@footnote{https://bugs.python.org/issue43882}: The presence of newline or tab characters in parts of a URL
could allow some forms of attacks.

Following the controlling specification for URLs defined by WHATWG
@ref{126,,urllib.parse()} now removes ASCII newlines and tabs from URLs,
preventing such attacks.

@item 
bpo-43472@footnote{https://bugs.python.org/issue43472}: Ensures interpreter-level audit hooks receive the
@code{cpython.PyInterpreterState_New} event when called through the
@code{_xxsubinterpreters} module.

@item 
bpo-43362@footnote{https://bugs.python.org/issue43362}: Fix invalid free in _sha3 module. The issue was introduced in
3.10.0a1. Python 3.9 and earlier are not affected.

@item 
bpo-43762@footnote{https://bugs.python.org/issue43762}: Add audit events for @code{sqlite3.connect/handle()},
@ref{2c3,,sqlite3.Connection.enable_load_extension()}, and
@ref{2c4,,sqlite3.Connection.load_extension()}. Patch by Erlend E. Aasland.

@item 
bpo-43756@footnote{https://bugs.python.org/issue43756}: Add new audit event @code{glob.glob/2} to incorporate the new
@emph{root_dir} and @emph{dir_fd} arguments added to @ref{288,,glob.glob()} and
@ref{289,,glob.iglob()}.

@item 
bpo-36384@footnote{https://bugs.python.org/issue36384}: @ref{9e,,ipaddress} module no longer accepts any leading zeros in
IPv4 address strings. Leading zeros are ambiguous and interpreted as octal
notation by some libraries. For example the legacy function
@ref{464,,socket.inet_aton()} treats leading zeros as octal notation. glibc
implementation of modern @ref{465,,inet_pton()} does not accept any
leading zeros. For a while the @ref{9e,,ipaddress} module used to accept
ambiguous leading zeros.

@item 
bpo-43075@footnote{https://bugs.python.org/issue43075}: Fix Regular Expression Denial of Service (ReDoS) vulnerability
in @ref{1131,,urllib.request.AbstractBasicAuthHandler}.  The ReDoS-vulnerable
regex has quadratic worst-case complexity and it allows cause a denial of
service when identifying crafted invalid RFCs. This ReDoS issue is on the
client side and needs remote attackers to control the HTTP server.

@item 
bpo-42800@footnote{https://bugs.python.org/issue42800}: Audit hooks are now fired for frame.f_code, traceback.tb_frame,
and generator code/frame attribute access.

@item 
bpo-37363@footnote{https://bugs.python.org/issue37363}: Add audit events to the @ref{8f,,http.client} module.
@end itemize

@node Core and Builtins<7>,Library<7>,Security<2>,Python 3 10 0 beta 1
@anchor{whatsnew/changelog id45}@anchor{1132}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-43977@footnote{https://bugs.python.org/issue43977}: Prevent classes being both a sequence and a mapping when
pattern matching.

@item 
bpo-43977@footnote{https://bugs.python.org/issue43977}: Use @ref{1133,,tp_flags} on the class object to
determine if the subject is a sequence or mapping when pattern matching.
Avoids the need to import @ref{1d,,collections.abc} when pattern matching.

@item 
bpo-43892@footnote{https://bugs.python.org/issue43892}: Restore proper validation of complex literal value patterns
when parsing @code{match} blocks.

@item 
bpo-43933@footnote{https://bugs.python.org/issue43933}: Set frame.f_lineno to the line number of the ‘with’ kweyword
when executing the call to @code{__exit__}.

@item 
bpo-43933@footnote{https://bugs.python.org/issue43933}: If the current position in a frame has no line number then set
the f_lineno attribute to None, instead of -1, to conform to PEP 626. This
should not normally be possible, but might occur in some unusual
circumstances.

@item 
bpo-43963@footnote{https://bugs.python.org/issue43963}: Importing the @code{_signal} module in a subinterpreter has no
longer side effects.

@item 
bpo-42739@footnote{https://bugs.python.org/issue42739}: The internal representation of line number tables is changed to
not use sentinels, and an explicit length parameter is added to the out of
process API function @code{PyLineTable_InitAddressRange}. This makes the
handling of line number tables more robust in some circumstances.

@item 
bpo-43908@footnote{https://bugs.python.org/issue43908}: Make @ref{d8,,re} types immutable. Patch by Erlend E. Aasland.

@item 
bpo-43908@footnote{https://bugs.python.org/issue43908}: Make the @ref{25e,,array.array} type immutable. Patch by Erlend
E. Aasland.

@item 
bpo-43901@footnote{https://bugs.python.org/issue43901}: Change class and module objects to lazy-create empty
annotations dicts on demand.  The annotations dicts are stored in the
object’s __dict__ for backwards compatibility.

@item 
bpo-43892@footnote{https://bugs.python.org/issue43892}: Match patterns now use new dedicated AST nodes (@code{MatchValue},
@code{MatchSingleton}, @code{MatchSequence}, @code{MatchStar}, @code{MatchMapping},
@code{MatchClass}) rather than reusing expression AST nodes. @code{MatchAs} and
@code{MatchOr} are now defined as pattern nodes rather than as expression
nodes. Patch by Nick Coghlan.

@item 
bpo-42725@footnote{https://bugs.python.org/issue42725}: Usage of @code{await}/@code{yield}/@code{yield from} and named
expressions within an annotation is now forbidden when PEP 563 is
activated.

@item 
bpo-43754@footnote{https://bugs.python.org/issue43754}: When performing structural pattern matching (
@geindex Python Enhancement Proposals; PEP 634
PEP 634@footnote{https://www.python.org/dev/peps/pep-0634}),
captured names are now left unbound until the @emph{entire} pattern has matched
successfully.

@item 
bpo-42737@footnote{https://bugs.python.org/issue42737}: Annotations for complex targets (everything beside simple
names) no longer cause any runtime effects with @code{from __future__ import
annotations}.

@item 
bpo-43914@footnote{https://bugs.python.org/issue43914}: @ref{21a,,SyntaxError} exceptions raised by the interpreter will
highlight the full error range of the expression that consistutes the
syntax error itself, instead of just where the problem is detected. Patch
by Pablo Galindo.

@item 
bpo-38605@footnote{https://bugs.python.org/issue38605}: Revert making @code{from __future__ import annotations} the
default. This follows the Steering Council decision to postpone PEP 563
changes to at least Python 3.11. See the original email for more
information regarding the decision:
@indicateurl{https://mail.python.org/archives/list/python-dev@@python.org/thread/CLVXXPQ2T2LQ5MP2Y53VVQFCXYWQJHKZ/}.
Patch by Pablo Galindo.

@item 
bpo-43475@footnote{https://bugs.python.org/issue43475}: Hashes of NaN values now depend on object identity.  Formerly,
they always hashed to 0 even though NaN values are not equal to one
another.  Having the same hash for unequal values caused pile-ups in hash
tables.

@item 
bpo-43859@footnote{https://bugs.python.org/issue43859}: Improve the error message for @ref{21c,,IndentationError}
exceptions. Patch by Pablo Galindo

@item 
bpo-41323@footnote{https://bugs.python.org/issue41323}: Constant tuple folding in bytecode optimizer now reuses tuple
in constant table.

@item 
bpo-43846@footnote{https://bugs.python.org/issue43846}: Data stack usage is much reduced for large literal and call
expressions.

@item 
bpo-38530@footnote{https://bugs.python.org/issue38530}: When printing @ref{21f,,NameError} raised by the interpreter,
@code{PyErr_Display()} will offer suggestions of similar variable names
in the function that the exception was raised from. Patch by Pablo Galindo

@item 
bpo-43823@footnote{https://bugs.python.org/issue43823}: Improve syntax errors for invalid dictionary literals. Patch by
Pablo Galindo.

@item 
bpo-43822@footnote{https://bugs.python.org/issue43822}: Improve syntax errors in the parser for missing commas between
expressions. Patch by Pablo Galindo.

@item 
bpo-43798@footnote{https://bugs.python.org/issue43798}: @ref{1134,,ast.alias} nodes now include source location metadata
attributes e.g. lineno, col_offset.

@item 
bpo-43797@footnote{https://bugs.python.org/issue43797}: Improve @code{SyntaxError} error messages for invalid comparisons.
Patch by Pablo Galindo.

@item 
bpo-43760@footnote{https://bugs.python.org/issue43760}: Move the flag for checking whether tracing is enabled to the C
stack, from the heap. Should speed up dispatch in the interpreter.

@item 
bpo-43682@footnote{https://bugs.python.org/issue43682}: Static methods (@ref{253,,@@staticmethod}) and class
methods (@ref{254,,@@classmethod}) now inherit the method
attributes (@code{__module__}, @code{__name__}, @code{__qualname__}, @code{__doc__},
@code{__annotations__}) and have a new @code{__wrapped__} attribute. Patch by
Victor Stinner.

@item 
bpo-43751@footnote{https://bugs.python.org/issue43751}: Fixed a bug where @code{anext(ait, default)} would erroneously
return None.

@item 
bpo-42128@footnote{https://bugs.python.org/issue42128}: @ref{1135,,__match_args__} is no longer allowed to be a
list.

@item 
bpo-43683@footnote{https://bugs.python.org/issue43683}: Add GEN_START opcode. Marks start of generator, including
async, or coroutine and handles sending values to a newly created
generator or coroutine.

@item 
bpo-43105@footnote{https://bugs.python.org/issue43105}: Importlib now resolves relative paths when creating module spec
objects from file locations.

@item 
bpo-43682@footnote{https://bugs.python.org/issue43682}: Static methods (@ref{253,,@@staticmethod}) are now
callable as regular functions. Patch by Victor Stinner.

@item 
bpo-42609@footnote{https://bugs.python.org/issue42609}: Prevented crashes in the AST validator and optimizer when
compiling some absurdly long expressions like @code{"+0"*1000000}.
@ref{8c4,,RecursionError} is now raised instead.

@item 
bpo-38530@footnote{https://bugs.python.org/issue38530}: When printing @ref{163,,AttributeError}, @code{PyErr_Display()}
will offer suggestions of similar attribute names in the object that the
exception was raised from. Patch by Pablo Galindo
@end itemize

@node Library<7>,Documentation<8>,Core and Builtins<7>,Python 3 10 0 beta 1
@anchor{whatsnew/changelog id46}@anchor{1136}
@subsubsection Library



@itemize -

@item 
bpo-44015@footnote{https://bugs.python.org/issue44015}: In @@dataclass(), raise a TypeError if KW_ONLY is specified more
than once.

@item 
bpo-25478@footnote{https://bugs.python.org/issue25478}: Added a @emph{total()} method to collections.Counter() to compute
the sum of the counts.

@item 
bpo-43733@footnote{https://bugs.python.org/issue43733}: Change @ref{1137,,netrc.netrc} to use UTF-8 encoding before using
locale encoding.

@item 
bpo-43979@footnote{https://bugs.python.org/issue43979}: Removed an unnecessary list comprehension before looping from
@ref{2e4,,urllib.parse.parse_qsl()}.  Patch by Christoph Zwerschke and Dong-hee
Na.

@item 
bpo-43993@footnote{https://bugs.python.org/issue43993}: Update bundled pip to 21.1.1.

@item 
bpo-43957@footnote{https://bugs.python.org/issue43957}: [Enum] Deprecate @code{TypeError} when non-member is used in a
containment check; In 3.12 @code{True} or @code{False} will be returned instead,
and containment will return @code{True} if the value is either a member of
that enum or one of its members’ value.

@item 
bpo-42904@footnote{https://bugs.python.org/issue42904}: For backwards compatibility with previous minor versions of
Python, if @ref{432,,typing.get_type_hints()} receives no namespace dictionary
arguments, @ref{432,,typing.get_type_hints()} will search through the global
then local namespaces during evaluation of stringized type annotations
(string forward references) inside a class.

@item 
bpo-43945@footnote{https://bugs.python.org/issue43945}: [Enum] Deprecate non-standard mixin format() behavior: in 3.12
the enum member, not the member’s value, will be used for format() calls.

@item 
bpo-41139@footnote{https://bugs.python.org/issue41139}: Deprecate undocumented @code{cgi.log()} API.

@item 
bpo-43937@footnote{https://bugs.python.org/issue43937}: Fixed the @ref{11d,,turtle} module working with non-default root
window.

@item 
bpo-43930@footnote{https://bugs.python.org/issue43930}: Update bundled pip to 21.1 and setuptools to 56.0.0

@item 
bpo-43907@footnote{https://bugs.python.org/issue43907}: Fix a bug in the pure-Python pickle implementation when using
protocol 5, where bytearray instances that occur several time in the
pickled object graph would incorrectly unpickle into repeated copies of
the bytearray object.

@item 
bpo-43926@footnote{https://bugs.python.org/issue43926}: In @code{importlib.metadata}, provide a uniform interface to
@code{Description}, allow for any field to be encoded with multiline values,
remove continuation lines from multiline values, and add a @code{.json}
property for easy access to the PEP 566 JSON-compatible form. Sync with
@code{importlib_metadata 4.0}.

@item 
bpo-43920@footnote{https://bugs.python.org/issue43920}: OpenSSL 3.0.0: @ref{a5f,,load_verify_locations()}
now returns a consistent error message when cadata contains no valid
certificate.

@item 
bpo-43607@footnote{https://bugs.python.org/issue43607}: @ref{124,,urllib} can now convert Windows paths with @code{\\?\}
prefixes into URL paths.

@item 
bpo-43817@footnote{https://bugs.python.org/issue43817}: Add @ref{295,,inspect.get_annotations()}, which safely computes the
annotations defined on an object.  It works around the quirks of accessing
the annotations from various types of objects, and makes very few
assumptions about the object passed in. @ref{295,,inspect.get_annotations()}
can also correctly un-stringize stringized annotations.

@ref{1c1,,inspect.signature()}, @code{inspect.from_callable()}, and
@code{inspect.from_function()} now call @ref{295,,inspect.get_annotations()} to
retrieve annotations.  This means @ref{1c1,,inspect.signature()} and
@code{inspect.from_callable()} can now un-stringize stringized annotations,
too.

@item 
bpo-43284@footnote{https://bugs.python.org/issue43284}: platform.win32_ver derives the windows version from
sys.getwindowsversion().platform_version which in turn derives the version
from kernel32.dll (which can be of a different version than Windows
itself). Therefore change the platform.win32_ver to determine the version
using the platform module’s _syscmd_ver private function to return an
accurate version.

@item 
bpo-42854@footnote{https://bugs.python.org/issue42854}: The @ref{ee,,ssl} module now uses @code{SSL_read_ex} and
@code{SSL_write_ex} internally. The functions support reading and writing of
data larger than 2 GB. Writing zero-length data no longer fails with a
protocol violation error.

@item 
bpo-42333@footnote{https://bugs.python.org/issue42333}: Port @code{_ssl} extension module to multiphase initialization.

@item 
bpo-43880@footnote{https://bugs.python.org/issue43880}: @ref{ee,,ssl} now raises DeprecationWarning for OP_NO_SSL/TLS*
options, old TLS versions, old protocols, and other features that have
been deprecated since Python 3.6, 3.7, or OpenSSL 1.1.0.

@item 
bpo-41559@footnote{https://bugs.python.org/issue41559}: 
@geindex Python Enhancement Proposals; PEP 612
PEP 612@footnote{https://www.python.org/dev/peps/pep-0612} is now implemented purely in Python; builtin
@code{types.GenericAlias} objects no longer include @code{typing.ParamSpec} in
@code{__parameters__} (with the exception of @code{collections.abc.Callable}‘s
@code{GenericAlias}). This means previously invalid uses of @code{ParamSpec}
(such as @code{list[P]}) which worked in earlier versions of Python 3.10
alpha, will now raise @code{TypeError} during substitution.

@item 
bpo-43867@footnote{https://bugs.python.org/issue43867}: The @ref{b3,,multiprocessing} @code{Server} class now explicitly
catches @ref{892,,SystemExit} and closes the client connection in this case.
It happens when the @code{Server.serve_client()} method reaches the end of
file (EOF).

@item 
bpo-40443@footnote{https://bugs.python.org/issue40443}: Remove unused imports: pyclbr no longer uses copy, and typing
no longer uses ast. Patch by Victor Stinner.

@item 
bpo-43820@footnote{https://bugs.python.org/issue43820}: Remove an unneeded copy of the namespace passed to
dataclasses.make_dataclass().

@item 
bpo-43787@footnote{https://bugs.python.org/issue43787}: Add @code{__iter__()} method to @ref{455,,bz2.BZ2File},
@ref{43f,,gzip.GzipFile}, and @ref{b93,,lzma.LZMAFile}. It makes iterating
them about 2x faster. Patch by Inada Naoki.

@item 
bpo-43680@footnote{https://bugs.python.org/issue43680}: Deprecate io.OpenWrapper and _pyio.OpenWrapper: use io.open and
_pyio.open instead. Until Python 3.9, _pyio.open was not a static method
and builtins.open was set to OpenWrapper to not become a bound method when
set to a class variable. _io.open is a built-in function whereas
_pyio.open is a Python function. In Python 3.10, _pyio.open() is now a
static method, and builtins.open() is now io.open().

@item 
bpo-43680@footnote{https://bugs.python.org/issue43680}: The Python @code{_pyio.open()} function becomes a static method
to behave as @ref{1c9,,io.open()} built-in function: don’t become a bound
method when stored as a class variable. It becomes possible since static
methods are now callable in Python 3.10. Moreover,
@code{_pyio.OpenWrapper()} becomes a simple alias to @code{_pyio.open()}.
Patch by Victor Stinner.

@item 
bpo-41515@footnote{https://bugs.python.org/issue41515}: Fix @ref{5bb,,KeyError} raised in @ref{432,,typing.get_type_hints()} due
to synthetic modules that don’t appear in @code{sys.modules}.

@item 
bpo-43776@footnote{https://bugs.python.org/issue43776}: When @ref{5ae,,subprocess.Popen} args are provided as a string or
as @ref{522,,pathlib.Path}, the Popen instance repr now shows the right
thing.

@item 
bpo-42248@footnote{https://bugs.python.org/issue42248}: [Enum] ensure exceptions raised in @code{_missing__} are released

@item 
bpo-43744@footnote{https://bugs.python.org/issue43744}: fix issue with enum member name matching the start of a private
variable name

@item 
bpo-43772@footnote{https://bugs.python.org/issue43772}: Fixed the return value of @code{TypeVar.__ror__}. Patch by Jelle
Zijlstra.

@item 
bpo-43764@footnote{https://bugs.python.org/issue43764}: Add match_args parameter to @@dataclass decorator to allow
suppression of __match_args__ generation.

@item 
bpo-43799@footnote{https://bugs.python.org/issue43799}: OpenSSL 3.0.0: define @code{OPENSSL_API_COMPAT} 1.1.1 to suppress
deprecation warnings. Python requires OpenSSL 1.1.1 APIs.

@item 
bpo-43478@footnote{https://bugs.python.org/issue43478}: Mocks can no longer be used as the specs for other Mocks. As a
result, an already-mocked object cannot have an attribute mocked using
@code{autospec=True} or be the subject of a @code{create_autospec(...)} call.
This can uncover bugs in tests since these Mock-derived Mocks will always
pass certain tests (e.g. @ref{236,,isinstance()}) and builtin assert functions
(e.g. assert_called_once_with) will unconditionally pass.

@item 
bpo-43794@footnote{https://bugs.python.org/issue43794}: Add @ref{2ba,,ssl.OP_IGNORE_UNEXPECTED_EOF} constants (OpenSSL
3.0.0)

@item 
bpo-43785@footnote{https://bugs.python.org/issue43785}: Improve @code{bz2.BZ2File} performance by removing the RLock from
BZ2File. This makes BZ2File thread unsafe in the face of multiple
simultaneous readers or writers, just like its equivalent classes in
@ref{87,,gzip} and @ref{a9,,lzma} have always been.  Patch by Inada Naoki.

@item 
bpo-43789@footnote{https://bugs.python.org/issue43789}: OpenSSL 3.0.0: Don’t call the password callback function a
second time when first call has signaled an error condition.

@item 
bpo-43788@footnote{https://bugs.python.org/issue43788}: The header files for @ref{ee,,ssl} error codes are now OpenSSL
version-specific. Exceptions will now show correct reason and library
codes. The @code{make_ssl_data.py} script has been rewritten to use OpenSSL’s
text file with error codes.

@item 
bpo-43766@footnote{https://bugs.python.org/issue43766}: Implement 
@geindex Python Enhancement Proposals; PEP 647
PEP 647@footnote{https://www.python.org/dev/peps/pep-0647} in the @ref{120,,typing} module by adding
@code{TypeGuard}.

@item 
bpo-25264@footnote{https://bugs.python.org/issue25264}: @ref{2a0,,os.path.realpath()} now accepts a @emph{strict} keyword-only
argument. When set to @code{True}, @ref{2a1,,OSError} is raised if a path doesn’t
exist or a symlink loop is encountered.

@item 
bpo-43780@footnote{https://bugs.python.org/issue43780}: In @code{importlib.metadata}, incorporate changes from
importlib_metadata 3.10: Add mtime-based caching during distribution
discovery. Flagged use of dict result from @code{entry_points()} as
deprecated.

@item 
bpo-47383@footnote{https://bugs.python.org/issue47383}: The @code{P.args} and @code{P.kwargs} attributes of
@ref{23a,,typing.ParamSpec} are now instances of the new classes
@ref{23d,,typing.ParamSpecArgs} and @ref{23e,,typing.ParamSpecKwargs}, which
enables a more useful @code{repr()}. Patch by Jelle Zijlstra.

@item 
bpo-43731@footnote{https://bugs.python.org/issue43731}: Add an @code{encoding} parameter @code{logging.fileConfig()}.

@item 
bpo-43712@footnote{https://bugs.python.org/issue43712}: Add @code{encoding} and @code{errors} parameters to
@ref{280,,fileinput.input()} and @ref{1bc,,fileinput.FileInput}.

@item 
bpo-38659@footnote{https://bugs.python.org/issue38659}: A @code{simple_enum} decorator is added to the @code{enum} module to
convert a normal class into an Enum. @code{test_simple_enum} added to test
simple enums against a corresponding normal Enum.  Standard library
modules updated to use @code{simple_enum}.

@item 
bpo-43764@footnote{https://bugs.python.org/issue43764}: Fix an issue where @ref{1135,,__match_args__} generation
could fail for some @ref{2e,,dataclasses}.

@item 
bpo-43752@footnote{https://bugs.python.org/issue43752}: Fix @ref{ed,,sqlite3} regression for zero-sized blobs with
converters, where @code{b""} was returned instead of @code{None}. The regression
was introduced by GH-24723. Patch by Erlend E. Aasland.

@item 
bpo-43655@footnote{https://bugs.python.org/issue43655}: @ref{10c,,tkinter} dialog windows are now recognized as dialogs by
window managers on macOS and X Window.

@item 
bpo-43723@footnote{https://bugs.python.org/issue43723}: The following @code{threading} methods are now deprecated and
should be replaced:


@itemize -

@item 
@code{currentThread} => @ref{31e,,threading.current_thread()}

@item 
@code{activeCount} => @ref{31f,,threading.active_count()}

@item 
@code{Condition.notifyAll} => @ref{320,,threading.Condition.notify_all()}

@item 
@code{Event.isSet} => @ref{321,,threading.Event.is_set()}

@item 
@code{Thread.setName} => @ref{322,,threading.Thread.name}

@item 
@code{thread.getName} => @ref{322,,threading.Thread.name}

@item 
@code{Thread.isDaemon} => @ref{323,,threading.Thread.daemon}

@item 
@code{Thread.setDaemon} => @ref{323,,threading.Thread.daemon}
@end itemize

Patch by Jelle Zijlstra.

@item 
bpo-2135@footnote{https://bugs.python.org/issue2135}: Deprecate find_module() and find_loader() implementations in
importlib and zipimport.

@item 
bpo-43534@footnote{https://bugs.python.org/issue43534}: @ref{1138,,turtle.textinput()} and @ref{1139,,turtle.numinput()} create now
a transient window working on behalf of the canvas window.

@item 
bpo-43532@footnote{https://bugs.python.org/issue43532}: Add the ability to specify keyword-only fields to dataclasses.
These fields will become keyword-only arguments to the generated __init__.

@item 
bpo-43522@footnote{https://bugs.python.org/issue43522}: Fix problem with
@ref{6ab,,hostname_checks_common_name}. OpenSSL does not copy
hostflags from @emph{struct SSL_CTX} to @emph{struct SSL}.

@item 
bpo-8978@footnote{https://bugs.python.org/issue8978}: Improve error message for @ref{9e5,,tarfile.open()} when @ref{a9,,lzma}
/ @ref{12,,bz2} are unavailable.  Patch by Anthony Sottile.

@item 
bpo-42967@footnote{https://bugs.python.org/issue42967}: Allow @ref{2f0,,bytes} @code{separator} argument in
@code{urllib.parse.parse_qs} and @code{urllib.parse.parse_qsl} when parsing
@ref{2ef,,str} query strings. Previously, this raised a @code{TypeError}.

@item 
bpo-43296@footnote{https://bugs.python.org/issue43296}: Improve @ref{ed,,sqlite3} error handling: @code{sqlite3_value_blob()}
errors that set @code{SQLITE_NOMEM} now raise @ref{10de,,MemoryError}. Patch by
Erlend E. Aasland.

@item 
bpo-43312@footnote{https://bugs.python.org/issue43312}: New functions @ref{113a,,sysconfig.get_preferred_scheme()} and
@ref{113b,,sysconfig.get_default_scheme()} are added to query a platform for its
preferred “user”, “home”, and “prefix” (default) scheme names.

@item 
bpo-43265@footnote{https://bugs.python.org/issue43265}: Improve @ref{6a7,,sqlite3.Connection.backup()} error handling. The
error message for non-existent target database names is now @code{unknown
database <database name>} instead of @code{SQL logic error}. Patch by Erlend
E. Aasland.

@item 
bpo-41282@footnote{https://bugs.python.org/issue41282}: Install schemes in @ref{49,,distutils.command.install} are now
loaded from @ref{f8,,sysconfig}.

@item 
bpo-41282@footnote{https://bugs.python.org/issue41282}: @ref{5e,,distutils.sysconfig} has been merged to @ref{f8,,sysconfig}.

@item 
bpo-43176@footnote{https://bugs.python.org/issue43176}: Fixed processing of a dataclass that inherits from a frozen
dataclass with no fields.  It is now correctly detected as an error.

@item 
bpo-43080@footnote{https://bugs.python.org/issue43080}: @ref{cd,,pprint} now has support for
@ref{276,,dataclasses.dataclass}. Patch by Lewis Gaul.

@item 
bpo-39950@footnote{https://bugs.python.org/issue39950}: Add @ref{2a4,,pathlib.Path.hardlink_to()} method that supersedes
@code{link_to()}. The new method has the same argument order as @code{symlink_to()}.

@item 
bpo-42904@footnote{https://bugs.python.org/issue42904}: @ref{432,,typing.get_type_hints()} now checks the local namespace of
a class when evaluating 
@geindex Python Enhancement Proposals; PEP 563
PEP 563@footnote{https://www.python.org/dev/peps/pep-0563} annotations inside said class.

@item 
bpo-42269@footnote{https://bugs.python.org/issue42269}: Add @code{slots} parameter to @code{dataclasses.dataclass} decorator
to automatically generate @code{__slots__} for class. Patch provided by Yurii
Karabas.

@item 
bpo-39529@footnote{https://bugs.python.org/issue39529}: Deprecated use of @ref{314,,asyncio.get_event_loop()} without
running event loop. Emit deprecation warning for @ref{9,,asyncio} functions
which implicitly create a @ref{316,,Future} or
@ref{317,,Task} objects if there is no running event loop and no
explicit @emph{loop} argument is passed: @ref{318,,ensure_future()},
@ref{319,,wrap_future()}, @ref{31a,,gather()},
@ref{31b,,shield()}, @ref{31c,,as_completed()} and constructors of
@ref{316,,Future}, @ref{317,,Task},
@ref{31d,,StreamReader}, @code{StreamReaderProtocol}.

@item 
bpo-18369@footnote{https://bugs.python.org/issue18369}: Certificate and PrivateKey classes were added to the ssl
module. Certificates and keys can now be loaded from memory buffer, too.

@item 
bpo-41486@footnote{https://bugs.python.org/issue41486}: Use a new output buffer management code for @ref{12,,bz2} /
@ref{a9,,lzma} / @ref{14b,,zlib} modules, and add @code{.readall()} function to
@code{_compression.DecompressReader} class. These bring some performance
improvements. Patch by Ma Lin.

@item 
bpo-31870@footnote{https://bugs.python.org/issue31870}: The @ref{2c0,,ssl.get_server_certificate()} function now has a
@emph{timeout} parameter.

@item 
bpo-41735@footnote{https://bugs.python.org/issue41735}: Fix thread locks in zlib module may go wrong in rare case.
Patch by Ma Lin.

@item 
bpo-36470@footnote{https://bugs.python.org/issue36470}: Fix dataclasses with @code{InitVar}s and
@ref{113c,,replace()}. Patch by Claudiu Popa.

@item 
bpo-40849@footnote{https://bugs.python.org/issue40849}: Expose X509_V_FLAG_PARTIAL_CHAIN ssl flag

@item 
bpo-35114@footnote{https://bugs.python.org/issue35114}: @ref{113d,,ssl.RAND_status()} now returns a boolean value (as
documented) instead of @code{1} or @code{0}.

@item 
bpo-39906@footnote{https://bugs.python.org/issue39906}: @ref{2a7,,pathlib.Path.stat()} and @ref{2a8,,chmod()} now
accept a @emph{follow_symlinks} keyword-only argument for consistency with
corresponding functions in the @ref{c0,,os} module.

@item 
bpo-39899@footnote{https://bugs.python.org/issue39899}: @ref{520,,os.path.expanduser()} now refuses to guess Windows home
directories if the basename of current user’s home directory does not
match their username.

@ref{9a9,,pathlib.Path.expanduser()} and @ref{9aa,,home()} now
consistently raise @ref{5af,,RuntimeError} exception when a home directory
cannot be resolved. Previously a @ref{5bb,,KeyError} exception could be raised
on Windows when the @code{"USERNAME"}  environment variable was unset.

@item 
bpo-36076@footnote{https://bugs.python.org/issue36076}: Added SNI support to @ref{2c0,,ssl.get_server_certificate()}.

@item 
bpo-38490@footnote{https://bugs.python.org/issue38490}: Covariance, Pearson’s correlation, and simple linear regression
functionality was added to statistics module. Patch by Tymoteusz Wołodźko.

@item 
bpo-33731@footnote{https://bugs.python.org/issue33731}: Provide a locale.localize() function, which converts a
normalized number string into a locale format.

@item 
bpo-32745@footnote{https://bugs.python.org/issue32745}: Fix a regression in the handling of ctypes’
@ref{113e,,ctypes.c_wchar_p} type: embedded null characters would cause a
@ref{17c,,ValueError} to be raised. Patch by Zackery Spytz.
@end itemize

@node Documentation<8>,Tests<7>,Library<7>,Python 3 10 0 beta 1
@anchor{whatsnew/changelog id47}@anchor{113f}
@subsubsection Documentation



@itemize -

@item 
bpo-43987@footnote{https://bugs.python.org/issue43987}: Add “Annotations Best Practices” document as a new HOWTO.

@item 
bpo-43977@footnote{https://bugs.python.org/issue43977}: Document the new @ref{110a,,Py_TPFLAGS_MAPPING} and
@ref{110b,,Py_TPFLAGS_SEQUENCE} type flags.

@item 
bpo-43959@footnote{https://bugs.python.org/issue43959}: The documentation on the PyContextVar C-API was clarified.

@item 
bpo-43938@footnote{https://bugs.python.org/issue43938}: Update dataclasses documentation to express that
FrozenInstanceError is derived from AttributeError.

@item 
bpo-43778@footnote{https://bugs.python.org/issue43778}: Fix the Sphinx glossary_search extension: create the _static/
sub-directory if it doesn’t exist.

@item 
bpo-43755@footnote{https://bugs.python.org/issue43755}: Update documentation to reflect that unparenthesized lambda
expressions can no longer be the expression part in an @code{if} clause in
comprehensions and generator expressions since Python 3.9.

@item 
bpo-43739@footnote{https://bugs.python.org/issue43739}: Fixing the example code in Doc/extending/extending.rst to
declare and initialize the pmodule variable to be of the right type.
@end itemize

@node Tests<7>,Build<7>,Documentation<8>,Python 3 10 0 beta 1
@anchor{whatsnew/changelog id48}@anchor{1140}
@subsubsection Tests



@itemize -

@item 
bpo-43961@footnote{https://bugs.python.org/issue43961}: Fix test_logging.test_namer_rotator_inheritance() on Windows:
use @ref{c86,,os.replace()} rather than @ref{c85,,os.rename()}. Patch by Victor
Stinner.

@item 
bpo-43842@footnote{https://bugs.python.org/issue43842}: Fix a race condition in the SMTP test of test_logging. Don’t
close a file descriptor (socket) from a different thread while
asyncore.loop() is polling the file descriptor. Patch by Victor Stinner.

@item 
bpo-43843@footnote{https://bugs.python.org/issue43843}: @code{test.libregrtest} now marks a test as ENV_CHANGED
(altered the execution environment) if a thread raises an exception but
does not catch it. It sets a hook on @ref{2d1,,threading.excepthook()}. Use
@code{--fail-env-changed} option to mark the test as failed. Patch by Victor
Stinner.

@item 
bpo-43811@footnote{https://bugs.python.org/issue43811}: Tests multiple OpenSSL versions on GitHub Actions. Use ccache
to speed up testing.

@item 
bpo-43791@footnote{https://bugs.python.org/issue43791}: OpenSSL 3.0.0: Disable testing of legacy protocols TLS 1.0 and
1.1. Tests are failing with TLSV1_ALERT_INTERNAL_ERROR.
@end itemize

@node Build<7>,Windows<7>,Tests<7>,Python 3 10 0 beta 1
@anchor{whatsnew/changelog id49}@anchor{1141}
@subsubsection Build



@itemize -

@item 
bpo-43567@footnote{https://bugs.python.org/issue43567}: Improved generated code refresh (AST/tokens/opcodes/keywords)
on Windows.

@item 
bpo-43669@footnote{https://bugs.python.org/issue43669}: Implement 
@geindex Python Enhancement Proposals; PEP 644
PEP 644@footnote{https://www.python.org/dev/peps/pep-0644}. Python now requires OpenSSL 1.1.1 or
newer.
@end itemize

@node Windows<7>,macOS<6>,Build<7>,Python 3 10 0 beta 1
@anchor{whatsnew/changelog id50}@anchor{1142}
@subsubsection Windows



@itemize -

@item 
bpo-35306@footnote{https://bugs.python.org/issue35306}: Adds additional arguments to @ref{1143,,os.startfile()} function.

@item 
bpo-43538@footnote{https://bugs.python.org/issue43538}: Avoid raising errors from @ref{523,,pathlib.Path.exists()} when
passed an invalid filename.

@item 
bpo-38822@footnote{https://bugs.python.org/issue38822}: Fixed @ref{516,,os.stat()} failing on inaccessible directories with
a trailing slash, rather than falling back to the parent directory’s
metadata. This implicitly affected @ref{51a,,os.path.exists()} and
@ref{51c,,os.path.isdir()}.

@item 
bpo-26227@footnote{https://bugs.python.org/issue26227}: Fixed decoding of host names in @ref{1144,,socket.gethostbyaddr()}
and @ref{1145,,socket.gethostbyname_ex()}.

@item 
bpo-40432@footnote{https://bugs.python.org/issue40432}: Updated pegen regeneration script on Windows to find and use
Python 3.8 or higher.  Prior to this, pegen regeneration already required
3.8 or higher, but the script may have used lower versions of Python.

@item 
bpo-43745@footnote{https://bugs.python.org/issue43745}: Actually updates Windows release to OpenSSL 1.1.1k. Earlier
releases were mislabelled and actually included 1.1.1i again.

@item 
bpo-43652@footnote{https://bugs.python.org/issue43652}: Update Tcl and Tk to 8.6.11 in Windows installer.

@item 
bpo-43492@footnote{https://bugs.python.org/issue43492}: Upgrade Windows installer to use SQLite 3.35.5.

@item 
bpo-30555@footnote{https://bugs.python.org/issue30555}: Fix @code{WindowsConsoleIO} errors in the presence of fd
redirection. Patch by Segev Finer.
@end itemize

@node macOS<6>,IDLE<7>,Windows<7>,Python 3 10 0 beta 1
@anchor{whatsnew/changelog id51}@anchor{1146}
@subsubsection macOS



@itemize -

@item 
bpo-42119@footnote{https://bugs.python.org/issue42119}: Fix check for macOS SDK paths when building Python. Narrow
search to match contents of SDKs, namely only files in
@code{/System/Library}, @code{/System/IOSSupport}, and @code{/usr} other than
@code{/usr/local}. Previously, anything under @code{/System} was assumed to be
in an SDK which causes problems with the new file system layout in 10.15+
where user file systems may appear to be mounted under @code{/System}.  Paths
in @code{/Library} were also incorrectly treated as SDK locations.

@item 
bpo-43568@footnote{https://bugs.python.org/issue43568}: Drop support for MACOSX_DEPLOYMENT_TARGET < 10.3

@item 
bpo-44009@footnote{https://bugs.python.org/issue44009}: Provide “python3.x-intel64” executable to allow reliably
forcing macOS universal2 framework builds to run under Rosetta 2 Intel-64
emulation on Apple Silicon Macs.  This can be useful for testing or when
universal2 wheels are not yet available.

@item 
bpo-43851@footnote{https://bugs.python.org/issue43851}: Build SQLite with @code{SQLITE_OMIT_AUTOINIT} on macOS. Patch by
Erlend E. Aasland.

@item 
bpo-43492@footnote{https://bugs.python.org/issue43492}: Update macOS installer to use SQLite 3.35.4.

@item 
bpo-42235@footnote{https://bugs.python.org/issue42235}: @code{Mac/BuildScript/build-installer.py} will now use
“–enable-optimizations” and @code{--with-lto} when building on macOS 10.15
or later.
@end itemize

@node IDLE<7>,C API<7>,macOS<6>,Python 3 10 0 beta 1
@anchor{whatsnew/changelog id52}@anchor{1147}
@subsubsection IDLE



@itemize -

@item 
bpo-37903@footnote{https://bugs.python.org/issue37903}: Add mouse actions to the shell sidebar.  Left click and
optional drag selects one or more lines, as with the editor line number
sidebar.  Right click after selecting raises a context menu with ‘copy
with prompts’.  This zips together prompts from the sidebar with lines
from the selected text.

@item 
bpo-43981@footnote{https://bugs.python.org/issue43981}: Fix reference leak in test_sidebar and test_squeezer. Patches
by Terry Jan Reedy and Pablo Galindo

@item 
bpo-37892@footnote{https://bugs.python.org/issue37892}: Indent IDLE Shell input with spaces instead of tabs

@item 
bpo-43655@footnote{https://bugs.python.org/issue43655}: IDLE dialog windows are now recognized as dialogs by window
managers on macOS and X Window.

@item 
bpo-37903@footnote{https://bugs.python.org/issue37903}: IDLE’s shell now shows prompts in a separate side-bar.
@end itemize

@node C API<7>,,IDLE<7>,Python 3 10 0 beta 1
@anchor{whatsnew/changelog id53}@anchor{1148}
@subsubsection C API



@itemize -

@item 
bpo-43916@footnote{https://bugs.python.org/issue43916}: Add a new @code{Py_TPFLAGS_DISALLOW_INSTANTIATION} type flag
to disallow creating type instances. Patch by Victor Stinner.

@item 
bpo-43774@footnote{https://bugs.python.org/issue43774}: Remove the now unused @code{PYMALLOC_DEBUG} macro. Debug hooks on
memory allocators are now installed by default if Python is built in debug
mode (if @code{Py_DEBUG} macro is defined). Moreover, they can now be used on
Python build in release mode (ex: using @code{PYTHONMALLOC=debug} environment
variable).

@item 
bpo-43962@footnote{https://bugs.python.org/issue43962}: _PyInterpreterState_IDIncref() now calls
_PyInterpreterState_IDInitref() and always increments id_refcount.
Previously, calling _xxsubinterpreters.get_current() could create an
id_refcount inconsistency when a _xxsubinterpreters.InterpreterID object
was deallocated. Patch by Victor Stinner.

@item 
bpo-28254@footnote{https://bugs.python.org/issue28254}: Add new C-API functions to control the state of the garbage
collector: @ref{37b,,PyGC_Enable()}, @ref{37c,,PyGC_Disable()},
@ref{37d,,PyGC_IsEnabled()}, corresponding to the functions in the
@ref{80,,gc} module.

@item 
bpo-43908@footnote{https://bugs.python.org/issue43908}: Introduce @ref{1ef,,Py_TPFLAGS_IMMUTABLETYPE} flag for immutable
type objects, and modify @ref{1ec,,PyType_Ready()} to set it for static
types. Patch by Erlend E. Aasland.

@item 
bpo-43795@footnote{https://bugs.python.org/issue43795}: @ref{a1f,,PyMem_Calloc()} is now available in the limited C API
(@code{Py_LIMITED_API}).

@item 
bpo-43868@footnote{https://bugs.python.org/issue43868}: @ref{3a2,,PyOS_ReadlineFunctionPointer()} is no longer exported by
limited C API headers and by @code{python3.dll} on Windows. Like any function
that takes @code{FILE*}, it is not part of the stable ABI.

@item 
bpo-43795@footnote{https://bugs.python.org/issue43795}: Stable ABI and limited API definitions are generated from a
central manifest (
@geindex Python Enhancement Proposals; PEP 652
PEP 652@footnote{https://www.python.org/dev/peps/pep-0652}).

@item 
bpo-43753@footnote{https://bugs.python.org/issue43753}: Add the @ref{377,,Py_Is(x@comma{} y)} function to test if the
@emph{x} object is the @emph{y} object, the same as @code{x is y} in Python. Add also
the @ref{378,,Py_IsNone()}, @ref{379,,Py_IsTrue()}, @ref{37a,,Py_IsFalse()}
functions to test if an object is, respectively, the @code{None} singleton,
the @code{True} singleton or the @code{False} singleton. Patch by Victor
Stinner.
@end itemize

@node Python 3 10 0 alpha 7,Python 3 10 0 alpha 6,Python 3 10 0 beta 1,Changelog
@anchor{whatsnew/changelog python-3-10-0-alpha-7}@anchor{1149}
@subsection Python 3.10.0 alpha 7


@emph{Release date: 2021-04-05}

@menu
* Security: Security<3>. 
* Core and Builtins: Core and Builtins<8>. 
* Library: Library<8>. 
* Documentation: Documentation<9>. 
* Tests: Tests<8>. 
* Build: Build<8>. 
* Windows: Windows<8>. 
* IDLE: IDLE<8>. 
* C API: C API<8>. 

@end menu

@node Security<3>,Core and Builtins<8>,,Python 3 10 0 alpha 7
@anchor{whatsnew/changelog id54}@anchor{114a}
@subsubsection Security



@itemize -

@item 
bpo-42988@footnote{https://bugs.python.org/issue42988}: CVE-2021-3426: Remove the @code{getfile} feature of the
@ref{d4,,pydoc} module which could be abused to read arbitrary files on the
disk (directory traversal vulnerability). Moreover, even source code of
Python modules can contain sensitive data like passwords. Vulnerability
reported by David Schwörer.

@item 
bpo-43285@footnote{https://bugs.python.org/issue43285}: @ref{7e,,ftplib} no longer trusts the IP address value returned
from the server in response to the PASV command by default.  This prevents
a malicious FTP server from using the response to probe IPv4 address and
port combinations on the client network.

Code that requires the former vulnerable behavior may set a
@code{trust_server_pasv_ipv4_address} attribute on their @ref{3e9,,ftplib.FTP}
instances to @code{True} to re-enable it.

@item 
bpo-43439@footnote{https://bugs.python.org/issue43439}: Add audit hooks for @ref{284,,gc.get_objects()},
@ref{285,,gc.get_referrers()} and @ref{286,,gc.get_referents()}. Patch by Pablo
Galindo.
@end itemize

@node Core and Builtins<8>,Library<8>,Security<3>,Python 3 10 0 alpha 7
@anchor{whatsnew/changelog id55}@anchor{114b}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-27129@footnote{https://bugs.python.org/issue27129}: Update CPython bytecode magic number.

@item 
bpo-43672@footnote{https://bugs.python.org/issue43672}: Raise ImportWarning when calling find_loader().

@item 
bpo-43660@footnote{https://bugs.python.org/issue43660}: Fix crash that happens when replacing @code{sys.stderr} with a
callable that can remove the object while an exception is being printed.
Patch by Pablo Galindo.

@item 
bpo-27129@footnote{https://bugs.python.org/issue27129}: The bytecode interpreter uses instruction, rather byte, offsets
internally. This reduces the number of EXTENDED_ARG instructions needed
and streamlines instruction dispatch a bit.

@item 
bpo-40645@footnote{https://bugs.python.org/issue40645}: Fix reference leak in the @code{_hashopenssl} extension. Patch
by Pablo Galindo.

@item 
bpo-42134@footnote{https://bugs.python.org/issue42134}: Calls to find_module() by the import system now raise
ImportWarning.

@item 
bpo-41064@footnote{https://bugs.python.org/issue41064}: Improve the syntax error for invalid usage of double starred
elements (‘**’) in f-strings. Patch by Pablo Galindo.

@item 
bpo-43575@footnote{https://bugs.python.org/issue43575}: Speed up calls to @code{map()} by using the 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590}
@code{vectorcall} calling convention. Patch by Dong-hee Na.

@item 
bpo-42137@footnote{https://bugs.python.org/issue42137}: The import system now prefers using @code{__spec__} for
@code{ModuleType.__repr__} over @code{module_repr()}.

@item 
bpo-43452@footnote{https://bugs.python.org/issue43452}: Added micro-optimizations to @code{_PyType_Lookup()} to improve
cache lookup performance in the common case of cache hits.

@item 
bpo-43555@footnote{https://bugs.python.org/issue43555}: Report the column offset for @ref{21a,,SyntaxError} for invalid
line continuation characters. Patch by Pablo Galindo.

@item 
bpo-43517@footnote{https://bugs.python.org/issue43517}: Fix misdetection of circular imports when using @code{from pkg.mod
import attr}, which caused false positives in non-trivial multi-threaded
code.

@item 
bpo-43497@footnote{https://bugs.python.org/issue43497}: Emit SyntaxWarnings for assertions with tuple constants, this
is a regression introduced in python3.7

@item 
bpo-39316@footnote{https://bugs.python.org/issue39316}: Tracing now has correct line numbers for attribute accesses
when the attribute is on a different line from the object. Improves
debugging and profiling for multi-line method chains.

@item 
bpo-35883@footnote{https://bugs.python.org/issue35883}: Python no longer fails at startup with a fatal error if a
command line argument contains an invalid Unicode character. The
@ref{6fe,,Py_DecodeLocale()} function now escapes byte sequences which would
be decoded as Unicode characters outside the [U+0000; U+10ffff] range.

@item 
bpo-43410@footnote{https://bugs.python.org/issue43410}: Fix a bug that was causing the parser to crash when emitting
syntax errors when reading input from stdin. Patch by Pablo Galindo

@item 
bpo-43406@footnote{https://bugs.python.org/issue43406}: Fix a possible race condition where @code{PyErr_CheckSignals}
tries to execute a non-Python signal handler.

@item 
bpo-42128@footnote{https://bugs.python.org/issue42128}: Add @code{__match_args__} to @code{structsequence} based
classes. Patch by Pablo Galindo.

@item 
bpo-43390@footnote{https://bugs.python.org/issue43390}: CPython now sets the @code{SA_ONSTACK} flag in @code{PyOS_setsig} for
the VM’s default signal handlers.  This is friendlier to other in-process
code that an extension module or embedding use could pull in (such as
Golang’s cgo) where tiny thread stacks are the norm and @code{sigaltstack()}
has been used to provide for signal handlers.  This is a no-op change for
the vast majority of processes that don’t use sigaltstack.

@item 
bpo-43287@footnote{https://bugs.python.org/issue43287}: Speed up calls to @code{filter()} by using the 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590}
@code{vectorcall} calling convention. Patch by Dong-hee Na.

@item 
bpo-37448@footnote{https://bugs.python.org/issue37448}: Add a radix tree based memory map to track in-use obmalloc
arenas. Use to replace the old implementation of address_in_range(). The
radix tree approach makes it easy to increase pool sizes beyond the OS
page size. Boosting the pool and arena size allows obmalloc to handle a
significantly higher percentage of requests from its ultra-fast paths.

It also has the advantage of eliminating the memory unsanitary behavior of
the previous address_in_range(). The old address_in_range() was marked
with the annotations _Py_NO_SANITIZE_ADDRESS, _Py_NO_SANITIZE_THREAD, and
_Py_NO_SANITIZE_MEMORY. Those annotations are no longer needed.

To disable the radix tree map, set a preprocessor flag as follows:
@code{-DWITH_PYMALLOC_RADIX_TREE=0}.

Co-authored-by: Tim Peters <@email{tim.peters@@gmail.com}>

@item 
bpo-29988@footnote{https://bugs.python.org/issue29988}: Only handle asynchronous exceptions and requests to drop the
GIL when returning from a call or on the back edges of loops. Makes sure
that @code{__exit__()} is always called in with statements, even for
interrupts.
@end itemize

@node Library<8>,Documentation<9>,Core and Builtins<8>,Python 3 10 0 alpha 7
@anchor{whatsnew/changelog id56}@anchor{114c}
@subsubsection Library



@itemize -

@item 
bpo-43720@footnote{https://bugs.python.org/issue43720}: Document various stdlib deprecations in imp, pkgutil, and
importlib.util for removal in Python 3.12.

@item 
bpo-43433@footnote{https://bugs.python.org/issue43433}: @ref{56c,,xmlrpc.client.ServerProxy} no longer ignores query and
fragment in the URL of the server.

@item 
bpo-31956@footnote{https://bugs.python.org/issue31956}: The @ref{25d,,index()} method of @ref{25e,,array.array}
now has optional @emph{start} and @emph{stop} parameters.

@item 
bpo-40066@footnote{https://bugs.python.org/issue40066}: Enum: adjust @code{repr()} to show only enum and member name (not
value, nor angle brackets) and @code{str()} to show only member name.  Update
and improve documentation to match.

@item 
bpo-42136@footnote{https://bugs.python.org/issue42136}: Deprecate all module_repr() methods found in importlib as their
use is being phased out by Python 3.12.

@item 
bpo-35930@footnote{https://bugs.python.org/issue35930}: Raising an exception raised in a “future” instance will create
reference cycles.

@item 
bpo-41369@footnote{https://bugs.python.org/issue41369}: Finish updating the vendored libmpdec to version 2.5.1.  Patch
by Stefan Krah.

@item 
bpo-43422@footnote{https://bugs.python.org/issue43422}: Revert the _decimal C API which was added in bpo-41324@footnote{https://bugs.python.org/issue41324}.

@item 
bpo-43577@footnote{https://bugs.python.org/issue43577}: Fix deadlock when using @ref{2bf,,ssl.SSLContext} debug callback
with @ref{114d,,ssl.SSLContext.sni_callback()}.

@item 
bpo-43571@footnote{https://bugs.python.org/issue43571}: It’s now possible to create MPTCP sockets with IPPROTO_MPTCP

@item 
bpo-43542@footnote{https://bugs.python.org/issue43542}: @code{image/heic} and @code{image/heif} were added to
@ref{ae,,mimetypes}.

@item 
bpo-40645@footnote{https://bugs.python.org/issue40645}: The @ref{8a,,hmac} module now uses OpenSSL’s HMAC implementation
when digestmod argument is a hash name or builtin hash function.

@item 
bpo-43510@footnote{https://bugs.python.org/issue43510}: Implement 
@geindex Python Enhancement Proposals; PEP 597
PEP 597@footnote{https://www.python.org/dev/peps/pep-0597}: Add @code{EncodingWarning} warning, @code{-X
warn_default_encoding} option, 
@geindex PYTHONWARNDEFAULTENCODING
@geindex environment variable; PYTHONWARNDEFAULTENCODING
@ref{230,,PYTHONWARNDEFAULTENCODING}
environment variable and @code{encoding="locale"} argument value.

@item 
bpo-43521@footnote{https://bugs.python.org/issue43521}: @code{ast.unparse} can now render NaNs and empty sets.

@item 
bpo-42914@footnote{https://bugs.python.org/issue42914}: @ref{2ac,,pprint.pprint()} gains a new boolean
@code{underscore_numbers} optional argument to emit integers with thousands
separated by an underscore character for improved readability (for example
@code{1_000_000} instead of @code{1000000}).

@item 
bpo-41361@footnote{https://bugs.python.org/issue41361}: @ref{114e,,rotate()} calls are now slightly faster
due to faster argument parsing.

@item 
bpo-43423@footnote{https://bugs.python.org/issue43423}: @code{subprocess.communicate()} no longer raises an IndexError
when there is an empty stdout or stderr IO buffer during a timeout on
Windows.

@item 
bpo-27820@footnote{https://bugs.python.org/issue27820}: Fixed long-standing bug of smtplib.SMTP where doing AUTH LOGIN
with initial_response_ok=False will fail.

The cause is that SMTP.auth_login _always_ returns a password if provided
with a challenge string, thus non-compliant with the standard for AUTH
LOGIN.

Also fixes bug with the test for smtpd.

@item 
bpo-43445@footnote{https://bugs.python.org/issue43445}: Add frozen modules to @ref{2c7,,sys.stdlib_module_names}. For
example, add @code{"_frozen_importlib"} and @code{"_frozen_importlib_external"}
names.

@item 
bpo-43245@footnote{https://bugs.python.org/issue43245}: Add keyword arguments support to @code{ChainMap.new_child()}.

@item 
bpo-29982@footnote{https://bugs.python.org/issue29982}: Add optional parameter @emph{ignore_cleanup_errors} to
@ref{de5,,tempfile.TemporaryDirectory()} and allow multiple @code{cleanup()}
attempts. Contributed by C.A.M. Gerlach.

@item 
bpo-43428@footnote{https://bugs.python.org/issue43428}: Include changes from importlib_metadata 3.7@footnote{https://importlib-metadata.readthedocs.io/en/latest/history.html#v3-7-0}:

Performance enhancements to distribution discovery.

@code{entry_points} only returns unique distributions.

Introduces new @code{EntryPoints} object for containing a set of entry points
with convenience methods for selecting entry points by group or name.
@code{entry_points} now returns this object if selection parameters are
supplied but continues to return a dict object for compatibility. Users
are encouraged to rely on the selection interface. The dict object result
is likely to be deprecated in the future.

Added packages_distributions function to return a mapping of packages to
the distributions that provide them.

@item 
bpo-43332@footnote{https://bugs.python.org/issue43332}: Improves the networking efficiency of @ref{8f,,http.client} when
using a proxy via @code{set_tunnel()}.  Fewer small send
calls are made during connection setup.

@item 
bpo-43420@footnote{https://bugs.python.org/issue43420}: Improve performance of @ref{172,,fractions.Fraction} arithmetics
for large components.  Contributed by Sergey B. Kirpichev.

@item 
bpo-43356@footnote{https://bugs.python.org/issue43356}: Allow passing a signal number to @code{_thread.interrupt_main()}.

@item 
bpo-43399@footnote{https://bugs.python.org/issue43399}: Fix @code{ElementTree.extend} not working on iterators when using
the Python implementation

@item 
bpo-43369@footnote{https://bugs.python.org/issue43369}: Improve @ref{ed,,sqlite3} error handling: If
@code{sqlite3_column_text()} and @code{sqlite3_column_blob()} set
@code{SQLITE_NOMEM}, @ref{10de,,MemoryError} is now raised. Patch by Erlend E.
Aasland.

@item 
bpo-43368@footnote{https://bugs.python.org/issue43368}: Fix a regression introduced in GH-24562, where an empty
bytestring was fetched as @code{None} instead of @code{b'@w{'}} in @ref{ed,,sqlite3}.
Patch by Mariusz Felisiak.

@item 
bpo-41282@footnote{https://bugs.python.org/issue41282}: Fixed stacklevel of @code{DeprecationWarning} emitted from
@code{import distutils}.

@item 
bpo-42129@footnote{https://bugs.python.org/issue42129}: @code{importlib.resources} now honors namespace packages, merging
resources from each location in the namespace as introduced in
@code{importlib_resources} 3.2 and including incidental changes through
5.0.3.

@item 
bpo-43295@footnote{https://bugs.python.org/issue43295}: @ref{114f,,datetime.datetime.strptime()} now raises @code{ValueError}
instead of @code{IndexError} when matching @code{'z'} with the @code{%z} format
specifier.

@item 
bpo-43125@footnote{https://bugs.python.org/issue43125}: Return empty string if base64mime.body_encode receive empty
bytes

@item 
bpo-43084@footnote{https://bugs.python.org/issue43084}: @ref{1150,,curses.window.enclose()} returns now @code{True} or @code{False}
(as was documented) instead of @code{1} or @code{0}.

@item 
bpo-42994@footnote{https://bugs.python.org/issue42994}: Add MIME types for opus, AAC, 3gpp and 3gpp2

@item 
bpo-14678@footnote{https://bugs.python.org/issue14678}: Add an invalidate_caches() method to the zipimport.zipimporter
class to support importlib.invalidate_caches(). Patch by Desmond Cheong.

@item 
bpo-42782@footnote{https://bugs.python.org/issue42782}: Fail fast in @ref{572,,shutil.move()} to avoid creating
destination directories on failure.

@item 
bpo-40066@footnote{https://bugs.python.org/issue40066}: Enum’s @ref{27d,,repr()} and @ref{2ef,,str()} have changed: @ref{27d,,repr()} is now
@emph{EnumClass.MemberName} and @ref{2ef,,str()} is @emph{MemberName}.  Additionally, stdlib
Enum’s whose contents are available as module attributes, such as
@code{RegexFlag.IGNORECASE}, have their @ref{27d,,repr()} as @emph{module.name}, e.g.
@ref{1151,,re.IGNORECASE}.

@item 
bpo-26053@footnote{https://bugs.python.org/issue26053}: Fixed bug where the @ref{c4,,pdb} interactive run command echoed
the args from the shell command line, even if those have been overridden
at the pdb prompt.

@item 
bpo-24160@footnote{https://bugs.python.org/issue24160}: Fixed bug where breakpoints did not persist across multiple
debugger sessions in @ref{c4,,pdb}’s interactive mode.

@item 
bpo-40701@footnote{https://bugs.python.org/issue40701}: When the @ref{1152,,tempfile.tempdir} global variable is set to a
value of type bytes, it is now handled consistently.  Previously
exceptions could be raised from some tempfile APIs when the directory did
not already exist in this situation.  Also ensures that the
@ref{1153,,tempfile.gettempdir()} and @ref{1154,,tempfile.gettempdirb()} functions
@emph{always} return @code{str} and @code{bytes} respectively.

@item 
bpo-39342@footnote{https://bugs.python.org/issue39342}: Expose @code{X509_V_FLAG_ALLOW_PROXY_CERTS} as
@ref{1155,,VERIFY_ALLOW_PROXY_CERTS} to allow proxy certificate
validation as explained in
@indicateurl{https://www.openssl.org/docs/man1.1.1/man7/proxy-certificates.html}.

@item 
bpo-31861@footnote{https://bugs.python.org/issue31861}: Add builtins.aiter and builtins.anext. Patch by Joshua Bronson
(@@jab), Daniel Pope (@@lordmauve), and Justin Wang (@@justin39).
@end itemize

@node Documentation<9>,Tests<8>,Library<8>,Python 3 10 0 alpha 7
@anchor{whatsnew/changelog id57}@anchor{1156}
@subsubsection Documentation



@itemize -

@item 
bpo-43199@footnote{https://bugs.python.org/issue43199}: Answer “Why is there no goto?” in the Design and History FAQ.

@item 
bpo-43407@footnote{https://bugs.python.org/issue43407}: Clarified that a result from @ref{9ed,,time.monotonic()},
@ref{5a0,,time.perf_counter()}, @ref{5a1,,time.process_time()}, or
@ref{42a,,time.thread_time()} can be compared with the result from any
following call to the same function - not just the next immediate call.

@item 
bpo-43354@footnote{https://bugs.python.org/issue43354}: Fix type documentation for @code{Fault.faultCode}; the type has to
be @code{int} instead of @code{str}.

@item 
bpo-41933@footnote{https://bugs.python.org/issue41933}: Clarified wording of s * n in the Common Sequence Operations
@end itemize

@node Tests<8>,Build<8>,Documentation<9>,Python 3 10 0 alpha 7
@anchor{whatsnew/changelog id58}@anchor{1157}
@subsubsection Tests



@itemize -

@item 
bpo-37945@footnote{https://bugs.python.org/issue37945}: Fix test_getsetlocale_issue1813() of test_locale: skip the test
if @code{setlocale()} fails. Patch by Victor Stinner.

@item 
bpo-41561@footnote{https://bugs.python.org/issue41561}: Add workaround for Ubuntu’s custom OpenSSL security level
policy.
@end itemize

@node Build<8>,Windows<8>,Tests<8>,Python 3 10 0 alpha 7
@anchor{whatsnew/changelog id59}@anchor{1158}
@subsubsection Build



@itemize -

@item 
bpo-43179@footnote{https://bugs.python.org/issue43179}: Introduce and correctly use ALIGNOF_X in place of SIZEOF_X for
alignment-related code in optimized string routines. Patch by Jessica
Clarke.

@item 
bpo-43631@footnote{https://bugs.python.org/issue43631}: Update macOS, Windows, and CI to OpenSSL 1.1.1k.

@item 
bpo-43617@footnote{https://bugs.python.org/issue43617}: Improve configure.ac: Check for presence of autoconf-archive
package and remove our copies of M4 macros.

@item 
bpo-43466@footnote{https://bugs.python.org/issue43466}: The @code{configure} script now supports @code{--with-openssl-rpath}
option.

@item 
bpo-43372@footnote{https://bugs.python.org/issue43372}: Use @code{_freeze_importlib} to generate code for the
@code{__hello__} module. This approach ensures the code matches the
interpreter version.  Previously, PYTHON_FOR_REGEN was used to generate
the code, which might be wrong.  The marshal format for code objects has
changed with bpo-42246@footnote{https://bugs.python.org/issue42246}, commit 877df851. Update the code and the expected
code sizes in ctypes test_frozentable.
@end itemize

@node Windows<8>,IDLE<8>,Build<8>,Python 3 10 0 alpha 7
@anchor{whatsnew/changelog id60}@anchor{1159}
@subsubsection Windows



@itemize -

@item 
bpo-43440@footnote{https://bugs.python.org/issue43440}: Build @ref{ed,,sqlite3} with the @code{R*Tree} module enabled. Patch
by Erlend E. Aasland.
@end itemize

@node IDLE<8>,C API<8>,Windows<8>,Python 3 10 0 alpha 7
@anchor{whatsnew/changelog id61}@anchor{115a}
@subsubsection IDLE



@itemize -

@item 
bpo-42225@footnote{https://bugs.python.org/issue42225}: Document that IDLE can fail on Unix either from misconfigured
IP masquerade rules or failure displaying complex colored (non-ascii)
characters.
@end itemize

@node C API<8>,,IDLE<8>,Python 3 10 0 alpha 7
@anchor{whatsnew/changelog id62}@anchor{115b}
@subsubsection C API



@itemize -

@item 
bpo-43688@footnote{https://bugs.python.org/issue43688}: The limited C API is now supported if Python is built in debug
mode (if the @code{Py_DEBUG} macro is defined). In the limited C API, the
@ref{372,,Py_INCREF()} and @ref{373,,Py_DECREF()} functions are now implemented
as opaque function calls, rather than accessing directly the
@ref{374,,PyObject.ob_refcnt} member, if Python is built in debug mode
and the @code{Py_LIMITED_API} macro targets Python 3.10 or newer. It became
possible to support the limited C API in debug mode because the
@ref{375,,PyObject} structure is the same in release and debug mode since
Python 3.8 (see bpo-36465@footnote{https://bugs.python.org/issue36465}).

The limited C API is still not supported in the @code{--with-trace-refs}
special build (@code{Py_TRACE_REFS} macro).

Patch by Victor Stinner.

@item 
bpo-43244@footnote{https://bugs.python.org/issue43244}: Remove the @code{pyarena.h} header file with functions:


@itemize *

@item 
@code{PyArena_New()}

@item 
@code{PyArena_Free()}

@item 
@code{PyArena_Malloc()}

@item 
@code{PyArena_AddPyObject()}
@end itemize

These functions were undocumented, excluded from the limited C API, and
were only used internally by the compiler. Patch by Victor Stinner.

@item 
bpo-43244@footnote{https://bugs.python.org/issue43244}: Remove the compiler and parser functions using @code{struct _mod}
type, because the public AST C API was removed:


@itemize *

@item 
@code{PyAST_Compile()}

@item 
@code{PyAST_CompileEx()}

@item 
@code{PyAST_CompileObject()}

@item 
@code{PyFuture_FromAST()}

@item 
@code{PyFuture_FromASTObject()}

@item 
@code{PyParser_ASTFromFile()}

@item 
@code{PyParser_ASTFromFileObject()}

@item 
@code{PyParser_ASTFromFilename()}

@item 
@code{PyParser_ASTFromString()}

@item 
@code{PyParser_ASTFromStringObject()}
@end itemize

These functions were undocumented and excluded from the limited C API.
Patch by Victor Stinner.

@item 
bpo-43244@footnote{https://bugs.python.org/issue43244}: Remove @code{ast.h}, @code{asdl.h}, and @code{Python-ast.h} header
files. These functions were undocumented and excluded from the limited C
API. Most names defined by these header files were not prefixed by @code{Py}
and so could create names conflicts. For example, @code{Python-ast.h} defined
a @code{Yield} macro which was conflict with the @code{Yield} name used by the
Windows @code{<winbase.h>} header. Use the Python @ref{7,,ast} module instead.
Patch by Victor Stinner.

@item 
bpo-43541@footnote{https://bugs.python.org/issue43541}: Fix a @code{PyEval_EvalCodeEx()} regression: fix reference
counting on builtins. Patch by Victor Stinner.

@item 
bpo-43244@footnote{https://bugs.python.org/issue43244}: Remove the @code{symtable.h} header file and the undocumented
functions:


@itemize *

@item 
@code{PyST_GetScope()}

@item 
@code{PySymtable_Build()}

@item 
@code{PySymtable_BuildObject()}

@item 
@code{PySymtable_Free()}

@item 
@code{Py_SymtableString()}

@item 
@code{Py_SymtableStringObject()}
@end itemize

The @code{Py_SymtableString()} function was part the stable ABI by mistake
but it could not be used, because the @code{symtable.h} header file was
excluded from the limited C API.

The Python @ref{f6,,symtable} module remains available and is unchanged.

Patch by Victor Stinner.

@item 
bpo-43244@footnote{https://bugs.python.org/issue43244}: Remove the @code{PyAST_Validate()} function. It is no longer
possible to build a AST object (@code{mod_ty} type) with the public C API.
The function was already excluded from the limited C API (
@geindex Python Enhancement Proposals; PEP 384
PEP 384@footnote{https://www.python.org/dev/peps/pep-0384}).
Patch by Victor Stinner.
@end itemize

@node Python 3 10 0 alpha 6,Python 3 10 0 alpha 5,Python 3 10 0 alpha 7,Changelog
@anchor{whatsnew/changelog python-3-10-0-alpha-6}@anchor{115c}
@subsection Python 3.10.0 alpha 6


@emph{Release date: 2021-03-01}

@menu
* Security: Security<4>. 
* Core and Builtins: Core and Builtins<9>. 
* Library: Library<9>. 
* Documentation: Documentation<10>. 
* Tests: Tests<9>. 
* Build: Build<9>. 
* Windows: Windows<9>. 
* macOS: macOS<7>. 
* IDLE: IDLE<9>. 
* C API: C API<9>. 

@end menu

@node Security<4>,Core and Builtins<9>,,Python 3 10 0 alpha 6
@anchor{whatsnew/changelog id63}@anchor{115d}
@subsubsection Security



@itemize -

@item 
bpo-42967@footnote{https://bugs.python.org/issue42967}: Fix web cache poisoning vulnerability by defaulting the query
args separator to @code{&}, and allowing the user to choose a custom
separator.
@end itemize

@node Core and Builtins<9>,Library<9>,Security<4>,Python 3 10 0 alpha 6
@anchor{whatsnew/changelog id64}@anchor{115e}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-43321@footnote{https://bugs.python.org/issue43321}: Fix @code{SystemError} raised when @code{PyArg_Parse*()} is used with
@code{#} but without @code{PY_SSIZE_T_CLEAN} defined.

@item 
bpo-36346@footnote{https://bugs.python.org/issue36346}: @code{PyArg_Parse*()} functions now emits @code{DeprecationWarning}
when @code{u} or @code{Z} format is used. See 
@geindex Python Enhancement Proposals; PEP 623
PEP 623@footnote{https://www.python.org/dev/peps/pep-0623} for detail.

@item 
bpo-43277@footnote{https://bugs.python.org/issue43277}: Add a new @ref{36f,,PySet_CheckExact()} function to the C-API to
check if an object is an instance of @ref{370,,set} but not an instance of a
subtype. Patch by Pablo Galindo.

@item 
bpo-42990@footnote{https://bugs.python.org/issue42990}: The @ref{349,,types.FunctionType} constructor now inherits the
current builtins if the @emph{globals} dictionary has no @code{"__builtins__"}
key, rather than using @code{@{"None": None@}} as builtins: same behavior as
@ref{34a,,eval()} and @ref{34b,,exec()} functions. Defining a function with @code{def
function(...): ...} in Python is not affected, globals cannot be
overridden with this syntax: it also inherits the current builtins. Patch
by Victor Stinner.

@item 
bpo-42990@footnote{https://bugs.python.org/issue42990}: Functions have a new @code{__builtins__} attribute which is used
to look for builtin symbols when a function is executed, instead of
looking into @code{__globals__['__builtins__']}. Patch by Mark Shannon and
Victor Stinner.

@item 
bpo-43149@footnote{https://bugs.python.org/issue43149}: Improve the error message in the parser for exception groups
without parentheses. Patch by Pablo Galindo.

@item 
bpo-43121@footnote{https://bugs.python.org/issue43121}: Fixed an incorrect @ref{21a,,SyntaxError} message for missing comma
in literals. Patch by Pablo Galindo.

@item 
bpo-42819@footnote{https://bugs.python.org/issue42819}: @ref{d9,,readline}: Explicitly disable bracketed paste in the
interactive interpreter, even if it’s set in the inputrc, is enabled by
default (eg GNU Readline 8.1), or a user calls
@code{readline.read_init_file()}. The Python REPL has not implemented
bracketed paste support. Also, bracketed mode writes the @code{"\x1b[?2004h"}
escape sequence into stdout which causes test failures in applications
that don’t support it. It can still be explicitly enabled by calling
@code{readline.parse_and_bind("set enable-bracketed-paste on")}. Patch by
Dustin Rodrigues.

@item 
bpo-42808@footnote{https://bugs.python.org/issue42808}: Simple calls to @code{type(object)} are now faster due to the
@code{vectorcall} calling convention. Patch by Dennis Sweeney.

@item 
bpo-42217@footnote{https://bugs.python.org/issue42217}: Make the compiler merges same co_code and co_linetable objects
in a module like already did for co_consts.

@item 
bpo-41972@footnote{https://bugs.python.org/issue41972}: Substring search functions such as @code{str1 in str2} and
@code{str2.find(str1)} now sometimes use the “Two-Way” string comparison
algorithm to avoid quadratic behavior on long strings.

@item 
bpo-42128@footnote{https://bugs.python.org/issue42128}: Implement 
@geindex Python Enhancement Proposals; PEP 634
PEP 634@footnote{https://www.python.org/dev/peps/pep-0634} (structural pattern matching). Patch by
Brandt Bucher.

@item 
bpo-40692@footnote{https://bugs.python.org/issue40692}: In the @ref{3da,,concurrent.futures.ProcessPoolExecutor},
validate that @code{multiprocess.synchronize()} is available on a given
platform and rely on that check in the @ref{20,,concurrent.futures} test
suite so we can run tests that are unrelated to
@code{ProcessPoolExecutor} on those platforms.

@item 
bpo-38302@footnote{https://bugs.python.org/issue38302}: If @ref{24b,,object.__ipow__()} returns @ref{24c,,NotImplemented}, the
operator will correctly fall back to @ref{24d,,object.__pow__()} and
@ref{24e,,object.__rpow__()} as expected.
@end itemize

@node Library<9>,Documentation<10>,Core and Builtins<9>,Python 3 10 0 alpha 6
@anchor{whatsnew/changelog id65}@anchor{115f}
@subsubsection Library



@itemize -

@item 
bpo-43316@footnote{https://bugs.python.org/issue43316}: The @code{python -m gzip} command line application now properly
fails when detecting an unsupported extension. It exits with a non-zero
exit code and prints an error message to stderr.

@item 
bpo-43317@footnote{https://bugs.python.org/issue43317}: Set the chunk size for the @code{gzip} module main function to
io.DEFAULT_BUFFER_SIZE. This is slightly faster than the 1024 bytes
constant that was used previously.

@item 
bpo-43146@footnote{https://bugs.python.org/issue43146}: Handle None in single-arg versions of
@ref{2d5,,print_exception()} and
@ref{2d3,,format_exception()}.

@item 
bpo-43260@footnote{https://bugs.python.org/issue43260}: Fix TextIOWrapper can not flush internal buffer forever after
very large text is written.

@item 
bpo-43258@footnote{https://bugs.python.org/issue43258}: Prevent needless allocation of @ref{ed,,sqlite3} aggregate
function context when no rows match an aggregate query. Patch by Erlend E.
Aasland.

@item 
bpo-43251@footnote{https://bugs.python.org/issue43251}: Improve @ref{ed,,sqlite3} error handling:
@code{sqlite3_column_name()} failures now result in @ref{10de,,MemoryError}. Patch
by Erlend E. Aasland.

@item 
bpo-40956@footnote{https://bugs.python.org/issue40956}: Fix segfault in @ref{6a7,,sqlite3.Connection.backup()} if no
argument was provided. The regression was introduced by GH-23838. Patch by
Erlend E. Aasland.

@item 
bpo-43172@footnote{https://bugs.python.org/issue43172}: The readline module now passes its tests when built directly
against libedit. Existing irreconcilable API differences remain in
@ref{1160,,readline.get_begidx()} and @ref{1161,,readline.get_endidx()} behavior based
on libreadline vs libedit use.

@item 
bpo-43163@footnote{https://bugs.python.org/issue43163}: Fix a bug in @ref{1b,,codeop} that was causing it to not ask for
more input when multi-line snippets have unclosed parentheses. Patch by
Pablo Galindo

@item 
bpo-43162@footnote{https://bugs.python.org/issue43162}: deprecate unsupported ability to access enum members as
attributes of other enum members

@item 
bpo-43146@footnote{https://bugs.python.org/issue43146}: Fix recent regression in None argument handling in
@ref{11a,,traceback} module functions.

@item 
bpo-43102@footnote{https://bugs.python.org/issue43102}: The namedtuple __new__ method had its __builtins__ set to None
instead of an actual dictionary.  This created problems for introspection
tools.

@item 
bpo-43106@footnote{https://bugs.python.org/issue43106}: Added @ref{29b,,O_EVTONLY}, @ref{29c,,O_FSYNC},
@ref{29d,,O_SYMLINK} and @ref{29e,,O_NOFOLLOW_ANY} for macOS. Patch by
Dong-hee Na.

@item 
bpo-42960@footnote{https://bugs.python.org/issue42960}: Adds @ref{1162,,resource.RLIMIT_KQUEUES} constant from FreeBSD to
the @ref{db,,resource} module.

@item 
bpo-42151@footnote{https://bugs.python.org/issue42151}: Make the pure Python implementation of
@ref{13e,,xml.etree.ElementTree} behave the same as the C implementation
(@code{_elementree}) regarding default attribute values (by not setting
@code{specified_attributes=1}).

@item 
bpo-29753@footnote{https://bugs.python.org/issue29753}: In ctypes, now packed bitfields are calculated properly and the
first item of packed bitfields is now shrank correctly.
@end itemize

@node Documentation<10>,Tests<9>,Library<9>,Python 3 10 0 alpha 6
@anchor{whatsnew/changelog id66}@anchor{1163}
@subsubsection Documentation



@itemize -

@item 
bpo-27646@footnote{https://bugs.python.org/issue27646}: Clarify that ‘yield from <expr>’ works with any iterable, not
just iterators.

@item 
bpo-36346@footnote{https://bugs.python.org/issue36346}: Update some deprecated unicode APIs which are documented as
“will be removed in 4.0” to “3.12”. See 
@geindex Python Enhancement Proposals; PEP 623
PEP 623@footnote{https://www.python.org/dev/peps/pep-0623} for detail.
@end itemize

@node Tests<9>,Build<9>,Documentation<10>,Python 3 10 0 alpha 6
@anchor{whatsnew/changelog id67}@anchor{1164}
@subsubsection Tests



@itemize -

@item 
bpo-43288@footnote{https://bugs.python.org/issue43288}: Fix test_importlib to correctly skip Unicode file tests if the
filesystem does not support them.
@end itemize

@node Build<9>,Windows<9>,Tests<9>,Python 3 10 0 alpha 6
@anchor{whatsnew/changelog id68}@anchor{1165}
@subsubsection Build



@itemize -

@item 
bpo-43174@footnote{https://bugs.python.org/issue43174}: Windows build now uses @code{/utf-8} compiler option.

@item 
bpo-43103@footnote{https://bugs.python.org/issue43103}: Add a new configure @code{--without-static-libpython} option to
not build the @code{libpythonMAJOR.MINOR.a} static library and not install
the @code{python.o} object file.

@item 
bpo-13501@footnote{https://bugs.python.org/issue13501}: The configure script can now use @emph{libedit} instead of
@emph{readline} with the command line option @code{--with-readline=editline}.

@item 
bpo-42603@footnote{https://bugs.python.org/issue42603}: Make configure script use pkg-config to detect the location of
Tcl/Tk headers and libraries, used to build tkinter.

On macOS, a Tcl/Tk configuration provided by pkg-config will be preferred
over Tcl/Tk frameworks installed in @code{/@{System/,@}Library/Frameworks}. If
both exist and the latter is preferred, the appropriate @code{--with-tcltk-*}
configuration options need to be explicitly set.

@item 
bpo-39448@footnote{https://bugs.python.org/issue39448}: Add the “regen-frozen” makefile target that regenerates the
code for the frozen @code{__hello__} module.
@end itemize

@node Windows<9>,macOS<7>,Build<9>,Python 3 10 0 alpha 6
@anchor{whatsnew/changelog id69}@anchor{1166}
@subsubsection Windows



@itemize -

@item 
bpo-43155@footnote{https://bugs.python.org/issue43155}: @code{PyCMethod_New()} is now present in @code{python3.lib}.
@end itemize

@node macOS<7>,IDLE<9>,Windows<9>,Python 3 10 0 alpha 6
@anchor{whatsnew/changelog id70}@anchor{1167}
@subsubsection macOS



@itemize -

@item 
bpo-41837@footnote{https://bugs.python.org/issue41837}: Update macOS installer build to use OpenSSL 1.1.1j.
@end itemize

@node IDLE<9>,C API<9>,macOS<7>,Python 3 10 0 alpha 6
@anchor{whatsnew/changelog id71}@anchor{1168}
@subsubsection IDLE



@itemize -

@item 
bpo-43283@footnote{https://bugs.python.org/issue43283}: Document why printing to IDLE’s Shell is often slower than
printing to a system terminal and that it can be made faster by
pre-formatting a single string before printing.
@end itemize

@node C API<9>,,IDLE<9>,Python 3 10 0 alpha 6
@anchor{whatsnew/changelog id72}@anchor{1169}
@subsubsection C API



@itemize -

@item 
bpo-43278@footnote{https://bugs.python.org/issue43278}: Always put compiler and system information on the first line of
the REPL welcome message.

@item 
bpo-43270@footnote{https://bugs.python.org/issue43270}: Remove the private @code{_PyErr_OCCURRED()} macro: use the public
@ref{116a,,PyErr_Occurred()} function instead.

@item 
bpo-35134@footnote{https://bugs.python.org/issue35134}: Move odictobject.h, parser_interface.h, picklebufobject.h,
pydebug.h, and pyfpe.h into the cpython/ directory. They must not be
included directly, as they are already included by Python.h: @ref{1f5,,Include Files}.

@item 
bpo-35134@footnote{https://bugs.python.org/issue35134}: Move pyarena.h, pyctype.h, and pytime.h into the cpython/
directory. They must not be included directly, as they are already
included by Python.h: @ref{1f5,,Include Files}.

@item 
bpo-40170@footnote{https://bugs.python.org/issue40170}: @code{PyExceptionClass_Name()} is now always declared as a
function, in order to hide implementation details. The macro accessed
@ref{116b,,PyTypeObject.tp_name} directly.  Patch by Erlend E. Aasland.

@item 
bpo-43239@footnote{https://bugs.python.org/issue43239}: The @code{PyCFunction_New()} function is now exported in the
ABI when compiled with @code{-fvisibility=hidden}.

@item 
bpo-40170@footnote{https://bugs.python.org/issue40170}: @ref{116c,,PyIter_Check()} is now always declared as a function, in
order to hide implementation details. The macro accessed
@ref{1029,,PyTypeObject.tp_iternext} directly. Patch by Erlend E. Aasland.

@item 
bpo-40170@footnote{https://bugs.python.org/issue40170}: Convert @ref{116d,,PyDescr_IsData()} macro to a function to hide
implementation details: The macro accessed
@ref{116e,,PyTypeObject.tp_descr_set} directly. Patch by Erlend E.
Aasland.

@item 
bpo-43181@footnote{https://bugs.python.org/issue43181}: Convert @ref{116f,,PyObject_TypeCheck()} macro to a static inline
function. Patch by Erlend E. Aasland.
@end itemize

@node Python 3 10 0 alpha 5,Python 3 10 0 alpha 4,Python 3 10 0 alpha 6,Changelog
@anchor{whatsnew/changelog python-3-10-0-alpha-5}@anchor{1170}
@subsection Python 3.10.0 alpha 5


@emph{Release date: 2021-02-02}

@menu
* Security: Security<5>. 
* Core and Builtins: Core and Builtins<10>. 
* Library: Library<10>. 
* Documentation: Documentation<11>. 
* Tests: Tests<10>. 
* Build: Build<10>. 
* Windows: Windows<10>. 
* macOS: macOS<8>. 
* IDLE: IDLE<10>. 
* C API: C API<10>. 

@end menu

@node Security<5>,Core and Builtins<10>,,Python 3 10 0 alpha 5
@anchor{whatsnew/changelog id73}@anchor{1171}
@subsubsection Security



@itemize -

@item 
bpo-42938@footnote{https://bugs.python.org/issue42938}: Avoid static buffers when computing the repr of
@ref{1172,,ctypes.c_double} and @ref{1173,,ctypes.c_longdouble} values.
@end itemize

@node Core and Builtins<10>,Library<10>,Security<5>,Python 3 10 0 alpha 5
@anchor{whatsnew/changelog id74}@anchor{1174}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-42990@footnote{https://bugs.python.org/issue42990}: Refactor the @code{PyEval_} family of functions.


@itemize *

@item 
An new function @code{_PyEval_Vector} is added to simplify calls to Python from C.

@item 
@code{_PyEval_EvalCodeWithName} is removed

@item 
@code{PyEval_EvalCodeEx} is retained as part of the API, but is not used internally
@end itemize

@item 
bpo-38631@footnote{https://bugs.python.org/issue38631}: Replace @ref{48f,,Py_FatalError()} calls in the compiler with
regular @ref{891,,SystemError} exceptions. Patch by Victor Stinner.

@item 
bpo-42997@footnote{https://bugs.python.org/issue42997}: Improve error message for missing “:” before blocks. Patch by
Pablo Galindo.

@item 
bpo-43017@footnote{https://bugs.python.org/issue43017}: Improve error message in the parser when using un-parenthesised
tuples in comprehensions. Patch by Pablo Galindo.

@item 
bpo-42986@footnote{https://bugs.python.org/issue42986}: Fix parser crash when reporting syntax errors in f-string with
newlines. Patch by Pablo Galindo.

@item 
bpo-40176@footnote{https://bugs.python.org/issue40176}: Syntax errors for unterminated string literals now point to the
start of the string instead of reporting EOF/EOL.

@item 
bpo-42927@footnote{https://bugs.python.org/issue42927}: The inline cache for @code{LOAD_ATTR} now also optimizes access to
attributes defined by @code{__slots__}. This makes reading such attribute up
to 30% faster.

@item 
bpo-42864@footnote{https://bugs.python.org/issue42864}: Improve error messages in the parser when parentheses are not
closed. Patch by Pablo Galindo.

@item 
bpo-42924@footnote{https://bugs.python.org/issue42924}: Fix @code{bytearray} repetition incorrectly copying data from the
start of the buffer, even if the data is offset within the buffer (e.g.
after reassigning a slice at the start of the @code{bytearray} to a shorter
byte string).

@item 
bpo-42882@footnote{https://bugs.python.org/issue42882}: Fix the @code{_PyUnicode_FromId()} function
(_Py_IDENTIFIER(var) API) when @ref{38d,,Py_Initialize()} /
@ref{eed,,Py_Finalize()} is called multiple times: preserve
@code{_PyRuntime.unicode_ids.next_index} value.

@item 
bpo-42827@footnote{https://bugs.python.org/issue42827}: Fix a crash when working out the error line of a
@ref{21a,,SyntaxError} in some multi-line expressions.

@item 
bpo-42823@footnote{https://bugs.python.org/issue42823}: frame.f_lineno is correct even if frame.f_trace is set to True

@item 
bpo-37324@footnote{https://bugs.python.org/issue37324}: Remove deprecated aliases to
@ref{33d,,Collections Abstract Base Classes} from the @ref{1c,,collections}
module.

@item 
bpo-41994@footnote{https://bugs.python.org/issue41994}: Fixed possible leak in @code{import} when @code{sys.modules} is not a
@code{dict}.

@item 
bpo-27772@footnote{https://bugs.python.org/issue27772}: In string formatting, preceding the @emph{width} field by @code{'0'} no
longer affects the default alignment for strings.
@end itemize

@node Library<10>,Documentation<11>,Core and Builtins<10>,Python 3 10 0 alpha 5
@anchor{whatsnew/changelog id75}@anchor{1175}
@subsubsection Library



@itemize -

@item 
bpo-43108@footnote{https://bugs.python.org/issue43108}: Fixed a reference leak in the @ref{2a,,curses} module. Patch by
Pablo Galindo

@item 
bpo-43077@footnote{https://bugs.python.org/issue43077}: Update the bundled pip to 21.0.1 and setuptools to 52.0.0.

@item 
bpo-41282@footnote{https://bugs.python.org/issue41282}: Deprecate @code{distutils} in documentation and add warning on
import.

@item 
bpo-43014@footnote{https://bugs.python.org/issue43014}: Improve performance of @ref{118,,tokenize} by 20-30%.  Patch by
Anthony Sottile.

@item 
bpo-42323@footnote{https://bugs.python.org/issue42323}: Fix @ref{403,,math.nextafter()} for NaN on AIX.

@item 
bpo-42955@footnote{https://bugs.python.org/issue42955}: Add @ref{2c7,,sys.stdlib_module_names}, containing the list of the
standard library module names. Patch by Victor Stinner.

@item 
bpo-42944@footnote{https://bugs.python.org/issue42944}: Fix @code{random.Random.sample} when @code{counts} argument is not
@code{None}.

@item 
bpo-42934@footnote{https://bugs.python.org/issue42934}: Use @ref{9f4,,TracebackException}’s new @code{compact}
param in @ref{d04,,TestResult} to reduce time and memory consumed
by traceback formatting.

@item 
bpo-42931@footnote{https://bugs.python.org/issue42931}: Add @code{randbytes()} to @code{random.__all__}.

@item 
bpo-38250@footnote{https://bugs.python.org/issue38250}: [Enum] Flags consisting of a single bit are now considered
canonical, and will be the only flags returned from listing and iterating
over a Flag class or a Flag member.  Multi-bit flags are considered
aliases; they will be returned from lookups and operations that result in
their value. Iteration for both Flag and Flag members is in definition
order.

@item 
bpo-42877@footnote{https://bugs.python.org/issue42877}: Added the @code{compact} parameter to the constructor of
@ref{9f4,,traceback.TracebackException} to reduce time and memory for use
cases that only need to call @code{TracebackException.format()} and
@code{TracebackException.format_exception_only()}.

@item 
bpo-42923@footnote{https://bugs.python.org/issue42923}: The @ref{48f,,Py_FatalError()} function and the
@ref{78,,faulthandler} module now dump the list of extension modules on a
fatal error.

@item 
bpo-42848@footnote{https://bugs.python.org/issue42848}: Removed recursion from @ref{9f4,,TracebackException}
to allow it to handle long exception chains.

@item 
bpo-42901@footnote{https://bugs.python.org/issue42901}: [Enum] move member creation from @code{EnumMeta.__new__} to
@code{_proto_member.__set_name__}, allowing members to be created and visible
in @code{__init_subclass__}.

@item 
bpo-42780@footnote{https://bugs.python.org/issue42780}: Fix os.set_inheritable() for O_PATH file descriptors on Linux.

@item 
bpo-42866@footnote{https://bugs.python.org/issue42866}: Fix a reference leak in the @code{getcodec()} function of CJK
codecs. Patch by Victor Stinner.

@item 
bpo-42846@footnote{https://bugs.python.org/issue42846}: Convert the 6 CJK codec extension modules (_codecs_cn,
_codecs_hk, _codecs_iso2022, _codecs_jp, _codecs_kr and _codecs_tw) to the
multiphase initialization API (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}). Patch by Victor Stinner.

@item 
bpo-42851@footnote{https://bugs.python.org/issue42851}: remove __init_subclass__ support for Enum members

@item 
bpo-42834@footnote{https://bugs.python.org/issue42834}: Make internal caches of the @code{_json} module compatible with
subinterpreters.

@item 
bpo-41748@footnote{https://bugs.python.org/issue41748}: Fix HTMLParser parsing rules for element attributes containing
commas with spaces. Patch by Karl Dubost.

@item 
bpo-40810@footnote{https://bugs.python.org/issue40810}: Require SQLite 3.7.15 or newer. Patch by Erlend E. Aasland.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Convert the _multibytecodec extension module (CJK codecs) to
multi-phase initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).  Patch by Erlend E. Aasland.

@item 
bpo-42802@footnote{https://bugs.python.org/issue42802}: The distutils @code{bdist_wininst} command deprecated in Python
3.8 has been removed. The distutils @code{bdist_wheel} command is now
recommended to distribute binary packages on Windows.

@item 
bpo-24464@footnote{https://bugs.python.org/issue24464}: The undocumented built-in function
@code{sqlite3.enable_shared_cache} is now deprecated, scheduled for removal
in Python 3.12.  Its use is strongly discouraged by the SQLite3
documentation.  Patch by Erlend E. Aasland.

@item 
bpo-42384@footnote{https://bugs.python.org/issue42384}: Make pdb populate sys.path[0] exactly the same as regular
python execution.

@item 
bpo-42383@footnote{https://bugs.python.org/issue42383}: Fix pdb: previously pdb would fail to restart the debugging
target if it was specified using a relative path and the current directory
changed.

@item 
bpo-42005@footnote{https://bugs.python.org/issue42005}: Fix CLI of @ref{26,,cProfile} and @ref{ce,,profile} to catch
@ref{bf2,,BrokenPipeError}.

@item 
bpo-41604@footnote{https://bugs.python.org/issue41604}: Don’t decrement the reference count of the previous user_ptr
when set_panel_userptr fails.

@item 
bpo-41149@footnote{https://bugs.python.org/issue41149}: Allow executing callables that have a boolean value of
@code{False} when passed to @code{Threading.thread} as the target. Patch
contributed by Barney Stratford.

@item 
bpo-38307@footnote{https://bugs.python.org/issue38307}: Add an ‘end_lineno’ attribute to the Class and Function objects
that appear in the tree returned by pyclbr functions.  This and the
existing ‘lineno’ attribute define the extent of class and def statements.
Patch by Aviral Srivastava.

@item 
bpo-39273@footnote{https://bugs.python.org/issue39273}: The @code{BUTTON5_*} constants are now exposed in the
@ref{2a,,curses} module if available.

@item 
bpo-33289@footnote{https://bugs.python.org/issue33289}: Correct call to @ref{10d,,tkinter.colorchooser} to return RGB
triplet of ints instead of floats.  Patch by Cheryl Sabella.
@end itemize

@node Documentation<11>,Tests<10>,Library<10>,Python 3 10 0 alpha 5
@anchor{whatsnew/changelog id76}@anchor{1176}
@subsubsection Documentation



@itemize -

@item 
bpo-40304@footnote{https://bugs.python.org/issue40304}: Fix doc for type(name, bases, dict).  Patch by Boris
Verkhovskiy and Éric Araujo.

@item 
bpo-42811@footnote{https://bugs.python.org/issue42811}: Updated importlib.utils.resolve_name() doc to use
__spec__.parent instead of __package__. (Thanks Yair Frid.)
@end itemize

@node Tests<10>,Build<10>,Documentation<11>,Python 3 10 0 alpha 5
@anchor{whatsnew/changelog id77}@anchor{1177}
@subsubsection Tests



@itemize -

@item 
bpo-40823@footnote{https://bugs.python.org/issue40823}: Use @code{unittest.TestLoader().loadTestsFromTestCase()} instead
of @code{unittest.makeSuite()} in @ref{ed,,sqlite3} tests. Patch by Erlend E.
Aasland.

@item 
bpo-40810@footnote{https://bugs.python.org/issue40810}: In @ref{ed,,sqlite3}, fix @code{CheckTraceCallbackContent} for SQLite
pre 3.7.15.
@end itemize

@node Build<10>,Windows<10>,Tests<10>,Python 3 10 0 alpha 5
@anchor{whatsnew/changelog id78}@anchor{1178}
@subsubsection Build



@itemize -

@item 
bpo-43031@footnote{https://bugs.python.org/issue43031}: Pass @code{--timeout=$(TESTTIMEOUT)} option to the default profile
task @code{./python -m test --pgo} command.

@item 
bpo-36143@footnote{https://bugs.python.org/issue36143}: @code{make regen-all} now also runs @code{regen-keyword}. Patch by
Victor Stinner.

@item 
bpo-42874@footnote{https://bugs.python.org/issue42874}: Removed the grep -q and -E flags in the tzpath validation
section of the configure script to better accommodate users of some
platforms (specifically Solaris 10).

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Add library search path by wr-cc in add_cross_compiling_paths()
for VxWorks.

@item 
bpo-42856@footnote{https://bugs.python.org/issue42856}: Add @code{--with-wheel-pkg-dir=PATH} option to the @code{./configure}
script. If specified, the @ref{75,,ensurepip} module looks for @code{setuptools}
and @code{pip} wheel packages in this directory: if both are present, these
wheel packages are used instead of ensurepip bundled wheel packages.

Some Linux distribution packaging policies recommend against bundling
dependencies. For example, Fedora installs wheel packages in the
@code{/usr/share/python-wheels/} directory and don’t install the
@code{ensurepip._bundled} package.
@end itemize

@node Windows<10>,macOS<8>,Build<10>,Python 3 10 0 alpha 5
@anchor{whatsnew/changelog id79}@anchor{1179}
@subsubsection Windows



@itemize -

@item 
bpo-41837@footnote{https://bugs.python.org/issue41837}: Updated Windows installer to include OpenSSL 1.1.1i

@item 
bpo-42584@footnote{https://bugs.python.org/issue42584}: Upgrade Windows installer to use SQLite 3.34.0.
@end itemize

@node macOS<8>,IDLE<10>,Windows<10>,Python 3 10 0 alpha 5
@anchor{whatsnew/changelog id80}@anchor{117a}
@subsubsection macOS



@itemize -

@item 
bpo-42504@footnote{https://bugs.python.org/issue42504}: Ensure that the value of
sysconfig.get_config_var(‘MACOSX_DEPLOYMENT_TARGET’) is always a string,
even in when the value is parsable as an integer.
@end itemize

@node IDLE<10>,C API<10>,macOS<8>,Python 3 10 0 alpha 5
@anchor{whatsnew/changelog id81}@anchor{117b}
@subsubsection IDLE



@itemize -

@item 
bpo-43008@footnote{https://bugs.python.org/issue43008}: Make IDLE invoke @ref{28e,,sys.excepthook()} in normal, 2-process
mode. Patch by Ken Hilton.

@item 
bpo-33065@footnote{https://bugs.python.org/issue33065}: Fix problem debugging user classes with __repr__ method.

@item 
bpo-23544@footnote{https://bugs.python.org/issue23544}: Disable Debug=>Stack Viewer when user code is running or
Debugger is active, to prevent hang or crash.  Patch by Zackery Spytz.

@item 
bpo-32631@footnote{https://bugs.python.org/issue32631}: Finish zzdummy example extension module: make menu entries
work; add docstrings and tests with 100% coverage.
@end itemize

@node C API<10>,,IDLE<10>,Python 3 10 0 alpha 5
@anchor{whatsnew/changelog id82}@anchor{117c}
@subsubsection C API



@itemize -

@item 
bpo-42979@footnote{https://bugs.python.org/issue42979}: When Python is built in debug mode (with C assertions), calling
a type slot like @code{sq_length} (@code{__len__()} in Python) now fails with a
fatal error if the slot succeeded with an exception set, or failed with no
exception set. The error message contains the slot, the type name, and the
current exception (if an exception is set). Patch by Victor Stinner.

@item 
bpo-43030@footnote{https://bugs.python.org/issue43030}: Fixed a compiler warning in @ref{117d,,Py_UNICODE_ISSPACE()} on
platforms with signed @code{wchar_t}.
@end itemize

@node Python 3 10 0 alpha 4,Python 3 10 0 alpha 3,Python 3 10 0 alpha 5,Changelog
@anchor{whatsnew/changelog python-3-10-0-alpha-4}@anchor{117e}
@subsection Python 3.10.0 alpha 4


@emph{Release date: 2021-01-04}

@menu
* Core and Builtins: Core and Builtins<11>. 
* Library: Library<11>. 
* Documentation: Documentation<12>. 
* Tests: Tests<11>. 
* Build: Build<11>. 
* macOS: macOS<9>. 
* Tools/Demos: Tools/Demos<2>. 
* C API: C API<11>. 

@end menu

@node Core and Builtins<11>,Library<11>,,Python 3 10 0 alpha 4
@anchor{whatsnew/changelog id83}@anchor{117f}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-42814@footnote{https://bugs.python.org/issue42814}: Fix undefined behavior in @code{Objects/genericaliasobject.c}.

@item 
bpo-42806@footnote{https://bugs.python.org/issue42806}: Fix the column offsets for f-strings @ref{7,,ast} nodes
surrounded by parentheses and for nodes that spawn multiple lines. Patch
by Pablo Galindo.

@item 
bpo-40631@footnote{https://bugs.python.org/issue40631}: Fix regression where a single parenthesized starred expression
was a valid assignment target.

@item 
bpo-27794@footnote{https://bugs.python.org/issue27794}: Improve the error message for failed writes/deletes to property
objects. When possible, the attribute name is now shown. Patch provided by
Yurii Karabas.

@item 
bpo-42745@footnote{https://bugs.python.org/issue42745}: Make the type attribute lookup cache per-interpreter. Patch by
Victor Stinner.

@item 
bpo-42246@footnote{https://bugs.python.org/issue42246}: Jumps to jumps are not eliminated when it would break PEP 626.

@item 
bpo-42246@footnote{https://bugs.python.org/issue42246}: Make sure that the @code{f_lasti} and @code{f_lineno} attributes of a
frame are set correctly when an exception is raised or re-raised. Required
for PEP 626.

@item 
bpo-32381@footnote{https://bugs.python.org/issue32381}: The coding cookie (ex: @code{# coding: latin1}) is now ignored in
the command passed to the @ref{1180,,-c} command line option. Patch by
Victor Stinner.

@item 
bpo-30858@footnote{https://bugs.python.org/issue30858}: Improve error location in expressions that contain assignments.
Patch by Pablo Galindo and Lysandros Nikolaou.

@item 
bpo-42615@footnote{https://bugs.python.org/issue42615}: Remove jump commands made redundant by the deletion of
unreachable bytecode blocks

@item 
bpo-42639@footnote{https://bugs.python.org/issue42639}: Make the @ref{b,,atexit} module state per-interpreter. It is now
safe have more than one @ref{b,,atexit} module instance. Patch by Dong-hee
Na and Victor Stinner.

@item 
bpo-32381@footnote{https://bugs.python.org/issue32381}: Fix encoding name when running a @code{.pyc} file on Windows:
@ref{1181,,PyRun_SimpleFileExFlags()} now uses the correct encoding to
decode the filename.

@item 
bpo-42195@footnote{https://bugs.python.org/issue42195}: The @code{__args__} of the parameterized generics for
@ref{23c,,typing.Callable} and @ref{269,,collections.abc.Callable} are now
consistent.  The @code{__args__} for @ref{269,,collections.abc.Callable} are
now flattened while @ref{23c,,typing.Callable}’s have not changed.  To allow
this change, @ref{26a,,types.GenericAlias} can now be subclassed and
@code{collections.abc.Callable}’s @code{__class_getitem__} will now return a
subclass of @code{types.GenericAlias}.  Tests for typing were also updated to
not subclass things like @code{Callable[..., T]} as that is not a valid base
class.  Finally, both @code{Callable}s no longer validate their
@code{argtypes}, in @code{Callable[[argtypes], resulttype]} to prepare for
@geindex Python Enhancement Proposals; PEP 612
PEP 612@footnote{https://www.python.org/dev/peps/pep-0612}.  Patch by Ken Jin.

@item 
bpo-40137@footnote{https://bugs.python.org/issue40137}: Convert functools module to use
@ref{36d,,PyType_FromModuleAndSpec()}.

@item 
bpo-40077@footnote{https://bugs.python.org/issue40077}: Convert @ref{6,,array} to use heap types, and establish module
state for these.

@item 
bpo-42008@footnote{https://bugs.python.org/issue42008}: Fix _random.Random() seeding.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @code{pyexpat} extension module to multi-phase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-40521@footnote{https://bugs.python.org/issue40521}: Make the Unicode dictionary of interned strings compatible with
subinterpreters. Patch by Victor Stinner.

@item 
bpo-39465@footnote{https://bugs.python.org/issue39465}: Make @code{_PyUnicode_FromId()} function compatible with
subinterpreters. Each interpreter now has an array of identifier objects
(interned strings decoded from UTF-8). Patch by Victor Stinner.
@end itemize

@node Library<11>,Documentation<12>,Core and Builtins<11>,Python 3 10 0 alpha 4
@anchor{whatsnew/changelog id84}@anchor{1182}
@subsubsection Library



@itemize -

@item 
bpo-42257@footnote{https://bugs.python.org/issue42257}: Handle empty string in variable executable in
platform.libc_ver()

@item 
bpo-42772@footnote{https://bugs.python.org/issue42772}: randrange() now raises a TypeError when step is specified
without a stop argument.  Formerly, it silently ignored the step argument.

@item 
bpo-42759@footnote{https://bugs.python.org/issue42759}: Fixed equality comparison of @code{tkinter.Variable} and
@ref{1183,,tkinter.font.Font}. Objects which belong to different Tcl
interpreters are now always different, even if they have the same name.

@item 
bpo-42756@footnote{https://bugs.python.org/issue42756}: Configure LMTP Unix-domain socket to use socket global default
timeout when a timeout is not explicitly provided.

@item 
bpo-23328@footnote{https://bugs.python.org/issue23328}: Allow / character in username, password fields on _PROXY
envars.

@item 
bpo-42740@footnote{https://bugs.python.org/issue42740}: @ref{344,,typing.get_args()} and @ref{55c,,typing.get_origin()} now
support 
@geindex Python Enhancement Proposals; PEP 604
PEP 604@footnote{https://www.python.org/dev/peps/pep-0604} union types and 
@geindex Python Enhancement Proposals; PEP 612
PEP 612@footnote{https://www.python.org/dev/peps/pep-0612} additions to @code{Callable}.

@item 
bpo-42655@footnote{https://bugs.python.org/issue42655}: @ref{f4,,subprocess} @emph{extra_groups} is now correctly passed into
setgroups() system call.

@item 
bpo-42727@footnote{https://bugs.python.org/issue42727}: @code{EnumMeta.__prepare__} now accepts @code{**kwds} to properly
support @code{__init_subclass__}

@item 
bpo-38308@footnote{https://bugs.python.org/issue38308}: Add optional @emph{weights} to @emph{statistics.harmonic_mean()}.

@item 
bpo-42721@footnote{https://bugs.python.org/issue42721}: When simple query dialogs (@ref{114,,tkinter.simpledialog}),
message boxes (@ref{112,,tkinter.messagebox}) or color choose dialog
(@ref{10d,,tkinter.colorchooser}) are created without arguments @emph{master} and
@emph{parent}, and the default root window is not yet created, and
@code{NoDefaultRoot()} was not called, a new temporal hidden root
window will be created automatically. It will not be set as the default
root window and will be destroyed right after closing the dialog window.
It will help to use these simple dialog windows in programs which do not
need other GUI.

@item 
bpo-25246@footnote{https://bugs.python.org/issue25246}: Optimized @ref{1184,,collections.deque.remove()}.

@item 
bpo-35728@footnote{https://bugs.python.org/issue35728}: Added a root parameter to @ref{1185,,tkinter.font.nametofont()}.

@item 
bpo-15303@footnote{https://bugs.python.org/issue15303}: @ref{10c,,tkinter} supports now widgets with boolean value False.

@item 
bpo-42681@footnote{https://bugs.python.org/issue42681}: Fixed range checks for color and pair numbers in @ref{2a,,curses}.

@item 
bpo-42685@footnote{https://bugs.python.org/issue42685}: Improved placing of simple query windows in Tkinter (such as
@ref{1186,,tkinter.simpledialog.askinteger()}). They are now centered at the
center of the parent window if it is specified and shown, otherwise at the
center of the screen.

@item 
bpo-9694@footnote{https://bugs.python.org/issue9694}: Argparse help no longer uses the confusing phrase, “optional
arguments”. It uses “options” instead.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @ref{2,,_thread} extension module to the multiphase
initialization API (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}) and convert its static types to heap
types.

@item 
bpo-37961@footnote{https://bugs.python.org/issue37961}: Fix crash in @code{tracemalloc.Traceback.__repr__()} (regressed
in Python 3.9).

@item 
bpo-42630@footnote{https://bugs.python.org/issue42630}: @ref{10c,,tkinter} functions and constructors which need a default
root window raise now @ref{5af,,RuntimeError} with descriptive message instead
of obscure @ref{163,,AttributeError} or @ref{21f,,NameError} if it is not created
yet or cannot be created automatically.

@item 
bpo-42639@footnote{https://bugs.python.org/issue42639}: @code{atexit._run_exitfuncs()} now logs callback exceptions
using @ref{18d,,sys.unraisablehook}, rather than logging them directly into
@ref{42d,,sys.stderr} and raise the last exception.

@item 
bpo-42644@footnote{https://bugs.python.org/issue42644}: @code{logging.disable} will now validate the types and value of
its parameter. It also now accepts strings representing the levels (as
does @code{loging.setLevel}) instead of only the numerical values.

@item 
bpo-42639@footnote{https://bugs.python.org/issue42639}: At Python exit, if a callback registered with
@ref{343,,atexit.register()} fails, its exception is now logged. Previously,
only some exceptions were logged, and the last exception was always
silently ignored.

@item 
bpo-36541@footnote{https://bugs.python.org/issue36541}: Fixed lib2to3.pgen2 to be able to parse PEP-570 positional only
argument syntax.

@item 
bpo-42382@footnote{https://bugs.python.org/issue42382}: In @code{importlib.metadata}: -  @code{EntryPoint} objects now expose
a @code{.dist} object referencing the @code{Distribution} when constructed from
a @code{Distribution}. - Add support for package discovery under package
normalization rules. - The object returned by @code{metadata()} now has a
formally-defined protocol called @code{PackageMetadata} with declared support
for the @code{.get_all()} method. - Synced with importlib_metadata 3.3.

@item 
bpo-41877@footnote{https://bugs.python.org/issue41877}: A check is added against misspellings of autospect, auto_spec
and set_spec being passed as arguments to patch, patch.object and
create_autospec.

@item 
bpo-39717@footnote{https://bugs.python.org/issue39717}: [tarfile] update nested exception raising to use @code{from None}
or @code{from e}

@item 
bpo-41877@footnote{https://bugs.python.org/issue41877}: AttributeError for suspected misspellings of assertions on
mocks are now pointing out that the cause are misspelled assertions and
also what to do if the misspelling is actually an intended attribute name.
The unittest.mock document is also updated to reflect the current set of
recognised misspellings.

@item 
bpo-41559@footnote{https://bugs.python.org/issue41559}: Implemented 
@geindex Python Enhancement Proposals; PEP 612
PEP 612@footnote{https://www.python.org/dev/peps/pep-0612}: added @code{ParamSpec} and @code{Concatenate}
to @ref{120,,typing}.  Patch by Ken Jin.

@item 
bpo-42385@footnote{https://bugs.python.org/issue42385}: StrEnum: fix _generate_next_value_ to return a str

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Define THREAD_STACK_SIZE for VxWorks.

@item 
bpo-34750@footnote{https://bugs.python.org/issue34750}: [Enum] @code{_EnumDict.update()} is now supported

@item 
bpo-42517@footnote{https://bugs.python.org/issue42517}: Enum: private names do not become members / do not generate
errors – they remain normal attributes

@item 
bpo-42678@footnote{https://bugs.python.org/issue42678}: @code{Enum}: call @code{__init_subclass__} after members have been
added

@item 
bpo-28964@footnote{https://bugs.python.org/issue28964}: @ref{754,,ast.literal_eval()} adds line number information (if
available) in error message for malformed nodes.

@item 
bpo-42470@footnote{https://bugs.python.org/issue42470}: @ref{ddd,,random.sample()} no longer warns on a sequence which is
also a set.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: @code{posixpath.expanduser()} returns the input @emph{path} unchanged
if user home directory is None on VxWorks.

@item 
bpo-42388@footnote{https://bugs.python.org/issue42388}: Fix subprocess.check_output(…, input=None) behavior when
text=True to be consistent with that of the documentation and
universal_newlines=True.

@item 
bpo-34463@footnote{https://bugs.python.org/issue34463}: Fixed discrepancy between @ref{11a,,traceback} and the interpreter
in formatting of SyntaxError with lineno not set (@ref{11a,,traceback} was
changed to match interpreter).

@item 
bpo-42393@footnote{https://bugs.python.org/issue42393}: Raise @ref{347,,OverflowError} instead of silent truncation in
@ref{346,,socket.ntohs()} and @ref{345,,socket.htons()}.  Silent truncation was
deprecated in Python 3.7. Patch by Erlend E. Aasland

@item 
bpo-42222@footnote{https://bugs.python.org/issue42222}: Harmonized @ref{305,,random.randrange()} argument handling to match
@ref{438,,range()}.


@itemize *

@item 
The integer test and conversion in @code{randrange()} now uses
@ref{1187,,operator.index()}.

@item 
Non-integer arguments to @code{randrange()} are deprecated.

@item 
The @code{ValueError} is deprecated in favor of a @code{TypeError}.

@item 
It now runs a little faster than before.
@end itemize

(Contributed by Raymond Hettinger and Serhiy Storchaka.)

@item 
bpo-42163@footnote{https://bugs.python.org/issue42163}: Restore compatibility for @code{uname_result} around deepcopy and
_replace.

@item 
bpo-42090@footnote{https://bugs.python.org/issue42090}: @code{zipfile.Path.joinpath} now accepts arbitrary arguments, same
as @code{pathlib.Path.joinpath}.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the _csv module to the multi-phase initialization API
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-42059@footnote{https://bugs.python.org/issue42059}: @ref{2de,,typing.TypedDict} types created using the alternative
call-style syntax now correctly respect the @code{total} keyword argument
when setting their @code{__required_keys__} and @code{__optional_keys__} class
attributes.

@item 
bpo-41960@footnote{https://bugs.python.org/issue41960}: Add @code{globalns} and @code{localns} parameters to the
@ref{1c1,,inspect.signature()} and @ref{1c3,,inspect.Signature.from_callable()}.

@item 
bpo-41907@footnote{https://bugs.python.org/issue41907}: fix @code{format()} behavior for @code{IntFlag}

@item 
bpo-41891@footnote{https://bugs.python.org/issue41891}: Ensure asyncio.wait_for waits for task completion

@item 
bpo-24792@footnote{https://bugs.python.org/issue24792}: Fixed bug where @code{zipimporter} sometimes reports an
incorrect cause of import errors.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Fix site and sysconfig modules for VxWorks RTOS which has no
home directories.

@item 
bpo-41462@footnote{https://bugs.python.org/issue41462}: Add @ref{9a1,,os.set_blocking()} support for VxWorks RTOS.

@item 
bpo-40219@footnote{https://bugs.python.org/issue40219}: Lowered @code{tkinter.ttk.LabeledScale} dummy widget to
prevent hiding part of the content label.

@item 
bpo-37193@footnote{https://bugs.python.org/issue37193}: Fixed memory leak in @code{socketserver.ThreadingMixIn} introduced
in Python 3.7.

@item 
bpo-39068@footnote{https://bugs.python.org/issue39068}: Fix initialization race condition in @code{a85encode()} and
@code{b85encode()} in @ref{d,,base64}. Patch by Brandon Stansbury.
@end itemize

@node Documentation<12>,Tests<11>,Library<11>,Python 3 10 0 alpha 4
@anchor{whatsnew/changelog id85}@anchor{1188}
@subsubsection Documentation



@itemize -

@item 
bpo-17140@footnote{https://bugs.python.org/issue17140}: Add documentation for the
@ref{1189,,multiprocessing.pool.ThreadPool} class.

@item 
bpo-34398@footnote{https://bugs.python.org/issue34398}: Prominently feature listings from the glossary in documentation
search results. Patch by Ammar Askar.
@end itemize

@node Tests<11>,Build<11>,Documentation<12>,Python 3 10 0 alpha 4
@anchor{whatsnew/changelog id86}@anchor{118a}
@subsubsection Tests



@itemize -

@item 
bpo-42794@footnote{https://bugs.python.org/issue42794}: Update test_nntplib to use official group name of news.aioe.org
for testing. Patch by Dong-hee Na.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Skip some asyncio tests on VxWorks.

@item 
bpo-42641@footnote{https://bugs.python.org/issue42641}: Enhance @code{test_select.test_select()}: it now takes 500
milliseconds rather than 10 seconds. Use Python rather than a shell to
make the test more portable.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Skip some tests in _test_all_chown_common() on VxWorks.

@item 
bpo-42199@footnote{https://bugs.python.org/issue42199}: Fix bytecode helper assertNotInBytecode.

@item 
bpo-41443@footnote{https://bugs.python.org/issue41443}: Add more attribute checking in test_posix.py

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Disable os.popen and impacted tests on VxWorks

@item 
bpo-41439@footnote{https://bugs.python.org/issue41439}: Port test_ssl and test_uuid to VxWorks RTOS.
@end itemize

@node Build<11>,macOS<9>,Tests<11>,Python 3 10 0 alpha 4
@anchor{whatsnew/changelog id87}@anchor{118b}
@subsubsection Build



@itemize -

@item 
bpo-42692@footnote{https://bugs.python.org/issue42692}: Fix __builtin_available check on older compilers. Patch by
Joshua Root.

@item 
bpo-27640@footnote{https://bugs.python.org/issue27640}: Added @code{--disable-test-modules} option to the @code{configure}
script: don’t build nor install test modules. Patch by Xavier de Gaye,
Thomas Petazzoni and Peixing Xin.

@item 
bpo-42604@footnote{https://bugs.python.org/issue42604}: Now all platforms use a value for the “EXT_SUFFIX” build
variable derived from SOABI (for instance in freeBSD, “EXT_SUFFIX” is now
“.cpython-310d.so” instead of “.so”). Previously only Linux, Mac and
VxWorks were using a value for “EXT_SUFFIX” that included “SOABI”.

@item 
bpo-42598@footnote{https://bugs.python.org/issue42598}: Fix implicit function declarations in configure which could
have resulted in incorrect configuration checks.  Patch contributed by
Joshua Root.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Enable libpython3.so for VxWorks.

@item 
bpo-29076@footnote{https://bugs.python.org/issue29076}: Add fish shell support to macOS installer.
@end itemize

@node macOS<9>,Tools/Demos<2>,Build<11>,Python 3 10 0 alpha 4
@anchor{whatsnew/changelog id88}@anchor{118c}
@subsubsection macOS



@itemize -

@item 
bpo-42361@footnote{https://bugs.python.org/issue42361}: Update macOS installer build to use Tcl/Tk 8.6.11 (rc2,
expected to be final release).

@item 
bpo-41837@footnote{https://bugs.python.org/issue41837}: Update macOS installer build to use OpenSSL 1.1.1i.

@item 
bpo-42584@footnote{https://bugs.python.org/issue42584}: Update macOS installer to use SQLite 3.34.0.
@end itemize

@node Tools/Demos<2>,C API<11>,macOS<9>,Python 3 10 0 alpha 4
@anchor{whatsnew/changelog id89}@anchor{118d}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-42726@footnote{https://bugs.python.org/issue42726}: Fixed Python 3 compatibility issue with gdb/libpython.py
handling of attribute dictionaries.

@item 
bpo-42613@footnote{https://bugs.python.org/issue42613}: Fix @code{freeze.py} tool to use the prope config and library
directories. Patch by Victor Stinner.
@end itemize

@node C API<11>,,Tools/Demos<2>,Python 3 10 0 alpha 4
@anchor{whatsnew/changelog id90}@anchor{118e}
@subsubsection C API



@itemize -

@item 
bpo-42591@footnote{https://bugs.python.org/issue42591}: Export the @code{Py_FrozenMain()} function: fix a Python 3.9.0
regression. Python 3.9 uses @code{-fvisibility=hidden} and the function was
not exported explicitly and so not exported.

@item 
bpo-32381@footnote{https://bugs.python.org/issue32381}: Remove the private @code{_Py_fopen()} function which is no
longer needed. Use @code{_Py_wfopen()} or @code{_Py_fopen_obj()}
instead. Patch by Victor Stinner.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @ref{db,,resource} extension module to module state

@item 
bpo-42111@footnote{https://bugs.python.org/issue42111}: Update the @code{xxlimited} module to be a better example of how
to use the limited C API.

@item 
bpo-40052@footnote{https://bugs.python.org/issue40052}: Fix an alignment build warning/error in function
@code{PyVectorcall_Function()}. Patch by Andreas Schneider, Antoine Pitrou
and Petr Viktorin.
@end itemize

@node Python 3 10 0 alpha 3,Python 3 10 0 alpha 2,Python 3 10 0 alpha 4,Changelog
@anchor{whatsnew/changelog python-3-10-0-alpha-3}@anchor{118f}
@subsection Python 3.10.0 alpha 3


@emph{Release date: 2020-12-07}

@menu
* Security: Security<6>. 
* Core and Builtins: Core and Builtins<12>. 
* Library: Library<12>. 
* Documentation: Documentation<13>. 
* Tests: Tests<12>. 
* Build: Build<12>. 
* Windows: Windows<11>. 
* macOS: macOS<10>. 
* IDLE: IDLE<11>. 
* Tools/Demos: Tools/Demos<3>. 
* C API: C API<12>. 

@end menu

@node Security<6>,Core and Builtins<12>,,Python 3 10 0 alpha 3
@anchor{whatsnew/changelog id91}@anchor{1190}
@subsubsection Security



@itemize -

@item 
bpo-40791@footnote{https://bugs.python.org/issue40791}: Add @code{volatile} to the accumulator variable in
@code{hmac.compare_digest}, making constant-time-defeating optimizations less
likely.
@end itemize

@node Core and Builtins<12>,Library<12>,Security<6>,Python 3 10 0 alpha 3
@anchor{whatsnew/changelog id92}@anchor{1191}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-42576@footnote{https://bugs.python.org/issue42576}: @code{types.GenericAlias} will now raise a @code{TypeError} when
attempting to initialize with a keyword argument.  Previously, this would
cause the interpreter to crash if the interpreter was compiled with debug
symbols. This does not affect interpreters compiled for release.  Patch by
Ken Jin.

@item 
bpo-42536@footnote{https://bugs.python.org/issue42536}: Several built-in and standard library types now ensure that
their internal result tuples are always tracked by the @ref{1192,,garbage collector}:


@itemize -

@item 
@ref{4e5,,collections.OrderedDict.items()}

@item 
@ref{245,,dict.items()}

@item 
@ref{fdc,,enumerate()}

@item 
@ref{e77,,functools.reduce()}

@item 
@ref{ea8,,itertools.combinations()}

@item 
@ref{e31,,itertools.combinations_with_replacement()}

@item 
@ref{1193,,itertools.permutations()}

@item 
@ref{ea9,,itertools.product()}

@item 
@ref{e43,,itertools.zip_longest()}

@item 
@ref{247,,zip()}
@end itemize

Previously, they could have become untracked by a prior garbage
collection. Patch by Brandt Bucher.

@item 
bpo-42500@footnote{https://bugs.python.org/issue42500}: Improve handling of exceptions near recursion limit. Converts a
number of Fatal Errors in RecursionErrors.

@item 
bpo-42246@footnote{https://bugs.python.org/issue42246}: PEP 626: After a return, the f_lineno attribute of a frame is
always the last line executed.

@item 
bpo-42435@footnote{https://bugs.python.org/issue42435}: Speed up comparison of bytes objects with non-bytes objects
when option @ref{6dc,,-b} is specified. Speed up comparison of bytarray
objects with non-buffer object.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @code{_warnings} extension module to the multi-phase
initialization API (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}). Patch by Victor Stinner.

@item 
bpo-41686@footnote{https://bugs.python.org/issue41686}: On Windows, the @code{SIGINT} event, @code{_PyOS_SigintEvent()}, is
now created even if Python is configured to not install signal handlers
(if @ref{3a1,,PyConfig.install_signal_handlers} equals to 0, or
@code{Py_InitializeEx(0)}).

@item 
bpo-42381@footnote{https://bugs.python.org/issue42381}: Allow assignment expressions in set literals and set
comprehensions as per PEP 572. Patch by Pablo Galindo.

@item 
bpo-42202@footnote{https://bugs.python.org/issue42202}: Change function parameters annotations internal representation
to tuple of strings. Patch provided by Yurii Karabas.

@item 
bpo-42374@footnote{https://bugs.python.org/issue42374}: Fix a regression introduced by the new parser, where an
unparenthesized walrus operator was not allowed within generator
expressions.

@item 
bpo-42316@footnote{https://bugs.python.org/issue42316}: Allow an unparenthesized walrus in subscript indexes.

@item 
bpo-42349@footnote{https://bugs.python.org/issue42349}: Make sure that the compiler front-end produces a well-formed
control flow graph. Be be more aggressive in the compiler back-end, as it
is now safe to do so.

@item 
bpo-42296@footnote{https://bugs.python.org/issue42296}: On Windows, fix a regression in signal handling which prevented
to interrupt a program using CTRL+C. The signal handler can be run in a
thread different than the Python thread, in which case the test deciding
if the thread can handle signals is wrong.

@item 
bpo-42332@footnote{https://bugs.python.org/issue42332}: @ref{26a,,types.GenericAlias} objects can now be the targets of
weakrefs.

@item 
bpo-42282@footnote{https://bugs.python.org/issue42282}: Optimise constant subexpressions that appear as part of named
expressions (previously the AST optimiser did not descend into named
expressions). Patch by Nick Coghlan.

@item 
bpo-42266@footnote{https://bugs.python.org/issue42266}: Fixed a bug with the LOAD_ATTR opcode cache that was not
respecting monkey-patching a class-level attribute to make it a
descriptor. Patch by Pablo Galindo.

@item 
bpo-40077@footnote{https://bugs.python.org/issue40077}: Convert @ref{d5,,queue} to use heap types.

@item 
bpo-42246@footnote{https://bugs.python.org/issue42246}: Improved accuracy of line tracing events and f_lineno attribute
of Frame objects. See PEP 626 for details.

@item 
bpo-40077@footnote{https://bugs.python.org/issue40077}: Convert @ref{af,,mmap} to use heap types.

@item 
bpo-42233@footnote{https://bugs.python.org/issue42233}: Allow @code{GenericAlias} objects to use @ref{238,,union type expressions}. This allows expressions like @code{list[int] |
dict[float, str]} where previously a @code{TypeError} would have been
thrown.  This also fixes union type expressions not de-duplicating
@code{GenericAlias} objects. (Contributed by Ken Jin in bpo-42233@footnote{https://bugs.python.org/issue42233}.)

@item 
bpo-26131@footnote{https://bugs.python.org/issue26131}: The import system triggers a @ref{304,,ImportWarning} when it falls back
to using @code{load_module()}.
@end itemize

@node Library<12>,Documentation<13>,Core and Builtins<12>,Python 3 10 0 alpha 3
@anchor{whatsnew/changelog id93}@anchor{1194}
@subsubsection Library



@itemize -

@item 
bpo-5054@footnote{https://bugs.python.org/issue5054}: CGIHTTPRequestHandler.run_cgi() HTTP_ACCEPT improperly parsed.
Replace the special purpose getallmatchingheaders with generic get_all
method and add relevant tests.

Original Patch by Martin Panter. Modified by Senthil Kumaran.

@item 
bpo-42562@footnote{https://bugs.python.org/issue42562}: Fix issue when dis failed to parse function that has no line
numbers. Patch provided by Yurii Karabas.

@item 
bpo-17735@footnote{https://bugs.python.org/issue17735}: @code{inspect.findsource()} now raises @ref{2a1,,OSError} instead of
@ref{1066,,IndexError} when @code{co_lineno} of a code object is greater than
the file length. This can happen, for example, when a file is edited after
it was imported.  PR by Irit Katriel.

@item 
bpo-42116@footnote{https://bugs.python.org/issue42116}: Fix handling of trailing comments by @ref{1105,,inspect.getsource()}.

@item 
bpo-42532@footnote{https://bugs.python.org/issue42532}: Remove unexpected call of @code{__bool__} when passing a
@code{spec_arg} argument to a Mock.

@item 
bpo-38200@footnote{https://bugs.python.org/issue38200}: Added itertools.pairwise()

@item 
bpo-41818@footnote{https://bugs.python.org/issue41818}: Fix test_master_read() so that it succeeds on all platforms
that either raise OSError or return b”” upon reading from master.

@item 
bpo-42487@footnote{https://bugs.python.org/issue42487}: ChainMap.__iter__ no longer calls __getitem__ on underlying
maps

@item 
bpo-42482@footnote{https://bugs.python.org/issue42482}: @ref{9f4,,TracebackException} no longer holds a
reference to the exception’s traceback object. Consequently, instances of
TracebackException for equivalent but non-equal exceptions now compare as
equal.

@item 
bpo-41818@footnote{https://bugs.python.org/issue41818}: Make test_openpty() avoid unexpected success due to number of
rows and/or number of columns being == 0.

@item 
bpo-42392@footnote{https://bugs.python.org/issue42392}: Remove loop parameter from @code{asyncio.subprocess} and
@code{asyncio.tasks} functions. Patch provided by Yurii Karabas.

@item 
bpo-42392@footnote{https://bugs.python.org/issue42392}: Remove loop parameter from @code{asyncio.open_connection} and
@code{asyncio.start_server} functions. Patch provided by Yurii Karabas.

@item 
bpo-28468@footnote{https://bugs.python.org/issue28468}: Add @ref{2aa,,platform.freedesktop_os_release()} function to parse
freedesktop.org @code{os-release} files.

@item 
bpo-42299@footnote{https://bugs.python.org/issue42299}: Removed the @code{formatter} module, which was deprecated in
Python 3.4. It is somewhat obsolete, little used, and not tested. It was
originally scheduled to be removed in Python 3.6, but such removals were
delayed until after Python 2.7 EOL. Existing users should copy whatever
classes they use into their code. Patch by Dong-hee Na and and Terry J.
Reedy.

@item 
bpo-26131@footnote{https://bugs.python.org/issue26131}: Deprecate zipimport.zipimporter.load_module() in favour of
exec_module().

@item 
bpo-41818@footnote{https://bugs.python.org/issue41818}: Updated tests for the pty library. test_basic() has been
changed to test_openpty(); this additionally checks if slave termios and
slave winsize are being set properly by pty.openpty(). In order to add
support for FreeBSD, NetBSD, OpenBSD, and Darwin, this also adds
test_master_read(), which demonstrates that pty.spawn() should not depend
on an OSError to exit from its copy loop.

@item 
bpo-42392@footnote{https://bugs.python.org/issue42392}: Remove loop parameter from @code{__init__} in all
@code{asyncio.locks} and @code{asyncio.Queue} classes. Patch provided by Yurii
Karabas.

@item 
bpo-15450@footnote{https://bugs.python.org/issue15450}: Make @ref{abd,,filecmp.dircmp} respect subclassing. Now the
@ref{1195,,filecmp.dircmp.subdirs} behaves as expected when subclassing
dircmp.

@item 
bpo-42413@footnote{https://bugs.python.org/issue42413}: The exception @ref{2b7,,socket.timeout} is now an alias of
@ref{2b8,,TimeoutError}.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Support signal module on VxWorks.

@item 
bpo-42406@footnote{https://bugs.python.org/issue42406}: We fixed an issue in @code{pickle.whichmodule} in which importing
@ref{b3,,multiprocessing} could change the how pickle identifies which module an
object belongs to, potentially breaking the unpickling of those objects.

@item 
bpo-42403@footnote{https://bugs.python.org/issue42403}: Simplify the @ref{96,,importlib} external bootstrap code:
@code{importlib._bootstrap_external} now uses regular imports to import
builtin modules. When it is imported, the builtin @ref{3b7,,__import__()}
function is already fully working and so can be used to import builtin
modules like @ref{f7,,sys}. Patch by Victor Stinner.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Convert _sre module types to heap types (PEP 384). Patch by
Erlend E. Aasland.

@item 
bpo-42375@footnote{https://bugs.python.org/issue42375}: subprocess module update for DragonFlyBSD support.

@item 
bpo-41713@footnote{https://bugs.python.org/issue41713}: Port the @code{_signal} extension module to the multi-phase
initialization API (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}). Patch by Victor Stinner and Mohamed
Koubaa.

@item 
bpo-37205@footnote{https://bugs.python.org/issue37205}: @ref{5fc,,time.time()}, @ref{5a0,,time.perf_counter()} and
@ref{9ed,,time.monotonic()} functions can no longer fail with a Python fatal
error, instead raise a regular Python exception on failure.

@item 
bpo-42328@footnote{https://bugs.python.org/issue42328}: Fixed @ref{1196,,tkinter.ttk.Style.map()}. The function accepts now
the representation of the default state as empty sequence (as returned by
@code{Style.map()}). The structure of the result is now the same on all
platform and does not depend on the value of @code{wantobjects}.

@item 
bpo-42345@footnote{https://bugs.python.org/issue42345}: Fix various issues with @code{typing.Literal} parameter handling
(flatten, deduplicate, use type to cache key). Patch provided by Yurii
Karabas.

@item 
bpo-37205@footnote{https://bugs.python.org/issue37205}: @ref{5a0,,time.perf_counter()} on Windows and
@ref{9ed,,time.monotonic()} on macOS are now system-wide. Previously, they
used an offset computed at startup to reduce the precision loss caused by
the float type. Use @ref{600,,time.perf_counter_ns()} and
@ref{5ff,,time.monotonic_ns()} added in Python 3.7 to avoid this precision
loss.

@item 
bpo-42318@footnote{https://bugs.python.org/issue42318}: Fixed support of non-BMP characters in @ref{10c,,tkinter} on macOS.

@item 
bpo-42350@footnote{https://bugs.python.org/issue42350}: Fix the @ref{44b,,threading.Thread} class at fork: do nothing if
the thread is already stopped (ex: fork called at Python exit).
Previously, an error was logged in the child process.

@item 
bpo-42333@footnote{https://bugs.python.org/issue42333}: Port _ssl extension module to heap types.

@item 
bpo-42014@footnote{https://bugs.python.org/issue42014}: The @code{onerror} callback from @code{shutil.rmtree} now receives
correct function when @code{os.open} fails.

@item 
bpo-42237@footnote{https://bugs.python.org/issue42237}: Fix @ref{62c,,os.sendfile()} on illumos.

@item 
bpo-42308@footnote{https://bugs.python.org/issue42308}: Add @ref{2d0,,threading.__excepthook__} to allow retrieving the
original value of @ref{2d1,,threading.excepthook()} in case it is set to a
broken or a different value. Patch by Mario Corchero.

@item 
bpo-42131@footnote{https://bugs.python.org/issue42131}: Implement PEP 451/spec methods on zipimport.zipimporter:
find_spec(), create_module(), and exec_module().

This also allows for the documented deprecation of find_loader(),
find_module(), and load_module().

@item 
bpo-41877@footnote{https://bugs.python.org/issue41877}: Mock objects which are not unsafe will now raise an
AttributeError if an attribute with the prefix asert, aseert, or assrt is
accessed, in addition to this already happening for the prefixes assert or
assret.

@item 
bpo-42264@footnote{https://bugs.python.org/issue42264}: @code{sqlite3.OptimizedUnicode} has been undocumented and obsolete
since Python 3.3, when it was made an alias to @ref{2ef,,str}.  It is now
deprecated, scheduled for removal in Python 3.12.

@item 
bpo-42251@footnote{https://bugs.python.org/issue42251}: Added @ref{2cc,,threading.gettrace()} and
@ref{2cd,,threading.getprofile()} to retrieve the functions set by
@ref{2ce,,threading.settrace()} and @ref{2cf,,threading.setprofile()} respectively.
Patch by Mario Corchero.

@item 
bpo-42249@footnote{https://bugs.python.org/issue42249}: Fixed writing binary Plist files larger than 4 GiB.

@item 
bpo-42236@footnote{https://bugs.python.org/issue42236}: On Unix, the @ref{10f6,,os.device_encoding()} function now returns
@code{'UTF-8'} rather than the device encoding if the @ref{5f2,,Python UTF-8 Mode} is enabled.

@item 
bpo-41754@footnote{https://bugs.python.org/issue41754}: webbrowser: Ignore @emph{NotADirectoryError} when calling
@code{xdg-settings}.

@item 
bpo-42183@footnote{https://bugs.python.org/issue42183}: Fix a stack overflow error for asyncio Task or Future repr().

The overflow occurs under some circumstances when a Task or Future
recursively returns itself.

@item 
bpo-42140@footnote{https://bugs.python.org/issue42140}: Improve asyncio.wait function to create the futures set just
one time.

@item 
bpo-42133@footnote{https://bugs.python.org/issue42133}: Update various modules in the stdlib to fall back on
@code{__spec__.loader} when @ref{baf,,__loader__} isn’t defined on a module.

@item 
bpo-26131@footnote{https://bugs.python.org/issue26131}: The @code{load_module()} methods found in importlib now trigger a
DeprecationWarning.

@item 
bpo-39825@footnote{https://bugs.python.org/issue39825}: Windows: Change @code{sysconfig.get_config_var('EXT_SUFFIX')} to
the expected full @code{platform_tag.extension} format. Previously it was
hard-coded to @code{.pyd}, now it is compatible with @code{distutils.sysconfig}
and will result in something like @code{.cp38-win_amd64.pyd}. This brings
windows into conformance with the other platforms.

@item 
bpo-26389@footnote{https://bugs.python.org/issue26389}: The @ref{2d3,,traceback.format_exception()},
@ref{2d4,,traceback.format_exception_only()}, and
@ref{2d5,,traceback.print_exception()} functions can now take an exception
object as a positional-only argument.

@item 
bpo-41889@footnote{https://bugs.python.org/issue41889}: Enum: fix regression involving inheriting a multiply-inherited
enum

@item 
bpo-41861@footnote{https://bugs.python.org/issue41861}: Convert @ref{ed,,sqlite3} to use heap types (PEP 384). Patch by
Erlend E. Aasland.

@item 
bpo-40624@footnote{https://bugs.python.org/issue40624}: Added support for the XPath @code{!=} operator in xml.etree

@item 
bpo-28850@footnote{https://bugs.python.org/issue28850}: Fix @ref{1197,,pprint.PrettyPrinter.format()} overrides being ignored
for contents of small containers. The @code{pprint._safe_repr()} function
was removed.

@item 
bpo-41625@footnote{https://bugs.python.org/issue41625}: Expose the @code{splice()} as @ref{29a,,os.splice()} in the
@ref{c0,,os} module. Patch by Pablo Galindo

@item 
bpo-34215@footnote{https://bugs.python.org/issue34215}: Clarify the error message for
@ref{1198,,asyncio.IncompleteReadError} when @code{expected} is @code{None}.

@item 
bpo-41543@footnote{https://bugs.python.org/issue41543}: Add async context manager support for contextlib.nullcontext.

@item 
bpo-21041@footnote{https://bugs.python.org/issue21041}: @ref{2a3,,pathlib.PurePath.parents} now supports negative
indexing. Patch contributed by Yaroslav Pankovych.

@item 
bpo-41332@footnote{https://bugs.python.org/issue41332}: Added missing connect_accepted_socket() method to
@code{asyncio.AbstractEventLoop}.

@item 
bpo-12800@footnote{https://bugs.python.org/issue12800}: Extracting a symlink from a tarball should succeed and
overwrite the symlink if it already exists. The fix is to remove the
existing file or symlink before extraction. Based on patch by Chris AtLee,
Jeffrey Kintscher, and Senthil Kumaran.

@item 
bpo-40968@footnote{https://bugs.python.org/issue40968}: @ref{127,,urllib.request} and @ref{8f,,http.client} now send
@code{http/1.1} ALPN extension during TLS handshake when no custom context is
supplied.

@item 
bpo-41001@footnote{https://bugs.python.org/issue41001}: Add func:@ref{299,,os.eventfd} to provide a low level interface for
Linux’s event notification file descriptor.

@item 
bpo-40816@footnote{https://bugs.python.org/issue40816}: Add AsyncContextDecorator to contextlib to support async
context manager as a decorator.

@item 
bpo-40550@footnote{https://bugs.python.org/issue40550}: Fix time-of-check/time-of-action issue in
subprocess.Popen.send_signal.

@item 
bpo-39411@footnote{https://bugs.python.org/issue39411}: Add an @code{is_async} identifier to @ref{d3,,pyclbr}’s @code{Function}
objects. Patch by Batuhan Taskaya

@item 
bpo-35498@footnote{https://bugs.python.org/issue35498}: Add slice support to @ref{2a3,,pathlib.PurePath.parents}.
@end itemize

@node Documentation<13>,Tests<12>,Library<12>,Python 3 10 0 alpha 3
@anchor{whatsnew/changelog id94}@anchor{1199}
@subsubsection Documentation



@itemize -

@item 
bpo-42238@footnote{https://bugs.python.org/issue42238}: Tentative to deprecate @code{make suspicious} by first removing it
from the CI and documentation builds, but keeping it around for manual
uses.

@item 
bpo-42153@footnote{https://bugs.python.org/issue42153}: Fix the URL for the IMAP protocol documents.

@item 
bpo-41028@footnote{https://bugs.python.org/issue41028}: Language and version switchers, previously maintained in every
cpython branches, are now handled by docsbuild-script.
@end itemize

@node Tests<12>,Build<12>,Documentation<13>,Python 3 10 0 alpha 3
@anchor{whatsnew/changelog id95}@anchor{119a}
@subsubsection Tests



@itemize -

@item 
bpo-41473@footnote{https://bugs.python.org/issue41473}: Re-enable test_gdb on gdb 9.2 and newer:
@indicateurl{https://bugzilla.redhat.com/show_bug.cgi?id=1866884} bug is fixed in gdb
10.1.

@item 
bpo-42553@footnote{https://bugs.python.org/issue42553}: Fix @code{test_asyncio.test_call_later()} race condition: don’t
measure asyncio performance in the @code{call_later()} unit test. The test
failed randomly on the CI.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Fix test_netrc on VxWorks: create temporary directories using
temp_cwd().

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: skip test_getaddrinfo_ipv6_scopeid_symbolic and
test_getnameinfo_ipv6_scopeid_symbolic on VxWorks

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: skip test_test of test_mailcap on VxWorks

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: add shell requirement for test_pipes

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: skip some tests related to fifo on VxWorks

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Fix test_doctest.py failures for VxWorks.

@item 
bpo-40754@footnote{https://bugs.python.org/issue40754}: Include @code{_testinternalcapi} module in Windows installer for
test suite

@item 
bpo-41561@footnote{https://bugs.python.org/issue41561}: test_ssl: skip test_min_max_version_mismatch when TLS 1.0 is
not available

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Fix os module failures for VxWorks RTOS.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Fix fifo test cases for VxWorks RTOS.
@end itemize

@node Build<12>,Windows<11>,Tests<12>,Python 3 10 0 alpha 3
@anchor{whatsnew/changelog id96}@anchor{119b}
@subsubsection Build



@itemize -

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: remove libnet dependency from detect_socket() for VxWorks

@item 
bpo-42398@footnote{https://bugs.python.org/issue42398}: Fix a race condition in “make regen-all” when make -jN option
is used to run jobs in parallel. The clinic.py script now only use atomic
write to write files. Moveover, generated files are now left unchanged if
the content does not change, to not change the file modification time.

@item 
bpo-41617@footnote{https://bugs.python.org/issue41617}: Fix building @code{pycore_bitutils.h} internal header on old clang
version without @code{__builtin_bswap16()} (ex: Xcode 4.6.3 on Mac OS X
10.7). Patch by Joshua Root and Victor Stinner.

@item 
bpo-38823@footnote{https://bugs.python.org/issue38823}: It is no longer possible to build the @code{_ctypes} extension
module without @code{wchar_t} type: remove @code{CTYPES_UNICODE} macro.
Anyway, the @code{wchar_t} type is required to build Python. Patch by
Victor Stinner.

@item 
bpo-42087@footnote{https://bugs.python.org/issue42087}: Support was removed for AIX 5.3 and below. See bpo-40680@footnote{https://bugs.python.org/issue40680}.

@item 
bpo-40998@footnote{https://bugs.python.org/issue40998}: Addressed three compiler warnings found by undefined behavior
sanitizer (ubsan).
@end itemize

@node Windows<11>,macOS<10>,Build<12>,Python 3 10 0 alpha 3
@anchor{whatsnew/changelog id97}@anchor{119c}
@subsubsection Windows



@itemize -

@item 
bpo-42120@footnote{https://bugs.python.org/issue42120}: Remove macro definition of @code{copysign} (to @code{_copysign}) in
headers.

@item 
bpo-38506@footnote{https://bugs.python.org/issue38506}: The Windows launcher now properly handles Python 3.10 when
listing installed Python versions.
@end itemize

@node macOS<10>,IDLE<11>,Windows<11>,Python 3 10 0 alpha 3
@anchor{whatsnew/changelog id98}@anchor{119d}
@subsubsection macOS



@itemize -

@item 
bpo-42504@footnote{https://bugs.python.org/issue42504}: Fix build on macOS Big Sur when MACOSX_DEPLOYMENT_TARGET=11

@item 
bpo-41116@footnote{https://bugs.python.org/issue41116}: Ensure distutils.unixxcompiler.find_library_file can find
system provided libraries on macOS 11.

@item 
bpo-41100@footnote{https://bugs.python.org/issue41100}: Add support for macOS 11 and Apple Silicon systems.

It is now possible to build “Universal 2” binaries using
“–enable-universalsdk –with-universal-archs=universal2”.

Binaries build on later macOS versions can be deployed back to older
versions (tested up to macOS 10.9), when using the correct deployment
target. This is tested using Xcode 11 and later.

@item 
bpo-42232@footnote{https://bugs.python.org/issue42232}: Added Darwin specific madvise options to mmap module.

@item 
bpo-38443@footnote{https://bugs.python.org/issue38443}: The @code{--enable-universalsdk} and @code{--with-universal-archs}
options for the configure script now check that the specified
architectures can be used.
@end itemize

@node IDLE<11>,Tools/Demos<3>,macOS<10>,Python 3 10 0 alpha 3
@anchor{whatsnew/changelog id99}@anchor{119e}
@subsubsection IDLE



@itemize -

@item 
bpo-42508@footnote{https://bugs.python.org/issue42508}: Keep IDLE running on macOS.  Remove obsolete workaround that
prevented running files with shortcuts when using new universal2
installers built on macOS 11.

@item 
bpo-42426@footnote{https://bugs.python.org/issue42426}: Fix reporting offset of the RE error in searchengine.

@item 
bpo-42415@footnote{https://bugs.python.org/issue42415}: Get docstrings for IDLE calltips more often by using
inspect.getdoc.
@end itemize

@node Tools/Demos<3>,C API<12>,IDLE<11>,Python 3 10 0 alpha 3
@anchor{whatsnew/changelog id100}@anchor{119f}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-42212@footnote{https://bugs.python.org/issue42212}: The smelly.py script now also checks the Python dynamic library
and extension modules, not only the Python static library. Make also the
script more verbose: explain what it does.

@item 
bpo-36310@footnote{https://bugs.python.org/issue36310}: Allow @code{Tools/i18n/pygettext.py} to detect calls to
@code{gettext} in f-strings.
@end itemize

@node C API<12>,,Tools/Demos<3>,Python 3 10 0 alpha 3
@anchor{whatsnew/changelog id101}@anchor{11a0}
@subsubsection C API



@itemize -

@item 
bpo-42423@footnote{https://bugs.python.org/issue42423}: The @ref{36c,,PyType_FromSpecWithBases()} and
@ref{36d,,PyType_FromModuleAndSpec()} functions now accept a single class as
the @emph{bases} argument.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @ref{e0,,select} extension module to multiphase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _posixsubprocess extension module to multiphase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _posixshmem extension module to multiphase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489})

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _struct extension module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489})

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @ref{ec,,spwd} extension module to multiphase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489})

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @ref{80,,gc} extension module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489})

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _queue extension module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489})

@item 
bpo-39573@footnote{https://bugs.python.org/issue39573}: Convert @ref{1f1,,Py_TYPE()} and @ref{1f3,,Py_SIZE()} back to macros
to allow using them as an l-value. Many third party C extension modules
rely on the ability of using Py_TYPE() and Py_SIZE() to set an object type
and size: @code{Py_TYPE(obj) = type;} and @code{Py_SIZE(obj) = size;}.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @ref{f6,,symtable} extension module to multiphase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489})

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @ref{86,,grp} and @ref{d1,,pwd} extension modules to
multiphase initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489})

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _random extension module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489})

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _hashlib extension module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489})

@item 
bpo-41713@footnote{https://bugs.python.org/issue41713}: Removed the undocumented @code{PyOS_InitInterrupts()} function.
Initializing Python already implicitly installs signal handlers: see
@ref{3a1,,PyConfig.install_signal_handlers}. Patch by Victor Stinner.

@item 
bpo-40170@footnote{https://bugs.python.org/issue40170}: The @code{Py_TRASHCAN_BEGIN} macro no longer accesses PyTypeObject
attributes, but now can get the condition by calling the new private
@code{_PyTrash_cond()} function which hides implementation details.

@item 
bpo-42260@footnote{https://bugs.python.org/issue42260}: @ref{387,,Py_GetPath()}, @ref{388,,Py_GetPrefix()},
@ref{389,,Py_GetExecPrefix()}, @ref{38a,,Py_GetProgramFullPath()},
@ref{38b,,Py_GetPythonHome()} and @ref{38c,,Py_GetProgramName()} functions now
return @code{NULL} if called before @ref{38d,,Py_Initialize()} (before Python is
initialized). Use the new @ref{207,,Python Initialization Configuration API} to get the @ref{38e,,Python Path Configuration.}. Patch by Victor Stinner.

@item 
bpo-42260@footnote{https://bugs.python.org/issue42260}: The @code{PyConfig_Read()} function now only parses
@ref{11a1,,PyConfig.argv} arguments once: @ref{11a2,,PyConfig.parse_argv}
is set to @code{2} after arguments are parsed. Since Python arguments are
strippped from @ref{11a1,,PyConfig.argv}, parsing arguments twice would
parse the application options as Python options.

@item 
bpo-42262@footnote{https://bugs.python.org/issue42262}: Added @ref{36a,,Py_NewRef()} and @ref{36b,,Py_XNewRef()} functions to
increment the reference count of an object and return the object. Patch by
Victor Stinner.

@item 
bpo-42260@footnote{https://bugs.python.org/issue42260}: When @ref{38d,,Py_Initialize()} is called twice, the second call
now updates more @ref{f7,,sys} attributes for the configuration, rather than
only @ref{e15,,sys.argv}. Patch by Victor Stinner.

@item 
bpo-41832@footnote{https://bugs.python.org/issue41832}: The @ref{36d,,PyType_FromModuleAndSpec()} function now accepts
NULL @code{tp_doc} slot.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Added @ref{368,,PyModule_AddObjectRef()} function: similar to
@ref{369,,PyModule_AddObject()} but don’t steal a reference to the value on
success. Patch by Victor Stinner.

@item 
bpo-42171@footnote{https://bugs.python.org/issue42171}: The @code{METH_FASTCALL} calling convention is added to the
limited API. The functions  @ref{47c,,PyModule_AddType()},
@ref{36d,,PyType_FromModuleAndSpec()}, @ref{474,,PyType_GetModule()} and
@ref{475,,PyType_GetModuleState()} are added to the limited API on Windows.

@item 
bpo-42085@footnote{https://bugs.python.org/issue42085}: Add dedicated entry to PyAsyncMethods for sending values

@item 
bpo-41073@footnote{https://bugs.python.org/issue41073}: @ref{36e,,PyType_GetSlot()} can now accept static types.

@item 
bpo-30459@footnote{https://bugs.python.org/issue30459}: @ref{38f,,PyList_SET_ITEM()}, @ref{390,,PyTuple_SET_ITEM()} and
@ref{391,,PyCell_SET()} macros can no longer be used as l-value or r-value.
For example, @code{x = PyList_SET_ITEM(a, b, c)} and @code{PyList_SET_ITEM(a, b,
c) = x} now fail with a compiler error. It prevents bugs like @code{if
(PyList_SET_ITEM (a, b, c) < 0) ...} test. Patch by Zackery Spytz and
Victor Stinner.
@end itemize

@node Python 3 10 0 alpha 2,Python 3 10 0 alpha 1,Python 3 10 0 alpha 3,Changelog
@anchor{whatsnew/changelog python-3-10-0-alpha-2}@anchor{11a3}
@subsection Python 3.10.0 alpha 2


@emph{Release date: 2020-11-03}

@menu
* Security: Security<7>. 
* Core and Builtins: Core and Builtins<13>. 
* Library: Library<13>. 
* Documentation: Documentation<14>. 
* Tests: Tests<13>. 
* Build: Build<13>. 
* Windows: Windows<12>. 
* macOS: macOS<11>. 
* IDLE: IDLE<12>. 
* C API: C API<13>. 

@end menu

@node Security<7>,Core and Builtins<13>,,Python 3 10 0 alpha 2
@anchor{whatsnew/changelog id102}@anchor{11a4}
@subsubsection Security



@itemize -

@item 
bpo-42103@footnote{https://bugs.python.org/issue42103}: Prevented potential DoS attack via CPU and RAM exhaustion when
processing malformed Apple Property List files in binary format.

@item 
bpo-42051@footnote{https://bugs.python.org/issue42051}: The @ref{ca,,plistlib} module no longer accepts entity
declarations in XML plist files to avoid XML vulnerabilities. This should
not affect users as entity declarations are not used in regular plist
files.
@end itemize

@node Core and Builtins<13>,Library<13>,Security<7>,Python 3 10 0 alpha 2
@anchor{whatsnew/changelog id103}@anchor{11a5}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-42236@footnote{https://bugs.python.org/issue42236}: If the @code{nl_langinfo(CODESET)} function returns an empty
string, Python now uses UTF-8 as the filesystem encoding. Patch by Victor
Stinner.

@item 
bpo-42218@footnote{https://bugs.python.org/issue42218}: Fixed a bug in the PEG parser that was causing crashes in debug
mode. Now errors are checked in left-recursive rules to avoid cases where
such errors do not get handled in time and appear as long-distance crashes
in other places.

@item 
bpo-42214@footnote{https://bugs.python.org/issue42214}: Fixed a possible crash in the PEG parser when checking for the
‘!=’ token in the @code{barry_as_flufl} rule. Patch by Pablo Galindo.

@item 
bpo-42206@footnote{https://bugs.python.org/issue42206}: Propagate and raise the errors caused by
@code{PyAST_Validate()} in the parser.

@item 
bpo-41796@footnote{https://bugs.python.org/issue41796}: The @ref{7,,ast} module internal state is now per interpreter.
Patch by Victor Stinner.

@item 
bpo-42143@footnote{https://bugs.python.org/issue42143}: Fix handling of errors during creation of @code{PyFunctionObject},
which resulted in operations on uninitialized memory. Patch by Yonatan
Goldschmidt.

@item 
bpo-41659@footnote{https://bugs.python.org/issue41659}: Fix a bug in the parser, where a curly brace following a
@code{primary} didn’t fail immediately. This led to invalid expressions like @code{a
@{b@}} to throw a @ref{21a,,SyntaxError} with a wrong offset, or invalid
expressions ending with a curly brace like @code{a @{} to not fail immediately
in the REPL.

@item 
bpo-42150@footnote{https://bugs.python.org/issue42150}: Fix possible buffer overflow in the new parser when checking
for continuation lines. Patch by Pablo Galindo.

@item 
bpo-42123@footnote{https://bugs.python.org/issue42123}: Run the parser two times. On the first run, disable all the
rules that only generate better error messages to gain performance. If
there’s a parse failure, run the parser a second time with those enabled.

@item 
bpo-42093@footnote{https://bugs.python.org/issue42093}: The @code{LOAD_ATTR} instruction now uses new “per opcode cache”
mechanism and it is about 36% faster now. Patch by Pablo Galindo and Yury
Selivanov.

@item 
bpo-42030@footnote{https://bugs.python.org/issue42030}: Support for the legacy AIX-specific shared library loading
support has been removed. All versions of AIX since 4.3 have supported and
defaulted to using the common Unix mechanism instead.

@item 
bpo-41984@footnote{https://bugs.python.org/issue41984}: The garbage collector now tracks all user-defined classes.
Patch by Brandt Bucher.

@item 
bpo-41993@footnote{https://bugs.python.org/issue41993}: Fixed potential issues with removing not completely initialized
module from @code{sys.modules} when import fails.

@item 
bpo-41979@footnote{https://bugs.python.org/issue41979}: Star-unpacking is now allowed for with item’s targets in the
PEG parser.

@item 
bpo-41974@footnote{https://bugs.python.org/issue41974}: Removed special methods @code{__int__}, @code{__float__},
@code{__floordiv__}, @code{__mod__}, @code{__divmod__}, @code{__rfloordiv__},
@code{__rmod__} and @code{__rdivmod__} of the @ref{33b,,complex} class.  They
always raised a @ref{162,,TypeError}.

@item 
bpo-41902@footnote{https://bugs.python.org/issue41902}: Micro optimization when compute
@ref{11a6,,sq_item} and
@ref{11a7,,mp_subscript} of @ref{438,,range}. Patch by
Dong-hee Na.

@item 
bpo-41894@footnote{https://bugs.python.org/issue41894}: When loading a native module and a load failure occurs, prevent
a possible UnicodeDecodeError when not running in a UTF-8 locale by
decoding the load error message using the current locale’s encoding.

@item 
bpo-41902@footnote{https://bugs.python.org/issue41902}: Micro optimization for range.index if step is 1. Patch by
Dong-hee Na.

@item 
bpo-41435@footnote{https://bugs.python.org/issue41435}: Add @ref{11a8,,sys._current_exceptions()} function to retrieve a
dictionary mapping each thread’s identifier to the topmost exception
currently active in that thread at the time the function is called.

@item 
bpo-38605@footnote{https://bugs.python.org/issue38605}: Enable @code{from __future__ import annotations} (
@geindex Python Enhancement Proposals; PEP 563
PEP 563@footnote{https://www.python.org/dev/peps/pep-0563}) by
default. The values found in @code{__annotations__} dicts are now
strings, e.g. @code{@{"x": "int"@}} instead of @code{@{"x": int@}}.
@end itemize

@node Library<13>,Documentation<14>,Core and Builtins<13>,Python 3 10 0 alpha 2
@anchor{whatsnew/changelog id104}@anchor{11a9}
@subsubsection Library



@itemize -

@item 
bpo-35455@footnote{https://bugs.python.org/issue35455}: On Solaris, @ref{42a,,thread_time()} is now implemented with
@code{gethrvtime()} because @code{clock_gettime(CLOCK_THREAD_CPUTIME_ID)} is not
always available. Patch by Jakub Kulik.

@item 
bpo-42233@footnote{https://bugs.python.org/issue42233}: The @ref{27d,,repr()} of @ref{120,,typing} types containing
@ref{268,,Generic Alias Types} previously did not show
the parameterized types in the @code{GenericAlias}.  They have now been
changed to do so.

@item 
bpo-29566@footnote{https://bugs.python.org/issue29566}: @code{binhex.binhex()} consistently writes macOS 9 line endings.

@item 
bpo-26789@footnote{https://bugs.python.org/issue26789}: The @ref{1104,,logging.FileHandler} class now keeps a reference to
the builtin @ref{1c8,,open()} function to be able to open or reopen the file
during Python finalization. Fix errors like: @code{NameError: name 'open' is
not defined}. Patch by Victor Stinner.

@item 
bpo-42157@footnote{https://bugs.python.org/issue42157}: Removed the @code{unicodedata.ucnhash_CAPI} attribute which was an
internal PyCapsule object. The related private @code{_PyUnicode_Name_CAPI}
structure was moved to the internal C API. Patch by Victor Stinner.

@item 
bpo-42157@footnote{https://bugs.python.org/issue42157}: Convert the @ref{121,,unicodedata} extension module to the
multiphase initialization API (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}) and convert the
@code{unicodedata.UCD} static type to a heap type. Patch by Mohamed Koubaa
and Victor Stinner.

@item 
bpo-42146@footnote{https://bugs.python.org/issue42146}: Fix memory leak in @ref{5ae,,subprocess.Popen()} in case an uid
(gid) specified in @code{user} (@code{group}, @code{extra_groups}) overflows @code{uid_t}
(@code{gid_t}).

@item 
bpo-42103@footnote{https://bugs.python.org/issue42103}: @code{InvalidFileException} and @ref{8c4,,RecursionError}
are now the only errors caused by loading malformed binary Plist file
(previously ValueError and TypeError could be raised in some specific
cases).

@item 
bpo-41490@footnote{https://bugs.python.org/issue41490}: In @code{importlib.resources}, @code{.path} method is more aggressive
about releasing handles to zipfile objects early, enabling use-cases like
certifi to leave the context open but delete the underlying zip file.

@item 
bpo-41052@footnote{https://bugs.python.org/issue41052}: Pickling heap types implemented in C with protocols 0 and 1
raises now an error instead of producing incorrect data.

@item 
bpo-42089@footnote{https://bugs.python.org/issue42089}: In @code{importlib.metadata.PackageNotFoundError}, make reference
to the package metadata being missing to improve the user experience.

@item 
bpo-41491@footnote{https://bugs.python.org/issue41491}: plistlib: fix parsing XML plists with hexadecimal integer
values

@item 
bpo-42065@footnote{https://bugs.python.org/issue42065}: Fix an incorrectly formatted error from
@code{_codecs.charmap_decode()} when called with a mapped value outside the
range of valid Unicode code points. PR by Max Bernstein.

@item 
bpo-41966@footnote{https://bugs.python.org/issue41966}: Fix pickling pure Python @ref{364,,datetime.time} subclasses.
Patch by Dean Inwood.

@item 
bpo-19270@footnote{https://bugs.python.org/issue19270}: @ref{11aa,,sched.scheduler.cancel()} will now cancel the correct
event, if two events with same priority are scheduled for the same time.
Patch by Bar Harel.

@item 
bpo-28660@footnote{https://bugs.python.org/issue28660}: @ref{111c,,textwrap.wrap()} now attempts to break long words after
hyphens when @code{break_long_words=True} and @code{break_on_hyphens=True}.

@item 
bpo-35823@footnote{https://bugs.python.org/issue35823}: Use @code{vfork()} instead of @code{fork()} for
@ref{5ae,,subprocess.Popen()} on Linux to improve performance in cases where it
is deemed safe.

@item 
bpo-42043@footnote{https://bugs.python.org/issue42043}: Add support for @code{zipfile.Path} inheritance.
@code{zipfile.Path.is_file()} now returns False for non-existent names.
@code{zipfile.Path} objects now expose a @code{.filename} attribute and rely on
that to resolve @code{.name} and @code{.parent} when the @code{Path} object is at
the root of the zipfile.

@item 
bpo-42021@footnote{https://bugs.python.org/issue42021}: Fix possible ref leaks in @ref{ed,,sqlite3} module init.

@item 
bpo-39101@footnote{https://bugs.python.org/issue39101}: Fixed tests using IsolatedAsyncioTestCase from hanging on
BaseExceptions.

@item 
bpo-41976@footnote{https://bugs.python.org/issue41976}: Fixed a bug that was causing @ref{8a0,,ctypes.util.find_library()}
to return @code{None} when triying to locate a library in an environment when
gcc>=9 is available and @code{ldconfig} is not. Patch by Pablo Galindo

@item 
bpo-41943@footnote{https://bugs.python.org/issue41943}: Fix bug where TestCase.assertLogs doesn’t correctly filter
messages by level.

@item 
bpo-41923@footnote{https://bugs.python.org/issue41923}: Implement 
@geindex Python Enhancement Proposals; PEP 613
PEP 613@footnote{https://www.python.org/dev/peps/pep-0613}, introducing @ref{11ab,,typing.TypeAlias}
annotation.

@item 
bpo-41905@footnote{https://bugs.python.org/issue41905}: A new function in abc: @emph{update_abstractmethods} to re-calculate
an abstract class’s abstract status. In addition, @emph{dataclass} has been
changed to call this function.

@item 
bpo-23706@footnote{https://bugs.python.org/issue23706}: Added @emph{newline} parameter to @code{pathlib.Path.write_text()}.

@item 
bpo-41876@footnote{https://bugs.python.org/issue41876}: Tkinter font class repr uses font name

@item 
bpo-41831@footnote{https://bugs.python.org/issue41831}: @code{str()} for the @code{type} attribute of the @code{tkinter.Event}
object always returns now the numeric code returned by Tk instead of the
name of the event type.

@item 
bpo-39337@footnote{https://bugs.python.org/issue39337}: @code{encodings.normalize_encoding()} now ignores non-ASCII
characters.

@item 
bpo-41747@footnote{https://bugs.python.org/issue41747}: Ensure all methods that generated from
@ref{276,,dataclasses.dataclass()} objects now have the proper @code{__qualname__}
attribute referring to the class they belong to. Patch by Batuhan Taskaya.

@item 
bpo-30681@footnote{https://bugs.python.org/issue30681}: Handle exceptions caused by unparsable date headers when using
email “default” policy.  Patch by Tim Bell, Georges Toth

@item 
bpo-41586@footnote{https://bugs.python.org/issue41586}: Add F_SETPIPE_SZ and F_GETPIPE_SZ to fcntl module. Allow
setting pipesize on subprocess.Popen.

@item 
bpo-41229@footnote{https://bugs.python.org/issue41229}: Add @code{contextlib.aclosing} for deterministic cleanup of async
generators which is analogous to @code{contextlib.closing} for non-async
generators. Patch by Joongi Kim and John Belmonte.

@item 
bpo-16396@footnote{https://bugs.python.org/issue16396}: Allow @code{ctypes.wintypes} to be imported on non-Windows
systems.

@item 
bpo-4356@footnote{https://bugs.python.org/issue4356}: Add a key function to the bisect module.

@item 
bpo-40592@footnote{https://bugs.python.org/issue40592}: @ref{11ac,,shutil.which()} now ignores empty entries in
@geindex PATHEXT
@geindex environment variable; PATHEXT
@code{PATHEXT} instead of treating them as a match.

@item 
bpo-40492@footnote{https://bugs.python.org/issue40492}: Fix @code{--outfile} for @ref{26,,cProfile} / @ref{ce,,profile} not
writing the output file in the original directory when the program being
profiled changes the working directory.  PR by Anthony Sottile.

@item 
bpo-34204@footnote{https://bugs.python.org/issue34204}: The @ref{e2,,shelve} module now uses
@ref{2b0,,pickle.DEFAULT_PROTOCOL} by default instead of @ref{c5,,pickle}
protocol @code{3}.

@item 
bpo-27321@footnote{https://bugs.python.org/issue27321}: Fixed KeyError exception when flattening an email to a string
attempts to replace a non-existent Content-Transfer-Encoding header.

@item 
bpo-38976@footnote{https://bugs.python.org/issue38976}: The @ref{90,,http.cookiejar} module now supports the parsing of
cookies in CURL-style cookiejar files through MozillaCookieJar on all
platforms. Previously, such cookie entries would be silently ignored when
loading a cookiejar with such entries.

Additionally, the HTTP Only attribute is persisted in the object, and will
be correctly written to file if the MozillaCookieJar object is
subsequently dumped.
@end itemize

@node Documentation<14>,Tests<13>,Library<13>,Python 3 10 0 alpha 2
@anchor{whatsnew/changelog id105}@anchor{11ad}
@subsubsection Documentation



@itemize -

@item 
bpo-42061@footnote{https://bugs.python.org/issue42061}: Document __format__ functionality for IP addresses.

@item 
bpo-41910@footnote{https://bugs.python.org/issue41910}: Document the default implementation of @ref{618,,object.__eq__}.

@item 
bpo-42010@footnote{https://bugs.python.org/issue42010}: Clarify that subscription expressions are also valid for
certain @ref{11ae,,classes} and @ref{11af,,types} in the standard
library, and for user-defined classes and types if the classmethod
@code{__class_getitem__()} is provided.

@item 
bpo-41805@footnote{https://bugs.python.org/issue41805}: Documented @ref{268,,generic alias type} and
@ref{26a,,types.GenericAlias}. Also added an entry in glossary for
@ref{11b0,,generic types}.

@item 
bpo-39693@footnote{https://bugs.python.org/issue39693}: Fix tarfile’s extractfile documentation

@item 
bpo-39416@footnote{https://bugs.python.org/issue39416}: Document some restrictions on the default string
representations of numeric classes.
@end itemize

@node Tests<13>,Build<13>,Documentation<14>,Python 3 10 0 alpha 2
@anchor{whatsnew/changelog id106}@anchor{11b1}
@subsubsection Tests



@itemize -

@item 
bpo-41739@footnote{https://bugs.python.org/issue41739}: Fix test_logging.test_race_between_set_target_and_flush(): the
test now waits until all threads complete to avoid leaking running
threads.

@item 
bpo-41970@footnote{https://bugs.python.org/issue41970}: Avoid a test failure in @code{test_lib2to3} if the module has
already imported at the time the test executes. Patch by Pablo Galindo.

@item 
bpo-41944@footnote{https://bugs.python.org/issue41944}: Tests for CJK codecs no longer call @code{eval()} on content
received via HTTP.

@item 
bpo-41306@footnote{https://bugs.python.org/issue41306}: Fixed a failure in @code{test_tk.test_widgets.ScaleTest} happening
when executing the test with Tk 8.6.10.
@end itemize

@node Build<13>,Windows<12>,Tests<13>,Python 3 10 0 alpha 2
@anchor{whatsnew/changelog id107}@anchor{11b2}
@subsubsection Build



@itemize -

@item 
bpo-38980@footnote{https://bugs.python.org/issue38980}: Add @code{-fno-semantic-interposition} to both the compile and
link line when building with @code{--enable-optimizations}. Patch by Victor
Stinner and Pablo Galindo.
@end itemize

@node Windows<12>,macOS<11>,Build<13>,Python 3 10 0 alpha 2
@anchor{whatsnew/changelog id108}@anchor{11b3}
@subsubsection Windows



@itemize -

@item 
bpo-38439@footnote{https://bugs.python.org/issue38439}: Updates the icons for IDLE in the Windows Store package.

@item 
bpo-38252@footnote{https://bugs.python.org/issue38252}: Use 8-byte step to detect ASCII sequence in 64-bit Windows
build.

@item 
bpo-39107@footnote{https://bugs.python.org/issue39107}: Update Tcl and Tk to 8.6.10 in Windows installer.

@item 
bpo-41557@footnote{https://bugs.python.org/issue41557}: Update Windows installer to use SQLite 3.33.0.

@item 
bpo-38324@footnote{https://bugs.python.org/issue38324}: Avoid Unicode errors when accessing certain locale data on
Windows.
@end itemize

@node macOS<11>,IDLE<12>,Windows<12>,Python 3 10 0 alpha 2
@anchor{whatsnew/changelog id109}@anchor{11b4}
@subsubsection macOS



@itemize -

@item 
bpo-41471@footnote{https://bugs.python.org/issue41471}: Ignore invalid prefix lengths in system proxy excludes.
@end itemize

@node IDLE<12>,C API<13>,macOS<11>,Python 3 10 0 alpha 2
@anchor{whatsnew/changelog id110}@anchor{11b5}
@subsubsection IDLE



@itemize -

@item 
bpo-33987@footnote{https://bugs.python.org/issue33987}: Mostly finish using ttk widgets, mainly for editor, settings,
and searches. Some patches by Mark Roseman.

@item 
bpo-40511@footnote{https://bugs.python.org/issue40511}: Typing opening and closing parentheses inside the parentheses
of a function call will no longer cause unnecessary “flashing” off and on
of an existing open call-tip, e.g. when typed in a string literal.

@item 
bpo-38439@footnote{https://bugs.python.org/issue38439}: Add a 256×256 pixel IDLE icon to the Windows .ico file. Created
by Andrew Clover. Remove the low-color gif variations from the .ico file.
@end itemize

@node C API<13>,,IDLE<12>,Python 3 10 0 alpha 2
@anchor{whatsnew/changelog id111}@anchor{11b6}
@subsubsection C API



@itemize -

@item 
bpo-42157@footnote{https://bugs.python.org/issue42157}: The private @code{_PyUnicode_Name_CAPI} structure of the PyCapsule
API @code{unicodedata.ucnhash_CAPI} has been moved to the internal C API.
Patch by Victor Stinner.

@item 
bpo-42015@footnote{https://bugs.python.org/issue42015}: Fix potential crash in deallocating method objects when
dynamically allocated @code{PyMethodDef}’s lifetime is managed through the
@code{self} argument of a @code{PyCFunction}.

@item 
bpo-40423@footnote{https://bugs.python.org/issue40423}: The @ref{f4,,subprocess} module and @code{os.closerange} will now use
the @code{close_range(low, high, flags)} syscall when it is available for
more efficient closing of ranges of descriptors.

@item 
bpo-41845@footnote{https://bugs.python.org/issue41845}: @ref{11b7,,PyObject_GenericGetDict()} is available again in the
limited API when targeting 3.10 or later.

@item 
bpo-40422@footnote{https://bugs.python.org/issue40422}: Add @code{_Py_closerange} function to provide performant closing of
a range of file descriptors.

@item 
bpo-41986@footnote{https://bugs.python.org/issue41986}: @code{Py_FileSystemDefaultEncodeErrors} and
@code{Py_UTF8Mode} are available again in limited API.

@item 
bpo-41756@footnote{https://bugs.python.org/issue41756}: Add @code{PyIter_Send} function to allow sending value into
generator/coroutine/iterator without raising StopIteration exception to
signal return.

@item 
bpo-41784@footnote{https://bugs.python.org/issue41784}: Added @code{PyUnicode_AsUTF8AndSize} to the limited C API.
@end itemize

@node Python 3 10 0 alpha 1,Python 3 9 0 beta 1,Python 3 10 0 alpha 2,Changelog
@anchor{whatsnew/changelog python-3-10-0-alpha-1}@anchor{11b8}
@subsection Python 3.10.0 alpha 1


@emph{Release date: 2020-10-05}

@menu
* Security: Security<8>. 
* Core and Builtins: Core and Builtins<14>. 
* Library: Library<14>. 
* Documentation: Documentation<15>. 
* Tests: Tests<14>. 
* Build: Build<14>. 
* Windows: Windows<13>. 
* macOS: macOS<12>. 
* IDLE: IDLE<13>. 
* C API: C API<14>. 

@end menu

@node Security<8>,Core and Builtins<14>,,Python 3 10 0 alpha 1
@anchor{whatsnew/changelog id112}@anchor{11b9}
@subsubsection Security



@itemize -

@item 
bpo-41304@footnote{https://bugs.python.org/issue41304}: Fixes @code{python3x._pth} being ignored on Windows, caused by the
fix for bpo-29778@footnote{https://bugs.python.org/issue29778} (CVE-2020-15801).

@item 
bpo-41162@footnote{https://bugs.python.org/issue41162}: Audit hooks are now cleared later during finalization to avoid
missing events.

@item 
bpo-29778@footnote{https://bugs.python.org/issue29778}: Ensure @code{python3.dll} is loaded from correct locations
when Python is embedded (CVE-2020-15523).

@item 
bpo-41004@footnote{https://bugs.python.org/issue41004}: The __hash__() methods of  ipaddress.IPv4Interface and
ipaddress.IPv6Interface incorrectly generated constant hash values of 32
and 128 respectively. This resulted in always causing hash collisions. The
fix uses hash() to generate hash values for the tuple of (address, mask
length, network address).

@item 
bpo-39603@footnote{https://bugs.python.org/issue39603}: Prevent http header injection by rejecting control characters
in http.client.putrequest(…).
@end itemize

@node Core and Builtins<14>,Library<14>,Security<8>,Python 3 10 0 alpha 1
@anchor{whatsnew/changelog id113}@anchor{11ba}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-41909@footnote{https://bugs.python.org/issue41909}: Fixed stack overflow in @ref{237,,issubclass()} and
@ref{236,,isinstance()} when getting the @code{__bases__} attribute leads to
infinite recursion.

@item 
bpo-41922@footnote{https://bugs.python.org/issue41922}: Speed up calls to @code{reversed()} by using the 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590}
@code{vectorcall} calling convention. Patch by Dong-hee Na.

@item 
bpo-41873@footnote{https://bugs.python.org/issue41873}: Calls to @code{float()} are now faster due to the @code{vectorcall}
calling convention. Patch by Dennis Sweeney.

@item 
bpo-41870@footnote{https://bugs.python.org/issue41870}: Speed up calls to @code{bool()} by using the 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590}
@code{vectorcall} calling convention. Patch by Dong-hee Na.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @code{_bisect} module to the multi-phase
initialization API (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-39934@footnote{https://bugs.python.org/issue39934}: Correctly count control blocks in ‘except’ in compiler. Ensures
that a syntax error, rather a fatal error, occurs for deeply nested, named
exception handlers.

@item 
bpo-41780@footnote{https://bugs.python.org/issue41780}: Fix @code{__dir__()} of @ref{26a,,types.GenericAlias}. Patch by
Batuhan Taskaya.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @code{_lsprof} extension module to multi-phase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @ref{17,,cmath} extension module to multi-phase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @code{_scproxy} extension module to multi-phase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @ref{fe,,termios} extension module to multi-phase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Convert the @code{_sha256} extension module types to heap
types.

@item 
bpo-41690@footnote{https://bugs.python.org/issue41690}: Fix a possible stack overflow in the parser when parsing
functions and classes with a huge amount of arguments. Patch by Pablo
Galindo.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @code{_overlapped} extension module to multi-phase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @code{_curses_panel} extension module to multi-phase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @code{_opcode} extension module to multi-phase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-41681@footnote{https://bugs.python.org/issue41681}: Fixes the wrong error description in the error raised by using
2 @code{,} in format string in f-string and @ref{784,,str.format()}.

@item 
bpo-41675@footnote{https://bugs.python.org/issue41675}: The implementation of @ref{cca,,signal.siginterrupt()} now uses
@code{sigaction()} (if it is available in the system) instead of the
deprecated @code{siginterrupt()}. Patch by Pablo Galindo.

@item 
bpo-41670@footnote{https://bugs.python.org/issue41670}: Prevent line trace being skipped on platforms not compiled with
@code{USE_COMPUTED_GOTOS}. Fixes issue where some lines nested within a
try-except block were not being traced on Windows.

@item 
bpo-41654@footnote{https://bugs.python.org/issue41654}: Fix a crash that occurred when destroying subclasses of
@ref{10de,,MemoryError}. Patch by Pablo Galindo.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @ref{14b,,zlib} extension module to multi-phase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-41631@footnote{https://bugs.python.org/issue41631}: The @code{_ast} module uses again a global state. Using a module
state per module instance is causing subtle practical problems. For
example, the Mercurial project replaces the @code{__import__()} function to
implement lazy import, whereas Python expected that @code{import _ast} always
return a fully initialized @code{_ast} module.

@item 
bpo-40077@footnote{https://bugs.python.org/issue40077}: Convert @code{_operator} to use @ref{469,,PyType_FromSpec()}.

@item 
bpo-1653741@footnote{https://bugs.python.org/issue1653741}: Port @code{_sha3} to multi-phase init.  Convert static types
to heap types.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @code{_blake2} extension module to the multi-phase
initialization API (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-41533@footnote{https://bugs.python.org/issue41533}: Free the stack allocated in @code{va_build_stack} if
@code{do_mkstack} fails and the stack is not a @code{small_stack}.

@item 
bpo-41531@footnote{https://bugs.python.org/issue41531}: Fix a bug that was dropping keys when compiling dict literals
with more than 0xFFFF elements. Patch by Pablo Galindo.

@item 
bpo-41525@footnote{https://bugs.python.org/issue41525}: The output of @code{python --help} contains now only ASCII
characters.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @code{_sha1}, @code{_sha512}, and @code{_md5}
extension modules to multi-phase initialization API (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-41431@footnote{https://bugs.python.org/issue41431}: Optimize @code{dict_merge()} for copying dict (e.g. @code{dict(d)}
and @code{@{@}.update(d)}).

@item 
bpo-41428@footnote{https://bugs.python.org/issue41428}: Implement PEP 604. This supports (int | str) etc. in place of
Union[str, int].

@item 
bpo-41340@footnote{https://bugs.python.org/issue41340}: Removed fallback implementation for @code{strdup}.

@item 
bpo-38156@footnote{https://bugs.python.org/issue38156}: Handle interrupts that come after EOF correctly in
@code{PyOS_StdioReadline}.

@item 
bpo-41342@footnote{https://bugs.python.org/issue41342}: @ref{e75,,round()} with integer argument is now faster (9–60%).

@item 
bpo-41334@footnote{https://bugs.python.org/issue41334}: Constructors @ref{2ef,,str()}, @ref{2f0,,bytes()} and @ref{2f1,,bytearray()}
are now faster (around 30–40% for small objects).

@item 
bpo-41295@footnote{https://bugs.python.org/issue41295}: Resolve a regression in CPython 3.8.4 where defining
“__setattr__” in a multi-inheritance setup and calling up the hierarchy
chain could fail if builtins/extension types were involved in the base
types.

@item 
bpo-41323@footnote{https://bugs.python.org/issue41323}: Bytecode optimizations are performed directly on the control
flow graph. This will result in slightly more compact code objects in some
circumstances.

@item 
bpo-41247@footnote{https://bugs.python.org/issue41247}: Always cache the running loop holder when running
@code{asyncio.set_running_loop}.

@item 
bpo-41252@footnote{https://bugs.python.org/issue41252}: Fix incorrect refcounting in _ssl.c’s
@code{_servername_callback()}.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @ref{b3,,multiprocessing} to multi-phase initialization

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @code{winapi} to multiphase initialization

@item 
bpo-41215@footnote{https://bugs.python.org/issue41215}: Use non-NULL default values in the PEG parser keyword list to
overcome a bug that was preventing Python from being properly compiled
when using the XLC compiler. Patch by Pablo Galindo.

@item 
bpo-41218@footnote{https://bugs.python.org/issue41218}: Python 3.8.3 had a regression where compiling with
ast.PyCF_ALLOW_TOP_LEVEL_AWAIT would aggressively mark list comprehension
with CO_COROUTINE. Now only list comprehension making use of async/await
will tagged as so.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @ref{78,,faulthandler} to multiphase initialization.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @code{sha256} to multiphase initialization

@item 
bpo-41175@footnote{https://bugs.python.org/issue41175}: Guard against a NULL pointer dereference within bytearrayobject
triggered by the @code{bytearray() + bytearray()} operation.

@item 
bpo-41100@footnote{https://bugs.python.org/issue41100}: add arm64 to the allowable Mac OS arches in mpdecimal.h

@item 
bpo-41094@footnote{https://bugs.python.org/issue41094}: Fix decoding errors with audit when open files with non-ASCII
names on non-UTF-8 locale.

@item 
bpo-39960@footnote{https://bugs.python.org/issue39960}: The “hackcheck” that prevents sneaking around a type’s
__setattr__() by calling the superclass method was rewritten to allow C
implemented heap types.

@item 
bpo-41084@footnote{https://bugs.python.org/issue41084}: Prefix the error message with ‘f-string: ‘, when parsing an
f-string expression which throws a @ref{21a,,SyntaxError}.

@item 
bpo-40521@footnote{https://bugs.python.org/issue40521}: Empty frozensets are no longer singletons.

@item 
bpo-41076@footnote{https://bugs.python.org/issue41076}: Pre-feed the parser with the location of the f-string
expression, not the f-string itself, which allows us to skip the shifting
of the AST node locations after the parsing is completed.

@item 
bpo-41056@footnote{https://bugs.python.org/issue41056}: Fixes a reference to deallocated stack space during startup
when constructing sys.path involving a relative symlink when code was
supplied via -c.  (discovered via Coverity)

@item 
bpo-41061@footnote{https://bugs.python.org/issue41061}: Fix incorrect expressions and asserts in hashtable code and
tests.

@item 
bpo-41052@footnote{https://bugs.python.org/issue41052}: Opt out serialization/deserialization for _random.Random

@item 
bpo-40939@footnote{https://bugs.python.org/issue40939}: Rename @code{PyPegen*} functions to @code{PyParser*}, so that we can
remove the old set of @code{PyParser*} functions that were using the old
parser, but keep everything backwards-compatible.

@item 
bpo-35975@footnote{https://bugs.python.org/issue35975}: Stefan Behnel reported that cf_feature_version is used even
when PyCF_ONLY_AST is not set. This is against the intention and against
the documented behavior, so it’s been fixed.

@item 
bpo-40939@footnote{https://bugs.python.org/issue40939}: Remove the remaining files from the old parser and the
@code{symbol} module.

@item 
bpo-40077@footnote{https://bugs.python.org/issue40077}: Convert @code{_bz2} to use @ref{469,,PyType_FromSpec()}.

@item 
bpo-41006@footnote{https://bugs.python.org/issue41006}: The @code{encodings.latin_1} module is no longer imported at
startup. Now it is only imported when it is the filesystem encoding or the
stdio encoding.

@item 
bpo-40636@footnote{https://bugs.python.org/issue40636}: @ref{247,,zip()} now supports 
@geindex Python Enhancement Proposals; PEP 618
PEP 618@footnote{https://www.python.org/dev/peps/pep-0618}’s @code{strict} parameter,
which raises a @ref{17c,,ValueError} if the arguments are exhausted at
different lengths. Patch by Brandt Bucher.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @code{_gdbm} to multiphase initialization.

@item 
bpo-40985@footnote{https://bugs.python.org/issue40985}: Fix a bug that caused the @ref{21a,,SyntaxError} text to be empty
when a file ends with a line ending in a line continuation character (i.e.
backslash). The error text should contain the text of the last line.

@item 
bpo-40958@footnote{https://bugs.python.org/issue40958}: Fix a possible buffer overflow in the PEG parser when gathering
information for emitting syntax errors. Patch by Pablo Galindo.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @code{_dbm} to multiphase initialization.

@item 
bpo-40957@footnote{https://bugs.python.org/issue40957}: Fix refleak in _Py_fopen_obj() when PySys_Audit() fails

@item 
bpo-40950@footnote{https://bugs.python.org/issue40950}: Add a state to the @ref{bb,,nis} module (
@geindex Python Enhancement Proposals; PEP 3121
PEP 3121@footnote{https://www.python.org/dev/peps/pep-3121}) and apply
the multiphase initialization. Patch by Dong-hee Na.

@item 
bpo-40947@footnote{https://bugs.python.org/issue40947}: The Python @ref{38e,,Path Configuration} now
takes @ref{11bb,,PyConfig.platlibdir} in account.

@item 
bpo-40939@footnote{https://bugs.python.org/issue40939}: Remove the old parser, the @code{parser} module and all
associated support code, command-line options and environment variables.
Patch by Pablo Galindo.

@item 
bpo-40847@footnote{https://bugs.python.org/issue40847}: Fix a bug where a line with only a line continuation character
is not considered a blank line at tokenizer level. In such cases, more
than a single @code{NEWLINE} token was emitted. The old parser was working
around the issue, but the new parser threw a @ref{21a,,SyntaxError} for valid
input due to this. For example, an empty line following a line
continuation character was interpreted as a @ref{21a,,SyntaxError}.

@item 
bpo-40890@footnote{https://bugs.python.org/issue40890}: Each dictionary view now has a @code{mapping} attribute that
provides a @ref{246,,types.MappingProxyType} wrapping the original
dictionary.  Patch contributed by Dennis Sweeney.

@item 
bpo-40889@footnote{https://bugs.python.org/issue40889}: Improved the performance of symmetric difference operations on
dictionary item views.  Patch by Dennis Sweeney.

@item 
bpo-40904@footnote{https://bugs.python.org/issue40904}: Fix possible segfault in the new PEG parser when parsing
f-string containing yield statements with no value (@code{f"@{yield@}"}).
Patch by Pablo Galindo

@item 
bpo-40903@footnote{https://bugs.python.org/issue40903}: Fixed a possible segfault in the new PEG parser when producing
error messages for invalid assignments of the form @code{p=p=}. Patch by
Pablo Galindo

@item 
bpo-40880@footnote{https://bugs.python.org/issue40880}: Fix invalid memory read in the new parser when checking
newlines in string literals. Patch by Pablo Galindo.

@item 
bpo-40883@footnote{https://bugs.python.org/issue40883}: Fix memory leak in when parsing f-strings in the new parser.
Patch by Pablo Galindo

@item 
bpo-40870@footnote{https://bugs.python.org/issue40870}: Raise @ref{17c,,ValueError} when validating custom AST’s where the
constants @code{True}, @code{False} and @code{None} are used within a
@ref{11bc,,ast.Name} node.

@item 
bpo-40854@footnote{https://bugs.python.org/issue40854}: Allow overriding @ref{42c,,sys.platlibdir} via a new
@geindex PYTHONPLATLIBDIR
@geindex environment variable; PYTHONPLATLIBDIR
@ref{11bd,,PYTHONPLATLIBDIR} environment variable.

@item 
bpo-40826@footnote{https://bugs.python.org/issue40826}: Fix GIL usage in @code{PyOS_Readline()}: lock the GIL to set
an exception and pass the Python thread state when checking if there is a
pending signal.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @ref{79,,fcntl} to multiphase initialization.

@item 
bpo-19468@footnote{https://bugs.python.org/issue19468}: Delete unnecessary instance check in importlib.reload(). Patch
by Furkan Önder.

@item 
bpo-40824@footnote{https://bugs.python.org/issue40824}: Unexpected errors in calling the @code{__iter__} method are no
longer masked by @code{TypeError} in the @ref{2fc,,in} operator and functions
@ref{3c0,,contains()}, @ref{3c1,,indexOf()} and
@ref{3c2,,countOf()} of the @ref{be,,operator} module.

@item 
bpo-40792@footnote{https://bugs.python.org/issue40792}: Attributes @code{start}, @code{stop} and @code{step} of the
@ref{438,,range} object now always has exact type @ref{1b1,,int}.  Previously,
they could have been an instance of a subclass of @code{int}.

@item 
bpo-40780@footnote{https://bugs.python.org/issue40780}: Fix a corner case where g-style string formatting of a float
failed to remove trailing zeros.

@item 
bpo-38964@footnote{https://bugs.python.org/issue38964}: When there’s a @ref{21a,,SyntaxError} in the expression part of an
fstring, the filename attribute of the @ref{21a,,SyntaxError} gets correctly
set to the name of the file the fstring resides in.

@item 
bpo-40750@footnote{https://bugs.python.org/issue40750}: Support the “-d” debug flag in the new PEG parser. Patch by
Pablo Galindo

@item 
bpo-40217@footnote{https://bugs.python.org/issue40217}: Instances of types created with
@ref{36c,,PyType_FromSpecWithBases()} will no longer automatically visit
their class object when traversing references in the garbage collector.
The user is expected to manually visit the object’s class. Patch by Pablo
Galindo.

@item 
bpo-39573@footnote{https://bugs.python.org/issue39573}: @ref{1f1,,Py_TYPE()} is changed to the inline static function.
Patch by Dong-hee Na.

@item 
bpo-40696@footnote{https://bugs.python.org/issue40696}: Fix a hang that can arise after @ref{11be,,generator.throw()} due to
a cycle in the exception context chain.

@item 
bpo-40521@footnote{https://bugs.python.org/issue40521}: Each interpreter now its has own free lists, singletons and
caches:


@itemize *

@item 
Free lists: float, tuple, list, dict, frame, context,
asynchronous generator, MemoryError.

@item 
Singletons: empty tuple, empty bytes string, empty Unicode string,
single byte character, single Unicode (latin1) character.

@item 
Slice cache.
@end itemize

They are no longer shared by all interpreters.

@item 
bpo-40679@footnote{https://bugs.python.org/issue40679}: Certain @ref{162,,TypeError} messages about missing or extra
arguments now include the function’s @ref{11bf,,qualified name}.  Patch by
Dennis Sweeney.

@item 
bpo-29590@footnote{https://bugs.python.org/issue29590}: Make the stack trace correct after calling
@ref{11be,,generator.throw()} on a generator that has yielded from a @code{yield
from}.

@item 
bpo-4022@footnote{https://bugs.python.org/issue4022}: Improve performance of generators by not raising internal
StopIteration.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @ref{af,,mmap} to multiphase initialization.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @code{_lzma} to multiphase initialization.

@item 
bpo-37999@footnote{https://bugs.python.org/issue37999}: Builtin and extension functions that take integer arguments no
longer accept @ref{248,,Decimal}s, @ref{172,,Fraction}s and other objects that can be converted to integers only with a loss
(e.g. that have the @ref{249,,__int__()} method but do not have the
@ref{24a,,__index__()} method).

@item 
bpo-29882@footnote{https://bugs.python.org/issue29882}: Add @ref{242,,int.bit_count()}, counting the number of ones in the
binary representation of an integer. Patch by Niklas Fiekas.

@item 
bpo-36982@footnote{https://bugs.python.org/issue36982}: Use ncurses extended color functions when available to support
terminals with 256 colors, and add the new function
@ref{273,,curses.has_extended_color_support()} to indicate whether extended
color support is provided by the underlying ncurses library.

@item 
bpo-19569@footnote{https://bugs.python.org/issue19569}: Add the private macros @code{_Py_COMP_DIAG_PUSH},
@code{_Py_COMP_DIAG_IGNORE_DEPR_DECLS}, and @code{_Py_COMP_DIAG_POP}.

@item 
bpo-26680@footnote{https://bugs.python.org/issue26680}: The int type now supports the x.is_integer() method for
compatibility with float.
@end itemize

@node Library<14>,Documentation<15>,Core and Builtins<14>,Python 3 10 0 alpha 1
@anchor{whatsnew/changelog id114}@anchor{11c0}
@subsubsection Library



@itemize -

@item 
bpo-41900@footnote{https://bugs.python.org/issue41900}: C14N 2.0 serialisation in xml.etree.ElementTree failed for
unprefixed attributes when a default namespace was defined.

@item 
bpo-41887@footnote{https://bugs.python.org/issue41887}: Strip leading spaces and tabs on @ref{754,,ast.literal_eval()}. Also
document stripping of spaces and tabs for @ref{34a,,eval()}.

@item 
bpo-41773@footnote{https://bugs.python.org/issue41773}: Note in documentation that @ref{80f,,random.choices()} doesn’t
support non-finite weights, raise @ref{17c,,ValueError} when given non-finite
weights.

@item 
bpo-41840@footnote{https://bugs.python.org/issue41840}: Fix a bug in the @ref{f6,,symtable} module that was causing
module-scope global variables to not be reported as both local and global.
Patch by Pablo Galindo.

@item 
bpo-41842@footnote{https://bugs.python.org/issue41842}: Add @ref{266,,codecs.unregister()} function to unregister a codec
search function.

@item 
bpo-40564@footnote{https://bugs.python.org/issue40564}: In @code{zipfile.Path}, mutate the passed ZipFile object type
instead of making a copy. Prevents issues when both the local copy and the
caller’s copy attempt to close the same file handle.

@item 
bpo-40670@footnote{https://bugs.python.org/issue40670}: More reliable validation of statements in
@ref{11c1,,timeit.Timer}. It now accepts “empty” statements (only whitespaces
and comments) and rejects misindentent statements.

@item 
bpo-41833@footnote{https://bugs.python.org/issue41833}: The @ref{44b,,threading.Thread} constructor now uses the target
name if the @emph{target} argument is specified but the @emph{name} argument is
omitted.

@item 
bpo-41817@footnote{https://bugs.python.org/issue41817}: fix @code{tkinter.EventType} Enum so all members are strings, and
none are tuples

@item 
bpo-41810@footnote{https://bugs.python.org/issue41810}: @ref{2d7,,types.EllipsisType}, @ref{2d9,,types.NotImplementedType}
and @ref{2d8,,types.NoneType} have been reintroduced, providing a new set of
types readily interpretable by static type checkers.

@item 
bpo-41815@footnote{https://bugs.python.org/issue41815}: Fix SQLite3 segfault when backing up closed database. Patch
contributed by Peter David McCormick.

@item 
bpo-41816@footnote{https://bugs.python.org/issue41816}: StrEnum added: it ensures that all members are already strings
or string candidates

@item 
bpo-41517@footnote{https://bugs.python.org/issue41517}: fix bug allowing Enums to be extended via multiple inheritance

@item 
bpo-39587@footnote{https://bugs.python.org/issue39587}: use the correct mix-in data type when constructing Enums

@item 
bpo-41792@footnote{https://bugs.python.org/issue41792}: Add is_typeddict function to typing.py to check if a type is a
TypedDict class

Previously there was no way to check that without using private API. See
the @code{relevant issue in python/typing}

@item 
bpo-41789@footnote{https://bugs.python.org/issue41789}: Honor @ref{5a5,,object} overrides in @code{Enum} class creation
(specifically, @code{__str__}, @code{__repr__}, @code{__format__}, and @code{__reduce_ex__}).

@item 
bpo-32218@footnote{https://bugs.python.org/issue32218}: @ref{7e7,,enum.Flag} and @ref{11c2,,enum.IntFlag} members are now iterable

@item 
bpo-39651@footnote{https://bugs.python.org/issue39651}: Fix a race condition in the @code{call_soon_threadsafe()} method
of @code{asyncio.ProactorEventLoop}: do nothing if the self-pipe socket has
been closed.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @code{mashal} extension module to the multi-phase
initialization API (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port the @code{_string} extension module to the multi-phase
initialization API (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-41732@footnote{https://bugs.python.org/issue41732}: Added an @ref{11c3,,iterator} to @ref{573,,memoryview}.

@item 
bpo-41720@footnote{https://bugs.python.org/issue41720}: Fixed @code{turtle.Vec2D.__rmul__()} for arguments which are not
int or float.

@item 
bpo-41696@footnote{https://bugs.python.org/issue41696}: Fix handling of debug mode in @ref{3d1,,asyncio.run()}. This allows
setting @code{PYTHONASYNCIODEBUG} or @code{-X dev} to enable asyncio debug mode
when using @ref{3d1,,asyncio.run()}.

@item 
bpo-41687@footnote{https://bugs.python.org/issue41687}: Fix implementation of sendfile to be compatible with Solaris.

@item 
bpo-41662@footnote{https://bugs.python.org/issue41662}: No longer override exceptions raised in @code{__len__()} of a
sequence of parameters in @ref{ed,,sqlite3} with
@ref{186,,ProgrammingError}.

@item 
bpo-39010@footnote{https://bugs.python.org/issue39010}: Restarting a @code{ProactorEventLoop} on Windows no longer logs
spurious @code{ConnectionResetErrors}.

@item 
bpo-41638@footnote{https://bugs.python.org/issue41638}: @ref{186,,ProgrammingError} message for absent parameter
in @ref{ed,,sqlite3} contains now the name of the parameter instead of its
index when parameters are supplied as a dict.

@item 
bpo-41662@footnote{https://bugs.python.org/issue41662}: Fixed crash when mutate list of parameters during iteration in
@ref{ed,,sqlite3}.

@item 
bpo-41513@footnote{https://bugs.python.org/issue41513}: Improved the accuracy of math.hypot().  Internally, each step
is computed with extra precision so that the result is now almost always
correctly rounded.

@item 
bpo-41609@footnote{https://bugs.python.org/issue41609}: The pdb whatis command correctly reports instance methods as
‘Method’ rather than ‘Function’.

@item 
bpo-39994@footnote{https://bugs.python.org/issue39994}: Fixed pprint’s handling of dict subclasses that override
__repr__.

@item 
bpo-32751@footnote{https://bugs.python.org/issue32751}: When cancelling the task due to a timeout,
@ref{3d5,,asyncio.wait_for()} will now wait until the cancellation is complete
also in the case when @emph{timeout} is <= 0, like it does with positive
timeouts.

@item 
bpo-37658@footnote{https://bugs.python.org/issue37658}: @ref{3d5,,asyncio.wait_for()} now properly handles races between
cancellation of itself and the completion of the wrapped awaitable.

@item 
bpo-40782@footnote{https://bugs.python.org/issue40782}: Change the method asyncio.AbstractEventLoop.run_in_executor to
not be a coroutine.

@item 
bpo-41520@footnote{https://bugs.python.org/issue41520}: Fix @ref{1b,,codeop} regression that prevented turning compile
warnings into errors.

@item 
bpo-41528@footnote{https://bugs.python.org/issue41528}: turtle uses math module functions to convert degrees to radians
and vice versa and to calculate vector norm

@item 
bpo-41513@footnote{https://bugs.python.org/issue41513}: Minor algorithmic improvement to math.hypot() and math.dist()
giving small gains in speed and accuracy.

@item 
bpo-41503@footnote{https://bugs.python.org/issue41503}: Fixed a race between setTarget and flush in
logging.handlers.MemoryHandler.

@item 
bpo-41497@footnote{https://bugs.python.org/issue41497}: Fix potential UnicodeDecodeError in dis module.

@item 
bpo-41467@footnote{https://bugs.python.org/issue41467}: On Windows, fix asyncio @code{recv_into()} return value when the
socket/pipe is closed (@ref{bf2,,BrokenPipeError}): return @code{0} rather than
an empty byte string (@code{b'@w{'}}).

@item 
bpo-41425@footnote{https://bugs.python.org/issue41425}: Make tkinter doc example runnable.

@item 
bpo-41421@footnote{https://bugs.python.org/issue41421}: Make an algebraic simplification to random.paretovariate().  It
now is slightly less subject to round-off error and is slightly faster.
Inputs that used to cause ZeroDivisionError now cause an OverflowError
instead.

@item 
bpo-41440@footnote{https://bugs.python.org/issue41440}: Add @ref{298,,os.cpu_count()} support for VxWorks RTOS.

@item 
bpo-41316@footnote{https://bugs.python.org/issue41316}: Fix the @ref{fb,,tarfile} module to write only basename of TAR
file to GZIP compression header.

@item 
bpo-41384@footnote{https://bugs.python.org/issue41384}: Raise TclError instead of TypeError when an unknown option is
passed to tkinter.OptionMenu.

@item 
bpo-41317@footnote{https://bugs.python.org/issue41317}: Use add_done_callback() in asyncio.loop.sock_accept() to
unsubscribe reader early on cancellation.

@item 
bpo-41364@footnote{https://bugs.python.org/issue41364}: Reduce import overhead of @ref{12b,,uuid}.

@item 
bpo-35328@footnote{https://bugs.python.org/issue35328}: Set the environment variable @code{VIRTUAL_ENV_PROMPT} at
@ref{12c,,venv} activation.

@item 
bpo-41341@footnote{https://bugs.python.org/issue41341}: Recursive evaluation of @ref{1090,,typing.ForwardRef} in
@code{get_type_hints}.

@item 
bpo-41344@footnote{https://bugs.python.org/issue41344}: Prevent creating @code{shared_memory.SharedMemory} objects
with @code{size=0}.

@item 
bpo-41333@footnote{https://bugs.python.org/issue41333}: @code{collections.OrderedDict.pop()} is now 2 times faster.

@item 
bpo-41288@footnote{https://bugs.python.org/issue41288}: Unpickling invalid NEWOBJ_EX opcode with the C implementation
raises now UnpicklingError instead of crashing.

@item 
bpo-39017@footnote{https://bugs.python.org/issue39017}: Avoid infinite loop when reading specially crafted TAR files
using the tarfile module (CVE-2019-20907).

@item 
bpo-41273@footnote{https://bugs.python.org/issue41273}: Speed up any transport using @code{_ProactorReadPipeTransport} by
calling @code{recv_into} instead of @code{recv}, thus not creating a new buffer
for each @code{recv} call in the transport’s read loop.

@item 
bpo-41235@footnote{https://bugs.python.org/issue41235}: Fix the error handling in
@ref{cda,,ssl.SSLContext.load_dh_params()}.

@item 
bpo-41207@footnote{https://bugs.python.org/issue41207}: In distutils.spawn, restore expectation that DistutilsExecError
is raised when the command is not found.

@item 
bpo-29727@footnote{https://bugs.python.org/issue29727}: Register @ref{25e,,array.array} as a
@ref{933,,MutableSequence}. Patch by Pablo Galindo.

@item 
bpo-39168@footnote{https://bugs.python.org/issue39168}: Remove the @code{__new__} method of @ref{11c4,,typing.Generic}.

@item 
bpo-41194@footnote{https://bugs.python.org/issue41194}: Fix a crash in the @code{_ast} module: it can no longer be loaded
more than once. It now uses a global state rather than a module state.

@item 
bpo-41195@footnote{https://bugs.python.org/issue41195}: Add read-only ssl.SSLContext.security_level attribute to
retrieve the context’s security level.

@item 
bpo-41193@footnote{https://bugs.python.org/issue41193}: The @code{write_history()} atexit function of the readline
completer now ignores any @ref{2a1,,OSError} to ignore error if the filesystem
is read-only, instead of only ignoring @ref{a37,,FileNotFoundError} and
@ref{895,,PermissionError}.

@item 
bpo-41182@footnote{https://bugs.python.org/issue41182}: selector: use DefaultSelector based upon implementation

@item 
bpo-41161@footnote{https://bugs.python.org/issue41161}: The decimal module now requires libmpdec-2.5.0. Users of
–with-system-libmpdec should update their system library.

@item 
bpo-40874@footnote{https://bugs.python.org/issue40874}: The decimal module now requires libmpdec-2.5.0.

@item 
bpo-41138@footnote{https://bugs.python.org/issue41138}: Fixed the @ref{119,,trace} module CLI for Python source files with
non-UTF-8 encoding.

@item 
bpo-31082@footnote{https://bugs.python.org/issue31082}: Use the term “iterable” in the docstring for
@ref{e77,,functools.reduce()}.

@item 
bpo-40521@footnote{https://bugs.python.org/issue40521}: Remove freelist from collections.deque().

@item 
bpo-31938@footnote{https://bugs.python.org/issue31938}: Fix default-value signatures of several functions in the
@ref{e0,,select} module - by Anthony Sottile.

@item 
bpo-41068@footnote{https://bugs.python.org/issue41068}: Fixed reading files with non-ASCII names from ZIP archive
directly after writing them.

@item 
bpo-41058@footnote{https://bugs.python.org/issue41058}: @code{pdb.find_function()} now correctly determines the source
file encoding.

@item 
bpo-41056@footnote{https://bugs.python.org/issue41056}: Invalid file descriptor values are now prevented from being
passed to os.fpathconf. (discovered by Coverity)

@item 
bpo-41056@footnote{https://bugs.python.org/issue41056}: Fix a NULL pointer dereference within the ssl module during a
MemoryError in the keylog callback. (discovered by Coverity)

@item 
bpo-41056@footnote{https://bugs.python.org/issue41056}: Fixed an instance where a MemoryError within the zoneinfo
module might not be reported or not reported at its source. (found by
Coverity)

@item 
bpo-41048@footnote{https://bugs.python.org/issue41048}: @ref{11c5,,mimetypes.read_mime_types()} function reads the rule file
using UTF-8 encoding, not the locale encoding. Patch by Srinivas Reddy
Thatiparthy.

@item 
bpo-41043@footnote{https://bugs.python.org/issue41043}: Fixed the use of @ref{288,,glob()} in the stdlib: literal part
of the path is now always correctly escaped.

@item 
bpo-41025@footnote{https://bugs.python.org/issue41025}: Fixed an issue preventing the C implementation of
@ref{3c5,,zoneinfo.ZoneInfo} from being subclassed.

@item 
bpo-35018@footnote{https://bugs.python.org/issue35018}: Add the @ref{2e8,,xml.sax.handler.LexicalHandler} class that is
present in other SAX XML implementations.

@item 
bpo-41002@footnote{https://bugs.python.org/issue41002}: Improve performance of HTTPResponse.read with a given amount.
Patch by Bruce Merry.

@item 
bpo-40448@footnote{https://bugs.python.org/issue40448}: @ref{75,,ensurepip} now disables the use of @code{pip} cache when
installing the bundled versions of @code{pip} and @code{setuptools}.  Patch by
Krzysztof Konopko.

@item 
bpo-40967@footnote{https://bugs.python.org/issue40967}: Removed @code{asyncio.Task.current_task()} and
@code{asyncio.Task.all_tasks()}. Patch contributed by Rémi Lapeyre.

@item 
bpo-40924@footnote{https://bugs.python.org/issue40924}: Ensure @code{importlib.resources.path} returns an extant path for
the SourceFileLoader’s resource reader. Avoids the regression identified
in master while a long-term solution is devised.

@item 
bpo-40955@footnote{https://bugs.python.org/issue40955}: Fix a minor memory leak in @ref{f4,,subprocess} module when
extra_groups was specified.

@item 
bpo-40855@footnote{https://bugs.python.org/issue40855}: The standard deviation and variance functions in the statistics
module were ignoring their mu and xbar arguments.

@item 
bpo-40939@footnote{https://bugs.python.org/issue40939}: Use the new PEG parser when generating the stdlib
@ref{a2,,keyword} module.

@item 
bpo-23427@footnote{https://bugs.python.org/issue23427}: Add @ref{2c6,,sys.orig_argv} attribute: the list of the original
command line arguments passed to the Python executable.

@item 
bpo-33689@footnote{https://bugs.python.org/issue33689}: Ignore empty or whitespace-only lines in .pth files. This
matches the documentated behavior. Before, empty lines caused the
site-packages dir to appear multiple times in sys.path. By Ido Michael,
contributors Malcolm Smith and Tal Einat.

@item 
bpo-40884@footnote{https://bugs.python.org/issue40884}: Added a @code{defaults} parameter to @ref{11c6,,logging.Formatter}, to
allow specifying default values for custom fields. Patch by Asaf Alon and
Bar Harel.

@item 
bpo-40876@footnote{https://bugs.python.org/issue40876}: Clarify error message in the @ref{28,,csv} module.

@item 
bpo-39791@footnote{https://bugs.python.org/issue39791}: Refresh importlib.metadata from importlib_metadata 1.6.1.

@item 
bpo-40807@footnote{https://bugs.python.org/issue40807}: Stop codeop._maybe_compile, used by code.InteractiveInterpreter
(and IDLE). from emitting each warning three times.

@item 
bpo-32604@footnote{https://bugs.python.org/issue32604}: Fix reference leak in the @ref{e0,,select} module when the module
is imported in a subinterpreter.

@item 
bpo-39791@footnote{https://bugs.python.org/issue39791}: Built-in loaders (SourceFileLoader and ZipImporter) now supply
@code{TraversableResources} implementations for @code{ResourceReader}, and the
fallback function has been removed.

@item 
bpo-39314@footnote{https://bugs.python.org/issue39314}: @code{rlcompleter.Completer} and the standard Python shell
now close the parenthesis for functions that take no arguments. Patch
contributed by Rémi Lapeyre.

@item 
bpo-17005@footnote{https://bugs.python.org/issue17005}: The topological sort functionality that was introduced
initially in the @ref{7f,,functools} module has been moved to a new
@ref{85,,graphlib} module to better accommodate the new tools and keep the
original scope of the @ref{7f,,functools} module. Patch by Pablo Galindo

@item 
bpo-40834@footnote{https://bugs.python.org/issue40834}: Fix truncate when sending str object
with_xxsubinterpreters.channel_send.

@item 
bpo-40755@footnote{https://bugs.python.org/issue40755}: Add rich comparisons to collections.Counter().

@item 
bpo-26407@footnote{https://bugs.python.org/issue26407}: Unexpected errors in calling the @code{__iter__} method are no
longer masked by @code{TypeError} in @ref{11c7,,csv.reader()},
@code{csv.writer.writerow()} and @code{csv.writer.writerows()}.

@item 
bpo-39384@footnote{https://bugs.python.org/issue39384}: Fixed email.contentmanager to allow set_content() to set a null
string.

@item 
bpo-40744@footnote{https://bugs.python.org/issue40744}: The @ref{ed,,sqlite3} module uses SQLite API functions that
require SQLite v3.7.3 or higher.  This patch removes support for older
SQLite versions, and explicitly requires SQLite 3.7.3 both at build,
compile and runtime.  Patch by Sergey Fedoseev and Erlend E. Aasland.

@item 
bpo-40777@footnote{https://bugs.python.org/issue40777}: Initialize PyDateTime_IsoCalendarDateType.tp_base at run-time
to avoid errors on some compilers.

@item 
bpo-38488@footnote{https://bugs.python.org/issue38488}: Update ensurepip to install pip 20.1.1 and setuptools 47.1.0.

@item 
bpo-40792@footnote{https://bugs.python.org/issue40792}: The result of @ref{1187,,operator.index()} now always has exact type
@ref{1b1,,int}. Previously, the result could have been an instance of a
subclass of @code{int}.

@item 
bpo-40767@footnote{https://bugs.python.org/issue40767}: @ref{130,,webbrowser} now properly finds the default browser in
pure Wayland systems by checking the WAYLAND_DISPLAY environment variable.
Patch contributed by Jérémy Attali.

@item 
bpo-40791@footnote{https://bugs.python.org/issue40791}: @code{hashlib.compare_digest()} uses OpenSSL’s
@code{CRYPTO_memcmp()} function when OpenSSL is available.

@item 
bpo-40795@footnote{https://bugs.python.org/issue40795}: @ref{29,,ctypes} module: If ctypes fails to convert the result of
a callback or if a ctypes callback function raises an exception,
sys.unraisablehook is now called with an exception set. Previously, the
error was logged into stderr by @ref{11c8,,PyErr_Print()}.

@item 
bpo-16995@footnote{https://bugs.python.org/issue16995}: Add @ref{261,,base64.b32hexencode()} and @ref{262,,base64.b32hexdecode()}
to support the Base32 Encoding with Extended Hex Alphabet.

@item 
bpo-30008@footnote{https://bugs.python.org/issue30008}: Fix @ref{ee,,ssl} code to be compatible with OpenSSL 1.1.x builds
that use @code{no-deprecated} and @code{--api=1.1.0}.

@item 
bpo-30064@footnote{https://bugs.python.org/issue30064}: Fix asyncio @code{loop.sock_*} race condition issue

@item 
bpo-40759@footnote{https://bugs.python.org/issue40759}: Deprecate the @code{symbol} module.

@item 
bpo-40756@footnote{https://bugs.python.org/issue40756}: The second argument (extra) of @code{LoggerAdapter.__init__} now
defaults to None.

@item 
bpo-37129@footnote{https://bugs.python.org/issue37129}: Add a new @ref{11c9,,os.RWF_APPEND} flag for @ref{689,,os.pwritev()}.

@item 
bpo-40737@footnote{https://bugs.python.org/issue40737}: Fix possible reference leak for @ref{ed,,sqlite3} initialization.

@item 
bpo-40726@footnote{https://bugs.python.org/issue40726}: Handle cases where the @code{end_lineno} is @code{None} on
@ref{11ca,,ast.increment_lineno()}.

@item 
bpo-40698@footnote{https://bugs.python.org/issue40698}: @ref{37,,distutils} upload creates SHA2-256 and Blake2b-256
digests. MD5 digests is skipped if platform blocks MD5.

@item 
bpo-40695@footnote{https://bugs.python.org/issue40695}: @ref{88,,hashlib} no longer falls back to builtin hash
implementations when OpenSSL provides a hash digest and the algorithm is
blocked by security policy.

@item 
bpo-9216@footnote{https://bugs.python.org/issue9216}: func:@ref{11cb,,hashlib.new} passed @code{usedforsecurity} to OpenSSL EVP
constructor @code{_hashlib.new()}. test_hashlib and test_smtplib handle
strict security policy better.

@item 
bpo-40614@footnote{https://bugs.python.org/issue40614}: @ref{4d3,,ast.parse()} will not parse self documenting expressions
in f-strings when passed @code{feature_version} is less than @code{(3, 8)}.

@item 
bpo-40626@footnote{https://bugs.python.org/issue40626}: Add h5 file extension as MIME Type application/x-hdf5, as per
HDF Group recommendation for HDF5 formatted data files. Patch contributed
by Mark Schwab.

@item 
bpo-25920@footnote{https://bugs.python.org/issue25920}: On macOS, when building Python for macOS 10.4 and older, which
wasn’t the case for python.org macOS installer, @ref{11cc,,socket.getaddrinfo()}
no longer uses an internal lock to prevent race conditions when calling
@code{getaddrinfo()} which is thread-safe since macOS 10.5. Python 3.9
requires macOS 10.6 or newer. The internal lock caused random hang on fork
when another thread was calling @ref{11cc,,socket.getaddrinfo()}. The lock was
also used on FreeBSD older than 5.3, OpenBSD older than 201311 and NetBSD
older than 4.

@item 
bpo-40671@footnote{https://bugs.python.org/issue40671}: Prepare @code{_hashlib} for 
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489} and use
@ref{47c,,PyModule_AddType()}.

@item 
bpo-32309@footnote{https://bugs.python.org/issue32309}: Added a new @ref{3cf,,coroutine} @ref{3d3,,asyncio.to_thread()}. It is
mainly used for running IO-bound functions in a separate thread to avoid
blocking the event loop, and essentially works as a high-level version of
@ref{3d4,,run_in_executor()} that can directly take keyword
arguments.

@item 
bpo-36543@footnote{https://bugs.python.org/issue36543}: Restored the deprecated @code{xml.etree.cElementTree} module.

@item 
bpo-40611@footnote{https://bugs.python.org/issue40611}: @ref{11cd,,MAP_POPULATE} constant has now been added to the
list of exported @ref{af,,mmap} module flags.

@item 
bpo-39881@footnote{https://bugs.python.org/issue39881}: PEP 554 for use in the test suite. (Patch By Joannah Nanjekye)

@item 
bpo-13097@footnote{https://bugs.python.org/issue13097}: @code{ctypes} now raises an @code{ArgumentError} when a callback is
invoked with more than 1024 arguments.

@item 
bpo-39385@footnote{https://bugs.python.org/issue39385}: A new test assertion context-manager,
@code{unittest.assertNoLogs()} will ensure a given block of code emits no
log messages using the logging module. Contributed by Kit Yan Choi.

@item 
bpo-23082@footnote{https://bugs.python.org/issue23082}: Updated the error message and docs of PurePath.relative_to() to
better reflect the function behaviour.

@item 
bpo-40318@footnote{https://bugs.python.org/issue40318}: Use SQLite3 trace v2 API, if it is available.

@item 
bpo-40105@footnote{https://bugs.python.org/issue40105}: ZipFile truncates files to avoid corruption when a shorter
comment is provided in append (“a”) mode. Patch by Jan Mazur.

@item 
bpo-40084@footnote{https://bugs.python.org/issue40084}: Fix @code{Enum.__dir__}: dir(Enum.member) now includes attributes
as well as methods.

@item 
bpo-31122@footnote{https://bugs.python.org/issue31122}: ssl.wrap_socket() now raises ssl.SSLEOFError rather than
OSError when peer closes connection during TLS negotiation

@item 
bpo-39728@footnote{https://bugs.python.org/issue39728}: fix default @code{_missing_} so a duplicate @ref{17c,,ValueError} is not set
as the @code{__context__} of the original @ref{17c,,ValueError}

@item 
bpo-39244@footnote{https://bugs.python.org/issue39244}: Fixed @code{multiprocessing.context.get_all_start_methods} to
properly return the default method first on macOS.

@item 
bpo-39040@footnote{https://bugs.python.org/issue39040}: Fix parsing of invalid mime headers parameters by collapsing
whitespace between encoded words in a bare-quote-string.

@item 
bpo-38731@footnote{https://bugs.python.org/issue38731}: Add @code{--quiet} option to command-line interface of
@ref{d2,,py_compile}. Patch by Gregory Schevchenko.

@item 
bpo-35714@footnote{https://bugs.python.org/issue35714}: @ref{eb6,,struct.error} is now raised if there is a null character
in a @ref{f3,,struct} format string.

@item 
bpo-38144@footnote{https://bugs.python.org/issue38144}: Added the @emph{root_dir} and @emph{dir_fd} parameters in
@ref{288,,glob.glob()}.

@item 
bpo-26543@footnote{https://bugs.python.org/issue26543}: Fix @code{IMAP4.noop()} when debug mode is enabled (ex:
@code{imaplib.Debug = 3}).

@item 
bpo-12178@footnote{https://bugs.python.org/issue12178}: @ref{ff4,,csv.writer()} now correctly escapes @emph{escapechar} when
input contains @emph{escapechar}.  Patch by Catalin Iacob, Berker Peksag, and
Itay Elbirt.

@item 
bpo-36290@footnote{https://bugs.python.org/issue36290}: AST nodes are now raising @ref{162,,TypeError} on conflicting
keyword arguments. Patch contributed by Rémi Lapeyre.

@item 
bpo-33944@footnote{https://bugs.python.org/issue33944}: Added site.py site-packages tracing in verbose mode.

@item 
bpo-35078@footnote{https://bugs.python.org/issue35078}: Refactor formatweekday, formatmonthname methods in
LocaleHTMLCalendar and LocaleTextCalendar classes in calendar module to
call the base class methods.This enables customizable CSS classes for
LocaleHTMLCalendar. Patch by Srinivas Reddy Thatiparthy

@item 
bpo-29620@footnote{https://bugs.python.org/issue29620}: @ref{d01,,assertWarns()} no longer raises a
@code{RuntimeException} when accessing a module’s @code{__warningregistry__}
causes importation of a new module, or when a new module is imported in
another thread. Patch by Kernc.

@item 
bpo-31844@footnote{https://bugs.python.org/issue31844}: Remove @code{ParserBase.error()} method from the private and
undocumented @code{_markupbase} module.  @ref{33c,,html.parser.HTMLParser} is
the only subclass of @code{ParserBase} and its @code{error()} implementation was
deprecated in Python 3.4 and removed in Python 3.5.

@item 
bpo-34226@footnote{https://bugs.python.org/issue34226}: Fix @ref{2e6,,cgi.parse_multipart} without content_length. Patch by
Roger Duran

@item 
bpo-33660@footnote{https://bugs.python.org/issue33660}: Fix pathlib.PosixPath to resolve a relative path located on the
root directory properly.

@item 
bpo-28557@footnote{https://bugs.python.org/issue28557}: Improve the error message for a misbehaving @code{rawio.readinto}

@item 
bpo-26680@footnote{https://bugs.python.org/issue26680}: The d.is_integer() method is added to the Decimal type, for
compatibility with other number types.

@item 
bpo-26680@footnote{https://bugs.python.org/issue26680}: The x.is_integer() method is incorporated into the abstract
types of the numeric tower, Real, Rational and Integral, with appropriate
default implementations.
@end itemize

@node Documentation<15>,Tests<14>,Library<14>,Python 3 10 0 alpha 1
@anchor{whatsnew/changelog id115}@anchor{11ce}
@subsubsection Documentation



@itemize -

@item 
bpo-41428@footnote{https://bugs.python.org/issue41428}: Add documentation for 
@geindex Python Enhancement Proposals; PEP 604
PEP 604@footnote{https://www.python.org/dev/peps/pep-0604} (Allow writing union types as
@code{X | Y}).

@item 
bpo-41774@footnote{https://bugs.python.org/issue41774}: In Programming FAQ “Sequences (Tuples/Lists)” section, add “How
do you remove multiple items from a list”.

@item 
bpo-35293@footnote{https://bugs.python.org/issue35293}: Fix RemovedInSphinx40Warning when building the documentation.
Patch by Dong-hee Na.

@item 
bpo-37149@footnote{https://bugs.python.org/issue37149}: Change Shipman tkinter doc link from archive.org to TkDocs.
(The doc has been removed from the NMT server.)  The new link responds
much faster and includes a short explanatory note.

@item 
bpo-41726@footnote{https://bugs.python.org/issue41726}: Update the refcounts info of @code{PyType_FromModuleAndSpec}.

@item 
bpo-41624@footnote{https://bugs.python.org/issue41624}: Fix the signature of @ref{11cf,,typing.Coroutine}.

@item 
bpo-40204@footnote{https://bugs.python.org/issue40204}: Enable Sphinx 3.2 @code{c_allow_pre_v3} option and disable
@code{c_warn_on_allowed_pre_v3} option to make the documentation compatible
with Sphinx 2 and Sphinx 3.

@item 
bpo-41045@footnote{https://bugs.python.org/issue41045}: Add documentation for debug feature of f-strings.

@item 
bpo-41314@footnote{https://bugs.python.org/issue41314}: Changed the release when @code{from __future__ import annotations}
becomes the default from @code{4.0} to @code{3.10} (following a change in PEP
563).

@item 
bpo-40979@footnote{https://bugs.python.org/issue40979}: Refactored typing.rst, arranging more than 70 classes,
functions, and decorators into new sub-sections.

@item 
bpo-40552@footnote{https://bugs.python.org/issue40552}: Fix in tutorial section 4.2. Code snippet is now correct.

@item 
bpo-39883@footnote{https://bugs.python.org/issue39883}: Make code, examples, and recipes in the Python documentation be
licensed under the more permissive BSD0 license in addition to the
existing Python 2.0 license.

@item 
bpo-37703@footnote{https://bugs.python.org/issue37703}: Updated Documentation to comprehensively elaborate on the
behaviour of gather.cancel()
@end itemize

@node Tests<14>,Build<14>,Documentation<15>,Python 3 10 0 alpha 1
@anchor{whatsnew/changelog id116}@anchor{11d0}
@subsubsection Tests



@itemize -

@item 
bpo-41939@footnote{https://bugs.python.org/issue41939}: Fix test_site.test_license_exists_at_url(): call
@code{urllib.request.urlcleanup()} to reset the global
@code{urllib.request._opener}. Patch by Victor Stinner.

@item 
bpo-41731@footnote{https://bugs.python.org/issue41731}: Make test_cmd_line_script pass with option ‘-vv’.

@item 
bpo-41602@footnote{https://bugs.python.org/issue41602}: Add tests for SIGINT handling in the runpy module.

@item 
bpo-41521@footnote{https://bugs.python.org/issue41521}: @ref{100,,test.support}: Rename @code{blacklist} parameter of
@ref{11d1,,check__all__()} to @code{not_exported}.

@item 
bpo-41477@footnote{https://bugs.python.org/issue41477}: Make ctypes optional in test_genericalias.

@item 
bpo-41085@footnote{https://bugs.python.org/issue41085}: Fix integer overflow in the @ref{25d,,array.array.index()} method on
64-bit Windows for index larger than @code{2**31}.

@item 
bpo-41069@footnote{https://bugs.python.org/issue41069}: @code{test.support.TESTFN} and the current directory for tests
when run via @code{test.regrtest} contain now non-ascii characters if
possible.

@item 
bpo-38377@footnote{https://bugs.python.org/issue38377}: On Linux, skip tests using multiprocessing if the current user
cannot create a file in @code{/dev/shm/} directory. Add the
@ref{11d2,,skip_if_broken_multiprocessing_synchronize()} function
to the @ref{100,,test.support} module.

@item 
bpo-41009@footnote{https://bugs.python.org/issue41009}: Fix use of @code{support.require_@{linux|mac|freebsd@}_version()}
decorators as class decorator.

@item 
bpo-41003@footnote{https://bugs.python.org/issue41003}: Fix @code{test_copyreg} when @code{numpy} is installed:
@code{test.pickletester} now saves/restores warnings filters when importing
@code{numpy}, to ignore filters installed by @code{numpy}.

@item 
bpo-40964@footnote{https://bugs.python.org/issue40964}: Disable remote @ref{93,,imaplib} tests, host cyrus.andrew.cmu.edu
is blocking incoming connections.

@item 
bpo-40927@footnote{https://bugs.python.org/issue40927}: Fix test_binhex when run twice: it now uses
import_fresh_module() to ensure that it raises DeprecationWarning each
time.

@item 
bpo-17258@footnote{https://bugs.python.org/issue17258}: Skip some @ref{b3,,multiprocessing} tests when MD5 hash digest is
blocked.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Increase LOOPBACK_TIMEOUT to 10 for VxWorks RTOS.

@item 
bpo-38169@footnote{https://bugs.python.org/issue38169}: Increase code coverage for SharedMemory and ShareableList

@item 
bpo-34401@footnote{https://bugs.python.org/issue34401}: Make test_gdb properly run on HP-UX. Patch by Michael Osipov.
@end itemize

@node Build<14>,Windows<13>,Tests<14>,Python 3 10 0 alpha 1
@anchor{whatsnew/changelog id117}@anchor{11d3}
@subsubsection Build



@itemize -

@item 
bpo-38249@footnote{https://bugs.python.org/issue38249}: Update @ref{6e8,,Py_UNREACHABLE} to use __builtin_unreachable()
if only the compiler is able to use it. Patch by Dong-hee Na.

@item 
bpo-41617@footnote{https://bugs.python.org/issue41617}: Fix @code{pycore_bitutils.h} header file to support old clang
versions: @code{__builtin_bswap16()} is not available in LLVM clang 3.0.

@item 
bpo-40204@footnote{https://bugs.python.org/issue40204}: Pin Sphinx version to 2.3.1 in @code{Doc/Makefile}.

@item 
bpo-36020@footnote{https://bugs.python.org/issue36020}: The C99 functions @code{snprintf()} and @code{vsnprintf()}
are now required to build Python.

@item 
bpo-40684@footnote{https://bugs.python.org/issue40684}: @code{make install} now uses the @code{PLATLIBDIR} variable for the
destination @code{lib-dynload/} directory when @code{./configure
--with-platlibdir} is used.

@item 
bpo-40683@footnote{https://bugs.python.org/issue40683}: Fixed an issue where the @ref{14c,,zoneinfo} module and its tests
were not included when Python is installed with @code{make}.
@end itemize

@node Windows<13>,macOS<12>,Build<14>,Python 3 10 0 alpha 1
@anchor{whatsnew/changelog id118}@anchor{11d4}
@subsubsection Windows



@itemize -

@item 
bpo-41744@footnote{https://bugs.python.org/issue41744}: Fixes automatic import of props file when using the Nuget
package.

@item 
bpo-41627@footnote{https://bugs.python.org/issue41627}: The user site directory for 32-bit now includes a @code{-32}
suffix to distinguish it from the 64-bit interpreter’s directory.

@item 
bpo-41526@footnote{https://bugs.python.org/issue41526}: Fixed layout of final page of the installer by removing the
special thanks to Mark Hammond (with his permission).

@item 
bpo-41492@footnote{https://bugs.python.org/issue41492}: Fixes the description that appears in UAC prompts.

@item 
bpo-40948@footnote{https://bugs.python.org/issue40948}: Improve post-install message to direct people to the “py”
command.

@item 
bpo-41412@footnote{https://bugs.python.org/issue41412}: The installer will now fail to install on Windows 7 and Windows
8. Further, the UCRT dependency is now always downloaded on demand.

@item 
bpo-40741@footnote{https://bugs.python.org/issue40741}: Update Windows release to include SQLite 3.32.3.

@item 
bpo-41142@footnote{https://bugs.python.org/issue41142}: @ref{b1,,msilib} now supports creating CAB files with non-ASCII
file path and adding files with non-ASCII file path to them.

@item 
bpo-41074@footnote{https://bugs.python.org/issue41074}: Fixed support of non-ASCII names in functions
@ref{11d5,,msilib.OpenDatabase()} and @ref{11d6,,msilib.init_database()} and non-ASCII
SQL in method @ref{11d7,,msilib.Database.OpenView()}.

@item 
bpo-41039@footnote{https://bugs.python.org/issue41039}: Stable ABI redirection DLL (python3.dll) now uses @code{#pragma
comment(linker)} for re-exporting.

@item 
bpo-40164@footnote{https://bugs.python.org/issue40164}: Updates Windows OpenSSL to 1.1.1g

@item 
bpo-39631@footnote{https://bugs.python.org/issue39631}: Changes the registered MIME type for @code{.py} files on Windows
to @code{text/x-python} instead of @code{text/plain}.

@item 
bpo-40677@footnote{https://bugs.python.org/issue40677}: Manually define IO_REPARSE_TAG_APPEXECLINK in case some old
Windows SDK doesn’t have it.

@item 
bpo-37556@footnote{https://bugs.python.org/issue37556}: Extend py.exe help to mention overrides via venv, shebang,
environmental variables & ini files.
@end itemize

@node macOS<12>,IDLE<13>,Windows<13>,Python 3 10 0 alpha 1
@anchor{whatsnew/changelog id119}@anchor{11d8}
@subsubsection macOS



@itemize -

@item 
bpo-41557@footnote{https://bugs.python.org/issue41557}: Update macOS installer to use SQLite 3.33.0.

@item 
bpo-39580@footnote{https://bugs.python.org/issue39580}: Avoid opening Finder window if running installer from the
command line. Patch contributed by Rick Heil.

@item 
bpo-41100@footnote{https://bugs.python.org/issue41100}: Fix configure error when building on macOS 11. Note that the
current Python release was released shortly after the first developer
preview of macOS 11 (Big Sur); there are other known issues with building
and running on the developer preview. Big Sur is expected to be fully
supported in a future bugfix release of Python 3.8.x and with 3.9.0.

@item 
bpo-40741@footnote{https://bugs.python.org/issue40741}: Update macOS installer to use SQLite 3.32.3.

@item 
bpo-41005@footnote{https://bugs.python.org/issue41005}: fixed an XDG settings issue not allowing macos to open browser
in webbrowser.py

@item 
bpo-40741@footnote{https://bugs.python.org/issue40741}: Update macOS installer to use SQLite 3.32.2.
@end itemize

@node IDLE<13>,C API<14>,macOS<12>,Python 3 10 0 alpha 1
@anchor{whatsnew/changelog id120}@anchor{11d9}
@subsubsection IDLE



@itemize -

@item 
bpo-41775@footnote{https://bugs.python.org/issue41775}: Use ‘IDLE Shell’ as shell title

@item 
bpo-35764@footnote{https://bugs.python.org/issue35764}: Rewrite the Calltips doc section.

@item 
bpo-40181@footnote{https://bugs.python.org/issue40181}: In calltips, stop reminding that ‘/’ marks the end of
positional-only arguments.

@item 
bpo-41468@footnote{https://bugs.python.org/issue41468}: Improve IDLE run crash error message (which users should never
see).

@item 
bpo-41373@footnote{https://bugs.python.org/issue41373}: Save files loaded with no line ending, as when blank, or
different line endings, by setting its line ending to the system default.
Fix regression in 3.8.4 and 3.9.0b4.

@item 
bpo-41300@footnote{https://bugs.python.org/issue41300}: Save files with non-ascii chars. Fix regression released in
3.9.0b4 and 3.8.4.

@item 
bpo-37765@footnote{https://bugs.python.org/issue37765}: Add keywords to module name completion list.  Rewrite
Completions section of IDLE doc.

@item 
bpo-41152@footnote{https://bugs.python.org/issue41152}: The encoding of @code{stdin}, @code{stdout} and @code{stderr} in IDLE is
now always UTF-8.

@item 
bpo-41144@footnote{https://bugs.python.org/issue41144}: Make Open Module open a special module such as os.path.

@item 
bpo-39885@footnote{https://bugs.python.org/issue39885}: Make context menu Cut and Copy work again when right-clicking
within a selection.

@item 
bpo-40723@footnote{https://bugs.python.org/issue40723}: Make test_idle pass when run after import.
@end itemize

@node C API<14>,,IDLE<13>,Python 3 10 0 alpha 1
@anchor{whatsnew/changelog id121}@anchor{11da}
@subsubsection C API



@itemize -

@item 
bpo-41936@footnote{https://bugs.python.org/issue41936}: Removed undocumented macros @code{Py_ALLOW_RECURSION} and
@code{Py_END_ALLOW_RECURSION} and the @code{recursion_critical} field of the
@ref{3a0,,PyInterpreterState} structure.

@item 
bpo-41692@footnote{https://bugs.python.org/issue41692}: The @code{PyUnicode_InternImmortal()} function is now deprecated
and will be removed in Python 3.12: use @ref{393,,PyUnicode_InternInPlace()}
instead. Patch by Victor Stinner.

@item 
bpo-41842@footnote{https://bugs.python.org/issue41842}: Add @ref{365,,PyCodec_Unregister()} function to unregister a codec
search function.

@item 
bpo-41834@footnote{https://bugs.python.org/issue41834}: Remove the @code{_Py_CheckRecursionLimit} variable: it has been
replaced by @code{ceval.recursion_limit} of the @ref{3a0,,PyInterpreterState}
structure. Patch by Victor Stinner.

@item 
bpo-41689@footnote{https://bugs.python.org/issue41689}: Types created with @ref{469,,PyType_FromSpec()} now make any
signature in their @code{tp_doc} slot accessible from @code{__text_signature__}.

@item 
bpo-41524@footnote{https://bugs.python.org/issue41524}: Fix bug in PyOS_mystrnicmp and PyOS_mystricmp that incremented
pointers beyond the end of a string.

@item 
bpo-41324@footnote{https://bugs.python.org/issue41324}: Add a minimal decimal capsule API.  The API supports fast
conversions between Decimals up to 38 digits and their triple
representation as a C struct.

@item 
bpo-30155@footnote{https://bugs.python.org/issue30155}: Add @ref{361,,PyDateTime_DATE_GET_TZINFO()} and
@ref{362,,PyDateTime_TIME_GET_TZINFO()} macros for accessing the @code{tzinfo}
attributes of @ref{363,,datetime.datetime} and @ref{364,,datetime.time}
objects.

@item 
bpo-40170@footnote{https://bugs.python.org/issue40170}: Revert @ref{11db,,PyType_HasFeature()} change: it reads again
directly the @ref{1133,,PyTypeObject.tp_flags} member when the limited C
API is not used, rather than always calling @ref{11dc,,PyType_GetFlags()}
which hides implementation details.

@item 
bpo-41123@footnote{https://bugs.python.org/issue41123}: Remove @code{PyUnicode_AsUnicodeCopy}.

@item 
bpo-41123@footnote{https://bugs.python.org/issue41123}: Removed @code{PyLong_FromUnicode()}.

@item 
bpo-41123@footnote{https://bugs.python.org/issue41123}: Removed @code{PyUnicode_GetMax()}.

@item 
bpo-41123@footnote{https://bugs.python.org/issue41123}: Removed @code{Py_UNICODE_str*} functions manipulating
@code{Py_UNICODE*} strings.

@item 
bpo-41103@footnote{https://bugs.python.org/issue41103}: @code{PyObject_AsCharBuffer()}, @code{PyObject_AsReadBuffer()},
@code{PyObject_CheckReadBuffer()}, and @code{PyObject_AsWriteBuffer()} are
removed. Please migrate to new buffer protocol;
@ref{1dd,,PyObject_GetBuffer()} and @ref{1e5,,PyBuffer_Release()}.

@item 
bpo-36346@footnote{https://bugs.python.org/issue36346}: Raises DeprecationWarning for @code{PyUnicode_FromUnicode(NULL,
size)} and @code{PyUnicode_FromStringAndSize(NULL, size)} with @code{size > 0}.

@item 
bpo-36346@footnote{https://bugs.python.org/issue36346}: Mark @code{Py_UNICODE_COPY}, @code{Py_UNICODE_FILL},
@code{PyUnicode_WSTR_LENGTH}, @code{PyUnicode_FromUnicode},
@code{PyUnicode_AsUnicode}, and @code{PyUnicode_AsUnicodeAndSize} as deprecated
in C. Remove @code{Py_UNICODE_MATCH} which was deprecated and broken since
Python 3.3.

@item 
bpo-40989@footnote{https://bugs.python.org/issue40989}: The @code{PyObject_INIT()} and @code{PyObject_INIT_VAR()}
macros become aliases to, respectively, @ref{581,,PyObject_Init()} and
@ref{11dd,,PyObject_InitVar()} functions.

@item 
bpo-36020@footnote{https://bugs.python.org/issue36020}: On Windows, @code{#include "pyerrors.h"} no longer defines
@code{snprintf} and @code{vsnprintf} macros.

@item 
bpo-40943@footnote{https://bugs.python.org/issue40943}: The @code{PY_SSIZE_T_CLEAN} macro must now be defined to use
@ref{37f,,PyArg_ParseTuple()} and @ref{380,,Py_BuildValue()} formats which use
@code{#}: @code{es#}, @code{et#}, @code{s#}, @code{u#}, @code{y#}, @code{z#}, @code{U#} and
@code{Z#}. See @ref{381,,Parsing arguments and building values} and
the 
@geindex Python Enhancement Proposals; PEP 353
PEP 353@footnote{https://www.python.org/dev/peps/pep-0353}.

@item 
bpo-40910@footnote{https://bugs.python.org/issue40910}: Export explicitly the @ref{11de,,Py_GetArgcArgv()} function to the
C API and document the function. Previously, it was exported implicitly
which no longer works since Python is built with @code{-fvisibility=hidden}.

@item 
bpo-40724@footnote{https://bugs.python.org/issue40724}: Allow defining buffer slots in type specs.

@item 
bpo-40679@footnote{https://bugs.python.org/issue40679}: Fix a @code{_PyEval_EvalCode()} crash if @emph{qualname} argument is
NULL.

@item 
bpo-40839@footnote{https://bugs.python.org/issue40839}: Calling @ref{384,,PyDict_GetItem()} without @ref{385,,GIL} held had
been allowed for historical reason. It is no longer allowed.

@item 
bpo-40826@footnote{https://bugs.python.org/issue40826}: @code{PyOS_InterruptOccurred()} now fails with a fatal error
if it is called with the GIL released.

@item 
bpo-40792@footnote{https://bugs.python.org/issue40792}: The result of @ref{35f,,PyNumber_Index()} now always has exact
type @ref{1b1,,int}. Previously, the result could have been an instance of a
subclass of @code{int}.

@item 
bpo-39573@footnote{https://bugs.python.org/issue39573}: Convert @ref{382,,Py_REFCNT()} and @ref{1f3,,Py_SIZE()} macros to
static inline functions. They cannot be used as l-value anymore: use
@ref{383,,Py_SET_REFCNT()} and @ref{1f4,,Py_SET_SIZE()} to set an object
reference count and size. This change is backward incompatible on purpose,
to prepare the C API for an opaque @ref{375,,PyObject} structure.

@item 
bpo-40703@footnote{https://bugs.python.org/issue40703}: The PyType_FromSpec*() functions no longer overwrite the type’s
“__module__” attribute if it is set via “Py_tp_members” or “Py_tp_getset”.

@item 
bpo-39583@footnote{https://bugs.python.org/issue39583}: Remove superfluous “extern C” declarations from
@code{Include/cpython/*.h}.
@end itemize

@node Python 3 9 0 beta 1,Python 3 9 0 alpha 6,Python 3 10 0 alpha 1,Changelog
@anchor{whatsnew/changelog python-3-9-0-beta-1}@anchor{11df}
@subsection Python 3.9.0 beta 1


@emph{Release date: 2020-05-19}

@menu
* Security: Security<9>. 
* Core and Builtins: Core and Builtins<15>. 
* Library: Library<15>. 
* Documentation: Documentation<16>. 
* Tests: Tests<15>. 
* Build: Build<15>. 
* Windows: Windows<14>. 
* macOS: macOS<13>. 
* Tools/Demos: Tools/Demos<4>. 
* C API: C API<15>. 

@end menu

@node Security<9>,Core and Builtins<15>,,Python 3 9 0 beta 1
@anchor{whatsnew/changelog id122}@anchor{11e0}
@subsubsection Security



@itemize -

@item 
bpo-40501@footnote{https://bugs.python.org/issue40501}: @ref{12b,,uuid} no longer uses @ref{29,,ctypes} to load
@code{libuuid} or @code{rpcrt4.dll} at runtime.
@end itemize

@node Core and Builtins<15>,Library<15>,Security<9>,Python 3 9 0 beta 1
@anchor{whatsnew/changelog id123}@anchor{11e1}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-40663@footnote{https://bugs.python.org/issue40663}: Correctly generate annotations where parentheses are omitted
but required (e.g: @code{Type[(str, int, *other))]}.

@item 
bpo-40596@footnote{https://bugs.python.org/issue40596}: Fixed @ref{11e2,,str.isidentifier()} for non-canonicalized strings
containing non-BMP characters on Windows.

@item 
bpo-40593@footnote{https://bugs.python.org/issue40593}: Improved syntax errors for invalid characters in source code.

@item 
bpo-40585@footnote{https://bugs.python.org/issue40585}: Fixed a bug when using @ref{11e3,,codeop.compile_command()} that was
causing exceptions to be swallowed with the new parser. Patch by Pablo
Galindo

@item 
bpo-40566@footnote{https://bugs.python.org/issue40566}: Apply 
@geindex Python Enhancement Proposals; PEP 573
PEP 573@footnote{https://www.python.org/dev/peps/pep-0573} to @ref{3,,abc}.

@item 
bpo-40502@footnote{https://bugs.python.org/issue40502}: Initialize @code{n->n_col_offset}. (Patch by Joannah Nanjekye)

@item 
bpo-40527@footnote{https://bugs.python.org/issue40527}: Fix command line argument parsing: no longer write errors
multiple times into stderr.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @ref{77,,errno} to multiphase initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-40523@footnote{https://bugs.python.org/issue40523}: Add pass-throughs for @ref{c14,,hash()} and @ref{2f6,,reversed()} to
@ref{567,,weakref.proxy} objects. Patch by Pablo Galindo.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @ref{f9,,syslog} to multiphase initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-40246@footnote{https://bugs.python.org/issue40246}: Reporting a specialised error message for invalid string
prefixes, which was introduced in bpo-40246@footnote{https://bugs.python.org/issue40246}, is being reverted due to
backwards compatibility concerns for strings that immediately follow a
reserved keyword without whitespace between them. Constructs like
@code{bg="#d00" if clear else"#fca"} were failing to parse, which is not an
acceptable breakage on such short notice.

@item 
bpo-40417@footnote{https://bugs.python.org/issue40417}: Fix imp module deprecation warning when PyImport_ReloadModule
is called. Patch by Robert Rouhani.

@item 
bpo-40408@footnote{https://bugs.python.org/issue40408}: Fixed support of nested type variables in GenericAlias (e.g.
@code{list[list[T]]}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _stat module to multiphase initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-29587@footnote{https://bugs.python.org/issue29587}: Enable implicit exception chaining when calling
@ref{11be,,generator.throw()}.

@item 
bpo-40328@footnote{https://bugs.python.org/issue40328}: Add tools for generating mappings headers for CJKCodecs.

@item 
bpo-40228@footnote{https://bugs.python.org/issue40228}: Setting frame.f_lineno is now robust w.r.t. changes in the
source-to-bytecode compiler

@item 
bpo-38880@footnote{https://bugs.python.org/issue38880}: Added the ability to list interpreters associated with channel
ends in the internal subinterpreters module.

@item 
bpo-37986@footnote{https://bugs.python.org/issue37986}: Improve performance of @ref{437,,PyLong_FromDouble()} for values
that fit into long.
@end itemize

@node Library<15>,Documentation<16>,Core and Builtins<15>,Python 3 9 0 beta 1
@anchor{whatsnew/changelog id124}@anchor{11e4}
@subsubsection Library



@itemize -

@item 
bpo-40662@footnote{https://bugs.python.org/issue40662}: Fixed @ref{4d2,,ast.get_source_segment()} for ast nodes that have
incomplete location information. Patch by Irit Katriel.

@item 
bpo-40665@footnote{https://bugs.python.org/issue40665}: Convert @ref{10,,bisect} to use Argument Clinic.

@item 
bpo-40536@footnote{https://bugs.python.org/issue40536}: Added the @ref{11e5,,available_timezones()} function to the
@ref{14c,,zoneinfo} module. Patch by Paul Ganssle.

@item 
bpo-40645@footnote{https://bugs.python.org/issue40645}: The @code{hmac.HMAC} exposes internal implementation details.
The attributes @code{digest_cons}, @code{inner}, and @code{outer} are deprecated
and will be removed in the future.

@item 
bpo-40645@footnote{https://bugs.python.org/issue40645}: The internal module @code{_hashlib} wraps and exposes OpenSSL’s
HMAC API. The new code will be used in Python 3.10 after the internal
implementation details of the pure Python HMAC module are no longer part
of the public API.

@item 
bpo-40637@footnote{https://bugs.python.org/issue40637}: Builtin hash modules can now be disabled or selectively enabled
with @code{configure --with-builtin-hashlib-hashes=sha3,blake1} or
@code{--without-builtin-hashlib-hashes}.

@item 
bpo-37630@footnote{https://bugs.python.org/issue37630}: The @ref{88,,hashlib} module can now use SHA3 hashes and SHAKE XOF
from OpenSSL when available.

@item 
bpo-40479@footnote{https://bugs.python.org/issue40479}: The @ref{88,,hashlib} now compiles with OpenSSL 3.0.0-alpha2.

@item 
bpo-40257@footnote{https://bugs.python.org/issue40257}: Revert changes to @ref{500,,inspect.getdoc()}.

@item 
bpo-40607@footnote{https://bugs.python.org/issue40607}: When cancelling a task due to timeout, @ref{3d5,,asyncio.wait_for()}
will now propagate the exception if an error happens during cancellation.
Patch by Roman Skurikhin.

@item 
bpo-40612@footnote{https://bugs.python.org/issue40612}: Fix edge cases in SyntaxError formatting. If the offset is <=
0, no caret is printed. If the offset is > line length, the caret is
printed pointing just after the last character.

@item 
bpo-40597@footnote{https://bugs.python.org/issue40597}: If text content lines are longer than policy.max_line_length,
always use a content-encoding to make sure they are wrapped.

@item 
bpo-40571@footnote{https://bugs.python.org/issue40571}: Added functools.cache() as a simpler, more discoverable way to
access the unbounded cache variant of lru_cache(maxsize=None).

@item 
bpo-40503@footnote{https://bugs.python.org/issue40503}: 
@geindex Python Enhancement Proposals; PEP 615
PEP 615@footnote{https://www.python.org/dev/peps/pep-0615}, the @ref{14c,,zoneinfo} module. Adds support for the
IANA time zone database.

@item 
bpo-40397@footnote{https://bugs.python.org/issue40397}: Removed attributes @code{__args__} and @code{__parameters__} from
special generic aliases like @code{typing.List} (not subscripted).

@item 
bpo-40549@footnote{https://bugs.python.org/issue40549}: Convert posixmodule.c (“posix” or “nt” module) to the
multiphase initialization (PEP 489).

@item 
bpo-31033@footnote{https://bugs.python.org/issue31033}: Add a @code{msg} argument to @code{Future.cancel()} and
@code{Task.cancel()}.

@item 
bpo-40541@footnote{https://bugs.python.org/issue40541}: Added an optional @emph{counts} parameter to random.sample().

@item 
bpo-40515@footnote{https://bugs.python.org/issue40515}: The @ref{ee,,ssl} and @ref{88,,hashlib} modules now actively check
that OpenSSL is build with thread support. Python 3.7.0 made thread
support mandatory and no longer works safely with a no-thread builds.

@item 
bpo-31033@footnote{https://bugs.python.org/issue31033}: When a @ref{317,,asyncio.Task} is cancelled, the exception
traceback now chains all the way back to where the task was first
interrupted.

@item 
bpo-40504@footnote{https://bugs.python.org/issue40504}: @ref{4f3,,functools.lru_cache()} objects can now be the targets of
weakrefs.

@item 
bpo-40559@footnote{https://bugs.python.org/issue40559}: Fix possible memory leak in the C implementation of
@ref{317,,asyncio.Task}.

@item 
bpo-40480@footnote{https://bugs.python.org/issue40480}: @code{fnmatch.fnmatch()} could take exponential time in the
presence of multiple @code{*} pattern characters.  This was repaired by
generating more elaborate regular expressions to avoid futile
backtracking.

@item 
bpo-40495@footnote{https://bugs.python.org/issue40495}: @ref{1f,,compileall} is now able to use hardlinks to prevent
duplicates in a case when @code{.pyc} files for different optimization levels
have the same content.

@item 
bpo-40457@footnote{https://bugs.python.org/issue40457}: The ssl module now support OpenSSL builds without TLS 1.0 and
1.1 methods.

@item 
bpo-40355@footnote{https://bugs.python.org/issue40355}: Improve error reporting in @ref{754,,ast.literal_eval()} in the
presence of malformed @ref{11e6,,ast.Dict} nodes instead of silently ignoring
any non-conforming elements. Patch by Curtis Bucher.

@item 
bpo-40465@footnote{https://bugs.python.org/issue40465}: Deprecated the optional @emph{random} argument to
@emph{random.shuffle()}.

@item 
bpo-40459@footnote{https://bugs.python.org/issue40459}: @ref{11e7,,platform.win32_ver()} now produces correct @emph{ptype} strings
instead of empty strings.

@item 
bpo-39435@footnote{https://bugs.python.org/issue39435}: The first argument of @ref{85e,,pickle.loads()} is now
positional-only.

@item 
bpo-39305@footnote{https://bugs.python.org/issue39305}: Update @ref{bc,,nntplib} to merge @ref{409,,nntplib.NNTP} and
@code{nntplib._NNTPBase}. Patch by Dong-hee Na.

@item 
bpo-32494@footnote{https://bugs.python.org/issue32494}: Update @ref{32,,dbm.gnu} to use gdbm_count if possible when
calling @ref{4a6,,len()}. Patch by Dong-hee Na.

@item 
bpo-40453@footnote{https://bugs.python.org/issue40453}: Add @code{isolated=True} keyword-only parameter to
@code{_xxsubinterpreters.create()}. An isolated subinterpreter cannot spawn
threads, spawn a child process or call @code{os.fork()}.

@item 
bpo-40286@footnote{https://bugs.python.org/issue40286}: Remove @code{_random.Random.randbytes()}: the C implementation of
@code{randbytes()}. Implement the method in Python to ease subclassing:
@code{randbytes()} now directly reuses @code{getrandbits()}.

@item 
bpo-40394@footnote{https://bugs.python.org/issue40394}: Added default arguments to
@ref{11e8,,difflib.SequenceMatcher.find_longest_match()}.

@item 
bpo-39995@footnote{https://bugs.python.org/issue39995}: Fix a race condition in concurrent.futures._ThreadWakeup:
access to _ThreadWakeup is now protected with the shutdown lock.

@item 
bpo-30966@footnote{https://bugs.python.org/issue30966}: @code{Process.shutdown(wait=True)} of @ref{20,,concurrent.futures}
now closes explicitly the result queue.

@item 
bpo-30966@footnote{https://bugs.python.org/issue30966}: Add a new @ref{407,,close()} method to
the @ref{406,,SimpleQueue} class to explicitly close the
queue.

@item 
bpo-39966@footnote{https://bugs.python.org/issue39966}: Revert bpo-25597@footnote{https://bugs.python.org/issue25597}. @ref{a01,,unittest.mock.MagicMock} with wraps’
set uses default return values for magic methods.

@item 
bpo-39791@footnote{https://bugs.python.org/issue39791}: Added @code{files()} function to importlib.resources with support
for subdirectories in package data, matching backport in
importlib_resources 1.5.

@item 
bpo-40375@footnote{https://bugs.python.org/issue40375}: @ref{3f6,,imaplib.IMAP4.unselect()} is added. Patch by Dong-hee Na.

@item 
bpo-40389@footnote{https://bugs.python.org/issue40389}: @code{repr()} now returns @code{typing.Optional[T]} when called for
@code{typing.Union} of two types, one of which is @code{NoneType}.

@item 
bpo-40291@footnote{https://bugs.python.org/issue40291}: Add support for CAN_J1939 sockets (available on Linux 5.4+)

@item 
bpo-40273@footnote{https://bugs.python.org/issue40273}: @ref{246,,types.MappingProxyType} is now reversible.

@item 
bpo-39075@footnote{https://bugs.python.org/issue39075}: The repr for @ref{41b,,types.SimpleNamespace} is now insertion
ordered rather than alphabetical.

@item 
bpo-40192@footnote{https://bugs.python.org/issue40192}: On AIX, @ref{42a,,thread_time()} is now implemented with
@code{thread_cputime()} which has nanosecond resolution, rather than
@code{clock_gettime(CLOCK_THREAD_CPUTIME_ID)} which has a resolution of 10
milliseconds. Patch by Batuhan Taskaya.

@item 
bpo-40025@footnote{https://bugs.python.org/issue40025}: Raise TypeError when _generate_next_value_ is defined after
members. Patch by Ethan Onstott.

@item 
bpo-39058@footnote{https://bugs.python.org/issue39058}: In the argparse module, the repr for Namespace() and other
argument holders now displayed in the order attributes were added.
Formerly, it displayed in alphabetical order even though argument order is
preserved the user visible parts of the module.

@item 
bpo-24416@footnote{https://bugs.python.org/issue24416}: The @code{isocalendar()} methods of @ref{3e2,,datetime.date} and
@ref{363,,datetime.datetime} now return a @ref{ce9,,named tuple} instead of a
@ref{3e5,,tuple}.
@end itemize

@node Documentation<16>,Tests<15>,Library<15>,Python 3 9 0 beta 1
@anchor{whatsnew/changelog id125}@anchor{11e9}
@subsubsection Documentation



@itemize -

@item 
bpo-34790@footnote{https://bugs.python.org/issue34790}: Add version of removal for explicit passing of coros to
@ref{440,,asyncio.wait()}’s documentation

@item 
bpo-40561@footnote{https://bugs.python.org/issue40561}: Provide docstrings for webbrowser open functions.

@item 
bpo-40499@footnote{https://bugs.python.org/issue40499}: Mention that @ref{440,,asyncio.wait()} requires a non-empty set of
awaitables.

@item 
bpo-39705@footnote{https://bugs.python.org/issue39705}: Tutorial example for sorted() in the Loop Techniques section is
given a better explanation. Also a new example is included to explain
sorted()’s basic behavior.

@item 
bpo-39435@footnote{https://bugs.python.org/issue39435}: Fix an incorrect signature for @ref{85e,,pickle.loads()} in the docs
@end itemize

@node Tests<15>,Build<15>,Documentation<16>,Python 3 9 0 beta 1
@anchor{whatsnew/changelog id126}@anchor{11ea}
@subsubsection Tests



@itemize -

@item 
bpo-40055@footnote{https://bugs.python.org/issue40055}: distutils.tests now saves/restores warnings filters to leave
them unchanged. Importing tests imports docutils which imports
pkg_resources which adds a warnings filter.

@item 
bpo-40436@footnote{https://bugs.python.org/issue40436}: test_gdb and test.pythoninfo now check gdb command exit code.
@end itemize

@node Build<15>,Windows<14>,Tests<15>,Python 3 9 0 beta 1
@anchor{whatsnew/changelog id127}@anchor{11eb}
@subsubsection Build



@itemize -

@item 
bpo-40653@footnote{https://bugs.python.org/issue40653}: Move _dirnameW out of HAVE_SYMLINK to fix a potential compiling
issue.

@item 
bpo-40514@footnote{https://bugs.python.org/issue40514}: Add @code{--with-experimental-isolated-subinterpreters} build
option to @code{configure}: better isolate subinterpreters, experimental
build mode.
@end itemize

@node Windows<14>,macOS<13>,Build<15>,Python 3 9 0 beta 1
@anchor{whatsnew/changelog id128}@anchor{11ec}
@subsubsection Windows



@itemize -

@item 
bpo-40650@footnote{https://bugs.python.org/issue40650}: Include winsock2.h in pytime.c for timeval.

@item 
bpo-40458@footnote{https://bugs.python.org/issue40458}: Increase reserved stack space to prevent overflow crash on
Windows.

@item 
bpo-39148@footnote{https://bugs.python.org/issue39148}: Add IPv6 support to @ref{9,,asyncio} datagram endpoints in
ProactorEventLoop. Change the raised exception for unknown address
families to ValueError as it’s not coming from Windows API.
@end itemize

@node macOS<13>,Tools/Demos<4>,Windows<14>,Python 3 9 0 beta 1
@anchor{whatsnew/changelog id129}@anchor{11ed}
@subsubsection macOS



@itemize -

@item 
bpo-34956@footnote{https://bugs.python.org/issue34956}: When building Python on macOS from source, @code{_tkinter} now
links with non-system Tcl and Tk frameworks if they are installed in
@code{/Library/Frameworks}, as had been the case on older releases of macOS.
If a macOS SDK is explicitly configured, by using
@code{--enable-universalsdk=} or @code{-isysroot}, only the SDK itself is
searched. The default behavior can still be overridden with
@code{--with-tcltk-includes} and @code{--with-tcltk-libs}.

@item 
bpo-35569@footnote{https://bugs.python.org/issue35569}: Expose RFC 3542 IPv6 socket options.
@end itemize

@node Tools/Demos<4>,C API<15>,macOS<13>,Python 3 9 0 beta 1
@anchor{whatsnew/changelog id130}@anchor{11ee}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-40479@footnote{https://bugs.python.org/issue40479}: Update multissltest helper to test with latest OpenSSL 1.0.2,
1.1.0, 1.1.1, and 3.0.0-alpha.

@item 
bpo-40431@footnote{https://bugs.python.org/issue40431}: Fix a syntax typo in @code{turtledemo} that now raises a
@code{SyntaxError}.

@item 
bpo-40163@footnote{https://bugs.python.org/issue40163}: Fix multissltest tool. OpenSSL has changed download URL for old
releases. The multissltest tool now tries to download from current and old
download URLs.
@end itemize

@node C API<15>,,Tools/Demos<4>,Python 3 9 0 beta 1
@anchor{whatsnew/changelog id131}@anchor{11ef}
@subsubsection C API



@itemize -

@item 
bpo-39465@footnote{https://bugs.python.org/issue39465}: Remove the @code{_PyUnicode_ClearStaticStrings()} function from
the C API.

@item 
bpo-38787@footnote{https://bugs.python.org/issue38787}: Add PyCFunction_CheckExact() macro for exact type checks now
that we allow subtypes of PyCFunction, as well as PyCMethod_CheckExact()
and PyCMethod_Check() for the new PyCMethod subtype.

@item 
bpo-40545@footnote{https://bugs.python.org/issue40545}: Declare @code{_PyErr_GetTopmostException()} with @code{PyAPI_FUNC()}
to properly export the function in the C API. The function remains private
(@code{_Py}) prefix.

@item 
bpo-40412@footnote{https://bugs.python.org/issue40412}: Nullify inittab_copy during finalization, preventing future
interpreter initializations in an embedded situation from crashing. Patch
by Gregory Szorc.

@item 
bpo-40429@footnote{https://bugs.python.org/issue40429}: The @ref{1fb,,PyThreadState_GetFrame()} function now returns a
strong reference to the frame.

@item 
bpo-40428@footnote{https://bugs.python.org/issue40428}: Remove the following functions from the C API. Call
@ref{49a,,PyGC_Collect()} explicitly to free all free lists.


@itemize *

@item 
@code{PyAsyncGen_ClearFreeLists()}

@item 
@code{PyContext_ClearFreeList()}

@item 
@code{PyDict_ClearFreeList()}

@item 
@code{PyFloat_ClearFreeList()}

@item 
@code{PyFrame_ClearFreeList()}

@item 
@code{PyList_ClearFreeList()}

@item 
@code{PySet_ClearFreeList()}

@item 
@code{PyTuple_ClearFreeList()}
@end itemize

@item 
bpo-40421@footnote{https://bugs.python.org/issue40421}: New @ref{1f7,,PyFrame_GetBack()} function: get the frame next
outer frame.

@item 
bpo-40421@footnote{https://bugs.python.org/issue40421}: New @ref{1f8,,PyFrame_GetCode()} function: return a borrowed
reference to the frame code.

@item 
bpo-40217@footnote{https://bugs.python.org/issue40217}: Ensure that instances of types created with
@ref{36c,,PyType_FromSpecWithBases()} will visit its class object when
traversing references in the garbage collector (implemented as an
extension of the provided @ref{1ee,,tp_traverse}). Patch by
Pablo Galindo.

@item 
bpo-38787@footnote{https://bugs.python.org/issue38787}: Module C state is now accessible from C-defined heap type
methods (
@geindex Python Enhancement Proposals; PEP 573
PEP 573@footnote{https://www.python.org/dev/peps/pep-0573}). Patch by Marcel Plch and Petr Viktorin.
@end itemize

@node Python 3 9 0 alpha 6,Python 3 9 0 alpha 5,Python 3 9 0 beta 1,Changelog
@anchor{whatsnew/changelog python-3-9-0-alpha-6}@anchor{11f0}
@subsection Python 3.9.0 alpha 6


@emph{Release date: 2020-04-27}

@menu
* Security: Security<10>. 
* Core and Builtins: Core and Builtins<16>. 
* Library: Library<16>. 
* Documentation: Documentation<17>. 
* Tests: Tests<16>. 
* Build: Build<16>. 
* Windows: Windows<15>. 
* macOS: macOS<14>. 
* IDLE: IDLE<14>. 
* Tools/Demos: Tools/Demos<5>. 
* C API: C API<16>. 

@end menu

@node Security<10>,Core and Builtins<16>,,Python 3 9 0 alpha 6
@anchor{whatsnew/changelog id132}@anchor{11f1}
@subsubsection Security



@itemize -

@item 
bpo-40121@footnote{https://bugs.python.org/issue40121}: Fixes audit events raised on creating a new socket.

@item 
bpo-39073@footnote{https://bugs.python.org/issue39073}: Disallow CR or LF in email.headerregistry.Address arguments to
guard against header injection attacks.

@item 
bpo-39503@footnote{https://bugs.python.org/issue39503}: CVE-2020-8492: The
@ref{1131,,AbstractBasicAuthHandler} class of the
@ref{127,,urllib.request} module uses an inefficient regular expression which
can be exploited by an attacker to cause a denial of service. Fix the
regex to prevent the catastrophic backtracking. Vulnerability reported by
Ben Caller and Matt Schwager.
@end itemize

@node Core and Builtins<16>,Library<16>,Security<10>,Python 3 9 0 alpha 6
@anchor{whatsnew/changelog id133}@anchor{11f2}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-40313@footnote{https://bugs.python.org/issue40313}: Improve the performance of bytes.hex().

@item 
bpo-40334@footnote{https://bugs.python.org/issue40334}: Switch to a new parser, based on PEG.  For more details see PEP
617. To temporarily switch back to the old parser, use @code{-X oldparser} or
@code{PYTHONOLDPARSER=1}.  In Python 3.10 we will remove the old parser
completely, including the @code{parser} module (already deprecated) and
anything that depends on it.

@item 
bpo-40267@footnote{https://bugs.python.org/issue40267}: Fix the tokenizer to display the correct error message, when
there is a SyntaxError on the last input character and no newline follows.
It used to be @code{unexpected EOF while parsing}, while it should be @code{invalid
syntax}.

@item 
bpo-39522@footnote{https://bugs.python.org/issue39522}: Correctly unparse explicit @code{u} prefix for strings when
postponed evaluation for annotations activated. Patch by Batuhan Taskaya.

@item 
bpo-40246@footnote{https://bugs.python.org/issue40246}: Report a specialized error message, @code{invalid string prefix},
when the tokenizer encounters a string with an invalid prefix.

@item 
bpo-40082@footnote{https://bugs.python.org/issue40082}: Fix the signal handler: it now always uses the main
interpreter, rather than trying to get the current Python thread state.

@item 
bpo-37388@footnote{https://bugs.python.org/issue37388}: str.encode() and str.decode() no longer check the encoding and
errors in development mode or in debug mode during Python finalization.
The codecs machinery can no longer work on very late calls to str.encode()
and str.decode().

@item 
bpo-40077@footnote{https://bugs.python.org/issue40077}: Fix possible refleaks in @code{_json}, memo of PyScannerObject
should be traversed.

@item 
bpo-37207@footnote{https://bugs.python.org/issue37207}: Speed up calls to @code{dict()} by using the 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590}
@code{vectorcall} calling convention.

@item 
bpo-40141@footnote{https://bugs.python.org/issue40141}: Add column and line information to @code{ast.keyword} nodes. Patch
by Pablo Galindo.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @ref{db,,resource} to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port @ref{ad,,math} to multiphase initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _uuid module to multiphase initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-40077@footnote{https://bugs.python.org/issue40077}: Convert json module to use @ref{469,,PyType_FromSpec()}.

@item 
bpo-40067@footnote{https://bugs.python.org/issue40067}: Improve the error message for multiple star expressions in an
assignment. Patch by Furkan Onder

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _functools module to multiphase initialization (PEP
489). Patch by Paulo Henrique Silva.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port operator module to multiphase initialization (PEP 489).
Patch by Paulo Henrique Silva.

@item 
bpo-20526@footnote{https://bugs.python.org/issue20526}: Fix @ref{11f3,,PyThreadState_Clear()}. @code{PyThreadState.frame} is
a borrowed reference, not a strong reference: @code{PyThreadState_Clear()}
must not call @code{Py_CLEAR(tstate->frame)}.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port time module to multiphase initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).
Patch by Paulo Henrique Silva.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _weakref extension module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-40020@footnote{https://bugs.python.org/issue40020}: Fix a leak and subsequent crash in parsetok.c caused by realloc
misuse on a rare codepath.

@item 
bpo-39939@footnote{https://bugs.python.org/issue39939}: Added str.removeprefix and str.removesuffix methods and
corresponding bytes, bytearray, and collections.UserString methods to
remove affixes from a string if present. See 
@geindex Python Enhancement Proposals; PEP 616
PEP 616@footnote{https://www.python.org/dev/peps/pep-0616} for a full
description. Patch by Dennis Sweeney.

@item 
bpo-39481@footnote{https://bugs.python.org/issue39481}: Implement PEP 585. This supports list[int], tuple[str, …]
etc.

@item 
bpo-32894@footnote{https://bugs.python.org/issue32894}: Support unparsing of infinity numbers in postponed annotations.
Patch by Batuhan Taşkaya.

@item 
bpo-37207@footnote{https://bugs.python.org/issue37207}: Speed up calls to @code{list()} by using the 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590}
@code{vectorcall} calling convention. Patch by Mark Shannon.
@end itemize

@node Library<16>,Documentation<17>,Core and Builtins<16>,Python 3 9 0 alpha 6
@anchor{whatsnew/changelog id134}@anchor{11f4}
@subsubsection Library



@itemize -

@item 
bpo-40398@footnote{https://bugs.python.org/issue40398}: @ref{344,,typing.get_args()} now always returns an empty tuple for
special generic aliases.

@item 
bpo-40396@footnote{https://bugs.python.org/issue40396}: Functions @ref{55c,,typing.get_origin()}, @ref{344,,typing.get_args()}
and @ref{432,,typing.get_type_hints()} support now generic aliases like
@code{list[int]}.

@item 
bpo-38061@footnote{https://bugs.python.org/issue38061}: Optimize the @ref{f4,,subprocess} module on FreeBSD using
@code{closefrom()}. A single @code{close(fd)} syscall is cheap, but when
@code{sysconf(_SC_OPEN_MAX)} is high, the loop calling @code{close(fd)} on each
file descriptor can take several milliseconds.

The workaround on FreeBSD to improve performance was to load and mount the
fdescfs kernel module, but this is not enabled by default.

Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans
(kevans) and Kubilay Kocak (koobs):
@indicateurl{https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274}

@item 
bpo-38061@footnote{https://bugs.python.org/issue38061}: On FreeBSD, @code{os.closerange(fd_low, fd_high)} now calls
@code{closefrom(fd_low)} if @emph{fd_high} is greater than or equal to
@code{sysconf(_SC_OPEN_MAX)}.

Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans
(kevans) and Kubilay Kocak (koobs):
@indicateurl{https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274}

@item 
bpo-40360@footnote{https://bugs.python.org/issue40360}: The @ref{a3,,lib2to3} module is pending deprecation due to
@geindex Python Enhancement Proposals; PEP 617
PEP 617@footnote{https://www.python.org/dev/peps/pep-0617}.

@item 
bpo-40138@footnote{https://bugs.python.org/issue40138}: Fix the Windows implementation of @ref{8f6,,os.waitpid()} for exit
code larger than @code{INT_MAX >> 8}. The exit status is now interpreted as
an unsigned number.

@item 
bpo-39942@footnote{https://bugs.python.org/issue39942}: Set “__main__” as the default module name when “__name__” is
missing in @ref{10ba,,typing.TypeVar}. Patch by Weipeng Hong.

@item 
bpo-40275@footnote{https://bugs.python.org/issue40275}: The @ref{a6,,logging} package is now imported lazily in
@ref{122,,unittest} only when the @ref{2e1,,assertLogs()}
assertion is used.

@item 
bpo-40275@footnote{https://bugs.python.org/issue40275}: The @ref{9,,asyncio} package is now imported lazily in
@ref{122,,unittest} only when the @ref{1ac,,IsolatedAsyncioTestCase}
class is used.

@item 
bpo-40330@footnote{https://bugs.python.org/issue40330}: In @code{ShareableList.__setitem__()}, check the size of a new
string item after encoding it to utf-8, not before.

@item 
bpo-40148@footnote{https://bugs.python.org/issue40148}: Added @code{pathlib.Path.with_stem()} to create a new Path
with the stem replaced.

@item 
bpo-40325@footnote{https://bugs.python.org/issue40325}: Deprecated support for set objects in random.sample().

@item 
bpo-40257@footnote{https://bugs.python.org/issue40257}: Improved help for the @ref{120,,typing} module. Docstrings are now
shown for all special forms and special generic aliases (like @code{Union}
and @code{List}). Using @code{help()} with generic alias like @code{List[int]} will
show the help for the correspondent concrete type (@code{list} in this case).

@item 
bpo-40257@footnote{https://bugs.python.org/issue40257}: func:@ref{500,,inspect.getdoc} no longer returns docstring inherited
from the type of the object or from parent class if it is a class if it is
not defined in the object itself. In @ref{d4,,pydoc} the documentation string
is now shown not only for class, function, method etc, but for any object
that has its own @code{__doc__} attribute.

@item 
bpo-40287@footnote{https://bugs.python.org/issue40287}: Fixed @code{SpooledTemporaryFile.seek()} to return the position.

@item 
bpo-40290@footnote{https://bugs.python.org/issue40290}: Added zscore() to statistics.NormalDist().

@item 
bpo-40282@footnote{https://bugs.python.org/issue40282}: Allow @code{random.getrandbits(0)} to succeed and to return 0.

@item 
bpo-40286@footnote{https://bugs.python.org/issue40286}: Add @ref{11f5,,random.randbytes()} function and
@code{random.Random.randbytes()} method to generate random bytes.

@item 
bpo-40277@footnote{https://bugs.python.org/issue40277}: @ref{3e4,,collections.namedtuple()} now provides a human-readable
repr for its field accessors.

@item 
bpo-40270@footnote{https://bugs.python.org/issue40270}: The included copy of sqlite3 on Windows is now compiled with
the json extension. This allows the use of functions such as
@code{json_object}.

@item 
bpo-29255@footnote{https://bugs.python.org/issue29255}: Wait in @code{KqueueSelector.select} when no fds are registered

@item 
bpo-40260@footnote{https://bugs.python.org/issue40260}: Ensure @ref{b0,,modulefinder} uses @ref{11f6,,io.open_code()} and
respects coding comments.

@item 
bpo-40234@footnote{https://bugs.python.org/issue40234}: Allow again to spawn daemon threads in subinterpreters (revert
change which denied them).

@item 
bpo-39207@footnote{https://bugs.python.org/issue39207}: Workers in @ref{3da,,ProcessPoolExecutor} are
now spawned on demand, only when there are no available idle workers to
reuse. This optimizes startup overhead and reduces the amount of lost CPU
time to idle workers. Patch by Kyle Stanley.

@item 
bpo-40091@footnote{https://bugs.python.org/issue40091}: Fix a hang at fork in the logging module: the new private
_at_fork_reinit() method is now used to reinitialize locks at fork in the
child process.

@item 
bpo-40149@footnote{https://bugs.python.org/issue40149}: Implement traverse and clear slots in _abc._abc_data type.

@item 
bpo-40208@footnote{https://bugs.python.org/issue40208}: Remove deprecated @code{symtable.SymbolTable.has_exec()}.

@item 
bpo-40196@footnote{https://bugs.python.org/issue40196}: Fix a bug in the @ref{f6,,symtable} module that was causing
incorrectly report global variables as local. Patch by Pablo Galindo.

@item 
bpo-40190@footnote{https://bugs.python.org/issue40190}: Add support for @code{_SC_AIX_REALMEM} to @code{posix.sysconf()}.

@item 
bpo-40182@footnote{https://bugs.python.org/issue40182}: Removed the @code{_field_types} attribute of the
@ref{457,,typing.NamedTuple} class.

@item 
bpo-36517@footnote{https://bugs.python.org/issue36517}: Multiple inheritance with @ref{457,,typing.NamedTuple} now raises
an error instead of silently ignoring other types.

@item 
bpo-40126@footnote{https://bugs.python.org/issue40126}: Fixed reverting multiple patches in unittest.mock. Patcher’s
@code{__exit__()} is now never called if its @code{__enter__()} is failed.
Returning true from @code{__exit__()} silences now the exception.

@item 
bpo-40094@footnote{https://bugs.python.org/issue40094}: CGIHTTPRequestHandler of http.server now logs the CGI script
exit code, rather than the CGI script exit status of os.waitpid(). For
example, if the script is killed by signal 11, it now logs: “CGI script
exit code -11.”

@item 
bpo-40108@footnote{https://bugs.python.org/issue40108}: Improve the error message when triying to import a module using
@ref{dd,,runpy} and incorrently use the “.py” extension at the end of the
module name. Patch by Pablo Galindo.

@item 
bpo-40094@footnote{https://bugs.python.org/issue40094}: Add @ref{411,,os.waitstatus_to_exitcode()} function: convert a wait
status to an exit code.

@item 
bpo-40089@footnote{https://bugs.python.org/issue40089}: Fix threading._after_fork(): if fork was not called by a thread
spawned by threading.Thread, threading._after_fork() now creates a
_MainThread instance for _main_thread, instead of a _DummyThread instance.

@item 
bpo-40089@footnote{https://bugs.python.org/issue40089}: Add a private @code{_at_fork_reinit()} method to
@code{_thread.Lock}, @code{_thread.RLock}, @ref{e09,,threading.RLock}
and @ref{cf0,,threading.Condition} classes: reinitialize the lock at fork in
the child process, reset the lock to the unlocked state. Rename also the
private @code{_reset_internal_locks()} method of @ref{cf3,,threading.Event} to
@code{_at_fork_reinit()}.

@item 
bpo-25780@footnote{https://bugs.python.org/issue25780}: Expose @ref{425,,CAN_RAW_JOIN_FILTERS} in the
@ref{ea,,socket} module.

@item 
bpo-39503@footnote{https://bugs.python.org/issue39503}: @ref{1131,,AbstractBasicAuthHandler} of
@ref{127,,urllib.request} now parses all WWW-Authenticate HTTP headers and
accepts multiple challenges per header: use the realm of the first Basic
challenge.

@item 
bpo-39812@footnote{https://bugs.python.org/issue39812}: Removed daemon threads from @ref{20,,concurrent.futures} by adding
an internal @code{threading._register_atexit()}, which calls registered
functions prior to joining all non-daemon threads. This allows for
compatibility with subinterpreters, which don’t support daemon threads.

@item 
bpo-40050@footnote{https://bugs.python.org/issue40050}: Fix @code{importlib._bootstrap_external}: avoid creating a new
@code{winreg} builtin module if it’s already available in
@ref{11f7,,sys.modules}, and remove redundant imports.

@item 
bpo-40014@footnote{https://bugs.python.org/issue40014}: Fix @code{os.getgrouplist()}: if @code{getgrouplist()} function fails
because the group list is too small, retry with a larger group list. On
failure, the glibc implementation of @code{getgrouplist()} sets @code{ngroups}
to the total number of groups. For other implementations, double the group
list size.

@item 
bpo-40017@footnote{https://bugs.python.org/issue40017}: Add @ref{11f8,,time.CLOCK_TAI} constant if the operating system
support it.

@item 
bpo-40016@footnote{https://bugs.python.org/issue40016}: In re docstring, clarify the relationship between inline and
argument compile flags.

@item 
bpo-39953@footnote{https://bugs.python.org/issue39953}: Update internal table of OpenSSL error codes in the @code{ssl}
module.

@item 
bpo-36144@footnote{https://bugs.python.org/issue36144}: Added 
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584} operators to
@ref{11f9,,weakref.WeakValueDictionary}.

@item 
bpo-36144@footnote{https://bugs.python.org/issue36144}: Added 
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584} operators to
@ref{11fa,,weakref.WeakKeyDictionary}.

@item 
bpo-38891@footnote{https://bugs.python.org/issue38891}: Fix linear runtime behaviour of the @code{__getitem__} and
@code{__setitem__} methods in
@ref{11fb,,multiprocessing.shared_memory.ShareableList}. This avoids
quadratic performance when iterating a @code{ShareableList}. Patch by Thomas
Krennwallner.

@item 
bpo-39682@footnote{https://bugs.python.org/issue39682}: Remove undocumented support for @emph{closing} a @ref{522,,pathlib.Path}
object via its context manager. The context manager magic methods remain,
but they are now a no-op, making @code{Path} objects immutable.

@item 
bpo-36144@footnote{https://bugs.python.org/issue36144}: Added 
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584} operators (@code{|} and @code{|=}) to
@ref{758,,collections.ChainMap}.

@item 
bpo-39011@footnote{https://bugs.python.org/issue39011}: Normalization of line endings in ElementTree attributes was
removed, as line endings which were replaced by entity numbers should be
preserved in original form.

@item 
bpo-38410@footnote{https://bugs.python.org/issue38410}: Properly handle @ref{11fc,,sys.audit()} failures in
@ref{11fd,,sys.set_asyncgen_hooks()}.

@item 
bpo-36541@footnote{https://bugs.python.org/issue36541}: lib2to3 now recognizes named assignment expressions (the walrus
operator, @code{:=})

@item 
bpo-35967@footnote{https://bugs.python.org/issue35967}: In platform, delay the invocation of ‘uname -p’ until the
processor attribute is requested.

@item 
bpo-35113@footnote{https://bugs.python.org/issue35113}: @ref{1105,,inspect.getsource()} now returns correct source code for
inner class with same name as module level class. Decorators are also
returned as part of source of the class. Patch by Karthikeyan
Singaravelan.

@item 
bpo-33262@footnote{https://bugs.python.org/issue33262}: Deprecate passing None as an argument for
@ref{443,,shlex.split()}’s @code{s} parameter.  Patch by Zackery Spytz.

@item 
bpo-31758@footnote{https://bugs.python.org/issue31758}: Prevent crashes when using an uninitialized
@code{_elementtree.XMLParser} object. Patch by Oren Milman.
@end itemize

@node Documentation<17>,Tests<16>,Library<16>,Python 3 9 0 alpha 6
@anchor{whatsnew/changelog id135}@anchor{11fe}
@subsubsection Documentation



@itemize -

@item 
bpo-27635@footnote{https://bugs.python.org/issue27635}: The pickle documentation incorrectly claimed that @code{__new__}
isn’t called by default when unpickling.

@item 
bpo-39879@footnote{https://bugs.python.org/issue39879}: Updated @ref{11ff,,Data model} docs to include @ref{3b0,,dict()} insertion
order preservation. Patch by Furkan Onder and Samy Lahfa.

@item 
bpo-38387@footnote{https://bugs.python.org/issue38387}: Document @ref{1200,,PyDoc_STRVAR} macro in the C-API reference.

@item 
bpo-13743@footnote{https://bugs.python.org/issue13743}: Some methods within xml.dom.minidom.Element class are now
better documented.
@end itemize

@node Tests<16>,Build<16>,Documentation<17>,Python 3 9 0 alpha 6
@anchor{whatsnew/changelog id136}@anchor{1201}
@subsubsection Tests



@itemize -

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Set expected default encoding in test_c_locale_coercion.py for
VxWorks RTOS.

@item 
bpo-40162@footnote{https://bugs.python.org/issue40162}: Update Travis CI configuration to OpenSSL 1.1.1f.

@item 
bpo-40146@footnote{https://bugs.python.org/issue40146}: Update OpenSSL to 1.1.1f in Azure Pipelines.

@item 
bpo-40094@footnote{https://bugs.python.org/issue40094}: Add @ref{1202,,test.support.wait_process()} function.

@item 
bpo-40003@footnote{https://bugs.python.org/issue40003}: @code{test.bisect_cmd} now copies Python command line options like
@code{-O} or @code{-W}. Moreover, emit a warning if @code{test.bisect_cmd} is used
with @code{-w}/@code{--verbose2} option.

@item 
bpo-39380@footnote{https://bugs.python.org/issue39380}: Add the encoding in @ref{3e9,,ftplib.FTP} and
@ref{3ea,,ftplib.FTP_TLS} to the constructor as keyword-only and change the
default from @code{latin-1} to @code{utf-8} to follow 
@geindex RFC; RFC 2640
RFC 2640@footnote{https://tools.ietf.org/html/rfc2640.html}.

@item 
bpo-39793@footnote{https://bugs.python.org/issue39793}: Use the same domain when testing @code{make_msgid}. Patch by
Batuhan Taskaya.

@item 
bpo-1812@footnote{https://bugs.python.org/issue1812}: Fix newline handling in doctest.testfile when loading from a
package whose loader has a get_data method. Patch by Peter Donis.
@end itemize

@node Build<16>,Windows<15>,Tests<16>,Python 3 9 0 alpha 6
@anchor{whatsnew/changelog id137}@anchor{1203}
@subsubsection Build



@itemize -

@item 
bpo-38360@footnote{https://bugs.python.org/issue38360}: Support single-argument form of macOS -isysroot flag.

@item 
bpo-40158@footnote{https://bugs.python.org/issue40158}: Fix CPython MSBuild Properties in NuGet Package
(build/native/python.props)

@item 
bpo-38527@footnote{https://bugs.python.org/issue38527}: Fix configure check on Solaris for “float word ordering”:
sometimes, the correct “grep” command was not being used. Patch by Arnon
Yaari.
@end itemize

@node Windows<15>,macOS<14>,Build<16>,Python 3 9 0 alpha 6
@anchor{whatsnew/changelog id138}@anchor{1204}
@subsubsection Windows



@itemize -

@item 
bpo-40164@footnote{https://bugs.python.org/issue40164}: Updates Windows to OpenSSL 1.1.1f

@item 
bpo-8901@footnote{https://bugs.python.org/issue8901}: Ignore the Windows registry when the @code{-E} option is used.
@end itemize

@node macOS<14>,IDLE<14>,Windows<15>,Python 3 9 0 alpha 6
@anchor{whatsnew/changelog id139}@anchor{1205}
@subsubsection macOS



@itemize -

@item 
bpo-38329@footnote{https://bugs.python.org/issue38329}: python.org macOS installers now update the Current version
symlink of /Library/Frameworks/Python.framework/Versions for 3.9 installs.
Previously, Current was only updated for Python 2.x installs. This should
make it easier to embed Python 3 into other macOS applications.

@item 
bpo-40164@footnote{https://bugs.python.org/issue40164}: Update macOS installer builds to use OpenSSL 1.1.1g.
@end itemize

@node IDLE<14>,Tools/Demos<5>,macOS<14>,Python 3 9 0 alpha 6
@anchor{whatsnew/changelog id140}@anchor{1206}
@subsubsection IDLE



@itemize -

@item 
bpo-38439@footnote{https://bugs.python.org/issue38439}: Add a 256×256 pixel IDLE icon to support more modern
environments. Created by Andrew Clover. Delete the unused macOS idle.icns
icon file.

@item 
bpo-38689@footnote{https://bugs.python.org/issue38689}: IDLE will no longer freeze when inspect.signature fails when
fetching a calltip.
@end itemize

@node Tools/Demos<5>,C API<16>,IDLE<14>,Python 3 9 0 alpha 6
@anchor{whatsnew/changelog id141}@anchor{1207}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-40385@footnote{https://bugs.python.org/issue40385}: Removed the checkpyc.py tool. Please see compileall without
force mode as a potential alternative.

@item 
bpo-40179@footnote{https://bugs.python.org/issue40179}: Fixed translation of @code{#elif} in Argument Clinic.

@item 
bpo-40094@footnote{https://bugs.python.org/issue40094}: Fix @code{which.py} script exit code: it now uses
@ref{411,,os.waitstatus_to_exitcode()} to convert @ref{fbb,,os.system()} exit status
into an exit code.
@end itemize

@node C API<16>,,Tools/Demos<5>,Python 3 9 0 alpha 6
@anchor{whatsnew/changelog id142}@anchor{1208}
@subsubsection C API



@itemize -

@item 
bpo-40241@footnote{https://bugs.python.org/issue40241}: Move the @code{PyGC_Head} structure to the internal C API.

@item 
bpo-40170@footnote{https://bugs.python.org/issue40170}: Convert @ref{490,,PyObject_IS_GC()} macro to a function to hide
implementation details.

@item 
bpo-40241@footnote{https://bugs.python.org/issue40241}: Add the functions @ref{47d,,PyObject_GC_IsTracked()} and
@ref{47e,,PyObject_GC_IsFinalized()} to the public API to allow to query if
Python objects are being currently tracked or have been already finalized
by the garbage collector respectively. Patch by Pablo Galindo.

@item 
bpo-40170@footnote{https://bugs.python.org/issue40170}: The @code{PyObject_NEW()} macro becomes an alias to the
@ref{491,,PyObject_New()} macro, and the @code{PyObject_NEW_VAR()} macro
becomes an alias to the @ref{492,,PyObject_NewVar()} macro, to hide
implementation details. They no longer access directly the
@ref{493,,PyTypeObject.tp_basicsize} member.

@item 
bpo-40170@footnote{https://bugs.python.org/issue40170}: @ref{11db,,PyType_HasFeature()} now always calls
@ref{11dc,,PyType_GetFlags()} to hide implementation details. Previously, it
accessed directly the @ref{1133,,PyTypeObject.tp_flags} member when the
limited C API was not used.

@item 
bpo-40170@footnote{https://bugs.python.org/issue40170}: Convert the @code{PyObject_GET_WEAKREFS_LISTPTR()} macro to a
function to hide implementation details: the macro accessed directly to
the @ref{494,,PyTypeObject.tp_weaklistoffset} member.

@item 
bpo-40170@footnote{https://bugs.python.org/issue40170}: Convert @ref{1dc,,PyObject_CheckBuffer()} macro to a function to
hide implementation details: the macro accessed directly the
@ref{495,,PyTypeObject.tp_as_buffer} member.

@item 
bpo-40170@footnote{https://bugs.python.org/issue40170}: Always declare @ref{496,,PyIndex_Check()} as an opaque function to
hide implementation details: remove @code{PyIndex_Check()} macro. The macro
accessed directly the @ref{497,,PyTypeObject.tp_as_number} member.

@item 
bpo-39947@footnote{https://bugs.python.org/issue39947}: Add @ref{478,,PyThreadState_GetID()} function: get the unique
identifier of a Python thread state.
@end itemize

@node Python 3 9 0 alpha 5,Python 3 9 0 alpha 4,Python 3 9 0 alpha 6,Changelog
@anchor{whatsnew/changelog python-3-9-0-alpha-5}@anchor{1209}
@subsection Python 3.9.0 alpha 5


@emph{Release date: 2020-03-23}

@menu
* Security: Security<11>. 
* Core and Builtins: Core and Builtins<17>. 
* Library: Library<17>. 
* Documentation: Documentation<18>. 
* Tests: Tests<17>. 
* Build: Build<17>. 
* Windows: Windows<16>. 
* macOS: macOS<15>. 
* IDLE: IDLE<15>. 
* Tools/Demos: Tools/Demos<6>. 
* C API: C API<17>. 

@end menu

@node Security<11>,Core and Builtins<17>,,Python 3 9 0 alpha 5
@anchor{whatsnew/changelog id143}@anchor{120a}
@subsubsection Security



@itemize -

@item 
bpo-38576@footnote{https://bugs.python.org/issue38576}: Disallow control characters in hostnames in http.client,
addressing CVE-2019-18348. Such potentially malicious header injection
URLs now cause a InvalidURL to be raised.
@end itemize

@node Core and Builtins<17>,Library<17>,Security<11>,Python 3 9 0 alpha 5
@anchor{whatsnew/changelog id144}@anchor{120b}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-40010@footnote{https://bugs.python.org/issue40010}: Optimize pending calls in multithreaded applications. If a
thread different than the main thread schedules a pending call
(@ref{488,,Py_AddPendingCall()}), the bytecode evaluation loop is no longer
interrupted at each bytecode instruction to check for pending calls which
cannot be executed. Only the main thread can execute pending calls.

Previously, the bytecode evaluation loop was interrupted at each
instruction until the main thread executes pending calls.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _weakref extension module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _collections module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-40010@footnote{https://bugs.python.org/issue40010}: Optimize signal handling in multithreaded applications. If a
thread different than the main thread gets a signal, the bytecode
evaluation loop is no longer interrupted at each bytecode instruction to
check for pending signals which cannot be handled. Only the main thread of
the main interpreter can handle signals.

Previously, the bytecode evaluation loop was interrupted at each
instruction until the main thread handles signals.

@item 
bpo-39984@footnote{https://bugs.python.org/issue39984}: If @ref{488,,Py_AddPendingCall()} is called in a subinterpreter,
the function is now scheduled to be called from the subinterpreter, rather
than being called from the main interpreter. Each subinterpreter now has
its own list of scheduled calls.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _heapq module to multiphase initialization.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port itertools module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-37207@footnote{https://bugs.python.org/issue37207}: Speed up calls to @code{frozenset()} by using the 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590}
@code{vectorcall} calling convention. Patch by Dong-hee Na.

@item 
bpo-39984@footnote{https://bugs.python.org/issue39984}: subinterpreters: Move
@code{_PyRuntimeState.ceval.tracing_possible} to
@code{PyInterpreterState.ceval.tracing_possible}: each interpreter now has
its own variable.

@item 
bpo-37207@footnote{https://bugs.python.org/issue37207}: Speed up calls to @code{set()} by using the 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590}
@code{vectorcall} calling convention. Patch by Dong-hee Na.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _statistics module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-39968@footnote{https://bugs.python.org/issue39968}: Use inline function to replace extension modules’
get_module_state macros.

@item 
bpo-39965@footnote{https://bugs.python.org/issue39965}: Correctly raise @code{SyntaxError} if @emph{await} is used inside
non-async functions and @code{PyCF_ALLOW_TOP_LEVEL_AWAIT} is set (like in the
asyncio REPL). Patch by Pablo Galindo.

@item 
bpo-39562@footnote{https://bugs.python.org/issue39562}: Allow executing asynchronous comprehensions on the top level
when the @code{PyCF_ALLOW_TOP_LEVEL_AWAIT} flag is given. Patch by Batuhan
Taskaya.

@item 
bpo-37207@footnote{https://bugs.python.org/issue37207}: Speed up calls to @code{tuple()} by using the 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590}
@code{vectorcall} calling convention. Patch by Dong-hee Na.

@item 
bpo-38373@footnote{https://bugs.python.org/issue38373}: Changed list overallocation strategy. It no longer
overallocates if the new size is closer to overallocated size than to the
old size and adds padding.

@item 
bpo-39926@footnote{https://bugs.python.org/issue39926}: Update Unicode database to Unicode version 13.0.0.

@item 
bpo-19466@footnote{https://bugs.python.org/issue19466}: Clear the frames of daemon threads earlier during the Python
shutdown to call objects destructors. So “unclosed file” resource warnings
are now emitted for daemon threads in a more reliable way.

@item 
bpo-38894@footnote{https://bugs.python.org/issue38894}: Fix a bug that was causing incomplete results when calling
@code{pathlib.Path.glob} in the presence of symlinks that point to files
where the user does not have read access. Patch by Pablo Galindo and Matt
Wozniski.

@item 
bpo-39877@footnote{https://bugs.python.org/issue39877}: Fix @ref{5a9,,PyEval_RestoreThread()} random crash at exit with
daemon threads. It now accesses the @code{_PyRuntime} variable directly
instead of using @code{tstate->interp->runtime}, since @code{tstate} can be a
dangling pointer after @ref{eed,,Py_Finalize()} has been called. Moreover,
the daemon thread now exits before trying to take the GIL.

@item 
bpo-39871@footnote{https://bugs.python.org/issue39871}: Fix a possible @ref{891,,SystemError} in
@code{math.@{atan2,copysign,remainder@}()} when the first argument cannot be
converted to a @ref{256,,float}. Patch by Zackery Spytz.

@item 
bpo-39776@footnote{https://bugs.python.org/issue39776}: Fix race condition where threads created by PyGILState_Ensure()
could get a duplicate id.

This affects consumers of tstate->id like the contextvar caching
machinery, which could return invalid cached objects under heavy thread
load (observed in embedded scenarios).

@item 
bpo-39778@footnote{https://bugs.python.org/issue39778}: Fixed a crash due to incorrect handling of weak references in
@code{collections.OrderedDict} classes. Patch by Pablo Galindo.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port audioop extension module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-39702@footnote{https://bugs.python.org/issue39702}: Relax @ref{1bf,,decorator} grammar restrictions to allow any valid
expression (
@geindex Python Enhancement Proposals; PEP 614
PEP 614@footnote{https://www.python.org/dev/peps/pep-0614}).

@item 
bpo-38091@footnote{https://bugs.python.org/issue38091}: Tweak import deadlock detection code to not deadlock itself.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _locale extension module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-39087@footnote{https://bugs.python.org/issue39087}: Optimize @ref{6ef,,PyUnicode_AsUTF8()} and
@ref{367,,PyUnicode_AsUTF8AndSize()} slightly when they need to create
internal UTF-8 cache.

@item 
bpo-39520@footnote{https://bugs.python.org/issue39520}: Fix unparsing of ext slices with no items (@code{foo[:,]}). Patch
by Batuhan Taskaya.

@item 
bpo-39220@footnote{https://bugs.python.org/issue39220}: Do not optimize annotations if ‘from __future__ import
annotations’ is used. Patch by Pablo Galindo.

@item 
bpo-35712@footnote{https://bugs.python.org/issue35712}: Using @ref{24c,,NotImplemented} in a boolean context has been
deprecated. Patch contributed by Josh Rosenberg.

@item 
bpo-22490@footnote{https://bugs.python.org/issue22490}: Don’t leak environment variable @code{__PYVENV_LAUNCHER__} into
the interpreter session on macOS.
@end itemize

@node Library<17>,Documentation<18>,Core and Builtins<17>,Python 3 9 0 alpha 5
@anchor{whatsnew/changelog id145}@anchor{120c}
@subsubsection Library



@itemize -

@item 
bpo-39830@footnote{https://bugs.python.org/issue39830}: Add @ref{1111,,zipfile.Path} to @code{__all__} in the @ref{149,,zipfile}
module.

@item 
bpo-40000@footnote{https://bugs.python.org/issue40000}: Improved error messages for validation of @code{ast.Constant}
nodes. Patch by Batuhan Taskaya.

@item 
bpo-39999@footnote{https://bugs.python.org/issue39999}: @code{__module__} of the AST node classes is now set to “ast”
instead of “_ast”. Added docstrings for dummy AST node classes and
deprecated attributes.

@item 
bpo-39991@footnote{https://bugs.python.org/issue39991}: @ref{6d9,,uuid.getnode()} now skips IPv6 addresses with the same
string length than a MAC address (17 characters): only use MAC addresses.

@item 
bpo-39988@footnote{https://bugs.python.org/issue39988}: Deprecated @code{ast.AugLoad} and @code{ast.AugStore} node classes
because they are no longer used.

@item 
bpo-39656@footnote{https://bugs.python.org/issue39656}: Ensure @code{bin/python3.#} is always present in virtual
environments on POSIX platforms - by Anthony Sottile.

@item 
bpo-39969@footnote{https://bugs.python.org/issue39969}: Deprecated @code{ast.Param} node class because it’s no longer
used. Patch by Batuhan Taskaya.

@item 
bpo-39360@footnote{https://bugs.python.org/issue39360}: Ensure all workers exit when finalizing a
@code{multiprocessing.Pool} implicitly via the module finalization
handlers of multiprocessing. This fixes a deadlock situation that can be
experienced when the Pool is not properly finalized via the context
manager or a call to @code{multiprocessing.Pool.terminate}. Patch by Batuhan
Taskaya and Pablo Galindo.

@item 
bpo-35370@footnote{https://bugs.python.org/issue35370}: sys.settrace(), sys.setprofile() and _lsprof.Profiler.enable()
now properly report @ref{120d,,PySys_Audit()} error if “sys.setprofile” or
“sys.settrace” audit event is denied.

@item 
bpo-39936@footnote{https://bugs.python.org/issue39936}: AIX: Fix _aix_support module when the subprocess is not
available, when building Python from scratch. It now uses new private
_bootsubprocess module, rather than having two implementations depending
if subprocess is available or not. So _aix_support.aix_platform() result
is now the same if subprocess is available or not.

@item 
bpo-36144@footnote{https://bugs.python.org/issue36144}: @ref{4e5,,collections.OrderedDict} now implements @code{|} and
@code{|=} (
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584}).

@item 
bpo-39652@footnote{https://bugs.python.org/issue39652}: The column name found in @code{sqlite3.Cursor.description} is now
truncated on the first ‘[’ only if the PARSE_COLNAMES option is set.

@item 
bpo-39915@footnote{https://bugs.python.org/issue39915}: Ensure @ref{120e,,unittest.mock.AsyncMock.await_args_list} has call
objects in the order of awaited arguments instead of using
@ref{120f,,unittest.mock.Mock.call_args} which has the last value of the call.
Patch by Karthikeyan Singaravelan.

@item 
bpo-36144@footnote{https://bugs.python.org/issue36144}: Updated @ref{d60,,os.environ} and @ref{db8,,os.environb} to support
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584}’s merge (@code{|}) and update (@code{|=}) operators.

@item 
bpo-38662@footnote{https://bugs.python.org/issue38662}: The @code{ensurepip} module now invokes @code{pip} via the @code{runpy}
module. Hence it is no longer tightly coupled with the internal API of the
bundled @code{pip} version, allowing easier updates to a newer @code{pip}
version both internally and for distributors.

@item 
bpo-38075@footnote{https://bugs.python.org/issue38075}: Fix the @code{random.Random.seed()} method when a @ref{2f7,,bool}
is passed as the seed.

@item 
bpo-39916@footnote{https://bugs.python.org/issue39916}: More reliable use of @code{os.scandir()} in @code{Path.glob()}. It no
longer emits a ResourceWarning when interrupted.

@item 
bpo-39850@footnote{https://bugs.python.org/issue39850}: @ref{b3,,multiprocessing} now supports abstract socket addresses
(if abstract sockets are supported in the running platform). When creating
arbitrary addresses (like when default-constructing
@ref{1210,,multiprocessing.connection.Listener} objects) abstract sockets are
preferred to avoid the case when the temporary-file-generated address is
too large for an AF_UNIX socket address. Patch by Pablo Galindo.

@item 
bpo-36287@footnote{https://bugs.python.org/issue36287}: @ref{3cb,,ast.dump()} no longer outputs optional fields and
attributes with default values. The default values for optional fields and
attributes of AST nodes are now set as class attributes (e.g.
@code{Constant.kind} is set to @code{None}).

@item 
bpo-39889@footnote{https://bugs.python.org/issue39889}: Fixed @ref{3cc,,ast.unparse()} for extended slices containing a
single element (e.g. @code{a[i:j,]}). Remove redundant tuples when index with
a tuple (e.g. @code{a[i, j]}).

@item 
bpo-39828@footnote{https://bugs.python.org/issue39828}: Fix @ref{a1,,json.tool} to catch @ref{bf2,,BrokenPipeError}. Patch by
Dong-hee Na.

@item 
bpo-13487@footnote{https://bugs.python.org/issue13487}: Avoid a possible @emph{“RuntimeError: dictionary changed size during
iteration”} from @ref{10e3,,inspect.getmodule()} when it tried to loop through
@ref{11f7,,sys.modules}.

@item 
bpo-39674@footnote{https://bugs.python.org/issue39674}: Revert “bpo-37330@footnote{https://bugs.python.org/issue37330}: open() no longer accept ‘U’ in file mode”.
The “U” mode of open() is kept in Python 3.9 to ease transition from
Python 2.7, but will be removed in Python 3.10.

@item 
bpo-28577@footnote{https://bugs.python.org/issue28577}: The hosts method on 32-bit prefix length IPv4Networks and
128-bit prefix IPv6Networks now returns a list containing the single
Address instead of an empty list.

@item 
bpo-39826@footnote{https://bugs.python.org/issue39826}: Add getConnection method to logging HTTPHandler to enable
custom connections.

@item 
bpo-39763@footnote{https://bugs.python.org/issue39763}: Reimplement @code{distutils.spawn.spawn()} function with the
@ref{f4,,subprocess} module.

@item 
bpo-39794@footnote{https://bugs.python.org/issue39794}: Add –without-decimal-contextvar build option.  This enables a
thread-local rather than a coroutine local context.

@item 
bpo-36144@footnote{https://bugs.python.org/issue36144}: @ref{d63,,collections.defaultdict} now implements @code{|}
(
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584}).

@item 
bpo-39517@footnote{https://bugs.python.org/issue39517}: Fix runpy.run_path() when using pathlike objects

@item 
bpo-39775@footnote{https://bugs.python.org/issue39775}: Change @code{inspect.Signature.parameters} back to
@code{collections.OrderedDict}. This was changed to @code{dict} in Python
3.9.0a4.

@item 
bpo-39678@footnote{https://bugs.python.org/issue39678}: Refactor queue_manager in
@ref{3da,,concurrent.futures.ProcessPoolExecutor} to make it easier to
maintain.

@item 
bpo-39764@footnote{https://bugs.python.org/issue39764}: Fix AttributeError when calling get_stack on a PyAsyncGenObject
Task

@item 
bpo-39769@footnote{https://bugs.python.org/issue39769}: The @ref{645,,compileall.compile_dir()} function’s @emph{ddir} parameter
and the compileall command line flag @code{-d} no longer write the wrong
pathname to the generated pyc file for submodules beneath the root of the
directory tree being compiled.  This fixes a regression introduced with
Python 3.5.

@item 
bpo-36144@footnote{https://bugs.python.org/issue36144}: @ref{246,,types.MappingProxyType} objects now support the merge
(@code{|}) operator from 
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584}.

@item 
bpo-38691@footnote{https://bugs.python.org/issue38691}: The @ref{96,,importlib} module now ignores the
@geindex PYTHONCASEOK
@geindex environment variable; PYTHONCASEOK
@ref{45f,,PYTHONCASEOK} environment variable when the @ref{460,,-E} or
@ref{461,,-I} command line options are being used.

@item 
bpo-39719@footnote{https://bugs.python.org/issue39719}: Remove @code{tempfile.SpooledTemporaryFile.softspace()} as files
no longer have the @code{softspace} attribute in Python 3. Patch by Shantanu.

@item 
bpo-39667@footnote{https://bugs.python.org/issue39667}: Improve pathlib.Path compatibility on zipfile.Path and correct
performance degradation as found in zipp 3.0.

@item 
bpo-39638@footnote{https://bugs.python.org/issue39638}: Keep ASDL signatures in the docstrings for @code{AST} nodes. Patch
by Batuhan Taskaya

@item 
bpo-39639@footnote{https://bugs.python.org/issue39639}: Deprecated @code{ast.Suite} node class because it’s no longer
used. Patch by Batuhan Taskaya.

@item 
bpo-39609@footnote{https://bugs.python.org/issue39609}: Add thread_name_prefix to default asyncio executor

@item 
bpo-39548@footnote{https://bugs.python.org/issue39548}: Fix handling of header in
@ref{1211,,urllib.request.AbstractDigestAuthHandler} when the optional
@code{qop} parameter is not present.

@item 
bpo-39509@footnote{https://bugs.python.org/issue39509}: HTTP status codes @code{103 EARLY_HINTS} and @code{425 TOO_EARLY} are
added to @ref{3ef,,http.HTTPStatus}. Patch by Dong-hee Na.

@item 
bpo-39507@footnote{https://bugs.python.org/issue39507}: Adding HTTP status 418 “I’m a Teapot” to HTTPStatus in http
library. Patch by Ross Rhodes.

@item 
bpo-39495@footnote{https://bugs.python.org/issue39495}: Remove default value from @emph{attrs} parameter of
@ref{1212,,xml.etree.ElementTree.TreeBuilder.start()} for consistency between
Python and C implementations.

@item 
bpo-38971@footnote{https://bugs.python.org/issue38971}: Open issue in the BPO indicated a desire to make the
implementation of codecs.open() at parity with io.open(), which implements
a try/except to assure file stream gets closed before an exception is
raised.

@item 
bpo-38641@footnote{https://bugs.python.org/issue38641}: Added starred expressions support to @code{return} and @code{yield}
statements for @code{lib2to3}. Patch by Vlad Emelianov.

@item 
bpo-37534@footnote{https://bugs.python.org/issue37534}: When using minidom module to generate XML documents the ability
to add Standalone Document Declaration is added. All the changes are made
to generate a document in compliance with Extensible Markup Language (XML)
1.0 (Fifth Edition) W3C Recommendation (available here:
@indicateurl{https://www.w3.org/TR/xml/#sec-prolog-dtd}).

@item 
bpo-34788@footnote{https://bugs.python.org/issue34788}: Add support for scoped IPv6 addresses to @ref{9e,,ipaddress}.
Patch by Oleksandr Pavliuk.

@item 
bpo-34822@footnote{https://bugs.python.org/issue34822}: Simplified AST for subscription. Simple indices are now
represented by their value, extended slices are represented as tuples.
@ref{7,,ast} classes @code{Index} and @code{ExtSlice} are considered deprecated
and will be removed in future Python versions. In the meantime,
@code{Index(value)} now returns a @code{value} itself, @code{ExtSlice(slices)}
returns @code{Tuple(slices, Load())}.
@end itemize

@node Documentation<18>,Tests<17>,Library<17>,Python 3 9 0 alpha 5
@anchor{whatsnew/changelog id146}@anchor{1213}
@subsubsection Documentation



@itemize -

@item 
bpo-39868@footnote{https://bugs.python.org/issue39868}: Updated the Language Reference for 
@geindex Python Enhancement Proposals; PEP 572
PEP 572@footnote{https://www.python.org/dev/peps/pep-0572}.

@item 
bpo-13790@footnote{https://bugs.python.org/issue13790}: Change ‘string’ to ‘specification’ in format doc.

@item 
bpo-17422@footnote{https://bugs.python.org/issue17422}: The language reference no longer restricts default class
namespaces to dicts only.

@item 
bpo-39530@footnote{https://bugs.python.org/issue39530}: Fix misleading documentation about mixed-type numeric
comparisons.

@item 
bpo-39718@footnote{https://bugs.python.org/issue39718}: Update @ref{117,,token} documentation to reflect additions in
Python 3.8

@item 
bpo-39677@footnote{https://bugs.python.org/issue39677}: Changed operand name of @strong{MAKE_FUNCTION} from @emph{argc} to
@emph{flags} for module @ref{36,,dis}
@end itemize

@node Tests<17>,Build<17>,Documentation<18>,Python 3 9 0 alpha 5
@anchor{whatsnew/changelog id147}@anchor{1214}
@subsubsection Tests



@itemize -

@item 
bpo-40019@footnote{https://bugs.python.org/issue40019}: test_gdb now skips tests if it detects that gdb failed to read
debug information because the Python binary is optimized.

@item 
bpo-27807@footnote{https://bugs.python.org/issue27807}: @code{test_site.test_startup_imports()} is now skipped if a path
of @ref{489,,sys.path} contains a @code{.pth} file.

@item 
bpo-26067@footnote{https://bugs.python.org/issue26067}: Do not fail test_shutil test_chown test when uid or gid of user
cannot be resolved to a name.

@item 
bpo-39855@footnote{https://bugs.python.org/issue39855}: test_subprocess.test_user() now skips the test on an user name
if the user name doesn’t exist. For example, skip the test if the user
“nobody” doesn’t exist on Linux.
@end itemize

@node Build<17>,Windows<16>,Tests<17>,Python 3 9 0 alpha 5
@anchor{whatsnew/changelog id148}@anchor{1215}
@subsubsection Build



@itemize -

@item 
bpo-39761@footnote{https://bugs.python.org/issue39761}: Fix build with DTrace but without additional DFLAGS.

@item 
bpo-39763@footnote{https://bugs.python.org/issue39763}: setup.py now uses a basic implementation of the
@ref{f4,,subprocess} module if the @ref{f4,,subprocess} module is not available:
before required C extension modules are built.

@item 
bpo-1294959@footnote{https://bugs.python.org/issue1294959}: Add @code{--with-platlibdir} option to the configure script:
name of the platform-specific library directory, stored in the new
@ref{42c,,sys.platlibdir} attribute. It is used to build the path of
platform-specific extension modules and the path of the standard library.
It is equal to @code{"lib"} on most platforms. On Fedora and SuSE, it is
equal to @code{"lib64"} on 64-bit platforms. Patch by Jan Matějek, Matěj
Cepl, Charalampos Stratakis and Victor Stinner.
@end itemize

@node Windows<16>,macOS<15>,Build<17>,Python 3 9 0 alpha 5
@anchor{whatsnew/changelog id149}@anchor{1216}
@subsubsection Windows



@itemize -

@item 
bpo-39930@footnote{https://bugs.python.org/issue39930}: Ensures the required @code{vcruntime140.dll} is included in
install packages.

@item 
bpo-39847@footnote{https://bugs.python.org/issue39847}: Avoid hang when computer is hibernated whilst waiting for a
mutex (for lock-related objects from @ref{109,,threading}) around 49-day
uptime.

@item 
bpo-38597@footnote{https://bugs.python.org/issue38597}: @ref{37,,distutils} will no longer statically link
@code{vcruntime140.dll} when a redistributable version is unavailable.
All future releases of CPython will include a copy of this DLL to ensure
distributed extensions can continue to load.

@item 
bpo-38380@footnote{https://bugs.python.org/issue38380}: Update Windows builds to use SQLite 3.31.1

@item 
bpo-39789@footnote{https://bugs.python.org/issue39789}: Update Windows release build machines to Visual Studio 2019
(MSVC 14.2).

@item 
bpo-34803@footnote{https://bugs.python.org/issue34803}: Package for nuget.org now includes repository reference and
bundled icon image.
@end itemize

@node macOS<15>,IDLE<15>,Windows<16>,Python 3 9 0 alpha 5
@anchor{whatsnew/changelog id150}@anchor{1217}
@subsubsection macOS



@itemize -

@item 
bpo-38380@footnote{https://bugs.python.org/issue38380}: Update macOS builds to use SQLite 3.31.1
@end itemize

@node IDLE<15>,Tools/Demos<6>,macOS<15>,Python 3 9 0 alpha 5
@anchor{whatsnew/changelog id151}@anchor{1218}
@subsubsection IDLE



@itemize -

@item 
bpo-27115@footnote{https://bugs.python.org/issue27115}: For ‘Go to Line’, use a Query box subclass with IDLE standard
behavior and improved error checking.

@item 
bpo-39885@footnote{https://bugs.python.org/issue39885}: Since clicking to get an IDLE context menu moves the cursor,
any text selection should be and now is cleared.

@item 
bpo-39852@footnote{https://bugs.python.org/issue39852}: Edit “Go to line” now clears any selection, preventing
accidental deletion. It also updates Ln and Col on the status bar.

@item 
bpo-39781@footnote{https://bugs.python.org/issue39781}: Selecting code context lines no longer causes a jump.
@end itemize

@node Tools/Demos<6>,C API<17>,IDLE<15>,Python 3 9 0 alpha 5
@anchor{whatsnew/changelog id152}@anchor{1219}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-36184@footnote{https://bugs.python.org/issue36184}: Port python-gdb.py to FreeBSD. python-gdb.py now checks for
“take_gil” function name to check if a frame tries to acquire the GIL,
instead of checking for “pthread_cond_timedwait” which is specific to
Linux and can be a different condition than the GIL.

@item 
bpo-38080@footnote{https://bugs.python.org/issue38080}: Added support to fix @code{getproxies} in the
@code{lib2to3.fixes.fix_urllib} module. Patch by José Roberto Meza
Cabrera.
@end itemize

@node C API<17>,,Tools/Demos<6>,Python 3 9 0 alpha 5
@anchor{whatsnew/changelog id153}@anchor{121a}
@subsubsection C API



@itemize -

@item 
bpo-40024@footnote{https://bugs.python.org/issue40024}: Add @ref{47c,,PyModule_AddType()} helper function: add a type to a
module. Patch by Dong-hee Na.

@item 
bpo-39946@footnote{https://bugs.python.org/issue39946}: Remove @code{_PyRuntime.getframe} hook and remove
@code{_PyThreadState_GetFrame} macro which was an alias to
@code{_PyRuntime.getframe}. They were only exposed by the internal C API.
Remove also @code{PyThreadFrameGetter} type.

@item 
bpo-39947@footnote{https://bugs.python.org/issue39947}: Add @ref{1fb,,PyThreadState_GetFrame()} function: get the current
frame of a Python thread state.

@item 
bpo-37207@footnote{https://bugs.python.org/issue37207}: Add _PyArg_NoKwnames helper function. Patch by Dong-hee Na.

@item 
bpo-39947@footnote{https://bugs.python.org/issue39947}: Add @ref{476,,PyThreadState_GetInterpreter()}: get the interpreter
of a Python thread state.

@item 
bpo-39947@footnote{https://bugs.python.org/issue39947}: Add @ref{477,,PyInterpreterState_Get()} function to the limited C
API.

@item 
bpo-35370@footnote{https://bugs.python.org/issue35370}: If @ref{120d,,PySys_Audit()} fails in @ref{1031,,PyEval_SetProfile()}
or @ref{1032,,PyEval_SetTrace()}, log the error as an unraisable exception.

@item 
bpo-39947@footnote{https://bugs.python.org/issue39947}: Move the static inline function flavor of
Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() to the internal C API:
they access PyThreadState attributes. The limited C API provides regular
functions which hide implementation details.

@item 
bpo-39947@footnote{https://bugs.python.org/issue39947}: Py_TRASHCAN_BEGIN_CONDITION and Py_TRASHCAN_END macro no longer
access PyThreadState attributes, but call new private _PyTrash_begin() and
_PyTrash_end() functions which hide implementation details.

@item 
bpo-39884@footnote{https://bugs.python.org/issue39884}: @ref{121b,,PyDescr_NewMethod()} and @code{PyCFunction_NewEx()} now
include the method name in the SystemError “bad call flags” error message
to ease debug.

@item 
bpo-39877@footnote{https://bugs.python.org/issue39877}: Deprecated @ref{441,,PyEval_InitThreads()} and
@ref{442,,PyEval_ThreadsInitialized()}. Calling @ref{441,,PyEval_InitThreads()}
now does nothing.

@item 
bpo-38249@footnote{https://bugs.python.org/issue38249}: @ref{6e8,,Py_UNREACHABLE} is now implemented with
@code{__builtin_unreachable()} and analogs in release mode.

@item 
bpo-38643@footnote{https://bugs.python.org/issue38643}: @ref{121c,,PyNumber_ToBase()} now raises a @ref{891,,SystemError}
instead of crashing when called with invalid base.

@item 
bpo-39882@footnote{https://bugs.python.org/issue39882}: The @ref{48f,,Py_FatalError()} function is replaced with a macro
which logs automatically the name of the current function, unless the
@code{Py_LIMITED_API} macro is defined.

@item 
bpo-39824@footnote{https://bugs.python.org/issue39824}: Extension modules: @ref{481,,m_traverse},
@ref{482,,m_clear} and @ref{483,,m_free}
functions of @ref{484,,PyModuleDef} are no longer called if the module
state was requested but is not allocated yet. This is the case immediately
after the module is created and before the module is executed
(@ref{485,,Py_mod_exec} function). More precisely, these functions are not
called if @ref{486,,m_size} is greater than 0 and the module
state (as returned by @ref{487,,PyModule_GetState()}) is @code{NULL}.

Extension modules without module state (@code{m_size <= 0}) are not affected.

@item 
bpo-38913@footnote{https://bugs.python.org/issue38913}: Fixed segfault in @code{Py_BuildValue()} called with a format
containing “#” and undefined PY_SSIZE_T_CLEAN whwn an exception is set.

@item 
bpo-38500@footnote{https://bugs.python.org/issue38500}: Add a private API to get and set the frame evaluation function:
add @ref{121d,,_PyInterpreterState_GetEvalFrameFunc()} and
@ref{121e,,_PyInterpreterState_SetEvalFrameFunc()} C functions. The
@ref{121f,,_PyFrameEvalFunction} function type now takes a @emph{tstate}
parameter.
@end itemize

@node Python 3 9 0 alpha 4,Python 3 9 0 alpha 3,Python 3 9 0 alpha 5,Changelog
@anchor{whatsnew/changelog python-3-9-0-alpha-4}@anchor{1220}
@subsection Python 3.9.0 alpha 4


@emph{Release date: 2020-02-25}

@menu
* Security: Security<12>. 
* Core and Builtins: Core and Builtins<18>. 
* Library: Library<18>. 
* Documentation: Documentation<19>. 
* Tests: Tests<18>. 
* Build: Build<18>. 
* Windows: Windows<17>. 
* IDLE: IDLE<16>. 
* C API: C API<18>. 

@end menu

@node Security<12>,Core and Builtins<18>,,Python 3 9 0 alpha 4
@anchor{whatsnew/changelog id154}@anchor{1221}
@subsubsection Security



@itemize -

@item 
bpo-39184@footnote{https://bugs.python.org/issue39184}: Add audit events to functions in @ref{79,,fcntl}, @ref{b2,,msvcrt}, @ref{c0,,os},
@ref{db,,resource}, @ref{e4,,shutil}, @ref{e5,,signal} and @ref{f9,,syslog}.

@item 
bpo-39401@footnote{https://bugs.python.org/issue39401}: Avoid unsafe DLL load at startup on Windows 7 and earlier.

@item 
bpo-39184@footnote{https://bugs.python.org/issue39184}: Add audit events to command execution functions in os and pty
modules.
@end itemize

@node Core and Builtins<18>,Library<18>,Security<12>,Python 3 9 0 alpha 4
@anchor{whatsnew/changelog id155}@anchor{1222}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-39382@footnote{https://bugs.python.org/issue39382}: Fix a use-after-free in the single inheritance path of
@code{issubclass()}, when the @code{__bases__} of an object has a single
reference, and so does its first item. Patch by Yonatan Goldschmidt.

@item 
bpo-39573@footnote{https://bugs.python.org/issue39573}: Update clinic tool to use @ref{112d,,Py_IS_TYPE()}. Patch by
Dong-hee Na.

@item 
bpo-39619@footnote{https://bugs.python.org/issue39619}: Enable use of @ref{1223,,os.chroot()} on HP-UX systems.

@item 
bpo-39573@footnote{https://bugs.python.org/issue39573}: Add @ref{112d,,Py_IS_TYPE()} static inline function to check
whether the object @emph{o} type is @emph{type}.

@item 
bpo-39606@footnote{https://bugs.python.org/issue39606}: Fix regression caused by fix for bpo-39386@footnote{https://bugs.python.org/issue39386}, that prevented
calling @code{aclose} on an async generator that had already been closed or
exhausted.

@item 
bpo-39579@footnote{https://bugs.python.org/issue39579}: Change the ending column offset of @code{Attribute} nodes
constructed in @code{ast_for_dotted_name} to point at the end of the current
node and not at the end of the last @code{NAME} node.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _crypt extension module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _contextvars extension module to multiphase
initialization (
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-39510@footnote{https://bugs.python.org/issue39510}: Fix segfault in @code{readinto()} method on closed BufferedReader.

@item 
bpo-39502@footnote{https://bugs.python.org/issue39502}: Fix @ref{1224,,time.localtime()} on 64-bit AIX  to support years
before 1902 and after 2038. Patch by M Felt.

@item 
bpo-39492@footnote{https://bugs.python.org/issue39492}: Fix a reference cycle in the C Pickler that was preventing the
garbage collection of deleted, pickled objects.

@item 
bpo-39453@footnote{https://bugs.python.org/issue39453}: Fixed a possible crash in @code{list.__contains__()} when a list
is changed during comparing items. Patch by Dong-hee Na.

@item 
bpo-39434@footnote{https://bugs.python.org/issue39434}: @ref{43b,,floor division} of float operation now has a better
performance. Also the message of @ref{43c,,ZeroDivisionError} for this
operation is updated. Patch by Dong-hee Na.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _codecs extension module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _bz2 extension module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _abc extension module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-39320@footnote{https://bugs.python.org/issue39320}: Replace two complex bytecodes for building dicts with two
simpler ones. The new bytecodes @code{DICT_MERGE} and @code{DICT_UPDATE} have
been added The old bytecodes @code{BUILD_MAP_UNPACK} and
@code{BUILD_MAP_UNPACK_WITH_CALL} have been removed.

@item 
bpo-39219@footnote{https://bugs.python.org/issue39219}: Syntax errors raised in the tokenizer now always set correct
“text” and “offset” attributes.

@item 
bpo-36051@footnote{https://bugs.python.org/issue36051}: Drop the GIL during large @code{bytes.join} operations. Patch by
Bruce Merry.

@item 
bpo-38960@footnote{https://bugs.python.org/issue38960}: Fix DTrace build issues on FreeBSD. Patch by David Carlier.

@item 
bpo-37207@footnote{https://bugs.python.org/issue37207}: Speed up calls to @code{range()} by about 30%, by using the PEP
590 @code{vectorcall} calling convention. Patch by Mark Shannon.

@item 
bpo-36144@footnote{https://bugs.python.org/issue36144}: @ref{3b0,,dict} (and @ref{1225,,collections.UserDict}) objects now
support PEP 584’s merge (@code{|}) and update (@code{|=}) operators. Patch by
Brandt Bucher.

@item 
bpo-32856@footnote{https://bugs.python.org/issue32856}: Optimized the idiom for assignment a temporary variable in
comprehensions. Now @code{for y in [expr]} in comprehensions is as fast as a
simple assignment @code{y = expr}.
@end itemize

@node Library<18>,Documentation<19>,Core and Builtins<18>,Python 3 9 0 alpha 4
@anchor{whatsnew/changelog id156}@anchor{1226}
@subsubsection Library



@itemize -

@item 
bpo-30566@footnote{https://bugs.python.org/issue30566}: Fix @ref{1066,,IndexError} when trying to decode an invalid string
with punycode codec.

@item 
bpo-39649@footnote{https://bugs.python.org/issue39649}: Remove obsolete check for @code{__args__} in
bdb.Bdb.format_stack_entry.

@item 
bpo-39648@footnote{https://bugs.python.org/issue39648}: Expanded @ref{401,,math.gcd()} and @ref{402,,math.lcm()} to handle
multiple arguments.

@item 
bpo-39681@footnote{https://bugs.python.org/issue39681}: Fix a regression where the C pickle module wouldn’t allow
unpickling from a file-like object that doesn’t expose a readinto()
method.

@item 
bpo-35950@footnote{https://bugs.python.org/issue35950}: Raise @ref{1227,,io.UnsupportedOperation} in
@code{io.BufferedReader.truncate()} when it is called on a read-only
@ref{daf,,io.BufferedReader} instance.

@item 
bpo-39479@footnote{https://bugs.python.org/issue39479}: Add @ref{402,,math.lcm()} function: least common multiple.

@item 
bpo-39674@footnote{https://bugs.python.org/issue39674}: Revert “Do not expose abstract collection classes in the
collections module” change (bpo-25988@footnote{https://bugs.python.org/issue25988}). Aliases to ABC like
collections.Mapping are kept in Python 3.9 to ease transition from Python
2.7, but will be removed in Python 3.10.

@item 
bpo-39104@footnote{https://bugs.python.org/issue39104}: Fix hanging ProcessPoolExcutor on @code{shutdown(wait=False)} when
a task has failed pickling.

@item 
bpo-39627@footnote{https://bugs.python.org/issue39627}: Fixed TypedDict totality check for inherited keys.

@item 
bpo-39474@footnote{https://bugs.python.org/issue39474}: Fixed starting position of AST for expressions like @code{(a)(b)},
@code{(a)[b]} and @code{(a).b}.

@item 
bpo-21016@footnote{https://bugs.python.org/issue21016}: The @ref{d4,,pydoc} and @ref{119,,trace} modules now use the
@ref{f8,,sysconfig} module to get the path to the Python standard library, to
support uncommon installation path like @code{/usr/lib64/python3.9/} on
Fedora. Patch by Jan Matějek.

@item 
bpo-39590@footnote{https://bugs.python.org/issue39590}: Collections.deque now holds strong references during
deque.__contains__ and deque.count, fixing crashes.

@item 
bpo-39586@footnote{https://bugs.python.org/issue39586}: The distutils @code{bdist_msi} command is deprecated in Python
3.9, use @code{bdist_wheel} (wheel packages) instead.

@item 
bpo-39595@footnote{https://bugs.python.org/issue39595}: Improved performance of zipfile.Path for files with a large
number of entries. Also improved performance and fixed minor issue as
published with importlib_metadata 1.5@footnote{https://importlib-metadata.readthedocs.io/en/latest/changelog%20(links).html#v1-5-0}.

@item 
bpo-39350@footnote{https://bugs.python.org/issue39350}: Fix regression in @ref{172,,fractions.Fraction} if the numerator
and/or the denominator is an @ref{1b1,,int} subclass. The @ref{401,,math.gcd()}
function is now used to normalize the @emph{numerator} and @emph{denominator}.
@ref{401,,math.gcd()} always return a @ref{1b1,,int} type. Previously, the GCD
type depended on @emph{numerator} and @emph{denominator}.

@item 
bpo-39567@footnote{https://bugs.python.org/issue39567}: Added audit for @ref{8de,,os.walk()}, @ref{683,,os.fwalk()},
@ref{1228,,pathlib.Path.glob()} and @ref{1229,,pathlib.Path.rglob()}.

@item 
bpo-39559@footnote{https://bugs.python.org/issue39559}: Remove unused, undocumented argument @code{getters} from
@ref{6d9,,uuid.getnode()}

@item 
bpo-38149@footnote{https://bugs.python.org/issue38149}: @ref{11fc,,sys.audit()} is now called only once per call of
@ref{288,,glob.glob()} and @ref{289,,glob.iglob()}.

@item 
bpo-39546@footnote{https://bugs.python.org/issue39546}: Fix a regression in @ref{91c,,ArgumentParser} where
@code{allow_abbrev=False} was ignored for long options that used a prefix
character other than “-“.

@item 
bpo-39450@footnote{https://bugs.python.org/issue39450}: Striped whitespace from docstring before returning it from
@code{unittest.case.shortDescription()}.

@item 
bpo-12915@footnote{https://bugs.python.org/issue12915}: A new function @code{resolve_name} has been added to the
@code{pkgutil} module. This resolves a string of the form @code{'a.b.c.d'} or
@code{'a.b:c.d'} to an object. In the example, @code{a.b} is a package/module
and @code{c.d} is an object within that package/module reached via recursive
attribute access.

@item 
bpo-39353@footnote{https://bugs.python.org/issue39353}: The @ref{1b8,,binascii.crc_hqx()} function is no longer deprecated.

@item 
bpo-39493@footnote{https://bugs.python.org/issue39493}: Mark @code{typing.IO.closed} as a property

@item 
bpo-39491@footnote{https://bugs.python.org/issue39491}: Add @ref{431,,typing.Annotated} and @code{include_extras} parameter
to @ref{432,,typing.get_type_hints()} as part of 
@geindex Python Enhancement Proposals; PEP 593
PEP 593@footnote{https://www.python.org/dev/peps/pep-0593}. Patch by Till
Varoquaux, documentation by Till Varoquaux and Konstantin Kashin.

@item 
bpo-39485@footnote{https://bugs.python.org/issue39485}: Fix a bug in @ref{122a,,unittest.mock.create_autospec()} that would
complain about the wrong number of arguments for custom descriptors
defined in an extension module returning functions.

@item 
bpo-38932@footnote{https://bugs.python.org/issue38932}: Mock fully resets child objects on reset_mock(). Patch by
Vegard Stikbakke

@item 
bpo-39082@footnote{https://bugs.python.org/issue39082}: Allow AsyncMock to correctly patch static/class methods

@item 
bpo-39432@footnote{https://bugs.python.org/issue39432}: Implement PEP-489 algorithm for non-ascii “PyInit_…” symbol
names in distutils to make it export the correct init symbol also on
Windows.

@item 
bpo-18819@footnote{https://bugs.python.org/issue18819}: Omit @code{devmajor} and @code{devminor} fields for non-device files
in @ref{fb,,tarfile} archives, enabling bit-for-bit compatibility with GNU
@code{tar(1)}.

@item 
bpo-39349@footnote{https://bugs.python.org/issue39349}: Added a new @emph{cancel_futures} parameter to
@ref{3d9,,concurrent.futures.Executor.shutdown()} that cancels all pending
futures which have not started running, instead of waiting for them to
complete before shutting down the executor.

@item 
bpo-39274@footnote{https://bugs.python.org/issue39274}: @code{bool(fraction.Fraction)} now returns a boolean even if
(numerator != 0) does not return a boolean (ex: numpy number).

@item 
bpo-34793@footnote{https://bugs.python.org/issue34793}: Remove support for @code{with (await asyncio.lock):} and @code{with
(yield from asyncio.lock):}.  The same is correct for
@code{asyncio.Condition} and @code{asyncio.Semaphore}.

@item 
bpo-25597@footnote{https://bugs.python.org/issue25597}: Ensure, if @code{wraps} is supplied to
@ref{a01,,unittest.mock.MagicMock}, it is used to calculate return values
for the magic methods instead of using the default return values. Patch by
Karthikeyan Singaravelan.

@item 
bpo-36350@footnote{https://bugs.python.org/issue36350}: @ref{122b,,inspect.Signature.parameters} and
@ref{3fc,,inspect.BoundArguments.arguments} are now dicts instead of OrderedDicts.
Patch contributed by Rémi Lapeyre.

@item 
bpo-35727@footnote{https://bugs.python.org/issue35727}: Fix sys.exit() and sys.exit(None) exit code propagation when
used in multiprocessing.Process.

@item 
bpo-32173@footnote{https://bugs.python.org/issue32173}: * Add @code{lazycache} function to @code{__all__}. * Use @ref{122c,,dict.clear} to
clear the cache. * Refactoring @code{getline} function and @code{checkcache}
function.
@end itemize

@node Documentation<19>,Tests<18>,Library<18>,Python 3 9 0 alpha 4
@anchor{whatsnew/changelog id157}@anchor{122d}
@subsubsection Documentation



@itemize -

@item 
bpo-17422@footnote{https://bugs.python.org/issue17422}: The language reference now specifies restrictions on class
namespaces. Adapted from a patch by Ethan Furman.

@item 
bpo-39572@footnote{https://bugs.python.org/issue39572}: Updated documentation of @code{total} flag of @code{TypedDict}.

@item 
bpo-39654@footnote{https://bugs.python.org/issue39654}: In pyclbr doc, update ‘class’ to ‘module’ where appropriate and
add readmodule comment. Patch by Hakan Çelik.

@item 
bpo-39153@footnote{https://bugs.python.org/issue39153}: Clarify refcounting semantics for the following functions: -
PyObject_SetItem - PyMapping_SetItemString - PyDict_SetItem -
PyDict_SetItemString

@item 
bpo-39392@footnote{https://bugs.python.org/issue39392}: Explain that when filling with turtle, overlap regions may be
left unfilled.

@item 
bpo-39369@footnote{https://bugs.python.org/issue39369}: Update mmap readline method description. The fact that the
readline method does update the file position should not be ignored since
this might give the impression for the programmer that it doesn’t update
it.

@item 
bpo-9056@footnote{https://bugs.python.org/issue9056}: Include subsection in TOC for PDF version of docs.
@end itemize

@node Tests<18>,Build<18>,Documentation<19>,Python 3 9 0 alpha 4
@anchor{whatsnew/changelog id158}@anchor{122e}
@subsubsection Tests



@itemize -

@item 
bpo-38325@footnote{https://bugs.python.org/issue38325}: Skip tests on non-BMP characters of test_winconsoleio.

@item 
bpo-39502@footnote{https://bugs.python.org/issue39502}: Skip test_zipfile.test_add_file_after_2107() if
@ref{1224,,time.localtime()} fails with @ref{347,,OverflowError}. It is the case on
AIX 6.1 for example.
@end itemize

@node Build<18>,Windows<17>,Tests<18>,Python 3 9 0 alpha 4
@anchor{whatsnew/changelog id159}@anchor{122f}
@subsubsection Build



@itemize -

@item 
bpo-39489@footnote{https://bugs.python.org/issue39489}: Remove @code{COUNT_ALLOCS} special build.
@end itemize

@node Windows<17>,IDLE<16>,Build<18>,Python 3 9 0 alpha 4
@anchor{whatsnew/changelog id160}@anchor{1230}
@subsubsection Windows



@itemize -

@item 
bpo-39553@footnote{https://bugs.python.org/issue39553}: Delete unused code related to SxS manifests.

@item 
bpo-39439@footnote{https://bugs.python.org/issue39439}: Honor the Python path when a virtualenv is active on Windows.

@item 
bpo-39393@footnote{https://bugs.python.org/issue39393}: Improve the error message when attempting to load a DLL with
unresolved dependencies.

@item 
bpo-38883@footnote{https://bugs.python.org/issue38883}: @ref{9aa,,home()} and
@ref{9a9,,expanduser()} on Windows now prefer
@geindex USERPROFILE
@geindex environment variable; USERPROFILE
@code{USERPROFILE} and no longer use 
@geindex HOME
@geindex environment variable; HOME
@code{HOME}, which is not
normally set for regular user accounts. This makes them again behave like
@ref{520,,os.path.expanduser()}, which was changed to ignore 
@geindex HOME
@geindex environment variable; HOME
@code{HOME} in
3.8, see bpo-36264@footnote{https://bugs.python.org/issue36264}.

@item 
bpo-39185@footnote{https://bugs.python.org/issue39185}: The build.bat script has additional options for very-quiet
output (-q) and very-verbose output (-vv)
@end itemize

@node IDLE<16>,C API<18>,Windows<17>,Python 3 9 0 alpha 4
@anchor{whatsnew/changelog id161}@anchor{1231}
@subsubsection IDLE



@itemize -

@item 
bpo-39663@footnote{https://bugs.python.org/issue39663}: Add tests for pyparse find_good_parse_start().

@item 
bpo-39600@footnote{https://bugs.python.org/issue39600}: In the font configuration window, remove duplicated font names.

@item 
bpo-30780@footnote{https://bugs.python.org/issue30780}: Add remaining configdialog tests for buttons and highlights and
keys tabs.

@item 
bpo-39388@footnote{https://bugs.python.org/issue39388}: IDLE Settings Cancel button now cancels pending changes

@item 
bpo-38792@footnote{https://bugs.python.org/issue38792}: Close an IDLE shell calltip if a @ref{4ca,,KeyboardInterrupt} or
shell restart occurs.  Patch by Zackery Spytz.
@end itemize

@node C API<18>,,IDLE<16>,Python 3 9 0 alpha 4
@anchor{whatsnew/changelog id162}@anchor{1232}
@subsubsection C API



@itemize -

@item 
bpo-35081@footnote{https://bugs.python.org/issue35081}: Move the @code{bytes_methods.h} header file to the internal C API
as @code{pycore_bytes_methods.h}: it only contains private symbols (prefixed
by @code{_Py}), except of the @code{PyDoc_STRVAR_shared()} macro.

@item 
bpo-35081@footnote{https://bugs.python.org/issue35081}: Move the @code{dtoa.h} header file to the internal C API as
@code{pycore_dtoa.h}: it only contains private functions (prefixed by
@code{_Py}). The @ref{ad,,math} and @ref{17,,cmath} modules must now be compiled
with the @code{Py_BUILD_CORE} macro defined.

@item 
bpo-39573@footnote{https://bugs.python.org/issue39573}: Add @ref{1f4,,Py_SET_SIZE()} function to set the size of an
object.

@item 
bpo-39500@footnote{https://bugs.python.org/issue39500}: @ref{1233,,PyUnicode_IsIdentifier()} does not call
@ref{48f,,Py_FatalError()} anymore if the string is not ready.

@item 
bpo-39573@footnote{https://bugs.python.org/issue39573}: Add @ref{1f2,,Py_SET_TYPE()} function to set the type of an
object.

@item 
bpo-39573@footnote{https://bugs.python.org/issue39573}: Add a @ref{383,,Py_SET_REFCNT()} function to set the reference
counter of an object.

@item 
bpo-39542@footnote{https://bugs.python.org/issue39542}: Convert @ref{11db,,PyType_HasFeature()}, @ref{1234,,PyType_Check()} and
@ref{1235,,PyType_CheckExact()} macros to static inline functions.

@item 
bpo-39542@footnote{https://bugs.python.org/issue39542}: In the limited C API, @code{PyObject_INIT()} and
@code{PyObject_INIT_VAR()} are now defined as aliases to
@ref{581,,PyObject_Init()} and @ref{11dd,,PyObject_InitVar()} to make their
implementation opaque. It avoids to leak implementation details in the
limited C API. Exclude the following functions from the limited C API:
@code{_Py_NewReference()}, @code{_Py_ForgetReference()},
@code{_PyTraceMalloc_NewReference()} and @code{_Py_GetRefTotal()}.

@item 
bpo-39542@footnote{https://bugs.python.org/issue39542}: Exclude trashcan mechanism from the limited C API: it requires
access to PyTypeObject and PyThreadState structure fields, whereas these
structures are opaque in the limited C API.

@item 
bpo-39511@footnote{https://bugs.python.org/issue39511}: The @ref{11f3,,PyThreadState_Clear()} function now calls the
@code{PyThreadState.on_delete} callback. Previously, that happened in
@ref{1236,,PyThreadState_Delete()}.

@item 
bpo-38076@footnote{https://bugs.python.org/issue38076}: Fix to clear the interpreter state only after clearing module
globals to guarantee module state access from C Extensions during runtime
destruction

@item 
bpo-39245@footnote{https://bugs.python.org/issue39245}: The Vectorcall API (PEP 590) was made public, adding the
functions @code{PyObject_Vectorcall}, @code{PyObject_VectorcallMethod},
@code{PyVectorcall_Function}, @code{PyObject_CallOneArg},
@code{PyObject_CallMethodNoArgs}, @code{PyObject_CallMethodOneArg},
@code{PyObject_FastCallDict}, and the flag @code{Py_TPFLAGS_HAVE_VECTORCALL}.
@end itemize

@node Python 3 9 0 alpha 3,Python 3 9 0 alpha 2,Python 3 9 0 alpha 4,Changelog
@anchor{whatsnew/changelog python-3-9-0-alpha-3}@anchor{1237}
@subsection Python 3.9.0 alpha 3


@emph{Release date: 2020-01-24}

@menu
* Core and Builtins: Core and Builtins<19>. 
* Library: Library<19>. 
* Documentation: Documentation<20>. 
* Build: Build<19>. 
* IDLE: IDLE<17>. 
* C API: C API<19>. 

@end menu

@node Core and Builtins<19>,Library<19>,,Python 3 9 0 alpha 3
@anchor{whatsnew/changelog id163}@anchor{1238}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-39427@footnote{https://bugs.python.org/issue39427}: Document all possibilities for the @code{-X} options in the
command line help section. Patch by Pablo Galindo.

@item 
bpo-39421@footnote{https://bugs.python.org/issue39421}: Fix possible crashes when operating with the functions in the
@ref{89,,heapq} module and custom comparison operators.

@item 
bpo-39386@footnote{https://bugs.python.org/issue39386}: Prevent double awaiting of async iterator.

@item 
bpo-17005@footnote{https://bugs.python.org/issue17005}: Add @code{functools.TopologicalSorter} to the
@ref{7f,,functools} module to offers functionality to perform topological
sorting of graphs. Patch by Pablo Galindo, Tim Peters and Larry Hastings.

@item 
bpo-39320@footnote{https://bugs.python.org/issue39320}: Replace four complex bytecodes for building sequences with
three simpler ones.

The following four bytecodes have been removed:


@itemize *

@item 
BUILD_LIST_UNPACK

@item 
BUILD_TUPLE_UNPACK

@item 
BUILD_SET_UNPACK

@item 
BUILD_TUPLE_UNPACK_WITH_CALL
@end itemize

The following three bytecodes have been added:


@itemize *

@item 
LIST_TO_TUPLE

@item 
LIST_EXTEND

@item 
SET_UPDATE
@end itemize

@item 
bpo-39336@footnote{https://bugs.python.org/issue39336}: Import loaders which publish immutable module objects can now
publish immutable packages in addition to individual modules.

@item 
bpo-39322@footnote{https://bugs.python.org/issue39322}: Added a new function @ref{3ec,,gc.is_finalized()} to check if an
object has been finalized by the garbage collector. Patch by Pablo
Galindo.

@item 
bpo-39048@footnote{https://bugs.python.org/issue39048}: Improve the displayed error message when incorrect types are
passed to @code{async with} statements by looking up the @code{__aenter__()}
special method before the @code{__aexit__()} special method when entering
an asynchronous context manager. Patch by Géry Ogam.

@item 
bpo-39235@footnote{https://bugs.python.org/issue39235}: Fix AST end location for lone generator expression in function
call, e.g. f(i for i in a).

@item 
bpo-39209@footnote{https://bugs.python.org/issue39209}: Correctly handle multi-line tokens in interactive mode. Patch
by Pablo Galindo.

@item 
bpo-1635741@footnote{https://bugs.python.org/issue1635741}: Port _json extension module to multiphase initialization
(
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}).

@item 
bpo-39216@footnote{https://bugs.python.org/issue39216}: Fix constant folding optimization for positional only arguments
- by Anthony Sottile.

@item 
bpo-39215@footnote{https://bugs.python.org/issue39215}: Fix @code{SystemError} when nested function has annotation on
positional-only argument - by Anthony Sottile.

@item 
bpo-39200@footnote{https://bugs.python.org/issue39200}: Correct the error message when calling the @ref{a7d,,min()} or
@ref{a7e,,max()} with no arguments. Patch by Dong-hee Na.

@item 
bpo-39200@footnote{https://bugs.python.org/issue39200}: Correct the error message when trying to construct
@ref{438,,range} objects with no arguments. Patch by Pablo Galindo.

@item 
bpo-39166@footnote{https://bugs.python.org/issue39166}: Fix incorrect line execution reporting in trace functions when
tracing the last iteration of asynchronous for loops. Patch by Pablo
Galindo.

@item 
bpo-39114@footnote{https://bugs.python.org/issue39114}: Fix incorrect line execution reporting in trace functions when
tracing exception handlers with name binding. Patch by Pablo Galindo.

@item 
bpo-39156@footnote{https://bugs.python.org/issue39156}: Split the COMPARE_OP bytecode instruction into four distinct
instructions.


@itemize *

@item 
COMPARE_OP for rich comparisons

@item 
IS_OP for ‘is’ and ‘is not’ tests

@item 
CONTAINS_OP for ‘in’ and ‘is not’ tests

@item 
JUMP_IF_NOT_EXC_MATCH for checking exceptions in ‘try-except’ statements.
@end itemize

This improves the clarity of the interpreter and should provide a modest
speedup.

@item 
bpo-38588@footnote{https://bugs.python.org/issue38588}: Fix possible crashes in dict and list when calling
@ref{1239,,PyObject_RichCompareBool()}.

@item 
bpo-13601@footnote{https://bugs.python.org/issue13601}: By default, @code{sys.stderr} is line-buffered now, even if
@code{stderr} is redirected to a file. You can still make @code{sys.stderr}
unbuffered by passing the @ref{123a,,-u} command-line option or setting the
@geindex PYTHONUNBUFFERED
@geindex environment variable; PYTHONUNBUFFERED
@ref{123b,,PYTHONUNBUFFERED} environment variable.

(Contributed by Jendrik Seipp in bpo-13601@footnote{https://bugs.python.org/issue13601}.)

@item 
bpo-38610@footnote{https://bugs.python.org/issue38610}: Fix possible crashes in several list methods by holding strong
references to list elements when calling
@ref{1239,,PyObject_RichCompareBool()}.

@item 
bpo-32021@footnote{https://bugs.python.org/issue32021}: Include brotli .br encoding in mimetypes encodings_map
@end itemize

@node Library<19>,Documentation<20>,Core and Builtins<19>,Python 3 9 0 alpha 3
@anchor{whatsnew/changelog id164}@anchor{123c}
@subsubsection Library



@itemize -

@item 
bpo-39430@footnote{https://bugs.python.org/issue39430}: Fixed race condition in lazy imports in @ref{fb,,tarfile}.

@item 
bpo-39413@footnote{https://bugs.python.org/issue39413}: The @ref{40f,,os.unsetenv()} function is now also available on
Windows.

@item 
bpo-39390@footnote{https://bugs.python.org/issue39390}: Fixed a regression with the @code{ignore} callback of
@ref{538,,shutil.copytree()}. The argument types are now str and List[str]
again.

@item 
bpo-39395@footnote{https://bugs.python.org/issue39395}: The @ref{410,,os.putenv()} and @ref{40f,,os.unsetenv()} functions are now
always available.

@item 
bpo-39406@footnote{https://bugs.python.org/issue39406}: If @code{setenv()} C function is available, @ref{410,,os.putenv()} is
now implemented with @code{setenv()} instead of @code{putenv()}, so Python
doesn’t have to handle the environment variable memory.

@item 
bpo-39396@footnote{https://bugs.python.org/issue39396}: Fix @code{math.nextafter(-0.0, +0.0)} on AIX 7.1.

@item 
bpo-29435@footnote{https://bugs.python.org/issue29435}: Allow @ref{1096,,tarfile.is_tarfile()} to be used with file and
file-like objects, like @ref{ebc,,zipfile.is_zipfile()}. Patch by William
Woodruff.

@item 
bpo-39377@footnote{https://bugs.python.org/issue39377}: Removed @code{encoding} option from @ref{456,,json.loads()}.  It has
been deprecated since Python 3.1.

@item 
bpo-39389@footnote{https://bugs.python.org/issue39389}: Write accurate compression level metadata in @ref{87,,gzip}
archives, rather than always signaling maximum compression.

@item 
bpo-39366@footnote{https://bugs.python.org/issue39366}: The previously deprecated @code{xpath()} and @code{xgtitle()} methods
of @ref{409,,nntplib.NNTP} have been removed.

@item 
bpo-39357@footnote{https://bugs.python.org/issue39357}: Remove the @emph{buffering} parameter of @ref{455,,bz2.BZ2File}. Since
Python 3.0, it was ignored and using it was emitting
@ref{1b2,,DeprecationWarning}. Pass an open file object, to control how the
file is opened. The @emph{compresslevel} parameter becomes keyword-only.

@item 
bpo-39353@footnote{https://bugs.python.org/issue39353}: Deprecate binhex4 and hexbin4 standards. Deprecate the
@code{binhex} module and the following @ref{f,,binascii} functions:
@code{b2a_hqx()}, @code{a2b_hqx()},
@code{rlecode_hqx()}, @code{rledecode_hqx()},
@ref{1b8,,crc_hqx()}.

@item 
bpo-39351@footnote{https://bugs.python.org/issue39351}: Remove @code{base64.encodestring()} and @code{base64.decodestring()},
aliases deprecated since Python 3.1: use @ref{453,,base64.encodebytes()} and
@ref{454,,base64.decodebytes()} instead.

@item 
bpo-39350@footnote{https://bugs.python.org/issue39350}: Remove @code{fractions.gcd()} function, deprecated since Python
3.5 (bpo-22486@footnote{https://bugs.python.org/issue22486}): use @ref{401,,math.gcd()} instead.

@item 
bpo-39329@footnote{https://bugs.python.org/issue39329}: @ref{423,,LMTP} constructor now has an optional
@emph{timeout} parameter. Patch by Dong-hee Na.

@item 
bpo-39313@footnote{https://bugs.python.org/issue39313}: Add a new @code{exec_function} option (@emph{–exec-function} in the
CLI) to @code{RefactoringTool} for making @code{exec} a function. Patch by
Batuhan Taskaya.

@item 
bpo-39259@footnote{https://bugs.python.org/issue39259}: @ref{3ea,,FTP_TLS} and @ref{3ea,,FTP_TLS} now raise
a @ref{17c,,ValueError} if the given timeout for their constructor is zero
to prevent the creation of a non-blocking socket. Patch by Dong-hee Na.

@item 
bpo-39259@footnote{https://bugs.python.org/issue39259}: @ref{421,,SMTP} and @ref{422,,SMTP_SSL} now raise
a @ref{17c,,ValueError} if the given timeout for their constructor is zero
to prevent the creation of a non-blocking socket. Patch by Dong-hee Na.

@item 
bpo-39310@footnote{https://bugs.python.org/issue39310}: Add @ref{404,,math.ulp()}: return the value of the least significant
bit of a float.

@item 
bpo-39297@footnote{https://bugs.python.org/issue39297}: Improved performance of importlib.metadata distribution
discovery and resilients to inaccessible sys.path entries
(importlib_metadata v1.4.0).

@item 
bpo-39259@footnote{https://bugs.python.org/issue39259}: @ref{409,,NNTP} and @ref{40a,,NNTP_SSL} now raise
a @ref{17c,,ValueError} if the given timeout for their constructor is zero
to prevent the creation of a non-blocking socket. Patch by Dong-hee Na.

@item 
bpo-38901@footnote{https://bugs.python.org/issue38901}: When you specify prompt=’.’ or equivalently python -m venv
–prompt . … the basename of the current directory is used to set the
created venv’s prompt when it’s activated.

@item 
bpo-39288@footnote{https://bugs.python.org/issue39288}: Add @ref{403,,math.nextafter()}: return the next floating-point
value after @emph{x} towards @emph{y}.

@item 
bpo-39259@footnote{https://bugs.python.org/issue39259}: @ref{418,,POP3} and @ref{419,,POP3_SSL} now raise a
@ref{17c,,ValueError} if the given timeout for their constructor is zero to
prevent the creation of a non-blocking socket. Patch by Dong-hee Na.

@item 
bpo-39242@footnote{https://bugs.python.org/issue39242}: Updated the Gmane domain from news.gmane.org to news.gmane.io
which is used for examples of @ref{409,,NNTP} news reader server
and nntplib tests.

@item 
bpo-35292@footnote{https://bugs.python.org/issue35292}: Proxy the @code{SimpleHTTPRequestHandler.guess_type} to
@ref{123d,,mimetypes.guess_type} so the @ref{123e,,mimetypes.init} is called lazily to avoid
unnecessary costs when @ref{92,,http.server} module is imported.

@item 
bpo-39239@footnote{https://bugs.python.org/issue39239}: The @ref{45d,,select.epoll.unregister()} method no longer ignores
the @ref{45e,,EBADF} error.

@item 
bpo-38907@footnote{https://bugs.python.org/issue38907}: In http.server script, restore binding to IPv4 on Windows.

@item 
bpo-39152@footnote{https://bugs.python.org/issue39152}: Fix ttk.Scale.configure([name]) to return configuration tuple
for name or all options.  Giovanni Lombardo contributed part of the patch.

@item 
bpo-39198@footnote{https://bugs.python.org/issue39198}: If an exception were to be thrown in @code{Logger.isEnabledFor}
(say, by asyncio timeouts or stopit) , the @ref{a6,,logging} global lock may not
be released appropriately, resulting in deadlock.  This change wraps that
block of code with @code{try...finally} to ensure the lock is released.

@item 
bpo-39191@footnote{https://bugs.python.org/issue39191}: Perform a check for running loop before starting a new task in
@code{loop.run_until_complete()} to fail fast; it prevents the side effect of
new task spawning before exception raising.

@item 
bpo-38871@footnote{https://bugs.python.org/issue38871}: Correctly parenthesize filter-based statements that contain
lambda expressions in mod:@ref{a3,,lib2to3}. Patch by Dong-hee Na.

@item 
bpo-39142@footnote{https://bugs.python.org/issue39142}: A change was made to logging.config.dictConfig to avoid
converting instances of named tuples to ConvertingTuple. It’s assumed that
named tuples are too specialised to be treated like ordinary tuples; if a
user of named tuples requires ConvertingTuple functionality, they will
have to implement that themselves in their named tuple class.

@item 
bpo-39158@footnote{https://bugs.python.org/issue39158}: ast.literal_eval() now supports empty sets.

@item 
bpo-39129@footnote{https://bugs.python.org/issue39129}: Fix import path for @code{asyncio.TimeoutError}

@item 
bpo-39057@footnote{https://bugs.python.org/issue39057}: @code{urllib.request.proxy_bypass_environment()} now ignores
leading dots and no longer ignores a trailing newline.

@item 
bpo-39056@footnote{https://bugs.python.org/issue39056}: Fixed handling invalid warning category in the -W option.  No
longer import the re module if it is not needed.

@item 
bpo-39055@footnote{https://bugs.python.org/issue39055}: @ref{123f,,base64.b64decode()} with @code{validate=True} raises now a
binascii.Error if the input ends with a single @code{\n}.

@item 
bpo-21600@footnote{https://bugs.python.org/issue21600}: Fix @code{mock.patch.stopall()} to stop active patches that were
created with @code{mock.patch.dict()}.

@item 
bpo-39019@footnote{https://bugs.python.org/issue39019}: Implement dummy @code{__class_getitem__} for
@ref{cec,,tempfile.SpooledTemporaryFile}.

@item 
bpo-39019@footnote{https://bugs.python.org/issue39019}: Implement dummy @code{__class_getitem__} for @code{subprocess.Popen},
@code{subprocess.CompletedProcess}

@item 
bpo-38914@footnote{https://bugs.python.org/issue38914}: Adjusted the wording of the warning issued by distutils’
@code{check} command when the @code{author} and @code{maintainer} fields are
supplied but no corresponding e-mail field (@code{author_email} or
@code{maintainer_email}) is found. The wording now reflects the fact that
these fields are suggested, but not required. Patch by Juergen Gmach.

@item 
bpo-38878@footnote{https://bugs.python.org/issue38878}: Fixed __subclasshook__ of @ref{795,,os.PathLike} to return a
correct result upon inheritance. Patch by Bar Harel.

@item 
bpo-38615@footnote{https://bugs.python.org/issue38615}: @ref{3f2,,IMAP4} and @ref{3f3,,IMAP4_SSL} now
have an optional @emph{timeout} parameter for their constructors. Also, the
@ref{3f4,,open()} method now has an optional @emph{timeout} parameter
with this change. The overridden methods of @ref{3f3,,IMAP4_SSL}
and @ref{3f5,,IMAP4_stream} were applied to this change. Patch by
Dong-hee Na.

@item 
bpo-35182@footnote{https://bugs.python.org/issue35182}: Fixed @code{Popen.communicate()} subsequent call crash when the
child process has already closed any piped standard stream, but still
continues to be running. Patch by Andriy Maletsky.

@item 
bpo-38630@footnote{https://bugs.python.org/issue38630}: On Unix, @ref{1240,,subprocess.Popen.send_signal()} now polls the
process status. Polling reduces the risk of sending a signal to the wrong
process if the process completed, the @ref{1241,,subprocess.Popen.returncode}
attribute is still @code{None}, and the pid has been reassigned (recycled) to
a new different process.

@item 
bpo-38536@footnote{https://bugs.python.org/issue38536}: Removes trailing space in formatted currency with
@code{international=True} and a locale with symbol following value. E.g.
@code{locale.currency(12.34, international=True)} returned @code{'12,34 EUR '}
instead of @code{'12,34 EUR'}.

@item 
bpo-38473@footnote{https://bugs.python.org/issue38473}: Use signature from inner mock for autospecced methods attached
with @code{unittest.mock.attach_mock()}. Patch by Karthikeyan Singaravelan.

@item 
bpo-38361@footnote{https://bugs.python.org/issue38361}: Fixed an issue where @code{ident} could include a leading path
separator when @ref{1242,,syslog.openlog()} was called without arguments.

@item 
bpo-38293@footnote{https://bugs.python.org/issue38293}: Add @ref{69a,,copy.copy()} and @ref{69b,,copy.deepcopy()} support to
@ref{501,,property()} objects.

@item 
bpo-37958@footnote{https://bugs.python.org/issue37958}: Added the pstats.Stats.get_profile_dict() method to return the
profile data as a StatsProfile instance.

@item 
bpo-28367@footnote{https://bugs.python.org/issue28367}: Termios magic constants for the following baud rates:   -
B500000   - B576000   - B921600   - B1000000   - B1152000   - B1500000   -
B2000000   - B2500000   - B3000000   - B3500000   - B4000000 Patch by
Andrey Smirnov
@end itemize

@node Documentation<20>,Build<19>,Library<19>,Python 3 9 0 alpha 3
@anchor{whatsnew/changelog id165}@anchor{1243}
@subsubsection Documentation



@itemize -

@item 
bpo-39381@footnote{https://bugs.python.org/issue39381}: Mention in docs that @ref{314,,asyncio.get_event_loop()} implicitly
creates new event loop only if called from the main thread.

@item 
bpo-38918@footnote{https://bugs.python.org/issue38918}: Add an entry for @code{__module__} in the “function” & “method”
sections of the inspect docs types and members table@footnote{https://docs.python.org/3/library/inspect.html#types-and-members}

@item 
bpo-3530@footnote{https://bugs.python.org/issue3530}: In the @ref{7,,ast} module documentation, fix a misleading
@code{NodeTransformer} example and add advice on when to use the
@code{fix_missing_locations} function.
@end itemize

@node Build<19>,IDLE<17>,Documentation<20>,Python 3 9 0 alpha 3
@anchor{whatsnew/changelog id166}@anchor{1244}
@subsubsection Build



@itemize -

@item 
bpo-39395@footnote{https://bugs.python.org/issue39395}: On non-Windows platforms, the @code{setenv()} and
@code{unsetenv()} functions are now required to build Python.

@item 
bpo-39160@footnote{https://bugs.python.org/issue39160}: Updated the documentation in @code{/configure --help} to show
default values, reference documentation where required and add additional
explanation where needed.

@item 
bpo-39144@footnote{https://bugs.python.org/issue39144}: The ctags and etags build targets both include Modules/_ctypes
and Python standard library source files.
@end itemize

@node IDLE<17>,C API<19>,Build<19>,Python 3 9 0 alpha 3
@anchor{whatsnew/changelog id167}@anchor{1245}
@subsubsection IDLE



@itemize -

@item 
bpo-39050@footnote{https://bugs.python.org/issue39050}: Make IDLE Settings dialog Help button work again.

@item 
bpo-34118@footnote{https://bugs.python.org/issue34118}: Tag memoryview, range, and tuple as classes, the same as list,
etcetera, in the library manual built-in functions list.

@item 
bpo-32989@footnote{https://bugs.python.org/issue32989}: Add tests for editor newline_and_indent_event method. Remove
dead code from pyparse find_good_parse_start method.
@end itemize

@node C API<19>,,IDLE<17>,Python 3 9 0 alpha 3
@anchor{whatsnew/changelog id168}@anchor{1246}
@subsubsection C API



@itemize -

@item 
bpo-39372@footnote{https://bugs.python.org/issue39372}: Clean header files of interfaces defined but with no
implementation. The public API symbols being removed are:
@code{_PyBytes_InsertThousandsGroupingLocale},
@code{_PyBytes_InsertThousandsGrouping}, @code{_Py_InitializeFromArgs},
@code{_Py_InitializeFromWideArgs}, @code{_PyFloat_Repr}, @code{_PyFloat_Digits},
@code{_PyFloat_DigitsInit}, @code{PyFrame_ExtendStack},
@code{_PyAIterWrapper_Type}, @code{PyNullImporter_Type}, @code{PyCmpWrapper_Type},
@code{PySortWrapper_Type}, @code{PyNoArgsFunction}.

@item 
bpo-39164@footnote{https://bugs.python.org/issue39164}: Add a private @code{_PyErr_GetExcInfo()} function to retrieve
exception information of the specified Python thread state.
@end itemize

@node Python 3 9 0 alpha 2,Python 3 9 0 alpha 1,Python 3 9 0 alpha 3,Changelog
@anchor{whatsnew/changelog python-3-9-0-alpha-2}@anchor{1247}
@subsection Python 3.9.0 alpha 2


@emph{Release date: 2019-12-18}

@menu
* Security: Security<13>. 
* Core and Builtins: Core and Builtins<20>. 
* Library: Library<20>. 
* Documentation: Documentation<21>. 
* Tests: Tests<19>. 
* Build: Build<20>. 
* Windows: Windows<18>. 
* macOS: macOS<16>. 
* IDLE: IDLE<18>. 
* C API: C API<20>. 

@end menu

@node Security<13>,Core and Builtins<20>,,Python 3 9 0 alpha 2
@anchor{whatsnew/changelog id169}@anchor{1248}
@subsubsection Security



@itemize -

@item 
bpo-38945@footnote{https://bugs.python.org/issue38945}: Newline characters have been escaped when performing uu
encoding to prevent them from overflowing into to content section of the
encoded file. This prevents malicious or accidental modification of data
during the decoding process.

@item 
bpo-37228@footnote{https://bugs.python.org/issue37228}: Due to significant security concerns, the @emph{reuse_address}
parameter of @ref{1b9,,asyncio.loop.create_datagram_endpoint()} is no longer
supported. This is because of the behavior of @code{SO_REUSEADDR} in UDP. For
more details, see the documentation for
@code{loop.create_datagram_endpoint()}. (Contributed by Kyle Stanley, Antoine
Pitrou, and Yury Selivanov in bpo-37228@footnote{https://bugs.python.org/issue37228}.)

@item 
bpo-38804@footnote{https://bugs.python.org/issue38804}: Fixes a ReDoS vulnerability in @ref{90,,http.cookiejar}. Patch by
Ben Caller.
@end itemize

@node Core and Builtins<20>,Library<20>,Security<13>,Python 3 9 0 alpha 2
@anchor{whatsnew/changelog id170}@anchor{1249}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-39028@footnote{https://bugs.python.org/issue39028}: Slightly improve the speed of keyword argument parsing with
many kwargs by strengthening the assumption that kwargs are interned
strings.

@item 
bpo-39080@footnote{https://bugs.python.org/issue39080}: Fix the value of @emph{end_col_offset} for Starred Expression AST
nodes when they are among the elements in the @emph{args} attribute of Call AST
nodes.

@item 
bpo-39031@footnote{https://bugs.python.org/issue39031}: When parsing an “elif” node, lineno and col_offset of the node
now point to the “elif” keyword and not to its condition, making it
consistent with the “if” node. Patch by Lysandros Nikolaou.

@item 
bpo-20443@footnote{https://bugs.python.org/issue20443}: In Python 3.9.0a1, sys.argv[0] was made an absolute path if a
filename was specified on the command line. Revert this change, since most
users expect sys.argv to be unmodified.

@item 
bpo-39008@footnote{https://bugs.python.org/issue39008}: @ref{120d,,PySys_Audit()} now requires @code{Py_ssize_t} to be used
for size arguments in the format string, regardless of whether
@code{PY_SSIZE_T_CLEAN} was defined at include time.

@item 
bpo-38673@footnote{https://bugs.python.org/issue38673}: In REPL mode, don’t switch to PS2 if the line starts with
comment or whitespace. Based on work by Batuhan Taşkaya.

@item 
bpo-38922@footnote{https://bugs.python.org/issue38922}: Calling @code{replace} on a code object now raises the
@code{code.__new__} audit event.

@item 
bpo-38920@footnote{https://bugs.python.org/issue38920}: Add audit hooks for when @ref{28e,,sys.excepthook()} and
@ref{18d,,sys.unraisablehook()} are invoked.

@item 
bpo-38892@footnote{https://bugs.python.org/issue38892}: Improve documentation for audit events table and functions.

@item 
bpo-38852@footnote{https://bugs.python.org/issue38852}: Set the thread stack size to 8 Mb for debug builds on android
platforms.

@item 
bpo-38858@footnote{https://bugs.python.org/issue38858}: Each Python subinterpreter now has its own “small integer
singletons”: numbers in [-5; 257] range. It is no longer possible to
change the number of small integers at build time by overriding
@code{NSMALLNEGINTS} and @code{NSMALLPOSINTS} macros: macros should now be
modified manually in @code{pycore_pystate.h} header file.

@item 
bpo-36854@footnote{https://bugs.python.org/issue36854}: The garbage collector state becomes per interpreter
(@code{PyInterpreterState.gc}), rather than being global
(@code{_PyRuntimeState.gc}).

@item 
bpo-38835@footnote{https://bugs.python.org/issue38835}: The @code{PyFPE_START_PROTECT()} and @code{PyFPE_END_PROTECT()}
macros are empty: they have been doing nothing for the last year, so stop
using them.

@item 
bpo-38328@footnote{https://bugs.python.org/issue38328}: Sped up the creation time of constant @ref{43a,,list} and
@ref{370,,set} displays. Patch by Brandt Bucher.

@item 
bpo-38707@footnote{https://bugs.python.org/issue38707}: @code{MainThread.native_id} is now correctly reset in child
processes spawned using @ref{681,,multiprocessing.Process}, instead of
retaining the parent’s value.

@item 
bpo-38629@footnote{https://bugs.python.org/issue38629}: Added @code{__floor__} and @code{__ceil__} methods to float object.
Patch by Batuhan Taşkaya.

@item 
bpo-27145@footnote{https://bugs.python.org/issue27145}: int + int and int - int operators can now return small integer
singletons. Patch by hongweipeng.

@item 
bpo-38021@footnote{https://bugs.python.org/issue38021}: Provide a platform tag for AIX that is sufficient for PEP425
binary distribution identification. Patch by Michael Felt.

@item 
bpo-35409@footnote{https://bugs.python.org/issue35409}: Ignore GeneratorExit exceptions when throwing an exception into
the aclose coroutine of an asynchronous generator.

@item 
bpo-33387@footnote{https://bugs.python.org/issue33387}: Removed WITH_CLEANUP_START, WITH_CLEANUP_FINISH, BEGIN_FINALLY,
END_FINALLY, CALL_FINALLY and POP_FINALLY bytecodes. Replaced with RERAISE
and WITH_EXCEPT_START bytecodes. The compiler now generates different code
for exceptional and non-exceptional branches for ‘with’ and ‘try-except’
statements. For ‘try-finally’ statements the ‘finally’ block is replicated
for each exit from the ‘try’ body.
@end itemize

@node Library<20>,Documentation<21>,Core and Builtins<20>,Python 3 9 0 alpha 2
@anchor{whatsnew/changelog id171}@anchor{124a}
@subsubsection Library



@itemize -

@item 
bpo-39033@footnote{https://bugs.python.org/issue39033}: Fix @ref{21f,,NameError} in @ref{14a,,zipimport}. Patch by Karthikeyan
Singaravelan.

@item 
bpo-39022@footnote{https://bugs.python.org/issue39022}: Update importlib.metadata to include improvements from
importlib_metadata 1.3 including better serialization of EntryPoints and
improved documentation for custom finders.

@item 
bpo-39006@footnote{https://bugs.python.org/issue39006}: Fix asyncio when the ssl module is missing: only check for
ssl.SSLSocket instance if the ssl module is available.

@item 
bpo-38708@footnote{https://bugs.python.org/issue38708}: Fix a potential IndexError in email parser when parsing an
empty msg-id.

@item 
bpo-38698@footnote{https://bugs.python.org/issue38698}: Add a new @code{InvalidMessageID} token to email parser to
represent invalid Message-ID headers.  Also, add defects when there is
remaining value after parsing the header.

@item 
bpo-38994@footnote{https://bugs.python.org/issue38994}: Implement @code{__class_getitem__} for @code{os.PathLike},
@code{pathlib.Path}.

@item 
bpo-38979@footnote{https://bugs.python.org/issue38979}: Return class from @code{ContextVar.__class_getitem__} to simplify
subclassing.

@item 
bpo-38978@footnote{https://bugs.python.org/issue38978}: Implement @code{__class_getitem__} on asyncio objects (Future,
Task, Queue). Patch by Batuhan Taskaya.

@item 
bpo-38916@footnote{https://bugs.python.org/issue38916}: @ref{25e,,array.array}: Remove @code{tostring()} and
@code{fromstring()} methods. They were aliases to @code{tobytes()} and
@code{frombytes()}, deprecated since Python 3.2.

@item 
bpo-38986@footnote{https://bugs.python.org/issue38986}: Make repr of C accelerated TaskWakeupMethWrapper the same as of
pure Python version.

@item 
bpo-38982@footnote{https://bugs.python.org/issue38982}: Fix asyncio @code{PidfdChildWatcher}: handle @code{waitpid()} error.
If @code{waitpid()} is called elsewhere, @code{waitpid()} call fails with
@ref{bee,,ChildProcessError}: use return code 255 in this case, and log a
warning. It ensures that the pidfd file descriptor is closed if this error
occurs.

@item 
bpo-38529@footnote{https://bugs.python.org/issue38529}: Drop too noisy asyncio warning about deletion of a stream
without explicit @code{.close()} call.

@item 
bpo-27413@footnote{https://bugs.python.org/issue27413}: Added ability to pass through @code{ensure_ascii} options to
json.dumps in the @code{json.tool} command-line interface.

@item 
bpo-38634@footnote{https://bugs.python.org/issue38634}: The @ref{d9,,readline} module now detects if Python is linked to
libedit at runtime on all platforms.  Previously, the check was only done
on macOS.

@item 
bpo-33684@footnote{https://bugs.python.org/issue33684}: Fix @code{json.tool} failed to read a JSON file with non-ASCII
characters when locale encoding is not UTF-8.

@item 
bpo-38698@footnote{https://bugs.python.org/issue38698}: Prevent UnboundLocalError to pop up in parse_message_id.

parse_message_id() was improperly using a token defined inside an
exception handler, which was raising @ref{1067,,UnboundLocalError} on parsing an
invalid value. Patch by Claudiu Popa.

@item 
bpo-38927@footnote{https://bugs.python.org/issue38927}: Use @code{python -m pip} instead of @code{pip} to upgrade
dependencies in venv.

@item 
bpo-26730@footnote{https://bugs.python.org/issue26730}: Fix @code{SpooledTemporaryFile.rollover()} might corrupt the file
when it is in text mode. Patch by Serhiy Storchaka.

@item 
bpo-38881@footnote{https://bugs.python.org/issue38881}: random.choices() now raises a ValueError when all the weights
are zero.

@item 
bpo-38876@footnote{https://bugs.python.org/issue38876}: Raise pickle.UnpicklingError when loading an item from memo for
invalid input.

The previous code was raising a @ref{5bb,,KeyError} for both the Python and C
implementation. This was caused by the specified index of an invalid input
which did not exist in the memo structure, where the pickle stores what
objects it has seen. The malformed input would have caused either a
@code{BINGET} or @code{LONG_BINGET} load from the memo, leading to a @ref{5bb,,KeyError} as
the determined index was bogus. Patch by Claudiu Popa

@item 
bpo-38688@footnote{https://bugs.python.org/issue38688}: Calling func:@ref{538,,shutil.copytree} to copy a directory tree from
one directory to another subdirectory resulted in an endless loop and a
RecursionError. A fix was added to consume an iterator and create the list
of the entries to be copied, avoiding the recursion for newly created
directories. Patch by Bruno P. Kinoshita.

@item 
bpo-38863@footnote{https://bugs.python.org/issue38863}: Improve @code{is_cgi()} function in @ref{92,,http.server}, which
enables processing the case that cgi directory is a child of another
directory other than root.

@item 
bpo-37838@footnote{https://bugs.python.org/issue37838}: @ref{432,,typing.get_type_hints()} properly handles functions
decorated with @ref{adb,,functools.wraps()}.

@item 
bpo-38870@footnote{https://bugs.python.org/issue38870}: Expose @ref{3cc,,ast.unparse()} as a function of the @ref{7,,ast}
module that can be used to unparse an @ref{3cd,,ast.AST} object and produce
a string with code that would produce an equivalent @ref{3cd,,ast.AST}
object when parsed. Patch by Pablo Galindo and Batuhan Taskaya.

@item 
bpo-38859@footnote{https://bugs.python.org/issue38859}: AsyncMock now returns StopAsyncIteration on the exhaustion of a
side_effects iterable. Since PEP-479 its Impossible to raise a
StopIteration exception from a coroutine.

@item 
bpo-38857@footnote{https://bugs.python.org/issue38857}: AsyncMock fix for return values that are awaitable types.  This
also covers side_effect iterable values that happened to be awaitable, and
wraps callables that return an awaitable type. Before these awaitables
were being awaited instead of being returned as is.

@item 
bpo-38834@footnote{https://bugs.python.org/issue38834}: @ref{2de,,typing.TypedDict} subclasses now track which keys are
optional using the @code{__required_keys__} and @code{__optional_keys__}
attributes, to enable runtime validation by downstream projects.  Patch by
Zac Hatfield-Dodds.

@item 
bpo-38821@footnote{https://bugs.python.org/issue38821}: Fix unhandled exceptions in @ref{5,,argparse} when
internationalizing error messages for arguments with @code{nargs} set to
special (non-integer) values. Patch by Federico Bond.

@item 
bpo-38820@footnote{https://bugs.python.org/issue38820}: Make Python compatible with OpenSSL 3.0.0.
@ref{b39,,ssl.SSLSocket.getpeercert()} no longer returns IPv6 addresses with a
trailing new line.

@item 
bpo-38811@footnote{https://bugs.python.org/issue38811}: Fix an unhandled exception in @ref{c3,,pathlib} when
@ref{c82,,os.link()} is missing. Patch by Toke Høiland-Jørgensen.

@item 
bpo-38686@footnote{https://bugs.python.org/issue38686}: Added support for multiple @code{qop} values in
@ref{1211,,urllib.request.AbstractDigestAuthHandler}.

@item 
bpo-38712@footnote{https://bugs.python.org/issue38712}: Add the Linux-specific @ref{41f,,signal.pidfd_send_signal()}
function, which allows sending a signal to a process identified by a file
descriptor rather than a pid.

@item 
bpo-38348@footnote{https://bugs.python.org/issue38348}: Add @code{-i} and @code{--indent} (indentation level), and
@code{--no-type-comments} (type comments) command line options to ast parsing
tool.

@item 
bpo-37523@footnote{https://bugs.python.org/issue37523}: Change @code{zipfile.ZipExtFile} to raise @code{ValueError} when
trying to access the underlying file object after it has been closed. This
new behavior is consistent with how accessing closed files is handled in
other parts of Python.

@item 
bpo-38045@footnote{https://bugs.python.org/issue38045}: Improve the performance of @code{enum._decompose()} in
@ref{76,,enum}. Patch by hongweipeng.

@item 
bpo-36820@footnote{https://bugs.python.org/issue36820}: Break cycle generated when saving an exception in socket.py,
codeop.py and dyld.py as they keep alive not only the exception but user
objects through the @code{__traceback__} attribute. Patch by Mario Corchero.

@item 
bpo-36406@footnote{https://bugs.python.org/issue36406}: Handle namespace packages in @ref{63,,doctest}. Patch by
Karthikeyan Singaravelan.

@item 
bpo-34776@footnote{https://bugs.python.org/issue34776}: Fix dataclasses to support forward references in type
annotations

@item 
bpo-20928@footnote{https://bugs.python.org/issue20928}: ElementTree supports recursive XInclude processing.  Patch by
Stefan Behnel.

@item 
bpo-29636@footnote{https://bugs.python.org/issue29636}: Add whitespace options for formatting JSON with the
@code{json.tool} CLI. The following mutually exclusive options are now
supported: @code{--indent} for setting the indent level in spaces; @code{--tab}
for indenting with tabs; @code{--no-indent} for suppressing newlines; and
@code{--compact} for suppressing all whitespace. The default behavior remains
the same as @code{--indent=4}.
@end itemize

@node Documentation<21>,Tests<19>,Library<20>,Python 3 9 0 alpha 2
@anchor{whatsnew/changelog id172}@anchor{124b}
@subsubsection Documentation



@itemize -

@item 
bpo-38928@footnote{https://bugs.python.org/issue38928}: Correct when venv’s @code{upgrade_dependencies()} and
@code{--upgrade-deps} are added.

@item 
bpo-38899@footnote{https://bugs.python.org/issue38899}: Update documentation to state that to activate virtual
environments under fish one should use @code{source}, not @code{} as documented at
@indicateurl{https://fishshell.com/docs/current/commands.html#source}.

@item 
bpo-22377@footnote{https://bugs.python.org/issue22377}: Improves documentation of the values that
@ref{114f,,datetime.datetime.strptime()} accepts for @code{%Z}. Patch by Karl
Dubost.
@end itemize

@node Tests<19>,Build<20>,Documentation<21>,Python 3 9 0 alpha 2
@anchor{whatsnew/changelog id173}@anchor{124c}
@subsubsection Tests



@itemize -

@item 
bpo-38546@footnote{https://bugs.python.org/issue38546}: Fix test_ressources_gced_in_workers() of
test_concurrent_futures: explicitly stop the manager to prevent leaking a
child process running in the background after the test completes.

@item 
bpo-38546@footnote{https://bugs.python.org/issue38546}: Multiprocessing and concurrent.futures tests now stop the
resource tracker process when tests complete.

@item 
bpo-38614@footnote{https://bugs.python.org/issue38614}: Replace hardcoded timeout constants in tests with new
@ref{100,,test.support} constants: @ref{124d,,LOOPBACK_TIMEOUT},
@ref{124e,,INTERNET_TIMEOUT},
@ref{1120,,SHORT_TIMEOUT} and
@ref{124f,,LONG_TIMEOUT}. It becomes easier to adjust these four
timeout constants for all tests at once, rather than having to adjust
every single test file.

@item 
bpo-38547@footnote{https://bugs.python.org/issue38547}: Fix test_pty: if the process is the session leader, closing the
master file descriptor raises a SIGHUP signal: simply ignore SIGHUP when
running the tests.

@item 
bpo-38992@footnote{https://bugs.python.org/issue38992}: Fix a test for @ref{f3a,,math.fsum()} that was failing due to
constant folding.

@item 
bpo-38991@footnote{https://bugs.python.org/issue38991}: @ref{100,,test.support}:
@code{run_python_until_end()},
@code{assert_python_ok()} and
@code{assert_python_failure()} functions no longer strip
whitespaces from stderr. Remove @code{test.support.strip_python_stderr()}
function.

@item 
bpo-38965@footnote{https://bugs.python.org/issue38965}: Fix test_faulthandler on GCC 10. Use the “volatile” keyword in
@code{faulthandler._stack_overflow()} to prevent tail call optimization on
any compiler, rather than relying on compiler specific pragma.

@item 
bpo-38875@footnote{https://bugs.python.org/issue38875}: test_capi: trashcan tests now require the test “cpu” resource.

@item 
bpo-38841@footnote{https://bugs.python.org/issue38841}: Skip asyncio test_create_datagram_endpoint_existing_sock_unix
on platforms lacking a functional bind() for named unix domain sockets.

@item 
bpo-38692@footnote{https://bugs.python.org/issue38692}: Skip the test_posix.test_pidfd_open() test if
@code{os.pidfd_open()} fails with a @ref{895,,PermissionError}. This situation
can happen in a Linux sandbox using a syscall whitelist which doesn’t
allow the @code{pidfd_open()} syscall yet.

@item 
bpo-38839@footnote{https://bugs.python.org/issue38839}: Fix some unused functions in tests. Patch by Adam Johnson.

@item 
bpo-38669@footnote{https://bugs.python.org/issue38669}: Raise @ref{162,,TypeError} when passing target as a string with
@ref{1250,,unittest.mock.patch.object()}.

@item 
bpo-37957@footnote{https://bugs.python.org/issue37957}: test.regrtest now can receive a list of test patterns to ignore
(using the -i/–ignore argument) or a file with a list of patterns to
ignore (using the –ignore-file argument). Patch by Pablo Galindo.
@end itemize

@node Build<20>,Windows<18>,Tests<19>,Python 3 9 0 alpha 2
@anchor{whatsnew/changelog id174}@anchor{1251}
@subsubsection Build



@itemize -

@item 
bpo-37404@footnote{https://bugs.python.org/issue37404}: @ref{9,,asyncio} now raises @code{TyperError} when calling
incompatible methods with an @ref{3d6,,ssl.SSLSocket} socket.  Patch by Ido
Michael.

@item 
bpo-36500@footnote{https://bugs.python.org/issue36500}: Added an optional “regen” project to the Visual Studio solution
that will regenerate all grammar, tokens, and opcodes.
@end itemize

@node Windows<18>,macOS<16>,Build<20>,Python 3 9 0 alpha 2
@anchor{whatsnew/changelog id175}@anchor{1252}
@subsubsection Windows



@itemize -

@item 
bpo-39007@footnote{https://bugs.python.org/issue39007}: Add auditing events to functions in @ref{131,,winreg}.

@item 
bpo-33125@footnote{https://bugs.python.org/issue33125}: Add support for building and releasing Windows ARM64 packages.
@end itemize

@node macOS<16>,IDLE<18>,Windows<18>,Python 3 9 0 alpha 2
@anchor{whatsnew/changelog id176}@anchor{1253}
@subsubsection macOS



@itemize -

@item 
bpo-37931@footnote{https://bugs.python.org/issue37931}: Fixed a crash on OSX dynamic builds that occurred when
re-initializing the posix module after a Py_Finalize if the environment
had changed since the previous @code{import posix}. Patch by Benoît Hudson.
@end itemize

@node IDLE<18>,C API<20>,macOS<16>,Python 3 9 0 alpha 2
@anchor{whatsnew/changelog id177}@anchor{1254}
@subsubsection IDLE



@itemize -

@item 
bpo-38944@footnote{https://bugs.python.org/issue38944}: Escape key now closes IDLE completion windows.  Patch by Johnny
Najera.

@item 
bpo-38943@footnote{https://bugs.python.org/issue38943}: Fix IDLE autocomplete windows not always appearing on some
systems. Patch by Johnny Najera.

@item 
bpo-38862@footnote{https://bugs.python.org/issue38862}: ‘Strip Trailing Whitespace’ on the Format menu removes extra
newlines at the end of non-shell files.

@item 
bpo-38636@footnote{https://bugs.python.org/issue38636}: Fix IDLE Format menu tab toggle and file indent width. These
functions (default shortcuts Alt-T and Alt-U) were mistakenly disabled in
3.7.5 and 3.8.0.
@end itemize

@node C API<20>,,IDLE<18>,Python 3 9 0 alpha 2
@anchor{whatsnew/changelog id178}@anchor{1255}
@subsubsection C API



@itemize -

@item 
bpo-38896@footnote{https://bugs.python.org/issue38896}: Remove @code{PyUnicode_ClearFreeList()} function: the Unicode free
list has been removed in Python 3.3.

@item 
bpo-37340@footnote{https://bugs.python.org/issue37340}: Remove @code{PyMethod_ClearFreeList()} and
@code{PyCFunction_ClearFreeList()} functions: the free lists of bound method
objects have been removed.

@item 
bpo-38835@footnote{https://bugs.python.org/issue38835}: Exclude @code{PyFPE_START_PROTECT()} and @code{PyFPE_END_PROTECT()}
macros of @code{pyfpe.h} from @code{Py_LIMITED_API} (stable API).
@end itemize

@node Python 3 9 0 alpha 1,Python 3 8 0 beta 1,Python 3 9 0 alpha 2,Changelog
@anchor{whatsnew/changelog python-3-9-0-alpha-1}@anchor{1256}
@subsection Python 3.9.0 alpha 1


@emph{Release date: 2019-11-19}

@menu
* Security: Security<14>. 
* Core and Builtins: Core and Builtins<21>. 
* Library: Library<21>. 
* Documentation: Documentation<22>. 
* Tests: Tests<20>. 
* Build: Build<21>. 
* Windows: Windows<19>. 
* macOS: macOS<17>. 
* IDLE: IDLE<19>. 
* Tools/Demos: Tools/Demos<7>. 
* C API: C API<21>. 

@end menu

@node Security<14>,Core and Builtins<21>,,Python 3 9 0 alpha 1
@anchor{whatsnew/changelog id179}@anchor{1257}
@subsubsection Security



@itemize -

@item 
bpo-38722@footnote{https://bugs.python.org/issue38722}: @ref{dd,,runpy} now uses @ref{11f6,,io.open_code()} to open code files.
Patch by Jason Killen.

@item 
bpo-38622@footnote{https://bugs.python.org/issue38622}: Add additional audit events for the @ref{29,,ctypes} module.

@item 
bpo-38418@footnote{https://bugs.python.org/issue38418}: Fixes audit event for @ref{fbb,,os.system()} to be named
@code{os.system}.

@item 
bpo-38243@footnote{https://bugs.python.org/issue38243}: Escape the server title of
@ref{1258,,xmlrpc.server.DocXMLRPCServer} when rendering the document page as
HTML. (Contributed by Dong-hee Na in bpo-38243@footnote{https://bugs.python.org/issue38243}.)

@item 
bpo-38174@footnote{https://bugs.python.org/issue38174}: Update vendorized expat library version to 2.2.8, which
resolves CVE-2019-15903.

@item 
bpo-37764@footnote{https://bugs.python.org/issue37764}: Fixes email._header_value_parser.get_unstructured going into an
infinite loop for a specific case in which the email header does not have
trailing whitespace, and the case in which it contains an invalid encoded
word. Patch by Ashwin Ramaswami.

@item 
bpo-37461@footnote{https://bugs.python.org/issue37461}: Fix an infinite loop when parsing specially crafted email
headers. Patch by Abhilash Raj.

@item 
bpo-37363@footnote{https://bugs.python.org/issue37363}: Adds audit events for the range of supported run commands (see
@ref{1259,,Command line and environment}).

@item 
bpo-37463@footnote{https://bugs.python.org/issue37463}: ssl.match_hostname() no longer accepts IPv4 addresses with
additional text after the address and only quad-dotted notation without
trailing whitespaces. Some inet_aton() implementations ignore whitespace
and all data after whitespace, e.g. ‘127.0.0.1 whatever’.

@item 
bpo-37363@footnote{https://bugs.python.org/issue37363}: Adds audit events for @ref{75,,ensurepip}, @ref{7e,,ftplib},
@ref{84,,glob}, @ref{93,,imaplib}, @ref{bc,,nntplib}, @ref{c4,,pdb}, @ref{cb,,poplib},
@ref{e4,,shutil}, @ref{e8,,smtplib}, @ref{ed,,sqlite3}, @ref{f4,,subprocess},
@ref{fc,,telnetlib}, @ref{fd,,tempfile} and @ref{130,,webbrowser}, as well as
@ref{c8d,,os.listdir()}, @ref{576,,os.scandir()} and @ref{5db,,breakpoint()}.

@item 
bpo-37364@footnote{https://bugs.python.org/issue37364}: @ref{11f6,,io.open_code()} is now used when reading @code{.pth}
files.

@item 
bpo-34631@footnote{https://bugs.python.org/issue34631}: Updated OpenSSL to 1.1.1c in Windows installer

@item 
bpo-34155@footnote{https://bugs.python.org/issue34155}: Fix parsing of invalid email addresses with more than one @code{@@}
(e.g. a@@b@@c.com.) to not return the part before 2nd @code{@@} as valid email
address. Patch by maxking & jpic.
@end itemize

@node Core and Builtins<21>,Library<21>,Security<14>,Python 3 9 0 alpha 1
@anchor{whatsnew/changelog id180}@anchor{125a}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-38631@footnote{https://bugs.python.org/issue38631}: Replace @code{Py_FatalError()} call with a regular
@ref{5af,,RuntimeError} exception in @code{float.__getformat__()}.

@item 
bpo-38639@footnote{https://bugs.python.org/issue38639}: Optimized @ref{f28,,math.floor()}, @ref{f29,,math.ceil()} and
@ref{f2a,,math.trunc()} for floats.

@item 
bpo-38640@footnote{https://bugs.python.org/issue38640}: Fixed a bug in the compiler that was causing to raise in the
presence of break statements and continue statements inside always false
while loops. Patch by Pablo Galindo.

@item 
bpo-38613@footnote{https://bugs.python.org/issue38613}: Optimized some set operations (e.g. @code{|}, @code{^}, and @code{-}) of
@code{dict_keys}. @code{d.keys() | other} was slower than @code{set(d) | other} but
they are almost same performance for now.

@item 
bpo-28029@footnote{https://bugs.python.org/issue28029}: @code{"".replace("", s, n)} now returns @code{s} instead of an empty
string for all non-zero @code{n}.  There are similar changes for
@ref{2f0,,bytes} and @ref{2f1,,bytearray} objects.

@item 
bpo-38535@footnote{https://bugs.python.org/issue38535}: Fixed line numbers and column offsets for AST nodes for calls
without arguments in decorators.

@item 
bpo-38525@footnote{https://bugs.python.org/issue38525}: Fix a segmentation fault when using reverse iterators of empty
@code{dict} objects. Patch by Dong-hee Na and Inada Naoki.

@item 
bpo-38465@footnote{https://bugs.python.org/issue38465}: @ref{2f1,,bytearray}, @ref{25e,,array} and
@ref{511,,mmap} objects allow now to export more than @code{2**31}
buffers at a time.

@item 
bpo-38469@footnote{https://bugs.python.org/issue38469}: Fixed a bug where the scope of named expressions was not being
resolved correctly in the presence of the @emph{global} keyword. Patch by Pablo
Galindo.

@item 
bpo-38437@footnote{https://bugs.python.org/issue38437}: Activate the @code{GC_DEBUG} macro for debug builds of the
interpreter (when @code{Py_DEBUG} is set). Patch by Pablo Galindo.

@item 
bpo-38379@footnote{https://bugs.python.org/issue38379}: When the garbage collector makes a collection in which some
objects resurrect (they are reachable from outside the isolated cycles
after the finalizers have been executed), do not block the collection of
all objects that are still unreachable. Patch by Pablo Galindo and Tim
Peters.

@item 
bpo-38379@footnote{https://bugs.python.org/issue38379}: When cyclic garbage collection (gc) runs finalizers that
resurrect unreachable objects, the current gc run ends, without collecting
any cyclic trash.  However, the statistics reported by @code{collect()} and
@code{get_stats()} claimed that all cyclic trash found was collected, and
that the resurrected objects were collected.   Changed the stats to report
that none were collected.

@item 
bpo-38392@footnote{https://bugs.python.org/issue38392}: In debug mode, @ref{103a,,PyObject_GC_Track()} now calls
@code{tp_traverse()} of the object type to ensure that the object is valid:
test that objects visited by @code{tp_traverse()} are valid.

@item 
bpo-38210@footnote{https://bugs.python.org/issue38210}: Remove unnecessary intersection and update set operation in
dictview with empty set. (Contributed by Dong-hee Na in bpo-38210@footnote{https://bugs.python.org/issue38210}.)

@item 
bpo-38402@footnote{https://bugs.python.org/issue38402}: Check the error from the system’s underlying @code{crypt} or
@code{crypt_r}.

@item 
bpo-37474@footnote{https://bugs.python.org/issue37474}: On FreeBSD, Python no longer calls @code{fedisableexcept()} at
startup to control the floating point control mode. The call became
useless since FreeBSD 6: it became the default mode.

@item 
bpo-38006@footnote{https://bugs.python.org/issue38006}: Fix a bug due to the interaction of weakrefs and the cyclic
garbage collector. We must clear any weakrefs in garbage in order to
prevent their callbacks from executing and causing a crash.

@item 
bpo-38317@footnote{https://bugs.python.org/issue38317}: Fix warnings options priority: @code{PyConfig.warnoptions} has the
highest priority, as stated in the 
@geindex Python Enhancement Proposals; PEP 587
PEP 587@footnote{https://www.python.org/dev/peps/pep-0587}.

@item 
bpo-38310@footnote{https://bugs.python.org/issue38310}: Predict @code{BUILD_MAP_UNPACK_WITH_CALL} -> @code{CALL_FUNCTION_EX}
opcode pairs in the main interpreter loop. Patch by Brandt Bucher.

@item 
bpo-36871@footnote{https://bugs.python.org/issue36871}: Improve error handling for the assert_has_calls and
assert_has_awaits methods of mocks. Fixed a bug where any errors
encountered while binding the expected calls to the mock’s spec were
silently swallowed, leading to misleading error output.

@item 
bpo-11410@footnote{https://bugs.python.org/issue11410}: Better control over symbol visibility is provided through use
of the visibility attributes available in gcc >= 4.0, provided in a
uniform way across POSIX and Windows. The POSIX build files have been
updated to compile with -fvisibility=hidden, minimising exported symbols.

@item 
bpo-38219@footnote{https://bugs.python.org/issue38219}: Optimized the @ref{3b0,,dict} constructor and the
@ref{fc3,,update()} method for the case when the argument is a dict.

@item 
bpo-38236@footnote{https://bugs.python.org/issue38236}: Python now dumps path configuration if it fails to import the
Python codecs of the filesystem and stdio encodings.

@item 
bpo-38013@footnote{https://bugs.python.org/issue38013}: Allow to call @code{async_generator_athrow().throw(...)} even for
non-started async generator helper. It fixes annoying warning at the end
of @ref{3d1,,asyncio.run()} call.

@item 
bpo-38124@footnote{https://bugs.python.org/issue38124}: Fix an off-by-one error in PyState_AddModule that could cause
out-of-bounds memory access.

@item 
bpo-38116@footnote{https://bugs.python.org/issue38116}: The select module is now PEP-384 compliant and no longer has
static state

@item 
bpo-38113@footnote{https://bugs.python.org/issue38113}: ast module updated to PEP-384 and all statics removed

@item 
bpo-38076@footnote{https://bugs.python.org/issue38076}: The struct module is now PEP-384 compatible

@item 
bpo-38075@footnote{https://bugs.python.org/issue38075}: The random module is now PEP-384 compatible

@item 
bpo-38074@footnote{https://bugs.python.org/issue38074}: zlib module made PEP-384 compatible

@item 
bpo-38073@footnote{https://bugs.python.org/issue38073}: Make pwd extension module PEP-384 compatible

@item 
bpo-38072@footnote{https://bugs.python.org/issue38072}: grp module made PEP-384 compatible

@item 
bpo-38069@footnote{https://bugs.python.org/issue38069}: Make _posixsubprocess PEP-384 compatible

@item 
bpo-38071@footnote{https://bugs.python.org/issue38071}: Make termios extension module PEP-384 compatible

@item 
bpo-38005@footnote{https://bugs.python.org/issue38005}: Fixed comparing and creating of InterpreterID and ChannelID.

@item 
bpo-36946@footnote{https://bugs.python.org/issue36946}: Fix possible signed integer overflow when handling slices.
Patch by hongweipeng.

@item 
bpo-37994@footnote{https://bugs.python.org/issue37994}: Fixed silencing arbitrary errors if an attribute lookup fails
in several sites. Only AttributeError should be silenced.

@item 
bpo-8425@footnote{https://bugs.python.org/issue8425}: Optimize set difference_update for the case when the other set
is much larger than the base set.  (Suggested by Evgeny Kapun with code
contributed by Michele Orrù).

@item 
bpo-37966@footnote{https://bugs.python.org/issue37966}: The implementation of @ref{55e,,is_normalized()} has
been greatly sped up on strings that aren’t normalized, by implementing
the full normalization-quick-check algorithm from the Unicode standard.

@item 
bpo-37947@footnote{https://bugs.python.org/issue37947}: Adjust correctly the recursion level in the symtable generation
for named expressions. Patch by Pablo Galindo.

@item 
bpo-37812@footnote{https://bugs.python.org/issue37812}: The @code{CHECK_SMALL_INT} macro used inside
@code{Object/longobject.c} has been replaced with an explicit @code{return}
at each call site.

@item 
bpo-37751@footnote{https://bugs.python.org/issue37751}: Fix @ref{466,,codecs.lookup()} to normalize the encoding name the
same way than @code{encodings.normalize_encoding()}, except that
@ref{466,,codecs.lookup()} also converts the name to lower case.

@item 
bpo-37830@footnote{https://bugs.python.org/issue37830}: Fixed compilation of @ref{5cb,,break} and @ref{4c0,,continue}
in the @ref{4c1,,finally} block when the corresponding @ref{f6d,,try}
block contains @ref{4c6,,return} with a non-constant value.

@item 
bpo-20490@footnote{https://bugs.python.org/issue20490}: Improve import error message for partially initialized module
on circular @code{from} imports - by Anthony Sottile.

@item 
bpo-37840@footnote{https://bugs.python.org/issue37840}: Fix handling of negative indices in
@ref{11a6,,sq_item} of @ref{2f1,,bytearray}. Patch by
Sergey Fedoseev.

@item 
bpo-37802@footnote{https://bugs.python.org/issue37802}: Slightly improve performance of
@ref{125b,,PyLong_FromUnsignedLong()}, @ref{125c,,PyLong_FromUnsignedLongLong()}
and @ref{125d,,PyLong_FromSize_t()}. Patch by Sergey Fedoseev.

@item 
bpo-37409@footnote{https://bugs.python.org/issue37409}: Ensure explicit relative imports from interactive sessions and
scripts (having no parent package) always raise ImportError, rather than
treating the current module as the package. Patch by Ben Lewis.

@item 
bpo-32912@footnote{https://bugs.python.org/issue32912}: Reverted bpo-32912@footnote{https://bugs.python.org/issue32912}: emitting @ref{4c7,,SyntaxWarning} instead
of @ref{1b2,,DeprecationWarning} for invalid escape sequences in string and
bytes literals.

@item 
bpo-37757@footnote{https://bugs.python.org/issue37757}: 
@geindex Python Enhancement Proposals; PEP 572
PEP 572@footnote{https://www.python.org/dev/peps/pep-0572}: As described in the PEP, assignment expressions now
raise @ref{21a,,SyntaxError} when their interaction with comprehension scoping
results in an ambiguous target scope.

The @code{TargetScopeError} subclass originally proposed by the PEP has been
removed in favour of just raising regular syntax errors for the disallowed
cases.

@item 
bpo-36279@footnote{https://bugs.python.org/issue36279}: Fix potential use of uninitialized memory in @ref{8f2,,os.wait3()}.

@item 
bpo-36311@footnote{https://bugs.python.org/issue36311}: Decoding bytes objects larger than 2GiB is faster and no longer
fails when a multibyte characters spans a chunk boundary.

@item 
bpo-34880@footnote{https://bugs.python.org/issue34880}: The @ref{46d,,assert} statement now works properly if the
@ref{46e,,AssertionError} exception is being shadowed. Patch by Zackery Spytz.

@item 
bpo-37340@footnote{https://bugs.python.org/issue37340}: Removed object cache (@code{free_list}) for bound method objects.
Temporary bound method objects are less used than before thanks to the
@code{LOAD_METHOD} opcode and the @code{_PyObject_VectorcallMethod} C API.

@item 
bpo-37648@footnote{https://bugs.python.org/issue37648}: Fixed minor inconsistency in @code{list.__contains__()},
@code{tuple.__contains__()} and a few other places. The collection’s item
is now always at the left and the needle is on the right of @code{==}.

@item 
bpo-37444@footnote{https://bugs.python.org/issue37444}: Update differing exception between @code{builtins.__import__()}
and @ref{c02,,importlib.__import__()}.

@item 
bpo-37619@footnote{https://bugs.python.org/issue37619}: When adding a wrapper descriptor from one class to a different
class (for example, setting @code{__add__ = str.__add__} on an @code{int}
subclass), an exception is correctly raised when the operator is called.

@item 
bpo-37593@footnote{https://bugs.python.org/issue37593}: Swap the positions of the @emph{posonlyargs} and @emph{args} parameters
in the constructor of @code{ast.parameters} nodes.

@item 
bpo-37543@footnote{https://bugs.python.org/issue37543}: Optimized pymalloc for non PGO build.

@item 
bpo-37537@footnote{https://bugs.python.org/issue37537}: Compute allocated pymalloc blocks inside
_Py_GetAllocatedBlocks().  This slows down _Py_GetAllocatedBlocks() but
gives a small speedup to _PyObject_Malloc() and _PyObject_Free().

@item 
bpo-37467@footnote{https://bugs.python.org/issue37467}: Fix @ref{28e,,sys.excepthook()} and @code{PyErr_Display()} if a
filename is a bytes string. For example, for a SyntaxError exception where
the filename attribute is a bytes string.

@item 
bpo-37433@footnote{https://bugs.python.org/issue37433}: Fix @code{SyntaxError} indicator printing too many spaces for
multi-line strings - by Anthony Sottile.

@item 
bpo-37417@footnote{https://bugs.python.org/issue37417}: @code{bytearray.extend()} now correctly handles errors that
arise during iteration. Patch by Brandt Bucher.

@item 
bpo-37414@footnote{https://bugs.python.org/issue37414}: The undocumented @code{sys.callstats()} function has been removed.
Since Python 3.7, it was deprecated and always returned @code{None}. It
required a special build option @code{CALL_PROFILE} which was already removed
in Python 3.7.

@item 
bpo-37392@footnote{https://bugs.python.org/issue37392}: Remove @code{sys.getcheckinterval()} and
@code{sys.setcheckinterval()} functions. They were deprecated since Python
3.2. Use @ref{448,,sys.getswitchinterval()} and @ref{449,,sys.setswitchinterval()}
instead. Remove also @code{check_interval} field of the
@code{PyInterpreterState} structure.

@item 
bpo-37388@footnote{https://bugs.python.org/issue37388}: In development mode and in debug build, @emph{encoding} and @emph{errors}
arguments are now checked on string encoding and decoding operations.
Examples: @ref{1c8,,open()}, @ref{3ba,,str.encode()} and @ref{3bb,,bytes.decode()}.

By default, for best performances, the @emph{errors} argument is only checked
at the first encoding/decoding error, and the @emph{encoding} argument is
sometimes ignored for empty strings.

@item 
bpo-37348@footnote{https://bugs.python.org/issue37348}: Optimized decoding short ASCII string with UTF-8 and ascii
codecs. @code{b"foo".decode()} is about 15% faster.  Patch by Inada Naoki.

@item 
bpo-24214@footnote{https://bugs.python.org/issue24214}: Improved support of the surrogatepass error handler in the
UTF-8 and UTF-16 incremental decoders.

@item 
bpo-37330@footnote{https://bugs.python.org/issue37330}: @ref{1c8,,open()}, @ref{1c9,,io.open()}, @ref{1ca,,codecs.open()} and
@ref{1bc,,fileinput.FileInput} no longer accept @code{'U'} (“universal
newline”) in the file mode. This flag was deprecated since Python 3.3.

@item 
bpo-35224@footnote{https://bugs.python.org/issue35224}: Reverse evaluation order of key: value in dict comprehensions
as proposed in PEP 572. I.e. in @code{@{k: v for ...@}}, @code{k} will be
evaluated before @code{v}.

@item 
bpo-37316@footnote{https://bugs.python.org/issue37316}: Fix the @ref{120d,,PySys_Audit()} call in @ref{511,,mmap.mmap}.

@item 
bpo-37300@footnote{https://bugs.python.org/issue37300}: Remove an unnecessary Py_XINCREF in classobject.c.

@item 
bpo-37269@footnote{https://bugs.python.org/issue37269}: Fix a bug in the peephole optimizer that was not treating
correctly constant conditions with binary operators. Patch by Pablo
Galindo.

@item 
bpo-20443@footnote{https://bugs.python.org/issue20443}: Python now gets the absolute path of the script filename
specified on the command line (ex: “python3 script.py”): the __file__
attribute of the __main__ module and sys.path[0] become an absolute path,
rather than a relative path.

@item 
bpo-37257@footnote{https://bugs.python.org/issue37257}: Python’s small object allocator (@code{obmalloc.c}) now allows (no
more than) one empty arena to remain available for immediate reuse,
without returning it to the OS.  This prevents thrashing in simple loops
where an arena could be created and destroyed anew on each iteration.

@item 
bpo-37231@footnote{https://bugs.python.org/issue37231}: The dispatching of type slots to special methods (for example
calling @code{__mul__} when doing @code{x * y}) has been made faster.

@item 
bpo-36974@footnote{https://bugs.python.org/issue36974}: Implemented separate vectorcall functions for every calling
convention of builtin functions and methods. This improves performance for
calls.

@item 
bpo-37213@footnote{https://bugs.python.org/issue37213}: Handle correctly negative line offsets in the peephole
optimizer. Patch by Pablo Galindo.

@item 
bpo-37219@footnote{https://bugs.python.org/issue37219}: Remove erroneous optimization for empty set differences.

@item 
bpo-15913@footnote{https://bugs.python.org/issue15913}: Implement @ref{1df,,PyBuffer_SizeFromFormat()} function
(previously documented but not implemented): call @ref{125e,,struct.calcsize()}.
Patch by Joannah Nanjekye.

@item 
bpo-36922@footnote{https://bugs.python.org/issue36922}: Slot functions optimize any callable with
@code{Py_TPFLAGS_METHOD_DESCRIPTOR} instead of only instances of
@code{function}.

@item 
bpo-36974@footnote{https://bugs.python.org/issue36974}: The slot @code{tp_vectorcall_offset} is inherited unconditionally
to support @code{super().__call__()} when the base class uses vectorcall.

@item 
bpo-37160@footnote{https://bugs.python.org/issue37160}: @ref{54f,,threading.get_native_id()} now also supports NetBSD.

@item 
bpo-37077@footnote{https://bugs.python.org/issue37077}: Add @ref{54f,,threading.get_native_id()} support for AIX. Patch by
M. Felt

@item 
bpo-36781@footnote{https://bugs.python.org/issue36781}: @ref{50d,,sum()} has been optimized for boolean values.

@item 
bpo-34556@footnote{https://bugs.python.org/issue34556}: Add @code{--upgrade-deps} to venv module. Patch by Cooper Ry Lees

@item 
bpo-20523@footnote{https://bugs.python.org/issue20523}: @code{pdb.Pdb} supports ~/.pdbrc in Windows 7. Patch by Tim Hopper
and Dan Lidral-Porter.

@item 
bpo-35551@footnote{https://bugs.python.org/issue35551}: Updated encodings: - Removed the “tis260” encoding, which was
an alias for the nonexistent “tactis” codec. - Added “mac_centeuro” as an
alias for the mac_latin2 encoding.

@item 
bpo-19072@footnote{https://bugs.python.org/issue19072}: The @ref{254,,classmethod} decorator can now wrap other
descriptors such as property objects.  Adapted from a patch written by
Graham Dumpleton.

@item 
bpo-27575@footnote{https://bugs.python.org/issue27575}: Improve speed of dictview intersection by directly using set
intersection logic. Patch by David Su.

@item 
bpo-30773@footnote{https://bugs.python.org/issue30773}: Prohibit parallel running of aclose() / asend() / athrow(). Fix
ag_running to reflect the actual running status of the AG.
@end itemize

@node Library<21>,Documentation<22>,Core and Builtins<21>,Python 3 9 0 alpha 1
@anchor{whatsnew/changelog id181}@anchor{125f}
@subsubsection Library



@itemize -

@item 
bpo-36589@footnote{https://bugs.python.org/issue36589}: The @ref{94b,,curses.update_lines_cols()} function now returns
@code{None} instead of @code{1} on success.

@item 
bpo-38807@footnote{https://bugs.python.org/issue38807}: Update @ref{162,,TypeError} messages for @ref{1260,,os.path.join()} to
include @ref{795,,os.PathLike} objects as acceptable input types.

@item 
bpo-38724@footnote{https://bugs.python.org/issue38724}: Add a repr for @code{subprocess.Popen} objects. Patch by Andrey
Doroschenko.

@item 
bpo-38786@footnote{https://bugs.python.org/issue38786}: pydoc now recognizes and parses HTTPS URLs. Patch by python273.

@item 
bpo-38785@footnote{https://bugs.python.org/issue38785}: Prevent asyncio from crashing if parent @code{__init__} is not
called from a constructor of object derived from @code{asyncio.Future}.

@item 
bpo-38723@footnote{https://bugs.python.org/issue38723}: @ref{c4,,pdb} now uses @ref{11f6,,io.open_code()} to trigger auditing
events.

@item 
bpo-27805@footnote{https://bugs.python.org/issue27805}: Allow opening pipes and other non-seekable files in append mode
with @ref{1c8,,open()}.

@item 
bpo-38438@footnote{https://bugs.python.org/issue38438}: Simplify the @ref{5,,argparse} usage message for @code{nargs="*"}.

@item 
bpo-38761@footnote{https://bugs.python.org/issue38761}: WeakSet is now registered as a collections.abc.MutableSet.

@item 
bpo-38716@footnote{https://bugs.python.org/issue38716}: logging: change RotatingHandler namer and rotator to
class-level attributes. This stops __init__ from setting them to None in
the case where a subclass defines them with eponymous methods.

@item 
bpo-38713@footnote{https://bugs.python.org/issue38713}: Add @ref{40e,,os.P_PIDFD} constant, which may be passed to
@ref{8f5,,os.waitid()} to wait on a Linux process file descriptor.

@item 
bpo-38692@footnote{https://bugs.python.org/issue38692}: Add @ref{3d2,,asyncio.PidfdChildWatcher}, a Linux-specific child
watcher implementation that polls process file descriptors.

@item 
bpo-38692@footnote{https://bugs.python.org/issue38692}: Expose the Linux @code{pidfd_open} syscall as
@ref{3a6,,os.pidfd_open()}.

@item 
bpo-38602@footnote{https://bugs.python.org/issue38602}: Added constants @code{F_OFD_GETLK},
@code{F_OFD_SETLK} and @code{F_OFD_SETLKW} to the
@ref{79,,fcntl} module. Patch by Dong-hee Na.

@item 
bpo-38334@footnote{https://bugs.python.org/issue38334}: Fixed seeking backward on an encrypted
@code{zipfile.ZipExtFile}.

@item 
bpo-38312@footnote{https://bugs.python.org/issue38312}: Add @ref{3dc,,curses.get_escdelay()}, @ref{3dd,,curses.set_escdelay()},
@ref{3de,,curses.get_tabsize()}, and @ref{3df,,curses.set_tabsize()} functions - by
Anthony Sottile.

@item 
bpo-38586@footnote{https://bugs.python.org/issue38586}: Now @ref{678,,fileConfig()} correctly sets the .name
of handlers loaded.

@item 
bpo-38565@footnote{https://bugs.python.org/issue38565}: Add new cache_parameters() method for functools.lru_cache() to
better support pickling.

@item 
bpo-34679@footnote{https://bugs.python.org/issue34679}: asynci.ProactorEventLoop.close() now only calls
signal.set_wakeup_fd() in the main thread.

@item 
bpo-31202@footnote{https://bugs.python.org/issue31202}: The case the result of @code{pathlib.WindowsPath.glob()} matches
now the case of the pattern for literal parts.

@item 
bpo-36321@footnote{https://bugs.python.org/issue36321}: Remove misspelled attribute.  The 3.8 changelog noted that this
would be removed in 3.9.

@item 
bpo-38521@footnote{https://bugs.python.org/issue38521}: Fixed erroneous equality comparison in statistics.NormalDist().

@item 
bpo-38493@footnote{https://bugs.python.org/issue38493}: Added @ref{40c,,CLD_KILLED} and @ref{40d,,CLD_STOPPED} for
@code{si_code}. Patch by Dong-hee Na.

@item 
bpo-38478@footnote{https://bugs.python.org/issue38478}: Fixed a bug in @code{inspect.signature.bind()} that was causing
it to fail when handling a keyword argument with same name as
positional-only parameter. Patch by Pablo Galindo.

@item 
bpo-33604@footnote{https://bugs.python.org/issue33604}: Fixed @ref{ac7,,hmac.new} and @code{hmac.HMAC} to raise TypeError instead of
ValueError when the digestmod parameter, now required in 3.8, is omitted.
Also clarified the hmac module documentation and docstrings.

@item 
bpo-38378@footnote{https://bugs.python.org/issue38378}: Parameters @emph{out} and @emph{in} of @ref{62c,,os.sendfile()} was renamed to
@emph{out_fd} and @emph{in_fd}.

@item 
bpo-38417@footnote{https://bugs.python.org/issue38417}: Added support for setting the umask in the child process to the
subprocess module on POSIX systems.

@item 
bpo-38449@footnote{https://bugs.python.org/issue38449}: Revert GH-15522, which introduces a regression in
@ref{123d,,mimetypes.guess_type()} due to improper handling of filenames as
urls.

@item 
bpo-38431@footnote{https://bugs.python.org/issue38431}: Fix @code{__repr__} method for @code{dataclasses.InitVar} to
support typing objects, patch by Samuel Colvin.

@item 
bpo-38109@footnote{https://bugs.python.org/issue38109}: Add missing @ref{b3c,,stat.S_IFDOOR}, @ref{b3d,,stat.S_IFPORT},
@ref{b3e,,stat.S_IFWHT}, @ref{1261,,stat.S_ISDOOR()}, @ref{1262,,stat.S_ISPORT()}, and
@ref{1263,,stat.S_ISWHT()} values to the Python implementation of @ref{ef,,stat}.

@item 
bpo-38422@footnote{https://bugs.python.org/issue38422}: Clarify docstrings of pathlib suffix(es)

@item 
bpo-38405@footnote{https://bugs.python.org/issue38405}: Nested subclasses of @ref{457,,typing.NamedTuple} are now
pickleable.

@item 
bpo-38332@footnote{https://bugs.python.org/issue38332}: Prevent @ref{5bb,,KeyError} thrown by @code{_encoded_words.decode()}
when given an encoded-word with invalid content-type encoding from
propagating all the way to @code{email.message.get()}.

@item 
bpo-38371@footnote{https://bugs.python.org/issue38371}: Deprecated the @code{split()} method in
@code{_tkinter.TkappType} in favour of the @code{splitlist()} method which
has more consistent and predicable behavior.

@item 
bpo-38341@footnote{https://bugs.python.org/issue38341}: Add @ref{1264,,smtplib.SMTPNotSupportedError} to the @ref{e8,,smtplib}
exported names.

@item 
bpo-38319@footnote{https://bugs.python.org/issue38319}: sendfile() used in socket and shutil modules was raising
OverflowError for files >= 2GiB on 32-bit architectures.  (patch by
Giampaolo Rodola)

@item 
bpo-38242@footnote{https://bugs.python.org/issue38242}: Revert the new asyncio Streams API

@item 
bpo-13153@footnote{https://bugs.python.org/issue13153}: OS native encoding is now used for converting between Python
strings and Tcl objects.  This allows to display, copy and paste to
clipboard emoji and other non-BMP characters.  Converting strings from Tcl
to Python and back now never fails (except MemoryError).

@item 
bpo-38019@footnote{https://bugs.python.org/issue38019}: Correctly handle pause/resume reading of closed asyncio unix
pipe.

@item 
bpo-38163@footnote{https://bugs.python.org/issue38163}: Child mocks will now detect their type as either synchronous or
asynchronous, asynchronous child mocks will be AsyncMocks and synchronous
child mocks will be either MagicMock or Mock (depending on their parent
type).

@item 
bpo-38161@footnote{https://bugs.python.org/issue38161}: Removes _AwaitEvent from AsyncMock.

@item 
bpo-38216@footnote{https://bugs.python.org/issue38216}: Allow the rare code that wants to send invalid http requests
from the @ref{8f,,http.client} library a way to do so.  The fixes for bpo-30458@footnote{https://bugs.python.org/issue30458}
led to breakage for some projects that were relying on this ability to
test their own behavior in the face of bad requests.

@item 
bpo-28286@footnote{https://bugs.python.org/issue28286}: Deprecate opening @ref{43f,,GzipFile} for writing
implicitly.  Always specify the @emph{mode} argument for writing.

@item 
bpo-38108@footnote{https://bugs.python.org/issue38108}: Any synchronous magic methods on an AsyncMock now return a
MagicMock. Any asynchronous magic methods on a MagicMock now return an
AsyncMock.

@item 
bpo-38265@footnote{https://bugs.python.org/issue38265}: Update the @emph{length} parameter of @ref{688,,os.pread()} to accept
@code{Py_ssize_t} instead of int.

@item 
bpo-38112@footnote{https://bugs.python.org/issue38112}: @ref{1f,,compileall} has a higher default recursion limit and new
command-line arguments for path manipulation, symlinks handling, and
multiple optimization levels.

@item 
bpo-38248@footnote{https://bugs.python.org/issue38248}: asyncio: Fix inconsistent immediate Task cancellation

@item 
bpo-38237@footnote{https://bugs.python.org/issue38237}: The arguments for the builtin pow function are more
descriptive. They can now also be passed in as keywords.

@item 
bpo-34002@footnote{https://bugs.python.org/issue34002}: Improve efficiency in parts of email package by changing
while-pop to a for loop, using isdisjoint instead of set intersections.

@item 
bpo-38191@footnote{https://bugs.python.org/issue38191}: Constructors of @ref{457,,NamedTuple} and
@ref{2de,,TypedDict} types now accept arbitrary keyword argument
names, including “cls”, “self”, “typename”, “_typename”, “fields” and
“_fields”.

@item 
bpo-38155@footnote{https://bugs.python.org/issue38155}: Add @code{__all__} to  @ref{2f,,datetime}. Patch by Tahia Khan.

@item 
bpo-38185@footnote{https://bugs.python.org/issue38185}: Fixed case-insensitive string comparison in
@ref{9dd,,sqlite3.Row} indexing.

@item 
bpo-38136@footnote{https://bugs.python.org/issue38136}: Changes AsyncMock call count and await count to be two
different counters. Now await count only counts when a coroutine has been
awaited, not when it has been called, and vice-versa. Update the
documentation around this.

@item 
bpo-37828@footnote{https://bugs.python.org/issue37828}: Fix default mock name in
@ref{843,,unittest.mock.Mock.assert_called()} exceptions. Patch by Abraham
Toriz Cruz.

@item 
bpo-38175@footnote{https://bugs.python.org/issue38175}: Fix a memory leak in comparison of @ref{9dd,,sqlite3.Row}
objects.

@item 
bpo-33936@footnote{https://bugs.python.org/issue33936}: _hashlib no longer calls obsolete OpenSSL initialization
function with OpenSSL 1.1.0+.

@item 
bpo-34706@footnote{https://bugs.python.org/issue34706}: Preserve subclassing in inspect.Signature.from_callable.

@item 
bpo-38153@footnote{https://bugs.python.org/issue38153}: Names of hashing algorithms from OpenSSL are now normalized to
follow Python’s naming conventions. For example OpenSSL uses sha3-512
instead of sha3_512 or blake2b512 instead of blake2b.

@item 
bpo-38115@footnote{https://bugs.python.org/issue38115}: Fix a bug in dis.findlinestarts() where it would return invalid
bytecode offsets. Document that a code object’s co_lnotab can contain
invalid bytecode offsets.

@item 
bpo-38148@footnote{https://bugs.python.org/issue38148}: Add slots to @ref{9,,asyncio} transport classes, which can reduce
memory usage.

@item 
bpo-38142@footnote{https://bugs.python.org/issue38142}: The _hashlib OpenSSL wrapper extension module is now PEP-384
compliant.

@item 
bpo-9216@footnote{https://bugs.python.org/issue9216}: hashlib constructors now support usedforsecurity flag to signal
that a hashing algorithm is not used in a security context.

@item 
bpo-36991@footnote{https://bugs.python.org/issue36991}: Fixes a potential incorrect AttributeError exception escaping
ZipFile.extract() in some unsupported input error situations.

@item 
bpo-38134@footnote{https://bugs.python.org/issue38134}: Remove obsolete copy of PBKDF2_HMAC_fast. All supported OpenSSL
versions contain a fast implementation.

@item 
bpo-38132@footnote{https://bugs.python.org/issue38132}: The OpenSSL hashlib wrapper uses a simpler implementation.
Several Macros and pointless caches are gone. The hash name now comes from
OpenSSL’s EVP. The algorithm name stays the same, except it is now always
lower case.

@item 
bpo-38008@footnote{https://bugs.python.org/issue38008}: Fix parent class check in protocols to correctly identify the
module that provides a builtin protocol, instead of assuming they all come
from the @ref{1d,,collections.abc} module

@item 
bpo-34037@footnote{https://bugs.python.org/issue34037}: For @ref{9,,asyncio}, add a new coroutine
@code{loop.shutdown_default_executor()}. The new coroutine provides an API
to schedule an executor shutdown that waits on the threadpool to finish
closing. Also, @ref{3d1,,asyncio.run()} has been updated to utilize the new
coroutine. Patch by Kyle Stanley.

@item 
bpo-37405@footnote{https://bugs.python.org/issue37405}: Fixed regression bug for socket.getsockname() for non-CAN_ISOTP
AF_CAN address family sockets by returning a 1-tuple instead of string.

@item 
bpo-38121@footnote{https://bugs.python.org/issue38121}: Update parameter names on functions in importlib.metadata
matching the changes in the 0.22 release of importlib_metadata.

@item 
bpo-38110@footnote{https://bugs.python.org/issue38110}: The os.closewalk() implementation now uses the libc fdwalk()
API on platforms where it is available.

@item 
bpo-38093@footnote{https://bugs.python.org/issue38093}: Fixes AsyncMock so it doesn’t crash when used with
AsyncContextManagers or AsyncIterators.

@item 
bpo-37488@footnote{https://bugs.python.org/issue37488}: Add warning to @code{datetime.utctimetuple()},
@code{datetime.utcnow()} and @code{datetime.utcfromtimestamp()} .

@item 
bpo-35640@footnote{https://bugs.python.org/issue35640}: Allow passing a @ref{63d,,path-like object} as @code{directory}
argument to the @ref{666,,http.server.SimpleHTTPRequestHandler} class. Patch
by Géry Ogam.

@item 
bpo-38086@footnote{https://bugs.python.org/issue38086}: Update importlib.metadata with changes from importlib_metadata 0.21@footnote{https://gitlab.com/python-devs/importlib_metadata/blob/0.21/importlib_metadata/docs/changelog.rst}.

@item 
bpo-37251@footnote{https://bugs.python.org/issue37251}: Remove @code{__code__} check in AsyncMock that incorrectly evaluated
function specs as async objects but failed to evaluate classes with
@code{__await__} but no @code{__code__} attribute defined as async objects.

@item 
bpo-38037@footnote{https://bugs.python.org/issue38037}: Fix reference counters in the @ref{e5,,signal} module.

@item 
bpo-38066@footnote{https://bugs.python.org/issue38066}: Hide internal asyncio.Stream methods: feed_eof(), feed_data(),
set_exception() and set_transport().

@item 
bpo-38059@footnote{https://bugs.python.org/issue38059}: inspect.py now uses sys.exit() instead of exit()

@item 
bpo-38049@footnote{https://bugs.python.org/issue38049}: Added command-line interface for the @ref{7,,ast} module.

@item 
bpo-37953@footnote{https://bugs.python.org/issue37953}: In @ref{120,,typing}, improved the @code{__hash__} and @code{__eq__}
methods for @code{ForwardReferences}.

@item 
bpo-38026@footnote{https://bugs.python.org/issue38026}: Fixed @ref{de8,,inspect.getattr_static()} used @code{isinstance} while
it should avoid dynamic lookup.

@item 
bpo-35923@footnote{https://bugs.python.org/issue35923}: Update @ref{7f7,,importlib.machinery.BuiltinImporter} to use
@code{loader._ORIGIN} instead of a hardcoded value. Patch by Dong-hee Na.

@item 
bpo-38010@footnote{https://bugs.python.org/issue38010}: In @code{importlib.metadata} sync with @code{importlib_metadata}
0.20, clarifying behavior of @code{files()} and fixing issue where only one
requirement was returned for @code{requires()} on @code{dist-info} packages.

@item 
bpo-38006@footnote{https://bugs.python.org/issue38006}: weakref.WeakValueDictionary defines a local remove() function
used as callback for weak references. This function was created with a
closure. Modify the implementation to avoid the closure.

@item 
bpo-37995@footnote{https://bugs.python.org/issue37995}: Added the @emph{indent} option to @ref{3cb,,ast.dump()} which allows it
to produce a multiline indented output.

@item 
bpo-34410@footnote{https://bugs.python.org/issue34410}: Fixed a crash in the @code{tee()} iterator when re-enter it.
RuntimeError is now raised in this case.

@item 
bpo-37140@footnote{https://bugs.python.org/issue37140}: Fix a ctypes regression of Python 3.8. When a ctypes.Structure
is passed by copy to a function, ctypes internals created a temporary
object which had the side effect of calling the structure finalizer
(__del__) twice. The Python semantics requires a finalizer to be called
exactly once. Fix ctypes internals to no longer call the finalizer twice.

@item 
bpo-37587@footnote{https://bugs.python.org/issue37587}: @code{_json.scanstring} is now up to 3x faster when there are many
backslash escaped characters in the JSON string.

@item 
bpo-37834@footnote{https://bugs.python.org/issue37834}: Prevent shutil.rmtree exception when built on non-Windows
system without fd system call support, like older versions of macOS.

@item 
bpo-10978@footnote{https://bugs.python.org/issue10978}: Semaphores and BoundedSemaphores can now release more than one
waiting thread at a time.

@item 
bpo-37972@footnote{https://bugs.python.org/issue37972}: Subscripts to the @ref{1265,,unittest.mock.call} objects now receive the
same chaining mechanism as any other custom attributes, so that the
following usage no longer raises a @ref{162,,TypeError}:

call().foo().__getitem__(‘bar’)

Patch by blhsing

@item 
bpo-37965@footnote{https://bugs.python.org/issue37965}: Fix C compiler warning caused by
distutils.ccompiler.CCompiler.has_function.

@item 
bpo-37964@footnote{https://bugs.python.org/issue37964}: Add @code{F_GETPATH} command to @ref{79,,fcntl}.

@item 
bpo-37960@footnote{https://bugs.python.org/issue37960}: @code{repr()} of buffered and text streams now silences only
expected exceptions when get the value of “name” and “mode” attributes.

@item 
bpo-37961@footnote{https://bugs.python.org/issue37961}: Add a @code{total_nframe} field to the traces collected by the
tracemalloc module. This field indicates the original number of frames
before it was truncated.

@item 
bpo-37951@footnote{https://bugs.python.org/issue37951}: Most features of the subprocess module now work again in
subinterpreters. Only @emph{preexec_fn} is restricted in subinterpreters.

@item 
bpo-36205@footnote{https://bugs.python.org/issue36205}: Fix the rusage implementation of time.process_time() to
correctly report the sum of the system and user CPU time.

@item 
bpo-37950@footnote{https://bugs.python.org/issue37950}: Fix @ref{3cb,,ast.dump()} when call with incompletely initialized
node.

@item 
bpo-34679@footnote{https://bugs.python.org/issue34679}: Restores instantiation of Windows IOCP event loops from the
non-main thread.

@item 
bpo-36917@footnote{https://bugs.python.org/issue36917}: Add default implementation of the
@code{ast.NodeVisitor.visit_Constant()} method which emits a deprecation
warning and calls corresponding methody @code{visit_Num()}, @code{visit_Str()},
etc.

@item 
bpo-37798@footnote{https://bugs.python.org/issue37798}: Update test_statistics.py to verify that the statistics module
works well for both C and Python implementations. Patch by Dong-hee Na

@item 
bpo-26589@footnote{https://bugs.python.org/issue26589}: Added a new status code to the http module: 451
UNAVAILABLE_FOR_LEGAL_REASONS

@item 
bpo-37915@footnote{https://bugs.python.org/issue37915}: Fix a segmentation fault that appeared when comparing instances
of @code{datetime.timezone} and @code{datetime.tzinfo} objects. Patch by Pablo
Galindo.

@item 
bpo-32554@footnote{https://bugs.python.org/issue32554}: Deprecate having random.seed() call hash on arbitrary types.

@item 
bpo-9938@footnote{https://bugs.python.org/issue9938}: Add optional keyword argument @code{exit_on_error} for
@code{ArgumentParser}.

@item 
bpo-37851@footnote{https://bugs.python.org/issue37851}: The @ref{78,,faulthandler} module no longer allocates its
alternative stack at Python startup. Now the stack is only allocated at
the first faulthandler usage.

@item 
bpo-32793@footnote{https://bugs.python.org/issue32793}: Fix a duplicated debug message when
@ref{1266,,smtplib.SMTP.connect()} is called.

@item 
bpo-37885@footnote{https://bugs.python.org/issue37885}: venv: Don’t generate unset variable warning on deactivate.

@item 
bpo-37868@footnote{https://bugs.python.org/issue37868}: Fix dataclasses.is_dataclass when given an instance that never
raises AttributeError in __getattr__.  That is, an object that returns
something for __dataclass_fields__ even if it’s not a dataclass.

@item 
bpo-37811@footnote{https://bugs.python.org/issue37811}: Fix @code{socket} module’s @code{socket.connect(address)} function
being unable to establish connection in case of interrupted system call.
The problem was observed on all OSes which @code{poll(2)} system call can
take only non-negative integers and -1 as a timeout value.

@item 
bpo-37863@footnote{https://bugs.python.org/issue37863}: Optimizations for Fraction.__hash__ suggested by Tim Peters.

@item 
bpo-21131@footnote{https://bugs.python.org/issue21131}: Fix @code{faulthandler.register(chain=True)} stack. faulthandler
now allocates a dedicated stack of @code{SIGSTKSZ*2} bytes, instead of just
@code{SIGSTKSZ} bytes. Calling the previous signal handler in faulthandler
signal handler uses more than @code{SIGSTKSZ} bytes of stack memory on some
platforms.

@item 
bpo-37798@footnote{https://bugs.python.org/issue37798}: Add C fastpath for statistics.NormalDist.inv_cdf() Patch by
Dong-hee Na

@item 
bpo-37804@footnote{https://bugs.python.org/issue37804}: Remove the deprecated method @code{threading.Thread.isAlive()}.
Patch by Dong-hee Na.

@item 
bpo-37819@footnote{https://bugs.python.org/issue37819}: Add Fraction.as_integer_ratio() to match the corresponding
methods in bool, int, float, and decimal.

@item 
bpo-14465@footnote{https://bugs.python.org/issue14465}: Add an xml.etree.ElementTree.indent() function for
pretty-printing XML trees. Contributed by Stefan Behnel.

@item 
bpo-37810@footnote{https://bugs.python.org/issue37810}: Fix @ref{35,,difflib} @code{?} hint in diff output when dealing with
tabs.  Patch by Anthony Sottile.

@item 
bpo-37772@footnote{https://bugs.python.org/issue37772}: In @code{zipfile.Path}, when adding implicit dirs, ensure that
ancestral directories are added and that duplicates are excluded.

@item 
bpo-18578@footnote{https://bugs.python.org/issue18578}: Renamed and documented @code{test.bytecode_helper} as
@ref{101,,test.support.bytecode_helper}. Patch by Joannah Nanjekye.

@item 
bpo-37785@footnote{https://bugs.python.org/issue37785}: Fix xgettext warnings in @ref{5,,argparse}.

@item 
bpo-34488@footnote{https://bugs.python.org/issue34488}: @code{writelines()} method of @ref{a15,,io.BytesIO} is now
slightly faster when many small lines are passed. Patch by Sergey
Fedoseev.

@item 
bpo-37449@footnote{https://bugs.python.org/issue37449}: @ref{75,,ensurepip} now uses @ref{1267,,importlib.resources.read_binary()} to
read data instead of @ref{1268,,pkgutil.get_data()}. Patch by Joannah Nanjekye.

@item 
bpo-28292@footnote{https://bugs.python.org/issue28292}: Mark calendar.py helper functions as being private.  The
follows PEP 8 guidance to maintain the style conventions in the module and
it addresses a known case of user confusion.

@item 
bpo-18049@footnote{https://bugs.python.org/issue18049}: Add definition of THREAD_STACK_SIZE for AIX in
Python/thread_pthread.h The default thread stacksize caused crashes with
the default recursion limit Patch by M Felt

@item 
bpo-37742@footnote{https://bugs.python.org/issue37742}: The logging.getLogger() API now returns the root logger when
passed the name ‘root’, whereas previously it returned a non-root logger
named ‘root’. This could affect cases where user code explicitly wants a
non-root logger named ‘root’, or instantiates a logger using
logging.getLogger(__name__) in some top-level module called ‘root.py’.

@item 
bpo-37738@footnote{https://bugs.python.org/issue37738}: Fix the implementation of curses @code{addch(str, color_pair)}:
pass the color pair to @code{setcchar()}, instead of always passing 0 as the
color pair.

@item 
bpo-37723@footnote{https://bugs.python.org/issue37723}: Fix performance regression on regular expression parsing with
huge character sets. Patch by Yann Vaginay.

@item 
bpo-35943@footnote{https://bugs.python.org/issue35943}: The function @ref{6e7,,PyImport_GetModule()} now ensures any
module it returns is fully initialized. Patch by Joannah Nanjekye.

@item 
bpo-32178@footnote{https://bugs.python.org/issue32178}: Fix IndexError in @ref{64,,email} package when trying to parse
invalid address fields starting with @code{:}.

@item 
bpo-37268@footnote{https://bugs.python.org/issue37268}: The @code{parser} module is deprecated and will be removed in
future versions of Python.

@item 
bpo-11953@footnote{https://bugs.python.org/issue11953}: Completing WSA* error codes in @ref{ea,,socket}.

@item 
bpo-37685@footnote{https://bugs.python.org/issue37685}: Fixed comparisons of @ref{4c8,,datetime.timedelta} and
@ref{c51,,datetime.timezone}.

@item 
bpo-37697@footnote{https://bugs.python.org/issue37697}: Synchronize @code{importlib.metadata} with importlib_metadata 0.19@footnote{https://gitlab.com/python-devs/importlib_metadata/-/milestones/20},
improving handling of EGG-INFO files and fixing a crash when entry point
names contained colons.

@item 
bpo-37695@footnote{https://bugs.python.org/issue37695}: Correct @ref{c30,,curses.unget_wch()} error message.  Patch by
Anthony Sottile.

@item 
bpo-37689@footnote{https://bugs.python.org/issue37689}: Add @code{is_relative_to()} in @code{PurePath} to determine
whether or not one path is relative to another.

@item 
bpo-29553@footnote{https://bugs.python.org/issue29553}: Fixed @ref{1269,,argparse.ArgumentParser.format_usage()} for mutually
exclusive groups. Patch by Andrew Nester.

@item 
bpo-37691@footnote{https://bugs.python.org/issue37691}: Let math.dist() accept coordinates as sequences (or iterables)
rather than just tuples.

@item 
bpo-37685@footnote{https://bugs.python.org/issue37685}: Fixed @code{__eq__}, @code{__lt__} etc implementations in some
classes. They now return @ref{24c,,NotImplemented} for unsupported type of
the other operand. This allows the other operand to play role (for example
the equality comparison with @ref{126a,,ANY} will return
@code{True}).

@item 
bpo-37354@footnote{https://bugs.python.org/issue37354}: Make Activate.ps1 Powershell script static to allow for signing
it.

@item 
bpo-37664@footnote{https://bugs.python.org/issue37664}: Update wheels bundled with ensurepip (pip 19.2.3 and setuptools
41.2.0)

@item 
bpo-37663@footnote{https://bugs.python.org/issue37663}: Bring consistency to venv shell activation scripts by always
using __VENV_PROMPT__.

@item 
bpo-37642@footnote{https://bugs.python.org/issue37642}: Allowed the pure Python implementation of
@ref{c51,,datetime.timezone} to represent sub-minute offsets close to
minimum and maximum boundaries, specifically in the ranges (23:59, 24:00)
and (-23:59, 24:00). Patch by Ngalim Siregar

@item 
bpo-36161@footnote{https://bugs.python.org/issue36161}: In @ref{cc,,posix}, use @code{ttyname_r} instead of @code{ttyname} for
thread safety.

@item 
bpo-36324@footnote{https://bugs.python.org/issue36324}: Make internal attributes for statistics.NormalDist() private.

@item 
bpo-37555@footnote{https://bugs.python.org/issue37555}: Fix @code{NonCallableMock._call_matcher} returning tuple instead of
@code{_Call} object when @code{self._spec_signature} exists. Patch by Elizabeth
Uselton

@item 
bpo-29446@footnote{https://bugs.python.org/issue29446}: Make @code{from tkinter import *} import only the expected objects.

@item 
bpo-16970@footnote{https://bugs.python.org/issue16970}: Adding a value error when an invalid value in passed to nargs
Patch by Robert Leenders

@item 
bpo-34443@footnote{https://bugs.python.org/issue34443}: Exceptions from @ref{76,,enum} now use the @code{__qualname} of the
enum class in the exception message instead of the @code{__name__}.

@item 
bpo-37491@footnote{https://bugs.python.org/issue37491}: Fix @code{IndexError} when parsing email headers with unexpectedly
ending bare-quoted string value. Patch by Abhilash Raj.

@item 
bpo-37587@footnote{https://bugs.python.org/issue37587}: Make json.loads faster for long strings. (Patch by Marco
Paolini)

@item 
bpo-18378@footnote{https://bugs.python.org/issue18378}: Recognize “UTF-8” as a valid value for LC_CTYPE in
locale._parse_localename.

@item 
bpo-37579@footnote{https://bugs.python.org/issue37579}: Return @ref{24c,,NotImplemented} in Python implementation of
@code{__eq__} for @ref{4c8,,timedelta} and @ref{364,,time}
when the other object being compared is not of the same type to match C
implementation. Patch by Karthikeyan Singaravelan.

@item 
bpo-21478@footnote{https://bugs.python.org/issue21478}: Record calls to parent when autospecced object is attached to a
mock using @code{unittest.mock.attach_mock()}. Patch by Karthikeyan
Singaravelan.

@item 
bpo-37531@footnote{https://bugs.python.org/issue37531}: “python3 -m test -jN –timeout=TIMEOUT” now kills a worker
process if it runs longer than @emph{TIMEOUT} seconds.

@item 
bpo-37482@footnote{https://bugs.python.org/issue37482}: Fix serialization of display name in originator or destination
address fields with both encoded words and special chars.

@item 
bpo-36993@footnote{https://bugs.python.org/issue36993}: Improve error reporting for corrupt zip files with bad zip64
extra data. Patch by Daniel Hillier.

@item 
bpo-37502@footnote{https://bugs.python.org/issue37502}: pickle.loads() no longer raises TypeError when the buffers
argument is set to None

@item 
bpo-37520@footnote{https://bugs.python.org/issue37520}: Correct behavior for zipfile.Path.parent when the path object
identifies a subdirectory.

@item 
bpo-18374@footnote{https://bugs.python.org/issue18374}: Fix the @code{.col_offset} attribute of nested @ref{126b,,ast.BinOp}
instances which had a too large value in some situations.

@item 
bpo-37424@footnote{https://bugs.python.org/issue37424}: Fixes a possible hang when using a timeout on
@ref{6b7,,subprocess.run()} while capturing output.  If the child process spawned
its own children or otherwise connected its stdout or stderr handles with
another process, we could hang after the timeout was reached and our child
was killed when attempting to read final output from the pipes.

@item 
bpo-37421@footnote{https://bugs.python.org/issue37421}: Fix @code{multiprocessing.util.get_temp_dir()} finalizer: clear
also the ‘tempdir’ configuration of the current process, so next call to
@code{get_temp_dir()} will create a new temporary directory, rather than
reusing the removed temporary directory.

@item 
bpo-37481@footnote{https://bugs.python.org/issue37481}: The distutils @code{bdist_wininst} command is deprecated in Python
3.8, use @code{bdist_wheel} (wheel packages) instead.

@item 
bpo-37479@footnote{https://bugs.python.org/issue37479}: When @code{Enum.__str__} is overridden in a derived class, the
override will be used by @code{Enum.__format__} regardless of whether mixin
classes are present.

@item 
bpo-37440@footnote{https://bugs.python.org/issue37440}: http.client now enables TLS 1.3 post-handshake authentication
for default context or if a cert_file is passed to HTTPSConnection.

@item 
bpo-37437@footnote{https://bugs.python.org/issue37437}: Update vendorized expat version to 2.2.7.

@item 
bpo-37428@footnote{https://bugs.python.org/issue37428}: SSLContext.post_handshake_auth = True no longer sets
SSL_VERIFY_POST_HANDSHAKE verify flag for client connections. Although the
option is documented as ignored for clients, OpenSSL implicitly enables
cert chain validation when the flag is set.

@item 
bpo-37420@footnote{https://bugs.python.org/issue37420}: @ref{ca1,,os.sched_setaffinity()} now correctly handles errors that
arise during iteration over its @code{mask} argument. Patch by Brandt Bucher.

@item 
bpo-37412@footnote{https://bugs.python.org/issue37412}: The @ref{5ad,,os.getcwdb()} function now uses the UTF-8 encoding on
Windows, rather than the ANSI code page: see 
@geindex Python Enhancement Proposals; PEP 529
PEP 529@footnote{https://www.python.org/dev/peps/pep-0529} for the rationale.
The function is no longer deprecated on Windows.

@item 
bpo-37406@footnote{https://bugs.python.org/issue37406}: The sqlite3 module now raises TypeError, rather than
ValueError, if operation argument type is not str: execute(),
executemany() and calling a connection.

@item 
bpo-29412@footnote{https://bugs.python.org/issue29412}: Fix IndexError in parsing a header value ending unexpectedly.
Patch by Abhilash Raj.

@item 
bpo-36546@footnote{https://bugs.python.org/issue36546}: The @emph{dist} argument for statistics.quantiles() is now
positional only. The current name doesn’t reflect that the argument can be
either a dataset or a distribution.  Marking the parameter as positional
avoids confusion and makes it possible to change the name later.

@item 
bpo-37394@footnote{https://bugs.python.org/issue37394}: Fix a bug that was causing the @ref{d5,,queue} module to fail if
the accelerator module was not available. Patch by Pablo Galindo.

@item 
bpo-37376@footnote{https://bugs.python.org/issue37376}: @ref{cd,,pprint} now has support for
@ref{41b,,types.SimpleNamespace}. Patch by Carl Bordum Hansen.

@item 
bpo-26967@footnote{https://bugs.python.org/issue26967}: An @ref{91c,,ArgumentParser} with
@code{allow_abbrev=False} no longer disables grouping of short flags, such as
@code{-vv}, but only disables abbreviation of long flags as documented. Patch
by Zac Hatfield-Dodds.

@item 
bpo-37212@footnote{https://bugs.python.org/issue37212}: @ref{1265,,unittest.mock.call()} now preserves the order of keyword
arguments in repr output. Patch by Karthikeyan Singaravelan.

@item 
bpo-37372@footnote{https://bugs.python.org/issue37372}: Fix error unpickling datetime.time objects from Python 2 with
seconds>=24. Patch by Justin Blanchard.

@item 
bpo-37345@footnote{https://bugs.python.org/issue37345}: Add formal support for UDPLITE sockets. Support was present
before, but it is now easier to detect support with @code{hasattr(socket,
'IPPROTO_UDPLITE')} and there are constants defined for each of the
values needed: @code{socket.IPPROTO_UDPLITE},
@code{UDPLITE_SEND_CSCOV}, and @code{UDPLITE_RECV_CSCOV}. Patch by
Gabe Appleton.

@item 
bpo-37358@footnote{https://bugs.python.org/issue37358}: Optimized @code{functools.partial} by using vectorcall.

@item 
bpo-37347@footnote{https://bugs.python.org/issue37347}: @ref{126c,,sqlite3.Connection.create_aggregate()},
@ref{126d,,sqlite3.Connection.create_function()},
@ref{183,,sqlite3.Connection.set_authorizer()},
@ref{126e,,sqlite3.Connection.set_progress_handler()}
@ref{cd6,,sqlite3.Connection.set_trace_callback()} methods lead to segfaults if
some of these methods are called twice with an equal object but not the
same. Now callbacks are stored more carefully. Patch by Aleksandr Balezin.

@item 
bpo-37163@footnote{https://bugs.python.org/issue37163}: The @emph{obj} argument of @ref{113c,,dataclasses.replace()} is
positional-only now.

@item 
bpo-37085@footnote{https://bugs.python.org/issue37085}: Add the optional Linux SocketCAN Broadcast Manager constants,
used as flags to configure the BCM behaviour, in the socket module.  Patch
by Karl Ding.

@item 
bpo-37328@footnote{https://bugs.python.org/issue37328}: @code{HTMLParser.unescape} is removed.  It was undocumented and
deprecated since Python 3.4.

@item 
bpo-37305@footnote{https://bugs.python.org/issue37305}: Add .webmanifest -> application/manifest+json to list of
recognized file types and content type headers

@item 
bpo-37320@footnote{https://bugs.python.org/issue37320}: @code{aifc.openfp()} alias to @code{aifc.open()}, @code{sunau.openfp()}
alias to @code{sunau.open()}, and @code{wave.openfp()} alias to @code{wave.open()}
have been removed. They were deprecated since Python 3.7.

@item 
bpo-37315@footnote{https://bugs.python.org/issue37315}: Deprecated accepting floats with integral value (like @code{5.0})
in @ref{43e,,math.factorial()}.

@item 
bpo-37312@footnote{https://bugs.python.org/issue37312}: @code{_dummy_thread} and @code{dummy_threading} modules have been
removed. These modules were deprecated since Python 3.7 which requires
threading support.

@item 
bpo-33972@footnote{https://bugs.python.org/issue33972}: Email with single part but content-type set to @code{multipart/*}
doesn’t raise AttributeError anymore.

@item 
bpo-37280@footnote{https://bugs.python.org/issue37280}: Use threadpool for reading from file for sendfile fallback
mode.

@item 
bpo-37279@footnote{https://bugs.python.org/issue37279}: Fix asyncio sendfile support when sendfile sends extra data in
fallback mode.

@item 
bpo-19865@footnote{https://bugs.python.org/issue19865}: @ref{126f,,ctypes.create_unicode_buffer()} now also supports
non-BMP characters on platforms with 16-bit @code{wchar_t} (for
example, Windows and AIX).

@item 
bpo-37266@footnote{https://bugs.python.org/issue37266}: In a subinterpreter, spawning a daemon thread now raises an
exception. Daemon threads were never supported in subinterpreters.
Previously, the subinterpreter finalization crashed with a Python fatal
error if a daemon thread was still running.

@item 
bpo-37210@footnote{https://bugs.python.org/issue37210}: Allow pure Python implementation of @ref{c5,,pickle} to work even
when the C @code{_pickle} module is unavailable.

@item 
bpo-21872@footnote{https://bugs.python.org/issue21872}: Fix @ref{a9,,lzma}: module decompresses data incompletely. When
decompressing a FORMAT_ALONE format file, and it doesn’t have the end
marker, sometimes the last one to dozens bytes can’t be output. Patch by
Ma Lin.

@item 
bpo-35922@footnote{https://bugs.python.org/issue35922}: Fix @code{RobotFileParser.crawl_delay()} and
@code{RobotFileParser.request_rate()} to return @code{None} rather than raise
@ref{163,,AttributeError} when no relevant rule is defined in the robots.txt
file.  Patch by Rémi Lapeyre.

@item 
bpo-35766@footnote{https://bugs.python.org/issue35766}: Change the format of feature_version to be a (major, minor)
tuple.

@item 
bpo-36607@footnote{https://bugs.python.org/issue36607}: Eliminate @ref{5af,,RuntimeError} raised by
@ref{459,,asyncio.all_tasks()} if internal tasks weak set is changed by
another thread during iteration.

@item 
bpo-18748@footnote{https://bugs.python.org/issue18748}: @code{_pyio.IOBase} destructor now does nothing if getting
the @code{closed} attribute fails to better mimic @code{_io.IOBase}
finalizer.

@item 
bpo-36402@footnote{https://bugs.python.org/issue36402}: Fix a race condition at Python shutdown when waiting for
threads. Wait until the Python thread state of all non-daemon threads get
deleted (join all non-daemon threads), rather than just wait until
non-daemon Python threads complete.

@item 
bpo-37206@footnote{https://bugs.python.org/issue37206}: Default values which cannot be represented as Python objects no
longer improperly represented as @code{None} in function signatures.

@item 
bpo-37111@footnote{https://bugs.python.org/issue37111}: Added @code{encoding} and @code{errors} keyword parameters to
@code{logging.basicConfig}.

@item 
bpo-12144@footnote{https://bugs.python.org/issue12144}: Ensure cookies with @code{expires} attribute are handled in
@code{CookieJar.make_cookies()}.

@item 
bpo-34886@footnote{https://bugs.python.org/issue34886}: Fix an unintended ValueError from @ref{6b7,,subprocess.run()} when
checking for conflicting @ref{e73,,input} and @code{stdin} or @code{capture_output} and
@code{stdout} or @code{stderr} args when they were explicitly provided but with
@ref{182,,None} values within a passed in @code{**kwargs} dict rather than as passed
directly by name. Patch contributed by Rémi Lapeyre.

@item 
bpo-37173@footnote{https://bugs.python.org/issue37173}: The exception message for @code{inspect.getfile()} now correctly
reports the passed class rather than the builtins module.

@item 
bpo-37178@footnote{https://bugs.python.org/issue37178}: Give math.perm() a one argument form that means the same as
math.factorial().

@item 
bpo-37178@footnote{https://bugs.python.org/issue37178}: For math.perm(n, k), let k default to n, giving the same result
as factorial.

@item 
bpo-37165@footnote{https://bugs.python.org/issue37165}: Converted _collections._count_elements to use the Argument
Clinic.

@item 
bpo-34767@footnote{https://bugs.python.org/issue34767}: Do not always create a @ref{7d3,,collections.deque} in
@ref{58b,,asyncio.Lock}.

@item 
bpo-37158@footnote{https://bugs.python.org/issue37158}: Speed-up statistics.fmean() by switching from a function to a
generator.

@item 
bpo-34282@footnote{https://bugs.python.org/issue34282}: Remove @code{Enum._convert} method, deprecated in 3.8.

@item 
bpo-37150@footnote{https://bugs.python.org/issue37150}: @code{argparse._ActionsContainer.add_argument} now throws error, if
someone accidentally pass FileType class object instead of instance of
FileType as @ref{89e,,type} argument

@item 
bpo-28724@footnote{https://bugs.python.org/issue28724}: The socket module now has the @ref{427,,socket.send_fds()} and
@code{socket.recv.fds()} methods. Contributed by Joannah Nanjekye, Shinya
Okano and Victor Stinner.

@item 
bpo-35621@footnote{https://bugs.python.org/issue35621}: Support running asyncio subprocesses when execution event loop
in a thread on UNIX.

@item 
bpo-36520@footnote{https://bugs.python.org/issue36520}: Lengthy email headers with UTF-8 characters are now properly
encoded when they are folded. Patch by Jeffrey Kintscher.

@item 
bpo-30835@footnote{https://bugs.python.org/issue30835}: Fixed a bug in email parsing where a message with invalid bytes
in content-transfer-encoding of a multipart message can cause an
AttributeError. Patch by Andrew Donnellan.

@item 
bpo-31163@footnote{https://bugs.python.org/issue31163}: pathlib.Path instance’s rename and replace methods now return
the new Path instance.

@item 
bpo-25068@footnote{https://bugs.python.org/issue25068}: @ref{1270,,urllib.request.ProxyHandler} now lowercases the keys of
the passed dictionary.

@item 
bpo-26185@footnote{https://bugs.python.org/issue26185}: Fix @ref{27d,,repr()} on empty @code{ZipInfo} object. Patch by
Mickaël Schoentgen.

@item 
bpo-21315@footnote{https://bugs.python.org/issue21315}: Email headers containing RFC2047 encoded words are parsed
despite the missing whitespace, and a defect registered. Also missing
trailing whitespace after encoded words is now registered as a defect.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Port test_datetime to VxWorks: skip zoneinfo tests on VxWorks

@item 
bpo-35805@footnote{https://bugs.python.org/issue35805}: Add parser for Message-ID header and add it to default
HeaderRegistry. This should prevent folding of Message-ID using RFC 2048
encoded words.

@item 
bpo-36871@footnote{https://bugs.python.org/issue36871}: Ensure method signature is used instead of constructor
signature of a class while asserting mock object against method calls.
Patch by Karthikeyan Singaravelan.

@item 
bpo-35070@footnote{https://bugs.python.org/issue35070}: posix.getgrouplist() now works correctly when the user belongs
to NGROUPS_MAX supplemental groups. Patch by Jeffrey Kintscher.

@item 
bpo-31783@footnote{https://bugs.python.org/issue31783}: Fix race condition in ThreadPoolExecutor when worker threads
are created during interpreter shutdown.

@item 
bpo-36582@footnote{https://bugs.python.org/issue36582}: Fix @code{UserString.encode()} to correctly return @code{bytes}
rather than a @code{UserString} instance.

@item 
bpo-32424@footnote{https://bugs.python.org/issue32424}: Deprecate xml.etree.ElementTree.Element.copy() in favor of
copy.copy().

Patch by Gordon P. Hemsley

@item 
bpo-36564@footnote{https://bugs.python.org/issue36564}: Fix infinite loop in email header folding logic that would be
triggered when an email policy’s max_line_length is not long enough to
include the required markup and any values in the message. Patch by Paul
Ganssle

@item 
bpo-36543@footnote{https://bugs.python.org/issue36543}: Removed methods Element.getchildren(), Element.getiterator()
and ElementTree.getiterator() and the xml.etree.cElementTree module.

@item 
bpo-36409@footnote{https://bugs.python.org/issue36409}: Remove the old plistlib API deprecated in Python 3.4

@item 
bpo-36302@footnote{https://bugs.python.org/issue36302}: distutils sorts source file lists so that Extension .so files
build more reproducibly by default

@item 
bpo-36250@footnote{https://bugs.python.org/issue36250}: Ignore @code{ValueError} from @code{signal} with @code{interaction} in
non-main thread.

@item 
bpo-36046@footnote{https://bugs.python.org/issue36046}: Added @code{user}, @code{group} and @code{extra_groups} parameters to
the subprocess.Popen constructor. Patch by Patrick McLean.

@item 
bpo-32627@footnote{https://bugs.python.org/issue32627}: Fix compile error when @code{_uuid} headers conflicting included.

@item 
bpo-35800@footnote{https://bugs.python.org/issue35800}: Deprecate @code{smtpd.MailmanProxy} ready for future removal.

@item 
bpo-35168@footnote{https://bugs.python.org/issue35168}: @ref{1271,,shlex.shlex.punctuation_chars} is now a read-only
property.

@item 
bpo-8538@footnote{https://bugs.python.org/issue8538}: Add support for boolean actions like @code{--foo} and @code{--no-foo}
to argparse. Patch contributed by Rémi Lapeyre.

@item 
bpo-20504@footnote{https://bugs.python.org/issue20504}: Fixes a bug in @ref{14,,cgi} module when a multipart/form-data
request has no @code{Content-Length} header.

@item 
bpo-25988@footnote{https://bugs.python.org/issue25988}: The abstract base classes in @ref{1d,,collections.abc} no longer
are exposed in the regular @ref{1c,,collections} module.

@item 
bpo-11122@footnote{https://bugs.python.org/issue11122}: Distutils won’t check for rpmbuild in specified paths only.

@item 
bpo-34775@footnote{https://bugs.python.org/issue34775}: Division handling of PurePath now returns NotImplemented
instead of raising a TypeError when passed something other than an
instance of str or PurePath. Patch by Roger Aiudi.

@item 
bpo-34749@footnote{https://bugs.python.org/issue34749}: @ref{1272,,binascii.a2b_base64()} is now up to 2 times faster. Patch
by Sergey Fedoseev.

@item 
bpo-34519@footnote{https://bugs.python.org/issue34519}: Add additional aliases for HP Roman 8. Patch by Michael Osipov.

@item 
bpo-28009@footnote{https://bugs.python.org/issue28009}: Fix uuid.getnode() on platforms with ‘.’ as MAC Addr delimiter
as well fix for MAC Addr format that omits a leading 0 in MAC Addr values.
Currently, AIX is the only know platform with these settings. Patch by
Michael Felt.

@item 
bpo-30618@footnote{https://bugs.python.org/issue30618}: Add @ref{413,,readlink()}. Patch by Girts Folkmanis.

@item 
bpo-32498@footnote{https://bugs.python.org/issue32498}: Made @ref{1273,,urllib.parse.unquote()} accept bytes in addition to
strings. Patch by Stein Karlsen.

@item 
bpo-33348@footnote{https://bugs.python.org/issue33348}: lib2to3 now recognizes expressions after @code{*} and @code{**} like in
@code{f(*[] or [])}.

@item 
bpo-32689@footnote{https://bugs.python.org/issue32689}: Update @ref{572,,shutil.move()} function to allow for Path objects
to be used as source argument. Patch by Emily Morehouse and Maxwell
“5.13b” McKinnon.

@item 
bpo-32820@footnote{https://bugs.python.org/issue32820}: Added __format__ to IPv4 and IPv6 classes.  Always outputs a
fully zero- padded string. Supports b/x/n modifiers (bin/hex/native
format).  Native format for IPv4 is bin, native format for IPv6 is hex.
Also supports ‘#’ and ‘_’ modifiers.

@item 
bpo-27657@footnote{https://bugs.python.org/issue27657}: Fix urllib.parse.urlparse() with numeric paths. A string like
“path:80” is no longer parsed as a path but as a scheme (“path”) and a
path (“80”).

@item 
bpo-4963@footnote{https://bugs.python.org/issue4963}: Fixed non-deterministic behavior related to mimetypes extension
mapping and module reinitialization.
@end itemize

@node Documentation<22>,Tests<20>,Library<21>,Python 3 9 0 alpha 1
@anchor{whatsnew/changelog id182}@anchor{1274}
@subsubsection Documentation



@itemize -

@item 
bpo-21767@footnote{https://bugs.python.org/issue21767}: Explicitly mention abc support in functools.singledispatch

@item 
bpo-38816@footnote{https://bugs.python.org/issue38816}: Provides more details about the interaction between
@code{fork()} and CPython’s runtime, focusing just on the C-API.  This
includes cautions about where @code{fork()} should and shouldn’t be
called.

@item 
bpo-38351@footnote{https://bugs.python.org/issue38351}: Modernize @ref{64,,email} examples from %-formatting to f-strings.

@item 
bpo-38778@footnote{https://bugs.python.org/issue38778}: Document the fact that @ref{5af,,RuntimeError} is raised if
@ref{bb9,,os.fork()} is called in a subinterpreter.

@item 
bpo-38592@footnote{https://bugs.python.org/issue38592}: Add Brazilian Portuguese to the language switcher at Python
Documentation website.

@item 
bpo-38294@footnote{https://bugs.python.org/issue38294}: Add list of no-longer-escaped chars to re.escape documentation

@item 
bpo-38053@footnote{https://bugs.python.org/issue38053}: Modernized the plistlib documentation

@item 
bpo-26868@footnote{https://bugs.python.org/issue26868}: Fix example usage of @ref{369,,PyModule_AddObject()} to properly
handle errors.

@item 
bpo-36797@footnote{https://bugs.python.org/issue36797}: Fix a dead link in the distutils API Reference.

@item 
bpo-37977@footnote{https://bugs.python.org/issue37977}: Warn more strongly and clearly about pickle insecurity

@item 
bpo-37979@footnote{https://bugs.python.org/issue37979}: Added a link to dateutil.parser.isoparse in the
datetime.fromisoformat documentation. Patch by Paul Ganssle

@item 
bpo-12707@footnote{https://bugs.python.org/issue12707}: Deprecate info(), geturl(), getcode() methods in favor of the
headers, url, and status properties, respectively, for HTTPResponse and
addinfourl. Also deprecate the code attribute of addinfourl in favor of
the status attribute. Patch by Ashwin Ramaswami

@item 
bpo-37937@footnote{https://bugs.python.org/issue37937}: Mention @code{frame.f_trace} in @ref{1034,,sys.settrace()} docs.

@item 
bpo-37878@footnote{https://bugs.python.org/issue37878}: Make @ref{1275,,PyThreadState_DeleteCurrent()} Internal.

@item 
bpo-37759@footnote{https://bugs.python.org/issue37759}: Beginning edits to Whatsnew 3.8

@item 
bpo-37726@footnote{https://bugs.python.org/issue37726}: Stop recommending getopt in the tutorial for command line
argument parsing and promote argparse.

@item 
bpo-32910@footnote{https://bugs.python.org/issue32910}: Remove implementation-specific behaviour of how venv’s
Deactivate works.

@item 
bpo-37256@footnote{https://bugs.python.org/issue37256}: Fix wording of arguments for @code{Request} in
@ref{127,,urllib.request}

@item 
bpo-37284@footnote{https://bugs.python.org/issue37284}: Add a brief note to indicate that any new
@code{sys.implementation} required attributes must go through the PEP
process.

@item 
bpo-30088@footnote{https://bugs.python.org/issue30088}: Documented that @ref{eaa,,mailbox.Maildir} constructor doesn’t
attempt to verify the maildir folder layout correctness. Patch by
Sviatoslav Sydorenko.

@item 
bpo-37521@footnote{https://bugs.python.org/issue37521}: Fix @ref{96,,importlib} examples to insert any newly created modules
via importlib.util.module_from_spec() immediately into sys.modules instead
of after calling loader.exec_module().

Thanks to Benjamin Mintz for finding the bug.

@item 
bpo-37456@footnote{https://bugs.python.org/issue37456}: Slash (‘/’) is now part of syntax.

@item 
bpo-37487@footnote{https://bugs.python.org/issue37487}: Fix PyList_GetItem index description to include 0.

@item 
bpo-37149@footnote{https://bugs.python.org/issue37149}: Replace the dead link to the Tkinter 8.5 reference by John
Shipman, New Mexico Tech, with a link to the archive.org copy.

@item 
bpo-37478@footnote{https://bugs.python.org/issue37478}: Added possible exceptions to the description of os.chdir().

@item 
bpo-34903@footnote{https://bugs.python.org/issue34903}: Documented that in @ref{114f,,datetime.datetime.strptime()}, the
leading zero in some two-digit formats is optional. Patch by Mike Gleen.

@item 
bpo-36260@footnote{https://bugs.python.org/issue36260}: Add decompression pitfalls to zipfile module documentation.

@item 
bpo-37004@footnote{https://bugs.python.org/issue37004}: In the documentation for difflib, a note was added explicitly
warning that the results of SequenceMatcher’s ratio method may depend on
the order of the input strings.

@item 
bpo-36960@footnote{https://bugs.python.org/issue36960}: Restructured the @ref{2f,,datetime} docs in the interest of making
them more user-friendly and improving readability. Patch by Brad Solomon.

@item 
bpo-36487@footnote{https://bugs.python.org/issue36487}: Make C-API docs clear about what the “main” interpreter is.

@item 
bpo-23460@footnote{https://bugs.python.org/issue23460}: The documentation for decimal string formatting using the @code{:g}
specifier has been updated to reflect the correct exponential notation
cutoff point. Original patch contributed by Tuomas Suutari.

@item 
bpo-35803@footnote{https://bugs.python.org/issue35803}: Document and test that @code{tempfile} functions may accept a
@ref{63d,,path-like object} for the @code{dir} argument.  Patch by Anthony
Sottile.

@item 
bpo-33944@footnote{https://bugs.python.org/issue33944}: Added a note about the intended use of code in .pth files.

@item 
bpo-34293@footnote{https://bugs.python.org/issue34293}: Fix the Doc/Makefile regarding PAPER environment variable and
PDF builds

@item 
bpo-25237@footnote{https://bugs.python.org/issue25237}: Add documentation for tkinter modules
@end itemize

@node Tests<20>,Build<21>,Documentation<22>,Python 3 9 0 alpha 1
@anchor{whatsnew/changelog id183}@anchor{1276}
@subsubsection Tests



@itemize -

@item 
bpo-38614@footnote{https://bugs.python.org/issue38614}: Fix test_communicate() of test_asyncio.test_subprocess: use
@code{support.LONG_TIMEOUT} (5 minutes), instead of just 1 minute.

@item 
bpo-38614@footnote{https://bugs.python.org/issue38614}: Add timeout constants to @ref{100,,test.support}:
@ref{124d,,LOOPBACK_TIMEOUT},
@ref{124e,,INTERNET_TIMEOUT},
@ref{1120,,SHORT_TIMEOUT} and
@ref{124f,,LONG_TIMEOUT}.

@item 
bpo-38502@footnote{https://bugs.python.org/issue38502}: test.regrtest now uses process groups in the multiprocessing
mode (-jN command line option) if process groups are available: if
@ref{1277,,os.setsid()} and @ref{1278,,os.killpg()} functions are available.

@item 
bpo-35998@footnote{https://bugs.python.org/issue35998}: Fix a race condition in test_asyncio.test_start_tls_server_1().
Previously, there was a race condition between the test main() function
which replaces the protocol and the test ServerProto protocol which sends
ANSWER once it gets HELLO. Now, only the test main() function is
responsible to send data, ServerProto no longer sends data.

@item 
bpo-38470@footnote{https://bugs.python.org/issue38470}: Fix @code{test_compileall.test_compile_dir_maxlevels()} on Windows
without long path support: only create 3 subdirectories instead of between
20 and 100 subdirectories.

@item 
bpo-37531@footnote{https://bugs.python.org/issue37531}: On timeout, regrtest no longer attempts to call
@code{popen.communicate()} again: it can hang until all child processes using
stdout and stderr pipes completes. Kill the worker process and ignores its
output. Change also the faulthandler timeout of the main process from 1
minute to 5 minutes, for Python slowest buildbots.

@item 
bpo-38239@footnote{https://bugs.python.org/issue38239}: Fix test_gdb for Link Time Optimization (LTO) builds.

@item 
bpo-38275@footnote{https://bugs.python.org/issue38275}: test_ssl now handles disabled TLS/SSL versions better.
OpenSSL’s crypto policy and run-time settings are recognized and tests for
disabled versions are skipped. Tests also accept more TLS minimum_versions
for platforms that override OpenSSL’s default with strict settings.

@item 
bpo-38271@footnote{https://bugs.python.org/issue38271}: The private keys for test_ssl were encrypted with 3DES in
traditional PKCS#5 format. 3DES and the digest algorithm of PKCS#5 are
blocked by some strict crypto policies. Use PKCS#8 format with AES256
encryption instead.

@item 
bpo-38270@footnote{https://bugs.python.org/issue38270}: test.support now has a helper function to check for
availability of a hash digest function. Several tests are refactored avoid
MD5 and use SHA256 instead. Other tests are marked to use MD5 and skipped
when MD5 is disabled.

@item 
bpo-37123@footnote{https://bugs.python.org/issue37123}: Multiprocessing test test_mymanager() now also expects
-SIGTERM, not only exitcode 0. BaseManager._finalize_manager() sends
SIGTERM to the manager process if it takes longer than 1 second to stop,
which happens on slow buildbots.

@item 
bpo-38212@footnote{https://bugs.python.org/issue38212}: Multiprocessing tests: increase
test_queue_feeder_donot_stop_onexc() timeout from 1 to 60 seconds.

@item 
bpo-38117@footnote{https://bugs.python.org/issue38117}: Test with OpenSSL 1.1.1d

@item 
bpo-38018@footnote{https://bugs.python.org/issue38018}: Increase code coverage for multiprocessing.shared_memory.

@item 
bpo-37805@footnote{https://bugs.python.org/issue37805}: Add tests for json.dump(…, skipkeys=True). Patch by Dong-hee
Na.

@item 
bpo-37531@footnote{https://bugs.python.org/issue37531}: Enhance regrtest multiprocess timeout: write a message when
killing a worker process, catch popen.kill() and popen.wait() exceptions,
put a timeout on the second call to popen.communicate().

@item 
bpo-37876@footnote{https://bugs.python.org/issue37876}: Add tests for ROT-13 codec.

@item 
bpo-36833@footnote{https://bugs.python.org/issue36833}: Added tests for PyDateTime_xxx_GET_xxx() macros of the C API of
the @ref{2f,,datetime} module. Patch by Joannah Nanjekye.

@item 
bpo-37558@footnote{https://bugs.python.org/issue37558}: Fix test_shared_memory_cleaned_after_process_termination name
handling

@item 
bpo-37526@footnote{https://bugs.python.org/issue37526}: Add @code{test.support.catch_threading_exception()}: context
manager catching @ref{44b,,threading.Thread} exception using
@ref{2d1,,threading.excepthook()}.

@item 
bpo-37421@footnote{https://bugs.python.org/issue37421}: test_concurrent_futures now explicitly stops the ForkServer
instance if it’s running.

@item 
bpo-37421@footnote{https://bugs.python.org/issue37421}: multiprocessing tests now stop the ForkServer instance if it’s
running: close the “alive” file descriptor to ask the server to stop and
then remove its UNIX address.

@item 
bpo-37421@footnote{https://bugs.python.org/issue37421}: test_distutils.test_build_ext() is now able to remove the
temporary directory on Windows: don’t import the newly built C extension
(“xx”) in the current process, but test it in a separated process.

@item 
bpo-37421@footnote{https://bugs.python.org/issue37421}: test_concurrent_futures now cleans up multiprocessing to remove
immediately temporary directories created by
multiprocessing.util.get_temp_dir().

@item 
bpo-37421@footnote{https://bugs.python.org/issue37421}: test_winconsoleio doesn’t leak a temporary file anymore: use
tempfile.TemporaryFile() to remove it when the test completes.

@item 
bpo-37421@footnote{https://bugs.python.org/issue37421}: multiprocessing tests now explicitly call @code{_run_finalizers()}
to immediately remove temporary directories created by tests.

@item 
bpo-37421@footnote{https://bugs.python.org/issue37421}: urllib.request tests now call
@ref{1279,,urlcleanup()} to remove temporary files created by
@code{urlretrieve()} tests and to clear the @code{_opener} global variable set
by @code{urlopen()} and functions calling indirectly @code{urlopen()}.

@item 
bpo-37472@footnote{https://bugs.python.org/issue37472}: Remove @code{Lib/test/outstanding_bugs.py}.

@item 
bpo-37199@footnote{https://bugs.python.org/issue37199}: Fix test failures when IPv6 is unavailable or disabled.

@item 
bpo-19696@footnote{https://bugs.python.org/issue19696}: Replace deprecated method “random.choose” with “random.choice”
in “test_pkg_import.py”.

@item 
bpo-37335@footnote{https://bugs.python.org/issue37335}: Remove no longer necessary code from c locale coercion tests

@item 
bpo-37421@footnote{https://bugs.python.org/issue37421}: Fix test_shutil to no longer leak temporary files.

@item 
bpo-37411@footnote{https://bugs.python.org/issue37411}: Fix test_wsgiref.testEnviron() to no longer depend on the
environment variables (don’t fail if “X” variable is set).

@item 
bpo-37400@footnote{https://bugs.python.org/issue37400}: Fix test_os.test_chown(): use os.getgroups() rather than
grp.getgrall() to get groups. Rename also the test to test_chown_gid().

@item 
bpo-37359@footnote{https://bugs.python.org/issue37359}: Add –cleanup option to python3 -m test to remove
@code{test_python_*} directories of previous failed jobs. Add “make
cleantest” to run @code{python3 -m test --cleanup}.

@item 
bpo-37362@footnote{https://bugs.python.org/issue37362}: test_gdb no longer fails if it gets an “unexpected” message on
stderr: it now ignores stderr. The purpose of test_gdb is to test that
python-gdb.py commands work as expected, not to test gdb.

@item 
bpo-35998@footnote{https://bugs.python.org/issue35998}: Avoid TimeoutError in test_asyncio: test_start_tls_server_1()

@item 
bpo-37278@footnote{https://bugs.python.org/issue37278}: Fix test_asyncio ProactorLoopCtrlC: join the thread to prevent
leaking a running thread and leaking a reference.

@item 
bpo-37261@footnote{https://bugs.python.org/issue37261}: Fix @ref{127a,,test.support.catch_unraisable_exception()}: its
__exit__() method now ignores unraisable exception raised when clearing
its @code{unraisable} attribute.

@item 
bpo-37069@footnote{https://bugs.python.org/issue37069}: regrtest now uses @ref{18d,,sys.unraisablehook()} to mark a test as
“environment altered” (ENV_CHANGED) if it emits an “unraisable exception”.
Moreover, regrtest logs a warning in this case.

Use @code{python3 -m test --fail-env-changed} to catch unraisable exceptions
in tests.

@item 
bpo-37252@footnote{https://bugs.python.org/issue37252}: Fix assertions in @code{test_close} and
@code{test_events_mask_overflow} devpoll tests.

@item 
bpo-37169@footnote{https://bugs.python.org/issue37169}: Rewrite @code{_PyObject_IsFreed()} unit tests.

@item 
bpo-37153@footnote{https://bugs.python.org/issue37153}: @code{test_venv.test_multiprocessing()} now explicitly calls
@code{pool.terminate()} to wait until the pool completes.

@item 
bpo-34001@footnote{https://bugs.python.org/issue34001}: Make test_ssl pass with LibreSSL. LibreSSL handles minimum and
maximum TLS version differently than OpenSSL.

@item 
bpo-36919@footnote{https://bugs.python.org/issue36919}: Make @code{test_source_encoding.test_issue2301} implementation
independent. The test will work now for both CPython and IronPython.

@item 
bpo-30202@footnote{https://bugs.python.org/issue30202}: Update @code{test.test_importlib.test_abc} to test
@code{find_spec()}.

@item 
bpo-28009@footnote{https://bugs.python.org/issue28009}: Modify the test_uuid logic to test when a program is available
AND can be used to obtain a MACADDR as basis for an UUID. Patch by M. Felt

@item 
bpo-34596@footnote{https://bugs.python.org/issue34596}: Fallback to a default reason when @ref{127b,,unittest.skip()} is
uncalled. Patch by Naitree Zhu.
@end itemize

@node Build<21>,Windows<19>,Tests<20>,Python 3 9 0 alpha 1
@anchor{whatsnew/changelog id184}@anchor{127c}
@subsubsection Build



@itemize -

@item 
bpo-38809@footnote{https://bugs.python.org/issue38809}: On Windows, build scripts will now recognize and use python.exe
from an active virtual env.

@item 
bpo-38684@footnote{https://bugs.python.org/issue38684}: Fix _hashlib build when Blake2 is disabled, but OpenSSL
supports it.

@item 
bpo-38468@footnote{https://bugs.python.org/issue38468}: Misc/python-config.in now uses @code{getvar()} for all still
existing @ref{bbc,,sysconfig.get_config_var()} calls. Patch by Joannah Nanjekye.

@item 
bpo-37415@footnote{https://bugs.python.org/issue37415}: Fix stdatomic.h header check for ICC compiler: the ICC
implementation lacks atomic_uintptr_t type which is needed by Python.

@item 
bpo-38301@footnote{https://bugs.python.org/issue38301}: In Solaris family, we must be sure to use @code{-D_REENTRANT}.
Patch by Jesús Cea Avión.

@item 
bpo-36002@footnote{https://bugs.python.org/issue36002}: Locate @code{llvm-profdata} and @code{llvm-ar} binaries using
@code{AC_PATH_TOOL} rather than @code{AC_PATH_TARGET_TOOL}.

@item 
bpo-37936@footnote{https://bugs.python.org/issue37936}: The @code{.gitignore} file systematically keeps “rooted”, with
a non-trailing slash, all the rules that are meant to apply to files in a
specific place in the repo.  Previously, when the intended file to ignore
happened to be at the root of the repo, we’d most often accidentally also
ignore files and directories with the same name anywhere in the tree.

@item 
bpo-37760@footnote{https://bugs.python.org/issue37760}: The @code{Tools/unicode/makeunicodedata.py} script, which is
used for converting information from the Unicode Character Database into
generated code and data used by the methods of @ref{2ef,,str} and by the
@ref{121,,unicodedata} module, now handles each character’s data as a
@code{dataclass} with named attributes, rather than a length-18 list of
different fields.

@item 
bpo-37936@footnote{https://bugs.python.org/issue37936}: The @code{.gitignore} file no longer applies to any files that
are in fact tracked in the Git repository.  Patch by Greg Price.

@item 
bpo-37725@footnote{https://bugs.python.org/issue37725}: Change “clean” makefile target to also clean the program guided
optimization (PGO) data.  Previously you would have to use “make clean”
and “make profile-removal”, or “make clobber”.

@item 
bpo-37707@footnote{https://bugs.python.org/issue37707}: Mark some individual tests to skip when –pgo is used.  The
tests marked increase the PGO task time significantly and likely don’t
help improve optimization of the final executable.

@item 
bpo-36044@footnote{https://bugs.python.org/issue36044}: Reduce the number of unit tests run for the PGO generation
task.  This speeds up the task by a factor of about 15x.  Running the full
unit test suite is slow.  This change may result in a slightly less
optimized build since not as many code branches will be executed.  If you
are willing to wait for the much slower build, the old behavior can be
restored using ‘./configure [..] PROFILE_TASK=”-m test –pgo-extended”’.
We make no guarantees as to which PGO task set produces a faster build.
Users who care should run their own relevant benchmarks as results can
depend on the environment, workload, and compiler tool chain.

@item 
bpo-37468@footnote{https://bugs.python.org/issue37468}: @code{make install} no longer installs @code{wininst-*.exe} files
used by distutils bdist_wininst: bdist_wininst only works on Windows.

@item 
bpo-37189@footnote{https://bugs.python.org/issue37189}: Many @code{PyRun_XXX()} functions like @ref{127d,,PyRun_String()} were
no longer exported in @code{libpython38.dll} by mistake. Export them again to
fix the ABI compatibility.

@item 
bpo-25361@footnote{https://bugs.python.org/issue25361}: Enables use of SSE2 instructions in Windows 32-bit build.

@item 
bpo-36210@footnote{https://bugs.python.org/issue36210}: Update optional extension module detection for AIX. ossaudiodev
and spwd are not applicable for AIX, and are no longer reported as
missing. 3rd-party packaging of ncurses (with ASIS support) conflicts with
officially supported AIX curses library, so configure AIX to use
libcurses.a. However, skip trying to build _curses_panel.

patch by M Felt
@end itemize

@node Windows<19>,macOS<17>,Build<21>,Python 3 9 0 alpha 1
@anchor{whatsnew/changelog id185}@anchor{127e}
@subsubsection Windows



@itemize -

@item 
bpo-38589@footnote{https://bugs.python.org/issue38589}: Fixes HTML Help shortcut when Windows is not installed to C
drive

@item 
bpo-38453@footnote{https://bugs.python.org/issue38453}: Ensure ntpath.realpath() correctly resolves relative paths.

@item 
bpo-38519@footnote{https://bugs.python.org/issue38519}: Restores the internal C headers that were missing from the
nuget.org and Microsoft Store packages.

@item 
bpo-38492@footnote{https://bugs.python.org/issue38492}: Remove @code{pythonw.exe} dependency on the Microsoft C++ runtime.

@item 
bpo-38344@footnote{https://bugs.python.org/issue38344}: Fix error message in activate.bat

@item 
bpo-38359@footnote{https://bugs.python.org/issue38359}: Ensures @code{pyw.exe} launcher reads correct registry key.

@item 
bpo-38355@footnote{https://bugs.python.org/issue38355}: Fixes @code{ntpath.realpath} failing on @code{sys.executable}.

@item 
bpo-38117@footnote{https://bugs.python.org/issue38117}: Update bundled OpenSSL to 1.1.1d

@item 
bpo-38092@footnote{https://bugs.python.org/issue38092}: Reduce overhead when using multiprocessing in a Windows virtual
environment.

@item 
bpo-38133@footnote{https://bugs.python.org/issue38133}: Allow py.exe launcher to locate installations from the
Microsoft Store and improve display of active virtual environments.

@item 
bpo-38114@footnote{https://bugs.python.org/issue38114}: The @code{pip.ini} is no longer included in the Nuget package.

@item 
bpo-32592@footnote{https://bugs.python.org/issue32592}: Set Windows 8 as the minimum required version for API support

@item 
bpo-36634@footnote{https://bugs.python.org/issue36634}: @ref{298,,os.cpu_count()} now returns active processors rather than
maximum processors.

@item 
bpo-36634@footnote{https://bugs.python.org/issue36634}: venv activate.bat now works when the existing variables contain
double quote characters.

@item 
bpo-38081@footnote{https://bugs.python.org/issue38081}: Prevent error calling @ref{2a0,,os.path.realpath()} on @code{'NUL'}.

@item 
bpo-38087@footnote{https://bugs.python.org/issue38087}: Fix case sensitivity in test_pathlib and test_ntpath.

@item 
bpo-38088@footnote{https://bugs.python.org/issue38088}: Fixes distutils not finding vcruntime140.dll with only the v142
toolset installed.

@item 
bpo-37283@footnote{https://bugs.python.org/issue37283}: Ensure command-line and unattend.xml setting override
previously detected states in Windows installer.

@item 
bpo-38030@footnote{https://bugs.python.org/issue38030}: Fixes @ref{516,,os.stat()} failing for block devices on Windows

@item 
bpo-38020@footnote{https://bugs.python.org/issue38020}: Fixes potential crash when calling @ref{414,,os.readlink()} (or
indirectly through @ref{2a0,,realpath()}) on a file that is not a
supported link.

@item 
bpo-37705@footnote{https://bugs.python.org/issue37705}: Improve the implementation of @code{winerror_to_errno()}.

@item 
bpo-37549@footnote{https://bugs.python.org/issue37549}: @ref{127f,,os.dup()} no longer fails for standard streams on Windows
7.

@item 
bpo-1311@footnote{https://bugs.python.org/issue1311}: The @code{nul} file on Windows now returns True from
@ref{51a,,exists()} and a valid result from @ref{516,,os.stat()} with
@code{S_IFCHR} set.

@item 
bpo-9949@footnote{https://bugs.python.org/issue9949}: Enable support for following symlinks in @code{os.realpath()}.

@item 
bpo-37834@footnote{https://bugs.python.org/issue37834}: Treat all name surrogate reparse points on Windows in
@ref{517,,os.lstat()} and other reparse points as regular files in
@ref{516,,os.stat()}.

@item 
bpo-36266@footnote{https://bugs.python.org/issue36266}: Add the module name in the formatted error message when DLL
load fail happens during module import in
@code{_PyImport_FindSharedFuncptrWindows()}. Patch by Srinivas Nyayapati.

@item 
bpo-25172@footnote{https://bugs.python.org/issue25172}: Trying to import the @ref{27,,crypt} module on Windows will result
in an @ref{3b8,,ImportError} with a message explaining that the module isn’t
supported on Windows. On other platforms, if the underlying @code{_crypt}
module is not available, the ImportError will include a message explaining
the problem.

@item 
bpo-37778@footnote{https://bugs.python.org/issue37778}: Fixes the icons used for file associations to the Microsoft
Store package.

@item 
bpo-37734@footnote{https://bugs.python.org/issue37734}: Fix use of registry values to launch Python from Microsoft
Store app.

@item 
bpo-37702@footnote{https://bugs.python.org/issue37702}: Fix memory leak on Windows in creating an SSLContext object or
running urllib.request.urlopen(’@indicateurl{https://}…’).

@item 
bpo-37672@footnote{https://bugs.python.org/issue37672}: Switch Windows Store package’s pip to use bundled
@code{pip.ini} instead of 
@geindex PIP_USER
@geindex environment variable; PIP_USER
@code{PIP_USER} variable.

@item 
bpo-10945@footnote{https://bugs.python.org/issue10945}: Officially drop support for creating bdist_wininst installers
on non-Windows systems.

@item 
bpo-37445@footnote{https://bugs.python.org/issue37445}: Include the @code{FORMAT_MESSAGE_IGNORE_INSERTS} flag in
@code{FormatMessageW()} calls.

@item 
bpo-37369@footnote{https://bugs.python.org/issue37369}: Fixes path for @ref{585,,sys.executable} when running from the
Microsoft Store.

@item 
bpo-37380@footnote{https://bugs.python.org/issue37380}: Don’t collect unfinished processes with @code{subprocess._active}
on Windows to cleanup later. Patch by Ruslan Kuprieiev.

@item 
bpo-37351@footnote{https://bugs.python.org/issue37351}: Removes libpython38.a from standard Windows distribution.

@item 
bpo-35360@footnote{https://bugs.python.org/issue35360}: Update Windows builds to use SQLite 3.28.0.

@item 
bpo-37267@footnote{https://bugs.python.org/issue37267}: On Windows, @ref{127f,,os.dup()} no longer creates an inheritable fd
when handling a character file.

@item 
bpo-36779@footnote{https://bugs.python.org/issue36779}: Ensure @code{time.tzname} is correct on Windows when the active
code page is set to CP_UTF7 or CP_UTF8.

@item 
bpo-32587@footnote{https://bugs.python.org/issue32587}: Make @ref{1280,,winreg.REG_MULTI_SZ} support zero-length strings.

@item 
bpo-28269@footnote{https://bugs.python.org/issue28269}: Replace use of @code{strcasecmp()} for the system function
@code{_stricmp()}. Patch by Minmin Gong.

@item 
bpo-36590@footnote{https://bugs.python.org/issue36590}: Add native Bluetooth RFCOMM support to socket module.
@end itemize

@node macOS<17>,IDLE<19>,Windows<19>,Python 3 9 0 alpha 1
@anchor{whatsnew/changelog id186}@anchor{1281}
@subsubsection macOS



@itemize -

@item 
bpo-38117@footnote{https://bugs.python.org/issue38117}: Updated OpenSSL to 1.1.1d in macOS installer.

@item 
bpo-38089@footnote{https://bugs.python.org/issue38089}: Move Azure Pipelines to latest VM versions and make macOS tests
optional

@item 
bpo-18049@footnote{https://bugs.python.org/issue18049}: Increase the default stack size of threads from 5MB to 16MB on
macOS, to match the stack size of the main thread. This avoids crashes on
deep recursion in threads.

@item 
bpo-34602@footnote{https://bugs.python.org/issue34602}: Avoid test suite failures on macOS by no longer calling
resource.setrlimit to increase the process stack size limit at runtime.
The runtime change is no longer needed since the interpreter is being
built with a larger default stack size.

@item 
bpo-35360@footnote{https://bugs.python.org/issue35360}: Update macOS installer to use SQLite 3.28.0.

@item 
bpo-34631@footnote{https://bugs.python.org/issue34631}: Updated OpenSSL to 1.1.1c in macOS installer.
@end itemize

@node IDLE<19>,Tools/Demos<7>,macOS<17>,Python 3 9 0 alpha 1
@anchor{whatsnew/changelog id187}@anchor{1282}
@subsubsection IDLE



@itemize -

@item 
bpo-26353@footnote{https://bugs.python.org/issue26353}: Stop adding newline when saving an IDLE shell window.

@item 
bpo-4630@footnote{https://bugs.python.org/issue4630}: Add an option to toggle IDLE’s cursor blink for shell, editor,
and output windows.  See Settings, General, Window Preferences, Cursor
Blink. Patch by Zackery Spytz.

@item 
bpo-38598@footnote{https://bugs.python.org/issue38598}: Do not try to compile IDLE shell or output windows

@item 
bpo-36698@footnote{https://bugs.python.org/issue36698}: IDLE no longer fails when write non-encodable characters to
stderr.  It now escapes them with a backslash, as the regular Python
interpreter. Added the @code{errors} field to the standard streams.

@item 
bpo-35379@footnote{https://bugs.python.org/issue35379}: When exiting IDLE, catch any AttributeError.  One happens when
EditorWindow.close is called twice.  Printing a traceback, when IDLE is
run from a terminal, is useless and annoying.

@item 
bpo-38183@footnote{https://bugs.python.org/issue38183}: To avoid problems, test_idle ignores the user config directory.
It no longer tries to create or access .idlerc or any files within. Users
must run IDLE to discover problems with saving settings.

@item 
bpo-38077@footnote{https://bugs.python.org/issue38077}: IDLE no longer adds ‘argv’ to the user namespace when
initializing it.  This bug only affected 3.7.4 and 3.8.0b2 to 3.8.0b4.

@item 
bpo-38041@footnote{https://bugs.python.org/issue38041}: Shell restart lines now fill the window width, always start
with ‘=’, and avoid wrapping unnecessarily. The line will still wrap if
the included file name is long relative to the width.

@item 
bpo-35771@footnote{https://bugs.python.org/issue35771}: To avoid occasional spurious test_idle failures on slower
machines, increase the @code{hover_delay} in test_tooltip.

@item 
bpo-37824@footnote{https://bugs.python.org/issue37824}: Properly handle user input warnings in IDLE shell. Cease
turning SyntaxWarnings into SyntaxErrors.

@item 
bpo-37929@footnote{https://bugs.python.org/issue37929}: IDLE Settings dialog now closes properly when there is no shell
window.

@item 
bpo-37902@footnote{https://bugs.python.org/issue37902}: Add mousewheel scrolling for IDLE module, path, and stack
browsers. Patch by George Zhang.

@item 
bpo-37849@footnote{https://bugs.python.org/issue37849}: Fixed completions list appearing too high or low when shown
above the current line.

@item 
bpo-36419@footnote{https://bugs.python.org/issue36419}: Refactor IDLE autocomplete and improve testing.

@item 
bpo-37748@footnote{https://bugs.python.org/issue37748}: Reorder the Run menu.  Put the most common choice, Run Module,
at the top.

@item 
bpo-37692@footnote{https://bugs.python.org/issue37692}: Improve highlight config sample with example shell interaction
and better labels for shell elements.

@item 
bpo-37628@footnote{https://bugs.python.org/issue37628}: Settings dialog no longer expands with font size.

@item 
bpo-37627@footnote{https://bugs.python.org/issue37627}: Initialize the Customize Run dialog with the command line
arguments most recently entered before.  The user can optionally edit
before submitting them.

@item 
bpo-33610@footnote{https://bugs.python.org/issue33610}: Fix code context not showing the correct context when first
toggled on.

@item 
bpo-37530@footnote{https://bugs.python.org/issue37530}: Optimize code context to reduce unneeded background activity.
Font and highlight changes now occur along with text changes instead of
after a random delay.

@item 
bpo-27452@footnote{https://bugs.python.org/issue27452}: Cleanup @code{config.py} by inlining @code{RemoveFile} and
simplifying the handling of @code{file} in @code{CreateConfigHandlers}.

@item 
bpo-37325@footnote{https://bugs.python.org/issue37325}: Fix tab focus traversal order for help source and custom run
dialogs.

@item 
bpo-37321@footnote{https://bugs.python.org/issue37321}: Both subprocess connection error messages now refer to the
‘Startup failure’ section of the IDLE doc.

@item 
bpo-17535@footnote{https://bugs.python.org/issue17535}: Add optional line numbers for IDLE editor windows.  Windows
open without line numbers unless set otherwise in the General tab of the
configuration dialog.

@item 
bpo-26806@footnote{https://bugs.python.org/issue26806}: To compensate for stack frames added by IDLE and avoid possible
problems with low recursion limits, add 30 to limits in the user code
execution process.  Subtract 30 when reporting recursion limits to make
this addition mostly transparent.

@item 
bpo-37177@footnote{https://bugs.python.org/issue37177}: Properly ‘attach’ search dialogs to their main window so that
they behave like other dialogs and do not get hidden behind their main
window.

@item 
bpo-37039@footnote{https://bugs.python.org/issue37039}: Adjust “Zoom Height” to individual screens by momentarily
maximizing the window on first use with a particular screen.  Changing
screen settings may invalidate the saved height.  While a window is
maximized, “Zoom Height” has no effect.

@item 
bpo-35763@footnote{https://bugs.python.org/issue35763}: Make calltip reminder about ‘/’ meaning positional-only less
obtrusive by only adding it when there is room on the first line.

@item 
bpo-5680@footnote{https://bugs.python.org/issue5680}: Add ‘Run… Customized’ to the Run menu to run a module with
customized settings.  Any ‘command line arguments’ entered are added to
sys.argv. One can suppress the normal Shell main module restart.

@item 
bpo-36390@footnote{https://bugs.python.org/issue36390}: Gather Format menu functions into format.py.  Combine
paragraph.py, rstrip.py, and format methods from editor.py.
@end itemize

@node Tools/Demos<7>,C API<21>,IDLE<19>,Python 3 9 0 alpha 1
@anchor{whatsnew/changelog id188}@anchor{1283}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-38118@footnote{https://bugs.python.org/issue38118}: Update Valgrind suppression file to ignore a false alarm in
@ref{1284,,PyUnicode_Decode()} when using GCC builtin strcmp().

@item 
bpo-38347@footnote{https://bugs.python.org/issue38347}: pathfix.py: Assume all files that end on ‘.py’ are Python
scripts when working recursively.

@item 
bpo-37803@footnote{https://bugs.python.org/issue37803}: pdb’s @code{--help} and @code{--version} long options now work.

@item 
bpo-37942@footnote{https://bugs.python.org/issue37942}: Improve ArgumentClinic converter for floats.

@item 
bpo-37704@footnote{https://bugs.python.org/issue37704}: Remove @code{Tools/scripts/h2py.py}: use cffi to access a C API in
Python.

@item 
bpo-37675@footnote{https://bugs.python.org/issue37675}: 2to3 now works when run from a zipped standard library.

@item 
bpo-37034@footnote{https://bugs.python.org/issue37034}: Argument Clinic now uses the argument name on errors with
keyword-only argument instead of their position. Patch contributed by Rémi
Lapeyre.

@item 
bpo-37064@footnote{https://bugs.python.org/issue37064}: Add option -k to pathscript.py script: preserve shebang flags.
Add option -a to pathscript.py script: add flags.
@end itemize

@node C API<21>,,Tools/Demos<7>,Python 3 9 0 alpha 1
@anchor{whatsnew/changelog id189}@anchor{1285}
@subsubsection C API



@itemize -

@item 
bpo-37633@footnote{https://bugs.python.org/issue37633}: Re-export some function compatibility wrappers for macros in
@code{pythonrun.h}.

@item 
bpo-38644@footnote{https://bugs.python.org/issue38644}: Provide @ref{47a,,Py_EnterRecursiveCall()} and
@ref{47b,,Py_LeaveRecursiveCall()} as regular functions for the limited API.
Previously, there were defined as macros, but these macros didn’t work
with the limited API which cannot access @code{PyThreadState.recursion_depth}
field. Remove @code{_Py_CheckRecursionLimit} from the stable ABI.

@item 
bpo-38650@footnote{https://bugs.python.org/issue38650}: The global variable @ref{48b,,PyStructSequence_UnnamedField} is
now a constant and refers to a constant string.

@item 
bpo-38540@footnote{https://bugs.python.org/issue38540}: Fixed possible leak in @ref{1286,,PyArg_Parse()} and similar
functions for format units @code{"es#"} and @code{"et#"} when the macro
@code{PY_SSIZE_T_CLEAN} is not defined.

@item 
bpo-38395@footnote{https://bugs.python.org/issue38395}: Fix a crash in @ref{567,,weakref.proxy} objects due to incorrect
lifetime management when calling some associated methods that may delete
the last reference to object being referenced by the proxy. Patch by Pablo
Galindo.

@item 
bpo-36389@footnote{https://bugs.python.org/issue36389}: The @code{_PyObject_CheckConsistency()} function is now also
available in release mode. For example, it can be used to debug a crash in
the @code{visit_decref()} function of the GC.

@item 
bpo-38266@footnote{https://bugs.python.org/issue38266}: Revert the removal of PyThreadState_DeleteCurrent() with
documentation.

@item 
bpo-38303@footnote{https://bugs.python.org/issue38303}: Update audioop extension module to use the stable ABI
(PEP-384). Patch by Tyler Kieft.

@item 
bpo-38234@footnote{https://bugs.python.org/issue38234}: @ref{202,,Py_SetPath()} now sets @ref{585,,sys.executable} to the
program full path (@ref{38a,,Py_GetProgramFullPath()}) rather than to the
program name (@ref{38c,,Py_GetProgramName()}).

@item 
bpo-38234@footnote{https://bugs.python.org/issue38234}: Python ignored arguments passed to @ref{202,,Py_SetPath()},
@ref{204,,Py_SetPythonHome()} and @ref{203,,Py_SetProgramName()}: fix Python
initialization to use specified arguments.

@item 
bpo-38205@footnote{https://bugs.python.org/issue38205}: The @ref{6e8,,Py_UNREACHABLE()} macro now calls
@ref{48f,,Py_FatalError()}.

@item 
bpo-38140@footnote{https://bugs.python.org/issue38140}: Make dict and weakref offsets opaque for C heap types by
passing the offsets through PyMemberDef

@item 
bpo-15088@footnote{https://bugs.python.org/issue15088}: The C function @code{PyGen_NeedsFinalizing} has been removed. It
was not documented, tested or used anywhere within CPython after the
implementation of 
@geindex Python Enhancement Proposals; PEP 442
PEP 442@footnote{https://www.python.org/dev/peps/pep-0442}. Patch by Joannah Nanjekye. (Patch by Joannah
Nanjekye)

@item 
bpo-36763@footnote{https://bugs.python.org/issue36763}: Options added by @code{PySys_AddXOption()} are now handled the
same way than @code{PyConfig.xoptions} and command line @code{-X} options.

@item 
bpo-37926@footnote{https://bugs.python.org/issue37926}: Fix a crash in @code{PySys_SetArgvEx(0, NULL, 0)}.

@item 
bpo-37879@footnote{https://bugs.python.org/issue37879}: Fix subtype_dealloc to suppress the type decref when the base
type is a C heap type

@item 
bpo-37645@footnote{https://bugs.python.org/issue37645}: Add @code{_PyObject_FunctionStr()} to get a user-friendly
string representation of a function-like object. Patch by Jeroen Demeyer.

@item 
bpo-29548@footnote{https://bugs.python.org/issue29548}: The functions @code{PyEval_CallObject}, @code{PyEval_CallFunction},
@code{PyEval_CallMethod} and @code{PyEval_CallObjectWithKeywords} are
deprecated. Use @ref{46a,,PyObject_Call()} and its variants instead.

@item 
bpo-37151@footnote{https://bugs.python.org/issue37151}: @code{PyCFunction_Call} is now a deprecated alias of
@ref{46a,,PyObject_Call()}.

@item 
bpo-37540@footnote{https://bugs.python.org/issue37540}: The vectorcall protocol now requires that the caller passes
only strings as keyword names.

@item 
bpo-37207@footnote{https://bugs.python.org/issue37207}: The vectorcall protocol is now enabled for @code{type} objects:
set @code{tp_vectorcall} to a vectorcall function to be used instead of
@code{tp_new} and @code{tp_init} when calling the class itself.

@item 
bpo-21120@footnote{https://bugs.python.org/issue21120}: Exclude Python-ast.h, ast.h and asdl.h from the limited API.

@item 
bpo-37483@footnote{https://bugs.python.org/issue37483}: Add new function @code{_PyObject_CallOneArg} for calling an object
with one positional argument.

@item 
bpo-36763@footnote{https://bugs.python.org/issue36763}: Add @code{PyConfig_SetWideStringList()} function.

@item 
bpo-37337@footnote{https://bugs.python.org/issue37337}: Add fast functions for calling methods:
@code{_PyObject_VectorcallMethod()}, @code{_PyObject_CallMethodNoArgs()}
and @code{_PyObject_CallMethodOneArg()}.

@item 
bpo-28805@footnote{https://bugs.python.org/issue28805}: The @ref{1287,,METH_FASTCALL} calling convention has been
documented.

@item 
bpo-37221@footnote{https://bugs.python.org/issue37221}: The new function @ref{583,,PyCode_NewWithPosOnlyArgs()} allows to
create code objects like @ref{584,,PyCode_New()}, but with an extra
@emph{posonlyargcount} parameter for indicating the number of positonal-only
arguments.

@item 
bpo-37215@footnote{https://bugs.python.org/issue37215}: Fix dtrace issue introduce by bpo-36842@footnote{https://bugs.python.org/issue36842}

@item 
bpo-37194@footnote{https://bugs.python.org/issue37194}: Add a new public @ref{479,,PyObject_CallNoArgs()} function to the
C API: call a callable Python object without any arguments. It is the most
efficient way to call a callback without any argument. On x86-64, for
example, @code{PyObject_CallFunctionObjArgs(func, NULL)} allocates 960 bytes
on the stack per call, whereas @code{PyObject_CallNoArgs(func)} only
allocates 624 bytes per call.

@item 
bpo-37170@footnote{https://bugs.python.org/issue37170}: Fix the cast on error in
@ref{1288,,PyLong_AsUnsignedLongLongMask()}.

@item 
bpo-35381@footnote{https://bugs.python.org/issue35381}: Convert posixmodule.c statically allocated types
@code{DirEntryType} and @code{ScandirIteratorType} to heap-allocated types.

@item 
bpo-34331@footnote{https://bugs.python.org/issue34331}: Use singular/plural noun in error message when instantiating an
abstract class with non-overriden abstract method(s).
@end itemize

@node Python 3 8 0 beta 1,Python 3 8 0 alpha 4,Python 3 9 0 alpha 1,Changelog
@anchor{whatsnew/changelog python-3-8-0-beta-1}@anchor{1289}
@subsection Python 3.8.0 beta 1


@emph{Release date: 2019-06-04}

@menu
* Security: Security<15>. 
* Core and Builtins: Core and Builtins<22>. 
* Library: Library<22>. 
* Documentation: Documentation<23>. 
* Tests: Tests<21>. 
* Build: Build<22>. 
* Windows: Windows<20>. 
* macOS: macOS<18>. 
* IDLE: IDLE<20>. 
* Tools/Demos: Tools/Demos<8>. 
* C API: C API<22>. 

@end menu

@node Security<15>,Core and Builtins<22>,,Python 3 8 0 beta 1
@anchor{whatsnew/changelog id190}@anchor{128a}
@subsubsection Security



@itemize -

@item 
bpo-35907@footnote{https://bugs.python.org/issue35907}: CVE-2019-9948: Avoid file reading by disallowing
@code{local-file://} and @code{local_file://} URL schemes in
@code{URLopener().open()} and @code{URLopener().retrieve()} of
@ref{127,,urllib.request}.

@item 
bpo-33529@footnote{https://bugs.python.org/issue33529}: Prevent fold function used in email header encoding from
entering infinite loop when there are too many non-ASCII characters in a
header.

@item 
bpo-33164@footnote{https://bugs.python.org/issue33164}: Updated blake2 implementation which uses secure memset
implementation provided by platform.
@end itemize

@node Core and Builtins<22>,Library<22>,Security<15>,Python 3 8 0 beta 1
@anchor{whatsnew/changelog id191}@anchor{128b}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-35814@footnote{https://bugs.python.org/issue35814}: Allow unpacking in the right hand side of annotated
assignments. In particular, @code{t: Tuple[int, ...] = x, y, *z} is now
allowed.

@item 
bpo-37126@footnote{https://bugs.python.org/issue37126}: All structseq objects are now tracked by the garbage collector.
Patch by Pablo Galindo.

@item 
bpo-37122@footnote{https://bugs.python.org/issue37122}: Make the @emph{co_argcount} attribute of code objects represent the
total number of positional arguments (including positional-only
arguments). The value of @emph{co_posonlyargcount} can be used to distinguish
which arguments are positional only, and the difference (@emph{co_argcount} -
@emph{co_posonlyargcount}) is the number of positional-or-keyword arguments.
Patch by Pablo Galindo.

@item 
bpo-20092@footnote{https://bugs.python.org/issue20092}: Constructors of @ref{1b1,,int}, @ref{256,,float} and
@ref{33b,,complex} will now use the @ref{24a,,__index__()} special
method, if available and the corresponding method @ref{249,,__int__()},
@ref{4c3,,__float__()} or @ref{4c4,,__complex__()} is not available.

@item 
bpo-37087@footnote{https://bugs.python.org/issue37087}: Add native thread ID (TID) support to OpenBSD.

@item 
bpo-26219@footnote{https://bugs.python.org/issue26219}: Implemented per opcode cache mechanism and @code{LOAD_GLOBAL}
instruction use it. @code{LOAD_GLOBAL} is now about 40% faster. Contributed
by Yury Selivanov, and Inada Naoki.

@item 
bpo-37072@footnote{https://bugs.python.org/issue37072}: Fix crash in PyAST_FromNodeObject() when flags is NULL.

@item 
bpo-37029@footnote{https://bugs.python.org/issue37029}: Freeing a great many small objects could take time quadratic in
the number of arenas, due to using linear search to keep @code{obmalloc.c}’s
list of usable arenas sorted by order of number of free memory pools.
This is accomplished without search now, leaving the worst-case time
linear in the number of arenas.  For programs where this quite visibly
matters (typically with more than 100 thousand small objects alive
simultaneously), this can greatly reduce the time needed to release their
memory.

@item 
bpo-26423@footnote{https://bugs.python.org/issue26423}: Fix possible overflow in @code{wrap_lenfunc()} when @code{sizeof(long)
< sizeof(Py_ssize_t)} (e.g., 64-bit Windows).

@item 
bpo-37050@footnote{https://bugs.python.org/issue37050}: Improve the AST for “debug” f-strings, which use ‘=’ to print
out the source of the expression being evaluated.  Delete expr_text from
the FormattedValue node, and instead use a Constant string node (possibly
merged with adjacent constant expressions inside the f-string).

@item 
bpo-22385@footnote{https://bugs.python.org/issue22385}: The @ref{8c1,,bytes.hex}, @ref{8c2,,bytearray.hex}, and @ref{8c3,,memoryview.hex} methods
as well as the @ref{128c,,binascii.hexlify} and @code{b2a_hex} functions now have the
ability to include an optional separator between hex bytes.  This
functionality was inspired by MicroPython’s hexlify implementation.

@item 
bpo-26836@footnote{https://bugs.python.org/issue26836}: Add @ref{515,,os.memfd_create()}.

@item 
bpo-37032@footnote{https://bugs.python.org/issue37032}: Added new @code{replace()} method to the code type
(@ref{4cb,,types.CodeType}).

@item 
bpo-37007@footnote{https://bugs.python.org/issue37007}: Implement @ref{53e,,socket.if_nameindex()},
@ref{53f,,socket.if_nametoindex()}, and @ref{540,,socket.if_indextoname()} on
Windows.

@item 
bpo-36829@footnote{https://bugs.python.org/issue36829}: @ref{128d,,PyErr_WriteUnraisable()} now creates a traceback object
if there is no current traceback. Moreover, call
@ref{128e,,PyErr_NormalizeException()} and @ref{128f,,PyException_SetTraceback()}
to normalize the exception value. Ignore any error.

@item 
bpo-36878@footnote{https://bugs.python.org/issue36878}: Only accept text after @code{# type: ignore} if the first character
is ASCII. This is to disallow things like @code{# type: ignoreé}.

@item 
bpo-36878@footnote{https://bugs.python.org/issue36878}: Store text appearing after a @code{# type: ignore} comment in the
AST. For example a type ignore like @code{# type: ignore[E1000]} will have the
string @code{"[E1000]"} stored in its AST node.

@item 
bpo-2180@footnote{https://bugs.python.org/issue2180}: Treat line continuation at EOF as a @code{SyntaxError} by Anthony
Sottile.

@item 
bpo-36907@footnote{https://bugs.python.org/issue36907}: Fix a crash when calling a C function with a keyword dict
(@code{f(**kwargs)}) and changing the dict @code{kwargs} while that function is
running.

@item 
bpo-36946@footnote{https://bugs.python.org/issue36946}: Fix possible signed integer overflow when handling slices.

@item 
bpo-36826@footnote{https://bugs.python.org/issue36826}: Add NamedExpression kind support to ast_unparse.c

@item 
bpo-1875@footnote{https://bugs.python.org/issue1875}: A @ref{21a,,SyntaxError} is now raised if a code blocks that will be
optimized away (e.g. if conditions that are always false) contains syntax
errors. Patch by Pablo Galindo.

@item 
bpo-36027@footnote{https://bugs.python.org/issue36027}: Allow computation of modular inverses via three-argument
@code{pow}: the second argument is now permitted to be negative in the case
where the first and third arguments are relatively prime.

@item 
bpo-36861@footnote{https://bugs.python.org/issue36861}: Update the Unicode database to version 12.1.0.

@item 
bpo-28866@footnote{https://bugs.python.org/issue28866}: Avoid caching attributes of classes which type defines mro() to
avoid a hard cache invalidation problem.

@item 
bpo-36851@footnote{https://bugs.python.org/issue36851}: The @code{FrameType} stack is now correctly cleaned up if the
execution ends with a return and the stack is not empty.

@item 
bpo-34616@footnote{https://bugs.python.org/issue34616}: The @code{compile()} builtin functions now support the
@code{ast.PyCF_ALLOW_TOP_LEVEL_AWAIT} flag,  which allow to compile sources
that  contains top-level @code{await}, @code{async with} or @code{async for}. This
is useful to evaluate async-code from with an already async functions; for
example in a custom REPL.

@item 
bpo-36842@footnote{https://bugs.python.org/issue36842}: Implement PEP 578, adding sys.audit, io.open_code and related
APIs.

@item 
bpo-27639@footnote{https://bugs.python.org/issue27639}: Correct return type for UserList slicing operations. Patch by
Michael Blahay, Erick Cervantes, and vaultah

@item 
bpo-36737@footnote{https://bugs.python.org/issue36737}: Move PyRuntimeState.warnings into per-interpreter state (via
“module state”).

@item 
bpo-36793@footnote{https://bugs.python.org/issue36793}: Removed @code{__str__} implementations from builtin types
@ref{2f7,,bool}, @ref{1b1,,int}, @ref{256,,float}, @ref{33b,,complex} and few
classes from the standard library. They now inherit @code{__str__()} from
@ref{5a5,,object}.

@item 
bpo-36817@footnote{https://bugs.python.org/issue36817}: Add a @code{=} feature f-strings for debugging. This can precede
@code{!s}, @code{!r}, or @code{!a}. It produces the text of the expression,
followed by an equal sign, followed by the repr of the value of the
expression. So @code{f'@{3*9+15=@}'} would be equal to the string
@code{'3*9+15=42'}.  If @code{=} is specified, the default conversion is set to
@code{!r}, unless a format spec is given, in which case the formatting
behavior is unchanged, and __format__ will be used.

@item 
bpo-24048@footnote{https://bugs.python.org/issue24048}: Save the live exception during import.c’s @code{remove_module()}.

@item 
bpo-27987@footnote{https://bugs.python.org/issue27987}: pymalloc returns memory blocks aligned by 16 bytes, instead of
8 bytes, on 64-bit platforms to conform x86-64 ABI. Recent compilers
assume this alignment more often. Patch by Inada Naoki.

@item 
bpo-36601@footnote{https://bugs.python.org/issue36601}: A long-since-meaningless check for @code{getpid() == main_pid} was
removed from Python’s internal C signal handler.

@item 
bpo-36594@footnote{https://bugs.python.org/issue36594}: Fix incorrect use of @code{%p} in format strings. Patch by Zackery
Spytz.

@item 
bpo-36045@footnote{https://bugs.python.org/issue36045}: builtins.help() now prefixes @code{async} for async functions

@item 
bpo-36084@footnote{https://bugs.python.org/issue36084}: Add native thread ID (TID) to threading.Thread objects
(supported platforms: Windows, FreeBSD, Linux, macOS)

@item 
bpo-36035@footnote{https://bugs.python.org/issue36035}: Added fix for broken symlinks in combination with pathlib

@item 
bpo-35983@footnote{https://bugs.python.org/issue35983}: Added new trashcan macros to deal with a double deallocation
that could occur when the @code{tp_dealloc} of a subclass calls the
@code{tp_dealloc} of a base class and that base class uses the trashcan
mechanism. Patch by Jeroen Demeyer.

@item 
bpo-20602@footnote{https://bugs.python.org/issue20602}: Do not clear @ref{22f,,sys.flags} and @ref{1290,,sys.float_info}
during shutdown. Patch by Zackery Spytz.

@item 
bpo-26826@footnote{https://bugs.python.org/issue26826}: Expose @code{copy_file_range()} as a low level API in the
@ref{c0,,os} module.

@item 
bpo-32388@footnote{https://bugs.python.org/issue32388}: Remove cross-version binary compatibility requirement in
tp_flags.

@item 
bpo-31862@footnote{https://bugs.python.org/issue31862}: Port binascii to PEP 489 multiphase initialization. Patch by
Marcel Plch.
@end itemize

@node Library<22>,Documentation<23>,Core and Builtins<22>,Python 3 8 0 beta 1
@anchor{whatsnew/changelog id192}@anchor{1291}
@subsubsection Library



@itemize -

@item 
bpo-37128@footnote{https://bugs.python.org/issue37128}: Added @ref{19c,,math.perm()}.

@item 
bpo-37120@footnote{https://bugs.python.org/issue37120}: Add SSLContext.num_tickets to control the number of TLSv1.3
session tickets.

@item 
bpo-12202@footnote{https://bugs.python.org/issue12202}: Fix the error handling in
@ref{1292,,msilib.SummaryInformation.GetProperty()}. Patch by Zackery Spytz.

@item 
bpo-26835@footnote{https://bugs.python.org/issue26835}: The fcntl module now contains file sealing constants for
sealing of memfds.

@item 
bpo-29262@footnote{https://bugs.python.org/issue29262}: Add @code{get_origin()} and @code{get_args()} introspection helpers
to @code{typing} module.

@item 
bpo-12639@footnote{https://bugs.python.org/issue12639}: @ref{1293,,msilib.Directory.start_component()} no longer fails if
@emph{keyfile} is not @code{None}.

@item 
bpo-36999@footnote{https://bugs.python.org/issue36999}: Add the @code{asyncio.Task.get_coro()} method to publicly expose
the tasks’s coroutine object.

@item 
bpo-35246@footnote{https://bugs.python.org/issue35246}: Make @ref{591,,asyncio.create_subprocess_exec()} accept path-like
arguments.

@item 
bpo-35279@footnote{https://bugs.python.org/issue35279}: Change default @emph{max_workers} of @code{ThreadPoolExecutor} from
@code{cpu_count() * 5} to @code{min(32, cpu_count() + 4)}.  Previous value was
unreasonably large on many cores machines.

@item 
bpo-37076@footnote{https://bugs.python.org/issue37076}: @ref{1294,,_thread.start_new_thread()} now logs uncaught exception
raised by the function using @ref{18d,,sys.unraisablehook()}, rather than
@ref{28e,,sys.excepthook()}, so the hook gets access to the function which
raised the exception.

@item 
bpo-33725@footnote{https://bugs.python.org/issue33725}: On macOS, the @ref{b3,,multiprocessing} module now uses @emph{spawn}
start method by default.

@item 
bpo-37054@footnote{https://bugs.python.org/issue37054}: Fix destructor @code{_pyio.BytesIO} and
@code{_pyio.TextIOWrapper}: initialize their @code{_buffer} attribute as
soon as possible (in the class body), because it’s used by @code{__del__()}
which calls @code{close()}.

@item 
bpo-37058@footnote{https://bugs.python.org/issue37058}: PEP 544: Add @code{Protocol} and @code{@@runtime_checkable} to the
@code{typing} module.

@item 
bpo-36933@footnote{https://bugs.python.org/issue36933}: The functions @code{sys.set_coroutine_wrapper} and
@code{sys.get_coroutine_wrapper} that were deprecated and marked for removal
in 3.8 have been removed.

@item 
bpo-37047@footnote{https://bugs.python.org/issue37047}: Handle late binding and attribute access in
@ref{560,,unittest.mock.AsyncMock} setup for autospeccing. Document newly
implemented async methods in @ref{a01,,unittest.mock.MagicMock}.

@item 
bpo-37049@footnote{https://bugs.python.org/issue37049}: PEP 589: Add @code{TypedDict} to the @code{typing} module.

@item 
bpo-37046@footnote{https://bugs.python.org/issue37046}: PEP 586: Add @code{Literal} to the @code{typing} module.

@item 
bpo-37045@footnote{https://bugs.python.org/issue37045}: PEP 591: Add @code{Final} qualifier and @code{@@final} decorator to
the @code{typing} module.

@item 
bpo-37035@footnote{https://bugs.python.org/issue37035}: Don’t log OSError based exceptions if a fatal error has
occurred in asyncio transport. Peer can generate almost any OSError, user
cannot avoid these exceptions by fixing own code. Errors are still
propagated to user code, it’s just logging them is pointless and pollute
asyncio logs.

@item 
bpo-37001@footnote{https://bugs.python.org/issue37001}: @ref{1295,,symtable.symtable()} now accepts the same input types for
source code as the built-in @ref{4e2,,compile()} function. Patch by Dino
Viehland.

@item 
bpo-37028@footnote{https://bugs.python.org/issue37028}: Implement asyncio REPL

@item 
bpo-37027@footnote{https://bugs.python.org/issue37027}: Return safe to use proxy socket object from
transport.get_extra_info(‘socket’)

@item 
bpo-32528@footnote{https://bugs.python.org/issue32528}: Make asyncio.CancelledError a BaseException.

This will address the common mistake many asyncio users make: an “except
Exception” clause breaking Tasks cancellation.

In addition to this change, we stop inheriting asyncio.TimeoutError and
asyncio.InvalidStateError from their concurrent.futures.* counterparts.
There’s no point for these exceptions to share the inheritance chain.

@item 
bpo-1230540@footnote{https://bugs.python.org/issue1230540}: Add a new @ref{2d1,,threading.excepthook()} function which handles
uncaught @ref{54e,,threading.Thread.run()} exception. It can be overridden to
control how uncaught @ref{54e,,threading.Thread.run()} exceptions are handled.

@item 
bpo-36996@footnote{https://bugs.python.org/issue36996}: Handle @ref{a02,,unittest.mock.patch()} used as a decorator on async
functions.

@item 
bpo-37008@footnote{https://bugs.python.org/issue37008}: Add support for calling @ref{252,,next()} with the mock resulting
from @ref{1296,,unittest.mock.mock_open()}

@item 
bpo-27737@footnote{https://bugs.python.org/issue27737}: Allow whitespace only header encoding in @code{email.header} - by
Batuhan Taskaya

@item 
bpo-36969@footnote{https://bugs.python.org/issue36969}: PDB command @code{args} now  display positional only arguments.
Patch contributed by Rémi Lapeyre.

@item 
bpo-36969@footnote{https://bugs.python.org/issue36969}: PDB command @code{args} now  display keyword only arguments. Patch
contributed by Rémi Lapeyre.

@item 
bpo-36983@footnote{https://bugs.python.org/issue36983}: Add missing names to @code{typing.__all__}: @code{ChainMap},
@code{ForwardRef}, @code{OrderedDict} - by Anthony Sottile.

@item 
bpo-36972@footnote{https://bugs.python.org/issue36972}: Add SupportsIndex protocol to the typing module to allow type
checking to detect classes that can be passed to @ref{e6f,,hex()}, @ref{e6e,,oct()} and
@ref{e4e,,bin()}.

@item 
bpo-32972@footnote{https://bugs.python.org/issue32972}: Implement @code{unittest.IsolatedAsyncioTestCase} to help testing
asyncio-based code.

@item 
bpo-36952@footnote{https://bugs.python.org/issue36952}: @ref{280,,fileinput.input()} and @ref{1bc,,fileinput.FileInput}
@strong{bufsize} argument has been removed (was deprecated and ignored since
Python 3.6), and as a result the @strong{mode} and @strong{openhook} arguments have
been made keyword-only.

@item 
bpo-36952@footnote{https://bugs.python.org/issue36952}: Starting with Python 3.3, importing ABCs from
@ref{1c,,collections} is deprecated, and import should be done from
@ref{1d,,collections.abc}. Still being able to import from @ref{1c,,collections}
was marked for removal in 3.8, but has been delayed to 3.9; documentation
and @code{DeprecationWarning} clarified.

@item 
bpo-36949@footnote{https://bugs.python.org/issue36949}: Implement __repr__ for WeakSet objects.

@item 
bpo-36948@footnote{https://bugs.python.org/issue36948}: Fix @ref{21f,,NameError} in
@ref{1297,,urllib.request.URLopener.retrieve()}. Patch by Karthikeyan
Singaravelan.

@item 
bpo-33524@footnote{https://bugs.python.org/issue33524}: Fix the folding of email header when the max_line_length is 0
or None and the header contains non-ascii characters.  Contributed by
Licht Takeuchi (@@Licht-T).

@item 
bpo-24564@footnote{https://bugs.python.org/issue24564}: @ref{cc3,,shutil.copystat()} now ignores @ref{1298,,errno.EINVAL} on
@ref{c9a,,os.setxattr()} which may occur when copying files on filesystems
without extended attributes support.

Original patch by Giampaolo Rodola, updated by Ying Wang.

@item 
bpo-36888@footnote{https://bugs.python.org/issue36888}: Python child processes can now access the status of their
parent process using multiprocessing.process.parent_process

@item 
bpo-36921@footnote{https://bugs.python.org/issue36921}: Deprecate @code{@@coroutine} for sake of @code{async def}.

@item 
bpo-25652@footnote{https://bugs.python.org/issue25652}: Fix bug in @code{__rmod__} of @code{UserString} - by Batuhan Taskaya.

@item 
bpo-36916@footnote{https://bugs.python.org/issue36916}: Remove a message about an unhandled exception in a task when
writer.write() is used without await and writer.drain() fails with an
exception.

@item 
bpo-36889@footnote{https://bugs.python.org/issue36889}: Introduce @code{asyncio.Stream} class that merges
@ref{31d,,asyncio.StreamReader} and @ref{1299,,asyncio.StreamWriter}
functionality. @code{asyncio.Stream} can work in readonly, writeonly and
readwrite modes. Provide @code{asyncio.connect()},
@code{asyncio.connect_unix()}, @code{asyncio.connect_read_pipe()} and
@code{asyncio.connect_write_pipe()} factories to open
@code{asyncio.Stream} connections. Provide @code{asyncio.StreamServer}
and @code{UnixStreamServer} to serve servers with asyncio.Stream API.
Modify @ref{592,,asyncio.create_subprocess_shell()} and
@ref{591,,asyncio.create_subprocess_exec()} to use @code{asyncio.Stream}
instead of deprecated @code{StreamReader} and @code{StreamWriter}.
Deprecate @ref{31d,,asyncio.StreamReader} and @ref{1299,,asyncio.StreamWriter}.
Deprecate usage of private classes, e.g.
@code{asyncio.FlowControlMixing} and
@code{asyncio.StreamReaderProtocol} outside of asyncio package.

@item 
bpo-36845@footnote{https://bugs.python.org/issue36845}: Added validation of integer prefixes to the construction of IP
networks and interfaces in the ipaddress module.

@item 
bpo-23378@footnote{https://bugs.python.org/issue23378}: Add an extend action to argparser.

@item 
bpo-36867@footnote{https://bugs.python.org/issue36867}: Fix a bug making a SharedMemoryManager instance and its parent
process use two separate resource_tracker processes.

@item 
bpo-23896@footnote{https://bugs.python.org/issue23896}: Adds a grammar to lib2to3.pygram that contains exec as a
function not as statement.

@item 
bpo-36895@footnote{https://bugs.python.org/issue36895}: The function @code{time.clock()} was deprecated in 3.3 in favor of
@code{time.perf_counter()} and marked for removal in 3.8, it has removed.

@item 
bpo-35545@footnote{https://bugs.python.org/issue35545}: Fix asyncio discarding IPv6 scopes when ensuring hostname
resolutions internally

@item 
bpo-36887@footnote{https://bugs.python.org/issue36887}: Add new function @ref{50e,,math.isqrt()} to compute integer square
roots.

@item 
bpo-34632@footnote{https://bugs.python.org/issue34632}: Introduce the @code{importlib.metadata} module with (provisional)
support for reading metadata from third-party packages.

@item 
bpo-36878@footnote{https://bugs.python.org/issue36878}: When using @code{type_comments=True} in @ref{4d3,,ast.parse}, treat @code{# type:
ignore} followed by a non-alphanumeric character and then arbitrary text
as a type ignore, instead of requiring nothing but whitespace or another
comment. This is to permit formations such as @code{# type: ignore[E1000]}.

@item 
bpo-36778@footnote{https://bugs.python.org/issue36778}: @code{cp65001} encoding (Windows code page 65001) becomes an alias
to @code{utf_8} encoding.

@item 
bpo-36867@footnote{https://bugs.python.org/issue36867}: The multiprocessing.resource_tracker replaces the
multiprocessing.semaphore_tracker module. Other than semaphores,
resource_tracker also tracks shared_memory segments.

@item 
bpo-30262@footnote{https://bugs.python.org/issue30262}: The @code{Cache} and @code{Statement} objects of the @ref{ed,,sqlite3}
module are not exposed to the user.  Patch by Aviv Palivoda.

@item 
bpo-24538@footnote{https://bugs.python.org/issue24538}: In @ref{cc3,,shutil.copystat()}, first copy extended file attributes and
then file permissions, since extended attributes can only be set on the
destination while it is still writeable.

@item 
bpo-36829@footnote{https://bugs.python.org/issue36829}: Add new @ref{18d,,sys.unraisablehook()} function which can be
overridden to control how “unraisable exceptions” are handled. It is
called when an exception has occurred but there is no way for Python to
handle it. For example, when a destructor raises an exception or during
garbage collection (@ref{54b,,gc.collect()}).

@item 
bpo-36832@footnote{https://bugs.python.org/issue36832}: Introducing @code{zipfile.Path}, a pathlib-compatible wrapper for
traversing zip files.

@item 
bpo-36814@footnote{https://bugs.python.org/issue36814}: Fix an issue where os.posix_spawnp() would incorrectly raise a
TypeError when file_actions is None.

@item 
bpo-33110@footnote{https://bugs.python.org/issue33110}: Handle exceptions raised by functions added by
concurrent.futures add_done_callback correctly when the Future has already
completed.

@item 
bpo-26903@footnote{https://bugs.python.org/issue26903}: Limit @code{max_workers} in @code{ProcessPoolExecutor} to 61 to work
around a WaitForMultipleObjects limitation.

@item 
bpo-36813@footnote{https://bugs.python.org/issue36813}: Fix @ref{993,,QueueListener} to call
@code{queue.task_done()} upon stopping. Patch by Bar Harel.

@item 
bpo-36806@footnote{https://bugs.python.org/issue36806}: Forbid creation of asyncio stream objects like StreamReader,
StreamWriter, Process, and their protocols outside of asyncio package.

@item 
bpo-36802@footnote{https://bugs.python.org/issue36802}: Provide both sync and async calls for StreamWriter.write() and
StreamWriter.close()

@item 
bpo-36801@footnote{https://bugs.python.org/issue36801}: Properly handle SSL connection closing in asyncio
StreamWriter.drain() call.

@item 
bpo-36785@footnote{https://bugs.python.org/issue36785}: Implement PEP 574 (pickle protocol 5 with out-of-band buffers).

@item 
bpo-36772@footnote{https://bugs.python.org/issue36772}: functools.lru_cache() can now be used as a straight decorator
in addition to its existing usage as a function that returns a decorator.

@item 
bpo-6584@footnote{https://bugs.python.org/issue6584}: Add a @ref{4fd,,BadGzipFile} exception to the @ref{87,,gzip}
module.

@item 
bpo-36748@footnote{https://bugs.python.org/issue36748}: Optimized write buffering in C implementation of
@code{TextIOWrapper}. Writing ASCII string to @code{TextIOWrapper} with ascii,
latin1, or utf-8 encoding is about 20% faster.  Patch by Inada Naoki.

@item 
bpo-8138@footnote{https://bugs.python.org/issue8138}: Don’t mark @code{wsgiref.simple_server.SimpleServer} as
multi-threaded since @code{wsgiref.simple_server.WSGIServer} is
single-threaded.

@item 
bpo-22640@footnote{https://bugs.python.org/issue22640}: @ref{534,,py_compile.compile()} now supports silent mode. Patch by
Joannah Nanjekye

@item 
bpo-29183@footnote{https://bugs.python.org/issue29183}: Fix double exceptions in @ref{129a,,wsgiref.handlers.BaseHandler}
by calling its @code{close()} method only
when no exception is raised.

@item 
bpo-36548@footnote{https://bugs.python.org/issue36548}: Improved the repr of regular expression flags.

@item 
bpo-36542@footnote{https://bugs.python.org/issue36542}: The signature of Python functions can now be overridden by
specifying the @code{__text_signature__} attribute.

@item 
bpo-36533@footnote{https://bugs.python.org/issue36533}: Reinitialize logging.Handler locks in forked child processes
instead of attempting to acquire them all in the parent before forking
only to be released in the child process.  The acquire/release pattern was
leading to deadlocks in code that has implemented any form of chained
logging handlers that depend upon one another as the lock acquisition
order cannot be guaranteed.

@item 
bpo-35252@footnote{https://bugs.python.org/issue35252}: Throw a TypeError instead of an AssertionError when using an
invalid type annotation with singledispatch.

@item 
bpo-35900@footnote{https://bugs.python.org/issue35900}: Allow reduction methods to return a 6-item tuple where the 6th
item specifies a custom state-setting method that’s called instead of the
regular @code{__setstate__} method.

@item 
bpo-35900@footnote{https://bugs.python.org/issue35900}: enable custom reduction callback registration for functions and
classes in _pickle.c, using the new Pickler’s attribute
@code{reducer_override}

@item 
bpo-36368@footnote{https://bugs.python.org/issue36368}: Fix a bug crashing SharedMemoryManager instances in interactive
sessions after a ctrl-c (KeyboardInterrupt) was sent

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Fix mmap fail for VxWorks

@item 
bpo-27497@footnote{https://bugs.python.org/issue27497}: @ref{da0,,csv.DictWriter.writeheader()} now returns the return value
of the underlying @code{csv.Writer.writerow()} method. Patch contributed by
Ashish Nitin Patil.

@item 
bpo-36239@footnote{https://bugs.python.org/issue36239}: Parsing .mo files now ignores comments starting and ending with
#-#-#-#-#.

@item 
bpo-26707@footnote{https://bugs.python.org/issue26707}: Enable plistlib to read and write binary plist files that were
created as a KeyedArchive file. Specifically, this allows the plistlib to
process 0x80 tokens as UID objects.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Add posix module support for VxWorks.

@item 
bpo-35125@footnote{https://bugs.python.org/issue35125}: Asyncio: Remove inner callback on outer cancellation in shield

@item 
bpo-35721@footnote{https://bugs.python.org/issue35721}: Fix @code{asyncio.SelectorEventLoop.subprocess_exec()} leaks
file descriptors if @code{Popen} fails and called with
@code{stdin=subprocess.PIPE}. Patch by Niklas Fiekas.

@item 
bpo-31855@footnote{https://bugs.python.org/issue31855}: @ref{1296,,unittest.mock.mock_open()} results now respects the
argument of read([size]). Patch contributed by Rémi Lapeyre.

@item 
bpo-35431@footnote{https://bugs.python.org/issue35431}: Implement @ref{19b,,math.comb()} that returns binomial coefficient,
that computes the number of ways to choose k items from n items without
repetition and without order. Patch by Yash Aggarwal and Keller Fuchs.

@item 
bpo-26660@footnote{https://bugs.python.org/issue26660}: Fixed permission errors in
@ref{de5,,TemporaryDirectory} clean up. Previously
@code{TemporaryDirectory.cleanup()} failed when non-writeable or
non-searchable files or directories were created inside a temporary
directory.

@item 
bpo-34271@footnote{https://bugs.python.org/issue34271}: Add debugging helpers to ssl module. It’s now possible to dump
key material and to trace TLS protocol. The default and stdlib contexts
also support SSLKEYLOGFILE env var.

@item 
bpo-26467@footnote{https://bugs.python.org/issue26467}: Added AsyncMock to support using unittest to mock asyncio
coroutines. Patch by Lisa Roach.

@item 
bpo-33569@footnote{https://bugs.python.org/issue33569}: dataclasses.InitVar: Exposes the type used to create the init
var.

@item 
bpo-34424@footnote{https://bugs.python.org/issue34424}: Fix serialization of messages containing encoded strings when
the policy.linesep is set to a multi-character string. Patch by Jens
Troeger.

@item 
bpo-34303@footnote{https://bugs.python.org/issue34303}: Performance of @ref{e77,,functools.reduce()} is slightly improved.
Patch by Sergey Fedoseev.

@item 
bpo-33361@footnote{https://bugs.python.org/issue33361}: Fix a bug in @ref{129b,,codecs.StreamRecoder} where seeking might
leave old data in a buffer and break subsequent read calls. Patch by Ammar
Askar.

@item 
bpo-22454@footnote{https://bugs.python.org/issue22454}: The @ref{e3,,shlex} module now exposes @ref{536,,shlex.join()}, the
inverse of @ref{443,,shlex.split()}. Patch by Bo Bayles.

@item 
bpo-31922@footnote{https://bugs.python.org/issue31922}: @code{asyncio.AbstractEventLoop.create_datagram_endpoint()}: Do
not connect UDP socket when broadcast is allowed. This allows to receive
replies after a UDP broadcast.

@item 
bpo-24882@footnote{https://bugs.python.org/issue24882}: Change ThreadPoolExecutor to use existing idle threads before
spinning up new ones.

@item 
bpo-31961@footnote{https://bugs.python.org/issue31961}: Added support for bytes and path-like objects in
@ref{5ae,,subprocess.Popen()} on Windows.  The @emph{args} parameter now accepts a
@ref{63d,,path-like object} if @emph{shell} is @code{False} and a sequence containing
bytes and path-like objects. The @emph{executable} parameter now accepts a
bytes and @ref{63d,,path-like object}. The @emph{cwd} parameter now accepts a
bytes object. Based on patch by Anders Lorentsen.

@item 
bpo-33123@footnote{https://bugs.python.org/issue33123}: @ref{129c,,pathlib.Path.unlink} now accepts a @emph{missing_ok}
parameter to avoid a @ref{a37,,FileNotFoundError} from being raised. Patch by
Robert Buchholz.

@item 
bpo-32941@footnote{https://bugs.python.org/issue32941}: Allow @ref{511,,mmap.mmap} objects to access the madvise() system
call (through @ref{512,,mmap.mmap.madvise()}).

@item 
bpo-22102@footnote{https://bugs.python.org/issue22102}: Added support for ZIP files with disks set to 0. Such files are
commonly created by builtin tools on Windows when use ZIP64 extension.
Patch by Francisco Facioni.

@item 
bpo-32515@footnote{https://bugs.python.org/issue32515}: trace.py can now run modules via python3 -m trace -t –module
module_name

@item 
bpo-32299@footnote{https://bugs.python.org/issue32299}: Changed @ref{129d,,unittest.mock.patch.dict()} to return the patched
dictionary when used as context manager. Patch by Vadim Tsander.

@item 
bpo-27141@footnote{https://bugs.python.org/issue27141}: Added a @code{__copy__()} to @code{collections.UserList} and
@code{collections.UserDict} in order to correctly implement shallow copying
of the objects. Patch by Bar Harel.

@item 
bpo-31829@footnote{https://bugs.python.org/issue31829}: @code{\r}, @code{\0} and @code{\x1a} (end-of-file on Windows) are now
escaped in protocol 0 pickles of Unicode strings. This allows to load them
without loss from files open in text mode in Python 2.

@item 
bpo-23395@footnote{https://bugs.python.org/issue23395}: @code{_thread.interrupt_main()} now avoids setting the Python
error status if the @code{SIGINT} signal is ignored or not handled by Python.
@end itemize

@node Documentation<23>,Tests<21>,Library<22>,Python 3 8 0 beta 1
@anchor{whatsnew/changelog id193}@anchor{129e}
@subsubsection Documentation



@itemize -

@item 
bpo-36896@footnote{https://bugs.python.org/issue36896}: Clarify that some types have unstable constructor signature
between Python versions.

@item 
bpo-36686@footnote{https://bugs.python.org/issue36686}: Improve documentation of the stdin, stdout, and stderr
arguments of the @code{asyncio.subprocess_exec} function to specify which
values are supported. Also mention that decoding as text is not supported.

Add a few tests to verify that the various values passed to the std*
arguments actually work.

@item 
bpo-36984@footnote{https://bugs.python.org/issue36984}: Improve version added references in @code{typing} module - by
Anthony Sottile.

@item 
bpo-36868@footnote{https://bugs.python.org/issue36868}: What’s new now mentions SSLContext.hostname_checks_common_name
instead of SSLContext.host_flags.

@item 
bpo-35924@footnote{https://bugs.python.org/issue35924}: Add a note to the @code{curses.addstr()} documentation to warn
that multiline strings can cause segfaults because of an ncurses bug.

@item 
bpo-36783@footnote{https://bugs.python.org/issue36783}: Added C API Documentation for Time_FromTimeAndFold and
PyDateTime_FromDateAndTimeAndFold as per PEP 495. Patch by Edison
Abahurire.

@item 
bpo-36797@footnote{https://bugs.python.org/issue36797}: More of the legacy distutils documentation has been either
pruned, or else more clearly marked as being retained solely until the
setuptools documentation covers it independently.

@item 
bpo-22865@footnote{https://bugs.python.org/issue22865}: Add detail to the documentation on the @ref{b03,,pty.spawn} function.

@item 
bpo-35397@footnote{https://bugs.python.org/issue35397}: Remove deprecation and document urllib.parse.unwrap(). Patch
contributed by Rémi Lapeyre.

@item 
bpo-32995@footnote{https://bugs.python.org/issue32995}: Added the context variable in glossary.

@item 
bpo-33519@footnote{https://bugs.python.org/issue33519}: Clarify that @ref{24,,copy()} is not part of the @code{MutableSequence} ABC.

@item 
bpo-33482@footnote{https://bugs.python.org/issue33482}: Make @code{codecs.StreamRecoder.writelines} take a list of bytes.

@item 
bpo-25735@footnote{https://bugs.python.org/issue25735}: Added documentation for func factorial to indicate that returns
integer values

@item 
bpo-20285@footnote{https://bugs.python.org/issue20285}: Expand object.__doc__ (docstring) to make it clearer. Modify
pydoc.py so that help(object) lists object methods (for other classes,
help omits methods of the object base class.)
@end itemize

@node Tests<21>,Build<22>,Documentation<23>,Python 3 8 0 beta 1
@anchor{whatsnew/changelog id194}@anchor{129f}
@subsubsection Tests



@itemize -

@item 
bpo-37069@footnote{https://bugs.python.org/issue37069}: Modify test_coroutines, test_cprofile, test_generators,
test_raise, test_ssl and test_yield_from to use
@ref{127a,,test.support.catch_unraisable_exception()} rather than
@ref{12a0,,test.support.captured_stderr()}.

@item 
bpo-37098@footnote{https://bugs.python.org/issue37098}: Fix test_memfd_create on older Linux Kernels.

@item 
bpo-37081@footnote{https://bugs.python.org/issue37081}: Test with OpenSSL 1.1.1c

@item 
bpo-36829@footnote{https://bugs.python.org/issue36829}: Add @ref{127a,,test.support.catch_unraisable_exception()}: context
manager catching unraisable exception using @ref{18d,,sys.unraisablehook()}.

@item 
bpo-36915@footnote{https://bugs.python.org/issue36915}: The main regrtest process now always removes all temporary
directories of worker processes even if they crash or if they are killed
on KeyboardInterrupt (CTRL+c).

@item 
bpo-36719@footnote{https://bugs.python.org/issue36719}: “python3 -m test -jN …” now continues the execution of next
tests when a worker process crash (CHILD_ERROR state). Previously, the
test suite stopped immediately. Use –failfast to stop at the first error.

@item 
bpo-36816@footnote{https://bugs.python.org/issue36816}: Update Lib/test/selfsigned_pythontestdotnet.pem to match
self-signed.pythontest.net’s new TLS certificate.

@item 
bpo-35925@footnote{https://bugs.python.org/issue35925}: Skip httplib and nntplib networking tests when they would
otherwise fail due to a modern OS or distro with a default OpenSSL policy
of rejecting connections to servers with weak certificates.

@item 
bpo-36782@footnote{https://bugs.python.org/issue36782}: Add tests for several C API functions in the @ref{2f,,datetime}
module. Patch by Edison Abahurire.

@item 
bpo-36342@footnote{https://bugs.python.org/issue36342}: Fix test_multiprocessing in test_venv if platform lacks
functioning sem_open.
@end itemize

@node Build<22>,Windows<20>,Tests<21>,Python 3 8 0 beta 1
@anchor{whatsnew/changelog id195}@anchor{12a1}
@subsubsection Build



@itemize -

@item 
bpo-36721@footnote{https://bugs.python.org/issue36721}: To embed Python into an application, a new @code{--embed} option
must be passed to @code{python3-config --libs --embed} to get @code{-lpython3.8}
(link the application to libpython). To support both 3.8 and older, try
@code{python3-config --libs --embed} first and fallback to @code{python3-config
--libs} (without @code{--embed}) if the previous command fails.

Add a pkg-config @code{python-3.8-embed} module to embed Python into an
application: @code{pkg-config python-3.8-embed --libs} includes
@code{-lpython3.8}. To support both 3.8 and older, try @code{pkg-config
python-X.Y-embed --libs} first and fallback to @code{pkg-config python-X.Y
--libs} (without @code{--embed}) if the previous command fails (replace
@code{X.Y} with the Python version).

On the other hand, @code{pkg-config python3.8 --libs} no longer contains
@code{-lpython3.8}. C extensions must not be linked to libpython (except on
Android, case handled by the script); this change is backward incompatible
on purpose.

@item 
bpo-36786@footnote{https://bugs.python.org/issue36786}: “make install” now runs compileall in parallel.
@end itemize

@node Windows<20>,macOS<18>,Build<22>,Python 3 8 0 beta 1
@anchor{whatsnew/changelog id196}@anchor{12a2}
@subsubsection Windows



@itemize -

@item 
bpo-36965@footnote{https://bugs.python.org/issue36965}: include of STATUS_CONTROL_C_EXIT without depending on MSC
compiler

@item 
bpo-35926@footnote{https://bugs.python.org/issue35926}: Update to OpenSSL 1.1.1b for Windows.

@item 
bpo-29883@footnote{https://bugs.python.org/issue29883}: Add Windows support for UDP transports for the Proactor Event
Loop. Patch by Adam Meily.

@item 
bpo-33407@footnote{https://bugs.python.org/issue33407}: The @ref{5c5,,Py_DEPRECATED()} macro has been implemented for
MSVC.
@end itemize

@node macOS<18>,IDLE<20>,Windows<20>,Python 3 8 0 beta 1
@anchor{whatsnew/changelog id197}@anchor{12a3}
@subsubsection macOS



@itemize -

@item 
bpo-36231@footnote{https://bugs.python.org/issue36231}: Support building Python on macOS without /usr/include
installed. As of macOS 10.14, system header files are only available
within an SDK provided by either the Command Line Tools or the Xcode app.
@end itemize

@node IDLE<20>,Tools/Demos<8>,macOS<18>,Python 3 8 0 beta 1
@anchor{whatsnew/changelog id198}@anchor{12a4}
@subsubsection IDLE



@itemize -

@item 
bpo-35610@footnote{https://bugs.python.org/issue35610}: Replace now redundant .context_use_ps1 with .prompt_last_line.
This finishes change started in bpo-31858@footnote{https://bugs.python.org/issue31858}.

@item 
bpo-37038@footnote{https://bugs.python.org/issue37038}: Make idlelib.run runnable; add test clause.

@item 
bpo-36958@footnote{https://bugs.python.org/issue36958}: Print any argument other than None or int passed to SystemExit
or sys.exit().

@item 
bpo-36807@footnote{https://bugs.python.org/issue36807}: When saving a file, call os.fsync() so bits are flushed to e.g.
USB drive.

@item 
bpo-32411@footnote{https://bugs.python.org/issue32411}: In browser.py, remove extraneous sorting by line number since
dictionary was created in line number order.
@end itemize

@node Tools/Demos<8>,C API<22>,IDLE<20>,Python 3 8 0 beta 1
@anchor{whatsnew/changelog id199}@anchor{12a5}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-37053@footnote{https://bugs.python.org/issue37053}: Handle strings like u”bar” correctly in
Tools/parser/unparse.py. Patch by Chih-Hsuan Yen.
@end itemize

@node C API<22>,,Tools/Demos<8>,Python 3 8 0 beta 1
@anchor{whatsnew/changelog id200}@anchor{12a6}
@subsubsection C API



@itemize -

@item 
bpo-36763@footnote{https://bugs.python.org/issue36763}: Implement the 
@geindex Python Enhancement Proposals; PEP 587
PEP 587@footnote{https://www.python.org/dev/peps/pep-0587} “Python Initialization Configuration”.

@item 
bpo-36379@footnote{https://bugs.python.org/issue36379}: Fix crashes when attempting to use the @emph{modulo} parameter when
@code{__ipow__} is implemented in C.

@item 
bpo-37107@footnote{https://bugs.python.org/issue37107}: Update @ref{12a7,,PyObject_CallMethodObjArgs()} and
@code{_PyObject_CallMethodIdObjArgs} to use @code{_PyObject_GetMethod} to avoid
creating a bound method object in many cases. Patch by Michael J.
Sullivan.

@item 
bpo-36974@footnote{https://bugs.python.org/issue36974}: Implement 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590}: Vectorcall: a fast calling protocol for
CPython. This is a new protocol to optimize calls of custom callable
objects.

@item 
bpo-36763@footnote{https://bugs.python.org/issue36763}: @code{Py_Main()} now returns the exitcode rather than calling
@code{Py_Exit(exitcode)} when calling @code{PyErr_Print()} if the current
exception type is @code{SystemExit}.

@item 
bpo-36922@footnote{https://bugs.python.org/issue36922}: Add new type flag @code{Py_TPFLAGS_METHOD_DESCRIPTOR} for objects
behaving like unbound methods. These are objects supporting the
optimization given by the @code{LOAD_METHOD}/@code{CALL_METHOD} opcodes. See PEP
590.

@item 
bpo-36728@footnote{https://bugs.python.org/issue36728}: The @code{PyEval_ReInitThreads()} function has been removed
from the C API. It should not be called explicitly: use
@ref{5c1,,PyOS_AfterFork_Child()} instead.
@end itemize

@node Python 3 8 0 alpha 4,Python 3 8 0 alpha 3,Python 3 8 0 beta 1,Changelog
@anchor{whatsnew/changelog python-3-8-0-alpha-4}@anchor{12a8}
@subsection Python 3.8.0 alpha 4


@emph{Release date: 2019-05-06}

@menu
* Security: Security<16>. 
* Core and Builtins: Core and Builtins<23>. 
* Library: Library<23>. 
* Documentation: Documentation<24>. 
* Tests: Tests<22>. 
* Build: Build<23>. 
* Windows: Windows<21>. 
* macOS: macOS<19>. 
* IDLE: IDLE<21>. 
* Tools/Demos: Tools/Demos<9>. 
* C API: C API<23>. 

@end menu

@node Security<16>,Core and Builtins<23>,,Python 3 8 0 alpha 4
@anchor{whatsnew/changelog id201}@anchor{12a9}
@subsubsection Security



@itemize -

@item 
bpo-36742@footnote{https://bugs.python.org/issue36742}: Fixes mishandling of pre-normalization characters in
urlsplit().

@item 
bpo-30458@footnote{https://bugs.python.org/issue30458}: Address CVE-2019-9740 by disallowing URL paths with embedded
whitespace or control characters through into the underlying http client
request.  Such potentially malicious header injection URLs now cause an
http.client.InvalidURL exception to be raised.

@item 
bpo-35755@footnote{https://bugs.python.org/issue35755}: @ref{11ac,,shutil.which()} now uses @code{os.confstr("CS_PATH")} if
available and if the 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} environment variable is not set.
Remove also the current directory from @code{posixpath.defpath}. On Unix,
@ref{11ac,,shutil.which()} and the @ref{f4,,subprocess} module no longer search the
executable in the current directory if the 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} environment
variable is not set.
@end itemize

@node Core and Builtins<23>,Library<23>,Security<16>,Python 3 8 0 alpha 4
@anchor{whatsnew/changelog id202}@anchor{12aa}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-36722@footnote{https://bugs.python.org/issue36722}: In debug build, import now also looks for C extensions compiled
in release mode and for C extensions compiled in the stable ABI.

@item 
bpo-32849@footnote{https://bugs.python.org/issue32849}: Fix Python Initialization code on FreeBSD to detect properly
when stdin file descriptor (fd 0) is invalid.

@item 
bpo-36623@footnote{https://bugs.python.org/issue36623}: Remove parser headers and related function declarations that
lack implementations after the removal of pgen.

@item 
bpo-20180@footnote{https://bugs.python.org/issue20180}: @code{dict.pop()} is now up to 33% faster thanks to Argument
Clinic. Patch by Inada Naoki.

@item 
bpo-36611@footnote{https://bugs.python.org/issue36611}: Debug memory allocators: disable serialno field by default from
debug hooks on Python memory allocators to reduce the memory footprint by
5%. Enable @ref{11b,,tracemalloc} to get the traceback where a memory block
has been allocated when a fatal memory error is logged to decide where to
put a breakpoint. Compile Python with @code{PYMEM_DEBUG_SERIALNO} defined to
get back the field.

@item 
bpo-36588@footnote{https://bugs.python.org/issue36588}: On AIX, @ref{5a6,,sys.platform} doesn’t contain the major version
anymore. Always return @code{'aix'}, instead of @code{'aix3'} .. @code{'aix7'}.
Since older Python versions include the version number, it is recommended
to always use @code{sys.platform.startswith('aix')}. Contributed by M. Felt.

@item 
bpo-36549@footnote{https://bugs.python.org/issue36549}: Change str.capitalize to use titlecase for the first character
instead of uppercase.

@item 
bpo-36540@footnote{https://bugs.python.org/issue36540}: Implement 
@geindex Python Enhancement Proposals; PEP 570
PEP 570@footnote{https://www.python.org/dev/peps/pep-0570} (Python positional-only parameters). Patch
by Pablo Galindo.

@item 
bpo-36475@footnote{https://bugs.python.org/issue36475}: @ref{5a7,,PyEval_AcquireLock()} and @ref{5a8,,PyEval_AcquireThread()}
now terminate the current thread if called while the interpreter is
finalizing, making them consistent with @ref{5a9,,PyEval_RestoreThread()},
@ref{5aa,,Py_END_ALLOW_THREADS()}, and @ref{5ab,,PyGILState_Ensure()}.

@item 
bpo-36504@footnote{https://bugs.python.org/issue36504}: Fix signed integer overflow in _ctypes.c’s
@code{PyCArrayType_new()}.

@item 
bpo-20844@footnote{https://bugs.python.org/issue20844}: Fix running script with encoding cookie and LF line ending may
fail on Windows.

@item 
bpo-24214@footnote{https://bugs.python.org/issue24214}: Fixed support of the surrogatepass error handler in the UTF-8
incremental decoder.

@item 
bpo-36452@footnote{https://bugs.python.org/issue36452}: Changing @code{dict} keys during iteration of the dict itself,
@code{keys()}, @code{values()}, or @code{items()} will now be detected in certain
corner cases where keys are deleted/added so that the number of keys isn’t
changed. A @ref{5af,,RuntimeError} will be raised after @code{len(dict)} iterations.
Contributed by Thomas Perl.

@item 
bpo-36459@footnote{https://bugs.python.org/issue36459}: Fix a possible double @code{PyMem_FREE()} due to tokenizer.c’s
@code{tok_nextc()}.

@item 
bpo-36433@footnote{https://bugs.python.org/issue36433}: Fixed TypeError message in classmethoddescr_call.

@item 
bpo-36430@footnote{https://bugs.python.org/issue36430}: Fix a possible reference leak in @ref{e33,,itertools.count()}.

@item 
bpo-36440@footnote{https://bugs.python.org/issue36440}: Include node names in @code{ParserError} messages, instead of
numeric IDs. Patch by A. Skrobov.

@item 
bpo-36143@footnote{https://bugs.python.org/issue36143}: Regenerate @ref{a2,,keyword} from the Grammar and Tokens file
using pgen. Patch by Pablo Galindo.

@item 
bpo-18372@footnote{https://bugs.python.org/issue18372}: Add missing @ref{103a,,PyObject_GC_Track()} calls in the
@ref{c5,,pickle} module. Patch by Zackery Spytz.
@end itemize

@node Library<23>,Documentation<24>,Core and Builtins<23>,Python 3 8 0 alpha 4
@anchor{whatsnew/changelog id203}@anchor{12ab}
@subsubsection Library



@itemize -

@item 
bpo-35952@footnote{https://bugs.python.org/issue35952}: Fix pythoninfo when the compiler is missing.

@item 
bpo-28238@footnote{https://bugs.python.org/issue28238}: The @code{.find*()} methods of xml.etree.ElementTree can now
search for wildcards like @code{@{*@}tag} and @code{@{ns@}*} that match a tag in any
namespace or all tags in a namespace.  Patch by Stefan Behnel.

@item 
bpo-26978@footnote{https://bugs.python.org/issue26978}: @code{pathlib.path.link_to()} is now implemented. It creates a hard
link pointing to a path.

@item 
bpo-1613500@footnote{https://bugs.python.org/issue1613500}: @ref{1bc,,fileinput.FileInput} now uses the input file mode to
correctly set the output file mode (previously it was hardcoded to
@code{'w'})  when @code{inplace=True} is passed to its constructor.

@item 
bpo-36734@footnote{https://bugs.python.org/issue36734}: Fix compilation of @code{faulthandler.c} on HP-UX. Initialize
@code{stack_t current_stack} to zero using @code{memset()}.

@item 
bpo-13611@footnote{https://bugs.python.org/issue13611}: The xml.etree.ElementTree packages gained support for C14N 2.0
serialisation. Patch by Stefan Behnel.

@item 
bpo-36669@footnote{https://bugs.python.org/issue36669}: Add missing matrix multiplication operator support to
weakref.proxy.

@item 
bpo-36676@footnote{https://bugs.python.org/issue36676}: The XMLParser() in xml.etree.ElementTree provides namespace
prefix context to the parser target if it defines the callback methods
“start_ns()” and/or “end_ns()”. Patch by Stefan Behnel.

@item 
bpo-36673@footnote{https://bugs.python.org/issue36673}: The TreeBuilder and XMLPullParser in xml.etree.ElementTree
gained support for parsing comments and processing instructions. Patch by
Stefan Behnel.

@item 
bpo-36650@footnote{https://bugs.python.org/issue36650}: The C version of functools.lru_cache() was treating calls with
an empty @code{**kwargs} dictionary as being distinct from calls with no
keywords at all. This did not result in an incorrect answer, but it did
trigger an unexpected cache miss.

@item 
bpo-28552@footnote{https://bugs.python.org/issue28552}: Fix @ref{5e,,distutils.sysconfig} if @ref{585,,sys.executable} is
@code{None} or an empty string: use @ref{12ac,,os.getcwd()} to initialize
@code{project_base}.  Fix also the distutils build command: don’t use
@ref{585,,sys.executable} if it is @code{None} or an empty string.

@item 
bpo-35755@footnote{https://bugs.python.org/issue35755}: @ref{11ac,,shutil.which()} and
@code{distutils.spawn.find_executable()} now use @code{os.confstr("CS_PATH")}
if available instead of @ref{12ad,,os.defpath}, if the @code{PATH} environment
variable is not set. Moreover, don’t use @code{os.confstr("CS_PATH")} nor
@ref{12ad,,os.defpath} if the @code{PATH} environment variable is set to an empty
string.

@item 
bpo-25430@footnote{https://bugs.python.org/issue25430}: improve performance of @code{IPNetwork.__contains__()}

@item 
bpo-30485@footnote{https://bugs.python.org/issue30485}: Path expressions in xml.etree.ElementTree can now avoid
explicit namespace prefixes for tags (or the “@{namespace@}tag” notation) by
passing a default namespace with an empty string prefix.

@item 
bpo-36613@footnote{https://bugs.python.org/issue36613}: Fix @ref{9,,asyncio} wait() not removing callback if exception

@item 
bpo-36598@footnote{https://bugs.python.org/issue36598}: Fix @code{isinstance} check for Mock objects with spec when the
code is executed under tracing. Patch by Karthikeyan Singaravelan.

@item 
bpo-18748@footnote{https://bugs.python.org/issue18748}: In development mode (@ref{159,,-X} @code{dev}) and in debug build,
the @ref{503,,io.IOBase} destructor now logs @code{close()} exceptions. These
exceptions are silent by default in release mode.

@item 
bpo-36575@footnote{https://bugs.python.org/issue36575}: The @code{_lsprof} module now uses internal timer same to
@code{time.perf_counter()} by default. @code{gettimeofday(2)} was used on Unix.
New timer has better resolution on most Unix platforms and timings are no
longer impacted by system clock updates since @code{perf_counter()} is
monotonic. Patch by Inada Naoki.

@item 
bpo-33461@footnote{https://bugs.python.org/issue33461}: @code{json.loads} now emits @code{DeprecationWarning} when
@code{encoding} option is specified.  Patch by Matthias Bussonnier.

@item 
bpo-36559@footnote{https://bugs.python.org/issue36559}: The random module now prefers the lean internal _sha512 module
over hashlib for seed(version=2) to optimize import time.

@item 
bpo-17561@footnote{https://bugs.python.org/issue17561}: Set backlog=None as the default for socket.create_server.

@item 
bpo-34373@footnote{https://bugs.python.org/issue34373}: Fix @ref{d8a,,time.mktime()} error handling on AIX for year before
1970.

@item 
bpo-36232@footnote{https://bugs.python.org/issue36232}: Improve error message when trying to open existing DBM database
that actually doesn’t exist. Patch by Marco Rougeth.

@item 
bpo-36546@footnote{https://bugs.python.org/issue36546}: Add statistics.quantiles()

@item 
bpo-36050@footnote{https://bugs.python.org/issue36050}: Optimized @code{http.client.HTTPResponse.read()} for large
response. Patch by Inada Naoki.

@item 
bpo-36522@footnote{https://bugs.python.org/issue36522}: If @emph{debuglevel} is set to >0 in @ref{8f,,http.client}, print all
values for headers with multiple values for the same header name. Patch by
Matt Houglum.

@item 
bpo-36492@footnote{https://bugs.python.org/issue36492}: Deprecated passing required arguments like @emph{func} as keyword
arguments in functions which should accept arbitrary keyword arguments and
pass them to other function. Arbitrary keyword arguments (even with names
“self” and “func”) can now be passed to these functions if the required
arguments are passed as positional arguments.

@item 
bpo-27181@footnote{https://bugs.python.org/issue27181}: Add statistics.geometric_mean().

@item 
bpo-30427@footnote{https://bugs.python.org/issue30427}: @code{os.path.normcase()} relies on @code{os.fspath()} to check the
type of its argument. Redundant checks have been removed from its
@code{posixpath.normcase()} and @code{ntpath.normcase()} implementations. Patch
by Wolfgang Maier.

@item 
bpo-36385@footnote{https://bugs.python.org/issue36385}: Stop rejecting IPv4 octets for being ambiguously octal. Leading
zeros are ignored, and no longer are assumed to specify octal octets.
Octets are always decimal numbers. Octets must still be no more than three
digits, including leading zeroes.

@item 
bpo-36434@footnote{https://bugs.python.org/issue36434}: Errors during writing to a ZIP file no longer prevent to
properly close it.

@item 
bpo-36407@footnote{https://bugs.python.org/issue36407}: Fixed wrong indentation writing for CDATA section in
xml.dom.minidom. Patch by Vladimir Surjaninov.

@item 
bpo-36326@footnote{https://bugs.python.org/issue36326}: inspect.getdoc() can now find docstrings for member objects
when __slots__ is a dictionary.

@item 
bpo-36366@footnote{https://bugs.python.org/issue36366}: Calling @code{stop()} on an unstarted or stopped
@ref{a02,,unittest.mock.patch()} object will now return @ref{182,,None} instead of
raising @ref{5af,,RuntimeError}, making the method idempotent. Patch by
Karthikeyan Singaravelan.

@item 
bpo-36348@footnote{https://bugs.python.org/issue36348}: The @code{imap.IMAP4.logout()} method no longer ignores silently
arbitrary exceptions.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Add time module support and fix test_time faiures for VxWorks.

@item 
bpo-36227@footnote{https://bugs.python.org/issue36227}: Added support for keyword arguments @code{default_namespace} and
@code{xml_declaration} in functions ElementTree.tostring() and
ElementTree.tostringlist().

@item 
bpo-36004@footnote{https://bugs.python.org/issue36004}: Added new alternate constructors
@ref{4f0,,datetime.date.fromisocalendar()} and
@ref{4f1,,datetime.datetime.fromisocalendar()}, which construct date objects
from ISO year, week number and weekday; these are the inverse of each
class’s @code{isocalendar} method. Patch by Paul Ganssle.

@item 
bpo-35936@footnote{https://bugs.python.org/issue35936}: @ref{b0,,modulefinder} no longer depends on the deprecated
@ref{95,,imp} module, and the initializer for
@ref{12ae,,modulefinder.ModuleFinder} now has immutable default arguments.
Patch by Brandt Bucher.

@item 
bpo-35376@footnote{https://bugs.python.org/issue35376}: @ref{b0,,modulefinder} correctly handles modules that have the
same name as a bad package. Patch by Brandt Bucher.

@item 
bpo-17396@footnote{https://bugs.python.org/issue17396}: @ref{b0,,modulefinder} no longer crashes when encountering syntax
errors in followed imports. Patch by Brandt Bucher.

@item 
bpo-35934@footnote{https://bugs.python.org/issue35934}: Added @ref{53c,,create_server()} and
@ref{53d,,has_dualstack_ipv6()} convenience functions to automate the
necessary tasks usually involved when creating a server socket, including
accepting both IPv4 and IPv6 connections on the same socket.  (Contributed
by Giampaolo Rodola in bpo-17561@footnote{https://bugs.python.org/issue17561}.)

@item 
bpo-23078@footnote{https://bugs.python.org/issue23078}: Add support for @ref{254,,classmethod()} and @ref{253,,staticmethod()} to
@ref{122a,,unittest.mock.create_autospec()}.  Initial patch by Felipe Ochoa.

@item 
bpo-35416@footnote{https://bugs.python.org/issue35416}: Fix potential resource warnings in distutils. Patch by Mickaël
Schoentgen.

@item 
bpo-25451@footnote{https://bugs.python.org/issue25451}: Add transparency methods to @code{tkinter.PhotoImage}.  Patch
by Zackery Spytz.

@item 
bpo-35082@footnote{https://bugs.python.org/issue35082}: Don’t return deleted attributes when calling dir on a
@ref{561,,unittest.mock.Mock}.

@item 
bpo-34547@footnote{https://bugs.python.org/issue34547}: @ref{129a,,wsgiref.handlers.BaseHandler} now handles abrupt client
connection terminations gracefully. Patch by Petter Strandmark.

@item 
bpo-31658@footnote{https://bugs.python.org/issue31658}: @ref{12af,,xml.sax.parse()} now supports @ref{63d,,path-like}. Patch by Mickaël Schoentgen.

@item 
bpo-34139@footnote{https://bugs.python.org/issue34139}: Remove stale unix datagram socket before binding

@item 
bpo-33530@footnote{https://bugs.python.org/issue33530}: Implemented Happy Eyeballs in @code{asyncio.create_connection()}.
Added two new arguments, @emph{happy_eyeballs_delay} and @emph{interleave}, to
specify Happy Eyeballs behavior.

@item 
bpo-33291@footnote{https://bugs.python.org/issue33291}: Do not raise AttributeError when calling the inspect functions
isgeneratorfunction, iscoroutinefunction, isasyncgenfunction on a method
created from an arbitrary callable. Instead, return False.

@item 
bpo-31310@footnote{https://bugs.python.org/issue31310}: Fix the multiprocessing.semaphore_tracker so it is reused by
child processes

@item 
bpo-31292@footnote{https://bugs.python.org/issue31292}: Fix @code{setup.py check --restructuredtext} for files containing
@code{include} directives.
@end itemize

@node Documentation<24>,Tests<22>,Library<23>,Python 3 8 0 alpha 4
@anchor{whatsnew/changelog id204}@anchor{12b0}
@subsubsection Documentation



@itemize -

@item 
bpo-36625@footnote{https://bugs.python.org/issue36625}: Remove obsolete comments from docstrings in fractions.Fraction

@item 
bpo-30840@footnote{https://bugs.python.org/issue30840}: Document relative imports

@item 
bpo-36523@footnote{https://bugs.python.org/issue36523}: Add docstring for io.IOBase.writelines().

@item 
bpo-36425@footnote{https://bugs.python.org/issue36425}: New documentation translation: Simplified Chinese@footnote{https://docs.python.org/zh-cn/}.

@item 
bpo-36345@footnote{https://bugs.python.org/issue36345}: Avoid the duplication of code from @code{Tools/scripts/serve.py}
in using the @code{literalinclude} directive for the basic
wsgiref-based web server in the documentation of @ref{133,,wsgiref}.
Contributed by Stéphane Wirtel.

@item 
bpo-36345@footnote{https://bugs.python.org/issue36345}: Using the code of the @code{Tools/scripts/serve.py} script as an
example in the @ref{133,,wsgiref} documentation.  Contributed by Stéphane
Wirtel.

@item 
bpo-36157@footnote{https://bugs.python.org/issue36157}: Added Documention for  PyInterpreterState_Main().

@item 
bpo-33043@footnote{https://bugs.python.org/issue33043}: Updates the docs.python.org page with the addition of a
‘Contributing to Docs’ link at the end of the page (between ‘Reporting
Bugs’ and ‘About Documentation’). Updates the ‘Found a Bug’ page with
additional links and information in the Documentation Bugs section.

@item 
bpo-35581@footnote{https://bugs.python.org/issue35581}: @@typing.type_check_only now allows type stubs to mark functions
and classes not available during runtime.

@item 
bpo-33832@footnote{https://bugs.python.org/issue33832}: Add glossary entry for ‘magic method’.

@item 
bpo-32913@footnote{https://bugs.python.org/issue32913}: Added re.Match.groupdict example to regex HOWTO.
@end itemize

@node Tests<22>,Build<23>,Documentation<24>,Python 3 8 0 alpha 4
@anchor{whatsnew/changelog id205}@anchor{12b1}
@subsubsection Tests



@itemize -

@item 
bpo-36719@footnote{https://bugs.python.org/issue36719}: regrtest now always detects uncollectable objects. Previously,
the check was only enabled by @code{--findleaks}. The check now also works
with @code{-jN/--multiprocess N}. @code{--findleaks} becomes a deprecated alias
to @code{--fail-env-changed}.

@item 
bpo-36725@footnote{https://bugs.python.org/issue36725}: When using multiprocessing mode (-jN), regrtest now better
reports errors if a worker process fails, and it exits immediately on a
worker thread failure or when interrupted.

@item 
bpo-36454@footnote{https://bugs.python.org/issue36454}: Change test_time.test_monotonic() to test only the lower bound
of elapsed time after a sleep command rather than the upper bound. This
prevents unnecessary test failures on slow buildbots. Patch by Victor
Stinner.

@item 
bpo-32424@footnote{https://bugs.python.org/issue32424}: Improve test coverage for xml.etree.ElementTree. Patch by
Gordon P. Hemsley.

@item 
bpo-32424@footnote{https://bugs.python.org/issue32424}: Fix typo in test_cyclic_gc() test for xml.etree.ElementTree.
Patch by Gordon P. Hemsley.

@item 
bpo-36635@footnote{https://bugs.python.org/issue36635}: Add a new @code{_testinternalcapi} module to test the internal
C API.

@item 
bpo-36629@footnote{https://bugs.python.org/issue36629}: Fix @code{test_imap4_host_default_value()} of @code{test_imaplib}:
catch also @ref{12b2,,errno.ENETUNREACH} error.

@item 
bpo-36611@footnote{https://bugs.python.org/issue36611}: Fix @code{test_sys.test_getallocatedblocks()} when
@ref{11b,,tracemalloc} is enabled.

@item 
bpo-36560@footnote{https://bugs.python.org/issue36560}: Fix reference leak hunting in regrtest: compute also deltas (of
reference count, allocated memory blocks, file descriptor count) during
warmup, to ensure that everything is initialized before starting to hunt
reference leaks.

@item 
bpo-36565@footnote{https://bugs.python.org/issue36565}: Fix reference hunting (@code{python3 -m test -R 3:3}) when Python
has no built-in abc module.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Port test_resource to VxWorks: skip tests cases setting
RLIMIT_FSIZE and RLIMIT_CPU.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Fix test_tabnanny on VxWorks: adjust ENOENT error message.

@item 
bpo-36436@footnote{https://bugs.python.org/issue36436}: Fix @code{_testcapi.pymem_buffer_overflow()}: handle memory
allocation failure.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Fix test_utf8_mode on VxWorks: Python always use UTF-8 on
VxWorks.

@item 
bpo-36341@footnote{https://bugs.python.org/issue36341}: Fix tests that may fail with PermissionError upon calling
bind() on AF_UNIX sockets.
@end itemize

@node Build<23>,Windows<21>,Tests<22>,Python 3 8 0 alpha 4
@anchor{whatsnew/changelog id206}@anchor{12b3}
@subsubsection Build



@itemize -

@item 
bpo-36747@footnote{https://bugs.python.org/issue36747}: Remove the stale scriptsinstall Makefile target.

@item 
bpo-21536@footnote{https://bugs.python.org/issue21536}: On Unix, C extensions are no longer linked to libpython except
on Android and Cygwin.

It is now possible for a statically linked Python to load a C extension
built using a shared library Python.

When Python is embedded, @code{libpython} must not be loaded with
@code{RTLD_LOCAL}, but @code{RTLD_GLOBAL} instead. Previously, using
@code{RTLD_LOCAL}, it was already not possible to load C extensions which
were not linked to @code{libpython}, such as C extensions of the standard
library built by the @code{*shared*} section of @code{Modules/Setup}.

distutils, python-config and python-config.py have been modified.

@item 
bpo-36707@footnote{https://bugs.python.org/issue36707}: @code{./configure --with-pymalloc} no longer adds the @code{m} flag
to SOABI (sys.implementation.cache_tag). Enabling or disabling pymalloc
has no impact on the ABI.

@item 
bpo-36635@footnote{https://bugs.python.org/issue36635}: Change @code{PyAPI_FUNC(type)}, @code{PyAPI_DATA(type)} and
@code{PyMODINIT_FUNC} macros of @code{pyport.h} when @code{Py_BUILD_CORE_MODULE} is
defined. The @code{Py_BUILD_CORE_MODULE} define must be now be used to build
a C extension as a dynamic library accessing Python internals: export the
PyInit_xxx() function in DLL exports on Windows.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Don’t build the @code{_crypt} extension on VxWorks.

@item 
bpo-36618@footnote{https://bugs.python.org/issue36618}: Add @code{-fmax-type-align=8} to CFLAGS when clang compiler is
detected. The pymalloc memory allocator aligns memory on 8 bytes. On
x86-64, clang expects alignment on 16 bytes by default and so uses MOVAPS
instruction which can lead to segmentation fault. Instruct clang that
Python is limited to alignment on 8 bytes to use MOVUPS instruction
instead: slower but don’t trigger a SIGSEGV if the memory is not aligned
on 16 bytes. Sadly, the flag must be added to @code{CFLAGS} and not just
@code{CFLAGS_NODIST}, since third party C extensions can have the same issue.

@item 
bpo-36605@footnote{https://bugs.python.org/issue36605}: @code{make tags} and @code{make TAGS} now also parse
@code{Modules/_io/*.c} and @code{Modules/_io/*.h}.

@item 
bpo-36465@footnote{https://bugs.python.org/issue36465}: Release builds and debug builds are now ABI compatible:
defining the @code{Py_DEBUG} macro no longer implies the @code{Py_TRACE_REFS}
macro, which introduces the only ABI incompatibility. The
@code{Py_TRACE_REFS} macro, which adds the @code{sys.getobjects()} function
and the 
@geindex PYTHONDUMPREFS
@geindex environment variable; PYTHONDUMPREFS
@ref{4ad,,PYTHONDUMPREFS} environment variable, can be set using
the new @code{./configure --with-trace-refs} build option.

@item 
bpo-36577@footnote{https://bugs.python.org/issue36577}: setup.py now correctly reports missing OpenSSL headers and
libraries again.

@item 
bpo-36544@footnote{https://bugs.python.org/issue36544}: Fix regression introduced in bpo-36146@footnote{https://bugs.python.org/issue36146} refactoring setup.py

@item 
bpo-36508@footnote{https://bugs.python.org/issue36508}: @code{python-config --ldflags} no longer includes flags of the
@code{LINKFORSHARED} variable. The @code{LINKFORSHARED} variable must only be
used to build executables.

@item 
bpo-36503@footnote{https://bugs.python.org/issue36503}: Remove references to “aix3” and “aix4”. Patch by M. Felt.
@end itemize

@node Windows<21>,macOS<19>,Build<23>,Python 3 8 0 alpha 4
@anchor{whatsnew/changelog id207}@anchor{12b4}
@subsubsection Windows



@itemize -

@item 
bpo-35920@footnote{https://bugs.python.org/issue35920}: Added platform.win32_edition() and platform.win32_is_iot().
Added support for cross-compiling packages for Windows ARM32. Skip tests
that are not expected to work on Windows IoT Core ARM32.

@item 
bpo-36649@footnote{https://bugs.python.org/issue36649}: Remove trailing spaces for registry keys when installed via the
Store.

@item 
bpo-34144@footnote{https://bugs.python.org/issue34144}: Fixed activate.bat to correctly update codepage when chcp.com
returns dots in output. Patch by Lorenz Mende.

@item 
bpo-36509@footnote{https://bugs.python.org/issue36509}: Added preset-iot layout for Windows IoT ARM containers. This
layout doesn’t contain UI components like tkinter or IDLE. It also doesn’t
contain files to support on-target builds since Windows ARM32 builds must
be cross-compiled when using MSVC.

@item 
bpo-35941@footnote{https://bugs.python.org/issue35941}: enum_certificates function of the ssl module now returns
certificates from all available certificate stores inside windows in a
query instead of returning only certificates from the system wide
certificate store. This includes certificates from these certificate
stores: local machine, local machine enterprise, local machine group
policy, current user, current user group policy, services, users.
ssl.enum_crls() function is changed in the same way to return all
certificate revocation lists inside the windows certificate revocation
list stores.

@item 
bpo-36441@footnote{https://bugs.python.org/issue36441}: Fixes creating a venv when debug binaries are installed.

@item 
bpo-36085@footnote{https://bugs.python.org/issue36085}: Enable better DLL resolution on Windows by using safe DLL
search paths and adding @ref{4ee,,os.add_dll_directory()}.

@item 
bpo-36010@footnote{https://bugs.python.org/issue36010}: Add the venv standard library module to the nuget distribution
for Windows.

@item 
bpo-29515@footnote{https://bugs.python.org/issue29515}: Add the following socket module constants on Windows:
IPPROTO_AH IPPROTO_CBT IPPROTO_DSTOPTS IPPROTO_EGP IPPROTO_ESP
IPPROTO_FRAGMENT IPPROTO_GGP IPPROTO_HOPOPTS IPPROTO_ICLFXBM
IPPROTO_ICMPV6 IPPROTO_IDP IPPROTO_IGMP IPPROTO_IGP IPPROTO_IPV4
IPPROTO_IPV6 IPPROTO_L2TP IPPROTO_MAX IPPROTO_ND IPPROTO_NONE IPPROTO_PGM
IPPROTO_PIM IPPROTO_PUP IPPROTO_RDP IPPROTO_ROUTING IPPROTO_SCTP
IPPROTO_ST

@item 
bpo-35947@footnote{https://bugs.python.org/issue35947}: Added current version of libffi to cpython-source-deps. Change
_ctypes to use current version of libffi on Windows.

@item 
bpo-34060@footnote{https://bugs.python.org/issue34060}: Report system load when running test suite on Windows. Patch by
Ammar Askar. Based on prior work by Jeremy Kloth.

@item 
bpo-31512@footnote{https://bugs.python.org/issue31512}: With the Windows 10 Creators Update, non-elevated users can now
create symlinks as long as the computer has Developer Mode enabled.
@end itemize

@node macOS<19>,IDLE<21>,Windows<21>,Python 3 8 0 alpha 4
@anchor{whatsnew/changelog id208}@anchor{12b5}
@subsubsection macOS



@itemize -

@item 
bpo-34602@footnote{https://bugs.python.org/issue34602}: Avoid failures setting macOS stack resource limit with
resource.setrlimit. This reverts an earlier fix for bpo-18075@footnote{https://bugs.python.org/issue18075} which forced
a non-default stack size when building the interpreter executable on
macOS.
@end itemize

@node IDLE<21>,Tools/Demos<9>,macOS<19>,Python 3 8 0 alpha 4
@anchor{whatsnew/changelog id209}@anchor{12b6}
@subsubsection IDLE



@itemize -

@item 
bpo-36429@footnote{https://bugs.python.org/issue36429}: Fix starting IDLE with pyshell. Add idlelib.pyshell alias at
top; remove pyshell alias at bottom. Remove obsolete __name__==’__main__’
command.
@end itemize

@node Tools/Demos<9>,C API<23>,IDLE<21>,Python 3 8 0 alpha 4
@anchor{whatsnew/changelog id210}@anchor{12b7}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-14546@footnote{https://bugs.python.org/issue14546}: Fix the argument handling in Tools/scripts/lll.py.
@end itemize

@node C API<23>,,Tools/Demos<9>,Python 3 8 0 alpha 4
@anchor{whatsnew/changelog id211}@anchor{12b8}
@subsubsection C API



@itemize -

@item 
bpo-36763@footnote{https://bugs.python.org/issue36763}: Fix memory leak in @ref{205,,Py_SetStandardStreamEncoding()}:
release memory if the function is called twice.

@item 
bpo-36641@footnote{https://bugs.python.org/issue36641}: PyDoc_VAR(name) and
@ref{1200,,PyDoc_STRVAR}(name,  str) now create @code{static const char name[]}
instead of @code{static char name[]}.  Patch by Inada Naoki.

@item 
bpo-36389@footnote{https://bugs.python.org/issue36389}: Change the value of @code{CLEANBYTE}, @code{DEADDYTE} and
@code{FORBIDDENBYTE} internal constants used by debug hooks on Python memory
allocators (@ref{7b0,,PyMem_SetupDebugHooks()} function). Byte patterns
@code{0xCB}, @code{0xDB} and @code{0xFB} have been replaced with @code{0xCD}, @code{0xDD}
and @code{0xFD} to use the same values than Windows CRT debug @code{malloc()}
and @code{free()}.

@item 
bpo-36443@footnote{https://bugs.python.org/issue36443}: Since Python 3.7.0, calling @ref{6fe,,Py_DecodeLocale()} before
@ref{38d,,Py_Initialize()} produces mojibake if the @code{LC_CTYPE} locale is
coerced and/or if the UTF-8 Mode is enabled by the user configuration. The
LC_CTYPE coercion and UTF-8 Mode are now disabled by default to fix the
mojibake issue. They must now be enabled explicitly (opt-in) using the new
@code{_Py_PreInitialize()} API with @code{_PyPreConfig}.

@item 
bpo-36025@footnote{https://bugs.python.org/issue36025}: Fixed an accidental change to the datetime C API where the
arguments to the @ref{12b9,,PyDate_FromTimestamp()} function were incorrectly
interpreted as a single timestamp rather than an arguments tuple, which
causes existing code to start raising @ref{162,,TypeError}. The
backwards-incompatible change was only present in alpha releases of Python
3.8. Patch by Paul Ganssle.

@item 
bpo-35810@footnote{https://bugs.python.org/issue35810}: Modify @code{PyObject_Init} to correctly increase the refcount of
heap- allocated Type objects. Also fix the refcounts of the heap-allocated
types that were either doing this manually or not decreasing the type’s
refcount in tp_dealloc
@end itemize

@node Python 3 8 0 alpha 3,Python 3 8 0 alpha 2,Python 3 8 0 alpha 4,Changelog
@anchor{whatsnew/changelog python-3-8-0-alpha-3}@anchor{12ba}
@subsection Python 3.8.0 alpha 3


@emph{Release date: 2019-03-25}

@menu
* Security: Security<17>. 
* Core and Builtins: Core and Builtins<24>. 
* Library: Library<24>. 
* Documentation: Documentation<25>. 
* Tests: Tests<23>. 
* Build: Build<24>. 
* Windows: Windows<22>. 
* IDLE: IDLE<22>. 
* Tools/Demos: Tools/Demos<10>. 
* C API: C API<24>. 

@end menu

@node Security<17>,Core and Builtins<24>,,Python 3 8 0 alpha 3
@anchor{whatsnew/changelog id212}@anchor{12bb}
@subsubsection Security



@itemize -

@item 
bpo-36216@footnote{https://bugs.python.org/issue36216}: Changes urlsplit() to raise ValueError when the URL contains
characters that decompose under IDNA encoding (NFKC-normalization) into
characters that affect how the URL is parsed.

@item 
bpo-35121@footnote{https://bugs.python.org/issue35121}: Don’t send cookies of domain A without Domain attribute to
domain B when domain A is a suffix match of domain B while using a
cookiejar with @ref{12bc,,http.cookiejar.DefaultCookiePolicy} policy. Patch
by Karthikeyan Singaravelan.
@end itemize

@node Core and Builtins<24>,Library<24>,Security<17>,Python 3 8 0 alpha 3
@anchor{whatsnew/changelog id213}@anchor{12bd}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-36421@footnote{https://bugs.python.org/issue36421}: Fix a possible double decref in _ctypes.c’s
@code{PyCArrayType_new()}.

@item 
bpo-36412@footnote{https://bugs.python.org/issue36412}: Fix a possible crash when creating a new dictionary.

@item 
bpo-36398@footnote{https://bugs.python.org/issue36398}: Fix a possible crash in @code{structseq_repr()}.

@item 
bpo-36256@footnote{https://bugs.python.org/issue36256}: Fix bug in parsermodule when parsing a state in a DFA that has
two or more arcs with labels of the same type. Patch by Pablo Galindo.

@item 
bpo-36365@footnote{https://bugs.python.org/issue36365}: repr(structseq) is no longer limited to 512 bytes.

@item 
bpo-36374@footnote{https://bugs.python.org/issue36374}: Fix a possible null pointer dereference in
@code{merge_consts_recursive()}. Patch by Zackery Spytz.

@item 
bpo-36236@footnote{https://bugs.python.org/issue36236}: At Python initialization, the current directory is no longer
prepended to @ref{489,,sys.path} if it has been removed.

@item 
bpo-36352@footnote{https://bugs.python.org/issue36352}: Python initialization now fails with an error, rather than
silently truncating paths, if a path is too long.

@item 
bpo-36301@footnote{https://bugs.python.org/issue36301}: Python initialization now fails if decoding @code{pybuilddir.txt}
configuration file fails at startup.

@item 
bpo-36333@footnote{https://bugs.python.org/issue36333}: Fix leak in _PyRuntimeState_Fini. Contributed by Stéphane
Wirtel.

@item 
bpo-36332@footnote{https://bugs.python.org/issue36332}: The builtin @ref{4e2,,compile()} can now handle AST objects that
contain assignment expressions. Patch by Pablo Galindo.

@item 
bpo-36282@footnote{https://bugs.python.org/issue36282}: Improved error message for too much positional arguments in
some builtin functions.

@item 
bpo-30040@footnote{https://bugs.python.org/issue30040}: New empty dict uses fewer memory for now.  It used more memory
than empty dict created by @code{dict.clear()}.  And empty dict creation and
deletion is about 2x faster.  Patch by Inada Naoki.

@item 
bpo-36262@footnote{https://bugs.python.org/issue36262}: Fix an unlikely memory leak on conversion from string to float
in the function @code{_Py_dg_strtod()} used by @code{float(str)},
@code{complex(str)}, @ref{85d,,pickle.load()}, @ref{12be,,marshal.load()}, etc.

@item 
bpo-36252@footnote{https://bugs.python.org/issue36252}: Update Unicode databases to version 12.0.0.

@item 
bpo-36218@footnote{https://bugs.python.org/issue36218}: Fix a segfault occurring when sorting a list of heterogeneous
values. Patch contributed by Rémi Lapeyre and Elliot Gorokhovsky.

@item 
bpo-36188@footnote{https://bugs.python.org/issue36188}: Cleaned up left-over vestiges of Python 2 unbound method
handling in method objects and documentation. Patch by Martijn Pieters

@item 
bpo-36124@footnote{https://bugs.python.org/issue36124}: Add a new interpreter-specific dict and expose it in the C-API
via PyInterpreterState_GetDict().  This parallels PyThreadState_GetDict().
However, extension modules should continue using PyModule_GetState() for
their own internal per-interpreter state.

@item 
bpo-35975@footnote{https://bugs.python.org/issue35975}: Add a @code{feature_version} flag to @code{ast.parse()} (documented)
and @code{compile()} (hidden) that allows tweaking the parser to support
older versions of the grammar. In particular, if @code{feature_version} is 5
or 6, the hacks for the @code{async} and @code{await} keyword from PEP 492 are
reinstated. (For 7 or higher, these are unconditionally treated as
keywords, but they are still special tokens rather than @code{NAME} tokens
that the parser driver recognizes.)

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Use UTF-8 as the system encoding on VxWorks.

@item 
bpo-36048@footnote{https://bugs.python.org/issue36048}: The @ref{24a,,__index__()} special method will be used
instead of @ref{249,,__int__()} for implicit conversion of Python
numbers to C integers.  Using the @code{__int__()} method in implicit
conversions has been deprecated.

@item 
bpo-35808@footnote{https://bugs.python.org/issue35808}: Retire pgen and use a modified version of pgen2 to generate the
parser. Patch by Pablo Galindo.
@end itemize

@node Library<24>,Documentation<25>,Core and Builtins<24>,Python 3 8 0 alpha 3
@anchor{whatsnew/changelog id214}@anchor{12bf}
@subsubsection Library



@itemize -

@item 
bpo-36401@footnote{https://bugs.python.org/issue36401}: The class documentation created by pydoc now has a separate
section for readonly properties.

@item 
bpo-36320@footnote{https://bugs.python.org/issue36320}: The typing.NamedTuple() class has deprecated the _field_types
attribute in favor of the __annotations__ attribute which carried the same
information. Also, both attributes were converted from OrderedDict to a
regular dict.

@item 
bpo-34745@footnote{https://bugs.python.org/issue34745}: Fix @ref{9,,asyncio} ssl memory issues caused by circular
references

@item 
bpo-36324@footnote{https://bugs.python.org/issue36324}: Add method to statistics.NormalDist for computing the inverse
cumulative normal distribution.

@item 
bpo-36321@footnote{https://bugs.python.org/issue36321}: collections.namedtuple() misspelled the name of an attribute.
To be consistent with typing.NamedTuple, the attribute name should have
been “_field_defaults” instead of “_fields_defaults”.  For backwards
compatibility, both spellings are now created.  The misspelled version may
be removed in the future.

@item 
bpo-36297@footnote{https://bugs.python.org/issue36297}: “unicode_internal” codec is removed.  It was deprecated since
Python 3.3. Patch by Inada Naoki.

@item 
bpo-36298@footnote{https://bugs.python.org/issue36298}: Raise ModuleNotFoundError in pyclbr when a module can’t be
found. Thanks to ‘mental’ for the bug report.

@item 
bpo-36268@footnote{https://bugs.python.org/issue36268}: Switch the default format used for writing tars with
mod:@ref{fb,,tarfile} to the modern POSIX.1-2001 pax standard, from the
vendor-specific GNU. Contributed by C.A.M. Gerlach.

@item 
bpo-36285@footnote{https://bugs.python.org/issue36285}: Fix integer overflows in the array module. Patch by Stephan
Hohe.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Add _signal module support for VxWorks.

@item 
bpo-36272@footnote{https://bugs.python.org/issue36272}: @ref{a6,,logging} does not silently ignore RecursionError anymore.
Patch contributed by Rémi Lapeyre.

@item 
bpo-36280@footnote{https://bugs.python.org/issue36280}: Add a kind field to ast.Constant. It is ‘u’ if the literal has
a ‘u’ prefix (i.e. a Python 2 style unicode literal), else None.

@item 
bpo-35931@footnote{https://bugs.python.org/issue35931}: The @ref{c4,,pdb} @code{debug} command now gracefully handles all
exceptions.

@item 
bpo-36251@footnote{https://bugs.python.org/issue36251}: Fix format strings used for stderrprinter and re.Match reprs.
Patch by Stephan Hohe.

@item 
bpo-36235@footnote{https://bugs.python.org/issue36235}: Fix @code{CFLAGS} in @code{customize_compiler()} of
@code{distutils.sysconfig}: when the @code{CFLAGS} environment variable is
defined, don’t override @code{CFLAGS} variable with the @code{OPT} variable
anymore. Initial patch written by David Malcolm.

@item 
bpo-35807@footnote{https://bugs.python.org/issue35807}: Update ensurepip to install pip 19.0.3 and setuptools 40.8.0.

@item 
bpo-36139@footnote{https://bugs.python.org/issue36139}: Release GIL when closing @ref{511,,mmap} objects.

@item 
bpo-36179@footnote{https://bugs.python.org/issue36179}: Fix two unlikely reference leaks in _hashopenssl. The leaks
only occur in out-of-memory cases.

@item 
bpo-36169@footnote{https://bugs.python.org/issue36169}: Add overlap() method to statistics.NormalDist.  Computes the
overlapping coefficient for two normal distributions.

@item 
bpo-36103@footnote{https://bugs.python.org/issue36103}: Default buffer size used by @code{shutil.copyfileobj()} is changed
from 16 KiB to 64 KiB on non-Windows platform to reduce system call
overhead. Contributed by Inada Naoki.

@item 
bpo-36130@footnote{https://bugs.python.org/issue36130}: Fix @code{pdb} with @code{skip=...} when stepping into a frame
without a @code{__name__} global.  Patch by Anthony Sottile.

@item 
bpo-35652@footnote{https://bugs.python.org/issue35652}: shutil.copytree(copy_function=…) erroneously pass DirEntry
instead of a path string.

@item 
bpo-35178@footnote{https://bugs.python.org/issue35178}: Ensure custom @ref{12c0,,warnings.formatwarning()} function can
receive @code{line} as positional argument. Based on patch by Tashrif Billah.

@item 
bpo-36106@footnote{https://bugs.python.org/issue36106}: Resolve potential name clash with libm’s sinpi(). Patch by
Dmitrii Pasechnik.

@item 
bpo-36091@footnote{https://bugs.python.org/issue36091}: Clean up reference to async generator in Lib/types. Patch by
Henry Chen.

@item 
bpo-36043@footnote{https://bugs.python.org/issue36043}: @code{FileCookieJar} supports @ref{63d,,path-like object}.
Contributed by Stéphane Wirtel

@item 
bpo-35899@footnote{https://bugs.python.org/issue35899}: Enum has been fixed to correctly handle empty strings and
strings with non-Latin characters (ie. ‘α’, ‘א’) without crashing.
Original patch contributed by Maxwell. Assisted by Stéphane Wirtel.

@item 
bpo-21269@footnote{https://bugs.python.org/issue21269}: Add @code{args} and @code{kwargs} properties to mock call objects.
Contributed by Kumar Akshay.

@item 
bpo-30670@footnote{https://bugs.python.org/issue30670}: @ref{532,,pprint.pp} has been added to pretty-print objects with
dictionary keys being sorted with their insertion order by default.
Parameter @emph{sort_dicts} has been added to @ref{2ac,,pprint.pprint}, @ref{b01,,pprint.pformat}
and @ref{b00,,pprint.PrettyPrinter}. Contributed by Rémi Lapeyre.

@item 
bpo-35843@footnote{https://bugs.python.org/issue35843}: Implement @code{__getitem__} for @code{_NamespacePath}.  Patch by
Anthony Sottile.

@item 
bpo-35802@footnote{https://bugs.python.org/issue35802}: Clean up code which checked presence of @code{os.stat} /
@code{os.lstat} / @code{os.chmod} which are always present.  Patch by Anthony
Sottile.

@item 
bpo-35715@footnote{https://bugs.python.org/issue35715}: Librates the return value of a ProcessPoolExecutor
_process_worker after it’s no longer needed to free memory

@item 
bpo-35493@footnote{https://bugs.python.org/issue35493}: Use @ref{c72,,multiprocessing.connection.wait()} instead of polling
each 0.2 seconds for worker updates in @code{multiprocessing.Pool}.
Patch by Pablo Galindo.

@item 
bpo-35661@footnote{https://bugs.python.org/issue35661}: Store the venv prompt in pyvenv.cfg.

@item 
bpo-35121@footnote{https://bugs.python.org/issue35121}: Don’t set cookie for a request when the request path is a
prefix match of the cookie’s path attribute but doesn’t end with “/”.
Patch by Karthikeyan Singaravelan.

@item 
bpo-21478@footnote{https://bugs.python.org/issue21478}: Calls to a child function created with
@ref{122a,,unittest.mock.create_autospec()} should propagate to the parent.
Patch by Karthikeyan Singaravelan.

@item 
bpo-35198@footnote{https://bugs.python.org/issue35198}: Fix C++ extension compilation on AIX
@end itemize

@node Documentation<25>,Tests<23>,Library<24>,Python 3 8 0 alpha 3
@anchor{whatsnew/changelog id215}@anchor{12c1}
@subsubsection Documentation



@itemize -

@item 
bpo-36329@footnote{https://bugs.python.org/issue36329}: Declare the path of the Python binary for the usage of
@code{Tools/scripts/serve.py} when executing @code{make -C Doc/ serve}.
Contributed by Stéphane Wirtel

@item 
bpo-36138@footnote{https://bugs.python.org/issue36138}: Improve documentation about converting datetime.timedelta to
scalars.

@item 
bpo-21314@footnote{https://bugs.python.org/issue21314}: A new entry was added to the Core Language Section of the
Programming FAQ, which explaines the usage of slash(/) in the signature of
a function. Patch by Lysandros Nikolaou
@end itemize

@node Tests<23>,Build<24>,Documentation<25>,Python 3 8 0 alpha 3
@anchor{whatsnew/changelog id216}@anchor{12c2}
@subsubsection Tests



@itemize -

@item 
bpo-36234@footnote{https://bugs.python.org/issue36234}: test_posix.PosixUidGidTests: add tests for invalid uid/gid type
(str). Initial patch written by David Malcolm.

@item 
bpo-29571@footnote{https://bugs.python.org/issue29571}: Fix @code{test_re.test_locale_flag()}:  use
@code{locale.getpreferredencoding()} rather than @code{locale.getlocale()} to
get the locale encoding. With some locales, @code{locale.getlocale()} returns
the wrong encoding.

@item 
bpo-36123@footnote{https://bugs.python.org/issue36123}: Fix race condition in test_socket.
@end itemize

@node Build<24>,Windows<22>,Tests<23>,Python 3 8 0 alpha 3
@anchor{whatsnew/changelog id217}@anchor{12c3}
@subsubsection Build



@itemize -

@item 
bpo-36356@footnote{https://bugs.python.org/issue36356}: Fix leaks that led to build failure when configured with
address sanitizer.

@item 
bpo-36146@footnote{https://bugs.python.org/issue36146}: Add @code{TEST_EXTENSIONS} constant to @code{setup.py} to allow to
not build test extensions like @code{_testcapi}.

@item 
bpo-36146@footnote{https://bugs.python.org/issue36146}: Fix setup.py on macOS: only add @code{/usr/include/ffi} to include
directories of _ctypes, not for all extensions.

@item 
bpo-31904@footnote{https://bugs.python.org/issue31904}: Enable build system to cross-build for VxWorks RTOS.
@end itemize

@node Windows<22>,IDLE<22>,Build<24>,Python 3 8 0 alpha 3
@anchor{whatsnew/changelog id218}@anchor{12c4}
@subsubsection Windows



@itemize -

@item 
bpo-36312@footnote{https://bugs.python.org/issue36312}: Fixed decoders for the following code pages: 50220, 50221,
50222, 50225, 50227, 50229, 57002 through 57011, 65000 and 42.

@item 
bpo-36264@footnote{https://bugs.python.org/issue36264}: Don’t honor POSIX @code{HOME} in @code{os.path.expanduser} on
windows.  Patch by Anthony Sottile.

@item 
bpo-24643@footnote{https://bugs.python.org/issue24643}: Fix name collisions due to @code{#define timezone _timezone} in
PC/pyconfig.h.
@end itemize

@node IDLE<22>,Tools/Demos<10>,Windows<22>,Python 3 8 0 alpha 3
@anchor{whatsnew/changelog id219}@anchor{12c5}
@subsubsection IDLE



@itemize -

@item 
bpo-36405@footnote{https://bugs.python.org/issue36405}: Use dict unpacking in idlelib.

@item 
bpo-36396@footnote{https://bugs.python.org/issue36396}: Remove fgBg param of idlelib.config.GetHighlight(). This param
was only used twice and changed the return type.

@item 
bpo-36176@footnote{https://bugs.python.org/issue36176}: Fix IDLE autocomplete & calltip popup colors. Prevent conflicts
with Linux dark themes (and slightly darken calltip background).

@item 
bpo-23205@footnote{https://bugs.python.org/issue23205}: For the grep module, add tests for findfiles, refactor
findfiles to be a module-level function, and refactor findfiles to use
os.walk.

@item 
bpo-23216@footnote{https://bugs.python.org/issue23216}: Add docstrings to IDLE search modules.

@item 
bpo-36152@footnote{https://bugs.python.org/issue36152}: Remove colorizer.ColorDelegator.close_when_done and the
corresponding argument of .close().  In IDLE, both have always been None
or False since 2007.

@item 
bpo-32129@footnote{https://bugs.python.org/issue32129}: Avoid blurry IDLE application icon on macOS with Tk 8.6. Patch
by Kevin Walzer.

@item 
bpo-36096@footnote{https://bugs.python.org/issue36096}: Refactor class variables to instance variables in colorizer.

@item 
bpo-30348@footnote{https://bugs.python.org/issue30348}: Increase test coverage of idlelib.autocomplete by 30%. Patch by
Louie Lu
@end itemize

@node Tools/Demos<10>,C API<24>,IDLE<22>,Python 3 8 0 alpha 3
@anchor{whatsnew/changelog id220}@anchor{12c6}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-35132@footnote{https://bugs.python.org/issue35132}: Fix py-list and py-bt commands of python-gdb.py on gdb7.

@item 
bpo-32217@footnote{https://bugs.python.org/issue32217}: Fix freeze script on Windows.
@end itemize

@node C API<24>,,Tools/Demos<10>,Python 3 8 0 alpha 3
@anchor{whatsnew/changelog id221}@anchor{12c7}
@subsubsection C API



@itemize -

@item 
bpo-36381@footnote{https://bugs.python.org/issue36381}: Raise @code{DeprecationWarning} when ‘#’ formats are used for
building or parsing values without @code{PY_SSIZE_T_CLEAN}.

@item 
bpo-36142@footnote{https://bugs.python.org/issue36142}: The whole coreconfig.h header is now excluded from
Py_LIMITED_API. Move functions definitions into a new internal
pycore_coreconfig.h header.
@end itemize

@node Python 3 8 0 alpha 2,Python 3 8 0 alpha 1,Python 3 8 0 alpha 3,Changelog
@anchor{whatsnew/changelog python-3-8-0-alpha-2}@anchor{12c8}
@subsection Python 3.8.0 alpha 2


@emph{Release date: 2019-02-25}

@menu
* Core and Builtins: Core and Builtins<25>. 
* Library: Library<25>. 
* Documentation: Documentation<26>. 
* Tests: Tests<24>. 
* Windows: Windows<23>. 
* IDLE: IDLE<23>. 

@end menu

@node Core and Builtins<25>,Library<25>,,Python 3 8 0 alpha 2
@anchor{whatsnew/changelog id222}@anchor{12c9}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-36052@footnote{https://bugs.python.org/issue36052}: Raise a @ref{21a,,SyntaxError} when assigning a value to
@ref{257,,__debug__} with the Assignment Operator. Contributed by Stéphane Wirtel
and Pablo Galindo.

@item 
bpo-36012@footnote{https://bugs.python.org/issue36012}: Doubled the speed of class variable writes.  When a non-dunder
attribute was updated, there was an unnecessary call to update slots.

@item 
bpo-35942@footnote{https://bugs.python.org/issue35942}: The error message emitted when returning invalid types from
@code{__fspath__} in interfaces that allow passing @ref{795,,PathLike}
objects has been improved and now it does explain the origin of the error.

@item 
bpo-36016@footnote{https://bugs.python.org/issue36016}: @code{gc.get_objects} can now receive an optional parameter
indicating a generation to get objects from. Patch by Pablo Galindo.

@item 
bpo-1054041@footnote{https://bugs.python.org/issue1054041}: When the main interpreter exits due to an uncaught
KeyboardInterrupt, the process now exits in the appropriate manner for its
parent process to detect that a SIGINT or ^C terminated the process.  This
allows shells and batch scripts to understand that the user has asked them
to stop.

@item 
bpo-35992@footnote{https://bugs.python.org/issue35992}: Fix @code{__class_getitem__()} not being called on a class with a
custom non-subscriptable metaclass.

@item 
bpo-35993@footnote{https://bugs.python.org/issue35993}: Fix a crash on fork when using subinterpreters. Contributed by
Stéphane Wirtel

@item 
bpo-35991@footnote{https://bugs.python.org/issue35991}: Fix a potential double free in Modules/_randommodule.c.

@item 
bpo-35961@footnote{https://bugs.python.org/issue35961}: Fix a crash in slice_richcompare(): use strong references
rather than stolen references for the two temporary internal tuples.

@item 
bpo-35911@footnote{https://bugs.python.org/issue35911}: Enable the creation of cell objects by adding a
@code{cell.__new__} method, and expose the type @code{cell} in @code{Lib/types.py}
under the name CellType. Patch by Pierre Glaser.

@item 
bpo-12822@footnote{https://bugs.python.org/issue12822}: Use monotonic clock for @code{pthread_cond_timedwait} when
@code{pthread_condattr_setclock} and @code{CLOCK_MONOTONIC} are available.

@item 
bpo-15248@footnote{https://bugs.python.org/issue15248}: The compiler emits now syntax warnings in the case when a comma
is likely missed before tuple or list.

@item 
bpo-35886@footnote{https://bugs.python.org/issue35886}: The implementation of PyInterpreterState has been moved into
the internal header files (guarded by Py_BUILD_CORE).

@item 
bpo-31506@footnote{https://bugs.python.org/issue31506}: Clarify the errors reported when @code{object.__new__} and
@code{object.__init__} receive more than one argument. Contributed by Sanyam
Khurana.

@item 
bpo-35724@footnote{https://bugs.python.org/issue35724}: Signal-handling is now guaranteed to happen relative to the
main interpreter.

@item 
bpo-33608@footnote{https://bugs.python.org/issue33608}: We added a new internal _Py_AddPendingCall() that operates
relative to the provided interpreter.  This allows us to use the existing
implementation to ask another interpreter to do work that cannot be done
in the current interpreter, like decref an object the other interpreter
owns.  The existing Py_AddPendingCall() only operates relative to the main
interpreter.

@item 
bpo-33989@footnote{https://bugs.python.org/issue33989}: Fix a possible crash in @ref{705,,list.sort()} when sorting objects
with @code{ob_type->tp_richcompare == NULL}.  Patch by Zackery Spytz.
@end itemize

@node Library<25>,Documentation<26>,Core and Builtins<25>,Python 3 8 0 alpha 2
@anchor{whatsnew/changelog id223}@anchor{12ca}
@subsubsection Library



@itemize -

@item 
bpo-35512@footnote{https://bugs.python.org/issue35512}: @ref{129d,,unittest.mock.patch.dict()} used as a decorator with
string target resolves the target during function call instead of during
decorator construction. Patch by Karthikeyan Singaravelan.

@item 
bpo-36018@footnote{https://bugs.python.org/issue36018}: Add statistics.NormalDist, a tool for creating and manipulating
normal distributions of random variable.  Features a composite class that
treats the mean and standard deviation of measurement data as single
entity.

@item 
bpo-35904@footnote{https://bugs.python.org/issue35904}: Added statistics.fmean() as a faster, floating point variant of
the existing mean() function.

@item 
bpo-35918@footnote{https://bugs.python.org/issue35918}: Removed broken @code{has_key} method from
multiprocessing.managers.SyncManager.dict. Contributed by Rémi Lapeyre.

@item 
bpo-18283@footnote{https://bugs.python.org/issue18283}: Add support for bytes to @ref{11ac,,shutil.which()}.

@item 
bpo-35960@footnote{https://bugs.python.org/issue35960}: Fix @ref{12cb,,dataclasses.field()} throwing away empty mapping
objects passed as metadata.

@item 
bpo-35500@footnote{https://bugs.python.org/issue35500}: Write expected and actual call parameters on separate lines in
@ref{12cc,,unittest.mock.Mock.assert_called_with()} assertion errors.
Contributed by Susan Su.

@item 
bpo-35931@footnote{https://bugs.python.org/issue35931}: The @ref{c4,,pdb} @code{debug} command now gracefully handles syntax
errors.

@item 
bpo-24209@footnote{https://bugs.python.org/issue24209}: In http.server script, rely on getaddrinfo to bind to preferred
address based on the bind parameter. Now default bind or binding to a name
may bind to IPv6 or dual-stack, depending on the environment.

@item 
bpo-35321@footnote{https://bugs.python.org/issue35321}: Set @code{__spec__.origin} of  @code{_frozen_importlib} to frozen so
that it matches the behavior of @code{_frozen_importlib_external}.  Patch by
Nina Zakharenko.

@item 
bpo-35378@footnote{https://bugs.python.org/issue35378}: Fix a reference issue inside @code{multiprocessing.Pool} that
caused the pool to remain alive if it was deleted without being closed or
terminated explicitly. A new strong reference is added to the pool
iterators to link the lifetime of the pool to the lifetime of its
iterators so the pool does not get destroyed if a pool iterator is still
alive.

@item 
bpo-34294@footnote{https://bugs.python.org/issue34294}: re module, fix wrong capturing groups in rare cases.
@ref{dd2,,re.search()}, @ref{bbb,,re.findall()}, @ref{72e,,re.sub()} and other functions
that scan through string looking for a match, should reset capturing
groups between two match attempts. Patch by Ma Lin.

@item 
bpo-35615@footnote{https://bugs.python.org/issue35615}: @ref{12f,,weakref}: Fix a RuntimeError when copying a
WeakKeyDictionary or a WeakValueDictionary, due to some keys or values
disappearing while iterating.

@item 
bpo-35606@footnote{https://bugs.python.org/issue35606}: Implement @ref{50c,,math.prod()} as analogous function to
@ref{50d,,sum()} that returns the product of a ‘start’ value (default: 1) times
an iterable of numbers. Patch by Pablo Galindo.

@item 
bpo-32417@footnote{https://bugs.python.org/issue32417}: Performing arithmetic between @ref{363,,datetime.datetime}
subclasses and @ref{4c8,,datetime.timedelta} now returns an object of the
same type as the @ref{363,,datetime.datetime} subclass. As a result,
@ref{4c9,,datetime.datetime.astimezone()} and alternate constructors like
@ref{12cd,,datetime.datetime.now()} and @code{datetime.fromtimestamp()} called
with a @code{tz} argument now @emph{also} retain their subclass.

@item 
bpo-35153@footnote{https://bugs.python.org/issue35153}: Add @emph{headers} optional keyword-only parameter to
@ref{56c,,xmlrpc.client.ServerProxy}, @code{xmlrpc.client.Transport} and
@code{xmlrpc.client.SafeTransport}.  Patch by Cédric Krier.

@item 
bpo-34572@footnote{https://bugs.python.org/issue34572}: Fix C implementation of pickle.loads to use importlib’s locking
mechanisms, and thereby avoid using partially-loaded modules. Patch by Tim
Burgess.
@end itemize

@node Documentation<26>,Tests<24>,Library<25>,Python 3 8 0 alpha 2
@anchor{whatsnew/changelog id224}@anchor{12ce}
@subsubsection Documentation



@itemize -

@item 
bpo-36083@footnote{https://bugs.python.org/issue36083}: Fix formatting of –check-hash-based-pycs options in the
manpage Synopsis.

@item 
bpo-36007@footnote{https://bugs.python.org/issue36007}: Bump minimum sphinx version to 1.8.  Patch by Anthony Sottile.

@item 
bpo-22062@footnote{https://bugs.python.org/issue22062}: Update documentation and docstrings for pathlib. Original patch
by Mike Short.
@end itemize

@node Tests<24>,Windows<23>,Documentation<26>,Python 3 8 0 alpha 2
@anchor{whatsnew/changelog id225}@anchor{12cf}
@subsubsection Tests



@itemize -

@item 
bpo-27313@footnote{https://bugs.python.org/issue27313}: Avoid test_ttk_guionly ComboboxTest failure with macOS Cocoa
Tk.

@item 
bpo-36019@footnote{https://bugs.python.org/issue36019}: Add test.support.TEST_HTTP_URL and replace references of
@indicateurl{http://www.example.com} by this new constant. Contributed by Stéphane
Wirtel.

@item 
bpo-36037@footnote{https://bugs.python.org/issue36037}: Fix test_ssl for strict OpenSSL configuration like RHEL8 strict
crypto policy. Use older TLS version for minimum TLS version of the server
SSL context if needed, to test TLS version older than default minimum TLS
version.

@item 
bpo-35798@footnote{https://bugs.python.org/issue35798}: Added @code{test.support.check_syntax_warning()}.

@item 
bpo-35505@footnote{https://bugs.python.org/issue35505}: Make test_imap4_host_default_value independent on whether the
local IMAP server is running.

@item 
bpo-35917@footnote{https://bugs.python.org/issue35917}: multiprocessing: provide unit tests for SyncManager and
SharedMemoryManager classes + all the shareable types which are supposed
to be supported by them.  (patch by Giampaolo Rodola)

@item 
bpo-35704@footnote{https://bugs.python.org/issue35704}: Skip @code{test_shutil.test_unpack_archive_xztar} to prevent a
MemoryError on 32-bit AIX when MAXDATA setting is less than 0x20000000.

Patch by Michael Felt (aixtools)

@item 
bpo-34720@footnote{https://bugs.python.org/issue34720}: Assert m_state != NULL to mimic GC traversal functions that do
not correctly handle module creation when the module state has not been
created.
@end itemize

@node Windows<23>,IDLE<23>,Tests<24>,Python 3 8 0 alpha 2
@anchor{whatsnew/changelog id226}@anchor{12d0}
@subsubsection Windows



@itemize -

@item 
bpo-35976@footnote{https://bugs.python.org/issue35976}: Added ARM build support to Windows build files in PCBuild.

@item 
bpo-35692@footnote{https://bugs.python.org/issue35692}: @code{pathlib} no longer raises when checking file and directory
existence on drives that are not ready

@item 
bpo-35872@footnote{https://bugs.python.org/issue35872}: Uses the base Python executable when invoking venv in a virtual
environment

@item 
bpo-35873@footnote{https://bugs.python.org/issue35873}: Prevents venv paths being inherited by child processes

@item 
bpo-35299@footnote{https://bugs.python.org/issue35299}: Fix sysconfig detection of the source directory and distutils
handling of pyconfig.h during PGO profiling
@end itemize

@node IDLE<23>,,Windows<23>,Python 3 8 0 alpha 2
@anchor{whatsnew/changelog id227}@anchor{12d1}
@subsubsection IDLE



@itemize -

@item 
bpo-24310@footnote{https://bugs.python.org/issue24310}: IDLE – Document settings dialog font tab sample.

@item 
bpo-35833@footnote{https://bugs.python.org/issue35833}: Revise IDLE doc for control codes sent to Shell. Add a code
example block.

@item 
bpo-35689@footnote{https://bugs.python.org/issue35689}: Add docstrings and unittests for colorizer.py.
@end itemize

@node Python 3 8 0 alpha 1,Python 3 7 0 final,Python 3 8 0 alpha 2,Changelog
@anchor{whatsnew/changelog python-3-8-0-alpha-1}@anchor{12d2}
@subsection Python 3.8.0 alpha 1


@emph{Release date: 2019-02-03}

@menu
* Security: Security<18>. 
* Core and Builtins: Core and Builtins<26>. 
* Library: Library<26>. 
* Documentation: Documentation<27>. 
* Tests: Tests<25>. 
* Build: Build<25>. 
* Windows: Windows<24>. 
* macOS: macOS<20>. 
* IDLE: IDLE<24>. 
* Tools/Demos: Tools/Demos<11>. 
* C API: C API<25>. 

@end menu

@node Security<18>,Core and Builtins<26>,,Python 3 8 0 alpha 1
@anchor{whatsnew/changelog id228}@anchor{12d3}
@subsubsection Security



@itemize -

@item 
bpo-35746@footnote{https://bugs.python.org/issue35746}: [CVE-2019-5010] Fix a NULL pointer deref in ssl module. The
cert parser did not handle CRL distribution points with empty DP or URI
correctly. A malicious or buggy certificate can result into segfault.
Vulnerability (TALOS-2018-0758) reported by Colin Read and Nicolas Edet of
Cisco.

@item 
bpo-34812@footnote{https://bugs.python.org/issue34812}: The @ref{461,,-I} command line option (run Python in isolated
mode) is now also copied by the @ref{b3,,multiprocessing} and
@ref{37,,distutils} modules when spawning child processes. Previously, only
@ref{460,,-E} and @ref{f15,,-s} options (enabled by @ref{461,,-I}) were
copied.

@item 
bpo-34791@footnote{https://bugs.python.org/issue34791}: The xml.sax and xml.dom.domreg no longer use environment
variables to override parser implementations when
sys.flags.ignore_environment is set by -E or -I arguments.

@item 
bpo-17239@footnote{https://bugs.python.org/issue17239}: The xml.sax and xml.dom.minidom parsers no longer processes
external entities by default. External DTD and ENTITY declarations no
longer load files or create network connections.

@item 
bpo-34623@footnote{https://bugs.python.org/issue34623}: CVE-2018-14647: The C accelerated _elementtree module now
initializes hash randomization salt from _Py_HashSecret instead of
libexpat’s default CSPRNG.

@item 
bpo-34405@footnote{https://bugs.python.org/issue34405}: Updated to OpenSSL 1.1.0i for Windows builds.

@item 
bpo-33871@footnote{https://bugs.python.org/issue33871}: Fixed sending the part of the file in @ref{62c,,os.sendfile()} on
macOS.  Using the @emph{trailers} argument could cause sending more bytes from
the input file than was specified.

@item 
bpo-32533@footnote{https://bugs.python.org/issue32533}: Fixed thread-safety of error handling in _ssl.

@item 
bpo-33136@footnote{https://bugs.python.org/issue33136}: Harden ssl module against LibreSSL CVE-2018-8970.
X509_VERIFY_PARAM_set1_host() is called with an explicit namelen. A new
test ensures that NULL bytes are not allowed.

@item 
bpo-33001@footnote{https://bugs.python.org/issue33001}: Minimal fix to prevent buffer overrun in os.symlink on Windows

@item 
bpo-32981@footnote{https://bugs.python.org/issue32981}: Regexes in difflib and poplib were vulnerable to catastrophic
backtracking. These regexes formed potential DOS vectors (REDOS). They
have been refactored. This resolves CVE-2018-1060 and CVE-2018-1061. Patch
by Jamie Davis.

@item 
bpo-28414@footnote{https://bugs.python.org/issue28414}: The ssl module now allows users to perform their own IDN
en/decoding when using SNI.
@end itemize

@node Core and Builtins<26>,Library<26>,Security<18>,Python 3 8 0 alpha 1
@anchor{whatsnew/changelog id229}@anchor{12d4}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-35877@footnote{https://bugs.python.org/issue35877}: Make parenthesis optional for named expressions in while
statement. Patch by Karthikeyan Singaravelan.

@item 
bpo-35814@footnote{https://bugs.python.org/issue35814}: Allow same right hand side expressions in annotated assignments
as in normal ones. In particular, @code{x: Tuple[int, int] = 1, 2} (without
parentheses on the right) is now allowed.

@item 
bpo-35766@footnote{https://bugs.python.org/issue35766}: Add the option to parse PEP 484 type comments in the ast
module. (Off by default.) This is merging the key functionality of the
third party fork thereof,
[typed_ast](@indicateurl{https://github.com/python/typed_ast}).

@item 
bpo-35713@footnote{https://bugs.python.org/issue35713}: Reorganize Python initialization to get working exceptions and
sys.stderr earlier.

@item 
bpo-33416@footnote{https://bugs.python.org/issue33416}: Add end line and end column position information to the Python
AST nodes. This is a C-level backwards incompatible change.

@item 
bpo-35720@footnote{https://bugs.python.org/issue35720}: Fixed a minor memory leak in pymain_parse_cmdline_impl function
in Modules/main.c

@item 
bpo-35634@footnote{https://bugs.python.org/issue35634}: @code{func(**kwargs)} will now raise an error when @code{kwargs} is a
mapping containing multiple entries with the same key. An error was
already raised when other keyword arguments are passed before @code{**kwargs}
since Python 3.6.

@item 
bpo-35623@footnote{https://bugs.python.org/issue35623}: Fix a crash when sorting very long lists. Patch by Stephan
Hohe.

@item 
bpo-35214@footnote{https://bugs.python.org/issue35214}: clang Memory Sanitizer build instrumentation was added to work
around false positives from posix, socket, time, test_io, and
test_faulthandler.

@item 
bpo-35560@footnote{https://bugs.python.org/issue35560}: Fix an assertion error in @ref{785,,format()} in debug build for
floating point formatting with “n” format, zero padding and small width.
Release build is not impacted. Patch by Karthikeyan Singaravelan.

@item 
bpo-35552@footnote{https://bugs.python.org/issue35552}: Format characters @code{%s} and @code{%V} in
@ref{398,,PyUnicode_FromFormat()} and @code{%s} in @ref{12d5,,PyBytes_FromFormat()}
no longer read memory past the limit if @emph{precision} is specified.

@item 
bpo-35504@footnote{https://bugs.python.org/issue35504}: Fix segfaults and @ref{891,,SystemError}s when deleting certain
attributes. Patch by Zackery Spytz.

@item 
bpo-35504@footnote{https://bugs.python.org/issue35504}: Fixed a SystemError when delete the characters_written
attribute of an OSError.

@item 
bpo-35494@footnote{https://bugs.python.org/issue35494}: Improved syntax error messages for unbalanced parentheses in
f-string.

@item 
bpo-35444@footnote{https://bugs.python.org/issue35444}: Fixed error handling in pickling methods when fail to look up
builtin “getattr”. Sped up pickling iterators.

@item 
bpo-35436@footnote{https://bugs.python.org/issue35436}: Fix various issues with memory allocation error handling.
Patch by Zackery Spytz.

@item 
bpo-35423@footnote{https://bugs.python.org/issue35423}: Separate the signal handling trigger in the eval loop from the
“pending calls” machinery. There is no semantic change and the difference
in performance is insignificant.

@item 
bpo-35357@footnote{https://bugs.python.org/issue35357}: Internal attributes’ names of unittest.mock._Call and
unittest.mock.MagicProxy (name, parent & from_kall) are now prefixed with
_mock_ in order to prevent clashes with widely used object attributes.
Fixed minor typo in test function name.

@item 
bpo-35372@footnote{https://bugs.python.org/issue35372}: Fixed the code page decoder for input longer than 2 GiB
containing undecodable bytes.

@item 
bpo-35336@footnote{https://bugs.python.org/issue35336}: Fix PYTHONCOERCECLOCALE=1 environment variable: only coerce the
C locale if the LC_CTYPE locale is “C”.

@item 
bpo-31241@footnote{https://bugs.python.org/issue31241}: The @emph{lineno} and @emph{col_offset} attributes of AST nodes for list
comprehensions, generator expressions and tuples are now point to the
opening parenthesis or square brace. For tuples without parenthesis they
point to the position of the first item.

@item 
bpo-33954@footnote{https://bugs.python.org/issue33954}: For @ref{784,,str.format()}, @code{float.__format__()} and
@code{complex.__format__()} methods for non-ASCII decimal point when using
the “n” formatter.

@item 
bpo-35269@footnote{https://bugs.python.org/issue35269}: Fix a possible segfault involving a newly-created coroutine.
Patch by Zackery Spytz.

@item 
bpo-35224@footnote{https://bugs.python.org/issue35224}: Implement 
@geindex Python Enhancement Proposals; PEP 572
PEP 572@footnote{https://www.python.org/dev/peps/pep-0572} (assignment expressions). Patch by Emily
Morehouse.

@item 
bpo-32492@footnote{https://bugs.python.org/issue32492}: Speed up @code{namedtuple} attribute access by 1.6x using a C
fast-path for the name descriptors. Patch by Pablo Galindo.

@item 
bpo-35214@footnote{https://bugs.python.org/issue35214}: Fixed an out of bounds memory access when parsing a truncated
unicode escape sequence at the end of a string such as @code{'\N'}.  It would
read one byte beyond the end of the memory allocation.

@item 
bpo-35214@footnote{https://bugs.python.org/issue35214}: The interpreter and extension modules have had annotations
added so that they work properly under clang’s Memory Sanitizer.  A new
configure flag –with-memory-sanitizer has been added to make test builds
of this nature easier to perform.

@item 
bpo-35193@footnote{https://bugs.python.org/issue35193}: Fix an off by one error in the bytecode peephole optimizer
where it could read bytes beyond the end of bounds of an array when
removing unreachable code. This bug was present in every release of Python
3.6 and 3.7 until now.

@item 
bpo-35169@footnote{https://bugs.python.org/issue35169}: Improved error messages for forbidden assignments.

@item 
bpo-34022@footnote{https://bugs.python.org/issue34022}: Fix handling of hash-based bytecode files in @ref{14a,,zipimport}.
Patch by Elvis Pranskevichus.

@item 
bpo-28401@footnote{https://bugs.python.org/issue28401}: Debug builds will no longer to attempt to import extension
modules built for the ABI as they were never compatible to begin with.
Patch by Stefano Rivera.

@item 
bpo-29341@footnote{https://bugs.python.org/issue29341}: Clarify in the docstrings of @ref{c0,,os} methods that path-like
objects are also accepted as input parameters.

@item 
bpo-35050@footnote{https://bugs.python.org/issue35050}: @ref{ea,,socket}: Fix off-by-one bug in length check for
@code{AF_ALG} name and type.

@item 
bpo-29743@footnote{https://bugs.python.org/issue29743}: Raise @ref{17c,,ValueError} instead of @ref{347,,OverflowError} in case
of a negative @code{_length_} in a @ref{12d6,,ctypes.Array} subclass.  Also
raise @ref{162,,TypeError} instead of @ref{163,,AttributeError} for non-integer
@code{_length_}. Original patch by Oren Milman.

@item 
bpo-16806@footnote{https://bugs.python.org/issue16806}: Fix @code{lineno} and @code{col_offset} for multi-line string tokens.

@item 
bpo-35029@footnote{https://bugs.python.org/issue35029}: @ref{4c7,,SyntaxWarning} raised as an exception at code generation
time will be now replaced with a @ref{21a,,SyntaxError} for better error
reporting.

@item 
bpo-34983@footnote{https://bugs.python.org/issue34983}: Expose @ref{12d7,,symtable.Symbol.is_nonlocal()} in the symtable
module. Patch by Pablo Galindo.

@item 
bpo-34974@footnote{https://bugs.python.org/issue34974}: @ref{2f0,,bytes} and @ref{2f1,,bytearray} constructors no longer
convert unexpected exceptions (e.g. @ref{10de,,MemoryError} and
@ref{4ca,,KeyboardInterrupt}) to @ref{162,,TypeError}.

@item 
bpo-34939@footnote{https://bugs.python.org/issue34939}: Allow annotated names in module namespace that are declared
global before the annotation happens. Patch by Pablo Galindo.

@item 
bpo-34973@footnote{https://bugs.python.org/issue34973}: Fixed crash in @ref{2f0,,bytes()} when the @ref{43a,,list} argument is
mutated while it is iterated.

@item 
bpo-34876@footnote{https://bugs.python.org/issue34876}: The @emph{lineno} and @emph{col_offset} attributes of the AST for
decorated function and class refer now to the position of the
corresponding @code{def}, @code{async def} and @code{class} instead of the position
of the first decorator. This leads to more correct line reporting in
tracing. This is the only case when the position of child AST nodes can
precede the position of the parent AST node.

@item 
bpo-34879@footnote{https://bugs.python.org/issue34879}: Fix a possible null pointer dereference in bytesobject.c.
Patch by Zackery Spytz.

@item 
bpo-34784@footnote{https://bugs.python.org/issue34784}: Fix the implementation of PyStructSequence_NewType in order to
create heap allocated StructSequences.

@item 
bpo-32912@footnote{https://bugs.python.org/issue32912}: A @ref{4c7,,SyntaxWarning} is now emitted instead of a
@ref{1b2,,DeprecationWarning} for invalid escape sequences in string and bytes
literals.

@item 
bpo-34854@footnote{https://bugs.python.org/issue34854}: Fixed a crash in compiling string annotations containing a
lambda with a keyword-only argument that doesn’t have a default value.

@item 
bpo-34850@footnote{https://bugs.python.org/issue34850}: The compiler now produces a @ref{4c7,,SyntaxWarning} when identity
checks (@code{is} and @code{is not}) are used with certain types of literals
(e.g. strings, ints).  These can often work by accident in CPython, but
are not guaranteed by the language spec.  The warning advises users to use
equality tests (@code{==} and @code{!=}) instead.

@item 
bpo-34824@footnote{https://bugs.python.org/issue34824}: Fix a possible null pointer dereference in Modules/_ssl.c.
Patch by Zackery Spytz.

@item 
bpo-30156@footnote{https://bugs.python.org/issue30156}: The C function @code{property_descr_get()} uses a “cached” tuple
to optimize function calls. But this tuple can be discovered in debug mode
with @code{sys.getobjects()}. Remove the optimization, it’s not really
worth it and it causes 3 different crashes last years.

@item 
bpo-34762@footnote{https://bugs.python.org/issue34762}: Fix contextvars C API to use PyObject* pointer types.

@item 
bpo-34751@footnote{https://bugs.python.org/issue34751}: The hash function for tuples is now based on xxHash which gives
better collision results on (formerly) pathological cases. Additionally,
on 64-bit systems it improves tuple hashes in general. Patch by Jeroen
Demeyer with substantial contributions by Tim Peters.

@item 
bpo-34735@footnote{https://bugs.python.org/issue34735}: Fix a memory leak in Modules/timemodule.c.  Patch by Zackery
Spytz.

@item 
bpo-34683@footnote{https://bugs.python.org/issue34683}: Fixed a bug where some SyntaxError error pointed to locations
that were off-by-one.

@item 
bpo-34651@footnote{https://bugs.python.org/issue34651}: Only allow the main interpreter to fork.  The avoids the
possibility of affecting the main interpreter, which is critical to
operation of the runtime.

@item 
bpo-34653@footnote{https://bugs.python.org/issue34653}: Remove unused function PyParser_SimpleParseStringFilename.

@item 
bpo-32236@footnote{https://bugs.python.org/issue32236}: Warn that line buffering is not supported if @ref{1c8,,open()} is
called with binary mode and @code{buffering=1}.

@item 
bpo-34641@footnote{https://bugs.python.org/issue34641}: Further restrict the syntax of the left-hand side of keyword
arguments in function calls. In particular, @code{f((keyword)=arg)} is now
disallowed.

@item 
bpo-34637@footnote{https://bugs.python.org/issue34637}: Make the @emph{start} argument to @emph{sum()} visible as a keyword
argument.

@item 
bpo-1621@footnote{https://bugs.python.org/issue1621}: Do not assume signed integer overflow behavior (C undefined
behavior) when performing set hash table resizing.

@item 
bpo-34588@footnote{https://bugs.python.org/issue34588}: Fix an off-by-one in the recursive call pruning feature of
traceback formatting.

@item 
bpo-34485@footnote{https://bugs.python.org/issue34485}: On Windows, the LC_CTYPE is now set to the user preferred
locale at startup. Previously, the LC_CTYPE locale was “C” at startup, but
changed when calling setlocale(LC_CTYPE, “”) or setlocale(LC_ALL, “”).

@item 
bpo-34485@footnote{https://bugs.python.org/issue34485}: Standard streams like sys.stdout now use the “surrogateescape”
error handler, instead of “strict”, on the POSIX locale (when the C locale
is not coerced and the UTF-8 Mode is disabled).

@item 
bpo-34485@footnote{https://bugs.python.org/issue34485}: Fix the error handler of standard streams like sys.stdout:
PYTHONIOENCODING=”:” is now ignored instead of setting the error handler
to “strict”.

@item 
bpo-34485@footnote{https://bugs.python.org/issue34485}: Python now gets the locale encoding with C code to initialize
the encoding of standard streams like sys.stdout. Moreover, the encoding
is now initialized to the Python codec name to get a normalized encoding
name and to ensure that the codec is loaded. The change avoids importing
_bootlocale and _locale modules at startup by default.

@item 
bpo-34527@footnote{https://bugs.python.org/issue34527}: On FreeBSD, Py_DecodeLocale() and Py_EncodeLocale() now also
forces the ASCII encoding if the LC_CTYPE locale is “POSIX”, not only if
the LC_CTYPE locale is “C”.

@item 
bpo-34527@footnote{https://bugs.python.org/issue34527}: The UTF-8 Mode is now also enabled by the “POSIX” locale, not
only by the “C” locale.

@item 
bpo-34403@footnote{https://bugs.python.org/issue34403}: On HP-UX with C or POSIX locale, sys.getfilesystemencoding()
now returns “ascii” instead of “roman8” (when the UTF-8 Mode is disabled
and the C locale is not coerced).

@item 
bpo-34523@footnote{https://bugs.python.org/issue34523}: The Python filesystem encoding is now read earlier during the
Python initialization.

@item 
bpo-12458@footnote{https://bugs.python.org/issue12458}: Tracebacks show now correct line number for subexpressions in
multiline expressions. Tracebacks show now the line number of the first
line for multiline expressions instead of the line number of the last
subexpression.

@item 
bpo-34408@footnote{https://bugs.python.org/issue34408}: Prevent a null pointer dereference and resource leakage in
@code{PyInterpreterState_New()}.

@item 
bpo-34400@footnote{https://bugs.python.org/issue34400}: Fix undefined behavior in parsetok.c.  Patch by Zackery Spytz.

@item 
bpo-33073@footnote{https://bugs.python.org/issue33073}: Added as_integer_ratio to ints to make them more interoperable
with floats.

@item 
bpo-34377@footnote{https://bugs.python.org/issue34377}: Update valgrind suppression list to use
@code{_PyObject_Free}/@code{_PyObject_Realloc} instead of
@code{PyObject_Free}/@code{PyObject_Realloc}.

@item 
bpo-34353@footnote{https://bugs.python.org/issue34353}: Added the “socket” option in the @ref{ce3,,stat.filemode()} Python
implementation to match the C implementation.

@item 
bpo-34320@footnote{https://bugs.python.org/issue34320}: Fix @code{dict(od)} didn’t copy iteration order of OrderedDict.

@item 
bpo-34113@footnote{https://bugs.python.org/issue34113}: Fixed crash on debug builds when opcode stack was adjusted with
negative numbers. Patch by Constantin Petrisor.

@item 
bpo-34100@footnote{https://bugs.python.org/issue34100}: Compiler now merges constants in tuples and frozensets
recursively. Code attributes like @code{co_names} are merged too.

@item 
bpo-34151@footnote{https://bugs.python.org/issue34151}: Performance of list concatenation, repetition and slicing
operations is slightly improved. Patch by Sergey Fedoseev.

@item 
bpo-34170@footnote{https://bugs.python.org/issue34170}: -X dev: it is now possible to override the memory allocator
using PYTHONMALLOC even if the developer mode is enabled.

@item 
bpo-33237@footnote{https://bugs.python.org/issue33237}: Improved @ref{163,,AttributeError} message for partially
initialized module.

@item 
bpo-34149@footnote{https://bugs.python.org/issue34149}: Fix min and max functions to get default behavior when key is
None.

@item 
bpo-34125@footnote{https://bugs.python.org/issue34125}: Profiling of unbound built-in methods now works when
@code{**kwargs} is given.

@item 
bpo-34141@footnote{https://bugs.python.org/issue34141}: Optimized pickling atomic types (None, bool, int, float, bytes,
str).

@item 
bpo-34126@footnote{https://bugs.python.org/issue34126}: Fix crashes when profiling certain invalid calls of unbound
methods. Patch by Jeroen Demeyer.

@item 
bpo-24618@footnote{https://bugs.python.org/issue24618}: Fixed reading invalid memory when create the code object with
too small varnames tuple or too large argument counts.

@item 
bpo-34068@footnote{https://bugs.python.org/issue34068}: In @ref{12d8,,io.IOBase.close()}, ensure that the
@ref{12d9,,closed} attribute is not set with a live exception.
Patch by Zackery Spytz and Serhiy Storchaka.

@item 
bpo-34087@footnote{https://bugs.python.org/issue34087}: Fix buffer overflow while converting unicode to numeric values.

@item 
bpo-34080@footnote{https://bugs.python.org/issue34080}: Fixed a memory leak in the compiler when it raised some
uncommon errors during tokenizing.

@item 
bpo-34066@footnote{https://bugs.python.org/issue34066}: Disabled interruption by Ctrl-C between calling @code{open()} and
entering a @strong{with} block in @code{with open()}.

@item 
bpo-34042@footnote{https://bugs.python.org/issue34042}: Fix dict.copy() to maintain correct total refcount (as reported
by sys.gettotalrefcount()).

@item 
bpo-33418@footnote{https://bugs.python.org/issue33418}: Fix potential memory leak in function object when it creates
reference cycle.

@item 
bpo-33985@footnote{https://bugs.python.org/issue33985}: Implement contextvars.ContextVar.name attribute.

@item 
bpo-33956@footnote{https://bugs.python.org/issue33956}: Update vendored Expat library copy to version 2.2.5.

@item 
bpo-24596@footnote{https://bugs.python.org/issue24596}: Decref the module object in @ref{1181,,PyRun_SimpleFileExFlags()}
before calling @ref{11c8,,PyErr_Print()}.  Patch by Zackery Spytz.

@item 
bpo-33451@footnote{https://bugs.python.org/issue33451}: Close directly executed pyc files before calling
@code{PyEval_EvalCode()}.

@item 
bpo-1617161@footnote{https://bugs.python.org/issue1617161}: The hash of @code{BuiltinMethodType} instances (methods of
built-in classes) now depends on the hash of the identity of @emph{__self__}
instead of its value. The hash and equality of @code{ModuleType} and
@code{MethodWrapperType} instances (methods of user-defined classes and
some methods of built-in classes like @code{str.__add__}) now depend on the
hash and equality of the identity of @emph{__self__} instead of its value.
@code{MethodWrapperType} instances no longer support ordering.

@item 
bpo-33824@footnote{https://bugs.python.org/issue33824}: Fix “LC_ALL=C python3.7 -V”: reset properly the command line
parser when the encoding changes after reading the Python configuration.

@item 
bpo-33803@footnote{https://bugs.python.org/issue33803}: Fix a crash in hamt.c caused by enabling GC tracking for an
object that hadn’t all of its fields set to NULL.

@item 
bpo-33738@footnote{https://bugs.python.org/issue33738}: Seven macro incompatibilities with the Limited API were fixed,
and the macros @ref{116c,,PyIter_Check()}, @ref{496,,PyIndex_Check()} and
@code{PyExceptionClass_Name()} were added as functions. A script for
automatic macro checks was added.

@item 
bpo-33786@footnote{https://bugs.python.org/issue33786}: Fix asynchronous generators to handle GeneratorExit in athrow()
correctly

@item 
bpo-30167@footnote{https://bugs.python.org/issue30167}: @code{PyRun_SimpleFileExFlags} removes @code{__cached__} from module
in addition to @code{__file__}.

@item 
bpo-33706@footnote{https://bugs.python.org/issue33706}: Fix a crash in Python initialization when parsing the command
line options. Thanks Christoph Gohlke for the bug report and the fix!

@item 
bpo-33597@footnote{https://bugs.python.org/issue33597}: Reduce @code{PyGC_Head} size from 3 words to 2 words.

@item 
bpo-30654@footnote{https://bugs.python.org/issue30654}: Fixed reset of the SIGINT handler to SIG_DFL on interpreter
shutdown even when there was a custom handler set previously. Patch by
Philipp Kerling.

@item 
bpo-33622@footnote{https://bugs.python.org/issue33622}: Fixed a leak when the garbage collector fails to add an object
with the @code{__del__} method or referenced by it into the
@ref{d66,,gc.garbage} list. @ref{49a,,PyGC_Collect()} can now be called when an
exception is set and preserves it.

@item 
bpo-33462@footnote{https://bugs.python.org/issue33462}: Make dict and dict views reversible. Patch by Rémi Lapeyre.

@item 
bpo-23722@footnote{https://bugs.python.org/issue23722}: A @ref{5af,,RuntimeError} is now raised when the custom metaclass
doesn’t provide the @code{__classcell__} entry in the namespace passed to
@code{type.__new__}.  A @ref{1b2,,DeprecationWarning} was emitted in Python
3.6–3.7.

@item 
bpo-33499@footnote{https://bugs.python.org/issue33499}: Add 
@geindex PYTHONPYCACHEPREFIX
@geindex environment variable; PYTHONPYCACHEPREFIX
@ref{4aa,,PYTHONPYCACHEPREFIX} environment variable and
@ref{159,,-X} @code{pycache_prefix} command-line option to set an alternate
root directory for writing module bytecode cache files.

@item 
bpo-25711@footnote{https://bugs.python.org/issue25711}: The @ref{14a,,zipimport} module has been rewritten in pure Python.

@item 
bpo-33509@footnote{https://bugs.python.org/issue33509}: Fix module_globals parameter of warnings.warn_explicit(): don’t
crash if module_globals is not a dict.

@item 
bpo-31849@footnote{https://bugs.python.org/issue31849}: Fix signed/unsigned comparison warning in pyhash.c.

@item 
bpo-33475@footnote{https://bugs.python.org/issue33475}: Fixed miscellaneous bugs in converting annotations to strings
and optimized parentheses in the string representation.

@item 
bpo-20104@footnote{https://bugs.python.org/issue20104}: Added support for the @code{setpgroup}, @code{resetids}, @code{setsigmask},
@code{setsigdef} and @code{scheduler} parameters of @code{posix_spawn}. Patch by Pablo
Galindo.

@item 
bpo-33391@footnote{https://bugs.python.org/issue33391}: Fix a leak in set_symmetric_difference().

@item 
bpo-33363@footnote{https://bugs.python.org/issue33363}: Raise a SyntaxError for @code{async with} and @code{async for}
statements outside of async functions.

@item 
bpo-28055@footnote{https://bugs.python.org/issue28055}: Fix unaligned accesses in siphash24(). Patch by Rolf Eike Beer.

@item 
bpo-33128@footnote{https://bugs.python.org/issue33128}: Fix a bug that causes PathFinder to appear twice on
sys.meta_path. Patch by Pablo Galindo Salgado.

@item 
bpo-33331@footnote{https://bugs.python.org/issue33331}: Modules imported last are now cleared first at interpreter
shutdown.

@item 
bpo-33312@footnote{https://bugs.python.org/issue33312}: Fixed clang ubsan (undefined behavior sanitizer) warnings in
dictobject.c by adjusting how the internal struct _dictkeysobject shared
keys structure is declared.

@item 
bpo-33305@footnote{https://bugs.python.org/issue33305}: Improved syntax error messages for invalid numerical literals.

@item 
bpo-33306@footnote{https://bugs.python.org/issue33306}: Improved syntax error messages for unbalanced parentheses.

@item 
bpo-33234@footnote{https://bugs.python.org/issue33234}: The list constructor will pre-size and not over-allocate when
the input length is known.

@item 
bpo-33270@footnote{https://bugs.python.org/issue33270}: Intern the names for all anonymous code objects.  Patch by
Zackery Spytz.

@item 
bpo-30455@footnote{https://bugs.python.org/issue30455}: The C and Python code and the documentation related to tokens
are now generated from a single source file @code{Grammar/Tokens}.

@item 
bpo-33176@footnote{https://bugs.python.org/issue33176}: Add a @code{toreadonly()} method to memoryviews.

@item 
bpo-33231@footnote{https://bugs.python.org/issue33231}: Fix potential memory leak in @code{normalizestring()}.

@item 
bpo-33205@footnote{https://bugs.python.org/issue33205}: Change dict growth function from
@code{round_up_to_power_2(used*2+hashtable_size/2)} to
@code{round_up_to_power_2(used*3)}.  Previously, dict is shrinked only when
@code{used == 0}. Now dict has more chance to be shrinked.

@item 
bpo-29922@footnote{https://bugs.python.org/issue29922}: Improved error messages in ‘async with’ when @code{__aenter__()}
or @code{__aexit__()} return non-awaitable object.

@item 
bpo-33199@footnote{https://bugs.python.org/issue33199}: Fix @code{ma_version_tag} in dict implementation is uninitialized
when copying from key-sharing dict.

@item 
bpo-33053@footnote{https://bugs.python.org/issue33053}: When using the -m switch, sys.path[0] is now explicitly
expanded as the @emph{starting} working directory, rather than being left as
the empty path (which allows imports from the current working directory at
the time of the import)

@item 
bpo-33138@footnote{https://bugs.python.org/issue33138}: Changed standard error message for non-pickleable and
non-copyable types. It now says “cannot pickle” instead of “can’t pickle”
or “cannot serialize”.

@item 
bpo-33018@footnote{https://bugs.python.org/issue33018}: Improve consistency of errors raised by @code{issubclass()} when
called with a non-class and an abstract base class as the first and second
arguments, respectively. Patch by Josh Bronson.

@item 
bpo-33083@footnote{https://bugs.python.org/issue33083}: @code{math.factorial} no longer accepts arguments that are not
int-like. Patch by Pablo Galindo.

@item 
bpo-33041@footnote{https://bugs.python.org/issue33041}: Added new opcode @ref{5cc,,END_ASYNC_FOR} and fixes the
following issues:


@itemize *

@item 
Setting global @ref{12da,,StopAsyncIteration} no longer breaks @code{async for}
loops.

@item 
Jumping into an @code{async for} loop is now disabled.

@item 
Jumping out of an @code{async for} loop no longer corrupts the stack.
@end itemize

@item 
bpo-25750@footnote{https://bugs.python.org/issue25750}: Fix rare Python crash due to bad refcounting in
@code{type_getattro()} if a descriptor deletes itself from the class. Patch
by Jeroen Demeyer.

@item 
bpo-33041@footnote{https://bugs.python.org/issue33041}: Fixed bytecode generation for “async for” with a complex
target. A StopAsyncIteration raised on assigning or unpacking will be now
propagated instead of stopping the iteration.

@item 
bpo-33026@footnote{https://bugs.python.org/issue33026}: Fixed jumping out of “with” block by setting f_lineno.

@item 
bpo-33005@footnote{https://bugs.python.org/issue33005}: Fix a crash on fork when using a custom memory allocator (ex:
using PYTHONMALLOC env var). _PyGILState_Reinit() and
_PyInterpreterState_Enable() now use the default RAW memory allocator to
allocate a new interpreters mutex on fork.

@item 
bpo-32911@footnote{https://bugs.python.org/issue32911}: Due to unexpected compatibility issues discovered during
downstream beta testing, reverted bpo-29463@footnote{https://bugs.python.org/issue29463}. @code{docstring} field is
removed from Module, ClassDef, FunctionDef, and AsyncFunctionDef ast nodes
which was added in 3.7a1.  Docstring expression is restored as a first
statement in their body. Based on patch by Inada Naoki.

@item 
bpo-17288@footnote{https://bugs.python.org/issue17288}: Prevent jumps from ‘return’ and ‘exception’ trace events.

@item 
bpo-32946@footnote{https://bugs.python.org/issue32946}: Importing names from already imported module with “from …
import …” is now 30% faster if the module is not a package.

@item 
bpo-32932@footnote{https://bugs.python.org/issue32932}: Make error message more revealing when there are non-str
objects in @code{__all__}.

@item 
bpo-32925@footnote{https://bugs.python.org/issue32925}: Optimized iterating and containing test for literal lists
consisting of non-constants: @code{x in [a, b]} and @code{for x in [a, b]}. The
case of all constant elements already was optimized.

@item 
bpo-32889@footnote{https://bugs.python.org/issue32889}: Update Valgrind suppression list to account for the rename of
@code{Py_ADDRESS_IN_RANG} to @code{address_in_range}.

@item 
bpo-32836@footnote{https://bugs.python.org/issue32836}: Don’t use temporary variables in cases of list/dict/set
comprehensions

@item 
bpo-31356@footnote{https://bugs.python.org/issue31356}: Remove the new API added in bpo-31356@footnote{https://bugs.python.org/issue31356} (gc.ensure_disabled()
context manager).

@item 
bpo-32305@footnote{https://bugs.python.org/issue32305}: For namespace packages, ensure that both @code{__file__} and
@code{__spec__.origin} are set to None.

@item 
bpo-32303@footnote{https://bugs.python.org/issue32303}: Make sure @code{__spec__.loader} matches @code{__loader__} for
namespace packages.

@item 
bpo-32711@footnote{https://bugs.python.org/issue32711}: Fix the warning messages for Python/ast_unparse.c. Patch by
Stéphane Wirtel

@item 
bpo-32583@footnote{https://bugs.python.org/issue32583}: Fix possible crashing in builtin Unicode decoders caused by
write out-of-bound errors when using customized decode error handlers.

@item 
bpo-32489@footnote{https://bugs.python.org/issue32489}: A @ref{4c0,,continue} statement is now allowed in the
@ref{4c1,,finally} clause.

@item 
bpo-17611@footnote{https://bugs.python.org/issue17611}: Simplified the interpreter loop by moving the logic of
unrolling the stack of blocks into the compiler. The compiler emits now
explicit instructions for adjusting the stack of values and calling the
cleaning up code for @ref{5cb,,break}, @ref{4c0,,continue} and
@ref{4c6,,return}.

Removed opcodes @code{BREAK_LOOP}, @code{CONTINUE_LOOP},
@code{SETUP_LOOP} and @code{SETUP_EXCEPT}. Added new opcodes
@code{ROT_FOUR}, @code{BEGIN_FINALLY} and @code{CALL_FINALLY} and
@code{POP_FINALLY}. Changed the behavior of @code{END_FINALLY} and
@code{WITH_CLEANUP_START}.

@item 
bpo-32285@footnote{https://bugs.python.org/issue32285}: New function unicodedata.is_normalized, which can check whether
a string is in a specific normal form.

@item 
bpo-10544@footnote{https://bugs.python.org/issue10544}: Yield expressions are now disallowed in comprehensions and
generator expressions except the expression for the outermost iterable.

@item 
bpo-32117@footnote{https://bugs.python.org/issue32117}: Iterable unpacking is now allowed without parentheses in yield
and return statements, e.g. @code{yield 1, 2, 3, *rest}. Thanks to David
Cuthbert for the change and Jordan Chapman for added tests.

@item 
bpo-31902@footnote{https://bugs.python.org/issue31902}: Fix the @code{col_offset} attribute for ast nodes
@code{ast.AsyncFor}, @code{ast.AsyncFunctionDef}, and @code{ast.AsyncWith}.
Previously, @code{col_offset} pointed to the keyword after @code{async}.

@item 
bpo-25862@footnote{https://bugs.python.org/issue25862}: Fix assertion failures in the @code{tell()} method of
@code{io.TextIOWrapper}. Patch by Zackery Spytz.

@item 
bpo-21983@footnote{https://bugs.python.org/issue21983}: Fix a crash in @ref{12db,,ctypes.cast()} in case the type argument is a
ctypes structured data type. Patch by Eryk Sun and Oren Milman.

@item 
bpo-31577@footnote{https://bugs.python.org/issue31577}: Fix a crash in @ref{c8a,,os.utime()} in case of a bad ns argument. Patch
by Oren Milman.

@item 
bpo-29832@footnote{https://bugs.python.org/issue29832}: Remove references to ‘getsockaddrarg’ from various socket error
messages. Patch by Oren Milman.
@end itemize

@node Library<26>,Documentation<27>,Core and Builtins<26>,Python 3 8 0 alpha 1
@anchor{whatsnew/changelog id230}@anchor{12dc}
@subsubsection Library



@itemize -

@item 
bpo-35845@footnote{https://bugs.python.org/issue35845}: Add ‘order’ parameter to memoryview.tobytes().

@item 
bpo-35864@footnote{https://bugs.python.org/issue35864}: The _asdict() method for collections.namedtuple now returns a
regular dict instead of an OrderedDict.

@item 
bpo-35537@footnote{https://bugs.python.org/issue35537}: An ExitStack is now used internally within subprocess.Popen to
clean up pipe file handles. No behavior change in normal operation. But if
closing one handle were ever to cause an exception, the others will now be
closed instead of leaked.  (patch by Giampaolo Rodola)

@item 
bpo-35847@footnote{https://bugs.python.org/issue35847}: RISC-V needed the CTYPES_PASS_BY_REF_HACK.  Fixes ctypes
Structure test_pass_by_value.

@item 
bpo-35813@footnote{https://bugs.python.org/issue35813}: Shared memory submodule added to multiprocessing to avoid need
for serialization between processes

@item 
bpo-35780@footnote{https://bugs.python.org/issue35780}: Fix lru_cache() errors arising in recursive, reentrant, or
multi-threaded code. These errors could result in orphan links and in the
cache being trapped in a state with fewer than the specified maximum
number of links. Fix handling of negative maxsize which should have been
treated as zero. Fix errors in toggling the “full” status flag. Fix
misordering of links when errors are encountered.  Sync-up the C code and
pure Python code for the space saving path in functions with a single
positional argument. In this common case, the space overhead of an lru
cache entry is reduced by almost half.  Fix counting of cache misses. In
error cases, the miss count was out of sync with the actual number of
times the underlying user function was called.

@item 
bpo-35537@footnote{https://bugs.python.org/issue35537}: @ref{56e,,os.posix_spawn()} and @ref{12dd,,os.posix_spawnp()} now have a
@emph{setsid} parameter.

@item 
bpo-23846@footnote{https://bugs.python.org/issue23846}: @ref{4da,,asyncio.ProactorEventLoop} now catches and logs send
errors when the self-pipe is full.

@item 
bpo-34323@footnote{https://bugs.python.org/issue34323}: @ref{9,,asyncio}: Enhance @code{IocpProactor.close()} log: wait 1
second before the first log, then log every second. Log also the number of
seconds since @code{close()} was called.

@item 
bpo-35674@footnote{https://bugs.python.org/issue35674}: Add a new @ref{12dd,,os.posix_spawnp()} function. Patch by Joannah
Nanjekye.

@item 
bpo-35733@footnote{https://bugs.python.org/issue35733}: @code{ast.Constant(boolean)} no longer an instance of
@code{ast.Num}.  Patch by Anthony Sottile.

@item 
bpo-35726@footnote{https://bugs.python.org/issue35726}: QueueHandler.prepare() now makes a copy of the record before
modifying and enqueueing it, to avoid affecting other handlers in the
chain.

@item 
bpo-35719@footnote{https://bugs.python.org/issue35719}: Sped up multi-argument @ref{ad,,math} functions atan2(),
copysign(), remainder() and hypot() by 1.3–2.5 times.

@item 
bpo-35717@footnote{https://bugs.python.org/issue35717}: Fix KeyError exception raised when using enums and compile.
Patch contributed by Rémi Lapeyre.

@item 
bpo-35699@footnote{https://bugs.python.org/issue35699}: Fixed detection of Visual Studio Build Tools 2017 in distutils

@item 
bpo-32710@footnote{https://bugs.python.org/issue32710}: Fix memory leaks in asyncio ProactorEventLoop on overlapped
operation failure.

@item 
bpo-35702@footnote{https://bugs.python.org/issue35702}: The @ref{554,,time.CLOCK_UPTIME_RAW} constant is now available for
macOS 10.12.

@item 
bpo-32710@footnote{https://bugs.python.org/issue32710}: Fix a memory leak in asyncio in the ProactorEventLoop when
@code{ReadFile()} or @code{WSASend()} overlapped operation fail immediately:
release the internal buffer.

@item 
bpo-35682@footnote{https://bugs.python.org/issue35682}: Fix @code{asyncio.ProactorEventLoop.sendfile()}: don’t attempt to
set the result of an internal future if it’s already done.

@item 
bpo-35283@footnote{https://bugs.python.org/issue35283}: Add a deprecated warning for the
@code{threading.Thread.isAlive()} method. Patch by Dong-hee Na.

@item 
bpo-35664@footnote{https://bugs.python.org/issue35664}: Improve operator.itemgetter() performance by 33% with optimized
argument handling and with adding a fast path for the common case of a
single non-negative integer index into a tuple (which is the typical use
case in the standard library).

@item 
bpo-35643@footnote{https://bugs.python.org/issue35643}: Fixed a SyntaxWarning: invalid escape sequence in
Modules/_sha3/cleanup.py. Patch by Mickaël Schoentgen.

@item 
bpo-35619@footnote{https://bugs.python.org/issue35619}: Improved support of custom data descriptors in @ref{3bc,,help()} and
@ref{d4,,pydoc}.

@item 
bpo-28503@footnote{https://bugs.python.org/issue28503}: The @ref{27,,crypt} module now internally uses the @code{crypt_r()} library
function instead of @ref{27,,crypt()} when available.

@item 
bpo-35614@footnote{https://bugs.python.org/issue35614}: Fixed help() on metaclasses. Patch by Sanyam Khurana.

@item 
bpo-35568@footnote{https://bugs.python.org/issue35568}: Expose @code{raise(signum)} as @code{raise_signal}

@item 
bpo-35588@footnote{https://bugs.python.org/issue35588}: The floor division and modulo operations and the @ref{4a8,,divmod()}
function on @ref{172,,fractions.Fraction} types are 2–4x faster. Patch by
Stefan Behnel.

@item 
bpo-35585@footnote{https://bugs.python.org/issue35585}: Speed-up building enums by value, e.g. http.HTTPStatus(200).

@item 
bpo-30561@footnote{https://bugs.python.org/issue30561}: random.gammavariate(1.0, beta) now computes the same result as
random.expovariate(1.0 / beta).  This synchronizes the two algorithms and
eliminates some idiosyncrasies in the old implementation.  It does however
produce a difference stream of random variables than it used to.

@item 
bpo-35537@footnote{https://bugs.python.org/issue35537}: The @ref{f4,,subprocess} module can now use the
@ref{56e,,os.posix_spawn()} function in some cases for better performance.

@item 
bpo-35526@footnote{https://bugs.python.org/issue35526}: Delaying the ‘joke’ of barry_as_FLUFL.mandatory to Python
version 4.0

@item 
bpo-35523@footnote{https://bugs.python.org/issue35523}: Remove @ref{29,,ctypes} callback workaround: no longer create a
callback at startup. Avoid SELinux alert on @code{import ctypes} and @code{import
uuid}.

@item 
bpo-31784@footnote{https://bugs.python.org/issue31784}: @ref{6da,,uuid.uuid1()} now calls @ref{602,,time.time_ns()} rather than
@code{int(time.time() * 1e9)}.

@item 
bpo-35513@footnote{https://bugs.python.org/issue35513}: @code{TextTestRunner} of
@code{unittest.runner} now uses @ref{5a0,,time.perf_counter()} rather than
@ref{5fc,,time.time()} to measure the execution time of a test:
@ref{5fc,,time.time()} can go backwards, whereas @ref{5a0,,time.perf_counter()} is
monotonic.

@item 
bpo-35502@footnote{https://bugs.python.org/issue35502}: Fixed reference leaks in
@ref{56a,,xml.etree.ElementTree.TreeBuilder} in case of unfinished building
of the tree (in particular when an error was raised during parsing XML).

@item 
bpo-35348@footnote{https://bugs.python.org/issue35348}: Make @ref{12de,,platform.architecture()} parsing of @code{file} command
output more reliable: add the @code{-b} option to the @code{file} command to
omit the filename, force the usage of the C locale, and search also the
“shared object” pattern.

@item 
bpo-35491@footnote{https://bugs.python.org/issue35491}: @ref{b3,,multiprocessing}: Add @code{Pool.__repr__()} and enhance
@code{BaseProcess.__repr__()} (add pid and parent pid) to ease debugging.
Pool state constant values are now strings instead of integers, for
example @code{RUN} value becomes @code{'RUN'} instead of @code{0}.

@item 
bpo-35477@footnote{https://bugs.python.org/issue35477}: @code{multiprocessing.Pool.__enter__()} now fails if the pool is
not running: @code{with pool:} fails if used more than once.

@item 
bpo-31446@footnote{https://bugs.python.org/issue31446}: Copy command line that was passed to CreateProcessW since this
function can change the content of the input buffer.

@item 
bpo-35471@footnote{https://bugs.python.org/issue35471}: Python 2.4 dropped MacOS 9 support. The macpath module was
deprecated in Python 3.7. The module is now removed.

@item 
bpo-23057@footnote{https://bugs.python.org/issue23057}: Unblock Proactor event loop when keyboard interrupt is received
on Windows

@item 
bpo-35052@footnote{https://bugs.python.org/issue35052}: Fix xml.dom.minidom cloneNode() on a document with an entity:
pass the correct arguments to the user data handler of an entity.

@item 
bpo-20239@footnote{https://bugs.python.org/issue20239}: Allow repeated assignment deletion of
@ref{561,,unittest.mock.Mock} attributes. Patch by Pablo Galindo.

@item 
bpo-17185@footnote{https://bugs.python.org/issue17185}: Set @code{__signature__} on mock for @ref{9c,,inspect} to get
signature. Patch by Karthikeyan Singaravelan.

@item 
bpo-35445@footnote{https://bugs.python.org/issue35445}: Memory errors during creating posix.environ no longer ignored.

@item 
bpo-35415@footnote{https://bugs.python.org/issue35415}: Validate fileno= argument to socket.socket().

@item 
bpo-35424@footnote{https://bugs.python.org/issue35424}: @code{multiprocessing.Pool} destructor now emits
@ref{77b,,ResourceWarning} if the pool is still running.

@item 
bpo-35330@footnote{https://bugs.python.org/issue35330}: When a @code{Mock} instance was used to wrap an object, if
@code{side_effect} is used in one of the mocks of it methods, don’t call the
original implementation and return the result of using the side effect the
same way that it is done with return_value.

@item 
bpo-35346@footnote{https://bugs.python.org/issue35346}: Drop Mac OS 9 and Rhapsody support from the @ref{c9,,platform}
module. Rhapsody last release was in 2000. Mac OS 9 last release was in
2001.

@item 
bpo-10496@footnote{https://bugs.python.org/issue10496}: @code{check_environ()} of
@code{distutils.utils} now catches @ref{5bb,,KeyError} on calling
@ref{12df,,pwd.getpwuid()}: don’t create the @code{HOME} environment variable in
this case.

@item 
bpo-10496@footnote{https://bugs.python.org/issue10496}: @code{posixpath.expanduser()} now returns the input @emph{path}
unchanged if the @code{HOME} environment variable is not set and the current
user has no home directory (if the current user identifier doesn’t exist
in the password database). This change fix the @ref{e6,,site} module if the
current user doesn’t exist in the password database (if the user has no
home directory).

@item 
bpo-35389@footnote{https://bugs.python.org/issue35389}: @ref{12e0,,platform.libc_ver()} now uses
@code{os.confstr('CS_GNU_LIBC_VERSION')} if available and the @emph{executable}
parameter is not set.

@item 
bpo-35394@footnote{https://bugs.python.org/issue35394}: Add empty slots to asyncio abstract protocols.

@item 
bpo-35310@footnote{https://bugs.python.org/issue35310}: Fix a bug in @ref{8fc,,select.select()} where, in some cases, the
file descriptor sequences were returned unmodified after a signal
interruption, even though the file descriptors might not be ready yet.
@ref{8fc,,select.select()} will now always return empty lists if a timeout has
occurred.  Patch by Oran Avraham.

@item 
bpo-35380@footnote{https://bugs.python.org/issue35380}: Enable TCP_NODELAY on Windows for proactor asyncio event loop.

@item 
bpo-35341@footnote{https://bugs.python.org/issue35341}: Add generic version of @code{collections.OrderedDict} to the
@code{typing} module. Patch by Ismo Toijala.

@item 
bpo-35371@footnote{https://bugs.python.org/issue35371}: Fixed possible crash in @code{os.utime()} on Windows when pass
incorrect arguments.

@item 
bpo-35346@footnote{https://bugs.python.org/issue35346}: @ref{12e1,,platform.uname()} now redirects @code{stderr} to
@ref{12e2,,os.devnull} when running external programs like @code{cmd /c ver}.

@item 
bpo-35066@footnote{https://bugs.python.org/issue35066}: Previously, calling the strftime() method on a datetime object
with a trailing ‘%’ in the format string would result in an exception.
However, this only occurred when the datetime C module was being used; the
python implementation did not match this behavior. Datetime is now PEP-399
compliant, and will not throw an exception on a trailing ‘%’.

@item 
bpo-35345@footnote{https://bugs.python.org/issue35345}: The function @code{platform.popen} has been removed, it was
deprecated since Python 3.3: use @ref{59f,,os.popen()} instead.

@item 
bpo-35344@footnote{https://bugs.python.org/issue35344}: On macOS, @ref{12e3,,platform.platform()} now uses
@ref{12e4,,platform.mac_ver()}, if it returns a non-empty release string, to get
the macOS version rather than the darwin version.

@item 
bpo-35312@footnote{https://bugs.python.org/issue35312}: Make @code{lib2to3.pgen2.parse.ParseError} round-trip pickle-able.
Patch by Anthony Sottile.

@item 
bpo-35308@footnote{https://bugs.python.org/issue35308}: Fix regression in @code{webbrowser} where default browsers may be
preferred over browsers in the @code{BROWSER} environment variable.

@item 
bpo-24746@footnote{https://bugs.python.org/issue24746}: Avoid stripping trailing whitespace in doctest fancy diff.
Original patch by R. David Murray & Jairo Trad. Enhanced by Sanyam
Khurana.

@item 
bpo-28604@footnote{https://bugs.python.org/issue28604}: @ref{739,,locale.localeconv()} now sets temporarily the @code{LC_CTYPE}
locale to the @code{LC_MONETARY} locale if the two locales are different and
monetary strings are non-ASCII. This temporary change affects other
threads.

@item 
bpo-35277@footnote{https://bugs.python.org/issue35277}: Update ensurepip to install pip 18.1 and setuptools 40.6.2.

@item 
bpo-24209@footnote{https://bugs.python.org/issue24209}: Adds IPv6 support when invoking http.server directly.

@item 
bpo-35226@footnote{https://bugs.python.org/issue35226}: Recursively check arguments when testing for equality of
@ref{1265,,unittest.mock.call} objects and add note that tracking of
parameters used to create ancestors of mocks in @code{mock_calls} is not
possible.

@item 
bpo-29564@footnote{https://bugs.python.org/issue29564}: The warnings module now suggests to enable tracemalloc if the
source is specified, the tracemalloc module is available, but tracemalloc
is not tracing memory allocations.

@item 
bpo-35189@footnote{https://bugs.python.org/issue35189}: Modify the following fnctl function to retry if interrupted by
a signal (EINTR): flock, lockf, fnctl

@item 
bpo-30064@footnote{https://bugs.python.org/issue30064}: Use add_done_callback() in sock_* asyncio API to unsubscribe
reader/writer early on calcellation.

@item 
bpo-35186@footnote{https://bugs.python.org/issue35186}: Removed the “built with” comment added when @code{setup.py upload}
is used with either @code{bdist_rpm} or @code{bdist_dumb}.

@item 
bpo-35152@footnote{https://bugs.python.org/issue35152}: Allow sending more than 2 GB at once on a multiprocessing
connection on non-Windows systems.

@item 
bpo-35062@footnote{https://bugs.python.org/issue35062}: Fix incorrect parsing of
@code{_io.IncrementalNewlineDecoder}’s @emph{translate} argument.

@item 
bpo-35065@footnote{https://bugs.python.org/issue35065}: Remove @code{StreamReaderProtocol._untrack_reader}. The call to
@code{_untrack_reader} is currently performed too soon, causing the protocol to
forget about the reader before @code{connection_lost} can run and feed the EOF
to the reader.

@item 
bpo-34160@footnote{https://bugs.python.org/issue34160}: ElementTree and minidom now preserve the attribute order
specified by the user.

@item 
bpo-35079@footnote{https://bugs.python.org/issue35079}: Improve difflib.SequenceManager.get_matching_blocks doc by
adding ‘non-overlapping’ and changing ‘!=’ to ‘<’.

@item 
bpo-33710@footnote{https://bugs.python.org/issue33710}: Deprecated @code{l*gettext()} functions and methods in the
@ref{83,,gettext} module. They return encoded bytes instead of Unicode
strings and are artifacts from Python 2 times. Also deprecated functions
and methods related to setting the charset for @code{l*gettext()} functions
and methods.

@item 
bpo-35017@footnote{https://bugs.python.org/issue35017}: @ref{cd4,,socketserver.BaseServer.serve_forever()} now exits
immediately if it’s @ref{12e5,,shutdown()} method is
called while it is polling for new events.

@item 
bpo-35024@footnote{https://bugs.python.org/issue35024}: @ref{96,,importlib} no longer logs @code{wrote} redundantly
after @code{(created|could not create)} is already logged.
Patch by Quentin Agren.

@item 
bpo-35047@footnote{https://bugs.python.org/issue35047}: @code{unittest.mock} now includes mock calls in exception messages
if @code{assert_not_called}, @code{assert_called_once}, or
@code{assert_called_once_with} fails. Patch by Petter Strandmark.

@item 
bpo-31047@footnote{https://bugs.python.org/issue31047}: Fix @code{ntpath.abspath} regression where it didn’t remove a
trailing separator on Windows. Patch by Tim Graham.

@item 
bpo-35053@footnote{https://bugs.python.org/issue35053}: tracemalloc now tries to update the traceback when an object is
reused from a “free list” (optimization for faster object creation, used
by the builtin list type for example).

@item 
bpo-31553@footnote{https://bugs.python.org/issue31553}: Add the –json-lines option to json.tool. Patch by hongweipeng.

@item 
bpo-34794@footnote{https://bugs.python.org/issue34794}: Fixed a leak in Tkinter when pass the Python wrapper around
Tcl_Obj back to Tcl/Tk.

@item 
bpo-34909@footnote{https://bugs.python.org/issue34909}: Enum: fix grandchildren subclassing when parent mixed with
concrete data types.

@item 
bpo-35022@footnote{https://bugs.python.org/issue35022}: @ref{a01,,unittest.mock.MagicMock} now supports the
@code{__fspath__} method (from @ref{795,,os.PathLike}).

@item 
bpo-35008@footnote{https://bugs.python.org/issue35008}: Fixed references leaks when call the @code{__setstate__()} method
of @ref{44e,,xml.etree.ElementTree.Element} in the C implementation for
already initialized element.

@item 
bpo-23420@footnote{https://bugs.python.org/issue23420}: Verify the value for the parameter ‘-s’ of the cProfile CLI.
Patch by Robert Kuska

@item 
bpo-33947@footnote{https://bugs.python.org/issue33947}: dataclasses now handle recursive reprs without raising
RecursionError.

@item 
bpo-34890@footnote{https://bugs.python.org/issue34890}: Make @ref{979,,inspect.iscoroutinefunction()},
@ref{12e6,,inspect.isgeneratorfunction()} and @ref{12e7,,inspect.isasyncgenfunction()}
work with @ref{a3f,,functools.partial()}. Patch by Pablo Galindo.

@item 
bpo-34521@footnote{https://bugs.python.org/issue34521}: Use @ref{12e8,,socket.CMSG_SPACE()} to calculate ancillary data size
instead of @ref{12e9,,socket.CMSG_LEN()} in
@code{multiprocessing.reduction.recvfds()} as 
@geindex RFC; RFC 3542
RFC 3542@footnote{https://tools.ietf.org/html/rfc3542.html} requires the use
of the former for portable applications.

@item 
bpo-31522@footnote{https://bugs.python.org/issue31522}: The @code{mailbox.mbox.get_string} function @emph{from_} parameter can
now successfully be set to a non-default value.

@item 
bpo-34970@footnote{https://bugs.python.org/issue34970}: Protect tasks weak set manipulation in @code{asyncio.all_tasks()}

@item 
bpo-34969@footnote{https://bugs.python.org/issue34969}: gzip: Add –fast, –best on the gzip CLI, these parameters will
be used for the fast compression method (quick) or the best method
compress (slower, but smaller file). Also, change the default compression
level to 6 (tradeoff).

@item 
bpo-16965@footnote{https://bugs.python.org/issue16965}: The @ref{12ea,,2to3} @ref{12eb,,execfile} fixer now opens the file
with mode @code{'rb'}.  Patch by Zackery Spytz.

@item 
bpo-34966@footnote{https://bugs.python.org/issue34966}: @ref{d4,,pydoc} now supports aliases not only to methods defined
in the end class, but also to inherited methods.  The docstring is not
duplicated for aliases.

@item 
bpo-34926@footnote{https://bugs.python.org/issue34926}: @ref{12ec,,mimetypes.MimeTypes.guess_type()} now accepts
@ref{63d,,path-like object} in addition to url strings. Patch by Mayank
Asthana.

@item 
bpo-23831@footnote{https://bugs.python.org/issue23831}: Add @code{moveto()} method to the @code{tkinter.Canvas} widget. Patch
by Juliette Monsel.

@item 
bpo-34941@footnote{https://bugs.python.org/issue34941}: Methods @code{find()}, @code{findtext()} and @code{findall()} of the
@code{Element} class in the @ref{13e,,xml.etree.ElementTree} module are now able
to find children which are instances of @code{Element} subclasses.

@item 
bpo-32680@footnote{https://bugs.python.org/issue32680}: @ref{421,,smtplib.SMTP} objects now always have a @code{sock}
attribute present

@item 
bpo-34769@footnote{https://bugs.python.org/issue34769}: Fix for async generators not finalizing when event loop is in
debug mode and garbage collector runs in another thread.

@item 
bpo-34936@footnote{https://bugs.python.org/issue34936}: Fix @code{TclError} in @code{tkinter.Spinbox.selection_element()}.
Patch by Juliette Monsel.

@item 
bpo-34829@footnote{https://bugs.python.org/issue34829}: Add methods @code{selection_from}, @code{selection_range},
@code{selection_present} and @code{selection_to} to the @code{tkinter.Spinbox} for
consistency with the @code{tkinter.Entry} widget. Patch by Juliette Monsel.

@item 
bpo-34911@footnote{https://bugs.python.org/issue34911}: Added @emph{secure_protocols} argument to
@emph{http.cookiejar.DefaultCookiePolicy} to allow for tweaking of protocols
and also to add support by default for @emph{wss}, the secure websocket
protocol.

@item 
bpo-34922@footnote{https://bugs.python.org/issue34922}: Fixed integer overflow in the @ref{12ed,,digest()}
and @ref{12ee,,hexdigest()} methods for the SHAKE algorithm in
the @ref{88,,hashlib} module.

@item 
bpo-34925@footnote{https://bugs.python.org/issue34925}: 25% speedup in argument parsing for the functions in the bisect
module.

@item 
bpo-34900@footnote{https://bugs.python.org/issue34900}: Fixed @ref{10f9,,unittest.TestCase.debug()} when used to call test
methods with subtests.  Patch by Bruno Oliveira.

@item 
bpo-34844@footnote{https://bugs.python.org/issue34844}: logging.Formatter enhancement - Ensure styles and fmt matches
in logging.Formatter - Added validate method in each format style class:
StrFormatStyle, PercentStyle, StringTemplateStyle. - This method is called
in the constructor of logging.Formatter class - Also re-raise the KeyError
in the format method of each style class, so it would a bit clear that
it’s an error with the invalid format fields.

@item 
bpo-34897@footnote{https://bugs.python.org/issue34897}: Adjust test.support.missing_compiler_executable check so that a
nominal command name of “” is ignored. Patch by Michael Felt.

@item 
bpo-34871@footnote{https://bugs.python.org/issue34871}: Fix inspect module polluted @code{sys.modules} when parsing
@code{__text_signature__} of callable.

@item 
bpo-34898@footnote{https://bugs.python.org/issue34898}: Add @code{mtime} argument to @ref{4fc,,gzip.compress} for reproducible
output. Patch by Guo Ci Teo.

@item 
bpo-28441@footnote{https://bugs.python.org/issue28441}: On Cygwin and MinGW, ensure that @code{sys.executable} always
includes the full filename in the path, including the @code{.exe} suffix
(unless it is a symbolic link).

@item 
bpo-34866@footnote{https://bugs.python.org/issue34866}: Adding @code{max_num_fields} to @code{cgi.FieldStorage} to make DOS
attacks harder by limiting the number of @code{MiniFieldStorage} objects
created by @code{FieldStorage}.

@item 
bpo-34711@footnote{https://bugs.python.org/issue34711}: http.server ensures it reports HTTPStatus.NOT_FOUND when the
local path ends with “/” and is not a directory, even if the underlying OS
(e.g. AIX) accepts such paths as a valid file reference. Patch by Michael
Felt.

@item 
bpo-34872@footnote{https://bugs.python.org/issue34872}: Fix self-cancellation in C implementation of asyncio.Task

@item 
bpo-34849@footnote{https://bugs.python.org/issue34849}: Don’t log waiting for @code{selector.select} in asyncio loop
iteration. The waiting is pretty normal for any asyncio program, logging
its time just adds a noise to logs without any useful information
provided.

@item 
bpo-34022@footnote{https://bugs.python.org/issue34022}: The 
@geindex SOURCE_DATE_EPOCH
@geindex environment variable; SOURCE_DATE_EPOCH
@code{SOURCE_DATE_EPOCH} environment variable no longer
overrides the value of the @emph{invalidation_mode} argument to
@ref{534,,py_compile.compile()}, and determines its default value instead.

@item 
bpo-34819@footnote{https://bugs.python.org/issue34819}: Use a monotonic clock to compute timeouts in
@code{Executor.map()} and @code{as_completed()}, in order to prevent
timeouts from deviating when the system clock is adjusted.

@item 
bpo-34758@footnote{https://bugs.python.org/issue34758}: Add .wasm -> application/wasm to list of recognized file types
and content type headers

@item 
bpo-34789@footnote{https://bugs.python.org/issue34789}: @ref{12ef,,xml.sax.make_parser()} now accepts any iterable as its
@emph{parser_list} argument.  Patch by Andrés Delfino.

@item 
bpo-34334@footnote{https://bugs.python.org/issue34334}: In @code{QueueHandler}, clear @code{exc_text} from
@code{LogRecord} to prevent traceback from being written twice.

@item 
bpo-34687@footnote{https://bugs.python.org/issue34687}: On Windows, asyncio now uses ProactorEventLoop, instead of
SelectorEventLoop, by default.

@item 
bpo-5950@footnote{https://bugs.python.org/issue5950}: Support reading zip files with archive comments in
@ref{14a,,zipimport}.

@item 
bpo-32892@footnote{https://bugs.python.org/issue32892}: The parser now represents all constants as
@ref{588,,ast.Constant} instead of using specific constant AST types
(@code{Num}, @code{Str}, @code{Bytes}, @code{NameConstant} and @code{Ellipsis}). These
classes are considered deprecated and will be removed in future Python
versions.

@item 
bpo-34728@footnote{https://bugs.python.org/issue34728}: Add deprecation warning when @code{loop} is used in methods:
@ref{58a,,asyncio.sleep}, @ref{440,,asyncio.wait} and @ref{3d5,,asyncio.wait_for}.

@item 
bpo-34738@footnote{https://bugs.python.org/issue34738}: ZIP files created by @ref{37,,distutils} will now include entries
for directories.

@item 
bpo-34659@footnote{https://bugs.python.org/issue34659}: Add an optional @emph{initial} argument to itertools.accumulate().

@item 
bpo-29577@footnote{https://bugs.python.org/issue29577}: Support multiple mixin classes when creating Enums.

@item 
bpo-34670@footnote{https://bugs.python.org/issue34670}: Add SSLContext.post_handshake_auth and
SSLSocket.verify_client_post_handshake for TLS 1.3’s post handshake
authentication feature.

@item 
bpo-32718@footnote{https://bugs.python.org/issue32718}: The Activate.ps1 script from venv works with PowerShell Core
6.1 and is now available under all operating systems.

@item 
bpo-31177@footnote{https://bugs.python.org/issue31177}: Fix bug that prevented using @ref{845,,reset_mock} on mock instances with deleted attributes

@item 
bpo-34672@footnote{https://bugs.python.org/issue34672}: Add a workaround, so the @code{'Z'} @ref{d8b,,time.strftime()}
specifier on the musl C library can work in some cases.

@item 
bpo-34666@footnote{https://bugs.python.org/issue34666}: Implement @code{asyncio.StreamWriter.awrite} and
@code{asyncio.StreamWriter.aclose()} coroutines.  Methods are needed for
providing a consistent stream API with control flow switched on by
default.

@item 
bpo-6721@footnote{https://bugs.python.org/issue6721}: Acquire the logging module’s commonly used internal locks while
fork()ing to avoid deadlocks in the child process.

@item 
bpo-34658@footnote{https://bugs.python.org/issue34658}: Fix a rare interpreter unhandled exception state SystemError
only seen when using subprocess with a preexec_fn while an after_parent
handler has been registered with os.register_at_fork and the fork system
call fails.

@item 
bpo-34652@footnote{https://bugs.python.org/issue34652}: Ensure @ref{12f0,,os.lchmod()} is never defined on Linux.

@item 
bpo-34638@footnote{https://bugs.python.org/issue34638}: Store a weak reference to stream reader to break strong
references loop between reader and protocol.  It allows to detect and
close the socket if the stream is deleted (garbage collected) without
@code{close()} call.

@item 
bpo-34536@footnote{https://bugs.python.org/issue34536}: @code{Enum._missing_}:  raise @ref{17c,,ValueError} if None returned and
@ref{162,,TypeError} if non-member is returned.

@item 
bpo-34636@footnote{https://bugs.python.org/issue34636}: Speed up re scanning of many non-matching characters for s w
and d within bytes objects. (microoptimization)

@item 
bpo-24412@footnote{https://bugs.python.org/issue24412}: Add @ref{562,,addModuleCleanup()} and
@ref{563,,addClassCleanup()} to unittest to support
cleanups for @code{setUpModule()} and
@ref{564,,setUpClass()}. Patch by Lisa Roach.

@item 
bpo-34630@footnote{https://bugs.python.org/issue34630}: Don’t log SSL certificate errors in asyncio code (connection
error logging is skipped already).

@item 
bpo-32490@footnote{https://bugs.python.org/issue32490}: Prevent filename duplication in @ref{f4,,subprocess} exception
messages.  Patch by Zackery Spytz.

@item 
bpo-34363@footnote{https://bugs.python.org/issue34363}: dataclasses.asdict() and .astuple() now handle namedtuples
correctly.

@item 
bpo-34625@footnote{https://bugs.python.org/issue34625}: Update vendorized expat library version to 2.2.6.

@item 
bpo-32270@footnote{https://bugs.python.org/issue32270}: The subprocess module no longer mistakenly closes redirected
fds even when they were in pass_fds when outside of the default @{0, 1, 2@}
set.

@item 
bpo-34622@footnote{https://bugs.python.org/issue34622}: Create a dedicated @code{asyncio.CancelledError},
@code{asyncio.InvalidStateError} and @code{asyncio.TimeoutError} exception
classes.  Inherit them from corresponding exceptions from
@code{concurrent.futures} package. Extract @code{asyncio} exceptions into a
separate file.

@item 
bpo-34610@footnote{https://bugs.python.org/issue34610}: Fixed iterator of @code{multiprocessing.managers.DictProxy}.

@item 
bpo-34421@footnote{https://bugs.python.org/issue34421}: Fix distutils logging for non-ASCII strings.  This caused
installation issues on Windows.

@item 
bpo-34604@footnote{https://bugs.python.org/issue34604}: Fix possible mojibake in the error message of @ref{12f1,,pwd.getpwnam}
and @ref{12f2,,grp.getgrnam} using string representation because of invisible
characters or trailing whitespaces. Patch by William Grzybowski.

@item 
bpo-30977@footnote{https://bugs.python.org/issue30977}: Make uuid.UUID use @code{__slots__} to reduce its memory
footprint. Based on original patch by Wouter Bolsterlee.

@item 
bpo-34574@footnote{https://bugs.python.org/issue34574}: OrderedDict iterators are not exhausted during pickling
anymore. Patch by Sergey Fedoseev.

@item 
bpo-8110@footnote{https://bugs.python.org/issue8110}: Refactored @ref{f4,,subprocess} to check for Windows-specific
modules rather than @code{sys.platform == 'win32'}.

@item 
bpo-34530@footnote{https://bugs.python.org/issue34530}: @code{distutils.spawn.find_executable()} now falls back on
@ref{12ad,,os.defpath} if the @code{PATH} environment variable is not set.

@item 
bpo-34563@footnote{https://bugs.python.org/issue34563}: On Windows, fix multiprocessing.Connection for very large read:
fix _winapi.PeekNamedPipe() and _winapi.ReadFile() for read larger than
INT_MAX (usually @code{2**31-1}).

@item 
bpo-34558@footnote{https://bugs.python.org/issue34558}: Correct typo in Lib/ctypes/_aix.py

@item 
bpo-34282@footnote{https://bugs.python.org/issue34282}: Move @code{Enum._convert} to @code{EnumMeta._convert_} and fix enum
members getting shadowed by parent attributes.

@item 
bpo-22872@footnote{https://bugs.python.org/issue22872}: When the queue is closed, @ref{17c,,ValueError} is now raised by
@ref{12f3,,multiprocessing.Queue.put()} and @ref{12f4,,multiprocessing.Queue.get()}
instead of @ref{46e,,AssertionError} and @ref{2a1,,OSError}, respectively. Patch
by Zackery Spytz.

@item 
bpo-34515@footnote{https://bugs.python.org/issue34515}: Fix parsing non-ASCII identifiers in
@code{lib2to3.pgen2.tokenize} (PEP 3131).

@item 
bpo-13312@footnote{https://bugs.python.org/issue13312}: Avoids a possible integer underflow (undefined behavior) in the
time module’s year handling code when passed a very low negative year
value.

@item 
bpo-34472@footnote{https://bugs.python.org/issue34472}: Improved compatibility for streamed files in @ref{149,,zipfile}.
Previously an optional signature was not being written and certain ZIP
applications were not supported. Patch by Silas Sewell.

@item 
bpo-34454@footnote{https://bugs.python.org/issue34454}: Fix the .fromisoformat() methods of datetime types crashing
when given unicode with non-UTF-8-encodable code points.  Specifically,
datetime.fromisoformat() now accepts surrogate unicode code points used as
the separator. Report and tests by Alexey Izbyshev, patch by Paul Ganssle.

@item 
bpo-6700@footnote{https://bugs.python.org/issue6700}: Fix inspect.getsourcelines for module level frames/tracebacks.
Patch by Vladimir Matveev.

@item 
bpo-34171@footnote{https://bugs.python.org/issue34171}: Running the @ref{119,,trace} module no longer creates the
@code{trace.cover} file.

@item 
bpo-34441@footnote{https://bugs.python.org/issue34441}: Fix crash when an @code{ABC}-derived class with invalid
@code{__subclasses__} is passed as the second argument to
@ref{237,,issubclass()}. Patch by Alexey Izbyshev.

@item 
bpo-34427@footnote{https://bugs.python.org/issue34427}: Fix infinite loop in @code{a.extend(a)} for @code{MutableSequence}
subclasses.

@item 
bpo-34412@footnote{https://bugs.python.org/issue34412}: Make @ref{12f5,,signal.strsignal()} work on HP-UX. Patch by Michael
Osipov.

@item 
bpo-20849@footnote{https://bugs.python.org/issue20849}: shutil.copytree now accepts a new @code{dirs_exist_ok} keyword
argument. Patch by Josh Bronson.

@item 
bpo-31715@footnote{https://bugs.python.org/issue31715}: Associate @code{.mjs} file extension with
@code{application/javascript} MIME Type.

@item 
bpo-34384@footnote{https://bugs.python.org/issue34384}: @ref{414,,os.readlink()} now accepts @ref{63d,,path-like} and @ref{2f0,,bytes} objects on Windows.

@item 
bpo-22602@footnote{https://bugs.python.org/issue22602}: The UTF-7 decoder now raises @ref{51f,,UnicodeDecodeError} for
ill-formed sequences starting with “+” (as specified in RFC 2152).  Patch
by Zackery Spytz.

@item 
bpo-2122@footnote{https://bugs.python.org/issue2122}: The @ref{5b7,,mmap.flush()} method now returns
@code{None} on success, raises an exception on error under all platforms.

@item 
bpo-34341@footnote{https://bugs.python.org/issue34341}: Appending to the ZIP archive with the ZIP64 extension no longer
grows the size of extra fields of existing entries.

@item 
bpo-34333@footnote{https://bugs.python.org/issue34333}: Fix %-formatting in @ref{12f6,,pathlib.PurePath.with_suffix()} when
formatting an error message.

@item 
bpo-18540@footnote{https://bugs.python.org/issue18540}: The @ref{3f2,,imaplib.IMAP4} and @ref{3f3,,imaplib.IMAP4_SSL}
classes now resolve to the local host IP correctly when the default value
of @emph{host} parameter (@code{'@w{'}}) is used.

@item 
bpo-26502@footnote{https://bugs.python.org/issue26502}: Implement @code{traceback.FrameSummary.__len__()} method to
preserve compatibility with the old tuple API.

@item 
bpo-34318@footnote{https://bugs.python.org/issue34318}: @ref{cff,,assertRaises()},
@ref{d00,,assertRaisesRegex()},
@ref{d01,,assertWarns()} and
@ref{d02,,assertWarnsRegex()} no longer success if the
passed callable is None. They no longer ignore unknown keyword arguments
in the context manager mode. A DeprecationWarning was raised in these
cases since Python 3.5.

@item 
bpo-9372@footnote{https://bugs.python.org/issue9372}: Deprecate @code{__getitem__()} methods of
@ref{1ba,,xml.dom.pulldom.DOMEventStream}, @ref{1bb,,wsgiref.util.FileWrapper}
and @ref{1bc,,fileinput.FileInput}.

@item 
bpo-33613@footnote{https://bugs.python.org/issue33613}: Fix a race condition in @code{multiprocessing.semaphore_tracker}
when the tracker receives SIGINT before it can register signal handlers
for ignoring it.

@item 
bpo-34248@footnote{https://bugs.python.org/issue34248}: Report filename in the exception raised when the database file
cannot be opened by @ref{12f7,,dbm.gnu.open()} and @ref{12f8,,dbm.ndbm.open()} due to
OS-related error. Patch by Zsolt Cserna.

@item 
bpo-33089@footnote{https://bugs.python.org/issue33089}: Add math.dist() to compute the Euclidean distance between two
points.

@item 
bpo-34246@footnote{https://bugs.python.org/issue34246}: @ref{9c4,,smtplib.SMTP.send_message()} no longer modifies the
content of the @emph{mail_options} argument. Patch by Pablo S. Blum de Aguiar.

@item 
bpo-31047@footnote{https://bugs.python.org/issue31047}: Fix @code{ntpath.abspath} for invalid paths on windows. Patch by
Franz Woellert.

@item 
bpo-32321@footnote{https://bugs.python.org/issue32321}: Add pure Python fallback for functools.reduce. Patch by Robert
Wright.

@item 
bpo-34270@footnote{https://bugs.python.org/issue34270}: The default asyncio task class now always has a name which can
be get or set using two new methods (@ref{4df,,get_name()} and
@ref{4de,,set_name()}) and is visible in the @ref{27d,,repr()} output.
An initial name can also be set using the new @code{name} keyword argument to
@ref{4dc,,asyncio.create_task()} or the
@code{create_task()} method of the event loop.
If no initial name is set, the default Task implementation generates a
name like @code{Task-1} using a monotonic counter.

@item 
bpo-34263@footnote{https://bugs.python.org/issue34263}: asyncio’s event loop will not pass timeouts longer than one day
to epoll/select etc.

@item 
bpo-34035@footnote{https://bugs.python.org/issue34035}: Fix several AttributeError in zipfile seek() methods. Patch by
Mickaël Schoentgen.

@item 
bpo-32215@footnote{https://bugs.python.org/issue32215}: Fix performance regression in @ref{ed,,sqlite3} when a DML
statement appeared in a different line than the rest of the SQL query.

@item 
bpo-34075@footnote{https://bugs.python.org/issue34075}: Deprecate passing non-ThreadPoolExecutor instances to
@code{AbstractEventLoop.set_default_executor()}.

@item 
bpo-34251@footnote{https://bugs.python.org/issue34251}: Restore @code{msilib.Win64} to preserve backwards compatibility
since it’s already used by @ref{37,,distutils}’ @code{bdist_msi} command.

@item 
bpo-19891@footnote{https://bugs.python.org/issue19891}: Ignore errors caused by missing / non-writable homedir while
writing history during exit of an interactive session.  Patch by Anthony
Sottile.

@item 
bpo-33089@footnote{https://bugs.python.org/issue33089}: Enhanced math.hypot() to support more than two dimensions.

@item 
bpo-34228@footnote{https://bugs.python.org/issue34228}: tracemalloc: PYTHONTRACEMALLOC=0 environment variable and -X
tracemalloc=0 command line option are now allowed to disable explicitly
tracemalloc at startup.

@item 
bpo-13041@footnote{https://bugs.python.org/issue13041}: Use @ref{c96,,shutil.get_terminal_size()} to calculate the terminal
width correctly in the @code{argparse.HelpFormatter} class.  Initial patch by
Zbyszek Jędrzejewski-Szmek.

@item 
bpo-34213@footnote{https://bugs.python.org/issue34213}: Allow frozen dataclasses to have a field named “object”.
Previously this conflicted with an internal use of “object”.

@item 
bpo-34052@footnote{https://bugs.python.org/issue34052}: @ref{126c,,sqlite3.Connection.create_aggregate()},
@ref{126d,,sqlite3.Connection.create_function()},
@ref{183,,sqlite3.Connection.set_authorizer()},
@ref{126e,,sqlite3.Connection.set_progress_handler()} methods raises TypeError
when unhashable objects are passed as callable. These methods now don’t
pass such objects to SQLite API. Previous behavior could lead to
segfaults. Patch by Sergey Fedoseev.

@item 
bpo-34197@footnote{https://bugs.python.org/issue34197}: Attributes @emph{skipinitialspace}, @emph{doublequote} and @emph{strict} of
the @emph{dialect} attribute of the @ref{28,,csv} reader are now @ref{2f7,,bool}
instances instead of integers 0 or 1.

@item 
bpo-32788@footnote{https://bugs.python.org/issue32788}: Errors other than @ref{162,,TypeError} raised in methods
@code{__adapt__()} and @code{__conform__()} in the @ref{ed,,sqlite3} module are now
propagated to the user.

@item 
bpo-21446@footnote{https://bugs.python.org/issue21446}: The @ref{12f9,,reload} fixer now uses @ref{66a,,importlib.reload()}
instead of deprecated @ref{e78,,imp.reload()}.

@item 
bpo-940286@footnote{https://bugs.python.org/issue940286}: pydoc’s @code{Helper.showtopic()} method now prints the cross
references of a topic correctly.

@item 
bpo-34164@footnote{https://bugs.python.org/issue34164}: @ref{bb6,,base64.b32decode()} could raise UnboundLocalError or
OverflowError for incorrect padding.  Now it always raises
@code{base64.Error} in these cases.

@item 
bpo-33729@footnote{https://bugs.python.org/issue33729}: Fixed issues with arguments parsing in @ref{88,,hashlib}.

@item 
bpo-34097@footnote{https://bugs.python.org/issue34097}: ZipFile can zip files older than 1980-01-01 and newer than
2107-12-31 using a new @code{strict_timestamps} parameter at the cost of
setting the timestamp to the limit.

@item 
bpo-34108@footnote{https://bugs.python.org/issue34108}: Remove extraneous CR in 2to3 refactor.

@item 
bpo-34070@footnote{https://bugs.python.org/issue34070}: Make sure to only check if the handle is a tty, when opening a
file with @code{buffering=-1}.

@item 
bpo-27494@footnote{https://bugs.python.org/issue27494}: Reverted bpo-27494@footnote{https://bugs.python.org/issue27494}. 2to3 rejects now a trailing comma in
generator expressions.

@item 
bpo-33967@footnote{https://bugs.python.org/issue33967}: functools.singledispatch now raises TypeError instead of
IndexError when no positional arguments are passed.

@item 
bpo-34041@footnote{https://bugs.python.org/issue34041}: Add the parameter @emph{deterministic} to the
@ref{126d,,sqlite3.Connection.create_function()} method. Patch by Sergey
Fedoseev.

@item 
bpo-34056@footnote{https://bugs.python.org/issue34056}: Ensure the loader shim created by @code{imp.load_module} always
returns bytes from its @code{get_data()} function. This fixes using
@code{imp.load_module} with 
@geindex Python Enhancement Proposals; PEP 552
PEP 552@footnote{https://www.python.org/dev/peps/pep-0552} hash-based pycs.

@item 
bpo-34054@footnote{https://bugs.python.org/issue34054}: The multiprocessing module now uses the monotonic clock
@ref{9ed,,time.monotonic()} instead of the system clock @ref{5fc,,time.time()} to
implement timeout.

@item 
bpo-34043@footnote{https://bugs.python.org/issue34043}: Optimize tarfile uncompress performance about 15% when gzip is
used.

@item 
bpo-34044@footnote{https://bugs.python.org/issue34044}: @code{subprocess.Popen} now copies the @emph{startupinfo} argument to
leave it unchanged: it will modify the copy, so that the same
@code{STARTUPINFO} object can be used multiple times.

@item 
bpo-34010@footnote{https://bugs.python.org/issue34010}: Fixed a performance regression for reading streams with
tarfile. The buffered read should use a list, instead of appending to a
bytes object.

@item 
bpo-34019@footnote{https://bugs.python.org/issue34019}: webbrowser: Correct the arguments passed to Opera Browser when
opening a new URL using the @code{webbrowser} module. Patch by Bumsik Kim.

@item 
bpo-34003@footnote{https://bugs.python.org/issue34003}: csv.DictReader now creates dicts instead of OrderedDicts. Patch
by Michael Selik.

@item 
bpo-33978@footnote{https://bugs.python.org/issue33978}: Closed existing logging handlers before reconfiguration via
fileConfig and dictConfig. Patch by Karthikeyan Singaravelan.

@item 
bpo-14117@footnote{https://bugs.python.org/issue14117}: Make minor tweaks to turtledemo. The ‘wikipedia’ example is now
‘rosette’, describing what it draws.  The ‘penrose’ print output is
reduced. The’1024’ output of ‘tree’ is eliminated.

@item 
bpo-33974@footnote{https://bugs.python.org/issue33974}: Fixed passing lists and tuples of strings containing special
characters @code{"}, @code{\}, @code{@{}, @code{@}} and @code{\n} as options to
@ref{116,,ttk} widgets.

@item 
bpo-27500@footnote{https://bugs.python.org/issue27500}: Fix getaddrinfo to resolve IPv6 addresses correctly.

@item 
bpo-24567@footnote{https://bugs.python.org/issue24567}: Improve random.choices() to handle subnormal input weights that
could occasionally trigger an IndexError.

@item 
bpo-33871@footnote{https://bugs.python.org/issue33871}: Fixed integer overflow in @ref{687,,os.readv()}, @ref{68a,,os.writev()},
@ref{686,,os.preadv()} and @ref{689,,os.pwritev()} and in @ref{62c,,os.sendfile()} with
@emph{headers} or @emph{trailers} arguments (on BSD-based OSes and macOS).

@item 
bpo-25007@footnote{https://bugs.python.org/issue25007}: Add @ref{69a,,copy.copy()} and @ref{69b,,copy.deepcopy()} support to zlib
compressors and decompressors.  Patch by Zackery Spytz.

@item 
bpo-33929@footnote{https://bugs.python.org/issue33929}: multiprocessing: Fix a race condition in Popen of
multiprocessing.popen_spawn_win32. The child process now duplicates the
read end of pipe instead of “stealing” it. Previously, the read end of
pipe was “stolen” by the child process, but it leaked a handle if the
child process had been terminated before it could steal the handle from
the parent process.

@item 
bpo-33899@footnote{https://bugs.python.org/issue33899}: Tokenize module now implicitly emits a NEWLINE when provided
with input that does not have a trailing new line.  This behavior now
matches what the C tokenizer does internally.  Contributed by Ammar Askar.

@item 
bpo-33897@footnote{https://bugs.python.org/issue33897}: Added a ‘force’ keyword argument to logging.basicConfig().

@item 
bpo-33695@footnote{https://bugs.python.org/issue33695}: @ref{538,,shutil.copytree()} uses @ref{576,,os.scandir()} function and
all copy functions depending from it use cached @ref{516,,os.stat()} values.
The speedup for copying a directory with 8000 files is around +9% on
Linux, +20% on Windows and + 30% on a Windows SMB share. Also the number
of @ref{516,,os.stat()} syscalls is reduced by 38% making
@ref{538,,shutil.copytree()} especially faster on network filesystems.
(Contributed by Giampaolo Rodola’ in bpo-33695@footnote{https://bugs.python.org/issue33695}.)

@item 
bpo-33916@footnote{https://bugs.python.org/issue33916}: bz2 and lzma: When Decompressor.__init__() is called twice,
free the old lock to not leak memory.

@item 
bpo-32568@footnote{https://bugs.python.org/issue32568}: Make select.epoll() and its documentation consistent regarding
@emph{sizehint} and @emph{flags}.

@item 
bpo-33833@footnote{https://bugs.python.org/issue33833}: Fixed bug in asyncio where ProactorSocketTransport logs
AssertionError if force closed during write.

@item 
bpo-33663@footnote{https://bugs.python.org/issue33663}: Convert content length to string before putting to header.

@item 
bpo-33721@footnote{https://bugs.python.org/issue33721}: @ref{c1,,os.path} functions that return a boolean result like
@ref{51a,,exists()}, @ref{51b,,lexists()}, @ref{51c,,isdir()},
@ref{51d,,isfile()}, @ref{518,,islink()}, and
@ref{51e,,ismount()}, and @ref{522,,pathlib.Path} methods that return a
boolean result like @ref{523,,exists()},
@ref{524,,is_dir()}, @ref{525,,is_file()},
@ref{526,,is_mount()}, @ref{527,,is_symlink()},
@ref{528,,is_block_device()},
@ref{529,,is_char_device()}, @ref{52a,,is_fifo()},
@ref{52b,,is_socket()} now return @code{False} instead of raising
@ref{17c,,ValueError} or its subclasses @ref{185,,UnicodeEncodeError} and
@ref{51f,,UnicodeDecodeError} for paths that contain characters or bytes
unrepresentable at the OS level.

@item 
bpo-26544@footnote{https://bugs.python.org/issue26544}: Fixed implementation of @ref{12e0,,platform.libc_ver()}. It almost
always returned version ‘2.9’ for glibc.

@item 
bpo-33843@footnote{https://bugs.python.org/issue33843}: Remove deprecated @code{cgi.escape}, @code{cgi.parse_qs} and
@code{cgi.parse_qsl}.

@item 
bpo-33842@footnote{https://bugs.python.org/issue33842}: Remove @code{tarfile.filemode} which is deprecated since Python
3.3.

@item 
bpo-30167@footnote{https://bugs.python.org/issue30167}: Prevent site.main() exception if PYTHONSTARTUP is set. Patch by
Steve Weber.

@item 
bpo-33805@footnote{https://bugs.python.org/issue33805}: Improve error message of dataclasses.replace() when an InitVar
is not specified

@item 
bpo-33687@footnote{https://bugs.python.org/issue33687}: Fix the call to @code{os.chmod()} for @code{uu.decode()} if a mode is
given or decoded. Patch by Timo Furrer.

@item 
bpo-33812@footnote{https://bugs.python.org/issue33812}: Datetime instance d with non-None tzinfo, but with
d.tzinfo.utcoffset(d) returning None is now treated as naive by the
astimezone() method.

@item 
bpo-32108@footnote{https://bugs.python.org/issue32108}: In configparser, don’t clear section when it is assigned to
itself.

@item 
bpo-27397@footnote{https://bugs.python.org/issue27397}: Make email module properly handle invalid-length base64
strings.

@item 
bpo-33578@footnote{https://bugs.python.org/issue33578}: Implement multibyte encoder/decoder state methods

@item 
bpo-30805@footnote{https://bugs.python.org/issue30805}: Avoid race condition with debug logging

@item 
bpo-33476@footnote{https://bugs.python.org/issue33476}: Fix _header_value_parser.py when address group is missing final
‘;’. Contributed by Enrique Perez-Terron

@item 
bpo-33694@footnote{https://bugs.python.org/issue33694}: asyncio: Fix a race condition causing data loss on
pause_reading()/resume_reading() when using the ProactorEventLoop.

@item 
bpo-32493@footnote{https://bugs.python.org/issue32493}: Correct test for @code{uuid_enc_be} availability in
@code{configure.ac}. Patch by Michael Felt.

@item 
bpo-33792@footnote{https://bugs.python.org/issue33792}: Add asyncio.WindowsSelectorEventLoopPolicy and
asyncio.WindowsProactorEventLoopPolicy.

@item 
bpo-33274@footnote{https://bugs.python.org/issue33274}: W3C DOM Level 1 specifies return value of
Element.removeAttributeNode() as “The Attr node that was removed.”
xml.dom.minidom now complies with this requirement.

@item 
bpo-33778@footnote{https://bugs.python.org/issue33778}: Update @code{unicodedata}’s database to Unicode version 11.0.0.

@item 
bpo-33165@footnote{https://bugs.python.org/issue33165}: Added a stacklevel parameter to logging calls to allow use of
wrapper/helper functions for logging APIs.

@item 
bpo-33770@footnote{https://bugs.python.org/issue33770}: improve base64 exception message for encoded inputs of invalid
length

@item 
bpo-33769@footnote{https://bugs.python.org/issue33769}: asyncio/start_tls: Fix error message; cancel callbacks in case
of an unhandled error; mark SSLTransport as closed if it is aborted.

@item 
bpo-33767@footnote{https://bugs.python.org/issue33767}: The concatenation (@code{+}) and repetition (@code{*}) sequence
operations now raise @ref{162,,TypeError} instead of @ref{891,,SystemError} when
performed on @ref{511,,mmap.mmap} objects.  Patch by Zackery Spytz.

@item 
bpo-33734@footnote{https://bugs.python.org/issue33734}: asyncio/ssl: Fix AttributeError, increase default handshake
timeout

@item 
bpo-31014@footnote{https://bugs.python.org/issue31014}: Fixed creating a controller for @ref{130,,webbrowser} when a user
specifies a path to an entry in the BROWSER environment variable.  Based
on patch by John Still.

@item 
bpo-2504@footnote{https://bugs.python.org/issue2504}: Add gettext.pgettext() and variants.

@item 
bpo-33197@footnote{https://bugs.python.org/issue33197}: Add description property for _ParameterKind

@item 
bpo-32751@footnote{https://bugs.python.org/issue32751}: When cancelling the task due to a timeout,
@ref{3d5,,asyncio.wait_for()} will now wait until the cancellation is complete.

@item 
bpo-32684@footnote{https://bugs.python.org/issue32684}: Fix gather to propagate cancellation of itself even with
return_exceptions.

@item 
bpo-33654@footnote{https://bugs.python.org/issue33654}: Support protocol type switching in SSLTransport.set_protocol().

@item 
bpo-33674@footnote{https://bugs.python.org/issue33674}: Pause the transport as early as possible to further reduce the
risk of data_received() being called before connection_made().

@item 
bpo-33671@footnote{https://bugs.python.org/issue33671}: @ref{56f,,shutil.copyfile()}, @ref{570,,shutil.copy()},
@ref{571,,shutil.copy2()}, @ref{538,,shutil.copytree()} and @ref{572,,shutil.move()} use
platform-specific fast-copy syscalls on Linux and macOS in order to copy
the file more efficiently. On Windows @ref{56f,,shutil.copyfile()} uses a
bigger default buffer size (1 MiB instead of 16 KiB) and a
@ref{573,,memoryview()}-based variant of @ref{574,,shutil.copyfileobj()} is used.
The speedup for copying a 512MiB file is about +26% on Linux, +50% on
macOS and +40% on Windows. Also, much less CPU cycles are consumed.
(Contributed by Giampaolo Rodola’ in bpo-25427@footnote{https://bugs.python.org/issue25427}.)

@item 
bpo-33674@footnote{https://bugs.python.org/issue33674}: Fix a race condition in SSLProtocol.connection_made() of
asyncio.sslproto: start immediately the handshake instead of using
call_soon(). Previously, data_received() could be called before the
handshake started, causing the handshake to hang or fail.

@item 
bpo-31647@footnote{https://bugs.python.org/issue31647}: Fixed bug where calling write_eof() on a
_SelectorSocketTransport after it’s already closed raises AttributeError.

@item 
bpo-32610@footnote{https://bugs.python.org/issue32610}: Make asyncio.all_tasks() return only pending tasks.

@item 
bpo-32410@footnote{https://bugs.python.org/issue32410}: Avoid blocking on file IO in sendfile fallback code

@item 
bpo-33469@footnote{https://bugs.python.org/issue33469}: Fix RuntimeError after closing loop that used run_in_executor

@item 
bpo-33672@footnote{https://bugs.python.org/issue33672}: Fix Task.__repr__ crash with Cython’s bogus coroutines

@item 
bpo-33654@footnote{https://bugs.python.org/issue33654}: Fix transport.set_protocol() to support switching between
asyncio.Protocol and asyncio.BufferedProtocol.  Fix loop.start_tls() to
work with asyncio.BufferedProtocols.

@item 
bpo-33652@footnote{https://bugs.python.org/issue33652}: Pickles of type variables and subscripted generics are now
future-proof and compatible with older Python versions.

@item 
bpo-32493@footnote{https://bugs.python.org/issue32493}: Fixed @ref{6da,,uuid.uuid1()} on FreeBSD.

@item 
bpo-33238@footnote{https://bugs.python.org/issue33238}: Add @code{InvalidStateError} to @ref{20,,concurrent.futures}.
@code{Future.set_result} and @code{Future.set_exception} now raise
@code{InvalidStateError} if the futures are not pending or running. Patch by
Jason Haydaman.

@item 
bpo-33618@footnote{https://bugs.python.org/issue33618}: Finalize and document preliminary and experimental TLS 1.3
support with OpenSSL 1.1.1

@item 
bpo-33625@footnote{https://bugs.python.org/issue33625}: Release GIL on @ref{12f2,,grp.getgrnam}, @ref{877,,grp.getgrgid}, @ref{12f1,,pwd.getpwnam}
and @ref{12df,,pwd.getpwuid} if reentrant variants of these functions are available.
Patch by William Grzybowski.

@item 
bpo-33623@footnote{https://bugs.python.org/issue33623}: Fix possible SIGSGV when asyncio.Future is created in __del__

@item 
bpo-11874@footnote{https://bugs.python.org/issue11874}: Use a better regex when breaking usage into wrappable parts.
Avoids bogus assertion errors from custom metavar strings.

@item 
bpo-30877@footnote{https://bugs.python.org/issue30877}: Fixed a bug in the Python implementation of the JSON decoder
that prevented the cache of parsed strings from clearing after finishing
the decoding. Based on patch by c-fos.

@item 
bpo-33604@footnote{https://bugs.python.org/issue33604}: Remove HMAC default to md5 marked for removal in 3.8 (removal
originally planned in 3.6, bump to 3.8 in gh-7062).

@item 
bpo-33582@footnote{https://bugs.python.org/issue33582}: Emit a deprecation warning for inspect.formatargspec

@item 
bpo-21145@footnote{https://bugs.python.org/issue21145}: Add @code{functools.cached_property} decorator, for computed
properties cached for the life of the instance.

@item 
bpo-33570@footnote{https://bugs.python.org/issue33570}: Change TLS 1.3 cipher suite settings for compatibility with
OpenSSL 1.1.1-pre6 and newer. OpenSSL 1.1.1 will have TLS 1.3 ciphers
enabled by default.

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Do not simplify arguments to @ref{235,,typing.Union}. Now
@code{Union[Manager, Employee]} is not simplified to @code{Employee} at runtime.
Such simplification previously caused several bugs and limited
possibilities for introspection.

@item 
bpo-12486@footnote{https://bugs.python.org/issue12486}: @ref{12fa,,tokenize.generate_tokens()} is now documented as a public
API to tokenize unicode strings. It was previously present but
undocumented.

@item 
bpo-33540@footnote{https://bugs.python.org/issue33540}: Add a new @code{block_on_close} class attribute to
@code{ForkingMixIn} and @code{ThreadingMixIn} classes of @ref{eb,,socketserver}.

@item 
bpo-33548@footnote{https://bugs.python.org/issue33548}: tempfile._candidate_tempdir_list should consider common TEMP
locations

@item 
bpo-33109@footnote{https://bugs.python.org/issue33109}: argparse subparsers are once again not required by default,
reverting the change in behavior introduced by bpo-26510@footnote{https://bugs.python.org/issue26510} in 3.7.0a2.

@item 
bpo-33541@footnote{https://bugs.python.org/issue33541}: Remove unused private method @code{_strptime.LocaleTime.__pad}
(a.k.a. @code{_LocaleTime__pad}).

@item 
bpo-33536@footnote{https://bugs.python.org/issue33536}: dataclasses.make_dataclass now checks for invalid field names
and duplicate fields. Also, added a check for invalid field
specifications.

@item 
bpo-33542@footnote{https://bugs.python.org/issue33542}: Prevent @code{uuid.get_node} from using a DUID instead of a MAC on
Windows. Patch by Zvi Effron

@item 
bpo-26819@footnote{https://bugs.python.org/issue26819}: Fix race condition with @code{ReadTransport.resume_reading} in
Windows proactor event loop.

@item 
Fix failure in @ref{432,,typing.get_type_hints()} when ClassVar was provided as a
string forward reference.

@item 
bpo-33516@footnote{https://bugs.python.org/issue33516}: @ref{a01,,unittest.mock.MagicMock} now supports the @code{__round__}
magic method.

@item 
bpo-28612@footnote{https://bugs.python.org/issue28612}: Added support for Site Maps to urllib’s @code{RobotFileParser} as
@ref{12fb,,RobotFileParser.site_maps()}. Patch by Lady Red, based
on patch by Peter Wirtz.

@item 
bpo-28167@footnote{https://bugs.python.org/issue28167}: Remove platform.linux_distribution, which was deprecated since
3.5.

@item 
bpo-33504@footnote{https://bugs.python.org/issue33504}: Switch the default dictionary implementation for
@ref{21,,configparser} from @ref{4e5,,collections.OrderedDict} to the standard
@ref{3b0,,dict} type.

@item 
bpo-33505@footnote{https://bugs.python.org/issue33505}: Optimize asyncio.ensure_future() by reordering if checks: 1.17x
faster.

@item 
bpo-33497@footnote{https://bugs.python.org/issue33497}: Add errors param to cgi.parse_multipart and make an encoding in
FieldStorage use the given errors (needed for Twisted).  Patch by Amber
Brown.

@item 
bpo-29235@footnote{https://bugs.python.org/issue29235}: The @code{cProfile.Profile} class can now be used as a
context manager. Patch by Scott Sanderson.

@item 
bpo-33495@footnote{https://bugs.python.org/issue33495}: Change dataclasses.Fields repr to use the repr of each of its
members, instead of str.  This makes it more clear what each field
actually represents.  This is especially true for the ‘type’ member.

@item 
bpo-26103@footnote{https://bugs.python.org/issue26103}: Correct @code{inspect.isdatadescriptor} to look for @code{__set__} or
@code{__delete__}.  Patch by Aaron Hall.

@item 
bpo-29209@footnote{https://bugs.python.org/issue29209}: Removed the @code{doctype()} method and the @emph{html} parameter of
the constructor of @ref{569,,XMLParser}.  The
@code{doctype()} method defined in a subclass will no longer be called.
Deprecated methods @code{getchildren()} and @code{getiterator()} in the
@ref{13e,,ElementTree} module emit now a @ref{1b2,,DeprecationWarning}
instead of @ref{3a9,,PendingDeprecationWarning}.

@item 
bpo-33453@footnote{https://bugs.python.org/issue33453}: Fix dataclasses to work if using literal string type
annotations or if using PEP 563 “Postponed Evaluation of Annotations”.
Only specific string prefixes are detected for both ClassVar (“ClassVar”
and “typing.ClassVar”) and InitVar (“InitVar” and “dataclasses.InitVar”).

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Minor fixes in typing module: add annotations to
@code{NamedTuple.__new__}, pass @code{*args} and @code{**kwds} in
@code{Generic.__new__}.  Original PRs by Paulius Šarka and Chad Dombrova.

@item 
bpo-33365@footnote{https://bugs.python.org/issue33365}: Print the header values besides the header keys instead just
the header keys if @emph{debuglevel} is set to >0 in @ref{8f,,http.client}. Patch
by Marco Strigl.

@item 
bpo-20087@footnote{https://bugs.python.org/issue20087}: Updated alias mapping with glibc 2.27 supported locales.

@item 
bpo-33422@footnote{https://bugs.python.org/issue33422}: Fix trailing quotation marks getting deleted when looking up
byte/string literals on pydoc. Patch by Andrés Delfino.

@item 
bpo-28167@footnote{https://bugs.python.org/issue28167}: The function @code{platform.linux_distribution} and
@code{platform.dist} now trigger a @code{DeprecationWarning} and have been
marked for removal in Python 3.8

@item 
bpo-33281@footnote{https://bugs.python.org/issue33281}: Fix ctypes.util.find_library regression on macOS.

@item 
bpo-33311@footnote{https://bugs.python.org/issue33311}: Text and html output generated by cgitb does not display
parentheses if the current call is done directly in the module. Patch by
Stéphane Blondon.

@item 
bpo-27300@footnote{https://bugs.python.org/issue27300}: The file classes in @emph{tempfile} now accept an @emph{errors} parameter
that complements the already existing @emph{encoding}.  Patch by Stephan Hohe.

@item 
bpo-32933@footnote{https://bugs.python.org/issue32933}: @ref{1296,,unittest.mock.mock_open()} now supports iteration over the
file contents. Patch by Tony Flury.

@item 
bpo-33217@footnote{https://bugs.python.org/issue33217}: Raise @ref{162,,TypeError} when looking up non-Enum objects in Enum
classes and Enum members.

@item 
bpo-33197@footnote{https://bugs.python.org/issue33197}: Update error message when constructing invalid
inspect.Parameters Patch by Dong-hee Na.

@item 
bpo-33383@footnote{https://bugs.python.org/issue33383}: Fixed crash in the get() method of the @ref{33,,dbm.ndbm} database
object when it is called with a single argument.

@item 
bpo-33375@footnote{https://bugs.python.org/issue33375}: The warnings module now finds the Python file associated with a
warning from the code object, rather than the frame’s global namespace.
This is consistent with how tracebacks and pdb find filenames, and should
work better for dynamically executed code.

@item 
bpo-33336@footnote{https://bugs.python.org/issue33336}: @code{imaplib} now allows @code{MOVE} command in @code{IMAP4.uid()} (RFC
6851: IMAP MOVE Extension) and potentially as a name of supported method
of @code{IMAP4} object.

@item 
bpo-32455@footnote{https://bugs.python.org/issue32455}: Added @emph{jump} parameter to @ref{aae,,dis.stack_effect()}.

@item 
bpo-27485@footnote{https://bugs.python.org/issue27485}: Rename and deprecate undocumented functions in
@ref{126,,urllib.parse()}.

@item 
bpo-33332@footnote{https://bugs.python.org/issue33332}: Add @code{signal.valid_signals()} to expose the POSIX sigfillset()
functionality.

@item 
bpo-33251@footnote{https://bugs.python.org/issue33251}: @code{ConfigParser.items()} was fixed so that key-value pairs passed
in via @ref{12fc,,vars} are not included in the resulting output.

@item 
bpo-33329@footnote{https://bugs.python.org/issue33329}: Fix multiprocessing regression on newer glibcs

@item 
bpo-33334@footnote{https://bugs.python.org/issue33334}: @ref{aae,,dis.stack_effect()} now supports all defined opcodes
including NOP and EXTENDED_ARG.

@item 
bpo-991266@footnote{https://bugs.python.org/issue991266}: Fix quoting of the @code{Comment} attribute of
@ref{12fd,,http.cookies.SimpleCookie}.

@item 
bpo-33131@footnote{https://bugs.python.org/issue33131}: Upgrade bundled version of pip to 10.0.1.

@item 
bpo-33308@footnote{https://bugs.python.org/issue33308}: Fixed a crash in the @code{parser} module when converting an ST
object to a tree of tuples or lists with @code{line_info=False} and
@code{col_info=True}.

@item 
bpo-23403@footnote{https://bugs.python.org/issue23403}: lib2to3 now uses pickle protocol 4 for pre-computed grammars.

@item 
bpo-33266@footnote{https://bugs.python.org/issue33266}: lib2to3 now recognizes @code{rf'...'} strings.

@item 
bpo-11594@footnote{https://bugs.python.org/issue11594}: Ensure line-endings are respected when using lib2to3.

@item 
bpo-33254@footnote{https://bugs.python.org/issue33254}: Have @ref{12fe,,importlib.resources.contents()} and
@ref{12ff,,importlib.abc.ResourceReader.contents()} return an @ref{dcc,,iterable}
instead of an @ref{11c3,,iterator}.

@item 
bpo-33265@footnote{https://bugs.python.org/issue33265}: @code{contextlib.ExitStack} and @code{contextlib.AsyncExitStack} now
use a method instead of a wrapper function for exit callbacks.

@item 
bpo-33263@footnote{https://bugs.python.org/issue33263}: Fix FD leak in @code{_SelectorSocketTransport}  Patch by Vlad
Starostin.

@item 
bpo-33256@footnote{https://bugs.python.org/issue33256}: Fix display of @code{<module>} call in the html produced by
@code{cgitb.html()}. Patch by Stéphane Blondon.

@item 
bpo-33144@footnote{https://bugs.python.org/issue33144}: @code{random.Random()} and its subclassing mechanism got optimized
to check only once at class/subclass instantiation time whether its
@code{getrandbits()} method can be relied on by other methods, including
@code{randrange()}, for the generation of arbitrarily large random integers.
Patch by Wolfgang Maier.

@item 
bpo-33185@footnote{https://bugs.python.org/issue33185}: Fixed regression when running pydoc with the @ref{611,,-m}
switch. (The regression was introduced in 3.7.0b3 by the resolution of
bpo-33053@footnote{https://bugs.python.org/issue33053})

This fix also changed pydoc to add @code{os.getcwd()} to @ref{489,,sys.path}
when necessary, rather than adding @code{"."}.

@item 
bpo-29613@footnote{https://bugs.python.org/issue29613}: Added support for the @code{SameSite} cookie flag to the
@code{http.cookies} module.

@item 
bpo-33169@footnote{https://bugs.python.org/issue33169}: Delete entries of @code{None} in @ref{74c,,sys.path_importer_cache}
when @code{importlib.machinery.invalidate_caches()} is called.

@item 
bpo-33203@footnote{https://bugs.python.org/issue33203}: @code{random.Random.choice()} now raises @code{IndexError} for empty
sequences consistently even when called from subclasses without a
@code{getrandbits()} implementation.

@item 
bpo-33224@footnote{https://bugs.python.org/issue33224}: Update difflib.mdiff() for 
@geindex Python Enhancement Proposals; PEP 479
PEP 479@footnote{https://www.python.org/dev/peps/pep-0479}.  Convert an uncaught
StopIteration in a generator into a return-statement.

@item 
bpo-33209@footnote{https://bugs.python.org/issue33209}: End framing at the end of C implementation of
@ref{1300,,pickle.Pickler.dump()}.

@item 
bpo-32861@footnote{https://bugs.python.org/issue32861}: The urllib.robotparser’s @code{__str__} representation now
includes wildcard entries and the “Crawl-delay” and “Request-rate” fields.
Also removes extra newlines that were being appended to the end of the
string. Patch by Michael Lazar.

@item 
bpo-23403@footnote{https://bugs.python.org/issue23403}: @code{DEFAULT_PROTOCOL} in @ref{c5,,pickle} was bumped to 4. Protocol
4 is described in 
@geindex Python Enhancement Proposals; PEP 3154
PEP 3154@footnote{https://www.python.org/dev/peps/pep-3154} and available since Python 3.4. It offers
better performance and smaller size compared to protocol 3 introduced in
Python 3.0.

@item 
bpo-20104@footnote{https://bugs.python.org/issue20104}: Improved error handling and fixed a reference leak in
@ref{56e,,os.posix_spawn()}.

@item 
bpo-33106@footnote{https://bugs.python.org/issue33106}: Deleting a key from a read-only dbm database raises module
specific error instead of KeyError.

@item 
bpo-33175@footnote{https://bugs.python.org/issue33175}: In dataclasses, Field.__set_name__ now looks up the
__set_name__ special method on the class, not the instance, of the default
value.

@item 
bpo-32380@footnote{https://bugs.python.org/issue32380}: Create functools.singledispatchmethod to support generic single
dispatch on descriptors and methods.

@item 
bpo-33141@footnote{https://bugs.python.org/issue33141}: Have Field objects pass through __set_name__ to their default
values, if they have their own __set_name__.

@item 
bpo-33096@footnote{https://bugs.python.org/issue33096}: Allow ttk.Treeview.insert to insert iid that has a false
boolean value. Note iid=0 and iid=False would be same. Patch by Garvit
Khatri.

@item 
bpo-32873@footnote{https://bugs.python.org/issue32873}: Treat type variables and special typing forms as immutable by
copy and pickle.  This fixes several minor issues and inconsistencies, and
improves backwards compatibility with Python 3.6.

@item 
bpo-33134@footnote{https://bugs.python.org/issue33134}: When computing dataclass’s __hash__, use the lookup table to
contain the function which returns the __hash__ value.  This is an
improvement over looking up a string, and then testing that string to see
what to do.

@item 
bpo-33127@footnote{https://bugs.python.org/issue33127}: The ssl module now compiles with LibreSSL 2.7.1.

@item 
bpo-32505@footnote{https://bugs.python.org/issue32505}: Raise TypeError if a member variable of a dataclass is of type
Field, but doesn’t have a type annotation.

@item 
bpo-33078@footnote{https://bugs.python.org/issue33078}: Fix the failure on OSX caused by the tests relying on
sem_getvalue

@item 
bpo-33116@footnote{https://bugs.python.org/issue33116}: Add ‘Field’ to dataclasses.__all__.

@item 
bpo-32896@footnote{https://bugs.python.org/issue32896}: Fix an error where subclassing a dataclass with a field that
uses a default_factory would generate an incorrect class.

@item 
bpo-33100@footnote{https://bugs.python.org/issue33100}: Dataclasses: If a field has a default value that’s a
MemberDescriptorType, then it’s from that field being in __slots__, not an
actual default value.

@item 
bpo-32953@footnote{https://bugs.python.org/issue32953}: If a non-dataclass inherits from a frozen dataclass, allow
attributes to be added to the derived class.  Only attributes from the
frozen dataclass cannot be assigned to.  Require all dataclasses in a
hierarchy to be either all frozen or all non-frozen.

@item 
bpo-33097@footnote{https://bugs.python.org/issue33097}: Raise RuntimeError when @code{executor.submit} is called during
interpreter shutdown.

@item 
bpo-32968@footnote{https://bugs.python.org/issue32968}: Modulo and floor division involving Fraction and float should
return float.

@item 
bpo-33061@footnote{https://bugs.python.org/issue33061}: Add missing @code{NoReturn} to @code{__all__} in typing.py

@item 
bpo-33078@footnote{https://bugs.python.org/issue33078}: Fix the size handling in multiprocessing.Queue when a pickling
error occurs.

@item 
bpo-33064@footnote{https://bugs.python.org/issue33064}: lib2to3 now properly supports trailing commas after @code{*args}
and @code{**kwargs} in function signatures.

@item 
bpo-33056@footnote{https://bugs.python.org/issue33056}: FIX properly close leaking fds in
concurrent.futures.ProcessPoolExecutor.

@item 
bpo-33021@footnote{https://bugs.python.org/issue33021}: Release the GIL during fstat() calls, avoiding hang of all
threads when calling mmap.mmap(), os.urandom(), and random.seed().  Patch
by Nir Soffer.

@item 
bpo-31804@footnote{https://bugs.python.org/issue31804}: Avoid failing in multiprocessing.Process if the standard
streams are closed or None at exit.

@item 
bpo-33034@footnote{https://bugs.python.org/issue33034}: Providing an explicit error message when casting the port
property to anything that is not an integer value using @code{urlparse()} and
@code{urlsplit()}. Patch by Matt Eaton.

@item 
bpo-30249@footnote{https://bugs.python.org/issue30249}: Improve struct.unpack_from() exception messages for problems
with the buffer size and offset.

@item 
bpo-33037@footnote{https://bugs.python.org/issue33037}: Skip sending/receiving data after SSL transport closing.

@item 
bpo-27683@footnote{https://bugs.python.org/issue27683}: Fix a regression in @ref{9e,,ipaddress} that result of
@code{hosts()} is empty when the network is constructed by a tuple
containing an integer mask and only 1 bit left for addresses.

@item 
bpo-22674@footnote{https://bugs.python.org/issue22674}: Add the strsignal() function in the signal module that returns
the system description of the given signal, as returned by strsignal(3).

@item 
bpo-32999@footnote{https://bugs.python.org/issue32999}: Fix C implementation of @code{ABC.__subclasscheck__(cls,
subclass)} crashed when @code{subclass} is not a type object.

@item 
bpo-33009@footnote{https://bugs.python.org/issue33009}: Fix inspect.signature() for single-parameter partialmethods.

@item 
bpo-32969@footnote{https://bugs.python.org/issue32969}: Expose several missing constants in zlib and fix corresponding
documentation.

@item 
bpo-32056@footnote{https://bugs.python.org/issue32056}: Improved exceptions raised for invalid number of channels and
sample width when read an audio file in modules @ref{4,,aifc}, @ref{12e,,wave}
and @ref{f5,,sunau}.

@item 
bpo-32970@footnote{https://bugs.python.org/issue32970}: Improved disassembly of the MAKE_FUNCTION instruction.

@item 
bpo-32844@footnote{https://bugs.python.org/issue32844}: Fix wrong redirection of a low descriptor (0 or 1) to stderr in
subprocess if another low descriptor is closed.

@item 
bpo-32960@footnote{https://bugs.python.org/issue32960}: For dataclasses, disallow inheriting frozen from non-frozen
classes, and also disallow inheriting non-frozen from frozen classes. This
restriction will be relaxed at a future date.

@item 
bpo-32713@footnote{https://bugs.python.org/issue32713}: Fixed tarfile.itn handling of out-of-bounds float values. Patch
by Joffrey Fuhrer.

@item 
bpo-32257@footnote{https://bugs.python.org/issue32257}: The ssl module now contains OP_NO_RENEGOTIATION constant,
available with OpenSSL 1.1.0h or 1.1.1.

@item 
bpo-32951@footnote{https://bugs.python.org/issue32951}: Direct instantiation of SSLSocket and SSLObject objects is now
prohibited. The constructors were never documented, tested, or designed as
public constructors. Users were suppose to use ssl.wrap_socket() or
SSLContext.

@item 
bpo-32929@footnote{https://bugs.python.org/issue32929}: Remove the tri-state parameter “hash”, and add the boolean
“unsafe_hash”. If unsafe_hash is True, add a __hash__ function, but if a
__hash__ exists, raise TypeError.  If unsafe_hash is False, add a __hash__
based on the values of eq= and frozen=.  The unsafe_hash=False behavior is
the same as the old hash=None behavior.  unsafe_hash=False is the default,
just as hash=None used to be.

@item 
bpo-32947@footnote{https://bugs.python.org/issue32947}: Add OP_ENABLE_MIDDLEBOX_COMPAT and test workaround for TLSv1.3
for future compatibility with OpenSSL 1.1.1.

@item 
bpo-32146@footnote{https://bugs.python.org/issue32146}: Document the interaction between frozen executables and the
spawn and forkserver start methods in multiprocessing.

@item 
bpo-30622@footnote{https://bugs.python.org/issue30622}: The ssl module now detects missing NPN support in LibreSSL.

@item 
bpo-32922@footnote{https://bugs.python.org/issue32922}: dbm.open() now encodes filename with the filesystem encoding
rather than default encoding.

@item 
bpo-32759@footnote{https://bugs.python.org/issue32759}: Free unused arenas in multiprocessing.heap.

@item 
bpo-32859@footnote{https://bugs.python.org/issue32859}: In @code{os.dup2}, don’t check every call whether the @code{dup3}
syscall exists or not.

@item 
bpo-32556@footnote{https://bugs.python.org/issue32556}: nt._getfinalpathname, nt._getvolumepathname and
nt._getdiskusage now correctly convert from bytes.

@item 
bpo-21060@footnote{https://bugs.python.org/issue21060}: Rewrite confusing message from setup.py upload from “No dist
file created in earlier command” to the more helpful “Must create and
upload files in one command”.

@item 
bpo-32857@footnote{https://bugs.python.org/issue32857}: In @ref{10c,,tkinter}, @code{after_cancel(None)} now raises a
@ref{17c,,ValueError} instead of canceling the first scheduled function.
Patch by Cheryl Sabella.

@item 
bpo-32852@footnote{https://bugs.python.org/issue32852}: Make sure sys.argv remains as a list when running trace.

@item 
bpo-31333@footnote{https://bugs.python.org/issue31333}: @code{_abc} module is added.  It is a speedup module with C
implementations for various functions and methods in @code{abc}.  Creating an
ABC subclass and calling @code{isinstance} or @code{issubclass} with an ABC
subclass are up to 1.5x faster. In addition, this makes Python start-up up
to 10% faster.

Note that the new implementation hides internal registry and caches,
previously accessible via private attributes @code{_abc_registry},
@code{_abc_cache}, and @code{_abc_negative_cache}.  There are three debugging
helper methods that can be used instead @code{_dump_registry},
@code{_abc_registry_clear}, and @code{_abc_caches_clear}.

@item 
bpo-32841@footnote{https://bugs.python.org/issue32841}: Fixed @ref{58d,,asyncio.Condition} issue which silently ignored
cancellation after notifying and cancelling a conditional lock. Patch by
Bar Harel.

@item 
bpo-32819@footnote{https://bugs.python.org/issue32819}: ssl.match_hostname() has been simplified and no longer depends
on re and ipaddress module for wildcard and IP addresses. Error reporting
for invalid wildcards has been improved.

@item 
bpo-19675@footnote{https://bugs.python.org/issue19675}: @code{multiprocessing.Pool} no longer leaks processes if its
initialization fails.

@item 
bpo-32394@footnote{https://bugs.python.org/issue32394}: socket: Remove
TCP_FASTOPEN,TCP_KEEPCNT,TCP_KEEPIDLE,TCP_KEEPINTVL flags on older version
Windows during run-time.

@item 
bpo-31787@footnote{https://bugs.python.org/issue31787}: Fixed refleaks of @code{__init__()} methods in various modules.
(Contributed by Oren Milman)

@item 
bpo-30157@footnote{https://bugs.python.org/issue30157}: Fixed guessing quote and delimiter in csv.Sniffer.sniff() when
only the last field is quoted.  Patch by Jake Davis.

@item 
bpo-30688@footnote{https://bugs.python.org/issue30688}: Added support of @code{\N@{name@}} escapes in regular expressions.
Based on patch by Jonathan Eunice.

@item 
bpo-32792@footnote{https://bugs.python.org/issue32792}: collections.ChainMap() preserves the order of the underlying
mappings.

@item 
bpo-32775@footnote{https://bugs.python.org/issue32775}: @ref{1301,,fnmatch.translate()} no longer produces patterns which
contain set operations. Sets starting with ‘[’ or containing ‘–’, ‘&&’,
‘~~’ or ‘||’ will be interpreted differently in regular expressions in
future versions. Currently they emit warnings. fnmatch.translate() now
avoids producing patterns containing such sets by accident.

@item 
bpo-32622@footnote{https://bugs.python.org/issue32622}: Implement native fast sendfile for Windows proactor event loop.

@item 
bpo-32777@footnote{https://bugs.python.org/issue32777}: Fix a rare but potential pre-exec child process deadlock in
subprocess on POSIX systems when marking file descriptors inheritable on
exec in the child process.  This bug appears to have been introduced in
3.4.

@item 
bpo-32647@footnote{https://bugs.python.org/issue32647}: The ctypes module used to depend on indirect linking for
dlopen. The shared extension is now explicitly linked against libdl on
platforms with dl.

@item 
bpo-32749@footnote{https://bugs.python.org/issue32749}: A @ref{31,,dbm.dumb} database opened with flags ‘r’ is now
read-only. @ref{5b4,,dbm.dumb.open()} with flags ‘r’ and ‘w’ no longer creates
a database if it does not exist.

@item 
bpo-32741@footnote{https://bugs.python.org/issue32741}: Implement @code{asyncio.TimerHandle.when()} method.

@item 
bpo-32691@footnote{https://bugs.python.org/issue32691}: Use mod_spec.parent when running modules with pdb

@item 
bpo-32734@footnote{https://bugs.python.org/issue32734}: Fixed @code{asyncio.Lock()} safety issue which allowed acquiring
and locking the same lock multiple times, without it being free. Patch by
Bar Harel.

@item 
bpo-32727@footnote{https://bugs.python.org/issue32727}: Do not include name field in SMTP envelope from address. Patch
by Stéphane Wirtel

@item 
bpo-31453@footnote{https://bugs.python.org/issue31453}: Add TLSVersion constants and SSLContext.maximum_version /
minimum_version attributes. The new API wraps OpenSSL 1.1
@indicateurl{https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_min_proto_version.html}
feature.

@item 
bpo-24334@footnote{https://bugs.python.org/issue24334}: Internal implementation details of ssl module were cleaned up.
The SSLSocket has one less layer of indirection. Owner and session
information are now handled by the SSLSocket and SSLObject constructor.
Channel binding implementation has been simplified.

@item 
bpo-31848@footnote{https://bugs.python.org/issue31848}: Fix the error handling in Aifc_read.initfp() when the SSND
chunk is not found. Patch by Zackery Spytz.

@item 
bpo-32585@footnote{https://bugs.python.org/issue32585}: Add Ttk spinbox widget to @ref{116,,tkinter.ttk}.  Patch by Alan D
Moore.

@item 
bpo-32512@footnote{https://bugs.python.org/issue32512}: @ref{ce,,profile} CLI accepts @code{-m module_name} as an alternative
to script path.

@item 
bpo-8525@footnote{https://bugs.python.org/issue8525}: help() on a type now displays builtin subclasses. This is
intended primarily to help with notification of more specific exception
subclasses.

Patch by Sanyam Khurana.

@item 
bpo-31639@footnote{https://bugs.python.org/issue31639}: http.server now exposes a ThreadingHTTPServer class and uses it
when the module is run with @code{-m} to cope with web browsers pre-opening
sockets.

@item 
bpo-29877@footnote{https://bugs.python.org/issue29877}: compileall: import ProcessPoolExecutor only when needed,
preventing hangs on low resource platforms

@item 
bpo-32221@footnote{https://bugs.python.org/issue32221}: Various functions returning tuple containing IPv6 addresses now
omit @code{%scope} part since the same information is already encoded in
@emph{scopeid} tuple item. Especially this speeds up @code{socket.recvfrom()}
when it receives multicast packet since useless resolving of network
interface name is omitted.

@item 
bpo-32147@footnote{https://bugs.python.org/issue32147}: @ref{1302,,binascii.unhexlify()} is now up to 2 times faster. Patch
by Sergey Fedoseev.

@item 
bpo-30693@footnote{https://bugs.python.org/issue30693}: The TarFile class now recurses directories in a reproducible
way.

@item 
bpo-30693@footnote{https://bugs.python.org/issue30693}: The ZipFile class now recurses directories in a reproducible
way.

@item 
bpo-31680@footnote{https://bugs.python.org/issue31680}: Added @ref{4eb,,curses.ncurses_version}.

@item 
bpo-31908@footnote{https://bugs.python.org/issue31908}: Fix output of cover files for @code{trace} module command-line
tool. Previously emitted cover files only when @code{--missing} option was
used. Patch by Michael Selik.

@item 
bpo-31608@footnote{https://bugs.python.org/issue31608}: Raise a @code{TypeError} instead of crashing if a
@code{collections.deque} subclass returns a non-deque from @code{__new__}. Patch
by Oren Milman.

@item 
bpo-31425@footnote{https://bugs.python.org/issue31425}: Add support for sockets of the AF_QIPCRTR address family,
supported by the Linux kernel. This is used to communicate with services,
such as GPS or radio, running on Qualcomm devices. Patch by Bjorn
Andersson.

@item 
bpo-22005@footnote{https://bugs.python.org/issue22005}: Implemented unpickling instances of
@ref{363,,datetime}, @ref{3e2,,date} and
@ref{364,,time} pickled by Python 2. @code{encoding='latin1'} should
be used for successful decoding.

@item 
bpo-27645@footnote{https://bugs.python.org/issue27645}: @ref{18b,,sqlite3.Connection} now exposes a
@ref{6a7,,backup} method, if the underlying SQLite
library is at version 3.6.11 or higher.  Patch by Lele Gaifax.

@item 
bpo-16865@footnote{https://bugs.python.org/issue16865}: Support arrays >=2GiB in @ref{29,,ctypes}.  Patch by Segev Finer.

@item 
bpo-31508@footnote{https://bugs.python.org/issue31508}: Removed support of arguments in
@ref{5b1,,tkinter.ttk.Treeview.selection}. It was deprecated in 3.6.  Use
specialized methods like @code{selection_set} for changing the selection.

@item 
bpo-29456@footnote{https://bugs.python.org/issue29456}: Fix bugs in hangul normalization: u1176, u11a7 and u11c3
@end itemize

@node Documentation<27>,Tests<25>,Library<26>,Python 3 8 0 alpha 1
@anchor{whatsnew/changelog id231}@anchor{1303}
@subsubsection Documentation



@itemize -

@item 
bpo-21257@footnote{https://bugs.python.org/issue21257}: Document @ref{1304,,http.client.parse_headers()}.

@item 
bpo-34764@footnote{https://bugs.python.org/issue34764}: Improve example of iter() with 2nd sentinel argument.

@item 
bpo-35564@footnote{https://bugs.python.org/issue35564}: Explicitly set master_doc variable in conf.py for compliance
with Sphinx 2.0

@item 
bpo-35511@footnote{https://bugs.python.org/issue35511}: Specified that profile.Profile class doesn’t not support enable
or disable methods. Also, elaborated that Profile object as a context
manager is only supported in cProfile module.

@item 
bpo-10536@footnote{https://bugs.python.org/issue10536}: Enhance the gettext docs. Patch by Éric Araujo

@item 
bpo-35089@footnote{https://bugs.python.org/issue35089}: Remove mention of @code{typing.io} and @code{typing.re}. Their types
should be imported from @code{typing} directly.

@item 
bpo-35038@footnote{https://bugs.python.org/issue35038}: Fix the documentation about an unexisting @code{f_restricted}
attribute in the frame object. Patch by Stéphane Wirtel

@item 
bpo-35042@footnote{https://bugs.python.org/issue35042}: Replace PEP XYZ by the pep role and allow to use the direct
links to the PEPs.

@item 
bpo-35044@footnote{https://bugs.python.org/issue35044}: Fix the documentation with the role @code{exc} for the
appropriated exception. Patch by Stéphane Wirtel

@item 
bpo-35035@footnote{https://bugs.python.org/issue35035}: Rename documentation for @ref{71,,email.utils} to
@code{email.utils.rst}.

@item 
bpo-34967@footnote{https://bugs.python.org/issue34967}: Use app.add_object_type() instead of the deprecated Sphinx
function app.description_unit()

@item 
bpo-34913@footnote{https://bugs.python.org/issue34913}: Add documentation about the new command line interface of the
gzip module.

@item 
bpo-32174@footnote{https://bugs.python.org/issue32174}: chm document displays non-ASCII charaters properly on some MBCS
Windows systems.

@item 
bpo-11233@footnote{https://bugs.python.org/issue11233}: Create availability directive for documentation.  Original
patch by Georg Brandl.

@item 
bpo-34790@footnote{https://bugs.python.org/issue34790}: Document how passing coroutines to asyncio.wait() can be
confusing.

@item 
bpo-34552@footnote{https://bugs.python.org/issue34552}: Make clear that @code{==} operator sometimes is equivalent to
@code{is}. The @code{<}, @code{<=}, @code{>} and @code{>=} operators are only defined where
they make sense.

@item 
bpo-28617@footnote{https://bugs.python.org/issue28617}: Fixed info in the stdtypes docs concerning the types that
support membership tests.

@item 
bpo-20177@footnote{https://bugs.python.org/issue20177}: Migrate datetime.date.fromtimestamp to Argument Clinic. Patch
by Tim Hoffmann.

@item 
bpo-34065@footnote{https://bugs.python.org/issue34065}: Fix wrongly written basicConfig documentation markup syntax

@item 
bpo-33460@footnote{https://bugs.python.org/issue33460}: replaced ellipsis with correct error codes in tutorial chapter
3.

@item 
bpo-33847@footnote{https://bugs.python.org/issue33847}: Add ‘@@’ operator entry to index.

@item 
bpo-33409@footnote{https://bugs.python.org/issue33409}: Clarified the relationship between 
@geindex Python Enhancement Proposals; PEP 538
PEP 538@footnote{https://www.python.org/dev/peps/pep-0538}’s
PYTHONCOERCECLOCALE and PEP 540’s PYTHONUTF8 mode.

@item 
bpo-33197@footnote{https://bugs.python.org/issue33197}: Add versionadded tag to the documentation of
ParameterKind.description

@item 
bpo-17045@footnote{https://bugs.python.org/issue17045}: Improve the C-API doc for PyTypeObject.  This includes adding
several quick-reference tables and a lot of missing slot/typedef entries.
The existing entries were also cleaned up with a slightly more consistent
format.

@item 
bpo-33736@footnote{https://bugs.python.org/issue33736}: Improve the documentation of @ref{636,,asyncio.open_connection()},
@ref{637,,asyncio.start_server()} and their UNIX socket counterparts.

@item 
bpo-23859@footnote{https://bugs.python.org/issue23859}: Document that @ref{440,,asyncio.wait()} does not cancel its futures on
timeout.

@item 
bpo-32436@footnote{https://bugs.python.org/issue32436}: Document 
@geindex Python Enhancement Proposals; PEP 567
PEP 567@footnote{https://www.python.org/dev/peps/pep-0567} changes to asyncio.

@item 
bpo-33604@footnote{https://bugs.python.org/issue33604}: Update HMAC md5 default to a DeprecationWarning, bump removal
to 3.8.

@item 
bpo-33594@footnote{https://bugs.python.org/issue33594}: Document @code{getargspec}, @code{from_function} and @code{from_builtin}
as deprecated in their respective docstring, and include version since
deprecation in DeprecationWarning message.

@item 
bpo-33503@footnote{https://bugs.python.org/issue33503}: Fix broken pypi link

@item 
bpo-33421@footnote{https://bugs.python.org/issue33421}: Add missing documentation for @code{typing.AsyncContextManager}.

@item 
bpo-33487@footnote{https://bugs.python.org/issue33487}: BZ2file now emit a DeprecationWarning when buffering=None is
passed, the deprecation message and documentation also now explicitly
state it is deprecated since 3.0.

@item 
bpo-33378@footnote{https://bugs.python.org/issue33378}: Add Korean language switcher for @indicateurl{https://docs.python.org/3/}

@item 
bpo-33276@footnote{https://bugs.python.org/issue33276}: Clarify that the @code{__path__} attribute on modules cannot be
just any value.

@item 
bpo-33201@footnote{https://bugs.python.org/issue33201}: Modernize documentation for writing C extension types.

@item 
bpo-33195@footnote{https://bugs.python.org/issue33195}: Deprecate @code{Py_UNICODE} usage in @code{c-api/arg} document.
@code{Py_UNICODE} related APIs are deprecated since Python 3.3, but it is
missed in the document.

@item 
bpo-33126@footnote{https://bugs.python.org/issue33126}: Document PyBuffer_ToContiguous().

@item 
bpo-27212@footnote{https://bugs.python.org/issue27212}: Modify documentation for the @code{islice()} recipe to consume
initial values up to the start index.

@item 
bpo-28247@footnote{https://bugs.python.org/issue28247}: Update @ref{148,,zipapp} documentation to describe how to make
standalone applications.

@item 
bpo-18802@footnote{https://bugs.python.org/issue18802}: Documentation changes for ipaddress.  Patch by Jon Foster and
Berker Peksag.

@item 
bpo-27428@footnote{https://bugs.python.org/issue27428}: Update documentation to clarify that @code{WindowsRegistryFinder}
implements @code{MetaPathFinder}. (Patch by Himanshu Lakhara)

@item 
bpo-28124@footnote{https://bugs.python.org/issue28124}: The ssl module function ssl.wrap_socket() has been
de-emphasized and deprecated in favor of the more secure and efficient
SSLContext.wrap_socket() method.

@item 
bpo-17232@footnote{https://bugs.python.org/issue17232}: Clarify docs for -O and -OO.  Patch by Terry Reedy.

@item 
bpo-32436@footnote{https://bugs.python.org/issue32436}: Add documentation for the contextvars module (PEP 567).

@item 
bpo-32800@footnote{https://bugs.python.org/issue32800}: Update link to w3c doc for xml default namespaces.

@item 
bpo-11015@footnote{https://bugs.python.org/issue11015}: Update @ref{100,,test.support} documentation.

@item 
bpo-32613@footnote{https://bugs.python.org/issue32613}: Update the faq/windows.html to use the py command from PEP 397
instead of python.

@item 
bpo-8722@footnote{https://bugs.python.org/issue8722}: Document @code{__getattr__()} behavior when property @code{get()}
method raises @ref{163,,AttributeError}.

@item 
bpo-32614@footnote{https://bugs.python.org/issue32614}: Modify RE examples in documentation to use raw strings to
prevent @ref{1b2,,DeprecationWarning} and add text to REGEX HOWTO to highlight
the deprecation.

@item 
bpo-20709@footnote{https://bugs.python.org/issue20709}: Remove the paragraph where we explain that os.utime() does not
support a directory as path under Windows. Patch by Jan-Philip Gehrcke

@item 
bpo-32722@footnote{https://bugs.python.org/issue32722}: Remove the bad example in the tutorial of the Generator
Expression. Patch by Stéphane Wirtel

@item 
bpo-31972@footnote{https://bugs.python.org/issue31972}: Improve docstrings for @ref{1c4,,pathlib.PurePath} subclasses.

@item 
bpo-30607@footnote{https://bugs.python.org/issue30607}: Use the externalized @code{python-docs-theme} package when
building the documentation.

@item 
bpo-8243@footnote{https://bugs.python.org/issue8243}: Add a note about curses.addch and curses.addstr exception
behavior when writing outside a window, or pad.

@item 
bpo-32337@footnote{https://bugs.python.org/issue32337}: Update documentation related with @code{dict} order.

@item 
bpo-25041@footnote{https://bugs.python.org/issue25041}: Document @code{AF_PACKET} in the @ref{ea,,socket} module.

@item 
bpo-31432@footnote{https://bugs.python.org/issue31432}: Clarify meaning of CERT_NONE, CERT_OPTIONAL, and CERT_REQUIRED
flags for ssl.SSLContext.verify_mode.
@end itemize

@node Tests<25>,Build<25>,Documentation<27>,Python 3 8 0 alpha 1
@anchor{whatsnew/changelog id232}@anchor{1305}
@subsubsection Tests



@itemize -

@item 
bpo-35772@footnote{https://bugs.python.org/issue35772}: Fix sparse file tests of test_tarfile on ppc64 with the tmpfs
filesystem. Fix the function testing if the filesystem supports sparse
files: create a file which contains data and “holes”, instead of creating
a file which contains no data. tmpfs effective block size is a page size
(tmpfs lives in the page cache). RHEL uses 64 KiB pages on aarch64, ppc64,
ppc64le, only s390x and x86_64 use 4 KiB pages, whereas the test punch
holes of 4 KiB.

@item 
bpo-35045@footnote{https://bugs.python.org/issue35045}: Make ssl tests less strict and also accept TLSv1 as system
default. The changes unbreaks test_min_max_version on Fedora 29.

@item 
bpo-32710@footnote{https://bugs.python.org/issue32710}: @code{test_asyncio/test_sendfile.py} now resets the event loop
policy using @code{tearDownModule()} as done in other tests, to prevent a
warning when running tests on Windows.

@item 
bpo-33717@footnote{https://bugs.python.org/issue33717}: test.pythoninfo now logs information of all clocks, not only
time.time() and time.perf_counter().

@item 
bpo-35488@footnote{https://bugs.python.org/issue35488}: Add a test to pathlib’s Path.match() to verify it does not
support glob-style ** recursive pattern matching.

@item 
bpo-31731@footnote{https://bugs.python.org/issue31731}: Fix a race condition in @code{check_interrupted_write()} of
test_io: create directly the thread with SIGALRM signal blocked, rather
than blocking the signal later from the thread. Previously, it was
possible that the thread gets the signal before the signal is blocked.

@item 
bpo-35424@footnote{https://bugs.python.org/issue35424}: Fix test_multiprocessing_main_handling: use
@code{multiprocessing.Pool} with a context manager and then explicitly
join the pool.

@item 
bpo-35519@footnote{https://bugs.python.org/issue35519}: Rename @code{test.bisect} module to @code{test.bisect_cmd} to
avoid conflict with @ref{10,,bisect} module when running directly a test like
@code{./python Lib/test/test_xmlrpc.py}.

@item 
bpo-35513@footnote{https://bugs.python.org/issue35513}: Replace @ref{5fc,,time.time()} with @ref{9ed,,time.monotonic()} in tests
to measure time delta.

@item 
bpo-34279@footnote{https://bugs.python.org/issue34279}: @ref{1306,,test.support.run_unittest()} no longer raise
@code{TestDidNotRun} if the test result contains skipped tests. The
exception is now only raised if no test have been run and no test have
been skipped.

@item 
bpo-35412@footnote{https://bugs.python.org/issue35412}: Add testcase to @code{test_future4}: check unicode literal.

@item 
bpo-26704@footnote{https://bugs.python.org/issue26704}: Added test demonstrating double-patching of an instance method.
Patch by Anthony Sottile.

@item 
bpo-33725@footnote{https://bugs.python.org/issue33725}: test_multiprocessing_fork may crash on recent versions of
macOS.  Until the issue is resolved, skip the test on macOS.

@item 
bpo-35352@footnote{https://bugs.python.org/issue35352}: Modify test_asyncio to use the certificate set from the test
directory.

@item 
bpo-35317@footnote{https://bugs.python.org/issue35317}: Fix @code{mktime()} overflow error in @code{test_email}: run
@code{test_localtime_daylight_true_dst_true()} and
@code{test_localtime_daylight_false_dst_true()} with a specific timezone.

@item 
bpo-21263@footnote{https://bugs.python.org/issue21263}: After several reports that test_gdb does not work properly on
macOS and since gdb is not shipped by default anymore, test_gdb is now
skipped on macOS when LLVM Clang has been used to compile Python. Patch by
Lysandros Nikolaou

@item 
bpo-34279@footnote{https://bugs.python.org/issue34279}: regrtest issue a warning when no tests have been executed in a
particular test file. Also, a new final result state is issued if no test
have been executed across all test files. Patch by Pablo Galindo.

@item 
bpo-34962@footnote{https://bugs.python.org/issue34962}: make docstest in Doc now passes., and is enforced in CI

@item 
bpo-23596@footnote{https://bugs.python.org/issue23596}: Use argparse for the command line of the gzip module. Patch by
Antony Lee

@item 
bpo-34537@footnote{https://bugs.python.org/issue34537}: Fix @code{test_gdb.test_strings()} when @code{LC_ALL=C} and GDB was
compiled with Python 3.6 or earlier.

@item 
bpo-34587@footnote{https://bugs.python.org/issue34587}: test_socket: Remove RDSTest.testCongestion(). The test tries to
fill the receiver’s socket buffer and expects an error. But the RDS
protocol doesn’t require that. Moreover, the Linux implementation of RDS
expects that the producer of the messages reduces its rate, it’s not the
role of the receiver to trigger an error. The test fails on Fedora 28 by
design, so just remove it.

@item 
bpo-34661@footnote{https://bugs.python.org/issue34661}: Fix test_shutil if unzip doesn’t support -t.

@item 
bpo-34200@footnote{https://bugs.python.org/issue34200}: Fixed non-deterministic flakiness of test_pkg by not using the
scary test.support.module_cleanup() logic to save and restore sys.modules
contents between test cases.

@item 
bpo-34569@footnote{https://bugs.python.org/issue34569}: The experimental PEP 554 data channels now correctly pass
negative PyLong objects between subinterpreters on 32-bit systems. Patch
by Michael Felt.

@item 
bpo-34594@footnote{https://bugs.python.org/issue34594}: Fix usage of hardcoded @code{errno} values in the tests.

@item 
bpo-34579@footnote{https://bugs.python.org/issue34579}: Fix test_embed for AIX Patch by Michael Felt

@item 
bpo-34542@footnote{https://bugs.python.org/issue34542}: Use 3072 RSA keys and SHA-256 signature for test certs and
keys.

@item 
bpo-11193@footnote{https://bugs.python.org/issue11193}: Remove special condition for AIX in
@code{test_subprocess.test_undecodable_env}

@item 
bpo-34347@footnote{https://bugs.python.org/issue34347}: Fix @code{test_utf8_mode.test_cmd_line} for AIX

@item 
bpo-34490@footnote{https://bugs.python.org/issue34490}: On AIX with AF_UNIX family sockets getsockname() does not
provide ‘sockname’, so skip calls to transport.get_extra_info(‘sockname’)

@item 
bpo-34391@footnote{https://bugs.python.org/issue34391}: Fix ftplib test for TLS 1.3 by reading from data socket.

@item 
bpo-11192@footnote{https://bugs.python.org/issue11192}: Fix @code{test_socket} on AIX 6.1 and later IPv6 zone id supports
only supported by inet_pton6_zone() Switch to runtime-based
platform.system() to establish current platform  rather than build-time
based sys.platform()

@item 
bpo-34399@footnote{https://bugs.python.org/issue34399}: Update all RSA keys and DH params to use at least 2048 bits.

@item 
bpo-34373@footnote{https://bugs.python.org/issue34373}: Fix @code{test_mktime} and @code{test_pthread_getcpuclickid} tests
for AIX Add range checking for @code{_PyTime_localtime} for AIX Patch by
Michael Felt

@item 
bpo-11191@footnote{https://bugs.python.org/issue11191}: Skip the distutils test ‘test_search_cpp’ when using XLC as
compiler patch by aixtools (Michael Felt)

@item 
Improved an error message when mock assert_has_calls fails.

@item 
bpo-33746@footnote{https://bugs.python.org/issue33746}: Fix test_unittest when run in verbose mode.

@item 
bpo-33901@footnote{https://bugs.python.org/issue33901}: Fix test_dbm_gnu on macOS with gdbm 1.15: add a larger value to
make sure that the file size changes.

@item 
bpo-33873@footnote{https://bugs.python.org/issue33873}: Fix a bug in @code{regrtest} that caused an extra test to run if
–huntrleaks/-R was used. Exit with error in case that invalid parameters
are specified to –huntrleaks/-R (at least one warmup run and one
repetition must be used).

@item 
bpo-33562@footnote{https://bugs.python.org/issue33562}: Check that a global asyncio event loop policy is not left
behind by any tests.

@item 
bpo-33655@footnote{https://bugs.python.org/issue33655}: Ignore test_posix_fallocate failures on BSD platforms that
might be due to running on ZFS.

@item 
bpo-32962@footnote{https://bugs.python.org/issue32962}: Fixed test_gdb when Python is compiled with flags -mcet
-fcf-protection -O0.

@item 
bpo-33358@footnote{https://bugs.python.org/issue33358}: Fix @code{test_embed.test_pre_initialization_sys_options()} when
the interpreter is built with @code{--enable-shared}.

@item 
bpo-32872@footnote{https://bugs.python.org/issue32872}: Avoid regrtest compatibility issue with namespace packages.

@item 
bpo-32517@footnote{https://bugs.python.org/issue32517}: Fix failing @code{test_asyncio} on macOS 10.12.2+ due to transport
of @code{KqueueSelector} loop was not being closed.

@item 
bpo-32663@footnote{https://bugs.python.org/issue32663}: Making sure the @code{SMTPUTF8SimTests} class of tests gets run in
test_smtplib.py.

@item 
bpo-27643@footnote{https://bugs.python.org/issue27643}: Test_C test case needs “signed short” bitfields, but the IBM
XLC compiler (on AIX) does not support this Skip the code and test when
AIX and XLC are used

Applicable to Python2-2.7 and later

@item 
bpo-19417@footnote{https://bugs.python.org/issue19417}: Add test_bdb.py.

@item 
bpo-31809@footnote{https://bugs.python.org/issue31809}: Add tests to verify connection with secp ECDH curves.
@end itemize

@node Build<25>,Windows<24>,Tests<25>,Python 3 8 0 alpha 1
@anchor{whatsnew/changelog id233}@anchor{1307}
@subsubsection Build



@itemize -

@item 
bpo-34691@footnote{https://bugs.python.org/issue34691}: The _contextvars module is now built into the core Python
library on Windows.

@item 
bpo-35683@footnote{https://bugs.python.org/issue35683}: Improved Azure Pipelines build steps and now verifying layouts
correctly

@item 
bpo-35642@footnote{https://bugs.python.org/issue35642}: Remove asynciomodule.c from pythoncore.vcxproj

@item 
bpo-35550@footnote{https://bugs.python.org/issue35550}: Fix incorrect Solaris #ifdef checks to look for __sun && __SVR4
instead of sun when compiling.

@item 
bpo-35499@footnote{https://bugs.python.org/issue35499}: @code{make profile-opt} no longer replaces @code{CFLAGS_NODIST} with
@code{CFLAGS}. It now adds profile-guided optimization (PGO) flags to
@code{CFLAGS_NODIST}: existing @code{CFLAGS_NODIST} flags are kept.

@item 
bpo-35257@footnote{https://bugs.python.org/issue35257}: Avoid leaking the linker flags from Link Time Optimizations
(LTO) into distutils when compiling C extensions.

@item 
bpo-35351@footnote{https://bugs.python.org/issue35351}: When building Python with clang and LTO, LTO flags are no
longer passed into CFLAGS to build third-party C extensions through
distutils.

@item 
bpo-35139@footnote{https://bugs.python.org/issue35139}: Fix a compiler error when statically linking @code{pyexpat} in
@code{Modules/Setup}.

@item 
bpo-35059@footnote{https://bugs.python.org/issue35059}: PCbuild: Set InlineFunctionExpansion to OnlyExplicitInline
(“/Ob1” option) in pyproject.props in Debug mode to expand functions
marked as inline. This change should make Python compiled in Debug mode a
little bit faster on Windows.

@item 
bpo-35011@footnote{https://bugs.python.org/issue35011}: Restores the use of pyexpatns.h to isolate our embedded copy of
the expat C library so that its symbols do not conflict at link or dynamic
loading time with an embedding application or other extension modules with
their own version of libexpat.

@item 
bpo-28015@footnote{https://bugs.python.org/issue28015}: Have –with-lto works correctly with clang.

@item 
bpo-34765@footnote{https://bugs.python.org/issue34765}: Update the outdated install-sh file to the latest revision from
automake v1.16.1

@item 
bpo-34585@footnote{https://bugs.python.org/issue34585}: Check for floating-point byte order in configure.ac using
compilation tests instead of executing code, so that these checks work in
cross-compiled builds.

@item 
bpo-34710@footnote{https://bugs.python.org/issue34710}: Fixed SSL module build with OpenSSL & pedantic CFLAGS.

@item 
bpo-34582@footnote{https://bugs.python.org/issue34582}: Add JUnit XML output for regression tests and update Azure
DevOps builds.

@item 
bpo-34081@footnote{https://bugs.python.org/issue34081}: Make Sphinx warnings as errors in the Docs Makefile.

@item 
bpo-34555@footnote{https://bugs.python.org/issue34555}: Fix for case where it was not possible to have both
@code{HAVE_LINUX_VM_SOCKETS_H} and @code{HAVE_SOCKADDR_ALG} be undefined.

@item 
bpo-33015@footnote{https://bugs.python.org/issue33015}: Fix an undefined behaviour in the pthread implementation of
@code{PyThread_start_new_thread()}: add a function wrapper to always
return @code{NULL}.

@item 
bpo-34245@footnote{https://bugs.python.org/issue34245}: The Python shared library is now installed with write
permission (mode 0755), which is the standard way of installing such
libraries.

@item 
bpo-34121@footnote{https://bugs.python.org/issue34121}: Fix detection of C11 atomic support on clang.

@item 
bpo-32430@footnote{https://bugs.python.org/issue32430}: Rename Modules/Setup.dist to Modules/Setup, and remove the
necessity to copy the former manually to the latter when updating the
local source tree.

@item 
bpo-30345@footnote{https://bugs.python.org/issue30345}: Add -g to LDFLAGS when compiling with LTO to get debug symbols.

@item 
bpo-5755@footnote{https://bugs.python.org/issue5755}: Move @code{-Wstrict-prototypes} option to @code{CFLAGS_NODIST} from
@code{OPT}. This option emitted annoying warnings when building extension
modules written in C++.

@item 
bpo-33614@footnote{https://bugs.python.org/issue33614}: Ensures module definition files for the stable ABI on Windows
are correctly regenerated.

@item 
bpo-33648@footnote{https://bugs.python.org/issue33648}: The –with-c-locale-warning configuration flag has been
removed. It has had no effect for about a year.

@item 
bpo-33522@footnote{https://bugs.python.org/issue33522}: Enable CI builds on Visual Studio Team Services at
@indicateurl{https://python.visualstudio.com/cpython}

@item 
bpo-33512@footnote{https://bugs.python.org/issue33512}: configure’s check for “long double” has been simplified

@item 
bpo-33483@footnote{https://bugs.python.org/issue33483}: C compiler is now correctly detected from the standard
environment variables. –without-gcc and –with-icc options have been
removed.

@item 
bpo-33394@footnote{https://bugs.python.org/issue33394}: Enable the verbose build for extension modules, when GNU make
is passed macros on the command line.

@item 
bpo-33393@footnote{https://bugs.python.org/issue33393}: Update config.guess and config.sub files.

@item 
bpo-33377@footnote{https://bugs.python.org/issue33377}: Add new triplets for mips r6 and riscv variants (used in
extension suffixes).

@item 
bpo-32232@footnote{https://bugs.python.org/issue32232}: By default, modules configured in @code{Modules/Setup} are no longer
built with @code{-DPy_BUILD_CORE}. Instead, modules that specifically need that
preprocessor definition include it in their individual entries.

@item 
bpo-33182@footnote{https://bugs.python.org/issue33182}: The embedding tests can once again be built with clang 6.0

@item 
bpo-33163@footnote{https://bugs.python.org/issue33163}: Upgrade pip to 9.0.3 and setuptools to v39.0.1.

@item 
bpo-33012@footnote{https://bugs.python.org/issue33012}: gcc 8 has added a new warning heuristic to detect invalid
function casts and a stock python build seems to hit that warning quite
often.  The most common is the cast of a METH_NOARGS function (that uses
just one argument) to a PyCFunction. Fix this by adding a dummy argument
to all functions that implement METH_NOARGS.

@item 
bpo-32898@footnote{https://bugs.python.org/issue32898}: Fix the python debug build when using COUNT_ALLOCS.

@item 
bpo-29442@footnote{https://bugs.python.org/issue29442}: Replace optparse with argparse in setup.py
@end itemize

@node Windows<24>,macOS<20>,Build<25>,Python 3 8 0 alpha 1
@anchor{whatsnew/changelog id234}@anchor{1308}
@subsubsection Windows



@itemize -

@item 
bpo-35890@footnote{https://bugs.python.org/issue35890}: Fix API calling consistency of GetVersionEx and wcstok.

@item 
bpo-32560@footnote{https://bugs.python.org/issue32560}: The @code{py} launcher now forwards its @code{STARTUPINFO} structure
to child processes.

@item 
bpo-35854@footnote{https://bugs.python.org/issue35854}: Fix EnvBuilder and –symlinks in venv on Windows

@item 
bpo-35811@footnote{https://bugs.python.org/issue35811}: Avoid propagating venv settings when launching via py.exe

@item 
bpo-35797@footnote{https://bugs.python.org/issue35797}: Fix default executable used by the multiprocessing module

@item 
bpo-35758@footnote{https://bugs.python.org/issue35758}: Allow building on ARM with MSVC.

@item 
bpo-29734@footnote{https://bugs.python.org/issue29734}: Fix handle leaks in os.stat on Windows.

@item 
bpo-35596@footnote{https://bugs.python.org/issue35596}: Use unchecked PYCs for the embeddable distro to avoid zipimport
restrictions.

@item 
bpo-35596@footnote{https://bugs.python.org/issue35596}: Fix vcruntime140.dll being added to embeddable distro multiple
times.

@item 
bpo-35402@footnote{https://bugs.python.org/issue35402}: Update Windows build to use Tcl and Tk 8.6.9

@item 
bpo-35401@footnote{https://bugs.python.org/issue35401}: Updates Windows build to OpenSSL 1.1.0j

@item 
bpo-34977@footnote{https://bugs.python.org/issue34977}: venv on Windows will now use a python.exe redirector rather
than copying the actual binaries from the base environment.

@item 
bpo-34977@footnote{https://bugs.python.org/issue34977}: Adds support for building a Windows App Store package

@item 
bpo-35067@footnote{https://bugs.python.org/issue35067}: Remove _distutils_findvs module and use vswhere.exe instead.

@item 
bpo-32557@footnote{https://bugs.python.org/issue32557}: Allow shutil.disk_usage to take a file path on Windows

@item 
bpo-34770@footnote{https://bugs.python.org/issue34770}: Fix a possible null pointer dereference in pyshellext.cpp.

@item 
bpo-34603@footnote{https://bugs.python.org/issue34603}: Fix returning structs from functions produced by MSVC

@item 
bpo-34581@footnote{https://bugs.python.org/issue34581}: Guard MSVC-specific code in socketmodule.c with @code{#ifdef
_MSC_VER}.

@item 
bpo-34532@footnote{https://bugs.python.org/issue34532}: Fixes exit code of list version arguments for py.exe.

@item 
bpo-34062@footnote{https://bugs.python.org/issue34062}: Fixed the ‘–list’ and ‘–list-paths’ arguments for the py.exe
launcher

@item 
bpo-34225@footnote{https://bugs.python.org/issue34225}: Ensure INCLUDE and LIB directories do not end with a backslash.

@item 
bpo-34011@footnote{https://bugs.python.org/issue34011}: A suite of code has been changed which copied across DLLs and
init.tcl from the running Python location into a venv being created. These
copies are needed only when running from a Python source build, and the
copying code is now only run when that is the case, rather than whenever a
venv is created.

@item 
bpo-34006@footnote{https://bugs.python.org/issue34006}: Revert line length limit for Windows help docs. The line-length
limit is not needed because the pages appear in a separate app rather than
on a browser tab.  It can also interact badly with the DPI setting.

@item 
bpo-31546@footnote{https://bugs.python.org/issue31546}: Restore running PyOS_InputHook while waiting for user input at
the prompt. The restores integration of interactive GUI windows (such as
Matplotlib figures) with the prompt on Windows.

@item 
bpo-30237@footnote{https://bugs.python.org/issue30237}: Output error when ReadConsole is canceled by
CancelSynchronousIo instead of crashing.

@item 
bpo-33895@footnote{https://bugs.python.org/issue33895}: GIL is released while calling functions that acquire Windows
loader lock.

@item 
bpo-33720@footnote{https://bugs.python.org/issue33720}: Reduces maximum marshal recursion depth on release builds.

@item 
bpo-29097@footnote{https://bugs.python.org/issue29097}: Fix bug where @code{datetime.fromtimestamp()} erroneously throws
an @ref{2a1,,OSError} on Windows for values between 0 and 86400. Patch by
Ammar Askar.

@item 
bpo-33316@footnote{https://bugs.python.org/issue33316}: PyThread_release_lock always fails

@item 
bpo-33184@footnote{https://bugs.python.org/issue33184}: Update Windows installer to use OpenSSL 1.1.0h.

@item 
bpo-32890@footnote{https://bugs.python.org/issue32890}: Fix usage of GetLastError() instead of errno in os.execve() and
os.truncate().

@item 
bpo-33016@footnote{https://bugs.python.org/issue33016}: Fix potential use of uninitialized memory in
nt._getfinalpathname

@item 
bpo-32903@footnote{https://bugs.python.org/issue32903}: Fix a memory leak in os.chdir() on Windows if the current
directory is set to a UNC path.

@item 
bpo-32901@footnote{https://bugs.python.org/issue32901}: Update Tcl and Tk versions to 8.6.8

@item 
bpo-31966@footnote{https://bugs.python.org/issue31966}: Fixed WindowsConsoleIO.write() for writing empty data.

@item 
bpo-32409@footnote{https://bugs.python.org/issue32409}: Ensures activate.bat can handle Unicode contents.

@item 
bpo-32457@footnote{https://bugs.python.org/issue32457}: Improves handling of denormalized executable path when
launching Python.

@item 
bpo-32370@footnote{https://bugs.python.org/issue32370}: Use the correct encoding for ipconfig output in the uuid
module. Patch by Segev Finer.

@item 
bpo-29248@footnote{https://bugs.python.org/issue29248}: Fix @ref{414,,os.readlink()} on Windows, which was mistakenly
treating the @code{PrintNameOffset} field of the reparse data buffer as a
number of characters instead of bytes. Patch by Craig Holmquist and SSE4.

@item 
bpo-1104@footnote{https://bugs.python.org/issue1104}: Correctly handle string length in
@code{msilib.SummaryInfo.GetProperty()} to prevent it from truncating the
last character.
@end itemize

@node macOS<20>,IDLE<24>,Windows<24>,Python 3 8 0 alpha 1
@anchor{whatsnew/changelog id235}@anchor{1309}
@subsubsection macOS



@itemize -

@item 
bpo-35401@footnote{https://bugs.python.org/issue35401}: Update macOS installer to use OpenSSL 1.1.0j.

@item 
bpo-35025@footnote{https://bugs.python.org/issue35025}: Properly guard the use of the @code{CLOCK_GETTIME} et al. macros
in @code{timemodule} on macOS.

@item 
bpo-24658@footnote{https://bugs.python.org/issue24658}: On macOS, fix reading from and writing into a file with a size
larger than 2 GiB.

@item 
bpo-34405@footnote{https://bugs.python.org/issue34405}: Update to OpenSSL 1.1.0i for macOS installer builds.

@item 
bpo-33635@footnote{https://bugs.python.org/issue33635}: In macOS stat on some file descriptors (/dev/fd/3 f.e) will
result in bad file descriptor OSError. Guard against this exception was
added in is_dir, is_file and similar methods. DirEntry.is_dir can also
throw this exception so _RecursiveWildcardSelector._iterate_directories
was also extended with the same error ignoring pattern.

@item 
bpo-13631@footnote{https://bugs.python.org/issue13631}: The .editrc file in user’s home directory is now processed
correctly during the readline initialization through editline emulation on
macOS.

@item 
bpo-33184@footnote{https://bugs.python.org/issue33184}: Update macOS installer build to use OpenSSL 1.1.0h.

@item 
bpo-32726@footnote{https://bugs.python.org/issue32726}: Build and link with private copy of Tcl/Tk 8.6 for the macOS
10.6+ installer. The 10.9+ installer variant already does this.  This
means that the Python 3.7 provided by the python.org macOS installers no
longer need or use any external versions of Tcl/Tk, either system-provided
or user-installed, such as ActiveTcl.

@item 
bpo-32901@footnote{https://bugs.python.org/issue32901}: Update macOS 10.9+ installer to Tcl/Tk 8.6.8.

@item 
bpo-31903@footnote{https://bugs.python.org/issue31903}: In @code{_scproxy}, drop the GIL when calling into
@code{SystemConfiguration} to avoid deadlocks.
@end itemize

@node IDLE<24>,Tools/Demos<11>,macOS<20>,Python 3 8 0 alpha 1
@anchor{whatsnew/changelog id236}@anchor{130a}
@subsubsection IDLE



@itemize -

@item 
bpo-35770@footnote{https://bugs.python.org/issue35770}: IDLE macosx deletes Options => Configure IDLE. It previously
deleted Window => Zoom Height by mistake. (Zoom Height is now on the
Options menu).  On Mac, the settings dialog is accessed via Preferences on
the IDLE menu.

@item 
bpo-35769@footnote{https://bugs.python.org/issue35769}: Change IDLE’s new file name from ‘Untitled’ to ‘untitled’

@item 
bpo-35660@footnote{https://bugs.python.org/issue35660}: Fix imports in idlelib.window.

@item 
bpo-35641@footnote{https://bugs.python.org/issue35641}: Proper format @code{calltip} when the function has no docstring.

@item 
bpo-33987@footnote{https://bugs.python.org/issue33987}: Use ttk Frame for ttk widgets.

@item 
bpo-34055@footnote{https://bugs.python.org/issue34055}: Fix erroneous ‘smart’ indents and newlines in IDLE Shell.

@item 
bpo-35591@footnote{https://bugs.python.org/issue35591}: Find Selection now works when selection not found.

@item 
bpo-35196@footnote{https://bugs.python.org/issue35196}: Speed up squeezer line counting.

@item 
bpo-35598@footnote{https://bugs.python.org/issue35598}: Update config_key: use PEP 8 names and ttk widgets, make some
objects global, and add tests.

@item 
bpo-28097@footnote{https://bugs.python.org/issue28097}: Add Previous/Next History entries to Shell menu.

@item 
bpo-35208@footnote{https://bugs.python.org/issue35208}: Squeezer now properly counts wrapped lines before newlines.

@item 
bpo-35555@footnote{https://bugs.python.org/issue35555}: Gray out Code Context menu entry when it’s not applicable.

@item 
bpo-35521@footnote{https://bugs.python.org/issue35521}: Document the IDLE editor code context feature. Add some
internal references within the IDLE doc.

@item 
bpo-22703@footnote{https://bugs.python.org/issue22703}: The Code Context menu label now toggles between Show/Hide Code
Context. The Zoom Height menu now toggles between Zoom/Restore Height.
Zoom Height has moved from the Window menu to the Options menu.

@item 
bpo-35213@footnote{https://bugs.python.org/issue35213}: Where appropriate, use ‘macOS’ in idlelib.

@item 
bpo-34864@footnote{https://bugs.python.org/issue34864}: On macOS, warn if the system preference “Prefer tabs when
opening documents” is set to “Always”.

@item 
bpo-34864@footnote{https://bugs.python.org/issue34864}: Document two IDLE on MacOS issues. The System Preferences Dock
“prefer tabs always” setting disables some IDLE features.  Menus are a bit
different than as described for Windows and Linux.

@item 
bpo-35202@footnote{https://bugs.python.org/issue35202}: Remove unused imports from lib/idlelib

@item 
bpo-33000@footnote{https://bugs.python.org/issue33000}: Document that IDLE’s shell has no line limit. A program that
runs indefinitely can overfill memory.

@item 
bpo-23220@footnote{https://bugs.python.org/issue23220}: Explain how IDLE’s Shell displays output.

@item 
bpo-35099@footnote{https://bugs.python.org/issue35099}: Improve the doc about IDLE running user code.   The section is
renamed from “IDLE – console differences” is renamed “Running user code”.
It mostly covers the implications of using custom sys.stdxxx objects.

@item 
bpo-35097@footnote{https://bugs.python.org/issue35097}: Add IDLE doc subsection explaining editor windows. Topics
include opening, title and status bar, .py* extension, and running.

@item 
bpo-35093@footnote{https://bugs.python.org/issue35093}: Document the IDLE document viewer in the IDLE doc. Add a
paragraph in “Help and preferences”, “Help sources” subsection.

@item 
bpo-35088@footnote{https://bugs.python.org/issue35088}: Update idlelib.help.copy_string docstring. We now use git and
backporting instead of hg and forward merging.

@item 
bpo-35087@footnote{https://bugs.python.org/issue35087}: Update idlelib help files for the current doc build. The main
change is the elimination of chapter-section numbers.

@item 
bpo-34548@footnote{https://bugs.python.org/issue34548}: Use configured color theme for read-only text views.

@item 
bpo-1529353@footnote{https://bugs.python.org/issue1529353}: Enable “squeezing” of long outputs in the shell, to avoid
performance degradation and to clean up the history without losing it.
Squeezed outputs may be copied, viewed in a separate window, and
“unsqueezed”.

@item 
bpo-34047@footnote{https://bugs.python.org/issue34047}: Fixed mousewheel scrolling direction on macOS.

@item 
bpo-34275@footnote{https://bugs.python.org/issue34275}: Make IDLE calltips always visible on Mac. Some MacOS-tk
combinations need .update_idletasks(). Patch by Kevin Walzer.

@item 
bpo-34120@footnote{https://bugs.python.org/issue34120}: Fix unresponsiveness after closing certain windows and dialogs.

@item 
bpo-33975@footnote{https://bugs.python.org/issue33975}: Avoid small type when running htests. Since part of the purpose
of human-viewed tests is to determine that widgets look right, it is
important that they look the same for testing as when running IDLE.

@item 
bpo-33905@footnote{https://bugs.python.org/issue33905}: Add test for idlelib.stackview.StackBrowser.

@item 
bpo-33924@footnote{https://bugs.python.org/issue33924}: Change mainmenu.menudefs key ‘windows’ to ‘window’. Every other
menudef key is lowercase version of main menu entry.

@item 
bpo-33906@footnote{https://bugs.python.org/issue33906}: Rename idlelib.windows as window Match Window on the main menu
and remove last plural module name.

@item 
bpo-33917@footnote{https://bugs.python.org/issue33917}: Fix and document idlelib/idle_test/template.py. The revised
file compiles, runs, and tests OK.  idle_test/README.txt explains how to
use it to create new IDLE test files.

@item 
bpo-33904@footnote{https://bugs.python.org/issue33904}: IDLE: In rstrip, rename class RstripExtension as Rstrip

@item 
bpo-33907@footnote{https://bugs.python.org/issue33907}: For consistency and clarity, rename an IDLE module and classes.
Module calltips and its class CallTips are now calltip and Calltip. In
module calltip_w, class CallTip is now CalltipWindow.

@item 
bpo-33856@footnote{https://bugs.python.org/issue33856}: Add “help” in the welcome message of IDLE

@item 
bpo-33839@footnote{https://bugs.python.org/issue33839}: IDLE: refactor ToolTip and CallTip and add documentation and
tests

@item 
bpo-33855@footnote{https://bugs.python.org/issue33855}: Minimally test all IDLE modules. Add missing files, import
module, instantiate classes, and check coverage. Check existing files.

@item 
bpo-33656@footnote{https://bugs.python.org/issue33656}: On Windows, add API call saying that tk scales for DPI. On
Windows 8.1+ or 10, with DPI compatibility properties of the Python binary
unchanged, and a monitor resolution greater than 96 DPI, this should make
text and lines sharper.  It should otherwise have no effect.

@item 
bpo-33768@footnote{https://bugs.python.org/issue33768}: Clicking on a context line moves that line to the top of the
editor window.

@item 
bpo-33763@footnote{https://bugs.python.org/issue33763}: IDLE: Use read-only text widget for code context instead of
label widget.

@item 
bpo-33664@footnote{https://bugs.python.org/issue33664}: Scroll IDLE editor text by lines. Previously, the mouse wheel
and scrollbar slider moved text by a fixed number of pixels, resulting in
partial lines at the top of the editor box.  The change also applies to
the shell and grep output windows, but not to read-only text views.

@item 
bpo-33679@footnote{https://bugs.python.org/issue33679}: Enable theme-specific color configuration for Code Context. Use
the Highlights tab to see the setting for built-in themes or add settings
to custom themes.

@item 
bpo-33642@footnote{https://bugs.python.org/issue33642}: Display up to maxlines non-blank lines for Code Context. If
there is no current context, show a single blank line.

@item 
bpo-33628@footnote{https://bugs.python.org/issue33628}: IDLE: Cleanup codecontext.py and its test.

@item 
bpo-33564@footnote{https://bugs.python.org/issue33564}: IDLE’s code context now recognizes async as a block opener.

@item 
bpo-21474@footnote{https://bugs.python.org/issue21474}: Update word/identifier definition from ascii to unicode. In
text and entry boxes, this affects selection by double-click, movement
left/right by control-left/right, and deletion left/right by
control-BACKSPACE/DEL.

@item 
bpo-33204@footnote{https://bugs.python.org/issue33204}: IDLE: consistently color invalid string prefixes. A ‘u’ string
prefix cannot be paired with either ‘r’ or ‘f’. Consistently color as much
of the prefix, starting at the right, as is valid. Revise and extend
colorizer test.

@item 
bpo-32984@footnote{https://bugs.python.org/issue32984}: Set @code{__file__} while running a startup file.  Like Python,
IDLE optionally runs one startup file in the Shell window before
presenting the first interactive input prompt.  For IDLE, @code{-s} runs a
file named in environmental variable  
@geindex IDLESTARTUP
@geindex environment variable; IDLESTARTUP
@code{IDLESTARTUP} or
@geindex PYTHONSTARTUP
@geindex environment variable; PYTHONSTARTUP
@ref{b4d,,PYTHONSTARTUP}; @code{-r file} runs @code{file}.  Python sets
@code{__file__} to the startup file name before running the file and unsets
it before the first prompt.  IDLE now does the same when run normally,
without the @code{-n} option.

@item 
bpo-32940@footnote{https://bugs.python.org/issue32940}: Simplify and rename StringTranslatePseudoMapping in pyparse.

@item 
bpo-32916@footnote{https://bugs.python.org/issue32916}: Change @code{str} to @code{code} in pyparse.

@item 
bpo-32905@footnote{https://bugs.python.org/issue32905}: Remove unused code in pyparse module.

@item 
bpo-32874@footnote{https://bugs.python.org/issue32874}: Add tests for pyparse.

@item 
bpo-32837@footnote{https://bugs.python.org/issue32837}: Using the system and place-dependent default encoding for
open() is a bad idea for IDLE’s system and location-independent files.

@item 
bpo-32826@footnote{https://bugs.python.org/issue32826}: Add “encoding=utf-8” to open() in IDLE’s test_help_about. GUI
test test_file_buttons() only looks at initial ascii-only lines, but
failed on systems where open() defaults to ‘ascii’ because readline()
internally reads and decodes far enough ahead to encounter a non-ascii
character in CREDITS.txt.

@item 
bpo-32831@footnote{https://bugs.python.org/issue32831}: Add docstrings and tests for codecontext.

@item 
bpo-32765@footnote{https://bugs.python.org/issue32765}: Update configdialog General tab docstring to add new widgets to
the widget list.
@end itemize

@node Tools/Demos<11>,C API<25>,IDLE<24>,Python 3 8 0 alpha 1
@anchor{whatsnew/changelog id237}@anchor{130b}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-35884@footnote{https://bugs.python.org/issue35884}: Add a benchmark script for timing various ways to access
variables: @code{Tools/scripts/var_access_benchmark.py}.

@item 
bpo-34989@footnote{https://bugs.python.org/issue34989}: python-gdb.py now handles errors on computing the line number
of a Python frame.

@item 
bpo-20260@footnote{https://bugs.python.org/issue20260}: Argument Clinic now has non-bitwise unsigned int converters.

@item 
bpo-32962@footnote{https://bugs.python.org/issue32962}: python-gdb now catches @code{UnicodeDecodeError} exceptions when
calling @code{string()}.

@item 
bpo-32962@footnote{https://bugs.python.org/issue32962}: python-gdb now catches ValueError on read_var(): when Python
has no debug symbols for example.

@item 
bpo-33189@footnote{https://bugs.python.org/issue33189}: @code{pygettext.py} now recognizes only literal strings as
docstrings and translatable strings, and rejects bytes literals and
f-string expressions.

@item 
bpo-31920@footnote{https://bugs.python.org/issue31920}: Fixed handling directories as arguments in the @code{pygettext}
script. Based on patch by Oleg Krasnikov.

@item 
bpo-29673@footnote{https://bugs.python.org/issue29673}: Fix pystackv and pystack gdbinit macros.

@item 
bpo-25427@footnote{https://bugs.python.org/issue25427}: Remove the pyvenv script in favor of @code{python3 -m venv} in
order to lower confusion as to what Python interpreter a virtual
environment will be created for.

@item 
bpo-32885@footnote{https://bugs.python.org/issue32885}: Add an @code{-n} flag for @code{Tools/scripts/pathfix.py} to disable
automatic backup creation (files with @code{~} suffix).

@item 
bpo-32222@footnote{https://bugs.python.org/issue32222}: Fix pygettext not extracting docstrings for functions with type
annotated arguments. Patch by Toby Harradine.

@item 
bpo-31583@footnote{https://bugs.python.org/issue31583}: Fix 2to3 for using with –add-suffix option but without
–output-dir option for relative path to files in current directory.
@end itemize

@node C API<25>,,Tools/Demos<11>,Python 3 8 0 alpha 1
@anchor{whatsnew/changelog id238}@anchor{130c}
@subsubsection C API



@itemize -

@item 
bpo-35713@footnote{https://bugs.python.org/issue35713}: The @code{PyByteArray_Init()} and @code{PyByteArray_Fini()}
functions have been removed. They did nothing since Python 2.7.4 and
Python 3.2.0, were excluded from the limited API (stable ABI), and were
not documented.

@item 
bpo-33817@footnote{https://bugs.python.org/issue33817}: Fixed @ref{130d,,_PyBytes_Resize()} for empty bytes objects.

@item 
bpo-35322@footnote{https://bugs.python.org/issue35322}: Fix memory leak in @ref{701,,PyUnicode_EncodeLocale()} and
@ref{130e,,PyUnicode_EncodeFSDefault()} on error handling.

@item 
bpo-35059@footnote{https://bugs.python.org/issue35059}: The following C macros have been converted to static inline
functions: @ref{372,,Py_INCREF()}, @ref{373,,Py_DECREF()}, @ref{57b,,Py_XINCREF()},
@ref{1fd,,Py_XDECREF()}, @code{PyObject_INIT()},
@code{PyObject_INIT_VAR()}.

@item 
bpo-35296@footnote{https://bugs.python.org/issue35296}: @code{make install} now also installs the internal API:
@code{Include/internal/*.h} header files.

@item 
bpo-35081@footnote{https://bugs.python.org/issue35081}: Internal APIs surrounded by @code{#ifdef Py_BUILD_CORE} have been
moved from @code{Include/*.h} headers to new header files
@code{Include/internal/pycore_*.h}.

@item 
bpo-35259@footnote{https://bugs.python.org/issue35259}: Conditionally declare @ref{862,,Py_FinalizeEx()} (new in 3.6)
based on Py_LIMITED_API. Patch by Arthur Neufeld.

@item 
bpo-35081@footnote{https://bugs.python.org/issue35081}: The @code{_PyObject_GC_TRACK()} and
@code{_PyObject_GC_UNTRACK()} macros have been removed from the public C
API.

@item 
bpo-35134@footnote{https://bugs.python.org/issue35134}: Creation of a new @code{Include/cpython/} subdirectory.

@item 
bpo-34725@footnote{https://bugs.python.org/issue34725}: Adds _Py_SetProgramFullPath so embedders may override
sys.executable

@item 
bpo-34910@footnote{https://bugs.python.org/issue34910}: Ensure that @ref{130f,,PyObject_Print()} always returns @code{-1} on
error.  Patch by Zackery Spytz.

@item 
bpo-34523@footnote{https://bugs.python.org/issue34523}: Py_DecodeLocale() and Py_EncodeLocale() now use the UTF-8
encoding on Windows if Py_LegacyWindowsFSEncodingFlag is zero.

@item 
bpo-34193@footnote{https://bugs.python.org/issue34193}: Fix pluralization in TypeError messages in getargs.c and
typeobject.c: ‘1 argument’ instead of ‘1 arguments’ and ‘1 element’
instead of ‘1 elements’.

@item 
bpo-34127@footnote{https://bugs.python.org/issue34127}: Return grammatically correct error message based on argument
count. Patch by Karthikeyan Singaravelan.

@item 
bpo-23927@footnote{https://bugs.python.org/issue23927}: Fixed @ref{891,,SystemError} in
@ref{863,,PyArg_ParseTupleAndKeywords()} when the @code{w*} format unit is used
for optional parameter.

@item 
bpo-32455@footnote{https://bugs.python.org/issue32455}: Added @code{PyCompile_OpcodeStackEffectWithJump()}.

@item 
bpo-34008@footnote{https://bugs.python.org/issue34008}: Py_Main() can again be called after Py_Initialize(), as in
Python 3.6.

@item 
bpo-32500@footnote{https://bugs.python.org/issue32500}: Fixed error messages for @ref{1310,,PySequence_Size()},
@ref{1311,,PySequence_GetItem()}, @ref{1312,,PySequence_SetItem()} and
@ref{1313,,PySequence_DelItem()} called with a mapping and
@ref{1314,,PyMapping_Size()} called with a sequence.

@item 
bpo-33818@footnote{https://bugs.python.org/issue33818}: @code{PyExceptionClass_Name()} will now return @code{const char
*} instead of @code{char *}.

@item 
bpo-33042@footnote{https://bugs.python.org/issue33042}: Embedding applications may once again call
PySys_ResetWarnOptions, PySys_AddWarnOption, and PySys_AddXOption prior to
calling Py_Initialize.

@item 
bpo-32374@footnote{https://bugs.python.org/issue32374}: Document that m_traverse for multi-phase initialized modules
can be called with m_state=NULL, and add a sanity check

@item 
bpo-30863@footnote{https://bugs.python.org/issue30863}: @ref{1315,,PyUnicode_AsWideChar()} and
@ref{39f,,PyUnicode_AsWideCharString()} no longer cache the @code{wchar_t*}
representation of string objects.
@end itemize

@node Python 3 7 0 final,Python 3 7 0 release candidate 1,Python 3 8 0 alpha 1,Changelog
@anchor{whatsnew/changelog python-3-7-0-final}@anchor{1316}
@subsection Python 3.7.0 final


@emph{Release date: 2018-06-27}

@menu
* Library: Library<27>. 
* C API: C API<26>. 

@end menu

@node Library<27>,C API<26>,,Python 3 7 0 final
@anchor{whatsnew/changelog id239}@anchor{1317}
@subsubsection Library



@itemize -

@item 
bpo-33851@footnote{https://bugs.python.org/issue33851}: Fix @ref{1318,,ast.get_docstring()} for a node that lacks a
docstring.
@end itemize

@node C API<26>,,Library<27>,Python 3 7 0 final
@anchor{whatsnew/changelog id240}@anchor{1319}
@subsubsection C API



@itemize -

@item 
bpo-33932@footnote{https://bugs.python.org/issue33932}: Calling Py_Initialize() twice does nothing, instead of failing
with a fatal error: restore the Python 3.6 behaviour.
@end itemize

@node Python 3 7 0 release candidate 1,Python 3 7 0 beta 5,Python 3 7 0 final,Changelog
@anchor{whatsnew/changelog python-3-7-0-release-candidate-1}@anchor{131a}
@subsection Python 3.7.0 release candidate 1


@emph{Release date: 2018-06-12}

@menu
* Core and Builtins: Core and Builtins<27>. 
* Library: Library<28>. 
* Documentation: Documentation<28>. 
* Build: Build<26>. 
* Windows: Windows<25>. 
* IDLE: IDLE<25>. 

@end menu

@node Core and Builtins<27>,Library<28>,,Python 3 7 0 release candidate 1
@anchor{whatsnew/changelog id241}@anchor{131b}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-33803@footnote{https://bugs.python.org/issue33803}: Fix a crash in hamt.c caused by enabling GC tracking for an
object that hadn’t all of its fields set to NULL.

@item 
bpo-33706@footnote{https://bugs.python.org/issue33706}: Fix a crash in Python initialization when parsing the command
line options. Thanks Christoph Gohlke for the bug report and the fix!

@item 
bpo-30654@footnote{https://bugs.python.org/issue30654}: Fixed reset of the SIGINT handler to SIG_DFL on interpreter
shutdown even when there was a custom handler set previously. Patch by
Philipp Kerling.

@item 
bpo-31849@footnote{https://bugs.python.org/issue31849}: Fix signed/unsigned comparison warning in pyhash.c.
@end itemize

@node Library<28>,Documentation<28>,Core and Builtins<27>,Python 3 7 0 release candidate 1
@anchor{whatsnew/changelog id242}@anchor{131c}
@subsubsection Library



@itemize -

@item 
bpo-30167@footnote{https://bugs.python.org/issue30167}: Prevent site.main() exception if PYTHONSTARTUP is set. Patch by
Steve Weber.

@item 
bpo-33812@footnote{https://bugs.python.org/issue33812}: Datetime instance d with non-None tzinfo, but with
d.tzinfo.utcoffset(d) returning None is now treated as naive by the
astimezone() method.

@item 
bpo-30805@footnote{https://bugs.python.org/issue30805}: Avoid race condition with debug logging

@item 
bpo-33694@footnote{https://bugs.python.org/issue33694}: asyncio: Fix a race condition causing data loss on
pause_reading()/resume_reading() when using the ProactorEventLoop.

@item 
bpo-32493@footnote{https://bugs.python.org/issue32493}: Correct test for @code{uuid_enc_be} availability in
@code{configure.ac}. Patch by Michael Felt.

@item 
bpo-33792@footnote{https://bugs.python.org/issue33792}: Add asyncio.WindowsSelectorEventLoopPolicy and
asyncio.WindowsProactorEventLoopPolicy.

@item 
bpo-33778@footnote{https://bugs.python.org/issue33778}: Update @code{unicodedata}’s database to Unicode version 11.0.0.

@item 
bpo-33770@footnote{https://bugs.python.org/issue33770}: improve base64 exception message for encoded inputs of invalid
length

@item 
bpo-33769@footnote{https://bugs.python.org/issue33769}: asyncio/start_tls: Fix error message; cancel callbacks in case
of an unhandled error; mark SSLTransport as closed if it is aborted.

@item 
bpo-33767@footnote{https://bugs.python.org/issue33767}: The concatenation (@code{+}) and repetition (@code{*}) sequence
operations now raise @ref{162,,TypeError} instead of @ref{891,,SystemError} when
performed on @ref{511,,mmap.mmap} objects.  Patch by Zackery Spytz.

@item 
bpo-33734@footnote{https://bugs.python.org/issue33734}: asyncio/ssl: Fix AttributeError, increase default handshake
timeout

@item 
bpo-11874@footnote{https://bugs.python.org/issue11874}: Use a better regex when breaking usage into wrappable parts.
Avoids bogus assertion errors from custom metavar strings.

@item 
bpo-33582@footnote{https://bugs.python.org/issue33582}: Emit a deprecation warning for inspect.formatargspec
@end itemize

@node Documentation<28>,Build<26>,Library<28>,Python 3 7 0 release candidate 1
@anchor{whatsnew/changelog id243}@anchor{131d}
@subsubsection Documentation



@itemize -

@item 
bpo-33409@footnote{https://bugs.python.org/issue33409}: Clarified the relationship between 
@geindex Python Enhancement Proposals; PEP 538
PEP 538@footnote{https://www.python.org/dev/peps/pep-0538}’s
PYTHONCOERCECLOCALE and PEP 540’s PYTHONUTF8 mode.

@item 
bpo-33736@footnote{https://bugs.python.org/issue33736}: Improve the documentation of @ref{636,,asyncio.open_connection()},
@ref{637,,asyncio.start_server()} and their UNIX socket counterparts.

@item 
bpo-31432@footnote{https://bugs.python.org/issue31432}: Clarify meaning of CERT_NONE, CERT_OPTIONAL, and CERT_REQUIRED
flags for ssl.SSLContext.verify_mode.
@end itemize

@node Build<26>,Windows<25>,Documentation<28>,Python 3 7 0 release candidate 1
@anchor{whatsnew/changelog id244}@anchor{131e}
@subsubsection Build



@itemize -

@item 
bpo-5755@footnote{https://bugs.python.org/issue5755}: Move @code{-Wstrict-prototypes} option to @code{CFLAGS_NODIST} from
@code{OPT}. This option emitted annoying warnings when building extension
modules written in C++.
@end itemize

@node Windows<25>,IDLE<25>,Build<26>,Python 3 7 0 release candidate 1
@anchor{whatsnew/changelog id245}@anchor{131f}
@subsubsection Windows



@itemize -

@item 
bpo-33720@footnote{https://bugs.python.org/issue33720}: Reduces maximum marshal recursion depth on release builds.
@end itemize

@node IDLE<25>,,Windows<25>,Python 3 7 0 release candidate 1
@anchor{whatsnew/changelog id246}@anchor{1320}
@subsubsection IDLE



@itemize -

@item 
bpo-33656@footnote{https://bugs.python.org/issue33656}: On Windows, add API call saying that tk scales for DPI. On
Windows 8.1+ or 10, with DPI compatibility properties of the Python binary
unchanged, and a monitor resolution greater than 96 DPI, this should make
text and lines sharper.  It should otherwise have no effect.

@item 
bpo-33768@footnote{https://bugs.python.org/issue33768}: Clicking on a context line moves that line to the top of the
editor window.

@item 
bpo-33763@footnote{https://bugs.python.org/issue33763}: IDLE: Use read-only text widget for code context instead of
label widget.

@item 
bpo-33664@footnote{https://bugs.python.org/issue33664}: Scroll IDLE editor text by lines. Previously, the mouse wheel
and scrollbar slider moved text by a fixed number of pixels, resulting in
partial lines at the top of the editor box.  The change also applies to
the shell and grep output windows, but not to read-only text views.

@item 
bpo-33679@footnote{https://bugs.python.org/issue33679}: Enable theme-specific color configuration for Code Context. Use
the Highlights tab to see the setting for built-in themes or add settings
to custom themes.

@item 
bpo-33642@footnote{https://bugs.python.org/issue33642}: Display up to maxlines non-blank lines for Code Context. If
there is no current context, show a single blank line.
@end itemize

@node Python 3 7 0 beta 5,Python 3 7 0 beta 4,Python 3 7 0 release candidate 1,Changelog
@anchor{whatsnew/changelog python-3-7-0-beta-5}@anchor{1321}
@subsection Python 3.7.0 beta 5


@emph{Release date: 2018-05-30}

@menu
* Core and Builtins: Core and Builtins<28>. 
* Library: Library<29>. 
* Documentation: Documentation<29>. 
* Tests: Tests<26>. 
* Build: Build<27>. 
* macOS: macOS<21>. 
* IDLE: IDLE<26>. 

@end menu

@node Core and Builtins<28>,Library<29>,,Python 3 7 0 beta 5
@anchor{whatsnew/changelog id247}@anchor{1322}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-33622@footnote{https://bugs.python.org/issue33622}: Fixed a leak when the garbage collector fails to add an object
with the @code{__del__} method or referenced by it into the
@ref{d66,,gc.garbage} list. @ref{49a,,PyGC_Collect()} can now be called when an
exception is set and preserves it.

@item 
bpo-33509@footnote{https://bugs.python.org/issue33509}: Fix module_globals parameter of warnings.warn_explicit(): don’t
crash if module_globals is not a dict.

@item 
bpo-20104@footnote{https://bugs.python.org/issue20104}: The new @ref{56e,,os.posix_spawn} added in 3.7.0b1 was removed as we are
still working on what the API should look like.  Expect this in 3.8
instead.

@item 
bpo-33475@footnote{https://bugs.python.org/issue33475}: Fixed miscellaneous bugs in converting annotations to strings
and optimized parentheses in the string representation.

@item 
bpo-33391@footnote{https://bugs.python.org/issue33391}: Fix a leak in set_symmetric_difference().

@item 
bpo-28055@footnote{https://bugs.python.org/issue28055}: Fix unaligned accesses in siphash24(). Patch by Rolf Eike Beer.

@item 
bpo-32911@footnote{https://bugs.python.org/issue32911}: Due to unexpected compatibility issues discovered during
downstream beta testing, reverted bpo-29463@footnote{https://bugs.python.org/issue29463}. @code{docstring} field is
removed from Module, ClassDef, FunctionDef, and AsyncFunctionDef ast nodes
which was added in 3.7a1.  Docstring expression is restored as a first
statement in their body. Based on patch by Inada Naoki.

@item 
bpo-21983@footnote{https://bugs.python.org/issue21983}: Fix a crash in @ref{12db,,ctypes.cast()} in case the type argument is a
ctypes structured data type. Patch by Eryk Sun and Oren Milman.
@end itemize

@node Library<29>,Documentation<29>,Core and Builtins<28>,Python 3 7 0 beta 5
@anchor{whatsnew/changelog id248}@anchor{1323}
@subsubsection Library



@itemize -

@item 
bpo-32751@footnote{https://bugs.python.org/issue32751}: When cancelling the task due to a timeout,
@ref{3d5,,asyncio.wait_for()} will now wait until the cancellation is complete.

@item 
bpo-32684@footnote{https://bugs.python.org/issue32684}: Fix gather to propagate cancellation of itself even with
return_exceptions.

@item 
bpo-33654@footnote{https://bugs.python.org/issue33654}: Support protocol type switching in SSLTransport.set_protocol().

@item 
bpo-33674@footnote{https://bugs.python.org/issue33674}: Pause the transport as early as possible to further reduce the
risk of data_received() being called before connection_made().

@item 
bpo-33674@footnote{https://bugs.python.org/issue33674}: Fix a race condition in SSLProtocol.connection_made() of
asyncio.sslproto: start immediately the handshake instead of using
call_soon(). Previously, data_received() could be called before the
handshake started, causing the handshake to hang or fail.

@item 
bpo-31647@footnote{https://bugs.python.org/issue31647}: Fixed bug where calling write_eof() on a
_SelectorSocketTransport after it’s already closed raises AttributeError.

@item 
bpo-32610@footnote{https://bugs.python.org/issue32610}: Make asyncio.all_tasks() return only pending tasks.

@item 
bpo-32410@footnote{https://bugs.python.org/issue32410}: Avoid blocking on file IO in sendfile fallback code

@item 
bpo-33469@footnote{https://bugs.python.org/issue33469}: Fix RuntimeError after closing loop that used run_in_executor

@item 
bpo-33672@footnote{https://bugs.python.org/issue33672}: Fix Task.__repr__ crash with Cython’s bogus coroutines

@item 
bpo-33654@footnote{https://bugs.python.org/issue33654}: Fix transport.set_protocol() to support switching between
asyncio.Protocol and asyncio.BufferedProtocol.  Fix loop.start_tls() to
work with asyncio.BufferedProtocols.

@item 
bpo-33652@footnote{https://bugs.python.org/issue33652}: Pickles of type variables and subscripted generics are now
future-proof and compatible with older Python versions.

@item 
bpo-32493@footnote{https://bugs.python.org/issue32493}: Fixed @ref{6da,,uuid.uuid1()} on FreeBSD.

@item 
bpo-33618@footnote{https://bugs.python.org/issue33618}: Finalize and document preliminary and experimental TLS 1.3
support with OpenSSL 1.1.1

@item 
bpo-33623@footnote{https://bugs.python.org/issue33623}: Fix possible SIGSGV when asyncio.Future is created in __del__

@item 
bpo-30877@footnote{https://bugs.python.org/issue30877}: Fixed a bug in the Python implementation of the JSON decoder
that prevented the cache of parsed strings from clearing after finishing
the decoding. Based on patch by c-fos.

@item 
bpo-33570@footnote{https://bugs.python.org/issue33570}: Change TLS 1.3 cipher suite settings for compatibility with
OpenSSL 1.1.1-pre6 and newer. OpenSSL 1.1.1 will have TLS 1.3 ciphers
enabled by default.

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Do not simplify arguments to @ref{235,,typing.Union}. Now
@code{Union[Manager, Employee]} is not simplified to @code{Employee} at runtime.
Such simplification previously caused several bugs and limited
possibilities for introspection.

@item 
bpo-33540@footnote{https://bugs.python.org/issue33540}: Add a new @code{block_on_close} class attribute to
@code{ForkingMixIn} and @code{ThreadingMixIn} classes of @ref{eb,,socketserver}.

@item 
bpo-33548@footnote{https://bugs.python.org/issue33548}: tempfile._candidate_tempdir_list should consider common TEMP
locations

@item 
bpo-33109@footnote{https://bugs.python.org/issue33109}: argparse subparsers are once again not required by default,
reverting the change in behavior introduced by bpo-26510@footnote{https://bugs.python.org/issue26510} in 3.7.0a2.

@item 
bpo-33536@footnote{https://bugs.python.org/issue33536}: dataclasses.make_dataclass now checks for invalid field names
and duplicate fields. Also, added a check for invalid field
specifications.

@item 
bpo-33542@footnote{https://bugs.python.org/issue33542}: Prevent @code{uuid.get_node} from using a DUID instead of a MAC on
Windows. Patch by Zvi Effron

@item 
bpo-26819@footnote{https://bugs.python.org/issue26819}: Fix race condition with @code{ReadTransport.resume_reading} in
Windows proactor event loop.

@item 
Fix failure in @ref{432,,typing.get_type_hints()} when ClassVar was provided as a
string forward reference.

@item 
bpo-33505@footnote{https://bugs.python.org/issue33505}: Optimize asyncio.ensure_future() by reordering if checks: 1.17x
faster.

@item 
bpo-33497@footnote{https://bugs.python.org/issue33497}: Add errors param to cgi.parse_multipart and make an encoding in
FieldStorage use the given errors (needed for Twisted).  Patch by Amber
Brown.

@item 
bpo-33495@footnote{https://bugs.python.org/issue33495}: Change dataclasses.Fields repr to use the repr of each of its
members, instead of str.  This makes it more clear what each field
actually represents.  This is especially true for the ‘type’ member.

@item 
bpo-33453@footnote{https://bugs.python.org/issue33453}: Fix dataclasses to work if using literal string type
annotations or if using PEP 563 “Postponed Evaluation of Annotations”.
Only specific string prefixes are detected for both ClassVar (“ClassVar”
and “typing.ClassVar”) and InitVar (“InitVar” and “dataclasses.InitVar”).

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Minor fixes in typing module: add annotations to
@code{NamedTuple.__new__}, pass @code{*args} and @code{**kwds} in
@code{Generic.__new__}.  Original PRs by Paulius Šarka and Chad Dombrova.

@item 
bpo-20087@footnote{https://bugs.python.org/issue20087}: Updated alias mapping with glibc 2.27 supported locales.

@item 
bpo-33422@footnote{https://bugs.python.org/issue33422}: Fix trailing quotation marks getting deleted when looking up
byte/string literals on pydoc. Patch by Andrés Delfino.

@item 
bpo-28167@footnote{https://bugs.python.org/issue28167}: The function @code{platform.linux_distribution} and
@code{platform.dist} now trigger a @code{DeprecationWarning} and have been
marked for removal in Python 3.8

@item 
bpo-33197@footnote{https://bugs.python.org/issue33197}: Update error message when constructing invalid
inspect.Parameters Patch by Dong-hee Na.

@item 
bpo-33263@footnote{https://bugs.python.org/issue33263}: Fix FD leak in @code{_SelectorSocketTransport}  Patch by Vlad
Starostin.

@item 
bpo-32861@footnote{https://bugs.python.org/issue32861}: The urllib.robotparser’s @code{__str__} representation now
includes wildcard entries and the “Crawl-delay” and “Request-rate” fields.
Patch by Michael Lazar.

@item 
bpo-32257@footnote{https://bugs.python.org/issue32257}: The ssl module now contains OP_NO_RENEGOTIATION constant,
available with OpenSSL 1.1.0h or 1.1.1.

@item 
bpo-16865@footnote{https://bugs.python.org/issue16865}: Support arrays >=2GiB in @ref{29,,ctypes}.  Patch by Segev Finer.
@end itemize

@node Documentation<29>,Tests<26>,Library<29>,Python 3 7 0 beta 5
@anchor{whatsnew/changelog id249}@anchor{1324}
@subsubsection Documentation



@itemize -

@item 
bpo-23859@footnote{https://bugs.python.org/issue23859}: Document that @ref{440,,asyncio.wait()} does not cancel its futures on
timeout.

@item 
bpo-32436@footnote{https://bugs.python.org/issue32436}: Document 
@geindex Python Enhancement Proposals; PEP 567
PEP 567@footnote{https://www.python.org/dev/peps/pep-0567} changes to asyncio.

@item 
bpo-33604@footnote{https://bugs.python.org/issue33604}: Update HMAC md5 default to a DeprecationWarning, bump removal
to 3.8.

@item 
bpo-33503@footnote{https://bugs.python.org/issue33503}: Fix broken pypi link

@item 
bpo-33421@footnote{https://bugs.python.org/issue33421}: Add missing documentation for @code{typing.AsyncContextManager}.
@end itemize

@node Tests<26>,Build<27>,Documentation<29>,Python 3 7 0 beta 5
@anchor{whatsnew/changelog id250}@anchor{1325}
@subsubsection Tests



@itemize -

@item 
bpo-33655@footnote{https://bugs.python.org/issue33655}: Ignore test_posix_fallocate failures on BSD platforms that
might be due to running on ZFS.

@item 
bpo-32604@footnote{https://bugs.python.org/issue32604}: Remove the _xxsubinterpreters module (meant for testing) and
associated helpers.  This module was originally added recently in 3.7b1.
@end itemize

@node Build<27>,macOS<21>,Tests<26>,Python 3 7 0 beta 5
@anchor{whatsnew/changelog id251}@anchor{1326}
@subsubsection Build



@itemize -

@item 
bpo-33614@footnote{https://bugs.python.org/issue33614}: Ensures module definition files for the stable ABI on Windows
are correctly regenerated.

@item 
bpo-33522@footnote{https://bugs.python.org/issue33522}: Enable CI builds on Visual Studio Team Services at
@indicateurl{https://python.visualstudio.com/cpython}

@item 
bpo-33012@footnote{https://bugs.python.org/issue33012}: Add @code{-Wno-cast-function-type} for gcc 8 for silencing
warnings about function casts like casting to PyCFunction in method
definition lists.
@end itemize

@node macOS<21>,IDLE<26>,Build<27>,Python 3 7 0 beta 5
@anchor{whatsnew/changelog id252}@anchor{1327}
@subsubsection macOS



@itemize -

@item 
bpo-13631@footnote{https://bugs.python.org/issue13631}: The .editrc file in user’s home directory is now processed
correctly during the readline initialization through editline emulation on
macOS.
@end itemize

@node IDLE<26>,,macOS<21>,Python 3 7 0 beta 5
@anchor{whatsnew/changelog id253}@anchor{1328}
@subsubsection IDLE



@itemize -

@item 
bpo-33628@footnote{https://bugs.python.org/issue33628}: IDLE: Cleanup codecontext.py and its test.

@item 
bpo-33564@footnote{https://bugs.python.org/issue33564}: IDLE’s code context now recognizes async as a block opener.

@item 
bpo-32831@footnote{https://bugs.python.org/issue32831}: Add docstrings and tests for codecontext.
@end itemize

@node Python 3 7 0 beta 4,Python 3 7 0 beta 3,Python 3 7 0 beta 5,Changelog
@anchor{whatsnew/changelog python-3-7-0-beta-4}@anchor{1329}
@subsection Python 3.7.0 beta 4


@emph{Release date: 2018-05-02}

@menu
* Core and Builtins: Core and Builtins<29>. 
* Library: Library<30>. 
* Documentation: Documentation<30>. 
* Tests: Tests<27>. 
* Build: Build<28>. 
* Windows: Windows<26>. 
* macOS: macOS<22>. 
* IDLE: IDLE<27>. 
* Tools/Demos: Tools/Demos<12>. 

@end menu

@node Core and Builtins<29>,Library<30>,,Python 3 7 0 beta 4
@anchor{whatsnew/changelog id254}@anchor{132a}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-33363@footnote{https://bugs.python.org/issue33363}: Raise a SyntaxError for @code{async with} and @code{async for}
statements outside of async functions.

@item 
bpo-33128@footnote{https://bugs.python.org/issue33128}: Fix a bug that causes PathFinder to appear twice on
sys.meta_path. Patch by Pablo Galindo Salgado.

@item 
bpo-33312@footnote{https://bugs.python.org/issue33312}: Fixed clang ubsan (undefined behavior sanitizer) warnings in
dictobject.c by adjusting how the internal struct _dictkeysobject shared
keys structure is declared.

@item 
bpo-33231@footnote{https://bugs.python.org/issue33231}: Fix potential memory leak in @code{normalizestring()}.

@item 
bpo-33205@footnote{https://bugs.python.org/issue33205}: Change dict growth function from
@code{round_up_to_power_2(used*2+hashtable_size/2)} to
@code{round_up_to_power_2(used*3)}.  Previously, dict is shrinked only when
@code{used == 0}. Now dict has more chance to be shrinked.

@item 
bpo-29922@footnote{https://bugs.python.org/issue29922}: Improved error messages in ‘async with’ when @code{__aenter__()}
or @code{__aexit__()} return non-awaitable object.

@item 
bpo-33199@footnote{https://bugs.python.org/issue33199}: Fix @code{ma_version_tag} in dict implementation is uninitialized
when copying from key-sharing dict.
@end itemize

@node Library<30>,Documentation<30>,Core and Builtins<29>,Python 3 7 0 beta 4
@anchor{whatsnew/changelog id255}@anchor{132b}
@subsubsection Library



@itemize -

@item 
bpo-33281@footnote{https://bugs.python.org/issue33281}: Fix ctypes.util.find_library regression on macOS.

@item 
bpo-33383@footnote{https://bugs.python.org/issue33383}: Fixed crash in the get() method of the @ref{33,,dbm.ndbm} database
object when it is called with a single argument.

@item 
bpo-33329@footnote{https://bugs.python.org/issue33329}: Fix multiprocessing regression on newer glibcs

@item 
bpo-991266@footnote{https://bugs.python.org/issue991266}: Fix quoting of the @code{Comment} attribute of
@ref{12fd,,http.cookies.SimpleCookie}.

@item 
bpo-33131@footnote{https://bugs.python.org/issue33131}: Upgrade bundled version of pip to 10.0.1.

@item 
bpo-33308@footnote{https://bugs.python.org/issue33308}: Fixed a crash in the @code{parser} module when converting an ST
object to a tree of tuples or lists with @code{line_info=False} and
@code{col_info=True}.

@item 
bpo-33266@footnote{https://bugs.python.org/issue33266}: lib2to3 now recognizes @code{rf'...'} strings.

@item 
bpo-11594@footnote{https://bugs.python.org/issue11594}: Ensure line-endings are respected when using lib2to3.

@item 
bpo-33254@footnote{https://bugs.python.org/issue33254}: Have @ref{12fe,,importlib.resources.contents()} and
@ref{12ff,,importlib.abc.ResourceReader.contents()} return an @ref{dcc,,iterable}
instead of an @ref{11c3,,iterator}.

@item 
bpo-33256@footnote{https://bugs.python.org/issue33256}: Fix display of @code{<module>} call in the html produced by
@code{cgitb.html()}. Patch by Stéphane Blondon.

@item 
bpo-33185@footnote{https://bugs.python.org/issue33185}: Fixed regression when running pydoc with the @ref{611,,-m}
switch. (The regression was introduced in 3.7.0b3 by the resolution of
bpo-33053@footnote{https://bugs.python.org/issue33053}) This fix also changed pydoc to add @code{os.getcwd()} to
@ref{489,,sys.path} when necessary, rather than adding @code{"."}.

@item 
bpo-33169@footnote{https://bugs.python.org/issue33169}: Delete entries of @code{None} in @ref{74c,,sys.path_importer_cache}
when @code{importlib.machinery.invalidate_caches()} is called.

@item 
bpo-33217@footnote{https://bugs.python.org/issue33217}: Deprecate looking up non-Enum objects in Enum classes and Enum
members (will raise @ref{162,,TypeError} in 3.8+).

@item 
bpo-33203@footnote{https://bugs.python.org/issue33203}: @code{random.Random.choice()} now raises @code{IndexError} for empty
sequences consistently even when called from subclasses without a
@code{getrandbits()} implementation.

@item 
bpo-33224@footnote{https://bugs.python.org/issue33224}: Update difflib.mdiff() for 
@geindex Python Enhancement Proposals; PEP 479
PEP 479@footnote{https://www.python.org/dev/peps/pep-0479}.  Convert an uncaught
StopIteration in a generator into a return-statement.

@item 
bpo-33209@footnote{https://bugs.python.org/issue33209}: End framing at the end of C implementation of
@ref{1300,,pickle.Pickler.dump()}.

@item 
bpo-20104@footnote{https://bugs.python.org/issue20104}: Improved error handling and fixed a reference leak in
@ref{56e,,os.posix_spawn()}.

@item 
bpo-33175@footnote{https://bugs.python.org/issue33175}: In dataclasses, Field.__set_name__ now looks up the
__set_name__ special method on the class, not the instance, of the default
value.

@item 
bpo-33097@footnote{https://bugs.python.org/issue33097}: Raise RuntimeError when @code{executor.submit} is called during
interpreter shutdown.

@item 
bpo-31908@footnote{https://bugs.python.org/issue31908}: Fix output of cover files for @code{trace} module command-line
tool. Previously emitted cover files only when @code{--missing} option was
used. Patch by Michael Selik.
@end itemize

@node Documentation<30>,Tests<27>,Library<30>,Python 3 7 0 beta 4
@anchor{whatsnew/changelog id256}@anchor{132c}
@subsubsection Documentation



@itemize -

@item 
bpo-33378@footnote{https://bugs.python.org/issue33378}: Add Korean language switcher for @indicateurl{https://docs.python.org/3/}

@item 
bpo-33276@footnote{https://bugs.python.org/issue33276}: Clarify that the @code{__path__} attribute on modules cannot be
just any value.

@item 
bpo-33201@footnote{https://bugs.python.org/issue33201}: Modernize documentation for writing C extension types.

@item 
bpo-33195@footnote{https://bugs.python.org/issue33195}: Deprecate @code{Py_UNICODE} usage in @code{c-api/arg} document.
@code{Py_UNICODE} related APIs are deprecated since Python 3.3, but it is
missed in the document.

@item 
bpo-8243@footnote{https://bugs.python.org/issue8243}: Add a note about curses.addch and curses.addstr exception
behavior when writing outside a window, or pad.

@item 
bpo-32337@footnote{https://bugs.python.org/issue32337}: Update documentation related with @code{dict} order.
@end itemize

@node Tests<27>,Build<28>,Documentation<30>,Python 3 7 0 beta 4
@anchor{whatsnew/changelog id257}@anchor{132d}
@subsubsection Tests



@itemize -

@item 
bpo-33358@footnote{https://bugs.python.org/issue33358}: Fix @code{test_embed.test_pre_initialization_sys_options()} when
the interpreter is built with @code{--enable-shared}.
@end itemize

@node Build<28>,Windows<26>,Tests<27>,Python 3 7 0 beta 4
@anchor{whatsnew/changelog id258}@anchor{132e}
@subsubsection Build



@itemize -

@item 
bpo-33394@footnote{https://bugs.python.org/issue33394}: Enable the verbose build for extension modules, when GNU make
is passed macros on the command line.

@item 
bpo-33393@footnote{https://bugs.python.org/issue33393}: Update config.guess and config.sub files.

@item 
bpo-33377@footnote{https://bugs.python.org/issue33377}: Add new triplets for mips r6 and riscv variants (used in
extension suffixes).

@item 
bpo-32232@footnote{https://bugs.python.org/issue32232}: By default, modules configured in @code{Modules/Setup} are no longer
built with @code{-DPy_BUILD_CORE}. Instead, modules that specifically need that
preprocessor definition include it in their individual entries.

@item 
bpo-33182@footnote{https://bugs.python.org/issue33182}: The embedding tests can once again be built with clang 6.0
@end itemize

@node Windows<26>,macOS<22>,Build<28>,Python 3 7 0 beta 4
@anchor{whatsnew/changelog id259}@anchor{132f}
@subsubsection Windows



@itemize -

@item 
bpo-33184@footnote{https://bugs.python.org/issue33184}: Update Windows installer to use OpenSSL 1.1.0h.
@end itemize

@node macOS<22>,IDLE<27>,Windows<26>,Python 3 7 0 beta 4
@anchor{whatsnew/changelog id260}@anchor{1330}
@subsubsection macOS



@itemize -

@item 
bpo-33184@footnote{https://bugs.python.org/issue33184}: Update macOS installer build to use OpenSSL 1.1.0h.
@end itemize

@node IDLE<27>,Tools/Demos<12>,macOS<22>,Python 3 7 0 beta 4
@anchor{whatsnew/changelog id261}@anchor{1331}
@subsubsection IDLE



@itemize -

@item 
bpo-21474@footnote{https://bugs.python.org/issue21474}: Update word/identifier definition from ascii to unicode. In
text and entry boxes, this affects selection by double-click, movement
left/right by control-left/right, and deletion left/right by
control-BACKSPACE/DEL.

@item 
bpo-33204@footnote{https://bugs.python.org/issue33204}: IDLE: consistently color invalid string prefixes. A ‘u’ string
prefix cannot be paired with either ‘r’ or ‘f’. Consistently color as much
of the prefix, starting at the right, as is valid. Revise and extend
colorizer test.
@end itemize

@node Tools/Demos<12>,,IDLE<27>,Python 3 7 0 beta 4
@anchor{whatsnew/changelog id262}@anchor{1332}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-33189@footnote{https://bugs.python.org/issue33189}: @code{pygettext.py} now recognizes only literal strings as
docstrings and translatable strings, and rejects bytes literals and
f-string expressions.

@item 
bpo-31920@footnote{https://bugs.python.org/issue31920}: Fixed handling directories as arguments in the @code{pygettext}
script. Based on patch by Oleg Krasnikov.

@item 
bpo-29673@footnote{https://bugs.python.org/issue29673}: Fix pystackv and pystack gdbinit macros.

@item 
bpo-31583@footnote{https://bugs.python.org/issue31583}: Fix 2to3 for using with –add-suffix option but without
–output-dir option for relative path to files in current directory.
@end itemize

@node Python 3 7 0 beta 3,Python 3 7 0 beta 2,Python 3 7 0 beta 4,Changelog
@anchor{whatsnew/changelog python-3-7-0-beta-3}@anchor{1333}
@subsection Python 3.7.0 beta 3


@emph{Release date: 2018-03-29}

@menu
* Security: Security<19>. 
* Core and Builtins: Core and Builtins<30>. 
* Library: Library<31>. 
* Documentation: Documentation<31>. 
* Tests: Tests<28>. 
* Build: Build<29>. 
* Windows: Windows<27>. 
* macOS: macOS<23>. 
* IDLE: IDLE<28>. 
* Tools/Demos: Tools/Demos<13>. 
* C API: C API<27>. 

@end menu

@node Security<19>,Core and Builtins<30>,,Python 3 7 0 beta 3
@anchor{whatsnew/changelog id263}@anchor{1334}
@subsubsection Security



@itemize -

@item 
bpo-33136@footnote{https://bugs.python.org/issue33136}: Harden ssl module against LibreSSL CVE-2018-8970.
X509_VERIFY_PARAM_set1_host() is called with an explicit namelen. A new
test ensures that NULL bytes are not allowed.

@item 
bpo-33001@footnote{https://bugs.python.org/issue33001}: Minimal fix to prevent buffer overrun in os.symlink on Windows

@item 
bpo-32981@footnote{https://bugs.python.org/issue32981}: Regexes in difflib and poplib were vulnerable to catastrophic
backtracking. These regexes formed potential DOS vectors (REDOS). They
have been refactored. This resolves CVE-2018-1060 and CVE-2018-1061. Patch
by Jamie Davis.
@end itemize

@node Core and Builtins<30>,Library<31>,Security<19>,Python 3 7 0 beta 3
@anchor{whatsnew/changelog id264}@anchor{1335}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-33053@footnote{https://bugs.python.org/issue33053}: When using the -m switch, sys.path[0] is now explicitly
expanded as the @emph{starting} working directory, rather than being left as
the empty path (which allows imports from the current working directory at
the time of the import)

@item 
bpo-33018@footnote{https://bugs.python.org/issue33018}: Improve consistency of errors raised by @code{issubclass()} when
called with a non-class and an abstract base class as the first and second
arguments, respectively. Patch by Josh Bronson.

@item 
bpo-33041@footnote{https://bugs.python.org/issue33041}: Fixed jumping when the function contains an @code{async for} loop.

@item 
bpo-33026@footnote{https://bugs.python.org/issue33026}: Fixed jumping out of “with” block by setting f_lineno.

@item 
bpo-33005@footnote{https://bugs.python.org/issue33005}: Fix a crash on fork when using a custom memory allocator (ex:
using PYTHONMALLOC env var). _PyGILState_Reinit() and
_PyInterpreterState_Enable() now use the default RAW memory allocator to
allocate a new interpreters mutex on fork.

@item 
bpo-17288@footnote{https://bugs.python.org/issue17288}: Prevent jumps from ‘return’ and ‘exception’ trace events.

@item 
bpo-32836@footnote{https://bugs.python.org/issue32836}: Don’t use temporary variables in cases of list/dict/set
comprehensions
@end itemize

@node Library<31>,Documentation<31>,Core and Builtins<30>,Python 3 7 0 beta 3
@anchor{whatsnew/changelog id265}@anchor{1336}
@subsubsection Library



@itemize -

@item 
bpo-33141@footnote{https://bugs.python.org/issue33141}: Have Field objects pass through __set_name__ to their default
values, if they have their own __set_name__.

@item 
bpo-33096@footnote{https://bugs.python.org/issue33096}: Allow ttk.Treeview.insert to insert iid that has a false
boolean value. Note iid=0 and iid=False would be same. Patch by Garvit
Khatri.

@item 
bpo-32873@footnote{https://bugs.python.org/issue32873}: Treat type variables and special typing forms as immutable by
copy and pickle.  This fixes several minor issues and inconsistencies, and
improves backwards compatibility with Python 3.6.

@item 
bpo-33134@footnote{https://bugs.python.org/issue33134}: When computing dataclass’s __hash__, use the lookup table to
contain the function which returns the __hash__ value.  This is an
improvement over looking up a string, and then testing that string to see
what to do.

@item 
bpo-33127@footnote{https://bugs.python.org/issue33127}: The ssl module now compiles with LibreSSL 2.7.1.

@item 
bpo-32505@footnote{https://bugs.python.org/issue32505}: Raise TypeError if a member variable of a dataclass is of type
Field, but doesn’t have a type annotation.

@item 
bpo-33078@footnote{https://bugs.python.org/issue33078}: Fix the failure on OSX caused by the tests relying on
sem_getvalue

@item 
bpo-33116@footnote{https://bugs.python.org/issue33116}: Add ‘Field’ to dataclasses.__all__.

@item 
bpo-32896@footnote{https://bugs.python.org/issue32896}: Fix an error where subclassing a dataclass with a field that
uses a default_factory would generate an incorrect class.

@item 
bpo-33100@footnote{https://bugs.python.org/issue33100}: Dataclasses: If a field has a default value that’s a
MemberDescriptorType, then it’s from that field being in __slots__, not an
actual default value.

@item 
bpo-32953@footnote{https://bugs.python.org/issue32953}: If a non-dataclass inherits from a frozen dataclass, allow
attributes to be added to the derived class.  Only attributes from the
frozen dataclass cannot be assigned to.  Require all dataclasses in a
hierarchy to be either all frozen or all non-frozen.

@item 
bpo-33061@footnote{https://bugs.python.org/issue33061}: Add missing @code{NoReturn} to @code{__all__} in typing.py

@item 
bpo-33078@footnote{https://bugs.python.org/issue33078}: Fix the size handling in multiprocessing.Queue when a pickling
error occurs.

@item 
bpo-33064@footnote{https://bugs.python.org/issue33064}: lib2to3 now properly supports trailing commas after @code{*args}
and @code{**kwargs} in function signatures.

@item 
bpo-33056@footnote{https://bugs.python.org/issue33056}: FIX properly close leaking fds in
concurrent.futures.ProcessPoolExecutor.

@item 
bpo-33021@footnote{https://bugs.python.org/issue33021}: Release the GIL during fstat() calls, avoiding hang of all
threads when calling mmap.mmap(), os.urandom(), and random.seed().  Patch
by Nir Soffer.

@item 
bpo-31804@footnote{https://bugs.python.org/issue31804}: Avoid failing in multiprocessing.Process if the standard
streams are closed or None at exit.

@item 
bpo-33037@footnote{https://bugs.python.org/issue33037}: Skip sending/receiving data after SSL transport closing.

@item 
bpo-27683@footnote{https://bugs.python.org/issue27683}: Fix a regression in @ref{9e,,ipaddress} that result of
@code{hosts()} is empty when the network is constructed by a tuple
containing an integer mask and only 1 bit left for addresses.

@item 
bpo-32999@footnote{https://bugs.python.org/issue32999}: Fix C implementation of @code{ABC.__subclasscheck__(cls,
subclass)} crashed when @code{subclass} is not a type object.

@item 
bpo-33009@footnote{https://bugs.python.org/issue33009}: Fix inspect.signature() for single-parameter partialmethods.

@item 
bpo-32969@footnote{https://bugs.python.org/issue32969}: Expose several missing constants in zlib and fix corresponding
documentation.

@item 
bpo-32056@footnote{https://bugs.python.org/issue32056}: Improved exceptions raised for invalid number of channels and
sample width when read an audio file in modules @ref{4,,aifc}, @ref{12e,,wave}
and @ref{f5,,sunau}.

@item 
bpo-32844@footnote{https://bugs.python.org/issue32844}: Fix wrong redirection of a low descriptor (0 or 1) to stderr in
subprocess if another low descriptor is closed.

@item 
bpo-32857@footnote{https://bugs.python.org/issue32857}: In @ref{10c,,tkinter}, @code{after_cancel(None)} now raises a
@ref{17c,,ValueError} instead of canceling the first scheduled function.
Patch by Cheryl Sabella.

@item 
bpo-31639@footnote{https://bugs.python.org/issue31639}: http.server now exposes a ThreadedHTTPServer class and uses it
when the module is run with @code{-m} to cope with web browsers pre-opening
sockets.

@item 
bpo-27645@footnote{https://bugs.python.org/issue27645}: @ref{18b,,sqlite3.Connection} now exposes a
@ref{6a7,,backup} method, if the underlying SQLite
library is at version 3.6.11 or higher.  Patch by Lele Gaifax.
@end itemize

@node Documentation<31>,Tests<28>,Library<31>,Python 3 7 0 beta 3
@anchor{whatsnew/changelog id266}@anchor{1337}
@subsubsection Documentation



@itemize -

@item 
bpo-33126@footnote{https://bugs.python.org/issue33126}: Document PyBuffer_ToContiguous().

@item 
bpo-27212@footnote{https://bugs.python.org/issue27212}: Modify documentation for the @code{islice()} recipe to consume
initial values up to the start index.

@item 
bpo-28247@footnote{https://bugs.python.org/issue28247}: Update @ref{148,,zipapp} documentation to describe how to make
standalone applications.

@item 
bpo-18802@footnote{https://bugs.python.org/issue18802}: Documentation changes for ipaddress.  Patch by Jon Foster and
Berker Peksag.

@item 
bpo-27428@footnote{https://bugs.python.org/issue27428}: Update documentation to clarify that @code{WindowsRegistryFinder}
implements @code{MetaPathFinder}. (Patch by Himanshu Lakhara)
@end itemize

@node Tests<28>,Build<29>,Documentation<31>,Python 3 7 0 beta 3
@anchor{whatsnew/changelog id267}@anchor{1338}
@subsubsection Tests



@itemize -

@item 
bpo-32872@footnote{https://bugs.python.org/issue32872}: Avoid regrtest compatibility issue with namespace packages.

@item 
bpo-32517@footnote{https://bugs.python.org/issue32517}: Fix failing @code{test_asyncio} on macOS 10.12.2+ due to transport
of @code{KqueueSelector} loop was not being closed.

@item 
bpo-19417@footnote{https://bugs.python.org/issue19417}: Add test_bdb.py.
@end itemize

@node Build<29>,Windows<27>,Tests<28>,Python 3 7 0 beta 3
@anchor{whatsnew/changelog id268}@anchor{1339}
@subsubsection Build



@itemize -

@item 
bpo-33163@footnote{https://bugs.python.org/issue33163}: Upgrade pip to 9.0.3 and setuptools to v39.0.1.
@end itemize

@node Windows<27>,macOS<23>,Build<29>,Python 3 7 0 beta 3
@anchor{whatsnew/changelog id269}@anchor{133a}
@subsubsection Windows



@itemize -

@item 
bpo-33016@footnote{https://bugs.python.org/issue33016}: Fix potential use of uninitialized memory in
nt._getfinalpathname

@item 
bpo-32903@footnote{https://bugs.python.org/issue32903}: Fix a memory leak in os.chdir() on Windows if the current
directory is set to a UNC path.
@end itemize

@node macOS<23>,IDLE<28>,Windows<27>,Python 3 7 0 beta 3
@anchor{whatsnew/changelog id270}@anchor{133b}
@subsubsection macOS



@itemize -

@item 
bpo-32726@footnote{https://bugs.python.org/issue32726}: Build and link with private copy of Tcl/Tk 8.6 for the macOS
10.6+ installer. The 10.9+ installer variant already does this.  This
means that the Python 3.7 provided by the python.org macOS installers no
longer need or use any external versions of Tcl/Tk, either system-provided
or user-installed, such as ActiveTcl.
@end itemize

@node IDLE<28>,Tools/Demos<13>,macOS<23>,Python 3 7 0 beta 3
@anchor{whatsnew/changelog id271}@anchor{133c}
@subsubsection IDLE



@itemize -

@item 
bpo-32984@footnote{https://bugs.python.org/issue32984}: Set @code{__file__} while running a startup file.  Like Python,
IDLE optionally runs one startup file in the Shell window before
presenting the first interactive input prompt.  For IDLE, @code{-s} runs a
file named in environmental variable  
@geindex IDLESTARTUP
@geindex environment variable; IDLESTARTUP
@code{IDLESTARTUP} or
@geindex PYTHONSTARTUP
@geindex environment variable; PYTHONSTARTUP
@ref{b4d,,PYTHONSTARTUP}; @code{-r file} runs @code{file}.  Python sets
@code{__file__} to the startup file name before running the file and unsets
it before the first prompt.  IDLE now does the same when run normally,
without the @code{-n} option.

@item 
bpo-32940@footnote{https://bugs.python.org/issue32940}: Simplify and rename StringTranslatePseudoMapping in pyparse.
@end itemize

@node Tools/Demos<13>,C API<27>,IDLE<28>,Python 3 7 0 beta 3
@anchor{whatsnew/changelog id272}@anchor{133d}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-32885@footnote{https://bugs.python.org/issue32885}: Add an @code{-n} flag for @code{Tools/scripts/pathfix.py} to disable
automatic backup creation (files with @code{~} suffix).
@end itemize

@node C API<27>,,Tools/Demos<13>,Python 3 7 0 beta 3
@anchor{whatsnew/changelog id273}@anchor{133e}
@subsubsection C API



@itemize -

@item 
bpo-33042@footnote{https://bugs.python.org/issue33042}: Embedding applications may once again call
PySys_ResetWarnOptions, PySys_AddWarnOption, and PySys_AddXOption prior to
calling Py_Initialize.

@item 
bpo-32374@footnote{https://bugs.python.org/issue32374}: Document that m_traverse for multi-phase initialized modules
can be called with m_state=NULL, and add a sanity check
@end itemize

@node Python 3 7 0 beta 2,Python 3 7 0 beta 1,Python 3 7 0 beta 3,Changelog
@anchor{whatsnew/changelog python-3-7-0-beta-2}@anchor{133f}
@subsection Python 3.7.0 beta 2


@emph{Release date: 2018-02-27}

@menu
* Security: Security<20>. 
* Core and Builtins: Core and Builtins<31>. 
* Library: Library<32>. 
* Documentation: Documentation<32>. 
* Tests: Tests<29>. 
* Build: Build<30>. 
* Windows: Windows<28>. 
* macOS: macOS<24>. 
* IDLE: IDLE<29>. 
* Tools/Demos: Tools/Demos<14>. 

@end menu

@node Security<20>,Core and Builtins<31>,,Python 3 7 0 beta 2
@anchor{whatsnew/changelog id274}@anchor{1340}
@subsubsection Security



@itemize -

@item 
bpo-28414@footnote{https://bugs.python.org/issue28414}: The ssl module now allows users to perform their own IDN
en/decoding when using SNI.
@end itemize

@node Core and Builtins<31>,Library<32>,Security<20>,Python 3 7 0 beta 2
@anchor{whatsnew/changelog id275}@anchor{1341}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-32889@footnote{https://bugs.python.org/issue32889}: Update Valgrind suppression list to account for the rename of
@code{Py_ADDRESS_IN_RANG} to @code{address_in_range}.

@item 
bpo-31356@footnote{https://bugs.python.org/issue31356}: Remove the new API added in bpo-31356@footnote{https://bugs.python.org/issue31356} (gc.ensure_disabled()
context manager).

@item 
bpo-32305@footnote{https://bugs.python.org/issue32305}: For namespace packages, ensure that both @code{__file__} and
@code{__spec__.origin} are set to None.

@item 
bpo-32303@footnote{https://bugs.python.org/issue32303}: Make sure @code{__spec__.loader} matches @code{__loader__} for
namespace packages.

@item 
bpo-32711@footnote{https://bugs.python.org/issue32711}: Fix the warning messages for Python/ast_unparse.c. Patch by
Stéphane Wirtel

@item 
bpo-32583@footnote{https://bugs.python.org/issue32583}: Fix possible crashing in builtin Unicode decoders caused by
write out-of-bound errors when using customized decode error handlers.
@end itemize

@node Library<32>,Documentation<32>,Core and Builtins<31>,Python 3 7 0 beta 2
@anchor{whatsnew/changelog id276}@anchor{1342}
@subsubsection Library



@itemize -

@item 
bpo-32960@footnote{https://bugs.python.org/issue32960}: For dataclasses, disallow inheriting frozen from non-frozen
classes, and also disallow inheriting non-frozen from frozen classes. This
restriction will be relaxed at a future date.

@item 
bpo-32713@footnote{https://bugs.python.org/issue32713}: Fixed tarfile.itn handling of out-of-bounds float values. Patch
by Joffrey Fuhrer.

@item 
bpo-32951@footnote{https://bugs.python.org/issue32951}: Direct instantiation of SSLSocket and SSLObject objects is now
prohibited. The constructors were never documented, tested, or designed as
public constructors. Users were suppose to use ssl.wrap_socket() or
SSLContext.

@item 
bpo-32929@footnote{https://bugs.python.org/issue32929}: Remove the tri-state parameter “hash”, and add the boolean
“unsafe_hash”. If unsafe_hash is True, add a __hash__ function, but if a
__hash__ exists, raise TypeError.  If unsafe_hash is False, add a __hash__
based on the values of eq= and frozen=.  The unsafe_hash=False behavior is
the same as the old hash=None behavior.  unsafe_hash=False is the default,
just as hash=None used to be.

@item 
bpo-32947@footnote{https://bugs.python.org/issue32947}: Add OP_ENABLE_MIDDLEBOX_COMPAT and test workaround for TLSv1.3
for future compatibility with OpenSSL 1.1.1.

@item 
bpo-30622@footnote{https://bugs.python.org/issue30622}: The ssl module now detects missing NPN support in LibreSSL.

@item 
bpo-32922@footnote{https://bugs.python.org/issue32922}: dbm.open() now encodes filename with the filesystem encoding
rather than default encoding.

@item 
bpo-32859@footnote{https://bugs.python.org/issue32859}: In @code{os.dup2}, don’t check every call whether the @code{dup3}
syscall exists or not.

@item 
bpo-32556@footnote{https://bugs.python.org/issue32556}: nt._getfinalpathname, nt._getvolumepathname and
nt._getdiskusage now correctly convert from bytes.

@item 
bpo-25988@footnote{https://bugs.python.org/issue25988}: Emit a @ref{1b2,,DeprecationWarning} when using or importing an ABC
directly from @ref{1c,,collections} rather than from @ref{1d,,collections.abc}.

@item 
bpo-21060@footnote{https://bugs.python.org/issue21060}: Rewrite confusing message from setup.py upload from “No dist
file created in earlier command” to the more helpful “Must create and
upload files in one command”.

@item 
bpo-32852@footnote{https://bugs.python.org/issue32852}: Make sure sys.argv remains as a list when running trace.

@item 
bpo-31333@footnote{https://bugs.python.org/issue31333}: @code{_abc} module is added.  It is a speedup module with C
implementations for various functions and methods in @code{abc}.  Creating an
ABC subclass and calling @code{isinstance} or @code{issubclass} with an ABC
subclass are up to 1.5x faster. In addition, this makes Python start-up up
to 10% faster. Note that the new implementation hides internal registry
and caches, previously accessible via private attributes
@code{_abc_registry}, @code{_abc_cache}, and @code{_abc_negative_cache}.  There are
three debugging helper methods that can be used instead
@code{_dump_registry}, @code{_abc_registry_clear}, and @code{_abc_caches_clear}.

@item 
bpo-32841@footnote{https://bugs.python.org/issue32841}: Fixed @ref{58d,,asyncio.Condition} issue which silently ignored
cancellation after notifying and cancelling a conditional lock. Patch by
Bar Harel.

@item 
bpo-32819@footnote{https://bugs.python.org/issue32819}: ssl.match_hostname() has been simplified and no longer depends
on re and ipaddress module for wildcard and IP addresses. Error reporting
for invalid wildcards has been improved.

@item 
bpo-32394@footnote{https://bugs.python.org/issue32394}: socket: Remove
TCP_FASTOPEN,TCP_KEEPCNT,TCP_KEEPIDLE,TCP_KEEPINTVL flags on older version
Windows during run-time.

@item 
bpo-31787@footnote{https://bugs.python.org/issue31787}: Fixed refleaks of @code{__init__()} methods in various modules.
(Contributed by Oren Milman)

@item 
bpo-30157@footnote{https://bugs.python.org/issue30157}: Fixed guessing quote and delimiter in csv.Sniffer.sniff() when
only the last field is quoted.  Patch by Jake Davis.

@item 
bpo-32792@footnote{https://bugs.python.org/issue32792}: collections.ChainMap() preserves the order of the underlying
mappings.

@item 
bpo-32775@footnote{https://bugs.python.org/issue32775}: @ref{1301,,fnmatch.translate()} no longer produces patterns which
contain set operations. Sets starting with ‘[’ or containing ‘–’, ‘&&’,
‘~~’ or ‘||’ will be interpreted differently in regular expressions in
future versions. Currently they emit warnings. fnmatch.translate() now
avoids producing patterns containing such sets by accident.

@item 
bpo-32622@footnote{https://bugs.python.org/issue32622}: Implement native fast sendfile for Windows proactor event loop.

@item 
bpo-32777@footnote{https://bugs.python.org/issue32777}: Fix a rare but potential pre-exec child process deadlock in
subprocess on POSIX systems when marking file descriptors inheritable on
exec in the child process.  This bug appears to have been introduced in
3.4.

@item 
bpo-32647@footnote{https://bugs.python.org/issue32647}: The ctypes module used to depend on indirect linking for
dlopen. The shared extension is now explicitly linked against libdl on
platforms with dl.

@item 
bpo-32741@footnote{https://bugs.python.org/issue32741}: Implement @code{asyncio.TimerHandle.when()} method.

@item 
bpo-32691@footnote{https://bugs.python.org/issue32691}: Use mod_spec.parent when running modules with pdb

@item 
bpo-32734@footnote{https://bugs.python.org/issue32734}: Fixed @code{asyncio.Lock()} safety issue which allowed acquiring
and locking the same lock multiple times, without it being free. Patch by
Bar Harel.

@item 
bpo-32727@footnote{https://bugs.python.org/issue32727}: Do not include name field in SMTP envelope from address. Patch
by Stéphane Wirtel

@item 
bpo-31453@footnote{https://bugs.python.org/issue31453}: Add TLSVersion constants and SSLContext.maximum_version /
minimum_version attributes. The new API wraps OpenSSL 1.1
@indicateurl{https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_min_proto_version.html}
feature.

@item 
bpo-24334@footnote{https://bugs.python.org/issue24334}: Internal implementation details of ssl module were cleaned up.
The SSLSocket has one less layer of indirection. Owner and session
information are now handled by the SSLSocket and SSLObject constructor.
Channel binding implementation has been simplified.

@item 
bpo-31848@footnote{https://bugs.python.org/issue31848}: Fix the error handling in Aifc_read.initfp() when the SSND
chunk is not found. Patch by Zackery Spytz.

@item 
bpo-32585@footnote{https://bugs.python.org/issue32585}: Add Ttk spinbox widget to @ref{116,,tkinter.ttk}.  Patch by Alan D
Moore.

@item 
bpo-32221@footnote{https://bugs.python.org/issue32221}: Various functions returning tuple containing IPv6 addresses now
omit @code{%scope} part since the same information is already encoded in
@emph{scopeid} tuple item. Especially this speeds up @code{socket.recvfrom()}
when it receives multicast packet since useless resolving of network
interface name is omitted.

@item 
bpo-30693@footnote{https://bugs.python.org/issue30693}: The TarFile class now recurses directories in a reproducible
way.

@item 
bpo-30693@footnote{https://bugs.python.org/issue30693}: The ZipFile class now recurses directories in a reproducible
way.
@end itemize

@node Documentation<32>,Tests<29>,Library<32>,Python 3 7 0 beta 2
@anchor{whatsnew/changelog id277}@anchor{1343}
@subsubsection Documentation



@itemize -

@item 
bpo-28124@footnote{https://bugs.python.org/issue28124}: The ssl module function ssl.wrap_socket() has been
de-emphasized and deprecated in favor of the more secure and efficient
SSLContext.wrap_socket() method.

@item 
bpo-17232@footnote{https://bugs.python.org/issue17232}: Clarify docs for -O and -OO.  Patch by Terry Reedy.

@item 
bpo-32436@footnote{https://bugs.python.org/issue32436}: Add documentation for the contextvars module (PEP 567).

@item 
bpo-32800@footnote{https://bugs.python.org/issue32800}: Update link to w3c doc for xml default namespaces.

@item 
bpo-11015@footnote{https://bugs.python.org/issue11015}: Update @ref{100,,test.support} documentation.

@item 
bpo-8722@footnote{https://bugs.python.org/issue8722}: Document @code{__getattr__()} behavior when property @code{get()}
method raises @ref{163,,AttributeError}.

@item 
bpo-32614@footnote{https://bugs.python.org/issue32614}: Modify RE examples in documentation to use raw strings to
prevent @ref{1b2,,DeprecationWarning} and add text to REGEX HOWTO to highlight
the deprecation.

@item 
bpo-31972@footnote{https://bugs.python.org/issue31972}: Improve docstrings for @ref{1c4,,pathlib.PurePath} subclasses.
@end itemize

@node Tests<29>,Build<30>,Documentation<32>,Python 3 7 0 beta 2
@anchor{whatsnew/changelog id278}@anchor{1344}
@subsubsection Tests



@itemize -

@item 
bpo-31809@footnote{https://bugs.python.org/issue31809}: Add tests to verify connection with secp ECDH curves.
@end itemize

@node Build<30>,Windows<28>,Tests<29>,Python 3 7 0 beta 2
@anchor{whatsnew/changelog id279}@anchor{1345}
@subsubsection Build



@itemize -

@item 
bpo-32898@footnote{https://bugs.python.org/issue32898}: Fix the python debug build when using COUNT_ALLOCS.
@end itemize

@node Windows<28>,macOS<24>,Build<30>,Python 3 7 0 beta 2
@anchor{whatsnew/changelog id280}@anchor{1346}
@subsubsection Windows



@itemize -

@item 
bpo-32901@footnote{https://bugs.python.org/issue32901}: Update Tcl and Tk versions to 8.6.8

@item 
bpo-31966@footnote{https://bugs.python.org/issue31966}: Fixed WindowsConsoleIO.write() for writing empty data.

@item 
bpo-32409@footnote{https://bugs.python.org/issue32409}: Ensures activate.bat can handle Unicode contents.

@item 
bpo-32457@footnote{https://bugs.python.org/issue32457}: Improves handling of denormalized executable path when
launching Python.

@item 
bpo-32370@footnote{https://bugs.python.org/issue32370}: Use the correct encoding for ipconfig output in the uuid
module. Patch by Segev Finer.

@item 
bpo-29248@footnote{https://bugs.python.org/issue29248}: Fix @ref{414,,os.readlink()} on Windows, which was mistakenly
treating the @code{PrintNameOffset} field of the reparse data buffer as a
number of characters instead of bytes. Patch by Craig Holmquist and SSE4.
@end itemize

@node macOS<24>,IDLE<29>,Windows<28>,Python 3 7 0 beta 2
@anchor{whatsnew/changelog id281}@anchor{1347}
@subsubsection macOS



@itemize -

@item 
bpo-32901@footnote{https://bugs.python.org/issue32901}: Update macOS 10.9+ installer to Tcl/Tk 8.6.8.
@end itemize

@node IDLE<29>,Tools/Demos<14>,macOS<24>,Python 3 7 0 beta 2
@anchor{whatsnew/changelog id282}@anchor{1348}
@subsubsection IDLE



@itemize -

@item 
bpo-32916@footnote{https://bugs.python.org/issue32916}: Change @code{str} to @code{code} in pyparse.

@item 
bpo-32905@footnote{https://bugs.python.org/issue32905}: Remove unused code in pyparse module.

@item 
bpo-32874@footnote{https://bugs.python.org/issue32874}: Add tests for pyparse.

@item 
bpo-32837@footnote{https://bugs.python.org/issue32837}: Using the system and place-dependent default encoding for
open() is a bad idea for IDLE’s system and location-independent files.

@item 
bpo-32826@footnote{https://bugs.python.org/issue32826}: Add “encoding=utf-8” to open() in IDLE’s test_help_about. GUI
test test_file_buttons() only looks at initial ascii-only lines, but
failed on systems where open() defaults to ‘ascii’ because readline()
internally reads and decodes far enough ahead to encounter a non-ascii
character in CREDITS.txt.

@item 
bpo-32765@footnote{https://bugs.python.org/issue32765}: Update configdialog General tab docstring to add new widgets to
the widget list.
@end itemize

@node Tools/Demos<14>,,IDLE<29>,Python 3 7 0 beta 2
@anchor{whatsnew/changelog id283}@anchor{1349}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-32222@footnote{https://bugs.python.org/issue32222}: Fix pygettext not extracting docstrings for functions with type
annotated arguments. Patch by Toby Harradine.
@end itemize

@node Python 3 7 0 beta 1,Python 3 7 0 alpha 4,Python 3 7 0 beta 2,Changelog
@anchor{whatsnew/changelog python-3-7-0-beta-1}@anchor{134a}
@subsection Python 3.7.0 beta 1


@emph{Release date: 2018-01-30}

@menu
* Core and Builtins: Core and Builtins<32>. 
* Library: Library<33>. 
* Documentation: Documentation<33>. 
* Tests: Tests<30>. 
* Build: Build<31>. 
* Windows: Windows<29>. 
* macOS: macOS<25>. 
* C API: C API<28>. 

@end menu

@node Core and Builtins<32>,Library<33>,,Python 3 7 0 beta 1
@anchor{whatsnew/changelog id284}@anchor{134b}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-32703@footnote{https://bugs.python.org/issue32703}: Fix coroutine’s ResourceWarning when there’s an active error
set when it’s being finalized.

@item 
bpo-32650@footnote{https://bugs.python.org/issue32650}: Pdb and other debuggers dependent on bdb.py will correctly step
over (next command) native coroutines. Patch by Pablo Galindo.

@item 
bpo-28685@footnote{https://bugs.python.org/issue28685}: Optimize list.sort() and sorted() by using type specialized
comparisons when possible.

@item 
bpo-32685@footnote{https://bugs.python.org/issue32685}: Improve suggestion when the Python 2 form of print statement is
either present on the same line as the header of a compound statement or
else terminated by a semi-colon instead of a newline. Patch by Nitish
Chandra.

@item 
bpo-32697@footnote{https://bugs.python.org/issue32697}: Python now explicitly preserves the definition order of
keyword-only parameters.  It’s always preserved their order, but this
behavior was never guaranteed before; this behavior is now guaranteed and
tested.

@item 
bpo-32690@footnote{https://bugs.python.org/issue32690}: The locals() dictionary now displays in the lexical order that
variables were defined.  Previously, the order was reversed.

@item 
bpo-32677@footnote{https://bugs.python.org/issue32677}: Add @code{.isascii()} method to @code{str}, @code{bytes} and
@code{bytearray}. It can be used to test that string contains only ASCII
characters.

@item 
bpo-32670@footnote{https://bugs.python.org/issue32670}: Enforce 
@geindex Python Enhancement Proposals; PEP 479
PEP 479@footnote{https://www.python.org/dev/peps/pep-0479} for all code. This means that manually
raising a StopIteration exception from a generator is prohibited for all
code, regardless of whether ‘from __future__ import generator_stop’ was
used or not.

@item 
bpo-32591@footnote{https://bugs.python.org/issue32591}: Added built-in support for tracking the origin of coroutine
objects; see sys.set_coroutine_origin_tracking_depth and
CoroutineType.cr_origin. This replaces the asyncio debug mode’s use of
coroutine wrapping for native coroutine objects.

@item 
bpo-31368@footnote{https://bugs.python.org/issue31368}: Expose preadv and pwritev system calls in the os module. Patch
by Pablo Galindo

@item 
bpo-32544@footnote{https://bugs.python.org/issue32544}: @code{hasattr(obj, name)} and @code{getattr(obj, name, default)} are
about 4 times faster than before when @code{name} is not found and @code{obj}
doesn’t override @code{__getattr__} or @code{__getattribute__}.

@item 
bpo-26163@footnote{https://bugs.python.org/issue26163}: Improved frozenset() hash to create more distinct hash values
when faced with datasets containing many similar values.

@item 
bpo-32550@footnote{https://bugs.python.org/issue32550}: Remove the STORE_ANNOTATION bytecode.

@item 
bpo-20104@footnote{https://bugs.python.org/issue20104}: Expose posix_spawn as a low level API in the os module.
(removed before 3.7.0rc1)

@item 
bpo-24340@footnote{https://bugs.python.org/issue24340}: Fixed estimation of the code stack size.

@item 
bpo-32436@footnote{https://bugs.python.org/issue32436}: Implement 
@geindex Python Enhancement Proposals; PEP 567
PEP 567@footnote{https://www.python.org/dev/peps/pep-0567} Context Variables.

@item 
bpo-18533@footnote{https://bugs.python.org/issue18533}: @code{repr()} on a dict containing its own @code{values()} or
@code{items()} no longer raises @code{RecursionError}; OrderedDict similarly.
Instead, use @code{...}, as for other recursive structures.  Patch by Ben
North.

@item 
bpo-20891@footnote{https://bugs.python.org/issue20891}: Py_Initialize() now creates the GIL. The GIL is no longer
created “on demand” to fix a race condition when PyGILState_Ensure() is
called in a non-Python thread.

@item 
bpo-32028@footnote{https://bugs.python.org/issue32028}: Leading whitespace is now correctly ignored when generating
suggestions for converting Py2 print statements to Py3 builtin print
function calls. Patch by Sanyam Khurana.

@item 
bpo-31179@footnote{https://bugs.python.org/issue31179}: Make dict.copy() up to 5.5 times faster.

@item 
bpo-31113@footnote{https://bugs.python.org/issue31113}: Get rid of recursion in the compiler for normal control flow.
@end itemize

@node Library<33>,Documentation<33>,Core and Builtins<32>,Python 3 7 0 beta 1
@anchor{whatsnew/changelog id285}@anchor{134c}
@subsubsection Library



@itemize -

@item 
bpo-25988@footnote{https://bugs.python.org/issue25988}: Deprecate exposing the contents of collections.abc in the
regular collections module.

@item 
bpo-31429@footnote{https://bugs.python.org/issue31429}: The default cipher suite selection of the ssl module now uses a
blacklist approach rather than a hard-coded whitelist. Python no longer
re-enables ciphers that have been blocked by OpenSSL security update.
Default cipher suite selection can be configured on compile time.

@item 
bpo-30306@footnote{https://bugs.python.org/issue30306}: contextlib.contextmanager now releases the arguments passed to
the underlying generator as soon as the context manager is entered.
Previously it would keep them alive for as long as the context manager was
alive, even when not being used as a function decorator. Patch by Martin
Teichmann.

@item 
bpo-21417@footnote{https://bugs.python.org/issue21417}: Added support for setting the compression level for
zipfile.ZipFile.

@item 
bpo-32251@footnote{https://bugs.python.org/issue32251}: Implement asyncio.BufferedProtocol (provisional API).

@item 
bpo-32513@footnote{https://bugs.python.org/issue32513}: In dataclasses, allow easier overriding of dunder methods
without specifying decorator parameters.

@item 
bpo-32660@footnote{https://bugs.python.org/issue32660}: @ref{fe,,termios} makes available @code{FIONREAD}, @code{FIONCLEX},
@code{FIOCLEX}, @code{FIOASYNC} and @code{FIONBIO} also under Solaris/derivatives.

@item 
bpo-27931@footnote{https://bugs.python.org/issue27931}: Fix email address header parsing error when the username is an
empty quoted string. Patch by Xiang Zhang.

@item 
bpo-32659@footnote{https://bugs.python.org/issue32659}: Under Solaris and derivatives, @ref{134d,,os.stat_result} provides
a st_fstype attribute.

@item 
bpo-32662@footnote{https://bugs.python.org/issue32662}: Implement Server.start_serving(), Server.serve_forever(), and
Server.is_serving() methods.  Add ‘start_serving’ keyword parameter to
loop.create_server() and loop.create_unix_server().

@item 
bpo-32391@footnote{https://bugs.python.org/issue32391}: Implement @ref{629,,asyncio.StreamWriter.wait_closed()} and
@ref{62a,,asyncio.StreamWriter.is_closing()} methods

@item 
bpo-32643@footnote{https://bugs.python.org/issue32643}: Make Task._step, Task._wakeup and Future._schedule_callbacks
methods private.

@item 
bpo-32630@footnote{https://bugs.python.org/issue32630}: Refactor decimal module to use contextvars to store decimal
context.

@item 
bpo-32622@footnote{https://bugs.python.org/issue32622}: Add @code{asyncio.AbstractEventLoop.sendfile()} method.

@item 
bpo-32304@footnote{https://bugs.python.org/issue32304}: distutils’ upload command no longer corrupts tar files ending
with a CR byte, and no longer tries to convert CR to CRLF in any of the
upload text fields.

@item 
bpo-32502@footnote{https://bugs.python.org/issue32502}: uuid.uuid1 no longer raises an exception if a 64-bit hardware
address is encountered.

@item 
bpo-32596@footnote{https://bugs.python.org/issue32596}: @code{concurrent.futures} imports @code{ThreadPoolExecutor} and
@code{ProcessPoolExecutor} lazily (using 
@geindex Python Enhancement Proposals; PEP 562
PEP 562@footnote{https://www.python.org/dev/peps/pep-0562}). It makes @code{import
asyncio} about 15% faster because asyncio uses only
@code{ThreadPoolExecutor} by default.

@item 
bpo-31801@footnote{https://bugs.python.org/issue31801}: Add @code{_ignore_} to @code{Enum} so temporary variables can be used
during class construction without being turned into members.

@item 
bpo-32576@footnote{https://bugs.python.org/issue32576}: Use queue.SimpleQueue() in places where it can be invoked from
a weakref callback.

@item 
bpo-32574@footnote{https://bugs.python.org/issue32574}: Fix memory leak in asyncio.Queue, when the queue has limited
size and it is full, the cancelation of queue.put() can cause a memory
leak. Patch by: José Melero.

@item 
bpo-32521@footnote{https://bugs.python.org/issue32521}: The nis module is now compatible with new libnsl and headers
location.

@item 
bpo-32467@footnote{https://bugs.python.org/issue32467}: collections.abc.ValuesView now inherits from
collections.abc.Collection.

@item 
bpo-32473@footnote{https://bugs.python.org/issue32473}: Improve ABCMeta._dump_registry() output readability

@item 
bpo-32102@footnote{https://bugs.python.org/issue32102}: New argument @code{capture_output} for subprocess.run

@item 
bpo-32521@footnote{https://bugs.python.org/issue32521}: glibc has removed Sun RPC. Use replacement libtirpc headers and
library in nis module.

@item 
bpo-32493@footnote{https://bugs.python.org/issue32493}: UUID module fixes build for FreeBSD/OpenBSD

@item 
bpo-32503@footnote{https://bugs.python.org/issue32503}: Pickling with protocol 4 no longer creates too small frames.

@item 
bpo-29237@footnote{https://bugs.python.org/issue29237}: Create enum for pstats sorting options

@item 
bpo-32454@footnote{https://bugs.python.org/issue32454}: Add close(fd) function to the socket module.

@item 
bpo-25942@footnote{https://bugs.python.org/issue25942}: The subprocess module is now more graceful when handling a
Ctrl-C KeyboardInterrupt during subprocess.call, subprocess.run, or a
Popen context manager.  It now waits a short amount of time for the child
(presumed to have also gotten the SIGINT) to exit, before continuing the
KeyboardInterrupt exception handling.  This still includes a SIGKILL in
the call() and run() APIs, but at least the child had a chance first.

@item 
bpo-32433@footnote{https://bugs.python.org/issue32433}: The hmac module now has hmac.digest(), which provides an
optimized HMAC digest.

@item 
bpo-28134@footnote{https://bugs.python.org/issue28134}: Sockets now auto-detect family, type and protocol from file
descriptor by default.

@item 
bpo-32404@footnote{https://bugs.python.org/issue32404}: Fix bug where @ref{134e,,datetime.datetime.fromtimestamp()} did not
call __new__ in @ref{363,,datetime.datetime} subclasses.

@item 
bpo-32403@footnote{https://bugs.python.org/issue32403}: Improved speed of @ref{3e2,,datetime.date} and
@ref{363,,datetime.datetime} alternate constructors.

@item 
bpo-32228@footnote{https://bugs.python.org/issue32228}: Ensure that @code{truncate()} preserves the file position (as
reported by @code{tell()}) after writes longer than the buffer size.

@item 
bpo-32410@footnote{https://bugs.python.org/issue32410}: Implement @code{loop.sock_sendfile} for asyncio event loop.

@item 
bpo-22908@footnote{https://bugs.python.org/issue22908}: Added seek and tell to the ZipExtFile class. This only works if
the file object used to open the zipfile is seekable.

@item 
bpo-32373@footnote{https://bugs.python.org/issue32373}: Add socket.getblocking() method.

@item 
bpo-32248@footnote{https://bugs.python.org/issue32248}: Add @ref{9a,,importlib.resources} and
@ref{61b,,importlib.abc.ResourceReader} as the unified API for reading
resources contained within packages.  Loaders wishing to support resource
reading must implement the @code{get_resource_reader()} method.
File-based and zipimport-based loaders both implement these APIs.
@ref{71f,,importlib.abc.ResourceLoader} is deprecated in favor of these new
APIs.

@item 
bpo-32320@footnote{https://bugs.python.org/issue32320}: collections.namedtuple() now supports default values.

@item 
bpo-29302@footnote{https://bugs.python.org/issue29302}: Add contextlib.AsyncExitStack. Patch by Alexander Mohr and Ilya
Kulakov.

@item 
bpo-31961@footnote{https://bugs.python.org/issue31961}: @emph{Removed in Python 3.7.0b2.} The @emph{args} argument of
subprocess.Popen can now be a @ref{63d,,path-like object}. If @emph{args} is given
as a sequence, it’s first element can now be a @ref{63d,,path-like object} as
well.

@item 
bpo-31900@footnote{https://bugs.python.org/issue31900}: The @ref{739,,locale.localeconv()} function now sets temporarily the
@code{LC_CTYPE} locale to the @code{LC_NUMERIC} locale to decode
@code{decimal_point} and @code{thousands_sep} byte strings if they are non-ASCII
or longer than 1 byte, and the @code{LC_NUMERIC} locale is different than the
@code{LC_CTYPE} locale. This temporary change affects other threads. Same
change for the @ref{784,,str.format()} method when formatting a number
(@ref{1b1,,int}, @ref{256,,float}, @ref{256,,float} and subclasses) with the
@code{n} type (ex: @code{'@{:n@}'.format(1234)}).

@item 
bpo-31853@footnote{https://bugs.python.org/issue31853}: Use super().method instead of socket.method in SSLSocket.  They
were there most likely for legacy reasons.

@item 
bpo-31399@footnote{https://bugs.python.org/issue31399}: The ssl module now uses OpenSSL’s X509_VERIFY_PARAM_set1_host()
and X509_VERIFY_PARAM_set1_ip() API to verify hostname and IP addresses.
Subject common name fallback can be disabled with
SSLContext.hostname_checks_common_name.

@item 
bpo-14976@footnote{https://bugs.python.org/issue14976}: Add a queue.SimpleQueue class, an unbounded FIFO queue with a
reentrant C implementation of put().
@end itemize

@node Documentation<33>,Tests<30>,Library<33>,Python 3 7 0 beta 1
@anchor{whatsnew/changelog id286}@anchor{134f}
@subsubsection Documentation



@itemize -

@item 
bpo-32724@footnote{https://bugs.python.org/issue32724}: Add references to some commands in the documentation of Pdb.
Patch by Stéphane Wirtel

@item 
bpo-32649@footnote{https://bugs.python.org/issue32649}: Complete the C API documentation, profiling and tracing part
with the newly added per-opcode events.

@item 
bpo-17799@footnote{https://bugs.python.org/issue17799}: Explain real behaviour of sys.settrace and sys.setprofile and
their C-API counterparts regarding which type of events are received in
each function. Patch by Pablo Galindo Salgado.
@end itemize

@node Tests<30>,Build<31>,Documentation<33>,Python 3 7 0 beta 1
@anchor{whatsnew/changelog id287}@anchor{1350}
@subsubsection Tests



@itemize -

@item 
bpo-32721@footnote{https://bugs.python.org/issue32721}: Fix test_hashlib to not fail if the _md5 module is not built.

@item 
bpo-28414@footnote{https://bugs.python.org/issue28414}: Add test cases for IDNA 2003 and 2008 host names. IDNA 2003
internationalized host names are working since bpo-31399@footnote{https://bugs.python.org/issue31399} has landed. IDNA
2008 are still broken.

@item 
bpo-32604@footnote{https://bugs.python.org/issue32604}: Add a new “_xxsubinterpreters” extension module that exposes
the existing subinterpreter C-API and a new cross-interpreter data sharing
mechanism. The module is primarily intended for more thorough testing of
the existing subinterpreter support. Note that the _xxsubinterpreters
module has been removed in 3.7.0rc1.

@item 
bpo-32602@footnote{https://bugs.python.org/issue32602}: Add test certs and test for ECDSA cert and EC/RSA dual mode.

@item 
bpo-32549@footnote{https://bugs.python.org/issue32549}: On Travis CI, Python now Compiles and uses a local copy of
OpenSSL 1.1.0g for testing.
@end itemize

@node Build<31>,Windows<29>,Tests<30>,Python 3 7 0 beta 1
@anchor{whatsnew/changelog id288}@anchor{1351}
@subsubsection Build



@itemize -

@item 
bpo-32635@footnote{https://bugs.python.org/issue32635}: Fix segfault of the crypt module when libxcrypt is provided
instead of libcrypt at the system.

@item 
bpo-32598@footnote{https://bugs.python.org/issue32598}: Use autoconf to detect OpenSSL libs, headers and supported
features. The ax_check_openssl M4 macro uses pkg-config to locate OpenSSL
and falls back to manual search.

@item 
bpo-32593@footnote{https://bugs.python.org/issue32593}: Drop support of FreeBSD 9 and older.

@item 
bpo-29708@footnote{https://bugs.python.org/issue29708}: If the 
@geindex SOURCE_DATE_EPOCH
@geindex environment variable; SOURCE_DATE_EPOCH
@code{SOURCE_DATE_EPOCH} environment variable is set,
@ref{d2,,py_compile} will always create hash-based @code{.pyc} files.
@end itemize

@node Windows<29>,macOS<25>,Build<31>,Python 3 7 0 beta 1
@anchor{whatsnew/changelog id289}@anchor{1352}
@subsubsection Windows



@itemize -

@item 
bpo-32588@footnote{https://bugs.python.org/issue32588}: Create standalone _distutils_findvs module and add missing
_queue module to installer.

@item 
bpo-29911@footnote{https://bugs.python.org/issue29911}: Ensure separate Modify and Uninstall buttons are displayed.

@item 
bpo-32507@footnote{https://bugs.python.org/issue32507}: Use app-local UCRT install rather than the proper update for
old versions of Windows.
@end itemize

@node macOS<25>,C API<28>,Windows<29>,Python 3 7 0 beta 1
@anchor{whatsnew/changelog id290}@anchor{1353}
@subsubsection macOS



@itemize -

@item 
bpo-32726@footnote{https://bugs.python.org/issue32726}: Provide an additional, more modern macOS installer variant that
supports macOS 10.9+ systems in 64-bit mode only.  Upgrade the supplied
third-party libraries to OpenSSL 1.1.0g and to SQLite 3.22.0.  The 10.9+
installer now links with and supplies its own copy of Tcl/Tk 8.6.

@item 
bpo-28440@footnote{https://bugs.python.org/issue28440}: No longer add /Library/Python/3.x/site-packages to sys.path for
macOS framework builds to avoid future conflicts.
@end itemize

@node C API<28>,,macOS<25>,Python 3 7 0 beta 1
@anchor{whatsnew/changelog id291}@anchor{1354}
@subsubsection C API



@itemize -

@item 
bpo-32681@footnote{https://bugs.python.org/issue32681}: Fix uninitialized variable ‘res’ in the C implementation of
os.dup2. Patch by Stéphane Wirtel

@item 
bpo-10381@footnote{https://bugs.python.org/issue10381}: Add C API access to the @code{datetime.timezone} constructor and
@code{datetime.timzone.UTC} singleton.
@end itemize

@node Python 3 7 0 alpha 4,Python 3 7 0 alpha 3,Python 3 7 0 beta 1,Changelog
@anchor{whatsnew/changelog python-3-7-0-alpha-4}@anchor{1355}
@subsection Python 3.7.0 alpha 4


@emph{Release date: 2018-01-08}

@menu
* Core and Builtins: Core and Builtins<33>. 
* Library: Library<34>. 
* Documentation: Documentation<34>. 
* Tests: Tests<31>. 
* Windows: Windows<30>. 
* Tools/Demos: Tools/Demos<15>. 
* C API: C API<29>. 

@end menu

@node Core and Builtins<33>,Library<34>,,Python 3 7 0 alpha 4
@anchor{whatsnew/changelog id292}@anchor{1356}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-31975@footnote{https://bugs.python.org/issue31975}: The default warning filter list now starts with a
“default::DeprecationWarning:__main__” entry, so deprecation warnings are
once again shown by default in single-file scripts and at the interactive
prompt.

@item 
bpo-32226@footnote{https://bugs.python.org/issue32226}: @code{__class_getitem__} is now an automatic class method.

@item 
bpo-32399@footnote{https://bugs.python.org/issue32399}: Add AIX uuid library support for RFC4122 using uuid_create() in
libc.a

@item 
bpo-32390@footnote{https://bugs.python.org/issue32390}: Fix the compilation failure on AIX after the f_fsid field has
been added to the object returned by os.statvfs() (bpo-32143@footnote{https://bugs.python.org/issue32143}). Original
patch by Michael Felt.

@item 
bpo-32379@footnote{https://bugs.python.org/issue32379}: Make MRO computation faster when a class inherits from a single
base.

@item 
bpo-32259@footnote{https://bugs.python.org/issue32259}: The error message of a TypeError raised when unpack
non-iterable is now more specific.

@item 
bpo-27169@footnote{https://bugs.python.org/issue27169}: The @code{__debug__} constant is now optimized out at compile
time. This fixes also bpo-22091@footnote{https://bugs.python.org/issue22091}.

@item 
bpo-32329@footnote{https://bugs.python.org/issue32329}: The @ref{1357,,-R} option now turns on hash randomization when
the 
@geindex PYTHONHASHSEED
@geindex environment variable; PYTHONHASHSEED
@ref{c15,,PYTHONHASHSEED} environment variable is set to @code{0}.
Previously, the option was ignored. Moreover,
@code{sys.flags.hash_randomization} is now properly set to 0 when hash
randomization is turned off by @code{PYTHONHASHSEED=0}.

@item 
bpo-30416@footnote{https://bugs.python.org/issue30416}: The optimizer is now protected from spending much time doing
complex calculations and consuming much memory for creating large
constants in constant folding. Increased limits for constants that can be
produced in constant folding.

@item 
bpo-32282@footnote{https://bugs.python.org/issue32282}: Fix an unnecessary ifdef in the include of VersionHelpers.h in
socketmodule on Windows.

@item 
bpo-30579@footnote{https://bugs.python.org/issue30579}: Implement TracebackType.__new__ to allow Python-level creation
of traceback objects, and make TracebackType.tb_next mutable.

@item 
bpo-32260@footnote{https://bugs.python.org/issue32260}: Don’t byte swap the input keys to the SipHash algorithm on
big-endian platforms. This should ensure siphash gives consistent results
across platforms.

@item 
bpo-31506@footnote{https://bugs.python.org/issue31506}: Improve the error message logic for object.__new__ and
object.__init__. Patch by Sanyam Khurana.

@item 
bpo-20361@footnote{https://bugs.python.org/issue20361}: @code{-b} and @code{-bb} now inject @code{'default::BytesWarning'} and
@code{error::BytesWarning} entries into @code{sys.warnoptions}, ensuring that
they take precedence over any other warning filters configured via the
@code{-W} option or the @code{PYTHONWARNINGS} environment variable.

@item 
bpo-32230@footnote{https://bugs.python.org/issue32230}: @code{-X dev} now injects a @code{'default'} entry into
sys.warnoptions, ensuring that it behaves identically to actually passing
@code{-Wdefault} at the command line.

@item 
bpo-29240@footnote{https://bugs.python.org/issue29240}: Add a new UTF-8 mode: implementation of the 
@geindex Python Enhancement Proposals; PEP 540
PEP 540@footnote{https://www.python.org/dev/peps/pep-0540}.

@item 
bpo-32226@footnote{https://bugs.python.org/issue32226}: 
@geindex Python Enhancement Proposals; PEP 560
PEP 560@footnote{https://www.python.org/dev/peps/pep-0560}: Add support for @code{__mro_entries__} and
@code{__class_getitem__}. Implemented by Ivan Levkivskyi.

@item 
bpo-32225@footnote{https://bugs.python.org/issue32225}: 
@geindex Python Enhancement Proposals; PEP 562
PEP 562@footnote{https://www.python.org/dev/peps/pep-0562}: Add support for module @code{__getattr__} and
@code{__dir__}. Implemented by Ivan Levkivskyi.

@item 
bpo-31901@footnote{https://bugs.python.org/issue31901}: The @ref{b,,atexit} module now has its callback stored per
interpreter.

@item 
bpo-31650@footnote{https://bugs.python.org/issue31650}: Implement 
@geindex Python Enhancement Proposals; PEP 552
PEP 552@footnote{https://www.python.org/dev/peps/pep-0552} (Deterministic pycs). Python now supports
invalidating bytecode cache files bashed on a source content hash rather
than source last-modified time.

@item 
bpo-29469@footnote{https://bugs.python.org/issue29469}: Move constant folding from bytecode layer to AST layer.
Original patch by Eugene Toder.
@end itemize

@node Library<34>,Documentation<34>,Core and Builtins<33>,Python 3 7 0 alpha 4
@anchor{whatsnew/changelog id293}@anchor{1358}
@subsubsection Library



@itemize -

@item 
bpo-32506@footnote{https://bugs.python.org/issue32506}: Now that dict is defined as keeping insertion order, drop
OrderedDict and just use plain dict.

@item 
bpo-32279@footnote{https://bugs.python.org/issue32279}: Add params to dataclasses.make_dataclasses(): init, repr, eq,
order, hash, and frozen.  Pass them through to dataclass().

@item 
bpo-32278@footnote{https://bugs.python.org/issue32278}: Make type information optional on dataclasses.make_dataclass().
If omitted, the string ‘typing.Any’ is used.

@item 
bpo-32499@footnote{https://bugs.python.org/issue32499}: Add dataclasses.is_dataclass(obj), which returns True if obj is
a dataclass or an instance of one.

@item 
bpo-32468@footnote{https://bugs.python.org/issue32468}: Improve frame repr() to mention filename, code name and current
line number.

@item 
bpo-23749@footnote{https://bugs.python.org/issue23749}: asyncio: Implement loop.start_tls()

@item 
bpo-32441@footnote{https://bugs.python.org/issue32441}: Return the new file descriptor (i.e., the second argument) from
@code{os.dup2}. Previously, @code{None} was always returned.

@item 
bpo-32422@footnote{https://bugs.python.org/issue32422}: @code{functools.lru_cache} uses less memory (3 words for each
cached key) and takes about 1/3 time for cyclic GC.

@item 
bpo-31721@footnote{https://bugs.python.org/issue31721}: Prevent Python crash from happening when Future._log_traceback
is set to True manually.  Now it can only be set to False, or a ValueError
is raised.

@item 
bpo-32415@footnote{https://bugs.python.org/issue32415}: asyncio: Add Task.get_loop() and Future.get_loop()

@item 
bpo-26133@footnote{https://bugs.python.org/issue26133}: Don’t unsubscribe signals in asyncio UNIX event loop on
interpreter shutdown.

@item 
bpo-32363@footnote{https://bugs.python.org/issue32363}: Make asyncio.Task.set_exception() and set_result() raise
NotImplementedError. Task._step() and Future.__await__() raise proper
exceptions when they are in an invalid state, instead of raising an
AssertionError.

@item 
bpo-32357@footnote{https://bugs.python.org/issue32357}: Optimize asyncio.iscoroutine() and loop.create_task() for
non-native coroutines (e.g. async/await compiled with Cython).
‘loop.create_task(python_coroutine)’ used to be 20% faster than
‘loop.create_task(cython_coroutine)’.  Now, the latter is as fast.

@item 
bpo-32356@footnote{https://bugs.python.org/issue32356}: asyncio.transport.resume_reading() and pause_reading() are now
idempotent. New transport.is_reading() method is added.

@item 
bpo-32355@footnote{https://bugs.python.org/issue32355}: Optimize asyncio.gather(); now up to 15% faster.

@item 
bpo-32351@footnote{https://bugs.python.org/issue32351}: Use fastpath in asyncio.sleep if delay<0 (2x boost)

@item 
bpo-32348@footnote{https://bugs.python.org/issue32348}: Optimize asyncio.Future schedule/add/remove callback.  The
optimization shows 3-6% performance improvements of async/await code.

@item 
bpo-32331@footnote{https://bugs.python.org/issue32331}: Fix socket.settimeout() and socket.setblocking() to keep
socket.type as is. Fix socket.socket() constructor to reset any bit flags
applied to socket’s type.  This change only affects OSes that have
SOCK_NONBLOCK and/or SOCK_CLOEXEC.

@item 
bpo-32248@footnote{https://bugs.python.org/issue32248}: Add @ref{61b,,importlib.abc.ResourceReader} as an ABC for loaders
to provide a unified API for reading resources contained within packages.
Also add @ref{9a,,importlib.resources} as the port of
@code{importlib_resources}.

@item 
bpo-32311@footnote{https://bugs.python.org/issue32311}: Implement asyncio.create_task(coro) shortcut

@item 
bpo-32327@footnote{https://bugs.python.org/issue32327}: Convert asyncio functions that were documented as coroutines to
coroutines. Affected functions: loop.sock_sendall, loop.sock_recv,
loop.sock_accept, loop.getaddrinfo, loop.getnameinfo.

@item 
bpo-32323@footnote{https://bugs.python.org/issue32323}: @ref{88f,,urllib.parse.urlsplit()} does not convert zone-id
(scope) to lower case for scoped IPv6 addresses in hostnames now.

@item 
bpo-32302@footnote{https://bugs.python.org/issue32302}: Fix bdist_wininst of distutils for CRT v142: it binary
compatible with CRT v140.

@item 
bpo-29711@footnote{https://bugs.python.org/issue29711}: Fix @code{stop_serving} in asyncio proactor loop kill all
listening servers

@item 
bpo-32308@footnote{https://bugs.python.org/issue32308}: @ref{72e,,re.sub()} now replaces empty matches adjacent to a
previous non-empty match.

@item 
bpo-29970@footnote{https://bugs.python.org/issue29970}: Abort asyncio SSLProtocol connection if handshake not complete
within 10 seconds.

@item 
bpo-32314@footnote{https://bugs.python.org/issue32314}: Implement asyncio.run().

@item 
bpo-17852@footnote{https://bugs.python.org/issue17852}: Revert incorrect fix based on misunderstanding of
_Py_PyAtExit() semantics.

@item 
bpo-32296@footnote{https://bugs.python.org/issue32296}: Implement asyncio._get_running_loop() and get_event_loop() in
C. This makes them 4x faster.

@item 
bpo-32250@footnote{https://bugs.python.org/issue32250}: Implement @code{asyncio.current_task()} and
@code{asyncio.all_tasks()}. Add helpers intended to be used by alternative
task implementations: @code{asyncio._register_task}, @code{asyncio._enter_task},
@code{asyncio._leave_task} and @code{asyncio._unregister_task}. Deprecate
@code{asyncio.Task.current_task()} and @code{asyncio.Task.all_tasks()}.

@item 
bpo-32255@footnote{https://bugs.python.org/issue32255}: A single empty field is now always quoted when written into a
CSV file. This allows to distinguish an empty row from a row consisting of
a single empty field. Patch by Licht Takeuchi.

@item 
bpo-32277@footnote{https://bugs.python.org/issue32277}: Raise @code{NotImplementedError} instead of @code{SystemError} on
platforms where @code{chmod(..., follow_symlinks=False)} is not supported.
Patch by Anthony Sottile.

@item 
bpo-30050@footnote{https://bugs.python.org/issue30050}: New argument warn_on_full_buffer to signal.set_wakeup_fd lets
you control whether Python prints a warning on stderr when the wakeup fd
buffer overflows.

@item 
bpo-29137@footnote{https://bugs.python.org/issue29137}: The @code{fpectl} library has been removed. It was never enabled
by default, never worked correctly on x86-64, and it changed the Python
ABI in ways that caused unexpected breakage of C extensions.

@item 
bpo-32273@footnote{https://bugs.python.org/issue32273}: Move asyncio.test_utils to test.test_asyncio.

@item 
bpo-32272@footnote{https://bugs.python.org/issue32272}: Remove asyncio.async() function.

@item 
bpo-32269@footnote{https://bugs.python.org/issue32269}: Add asyncio.get_running_loop() function.

@item 
bpo-32265@footnote{https://bugs.python.org/issue32265}: All class and static methods of builtin types now are correctly
classified by inspect.classify_class_attrs() and grouped in pydoc ouput.
Added types.ClassMethodDescriptorType for unbound class methods of builtin
types.

@item 
bpo-32253@footnote{https://bugs.python.org/issue32253}: Deprecate @code{yield from lock}, @code{await lock}, @code{with (yield
from lock)} and @code{with await lock} for asyncio synchronization
primitives.

@item 
bpo-22589@footnote{https://bugs.python.org/issue22589}: Changed MIME type of .bmp from ‘image/x-ms-bmp’ to ‘image/bmp’

@item 
bpo-32193@footnote{https://bugs.python.org/issue32193}: Convert asyncio to use @emph{async/await} syntax. Old styled @code{yield
from} is still supported too.

@item 
bpo-32206@footnote{https://bugs.python.org/issue32206}: Add support to run modules with pdb

@item 
bpo-32227@footnote{https://bugs.python.org/issue32227}: @code{functools.singledispatch} now supports registering
implementations using type annotations.

@item 
bpo-15873@footnote{https://bugs.python.org/issue15873}: Added new alternate constructors
@ref{650,,datetime.datetime.fromisoformat()},
@ref{1359,,datetime.time.fromisoformat()} and
@ref{135a,,datetime.date.fromisoformat()} as the inverse operation of each
classes’s respective @code{isoformat} methods.

@item 
bpo-32199@footnote{https://bugs.python.org/issue32199}: The getnode() ip getter now uses ‘ip link’ instead of ‘ip link
list’.

@item 
bpo-32143@footnote{https://bugs.python.org/issue32143}: os.statvfs() includes the f_fsid field from statvfs(2)

@item 
bpo-26439@footnote{https://bugs.python.org/issue26439}: Fix ctypes.util.find_library() for AIX by implementing
ctypes._aix.find_library() Patch by: Michael Felt

@item 
bpo-31993@footnote{https://bugs.python.org/issue31993}: The pickler now uses less memory when serializing large bytes
and str objects into a file.  Pickles created with protocol 4 will require
less memory for unpickling large bytes and str objects.

@item 
bpo-27456@footnote{https://bugs.python.org/issue27456}: Ensure TCP_NODELAY is set on Linux. Tests by Victor Stinner.

@item 
bpo-31778@footnote{https://bugs.python.org/issue31778}: ast.literal_eval() is now more strict. Addition and subtraction
of arbitrary numbers no longer allowed.

@item 
bpo-31802@footnote{https://bugs.python.org/issue31802}: Importing native path module (@code{posixpath}, @code{ntpath}) now
works even if the @code{os} module still is not imported.

@item 
bpo-30241@footnote{https://bugs.python.org/issue30241}: Add contextlib.AbstractAsyncContextManager. Patch by Jelle
Zijlstra.

@item 
bpo-31699@footnote{https://bugs.python.org/issue31699}: Fix deadlocks in
@ref{3da,,concurrent.futures.ProcessPoolExecutor} when task arguments or
results cause pickling or unpickling errors. This should make sure that
calls to the @code{ProcessPoolExecutor} API always eventually return.

@item 
bpo-15216@footnote{https://bugs.python.org/issue15216}: @code{TextIOWrapper.reconfigure()} supports changing @emph{encoding},
@emph{errors}, and @emph{newline}.
@end itemize

@node Documentation<34>,Tests<31>,Library<34>,Python 3 7 0 alpha 4
@anchor{whatsnew/changelog id294}@anchor{135b}
@subsubsection Documentation



@itemize -

@item 
bpo-32418@footnote{https://bugs.python.org/issue32418}: Add get_loop() method to Server and AbstractServer classes.
@end itemize

@node Tests<31>,Windows<30>,Documentation<34>,Python 3 7 0 alpha 4
@anchor{whatsnew/changelog id295}@anchor{135c}
@subsubsection Tests



@itemize -

@item 
bpo-32252@footnote{https://bugs.python.org/issue32252}: Fix faulthandler_suppress_crash_report() used to prevent core
dump files when testing crashes. getrlimit() returns zero on success.

@item 
bpo-32002@footnote{https://bugs.python.org/issue32002}: Adjust C locale coercion testing for the empty locale and POSIX
locale cases to more readily adjust to platform dependent behaviour.
@end itemize

@node Windows<30>,Tools/Demos<15>,Tests<31>,Python 3 7 0 alpha 4
@anchor{whatsnew/changelog id296}@anchor{135d}
@subsubsection Windows



@itemize -

@item 
bpo-19764@footnote{https://bugs.python.org/issue19764}: Implement support for @code{subprocess.Popen(close_fds=True)} on
Windows. Patch by Segev Finer.
@end itemize

@node Tools/Demos<15>,C API<29>,Windows<30>,Python 3 7 0 alpha 4
@anchor{whatsnew/changelog id297}@anchor{135e}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-24960@footnote{https://bugs.python.org/issue24960}: 2to3 and lib2to3 can now read pickled grammar files using
pkgutil.get_data() rather than probing the filesystem. This lets 2to3 and
lib2to3 work when run from a zipfile.
@end itemize

@node C API<29>,,Tools/Demos<15>,Python 3 7 0 alpha 4
@anchor{whatsnew/changelog id298}@anchor{135f}
@subsubsection C API



@itemize -

@item 
bpo-32030@footnote{https://bugs.python.org/issue32030}: Py_Initialize() doesn’t reset the memory allocators to default
if the @code{PYTHONMALLOC} environment variable is not set.

@item 
bpo-29084@footnote{https://bugs.python.org/issue29084}: Undocumented C API for OrderedDict has been excluded from the
limited C API. It was added by mistake and actually never worked in the
limited C API.

@item 
bpo-32264@footnote{https://bugs.python.org/issue32264}: Moved the pygetopt.h header into internal/, since it has no
public APIs.

@item 
bpo-32241@footnote{https://bugs.python.org/issue32241}: @ref{203,,Py_SetProgramName()} and @ref{204,,Py_SetPythonHome()} now
take the @code{const wchar *} arguments instead of @code{wchar *}.
@end itemize

@node Python 3 7 0 alpha 3,Python 3 7 0 alpha 2,Python 3 7 0 alpha 4,Changelog
@anchor{whatsnew/changelog python-3-7-0-alpha-3}@anchor{1360}
@subsection Python 3.7.0 alpha 3


@emph{Release date: 2017-12-05}

@menu
* Core and Builtins: Core and Builtins<34>. 
* Library: Library<35>. 
* Documentation: Documentation<35>. 
* Tests: Tests<32>. 
* Build: Build<32>. 
* Windows: Windows<31>. 
* macOS: macOS<26>. 
* IDLE: IDLE<30>. 
* Tools/Demos: Tools/Demos<16>. 
* C API: C API<30>. 

@end menu

@node Core and Builtins<34>,Library<35>,,Python 3 7 0 alpha 3
@anchor{whatsnew/changelog id299}@anchor{1361}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-32176@footnote{https://bugs.python.org/issue32176}: co_flags.CO_NOFREE is now always set correctly by the code
object constructor based on freevars and cellvars, rather than needing to
be set correctly by the caller. This ensures it will be cleared
automatically when additional cell references are injected into a modified
code object and function.

@item 
bpo-10544@footnote{https://bugs.python.org/issue10544}: Yield expressions are now deprecated in comprehensions and
generator expressions. They are still permitted in the definition of the
outermost iterable, as that is evaluated directly in the enclosing scope.

@item 
bpo-32137@footnote{https://bugs.python.org/issue32137}: The repr of deeply nested dict now raises a RecursionError
instead of crashing due to a stack overflow.

@item 
bpo-32096@footnote{https://bugs.python.org/issue32096}: Revert memory allocator changes in the C API: move structures
back from _PyRuntime to Objects/obmalloc.c. The memory allocators are once
again initialized statically, and so PyMem_RawMalloc() and
Py_DecodeLocale() can be called before _PyRuntime_Initialize().

@item 
bpo-32043@footnote{https://bugs.python.org/issue32043}: Add a new “developer mode”: new “-X dev” command line option to
enable debug checks at runtime.

@item 
bpo-32023@footnote{https://bugs.python.org/issue32023}: SyntaxError is now correctly raised when a generator expression
without parenthesis is used instead of an inheritance list in a class
definition. The duplication of the parentheses can be omitted only on
calls.

@item 
bpo-32012@footnote{https://bugs.python.org/issue32012}: SyntaxError is now correctly raised when a generator expression
without parenthesis is passed as an argument, but followed by a trailing
comma. A generator expression always needs to be directly inside a set of
parentheses and cannot have a comma on either side.

@item 
bpo-28180@footnote{https://bugs.python.org/issue28180}: A new internal @code{_Py_SetLocaleFromEnv(category)} helper
function has been added in order to improve the consistency of behaviour
across different @code{libc} implementations (e.g. Android doesn’t support
setting the locale from the environment by default).

@item 
bpo-31949@footnote{https://bugs.python.org/issue31949}: Fixed several issues in printing tracebacks
(PyTraceBack_Print()). Setting sys.tracebacklimit to 0 or less now
suppresses printing tracebacks. Setting sys.tracebacklimit to None now
causes using the default limit. Setting sys.tracebacklimit to an integer
larger than LONG_MAX now means using the limit LONG_MAX rather than the
default limit. Fixed integer overflows in the case of more than @code{2**31}
traceback items on Windows. Fixed output errors handling.

@item 
bpo-30696@footnote{https://bugs.python.org/issue30696}: Fix the interactive interpreter looping endlessly when no
memory.

@item 
bpo-20047@footnote{https://bugs.python.org/issue20047}: Bytearray methods partition() and rpartition() now accept only
bytes-like objects as separator, as documented.  In particular they now
raise TypeError rather of returning a bogus result when an integer is
passed as a separator.

@item 
bpo-21720@footnote{https://bugs.python.org/issue21720}: BytesWarning no longer emitted when the @emph{fromlist} argument of
@code{__import__()} or the @code{__all__} attribute of the module contain bytes
instances.

@item 
bpo-31845@footnote{https://bugs.python.org/issue31845}: Environment variables are once more read correctly at
interpreter startup.

@item 
bpo-28936@footnote{https://bugs.python.org/issue28936}: Ensure that lexically first syntax error involving a parameter
and @code{global} or @code{nonlocal} is detected first at a given scope. Patch
by Ivan Levkivskyi.

@item 
bpo-31825@footnote{https://bugs.python.org/issue31825}: Fixed OverflowError in the ‘unicode-escape’ codec and in
codecs.escape_decode() when decode an escaped non-ascii byte.

@item 
bpo-31618@footnote{https://bugs.python.org/issue31618}: The per-frame tracing logic added in 3.7a1 has been altered so
that @code{frame->f_lineno} is updated before either @code{"line"} or
@code{"opcode"} events are emitted. Previously, opcode events were emitted
first, and therefore would occasionally see stale line numbers on the
frame. The behavior of this feature has changed slightly as a result: when
both @code{f_trace_lines} and @code{f_trace_opcodes} are enabled, line events
now occur first.

@item 
bpo-28603@footnote{https://bugs.python.org/issue28603}: Print the full context/cause chain of exceptions on interpreter
exit, even if an exception in the chain is unhashable or compares equal to
later ones. Patch by Zane Bitter.

@item 
bpo-31786@footnote{https://bugs.python.org/issue31786}: Fix timeout rounding in the select module to round correctly
negative timeouts between -1.0 and 0.0. The functions now block waiting
for events as expected. Previously, the call was incorrectly non-blocking.
Patch by Pablo Galindo.

@item 
bpo-31781@footnote{https://bugs.python.org/issue31781}: Prevent crashes when calling methods of an uninitialized
@code{zipimport.zipimporter} object. Patch by Oren Milman.

@item 
bpo-30399@footnote{https://bugs.python.org/issue30399}: Standard repr() of BaseException with a single argument no
longer contains redundant trailing comma.

@item 
bpo-31626@footnote{https://bugs.python.org/issue31626}: Fixed a bug in debug memory allocator.  There was a write to
freed memory after shrinking a memory block.

@item 
bpo-30817@footnote{https://bugs.python.org/issue30817}: @code{PyErr_PrintEx()} clears now the ignored exception that may be
raised by @code{_PySys_SetObjectId()}, for example when no memory.
@end itemize

@node Library<35>,Documentation<35>,Core and Builtins<34>,Python 3 7 0 alpha 3
@anchor{whatsnew/changelog id300}@anchor{1362}
@subsubsection Library



@itemize -

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Two minor fixes for @code{typing} module: allow shallow copying
instances of generic classes, improve interaction of @code{__init_subclass__}
with generics. Original PRs by Ivan Levkivskyi.

@item 
bpo-32214@footnote{https://bugs.python.org/issue32214}: PEP 557, Data Classes. Provides a decorator which adds
boilerplate methods to classes which use type annotations so specify
fields.

@item 
bpo-27240@footnote{https://bugs.python.org/issue27240}: The header folding algorithm for the new email policies has
been rewritten, which also fixes bpo-30788@footnote{https://bugs.python.org/issue30788}, bpo-31831@footnote{https://bugs.python.org/issue31831}, and bpo-32182@footnote{https://bugs.python.org/issue32182}.  In
particular, RFC2231 folding is now done correctly.

@item 
bpo-32186@footnote{https://bugs.python.org/issue32186}: io.FileIO.readall() and io.FileIO.read() now release the GIL
when getting the file size. Fixed hang of all threads with inaccessible
NFS server. Patch by Nir Soffer.

@item 
bpo-321010@footnote{https://bugs.python.org/issue321010}: Add @code{sys.flags.dev_mode} flag

@item 
bpo-32154@footnote{https://bugs.python.org/issue32154}: The @code{asyncio.windows_utils.socketpair()} function has been
removed: use directly @ref{731,,socket.socketpair()} which is available on all
platforms since Python 3.5 (before, it wasn’t available on Windows).
@code{asyncio.windows_utils.socketpair()} was just an alias to
@code{socket.socketpair} on Python 3.5 and newer.

@item 
bpo-32089@footnote{https://bugs.python.org/issue32089}: warnings: In development (-X dev) and debug mode (pydebug
build), use the “default” action for ResourceWarning, rather than the
“always” action, in the default warnings filters.

@item 
bpo-32107@footnote{https://bugs.python.org/issue32107}: @code{uuid.getnode()} now preferentially returns universally
administered MAC addresses if available, over locally administered MAC
addresses.  This makes a better guarantee for global uniqueness of UUIDs
returned from @code{uuid.uuid1()}.  If only locally administered MAC
addresses are available, the first such one found is returned.

@item 
bpo-23033@footnote{https://bugs.python.org/issue23033}: Wildcard is now supported in hostname when it is one and only
character in the left most segment of hostname in second argument of
@ref{333,,ssl.match_hostname()}.  Patch by Mandeep Singh.

@item 
bpo-12239@footnote{https://bugs.python.org/issue12239}: Make @ref{1292,,msilib.SummaryInformation.GetProperty()} return
@code{None} when the value of property is @code{VT_EMPTY}.  Initial patch by
Mark Mc Mahon.

@item 
bpo-28334@footnote{https://bugs.python.org/issue28334}: Use @ref{520,,os.path.expanduser()} to find the @code{~/.netrc} file in
@ref{1137,,netrc.netrc}.  If it does not exist, @ref{a37,,FileNotFoundError} is
raised.  Patch by Dimitri Merejkowsky.

@item 
bpo-32121@footnote{https://bugs.python.org/issue32121}: Made @code{tracemalloc.Traceback} behave more like the traceback
module, sorting the frames from oldest to most recent.
@code{Traceback.format()} now accepts negative @emph{limit}, truncating the result
to the @code{abs(limit)} oldest frames. To get the old behaviour, one can use
the new @emph{most_recent_first} argument to @code{Traceback.format()}. (Patch by
Jesse Bakker.)

@item 
bpo-31325@footnote{https://bugs.python.org/issue31325}: Fix wrong usage of @ref{3e4,,collections.namedtuple()} in the
@ref{1363,,RobotFileParser.parse()}
method. Initial patch by Robin Wellner.

@item 
bpo-12382@footnote{https://bugs.python.org/issue12382}: @ref{11d5,,msilib.OpenDatabase()} now raises a better exception
message when it couldn’t open or create an MSI file.  Initial patch by
William Tisäter.

@item 
bpo-19610@footnote{https://bugs.python.org/issue19610}: @code{setup()} now warns about invalid types for some fields. The
@code{distutils.dist.Distribution} class now warns when @code{classifiers},
@code{keywords} and @code{platforms} fields are not specified as a list or a
string.

@item 
bpo-32071@footnote{https://bugs.python.org/issue32071}: Added the @code{-k} command-line option to @code{python -m unittest}
to run only tests that match the given pattern(s).

@item 
bpo-10049@footnote{https://bugs.python.org/issue10049}: Added @emph{nullcontext} no-op context manager to contextlib. This
provides a simpler and faster alternative to ExitStack() when handling
optional context managers.

@item 
bpo-28684@footnote{https://bugs.python.org/issue28684}: The new test.support.skip_unless_bind_unix_socket() decorator
is used here to skip asyncio tests that fail because the platform lacks a
functional bind() function for unix domain sockets (as it is the case for
non root users on the recent Android versions that run now SELinux in
enforcing mode).

@item 
bpo-32110@footnote{https://bugs.python.org/issue32110}: @code{codecs.StreamReader.read(n)} now returns not more than @emph{n}
characters/bytes for non-negative @emph{n}. This makes it compatible with
@code{read()} methods of other file-like objects.

@item 
bpo-27535@footnote{https://bugs.python.org/issue27535}: The warnings module doesn’t leak memory anymore in the hidden
warnings registry for the “ignore” action of warnings filters.
warn_explicit() function doesn’t add the warning key to the registry
anymore for the “ignore” action.

@item 
bpo-32088@footnote{https://bugs.python.org/issue32088}: warnings:  When Python is build is debug mode (@code{Py_DEBUG}),
@ref{1b2,,DeprecationWarning}, @ref{3a9,,PendingDeprecationWarning} and
@ref{304,,ImportWarning} warnings are now displayed by default.

@item 
bpo-1647489@footnote{https://bugs.python.org/issue1647489}: Fixed searching regular expression patterns that could match
an empty string. Non-empty string can now be correctly found after
matching an empty string.

@item 
bpo-25054@footnote{https://bugs.python.org/issue25054}: Added support of splitting on a pattern that could match an
empty string.

@item 
bpo-32072@footnote{https://bugs.python.org/issue32072}: Fixed issues with binary plists: Fixed saving bytearrays.
Identical objects will be saved only once. Equal references will be load
as identical objects. Added support for saving and loading recursive data
structures.

@item 
bpo-32069@footnote{https://bugs.python.org/issue32069}: Drop legacy SSL transport from asyncio, ssl.MemoryBIO is always
used anyway.

@item 
bpo-32066@footnote{https://bugs.python.org/issue32066}: asyncio: Support pathlib.Path in create_unix_connection; sock
arg should be optional

@item 
bpo-32046@footnote{https://bugs.python.org/issue32046}: Updates 2to3 to convert from operator.isCallable(obj) to
callable(obj). Patch by Dong-hee Na.

@item 
bpo-32018@footnote{https://bugs.python.org/issue32018}: inspect.signature should follow 
@geindex Python Enhancement Proposals; PEP 8
PEP 8@footnote{https://www.python.org/dev/peps/pep-0008}, if the parameter has
an annotation and a default value. Patch by Dong-hee Na.

@item 
bpo-32025@footnote{https://bugs.python.org/issue32025}: Add time.thread_time() and time.thread_time_ns()

@item 
bpo-32037@footnote{https://bugs.python.org/issue32037}: Integers that fit in a signed 32-bit integer will be now
pickled with protocol 0 using the INT opcode.  This will decrease the size
of a pickle, speed up pickling and unpickling, and make these integers be
unpickled as int instances in Python 2.

@item 
bpo-32034@footnote{https://bugs.python.org/issue32034}: Make asyncio.IncompleteReadError and LimitOverrunError
pickleable.

@item 
bpo-32015@footnote{https://bugs.python.org/issue32015}: Fixed the looping of asyncio in the case of reconnection the
socket during waiting async read/write from/to the socket.

@item 
bpo-32011@footnote{https://bugs.python.org/issue32011}: Restored support of loading marshal files with the TYPE_INT64
code. These files can be produced in Python 2.7.

@item 
bpo-28369@footnote{https://bugs.python.org/issue28369}: Enhance add_reader/writer check that socket is not used by some
transport. Before, only cases when add_reader/writer were called with an
int FD were supported.  Now the check is implemented correctly for all
file-like objects.

@item 
bpo-31976@footnote{https://bugs.python.org/issue31976}: Fix race condition when flushing a file is slow, which can
cause a segfault if closing the file from another thread.

@item 
bpo-31985@footnote{https://bugs.python.org/issue31985}: Formally deprecated aifc.openfp, sunau.openfp, and wave.openfp.
Since change 7bc817d5ba917528e8bd07ec461c635291e7b06a in 1993, openfp in
each of the three modules had been pointing to that module’s open function
as a matter of backwards compatibility, though it had been both untested
and undocumented.

@item 
bpo-21862@footnote{https://bugs.python.org/issue21862}: cProfile command line now accepts @code{-m module_name} as an
alternative to script path. Patch by Sanyam Khurana.

@item 
bpo-31970@footnote{https://bugs.python.org/issue31970}: Reduce performance overhead of asyncio debug mode.

@item 
bpo-31843@footnote{https://bugs.python.org/issue31843}: @emph{database} argument of sqlite3.connect() now accepts a
@ref{63d,,path-like object}, instead of just a string.

@item 
bpo-31945@footnote{https://bugs.python.org/issue31945}: Add Configurable @emph{blocksize} to @code{HTTPConnection} and
@code{HTTPSConnection} for improved upload throughput.  Patch by Nir Soffer.

@item 
bpo-31943@footnote{https://bugs.python.org/issue31943}: Add a @code{cancelled()} method to @ref{1364,,asyncio.Handle}.  Patch
by Marat Sharafutdinov.

@item 
bpo-9678@footnote{https://bugs.python.org/issue9678}: Fixed determining the MAC address in the uuid module: Using
ifconfig on NetBSD and OpenBSD. Using arp on Linux, FreeBSD, NetBSD and
OpenBSD. Based on patch by Takayuki Shimizukawa.

@item 
bpo-30057@footnote{https://bugs.python.org/issue30057}: Fix potential missed signal in signal.signal().

@item 
bpo-31933@footnote{https://bugs.python.org/issue31933}: Fix Blake2 params leaf_size and node_offset on big endian
platforms. Patch by Jack O’Connor.

@item 
bpo-21423@footnote{https://bugs.python.org/issue21423}: Add an initializer argument to @{Process,Thread@}PoolExecutor

@item 
bpo-31927@footnote{https://bugs.python.org/issue31927}: Fixed compilation of the socket module on NetBSD 8.  Fixed
assertion failure or reading arbitrary data when parse a AF_BLUETOOTH
address on NetBSD and DragonFly BSD.

@item 
bpo-27666@footnote{https://bugs.python.org/issue27666}: Fixed stack corruption in curses.box() and curses.ungetmouse()
when the size of types chtype or mmask_t is less than the size of C long.
curses.box() now accepts characters as arguments.  Based on patch by Steve
Fink.

@item 
bpo-31917@footnote{https://bugs.python.org/issue31917}: Add 3 new clock identifiers: @ref{6bf,,time.CLOCK_BOOTTIME},
@ref{6c0,,time.CLOCK_PROF} and @ref{6c1,,time.CLOCK_UPTIME}.

@item 
bpo-31897@footnote{https://bugs.python.org/issue31897}: plistlib now catches more errors when read binary plists and
raises InvalidFileException instead of unexpected exceptions.

@item 
bpo-25720@footnote{https://bugs.python.org/issue25720}: Fix the method for checking pad state of curses WINDOW. Patch
by Masayuki Yamamoto.

@item 
bpo-31893@footnote{https://bugs.python.org/issue31893}: Fixed the layout of the kqueue_event structure on OpenBSD and
NetBSD. Fixed the comparison of the kqueue_event objects.

@item 
bpo-31891@footnote{https://bugs.python.org/issue31891}: Fixed building the curses module on NetBSD.

@item 
bpo-31884@footnote{https://bugs.python.org/issue31884}: added required constants to subprocess module for setting
priority on windows

@item 
bpo-28281@footnote{https://bugs.python.org/issue28281}: Remove year (1-9999) limits on the Calendar.weekday() function.
Patch by Mark Gollahon.

@item 
bpo-31702@footnote{https://bugs.python.org/issue31702}: crypt.mksalt() now allows to specify the number of rounds for
SHA-256 and SHA-512 hashing.

@item 
bpo-30639@footnote{https://bugs.python.org/issue30639}: @ref{1365,,inspect.getfile()} no longer computes the repr of unknown
objects to display in an error message, to protect against badly behaved
custom reprs.

@item 
bpo-30768@footnote{https://bugs.python.org/issue30768}: Fix the pthread+semaphore implementation of
PyThread_acquire_lock_timed() when called with timeout > 0 and
intr_flag=0: recompute the timeout if sem_timedwait() is interrupted by a
signal (EINTR). See also the 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475}.

@item 
bpo-31854@footnote{https://bugs.python.org/issue31854}: Add @code{mmap.ACCESS_DEFAULT} constant.

@item 
bpo-31834@footnote{https://bugs.python.org/issue31834}: Use optimized code for BLAKE2 only with SSSE3+. The pure SSE2
implementation is slower than the pure C reference implementation.

@item 
bpo-28292@footnote{https://bugs.python.org/issue28292}: Calendar.itermonthdates() will now consistently raise an
exception when a date falls outside of the 0001-01-01 through 9999-12-31
range.  To support applications that cannot tolerate such exceptions, the
new methods itermonthdays3() and itermonthdays4() are added.  The new
methods return tuples and are not restricted by the range supported by
datetime.date.

@item 
bpo-28564@footnote{https://bugs.python.org/issue28564}: The shutil.rmtree() function has been sped up to 20–40%. This
was done using the os.scandir() function.

@item 
bpo-28416@footnote{https://bugs.python.org/issue28416}: Instances of pickle.Pickler subclass with the persistent_id()
method and pickle.Unpickler subclass with the persistent_load() method no
longer create reference cycles.

@item 
bpo-31653@footnote{https://bugs.python.org/issue31653}: Don’t release the GIL if we can acquire a multiprocessing
semaphore immediately.

@item 
bpo-28326@footnote{https://bugs.python.org/issue28326}: Fix multiprocessing.Process when stdout and/or stderr is closed
or None.

@item 
bpo-20825@footnote{https://bugs.python.org/issue20825}: Add @code{subnet_of} and @code{superset_of} containment tests to
@ref{66f,,ipaddress.IPv6Network} and @ref{670,,ipaddress.IPv4Network}. Patch
by Michel Albert and Cheryl Sabella.

@item 
bpo-31827@footnote{https://bugs.python.org/issue31827}: Remove the os.stat_float_times() function. It was introduced in
Python 2.3 for backward compatibility with Python 2.2, and was deprecated
since Python 3.1.

@item 
bpo-31756@footnote{https://bugs.python.org/issue31756}: Add a @code{subprocess.Popen(text=False)} keyword argument to
@ref{f4,,subprocess} functions to be more explicit about when the library should
attempt to decode outputs into text. Patch by Andrew Clegg.

@item 
bpo-31819@footnote{https://bugs.python.org/issue31819}: Add AbstractEventLoop.sock_recv_into().

@item 
bpo-31457@footnote{https://bugs.python.org/issue31457}: If nested log adapters are used, the inner @code{process()}
methods are no longer omitted.

@item 
bpo-31457@footnote{https://bugs.python.org/issue31457}: The @code{manager} property on LoggerAdapter objects is now
properly settable.

@item 
bpo-31806@footnote{https://bugs.python.org/issue31806}: Fix timeout rounding in time.sleep(), threading.Lock.acquire()
and socket.socket.settimeout() to round correctly negative timeouts
between -1.0 and 0.0. The functions now block waiting for events as
expected. Previously, the call was incorrectly non-blocking. Patch by
Pablo Galindo.

@item 
bpo-31803@footnote{https://bugs.python.org/issue31803}: time.clock() and time.get_clock_info(‘clock’) now emit a
DeprecationWarning warning.

@item 
bpo-31800@footnote{https://bugs.python.org/issue31800}: Extended support for parsing UTC offsets. strptime ‘%z’ can now
parse the output generated by datetime.isoformat, including seconds and
microseconds.

@item 
bpo-28603@footnote{https://bugs.python.org/issue28603}: traceback: Fix a TypeError that occurred during printing of
exception tracebacks when either the current exception or an exception in
its context/cause chain is unhashable. Patch by Zane Bitter.

@item 
bpo-30541@footnote{https://bugs.python.org/issue30541}: Add new function to seal a mock and prevent the automatically
creation of child mocks. Patch by Mario Corchero.

@item 
bpo-31784@footnote{https://bugs.python.org/issue31784}: Implement the 
@geindex Python Enhancement Proposals; PEP 564
PEP 564@footnote{https://www.python.org/dev/peps/pep-0564}, add new 6 new functions with
nanosecond resolution to the @ref{10a,,time} module:
@ref{5fd,,clock_gettime_ns()}, @ref{5fe,,clock_settime_ns()},
@ref{5ff,,monotonic_ns()}, @ref{600,,perf_counter_ns()},
@ref{601,,process_time_ns()}, @ref{602,,time_ns()}.

@item 
bpo-30143@footnote{https://bugs.python.org/issue30143}: 2to3 now generates a code that uses abstract collection classes
from collections.abc rather than collections.

@item 
bpo-31770@footnote{https://bugs.python.org/issue31770}: Prevent a crash when calling the @code{__init__()} method of a
@code{sqlite3.Cursor} object more than once. Patch by Oren Milman.

@item 
bpo-31764@footnote{https://bugs.python.org/issue31764}: Prevent a crash in @code{sqlite3.Cursor.close()} in case the
@code{Cursor} object is uninitialized. Patch by Oren Milman.

@item 
bpo-31752@footnote{https://bugs.python.org/issue31752}: Fix possible crash in timedelta constructor called with custom
integers.

@item 
bpo-31620@footnote{https://bugs.python.org/issue31620}: an empty asyncio.Queue now doesn’t leak memory when queue.get
pollers timeout

@item 
bpo-31690@footnote{https://bugs.python.org/issue31690}: Allow the flags re.ASCII, re.LOCALE, and re.UNICODE to be used
as group flags for regular expressions.

@item 
bpo-30349@footnote{https://bugs.python.org/issue30349}: FutureWarning is now emitted if a regular expression contains
character set constructs that will change semantically in the future
(nested sets and set operations).

@item 
bpo-31664@footnote{https://bugs.python.org/issue31664}: Added support for the Blowfish hashing in the crypt module.

@item 
bpo-31632@footnote{https://bugs.python.org/issue31632}: Fix method set_protocol() of class _SSLProtocolTransport in
asyncio module. This method was previously modifying a wrong reference to
the protocol.

@item 
bpo-15037@footnote{https://bugs.python.org/issue15037}: Added a workaround for getkey() in curses for ncurses 5.7 and
earlier.

@item 
bpo-31307@footnote{https://bugs.python.org/issue31307}: Allow use of bytes objects for arguments to
@ref{1366,,configparser.ConfigParser.read()}. Patch by Vincent Michel.

@item 
bpo-31334@footnote{https://bugs.python.org/issue31334}: Fix @code{poll.poll([timeout])} in the @code{select} module for
arbitrary negative timeouts on all OSes where it can only be a
non-negative integer or -1. Patch by Riccardo Coccioli.

@item 
bpo-31310@footnote{https://bugs.python.org/issue31310}: multiprocessing’s semaphore tracker should be launched again if
crashed.

@item 
bpo-31308@footnote{https://bugs.python.org/issue31308}: Make multiprocessing’s forkserver process immune to Ctrl-C and
other user interruptions. If it crashes, restart it when necessary.

@item 
bpo-31245@footnote{https://bugs.python.org/issue31245}: Added support for AF_UNIX socket in asyncio
@code{create_datagram_endpoint}.

@item 
bpo-30553@footnote{https://bugs.python.org/issue30553}: Add HTTP/2 status code 421 (Misdirected Request) to
@ref{3ef,,http.HTTPStatus}. Patch by Vitor Pereira.
@end itemize

@node Documentation<35>,Tests<32>,Library<35>,Python 3 7 0 alpha 3
@anchor{whatsnew/changelog id301}@anchor{1367}
@subsubsection Documentation



@itemize -

@item 
bpo-32105@footnote{https://bugs.python.org/issue32105}: Added asyncio.BaseEventLoop.connect_accepted_socket
versionadded marker.
@end itemize

@node Tests<32>,Build<32>,Documentation<35>,Python 3 7 0 alpha 3
@anchor{whatsnew/changelog id302}@anchor{1368}
@subsubsection Tests



@itemize -

@item 
bpo-31380@footnote{https://bugs.python.org/issue31380}: Skip test_httpservers test_undecodable_file on macOS: fails on
APFS.

@item 
bpo-31705@footnote{https://bugs.python.org/issue31705}: Skip test_socket.test_sha256() on Linux kernel older than 4.5.
The test fails with ENOKEY on kernel 3.10 (on ppc64le). A fix was merged
into the kernel 4.5.

@item 
bpo-32138@footnote{https://bugs.python.org/issue32138}: Skip on Android test_faulthandler tests that raise SIGSEGV and
remove the test.support.requires_android_level decorator.

@item 
bpo-32136@footnote{https://bugs.python.org/issue32136}: The runtime embedding tests have been split out from
@code{Lib/test/test_capi.py} into a new @code{Lib/test/test_embed.py} file.

@item 
bpo-28668@footnote{https://bugs.python.org/issue28668}: test.support.requires_multiprocessing_queue is removed. Skip
tests with test.support.import_module(‘multiprocessing.synchronize’)
instead when the semaphore implementation is broken or missing.

@item 
bpo-32126@footnote{https://bugs.python.org/issue32126}: Skip test_get_event_loop_new_process in
test.test_asyncio.test_events when sem_open() is not functional.

@item 
bpo-31174@footnote{https://bugs.python.org/issue31174}: Fix test_tools.test_unparse: DirectoryTestCase now stores the
names sample to always test the same files. It prevents false alarms when
hunting reference leaks.
@end itemize

@node Build<32>,Windows<31>,Tests<32>,Python 3 7 0 alpha 3
@anchor{whatsnew/changelog id303}@anchor{1369}
@subsubsection Build



@itemize -

@item 
bpo-28538@footnote{https://bugs.python.org/issue28538}: Revert the previous changes, the if_nameindex structure is
defined by Unified Headers.

@item 
bpo-28762@footnote{https://bugs.python.org/issue28762}: Revert the last commit, the F_LOCK macro is defined by Android
Unified Headers.

@item 
bpo-29040@footnote{https://bugs.python.org/issue29040}: Support building Android with Unified Headers. The first NDK
release to support Unified Headers is android-ndk-r14.

@item 
bpo-32059@footnote{https://bugs.python.org/issue32059}: @code{detect_modules()} in @code{setup.py} now also searches the
sysroot paths when cross-compiling.

@item 
bpo-31957@footnote{https://bugs.python.org/issue31957}: Fixes Windows SDK version detection when building for Windows.

@item 
bpo-31609@footnote{https://bugs.python.org/issue31609}: Fixes quotes in PCbuild/clean.bat

@item 
bpo-31934@footnote{https://bugs.python.org/issue31934}: Abort the build when building out of a not clean source tree.

@item 
bpo-31926@footnote{https://bugs.python.org/issue31926}: Fixed Argument Clinic sometimes causing compilation errors when
there was more than one function and/or method in a .c file with the same
name.

@item 
bpo-28791@footnote{https://bugs.python.org/issue28791}: Update Windows builds to use SQLite 3.21.0.

@item 
bpo-28791@footnote{https://bugs.python.org/issue28791}: Update OS X installer to use SQLite 3.21.0.

@item 
bpo-28643@footnote{https://bugs.python.org/issue28643}: Record profile-opt build progress with stamp files.

@item 
bpo-31866@footnote{https://bugs.python.org/issue31866}: Finish removing support for AtheOS.
@end itemize

@node Windows<31>,macOS<26>,Build<32>,Python 3 7 0 alpha 3
@anchor{whatsnew/changelog id304}@anchor{136a}
@subsubsection Windows



@itemize -

@item 
bpo-1102@footnote{https://bugs.python.org/issue1102}: Return @code{None} when @code{View.Fetch()} returns
@code{ERROR_NO_MORE_ITEMS} instead of raising @code{MSIError}. Initial patch by
Anthony Tuininga.

@item 
bpo-31944@footnote{https://bugs.python.org/issue31944}: Fixes Modify button in Apps and Features dialog.

@item 
bpo-20486@footnote{https://bugs.python.org/issue20486}: Implement the @code{Database.Close()} method to help closing MSI
database objects.

@item 
bpo-31857@footnote{https://bugs.python.org/issue31857}: Make the behavior of USE_STACKCHECK deterministic in a
multi-threaded environment.
@end itemize

@node macOS<26>,IDLE<30>,Windows<31>,Python 3 7 0 alpha 3
@anchor{whatsnew/changelog id305}@anchor{136b}
@subsubsection macOS



@itemize -

@item 
bpo-31392@footnote{https://bugs.python.org/issue31392}: Update macOS installer to use OpenSSL 1.0.2m
@end itemize

@node IDLE<30>,Tools/Demos<16>,macOS<26>,Python 3 7 0 alpha 3
@anchor{whatsnew/changelog id306}@anchor{136c}
@subsubsection IDLE



@itemize -

@item 
bpo-32207@footnote{https://bugs.python.org/issue32207}: Improve tk event exception tracebacks in IDLE. When tk event
handling is driven by IDLE’s run loop, a confusing and distracting
queue.EMPTY traceback context is no longer added to tk event exception
tracebacks.  The traceback is now the same as when event handling is
driven by user code.  Patch based on a suggestion by Serhiy Storchaka.

@item 
bpo-32164@footnote{https://bugs.python.org/issue32164}: Delete unused file idlelib/tabbedpages.py. Use of TabbedPageSet
in configdialog was replaced by ttk.Notebook.

@item 
bpo-32100@footnote{https://bugs.python.org/issue32100}: IDLE: Fix old and new bugs in pathbrowser; improve tests. Patch
mostly by Cheryl Sabella.

@item 
bpo-31858@footnote{https://bugs.python.org/issue31858}: IDLE – Restrict shell prompt manipulation to the shell. Editor
and output windows only see an empty last prompt line.  This simplifies
the code and fixes a minor bug when newline is inserted. Sys.ps1, if
present, is read on Shell start-up, but is not set or changed.

@item 
bpo-31860@footnote{https://bugs.python.org/issue31860}: The font sample in the IDLE configuration dialog is now
editable. Changes persist while IDLE remains open

@item 
bpo-31836@footnote{https://bugs.python.org/issue31836}: Test_code_module now passes if run after test_idle, which sets
ps1. The code module uses sys.ps1 if present or sets it to ‘>>> ‘ if not.
Test_code_module now properly tests both behaviors.  Ditto for ps2.

@item 
bpo-28603@footnote{https://bugs.python.org/issue28603}: Fix a TypeError that caused a shell restart when printing a
traceback that includes an exception that is unhashable. Patch by Zane
Bitter.

@item 
bpo-13802@footnote{https://bugs.python.org/issue13802}: Use non-Latin characters in the IDLE’s Font settings sample.
Even if one selects a font that defines a limited subset of the unicode
Basic Multilingual Plane, tcl/tk will use other fonts that define a
character. The expanded example give users of non-Latin characters a
better idea of what they might see in IDLE’s shell and editors. To make
room for the expanded sample, frames on the Font tab are re-arranged.  The
Font/Tabs help explains a bit about the additions.
@end itemize

@node Tools/Demos<16>,C API<30>,IDLE<30>,Python 3 7 0 alpha 3
@anchor{whatsnew/changelog id307}@anchor{136d}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-32159@footnote{https://bugs.python.org/issue32159}: Remove CVS and Subversion tools: remove svneol.py and
treesync.py scripts. CPython migrated from CVS to Subversion, to
Mercurial, and then to Git. CVS and Subversion are no longer used to
develop CPython.

@item 
bpo-30722@footnote{https://bugs.python.org/issue30722}: Make redemo work with Python 3.6 and newer versions. Also,
remove the @code{LOCALE} option since it doesn’t work with string patterns in
Python 3. Patch by Christoph Sarnowski.
@end itemize

@node C API<30>,,Tools/Demos<16>,Python 3 7 0 alpha 3
@anchor{whatsnew/changelog id308}@anchor{136e}
@subsubsection C API



@itemize -

@item 
bpo-20891@footnote{https://bugs.python.org/issue20891}: Fix PyGILState_Ensure(). When PyGILState_Ensure() is called in
a non-Python thread before PyEval_InitThreads(), only call
PyEval_InitThreads() after calling PyThreadState_New() to fix a crash.

@item 
bpo-32125@footnote{https://bugs.python.org/issue32125}: The @code{Py_UseClassExceptionsFlag} flag has been removed. It was
deprecated and wasn’t used anymore since Python 2.0.

@item 
bpo-25612@footnote{https://bugs.python.org/issue25612}: Move the current exception state from the frame object to the
co-routine. This simplifies the interpreter and fixes a couple of obscure
bugs caused by having swap exception state when entering or exiting a
generator.

@item 
bpo-23699@footnote{https://bugs.python.org/issue23699}: Add Py_RETURN_RICHCOMPARE macro to reduce boilerplate code in
rich comparison functions.

@item 
bpo-30697@footnote{https://bugs.python.org/issue30697}: The @code{PyExc_RecursionErrorInst} singleton is removed and
@code{PyErr_NormalizeException()} does not use it anymore. This singleton is
persistent and its members being never cleared may cause a segfault during
finalization of the interpreter. See also bpo-22898@footnote{https://bugs.python.org/issue22898}.
@end itemize

@node Python 3 7 0 alpha 2,Python 3 7 0 alpha 1,Python 3 7 0 alpha 3,Changelog
@anchor{whatsnew/changelog python-3-7-0-alpha-2}@anchor{136f}
@subsection Python 3.7.0 alpha 2


@emph{Release date: 2017-10-16}

@menu
* Core and Builtins: Core and Builtins<35>. 
* Library: Library<36>. 
* Documentation: Documentation<36>. 
* Build: Build<33>. 
* IDLE: IDLE<31>. 
* C API: C API<31>. 

@end menu

@node Core and Builtins<35>,Library<36>,,Python 3 7 0 alpha 2
@anchor{whatsnew/changelog id309}@anchor{1370}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-31558@footnote{https://bugs.python.org/issue31558}: @code{gc.freeze()} is a new API that allows for moving all objects
currently tracked by the garbage collector to a permanent generation,
effectively removing them from future collection events. This can be used
to protect those objects from having their PyGC_Head mutated. In effect,
this enables great copy-on-write stability at fork().

@item 
bpo-31642@footnote{https://bugs.python.org/issue31642}: Restored blocking “from package import module” by setting
sys.modules[“package.module”] to None.

@item 
bpo-31708@footnote{https://bugs.python.org/issue31708}: Allow use of asynchronous generator expressions in synchronous
functions.

@item 
bpo-31709@footnote{https://bugs.python.org/issue31709}: Drop support of asynchronous __aiter__.

@item 
bpo-30404@footnote{https://bugs.python.org/issue30404}: The -u option now makes the stdout and stderr streams
unbuffered rather than line-buffered.

@item 
bpo-31619@footnote{https://bugs.python.org/issue31619}: Fixed a ValueError when convert a string with large number of
underscores to integer with binary base.

@item 
bpo-31602@footnote{https://bugs.python.org/issue31602}: Fix an assertion failure in @code{zipimporter.get_source()} in case
of a bad @ref{85a,,zlib.decompress()}. Patch by Oren Milman.

@item 
bpo-31592@footnote{https://bugs.python.org/issue31592}: Fixed an assertion failure in Python parser in case of a bad
@ref{19a,,unicodedata.normalize()}. Patch by Oren Milman.

@item 
bpo-31588@footnote{https://bugs.python.org/issue31588}: Raise a @ref{162,,TypeError} with a helpful error message when class
creation fails due to a metaclass with a bad @code{__prepare__()} method.
Patch by Oren Milman.

@item 
bpo-31574@footnote{https://bugs.python.org/issue31574}: Importlib was instrumented with two dtrace probes to profile
import timing.

@item 
bpo-31566@footnote{https://bugs.python.org/issue31566}: Fix an assertion failure in @code{_warnings.warn()} in case of a bad
@code{__name__} global. Patch by Oren Milman.

@item 
bpo-31506@footnote{https://bugs.python.org/issue31506}: Improved the error message logic for object.__new__ and
object.__init__.

@item 
bpo-31505@footnote{https://bugs.python.org/issue31505}: Fix an assertion failure in @ref{a0,,json}, in case
@code{_json.make_encoder()} received a bad @code{encoder()} argument. Patch by Oren
Milman.

@item 
bpo-31492@footnote{https://bugs.python.org/issue31492}: Fix assertion failures in case of failing to import from a
module with a bad @code{__name__} attribute, and in case of failing to access
an attribute of such a module. Patch by Oren Milman.

@item 
bpo-31478@footnote{https://bugs.python.org/issue31478}: Fix an assertion failure in @code{_random.Random.seed()} in case the
argument has a bad @code{__abs__()} method. Patch by Oren Milman.

@item 
bpo-31336@footnote{https://bugs.python.org/issue31336}: Speed up class creation by 10-20% by reducing the overhead in
the necessary special method lookups.  Patch by Stefan Behnel.

@item 
bpo-31415@footnote{https://bugs.python.org/issue31415}: Add @code{-X importtime} option to show how long each import
takes. It can be used to optimize application’s startup time.  Support the
@geindex PYTHONPROFILEIMPORTTIME
@geindex environment variable; PYTHONPROFILEIMPORTTIME
@ref{612,,PYTHONPROFILEIMPORTTIME} as an equivalent way to enable this.

@item 
bpo-31410@footnote{https://bugs.python.org/issue31410}: Optimized calling wrapper and classmethod descriptors.

@item 
bpo-31353@footnote{https://bugs.python.org/issue31353}: 
@geindex Python Enhancement Proposals; PEP 553
PEP 553@footnote{https://www.python.org/dev/peps/pep-0553} - Add a new built-in called @code{breakpoint()} which
calls @code{sys.breakpointhook()}.  By default this imports @code{pdb} and calls
@code{pdb.set_trace()}, but users may override @code{sys.breakpointhook()} to
call whatever debugger they want.  The original value of the hook is saved
in @code{sys.__breakpointhook__}.

@item 
bpo-17852@footnote{https://bugs.python.org/issue17852}: Maintain a list of open buffered files, flush them before
exiting the interpreter.  Based on a patch from Armin Rigo.

@item 
bpo-31315@footnote{https://bugs.python.org/issue31315}: Fix an assertion failure in imp.create_dynamic(), when
spec.name is not a string. Patch by Oren Milman.

@item 
bpo-31311@footnote{https://bugs.python.org/issue31311}: Fix a crash in the @code{__setstate__()} method of
@ref{1371,,ctypes._CData}, in case of a bad @code{__dict__}. Patch by Oren Milman.

@item 
bpo-31293@footnote{https://bugs.python.org/issue31293}: Fix crashes in true division and multiplication of a timedelta
object by a float with a bad as_integer_ratio() method. Patch by Oren
Milman.

@item 
bpo-31285@footnote{https://bugs.python.org/issue31285}: Fix an assertion failure in @ref{84b,,warnings.warn_explicit}, when the
return value of the received loader’s get_source() has a bad splitlines()
method. Patch by Oren Milman.

@item 
bpo-30406@footnote{https://bugs.python.org/issue30406}: Make @code{async} and @code{await} proper keywords, as specified in
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492}.
@end itemize

@node Library<36>,Documentation<36>,Core and Builtins<35>,Python 3 7 0 alpha 2
@anchor{whatsnew/changelog id310}@anchor{1372}
@subsubsection Library



@itemize -

@item 
bpo-30058@footnote{https://bugs.python.org/issue30058}: Fixed buffer overflow in select.kqueue.control().

@item 
bpo-31672@footnote{https://bugs.python.org/issue31672}: @code{idpattern} in @code{string.Template} matched some non-ASCII
characters. Now it uses @code{-i} regular expression local flag to avoid
non-ASCII characters.

@item 
bpo-31701@footnote{https://bugs.python.org/issue31701}: On Windows, faulthandler.enable() now ignores MSC and COM
exceptions.

@item 
bpo-31728@footnote{https://bugs.python.org/issue31728}: Prevent crashes in @code{_elementtree} due to unsafe cleanup of
@code{Element.text} and @code{Element.tail}. Patch by Oren Milman.

@item 
bpo-31671@footnote{https://bugs.python.org/issue31671}: Now @code{re.compile()} converts passed RegexFlag to normal int
object before compiling. bm_regex_compile benchmark shows 14% performance
improvements.

@item 
bpo-30397@footnote{https://bugs.python.org/issue30397}: The types of compiled regular objects and match objects are now
exposed as @code{re.Pattern} and @code{re.Match}.  This adds information in pydoc
output for the re module.

@item 
bpo-31675@footnote{https://bugs.python.org/issue31675}: Fixed memory leaks in Tkinter’s methods splitlist() and split()
when pass a string larger than 2 GiB.

@item 
bpo-31673@footnote{https://bugs.python.org/issue31673}: Fixed typo in the name of Tkinter’s method adderrorinfo().

@item 
bpo-31648@footnote{https://bugs.python.org/issue31648}: Improvements to path predicates in ElementTree: Allow
whitespace around predicate parts, i.e. “[a = ‘text’]” instead of
requiring the less readable “[a=’text’]”. Add support for text comparison
of the current node, like “[.=’text’]”. Patch by Stefan Behnel.

@item 
bpo-30806@footnote{https://bugs.python.org/issue30806}: Fix the string representation of a netrc object.

@item 
bpo-31638@footnote{https://bugs.python.org/issue31638}: Add optional argument @code{compressed} to
@code{zipapp.create_archive}, and add option @code{--compress} to the command
line interface of @code{zipapp}.

@item 
bpo-25351@footnote{https://bugs.python.org/issue25351}: Avoid venv activate failures with undefined variables

@item 
bpo-20519@footnote{https://bugs.python.org/issue20519}: Avoid ctypes use (if possible) and improve import time for
uuid.

@item 
bpo-28293@footnote{https://bugs.python.org/issue28293}: The regular expression cache is no longer completely dumped
when it is full.

@item 
bpo-31596@footnote{https://bugs.python.org/issue31596}: Added pthread_getcpuclockid() to the time module

@item 
bpo-27494@footnote{https://bugs.python.org/issue27494}: Make 2to3 accept a trailing comma in generator expressions. For
example, @code{set(x for x in [],)} is now allowed.

@item 
bpo-30347@footnote{https://bugs.python.org/issue30347}: Stop crashes when concurrently iterate over itertools.groupby()
iterators.

@item 
bpo-30346@footnote{https://bugs.python.org/issue30346}: An iterator produced by itertools.groupby() iterator now
becomes exhausted after advancing the groupby iterator.

@item 
bpo-31556@footnote{https://bugs.python.org/issue31556}: Cancel asyncio.wait_for future faster if timeout <= 0

@item 
bpo-31540@footnote{https://bugs.python.org/issue31540}: Allow passing a context object in
@ref{3da,,concurrent.futures.ProcessPoolExecutor} constructor. Also, free
job resources in @ref{3da,,concurrent.futures.ProcessPoolExecutor} earlier
to improve memory usage when a worker waits for new jobs.

@item 
bpo-31516@footnote{https://bugs.python.org/issue31516}: @code{threading.current_thread()} should not return a dummy thread
at shutdown.

@item 
bpo-31525@footnote{https://bugs.python.org/issue31525}: In the sqlite module, require the sqlite3_prepare_v2 API. Thus,
the sqlite module now requires sqlite version at least 3.3.9.

@item 
bpo-26510@footnote{https://bugs.python.org/issue26510}: argparse subparsers are now required by default.  This matches
behaviour in Python 2. For optional subparsers, use the new parameter
@code{add_subparsers(required=False)}. Patch by Anthony Sottile. (As of
3.7.0rc1, the default was changed to not required as had been the case
since Python 3.3.)

@item 
bpo-27541@footnote{https://bugs.python.org/issue27541}: Reprs of subclasses of some collection and iterator classes
(@ref{2f1,,bytearray}, @ref{25e,,array.array}, @ref{7d3,,collections.deque},
@ref{d63,,collections.defaultdict}, @ref{e33,,itertools.count}, @ref{1373,,itertools.repeat}) now
contain actual type name insteads of hardcoded name of the base class.

@item 
bpo-31351@footnote{https://bugs.python.org/issue31351}: python -m ensurepip now exits with non-zero exit code if pip
bootstrapping has failed.

@item 
bpo-31389@footnote{https://bugs.python.org/issue31389}: @code{pdb.set_trace()} now takes an optional keyword-only argument
@code{header}. If given, this is printed to the console just before debugging
begins.
@end itemize

@node Documentation<36>,Build<33>,Library<36>,Python 3 7 0 alpha 2
@anchor{whatsnew/changelog id311}@anchor{1374}
@subsubsection Documentation



@itemize -

@item 
bpo-31537@footnote{https://bugs.python.org/issue31537}: Fix incorrect usage of @code{get_history_length} in readline
documentation example code. Patch by Brad Smith.

@item 
bpo-30085@footnote{https://bugs.python.org/issue30085}: The operator functions without double underscores are preferred
for clarity. The one with underscores are only kept for
back-compatibility.
@end itemize

@node Build<33>,IDLE<31>,Documentation<36>,Python 3 7 0 alpha 2
@anchor{whatsnew/changelog id312}@anchor{1375}
@subsubsection Build



@itemize -

@item 
bpo-31696@footnote{https://bugs.python.org/issue31696}: Improve compiler version information in @ref{86c,,sys.version}
when Python is built with Clang.

@item 
bpo-31625@footnote{https://bugs.python.org/issue31625}: Stop using ranlib on static libraries. Instead, we assume ar
supports the ‘s’ flag.

@item 
bpo-31624@footnote{https://bugs.python.org/issue31624}: Remove support for BSD/OS.

@item 
bpo-22140@footnote{https://bugs.python.org/issue22140}: Prevent double substitution of prefix in python-config.sh.

@item 
bpo-31569@footnote{https://bugs.python.org/issue31569}: Correct PCBuild/ case to PCbuild/ in build scripts and
documentation.

@item 
bpo-31536@footnote{https://bugs.python.org/issue31536}: Avoid wholesale rebuild after @code{make regen-all} if nothing
changed.
@end itemize

@node IDLE<31>,C API<31>,Build<33>,Python 3 7 0 alpha 2
@anchor{whatsnew/changelog id313}@anchor{1376}
@subsubsection IDLE



@itemize -

@item 
bpo-31460@footnote{https://bugs.python.org/issue31460}: Simplify the API of IDLE’s Module Browser. Passing a widget
instead of an flist with a root widget opens the option of creating a
browser frame that is only part of a window.  Passing a full file name
instead of pieces assumed to come from a .py file opens the possibility of
browsing python files that do not end in .py.

@item 
bpo-31649@footnote{https://bugs.python.org/issue31649}: IDLE - Make _htest, _utest parameters keyword only.

@item 
bpo-31559@footnote{https://bugs.python.org/issue31559}: Remove test order dependence in idle_test.test_browser.

@item 
bpo-31459@footnote{https://bugs.python.org/issue31459}: Rename IDLE’s module browser from Class Browser to Module
Browser. The original module-level class and method browser became a
module browser, with the addition of module-level functions, years ago.
Nested classes and functions were added yesterday.  For
back-compatibility, the virtual event <<open-class-browser>>, which
appears on the Keys tab of the Settings dialog, is not changed. Patch by
Cheryl Sabella.

@item 
bpo-31500@footnote{https://bugs.python.org/issue31500}: Default fonts now are scaled on HiDPI displays.

@item 
bpo-1612262@footnote{https://bugs.python.org/issue1612262}: IDLE module browser now shows nested classes and functions.
Original patches for code and tests by Guilherme Polo and Cheryl Sabella,
respectively.
@end itemize

@node C API<31>,,IDLE<31>,Python 3 7 0 alpha 2
@anchor{whatsnew/changelog id314}@anchor{1377}
@subsubsection C API



@itemize -

@item 
bpo-28280@footnote{https://bugs.python.org/issue28280}: Make @code{PyMapping_Keys()}, @code{PyMapping_Values()} and
@code{PyMapping_Items()} always return a @ref{43a,,list} (rather than a @ref{43a,,list} or a
@ref{3e5,,tuple}). Patch by Oren Milman.

@item 
bpo-31532@footnote{https://bugs.python.org/issue31532}: Fix memory corruption due to allocator mix in getpath.c between
Py_GetPath() and Py_SetPath()

@item 
bpo-25658@footnote{https://bugs.python.org/issue25658}: Implement 
@geindex Python Enhancement Proposals; PEP 539
PEP 539@footnote{https://www.python.org/dev/peps/pep-0539} for Thread Specific Storage (TSS) API: it
is a new Thread Local Storage (TLS) API to CPython which would supersede
use of the existing TLS API within the CPython interpreter, while
deprecating the existing API. PEP written by Erik M. Bray, patch by
Masayuki Yamamoto.
@end itemize

@node Python 3 7 0 alpha 1,Python 3 6 6 final,Python 3 7 0 alpha 2,Changelog
@anchor{whatsnew/changelog python-3-7-0-alpha-1}@anchor{1378}
@subsection Python 3.7.0 alpha 1


@emph{Release date: 2017-09-19}

@menu
* Security: Security<21>. 
* Core and Builtins: Core and Builtins<36>. 
* Library: Library<37>. 
* Documentation: Documentation<37>. 
* Tests: Tests<33>. 
* Build: Build<34>. 
* Windows: Windows<32>. 
* IDLE: IDLE<32>. 
* Tools/Demos: Tools/Demos<17>. 
* C API: C API<32>. 

@end menu

@node Security<21>,Core and Builtins<36>,,Python 3 7 0 alpha 1
@anchor{whatsnew/changelog id315}@anchor{1379}
@subsubsection Security



@itemize -

@item 
bpo-29781@footnote{https://bugs.python.org/issue29781}: SSLObject.version() now correctly returns None when handshake
over BIO has not been performed yet.

@item 
bpo-29505@footnote{https://bugs.python.org/issue29505}: Add fuzz tests for float(str), int(str), unicode(str); for
oss-fuzz.

@item 
bpo-30947@footnote{https://bugs.python.org/issue30947}: Upgrade libexpat embedded copy from version 2.2.1 to 2.2.3 to
get security fixes.

@item 
bpo-30730@footnote{https://bugs.python.org/issue30730}: Prevent environment variables injection in subprocess on
Windows.  Prevent passing other environment variables and command
arguments.

@item 
bpo-30694@footnote{https://bugs.python.org/issue30694}: Upgrade expat copy from 2.2.0 to 2.2.1 to get fixes of multiple
security vulnerabilities including: CVE-2017-9233 (External entity
infinite loop DoS), CVE-2016-9063 (Integer overflow, re-fix),
CVE-2016-0718 (Fix regression bugs from 2.2.0’s fix to CVE-2016-0718) and
CVE-2012-0876 (Counter hash flooding with SipHash). Note: the
CVE-2016-5300 (Use os-specific entropy sources like getrandom) doesn’t
impact Python, since Python already gets entropy from the OS to set the
expat secret using @code{XML_SetHashSalt()}.

@item 
bpo-30500@footnote{https://bugs.python.org/issue30500}: Fix urllib.parse.splithost() to correctly parse fragments. For
example, @code{splithost('//127.0.0.1#@@evil.com/')} now correctly returns the
@code{127.0.0.1} host, instead of treating @code{@@evil.com} as the host in an
authentication (@code{login@@host}).

@item 
bpo-29591@footnote{https://bugs.python.org/issue29591}: Update expat copy from 2.1.1 to 2.2.0 to get fixes of
CVE-2016-0718 and CVE-2016-4472. See
@indicateurl{https://sourceforge.net/p/expat/bugs/537/} for more information.
@end itemize

@node Core and Builtins<36>,Library<37>,Security<21>,Python 3 7 0 alpha 1
@anchor{whatsnew/changelog id316}@anchor{137a}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-31490@footnote{https://bugs.python.org/issue31490}: Fix an assertion failure in @ref{29,,ctypes} class definition, in case
the class has an attribute whose name is specified in @code{_anonymous_} but
not in @code{_fields_}. Patch by Oren Milman.

@item 
bpo-31471@footnote{https://bugs.python.org/issue31471}: Fix an assertion failure in @ref{5ae,,subprocess.Popen()} on Windows, in
case the env argument has a bad keys() method. Patch by Oren Milman.

@item 
bpo-31418@footnote{https://bugs.python.org/issue31418}: Fix an assertion failure in @code{PyErr_WriteUnraisable()} in case
of an exception with a bad @code{__module__} attribute. Patch by Oren Milman.

@item 
bpo-31416@footnote{https://bugs.python.org/issue31416}: Fix assertion failures in case of a bad warnings.filters or
warnings.defaultaction. Patch by Oren Milman.

@item 
bpo-28411@footnote{https://bugs.python.org/issue28411}: Change direct usage of PyInterpreterState.modules to
PyImport_GetModuleDict(). Also introduce more uniformity in other code
that deals with sys.modules. This helps reduce complications when working
on sys.modules.

@item 
bpo-28411@footnote{https://bugs.python.org/issue28411}: Switch to the abstract API when dealing with
@code{PyInterpreterState.modules}. This allows later support for all dict
subclasses and other Mapping implementations.  Also add a
@code{PyImport_GetModule()} function to reduce a bunch of duplicated code.

@item 
bpo-31411@footnote{https://bugs.python.org/issue31411}: Raise a TypeError instead of SystemError in case
warnings.onceregistry is not a dictionary. Patch by Oren Milman.

@item 
bpo-31344@footnote{https://bugs.python.org/issue31344}: For finer control of tracing behaviour when testing the
interpreter, two new frame attributes have been added to control the
emission of particular trace events: @code{f_trace_lines} (@code{True} by
default) to turn off per-line trace events; and @code{f_trace_opcodes}
(@code{False} by default) to turn on per-opcode trace events.

@item 
bpo-31373@footnote{https://bugs.python.org/issue31373}: Fix several possible instances of undefined behavior due to
floating-point demotions.

@item 
bpo-30465@footnote{https://bugs.python.org/issue30465}: Location information (@code{lineno} and @code{col_offset}) in
f-strings is now (mostly) correct.  This fixes tools like flake8 from
showing warnings on the wrong line (typically the first line of the file).

@item 
bpo-30860@footnote{https://bugs.python.org/issue30860}: Consolidate CPython’s global runtime state under a single
struct.  This improves discoverability of the runtime state.

@item 
bpo-31347@footnote{https://bugs.python.org/issue31347}: Fix possible undefined behavior in _PyObject_FastCall_Prepend.

@item 
bpo-31343@footnote{https://bugs.python.org/issue31343}: Include sys/sysmacros.h for major(), minor(), and makedev().
GNU C libray plans to remove the functions from sys/types.h.

@item 
bpo-31291@footnote{https://bugs.python.org/issue31291}: Fix an assertion failure in @ref{137b,,zipimport.zipimporter.get_data} on
Windows, when the return value of @code{pathname.replace('/','\\')} isn’t a
string. Patch by Oren Milman.

@item 
bpo-31271@footnote{https://bugs.python.org/issue31271}: Fix an assertion failure in the write() method of
@ref{889,,io.TextIOWrapper}, when the encoder doesn’t return a bytes object. Patch
by Oren Milman.

@item 
bpo-31243@footnote{https://bugs.python.org/issue31243}: Fix a crash in some methods of @ref{889,,io.TextIOWrapper}, when the
decoder’s state is invalid. Patch by Oren Milman.

@item 
bpo-30721@footnote{https://bugs.python.org/issue30721}: @code{print} now shows correct usage hint for using Python 2
redirection syntax.  Patch by Sanyam Khurana.

@item 
bpo-31070@footnote{https://bugs.python.org/issue31070}: Fix a race condition in importlib _get_module_lock().

@item 
bpo-30747@footnote{https://bugs.python.org/issue30747}: Add a non-dummy implementation of _Py_atomic_store and
_Py_atomic_load on MSVC.

@item 
bpo-31095@footnote{https://bugs.python.org/issue31095}: Fix potential crash during GC caused by @code{tp_dealloc} which
doesn’t call @code{PyObject_GC_UnTrack()}.

@item 
bpo-31071@footnote{https://bugs.python.org/issue31071}: Avoid masking original TypeError in call with * unpacking when
other arguments are passed.

@item 
bpo-30978@footnote{https://bugs.python.org/issue30978}: str.format_map() now passes key lookup exceptions through.
Previously any exception was replaced with a KeyError exception.

@item 
bpo-30808@footnote{https://bugs.python.org/issue30808}: Use _Py_atomic API for concurrency-sensitive signal state.

@item 
bpo-30876@footnote{https://bugs.python.org/issue30876}: Relative import from unloaded package now reimports the package
instead of failing with SystemError.  Relative import from non-package now
fails with ImportError rather than SystemError.

@item 
bpo-30703@footnote{https://bugs.python.org/issue30703}: Improve signal delivery. Avoid using Py_AddPendingCall from
signal handler, to avoid calling signal-unsafe functions. The tests I’m
adding here fail without the rest of the patch, on Linux and OS X. This
means our signal delivery logic had defects (some signals could be lost).

@item 
bpo-30765@footnote{https://bugs.python.org/issue30765}: Avoid blocking in pthread_mutex_lock() when
PyThread_acquire_lock() is asked not to block.

@item 
bpo-31161@footnote{https://bugs.python.org/issue31161}: Make sure the ‘Missing parentheses’ syntax error message is
only applied to SyntaxError, not to subclasses. Patch by Martijn Pieters.

@item 
bpo-30814@footnote{https://bugs.python.org/issue30814}: Fixed a race condition when import a submodule from a package.

@item 
bpo-30736@footnote{https://bugs.python.org/issue30736}: The internal unicodedata database has been upgraded to Unicode
10.0.

@item 
bpo-30604@footnote{https://bugs.python.org/issue30604}: Move co_extra_freefuncs from per-thread to per-interpreter to
avoid crashes.

@item 
bpo-30597@footnote{https://bugs.python.org/issue30597}: @code{print} now shows expected input in custom error message when
used as a Python 2 statement. Patch by Sanyam Khurana.

@item 
bpo-30682@footnote{https://bugs.python.org/issue30682}: Removed a too-strict assertion that failed for certain
f-strings, such as eval(“f’\n’”) and eval(“f’\r’”).

@item 
bpo-30501@footnote{https://bugs.python.org/issue30501}: The compiler now produces more optimal code for complex
condition expressions in the “if”, “while” and “assert” statement, the
“if” expression, and generator expressions and comprehensions.

@item 
bpo-28180@footnote{https://bugs.python.org/issue28180}: Implement 
@geindex Python Enhancement Proposals; PEP 538
PEP 538@footnote{https://www.python.org/dev/peps/pep-0538} (legacy C locale coercion). This means
that when a suitable coercion target locale is available, both the core
interpreter and locale-aware C extensions will assume the use of UTF-8 as
the default text encoding, rather than ASCII.

@item 
bpo-30486@footnote{https://bugs.python.org/issue30486}: Allows setting cell values for __closure__. Patch by Lisa
Roach.

@item 
bpo-30537@footnote{https://bugs.python.org/issue30537}: itertools.islice now accepts integer-like objects (having an
__index__ method) as start, stop, and slice arguments

@item 
bpo-25324@footnote{https://bugs.python.org/issue25324}: Tokens needed for parsing in Python moved to C. @code{COMMENT},
@code{NL} and @code{ENCODING}. This way the tokens and tok_names in the token
module don’t get changed when you import the tokenize module.

@item 
bpo-29104@footnote{https://bugs.python.org/issue29104}: Fixed parsing backslashes in f-strings.

@item 
bpo-27945@footnote{https://bugs.python.org/issue27945}: Fixed various segfaults with dict when input collections are
mutated during searching, inserting or comparing.  Based on patches by
Duane Griffin and Tim Mitchell.

@item 
bpo-25794@footnote{https://bugs.python.org/issue25794}: Fixed type.__setattr__() and type.__delattr__() for
non-interned attribute names.  Based on patch by Eryk Sun.

@item 
bpo-30039@footnote{https://bugs.python.org/issue30039}: If a KeyboardInterrupt happens when the interpreter is in the
middle of resuming a chain of nested ‘yield from’ or ‘await’ calls, it’s
now correctly delivered to the innermost frame.

@item 
bpo-28974@footnote{https://bugs.python.org/issue28974}: @code{object.__format__(x, '@w{'})} is now equivalent to @code{str(x)}
rather than @code{format(str(self), '@w{'})}.

@item 
bpo-30024@footnote{https://bugs.python.org/issue30024}: Circular imports involving absolute imports with binding a
submodule to a name are now supported.

@item 
bpo-12414@footnote{https://bugs.python.org/issue12414}: sys.getsizeof() on a code object now returns the sizes which
includes the code struct and sizes of objects which it references. Patch
by Dong-hee Na.

@item 
bpo-29839@footnote{https://bugs.python.org/issue29839}: len() now raises ValueError rather than OverflowError if
__len__() returned a large negative integer.

@item 
bpo-11913@footnote{https://bugs.python.org/issue11913}: README.rst is now included in the list of distutils standard
READMEs and therefore included in source distributions.

@item 
bpo-29914@footnote{https://bugs.python.org/issue29914}: Fixed default implementations of __reduce__ and
__reduce_ex__(). object.__reduce__() no longer takes arguments,
object.__reduce_ex__() now requires one argument.

@item 
bpo-29949@footnote{https://bugs.python.org/issue29949}: Fix memory usage regression of set and frozenset object.

@item 
bpo-29935@footnote{https://bugs.python.org/issue29935}: Fixed error messages in the index() method of tuple, list and
deque when pass indices of wrong type.

@item 
bpo-29816@footnote{https://bugs.python.org/issue29816}: Shift operation now has less opportunity to raise
OverflowError. ValueError always is raised rather than OverflowError for
negative counts. Shifting zero with non-negative count always returns
zero.

@item 
bpo-24821@footnote{https://bugs.python.org/issue24821}: Fixed the slowing down to 25 times in the searching of some
unlucky Unicode characters.

@item 
bpo-29102@footnote{https://bugs.python.org/issue29102}: Add a unique ID to PyInterpreterState.  This makes it easier to
identify each subinterpreter.

@item 
bpo-29894@footnote{https://bugs.python.org/issue29894}: The deprecation warning is emitted if __complex__ returns an
instance of a strict subclass of complex.  In a future versions of Python
this can be an error.

@item 
bpo-29859@footnote{https://bugs.python.org/issue29859}: Show correct error messages when any of the pthread_* calls in
thread_pthread.h fails.

@item 
bpo-29849@footnote{https://bugs.python.org/issue29849}: Fix a memory leak when an ImportError is raised during from
import.

@item 
bpo-28856@footnote{https://bugs.python.org/issue28856}: Fix an oversight that %b format for bytes should support
objects follow the buffer protocol.

@item 
bpo-29723@footnote{https://bugs.python.org/issue29723}: The @code{sys.path[0]} initialization change for bpo-29139@footnote{https://bugs.python.org/issue29139} caused
a regression by revealing an inconsistency in how sys.path is initialized
when executing @code{__main__} from a zipfile, directory, or other import
location. The interpreter now consistently avoids ever adding the import
location’s parent directory to @code{sys.path}, and ensures no other
@code{sys.path} entries are inadvertently modified when inserting the import
location named on the command line.

@item 
bpo-29568@footnote{https://bugs.python.org/issue29568}: Escaped percent “%%” in the format string for classic string
formatting no longer allows any characters between two percents.

@item 
bpo-29714@footnote{https://bugs.python.org/issue29714}: Fix a regression that bytes format may fail when containing
zero bytes inside.

@item 
bpo-29695@footnote{https://bugs.python.org/issue29695}: bool(), float(), list() and tuple() no longer take keyword
arguments. The first argument of int() can now be passes only as
positional argument.

@item 
bpo-28893@footnote{https://bugs.python.org/issue28893}: Set correct __cause__ for errors about invalid awaitables
returned from __aiter__ and __anext__.

@item 
bpo-28876@footnote{https://bugs.python.org/issue28876}: @code{bool(range)} works even if @code{len(range)} raises
@ref{347,,OverflowError}.

@item 
bpo-29683@footnote{https://bugs.python.org/issue29683}: Fixes to memory allocation in _PyCode_SetExtra.  Patch by Brian
Coleman.

@item 
bpo-29684@footnote{https://bugs.python.org/issue29684}: Fix minor regression of PyEval_CallObjectWithKeywords. It
should raise TypeError when kwargs is not a dict.  But it might cause segv
when args=NULL and kwargs is not a dict.

@item 
bpo-28598@footnote{https://bugs.python.org/issue28598}: Support __rmod__ for subclasses of str being called before
str.__mod__. Patch by Martijn Pieters.

@item 
bpo-29607@footnote{https://bugs.python.org/issue29607}: Fix stack_effect computation for CALL_FUNCTION_EX. Patch by
Matthieu Dartiailh.

@item 
bpo-29602@footnote{https://bugs.python.org/issue29602}: Fix incorrect handling of signed zeros in complex constructor
for complex subclasses and for inputs having a __complex__ method. Patch
by Serhiy Storchaka.

@item 
bpo-29347@footnote{https://bugs.python.org/issue29347}: Fixed possibly dereferencing undefined pointers when creating
weakref objects.

@item 
bpo-29463@footnote{https://bugs.python.org/issue29463}: Add @code{docstring} field to Module, ClassDef, FunctionDef, and
AsyncFunctionDef ast nodes.  docstring is not first stmt in their body
anymore.  It affects @code{co_firstlineno} and @code{co_lnotab} of code object
for module and class. (Reverted in bpo-32911@footnote{https://bugs.python.org/issue32911}.)

@item 
bpo-29438@footnote{https://bugs.python.org/issue29438}: Fixed use-after-free problem in key sharing dict.

@item 
bpo-29546@footnote{https://bugs.python.org/issue29546}: Set the ‘path’ and ‘name’ attribute on ImportError for @code{from
... import ...}.

@item 
bpo-29546@footnote{https://bugs.python.org/issue29546}: Improve from-import error message with location

@item 
bpo-29478@footnote{https://bugs.python.org/issue29478}: If max_line_length=None is specified while using the Compat32
policy, it is no longer ignored.  Patch by Mircea Cosbuc.

@item 
bpo-29319@footnote{https://bugs.python.org/issue29319}: Prevent RunMainFromImporter overwriting sys.path[0].

@item 
bpo-29337@footnote{https://bugs.python.org/issue29337}: Fixed possible BytesWarning when compare the code objects.
Warnings could be emitted at compile time.

@item 
bpo-29327@footnote{https://bugs.python.org/issue29327}: Fixed a crash when pass the iterable keyword argument to
sorted().

@item 
bpo-29034@footnote{https://bugs.python.org/issue29034}: Fix memory leak and use-after-free in os module
(path_converter).

@item 
bpo-29159@footnote{https://bugs.python.org/issue29159}: Fix regression in bytes(x) when x.__index__() raises Exception.

@item 
bpo-29049@footnote{https://bugs.python.org/issue29049}: Call _PyObject_GC_TRACK() lazily when calling Python function.
Calling function is up to 5% faster.

@item 
bpo-28927@footnote{https://bugs.python.org/issue28927}: bytes.fromhex() and bytearray.fromhex() now ignore all ASCII
whitespace, not only spaces.  Patch by Robert Xiao.

@item 
bpo-28932@footnote{https://bugs.python.org/issue28932}: Do not include <sys/random.h> if it does not exist.

@item 
bpo-25677@footnote{https://bugs.python.org/issue25677}: Correct the positioning of the syntax error caret for indented
blocks. Based on patch by Michael Layzell.

@item 
bpo-29000@footnote{https://bugs.python.org/issue29000}: Fixed bytes formatting of octals with zero padding in alternate
form.

@item 
bpo-18896@footnote{https://bugs.python.org/issue18896}: Python function can now have more than 255 parameters.
collections.namedtuple() now supports tuples with more than 255 elements.

@item 
bpo-28596@footnote{https://bugs.python.org/issue28596}: The preferred encoding is UTF-8 on Android. Patch written by
Chi Hsuan Yen.

@item 
bpo-22257@footnote{https://bugs.python.org/issue22257}: Clean up interpreter startup (see 
@geindex Python Enhancement Proposals; PEP 432
PEP 432@footnote{https://www.python.org/dev/peps/pep-0432}).

@item 
bpo-26919@footnote{https://bugs.python.org/issue26919}: On Android, operating system data is now always encoded/decoded
to/from UTF-8, instead of the locale encoding to avoid inconsistencies
with os.fsencode() and os.fsdecode() which are already using UTF-8.

@item 
bpo-28991@footnote{https://bugs.python.org/issue28991}: functools.lru_cache() was susceptible to an obscure reentrancy
bug triggerable by a monkey-patched len() function.

@item 
bpo-28147@footnote{https://bugs.python.org/issue28147}: Fix a memory leak in split-table dictionaries: setattr() must
not convert combined table into split table. Patch written by INADA Naoki.

@item 
bpo-28739@footnote{https://bugs.python.org/issue28739}: f-string expressions are no longer accepted as docstrings and
by ast.literal_eval() even if they do not include expressions.

@item 
bpo-28512@footnote{https://bugs.python.org/issue28512}: Fixed setting the offset attribute of SyntaxError by
PyErr_SyntaxLocationEx() and PyErr_SyntaxLocationObject().

@item 
bpo-28918@footnote{https://bugs.python.org/issue28918}: Fix the cross compilation of xxlimited when Python has been
built with Py_DEBUG defined.

@item 
bpo-23722@footnote{https://bugs.python.org/issue23722}: Rather than silently producing a class that doesn’t support
zero-argument @code{super()} in methods, failing to pass the new
@code{__classcell__} namespace entry up to @code{type.__new__} now results in a
@code{DeprecationWarning} and a class that supports zero-argument
@code{super()}.

@item 
bpo-28797@footnote{https://bugs.python.org/issue28797}: Modifying the class __dict__ inside the __set_name__ method of
a descriptor that is used inside that class no longer prevents calling the
__set_name__ method of other descriptors.

@item 
bpo-28799@footnote{https://bugs.python.org/issue28799}: Remove the @code{PyEval_GetCallStats()} function and deprecate the
untested and undocumented @code{sys.callstats()} function. Remove the
@code{CALL_PROFILE} special build: use the @ref{1033,,sys.setprofile()} function,
@ref{26,,cProfile} or @ref{ce,,profile} to profile function calls.

@item 
bpo-12844@footnote{https://bugs.python.org/issue12844}: More than 255 arguments can now be passed to a function.

@item 
bpo-28782@footnote{https://bugs.python.org/issue28782}: Fix a bug in the implementation @code{yield from} when checking if
the next instruction is YIELD_FROM. Regression introduced by WORDCODE
(bpo-26647@footnote{https://bugs.python.org/issue26647}).

@item 
bpo-28774@footnote{https://bugs.python.org/issue28774}: Fix error position of the unicode error in ASCII and Latin1
encoders when a string returned by the error handler contains multiple
non-encodable characters (non-ASCII for the ASCII codec, characters out of
the U+0000-U+00FF range for Latin1).

@item 
bpo-28731@footnote{https://bugs.python.org/issue28731}: Optimize _PyDict_NewPresized() to create correct size dict.
Improve speed of dict literal with constant keys up to 30%.

@item 
bpo-28532@footnote{https://bugs.python.org/issue28532}: Show sys.version when -V option is supplied twice.

@item 
bpo-27100@footnote{https://bugs.python.org/issue27100}: The with-statement now checks for __enter__ before it checks
for __exit__. This gives less confusing error messages when both methods
are missing. Patch by Jonathan Ellington.

@item 
bpo-28746@footnote{https://bugs.python.org/issue28746}: Fix the set_inheritable() file descriptor method on platforms
that do not have the ioctl FIOCLEX and FIONCLEX commands.

@item 
bpo-26920@footnote{https://bugs.python.org/issue26920}: Fix not getting the locale’s charset upon initializing the
interpreter, on platforms that do not have langinfo.

@item 
bpo-28648@footnote{https://bugs.python.org/issue28648}: Fixed crash in Py_DecodeLocale() in debug build on Mac OS X
when decode astral characters.  Patch by Xiang Zhang.

@item 
bpo-28665@footnote{https://bugs.python.org/issue28665}: Improve speed of the STORE_DEREF opcode by 40%.

@item 
bpo-19398@footnote{https://bugs.python.org/issue19398}: Extra slash no longer added to sys.path components in case of
empty compile-time PYTHONPATH components.

@item 
bpo-28621@footnote{https://bugs.python.org/issue28621}: Sped up converting int to float by reusing faster bits counting
implementation.  Patch by Adrian Wielgosik.

@item 
bpo-28580@footnote{https://bugs.python.org/issue28580}: Optimize iterating split table values. Patch by Xiang Zhang.

@item 
bpo-28583@footnote{https://bugs.python.org/issue28583}: PyDict_SetDefault didn’t combine split table when needed. Patch
by Xiang Zhang.

@item 
bpo-28128@footnote{https://bugs.python.org/issue28128}: Deprecation warning for invalid str and byte escape sequences
now prints better information about where the error occurs. Patch by
Serhiy Storchaka and Eric Smith.

@item 
bpo-28509@footnote{https://bugs.python.org/issue28509}: dict.update() no longer allocate unnecessary large memory.

@item 
bpo-28426@footnote{https://bugs.python.org/issue28426}: Fixed potential crash in PyUnicode_AsDecodedObject() in debug
build.

@item 
bpo-28517@footnote{https://bugs.python.org/issue28517}: Fixed of-by-one error in the peephole optimizer that caused
keeping unreachable code.

@item 
bpo-28214@footnote{https://bugs.python.org/issue28214}: Improved exception reporting for problematic __set_name__
attributes.

@item 
bpo-23782@footnote{https://bugs.python.org/issue23782}: Fixed possible memory leak in _PyTraceback_Add() and exception
loss in PyTraceBack_Here().

@item 
bpo-28183@footnote{https://bugs.python.org/issue28183}: Optimize and cleanup dict iteration.

@item 
bpo-26081@footnote{https://bugs.python.org/issue26081}: Added C implementation of asyncio.Future. Original patch by
Yury Selivanov.

@item 
bpo-28379@footnote{https://bugs.python.org/issue28379}: Added sanity checks and tests for PyUnicode_CopyCharacters().
Patch by Xiang Zhang.

@item 
bpo-28376@footnote{https://bugs.python.org/issue28376}: The type of long range iterator is now registered as Iterator.
Patch by Oren Milman.

@item 
bpo-28376@footnote{https://bugs.python.org/issue28376}: Creating instances of range_iterator by calling range_iterator
type now is disallowed.  Calling iter() on range instance is the only way.
Patch by Oren Milman.

@item 
bpo-26906@footnote{https://bugs.python.org/issue26906}: Resolving special methods of uninitialized type now causes
implicit initialization of the type instead of a fail.

@item 
bpo-18287@footnote{https://bugs.python.org/issue18287}: PyType_Ready() now checks that tp_name is not NULL. Original
patch by Niklas Koep.

@item 
bpo-24098@footnote{https://bugs.python.org/issue24098}: Fixed possible crash when AST is changed in process of
compiling it.

@item 
bpo-28201@footnote{https://bugs.python.org/issue28201}: Dict reduces possibility of 2nd conflict in hash table when
hashes have same lower bits.

@item 
bpo-28350@footnote{https://bugs.python.org/issue28350}: String constants with null character no longer interned.

@item 
bpo-26617@footnote{https://bugs.python.org/issue26617}: Fix crash when GC runs during weakref callbacks.

@item 
bpo-27942@footnote{https://bugs.python.org/issue27942}: String constants now interned recursively in tuples and
frozensets.

@item 
bpo-28289@footnote{https://bugs.python.org/issue28289}: ImportError.__init__ now resets not specified attributes.

@item 
bpo-21578@footnote{https://bugs.python.org/issue21578}: Fixed misleading error message when ImportError called with
invalid keyword args.

@item 
bpo-28203@footnote{https://bugs.python.org/issue28203}: Fix incorrect type in complex(1.0, @{2:3@}) error message. Patch
by Soumya Sharma.

@item 
bpo-28086@footnote{https://bugs.python.org/issue28086}: Single var-positional argument of tuple subtype was passed
unscathed to the C-defined function.  Now it is converted to exact tuple.

@item 
bpo-28214@footnote{https://bugs.python.org/issue28214}: Now __set_name__ is looked up on the class instead of the
instance.

@item 
bpo-27955@footnote{https://bugs.python.org/issue27955}: Fallback on reading /dev/urandom device when the getrandom()
syscall fails with EPERM, for example when blocked by SECCOMP.

@item 
bpo-28192@footnote{https://bugs.python.org/issue28192}: Don’t import readline in isolated mode.

@item 
bpo-27441@footnote{https://bugs.python.org/issue27441}: Remove some redundant assignments to ob_size in longobject.c.
Thanks Oren Milman.

@item 
bpo-27222@footnote{https://bugs.python.org/issue27222}: Clean up redundant code in long_rshift function. Thanks Oren
Milman.

@item 
Upgrade internal unicode databases to Unicode version 9.0.0.

@item 
bpo-28131@footnote{https://bugs.python.org/issue28131}: Fix a regression in zipimport’s compile_source().  zipimport
should use the same optimization level as the interpreter.

@item 
bpo-28126@footnote{https://bugs.python.org/issue28126}: Replace Py_MEMCPY with memcpy(). Visual Studio can properly
optimize memcpy().

@item 
bpo-28120@footnote{https://bugs.python.org/issue28120}: Fix dict.pop() for splitted dictionary when trying to remove a
“pending key” (Not yet inserted in split-table). Patch by Xiang Zhang.

@item 
bpo-26182@footnote{https://bugs.python.org/issue26182}: Raise DeprecationWarning when async and await keywords are used
as variable/attribute/class/function name.

@item 
bpo-26182@footnote{https://bugs.python.org/issue26182}: Fix a refleak in code that raises DeprecationWarning.

@item 
bpo-28721@footnote{https://bugs.python.org/issue28721}: Fix asynchronous generators aclose() and athrow() to handle
StopAsyncIteration propagation properly.

@item 
bpo-26110@footnote{https://bugs.python.org/issue26110}: Speed-up method calls: add LOAD_METHOD and CALL_METHOD opcodes.
@end itemize

@node Library<37>,Documentation<37>,Core and Builtins<36>,Python 3 7 0 alpha 1
@anchor{whatsnew/changelog id317}@anchor{137c}
@subsubsection Library



@itemize -

@item 
bpo-31499@footnote{https://bugs.python.org/issue31499}: xml.etree: Fix a crash when a parser is part of a reference
cycle.

@item 
bpo-31482@footnote{https://bugs.python.org/issue31482}: @code{random.seed()} now works with bytes in version=1

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: typing.get_type_hints now finds the right globalns for classes
and modules by default (when no @code{globalns} was specified by the caller).

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Speed improvements to the @code{typing} module.  Original PRs by
Ivan Levkivskyi and Mitar.

@item 
bpo-31544@footnote{https://bugs.python.org/issue31544}: The C accelerator module of ElementTree ignored exceptions
raised when looking up TreeBuilder target methods in XMLParser().

@item 
bpo-31234@footnote{https://bugs.python.org/issue31234}: socket.create_connection() now fixes manually a reference
cycle: clear the variable storing the last exception on success.

@item 
bpo-31457@footnote{https://bugs.python.org/issue31457}: LoggerAdapter objects can now be nested.

@item 
bpo-31431@footnote{https://bugs.python.org/issue31431}: SSLContext.check_hostname now automatically sets
SSLContext.verify_mode to ssl.CERT_REQUIRED instead of failing with a
ValueError.

@item 
bpo-31233@footnote{https://bugs.python.org/issue31233}: socketserver.ThreadingMixIn now keeps a list of non-daemonic
threads to wait until all these threads complete in server_close().

@item 
bpo-28638@footnote{https://bugs.python.org/issue28638}: Changed the implementation strategy for
collections.namedtuple() to substantially reduce the use of exec() in
favor of precomputed methods. As a result, the @emph{verbose} parameter and
@emph{_source} attribute are no longer supported.  The benefits include 1)
having a smaller memory footprint for applications using multiple named
tuples, 2) faster creation of the named tuple class (approx 4x to 6x
depending on how it is measured), and 3) minor speed-ups for instance
creation using __new__, _make, and _replace.  (The primary patch
contributor is Jelle Zijlstra with further improvements by INADA Naoki,
Serhiy Storchaka, and Raymond Hettinger.)

@item 
bpo-31400@footnote{https://bugs.python.org/issue31400}: Improves SSL error handling to avoid losing error numbers.

@item 
bpo-27629@footnote{https://bugs.python.org/issue27629}: Make return types of SSLContext.wrap_bio() and
SSLContext.wrap_socket() customizable.

@item 
bpo-28958@footnote{https://bugs.python.org/issue28958}: ssl.SSLContext() now uses OpenSSL error information when a
context cannot be instantiated.

@item 
bpo-28182@footnote{https://bugs.python.org/issue28182}: The SSL module now raises SSLCertVerificationError when OpenSSL
fails to verify the peer’s certificate. The exception contains more
information about the error.

@item 
bpo-27340@footnote{https://bugs.python.org/issue27340}: SSLSocket.sendall() now uses memoryview to create slices of
data. This fixes support for all bytes-like object. It is also more
efficient and avoids costly copies.

@item 
bpo-14191@footnote{https://bugs.python.org/issue14191}: A new function
@code{argparse.ArgumentParser.parse_intermixed_args} provides the ability to
parse command lines where there user intermixes options and positional
arguments.

@item 
bpo-31178@footnote{https://bugs.python.org/issue31178}: Fix string concatenation bug in rare error path in the
subprocess module

@item 
bpo-31350@footnote{https://bugs.python.org/issue31350}: Micro-optimize @code{asyncio._get_running_loop()} to become up
to 10% faster.

@item 
bpo-31170@footnote{https://bugs.python.org/issue31170}: expat: Update libexpat from 2.2.3 to 2.2.4. Fix copying of
partial characters for UTF-8 input (libexpat bug 115):
@indicateurl{https://github.com/libexpat/libexpat/issues/115}

@item 
bpo-29136@footnote{https://bugs.python.org/issue29136}: Add TLS 1.3 cipher suites and OP_NO_TLSv1_3.

@item 
bpo-1198569@footnote{https://bugs.python.org/issue1198569}: @code{string.Template} subclasses can optionally define
@code{braceidpattern} if they want to specify different placeholder patterns
inside and outside the braces.  If None (the default) it falls back to
@code{idpattern}.

@item 
bpo-31326@footnote{https://bugs.python.org/issue31326}: concurrent.futures.ProcessPoolExecutor.shutdown() now
explicitly closes the call queue. Moreover, shutdown(wait=True) now also
join the call queue thread, to prevent leaking a dangling thread.

@item 
bpo-27144@footnote{https://bugs.python.org/issue27144}: The @code{map()} and @code{as_completed()} iterators in
@code{concurrent.futures} now avoid keeping a reference to yielded objects.

@item 
bpo-31281@footnote{https://bugs.python.org/issue31281}: Fix @code{fileinput.FileInput(files, inplace=True)} when @code{files}
contain @code{pathlib.Path} objects.

@item 
bpo-10746@footnote{https://bugs.python.org/issue10746}: Fix ctypes producing wrong 
@geindex Python Enhancement Proposals; PEP 3118
PEP 3118@footnote{https://www.python.org/dev/peps/pep-3118} type codes for integer
types.

@item 
bpo-27584@footnote{https://bugs.python.org/issue27584}: @code{AF_VSOCK} has been added to the socket interface which
allows communication between virtual machines and their host.

@item 
bpo-22536@footnote{https://bugs.python.org/issue22536}: The subprocess module now sets the filename when
FileNotFoundError is raised on POSIX systems due to the executable or cwd
not being found.

@item 
bpo-29741@footnote{https://bugs.python.org/issue29741}: Update some methods in the _pyio module to also accept integer
types. Patch by Oren Milman.

@item 
bpo-31249@footnote{https://bugs.python.org/issue31249}: concurrent.futures: WorkItem.run() used by ThreadPoolExecutor
now breaks a reference cycle between an exception object and the WorkItem
object.

@item 
bpo-31247@footnote{https://bugs.python.org/issue31247}: xmlrpc.server now explicitly breaks reference cycles when using
sys.exc_info() in code handling exceptions.

@item 
bpo-23835@footnote{https://bugs.python.org/issue23835}: configparser: reading defaults in the @code{ConfigParser()}
constructor is now using @code{read_dict()}, making its behavior consistent
with the rest of the parser.  Non-string keys and values in the defaults
dictionary are now being implicitly converted to strings.  Patch by James
Tocknell.

@item 
bpo-31238@footnote{https://bugs.python.org/issue31238}: pydoc: the stop() method of the private ServerThread class now
waits until DocServer.serve_until_quit() completes and then explicitly
sets its docserver attribute to None to break a reference cycle.

@item 
bpo-5001@footnote{https://bugs.python.org/issue5001}: Many asserts in @ref{b3,,multiprocessing} are now more informative, and
some error types have been changed to more specific ones.

@item 
bpo-31109@footnote{https://bugs.python.org/issue31109}: Convert zipimport to use Argument Clinic.

@item 
bpo-30102@footnote{https://bugs.python.org/issue30102}: The ssl and hashlib modules now call
OPENSSL_add_all_algorithms_noconf() on OpenSSL < 1.1.0. The function
detects CPU features and enables optimizations on some CPU architectures
such as POWER8. Patch is based on research from Gustavo Serra Scalet.

@item 
bpo-18966@footnote{https://bugs.python.org/issue18966}: Non-daemonic threads created by a multiprocessing.Process are
now joined on child exit.

@item 
bpo-31183@footnote{https://bugs.python.org/issue31183}: @ref{36,,dis} now works with asynchronous generator and coroutine
objects. Patch by George Collins based on diagnosis by Luciano Ramalho.

@item 
bpo-5001@footnote{https://bugs.python.org/issue5001}: There are a number of uninformative asserts in the
@ref{b3,,multiprocessing} module, as noted in issue 5001. This change fixes two of
the most potentially problematic ones, since they are in error-reporting
code, in the @code{multiprocessing.managers.convert_to_error} function. (It
also makes more informative a ValueError message.) The only potentially
problematic change is that the AssertionError is now a TypeError; however,
this should also help distinguish it from an AssertionError being
@emph{reported} by the function/its caller (such as in issue 31169). - Patch by
Allen W. Smith (drallensmith on github).

@item 
bpo-31185@footnote{https://bugs.python.org/issue31185}: Fixed miscellaneous errors in asyncio speedup module.

@item 
bpo-31151@footnote{https://bugs.python.org/issue31151}: socketserver.ForkingMixIn.server_close() now waits until all
child processes completed to prevent leaking zombie processes.

@item 
bpo-31072@footnote{https://bugs.python.org/issue31072}: Add an @code{include_file} parameter to
@code{zipapp.create_archive()}

@item 
bpo-24700@footnote{https://bugs.python.org/issue24700}: Optimize array.array comparison. It is now from 10x up to 70x
faster when comparing arrays holding values of the same integer type.

@item 
bpo-31135@footnote{https://bugs.python.org/issue31135}: ttk: fix the destroy() method of LabeledScale and OptionMenu
classes. Call the parent destroy() method even if the used attribute
doesn’t exist. The LabeledScale.destroy() method now also explicitly
clears label and scale attributes to help the garbage collector to destroy
all widgets.

@item 
bpo-31107@footnote{https://bugs.python.org/issue31107}: Fix @code{copyreg._slotnames()} mangled attribute calculation for
classes whose name begins with an underscore. Patch by Shane Harvey.

@item 
bpo-31080@footnote{https://bugs.python.org/issue31080}: Allow @ref{678,,logging.config.fileConfig} to accept kwargs and/or args.

@item 
bpo-30897@footnote{https://bugs.python.org/issue30897}: @code{pathlib.Path} objects now include an @code{is_mount()} method
(only implemented on POSIX).  This is similar to @code{os.path.ismount(p)}.
Patch by Cooper Ry Lees.

@item 
bpo-31061@footnote{https://bugs.python.org/issue31061}: Fixed a crash when using asyncio and threads.

@item 
bpo-30987@footnote{https://bugs.python.org/issue30987}: Added support for CAN ISO-TP protocol in the socket module.

@item 
bpo-30522@footnote{https://bugs.python.org/issue30522}: Added a @code{setStream} method to @code{logging.StreamHandler} to
allow the stream to be set after creation.

@item 
bpo-30502@footnote{https://bugs.python.org/issue30502}: Fix handling of long oids in ssl.  Based on patch by Christian
Heimes.

@item 
bpo-5288@footnote{https://bugs.python.org/issue5288}: Support tzinfo objects with sub-minute offsets.

@item 
bpo-30919@footnote{https://bugs.python.org/issue30919}: Fix shared memory performance regression in multiprocessing in
3.x. Shared memory used anonymous memory mappings in 2.x, while 3.x mmaps
actual files. Try to be careful to do as little disk I/O as possible.

@item 
bpo-26732@footnote{https://bugs.python.org/issue26732}: Fix too many fds in processes started with the “forkserver”
method. A child process would inherit as many fds as the number of
still-running children.

@item 
bpo-29403@footnote{https://bugs.python.org/issue29403}: Fix @code{unittest.mock}’s autospec to not fail on method-bound
builtin functions.  Patch by Aaron Gallagher.

@item 
bpo-30961@footnote{https://bugs.python.org/issue30961}: Fix decrementing a borrowed reference in tracemalloc.

@item 
bpo-19896@footnote{https://bugs.python.org/issue19896}: Fix multiprocessing.sharedctypes to recognize typecodes @code{'q'}
and @code{'Q'}.

@item 
bpo-30946@footnote{https://bugs.python.org/issue30946}: Remove obsolete code in readline module for platforms where GNU
readline is older than 2.1 or where select() is not available.

@item 
bpo-25684@footnote{https://bugs.python.org/issue25684}: Change @code{ttk.OptionMenu} radiobuttons to be unique across
instances of @code{OptionMenu}.

@item 
bpo-30886@footnote{https://bugs.python.org/issue30886}: Fix multiprocessing.Queue.join_thread(): it now waits until the
thread completes, even if the thread was started by the same process which
created the queue.

@item 
bpo-29854@footnote{https://bugs.python.org/issue29854}: Fix segfault in readline when using readline’s history-size
option.  Patch by Nir Soffer.

@item 
bpo-30794@footnote{https://bugs.python.org/issue30794}: Added multiprocessing.Process.kill method to terminate using
the SIGKILL signal on Unix.

@item 
bpo-30319@footnote{https://bugs.python.org/issue30319}: socket.close() now ignores ECONNRESET error.

@item 
bpo-30828@footnote{https://bugs.python.org/issue30828}: Fix out of bounds write in
@code{asyncio.CFuture.remove_done_callback()}.

@item 
bpo-30302@footnote{https://bugs.python.org/issue30302}: Use keywords in the @code{repr} of @code{datetime.timedelta}.

@item 
bpo-30807@footnote{https://bugs.python.org/issue30807}: signal.setitimer() may disable the timer when passed a tiny
value. Tiny values (such as 1e-6) are valid non-zero values for
setitimer(), which is specified as taking microsecond-resolution
intervals. However, on some platform, our conversion routine could convert
1e-6 into a zero interval, therefore disabling the timer instead of
(re-)scheduling it.

@item 
bpo-30441@footnote{https://bugs.python.org/issue30441}: Fix bug when modifying os.environ while iterating over it

@item 
bpo-29585@footnote{https://bugs.python.org/issue29585}: Avoid importing @code{sysconfig} from @code{site} to improve startup
speed. Python startup is about 5% faster on Linux and 30% faster on macOS.

@item 
bpo-29293@footnote{https://bugs.python.org/issue29293}: Add missing parameter “n” on
multiprocessing.Condition.notify(). The doc claims
multiprocessing.Condition behaves like threading.Condition, but its
notify() method lacked the optional “n” argument (to specify the number of
sleepers to wake up) that threading.Condition.notify() accepts.

@item 
bpo-30532@footnote{https://bugs.python.org/issue30532}: Fix email header value parser dropping folding white space in
certain cases.

@item 
bpo-30596@footnote{https://bugs.python.org/issue30596}: Add a @code{close()} method to @code{multiprocessing.Process}.

@item 
bpo-9146@footnote{https://bugs.python.org/issue9146}: Fix a segmentation fault in _hashopenssl when standard hash
functions such as md5 are not available in the linked OpenSSL library.  As
in some special FIPS-140 build environments.

@item 
bpo-29169@footnote{https://bugs.python.org/issue29169}: Update zlib to 1.2.11.

@item 
bpo-30119@footnote{https://bugs.python.org/issue30119}: ftplib.FTP.putline() now throws ValueError on commands that
contains CR or LF. Patch by Dong-hee Na.

@item 
bpo-30879@footnote{https://bugs.python.org/issue30879}: os.listdir() and os.scandir() now emit bytes names when called
with bytes-like argument.

@item 
bpo-30746@footnote{https://bugs.python.org/issue30746}: Prohibited the ‘=’ character in environment variable names in
@code{os.putenv()} and @code{os.spawn*()}.

@item 
bpo-30664@footnote{https://bugs.python.org/issue30664}: The description of a unittest subtest now preserves the order
of keyword arguments of TestCase.subTest().

@item 
bpo-21071@footnote{https://bugs.python.org/issue21071}: struct.Struct.format type is now @ref{2ef,,str} instead of
@ref{2f0,,bytes}.

@item 
bpo-29212@footnote{https://bugs.python.org/issue29212}: Fix concurrent.futures.thread.ThreadPoolExecutor threads to
have a non repr() based thread name by default when no thread_name_prefix
is supplied. They will now identify themselves as
“ThreadPoolExecutor-y_n”.

@item 
bpo-29755@footnote{https://bugs.python.org/issue29755}: Fixed the lgettext() family of functions in the gettext module.
They now always return bytes.

@item 
bpo-30616@footnote{https://bugs.python.org/issue30616}: Functional API of enum allows to create empty enums. Patched by
Dong-hee Na

@item 
bpo-30038@footnote{https://bugs.python.org/issue30038}: Fix race condition between signal delivery and wakeup file
descriptor. Patch by Nathaniel Smith.

@item 
bpo-23894@footnote{https://bugs.python.org/issue23894}: lib2to3 now recognizes @code{rb'...'} and @code{f'...'} strings.

@item 
bpo-24744@footnote{https://bugs.python.org/issue24744}: pkgutil.walk_packages function now raises ValueError if @emph{path}
is a string. Patch by Sanyam Khurana.

@item 
bpo-24484@footnote{https://bugs.python.org/issue24484}: Avoid race condition in multiprocessing cleanup.

@item 
bpo-30589@footnote{https://bugs.python.org/issue30589}: Fix multiprocessing.Process.exitcode to return the opposite of
the signal number when the process is killed by a signal (instead of 255)
when using the “forkserver” method.

@item 
bpo-28994@footnote{https://bugs.python.org/issue28994}: The traceback no longer displayed for SystemExit raised in a
callback registered by atexit.

@item 
bpo-30508@footnote{https://bugs.python.org/issue30508}: Don’t log exceptions if Task/Future “cancel()” method was
called.

@item 
bpo-30645@footnote{https://bugs.python.org/issue30645}: Fix path calculation in @code{imp.load_package()}, fixing it for
cases when a package is only shipped with bytecodes. Patch by Alexandru
Ardelean.

@item 
bpo-11822@footnote{https://bugs.python.org/issue11822}: The dis.dis() function now is able to disassemble nested code
objects.

@item 
bpo-30624@footnote{https://bugs.python.org/issue30624}: selectors does not take KeyboardInterrupt and SystemExit into
account, leaving a fd in a bad state in case of error. Patch by Giampaolo
Rodola’.

@item 
bpo-30595@footnote{https://bugs.python.org/issue30595}: multiprocessing.Queue.get() with a timeout now polls its reader
in non-blocking mode if it succeeded to acquire the lock but the acquire
took longer than the timeout.

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Updates to typing module: Add generic AsyncContextManager, add
support for ContextManager on all versions. Original PRs by Jelle Zijlstra
and Ivan Levkivskyi

@item 
bpo-30605@footnote{https://bugs.python.org/issue30605}: re.compile() no longer raises a BytesWarning when compiling a
bytes instance with misplaced inline modifier.  Patch by Roy Williams.

@item 
bpo-29870@footnote{https://bugs.python.org/issue29870}: Fix ssl sockets leaks when connection is aborted in asyncio/ssl
implementation. Patch by Michaël Sghaïer.

@item 
bpo-29743@footnote{https://bugs.python.org/issue29743}: Closing transport during handshake process leaks open socket.
Patch by Nikolay Kim

@item 
bpo-27585@footnote{https://bugs.python.org/issue27585}: Fix waiter cancellation in asyncio.Lock. Patch by Mathieu
Sornay.

@item 
bpo-30014@footnote{https://bugs.python.org/issue30014}: modify() method of poll(), epoll() and devpoll() based classes
of selectors module is around 10% faster.  Patch by Giampaolo Rodola’.

@item 
bpo-30418@footnote{https://bugs.python.org/issue30418}: On Windows, subprocess.Popen.communicate() now also ignore
EINVAL on stdin.write() if the child process is still running but closed
the pipe.

@item 
bpo-30463@footnote{https://bugs.python.org/issue30463}: Addded empty __slots__ to abc.ABC.  This allows subclassers to
deny __dict__ and __weakref__ creation.  Patch by Aaron Hall.

@item 
bpo-30520@footnote{https://bugs.python.org/issue30520}: Loggers are now pickleable.

@item 
bpo-30557@footnote{https://bugs.python.org/issue30557}: faulthandler now correctly filters and displays exception codes
on Windows

@item 
bpo-30526@footnote{https://bugs.python.org/issue30526}: Add TextIOWrapper.reconfigure() and a
TextIOWrapper.write_through attribute.

@item 
bpo-30245@footnote{https://bugs.python.org/issue30245}: Fix possible overflow when organize struct.pack_into error
message.  Patch by Yuan Liu.

@item 
bpo-30378@footnote{https://bugs.python.org/issue30378}: Fix the problem that logging.handlers.SysLogHandler cannot
handle IPv6 addresses.

@item 
bpo-16500@footnote{https://bugs.python.org/issue16500}: Allow registering at-fork handlers.

@item 
bpo-30470@footnote{https://bugs.python.org/issue30470}: Deprecate invalid ctypes call protection on Windows.  Patch by
Mariatta Wijaya.

@item 
bpo-30414@footnote{https://bugs.python.org/issue30414}: multiprocessing.Queue._feed background running thread do not
break from main loop on exception.

@item 
bpo-30003@footnote{https://bugs.python.org/issue30003}: Fix handling escape characters in HZ codec.  Based on patch by
Ma Lin.

@item 
bpo-30149@footnote{https://bugs.python.org/issue30149}: inspect.signature() now supports callables with
variable-argument parameters wrapped with partialmethod. Patch by Dong-hee
Na.

@item 
bpo-30436@footnote{https://bugs.python.org/issue30436}: importlib.find_spec() raises ModuleNotFoundError instead of
AttributeError if the specified parent module is not a package (i.e. lacks
a __path__ attribute).

@item 
bpo-30301@footnote{https://bugs.python.org/issue30301}: Fix AttributeError when using SimpleQueue.empty() under @emph{spawn}
and @emph{forkserver} start methods.

@item 
bpo-30375@footnote{https://bugs.python.org/issue30375}: Warnings emitted when compile a regular expression now always
point to the line in the user code.  Previously they could point into
inners of the re module if emitted from inside of groups or conditionals.

@item 
bpo-30329@footnote{https://bugs.python.org/issue30329}: imaplib and poplib now catch the Windows socket WSAEINVAL error
(code 10022) on shutdown(SHUT_RDWR): An invalid operation was attempted.
This error occurs sometimes on SSL connections.

@item 
bpo-29196@footnote{https://bugs.python.org/issue29196}: Removed previously deprecated in Python 2.4 classes Plist, Dict
and _InternalDict in the plistlib module.  Dict values in the result of
functions readPlist() and readPlistFromBytes() are now normal dicts.  You
no longer can use attribute access to access items of these dictionaries.

@item 
bpo-9850@footnote{https://bugs.python.org/issue9850}: The @code{macpath} is now deprecated and will be removed in
Python 3.8.

@item 
bpo-30299@footnote{https://bugs.python.org/issue30299}: Compiling regular expression in debug mode on CPython now
displays the compiled bytecode in human readable form.

@item 
bpo-30048@footnote{https://bugs.python.org/issue30048}: Fixed @code{Task.cancel()} can be ignored when the task is running
coroutine and the coroutine returned without any more @code{await}.

@item 
bpo-30266@footnote{https://bugs.python.org/issue30266}: contextlib.AbstractContextManager now supports
anti-registration by setting __enter__ = None or __exit__ = None,
following the pattern introduced in bpo-25958@footnote{https://bugs.python.org/issue25958}. Patch by Jelle Zijlstra.

@item 
bpo-30340@footnote{https://bugs.python.org/issue30340}: Enhanced regular expressions optimization. This increased the
performance of matching some patterns up to 25 times.

@item 
bpo-30298@footnote{https://bugs.python.org/issue30298}: Weaken the condition of deprecation warnings for inline
modifiers. Now allowed several subsequential inline modifiers at the start
of the pattern (e.g. @code{'(?i)(?s)...'}).  In verbose mode whitespaces and
comments now are allowed before and between inline modifiers (e.g. @code{'(?x)
(?i) (?s)...'}).

@item 
bpo-30285@footnote{https://bugs.python.org/issue30285}: Optimized case-insensitive matching and searching of regular
expressions.

@item 
bpo-29990@footnote{https://bugs.python.org/issue29990}: Fix range checking in GB18030 decoder.  Original patch by Ma
Lin.

@item 
bpo-29979@footnote{https://bugs.python.org/issue29979}: rewrite cgi.parse_multipart, reusing the FieldStorage class and
making its results consistent with those of FieldStorage for
multipart/form-data requests. Patch by Pierre Quentel.

@item 
bpo-30243@footnote{https://bugs.python.org/issue30243}: Removed the __init__ methods of _json’s scanner and encoder.
Misusing them could cause memory leaks or crashes.  Now scanner and
encoder objects are completely initialized in the __new__ methods.

@item 
bpo-30215@footnote{https://bugs.python.org/issue30215}: Compiled regular expression objects with the re.LOCALE flag no
longer depend on the locale at compile time.  Only the locale at matching
time affects the result of matching.

@item 
bpo-30185@footnote{https://bugs.python.org/issue30185}: Avoid KeyboardInterrupt tracebacks in forkserver helper process
when Ctrl-C is received.

@item 
bpo-30103@footnote{https://bugs.python.org/issue30103}: binascii.b2a_uu() and uu.encode() now support using @code{'`'} as
zero instead of space.

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Various updates to typing module: add typing.NoReturn type, use
WrapperDescriptorType, minor bug-fixes.  Original PRs by Jim
Fasarakis-Hilliard and Ivan Levkivskyi.

@item 
bpo-30205@footnote{https://bugs.python.org/issue30205}: Fix getsockname() for unbound AF_UNIX sockets on Linux.

@item 
bpo-30228@footnote{https://bugs.python.org/issue30228}: The seek() and tell() methods of io.FileIO now set the internal
seekable attribute to avoid one syscall on open() (in buffered or text
mode).

@item 
bpo-30190@footnote{https://bugs.python.org/issue30190}: unittest’s assertAlmostEqual and assertNotAlmostEqual provide a
better message in case of failure which includes the difference between
left and right arguments.  (patch by Giampaolo Rodola’)

@item 
bpo-30101@footnote{https://bugs.python.org/issue30101}: Add support for curses.A_ITALIC.

@item 
bpo-29822@footnote{https://bugs.python.org/issue29822}: inspect.isabstract() now works during __init_subclass__.  Patch
by Nate Soares.

@item 
bpo-29960@footnote{https://bugs.python.org/issue29960}: Preserve generator state when _random.Random.setstate() raises
an exception. Patch by Bryan Olson.

@item 
bpo-30070@footnote{https://bugs.python.org/issue30070}: Fixed leaks and crashes in errors handling in the parser
module.

@item 
bpo-22352@footnote{https://bugs.python.org/issue22352}: Column widths in the output of dis.dis() are now adjusted for
large line numbers and instruction offsets.

@item 
bpo-30061@footnote{https://bugs.python.org/issue30061}: Fixed crashes in IOBase methods __next__() and readlines() when
readline() or __next__() respectively return non-sizeable object. Fixed
possible other errors caused by not checking results of PyObject_Size(),
PySequence_Size(), or PyMapping_Size().

@item 
bpo-30218@footnote{https://bugs.python.org/issue30218}: Fix PathLike support for shutil.unpack_archive. Patch by Jelle
Zijlstra.

@item 
bpo-10076@footnote{https://bugs.python.org/issue10076}: Compiled regular expression and match objects in the re module
now support copy.copy() and copy.deepcopy() (they are considered atomic).

@item 
bpo-30068@footnote{https://bugs.python.org/issue30068}: _io._IOBase.readlines will check if it’s closed first when hint
is present.

@item 
bpo-29694@footnote{https://bugs.python.org/issue29694}: Fixed race condition in pathlib mkdir with flags parents=True.
Patch by Armin Rigo.

@item 
bpo-29692@footnote{https://bugs.python.org/issue29692}: Fixed arbitrary unchaining of RuntimeError exceptions in
contextlib.contextmanager.  Patch by Siddharth Velankar.

@item 
bpo-26187@footnote{https://bugs.python.org/issue26187}: Test that sqlite3 trace callback is not called multiple times
when schema is changing.  Indirectly fixed by switching to use
sqlite3_prepare_v2() in bpo-9303@footnote{https://bugs.python.org/issue9303}.  Patch by Aviv Palivoda.

@item 
bpo-30017@footnote{https://bugs.python.org/issue30017}: Allowed calling the close() method of the zip entry writer
object multiple times.  Writing to a closed writer now always produces a
ValueError.

@item 
bpo-29998@footnote{https://bugs.python.org/issue29998}: Pickling and copying ImportError now preserves name and path
attributes.

@item 
bpo-29995@footnote{https://bugs.python.org/issue29995}: re.escape() now escapes only regex special characters.

@item 
bpo-29962@footnote{https://bugs.python.org/issue29962}: Add math.remainder operation, implementing remainder as
specified in IEEE 754.

@item 
bpo-29649@footnote{https://bugs.python.org/issue29649}: Improve struct.pack_into() exception messages for problems with
the buffer size and offset.  Patch by Andrew Nester.

@item 
bpo-29654@footnote{https://bugs.python.org/issue29654}: Support If-Modified-Since HTTP header (browser cache).  Patch
by Pierre Quentel.

@item 
bpo-29931@footnote{https://bugs.python.org/issue29931}: Fixed comparison check for ipaddress.ip_interface objects.
Patch by Sanjay Sundaresan.

@item 
bpo-29953@footnote{https://bugs.python.org/issue29953}: Fixed memory leaks in the replace() method of datetime and time
objects when pass out of bound fold argument.

@item 
bpo-29942@footnote{https://bugs.python.org/issue29942}: Fix a crash in itertools.chain.from_iterable when encountering
long runs of empty iterables.

@item 
bpo-10030@footnote{https://bugs.python.org/issue10030}: Sped up reading encrypted ZIP files by 2 times.

@item 
bpo-29204@footnote{https://bugs.python.org/issue29204}: Element.getiterator() and the html parameter of XMLParser()
were deprecated only in the documentation (since Python 3.2 and 3.4
correspondingly). Now using them emits a deprecation warning.

@item 
bpo-27863@footnote{https://bugs.python.org/issue27863}: Fixed multiple crashes in ElementTree caused by race conditions
and wrong types.

@item 
bpo-25996@footnote{https://bugs.python.org/issue25996}: Added support of file descriptors in os.scandir() on Unix.
os.fwalk() is sped up by 2 times by using os.scandir().

@item 
bpo-28699@footnote{https://bugs.python.org/issue28699}: Fixed a bug in pools in multiprocessing.pool that raising an
exception at the very first of an iterable may swallow the exception or
make the program hang. Patch by Davin Potts and Xiang Zhang.

@item 
bpo-23890@footnote{https://bugs.python.org/issue23890}: unittest.TestCase.assertRaises() now manually breaks a
reference cycle to not keep objects alive longer than expected.

@item 
bpo-29901@footnote{https://bugs.python.org/issue29901}: The zipapp module now supports general path-like objects, not
just pathlib.Path.

@item 
bpo-25803@footnote{https://bugs.python.org/issue25803}: Avoid incorrect errors raised by Path.mkdir(exist_ok=True) when
the OS gives priority to errors such as EACCES over EEXIST.

@item 
bpo-29861@footnote{https://bugs.python.org/issue29861}: Release references to tasks, their arguments and their results
as soon as they are finished in multiprocessing.Pool.

@item 
bpo-19930@footnote{https://bugs.python.org/issue19930}: The mode argument of os.makedirs() no longer affects the file
permission bits of newly-created intermediate-level directories.

@item 
bpo-29884@footnote{https://bugs.python.org/issue29884}: faulthandler: Restore the old sigaltstack during teardown.
Patch by Christophe Zeitouny.

@item 
bpo-25455@footnote{https://bugs.python.org/issue25455}: Fixed crashes in repr of recursive buffered file-like objects.

@item 
bpo-29800@footnote{https://bugs.python.org/issue29800}: Fix crashes in partial.__repr__ if the keys of partial.keywords
are not strings.  Patch by Michael Seifert.

@item 
bpo-8256@footnote{https://bugs.python.org/issue8256}: Fixed possible failing or crashing input() if attributes
“encoding” or “errors” of sys.stdin or sys.stdout are not set or are not
strings.

@item 
bpo-28692@footnote{https://bugs.python.org/issue28692}: Using non-integer value for selecting a plural form in gettext
is now deprecated.

@item 
bpo-26121@footnote{https://bugs.python.org/issue26121}: Use C library implementation for math functions erf() and
erfc().

@item 
bpo-29619@footnote{https://bugs.python.org/issue29619}: os.stat() and os.DirEntry.inode() now convert inode (st_ino)
using unsigned integers.

@item 
bpo-28298@footnote{https://bugs.python.org/issue28298}: Fix a bug that prevented array ‘Q’, ‘L’ and ‘I’ from accepting
big intables (objects that have __int__) as elements.

@item 
bpo-29645@footnote{https://bugs.python.org/issue29645}: Speed up importing the webbrowser module.
webbrowser.register() is now thread-safe.

@item 
bpo-28231@footnote{https://bugs.python.org/issue28231}: The zipfile module now accepts path-like objects for external
paths.

@item 
bpo-26915@footnote{https://bugs.python.org/issue26915}: index() and count() methods of collections.abc.Sequence now
check identity before checking equality when do comparisons.

@item 
bpo-28682@footnote{https://bugs.python.org/issue28682}: Added support for bytes paths in os.fwalk().

@item 
bpo-29728@footnote{https://bugs.python.org/issue29728}: Add new @code{socket.TCP_NOTSENT_LOWAT} (Linux 3.12) constant.
Patch by Nathaniel J. Smith.

@item 
bpo-29623@footnote{https://bugs.python.org/issue29623}: Allow use of path-like object as a single argument in
ConfigParser.read(). Patch by David Ellis.

@item 
bpo-9303@footnote{https://bugs.python.org/issue9303}: Migrate sqlite3 module to _v2 API.  Patch by Aviv Palivoda.

@item 
bpo-28963@footnote{https://bugs.python.org/issue28963}: Fix out of bound iteration in
asyncio.Future.remove_done_callback implemented in C.

@item 
bpo-29704@footnote{https://bugs.python.org/issue29704}: asyncio.subprocess.SubprocessStreamProtocol no longer closes
before all pipes are closed.

@item 
bpo-29271@footnote{https://bugs.python.org/issue29271}: Fix Task.current_task and Task.all_tasks implemented in C to
accept None argument as their pure Python implementation.

@item 
bpo-29703@footnote{https://bugs.python.org/issue29703}: Fix asyncio to support instantiation of new event loops in
child processes.

@item 
bpo-29615@footnote{https://bugs.python.org/issue29615}: SimpleXMLRPCDispatcher no longer chains KeyError (or any other
exception) to exception(s) raised in the dispatched methods. Patch by Petr
Motejlek.

@item 
bpo-7769@footnote{https://bugs.python.org/issue7769}: Method register_function() of
xmlrpc.server.SimpleXMLRPCDispatcher and its subclasses can now be used as
a decorator.

@item 
bpo-29376@footnote{https://bugs.python.org/issue29376}: Fix assertion error in threading._DummyThread.is_alive().

@item 
bpo-28624@footnote{https://bugs.python.org/issue28624}: Add a test that checks that cwd parameter of Popen() accepts
PathLike objects.  Patch by Sayan Chowdhury.

@item 
bpo-28518@footnote{https://bugs.python.org/issue28518}: Start a transaction implicitly before a DML statement. Patch by
Aviv Palivoda.

@item 
bpo-29742@footnote{https://bugs.python.org/issue29742}: get_extra_info() raises exception if get called on closed ssl
transport. Patch by Nikolay Kim.

@item 
bpo-16285@footnote{https://bugs.python.org/issue16285}: urllib.parse.quote is now based on RFC 3986 and hence includes
‘~’ in the set of characters that is not quoted by default. Patch by
Christian Theune and Ratnadeep Debnath.

@item 
bpo-29532@footnote{https://bugs.python.org/issue29532}: Altering a kwarg dictionary passed to functools.partial() no
longer affects a partial object after creation.

@item 
bpo-29110@footnote{https://bugs.python.org/issue29110}: Fix file object leak in aifc.open() when file is given as a
filesystem path and is not in valid AIFF format. Patch by Anthony Zhang.

@item 
bpo-22807@footnote{https://bugs.python.org/issue22807}: Add uuid.SafeUUID and uuid.UUID.is_safe to relay information
from the platform about whether generated UUIDs are generated with a
multiprocessing safe method.

@item 
bpo-29576@footnote{https://bugs.python.org/issue29576}: Improve some deprecations in importlib. Some deprecated methods
now emit DeprecationWarnings and have better descriptive messages.

@item 
bpo-29534@footnote{https://bugs.python.org/issue29534}: Fixed different behaviour of Decimal.from_float() for _decimal
and _pydecimal. Thanks Andrew Nester.

@item 
bpo-10379@footnote{https://bugs.python.org/issue10379}: locale.format_string now supports the ‘monetary’ keyword
argument, and locale.format is deprecated.

@item 
bpo-29851@footnote{https://bugs.python.org/issue29851}: importlib.reload() now raises ModuleNotFoundError if the module
lacks a spec.

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Various updates to typing module: typing.Counter,
typing.ChainMap, improved ABC caching, etc. Original PRs by Jelle
Zijlstra, Ivan Levkivskyi, Manuel Krebber, and Łukasz Langa.

@item 
bpo-29100@footnote{https://bugs.python.org/issue29100}: Fix datetime.fromtimestamp() regression introduced in Python
3.6.0: check minimum and maximum years.

@item 
bpo-29416@footnote{https://bugs.python.org/issue29416}: Prevent infinite loop in pathlib.Path.mkdir

@item 
bpo-29444@footnote{https://bugs.python.org/issue29444}: Fixed out-of-bounds buffer access in the group() method of the
match object. Based on patch by WGH.

@item 
bpo-29377@footnote{https://bugs.python.org/issue29377}: Add WrapperDescriptorType, MethodWrapperType, and
MethodDescriptorType built-in types to types module. Original patch by
Manuel Krebber.

@item 
bpo-29218@footnote{https://bugs.python.org/issue29218}: Unused install_misc command is now removed.  It has been
documented as unused since 2000.  Patch by Eric N. Vander Weele.

@item 
bpo-29368@footnote{https://bugs.python.org/issue29368}: The extend() method is now called instead of the append()
method when unpickle collections.deque and other list-like objects. This
can speed up unpickling to 2 times.

@item 
bpo-29338@footnote{https://bugs.python.org/issue29338}: The help of a builtin or extension class now includes the
constructor signature if __text_signature__ is provided for the class.

@item 
bpo-29335@footnote{https://bugs.python.org/issue29335}: Fix subprocess.Popen.wait() when the child process has exited
to a stopped instead of terminated state (ex: when under ptrace).

@item 
bpo-29290@footnote{https://bugs.python.org/issue29290}: Fix a regression in argparse that help messages would wrap at
non-breaking spaces.

@item 
bpo-28735@footnote{https://bugs.python.org/issue28735}: Fixed the comparison of mock.MagickMock with mock.ANY.

@item 
bpo-29197@footnote{https://bugs.python.org/issue29197}: Removed deprecated function ntpath.splitunc().

@item 
bpo-29210@footnote{https://bugs.python.org/issue29210}: Removed support of deprecated argument “exclude” in
tarfile.TarFile.add().

@item 
bpo-29219@footnote{https://bugs.python.org/issue29219}: Fixed infinite recursion in the repr of uninitialized
ctypes.CDLL instances.

@item 
bpo-29192@footnote{https://bugs.python.org/issue29192}: Removed deprecated features in the http.cookies module.

@item 
bpo-29193@footnote{https://bugs.python.org/issue29193}: A format string argument for string.Formatter.format() is now
positional-only.

@item 
bpo-29195@footnote{https://bugs.python.org/issue29195}: Removed support of deprecated undocumented keyword arguments in
methods of regular expression objects.

@item 
bpo-28969@footnote{https://bugs.python.org/issue28969}: Fixed race condition in C implementation of
functools.lru_cache. KeyError could be raised when cached function with
full cache was simultaneously called from different threads with the same
uncached arguments.

@item 
bpo-20804@footnote{https://bugs.python.org/issue20804}: The unittest.mock.sentinel attributes now preserve their
identity when they are copied or pickled.

@item 
bpo-29142@footnote{https://bugs.python.org/issue29142}: In urllib.request, suffixes in no_proxy environment variable
with leading dots could match related hostnames again (e.g. .b.c matches
a.b.c). Patch by Milan Oberkirch.

@item 
bpo-28961@footnote{https://bugs.python.org/issue28961}: Fix unittest.mock._Call helper: don’t ignore the name parameter
anymore. Patch written by Jiajun Huang.

@item 
bpo-15812@footnote{https://bugs.python.org/issue15812}: inspect.getframeinfo() now correctly shows the first line of a
context. Patch by Sam Breese.

@item 
bpo-28985@footnote{https://bugs.python.org/issue28985}: Update authorizer constants in sqlite3 module. Patch by
Dingyuan Wang.

@item 
bpo-29079@footnote{https://bugs.python.org/issue29079}: Prevent infinite loop in pathlib.resolve() on Windows

@item 
bpo-13051@footnote{https://bugs.python.org/issue13051}: Fixed recursion errors in large or resized
curses.textpad.Textbox.  Based on patch by Tycho Andersen.

@item 
bpo-9770@footnote{https://bugs.python.org/issue9770}: curses.ascii predicates now work correctly with negative
integers.

@item 
bpo-28427@footnote{https://bugs.python.org/issue28427}: old keys should not remove new values from WeakValueDictionary
when collecting from another thread.

@item 
bpo-28923@footnote{https://bugs.python.org/issue28923}: Remove editor artifacts from Tix.py.

@item 
bpo-28871@footnote{https://bugs.python.org/issue28871}: Fixed a crash when deallocate deep ElementTree.

@item 
bpo-19542@footnote{https://bugs.python.org/issue19542}: Fix bugs in WeakValueDictionary.setdefault() and
WeakValueDictionary.pop() when a GC collection happens in another thread.

@item 
bpo-20191@footnote{https://bugs.python.org/issue20191}: Fixed a crash in resource.prlimit() when passing a sequence
that doesn’t own its elements as limits.

@item 
bpo-16255@footnote{https://bugs.python.org/issue16255}: subprocess.Popen uses /system/bin/sh on Android as the shell,
instead of /bin/sh.

@item 
bpo-28779@footnote{https://bugs.python.org/issue28779}: multiprocessing.set_forkserver_preload() would crash the
forkserver process if a preloaded module instantiated some multiprocessing
objects such as locks.

@item 
bpo-26937@footnote{https://bugs.python.org/issue26937}: The chown() method of the tarfile.TarFile class does not fail
now when the grp module cannot be imported, as for example on Android
platforms.

@item 
bpo-28847@footnote{https://bugs.python.org/issue28847}: dbm.dumb now supports reading read-only files and no longer
writes the index file when it is not changed.  A deprecation warning is
now emitted if the index file is missed and recreated in the ‘r’ and ‘w’
modes (will be an error in future Python releases).

@item 
bpo-27030@footnote{https://bugs.python.org/issue27030}: Unknown escapes consisting of @code{'\'} and an ASCII letter in
re.sub() replacement templates regular expressions now are errors.

@item 
bpo-28835@footnote{https://bugs.python.org/issue28835}: Fix a regression introduced in warnings.catch_warnings(): call
warnings.showwarning() if it was overridden inside the context manager.

@item 
bpo-27172@footnote{https://bugs.python.org/issue27172}: To assist with upgrades from 2.7, the previously documented
deprecation of @code{inspect.getfullargspec()} has been reversed. This
decision may be revisited again after the Python 2.7 branch is no longer
officially supported.

@item 
bpo-28740@footnote{https://bugs.python.org/issue28740}: Add sys.getandroidapilevel(): return the build time API version
of Android as an integer. Function only available on Android.

@item 
bpo-26273@footnote{https://bugs.python.org/issue26273}: Add new @code{socket.TCP_CONGESTION} (Linux 2.6.13) and
@code{socket.TCP_USER_TIMEOUT} (Linux 2.6.37) constants. Patch written by
Omar Sandoval.

@item 
bpo-28752@footnote{https://bugs.python.org/issue28752}: Restored the __reduce__() methods of datetime objects.

@item 
bpo-28727@footnote{https://bugs.python.org/issue28727}: Regular expression patterns, _sre.SRE_Pattern objects created
by re.compile(), become comparable (only x==y and x!=y operators). This
change should fix the bpo-18383@footnote{https://bugs.python.org/issue18383}: don’t duplicate warning filters when
the warnings module is reloaded (thing usually only done in unit tests).

@item 
bpo-20572@footnote{https://bugs.python.org/issue20572}: Remove the subprocess.Popen.wait endtime parameter.  It was
deprecated in 3.4 and undocumented prior to that.

@item 
bpo-25659@footnote{https://bugs.python.org/issue25659}: In ctypes, prevent a crash calling the from_buffer() and
from_buffer_copy() methods on abstract classes like Array.

@item 
bpo-28548@footnote{https://bugs.python.org/issue28548}: In the “http.server” module, parse the protocol version if
possible, to avoid using HTTP 0.9 in some error responses.

@item 
bpo-19717@footnote{https://bugs.python.org/issue19717}: Makes Path.resolve() succeed on paths that do not exist. Patch
by Vajrasky Kok

@item 
bpo-28563@footnote{https://bugs.python.org/issue28563}: Fixed possible DoS and arbitrary code execution when handle
plural form selections in the gettext module.  The expression parser now
supports exact syntax supported by GNU gettext.

@item 
bpo-28387@footnote{https://bugs.python.org/issue28387}: Fixed possible crash in _io.TextIOWrapper deallocator when the
garbage collector is invoked in other thread.  Based on patch by Sebastian
Cufre.

@item 
bpo-27517@footnote{https://bugs.python.org/issue27517}: LZMA compressor and decompressor no longer raise exceptions if
given empty data twice.  Patch by Benjamin Fogle.

@item 
bpo-28549@footnote{https://bugs.python.org/issue28549}: Fixed segfault in curses’s addch() with ncurses6.

@item 
bpo-28449@footnote{https://bugs.python.org/issue28449}: tarfile.open() with mode “r” or “r:” now tries to open a tar
file with compression before trying to open it without compression.
Otherwise it had 50% chance failed with ignore_zeros=True.

@item 
bpo-23262@footnote{https://bugs.python.org/issue23262}: The webbrowser module now supports Firefox 36+ and derived
browsers.  Based on patch by Oleg Broytman.

@item 
bpo-24241@footnote{https://bugs.python.org/issue24241}: The webbrowser in an X environment now prefers using the
default browser directly. Also, the webbrowser register() function now has
a documented ‘preferred’ argument, to specify browsers to be returned by
get() with no arguments. Patch by David Steele

@item 
bpo-27939@footnote{https://bugs.python.org/issue27939}: Fixed bugs in tkinter.ttk.LabeledScale and tkinter.Scale caused
by representing the scale as float value internally in Tk.  tkinter.IntVar
now works if float value is set to underlying Tk variable.

@item 
bpo-28255@footnote{https://bugs.python.org/issue28255}: calendar.TextCalendar.prweek() no longer prints a space after a
weeks’s calendar.  calendar.TextCalendar.pryear() no longer prints
redundant newline after a year’s calendar.  Based on patch by Xiang Zhang.

@item 
bpo-28255@footnote{https://bugs.python.org/issue28255}: calendar.TextCalendar.prmonth() no longer prints a space at the
start of new line after printing a month’s calendar.  Patch by Xiang
Zhang.

@item 
bpo-20491@footnote{https://bugs.python.org/issue20491}: The textwrap.TextWrapper class now honors non-breaking spaces.
Based on patch by Kaarle Ritvanen.

@item 
bpo-28353@footnote{https://bugs.python.org/issue28353}: os.fwalk() no longer fails on broken links.

@item 
bpo-28430@footnote{https://bugs.python.org/issue28430}: Fix iterator of C implemented asyncio.Future doesn’t accept
non-None value is passed to it.send(val).

@item 
bpo-27025@footnote{https://bugs.python.org/issue27025}: Generated names for Tkinter widgets now start by the “!” prefix
for readability.

@item 
bpo-25464@footnote{https://bugs.python.org/issue25464}: Fixed HList.header_exists() in tkinter.tix module by addin a
workaround to Tix library bug.

@item 
bpo-28488@footnote{https://bugs.python.org/issue28488}: shutil.make_archive() no longer adds entry “./” to ZIP archive.

@item 
bpo-25953@footnote{https://bugs.python.org/issue25953}: re.sub() now raises an error for invalid numerical group
reference in replacement template even if the pattern is not found in the
string.  Error message for invalid group reference now includes the group
index and the position of the reference. Based on patch by SilentGhost.

@item 
bpo-28469@footnote{https://bugs.python.org/issue28469}: timeit now uses the sequence 1, 2, 5, 10, 20, 50,… instead of
1, 10, 100,… for autoranging.

@item 
bpo-28115@footnote{https://bugs.python.org/issue28115}: Command-line interface of the zipfile module now uses argparse.
Added support of long options.

@item 
bpo-18219@footnote{https://bugs.python.org/issue18219}: Optimize csv.DictWriter for large number of columns. Patch by
Mariatta Wijaya.

@item 
bpo-28448@footnote{https://bugs.python.org/issue28448}: Fix C implemented asyncio.Future didn’t work on Windows.

@item 
bpo-23214@footnote{https://bugs.python.org/issue23214}: In the “io” module, the argument to BufferedReader and
BytesIO’s read1() methods is now optional and can be -1, matching the
BufferedIOBase specification.

@item 
bpo-28480@footnote{https://bugs.python.org/issue28480}: Fix error building socket module when multithreading is
disabled.

@item 
bpo-28240@footnote{https://bugs.python.org/issue28240}: timeit: remove @code{-c/--clock} and @code{-t/--time} command line
options which were deprecated since Python 3.3.

@item 
bpo-28240@footnote{https://bugs.python.org/issue28240}: timeit now repeats the benchmarks 5 times instead of only 3 to
make benchmarks more reliable.

@item 
bpo-28240@footnote{https://bugs.python.org/issue28240}: timeit autorange now uses a single loop iteration if the
benchmark takes less than 10 seconds, instead of 10 iterations. “python3
-m timeit -s ‘import time’ ‘time.sleep(1)’” now takes 4 seconds instead of
40 seconds.

@item 
Distutils.sdist now looks for README and setup.py files with case
sensitivity. This behavior matches that found in Setuptools 6.0 and later.
See setuptools 100@footnote{https://github.com/pypa/setuptools/issues/100} for
rationale.

@item 
bpo-24452@footnote{https://bugs.python.org/issue24452}: Make webbrowser support Chrome on Mac OS X. Patch by Ned
Batchelder.

@item 
bpo-20766@footnote{https://bugs.python.org/issue20766}: Fix references leaked by pdb in the handling of SIGINT
handlers.

@item 
bpo-27998@footnote{https://bugs.python.org/issue27998}: Fixed bytes path support in os.scandir() on Windows. Patch by
Eryk Sun.

@item 
bpo-28317@footnote{https://bugs.python.org/issue28317}: The disassembler now decodes FORMAT_VALUE argument.

@item 
bpo-28380@footnote{https://bugs.python.org/issue28380}: unittest.mock Mock autospec functions now properly support
assert_called, assert_not_called, and assert_called_once.

@item 
bpo-28229@footnote{https://bugs.python.org/issue28229}: lzma module now supports pathlib.

@item 
bpo-28321@footnote{https://bugs.python.org/issue28321}: Fixed writing non-BMP characters with binary format in
plistlib.

@item 
bpo-28225@footnote{https://bugs.python.org/issue28225}: bz2 module now supports pathlib.  Initial patch by Ethan
Furman.

@item 
bpo-28227@footnote{https://bugs.python.org/issue28227}: gzip now supports pathlib.  Patch by Ethan Furman.

@item 
bpo-28332@footnote{https://bugs.python.org/issue28332}: Deprecated silent truncations in socket.htons and socket.ntohs.
Original patch by Oren Milman.

@item 
bpo-27358@footnote{https://bugs.python.org/issue27358}: Optimized merging var-keyword arguments and improved error
message when passing a non-mapping as a var-keyword argument.

@item 
bpo-28257@footnote{https://bugs.python.org/issue28257}: Improved error message when passing a non-iterable as a
var-positional argument.  Added opcode BUILD_TUPLE_UNPACK_WITH_CALL.

@item 
bpo-28322@footnote{https://bugs.python.org/issue28322}: Fixed possible crashes when unpickle itertools objects from
incorrect pickle data.  Based on patch by John Leitch.

@item 
bpo-28228@footnote{https://bugs.python.org/issue28228}: imghdr now supports pathlib.

@item 
bpo-28226@footnote{https://bugs.python.org/issue28226}: compileall now supports pathlib.

@item 
bpo-28314@footnote{https://bugs.python.org/issue28314}: Fix function declaration (C flags) for the getiterator() method
of xml.etree.ElementTree.Element.

@item 
bpo-28148@footnote{https://bugs.python.org/issue28148}: Stop using localtime() and gmtime() in the time module.
Introduced platform independent _PyTime_localtime API that is similar to
POSIX localtime_r, but available on all platforms.  Patch by Ed Schouten.

@item 
bpo-28253@footnote{https://bugs.python.org/issue28253}: Fixed calendar functions for extreme months: 0001-01 and
9999-12. Methods itermonthdays() and itermonthdays2() are reimplemented so
that they don’t call itermonthdates() which can cause datetime.date
under/overflow.

@item 
bpo-28275@footnote{https://bugs.python.org/issue28275}: Fixed possible use after free in the decompress() methods of
the LZMADecompressor and BZ2Decompressor classes. Original patch by John
Leitch.

@item 
bpo-27897@footnote{https://bugs.python.org/issue27897}: Fixed possible crash in sqlite3.Connection.create_collation()
if pass invalid string-like object as a name.  Patch by Xiang Zhang.

@item 
bpo-18844@footnote{https://bugs.python.org/issue18844}: random.choices() now has k as a keyword-only argument to
improve the readability of common cases and come into line with the
signature used in other languages.

@item 
bpo-18893@footnote{https://bugs.python.org/issue18893}: Fix invalid exception handling in Lib/ctypes/macholib/dyld.py.
Patch by Madison May.

@item 
bpo-27611@footnote{https://bugs.python.org/issue27611}: Fixed support of default root window in the tkinter.tix module.
Added the master parameter in the DisplayStyle constructor.

@item 
bpo-27348@footnote{https://bugs.python.org/issue27348}: In the traceback module, restore the formatting of exception
messages like “Exception: None”.  This fixes a regression introduced in
3.5a2.

@item 
bpo-25651@footnote{https://bugs.python.org/issue25651}: Allow falsy values to be used for msg parameter of subTest().

@item 
bpo-27778@footnote{https://bugs.python.org/issue27778}: Fix a memory leak in os.getrandom() when the getrandom() is
interrupted by a signal and a signal handler raises a Python exception.

@item 
bpo-28200@footnote{https://bugs.python.org/issue28200}: Fix memory leak on Windows in the os module (fix
path_converter() function).

@item 
bpo-25400@footnote{https://bugs.python.org/issue25400}: RobotFileParser now correctly returns default values for
crawl_delay and request_rate.  Initial patch by Peter Wirtz.

@item 
bpo-27932@footnote{https://bugs.python.org/issue27932}: Prevent memory leak in win32_ver().

@item 
Fix UnboundLocalError in socket._sendfile_use_sendfile.

@item 
bpo-28075@footnote{https://bugs.python.org/issue28075}: Check for ERROR_ACCESS_DENIED in Windows implementation of
os.stat().  Patch by Eryk Sun.

@item 
bpo-22493@footnote{https://bugs.python.org/issue22493}: Warning message emitted by using inline flags in the middle of
regular expression now contains a (truncated) regex pattern. Patch by Tim
Graham.

@item 
bpo-25270@footnote{https://bugs.python.org/issue25270}: Prevent codecs.escape_encode() from raising SystemError when an
empty bytestring is passed.

@item 
bpo-28181@footnote{https://bugs.python.org/issue28181}: Get antigravity over HTTPS. Patch by Kaartic Sivaraam.

@item 
bpo-25895@footnote{https://bugs.python.org/issue25895}: Enable WebSocket URL schemes in urllib.parse.urljoin. Patch by
Gergely Imreh and Markus Holtermann.

@item 
bpo-28114@footnote{https://bugs.python.org/issue28114}: Fix a crash in parse_envlist() when env contains byte strings.
Patch by Eryk Sun.

@item 
bpo-27599@footnote{https://bugs.python.org/issue27599}: Fixed buffer overrun in binascii.b2a_qp() and
binascii.a2b_qp().

@item 
bpo-27906@footnote{https://bugs.python.org/issue27906}: Fix socket accept exhaustion during high TCP traffic. Patch by
Kevin Conway.

@item 
bpo-28174@footnote{https://bugs.python.org/issue28174}: Handle when SO_REUSEPORT isn’t properly supported. Patch by
Seth Michael Larson.

@item 
bpo-26654@footnote{https://bugs.python.org/issue26654}: Inspect functools.partial in asyncio.Handle.__repr__. Patch by
iceboy.

@item 
bpo-26909@footnote{https://bugs.python.org/issue26909}: Fix slow pipes IO in asyncio. Patch by INADA Naoki.

@item 
bpo-28176@footnote{https://bugs.python.org/issue28176}: Fix callbacks race in asyncio.SelectorLoop.sock_connect.

@item 
bpo-27759@footnote{https://bugs.python.org/issue27759}: Fix selectors incorrectly retain invalid file descriptors.
Patch by Mark Williams.

@item 
bpo-28325@footnote{https://bugs.python.org/issue28325}: Remove vestigial MacOS 9 macurl2path module and its tests.

@item 
bpo-28368@footnote{https://bugs.python.org/issue28368}: Refuse monitoring processes if the child watcher has no loop
attached. Patch by Vincent Michel.

@item 
bpo-28369@footnote{https://bugs.python.org/issue28369}: Raise RuntimeError when transport’s FD is used with add_reader,
add_writer, etc.

@item 
bpo-28370@footnote{https://bugs.python.org/issue28370}: Speedup asyncio.StreamReader.readexactly. Patch by Коренберг
Марк.

@item 
bpo-28371@footnote{https://bugs.python.org/issue28371}: Deprecate passing asyncio.Handles to run_in_executor.

@item 
bpo-28372@footnote{https://bugs.python.org/issue28372}: Fix asyncio to support formatting of non-python coroutines.

@item 
bpo-28399@footnote{https://bugs.python.org/issue28399}: Remove UNIX socket from FS before binding. Patch by Коренберг
Марк.

@item 
bpo-27972@footnote{https://bugs.python.org/issue27972}: Prohibit Tasks to await on themselves.

@item 
bpo-24142@footnote{https://bugs.python.org/issue24142}: Reading a corrupt config file left configparser in an invalid
state. Original patch by Florian Höch.

@item 
bpo-29581@footnote{https://bugs.python.org/issue29581}: ABCMeta.__new__ now accepts @code{**kwargs}, allowing abstract
base classes to use keyword parameters in __init_subclass__. Patch by Nate
Soares.

@item 
bpo-25532@footnote{https://bugs.python.org/issue25532}: inspect.unwrap() will now only try to unwrap an object
sys.getrecursionlimit() times, to protect against objects which create a
new object on every attribute access.

@item 
bpo-30177@footnote{https://bugs.python.org/issue30177}: path.resolve(strict=False) no longer cuts the path after the
first element not present in the filesystem.  Patch by Antoine Pietri.
@end itemize

@node Documentation<37>,Tests<33>,Library<37>,Python 3 7 0 alpha 1
@anchor{whatsnew/changelog id318}@anchor{137d}
@subsubsection Documentation



@itemize -

@item 
bpo-31294@footnote{https://bugs.python.org/issue31294}: Fix incomplete code snippet in the ZeroMQSocketListener and
ZeroMQSocketHandler examples and adapt them to Python 3.

@item 
bpo-21649@footnote{https://bugs.python.org/issue21649}: Add RFC 7525 and Mozilla server side TLS links to SSL
documentation.

@item 
bpo-31128@footnote{https://bugs.python.org/issue31128}: Allow the pydoc server to bind to arbitrary hostnames.

@item 
bpo-30803@footnote{https://bugs.python.org/issue30803}: Clarify doc on truth value testing. Original patch by Peter
Thomassen.

@item 
bpo-30176@footnote{https://bugs.python.org/issue30176}: Add missing attribute related constants in curses
documentation.

@item 
bpo-30052@footnote{https://bugs.python.org/issue30052}: the link targets for @ref{2f0,,bytes()} and @ref{2f1,,bytearray()} are
now their respective type definitions, rather than the corresponding
builtin function entries. Use @ref{137e,,bytes} and
@ref{137f,,bytearray} to reference the latter. In order to
ensure this and future cross-reference updates are applied automatically,
the daily documentation builds now disable the default output caching
features in Sphinx.

@item 
bpo-26985@footnote{https://bugs.python.org/issue26985}: Add missing info of code object in inspect documentation.

@item 
bpo-19824@footnote{https://bugs.python.org/issue19824}: Improve the documentation for, and links to, template strings
by emphasizing their utility for internationalization, and by clarifying
some usage constraints. (See also: bpo-20314@footnote{https://bugs.python.org/issue20314}, bpo-12518@footnote{https://bugs.python.org/issue12518})

@item 
bpo-28929@footnote{https://bugs.python.org/issue28929}: Link the documentation to its source file on GitHub.

@item 
bpo-25008@footnote{https://bugs.python.org/issue25008}: Document smtpd.py as effectively deprecated and add a pointer
to aiosmtpd, a third-party asyncio-based replacement.

@item 
bpo-26355@footnote{https://bugs.python.org/issue26355}: Add canonical header link on each page to corresponding major
version of the documentation. Patch by Matthias Bussonnier.

@item 
bpo-29349@footnote{https://bugs.python.org/issue29349}: Fix Python 2 syntax in code for building the documentation.

@item 
bpo-23722@footnote{https://bugs.python.org/issue23722}: The data model reference and the porting section in the 3.6
What’s New guide now cover the additional @code{__classcell__} handling
needed for custom metaclasses to fully support 
@geindex Python Enhancement Proposals; PEP 487
PEP 487@footnote{https://www.python.org/dev/peps/pep-0487} and
zero-argument @code{super()}.

@item 
bpo-28513@footnote{https://bugs.python.org/issue28513}: Documented command-line interface of zipfile.
@end itemize

@node Tests<33>,Build<34>,Documentation<37>,Python 3 7 0 alpha 1
@anchor{whatsnew/changelog id319}@anchor{1380}
@subsubsection Tests



@itemize -

@item 
bpo-29639@footnote{https://bugs.python.org/issue29639}: test.support.HOST is now “localhost”, a new HOSTv4 constant has
been added for your @code{127.0.0.1} needs, similar to the existing HOSTv6
constant.

@item 
bpo-31320@footnote{https://bugs.python.org/issue31320}: Silence traceback in test_ssl

@item 
bpo-31346@footnote{https://bugs.python.org/issue31346}: Prefer PROTOCOL_TLS_CLIENT and PROTOCOL_TLS_SERVER protocols
for SSLContext.

@item 
bpo-25674@footnote{https://bugs.python.org/issue25674}: Remove sha256.tbs-internet.com ssl test

@item 
bpo-30715@footnote{https://bugs.python.org/issue30715}: Address ALPN callback changes for OpenSSL 1.1.0f. The latest
version behaves like OpenSSL 1.0.2 and no longer aborts handshake.

@item 
bpo-30822@footnote{https://bugs.python.org/issue30822}: regrtest: Exclude tzdata from regrtest –all. When running the
test suite using –use=all / -u all, exclude tzdata since it makes
test_datetime too slow (15-20 min on some buildbots) which then times out
on some buildbots. Fix also regrtest command line parser to allow passing
-u extralargefile to run test_zipfile64.

@item 
bpo-30695@footnote{https://bugs.python.org/issue30695}: Add the @code{set_nomemory(start, stop)} and @code{remove_mem_hooks()}
functions to the _testcapi module.

@item 
bpo-30357@footnote{https://bugs.python.org/issue30357}: test_thread: setUp() now uses support.threading_setup() and
support.threading_cleanup() to wait until threads complete to avoid random
side effects on following tests. Initial patch written by Grzegorz
Grzywacz.

@item 
bpo-30197@footnote{https://bugs.python.org/issue30197}: Enhanced functions swap_attr() and swap_item() in the
test.support module. They now work when delete replaced attribute or item
inside the with statement.  The old value of the attribute or item (or
None if it doesn’t exist) now will be assigned to the target of the “as”
clause, if there is one.

@item 
bpo-24932@footnote{https://bugs.python.org/issue24932}: Use proper command line parsing in _testembed

@item 
bpo-28950@footnote{https://bugs.python.org/issue28950}: Disallow -j0 to be combined with -T/-l in regrtest command line
arguments.

@item 
bpo-28683@footnote{https://bugs.python.org/issue28683}: Fix the tests that bind() a unix socket and raise
PermissionError on Android for a non-root user.

@item 
bpo-26936@footnote{https://bugs.python.org/issue26936}: Fix the test_socket failures on Android - getservbyname(),
getservbyport() and getaddrinfo() are broken on some Android API levels.

@item 
bpo-28666@footnote{https://bugs.python.org/issue28666}: Now test.support.rmtree is able to remove unwritable or
unreadable directories.

@item 
bpo-23839@footnote{https://bugs.python.org/issue23839}: Various caches now are cleared before running every test file.

@item 
bpo-26944@footnote{https://bugs.python.org/issue26944}: Fix test_posix for Android where ‘id -G’ is entirely wrong or
missing the effective gid.

@item 
bpo-28409@footnote{https://bugs.python.org/issue28409}: regrtest: fix the parser of command line arguments.

@item 
bpo-28217@footnote{https://bugs.python.org/issue28217}: Adds _testconsole module to test console input.

@item 
bpo-26939@footnote{https://bugs.python.org/issue26939}: Add the support.setswitchinterval() function to fix
test_functools hanging on the Android armv7 qemu emulator.
@end itemize

@node Build<34>,Windows<32>,Tests<33>,Python 3 7 0 alpha 1
@anchor{whatsnew/changelog id320}@anchor{1381}
@subsubsection Build



@itemize -

@item 
bpo-31354@footnote{https://bugs.python.org/issue31354}: Allow –with-lto to be used on all builds, not just @code{make
profile-opt}.

@item 
bpo-31370@footnote{https://bugs.python.org/issue31370}: Remove support for building –without-threads. This option is
not really useful anymore in the 21st century. Removing lots of
conditional paths allows us to simplify the code base, including in
difficult to maintain low-level internal code.

@item 
bpo-31341@footnote{https://bugs.python.org/issue31341}: Per 
@geindex Python Enhancement Proposals; PEP 11
PEP 11@footnote{https://www.python.org/dev/peps/pep-0011}, support for the IRIX operating system was
removed.

@item 
bpo-30854@footnote{https://bugs.python.org/issue30854}: Fix compile error when compiling –without-threads. Patch by
Masayuki Yamamoto.

@item 
bpo-30687@footnote{https://bugs.python.org/issue30687}: Locate msbuild.exe on Windows when building rather than
vcvarsall.bat

@item 
bpo-20210@footnote{https://bugs.python.org/issue20210}: Support the @emph{disabled} marker in Setup files. Extension modules
listed after this marker are not built at all, neither by the Makefile nor
by setup.py.

@item 
bpo-29941@footnote{https://bugs.python.org/issue29941}: Add @code{--with-assertions} configure flag to explicitly enable C
@code{assert()} checks. Defaults to off. @code{--with-pydebug} implies
@code{--with-assertions}.

@item 
bpo-28787@footnote{https://bugs.python.org/issue28787}: Fix out-of-tree builds of Python when configured with
@code{--with--dtrace}.

@item 
bpo-29243@footnote{https://bugs.python.org/issue29243}: Prevent unnecessary rebuilding of Python during @code{make test},
@code{make install} and some other make targets when configured with
@code{--enable-optimizations}.

@item 
bpo-23404@footnote{https://bugs.python.org/issue23404}: Don’t regenerate generated files based on file modification
time anymore: the action is now explicit. Replace @code{make touch} with
@code{make regen-all}.

@item 
bpo-29643@footnote{https://bugs.python.org/issue29643}: Fix @code{--enable-optimization} didn’t work.

@item 
bpo-27593@footnote{https://bugs.python.org/issue27593}: sys.version and the platform module python_build(),
python_branch(), and python_revision() functions now use git information
rather than hg when building from a repo.

@item 
bpo-29572@footnote{https://bugs.python.org/issue29572}: Update Windows build and OS X installers to use OpenSSL 1.0.2k.

@item 
bpo-27659@footnote{https://bugs.python.org/issue27659}: Prohibit implicit C function declarations: use
@code{-Werror=implicit-function-declaration} when possible (GCC and Clang,
but it depends on the compiler version). Patch written by Chi Hsuan Yen.

@item 
bpo-29384@footnote{https://bugs.python.org/issue29384}: Remove old Be OS helper scripts.

@item 
bpo-26851@footnote{https://bugs.python.org/issue26851}: Set Android compilation and link flags.

@item 
bpo-28768@footnote{https://bugs.python.org/issue28768}: Fix implicit declaration of function _setmode. Patch by
Masayuki Yamamoto

@item 
bpo-29080@footnote{https://bugs.python.org/issue29080}: Removes hard dependency on hg.exe from PCBuild/build.bat

@item 
bpo-23903@footnote{https://bugs.python.org/issue23903}: Added missed names to PC/python3.def.

@item 
bpo-28762@footnote{https://bugs.python.org/issue28762}: lockf() is available on Android API level 24, but the F_LOCK
macro is not defined in android-ndk-r13.

@item 
bpo-28538@footnote{https://bugs.python.org/issue28538}: Fix the compilation error that occurs because if_nameindex() is
available on Android API level 24, but the if_nameindex structure is not
defined.

@item 
bpo-20211@footnote{https://bugs.python.org/issue20211}: Do not add the directory for installing C header files and the
directory for installing object code libraries to the cross compilation
search paths. Original patch by Thomas Petazzoni.

@item 
bpo-28849@footnote{https://bugs.python.org/issue28849}: Do not define sys.implementation._multiarch on Android.

@item 
bpo-10656@footnote{https://bugs.python.org/issue10656}: Fix out-of-tree building on AIX.  Patch by Tristan Carel and
Michael Haubenwallner.

@item 
bpo-26359@footnote{https://bugs.python.org/issue26359}: Rename –with-optimiations to –enable-optimizations.

@item 
bpo-28444@footnote{https://bugs.python.org/issue28444}: Fix missing extensions modules when cross compiling.

@item 
bpo-28208@footnote{https://bugs.python.org/issue28208}: Update Windows build and OS X installers to use SQLite 3.14.2.

@item 
bpo-28248@footnote{https://bugs.python.org/issue28248}: Update Windows build and OS X installers to use OpenSSL 1.0.2j.

@item 
bpo-21124@footnote{https://bugs.python.org/issue21124}: Fix building the _struct module on Cygwin by passing @code{NULL}
instead of @code{&PyType_Type} to PyVarObject_HEAD_INIT.  Patch by Masayuki
Yamamoto.

@item 
bpo-13756@footnote{https://bugs.python.org/issue13756}: Fix building extensions modules on Cygwin.  Patch by Roumen
Petrov, based on original patch by Jason Tishler.

@item 
bpo-21085@footnote{https://bugs.python.org/issue21085}: Add configure check for siginfo_t.si_band, which Cygwin does
not provide. Patch by Masayuki Yamamoto with review and rebase by Erik
Bray.

@item 
bpo-28258@footnote{https://bugs.python.org/issue28258}: Fixed build with Estonian locale (python-config and distclean
targets in Makefile).  Patch by Arfrever Frehtes Taifersar Arahesis.

@item 
bpo-26661@footnote{https://bugs.python.org/issue26661}: setup.py now detects system libffi with multiarch wrapper.

@item 
bpo-27979@footnote{https://bugs.python.org/issue27979}: A full copy of libffi is no longer bundled for use when
building _ctypes on non-OSX UNIX platforms.  An installed copy of libffi
is now required when building _ctypes on such platforms.

@item 
bpo-15819@footnote{https://bugs.python.org/issue15819}: Remove redundant include search directory option for building
outside the source tree.

@item 
bpo-28676@footnote{https://bugs.python.org/issue28676}: Prevent missing ‘getentropy’ declaration warning on macOS.
Patch by Gareth Rees.
@end itemize

@node Windows<32>,IDLE<32>,Build<34>,Python 3 7 0 alpha 1
@anchor{whatsnew/changelog id321}@anchor{1382}
@subsubsection Windows



@itemize -

@item 
bpo-31392@footnote{https://bugs.python.org/issue31392}: Update Windows build to use OpenSSL 1.1.0f

@item 
bpo-30389@footnote{https://bugs.python.org/issue30389}: Adds detection of Visual Studio 2017 to distutils on Windows.

@item 
bpo-31358@footnote{https://bugs.python.org/issue31358}: zlib is no longer bundled in the CPython source, instead it is
downloaded on demand just like bz2, lzma, OpenSSL, Tcl/Tk, and SQLite.

@item 
bpo-31340@footnote{https://bugs.python.org/issue31340}: Change to building with MSVC v141 (included with Visual Studio
2017)

@item 
bpo-30581@footnote{https://bugs.python.org/issue30581}: os.cpu_count() now returns the correct number of processors on
Windows when the number of logical processors is greater than 64.

@item 
bpo-30916@footnote{https://bugs.python.org/issue30916}: Pre-build OpenSSL, Tcl and Tk and include the binaries in the
build.

@item 
bpo-30731@footnote{https://bugs.python.org/issue30731}: Add a missing xmlns to python.manifest so that it matches the
schema.

@item 
bpo-30291@footnote{https://bugs.python.org/issue30291}: Allow requiring 64-bit interpreters from py.exe using -64
suffix. Contributed by Steve (Gadget) Barnes.

@item 
bpo-30362@footnote{https://bugs.python.org/issue30362}: Adds list options (-0, -0p) to py.exe launcher. Contributed by
Steve Barnes.

@item 
bpo-23451@footnote{https://bugs.python.org/issue23451}: Fix socket deprecation warnings in socketmodule.c. Patch by
Segev Finer.

@item 
bpo-30450@footnote{https://bugs.python.org/issue30450}: The build process on Windows no longer depends on Subversion,
instead pulling external code from GitHub via a Python script.  If Python
3.6 is not found on the system (via @code{py -3.6}), NuGet is used to
download a copy of 32-bit Python.

@item 
bpo-29579@footnote{https://bugs.python.org/issue29579}: Removes readme.txt from the installer.

@item 
bpo-25778@footnote{https://bugs.python.org/issue25778}: winreg does not truncate string correctly (Patch by Eryk Sun)

@item 
bpo-28896@footnote{https://bugs.python.org/issue28896}: Deprecate WindowsRegistryFinder and disable it by default

@item 
bpo-28522@footnote{https://bugs.python.org/issue28522}: Fixes mishandled buffer reallocation in getpathp.c

@item 
bpo-28402@footnote{https://bugs.python.org/issue28402}: Adds signed catalog files for stdlib on Windows.

@item 
bpo-28333@footnote{https://bugs.python.org/issue28333}: Enables Unicode for ps1/ps2 and input() prompts. (Patch by Eryk
Sun)

@item 
bpo-28251@footnote{https://bugs.python.org/issue28251}: Improvements to help manuals on Windows.

@item 
bpo-28110@footnote{https://bugs.python.org/issue28110}: launcher.msi has different product codes between 32-bit and
64-bit

@item 
bpo-28161@footnote{https://bugs.python.org/issue28161}: Opening CON for write access fails

@item 
bpo-28162@footnote{https://bugs.python.org/issue28162}: WindowsConsoleIO readall() fails if first line starts with
Ctrl+Z

@item 
bpo-28163@footnote{https://bugs.python.org/issue28163}: WindowsConsoleIO fileno() passes wrong flags to _open_osfhandle

@item 
bpo-28164@footnote{https://bugs.python.org/issue28164}: _PyIO_get_console_type fails for various paths

@item 
bpo-28137@footnote{https://bugs.python.org/issue28137}: Renames Windows path file to ._pth

@item 
bpo-28138@footnote{https://bugs.python.org/issue28138}: Windows ._pth file should allow import site
@end itemize

@node IDLE<32>,Tools/Demos<17>,Windows<32>,Python 3 7 0 alpha 1
@anchor{whatsnew/changelog id322}@anchor{1383}
@subsubsection IDLE



@itemize -

@item 
bpo-31493@footnote{https://bugs.python.org/issue31493}: IDLE code context – fix code update and font update timers.
Canceling timers prevents a warning message when test_idle completes.

@item 
bpo-31488@footnote{https://bugs.python.org/issue31488}: IDLE - Update non-key options in former extension classes. When
applying configdialog changes, call .reload for each feature class. Change
ParenMatch so updated options affect existing instances attached to
existing editor windows.

@item 
bpo-31477@footnote{https://bugs.python.org/issue31477}: IDLE - Improve rstrip entry in doc. Strip trailing whitespace
strips more than blank spaces.  Multiline string literals are not skipped.

@item 
bpo-31480@footnote{https://bugs.python.org/issue31480}: IDLE - make tests pass with zzdummy extension disabled by
default.

@item 
bpo-31421@footnote{https://bugs.python.org/issue31421}: Document how IDLE runs tkinter programs. IDLE calls tcl/tk
update in the background in order to make live interaction and
experimentation with tkinter applications much easier.

@item 
bpo-31414@footnote{https://bugs.python.org/issue31414}: IDLE – fix tk entry box tests by deleting first. Adding to an
int entry is not the same as deleting and inserting because int(‘’) will
fail.

@item 
bpo-31051@footnote{https://bugs.python.org/issue31051}: Rearrange IDLE configdialog GenPage into Window, Editor, and
Help sections.

@item 
bpo-30617@footnote{https://bugs.python.org/issue30617}: IDLE - Add docstrings and tests for outwin subclass of editor.
Move some data and functions from the class to module level. Patch by
Cheryl Sabella.

@item 
bpo-31287@footnote{https://bugs.python.org/issue31287}: IDLE - Do not modify tkinter.message in test_configdialog.

@item 
bpo-27099@footnote{https://bugs.python.org/issue27099}: Convert IDLE’s built-in ‘extensions’ to regular features. About
10 IDLE features were implemented as supposedly optional extensions. Their
different behavior could be confusing or worse for users and not good for
maintenance. Hence the conversion. The main difference for users is that
user configurable key bindings for builtin features are now handled
uniformly. Now, editing a binding in a keyset only affects its value in
the keyset. All bindings are defined together in the system-specific
default keysets in config-extensions.def. All custom keysets are saved as
a whole in config-extension.cfg.  All take effect as soon as one clicks
Apply or Ok. The affected events are ‘<<force-open-completions>>’,
‘<<expand-word>>’, ‘<<force-open-calltip>>’, ‘<<flash-paren>>’,
‘<<format-paragraph>>’, ‘<<run-module>>’, ‘<<check-module>>’, and
‘<<zoom-height>>’.  Any (global) customizations made before 3.6.3 will not
affect their keyset-specific customization after 3.6.3. and vice versa.
Initial patch by Charles Wohlganger.

@item 
bpo-31206@footnote{https://bugs.python.org/issue31206}: IDLE: Factor HighPage(Frame) class from ConfigDialog. Patch by
Cheryl Sabella.

@item 
bpo-31001@footnote{https://bugs.python.org/issue31001}: Add tests for configdialog highlight tab.  Patch by Cheryl
Sabella.

@item 
bpo-31205@footnote{https://bugs.python.org/issue31205}: IDLE: Factor KeysPage(Frame) class from ConfigDialog.  The
slightly modified tests continue to pass.  Patch by Cheryl Sabella.

@item 
bpo-31130@footnote{https://bugs.python.org/issue31130}: IDLE – stop leaks in test_configdialog. Initial patch by
Victor Stinner.

@item 
bpo-31002@footnote{https://bugs.python.org/issue31002}: Add tests for configdialog keys tab. Patch by Cheryl Sabella.

@item 
bpo-19903@footnote{https://bugs.python.org/issue19903}: IDLE: Calltips use @ref{1c1,,inspect.signature} instead of
@ref{1c2,,inspect.getfullargspec}. This improves calltips for builtins converted to
use Argument Clinic. Patch by Louie Lu.

@item 
bpo-31083@footnote{https://bugs.python.org/issue31083}: IDLE - Add an outline of a TabPage class in configdialog.
Update existing classes to match outline. Initial patch by Cheryl Sabella.

@item 
bpo-31050@footnote{https://bugs.python.org/issue31050}: Factor GenPage(Frame) class from ConfigDialog. The slightly
modified tests continue to pass. Patch by Cheryl Sabella.

@item 
bpo-31004@footnote{https://bugs.python.org/issue31004}: IDLE - Factor FontPage(Frame) class from ConfigDialog. Slightly
modified tests continue to pass. Fix General tests. Patch mostly by Cheryl
Sabella.

@item 
bpo-30781@footnote{https://bugs.python.org/issue30781}: IDLE - Use ttk widgets in ConfigDialog. Patches by Terry Jan
Reedy and Cheryl Sabella.

@item 
bpo-31060@footnote{https://bugs.python.org/issue31060}: IDLE - Finish rearranging methods of ConfigDialog Grouping
methods pertaining to each tab and the buttons will aid writing tests and
improving the tabs and will enable splitting the groups into classes.

@item 
bpo-30853@footnote{https://bugs.python.org/issue30853}: IDLE – Factor a VarTrace class out of ConfigDialog. Instance
tracers manages pairs consisting of a tk variable and a callback function.
When tracing is turned on, setting the variable calls the function.  Test
coverage for the new class is 100%.

@item 
bpo-31003@footnote{https://bugs.python.org/issue31003}: IDLE: Add more tests for General tab.

@item 
bpo-30993@footnote{https://bugs.python.org/issue30993}: IDLE - Improve configdialog font page and tests. In
configdialog: Document causal pathways in create_font_tab docstring.
Simplify some attribute names. Move set_samples calls to var_changed_font
(idea from Cheryl Sabella).  Move related functions to positions after the
create widgets function. In test_configdialog: Fix test_font_set so not
order dependent.  Fix renamed test_indent_scale so it tests the widget.
Adjust tests for movement of set_samples call.  Add tests for load
functions.  Put all font tests in one class and tab indent tests in
another.  Except for two lines, these tests completely cover the related
functions.

@item 
bpo-30981@footnote{https://bugs.python.org/issue30981}: IDLE – Add more configdialog font page tests.

@item 
bpo-28523@footnote{https://bugs.python.org/issue28523}: IDLE: replace ‘colour’ with ‘color’ in configdialog.

@item 
bpo-30917@footnote{https://bugs.python.org/issue30917}: Add tests for idlelib.config.IdleConf. Increase coverage from
46% to 96%. Patch by Louie Lu.

@item 
bpo-30934@footnote{https://bugs.python.org/issue30934}: Document coverage details for idlelib tests. Add section to
idlelib/idle-test/README.txt. Include check that branches are taken both
ways. Exclude IDLE-specific code that does not run during unit tests.

@item 
bpo-30913@footnote{https://bugs.python.org/issue30913}: IDLE: Document ConfigDialog tk Vars, methods, and widgets in
docstrings This will facilitate improving the dialog and splitting up the
class. Original patch by Cheryl Sabella.

@item 
bpo-30899@footnote{https://bugs.python.org/issue30899}: IDLE: Add tests for ConfigParser subclasses in config. Patch by
Louie Lu.

@item 
bpo-30881@footnote{https://bugs.python.org/issue30881}: IDLE: Add docstrings to browser.py. Patch by Cheryl Sabella.

@item 
bpo-30851@footnote{https://bugs.python.org/issue30851}: IDLE: Remove unused variables in configdialog.  One is a
duplicate, one is set but cannot be altered by users. Patch by Cheryl
Sabella.

@item 
bpo-30870@footnote{https://bugs.python.org/issue30870}: IDLE: In Settings dialog, select font with Up, Down keys as
well as mouse. Initial patch by Louie Lu.

@item 
bpo-8231@footnote{https://bugs.python.org/issue8231}: IDLE: call config.IdleConf.GetUserCfgDir only once.

@item 
bpo-30779@footnote{https://bugs.python.org/issue30779}: IDLE: Factor ConfigChanges class from configdialog, put in
config; test. * In config, put dump test code in a function; run it and
unittest in   ‘if __name__ == ‘__main__’. * Add class config.ConfigChanges
based on changes_class_v4.py on bpo issue. * Add class
test_config.ChangesTest, partly using configdialog_tests_v1.py. * Revise
configdialog to use ConfigChanges; see tracker msg297804. * Revise
test_configdialog to match configdialog changes. * Remove configdialog
functions unused or moved to ConfigChanges. Cheryl Sabella contributed
parts of the patch.

@item 
bpo-30777@footnote{https://bugs.python.org/issue30777}: IDLE: configdialog - Add docstrings and fix comments. Patch by
Cheryl Sabella.

@item 
bpo-30495@footnote{https://bugs.python.org/issue30495}: IDLE: Improve textview with docstrings, PEP8 names, and more
tests. Patch by Cheryl Sabella.

@item 
bpo-30723@footnote{https://bugs.python.org/issue30723}: IDLE: Make several improvements to parenmatch. Add ‘parens’
style to highlight both opener and closer. Make ‘default’ style, which is
not default, a synonym for ‘opener’. Make time-delay work the same with
all styles. Add help for config dialog extensions tab, including help for
parenmatch. Add new tests.  Original patch by Charles Wohlganger.

@item 
bpo-30674@footnote{https://bugs.python.org/issue30674}: IDLE: add docstrings to grep module. Patch by Cheryl Sabella

@item 
bpo-21519@footnote{https://bugs.python.org/issue21519}: IDLE’s basic custom key entry dialog now detects duplicates
properly. Original patch by Saimadhav Heblikar.

@item 
bpo-29910@footnote{https://bugs.python.org/issue29910}: IDLE no longer deletes a character after commenting out a
region by a key shortcut.  Add @code{return 'break'} for this and other
potential conflicts between IDLE and default key bindings.

@item 
bpo-30728@footnote{https://bugs.python.org/issue30728}: Review and change idlelib.configdialog names. Lowercase method
and attribute names. Replace ‘colour’ with ‘color’, expand overly cryptic
names, delete unneeded underscores. Replace @code{import *} with specific
imports. Patches by Cheryl Sabella.

@item 
bpo-6739@footnote{https://bugs.python.org/issue6739}: IDLE: Verify user-entered key sequences by trying to bind them
with tk. Add tests for all 3 validation functions. Original patch by G
Polo.  Tests added by Cheryl Sabella.

@item 
bpo-15786@footnote{https://bugs.python.org/issue15786}: Fix several problems with IDLE’s autocompletion box. The
following should now work: clicking on selection box items; using the
scrollbar; selecting an item by hitting Return. Hangs on MacOSX should no
longer happen. Patch by Louie Lu.

@item 
bpo-25514@footnote{https://bugs.python.org/issue25514}: Add doc subsubsection about IDLE failure to start. Popup
no-connection message directs users to this section.

@item 
bpo-30642@footnote{https://bugs.python.org/issue30642}: Fix reference leaks in IDLE tests. Patches by Louie Lu and
Terry Jan Reedy.

@item 
bpo-30495@footnote{https://bugs.python.org/issue30495}: Add docstrings for textview.py and use PEP8 names. Patches by
Cheryl Sabella and Terry Jan Reedy.

@item 
bpo-30290@footnote{https://bugs.python.org/issue30290}: Help-about: use pep8 names and add tests. Increase coverage to
100%. Patches by Louie Lu, Cheryl Sabella, and Terry Jan Reedy.

@item 
bpo-30303@footnote{https://bugs.python.org/issue30303}: Add _utest option to textview; add new tests. Increase coverage
to 100%. Patches by Louie Lu and Terry Jan Reedy.

@item 
bpo-29071@footnote{https://bugs.python.org/issue29071}: IDLE colors f-string prefixes (but not invalid ur prefixes).

@item 
bpo-28572@footnote{https://bugs.python.org/issue28572}: Add 10% to coverage of IDLE’s test_configdialog. Update and
augment description of the configuration system.
@end itemize

@node Tools/Demos<17>,C API<32>,IDLE<32>,Python 3 7 0 alpha 1
@anchor{whatsnew/changelog id323}@anchor{1384}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-30983@footnote{https://bugs.python.org/issue30983}: gdb integration commands (py-bt, etc.) work on optimized shared
builds now, too.  
@geindex Python Enhancement Proposals; PEP 523
PEP 523@footnote{https://www.python.org/dev/peps/pep-0523} introduced _PyEval_EvalFrameDefault which
inlines PyEval_EvalFrameEx on non-debug shared builds.  This broke the
ability to use py-bt, py-up, and a few other Python-specific gdb
integrations. The problem is fixed by only looking for
_PyEval_EvalFrameDefault frames in python-gdb.py.  Original patch by Bruno
“Polaco” Penteado.

@item 
bpo-29748@footnote{https://bugs.python.org/issue29748}: Added the slice index converter in Argument Clinic.

@item 
bpo-24037@footnote{https://bugs.python.org/issue24037}: Argument Clinic now uses the converter @code{bool(accept=@{int@})}
rather than @ref{1b1,,int} for semantical booleans.  This avoids repeating the
default value for Python and C and will help in converting to @ref{2f7,,bool} in
future.

@item 
bpo-29367@footnote{https://bugs.python.org/issue29367}: python-gdb.py now supports also @code{method-wrapper}
(@code{wrapperobject}) objects.

@item 
bpo-28023@footnote{https://bugs.python.org/issue28023}: Fix python-gdb.py didn’t support new dict implementation.

@item 
bpo-15369@footnote{https://bugs.python.org/issue15369}: The pybench and pystone microbenchmark have been removed from
Tools. Please use the new Python benchmark suite
@indicateurl{https://github.com/python/performance} which is more reliable and includes
a portable version of pybench working on Python 2 and Python 3.

@item 
bpo-28102@footnote{https://bugs.python.org/issue28102}: The zipfile module CLI now prints usage to stderr. Patch by
Stephen J. Turnbull.
@end itemize

@node C API<32>,,Tools/Demos<17>,Python 3 7 0 alpha 1
@anchor{whatsnew/changelog id324}@anchor{1385}
@subsubsection C API



@itemize -

@item 
bpo-31338@footnote{https://bugs.python.org/issue31338}: Added the @code{Py_UNREACHABLE()} macro for code paths which are
never expected to be reached.  This and a few other useful macros are now
documented in the C API manual.

@item 
bpo-30832@footnote{https://bugs.python.org/issue30832}: Remove own implementation for thread-local storage. CPython has
provided the own implementation for thread-local storage (TLS) on
Python/thread.c, it’s used in the case which a platform has not supplied
native TLS.  However, currently all supported platforms (Windows and
pthreads) have provided native TLS and defined the Py_HAVE_NATIVE_TLS
macro with unconditional in any case.

@item 
bpo-30708@footnote{https://bugs.python.org/issue30708}: PyUnicode_AsWideCharString() now raises a ValueError if the
second argument is NULL and the wchar_t* string contains null characters.

@item 
bpo-16500@footnote{https://bugs.python.org/issue16500}: Deprecate PyOS_AfterFork() and add PyOS_BeforeFork(),
PyOS_AfterFork_Parent() and PyOS_AfterFork_Child().

@item 
bpo-6532@footnote{https://bugs.python.org/issue6532}: The type of results of PyThread_start_new_thread() and
PyThread_get_thread_ident(), and the id parameter of
PyThreadState_SetAsyncExc() changed from “long” to “unsigned long”.

@item 
bpo-27867@footnote{https://bugs.python.org/issue27867}: Function PySlice_GetIndicesEx() is deprecated and replaced with
a macro if Py_LIMITED_API is not set or set to the value between
0x03050400 and 0x03060000 (not including) or 0x03060100 or higher.  Added
functions PySlice_Unpack() and PySlice_AdjustIndices().

@item 
bpo-29083@footnote{https://bugs.python.org/issue29083}: Fixed the declaration of some public API functions.
PyArg_VaParse() and PyArg_VaParseTupleAndKeywords() were not available in
limited API. PyArg_ValidateKeywordArguments(), PyArg_UnpackTuple() and
Py_BuildValue() were not available in limited API of version < 3.3 when
PY_SSIZE_T_CLEAN is defined.

@item 
bpo-28769@footnote{https://bugs.python.org/issue28769}: The result of PyUnicode_AsUTF8AndSize() and PyUnicode_AsUTF8()
is now of type @code{const char *} rather of @code{char *}.

@item 
bpo-29058@footnote{https://bugs.python.org/issue29058}: All stable API extensions added after Python 3.2 are now
available only when Py_LIMITED_API is set to the PY_VERSION_HEX value of
the minimum Python version supporting this API.

@item 
bpo-28822@footnote{https://bugs.python.org/issue28822}: The index parameters @emph{start} and @emph{end} of PyUnicode_FindChar()
are now adjusted to behave like @code{str[start:end]}.

@item 
bpo-28808@footnote{https://bugs.python.org/issue28808}: PyUnicode_CompareWithASCIIString() now never raises exceptions.

@item 
bpo-28761@footnote{https://bugs.python.org/issue28761}: The fields name and doc of structures PyMemberDef, PyGetSetDef,
PyStructSequence_Field, PyStructSequence_Desc, and wrapperbase are now of
type @code{const char *} rather of @code{char *}.

@item 
bpo-28748@footnote{https://bugs.python.org/issue28748}: Private variable _Py_PackageContext is now of type @code{const char
*} rather of @code{char *}.

@item 
bpo-19569@footnote{https://bugs.python.org/issue19569}: Compiler warnings are now emitted if use most of deprecated
functions.

@item 
bpo-28426@footnote{https://bugs.python.org/issue28426}: Deprecated undocumented functions PyUnicode_AsEncodedObject(),
PyUnicode_AsDecodedObject(), PyUnicode_AsDecodedUnicode() and
PyUnicode_AsEncodedUnicode().
@end itemize

@node Python 3 6 6 final,Python 3 6 6 release candidate 1,Python 3 7 0 alpha 1,Changelog
@anchor{whatsnew/changelog python-3-6-6-final}@anchor{1386}
@subsection Python 3.6.6 final


@emph{Release date: 2018-06-27}

There were no new changes in version 3.6.6.

@node Python 3 6 6 release candidate 1,Python 3 6 5 final,Python 3 6 6 final,Changelog
@anchor{whatsnew/changelog python-3-6-6-release-candidate-1}@anchor{1387}
@subsection Python 3.6.6 release candidate 1


@emph{Release date: 2018-06-11}

@menu
* Core and Builtins: Core and Builtins<37>. 
* Library: Library<38>. 
* Documentation: Documentation<38>. 
* Tests: Tests<34>. 
* Build: Build<35>. 
* Windows: Windows<33>. 
* macOS: macOS<27>. 
* IDLE: IDLE<33>. 
* Tools/Demos: Tools/Demos<18>. 
* C API: C API<33>. 

@end menu

@node Core and Builtins<37>,Library<38>,,Python 3 6 6 release candidate 1
@anchor{whatsnew/changelog id325}@anchor{1388}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-33786@footnote{https://bugs.python.org/issue33786}: Fix asynchronous generators to handle GeneratorExit in athrow()
correctly

@item 
bpo-30654@footnote{https://bugs.python.org/issue30654}: Fixed reset of the SIGINT handler to SIG_DFL on interpreter
shutdown even when there was a custom handler set previously. Patch by
Philipp Kerling.

@item 
bpo-33622@footnote{https://bugs.python.org/issue33622}: Fixed a leak when the garbage collector fails to add an object
with the @code{__del__} method or referenced by it into the
@ref{d66,,gc.garbage} list. @ref{49a,,PyGC_Collect()} can now be called when an
exception is set and preserves it.

@item 
bpo-31849@footnote{https://bugs.python.org/issue31849}: Fix signed/unsigned comparison warning in pyhash.c.

@item 
bpo-33391@footnote{https://bugs.python.org/issue33391}: Fix a leak in set_symmetric_difference().

@item 
bpo-28055@footnote{https://bugs.python.org/issue28055}: Fix unaligned accesses in siphash24(). Patch by Rolf Eike Beer.

@item 
bpo-33231@footnote{https://bugs.python.org/issue33231}: Fix potential memory leak in @code{normalizestring()}.

@item 
bpo-29922@footnote{https://bugs.python.org/issue29922}: Improved error messages in ‘async with’ when @code{__aenter__()}
or @code{__aexit__()} return non-awaitable object.

@item 
bpo-33199@footnote{https://bugs.python.org/issue33199}: Fix @code{ma_version_tag} in dict implementation is uninitialized
when copying from key-sharing dict.

@item 
bpo-33041@footnote{https://bugs.python.org/issue33041}: Fixed jumping when the function contains an @code{async for} loop.

@item 
bpo-32282@footnote{https://bugs.python.org/issue32282}: Fix an unnecessary ifdef in the include of VersionHelpers.h in
socketmodule on Windows.

@item 
bpo-21983@footnote{https://bugs.python.org/issue21983}: Fix a crash in @ref{12db,,ctypes.cast()} in case the type argument is a
ctypes structured data type. Patch by Eryk Sun and Oren Milman.
@end itemize

@node Library<38>,Documentation<38>,Core and Builtins<37>,Python 3 6 6 release candidate 1
@anchor{whatsnew/changelog id326}@anchor{1389}
@subsubsection Library



@itemize -

@item 
bpo-30167@footnote{https://bugs.python.org/issue30167}: Prevent site.main() exception if PYTHONSTARTUP is set. Patch by
Steve Weber.

@item 
bpo-33812@footnote{https://bugs.python.org/issue33812}: Datetime instance d with non-None tzinfo, but with
d.tzinfo.utcoffset(d) returning None is now treated as naive by the
astimezone() method.

@item 
bpo-30805@footnote{https://bugs.python.org/issue30805}: Avoid race condition with debug logging

@item 
bpo-33767@footnote{https://bugs.python.org/issue33767}: The concatenation (@code{+}) and repetition (@code{*}) sequence
operations now raise @ref{162,,TypeError} instead of @ref{891,,SystemError} when
performed on @ref{511,,mmap.mmap} objects.  Patch by Zackery Spytz.

@item 
bpo-32684@footnote{https://bugs.python.org/issue32684}: Fix gather to propagate cancellation of itself even with
return_exceptions.

@item 
bpo-33674@footnote{https://bugs.python.org/issue33674}: Fix a race condition in SSLProtocol.connection_made() of
asyncio.sslproto: start immediately the handshake instead of using
call_soon(). Previously, data_received() could be called before the
handshake started, causing the handshake to hang or fail.

@item 
bpo-31647@footnote{https://bugs.python.org/issue31647}: Fixed bug where calling write_eof() on a
_SelectorSocketTransport after it’s already closed raises AttributeError.

@item 
bpo-33672@footnote{https://bugs.python.org/issue33672}: Fix Task.__repr__ crash with Cython’s bogus coroutines

@item 
bpo-33469@footnote{https://bugs.python.org/issue33469}: Fix RuntimeError after closing loop that used run_in_executor

@item 
bpo-11874@footnote{https://bugs.python.org/issue11874}: Use a better regex when breaking usage into wrappable parts.
Avoids bogus assertion errors from custom metavar strings.

@item 
bpo-30877@footnote{https://bugs.python.org/issue30877}: Fixed a bug in the Python implementation of the JSON decoder
that prevented the cache of parsed strings from clearing after finishing
the decoding. Based on patch by c-fos.

@item 
bpo-33548@footnote{https://bugs.python.org/issue33548}: tempfile._candidate_tempdir_list should consider common TEMP
locations

@item 
bpo-33542@footnote{https://bugs.python.org/issue33542}: Prevent @code{uuid.get_node} from using a DUID instead of a MAC on
Windows. Patch by Zvi Effron

@item 
bpo-26819@footnote{https://bugs.python.org/issue26819}: Fix race condition with @code{ReadTransport.resume_reading} in
Windows proactor event loop.

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Minor fixes in typing module: add annotations to
@code{NamedTuple.__new__}, pass @code{*args} and @code{**kwds} in
@code{Generic.__new__}.  Original PRs by Paulius Šarka and Chad Dombrova.

@item 
bpo-20087@footnote{https://bugs.python.org/issue20087}: Updated alias mapping with glibc 2.27 supported locales.

@item 
bpo-33422@footnote{https://bugs.python.org/issue33422}: Fix trailing quotation marks getting deleted when looking up
byte/string literals on pydoc. Patch by Andrés Delfino.

@item 
bpo-33197@footnote{https://bugs.python.org/issue33197}: Update error message when constructing invalid
inspect.Parameters Patch by Dong-hee Na.

@item 
bpo-33383@footnote{https://bugs.python.org/issue33383}: Fixed crash in the get() method of the @ref{33,,dbm.ndbm} database
object when it is called with a single argument.

@item 
bpo-33329@footnote{https://bugs.python.org/issue33329}: Fix multiprocessing regression on newer glibcs

@item 
bpo-991266@footnote{https://bugs.python.org/issue991266}: Fix quoting of the @code{Comment} attribute of
@ref{12fd,,http.cookies.SimpleCookie}.

@item 
bpo-33131@footnote{https://bugs.python.org/issue33131}: Upgrade bundled version of pip to 10.0.1.

@item 
bpo-33308@footnote{https://bugs.python.org/issue33308}: Fixed a crash in the @code{parser} module when converting an ST
object to a tree of tuples or lists with @code{line_info=False} and
@code{col_info=True}.

@item 
bpo-33263@footnote{https://bugs.python.org/issue33263}: Fix FD leak in @code{_SelectorSocketTransport}  Patch by Vlad
Starostin.

@item 
bpo-33256@footnote{https://bugs.python.org/issue33256}: Fix display of @code{<module>} call in the html produced by
@code{cgitb.html()}. Patch by Stéphane Blondon.

@item 
bpo-33203@footnote{https://bugs.python.org/issue33203}: @code{random.Random.choice()} now raises @code{IndexError} for empty
sequences consistently even when called from subclasses without a
@code{getrandbits()} implementation.

@item 
bpo-33224@footnote{https://bugs.python.org/issue33224}: Update difflib.mdiff() for 
@geindex Python Enhancement Proposals; PEP 479
PEP 479@footnote{https://www.python.org/dev/peps/pep-0479}.  Convert an uncaught
StopIteration in a generator into a return-statement.

@item 
bpo-33209@footnote{https://bugs.python.org/issue33209}: End framing at the end of C implementation of
@ref{1300,,pickle.Pickler.dump()}.

@item 
bpo-32861@footnote{https://bugs.python.org/issue32861}: The urllib.robotparser’s @code{__str__} representation now
includes wildcard entries and the “Crawl-delay” and “Request-rate” fields.
Patch by Michael Lazar.

@item 
bpo-33096@footnote{https://bugs.python.org/issue33096}: Allow ttk.Treeview.insert to insert iid that has a false
boolean value. Note iid=0 and iid=False would be same. Patch by Garvit
Khatri.

@item 
bpo-33127@footnote{https://bugs.python.org/issue33127}: The ssl module now compiles with LibreSSL 2.7.1.

@item 
bpo-33021@footnote{https://bugs.python.org/issue33021}: Release the GIL during fstat() calls, avoiding hang of all
threads when calling mmap.mmap(), os.urandom(), and random.seed().  Patch
by Nir Soffer.

@item 
bpo-27683@footnote{https://bugs.python.org/issue27683}: Fix a regression in @ref{9e,,ipaddress} that result of
@code{hosts()} is empty when the network is constructed by a tuple
containing an integer mask and only 1 bit left for addresses.

@item 
bpo-32844@footnote{https://bugs.python.org/issue32844}: Fix wrong redirection of a low descriptor (0 or 1) to stderr in
subprocess if another low descriptor is closed.

@item 
bpo-31908@footnote{https://bugs.python.org/issue31908}: Fix output of cover files for @code{trace} module command-line
tool. Previously emitted cover files only when @code{--missing} option was
used. Patch by Michael Selik.

@item 
bpo-31457@footnote{https://bugs.python.org/issue31457}: If nested log adapters are used, the inner @code{process()}
methods are no longer omitted.

@item 
bpo-16865@footnote{https://bugs.python.org/issue16865}: Support arrays >=2GiB in @ref{29,,ctypes}.  Patch by Segev Finer.

@item 
bpo-31238@footnote{https://bugs.python.org/issue31238}: pydoc: the stop() method of the private ServerThread class now
waits until DocServer.serve_until_quit() completes and then explicitly
sets its docserver attribute to None to break a reference cycle.
@end itemize

@node Documentation<38>,Tests<34>,Library<38>,Python 3 6 6 release candidate 1
@anchor{whatsnew/changelog id327}@anchor{138a}
@subsubsection Documentation



@itemize -

@item 
bpo-33503@footnote{https://bugs.python.org/issue33503}: Fix broken pypi link

@item 
bpo-33421@footnote{https://bugs.python.org/issue33421}: Add missing documentation for @code{typing.AsyncContextManager}.

@item 
bpo-33378@footnote{https://bugs.python.org/issue33378}: Add Korean language switcher for @indicateurl{https://docs.python.org/3/}

@item 
bpo-33276@footnote{https://bugs.python.org/issue33276}: Clarify that the @code{__path__} attribute on modules cannot be
just any value.

@item 
bpo-33201@footnote{https://bugs.python.org/issue33201}: Modernize documentation for writing C extension types.

@item 
bpo-33195@footnote{https://bugs.python.org/issue33195}: Deprecate @code{Py_UNICODE} usage in @code{c-api/arg} document.
@code{Py_UNICODE} related APIs are deprecated since Python 3.3, but it is
missed in the document.

@item 
bpo-33126@footnote{https://bugs.python.org/issue33126}: Document PyBuffer_ToContiguous().

@item 
bpo-27212@footnote{https://bugs.python.org/issue27212}: Modify documentation for the @code{islice()} recipe to consume
initial values up to the start index.

@item 
bpo-28247@footnote{https://bugs.python.org/issue28247}: Update @ref{148,,zipapp} documentation to describe how to make
standalone applications.

@item 
bpo-18802@footnote{https://bugs.python.org/issue18802}: Documentation changes for ipaddress.  Patch by Jon Foster and
Berker Peksag.

@item 
bpo-27428@footnote{https://bugs.python.org/issue27428}: Update documentation to clarify that @code{WindowsRegistryFinder}
implements @code{MetaPathFinder}. (Patch by Himanshu Lakhara)

@item 
bpo-8243@footnote{https://bugs.python.org/issue8243}: Add a note about curses.addch and curses.addstr exception
behavior when writing outside a window, or pad.

@item 
bpo-31432@footnote{https://bugs.python.org/issue31432}: Clarify meaning of CERT_NONE, CERT_OPTIONAL, and CERT_REQUIRED
flags for ssl.SSLContext.verify_mode.
@end itemize

@node Tests<34>,Build<35>,Documentation<38>,Python 3 6 6 release candidate 1
@anchor{whatsnew/changelog id328}@anchor{138b}
@subsubsection Tests



@itemize -

@item 
bpo-33655@footnote{https://bugs.python.org/issue33655}: Ignore test_posix_fallocate failures on BSD platforms that
might be due to running on ZFS.

@item 
bpo-19417@footnote{https://bugs.python.org/issue19417}: Add test_bdb.py.
@end itemize

@node Build<35>,Windows<33>,Tests<34>,Python 3 6 6 release candidate 1
@anchor{whatsnew/changelog id329}@anchor{138c}
@subsubsection Build



@itemize -

@item 
bpo-5755@footnote{https://bugs.python.org/issue5755}: Move @code{-Wstrict-prototypes} option to @code{CFLAGS_NODIST} from
@code{OPT}. This option emitted annoying warnings when building extension
modules written in C++.

@item 
bpo-33614@footnote{https://bugs.python.org/issue33614}: Ensures module definition files for the stable ABI on Windows
are correctly regenerated.

@item 
bpo-33522@footnote{https://bugs.python.org/issue33522}: Enable CI builds on Visual Studio Team Services at
@indicateurl{https://python.visualstudio.com/cpython}

@item 
bpo-33012@footnote{https://bugs.python.org/issue33012}: Add @code{-Wno-cast-function-type} for gcc 8 for silencing
warnings about function casts like casting to PyCFunction in method
definition lists.

@item 
bpo-33394@footnote{https://bugs.python.org/issue33394}: Enable the verbose build for extension modules, when GNU make
is passed macros on the command line.
@end itemize

@node Windows<33>,macOS<27>,Build<35>,Python 3 6 6 release candidate 1
@anchor{whatsnew/changelog id330}@anchor{138d}
@subsubsection Windows



@itemize -

@item 
bpo-33184@footnote{https://bugs.python.org/issue33184}: Update Windows installer to OpenSSL 1.0.2o.
@end itemize

@node macOS<27>,IDLE<33>,Windows<33>,Python 3 6 6 release candidate 1
@anchor{whatsnew/changelog id331}@anchor{138e}
@subsubsection macOS



@itemize -

@item 
bpo-33184@footnote{https://bugs.python.org/issue33184}: Update macOS installer build to use OpenSSL 1.0.2o.
@end itemize

@node IDLE<33>,Tools/Demos<18>,macOS<27>,Python 3 6 6 release candidate 1
@anchor{whatsnew/changelog id332}@anchor{138f}
@subsubsection IDLE



@itemize -

@item 
bpo-33656@footnote{https://bugs.python.org/issue33656}: On Windows, add API call saying that tk scales for DPI. On
Windows 8.1+ or 10, with DPI compatibility properties of the Python binary
unchanged, and a monitor resolution greater than 96 DPI, this should make
text and lines sharper.  It should otherwise have no effect.

@item 
bpo-33768@footnote{https://bugs.python.org/issue33768}: Clicking on a context line moves that line to the top of the
editor window.

@item 
bpo-33763@footnote{https://bugs.python.org/issue33763}: IDLE: Use read-only text widget for code context instead of
label widget.

@item 
bpo-33664@footnote{https://bugs.python.org/issue33664}: Scroll IDLE editor text by lines. Previously, the mouse wheel
and scrollbar slider moved text by a fixed number of pixels, resulting in
partial lines at the top of the editor box.  The change also applies to
the shell and grep output windows, but not to read-only text views.

@item 
bpo-33679@footnote{https://bugs.python.org/issue33679}: Enable theme-specific color configuration for Code Context. Use
the Highlights tab to see the setting for built-in themes or add settings
to custom themes.

@item 
bpo-33642@footnote{https://bugs.python.org/issue33642}: Display up to maxlines non-blank lines for Code Context. If
there is no current context, show a single blank line.

@item 
bpo-33628@footnote{https://bugs.python.org/issue33628}: IDLE: Cleanup codecontext.py and its test.

@item 
bpo-33564@footnote{https://bugs.python.org/issue33564}: IDLE’s code context now recognizes async as a block opener.

@item 
bpo-29706@footnote{https://bugs.python.org/issue29706}: IDLE now colors async and await as keywords in 3.6. They become
full keywords in 3.7.

@item 
bpo-21474@footnote{https://bugs.python.org/issue21474}: Update word/identifier definition from ascii to unicode. In
text and entry boxes, this affects selection by double-click, movement
left/right by control-left/right, and deletion left/right by
control-BACKSPACE/DEL.

@item 
bpo-33204@footnote{https://bugs.python.org/issue33204}: IDLE: consistently color invalid string prefixes. A ‘u’ string
prefix cannot be paired with either ‘r’ or ‘f’. Consistently color as much
of the prefix, starting at the right, as is valid. Revise and extend
colorizer test.

@item 
bpo-32831@footnote{https://bugs.python.org/issue32831}: Add docstrings and tests for codecontext.
@end itemize

@node Tools/Demos<18>,C API<33>,IDLE<33>,Python 3 6 6 release candidate 1
@anchor{whatsnew/changelog id333}@anchor{1390}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-33189@footnote{https://bugs.python.org/issue33189}: @code{pygettext.py} now recognizes only literal strings as
docstrings and translatable strings, and rejects bytes literals and
f-string expressions.

@item 
bpo-31920@footnote{https://bugs.python.org/issue31920}: Fixed handling directories as arguments in the @code{pygettext}
script. Based on patch by Oleg Krasnikov.

@item 
bpo-29673@footnote{https://bugs.python.org/issue29673}: Fix pystackv and pystack gdbinit macros.

@item 
bpo-32885@footnote{https://bugs.python.org/issue32885}: Add an @code{-n} flag for @code{Tools/scripts/pathfix.py} to disable
automatic backup creation (files with @code{~} suffix).

@item 
bpo-31583@footnote{https://bugs.python.org/issue31583}: Fix 2to3 for using with –add-suffix option but without
–output-dir option for relative path to files in current directory.
@end itemize

@node C API<33>,,Tools/Demos<18>,Python 3 6 6 release candidate 1
@anchor{whatsnew/changelog id334}@anchor{1391}
@subsubsection C API



@itemize -

@item 
bpo-32374@footnote{https://bugs.python.org/issue32374}: Document that m_traverse for multi-phase initialized modules
can be called with m_state=NULL, and add a sanity check
@end itemize

@node Python 3 6 5 final,Python 3 6 5 release candidate 1,Python 3 6 6 release candidate 1,Changelog
@anchor{whatsnew/changelog python-3-6-5-final}@anchor{1392}
@subsection Python 3.6.5 final


@emph{Release date: 2018-03-28}

@menu
* Tests: Tests<35>. 
* Build: Build<36>. 

@end menu

@node Tests<35>,Build<36>,,Python 3 6 5 final
@anchor{whatsnew/changelog id335}@anchor{1393}
@subsubsection Tests



@itemize -

@item 
bpo-32872@footnote{https://bugs.python.org/issue32872}: Avoid regrtest compatibility issue with namespace packages.
@end itemize

@node Build<36>,,Tests<35>,Python 3 6 5 final
@anchor{whatsnew/changelog id336}@anchor{1394}
@subsubsection Build



@itemize -

@item 
bpo-33163@footnote{https://bugs.python.org/issue33163}: Upgrade pip to 9.0.3 and setuptools to v39.0.1.
@end itemize

@node Python 3 6 5 release candidate 1,Python 3 6 4 final,Python 3 6 5 final,Changelog
@anchor{whatsnew/changelog python-3-6-5-release-candidate-1}@anchor{1395}
@subsection Python 3.6.5 release candidate 1


@emph{Release date: 2018-03-13}

@menu
* Security: Security<22>. 
* Core and Builtins: Core and Builtins<38>. 
* Library: Library<39>. 
* Documentation: Documentation<39>. 
* Tests: Tests<36>. 
* Build: Build<37>. 
* Windows: Windows<34>. 
* macOS: macOS<28>. 
* IDLE: IDLE<34>. 
* Tools/Demos: Tools/Demos<19>. 
* C API: C API<34>. 

@end menu

@node Security<22>,Core and Builtins<38>,,Python 3 6 5 release candidate 1
@anchor{whatsnew/changelog id337}@anchor{1396}
@subsubsection Security



@itemize -

@item 
bpo-33001@footnote{https://bugs.python.org/issue33001}: Minimal fix to prevent buffer overrun in os.symlink on Windows

@item 
bpo-32981@footnote{https://bugs.python.org/issue32981}: Regexes in difflib and poplib were vulnerable to catastrophic
backtracking. These regexes formed potential DOS vectors (REDOS). They
have been refactored. This resolves CVE-2018-1060 and CVE-2018-1061. Patch
by Jamie Davis.
@end itemize

@node Core and Builtins<38>,Library<39>,Security<22>,Python 3 6 5 release candidate 1
@anchor{whatsnew/changelog id338}@anchor{1397}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-33026@footnote{https://bugs.python.org/issue33026}: Fixed jumping out of “with” block by setting f_lineno.

@item 
bpo-17288@footnote{https://bugs.python.org/issue17288}: Prevent jumps from ‘return’ and ‘exception’ trace events.

@item 
bpo-32889@footnote{https://bugs.python.org/issue32889}: Update Valgrind suppression list to account for the rename of
@code{Py_ADDRESS_IN_RANG} to @code{address_in_range}.

@item 
bpo-32650@footnote{https://bugs.python.org/issue32650}: Pdb and other debuggers dependent on bdb.py will correctly step
over (next command) native coroutines. Patch by Pablo Galindo.

@item 
bpo-32685@footnote{https://bugs.python.org/issue32685}: Improve suggestion when the Python 2 form of print statement is
either present on the same line as the header of a compound statement or
else terminated by a semi-colon instead of a newline. Patch by Nitish
Chandra.

@item 
bpo-32583@footnote{https://bugs.python.org/issue32583}: Fix possible crashing in builtin Unicode decoders caused by
write out-of-bound errors when using customized decode error handlers.

@item 
bpo-26163@footnote{https://bugs.python.org/issue26163}: Improved frozenset() hash to create more distinct hash values
when faced with datasets containing many similar values.

@item 
bpo-27169@footnote{https://bugs.python.org/issue27169}: The @code{__debug__} constant is now optimized out at compile
time. This fixes also bpo-22091@footnote{https://bugs.python.org/issue22091}.

@item 
bpo-32329@footnote{https://bugs.python.org/issue32329}: @code{sys.flags.hash_randomization} is now properly set to 0 when
hash randomization is turned off by @code{PYTHONHASHSEED=0}.

@item 
bpo-30416@footnote{https://bugs.python.org/issue30416}: The optimizer is now protected from spending much time doing
complex calculations and consuming much memory for creating large
constants in constant folding.

@item 
bpo-18533@footnote{https://bugs.python.org/issue18533}: @code{repr()} on a dict containing its own @code{values()} or
@code{items()} no longer raises @code{RecursionError}; OrderedDict similarly.
Instead, use @code{...}, as for other recursive structures.  Patch by Ben
North.

@item 
bpo-32028@footnote{https://bugs.python.org/issue32028}: Leading whitespace is now correctly ignored when generating
suggestions for converting Py2 print statements to Py3 builtin print
function calls. Patch by Sanyam Khurana.

@item 
bpo-32137@footnote{https://bugs.python.org/issue32137}: The repr of deeply nested dict now raises a RecursionError
instead of crashing due to a stack overflow.
@end itemize

@node Library<39>,Documentation<39>,Core and Builtins<38>,Python 3 6 5 release candidate 1
@anchor{whatsnew/changelog id339}@anchor{1398}
@subsubsection Library



@itemize -

@item 
bpo-33064@footnote{https://bugs.python.org/issue33064}: lib2to3 now properly supports trailing commas after @code{*args}
and @code{**kwargs} in function signatures.

@item 
bpo-31804@footnote{https://bugs.python.org/issue31804}: Avoid failing in multiprocessing.Process if the standard
streams are closed or None at exit.

@item 
bpo-33037@footnote{https://bugs.python.org/issue33037}: Skip sending/receiving data after SSL transport closing.

@item 
bpo-30353@footnote{https://bugs.python.org/issue30353}: Fix ctypes pass-by-value for structs on 64-bit Cygwin/MinGW.

@item 
bpo-33009@footnote{https://bugs.python.org/issue33009}: Fix inspect.signature() for single-parameter partialmethods.

@item 
bpo-32969@footnote{https://bugs.python.org/issue32969}: Expose several missing constants in zlib and fix corresponding
documentation.

@item 
bpo-32713@footnote{https://bugs.python.org/issue32713}: Fixed tarfile.itn handling of out-of-bounds float values. Patch
by Joffrey Fuhrer.

@item 
bpo-30622@footnote{https://bugs.python.org/issue30622}: The ssl module now detects missing NPN support in LibreSSL.

@item 
bpo-32922@footnote{https://bugs.python.org/issue32922}: dbm.open() now encodes filename with the filesystem encoding
rather than default encoding.

@item 
bpo-32859@footnote{https://bugs.python.org/issue32859}: In @code{os.dup2}, don’t check every call whether the @code{dup3}
syscall exists or not.

@item 
bpo-21060@footnote{https://bugs.python.org/issue21060}: Rewrite confusing message from setup.py upload from “No dist
file created in earlier command” to the more helpful “Must create and
upload files in one command”.

@item 
bpo-32857@footnote{https://bugs.python.org/issue32857}: In @ref{10c,,tkinter}, @code{after_cancel(None)} now raises a
@ref{17c,,ValueError} instead of canceling the first scheduled function.
Patch by Cheryl Sabella.

@item 
bpo-32852@footnote{https://bugs.python.org/issue32852}: Make sure sys.argv remains as a list when running trace.

@item 
bpo-32841@footnote{https://bugs.python.org/issue32841}: Fixed @ref{58d,,asyncio.Condition} issue which silently ignored
cancellation after notifying and cancelling a conditional lock. Patch by
Bar Harel.

@item 
bpo-31787@footnote{https://bugs.python.org/issue31787}: Fixed refleaks of @code{__init__()} methods in various modules.
(Contributed by Oren Milman)

@item 
bpo-30157@footnote{https://bugs.python.org/issue30157}: Fixed guessing quote and delimiter in csv.Sniffer.sniff() when
only the last field is quoted.  Patch by Jake Davis.

@item 
bpo-32394@footnote{https://bugs.python.org/issue32394}: socket: Remove TCP_FASTOPEN, TCP_KEEPCNT flags on older version
Windows during run-time.

@item 
bpo-32777@footnote{https://bugs.python.org/issue32777}: Fix a rare but potential pre-exec child process deadlock in
subprocess on POSIX systems when marking file descriptors inheritable on
exec in the child process.  This bug appears to have been introduced in
3.4.

@item 
bpo-32647@footnote{https://bugs.python.org/issue32647}: The ctypes module used to depend on indirect linking for
dlopen. The shared extension is now explicitly linked against libdl on
platforms with dl.

@item 
bpo-32734@footnote{https://bugs.python.org/issue32734}: Fixed @code{asyncio.Lock()} safety issue which allowed acquiring
and locking the same lock multiple times, without it being free. Patch by
Bar Harel.

@item 
bpo-32727@footnote{https://bugs.python.org/issue32727}: Do not include name field in SMTP envelope from address. Patch
by Stéphane Wirtel

@item 
bpo-27931@footnote{https://bugs.python.org/issue27931}: Fix email address header parsing error when the username is an
empty quoted string. Patch by Xiang Zhang.

@item 
bpo-32304@footnote{https://bugs.python.org/issue32304}: distutils’ upload command no longer corrupts tar files ending
with a CR byte, and no longer tries to convert CR to CRLF in any of the
upload text fields.

@item 
bpo-32502@footnote{https://bugs.python.org/issue32502}: uuid.uuid1 no longer raises an exception if a 64-bit hardware
address is encountered.

@item 
bpo-31848@footnote{https://bugs.python.org/issue31848}: Fix the error handling in Aifc_read.initfp() when the SSND
chunk is not found. Patch by Zackery Spytz.

@item 
bpo-32555@footnote{https://bugs.python.org/issue32555}: On FreeBSD and Solaris, os.strerror() now always decode the
byte string from the current locale encoding, rather than using
ASCII/surrogateescape in some cases.

@item 
bpo-32521@footnote{https://bugs.python.org/issue32521}: The nis module is now compatible with new libnsl and headers
location.

@item 
bpo-32473@footnote{https://bugs.python.org/issue32473}: Improve ABCMeta._dump_registry() output readability

@item 
bpo-32521@footnote{https://bugs.python.org/issue32521}: glibc has removed Sun RPC. Use replacement libtirpc headers and
library in nis module.

@item 
bpo-32228@footnote{https://bugs.python.org/issue32228}: Ensure that @code{truncate()} preserves the file position (as
reported by @code{tell()}) after writes longer than the buffer size.

@item 
bpo-26133@footnote{https://bugs.python.org/issue26133}: Don’t unsubscribe signals in asyncio UNIX event loop on
interpreter shutdown.

@item 
bpo-32185@footnote{https://bugs.python.org/issue32185}: The SSL module no longer sends IP addresses in SNI TLS
extension on platforms with OpenSSL 1.0.2+ or inet_pton.

@item 
bpo-32323@footnote{https://bugs.python.org/issue32323}: @ref{88f,,urllib.parse.urlsplit()} does not convert zone-id
(scope) to lower case for scoped IPv6 addresses in hostnames now.

@item 
bpo-32302@footnote{https://bugs.python.org/issue32302}: Fix bdist_wininst of distutils for CRT v142: it binary
compatible with CRT v140.

@item 
bpo-32255@footnote{https://bugs.python.org/issue32255}: A single empty field is now always quoted when written into a
CSV file. This allows to distinguish an empty row from a row consisting of
a single empty field. Patch by Licht Takeuchi.

@item 
bpo-32277@footnote{https://bugs.python.org/issue32277}: Raise @code{NotImplementedError} instead of @code{SystemError} on
platforms where @code{chmod(..., follow_symlinks=False)} is not supported.
Patch by Anthony Sottile.

@item 
bpo-32199@footnote{https://bugs.python.org/issue32199}: The getnode() ip getter now uses ‘ip link’ instead of ‘ip link
list’.

@item 
bpo-27456@footnote{https://bugs.python.org/issue27456}: Ensure TCP_NODELAY is set on Linux. Tests by Victor Stinner.

@item 
bpo-31900@footnote{https://bugs.python.org/issue31900}: The @ref{739,,locale.localeconv()} function now sets temporarily the
@code{LC_CTYPE} locale to the @code{LC_NUMERIC} locale to decode
@code{decimal_point} and @code{thousands_sep} byte strings if they are non-ASCII
or longer than 1 byte, and the @code{LC_NUMERIC} locale is different than the
@code{LC_CTYPE} locale. This temporary change affects other threads. Same
change for the @ref{784,,str.format()} method when formatting a number
(@ref{1b1,,int}, @ref{256,,float}, @ref{256,,float} and subclasses) with the
@code{n} type (ex: @code{'@{:n@}'.format(1234)}).

@item 
bpo-31802@footnote{https://bugs.python.org/issue31802}: Importing native path module (@code{posixpath}, @code{ntpath}) now
works even if the @code{os} module still is not imported.
@end itemize

@node Documentation<39>,Tests<36>,Library<39>,Python 3 6 5 release candidate 1
@anchor{whatsnew/changelog id340}@anchor{1399}
@subsubsection Documentation



@itemize -

@item 
bpo-17232@footnote{https://bugs.python.org/issue17232}: Clarify docs for -O and -OO.  Patch by Terry Reedy.

@item 
bpo-32800@footnote{https://bugs.python.org/issue32800}: Update link to w3c doc for xml default namespaces.

@item 
bpo-8722@footnote{https://bugs.python.org/issue8722}: Document @code{__getattr__()} behavior when property @code{get()}
method raises @ref{163,,AttributeError}.

@item 
bpo-32614@footnote{https://bugs.python.org/issue32614}: Modify RE examples in documentation to use raw strings to
prevent @ref{1b2,,DeprecationWarning} and add text to REGEX HOWTO to highlight
the deprecation.

@item 
bpo-31972@footnote{https://bugs.python.org/issue31972}: Improve docstrings for @ref{1c4,,pathlib.PurePath} subclasses.

@item 
bpo-17799@footnote{https://bugs.python.org/issue17799}: Explain real behaviour of sys.settrace and sys.setprofile and
their C-API counterparts regarding which type of events are received in
each function. Patch by Pablo Galindo Salgado.
@end itemize

@node Tests<36>,Build<37>,Documentation<39>,Python 3 6 5 release candidate 1
@anchor{whatsnew/changelog id341}@anchor{139a}
@subsubsection Tests



@itemize -

@item 
bpo-32517@footnote{https://bugs.python.org/issue32517}: Fix failing @code{test_asyncio} on macOS 10.12.2+ due to transport
of @code{KqueueSelector} loop was not being closed.

@item 
bpo-32721@footnote{https://bugs.python.org/issue32721}: Fix test_hashlib to not fail if the _md5 module is not built.

@item 
bpo-32252@footnote{https://bugs.python.org/issue32252}: Fix faulthandler_suppress_crash_report() used to prevent core
dump files when testing crashes. getrlimit() returns zero on success.

@item 
bpo-31518@footnote{https://bugs.python.org/issue31518}: Debian Unstable has disabled TLS 1.0 and 1.1 for
SSLv23_METHOD(). Change TLS/SSL protocol of some tests to PROTOCOL_TLS or
PROTOCOL_TLSv1_2 to make them pass on Debian.
@end itemize

@node Build<37>,Windows<34>,Tests<36>,Python 3 6 5 release candidate 1
@anchor{whatsnew/changelog id342}@anchor{139b}
@subsubsection Build



@itemize -

@item 
bpo-32635@footnote{https://bugs.python.org/issue32635}: Fix segfault of the crypt module when libxcrypt is provided
instead of libcrypt at the system.
@end itemize

@node Windows<34>,macOS<28>,Build<37>,Python 3 6 5 release candidate 1
@anchor{whatsnew/changelog id343}@anchor{139c}
@subsubsection Windows



@itemize -

@item 
bpo-33016@footnote{https://bugs.python.org/issue33016}: Fix potential use of uninitialized memory in
nt._getfinalpathname

@item 
bpo-32903@footnote{https://bugs.python.org/issue32903}: Fix a memory leak in os.chdir() on Windows if the current
directory is set to a UNC path.

@item 
bpo-31966@footnote{https://bugs.python.org/issue31966}: Fixed WindowsConsoleIO.write() for writing empty data.

@item 
bpo-32409@footnote{https://bugs.python.org/issue32409}: Ensures activate.bat can handle Unicode contents.

@item 
bpo-32457@footnote{https://bugs.python.org/issue32457}: Improves handling of denormalized executable path when
launching Python.

@item 
bpo-32370@footnote{https://bugs.python.org/issue32370}: Use the correct encoding for ipconfig output in the uuid
module. Patch by Segev Finer.

@item 
bpo-29248@footnote{https://bugs.python.org/issue29248}: Fix @ref{414,,os.readlink()} on Windows, which was mistakenly
treating the @code{PrintNameOffset} field of the reparse data buffer as a
number of characters instead of bytes. Patch by Craig Holmquist and SSE4.

@item 
bpo-32588@footnote{https://bugs.python.org/issue32588}: Create standalone _distutils_findvs module.
@end itemize

@node macOS<28>,IDLE<34>,Windows<34>,Python 3 6 5 release candidate 1
@anchor{whatsnew/changelog id344}@anchor{139d}
@subsubsection macOS



@itemize -

@item 
bpo-32726@footnote{https://bugs.python.org/issue32726}: Provide an additional, more modern macOS installer variant that
supports macOS 10.9+ systems in 64-bit mode only. Upgrade the supplied
third-party libraries to OpenSSL 1.0.2n, XZ 5.2.3, and SQLite 3.22.0. The
10.9+ installer now links with and supplies its own copy of Tcl/Tk 8.6.8.
@end itemize

@node IDLE<34>,Tools/Demos<19>,macOS<28>,Python 3 6 5 release candidate 1
@anchor{whatsnew/changelog id345}@anchor{139e}
@subsubsection IDLE



@itemize -

@item 
bpo-32984@footnote{https://bugs.python.org/issue32984}: Set @code{__file__} while running a startup file.  Like Python,
IDLE optionally runs one startup file in the Shell window before
presenting the first interactive input prompt.  For IDLE, @code{-s} runs a
file named in environmental variable  
@geindex IDLESTARTUP
@geindex environment variable; IDLESTARTUP
@code{IDLESTARTUP} or
@geindex PYTHONSTARTUP
@geindex environment variable; PYTHONSTARTUP
@ref{b4d,,PYTHONSTARTUP}; @code{-r file} runs @code{file}.  Python sets
@code{__file__} to the startup file name before running the file and unsets
it before the first prompt.  IDLE now does the same when run normally,
without the @code{-n} option.

@item 
bpo-32940@footnote{https://bugs.python.org/issue32940}: Simplify and rename StringTranslatePseudoMapping in pyparse.

@item 
bpo-32916@footnote{https://bugs.python.org/issue32916}: Change @code{str} to @code{code} in pyparse.

@item 
bpo-32905@footnote{https://bugs.python.org/issue32905}: Remove unused code in pyparse module.

@item 
bpo-32874@footnote{https://bugs.python.org/issue32874}: Add tests for pyparse.

@item 
bpo-32837@footnote{https://bugs.python.org/issue32837}: Using the system and place-dependent default encoding for
open() is a bad idea for IDLE’s system and location-independent files.

@item 
bpo-32826@footnote{https://bugs.python.org/issue32826}: Add “encoding=utf-8” to open() in IDLE’s test_help_about. GUI
test test_file_buttons() only looks at initial ascii-only lines, but
failed on systems where open() defaults to ‘ascii’ because readline()
internally reads and decodes far enough ahead to encounter a non-ascii
character in CREDITS.txt.

@item 
bpo-32765@footnote{https://bugs.python.org/issue32765}: Update configdialog General tab docstring to add new widgets to
the widget list.
@end itemize

@node Tools/Demos<19>,C API<34>,IDLE<34>,Python 3 6 5 release candidate 1
@anchor{whatsnew/changelog id346}@anchor{139f}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-24960@footnote{https://bugs.python.org/issue24960}: 2to3 and lib2to3 can now read pickled grammar files using
pkgutil.get_data() rather than probing the filesystem. This lets 2to3 and
lib2to3 work when run from a zipfile.

@item 
bpo-32222@footnote{https://bugs.python.org/issue32222}: Fix pygettext not extracting docstrings for functions with type
annotated arguments. Patch by Toby Harradine.
@end itemize

@node C API<34>,,Tools/Demos<19>,Python 3 6 5 release candidate 1
@anchor{whatsnew/changelog id347}@anchor{13a0}
@subsubsection C API



@itemize -

@item 
bpo-29084@footnote{https://bugs.python.org/issue29084}: Undocumented C API for OrderedDict has been excluded from the
limited C API. It was added by mistake and actually never worked in the
limited C API.
@end itemize

@node Python 3 6 4 final,Python 3 6 4 release candidate 1,Python 3 6 5 release candidate 1,Changelog
@anchor{whatsnew/changelog python-3-6-4-final}@anchor{13a1}
@subsection Python 3.6.4 final


@emph{Release date: 2017-12-18}

There were no new code changes in version 3.6.4 since v3.6.4rc1.

@node Python 3 6 4 release candidate 1,Python 3 6 3 final,Python 3 6 4 final,Changelog
@anchor{whatsnew/changelog python-3-6-4-release-candidate-1}@anchor{13a2}
@subsection Python 3.6.4 release candidate 1


@emph{Release date: 2017-12-05}

@menu
* Core and Builtins: Core and Builtins<39>. 
* Library: Library<40>. 
* Documentation: Documentation<40>. 
* Tests: Tests<37>. 
* Build: Build<38>. 
* Windows: Windows<35>. 
* macOS: macOS<29>. 
* IDLE: IDLE<35>. 
* Tools/Demos: Tools/Demos<20>. 
* C API: C API<35>. 

@end menu

@node Core and Builtins<39>,Library<40>,,Python 3 6 4 release candidate 1
@anchor{whatsnew/changelog id348}@anchor{13a3}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-32176@footnote{https://bugs.python.org/issue32176}: co_flags.CO_NOFREE is now always set correctly by the code
object constructor based on freevars and cellvars, rather than needing to
be set correctly by the caller. This ensures it will be cleared
automatically when additional cell references are injected into a modified
code object and function.

@item 
bpo-31949@footnote{https://bugs.python.org/issue31949}: Fixed several issues in printing tracebacks
(PyTraceBack_Print()). Setting sys.tracebacklimit to 0 or less now
suppresses printing tracebacks. Setting sys.tracebacklimit to None now
causes using the default limit. Setting sys.tracebacklimit to an integer
larger than LONG_MAX now means using the limit LONG_MAX rather than the
default limit. Fixed integer overflows in the case of more than @code{2**31}
traceback items on Windows. Fixed output errors handling.

@item 
bpo-30696@footnote{https://bugs.python.org/issue30696}: Fix the interactive interpreter looping endlessly when no
memory.

@item 
bpo-20047@footnote{https://bugs.python.org/issue20047}: Bytearray methods partition() and rpartition() now accept only
bytes-like objects as separator, as documented.  In particular they now
raise TypeError rather of returning a bogus result when an integer is
passed as a separator.

@item 
bpo-31852@footnote{https://bugs.python.org/issue31852}: Fix a segmentation fault caused by a combination of the async
soft keyword and continuation lines.

@item 
bpo-21720@footnote{https://bugs.python.org/issue21720}: BytesWarning no longer emitted when the @emph{fromlist} argument of
@code{__import__()} or the @code{__all__} attribute of the module contain bytes
instances.

@item 
bpo-31825@footnote{https://bugs.python.org/issue31825}: Fixed OverflowError in the ‘unicode-escape’ codec and in
codecs.escape_decode() when decode an escaped non-ascii byte.

@item 
bpo-28603@footnote{https://bugs.python.org/issue28603}: Print the full context/cause chain of exceptions on interpreter
exit, even if an exception in the chain is unhashable or compares equal to
later ones. Patch by Zane Bitter.

@item 
bpo-31786@footnote{https://bugs.python.org/issue31786}: Fix timeout rounding in the select module to round correctly
negative timeouts between -1.0 and 0.0. The functions now block waiting
for events as expected. Previously, the call was incorrectly non-blocking.
Patch by Pablo Galindo.

@item 
bpo-31642@footnote{https://bugs.python.org/issue31642}: Restored blocking “from package import module” by setting
sys.modules[“package.module”] to None.

@item 
bpo-31626@footnote{https://bugs.python.org/issue31626}: Fixed a bug in debug memory allocator.  There was a write to
freed memory after shrinking a memory block.

@item 
bpo-31619@footnote{https://bugs.python.org/issue31619}: Fixed a ValueError when convert a string with large number of
underscores to integer with binary base.

@item 
bpo-31592@footnote{https://bugs.python.org/issue31592}: Fixed an assertion failure in Python parser in case of a bad
@ref{19a,,unicodedata.normalize()}. Patch by Oren Milman.

@item 
bpo-31588@footnote{https://bugs.python.org/issue31588}: Raise a @ref{162,,TypeError} with a helpful error message when class
creation fails due to a metaclass with a bad @code{__prepare__()} method.
Patch by Oren Milman.

@item 
bpo-31566@footnote{https://bugs.python.org/issue31566}: Fix an assertion failure in @code{_warnings.warn()} in case of a bad
@code{__name__} global. Patch by Oren Milman.

@item 
bpo-31505@footnote{https://bugs.python.org/issue31505}: Fix an assertion failure in @ref{a0,,json}, in case
@code{_json.make_encoder()} received a bad @code{encoder()} argument. Patch by Oren
Milman.

@item 
bpo-31492@footnote{https://bugs.python.org/issue31492}: Fix assertion failures in case of failing to import from a
module with a bad @code{__name__} attribute, and in case of failing to access
an attribute of such a module. Patch by Oren Milman.

@item 
bpo-31490@footnote{https://bugs.python.org/issue31490}: Fix an assertion failure in @ref{29,,ctypes} class definition, in case
the class has an attribute whose name is specified in @code{_anonymous_} but
not in @code{_fields_}. Patch by Oren Milman.

@item 
bpo-31478@footnote{https://bugs.python.org/issue31478}: Fix an assertion failure in @code{_random.Random.seed()} in case the
argument has a bad @code{__abs__()} method. Patch by Oren Milman.

@item 
bpo-31315@footnote{https://bugs.python.org/issue31315}: Fix an assertion failure in imp.create_dynamic(), when
spec.name is not a string. Patch by Oren Milman.

@item 
bpo-31311@footnote{https://bugs.python.org/issue31311}: Fix a crash in the @code{__setstate__()} method of
@ref{1371,,ctypes._CData}, in case of a bad @code{__dict__}. Patch by Oren Milman.

@item 
bpo-31293@footnote{https://bugs.python.org/issue31293}: Fix crashes in true division and multiplication of a timedelta
object by a float with a bad as_integer_ratio() method. Patch by Oren
Milman.

@item 
bpo-31285@footnote{https://bugs.python.org/issue31285}: Fix an assertion failure in @ref{84b,,warnings.warn_explicit}, when the
return value of the received loader’s get_source() has a bad splitlines()
method. Patch by Oren Milman.

@item 
bpo-30817@footnote{https://bugs.python.org/issue30817}: @code{PyErr_PrintEx()} clears now the ignored exception that may be
raised by @code{_PySys_SetObjectId()}, for example when no memory.
@end itemize

@node Library<40>,Documentation<40>,Core and Builtins<39>,Python 3 6 4 release candidate 1
@anchor{whatsnew/changelog id349}@anchor{13a4}
@subsubsection Library



@itemize -

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Two minor fixes for @code{typing} module: allow shallow copying
instances of generic classes, improve interaction of @code{__init_subclass__}
with generics. Original PRs by Ivan Levkivskyi.

@item 
bpo-27240@footnote{https://bugs.python.org/issue27240}: The header folding algorithm for the new email policies has
been rewritten, which also fixes bpo-30788@footnote{https://bugs.python.org/issue30788}, bpo-31831@footnote{https://bugs.python.org/issue31831}, and bpo-32182@footnote{https://bugs.python.org/issue32182}.  In
particular, RFC2231 folding is now done correctly.

@item 
bpo-32186@footnote{https://bugs.python.org/issue32186}: io.FileIO.readall() and io.FileIO.read() now release the GIL
when getting the file size. Fixed hang of all threads with inaccessible
NFS server. Patch by Nir Soffer.

@item 
bpo-12239@footnote{https://bugs.python.org/issue12239}: Make @ref{1292,,msilib.SummaryInformation.GetProperty()} return
@code{None} when the value of property is @code{VT_EMPTY}.  Initial patch by
Mark Mc Mahon.

@item 
bpo-31325@footnote{https://bugs.python.org/issue31325}: Fix wrong usage of @ref{3e4,,collections.namedtuple()} in the
@ref{1363,,RobotFileParser.parse()}
method. Initial patch by Robin Wellner.

@item 
bpo-12382@footnote{https://bugs.python.org/issue12382}: @ref{11d5,,msilib.OpenDatabase()} now raises a better exception
message when it couldn’t open or create an MSI file.  Initial patch by
William Tisäter.

@item 
bpo-32110@footnote{https://bugs.python.org/issue32110}: @code{codecs.StreamReader.read(n)} now returns not more than @emph{n}
characters/bytes for non-negative @emph{n}. This makes it compatible with
@code{read()} methods of other file-like objects.

@item 
bpo-32072@footnote{https://bugs.python.org/issue32072}: Fixed issues with binary plists: Fixed saving bytearrays.
Identical objects will be saved only once. Equal references will be load
as identical objects. Added support for saving and loading recursive data
structures.

@item 
bpo-32034@footnote{https://bugs.python.org/issue32034}: Make asyncio.IncompleteReadError and LimitOverrunError
pickleable.

@item 
bpo-32015@footnote{https://bugs.python.org/issue32015}: Fixed the looping of asyncio in the case of reconnection the
socket during waiting async read/write from/to the socket.

@item 
bpo-32011@footnote{https://bugs.python.org/issue32011}: Restored support of loading marshal files with the TYPE_INT64
code. These files can be produced in Python 2.7.

@item 
bpo-31970@footnote{https://bugs.python.org/issue31970}: Reduce performance overhead of asyncio debug mode.

@item 
bpo-9678@footnote{https://bugs.python.org/issue9678}: Fixed determining the MAC address in the uuid module: Using
ifconfig on NetBSD and OpenBSD. Using arp on Linux, FreeBSD, NetBSD and
OpenBSD. Based on patch by Takayuki Shimizukawa.

@item 
bpo-30057@footnote{https://bugs.python.org/issue30057}: Fix potential missed signal in signal.signal().

@item 
bpo-31933@footnote{https://bugs.python.org/issue31933}: Fix Blake2 params leaf_size and node_offset on big endian
platforms. Patch by Jack O’Connor.

@item 
bpo-31927@footnote{https://bugs.python.org/issue31927}: Fixed compilation of the socket module on NetBSD 8.  Fixed
assertion failure or reading arbitrary data when parse a AF_BLUETOOTH
address on NetBSD and DragonFly BSD.

@item 
bpo-27666@footnote{https://bugs.python.org/issue27666}: Fixed stack corruption in curses.box() and curses.ungetmouse()
when the size of types chtype or mmask_t is less than the size of C long.
curses.box() now accepts characters as arguments.  Based on patch by Steve
Fink.

@item 
bpo-31897@footnote{https://bugs.python.org/issue31897}: plistlib now catches more errors when read binary plists and
raises InvalidFileException instead of unexpected exceptions.

@item 
bpo-25720@footnote{https://bugs.python.org/issue25720}: Fix the method for checking pad state of curses WINDOW. Patch
by Masayuki Yamamoto.

@item 
bpo-31893@footnote{https://bugs.python.org/issue31893}: Fixed the layout of the kqueue_event structure on OpenBSD and
NetBSD. Fixed the comparison of the kqueue_event objects.

@item 
bpo-31891@footnote{https://bugs.python.org/issue31891}: Fixed building the curses module on NetBSD.

@item 
bpo-28416@footnote{https://bugs.python.org/issue28416}: Instances of pickle.Pickler subclass with the persistent_id()
method and pickle.Unpickler subclass with the persistent_load() method no
longer create reference cycles.

@item 
bpo-28326@footnote{https://bugs.python.org/issue28326}: Fix multiprocessing.Process when stdout and/or stderr is closed
or None.

@item 
bpo-31457@footnote{https://bugs.python.org/issue31457}: If nested log adapters are used, the inner @code{process()}
methods are no longer omitted.

@item 
bpo-31457@footnote{https://bugs.python.org/issue31457}: The @code{manager} property on LoggerAdapter objects is now
properly settable.

@item 
bpo-31806@footnote{https://bugs.python.org/issue31806}: Fix timeout rounding in time.sleep(), threading.Lock.acquire()
and socket.socket.settimeout() to round correctly negative timeouts
between -1.0 and 0.0. The functions now block waiting for events as
expected. Previously, the call was incorrectly non-blocking. Patch by
Pablo Galindo.

@item 
bpo-28603@footnote{https://bugs.python.org/issue28603}: traceback: Fix a TypeError that occurred during printing of
exception tracebacks when either the current exception or an exception in
its context/cause chain is unhashable. Patch by Zane Bitter.

@item 
bpo-30058@footnote{https://bugs.python.org/issue30058}: Fixed buffer overflow in select.kqueue.control().

@item 
bpo-31770@footnote{https://bugs.python.org/issue31770}: Prevent a crash when calling the @code{__init__()} method of a
@code{sqlite3.Cursor} object more than once. Patch by Oren Milman.

@item 
bpo-31672@footnote{https://bugs.python.org/issue31672}: @code{idpattern} in @code{string.Template} matched some non-ASCII
characters. Now it uses @code{-i} regular expression local flag to avoid
non-ASCII characters.

@item 
bpo-31764@footnote{https://bugs.python.org/issue31764}: Prevent a crash in @code{sqlite3.Cursor.close()} in case the
@code{Cursor} object is uninitialized. Patch by Oren Milman.

@item 
bpo-31752@footnote{https://bugs.python.org/issue31752}: Fix possible crash in timedelta constructor called with custom
integers.

@item 
bpo-31701@footnote{https://bugs.python.org/issue31701}: On Windows, faulthandler.enable() now ignores MSC and COM
exceptions.

@item 
bpo-31728@footnote{https://bugs.python.org/issue31728}: Prevent crashes in @code{_elementtree} due to unsafe cleanup of
@code{Element.text} and @code{Element.tail}. Patch by Oren Milman.

@item 
bpo-31620@footnote{https://bugs.python.org/issue31620}: an empty asyncio.Queue now doesn’t leak memory when queue.get
pollers timeout

@item 
bpo-31632@footnote{https://bugs.python.org/issue31632}: Fix method set_protocol() of class _SSLProtocolTransport in
asyncio module. This method was previously modifying a wrong reference to
the protocol.

@item 
bpo-31675@footnote{https://bugs.python.org/issue31675}: Fixed memory leaks in Tkinter’s methods splitlist() and split()
when pass a string larger than 2 GiB.

@item 
bpo-31673@footnote{https://bugs.python.org/issue31673}: Fixed typo in the name of Tkinter’s method adderrorinfo().

@item 
bpo-30806@footnote{https://bugs.python.org/issue30806}: Fix the string representation of a netrc object.

@item 
bpo-15037@footnote{https://bugs.python.org/issue15037}: Added a workaround for getkey() in curses for ncurses 5.7 and
earlier.

@item 
bpo-25351@footnote{https://bugs.python.org/issue25351}: Avoid venv activate failures with undefined variables

@item 
bpo-25532@footnote{https://bugs.python.org/issue25532}: inspect.unwrap() will now only try to unwrap an object
sys.getrecursionlimit() times, to protect against objects which create a
new object on every attribute access.

@item 
bpo-30347@footnote{https://bugs.python.org/issue30347}: Stop crashes when concurrently iterate over itertools.groupby()
iterators.

@item 
bpo-31516@footnote{https://bugs.python.org/issue31516}: @code{threading.current_thread()} should not return a dummy thread
at shutdown.

@item 
bpo-31351@footnote{https://bugs.python.org/issue31351}: python -m ensurepip now exits with non-zero exit code if pip
bootstrapping has failed.

@item 
bpo-31482@footnote{https://bugs.python.org/issue31482}: @code{random.seed()} now works with bytes in version=1

@item 
bpo-31334@footnote{https://bugs.python.org/issue31334}: Fix @code{poll.poll([timeout])} in the @code{select} module for
arbitrary negative timeouts on all OSes where it can only be a
non-negative integer or -1. Patch by Riccardo Coccioli.

@item 
bpo-31310@footnote{https://bugs.python.org/issue31310}: multiprocessing’s semaphore tracker should be launched again if
crashed.

@item 
bpo-31308@footnote{https://bugs.python.org/issue31308}: Make multiprocessing’s forkserver process immune to Ctrl-C and
other user interruptions. If it crashes, restart it when necessary.
@end itemize

@node Documentation<40>,Tests<37>,Library<40>,Python 3 6 4 release candidate 1
@anchor{whatsnew/changelog id350}@anchor{13a5}
@subsubsection Documentation



@itemize -

@item 
bpo-32105@footnote{https://bugs.python.org/issue32105}: Added asyncio.BaseEventLoop.connect_accepted_socket
versionadded marker.

@item 
bpo-31537@footnote{https://bugs.python.org/issue31537}: Fix incorrect usage of @code{get_history_length} in readline
documentation example code. Patch by Brad Smith.

@item 
bpo-30085@footnote{https://bugs.python.org/issue30085}: The operator functions without double underscores are preferred
for clarity. The one with underscores are only kept for
back-compatibility.
@end itemize

@node Tests<37>,Build<38>,Documentation<40>,Python 3 6 4 release candidate 1
@anchor{whatsnew/changelog id351}@anchor{13a6}
@subsubsection Tests



@itemize -

@item 
bpo-31380@footnote{https://bugs.python.org/issue31380}: Skip test_httpservers test_undecodable_file on macOS: fails on
APFS.

@item 
bpo-31705@footnote{https://bugs.python.org/issue31705}: Skip test_socket.test_sha256() on Linux kernel older than 4.5.
The test fails with ENOKEY on kernel 3.10 (on ppc64le). A fix was merged
into the kernel 4.5.

@item 
bpo-31174@footnote{https://bugs.python.org/issue31174}: Fix test_tools.test_unparse: DirectoryTestCase now stores the
names sample to always test the same files. It prevents false alarms when
hunting reference leaks.

@item 
bpo-30695@footnote{https://bugs.python.org/issue30695}: Add the @code{set_nomemory(start, stop)} and @code{remove_mem_hooks()}
functions to the _testcapi module.
@end itemize

@node Build<38>,Windows<35>,Tests<37>,Python 3 6 4 release candidate 1
@anchor{whatsnew/changelog id352}@anchor{13a7}
@subsubsection Build



@itemize -

@item 
bpo-32059@footnote{https://bugs.python.org/issue32059}: @code{detect_modules()} in @code{setup.py} now also searches the
sysroot paths when cross-compiling.

@item 
bpo-31957@footnote{https://bugs.python.org/issue31957}: Fixes Windows SDK version detection when building for Windows.

@item 
bpo-31609@footnote{https://bugs.python.org/issue31609}: Fixes quotes in PCbuild/clean.bat

@item 
bpo-31934@footnote{https://bugs.python.org/issue31934}: Abort the build when building out of a not clean source tree.

@item 
bpo-31926@footnote{https://bugs.python.org/issue31926}: Fixed Argument Clinic sometimes causing compilation errors when
there was more than one function and/or method in a .c file with the same
name.

@item 
bpo-28791@footnote{https://bugs.python.org/issue28791}: Update Windows builds to use SQLite 3.21.0.

@item 
bpo-28791@footnote{https://bugs.python.org/issue28791}: Update OS X installer to use SQLite 3.21.0.

@item 
bpo-22140@footnote{https://bugs.python.org/issue22140}: Prevent double substitution of prefix in python-config.sh.

@item 
bpo-31536@footnote{https://bugs.python.org/issue31536}: Avoid wholesale rebuild after @code{make regen-all} if nothing
changed.
@end itemize

@node Windows<35>,macOS<29>,Build<38>,Python 3 6 4 release candidate 1
@anchor{whatsnew/changelog id353}@anchor{13a8}
@subsubsection Windows



@itemize -

@item 
bpo-1102@footnote{https://bugs.python.org/issue1102}: Return @code{None} when @code{View.Fetch()} returns
@code{ERROR_NO_MORE_ITEMS} instead of raising @code{MSIError}. Initial patch by
Anthony Tuininga.

@item 
bpo-31944@footnote{https://bugs.python.org/issue31944}: Fixes Modify button in Apps and Features dialog.
@end itemize

@node macOS<29>,IDLE<35>,Windows<35>,Python 3 6 4 release candidate 1
@anchor{whatsnew/changelog id354}@anchor{13a9}
@subsubsection macOS



@itemize -

@item 
bpo-31392@footnote{https://bugs.python.org/issue31392}: Update macOS installer to use OpenSSL 1.0.2m
@end itemize

@node IDLE<35>,Tools/Demos<20>,macOS<29>,Python 3 6 4 release candidate 1
@anchor{whatsnew/changelog id355}@anchor{13aa}
@subsubsection IDLE



@itemize -

@item 
bpo-32207@footnote{https://bugs.python.org/issue32207}: Improve tk event exception tracebacks in IDLE. When tk event
handling is driven by IDLE’s run loop, a confusing and distracting
queue.EMPTY traceback context is no longer added to tk event exception
tracebacks.  The traceback is now the same as when event handling is
driven by user code.  Patch based on a suggestion by Serhiy Storchaka.

@item 
bpo-32164@footnote{https://bugs.python.org/issue32164}: Delete unused file idlelib/tabbedpages.py. Use of TabbedPageSet
in configdialog was replaced by ttk.Notebook.

@item 
bpo-32100@footnote{https://bugs.python.org/issue32100}: IDLE: Fix old and new bugs in pathbrowser; improve tests. Patch
mostly by Cheryl Sabella.

@item 
bpo-31858@footnote{https://bugs.python.org/issue31858}: IDLE – Restrict shell prompt manipulation to the shell. Editor
and output windows only see an empty last prompt line.  This simplifies
the code and fixes a minor bug when newline is inserted. Sys.ps1, if
present, is read on Shell start-up, but is not set or changed.

@item 
bpo-31860@footnote{https://bugs.python.org/issue31860}: The font sample in the IDLE configuration dialog is now
editable. Changes persist while IDLE remains open

@item 
bpo-31836@footnote{https://bugs.python.org/issue31836}: Test_code_module now passes if run after test_idle, which sets
ps1. The code module uses sys.ps1 if present or sets it to ‘>>> ‘ if not.
Test_code_module now properly tests both behaviors.  Ditto for ps2.

@item 
bpo-28603@footnote{https://bugs.python.org/issue28603}: Fix a TypeError that caused a shell restart when printing a
traceback that includes an exception that is unhashable. Patch by Zane
Bitter.

@item 
bpo-13802@footnote{https://bugs.python.org/issue13802}: Use non-Latin characters in the IDLE’s Font settings sample.
Even if one selects a font that defines a limited subset of the unicode
Basic Multilingual Plane, tcl/tk will use other fonts that define a
character. The expanded example give users of non-Latin characters a
better idea of what they might see in IDLE’s shell and editors. To make
room for the expanded sample, frames on the Font tab are re-arranged. The
Font/Tabs help explains a bit about the additions.

@item 
bpo-31460@footnote{https://bugs.python.org/issue31460}: Simplify the API of IDLE’s Module Browser. Passing a widget
instead of an flist with a root widget opens the option of creating a
browser frame that is only part of a window.  Passing a full file name
instead of pieces assumed to come from a .py file opens the possibility of
browsing python files that do not end in .py.

@item 
bpo-31649@footnote{https://bugs.python.org/issue31649}: IDLE - Make _htest, _utest parameters keyword only.

@item 
bpo-31559@footnote{https://bugs.python.org/issue31559}: Remove test order dependence in idle_test.test_browser.

@item 
bpo-31459@footnote{https://bugs.python.org/issue31459}: Rename IDLE’s module browser from Class Browser to Module
Browser. The original module-level class and method browser became a
module browser, with the addition of module-level functions, years ago.
Nested classes and functions were added yesterday.  For
back-compatibility, the virtual event <<open-class-browser>>, which
appears on the Keys tab of the Settings dialog, is not changed. Patch by
Cheryl Sabella.

@item 
bpo-31500@footnote{https://bugs.python.org/issue31500}: Default fonts now are scaled on HiDPI displays.

@item 
bpo-1612262@footnote{https://bugs.python.org/issue1612262}: IDLE module browser now shows nested classes and functions.
Original patches for code and tests by Guilherme Polo and Cheryl Sabella,
respectively.
@end itemize

@node Tools/Demos<20>,C API<35>,IDLE<35>,Python 3 6 4 release candidate 1
@anchor{whatsnew/changelog id356}@anchor{13ab}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-30722@footnote{https://bugs.python.org/issue30722}: Make redemo work with Python 3.6 and newer versions. Also,
remove the @code{LOCALE} option since it doesn’t work with string patterns in
Python 3. Patch by Christoph Sarnowski.
@end itemize

@node C API<35>,,Tools/Demos<20>,Python 3 6 4 release candidate 1
@anchor{whatsnew/changelog id357}@anchor{13ac}
@subsubsection C API



@itemize -

@item 
bpo-20891@footnote{https://bugs.python.org/issue20891}: Fix PyGILState_Ensure(). When PyGILState_Ensure() is called in
a non-Python thread before PyEval_InitThreads(), only call
PyEval_InitThreads() after calling PyThreadState_New() to fix a crash.

@item 
bpo-31532@footnote{https://bugs.python.org/issue31532}: Fix memory corruption due to allocator mix in getpath.c between
Py_GetPath() and Py_SetPath()

@item 
bpo-30697@footnote{https://bugs.python.org/issue30697}: The @code{PyExc_RecursionErrorInst} singleton is removed and
@code{PyErr_NormalizeException()} does not use it anymore. This singleton is
persistent and its members being never cleared may cause a segfault during
finalization of the interpreter. See also bpo-22898@footnote{https://bugs.python.org/issue22898}.
@end itemize

@node Python 3 6 3 final,Python 3 6 3 release candidate 1,Python 3 6 4 release candidate 1,Changelog
@anchor{whatsnew/changelog python-3-6-3-final}@anchor{13ad}
@subsection Python 3.6.3 final


@emph{Release date: 2017-10-03}

@menu
* Library: Library<41>. 
* Build: Build<39>. 

@end menu

@node Library<41>,Build<39>,,Python 3 6 3 final
@anchor{whatsnew/changelog id358}@anchor{13ae}
@subsubsection Library



@itemize -

@item 
bpo-31641@footnote{https://bugs.python.org/issue31641}: Re-allow arbitrary iterables in
@ref{13af,,concurrent.futures.as_completed()}. Fixes regression in 3.6.3rc1.
@end itemize

@node Build<39>,,Library<41>,Python 3 6 3 final
@anchor{whatsnew/changelog id359}@anchor{13b0}
@subsubsection Build



@itemize -

@item 
bpo-31662@footnote{https://bugs.python.org/issue31662}: Fix typos in Windows @code{uploadrelease.bat} script. Fix Windows
Doc build issues in @code{Doc/make.bat}.

@item 
bpo-31423@footnote{https://bugs.python.org/issue31423}: Fix building the PDF documentation with newer versions of
Sphinx.
@end itemize

@node Python 3 6 3 release candidate 1,Python 3 6 2 final,Python 3 6 3 final,Changelog
@anchor{whatsnew/changelog python-3-6-3-release-candidate-1}@anchor{13b1}
@subsection Python 3.6.3 release candidate 1


@emph{Release date: 2017-09-18}

@menu
* Security: Security<23>. 
* Core and Builtins: Core and Builtins<40>. 
* Library: Library<42>. 
* Documentation: Documentation<41>. 
* Tests: Tests<38>. 
* Build: Build<40>. 
* Windows: Windows<36>. 
* IDLE: IDLE<36>. 
* Tools/Demos: Tools/Demos<21>. 

@end menu

@node Security<23>,Core and Builtins<40>,,Python 3 6 3 release candidate 1
@anchor{whatsnew/changelog id360}@anchor{13b2}
@subsubsection Security



@itemize -

@item 
bpo-29781@footnote{https://bugs.python.org/issue29781}: SSLObject.version() now correctly returns None when handshake
over BIO has not been performed yet.

@item 
bpo-30947@footnote{https://bugs.python.org/issue30947}: Upgrade libexpat embedded copy from version 2.2.1 to 2.2.3 to
get security fixes.
@end itemize

@node Core and Builtins<40>,Library<42>,Security<23>,Python 3 6 3 release candidate 1
@anchor{whatsnew/changelog id361}@anchor{13b3}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-31471@footnote{https://bugs.python.org/issue31471}: Fix an assertion failure in @ref{5ae,,subprocess.Popen()} on Windows, in
case the env argument has a bad keys() method. Patch by Oren Milman.

@item 
bpo-31418@footnote{https://bugs.python.org/issue31418}: Fix an assertion failure in @code{PyErr_WriteUnraisable()} in case
of an exception with a bad @code{__module__} attribute. Patch by Oren Milman.

@item 
bpo-31416@footnote{https://bugs.python.org/issue31416}: Fix assertion failures in case of a bad warnings.filters or
warnings.defaultaction. Patch by Oren Milman.

@item 
bpo-31411@footnote{https://bugs.python.org/issue31411}: Raise a TypeError instead of SystemError in case
warnings.onceregistry is not a dictionary. Patch by Oren Milman.

@item 
bpo-31373@footnote{https://bugs.python.org/issue31373}: Fix several possible instances of undefined behavior due to
floating-point demotions.

@item 
bpo-30465@footnote{https://bugs.python.org/issue30465}: Location information (@code{lineno} and @code{col_offset}) in
f-strings is now (mostly) correct.  This fixes tools like flake8 from
showing warnings on the wrong line (typically the first line of the file).

@item 
bpo-31343@footnote{https://bugs.python.org/issue31343}: Include sys/sysmacros.h for major(), minor(), and makedev().
GNU C libray plans to remove the functions from sys/types.h.

@item 
bpo-31291@footnote{https://bugs.python.org/issue31291}: Fix an assertion failure in @ref{137b,,zipimport.zipimporter.get_data} on
Windows, when the return value of @code{pathname.replace('/','\\')} isn’t a
string. Patch by Oren Milman.

@item 
bpo-31271@footnote{https://bugs.python.org/issue31271}: Fix an assertion failure in the write() method of
@ref{889,,io.TextIOWrapper}, when the encoder doesn’t return a bytes object. Patch
by Oren Milman.

@item 
bpo-31243@footnote{https://bugs.python.org/issue31243}: Fix a crash in some methods of @ref{889,,io.TextIOWrapper}, when the
decoder’s state is invalid. Patch by Oren Milman.

@item 
bpo-30721@footnote{https://bugs.python.org/issue30721}: @code{print} now shows correct usage hint for using Python 2
redirection syntax.  Patch by Sanyam Khurana.

@item 
bpo-31070@footnote{https://bugs.python.org/issue31070}: Fix a race condition in importlib _get_module_lock().

@item 
bpo-31095@footnote{https://bugs.python.org/issue31095}: Fix potential crash during GC caused by @code{tp_dealloc} which
doesn’t call @code{PyObject_GC_UnTrack()}.

@item 
bpo-31071@footnote{https://bugs.python.org/issue31071}: Avoid masking original TypeError in call with * unpacking when
other arguments are passed.

@item 
bpo-30978@footnote{https://bugs.python.org/issue30978}: str.format_map() now passes key lookup exceptions through.
Previously any exception was replaced with a KeyError exception.

@item 
bpo-30808@footnote{https://bugs.python.org/issue30808}: Use _Py_atomic API for concurrency-sensitive signal state.

@item 
bpo-30876@footnote{https://bugs.python.org/issue30876}: Relative import from unloaded package now reimports the package
instead of failing with SystemError.  Relative import from non-package now
fails with ImportError rather than SystemError.

@item 
bpo-30703@footnote{https://bugs.python.org/issue30703}: Improve signal delivery. Avoid using Py_AddPendingCall from
signal handler, to avoid calling signal-unsafe functions. The tests I’m
adding here fail without the rest of the patch, on Linux and OS X. This
means our signal delivery logic had defects (some signals could be lost).

@item 
bpo-30765@footnote{https://bugs.python.org/issue30765}: Avoid blocking in pthread_mutex_lock() when
PyThread_acquire_lock() is asked not to block.

@item 
bpo-31161@footnote{https://bugs.python.org/issue31161}: Make sure the ‘Missing parentheses’ syntax error message is
only applied to SyntaxError, not to subclasses. Patch by Martijn Pieters.

@item 
bpo-30814@footnote{https://bugs.python.org/issue30814}: Fixed a race condition when import a submodule from a package.

@item 
bpo-30597@footnote{https://bugs.python.org/issue30597}: @code{print} now shows expected input in custom error message when
used as a Python 2 statement. Patch by Sanyam Khurana.
@end itemize

@node Library<42>,Documentation<41>,Core and Builtins<40>,Python 3 6 3 release candidate 1
@anchor{whatsnew/changelog id362}@anchor{13b4}
@subsubsection Library



@itemize -

@item 
bpo-31499@footnote{https://bugs.python.org/issue31499}: xml.etree: Fix a crash when a parser is part of a reference
cycle.

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: typing.get_type_hints now finds the right globalns for classes
and modules by default (when no @code{globalns} was specified by the caller).

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Speed improvements to the @code{typing} module.  Original PRs by
Ivan Levkivskyi and Mitar.

@item 
bpo-31544@footnote{https://bugs.python.org/issue31544}: The C accelerator module of ElementTree ignored exceptions
raised when looking up TreeBuilder target methods in XMLParser().

@item 
bpo-31234@footnote{https://bugs.python.org/issue31234}: socket.create_connection() now fixes manually a reference
cycle: clear the variable storing the last exception on success.

@item 
bpo-31457@footnote{https://bugs.python.org/issue31457}: LoggerAdapter objects can now be nested.

@item 
bpo-31400@footnote{https://bugs.python.org/issue31400}: Improves SSL error handling to avoid losing error numbers.

@item 
bpo-28958@footnote{https://bugs.python.org/issue28958}: ssl.SSLContext() now uses OpenSSL error information when a
context cannot be instantiated.

@item 
bpo-27340@footnote{https://bugs.python.org/issue27340}: SSLSocket.sendall() now uses memoryview to create slices of
data. This fixes support for all bytes-like object. It is also more
efficient and avoids costly copies.

@item 
bpo-31178@footnote{https://bugs.python.org/issue31178}: Fix string concatenation bug in rare error path in the
subprocess module

@item 
bpo-31350@footnote{https://bugs.python.org/issue31350}: Micro-optimize @code{asyncio._get_running_loop()} to become up
to 10% faster.

@item 
bpo-31170@footnote{https://bugs.python.org/issue31170}: expat: Update libexpat from 2.2.3 to 2.2.4. Fix copying of
partial characters for UTF-8 input (libexpat bug 115):
@indicateurl{https://github.com/libexpat/libexpat/issues/115}

@item 
bpo-29136@footnote{https://bugs.python.org/issue29136}: Add TLS 1.3 cipher suites and OP_NO_TLSv1_3.

@item 
bpo-29212@footnote{https://bugs.python.org/issue29212}: Fix concurrent.futures.thread.ThreadPoolExecutor threads to
have a non repr() based thread name by default when no thread_name_prefix
is supplied. They will now identify themselves as
“ThreadPoolExecutor-y_n”.

@item 
bpo-9146@footnote{https://bugs.python.org/issue9146}: Fix a segmentation fault in _hashopenssl when standard hash
functions such as md5 are not available in the linked OpenSSL library.  As
in some special FIPS-140 build environments.

@item 
bpo-27144@footnote{https://bugs.python.org/issue27144}: The @code{map()} and @code{as_completed()} iterators in
@code{concurrent.futures} now avoid keeping a reference to yielded objects.

@item 
bpo-10746@footnote{https://bugs.python.org/issue10746}: Fix ctypes producing wrong 
@geindex Python Enhancement Proposals; PEP 3118
PEP 3118@footnote{https://www.python.org/dev/peps/pep-3118} type codes for integer
types.

@item 
bpo-22536@footnote{https://bugs.python.org/issue22536}: The subprocess module now sets the filename when
FileNotFoundError is raised on POSIX systems due to the executable or cwd
not being found.

@item 
bpo-31249@footnote{https://bugs.python.org/issue31249}: concurrent.futures: WorkItem.run() used by ThreadPoolExecutor
now breaks a reference cycle between an exception object and the WorkItem
object.

@item 
bpo-31247@footnote{https://bugs.python.org/issue31247}: xmlrpc.server now explicitly breaks reference cycles when using
sys.exc_info() in code handling exceptions.

@item 
bpo-30102@footnote{https://bugs.python.org/issue30102}: The ssl and hashlib modules now call
OPENSSL_add_all_algorithms_noconf() on OpenSSL < 1.1.0. The function
detects CPU features and enables optimizations on some CPU architectures
such as POWER8. Patch is based on research from Gustavo Serra Scalet.

@item 
bpo-31185@footnote{https://bugs.python.org/issue31185}: Fixed miscellaneous errors in asyncio speedup module.

@item 
bpo-31135@footnote{https://bugs.python.org/issue31135}: ttk: fix the destroy() method of LabeledScale and OptionMenu
classes. Call the parent destroy() method even if the used attribute
doesn’t exist. The LabeledScale.destroy() method now also explicitly
clears label and scale attributes to help the garbage collector to destroy
all widgets.

@item 
bpo-31107@footnote{https://bugs.python.org/issue31107}: Fix @code{copyreg._slotnames()} mangled attribute calculation for
classes whose name begins with an underscore. Patch by Shane Harvey.

@item 
bpo-31061@footnote{https://bugs.python.org/issue31061}: Fixed a crash when using asyncio and threads.

@item 
bpo-30502@footnote{https://bugs.python.org/issue30502}: Fix handling of long oids in ssl.  Based on patch by Christian
Heimes.

@item 
bpo-30119@footnote{https://bugs.python.org/issue30119}: ftplib.FTP.putline() now throws ValueError on commands that
contains CR or LF. Patch by Dong-hee Na.

@item 
bpo-30595@footnote{https://bugs.python.org/issue30595}: multiprocessing.Queue.get() with a timeout now polls its reader
in non-blocking mode if it succeeded to acquire the lock but the acquire
took longer than the timeout.

@item 
bpo-29403@footnote{https://bugs.python.org/issue29403}: Fix @code{unittest.mock}’s autospec to not fail on method-bound
builtin functions.  Patch by Aaron Gallagher.

@item 
bpo-30961@footnote{https://bugs.python.org/issue30961}: Fix decrementing a borrowed reference in tracemalloc.

@item 
bpo-25684@footnote{https://bugs.python.org/issue25684}: Change @code{ttk.OptionMenu} radiobuttons to be unique across
instances of @code{OptionMenu}.

@item 
bpo-30886@footnote{https://bugs.python.org/issue30886}: Fix multiprocessing.Queue.join_thread(): it now waits until the
thread completes, even if the thread was started by the same process which
created the queue.

@item 
bpo-29854@footnote{https://bugs.python.org/issue29854}: Fix segfault in readline when using readline’s history-size
option.  Patch by Nir Soffer.

@item 
bpo-30319@footnote{https://bugs.python.org/issue30319}: socket.close() now ignores ECONNRESET error.

@item 
bpo-30828@footnote{https://bugs.python.org/issue30828}: Fix out of bounds write in
@code{asyncio.CFuture.remove_done_callback()}.

@item 
bpo-30807@footnote{https://bugs.python.org/issue30807}: signal.setitimer() may disable the timer when passed a tiny
value. Tiny values (such as 1e-6) are valid non-zero values for
setitimer(), which is specified as taking microsecond-resolution
intervals. However, on some platform, our conversion routine could convert
1e-6 into a zero interval, therefore disabling the timer instead of
(re-)scheduling it.

@item 
bpo-30441@footnote{https://bugs.python.org/issue30441}: Fix bug when modifying os.environ while iterating over it

@item 
bpo-30532@footnote{https://bugs.python.org/issue30532}: Fix email header value parser dropping folding white space in
certain cases.

@item 
bpo-30879@footnote{https://bugs.python.org/issue30879}: os.listdir() and os.scandir() now emit bytes names when called
with bytes-like argument.

@item 
bpo-30746@footnote{https://bugs.python.org/issue30746}: Prohibited the ‘=’ character in environment variable names in
@code{os.putenv()} and @code{os.spawn*()}.

@item 
bpo-29755@footnote{https://bugs.python.org/issue29755}: Fixed the lgettext() family of functions in the gettext module.
They now always return bytes.
@end itemize

@node Documentation<41>,Tests<38>,Library<42>,Python 3 6 3 release candidate 1
@anchor{whatsnew/changelog id363}@anchor{13b5}
@subsubsection Documentation



@itemize -

@item 
bpo-31294@footnote{https://bugs.python.org/issue31294}: Fix incomplete code snippet in the ZeroMQSocketListener and
ZeroMQSocketHandler examples and adapt them to Python 3.

@item 
bpo-21649@footnote{https://bugs.python.org/issue21649}: Add RFC 7525 and Mozilla server side TLS links to SSL
documentation.

@item 
bpo-30803@footnote{https://bugs.python.org/issue30803}: Clarify doc on truth value testing. Original patch by Peter
Thomassen.
@end itemize

@node Tests<38>,Build<40>,Documentation<41>,Python 3 6 3 release candidate 1
@anchor{whatsnew/changelog id364}@anchor{13b6}
@subsubsection Tests



@itemize -

@item 
bpo-31320@footnote{https://bugs.python.org/issue31320}: Silence traceback in test_ssl

@item 
bpo-25674@footnote{https://bugs.python.org/issue25674}: Remove sha256.tbs-internet.com ssl test

@item 
bpo-30715@footnote{https://bugs.python.org/issue30715}: Address ALPN callback changes for OpenSSL 1.1.0f. The latest
version behaves like OpenSSL 1.0.2 and no longer aborts handshake.

@item 
bpo-30822@footnote{https://bugs.python.org/issue30822}: regrtest: Exclude tzdata from regrtest –all. When running the
test suite using –use=all / -u all, exclude tzdata since it makes
test_datetime too slow (15-20 min on some buildbots) which then times out
on some buildbots. Fix also regrtest command line parser to allow passing
-u extralargefile to run test_zipfile64.
@end itemize

@node Build<40>,Windows<36>,Tests<38>,Python 3 6 3 release candidate 1
@anchor{whatsnew/changelog id365}@anchor{13b7}
@subsubsection Build



@itemize -

@item 
bpo-30854@footnote{https://bugs.python.org/issue30854}: Fix compile error when compiling –without-threads. Patch by
Masayuki Yamamoto.
@end itemize

@node Windows<36>,IDLE<36>,Build<40>,Python 3 6 3 release candidate 1
@anchor{whatsnew/changelog id366}@anchor{13b8}
@subsubsection Windows



@itemize -

@item 
bpo-30389@footnote{https://bugs.python.org/issue30389}: Adds detection of Visual Studio 2017 to distutils on Windows.

@item 
bpo-31340@footnote{https://bugs.python.org/issue31340}: Change to building with MSVC v141 (included with Visual Studio
2017)

@item 
bpo-30581@footnote{https://bugs.python.org/issue30581}: os.cpu_count() now returns the correct number of processors on
Windows when the number of logical processors is greater than 64.

@item 
bpo-30731@footnote{https://bugs.python.org/issue30731}: Add a missing xmlns to python.manifest so that it matches the
schema.
@end itemize

@node IDLE<36>,Tools/Demos<21>,Windows<36>,Python 3 6 3 release candidate 1
@anchor{whatsnew/changelog id367}@anchor{13b9}
@subsubsection IDLE



@itemize -

@item 
bpo-31493@footnote{https://bugs.python.org/issue31493}: IDLE code context – fix code update and font update timers.
Canceling timers prevents a warning message when test_idle completes.

@item 
bpo-31488@footnote{https://bugs.python.org/issue31488}: IDLE - Update non-key options in former extension classes. When
applying configdialog changes, call .reload for each feature class. Change
ParenMatch so updated options affect existing instances attached to
existing editor windows.

@item 
bpo-31477@footnote{https://bugs.python.org/issue31477}: IDLE - Improve rstrip entry in doc. Strip trailing whitespace
strips more than blank spaces.  Multiline string literals are not skipped.

@item 
bpo-31480@footnote{https://bugs.python.org/issue31480}: IDLE - make tests pass with zzdummy extension disabled by
default.

@item 
bpo-31421@footnote{https://bugs.python.org/issue31421}: Document how IDLE runs tkinter programs. IDLE calls tcl/tk
update in the background in order to make live interaction and
experimentation with tkinter applications much easier.

@item 
bpo-31414@footnote{https://bugs.python.org/issue31414}: IDLE – fix tk entry box tests by deleting first. Adding to an
int entry is not the same as deleting and inserting because int(‘’) will
fail.

@item 
bpo-31051@footnote{https://bugs.python.org/issue31051}: Rearrange IDLE configdialog GenPage into Window, Editor, and
Help sections.

@item 
bpo-30617@footnote{https://bugs.python.org/issue30617}: IDLE - Add docstrings and tests for outwin subclass of editor.
Move some data and functions from the class to module level. Patch by
Cheryl Sabella.

@item 
bpo-31287@footnote{https://bugs.python.org/issue31287}: IDLE - Do not modify tkinter.message in test_configdialog.

@item 
bpo-27099@footnote{https://bugs.python.org/issue27099}: Convert IDLE’s built-in ‘extensions’ to regular features. About
10 IDLE features were implemented as supposedly optional extensions. Their
different behavior could be confusing or worse for users and not good for
maintenance. Hence the conversion. The main difference for users is that
user configurable key bindings for builtin features are now handled
uniformly. Now, editing a binding in a keyset only affects its value in
the keyset. All bindings are defined together in the system-specific
default keysets in config-extensions.def. All custom keysets are saved as
a whole in config-extension.cfg.  All take effect as soon as one clicks
Apply or Ok. The affected events are ‘<<force-open-completions>>’,
‘<<expand-word>>’, ‘<<force-open-calltip>>’, ‘<<flash-paren>>’,
‘<<format-paragraph>>’, ‘<<run-module>>’, ‘<<check-module>>’, and
‘<<zoom-height>>’.  Any (global) customizations made before 3.6.3 will not
affect their keyset-specific customization after 3.6.3. and vice versa.
Initial patch by Charles Wohlganger.

@item 
bpo-31206@footnote{https://bugs.python.org/issue31206}: IDLE: Factor HighPage(Frame) class from ConfigDialog. Patch by
Cheryl Sabella.

@item 
bpo-31001@footnote{https://bugs.python.org/issue31001}: Add tests for configdialog highlight tab.  Patch by Cheryl
Sabella.

@item 
bpo-31205@footnote{https://bugs.python.org/issue31205}: IDLE: Factor KeysPage(Frame) class from ConfigDialog.  The
slightly modified tests continue to pass.  Patch by Cheryl Sabella.

@item 
bpo-31130@footnote{https://bugs.python.org/issue31130}: IDLE – stop leaks in test_configdialog. Initial patch by
Victor Stinner.

@item 
bpo-31002@footnote{https://bugs.python.org/issue31002}: Add tests for configdialog keys tab. Patch by Cheryl Sabella.

@item 
bpo-19903@footnote{https://bugs.python.org/issue19903}: IDLE: Calltips use @ref{1c1,,inspect.signature} instead of
@ref{1c2,,inspect.getfullargspec}. This improves calltips for builtins converted to
use Argument Clinic. Patch by Louie Lu.

@item 
bpo-31083@footnote{https://bugs.python.org/issue31083}: IDLE - Add an outline of a TabPage class in configdialog.
Update existing classes to match outline. Initial patch by Cheryl Sabella.

@item 
bpo-31050@footnote{https://bugs.python.org/issue31050}: Factor GenPage(Frame) class from ConfigDialog. The slightly
modified tests continue to pass. Patch by Cheryl Sabella.

@item 
bpo-31004@footnote{https://bugs.python.org/issue31004}: IDLE - Factor FontPage(Frame) class from ConfigDialog. Slightly
modified tests continue to pass. Fix General tests. Patch mostly by Cheryl
Sabella.

@item 
bpo-30781@footnote{https://bugs.python.org/issue30781}: IDLE - Use ttk widgets in ConfigDialog. Patches by Terry Jan
Reedy and Cheryl Sabella.

@item 
bpo-31060@footnote{https://bugs.python.org/issue31060}: IDLE - Finish rearranging methods of ConfigDialog Grouping
methods pertaining to each tab and the buttons will aid writing tests and
improving the tabs and will enable splitting the groups into classes.

@item 
bpo-30853@footnote{https://bugs.python.org/issue30853}: IDLE – Factor a VarTrace class out of ConfigDialog. Instance
tracers manages pairs consisting of a tk variable and a callback function.
When tracing is turned on, setting the variable calls the function.  Test
coverage for the new class is 100%.

@item 
bpo-31003@footnote{https://bugs.python.org/issue31003}: IDLE: Add more tests for General tab.

@item 
bpo-30993@footnote{https://bugs.python.org/issue30993}: IDLE - Improve configdialog font page and tests. In
configdialog: Document causal pathways in create_font_tab docstring.
Simplify some attribute names. Move set_samples calls to var_changed_font
(idea from Cheryl Sabella).  Move related functions to positions after the
create widgets function. In test_configdialog: Fix test_font_set so not
order dependent.  Fix renamed test_indent_scale so it tests the widget.
Adjust tests for movement of set_samples call.  Add tests for load
functions.  Put all font tests in one class and tab indent tests in
another.  Except for two lines, these tests completely cover the related
functions.

@item 
bpo-30981@footnote{https://bugs.python.org/issue30981}: IDLE – Add more configdialog font page tests.

@item 
bpo-28523@footnote{https://bugs.python.org/issue28523}: IDLE: replace ‘colour’ with ‘color’ in configdialog.

@item 
bpo-30917@footnote{https://bugs.python.org/issue30917}: Add tests for idlelib.config.IdleConf. Increase coverage from
46% to 96%. Patch by Louie Lu.

@item 
bpo-30934@footnote{https://bugs.python.org/issue30934}: Document coverage details for idlelib tests. Add section to
idlelib/idle-test/README.txt. Include check that branches are taken both
ways. Exclude IDLE-specific code that does not run during unit tests.

@item 
bpo-30913@footnote{https://bugs.python.org/issue30913}: IDLE: Document ConfigDialog tk Vars, methods, and widgets in
docstrings This will facilitate improving the dialog and splitting up the
class. Original patch by Cheryl Sabella.

@item 
bpo-30899@footnote{https://bugs.python.org/issue30899}: IDLE: Add tests for ConfigParser subclasses in config. Patch by
Louie Lu.

@item 
bpo-30881@footnote{https://bugs.python.org/issue30881}: IDLE: Add docstrings to browser.py. Patch by Cheryl Sabella.

@item 
bpo-30851@footnote{https://bugs.python.org/issue30851}: IDLE: Remove unused variables in configdialog.  One is a
duplicate, one is set but cannot be altered by users. Patch by Cheryl
Sabella.

@item 
bpo-30870@footnote{https://bugs.python.org/issue30870}: IDLE: In Settings dialog, select font with Up, Down keys as
well as mouse. Initial patch by Louie Lu.

@item 
bpo-8231@footnote{https://bugs.python.org/issue8231}: IDLE: call config.IdleConf.GetUserCfgDir only once.

@item 
bpo-30779@footnote{https://bugs.python.org/issue30779}: IDLE: Factor ConfigChanges class from configdialog, put in
config; test. * In config, put dump test code in a function; run it and
unittest in   ‘if __name__ == ‘__main__’. * Add class config.ConfigChanges
based on changes_class_v4.py on bpo issue. * Add class
test_config.ChangesTest, partly using configdialog_tests_v1.py. * Revise
configdialog to use ConfigChanges; see tracker msg297804. * Revise
test_configdialog to match configdialog changes. * Remove configdialog
functions unused or moved to ConfigChanges. Cheryl Sabella contributed
parts of the patch.

@item 
bpo-30777@footnote{https://bugs.python.org/issue30777}: IDLE: configdialog - Add docstrings and fix comments. Patch by
Cheryl Sabella.

@item 
bpo-30495@footnote{https://bugs.python.org/issue30495}: IDLE: Improve textview with docstrings, PEP8 names, and more
tests. Patch by Cheryl Sabella.

@item 
bpo-30723@footnote{https://bugs.python.org/issue30723}: IDLE: Make several improvements to parenmatch. Add ‘parens’
style to highlight both opener and closer. Make ‘default’ style, which is
not default, a synonym for ‘opener’. Make time-delay work the same with
all styles. Add help for config dialog extensions tab, including help for
parenmatch. Add new tests.  Original patch by Charles Wohlganger.

@item 
bpo-30674@footnote{https://bugs.python.org/issue30674}: IDLE: add docstrings to grep module. Patch by Cheryl Sabella

@item 
bpo-21519@footnote{https://bugs.python.org/issue21519}: IDLE’s basic custom key entry dialog now detects duplicates
properly. Original patch by Saimadhav Heblikar.

@item 
bpo-29910@footnote{https://bugs.python.org/issue29910}: IDLE no longer deletes a character after commenting out a
region by a key shortcut.  Add @code{return 'break'} for this and other
potential conflicts between IDLE and default key bindings.

@item 
bpo-30728@footnote{https://bugs.python.org/issue30728}: Review and change idlelib.configdialog names. Lowercase method
and attribute names. Replace ‘colour’ with ‘color’, expand overly cryptic
names, delete unneeded underscores. Replace @code{import *} with specific
imports. Patches by Cheryl Sabella.

@item 
bpo-6739@footnote{https://bugs.python.org/issue6739}: IDLE: Verify user-entered key sequences by trying to bind them
with tk. Add tests for all 3 validation functions. Original patch by G
Polo.  Tests added by Cheryl Sabella.
@end itemize

@node Tools/Demos<21>,,IDLE<36>,Python 3 6 3 release candidate 1
@anchor{whatsnew/changelog id368}@anchor{13ba}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-30983@footnote{https://bugs.python.org/issue30983}: gdb integration commands (py-bt, etc.) work on optimized shared
builds now, too.  
@geindex Python Enhancement Proposals; PEP 523
PEP 523@footnote{https://www.python.org/dev/peps/pep-0523} introduced _PyEval_EvalFrameDefault which
inlines PyEval_EvalFrameEx on non-debug shared builds.  This broke the
ability to use py-bt, py-up, and a few other Python-specific gdb
integrations. The problem is fixed by only looking for
_PyEval_EvalFrameDefault frames in python-gdb.py.  Original patch by Bruno
“Polaco” Penteado.
@end itemize

@node Python 3 6 2 final,Python 3 6 2 release candidate 2,Python 3 6 3 release candidate 1,Changelog
@anchor{whatsnew/changelog python-3-6-2-final}@anchor{13bb}
@subsection Python 3.6.2 final


@emph{Release date: 2017-07-17}

No changes since release candidate 2

@node Python 3 6 2 release candidate 2,Python 3 6 2 release candidate 1,Python 3 6 2 final,Changelog
@anchor{whatsnew/changelog python-3-6-2-release-candidate-2}@anchor{13bc}
@subsection Python 3.6.2 release candidate 2


@emph{Release date: 2017-07-07}

@menu
* Security: Security<24>. 

@end menu

@node Security<24>,,,Python 3 6 2 release candidate 2
@anchor{whatsnew/changelog id369}@anchor{13bd}
@subsubsection Security



@itemize -

@item 
bpo-30730@footnote{https://bugs.python.org/issue30730}: Prevent environment variables injection in subprocess on
Windows.  Prevent passing other environment variables and command
arguments.

@item 
bpo-30694@footnote{https://bugs.python.org/issue30694}: Upgrade expat copy from 2.2.0 to 2.2.1 to get fixes of multiple
security vulnerabilities including: CVE-2017-9233 (External entity
infinite loop DoS), CVE-2016-9063 (Integer overflow, re-fix),
CVE-2016-0718 (Fix regression bugs from 2.2.0’s fix to CVE-2016-0718) and
CVE-2012-0876 (Counter hash flooding with SipHash). Note: the
CVE-2016-5300 (Use os-specific entropy sources like getrandom) doesn’t
impact Python, since Python already gets entropy from the OS to set the
expat secret using @code{XML_SetHashSalt()}.

@item 
bpo-30500@footnote{https://bugs.python.org/issue30500}: Fix urllib.parse.splithost() to correctly parse fragments. For
example, @code{splithost('//127.0.0.1#@@evil.com/')} now correctly returns the
@code{127.0.0.1} host, instead of treating @code{@@evil.com} as the host in an
authentication (@code{login@@host}).
@end itemize

@node Python 3 6 2 release candidate 1,Python 3 6 1 final,Python 3 6 2 release candidate 2,Changelog
@anchor{whatsnew/changelog python-3-6-2-release-candidate-1}@anchor{13be}
@subsection Python 3.6.2 release candidate 1


@emph{Release date: 2017-06-17}

@menu
* Core and Builtins: Core and Builtins<41>. 
* Library: Library<43>. 
* Security: Security<25>. 
* Library: Library<44>. 
* IDLE: IDLE<37>. 
* C API: C API<36>. 
* Build: Build<41>. 
* Documentation: Documentation<42>. 
* Tools/Demos: Tools/Demos<22>. 
* Tests: Tests<39>. 
* Windows: Windows<37>. 

@end menu

@node Core and Builtins<41>,Library<43>,,Python 3 6 2 release candidate 1
@anchor{whatsnew/changelog id370}@anchor{13bf}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-30682@footnote{https://bugs.python.org/issue30682}: Removed a too-strict assertion that failed for certain
f-strings, such as eval(“f’\n’”) and eval(“f’\r’”).

@item 
bpo-30604@footnote{https://bugs.python.org/issue30604}: Move co_extra_freefuncs to not be per-thread to avoid crashes

@item 
bpo-29104@footnote{https://bugs.python.org/issue29104}: Fixed parsing backslashes in f-strings.

@item 
bpo-27945@footnote{https://bugs.python.org/issue27945}: Fixed various segfaults with dict when input collections are
mutated during searching, inserting or comparing.  Based on patches by
Duane Griffin and Tim Mitchell.

@item 
bpo-25794@footnote{https://bugs.python.org/issue25794}: Fixed type.__setattr__() and type.__delattr__() for
non-interned attribute names.  Based on patch by Eryk Sun.

@item 
bpo-30039@footnote{https://bugs.python.org/issue30039}: If a KeyboardInterrupt happens when the interpreter is in the
middle of resuming a chain of nested ‘yield from’ or ‘await’ calls, it’s
now correctly delivered to the innermost frame.

@item 
bpo-12414@footnote{https://bugs.python.org/issue12414}: sys.getsizeof() on a code object now returns the sizes which
includes the code struct and sizes of objects which it references. Patch
by Dong-hee Na.

@item 
bpo-29949@footnote{https://bugs.python.org/issue29949}: Fix memory usage regression of set and frozenset object.

@item 
bpo-29935@footnote{https://bugs.python.org/issue29935}: Fixed error messages in the index() method of tuple, list and
deque when pass indices of wrong type.

@item 
bpo-29859@footnote{https://bugs.python.org/issue29859}: Show correct error messages when any of the pthread_* calls in
thread_pthread.h fails.

@item 
bpo-28876@footnote{https://bugs.python.org/issue28876}: @code{bool(range)} works even if @code{len(range)} raises
@ref{347,,OverflowError}.

@item 
bpo-29600@footnote{https://bugs.python.org/issue29600}: Fix wrapping coroutine return values in StopIteration.

@item 
bpo-28856@footnote{https://bugs.python.org/issue28856}: Fix an oversight that %b format for bytes should support
objects follow the buffer protocol.

@item 
bpo-29714@footnote{https://bugs.python.org/issue29714}: Fix a regression that bytes format may fail when containing
zero bytes inside.

@item 
bpo-29478@footnote{https://bugs.python.org/issue29478}: If max_line_length=None is specified while using the Compat32
policy, it is no longer ignored.  Patch by Mircea Cosbuc.
@end itemize

@node Library<43>,Security<25>,Core and Builtins<41>,Python 3 6 2 release candidate 1
@anchor{whatsnew/changelog id371}@anchor{13c0}
@subsubsection Library



@itemize -

@item 
bpo-30616@footnote{https://bugs.python.org/issue30616}: Functional API of enum allows to create empty enums. Patched by
Dong-hee Na

@item 
bpo-30038@footnote{https://bugs.python.org/issue30038}: Fix race condition between signal delivery and wakeup file
descriptor. Patch by Nathaniel Smith.

@item 
bpo-23894@footnote{https://bugs.python.org/issue23894}: lib2to3 now recognizes @code{rb'...'} and @code{f'...'} strings.

@item 
bpo-23890@footnote{https://bugs.python.org/issue23890}: unittest.TestCase.assertRaises() now manually breaks a
reference cycle to not keep objects alive longer than expected.

@item 
bpo-30149@footnote{https://bugs.python.org/issue30149}: inspect.signature() now supports callables with
variable-argument parameters wrapped with partialmethod. Patch by Dong-hee
Na.

@item 
bpo-30645@footnote{https://bugs.python.org/issue30645}: Fix path calculation in imp.load_package(), fixing it for cases
when a package is only shipped with bytecodes. Patch by Alexandru
Ardelean.

@item 
bpo-29931@footnote{https://bugs.python.org/issue29931}: Fixed comparison check for ipaddress.ip_interface objects.
Patch by Sanjay Sundaresan.

@item 
bpo-30605@footnote{https://bugs.python.org/issue30605}: re.compile() no longer raises a BytesWarning when compiling a
bytes instance with misplaced inline modifier.  Patch by Roy Williams.
@end itemize

@node Security<25>,Library<44>,Library<43>,Python 3 6 2 release candidate 1
@anchor{whatsnew/changelog id372}@anchor{13c1}
@subsubsection Security



@itemize -

@item 
bpo-29591@footnote{https://bugs.python.org/issue29591}: Update expat copy from 2.1.1 to 2.2.0 to get fixes of
CVE-2016-0718 and CVE-2016-4472. See
@indicateurl{https://sourceforge.net/p/expat/bugs/537/} for more information.
@end itemize

@node Library<44>,IDLE<37>,Security<25>,Python 3 6 2 release candidate 1
@anchor{whatsnew/changelog id373}@anchor{13c2}
@subsubsection Library



@itemize -

@item 
bpo-24484@footnote{https://bugs.python.org/issue24484}: Avoid race condition in multiprocessing cleanup (#2159)

@item 
bpo-28994@footnote{https://bugs.python.org/issue28994}: The traceback no longer displayed for SystemExit raised in a
callback registered by atexit.

@item 
bpo-30508@footnote{https://bugs.python.org/issue30508}: Don’t log exceptions if Task/Future “cancel()” method was
called.

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Updates to typing module: Add generic AsyncContextManager, add
support for ContextManager on all versions. Original PRs by Jelle Zijlstra
and Ivan Levkivskyi

@item 
bpo-29870@footnote{https://bugs.python.org/issue29870}: Fix ssl sockets leaks when connection is aborted in asyncio/ssl
implementation. Patch by Michaël Sghaïer.

@item 
bpo-29743@footnote{https://bugs.python.org/issue29743}: Closing transport during handshake process leaks open socket.
Patch by Nikolay Kim

@item 
bpo-27585@footnote{https://bugs.python.org/issue27585}: Fix waiter cancellation in asyncio.Lock. Patch by Mathieu
Sornay.

@item 
bpo-30418@footnote{https://bugs.python.org/issue30418}: On Windows, subprocess.Popen.communicate() now also ignore
EINVAL on stdin.write() if the child process is still running but closed
the pipe.

@item 
bpo-29822@footnote{https://bugs.python.org/issue29822}: inspect.isabstract() now works during __init_subclass__.  Patch
by Nate Soares.

@item 
bpo-29581@footnote{https://bugs.python.org/issue29581}: ABCMeta.__new__ now accepts @code{**kwargs}, allowing abstract
base classes to use keyword parameters in __init_subclass__. Patch by Nate
Soares.

@item 
bpo-30557@footnote{https://bugs.python.org/issue30557}: faulthandler now correctly filters and displays exception codes
on Windows

@item 
bpo-30378@footnote{https://bugs.python.org/issue30378}: Fix the problem that logging.handlers.SysLogHandler cannot
handle IPv6 addresses.

@item 
bpo-29960@footnote{https://bugs.python.org/issue29960}: Preserve generator state when _random.Random.setstate() raises
an exception. Patch by Bryan Olson.

@item 
bpo-30414@footnote{https://bugs.python.org/issue30414}: multiprocessing.Queue._feed background running thread do not
break from main loop on exception.

@item 
bpo-30003@footnote{https://bugs.python.org/issue30003}: Fix handling escape characters in HZ codec.  Based on patch by
Ma Lin.

@item 
bpo-30301@footnote{https://bugs.python.org/issue30301}: Fix AttributeError when using SimpleQueue.empty() under @emph{spawn}
and @emph{forkserver} start methods.

@item 
bpo-30329@footnote{https://bugs.python.org/issue30329}: imaplib and poplib now catch the Windows socket WSAEINVAL error
(code 10022) on shutdown(SHUT_RDWR): An invalid operation was attempted.
This error occurs sometimes on SSL connections.

@item 
bpo-30375@footnote{https://bugs.python.org/issue30375}: Warnings emitted when compile a regular expression now always
point to the line in the user code.  Previously they could point into
inners of the re module if emitted from inside of groups or conditionals.

@item 
bpo-30048@footnote{https://bugs.python.org/issue30048}: Fixed @code{Task.cancel()} can be ignored when the task is running
coroutine and the coroutine returned without any more @code{await}.

@item 
bpo-30266@footnote{https://bugs.python.org/issue30266}: contextlib.AbstractContextManager now supports
anti-registration by setting __enter__ = None or __exit__ = None,
following the pattern introduced in bpo-25958@footnote{https://bugs.python.org/issue25958}. Patch by Jelle Zijlstra.

@item 
bpo-30298@footnote{https://bugs.python.org/issue30298}: Weaken the condition of deprecation warnings for inline
modifiers. Now allowed several subsequential inline modifiers at the start
of the pattern (e.g. @code{'(?i)(?s)...'}).  In verbose mode whitespaces and
comments now are allowed before and between inline modifiers (e.g. @code{'(?x)
(?i) (?s)...'}).

@item 
bpo-29990@footnote{https://bugs.python.org/issue29990}: Fix range checking in GB18030 decoder.  Original patch by Ma
Lin.

@item 
bpo-26293@footnote{https://bugs.python.org/issue26293}: Change resulted because of zipfile breakage. (See also:
bpo-29094@footnote{https://bugs.python.org/issue29094})

@item 
bpo-30243@footnote{https://bugs.python.org/issue30243}: Removed the __init__ methods of _json’s scanner and encoder.
Misusing them could cause memory leaks or crashes.  Now scanner and
encoder objects are completely initialized in the __new__ methods.

@item 
bpo-30185@footnote{https://bugs.python.org/issue30185}: Avoid KeyboardInterrupt tracebacks in forkserver helper process
when Ctrl-C is received.

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Various updates to typing module: add typing.NoReturn type, use
WrapperDescriptorType, minor bug-fixes.  Original PRs by Jim
Fasarakis-Hilliard and Ivan Levkivskyi.

@item 
bpo-30205@footnote{https://bugs.python.org/issue30205}: Fix getsockname() for unbound AF_UNIX sockets on Linux.

@item 
bpo-30070@footnote{https://bugs.python.org/issue30070}: Fixed leaks and crashes in errors handling in the parser
module.

@item 
bpo-30061@footnote{https://bugs.python.org/issue30061}: Fixed crashes in IOBase methods __next__() and readlines() when
readline() or __next__() respectively return non-sizeable object. Fixed
possible other errors caused by not checking results of PyObject_Size(),
PySequence_Size(), or PyMapping_Size().

@item 
bpo-30017@footnote{https://bugs.python.org/issue30017}: Allowed calling the close() method of the zip entry writer
object multiple times.  Writing to a closed writer now always produces a
ValueError.

@item 
bpo-30068@footnote{https://bugs.python.org/issue30068}: _io._IOBase.readlines will check if it’s closed first when hint
is present.

@item 
bpo-29694@footnote{https://bugs.python.org/issue29694}: Fixed race condition in pathlib mkdir with flags parents=True.
Patch by Armin Rigo.

@item 
bpo-29692@footnote{https://bugs.python.org/issue29692}: Fixed arbitrary unchaining of RuntimeError exceptions in
contextlib.contextmanager.  Patch by Siddharth Velankar.

@item 
bpo-29998@footnote{https://bugs.python.org/issue29998}: Pickling and copying ImportError now preserves name and path
attributes.

@item 
bpo-29953@footnote{https://bugs.python.org/issue29953}: Fixed memory leaks in the replace() method of datetime and time
objects when pass out of bound fold argument.

@item 
bpo-29942@footnote{https://bugs.python.org/issue29942}: Fix a crash in itertools.chain.from_iterable when encountering
long runs of empty iterables.

@item 
bpo-27863@footnote{https://bugs.python.org/issue27863}: Fixed multiple crashes in ElementTree caused by race conditions
and wrong types.

@item 
bpo-28699@footnote{https://bugs.python.org/issue28699}: Fixed a bug in pools in multiprocessing.pool that raising an
exception at the very first of an iterable may swallow the exception or
make the program hang. Patch by Davin Potts and Xiang Zhang.

@item 
bpo-25803@footnote{https://bugs.python.org/issue25803}: Avoid incorrect errors raised by Path.mkdir(exist_ok=True) when
the OS gives priority to errors such as EACCES over EEXIST.

@item 
bpo-29861@footnote{https://bugs.python.org/issue29861}: Release references to tasks, their arguments and their results
as soon as they are finished in multiprocessing.Pool.

@item 
bpo-29884@footnote{https://bugs.python.org/issue29884}: faulthandler: Restore the old sigaltstack during teardown.
Patch by Christophe Zeitouny.

@item 
bpo-25455@footnote{https://bugs.python.org/issue25455}: Fixed crashes in repr of recursive buffered file-like objects.

@item 
bpo-29800@footnote{https://bugs.python.org/issue29800}: Fix crashes in partial.__repr__ if the keys of partial.keywords
are not strings.  Patch by Michael Seifert.

@item 
bpo-29742@footnote{https://bugs.python.org/issue29742}: get_extra_info() raises exception if get called on closed ssl
transport. Patch by Nikolay Kim.

@item 
bpo-8256@footnote{https://bugs.python.org/issue8256}: Fixed possible failing or crashing input() if attributes
“encoding” or “errors” of sys.stdin or sys.stdout are not set or are not
strings.

@item 
bpo-28298@footnote{https://bugs.python.org/issue28298}: Fix a bug that prevented array ‘Q’, ‘L’ and ‘I’ from accepting
big intables (objects that have __int__) as elements.  Patch by Oren
Milman.

@item 
bpo-28231@footnote{https://bugs.python.org/issue28231}: The zipfile module now accepts path-like objects for external
paths.

@item 
bpo-26915@footnote{https://bugs.python.org/issue26915}: index() and count() methods of collections.abc.Sequence now
check identity before checking equality when do comparisons.

@item 
bpo-29615@footnote{https://bugs.python.org/issue29615}: SimpleXMLRPCDispatcher no longer chains KeyError (or any other
exception) to exception(s) raised in the dispatched methods. Patch by Petr
Motejlek.

@item 
bpo-30177@footnote{https://bugs.python.org/issue30177}: path.resolve(strict=False) no longer cuts the path after the
first element not present in the filesystem.  Patch by Antoine Pietri.
@end itemize

@node IDLE<37>,C API<36>,Library<44>,Python 3 6 2 release candidate 1
@anchor{whatsnew/changelog id374}@anchor{13c3}
@subsubsection IDLE



@itemize -

@item 
bpo-15786@footnote{https://bugs.python.org/issue15786}: Fix several problems with IDLE’s autocompletion box. The
following should now work: clicking on selection box items; using the
scrollbar; selecting an item by hitting Return. Hangs on MacOSX should no
longer happen. Patch by Louie Lu.

@item 
bpo-25514@footnote{https://bugs.python.org/issue25514}: Add doc subsubsection about IDLE failure to start. Popup
no-connection message directs users to this section.

@item 
bpo-30642@footnote{https://bugs.python.org/issue30642}: Fix reference leaks in IDLE tests. Patches by Louie Lu and
Terry Jan Reedy.

@item 
bpo-30495@footnote{https://bugs.python.org/issue30495}: Add docstrings for textview.py and use PEP8 names. Patches by
Cheryl Sabella and Terry Jan Reedy.

@item 
bpo-30290@footnote{https://bugs.python.org/issue30290}: Help-about: use pep8 names and add tests. Increase coverage to
100%. Patches by Louie Lu, Cheryl Sabella, and Terry Jan Reedy.

@item 
bpo-30303@footnote{https://bugs.python.org/issue30303}: Add _utest option to textview; add new tests. Increase coverage
to 100%. Patches by Louie Lu and Terry Jan Reedy.
@end itemize

@node C API<36>,Build<41>,IDLE<37>,Python 3 6 2 release candidate 1
@anchor{whatsnew/changelog id375}@anchor{13c4}
@subsubsection C API



@itemize -

@item 
bpo-27867@footnote{https://bugs.python.org/issue27867}: Function PySlice_GetIndicesEx() no longer replaced with a macro
if Py_LIMITED_API is not set.
@end itemize

@node Build<41>,Documentation<42>,C API<36>,Python 3 6 2 release candidate 1
@anchor{whatsnew/changelog id376}@anchor{13c5}
@subsubsection Build



@itemize -

@item 
bpo-29941@footnote{https://bugs.python.org/issue29941}: Add @code{--with-assertions} configure flag to explicitly enable C
@code{assert()} checks. Defaults to off. @code{--with-pydebug} implies
@code{--with-assertions}.

@item 
bpo-28787@footnote{https://bugs.python.org/issue28787}: Fix out-of-tree builds of Python when configured with
@code{--with--dtrace}.

@item 
bpo-29243@footnote{https://bugs.python.org/issue29243}: Prevent unnecessary rebuilding of Python during @code{make test},
@code{make install} and some other make targets when configured with
@code{--enable-optimizations}.

@item 
bpo-23404@footnote{https://bugs.python.org/issue23404}: Don’t regenerate generated files based on file modification
time anymore: the action is now explicit. Replace @code{make touch} with
@code{make regen-all}.

@item 
bpo-29643@footnote{https://bugs.python.org/issue29643}: Fix @code{--enable-optimization} didn’t work.
@end itemize

@node Documentation<42>,Tools/Demos<22>,Build<41>,Python 3 6 2 release candidate 1
@anchor{whatsnew/changelog id377}@anchor{13c6}
@subsubsection Documentation



@itemize -

@item 
bpo-30176@footnote{https://bugs.python.org/issue30176}: Add missing attribute related constants in curses
documentation.

@item 
bpo-30052@footnote{https://bugs.python.org/issue30052}: the link targets for @ref{2f0,,bytes()} and @ref{2f1,,bytearray()} are
now their respective type definitions, rather than the corresponding
builtin function entries. Use @ref{137e,,bytes} and
@ref{137f,,bytearray} to reference the latter. In order to
ensure this and future cross-reference updates are applied automatically,
the daily documentation builds now disable the default output caching
features in Sphinx.

@item 
bpo-26985@footnote{https://bugs.python.org/issue26985}: Add missing info of code object in inspect documentation.
@end itemize

@node Tools/Demos<22>,Tests<39>,Documentation<42>,Python 3 6 2 release candidate 1
@anchor{whatsnew/changelog id378}@anchor{13c7}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-29367@footnote{https://bugs.python.org/issue29367}: python-gdb.py now supports also @code{method-wrapper}
(@code{wrapperobject}) objects.
@end itemize

@node Tests<39>,Windows<37>,Tools/Demos<22>,Python 3 6 2 release candidate 1
@anchor{whatsnew/changelog id379}@anchor{13c8}
@subsubsection Tests



@itemize -

@item 
bpo-30357@footnote{https://bugs.python.org/issue30357}: test_thread: setUp() now uses support.threading_setup() and
support.threading_cleanup() to wait until threads complete to avoid random
side effects on following tests. Initial patch written by Grzegorz
Grzywacz.

@item 
bpo-30197@footnote{https://bugs.python.org/issue30197}: Enhanced functions swap_attr() and swap_item() in the
test.support module. They now work when delete replaced attribute or item
inside the with statement.  The old value of the attribute or item (or
None if it doesn’t exist) now will be assigned to the target of the “as”
clause, if there is one.
@end itemize

@node Windows<37>,,Tests<39>,Python 3 6 2 release candidate 1
@anchor{whatsnew/changelog id380}@anchor{13c9}
@subsubsection Windows



@itemize -

@item 
bpo-30687@footnote{https://bugs.python.org/issue30687}: Locate msbuild.exe on Windows when building rather than
vcvarsall.bat

@item 
bpo-30450@footnote{https://bugs.python.org/issue30450}: The build process on Windows no longer depends on Subversion,
instead pulling external code from GitHub via a Python script.  If Python
3.6 is not found on the system (via @code{py -3.6}), NuGet is used to
download a copy of 32-bit Python.
@end itemize

@node Python 3 6 1 final,Python 3 6 1 release candidate 1,Python 3 6 2 release candidate 1,Changelog
@anchor{whatsnew/changelog python-3-6-1-final}@anchor{13ca}
@subsection Python 3.6.1 final


@emph{Release date: 2017-03-21}

@menu
* Core and Builtins: Core and Builtins<42>. 
* Build: Build<42>. 

@end menu

@node Core and Builtins<42>,Build<42>,,Python 3 6 1 final
@anchor{whatsnew/changelog id381}@anchor{13cb}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-29723@footnote{https://bugs.python.org/issue29723}: The @code{sys.path[0]} initialization change for bpo-29139@footnote{https://bugs.python.org/issue29139} caused
a regression by revealing an inconsistency in how sys.path is initialized
when executing @code{__main__} from a zipfile, directory, or other import
location. The interpreter now consistently avoids ever adding the import
location’s parent directory to @code{sys.path}, and ensures no other
@code{sys.path} entries are inadvertently modified when inserting the import
location named on the command line.
@end itemize

@node Build<42>,,Core and Builtins<42>,Python 3 6 1 final
@anchor{whatsnew/changelog id382}@anchor{13cc}
@subsubsection Build



@itemize -

@item 
bpo-27593@footnote{https://bugs.python.org/issue27593}: fix format of git information used in sys.version

@item 
Fix incompatible comment in python.h
@end itemize

@node Python 3 6 1 release candidate 1,Python 3 6 0 final,Python 3 6 1 final,Changelog
@anchor{whatsnew/changelog python-3-6-1-release-candidate-1}@anchor{13cd}
@subsection Python 3.6.1 release candidate 1


@emph{Release date: 2017-03-04}

@menu
* Core and Builtins: Core and Builtins<43>. 
* Library: Library<45>. 
* IDLE: IDLE<38>. 
* Windows: Windows<38>. 
* C API: C API<37>. 
* Documentation: Documentation<43>. 
* Tests: Tests<40>. 
* Build: Build<43>. 

@end menu

@node Core and Builtins<43>,Library<45>,,Python 3 6 1 release candidate 1
@anchor{whatsnew/changelog id383}@anchor{13ce}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-28893@footnote{https://bugs.python.org/issue28893}: Set correct __cause__ for errors about invalid awaitables
returned from __aiter__ and __anext__.

@item 
bpo-29683@footnote{https://bugs.python.org/issue29683}: Fixes to memory allocation in _PyCode_SetExtra.  Patch by Brian
Coleman.

@item 
bpo-29684@footnote{https://bugs.python.org/issue29684}: Fix minor regression of PyEval_CallObjectWithKeywords. It
should raise TypeError when kwargs is not a dict.  But it might cause segv
when args=NULL and kwargs is not a dict.

@item 
bpo-28598@footnote{https://bugs.python.org/issue28598}: Support __rmod__ for subclasses of str being called before
str.__mod__. Patch by Martijn Pieters.

@item 
bpo-29607@footnote{https://bugs.python.org/issue29607}: Fix stack_effect computation for CALL_FUNCTION_EX. Patch by
Matthieu Dartiailh.

@item 
bpo-29602@footnote{https://bugs.python.org/issue29602}: Fix incorrect handling of signed zeros in complex constructor
for complex subclasses and for inputs having a __complex__ method. Patch
by Serhiy Storchaka.

@item 
bpo-29347@footnote{https://bugs.python.org/issue29347}: Fixed possibly dereferencing undefined pointers when creating
weakref objects.

@item 
bpo-29438@footnote{https://bugs.python.org/issue29438}: Fixed use-after-free problem in key sharing dict.

@item 
bpo-29319@footnote{https://bugs.python.org/issue29319}: Prevent RunMainFromImporter overwriting sys.path[0].

@item 
bpo-29337@footnote{https://bugs.python.org/issue29337}: Fixed possible BytesWarning when compare the code objects.
Warnings could be emitted at compile time.

@item 
bpo-29327@footnote{https://bugs.python.org/issue29327}: Fixed a crash when pass the iterable keyword argument to
sorted().

@item 
bpo-29034@footnote{https://bugs.python.org/issue29034}: Fix memory leak and use-after-free in os module
(path_converter).

@item 
bpo-29159@footnote{https://bugs.python.org/issue29159}: Fix regression in bytes(x) when x.__index__() raises Exception.

@item 
bpo-28932@footnote{https://bugs.python.org/issue28932}: Do not include <sys/random.h> if it does not exist.

@item 
bpo-25677@footnote{https://bugs.python.org/issue25677}: Correct the positioning of the syntax error caret for indented
blocks. Based on patch by Michael Layzell.

@item 
bpo-29000@footnote{https://bugs.python.org/issue29000}: Fixed bytes formatting of octals with zero padding in alternate
form.

@item 
bpo-26919@footnote{https://bugs.python.org/issue26919}: On Android, operating system data is now always encoded/decoded
to/from UTF-8, instead of the locale encoding to avoid inconsistencies
with os.fsencode() and os.fsdecode() which are already using UTF-8.

@item 
bpo-28991@footnote{https://bugs.python.org/issue28991}: functools.lru_cache() was susceptible to an obscure reentrancy
bug triggerable by a monkey-patched len() function.

@item 
bpo-28739@footnote{https://bugs.python.org/issue28739}: f-string expressions are no longer accepted as docstrings and
by ast.literal_eval() even if they do not include expressions.

@item 
bpo-28512@footnote{https://bugs.python.org/issue28512}: Fixed setting the offset attribute of SyntaxError by
PyErr_SyntaxLocationEx() and PyErr_SyntaxLocationObject().

@item 
bpo-28918@footnote{https://bugs.python.org/issue28918}: Fix the cross compilation of xxlimited when Python has been
built with Py_DEBUG defined.

@item 
bpo-28731@footnote{https://bugs.python.org/issue28731}: Optimize _PyDict_NewPresized() to create correct size dict.
Improve speed of dict literal with constant keys up to 30%.
@end itemize

@node Library<45>,IDLE<38>,Core and Builtins<43>,Python 3 6 1 release candidate 1
@anchor{whatsnew/changelog id384}@anchor{13cf}
@subsubsection Library



@itemize -

@item 
bpo-29169@footnote{https://bugs.python.org/issue29169}: Update zlib to 1.2.11.

@item 
bpo-29623@footnote{https://bugs.python.org/issue29623}: Allow use of path-like object as a single argument in
ConfigParser.read(). Patch by David Ellis.

@item 
bpo-28963@footnote{https://bugs.python.org/issue28963}: Fix out of bound iteration in
asyncio.Future.remove_done_callback implemented in C.

@item 
bpo-29704@footnote{https://bugs.python.org/issue29704}: asyncio.subprocess.SubprocessStreamProtocol no longer closes
before all pipes are closed.

@item 
bpo-29271@footnote{https://bugs.python.org/issue29271}: Fix Task.current_task and Task.all_tasks implemented in C to
accept None argument as their pure Python implementation.

@item 
bpo-29703@footnote{https://bugs.python.org/issue29703}: Fix asyncio to support instantiation of new event loops in
child processes.

@item 
bpo-29376@footnote{https://bugs.python.org/issue29376}: Fix assertion error in threading._DummyThread.is_alive().

@item 
bpo-28624@footnote{https://bugs.python.org/issue28624}: Add a test that checks that cwd parameter of Popen() accepts
PathLike objects.  Patch by Sayan Chowdhury.

@item 
bpo-28518@footnote{https://bugs.python.org/issue28518}: Start a transaction implicitly before a DML statement. Patch by
Aviv Palivoda.

@item 
bpo-29532@footnote{https://bugs.python.org/issue29532}: Altering a kwarg dictionary passed to functools.partial() no
longer affects a partial object after creation.

@item 
bpo-29110@footnote{https://bugs.python.org/issue29110}: Fix file object leak in aifc.open() when file is given as a
filesystem path and is not in valid AIFF format. Patch by Anthony Zhang.

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Various updates to typing module: typing.Counter,
typing.ChainMap, improved ABC caching, etc. Original PRs by Jelle
Zijlstra, Ivan Levkivskyi, Manuel Krebber, and Łukasz Langa.

@item 
bpo-29100@footnote{https://bugs.python.org/issue29100}: Fix datetime.fromtimestamp() regression introduced in Python
3.6.0: check minimum and maximum years.

@item 
bpo-29519@footnote{https://bugs.python.org/issue29519}: Fix weakref spewing exceptions during interpreter shutdown when
used with a rare combination of multiprocessing and custom codecs.

@item 
bpo-29416@footnote{https://bugs.python.org/issue29416}: Prevent infinite loop in pathlib.Path.mkdir

@item 
bpo-29444@footnote{https://bugs.python.org/issue29444}: Fixed out-of-bounds buffer access in the group() method of the
match object. Based on patch by WGH.

@item 
bpo-29335@footnote{https://bugs.python.org/issue29335}: Fix subprocess.Popen.wait() when the child process has exited
to a stopped instead of terminated state (ex: when under ptrace).

@item 
bpo-29290@footnote{https://bugs.python.org/issue29290}: Fix a regression in argparse that help messages would wrap at
non-breaking spaces.

@item 
bpo-28735@footnote{https://bugs.python.org/issue28735}: Fixed the comparison of mock.MagickMock with mock.ANY.

@item 
bpo-29316@footnote{https://bugs.python.org/issue29316}: Restore the provisional status of typing module, add
corresponding note to documentation. Patch by Ivan L.

@item 
bpo-29219@footnote{https://bugs.python.org/issue29219}: Fixed infinite recursion in the repr of uninitialized
ctypes.CDLL instances.

@item 
bpo-29011@footnote{https://bugs.python.org/issue29011}: Fix an important omission by adding Deque to the typing module.

@item 
bpo-28969@footnote{https://bugs.python.org/issue28969}: Fixed race condition in C implementation of
functools.lru_cache. KeyError could be raised when cached function with
full cache was simultaneously called from different threads with the same
uncached arguments.

@item 
bpo-29142@footnote{https://bugs.python.org/issue29142}: In urllib.request, suffixes in no_proxy environment variable
with leading dots could match related hostnames again (e.g. .b.c matches
a.b.c). Patch by Milan Oberkirch.

@item 
bpo-28961@footnote{https://bugs.python.org/issue28961}: Fix unittest.mock._Call helper: don’t ignore the name parameter
anymore. Patch written by Jiajun Huang.

@item 
bpo-29203@footnote{https://bugs.python.org/issue29203}: functools.lru_cache() now respects 
@geindex Python Enhancement Proposals; PEP 468
PEP 468@footnote{https://www.python.org/dev/peps/pep-0468} and preserves the
order of keyword arguments.  f(a=1, b=2) is now cached separately from
f(b=2, a=1) since both calls could potentially give different results.

@item 
bpo-15812@footnote{https://bugs.python.org/issue15812}: inspect.getframeinfo() now correctly shows the first line of a
context. Patch by Sam Breese.

@item 
bpo-29094@footnote{https://bugs.python.org/issue29094}: Offsets in a ZIP file created with extern file object and modes
“w” and “x” now are relative to the start of the file.

@item 
bpo-29085@footnote{https://bugs.python.org/issue29085}: Allow random.Random.seed() to use high quality OS randomness
rather than the pid and time.

@item 
bpo-29061@footnote{https://bugs.python.org/issue29061}: Fixed bug in secrets.randbelow() which would hang when given a
negative input.  Patch by Brendan Donegan.

@item 
bpo-29079@footnote{https://bugs.python.org/issue29079}: Prevent infinite loop in pathlib.resolve() on Windows

@item 
bpo-13051@footnote{https://bugs.python.org/issue13051}: Fixed recursion errors in large or resized
curses.textpad.Textbox.  Based on patch by Tycho Andersen.

@item 
bpo-29119@footnote{https://bugs.python.org/issue29119}: Fix weakrefs in the pure python version of
collections.OrderedDict move_to_end() method. Contributed by Andra
Bogildea.

@item 
bpo-9770@footnote{https://bugs.python.org/issue9770}: curses.ascii predicates now work correctly with negative
integers.

@item 
bpo-28427@footnote{https://bugs.python.org/issue28427}: old keys should not remove new values from WeakValueDictionary
when collecting from another thread.

@item 
bpo-28923@footnote{https://bugs.python.org/issue28923}: Remove editor artifacts from Tix.py.

@item 
bpo-29055@footnote{https://bugs.python.org/issue29055}: Neaten-up empty population error on random.choice() by
suppressing the upstream exception.

@item 
bpo-28871@footnote{https://bugs.python.org/issue28871}: Fixed a crash when deallocate deep ElementTree.

@item 
bpo-19542@footnote{https://bugs.python.org/issue19542}: Fix bugs in WeakValueDictionary.setdefault() and
WeakValueDictionary.pop() when a GC collection happens in another thread.

@item 
bpo-20191@footnote{https://bugs.python.org/issue20191}: Fixed a crash in resource.prlimit() when passing a sequence
that doesn’t own its elements as limits.

@item 
bpo-28779@footnote{https://bugs.python.org/issue28779}: multiprocessing.set_forkserver_preload() would crash the
forkserver process if a preloaded module instantiated some multiprocessing
objects such as locks.

@item 
bpo-28847@footnote{https://bugs.python.org/issue28847}: dbm.dumb now supports reading read-only files and no longer
writes the index file when it is not changed.

@item 
bpo-26937@footnote{https://bugs.python.org/issue26937}: The chown() method of the tarfile.TarFile class does not fail
now when the grp module cannot be imported, as for example on Android
platforms.
@end itemize

@node IDLE<38>,Windows<38>,Library<45>,Python 3 6 1 release candidate 1
@anchor{whatsnew/changelog id385}@anchor{13d0}
@subsubsection IDLE



@itemize -

@item 
bpo-29071@footnote{https://bugs.python.org/issue29071}: IDLE colors f-string prefixes (but not invalid ur prefixes).

@item 
bpo-28572@footnote{https://bugs.python.org/issue28572}: Add 10% to coverage of IDLE’s test_configdialog. Update and
augment description of the configuration system.
@end itemize

@node Windows<38>,C API<37>,IDLE<38>,Python 3 6 1 release candidate 1
@anchor{whatsnew/changelog id386}@anchor{13d1}
@subsubsection Windows



@itemize -

@item 
bpo-29579@footnote{https://bugs.python.org/issue29579}: Removes readme.txt from the installer

@item 
bpo-29326@footnote{https://bugs.python.org/issue29326}: Ignores blank lines in ._pth files (Patch by Alexey Izbyshev)

@item 
bpo-28164@footnote{https://bugs.python.org/issue28164}: Correctly handle special console filenames (patch by Eryk Sun)

@item 
bpo-29409@footnote{https://bugs.python.org/issue29409}: Implement 
@geindex Python Enhancement Proposals; PEP 529
PEP 529@footnote{https://www.python.org/dev/peps/pep-0529} for io.FileIO (Patch by Eryk Sun)

@item 
bpo-29392@footnote{https://bugs.python.org/issue29392}: Prevent crash when passing invalid arguments into msvcrt
module.

@item 
bpo-25778@footnote{https://bugs.python.org/issue25778}: winreg does not truncate string correctly (Patch by Eryk Sun)

@item 
bpo-28896@footnote{https://bugs.python.org/issue28896}: Deprecate WindowsRegistryFinder and disable it by default.
@end itemize

@node C API<37>,Documentation<43>,Windows<38>,Python 3 6 1 release candidate 1
@anchor{whatsnew/changelog id387}@anchor{13d2}
@subsubsection C API



@itemize -

@item 
bpo-27867@footnote{https://bugs.python.org/issue27867}: Function PySlice_GetIndicesEx() is replaced with a macro if
Py_LIMITED_API is not set or set to the value between 0x03050400 and
0x03060000 (not including) or 0x03060100 or higher.

@item 
bpo-29083@footnote{https://bugs.python.org/issue29083}: Fixed the declaration of some public API functions.
PyArg_VaParse() and PyArg_VaParseTupleAndKeywords() were not available in
limited API. PyArg_ValidateKeywordArguments(), PyArg_UnpackTuple() and
Py_BuildValue() were not available in limited API of version < 3.3 when
PY_SSIZE_T_CLEAN is defined.

@item 
bpo-29058@footnote{https://bugs.python.org/issue29058}: All stable API extensions added after Python 3.2 are now
available only when Py_LIMITED_API is set to the PY_VERSION_HEX value of
the minimum Python version supporting this API.
@end itemize

@node Documentation<43>,Tests<40>,C API<37>,Python 3 6 1 release candidate 1
@anchor{whatsnew/changelog id388}@anchor{13d3}
@subsubsection Documentation



@itemize -

@item 
bpo-28929@footnote{https://bugs.python.org/issue28929}: Link the documentation to its source file on GitHub.

@item 
bpo-25008@footnote{https://bugs.python.org/issue25008}: Document smtpd.py as effectively deprecated and add a pointer
to aiosmtpd, a third-party asyncio-based replacement.

@item 
bpo-26355@footnote{https://bugs.python.org/issue26355}: Add canonical header link on each page to corresponding major
version of the documentation. Patch by Matthias Bussonnier.

@item 
bpo-29349@footnote{https://bugs.python.org/issue29349}: Fix Python 2 syntax in code for building the documentation.
@end itemize

@node Tests<40>,Build<43>,Documentation<43>,Python 3 6 1 release candidate 1
@anchor{whatsnew/changelog id389}@anchor{13d4}
@subsubsection Tests



@itemize -

@item 
bpo-28087@footnote{https://bugs.python.org/issue28087}: Skip test_asyncore and test_eintr poll failures on macOS. Skip
some tests of select.poll when running on macOS due to unresolved issues
with the underlying system poll function on some macOS versions.

@item 
bpo-29571@footnote{https://bugs.python.org/issue29571}: to match the behaviour of the @code{re.LOCALE} flag,
test_re.test_locale_flag now uses @code{locale.getpreferredencoding(False)}
to determine the candidate encoding for the test regex (allowing it to
correctly skip the test when the default locale encoding is a multi-byte
encoding)

@item 
bpo-28950@footnote{https://bugs.python.org/issue28950}: Disallow -j0 to be combined with -T/-l in regrtest command line
arguments.

@item 
bpo-28683@footnote{https://bugs.python.org/issue28683}: Fix the tests that bind() a unix socket and raise
PermissionError on Android for a non-root user.

@item 
bpo-26939@footnote{https://bugs.python.org/issue26939}: Add the support.setswitchinterval() function to fix
test_functools hanging on the Android armv7 qemu emulator.
@end itemize

@node Build<43>,,Tests<40>,Python 3 6 1 release candidate 1
@anchor{whatsnew/changelog id390}@anchor{13d5}
@subsubsection Build



@itemize -

@item 
bpo-27593@footnote{https://bugs.python.org/issue27593}: sys.version and the platform module python_build(),
python_branch(), and python_revision() functions now use git information
rather than hg when building from a repo.

@item 
bpo-29572@footnote{https://bugs.python.org/issue29572}: Update Windows build and OS X installers to use OpenSSL 1.0.2k.

@item 
bpo-26851@footnote{https://bugs.python.org/issue26851}: Set Android compilation and link flags.

@item 
bpo-28768@footnote{https://bugs.python.org/issue28768}: Fix implicit declaration of function _setmode. Patch by
Masayuki Yamamoto

@item 
bpo-29080@footnote{https://bugs.python.org/issue29080}: Removes hard dependency on hg.exe from PCBuild/build.bat

@item 
bpo-23903@footnote{https://bugs.python.org/issue23903}: Added missed names to PC/python3.def.

@item 
bpo-28762@footnote{https://bugs.python.org/issue28762}: lockf() is available on Android API level 24, but the F_LOCK
macro is not defined in android-ndk-r13.

@item 
bpo-28538@footnote{https://bugs.python.org/issue28538}: Fix the compilation error that occurs because if_nameindex() is
available on Android API level 24, but the if_nameindex structure is not
defined.

@item 
bpo-20211@footnote{https://bugs.python.org/issue20211}: Do not add the directory for installing C header files and the
directory for installing object code libraries to the cross compilation
search paths. Original patch by Thomas Petazzoni.

@item 
bpo-28849@footnote{https://bugs.python.org/issue28849}: Do not define sys.implementation._multiarch on Android.
@end itemize

@node Python 3 6 0 final,Python 3 6 0 release candidate 2,Python 3 6 1 release candidate 1,Changelog
@anchor{whatsnew/changelog python-3-6-0-final}@anchor{13d6}
@subsection Python 3.6.0 final


@emph{Release date: 2016-12-23}

No changes since release candidate 2

@node Python 3 6 0 release candidate 2,Python 3 6 0 release candidate 1,Python 3 6 0 final,Changelog
@anchor{whatsnew/changelog python-3-6-0-release-candidate-2}@anchor{13d7}
@subsection Python 3.6.0 release candidate 2


@emph{Release date: 2016-12-16}

@menu
* Core and Builtins: Core and Builtins<44>. 
* Tools/Demos: Tools/Demos<23>. 
* Windows: Windows<39>. 
* Build: Build<44>. 

@end menu

@node Core and Builtins<44>,Tools/Demos<23>,,Python 3 6 0 release candidate 2
@anchor{whatsnew/changelog id391}@anchor{13d8}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-28147@footnote{https://bugs.python.org/issue28147}: Fix a memory leak in split-table dictionaries: setattr() must
not convert combined table into split table. Patch written by INADA Naoki.

@item 
bpo-28990@footnote{https://bugs.python.org/issue28990}: Fix asyncio SSL hanging if connection is closed before
handshake is completed. (Patch by HoHo-Ho)
@end itemize

@node Tools/Demos<23>,Windows<39>,Core and Builtins<44>,Python 3 6 0 release candidate 2
@anchor{whatsnew/changelog id392}@anchor{13d9}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-28770@footnote{https://bugs.python.org/issue28770}: Fix python-gdb.py for fastcalls.
@end itemize

@node Windows<39>,Build<44>,Tools/Demos<23>,Python 3 6 0 release candidate 2
@anchor{whatsnew/changelog id393}@anchor{13da}
@subsubsection Windows



@itemize -

@item 
bpo-28896@footnote{https://bugs.python.org/issue28896}: Deprecate WindowsRegistryFinder.
@end itemize

@node Build<44>,,Windows<39>,Python 3 6 0 release candidate 2
@anchor{whatsnew/changelog id394}@anchor{13db}
@subsubsection Build



@itemize -

@item 
bpo-28898@footnote{https://bugs.python.org/issue28898}: Prevent gdb build errors due to HAVE_LONG_LONG redefinition.
@end itemize

@node Python 3 6 0 release candidate 1,Python 3 6 0 beta 4,Python 3 6 0 release candidate 2,Changelog
@anchor{whatsnew/changelog python-3-6-0-release-candidate-1}@anchor{13dc}
@subsection Python 3.6.0 release candidate 1


@emph{Release date: 2016-12-06}

@menu
* Core and Builtins: Core and Builtins<45>. 
* Library: Library<46>. 
* C API: C API<38>. 
* Documentation: Documentation<44>. 
* Tools/Demos: Tools/Demos<24>. 

@end menu

@node Core and Builtins<45>,Library<46>,,Python 3 6 0 release candidate 1
@anchor{whatsnew/changelog id395}@anchor{13dd}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-23722@footnote{https://bugs.python.org/issue23722}: Rather than silently producing a class that doesn’t support
zero-argument @code{super()} in methods, failing to pass the new
@code{__classcell__} namespace entry up to @code{type.__new__} now results in a
@code{DeprecationWarning} and a class that supports zero-argument
@code{super()}.

@item 
bpo-28797@footnote{https://bugs.python.org/issue28797}: Modifying the class __dict__ inside the __set_name__ method of
a descriptor that is used inside that class no longer prevents calling the
__set_name__ method of other descriptors.

@item 
bpo-28782@footnote{https://bugs.python.org/issue28782}: Fix a bug in the implementation @code{yield from} when checking if
the next instruction is YIELD_FROM. Regression introduced by WORDCODE
(bpo-26647@footnote{https://bugs.python.org/issue26647}).
@end itemize

@node Library<46>,C API<38>,Core and Builtins<45>,Python 3 6 0 release candidate 1
@anchor{whatsnew/changelog id396}@anchor{13de}
@subsubsection Library



@itemize -

@item 
bpo-27030@footnote{https://bugs.python.org/issue27030}: Unknown escapes in re.sub() replacement template are allowed
again.  But they still are deprecated and will be disabled in 3.7.

@item 
bpo-28835@footnote{https://bugs.python.org/issue28835}: Fix a regression introduced in warnings.catch_warnings(): call
warnings.showwarning() if it was overridden inside the context manager.

@item 
bpo-27172@footnote{https://bugs.python.org/issue27172}: To assist with upgrades from 2.7, the previously documented
deprecation of @code{inspect.getfullargspec()} has been reversed. This
decision may be revisited again after the Python 2.7 branch is no longer
officially supported.

@item 
bpo-26273@footnote{https://bugs.python.org/issue26273}: Add new @code{socket.TCP_CONGESTION} (Linux 2.6.13) and
@code{socket.TCP_USER_TIMEOUT} (Linux 2.6.37) constants. Patch written by
Omar Sandoval.

@item 
bpo-24142@footnote{https://bugs.python.org/issue24142}: Reading a corrupt config file left configparser in an invalid
state. Original patch by Florian Höch.

@item 
bpo-28843@footnote{https://bugs.python.org/issue28843}: Fix asyncio C Task to handle exceptions __traceback__.
@end itemize

@node C API<38>,Documentation<44>,Library<46>,Python 3 6 0 release candidate 1
@anchor{whatsnew/changelog id397}@anchor{13df}
@subsubsection C API



@itemize -

@item 
bpo-28808@footnote{https://bugs.python.org/issue28808}: PyUnicode_CompareWithASCIIString() now never raises exceptions.
@end itemize

@node Documentation<44>,Tools/Demos<24>,C API<38>,Python 3 6 0 release candidate 1
@anchor{whatsnew/changelog id398}@anchor{13e0}
@subsubsection Documentation



@itemize -

@item 
bpo-23722@footnote{https://bugs.python.org/issue23722}: The data model reference and the porting section in the What’s
New guide now cover the additional @code{__classcell__} handling needed for
custom metaclasses to fully support 
@geindex Python Enhancement Proposals; PEP 487
PEP 487@footnote{https://www.python.org/dev/peps/pep-0487} and zero-argument
@code{super()}.
@end itemize

@node Tools/Demos<24>,,Documentation<44>,Python 3 6 0 release candidate 1
@anchor{whatsnew/changelog id399}@anchor{13e1}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-28023@footnote{https://bugs.python.org/issue28023}: Fix python-gdb.py didn’t support new dict implementation.
@end itemize

@node Python 3 6 0 beta 4,Python 3 6 0 beta 3,Python 3 6 0 release candidate 1,Changelog
@anchor{whatsnew/changelog python-3-6-0-beta-4}@anchor{13e2}
@subsection Python 3.6.0 beta 4


@emph{Release date: 2016-11-21}

@menu
* Core and Builtins: Core and Builtins<46>. 
* Library: Library<47>. 
* Documentation: Documentation<45>. 
* Tests: Tests<41>. 
* Build: Build<45>. 

@end menu

@node Core and Builtins<46>,Library<47>,,Python 3 6 0 beta 4
@anchor{whatsnew/changelog id400}@anchor{13e3}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-28532@footnote{https://bugs.python.org/issue28532}: Show sys.version when -V option is supplied twice.

@item 
bpo-27100@footnote{https://bugs.python.org/issue27100}: The with-statement now checks for __enter__ before it checks
for __exit__. This gives less confusing error messages when both methods
are missing. Patch by Jonathan Ellington.

@item 
bpo-28746@footnote{https://bugs.python.org/issue28746}: Fix the set_inheritable() file descriptor method on platforms
that do not have the ioctl FIOCLEX and FIONCLEX commands.

@item 
bpo-26920@footnote{https://bugs.python.org/issue26920}: Fix not getting the locale’s charset upon initializing the
interpreter, on platforms that do not have langinfo.

@item 
bpo-28648@footnote{https://bugs.python.org/issue28648}: Fixed crash in Py_DecodeLocale() in debug build on Mac OS X
when decode astral characters.  Patch by Xiang Zhang.

@item 
bpo-19398@footnote{https://bugs.python.org/issue19398}: Extra slash no longer added to sys.path components in case of
empty compile-time PYTHONPATH components.

@item 
bpo-28665@footnote{https://bugs.python.org/issue28665}: Improve speed of the STORE_DEREF opcode by 40%.

@item 
bpo-28583@footnote{https://bugs.python.org/issue28583}: PyDict_SetDefault didn’t combine split table when needed. Patch
by Xiang Zhang.

@item 
bpo-27243@footnote{https://bugs.python.org/issue27243}: Change PendingDeprecationWarning -> DeprecationWarning. As it
was agreed in the issue, __aiter__ returning an awaitable should result in
PendingDeprecationWarning in 3.5 and in DeprecationWarning in 3.6.

@item 
bpo-26182@footnote{https://bugs.python.org/issue26182}: Fix a refleak in code that raises DeprecationWarning.

@item 
bpo-28721@footnote{https://bugs.python.org/issue28721}: Fix asynchronous generators aclose() and athrow() to handle
StopAsyncIteration propagation properly.
@end itemize

@node Library<47>,Documentation<45>,Core and Builtins<46>,Python 3 6 0 beta 4
@anchor{whatsnew/changelog id401}@anchor{13e4}
@subsubsection Library



@itemize -

@item 
bpo-28752@footnote{https://bugs.python.org/issue28752}: Restored the __reduce__() methods of datetime objects.

@item 
bpo-28727@footnote{https://bugs.python.org/issue28727}: Regular expression patterns, _sre.SRE_Pattern objects created
by re.compile(), become comparable (only x==y and x!=y operators). This
change should fix the bpo-18383@footnote{https://bugs.python.org/issue18383}: don’t duplicate warning filters when
the warnings module is reloaded (thing usually only done in unit tests).

@item 
bpo-20572@footnote{https://bugs.python.org/issue20572}: The subprocess.Popen.wait method’s undocumented endtime
parameter now raises a DeprecationWarning.

@item 
bpo-25659@footnote{https://bugs.python.org/issue25659}: In ctypes, prevent a crash calling the from_buffer() and
from_buffer_copy() methods on abstract classes like Array.

@item 
bpo-19717@footnote{https://bugs.python.org/issue19717}: Makes Path.resolve() succeed on paths that do not exist. Patch
by Vajrasky Kok

@item 
bpo-28563@footnote{https://bugs.python.org/issue28563}: Fixed possible DoS and arbitrary code execution when handle
plural form selections in the gettext module.  The expression parser now
supports exact syntax supported by GNU gettext.

@item 
bpo-28387@footnote{https://bugs.python.org/issue28387}: Fixed possible crash in _io.TextIOWrapper deallocator when the
garbage collector is invoked in other thread.  Based on patch by Sebastian
Cufre.

@item 
bpo-28600@footnote{https://bugs.python.org/issue28600}: Optimize loop.call_soon.

@item 
bpo-28613@footnote{https://bugs.python.org/issue28613}: Fix get_event_loop() return the current loop if called from
coroutines/callbacks.

@item 
bpo-28634@footnote{https://bugs.python.org/issue28634}: Fix asyncio.isfuture() to support unittest.Mock.

@item 
bpo-26081@footnote{https://bugs.python.org/issue26081}: Fix refleak in _asyncio.Future.__iter__().throw.

@item 
bpo-28639@footnote{https://bugs.python.org/issue28639}: Fix inspect.isawaitable to always return bool Patch by Justin
Mayfield.

@item 
bpo-28652@footnote{https://bugs.python.org/issue28652}: Make loop methods reject socket kinds they do not support.

@item 
bpo-28653@footnote{https://bugs.python.org/issue28653}: Fix a refleak in functools.lru_cache.

@item 
bpo-28703@footnote{https://bugs.python.org/issue28703}: Fix asyncio.iscoroutinefunction to handle Mock objects.

@item 
bpo-28704@footnote{https://bugs.python.org/issue28704}: Fix create_unix_server to support Path-like objects (PEP 519).

@item 
bpo-28720@footnote{https://bugs.python.org/issue28720}: Add collections.abc.AsyncGenerator.
@end itemize

@node Documentation<45>,Tests<41>,Library<47>,Python 3 6 0 beta 4
@anchor{whatsnew/changelog id402}@anchor{13e5}
@subsubsection Documentation



@itemize -

@item 
bpo-28513@footnote{https://bugs.python.org/issue28513}: Documented command-line interface of zipfile.
@end itemize

@node Tests<41>,Build<45>,Documentation<45>,Python 3 6 0 beta 4
@anchor{whatsnew/changelog id403}@anchor{13e6}
@subsubsection Tests



@itemize -

@item 
bpo-28666@footnote{https://bugs.python.org/issue28666}: Now test.support.rmtree is able to remove unwritable or
unreadable directories.

@item 
bpo-23839@footnote{https://bugs.python.org/issue23839}: Various caches now are cleared before running every test file.
@end itemize

@node Build<45>,,Tests<41>,Python 3 6 0 beta 4
@anchor{whatsnew/changelog id404}@anchor{13e7}
@subsubsection Build



@itemize -

@item 
bpo-10656@footnote{https://bugs.python.org/issue10656}: Fix out-of-tree building on AIX.  Patch by Tristan Carel and
Michael Haubenwallner.

@item 
bpo-26359@footnote{https://bugs.python.org/issue26359}: Rename –with-optimiations to –enable-optimizations.

@item 
bpo-28676@footnote{https://bugs.python.org/issue28676}: Prevent missing ‘getentropy’ declaration warning on macOS.
Patch by Gareth Rees.
@end itemize

@node Python 3 6 0 beta 3,Python 3 6 0 beta 2,Python 3 6 0 beta 4,Changelog
@anchor{whatsnew/changelog python-3-6-0-beta-3}@anchor{13e8}
@subsection Python 3.6.0 beta 3


@emph{Release date: 2016-10-31}

@menu
* Core and Builtins: Core and Builtins<47>. 
* Library: Library<48>. 
* Windows: Windows<40>. 
* Build: Build<46>. 
* Tests: Tests<42>. 

@end menu

@node Core and Builtins<47>,Library<48>,,Python 3 6 0 beta 3
@anchor{whatsnew/changelog id405}@anchor{13e9}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-28128@footnote{https://bugs.python.org/issue28128}: Deprecation warning for invalid str and byte escape sequences
now prints better information about where the error occurs. Patch by
Serhiy Storchaka and Eric Smith.

@item 
bpo-28509@footnote{https://bugs.python.org/issue28509}: dict.update() no longer allocate unnecessary large memory.

@item 
bpo-28426@footnote{https://bugs.python.org/issue28426}: Fixed potential crash in PyUnicode_AsDecodedObject() in debug
build.

@item 
bpo-28517@footnote{https://bugs.python.org/issue28517}: Fixed of-by-one error in the peephole optimizer that caused
keeping unreachable code.

@item 
bpo-28214@footnote{https://bugs.python.org/issue28214}: Improved exception reporting for problematic __set_name__
attributes.

@item 
bpo-23782@footnote{https://bugs.python.org/issue23782}: Fixed possible memory leak in _PyTraceback_Add() and exception
loss in PyTraceBack_Here().

@item 
bpo-28471@footnote{https://bugs.python.org/issue28471}: Fix “Python memory allocator called without holding the GIL”
crash in socket.setblocking.
@end itemize

@node Library<48>,Windows<40>,Core and Builtins<47>,Python 3 6 0 beta 3
@anchor{whatsnew/changelog id406}@anchor{13ea}
@subsubsection Library



@itemize -

@item 
bpo-27517@footnote{https://bugs.python.org/issue27517}: LZMA compressor and decompressor no longer raise exceptions if
given empty data twice.  Patch by Benjamin Fogle.

@item 
bpo-28549@footnote{https://bugs.python.org/issue28549}: Fixed segfault in curses’s addch() with ncurses6.

@item 
bpo-28449@footnote{https://bugs.python.org/issue28449}: tarfile.open() with mode “r” or “r:” now tries to open a tar
file with compression before trying to open it without compression.
Otherwise it had 50% chance failed with ignore_zeros=True.

@item 
bpo-23262@footnote{https://bugs.python.org/issue23262}: The webbrowser module now supports Firefox 36+ and derived
browsers.  Based on patch by Oleg Broytman.

@item 
bpo-27939@footnote{https://bugs.python.org/issue27939}: Fixed bugs in tkinter.ttk.LabeledScale and tkinter.Scale caused
by representing the scale as float value internally in Tk.  tkinter.IntVar
now works if float value is set to underlying Tk variable.

@item 
bpo-18844@footnote{https://bugs.python.org/issue18844}: The various ways of specifying weights for random.choices() now
produce the same result sequences.

@item 
bpo-28255@footnote{https://bugs.python.org/issue28255}: calendar.TextCalendar().prmonth() no longer prints a space at
the start of new line after printing a month’s calendar.  Patch by Xiang
Zhang.

@item 
bpo-20491@footnote{https://bugs.python.org/issue20491}: The textwrap.TextWrapper class now honors non-breaking spaces.
Based on patch by Kaarle Ritvanen.

@item 
bpo-28353@footnote{https://bugs.python.org/issue28353}: os.fwalk() no longer fails on broken links.

@item 
bpo-28430@footnote{https://bugs.python.org/issue28430}: Fix iterator of C implemented asyncio.Future doesn’t accept
non-None value is passed to it.send(val).

@item 
bpo-27025@footnote{https://bugs.python.org/issue27025}: Generated names for Tkinter widgets now start by the “!” prefix
for readability.

@item 
bpo-25464@footnote{https://bugs.python.org/issue25464}: Fixed HList.header_exists() in tkinter.tix module by addin a
workaround to Tix library bug.

@item 
bpo-28488@footnote{https://bugs.python.org/issue28488}: shutil.make_archive() no longer adds entry “./” to ZIP archive.

@item 
bpo-25953@footnote{https://bugs.python.org/issue25953}: re.sub() now raises an error for invalid numerical group
reference in replacement template even if the pattern is not found in the
string.  Error message for invalid group reference now includes the group
index and the position of the reference. Based on patch by SilentGhost.

@item 
bpo-18219@footnote{https://bugs.python.org/issue18219}: Optimize csv.DictWriter for large number of columns. Patch by
Mariatta Wijaya.

@item 
bpo-28448@footnote{https://bugs.python.org/issue28448}: Fix C implemented asyncio.Future didn’t work on Windows.

@item 
bpo-28480@footnote{https://bugs.python.org/issue28480}: Fix error building socket module when multithreading is
disabled.

@item 
bpo-24452@footnote{https://bugs.python.org/issue24452}: Make webbrowser support Chrome on Mac OS X.

@item 
bpo-20766@footnote{https://bugs.python.org/issue20766}: Fix references leaked by pdb in the handling of SIGINT
handlers.

@item 
bpo-28492@footnote{https://bugs.python.org/issue28492}: Fix how StopIteration exception is raised in _asyncio.Future.

@item 
bpo-28500@footnote{https://bugs.python.org/issue28500}: Fix asyncio to handle async gens GC from another thread.

@item 
bpo-26923@footnote{https://bugs.python.org/issue26923}: Fix asyncio.Gather to refuse being cancelled once all children
are done. Patch by Johannes Ebke.

@item 
bpo-26796@footnote{https://bugs.python.org/issue26796}: Don’t configure the number of workers for default threadpool
executor. Initial patch by Hans Lawrenz.

@item 
bpo-28544@footnote{https://bugs.python.org/issue28544}: Implement asyncio.Task in C.
@end itemize

@node Windows<40>,Build<46>,Library<48>,Python 3 6 0 beta 3
@anchor{whatsnew/changelog id407}@anchor{13eb}
@subsubsection Windows



@itemize -

@item 
bpo-28522@footnote{https://bugs.python.org/issue28522}: Fixes mishandled buffer reallocation in getpathp.c
@end itemize

@node Build<46>,Tests<42>,Windows<40>,Python 3 6 0 beta 3
@anchor{whatsnew/changelog id408}@anchor{13ec}
@subsubsection Build



@itemize -

@item 
bpo-28444@footnote{https://bugs.python.org/issue28444}: Fix missing extensions modules when cross compiling.

@item 
bpo-28208@footnote{https://bugs.python.org/issue28208}: Update Windows build and OS X installers to use SQLite 3.14.2.

@item 
bpo-28248@footnote{https://bugs.python.org/issue28248}: Update Windows build and OS X installers to use OpenSSL 1.0.2j.
@end itemize

@node Tests<42>,,Build<46>,Python 3 6 0 beta 3
@anchor{whatsnew/changelog id409}@anchor{13ed}
@subsubsection Tests



@itemize -

@item 
bpo-26944@footnote{https://bugs.python.org/issue26944}: Fix test_posix for Android where ‘id -G’ is entirely wrong or
missing the effective gid.

@item 
bpo-28409@footnote{https://bugs.python.org/issue28409}: regrtest: fix the parser of command line arguments.
@end itemize

@node Python 3 6 0 beta 2,Python 3 6 0 beta 1,Python 3 6 0 beta 3,Changelog
@anchor{whatsnew/changelog python-3-6-0-beta-2}@anchor{13ee}
@subsection Python 3.6.0 beta 2


@emph{Release date: 2016-10-10}

@menu
* Core and Builtins: Core and Builtins<48>. 
* Library: Library<49>. 
* Windows: Windows<41>. 
* C API: C API<39>. 
* Build: Build<47>. 
* Tests: Tests<43>. 

@end menu

@node Core and Builtins<48>,Library<49>,,Python 3 6 0 beta 2
@anchor{whatsnew/changelog id410}@anchor{13ef}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-28183@footnote{https://bugs.python.org/issue28183}: Optimize and cleanup dict iteration.

@item 
bpo-26081@footnote{https://bugs.python.org/issue26081}: Added C implementation of asyncio.Future. Original patch by
Yury Selivanov.

@item 
bpo-28379@footnote{https://bugs.python.org/issue28379}: Added sanity checks and tests for PyUnicode_CopyCharacters().
Patch by Xiang Zhang.

@item 
bpo-28376@footnote{https://bugs.python.org/issue28376}: The type of long range iterator is now registered as Iterator.
Patch by Oren Milman.

@item 
bpo-28376@footnote{https://bugs.python.org/issue28376}: Creating instances of range_iterator by calling range_iterator
type now is deprecated.  Patch by Oren Milman.

@item 
bpo-28376@footnote{https://bugs.python.org/issue28376}: The constructor of range_iterator now checks that step is not
0. Patch by Oren Milman.

@item 
bpo-26906@footnote{https://bugs.python.org/issue26906}: Resolving special methods of uninitialized type now causes
implicit initialization of the type instead of a fail.

@item 
bpo-18287@footnote{https://bugs.python.org/issue18287}: PyType_Ready() now checks that tp_name is not NULL. Original
patch by Niklas Koep.

@item 
bpo-24098@footnote{https://bugs.python.org/issue24098}: Fixed possible crash when AST is changed in process of
compiling it.

@item 
bpo-28201@footnote{https://bugs.python.org/issue28201}: Dict reduces possibility of 2nd conflict in hash table when
hashes have same lower bits.

@item 
bpo-28350@footnote{https://bugs.python.org/issue28350}: String constants with null character no longer interned.

@item 
bpo-26617@footnote{https://bugs.python.org/issue26617}: Fix crash when GC runs during weakref callbacks.

@item 
bpo-27942@footnote{https://bugs.python.org/issue27942}: String constants now interned recursively in tuples and
frozensets.

@item 
bpo-21578@footnote{https://bugs.python.org/issue21578}: Fixed misleading error message when ImportError called with
invalid keyword args.

@item 
bpo-28203@footnote{https://bugs.python.org/issue28203}: Fix incorrect type in complex(1.0, @{2:3@}) error message. Patch
by Soumya Sharma.

@item 
bpo-28086@footnote{https://bugs.python.org/issue28086}: Single var-positional argument of tuple subtype was passed
unscathed to the C-defined function.  Now it is converted to exact tuple.

@item 
bpo-28214@footnote{https://bugs.python.org/issue28214}: Now __set_name__ is looked up on the class instead of the
instance.

@item 
bpo-27955@footnote{https://bugs.python.org/issue27955}: Fallback on reading /dev/urandom device when the getrandom()
syscall fails with EPERM, for example when blocked by SECCOMP.

@item 
bpo-28192@footnote{https://bugs.python.org/issue28192}: Don’t import readline in isolated mode.

@item 
Upgrade internal unicode databases to Unicode version 9.0.0.

@item 
bpo-28131@footnote{https://bugs.python.org/issue28131}: Fix a regression in zipimport’s compile_source().  zipimport
should use the same optimization level as the interpreter.

@item 
bpo-28126@footnote{https://bugs.python.org/issue28126}: Replace Py_MEMCPY with memcpy(). Visual Studio can properly
optimize memcpy().

@item 
bpo-28120@footnote{https://bugs.python.org/issue28120}: Fix dict.pop() for splitted dictionary when trying to remove a
“pending key” (Not yet inserted in split-table). Patch by Xiang Zhang.

@item 
bpo-26182@footnote{https://bugs.python.org/issue26182}: Raise DeprecationWarning when async and await keywords are used
as variable/attribute/class/function name.
@end itemize

@node Library<49>,Windows<41>,Core and Builtins<48>,Python 3 6 0 beta 2
@anchor{whatsnew/changelog id411}@anchor{13f0}
@subsubsection Library



@itemize -

@item 
bpo-27998@footnote{https://bugs.python.org/issue27998}: Fixed bytes path support in os.scandir() on Windows. Patch by
Eryk Sun.

@item 
bpo-28317@footnote{https://bugs.python.org/issue28317}: The disassembler now decodes FORMAT_VALUE argument.

@item 
bpo-26293@footnote{https://bugs.python.org/issue26293}: Fixed writing ZIP files that starts not from the start of the
file.  Offsets in ZIP file now are relative to the start of the archive in
conforming to the specification.

@item 
bpo-28380@footnote{https://bugs.python.org/issue28380}: unittest.mock Mock autospec functions now properly support
assert_called, assert_not_called, and assert_called_once.

@item 
bpo-27181@footnote{https://bugs.python.org/issue27181}: remove statistics.geometric_mean and defer until 3.7.

@item 
bpo-28229@footnote{https://bugs.python.org/issue28229}: lzma module now supports pathlib.

@item 
bpo-28321@footnote{https://bugs.python.org/issue28321}: Fixed writing non-BMP characters with binary format in
plistlib.

@item 
bpo-28225@footnote{https://bugs.python.org/issue28225}: bz2 module now supports pathlib.  Initial patch by Ethan
Furman.

@item 
bpo-28227@footnote{https://bugs.python.org/issue28227}: gzip now supports pathlib.  Patch by Ethan Furman.

@item 
bpo-27358@footnote{https://bugs.python.org/issue27358}: Optimized merging var-keyword arguments and improved error
message when passing a non-mapping as a var-keyword argument.

@item 
bpo-28257@footnote{https://bugs.python.org/issue28257}: Improved error message when passing a non-iterable as a
var-positional argument.  Added opcode BUILD_TUPLE_UNPACK_WITH_CALL.

@item 
bpo-28322@footnote{https://bugs.python.org/issue28322}: Fixed possible crashes when unpickle itertools objects from
incorrect pickle data.  Based on patch by John Leitch.

@item 
bpo-28228@footnote{https://bugs.python.org/issue28228}: imghdr now supports pathlib.

@item 
bpo-28226@footnote{https://bugs.python.org/issue28226}: compileall now supports pathlib.

@item 
bpo-28314@footnote{https://bugs.python.org/issue28314}: Fix function declaration (C flags) for the getiterator() method
of xml.etree.ElementTree.Element.

@item 
bpo-28148@footnote{https://bugs.python.org/issue28148}: Stop using localtime() and gmtime() in the time module.
Introduced platform independent _PyTime_localtime API that is similar to
POSIX localtime_r, but available on all platforms.  Patch by Ed Schouten.

@item 
bpo-28253@footnote{https://bugs.python.org/issue28253}: Fixed calendar functions for extreme months: 0001-01 and
9999-12. Methods itermonthdays() and itermonthdays2() are reimplemented so
that they don’t call itermonthdates() which can cause datetime.date
under/overflow.

@item 
bpo-28275@footnote{https://bugs.python.org/issue28275}: Fixed possible use after free in the decompress() methods of
the LZMADecompressor and BZ2Decompressor classes. Original patch by John
Leitch.

@item 
bpo-27897@footnote{https://bugs.python.org/issue27897}: Fixed possible crash in sqlite3.Connection.create_collation()
if pass invalid string-like object as a name.  Patch by Xiang Zhang.

@item 
bpo-18844@footnote{https://bugs.python.org/issue18844}: random.choices() now has k as a keyword-only argument to
improve the readability of common cases and come into line with the
signature used in other languages.

@item 
bpo-18893@footnote{https://bugs.python.org/issue18893}: Fix invalid exception handling in Lib/ctypes/macholib/dyld.py.
Patch by Madison May.

@item 
bpo-27611@footnote{https://bugs.python.org/issue27611}: Fixed support of default root window in the tkinter.tix module.
Added the master parameter in the DisplayStyle constructor.

@item 
bpo-27348@footnote{https://bugs.python.org/issue27348}: In the traceback module, restore the formatting of exception
messages like “Exception: None”.  This fixes a regression introduced in
3.5a2.

@item 
bpo-25651@footnote{https://bugs.python.org/issue25651}: Allow falsy values to be used for msg parameter of subTest().

@item 
bpo-27778@footnote{https://bugs.python.org/issue27778}: Fix a memory leak in os.getrandom() when the getrandom() is
interrupted by a signal and a signal handler raises a Python exception.

@item 
bpo-28200@footnote{https://bugs.python.org/issue28200}: Fix memory leak on Windows in the os module (fix
path_converter() function).

@item 
bpo-25400@footnote{https://bugs.python.org/issue25400}: RobotFileParser now correctly returns default values for
crawl_delay and request_rate.  Initial patch by Peter Wirtz.

@item 
bpo-27932@footnote{https://bugs.python.org/issue27932}: Prevent memory leak in win32_ver().

@item 
Fix UnboundLocalError in socket._sendfile_use_sendfile.

@item 
bpo-28075@footnote{https://bugs.python.org/issue28075}: Check for ERROR_ACCESS_DENIED in Windows implementation of
os.stat().  Patch by Eryk Sun.

@item 
bpo-22493@footnote{https://bugs.python.org/issue22493}: Warning message emitted by using inline flags in the middle of
regular expression now contains a (truncated) regex pattern. Patch by Tim
Graham.

@item 
bpo-25270@footnote{https://bugs.python.org/issue25270}: Prevent codecs.escape_encode() from raising SystemError when an
empty bytestring is passed.

@item 
bpo-28181@footnote{https://bugs.python.org/issue28181}: Get antigravity over HTTPS. Patch by Kaartic Sivaraam.

@item 
bpo-25895@footnote{https://bugs.python.org/issue25895}: Enable WebSocket URL schemes in urllib.parse.urljoin. Patch by
Gergely Imreh and Markus Holtermann.

@item 
bpo-28114@footnote{https://bugs.python.org/issue28114}: Fix a crash in parse_envlist() when env contains byte strings.
Patch by Eryk Sun.

@item 
bpo-27599@footnote{https://bugs.python.org/issue27599}: Fixed buffer overrun in binascii.b2a_qp() and
binascii.a2b_qp().

@item 
bpo-27906@footnote{https://bugs.python.org/issue27906}: Fix socket accept exhaustion during high TCP traffic. Patch by
Kevin Conway.

@item 
bpo-28174@footnote{https://bugs.python.org/issue28174}: Handle when SO_REUSEPORT isn’t properly supported. Patch by
Seth Michael Larson.

@item 
bpo-26654@footnote{https://bugs.python.org/issue26654}: Inspect functools.partial in asyncio.Handle.__repr__. Patch by
iceboy.

@item 
bpo-26909@footnote{https://bugs.python.org/issue26909}: Fix slow pipes IO in asyncio. Patch by INADA Naoki.

@item 
bpo-28176@footnote{https://bugs.python.org/issue28176}: Fix callbacks race in asyncio.SelectorLoop.sock_connect.

@item 
bpo-27759@footnote{https://bugs.python.org/issue27759}: Fix selectors incorrectly retain invalid file descriptors.
Patch by Mark Williams.

@item 
bpo-28368@footnote{https://bugs.python.org/issue28368}: Refuse monitoring processes if the child watcher has no loop
attached. Patch by Vincent Michel.

@item 
bpo-28369@footnote{https://bugs.python.org/issue28369}: Raise RuntimeError when transport’s FD is used with add_reader,
add_writer, etc.

@item 
bpo-28370@footnote{https://bugs.python.org/issue28370}: Speedup asyncio.StreamReader.readexactly. Patch by Коренберг
Марк.

@item 
bpo-28371@footnote{https://bugs.python.org/issue28371}: Deprecate passing asyncio.Handles to run_in_executor.

@item 
bpo-28372@footnote{https://bugs.python.org/issue28372}: Fix asyncio to support formatting of non-python coroutines.

@item 
bpo-28399@footnote{https://bugs.python.org/issue28399}: Remove UNIX socket from FS before binding. Patch by Коренберг
Марк.

@item 
bpo-27972@footnote{https://bugs.python.org/issue27972}: Prohibit Tasks to await on themselves.
@end itemize

@node Windows<41>,C API<39>,Library<49>,Python 3 6 0 beta 2
@anchor{whatsnew/changelog id412}@anchor{13f1}
@subsubsection Windows



@itemize -

@item 
bpo-28402@footnote{https://bugs.python.org/issue28402}: Adds signed catalog files for stdlib on Windows.

@item 
bpo-28333@footnote{https://bugs.python.org/issue28333}: Enables Unicode for ps1/ps2 and input() prompts. (Patch by Eryk
Sun)

@item 
bpo-28251@footnote{https://bugs.python.org/issue28251}: Improvements to help manuals on Windows.

@item 
bpo-28110@footnote{https://bugs.python.org/issue28110}: launcher.msi has different product codes between 32-bit and
64-bit

@item 
bpo-28161@footnote{https://bugs.python.org/issue28161}: Opening CON for write access fails

@item 
bpo-28162@footnote{https://bugs.python.org/issue28162}: WindowsConsoleIO readall() fails if first line starts with
Ctrl+Z

@item 
bpo-28163@footnote{https://bugs.python.org/issue28163}: WindowsConsoleIO fileno() passes wrong flags to _open_osfhandle

@item 
bpo-28164@footnote{https://bugs.python.org/issue28164}: _PyIO_get_console_type fails for various paths

@item 
bpo-28137@footnote{https://bugs.python.org/issue28137}: Renames Windows path file to ._pth

@item 
bpo-28138@footnote{https://bugs.python.org/issue28138}: Windows ._pth file should allow import site
@end itemize

@node C API<39>,Build<47>,Windows<41>,Python 3 6 0 beta 2
@anchor{whatsnew/changelog id413}@anchor{13f2}
@subsubsection C API



@itemize -

@item 
bpo-28426@footnote{https://bugs.python.org/issue28426}: Deprecated undocumented functions PyUnicode_AsEncodedObject(),
PyUnicode_AsDecodedObject(), PyUnicode_AsDecodedUnicode() and
PyUnicode_AsEncodedUnicode().
@end itemize

@node Build<47>,Tests<43>,C API<39>,Python 3 6 0 beta 2
@anchor{whatsnew/changelog id414}@anchor{13f3}
@subsubsection Build



@itemize -

@item 
bpo-28258@footnote{https://bugs.python.org/issue28258}: Fixed build with Estonian locale (python-config and distclean
targets in Makefile).  Patch by Arfrever Frehtes Taifersar Arahesis.

@item 
bpo-26661@footnote{https://bugs.python.org/issue26661}: setup.py now detects system libffi with multiarch wrapper.

@item 
bpo-15819@footnote{https://bugs.python.org/issue15819}: Remove redundant include search directory option for building
outside the source tree.
@end itemize

@node Tests<43>,,Build<47>,Python 3 6 0 beta 2
@anchor{whatsnew/changelog id415}@anchor{13f4}
@subsubsection Tests



@itemize -

@item 
bpo-28217@footnote{https://bugs.python.org/issue28217}: Adds _testconsole module to test console input.
@end itemize

@node Python 3 6 0 beta 1,Python 3 6 0 alpha 4,Python 3 6 0 beta 2,Changelog
@anchor{whatsnew/changelog python-3-6-0-beta-1}@anchor{13f5}
@subsection Python 3.6.0 beta 1


@emph{Release date: 2016-09-12}

@menu
* Core and Builtins: Core and Builtins<49>. 
* Library: Library<50>. 
* IDLE: IDLE<39>. 
* C API: C API<40>. 
* Tests: Tests<44>. 
* Build: Build<48>. 
* Tools/Demos: Tools/Demos<25>. 
* Windows: Windows<42>. 

@end menu

@node Core and Builtins<49>,Library<50>,,Python 3 6 0 beta 1
@anchor{whatsnew/changelog id416}@anchor{13f6}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-23722@footnote{https://bugs.python.org/issue23722}: The __class__ cell used by zero-argument super() is now
initialized from type.__new__ rather than __build_class__, so class
methods relying on that will now work correctly when called from metaclass
methods during class creation. Patch by Martin Teichmann.

@item 
bpo-25221@footnote{https://bugs.python.org/issue25221}: Fix corrupted result from PyLong_FromLong(0) when Python is
compiled with NSMALLPOSINTS = 0.

@item 
bpo-27080@footnote{https://bugs.python.org/issue27080}: Implement formatting support for 
@geindex Python Enhancement Proposals; PEP 515
PEP 515@footnote{https://www.python.org/dev/peps/pep-0515}.  Initial patch by
Chris Angelico.

@item 
bpo-27199@footnote{https://bugs.python.org/issue27199}: In tarfile, expose copyfileobj bufsize to improve throughput.
Patch by Jason Fried.

@item 
bpo-27948@footnote{https://bugs.python.org/issue27948}: In f-strings, only allow backslashes inside the braces (where
the expressions are).  This is a breaking change from the 3.6 alpha
releases, where backslashes are allowed anywhere in an f-string.  Also,
require that expressions inside f-strings be enclosed within literal
braces, and not escapes like @code{f'\x7b"hi"\x7d'}.

@item 
bpo-28046@footnote{https://bugs.python.org/issue28046}: Remove platform-specific directories from sys.path.

@item 
bpo-28071@footnote{https://bugs.python.org/issue28071}: Add early-out for differencing from an empty set.

@item 
bpo-25758@footnote{https://bugs.python.org/issue25758}: Prevents zipimport from unnecessarily encoding a filename
(patch by Eryk Sun)

@item 
bpo-25856@footnote{https://bugs.python.org/issue25856}: The __module__ attribute of extension classes and functions now
is interned. This leads to more compact pickle data with protocol 4.

@item 
bpo-27213@footnote{https://bugs.python.org/issue27213}: Rework CALL_FUNCTION* opcodes to produce shorter and more
efficient bytecode. Patch by Demur Rumed, design by Serhiy Storchaka,
reviewed by Serhiy Storchaka and Victor Stinner.

@item 
bpo-26331@footnote{https://bugs.python.org/issue26331}: Implement tokenizing support for 
@geindex Python Enhancement Proposals; PEP 515
PEP 515@footnote{https://www.python.org/dev/peps/pep-0515}. Patch by Georg
Brandl.

@item 
bpo-27999@footnote{https://bugs.python.org/issue27999}: Make “global after use” a SyntaxError, and ditto for nonlocal.
Patch by Ivan Levkivskyi.

@item 
bpo-28003@footnote{https://bugs.python.org/issue28003}: Implement 
@geindex Python Enhancement Proposals; PEP 525
PEP 525@footnote{https://www.python.org/dev/peps/pep-0525} – Asynchronous Generators.

@item 
bpo-27985@footnote{https://bugs.python.org/issue27985}: Implement 
@geindex Python Enhancement Proposals; PEP 526
PEP 526@footnote{https://www.python.org/dev/peps/pep-0526} – Syntax for Variable Annotations. Patch
by Ivan Levkivskyi.

@item 
bpo-26058@footnote{https://bugs.python.org/issue26058}: Add a new private version to the builtin dict type, incremented
at each dictionary creation and at each dictionary change. Implementation
of the PEP 509.

@item 
bpo-27364@footnote{https://bugs.python.org/issue27364}: A backslash-character pair that is not a valid escape sequence
now generates a DeprecationWarning.  Patch by Emanuel Barry.

@item 
bpo-27350@footnote{https://bugs.python.org/issue27350}: @ref{3b0,,dict} implementation is changed like PyPy. It is more compact
and preserves insertion order. (Concept developed by Raymond Hettinger and
patch by Inada Naoki.)

@item 
bpo-27911@footnote{https://bugs.python.org/issue27911}: Remove unnecessary error checks in
@code{exec_builtin_or_dynamic()}.

@item 
bpo-27078@footnote{https://bugs.python.org/issue27078}: Added BUILD_STRING opcode.  Optimized f-strings evaluation.

@item 
bpo-17884@footnote{https://bugs.python.org/issue17884}: Python now requires systems with inttypes.h and stdint.h

@item 
bpo-27961@footnote{https://bugs.python.org/issue27961}: Require platforms to support @code{long long}. Python hasn’t
compiled without @code{long long} for years, so this is basically a
formality.

@item 
bpo-27355@footnote{https://bugs.python.org/issue27355}: Removed support for Windows CE.  It was never finished, and
Windows CE is no longer a relevant platform for Python.

@item 
Implement 
@geindex Python Enhancement Proposals; PEP 523
PEP 523@footnote{https://www.python.org/dev/peps/pep-0523}.

@item 
bpo-27870@footnote{https://bugs.python.org/issue27870}: A left shift of zero by a large integer no longer attempts to
allocate large amounts of memory.

@item 
bpo-25402@footnote{https://bugs.python.org/issue25402}: In int-to-decimal-string conversion, improve the estimate of
the intermediate memory required, and remove an unnecessarily strict
overflow check. Patch by Serhiy Storchaka.

@item 
bpo-27214@footnote{https://bugs.python.org/issue27214}: In long_invert, be more careful about modifying object returned
by long_add, and remove an unnecessary check for small longs. Thanks Oren
Milman for analysis and patch.

@item 
bpo-27506@footnote{https://bugs.python.org/issue27506}: Support passing the bytes/bytearray.translate() “delete”
argument by keyword.

@item 
bpo-27812@footnote{https://bugs.python.org/issue27812}: Properly clear out a generator’s frame’s backreference to the
generator to prevent crashes in frame.clear().

@item 
bpo-27811@footnote{https://bugs.python.org/issue27811}: Fix a crash when a coroutine that has not been awaited is
finalized with warnings-as-errors enabled.

@item 
bpo-27587@footnote{https://bugs.python.org/issue27587}: Fix another issue found by PVS-Studio: Null pointer check after
use of ‘def’ in _PyState_AddModule(). Initial patch by Christian Heimes.

@item 
bpo-27792@footnote{https://bugs.python.org/issue27792}: The modulo operation applied to @code{bool} and other @code{int}
subclasses now always returns an @code{int}. Previously the return type
depended on the input values. Patch by Xiang Zhang.

@item 
bpo-26984@footnote{https://bugs.python.org/issue26984}: int() now always returns an instance of exact int.

@item 
bpo-25604@footnote{https://bugs.python.org/issue25604}: Fix a minor bug in integer true division; this bug could
potentially have caused off-by-one-ulp results on platforms with
unreliable ldexp implementations.

@item 
bpo-24254@footnote{https://bugs.python.org/issue24254}: Make class definition namespace ordered by default.

@item 
bpo-27662@footnote{https://bugs.python.org/issue27662}: Fix an overflow check in @code{List_New}: the original code was
checking against @code{Py_SIZE_MAX} instead of the correct upper bound of
@code{Py_SSIZE_T_MAX}. Patch by Xiang Zhang.

@item 
bpo-27782@footnote{https://bugs.python.org/issue27782}: Multi-phase extension module import now correctly allows the
@code{m_methods} field to be used to add module level functions to instances
of non-module types returned from @code{Py_create_mod}. Patch by Xiang Zhang.

@item 
bpo-27936@footnote{https://bugs.python.org/issue27936}: The round() function accepted a second None argument for some
types but not for others.  Fixed the inconsistency by accepting None for
all numeric types.

@item 
bpo-27487@footnote{https://bugs.python.org/issue27487}: Warn if a submodule argument to “python -m” or
runpy.run_module() is found in sys.modules after parent packages are
imported, but before the submodule is executed.

@item 
bpo-27157@footnote{https://bugs.python.org/issue27157}: Make only type() itself accept the one-argument form. Patch by
Eryk Sun and Emanuel Barry.

@item 
bpo-27558@footnote{https://bugs.python.org/issue27558}: Fix a SystemError in the implementation of “raise” statement.
In a brand new thread, raise a RuntimeError since there is no active
exception to reraise. Patch written by Xiang Zhang.

@item 
bpo-28008@footnote{https://bugs.python.org/issue28008}: Implement 
@geindex Python Enhancement Proposals; PEP 530
PEP 530@footnote{https://www.python.org/dev/peps/pep-0530} – asynchronous comprehensions.

@item 
bpo-27942@footnote{https://bugs.python.org/issue27942}: Fix memory leak in codeobject.c
@end itemize

@node Library<50>,IDLE<39>,Core and Builtins<49>,Python 3 6 0 beta 1
@anchor{whatsnew/changelog id417}@anchor{13f7}
@subsubsection Library



@itemize -

@item 
bpo-28732@footnote{https://bugs.python.org/issue28732}: Fix crash in os.spawnv() with no elements in args

@item 
bpo-28485@footnote{https://bugs.python.org/issue28485}: Always raise ValueError for negative
compileall.compile_dir(workers=…) parameter, even when multithreading is
unavailable.

@item 
bpo-28037@footnote{https://bugs.python.org/issue28037}: Use sqlite3_get_autocommit() instead of setting
Connection->inTransaction manually.

@item 
bpo-25283@footnote{https://bugs.python.org/issue25283}: Attributes tm_gmtoff and tm_zone are now available on all
platforms in the return values of time.localtime() and time.gmtime().

@item 
bpo-24454@footnote{https://bugs.python.org/issue24454}: Regular expression match object groups are now accessible using
__getitem__. “mo[x]” is equivalent to “mo.group(x)”.

@item 
bpo-10740@footnote{https://bugs.python.org/issue10740}: sqlite3 no longer implicitly commit an open transaction before
DDL statements.

@item 
bpo-17941@footnote{https://bugs.python.org/issue17941}: Add a @emph{module} parameter to collections.namedtuple().

@item 
bpo-22493@footnote{https://bugs.python.org/issue22493}: Inline flags now should be used only at the start of the
regular expression. Deprecation warning is emitted if uses them in the
middle of the regular expression.

@item 
bpo-26885@footnote{https://bugs.python.org/issue26885}: xmlrpc now supports unmarshalling additional data types used by
Apache XML-RPC implementation for numerics and None.

@item 
bpo-28070@footnote{https://bugs.python.org/issue28070}: Fixed parsing inline verbose flag in regular expressions.

@item 
bpo-19500@footnote{https://bugs.python.org/issue19500}: Add client-side SSL session resumption to the ssl module.

@item 
bpo-28022@footnote{https://bugs.python.org/issue28022}: Deprecate ssl-related arguments in favor of SSLContext. The
deprecation include manual creation of SSLSocket and certfile/keyfile (or
similar) in ftplib, httplib, imaplib, smtplib, poplib and urllib.

@item 
bpo-28043@footnote{https://bugs.python.org/issue28043}: SSLContext has improved default settings: OP_NO_SSLv2,
OP_NO_SSLv3, OP_NO_COMPRESSION, OP_CIPHER_SERVER_PREFERENCE,
OP_SINGLE_DH_USE, OP_SINGLE_ECDH_USE and HIGH ciphers without MD5.

@item 
bpo-24693@footnote{https://bugs.python.org/issue24693}: Changed some RuntimeError’s in the zipfile module to more
appropriate types. Improved some error messages and debugging output.

@item 
bpo-17909@footnote{https://bugs.python.org/issue17909}: @code{json.load} and @code{json.loads} now support binary input
encoded as UTF-8, UTF-16 or UTF-32. Patch by Serhiy Storchaka.

@item 
bpo-27137@footnote{https://bugs.python.org/issue27137}: the pure Python fallback implementation of
@code{functools.partial} now matches the behaviour of its accelerated C
counterpart for subclassing, pickling and text representation purposes.
Patch by Emanuel Barry and Serhiy Storchaka.

@item 
Fix possible integer overflows and crashes in the mmap module with unusual
usage patterns.

@item 
bpo-1703178@footnote{https://bugs.python.org/issue1703178}: Fix the ability to pass the –link-objects option to the
distutils build_ext command.

@item 
bpo-28019@footnote{https://bugs.python.org/issue28019}: itertools.count() no longer rounds non-integer step in range
between 1.0 and 2.0 to 1.

@item 
bpo-18401@footnote{https://bugs.python.org/issue18401}: Pdb now supports the ‘readrc’ keyword argument to control
whether .pdbrc files should be read.  Patch by Martin Matusiak and Sam
Kimbrel.

@item 
bpo-25969@footnote{https://bugs.python.org/issue25969}: Update the lib2to3 grammar to handle the unpacking
generalizations added in 3.5.

@item 
bpo-14977@footnote{https://bugs.python.org/issue14977}: mailcap now respects the order of the lines in the mailcap
files (“first match”), as required by RFC 1542.  Patch by Michael Lazar.

@item 
bpo-28082@footnote{https://bugs.python.org/issue28082}: Convert re flag constants to IntFlag.

@item 
bpo-28025@footnote{https://bugs.python.org/issue28025}: Convert all ssl module constants to IntEnum and IntFlags.
SSLContext properties now return flags and enums.

@item 
bpo-23591@footnote{https://bugs.python.org/issue23591}: Add Flag, IntFlag, and auto() to enum module.

@item 
bpo-433028@footnote{https://bugs.python.org/issue433028}: Added support of modifier spans in regular expressions.

@item 
bpo-24594@footnote{https://bugs.python.org/issue24594}: Validates persist parameter when opening MSI database

@item 
bpo-17582@footnote{https://bugs.python.org/issue17582}: xml.etree.ElementTree nows preserves whitespaces in attributes
(Patch by Duane Griffin.  Reviewed and approved by Stefan Behnel.)

@item 
bpo-28047@footnote{https://bugs.python.org/issue28047}: Fixed calculation of line length used for the base64 CTE in the
new email policies.

@item 
bpo-27576@footnote{https://bugs.python.org/issue27576}: Fix call order in OrderedDict.__init__().

@item 
email.generator.DecodedGenerator now supports the policy keyword.

@item 
bpo-28027@footnote{https://bugs.python.org/issue28027}: Remove undocumented modules from @code{Lib/plat-*}: IN, CDROM,
DLFCN, TYPES, CDIO, and STROPTS.

@item 
bpo-27445@footnote{https://bugs.python.org/issue27445}: Don’t pass str(_charset) to MIMEText.set_payload(). Patch by
Claude Paroz.

@item 
bpo-24277@footnote{https://bugs.python.org/issue24277}: The new email API is no longer provisional, and the docs have
been reorganized and rewritten to emphasize the new API.

@item 
bpo-22450@footnote{https://bugs.python.org/issue22450}: urllib now includes an @code{Accept: */*} header among the default
headers. This makes the results of REST API requests more consistent and
predictable especially when proxy servers are involved.

@item 
lib2to3.pgen3.driver.load_grammar() now creates a stable cache file
between runs given the same Grammar.txt input regardless of the hash
randomization setting.

@item 
bpo-28005@footnote{https://bugs.python.org/issue28005}: Allow ImportErrors in encoding implementation to propagate.

@item 
bpo-26667@footnote{https://bugs.python.org/issue26667}: Support path-like objects in importlib.util.

@item 
bpo-27570@footnote{https://bugs.python.org/issue27570}: Avoid zero-length memcpy() etc calls with null source pointers
in the “ctypes” and “array” modules.

@item 
bpo-22233@footnote{https://bugs.python.org/issue22233}: Break email header lines @emph{only} on the RFC specified CR and LF
characters, not on arbitrary unicode line breaks.  This also fixes a bug
in HTTP header parsing.

@item 
bpo-27331@footnote{https://bugs.python.org/issue27331}: The email.mime classes now all accept an optional policy
keyword.

@item 
bpo-27988@footnote{https://bugs.python.org/issue27988}: Fix email iter_attachments incorrect mutation of payload list.

@item 
bpo-16113@footnote{https://bugs.python.org/issue16113}: Add SHA-3 and SHAKE support to hashlib module.

@item 
Eliminate a tautological-pointer-compare warning in _scproxy.c.

@item 
bpo-27776@footnote{https://bugs.python.org/issue27776}: The @ref{17f,,os.urandom()} function does now block on Linux 3.17
and newer until the system urandom entropy pool is initialized to increase
the security. This change is part of the 
@geindex Python Enhancement Proposals; PEP 524
PEP 524@footnote{https://www.python.org/dev/peps/pep-0524}.

@item 
bpo-27778@footnote{https://bugs.python.org/issue27778}: Expose the Linux @code{getrandom()} syscall as a new
@ref{806,,os.getrandom()} function. This change is part of the 
@geindex Python Enhancement Proposals; PEP 524
PEP 524@footnote{https://www.python.org/dev/peps/pep-0524}.

@item 
bpo-27691@footnote{https://bugs.python.org/issue27691}: Fix ssl module’s parsing of GEN_RID subject alternative name
fields in X.509 certs.

@item 
bpo-18844@footnote{https://bugs.python.org/issue18844}: Add random.choices().

@item 
bpo-25761@footnote{https://bugs.python.org/issue25761}: Improved error reporting about truncated pickle data in C
implementation of unpickler.  UnpicklingError is now raised instead of
AttributeError and ValueError in some cases.

@item 
bpo-26798@footnote{https://bugs.python.org/issue26798}: Add BLAKE2 (blake2b and blake2s) to hashlib.

@item 
bpo-26032@footnote{https://bugs.python.org/issue26032}: Optimized globbing in pathlib by using os.scandir(); it is now
about 1.5–4 times faster.

@item 
bpo-25596@footnote{https://bugs.python.org/issue25596}: Optimized glob() and iglob() functions in the glob module; they
are now about 3–6 times faster.

@item 
bpo-27928@footnote{https://bugs.python.org/issue27928}: Add scrypt (password-based key derivation function) to hashlib
module (requires OpenSSL 1.1.0).

@item 
bpo-27850@footnote{https://bugs.python.org/issue27850}: Remove 3DES from ssl module’s default cipher list to counter
measure sweet32 attack (CVE-2016-2183).

@item 
bpo-27766@footnote{https://bugs.python.org/issue27766}: Add ChaCha20 Poly1305 to ssl module’s default cipher list.
(Required OpenSSL 1.1.0 or LibreSSL).

@item 
bpo-25387@footnote{https://bugs.python.org/issue25387}: Check return value of winsound.MessageBeep.

@item 
bpo-27866@footnote{https://bugs.python.org/issue27866}: Add SSLContext.get_ciphers() method to get a list of all
enabled ciphers.

@item 
bpo-27744@footnote{https://bugs.python.org/issue27744}: Add AF_ALG (Linux Kernel crypto) to socket module.

@item 
bpo-26470@footnote{https://bugs.python.org/issue26470}: Port ssl and hashlib module to OpenSSL 1.1.0.

@item 
bpo-11620@footnote{https://bugs.python.org/issue11620}: Fix support for SND_MEMORY in winsound.PlaySound.  Based on a
patch by Tim Lesher.

@item 
bpo-11734@footnote{https://bugs.python.org/issue11734}: Add support for IEEE 754 half-precision floats to the struct
module. Based on a patch by Eli Stevens.

@item 
bpo-27919@footnote{https://bugs.python.org/issue27919}: Deprecated @code{extra_path} distribution option in distutils
packaging.

@item 
bpo-23229@footnote{https://bugs.python.org/issue23229}: Add new @code{cmath} constants: @code{cmath.inf} and @code{cmath.nan} to
match @code{math.inf} and @code{math.nan}, and also @code{cmath.infj} and
@code{cmath.nanj} to match the format used by complex repr.

@item 
bpo-27842@footnote{https://bugs.python.org/issue27842}: The csv.DictReader now returns rows of type OrderedDict.
(Contributed by Steve Holden.)

@item 
Remove support for passing a file descriptor to os.access. It never worked
but previously didn’t raise.

@item 
bpo-12885@footnote{https://bugs.python.org/issue12885}: Fix error when distutils encounters symlink.

@item 
bpo-27881@footnote{https://bugs.python.org/issue27881}: Fixed possible bugs when setting
sqlite3.Connection.isolation_level. Based on patch by Xiang Zhang.

@item 
bpo-27861@footnote{https://bugs.python.org/issue27861}: Fixed a crash in sqlite3.Connection.cursor() when a factory
creates not a cursor.  Patch by Xiang Zhang.

@item 
bpo-19884@footnote{https://bugs.python.org/issue19884}: Avoid spurious output on OS X with Gnu Readline.

@item 
bpo-27706@footnote{https://bugs.python.org/issue27706}: Restore deterministic behavior of random.Random().seed() for
string seeds using seeding version 1.  Allows sequences of calls to
random() to exactly match those obtained in Python 2. Patch by Nofar
Schnider.

@item 
bpo-10513@footnote{https://bugs.python.org/issue10513}: Fix a regression in Connection.commit().  Statements should not
be reset after a commit.

@item 
bpo-12319@footnote{https://bugs.python.org/issue12319}: Chunked transfer encoding support added to
http.client.HTTPConnection requests.  The
urllib.request.AbstractHTTPHandler class does not enforce a Content-Length
header any more.  If a HTTP request has a file or iterable body, but no
Content-Length header, the library now falls back to use chunked
transfer-encoding.

@item 
A new version of typing.py from @indicateurl{https://github.com/python/typing}: -
Collection (only for 3.6) (bpo-27598@footnote{https://bugs.python.org/issue27598}) - Add FrozenSet to __all__
(upstream #261) - fix crash in _get_type_vars() (upstream #259) - Remove
the dict constraint in ForwardRef._eval_type (upstream #252)

@item 
bpo-27832@footnote{https://bugs.python.org/issue27832}: Make @code{_normalize} parameter to @code{Fraction} constructor
keyword-only, so that @code{Fraction(2, 3, 4)} now raises @code{TypeError}.

@item 
bpo-27539@footnote{https://bugs.python.org/issue27539}: Fix unnormalised @code{Fraction.__pow__} result in the case of
negative exponent and negative base.

@item 
bpo-21718@footnote{https://bugs.python.org/issue21718}: cursor.description is now available for queries using CTEs.

@item 
bpo-27819@footnote{https://bugs.python.org/issue27819}: In distutils sdists, simply produce the “gztar” (gzipped tar
format) distributions on all platforms unless “formats” is supplied.

@item 
bpo-2466@footnote{https://bugs.python.org/issue2466}: posixpath.ismount now correctly recognizes mount points which
the user does not have permission to access.

@item 
bpo-9998@footnote{https://bugs.python.org/issue9998}: On Linux, ctypes.util.find_library now looks in LD_LIBRARY_PATH
for shared libraries.

@item 
bpo-27573@footnote{https://bugs.python.org/issue27573}: exit message for code.interact is now configurable.

@item 
bpo-27930@footnote{https://bugs.python.org/issue27930}: Improved behaviour of logging.handlers.QueueListener. Thanks to
Paulo Andrade and Petr Viktorin for the analysis and patch.

@item 
bpo-6766@footnote{https://bugs.python.org/issue6766}: Distributed reference counting added to multiprocessing to
support nesting of shared values / proxy objects.

@item 
bpo-21201@footnote{https://bugs.python.org/issue21201}: Improves readability of multiprocessing error message.  Thanks
to Wojciech Walczak for patch.

@item 
asyncio: Add set_protocol / get_protocol to Transports.

@item 
bpo-27456@footnote{https://bugs.python.org/issue27456}: asyncio: Set TCP_NODELAY by default.
@end itemize

@node IDLE<39>,C API<40>,Library<50>,Python 3 6 0 beta 1
@anchor{whatsnew/changelog id418}@anchor{13f8}
@subsubsection IDLE



@itemize -

@item 
bpo-15308@footnote{https://bugs.python.org/issue15308}: Add ‘interrupt execution’ (^C) to Shell menu. Patch by Roger
Serwy, updated by Bayard Randel.

@item 
bpo-27922@footnote{https://bugs.python.org/issue27922}: Stop IDLE tests from ‘flashing’ gui widgets on the screen.

@item 
bpo-27891@footnote{https://bugs.python.org/issue27891}: Consistently group and sort imports within idlelib modules.

@item 
bpo-17642@footnote{https://bugs.python.org/issue17642}: add larger font sizes for classroom projection.

@item 
Add version to title of IDLE help window.

@item 
bpo-25564@footnote{https://bugs.python.org/issue25564}: In section on IDLE – console differences, mention that using
exec means that __builtins__ is defined for each statement.

@item 
bpo-27821@footnote{https://bugs.python.org/issue27821}: Fix 3.6.0a3 regression that prevented custom key sets from
being selected when no custom theme was defined.
@end itemize

@node C API<40>,Tests<44>,IDLE<39>,Python 3 6 0 beta 1
@anchor{whatsnew/changelog id419}@anchor{13f9}
@subsubsection C API



@itemize -

@item 
bpo-26900@footnote{https://bugs.python.org/issue26900}: Excluded underscored names and other private API from limited
API.

@item 
bpo-26027@footnote{https://bugs.python.org/issue26027}: Add support for path-like objects in PyUnicode_FSConverter() &
PyUnicode_FSDecoder().
@end itemize

@node Tests<44>,Build<48>,C API<40>,Python 3 6 0 beta 1
@anchor{whatsnew/changelog id420}@anchor{13fa}
@subsubsection Tests



@itemize -

@item 
bpo-27427@footnote{https://bugs.python.org/issue27427}: Additional tests for the math module. Patch by Francisco Couzo.

@item 
bpo-27953@footnote{https://bugs.python.org/issue27953}: Skip math and cmath tests that fail on OS X 10.4 due to a poor
libm implementation of tan.

@item 
bpo-26040@footnote{https://bugs.python.org/issue26040}: Improve test_math and test_cmath coverage and rigour. Patch by
Jeff Allen.

@item 
bpo-27787@footnote{https://bugs.python.org/issue27787}: Call gc.collect() before checking each test for “dangling
threads”, since the dangling threads are weak references.
@end itemize

@node Build<48>,Tools/Demos<25>,Tests<44>,Python 3 6 0 beta 1
@anchor{whatsnew/changelog id421}@anchor{13fb}
@subsubsection Build



@itemize -

@item 
bpo-27566@footnote{https://bugs.python.org/issue27566}: Fix clean target in freeze makefile (patch by Lisa Roach)

@item 
bpo-27705@footnote{https://bugs.python.org/issue27705}: Update message in validate_ucrtbase.py

@item 
bpo-27976@footnote{https://bugs.python.org/issue27976}: Deprecate building _ctypes with the bundled copy of libffi on
non-OSX UNIX platforms.

@item 
bpo-27983@footnote{https://bugs.python.org/issue27983}: Cause lack of llvm-profdata tool when using clang as required
for PGO linking to be a configure time error rather than make time when
@code{--with-optimizations} is enabled.  Also improve our ability to find the
llvm-profdata tool on MacOS and some Linuxes.

@item 
bpo-21590@footnote{https://bugs.python.org/issue21590}: Support for DTrace and SystemTap probes.

@item 
bpo-26307@footnote{https://bugs.python.org/issue26307}: The profile-opt build now applies PGO to the built-in modules.

@item 
bpo-26359@footnote{https://bugs.python.org/issue26359}: Add the –with-optimizations flag to turn on LTO and PGO build
support when available.

@item 
bpo-27917@footnote{https://bugs.python.org/issue27917}: Set platform triplets for Android builds.

@item 
bpo-25825@footnote{https://bugs.python.org/issue25825}: Update references to the $(LIBPL) installation path on AIX.
This path was changed in 3.2a4.

@item 
Update OS X installer to use SQLite 3.14.1 and XZ 5.2.2.

@item 
bpo-21122@footnote{https://bugs.python.org/issue21122}: Fix LTO builds on OS X.

@item 
bpo-17128@footnote{https://bugs.python.org/issue17128}: Build OS X installer with a private copy of OpenSSL. Also
provide a sample Install Certificates command script to install a set of
root certificates from the third-party certifi module.
@end itemize

@node Tools/Demos<25>,Windows<42>,Build<48>,Python 3 6 0 beta 1
@anchor{whatsnew/changelog id422}@anchor{13fc}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-27952@footnote{https://bugs.python.org/issue27952}: Get Tools/scripts/fixcid.py working with Python 3 and the
current “re” module, avoid invalid Python backslash escapes, and fix a bug
parsing escaped C quote signs.
@end itemize

@node Windows<42>,,Tools/Demos<25>,Python 3 6 0 beta 1
@anchor{whatsnew/changelog id423}@anchor{13fd}
@subsubsection Windows



@itemize -

@item 
bpo-28065@footnote{https://bugs.python.org/issue28065}: Update xz dependency to 5.2.2 and build it from source.

@item 
bpo-25144@footnote{https://bugs.python.org/issue25144}: Ensures TargetDir is set before continuing with custom install.

@item 
bpo-1602@footnote{https://bugs.python.org/issue1602}: Windows console doesn’t input or print Unicode (PEP 528)

@item 
bpo-27781@footnote{https://bugs.python.org/issue27781}: Change file system encoding on Windows to UTF-8 (PEP 529)

@item 
bpo-27731@footnote{https://bugs.python.org/issue27731}: Opt-out of MAX_PATH on Windows 10

@item 
bpo-6135@footnote{https://bugs.python.org/issue6135}: Adds encoding and errors parameters to subprocess.

@item 
bpo-27959@footnote{https://bugs.python.org/issue27959}: Adds oem encoding, alias ansi to mbcs, move aliasmbcs to codec
lookup.

@item 
bpo-27982@footnote{https://bugs.python.org/issue27982}: The functions of the winsound module now accept keyword
arguments.

@item 
bpo-20366@footnote{https://bugs.python.org/issue20366}: Build full text search support into SQLite on Windows.

@item 
bpo-27756@footnote{https://bugs.python.org/issue27756}: Adds new icons for Python files and processes on Windows.
Designs by Cherry Wang.

@item 
bpo-27883@footnote{https://bugs.python.org/issue27883}: Update sqlite to 3.14.1.0 on Windows.
@end itemize

@node Python 3 6 0 alpha 4,Python 3 6 0 alpha 3,Python 3 6 0 beta 1,Changelog
@anchor{whatsnew/changelog python-3-6-0-alpha-4}@anchor{13fe}
@subsection Python 3.6.0 alpha 4


@emph{Release date: 2016-08-15}

@menu
* Core and Builtins: Core and Builtins<50>. 
* Library: Library<51>. 
* IDLE: IDLE<40>. 
* Tests: Tests<45>. 
* Windows: Windows<43>. 
* Build: Build<49>. 

@end menu

@node Core and Builtins<50>,Library<51>,,Python 3 6 0 alpha 4
@anchor{whatsnew/changelog id424}@anchor{13ff}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-27704@footnote{https://bugs.python.org/issue27704}: Optimized creating bytes and bytearray from byte-like objects
and iterables. Speed up to 3 times for short objects.  Original patch by
Naoki Inada.

@item 
bpo-26823@footnote{https://bugs.python.org/issue26823}: Large sections of repeated lines in tracebacks are now
abbreviated as “[Previous line repeated @{count@} more times]” by the
builtin traceback rendering. Patch by Emanuel Barry.

@item 
bpo-27574@footnote{https://bugs.python.org/issue27574}: Decreased an overhead of parsing keyword arguments in functions
implemented with using Argument Clinic.

@item 
bpo-22557@footnote{https://bugs.python.org/issue22557}: Now importing already imported modules is up to 2.5 times
faster.

@item 
bpo-17596@footnote{https://bugs.python.org/issue17596}: Include <wincrypt.h> to help with Min GW building.

@item 
bpo-17599@footnote{https://bugs.python.org/issue17599}: On Windows, rename the privately defined REPARSE_DATA_BUFFER
structure to avoid conflicting with the definition from Min GW.

@item 
bpo-27507@footnote{https://bugs.python.org/issue27507}: Add integer overflow check in bytearray.extend().  Patch by
Xiang Zhang.

@item 
bpo-27581@footnote{https://bugs.python.org/issue27581}: Don’t rely on wrapping for overflow check in
PySequence_Tuple().  Patch by Xiang Zhang.

@item 
bpo-1621@footnote{https://bugs.python.org/issue1621}: Avoid signed integer overflow in list and tuple operations.
Patch by Xiang Zhang.

@item 
bpo-27419@footnote{https://bugs.python.org/issue27419}: Standard __import__() no longer look up “__import__” in globals
or builtins for importing submodules or “from import”.  Fixed a crash if
raise a warning about unabling to resolve package from __spec__ or
__package__.

@item 
bpo-27083@footnote{https://bugs.python.org/issue27083}: Respect the PYTHONCASEOK environment variable under Windows.

@item 
bpo-27514@footnote{https://bugs.python.org/issue27514}: Make having too many statically nested blocks a SyntaxError
instead of SystemError.

@item 
bpo-27366@footnote{https://bugs.python.org/issue27366}: Implemented 
@geindex Python Enhancement Proposals; PEP 487
PEP 487@footnote{https://www.python.org/dev/peps/pep-0487} (Simpler customization of class
creation). Upon subclassing, the __init_subclass__ classmethod is called
on the base class. Descriptors are initialized with __set_name__ after
class creation.
@end itemize

@node Library<51>,IDLE<40>,Core and Builtins<50>,Python 3 6 0 alpha 4
@anchor{whatsnew/changelog id425}@anchor{1400}
@subsubsection Library



@itemize -

@item 
bpo-26027@footnote{https://bugs.python.org/issue26027}: Add 
@geindex Python Enhancement Proposals; PEP 519
PEP 519@footnote{https://www.python.org/dev/peps/pep-0519}/__fspath__() support to the os and os.path
modules. Includes code from Jelle Zijlstra. (See also: bpo-27524@footnote{https://bugs.python.org/issue27524})

@item 
bpo-27598@footnote{https://bugs.python.org/issue27598}: Add Collections to collections.abc. Patch by Ivan Levkivskyi,
docs by Neil Girdhar.

@item 
bpo-25958@footnote{https://bugs.python.org/issue25958}: Support “anti-registration” of special methods from various
ABCs, like __hash__, __iter__ or __len__.  All these (and several more)
can be set to None in an implementation class and the behavior will be as
if the method is not defined at all. (Previously, this mechanism existed
only for __hash__, to make mutable classes unhashable.)  Code contributed
by Andrew Barnert and Ivan Levkivskyi.

@item 
bpo-16764@footnote{https://bugs.python.org/issue16764}: Support keyword arguments to zlib.decompress().  Patch by Xiang
Zhang.

@item 
bpo-27736@footnote{https://bugs.python.org/issue27736}: Prevent segfault after interpreter re-initialization due to ref
count problem introduced in code for bpo-27038@footnote{https://bugs.python.org/issue27038} in 3.6.0a3. Patch by
Xiang Zhang.

@item 
bpo-25628@footnote{https://bugs.python.org/issue25628}: The @emph{verbose} and @emph{rename} parameters for
collections.namedtuple are now keyword-only.

@item 
bpo-12345@footnote{https://bugs.python.org/issue12345}: Add mathematical constant tau to math and cmath. See also
@geindex Python Enhancement Proposals; PEP 628
PEP 628@footnote{https://www.python.org/dev/peps/pep-0628}.

@item 
bpo-26823@footnote{https://bugs.python.org/issue26823}: traceback.StackSummary.format now abbreviates large sections of
repeated lines as “[Previous line repeated @{count@} more times]” (this
change then further affects other traceback display operations in the
module). Patch by Emanuel Barry.

@item 
bpo-27664@footnote{https://bugs.python.org/issue27664}: Add to concurrent.futures.thread.ThreadPoolExecutor() the
ability to specify a thread name prefix.

@item 
bpo-27181@footnote{https://bugs.python.org/issue27181}: Add geometric_mean and harmonic_mean to statistics module.

@item 
bpo-27573@footnote{https://bugs.python.org/issue27573}: code.interact now prints an message when exiting.

@item 
bpo-6422@footnote{https://bugs.python.org/issue6422}: Add autorange method to timeit.Timer objects.

@item 
bpo-27773@footnote{https://bugs.python.org/issue27773}: Correct some memory management errors server_hostname in
_ssl.wrap_socket().

@item 
bpo-26750@footnote{https://bugs.python.org/issue26750}: unittest.mock.create_autospec() now works properly for
subclasses of property() and other data descriptors.  Removes the never
publicly used, never documented unittest.mock.DescriptorTypes tuple.

@item 
bpo-26754@footnote{https://bugs.python.org/issue26754}: Undocumented support of general bytes-like objects as path in
compile() and similar functions is now deprecated.

@item 
bpo-26800@footnote{https://bugs.python.org/issue26800}: Undocumented support of general bytes-like objects as paths in
os functions is now deprecated.

@item 
bpo-26981@footnote{https://bugs.python.org/issue26981}: Add _order_ compatibility shim to enum.Enum for Python 2/3 code
bases.

@item 
bpo-27661@footnote{https://bugs.python.org/issue27661}: Added tzinfo keyword argument to datetime.combine.

@item 
In the curses module, raise an error if window.getstr() or window.instr()
is passed a negative value.

@item 
bpo-27783@footnote{https://bugs.python.org/issue27783}: Fix possible usage of uninitialized memory in
operator.methodcaller.

@item 
bpo-27774@footnote{https://bugs.python.org/issue27774}: Fix possible Py_DECREF on unowned object in _sre.

@item 
bpo-27760@footnote{https://bugs.python.org/issue27760}: Fix possible integer overflow in binascii.b2a_qp.

@item 
bpo-27758@footnote{https://bugs.python.org/issue27758}: Fix possible integer overflow in the _csv module for large
record lengths.

@item 
bpo-27568@footnote{https://bugs.python.org/issue27568}: Prevent HTTPoxy attack (CVE-2016-1000110). Ignore the
HTTP_PROXY variable when REQUEST_METHOD environment is set, which
indicates that the script is in CGI mode.

@item 
bpo-7063@footnote{https://bugs.python.org/issue7063}: Remove dead code from the “array” module’s slice handling. Patch
by Chuck.

@item 
bpo-27656@footnote{https://bugs.python.org/issue27656}: Do not assume sched.h defines any SCHED_* constants.

@item 
bpo-27130@footnote{https://bugs.python.org/issue27130}: In the “zlib” module, fix handling of large buffers (typically
4 GiB) when compressing and decompressing.  Previously, inputs were
limited to 4 GiB, and compression and decompression operations did not
properly handle results of 4 GiB.

@item 
bpo-24773@footnote{https://bugs.python.org/issue24773}: Implemented 
@geindex Python Enhancement Proposals; PEP 495
PEP 495@footnote{https://www.python.org/dev/peps/pep-0495} (Local Time Disambiguation).

@item 
Expose the EPOLLEXCLUSIVE constant (when it is defined) in the select
module.

@item 
bpo-27567@footnote{https://bugs.python.org/issue27567}: Expose the EPOLLRDHUP and POLLRDHUP constants in the select
module.

@item 
bpo-1621@footnote{https://bugs.python.org/issue1621}: Avoid signed int negation overflow in the “audioop” module.

@item 
bpo-27533@footnote{https://bugs.python.org/issue27533}: Release GIL in nt._isdir

@item 
bpo-17711@footnote{https://bugs.python.org/issue17711}: Fixed unpickling by the persistent ID with protocol 0. Original
patch by Alexandre Vassalotti.

@item 
bpo-27522@footnote{https://bugs.python.org/issue27522}: Avoid an unintentional reference cycle in email.feedparser.

@item 
bpo-27512@footnote{https://bugs.python.org/issue27512}: Fix a segfault when os.fspath() called an __fspath__() method
that raised an exception. Patch by Xiang Zhang.
@end itemize

@node IDLE<40>,Tests<45>,Library<51>,Python 3 6 0 alpha 4
@anchor{whatsnew/changelog id426}@anchor{1401}
@subsubsection IDLE



@itemize -

@item 
bpo-27714@footnote{https://bugs.python.org/issue27714}: text_textview and test_autocomplete now pass when re-run in the
same process.  This occurs when test_idle fails when run with the -w
option but without -jn.  Fix warning from test_config.

@item 
bpo-27621@footnote{https://bugs.python.org/issue27621}: Put query response validation error messages in the query box
itself instead of in a separate messagebox.  Redo tests to match. Add Mac
OSX refinements. Original patch by Mark Roseman.

@item 
bpo-27620@footnote{https://bugs.python.org/issue27620}: Escape key now closes Query box as cancelled.

@item 
bpo-27609@footnote{https://bugs.python.org/issue27609}: IDLE: tab after initial whitespace should tab, not
autocomplete. This fixes problem with writing docstrings at least twice
indented.

@item 
bpo-27609@footnote{https://bugs.python.org/issue27609}: Explicitly return None when there are also non-None returns. In
a few cases, reverse a condition and eliminate a return.

@item 
bpo-25507@footnote{https://bugs.python.org/issue25507}: IDLE no longer runs buggy code because of its tkinter imports.
Users must include the same imports required to run directly in Python.

@item 
bpo-27173@footnote{https://bugs.python.org/issue27173}: Add ‘IDLE Modern Unix’ to the built-in key sets. Make the
default key set depend on the platform. Add tests for the changes to the
config module.

@item 
bpo-27452@footnote{https://bugs.python.org/issue27452}: add line counter and crc to IDLE configHandler test dump.
@end itemize

@node Tests<45>,Windows<43>,IDLE<40>,Python 3 6 0 alpha 4
@anchor{whatsnew/changelog id427}@anchor{1402}
@subsubsection Tests



@itemize -

@item 
bpo-25805@footnote{https://bugs.python.org/issue25805}: Skip a test in test_pkgutil as needed that doesn’t work when
@code{__name__ == __main__}. Patch by SilentGhost.

@item 
bpo-27472@footnote{https://bugs.python.org/issue27472}: Add test.support.unix_shell as the path to the default shell.

@item 
bpo-27369@footnote{https://bugs.python.org/issue27369}: In test_pyexpat, avoid testing an error message detail that
changed in Expat 2.2.0.

@item 
bpo-27594@footnote{https://bugs.python.org/issue27594}: Prevent assertion error when running test_ast with coverage
enabled: ensure code object has a valid first line number. Patch suggested
by Ivan Levkivskyi.
@end itemize

@node Windows<43>,Build<49>,Tests<45>,Python 3 6 0 alpha 4
@anchor{whatsnew/changelog id428}@anchor{1403}
@subsubsection Windows



@itemize -

@item 
bpo-27647@footnote{https://bugs.python.org/issue27647}: Update bundled Tcl/Tk to 8.6.6.

@item 
bpo-27610@footnote{https://bugs.python.org/issue27610}: Adds 
@geindex Python Enhancement Proposals; PEP 514
PEP 514@footnote{https://www.python.org/dev/peps/pep-0514} metadata to Windows installer

@item 
bpo-27469@footnote{https://bugs.python.org/issue27469}: Adds a shell extension to the launcher so that drag and drop
works correctly.

@item 
bpo-27309@footnote{https://bugs.python.org/issue27309}: Enables proper Windows styles in python[w].exe manifest.
@end itemize

@node Build<49>,,Windows<43>,Python 3 6 0 alpha 4
@anchor{whatsnew/changelog id429}@anchor{1404}
@subsubsection Build



@itemize -

@item 
bpo-27713@footnote{https://bugs.python.org/issue27713}: Suppress spurious build warnings when updating importlib’s
bootstrap files. Patch by Xiang Zhang

@item 
bpo-25825@footnote{https://bugs.python.org/issue25825}: Correct the references to Modules/python.exp, which is required
on AIX.  The references were accidentally changed in 3.5.0a1.

@item 
bpo-27453@footnote{https://bugs.python.org/issue27453}: CPP invocation in configure must use CPPFLAGS. Patch by Chi
Hsuan Yen.

@item 
bpo-27641@footnote{https://bugs.python.org/issue27641}: The configure script now inserts comments into the makefile to
prevent the pgen and _freeze_importlib executables from being
cross-compiled.

@item 
bpo-26662@footnote{https://bugs.python.org/issue26662}: Set PYTHON_FOR_GEN in configure as the Python program to be
used for file generation during the build.

@item 
bpo-10910@footnote{https://bugs.python.org/issue10910}: Avoid C++ compilation errors on FreeBSD and OS X. Also update
FreedBSD version checks for the original ctype UTF-8 workaround.
@end itemize

@node Python 3 6 0 alpha 3,Python 3 6 0 alpha 2,Python 3 6 0 alpha 4,Changelog
@anchor{whatsnew/changelog python-3-6-0-alpha-3}@anchor{1405}
@subsection Python 3.6.0 alpha 3


@emph{Release date: 2016-07-11}

@menu
* Core and Builtins: Core and Builtins<51>. 
* Library: Library<52>. 
* Security: Security<26>. 
* Library: Library<53>. 
* Security: Security<27>. 
* Library: Library<54>. 
* IDLE: IDLE<41>. 
* C API: C API<41>. 
* Build: Build<50>. 
* Tools/Demos: Tools/Demos<26>. 
* Documentation: Documentation<46>. 
* Tests: Tests<46>. 

@end menu

@node Core and Builtins<51>,Library<52>,,Python 3 6 0 alpha 3
@anchor{whatsnew/changelog id430}@anchor{1406}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-27473@footnote{https://bugs.python.org/issue27473}: Fixed possible integer overflow in bytes and bytearray
concatenations. Patch by Xiang Zhang.

@item 
bpo-23034@footnote{https://bugs.python.org/issue23034}: The output of a special Python build with defined COUNT_ALLOCS,
SHOW_ALLOC_COUNT or SHOW_TRACK_COUNT macros is now off by  default.  It
can be re-enabled using the “-X showalloccount” option.  It now outputs to
stderr instead of stdout.

@item 
bpo-27443@footnote{https://bugs.python.org/issue27443}: __length_hint__() of bytearray iterators no longer return a
negative integer for a resized bytearray.

@item 
bpo-27007@footnote{https://bugs.python.org/issue27007}: The fromhex() class methods of bytes and bytearray subclasses
now return an instance of corresponding subclass.
@end itemize

@node Library<52>,Security<26>,Core and Builtins<51>,Python 3 6 0 alpha 3
@anchor{whatsnew/changelog id431}@anchor{1407}
@subsubsection Library



@itemize -

@item 
bpo-26844@footnote{https://bugs.python.org/issue26844}: Fix error message for imp.find_module() to refer to ‘path’
instead of ‘name’. Patch by Lev Maximov.

@item 
bpo-23804@footnote{https://bugs.python.org/issue23804}: Fix SSL zero-length recv() calls to not block and not raise an
error about unclean EOF.

@item 
bpo-27466@footnote{https://bugs.python.org/issue27466}: Change time format returned by http.cookie.time2netscape,
confirming the netscape cookie format and making it consistent with
documentation.

@item 
bpo-21708@footnote{https://bugs.python.org/issue21708}: Deprecated dbm.dumb behavior that differs from common dbm
behavior: creating a database in ‘r’ and ‘w’ modes and modifying a
database in ‘r’ mode.

@item 
bpo-26721@footnote{https://bugs.python.org/issue26721}: Change the socketserver.StreamRequestHandler.wfile attribute to
implement BufferedIOBase. In particular, the write() method no longer does
partial writes.

@item 
bpo-22115@footnote{https://bugs.python.org/issue22115}: Added methods trace_add, trace_remove and trace_info in the
tkinter.Variable class.  They replace old methods trace_variable, trace,
trace_vdelete and trace_vinfo that use obsolete Tcl commands and might not
work in future versions of Tcl.  Fixed old tracing methods:
trace_vdelete() with wrong mode no longer break tracing, trace_vinfo() now
always returns a list of pairs of strings, tracing in the “u” mode now
works.

@item 
bpo-26243@footnote{https://bugs.python.org/issue26243}: Only the level argument to zlib.compress() is keyword argument
now.  The first argument is positional-only.

@item 
bpo-27038@footnote{https://bugs.python.org/issue27038}: Expose the DirEntry type as os.DirEntry. Code patch by Jelle
Zijlstra.

@item 
bpo-27186@footnote{https://bugs.python.org/issue27186}: Update os.fspath()/PyOS_FSPath() to check the return value of
__fspath__() to be either str or bytes.

@item 
bpo-18726@footnote{https://bugs.python.org/issue18726}: All optional parameters of the dump(), dumps(), load() and
loads() functions and JSONEncoder and JSONDecoder class constructors in
the json module are now keyword-only.

@item 
bpo-27319@footnote{https://bugs.python.org/issue27319}: Methods selection_set(), selection_add(), selection_remove()
and selection_toggle() of ttk.TreeView now allow passing multiple items as
multiple arguments instead of passing them as a tuple.  Deprecated
undocumented ability of calling the selection() method with arguments.

@item 
bpo-27079@footnote{https://bugs.python.org/issue27079}: Fixed curses.ascii functions isblank(), iscntrl() and
ispunct().

@item 
bpo-27294@footnote{https://bugs.python.org/issue27294}: Numerical state in the repr for Tkinter event objects is now
represented as a combination of known flags.

@item 
bpo-27177@footnote{https://bugs.python.org/issue27177}: Match objects in the re module now support index-like objects
as group indices.  Based on patches by Jeroen Demeyer and Xiang Zhang.

@item 
bpo-26754@footnote{https://bugs.python.org/issue26754}: Some functions (compile() etc) accepted a filename argument
encoded as an iterable of integers. Now only strings and byte-like objects
are accepted.

@item 
bpo-26536@footnote{https://bugs.python.org/issue26536}: socket.ioctl now supports SIO_LOOPBACK_FAST_PATH. Patch by
Daniel Stokes.

@item 
bpo-27048@footnote{https://bugs.python.org/issue27048}: Prevents distutils failing on Windows when environment
variables contain non-ASCII characters

@item 
bpo-27330@footnote{https://bugs.python.org/issue27330}: Fixed possible leaks in the ctypes module.

@item 
bpo-27238@footnote{https://bugs.python.org/issue27238}: Got rid of bare excepts in the turtle module.  Original patch
by Jelle Zijlstra.

@item 
bpo-27122@footnote{https://bugs.python.org/issue27122}: When an exception is raised within the context being managed by
a contextlib.ExitStack() and one of the exit stack generators catches and
raises it in a chain, do not re-raise the original exception when exiting,
let the new chained one through.  This avoids the 
@geindex Python Enhancement Proposals; PEP 479
PEP 479@footnote{https://www.python.org/dev/peps/pep-0479} bug described
in issue25782.
@end itemize

@node Security<26>,Library<53>,Library<52>,Python 3 6 0 alpha 3
@anchor{whatsnew/changelog id432}@anchor{1408}
@subsubsection Security



@itemize -

@item 
bpo-27278@footnote{https://bugs.python.org/issue27278}: Fix os.urandom() implementation using getrandom() on Linux.
Truncate size to INT_MAX and loop until we collected enough random bytes,
instead of casting a directly Py_ssize_t to int.
@end itemize

@node Library<53>,Security<27>,Security<26>,Python 3 6 0 alpha 3
@anchor{whatsnew/changelog id433}@anchor{1409}
@subsubsection Library



@itemize -

@item 
bpo-16864@footnote{https://bugs.python.org/issue16864}: sqlite3.Cursor.lastrowid now supports REPLACE statement.
Initial patch by Alex LordThorsen.

@item 
bpo-26386@footnote{https://bugs.python.org/issue26386}: Fixed ttk.TreeView selection operations with item id’s
containing spaces.

@item 
bpo-8637@footnote{https://bugs.python.org/issue8637}: Honor a pager set by the env var MANPAGER (in preference to one
set by the env var PAGER).
@end itemize

@node Security<27>,Library<54>,Library<53>,Python 3 6 0 alpha 3
@anchor{whatsnew/changelog id434}@anchor{140a}
@subsubsection Security



@itemize -

@item 
bpo-22636@footnote{https://bugs.python.org/issue22636}: Avoid shell injection problems with ctypes.util.find_library().
@end itemize

@node Library<54>,IDLE<41>,Security<27>,Python 3 6 0 alpha 3
@anchor{whatsnew/changelog id435}@anchor{140b}
@subsubsection Library



@itemize -

@item 
bpo-16182@footnote{https://bugs.python.org/issue16182}: Fix various functions in the “readline” module to use the
locale encoding, and fix get_begidx() and get_endidx() to return code
point indexes.

@item 
bpo-27392@footnote{https://bugs.python.org/issue27392}: Add loop.connect_accepted_socket(). Patch by Jim Fulton.
@end itemize

@node IDLE<41>,C API<41>,Library<54>,Python 3 6 0 alpha 3
@anchor{whatsnew/changelog id436}@anchor{140c}
@subsubsection IDLE



@itemize -

@item 
bpo-27477@footnote{https://bugs.python.org/issue27477}: IDLE search dialogs now use ttk widgets.

@item 
bpo-27173@footnote{https://bugs.python.org/issue27173}: Add ‘IDLE Modern Unix’ to the built-in key sets. Make the
default key set depend on the platform. Add tests for the changes to the
config module.

@item 
bpo-27452@footnote{https://bugs.python.org/issue27452}: make command line “idle-test> python test_help.py” work.
__file__ is relative when python is started in the file’s directory.

@item 
bpo-27452@footnote{https://bugs.python.org/issue27452}: add line counter and crc to IDLE configHandler test dump.

@item 
bpo-27380@footnote{https://bugs.python.org/issue27380}: IDLE: add query.py with base Query dialog and ttk widgets.
Module had subclasses SectionName, ModuleName, and HelpSource, which are
used to get information from users by configdialog and file =>Load Module.
Each subclass has itw own validity checks.  Using ModuleName allows users
to edit bad module names instead of starting over. Add tests and delete
the two files combined into the new one.

@item 
bpo-27372@footnote{https://bugs.python.org/issue27372}: Test_idle no longer changes the locale.

@item 
bpo-27365@footnote{https://bugs.python.org/issue27365}: Allow non-ascii chars in IDLE NEWS.txt, for contributor names.

@item 
bpo-27245@footnote{https://bugs.python.org/issue27245}: IDLE: Cleanly delete custom themes and key bindings.
Previously, when IDLE was started from a console or by import, a cascade
of warnings was emitted. Patch by Serhiy Storchaka.

@item 
bpo-24137@footnote{https://bugs.python.org/issue24137}: Run IDLE, test_idle, and htest with tkinter default root
disabled.  Fix code and tests that fail with this restriction.  Fix htests
to not create a second and redundant root and mainloop.

@item 
bpo-27310@footnote{https://bugs.python.org/issue27310}: Fix IDLE.app failure to launch on OS X due to vestigial import.
@end itemize

@node C API<41>,Build<50>,IDLE<41>,Python 3 6 0 alpha 3
@anchor{whatsnew/changelog id437}@anchor{140d}
@subsubsection C API



@itemize -

@item 
bpo-26754@footnote{https://bugs.python.org/issue26754}: PyUnicode_FSDecoder() accepted a filename argument encoded as
an iterable of integers. Now only strings and byte-like objects are
accepted.
@end itemize

@node Build<50>,Tools/Demos<26>,C API<41>,Python 3 6 0 alpha 3
@anchor{whatsnew/changelog id438}@anchor{140e}
@subsubsection Build



@itemize -

@item 
bpo-28066@footnote{https://bugs.python.org/issue28066}: Fix the logic that searches build directories for generated
include files when building outside the source tree.

@item 
bpo-27442@footnote{https://bugs.python.org/issue27442}: Expose the Android API level that python was built against, in
sysconfig.get_config_vars() as ‘ANDROID_API_LEVEL’.

@item 
bpo-27434@footnote{https://bugs.python.org/issue27434}: The interpreter that runs the cross-build, found in PATH, must
now be of the same feature version (e.g. 3.6) as the source being built.

@item 
bpo-26930@footnote{https://bugs.python.org/issue26930}: Update Windows builds to use OpenSSL 1.0.2h.

@item 
bpo-23968@footnote{https://bugs.python.org/issue23968}: Rename the platform directory from plat-$(MACHDEP) to
plat-$(PLATFORM_TRIPLET). Rename the config directory (LIBPL) from
config-$(LDVERSION) to config-$(LDVERSION)-$(PLATFORM_TRIPLET). Install
the platform specific _sysconfigdata module into the platform directory
and rename it to include the ABIFLAGS.

@item 
Don’t use largefile support for GNU/Hurd.
@end itemize

@node Tools/Demos<26>,Documentation<46>,Build<50>,Python 3 6 0 alpha 3
@anchor{whatsnew/changelog id439}@anchor{140f}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-27332@footnote{https://bugs.python.org/issue27332}: Fixed the type of the first argument of module-level functions
generated by Argument Clinic.  Patch by Petr Viktorin.

@item 
bpo-27418@footnote{https://bugs.python.org/issue27418}: Fixed Tools/importbench/importbench.py.
@end itemize

@node Documentation<46>,Tests<46>,Tools/Demos<26>,Python 3 6 0 alpha 3
@anchor{whatsnew/changelog id440}@anchor{1410}
@subsubsection Documentation



@itemize -

@item 
bpo-19489@footnote{https://bugs.python.org/issue19489}: Moved the search box from the sidebar to the header and footer
of each page. Patch by Ammar Askar.

@item 
bpo-27285@footnote{https://bugs.python.org/issue27285}: Update documentation to reflect the deprecation of @code{pyvenv}
and normalize on the term “virtual environment”. Patch by Steve Piercy.
@end itemize

@node Tests<46>,,Documentation<46>,Python 3 6 0 alpha 3
@anchor{whatsnew/changelog id441}@anchor{1411}
@subsubsection Tests



@itemize -

@item 
bpo-27027@footnote{https://bugs.python.org/issue27027}: Added test.support.is_android that is True when this is an
Android build.
@end itemize

@node Python 3 6 0 alpha 2,Python 3 6 0 alpha 1,Python 3 6 0 alpha 3,Changelog
@anchor{whatsnew/changelog python-3-6-0-alpha-2}@anchor{1412}
@subsection Python 3.6.0 alpha 2


@emph{Release date: 2016-06-13}

@menu
* Core and Builtins: Core and Builtins<52>. 
* Library: Library<55>. 
* Security: Security<28>. 
* Library: Library<56>. 
* Security: Security<29>. 
* Library: Library<57>. 
* IDLE: IDLE<42>. 
* Documentation: Documentation<47>. 
* Tests: Tests<47>. 
* Windows: Windows<44>. 
* Build: Build<51>. 
* Windows: Windows<45>. 
* C API: C API<42>. 
* Tools/Demos: Tools/Demos<27>. 

@end menu

@node Core and Builtins<52>,Library<55>,,Python 3 6 0 alpha 2
@anchor{whatsnew/changelog id442}@anchor{1413}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-27095@footnote{https://bugs.python.org/issue27095}: Simplified MAKE_FUNCTION and removed MAKE_CLOSURE opcodes.
Patch by Demur Rumed.

@item 
bpo-27190@footnote{https://bugs.python.org/issue27190}: Raise NotSupportedError if sqlite3 is older than 3.3.1. Patch
by Dave Sawyer.

@item 
bpo-27286@footnote{https://bugs.python.org/issue27286}: Fixed compiling BUILD_MAP_UNPACK_WITH_CALL opcode.  Calling
function with generalized unpacking (PEP 448) and conflicting keyword
names could cause undefined behavior.

@item 
bpo-27140@footnote{https://bugs.python.org/issue27140}: Added BUILD_CONST_KEY_MAP opcode.

@item 
bpo-27186@footnote{https://bugs.python.org/issue27186}: Add support for os.PathLike objects to open() (part of
@geindex Python Enhancement Proposals; PEP 519
PEP 519@footnote{https://www.python.org/dev/peps/pep-0519}).

@item 
bpo-27066@footnote{https://bugs.python.org/issue27066}: Fixed SystemError if a custom opener (for open()) returns a
negative number without setting an exception.

@item 
bpo-26983@footnote{https://bugs.python.org/issue26983}: float() now always return an instance of exact float. The
deprecation warning is emitted if __float__ returns an instance of a
strict subclass of float.  In a future versions of Python this can be an
error.

@item 
bpo-27097@footnote{https://bugs.python.org/issue27097}: Python interpreter is now about 7% faster due to optimized
instruction decoding.  Based on patch by Demur Rumed.

@item 
bpo-26647@footnote{https://bugs.python.org/issue26647}: Python interpreter now uses 16-bit wordcode instead of
bytecode. Patch by Demur Rumed.

@item 
bpo-23275@footnote{https://bugs.python.org/issue23275}: Allow assigning to an empty target list in round brackets: () =
iterable.

@item 
bpo-27243@footnote{https://bugs.python.org/issue27243}: Update the __aiter__ protocol: instead of returning an
awaitable that resolves to an asynchronous iterator, the asynchronous
iterator should be returned directly.  Doing the former will trigger a
PendingDeprecationWarning.
@end itemize

@node Library<55>,Security<28>,Core and Builtins<52>,Python 3 6 0 alpha 2
@anchor{whatsnew/changelog id443}@anchor{1414}
@subsubsection Library



@itemize -

@item 
Comment out socket (SO_REUSEPORT) and posix (O_SHLOCK, O_EXLOCK) constants
exposed on the API which are not implemented on GNU/Hurd. They would not
work at runtime anyway.

@item 
bpo-27025@footnote{https://bugs.python.org/issue27025}: Generated names for Tkinter widgets are now more meaningful and
recognizable.

@item 
bpo-25455@footnote{https://bugs.python.org/issue25455}: Fixed crashes in repr of recursive ElementTree.Element and
functools.partial objects.

@item 
bpo-27294@footnote{https://bugs.python.org/issue27294}: Improved repr for Tkinter event objects.

@item 
bpo-20508@footnote{https://bugs.python.org/issue20508}: Improve exception message of IPv@{4,6@}Network.__getitem__. Patch
by Gareth Rees.
@end itemize

@node Security<28>,Library<56>,Library<55>,Python 3 6 0 alpha 2
@anchor{whatsnew/changelog id444}@anchor{1415}
@subsubsection Security



@itemize -

@item 
bpo-26556@footnote{https://bugs.python.org/issue26556}: Update expat to 2.1.1, fixes CVE-2015-1283.

@item 
Fix TLS stripping vulnerability in smtplib, CVE-2016-0772. Reported by
Team Oststrom.
@end itemize

@node Library<56>,Security<29>,Security<28>,Python 3 6 0 alpha 2
@anchor{whatsnew/changelog id445}@anchor{1416}
@subsubsection Library



@itemize -

@item 
bpo-21386@footnote{https://bugs.python.org/issue21386}: Implement missing IPv4Address.is_global property.  It was
documented since 07a5610bae9d.  Initial patch by Roger Luethi.

@item 
bpo-27029@footnote{https://bugs.python.org/issue27029}: Removed deprecated support of universal newlines mode from
ZipFile.open().

@item 
bpo-27030@footnote{https://bugs.python.org/issue27030}: Unknown escapes consisting of @code{'\'} and an ASCII letter in
regular expressions now are errors.  The re.LOCALE flag now can be used
only with bytes patterns.

@item 
bpo-27186@footnote{https://bugs.python.org/issue27186}: Add os.PathLike support to DirEntry (part of 
@geindex Python Enhancement Proposals; PEP 519
PEP 519@footnote{https://www.python.org/dev/peps/pep-0519}).
Initial patch by Jelle Zijlstra.

@item 
bpo-20900@footnote{https://bugs.python.org/issue20900}: distutils register command now decodes HTTP responses
correctly.  Initial patch by ingrid.

@item 
bpo-27186@footnote{https://bugs.python.org/issue27186}: Add os.PathLike support to pathlib, removing its provisional
status (part of PEP 519). Initial patch by Dusty Phillips.

@item 
bpo-27186@footnote{https://bugs.python.org/issue27186}: Add support for os.PathLike objects to os.fsencode() and
os.fsdecode() (part of 
@geindex Python Enhancement Proposals; PEP 519
PEP 519@footnote{https://www.python.org/dev/peps/pep-0519}).

@item 
bpo-27186@footnote{https://bugs.python.org/issue27186}: Introduce os.PathLike and os.fspath() (part of 
@geindex Python Enhancement Proposals; PEP 519
PEP 519@footnote{https://www.python.org/dev/peps/pep-0519}).

@item 
A new version of typing.py provides several new classes and features:
@@overload outside stubs, Reversible, DefaultDict, Text, ContextManager,
Type[], NewType(), TYPE_CHECKING, and numerous bug fixes (note that some
of the new features are not yet implemented in mypy or other static
analyzers). Also classes for 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} (Awaitable, AsyncIterable,
AsyncIterator) have been added (in fact they made it into 3.5.1 but were
never mentioned).

@item 
bpo-25738@footnote{https://bugs.python.org/issue25738}: Stop http.server.BaseHTTPRequestHandler.send_error() from
sending a message body for 205 Reset Content.  Also, don’t send Content
header fields in responses that don’t have a body.  Patch by Susumu
Koshiba.

@item 
bpo-21313@footnote{https://bugs.python.org/issue21313}: Fix the “platform” module to tolerate when sys.version contains
truncated build information.
@end itemize

@node Security<29>,Library<57>,Library<56>,Python 3 6 0 alpha 2
@anchor{whatsnew/changelog id446}@anchor{1417}
@subsubsection Security



@itemize -

@item 
bpo-26839@footnote{https://bugs.python.org/issue26839}: On Linux, @ref{17f,,os.urandom()} now calls @code{getrandom()} with
@code{GRND_NONBLOCK} to fall back on reading @code{/dev/urandom} if the urandom
entropy pool is not initialized yet. Patch written by Colm Buckley.
@end itemize

@node Library<57>,IDLE<42>,Security<29>,Python 3 6 0 alpha 2
@anchor{whatsnew/changelog id447}@anchor{1418}
@subsubsection Library



@itemize -

@item 
bpo-23883@footnote{https://bugs.python.org/issue23883}: Added missing APIs to __all__ to match the documented APIs for
the following modules: cgi, mailbox, mimetypes, plistlib and smtpd.
Patches by Jacek Kołodziej.

@item 
bpo-27164@footnote{https://bugs.python.org/issue27164}: In the zlib module, allow decompressing raw Deflate streams
with a predefined zdict.  Based on patch by Xiang Zhang.

@item 
bpo-24291@footnote{https://bugs.python.org/issue24291}: Fix wsgiref.simple_server.WSGIRequestHandler to completely
write data to the client.  Previously it could do partial writes and
truncate data.  Also, wsgiref.handler.ServerHandler can now handle stdout
doing partial writes, but this is deprecated.

@item 
bpo-21272@footnote{https://bugs.python.org/issue21272}: Use _sysconfigdata.py to initialize distutils.sysconfig.

@item 
bpo-19611@footnote{https://bugs.python.org/issue19611}: @ref{9c,,inspect} now reports the implicit @code{.0} parameters
generated by the compiler for comprehension and generator expression
scopes as if they were positional-only parameters called @code{implicit0}.
Patch by Jelle Zijlstra.

@item 
bpo-26809@footnote{https://bugs.python.org/issue26809}: Add @code{__all__} to @ref{f1,,string}.  Patch by Emanuel Barry.

@item 
bpo-26373@footnote{https://bugs.python.org/issue26373}: subprocess.Popen.communicate now correctly ignores
BrokenPipeError when the child process dies before .communicate() is
called in more/all circumstances.

@item 
signal, socket, and ssl module IntEnum constant name lookups now return a
consistent name for values having multiple names.  Ex: signal.Signals(6)
now refers to itself as signal.SIGALRM rather than flipping between that
and signal.SIGIOT based on the interpreter’s hash randomization seed.

@item 
bpo-27167@footnote{https://bugs.python.org/issue27167}: Clarify the subprocess.CalledProcessError error message text
when the child process died due to a signal.

@item 
bpo-25931@footnote{https://bugs.python.org/issue25931}: Don’t define socketserver.Forking* names on platforms such as
Windows that do not support os.fork().

@item 
bpo-21776@footnote{https://bugs.python.org/issue21776}: distutils.upload now correctly handles HTTPError. Initial patch
by Claudiu Popa.

@item 
bpo-26526@footnote{https://bugs.python.org/issue26526}: Replace custom parse tree validation in the parser module with
a simple DFA validator.

@item 
bpo-27114@footnote{https://bugs.python.org/issue27114}: Fix SSLContext._load_windows_store_certs fails with
PermissionError

@item 
bpo-18383@footnote{https://bugs.python.org/issue18383}: Avoid creating duplicate filters when using filterwarnings and
simplefilter. Based on patch by Alex Shkop.

@item 
bpo-23026@footnote{https://bugs.python.org/issue23026}: winreg.QueryValueEx() now return an integer for REG_QWORD type.

@item 
bpo-26741@footnote{https://bugs.python.org/issue26741}: subprocess.Popen destructor now emits a ResourceWarning warning
if the child process is still running.

@item 
bpo-27056@footnote{https://bugs.python.org/issue27056}: Optimize pickle.load() and pickle.loads(), up to 10% faster to
deserialize a lot of small objects.

@item 
bpo-21271@footnote{https://bugs.python.org/issue21271}: New keyword only parameters in reset_mock call.
@end itemize

@node IDLE<42>,Documentation<47>,Library<57>,Python 3 6 0 alpha 2
@anchor{whatsnew/changelog id448}@anchor{1419}
@subsubsection IDLE



@itemize -

@item 
bpo-5124@footnote{https://bugs.python.org/issue5124}: Paste with text selected now replaces the selection on X11. This
matches how paste works on Windows, Mac, most modern Linux apps, and ttk
widgets. Original patch by Serhiy Storchaka.

@item 
bpo-24750@footnote{https://bugs.python.org/issue24750}: Switch all scrollbars in IDLE to ttk versions. Where needed,
minimal tests are added to cover changes.

@item 
bpo-24759@footnote{https://bugs.python.org/issue24759}: IDLE requires tk 8.5 and availability ttk widgets. Delete now
unneeded tk version tests and code for older versions. Add test for IDLE
syntax colorizer.

@item 
bpo-27239@footnote{https://bugs.python.org/issue27239}: idlelib.macosx.isXyzTk functions initialize as needed.

@item 
bpo-27262@footnote{https://bugs.python.org/issue27262}: move Aqua unbinding code, which enable context menus, to
macosx.

@item 
bpo-24759@footnote{https://bugs.python.org/issue24759}: Make clear in idlelib.idle_test.__init__ that the directory is
a private implementation of test.test_idle and tool for maintainers.

@item 
bpo-27196@footnote{https://bugs.python.org/issue27196}: Stop ‘ThemeChanged’ warnings when running IDLE tests. These
persisted after other warnings were suppressed in #20567. Apply Serhiy
Storchaka’s update_idletasks solution to four test files. Record this
additional advice in idle_test/README.txt

@item 
bpo-20567@footnote{https://bugs.python.org/issue20567}: Revise idle_test/README.txt with advice about avoiding tk
warning messages from tests.  Apply advice to several IDLE tests.

@item 
bpo-24225@footnote{https://bugs.python.org/issue24225}: Update idlelib/README.txt with new file names and event
handlers.

@item 
bpo-27156@footnote{https://bugs.python.org/issue27156}: Remove obsolete code not used by IDLE.

@item 
bpo-27117@footnote{https://bugs.python.org/issue27117}: Make colorizer htest and turtledemo work with dark themes. Move
code for configuring text widget colors to a new function.

@item 
bpo-24225@footnote{https://bugs.python.org/issue24225}: Rename many @code{idlelib/*.py} and @code{idle_test/test_*.py} files.
Edit files to replace old names with new names when the old name referred
to the module rather than the class it contained. See the issue and IDLE
section in What’s New in 3.6 for more.

@item 
bpo-26673@footnote{https://bugs.python.org/issue26673}: When tk reports font size as 0, change to size 10. Such fonts
on Linux prevented the configuration dialog from opening.

@item 
bpo-21939@footnote{https://bugs.python.org/issue21939}: Add test for IDLE’s percolator. Original patch by Saimadhav
Heblikar.

@item 
bpo-21676@footnote{https://bugs.python.org/issue21676}: Add test for IDLE’s replace dialog. Original patch by Saimadhav
Heblikar.

@item 
bpo-18410@footnote{https://bugs.python.org/issue18410}: Add test for IDLE’s search dialog. Original patch by Westley
Martínez.

@item 
bpo-21703@footnote{https://bugs.python.org/issue21703}: Add test for undo delegator.  Patch mostly by Saimadhav
Heblikar .

@item 
bpo-27044@footnote{https://bugs.python.org/issue27044}: Add ConfigDialog.remove_var_callbacks to stop memory leaks.

@item 
bpo-23977@footnote{https://bugs.python.org/issue23977}: Add more asserts to test_delegator.
@end itemize

@node Documentation<47>,Tests<47>,IDLE<42>,Python 3 6 0 alpha 2
@anchor{whatsnew/changelog id449}@anchor{141a}
@subsubsection Documentation



@itemize -

@item 
bpo-16484@footnote{https://bugs.python.org/issue16484}: Change the default PYTHONDOCS URL to “https:”, and fix the
resulting links to use lowercase.  Patch by Sean Rodman, test by Kaushik
Nadikuditi.

@item 
bpo-24136@footnote{https://bugs.python.org/issue24136}: Document the new 
@geindex Python Enhancement Proposals; PEP 448
PEP 448@footnote{https://www.python.org/dev/peps/pep-0448} unpacking syntax of 3.5.

@item 
bpo-22558@footnote{https://bugs.python.org/issue22558}: Add remaining doc links to source code for Python-coded
modules. Patch by Yoni Lavi.
@end itemize

@node Tests<47>,Windows<44>,Documentation<47>,Python 3 6 0 alpha 2
@anchor{whatsnew/changelog id450}@anchor{141b}
@subsubsection Tests



@itemize -

@item 
bpo-25285@footnote{https://bugs.python.org/issue25285}: regrtest now uses subprocesses when the -j1 command line option
is used: each test file runs in a fresh child process. Before, the -j1
option was ignored.

@item 
bpo-25285@footnote{https://bugs.python.org/issue25285}: Tools/buildbot/test.bat script now uses -j1 by default to run
each test file in fresh child process.
@end itemize

@node Windows<44>,Build<51>,Tests<47>,Python 3 6 0 alpha 2
@anchor{whatsnew/changelog id451}@anchor{141c}
@subsubsection Windows



@itemize -

@item 
bpo-27064@footnote{https://bugs.python.org/issue27064}: The py.exe launcher now defaults to Python 3. The Windows
launcher @code{py.exe} no longer prefers an installed Python 2 version over
Python 3 by default when used interactively.
@end itemize

@node Build<51>,Windows<45>,Windows<44>,Python 3 6 0 alpha 2
@anchor{whatsnew/changelog id452}@anchor{141d}
@subsubsection Build



@itemize -

@item 
bpo-27229@footnote{https://bugs.python.org/issue27229}: Fix the cross-compiling pgen rule for in-tree builds.  Patch by
Xavier de Gaye.

@item 
bpo-26930@footnote{https://bugs.python.org/issue26930}: Update OS X 10.5+ 32-bit-only installer to build and link with
OpenSSL 1.0.2h.
@end itemize

@node Windows<45>,C API<42>,Build<51>,Python 3 6 0 alpha 2
@anchor{whatsnew/changelog id453}@anchor{141e}
@subsubsection Windows



@itemize -

@item 
bpo-17500@footnote{https://bugs.python.org/issue17500}: Remove unused and outdated icons. (See also:
@indicateurl{https://github.com/python/pythondotorg/issues/945})
@end itemize

@node C API<42>,Tools/Demos<27>,Windows<45>,Python 3 6 0 alpha 2
@anchor{whatsnew/changelog id454}@anchor{141f}
@subsubsection C API



@itemize -

@item 
bpo-27186@footnote{https://bugs.python.org/issue27186}: Add the PyOS_FSPath() function (part of 
@geindex Python Enhancement Proposals; PEP 519
PEP 519@footnote{https://www.python.org/dev/peps/pep-0519}).

@item 
bpo-26282@footnote{https://bugs.python.org/issue26282}: PyArg_ParseTupleAndKeywords() now supports positional-only
parameters.
@end itemize

@node Tools/Demos<27>,,C API<42>,Python 3 6 0 alpha 2
@anchor{whatsnew/changelog id455}@anchor{1420}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-26282@footnote{https://bugs.python.org/issue26282}: Argument Clinic now supports positional-only and keyword
parameters in the same function.
@end itemize

@node Python 3 6 0 alpha 1,Python 3 5 5 final,Python 3 6 0 alpha 2,Changelog
@anchor{whatsnew/changelog python-3-6-0-alpha-1}@anchor{1421}
@subsection Python 3.6.0 alpha 1


@emph{Release date: 2016-05-16}

@menu
* Core and Builtins: Core and Builtins<53>. 
* Library: Library<58>. 
* Security: Security<30>. 
* Library: Library<59>. 
* Security: Security<31>. 
* Library: Library<60>. 
* Security: Security<32>. 
* Library: Library<61>. 
* IDLE: IDLE<43>. 
* Documentation: Documentation<48>. 
* Tests: Tests<48>. 
* Build: Build<52>. 
* Windows: Windows<46>. 
* Tools/Demos: Tools/Demos<28>. 
* C API: C API<43>. 

@end menu

@node Core and Builtins<53>,Library<58>,,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id456}@anchor{1422}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-20041@footnote{https://bugs.python.org/issue20041}: Fixed TypeError when frame.f_trace is set to None. Patch by
Xavier de Gaye.

@item 
bpo-26168@footnote{https://bugs.python.org/issue26168}: Fixed possible refleaks in failing Py_BuildValue() with the “N”
format unit.

@item 
bpo-26991@footnote{https://bugs.python.org/issue26991}: Fix possible refleak when creating a function with annotations.

@item 
bpo-27039@footnote{https://bugs.python.org/issue27039}: Fixed bytearray.remove() for values greater than 127.  Based on
patch by Joe Jevnik.

@item 
bpo-23640@footnote{https://bugs.python.org/issue23640}: int.from_bytes() no longer bypasses constructors for
subclasses.

@item 
bpo-27005@footnote{https://bugs.python.org/issue27005}: Optimized the float.fromhex() class method for exact float. It
is now 2 times faster.

@item 
bpo-18531@footnote{https://bugs.python.org/issue18531}: Single var-keyword argument of dict subtype was passed
unscathed to the C-defined function.  Now it is converted to exact dict.

@item 
bpo-26811@footnote{https://bugs.python.org/issue26811}: gc.get_objects() no longer contains a broken tuple with NULL
pointer.

@item 
bpo-20120@footnote{https://bugs.python.org/issue20120}: Use RawConfigParser for .pypirc parsing, removing support for
interpolation unintentionally added with move to Python 3. Behavior no
longer does any interpolation in .pypirc files, matching behavior in
Python 2.7 and Setuptools 19.0.

@item 
bpo-26249@footnote{https://bugs.python.org/issue26249}: Memory functions of the @ref{7ad,,PyMem_Malloc()} domain
(@code{PYMEM_DOMAIN_MEM}) now use the @ref{85c,,pymalloc allocator} rather than system @code{malloc()}. Applications calling
@ref{7ad,,PyMem_Malloc()} without holding the GIL can now crash: use
@code{PYTHONMALLOC=debug} environment variable to validate the usage of
memory allocators in your application.

@item 
bpo-26802@footnote{https://bugs.python.org/issue26802}: Optimize function calls only using unpacking like
@code{func(*tuple)} (no other positional argument, no keyword): avoid copying
the tuple. Patch written by Joe Jevnik.

@item 
bpo-26659@footnote{https://bugs.python.org/issue26659}: Make the builtin slice type support cycle collection.

@item 
bpo-26718@footnote{https://bugs.python.org/issue26718}: super.__init__ no longer leaks memory if called multiple times.
NOTE: A direct call of super.__init__ is not endorsed!

@item 
bpo-27138@footnote{https://bugs.python.org/issue27138}: Fix the doc comment for FileFinder.find_spec().

@item 
bpo-27147@footnote{https://bugs.python.org/issue27147}: Mention 
@geindex Python Enhancement Proposals; PEP 420
PEP 420@footnote{https://www.python.org/dev/peps/pep-0420} in the importlib docs.

@item 
bpo-25339@footnote{https://bugs.python.org/issue25339}: PYTHONIOENCODING now has priority over locale in setting the
error handler for stdin and stdout.

@item 
bpo-26494@footnote{https://bugs.python.org/issue26494}: Fixed crash on iterating exhausting iterators. Affected classes
are generic sequence iterators, iterators of str, bytes, bytearray, list,
tuple, set, frozenset, dict, OrderedDict, corresponding views and
os.scandir() iterator.

@item 
bpo-26574@footnote{https://bugs.python.org/issue26574}: Optimize @code{bytes.replace(b'@w{'}, b'.')} and
@code{bytearray.replace(b'@w{'}, b'.')}. Patch written by Josh Snider.

@item 
bpo-26581@footnote{https://bugs.python.org/issue26581}: If coding cookie is specified multiple times on a line in
Python source code file, only the first one is taken to account.

@item 
bpo-19711@footnote{https://bugs.python.org/issue19711}: Add tests for reloading namespace packages.

@item 
bpo-21099@footnote{https://bugs.python.org/issue21099}: Switch applicable importlib tests to use 
@geindex Python Enhancement Proposals; PEP 451
PEP 451@footnote{https://www.python.org/dev/peps/pep-0451} API.

@item 
bpo-26563@footnote{https://bugs.python.org/issue26563}: Debug hooks on Python memory allocators now raise a fatal error
if functions of the @ref{7ad,,PyMem_Malloc()} family are called without
holding the GIL.

@item 
bpo-26564@footnote{https://bugs.python.org/issue26564}: On error, the debug hooks on Python memory allocators now use
the @ref{11b,,tracemalloc} module to get the traceback where a memory block
was allocated.

@item 
bpo-26558@footnote{https://bugs.python.org/issue26558}: The debug hooks on Python memory allocator
@ref{7af,,PyObject_Malloc()} now detect when functions are called without
holding the GIL.

@item 
bpo-26516@footnote{https://bugs.python.org/issue26516}: Add 
@geindex PYTHONMALLOC
@geindex environment variable; PYTHONMALLOC
@ref{7ab,,PYTHONMALLOC} environment variable to set the
Python memory allocators and/or install debug hooks.

@item 
bpo-26516@footnote{https://bugs.python.org/issue26516}: The @ref{7b0,,PyMem_SetupDebugHooks()} function can now also be
used on Python compiled in release mode.

@item 
bpo-26516@footnote{https://bugs.python.org/issue26516}: The 
@geindex PYTHONMALLOCSTATS
@geindex environment variable; PYTHONMALLOCSTATS
@ref{1423,,PYTHONMALLOCSTATS} environment variable can now
also be used on Python compiled in release mode. It now has no effect if
set to an empty string.

@item 
bpo-26516@footnote{https://bugs.python.org/issue26516}: In debug mode, debug hooks are now also installed on Python
memory allocators when Python is configured without pymalloc.

@item 
bpo-26464@footnote{https://bugs.python.org/issue26464}: Fix str.translate() when string is ASCII and first replacements
removes character, but next replacement uses a non-ASCII character or a
string longer than 1 character. Regression introduced in Python 3.5.0.

@item 
bpo-22836@footnote{https://bugs.python.org/issue22836}: Ensure exception reports from PyErr_Display() and
PyErr_WriteUnraisable() are sensible even when formatting them produces
secondary errors.  This affects the reports produced by
sys.__excepthook__() and when __del__() raises an exception.

@item 
bpo-26302@footnote{https://bugs.python.org/issue26302}: Correct behavior to reject comma as a legal character for
cookie names.

@item 
bpo-26136@footnote{https://bugs.python.org/issue26136}: Upgrade the warning when a generator raises StopIteration from
PendingDeprecationWarning to DeprecationWarning.  Patch by Anish Shah.

@item 
bpo-26204@footnote{https://bugs.python.org/issue26204}: The compiler now ignores all constant statements: bytes, str,
int, float, complex, name constants (None, False, True), Ellipsis and
ast.Constant; not only str and int. For example, @code{1.0} is now ignored in
@code{def f(): 1.0}.

@item 
bpo-4806@footnote{https://bugs.python.org/issue4806}: Avoid masking the original TypeError exception when using star
(@code{*}) unpacking in function calls.  Based on patch by Hagen Fürstenau
and Daniel Urban.

@item 
bpo-26146@footnote{https://bugs.python.org/issue26146}: Add a new kind of AST node: @code{ast.Constant}. It can be used by
external AST optimizers, but the compiler does not emit directly such
node.

@item 
bpo-23601@footnote{https://bugs.python.org/issue23601}: Sped-up allocation of dict key objects by using Python’s small
object allocator.  (Contributed by Julian Taylor.)

@item 
bpo-18018@footnote{https://bugs.python.org/issue18018}: Import raises ImportError instead of SystemError if a relative
import is attempted without a known parent package.

@item 
bpo-25843@footnote{https://bugs.python.org/issue25843}: When compiling code, don’t merge constants if they are equal
but have a different types. For example, @code{f1, f2 = lambda: 1, lambda:
1.0} is now correctly compiled to two different functions: @code{f1()}
returns @code{1} (@code{int}) and @code{f2()} returns @code{1.0} (@code{float}), even if
@code{1} and @code{1.0} are equal.

@item 
bpo-26107@footnote{https://bugs.python.org/issue26107}: The format of the @code{co_lnotab} attribute of code objects
changes to support negative line number delta.

@item 
bpo-26154@footnote{https://bugs.python.org/issue26154}: Add a new private _PyThreadState_UncheckedGet() function to get
the current Python thread state, but don’t issue a fatal error if it is
NULL. This new function must be used instead of accessing directly the
_PyThreadState_Current variable.  The variable is no more exposed since
Python 3.5.1 to hide the exact implementation of atomic C types, to avoid
compiler issues.

@item 
bpo-25791@footnote{https://bugs.python.org/issue25791}: If __package__ != __spec__.parent or if neither __package__ or
__spec__ are defined then ImportWarning is raised.

@item 
bpo-22995@footnote{https://bugs.python.org/issue22995}: [UPDATE] Comment out the one of the pickleability tests in
_PyObject_GetState() due to regressions observed in Cython-based projects.

@item 
bpo-25961@footnote{https://bugs.python.org/issue25961}: Disallowed null characters in the type name.

@item 
bpo-25973@footnote{https://bugs.python.org/issue25973}: Fix segfault when an invalid nonlocal statement binds a name
starting with two underscores.

@item 
bpo-22995@footnote{https://bugs.python.org/issue22995}: Instances of extension types with a state that aren’t
subclasses of list or dict and haven’t implemented any pickle-related
methods (__reduce__, __reduce_ex__, __getnewargs__, __getnewargs_ex__, or
__getstate__), can no longer be pickled.  Including memoryview.

@item 
bpo-20440@footnote{https://bugs.python.org/issue20440}: Massive replacing unsafe attribute setting code with special
macro Py_SETREF.

@item 
bpo-25766@footnote{https://bugs.python.org/issue25766}: Special method __bytes__() now works in str subclasses.

@item 
bpo-25421@footnote{https://bugs.python.org/issue25421}: __sizeof__ methods of builtin types now use dynamic basic size.
This allows sys.getsize() to work correctly with their subclasses with
__slots__ defined.

@item 
bpo-25709@footnote{https://bugs.python.org/issue25709}: Fixed problem with in-place string concatenation and utf-8
cache.

@item 
bpo-5319@footnote{https://bugs.python.org/issue5319}: New Py_FinalizeEx() API allowing Python to set an exit status of
120 on failure to flush buffered streams.

@item 
bpo-25485@footnote{https://bugs.python.org/issue25485}: telnetlib.Telnet is now a context manager.

@item 
bpo-24097@footnote{https://bugs.python.org/issue24097}: Fixed crash in object.__reduce__() if slot name is freed inside
__getattr__.

@item 
bpo-24731@footnote{https://bugs.python.org/issue24731}: Fixed crash on converting objects with special methods
__bytes__, __trunc__, and __float__ returning instances of subclasses of
bytes, int, and float to subclasses of bytes, int, and float
correspondingly.

@item 
bpo-25630@footnote{https://bugs.python.org/issue25630}: Fix a possible segfault during argument parsing in functions
that accept filesystem paths.

@item 
bpo-23564@footnote{https://bugs.python.org/issue23564}: Fixed a partially broken sanity check in the _posixsubprocess
internals regarding how fds_to_pass were passed to the child.  The bug had
no actual impact as subprocess.py already avoided it.

@item 
bpo-25388@footnote{https://bugs.python.org/issue25388}: Fixed tokenizer crash when processing undecodable source code
with a null byte.

@item 
bpo-25462@footnote{https://bugs.python.org/issue25462}: The hash of the key now is calculated only once in most
operations in C implementation of OrderedDict.

@item 
bpo-22995@footnote{https://bugs.python.org/issue22995}: Default implementation of __reduce__ and __reduce_ex__ now
rejects builtin types with not defined __new__.

@item 
bpo-24802@footnote{https://bugs.python.org/issue24802}: Avoid buffer overreads when int(), float(), compile(), exec()
and eval() are passed bytes-like objects.  These objects are not
necessarily terminated by a null byte, but the functions assumed they
were.

@item 
bpo-25555@footnote{https://bugs.python.org/issue25555}: Fix parser and AST: fill lineno and col_offset of “arg” node
when compiling AST from Python objects.

@item 
bpo-24726@footnote{https://bugs.python.org/issue24726}: Fixed a crash and leaking NULL in repr() of OrderedDict that
was mutated by direct calls of dict methods.

@item 
bpo-25449@footnote{https://bugs.python.org/issue25449}: Iterating OrderedDict with keys with unstable hash now raises
KeyError in C implementations as well as in Python implementation.

@item 
bpo-25395@footnote{https://bugs.python.org/issue25395}: Fixed crash when highly nested OrderedDict structures were
garbage collected.

@item 
bpo-25401@footnote{https://bugs.python.org/issue25401}: Optimize bytes.fromhex() and bytearray.fromhex(): they are now
between 2x and 3.5x faster.

@item 
bpo-25399@footnote{https://bugs.python.org/issue25399}: Optimize bytearray % args using the new private _PyBytesWriter
API. Formatting is now between 2.5 and 5 times faster.

@item 
bpo-25274@footnote{https://bugs.python.org/issue25274}: sys.setrecursionlimit() now raises a RecursionError if the new
recursion limit is too low depending at the current recursion depth.
Modify also the “lower-water mark” formula to make it monotonic. This mark
is used to decide when the overflowed flag of the thread state is reset.

@item 
bpo-24402@footnote{https://bugs.python.org/issue24402}: Fix input() to prompt to the redirected stdout when
sys.stdout.fileno() fails.

@item 
bpo-25349@footnote{https://bugs.python.org/issue25349}: Optimize bytes % args using the new private _PyBytesWriter API.
Formatting is now up to 2 times faster.

@item 
bpo-24806@footnote{https://bugs.python.org/issue24806}: Prevent builtin types that are not allowed to be subclassed
from being subclassed through multiple inheritance.

@item 
bpo-25301@footnote{https://bugs.python.org/issue25301}: The UTF-8 decoder is now up to 15 times as fast for error
handlers: @code{ignore}, @code{replace} and @code{surrogateescape}.

@item 
bpo-24848@footnote{https://bugs.python.org/issue24848}: Fixed a number of bugs in UTF-7 decoding of misformed data.

@item 
bpo-25267@footnote{https://bugs.python.org/issue25267}: The UTF-8 encoder is now up to 75 times as fast for error
handlers: @code{ignore}, @code{replace}, @code{surrogateescape}, @code{surrogatepass}.
Patch co-written with Serhiy Storchaka.

@item 
bpo-25280@footnote{https://bugs.python.org/issue25280}: Import trace messages emitted in verbose (-v) mode are no
longer formatted twice.

@item 
bpo-25227@footnote{https://bugs.python.org/issue25227}: Optimize ASCII and latin1 encoders with the @code{surrogateescape}
error handler: the encoders are now up to 3 times as fast. Initial patch
written by Serhiy Storchaka.

@item 
bpo-25003@footnote{https://bugs.python.org/issue25003}: On Solaris 11.3 or newer, os.urandom() now uses the getrandom()
function instead of the getentropy() function. The getentropy() function
is blocking to generate very good quality entropy, os.urandom() doesn’t
need such high-quality entropy.

@item 
bpo-9232@footnote{https://bugs.python.org/issue9232}: Modify Python’s grammar to allow trailing commas in the argument
list of a function declaration.  For example, “def f(*, a = 3,): pass” is
now legal. Patch from Mark Dickinson.

@item 
bpo-24965@footnote{https://bugs.python.org/issue24965}: Implement 
@geindex Python Enhancement Proposals; PEP 498
PEP 498@footnote{https://www.python.org/dev/peps/pep-0498} “Literal String Interpolation”. This
allows you to embed expressions inside f-strings, which are converted to
normal strings at run time. Given x=3, then f’value=@{x@}’ == ‘value=3’.
Patch by Eric V. Smith.

@item 
bpo-26478@footnote{https://bugs.python.org/issue26478}: Fix semantic bugs when using binary operators with dictionary
views and tuples.

@item 
bpo-26171@footnote{https://bugs.python.org/issue26171}: Fix possible integer overflow and heap corruption in
zipimporter.get_data().

@item 
bpo-25660@footnote{https://bugs.python.org/issue25660}: Fix TAB key behaviour in REPL with readline.

@item 
bpo-26288@footnote{https://bugs.python.org/issue26288}: Optimize PyLong_AsDouble.

@item 
bpo-26289@footnote{https://bugs.python.org/issue26289}: Optimize floor and modulo division for single-digit longs.
Microbenchmarks show 2-2.5x improvement.  Built-in ‘divmod’ function is
now also ~10% faster. (See also: bpo-26315@footnote{https://bugs.python.org/issue26315})

@item 
bpo-25887@footnote{https://bugs.python.org/issue25887}: Raise a RuntimeError when a coroutine object is awaited more
than once.
@end itemize

@node Library<58>,Security<30>,Core and Builtins<53>,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id457}@anchor{1424}
@subsubsection Library



@itemize -

@item 
bpo-27057@footnote{https://bugs.python.org/issue27057}: Fix os.set_inheritable() on Android, ioctl() is blocked by
SELinux and fails with EACCESS. The function now falls back to fcntl().
Patch written by Michał Bednarski.

@item 
bpo-27014@footnote{https://bugs.python.org/issue27014}: Fix infinite recursion using typing.py.  Thanks to Kalle Tuure!

@item 
bpo-27031@footnote{https://bugs.python.org/issue27031}: Removed dummy methods in Tkinter widget classes: tk_menuBar()
and tk_bindForTraversal().

@item 
bpo-14132@footnote{https://bugs.python.org/issue14132}: Fix urllib.request redirect handling when the target only has a
query string.  Original fix by Ján Janech.

@item 
bpo-17214@footnote{https://bugs.python.org/issue17214}: The “urllib.request” module now percent-encodes non-ASCII bytes
found in redirect target URLs.  Some servers send Location header fields
with non-ASCII bytes, but “http.client” requires the request target to be
ASCII-encodable, otherwise a UnicodeEncodeError is raised.  Based on patch
by Christian Heimes.

@item 
bpo-27033@footnote{https://bugs.python.org/issue27033}: The default value of the decode_data parameter for
smtpd.SMTPChannel and smtpd.SMTPServer constructors is changed to False.

@item 
bpo-27034@footnote{https://bugs.python.org/issue27034}: Removed deprecated class asynchat.fifo.

@item 
bpo-26870@footnote{https://bugs.python.org/issue26870}: Added readline.set_auto_history(), which can stop entries being
automatically added to the history list.  Based on patch by Tyler
Crompton.

@item 
bpo-26039@footnote{https://bugs.python.org/issue26039}: zipfile.ZipFile.open() can now be used to write data into a ZIP
file, as well as for extracting data.  Patch by Thomas Kluyver.

@item 
bpo-26892@footnote{https://bugs.python.org/issue26892}: Honor debuglevel flag in urllib.request.HTTPHandler. Patch
contributed by Chi Hsuan Yen.

@item 
bpo-22274@footnote{https://bugs.python.org/issue22274}: In the subprocess module, allow stderr to be redirected to
stdout even when stdout is not redirected.  Patch by Akira Li.

@item 
bpo-26807@footnote{https://bugs.python.org/issue26807}: mock_open ‘files’ no longer error on readline at end of file.
Patch from Yolanda Robla.

@item 
bpo-25745@footnote{https://bugs.python.org/issue25745}: Fixed leaking a userptr in curses panel destructor.

@item 
bpo-26977@footnote{https://bugs.python.org/issue26977}: Removed unnecessary, and ignored, call to sum of squares helper
in statistics.pvariance.

@item 
bpo-26002@footnote{https://bugs.python.org/issue26002}: Use bisect in statistics.median instead of a linear search.
Patch by Upendra Kuma.

@item 
bpo-25974@footnote{https://bugs.python.org/issue25974}: Make use of new Decimal.as_integer_ratio() method in statistics
module. Patch by Stefan Krah.

@item 
bpo-26996@footnote{https://bugs.python.org/issue26996}: Add secrets module as described in 
@geindex Python Enhancement Proposals; PEP 506
PEP 506@footnote{https://www.python.org/dev/peps/pep-0506}.

@item 
bpo-26881@footnote{https://bugs.python.org/issue26881}: The modulefinder module now supports extended opcode arguments.

@item 
bpo-23815@footnote{https://bugs.python.org/issue23815}: Fixed crashes related to directly created instances of types in
_tkinter and curses.panel modules.

@item 
bpo-17765@footnote{https://bugs.python.org/issue17765}: weakref.ref() no longer silently ignores keyword arguments.
Patch by Georg Brandl.

@item 
bpo-26873@footnote{https://bugs.python.org/issue26873}: xmlrpc now raises ResponseError on unsupported type tags
instead of silently return incorrect result.

@item 
bpo-26915@footnote{https://bugs.python.org/issue26915}: The __contains__ methods in the collections ABCs now check for
identity before checking equality.  This better matches the behavior of
the concrete classes, allows sensible handling of NaNs, and makes it
easier to reason about container invariants.

@item 
bpo-26711@footnote{https://bugs.python.org/issue26711}: Fixed the comparison of plistlib.Data with other types.

@item 
bpo-24114@footnote{https://bugs.python.org/issue24114}: Fix an uninitialized variable in @code{ctypes.util}. The bug only
occurs on SunOS when the ctypes implementation searches for the @code{crle}
program.  Patch by Xiang Zhang.  Tested on SunOS by Kees Bos.

@item 
bpo-26864@footnote{https://bugs.python.org/issue26864}: In urllib.request, change the proxy bypass host checking
against no_proxy to be case-insensitive, and to not match unrelated host
names that happen to have a bypassed hostname as a suffix.  Patch by Xiang
Zhang.

@item 
bpo-24902@footnote{https://bugs.python.org/issue24902}: Print server URL on http.server startup.  Initial patch by
Felix Kaiser.

@item 
bpo-25788@footnote{https://bugs.python.org/issue25788}: fileinput.hook_encoded() now supports an “errors” argument for
passing to open.  Original patch by Joseph Hackman.

@item 
bpo-26634@footnote{https://bugs.python.org/issue26634}: recursive_repr() now sets __qualname__ of wrapper.  Patch by
Xiang Zhang.

@item 
bpo-26804@footnote{https://bugs.python.org/issue26804}: urllib.request will prefer lower_case proxy environment
variables over UPPER_CASE or Mixed_Case ones. Patch contributed by
Hans-Peter Jansen.

@item 
bpo-26837@footnote{https://bugs.python.org/issue26837}: assertSequenceEqual() now correctly outputs non-stringified
differing items (like bytes in the -b mode).  This affects
assertListEqual() and assertTupleEqual().

@item 
bpo-26041@footnote{https://bugs.python.org/issue26041}: Remove “will be removed in Python 3.7” from deprecation
messages of platform.dist() and platform.linux_distribution(). Patch by
Kumaripaba Miyurusara Athukorala.

@item 
bpo-26822@footnote{https://bugs.python.org/issue26822}: itemgetter, attrgetter and methodcaller objects no longer
silently ignore keyword arguments.

@item 
bpo-26733@footnote{https://bugs.python.org/issue26733}: Disassembling a class now disassembles class and static
methods. Patch by Xiang Zhang.

@item 
bpo-26801@footnote{https://bugs.python.org/issue26801}: Fix error handling in @ref{c96,,shutil.get_terminal_size()}, catch
@ref{163,,AttributeError} instead of @ref{21f,,NameError}. Patch written by
Emanuel Barry.

@item 
bpo-24838@footnote{https://bugs.python.org/issue24838}: tarfile’s ustar and gnu formats now correctly calculate name
and link field limits for multibyte character encodings like utf-8.
@end itemize

@node Security<30>,Library<59>,Library<58>,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id458}@anchor{1425}
@subsubsection Security



@itemize -

@item 
bpo-26657@footnote{https://bugs.python.org/issue26657}: Fix directory traversal vulnerability with http.server on
Windows.  This fixes a regression that was introduced in 3.3.4rc1 and
3.4.0rc1.  Based on patch by Philipp Hagemeister.
@end itemize

@node Library<59>,Security<31>,Security<30>,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id459}@anchor{1426}
@subsubsection Library



@itemize -

@item 
bpo-26717@footnote{https://bugs.python.org/issue26717}: Stop encoding Latin-1-ized WSGI paths with UTF-8.  Patch by
Anthony Sottile.

@item 
bpo-26782@footnote{https://bugs.python.org/issue26782}: Add STARTUPINFO to subprocess.__all__ on Windows.

@item 
bpo-26404@footnote{https://bugs.python.org/issue26404}: Add context manager to socketserver.  Patch by Aviv Palivoda.

@item 
bpo-26735@footnote{https://bugs.python.org/issue26735}: Fix @ref{17f,,os.urandom()} on Solaris 11.3 and newer when reading
more than 1,024 bytes: call @code{getrandom()} multiple times with a limit of
1024 bytes per call.

@item 
bpo-26585@footnote{https://bugs.python.org/issue26585}: Eliminate http.server._quote_html() and use
html.escape(quote=False).  Patch by Xiang Zhang.

@item 
bpo-26685@footnote{https://bugs.python.org/issue26685}: Raise OSError if closing a socket fails.

@item 
bpo-16329@footnote{https://bugs.python.org/issue16329}: Add .webm to mimetypes.types_map.  Patch by Giampaolo Rodola’.

@item 
bpo-13952@footnote{https://bugs.python.org/issue13952}: Add .csv to mimetypes.types_map.  Patch by Geoff Wilson.

@item 
bpo-26587@footnote{https://bugs.python.org/issue26587}: the site module now allows .pth files to specify files to be
added to sys.path (e.g. zip files).

@item 
bpo-25609@footnote{https://bugs.python.org/issue25609}: Introduce contextlib.AbstractContextManager and
typing.ContextManager.

@item 
bpo-26709@footnote{https://bugs.python.org/issue26709}: Fixed Y2038 problem in loading binary PLists.

@item 
bpo-23735@footnote{https://bugs.python.org/issue23735}: Handle terminal resizing with Readline 6.3+ by installing our
own SIGWINCH handler.  Patch by Eric Price.

@item 
bpo-25951@footnote{https://bugs.python.org/issue25951}: Change SSLSocket.sendall() to return None, as explicitly
documented for plain socket objects.  Patch by Aviv Palivoda.

@item 
bpo-26586@footnote{https://bugs.python.org/issue26586}: In http.server, respond with “413 Request header fields too
large” if there are too many header fields to parse, rather than killing
the connection and raising an unhandled exception.  Patch by Xiang Zhang.

@item 
bpo-26676@footnote{https://bugs.python.org/issue26676}: Added missing XMLPullParser to ElementTree.__all__.

@item 
bpo-22854@footnote{https://bugs.python.org/issue22854}: Change BufferedReader.writable() and BufferedWriter.readable()
to always return False.

@item 
bpo-26492@footnote{https://bugs.python.org/issue26492}: Exhausted iterator of array.array now conforms with the
behavior of iterators of other mutable sequences: it lefts exhausted even
if iterated array is extended.

@item 
bpo-26641@footnote{https://bugs.python.org/issue26641}: doctest.DocFileTest and doctest.testfile() now support packages
(module splitted into multiple directories) for the package parameter.

@item 
bpo-25195@footnote{https://bugs.python.org/issue25195}: Fix a regression in mock.MagicMock. _Call is a subclass of
tuple (changeset 3603bae63c13 only works for classes) so we need to
implement __ne__ ourselves.  Patch by Andrew Plummer.

@item 
bpo-26644@footnote{https://bugs.python.org/issue26644}: Raise ValueError rather than SystemError when a negative length
is passed to SSLSocket.recv() or read().

@item 
bpo-23804@footnote{https://bugs.python.org/issue23804}: Fix SSL recv(0) and read(0) methods to return zero bytes
instead of up to 1024.

@item 
bpo-26616@footnote{https://bugs.python.org/issue26616}: Fixed a bug in datetime.astimezone() method.

@item 
bpo-26637@footnote{https://bugs.python.org/issue26637}: The @ref{96,,importlib} module now emits an @ref{3b8,,ImportError}
rather than a @ref{162,,TypeError} if @ref{3b7,,__import__()} is tried during the
Python shutdown process but @ref{489,,sys.path} is already cleared (set to
@code{None}).

@item 
bpo-21925@footnote{https://bugs.python.org/issue21925}: @ref{12c0,,warnings.formatwarning()} now catches exceptions when
calling @ref{98a,,linecache.getline()} and
@ref{1427,,tracemalloc.get_object_traceback()} to be able to log
@ref{77b,,ResourceWarning} emitted late during the Python shutdown process.

@item 
bpo-23848@footnote{https://bugs.python.org/issue23848}: On Windows, faulthandler.enable() now also installs an
exception handler to dump the traceback of all Python threads on any
Windows exception, not only on UNIX signals (SIGSEGV, SIGFPE, SIGABRT).

@item 
bpo-26530@footnote{https://bugs.python.org/issue26530}: Add C functions @code{_PyTraceMalloc_Track()} and
@code{_PyTraceMalloc_Untrack()} to track memory blocks using the
@ref{11b,,tracemalloc} module. Add @code{_PyTraceMalloc_GetTraceback()} to
get the traceback of an object.

@item 
bpo-26588@footnote{https://bugs.python.org/issue26588}: The _tracemalloc now supports tracing memory allocations of
multiple address spaces (domains).

@item 
bpo-24266@footnote{https://bugs.python.org/issue24266}: Ctrl+C during Readline history search now cancels the search
mode when compiled with Readline 7.

@item 
bpo-26590@footnote{https://bugs.python.org/issue26590}: Implement a safe finalizer for the _socket.socket type. It now
releases the GIL to close the socket.

@item 
bpo-18787@footnote{https://bugs.python.org/issue18787}: spwd.getspnam() now raises a PermissionError if the user
doesn’t have privileges.

@item 
bpo-26560@footnote{https://bugs.python.org/issue26560}: Avoid potential ValueError in BaseHandler.start_response.
Initial patch by Peter Inglesby.

@item 
bpo-26567@footnote{https://bugs.python.org/issue26567}: Add a new function @ref{865,,PyErr_ResourceWarning()} function to
pass the destroyed object. Add a @emph{source} attribute to
@code{warnings.WarningMessage}. Add warnings._showwarnmsg() which uses
tracemalloc to get the traceback where source object was allocated.
@end itemize

@node Security<31>,Library<60>,Library<59>,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id460}@anchor{1428}
@subsubsection Security



@itemize -

@item 
bpo-26313@footnote{https://bugs.python.org/issue26313}: ssl.py _load_windows_store_certs fails if windows cert store is
empty. Patch by Baji.
@end itemize

@node Library<60>,Security<32>,Security<31>,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id461}@anchor{1429}
@subsubsection Library



@itemize -

@item 
bpo-26569@footnote{https://bugs.python.org/issue26569}: Fix @ref{142a,,pyclbr.readmodule()} and @ref{142b,,pyclbr.readmodule_ex()}
to support importing packages.

@item 
bpo-26499@footnote{https://bugs.python.org/issue26499}: Account for remaining Content-Length in HTTPResponse.readline()
and read1(). Based on patch by Silent Ghost. Also document that
HTTPResponse now supports these methods.

@item 
bpo-25320@footnote{https://bugs.python.org/issue25320}: Handle sockets in directories unittest discovery is scanning.
Patch from Victor van den Elzen.

@item 
bpo-16181@footnote{https://bugs.python.org/issue16181}: cookiejar.http2time() now returns None if year is higher than
datetime.MAXYEAR.

@item 
bpo-26513@footnote{https://bugs.python.org/issue26513}: Fixes platform module detection of Windows Server

@item 
bpo-23718@footnote{https://bugs.python.org/issue23718}: Fixed parsing time in week 0 before Jan 1.  Original patch by
Tamás Bence Gedai.

@item 
bpo-26323@footnote{https://bugs.python.org/issue26323}: Add Mock.assert_called() and Mock.assert_called_once() methods
to unittest.mock. Patch written by Amit Saha.

@item 
bpo-20589@footnote{https://bugs.python.org/issue20589}: Invoking Path.owner() and Path.group() on Windows now raise
NotImplementedError instead of ImportError.

@item 
bpo-26177@footnote{https://bugs.python.org/issue26177}: Fixed the keys() method for Canvas and Scrollbar widgets.

@item 
bpo-15068@footnote{https://bugs.python.org/issue15068}: Got rid of excessive buffering in fileinput. The bufsize
parameter is now deprecated and ignored.

@item 
bpo-19475@footnote{https://bugs.python.org/issue19475}: Added an optional argument timespec to the datetime isoformat()
method to choose the precision of the time component.

@item 
bpo-2202@footnote{https://bugs.python.org/issue2202}: Fix UnboundLocalError in
AbstractDigestAuthHandler.get_algorithm_impls. Initial patch by Mathieu
Dupuy.

@item 
bpo-26167@footnote{https://bugs.python.org/issue26167}: Minimized overhead in copy.copy() and copy.deepcopy().
Optimized copying and deepcopying bytearrays, NotImplemented, slices,
short lists, tuples, dicts, sets.

@item 
bpo-25718@footnote{https://bugs.python.org/issue25718}: Fixed pickling and copying the accumulate() iterator with total
is None.

@item 
bpo-26475@footnote{https://bugs.python.org/issue26475}: Fixed debugging output for regular expressions with the (?x)
flag.

@item 
bpo-26482@footnote{https://bugs.python.org/issue26482}: Allowed pickling recursive dequeues.

@item 
bpo-26335@footnote{https://bugs.python.org/issue26335}: Make mmap.write() return the number of bytes written like other
write methods.  Patch by Jakub Stasiak.

@item 
bpo-26457@footnote{https://bugs.python.org/issue26457}: Fixed the subnets() methods in IP network classes for the case
when resulting prefix length is equal to maximal prefix length. Based on
patch by Xiang Zhang.

@item 
bpo-26385@footnote{https://bugs.python.org/issue26385}: Remove the file if the internal open() call in
NamedTemporaryFile() fails. Patch by Silent Ghost.

@item 
bpo-26402@footnote{https://bugs.python.org/issue26402}: Fix XML-RPC client to retry when the server shuts down a
persistent connection.  This was a regression related to the new
http.client.RemoteDisconnected exception in 3.5.0a4.

@item 
bpo-25913@footnote{https://bugs.python.org/issue25913}: Leading @code{<~} is optional now in base64.a85decode() with
adobe=True.  Patch by Swati Jaiswal.

@item 
bpo-26186@footnote{https://bugs.python.org/issue26186}: Remove an invalid type check in importlib.util.LazyLoader.

@item 
bpo-26367@footnote{https://bugs.python.org/issue26367}: importlib.__import__() raises ImportError like
builtins.__import__() when @code{level} is specified but without an
accompanying package specified.

@item 
bpo-26309@footnote{https://bugs.python.org/issue26309}: In the “socketserver” module, shut down the request (closing
the connected socket) when verify_request() returns false.  Patch by Aviv
Palivoda.

@item 
bpo-23430@footnote{https://bugs.python.org/issue23430}: Change the socketserver module to only catch exceptions raised
from a request handler that are derived from Exception (instead of
BaseException). Therefore SystemExit and KeyboardInterrupt no longer
trigger the handle_error() method, and will now to stop a single-threaded
server.
@end itemize

@node Security<32>,Library<61>,Library<60>,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id462}@anchor{142c}
@subsubsection Security



@itemize -

@item 
bpo-25939@footnote{https://bugs.python.org/issue25939}: On Windows open the cert store readonly in
ssl.enum_certificates.
@end itemize

@node Library<61>,IDLE<43>,Security<32>,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id463}@anchor{142d}
@subsubsection Library



@itemize -

@item 
bpo-25995@footnote{https://bugs.python.org/issue25995}: os.walk() no longer uses FDs proportional to the tree depth.

@item 
bpo-25994@footnote{https://bugs.python.org/issue25994}: Added the close() method and the support of the context manager
protocol for the os.scandir() iterator.

@item 
bpo-23992@footnote{https://bugs.python.org/issue23992}: multiprocessing: make MapResult not fail-fast upon exception.

@item 
bpo-26243@footnote{https://bugs.python.org/issue26243}: Support keyword arguments to zlib.compress().  Patch by Aviv
Palivoda.

@item 
bpo-26117@footnote{https://bugs.python.org/issue26117}: The os.scandir() iterator now closes file descriptor not only
when the iteration is finished, but when it was failed with error.

@item 
bpo-25949@footnote{https://bugs.python.org/issue25949}: __dict__ for an OrderedDict instance is now created only when
needed.

@item 
bpo-25911@footnote{https://bugs.python.org/issue25911}: Restored support of bytes paths in os.walk() on Windows.

@item 
bpo-26045@footnote{https://bugs.python.org/issue26045}: Add UTF-8 suggestion to error message when posting a
non-Latin-1 string with http.client.

@item 
bpo-26039@footnote{https://bugs.python.org/issue26039}: Added zipfile.ZipInfo.from_file() and zipinfo.ZipInfo.is_dir().
Patch by Thomas Kluyver.

@item 
bpo-12923@footnote{https://bugs.python.org/issue12923}: Reset FancyURLopener’s redirect counter even if there is an
exception. Based on patches by Brian Brazil and Daniel Rocco.

@item 
bpo-25945@footnote{https://bugs.python.org/issue25945}: Fixed a crash when unpickle the functools.partial object with
wrong state. Fixed a leak in failed functools.partial constructor. “args”
and “keywords” attributes of functools.partial have now always types tuple
and dict correspondingly.

@item 
bpo-26202@footnote{https://bugs.python.org/issue26202}: copy.deepcopy() now correctly copies range() objects with
non-atomic attributes.

@item 
bpo-23076@footnote{https://bugs.python.org/issue23076}: Path.glob() now raises a ValueError if it’s called with an
invalid pattern. Patch by Thomas Nyberg.

@item 
bpo-19883@footnote{https://bugs.python.org/issue19883}: Fixed possible integer overflows in zipimport.

@item 
bpo-26227@footnote{https://bugs.python.org/issue26227}: On Windows, getnameinfo(), gethostbyaddr() and
gethostbyname_ex() functions of the socket module now decode the hostname
from the ANSI code page rather than UTF-8.

@item 
bpo-26099@footnote{https://bugs.python.org/issue26099}: The site module now writes an error into stderr if
sitecustomize module can be imported but executing the module raise an
ImportError. Same change for usercustomize.

@item 
bpo-26147@footnote{https://bugs.python.org/issue26147}: xmlrpc now works with strings not encodable with used non-UTF-8
encoding.

@item 
bpo-25935@footnote{https://bugs.python.org/issue25935}: Garbage collector now breaks reference loops with OrderedDict.

@item 
bpo-16620@footnote{https://bugs.python.org/issue16620}: Fixed AttributeError in msilib.Directory.glob().

@item 
bpo-26013@footnote{https://bugs.python.org/issue26013}: Added compatibility with broken protocol 2 pickles created in
old Python 3 versions (3.4.3 and lower).

@item 
bpo-26129@footnote{https://bugs.python.org/issue26129}: Deprecated accepting non-integers in grp.getgrgid().

@item 
bpo-25850@footnote{https://bugs.python.org/issue25850}: Use cross-compilation by default for 64-bit Windows.

@item 
bpo-25822@footnote{https://bugs.python.org/issue25822}: Add docstrings to the fields of urllib.parse results. Patch
contributed by Swati Jaiswal.

@item 
bpo-22642@footnote{https://bugs.python.org/issue22642}: Convert trace module option parsing mechanism to argparse.
Patch contributed by SilentGhost.

@item 
bpo-24705@footnote{https://bugs.python.org/issue24705}: Fix sysconfig._parse_makefile not expanding $@{@} vars appearing
before $() vars.

@item 
bpo-26069@footnote{https://bugs.python.org/issue26069}: Remove the deprecated apis in the trace module.

@item 
bpo-22138@footnote{https://bugs.python.org/issue22138}: Fix mock.patch behavior when patching descriptors. Restore
original values after patching. Patch contributed by Sean McCully.

@item 
bpo-25672@footnote{https://bugs.python.org/issue25672}: In the ssl module, enable the SSL_MODE_RELEASE_BUFFERS mode
option if it is safe to do so.

@item 
bpo-26012@footnote{https://bugs.python.org/issue26012}: Don’t traverse into symlinks for @code{**} pattern in
pathlib.Path.[r]glob().

@item 
bpo-24120@footnote{https://bugs.python.org/issue24120}: Ignore PermissionError when traversing a tree with
pathlib.Path.[r]glob(). Patch by Ulrich Petri.

@item 
bpo-21815@footnote{https://bugs.python.org/issue21815}: Accept ] characters in the data portion of imap responses, in
order to handle the flags with square brackets accepted and produced by
servers such as gmail.

@item 
bpo-25447@footnote{https://bugs.python.org/issue25447}: fileinput now uses sys.stdin as-is if it does not have a buffer
attribute (restores backward compatibility).

@item 
bpo-25971@footnote{https://bugs.python.org/issue25971}: Optimized creating Fractions from floats by 2 times and from
Decimals by 3 times.

@item 
bpo-25802@footnote{https://bugs.python.org/issue25802}: Document as deprecated the remaining implementations of
importlib.abc.Loader.load_module().

@item 
bpo-25928@footnote{https://bugs.python.org/issue25928}: Add Decimal.as_integer_ratio().

@item 
bpo-25447@footnote{https://bugs.python.org/issue25447}: Copying the lru_cache() wrapper object now always works,
independently from the type of the wrapped object (by returning the
original object unchanged).

@item 
bpo-25768@footnote{https://bugs.python.org/issue25768}: Have the functions in compileall return booleans instead of
ints and add proper documentation and tests for the return values.

@item 
bpo-24103@footnote{https://bugs.python.org/issue24103}: Fixed possible use after free in ElementTree.XMLPullParser.

@item 
bpo-25860@footnote{https://bugs.python.org/issue25860}: os.fwalk() no longer skips remaining directories when error
occurs. Original patch by Samson Lee.

@item 
bpo-25914@footnote{https://bugs.python.org/issue25914}: Fixed and simplified OrderedDict.__sizeof__.

@item 
bpo-25869@footnote{https://bugs.python.org/issue25869}: Optimized deepcopying ElementTree; it is now 20 times faster.

@item 
bpo-25873@footnote{https://bugs.python.org/issue25873}: Optimized iterating ElementTree.  Iterating elements
Element.iter() is now 40% faster, iterating text Element.itertext() is now
up to 2.5 times faster.

@item 
bpo-25902@footnote{https://bugs.python.org/issue25902}: Fixed various refcount issues in ElementTree iteration.

@item 
bpo-22227@footnote{https://bugs.python.org/issue22227}: The TarFile iterator is reimplemented using generator. This
implementation is simpler that using class.

@item 
bpo-25638@footnote{https://bugs.python.org/issue25638}: Optimized ElementTree.iterparse(); it is now 2x faster.
Optimized ElementTree parsing; it is now 10% faster.

@item 
bpo-25761@footnote{https://bugs.python.org/issue25761}: Improved detecting errors in broken pickle data.

@item 
bpo-25717@footnote{https://bugs.python.org/issue25717}: Restore the previous behaviour of tolerating most fstat()
errors when opening files.  This was a regression in 3.5a1, and stopped
anonymous temporary files from working in special cases.

@item 
bpo-24903@footnote{https://bugs.python.org/issue24903}: Fix regression in number of arguments compileall accepts when
‘-d’ is specified.  The check on the number of arguments has been dropped
completely as it never worked correctly anyway.

@item 
bpo-25764@footnote{https://bugs.python.org/issue25764}: In the subprocess module, preserve any exception caused by
fork() failure when preexec_fn is used.

@item 
bpo-25771@footnote{https://bugs.python.org/issue25771}: Tweak the exception message for importlib.util.resolve_name()
when ‘package’ isn’t specified but necessary.

@item 
bpo-6478@footnote{https://bugs.python.org/issue6478}: _strptime’s regexp cache now is reset after changing timezone
with time.tzset().

@item 
bpo-14285@footnote{https://bugs.python.org/issue14285}: When executing a package with the “python -m package” option,
and package initialization fails, a proper traceback is now reported.  The
“runpy” module now lets exceptions from package initialization pass back
to the caller, rather than raising ImportError.

@item 
bpo-19771@footnote{https://bugs.python.org/issue19771}: Also in runpy and the “-m” option, omit the irrelevant message
“… is a package and cannot be directly executed” if the package could
not even be initialized (e.g. due to a bad @code{*.pyc} file).

@item 
bpo-25177@footnote{https://bugs.python.org/issue25177}: Fixed problem with the mean of very small and very large
numbers. As a side effect, statistics.mean and statistics.variance should
be significantly faster.

@item 
bpo-25718@footnote{https://bugs.python.org/issue25718}: Fixed copying object with state with boolean value is false.

@item 
bpo-10131@footnote{https://bugs.python.org/issue10131}: Fixed deep copying of minidom documents.  Based on patch by
Marian Ganisin.

@item 
bpo-7990@footnote{https://bugs.python.org/issue7990}: dir() on ElementTree.Element now lists properties: “tag”,
“text”, “tail” and “attrib”.  Original patch by Santoso Wijaya.

@item 
bpo-25725@footnote{https://bugs.python.org/issue25725}: Fixed a reference leak in pickle.loads() when unpickling
invalid data including tuple instructions.

@item 
bpo-25663@footnote{https://bugs.python.org/issue25663}: In the Readline completer, avoid listing duplicate global
names, and search the global namespace before searching builtins.

@item 
bpo-25688@footnote{https://bugs.python.org/issue25688}: Fixed file leak in ElementTree.iterparse() raising an error.

@item 
bpo-23914@footnote{https://bugs.python.org/issue23914}: Fixed SystemError raised by unpickler on broken pickle data.

@item 
bpo-25691@footnote{https://bugs.python.org/issue25691}: Fixed crash on deleting ElementTree.Element attributes.

@item 
bpo-25624@footnote{https://bugs.python.org/issue25624}: ZipFile now always writes a ZIP_STORED header for directory
entries.  Patch by Dingyuan Wang.

@item 
bpo-25626@footnote{https://bugs.python.org/issue25626}: Change three zlib functions to accept sizes that fit in
Py_ssize_t, but internally cap those sizes to UINT_MAX.  This resolves a
regression in 3.5 where GzipFile.read() failed to read chunks larger than
2 or 4 GiB.  The change affects the zlib.Decompress.decompress()
max_length parameter, the zlib.decompress() bufsize parameter, and the
zlib.Decompress.flush() length parameter.

@item 
bpo-25583@footnote{https://bugs.python.org/issue25583}: Avoid incorrect errors raised by os.makedirs(exist_ok=True)
when the OS gives priority to errors such as EACCES over EEXIST.

@item 
bpo-25593@footnote{https://bugs.python.org/issue25593}: Change semantics of EventLoop.stop() in asyncio.

@item 
bpo-6973@footnote{https://bugs.python.org/issue6973}: When we know a subprocess.Popen process has died, do not allow
the send_signal(), terminate(), or kill() methods to do anything as they
could potentially signal a different process.

@item 
bpo-23883@footnote{https://bugs.python.org/issue23883}: Added missing APIs to __all__ to match the documented APIs for
the following modules: calendar, csv, enum, fileinput, ftplib, logging,
optparse, tarfile, threading and wave.  Also added a
test.support.check__all__() helper. Patches by Jacek Kołodziej, Mauro S.
M. Rodrigues and Joel Taddei.

@item 
bpo-25590@footnote{https://bugs.python.org/issue25590}: In the Readline completer, only call getattr() once per
attribute.  Also complete names of attributes such as properties and slots
which are listed by dir() but not yet created on an instance.

@item 
bpo-25498@footnote{https://bugs.python.org/issue25498}: Fix a crash when garbage-collecting ctypes objects created by
wrapping a memoryview.  This was a regression made in 3.5a1.  Based on
patch by Eryksun.

@item 
bpo-25584@footnote{https://bugs.python.org/issue25584}: Added “escape” to the __all__ list in the glob module.

@item 
bpo-25584@footnote{https://bugs.python.org/issue25584}: Fixed recursive glob() with patterns starting with @code{**}.

@item 
bpo-25446@footnote{https://bugs.python.org/issue25446}: Fix regression in smtplib’s AUTH LOGIN support.

@item 
bpo-18010@footnote{https://bugs.python.org/issue18010}: Fix the pydoc web server’s module search function to handle
exceptions from importing packages.

@item 
bpo-25554@footnote{https://bugs.python.org/issue25554}: Got rid of circular references in regular expression parsing.

@item 
bpo-18973@footnote{https://bugs.python.org/issue18973}: Command-line interface of the calendar module now uses argparse
instead of optparse.

@item 
bpo-25510@footnote{https://bugs.python.org/issue25510}: fileinput.FileInput.readline() now returns b’’ instead of ‘’ at
the end if the FileInput was opened with binary mode. Patch by Ryosuke
Ito.

@item 
bpo-25503@footnote{https://bugs.python.org/issue25503}: Fixed inspect.getdoc() for inherited docstrings of properties.
Original patch by John Mark Vandenberg.

@item 
bpo-25515@footnote{https://bugs.python.org/issue25515}: Always use os.urandom as a source of randomness in uuid.uuid4.

@item 
bpo-21827@footnote{https://bugs.python.org/issue21827}: Fixed textwrap.dedent() for the case when largest common
whitespace is a substring of smallest leading whitespace. Based on patch
by Robert Li.

@item 
bpo-25447@footnote{https://bugs.python.org/issue25447}: The lru_cache() wrapper objects now can be copied and pickled
(by returning the original object unchanged).

@item 
bpo-25390@footnote{https://bugs.python.org/issue25390}: typing: Don’t crash on Union[str, Pattern].

@item 
bpo-25441@footnote{https://bugs.python.org/issue25441}: asyncio: Raise error from drain() when socket is closed.

@item 
bpo-25410@footnote{https://bugs.python.org/issue25410}: Cleaned up and fixed minor bugs in C implementation of
OrderedDict.

@item 
bpo-25411@footnote{https://bugs.python.org/issue25411}: Improved Unicode support in SMTPHandler through better use of
the email package. Thanks to user simon04 for the patch.

@item 
Move the imp module from a PendingDeprecationWarning to
DeprecationWarning.

@item 
bpo-25407@footnote{https://bugs.python.org/issue25407}: Remove mentions of the formatter module being removed in Python
3.6.

@item 
bpo-25406@footnote{https://bugs.python.org/issue25406}: Fixed a bug in C implementation of OrderedDict.move_to_end()
that caused segmentation fault or hang in iterating after moving several
items to the start of ordered dict.

@item 
bpo-25382@footnote{https://bugs.python.org/issue25382}: pickletools.dis() now outputs implicit memo index for the
MEMOIZE opcode.

@item 
bpo-25357@footnote{https://bugs.python.org/issue25357}: Add an optional newline parameter to binascii.b2a_base64().
base64.b64encode() uses it to avoid a memory copy.

@item 
bpo-24164@footnote{https://bugs.python.org/issue24164}: Objects that need calling @code{__new__} with keyword arguments,
can now be pickled using pickle protocols older than protocol version 4.

@item 
bpo-25364@footnote{https://bugs.python.org/issue25364}: zipfile now works in threads disabled builds.

@item 
bpo-25328@footnote{https://bugs.python.org/issue25328}: smtpd’s SMTPChannel now correctly raises a ValueError if both
decode_data and enable_SMTPUTF8 are set to true.

@item 
bpo-16099@footnote{https://bugs.python.org/issue16099}: RobotFileParser now supports Crawl-delay and Request-rate
extensions.  Patch by Nikolay Bogoychev.

@item 
bpo-25316@footnote{https://bugs.python.org/issue25316}: distutils raises OSError instead of DistutilsPlatformError when
MSVC is not installed.

@item 
bpo-25380@footnote{https://bugs.python.org/issue25380}: Fixed protocol for the STACK_GLOBAL opcode in
pickletools.opcodes.

@item 
bpo-23972@footnote{https://bugs.python.org/issue23972}: Updates asyncio datagram create method allowing reuseport and
reuseaddr socket options to be set prior to binding the socket. Mirroring
the existing asyncio create_server method the reuseaddr option for
datagram sockets defaults to True if the O/S is ‘posix’ (except if the
platform is Cygwin). Patch by Chris Laws.

@item 
bpo-25304@footnote{https://bugs.python.org/issue25304}: Add asyncio.run_coroutine_threadsafe().  This lets you submit a
coroutine to a loop from another thread, returning a
concurrent.futures.Future.  By Vincent Michel.

@item 
bpo-25232@footnote{https://bugs.python.org/issue25232}: Fix CGIRequestHandler to split the query from the URL at the
first question mark (?) rather than the last. Patch from Xiang Zhang.

@item 
bpo-24657@footnote{https://bugs.python.org/issue24657}: Prevent CGIRequestHandler from collapsing slashes in the query
part of the URL as if it were a path. Patch from Xiang Zhang.

@item 
bpo-25287@footnote{https://bugs.python.org/issue25287}: Don’t add crypt.METHOD_CRYPT to crypt.methods if it’s not
supported. Check if it is supported, it may not be supported on OpenBSD
for example.

@item 
bpo-23600@footnote{https://bugs.python.org/issue23600}: Default implementation of tzinfo.fromutc() was returning wrong
results in some cases.

@item 
bpo-25203@footnote{https://bugs.python.org/issue25203}: Failed readline.set_completer_delims() no longer left the
module in inconsistent state.

@item 
bpo-25011@footnote{https://bugs.python.org/issue25011}: rlcompleter now omits private and special attribute names
unless the prefix starts with underscores.

@item 
bpo-25209@footnote{https://bugs.python.org/issue25209}: rlcompleter now can add a space or a colon after completed
keyword.

@item 
bpo-22241@footnote{https://bugs.python.org/issue22241}: timezone.utc name is now plain ‘UTC’, not ‘UTC-00:00’.

@item 
bpo-23517@footnote{https://bugs.python.org/issue23517}: fromtimestamp() and utcfromtimestamp() methods of
datetime.datetime now round microseconds to nearest with ties going to
nearest even integer (ROUND_HALF_EVEN), as round(float), instead of
rounding towards -Infinity (ROUND_FLOOR).

@item 
bpo-23552@footnote{https://bugs.python.org/issue23552}: Timeit now warns when there is substantial (4x) variance
between best and worst times. Patch from Serhiy Storchaka.

@item 
bpo-24633@footnote{https://bugs.python.org/issue24633}: site-packages/README -> README.txt.

@item 
bpo-24879@footnote{https://bugs.python.org/issue24879}: help() and pydoc can now list named tuple fields in the order
they were defined rather than alphabetically.  The ordering is determined
by the _fields attribute if present.

@item 
bpo-24874@footnote{https://bugs.python.org/issue24874}: Improve speed of itertools.cycle() and make its pickle more
compact.

@item 
Fix crash in itertools.cycle.__setstate__() when the first argument wasn’t
a list.

@item 
bpo-20059@footnote{https://bugs.python.org/issue20059}: urllib.parse raises ValueError on all invalid ports. Patch by
Martin Panter.

@item 
bpo-24360@footnote{https://bugs.python.org/issue24360}: Improve __repr__ of argparse.Namespace() for invalid
identifiers.  Patch by Matthias Bussonnier.

@item 
bpo-23426@footnote{https://bugs.python.org/issue23426}: run_setup was broken in distutils. Patch from Alexander
Belopolsky.

@item 
bpo-13938@footnote{https://bugs.python.org/issue13938}: 2to3 converts StringTypes to a tuple. Patch from Mark Hammond.

@item 
bpo-2091@footnote{https://bugs.python.org/issue2091}: open() accepted a ‘U’ mode string containing ‘+’, but ‘U’ can
only be used with ‘r’. Patch from Jeff Balogh and John O’Connor.

@item 
bpo-8585@footnote{https://bugs.python.org/issue8585}: improved tests for zipimporter2. Patch from Mark Lawrence.

@item 
bpo-18622@footnote{https://bugs.python.org/issue18622}: unittest.mock.mock_open().reset_mock would recurse infinitely.
Patch from Nicola Palumbo and Laurent De Buyst.

@item 
bpo-24426@footnote{https://bugs.python.org/issue24426}: Fast searching optimization in regular expressions now works
for patterns that starts with capturing groups.  Fast searching
optimization now can’t be disabled at compile time.

@item 
bpo-23661@footnote{https://bugs.python.org/issue23661}: unittest.mock side_effects can now be exceptions again. This
was a regression vs Python 3.4. Patch from Ignacio Rossi

@item 
bpo-13248@footnote{https://bugs.python.org/issue13248}: Remove deprecated inspect.getmoduleinfo function.

@item 
bpo-25578@footnote{https://bugs.python.org/issue25578}: Fix (another) memory leak in SSLSocket.getpeercer().

@item 
bpo-25530@footnote{https://bugs.python.org/issue25530}: Disable the vulnerable SSLv3 protocol by default when creating
ssl.SSLContext.

@item 
bpo-25569@footnote{https://bugs.python.org/issue25569}: Fix memory leak in SSLSocket.getpeercert().

@item 
bpo-25471@footnote{https://bugs.python.org/issue25471}: Sockets returned from accept() shouldn’t appear to be
nonblocking.

@item 
bpo-25319@footnote{https://bugs.python.org/issue25319}: When threading.Event is reinitialized, the underlying condition
should use a regular lock rather than a recursive lock.

@item 
Skip getaddrinfo if host is already resolved. Patch by A. Jesse Jiryu
Davis.

@item 
bpo-26050@footnote{https://bugs.python.org/issue26050}: Add asyncio.StreamReader.readuntil() method. Patch by Марк
Коренберг.

@item 
bpo-25924@footnote{https://bugs.python.org/issue25924}: Avoid unnecessary serialization of getaddrinfo(3) calls on OS X
versions 10.5 or higher.  Original patch by A. Jesse Jiryu Davis.

@item 
bpo-26406@footnote{https://bugs.python.org/issue26406}: Avoid unnecessary serialization of getaddrinfo(3) calls on
current versions of OpenBSD and NetBSD.  Patch by A. Jesse Jiryu Davis.

@item 
bpo-26848@footnote{https://bugs.python.org/issue26848}: Fix asyncio/subprocess.communicate() to handle empty input.
Patch by Jack O’Connor.

@item 
bpo-27040@footnote{https://bugs.python.org/issue27040}: Add loop.get_exception_handler method

@item 
bpo-27041@footnote{https://bugs.python.org/issue27041}: asyncio: Add loop.create_future method
@end itemize

@node IDLE<43>,Documentation<48>,Library<61>,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id464}@anchor{142e}
@subsubsection IDLE



@itemize -

@item 
bpo-20640@footnote{https://bugs.python.org/issue20640}: Add tests for idlelib.configHelpSourceEdit. Patch by Saimadhav
Heblikar.

@item 
In the ‘IDLE-console differences’ section of the IDLE doc, clarify how
running with IDLE affects sys.modules and the standard streams.

@item 
bpo-25507@footnote{https://bugs.python.org/issue25507}: fix incorrect change in IOBinding that prevented printing.
Augment IOBinding htest to include all major IOBinding functions.

@item 
bpo-25905@footnote{https://bugs.python.org/issue25905}: Revert unwanted conversion of ‘ to ’ RIGHT SINGLE QUOTATION
MARK in README.txt and open this and NEWS.txt with ‘ascii’. Re-encode
CREDITS.txt to utf-8 and open it with ‘utf-8’.

@item 
bpo-15348@footnote{https://bugs.python.org/issue15348}: Stop the debugger engine (normally in a user process) before
closing the debugger window (running in the IDLE process). This prevents
the RuntimeErrors that were being caught and ignored.

@item 
bpo-24455@footnote{https://bugs.python.org/issue24455}: Prevent IDLE from hanging when a) closing the shell while the
debugger is active (15347); b) closing the debugger with the [X] button
(15348); and c) activating the debugger when already active (24455). The
patch by Mark Roseman does this by making two changes. 1. Suspend and
resume the gui.interaction method with the tcl vwait mechanism intended
for this purpose (instead of root.mainloop & .quit). 2. In gui.run, allow
any existing interaction to terminate first.

@item 
Change ‘The program’ to ‘Your program’ in an IDLE ‘kill program?’ message
to make it clearer that the program referred to is the currently running
user program, not IDLE itself.

@item 
bpo-24750@footnote{https://bugs.python.org/issue24750}: Improve the appearance of the IDLE editor window status bar.
Patch by Mark Roseman.

@item 
bpo-25313@footnote{https://bugs.python.org/issue25313}: Change the handling of new built-in text color themes to better
address the compatibility problem introduced by the addition of IDLE Dark.
Consistently use the revised idleConf.CurrentTheme everywhere in idlelib.

@item 
bpo-24782@footnote{https://bugs.python.org/issue24782}: Extension configuration is now a tab in the IDLE Preferences
dialog rather than a separate dialog.  The former tabs are now a sorted
list.  Patch by Mark Roseman.

@item 
bpo-22726@footnote{https://bugs.python.org/issue22726}: Re-activate the config dialog help button with some content
about the other buttons and the new IDLE Dark theme.

@item 
bpo-24820@footnote{https://bugs.python.org/issue24820}: IDLE now has an ‘IDLE Dark’ built-in text color theme. It is
more or less IDLE Classic inverted, with a cobalt blue background.
Strings, comments, keywords, … are still green, red, orange, … . To
use it with IDLEs released before November 2015, hit the ‘Save as New
Custom Theme’ button and enter a new name, such as ‘Custom Dark’.  The
custom theme will work with any IDLE release, and can be modified.

@item 
bpo-25224@footnote{https://bugs.python.org/issue25224}: README.txt is now an idlelib index for IDLE developers and
curious users. The previous user content is now in the IDLE doc chapter.
‘IDLE’ now means ‘Integrated Development and Learning Environment’.

@item 
bpo-24820@footnote{https://bugs.python.org/issue24820}: Users can now set breakpoint colors in Settings -> Custom
Highlighting. Original patch by Mark Roseman.

@item 
bpo-24972@footnote{https://bugs.python.org/issue24972}: Inactive selection background now matches active selection
background, as configured by users, on all systems.  Found items are now
always highlighted on Windows.  Initial patch by Mark Roseman.

@item 
bpo-24570@footnote{https://bugs.python.org/issue24570}: Idle: make calltip and completion boxes appear on Macs affected
by a tk regression.  Initial patch by Mark Roseman.

@item 
bpo-24988@footnote{https://bugs.python.org/issue24988}: Idle ScrolledList context menus (used in debugger) now work on
Mac Aqua. Patch by Mark Roseman.

@item 
bpo-24801@footnote{https://bugs.python.org/issue24801}: Make right-click for context menu work on Mac Aqua. Patch by
Mark Roseman.

@item 
bpo-25173@footnote{https://bugs.python.org/issue25173}: Associate tkinter messageboxes with a specific widget. For Mac
OSX, make them a ‘sheet’.  Patch by Mark Roseman.

@item 
bpo-25198@footnote{https://bugs.python.org/issue25198}: Enhance the initial html viewer now used for Idle Help.
Properly indent fixed-pitch text (patch by Mark Roseman). Give code
snippet a very Sphinx-like light blueish-gray background. Re-use initial
width and height set by users for shell and editor. When the Table of
Contents (TOC) menu is used, put the section header at the top of the
screen.

@item 
bpo-25225@footnote{https://bugs.python.org/issue25225}: Condense and rewrite Idle doc section on text colors.

@item 
bpo-21995@footnote{https://bugs.python.org/issue21995}: Explain some differences between IDLE and console Python.

@item 
bpo-22820@footnote{https://bugs.python.org/issue22820}: Explain need for @emph{print} when running file from Idle editor.

@item 
bpo-25224@footnote{https://bugs.python.org/issue25224}: Doc: augment Idle feature list and no-subprocess section.

@item 
bpo-25219@footnote{https://bugs.python.org/issue25219}: Update doc for Idle command line options. Some were missing and
notes were not correct.

@item 
bpo-24861@footnote{https://bugs.python.org/issue24861}: Most of idlelib is private and subject to change. Use
idleib.idle.* to start Idle. See idlelib.__init__.__doc__.

@item 
bpo-25199@footnote{https://bugs.python.org/issue25199}: Idle: add synchronization comments for future maintainers.

@item 
bpo-16893@footnote{https://bugs.python.org/issue16893}: Replace help.txt with help.html for Idle doc display. The new
idlelib/help.html is rstripped Doc/build/html/library/idle.html. It looks
better than help.txt and will better document Idle as released. The
tkinter html viewer that works for this file was written by Rose Roseman.
The now unused EditorWindow.HelpDialog class and helt.txt file are
deprecated.

@item 
bpo-24199@footnote{https://bugs.python.org/issue24199}: Deprecate unused idlelib.idlever with possible removal in 3.6.

@item 
bpo-24790@footnote{https://bugs.python.org/issue24790}: Remove extraneous code (which also create 2 & 3 conflicts).
@end itemize

@node Documentation<48>,Tests<48>,IDLE<43>,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id465}@anchor{142f}
@subsubsection Documentation



@itemize -

@item 
bpo-26736@footnote{https://bugs.python.org/issue26736}: Used HTTPS for external links in the documentation if possible.

@item 
bpo-6953@footnote{https://bugs.python.org/issue6953}: Rework the Readline module documentation to group related
functions together, and add more details such as what underlying Readline
functions and variables are accessed.

@item 
bpo-23606@footnote{https://bugs.python.org/issue23606}: Adds note to ctypes documentation regarding cdll.msvcrt.

@item 
bpo-24952@footnote{https://bugs.python.org/issue24952}: Clarify the default size argument of stack_size() in the
“threading” and “_thread” modules. Patch from Mattip.

@item 
bpo-26014@footnote{https://bugs.python.org/issue26014}: Update 3.x packaging documentation: * “See also” links to the
new docs are now provided in the legacy pages * links to setuptools
documentation have been updated
@end itemize

@node Tests<48>,Build<52>,Documentation<48>,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id466}@anchor{1430}
@subsubsection Tests



@itemize -

@item 
bpo-21916@footnote{https://bugs.python.org/issue21916}: Added tests for the turtle module.  Patch by ingrid, Gregory
Loyse and Jelle Zijlstra.

@item 
bpo-26295@footnote{https://bugs.python.org/issue26295}: When using “python3 -m test –testdir=TESTDIR”, regrtest
doesn’t add “test.” prefix to test module names.

@item 
bpo-26523@footnote{https://bugs.python.org/issue26523}: The multiprocessing thread pool (multiprocessing.dummy.Pool)
was untested.

@item 
bpo-26015@footnote{https://bugs.python.org/issue26015}: Added new tests for pickling iterators of mutable sequences.

@item 
bpo-26325@footnote{https://bugs.python.org/issue26325}: Added test.support.check_no_resource_warning() to check that no
ResourceWarning is emitted.

@item 
bpo-25940@footnote{https://bugs.python.org/issue25940}: Changed test_ssl to use its internal local server more.  This
avoids relying on svn.python.org, which recently changed root certificate.

@item 
bpo-25616@footnote{https://bugs.python.org/issue25616}: Tests for OrderedDict are extracted from test_collections into
separate file test_ordered_dict.

@item 
bpo-25449@footnote{https://bugs.python.org/issue25449}: Added tests for OrderedDict subclasses.

@item 
bpo-25188@footnote{https://bugs.python.org/issue25188}: Add -P/–pgo to test.regrtest to suppress error output when
running the test suite for the purposes of a PGO build. Initial patch by
Alecsandru Patrascu.

@item 
bpo-22806@footnote{https://bugs.python.org/issue22806}: Add @code{python -m test --list-tests} command to list tests.

@item 
bpo-18174@footnote{https://bugs.python.org/issue18174}: @code{python -m test --huntrleaks ...} now also checks for leak of
file descriptors. Patch written by Richard Oudkerk.

@item 
bpo-25260@footnote{https://bugs.python.org/issue25260}: Fix @code{python -m test --coverage} on Windows. Remove the list
of ignored directories.

@item 
@code{PCbuild\rt.bat} now accepts an unlimited number of arguments to pass
along to regrtest.py.  Previously there was a limit of 9.

@item 
bpo-26583@footnote{https://bugs.python.org/issue26583}: Skip test_timestamp_overflow in test_import if bytecode files
cannot be written.
@end itemize

@node Build<52>,Windows<46>,Tests<48>,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id467}@anchor{1431}
@subsubsection Build



@itemize -

@item 
bpo-21277@footnote{https://bugs.python.org/issue21277}: Don’t try to link _ctypes with a ffi_convenience library.

@item 
bpo-26884@footnote{https://bugs.python.org/issue26884}: Fix linking extension modules for cross builds. Patch by Xavier
de Gaye.

@item 
bpo-26932@footnote{https://bugs.python.org/issue26932}: Fixed support of RTLD_* constants defined as enum values, not
via macros (in particular on Android).  Patch by Chi Hsuan Yen.

@item 
bpo-22359@footnote{https://bugs.python.org/issue22359}: Disable the rules for running _freeze_importlib and pgen when
cross-compiling.  The output of these programs is normally saved with the
source code anyway, and is still regenerated when doing a native build.
Patch by Xavier de Gaye.

@item 
bpo-21668@footnote{https://bugs.python.org/issue21668}: Link audioop, _datetime, _ctypes_test modules to libm, except
on Mac OS X. Patch written by Chi Hsuan Yen.

@item 
bpo-25702@footnote{https://bugs.python.org/issue25702}: A –with-lto configure option has been added that will enable
link time optimizations at build time during a make profile-opt. Some
compilers and toolchains are known to not produce stable code when using
LTO, be sure to test things thoroughly before relying on it. It can
provide a few % speed up over profile-opt alone.

@item 
bpo-26624@footnote{https://bugs.python.org/issue26624}: Adds validation of ucrtbase[d].dll version with warning for old
versions.

@item 
bpo-17603@footnote{https://bugs.python.org/issue17603}: Avoid error about nonexistent fileblocks.o file by using a
lower-level check for st_blocks in struct stat.

@item 
bpo-26079@footnote{https://bugs.python.org/issue26079}: Fixing the build output folder for tix-8.4.3.6. Patch by Bjoern
Thiel.

@item 
bpo-26465@footnote{https://bugs.python.org/issue26465}: Update Windows builds to use OpenSSL 1.0.2g.

@item 
bpo-25348@footnote{https://bugs.python.org/issue25348}: Added @code{--pgo} and @code{--pgo-job} arguments to
@code{PCbuild\build.bat} for building with Profile-Guided Optimization.  The
old @code{PCbuild\build_pgo.bat} script is removed.

@item 
bpo-25827@footnote{https://bugs.python.org/issue25827}: Add support for building with ICC to @code{configure}, including a
new @code{--with-icc} flag.

@item 
bpo-25696@footnote{https://bugs.python.org/issue25696}: Fix installation of Python on UNIX with make -j9.

@item 
bpo-24986@footnote{https://bugs.python.org/issue24986}: It is now possible to build Python on Windows without errors
when external libraries are not available.

@item 
bpo-24421@footnote{https://bugs.python.org/issue24421}: Compile Modules/_math.c once, before building extensions.
Previously it could fail to compile properly if the math and cmath builds
were concurrent.

@item 
bpo-26465@footnote{https://bugs.python.org/issue26465}: Update OS X 10.5+ 32-bit-only installer to build and link with
OpenSSL 1.0.2g.

@item 
bpo-26268@footnote{https://bugs.python.org/issue26268}: Update Windows builds to use OpenSSL 1.0.2f.

@item 
bpo-25136@footnote{https://bugs.python.org/issue25136}: Support Apple Xcode 7’s new textual SDK stub libraries.

@item 
bpo-24324@footnote{https://bugs.python.org/issue24324}: Do not enable unreachable code warnings when using gcc as the
option does not work correctly in older versions of gcc and has been
silently removed as of gcc-4.5.
@end itemize

@node Windows<46>,Tools/Demos<28>,Build<52>,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id468}@anchor{1432}
@subsubsection Windows



@itemize -

@item 
bpo-27053@footnote{https://bugs.python.org/issue27053}: Updates make_zip.py to correctly generate library ZIP file.

@item 
bpo-26268@footnote{https://bugs.python.org/issue26268}: Update the prepare_ssl.py script to handle OpenSSL releases
that don’t include the contents of the include directory (that is, 1.0.2e
and later).

@item 
bpo-26071@footnote{https://bugs.python.org/issue26071}: bdist_wininst created binaries fail to start and find 32bit
Python

@item 
bpo-26073@footnote{https://bugs.python.org/issue26073}: Update the list of magic numbers in launcher

@item 
bpo-26065@footnote{https://bugs.python.org/issue26065}: Excludes venv from library when generating embeddable distro.

@item 
bpo-25022@footnote{https://bugs.python.org/issue25022}: Removed very outdated PC/example_nt/ directory.
@end itemize

@node Tools/Demos<28>,C API<43>,Windows<46>,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id469}@anchor{1433}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-26799@footnote{https://bugs.python.org/issue26799}: Fix python-gdb.py: don’t get C types once when the Python code
is loaded, but get C types on demand. The C types can change if
python-gdb.py is loaded before the Python executable. Patch written by
Thomas Ilsche.

@item 
bpo-26271@footnote{https://bugs.python.org/issue26271}: Fix the Freeze tool to properly use flags passed through
configure. Patch by Daniel Shaulov.

@item 
bpo-26489@footnote{https://bugs.python.org/issue26489}: Add dictionary unpacking support to Tools/parser/unparse.py.
Patch by Guo Ci Teo.

@item 
bpo-26316@footnote{https://bugs.python.org/issue26316}: Fix variable name typo in Argument Clinic.

@item 
bpo-25440@footnote{https://bugs.python.org/issue25440}: Fix output of python-config –extension-suffix.

@item 
bpo-25154@footnote{https://bugs.python.org/issue25154}: The pyvenv script has been deprecated in favour of @code{python3 -m
venv}.
@end itemize

@node C API<43>,,Tools/Demos<28>,Python 3 6 0 alpha 1
@anchor{whatsnew/changelog id470}@anchor{1434}
@subsubsection C API



@itemize -

@item 
bpo-26312@footnote{https://bugs.python.org/issue26312}: SystemError is now raised in all programming bugs with using
PyArg_ParseTupleAndKeywords().  RuntimeError did raised before in some
programming bugs.

@item 
bpo-26198@footnote{https://bugs.python.org/issue26198}: ValueError is now raised instead of TypeError on buffer
overflow in parsing “es#” and “et#” format units.  SystemError is now
raised instead of TypeError on programmatical error in parsing format
string.
@end itemize

@node Python 3 5 5 final,Python 3 5 5 release candidate 1,Python 3 6 0 alpha 1,Changelog
@anchor{whatsnew/changelog python-3-5-5-final}@anchor{1435}
@subsection Python 3.5.5 final


@emph{Release date: 2018-02-04}

There were no new changes in version 3.5.5.

@node Python 3 5 5 release candidate 1,Python 3 5 4 final,Python 3 5 5 final,Changelog
@anchor{whatsnew/changelog python-3-5-5-release-candidate-1}@anchor{1436}
@subsection Python 3.5.5 release candidate 1


@emph{Release date: 2018-01-23}

@menu
* Security: Security<33>. 
* Core and Builtins: Core and Builtins<54>. 
* Library: Library<62>. 

@end menu

@node Security<33>,Core and Builtins<54>,,Python 3 5 5 release candidate 1
@anchor{whatsnew/changelog id471}@anchor{1437}
@subsubsection Security



@itemize -

@item 
bpo-32551@footnote{https://bugs.python.org/issue32551}: The @code{sys.path[0]} initialization change for bpo-29139@footnote{https://bugs.python.org/issue29139} caused
a regression by revealing an inconsistency in how sys.path is initialized
when executing @code{__main__} from a zipfile, directory, or other import
location. This is considered a potential security issue, as it may lead to
privileged processes unexpectedly loading code from user controlled
directories in situations where that was not previously the case. The
interpreter now consistently avoids ever adding the import location’s
parent directory to @code{sys.path}, and ensures no other @code{sys.path}
entries are inadvertently modified when inserting the import location
named on the command line. (Originally reported as bpo-29723@footnote{https://bugs.python.org/issue29723} against
Python 3.6rc1, but it was missed at the time that the then upcoming Python
3.5.4 release would also be affected)

@item 
bpo-30657@footnote{https://bugs.python.org/issue30657}: Fixed possible integer overflow in PyBytes_DecodeEscape,
CVE-2017-1000158. Original patch by Jay Bosamiya; rebased to Python 3 by
Miro Hrončok.

@item 
bpo-30947@footnote{https://bugs.python.org/issue30947}: Upgrade libexpat embedded copy from version 2.2.1 to 2.2.3 to
get security fixes.
@end itemize

@node Core and Builtins<54>,Library<62>,Security<33>,Python 3 5 5 release candidate 1
@anchor{whatsnew/changelog id472}@anchor{1438}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-31095@footnote{https://bugs.python.org/issue31095}: Fix potential crash during GC caused by @code{tp_dealloc} which
doesn’t call @code{PyObject_GC_UnTrack()}.
@end itemize

@node Library<62>,,Core and Builtins<54>,Python 3 5 5 release candidate 1
@anchor{whatsnew/changelog id473}@anchor{1439}
@subsubsection Library



@itemize -

@item 
bpo-32072@footnote{https://bugs.python.org/issue32072}: Fixed issues with binary plists: Fixed saving bytearrays.
Identical objects will be saved only once. Equal references will be load
as identical objects. Added support for saving and loading recursive data
structures.

@item 
bpo-31170@footnote{https://bugs.python.org/issue31170}: expat: Update libexpat from 2.2.3 to 2.2.4. Fix copying of
partial characters for UTF-8 input (libexpat bug 115):
@indicateurl{https://github.com/libexpat/libexpat/issues/115}
@end itemize

@node Python 3 5 4 final,Python 3 5 4 release candidate 1,Python 3 5 5 release candidate 1,Changelog
@anchor{whatsnew/changelog python-3-5-4-final}@anchor{143a}
@subsection Python 3.5.4 final


@emph{Release date: 2017-08-07}

@menu
* Library: Library<63>. 

@end menu

@node Library<63>,,,Python 3 5 4 final
@anchor{whatsnew/changelog id474}@anchor{143b}
@subsubsection Library



@itemize -

@item 
bpo-30119@footnote{https://bugs.python.org/issue30119}: ftplib.FTP.putline() now throws ValueError on commands that
contains CR or LF. Patch by Dong-hee Na.
@end itemize

@node Python 3 5 4 release candidate 1,Python 3 5 3 final,Python 3 5 4 final,Changelog
@anchor{whatsnew/changelog python-3-5-4-release-candidate-1}@anchor{143c}
@subsection Python 3.5.4 release candidate 1


@emph{Release date: 2017-07-23}

@menu
* Security: Security<34>. 
* Core and Builtins: Core and Builtins<55>. 
* Library: Library<64>. 
* Documentation: Documentation<49>. 
* Tests: Tests<49>. 
* Build: Build<53>. 
* Windows: Windows<47>. 
* C API: C API<44>. 

@end menu

@node Security<34>,Core and Builtins<55>,,Python 3 5 4 release candidate 1
@anchor{whatsnew/changelog id475}@anchor{143d}
@subsubsection Security



@itemize -

@item 
bpo-30730@footnote{https://bugs.python.org/issue30730}: Prevent environment variables injection in subprocess on
Windows.  Prevent passing other environment variables and command
arguments.

@item 
bpo-30694@footnote{https://bugs.python.org/issue30694}: Upgrade expat copy from 2.2.0 to 2.2.1 to get fixes of multiple
security vulnerabilities including: CVE-2017-9233 (External entity
infinite loop DoS), CVE-2016-9063 (Integer overflow, re-fix),
CVE-2016-0718 (Fix regression bugs from 2.2.0’s fix to CVE-2016-0718) and
CVE-2012-0876 (Counter hash flooding with SipHash). Note: the
CVE-2016-5300 (Use os-specific entropy sources like getrandom) doesn’t
impact Python, since Python already gets entropy from the OS to set the
expat secret using @code{XML_SetHashSalt()}.

@item 
bpo-30500@footnote{https://bugs.python.org/issue30500}: Fix urllib.parse.splithost() to correctly parse fragments. For
example, @code{splithost('//127.0.0.1#@@evil.com/')} now correctly returns the
@code{127.0.0.1} host, instead of treating @code{@@evil.com} as the host in an
authentication (@code{login@@host}).

@item 
bpo-29591@footnote{https://bugs.python.org/issue29591}: Update expat copy from 2.1.1 to 2.2.0 to get fixes of
CVE-2016-0718 and CVE-2016-4472. See
@indicateurl{https://sourceforge.net/p/expat/bugs/537/} for more information.
@end itemize

@node Core and Builtins<55>,Library<64>,Security<34>,Python 3 5 4 release candidate 1
@anchor{whatsnew/changelog id476}@anchor{143e}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-30876@footnote{https://bugs.python.org/issue30876}: Relative import from unloaded package now reimports the package
instead of failing with SystemError.  Relative import from non-package now
fails with ImportError rather than SystemError.

@item 
bpo-30765@footnote{https://bugs.python.org/issue30765}: Avoid blocking in pthread_mutex_lock() when
PyThread_acquire_lock() is asked not to block.

@item 
bpo-27945@footnote{https://bugs.python.org/issue27945}: Fixed various segfaults with dict when input collections are
mutated during searching, inserting or comparing.  Based on patches by
Duane Griffin and Tim Mitchell.

@item 
bpo-25794@footnote{https://bugs.python.org/issue25794}: Fixed type.__setattr__() and type.__delattr__() for
non-interned attribute names.  Based on patch by Eryk Sun.

@item 
bpo-29935@footnote{https://bugs.python.org/issue29935}: Fixed error messages in the index() method of tuple, list and
deque when pass indices of wrong type.

@item 
bpo-28876@footnote{https://bugs.python.org/issue28876}: @code{bool(range)} works even if @code{len(range)} raises
@ref{347,,OverflowError}.

@item 
bpo-29600@footnote{https://bugs.python.org/issue29600}: Fix wrapping coroutine return values in StopIteration.

@item 
bpo-29537@footnote{https://bugs.python.org/issue29537}: Restore runtime compatibility with bytecode files generated by
CPython 3.5.0 to 3.5.2, and adjust the eval loop to avoid the problems
that could be caused by the malformed variant of the
BUILD_MAP_UNPACK_WITH_CALL opcode that they may contain. Patch by Petr
Viktorin, Serhiy Storchaka, and Nick Coghlan.

@item 
bpo-28598@footnote{https://bugs.python.org/issue28598}: Support __rmod__ for subclasses of str being called before
str.__mod__. Patch by Martijn Pieters.

@item 
bpo-29602@footnote{https://bugs.python.org/issue29602}: Fix incorrect handling of signed zeros in complex constructor
for complex subclasses and for inputs having a __complex__ method. Patch
by Serhiy Storchaka.

@item 
bpo-29347@footnote{https://bugs.python.org/issue29347}: Fixed possibly dereferencing undefined pointers when creating
weakref objects.

@item 
bpo-29438@footnote{https://bugs.python.org/issue29438}: Fixed use-after-free problem in key sharing dict.

@item 
bpo-29319@footnote{https://bugs.python.org/issue29319}: Prevent RunMainFromImporter overwriting sys.path[0].

@item 
bpo-29337@footnote{https://bugs.python.org/issue29337}: Fixed possible BytesWarning when compare the code objects.
Warnings could be emitted at compile time.

@item 
bpo-29478@footnote{https://bugs.python.org/issue29478}: If max_line_length=None is specified while using the Compat32
policy, it is no longer ignored.  Patch by Mircea Cosbuc.
@end itemize

@node Library<64>,Documentation<49>,Core and Builtins<55>,Python 3 5 4 release candidate 1
@anchor{whatsnew/changelog id477}@anchor{143f}
@subsubsection Library



@itemize -

@item 
bpo-29403@footnote{https://bugs.python.org/issue29403}: Fix @code{unittest.mock}’s autospec to not fail on method-bound
builtin functions.  Patch by Aaron Gallagher.

@item 
bpo-30961@footnote{https://bugs.python.org/issue30961}: Fix decrementing a borrowed reference in tracemalloc.

@item 
bpo-30886@footnote{https://bugs.python.org/issue30886}: Fix multiprocessing.Queue.join_thread(): it now waits until the
thread completes, even if the thread was started by the same process which
created the queue.

@item 
bpo-29854@footnote{https://bugs.python.org/issue29854}: Fix segfault in readline when using readline’s history-size
option.  Patch by Nir Soffer.

@item 
bpo-30807@footnote{https://bugs.python.org/issue30807}: signal.setitimer() may disable the timer when passed a tiny
value. Tiny values (such as 1e-6) are valid non-zero values for
setitimer(), which is specified as taking microsecond-resolution
intervals. However, on some platform, our conversion routine could convert
1e-6 into a zero interval, therefore disabling the timer instead of
(re-)scheduling it.

@item 
bpo-30441@footnote{https://bugs.python.org/issue30441}: Fix bug when modifying os.environ while iterating over it

@item 
bpo-30532@footnote{https://bugs.python.org/issue30532}: Fix email header value parser dropping folding white space in
certain cases.

@item 
bpo-29169@footnote{https://bugs.python.org/issue29169}: Update zlib to 1.2.11.

@item 
bpo-30879@footnote{https://bugs.python.org/issue30879}: os.listdir() and os.scandir() now emit bytes names when called
with bytes-like argument.

@item 
bpo-30746@footnote{https://bugs.python.org/issue30746}: Prohibited the ‘=’ character in environment variable names in
@code{os.putenv()} and @code{os.spawn*()}.

@item 
bpo-29755@footnote{https://bugs.python.org/issue29755}: Fixed the lgettext() family of functions in the gettext module.
They now always return bytes.

@item 
bpo-30645@footnote{https://bugs.python.org/issue30645}: Fix path calculation in imp.load_package(), fixing it for cases
when a package is only shipped with bytecodes. Patch by Alexandru
Ardelean.

@item 
bpo-23890@footnote{https://bugs.python.org/issue23890}: unittest.TestCase.assertRaises() now manually breaks a
reference cycle to not keep objects alive longer than expected.

@item 
bpo-30149@footnote{https://bugs.python.org/issue30149}: inspect.signature() now supports callables with
variable-argument parameters wrapped with partialmethod. Patch by Dong-hee
Na.

@item 
bpo-29931@footnote{https://bugs.python.org/issue29931}: Fixed comparison check for ipaddress.ip_interface objects.
Patch by Sanjay Sundaresan.

@item 
bpo-24484@footnote{https://bugs.python.org/issue24484}: Avoid race condition in multiprocessing cleanup.

@item 
bpo-28994@footnote{https://bugs.python.org/issue28994}: The traceback no longer displayed for SystemExit raised in a
callback registered by atexit.

@item 
bpo-30508@footnote{https://bugs.python.org/issue30508}: Don’t log exceptions if Task/Future “cancel()” method was
called.

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Updates to typing module: Add generic AsyncContextManager, add
support for ContextManager on all versions. Original PRs by Jelle Zijlstra
and Ivan Levkivskyi

@item 
bpo-29870@footnote{https://bugs.python.org/issue29870}: Fix ssl sockets leaks when connection is aborted in asyncio/ssl
implementation. Patch by Michaël Sghaïer.

@item 
bpo-29743@footnote{https://bugs.python.org/issue29743}: Closing transport during handshake process leaks open socket.
Patch by Nikolay Kim

@item 
bpo-27585@footnote{https://bugs.python.org/issue27585}: Fix waiter cancellation in asyncio.Lock. Patch by Mathieu
Sornay.

@item 
bpo-30418@footnote{https://bugs.python.org/issue30418}: On Windows, subprocess.Popen.communicate() now also ignore
EINVAL on stdin.write() if the child process is still running but closed
the pipe.

@item 
bpo-30378@footnote{https://bugs.python.org/issue30378}: Fix the problem that logging.handlers.SysLogHandler cannot
handle IPv6 addresses.

@item 
bpo-29960@footnote{https://bugs.python.org/issue29960}: Preserve generator state when _random.Random.setstate() raises
an exception. Patch by Bryan Olson.

@item 
bpo-30414@footnote{https://bugs.python.org/issue30414}: multiprocessing.Queue._feed background running thread do not
break from main loop on exception.

@item 
bpo-30003@footnote{https://bugs.python.org/issue30003}: Fix handling escape characters in HZ codec.  Based on patch by
Ma Lin.

@item 
bpo-30301@footnote{https://bugs.python.org/issue30301}: Fix AttributeError when using SimpleQueue.empty() under @emph{spawn}
and @emph{forkserver} start methods.

@item 
bpo-30329@footnote{https://bugs.python.org/issue30329}: imaplib and poplib now catch the Windows socket WSAEINVAL error
(code 10022) on shutdown(SHUT_RDWR): An invalid operation was attempted.
This error occurs sometimes on SSL connections.

@item 
bpo-30375@footnote{https://bugs.python.org/issue30375}: Warnings emitted when compile a regular expression now always
point to the line in the user code.  Previously they could point into
inners of the re module if emitted from inside of groups or conditionals.

@item 
bpo-30048@footnote{https://bugs.python.org/issue30048}: Fixed @code{Task.cancel()} can be ignored when the task is running
coroutine and the coroutine returned without any more @code{await}.

@item 
bpo-29990@footnote{https://bugs.python.org/issue29990}: Fix range checking in GB18030 decoder.  Original patch by Ma
Lin.

@item 
bpo-26293@footnote{https://bugs.python.org/issue26293}: Change resulted because of zipfile breakage. (See also:
bpo-29094@footnote{https://bugs.python.org/issue29094})

@item 
bpo-30243@footnote{https://bugs.python.org/issue30243}: Removed the __init__ methods of _json’s scanner and encoder.
Misusing them could cause memory leaks or crashes.  Now scanner and
encoder objects are completely initialized in the __new__ methods.

@item 
bpo-30185@footnote{https://bugs.python.org/issue30185}: Avoid KeyboardInterrupt tracebacks in forkserver helper process
when Ctrl-C is received.

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Various updates to typing module: add typing.NoReturn type, use
WrapperDescriptorType, minor bug-fixes.  Original PRs by Jim
Fasarakis-Hilliard and Ivan Levkivskyi.

@item 
bpo-30205@footnote{https://bugs.python.org/issue30205}: Fix getsockname() for unbound AF_UNIX sockets on Linux.

@item 
bpo-30070@footnote{https://bugs.python.org/issue30070}: Fixed leaks and crashes in errors handling in the parser
module.

@item 
bpo-30061@footnote{https://bugs.python.org/issue30061}: Fixed crashes in IOBase methods __next__() and readlines() when
readline() or __next__() respectively return non-sizeable object. Fixed
possible other errors caused by not checking results of PyObject_Size(),
PySequence_Size(), or PyMapping_Size().

@item 
bpo-30068@footnote{https://bugs.python.org/issue30068}: _io._IOBase.readlines will check if it’s closed first when hint
is present.

@item 
bpo-29694@footnote{https://bugs.python.org/issue29694}: Fixed race condition in pathlib mkdir with flags parents=True.
Patch by Armin Rigo.

@item 
bpo-29692@footnote{https://bugs.python.org/issue29692}: Fixed arbitrary unchaining of RuntimeError exceptions in
contextlib.contextmanager.  Patch by Siddharth Velankar.

@item 
bpo-29998@footnote{https://bugs.python.org/issue29998}: Pickling and copying ImportError now preserves name and path
attributes.

@item 
bpo-29942@footnote{https://bugs.python.org/issue29942}: Fix a crash in itertools.chain.from_iterable when encountering
long runs of empty iterables.

@item 
bpo-27863@footnote{https://bugs.python.org/issue27863}: Fixed multiple crashes in ElementTree caused by race conditions
and wrong types.

@item 
bpo-28699@footnote{https://bugs.python.org/issue28699}: Fixed a bug in pools in multiprocessing.pool that raising an
exception at the very first of an iterable may swallow the exception or
make the program hang. Patch by Davin Potts and Xiang Zhang.

@item 
bpo-25803@footnote{https://bugs.python.org/issue25803}: Avoid incorrect errors raised by Path.mkdir(exist_ok=True) when
the OS gives priority to errors such as EACCES over EEXIST.

@item 
bpo-29861@footnote{https://bugs.python.org/issue29861}: Release references to tasks, their arguments and their results
as soon as they are finished in multiprocessing.Pool.

@item 
bpo-29884@footnote{https://bugs.python.org/issue29884}: faulthandler: Restore the old sigaltstack during teardown.
Patch by Christophe Zeitouny.

@item 
bpo-25455@footnote{https://bugs.python.org/issue25455}: Fixed crashes in repr of recursive buffered file-like objects.

@item 
bpo-29800@footnote{https://bugs.python.org/issue29800}: Fix crashes in partial.__repr__ if the keys of partial.keywords
are not strings.  Patch by Michael Seifert.

@item 
bpo-29742@footnote{https://bugs.python.org/issue29742}: get_extra_info() raises exception if get called on closed ssl
transport. Patch by Nikolay Kim.

@item 
bpo-8256@footnote{https://bugs.python.org/issue8256}: Fixed possible failing or crashing input() if attributes
“encoding” or “errors” of sys.stdin or sys.stdout are not set or are not
strings.

@item 
bpo-28298@footnote{https://bugs.python.org/issue28298}: Fix a bug that prevented array ‘Q’, ‘L’ and ‘I’ from accepting
big intables (objects that have __int__) as elements.  Patch by Oren
Milman.

@item 
bpo-29615@footnote{https://bugs.python.org/issue29615}: SimpleXMLRPCDispatcher no longer chains KeyError (or any other
exception) to exception(s) raised in the dispatched methods. Patch by Petr
Motejlek.

@item 
bpo-29704@footnote{https://bugs.python.org/issue29704}: asyncio.subprocess.SubprocessStreamProtocol no longer closes
before all pipes are closed.

@item 
bpo-29703@footnote{https://bugs.python.org/issue29703}: Fix asyncio to support instantiation of new event loops in
child processes.

@item 
bpo-29376@footnote{https://bugs.python.org/issue29376}: Fix assertion error in threading._DummyThread.is_alive().

@item 
bpo-29110@footnote{https://bugs.python.org/issue29110}: Fix file object leak in aifc.open() when file is given as a
filesystem path and is not in valid AIFF format. Patch by Anthony Zhang.

@item 
bpo-28961@footnote{https://bugs.python.org/issue28961}: Fix unittest.mock._Call helper: don’t ignore the name parameter
anymore. Patch written by Jiajun Huang.

@item 
bpo-29532@footnote{https://bugs.python.org/issue29532}: Altering a kwarg dictionary passed to functools.partial() no
longer affects a partial object after creation.

@item 
bpo-28556@footnote{https://bugs.python.org/issue28556}: Various updates to typing module: typing.Counter,
typing.ChainMap, improved ABC caching, etc. Original PRs by Jelle
Zijlstra, Ivan Levkivskyi, Manuel Krebber, and Łukasz Langa.

@item 
bpo-29100@footnote{https://bugs.python.org/issue29100}: Fix datetime.fromtimestamp() regression introduced in Python
3.6.0: check minimum and maximum years.

@item 
bpo-29519@footnote{https://bugs.python.org/issue29519}: Fix weakref spewing exceptions during interpreter shutdown when
used with a rare combination of multiprocessing and custom codecs.

@item 
bpo-29416@footnote{https://bugs.python.org/issue29416}: Prevent infinite loop in pathlib.Path.mkdir

@item 
bpo-29444@footnote{https://bugs.python.org/issue29444}: Fixed out-of-bounds buffer access in the group() method of the
match object. Based on patch by WGH.

@item 
bpo-29335@footnote{https://bugs.python.org/issue29335}: Fix subprocess.Popen.wait() when the child process has exited
to a stopped instead of terminated state (ex: when under ptrace).

@item 
bpo-29290@footnote{https://bugs.python.org/issue29290}: Fix a regression in argparse that help messages would wrap at
non-breaking spaces.

@item 
bpo-28735@footnote{https://bugs.python.org/issue28735}: Fixed the comparison of mock.MagickMock with mock.ANY.

@item 
bpo-29011@footnote{https://bugs.python.org/issue29011}: Fix an important omission by adding Deque to the typing module.

@item 
bpo-29219@footnote{https://bugs.python.org/issue29219}: Fixed infinite recursion in the repr of uninitialized
ctypes.CDLL instances.

@item 
bpo-28969@footnote{https://bugs.python.org/issue28969}: Fixed race condition in C implementation of
functools.lru_cache. KeyError could be raised when cached function with
full cache was simultaneously called from different threads with the same
uncached arguments.

@item 
bpo-29142@footnote{https://bugs.python.org/issue29142}: In urllib.request, suffixes in no_proxy environment variable
with leading dots could match related hostnames again (e.g. .b.c matches
a.b.c). Patch by Milan Oberkirch.
@end itemize

@node Documentation<49>,Tests<49>,Library<64>,Python 3 5 4 release candidate 1
@anchor{whatsnew/changelog id478}@anchor{1440}
@subsubsection Documentation



@itemize -

@item 
bpo-30176@footnote{https://bugs.python.org/issue30176}: Add missing attribute related constants in curses
documentation.

@item 
bpo-26985@footnote{https://bugs.python.org/issue26985}: Add missing info of code object in inspect documentation.

@item 
bpo-28929@footnote{https://bugs.python.org/issue28929}: Link the documentation to its source file on GitHub.

@item 
bpo-25008@footnote{https://bugs.python.org/issue25008}: Document smtpd.py as effectively deprecated and add a pointer
to aiosmtpd, a third-party asyncio-based replacement.

@item 
bpo-26355@footnote{https://bugs.python.org/issue26355}: Add canonical header link on each page to corresponding major
version of the documentation. Patch by Matthias Bussonnier.

@item 
bpo-29349@footnote{https://bugs.python.org/issue29349}: Fix Python 2 syntax in code for building the documentation.
@end itemize

@node Tests<49>,Build<53>,Documentation<49>,Python 3 5 4 release candidate 1
@anchor{whatsnew/changelog id479}@anchor{1441}
@subsubsection Tests



@itemize -

@item 
bpo-30822@footnote{https://bugs.python.org/issue30822}: Fix regrtest command line parser to allow passing -u
extralargefile to run test_zipfile64.

@item 
bpo-30383@footnote{https://bugs.python.org/issue30383}: regrtest: Enhance regrtest and backport features from the
master branch. Add options: –coverage, –testdir, –list-tests (list test
files, don’t run them), –list-cases (list test identifiers, don’t run
them, bpo-30523@footnote{https://bugs.python.org/issue30523}), –matchfile (load a list of test filters from a
text file, bpo-30540@footnote{https://bugs.python.org/issue30540}), –slowest (alias to –slow). Enhance output:
add timestamp, test result, currently running tests, “Tests result: xxx”
summary with total duration, etc. Fix reference leak hunting in regrtest,
–huntrleaks: regrtest now warms up caches, create explicitly all internal
singletons which are created on demand to prevent false positives when
checking for reference leaks. (bpo-30675@footnote{https://bugs.python.org/issue30675}).

@item 
bpo-30357@footnote{https://bugs.python.org/issue30357}: test_thread: setUp() now uses support.threading_setup() and
support.threading_cleanup() to wait until threads complete to avoid random
side effects on following tests. Initial patch written by Grzegorz
Grzywacz.

@item 
bpo-28087@footnote{https://bugs.python.org/issue28087}: Skip test_asyncore and test_eintr poll failures on macOS. Skip
some tests of select.poll when running on macOS due to unresolved issues
with the underlying system poll function on some macOS versions.

@item 
bpo-30197@footnote{https://bugs.python.org/issue30197}: Enhanced functions swap_attr() and swap_item() in the
test.support module. They now work when delete replaced attribute or item
inside the with statement.  The old value of the attribute or item (or
None if it doesn’t exist) now will be assigned to the target of the “as”
clause, if there is one.

@item 
bpo-29571@footnote{https://bugs.python.org/issue29571}: to match the behaviour of the @code{re.LOCALE} flag,
test_re.test_locale_flag now uses @code{locale.getpreferredencoding(False)}
to determine the candidate encoding for the test regex (allowing it to
correctly skip the test when the default locale encoding is a multi-byte
encoding)
@end itemize

@node Build<53>,Windows<47>,Tests<49>,Python 3 5 4 release candidate 1
@anchor{whatsnew/changelog id480}@anchor{1442}
@subsubsection Build



@itemize -

@item 
bpo-29243@footnote{https://bugs.python.org/issue29243}: Prevent unnecessary rebuilding of Python during @code{make test},
@code{make install} and some other make targets when configured with
@code{--enable-optimizations}.

@item 
bpo-23404@footnote{https://bugs.python.org/issue23404}: Don’t regenerate generated files based on file modification
time anymore: the action is now explicit. Replace @code{make touch} with
@code{make regen-all}.

@item 
bpo-29643@footnote{https://bugs.python.org/issue29643}: Fix @code{--enable-optimization} didn’t work.
@end itemize

@node Windows<47>,C API<44>,Build<53>,Python 3 5 4 release candidate 1
@anchor{whatsnew/changelog id481}@anchor{1443}
@subsubsection Windows



@itemize -

@item 
bpo-30687@footnote{https://bugs.python.org/issue30687}: Locate msbuild.exe on Windows when building rather than
vcvarsall.bat

@item 
bpo-29392@footnote{https://bugs.python.org/issue29392}: Prevent crash when passing invalid arguments into msvcrt
module.
@end itemize

@node C API<44>,,Windows<47>,Python 3 5 4 release candidate 1
@anchor{whatsnew/changelog id482}@anchor{1444}
@subsubsection C API



@itemize -

@item 
bpo-27867@footnote{https://bugs.python.org/issue27867}: Function PySlice_GetIndicesEx() is replaced with a macro if
Py_LIMITED_API is set to the value between 0x03050400 and 0x03060000 (not
including) or 0x03060100 or higher.

@item 
bpo-29083@footnote{https://bugs.python.org/issue29083}: Fixed the declaration of some public API functions.
PyArg_VaParse() and PyArg_VaParseTupleAndKeywords() were not available in
limited API. PyArg_ValidateKeywordArguments(), PyArg_UnpackTuple() and
Py_BuildValue() were not available in limited API of version < 3.3 when
PY_SSIZE_T_CLEAN is defined.
@end itemize

@node Python 3 5 3 final,Python 3 5 3 release candidate 1,Python 3 5 4 release candidate 1,Changelog
@anchor{whatsnew/changelog python-3-5-3-final}@anchor{1445}
@subsection Python 3.5.3 final


@emph{Release date: 2017-01-17}

There were no code changes between 3.5.3rc1 and 3.5.3 final.

@node Python 3 5 3 release candidate 1,Python 3 5 2 final,Python 3 5 3 final,Changelog
@anchor{whatsnew/changelog python-3-5-3-release-candidate-1}@anchor{1446}
@subsection Python 3.5.3 release candidate 1


@emph{Release date: 2017-01-02}

@menu
* Core and Builtins: Core and Builtins<56>. 
* Library: Library<65>. 
* Security: Security<35>. 
* Library: Library<66>. 
* Security: Security<36>. 
* Library: Library<67>. 
* IDLE: IDLE<44>. 
* C API: C API<45>. 
* Documentation: Documentation<50>. 
* Tests: Tests<50>. 
* Tools/Demos: Tools/Demos<29>. 
* Windows: Windows<48>. 
* Build: Build<54>. 

@end menu

@node Core and Builtins<56>,Library<65>,,Python 3 5 3 release candidate 1
@anchor{whatsnew/changelog id483}@anchor{1447}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-29073@footnote{https://bugs.python.org/issue29073}: bytearray formatting no longer truncates on first null byte.

@item 
bpo-28932@footnote{https://bugs.python.org/issue28932}: Do not include <sys/random.h> if it does not exist.

@item 
bpo-28147@footnote{https://bugs.python.org/issue28147}: Fix a memory leak in split-table dictionaries: setattr() must
not convert combined table into split table.

@item 
bpo-25677@footnote{https://bugs.python.org/issue25677}: Correct the positioning of the syntax error caret for indented
blocks. Based on patch by Michael Layzell.

@item 
bpo-29000@footnote{https://bugs.python.org/issue29000}: Fixed bytes formatting of octals with zero padding in alternate
form.

@item 
bpo-28512@footnote{https://bugs.python.org/issue28512}: Fixed setting the offset attribute of SyntaxError by
PyErr_SyntaxLocationEx() and PyErr_SyntaxLocationObject().

@item 
bpo-28991@footnote{https://bugs.python.org/issue28991}: functools.lru_cache() was susceptible to an obscure reentrancy
bug caused by a monkey-patched len() function.

@item 
bpo-28648@footnote{https://bugs.python.org/issue28648}: Fixed crash in Py_DecodeLocale() in debug build on Mac OS X
when decode astral characters.  Patch by Xiang Zhang.

@item 
bpo-19398@footnote{https://bugs.python.org/issue19398}: Extra slash no longer added to sys.path components in case of
empty compile-time PYTHONPATH components.

@item 
bpo-28426@footnote{https://bugs.python.org/issue28426}: Fixed potential crash in PyUnicode_AsDecodedObject() in debug
build.

@item 
bpo-23782@footnote{https://bugs.python.org/issue23782}: Fixed possible memory leak in _PyTraceback_Add() and exception
loss in PyTraceBack_Here().

@item 
bpo-28379@footnote{https://bugs.python.org/issue28379}: Added sanity checks and tests for PyUnicode_CopyCharacters().
Patch by Xiang Zhang.

@item 
bpo-28376@footnote{https://bugs.python.org/issue28376}: The type of long range iterator is now registered as Iterator.
Patch by Oren Milman.

@item 
bpo-28376@footnote{https://bugs.python.org/issue28376}: The constructor of range_iterator now checks that step is not
0. Patch by Oren Milman.

@item 
bpo-26906@footnote{https://bugs.python.org/issue26906}: Resolving special methods of uninitialized type now causes
implicit initialization of the type instead of a fail.

@item 
bpo-18287@footnote{https://bugs.python.org/issue18287}: PyType_Ready() now checks that tp_name is not NULL. Original
patch by Niklas Koep.

@item 
bpo-24098@footnote{https://bugs.python.org/issue24098}: Fixed possible crash when AST is changed in process of
compiling it.

@item 
bpo-28350@footnote{https://bugs.python.org/issue28350}: String constants with null character no longer interned.

@item 
bpo-26617@footnote{https://bugs.python.org/issue26617}: Fix crash when GC runs during weakref callbacks.

@item 
bpo-27942@footnote{https://bugs.python.org/issue27942}: String constants now interned recursively in tuples and
frozensets.

@item 
bpo-21578@footnote{https://bugs.python.org/issue21578}: Fixed misleading error message when ImportError called with
invalid keyword args.

@item 
bpo-28203@footnote{https://bugs.python.org/issue28203}: Fix incorrect type in error message from @code{complex(1.0,
@{2:3@})}. Patch by Soumya Sharma.

@item 
bpo-27955@footnote{https://bugs.python.org/issue27955}: Fallback on reading /dev/urandom device when the getrandom()
syscall fails with EPERM, for example when blocked by SECCOMP.

@item 
bpo-28131@footnote{https://bugs.python.org/issue28131}: Fix a regression in zipimport’s compile_source().  zipimport
should use the same optimization level as the interpreter.

@item 
bpo-25221@footnote{https://bugs.python.org/issue25221}: Fix corrupted result from PyLong_FromLong(0) when Python is
compiled with NSMALLPOSINTS = 0.

@item 
bpo-25758@footnote{https://bugs.python.org/issue25758}: Prevents zipimport from unnecessarily encoding a filename
(patch by Eryk Sun)

@item 
bpo-28189@footnote{https://bugs.python.org/issue28189}: dictitems_contains no longer swallows compare errors. (Patch by
Xiang Zhang)

@item 
bpo-27812@footnote{https://bugs.python.org/issue27812}: Properly clear out a generator’s frame’s backreference to the
generator to prevent crashes in frame.clear().

@item 
bpo-27811@footnote{https://bugs.python.org/issue27811}: Fix a crash when a coroutine that has not been awaited is
finalized with warnings-as-errors enabled.

@item 
bpo-27587@footnote{https://bugs.python.org/issue27587}: Fix another issue found by PVS-Studio: Null pointer check after
use of ‘def’ in _PyState_AddModule(). Initial patch by Christian Heimes.

@item 
bpo-26020@footnote{https://bugs.python.org/issue26020}: set literal evaluation order did not match documented
behaviour.

@item 
bpo-27782@footnote{https://bugs.python.org/issue27782}: Multi-phase extension module import now correctly allows the
@code{m_methods} field to be used to add module level functions to instances
of non-module types returned from @code{Py_create_mod}. Patch by Xiang Zhang.

@item 
bpo-27936@footnote{https://bugs.python.org/issue27936}: The round() function accepted a second None argument for some
types but not for others.  Fixed the inconsistency by accepting None for
all numeric types.

@item 
bpo-27487@footnote{https://bugs.python.org/issue27487}: Warn if a submodule argument to “python -m” or
runpy.run_module() is found in sys.modules after parent packages are
imported, but before the submodule is executed.

@item 
bpo-27558@footnote{https://bugs.python.org/issue27558}: Fix a SystemError in the implementation of “raise” statement.
In a brand new thread, raise a RuntimeError since there is no active
exception to reraise. Patch written by Xiang Zhang.

@item 
bpo-27419@footnote{https://bugs.python.org/issue27419}: Standard __import__() no longer look up “__import__” in globals
or builtins for importing submodules or “from import”.  Fixed handling an
error of non-string package name.

@item 
bpo-27083@footnote{https://bugs.python.org/issue27083}: Respect the PYTHONCASEOK environment variable under Windows.

@item 
bpo-27514@footnote{https://bugs.python.org/issue27514}: Make having too many statically nested blocks a SyntaxError
instead of SystemError.

@item 
bpo-27473@footnote{https://bugs.python.org/issue27473}: Fixed possible integer overflow in bytes and bytearray
concatenations. Patch by Xiang Zhang.

@item 
bpo-27507@footnote{https://bugs.python.org/issue27507}: Add integer overflow check in bytearray.extend().  Patch by
Xiang Zhang.

@item 
bpo-27581@footnote{https://bugs.python.org/issue27581}: Don’t rely on wrapping for overflow check in
PySequence_Tuple().  Patch by Xiang Zhang.

@item 
bpo-27443@footnote{https://bugs.python.org/issue27443}: __length_hint__() of bytearray iterators no longer return a
negative integer for a resized bytearray.

@item 
bpo-27942@footnote{https://bugs.python.org/issue27942}: Fix memory leak in codeobject.c
@end itemize

@node Library<65>,Security<35>,Core and Builtins<56>,Python 3 5 3 release candidate 1
@anchor{whatsnew/changelog id484}@anchor{1448}
@subsubsection Library



@itemize -

@item 
bpo-15812@footnote{https://bugs.python.org/issue15812}: inspect.getframeinfo() now correctly shows the first line of a
context. Patch by Sam Breese.

@item 
bpo-29094@footnote{https://bugs.python.org/issue29094}: Offsets in a ZIP file created with extern file object and modes
“w” and “x” now are relative to the start of the file.

@item 
bpo-13051@footnote{https://bugs.python.org/issue13051}: Fixed recursion errors in large or resized
curses.textpad.Textbox.  Based on patch by Tycho Andersen.

@item 
bpo-29119@footnote{https://bugs.python.org/issue29119}: Fix weakrefs in the pure python version of
collections.OrderedDict move_to_end() method. Contributed by Andra
Bogildea.

@item 
bpo-9770@footnote{https://bugs.python.org/issue9770}: curses.ascii predicates now work correctly with negative
integers.

@item 
bpo-28427@footnote{https://bugs.python.org/issue28427}: old keys should not remove new values from WeakValueDictionary
when collecting from another thread.

@item 
bpo-28923@footnote{https://bugs.python.org/issue28923}: Remove editor artifacts from Tix.py.

@item 
bpo-28871@footnote{https://bugs.python.org/issue28871}: Fixed a crash when deallocate deep ElementTree.

@item 
bpo-19542@footnote{https://bugs.python.org/issue19542}: Fix bugs in WeakValueDictionary.setdefault() and
WeakValueDictionary.pop() when a GC collection happens in another thread.

@item 
bpo-20191@footnote{https://bugs.python.org/issue20191}: Fixed a crash in resource.prlimit() when pass a sequence that
doesn’t own its elements as limits.

@item 
bpo-28779@footnote{https://bugs.python.org/issue28779}: multiprocessing.set_forkserver_preload() would crash the
forkserver process if a preloaded module instantiated some multiprocessing
objects such as locks.

@item 
bpo-28847@footnote{https://bugs.python.org/issue28847}: dbm.dumb now supports reading read-only files and no longer
writes the index file when it is not changed.

@item 
bpo-25659@footnote{https://bugs.python.org/issue25659}: In ctypes, prevent a crash calling the from_buffer() and
from_buffer_copy() methods on abstract classes like Array.

@item 
bpo-28732@footnote{https://bugs.python.org/issue28732}: Fix crash in os.spawnv() with no elements in args

@item 
bpo-28485@footnote{https://bugs.python.org/issue28485}: Always raise ValueError for negative
compileall.compile_dir(workers=…) parameter, even when multithreading is
unavailable.

@item 
bpo-28387@footnote{https://bugs.python.org/issue28387}: Fixed possible crash in _io.TextIOWrapper deallocator when the
garbage collector is invoked in other thread.  Based on patch by Sebastian
Cufre.

@item 
bpo-27517@footnote{https://bugs.python.org/issue27517}: LZMA compressor and decompressor no longer raise exceptions if
given empty data twice.  Patch by Benjamin Fogle.

@item 
bpo-28549@footnote{https://bugs.python.org/issue28549}: Fixed segfault in curses’s addch() with ncurses6.

@item 
bpo-28449@footnote{https://bugs.python.org/issue28449}: tarfile.open() with mode “r” or “r:” now tries to open a tar
file with compression before trying to open it without compression.
Otherwise it had 50% chance failed with ignore_zeros=True.

@item 
bpo-23262@footnote{https://bugs.python.org/issue23262}: The webbrowser module now supports Firefox 36+ and derived
browsers.  Based on patch by Oleg Broytman.

@item 
bpo-27939@footnote{https://bugs.python.org/issue27939}: Fixed bugs in tkinter.ttk.LabeledScale and tkinter.Scale caused
by representing the scale as float value internally in Tk.  tkinter.IntVar
now works if float value is set to underlying Tk variable.

@item 
bpo-28255@footnote{https://bugs.python.org/issue28255}: calendar.TextCalendar().prmonth() no longer prints a space at
the start of new line after printing a month’s calendar.  Patch by Xiang
Zhang.

@item 
bpo-20491@footnote{https://bugs.python.org/issue20491}: The textwrap.TextWrapper class now honors non-breaking spaces.
Based on patch by Kaarle Ritvanen.

@item 
bpo-28353@footnote{https://bugs.python.org/issue28353}: os.fwalk() no longer fails on broken links.

@item 
bpo-25464@footnote{https://bugs.python.org/issue25464}: Fixed HList.header_exists() in tkinter.tix module by addin a
workaround to Tix library bug.

@item 
bpo-28488@footnote{https://bugs.python.org/issue28488}: shutil.make_archive() no longer add entry “./” to ZIP archive.

@item 
bpo-24452@footnote{https://bugs.python.org/issue24452}: Make webbrowser support Chrome on Mac OS X.

@item 
bpo-20766@footnote{https://bugs.python.org/issue20766}: Fix references leaked by pdb in the handling of SIGINT
handlers.

@item 
bpo-26293@footnote{https://bugs.python.org/issue26293}: Fixed writing ZIP files that starts not from the start of the
file.  Offsets in ZIP file now are relative to the start of the archive in
conforming to the specification.

@item 
bpo-28321@footnote{https://bugs.python.org/issue28321}: Fixed writing non-BMP characters with binary format in
plistlib.

@item 
bpo-28322@footnote{https://bugs.python.org/issue28322}: Fixed possible crashes when unpickle itertools objects from
incorrect pickle data.  Based on patch by John Leitch.

@item 
Fix possible integer overflows and crashes in the mmap module with unusual
usage patterns.

@item 
bpo-1703178@footnote{https://bugs.python.org/issue1703178}: Fix the ability to pass the –link-objects option to the
distutils build_ext command.

@item 
bpo-28253@footnote{https://bugs.python.org/issue28253}: Fixed calendar functions for extreme months: 0001-01 and
9999-12. Methods itermonthdays() and itermonthdays2() are reimplemented so
that they don’t call itermonthdates() which can cause datetime.date
under/overflow.

@item 
bpo-28275@footnote{https://bugs.python.org/issue28275}: Fixed possible use after free in the decompress() methods of
the LZMADecompressor and BZ2Decompressor classes. Original patch by John
Leitch.

@item 
bpo-27897@footnote{https://bugs.python.org/issue27897}: Fixed possible crash in sqlite3.Connection.create_collation()
if pass invalid string-like object as a name.  Patch by Xiang Zhang.

@item 
bpo-18893@footnote{https://bugs.python.org/issue18893}: Fix invalid exception handling in Lib/ctypes/macholib/dyld.py.
Patch by Madison May.

@item 
bpo-27611@footnote{https://bugs.python.org/issue27611}: Fixed support of default root window in the tkinter.tix module.

@item 
bpo-27348@footnote{https://bugs.python.org/issue27348}: In the traceback module, restore the formatting of exception
messages like “Exception: None”.  This fixes a regression introduced in
3.5a2.

@item 
bpo-25651@footnote{https://bugs.python.org/issue25651}: Allow falsy values to be used for msg parameter of subTest().

@item 
bpo-27932@footnote{https://bugs.python.org/issue27932}: Prevent memory leak in win32_ver().

@item 
Fix UnboundLocalError in socket._sendfile_use_sendfile.

@item 
bpo-28075@footnote{https://bugs.python.org/issue28075}: Check for ERROR_ACCESS_DENIED in Windows implementation of
os.stat().  Patch by Eryk Sun.

@item 
bpo-25270@footnote{https://bugs.python.org/issue25270}: Prevent codecs.escape_encode() from raising SystemError when an
empty bytestring is passed.

@item 
bpo-28181@footnote{https://bugs.python.org/issue28181}: Get antigravity over HTTPS. Patch by Kaartic Sivaraam.

@item 
bpo-25895@footnote{https://bugs.python.org/issue25895}: Enable WebSocket URL schemes in urllib.parse.urljoin. Patch by
Gergely Imreh and Markus Holtermann.

@item 
bpo-27599@footnote{https://bugs.python.org/issue27599}: Fixed buffer overrun in binascii.b2a_qp() and
binascii.a2b_qp().

@item 
bpo-19003@footnote{https://bugs.python.org/issue19003}: m email.generator now replaces only @code{\r} and/or @code{\n} line
endings, per the RFC, instead of all unicode line endings.

@item 
bpo-28019@footnote{https://bugs.python.org/issue28019}: itertools.count() no longer rounds non-integer step in range
between 1.0 and 2.0 to 1.

@item 
bpo-25969@footnote{https://bugs.python.org/issue25969}: Update the lib2to3 grammar to handle the unpacking
generalizations added in 3.5.

@item 
bpo-14977@footnote{https://bugs.python.org/issue14977}: mailcap now respects the order of the lines in the mailcap
files (“first match”), as required by RFC 1542.  Patch by Michael Lazar.

@item 
bpo-24594@footnote{https://bugs.python.org/issue24594}: Validates persist parameter when opening MSI database

@item 
bpo-17582@footnote{https://bugs.python.org/issue17582}: xml.etree.ElementTree nows preserves whitespaces in attributes
(Patch by Duane Griffin.  Reviewed and approved by Stefan Behnel.)

@item 
bpo-28047@footnote{https://bugs.python.org/issue28047}: Fixed calculation of line length used for the base64 CTE in the
new email policies.

@item 
bpo-27445@footnote{https://bugs.python.org/issue27445}: Don’t pass str(_charset) to MIMEText.set_payload(). Patch by
Claude Paroz.

@item 
bpo-22450@footnote{https://bugs.python.org/issue22450}: urllib now includes an @code{Accept: */*} header among the default
headers. This makes the results of REST API requests more consistent and
predictable especially when proxy servers are involved.

@item 
lib2to3.pgen3.driver.load_grammar() now creates a stable cache file
between runs given the same Grammar.txt input regardless of the hash
randomization setting.

@item 
bpo-27570@footnote{https://bugs.python.org/issue27570}: Avoid zero-length memcpy() etc calls with null source pointers
in the “ctypes” and “array” modules.

@item 
bpo-22233@footnote{https://bugs.python.org/issue22233}: Break email header lines @emph{only} on the RFC specified CR and LF
characters, not on arbitrary unicode line breaks.  This also fixes a bug
in HTTP header parsing.

@item 
bpo-27988@footnote{https://bugs.python.org/issue27988}: Fix email iter_attachments incorrect mutation of payload list.

@item 
bpo-27691@footnote{https://bugs.python.org/issue27691}: Fix ssl module’s parsing of GEN_RID subject alternative name
fields in X.509 certs.

@item 
bpo-27850@footnote{https://bugs.python.org/issue27850}: Remove 3DES from ssl module’s default cipher list to counter
measure sweet32 attack (CVE-2016-2183).

@item 
bpo-27766@footnote{https://bugs.python.org/issue27766}: Add ChaCha20 Poly1305 to ssl module’s default cipher list.
(Required OpenSSL 1.1.0 or LibreSSL).

@item 
bpo-26470@footnote{https://bugs.python.org/issue26470}: Port ssl and hashlib module to OpenSSL 1.1.0.

@item 
Remove support for passing a file descriptor to os.access. It never worked
but previously didn’t raise.

@item 
bpo-12885@footnote{https://bugs.python.org/issue12885}: Fix error when distutils encounters symlink.

@item 
bpo-27881@footnote{https://bugs.python.org/issue27881}: Fixed possible bugs when setting
sqlite3.Connection.isolation_level. Based on patch by Xiang Zhang.

@item 
bpo-27861@footnote{https://bugs.python.org/issue27861}: Fixed a crash in sqlite3.Connection.cursor() when a factory
creates not a cursor.  Patch by Xiang Zhang.

@item 
bpo-19884@footnote{https://bugs.python.org/issue19884}: Avoid spurious output on OS X with Gnu Readline.

@item 
bpo-27706@footnote{https://bugs.python.org/issue27706}: Restore deterministic behavior of random.Random().seed() for
string seeds using seeding version 1.  Allows sequences of calls to
random() to exactly match those obtained in Python 2. Patch by Nofar
Schnider.

@item 
bpo-10513@footnote{https://bugs.python.org/issue10513}: Fix a regression in Connection.commit().  Statements should not
be reset after a commit.

@item 
A new version of typing.py from @indicateurl{https://github.com/python/typing}:
Collection (only for 3.6) (bpo-27598@footnote{https://bugs.python.org/issue27598}). Add FrozenSet to __all__
(upstream #261). Fix crash in _get_type_vars() (upstream #259). Remove the
dict constraint in ForwardRef._eval_type (upstream #252).

@item 
bpo-27539@footnote{https://bugs.python.org/issue27539}: Fix unnormalised @code{Fraction.__pow__} result in the case of
negative exponent and negative base.

@item 
bpo-21718@footnote{https://bugs.python.org/issue21718}: cursor.description is now available for queries using CTEs.

@item 
bpo-2466@footnote{https://bugs.python.org/issue2466}: posixpath.ismount now correctly recognizes mount points which
the user does not have permission to access.

@item 
bpo-27773@footnote{https://bugs.python.org/issue27773}: Correct some memory management errors server_hostname in
_ssl.wrap_socket().

@item 
bpo-26750@footnote{https://bugs.python.org/issue26750}: unittest.mock.create_autospec() now works properly for
subclasses of property() and other data descriptors.

@item 
In the curses module, raise an error if window.getstr() or window.instr()
is passed a negative value.

@item 
bpo-27783@footnote{https://bugs.python.org/issue27783}: Fix possible usage of uninitialized memory in
operator.methodcaller.

@item 
bpo-27774@footnote{https://bugs.python.org/issue27774}: Fix possible Py_DECREF on unowned object in _sre.

@item 
bpo-27760@footnote{https://bugs.python.org/issue27760}: Fix possible integer overflow in binascii.b2a_qp.

@item 
bpo-27758@footnote{https://bugs.python.org/issue27758}: Fix possible integer overflow in the _csv module for large
record lengths.

@item 
bpo-27568@footnote{https://bugs.python.org/issue27568}: Prevent HTTPoxy attack (CVE-2016-1000110). Ignore the
HTTP_PROXY variable when REQUEST_METHOD environment is set, which
indicates that the script is in CGI mode.

@item 
bpo-27656@footnote{https://bugs.python.org/issue27656}: Do not assume sched.h defines any SCHED_* constants.

@item 
bpo-27130@footnote{https://bugs.python.org/issue27130}: In the “zlib” module, fix handling of large buffers (typically
4 GiB) when compressing and decompressing.  Previously, inputs were
limited to 4 GiB, and compression and decompression operations did not
properly handle results of 4 GiB.

@item 
bpo-27533@footnote{https://bugs.python.org/issue27533}: Release GIL in nt._isdir

@item 
bpo-17711@footnote{https://bugs.python.org/issue17711}: Fixed unpickling by the persistent ID with protocol 0. Original
patch by Alexandre Vassalotti.

@item 
bpo-27522@footnote{https://bugs.python.org/issue27522}: Avoid an unintentional reference cycle in email.feedparser.

@item 
bpo-26844@footnote{https://bugs.python.org/issue26844}: Fix error message for imp.find_module() to refer to ‘path’
instead of ‘name’. Patch by Lev Maximov.

@item 
bpo-23804@footnote{https://bugs.python.org/issue23804}: Fix SSL zero-length recv() calls to not block and not raise an
error about unclean EOF.

@item 
bpo-27466@footnote{https://bugs.python.org/issue27466}: Change time format returned by http.cookie.time2netscape,
confirming the netscape cookie format and making it consistent with
documentation.

@item 
bpo-26664@footnote{https://bugs.python.org/issue26664}: Fix activate.fish by removing mis-use of @code{$}.

@item 
bpo-22115@footnote{https://bugs.python.org/issue22115}: Fixed tracing Tkinter variables: trace_vdelete() with wrong
mode no longer break tracing, trace_vinfo() now always returns a list of
pairs of strings, tracing in the “u” mode now works.

@item 
Fix a scoping issue in importlib.util.LazyLoader which triggered an
UnboundLocalError when lazy-loading a module that was already put into
sys.modules.

@item 
bpo-27079@footnote{https://bugs.python.org/issue27079}: Fixed curses.ascii functions isblank(), iscntrl() and
ispunct().

@item 
bpo-26754@footnote{https://bugs.python.org/issue26754}: Some functions (compile() etc) accepted a filename argument
encoded as an iterable of integers. Now only strings and byte-like objects
are accepted.

@item 
bpo-27048@footnote{https://bugs.python.org/issue27048}: Prevents distutils failing on Windows when environment
variables contain non-ASCII characters

@item 
bpo-27330@footnote{https://bugs.python.org/issue27330}: Fixed possible leaks in the ctypes module.

@item 
bpo-27238@footnote{https://bugs.python.org/issue27238}: Got rid of bare excepts in the turtle module.  Original patch
by Jelle Zijlstra.

@item 
bpo-27122@footnote{https://bugs.python.org/issue27122}: When an exception is raised within the context being managed by
a contextlib.ExitStack() and one of the exit stack generators catches and
raises it in a chain, do not re-raise the original exception when exiting,
let the new chained one through.  This avoids the 
@geindex Python Enhancement Proposals; PEP 479
PEP 479@footnote{https://www.python.org/dev/peps/pep-0479} bug described
in issue25782.
@end itemize

@node Security<35>,Library<66>,Library<65>,Python 3 5 3 release candidate 1
@anchor{whatsnew/changelog id485}@anchor{1449}
@subsubsection Security



@itemize -

@item 
bpo-27278@footnote{https://bugs.python.org/issue27278}: Fix os.urandom() implementation using getrandom() on Linux.
Truncate size to INT_MAX and loop until we collected enough random bytes,
instead of casting a directly Py_ssize_t to int.
@end itemize

@node Library<66>,Security<36>,Security<35>,Python 3 5 3 release candidate 1
@anchor{whatsnew/changelog id486}@anchor{144a}
@subsubsection Library



@itemize -

@item 
bpo-26386@footnote{https://bugs.python.org/issue26386}: Fixed ttk.TreeView selection operations with item id’s
containing spaces.
@end itemize

@node Security<36>,Library<67>,Library<66>,Python 3 5 3 release candidate 1
@anchor{whatsnew/changelog id487}@anchor{144b}
@subsubsection Security



@itemize -

@item 
bpo-22636@footnote{https://bugs.python.org/issue22636}: Avoid shell injection problems with ctypes.util.find_library().
@end itemize

@node Library<67>,IDLE<44>,Security<36>,Python 3 5 3 release candidate 1
@anchor{whatsnew/changelog id488}@anchor{144c}
@subsubsection Library



@itemize -

@item 
bpo-16182@footnote{https://bugs.python.org/issue16182}: Fix various functions in the “readline” module to use the
locale encoding, and fix get_begidx() and get_endidx() to return code
point indexes.

@item 
bpo-27392@footnote{https://bugs.python.org/issue27392}: Add loop.connect_accepted_socket(). Patch by Jim Fulton.

@item 
bpo-27930@footnote{https://bugs.python.org/issue27930}: Improved behaviour of logging.handlers.QueueListener. Thanks to
Paulo Andrade and Petr Viktorin for the analysis and patch.

@item 
bpo-21201@footnote{https://bugs.python.org/issue21201}: Improves readability of multiprocessing error message.  Thanks
to Wojciech Walczak for patch.

@item 
bpo-27456@footnote{https://bugs.python.org/issue27456}: asyncio: Set TCP_NODELAY by default.

@item 
bpo-27906@footnote{https://bugs.python.org/issue27906}: Fix socket accept exhaustion during high TCP traffic. Patch by
Kevin Conway.

@item 
bpo-28174@footnote{https://bugs.python.org/issue28174}: Handle when SO_REUSEPORT isn’t properly supported. Patch by
Seth Michael Larson.

@item 
bpo-26654@footnote{https://bugs.python.org/issue26654}: Inspect functools.partial in asyncio.Handle.__repr__. Patch by
iceboy.

@item 
bpo-26909@footnote{https://bugs.python.org/issue26909}: Fix slow pipes IO in asyncio. Patch by INADA Naoki.

@item 
bpo-28176@footnote{https://bugs.python.org/issue28176}: Fix callbacks race in asyncio.SelectorLoop.sock_connect.

@item 
bpo-27759@footnote{https://bugs.python.org/issue27759}: Fix selectors incorrectly retain invalid file descriptors.
Patch by Mark Williams.

@item 
bpo-28368@footnote{https://bugs.python.org/issue28368}: Refuse monitoring processes if the child watcher has no loop
attached. Patch by Vincent Michel.

@item 
bpo-28369@footnote{https://bugs.python.org/issue28369}: Raise RuntimeError when transport’s FD is used with add_reader,
add_writer, etc.

@item 
bpo-28370@footnote{https://bugs.python.org/issue28370}: Speedup asyncio.StreamReader.readexactly. Patch by Коренберг
Марк.

@item 
bpo-28371@footnote{https://bugs.python.org/issue28371}: Deprecate passing asyncio.Handles to run_in_executor.

@item 
bpo-28372@footnote{https://bugs.python.org/issue28372}: Fix asyncio to support formatting of non-python coroutines.

@item 
bpo-28399@footnote{https://bugs.python.org/issue28399}: Remove UNIX socket from FS before binding. Patch by Коренберг
Марк.

@item 
bpo-27972@footnote{https://bugs.python.org/issue27972}: Prohibit Tasks to await on themselves.

@item 
bpo-26923@footnote{https://bugs.python.org/issue26923}: Fix asyncio.Gather to refuse being cancelled once all children
are done. Patch by Johannes Ebke.

@item 
bpo-26796@footnote{https://bugs.python.org/issue26796}: Don’t configure the number of workers for default threadpool
executor. Initial patch by Hans Lawrenz.

@item 
bpo-28600@footnote{https://bugs.python.org/issue28600}: Optimize loop.call_soon().

@item 
bpo-28613@footnote{https://bugs.python.org/issue28613}: Fix get_event_loop() return the current loop if called from
coroutines/callbacks.

@item 
bpo-28639@footnote{https://bugs.python.org/issue28639}: Fix inspect.isawaitable to always return bool Patch by Justin
Mayfield.

@item 
bpo-28652@footnote{https://bugs.python.org/issue28652}: Make loop methods reject socket kinds they do not support.

@item 
bpo-28653@footnote{https://bugs.python.org/issue28653}: Fix a refleak in functools.lru_cache.

@item 
bpo-28703@footnote{https://bugs.python.org/issue28703}: Fix asyncio.iscoroutinefunction to handle Mock objects.

@item 
bpo-24142@footnote{https://bugs.python.org/issue24142}: Reading a corrupt config file left the parser in an invalid
state.  Original patch by Florian Höch.

@item 
bpo-28990@footnote{https://bugs.python.org/issue28990}: Fix SSL hanging if connection is closed before handshake
completed. (Patch by HoHo-Ho)
@end itemize

@node IDLE<44>,C API<45>,Library<67>,Python 3 5 3 release candidate 1
@anchor{whatsnew/changelog id489}@anchor{144d}
@subsubsection IDLE



@itemize -

@item 
bpo-15308@footnote{https://bugs.python.org/issue15308}: Add ‘interrupt execution’ (^C) to Shell menu. Patch by Roger
Serwy, updated by Bayard Randel.

@item 
bpo-27922@footnote{https://bugs.python.org/issue27922}: Stop IDLE tests from ‘flashing’ gui widgets on the screen.

@item 
Add version to title of IDLE help window.

@item 
bpo-25564@footnote{https://bugs.python.org/issue25564}: In section on IDLE – console differences, mention that using
exec means that __builtins__ is defined for each statement.

@item 
bpo-27714@footnote{https://bugs.python.org/issue27714}: text_textview and test_autocomplete now pass when re-run in the
same process.  This occurs when test_idle fails when run with the -w
option but without -jn.  Fix warning from test_config.

@item 
bpo-25507@footnote{https://bugs.python.org/issue25507}: IDLE no longer runs buggy code because of its tkinter imports.
Users must include the same imports required to run directly in Python.

@item 
bpo-27452@footnote{https://bugs.python.org/issue27452}: add line counter and crc to IDLE configHandler test dump.

@item 
bpo-27365@footnote{https://bugs.python.org/issue27365}: Allow non-ascii chars in IDLE NEWS.txt, for contributor names.

@item 
bpo-27245@footnote{https://bugs.python.org/issue27245}: IDLE: Cleanly delete custom themes and key bindings.
Previously, when IDLE was started from a console or by import, a cascade
of warnings was emitted. Patch by Serhiy Storchaka.
@end itemize

@node C API<45>,Documentation<50>,IDLE<44>,Python 3 5 3 release candidate 1
@anchor{whatsnew/changelog id490}@anchor{144e}
@subsubsection C API



@itemize -

@item 
bpo-28808@footnote{https://bugs.python.org/issue28808}: PyUnicode_CompareWithASCIIString() now never raises exceptions.

@item 
bpo-26754@footnote{https://bugs.python.org/issue26754}: PyUnicode_FSDecoder() accepted a filename argument encoded as
an iterable of integers. Now only strings and bytes-like objects are
accepted.
@end itemize

@node Documentation<50>,Tests<50>,C API<45>,Python 3 5 3 release candidate 1
@anchor{whatsnew/changelog id491}@anchor{144f}
@subsubsection Documentation



@itemize -

@item 
bpo-28513@footnote{https://bugs.python.org/issue28513}: Documented command-line interface of zipfile.
@end itemize

@node Tests<50>,Tools/Demos<29>,Documentation<50>,Python 3 5 3 release candidate 1
@anchor{whatsnew/changelog id492}@anchor{1450}
@subsubsection Tests



@itemize -

@item 
bpo-28950@footnote{https://bugs.python.org/issue28950}: Disallow -j0 to be combined with -T/-l/-M in regrtest command
line arguments.

@item 
bpo-28666@footnote{https://bugs.python.org/issue28666}: Now test.support.rmtree is able to remove unwritable or
unreadable directories.

@item 
bpo-23839@footnote{https://bugs.python.org/issue23839}: Various caches now are cleared before running every test file.

@item 
bpo-28409@footnote{https://bugs.python.org/issue28409}: regrtest: fix the parser of command line arguments.

@item 
bpo-27787@footnote{https://bugs.python.org/issue27787}: Call gc.collect() before checking each test for “dangling
threads”, since the dangling threads are weak references.

@item 
bpo-27369@footnote{https://bugs.python.org/issue27369}: In test_pyexpat, avoid testing an error message detail that
changed in Expat 2.2.0.
@end itemize

@node Tools/Demos<29>,Windows<48>,Tests<50>,Python 3 5 3 release candidate 1
@anchor{whatsnew/changelog id493}@anchor{1451}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-27952@footnote{https://bugs.python.org/issue27952}: Get Tools/scripts/fixcid.py working with Python 3 and the
current “re” module, avoid invalid Python backslash escapes, and fix a bug
parsing escaped C quote signs.

@item 
bpo-27332@footnote{https://bugs.python.org/issue27332}: Fixed the type of the first argument of module-level functions
generated by Argument Clinic.  Patch by Petr Viktorin.

@item 
bpo-27418@footnote{https://bugs.python.org/issue27418}: Fixed Tools/importbench/importbench.py.
@end itemize

@node Windows<48>,Build<54>,Tools/Demos<29>,Python 3 5 3 release candidate 1
@anchor{whatsnew/changelog id494}@anchor{1452}
@subsubsection Windows



@itemize -

@item 
bpo-28251@footnote{https://bugs.python.org/issue28251}: Improvements to help manuals on Windows.

@item 
bpo-28110@footnote{https://bugs.python.org/issue28110}: launcher.msi has different product codes between 32-bit and
64-bit

@item 
bpo-25144@footnote{https://bugs.python.org/issue25144}: Ensures TargetDir is set before continuing with custom install.

@item 
bpo-27469@footnote{https://bugs.python.org/issue27469}: Adds a shell extension to the launcher so that drag and drop
works correctly.

@item 
bpo-27309@footnote{https://bugs.python.org/issue27309}: Enabled proper Windows styles in python[w].exe manifest.
@end itemize

@node Build<54>,,Windows<48>,Python 3 5 3 release candidate 1
@anchor{whatsnew/changelog id495}@anchor{1453}
@subsubsection Build



@itemize -

@item 
bpo-29080@footnote{https://bugs.python.org/issue29080}: Removes hard dependency on hg.exe from PCBuild/build.bat

@item 
bpo-23903@footnote{https://bugs.python.org/issue23903}: Added missed names to PC/python3.def.

@item 
bpo-10656@footnote{https://bugs.python.org/issue10656}: Fix out-of-tree building on AIX.  Patch by Tristan Carel and
Michael Haubenwallner.

@item 
bpo-26359@footnote{https://bugs.python.org/issue26359}: Rename –with-optimiations to –enable-optimizations.

@item 
bpo-28444@footnote{https://bugs.python.org/issue28444}: Fix missing extensions modules when cross compiling.

@item 
bpo-28248@footnote{https://bugs.python.org/issue28248}: Update Windows build and OS X installers to use OpenSSL 1.0.2j.

@item 
bpo-28258@footnote{https://bugs.python.org/issue28258}: Fixed build with Estonian locale (python-config and distclean
targets in Makefile).  Patch by Arfrever Frehtes Taifersar Arahesis.

@item 
bpo-26661@footnote{https://bugs.python.org/issue26661}: setup.py now detects system libffi with multiarch wrapper.

@item 
bpo-28066@footnote{https://bugs.python.org/issue28066}: Fix the logic that searches build directories for generated
include files when building outside the source tree.

@item 
bpo-15819@footnote{https://bugs.python.org/issue15819}: Remove redundant include search directory option for building
outside the source tree.

@item 
bpo-27566@footnote{https://bugs.python.org/issue27566}: Fix clean target in freeze makefile (patch by Lisa Roach)

@item 
bpo-27705@footnote{https://bugs.python.org/issue27705}: Update message in validate_ucrtbase.py

@item 
bpo-27983@footnote{https://bugs.python.org/issue27983}: Cause lack of llvm-profdata tool when using clang as required
for PGO linking to be a configure time error rather than make time when
–with-optimizations is enabled.  Also improve our ability to find the
llvm-profdata tool on MacOS and some Linuxes.

@item 
bpo-26307@footnote{https://bugs.python.org/issue26307}: The profile-opt build now applies PGO to the built-in modules.

@item 
bpo-26359@footnote{https://bugs.python.org/issue26359}: Add the –with-optimizations configure flag.

@item 
bpo-27713@footnote{https://bugs.python.org/issue27713}: Suppress spurious build warnings when updating importlib’s
bootstrap files. Patch by Xiang Zhang

@item 
bpo-25825@footnote{https://bugs.python.org/issue25825}: Correct the references to Modules/python.exp and ld_so_aix,
which are required on AIX.  This updates references to an installation
path that was changed in 3.2a4, and undoes changed references to the build
tree that were made in 3.5.0a1.

@item 
bpo-27453@footnote{https://bugs.python.org/issue27453}: CPP invocation in configure must use CPPFLAGS. Patch by Chi
Hsuan Yen.

@item 
bpo-27641@footnote{https://bugs.python.org/issue27641}: The configure script now inserts comments into the makefile to
prevent the pgen and _freeze_importlib executables from being
cross-compiled.

@item 
bpo-26662@footnote{https://bugs.python.org/issue26662}: Set PYTHON_FOR_GEN in configure as the Python program to be
used for file generation during the build.

@item 
bpo-10910@footnote{https://bugs.python.org/issue10910}: Avoid C++ compilation errors on FreeBSD and OS X. Also update
FreedBSD version checks for the original ctype UTF-8 workaround.

@item 
bpo-28676@footnote{https://bugs.python.org/issue28676}: Prevent missing ‘getentropy’ declaration warning on macOS.
Patch by Gareth Rees.
@end itemize

@node Python 3 5 2 final,Python 3 5 2 release candidate 1,Python 3 5 3 release candidate 1,Changelog
@anchor{whatsnew/changelog python-3-5-2-final}@anchor{1454}
@subsection Python 3.5.2 final


@emph{Release date: 2016-06-26}

@menu
* Core and Builtins: Core and Builtins<57>. 
* Tests: Tests<51>. 
* IDLE: IDLE<45>. 

@end menu

@node Core and Builtins<57>,Tests<51>,,Python 3 5 2 final
@anchor{whatsnew/changelog id496}@anchor{1455}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-26930@footnote{https://bugs.python.org/issue26930}: Update Windows builds to use OpenSSL 1.0.2h.
@end itemize

@node Tests<51>,IDLE<45>,Core and Builtins<57>,Python 3 5 2 final
@anchor{whatsnew/changelog id497}@anchor{1456}
@subsubsection Tests



@itemize -

@item 
bpo-26867@footnote{https://bugs.python.org/issue26867}: Ubuntu’s openssl OP_NO_SSLv3 is forced on by default; fix test.
@end itemize

@node IDLE<45>,,Tests<51>,Python 3 5 2 final
@anchor{whatsnew/changelog id498}@anchor{1457}
@subsubsection IDLE



@itemize -

@item 
bpo-27365@footnote{https://bugs.python.org/issue27365}: Allow non-ascii in idlelib/NEWS.txt - minimal part for 3.5.2.
@end itemize

@node Python 3 5 2 release candidate 1,Python 3 5 1 final,Python 3 5 2 final,Changelog
@anchor{whatsnew/changelog python-3-5-2-release-candidate-1}@anchor{1458}
@subsection Python 3.5.2 release candidate 1


@emph{Release date: 2016-06-12}

@menu
* Core and Builtins: Core and Builtins<58>. 
* Security: Security<37>. 
* Library: Library<68>. 
* Security: Security<38>. 
* Library: Library<69>. 
* Security: Security<39>. 
* Library: Library<70>. 
* Security: Security<40>. 
* Library: Library<71>. 
* Security: Security<41>. 
* Library: Library<72>. 
* IDLE: IDLE<46>. 
* Documentation: Documentation<51>. 
* Tests: Tests<52>. 
* Build: Build<55>. 
* Windows: Windows<49>. 
* Tools/Demos: Tools/Demos<30>. 
* Windows: Windows<50>. 

@end menu

@node Core and Builtins<58>,Security<37>,,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id499}@anchor{1459}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-27066@footnote{https://bugs.python.org/issue27066}: Fixed SystemError if a custom opener (for open()) returns a
negative number without setting an exception.

@item 
bpo-20041@footnote{https://bugs.python.org/issue20041}: Fixed TypeError when frame.f_trace is set to None. Patch by
Xavier de Gaye.

@item 
bpo-26168@footnote{https://bugs.python.org/issue26168}: Fixed possible refleaks in failing Py_BuildValue() with the “N”
format unit.

@item 
bpo-26991@footnote{https://bugs.python.org/issue26991}: Fix possible refleak when creating a function with annotations.

@item 
bpo-27039@footnote{https://bugs.python.org/issue27039}: Fixed bytearray.remove() for values greater than 127.  Patch by
Joe Jevnik.

@item 
bpo-23640@footnote{https://bugs.python.org/issue23640}: int.from_bytes() no longer bypasses constructors for
subclasses.

@item 
bpo-26811@footnote{https://bugs.python.org/issue26811}: gc.get_objects() no longer contains a broken tuple with NULL
pointer.

@item 
bpo-20120@footnote{https://bugs.python.org/issue20120}: Use RawConfigParser for .pypirc parsing, removing support for
interpolation unintentionally added with move to Python 3. Behavior no
longer does any interpolation in .pypirc files, matching behavior in
Python 2.7 and Setuptools 19.0.

@item 
bpo-26659@footnote{https://bugs.python.org/issue26659}: Make the builtin slice type support cycle collection.

@item 
bpo-26718@footnote{https://bugs.python.org/issue26718}: super.__init__ no longer leaks memory if called multiple times.
NOTE: A direct call of super.__init__ is not endorsed!

@item 
bpo-25339@footnote{https://bugs.python.org/issue25339}: PYTHONIOENCODING now has priority over locale in setting the
error handler for stdin and stdout.

@item 
bpo-26494@footnote{https://bugs.python.org/issue26494}: Fixed crash on iterating exhausting iterators. Affected classes
are generic sequence iterators, iterators of str, bytes, bytearray, list,
tuple, set, frozenset, dict, OrderedDict, corresponding views and
os.scandir() iterator.

@item 
bpo-26581@footnote{https://bugs.python.org/issue26581}: If coding cookie is specified multiple times on a line in
Python source code file, only the first one is taken to account.

@item 
bpo-26464@footnote{https://bugs.python.org/issue26464}: Fix str.translate() when string is ASCII and first replacements
removes character, but next replacement uses a non-ASCII character or a
string longer than 1 character. Regression introduced in Python 3.5.0.

@item 
bpo-22836@footnote{https://bugs.python.org/issue22836}: Ensure exception reports from PyErr_Display() and
PyErr_WriteUnraisable() are sensible even when formatting them produces
secondary errors.  This affects the reports produced by
sys.__excepthook__() and when __del__() raises an exception.

@item 
bpo-26302@footnote{https://bugs.python.org/issue26302}: Correct behavior to reject comma as a legal character for
cookie names.

@item 
bpo-4806@footnote{https://bugs.python.org/issue4806}: Avoid masking the original TypeError exception when using star
(@code{*}) unpacking in function calls.  Based on patch by Hagen Fürstenau
and Daniel Urban.

@item 
bpo-27138@footnote{https://bugs.python.org/issue27138}: Fix the doc comment for FileFinder.find_spec().

@item 
bpo-26154@footnote{https://bugs.python.org/issue26154}: Add a new private _PyThreadState_UncheckedGet() function to get
the current Python thread state, but don’t issue a fatal error if it is
NULL. This new function must be used instead of accessing directly the
_PyThreadState_Current variable.  The variable is no more exposed since
Python 3.5.1 to hide the exact implementation of atomic C types, to avoid
compiler issues.

@item 
bpo-26194@footnote{https://bugs.python.org/issue26194}: Deque.insert() gave odd results for bounded deques that had
reached their maximum size.  Now an IndexError will be raised when
attempting to insert into a full deque.

@item 
bpo-25843@footnote{https://bugs.python.org/issue25843}: When compiling code, don’t merge constants if they are equal
but have a different types. For example, @code{f1, f2 = lambda: 1, lambda:
1.0} is now correctly compiled to two different functions: @code{f1()}
returns @code{1} (@code{int}) and @code{f2()} returns @code{1.0} (@code{int}), even if
@code{1} and @code{1.0} are equal.

@item 
bpo-22995@footnote{https://bugs.python.org/issue22995}: [UPDATE] Comment out the one of the pickleability tests in
_PyObject_GetState() due to regressions observed in Cython-based projects.

@item 
bpo-25961@footnote{https://bugs.python.org/issue25961}: Disallowed null characters in the type name.

@item 
bpo-25973@footnote{https://bugs.python.org/issue25973}: Fix segfault when an invalid nonlocal statement binds a name
starting with two underscores.

@item 
bpo-22995@footnote{https://bugs.python.org/issue22995}: Instances of extension types with a state that aren’t
subclasses of list or dict and haven’t implemented any pickle-related
methods (__reduce__, __reduce_ex__, __getnewargs__, __getnewargs_ex__, or
__getstate__), can no longer be pickled.  Including memoryview.

@item 
bpo-20440@footnote{https://bugs.python.org/issue20440}: Massive replacing unsafe attribute setting code with special
macro Py_SETREF.

@item 
bpo-25766@footnote{https://bugs.python.org/issue25766}: Special method __bytes__() now works in str subclasses.

@item 
bpo-25421@footnote{https://bugs.python.org/issue25421}: __sizeof__ methods of builtin types now use dynamic basic size.
This allows sys.getsize() to work correctly with their subclasses with
__slots__ defined.

@item 
bpo-25709@footnote{https://bugs.python.org/issue25709}: Fixed problem with in-place string concatenation and utf-8
cache.

@item 
bpo-27147@footnote{https://bugs.python.org/issue27147}: Mention 
@geindex Python Enhancement Proposals; PEP 420
PEP 420@footnote{https://www.python.org/dev/peps/pep-0420} in the importlib docs.

@item 
bpo-24097@footnote{https://bugs.python.org/issue24097}: Fixed crash in object.__reduce__() if slot name is freed inside
__getattr__.

@item 
bpo-24731@footnote{https://bugs.python.org/issue24731}: Fixed crash on converting objects with special methods
__bytes__, __trunc__, and __float__ returning instances of subclasses of
bytes, int, and float to subclasses of bytes, int, and float
correspondingly.

@item 
bpo-26478@footnote{https://bugs.python.org/issue26478}: Fix semantic bugs when using binary operators with dictionary
views and tuples.

@item 
bpo-26171@footnote{https://bugs.python.org/issue26171}: Fix possible integer overflow and heap corruption in
zipimporter.get_data().

@item 
bpo-25660@footnote{https://bugs.python.org/issue25660}: Fix TAB key behaviour in REPL with readline.

@item 
bpo-25887@footnote{https://bugs.python.org/issue25887}: Raise a RuntimeError when a coroutine object is awaited more
than once.

@item 
bpo-27243@footnote{https://bugs.python.org/issue27243}: Update the __aiter__ protocol: instead of returning an
awaitable that resolves to an asynchronous iterator, the asynchronous
iterator should be returned directly.  Doing the former will trigger a
PendingDeprecationWarning.
@end itemize

@node Security<37>,Library<68>,Core and Builtins<58>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id500}@anchor{145a}
@subsubsection Security



@itemize -

@item 
bpo-26556@footnote{https://bugs.python.org/issue26556}: Update expat to 2.1.1, fixes CVE-2015-1283.

@item 
Fix TLS stripping vulnerability in smtplib, CVE-2016-0772. Reported by
Team Oststrom
@end itemize

@node Library<68>,Security<38>,Security<37>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id501}@anchor{145b}
@subsubsection Library



@itemize -

@item 
bpo-21386@footnote{https://bugs.python.org/issue21386}: Implement missing IPv4Address.is_global property.  It was
documented since 07a5610bae9d.  Initial patch by Roger Luethi.

@item 
bpo-20900@footnote{https://bugs.python.org/issue20900}: distutils register command now decodes HTTP responses
correctly.  Initial patch by ingrid.

@item 
A new version of typing.py provides several new classes and features:
@@overload outside stubs, Reversible, DefaultDict, Text, ContextManager,
Type[], NewType(), TYPE_CHECKING, and numerous bug fixes (note that some
of the new features are not yet implemented in mypy or other static
analyzers). Also classes for 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} (Awaitable, AsyncIterable,
AsyncIterator) have been added (in fact they made it into 3.5.1 but were
never mentioned).

@item 
bpo-25738@footnote{https://bugs.python.org/issue25738}: Stop http.server.BaseHTTPRequestHandler.send_error() from
sending a message body for 205 Reset Content.  Also, don’t send Content
header fields in responses that don’t have a body.  Patch by Susumu
Koshiba.

@item 
bpo-21313@footnote{https://bugs.python.org/issue21313}: Fix the “platform” module to tolerate when sys.version contains
truncated build information.
@end itemize

@node Security<38>,Library<69>,Library<68>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id502}@anchor{145c}
@subsubsection Security



@itemize -

@item 
bpo-26839@footnote{https://bugs.python.org/issue26839}: On Linux, @ref{17f,,os.urandom()} now calls @code{getrandom()} with
@code{GRND_NONBLOCK} to fall back on reading @code{/dev/urandom} if the urandom
entropy pool is not initialized yet. Patch written by Colm Buckley.
@end itemize

@node Library<69>,Security<39>,Security<38>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id503}@anchor{145d}
@subsubsection Library



@itemize -

@item 
bpo-27164@footnote{https://bugs.python.org/issue27164}: In the zlib module, allow decompressing raw Deflate streams
with a predefined zdict.  Based on patch by Xiang Zhang.

@item 
bpo-24291@footnote{https://bugs.python.org/issue24291}: Fix wsgiref.simple_server.WSGIRequestHandler to completely
write data to the client.  Previously it could do partial writes and
truncate data.  Also, wsgiref.handler.ServerHandler can now handle stdout
doing partial writes, but this is deprecated.

@item 
bpo-26809@footnote{https://bugs.python.org/issue26809}: Add @code{__all__} to @ref{f1,,string}.  Patch by Emanuel Barry.

@item 
bpo-26373@footnote{https://bugs.python.org/issue26373}: subprocess.Popen.communicate now correctly ignores
BrokenPipeError when the child process dies before .communicate() is
called in more/all circumstances.

@item 
bpo-21776@footnote{https://bugs.python.org/issue21776}: distutils.upload now correctly handles HTTPError. Initial patch
by Claudiu Popa.

@item 
bpo-27114@footnote{https://bugs.python.org/issue27114}: Fix SSLContext._load_windows_store_certs fails with
PermissionError

@item 
bpo-18383@footnote{https://bugs.python.org/issue18383}: Avoid creating duplicate filters when using filterwarnings and
simplefilter. Based on patch by Alex Shkop.

@item 
bpo-27057@footnote{https://bugs.python.org/issue27057}: Fix os.set_inheritable() on Android, ioctl() is blocked by
SELinux and fails with EACCESS. The function now falls back to fcntl().
Patch written by Michał Bednarski.

@item 
bpo-27014@footnote{https://bugs.python.org/issue27014}: Fix infinite recursion using typing.py.  Thanks to Kalle Tuure!

@item 
bpo-14132@footnote{https://bugs.python.org/issue14132}: Fix urllib.request redirect handling when the target only has a
query string.  Original fix by Ján Janech.

@item 
bpo-17214@footnote{https://bugs.python.org/issue17214}: The “urllib.request” module now percent-encodes non-ASCII bytes
found in redirect target URLs.  Some servers send Location header fields
with non-ASCII bytes, but “http.client” requires the request target to be
ASCII-encodable, otherwise a UnicodeEncodeError is raised.  Based on patch
by Christian Heimes.

@item 
bpo-26892@footnote{https://bugs.python.org/issue26892}: Honor debuglevel flag in urllib.request.HTTPHandler. Patch
contributed by Chi Hsuan Yen.

@item 
bpo-22274@footnote{https://bugs.python.org/issue22274}: In the subprocess module, allow stderr to be redirected to
stdout even when stdout is not redirected.  Patch by Akira Li.

@item 
bpo-26807@footnote{https://bugs.python.org/issue26807}: mock_open ‘files’ no longer error on readline at end of file.
Patch from Yolanda Robla.

@item 
bpo-25745@footnote{https://bugs.python.org/issue25745}: Fixed leaking a userptr in curses panel destructor.

@item 
bpo-26977@footnote{https://bugs.python.org/issue26977}: Removed unnecessary, and ignored, call to sum of squares helper
in statistics.pvariance.

@item 
bpo-26881@footnote{https://bugs.python.org/issue26881}: The modulefinder module now supports extended opcode arguments.

@item 
bpo-23815@footnote{https://bugs.python.org/issue23815}: Fixed crashes related to directly created instances of types in
_tkinter and curses.panel modules.

@item 
bpo-17765@footnote{https://bugs.python.org/issue17765}: weakref.ref() no longer silently ignores keyword arguments.
Patch by Georg Brandl.

@item 
bpo-26873@footnote{https://bugs.python.org/issue26873}: xmlrpc now raises ResponseError on unsupported type tags
instead of silently return incorrect result.

@item 
bpo-26711@footnote{https://bugs.python.org/issue26711}: Fixed the comparison of plistlib.Data with other types.

@item 
bpo-24114@footnote{https://bugs.python.org/issue24114}: Fix an uninitialized variable in @code{ctypes.util}. The bug only
occurs on SunOS when the ctypes implementation searches for the @code{crle}
program.  Patch by Xiang Zhang.  Tested on SunOS by Kees Bos.

@item 
bpo-26864@footnote{https://bugs.python.org/issue26864}: In urllib.request, change the proxy bypass host checking
against no_proxy to be case-insensitive, and to not match unrelated host
names that happen to have a bypassed hostname as a suffix.  Patch by Xiang
Zhang.

@item 
bpo-26634@footnote{https://bugs.python.org/issue26634}: recursive_repr() now sets __qualname__ of wrapper.  Patch by
Xiang Zhang.

@item 
bpo-26804@footnote{https://bugs.python.org/issue26804}: urllib.request will prefer lower_case proxy environment
variables over UPPER_CASE or Mixed_Case ones. Patch contributed by
Hans-Peter Jansen.

@item 
bpo-26837@footnote{https://bugs.python.org/issue26837}: assertSequenceEqual() now correctly outputs non-stringified
differing items (like bytes in the -b mode).  This affects
assertListEqual() and assertTupleEqual().

@item 
bpo-26041@footnote{https://bugs.python.org/issue26041}: Remove “will be removed in Python 3.7” from deprecation
messages of platform.dist() and platform.linux_distribution(). Patch by
Kumaripaba Miyurusara Athukorala.

@item 
bpo-26822@footnote{https://bugs.python.org/issue26822}: itemgetter, attrgetter and methodcaller objects no longer
silently ignore keyword arguments.

@item 
bpo-26733@footnote{https://bugs.python.org/issue26733}: Disassembling a class now disassembles class and static
methods. Patch by Xiang Zhang.

@item 
bpo-26801@footnote{https://bugs.python.org/issue26801}: Fix error handling in @ref{c96,,shutil.get_terminal_size()}, catch
@ref{163,,AttributeError} instead of @ref{21f,,NameError}. Patch written by
Emanuel Barry.

@item 
bpo-24838@footnote{https://bugs.python.org/issue24838}: tarfile’s ustar and gnu formats now correctly calculate name
and link field limits for multibyte character encodings like utf-8.
@end itemize

@node Security<39>,Library<70>,Library<69>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id504}@anchor{145e}
@subsubsection Security



@itemize -

@item 
bpo-26657@footnote{https://bugs.python.org/issue26657}: Fix directory traversal vulnerability with http.server on
Windows.  This fixes a regression that was introduced in 3.3.4rc1 and
3.4.0rc1.  Based on patch by Philipp Hagemeister.
@end itemize

@node Library<70>,Security<40>,Security<39>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id505}@anchor{145f}
@subsubsection Library



@itemize -

@item 
bpo-26717@footnote{https://bugs.python.org/issue26717}: Stop encoding Latin-1-ized WSGI paths with UTF-8.  Patch by
Anthony Sottile.

@item 
bpo-26735@footnote{https://bugs.python.org/issue26735}: Fix @ref{17f,,os.urandom()} on Solaris 11.3 and newer when reading
more than 1,024 bytes: call @code{getrandom()} multiple times with a limit of
1024 bytes per call.

@item 
bpo-16329@footnote{https://bugs.python.org/issue16329}: Add .webm to mimetypes.types_map.  Patch by Giampaolo Rodola’.

@item 
bpo-13952@footnote{https://bugs.python.org/issue13952}: Add .csv to mimetypes.types_map.  Patch by Geoff Wilson.

@item 
bpo-26709@footnote{https://bugs.python.org/issue26709}: Fixed Y2038 problem in loading binary PLists.

@item 
bpo-23735@footnote{https://bugs.python.org/issue23735}: Handle terminal resizing with Readline 6.3+ by installing our
own SIGWINCH handler.  Patch by Eric Price.

@item 
bpo-26586@footnote{https://bugs.python.org/issue26586}: In http.server, respond with “413 Request header fields too
large” if there are too many header fields to parse, rather than killing
the connection and raising an unhandled exception.  Patch by Xiang Zhang.

@item 
bpo-22854@footnote{https://bugs.python.org/issue22854}: Change BufferedReader.writable() and BufferedWriter.readable()
to always return False.

@item 
bpo-25195@footnote{https://bugs.python.org/issue25195}: Fix a regression in mock.MagicMock. _Call is a subclass of
tuple (changeset 3603bae63c13 only works for classes) so we need to
implement __ne__ ourselves.  Patch by Andrew Plummer.

@item 
bpo-26644@footnote{https://bugs.python.org/issue26644}: Raise ValueError rather than SystemError when a negative length
is passed to SSLSocket.recv() or read().

@item 
bpo-23804@footnote{https://bugs.python.org/issue23804}: Fix SSL recv(0) and read(0) methods to return zero bytes
instead of up to 1024.

@item 
bpo-26616@footnote{https://bugs.python.org/issue26616}: Fixed a bug in datetime.astimezone() method.

@item 
bpo-21925@footnote{https://bugs.python.org/issue21925}: @ref{12c0,,warnings.formatwarning()} now catches exceptions on
@code{linecache.getline(...)} to be able to log @ref{77b,,ResourceWarning}
emitted late during the Python shutdown process.

@item 
bpo-24266@footnote{https://bugs.python.org/issue24266}: Ctrl+C during Readline history search now cancels the search
mode when compiled with Readline 7.

@item 
bpo-26560@footnote{https://bugs.python.org/issue26560}: Avoid potential ValueError in BaseHandler.start_response.
Initial patch by Peter Inglesby.
@end itemize

@node Security<40>,Library<71>,Library<70>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id506}@anchor{1460}
@subsubsection Security



@itemize -

@item 
bpo-26313@footnote{https://bugs.python.org/issue26313}: ssl.py _load_windows_store_certs fails if windows cert store is
empty. Patch by Baji.
@end itemize

@node Library<71>,Security<41>,Security<40>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id507}@anchor{1461}
@subsubsection Library



@itemize -

@item 
bpo-26569@footnote{https://bugs.python.org/issue26569}: Fix @ref{142a,,pyclbr.readmodule()} and @ref{142b,,pyclbr.readmodule_ex()}
to support importing packages.

@item 
bpo-26499@footnote{https://bugs.python.org/issue26499}: Account for remaining Content-Length in HTTPResponse.readline()
and read1(). Based on patch by Silent Ghost. Also document that
HTTPResponse now supports these methods.

@item 
bpo-25320@footnote{https://bugs.python.org/issue25320}: Handle sockets in directories unittest discovery is scanning.
Patch from Victor van den Elzen.

@item 
bpo-16181@footnote{https://bugs.python.org/issue16181}: cookiejar.http2time() now returns None if year is higher than
datetime.MAXYEAR.

@item 
bpo-26513@footnote{https://bugs.python.org/issue26513}: Fixes platform module detection of Windows Server

@item 
bpo-23718@footnote{https://bugs.python.org/issue23718}: Fixed parsing time in week 0 before Jan 1.  Original patch by
Tamás Bence Gedai.

@item 
bpo-20589@footnote{https://bugs.python.org/issue20589}: Invoking Path.owner() and Path.group() on Windows now raise
NotImplementedError instead of ImportError.

@item 
bpo-26177@footnote{https://bugs.python.org/issue26177}: Fixed the keys() method for Canvas and Scrollbar widgets.

@item 
bpo-15068@footnote{https://bugs.python.org/issue15068}: Got rid of excessive buffering in the fileinput module. The
bufsize parameter is no longer used.

@item 
bpo-2202@footnote{https://bugs.python.org/issue2202}: Fix UnboundLocalError in
AbstractDigestAuthHandler.get_algorithm_impls. Initial patch by Mathieu
Dupuy.

@item 
bpo-25718@footnote{https://bugs.python.org/issue25718}: Fixed pickling and copying the accumulate() iterator with total
is None.

@item 
bpo-26475@footnote{https://bugs.python.org/issue26475}: Fixed debugging output for regular expressions with the (?x)
flag.

@item 
bpo-26457@footnote{https://bugs.python.org/issue26457}: Fixed the subnets() methods in IP network classes for the case
when resulting prefix length is equal to maximal prefix length. Based on
patch by Xiang Zhang.

@item 
bpo-26385@footnote{https://bugs.python.org/issue26385}: Remove the file if the internal open() call in
NamedTemporaryFile() fails. Patch by Silent Ghost.

@item 
bpo-26402@footnote{https://bugs.python.org/issue26402}: Fix XML-RPC client to retry when the server shuts down a
persistent connection.  This was a regression related to the new
http.client.RemoteDisconnected exception in 3.5.0a4.

@item 
bpo-25913@footnote{https://bugs.python.org/issue25913}: Leading @code{<~} is optional now in base64.a85decode() with
adobe=True.  Patch by Swati Jaiswal.

@item 
bpo-26186@footnote{https://bugs.python.org/issue26186}: Remove an invalid type check in importlib.util.LazyLoader.

@item 
bpo-26367@footnote{https://bugs.python.org/issue26367}: importlib.__import__() raises SystemError like
builtins.__import__() when @code{level} is specified but without an
accompanying package specified.

@item 
bpo-26309@footnote{https://bugs.python.org/issue26309}: In the “socketserver” module, shut down the request (closing
the connected socket) when verify_request() returns false.  Patch by Aviv
Palivoda.
@end itemize

@node Security<41>,Library<72>,Library<71>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id508}@anchor{1462}
@subsubsection Security



@itemize -

@item 
bpo-25939@footnote{https://bugs.python.org/issue25939}: On Windows open the cert store readonly in
ssl.enum_certificates.
@end itemize

@node Library<72>,IDLE<46>,Security<41>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id509}@anchor{1463}
@subsubsection Library



@itemize -

@item 
bpo-25995@footnote{https://bugs.python.org/issue25995}: os.walk() no longer uses FDs proportional to the tree depth.

@item 
bpo-26117@footnote{https://bugs.python.org/issue26117}: The os.scandir() iterator now closes file descriptor not only
when the iteration is finished, but when it was failed with error.

@item 
bpo-25911@footnote{https://bugs.python.org/issue25911}: Restored support of bytes paths in os.walk() on Windows.

@item 
bpo-26045@footnote{https://bugs.python.org/issue26045}: Add UTF-8 suggestion to error message when posting a
non-Latin-1 string with http.client.

@item 
bpo-12923@footnote{https://bugs.python.org/issue12923}: Reset FancyURLopener’s redirect counter even if there is an
exception. Based on patches by Brian Brazil and Daniel Rocco.

@item 
bpo-25945@footnote{https://bugs.python.org/issue25945}: Fixed a crash when unpickle the functools.partial object with
wrong state. Fixed a leak in failed functools.partial constructor. “args”
and “keywords” attributes of functools.partial have now always types tuple
and dict correspondingly.

@item 
bpo-26202@footnote{https://bugs.python.org/issue26202}: copy.deepcopy() now correctly copies range() objects with
non-atomic attributes.

@item 
bpo-23076@footnote{https://bugs.python.org/issue23076}: Path.glob() now raises a ValueError if it’s called with an
invalid pattern. Patch by Thomas Nyberg.

@item 
bpo-19883@footnote{https://bugs.python.org/issue19883}: Fixed possible integer overflows in zipimport.

@item 
bpo-26227@footnote{https://bugs.python.org/issue26227}: On Windows, getnameinfo(), gethostbyaddr() and
gethostbyname_ex() functions of the socket module now decode the hostname
from the ANSI code page rather than UTF-8.

@item 
bpo-26147@footnote{https://bugs.python.org/issue26147}: xmlrpc now works with strings not encodable with used non-UTF-8
encoding.

@item 
bpo-25935@footnote{https://bugs.python.org/issue25935}: Garbage collector now breaks reference loops with OrderedDict.

@item 
bpo-16620@footnote{https://bugs.python.org/issue16620}: Fixed AttributeError in msilib.Directory.glob().

@item 
bpo-26013@footnote{https://bugs.python.org/issue26013}: Added compatibility with broken protocol 2 pickles created in
old Python 3 versions (3.4.3 and lower).

@item 
bpo-25850@footnote{https://bugs.python.org/issue25850}: Use cross-compilation by default for 64-bit Windows.

@item 
bpo-17633@footnote{https://bugs.python.org/issue17633}: Improve zipimport’s support for namespace packages.

@item 
bpo-24705@footnote{https://bugs.python.org/issue24705}: Fix sysconfig._parse_makefile not expanding $@{@} vars appearing
before $() vars.

@item 
bpo-22138@footnote{https://bugs.python.org/issue22138}: Fix mock.patch behavior when patching descriptors. Restore
original values after patching. Patch contributed by Sean McCully.

@item 
bpo-25672@footnote{https://bugs.python.org/issue25672}: In the ssl module, enable the SSL_MODE_RELEASE_BUFFERS mode
option if it is safe to do so.

@item 
bpo-26012@footnote{https://bugs.python.org/issue26012}: Don’t traverse into symlinks for @code{**} pattern in
pathlib.Path.[r]glob().

@item 
bpo-24120@footnote{https://bugs.python.org/issue24120}: Ignore PermissionError when traversing a tree with
pathlib.Path.[r]glob(). Patch by Ulrich Petri.

@item 
bpo-25447@footnote{https://bugs.python.org/issue25447}: fileinput now uses sys.stdin as-is if it does not have a buffer
attribute (restores backward compatibility).

@item 
bpo-25447@footnote{https://bugs.python.org/issue25447}: Copying the lru_cache() wrapper object now always works,
independently from the type of the wrapped object (by returning the
original object unchanged).

@item 
bpo-24103@footnote{https://bugs.python.org/issue24103}: Fixed possible use after free in ElementTree.XMLPullParser.

@item 
bpo-25860@footnote{https://bugs.python.org/issue25860}: os.fwalk() no longer skips remaining directories when error
occurs. Original patch by Samson Lee.

@item 
bpo-25914@footnote{https://bugs.python.org/issue25914}: Fixed and simplified OrderedDict.__sizeof__.

@item 
bpo-25902@footnote{https://bugs.python.org/issue25902}: Fixed various refcount issues in ElementTree iteration.

@item 
bpo-25717@footnote{https://bugs.python.org/issue25717}: Restore the previous behaviour of tolerating most fstat()
errors when opening files.  This was a regression in 3.5a1, and stopped
anonymous temporary files from working in special cases.

@item 
bpo-24903@footnote{https://bugs.python.org/issue24903}: Fix regression in number of arguments compileall accepts when
‘-d’ is specified.  The check on the number of arguments has been dropped
completely as it never worked correctly anyway.

@item 
bpo-25764@footnote{https://bugs.python.org/issue25764}: In the subprocess module, preserve any exception caused by
fork() failure when preexec_fn is used.

@item 
bpo-6478@footnote{https://bugs.python.org/issue6478}: _strptime’s regexp cache now is reset after changing timezone
with time.tzset().

@item 
bpo-14285@footnote{https://bugs.python.org/issue14285}: When executing a package with the “python -m package” option,
and package initialization fails, a proper traceback is now reported.  The
“runpy” module now lets exceptions from package initialization pass back
to the caller, rather than raising ImportError.

@item 
bpo-19771@footnote{https://bugs.python.org/issue19771}: Also in runpy and the “-m” option, omit the irrelevant message
“… is a package and cannot be directly executed” if the package could
not even be initialized (e.g. due to a bad @code{*.pyc} file).

@item 
bpo-25177@footnote{https://bugs.python.org/issue25177}: Fixed problem with the mean of very small and very large
numbers. As a side effect, statistics.mean and statistics.variance should
be significantly faster.

@item 
bpo-25718@footnote{https://bugs.python.org/issue25718}: Fixed copying object with state with boolean value is false.

@item 
bpo-10131@footnote{https://bugs.python.org/issue10131}: Fixed deep copying of minidom documents.  Based on patch by
Marian Ganisin.

@item 
bpo-25725@footnote{https://bugs.python.org/issue25725}: Fixed a reference leak in pickle.loads() when unpickling
invalid data including tuple instructions.

@item 
bpo-25663@footnote{https://bugs.python.org/issue25663}: In the Readline completer, avoid listing duplicate global
names, and search the global namespace before searching builtins.

@item 
bpo-25688@footnote{https://bugs.python.org/issue25688}: Fixed file leak in ElementTree.iterparse() raising an error.

@item 
bpo-23914@footnote{https://bugs.python.org/issue23914}: Fixed SystemError raised by unpickler on broken pickle data.

@item 
bpo-25691@footnote{https://bugs.python.org/issue25691}: Fixed crash on deleting ElementTree.Element attributes.

@item 
bpo-25624@footnote{https://bugs.python.org/issue25624}: ZipFile now always writes a ZIP_STORED header for directory
entries.  Patch by Dingyuan Wang.

@item 
Skip getaddrinfo if host is already resolved. Patch by A. Jesse Jiryu
Davis.

@item 
bpo-26050@footnote{https://bugs.python.org/issue26050}: Add asyncio.StreamReader.readuntil() method. Patch by Марк
Коренберг.

@item 
bpo-25924@footnote{https://bugs.python.org/issue25924}: Avoid unnecessary serialization of getaddrinfo(3) calls on OS X
versions 10.5 or higher.  Original patch by A. Jesse Jiryu Davis.

@item 
bpo-26406@footnote{https://bugs.python.org/issue26406}: Avoid unnecessary serialization of getaddrinfo(3) calls on
current versions of OpenBSD and NetBSD.  Patch by A. Jesse Jiryu Davis.

@item 
bpo-26848@footnote{https://bugs.python.org/issue26848}: Fix asyncio/subprocess.communicate() to handle empty input.
Patch by Jack O’Connor.

@item 
bpo-27040@footnote{https://bugs.python.org/issue27040}: Add loop.get_exception_handler method

@item 
bpo-27041@footnote{https://bugs.python.org/issue27041}: asyncio: Add loop.create_future method

@item 
bpo-27223@footnote{https://bugs.python.org/issue27223}: asyncio: Fix _read_ready and _write_ready to respect
_conn_lost. Patch by Łukasz Langa.

@item 
bpo-22970@footnote{https://bugs.python.org/issue22970}: asyncio: Fix inconsistency cancelling Condition.wait. Patch by
David Coles.
@end itemize

@node IDLE<46>,Documentation<51>,Library<72>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id510}@anchor{1464}
@subsubsection IDLE



@itemize -

@item 
bpo-5124@footnote{https://bugs.python.org/issue5124}: Paste with text selected now replaces the selection on X11. This
matches how paste works on Windows, Mac, most modern Linux apps, and ttk
widgets. Original patch by Serhiy Storchaka.

@item 
bpo-24759@footnote{https://bugs.python.org/issue24759}: Make clear in idlelib.idle_test.__init__ that the directory is
a private implementation of test.test_idle and tool for maintainers.

@item 
bpo-27196@footnote{https://bugs.python.org/issue27196}: Stop ‘ThemeChanged’ warnings when running IDLE tests. These
persisted after other warnings were suppressed in #20567. Apply Serhiy
Storchaka’s update_idletasks solution to four test files. Record this
additional advice in idle_test/README.txt

@item 
bpo-20567@footnote{https://bugs.python.org/issue20567}: Revise idle_test/README.txt with advice about avoiding tk
warning messages from tests.  Apply advice to several IDLE tests.

@item 
bpo-27117@footnote{https://bugs.python.org/issue27117}: Make colorizer htest and turtledemo work with dark themes. Move
code for configuring text widget colors to a new function.

@item 
bpo-26673@footnote{https://bugs.python.org/issue26673}: When tk reports font size as 0, change to size 10. Such fonts
on Linux prevented the configuration dialog from opening.

@item 
bpo-21939@footnote{https://bugs.python.org/issue21939}: Add test for IDLE’s percolator. Original patch by Saimadhav
Heblikar.

@item 
bpo-21676@footnote{https://bugs.python.org/issue21676}: Add test for IDLE’s replace dialog. Original patch by Saimadhav
Heblikar.

@item 
bpo-18410@footnote{https://bugs.python.org/issue18410}: Add test for IDLE’s search dialog. Original patch by Westley
Martínez.

@item 
bpo-21703@footnote{https://bugs.python.org/issue21703}: Add test for IDLE’s undo delegator. Original patch by Saimadhav
Heblikar .

@item 
bpo-27044@footnote{https://bugs.python.org/issue27044}: Add ConfigDialog.remove_var_callbacks to stop memory leaks.

@item 
bpo-23977@footnote{https://bugs.python.org/issue23977}: Add more asserts to test_delegator.

@item 
bpo-20640@footnote{https://bugs.python.org/issue20640}: Add tests for idlelib.configHelpSourceEdit. Patch by Saimadhav
Heblikar.

@item 
In the ‘IDLE-console differences’ section of the IDLE doc, clarify how
running with IDLE affects sys.modules and the standard streams.

@item 
bpo-25507@footnote{https://bugs.python.org/issue25507}: fix incorrect change in IOBinding that prevented printing.
Augment IOBinding htest to include all major IOBinding functions.

@item 
bpo-25905@footnote{https://bugs.python.org/issue25905}: Revert unwanted conversion of ‘ to ’ RIGHT SINGLE QUOTATION
MARK in README.txt and open this and NEWS.txt with ‘ascii’. Re-encode
CREDITS.txt to utf-8 and open it with ‘utf-8’.
@end itemize

@node Documentation<51>,Tests<52>,IDLE<46>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id511}@anchor{1465}
@subsubsection Documentation



@itemize -

@item 
bpo-19489@footnote{https://bugs.python.org/issue19489}: Moved the search box from the sidebar to the header and footer
of each page. Patch by Ammar Askar.

@item 
bpo-24136@footnote{https://bugs.python.org/issue24136}: Document the new 
@geindex Python Enhancement Proposals; PEP 448
PEP 448@footnote{https://www.python.org/dev/peps/pep-0448} unpacking syntax of 3.5.

@item 
bpo-26736@footnote{https://bugs.python.org/issue26736}: Used HTTPS for external links in the documentation if possible.

@item 
bpo-6953@footnote{https://bugs.python.org/issue6953}: Rework the Readline module documentation to group related
functions together, and add more details such as what underlying Readline
functions and variables are accessed.

@item 
bpo-23606@footnote{https://bugs.python.org/issue23606}: Adds note to ctypes documentation regarding cdll.msvcrt.

@item 
bpo-25500@footnote{https://bugs.python.org/issue25500}: Fix documentation to not claim that __import__ is searched for
in the global scope.

@item 
bpo-26014@footnote{https://bugs.python.org/issue26014}: Update 3.x packaging documentation: * “See also” links to the
new docs are now provided in the legacy pages * links to setuptools
documentation have been updated
@end itemize

@node Tests<52>,Build<55>,Documentation<51>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id512}@anchor{1466}
@subsubsection Tests



@itemize -

@item 
bpo-21916@footnote{https://bugs.python.org/issue21916}: Added tests for the turtle module.  Patch by ingrid, Gregory
Loyse and Jelle Zijlstra.

@item 
bpo-26523@footnote{https://bugs.python.org/issue26523}: The multiprocessing thread pool (multiprocessing.dummy.Pool)
was untested.

@item 
bpo-26015@footnote{https://bugs.python.org/issue26015}: Added new tests for pickling iterators of mutable sequences.

@item 
bpo-26325@footnote{https://bugs.python.org/issue26325}: Added test.support.check_no_resource_warning() to check that no
ResourceWarning is emitted.

@item 
bpo-25940@footnote{https://bugs.python.org/issue25940}: Changed test_ssl to use self-signed.pythontest.net.  This
avoids relying on svn.python.org, which recently changed root certificate.

@item 
bpo-25616@footnote{https://bugs.python.org/issue25616}: Tests for OrderedDict are extracted from test_collections into
separate file test_ordered_dict.

@item 
bpo-26583@footnote{https://bugs.python.org/issue26583}: Skip test_timestamp_overflow in test_import if bytecode files
cannot be written.
@end itemize

@node Build<55>,Windows<49>,Tests<52>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id513}@anchor{1467}
@subsubsection Build



@itemize -

@item 
bpo-26884@footnote{https://bugs.python.org/issue26884}: Fix linking extension modules for cross builds. Patch by Xavier
de Gaye.

@item 
bpo-22359@footnote{https://bugs.python.org/issue22359}: Disable the rules for running _freeze_importlib and pgen when
cross-compiling.  The output of these programs is normally saved with the
source code anyway, and is still regenerated when doing a native build.
Patch by Xavier de Gaye.

@item 
bpo-27229@footnote{https://bugs.python.org/issue27229}: Fix the cross-compiling pgen rule for in-tree builds.  Patch by
Xavier de Gaye.

@item 
bpo-21668@footnote{https://bugs.python.org/issue21668}: Link audioop, _datetime, _ctypes_test modules to libm, except
on Mac OS X. Patch written by Xavier de Gaye.

@item 
bpo-25702@footnote{https://bugs.python.org/issue25702}: A –with-lto configure option has been added that will enable
link time optimizations at build time during a make profile-opt. Some
compilers and toolchains are known to not produce stable code when using
LTO, be sure to test things thoroughly before relying on it. It can
provide a few % speed up over profile-opt alone.

@item 
bpo-26624@footnote{https://bugs.python.org/issue26624}: Adds validation of ucrtbase[d].dll version with warning for old
versions.

@item 
bpo-17603@footnote{https://bugs.python.org/issue17603}: Avoid error about nonexistent fileblocks.o file by using a
lower-level check for st_blocks in struct stat.

@item 
bpo-26079@footnote{https://bugs.python.org/issue26079}: Fixing the build output folder for tix-8.4.3.6. Patch by Bjoern
Thiel.

@item 
bpo-26465@footnote{https://bugs.python.org/issue26465}: Update Windows builds to use OpenSSL 1.0.2g.

@item 
bpo-24421@footnote{https://bugs.python.org/issue24421}: Compile Modules/_math.c once, before building extensions.
Previously it could fail to compile properly if the math and cmath builds
were concurrent.

@item 
bpo-25348@footnote{https://bugs.python.org/issue25348}: Added @code{--pgo} and @code{--pgo-job} arguments to
@code{PCbuild\build.bat} for building with Profile-Guided Optimization.  The
old @code{PCbuild\build_pgo.bat} script is now deprecated, and simply calls
@code{PCbuild\build.bat --pgo %*}.

@item 
bpo-25827@footnote{https://bugs.python.org/issue25827}: Add support for building with ICC to @code{configure}, including a
new @code{--with-icc} flag.

@item 
bpo-25696@footnote{https://bugs.python.org/issue25696}: Fix installation of Python on UNIX with make -j9.

@item 
bpo-26930@footnote{https://bugs.python.org/issue26930}: Update OS X 10.5+ 32-bit-only installer to build and link with
OpenSSL 1.0.2h.

@item 
bpo-26268@footnote{https://bugs.python.org/issue26268}: Update Windows builds to use OpenSSL 1.0.2f.

@item 
bpo-25136@footnote{https://bugs.python.org/issue25136}: Support Apple Xcode 7’s new textual SDK stub libraries.

@item 
bpo-24324@footnote{https://bugs.python.org/issue24324}: Do not enable unreachable code warnings when using gcc as the
option does not work correctly in older versions of gcc and has been
silently removed as of gcc-4.5.
@end itemize

@node Windows<49>,Tools/Demos<30>,Build<55>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id514}@anchor{1468}
@subsubsection Windows



@itemize -

@item 
bpo-27053@footnote{https://bugs.python.org/issue27053}: Updates make_zip.py to correctly generate library ZIP file.

@item 
bpo-26268@footnote{https://bugs.python.org/issue26268}: Update the prepare_ssl.py script to handle OpenSSL releases
that don’t include the contents of the include directory (that is, 1.0.2e
and later).

@item 
bpo-26071@footnote{https://bugs.python.org/issue26071}: bdist_wininst created binaries fail to start and find 32bit
Python

@item 
bpo-26073@footnote{https://bugs.python.org/issue26073}: Update the list of magic numbers in launcher

@item 
bpo-26065@footnote{https://bugs.python.org/issue26065}: Excludes venv from library when generating embeddable distro.
@end itemize

@node Tools/Demos<30>,Windows<50>,Windows<49>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id515}@anchor{1469}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-26799@footnote{https://bugs.python.org/issue26799}: Fix python-gdb.py: don’t get C types once when the Python code
is loaded, but get C types on demand. The C types can change if
python-gdb.py is loaded before the Python executable. Patch written by
Thomas Ilsche.

@item 
bpo-26271@footnote{https://bugs.python.org/issue26271}: Fix the Freeze tool to properly use flags passed through
configure. Patch by Daniel Shaulov.

@item 
bpo-26489@footnote{https://bugs.python.org/issue26489}: Add dictionary unpacking support to Tools/parser/unparse.py.
Patch by Guo Ci Teo.

@item 
bpo-26316@footnote{https://bugs.python.org/issue26316}: Fix variable name typo in Argument Clinic.
@end itemize

@node Windows<50>,,Tools/Demos<30>,Python 3 5 2 release candidate 1
@anchor{whatsnew/changelog id516}@anchor{146a}
@subsubsection Windows



@itemize -

@item 
bpo-17500@footnote{https://bugs.python.org/issue17500}: Remove unused and outdated icons. (See also:
@indicateurl{https://github.com/python/pythondotorg/issues/945})
@end itemize

@node Python 3 5 1 final,Python 3 5 1 release candidate 1,Python 3 5 2 release candidate 1,Changelog
@anchor{whatsnew/changelog python-3-5-1-final}@anchor{146b}
@subsection Python 3.5.1 final


@emph{Release date: 2015-12-06}

@menu
* Core and Builtins: Core and Builtins<59>. 
* Windows: Windows<51>. 

@end menu

@node Core and Builtins<59>,Windows<51>,,Python 3 5 1 final
@anchor{whatsnew/changelog id517}@anchor{146c}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-25709@footnote{https://bugs.python.org/issue25709}: Fixed problem with in-place string concatenation and utf-8
cache.
@end itemize

@node Windows<51>,,Core and Builtins<59>,Python 3 5 1 final
@anchor{whatsnew/changelog id518}@anchor{146d}
@subsubsection Windows



@itemize -

@item 
bpo-25715@footnote{https://bugs.python.org/issue25715}: Python 3.5.1 installer shows wrong upgrade path and incorrect
logic for launcher detection.
@end itemize

@node Python 3 5 1 release candidate 1,Python 3 5 0 final,Python 3 5 1 final,Changelog
@anchor{whatsnew/changelog python-3-5-1-release-candidate-1}@anchor{146e}
@subsection Python 3.5.1 release candidate 1


@emph{Release date: 2015-11-22}

@menu
* Core and Builtins: Core and Builtins<60>. 
* Library: Library<73>. 
* IDLE: IDLE<47>. 
* Documentation: Documentation<52>. 
* Tests: Tests<53>. 
* Build: Build<56>. 
* Windows: Windows<52>. 
* Tools/Demos: Tools/Demos<31>. 

@end menu

@node Core and Builtins<60>,Library<73>,,Python 3 5 1 release candidate 1
@anchor{whatsnew/changelog id519}@anchor{146f}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-25630@footnote{https://bugs.python.org/issue25630}: Fix a possible segfault during argument parsing in functions
that accept filesystem paths.

@item 
bpo-23564@footnote{https://bugs.python.org/issue23564}: Fixed a partially broken sanity check in the _posixsubprocess
internals regarding how fds_to_pass were passed to the child.  The bug had
no actual impact as subprocess.py already avoided it.

@item 
bpo-25388@footnote{https://bugs.python.org/issue25388}: Fixed tokenizer crash when processing undecodable source code
with a null byte.

@item 
bpo-25462@footnote{https://bugs.python.org/issue25462}: The hash of the key now is calculated only once in most
operations in C implementation of OrderedDict.

@item 
bpo-22995@footnote{https://bugs.python.org/issue22995}: Default implementation of __reduce__ and __reduce_ex__ now
rejects builtin types with not defined __new__.

@item 
bpo-25555@footnote{https://bugs.python.org/issue25555}: Fix parser and AST: fill lineno and col_offset of “arg” node
when compiling AST from Python objects.

@item 
bpo-24802@footnote{https://bugs.python.org/issue24802}: Avoid buffer overreads when int(), float(), compile(), exec()
and eval() are passed bytes-like objects.  These objects are not
necessarily terminated by a null byte, but the functions assumed they
were.

@item 
bpo-24726@footnote{https://bugs.python.org/issue24726}: Fixed a crash and leaking NULL in repr() of OrderedDict that
was mutated by direct calls of dict methods.

@item 
bpo-25449@footnote{https://bugs.python.org/issue25449}: Iterating OrderedDict with keys with unstable hash now raises
KeyError in C implementations as well as in Python implementation.

@item 
bpo-25395@footnote{https://bugs.python.org/issue25395}: Fixed crash when highly nested OrderedDict structures were
garbage collected.

@item 
bpo-25274@footnote{https://bugs.python.org/issue25274}: sys.setrecursionlimit() now raises a RecursionError if the new
recursion limit is too low depending at the current recursion depth.
Modify also the “lower-water mark” formula to make it monotonic. This mark
is used to decide when the overflowed flag of the thread state is reset.

@item 
bpo-24402@footnote{https://bugs.python.org/issue24402}: Fix input() to prompt to the redirected stdout when
sys.stdout.fileno() fails.

@item 
bpo-24806@footnote{https://bugs.python.org/issue24806}: Prevent builtin types that are not allowed to be subclassed
from being subclassed through multiple inheritance.

@item 
bpo-24848@footnote{https://bugs.python.org/issue24848}: Fixed a number of bugs in UTF-7 decoding of misformed data.

@item 
bpo-25280@footnote{https://bugs.python.org/issue25280}: Import trace messages emitted in verbose (-v) mode are no
longer formatted twice.

@item 
bpo-25003@footnote{https://bugs.python.org/issue25003}: On Solaris 11.3 or newer, os.urandom() now uses the getrandom()
function instead of the getentropy() function. The getentropy() function
is blocking to generate very good quality entropy, os.urandom() doesn’t
need such high-quality entropy.

@item 
bpo-25182@footnote{https://bugs.python.org/issue25182}: The stdprinter (used as sys.stderr before the io module is
imported at startup) now uses the backslashreplace error handler.

@item 
bpo-25131@footnote{https://bugs.python.org/issue25131}: Make the line number and column offset of set/dict literals and
comprehensions correspond to the opening brace.

@item 
bpo-25150@footnote{https://bugs.python.org/issue25150}: Hide the private _Py_atomic_xxx symbols from the public
Python.h header to fix a compilation error with OpenMP.
PyThreadState_GET() becomes an alias to PyThreadState_Get() to avoid ABI
incompatibilities.
@end itemize

@node Library<73>,IDLE<47>,Core and Builtins<60>,Python 3 5 1 release candidate 1
@anchor{whatsnew/changelog id520}@anchor{1470}
@subsubsection Library



@itemize -

@item 
bpo-25626@footnote{https://bugs.python.org/issue25626}: Change three zlib functions to accept sizes that fit in
Py_ssize_t, but internally cap those sizes to UINT_MAX.  This resolves a
regression in 3.5 where GzipFile.read() failed to read chunks larger than
2 or 4 GiB.  The change affects the zlib.Decompress.decompress()
max_length parameter, the zlib.decompress() bufsize parameter, and the
zlib.Decompress.flush() length parameter.

@item 
bpo-25583@footnote{https://bugs.python.org/issue25583}: Avoid incorrect errors raised by os.makedirs(exist_ok=True)
when the OS gives priority to errors such as EACCES over EEXIST.

@item 
bpo-25593@footnote{https://bugs.python.org/issue25593}: Change semantics of EventLoop.stop() in asyncio.

@item 
bpo-6973@footnote{https://bugs.python.org/issue6973}: When we know a subprocess.Popen process has died, do not allow
the send_signal(), terminate(), or kill() methods to do anything as they
could potentially signal a different process.

@item 
bpo-25590@footnote{https://bugs.python.org/issue25590}: In the Readline completer, only call getattr() once per
attribute.

@item 
bpo-25498@footnote{https://bugs.python.org/issue25498}: Fix a crash when garbage-collecting ctypes objects created by
wrapping a memoryview.  This was a regression made in 3.5a1.  Based on
patch by Eryksun.

@item 
bpo-25584@footnote{https://bugs.python.org/issue25584}: Added “escape” to the __all__ list in the glob module.

@item 
bpo-25584@footnote{https://bugs.python.org/issue25584}: Fixed recursive glob() with patterns starting with @code{**}.

@item 
bpo-25446@footnote{https://bugs.python.org/issue25446}: Fix regression in smtplib’s AUTH LOGIN support.

@item 
bpo-18010@footnote{https://bugs.python.org/issue18010}: Fix the pydoc web server’s module search function to handle
exceptions from importing packages.

@item 
bpo-25554@footnote{https://bugs.python.org/issue25554}: Got rid of circular references in regular expression parsing.

@item 
bpo-25510@footnote{https://bugs.python.org/issue25510}: fileinput.FileInput.readline() now returns b’’ instead of ‘’ at
the end if the FileInput was opened with binary mode. Patch by Ryosuke
Ito.

@item 
bpo-25503@footnote{https://bugs.python.org/issue25503}: Fixed inspect.getdoc() for inherited docstrings of properties.
Original patch by John Mark Vandenberg.

@item 
bpo-25515@footnote{https://bugs.python.org/issue25515}: Always use os.urandom as a source of randomness in uuid.uuid4.

@item 
bpo-21827@footnote{https://bugs.python.org/issue21827}: Fixed textwrap.dedent() for the case when largest common
whitespace is a substring of smallest leading whitespace. Based on patch
by Robert Li.

@item 
bpo-25447@footnote{https://bugs.python.org/issue25447}: The lru_cache() wrapper objects now can be copied and pickled
(by returning the original object unchanged).

@item 
bpo-25390@footnote{https://bugs.python.org/issue25390}: typing: Don’t crash on Union[str, Pattern].

@item 
bpo-25441@footnote{https://bugs.python.org/issue25441}: asyncio: Raise error from drain() when socket is closed.

@item 
bpo-25410@footnote{https://bugs.python.org/issue25410}: Cleaned up and fixed minor bugs in C implementation of
OrderedDict.

@item 
bpo-25411@footnote{https://bugs.python.org/issue25411}: Improved Unicode support in SMTPHandler through better use of
the email package. Thanks to user simon04 for the patch.

@item 
bpo-25407@footnote{https://bugs.python.org/issue25407}: Remove mentions of the formatter module being removed in Python
3.6.

@item 
bpo-25406@footnote{https://bugs.python.org/issue25406}: Fixed a bug in C implementation of OrderedDict.move_to_end()
that caused segmentation fault or hang in iterating after moving several
items to the start of ordered dict.

@item 
bpo-25364@footnote{https://bugs.python.org/issue25364}: zipfile now works in threads disabled builds.

@item 
bpo-25328@footnote{https://bugs.python.org/issue25328}: smtpd’s SMTPChannel now correctly raises a ValueError if both
decode_data and enable_SMTPUTF8 are set to true.

@item 
bpo-25316@footnote{https://bugs.python.org/issue25316}: distutils raises OSError instead of DistutilsPlatformError when
MSVC is not installed.

@item 
bpo-25380@footnote{https://bugs.python.org/issue25380}: Fixed protocol for the STACK_GLOBAL opcode in
pickletools.opcodes.

@item 
bpo-23972@footnote{https://bugs.python.org/issue23972}: Updates asyncio datagram create method allowing reuseport and
reuseaddr socket options to be set prior to binding the socket. Mirroring
the existing asyncio create_server method the reuseaddr option for
datagram sockets defaults to True if the O/S is ‘posix’ (except if the
platform is Cygwin). Patch by Chris Laws.

@item 
bpo-25304@footnote{https://bugs.python.org/issue25304}: Add asyncio.run_coroutine_threadsafe().  This lets you submit a
coroutine to a loop from another thread, returning a
concurrent.futures.Future.  By Vincent Michel.

@item 
bpo-25232@footnote{https://bugs.python.org/issue25232}: Fix CGIRequestHandler to split the query from the URL at the
first question mark (?) rather than the last. Patch from Xiang Zhang.

@item 
bpo-24657@footnote{https://bugs.python.org/issue24657}: Prevent CGIRequestHandler from collapsing slashes in the query
part of the URL as if it were a path. Patch from Xiang Zhang.

@item 
bpo-24483@footnote{https://bugs.python.org/issue24483}: C implementation of functools.lru_cache() now calculates key’s
hash only once.

@item 
bpo-22958@footnote{https://bugs.python.org/issue22958}: Constructor and update method of weakref.WeakValueDictionary
now accept the self and the dict keyword arguments.

@item 
bpo-22609@footnote{https://bugs.python.org/issue22609}: Constructor of collections.UserDict now accepts the self
keyword argument.

@item 
bpo-25111@footnote{https://bugs.python.org/issue25111}: Fixed comparison of traceback.FrameSummary.

@item 
bpo-25262@footnote{https://bugs.python.org/issue25262}: Added support for BINBYTES8 opcode in Python implementation of
unpickler. Highest 32 bits of 64-bit size for BINUNICODE8 and BINBYTES8
opcodes no longer silently ignored on 32-bit platforms in C
implementation.

@item 
bpo-25034@footnote{https://bugs.python.org/issue25034}: Fix string.Formatter problem with auto-numbering and nested
format_specs. Patch by Anthon van der Neut.

@item 
bpo-25233@footnote{https://bugs.python.org/issue25233}: Rewrite the guts of asyncio.Queue and asyncio.Semaphore to be
more understandable and correct.

@item 
bpo-25203@footnote{https://bugs.python.org/issue25203}: Failed readline.set_completer_delims() no longer left the
module in inconsistent state.

@item 
bpo-23600@footnote{https://bugs.python.org/issue23600}: Default implementation of tzinfo.fromutc() was returning wrong
results in some cases.

@item 
bpo-23329@footnote{https://bugs.python.org/issue23329}: Allow the ssl module to be built with older versions of
LibreSSL.

@item 
Prevent overflow in _Unpickler_Read.

@item 
bpo-25047@footnote{https://bugs.python.org/issue25047}: The XML encoding declaration written by Element Tree now
respects the letter case given by the user. This restores the ability to
write encoding names in uppercase like “UTF-8”, which worked in Python 2.

@item 
bpo-25135@footnote{https://bugs.python.org/issue25135}: Make deque_clear() safer by emptying the deque before clearing.
This helps avoid possible reentrancy issues.

@item 
bpo-19143@footnote{https://bugs.python.org/issue19143}: platform module now reads Windows version from kernel32.dll to
avoid compatibility shims.

@item 
bpo-25092@footnote{https://bugs.python.org/issue25092}: Fix datetime.strftime() failure when errno was already set to
EINVAL.

@item 
bpo-23517@footnote{https://bugs.python.org/issue23517}: Fix rounding in fromtimestamp() and utcfromtimestamp() methods
of datetime.datetime: microseconds are now rounded to nearest with ties
going to nearest even integer (ROUND_HALF_EVEN), instead of being rounding
towards minus infinity (ROUND_FLOOR). It’s important that these methods
use the same rounding mode than datetime.timedelta to keep the property:
(datetime(1970,1,1) + timedelta(seconds=t)) ==
datetime.utcfromtimestamp(t). It also the rounding mode used by
round(float) for example.

@item 
bpo-25155@footnote{https://bugs.python.org/issue25155}: Fix datetime.datetime.now() and datetime.datetime.utcnow() on
Windows to support date after year 2038. It was a regression introduced in
Python 3.5.0.

@item 
bpo-25108@footnote{https://bugs.python.org/issue25108}: Omitted internal frames in traceback functions print_stack(),
format_stack(), and extract_stack() called without arguments.

@item 
bpo-25118@footnote{https://bugs.python.org/issue25118}: Fix a regression of Python 3.5.0 in os.waitpid() on Windows.

@item 
bpo-24684@footnote{https://bugs.python.org/issue24684}: socket.socket.getaddrinfo() now calls
PyUnicode_AsEncodedString() instead of calling the encode() method of the
host, to handle correctly custom string with an encode() method which
doesn’t return a byte string. The encoder of the IDNA codec is now called
directly instead of calling the encode() method of the string.

@item 
bpo-25060@footnote{https://bugs.python.org/issue25060}: Correctly compute stack usage of the BUILD_MAP opcode.

@item 
bpo-24857@footnote{https://bugs.python.org/issue24857}: Comparing call_args to a long sequence now correctly returns a
boolean result instead of raising an exception.  Patch by A Kaptur.

@item 
bpo-23144@footnote{https://bugs.python.org/issue23144}: Make sure that HTMLParser.feed() returns all the data, even
when convert_charrefs is True.

@item 
bpo-24982@footnote{https://bugs.python.org/issue24982}: shutil.make_archive() with the “zip” format now adds entries
for directories (including empty directories) in ZIP file.

@item 
bpo-25019@footnote{https://bugs.python.org/issue25019}: Fixed a crash caused by setting non-string key of expat parser.
Based on patch by John Leitch.

@item 
bpo-16180@footnote{https://bugs.python.org/issue16180}: Exit pdb if file has syntax error, instead of trapping user in
an infinite loop.  Patch by Xavier de Gaye.

@item 
bpo-24891@footnote{https://bugs.python.org/issue24891}: Fix a race condition at Python startup if the file descriptor
of stdin (0), stdout (1) or stderr (2) is closed while Python is creating
sys.stdin, sys.stdout and sys.stderr objects. These attributes are now set
to None if the creation of the object failed, instead of raising an
OSError exception. Initial patch written by Marco Paolini.

@item 
bpo-24992@footnote{https://bugs.python.org/issue24992}: Fix error handling and a race condition (related to garbage
collection) in collections.OrderedDict constructor.

@item 
bpo-24881@footnote{https://bugs.python.org/issue24881}: Fixed setting binary mode in Python implementation of FileIO on
Windows and Cygwin.  Patch from Akira Li.

@item 
bpo-25578@footnote{https://bugs.python.org/issue25578}: Fix (another) memory leak in SSLSocket.getpeercer().

@item 
bpo-25530@footnote{https://bugs.python.org/issue25530}: Disable the vulnerable SSLv3 protocol by default when creating
ssl.SSLContext.

@item 
bpo-25569@footnote{https://bugs.python.org/issue25569}: Fix memory leak in SSLSocket.getpeercert().

@item 
bpo-25471@footnote{https://bugs.python.org/issue25471}: Sockets returned from accept() shouldn’t appear to be
nonblocking.

@item 
bpo-25319@footnote{https://bugs.python.org/issue25319}: When threading.Event is reinitialized, the underlying condition
should use a regular lock rather than a recursive lock.

@item 
bpo-21112@footnote{https://bugs.python.org/issue21112}: Fix regression in unittest.expectedFailure on subclasses. Patch
from Berker Peksag.

@item 
bpo-24764@footnote{https://bugs.python.org/issue24764}: cgi.FieldStorage.read_multi() now ignores the Content-Length
header in part headers. Patch written by Peter Landry and reviewed by
Pierre Quentel.

@item 
bpo-24913@footnote{https://bugs.python.org/issue24913}: Fix overrun error in deque.index(). Found by John Leitch and
Bryce Darling.

@item 
bpo-24774@footnote{https://bugs.python.org/issue24774}: Fix docstring in http.server.test. Patch from Chiu-Hsiang Hsu.

@item 
bpo-21159@footnote{https://bugs.python.org/issue21159}: Improve message in
configparser.InterpolationMissingOptionError. Patch from Łukasz Langa.

@item 
bpo-20362@footnote{https://bugs.python.org/issue20362}: Honour TestCase.longMessage correctly in assertRegex. Patch
from Ilia Kurenkov.

@item 
bpo-23572@footnote{https://bugs.python.org/issue23572}: Fixed functools.singledispatch on classes with falsy
metaclasses.  Patch by Ethan Furman.

@item 
asyncio: ensure_future() now accepts awaitable objects.
@end itemize

@node IDLE<47>,Documentation<52>,Library<73>,Python 3 5 1 release candidate 1
@anchor{whatsnew/changelog id521}@anchor{1471}
@subsubsection IDLE



@itemize -

@item 
bpo-15348@footnote{https://bugs.python.org/issue15348}: Stop the debugger engine (normally in a user process) before
closing the debugger window (running in the IDLE process). This prevents
the RuntimeErrors that were being caught and ignored.

@item 
bpo-24455@footnote{https://bugs.python.org/issue24455}: Prevent IDLE from hanging when a) closing the shell while the
debugger is active (15347); b) closing the debugger with the [X] button
(15348); and c) activating the debugger when already active (24455). The
patch by Mark Roseman does this by making two changes. 1. Suspend and
resume the gui.interaction method with the tcl vwait mechanism intended
for this purpose (instead of root.mainloop & .quit). 2. In gui.run, allow
any existing interaction to terminate first.

@item 
Change ‘The program’ to ‘Your program’ in an IDLE ‘kill program?’ message
to make it clearer that the program referred to is the currently running
user program, not IDLE itself.

@item 
bpo-24750@footnote{https://bugs.python.org/issue24750}: Improve the appearance of the IDLE editor window status bar.
Patch by Mark Roseman.

@item 
bpo-25313@footnote{https://bugs.python.org/issue25313}: Change the handling of new built-in text color themes to better
address the compatibility problem introduced by the addition of IDLE Dark.
Consistently use the revised idleConf.CurrentTheme everywhere in idlelib.

@item 
bpo-24782@footnote{https://bugs.python.org/issue24782}: Extension configuration is now a tab in the IDLE Preferences
dialog rather than a separate dialog.  The former tabs are now a sorted
list.  Patch by Mark Roseman.

@item 
bpo-22726@footnote{https://bugs.python.org/issue22726}: Re-activate the config dialog help button with some content
about the other buttons and the new IDLE Dark theme.

@item 
bpo-24820@footnote{https://bugs.python.org/issue24820}: IDLE now has an ‘IDLE Dark’ built-in text color theme. It is
more or less IDLE Classic inverted, with a cobalt blue background.
Strings, comments, keywords, … are still green, red, orange, … . To
use it with IDLEs released before November 2015, hit the ‘Save as New
Custom Theme’ button and enter a new name, such as ‘Custom Dark’.  The
custom theme will work with any IDLE release, and can be modified.

@item 
bpo-25224@footnote{https://bugs.python.org/issue25224}: README.txt is now an idlelib index for IDLE developers and
curious users. The previous user content is now in the IDLE doc chapter.
‘IDLE’ now means ‘Integrated Development and Learning Environment’.

@item 
bpo-24820@footnote{https://bugs.python.org/issue24820}: Users can now set breakpoint colors in Settings -> Custom
Highlighting. Original patch by Mark Roseman.

@item 
bpo-24972@footnote{https://bugs.python.org/issue24972}: Inactive selection background now matches active selection
background, as configured by users, on all systems.  Found items are now
always highlighted on Windows.  Initial patch by Mark Roseman.

@item 
bpo-24570@footnote{https://bugs.python.org/issue24570}: Idle: make calltip and completion boxes appear on Macs affected
by a tk regression.  Initial patch by Mark Roseman.

@item 
bpo-24988@footnote{https://bugs.python.org/issue24988}: Idle ScrolledList context menus (used in debugger) now work on
Mac Aqua. Patch by Mark Roseman.

@item 
bpo-24801@footnote{https://bugs.python.org/issue24801}: Make right-click for context menu work on Mac Aqua. Patch by
Mark Roseman.

@item 
bpo-25173@footnote{https://bugs.python.org/issue25173}: Associate tkinter messageboxes with a specific widget. For Mac
OSX, make them a ‘sheet’.  Patch by Mark Roseman.

@item 
bpo-25198@footnote{https://bugs.python.org/issue25198}: Enhance the initial html viewer now used for Idle Help.
Properly indent fixed-pitch text (patch by Mark Roseman). Give code
snippet a very Sphinx-like light blueish-gray background. Re-use initial
width and height set by users for shell and editor. When the Table of
Contents (TOC) menu is used, put the section header at the top of the
screen.

@item 
bpo-25225@footnote{https://bugs.python.org/issue25225}: Condense and rewrite Idle doc section on text colors.

@item 
bpo-21995@footnote{https://bugs.python.org/issue21995}: Explain some differences between IDLE and console Python.

@item 
bpo-22820@footnote{https://bugs.python.org/issue22820}: Explain need for @emph{print} when running file from Idle editor.

@item 
bpo-25224@footnote{https://bugs.python.org/issue25224}: Doc: augment Idle feature list and no-subprocess section.

@item 
bpo-25219@footnote{https://bugs.python.org/issue25219}: Update doc for Idle command line options. Some were missing and
notes were not correct.

@item 
bpo-24861@footnote{https://bugs.python.org/issue24861}: Most of idlelib is private and subject to change. Use
idleib.idle.* to start Idle. See idlelib.__init__.__doc__.

@item 
bpo-25199@footnote{https://bugs.python.org/issue25199}: Idle: add synchronization comments for future maintainers.

@item 
bpo-16893@footnote{https://bugs.python.org/issue16893}: Replace help.txt with help.html for Idle doc display. The new
idlelib/help.html is rstripped Doc/build/html/library/idle.html. It looks
better than help.txt and will better document Idle as released. The
tkinter html viewer that works for this file was written by Mark Roseman.
The now unused EditorWindow.HelpDialog class and helt.txt file are
deprecated.

@item 
bpo-24199@footnote{https://bugs.python.org/issue24199}: Deprecate unused idlelib.idlever with possible removal in 3.6.

@item 
bpo-24790@footnote{https://bugs.python.org/issue24790}: Remove extraneous code (which also create 2 & 3 conflicts).
@end itemize

@node Documentation<52>,Tests<53>,IDLE<47>,Python 3 5 1 release candidate 1
@anchor{whatsnew/changelog id522}@anchor{1472}
@subsubsection Documentation



@itemize -

@item 
bpo-22558@footnote{https://bugs.python.org/issue22558}: Add remaining doc links to source code for Python-coded
modules. Patch by Yoni Lavi.

@item 
bpo-12067@footnote{https://bugs.python.org/issue12067}: Rewrite Comparisons section in the Expressions chapter of the
language reference. Some of the details of comparing mixed types were
incorrect or ambiguous. NotImplemented is only relevant at a lower level
than the Expressions chapter. Added details of comparing range() objects,
and default behaviour and consistency suggestions for user-defined
classes. Patch from Andy Maier.

@item 
bpo-24952@footnote{https://bugs.python.org/issue24952}: Clarify the default size argument of stack_size() in the
“threading” and “_thread” modules. Patch from Mattip.

@item 
bpo-23725@footnote{https://bugs.python.org/issue23725}: Overhaul tempfile docs. Note deprecated status of mktemp. Patch
from Zbigniew Jędrzejewski-Szmek.

@item 
bpo-24808@footnote{https://bugs.python.org/issue24808}: Update the types of some PyTypeObject fields.  Patch by Joseph
Weston.

@item 
bpo-22812@footnote{https://bugs.python.org/issue22812}: Fix unittest discovery examples. Patch from Pam McA’Nulty.
@end itemize

@node Tests<53>,Build<56>,Documentation<52>,Python 3 5 1 release candidate 1
@anchor{whatsnew/changelog id523}@anchor{1473}
@subsubsection Tests



@itemize -

@item 
bpo-25449@footnote{https://bugs.python.org/issue25449}: Added tests for OrderedDict subclasses.

@item 
bpo-25099@footnote{https://bugs.python.org/issue25099}: Make test_compileall not fail when an entry on sys.path cannot
be written to (commonly seen in administrative installs on Windows).

@item 
bpo-23919@footnote{https://bugs.python.org/issue23919}: Prevents assert dialogs appearing in the test suite.

@item 
@code{PCbuild\rt.bat} now accepts an unlimited number of arguments to pass
along to regrtest.py.  Previously there was a limit of 9.
@end itemize

@node Build<56>,Windows<52>,Tests<53>,Python 3 5 1 release candidate 1
@anchor{whatsnew/changelog id524}@anchor{1474}
@subsubsection Build



@itemize -

@item 
bpo-24915@footnote{https://bugs.python.org/issue24915}: Add LLVM support for PGO builds and use the test suite to
generate the profile data. Initial patch by Alecsandru Patrascu of Intel.

@item 
bpo-24910@footnote{https://bugs.python.org/issue24910}: Windows MSIs now have unique display names.

@item 
bpo-24986@footnote{https://bugs.python.org/issue24986}: It is now possible to build Python on Windows without errors
when external libraries are not available.
@end itemize

@node Windows<52>,Tools/Demos<31>,Build<56>,Python 3 5 1 release candidate 1
@anchor{whatsnew/changelog id525}@anchor{1475}
@subsubsection Windows



@itemize -

@item 
bpo-25450@footnote{https://bugs.python.org/issue25450}: Updates shortcuts to start Python in installation directory.

@item 
bpo-25164@footnote{https://bugs.python.org/issue25164}: Changes default all-users install directory to match per-user
directory.

@item 
bpo-25143@footnote{https://bugs.python.org/issue25143}: Improves installer error messages for unsupported platforms.

@item 
bpo-25163@footnote{https://bugs.python.org/issue25163}: Display correct directory in installer when using non-default
settings.

@item 
bpo-25361@footnote{https://bugs.python.org/issue25361}: Disables use of SSE2 instructions in Windows 32-bit build

@item 
bpo-25089@footnote{https://bugs.python.org/issue25089}: Adds logging to installer for case where launcher is not
selected on upgrade.

@item 
bpo-25165@footnote{https://bugs.python.org/issue25165}: Windows uninstallation should not remove launcher if other
versions remain

@item 
bpo-25112@footnote{https://bugs.python.org/issue25112}: py.exe launcher is missing icons

@item 
bpo-25102@footnote{https://bugs.python.org/issue25102}: Windows installer does not precompile for -O or -OO.

@item 
bpo-25081@footnote{https://bugs.python.org/issue25081}: Makes Back button in installer go back to upgrade page when
upgrading.

@item 
bpo-25091@footnote{https://bugs.python.org/issue25091}: Increases font size of the installer.

@item 
bpo-25126@footnote{https://bugs.python.org/issue25126}: Clarifies that the non-web installer will download some
components.

@item 
bpo-25213@footnote{https://bugs.python.org/issue25213}: Restores requestedExecutionLevel to manifest to disable UAC
virtualization.

@item 
bpo-25022@footnote{https://bugs.python.org/issue25022}: Removed very outdated PC/example_nt/ directory.
@end itemize

@node Tools/Demos<31>,,Windows<52>,Python 3 5 1 release candidate 1
@anchor{whatsnew/changelog id526}@anchor{1476}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-25440@footnote{https://bugs.python.org/issue25440}: Fix output of python-config –extension-suffix.
@end itemize

@node Python 3 5 0 final,Python 3 5 0 release candidate 4,Python 3 5 1 release candidate 1,Changelog
@anchor{whatsnew/changelog python-3-5-0-final}@anchor{1477}
@subsection Python 3.5.0 final


@emph{Release date: 2015-09-13}

@menu
* Build: Build<57>. 

@end menu

@node Build<57>,,,Python 3 5 0 final
@anchor{whatsnew/changelog id527}@anchor{1478}
@subsubsection Build



@itemize -

@item 
bpo-25071@footnote{https://bugs.python.org/issue25071}: Windows installer should not require TargetDir parameter when
installing quietly.
@end itemize

@node Python 3 5 0 release candidate 4,Python 3 5 0 release candidate 3,Python 3 5 0 final,Changelog
@anchor{whatsnew/changelog python-3-5-0-release-candidate-4}@anchor{1479}
@subsection Python 3.5.0 release candidate 4


@emph{Release date: 2015-09-09}

@menu
* Library: Library<74>. 
* Build: Build<58>. 

@end menu

@node Library<74>,Build<58>,,Python 3 5 0 release candidate 4
@anchor{whatsnew/changelog id528}@anchor{147a}
@subsubsection Library



@itemize -

@item 
bpo-25029@footnote{https://bugs.python.org/issue25029}: Fixes MemoryError in test_strptime.
@end itemize

@node Build<58>,,Library<74>,Python 3 5 0 release candidate 4
@anchor{whatsnew/changelog id529}@anchor{147b}
@subsubsection Build



@itemize -

@item 
bpo-25027@footnote{https://bugs.python.org/issue25027}: Reverts partial-static build options and adds vcruntime140.dll
to Windows installation.
@end itemize

@node Python 3 5 0 release candidate 3,Python 3 5 0 release candidate 2,Python 3 5 0 release candidate 4,Changelog
@anchor{whatsnew/changelog python-3-5-0-release-candidate-3}@anchor{147c}
@subsection Python 3.5.0 release candidate 3


@emph{Release date: 2015-09-07}

@menu
* Core and Builtins: Core and Builtins<61>. 
* Library: Library<75>. 

@end menu

@node Core and Builtins<61>,Library<75>,,Python 3 5 0 release candidate 3
@anchor{whatsnew/changelog id530}@anchor{147d}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-24305@footnote{https://bugs.python.org/issue24305}: Prevent import subsystem stack frames from being counted by the
warnings.warn(stacklevel=) parameter.

@item 
bpo-24912@footnote{https://bugs.python.org/issue24912}: Prevent __class__ assignment to immutable built-in objects.

@item 
bpo-24975@footnote{https://bugs.python.org/issue24975}: Fix AST compilation for 
@geindex Python Enhancement Proposals; PEP 448
PEP 448@footnote{https://www.python.org/dev/peps/pep-0448} syntax.
@end itemize

@node Library<75>,,Core and Builtins<61>,Python 3 5 0 release candidate 3
@anchor{whatsnew/changelog id531}@anchor{147e}
@subsubsection Library



@itemize -

@item 
bpo-24917@footnote{https://bugs.python.org/issue24917}: time_strftime() buffer over-read.

@item 
bpo-24748@footnote{https://bugs.python.org/issue24748}: To resolve a compatibility problem found with py2exe and
pywin32, imp.load_dynamic() once again ignores previously loaded modules
to support Python modules replacing themselves with extension modules.
Patch by Petr Viktorin.

@item 
bpo-24635@footnote{https://bugs.python.org/issue24635}: Fixed a bug in typing.py where isinstance([], typing.Iterable)
would return True once, then False on subsequent calls.

@item 
bpo-24989@footnote{https://bugs.python.org/issue24989}: Fixed buffer overread in BytesIO.readline() if a position is
set beyond size.  Based on patch by John Leitch.

@item 
bpo-24913@footnote{https://bugs.python.org/issue24913}: Fix overrun error in deque.index(). Found by John Leitch and
Bryce Darling.
@end itemize

@node Python 3 5 0 release candidate 2,Python 3 5 0 release candidate 1,Python 3 5 0 release candidate 3,Changelog
@anchor{whatsnew/changelog python-3-5-0-release-candidate-2}@anchor{147f}
@subsection Python 3.5.0 release candidate 2


@emph{Release date: 2015-08-25}

@menu
* Core and Builtins: Core and Builtins<62>. 
* Library: Library<76>. 

@end menu

@node Core and Builtins<62>,Library<76>,,Python 3 5 0 release candidate 2
@anchor{whatsnew/changelog id532}@anchor{1480}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-24769@footnote{https://bugs.python.org/issue24769}: Interpreter now starts properly when dynamic loading is
disabled.  Patch by Petr Viktorin.

@item 
bpo-21167@footnote{https://bugs.python.org/issue21167}: NAN operations are now handled correctly when python is
compiled with ICC even if -fp-model strict is not specified.

@item 
bpo-24492@footnote{https://bugs.python.org/issue24492}: A “package” lacking a __name__ attribute when trying to perform
a @code{from .. import ...} statement will trigger an ImportError instead of
an AttributeError.
@end itemize

@node Library<76>,,Core and Builtins<62>,Python 3 5 0 release candidate 2
@anchor{whatsnew/changelog id533}@anchor{1481}
@subsubsection Library



@itemize -

@item 
bpo-24847@footnote{https://bugs.python.org/issue24847}: Removes vcruntime140.dll dependency from Tcl/Tk.

@item 
bpo-24839@footnote{https://bugs.python.org/issue24839}: platform._syscmd_ver raises DeprecationWarning

@item 
bpo-24867@footnote{https://bugs.python.org/issue24867}: Fix Task.get_stack() for ‘async def’ coroutines
@end itemize

@node Python 3 5 0 release candidate 1,Python 3 5 0 beta 4,Python 3 5 0 release candidate 2,Changelog
@anchor{whatsnew/changelog python-3-5-0-release-candidate-1}@anchor{1482}
@subsection Python 3.5.0 release candidate 1


@emph{Release date: 2015-08-09}

@menu
* Core and Builtins: Core and Builtins<63>. 
* Library: Library<77>. 
* IDLE: IDLE<48>. 
* Documentation: Documentation<53>. 
* Tests: Tests<54>. 

@end menu

@node Core and Builtins<63>,Library<77>,,Python 3 5 0 release candidate 1
@anchor{whatsnew/changelog id534}@anchor{1483}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-24667@footnote{https://bugs.python.org/issue24667}: Resize odict in all cases that the underlying dict resizes.
@end itemize

@node Library<77>,IDLE<48>,Core and Builtins<63>,Python 3 5 0 release candidate 1
@anchor{whatsnew/changelog id535}@anchor{1484}
@subsubsection Library



@itemize -

@item 
bpo-24824@footnote{https://bugs.python.org/issue24824}: Signatures of codecs.encode() and codecs.decode() now are
compatible with pydoc.

@item 
bpo-24634@footnote{https://bugs.python.org/issue24634}: Importing uuid should not try to load libc on Windows

@item 
bpo-24798@footnote{https://bugs.python.org/issue24798}: _msvccompiler.py doesn’t properly support manifests

@item 
bpo-4395@footnote{https://bugs.python.org/issue4395}: Better testing and documentation of binary operators. Patch by
Martin Panter.

@item 
bpo-23973@footnote{https://bugs.python.org/issue23973}: Update typing.py from GitHub repo.

@item 
bpo-23004@footnote{https://bugs.python.org/issue23004}: mock_open() now reads binary data correctly when the type of
read_data is bytes.  Initial patch by Aaron Hill.

@item 
bpo-23888@footnote{https://bugs.python.org/issue23888}: Handle fractional time in cookie expiry. Patch by ssh.

@item 
bpo-23652@footnote{https://bugs.python.org/issue23652}: Make it possible to compile the select module against the libc
headers from the Linux Standard Base, which do not include some EPOLL
macros.  Patch by Matt Frank.

@item 
bpo-22932@footnote{https://bugs.python.org/issue22932}: Fix timezones in email.utils.formatdate. Patch from Dmitry
Shachnev.

@item 
bpo-23779@footnote{https://bugs.python.org/issue23779}: imaplib raises TypeError if authenticator tries to abort. Patch
from Craig Holmquist.

@item 
bpo-23319@footnote{https://bugs.python.org/issue23319}: Fix ctypes.BigEndianStructure, swap correctly bytes. Patch
written by Matthieu Gautier.

@item 
bpo-23254@footnote{https://bugs.python.org/issue23254}: Document how to close the TCPServer listening socket. Patch
from Martin Panter.

@item 
bpo-19450@footnote{https://bugs.python.org/issue19450}: Update Windows and OS X installer builds to use SQLite 3.8.11.

@item 
bpo-17527@footnote{https://bugs.python.org/issue17527}: Add PATCH to wsgiref.validator. Patch from Luca Sbardella.

@item 
bpo-24791@footnote{https://bugs.python.org/issue24791}: Fix grammar regression for call syntax: ‘g(*a or b)’.
@end itemize

@node IDLE<48>,Documentation<53>,Library<77>,Python 3 5 0 release candidate 1
@anchor{whatsnew/changelog id536}@anchor{1485}
@subsubsection IDLE



@itemize -

@item 
bpo-23672@footnote{https://bugs.python.org/issue23672}: Allow Idle to edit and run files with astral chars in name.
Patch by Mohd Sanad Zaki Rizvi.

@item 
bpo-24745@footnote{https://bugs.python.org/issue24745}: Idle editor default font. Switch from Courier to
platform-sensitive TkFixedFont.  This should not affect current customized
font selections.  If there is a problem, edit
$HOME/.idlerc/config-main.cfg and remove ‘fontxxx’ entries from [Editor
Window].  Patch by Mark Roseman.

@item 
bpo-21192@footnote{https://bugs.python.org/issue21192}: Idle editor. When a file is run, put its name in the restart
bar. Do not print false prompts. Original patch by Adnan Umer.

@item 
bpo-13884@footnote{https://bugs.python.org/issue13884}: Idle menus. Remove tearoff lines. Patch by Roger Serwy.
@end itemize

@node Documentation<53>,Tests<54>,IDLE<48>,Python 3 5 0 release candidate 1
@anchor{whatsnew/changelog id537}@anchor{1486}
@subsubsection Documentation



@itemize -

@item 
bpo-24129@footnote{https://bugs.python.org/issue24129}: Clarify the reference documentation for name resolution. This
includes removing the assumption that readers will be familiar with the
name resolution scheme Python used prior to the introduction of lexical
scoping for function namespaces. Patch by Ivan Levkivskyi.

@item 
bpo-20769@footnote{https://bugs.python.org/issue20769}: Improve reload() docs. Patch by Dorian Pula.

@item 
bpo-23589@footnote{https://bugs.python.org/issue23589}: Remove duplicate sentence from the FAQ.  Patch by Yongzhi Pan.

@item 
bpo-24729@footnote{https://bugs.python.org/issue24729}: Correct IO tutorial to match implementation regarding encoding
parameter to open function.
@end itemize

@node Tests<54>,,Documentation<53>,Python 3 5 0 release candidate 1
@anchor{whatsnew/changelog id538}@anchor{1487}
@subsubsection Tests



@itemize -

@item 
bpo-24751@footnote{https://bugs.python.org/issue24751}: When running regrtest with the @code{-w} command line option, a
test run is no longer marked as a failure if all tests succeed when
re-run.
@end itemize

@node Python 3 5 0 beta 4,Python 3 5 0 beta 3,Python 3 5 0 release candidate 1,Changelog
@anchor{whatsnew/changelog python-3-5-0-beta-4}@anchor{1488}
@subsection Python 3.5.0 beta 4


@emph{Release date: 2015-07-26}

@menu
* Core and Builtins: Core and Builtins<64>. 
* Library: Library<78>. 
* Build: Build<59>. 

@end menu

@node Core and Builtins<64>,Library<78>,,Python 3 5 0 beta 4
@anchor{whatsnew/changelog id539}@anchor{1489}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-23573@footnote{https://bugs.python.org/issue23573}: Restored optimization of bytes.rfind() and bytearray.rfind()
for single-byte argument on Linux.

@item 
bpo-24569@footnote{https://bugs.python.org/issue24569}: Make 
@geindex Python Enhancement Proposals; PEP 448
PEP 448@footnote{https://www.python.org/dev/peps/pep-0448} dictionary evaluation more consistent.

@item 
bpo-24583@footnote{https://bugs.python.org/issue24583}: Fix crash when set is mutated while being updated.

@item 
bpo-24407@footnote{https://bugs.python.org/issue24407}: Fix crash when dict is mutated while being updated.

@item 
bpo-24619@footnote{https://bugs.python.org/issue24619}: New approach for tokenizing async/await. As a consequence, it
is now possible to have one-line ‘async def foo(): await ..’ functions.

@item 
bpo-24687@footnote{https://bugs.python.org/issue24687}: Plug refleak on SyntaxError in function parameters annotations.

@item 
bpo-15944@footnote{https://bugs.python.org/issue15944}: memoryview: Allow arbitrary formats when casting to bytes.
Patch by Martin Panter.
@end itemize

@node Library<78>,Build<59>,Core and Builtins<64>,Python 3 5 0 beta 4
@anchor{whatsnew/changelog id540}@anchor{148a}
@subsubsection Library



@itemize -

@item 
bpo-23441@footnote{https://bugs.python.org/issue23441}: rcompleter now prints a tab character instead of displaying
possible completions for an empty word.  Initial patch by Martin Sekera.

@item 
bpo-24683@footnote{https://bugs.python.org/issue24683}: Fixed crashes in _json functions called with arguments of
inappropriate type.

@item 
bpo-21697@footnote{https://bugs.python.org/issue21697}: shutil.copytree() now correctly handles symbolic links that
point to directories.  Patch by Eduardo Seabra and Thomas Kluyver.

@item 
bpo-14373@footnote{https://bugs.python.org/issue14373}: Fixed segmentation fault when gc.collect() is called during
constructing lru_cache (C implementation).

@item 
bpo-24695@footnote{https://bugs.python.org/issue24695}: Fix a regression in traceback.print_exception().  If
exc_traceback is None we shouldn’t print a traceback header like described
in the documentation.

@item 
bpo-24620@footnote{https://bugs.python.org/issue24620}: Random.setstate() now validates the value of state last
element.

@item 
bpo-22485@footnote{https://bugs.python.org/issue22485}: Fixed an issue that caused @ref{1105,,inspect.getsource} to return
incorrect results on nested functions.

@item 
bpo-22153@footnote{https://bugs.python.org/issue22153}: Improve unittest docs. Patch from Martin Panter and evilzero.

@item 
bpo-24580@footnote{https://bugs.python.org/issue24580}: Symbolic group references to open group in re patterns now are
explicitly forbidden as well as numeric group references.

@item 
bpo-24206@footnote{https://bugs.python.org/issue24206}: Fixed __eq__ and __ne__ methods of inspect classes.

@item 
bpo-24631@footnote{https://bugs.python.org/issue24631}: Fixed regression in the timeit module with multiline setup.

@item 
bpo-18622@footnote{https://bugs.python.org/issue18622}: unittest.mock.mock_open().reset_mock would recurse infinitely.
Patch from Nicola Palumbo and Laurent De Buyst.

@item 
bpo-23661@footnote{https://bugs.python.org/issue23661}: unittest.mock side_effects can now be exceptions again. This
was a regression vs Python 3.4. Patch from Ignacio Rossi

@item 
bpo-24608@footnote{https://bugs.python.org/issue24608}: chunk.Chunk.read() now always returns bytes, not str.

@item 
bpo-18684@footnote{https://bugs.python.org/issue18684}: Fixed reading out of the buffer in the re module.

@item 
bpo-24259@footnote{https://bugs.python.org/issue24259}: tarfile now raises a ReadError if an archive is truncated
inside a data segment.

@item 
bpo-15014@footnote{https://bugs.python.org/issue15014}: SMTP.auth() and SMTP.login() now support RFC 4954’s optional
initial-response argument to the SMTP AUTH command.

@item 
bpo-24669@footnote{https://bugs.python.org/issue24669}: Fix inspect.getsource() for ‘async def’ functions. Patch by Kai
Groner.

@item 
bpo-24688@footnote{https://bugs.python.org/issue24688}: ast.get_docstring() for ‘async def’ functions.
@end itemize

@node Build<59>,,Library<78>,Python 3 5 0 beta 4
@anchor{whatsnew/changelog id541}@anchor{148b}
@subsubsection Build



@itemize -

@item 
bpo-24603@footnote{https://bugs.python.org/issue24603}: Update Windows builds and OS X 10.5 installer to use OpenSSL
1.0.2d.
@end itemize

@node Python 3 5 0 beta 3,Python 3 5 0 beta 2,Python 3 5 0 beta 4,Changelog
@anchor{whatsnew/changelog python-3-5-0-beta-3}@anchor{148c}
@subsection Python 3.5.0 beta 3


@emph{Release date: 2015-07-05}

@menu
* Core and Builtins: Core and Builtins<65>. 
* Library: Library<79>. 
* Tests: Tests<55>. 
* Documentation: Documentation<54>. 
* Build: Build<60>. 

@end menu

@node Core and Builtins<65>,Library<79>,,Python 3 5 0 beta 3
@anchor{whatsnew/changelog id542}@anchor{148d}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-24467@footnote{https://bugs.python.org/issue24467}: Fixed possible buffer over-read in bytearray. The bytearray
object now always allocates place for trailing null byte and it’s buffer
now is always null-terminated.

@item 
Upgrade to Unicode 8.0.0.

@item 
bpo-24345@footnote{https://bugs.python.org/issue24345}: Add Py_tp_finalize slot for the stable ABI.

@item 
bpo-24400@footnote{https://bugs.python.org/issue24400}: Introduce a distinct type for 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} coroutines; add
types.CoroutineType, inspect.getcoroutinestate,
inspect.getcoroutinelocals; coroutines no longer use CO_GENERATOR flag;
sys.set_coroutine_wrapper works only for ‘async def’ coroutines;
inspect.iscoroutine no longer uses collections.abc.Coroutine, it’s
intended to test for pure ‘async def’ coroutines only; add new opcode:
GET_YIELD_FROM_ITER; fix generators wrapper used in types.coroutine to be
instance of collections.abc.Generator; collections.abc.Awaitable and
collections.abc.Coroutine can no longer be used to detect generator-based
coroutines–use inspect.isawaitable instead.

@item 
bpo-24450@footnote{https://bugs.python.org/issue24450}: Add gi_yieldfrom to generators and cr_await to coroutines.
Contributed by Benno Leslie and Yury Selivanov.

@item 
bpo-19235@footnote{https://bugs.python.org/issue19235}: Add new RecursionError exception. Patch by Georg Brandl.
@end itemize

@node Library<79>,Tests<55>,Core and Builtins<65>,Python 3 5 0 beta 3
@anchor{whatsnew/changelog id543}@anchor{148e}
@subsubsection Library



@itemize -

@item 
bpo-21750@footnote{https://bugs.python.org/issue21750}: mock_open.read_data can now be read from each instance, as it
could in Python 3.3.

@item 
bpo-24552@footnote{https://bugs.python.org/issue24552}: Fix use after free in an error case of the _pickle module.

@item 
bpo-24514@footnote{https://bugs.python.org/issue24514}: tarfile now tolerates number fields consisting of only
whitespace.

@item 
bpo-19176@footnote{https://bugs.python.org/issue19176}: Fixed doctype() related bugs in C implementation of
ElementTree. A deprecation warning no longer issued by XMLParser subclass
with default doctype() method.  Direct call of doctype() now issues a
warning.  Parser’s doctype() now is not called if target’s doctype() is
called.  Based on patch by Martin Panter.

@item 
bpo-20387@footnote{https://bugs.python.org/issue20387}: Restore semantic round-trip correctness in tokenize/untokenize
for tab-indented blocks.

@item 
bpo-24456@footnote{https://bugs.python.org/issue24456}: Fixed possible buffer over-read in adpcm2lin() and lin2adpcm()
functions of the audioop module.

@item 
bpo-24336@footnote{https://bugs.python.org/issue24336}: The contextmanager decorator now works with functions with
keyword arguments called “func” and “self”.  Patch by Martin Panter.

@item 
bpo-24522@footnote{https://bugs.python.org/issue24522}: Fix possible integer overflow in json accelerator module.

@item 
bpo-24489@footnote{https://bugs.python.org/issue24489}: ensure a previously set C errno doesn’t disturb cmath.polar().

@item 
bpo-24408@footnote{https://bugs.python.org/issue24408}: Fixed AttributeError in measure() and metrics() methods of
tkinter.Font.

@item 
bpo-14373@footnote{https://bugs.python.org/issue14373}: C implementation of functools.lru_cache() now can be used with
methods.

@item 
bpo-24347@footnote{https://bugs.python.org/issue24347}: Set KeyError if PyDict_GetItemWithError returns NULL.

@item 
bpo-24348@footnote{https://bugs.python.org/issue24348}: Drop superfluous incref/decref.

@item 
bpo-24359@footnote{https://bugs.python.org/issue24359}: Check for changed OrderedDict size during iteration.

@item 
bpo-24368@footnote{https://bugs.python.org/issue24368}: Support keyword arguments in OrderedDict methods.

@item 
bpo-24362@footnote{https://bugs.python.org/issue24362}: Simplify the C OrderedDict fast nodes resize logic.

@item 
bpo-24377@footnote{https://bugs.python.org/issue24377}: Fix a ref leak in OrderedDict.__repr__.

@item 
bpo-24369@footnote{https://bugs.python.org/issue24369}: Defend against key-changes during iteration.
@end itemize

@node Tests<55>,Documentation<54>,Library<79>,Python 3 5 0 beta 3
@anchor{whatsnew/changelog id544}@anchor{148f}
@subsubsection Tests



@itemize -

@item 
bpo-24373@footnote{https://bugs.python.org/issue24373}: _testmultiphase and xxlimited now use tp_traverse and
tp_finalize to avoid reference leaks encountered when combining tp_dealloc
with PyType_FromSpec (see bpo-16690@footnote{https://bugs.python.org/issue16690} for details)
@end itemize

@node Documentation<54>,Build<60>,Tests<55>,Python 3 5 0 beta 3
@anchor{whatsnew/changelog id545}@anchor{1490}
@subsubsection Documentation



@itemize -

@item 
bpo-24458@footnote{https://bugs.python.org/issue24458}: Update documentation to cover multi-phase initialization for
extension modules (PEP 489). Patch by Petr Viktorin.

@item 
bpo-24351@footnote{https://bugs.python.org/issue24351}: Clarify what is meant by “identifier” in the context of
string.Template instances.
@end itemize

@node Build<60>,,Documentation<54>,Python 3 5 0 beta 3
@anchor{whatsnew/changelog id546}@anchor{1491}
@subsubsection Build



@itemize -

@item 
bpo-24432@footnote{https://bugs.python.org/issue24432}: Update Windows builds and OS X 10.5 installer to use OpenSSL
1.0.2c.
@end itemize

@node Python 3 5 0 beta 2,Python 3 5 0 beta 1,Python 3 5 0 beta 3,Changelog
@anchor{whatsnew/changelog python-3-5-0-beta-2}@anchor{1492}
@subsection Python 3.5.0 beta 2


@emph{Release date: 2015-05-31}

@menu
* Core and Builtins: Core and Builtins<66>. 
* Library: Library<80>. 

@end menu

@node Core and Builtins<66>,Library<80>,,Python 3 5 0 beta 2
@anchor{whatsnew/changelog id547}@anchor{1493}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-24284@footnote{https://bugs.python.org/issue24284}: The startswith and endswith methods of the str class no longer
return True when finding the empty string and the indexes are completely
out of range.

@item 
bpo-24115@footnote{https://bugs.python.org/issue24115}: Update uses of PyObject_IsTrue(), PyObject_Not(),
PyObject_IsInstance(), PyObject_RichCompareBool() and _PyDict_Contains()
to check for and handle errors correctly.

@item 
bpo-24328@footnote{https://bugs.python.org/issue24328}: Fix importing one character extension modules.

@item 
bpo-11205@footnote{https://bugs.python.org/issue11205}: In dictionary displays, evaluate the key before the value.

@item 
bpo-24285@footnote{https://bugs.python.org/issue24285}: Fixed regression that prevented importing extension modules
from inside packages. Patch by Petr Viktorin.
@end itemize

@node Library<80>,,Core and Builtins<66>,Python 3 5 0 beta 2
@anchor{whatsnew/changelog id548}@anchor{1494}
@subsubsection Library



@itemize -

@item 
bpo-23247@footnote{https://bugs.python.org/issue23247}: Fix a crash in the StreamWriter.reset() of CJK codecs.

@item 
bpo-24270@footnote{https://bugs.python.org/issue24270}: Add math.isclose() and cmath.isclose() functions as per
@geindex Python Enhancement Proposals; PEP 485
PEP 485@footnote{https://www.python.org/dev/peps/pep-0485}. Contributed by Chris Barker and Tal Einat.

@item 
bpo-5633@footnote{https://bugs.python.org/issue5633}: Fixed timeit when the statement is a string and the setup is
not.

@item 
bpo-24326@footnote{https://bugs.python.org/issue24326}: Fixed audioop.ratecv() with non-default weightB argument.
Original patch by David Moore.

@item 
bpo-16991@footnote{https://bugs.python.org/issue16991}: Add a C implementation of OrderedDict.

@item 
bpo-23934@footnote{https://bugs.python.org/issue23934}: Fix inspect.signature to fail correctly for builtin types
lacking signature information.  Initial patch by James Powell.
@end itemize

@node Python 3 5 0 beta 1,Python 3 5 0 alpha 4,Python 3 5 0 beta 2,Changelog
@anchor{whatsnew/changelog python-3-5-0-beta-1}@anchor{1495}
@subsection Python 3.5.0 beta 1


@emph{Release date: 2015-05-24}

@menu
* Core and Builtins: Core and Builtins<67>. 
* Library: Library<81>. 
* IDLE: IDLE<49>. 
* Tests: Tests<56>. 
* Documentation: Documentation<55>. 
* Tools/Demos: Tools/Demos<32>. 

@end menu

@node Core and Builtins<67>,Library<81>,,Python 3 5 0 beta 1
@anchor{whatsnew/changelog id549}@anchor{1496}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-24276@footnote{https://bugs.python.org/issue24276}: Fixed optimization of property descriptor getter.

@item 
bpo-24268@footnote{https://bugs.python.org/issue24268}: PEP 489: Multi-phase extension module initialization. Patch by
Petr Viktorin.

@item 
bpo-23955@footnote{https://bugs.python.org/issue23955}: Add pyvenv.cfg option to suppress registry/environment lookup
for generating sys.path on Windows.

@item 
bpo-24257@footnote{https://bugs.python.org/issue24257}: Fixed system error in the comparison of faked
types.SimpleNamespace.

@item 
bpo-22939@footnote{https://bugs.python.org/issue22939}: Fixed integer overflow in iterator object.  Patch by Clement
Rouault.

@item 
bpo-23985@footnote{https://bugs.python.org/issue23985}: Fix a possible buffer overrun when deleting a slice from the
front of a bytearray and then appending some other bytes data.

@item 
bpo-24102@footnote{https://bugs.python.org/issue24102}: Fixed exception type checking in standard error handlers.

@item 
bpo-15027@footnote{https://bugs.python.org/issue15027}: The UTF-32 encoder is now 3x to 7x faster.

@item 
bpo-23290@footnote{https://bugs.python.org/issue23290}: Optimize set_merge() for cases where the target is empty.
(Contributed by Serhiy Storchaka.)

@item 
bpo-2292@footnote{https://bugs.python.org/issue2292}: PEP 448: Additional Unpacking Generalizations.

@item 
bpo-24096@footnote{https://bugs.python.org/issue24096}: Make warnings.warn_explicit more robust against mutation of the
warnings.filters list.

@item 
bpo-23996@footnote{https://bugs.python.org/issue23996}: Avoid a crash when a delegated generator raises an unnormalized
StopIteration exception.  Patch by Stefan Behnel.

@item 
bpo-23910@footnote{https://bugs.python.org/issue23910}: Optimize property() getter calls.  Patch by Joe Jevnik.

@item 
bpo-23911@footnote{https://bugs.python.org/issue23911}: Move path-based importlib bootstrap code to a separate frozen
module.

@item 
bpo-24192@footnote{https://bugs.python.org/issue24192}: Fix namespace package imports.

@item 
bpo-24022@footnote{https://bugs.python.org/issue24022}: Fix tokenizer crash when processing undecodable source code.

@item 
bpo-9951@footnote{https://bugs.python.org/issue9951}: Added a hex() method to bytes, bytearray, and memoryview.

@item 
bpo-22906@footnote{https://bugs.python.org/issue22906}: PEP 479: Change StopIteration handling inside generators.

@item 
bpo-24017@footnote{https://bugs.python.org/issue24017}: PEP 492: Coroutines with async and await syntax.
@end itemize

@node Library<81>,IDLE<49>,Core and Builtins<67>,Python 3 5 0 beta 1
@anchor{whatsnew/changelog id550}@anchor{1497}
@subsubsection Library



@itemize -

@item 
bpo-14373@footnote{https://bugs.python.org/issue14373}: Added C implementation of functools.lru_cache().  Based on
patches by Matt Joiner and Alexey Kachayev.

@item 
bpo-24230@footnote{https://bugs.python.org/issue24230}: The tempfile module now accepts bytes for prefix, suffix and
dir parameters and returns bytes in such situations (matching the os
module APIs).

@item 
bpo-22189@footnote{https://bugs.python.org/issue22189}: collections.UserString now supports __getnewargs__(),
__rmod__(), casefold(), format_map(), isprintable(), and maketrans().
Patch by Joe Jevnik.

@item 
bpo-24244@footnote{https://bugs.python.org/issue24244}: Prevents termination when an invalid format string is
encountered on Windows in strftime.

@item 
bpo-23973@footnote{https://bugs.python.org/issue23973}: PEP 484: Add the typing module.

@item 
bpo-23086@footnote{https://bugs.python.org/issue23086}: The collections.abc.Sequence() abstract base class added
@emph{start} and @emph{stop} parameters to the index() mixin. Patch by Devin
Jeanpierre.

@item 
bpo-20035@footnote{https://bugs.python.org/issue20035}: Replaced the @code{tkinter._fix} module used for setting up the
Tcl/Tk environment on Windows with a private function in the @code{_tkinter}
module that makes no permanent changes to the environment.

@item 
bpo-24257@footnote{https://bugs.python.org/issue24257}: Fixed segmentation fault in sqlite3.Row constructor with faked
cursor type.

@item 
bpo-15836@footnote{https://bugs.python.org/issue15836}: assertRaises(), assertRaisesRegex(), assertWarns() and
assertWarnsRegex() assertments now check the type of the first argument to
prevent possible user error.  Based on patch by Daniel Wagner-Hall.

@item 
bpo-9858@footnote{https://bugs.python.org/issue9858}: Add missing method stubs to _io.RawIOBase.  Patch by Laura
Rupprecht.

@item 
bpo-22955@footnote{https://bugs.python.org/issue22955}: attrgetter, itemgetter and methodcaller objects in the operator
module now support pickling.  Added readable and evaluable repr for these
objects. Based on patch by Josh Rosenberg.

@item 
bpo-22107@footnote{https://bugs.python.org/issue22107}: tempfile.gettempdir() and tempfile.mkdtemp() now try again when
a directory with the chosen name already exists on Windows as well as on
Unix. tempfile.mkstemp() now fails early if parent directory is not valid
(not exists or is a file) on Windows.

@item 
bpo-23780@footnote{https://bugs.python.org/issue23780}: Improved error message in os.path.join() with single argument.

@item 
bpo-6598@footnote{https://bugs.python.org/issue6598}: Increased time precision and random number range in
email.utils.make_msgid() to strengthen the uniqueness of the message ID.

@item 
bpo-24091@footnote{https://bugs.python.org/issue24091}: Fixed various crashes in corner cases in C implementation of
ElementTree.

@item 
bpo-21931@footnote{https://bugs.python.org/issue21931}: msilib.FCICreate() now raises TypeError in the case of a bad
argument instead of a ValueError with a bogus FCI error number. Patch by
Jeffrey Armstrong.

@item 
bpo-13866@footnote{https://bugs.python.org/issue13866}: @emph{quote_via} argument added to urllib.parse.urlencode.

@item 
bpo-20098@footnote{https://bugs.python.org/issue20098}: New mangle_from policy option for email, default True for
compat32, but False for all other policies.

@item 
bpo-24211@footnote{https://bugs.python.org/issue24211}: The email library now supports RFC 6532: it can generate
headers using utf-8 instead of encoded words.

@item 
bpo-16314@footnote{https://bugs.python.org/issue16314}: Added support for the LZMA compression in distutils.

@item 
bpo-21804@footnote{https://bugs.python.org/issue21804}: poplib now supports RFC 6856 (UTF8).

@item 
bpo-18682@footnote{https://bugs.python.org/issue18682}: Optimized pprint functions for builtin scalar types.

@item 
bpo-22027@footnote{https://bugs.python.org/issue22027}: smtplib now supports RFC 6531 (SMTPUTF8).

@item 
bpo-23488@footnote{https://bugs.python.org/issue23488}: Random generator objects now consume 2x less memory on 64-bit.

@item 
bpo-1322@footnote{https://bugs.python.org/issue1322}: platform.dist() and platform.linux_distribution() functions are
now deprecated.  Initial patch by Vajrasky Kok.

@item 
bpo-22486@footnote{https://bugs.python.org/issue22486}: Added the math.gcd() function.  The fractions.gcd() function
now is deprecated.  Based on patch by Mark Dickinson.

@item 
bpo-24064@footnote{https://bugs.python.org/issue24064}: Property() docstrings are now writeable. (Patch by Berker
Peksag.)

@item 
bpo-22681@footnote{https://bugs.python.org/issue22681}: Added support for the koi8_t encoding.

@item 
bpo-22682@footnote{https://bugs.python.org/issue22682}: Added support for the kz1048 encoding.

@item 
bpo-23796@footnote{https://bugs.python.org/issue23796}: peek and read1 methods of BufferedReader now raise ValueError
if they called on a closed object. Patch by John Hergenroeder.

@item 
bpo-21795@footnote{https://bugs.python.org/issue21795}: smtpd now supports the 8BITMIME extension whenever the new
@emph{decode_data} constructor argument is set to False.

@item 
bpo-24155@footnote{https://bugs.python.org/issue24155}: optimize heapq.heapify() for better cache performance when
heapifying large lists.

@item 
bpo-21800@footnote{https://bugs.python.org/issue21800}: imaplib now supports RFC 5161 (enable), RFC 6855
(utf8/internationalized email) and automatically encodes non-ASCII
usernames and passwords to UTF8.

@item 
bpo-20274@footnote{https://bugs.python.org/issue20274}: When calling a _sqlite.Connection, it now complains if passed
any keyword arguments.  Previously it silently ignored them.

@item 
bpo-20274@footnote{https://bugs.python.org/issue20274}: Remove ignored and erroneous “kwargs” parameters from three
METH_VARARGS methods on _sqlite.Connection.

@item 
bpo-24134@footnote{https://bugs.python.org/issue24134}: assertRaises(), assertRaisesRegex(), assertWarns() and
assertWarnsRegex() checks now emits a deprecation warning when callable is
None or keyword arguments except msg is passed in the context manager
mode.

@item 
bpo-24018@footnote{https://bugs.python.org/issue24018}: Add a collections.abc.Generator abstract base class.
Contributed by Stefan Behnel.

@item 
bpo-23880@footnote{https://bugs.python.org/issue23880}: Tkinter’s getint() and getdouble() now support Tcl_Obj.
Tkinter’s getdouble() now supports any numbers (in particular int).

@item 
bpo-22619@footnote{https://bugs.python.org/issue22619}: Added negative limit support in the traceback module. Based on
patch by Dmitry Kazakov.

@item 
bpo-24094@footnote{https://bugs.python.org/issue24094}: Fix possible crash in json.encode with poorly behaved dict
subclasses.

@item 
bpo-9246@footnote{https://bugs.python.org/issue9246}: On POSIX, os.getcwd() now supports paths longer than 1025 bytes.
Patch written by William Orr.

@item 
bpo-17445@footnote{https://bugs.python.org/issue17445}: add difflib.diff_bytes() to support comparison of byte strings
(fixes a regression from Python 2).

@item 
bpo-23917@footnote{https://bugs.python.org/issue23917}: Fall back to sequential compilation when ProcessPoolExecutor
doesn’t exist. Patch by Claudiu Popa.

@item 
bpo-23008@footnote{https://bugs.python.org/issue23008}: Fixed resolving attributes with boolean value is False in
pydoc.

@item 
Fix asyncio issue 235: LifoQueue and PriorityQueue’s put didn’t increment
unfinished tasks (this bug was introduced when JoinableQueue was merged
with Queue).

@item 
bpo-23908@footnote{https://bugs.python.org/issue23908}: os functions now reject paths with embedded null character on
Windows instead of silently truncating them.

@item 
bpo-23728@footnote{https://bugs.python.org/issue23728}: binascii.crc_hqx() could return an integer outside of the range
0-0xffff for empty data.

@item 
bpo-23887@footnote{https://bugs.python.org/issue23887}: urllib.error.HTTPError now has a proper repr() representation.
Patch by Berker Peksag.

@item 
asyncio: New event loop APIs: set_task_factory() and get_task_factory().

@item 
asyncio: async() function is deprecated in favour of ensure_future().

@item 
bpo-24178@footnote{https://bugs.python.org/issue24178}: asyncio.Lock, Condition, Semaphore, and BoundedSemaphore
support new ‘async with’ syntax.  Contributed by Yury Selivanov.

@item 
bpo-24179@footnote{https://bugs.python.org/issue24179}: Support ‘async for’ for asyncio.StreamReader. Contributed by
Yury Selivanov.

@item 
bpo-24184@footnote{https://bugs.python.org/issue24184}: Add AsyncIterator and AsyncIterable ABCs to collections.abc.
Contributed by Yury Selivanov.

@item 
bpo-22547@footnote{https://bugs.python.org/issue22547}: Implement informative __repr__ for inspect.BoundArguments.
Contributed by Yury Selivanov.

@item 
bpo-24190@footnote{https://bugs.python.org/issue24190}: Implement inspect.BoundArgument.apply_defaults() method.
Contributed by Yury Selivanov.

@item 
bpo-20691@footnote{https://bugs.python.org/issue20691}: Add ‘follow_wrapped’ argument to
inspect.Signature.from_callable() and inspect.signature(). Contributed by
Yury Selivanov.

@item 
bpo-24248@footnote{https://bugs.python.org/issue24248}: Deprecate inspect.Signature.from_function() and
inspect.Signature.from_builtin().

@item 
bpo-23898@footnote{https://bugs.python.org/issue23898}: Fix inspect.classify_class_attrs() to support attributes with
overloaded __eq__ and __bool__.  Patch by Mike Bayer.

@item 
bpo-24298@footnote{https://bugs.python.org/issue24298}: Fix inspect.signature() to correctly unwrap wrappers around
bound methods.
@end itemize

@node IDLE<49>,Tests<56>,Library<81>,Python 3 5 0 beta 1
@anchor{whatsnew/changelog id551}@anchor{1498}
@subsubsection IDLE



@itemize -

@item 
bpo-23184@footnote{https://bugs.python.org/issue23184}: remove unused names and imports in idlelib. Initial patch by Al
Sweigart.
@end itemize

@node Tests<56>,Documentation<55>,IDLE<49>,Python 3 5 0 beta 1
@anchor{whatsnew/changelog id552}@anchor{1499}
@subsubsection Tests



@itemize -

@item 
bpo-21520@footnote{https://bugs.python.org/issue21520}: test_zipfile no longer fails if the word ‘bad’ appears anywhere
in the name of the current directory.

@item 
bpo-9517@footnote{https://bugs.python.org/issue9517}: Move script_helper into the support package. Patch by Christie
Wilson.
@end itemize

@node Documentation<55>,Tools/Demos<32>,Tests<56>,Python 3 5 0 beta 1
@anchor{whatsnew/changelog id553}@anchor{149a}
@subsubsection Documentation



@itemize -

@item 
bpo-22155@footnote{https://bugs.python.org/issue22155}: Add File Handlers subsection with createfilehandler to tkinter
doc.  Remove obsolete example from FAQ.  Patch by Martin Panter.

@item 
bpo-24029@footnote{https://bugs.python.org/issue24029}: Document the name binding behavior for submodule imports.

@item 
bpo-24077@footnote{https://bugs.python.org/issue24077}: Fix typo in man page for -I command option: -s, not -S
@end itemize

@node Tools/Demos<32>,,Documentation<55>,Python 3 5 0 beta 1
@anchor{whatsnew/changelog id554}@anchor{149b}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-24000@footnote{https://bugs.python.org/issue24000}: Improved Argument Clinic’s mapping of converters to legacy
“format units”. Updated the documentation to match.

@item 
bpo-24001@footnote{https://bugs.python.org/issue24001}: Argument Clinic converters now use accept=@{type@} instead of
types=@{‘type’@} to specify the types the converter accepts.

@item 
bpo-23330@footnote{https://bugs.python.org/issue23330}: h2py now supports arbitrary filenames in #include.

@item 
bpo-24031@footnote{https://bugs.python.org/issue24031}: make patchcheck now supports git checkouts, too.
@end itemize

@node Python 3 5 0 alpha 4,Python 3 5 0 alpha 3,Python 3 5 0 beta 1,Changelog
@anchor{whatsnew/changelog python-3-5-0-alpha-4}@anchor{149c}
@subsection Python 3.5.0 alpha 4


@emph{Release date: 2015-04-19}

@menu
* Core and Builtins: Core and Builtins<68>. 
* Library: Library<82>. 
* Build: Build<61>. 
* Tests: Tests<57>. 
* Tools/Demos: Tools/Demos<33>. 
* C API: C API<46>. 

@end menu

@node Core and Builtins<68>,Library<82>,,Python 3 5 0 alpha 4
@anchor{whatsnew/changelog id555}@anchor{149d}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-22980@footnote{https://bugs.python.org/issue22980}: Under Linux, GNU/KFreeBSD and the Hurd, C extensions now
include the architecture triplet in the extension name, to make it easy to
test builds for different ABIs in the same working tree.  Under OS X, the
extension name now includes 
@geindex Python Enhancement Proposals; PEP 3149
PEP 3149@footnote{https://www.python.org/dev/peps/pep-3149}-style information.

@item 
bpo-22631@footnote{https://bugs.python.org/issue22631}: Added Linux-specific socket constant CAN_RAW_FD_FRAMES. Patch
courtesy of Joe Jevnik.

@item 
bpo-23731@footnote{https://bugs.python.org/issue23731}: Implement 
@geindex Python Enhancement Proposals; PEP 488
PEP 488@footnote{https://www.python.org/dev/peps/pep-0488}: removal of .pyo files.

@item 
bpo-23726@footnote{https://bugs.python.org/issue23726}: Don’t enable GC for user subclasses of non-GC types that don’t
add any new fields.  Patch by Eugene Toder.

@item 
bpo-23309@footnote{https://bugs.python.org/issue23309}: Avoid a deadlock at shutdown if a daemon thread is aborted
while it is holding a lock to a buffered I/O object, and the main thread
tries to use the same I/O object (typically stdout or stderr).  A fatal
error is emitted instead.

@item 
bpo-22977@footnote{https://bugs.python.org/issue22977}: Fixed formatting Windows error messages on Wine. Patch by
Martin Panter.

@item 
bpo-23466@footnote{https://bugs.python.org/issue23466}: %c, %o, %x, and %X in bytes formatting now raise TypeError on
non-integer input.

@item 
bpo-24044@footnote{https://bugs.python.org/issue24044}: Fix possible null pointer dereference in list.sort in out of
memory conditions.

@item 
bpo-21354@footnote{https://bugs.python.org/issue21354}: PyCFunction_New function is exposed by python DLL again.
@end itemize

@node Library<82>,Build<61>,Core and Builtins<68>,Python 3 5 0 alpha 4
@anchor{whatsnew/changelog id556}@anchor{149e}
@subsubsection Library



@itemize -

@item 
bpo-23840@footnote{https://bugs.python.org/issue23840}: tokenize.open() now closes the temporary binary file on error
to fix a resource warning.

@item 
bpo-16914@footnote{https://bugs.python.org/issue16914}: new debuglevel 2 in smtplib adds timestamps to debug output.

@item 
bpo-7159@footnote{https://bugs.python.org/issue7159}: urllib.request now supports sending auth credentials
automatically after the first 401.  This enhancement is a superset of the
enhancement from bpo-19494@footnote{https://bugs.python.org/issue19494} and supersedes that change.

@item 
bpo-23703@footnote{https://bugs.python.org/issue23703}: Fix a regression in urljoin() introduced in 901e4e52b20a. Patch
by Demian Brecht.

@item 
bpo-4254@footnote{https://bugs.python.org/issue4254}: Adds _curses.update_lines_cols().  Patch by Arnon Yaari

@item 
bpo-19933@footnote{https://bugs.python.org/issue19933}: Provide default argument for ndigits in round. Patch by
Vajrasky Kok.

@item 
bpo-23193@footnote{https://bugs.python.org/issue23193}: Add a numeric_owner parameter to tarfile.TarFile.extract and
tarfile.TarFile.extractall. Patch by Michael Vogt and Eric Smith.

@item 
bpo-23342@footnote{https://bugs.python.org/issue23342}: Add a subprocess.run() function than returns a CalledProcess
instance for a more consistent API than the existing call* functions.

@item 
bpo-21217@footnote{https://bugs.python.org/issue21217}: inspect.getsourcelines() now tries to compute the start and end
lines from the code object, fixing an issue when a lambda function is used
as decorator argument. Patch by Thomas Ballinger and Allison Kaptur.

@item 
bpo-24521@footnote{https://bugs.python.org/issue24521}: Fix possible integer overflows in the pickle module.

@item 
bpo-22931@footnote{https://bugs.python.org/issue22931}: Allow ‘[’ and ‘]’ in cookie values.

@item 
The keywords attribute of functools.partial is now always a dictionary.

@item 
bpo-23811@footnote{https://bugs.python.org/issue23811}: Add missing newline to the PyCompileError error message. Patch
by Alex Shkop.

@item 
bpo-21116@footnote{https://bugs.python.org/issue21116}: Avoid blowing memory when allocating a multiprocessing shared
array that’s larger than 50% of the available RAM.  Patch by Médéric
Boquien.

@item 
bpo-22982@footnote{https://bugs.python.org/issue22982}: Improve BOM handling when seeking to multiple positions of a
writable text file.

@item 
bpo-23464@footnote{https://bugs.python.org/issue23464}: Removed deprecated asyncio JoinableQueue.

@item 
bpo-23529@footnote{https://bugs.python.org/issue23529}: Limit the size of decompressed data when reading from GzipFile,
BZ2File or LZMAFile.  This defeats denial of service attacks using
compressed bombs (i.e. compressed payloads which decompress to a huge
size).  Patch by Martin Panter and Nikolaus Rath.

@item 
bpo-21859@footnote{https://bugs.python.org/issue21859}: Added Python implementation of io.FileIO.

@item 
bpo-23865@footnote{https://bugs.python.org/issue23865}: close() methods in multiple modules now are idempotent and more
robust at shutdown. If they need to release multiple resources, all are
released even if errors occur.

@item 
bpo-23400@footnote{https://bugs.python.org/issue23400}: Raise same exception on both Python 2 and 3 if sem_open is not
available. Patch by Davin Potts.

@item 
bpo-10838@footnote{https://bugs.python.org/issue10838}: The subprocess now module includes SubprocessError and
TimeoutError in its list of exported names for the users wild enough to
use @code{from subprocess import *}.

@item 
bpo-23411@footnote{https://bugs.python.org/issue23411}: Added DefragResult, ParseResult, SplitResult,
DefragResultBytes, ParseResultBytes, and SplitResultBytes to
urllib.parse.__all__. Patch by Martin Panter.

@item 
bpo-23881@footnote{https://bugs.python.org/issue23881}: urllib.request.ftpwrapper constructor now closes the socket if
the FTP connection failed to fix a ResourceWarning.

@item 
bpo-23853@footnote{https://bugs.python.org/issue23853}: @ref{904,,socket.socket.sendall()} does no more reset the socket
timeout each time data is sent successfully. The socket timeout is now the
maximum total duration to send all data.

@item 
bpo-22721@footnote{https://bugs.python.org/issue22721}: An order of multiline pprint output of set or dict containing
orderable and non-orderable elements no longer depends on iteration order
of set or dict.

@item 
bpo-15133@footnote{https://bugs.python.org/issue15133}: _tkinter.tkapp.getboolean() now supports Tcl_Obj and always
returns bool. tkinter.BooleanVar now validates input values (accepted
bool, int, str, and Tcl_Obj).  tkinter.BooleanVar.get() now always returns
bool.

@item 
bpo-10590@footnote{https://bugs.python.org/issue10590}: xml.sax.parseString() now supports string argument.

@item 
bpo-23338@footnote{https://bugs.python.org/issue23338}: Fixed formatting ctypes error messages on Cygwin. Patch by
Makoto Kato.

@item 
bpo-15582@footnote{https://bugs.python.org/issue15582}: inspect.getdoc() now follows inheritance chains.

@item 
bpo-2175@footnote{https://bugs.python.org/issue2175}: SAX parsers now support a character stream of InputSource
object.

@item 
bpo-16840@footnote{https://bugs.python.org/issue16840}: Tkinter now supports 64-bit integers added in Tcl 8.4 and
arbitrary precision integers added in Tcl 8.5.

@item 
bpo-23834@footnote{https://bugs.python.org/issue23834}: Fix socket.sendto(), use the C Py_ssize_t type to store the
result of sendto() instead of the C int type.

@item 
bpo-23618@footnote{https://bugs.python.org/issue23618}: @ref{8ff,,socket.socket.connect()} now waits until the connection
completes instead of raising @ref{8e3,,InterruptedError} if the connection is
interrupted by signals, signal handlers don’t raise an exception and the
socket is blocking or has a timeout. @ref{8ff,,socket.socket.connect()} still
raise @ref{8e3,,InterruptedError} for non-blocking sockets.

@item 
bpo-21526@footnote{https://bugs.python.org/issue21526}: Tkinter now supports new boolean type in Tcl 8.5.

@item 
bpo-23836@footnote{https://bugs.python.org/issue23836}: Fix the faulthandler module to handle reentrant calls to its
signal handlers.

@item 
bpo-23838@footnote{https://bugs.python.org/issue23838}: linecache now clears the cache and returns an empty result on
MemoryError.

@item 
bpo-10395@footnote{https://bugs.python.org/issue10395}: Added os.path.commonpath(). Implemented in posixpath and
ntpath. Based on patch by Rafik Draoui.

@item 
bpo-23611@footnote{https://bugs.python.org/issue23611}: Serializing more “lookupable” objects (such as unbound methods
or nested classes) now are supported with pickle protocols < 4.

@item 
bpo-13583@footnote{https://bugs.python.org/issue13583}: sqlite3.Row now supports slice indexing.

@item 
bpo-18473@footnote{https://bugs.python.org/issue18473}: Fixed 2to3 and 3to2 compatible pickle mappings.  Fixed
ambiguous reverse mappings.  Added many new mappings.  Import mapping is
no longer applied to modules already mapped with full name mapping.

@item 
bpo-23485@footnote{https://bugs.python.org/issue23485}: select.select() is now retried automatically with the
recomputed timeout when interrupted by a signal, except if the signal
handler raises an exception. This change is part of the 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475}.

@item 
bpo-23752@footnote{https://bugs.python.org/issue23752}: When built from an existing file descriptor, io.FileIO() now
only calls fstat() once. Before fstat() was called twice, which was not
necessary.

@item 
bpo-23704@footnote{https://bugs.python.org/issue23704}: collections.deque() objects now support __add__, __mul__, and
__imul__().

@item 
bpo-23171@footnote{https://bugs.python.org/issue23171}: csv.Writer.writerow() now supports arbitrary iterables.

@item 
bpo-23745@footnote{https://bugs.python.org/issue23745}: The new email header parser now handles duplicate MIME
parameter names without error, similar to how get_param behaves.

@item 
bpo-22117@footnote{https://bugs.python.org/issue22117}: Fix os.utime(), it now rounds the timestamp towards minus
infinity (-inf) instead of rounding towards zero.

@item 
bpo-23310@footnote{https://bugs.python.org/issue23310}: Fix MagicMock’s initializer to work with __methods__, just like
configure_mock().  Patch by Kasia Jachim.
@end itemize

@node Build<61>,Tests<57>,Library<82>,Python 3 5 0 alpha 4
@anchor{whatsnew/changelog id557}@anchor{149f}
@subsubsection Build



@itemize -

@item 
bpo-23817@footnote{https://bugs.python.org/issue23817}: FreeBSD now uses “1.0” in the SOVERSION as other operating
systems, instead of just “1”.

@item 
bpo-23501@footnote{https://bugs.python.org/issue23501}: Argument Clinic now generates code into separate files by
default.
@end itemize

@node Tests<57>,Tools/Demos<33>,Build<61>,Python 3 5 0 alpha 4
@anchor{whatsnew/changelog id558}@anchor{14a0}
@subsubsection Tests



@itemize -

@item 
bpo-23799@footnote{https://bugs.python.org/issue23799}: Added test.support.start_threads() for running and cleaning up
multiple threads.

@item 
bpo-22390@footnote{https://bugs.python.org/issue22390}: test.regrtest now emits a warning if temporary files or
directories are left after running a test.
@end itemize

@node Tools/Demos<33>,C API<46>,Tests<57>,Python 3 5 0 alpha 4
@anchor{whatsnew/changelog id559}@anchor{14a1}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-18128@footnote{https://bugs.python.org/issue18128}: pygettext now uses standard +NNNN format in the
POT-Creation-Date header.

@item 
bpo-23935@footnote{https://bugs.python.org/issue23935}: Argument Clinic’s understanding of format units accepting
bytes, bytearrays, and buffers is now consistent with both the
documentation and the implementation.

@item 
bpo-23944@footnote{https://bugs.python.org/issue23944}: Argument Clinic now wraps long impl prototypes at column 78.

@item 
bpo-20586@footnote{https://bugs.python.org/issue20586}: Argument Clinic now ensures that functions without docstrings
have signatures.

@item 
bpo-23492@footnote{https://bugs.python.org/issue23492}: Argument Clinic now generates argument parsing code with
PyArg_Parse instead of PyArg_ParseTuple if possible.

@item 
bpo-23500@footnote{https://bugs.python.org/issue23500}: Argument Clinic is now smarter about generating the “#ifndef”
(empty) definition of the methoddef macro: it’s only generated once, even
if Argument Clinic processes the same symbol multiple times, and it’s
emitted at the end of all processing rather than immediately after the
first use.
@end itemize

@node C API<46>,,Tools/Demos<33>,Python 3 5 0 alpha 4
@anchor{whatsnew/changelog id560}@anchor{14a2}
@subsubsection C API



@itemize -

@item 
bpo-23998@footnote{https://bugs.python.org/issue23998}: PyImport_ReInitLock() now checks for lock allocation error
@end itemize

@node Python 3 5 0 alpha 3,Python 3 5 0 alpha 2,Python 3 5 0 alpha 4,Changelog
@anchor{whatsnew/changelog python-3-5-0-alpha-3}@anchor{14a3}
@subsection Python 3.5.0 alpha 3


@emph{Release date: 2015-03-28}

@menu
* Core and Builtins: Core and Builtins<69>. 
* Library: Library<83>. 
* Build: Build<62>. 
* Tests: Tests<58>. 
* Tools/Demos: Tools/Demos<34>. 

@end menu

@node Core and Builtins<69>,Library<83>,,Python 3 5 0 alpha 3
@anchor{whatsnew/changelog id561}@anchor{14a4}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-23573@footnote{https://bugs.python.org/issue23573}: Increased performance of string search operations (str.find,
str.index, str.count, the in operator, str.split, str.partition) with
arguments of different kinds (UCS1, UCS2, UCS4).

@item 
bpo-23753@footnote{https://bugs.python.org/issue23753}: Python doesn’t support anymore platforms without stat() or
fstat(), these functions are always required.

@item 
bpo-23681@footnote{https://bugs.python.org/issue23681}: The -b option now affects comparisons of bytes with int.

@item 
bpo-23632@footnote{https://bugs.python.org/issue23632}: Memoryviews now allow tuple indexing (including for
multi-dimensional memoryviews).

@item 
bpo-23192@footnote{https://bugs.python.org/issue23192}: Fixed generator lambdas.  Patch by Bruno Cauet.

@item 
bpo-23629@footnote{https://bugs.python.org/issue23629}: Fix the default __sizeof__ implementation for variable-sized
objects.
@end itemize

@node Library<83>,Build<62>,Core and Builtins<69>,Python 3 5 0 alpha 3
@anchor{whatsnew/changelog id562}@anchor{14a5}
@subsubsection Library



@itemize -

@item 
bpo-14260@footnote{https://bugs.python.org/issue14260}: The groupindex attribute of regular expression pattern object
now is non-modifiable mapping.

@item 
bpo-23792@footnote{https://bugs.python.org/issue23792}: Ignore KeyboardInterrupt when the pydoc pager is active. This
mimics the behavior of the standard unix pagers, and prevents pipepager
from shutting down while the pager itself is still running.

@item 
bpo-23775@footnote{https://bugs.python.org/issue23775}: pprint() of OrderedDict now outputs the same representation as
repr().

@item 
bpo-23765@footnote{https://bugs.python.org/issue23765}: Removed IsBadStringPtr calls in ctypes

@item 
bpo-22364@footnote{https://bugs.python.org/issue22364}: Improved some re error messages using regex for hints.

@item 
bpo-23742@footnote{https://bugs.python.org/issue23742}: ntpath.expandvars() no longer loses unbalanced single quotes.

@item 
bpo-21717@footnote{https://bugs.python.org/issue21717}: The zipfile.ZipFile.open function now supports ‘x’ (exclusive
creation) mode.

@item 
bpo-21802@footnote{https://bugs.python.org/issue21802}: The reader in BufferedRWPair now is closed even when closing
writer failed in BufferedRWPair.close().

@item 
bpo-23622@footnote{https://bugs.python.org/issue23622}: Unknown escapes in regular expressions that consist of @code{'\'}
and ASCII letter now raise a deprecation warning and will be forbidden in
Python 3.6.

@item 
bpo-23671@footnote{https://bugs.python.org/issue23671}: string.Template now allows specifying the “self” parameter as a
keyword argument.  string.Formatter now allows specifying the “self” and
the “format_string” parameters as keyword arguments.

@item 
bpo-23502@footnote{https://bugs.python.org/issue23502}: The pprint module now supports mapping proxies.

@item 
bpo-17530@footnote{https://bugs.python.org/issue17530}: pprint now wraps long bytes objects and bytearrays.

@item 
bpo-22687@footnote{https://bugs.python.org/issue22687}: Fixed some corner cases in breaking words in tetxtwrap. Got rid
of quadratic complexity in breaking long words.

@item 
bpo-4727@footnote{https://bugs.python.org/issue4727}: The copy module now uses pickle protocol 4 (PEP 3154) and
supports copying of instances of classes whose __new__ method takes
keyword-only arguments.

@item 
bpo-23491@footnote{https://bugs.python.org/issue23491}: Added a zipapp module to support creating executable zip file
archives of Python code. Registered “.pyz” and “.pyzw” extensions on
Windows for these archives (PEP 441).

@item 
bpo-23657@footnote{https://bugs.python.org/issue23657}: Avoid explicit checks for str in zipapp, adding support for
pathlib.Path objects as arguments.

@item 
bpo-23688@footnote{https://bugs.python.org/issue23688}: Added support of arbitrary bytes-like objects and avoided
unnecessary copying of memoryview in gzip.GzipFile.write(). Original patch
by Wolfgang Maier.

@item 
bpo-23252@footnote{https://bugs.python.org/issue23252}: Added support for writing ZIP files to unseekable streams.

@item 
bpo-23647@footnote{https://bugs.python.org/issue23647}: Increase imaplib’s MAXLINE to accommodate modern mailbox sizes.

@item 
bpo-23539@footnote{https://bugs.python.org/issue23539}: If body is None, http.client.HTTPConnection.request now sets
Content-Length to 0 for PUT, POST, and PATCH headers to avoid 411 errors
from some web servers.

@item 
bpo-22351@footnote{https://bugs.python.org/issue22351}: The nntplib.NNTP constructor no longer leaves the connection
and socket open until the garbage collector cleans them up.  Patch by
Martin Panter.

@item 
bpo-23704@footnote{https://bugs.python.org/issue23704}: collections.deque() objects now support methods for index(),
insert(), and copy().  This allows deques to be registered as a
MutableSequence and it improves their substitutability for lists.

@item 
bpo-23715@footnote{https://bugs.python.org/issue23715}: @ref{908,,signal.sigwaitinfo()} and @ref{907,,signal.sigtimedwait()} are
now retried when interrupted by a signal not in the @emph{sigset} parameter, if
the signal handler does not raise an exception. signal.sigtimedwait()
recomputes the timeout with a monotonic clock when it is retried.

@item 
bpo-23001@footnote{https://bugs.python.org/issue23001}: Few functions in modules mmap, ossaudiodev, socket, ssl, and
codecs, that accepted only read-only bytes-like object now accept writable
bytes-like object too.

@item 
bpo-23646@footnote{https://bugs.python.org/issue23646}: If time.sleep() is interrupted by a signal, the sleep is now
retried with the recomputed delay, except if the signal handler raises an
exception (PEP 475).

@item 
bpo-23136@footnote{https://bugs.python.org/issue23136}: _strptime now uniformly handles all days in week 0, including
Dec 30 of previous year.  Based on patch by Jim Carroll.

@item 
bpo-23700@footnote{https://bugs.python.org/issue23700}: Iterator of NamedTemporaryFile now keeps a reference to
NamedTemporaryFile instance.  Patch by Bohuslav Kabrda.

@item 
bpo-22903@footnote{https://bugs.python.org/issue22903}: The fake test case created by unittest.loader when it fails
importing a test module is now picklable.

@item 
bpo-22181@footnote{https://bugs.python.org/issue22181}: On Linux, os.urandom() now uses the new getrandom() syscall if
available, syscall introduced in the Linux kernel 3.17. It is more
reliable and more secure, because it avoids the need of a file descriptor
and waits until the kernel has enough entropy.

@item 
bpo-2211@footnote{https://bugs.python.org/issue2211}: Updated the implementation of the http.cookies.Morsel class.
Setting attributes key, value and coded_value directly now is deprecated.
update() and setdefault() now transform and check keys.  Comparing for
equality now takes into account attributes key, value and coded_value.
copy() now returns a Morsel, not a dict.  repr() now contains all
attributes. Optimized checking keys and quoting values.  Added new tests.
Original patch by Demian Brecht.

@item 
bpo-18983@footnote{https://bugs.python.org/issue18983}: Allow selection of output units in timeit. Patch by Julian
Gindi.

@item 
bpo-23631@footnote{https://bugs.python.org/issue23631}: Fix traceback.format_list when a traceback has been mutated.

@item 
bpo-23568@footnote{https://bugs.python.org/issue23568}: Add rdivmod support to MagicMock() objects. Patch by Håkan
Lövdahl.

@item 
bpo-2052@footnote{https://bugs.python.org/issue2052}: Add charset parameter to HtmlDiff.make_file().

@item 
bpo-23668@footnote{https://bugs.python.org/issue23668}: Support os.truncate and os.ftruncate on Windows.

@item 
bpo-23138@footnote{https://bugs.python.org/issue23138}: Fixed parsing cookies with absent keys or values in cookiejar.
Patch by Demian Brecht.

@item 
bpo-23051@footnote{https://bugs.python.org/issue23051}: multiprocessing.Pool methods imap() and imap_unordered() now
handle exceptions raised by an iterator.  Patch by Alon Diamant and Davin
Potts.

@item 
bpo-23581@footnote{https://bugs.python.org/issue23581}: Add matmul support to MagicMock. Patch by Håkan Lövdahl.

@item 
bpo-23566@footnote{https://bugs.python.org/issue23566}: enable(), register(), dump_traceback() and
dump_traceback_later() functions of faulthandler now accept file
descriptors. Patch by Wei Wu.

@item 
bpo-22928@footnote{https://bugs.python.org/issue22928}: Disabled HTTP header injections in http.client. Original patch
by Demian Brecht.

@item 
bpo-23615@footnote{https://bugs.python.org/issue23615}: Modules bz2, tarfile and tokenize now can be reloaded with
imp.reload(). Patch by Thomas Kluyver.

@item 
bpo-23605@footnote{https://bugs.python.org/issue23605}: os.walk() now calls os.scandir() instead of os.listdir(). The
usage of os.scandir() reduces the number of calls to os.stat(). Initial
patch written by Ben Hoyt.
@end itemize

@node Build<62>,Tests<58>,Library<83>,Python 3 5 0 alpha 3
@anchor{whatsnew/changelog id563}@anchor{14a6}
@subsubsection Build



@itemize -

@item 
bpo-23585@footnote{https://bugs.python.org/issue23585}: make patchcheck will ensure the interpreter is built.
@end itemize

@node Tests<58>,Tools/Demos<34>,Build<62>,Python 3 5 0 alpha 3
@anchor{whatsnew/changelog id564}@anchor{14a7}
@subsubsection Tests



@itemize -

@item 
bpo-23583@footnote{https://bugs.python.org/issue23583}: Added tests for standard IO streams in IDLE.

@item 
bpo-22289@footnote{https://bugs.python.org/issue22289}: Prevent test_urllib2net failures due to ftp connection timeout.
@end itemize

@node Tools/Demos<34>,,Tests<58>,Python 3 5 0 alpha 3
@anchor{whatsnew/changelog id565}@anchor{14a8}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-22826@footnote{https://bugs.python.org/issue22826}: The result of open() in Tools/freeze/bkfile.py is now better
compatible with regular files (in particular it now supports the context
management protocol).
@end itemize

@node Python 3 5 0 alpha 2,Python 3 5 0 alpha 1,Python 3 5 0 alpha 3,Changelog
@anchor{whatsnew/changelog python-3-5-0-alpha-2}@anchor{14a9}
@subsection Python 3.5.0 alpha 2


@emph{Release date: 2015-03-09}

@menu
* Core and Builtins: Core and Builtins<70>. 
* Library: Library<84>. 
* Build: Build<63>. 
* C API: C API<47>. 
* Windows: Windows<53>. 

@end menu

@node Core and Builtins<70>,Library<84>,,Python 3 5 0 alpha 2
@anchor{whatsnew/changelog id566}@anchor{14aa}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-23571@footnote{https://bugs.python.org/issue23571}: PyObject_Call() and PyCFunction_Call() now raise a SystemError
if a function returns a result and raises an exception. The SystemError is
chained to the previous exception.
@end itemize

@node Library<84>,Build<63>,Core and Builtins<70>,Python 3 5 0 alpha 2
@anchor{whatsnew/changelog id567}@anchor{14ab}
@subsubsection Library



@itemize -

@item 
bpo-22524@footnote{https://bugs.python.org/issue22524}: New os.scandir() function, part of the 
@geindex Python Enhancement Proposals; PEP 471
PEP 471@footnote{https://www.python.org/dev/peps/pep-0471}:
“os.scandir() function – a better and faster directory iterator”. Patch
written by Ben Hoyt.

@item 
bpo-23103@footnote{https://bugs.python.org/issue23103}: Reduced the memory consumption of IPv4Address and IPv6Address.

@item 
bpo-21793@footnote{https://bugs.python.org/issue21793}: BaseHTTPRequestHandler again logs response code as numeric, not
as stringified enum.  Patch by Demian Brecht.

@item 
bpo-23476@footnote{https://bugs.python.org/issue23476}: In the ssl module, enable OpenSSL’s X509_V_FLAG_TRUSTED_FIRST
flag on certificate stores when it is available.

@item 
bpo-23576@footnote{https://bugs.python.org/issue23576}: Avoid stalling in SSL reads when EOF has been reached in the
SSL layer but the underlying connection hasn’t been closed.

@item 
bpo-23504@footnote{https://bugs.python.org/issue23504}: Added an __all__ to the types module.

@item 
bpo-23563@footnote{https://bugs.python.org/issue23563}: Optimized utility functions in urllib.parse.

@item 
bpo-7830@footnote{https://bugs.python.org/issue7830}: Flatten nested functools.partial.

@item 
bpo-20204@footnote{https://bugs.python.org/issue20204}: Added the __module__ attribute to _tkinter classes.

@item 
bpo-19980@footnote{https://bugs.python.org/issue19980}: Improved help() for non-recognized strings.  help(‘’) now shows
the help on str.  help(‘help’) now shows the help on help(). Original
patch by Mark Lawrence.

@item 
bpo-23521@footnote{https://bugs.python.org/issue23521}: Corrected pure python implementation of timedelta division.
Eliminated OverflowError from @code{timedelta * float} for some floats;
Corrected rounding in timedelta true division.

@item 
bpo-21619@footnote{https://bugs.python.org/issue21619}: Popen objects no longer leave a zombie after exit in the with
statement if the pipe was broken.  Patch by Martin Panter.

@item 
bpo-22936@footnote{https://bugs.python.org/issue22936}: Make it possible to show local variables in tracebacks for both
the traceback module and unittest.

@item 
bpo-15955@footnote{https://bugs.python.org/issue15955}: Add an option to limit the output size in bz2.decompress().
Patch by Nikolaus Rath.

@item 
bpo-6639@footnote{https://bugs.python.org/issue6639}: Module-level turtle functions no longer raise TclError after
closing the window.

@item 
bpo-814253@footnote{https://bugs.python.org/issue814253}: Group references and conditional group references now work in
lookbehind assertions in regular expressions. (See also: bpo-9179@footnote{https://bugs.python.org/issue9179})

@item 
bpo-23215@footnote{https://bugs.python.org/issue23215}: Multibyte codecs with custom error handlers that ignores errors
consumed too much memory and raised SystemError or MemoryError. Original
patch by Aleksi Torhamo.

@item 
bpo-5700@footnote{https://bugs.python.org/issue5700}: io.FileIO() called flush() after closing the file. flush() was
not called in close() if closefd=False.

@item 
bpo-23374@footnote{https://bugs.python.org/issue23374}: Fixed pydoc failure with non-ASCII files when stdout encoding
differs from file system encoding (e.g. on Mac OS).

@item 
bpo-23481@footnote{https://bugs.python.org/issue23481}: Remove RC4 from the SSL module’s default cipher list.

@item 
bpo-21548@footnote{https://bugs.python.org/issue21548}: Fix pydoc.synopsis() and pydoc.apropos() on modules with empty
docstrings.

@item 
bpo-22885@footnote{https://bugs.python.org/issue22885}: Fixed arbitrary code execution vulnerability in the dbm.dumb
module. Original patch by Claudiu Popa.

@item 
bpo-23239@footnote{https://bugs.python.org/issue23239}: ssl.match_hostname() now supports matching of IP addresses.

@item 
bpo-23146@footnote{https://bugs.python.org/issue23146}: Fix mishandling of absolute Windows paths with forward slashes
in pathlib.

@item 
bpo-23096@footnote{https://bugs.python.org/issue23096}: Pickle representation of floats with protocol 0 now is the same
for both Python and C implementations.

@item 
bpo-19105@footnote{https://bugs.python.org/issue19105}: pprint now more efficiently uses free space at the right.

@item 
bpo-14910@footnote{https://bugs.python.org/issue14910}: Add allow_abbrev parameter to argparse.ArgumentParser. Patch by
Jonathan Paugh, Steven Bethard, paul j3 and Daniel Eriksson.

@item 
bpo-21717@footnote{https://bugs.python.org/issue21717}: tarfile.open() now supports ‘x’ (exclusive creation) mode.

@item 
bpo-23344@footnote{https://bugs.python.org/issue23344}: marshal.dumps() is now 20-25% faster on average.

@item 
bpo-20416@footnote{https://bugs.python.org/issue20416}: marshal.dumps() with protocols 3 and 4 is now 40-50% faster on
average.

@item 
bpo-23421@footnote{https://bugs.python.org/issue23421}: Fixed compression in tarfile CLI.  Patch by wdv4758h.

@item 
bpo-23367@footnote{https://bugs.python.org/issue23367}: Fix possible overflows in the unicodedata module.

@item 
bpo-23361@footnote{https://bugs.python.org/issue23361}: Fix possible overflow in Windows subprocess creation code.

@item 
logging.handlers.QueueListener now takes a respect_handler_level keyword
argument which, if set to True, will pass messages to handlers taking
handler levels into account.

@item 
bpo-19705@footnote{https://bugs.python.org/issue19705}: turtledemo now has a visual sorting algorithm demo.  Original
patch from Jason Yeo.

@item 
bpo-23801@footnote{https://bugs.python.org/issue23801}: Fix issue where cgi.FieldStorage did not always ignore the
entire preamble to a multipart body.
@end itemize

@node Build<63>,C API<47>,Library<84>,Python 3 5 0 alpha 2
@anchor{whatsnew/changelog id568}@anchor{14ac}
@subsubsection Build



@itemize -

@item 
bpo-23445@footnote{https://bugs.python.org/issue23445}: pydebug builds now use “gcc -Og” where possible, to make the
resulting executable faster.

@item 
bpo-23686@footnote{https://bugs.python.org/issue23686}: Update OS X 10.5 installer build to use OpenSSL 1.0.2a.
@end itemize

@node C API<47>,Windows<53>,Build<63>,Python 3 5 0 alpha 2
@anchor{whatsnew/changelog id569}@anchor{14ad}
@subsubsection C API



@itemize -

@item 
bpo-20204@footnote{https://bugs.python.org/issue20204}: Deprecation warning is now raised for builtin types without the
__module__ attribute.
@end itemize

@node Windows<53>,,C API<47>,Python 3 5 0 alpha 2
@anchor{whatsnew/changelog id570}@anchor{14ae}
@subsubsection Windows



@itemize -

@item 
bpo-23465@footnote{https://bugs.python.org/issue23465}: Implement 
@geindex Python Enhancement Proposals; PEP 486
PEP 486@footnote{https://www.python.org/dev/peps/pep-0486} - Make the Python Launcher aware of
virtual environments. Patch by Paul Moore.

@item 
bpo-23437@footnote{https://bugs.python.org/issue23437}: Make user scripts directory versioned on Windows. Patch by Paul
Moore.
@end itemize

@node Python 3 5 0 alpha 1,,Python 3 5 0 alpha 2,Changelog
@anchor{whatsnew/changelog python-3-5-0-alpha-1}@anchor{14af}
@subsection Python 3.5.0 alpha 1


@emph{Release date: 2015-02-08}

@menu
* Core and Builtins: Core and Builtins<71>. 
* Library: Library<85>. 
* IDLE: IDLE<50>. 
* Build: Build<64>. 
* C API: C API<48>. 
* Documentation: Documentation<56>. 
* Tests: Tests<59>. 
* Tools/Demos: Tools/Demos<35>. 
* Windows: Windows<54>. 

@end menu

@node Core and Builtins<71>,Library<85>,,Python 3 5 0 alpha 1
@anchor{whatsnew/changelog id571}@anchor{14b0}
@subsubsection Core and Builtins



@itemize -

@item 
bpo-23285@footnote{https://bugs.python.org/issue23285}: PEP 475 - EINTR handling.

@item 
bpo-22735@footnote{https://bugs.python.org/issue22735}: Fix many edge cases (including crashes) involving custom mro()
implementations.

@item 
bpo-22896@footnote{https://bugs.python.org/issue22896}: Avoid using PyObject_AsCharBuffer(), PyObject_AsReadBuffer()
and PyObject_AsWriteBuffer().

@item 
bpo-21295@footnote{https://bugs.python.org/issue21295}: Revert some changes (bpo-16795@footnote{https://bugs.python.org/issue16795}) to AST line numbers and
column offsets that constituted a regression.

@item 
bpo-22986@footnote{https://bugs.python.org/issue22986}: Allow changing an object’s __class__ between a dynamic type and
static type in some cases.

@item 
bpo-15859@footnote{https://bugs.python.org/issue15859}: PyUnicode_EncodeFSDefault(), PyUnicode_EncodeMBCS() and
PyUnicode_EncodeCodePage() now raise an exception if the object is not a
Unicode object. For PyUnicode_EncodeFSDefault(), it was already the case
on platforms other than Windows. Patch written by Campbell Barton.

@item 
bpo-21408@footnote{https://bugs.python.org/issue21408}: The default __ne__() now returns NotImplemented if __eq__()
returned NotImplemented.  Original patch by Martin Panter.

@item 
bpo-23321@footnote{https://bugs.python.org/issue23321}: Fixed a crash in str.decode() when error handler returned
replacement string longer than malformed input data.

@item 
bpo-22286@footnote{https://bugs.python.org/issue22286}: The “backslashreplace” error handlers now works with decoding
and translating.

@item 
bpo-23253@footnote{https://bugs.python.org/issue23253}: Delay-load ShellExecute[AW] in os.startfile for reduced startup
overhead on Windows.

@item 
bpo-22038@footnote{https://bugs.python.org/issue22038}: pyatomic.h now uses stdatomic.h or GCC built-in functions for
atomic memory access if available. Patch written by Vitor de Lima and
Gustavo Temple.

@item 
bpo-20284@footnote{https://bugs.python.org/issue20284}: %-interpolation (aka printf) formatting added for bytes and
bytearray.

@item 
bpo-23048@footnote{https://bugs.python.org/issue23048}: Fix jumping out of an infinite while loop in the pdb.

@item 
bpo-20335@footnote{https://bugs.python.org/issue20335}: bytes constructor now raises TypeError when encoding or errors
is specified with non-string argument.  Based on patch by Renaud Blanch.

@item 
bpo-22834@footnote{https://bugs.python.org/issue22834}: If the current working directory ends up being set to a
non-existent directory then import will no longer raise FileNotFoundError.

@item 
bpo-22869@footnote{https://bugs.python.org/issue22869}: Move the interpreter startup & shutdown code to a new dedicated
pylifecycle.c module

@item 
bpo-22847@footnote{https://bugs.python.org/issue22847}: Improve method cache efficiency.

@item 
bpo-22335@footnote{https://bugs.python.org/issue22335}: Fix crash when trying to enlarge a bytearray to 0x7fffffff
bytes on a 32-bit platform.

@item 
bpo-22653@footnote{https://bugs.python.org/issue22653}: Fix an assertion failure in debug mode when doing a reentrant
dict insertion in debug mode.

@item 
bpo-22643@footnote{https://bugs.python.org/issue22643}: Fix integer overflow in Unicode case operations (upper, lower,
title, swapcase, casefold).

@item 
bpo-17636@footnote{https://bugs.python.org/issue17636}: Circular imports involving relative imports are now supported.

@item 
bpo-22604@footnote{https://bugs.python.org/issue22604}: Fix assertion error in debug mode when dividing a complex
number by (nan+0j).

@item 
bpo-21052@footnote{https://bugs.python.org/issue21052}: Do not raise ImportWarning when sys.path_hooks or sys.meta_path
are set to None.

@item 
bpo-16518@footnote{https://bugs.python.org/issue16518}: Use ‘bytes-like object required’ in error messages that
previously used the far more cryptic “‘x’ does not support the buffer
protocol.

@item 
bpo-22470@footnote{https://bugs.python.org/issue22470}: Fixed integer overflow issues in “backslashreplace”,
“xmlcharrefreplace”, and “surrogatepass” error handlers.

@item 
bpo-22540@footnote{https://bugs.python.org/issue22540}: speed up @code{PyObject_IsInstance} and @code{PyObject_IsSubclass} in the
common case that the second argument has metaclass @ref{89e,,type}.

@item 
bpo-18711@footnote{https://bugs.python.org/issue18711}: Add a new @code{PyErr_FormatV} function, similar to @code{PyErr_Format}
but accepting a @code{va_list} argument.

@item 
bpo-22520@footnote{https://bugs.python.org/issue22520}: Fix overflow checking when generating the repr of a unicode
object.

@item 
bpo-22519@footnote{https://bugs.python.org/issue22519}: Fix overflow checking in PyBytes_Repr.

@item 
bpo-22518@footnote{https://bugs.python.org/issue22518}: Fix integer overflow issues in latin-1 encoding.

@item 
bpo-16324@footnote{https://bugs.python.org/issue16324}: _charset parameter of MIMEText now also accepts
email.charset.Charset instances. Initial patch by Claude Paroz.

@item 
bpo-1764286@footnote{https://bugs.python.org/issue1764286}: Fix inspect.getsource() to support decorated functions. Patch
by Claudiu Popa.

@item 
bpo-18554@footnote{https://bugs.python.org/issue18554}: os.__all__ includes posix functions.

@item 
bpo-21391@footnote{https://bugs.python.org/issue21391}: Use os.path.abspath in the shutil module.

@item 
bpo-11471@footnote{https://bugs.python.org/issue11471}: avoid generating a JUMP_FORWARD instruction at the end of an
if-block if there is no else-clause.  Original patch by Eugene Toder.

@item 
bpo-22215@footnote{https://bugs.python.org/issue22215}: Now ValueError is raised instead of TypeError when str or bytes
argument contains not permitted null character or byte.

@item 
bpo-22258@footnote{https://bugs.python.org/issue22258}: Fix the internal function set_inheritable() on Illumos. This
platform exposes the function @code{ioctl(FIOCLEX)}, but calling it fails
with errno is ENOTTY: “Inappropriate ioctl for device”. set_inheritable()
now falls back to the slower @code{fcntl()} (@code{F_GETFD} and then
@code{F_SETFD}).

@item 
bpo-21389@footnote{https://bugs.python.org/issue21389}: Displaying the __qualname__ of the underlying function in the
repr of a bound method.

@item 
bpo-22206@footnote{https://bugs.python.org/issue22206}: Using pthread, PyThread_create_key() now sets errno to ENOMEM
and returns -1 (error) on integer overflow.

@item 
bpo-20184@footnote{https://bugs.python.org/issue20184}: Argument Clinic based signature introspection added for 30 of
the builtin functions.

@item 
bpo-22116@footnote{https://bugs.python.org/issue22116}: C functions and methods (of the ‘builtin_function_or_method’
type) can now be weakref’ed.  Patch by Wei Wu.

@item 
bpo-22077@footnote{https://bugs.python.org/issue22077}: Improve index error messages for bytearrays, bytes, lists, and
tuples by adding ‘or slices’. Added ‘, not <typename>’ for bytearrays.
Original patch by Claudiu Popa.

@item 
bpo-20179@footnote{https://bugs.python.org/issue20179}: Apply Argument Clinic to bytes and bytearray. Patch by Tal
Einat.

@item 
bpo-22082@footnote{https://bugs.python.org/issue22082}: Clear interned strings in slotdefs.

@item 
Upgrade Unicode database to Unicode 7.0.0.

@item 
bpo-21897@footnote{https://bugs.python.org/issue21897}: Fix a crash with the f_locals attribute with closure variables
when frame.clear() has been called.

@item 
bpo-21205@footnote{https://bugs.python.org/issue21205}: Add a new @code{__qualname__} attribute to generator, the
qualified name, and use it in the representation of a generator
(@code{repr(gen)}). The default name of the generator (@code{__name__}
attribute) is now get from the function instead of the code. Use
@code{gen.gi_code.co_name} to get the name of the code.

@item 
bpo-21669@footnote{https://bugs.python.org/issue21669}: With the aid of heuristics in SyntaxError.__init__, the parser
now attempts to generate more meaningful (or at least more search engine
friendly) error messages when “exec” and “print” are used as statements.

@item 
bpo-21642@footnote{https://bugs.python.org/issue21642}: In the conditional if-else expression, allow an integer written
with no space between itself and the @code{else} keyword (e.g. @code{True if
42else False}) to be valid syntax.

@item 
bpo-21523@footnote{https://bugs.python.org/issue21523}: Fix over-pessimistic computation of the stack effect of some
opcodes in the compiler.  This also fixes a quadratic compilation time
issue noticeable when compiling code with a large number of “and” and “or”
operators.

@item 
bpo-21418@footnote{https://bugs.python.org/issue21418}: Fix a crash in the builtin function super() when called without
argument and without current frame (ex: embedded Python).

@item 
bpo-21425@footnote{https://bugs.python.org/issue21425}: Fix flushing of standard streams in the interactive
interpreter.

@item 
bpo-21435@footnote{https://bugs.python.org/issue21435}: In rare cases, when running finalizers on objects in cyclic
trash a bad pointer dereference could occur due to a subtle flaw in
internal iteration logic.

@item 
bpo-21377@footnote{https://bugs.python.org/issue21377}: PyBytes_Concat() now tries to concatenate in-place when the
first argument has a reference count of 1.  Patch by Nikolaus Rath.

@item 
bpo-20355@footnote{https://bugs.python.org/issue20355}: -W command line options now have higher priority than the
PYTHONWARNINGS environment variable.  Patch by Arfrever.

@item 
bpo-21274@footnote{https://bugs.python.org/issue21274}: Define PATH_MAX for GNU/Hurd in Python/pythonrun.c.

@item 
bpo-20904@footnote{https://bugs.python.org/issue20904}: Support setting FPU precision on m68k.

@item 
bpo-21209@footnote{https://bugs.python.org/issue21209}: Fix sending tuples to custom generator objects with the yield
from syntax.

@item 
bpo-21193@footnote{https://bugs.python.org/issue21193}: pow(a, b, c) now raises ValueError rather than TypeError when b
is negative. Patch by Josh Rosenberg.

@item 
bpo-21176@footnote{https://bugs.python.org/issue21176}: PEP 465: Add the ‘@@’ operator for matrix multiplication.

@item 
bpo-21134@footnote{https://bugs.python.org/issue21134}: Fix segfault when str is called on an uninitialized
UnicodeEncodeError, UnicodeDecodeError, or UnicodeTranslateError object.

@item 
bpo-19537@footnote{https://bugs.python.org/issue19537}: Fix PyUnicode_DATA() alignment under m68k.  Patch by Andreas
Schwab.

@item 
bpo-20929@footnote{https://bugs.python.org/issue20929}: Add a type cast to avoid shifting a negative number.

@item 
bpo-20731@footnote{https://bugs.python.org/issue20731}: Properly position in source code files even if they are opened
in text mode. Patch by Serhiy Storchaka.

@item 
bpo-20637@footnote{https://bugs.python.org/issue20637}: Key-sharing now also works for instance dictionaries of
subclasses.  Patch by Peter Ingebretson.

@item 
bpo-8297@footnote{https://bugs.python.org/issue8297}: Attributes missing from modules now include the module name in
the error text.  Original patch by ysj.ray.

@item 
bpo-19995@footnote{https://bugs.python.org/issue19995}: %c, %o, %x, and %X now raise TypeError on non-integer input.

@item 
bpo-19655@footnote{https://bugs.python.org/issue19655}: The ASDL parser - used by the build process to generate code
for managing the Python AST in C - was rewritten. The new parser is self
contained and does not require to carry long the spark.py parser-generator
library; spark.py was removed from the source base.

@item 
bpo-12546@footnote{https://bugs.python.org/issue12546}: Allow @code{\x00} to be used as a fill character when using str,
int, float, and complex __format__ methods.

@item 
bpo-20480@footnote{https://bugs.python.org/issue20480}: Add ipaddress.reverse_pointer. Patch by Leon Weber.

@item 
bpo-13598@footnote{https://bugs.python.org/issue13598}: Modify string.Formatter to support auto-numbering of
replacement fields. It now matches the behavior of str.format() in this
regard. Patches by Phil Elson and Ramchandra Apte.

@item 
bpo-8931@footnote{https://bugs.python.org/issue8931}: Make alternate formatting (‘#’) for type ‘c’ raise an exception.
In versions prior to 3.5, ‘#’ with ‘c’ had no effect. Now specifying it is
an error. Patch by Torsten Landschoff.

@item 
bpo-23165@footnote{https://bugs.python.org/issue23165}: Perform overflow checks before allocating memory in the
_Py_char2wchar function.
@end itemize

@node Library<85>,IDLE<50>,Core and Builtins<71>,Python 3 5 0 alpha 1
@anchor{whatsnew/changelog id572}@anchor{14b1}
@subsubsection Library



@itemize -

@item 
bpo-23399@footnote{https://bugs.python.org/issue23399}: pyvenv creates relative symlinks where possible.

@item 
bpo-20289@footnote{https://bugs.python.org/issue20289}: cgi.FieldStorage() now supports the context management
protocol.

@item 
bpo-13128@footnote{https://bugs.python.org/issue13128}: Print response headers for CONNECT requests when debuglevel >
0. Patch by Demian Brecht.

@item 
bpo-15381@footnote{https://bugs.python.org/issue15381}: Optimized io.BytesIO to make less allocations and copyings.

@item 
bpo-22818@footnote{https://bugs.python.org/issue22818}: Splitting on a pattern that could match an empty string now
raises a warning.  Patterns that can only match empty strings are now
rejected.

@item 
bpo-23099@footnote{https://bugs.python.org/issue23099}: Closing io.BytesIO with exported buffer is rejected now to
prevent corrupting exported buffer.

@item 
bpo-23326@footnote{https://bugs.python.org/issue23326}: Removed __ne__ implementations.  Since fixing default __ne__
implementation in bpo-21408@footnote{https://bugs.python.org/issue21408} they are redundant.

@item 
bpo-23363@footnote{https://bugs.python.org/issue23363}: Fix possible overflow in itertools.permutations.

@item 
bpo-23364@footnote{https://bugs.python.org/issue23364}: Fix possible overflow in itertools.product.

@item 
bpo-23366@footnote{https://bugs.python.org/issue23366}: Fixed possible integer overflow in itertools.combinations.

@item 
bpo-23369@footnote{https://bugs.python.org/issue23369}: Fixed possible integer overflow in
_json.encode_basestring_ascii.

@item 
bpo-23353@footnote{https://bugs.python.org/issue23353}: Fix the exception handling of generators in
PyEval_EvalFrameEx(). At entry, save or swap the exception state even if
PyEval_EvalFrameEx() is called with throwflag=0. At exit, the exception
state is now always restored or swapped, not only if why is WHY_YIELD or
WHY_RETURN. Patch co-written with Antoine Pitrou.

@item 
bpo-14099@footnote{https://bugs.python.org/issue14099}: Restored support of writing ZIP files to tellable but
non-seekable streams.

@item 
bpo-14099@footnote{https://bugs.python.org/issue14099}: Writing to ZipFile and reading multiple ZipExtFiles is
threadsafe now.

@item 
bpo-19361@footnote{https://bugs.python.org/issue19361}: JSON decoder now raises JSONDecodeError instead of ValueError.

@item 
bpo-18518@footnote{https://bugs.python.org/issue18518}: timeit now rejects statements which can’t be compiled outside a
function or a loop (e.g. “return” or “break”).

@item 
bpo-23094@footnote{https://bugs.python.org/issue23094}: Fixed readline with frames in Python implementation of pickle.

@item 
bpo-23268@footnote{https://bugs.python.org/issue23268}: Fixed bugs in the comparison of ipaddress classes.

@item 
bpo-21408@footnote{https://bugs.python.org/issue21408}: Removed incorrect implementations of __ne__() which didn’t
returned NotImplemented if __eq__() returned NotImplemented.  The default
__ne__() now works correctly.

@item 
bpo-19996@footnote{https://bugs.python.org/issue19996}: @code{email.feedparser.FeedParser} now handles (malformed)
headers with no key rather than assuming the body has started.

@item 
bpo-20188@footnote{https://bugs.python.org/issue20188}: Support Application-Layer Protocol Negotiation (ALPN) in the
ssl module.

@item 
bpo-23133@footnote{https://bugs.python.org/issue23133}: Pickling of ipaddress objects now produces more compact and
portable representation.

@item 
bpo-23248@footnote{https://bugs.python.org/issue23248}: Update ssl error codes from latest OpenSSL git master.

@item 
bpo-23266@footnote{https://bugs.python.org/issue23266}: Much faster implementation of ipaddress.collapse_addresses()
when there are many non-consecutive addresses.

@item 
bpo-23098@footnote{https://bugs.python.org/issue23098}: 64-bit dev_t is now supported in the os module.

@item 
bpo-21817@footnote{https://bugs.python.org/issue21817}: When an exception is raised in a task submitted to a
ProcessPoolExecutor, the remote traceback is now displayed in the parent
process.  Patch by Claudiu Popa.

@item 
bpo-15955@footnote{https://bugs.python.org/issue15955}: Add an option to limit output size when decompressing LZMA
data.  Patch by Nikolaus Rath and Martin Panter.

@item 
bpo-23250@footnote{https://bugs.python.org/issue23250}: In the http.cookies module, capitalize “HttpOnly” and “Secure”
as they are written in the standard.

@item 
bpo-23063@footnote{https://bugs.python.org/issue23063}: In the distutils’ check command, fix parsing of reST with code
or code-block directives.

@item 
bpo-23209@footnote{https://bugs.python.org/issue23209}: selectors.BaseSelector.get_key() now raises a RuntimeError if
the selector is closed. And selectors.BaseSelector.close() now clears its
internal reference to the selector mapping to break a reference cycle.
Initial patch written by Martin Richard. (See also: bpo-23225@footnote{https://bugs.python.org/issue23225})

@item 
bpo-17911@footnote{https://bugs.python.org/issue17911}: Provide a way to seed the linecache for a PEP-302 module
without actually loading the code.

@item 
bpo-17911@footnote{https://bugs.python.org/issue17911}: Provide a new object API for traceback, including the ability
to not lookup lines at all until the traceback is actually rendered,
without any trace of the original objects being kept alive.

@item 
bpo-19777@footnote{https://bugs.python.org/issue19777}: Provide a home() classmethod on Path objects.  Contributed by
Victor Salgado and Mayank Tripathi.

@item 
bpo-23206@footnote{https://bugs.python.org/issue23206}: Make @code{json.dumps(..., ensure_ascii=False)} as fast as the
default case of @code{ensure_ascii=True}.  Patch by Naoki Inada.

@item 
bpo-23185@footnote{https://bugs.python.org/issue23185}: Add math.inf and math.nan constants.

@item 
bpo-23186@footnote{https://bugs.python.org/issue23186}: Add ssl.SSLObject.shared_ciphers() and
ssl.SSLSocket.shared_ciphers() to fetch the client’s list ciphers sent at
handshake.

@item 
bpo-23143@footnote{https://bugs.python.org/issue23143}: Remove compatibility with OpenSSLs older than 0.9.8.

@item 
bpo-23132@footnote{https://bugs.python.org/issue23132}: Improve performance and introspection support of comparison
methods created by functool.total_ordering.

@item 
bpo-19776@footnote{https://bugs.python.org/issue19776}: Add an expanduser() method on Path objects.

@item 
bpo-23112@footnote{https://bugs.python.org/issue23112}: Fix SimpleHTTPServer to correctly carry the query string and
fragment when it redirects to add a trailing slash.

@item 
bpo-21793@footnote{https://bugs.python.org/issue21793}: Added http.HTTPStatus enums (i.e. HTTPStatus.OK,
HTTPStatus.NOT_FOUND). Patch by Demian Brecht.

@item 
bpo-23093@footnote{https://bugs.python.org/issue23093}: In the io, module allow more operations to work on detached
streams.

@item 
bpo-23111@footnote{https://bugs.python.org/issue23111}: In the ftplib, make ssl.PROTOCOL_SSLv23 the default protocol
version.

@item 
bpo-22585@footnote{https://bugs.python.org/issue22585}: On OpenBSD 5.6 and newer, os.urandom() now calls getentropy(),
instead of reading /dev/urandom, to get pseudo-random bytes.

@item 
bpo-19104@footnote{https://bugs.python.org/issue19104}: pprint now produces evaluable output for wrapped strings.

@item 
bpo-23071@footnote{https://bugs.python.org/issue23071}: Added missing names to codecs.__all__.  Patch by Martin Panter.

@item 
bpo-22783@footnote{https://bugs.python.org/issue22783}: Pickling now uses the NEWOBJ opcode instead of the NEWOBJ_EX
opcode if possible.

@item 
bpo-15513@footnote{https://bugs.python.org/issue15513}: Added a __sizeof__ implementation for pickle classes.

@item 
bpo-19858@footnote{https://bugs.python.org/issue19858}: pickletools.optimize() now aware of the MEMOIZE opcode, can
produce more compact result and no longer produces invalid output if input
data contains MEMOIZE opcodes together with PUT or BINPUT opcodes.

@item 
bpo-22095@footnote{https://bugs.python.org/issue22095}: Fixed HTTPConnection.set_tunnel with default port.  The port
value in the host header was set to “None”.  Patch by Demian Brecht.

@item 
bpo-23016@footnote{https://bugs.python.org/issue23016}: A warning no longer produces an AttributeError when the program
is run with pythonw.exe.

@item 
bpo-21775@footnote{https://bugs.python.org/issue21775}: shutil.copytree(): fix crash when copying to VFAT. An exception
handler assumed that OSError objects always have a ‘winerror’ attribute.
That is not the case, so the exception handler itself raised
AttributeError when run on Linux (and, presumably, any other non-Windows
OS). Patch by Greg Ward.

@item 
bpo-1218234@footnote{https://bugs.python.org/issue1218234}: Fix inspect.getsource() to load updated source of reloaded
module. Initial patch by Berker Peksag.

@item 
bpo-21740@footnote{https://bugs.python.org/issue21740}: Support wrapped callables in doctest. Patch by Claudiu Popa.

@item 
bpo-23009@footnote{https://bugs.python.org/issue23009}: Make sure selectors.EpollSelector.select() works when no FD is
registered.

@item 
bpo-22959@footnote{https://bugs.python.org/issue22959}: In the constructor of http.client.HTTPSConnection, prefer the
context’s check_hostname attribute over the @emph{check_hostname} parameter.

@item 
bpo-22696@footnote{https://bugs.python.org/issue22696}: Add function @ref{9e1,,sys.is_finalizing()} to know about
interpreter shutdown.

@item 
bpo-16043@footnote{https://bugs.python.org/issue16043}: Add a default limit for the amount of data
xmlrpclib.gzip_decode will return. This resolves CVE-2013-1753.

@item 
bpo-14099@footnote{https://bugs.python.org/issue14099}: ZipFile.open() no longer reopen the underlying file.  Objects
returned by ZipFile.open() can now operate independently of the ZipFile
even if the ZipFile was created by passing in a file-like object as the
first argument to the constructor.

@item 
bpo-22966@footnote{https://bugs.python.org/issue22966}: Fix __pycache__ pyc file name clobber when pyc_compile is asked
to compile a source file containing multiple dots in the source file name.

@item 
bpo-21971@footnote{https://bugs.python.org/issue21971}: Update turtledemo doc and add module to the index.

@item 
bpo-21032@footnote{https://bugs.python.org/issue21032}: Fixed socket leak if HTTPConnection.getresponse() fails.
Original patch by Martin Panter.

@item 
bpo-22407@footnote{https://bugs.python.org/issue22407}: Deprecated the use of re.LOCALE flag with str patterns or
re.ASCII. It was newer worked.

@item 
bpo-22902@footnote{https://bugs.python.org/issue22902}: The “ip” command is now used on Linux to determine MAC address
in uuid.getnode().  Pach by Bruno Cauet.

@item 
bpo-22960@footnote{https://bugs.python.org/issue22960}: Add a context argument to xmlrpclib.ServerProxy constructor.

@item 
bpo-22389@footnote{https://bugs.python.org/issue22389}: Add contextlib.redirect_stderr().

@item 
bpo-21356@footnote{https://bugs.python.org/issue21356}: Make ssl.RAND_egd() optional to support LibreSSL. The
availability of the function is checked during the compilation. Patch
written by Bernard Spil.

@item 
bpo-22915@footnote{https://bugs.python.org/issue22915}: SAX parser now supports files opened with file descriptor or
bytes path.

@item 
bpo-22609@footnote{https://bugs.python.org/issue22609}: Constructors and update methods of mapping classes in the
collections module now accept the self keyword argument.

@item 
bpo-22940@footnote{https://bugs.python.org/issue22940}: Add readline.append_history_file.

@item 
bpo-19676@footnote{https://bugs.python.org/issue19676}: Added the “namereplace” error handler.

@item 
bpo-22788@footnote{https://bugs.python.org/issue22788}: Add @emph{context} parameter to logging.handlers.HTTPHandler.

@item 
bpo-22921@footnote{https://bugs.python.org/issue22921}: Allow SSLContext to take the @emph{hostname} parameter even if
OpenSSL doesn’t support SNI.

@item 
bpo-22894@footnote{https://bugs.python.org/issue22894}: TestCase.subTest() would cause the test suite to be stopped
when in failfast mode, even in the absence of failures.

@item 
bpo-22796@footnote{https://bugs.python.org/issue22796}: HTTP cookie parsing is now stricter, in order to protect
against potential injection attacks.

@item 
bpo-22370@footnote{https://bugs.python.org/issue22370}: Windows detection in pathlib is now more robust.

@item 
bpo-22841@footnote{https://bugs.python.org/issue22841}: Reject coroutines in asyncio add_signal_handler(). Patch by
Ludovic.Gasc.

@item 
bpo-19494@footnote{https://bugs.python.org/issue19494}: Added urllib.request.HTTPBasicPriorAuthHandler. Patch by Matej
Cepl.

@item 
bpo-22578@footnote{https://bugs.python.org/issue22578}: Added attributes to the re.error class.

@item 
bpo-22849@footnote{https://bugs.python.org/issue22849}: Fix possible double free in the io.TextIOWrapper constructor.

@item 
bpo-12728@footnote{https://bugs.python.org/issue12728}: Different Unicode characters having the same uppercase but
different lowercase are now matched in case-insensitive regular
expressions.

@item 
bpo-22821@footnote{https://bugs.python.org/issue22821}: Fixed fcntl() with integer argument on 64-bit big-endian
platforms.

@item 
bpo-21650@footnote{https://bugs.python.org/issue21650}: Add an @code{--sort-keys} option to json.tool CLI.

@item 
bpo-22824@footnote{https://bugs.python.org/issue22824}: Updated reprlib output format for sets to use set literals.
Patch contributed by Berker Peksag.

@item 
bpo-22824@footnote{https://bugs.python.org/issue22824}: Updated reprlib output format for arrays to display empty
arrays without an unnecessary empty list.  Suggested by Serhiy Storchaka.

@item 
bpo-22406@footnote{https://bugs.python.org/issue22406}: Fixed the uu_codec codec incorrectly ported to 3.x. Based on
patch by Martin Panter.

@item 
bpo-17293@footnote{https://bugs.python.org/issue17293}: uuid.getnode() now determines MAC address on AIX using netstat.
Based on patch by Aivars Kalvāns.

@item 
bpo-22769@footnote{https://bugs.python.org/issue22769}: Fixed ttk.Treeview.tag_has() when called without arguments.

@item 
bpo-22417@footnote{https://bugs.python.org/issue22417}: Verify certificates by default in httplib (PEP 476).

@item 
bpo-22775@footnote{https://bugs.python.org/issue22775}: Fixed unpickling of http.cookies.SimpleCookie with protocol 2
and above. Patch by Tim Graham.

@item 
bpo-22776@footnote{https://bugs.python.org/issue22776}: Brought excluded code into the scope of a try block in
SysLogHandler.emit().

@item 
bpo-22665@footnote{https://bugs.python.org/issue22665}: Add missing get_terminal_size and SameFileError to
shutil.__all__.

@item 
bpo-6623@footnote{https://bugs.python.org/issue6623}: Remove deprecated Netrc class in the ftplib module. Patch by
Matt Chaput.

@item 
bpo-17381@footnote{https://bugs.python.org/issue17381}: Fixed handling of case-insensitive ranges in regular
expressions.

@item 
bpo-22410@footnote{https://bugs.python.org/issue22410}: Module level functions in the re module now cache compiled
locale-dependent regular expressions taking into account the locale.

@item 
bpo-22759@footnote{https://bugs.python.org/issue22759}: Query methods on pathlib.Path() (exists(), is_dir(), etc.) now
return False when the underlying stat call raises NotADirectoryError.

@item 
bpo-8876@footnote{https://bugs.python.org/issue8876}: distutils now falls back to copying files when hard linking
doesn’t work. This allows use with special filesystems such as VirtualBox
shared folders.

@item 
bpo-22217@footnote{https://bugs.python.org/issue22217}: Implemented reprs of classes in the zipfile module.

@item 
bpo-22457@footnote{https://bugs.python.org/issue22457}: Honour load_tests in the start_dir of discovery.

@item 
bpo-18216@footnote{https://bugs.python.org/issue18216}: gettext now raises an error when a .mo file has an unsupported
major version number.  Patch by Aaron Hill.

@item 
bpo-13918@footnote{https://bugs.python.org/issue13918}: Provide a locale.delocalize() function which can remove
locale-specific number formatting from a string representing a number,
without then converting it to a specific type.  Patch by Cédric Krier.

@item 
bpo-22676@footnote{https://bugs.python.org/issue22676}: Make the pickling of global objects which don’t have a
__module__ attribute less slow.

@item 
bpo-18853@footnote{https://bugs.python.org/issue18853}: Fixed ResourceWarning in shlex.__nain__.

@item 
bpo-9351@footnote{https://bugs.python.org/issue9351}: Defaults set with set_defaults on an argparse subparser are no
longer ignored when also set on the parent parser.

@item 
bpo-7559@footnote{https://bugs.python.org/issue7559}: unittest test loading ImportErrors are reported as import errors
with their import exception rather than as attribute errors after the
import has already failed.

@item 
bpo-19746@footnote{https://bugs.python.org/issue19746}: Make it possible to examine the errors from unittest discovery
without executing the test suite. The new @code{errors} attribute on TestLoader
exposes these non-fatal errors encountered during discovery.

@item 
bpo-21991@footnote{https://bugs.python.org/issue21991}: Make email.headerregistry’s header ‘params’ attributes be
read-only (MappingProxyType).  Previously the dictionary was modifiable
but a new one was created on each access of the attribute.

@item 
bpo-22638@footnote{https://bugs.python.org/issue22638}: SSLv3 is now disabled throughout the standard library. It can
still be enabled by instantiating a SSLContext manually.

@item 
bpo-22641@footnote{https://bugs.python.org/issue22641}: In asyncio, the default SSL context for client connections is
now created using ssl.create_default_context(), for stronger security.

@item 
bpo-17401@footnote{https://bugs.python.org/issue17401}: Include closefd in io.FileIO repr.

@item 
bpo-21338@footnote{https://bugs.python.org/issue21338}: Add silent mode for compileall. quiet parameters of
compile_@{dir, file, path@} functions now have a multilevel value. Also, -q
option of the CLI now have a multilevel value. Patch by Thomas Kluyver.

@item 
bpo-20152@footnote{https://bugs.python.org/issue20152}: Convert the array and cmath modules to Argument Clinic.

@item 
bpo-18643@footnote{https://bugs.python.org/issue18643}: Add socket.socketpair() on Windows.

@item 
bpo-22435@footnote{https://bugs.python.org/issue22435}: Fix a file descriptor leak when socketserver bind fails.

@item 
bpo-13096@footnote{https://bugs.python.org/issue13096}: Fixed segfault in CTypes POINTER handling of large values.

@item 
bpo-11694@footnote{https://bugs.python.org/issue11694}: Raise ConversionError in xdrlib as documented.  Patch by Filip
Gruszczyński and Claudiu Popa.

@item 
bpo-19380@footnote{https://bugs.python.org/issue19380}: Optimized parsing of regular expressions.

@item 
bpo-1519638@footnote{https://bugs.python.org/issue1519638}: Now unmatched groups are replaced with empty strings in
re.sub() and re.subn().

@item 
bpo-18615@footnote{https://bugs.python.org/issue18615}: sndhdr.what/whathdr now return a namedtuple.

@item 
bpo-22462@footnote{https://bugs.python.org/issue22462}: Fix pyexpat’s creation of a dummy frame to make it appear in
exception tracebacks.

@item 
bpo-21965@footnote{https://bugs.python.org/issue21965}: Add support for in-memory SSL to the ssl module.  Patch by
Geert Jansen.

@item 
bpo-21173@footnote{https://bugs.python.org/issue21173}: Fix len() on a WeakKeyDictionary when .clear() was called with
an iterator alive.

@item 
bpo-11866@footnote{https://bugs.python.org/issue11866}: Eliminated race condition in the computation of names for new
threads.

@item 
bpo-21905@footnote{https://bugs.python.org/issue21905}: Avoid RuntimeError in pickle.whichmodule() when sys.modules is
mutated while iterating.  Patch by Olivier Grisel.

@item 
bpo-11271@footnote{https://bugs.python.org/issue11271}: concurrent.futures.Executor.map() now takes a @emph{chunksize}
argument to allow batching of tasks in child processes and improve
performance of ProcessPoolExecutor.  Patch by Dan O’Reilly.

@item 
bpo-21883@footnote{https://bugs.python.org/issue21883}: os.path.join() and os.path.relpath() now raise a TypeError with
more helpful error message for unsupported or mismatched types of
arguments.

@item 
bpo-22219@footnote{https://bugs.python.org/issue22219}: The zipfile module CLI now adds entries for directories
(including empty directories) in ZIP file.

@item 
bpo-22449@footnote{https://bugs.python.org/issue22449}: In the ssl.SSLContext.load_default_certs, consult the
environmental variables SSL_CERT_DIR and SSL_CERT_FILE on Windows.

@item 
bpo-22508@footnote{https://bugs.python.org/issue22508}: The email.__version__ variable has been removed; the email code
is no longer shipped separately from the stdlib, and __version__ hasn’t
been updated in several releases.

@item 
bpo-20076@footnote{https://bugs.python.org/issue20076}: Added non derived UTF-8 aliases to locale aliases table.

@item 
bpo-20079@footnote{https://bugs.python.org/issue20079}: Added locales supported in glibc 2.18 to locale alias table.

@item 
bpo-20218@footnote{https://bugs.python.org/issue20218}: Added convenience methods read_text/write_text and read_bytes/
write_bytes to pathlib.Path objects.

@item 
bpo-22396@footnote{https://bugs.python.org/issue22396}: On 32-bit AIX platform, don’t expose os.posix_fadvise() nor
os.posix_fallocate() because their prototypes in system headers are wrong.

@item 
bpo-22517@footnote{https://bugs.python.org/issue22517}: When an io.BufferedRWPair object is deallocated, clear its
weakrefs.

@item 
bpo-22437@footnote{https://bugs.python.org/issue22437}: Number of capturing groups in regular expression is no longer
limited by 100.

@item 
bpo-17442@footnote{https://bugs.python.org/issue17442}: InteractiveInterpreter now displays the full chained traceback
in its showtraceback method, to match the built in interactive
interpreter.

@item 
bpo-23392@footnote{https://bugs.python.org/issue23392}: Added tests for marshal C API that works with FILE*.

@item 
bpo-10510@footnote{https://bugs.python.org/issue10510}: distutils register and upload methods now use HTML standards
compliant CRLF line endings.

@item 
bpo-9850@footnote{https://bugs.python.org/issue9850}: Fixed macpath.join() for empty first component.  Patch by Oleg
Oshmyan.

@item 
bpo-5309@footnote{https://bugs.python.org/issue5309}: distutils’ build and build_ext commands now accept a @code{-j}
option to enable parallel building of extension modules.

@item 
bpo-22448@footnote{https://bugs.python.org/issue22448}: Improve canceled timer handles cleanup to prevent unbound
memory usage. Patch by Joshua Moore-Oliva.

@item 
bpo-22427@footnote{https://bugs.python.org/issue22427}: TemporaryDirectory no longer attempts to clean up twice when
used in the with statement in generator.

@item 
bpo-22362@footnote{https://bugs.python.org/issue22362}: Forbidden ambiguous octal escapes out of range 0-0o377 in
regular expressions.

@item 
bpo-20912@footnote{https://bugs.python.org/issue20912}: Now directories added to ZIP file have correct Unix and MS-DOS
directory attributes.

@item 
bpo-21866@footnote{https://bugs.python.org/issue21866}: ZipFile.close() no longer writes ZIP64 central directory
records if allowZip64 is false.

@item 
bpo-22278@footnote{https://bugs.python.org/issue22278}: Fix urljoin problem with relative urls, a regression observed
after changes to issue22118 were submitted.

@item 
bpo-22415@footnote{https://bugs.python.org/issue22415}: Fixed debugging output of the GROUPREF_EXISTS opcode in the re
module. Removed trailing spaces in debugging output.

@item 
bpo-22423@footnote{https://bugs.python.org/issue22423}: Unhandled exception in thread no longer causes unhandled
AttributeError when sys.stderr is None.

@item 
bpo-21332@footnote{https://bugs.python.org/issue21332}: Ensure that @code{bufsize=1} in subprocess.Popen() selects line
buffering, rather than block buffering.  Patch by Akira Li.

@item 
bpo-21091@footnote{https://bugs.python.org/issue21091}: Fix API bug: email.message.EmailMessage.is_attachment is now a
method.

@item 
bpo-21079@footnote{https://bugs.python.org/issue21079}: Fix email.message.EmailMessage.is_attachment to return the
correct result when the header has parameters as well as a value.

@item 
bpo-22247@footnote{https://bugs.python.org/issue22247}: Add NNTPError to nntplib.__all__.

@item 
bpo-22366@footnote{https://bugs.python.org/issue22366}: urllib.request.urlopen will accept a context object
(SSLContext) as an argument which will then be used for HTTPS connection.
Patch by Alex Gaynor.

@item 
bpo-4180@footnote{https://bugs.python.org/issue4180}: The warnings registries are now reset when the filters are
modified.

@item 
bpo-22419@footnote{https://bugs.python.org/issue22419}: Limit the length of incoming HTTP request in wsgiref server to
65536 bytes and send a 414 error code for higher lengths. Patch
contributed by Devin Cook.

@item 
Lax cookie parsing in http.cookies could be a security issue when combined
with non-standard cookie handling in some web browsers.  Reported by
Sergey Bobrov.

@item 
bpo-20537@footnote{https://bugs.python.org/issue20537}: logging methods now accept an exception instance as well as a
Boolean value or exception tuple. Thanks to Yury Selivanov for the patch.

@item 
bpo-22384@footnote{https://bugs.python.org/issue22384}: An exception in Tkinter callback no longer crashes the program
when it is run with pythonw.exe.

@item 
bpo-22168@footnote{https://bugs.python.org/issue22168}: Prevent turtle AttributeError with non-default Canvas on OS X.

@item 
bpo-21147@footnote{https://bugs.python.org/issue21147}: sqlite3 now raises an exception if the request contains a null
character instead of truncating it.  Based on patch by Victor Stinner.

@item 
bpo-13968@footnote{https://bugs.python.org/issue13968}: The glob module now supports recursive search in subdirectories
using the @code{**} pattern.

@item 
bpo-21951@footnote{https://bugs.python.org/issue21951}: Fixed a crash in Tkinter on AIX when called Tcl command with
empty string or tuple argument.

@item 
bpo-21951@footnote{https://bugs.python.org/issue21951}: Tkinter now most likely raises MemoryError instead of crash if
the memory allocation fails.

@item 
bpo-22338@footnote{https://bugs.python.org/issue22338}: Fix a crash in the json module on memory allocation failure.

@item 
bpo-12410@footnote{https://bugs.python.org/issue12410}: imaplib.IMAP4 now supports the context management protocol.
Original patch by Tarek Ziadé.

@item 
bpo-21270@footnote{https://bugs.python.org/issue21270}: We now override tuple methods in mock.call objects so that they
can be used as normal call attributes.

@item 
bpo-16662@footnote{https://bugs.python.org/issue16662}: load_tests() is now unconditionally run when it is present in a
package’s __init__.py.  TestLoader.loadTestsFromModule() still accepts
use_load_tests, but it is deprecated and ignored.  A new keyword-only
attribute @code{pattern} is added and documented.  Patch given by Robert
Collins, tweaked by Barry Warsaw.

@item 
bpo-22226@footnote{https://bugs.python.org/issue22226}: First letter no longer is stripped from the “status” key in the
result of Treeview.heading().

@item 
bpo-19524@footnote{https://bugs.python.org/issue19524}: Fixed resource leak in the HTTP connection when an invalid
response is received.  Patch by Martin Panter.

@item 
bpo-20421@footnote{https://bugs.python.org/issue20421}: Add a .version() method to SSL sockets exposing the actual
protocol version in use.

@item 
bpo-19546@footnote{https://bugs.python.org/issue19546}: configparser exceptions no longer expose implementation
details. Chained KeyErrors are removed, which leads to cleaner tracebacks.
Patch by Claudiu Popa.

@item 
bpo-22051@footnote{https://bugs.python.org/issue22051}: turtledemo no longer reloads examples to re-run them.
Initialization of variables and gui setup should be done in main(), which
is called each time a demo is run, but not on import.

@item 
bpo-21933@footnote{https://bugs.python.org/issue21933}: Turtledemo users can change the code font size with a menu
selection or control(command) ‘-’ or ‘+’ or control-mousewheel. Original
patch by Lita Cho.

@item 
bpo-21597@footnote{https://bugs.python.org/issue21597}: The separator between the turtledemo text pane and the drawing
canvas can now be grabbed and dragged with a mouse.  The code text pane
can be widened to easily view or copy the full width of the text.  The
canvas can be widened on small screens.  Original patches by Jan Kanis and
Lita Cho.

@item 
bpo-18132@footnote{https://bugs.python.org/issue18132}: Turtledemo buttons no longer disappear when the window is
shrunk.  Original patches by Jan Kanis and Lita Cho.

@item 
bpo-22043@footnote{https://bugs.python.org/issue22043}: time.monotonic() is now always available.
@code{threading.Lock.acquire()}, @code{threading.RLock.acquire()} and socket
operations now use a monotonic clock, instead of the system clock, when a
timeout is used.

@item 
bpo-21527@footnote{https://bugs.python.org/issue21527}: Add a default number of workers to ThreadPoolExecutor equal to
5 times the number of CPUs.  Patch by Claudiu Popa.

@item 
bpo-22216@footnote{https://bugs.python.org/issue22216}: smtplib now resets its state more completely after a quit.  The
most obvious consequence of the previous behavior was a STARTTLS failure
during a connect/starttls/quit/connect/starttls sequence.

@item 
bpo-22098@footnote{https://bugs.python.org/issue22098}: ctypes’ BigEndianStructure and LittleEndianStructure now define
an empty __slots__ so that subclasses don’t always get an instance dict.
Patch by Claudiu Popa.

@item 
bpo-22185@footnote{https://bugs.python.org/issue22185}: Fix an occasional RuntimeError in threading.Condition.wait()
caused by mutation of the waiters queue without holding the lock.  Patch
by Doug Zongker.

@item 
bpo-22287@footnote{https://bugs.python.org/issue22287}: On UNIX, _PyTime_gettimeofday() now uses
clock_gettime(CLOCK_REALTIME) if available. As a side effect, Python now
depends on the librt library on Solaris and on Linux (only with glibc
older than 2.17).

@item 
bpo-22182@footnote{https://bugs.python.org/issue22182}: Use e.args to unpack exceptions correctly in
distutils.file_util.move_file. Patch by Claudiu Popa.

@item 
The webbrowser module now uses subprocess’s start_new_session=True rather
than a potentially risky preexec_fn=os.setsid call.

@item 
bpo-22042@footnote{https://bugs.python.org/issue22042}: signal.set_wakeup_fd(fd) now raises an exception if the file
descriptor is in blocking mode.

@item 
bpo-16808@footnote{https://bugs.python.org/issue16808}: inspect.stack() now returns a named tuple instead of a tuple.
Patch by Daniel Shahaf.

@item 
bpo-22236@footnote{https://bugs.python.org/issue22236}: Fixed Tkinter images copying operations in NoDefaultRoot mode.

@item 
bpo-2527@footnote{https://bugs.python.org/issue2527}: Add a @emph{globals} argument to timeit functions, in order to
override the globals namespace in which the timed code is executed. Patch
by Ben Roberts.

@item 
bpo-22118@footnote{https://bugs.python.org/issue22118}: Switch urllib.parse to use RFC 3986 semantics for the
resolution of relative URLs, rather than RFCs 1808 and 2396. Patch by
Demian Brecht.

@item 
bpo-21549@footnote{https://bugs.python.org/issue21549}: Added the “members” parameter to TarFile.list().

@item 
bpo-19628@footnote{https://bugs.python.org/issue19628}: Allow compileall recursion depth to be specified with a -r
option.

@item 
bpo-15696@footnote{https://bugs.python.org/issue15696}: Add a __sizeof__ implementation for mmap objects on Windows.

@item 
bpo-22068@footnote{https://bugs.python.org/issue22068}: Avoided reference loops with Variables and Fonts in Tkinter.

@item 
bpo-22165@footnote{https://bugs.python.org/issue22165}: SimpleHTTPRequestHandler now supports undecodable file names.

@item 
bpo-15381@footnote{https://bugs.python.org/issue15381}: Optimized line reading in io.BytesIO.

@item 
bpo-8797@footnote{https://bugs.python.org/issue8797}: Raise HTTPError on failed Basic Authentication immediately.
Initial patch by Sam Bull.

@item 
bpo-20729@footnote{https://bugs.python.org/issue20729}: Restored the use of lazy iterkeys()/itervalues()/iteritems() in
the mailbox module.

@item 
bpo-21448@footnote{https://bugs.python.org/issue21448}: Changed FeedParser feed() to avoid O(N@w{^2}) behavior when
parsing long line. Original patch by Raymond Hettinger.

@item 
bpo-22184@footnote{https://bugs.python.org/issue22184}: The functools LRU Cache decorator factory now gives an earlier
and clearer error message when the user forgets the required parameters.

@item 
bpo-17923@footnote{https://bugs.python.org/issue17923}: glob() patterns ending with a slash no longer match non-dirs on
AIX.  Based on patch by Delhallt.

@item 
bpo-21725@footnote{https://bugs.python.org/issue21725}: Added support for RFC 6531 (SMTPUTF8) in smtpd.

@item 
bpo-22176@footnote{https://bugs.python.org/issue22176}: Update the ctypes module’s libffi to v3.1.  This release adds
support for the Linux AArch64 and POWERPC ELF ABIv2 little endian
architectures.

@item 
bpo-5411@footnote{https://bugs.python.org/issue5411}: Added support for the “xztar” format in the shutil module.

@item 
bpo-21121@footnote{https://bugs.python.org/issue21121}: Don’t force 3rd party C extensions to be built with
-Werror=declaration-after-statement.

@item 
bpo-21975@footnote{https://bugs.python.org/issue21975}: Fixed crash when using uninitialized sqlite3.Row (in particular
when unpickling pickled sqlite3.Row).  sqlite3.Row is now initialized in
the __new__() method.

@item 
bpo-20170@footnote{https://bugs.python.org/issue20170}: Convert posixmodule to use Argument Clinic.

@item 
bpo-21539@footnote{https://bugs.python.org/issue21539}: Add an @emph{exists_ok} argument to @code{Pathlib.mkdir()} to mimic
@code{mkdir -p} and @ref{68c,,os.makedirs()} functionality.  When true, ignore
FileExistsErrors.  Patch by Berker Peksag.

@item 
bpo-22127@footnote{https://bugs.python.org/issue22127}: Bypass IDNA for pure-ASCII host names in the socket module (in
particular for numeric IPs).

@item 
bpo-21047@footnote{https://bugs.python.org/issue21047}: set the default value for the @emph{convert_charrefs} argument of
HTMLParser to True.  Patch by Berker Peksag.

@item 
Add an __all__ to html.entities.

@item 
bpo-15114@footnote{https://bugs.python.org/issue15114}: the strict mode and argument of HTMLParser, HTMLParser.error,
and the HTMLParserError exception have been removed.

@item 
bpo-22085@footnote{https://bugs.python.org/issue22085}: Dropped support of Tk 8.3 in Tkinter.

@item 
bpo-21580@footnote{https://bugs.python.org/issue21580}: Now Tkinter correctly handles bytes arguments passed to Tk. In
particular this allows initializing images from binary data.

@item 
bpo-22003@footnote{https://bugs.python.org/issue22003}: When initialized from a bytes object, io.BytesIO() now defers
making a copy until it is mutated, improving performance and memory use on
some use cases. Patch by David Wilson.

@item 
bpo-22018@footnote{https://bugs.python.org/issue22018}: On Windows, signal.set_wakeup_fd() now also supports sockets. A
side effect is that Python depends to the WinSock library.

@item 
bpo-22054@footnote{https://bugs.python.org/issue22054}: Add os.get_blocking() and os.set_blocking() functions to get
and set the blocking mode of a file descriptor (False if the O_NONBLOCK
flag is set, True otherwise). These functions are not available on
Windows.

@item 
bpo-17172@footnote{https://bugs.python.org/issue17172}: Make turtledemo start as active on OS X even when run with
subprocess. Patch by Lita Cho.

@item 
bpo-21704@footnote{https://bugs.python.org/issue21704}: Fix build error for _multiprocessing when semaphores are not
available. Patch by Arfrever Frehtes Taifersar Arahesis.

@item 
bpo-20173@footnote{https://bugs.python.org/issue20173}: Convert sha1, sha256, sha512 and md5 to ArgumentClinic. Patch
by Vajrasky Kok.

@item 
Fix repr(_socket.socket) on Windows 64-bit: don’t fail with OverflowError
on closed socket. repr(socket.socket) already works fine.

@item 
bpo-22033@footnote{https://bugs.python.org/issue22033}: Reprs of most Python implemented classes now contain actual
class name instead of hardcoded one.

@item 
bpo-21947@footnote{https://bugs.python.org/issue21947}: The dis module can now disassemble generator-iterator objects
based on their gi_code attribute. Patch by Clement Rouault.

@item 
bpo-16133@footnote{https://bugs.python.org/issue16133}: The asynchat.async_chat.handle_read() method now ignores
BlockingIOError exceptions.

@item 
bpo-22044@footnote{https://bugs.python.org/issue22044}: Fixed premature DECREF in call_tzinfo_method. Patch by Tom
Flanagan.

@item 
bpo-19884@footnote{https://bugs.python.org/issue19884}: readline: Disable the meta modifier key if stdout is not a
terminal to not write the ANSI sequence @code{"\033[1034h"} into stdout. This
sequence is used on some terminal (ex: TERM=xterm-256color”) to enable
support of 8 bit characters.

@item 
bpo-4350@footnote{https://bugs.python.org/issue4350}: Removed a number of out-of-dated and non-working for a long time
Tkinter methods.

@item 
bpo-6167@footnote{https://bugs.python.org/issue6167}: Scrollbar.activate() now returns the name of active element if
the argument is not specified.  Scrollbar.set() now always accepts only 2
arguments.

@item 
bpo-15275@footnote{https://bugs.python.org/issue15275}: Clean up and speed up the ntpath module.

@item 
bpo-21888@footnote{https://bugs.python.org/issue21888}: plistlib’s load() and loads() now work if the fmt parameter is
specified.

@item 
bpo-22032@footnote{https://bugs.python.org/issue22032}: __qualname__ instead of __name__ is now always used to format
fully qualified class names of Python implemented classes.

@item 
bpo-22031@footnote{https://bugs.python.org/issue22031}: Reprs now always use hexadecimal format with the “0x” prefix
when contain an id in form ” at 0x…”.

@item 
bpo-22018@footnote{https://bugs.python.org/issue22018}: signal.set_wakeup_fd() now raises an OSError instead of a
ValueError on @code{fstat()} failure.

@item 
bpo-21044@footnote{https://bugs.python.org/issue21044}: tarfile.open() now handles fileobj with an integer ‘name’
attribute.  Based on patch by Antoine Pietri.

@item 
bpo-21966@footnote{https://bugs.python.org/issue21966}: Respect -q command-line option when code module is ran.

@item 
bpo-19076@footnote{https://bugs.python.org/issue19076}: Don’t pass the redundant ‘file’ argument to self.error().

@item 
bpo-16382@footnote{https://bugs.python.org/issue16382}: Improve exception message of warnings.warn() for bad category.
Initial patch by Phil Elson.

@item 
bpo-21932@footnote{https://bugs.python.org/issue21932}: os.read() now uses a @code{Py_ssize_t()} type instead of
int for the size to support reading more than 2 GB at once. On
Windows, the size is truncated to INT_MAX. As any call to os.read(), the
OS may read less bytes than the number of requested bytes.

@item 
bpo-21942@footnote{https://bugs.python.org/issue21942}: Fixed source file viewing in pydoc’s server mode on Windows.

@item 
bpo-11259@footnote{https://bugs.python.org/issue11259}: asynchat.async_chat().set_terminator() now raises a ValueError
if the number of received bytes is negative.

@item 
bpo-12523@footnote{https://bugs.python.org/issue12523}: asynchat.async_chat.push() now raises a TypeError if it doesn’t
get a bytes string

@item 
bpo-21707@footnote{https://bugs.python.org/issue21707}: Add missing kwonlyargcount argument to
ModuleFinder.replace_paths_in_code().

@item 
bpo-20639@footnote{https://bugs.python.org/issue20639}: calling Path.with_suffix(‘’) allows removing the suffix again.
Patch by July Tikhonov.

@item 
bpo-21714@footnote{https://bugs.python.org/issue21714}: Disallow the construction of invalid paths using
Path.with_name().  Original patch by Antony Lee.

@item 
bpo-15014@footnote{https://bugs.python.org/issue15014}: Added ‘auth’ method to smtplib to make implementing auth
mechanisms simpler, and used it internally in the login method.

@item 
bpo-21151@footnote{https://bugs.python.org/issue21151}: Fixed a segfault in the winreg module when @code{None} is passed
as a @code{REG_BINARY} value to SetValueEx.  Patch by John Ehresman.

@item 
bpo-21090@footnote{https://bugs.python.org/issue21090}: io.FileIO.readall() does not ignore I/O errors anymore. Before,
it ignored I/O errors if at least the first C call read() succeed.

@item 
bpo-5800@footnote{https://bugs.python.org/issue5800}: headers parameter of wsgiref.headers.Headers is now optional.
Initial patch by Pablo Torres Navarrete and SilentGhost.

@item 
bpo-21781@footnote{https://bugs.python.org/issue21781}: ssl.RAND_add() now supports strings longer than 2 GB.

@item 
bpo-21679@footnote{https://bugs.python.org/issue21679}: Prevent extraneous fstat() calls during open().  Patch by
Bohuslav Kabrda.

@item 
bpo-21863@footnote{https://bugs.python.org/issue21863}: cProfile now displays the module name of C extension functions,
in addition to their own name.

@item 
bpo-11453@footnote{https://bugs.python.org/issue11453}: asyncore: emit a ResourceWarning when an unclosed file_wrapper
object is destroyed. The destructor now closes the file if needed. The
close() method can now be called twice: the second call does nothing.

@item 
bpo-21858@footnote{https://bugs.python.org/issue21858}: Better handling of Python exceptions in the sqlite3 module.

@item 
bpo-21476@footnote{https://bugs.python.org/issue21476}: Make sure the email.parser.BytesParser TextIOWrapper is
discarded after parsing, so the input file isn’t unexpectedly closed.

@item 
bpo-20295@footnote{https://bugs.python.org/issue20295}: imghdr now recognizes OpenEXR format images.

@item 
bpo-21729@footnote{https://bugs.python.org/issue21729}: Used the “with” statement in the dbm.dumb module to ensure
files closing. Patch by Claudiu Popa.

@item 
bpo-21491@footnote{https://bugs.python.org/issue21491}: socketserver: Fix a race condition in child processes reaping.

@item 
bpo-21719@footnote{https://bugs.python.org/issue21719}: Added the @code{st_file_attributes} field to os.stat_result on
Windows.

@item 
bpo-21832@footnote{https://bugs.python.org/issue21832}: Require named tuple inputs to be exact strings.

@item 
bpo-21722@footnote{https://bugs.python.org/issue21722}: The distutils “upload” command now exits with a non-zero return
code when uploading fails.  Patch by Martin Dengler.

@item 
bpo-21723@footnote{https://bugs.python.org/issue21723}: asyncio.Queue: support any type of number (ex: float) for the
maximum size. Patch written by Vajrasky Kok.

@item 
bpo-21711@footnote{https://bugs.python.org/issue21711}: support for “site-python” directories has now been removed from
the site module (it was deprecated in 3.4).

@item 
bpo-17552@footnote{https://bugs.python.org/issue17552}: new socket.sendfile() method allowing a file to be sent over a
socket by using high-performance os.sendfile() on UNIX. Patch by Giampaolo
Rodola’.

@item 
bpo-18039@footnote{https://bugs.python.org/issue18039}: dbm.dump.open() now always creates a new database when the flag
has the value ‘n’.  Patch by Claudiu Popa.

@item 
bpo-21326@footnote{https://bugs.python.org/issue21326}: Add a new is_closed() method to asyncio.BaseEventLoop.
run_forever() and run_until_complete() methods of asyncio.BaseEventLoop
now raise an exception if the event loop was closed.

@item 
bpo-21766@footnote{https://bugs.python.org/issue21766}: Prevent a security hole in CGIHTTPServer by URL unquoting paths
before checking for a CGI script at that path.

@item 
bpo-21310@footnote{https://bugs.python.org/issue21310}: Fixed possible resource leak in failed open().

@item 
bpo-21256@footnote{https://bugs.python.org/issue21256}: Printout of keyword args should be in deterministic order in a
mock function call. This will help to write better doctests.

@item 
bpo-21677@footnote{https://bugs.python.org/issue21677}: Fixed chaining nonnormalized exceptions in io close() methods.

@item 
bpo-11709@footnote{https://bugs.python.org/issue11709}: Fix the pydoc.help function to not fail when sys.stdin is not a
valid file.

@item 
bpo-21515@footnote{https://bugs.python.org/issue21515}: tempfile.TemporaryFile now uses os.O_TMPFILE flag is available.

@item 
bpo-13223@footnote{https://bugs.python.org/issue13223}: Fix pydoc.writedoc so that the HTML documentation for methods
that use ‘self’ in the example code is generated correctly.

@item 
bpo-21463@footnote{https://bugs.python.org/issue21463}: In urllib.request, fix pruning of the FTP cache.

@item 
bpo-21618@footnote{https://bugs.python.org/issue21618}: The subprocess module could fail to close open fds that were
inherited by the calling process and already higher than POSIX resource
limits would otherwise allow.  On systems with a functioning /proc/self/fd
or /dev/fd interface the max is now ignored and all fds are closed.

@item 
bpo-20383@footnote{https://bugs.python.org/issue20383}: Introduce importlib.util.module_from_spec() as the preferred
way to create a new module.

@item 
bpo-21552@footnote{https://bugs.python.org/issue21552}: Fixed possible integer overflow of too long string lengths in
the tkinter module on 64-bit platforms.

@item 
bpo-14315@footnote{https://bugs.python.org/issue14315}: The zipfile module now ignores extra fields in the central
directory that are too short to be parsed instead of letting a
struct.unpack error bubble up as this “bad data” appears in many real
world zip files in the wild and is ignored by other zip tools.

@item 
bpo-13742@footnote{https://bugs.python.org/issue13742}: Added “key” and “reverse” parameters to heapq.merge(). (First
draft of patch contributed by Simon Sapin.)

@item 
bpo-21402@footnote{https://bugs.python.org/issue21402}: tkinter.ttk now works when default root window is not set.

@item 
bpo-3015@footnote{https://bugs.python.org/issue3015}: _tkinter.create() now creates tkapp object with wantobject=1 by
default.

@item 
bpo-10203@footnote{https://bugs.python.org/issue10203}: sqlite3.Row now truly supports sequence protocol.  In
particular it supports reverse() and negative indices.  Original patch by
Claudiu Popa.

@item 
bpo-18807@footnote{https://bugs.python.org/issue18807}: If copying (no symlinks) specified for a venv, then the python
interpreter aliases (python, python3) are now created by copying rather
than symlinking.

@item 
bpo-20197@footnote{https://bugs.python.org/issue20197}: Added support for the WebP image type in the imghdr module.
Patch by Fabrice Aneche and Claudiu Popa.

@item 
bpo-21513@footnote{https://bugs.python.org/issue21513}: Speedup some properties of IP addresses (IPv4Address,
IPv6Address) such as .is_private or .is_multicast.

@item 
bpo-21137@footnote{https://bugs.python.org/issue21137}: Improve the repr for threading.Lock() and its variants by
showing the “locked” or “unlocked” status.  Patch by Berker Peksag.

@item 
bpo-21538@footnote{https://bugs.python.org/issue21538}: The plistlib module now supports loading of binary plist files
when reference or offset size is not a power of two.

@item 
bpo-21455@footnote{https://bugs.python.org/issue21455}: Add a default backlog to socket.listen().

@item 
bpo-21525@footnote{https://bugs.python.org/issue21525}: Most Tkinter methods which accepted tuples now accept lists
too.

@item 
bpo-22166@footnote{https://bugs.python.org/issue22166}: With the assistance of a new internal _codecs._forget_codec
helping function, test_codecs now clears the encoding caches to avoid the
appearance of a reference leak

@item 
bpo-22236@footnote{https://bugs.python.org/issue22236}: Tkinter tests now don’t reuse default root window.  New root
window is created for every test class.

@item 
bpo-10744@footnote{https://bugs.python.org/issue10744}: Fix 
@geindex Python Enhancement Proposals; PEP 3118
PEP 3118@footnote{https://www.python.org/dev/peps/pep-3118} format strings on ctypes objects with a
nontrivial shape.

@item 
bpo-20826@footnote{https://bugs.python.org/issue20826}: Optimize ipaddress.collapse_addresses().

@item 
bpo-21487@footnote{https://bugs.python.org/issue21487}: Optimize ipaddress.summarize_address_range() and
ipaddress.@{IPv4Network,IPv6Network@}.subnets().

@item 
bpo-21486@footnote{https://bugs.python.org/issue21486}: Optimize parsing of netmasks in ipaddress.IPv4Network and
ipaddress.IPv6Network.

@item 
bpo-13916@footnote{https://bugs.python.org/issue13916}: Disallowed the surrogatepass error handler for non UTF-*
encodings.

@item 
bpo-20998@footnote{https://bugs.python.org/issue20998}: Fixed re.fullmatch() of repeated single character pattern with
ignore case. Original patch by Matthew Barnett.

@item 
bpo-21075@footnote{https://bugs.python.org/issue21075}: fileinput.FileInput now reads bytes from standard stream if
binary mode is specified.  Patch by Sam Kimbrel.

@item 
bpo-19775@footnote{https://bugs.python.org/issue19775}: Add a samefile() method to pathlib Path objects.  Initial patch
by Vajrasky Kok.

@item 
bpo-21226@footnote{https://bugs.python.org/issue21226}: Set up modules properly in PyImport_ExecCodeModuleObject (and
friends).

@item 
bpo-21398@footnote{https://bugs.python.org/issue21398}: Fix a unicode error in the pydoc pager when the documentation
contains characters not encodable to the stdout encoding.

@item 
bpo-16531@footnote{https://bugs.python.org/issue16531}: ipaddress.IPv4Network and ipaddress.IPv6Network now accept an
(address, netmask) tuple argument, so as to easily construct network
objects from existing addresses.

@item 
bpo-21156@footnote{https://bugs.python.org/issue21156}: importlib.abc.InspectLoader.source_to_code() is now a
staticmethod.

@item 
bpo-21424@footnote{https://bugs.python.org/issue21424}: Simplified and optimized heaqp.nlargest() and nmsmallest() to
make fewer tuple comparisons.

@item 
bpo-21396@footnote{https://bugs.python.org/issue21396}: Fix TextIOWrapper(…, write_through=True) to not force a
flush() on the underlying binary stream.  Patch by akira.

@item 
bpo-18314@footnote{https://bugs.python.org/issue18314}: Unlink now removes junctions on Windows. Patch by Kim Gräsman

@item 
bpo-21088@footnote{https://bugs.python.org/issue21088}: Bugfix for curses.window.addch() regression in 3.4.0. In
porting to Argument Clinic, the first two arguments were reversed.

@item 
bpo-21407@footnote{https://bugs.python.org/issue21407}: _decimal: The module now supports function signatures.

@item 
bpo-10650@footnote{https://bugs.python.org/issue10650}: Remove the non-standard ‘watchexp’ parameter from the
Decimal.quantize() method in the Python version.  It had never been
present in the C version.

@item 
bpo-21469@footnote{https://bugs.python.org/issue21469}: Reduced the risk of false positives in robotparser by checking
to make sure that robots.txt has been read or does not exist prior to
returning True in can_fetch().

@item 
bpo-19414@footnote{https://bugs.python.org/issue19414}: Have the OrderedDict mark deleted links as unusable. This gives
an early failure if the link is deleted during iteration.

@item 
bpo-21421@footnote{https://bugs.python.org/issue21421}: Add __slots__ to the MappingViews ABC. Patch by Josh Rosenberg.

@item 
bpo-21101@footnote{https://bugs.python.org/issue21101}: Eliminate double hashing in the C speed-up code for
collections.Counter().

@item 
bpo-21321@footnote{https://bugs.python.org/issue21321}: itertools.islice() now releases the reference to the source
iterator when the slice is exhausted.  Patch by Anton Afanasyev.

@item 
bpo-21057@footnote{https://bugs.python.org/issue21057}: TextIOWrapper now allows the underlying binary stream’s read()
or read1() method to return an arbitrary bytes-like object (such as a
memoryview). Patch by Nikolaus Rath.

@item 
bpo-20951@footnote{https://bugs.python.org/issue20951}: SSLSocket.send() now raises either SSLWantReadError or
SSLWantWriteError on a non-blocking socket if the operation would block.
Previously, it would return 0.  Patch by Nikolaus Rath.

@item 
bpo-13248@footnote{https://bugs.python.org/issue13248}: removed previously deprecated asyncore.dispatcher __getattr__
cheap inheritance hack.

@item 
bpo-9815@footnote{https://bugs.python.org/issue9815}: assertRaises now tries to clear references to local variables in
the exception’s traceback.

@item 
bpo-19940@footnote{https://bugs.python.org/issue19940}: ssl.cert_time_to_seconds() now interprets the given time string
in the UTC timezone (as specified in RFC 5280), not the local timezone.

@item 
bpo-13204@footnote{https://bugs.python.org/issue13204}: Calling sys.flags.__new__ would crash the interpreter, now it
raises a TypeError.

@item 
bpo-19385@footnote{https://bugs.python.org/issue19385}: Make operations on a closed dbm.dumb database always raise the
same exception.

@item 
bpo-21207@footnote{https://bugs.python.org/issue21207}: Detect when the os.urandom cached fd has been closed or
replaced, and open it anew.

@item 
bpo-21291@footnote{https://bugs.python.org/issue21291}: subprocess’s Popen.wait() is now thread safe so that multiple
threads may be calling wait() or poll() on a Popen instance at the same
time without losing the Popen.returncode value.

@item 
bpo-21127@footnote{https://bugs.python.org/issue21127}: Path objects can now be instantiated from str subclass
instances (such as @code{numpy.str_}).

@item 
bpo-15002@footnote{https://bugs.python.org/issue15002}: urllib.response object to use _TemporaryFileWrapper (and
_TemporaryFileCloser) facility. Provides a better way to handle file
descriptor close. Patch contributed by Christian Theune.

@item 
bpo-12220@footnote{https://bugs.python.org/issue12220}: mindom now raises a custom ValueError indicating it doesn’t
support spaces in URIs instead of letting a ‘split’ ValueError bubble up.

@item 
bpo-21068@footnote{https://bugs.python.org/issue21068}: The ssl.PROTOCOL* constants are now enum members.

@item 
bpo-21276@footnote{https://bugs.python.org/issue21276}: posixmodule: Don’t define USE_XATTRS on KFreeBSD and the Hurd.

@item 
bpo-21262@footnote{https://bugs.python.org/issue21262}: New method assert_not_called for Mock. It raises AssertionError
if the mock has been called.

@item 
bpo-21238@footnote{https://bugs.python.org/issue21238}: New keyword argument @code{unsafe} to Mock. It raises
@ref{163,,AttributeError} incase of an attribute startswith assert or assret.

@item 
bpo-20896@footnote{https://bugs.python.org/issue20896}: ssl.get_server_certificate() now uses PROTOCOL_SSLv23, not
PROTOCOL_SSLv3, for maximum compatibility.

@item 
bpo-21239@footnote{https://bugs.python.org/issue21239}: patch.stopall() didn’t work deterministically when the same
name was patched more than once.

@item 
bpo-21203@footnote{https://bugs.python.org/issue21203}: Updated fileConfig and dictConfig to remove inconsistencies.
Thanks to Jure Koren for the patch.

@item 
bpo-21222@footnote{https://bugs.python.org/issue21222}: Passing name keyword argument to mock.create_autospec now
works.

@item 
bpo-21197@footnote{https://bugs.python.org/issue21197}: Add lib64 -> lib symlink in venvs on 64-bit non-OS X POSIX.

@item 
bpo-17498@footnote{https://bugs.python.org/issue17498}: Some SMTP servers disconnect after certain errors, violating
strict RFC conformance.  Instead of losing the error code when we issue
the subsequent RSET, smtplib now returns the error code and defers raising
the SMTPServerDisconnected error until the next command is issued.

@item 
bpo-17826@footnote{https://bugs.python.org/issue17826}: setting an iterable side_effect on a mock function created by
create_autospec now works. Patch by Kushal Das.

@item 
bpo-7776@footnote{https://bugs.python.org/issue7776}: Fix @code{Host:} header and reconnection when using
http.client.HTTPConnection.set_tunnel(). Patch by Nikolaus Rath.

@item 
bpo-20968@footnote{https://bugs.python.org/issue20968}: unittest.mock.MagicMock now supports division. Patch by
Johannes Baiter.

@item 
bpo-21529@footnote{https://bugs.python.org/issue21529}: Fix arbitrary memory access in JSONDecoder.raw_decode with a
negative second parameter. Bug reported by Guido Vranken. (See also:
CVE-2014-4616)

@item 
bpo-21169@footnote{https://bugs.python.org/issue21169}: getpass now handles non-ascii characters that the input stream
encoding cannot encode by re-encoding using the replace error handler.

@item 
bpo-21171@footnote{https://bugs.python.org/issue21171}: Fixed undocumented filter API of the rot13 codec. Patch by
Berker Peksag.

@item 
bpo-20539@footnote{https://bugs.python.org/issue20539}: Improved math.factorial error message for large positive inputs
and changed exception type (OverflowError -> ValueError) for large
negative inputs.

@item 
bpo-21172@footnote{https://bugs.python.org/issue21172}: isinstance check relaxed from dict to collections.Mapping.

@item 
bpo-21155@footnote{https://bugs.python.org/issue21155}: asyncio.EventLoop.create_unix_server() now raises a ValueError
if path and sock are specified at the same time.

@item 
bpo-21136@footnote{https://bugs.python.org/issue21136}: Avoid unnecessary normalization of Fractions resulting from
power and other operations.  Patch by Raymond Hettinger.

@item 
bpo-17621@footnote{https://bugs.python.org/issue17621}: Introduce importlib.util.LazyLoader.

@item 
bpo-21076@footnote{https://bugs.python.org/issue21076}: signal module constants were turned into enums. Patch by
Giampaolo Rodola’.

@item 
bpo-20636@footnote{https://bugs.python.org/issue20636}: Improved the repr of Tkinter widgets.

@item 
bpo-19505@footnote{https://bugs.python.org/issue19505}: The items, keys, and values views of OrderedDict now support
reverse iteration using reversed().

@item 
bpo-21149@footnote{https://bugs.python.org/issue21149}: Improved thread-safety in logging cleanup during interpreter
shutdown. Thanks to Devin Jeanpierre for the patch.

@item 
bpo-21058@footnote{https://bugs.python.org/issue21058}: Fix a leak of file descriptor in
@ref{f44,,tempfile.NamedTemporaryFile()}, close the file descriptor if
@ref{1c9,,io.open()} fails

@item 
bpo-21200@footnote{https://bugs.python.org/issue21200}: Return None from pkgutil.get_loader() when __spec__ is missing.

@item 
bpo-21013@footnote{https://bugs.python.org/issue21013}: Enhance ssl.create_default_context() when used for server side
sockets to provide better security by default.

@item 
bpo-20145@footnote{https://bugs.python.org/issue20145}: @code{assertRaisesRegex} and @code{assertWarnsRegex} now raise a
TypeError if the second argument is not a string or compiled regex.

@item 
bpo-20633@footnote{https://bugs.python.org/issue20633}: Replace relative import by absolute import.

@item 
bpo-20980@footnote{https://bugs.python.org/issue20980}: Stop wrapping exception when using ThreadPool.

@item 
bpo-21082@footnote{https://bugs.python.org/issue21082}: In os.makedirs, do not set the process-wide umask. Note this
changes behavior of makedirs when exist_ok=True.

@item 
bpo-20990@footnote{https://bugs.python.org/issue20990}: Fix issues found by pyflakes for multiprocessing.

@item 
bpo-21015@footnote{https://bugs.python.org/issue21015}: SSL contexts will now automatically select an elliptic curve
for ECDH key exchange on OpenSSL 1.0.2 and later, and otherwise default to
“prime256v1”.

@item 
bpo-21000@footnote{https://bugs.python.org/issue21000}: Improve the command-line interface of json.tool.

@item 
bpo-20995@footnote{https://bugs.python.org/issue20995}: Enhance default ciphers used by the ssl module to enable better
security and prioritize perfect forward secrecy.

@item 
bpo-20884@footnote{https://bugs.python.org/issue20884}: Don’t assume that __file__ is defined on importlib.__init__.

@item 
bpo-21499@footnote{https://bugs.python.org/issue21499}: Ignore __builtins__ in several test_importlib.test_api tests.

@item 
bpo-20627@footnote{https://bugs.python.org/issue20627}: xmlrpc.client.ServerProxy is now a context manager.

@item 
bpo-19165@footnote{https://bugs.python.org/issue19165}: The formatter module now raises DeprecationWarning instead of
PendingDeprecationWarning.

@item 
bpo-13936@footnote{https://bugs.python.org/issue13936}: Remove the ability of datetime.time instances to be considered
false in boolean contexts.

@item 
bpo-18931@footnote{https://bugs.python.org/issue18931}: selectors module now supports /dev/poll on Solaris. Patch by
Giampaolo Rodola’.

@item 
bpo-19977@footnote{https://bugs.python.org/issue19977}: When the @code{LC_TYPE} locale is the POSIX locale (@code{C} locale),
@ref{5ee,,sys.stdin} and @ref{5ef,,sys.stdout} are now using the
@code{surrogateescape} error handler, instead of the @code{strict} error
handler.

@item 
bpo-20574@footnote{https://bugs.python.org/issue20574}: Implement incremental decoder for cp65001 code (Windows code
page 65001, Microsoft UTF-8).

@item 
bpo-20879@footnote{https://bugs.python.org/issue20879}: Delay the initialization of encoding and decoding tables for
base32, ascii85 and base85 codecs in the base64 module, and delay the
initialization of the unquote_to_bytes() table of the urllib.parse module,
to not waste memory if these modules are not used.

@item 
bpo-19157@footnote{https://bugs.python.org/issue19157}: Include the broadcast address in the usuable hosts for IPv6 in
ipaddress.

@item 
bpo-11599@footnote{https://bugs.python.org/issue11599}: When an external command (e.g. compiler) fails, distutils now
prints out the whole command line (instead of just the command name) if
the environment variable DISTUTILS_DEBUG is set.

@item 
bpo-4931@footnote{https://bugs.python.org/issue4931}: distutils should not produce unhelpful “error: None” messages
anymore. distutils.util.grok_environment_error is kept but doc-deprecated.

@item 
bpo-20875@footnote{https://bugs.python.org/issue20875}: Prevent possible gzip “‘read’ is not defined” NameError. Patch
by Claudiu Popa.

@item 
bpo-11558@footnote{https://bugs.python.org/issue11558}: @code{email.message.Message.attach} now returns a more useful
error message if @code{attach} is called on a message for which
@code{is_multipart} is False.

@item 
bpo-20283@footnote{https://bugs.python.org/issue20283}: RE pattern methods now accept the string keyword parameters as
documented. The pattern and source keyword parameters are left as
deprecated aliases.

@item 
bpo-20778@footnote{https://bugs.python.org/issue20778}: Fix modulefinder to work with bytecode-only modules.

@item 
bpo-20791@footnote{https://bugs.python.org/issue20791}: copy.copy() now doesn’t make a copy when the input is a bytes
object. Initial patch by Peter Otten.

@item 
bpo-19748@footnote{https://bugs.python.org/issue19748}: On AIX, time.mktime() now raises an OverflowError for year
outsize range [1902; 2037].

@item 
bpo-19573@footnote{https://bugs.python.org/issue19573}: inspect.signature: Use enum for parameter kind constants.

@item 
bpo-20726@footnote{https://bugs.python.org/issue20726}: inspect.signature: Make Signature and Parameter picklable.

@item 
bpo-17373@footnote{https://bugs.python.org/issue17373}: Add inspect.Signature.from_callable method.

@item 
bpo-20378@footnote{https://bugs.python.org/issue20378}: Improve repr of inspect.Signature and inspect.Parameter.

@item 
bpo-20816@footnote{https://bugs.python.org/issue20816}: Fix inspect.getcallargs() to raise correct TypeError for
missing keyword-only arguments. Patch by Jeremiah Lowin.

@item 
bpo-20817@footnote{https://bugs.python.org/issue20817}: Fix inspect.getcallargs() to fail correctly if more than 3
arguments are missing. Patch by Jeremiah Lowin.

@item 
bpo-6676@footnote{https://bugs.python.org/issue6676}: Ensure a meaningful exception is raised when attempting to parse
more than one XML document per pyexpat xmlparser instance. (Original
patches by Hirokazu Yamamoto and Amaury Forgeot d’Arc, with suggested
wording by David Gutteridge)

@item 
bpo-21117@footnote{https://bugs.python.org/issue21117}: Fix inspect.signature to better support functools.partial. Due
to the specifics of functools.partial implementation,
positional-or-keyword arguments passed as keyword arguments become
keyword-only.

@item 
bpo-20334@footnote{https://bugs.python.org/issue20334}: inspect.Signature and inspect.Parameter are now hashable.
Thanks to Antony Lee for bug reports and suggestions.

@item 
bpo-15916@footnote{https://bugs.python.org/issue15916}: doctest.DocTestSuite returns an empty unittest.TestSuite
instead of raising ValueError if it finds no tests

@item 
bpo-21209@footnote{https://bugs.python.org/issue21209}: Fix asyncio.tasks.CoroWrapper to workaround a bug in yield-from
implementation in CPythons prior to 3.4.1.

@item 
asyncio: Add gi_@{frame,running,code@} properties to CoroWrapper (upstream
bpo-163@footnote{https://bugs.python.org/issue163}).

@item 
bpo-21311@footnote{https://bugs.python.org/issue21311}: Avoid exception in _osx_support with non-standard compiler
configurations. Patch by John Szakmeister.

@item 
bpo-11571@footnote{https://bugs.python.org/issue11571}: Ensure that the turtle window becomes the topmost window when
launched on OS X.

@item 
bpo-21801@footnote{https://bugs.python.org/issue21801}: Validate that __signature__ is None or an instance of
Signature.

@item 
bpo-21923@footnote{https://bugs.python.org/issue21923}: Prevent AttributeError in
distutils.sysconfig.customize_compiler due to possible uninitialized
_config_vars.

@item 
bpo-21323@footnote{https://bugs.python.org/issue21323}: Fix http.server to again handle scripts in CGI subdirectories,
broken by the fix for security bpo-19435@footnote{https://bugs.python.org/issue19435}.  Patch by Zach Byrne.

@item 
bpo-22733@footnote{https://bugs.python.org/issue22733}: Fix ffi_prep_args not zero-extending argument values correctly
on 64-bit Windows.

@item 
bpo-23302@footnote{https://bugs.python.org/issue23302}: Default to TCP_NODELAY=1 upon establishing an HTTPConnection.
Removed use of hard-coded MSS as it’s an optimization that’s no longer
needed with Nagle disabled.
@end itemize

@node IDLE<50>,Build<64>,Library<85>,Python 3 5 0 alpha 1
@anchor{whatsnew/changelog id573}@anchor{14b2}
@subsubsection IDLE



@itemize -

@item 
bpo-20577@footnote{https://bugs.python.org/issue20577}: Configuration of the max line length for the FormatParagraph
extension has been moved from the General tab of the Idle preferences
dialog to the FormatParagraph tab of the Config Extensions dialog. Patch
by Tal Einat.

@item 
bpo-16893@footnote{https://bugs.python.org/issue16893}: Update Idle doc chapter to match current Idle and add new
information.

@item 
bpo-3068@footnote{https://bugs.python.org/issue3068}: Add Idle extension configuration dialog to Options menu. Changes
are written to HOME/.idlerc/config-extensions.cfg. Original patch by Tal
Einat.

@item 
bpo-16233@footnote{https://bugs.python.org/issue16233}: A module browser (File : Class Browser, Alt+C) requires an
editor window with a filename.  When Class Browser is requested otherwise,
from a shell, output window, or ‘Untitled’ editor, Idle no longer displays
an error box. It now pops up an Open Module box (Alt+M). If a valid name
is entered and a module is opened, a corresponding browser is also opened.

@item 
bpo-4832@footnote{https://bugs.python.org/issue4832}: Save As to type Python files automatically adds .py to the name
you enter (even if your system does not display it).  Some systems
automatically add .txt when type is Text files.

@item 
bpo-21986@footnote{https://bugs.python.org/issue21986}: Code objects are not normally pickled by the pickle module. To
match this, they are no longer pickled when running under Idle.

@item 
bpo-17390@footnote{https://bugs.python.org/issue17390}: Adjust Editor window title; remove ‘Python’, move version to
end.

@item 
bpo-14105@footnote{https://bugs.python.org/issue14105}: Idle debugger breakpoints no longer disappear when inserting or
deleting lines.

@item 
bpo-17172@footnote{https://bugs.python.org/issue17172}: Turtledemo can now be run from Idle. Currently, the entry is on
the Help menu, but it may move to Run. Patch by Ramchandra Apt and Lita
Cho.

@item 
bpo-21765@footnote{https://bugs.python.org/issue21765}: Add support for non-ascii identifiers to HyperParser.

@item 
bpo-21940@footnote{https://bugs.python.org/issue21940}: Add unittest for WidgetRedirector. Initial patch by Saimadhav
Heblikar.

@item 
bpo-18592@footnote{https://bugs.python.org/issue18592}: Add unittest for SearchDialogBase. Patch by Phil Webster.

@item 
bpo-21694@footnote{https://bugs.python.org/issue21694}: Add unittest for ParenMatch. Patch by Saimadhav Heblikar.

@item 
bpo-21686@footnote{https://bugs.python.org/issue21686}: add unittest for HyperParser. Original patch by Saimadhav
Heblikar.

@item 
bpo-12387@footnote{https://bugs.python.org/issue12387}: Add missing upper(lower)case versions of default Windows key
bindings for Idle so Caps Lock does not disable them. Patch by Roger
Serwy.

@item 
bpo-21695@footnote{https://bugs.python.org/issue21695}: Closing a Find-in-files output window while the search is still
in progress no longer closes Idle.

@item 
bpo-18910@footnote{https://bugs.python.org/issue18910}: Add unittest for textView. Patch by Phil Webster.

@item 
bpo-18292@footnote{https://bugs.python.org/issue18292}: Add unittest for AutoExpand. Patch by Saihadhav Heblikar.

@item 
bpo-18409@footnote{https://bugs.python.org/issue18409}: Add unittest for AutoComplete. Patch by Phil Webster.

@item 
bpo-21477@footnote{https://bugs.python.org/issue21477}: htest.py - Improve framework, complete set of tests. Patches by
Saimadhav Heblikar

@item 
bpo-18104@footnote{https://bugs.python.org/issue18104}: Add idlelib/idle_test/htest.py with a few sample tests to begin
consolidating and improving human-validated tests of Idle. Change other
files as needed to work with htest.  Running the module as __main__ runs
all tests.

@item 
bpo-21139@footnote{https://bugs.python.org/issue21139}: Change default paragraph width to 72, the 
@geindex Python Enhancement Proposals; PEP 8
PEP 8@footnote{https://www.python.org/dev/peps/pep-0008}
recommendation.

@item 
bpo-21284@footnote{https://bugs.python.org/issue21284}: Paragraph reformat test passes after user changes reformat
width.

@item 
bpo-17654@footnote{https://bugs.python.org/issue17654}: Ensure IDLE menus are customized properly on OS X for
non-framework builds and for all variants of Tk.

@item 
bpo-23180@footnote{https://bugs.python.org/issue23180}: Rename IDLE “Windows” menu item to “Window”. Patch by Al
Sweigart.
@end itemize

@node Build<64>,C API<48>,IDLE<50>,Python 3 5 0 alpha 1
@anchor{whatsnew/changelog id574}@anchor{14b3}
@subsubsection Build



@itemize -

@item 
bpo-15506@footnote{https://bugs.python.org/issue15506}: Use standard PKG_PROG_PKG_CONFIG autoconf macro in the
configure script.

@item 
bpo-22935@footnote{https://bugs.python.org/issue22935}: Allow the ssl module to be compiled if openssl doesn’t support
SSL 3.

@item 
bpo-22592@footnote{https://bugs.python.org/issue22592}: Drop support of the Borland C compiler to build Python. The
distutils module still supports it to build extensions.

@item 
bpo-22591@footnote{https://bugs.python.org/issue22591}: Drop support of MS-DOS, especially of the DJGPP compiler
(MS-DOS port of GCC).

@item 
bpo-16537@footnote{https://bugs.python.org/issue16537}: Check whether self.extensions is empty in setup.py. Patch by
Jonathan Hosmer.

@item 
bpo-22359@footnote{https://bugs.python.org/issue22359}: Remove incorrect uses of recursive make.  Patch by Jonas
Wagner.

@item 
bpo-21958@footnote{https://bugs.python.org/issue21958}: Define HAVE_ROUND when building with Visual Studio 2013 and
above.  Patch by Zachary Turner.

@item 
bpo-18093@footnote{https://bugs.python.org/issue18093}: the programs that embed the CPython runtime are now in a
separate “Programs” directory, rather than being kept in the Modules
directory.

@item 
bpo-15759@footnote{https://bugs.python.org/issue15759}: “make suspicious”, “make linkcheck” and “make doctest” in Doc/
now display special message when and only when there are failures.

@item 
bpo-21141@footnote{https://bugs.python.org/issue21141}: The Windows build process no longer attempts to find Perl,
instead relying on OpenSSL source being configured and ready to build.
The @code{PCbuild\build_ssl.py} script has been re-written and re-named to
@code{PCbuild\prepare_ssl.py}, and takes care of configuring OpenSSL source
for both 32 and 64 bit platforms.  OpenSSL sources obtained from
svn.python.org will always be pre-configured and ready to build.

@item 
bpo-21037@footnote{https://bugs.python.org/issue21037}: Add a build option to enable AddressSanitizer support.

@item 
bpo-19962@footnote{https://bugs.python.org/issue19962}: The Windows build process now creates “python.bat” in the root
of the source tree, which passes all arguments through to the most
recently built interpreter.

@item 
bpo-21285@footnote{https://bugs.python.org/issue21285}: Refactor and fix curses configure check to always search in a
ncursesw directory.

@item 
bpo-15234@footnote{https://bugs.python.org/issue15234}: For BerkeleyDB and Sqlite, only add the found library and
include directories if they aren’t already being searched. This avoids an
explicit runtime library dependency.

@item 
bpo-17861@footnote{https://bugs.python.org/issue17861}: Tools/scripts/generate_opcode_h.py automatically regenerates
Include/opcode.h from Lib/opcode.py if the latter gets any change.

@item 
bpo-20644@footnote{https://bugs.python.org/issue20644}: OS X installer build support for documentation build changes in
3.4.1: assume externally supplied sphinx-build is available in /usr/bin.

@item 
bpo-20022@footnote{https://bugs.python.org/issue20022}: Eliminate use of deprecated bundlebuilder in OS X builds.

@item 
bpo-15968@footnote{https://bugs.python.org/issue15968}: Incorporated Tcl, Tk, and Tix builds into the Windows build
solution.

@item 
bpo-17095@footnote{https://bugs.python.org/issue17095}: Fix Modules/Setup @emph{shared} support.

@item 
bpo-21811@footnote{https://bugs.python.org/issue21811}: Anticipated fixes to support OS X versions > 10.9.

@item 
bpo-21166@footnote{https://bugs.python.org/issue21166}: Prevent possible segfaults and other random failures of python
–generate-posix-vars in pybuilddir.txt build target.

@item 
bpo-18096@footnote{https://bugs.python.org/issue18096}: Fix library order returned by python-config.

@item 
bpo-17219@footnote{https://bugs.python.org/issue17219}: Add library build dir for Python extension cross-builds.

@item 
bpo-22919@footnote{https://bugs.python.org/issue22919}: Windows build updated to support VC 14.0 (Visual Studio 2015),
which will be used for the official release.

@item 
bpo-21236@footnote{https://bugs.python.org/issue21236}: Build _msi.pyd with cabinet.lib instead of fci.lib

@item 
bpo-17128@footnote{https://bugs.python.org/issue17128}: Use private version of OpenSSL for OS X 10.5+ installer.
@end itemize

@node C API<48>,Documentation<56>,Build<64>,Python 3 5 0 alpha 1
@anchor{whatsnew/changelog id575}@anchor{14b4}
@subsubsection C API



@itemize -

@item 
bpo-14203@footnote{https://bugs.python.org/issue14203}: Remove obsolete support for view==NULL in PyBuffer_FillInfo(),
bytearray_getbuffer(), bytesiobuf_getbuffer() and array_buffer_getbuf().
All functions now raise BufferError in that case.

@item 
bpo-22445@footnote{https://bugs.python.org/issue22445}: PyBuffer_IsContiguous() now implements precise contiguity
tests, compatible with NumPy’s NPY_RELAXED_STRIDES_CHECKING compilation
flag.  Previously the function reported false negatives for corner cases.

@item 
bpo-22079@footnote{https://bugs.python.org/issue22079}: PyType_Ready() now checks that statically allocated type has no
dynamically allocated bases.

@item 
bpo-22453@footnote{https://bugs.python.org/issue22453}: Removed non-documented macro PyObject_REPR().

@item 
bpo-18395@footnote{https://bugs.python.org/issue18395}: Rename @code{_Py_char2wchar()} to @ref{6fe,,Py_DecodeLocale()},
rename @code{_Py_wchar2char()} to @ref{6ff,,Py_EncodeLocale()}, and document
these functions.

@item 
bpo-21233@footnote{https://bugs.python.org/issue21233}: Add new C functions: PyMem_RawCalloc(), PyMem_Calloc(),
PyObject_Calloc(), _PyObject_GC_Calloc(). bytes(int) is now using
@code{calloc()} instead of @code{malloc()} for large objects which is faster and
use less memory.

@item 
bpo-20942@footnote{https://bugs.python.org/issue20942}: PyImport_ImportFrozenModuleObject() no longer sets __file__ to
match what importlib does; this affects _frozen_importlib as well as any
module loaded using imp.init_frozen().
@end itemize

@node Documentation<56>,Tests<59>,C API<48>,Python 3 5 0 alpha 1
@anchor{whatsnew/changelog id576}@anchor{14b5}
@subsubsection Documentation



@itemize -

@item 
bpo-19548@footnote{https://bugs.python.org/issue19548}: Update the codecs module documentation to better cover the
distinction between text encodings and other codecs, together with other
clarifications. Patch by Martin Panter.

@item 
bpo-22394@footnote{https://bugs.python.org/issue22394}: Doc/Makefile now supports @code{make venv PYTHON=../python} to
create a venv for generating the documentation, e.g., @code{make html
PYTHON=venv/bin/python3}.

@item 
bpo-21514@footnote{https://bugs.python.org/issue21514}: The documentation of the json module now refers to new JSON RFC
7159 instead of obsoleted RFC 4627.

@item 
bpo-21777@footnote{https://bugs.python.org/issue21777}: The binary sequence methods on bytes and bytearray are now
documented explicitly, rather than assuming users will be able to derive
the expected behaviour from the behaviour of the corresponding str
methods.

@item 
bpo-6916@footnote{https://bugs.python.org/issue6916}: undocument deprecated asynchat.fifo class.

@item 
bpo-17386@footnote{https://bugs.python.org/issue17386}: Expanded functionality of the @code{Doc/make.bat} script to make
it much more comparable to @code{Doc/Makefile}.

@item 
bpo-21312@footnote{https://bugs.python.org/issue21312}: Update the thread_foobar.h template file to include newer
threading APIs. Patch by Jack McCracken.

@item 
bpo-21043@footnote{https://bugs.python.org/issue21043}: Remove the recommendation for specific CA organizations and to
mention the ability to load the OS certificates.

@item 
bpo-20765@footnote{https://bugs.python.org/issue20765}: Add missing documentation for PurePath.with_name() and
PurePath.with_suffix().

@item 
bpo-19407@footnote{https://bugs.python.org/issue19407}: New package installation and distribution guides based on the
Python Packaging Authority tools. Existing guides have been retained as
legacy links from the distutils docs, as they still contain some required
reference material for tool developers that isn’t recorded anywhere else.

@item 
bpo-19697@footnote{https://bugs.python.org/issue19697}: Document cases where __main__.__spec__ is None.
@end itemize

@node Tests<59>,Tools/Demos<35>,Documentation<56>,Python 3 5 0 alpha 1
@anchor{whatsnew/changelog id577}@anchor{14b6}
@subsubsection Tests



@itemize -

@item 
bpo-18982@footnote{https://bugs.python.org/issue18982}: Add tests for CLI of the calendar module.

@item 
bpo-19548@footnote{https://bugs.python.org/issue19548}: Added some additional checks to test_codecs to ensure that
statements in the updated documentation remain accurate. Patch by Martin
Panter.

@item 
bpo-22838@footnote{https://bugs.python.org/issue22838}: All test_re tests now work with unittest test discovery.

@item 
bpo-22173@footnote{https://bugs.python.org/issue22173}: Update lib2to3 tests to use unittest test discovery.

@item 
bpo-16000@footnote{https://bugs.python.org/issue16000}: Convert test_curses to use unittest.

@item 
bpo-21456@footnote{https://bugs.python.org/issue21456}: Skip two tests in test_urllib2net.py if _ssl module not
present. Patch by Remi Pointel.

@item 
bpo-20746@footnote{https://bugs.python.org/issue20746}: Fix test_pdb to run in refleak mode (-R).  Patch by Xavier de
Gaye.

@item 
bpo-22060@footnote{https://bugs.python.org/issue22060}: test_ctypes has been somewhat cleaned up and simplified; it now
uses unittest test discovery to find its tests.

@item 
bpo-22104@footnote{https://bugs.python.org/issue22104}: regrtest.py no longer holds a reference to the suite of tests
loaded from test modules that don’t define test_main().

@item 
bpo-22111@footnote{https://bugs.python.org/issue22111}: Assorted cleanups in test_imaplib.  Patch by Milan Oberkirch.

@item 
bpo-22002@footnote{https://bugs.python.org/issue22002}: Added @code{load_package_tests} function to test.support and used
it to implement/augment test discovery in test_asyncio, test_email,
test_importlib, test_json, and test_tools.

@item 
bpo-21976@footnote{https://bugs.python.org/issue21976}: Fix test_ssl to accept LibreSSL version strings.  Thanks to
William Orr.

@item 
bpo-21918@footnote{https://bugs.python.org/issue21918}: Converted test_tools from a module to a package containing
separate test files for each tested script.

@item 
bpo-9554@footnote{https://bugs.python.org/issue9554}: Use modern unittest features in test_argparse. Initial patch by
Denver Coneybeare and Radu Voicilas.

@item 
bpo-20155@footnote{https://bugs.python.org/issue20155}: Changed HTTP method names in failing tests in test_httpservers
so that packet filtering software (specifically Windows Base Filtering
Engine) does not interfere with the transaction semantics expected by the
tests.

@item 
bpo-19493@footnote{https://bugs.python.org/issue19493}: Refactored the ctypes test package to skip tests explicitly
rather than silently.

@item 
bpo-18492@footnote{https://bugs.python.org/issue18492}: All resources are now allowed when tests are not run by
regrtest.py.

@item 
bpo-21634@footnote{https://bugs.python.org/issue21634}: Fix pystone micro-benchmark: use floor division instead of true
division to benchmark integers instead of floating point numbers. Set
pystone version to 1.2. Patch written by Lennart Regebro.

@item 
bpo-21605@footnote{https://bugs.python.org/issue21605}: Added tests for Tkinter images.

@item 
bpo-21493@footnote{https://bugs.python.org/issue21493}: Added test for ntpath.expanduser().  Original patch by Claudiu
Popa.

@item 
bpo-19925@footnote{https://bugs.python.org/issue19925}: Added tests for the spwd module. Original patch by Vajrasky
Kok.

@item 
bpo-21522@footnote{https://bugs.python.org/issue21522}: Added Tkinter tests for Listbox.itemconfigure(),
PanedWindow.paneconfigure(), and Menu.entryconfigure().

@item 
bpo-17756@footnote{https://bugs.python.org/issue17756}: Fix test_code test when run from the installed location.

@item 
bpo-17752@footnote{https://bugs.python.org/issue17752}: Fix distutils tests when run from the installed location.

@item 
bpo-18604@footnote{https://bugs.python.org/issue18604}: Consolidated checks for GUI availability.  All platforms now at
least check whether Tk can be instantiated when the GUI resource is
requested.

@item 
bpo-21275@footnote{https://bugs.python.org/issue21275}: Fix a socket test on KFreeBSD.

@item 
bpo-21223@footnote{https://bugs.python.org/issue21223}: Pass test_site/test_startup_imports when some of the extensions
are built as builtins.

@item 
bpo-20635@footnote{https://bugs.python.org/issue20635}: Added tests for Tk geometry managers.

@item 
Add test case for freeze.

@item 
bpo-20743@footnote{https://bugs.python.org/issue20743}: Fix a reference leak in test_tcl.

@item 
bpo-21097@footnote{https://bugs.python.org/issue21097}: Move test_namespace_pkgs into test_importlib.

@item 
bpo-21503@footnote{https://bugs.python.org/issue21503}: Use test_both() consistently in test_importlib.

@item 
bpo-20939@footnote{https://bugs.python.org/issue20939}: Avoid various network test failures due to new redirect of
@indicateurl{http://www.python.org/} to @indicateurl{https://www.python.org}: use
@indicateurl{http://www.example.com} instead.

@item 
bpo-20668@footnote{https://bugs.python.org/issue20668}: asyncio tests no longer rely on tests.txt file. (Patch by
Vajrasky Kok)

@item 
bpo-21093@footnote{https://bugs.python.org/issue21093}: Prevent failures of ctypes test_macholib on OS X if a copy of
libz exists in $HOME/lib or /usr/local/lib.

@item 
bpo-22770@footnote{https://bugs.python.org/issue22770}: Prevent some Tk segfaults on OS X when running gui tests.

@item 
bpo-23211@footnote{https://bugs.python.org/issue23211}: Workaround test_logging failure on some OS X 10.6 systems.

@item 
bpo-23345@footnote{https://bugs.python.org/issue23345}: Prevent test_ssl failures with large OpenSSL patch level values
(like 0.9.8zc).
@end itemize

@node Tools/Demos<35>,Windows<54>,Tests<59>,Python 3 5 0 alpha 1
@anchor{whatsnew/changelog id578}@anchor{14b7}
@subsubsection Tools/Demos



@itemize -

@item 
bpo-22314@footnote{https://bugs.python.org/issue22314}: pydoc now works when the LINES environment variable is set.

@item 
bpo-22615@footnote{https://bugs.python.org/issue22615}: Argument Clinic now supports the “type” argument for the int
converter. This permits using the int converter with enums and typedefs.

@item 
bpo-20076@footnote{https://bugs.python.org/issue20076}: The makelocalealias.py script no longer ignores UTF-8 mapping.

@item 
bpo-20079@footnote{https://bugs.python.org/issue20079}: The makelocalealias.py script now can parse the SUPPORTED file
from glibc sources and supports command line options for source paths.

@item 
bpo-22201@footnote{https://bugs.python.org/issue22201}: Command-line interface of the zipfile module now correctly
extracts ZIP files with directory entries.  Patch by Ryan Wilson.

@item 
bpo-22120@footnote{https://bugs.python.org/issue22120}: For functions using an unsigned integer return converter,
Argument Clinic now generates a cast to that type for the comparison to -1
in the generated code.  (This suppresses a compilation warning.)

@item 
bpo-18974@footnote{https://bugs.python.org/issue18974}: Tools/scripts/diff.py now uses argparse instead of optparse.

@item 
bpo-21906@footnote{https://bugs.python.org/issue21906}: Make Tools/scripts/md5sum.py work in Python 3. Patch by Zachary
Ware.

@item 
bpo-21629@footnote{https://bugs.python.org/issue21629}: Fix Argument Clinic’s “–converters” feature.

@item 
Add support for @code{yield from} to 2to3.

@item 
Add support for the 
@geindex Python Enhancement Proposals; PEP 465
PEP 465@footnote{https://www.python.org/dev/peps/pep-0465} matrix multiplication operator to 2to3.

@item 
bpo-16047@footnote{https://bugs.python.org/issue16047}: Fix module exception list and __file__ handling in freeze.
Patch by Meador Inge.

@item 
bpo-11824@footnote{https://bugs.python.org/issue11824}: Consider ABI tags in freeze. Patch by Meador Inge.

@item 
bpo-20535@footnote{https://bugs.python.org/issue20535}: PYTHONWARNING no longer affects the run_tests.py script. Patch
by Arfrever Frehtes Taifersar Arahesis.
@end itemize

@node Windows<54>,,Tools/Demos<35>,Python 3 5 0 alpha 1
@anchor{whatsnew/changelog id579}@anchor{14b8}
@subsubsection Windows



@itemize -

@item 
bpo-23260@footnote{https://bugs.python.org/issue23260}: Update Windows installer

@item 
The bundled version of Tcl/Tk has been updated to 8.6.3.  The most visible
result of this change is the addition of new native file dialogs when
running on Windows Vista or newer.  See Tcl/Tk’s TIP 432 for more
information.  Also, this version of Tcl/Tk includes support for Windows
10.

@item 
bpo-17896@footnote{https://bugs.python.org/issue17896}: The Windows build scripts now expect external library sources
to be in @code{PCbuild\..\externals} rather than @code{PCbuild\..\..}.

@item 
bpo-17717@footnote{https://bugs.python.org/issue17717}: The Windows build scripts now use a copy of NASM pulled from
svn.python.org to build OpenSSL.

@item 
bpo-21907@footnote{https://bugs.python.org/issue21907}: Improved the batch scripts provided for building Python.

@item 
bpo-22644@footnote{https://bugs.python.org/issue22644}: The bundled version of OpenSSL has been updated to 1.0.1j.

@item 
bpo-10747@footnote{https://bugs.python.org/issue10747}: Use versioned labels in the Windows start menu. Patch by Olive
Kilburn.

@item 
bpo-22980@footnote{https://bugs.python.org/issue22980}: .pyd files with a version and platform tag (for example,
“.cp35-win32.pyd”) will now be loaded in preference to those without tags.
@end itemize

@strong{(For information about older versions, consult the HISTORY file.)}

@node The Python Tutorial,Python Setup and Usage,What’s New in Python,Top
@anchor{tutorial/index doc}@anchor{14b9}@anchor{tutorial/index the-python-tutorial}@anchor{14ba}@anchor{tutorial/index tutorial-index}@anchor{14bb}
@chapter The Python Tutorial


Python is an easy to learn, powerful programming language. It has efficient
high-level data structures and a simple but effective approach to
object-oriented programming. Python’s elegant syntax and dynamic typing,
together with its interpreted nature, make it an ideal language for scripting
and rapid application development in many areas on most platforms.

The Python interpreter and the extensive standard library are freely available
in source or binary form for all major platforms from the Python web site,
@indicateurl{https://www.python.org/}, and may be freely distributed. The same site also
contains distributions of and pointers to many free third party Python modules,
programs and tools, and additional documentation.

The Python interpreter is easily extended with new functions and data types
implemented in C or C++ (or other languages callable from C). Python is also
suitable as an extension language for customizable applications.

This tutorial introduces the reader informally to the basic concepts and
features of the Python language and system. It helps to have a Python
interpreter handy for hands-on experience, but all examples are self-contained,
so the tutorial can be read off-line as well.

For a description of standard objects and modules, see @ref{14bc,,The Python Standard Library}.
@ref{14bd,,The Python Language Reference} gives a more formal definition of the language.  To write
extensions in C or C++, read @ref{14be,,Extending and Embedding the Python Interpreter} and
@ref{14bf,,Python/C API Reference Manual}. There are also several books covering Python in depth.

This tutorial does not attempt to be comprehensive and cover every single
feature, or even every commonly used feature. Instead, it introduces many of
Python’s most noteworthy features, and will give you a good idea of the
language’s flavor and style. After reading it, you will be able to read and
write Python modules and programs, and you will be ready to learn more about the
various Python library modules described in @ref{14bc,,The Python Standard Library}.

The @ref{14c0,,Glossary} is also worth going through.

@menu
* Whetting Your Appetite:: 
* Using the Python Interpreter:: 
* An Informal Introduction to Python:: 
* More Control Flow Tools:: 
* Data Structures:: 
* Modules:: 
* Input and Output:: 
* Errors and Exceptions:: 
* Classes:: 
* Brief Tour of the Standard Library:: 
* Brief Tour of the Standard Library — Part II:: 
* Virtual Environments and Packages:: 
* What Now?:: 
* Interactive Input Editing and History Substitution:: 
* Floating Point Arithmetic; Issues and Limitations: Floating Point Arithmetic Issues and Limitations. 
* Appendix:: 

@end menu

@node Whetting Your Appetite,Using the Python Interpreter,,The Python Tutorial
@anchor{tutorial/appetite doc}@anchor{14c1}@anchor{tutorial/appetite tut-intro}@anchor{14c2}@anchor{tutorial/appetite whetting-your-appetite}@anchor{14c3}
@section Whetting Your Appetite


If you do much work on computers, eventually you find that there’s some task
you’d like to automate.  For example, you may wish to perform a
search-and-replace over a large number of text files, or rename and rearrange a
bunch of photo files in a complicated way. Perhaps you’d like to write a small
custom database, or a specialized GUI application, or a simple game.

If you’re a professional software developer, you may have to work with several
C/C++/Java libraries but find the usual write/compile/test/re-compile cycle is
too slow.  Perhaps you’re writing a test suite for such a library and find
writing the testing code a tedious task.  Or maybe you’ve written a program that
could use an extension language, and you don’t want to design and implement a
whole new language for your application.

Python is just the language for you.

You could write a Unix shell script or Windows batch files for some of these
tasks, but shell scripts are best at moving around files and changing text data,
not well-suited for GUI applications or games. You could write a C/C++/Java
program, but it can take a lot of development time to get even a first-draft
program.  Python is simpler to use, available on Windows, macOS, and Unix
operating systems, and will help you get the job done more quickly.

Python is simple to use, but it is a real programming language, offering much
more structure and support for large programs than shell scripts or batch files
can offer.  On the other hand, Python also offers much more error checking than
C, and, being a @emph{very-high-level language}, it has high-level data types built
in, such as flexible arrays and dictionaries.  Because of its more general data
types Python is applicable to a much larger problem domain than Awk or even
Perl, yet many things are at least as easy in Python as in those languages.

Python allows you to split your program into modules that can be reused in other
Python programs.  It comes with a large collection of standard modules that you
can use as the basis of your programs — or as examples to start learning to
program in Python.  Some of these modules provide things like file I/O, system
calls, sockets, and even interfaces to graphical user interface toolkits like
Tk.

Python is an interpreted language, which can save you considerable time during
program development because no compilation and linking is necessary.  The
interpreter can be used interactively, which makes it easy to experiment with
features of the language, to write throw-away programs, or to test functions
during bottom-up program development. It is also a handy desk calculator.

Python enables programs to be written compactly and readably.  Programs written
in Python are typically much shorter than equivalent C,  C++, or Java programs,
for several reasons:


@itemize *

@item 
the high-level data types allow you to express complex operations in a single
statement;

@item 
statement grouping is done by indentation instead of beginning and ending
brackets;

@item 
no variable or argument declarations are necessary.
@end itemize

Python is @emph{extensible}: if you know how to program in C it is easy to add a new
built-in function or module to the interpreter, either to perform critical
operations at maximum speed, or to link Python programs to libraries that may
only be available in binary form (such as a vendor-specific graphics library).
Once you are really hooked, you can link the Python interpreter into an
application written in C and use it as an extension or command language for that
application.

By the way, the language is named after the BBC show “Monty Python’s Flying
Circus” and has nothing to do with reptiles.  Making references to Monty
Python skits in documentation is not only allowed, it is encouraged!

Now that you are all excited about Python, you’ll want to examine it in some
more detail.  Since the best way to learn a language is to use it, the tutorial
invites you to play with the Python interpreter as you read.

In the next chapter, the mechanics of using the interpreter are explained.  This
is rather mundane information, but essential for trying out the examples shown
later.

The rest of the tutorial introduces various features of the Python language and
system through examples, beginning with simple expressions, statements and data
types, through functions and modules, and finally touching upon advanced
concepts like exceptions and user-defined classes.

@node Using the Python Interpreter,An Informal Introduction to Python,Whetting Your Appetite,The Python Tutorial
@anchor{tutorial/interpreter doc}@anchor{14c4}@anchor{tutorial/interpreter tut-using}@anchor{14c5}@anchor{tutorial/interpreter using-the-python-interpreter}@anchor{14c6}
@section Using the Python Interpreter


@menu
* Invoking the Interpreter:: 
* The Interpreter and Its Environment:: 

@end menu

@node Invoking the Interpreter,The Interpreter and Its Environment,,Using the Python Interpreter
@anchor{tutorial/interpreter invoking-the-interpreter}@anchor{14c7}@anchor{tutorial/interpreter tut-invoking}@anchor{14c8}
@subsection Invoking the Interpreter


The Python interpreter is usually installed as @code{/usr/local/bin/python3.11}
on those machines where it is available; putting @code{/usr/local/bin} in your
Unix shell’s search path makes it possible to start it by typing the command:

@example
python3.11
@end example

to the shell. @footnote{@w{(1)} 
On Unix, the Python 3.x interpreter is by default not installed with the
executable named @code{python}, so that it does not conflict with a
simultaneously installed Python 2.x executable.
} Since the choice of the directory where the interpreter lives
is an installation option, other places are possible; check with your local
Python guru or system administrator.  (E.g., @code{/usr/local/python} is a
popular alternative location.)

On Windows machines where you have installed Python from the @ref{14c9,,Microsoft Store}, the @code{python3.11} command will be available. If you have
the @ref{be5,,py.exe launcher} installed, you can use the @code{py}
command. See @ref{14ca,,Excursus; Setting environment variables} for other ways to launch Python.

Typing an end-of-file character (@code{Control-D} on Unix, @code{Control-Z} on
Windows) at the primary prompt causes the interpreter to exit with a zero exit
status.  If that doesn’t work, you can exit the interpreter by typing the
following command: @code{quit()}.

The interpreter’s line-editing features include interactive editing, history
substitution and code completion on systems that support the GNU Readline@footnote{https://tiswww.case.edu/php/chet/readline/rltop.html} library.
Perhaps the quickest check to see whether command line editing is supported is
typing @code{Control-P} to the first Python prompt you get.  If it beeps, you
have command line editing; see Appendix @ref{14cb,,Interactive Input Editing and History Substitution} for an
introduction to the keys.  If nothing appears to happen, or if @code{^P} is
echoed, command line editing isn’t available; you’ll only be able to use
backspace to remove characters from the current line.

The interpreter operates somewhat like the Unix shell: when called with standard
input connected to a tty device, it reads and executes commands interactively;
when called with a file name argument or with a file as standard input, it reads
and executes a @emph{script} from that file.

A second way of starting the interpreter is @code{python -c command [arg] ...},
which executes the statement(s) in @emph{command}, analogous to the shell’s
@ref{1180,,-c} option.  Since Python statements often contain spaces or other
characters that are special to the shell, it is usually advised to quote
@emph{command} in its entirety with single quotes.

Some Python modules are also useful as scripts.  These can be invoked using
@code{python -m module [arg] ...}, which executes the source file for @emph{module} as
if you had spelled out its full name on the command line.

When a script file is used, it is sometimes useful to be able to run the script
and enter interactive mode afterwards.  This can be done by passing @ref{1018,,-i}
before the script.

All command line options are described in @ref{1259,,Command line and environment}.

@menu
* Argument Passing:: 
* Interactive Mode:: 

@end menu

@node Argument Passing,Interactive Mode,,Invoking the Interpreter
@anchor{tutorial/interpreter argument-passing}@anchor{14cc}@anchor{tutorial/interpreter tut-argpassing}@anchor{14cd}
@subsubsection Argument Passing


When known to the interpreter, the script name and additional arguments
thereafter are turned into a list of strings and assigned to the @code{argv}
variable in the @code{sys} module.  You can access this list by executing @code{import
sys}.  The length of the list is at least one; when no script and no arguments
are given, @code{sys.argv[0]} is an empty string.  When the script name is given as
@code{'-'} (meaning  standard input), @code{sys.argv[0]} is set to @code{'-'}.  When
@ref{1180,,-c} @emph{command} is used, @code{sys.argv[0]} is set to @code{'-c'}.  When
@ref{611,,-m} @emph{module} is used, @code{sys.argv[0]}  is set to the full name of the
located module.  Options found after  @ref{1180,,-c} @emph{command} or @ref{611,,-m}
@emph{module} are not consumed  by the Python interpreter’s option processing but
left in @code{sys.argv} for  the command or module to handle.

@node Interactive Mode,,Argument Passing,Invoking the Interpreter
@anchor{tutorial/interpreter interactive-mode}@anchor{14ce}@anchor{tutorial/interpreter tut-interactive}@anchor{b4b}
@subsubsection Interactive Mode


When commands are read from a tty, the interpreter is said to be in @emph{interactive
mode}.  In this mode it prompts for the next command with the @emph{primary prompt},
usually three greater-than signs (@code{>>>}); for continuation lines it prompts
with the @emph{secondary prompt}, by default three dots (@code{...}). The interpreter
prints a welcome message stating its version number and a copyright notice
before printing the first prompt:

@example
$ python3.11
Python 3.11 (default, April 4 2021, 09:25:04)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
@end example

@c XXX update for new releases

Continuation lines are needed when entering a multi-line construct. As an
example, take a look at this @ref{2fb,,if} statement:

@example
>>> the_world_is_flat = True
>>> if the_world_is_flat:
...     print("Be careful not to fall off!")
...
Be careful not to fall off!
@end example

For more on interactive mode, see @ref{14cf,,Interactive Mode}.

@node The Interpreter and Its Environment,,Invoking the Interpreter,Using the Python Interpreter
@anchor{tutorial/interpreter the-interpreter-and-its-environment}@anchor{14d0}@anchor{tutorial/interpreter tut-interp}@anchor{14d1}
@subsection The Interpreter and Its Environment


@menu
* Source Code Encoding:: 

@end menu

@node Source Code Encoding,,,The Interpreter and Its Environment
@anchor{tutorial/interpreter source-code-encoding}@anchor{14d2}@anchor{tutorial/interpreter tut-source-encoding}@anchor{14d3}
@subsubsection Source Code Encoding


By default, Python source files are treated as encoded in UTF-8.  In that
encoding, characters of most languages in the world can be used simultaneously
in string literals, identifiers and comments — although the standard library
only uses ASCII characters for identifiers, a convention that any portable code
should follow.  To display all these characters properly, your editor must
recognize that the file is UTF-8, and it must use a font that supports all the
characters in the file.

To declare an encoding other than the default one, a special comment line
should be added as the @emph{first} line of the file.  The syntax is as follows:

@example
# -*- coding: encoding -*-
@end example

where @emph{encoding} is one of the valid @ref{1a,,codecs} supported by Python.

For example, to declare that Windows-1252 encoding is to be used, the first
line of your source code file should be:

@example
# -*- coding: cp1252 -*-
@end example

One exception to the @emph{first line} rule is when the source code starts with a
@ref{14d4,,UNIX “shebang” line}.  In this case, the encoding
declaration should be added as the second line of the file.  For example:

@example
#!/usr/bin/env python3
# -*- coding: cp1252 -*-
@end example

@node An Informal Introduction to Python,More Control Flow Tools,Using the Python Interpreter,The Python Tutorial
@anchor{tutorial/introduction doc}@anchor{14d5}@anchor{tutorial/introduction an-informal-introduction-to-python}@anchor{14d6}@anchor{tutorial/introduction tut-informal}@anchor{14d7}
@section An Informal Introduction to Python


In the following examples, input and output are distinguished by the presence or
absence of prompts (@ref{14d8,,>>>} and @ref{14d9,,…}): to repeat the example, you must type
everything after the prompt, when the prompt appears; lines that do not begin
with a prompt are output from the interpreter. Note that a secondary prompt on a
line by itself in an example means you must type a blank line; this is used to
end a multi-line command.


@geindex # (hash); comment

Many of the examples in this manual, even those entered at the interactive
prompt, include comments.  Comments in Python start with the hash character,
@code{#}, and extend to the end of the physical line.  A comment may appear at the
start of a line or following whitespace or code, but not within a string
literal.  A hash character within a string literal is just a hash character.
Since comments are to clarify code and are not interpreted by Python, they may
be omitted when typing in examples.

Some examples:

@example
# this is the first comment
spam = 1  # and this is the second comment
          # ... and now a third!
text = "# This is not a comment because it's inside quotes."
@end example

@menu
* Using Python as a Calculator:: 
* First Steps Towards Programming:: 

@end menu

@node Using Python as a Calculator,First Steps Towards Programming,,An Informal Introduction to Python
@anchor{tutorial/introduction tut-calculator}@anchor{14da}@anchor{tutorial/introduction using-python-as-a-calculator}@anchor{14db}
@subsection Using Python as a Calculator


Let’s try some simple Python commands.  Start the interpreter and wait for the
primary prompt, @code{>>>}.  (It shouldn’t take long.)

@menu
* Numbers:: 
* Strings:: 
* Lists:: 

@end menu

@node Numbers,Strings,,Using Python as a Calculator
@anchor{tutorial/introduction numbers}@anchor{14dc}@anchor{tutorial/introduction tut-numbers}@anchor{14dd}
@subsubsection Numbers


The interpreter acts as a simple calculator: you can type an expression at it
and it will write the value.  Expression syntax is straightforward: the
operators @code{+}, @code{-}, @code{*} and @code{/} work just like in most other languages
(for example, Pascal or C); parentheses (@code{()}) can be used for grouping.
For example:

@example
>>> 2 + 2
4
>>> 50 - 5*6
20
>>> (50 - 5*6) / 4
5.0
>>> 8 / 5  # division always returns a floating point number
1.6
@end example

The integer numbers (e.g. @code{2}, @code{4}, @code{20}) have type @ref{1b1,,int},
the ones with a fractional part (e.g. @code{5.0}, @code{1.6}) have type
@ref{256,,float}.  We will see more about numeric types later in the tutorial.

Division (@code{/}) always returns a float.  To do @ref{43b,,floor division} and
get an integer result (discarding any fractional result) you can use the @code{//}
operator; to calculate the remainder you can use @code{%}:

@example
>>> 17 / 3  # classic division returns a float
5.666666666666667
>>>
>>> 17 // 3  # floor division discards the fractional part
5
>>> 17 % 3  # the % operator returns the remainder of the division
2
>>> 5 * 3 + 2  # floored quotient * divisor + remainder
17
@end example

With Python, it is possible to use the @code{**} operator to calculate powers @footnote{@w{(1)} 
Since @code{**} has higher precedence than @code{-}, @code{-3**2} will be
interpreted as @code{-(3**2)} and thus result in @code{-9}.  To avoid this
and get @code{9}, you can use @code{(-3)**2}.
}:

@example
>>> 5 ** 2  # 5 squared
25
>>> 2 ** 7  # 2 to the power of 7
128
@end example

The equal sign (@code{=}) is used to assign a value to a variable. Afterwards, no
result is displayed before the next interactive prompt:

@example
>>> width = 20
>>> height = 5 * 9
>>> width * height
900
@end example

If a variable is not “defined” (assigned a value), trying to use it will
give you an error:

@example
>>> n  # try to access an undefined variable
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'n' is not defined
@end example

There is full support for floating point; operators with mixed type operands
convert the integer operand to floating point:

@example
>>> 4 * 3.75 - 1
14.0
@end example

In interactive mode, the last printed expression is assigned to the variable
@code{_}.  This means that when you are using Python as a desk calculator, it is
somewhat easier to continue calculations, for example:

@example
>>> tax = 12.5 / 100
>>> price = 100.50
>>> price * tax
12.5625
>>> price + _
113.0625
>>> round(_, 2)
113.06
@end example

This variable should be treated as read-only by the user.  Don’t explicitly
assign a value to it — you would create an independent local variable with the
same name masking the built-in variable with its magic behavior.

In addition to @ref{1b1,,int} and @ref{256,,float}, Python supports other types of
numbers, such as @ref{248,,Decimal} and @ref{172,,Fraction}.
Python also has built-in support for @ref{14de,,complex numbers},
and uses the @code{j} or @code{J} suffix to indicate the imaginary part
(e.g. @code{3+5j}).

@node Strings,Lists,Numbers,Using Python as a Calculator
@anchor{tutorial/introduction strings}@anchor{14df}@anchor{tutorial/introduction tut-strings}@anchor{14e0}
@subsubsection Strings


Besides numbers, Python can also manipulate strings, which can be expressed
in several ways.  They can be enclosed in single quotes (@code{'...'}) or
double quotes (@code{"..."}) with the same result @footnote{@w{(2)} 
Unlike other languages, special characters such as @code{\n} have the
same meaning with both single (@code{'...'}) and double (@code{"..."}) quotes.
The only difference between the two is that within single quotes you don’t
need to escape @code{"} (but you have to escape @code{\'}) and vice versa.
}.  @code{\} can be used
to escape quotes:

@example
>>> 'spam eggs'  # single quotes
'spam eggs'
>>> 'doesn\'t'  # use \' to escape the single quote...
"doesn't"
>>> "doesn't"  # ...or use double quotes instead
"doesn't"
>>> '"Yes," they said.'
'"Yes," they said.'
>>> "\"Yes,\" they said."
'"Yes," they said.'
>>> '"Isn\'t," they said.'
'"Isn\'t," they said.'
@end example

In the interactive interpreter, the output string is enclosed in quotes and
special characters are escaped with backslashes.  While this might sometimes
look different from the input (the enclosing quotes could change), the two
strings are equivalent.  The string is enclosed in double quotes if
the string contains a single quote and no double quotes, otherwise it is
enclosed in single quotes.  The @ref{af6,,print()} function produces a more
readable output, by omitting the enclosing quotes and by printing escaped
and special characters:

@example
>>> '"Isn\'t," they said.'
'"Isn\'t," they said.'
>>> print('"Isn\'t," they said.')
"Isn't," they said.
>>> s = 'First line.\nSecond line.'  # \n means newline
>>> s  # without print(), \n is included in the output
'First line.\nSecond line.'
>>> print(s)  # with print(), \n produces a new line
First line.
Second line.
@end example

If you don’t want characters prefaced by @code{\} to be interpreted as
special characters, you can use @emph{raw strings} by adding an @code{r} before
the first quote:

@example
>>> print('C:\some\name')  # here \n means newline!
C:\some
ame
>>> print(r'C:\some\name')  # note the r before the quote
C:\some\name
@end example

String literals can span multiple lines.  One way is using triple-quotes:
@code{"""..."""} or @code{'@w{'}'...'@w{'}'}.  End of lines are automatically
included in the string, but it’s possible to prevent this by adding a @code{\} at
the end of the line.  The following example:

@example
print("""\
Usage: thingy [OPTIONS]
     -h                        Display this usage message
     -H hostname               Hostname to connect to
""")
@end example

produces the following output (note that the initial newline is not included):

@example
Usage: thingy [OPTIONS]
     -h                        Display this usage message
     -H hostname               Hostname to connect to
@end example

Strings can be concatenated (glued together) with the @code{+} operator, and
repeated with @code{*}:

@example
>>> # 3 times 'un', followed by 'ium'
>>> 3 * 'un' + 'ium'
'unununium'
@end example

Two or more @emph{string literals} (i.e. the ones enclosed between quotes) next
to each other are automatically concatenated.

@example
>>> 'Py' 'thon'
'Python'
@end example

This feature is particularly useful when you want to break long strings:

@example
>>> text = ('Put several strings within parentheses '
...         'to have them joined together.')
>>> text
'Put several strings within parentheses to have them joined together.'
@end example

This only works with two literals though, not with variables or expressions:

@example
>>> prefix = 'Py'
>>> prefix 'thon'  # can't concatenate a variable and a string literal
  File "<stdin>", line 1
    prefix 'thon'
                ^
SyntaxError: invalid syntax
>>> ('un' * 3) 'ium'
  File "<stdin>", line 1
    ('un' * 3) 'ium'
                   ^
SyntaxError: invalid syntax
@end example

If you want to concatenate variables or a variable and a literal, use @code{+}:

@example
>>> prefix + 'thon'
'Python'
@end example

Strings can be @emph{indexed} (subscripted), with the first character having index 0.
There is no separate character type; a character is simply a string of size
one:

@example
>>> word = 'Python'
>>> word[0]  # character in position 0
'P'
>>> word[5]  # character in position 5
'n'
@end example

Indices may also be negative numbers, to start counting from the right:

@example
>>> word[-1]  # last character
'n'
>>> word[-2]  # second-last character
'o'
>>> word[-6]
'P'
@end example

Note that since -0 is the same as 0, negative indices start from -1.

In addition to indexing, @emph{slicing} is also supported.  While indexing is used
to obtain individual characters, @emph{slicing} allows you to obtain substring:

@example
>>> word[0:2]  # characters from position 0 (included) to 2 (excluded)
'Py'
>>> word[2:5]  # characters from position 2 (included) to 5 (excluded)
'tho'
@end example

Slice indices have useful defaults; an omitted first index defaults to zero, an
omitted second index defaults to the size of the string being sliced.

@example
>>> word[:2]   # character from the beginning to position 2 (excluded)
'Py'
>>> word[4:]   # characters from position 4 (included) to the end
'on'
>>> word[-2:]  # characters from the second-last (included) to the end
'on'
@end example

Note how the start is always included, and the end always excluded.  This
makes sure that @code{s[:i] + s[i:]} is always equal to @code{s}:

@example
>>> word[:2] + word[2:]
'Python'
>>> word[:4] + word[4:]
'Python'
@end example

One way to remember how slices work is to think of the indices as pointing
@emph{between} characters, with the left edge of the first character numbered 0.
Then the right edge of the last character of a string of @emph{n} characters has
index @emph{n}, for example:

@example
 +---+---+---+---+---+---+
 | P | y | t | h | o | n |
 +---+---+---+---+---+---+
 0   1   2   3   4   5   6
-6  -5  -4  -3  -2  -1
@end example

The first row of numbers gives the position of the indices 0…6 in the string;
the second row gives the corresponding negative indices. The slice from @emph{i} to
@emph{j} consists of all characters between the edges labeled @emph{i} and @emph{j},
respectively.

For non-negative indices, the length of a slice is the difference of the
indices, if both are within bounds.  For example, the length of @code{word[1:3]} is
2.

Attempting to use an index that is too large will result in an error:

@example
>>> word[42]  # the word only has 6 characters
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: string index out of range
@end example

However, out of range slice indexes are handled gracefully when used for
slicing:

@example
>>> word[4:42]
'on'
>>> word[42:]
'@w{'}
@end example

Python strings cannot be changed — they are @ref{14e1,,immutable}.
Therefore, assigning to an indexed position in the string results in an error:

@example
>>> word[0] = 'J'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'str' object does not support item assignment
>>> word[2:] = 'py'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'str' object does not support item assignment
@end example

If you need a different string, you should create a new one:

@example
>>> 'J' + word[1:]
'Jython'
>>> word[:2] + 'py'
'Pypy'
@end example

The built-in function @ref{4a6,,len()} returns the length of a string:

@example
>>> s = 'supercalifragilisticexpialidocious'
>>> len(s)
34
@end example


@subsubheading See also



@table @asis

@item @ref{14e2,,Text Sequence Type — str}

Strings are examples of @emph{sequence types}, and support the common
operations supported by such types.

@item @ref{14e3,,String Methods}

Strings support a large number of methods for
basic transformations and searching.

@item @ref{4b0,,Formatted string literals}

String literals that have embedded expressions.

@item @ref{f1a,,Format String Syntax}

Information about string formatting with @ref{784,,str.format()}.

@item @ref{14e4,,printf-style String Formatting}

The old formatting operations invoked when strings are
the left operand of the @code{%} operator are described in more detail here.
@end table


@node Lists,,Strings,Using Python as a Calculator
@anchor{tutorial/introduction lists}@anchor{14e5}@anchor{tutorial/introduction tut-lists}@anchor{14e6}
@subsubsection Lists


Python knows a number of @emph{compound} data types, used to group together other
values.  The most versatile is the @emph{list}, which can be written as a list of
comma-separated values (items) between square brackets.  Lists might contain
items of different types, but usually the items all have the same type.

@example
>>> squares = [1, 4, 9, 16, 25]
>>> squares
[1, 4, 9, 16, 25]
@end example

Like strings (and all other built-in @ref{14e7,,sequence} types), lists can be
indexed and sliced:

@example
>>> squares[0]  # indexing returns the item
1
>>> squares[-1]
25
>>> squares[-3:]  # slicing returns a new list
[9, 16, 25]
@end example

All slice operations return a new list containing the requested elements.  This
means that the following slice returns a
@ref{14e8,,shallow copy} of the list:

@example
>>> squares[:]
[1, 4, 9, 16, 25]
@end example

Lists also support operations like concatenation:

@example
>>> squares + [36, 49, 64, 81, 100]
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
@end example

Unlike strings, which are @ref{14e1,,immutable}, lists are a @ref{14e9,,mutable}
type, i.e. it is possible to change their content:

@example
>>> cubes = [1, 8, 27, 65, 125]  # something's wrong here
>>> 4 ** 3  # the cube of 4 is 64, not 65!
64
>>> cubes[3] = 64  # replace the wrong value
>>> cubes
[1, 8, 27, 64, 125]
@end example

You can also add new items at the end of the list, by using
the @code{append()} @emph{method} (we will see more about methods later):

@example
>>> cubes.append(216)  # add the cube of 6
>>> cubes.append(7 ** 3)  # and the cube of 7
>>> cubes
[1, 8, 27, 64, 125, 216, 343]
@end example

Assignment to slices is also possible, and this can even change the size of the
list or clear it entirely:

@example
>>> letters = ['a', 'b', 'c', 'd', 'e', 'f', 'g']
>>> letters
['a', 'b', 'c', 'd', 'e', 'f', 'g']
>>> # replace some values
>>> letters[2:5] = ['C', 'D', 'E']
>>> letters
['a', 'b', 'C', 'D', 'E', 'f', 'g']
>>> # now remove them
>>> letters[2:5] = []
>>> letters
['a', 'b', 'f', 'g']
>>> # clear the list by replacing all the elements with an empty list
>>> letters[:] = []
>>> letters
[]
@end example

The built-in function @ref{4a6,,len()} also applies to lists:

@example
>>> letters = ['a', 'b', 'c', 'd']
>>> len(letters)
4
@end example

It is possible to nest lists (create lists containing other lists), for
example:

@example
>>> a = ['a', 'b', 'c']
>>> n = [1, 2, 3]
>>> x = [a, n]
>>> x
[['a', 'b', 'c'], [1, 2, 3]]
>>> x[0]
['a', 'b', 'c']
>>> x[0][1]
'b'
@end example

@node First Steps Towards Programming,,Using Python as a Calculator,An Informal Introduction to Python
@anchor{tutorial/introduction first-steps-towards-programming}@anchor{14ea}@anchor{tutorial/introduction tut-firststeps}@anchor{14eb}
@subsection First Steps Towards Programming


Of course, we can use Python for more complicated tasks than adding two and two
together.  For instance, we can write an initial sub-sequence of the
Fibonacci series@footnote{https://en.wikipedia.org/wiki/Fibonacci_number}
as follows:

@example
>>> # Fibonacci series:
... # the sum of two elements defines the next
... a, b = 0, 1
>>> while a < 10:
...     print(a)
...     a, b = b, a+b
...
0
1
1
2
3
5
8
@end example

This example introduces several new features.


@itemize *

@item 
The first line contains a @emph{multiple assignment}: the variables @code{a} and @code{b}
simultaneously get the new values 0 and 1.  On the last line this is used again,
demonstrating that the expressions on the right-hand side are all evaluated
first before any of the assignments take place.  The right-hand side expressions
are evaluated  from the left to the right.

@item 
The @ref{14ec,,while} loop executes as long as the condition (here: @code{a < 10})
remains true.  In Python, like in C, any non-zero integer value is true; zero is
false.  The condition may also be a string or list value, in fact any sequence;
anything with a non-zero length is true, empty sequences are false.  The test
used in the example is a simple comparison.  The standard comparison operators
are written the same as in C: @code{<} (less than), @code{>} (greater than), @code{==}
(equal to), @code{<=} (less than or equal to), @code{>=} (greater than or equal to)
and @code{!=} (not equal to).

@item 
The @emph{body} of the loop is @emph{indented}: indentation is Python’s way of grouping
statements.  At the interactive prompt, you have to type a tab or space(s) for
each indented line.  In practice you will prepare more complicated input
for Python with a text editor; all decent text editors have an auto-indent
facility.  When a compound statement is entered interactively, it must be
followed by a blank line to indicate completion (since the parser cannot
guess when you have typed the last line).  Note that each line within a basic
block must be indented by the same amount.

@item 
The @ref{af6,,print()} function writes the value of the argument(s) it is given.
It differs from just writing the expression you want to write (as we did
earlier in the calculator examples) in the way it handles multiple arguments,
floating point quantities, and strings.  Strings are printed without quotes,
and a space is inserted between items, so you can format things nicely, like
this:

@example
>>> i = 256*256
>>> print('The value of i is', i)
The value of i is 65536
@end example

The keyword argument @emph{end} can be used to avoid the newline after the output,
or end the output with a different string:

@example
>>> a, b = 0, 1
>>> while a < 1000:
...     print(a, end=',')
...     a, b = b, a+b
...
0,1,1,2,3,5,8,13,21,34,55,89,144,233,377,610,987,
@end example
@end itemize

@node More Control Flow Tools,Data Structures,An Informal Introduction to Python,The Python Tutorial
@anchor{tutorial/controlflow doc}@anchor{14ed}@anchor{tutorial/controlflow more-control-flow-tools}@anchor{14ee}@anchor{tutorial/controlflow tut-morecontrol}@anchor{14ef}
@section More Control Flow Tools


Besides the @ref{14ec,,while} statement just introduced, Python uses the usual
flow control statements known from other languages, with some twists.

@menu
* if Statements:: 
* for Statements:: 
* The range() Function: The range Function. 
* break and continue Statements@comma{} and else Clauses on Loops: break and continue Statements and else Clauses on Loops. 
* pass Statements:: 
* match Statements:: 
* Defining Functions:: 
* More on Defining Functions:: 
* Intermezzo; Coding Style: Intermezzo Coding Style. 

@end menu

@node if Statements,for Statements,,More Control Flow Tools
@anchor{tutorial/controlflow if-statements}@anchor{14f0}@anchor{tutorial/controlflow tut-if}@anchor{14f1}
@subsection @code{if} Statements


Perhaps the most well-known statement type is the @ref{2fb,,if} statement.  For
example:

@example
>>> x = int(input("Please enter an integer: "))
Please enter an integer: 42
>>> if x < 0:
...     x = 0
...     print('Negative changed to zero')
... elif x == 0:
...     print('Zero')
... elif x == 1:
...     print('Single')
... else:
...     print('More')
...
More
@end example

There can be zero or more @ref{14f2,,elif} parts, and the @ref{2fa,,else} part is
optional.  The keyword ‘@code{elif}’ is short for ‘else if’, and is useful
to avoid excessive indentation.  An  @code{if} … @code{elif} …
@code{elif} … sequence is a substitute for the @code{switch} or
@code{case} statements found in other languages.

If you’re comparing the same value to several constants, or checking for specific types or
attributes, you may also find the @code{match} statement useful. For more
details see @ref{14f3,,match Statements}.

@node for Statements,The range Function,if Statements,More Control Flow Tools
@anchor{tutorial/controlflow for-statements}@anchor{14f4}@anchor{tutorial/controlflow tut-for}@anchor{14f5}
@subsection @code{for} Statements


@geindex statement; for

The @ref{161,,for} statement in Python differs a bit from what you may be used
to in C or Pascal.  Rather than always iterating over an arithmetic progression
of numbers (like in Pascal), or giving the user the ability to define both the
iteration step and halting condition (as C), Python’s @code{for} statement
iterates over the items of any sequence (a list or a string), in the order that
they appear in the sequence.  For example (no pun intended):

@c One suggestion was to give a real C example here, but that may only serve to
@c confuse non-C programmers.

@example
>>> # Measure some strings:
... words = ['cat', 'window', 'defenestrate']
>>> for w in words:
...     print(w, len(w))
...
cat 3
window 6
defenestrate 12
@end example

Code that modifies a collection while iterating over that same collection can
be tricky to get right.  Instead, it is usually more straight-forward to loop
over a copy of the collection or to create a new collection:

@example
# Create a sample collection
users = @{'Hans': 'active', 'Éléonore': 'inactive', '景太郎': 'active'@}

# Strategy:  Iterate over a copy
for user, status in users.copy().items():
    if status == 'inactive':
        del users[user]

# Strategy:  Create a new collection
active_users = @{@}
for user, status in users.items():
    if status == 'active':
        active_users[user] = status
@end example

@node The range Function,break and continue Statements and else Clauses on Loops,for Statements,More Control Flow Tools
@anchor{tutorial/controlflow the-range-function}@anchor{14f6}@anchor{tutorial/controlflow tut-range}@anchor{14f7}
@subsection The @code{range()} Function


If you do need to iterate over a sequence of numbers, the built-in function
@ref{438,,range()} comes in handy.  It generates arithmetic progressions:

@example
>>> for i in range(5):
...     print(i)
...
0
1
2
3
4
@end example

The given end point is never part of the generated sequence; @code{range(10)} generates
10 values, the legal indices for items of a sequence of length 10.  It
is possible to let the range start at another number, or to specify a different
increment (even negative; sometimes this is called the ‘step’):

@example
>>> list(range(5, 10))
[5, 6, 7, 8, 9]

>>> list(range(0, 10, 3))
[0, 3, 6, 9]

>>> list(range(-10, -100, -30))
[-10, -40, -70]
@end example

To iterate over the indices of a sequence, you can combine @ref{438,,range()} and
@ref{4a6,,len()} as follows:

@example
>>> a = ['Mary', 'had', 'a', 'little', 'lamb']
>>> for i in range(len(a)):
...     print(i, a[i])
...
0 Mary
1 had
2 a
3 little
4 lamb
@end example

In most such cases, however, it is convenient to use the @ref{fdc,,enumerate()}
function, see @ref{14f8,,Looping Techniques}.

A strange thing happens if you just print a range:

@example
>>> range(10)
range(0, 10)
@end example

In many ways the object returned by @ref{438,,range()} behaves as if it is a list,
but in fact it isn’t. It is an object which returns the successive items of
the desired sequence when you iterate over it, but it doesn’t really make
the list, thus saving space.

We say such an object is @ref{dcc,,iterable}, that is, suitable as a target for
functions and constructs that expect something from which they can
obtain successive items until the supply is exhausted.  We have seen that
the @ref{161,,for} statement is such a construct, while an example of a function
that takes an iterable is @ref{50d,,sum()}:

@example
>>> sum(range(4))  # 0 + 1 + 2 + 3
6
@end example

Later we will see more functions that return iterables and take iterables as
arguments.  In chapter @ref{14f9,,Data Structures}, we will discuss in more detail about
@ref{43a,,list()}.

@node break and continue Statements and else Clauses on Loops,pass Statements,The range Function,More Control Flow Tools
@anchor{tutorial/controlflow break-and-continue-statements-and-else-clauses-on-loops}@anchor{14fa}@anchor{tutorial/controlflow tut-break}@anchor{14fb}
@subsection @code{break} and @code{continue} Statements, and @code{else} Clauses on Loops


The @ref{5cb,,break} statement, like in C, breaks out of the innermost enclosing
@ref{161,,for} or @ref{14ec,,while} loop.

Loop statements may have an @code{else} clause; it is executed when the loop
terminates through exhaustion of the iterable (with @ref{161,,for}) or when the
condition becomes false (with @ref{14ec,,while}), but not when the loop is
terminated by a @ref{5cb,,break} statement.  This is exemplified by the
following loop, which searches for prime numbers:

@example
>>> for n in range(2, 10):
...     for x in range(2, n):
...         if n % x == 0:
...             print(n, 'equals', x, '*', n//x)
...             break
...     else:
...         # loop fell through without finding a factor
...         print(n, 'is a prime number')
...
2 is a prime number
3 is a prime number
4 equals 2 * 2
5 is a prime number
6 equals 2 * 3
7 is a prime number
8 equals 2 * 4
9 equals 3 * 3
@end example

(Yes, this is the correct code.  Look closely: the @code{else} clause belongs to
the @ref{161,,for} loop, @strong{not} the @ref{2fb,,if} statement.)

When used with a loop, the @code{else} clause has more in common with the
@code{else} clause of a @ref{f6d,,try} statement than it does with that of
@ref{2fb,,if} statements: a @ref{f6d,,try} statement’s @code{else} clause runs
when no exception occurs, and a loop’s @code{else} clause runs when no @code{break}
occurs. For more on the @code{try} statement and exceptions, see
@ref{14fc,,Handling Exceptions}.

The @ref{4c0,,continue} statement, also borrowed from C, continues with the next
iteration of the loop:

@example
>>> for num in range(2, 10):
...     if num % 2 == 0:
...         print("Found an even number", num)
...         continue
...     print("Found an odd number", num)
...
Found an even number 2
Found an odd number 3
Found an even number 4
Found an odd number 5
Found an even number 6
Found an odd number 7
Found an even number 8
Found an odd number 9
@end example

@node pass Statements,match Statements,break and continue Statements and else Clauses on Loops,More Control Flow Tools
@anchor{tutorial/controlflow pass-statements}@anchor{14fd}@anchor{tutorial/controlflow tut-pass}@anchor{14fe}
@subsection @code{pass} Statements


The @ref{14ff,,pass} statement does nothing. It can be used when a statement is
required syntactically but the program requires no action. For example:

@example
>>> while True:
...     pass  # Busy-wait for keyboard interrupt (Ctrl+C)
...
@end example

This is commonly used for creating minimal classes:

@example
>>> class MyEmptyClass:
...     pass
...
@end example

Another place @ref{14ff,,pass} can be used is as a place-holder for a function or
conditional body when you are working on new code, allowing you to keep thinking
at a more abstract level.  The @code{pass} is silently ignored:

@example
>>> def initlog(*args):
...     pass   # Remember to implement this!
...
@end example

@node match Statements,Defining Functions,pass Statements,More Control Flow Tools
@anchor{tutorial/controlflow match-statements}@anchor{1500}@anchor{tutorial/controlflow tut-match}@anchor{14f3}
@subsection @code{match} Statements


A @ref{290,,match} statement takes an expression and compares its value to successive
patterns given as one or more case blocks.  This is superficially
similar to a switch statement in C, Java or JavaScript (and many
other languages), but it can also extract components (sequence elements or
object attributes) from the value into variables.

The simplest form compares a subject value against one or more literals:

@example
def http_error(status):
    match status:
        case 400:
            return "Bad request"
        case 404:
            return "Not found"
        case 418:
            return "I'm a teapot"
        case _:
            return "Something's wrong with the internet"
@end example

Note the last block: the “variable name” @code{_} acts as a @emph{wildcard} and
never fails to match. If no case matches, none of the branches is executed.

You can combine several literals in a single pattern using @code{|} (“or”):

@example
case 401 | 403 | 404:
    return "Not allowed"
@end example

Patterns can look like unpacking assignments, and can be used to bind
variables:

@example
# point is an (x, y) tuple
match point:
    case (0, 0):
        print("Origin")
    case (0, y):
        print(f"Y=@{y@}")
    case (x, 0):
        print(f"X=@{x@}")
    case (x, y):
        print(f"X=@{x@}, Y=@{y@}")
    case _:
        raise ValueError("Not a point")
@end example

Study that one carefully!  The first pattern has two literals, and can
be thought of as an extension of the literal pattern shown above.  But
the next two patterns combine a literal and a variable, and the
variable @emph{binds} a value from the subject (@code{point}).  The fourth
pattern captures two values, which makes it conceptually similar to
the unpacking assignment @code{(x, y) = point}.

If you are using classes to structure your data
you can use the class name followed by an argument list resembling a
constructor, but with the ability to capture attributes into variables:

@example
class Point:
    x: int
    y: int

def where_is(point):
    match point:
        case Point(x=0, y=0):
            print("Origin")
        case Point(x=0, y=y):
            print(f"Y=@{y@}")
        case Point(x=x, y=0):
            print(f"X=@{x@}")
        case Point():
            print("Somewhere else")
        case _:
            print("Not a point")
@end example

You can use positional parameters with some builtin classes that provide an
ordering for their attributes (e.g. dataclasses). You can also define a specific
position for attributes in patterns by setting the @code{__match_args__} special
attribute in your classes. If it’s set to (“x”, “y”), the following patterns are all
equivalent (and all bind the @code{y} attribute to the @code{var} variable):

@example
Point(1, var)
Point(1, y=var)
Point(x=1, y=var)
Point(y=var, x=1)
@end example

A recommended way to read patterns is to look at them as an extended form of what you
would put on the left of an assignment, to understand which variables would be set to
what.
Only the standalone names (like @code{var} above) are assigned to by a match statement.
Dotted names (like @code{foo.bar}), attribute names (the @code{x=} and @code{y=} above) or class names
(recognized by the “(…)” next to them like @code{Point} above) are never assigned to.

Patterns can be arbitrarily nested.  For example, if we have a short
list of points, we could match it like this:

@example
match points:
    case []:
        print("No points")
    case [Point(0, 0)]:
        print("The origin")
    case [Point(x, y)]:
        print(f"Single point @{x@}, @{y@}")
    case [Point(0, y1), Point(0, y2)]:
        print(f"Two on the Y axis at @{y1@}, @{y2@}")
    case _:
        print("Something else")
@end example

We can add an @code{if} clause to a pattern, known as a “guard”.  If the
guard is false, @code{match} goes on to try the next case block.  Note
that value capture happens before the guard is evaluated:

@example
match point:
    case Point(x, y) if x == y:
        print(f"Y=X at @{x@}")
    case Point(x, y):
        print(f"Not on the diagonal")
@end example

Several other key features of this statement:


@itemize -

@item 
Like unpacking assignments, tuple and list patterns have exactly the
same meaning and actually match arbitrary sequences.  An important
exception is that they don’t match iterators or strings.

@item 
Sequence patterns support extended unpacking: @code{[x, y, *rest]} and @code{(x, y,
*rest)} work similar to unpacking assignments.  The
name after @code{*} may also be @code{_}, so @code{(x, y, *_)} matches a sequence
of at least two items without binding the remaining items.

@item 
Mapping patterns: @code{@{"bandwidth": b, "latency": l@}} captures the
@code{"bandwidth"} and @code{"latency"} values from a dictionary.  Unlike sequence
patterns, extra keys are ignored.  An unpacking like @code{**rest} is also
supported.  (But @code{**_} would be redundant, so it is not allowed.)

@item 
Subpatterns may be captured using the @code{as} keyword:

@example
case (Point(x1, y1), Point(x2, y2) as p2): ...
@end example

will capture the second element of the input as @code{p2} (as long as the input is
a sequence of two points)

@item 
Most literals are compared by equality, however the singletons @code{True},
@code{False} and @code{None} are compared by identity.

@item 
Patterns may use named constants.  These must be dotted names
to prevent them from being interpreted as capture variable:

@example
from enum import Enum
class Color(Enum):
    RED = 'red'
    GREEN = 'green'
    BLUE = 'blue'

color = Color(input("Enter your choice of 'red', 'blue' or 'green': "))

match color:
    case Color.RED:
        print("I see red!")
    case Color.GREEN:
        print("Grass is green")
    case Color.BLUE:
        print("I'm feeling the blues :(")
@end example
@end itemize

For a more detailed explanation and additional examples, you can look into
@geindex Python Enhancement Proposals; PEP 636
PEP 636@footnote{https://www.python.org/dev/peps/pep-0636} which is written in a tutorial format.

@node Defining Functions,More on Defining Functions,match Statements,More Control Flow Tools
@anchor{tutorial/controlflow defining-functions}@anchor{1501}@anchor{tutorial/controlflow tut-functions}@anchor{1502}
@subsection Defining Functions


We can create a function that writes the Fibonacci series to an arbitrary
boundary:

@example
>>> def fib(n):    # write Fibonacci series up to n
...     """Print a Fibonacci series up to n."""
...     a, b = 0, 1
...     while a < n:
...         print(a, end=' ')
...         a, b = b, a+b
...     print()
...
>>> # Now call the function we just defined:
... fib(2000)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597
@end example

@geindex documentation strings
@geindex docstrings
@geindex strings@comma{} documentation

The keyword @ref{fb8,,def} introduces a function @emph{definition}.  It must be
followed by the function name and the parenthesized list of formal parameters.
The statements that form the body of the function start at the next line, and
must be indented.

The first statement of the function body can optionally be a string literal;
this string literal is the function’s documentation string, or @emph{docstring}.
(More about docstrings can be found in the section @ref{1503,,Documentation Strings}.)
There are tools which use docstrings to automatically produce online or printed
documentation, or to let the user interactively browse through code; it’s good
practice to include docstrings in code that you write, so make a habit of it.

The @emph{execution} of a function introduces a new symbol table used for the local
variables of the function.  More precisely, all variable assignments in a
function store the value in the local symbol table; whereas variable references
first look in the local symbol table, then in the local symbol tables of
enclosing functions, then in the global symbol table, and finally in the table
of built-in names. Thus, global variables and variables of enclosing functions
cannot be directly assigned a value within a function (unless, for global
variables, named in a @ref{1504,,global} statement, or, for variables of enclosing
functions, named in a @ref{e4d,,nonlocal} statement), although they may be
referenced.

The actual parameters (arguments) to a function call are introduced in the local
symbol table of the called function when it is called; thus, arguments are
passed using @emph{call by value} (where the @emph{value} is always an object @emph{reference},
not the value of the object). @footnote{@w{(1)} 
Actually, @emph{call by object reference} would be a better description,
since if a mutable object is passed, the caller will see any changes the
callee makes to it (items inserted into a list).
} When a function calls another function,
or calls itself recursively, a new
local symbol table is created for that call.

A function definition associates the function name with the function object in
the current symbol table.  The interpreter recognizes the object pointed to by
that name as a user-defined function.  Other names can also point to that same
function object and can also be used to access the function:

@example
>>> fib
<function fib at 10042ed0>
>>> f = fib
>>> f(100)
0 1 1 2 3 5 8 13 21 34 55 89
@end example

Coming from other languages, you might object that @code{fib} is not a function but
a procedure since it doesn’t return a value.  In fact, even functions without a
@ref{4c6,,return} statement do return a value, albeit a rather boring one.  This
value is called @code{None} (it’s a built-in name).  Writing the value @code{None} is
normally suppressed by the interpreter if it would be the only value written.
You can see it if you really want to using @ref{af6,,print()}:

@example
>>> fib(0)
>>> print(fib(0))
None
@end example

It is simple to write a function that returns a list of the numbers of the
Fibonacci series, instead of printing it:

@example
>>> def fib2(n):  # return Fibonacci series up to n
...     """Return a list containing the Fibonacci series up to n."""
...     result = []
...     a, b = 0, 1
...     while a < n:
...         result.append(a)    # see below
...         a, b = b, a+b
...     return result
...
>>> f100 = fib2(100)    # call it
>>> f100                # write the result
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]
@end example

This example, as usual, demonstrates some new Python features:


@itemize *

@item 
The @ref{4c6,,return} statement returns with a value from a function.
@code{return} without an expression argument returns @code{None}. Falling off
the end of a function also returns @code{None}.

@item 
The statement @code{result.append(a)} calls a @emph{method} of the list object
@code{result}.  A method is a function that ‘belongs’ to an object and is named
@code{obj.methodname}, where @code{obj} is some object (this may be an expression),
and @code{methodname} is the name of a method that is defined by the object’s type.
Different types define different methods.  Methods of different types may have
the same name without causing ambiguity.  (It is possible to define your own
object types and methods, using @emph{classes}, see @ref{1505,,Classes})
The method @code{append()} shown in the example is defined for list objects; it
adds a new element at the end of the list.  In this example it is equivalent to
@code{result = result + [a]}, but more efficient.
@end itemize

@node More on Defining Functions,Intermezzo Coding Style,Defining Functions,More Control Flow Tools
@anchor{tutorial/controlflow more-on-defining-functions}@anchor{1506}@anchor{tutorial/controlflow tut-defining}@anchor{1507}
@subsection More on Defining Functions


It is also possible to define functions with a variable number of arguments.
There are three forms, which can be combined.

@menu
* Default Argument Values:: 
* Keyword Arguments:: 
* Special parameters:: 
* Arbitrary Argument Lists:: 
* Unpacking Argument Lists:: 
* Lambda Expressions:: 
* Documentation Strings:: 
* Function Annotations:: 

@end menu

@node Default Argument Values,Keyword Arguments,,More on Defining Functions
@anchor{tutorial/controlflow default-argument-values}@anchor{1508}@anchor{tutorial/controlflow tut-defaultargs}@anchor{1509}
@subsubsection Default Argument Values


The most useful form is to specify a default value for one or more arguments.
This creates a function that can be called with fewer arguments than it is
defined to allow.  For example:

@example
def ask_ok(prompt, retries=4, reminder='Please try again!'):
    while True:
        ok = input(prompt)
        if ok in ('y', 'ye', 'yes'):
            return True
        if ok in ('n', 'no', 'nop', 'nope'):
            return False
        retries = retries - 1
        if retries < 0:
            raise ValueError('invalid user response')
        print(reminder)
@end example

This function can be called in several ways:


@itemize *

@item 
giving only the mandatory argument:
@code{ask_ok('Do you really want to quit?')}

@item 
giving one of the optional arguments:
@code{ask_ok('OK to overwrite the file?', 2)}

@item 
or even giving all arguments:
@code{ask_ok('OK to overwrite the file?', 2, 'Come on, only yes or no!')}
@end itemize

This example also introduces the @ref{2fc,,in} keyword. This tests whether or
not a sequence contains a certain value.

The default values are evaluated at the point of function definition in the
@emph{defining} scope, so that

@example
i = 5

def f(arg=i):
    print(arg)

i = 6
f()
@end example

will print @code{5}.

@strong{Important warning:}  The default value is evaluated only once. This makes a
difference when the default is a mutable object such as a list, dictionary, or
instances of most classes.  For example, the following function accumulates the
arguments passed to it on subsequent calls:

@example
def f(a, L=[]):
    L.append(a)
    return L

print(f(1))
print(f(2))
print(f(3))
@end example

This will print

@example
[1]
[1, 2]
[1, 2, 3]
@end example

If you don’t want the default to be shared between subsequent calls, you can
write the function like this instead:

@example
def f(a, L=None):
    if L is None:
        L = []
    L.append(a)
    return L
@end example

@node Keyword Arguments,Special parameters,Default Argument Values,More on Defining Functions
@anchor{tutorial/controlflow keyword-arguments}@anchor{150a}@anchor{tutorial/controlflow tut-keywordargs}@anchor{150b}
@subsubsection Keyword Arguments


Functions can also be called using @ref{85f,,keyword arguments}
of the form @code{kwarg=value}.  For instance, the following function:

@example
def parrot(voltage, state='a stiff', action='voom', type='Norwegian Blue'):
    print("-- This parrot wouldn't", action, end=' ')
    print("if you put", voltage, "volts through it.")
    print("-- Lovely plumage, the", type)
    print("-- It's", state, "!")
@end example

accepts one required argument (@code{voltage}) and three optional arguments
(@code{state}, @code{action}, and @code{type}).  This function can be called in any
of the following ways:

@example
parrot(1000)                                          # 1 positional argument
parrot(voltage=1000)                                  # 1 keyword argument
parrot(voltage=1000000, action='VOOOOOM')             # 2 keyword arguments
parrot(action='VOOOOOM', voltage=1000000)             # 2 keyword arguments
parrot('a million', 'bereft of life', 'jump')         # 3 positional arguments
parrot('a thousand', state='pushing up the daisies')  # 1 positional, 1 keyword
@end example

but all the following calls would be invalid:

@example
parrot()                     # required argument missing
parrot(voltage=5.0, 'dead')  # non-keyword argument after a keyword argument
parrot(110, voltage=220)     # duplicate value for the same argument
parrot(actor='John Cleese')  # unknown keyword argument
@end example

In a function call, keyword arguments must follow positional arguments.
All the keyword arguments passed must match one of the arguments
accepted by the function (e.g. @code{actor} is not a valid argument for the
@code{parrot} function), and their order is not important.  This also includes
non-optional arguments (e.g. @code{parrot(voltage=1000)} is valid too).
No argument may receive a value more than once.
Here’s an example that fails due to this restriction:

@example
>>> def function(a):
...     pass
...
>>> function(0, a=0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: function() got multiple values for argument 'a'
@end example

When a final formal parameter of the form @code{**name} is present, it receives a
dictionary (see @ref{5de,,Mapping Types — dict}) containing all keyword arguments except for
those corresponding to a formal parameter.  This may be combined with a formal
parameter of the form @code{*name} (described in the next subsection) which
receives a @ref{150c,,tuple} containing the positional
arguments beyond the formal parameter list.  (@code{*name} must occur
before @code{**name}.) For example, if we define a function like this:

@example
def cheeseshop(kind, *arguments, **keywords):
    print("-- Do you have any", kind, "?")
    print("-- I'm sorry, we're all out of", kind)
    for arg in arguments:
        print(arg)
    print("-" * 40)
    for kw in keywords:
        print(kw, ":", keywords[kw])
@end example

It could be called like this:

@example
cheeseshop("Limburger", "It's very runny, sir.",
           "It's really very, VERY runny, sir.",
           shopkeeper="Michael Palin",
           client="John Cleese",
           sketch="Cheese Shop Sketch")
@end example

and of course it would print:

@example
-- Do you have any Limburger ?
-- I'm sorry, we're all out of Limburger
It's very runny, sir.
It's really very, VERY runny, sir.
----------------------------------------
shopkeeper : Michael Palin
client : John Cleese
sketch : Cheese Shop Sketch
@end example

Note that the order in which the keyword arguments are printed is guaranteed
to match the order in which they were provided in the function call.

@node Special parameters,Arbitrary Argument Lists,Keyword Arguments,More on Defining Functions
@anchor{tutorial/controlflow special-parameters}@anchor{150d}
@subsubsection Special parameters


By default, arguments may be passed to a Python function either by position
or explicitly by keyword. For readability and performance, it makes sense to
restrict the way arguments can be passed so that a developer need only look
at the function definition to determine if items are passed by position, by
position or keyword, or by keyword.

A function definition may look like:

@example
def f(pos1, pos2, /, pos_or_kwd, *, kwd1, kwd2):
      -----------    ----------     ----------
        |             |                  |
        |        Positional or keyword   |
        |                                - Keyword only
         -- Positional only
@end example

where @code{/} and @code{*} are optional. If used, these symbols indicate the kind of
parameter by how the arguments may be passed to the function:
positional-only, positional-or-keyword, and keyword-only. Keyword parameters
are also referred to as named parameters.

@menu
* Positional-or-Keyword Arguments:: 
* Positional-Only Parameters:: 
* Keyword-Only Arguments:: 
* Function Examples:: 
* Recap:: 

@end menu

@node Positional-or-Keyword Arguments,Positional-Only Parameters,,Special parameters
@anchor{tutorial/controlflow positional-or-keyword-arguments}@anchor{150e}
@subsubsection Positional-or-Keyword Arguments


If @code{/} and @code{*} are not present in the function definition, arguments may
be passed to a function by position or by keyword.

@node Positional-Only Parameters,Keyword-Only Arguments,Positional-or-Keyword Arguments,Special parameters
@anchor{tutorial/controlflow positional-only-parameters}@anchor{150f}
@subsubsection Positional-Only Parameters


Looking at this in a bit more detail, it is possible to mark certain parameters
as @emph{positional-only}. If @emph{positional-only}, the parameters’ order matters, and
the parameters cannot be passed by keyword. Positional-only parameters are
placed before a @code{/} (forward-slash). The @code{/} is used to logically
separate the positional-only parameters from the rest of the parameters.
If there is no @code{/} in the function definition, there are no positional-only
parameters.

Parameters following the @code{/} may be @emph{positional-or-keyword} or @emph{keyword-only}.

@node Keyword-Only Arguments,Function Examples,Positional-Only Parameters,Special parameters
@anchor{tutorial/controlflow keyword-only-arguments}@anchor{1510}
@subsubsection Keyword-Only Arguments


To mark parameters as @emph{keyword-only}, indicating the parameters must be passed
by keyword argument, place an @code{*} in the arguments list just before the first
@emph{keyword-only} parameter.

@node Function Examples,Recap,Keyword-Only Arguments,Special parameters
@anchor{tutorial/controlflow function-examples}@anchor{1511}
@subsubsection Function Examples


Consider the following example function definitions paying close attention to the
markers @code{/} and @code{*}:

@example
>>> def standard_arg(arg):
...     print(arg)
...
>>> def pos_only_arg(arg, /):
...     print(arg)
...
>>> def kwd_only_arg(*, arg):
...     print(arg)
...
>>> def combined_example(pos_only, /, standard, *, kwd_only):
...     print(pos_only, standard, kwd_only)
@end example

The first function definition, @code{standard_arg}, the most familiar form,
places no restrictions on the calling convention and arguments may be
passed by position or keyword:

@example
>>> standard_arg(2)
2

>>> standard_arg(arg=2)
2
@end example

The second function @code{pos_only_arg} is restricted to only use positional
parameters as there is a @code{/} in the function definition:

@example
>>> pos_only_arg(1)
1

>>> pos_only_arg(arg=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: pos_only_arg() got some positional-only arguments passed as keyword arguments: 'arg'
@end example

The third function @code{kwd_only_args} only allows keyword arguments as indicated
by a @code{*} in the function definition:

@example
>>> kwd_only_arg(3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: kwd_only_arg() takes 0 positional arguments but 1 was given

>>> kwd_only_arg(arg=3)
3
@end example

And the last uses all three calling conventions in the same function
definition:

@example
>>> combined_example(1, 2, 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: combined_example() takes 2 positional arguments but 3 were given

>>> combined_example(1, 2, kwd_only=3)
1 2 3

>>> combined_example(1, standard=2, kwd_only=3)
1 2 3

>>> combined_example(pos_only=1, standard=2, kwd_only=3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: combined_example() got some positional-only arguments passed as keyword arguments: 'pos_only'
@end example

Finally, consider this function definition which has a potential collision between the positional argument @code{name}  and @code{**kwds} which has @code{name} as a key:

@example
def foo(name, **kwds):
    return 'name' in kwds
@end example

There is no possible call that will make it return @code{True} as the keyword @code{'name'}
will always bind to the first parameter. For example:

@example
>>> foo(1, **@{'name': 2@})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: foo() got multiple values for argument 'name'
>>>
@end example

But using @code{/} (positional only arguments), it is possible since it allows @code{name} as a positional argument and @code{'name'} as a key in the keyword arguments:

@example
def foo(name, /, **kwds):
    return 'name' in kwds
>>> foo(1, **@{'name': 2@})
True
@end example

In other words, the names of positional-only parameters can be used in
@code{**kwds} without ambiguity.

@node Recap,,Function Examples,Special parameters
@anchor{tutorial/controlflow recap}@anchor{1512}
@subsubsection Recap


The use case will determine which parameters to use in the function definition:

@example
def f(pos1, pos2, /, pos_or_kwd, *, kwd1, kwd2):
@end example

As guidance:


@itemize *

@item 
Use positional-only if you want the name of the parameters to not be
available to the user. This is useful when parameter names have no real
meaning, if you want to enforce the order of the arguments when the function
is called or if you need to take some positional parameters and arbitrary
keywords.

@item 
Use keyword-only when names have meaning and the function definition is
more understandable by being explicit with names or you want to prevent
users relying on the position of the argument being passed.

@item 
For an API, use positional-only to prevent breaking API changes
if the parameter’s name is modified in the future.
@end itemize

@node Arbitrary Argument Lists,Unpacking Argument Lists,Special parameters,More on Defining Functions
@anchor{tutorial/controlflow arbitrary-argument-lists}@anchor{1513}@anchor{tutorial/controlflow tut-arbitraryargs}@anchor{1514}
@subsubsection Arbitrary Argument Lists


@geindex * (asterisk); in function calls

Finally, the least frequently used option is to specify that a function can be
called with an arbitrary number of arguments.  These arguments will be wrapped
up in a tuple (see @ref{150c,,Tuples and Sequences}).  Before the variable number of arguments,
zero or more normal arguments may occur.

@example
def write_multiple_items(file, separator, *args):
    file.write(separator.join(args))
@end example

Normally, these @code{variadic} arguments will be last in the list of formal
parameters, because they scoop up all remaining input arguments that are
passed to the function. Any formal parameters which occur after the @code{*args}
parameter are ‘keyword-only’ arguments, meaning that they can only be used as
keywords rather than positional arguments.

@example
>>> def concat(*args, sep="/"):
...     return sep.join(args)
...
>>> concat("earth", "mars", "venus")
'earth/mars/venus'
>>> concat("earth", "mars", "venus", sep=".")
'earth.mars.venus'
@end example

@node Unpacking Argument Lists,Lambda Expressions,Arbitrary Argument Lists,More on Defining Functions
@anchor{tutorial/controlflow tut-unpacking-arguments}@anchor{1515}@anchor{tutorial/controlflow unpacking-argument-lists}@anchor{1516}
@subsubsection Unpacking Argument Lists


The reverse situation occurs when the arguments are already in a list or tuple
but need to be unpacked for a function call requiring separate positional
arguments.  For instance, the built-in @ref{438,,range()} function expects separate
@emph{start} and @emph{stop} arguments.  If they are not available separately, write the
function call with the  @code{*}-operator to unpack the arguments out of a list
or tuple:

@example
>>> list(range(3, 6))            # normal call with separate arguments
[3, 4, 5]
>>> args = [3, 6]
>>> list(range(*args))            # call with arguments unpacked from a list
[3, 4, 5]
@end example

@geindex **; in function calls

In the same fashion, dictionaries can deliver keyword arguments with the
@code{**}-operator:

@example
>>> def parrot(voltage, state='a stiff', action='voom'):
...     print("-- This parrot wouldn't", action, end=' ')
...     print("if you put", voltage, "volts through it.", end=' ')
...     print("E's", state, "!")
...
>>> d = @{"voltage": "four million", "state": "bleedin' demised", "action": "VOOM"@}
>>> parrot(**d)
-- This parrot wouldn't VOOM if you put four million volts through it. E's bleedin' demised !
@end example

@node Lambda Expressions,Documentation Strings,Unpacking Argument Lists,More on Defining Functions
@anchor{tutorial/controlflow lambda-expressions}@anchor{1517}@anchor{tutorial/controlflow tut-lambda}@anchor{1518}
@subsubsection Lambda Expressions


Small anonymous functions can be created with the @ref{e42,,lambda} keyword.
This function returns the sum of its two arguments: @code{lambda a, b: a+b}.
Lambda functions can be used wherever function objects are required.  They are
syntactically restricted to a single expression.  Semantically, they are just
syntactic sugar for a normal function definition.  Like nested function
definitions, lambda functions can reference variables from the containing
scope:

@example
>>> def make_incrementor(n):
...     return lambda x: x + n
...
>>> f = make_incrementor(42)
>>> f(0)
42
>>> f(1)
43
@end example

The above example uses a lambda expression to return a function.  Another use
is to pass a small function as an argument:

@example
>>> pairs = [(1, 'one'), (2, 'two'), (3, 'three'), (4, 'four')]
>>> pairs.sort(key=lambda pair: pair[1])
>>> pairs
[(4, 'four'), (1, 'one'), (3, 'three'), (2, 'two')]
@end example

@node Documentation Strings,Function Annotations,Lambda Expressions,More on Defining Functions
@anchor{tutorial/controlflow documentation-strings}@anchor{1519}@anchor{tutorial/controlflow tut-docstrings}@anchor{1503}
@subsubsection Documentation Strings


@geindex docstrings
@geindex documentation strings
@geindex strings@comma{} documentation

Here are some conventions about the content and formatting of documentation
strings.

The first line should always be a short, concise summary of the object’s
purpose.  For brevity, it should not explicitly state the object’s name or type,
since these are available by other means (except if the name happens to be a
verb describing a function’s operation).  This line should begin with a capital
letter and end with a period.

If there are more lines in the documentation string, the second line should be
blank, visually separating the summary from the rest of the description.  The
following lines should be one or more paragraphs describing the object’s calling
conventions, its side effects, etc.

The Python parser does not strip indentation from multi-line string literals in
Python, so tools that process documentation have to strip indentation if
desired.  This is done using the following convention. The first non-blank line
@emph{after} the first line of the string determines the amount of indentation for
the entire documentation string.  (We can’t use the first line since it is
generally adjacent to the string’s opening quotes so its indentation is not
apparent in the string literal.)  Whitespace “equivalent” to this indentation is
then stripped from the start of all lines of the string.  Lines that are
indented less should not occur, but if they occur all their leading whitespace
should be stripped.  Equivalence of whitespace should be tested after expansion
of tabs (to 8 spaces, normally).

Here is an example of a multi-line docstring:

@example
>>> def my_function():
...     """Do nothing, but document it.
...
...     No, really, it doesn't do anything.
...     """
...     pass
...
>>> print(my_function.__doc__)
Do nothing, but document it.

    No, really, it doesn't do anything.
@end example

@node Function Annotations,,Documentation Strings,More on Defining Functions
@anchor{tutorial/controlflow function-annotations}@anchor{151a}@anchor{tutorial/controlflow tut-annotations}@anchor{151b}
@subsubsection Function Annotations


@geindex function; annotations
@geindex ->; function annotations
@geindex ; (colon); function annotations

@ref{151c,,Function annotations} are completely optional metadata
information about the types used by user-defined functions (see 
@geindex Python Enhancement Proposals; PEP 3107
PEP 3107@footnote{https://www.python.org/dev/peps/pep-3107} and
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} for more information).

@ref{151d,,Annotations} are stored in the @code{__annotations__}
attribute of the function as a dictionary and have no effect on any other part of the
function.  Parameter annotations are defined by a colon after the parameter name, followed
by an expression evaluating to the value of the annotation.  Return annotations are
defined by a literal @code{->}, followed by an expression, between the parameter
list and the colon denoting the end of the @ref{fb8,,def} statement.  The
following example has a required argument, an optional argument, and the return
value annotated:

@example
>>> def f(ham: str, eggs: str = 'eggs') -> str:
...     print("Annotations:", f.__annotations__)
...     print("Arguments:", ham, eggs)
...     return ham + ' and ' + eggs
...
>>> f('spam')
Annotations: @{'ham': <class 'str'>, 'return': <class 'str'>, 'eggs': <class 'str'>@}
Arguments: spam eggs
'spam and eggs'
@end example

@node Intermezzo Coding Style,,More on Defining Functions,More Control Flow Tools
@anchor{tutorial/controlflow intermezzo-coding-style}@anchor{151e}@anchor{tutorial/controlflow tut-codingstyle}@anchor{151f}
@subsection Intermezzo: Coding Style


@geindex coding; style

Now that you are about to write longer, more complex pieces of Python, it is a
good time to talk about @emph{coding style}.  Most languages can be written (or more
concise, @emph{formatted}) in different styles; some are more readable than others.
Making it easy for others to read your code is always a good idea, and adopting
a nice coding style helps tremendously for that.

For Python, 
@geindex Python Enhancement Proposals; PEP 8
PEP 8@footnote{https://www.python.org/dev/peps/pep-0008} has emerged as the style guide that most projects adhere to;
it promotes a very readable and eye-pleasing coding style.  Every Python
developer should read it at some point; here are the most important points
extracted for you:


@itemize *

@item 
Use 4-space indentation, and no tabs.

4 spaces are a good compromise between small indentation (allows greater
nesting depth) and large indentation (easier to read).  Tabs introduce
confusion, and are best left out.

@item 
Wrap lines so that they don’t exceed 79 characters.

This helps users with small displays and makes it possible to have several
code files side-by-side on larger displays.

@item 
Use blank lines to separate functions and classes, and larger blocks of
code inside functions.

@item 
When possible, put comments on a line of their own.

@item 
Use docstrings.

@item 
Use spaces around operators and after commas, but not directly inside
bracketing constructs: @code{a = f(1, 2) + g(3, 4)}.

@item 
Name your classes and functions consistently; the convention is to use
@code{UpperCamelCase} for classes and @code{lowercase_with_underscores} for functions
and methods.  Always use @code{self} as the name for the first method argument
(see @ref{1520,,A First Look at Classes} for more on classes and methods).

@item 
Don’t use fancy encodings if your code is meant to be used in international
environments.  Python’s default, UTF-8, or even plain ASCII work best in any
case.

@item 
Likewise, don’t use non-ASCII characters in identifiers if there is only the
slightest chance people speaking a different language will read or maintain
the code.
@end itemize

@node Data Structures,Modules,More Control Flow Tools,The Python Tutorial
@anchor{tutorial/datastructures doc}@anchor{1521}@anchor{tutorial/datastructures data-structures}@anchor{1522}@anchor{tutorial/datastructures tut-structures}@anchor{14f9}
@section Data Structures


This chapter describes some things you’ve learned about already in more detail,
and adds some new things as well.

@menu
* More on Lists:: 
* The del statement:: 
* Tuples and Sequences:: 
* Sets:: 
* Dictionaries:: 
* Looping Techniques:: 
* More on Conditions:: 
* Comparing Sequences and Other Types:: 

@end menu

@node More on Lists,The del statement,,Data Structures
@anchor{tutorial/datastructures more-on-lists}@anchor{1523}@anchor{tutorial/datastructures tut-morelists}@anchor{1524}
@subsection More on Lists


The list data type has some more methods.  Here are all of the methods of list
objects:


@deffn {Method} list.append (x)

Add an item to the end of the list.  Equivalent to @code{a[len(a):] = [x]}.
@end deffn


@deffn {Method} list.extend (iterable)

Extend the list by appending all the items from the iterable.  Equivalent to
@code{a[len(a):] = iterable}.
@end deffn


@deffn {Method} list.insert (i, x)

Insert an item at a given position.  The first argument is the index of the
element before which to insert, so @code{a.insert(0, x)} inserts at the front of
the list, and @code{a.insert(len(a), x)} is equivalent to @code{a.append(x)}.
@end deffn


@deffn {Method} list.remove (x)

Remove the first item from the list whose value is equal to @emph{x}.  It raises a
@ref{17c,,ValueError} if there is no such item.
@end deffn


@deffn {Method} list.pop ([i])

Remove the item at the given position in the list, and return it.  If no index
is specified, @code{a.pop()} removes and returns the last item in the list.  (The
square brackets around the @emph{i} in the method signature denote that the parameter
is optional, not that you should type square brackets at that position.  You
will see this notation frequently in the Python Library Reference.)
@end deffn


@deffn {Method} list.clear ()

Remove all items from the list.  Equivalent to @code{del a[:]}.
@end deffn


@deffn {Method} list.index (x[, start[, end]])

Return zero-based index in the list of the first item whose value is equal to @emph{x}.
Raises a @ref{17c,,ValueError} if there is no such item.

The optional arguments @emph{start} and @emph{end} are interpreted as in the slice
notation and are used to limit the search to a particular subsequence of
the list.  The returned index is computed relative to the beginning of the full
sequence rather than the @emph{start} argument.
@end deffn


@deffn {Method} list.count (x)

Return the number of times @emph{x} appears in the list.
@end deffn


@deffn {Method} list.sort (*, key=None, reverse=False)

Sort the items of the list in place (the arguments can be used for sort
customization, see @ref{704,,sorted()} for their explanation).
@end deffn


@deffn {Method} list.reverse ()

Reverse the elements of the list in place.
@end deffn


@deffn {Method} list.copy ()

Return a shallow copy of the list.  Equivalent to @code{a[:]}.
@end deffn

An example that uses most of the list methods:

@example
>>> fruits = ['orange', 'apple', 'pear', 'banana', 'kiwi', 'apple', 'banana']
>>> fruits.count('apple')
2
>>> fruits.count('tangerine')
0
>>> fruits.index('banana')
3
>>> fruits.index('banana', 4)  # Find next banana starting a position 4
6
>>> fruits.reverse()
>>> fruits
['banana', 'apple', 'kiwi', 'banana', 'pear', 'apple', 'orange']
>>> fruits.append('grape')
>>> fruits
['banana', 'apple', 'kiwi', 'banana', 'pear', 'apple', 'orange', 'grape']
>>> fruits.sort()
>>> fruits
['apple', 'apple', 'banana', 'banana', 'grape', 'kiwi', 'orange', 'pear']
>>> fruits.pop()
'pear'
@end example

You might have noticed that methods like @code{insert}, @code{remove} or @code{sort} that
only modify the list have no return value printed – they return the default
@code{None}. @footnote{@w{(1)} 
Other languages may return the mutated object, which allows method
chaining, such as @code{d->insert("a")->remove("b")->sort();}.
}  This is a design principle for all mutable data structures in
Python.

Another thing you might notice is that not all data can be sorted or
compared.  For instance, @code{[None, 'hello', 10]} doesn’t sort because
integers can’t be compared to strings and @emph{None} can’t be compared to
other types.  Also, there are some types that don’t have a defined
ordering relation.  For example, @code{3+4j < 5+7j} isn’t a valid
comparison.

@menu
* Using Lists as Stacks:: 
* Using Lists as Queues:: 
* List Comprehensions: List Comprehensions<2>. 
* Nested List Comprehensions:: 

@end menu

@node Using Lists as Stacks,Using Lists as Queues,,More on Lists
@anchor{tutorial/datastructures tut-lists-as-stacks}@anchor{1525}@anchor{tutorial/datastructures using-lists-as-stacks}@anchor{1526}
@subsubsection Using Lists as Stacks


The list methods make it very easy to use a list as a stack, where the last
element added is the first element retrieved (“last-in, first-out”).  To add an
item to the top of the stack, use @code{append()}.  To retrieve an item from the
top of the stack, use @code{pop()} without an explicit index.  For example:

@example
>>> stack = [3, 4, 5]
>>> stack.append(6)
>>> stack.append(7)
>>> stack
[3, 4, 5, 6, 7]
>>> stack.pop()
7
>>> stack
[3, 4, 5, 6]
>>> stack.pop()
6
>>> stack.pop()
5
>>> stack
[3, 4]
@end example

@node Using Lists as Queues,List Comprehensions<2>,Using Lists as Stacks,More on Lists
@anchor{tutorial/datastructures tut-lists-as-queues}@anchor{1527}@anchor{tutorial/datastructures using-lists-as-queues}@anchor{1528}
@subsubsection Using Lists as Queues


It is also possible to use a list as a queue, where the first element added is
the first element retrieved (“first-in, first-out”); however, lists are not
efficient for this purpose.  While appends and pops from the end of list are
fast, doing inserts or pops from the beginning of a list is slow (because all
of the other elements have to be shifted by one).

To implement a queue, use @ref{7d3,,collections.deque} which was designed to
have fast appends and pops from both ends.  For example:

@example
>>> from collections import deque
>>> queue = deque(["Eric", "John", "Michael"])
>>> queue.append("Terry")           # Terry arrives
>>> queue.append("Graham")          # Graham arrives
>>> queue.popleft()                 # The first to arrive now leaves
'Eric'
>>> queue.popleft()                 # The second to arrive now leaves
'John'
>>> queue                           # Remaining queue in order of arrival
deque(['Michael', 'Terry', 'Graham'])
@end example

@node List Comprehensions<2>,Nested List Comprehensions,Using Lists as Queues,More on Lists
@anchor{tutorial/datastructures list-comprehensions}@anchor{1529}@anchor{tutorial/datastructures tut-listcomps}@anchor{152a}
@subsubsection List Comprehensions


List comprehensions provide a concise way to create lists.
Common applications are to make new lists where each element is the result of
some operations applied to each member of another sequence or iterable, or to
create a subsequence of those elements that satisfy a certain condition.

For example, assume we want to create a list of squares, like:

@example
>>> squares = []
>>> for x in range(10):
...     squares.append(x**2)
...
>>> squares
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
@end example

Note that this creates (or overwrites) a variable named @code{x} that still exists
after the loop completes.  We can calculate the list of squares without any
side effects using:

@example
squares = list(map(lambda x: x**2, range(10)))
@end example

or, equivalently:

@example
squares = [x**2 for x in range(10)]
@end example

which is more concise and readable.

A list comprehension consists of brackets containing an expression followed
by a @code{for} clause, then zero or more @code{for} or @code{if}
clauses.  The result will be a new list resulting from evaluating the expression
in the context of the @code{for} and @code{if} clauses which follow it.
For example, this listcomp combines the elements of two lists if they are not
equal:

@example
>>> [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]
[(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)]
@end example

and it’s equivalent to:

@example
>>> combs = []
>>> for x in [1,2,3]:
...     for y in [3,1,4]:
...         if x != y:
...             combs.append((x, y))
...
>>> combs
[(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)]
@end example

Note how the order of the @ref{161,,for} and @ref{2fb,,if} statements is the
same in both these snippets.

If the expression is a tuple (e.g. the @code{(x, y)} in the previous example),
it must be parenthesized.

@example
>>> vec = [-4, -2, 0, 2, 4]
>>> # create a new list with the values doubled
>>> [x*2 for x in vec]
[-8, -4, 0, 4, 8]
>>> # filter the list to exclude negative numbers
>>> [x for x in vec if x >= 0]
[0, 2, 4]
>>> # apply a function to all the elements
>>> [abs(x) for x in vec]
[4, 2, 0, 2, 4]
>>> # call a method on each element
>>> freshfruit = ['  banana', '  loganberry ', 'passion fruit  ']
>>> [weapon.strip() for weapon in freshfruit]
['banana', 'loganberry', 'passion fruit']
>>> # create a list of 2-tuples like (number, square)
>>> [(x, x**2) for x in range(6)]
[(0, 0), (1, 1), (2, 4), (3, 9), (4, 16), (5, 25)]
>>> # the tuple must be parenthesized, otherwise an error is raised
>>> [x, x**2 for x in range(6)]
  File "<stdin>", line 1, in <module>
    [x, x**2 for x in range(6)]
               ^
SyntaxError: invalid syntax
>>> # flatten a list using a listcomp with two 'for'
>>> vec = [[1,2,3], [4,5,6], [7,8,9]]
>>> [num for elem in vec for num in elem]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
@end example

List comprehensions can contain complex expressions and nested functions:

@example
>>> from math import pi
>>> [str(round(pi, i)) for i in range(1, 6)]
['3.1', '3.14', '3.142', '3.1416', '3.14159']
@end example

@node Nested List Comprehensions,,List Comprehensions<2>,More on Lists
@anchor{tutorial/datastructures nested-list-comprehensions}@anchor{152b}
@subsubsection Nested List Comprehensions


The initial expression in a list comprehension can be any arbitrary expression,
including another list comprehension.

Consider the following example of a 3x4 matrix implemented as a list of
3 lists of length 4:

@example
>>> matrix = [
...     [1, 2, 3, 4],
...     [5, 6, 7, 8],
...     [9, 10, 11, 12],
... ]
@end example

The following list comprehension will transpose rows and columns:

@example
>>> [[row[i] for row in matrix] for i in range(4)]
[[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]]
@end example

As we saw in the previous section, the nested listcomp is evaluated in
the context of the @ref{161,,for} that follows it, so this example is
equivalent to:

@example
>>> transposed = []
>>> for i in range(4):
...     transposed.append([row[i] for row in matrix])
...
>>> transposed
[[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]]
@end example

which, in turn, is the same as:

@example
>>> transposed = []
>>> for i in range(4):
...     # the following 3 lines implement the nested listcomp
...     transposed_row = []
...     for row in matrix:
...         transposed_row.append(row[i])
...     transposed.append(transposed_row)
...
>>> transposed
[[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]]
@end example

In the real world, you should prefer built-in functions to complex flow statements.
The @ref{247,,zip()} function would do a great job for this use case:

@example
>>> list(zip(*matrix))
[(1, 5, 9), (2, 6, 10), (3, 7, 11), (4, 8, 12)]
@end example

See @ref{1515,,Unpacking Argument Lists} for details on the asterisk in this line.

@node The del statement,Tuples and Sequences,More on Lists,Data Structures
@anchor{tutorial/datastructures the-del-statement}@anchor{152c}@anchor{tutorial/datastructures tut-del}@anchor{152d}
@subsection The @code{del} statement


There is a way to remove an item from a list given its index instead of its
value: the @ref{152e,,del} statement.  This differs from the @code{pop()} method
which returns a value.  The @code{del} statement can also be used to remove
slices from a list or clear the entire list (which we did earlier by assignment
of an empty list to the slice).  For example:

@example
>>> a = [-1, 1, 66.25, 333, 333, 1234.5]
>>> del a[0]
>>> a
[1, 66.25, 333, 333, 1234.5]
>>> del a[2:4]
>>> a
[1, 66.25, 1234.5]
>>> del a[:]
>>> a
[]
@end example

@ref{152e,,del} can also be used to delete entire variables:

@example
>>> del a
@end example

Referencing the name @code{a} hereafter is an error (at least until another value
is assigned to it).  We’ll find other uses for @ref{152e,,del} later.

@node Tuples and Sequences,Sets,The del statement,Data Structures
@anchor{tutorial/datastructures tuples-and-sequences}@anchor{152f}@anchor{tutorial/datastructures tut-tuples}@anchor{150c}
@subsection Tuples and Sequences


We saw that lists and strings have many common properties, such as indexing and
slicing operations.  They are two examples of @emph{sequence} data types (see
@ref{1530,,Sequence Types — list@comma{} tuple@comma{} range}).  Since Python is an evolving language, other sequence data
types may be added.  There is also another standard sequence data type: the
@emph{tuple}.

A tuple consists of a number of values separated by commas, for instance:

@example
>>> t = 12345, 54321, 'hello!'
>>> t[0]
12345
>>> t
(12345, 54321, 'hello!')
>>> # Tuples may be nested:
... u = t, (1, 2, 3, 4, 5)
>>> u
((12345, 54321, 'hello!'), (1, 2, 3, 4, 5))
>>> # Tuples are immutable:
... t[0] = 88888
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> # but they can contain mutable objects:
... v = ([1, 2, 3], [3, 2, 1])
>>> v
([1, 2, 3], [3, 2, 1])
@end example

As you see, on output tuples are always enclosed in parentheses, so that nested
tuples are interpreted correctly; they may be input with or without surrounding
parentheses, although often parentheses are necessary anyway (if the tuple is
part of a larger expression).  It is not possible to assign to the individual
items of a tuple, however it is possible to create tuples which contain mutable
objects, such as lists.

Though tuples may seem similar to lists, they are often used in different
situations and for different purposes.
Tuples are @ref{14e1,,immutable}, and usually contain a heterogeneous sequence of
elements that are accessed via unpacking (see later in this section) or indexing
(or even by attribute in the case of @ref{3e4,,namedtuples}).
Lists are @ref{14e9,,mutable}, and their elements are usually homogeneous and are
accessed by iterating over the list.

A special problem is the construction of tuples containing 0 or 1 items: the
syntax has some extra quirks to accommodate these.  Empty tuples are constructed
by an empty pair of parentheses; a tuple with one item is constructed by
following a value with a comma (it is not sufficient to enclose a single value
in parentheses). Ugly, but effective.  For example:

@example
>>> empty = ()
>>> singleton = 'hello',    # <-- note trailing comma
>>> len(empty)
0
>>> len(singleton)
1
>>> singleton
('hello',)
@end example

The statement @code{t = 12345, 54321, 'hello!'} is an example of @emph{tuple packing}:
the values @code{12345}, @code{54321} and @code{'hello!'} are packed together in a tuple.
The reverse operation is also possible:

@example
>>> x, y, z = t
@end example

This is called, appropriately enough, @emph{sequence unpacking} and works for any
sequence on the right-hand side.  Sequence unpacking requires that there are as
many variables on the left side of the equals sign as there are elements in the
sequence.  Note that multiple assignment is really just a combination of tuple
packing and sequence unpacking.

@node Sets,Dictionaries,Tuples and Sequences,Data Structures
@anchor{tutorial/datastructures sets}@anchor{1531}@anchor{tutorial/datastructures tut-sets}@anchor{1532}
@subsection Sets


Python also includes a data type for @emph{sets}.  A set is an unordered collection
with no duplicate elements.  Basic uses include membership testing and
eliminating duplicate entries.  Set objects also support mathematical operations
like union, intersection, difference, and symmetric difference.

Curly braces or the @ref{370,,set()} function can be used to create sets.  Note: to
create an empty set you have to use @code{set()}, not @code{@{@}}; the latter creates an
empty dictionary, a data structure that we discuss in the next section.

Here is a brief demonstration:

@example
>>> basket = @{'apple', 'orange', 'apple', 'pear', 'orange', 'banana'@}
>>> print(basket)                      # show that duplicates have been removed
@{'orange', 'banana', 'pear', 'apple'@}
>>> 'orange' in basket                 # fast membership testing
True
>>> 'crabgrass' in basket
False

>>> # Demonstrate set operations on unique letters from two words
...
>>> a = set('abracadabra')
>>> b = set('alacazam')
>>> a                                  # unique letters in a
@{'a', 'r', 'b', 'c', 'd'@}
>>> a - b                              # letters in a but not in b
@{'r', 'd', 'b'@}
>>> a | b                              # letters in a or b or both
@{'a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'@}
>>> a & b                              # letters in both a and b
@{'a', 'c'@}
>>> a ^ b                              # letters in a or b but not both
@{'r', 'd', 'b', 'm', 'z', 'l'@}
@end example

Similarly to @ref{152a,,list comprehensions}, set comprehensions
are also supported:

@example
>>> a = @{x for x in 'abracadabra' if x not in 'abc'@}
>>> a
@{'r', 'd'@}
@end example

@node Dictionaries,Looping Techniques,Sets,Data Structures
@anchor{tutorial/datastructures dictionaries}@anchor{1533}@anchor{tutorial/datastructures tut-dictionaries}@anchor{1534}
@subsection Dictionaries


Another useful data type built into Python is the @emph{dictionary} (see
@ref{5de,,Mapping Types — dict}). Dictionaries are sometimes found in other languages as
“associative memories” or “associative arrays”.  Unlike sequences, which are
indexed by a range of numbers, dictionaries are indexed by @emph{keys}, which can be
any immutable type; strings and numbers can always be keys.  Tuples can be used
as keys if they contain only strings, numbers, or tuples; if a tuple contains
any mutable object either directly or indirectly, it cannot be used as a key.
You can’t use lists as keys, since lists can be modified in place using index
assignments, slice assignments, or methods like @code{append()} and
@code{extend()}.

It is best to think of a dictionary as a set of @emph{key: value} pairs,
with the requirement that the keys are unique (within one dictionary). A pair of
braces creates an empty dictionary: @code{@{@}}. Placing a comma-separated list of
key:value pairs within the braces adds initial key:value pairs to the
dictionary; this is also the way dictionaries are written on output.

The main operations on a dictionary are storing a value with some key and
extracting the value given the key.  It is also possible to delete a key:value
pair with @code{del}. If you store using a key that is already in use, the old
value associated with that key is forgotten.  It is an error to extract a value
using a non-existent key.

Performing @code{list(d)} on a dictionary returns a list of all the keys
used in the dictionary, in insertion order (if you want it sorted, just use
@code{sorted(d)} instead). To check whether a single key is in the
dictionary, use the @ref{2fc,,in} keyword.

Here is a small example using a dictionary:

@example
>>> tel = @{'jack': 4098, 'sape': 4139@}
>>> tel['guido'] = 4127
>>> tel
@{'jack': 4098, 'sape': 4139, 'guido': 4127@}
>>> tel['jack']
4098
>>> del tel['sape']
>>> tel['irv'] = 4127
>>> tel
@{'jack': 4098, 'guido': 4127, 'irv': 4127@}
>>> list(tel)
['jack', 'guido', 'irv']
>>> sorted(tel)
['guido', 'irv', 'jack']
>>> 'guido' in tel
True
>>> 'jack' not in tel
False
@end example

The @ref{3b0,,dict()} constructor builds dictionaries directly from sequences of
key-value pairs:

@example
>>> dict([('sape', 4139), ('guido', 4127), ('jack', 4098)])
@{'sape': 4139, 'guido': 4127, 'jack': 4098@}
@end example

In addition, dict comprehensions can be used to create dictionaries from
arbitrary key and value expressions:

@example
>>> @{x: x**2 for x in (2, 4, 6)@}
@{2: 4, 4: 16, 6: 36@}
@end example

When the keys are simple strings, it is sometimes easier to specify pairs using
keyword arguments:

@example
>>> dict(sape=4139, guido=4127, jack=4098)
@{'sape': 4139, 'guido': 4127, 'jack': 4098@}
@end example

@node Looping Techniques,More on Conditions,Dictionaries,Data Structures
@anchor{tutorial/datastructures looping-techniques}@anchor{1535}@anchor{tutorial/datastructures tut-loopidioms}@anchor{14f8}
@subsection Looping Techniques


When looping through dictionaries, the key and corresponding value can be
retrieved at the same time using the @code{items()} method.

@example
>>> knights = @{'gallahad': 'the pure', 'robin': 'the brave'@}
>>> for k, v in knights.items():
...     print(k, v)
...
gallahad the pure
robin the brave
@end example

When looping through a sequence, the position index and corresponding value can
be retrieved at the same time using the @ref{fdc,,enumerate()} function.

@example
>>> for i, v in enumerate(['tic', 'tac', 'toe']):
...     print(i, v)
...
0 tic
1 tac
2 toe
@end example

To loop over two or more sequences at the same time, the entries can be paired
with the @ref{247,,zip()} function.

@example
>>> questions = ['name', 'quest', 'favorite color']
>>> answers = ['lancelot', 'the holy grail', 'blue']
>>> for q, a in zip(questions, answers):
...     print('What is your @{0@}?  It is @{1@}.'.format(q, a))
...
What is your name?  It is lancelot.
What is your quest?  It is the holy grail.
What is your favorite color?  It is blue.
@end example

To loop over a sequence in reverse, first specify the sequence in a forward
direction and then call the @ref{2f6,,reversed()} function.

@example
>>> for i in reversed(range(1, 10, 2)):
...     print(i)
...
9
7
5
3
1
@end example

To loop over a sequence in sorted order, use the @ref{704,,sorted()} function which
returns a new sorted list while leaving the source unaltered.

@example
>>> basket = ['apple', 'orange', 'apple', 'pear', 'orange', 'banana']
>>> for i in sorted(basket):
...     print(i)
...
apple
apple
banana
orange
orange
pear
@end example

Using @ref{370,,set()} on a sequence eliminates duplicate elements. The use of
@ref{704,,sorted()} in combination with @ref{370,,set()} over a sequence is an idiomatic
way to loop over unique elements of the sequence in sorted order.

@example
>>> basket = ['apple', 'orange', 'apple', 'pear', 'orange', 'banana']
>>> for f in sorted(set(basket)):
...     print(f)
...
apple
banana
orange
pear
@end example

It is sometimes tempting to change a list while you are looping over it;
however, it is often simpler and safer to create a new list instead.

@example
>>> import math
>>> raw_data = [56.2, float('NaN'), 51.7, 55.3, 52.5, float('NaN'), 47.8]
>>> filtered_data = []
>>> for value in raw_data:
...     if not math.isnan(value):
...         filtered_data.append(value)
...
>>> filtered_data
[56.2, 51.7, 55.3, 52.5, 47.8]
@end example

@node More on Conditions,Comparing Sequences and Other Types,Looping Techniques,Data Structures
@anchor{tutorial/datastructures more-on-conditions}@anchor{1536}@anchor{tutorial/datastructures tut-conditions}@anchor{1537}
@subsection More on Conditions


The conditions used in @code{while} and @code{if} statements can contain any
operators, not just comparisons.

The comparison operators @code{in} and @code{not in} are membership tests that
determine whether a value is in (or not in) a container.  The operators @code{is}
and @code{is not} compare whether two objects are really the same object.  All
comparison operators have the same priority, which is lower than that of all
numerical operators.

Comparisons can be chained.  For example, @code{a < b == c} tests whether @code{a} is
less than @code{b} and moreover @code{b} equals @code{c}.

Comparisons may be combined using the Boolean operators @code{and} and @code{or}, and
the outcome of a comparison (or of any other Boolean expression) may be negated
with @code{not}.  These have lower priorities than comparison operators; between
them, @code{not} has the highest priority and @code{or} the lowest, so that @code{A and
not B or C} is equivalent to @code{(A and (not B)) or C}. As always, parentheses
can be used to express the desired composition.

The Boolean operators @code{and} and @code{or} are so-called @emph{short-circuit}
operators: their arguments are evaluated from left to right, and evaluation
stops as soon as the outcome is determined.  For example, if @code{A} and @code{C} are
true but @code{B} is false, @code{A and B and C} does not evaluate the expression
@code{C}.  When used as a general value and not as a Boolean, the return value of a
short-circuit operator is the last evaluated argument.

It is possible to assign the result of a comparison or other Boolean expression
to a variable.  For example,

@example
>>> string1, string2, string3 = '@w{'}, 'Trondheim', 'Hammer Dance'
>>> non_null = string1 or string2 or string3
>>> non_null
'Trondheim'
@end example

Note that in Python, unlike C, assignment inside expressions must be done
explicitly with the
@ref{1538,,walrus operator} @code{:=}.
This avoids a common class of problems encountered in C programs: typing @code{=}
in an expression when @code{==} was intended.

@node Comparing Sequences and Other Types,,More on Conditions,Data Structures
@anchor{tutorial/datastructures comparing-sequences-and-other-types}@anchor{1539}@anchor{tutorial/datastructures tut-comparing}@anchor{153a}
@subsection Comparing Sequences and Other Types


Sequence objects typically may be compared to other objects with the same sequence
type. The comparison uses @emph{lexicographical} ordering: first the first two
items are compared, and if they differ this determines the outcome of the
comparison; if they are equal, the next two items are compared, and so on, until
either sequence is exhausted. If two items to be compared are themselves
sequences of the same type, the lexicographical comparison is carried out
recursively.  If all items of two sequences compare equal, the sequences are
considered equal. If one sequence is an initial sub-sequence of the other, the
shorter sequence is the smaller (lesser) one.  Lexicographical ordering for
strings uses the Unicode code point number to order individual characters.
Some examples of comparisons between sequences of the same type:

@example
(1, 2, 3)              < (1, 2, 4)
[1, 2, 3]              < [1, 2, 4]
'ABC' < 'C' < 'Pascal' < 'Python'
(1, 2, 3, 4)           < (1, 2, 4)
(1, 2)                 < (1, 2, -1)
(1, 2, 3)             == (1.0, 2.0, 3.0)
(1, 2, ('aa', 'ab'))   < (1, 2, ('abc', 'a'), 4)
@end example

Note that comparing objects of different types with @code{<} or @code{>} is legal
provided that the objects have appropriate comparison methods.  For example,
mixed numeric types are compared according to their numeric value, so 0 equals
0.0, etc.  Otherwise, rather than providing an arbitrary ordering, the
interpreter will raise a @ref{162,,TypeError} exception.

@node Modules,Input and Output,Data Structures,The Python Tutorial
@anchor{tutorial/modules doc}@anchor{153b}@anchor{tutorial/modules modules}@anchor{153c}@anchor{tutorial/modules tut-modules}@anchor{153d}
@section Modules


If you quit from the Python interpreter and enter it again, the definitions you
have made (functions and variables) are lost. Therefore, if you want to write a
somewhat longer program, you are better off using a text editor to prepare the
input for the interpreter and running it with that file as input instead.  This
is known as creating a @emph{script}.  As your program gets longer, you may want to
split it into several files for easier maintenance.  You may also want to use a
handy function that you’ve written in several programs without copying its
definition into each program.

To support this, Python has a way to put definitions in a file and use them in a
script or in an interactive instance of the interpreter. Such a file is called a
@emph{module}; definitions from a module can be @emph{imported} into other modules or into
the @emph{main} module (the collection of variables that you have access to in a
script executed at the top level and in calculator mode).

A module is a file containing Python definitions and statements.  The file name
is the module name with the suffix @code{.py} appended.  Within a module, the
module’s name (as a string) is available as the value of the global variable
@code{__name__}.  For instance, use your favorite text editor to create a file
called @code{fibo.py} in the current directory with the following contents:

@example
# Fibonacci numbers module

def fib(n):    # write Fibonacci series up to n
    a, b = 0, 1
    while a < n:
        print(a, end=' ')
        a, b = b, a+b
    print()

def fib2(n):   # return Fibonacci series up to n
    result = []
    a, b = 0, 1
    while a < n:
        result.append(a)
        a, b = b, a+b
    return result
@end example

Now enter the Python interpreter and import this module with the following
command:

@example
>>> import fibo
@end example

This does not enter the names of the functions defined in @code{fibo}  directly in
the current symbol table; it only enters the module name @code{fibo} there. Using
the module name you can access the functions:

@example
>>> fibo.fib(1000)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987
>>> fibo.fib2(100)
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]
>>> fibo.__name__
'fibo'
@end example

If you intend to use a function often you can assign it to a local name:

@example
>>> fib = fibo.fib
>>> fib(500)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
@end example

@menu
* More on Modules:: 
* Standard Modules:: 
* The dir() Function: The dir Function. 
* Packages:: 

@end menu

@node More on Modules,Standard Modules,,Modules
@anchor{tutorial/modules more-on-modules}@anchor{153e}@anchor{tutorial/modules tut-moremodules}@anchor{153f}
@subsection More on Modules


A module can contain executable statements as well as function definitions.
These statements are intended to initialize the module. They are executed only
the @emph{first} time the module name is encountered in an import statement. @footnote{@w{(1)} 
In fact function definitions are also ‘statements’ that are ‘executed’; the
execution of a module-level function definition enters the function name in
the module’s global symbol table.
}
(They are also run if the file is executed as a script.)

Each module has its own private symbol table, which is used as the global symbol
table by all functions defined in the module. Thus, the author of a module can
use global variables in the module without worrying about accidental clashes
with a user’s global variables. On the other hand, if you know what you are
doing you can touch a module’s global variables with the same notation used to
refer to its functions, @code{modname.itemname}.

Modules can import other modules.  It is customary but not required to place all
@ref{e35,,import} statements at the beginning of a module (or script, for that
matter).  The imported module names are placed in the importing module’s global
symbol table.

There is a variant of the @ref{e35,,import} statement that imports names from a
module directly into the importing module’s symbol table.  For example:

@example
>>> from fibo import fib, fib2
>>> fib(500)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
@end example

This does not introduce the module name from which the imports are taken in the
local symbol table (so in the example, @code{fibo} is not defined).

There is even a variant to import all names that a module defines:

@example
>>> from fibo import *
>>> fib(500)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
@end example

This imports all names except those beginning with an underscore (@code{_}).
In most cases Python programmers do not use this facility since it introduces
an unknown set of names into the interpreter, possibly hiding some things
you have already defined.

Note that in general the practice of importing @code{*} from a module or package is
frowned upon, since it often causes poorly readable code. However, it is okay to
use it to save typing in interactive sessions.

If the module name is followed by @code{as}, then the name
following @code{as} is bound directly to the imported module.

@example
>>> import fibo as fib
>>> fib.fib(500)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
@end example

This is effectively importing the module in the same way that @code{import fibo}
will do, with the only difference of it being available as @code{fib}.

It can also be used when utilising @ref{e51,,from} with similar effects:

@example
>>> from fibo import fib as fibonacci
>>> fibonacci(500)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
@end example

@cartouche
@quotation Note 
For efficiency reasons, each module is only imported once per interpreter
session.  Therefore, if you change your modules, you must restart the
interpreter – or, if it’s just one module you want to test interactively,
use @ref{66a,,importlib.reload()}, e.g. @code{import importlib;
importlib.reload(modulename)}.
@end quotation
@end cartouche

@menu
* Executing modules as scripts:: 
* The Module Search Path:: 
* “Compiled” Python files:: 

@end menu

@node Executing modules as scripts,The Module Search Path,,More on Modules
@anchor{tutorial/modules executing-modules-as-scripts}@anchor{1540}@anchor{tutorial/modules tut-modulesasscripts}@anchor{1541}
@subsubsection Executing modules as scripts


When you run a Python module with

@example
python fibo.py <arguments>
@end example

the code in the module will be executed, just as if you imported it, but with
the @code{__name__} set to @code{"__main__"}.  That means that by adding this code at
the end of your module:

@example
if __name__ == "__main__":
    import sys
    fib(int(sys.argv[1]))
@end example

you can make the file usable as a script as well as an importable module,
because the code that parses the command line only runs if the module is
executed as the “main” file:

@example
$ python fibo.py 50
0 1 1 2 3 5 8 13 21 34
@end example

If the module is imported, the code is not run:

@example
>>> import fibo
>>>
@end example

This is often used either to provide a convenient user interface to a module, or
for testing purposes (running the module as a script executes a test suite).

@node The Module Search Path,“Compiled” Python files,Executing modules as scripts,More on Modules
@anchor{tutorial/modules the-module-search-path}@anchor{1542}@anchor{tutorial/modules tut-searchpath}@anchor{1543}
@subsubsection The Module Search Path


@geindex module; search; path

When a module named @code{spam} is imported, the interpreter first searches for
a built-in module with that name. If not found, it then searches for a file
named @code{spam.py} in a list of directories given by the variable
@ref{489,,sys.path}.  @ref{489,,sys.path} is initialized from these locations:


@itemize *

@item 
The directory containing the input script (or the current directory when no
file is specified).

@item 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} (a list of directory names, with the same syntax as the
shell variable 
@geindex PATH
@geindex environment variable; PATH
@code{PATH}).

@item 
The installation-dependent default (by convention including a
@code{site-packages} directory, handled by the @ref{e6,,site} module).
@end itemize

@cartouche
@quotation Note 
On file systems which support symlinks, the directory containing the input
script is calculated after the symlink is followed. In other words the
directory containing the symlink is @strong{not} added to the module search path.
@end quotation
@end cartouche

After initialization, Python programs can modify @ref{489,,sys.path}.  The
directory containing the script being run is placed at the beginning of the
search path, ahead of the standard library path. This means that scripts in that
directory will be loaded instead of modules of the same name in the library
directory. This is an error unless the replacement is intended.  See section
@ref{1544,,Standard Modules} for more information.

@c %
@c Do we need stuff on zip files etc. ? DUBOIS

@node “Compiled” Python files,,The Module Search Path,More on Modules
@anchor{tutorial/modules compiled-python-files}@anchor{1545}
@subsubsection “Compiled” Python files


To speed up loading modules, Python caches the compiled version of each module
in the @code{__pycache__} directory under the name @code{module.@emph{version}.pyc},
where the version encodes the format of the compiled file; it generally contains
the Python version number.  For example, in CPython release 3.3 the compiled
version of spam.py would be cached as @code{__pycache__/spam.cpython-33.pyc}.  This
naming convention allows compiled modules from different releases and different
versions of Python to coexist.

Python checks the modification date of the source against the compiled version
to see if it’s out of date and needs to be recompiled.  This is a completely
automatic process.  Also, the compiled modules are platform-independent, so the
same library can be shared among systems with different architectures.

Python does not check the cache in two circumstances.  First, it always
recompiles and does not store the result for the module that’s loaded directly
from the command line.  Second, it does not check the cache if there is no
source module.  To support a non-source (compiled only) distribution, the
compiled module must be in the source directory, and there must not be a source
module.

Some tips for experts:


@itemize *

@item 
You can use the @ref{912,,-O} or @ref{913,,-OO} switches on the Python command
to reduce the size of a compiled module.  The @code{-O} switch removes assert
statements, the @code{-OO} switch removes both assert statements and __doc__
strings.  Since some programs may rely on having these available, you should
only use this option if you know what you’re doing.  “Optimized” modules have
an @code{opt-} tag and are usually smaller.  Future releases may
change the effects of optimization.

@item 
A program doesn’t run any faster when it is read from a @code{.pyc}
file than when it is read from a @code{.py} file; the only thing that’s faster
about @code{.pyc} files is the speed with which they are loaded.

@item 
The module @ref{1f,,compileall} can create .pyc files for all modules in a
directory.

@item 
There is more detail on this process, including a flow chart of the
decisions, in 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147}.
@end itemize

@node Standard Modules,The dir Function,More on Modules,Modules
@anchor{tutorial/modules standard-modules}@anchor{1546}@anchor{tutorial/modules tut-standardmodules}@anchor{1544}
@subsection Standard Modules


@geindex module; sys

Python comes with a library of standard modules, described in a separate
document, the Python Library Reference (“Library Reference” hereafter).  Some
modules are built into the interpreter; these provide access to operations that
are not part of the core of the language but are nevertheless built in, either
for efficiency or to provide access to operating system primitives such as
system calls.  The set of such modules is a configuration option which also
depends on the underlying platform.  For example, the @ref{131,,winreg} module is only
provided on Windows systems. One particular module deserves some attention:
@ref{f7,,sys}, which is built into every Python interpreter.  The variables
@code{sys.ps1} and @code{sys.ps2} define the strings used as primary and secondary
prompts:

@example
>>> import sys
>>> sys.ps1
'>>> '
>>> sys.ps2
'... '
>>> sys.ps1 = 'C> '
C> print('Yuck!')
Yuck!
C>
@end example

These two variables are only defined if the interpreter is in interactive mode.

The variable @code{sys.path} is a list of strings that determines the interpreter’s
search path for modules. It is initialized to a default path taken from the
environment variable 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH}, or from a built-in default if
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} is not set.  You can modify it using standard list
operations:

@example
>>> import sys
>>> sys.path.append('/ufs/guido/lib/python')
@end example

@node The dir Function,Packages,Standard Modules,Modules
@anchor{tutorial/modules the-dir-function}@anchor{1547}@anchor{tutorial/modules tut-dir}@anchor{1548}
@subsection The @code{dir()} Function


The built-in function @ref{f2f,,dir()} is used to find out which names a module
defines.  It returns a sorted list of strings:

@example
>>> import fibo, sys
>>> dir(fibo)
['__name__', 'fib', 'fib2']
>>> dir(sys)  
['__breakpointhook__', '__displayhook__', '__doc__', '__excepthook__',
 '__interactivehook__', '__loader__', '__name__', '__package__', '__spec__',
 '__stderr__', '__stdin__', '__stdout__', '__unraisablehook__',
 '_clear_type_cache', '_current_frames', '_debugmallocstats', '_framework',
 '_getframe', '_git', '_home', '_xoptions', 'abiflags', 'addaudithook',
 'api_version', 'argv', 'audit', 'base_exec_prefix', 'base_prefix',
 'breakpointhook', 'builtin_module_names', 'byteorder', 'call_tracing',
 'callstats', 'copyright', 'displayhook', 'dont_write_bytecode', 'exc_info',
 'excepthook', 'exec_prefix', 'executable', 'exit', 'flags', 'float_info',
 'float_repr_style', 'get_asyncgen_hooks', 'get_coroutine_origin_tracking_depth',
 'getallocatedblocks', 'getdefaultencoding', 'getdlopenflags',
 'getfilesystemencodeerrors', 'getfilesystemencoding', 'getprofile',
 'getrecursionlimit', 'getrefcount', 'getsizeof', 'getswitchinterval',
 'gettrace', 'hash_info', 'hexversion', 'implementation', 'int_info',
 'intern', 'is_finalizing', 'last_traceback', 'last_type', 'last_value',
 'maxsize', 'maxunicode', 'meta_path', 'modules', 'path', 'path_hooks',
 'path_importer_cache', 'platform', 'prefix', 'ps1', 'ps2', 'pycache_prefix',
 'set_asyncgen_hooks', 'set_coroutine_origin_tracking_depth', 'setdlopenflags',
 'setprofile', 'setrecursionlimit', 'setswitchinterval', 'settrace', 'stderr',
 'stdin', 'stdout', 'thread_info', 'unraisablehook', 'version', 'version_info',
 'warnoptions']
@end example

Without arguments, @ref{f2f,,dir()} lists the names you have defined currently:

@example
>>> a = [1, 2, 3, 4, 5]
>>> import fibo
>>> fib = fibo.fib
>>> dir()
['__builtins__', '__name__', 'a', 'fib', 'fibo', 'sys']
@end example

Note that it lists all types of names: variables, modules, functions, etc.

@geindex module; builtins

@ref{f2f,,dir()} does not list the names of built-in functions and variables.  If you
want a list of those, they are defined in the standard module
@ref{11,,builtins}:

@example
>>> import builtins
>>> dir(builtins)  
['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException',
 'BlockingIOError', 'BrokenPipeError', 'BufferError', 'BytesWarning',
 'ChildProcessError', 'ConnectionAbortedError', 'ConnectionError',
 'ConnectionRefusedError', 'ConnectionResetError', 'DeprecationWarning',
 'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False',
 'FileExistsError', 'FileNotFoundError', 'FloatingPointError',
 'FutureWarning', 'GeneratorExit', 'IOError', 'ImportError',
 'ImportWarning', 'IndentationError', 'IndexError', 'InterruptedError',
 'IsADirectoryError', 'KeyError', 'KeyboardInterrupt', 'LookupError',
 'MemoryError', 'NameError', 'None', 'NotADirectoryError', 'NotImplemented',
 'NotImplementedError', 'OSError', 'OverflowError',
 'PendingDeprecationWarning', 'PermissionError', 'ProcessLookupError',
 'ReferenceError', 'ResourceWarning', 'RuntimeError', 'RuntimeWarning',
 'StopIteration', 'SyntaxError', 'SyntaxWarning', 'SystemError',
 'SystemExit', 'TabError', 'TimeoutError', 'True', 'TypeError',
 'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError',
 'UnicodeError', 'UnicodeTranslateError', 'UnicodeWarning', 'UserWarning',
 'ValueError', 'Warning', 'ZeroDivisionError', '_', '__build_class__',
 '__debug__', '__doc__', '__import__', '__name__', '__package__', 'abs',
 'all', 'any', 'ascii', 'bin', 'bool', 'bytearray', 'bytes', 'callable',
 'chr', 'classmethod', 'compile', 'complex', 'copyright', 'credits',
 'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'exec', 'exit',
 'filter', 'float', 'format', 'frozenset', 'getattr', 'globals', 'hasattr',
 'hash', 'help', 'hex', 'id', 'input', 'int', 'isinstance', 'issubclass',
 'iter', 'len', 'license', 'list', 'locals', 'map', 'max', 'memoryview',
 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property',
 'quit', 'range', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice',
 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'vars',
 'zip']
@end example

@node Packages,,The dir Function,Modules
@anchor{tutorial/modules packages}@anchor{1549}@anchor{tutorial/modules tut-packages}@anchor{154a}
@subsection Packages


Packages are a way of structuring Python’s module namespace by using “dotted
module names”.  For example, the module name @code{A.B} designates a submodule
named @code{B} in a package named @code{A}.  Just like the use of modules saves the
authors of different modules from having to worry about each other’s global
variable names, the use of dotted module names saves the authors of multi-module
packages like NumPy or Pillow from having to worry about
each other’s module names.

Suppose you want to design a collection of modules (a “package”) for the uniform
handling of sound files and sound data.  There are many different sound file
formats (usually recognized by their extension, for example: @code{.wav},
@code{.aiff}, @code{.au}), so you may need to create and maintain a growing
collection of modules for the conversion between the various file formats.
There are also many different operations you might want to perform on sound data
(such as mixing, adding echo, applying an equalizer function, creating an
artificial stereo effect), so in addition you will be writing a never-ending
stream of modules to perform these operations.  Here’s a possible structure for
your package (expressed in terms of a hierarchical filesystem):

@example
sound/                          Top-level package
      __init__.py               Initialize the sound package
      formats/                  Subpackage for file format conversions
              __init__.py
              wavread.py
              wavwrite.py
              aiffread.py
              aiffwrite.py
              auread.py
              auwrite.py
              ...
      effects/                  Subpackage for sound effects
              __init__.py
              echo.py
              surround.py
              reverse.py
              ...
      filters/                  Subpackage for filters
              __init__.py
              equalizer.py
              vocoder.py
              karaoke.py
              ...
@end example

When importing the package, Python searches through the directories on
@code{sys.path} looking for the package subdirectory.

The @code{__init__.py} files are required to make Python treat directories
containing the file as packages.  This prevents directories with a common name,
such as @code{string}, unintentionally hiding valid modules that occur later
on the module search path. In the simplest case, @code{__init__.py} can just be
an empty file, but it can also execute initialization code for the package or
set the @code{__all__} variable, described later.

Users of the package can import individual modules from the package, for
example:

@example
import sound.effects.echo
@end example

This loads the submodule @code{sound.effects.echo}.  It must be referenced with
its full name.

@example
sound.effects.echo.echofilter(input, output, delay=0.7, atten=4)
@end example

An alternative way of importing the submodule is:

@example
from sound.effects import echo
@end example

This also loads the submodule @code{echo}, and makes it available without its
package prefix, so it can be used as follows:

@example
echo.echofilter(input, output, delay=0.7, atten=4)
@end example

Yet another variation is to import the desired function or variable directly:

@example
from sound.effects.echo import echofilter
@end example

Again, this loads the submodule @code{echo}, but this makes its function
@code{echofilter()} directly available:

@example
echofilter(input, output, delay=0.7, atten=4)
@end example

Note that when using @code{from package import item}, the item can be either a
submodule (or subpackage) of the package, or some  other name defined in the
package, like a function, class or variable.  The @code{import} statement first
tests whether the item is defined in the package; if not, it assumes it is a
module and attempts to load it.  If it fails to find it, an @ref{3b8,,ImportError}
exception is raised.

Contrarily, when using syntax like @code{import item.subitem.subsubitem}, each item
except for the last must be a package; the last item can be a module or a
package but can’t be a class or function or variable defined in the previous
item.

@menu
* Importing * From a Package:: 
* Intra-package References:: 
* Packages in Multiple Directories:: 

@end menu

@node Importing * From a Package,Intra-package References,,Packages
@anchor{tutorial/modules importing-from-a-package}@anchor{154b}@anchor{tutorial/modules tut-pkg-import-star}@anchor{154c}
@subsubsection Importing * From a Package


@geindex __all__

Now what happens when the user writes @code{from sound.effects import *}?  Ideally,
one would hope that this somehow goes out to the filesystem, finds which
submodules are present in the package, and imports them all.  This could take a
long time and importing sub-modules might have unwanted side-effects that should
only happen when the sub-module is explicitly imported.

The only solution is for the package author to provide an explicit index of the
package.  The @ref{e35,,import} statement uses the following convention: if a package’s
@code{__init__.py} code defines a list named @code{__all__}, it is taken to be the
list of module names that should be imported when @code{from package import *} is
encountered.  It is up to the package author to keep this list up-to-date when a
new version of the package is released.  Package authors may also decide not to
support it, if they don’t see a use for importing * from their package.  For
example, the file @code{sound/effects/__init__.py} could contain the following
code:

@example
__all__ = ["echo", "surround", "reverse"]
@end example

This would mean that @code{from sound.effects import *} would import the three
named submodules of the @code{sound} package.

If @code{__all__} is not defined, the statement @code{from sound.effects import *}
does @emph{not} import all submodules from the package @code{sound.effects} into the
current namespace; it only ensures that the package @code{sound.effects} has
been imported (possibly running any initialization code in @code{__init__.py})
and then imports whatever names are defined in the package.  This includes any
names defined (and submodules explicitly loaded) by @code{__init__.py}.  It
also includes any submodules of the package that were explicitly loaded by
previous @ref{e35,,import} statements.  Consider this code:

@example
import sound.effects.echo
import sound.effects.surround
from sound.effects import *
@end example

In this example, the @code{echo} and @code{surround} modules are imported in the
current namespace because they are defined in the @code{sound.effects} package
when the @code{from...import} statement is executed.  (This also works when
@code{__all__} is defined.)

Although certain modules are designed to export only names that follow certain
patterns when you use @code{import *}, it is still considered bad practice in
production code.

Remember, there is nothing wrong with using @code{from package import
specific_submodule}!  In fact, this is the recommended notation unless the
importing module needs to use submodules with the same name from different
packages.

@node Intra-package References,Packages in Multiple Directories,Importing * From a Package,Packages
@anchor{tutorial/modules id2}@anchor{154d}@anchor{tutorial/modules intra-package-references}@anchor{154e}
@subsubsection Intra-package References


When packages are structured into subpackages (as with the @code{sound} package
in the example), you can use absolute imports to refer to submodules of siblings
packages.  For example, if the module @code{sound.filters.vocoder} needs to use
the @code{echo} module in the @code{sound.effects} package, it can use @code{from
sound.effects import echo}.

You can also write relative imports, with the @code{from module import name} form
of import statement.  These imports use leading dots to indicate the current and
parent packages involved in the relative import.  From the @code{surround}
module for example, you might use:

@example
from . import echo
from .. import formats
from ..filters import equalizer
@end example

Note that relative imports are based on the name of the current module.  Since
the name of the main module is always @code{"__main__"}, modules intended for use
as the main module of a Python application must always use absolute imports.

@node Packages in Multiple Directories,,Intra-package References,Packages
@anchor{tutorial/modules packages-in-multiple-directories}@anchor{154f}
@subsubsection Packages in Multiple Directories


Packages support one more special attribute, @ref{1550,,__path__}.  This is
initialized to be a list containing the name of the directory holding the
package’s @code{__init__.py} before the code in that file is executed.  This
variable can be modified; doing so affects future searches for modules and
subpackages contained in the package.

While this feature is not often needed, it can be used to extend the set of
modules found in a package.

@node Input and Output,Errors and Exceptions,Modules,The Python Tutorial
@anchor{tutorial/inputoutput doc}@anchor{1551}@anchor{tutorial/inputoutput input-and-output}@anchor{1552}@anchor{tutorial/inputoutput tut-io}@anchor{1553}
@section Input and Output


There are several ways to present the output of a program; data can be printed
in a human-readable form, or written to a file for future use. This chapter will
discuss some of the possibilities.

@menu
* Fancier Output Formatting:: 
* Reading and Writing Files:: 

@end menu

@node Fancier Output Formatting,Reading and Writing Files,,Input and Output
@anchor{tutorial/inputoutput fancier-output-formatting}@anchor{1554}@anchor{tutorial/inputoutput tut-formatting}@anchor{1555}
@subsection Fancier Output Formatting


So far we’ve encountered two ways of writing values: @emph{expression statements} and
the @ref{af6,,print()} function.  (A third way is using the @code{write()} method
of file objects; the standard output file can be referenced as @code{sys.stdout}.
See the Library Reference for more information on this.)

Often you’ll want more control over the formatting of your output than simply
printing space-separated values. There are several ways to format output.


@itemize *

@item 
To use @ref{1556,,formatted string literals}, begin a string
with @code{f} or @code{F} before the opening quotation mark or triple quotation mark.
Inside this string, you can write a Python expression between @code{@{} and @code{@}}
characters that can refer to variables or literal values.

@example
>>> year = 2016
>>> event = 'Referendum'
>>> f'Results of the @{year@} @{event@}'
'Results of the 2016 Referendum'
@end example

@item 
The @ref{784,,str.format()} method of strings requires more manual
effort.  You’ll still use @code{@{} and @code{@}} to mark where a variable
will be substituted and can provide detailed formatting directives,
but you’ll also need to provide the information to be formatted.

@example
>>> yes_votes = 42_572_654
>>> no_votes = 43_132_495
>>> percentage = yes_votes / (yes_votes + no_votes)
>>> '@{:-9@} YES votes  @{:2.2%@}'.format(yes_votes, percentage)
' 42572654 YES votes  49.67%'
@end example

@item 
Finally, you can do all the string handling yourself by using string slicing and
concatenation operations to create any layout you can imagine.  The
string type has some methods that perform useful operations for padding
strings to a given column width.
@end itemize

When you don’t need fancy output but just want a quick display of some
variables for debugging purposes, you can convert any value to a string with
the @ref{27d,,repr()} or @ref{2ef,,str()} functions.

The @ref{2ef,,str()} function is meant to return representations of values which are
fairly human-readable, while @ref{27d,,repr()} is meant to generate representations
which can be read by the interpreter (or will force a @ref{21a,,SyntaxError} if
there is no equivalent syntax).  For objects which don’t have a particular
representation for human consumption, @ref{2ef,,str()} will return the same value as
@ref{27d,,repr()}.  Many values, such as numbers or structures like lists and
dictionaries, have the same representation using either function.  Strings, in
particular, have two distinct representations.

Some examples:

@example
>>> s = 'Hello, world.'
>>> str(s)
'Hello, world.'
>>> repr(s)
"'Hello, world.'"
>>> str(1/7)
'0.14285714285714285'
>>> x = 10 * 3.25
>>> y = 200 * 200
>>> s = 'The value of x is ' + repr(x) + ', and y is ' + repr(y) + '...'
>>> print(s)
The value of x is 32.5, and y is 40000...
>>> # The repr() of a string adds string quotes and backslashes:
... hello = 'hello, world\n'
>>> hellos = repr(hello)
>>> print(hellos)
'hello, world\n'
>>> # The argument to repr() may be any Python object:
... repr((x, y, ('spam', 'eggs')))
"(32.5, 40000, ('spam', 'eggs'))"
@end example

The @ref{f1,,string} module contains a @ref{6b5,,Template} class that offers
yet another way to substitute values into strings, using placeholders like
@code{$x} and replacing them with values from a dictionary, but offers much less
control of the formatting.

@menu
* Formatted String Literals:: 
* The String format() Method: The String format Method. 
* Manual String Formatting:: 
* Old string formatting:: 

@end menu

@node Formatted String Literals,The String format Method,,Fancier Output Formatting
@anchor{tutorial/inputoutput formatted-string-literals}@anchor{1557}@anchor{tutorial/inputoutput tut-f-strings}@anchor{1556}
@subsubsection Formatted String Literals


@ref{4b0,,Formatted string literals} (also called f-strings for
short) let you include the value of Python expressions inside a string by
prefixing the string with @code{f} or @code{F} and writing expressions as
@code{@{expression@}}.

An optional format specifier can follow the expression. This allows greater
control over how the value is formatted. The following example rounds pi to
three places after the decimal:

@example
>>> import math
>>> print(f'The value of pi is approximately @{math.pi:.3f@}.')
The value of pi is approximately 3.142.
@end example

Passing an integer after the @code{':'} will cause that field to be a minimum
number of characters wide.  This is useful for making columns line up.

@example
>>> table = @{'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 7678@}
>>> for name, phone in table.items():
...     print(f'@{name:10@} ==> @{phone:10d@}')
...
Sjoerd     ==>       4127
Jack       ==>       4098
Dcab       ==>       7678
@end example

Other modifiers can be used to convert the value before it is formatted.
@code{'!a'} applies @ref{f47,,ascii()}, @code{'!s'} applies @ref{2ef,,str()}, and @code{'!r'}
applies @ref{27d,,repr()}:

@example
>>> animals = 'eels'
>>> print(f'My hovercraft is full of @{animals@}.')
My hovercraft is full of eels.
>>> print(f'My hovercraft is full of @{animals!r@}.')
My hovercraft is full of 'eels'.
@end example

For a reference on these format specifications, see
the reference guide for the @ref{788,,Format Specification Mini-Language}.

@node The String format Method,Manual String Formatting,Formatted String Literals,Fancier Output Formatting
@anchor{tutorial/inputoutput the-string-format-method}@anchor{1558}@anchor{tutorial/inputoutput tut-string-format}@anchor{1559}
@subsubsection The String format() Method


Basic usage of the @ref{784,,str.format()} method looks like this:

@example
>>> print('We are the @{@} who say "@{@}!"'.format('knights', 'Ni'))
We are the knights who say "Ni!"
@end example

The brackets and characters within them (called format fields) are replaced with
the objects passed into the @ref{784,,str.format()} method.  A number in the
brackets can be used to refer to the position of the object passed into the
@ref{784,,str.format()} method.

@example
>>> print('@{0@} and @{1@}'.format('spam', 'eggs'))
spam and eggs
>>> print('@{1@} and @{0@}'.format('spam', 'eggs'))
eggs and spam
@end example

If keyword arguments are used in the @ref{784,,str.format()} method, their values
are referred to by using the name of the argument.

@example
>>> print('This @{food@} is @{adjective@}.'.format(
...       food='spam', adjective='absolutely horrible'))
This spam is absolutely horrible.
@end example

Positional and keyword arguments can be arbitrarily combined:

@example
>>> print('The story of @{0@}, @{1@}, and @{other@}.'.format('Bill', 'Manfred',
                                                       other='Georg'))
The story of Bill, Manfred, and Georg.
@end example

If you have a really long format string that you don’t want to split up, it
would be nice if you could reference the variables to be formatted by name
instead of by position.  This can be done by simply passing the dict and using
square brackets @code{'[]'} to access the keys.

@example
>>> table = @{'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 8637678@}
>>> print('Jack: @{0[Jack]:d@}; Sjoerd: @{0[Sjoerd]:d@}; '
...       'Dcab: @{0[Dcab]:d@}'.format(table))
Jack: 4098; Sjoerd: 4127; Dcab: 8637678
@end example

This could also be done by passing the table as keyword arguments with the ‘**’
notation.

@example
>>> table = @{'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 8637678@}
>>> print('Jack: @{Jack:d@}; Sjoerd: @{Sjoerd:d@}; Dcab: @{Dcab:d@}'.format(**table))
Jack: 4098; Sjoerd: 4127; Dcab: 8637678
@end example

This is particularly useful in combination with the built-in function
@ref{12fc,,vars()}, which returns a dictionary containing all local variables.

As an example, the following lines produce a tidily-aligned
set of columns giving integers and their squares and cubes:

@example
>>> for x in range(1, 11):
...     print('@{0:2d@} @{1:3d@} @{2:4d@}'.format(x, x*x, x*x*x))
...
 1   1    1
 2   4    8
 3   9   27
 4  16   64
 5  25  125
 6  36  216
 7  49  343
 8  64  512
 9  81  729
10 100 1000
@end example

For a complete overview of string formatting with @ref{784,,str.format()}, see
@ref{f1a,,Format String Syntax}.

@node Manual String Formatting,Old string formatting,The String format Method,Fancier Output Formatting
@anchor{tutorial/inputoutput manual-string-formatting}@anchor{155a}
@subsubsection Manual String Formatting


Here’s the same table of squares and cubes, formatted manually:

@example
>>> for x in range(1, 11):
...     print(repr(x).rjust(2), repr(x*x).rjust(3), end=' ')
...     # Note use of 'end' on previous line
...     print(repr(x*x*x).rjust(4))
...
 1   1    1
 2   4    8
 3   9   27
 4  16   64
 5  25  125
 6  36  216
 7  49  343
 8  64  512
 9  81  729
10 100 1000
@end example

(Note that the one space between each column was added by the
way @ref{af6,,print()} works: it always adds spaces between its arguments.)

The @ref{155b,,str.rjust()} method of string objects right-justifies a string in a
field of a given width by padding it with spaces on the left. There are
similar methods @ref{155c,,str.ljust()} and @ref{155d,,str.center()}. These methods do
not write anything, they just return a new string. If the input string is too
long, they don’t truncate it, but return it unchanged; this will mess up your
column lay-out but that’s usually better than the alternative, which would be
lying about a value. (If you really want truncation you can always add a
slice operation, as in @code{x.ljust(n)[:n]}.)

There is another method, @ref{155e,,str.zfill()}, which pads a numeric string on the
left with zeros.  It understands about plus and minus signs:

@example
>>> '12'.zfill(5)
'00012'
>>> '-3.14'.zfill(7)
'-003.14'
>>> '3.14159265359'.zfill(5)
'3.14159265359'
@end example

@node Old string formatting,,Manual String Formatting,Fancier Output Formatting
@anchor{tutorial/inputoutput old-string-formatting}@anchor{155f}
@subsubsection Old string formatting


The % operator (modulo) can also be used for string formatting. Given @code{'string'
% values}, instances of @code{%} in @code{string} are replaced with zero or more
elements of @code{values}. This operation is commonly known as string
interpolation. For example:

@example
>>> import math
>>> print('The value of pi is approximately %5.3f.' % math.pi)
The value of pi is approximately 3.142.
@end example

More information can be found in the @ref{14e4,,printf-style String Formatting} section.

@node Reading and Writing Files,,Fancier Output Formatting,Input and Output
@anchor{tutorial/inputoutput reading-and-writing-files}@anchor{1560}@anchor{tutorial/inputoutput tut-files}@anchor{1561}
@subsection Reading and Writing Files


@geindex built-in function; open
@geindex object; file

@ref{1c8,,open()} returns a @ref{d68,,file object}, and is most commonly used with
two arguments: @code{open(filename, mode)}.

@example
>>> f = open('workfile', 'w')
@end example

@c XXX str(f) is <io.TextIOWrapper object at 0x82e8dc4>
@c 
@c >>> print(f)
@c <open file 'workfile', mode 'w' at 80a0960>

The first argument is a string containing the filename.  The second argument is
another string containing a few characters describing the way in which the file
will be used.  @emph{mode} can be @code{'r'} when the file will only be read, @code{'w'}
for only writing (an existing file with the same name will be erased), and
@code{'a'} opens the file for appending; any data written to the file is
automatically added to the end.  @code{'r+'} opens the file for both reading and
writing. The @emph{mode} argument is optional; @code{'r'} will be assumed if it’s
omitted.

Normally, files are opened in @emph{text mode}, that means, you read and write
strings from and to the file, which are encoded in a specific encoding. If
encoding is not specified, the default is platform dependent (see
@ref{1c8,,open()}). @code{'b'} appended to the mode opens the file in
@emph{binary mode}: now the data is read and written in the form of bytes
objects.  This mode should be used for all files that don’t contain text.

In text mode, the default when reading is to convert platform-specific line
endings (@code{\n} on Unix, @code{\r\n} on Windows) to just @code{\n}.  When writing in
text mode, the default is to convert occurrences of @code{\n} back to
platform-specific line endings.  This behind-the-scenes modification
to file data is fine for text files, but will corrupt binary data like that in
@code{JPEG} or @code{EXE} files.  Be very careful to use binary mode when
reading and writing such files.

It is good practice to use the @ref{168,,with} keyword when dealing
with file objects.  The advantage is that the file is properly closed
after its suite finishes, even if an exception is raised at some
point.  Using @code{with} is also much shorter than writing
equivalent @ref{f6d,,try}-@ref{4c1,,finally} blocks:

@example
>>> with open('workfile') as f:
...     read_data = f.read()

>>> # We can check that the file has been automatically closed.
>>> f.closed
True
@end example

If you’re not using the @ref{168,,with} keyword, then you should call
@code{f.close()} to close the file and immediately free up any system
resources used by it.

@cartouche
@quotation Warning 
Calling @code{f.write()} without using the @code{with} keyword or calling
@code{f.close()} @strong{might} result in the arguments
of @code{f.write()} not being completely written to the disk, even if the
program exits successfully.
@end quotation
@end cartouche

@c See also https://bugs.python.org/issue17852

After a file object is closed, either by a @ref{168,,with} statement
or by calling @code{f.close()}, attempts to use the file object will
automatically fail.

@example
>>> f.close()
>>> f.read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: I/O operation on closed file.
@end example

@menu
* Methods of File Objects:: 
* Saving structured data with json:: 

@end menu

@node Methods of File Objects,Saving structured data with json,,Reading and Writing Files
@anchor{tutorial/inputoutput methods-of-file-objects}@anchor{1562}@anchor{tutorial/inputoutput tut-filemethods}@anchor{1563}
@subsubsection Methods of File Objects


The rest of the examples in this section will assume that a file object called
@code{f} has already been created.

To read a file’s contents, call @code{f.read(size)}, which reads some quantity of
data and returns it as a string (in text mode) or bytes object (in binary mode).
@emph{size} is an optional numeric argument.  When @emph{size} is omitted or negative, the
entire contents of the file will be read and returned; it’s your problem if the
file is twice as large as your machine’s memory. Otherwise, at most @emph{size}
characters (in text mode) or @emph{size} bytes (in binary mode) are read and returned.
If the end of the file has been reached, @code{f.read()} will return an empty
string (@code{'@w{'}}).

@example
>>> f.read()
'This is the entire file.\n'
>>> f.read()
'@w{'}
@end example

@code{f.readline()} reads a single line from the file; a newline character (@code{\n})
is left at the end of the string, and is only omitted on the last line of the
file if the file doesn’t end in a newline.  This makes the return value
unambiguous; if @code{f.readline()} returns an empty string, the end of the file
has been reached, while a blank line is represented by @code{'\n'}, a string
containing only a single newline.

@example
>>> f.readline()
'This is the first line of the file.\n'
>>> f.readline()
'Second line of the file\n'
>>> f.readline()
'@w{'}
@end example

For reading lines from a file, you can loop over the file object. This is memory
efficient, fast, and leads to simple code:

@example
>>> for line in f:
...     print(line, end='@w{'})
...
This is the first line of the file.
Second line of the file
@end example

If you want to read all the lines of a file in a list you can also use
@code{list(f)} or @code{f.readlines()}.

@code{f.write(string)} writes the contents of @emph{string} to the file, returning
the number of characters written.

@example
>>> f.write('This is a test\n')
15
@end example

Other types of objects need to be converted – either to a string (in text mode)
or a bytes object (in binary mode) – before writing them:

@example
>>> value = ('the answer', 42)
>>> s = str(value)  # convert the tuple to string
>>> f.write(s)
18
@end example

@code{f.tell()} returns an integer giving the file object’s current position in the file
represented as number of bytes from the beginning of the file when in binary mode and
an opaque number when in text mode.

To change the file object’s position, use @code{f.seek(offset, whence)}.  The position is computed
from adding @emph{offset} to a reference point; the reference point is selected by
the @emph{whence} argument.  A @emph{whence} value of 0 measures from the beginning
of the file, 1 uses the current file position, and 2 uses the end of the file as
the reference point.  @emph{whence} can be omitted and defaults to 0, using the
beginning of the file as the reference point.

@example
>>> f = open('workfile', 'rb+')
>>> f.write(b'0123456789abcdef')
16
>>> f.seek(5)      # Go to the 6th byte in the file
5
>>> f.read(1)
b'5'
>>> f.seek(-3, 2)  # Go to the 3rd byte before the end
13
>>> f.read(1)
b'd'
@end example

In text files (those opened without a @code{b} in the mode string), only seeks
relative to the beginning of the file are allowed (the exception being seeking
to the very file end with @code{seek(0, 2)}) and the only valid @emph{offset} values are
those returned from the @code{f.tell()}, or zero. Any other @emph{offset} value produces
undefined behaviour.

File objects have some additional methods, such as @code{isatty()} and
@code{truncate()} which are less frequently used; consult the Library
Reference for a complete guide to file objects.

@node Saving structured data with json,,Methods of File Objects,Reading and Writing Files
@anchor{tutorial/inputoutput saving-structured-data-with-json}@anchor{1564}@anchor{tutorial/inputoutput tut-json}@anchor{1565}
@subsubsection Saving structured data with @code{json}


@geindex module; json

Strings can easily be written to and read from a file.  Numbers take a bit more
effort, since the @code{read()} method only returns strings, which will have to
be passed to a function like @ref{1b1,,int()}, which takes a string like @code{'123'}
and returns its numeric value 123.  When you want to save more complex data
types like nested lists and dictionaries, parsing and serializing by hand
becomes complicated.

Rather than having users constantly writing and debugging code to save
complicated data types to files, Python allows you to use the popular data
interchange format called JSON (JavaScript Object Notation)@footnote{http://json.org}.  The standard module called @ref{a0,,json} can take Python
data hierarchies, and convert them to string representations; this process is
called @emph{serializing}.  Reconstructing the data from the string representation
is called @emph{deserializing}.  Between serializing and deserializing, the
string representing the object may have been stored in a file or data, or
sent over a network connection to some distant machine.

@cartouche
@quotation Note 
The JSON format is commonly used by modern applications to allow for data
exchange.  Many programmers are already familiar with it, which makes
it a good choice for interoperability.
@end quotation
@end cartouche

If you have an object @code{x}, you can view its JSON string representation with a
simple line of code:

@example
>>> import json
>>> x = [1, 'simple', 'list']
>>> json.dumps(x)
'[1, "simple", "list"]'
@end example

Another variant of the @ref{89b,,dumps()} function, called @ref{89a,,dump()},
simply serializes the object to a @ref{1566,,text file}.  So if @code{f} is a
@ref{1566,,text file} object opened for writing, we can do this:

@example
json.dump(x, f)
@end example

To decode the object again, if @code{f} is a @ref{1566,,text file} object which has
been opened for reading:

@example
x = json.load(f)
@end example

This simple serialization technique can handle lists and dictionaries, but
serializing arbitrary class instances in JSON requires a bit of extra effort.
The reference for the @ref{a0,,json} module contains an explanation of this.


@subsubheading See also


@ref{c5,,pickle} - the pickle module

Contrary to @ref{1565,,JSON}, @emph{pickle} is a protocol which allows
the serialization of arbitrarily complex Python objects.  As such, it is
specific to Python and cannot be used to communicate with applications
written in other languages.  It is also insecure by default:
deserializing pickle data coming from an untrusted source can execute
arbitrary code, if the data was crafted by a skilled attacker.


@node Errors and Exceptions,Classes,Input and Output,The Python Tutorial
@anchor{tutorial/errors doc}@anchor{1567}@anchor{tutorial/errors errors-and-exceptions}@anchor{1568}@anchor{tutorial/errors tut-errors}@anchor{1569}
@section Errors and Exceptions


Until now error messages haven’t been more than mentioned, but if you have tried
out the examples you have probably seen some.  There are (at least) two
distinguishable kinds of errors: @emph{syntax errors} and @emph{exceptions}.

@menu
* Syntax Errors:: 
* Exceptions:: 
* Handling Exceptions:: 
* Raising Exceptions:: 
* Exception Chaining:: 
* User-defined Exceptions:: 
* Defining Clean-up Actions:: 
* Predefined Clean-up Actions:: 
* Raising and Handling Multiple Unrelated Exceptions:: 

@end menu

@node Syntax Errors,Exceptions,,Errors and Exceptions
@anchor{tutorial/errors syntax-errors}@anchor{156a}@anchor{tutorial/errors tut-syntaxerrors}@anchor{156b}
@subsection Syntax Errors


Syntax errors, also known as parsing errors, are perhaps the most common kind of
complaint you get while you are still learning Python:

@example
>>> while True print('Hello world')
  File "<stdin>", line 1
    while True print('Hello world')
                   ^
SyntaxError: invalid syntax
@end example

The parser repeats the offending line and displays a little ‘arrow’ pointing at
the earliest point in the line where the error was detected.  The error is
caused by (or at least detected at) the token @emph{preceding} the arrow: in the
example, the error is detected at the function @ref{af6,,print()}, since a colon
(@code{':'}) is missing before it.  File name and line number are printed so you
know where to look in case the input came from a script.

@node Exceptions,Handling Exceptions,Syntax Errors,Errors and Exceptions
@anchor{tutorial/errors exceptions}@anchor{156c}@anchor{tutorial/errors tut-exceptions}@anchor{156d}
@subsection Exceptions


Even if a statement or expression is syntactically correct, it may cause an
error when an attempt is made to execute it. Errors detected during execution
are called @emph{exceptions} and are not unconditionally fatal: you will soon learn
how to handle them in Python programs.  Most exceptions are not handled by
programs, however, and result in error messages as shown here:

@example
>>> 10 * (1/0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero
>>> 4 + spam*3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'spam' is not defined
>>> '2' + 2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can only concatenate str (not "int") to str
@end example

The last line of the error message indicates what happened. Exceptions come in
different types, and the type is printed as part of the message: the types in
the example are @ref{43c,,ZeroDivisionError}, @ref{21f,,NameError} and @ref{162,,TypeError}.
The string printed as the exception type is the name of the built-in exception
that occurred.  This is true for all built-in exceptions, but need not be true
for user-defined exceptions (although it is a useful convention). Standard
exception names are built-in identifiers (not reserved keywords).

The rest of the line provides detail based on the type of exception and what
caused it.

The preceding part of the error message shows the context where the exception
occurred, in the form of a stack traceback. In general it contains a stack
traceback listing source lines; however, it will not display lines read from
standard input.

@ref{10e8,,Built-in Exceptions} lists the built-in exceptions and their meanings.

@node Handling Exceptions,Raising Exceptions,Exceptions,Errors and Exceptions
@anchor{tutorial/errors handling-exceptions}@anchor{156e}@anchor{tutorial/errors tut-handling}@anchor{14fc}
@subsection Handling Exceptions


It is possible to write programs that handle selected exceptions. Look at the
following example, which asks the user for input until a valid integer has been
entered, but allows the user to interrupt the program (using @code{Control-C} or
whatever the operating system supports); note that a user-generated interruption
is signalled by raising the @ref{4ca,,KeyboardInterrupt} exception.

@example
>>> while True:
...     try:
...         x = int(input("Please enter a number: "))
...         break
...     except ValueError:
...         print("Oops!  That was no valid number.  Try again...")
...
@end example

The @ref{f6d,,try} statement works as follows.


@itemize *

@item 
First, the @emph{try clause} (the statement(s) between the @ref{f6d,,try} and
@ref{16e,,except} keywords) is executed.

@item 
If no exception occurs, the @emph{except clause} is skipped and execution of the
@ref{f6d,,try} statement is finished.

@item 
If an exception occurs during execution of the @ref{f6d,,try} clause, the rest of the
clause is skipped.  Then, if its type matches the exception named after the
@ref{16e,,except} keyword, the @emph{except clause} is executed, and then execution
continues after the try/except block.

@item 
If an exception occurs which does not match the exception named in the @emph{except
clause}, it is passed on to outer @ref{f6d,,try} statements; if no handler is
found, it is an @emph{unhandled exception} and execution stops with a message as
shown above.
@end itemize

A @ref{f6d,,try} statement may have more than one @emph{except clause}, to specify
handlers for different exceptions.  At most one handler will be executed.
Handlers only handle exceptions that occur in the corresponding @emph{try clause},
not in other handlers of the same @code{try} statement.  An @emph{except clause}
may name multiple exceptions as a parenthesized tuple, for example:

@example
... except (RuntimeError, TypeError, NameError):
...     pass
@end example

A class in an @ref{16e,,except} clause is compatible with an exception if it is
the same class or a base class thereof (but not the other way around — an
@emph{except clause} listing a derived class is not compatible with a base class).
For example, the following code will print B, C, D in that order:

@example
class B(Exception):
    pass

class C(B):
    pass

class D(C):
    pass

for cls in [B, C, D]:
    try:
        raise cls()
    except D:
        print("D")
    except C:
        print("C")
    except B:
        print("B")
@end example

Note that if the @emph{except clauses} were reversed (with @code{except B} first), it
would have printed B, B, B — the first matching @emph{except clause} is triggered.

All exceptions inherit from @ref{15f,,BaseException}, and so it can be used to serve
as a wildcard. Use this with extreme caution, since it is easy to mask a real
programming error in this way!  It can also be used to print an error message and
then re-raise the exception (allowing a caller to handle the exception as well):

@example
import sys

try:
    f = open('myfile.txt')
    s = f.readline()
    i = int(s.strip())
except OSError as err:
    print("OS error: @{0@}".format(err))
except ValueError:
    print("Could not convert data to an integer.")
except BaseException as err:
    print(f"Unexpected @{err=@}, @{type(err)=@}")
    raise
@end example

Alternatively the last except clause may omit the exception name(s), however the exception
value must then be retrieved with @code{sys.exception()}.

The @ref{f6d,,try} … @ref{16e,,except} statement has an optional @emph{else
clause}, which, when present, must follow all @emph{except clauses}.  It is useful
for code that must be executed if the @emph{try clause} does not raise an exception.
For example:

@example
for arg in sys.argv[1:]:
    try:
        f = open(arg, 'r')
    except OSError:
        print('cannot open', arg)
    else:
        print(arg, 'has', len(f.readlines()), 'lines')
        f.close()
@end example

The use of the @code{else} clause is better than adding additional code to
the @ref{f6d,,try} clause because it avoids accidentally catching an exception
that wasn’t raised by the code being protected by the @code{try} …
@code{except} statement.

When an exception occurs, it may have an associated value, also known as the
exception’s @emph{argument}. The presence and type of the argument depend on the
exception type.

The @emph{except clause} may specify a variable after the exception name.  The
variable is bound to an exception instance with the arguments stored in
@code{instance.args}.  For convenience, the exception instance defines
@code{__str__()} so the arguments can be printed directly without having to
reference @code{.args}.  One may also instantiate an exception first before
raising it and add any attributes to it as desired.

@example
>>> try:
...     raise Exception('spam', 'eggs')
... except Exception as inst:
...     print(type(inst))    # the exception instance
...     print(inst.args)     # arguments stored in .args
...     print(inst)          # __str__ allows args to be printed directly,
...                          # but may be overridden in exception subclasses
...     x, y = inst.args     # unpack args
...     print('x =', x)
...     print('y =', y)
...
<class 'Exception'>
('spam', 'eggs')
('spam', 'eggs')
x = spam
y = eggs
@end example

If an exception has arguments, they are printed as the last part (‘detail’) of
the message for unhandled exceptions.

Exception handlers don’t just handle exceptions if they occur immediately in the
@emph{try clause}, but also if they occur inside functions that are called (even
indirectly) in the @emph{try clause}. For example:

@example
>>> def this_fails():
...     x = 1/0
...
>>> try:
...     this_fails()
... except ZeroDivisionError as err:
...     print('Handling run-time error:', err)
...
Handling run-time error: division by zero
@end example

@node Raising Exceptions,Exception Chaining,Handling Exceptions,Errors and Exceptions
@anchor{tutorial/errors raising-exceptions}@anchor{156f}@anchor{tutorial/errors tut-raising}@anchor{1570}
@subsection Raising Exceptions


The @ref{16d,,raise} statement allows the programmer to force a specified
exception to occur. For example:

@example
>>> raise NameError('HiThere')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: HiThere
@end example

The sole argument to @ref{16d,,raise} indicates the exception to be raised.
This must be either an exception instance or an exception class (a class that
derives from @ref{4d8,,Exception}).  If an exception class is passed, it will
be implicitly instantiated by calling its constructor with no arguments:

@example
raise ValueError  # shorthand for 'raise ValueError()'
@end example

If you need to determine whether an exception was raised but don’t intend to
handle it, a simpler form of the @ref{16d,,raise} statement allows you to
re-raise the exception:

@example
>>> try:
...     raise NameError('HiThere')
... except NameError:
...     print('An exception flew by!')
...     raise
...
An exception flew by!
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
NameError: HiThere
@end example

@node Exception Chaining,User-defined Exceptions,Raising Exceptions,Errors and Exceptions
@anchor{tutorial/errors exception-chaining}@anchor{1571}@anchor{tutorial/errors tut-exception-chaining}@anchor{1572}
@subsection Exception Chaining


The @ref{16d,,raise} statement allows an optional @ref{16d,,from} which enables
chaining exceptions. For example:

@example
# exc must be exception instance or None.
raise RuntimeError from exc
@end example

This can be useful when you are transforming exceptions. For example:

@example
>>> def func():
...     raise ConnectionError
...
>>> try:
...     func()
... except ConnectionError as exc:
...     raise RuntimeError('Failed to open database') from exc
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "<stdin>", line 2, in func
ConnectionError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
RuntimeError: Failed to open database
@end example

Exception chaining happens automatically when an exception is raised inside an
@ref{16e,,except} or @ref{4c1,,finally} section. This can be
disabled by using @code{from None} idiom:

@example
>>> try:
...     open('database.sqlite')
... except OSError:
...     raise RuntimeError from None
...
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
RuntimeError
@end example

For more information about chaining mechanics, see @ref{10e8,,Built-in Exceptions}.

@node User-defined Exceptions,Defining Clean-up Actions,Exception Chaining,Errors and Exceptions
@anchor{tutorial/errors tut-userexceptions}@anchor{1573}@anchor{tutorial/errors user-defined-exceptions}@anchor{1574}
@subsection User-defined Exceptions


Programs may name their own exceptions by creating a new exception class (see
@ref{1505,,Classes} for more about Python classes).  Exceptions should typically
be derived from the @ref{4d8,,Exception} class, either directly or indirectly.

Exception classes can be defined which do anything any other class can do, but
are usually kept simple, often only offering a number of attributes that allow
information about the error to be extracted by handlers for the exception.

Most exceptions are defined with names that end in “Error”, similar to the
naming of the standard exceptions.

Many standard modules define their own exceptions to report errors that may
occur in functions they define.  More information on classes is presented in
chapter @ref{1505,,Classes}.

@node Defining Clean-up Actions,Predefined Clean-up Actions,User-defined Exceptions,Errors and Exceptions
@anchor{tutorial/errors defining-clean-up-actions}@anchor{1575}@anchor{tutorial/errors tut-cleanup}@anchor{1576}
@subsection Defining Clean-up Actions


The @ref{f6d,,try} statement has another optional clause which is intended to
define clean-up actions that must be executed under all circumstances.  For
example:

@example
>>> try:
...     raise KeyboardInterrupt
... finally:
...     print('Goodbye, world!')
...
Goodbye, world!
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
KeyboardInterrupt
@end example

If a @ref{4c1,,finally} clause is present, the @code{finally}
clause will execute as the last task before the @ref{f6d,,try}
statement completes. The @code{finally} clause runs whether or
not the @code{try} statement produces an exception. The following
points discuss more complex cases when an exception occurs:


@itemize *

@item 
If an exception occurs during execution of the @code{try}
clause, the exception may be handled by an @ref{16e,,except}
clause. If the exception is not handled by an @code{except}
clause, the exception is re-raised after the @code{finally}
clause has been executed.

@item 
An exception could occur during execution of an @code{except}
or @code{else} clause. Again, the exception is re-raised after
the @code{finally} clause has been executed.

@item 
If the @code{finally} clause executes a @ref{5cb,,break},
@ref{4c0,,continue} or @ref{4c6,,return} statement, exceptions are not
re-raised.

@item 
If the @code{try} statement reaches a @ref{5cb,,break},
@ref{4c0,,continue} or @ref{4c6,,return} statement, the
@code{finally} clause will execute just prior to the
@code{break}, @code{continue} or @code{return}
statement’s execution.

@item 
If a @code{finally} clause includes a @code{return}
statement, the returned value will be the one from the
@code{finally} clause’s @code{return} statement, not the
value from the @code{try} clause’s @code{return}
statement.
@end itemize

For example:

@example
>>> def bool_return():
...     try:
...         return True
...     finally:
...         return False
...
>>> bool_return()
False
@end example

A more complicated example:

@example
>>> def divide(x, y):
...     try:
...         result = x / y
...     except ZeroDivisionError:
...         print("division by zero!")
...     else:
...         print("result is", result)
...     finally:
...         print("executing finally clause")
...
>>> divide(2, 1)
result is 2.0
executing finally clause
>>> divide(2, 0)
division by zero!
executing finally clause
>>> divide("2", "1")
executing finally clause
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in divide
TypeError: unsupported operand type(s) for /: 'str' and 'str'
@end example

As you can see, the @ref{4c1,,finally} clause is executed in any event.  The
@ref{162,,TypeError} raised by dividing two strings is not handled by the
@ref{16e,,except} clause and therefore re-raised after the @code{finally}
clause has been executed.

In real world applications, the @ref{4c1,,finally} clause is useful for
releasing external resources (such as files or network connections), regardless
of whether the use of the resource was successful.

@node Predefined Clean-up Actions,Raising and Handling Multiple Unrelated Exceptions,Defining Clean-up Actions,Errors and Exceptions
@anchor{tutorial/errors predefined-clean-up-actions}@anchor{1577}@anchor{tutorial/errors tut-cleanup-with}@anchor{1578}
@subsection Predefined Clean-up Actions


Some objects define standard clean-up actions to be undertaken when the object
is no longer needed, regardless of whether or not the operation using the object
succeeded or failed. Look at the following example, which tries to open a file
and print its contents to the screen.

@example
for line in open("myfile.txt"):
    print(line, end="")
@end example

The problem with this code is that it leaves the file open for an indeterminate
amount of time after this part of the code has finished executing.
This is not an issue in simple scripts, but can be a problem for larger
applications. The @ref{168,,with} statement allows objects like files to be
used in a way that ensures they are always cleaned up promptly and correctly.

@example
with open("myfile.txt") as f:
    for line in f:
        print(line, end="")
@end example

After the statement is executed, the file @emph{f} is always closed, even if a
problem was encountered while processing the lines. Objects which, like files,
provide predefined clean-up actions will indicate this in their documentation.

@node Raising and Handling Multiple Unrelated Exceptions,,Predefined Clean-up Actions,Errors and Exceptions
@anchor{tutorial/errors raising-and-handling-multiple-unrelated-exceptions}@anchor{1579}@anchor{tutorial/errors tut-exception-groups}@anchor{157a}
@subsection Raising and Handling Multiple Unrelated Exceptions


There are situations where it is necessary to report several exceptions that
have occurred. This it often the case in concurrency frameworks, when several
tasks may have failed in parallel, but there are also other use cases where
it is desirable to continue execution and collect multiple errors rather than
raise the first exception.

The builtin @ref{10db,,ExceptionGroup} wraps a list of exception instances so
that they can be raised together. It is an exception itself, so it can be
caught like any other exception.

@example
>>> def f():
...     excs = [OSError('error 1'), SystemError('error 2')]
...     raise ExceptionGroup('there were problems', excs)
...
>>> f()
  + Exception Group Traceback (most recent call last):
  |   File "<stdin>", line 1, in <module>
  |   File "<stdin>", line 3, in f
  | ExceptionGroup: there were problems
  +-+---------------- 1 ----------------
    | OSError: error 1
    +---------------- 2 ----------------
    | SystemError: error 2
    +------------------------------------
>>> try:
...     f()
... except Exception as e:
...     print(f'caught @{type(e)@}: e')
...
caught <class 'ExceptionGroup'>: e
>>>
@end example

By using @code{except*} instead of @code{except}, we can selectively
handle only the exceptions in the group that match a certain
type. In the following example, which shows a nested exception
group, each @code{except*} clause extracts from the group exceptions
of a certain type while letting all other exceptions propagate to
other clauses and eventually to be reraised.

@example
>>> def f():
...     raise ExceptionGroup("group1",
...                          [OSError(1),
...                           SystemError(2),
...                           ExceptionGroup("group2",
...                                          [OSError(3), RecursionError(4)])])
...
>>> try:
...     f()
... except* OSError as e:
...     print("There were OSErrors")
... except* SystemError as e:
...     print("There were SystemErrors")
...
There were OSErrors
There were SystemErrors
  + Exception Group Traceback (most recent call last):
  |   File "<stdin>", line 2, in <module>
  |   File "<stdin>", line 2, in f
  | ExceptionGroup: group1
  +-+---------------- 1 ----------------
    | ExceptionGroup: group2
    +-+---------------- 1 ----------------
      | RecursionError: 4
      +------------------------------------
>>>
@end example

Note that the exceptions nested in an exception group must be instances,
not types. This is because in practice the exceptions would typically
be ones that have already been raised and caught by the program, along
the following pattern:

@example
>>> excs = []
... for test in tests:
...     try:
...         test.run()
...     except Exception as e:
...         excs.append(e)
...
>>> if excs:
...    raise ExceptionGroup("Test Failures", excs)
...
@end example

@node Classes,Brief Tour of the Standard Library,Errors and Exceptions,The Python Tutorial
@anchor{tutorial/classes doc}@anchor{157b}@anchor{tutorial/classes classes}@anchor{157c}@anchor{tutorial/classes tut-classes}@anchor{1505}
@section Classes


Classes provide a means of bundling data and functionality together.  Creating
a new class creates a new @emph{type} of object, allowing new @emph{instances} of that
type to be made.  Each class instance can have attributes attached to it for
maintaining its state.  Class instances can also have methods (defined by its
class) for modifying its state.

Compared with other programming languages, Python’s class mechanism adds classes
with a minimum of new syntax and semantics.  It is a mixture of the class
mechanisms found in C++ and Modula-3.  Python classes provide all the standard
features of Object Oriented Programming: the class inheritance mechanism allows
multiple base classes, a derived class can override any methods of its base
class or classes, and a method can call the method of a base class with the same
name.  Objects can contain arbitrary amounts and kinds of data.  As is true for
modules, classes partake of the dynamic nature of Python: they are created at
runtime, and can be modified further after creation.

In C++ terminology, normally class members (including the data members) are
@emph{public} (except see below @ref{157d,,Private Variables}), and all member functions are
@emph{virtual}.  As in Modula-3, there are no shorthands for referencing the object’s
members from its methods: the method function is declared with an explicit first
argument representing the object, which is provided implicitly by the call.  As
in Smalltalk, classes themselves are objects.  This provides semantics for
importing and renaming.  Unlike C++ and Modula-3, built-in types can be used as
base classes for extension by the user.  Also, like in C++, most built-in
operators with special syntax (arithmetic operators, subscripting etc.) can be
redefined for class instances.

(Lacking universally accepted terminology to talk about classes, I will make
occasional use of Smalltalk and C++ terms.  I would use Modula-3 terms, since
its object-oriented semantics are closer to those of Python than C++, but I
expect that few readers have heard of it.)

@menu
* A Word About Names and Objects:: 
* Python Scopes and Namespaces:: 
* A First Look at Classes:: 
* Random Remarks:: 
* Inheritance:: 
* Private Variables:: 
* Odds and Ends:: 
* Iterators:: 
* Generators:: 
* Generator Expressions:: 

@end menu

@node A Word About Names and Objects,Python Scopes and Namespaces,,Classes
@anchor{tutorial/classes a-word-about-names-and-objects}@anchor{157e}@anchor{tutorial/classes tut-object}@anchor{157f}
@subsection A Word About Names and Objects


Objects have individuality, and multiple names (in multiple scopes) can be bound
to the same object.  This is known as aliasing in other languages.  This is
usually not appreciated on a first glance at Python, and can be safely ignored
when dealing with immutable basic types (numbers, strings, tuples).  However,
aliasing has a possibly surprising effect on the semantics of Python code
involving mutable objects such as lists, dictionaries, and most other types.
This is usually used to the benefit of the program, since aliases behave like
pointers in some respects.  For example, passing an object is cheap since only a
pointer is passed by the implementation; and if a function modifies an object
passed as an argument, the caller will see the change — this eliminates the
need for two different argument passing mechanisms as in Pascal.

@node Python Scopes and Namespaces,A First Look at Classes,A Word About Names and Objects,Classes
@anchor{tutorial/classes python-scopes-and-namespaces}@anchor{1580}@anchor{tutorial/classes tut-scopes}@anchor{1581}
@subsection Python Scopes and Namespaces


Before introducing classes, I first have to tell you something about Python’s
scope rules.  Class definitions play some neat tricks with namespaces, and you
need to know how scopes and namespaces work to fully understand what’s going on.
Incidentally, knowledge about this subject is useful for any advanced Python
programmer.

Let’s begin with some definitions.

A @emph{namespace} is a mapping from names to objects.  Most namespaces are currently
implemented as Python dictionaries, but that’s normally not noticeable in any
way (except for performance), and it may change in the future.  Examples of
namespaces are: the set of built-in names (containing functions such as @ref{1582,,abs()}, and
built-in exception names); the global names in a module; and the local names in
a function invocation.  In a sense the set of attributes of an object also form
a namespace.  The important thing to know about namespaces is that there is
absolutely no relation between names in different namespaces; for instance, two
different modules may both define a function @code{maximize} without confusion —
users of the modules must prefix it with the module name.

By the way, I use the word @emph{attribute} for any name following a dot — for
example, in the expression @code{z.real}, @code{real} is an attribute of the object
@code{z}.  Strictly speaking, references to names in modules are attribute
references: in the expression @code{modname.funcname}, @code{modname} is a module
object and @code{funcname} is an attribute of it.  In this case there happens to be
a straightforward mapping between the module’s attributes and the global names
defined in the module: they share the same namespace!  @footnote{@w{(1)} 
Except for one thing.  Module objects have a secret read-only attribute called
@ref{7a4,,__dict__} which returns the dictionary used to implement the module’s
namespace; the name @ref{7a4,,__dict__} is an attribute but not a global name.
Obviously, using this violates the abstraction of namespace implementation, and
should be restricted to things like post-mortem debuggers.
}

Attributes may be read-only or writable.  In the latter case, assignment to
attributes is possible.  Module attributes are writable: you can write
@code{modname.the_answer = 42}.  Writable attributes may also be deleted with the
@ref{152e,,del} statement.  For example, @code{del modname.the_answer} will remove
the attribute @code{the_answer} from the object named by @code{modname}.

Namespaces are created at different moments and have different lifetimes.  The
namespace containing the built-in names is created when the Python interpreter
starts up, and is never deleted.  The global namespace for a module is created
when the module definition is read in; normally, module namespaces also last
until the interpreter quits.  The statements executed by the top-level
invocation of the interpreter, either read from a script file or interactively,
are considered part of a module called @ref{1,,__main__}, so they have their own
global namespace.  (The built-in names actually also live in a module; this is
called @ref{11,,builtins}.)

The local namespace for a function is created when the function is called, and
deleted when the function returns or raises an exception that is not handled
within the function.  (Actually, forgetting would be a better way to describe
what actually happens.)  Of course, recursive invocations each have their own
local namespace.

A @emph{scope} is a textual region of a Python program where a namespace is directly
accessible.  “Directly accessible” here means that an unqualified reference to a
name attempts to find the name in the namespace.

Although scopes are determined statically, they are used dynamically. At any
time during execution, there are 3 or 4 nested scopes whose namespaces are
directly accessible:


@itemize *

@item 
the innermost scope, which is searched first, contains the local names

@item 
the scopes of any enclosing functions, which are searched starting with the
nearest enclosing scope, contains non-local, but also non-global names

@item 
the next-to-last scope contains the current module’s global names

@item 
the outermost scope (searched last) is the namespace containing built-in names
@end itemize

If a name is declared global, then all references and assignments go directly to
the middle scope containing the module’s global names.  To rebind variables
found outside of the innermost scope, the @ref{e4d,,nonlocal} statement can be
used; if not declared nonlocal, those variables are read-only (an attempt to
write to such a variable will simply create a @emph{new} local variable in the
innermost scope, leaving the identically named outer variable unchanged).

Usually, the local scope references the local names of the (textually) current
function.  Outside functions, the local scope references the same namespace as
the global scope: the module’s namespace. Class definitions place yet another
namespace in the local scope.

It is important to realize that scopes are determined textually: the global
scope of a function defined in a module is that module’s namespace, no matter
from where or by what alias the function is called.  On the other hand, the
actual search for names is done dynamically, at run time — however, the
language definition is evolving towards static name resolution, at “compile”
time, so don’t rely on dynamic name resolution!  (In fact, local variables are
already determined statically.)

A special quirk of Python is that – if no @ref{1504,,global} or @ref{e4d,,nonlocal}
statement is in effect – assignments to names always go into the innermost scope.
Assignments do not copy data — they just bind names to objects.  The same is true
for deletions: the statement @code{del x} removes the binding of @code{x} from the
namespace referenced by the local scope.  In fact, all operations that introduce
new names use the local scope: in particular, @ref{e35,,import} statements and
function definitions bind the module or function name in the local scope.

The @ref{1504,,global} statement can be used to indicate that particular
variables live in the global scope and should be rebound there; the
@ref{e4d,,nonlocal} statement indicates that particular variables live in
an enclosing scope and should be rebound there.

@menu
* Scopes and Namespaces Example:: 

@end menu

@node Scopes and Namespaces Example,,,Python Scopes and Namespaces
@anchor{tutorial/classes scopes-and-namespaces-example}@anchor{1583}@anchor{tutorial/classes tut-scopeexample}@anchor{1584}
@subsubsection Scopes and Namespaces Example


This is an example demonstrating how to reference the different scopes and
namespaces, and how @ref{1504,,global} and @ref{e4d,,nonlocal} affect variable
binding:

@example
def scope_test():
    def do_local():
        spam = "local spam"

    def do_nonlocal():
        nonlocal spam
        spam = "nonlocal spam"

    def do_global():
        global spam
        spam = "global spam"

    spam = "test spam"
    do_local()
    print("After local assignment:", spam)
    do_nonlocal()
    print("After nonlocal assignment:", spam)
    do_global()
    print("After global assignment:", spam)

scope_test()
print("In global scope:", spam)
@end example

The output of the example code is:

@example
After local assignment: test spam
After nonlocal assignment: nonlocal spam
After global assignment: nonlocal spam
In global scope: global spam
@end example

Note how the @emph{local} assignment (which is default) didn’t change @emph{scope_test}'s
binding of @emph{spam}.  The @ref{e4d,,nonlocal} assignment changed @emph{scope_test}'s
binding of @emph{spam}, and the @ref{1504,,global} assignment changed the module-level
binding.

You can also see that there was no previous binding for @emph{spam} before the
@ref{1504,,global} assignment.

@node A First Look at Classes,Random Remarks,Python Scopes and Namespaces,Classes
@anchor{tutorial/classes a-first-look-at-classes}@anchor{1585}@anchor{tutorial/classes tut-firstclasses}@anchor{1520}
@subsection A First Look at Classes


Classes introduce a little bit of new syntax, three new object types, and some
new semantics.

@menu
* Class Definition Syntax:: 
* Class Objects:: 
* Instance Objects:: 
* Method Objects:: 
* Class and Instance Variables:: 

@end menu

@node Class Definition Syntax,Class Objects,,A First Look at Classes
@anchor{tutorial/classes class-definition-syntax}@anchor{1586}@anchor{tutorial/classes tut-classdefinition}@anchor{1587}
@subsubsection Class Definition Syntax


The simplest form of class definition looks like this:

@example
class ClassName:
    <statement-1>
    .
    .
    .
    <statement-N>
@end example

Class definitions, like function definitions (@ref{fb8,,def} statements) must be
executed before they have any effect.  (You could conceivably place a class
definition in a branch of an @ref{2fb,,if} statement, or inside a function.)

In practice, the statements inside a class definition will usually be function
definitions, but other statements are allowed, and sometimes useful — we’ll
come back to this later.  The function definitions inside a class normally have
a peculiar form of argument list, dictated by the calling conventions for
methods — again, this is explained later.

When a class definition is entered, a new namespace is created, and used as the
local scope — thus, all assignments to local variables go into this new
namespace.  In particular, function definitions bind the name of the new
function here.

When a class definition is left normally (via the end), a @emph{class object} is
created.  This is basically a wrapper around the contents of the namespace
created by the class definition; we’ll learn more about class objects in the
next section.  The original local scope (the one in effect just before the class
definition was entered) is reinstated, and the class object is bound here to the
class name given in the class definition header (@code{ClassName} in the
example).

@node Class Objects,Instance Objects,Class Definition Syntax,A First Look at Classes
@anchor{tutorial/classes class-objects}@anchor{1588}@anchor{tutorial/classes tut-classobjects}@anchor{1589}
@subsubsection Class Objects


Class objects support two kinds of operations: attribute references and
instantiation.

@emph{Attribute references} use the standard syntax used for all attribute references
in Python: @code{obj.name}.  Valid attribute names are all the names that were in
the class’s namespace when the class object was created.  So, if the class
definition looked like this:

@example
class MyClass:
    """A simple example class"""
    i = 12345

    def f(self):
        return 'hello world'
@end example

then @code{MyClass.i} and @code{MyClass.f} are valid attribute references, returning
an integer and a function object, respectively. Class attributes can also be
assigned to, so you can change the value of @code{MyClass.i} by assignment.
@code{__doc__} is also a valid attribute, returning the docstring belonging to
the class: @code{"A simple example class"}.

Class @emph{instantiation} uses function notation.  Just pretend that the class
object is a parameterless function that returns a new instance of the class.
For example (assuming the above class):

@example
x = MyClass()
@end example

creates a new @emph{instance} of the class and assigns this object to the local
variable @code{x}.

The instantiation operation (“calling” a class object) creates an empty object.
Many classes like to create objects with instances customized to a specific
initial state. Therefore a class may define a special method named
@code{__init__()}, like this:

@example
def __init__(self):
    self.data = []
@end example

When a class defines an @code{__init__()} method, class instantiation
automatically invokes @code{__init__()} for the newly-created class instance.  So
in this example, a new, initialized instance can be obtained by:

@example
x = MyClass()
@end example

Of course, the @code{__init__()} method may have arguments for greater
flexibility.  In that case, arguments given to the class instantiation operator
are passed on to @code{__init__()}.  For example,

@example
>>> class Complex:
...     def __init__(self, realpart, imagpart):
...         self.r = realpart
...         self.i = imagpart
...
>>> x = Complex(3.0, -4.5)
>>> x.r, x.i
(3.0, -4.5)
@end example

@node Instance Objects,Method Objects,Class Objects,A First Look at Classes
@anchor{tutorial/classes instance-objects}@anchor{158a}@anchor{tutorial/classes tut-instanceobjects}@anchor{158b}
@subsubsection Instance Objects


Now what can we do with instance objects?  The only operations understood by
instance objects are attribute references.  There are two kinds of valid
attribute names: data attributes and methods.

@emph{data attributes} correspond to “instance variables” in Smalltalk, and to “data
members” in C++.  Data attributes need not be declared; like local variables,
they spring into existence when they are first assigned to.  For example, if
@code{x} is the instance of @code{MyClass} created above, the following piece of
code will print the value @code{16}, without leaving a trace:

@example
x.counter = 1
while x.counter < 10:
    x.counter = x.counter * 2
print(x.counter)
del x.counter
@end example

The other kind of instance attribute reference is a @emph{method}. A method is a
function that “belongs to” an object.  (In Python, the term method is not unique
to class instances: other object types can have methods as well.  For example,
list objects have methods called append, insert, remove, sort, and so on.
However, in the following discussion, we’ll use the term method exclusively to
mean methods of class instance objects, unless explicitly stated otherwise.)

@geindex object; method

Valid method names of an instance object depend on its class.  By definition,
all attributes of a class that are function  objects define corresponding
methods of its instances.  So in our example, @code{x.f} is a valid method
reference, since @code{MyClass.f} is a function, but @code{x.i} is not, since
@code{MyClass.i} is not.  But @code{x.f} is not the same thing as @code{MyClass.f} — it
is a @emph{method object}, not a function object.

@node Method Objects,Class and Instance Variables,Instance Objects,A First Look at Classes
@anchor{tutorial/classes method-objects}@anchor{158c}@anchor{tutorial/classes tut-methodobjects}@anchor{158d}
@subsubsection Method Objects


Usually, a method is called right after it is bound:

@example
x.f()
@end example

In the @code{MyClass} example, this will return the string @code{'hello world'}.
However, it is not necessary to call a method right away: @code{x.f} is a method
object, and can be stored away and called at a later time.  For example:

@example
xf = x.f
while True:
    print(xf())
@end example

will continue to print @code{hello world} until the end of time.

What exactly happens when a method is called?  You may have noticed that
@code{x.f()} was called without an argument above, even though the function
definition for @code{f()} specified an argument.  What happened to the argument?
Surely Python raises an exception when a function that requires an argument is
called without any — even if the argument isn’t actually used…

Actually, you may have guessed the answer: the special thing about methods is
that the instance object is passed as the first argument of the function.  In our
example, the call @code{x.f()} is exactly equivalent to @code{MyClass.f(x)}.  In
general, calling a method with a list of @emph{n} arguments is equivalent to calling
the corresponding function with an argument list that is created by inserting
the method’s instance object before the first argument.

If you still don’t understand how methods work, a look at the implementation can
perhaps clarify matters.  When a non-data attribute of an instance is
referenced, the instance’s class is searched.  If the name denotes a valid class
attribute that is a function object, a method object is created by packing
(pointers to) the instance object and the function object just found together in
an abstract object: this is the method object.  When the method object is called
with an argument list, a new argument list is constructed from the instance
object and the argument list, and the function object is called with this new
argument list.

@node Class and Instance Variables,,Method Objects,A First Look at Classes
@anchor{tutorial/classes class-and-instance-variables}@anchor{158e}@anchor{tutorial/classes tut-class-and-instance-variables}@anchor{158f}
@subsubsection Class and Instance Variables


Generally speaking, instance variables are for data unique to each instance
and class variables are for attributes and methods shared by all instances
of the class:

@example
class Dog:

    kind = 'canine'         # class variable shared by all instances

    def __init__(self, name):
        self.name = name    # instance variable unique to each instance

>>> d = Dog('Fido')
>>> e = Dog('Buddy')
>>> d.kind                  # shared by all dogs
'canine'
>>> e.kind                  # shared by all dogs
'canine'
>>> d.name                  # unique to d
'Fido'
>>> e.name                  # unique to e
'Buddy'
@end example

As discussed in @ref{157f,,A Word About Names and Objects}, shared data can have possibly surprising
effects with involving @ref{14e9,,mutable} objects such as lists and dictionaries.
For example, the @emph{tricks} list in the following code should not be used as a
class variable because just a single list would be shared by all @emph{Dog}
instances:

@example
class Dog:

    tricks = []             # mistaken use of a class variable

    def __init__(self, name):
        self.name = name

    def add_trick(self, trick):
        self.tricks.append(trick)

>>> d = Dog('Fido')
>>> e = Dog('Buddy')
>>> d.add_trick('roll over')
>>> e.add_trick('play dead')
>>> d.tricks                # unexpectedly shared by all dogs
['roll over', 'play dead']
@end example

Correct design of the class should use an instance variable instead:

@example
class Dog:

    def __init__(self, name):
        self.name = name
        self.tricks = []    # creates a new empty list for each dog

    def add_trick(self, trick):
        self.tricks.append(trick)

>>> d = Dog('Fido')
>>> e = Dog('Buddy')
>>> d.add_trick('roll over')
>>> e.add_trick('play dead')
>>> d.tricks
['roll over']
>>> e.tricks
['play dead']
@end example

@node Random Remarks,Inheritance,A First Look at Classes,Classes
@anchor{tutorial/classes random-remarks}@anchor{1590}@anchor{tutorial/classes tut-remarks}@anchor{1591}
@subsection Random Remarks


@c These should perhaps be placed more carefully...

If the same attribute name occurs in both an instance and in a class,
then attribute lookup prioritizes the instance:

@example
>>> class Warehouse:
        purpose = 'storage'
        region = 'west'

>>> w1 = Warehouse()
>>> print(w1.purpose, w1.region)
storage west
>>> w2 = Warehouse()
>>> w2.region = 'east'
>>> print(w2.purpose, w2.region)
storage east
@end example

Data attributes may be referenced by methods as well as by ordinary users
(“clients”) of an object.  In other words, classes are not usable to implement
pure abstract data types.  In fact, nothing in Python makes it possible to
enforce data hiding — it is all based upon convention.  (On the other hand,
the Python implementation, written in C, can completely hide implementation
details and control access to an object if necessary; this can be used by
extensions to Python written in C.)

Clients should use data attributes with care — clients may mess up invariants
maintained by the methods by stamping on their data attributes.  Note that
clients may add data attributes of their own to an instance object without
affecting the validity of the methods, as long as name conflicts are avoided —
again, a naming convention can save a lot of headaches here.

There is no shorthand for referencing data attributes (or other methods!) from
within methods.  I find that this actually increases the readability of methods:
there is no chance of confusing local variables and instance variables when
glancing through a method.

Often, the first argument of a method is called @code{self}.  This is nothing more
than a convention: the name @code{self} has absolutely no special meaning to
Python.  Note, however, that by not following the convention your code may be
less readable to other Python programmers, and it is also conceivable that a
@emph{class browser} program might be written that relies upon such a convention.

Any function object that is a class attribute defines a method for instances of
that class.  It is not necessary that the function definition is textually
enclosed in the class definition: assigning a function object to a local
variable in the class is also ok.  For example:

@example
# Function defined outside the class
def f1(self, x, y):
    return min(x, x+y)

class C:
    f = f1

    def g(self):
        return 'hello world'

    h = g
@end example

Now @code{f}, @code{g} and @code{h} are all attributes of class @code{C} that refer to
function objects, and consequently they are all methods of instances of
@code{C} — @code{h} being exactly equivalent to @code{g}.  Note that this practice
usually only serves to confuse the reader of a program.

Methods may call other methods by using method attributes of the @code{self}
argument:

@example
class Bag:
    def __init__(self):
        self.data = []

    def add(self, x):
        self.data.append(x)

    def addtwice(self, x):
        self.add(x)
        self.add(x)
@end example

Methods may reference global names in the same way as ordinary functions.  The
global scope associated with a method is the module containing its
definition.  (A class is never used as a global scope.)  While one
rarely encounters a good reason for using global data in a method, there are
many legitimate uses of the global scope: for one thing, functions and modules
imported into the global scope can be used by methods, as well as functions and
classes defined in it.  Usually, the class containing the method is itself
defined in this global scope, and in the next section we’ll find some good
reasons why a method would want to reference its own class.

Each value is an object, and therefore has a @emph{class} (also called its @emph{type}).
It is stored as @code{object.__class__}.

@node Inheritance,Private Variables,Random Remarks,Classes
@anchor{tutorial/classes inheritance}@anchor{1592}@anchor{tutorial/classes tut-inheritance}@anchor{1593}
@subsection Inheritance


Of course, a language feature would not be worthy of the name “class” without
supporting inheritance.  The syntax for a derived class definition looks like
this:

@example
class DerivedClassName(BaseClassName):
    <statement-1>
    .
    .
    .
    <statement-N>
@end example

The name @code{BaseClassName} must be defined in a scope containing the
derived class definition.  In place of a base class name, other arbitrary
expressions are also allowed.  This can be useful, for example, when the base
class is defined in another module:

@example
class DerivedClassName(modname.BaseClassName):
@end example

Execution of a derived class definition proceeds the same as for a base class.
When the class object is constructed, the base class is remembered.  This is
used for resolving attribute references: if a requested attribute is not found
in the class, the search proceeds to look in the base class.  This rule is
applied recursively if the base class itself is derived from some other class.

There’s nothing special about instantiation of derived classes:
@code{DerivedClassName()} creates a new instance of the class.  Method references
are resolved as follows: the corresponding class attribute is searched,
descending down the chain of base classes if necessary, and the method reference
is valid if this yields a function object.

Derived classes may override methods of their base classes.  Because methods
have no special privileges when calling other methods of the same object, a
method of a base class that calls another method defined in the same base class
may end up calling a method of a derived class that overrides it.  (For C++
programmers: all methods in Python are effectively @code{virtual}.)

An overriding method in a derived class may in fact want to extend rather than
simply replace the base class method of the same name. There is a simple way to
call the base class method directly: just call @code{BaseClassName.methodname(self,
arguments)}.  This is occasionally useful to clients as well.  (Note that this
only works if the base class is accessible as @code{BaseClassName} in the global
scope.)

Python has two built-in functions that work with inheritance:


@itemize *

@item 
Use @ref{236,,isinstance()} to check an instance’s type: @code{isinstance(obj, int)}
will be @code{True} only if @code{obj.__class__} is @ref{1b1,,int} or some class
derived from @ref{1b1,,int}.

@item 
Use @ref{237,,issubclass()} to check class inheritance: @code{issubclass(bool, int)}
is @code{True} since @ref{2f7,,bool} is a subclass of @ref{1b1,,int}.  However,
@code{issubclass(float, int)} is @code{False} since @ref{256,,float} is not a
subclass of @ref{1b1,,int}.
@end itemize

@menu
* Multiple Inheritance:: 

@end menu

@node Multiple Inheritance,,,Inheritance
@anchor{tutorial/classes multiple-inheritance}@anchor{1594}@anchor{tutorial/classes tut-multiple}@anchor{1595}
@subsubsection Multiple Inheritance


Python supports a form of multiple inheritance as well.  A class definition with
multiple base classes looks like this:

@example
class DerivedClassName(Base1, Base2, Base3):
    <statement-1>
    .
    .
    .
    <statement-N>
@end example

For most purposes, in the simplest cases, you can think of the search for
attributes inherited from a parent class as depth-first, left-to-right, not
searching twice in the same class where there is an overlap in the hierarchy.
Thus, if an attribute is not found in @code{DerivedClassName}, it is searched
for in @code{Base1}, then (recursively) in the base classes of @code{Base1},
and if it was not found there, it was searched for in @code{Base2}, and so on.

In fact, it is slightly more complex than that; the method resolution order
changes dynamically to support cooperative calls to @ref{78c,,super()}.  This
approach is known in some other multiple-inheritance languages as
call-next-method and is more powerful than the super call found in
single-inheritance languages.

Dynamic ordering is necessary because all cases of multiple inheritance exhibit
one or more diamond relationships (where at least one of the parent classes
can be accessed through multiple paths from the bottommost class).  For example,
all classes inherit from @ref{5a5,,object}, so any case of multiple inheritance
provides more than one path to reach @ref{5a5,,object}.  To keep the base classes
from being accessed more than once, the dynamic algorithm linearizes the search
order in a way that preserves the left-to-right ordering specified in each
class, that calls each parent only once, and that is monotonic (meaning that a
class can be subclassed without affecting the precedence order of its parents).
Taken together, these properties make it possible to design reliable and
extensible classes with multiple inheritance.  For more detail, see
@indicateurl{https://www.python.org/download/releases/2.3/mro/}.

@node Private Variables,Odds and Ends,Inheritance,Classes
@anchor{tutorial/classes private-variables}@anchor{1596}@anchor{tutorial/classes tut-private}@anchor{157d}
@subsection Private Variables


“Private” instance variables that cannot be accessed except from inside an
object don’t exist in Python.  However, there is a convention that is followed
by most Python code: a name prefixed with an underscore (e.g. @code{_spam}) should
be treated as a non-public part of the API (whether it is a function, a method
or a data member).  It should be considered an implementation detail and subject
to change without notice.

@geindex name; mangling

Since there is a valid use-case for class-private members (namely to avoid name
clashes of names with names defined by subclasses), there is limited support for
such a mechanism, called @emph{name mangling}.  Any identifier of the form
@code{__spam} (at least two leading underscores, at most one trailing underscore)
is textually replaced with @code{_classname__spam}, where @code{classname} is the
current class name with leading underscore(s) stripped.  This mangling is done
without regard to the syntactic position of the identifier, as long as it
occurs within the definition of a class.

Name mangling is helpful for letting subclasses override methods without
breaking intraclass method calls.  For example:

@example
class Mapping:
    def __init__(self, iterable):
        self.items_list = []
        self.__update(iterable)

    def update(self, iterable):
        for item in iterable:
            self.items_list.append(item)

    __update = update   # private copy of original update() method

class MappingSubclass(Mapping):

    def update(self, keys, values):
        # provides new signature for update()
        # but does not break __init__()
        for item in zip(keys, values):
            self.items_list.append(item)
@end example

The above example would work even if @code{MappingSubclass} were to introduce a
@code{__update} identifier since it is replaced with @code{_Mapping__update} in the
@code{Mapping} class  and @code{_MappingSubclass__update} in the @code{MappingSubclass}
class respectively.

Note that the mangling rules are designed mostly to avoid accidents; it still is
possible to access or modify a variable that is considered private.  This can
even be useful in special circumstances, such as in the debugger.

Notice that code passed to @code{exec()} or @code{eval()} does not consider the
classname of the invoking class to be the current class; this is similar to the
effect of the @code{global} statement, the effect of which is likewise restricted
to code that is byte-compiled together.  The same restriction applies to
@code{getattr()}, @code{setattr()} and @code{delattr()}, as well as when referencing
@code{__dict__} directly.

@node Odds and Ends,Iterators,Private Variables,Classes
@anchor{tutorial/classes odds-and-ends}@anchor{1597}@anchor{tutorial/classes tut-odds}@anchor{1598}
@subsection Odds and Ends


Sometimes it is useful to have a data type similar to the Pascal “record” or C
“struct”, bundling together a few named data items.  An empty class definition
will do nicely:

@example
class Employee:
    pass

john = Employee()  # Create an empty employee record

# Fill the fields of the record
john.name = 'John Doe'
john.dept = 'computer lab'
john.salary = 1000
@end example

A piece of Python code that expects a particular abstract data type can often be
passed a class that emulates the methods of that data type instead.  For
instance, if you have a function that formats some data from a file object, you
can define a class with methods @code{read()} and @code{readline()} that get the
data from a string buffer instead, and pass it as an argument.

@c (Unfortunately, this technique has its limitations: a class can't define
@c operations that are accessed by special syntax such as sequence subscripting
@c or arithmetic operators, and assigning such a "pseudo-file" to sys.stdin will
@c not cause the interpreter to read further input from it.)

Instance method objects have attributes, too: @code{m.__self__} is the instance
object with the method @code{m()}, and @code{m.__func__} is the function object
corresponding to the method.

@node Iterators,Generators,Odds and Ends,Classes
@anchor{tutorial/classes iterators}@anchor{1599}@anchor{tutorial/classes tut-iterators}@anchor{159a}
@subsection Iterators


By now you have probably noticed that most container objects can be looped over
using a @ref{161,,for} statement:

@example
for element in [1, 2, 3]:
    print(element)
for element in (1, 2, 3):
    print(element)
for key in @{'one':1, 'two':2@}:
    print(key)
for char in "123":
    print(char)
for line in open("myfile.txt"):
    print(line, end='@w{'})
@end example

This style of access is clear, concise, and convenient.  The use of iterators
pervades and unifies Python.  Behind the scenes, the @ref{161,,for} statement
calls @ref{251,,iter()} on the container object.  The function returns an iterator
object that defines the method @ref{e6d,,__next__()} which accesses
elements in the container one at a time.  When there are no more elements,
@ref{e6d,,__next__()} raises a @ref{736,,StopIteration} exception which tells the
@code{for} loop to terminate.  You can call the @ref{e6d,,__next__()} method
using the @ref{252,,next()} built-in function; this example shows how it all works:

@example
>>> s = 'abc'
>>> it = iter(s)
>>> it
<str_iterator object at 0x10c90e650>
>>> next(it)
'a'
>>> next(it)
'b'
>>> next(it)
'c'
>>> next(it)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    next(it)
StopIteration
@end example

Having seen the mechanics behind the iterator protocol, it is easy to add
iterator behavior to your classes.  Define an @code{__iter__()} method which
returns an object with a @ref{e6d,,__next__()} method.  If the class
defines @code{__next__()}, then @code{__iter__()} can just return @code{self}:

@example
class Reverse:
    """Iterator for looping over a sequence backwards."""
    def __init__(self, data):
        self.data = data
        self.index = len(data)

    def __iter__(self):
        return self

    def __next__(self):
        if self.index == 0:
            raise StopIteration
        self.index = self.index - 1
        return self.data[self.index]
@end example

@example
>>> rev = Reverse('spam')
>>> iter(rev)
<__main__.Reverse object at 0x00A1DB50>
>>> for char in rev:
...     print(char)
...
m
a
p
s
@end example

@node Generators,Generator Expressions,Iterators,Classes
@anchor{tutorial/classes generators}@anchor{159b}@anchor{tutorial/classes tut-generators}@anchor{159c}
@subsection Generators


@ref{bf7,,Generators} are a simple and powerful tool for creating iterators.  They
are written like regular functions but use the @ref{4c5,,yield} statement
whenever they want to return data.  Each time @ref{252,,next()} is called on it, the
generator resumes where it left off (it remembers all the data values and which
statement was last executed).  An example shows that generators can be trivially
easy to create:

@example
def reverse(data):
    for index in range(len(data)-1, -1, -1):
        yield data[index]
@end example

@example
>>> for char in reverse('golf'):
...     print(char)
...
f
l
o
g
@end example

Anything that can be done with generators can also be done with class-based
iterators as described in the previous section.  What makes generators so
compact is that the @code{__iter__()} and @ref{159d,,__next__()} methods
are created automatically.

Another key feature is that the local variables and execution state are
automatically saved between calls.  This made the function easier to write and
much more clear than an approach using instance variables like @code{self.index}
and @code{self.data}.

In addition to automatic method creation and saving program state, when
generators terminate, they automatically raise @ref{736,,StopIteration}. In
combination, these features make it easy to create iterators with no more effort
than writing a regular function.

@node Generator Expressions,,Generators,Classes
@anchor{tutorial/classes generator-expressions}@anchor{159e}@anchor{tutorial/classes tut-genexps}@anchor{159f}
@subsection Generator Expressions


Some simple generators can be coded succinctly as expressions using a syntax
similar to list comprehensions but with parentheses instead of square brackets.
These expressions are designed for situations where the generator is used right
away by an enclosing function.  Generator expressions are more compact but less
versatile than full generator definitions and tend to be more memory friendly
than equivalent list comprehensions.

Examples:

@example
>>> sum(i*i for i in range(10))                 # sum of squares
285

>>> xvec = [10, 20, 30]
>>> yvec = [7, 5, 3]
>>> sum(x*y for x,y in zip(xvec, yvec))         # dot product
260

>>> unique_words = set(word for line in page  for word in line.split())

>>> valedictorian = max((student.gpa, student.name) for student in graduates)

>>> data = 'golf'
>>> list(data[i] for i in range(len(data)-1, -1, -1))
['f', 'l', 'o', 'g']
@end example

@node Brief Tour of the Standard Library,Brief Tour of the Standard Library — Part II,Classes,The Python Tutorial
@anchor{tutorial/stdlib doc}@anchor{15a0}@anchor{tutorial/stdlib brief-tour-of-the-standard-library}@anchor{15a1}@anchor{tutorial/stdlib tut-brieftour}@anchor{15a2}
@section Brief Tour of the Standard Library


@menu
* Operating System Interface:: 
* File Wildcards:: 
* Command Line Arguments:: 
* Error Output Redirection and Program Termination:: 
* String Pattern Matching:: 
* Mathematics:: 
* Internet Access:: 
* Dates and Times:: 
* Data Compression:: 
* Performance Measurement:: 
* Quality Control:: 
* Batteries Included:: 

@end menu

@node Operating System Interface,File Wildcards,,Brief Tour of the Standard Library
@anchor{tutorial/stdlib operating-system-interface}@anchor{15a3}@anchor{tutorial/stdlib tut-os-interface}@anchor{15a4}
@subsection Operating System Interface


The @ref{c0,,os} module provides dozens of functions for interacting with the
operating system:

@example
>>> import os
>>> os.getcwd()      # Return the current working directory
'C:\\Python311'
>>> os.chdir('/server/accesslogs')   # Change current working directory
>>> os.system('mkdir today')   # Run the command mkdir in the system shell
0
@end example

Be sure to use the @code{import os} style instead of @code{from os import *}.  This
will keep @ref{8ee,,os.open()} from shadowing the built-in @ref{1c8,,open()} function which
operates much differently.

@geindex built-in function; help

The built-in @ref{f2f,,dir()} and @ref{3bc,,help()} functions are useful as interactive
aids for working with large modules like @ref{c0,,os}:

@example
>>> import os
>>> dir(os)
<returns a list of all module functions>
>>> help(os)
<returns an extensive manual page created from the module's docstrings>
@end example

For daily file and directory management tasks, the @ref{e4,,shutil} module provides
a higher level interface that is easier to use:

@example
>>> import shutil
>>> shutil.copyfile('data.db', 'archive.db')
'archive.db'
>>> shutil.move('/build/executables', 'installdir')
'installdir'
@end example

@node File Wildcards,Command Line Arguments,Operating System Interface,Brief Tour of the Standard Library
@anchor{tutorial/stdlib file-wildcards}@anchor{15a5}@anchor{tutorial/stdlib tut-file-wildcards}@anchor{15a6}
@subsection File Wildcards


The @ref{84,,glob} module provides a function for making file lists from directory
wildcard searches:

@example
>>> import glob
>>> glob.glob('*.py')
['primes.py', 'random.py', 'quote.py']
@end example

@node Command Line Arguments,Error Output Redirection and Program Termination,File Wildcards,Brief Tour of the Standard Library
@anchor{tutorial/stdlib command-line-arguments}@anchor{15a7}@anchor{tutorial/stdlib tut-command-line-arguments}@anchor{15a8}
@subsection Command Line Arguments


Common utility scripts often need to process command line arguments. These
arguments are stored in the @ref{f7,,sys} module’s @emph{argv} attribute as a list.  For
instance the following output results from running @code{python demo.py one two
three} at the command line:

@example
>>> import sys
>>> print(sys.argv)
['demo.py', 'one', 'two', 'three']
@end example

The @ref{5,,argparse} module provides a more sophisticated mechanism to process
command line arguments.  The following script extracts one or more filenames
and an optional number of lines to be displayed:

@example
import argparse

parser = argparse.ArgumentParser(prog = 'top',
    description = 'Show top lines from each file')
parser.add_argument('filenames', nargs='+')
parser.add_argument('-l', '--lines', type=int, default=10)
args = parser.parse_args()
print(args)
@end example

When run at the command line with @code{python top.py --lines=5 alpha.txt
beta.txt}, the script sets @code{args.lines} to @code{5} and @code{args.filenames}
to @code{['alpha.txt', 'beta.txt']}.

@node Error Output Redirection and Program Termination,String Pattern Matching,Command Line Arguments,Brief Tour of the Standard Library
@anchor{tutorial/stdlib error-output-redirection-and-program-termination}@anchor{15a9}@anchor{tutorial/stdlib tut-stderr}@anchor{15aa}
@subsection Error Output Redirection and Program Termination


The @ref{f7,,sys} module also has attributes for @emph{stdin}, @emph{stdout}, and @emph{stderr}.
The latter is useful for emitting warnings and error messages to make them
visible even when @emph{stdout} has been redirected:

@example
>>> sys.stderr.write('Warning, log file not found starting a new one\n')
Warning, log file not found starting a new one
@end example

The most direct way to terminate a script is to use @code{sys.exit()}.

@node String Pattern Matching,Mathematics,Error Output Redirection and Program Termination,Brief Tour of the Standard Library
@anchor{tutorial/stdlib string-pattern-matching}@anchor{15ab}@anchor{tutorial/stdlib tut-string-pattern-matching}@anchor{15ac}
@subsection String Pattern Matching


The @ref{d8,,re} module provides regular expression tools for advanced string
processing. For complex matching and manipulation, regular expressions offer
succinct, optimized solutions:

@example
>>> import re
>>> re.findall(r'\bf[a-z]*', 'which foot or hand fell fastest')
['foot', 'fell', 'fastest']
>>> re.sub(r'(\b[a-z]+) \1', r'\1', 'cat in the the hat')
'cat in the hat'
@end example

When only simple capabilities are needed, string methods are preferred because
they are easier to read and debug:

@example
>>> 'tea for too'.replace('too', 'two')
'tea for two'
@end example

@node Mathematics,Internet Access,String Pattern Matching,Brief Tour of the Standard Library
@anchor{tutorial/stdlib mathematics}@anchor{15ad}@anchor{tutorial/stdlib tut-mathematics}@anchor{15ae}
@subsection Mathematics


The @ref{ad,,math} module gives access to the underlying C library functions for
floating point math:

@example
>>> import math
>>> math.cos(math.pi / 4)
0.70710678118654757
>>> math.log(1024, 2)
10.0
@end example

The @ref{d7,,random} module provides tools for making random selections:

@example
>>> import random
>>> random.choice(['apple', 'pear', 'banana'])
'apple'
>>> random.sample(range(100), 10)   # sampling without replacement
[30, 83, 16, 4, 8, 81, 41, 50, 18, 33]
>>> random.random()    # random float
0.17970987693706186
>>> random.randrange(6)    # random integer chosen from range(6)
4
@end example

The @ref{f0,,statistics} module calculates basic statistical properties
(the mean, median, variance, etc.) of numeric data:

@example
>>> import statistics
>>> data = [2.75, 1.75, 1.25, 0.25, 0.5, 1.25, 3.5]
>>> statistics.mean(data)
1.6071428571428572
>>> statistics.median(data)
1.25
>>> statistics.variance(data)
1.3720238095238095
@end example

The SciPy project <@indicateurl{https://scipy.org}> has many other modules for numerical
computations.

@node Internet Access,Dates and Times,Mathematics,Brief Tour of the Standard Library
@anchor{tutorial/stdlib internet-access}@anchor{15af}@anchor{tutorial/stdlib tut-internet-access}@anchor{15b0}
@subsection Internet Access


There are a number of modules for accessing the internet and processing internet
protocols. Two of the simplest are @ref{127,,urllib.request} for retrieving data
from URLs and @ref{e8,,smtplib} for sending mail:

@example
>>> from urllib.request import urlopen
>>> with urlopen('http://worldtimeapi.org/api/timezone/etc/UTC.txt') as response:
...     for line in response:
...         line = line.decode()             # Convert bytes to a str
...         if line.startswith('datetime'):
...             print(line.rstrip())         # Remove trailing newline
...
datetime: 2022-01-01T01:36:47.689215+00:00

>>> import smtplib
>>> server = smtplib.SMTP('localhost')
>>> server.sendmail('soothsayer@@example.org', 'jcaesar@@example.org',
... """To: jcaesar@@example.org
... From: soothsayer@@example.org
...
... Beware the Ides of March.
... """)
>>> server.quit()
@end example

(Note that the second example needs a mailserver running on localhost.)

@node Dates and Times,Data Compression,Internet Access,Brief Tour of the Standard Library
@anchor{tutorial/stdlib dates-and-times}@anchor{15b1}@anchor{tutorial/stdlib tut-dates-and-times}@anchor{15b2}
@subsection Dates and Times


The @ref{2f,,datetime} module supplies classes for manipulating dates and times in
both simple and complex ways. While date and time arithmetic is supported, the
focus of the implementation is on efficient member extraction for output
formatting and manipulation.  The module also supports objects that are timezone
aware.

@example
>>> # dates are easily constructed and formatted
>>> from datetime import date
>>> now = date.today()
>>> now
datetime.date(2003, 12, 2)
>>> now.strftime("%m-%d-%y. %d %b %Y is a %A on the %d day of %B.")
'12-02-03. 02 Dec 2003 is a Tuesday on the 02 day of December.'

>>> # dates support calendar arithmetic
>>> birthday = date(1964, 7, 31)
>>> age = now - birthday
>>> age.days
14368
@end example

@node Data Compression,Performance Measurement,Dates and Times,Brief Tour of the Standard Library
@anchor{tutorial/stdlib data-compression}@anchor{15b3}@anchor{tutorial/stdlib tut-data-compression}@anchor{15b4}
@subsection Data Compression


Common data archiving and compression formats are directly supported by modules
including: @ref{14b,,zlib}, @ref{87,,gzip}, @ref{12,,bz2}, @ref{a9,,lzma}, @ref{149,,zipfile} and
@ref{fb,,tarfile}.

@example
>>> import zlib
>>> s = b'witch which has which witches wrist watch'
>>> len(s)
41
>>> t = zlib.compress(s)
>>> len(t)
37
>>> zlib.decompress(t)
b'witch which has which witches wrist watch'
>>> zlib.crc32(s)
226805979
@end example

@node Performance Measurement,Quality Control,Data Compression,Brief Tour of the Standard Library
@anchor{tutorial/stdlib performance-measurement}@anchor{15b5}@anchor{tutorial/stdlib tut-performance-measurement}@anchor{15b6}
@subsection Performance Measurement


Some Python users develop a deep interest in knowing the relative performance of
different approaches to the same problem. Python provides a measurement tool
that answers those questions immediately.

For example, it may be tempting to use the tuple packing and unpacking feature
instead of the traditional approach to swapping arguments. The @ref{10b,,timeit}
module quickly demonstrates a modest performance advantage:

@example
>>> from timeit import Timer
>>> Timer('t=a; a=b; b=t', 'a=1; b=2').timeit()
0.57535828626024577
>>> Timer('a,b = b,a', 'a=1; b=2').timeit()
0.54962537085770791
@end example

In contrast to @ref{10b,,timeit}’s fine level of granularity, the @ref{ce,,profile} and
@ref{cf,,pstats} modules provide tools for identifying time critical sections in
larger blocks of code.

@node Quality Control,Batteries Included,Performance Measurement,Brief Tour of the Standard Library
@anchor{tutorial/stdlib quality-control}@anchor{15b7}@anchor{tutorial/stdlib tut-quality-control}@anchor{15b8}
@subsection Quality Control


One approach for developing high quality software is to write tests for each
function as it is developed and to run those tests frequently during the
development process.

The @ref{63,,doctest} module provides a tool for scanning a module and validating
tests embedded in a program’s docstrings.  Test construction is as simple as
cutting-and-pasting a typical call along with its results into the docstring.
This improves the documentation by providing the user with an example and it
allows the doctest module to make sure the code remains true to the
documentation:

@example
def average(values):
    """Computes the arithmetic mean of a list of numbers.

    >>> print(average([20, 30, 70]))
    40.0
    """
    return sum(values) / len(values)

import doctest
doctest.testmod()   # automatically validate the embedded tests
@end example

The @ref{122,,unittest} module is not as effortless as the @ref{63,,doctest} module,
but it allows a more comprehensive set of tests to be maintained in a separate
file:

@example
import unittest

class TestStatisticalFunctions(unittest.TestCase):

    def test_average(self):
        self.assertEqual(average([20, 30, 70]), 40.0)
        self.assertEqual(round(average([1, 5, 7]), 1), 4.3)
        with self.assertRaises(ZeroDivisionError):
            average([])
        with self.assertRaises(TypeError):
            average(20, 30, 70)

unittest.main()  # Calling from the command line invokes all tests
@end example

@node Batteries Included,,Quality Control,Brief Tour of the Standard Library
@anchor{tutorial/stdlib batteries-included}@anchor{15b9}@anchor{tutorial/stdlib tut-batteries-included}@anchor{15ba}
@subsection Batteries Included


Python has a “batteries included” philosophy.  This is best seen through the
sophisticated and robust capabilities of its larger packages. For example:


@itemize *

@item 
The @ref{146,,xmlrpc.client} and @ref{147,,xmlrpc.server} modules make implementing
remote procedure calls into an almost trivial task.  Despite the modules
names, no direct knowledge or handling of XML is needed.

@item 
The @ref{64,,email} package is a library for managing email messages, including
MIME and other 
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html}-based message documents. Unlike @ref{e8,,smtplib} and
@ref{cb,,poplib} which actually send and receive messages, the email package has
a complete toolset for building or decoding complex message structures
(including attachments) and for implementing internet encoding and header
protocols.

@item 
The @ref{a0,,json} package provides robust support for parsing this
popular data interchange format.  The @ref{28,,csv} module supports
direct reading and writing of files in Comma-Separated Value format,
commonly supported by databases and spreadsheets.  XML processing is
supported by the @ref{13e,,xml.etree.ElementTree}, @ref{13b,,xml.dom} and
@ref{142,,xml.sax} packages. Together, these modules and packages
greatly simplify data interchange between Python applications and
other tools.

@item 
The @ref{ed,,sqlite3} module is a wrapper for the SQLite database
library, providing a persistent database that can be updated and
accessed using slightly nonstandard SQL syntax.

@item 
Internationalization is supported by a number of modules including
@ref{83,,gettext}, @ref{a5,,locale}, and the @ref{1a,,codecs} package.
@end itemize

@node Brief Tour of the Standard Library — Part II,Virtual Environments and Packages,Brief Tour of the Standard Library,The Python Tutorial
@anchor{tutorial/stdlib2 doc}@anchor{15bb}@anchor{tutorial/stdlib2 brief-tour-of-the-standard-library-part-ii}@anchor{15bc}@anchor{tutorial/stdlib2 tut-brieftourtwo}@anchor{15bd}
@section Brief Tour of the Standard Library — Part II


This second tour covers more advanced modules that support professional
programming needs.  These modules rarely occur in small scripts.

@menu
* Output Formatting:: 
* Templating:: 
* Working with Binary Data Record Layouts:: 
* Multi-threading: Multi-threading<2>. 
* Logging:: 
* Weak References:: 
* Tools for Working with Lists:: 
* Decimal Floating Point Arithmetic:: 

@end menu

@node Output Formatting,Templating,,Brief Tour of the Standard Library — Part II
@anchor{tutorial/stdlib2 output-formatting}@anchor{15be}@anchor{tutorial/stdlib2 tut-output-formatting}@anchor{15bf}
@subsection Output Formatting


The @ref{da,,reprlib} module provides a version of @ref{27d,,repr()} customized for
abbreviated displays of large or deeply nested containers:

@example
>>> import reprlib
>>> reprlib.repr(set('supercalifragilisticexpialidocious'))
"@{'a', 'c', 'd', 'e', 'f', 'g', ...@}"
@end example

The @ref{cd,,pprint} module offers more sophisticated control over printing both
built-in and user defined objects in a way that is readable by the interpreter.
When the result is longer than one line, the “pretty printer” adds line breaks
and indentation to more clearly reveal data structure:

@example
>>> import pprint
>>> t = [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta',
...     'yellow'], 'blue']]]
...
>>> pprint.pprint(t, width=30)
[[[['black', 'cyan'],
   'white',
   ['green', 'red']],
  [['magenta', 'yellow'],
   'blue']]]
@end example

The @ref{108,,textwrap} module formats paragraphs of text to fit a given screen
width:

@example
>>> import textwrap
>>> doc = """The wrap() method is just like fill() except that it returns
... a list of strings instead of one big string with newlines to separate
... the wrapped lines."""
...
>>> print(textwrap.fill(doc, width=40))
The wrap() method is just like fill()
except that it returns a list of strings
instead of one big string with newlines
to separate the wrapped lines.
@end example

The @ref{a5,,locale} module accesses a database of culture specific data formats.
The grouping attribute of locale’s format function provides a direct way of
formatting numbers with group separators:

@example
>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'English_United States.1252')
'English_United States.1252'
>>> conv = locale.localeconv()          # get a mapping of conventions
>>> x = 1234567.8
>>> locale.format("%d", x, grouping=True)
'1,234,567'
>>> locale.format_string("%s%.*f", (conv['currency_symbol'],
...                      conv['frac_digits'], x), grouping=True)
'$1,234,567.80'
@end example

@node Templating,Working with Binary Data Record Layouts,Output Formatting,Brief Tour of the Standard Library — Part II
@anchor{tutorial/stdlib2 templating}@anchor{15c0}@anchor{tutorial/stdlib2 tut-templating}@anchor{15c1}
@subsection Templating


The @ref{f1,,string} module includes a versatile @ref{6b5,,Template} class
with a simplified syntax suitable for editing by end-users.  This allows users
to customize their applications without having to alter the application.

The format uses placeholder names formed by @code{$} with valid Python identifiers
(alphanumeric characters and underscores).  Surrounding the placeholder with
braces allows it to be followed by more alphanumeric letters with no intervening
spaces.  Writing @code{$$} creates a single escaped @code{$}:

@example
>>> from string import Template
>>> t = Template('$@{village@}folk send $$10 to $cause.')
>>> t.substitute(village='Nottingham', cause='the ditch fund')
'Nottinghamfolk send $10 to the ditch fund.'
@end example

The @ref{15c2,,substitute()} method raises a @ref{5bb,,KeyError} when a
placeholder is not supplied in a dictionary or a keyword argument.  For
mail-merge style applications, user supplied data may be incomplete and the
@ref{15c3,,safe_substitute()} method may be more appropriate —
it will leave placeholders unchanged if data is missing:

@example
>>> t = Template('Return the $item to $owner.')
>>> d = dict(item='unladen swallow')
>>> t.substitute(d)
Traceback (most recent call last):
  ...
KeyError: 'owner'
>>> t.safe_substitute(d)
'Return the unladen swallow to $owner.'
@end example

Template subclasses can specify a custom delimiter.  For example, a batch
renaming utility for a photo browser may elect to use percent signs for
placeholders such as the current date, image sequence number, or file format:

@example
>>> import time, os.path
>>> photofiles = ['img_1074.jpg', 'img_1076.jpg', 'img_1077.jpg']
>>> class BatchRename(Template):
...     delimiter = '%'
>>> fmt = input('Enter rename style (%d-date %n-seqnum %f-format):  ')
Enter rename style (%d-date %n-seqnum %f-format):  Ashley_%n%f

>>> t = BatchRename(fmt)
>>> date = time.strftime('%d%b%y')
>>> for i, filename in enumerate(photofiles):
...     base, ext = os.path.splitext(filename)
...     newname = t.substitute(d=date, n=i, f=ext)
...     print('@{0@} --> @{1@}'.format(filename, newname))

img_1074.jpg --> Ashley_0.jpg
img_1076.jpg --> Ashley_1.jpg
img_1077.jpg --> Ashley_2.jpg
@end example

Another application for templating is separating program logic from the details
of multiple output formats.  This makes it possible to substitute custom
templates for XML files, plain text reports, and HTML web reports.

@node Working with Binary Data Record Layouts,Multi-threading<2>,Templating,Brief Tour of the Standard Library — Part II
@anchor{tutorial/stdlib2 tut-binary-formats}@anchor{15c4}@anchor{tutorial/stdlib2 working-with-binary-data-record-layouts}@anchor{15c5}
@subsection Working with Binary Data Record Layouts


The @ref{f3,,struct} module provides @ref{e25,,pack()} and
@ref{15c6,,unpack()} functions for working with variable length binary
record formats.  The following example shows
how to loop through header information in a ZIP file without using the
@ref{149,,zipfile} module.  Pack codes @code{"H"} and @code{"I"} represent two and four
byte unsigned numbers respectively.  The @code{"<"} indicates that they are
standard size and in little-endian byte order:

@example
import struct

with open('myfile.zip', 'rb') as f:
    data = f.read()

start = 0
for i in range(3):                      # show the first 3 file headers
    start += 14
    fields = struct.unpack('<IIIHH', data[start:start+16])
    crc32, comp_size, uncomp_size, filenamesize, extra_size = fields

    start += 16
    filename = data[start:start+filenamesize]
    start += filenamesize
    extra = data[start:start+extra_size]
    print(filename, hex(crc32), comp_size, uncomp_size)

    start += extra_size + comp_size     # skip to the next header
@end example

@node Multi-threading<2>,Logging,Working with Binary Data Record Layouts,Brief Tour of the Standard Library — Part II
@anchor{tutorial/stdlib2 multi-threading}@anchor{15c7}@anchor{tutorial/stdlib2 tut-multi-threading}@anchor{15c8}
@subsection Multi-threading


Threading is a technique for decoupling tasks which are not sequentially
dependent.  Threads can be used to improve the responsiveness of applications
that accept user input while other tasks run in the background.  A related use
case is running I/O in parallel with computations in another thread.

The following code shows how the high level @ref{109,,threading} module can run
tasks in background while the main program continues to run:

@example
import threading, zipfile

class AsyncZip(threading.Thread):
    def __init__(self, infile, outfile):
        threading.Thread.__init__(self)
        self.infile = infile
        self.outfile = outfile

    def run(self):
        f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED)
        f.write(self.infile)
        f.close()
        print('Finished background zip of:', self.infile)

background = AsyncZip('mydata.txt', 'myarchive.zip')
background.start()
print('The main program continues to run in foreground.')

background.join()    # Wait for the background task to finish
print('Main program waited until background was done.')
@end example

The principal challenge of multi-threaded applications is coordinating threads
that share data or other resources.  To that end, the threading module provides
a number of synchronization primitives including locks, events, condition
variables, and semaphores.

While those tools are powerful, minor design errors can result in problems that
are difficult to reproduce.  So, the preferred approach to task coordination is
to concentrate all access to a resource in a single thread and then use the
@ref{d5,,queue} module to feed that thread with requests from other threads.
Applications using @ref{f18,,Queue} objects for inter-thread communication and
coordination are easier to design, more readable, and more reliable.

@node Logging,Weak References,Multi-threading<2>,Brief Tour of the Standard Library — Part II
@anchor{tutorial/stdlib2 logging}@anchor{15c9}@anchor{tutorial/stdlib2 tut-logging}@anchor{15ca}
@subsection Logging


The @ref{a6,,logging} module offers a full featured and flexible logging system.
At its simplest, log messages are sent to a file or to @code{sys.stderr}:

@example
import logging
logging.debug('Debugging information')
logging.info('Informational message')
logging.warning('Warning:config file %s not found', 'server.conf')
logging.error('Error occurred')
logging.critical('Critical error -- shutting down')
@end example

This produces the following output:

@example
WARNING:root:Warning:config file server.conf not found
ERROR:root:Error occurred
CRITICAL:root:Critical error -- shutting down
@end example

By default, informational and debugging messages are suppressed and the output
is sent to standard error.  Other output options include routing messages
through email, datagrams, sockets, or to an HTTP Server.  New filters can select
different routing based on message priority: @code{DEBUG},
@code{INFO}, @code{WARNING}, @code{ERROR},
and @code{CRITICAL}.

The logging system can be configured directly from Python or can be loaded from
a user editable configuration file for customized logging without altering the
application.

@node Weak References,Tools for Working with Lists,Logging,Brief Tour of the Standard Library — Part II
@anchor{tutorial/stdlib2 tut-weak-references}@anchor{15cb}@anchor{tutorial/stdlib2 weak-references}@anchor{15cc}
@subsection Weak References


Python does automatic memory management (reference counting for most objects and
@ref{1192,,garbage collection} to eliminate cycles).  The memory is freed shortly
after the last reference to it has been eliminated.

This approach works fine for most applications but occasionally there is a need
to track objects only as long as they are being used by something else.
Unfortunately, just tracking them creates a reference that makes them permanent.
The @ref{12f,,weakref} module provides tools for tracking objects without creating a
reference.  When the object is no longer needed, it is automatically removed
from a weakref table and a callback is triggered for weakref objects.  Typical
applications include caching objects that are expensive to create:

@example
>>> import weakref, gc
>>> class A:
...     def __init__(self, value):
...         self.value = value
...     def __repr__(self):
...         return str(self.value)
...
>>> a = A(10)                   # create a reference
>>> d = weakref.WeakValueDictionary()
>>> d['primary'] = a            # does not create a reference
>>> d['primary']                # fetch the object if it is still alive
10
>>> del a                       # remove the one reference
>>> gc.collect()                # run garbage collection right away
0
>>> d['primary']                # entry was automatically removed
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    d['primary']                # entry was automatically removed
  File "C:/python311/lib/weakref.py", line 46, in __getitem__
    o = self.data[key]()
KeyError: 'primary'
@end example

@node Tools for Working with Lists,Decimal Floating Point Arithmetic,Weak References,Brief Tour of the Standard Library — Part II
@anchor{tutorial/stdlib2 tools-for-working-with-lists}@anchor{15cd}@anchor{tutorial/stdlib2 tut-list-tools}@anchor{15ce}
@subsection Tools for Working with Lists


Many data structure needs can be met with the built-in list type. However,
sometimes there is a need for alternative implementations with different
performance trade-offs.

The @ref{6,,array} module provides an @ref{25e,,array()} object that is like
a list that stores only homogeneous data and stores it more compactly.  The
following example shows an array of numbers stored as two byte unsigned binary
numbers (typecode @code{"H"}) rather than the usual 16 bytes per entry for regular
lists of Python int objects:

@example
>>> from array import array
>>> a = array('H', [4000, 10, 700, 22222])
>>> sum(a)
26932
>>> a[1:3]
array('H', [10, 700])
@end example

The @ref{1c,,collections} module provides a @ref{7d3,,deque()} object
that is like a list with faster appends and pops from the left side but slower
lookups in the middle. These objects are well suited for implementing queues
and breadth first tree searches:

@example
>>> from collections import deque
>>> d = deque(["task1", "task2", "task3"])
>>> d.append("task4")
>>> print("Handling", d.popleft())
Handling task1
@end example

@example
unsearched = deque([starting_node])
def breadth_first_search(unsearched):
    node = unsearched.popleft()
    for m in gen_moves(node):
        if is_goal(m):
            return m
        unsearched.append(m)
@end example

In addition to alternative list implementations, the library also offers other
tools such as the @ref{10,,bisect} module with functions for manipulating sorted
lists:

@example
>>> import bisect
>>> scores = [(100, 'perl'), (200, 'tcl'), (400, 'lua'), (500, 'python')]
>>> bisect.insort(scores, (300, 'ruby'))
>>> scores
[(100, 'perl'), (200, 'tcl'), (300, 'ruby'), (400, 'lua'), (500, 'python')]
@end example

The @ref{89,,heapq} module provides functions for implementing heaps based on
regular lists.  The lowest valued entry is always kept at position zero.  This
is useful for applications which repeatedly access the smallest element but do
not want to run a full list sort:

@example
>>> from heapq import heapify, heappop, heappush
>>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0]
>>> heapify(data)                      # rearrange the list into heap order
>>> heappush(data, -5)                 # add a new entry
>>> [heappop(data) for i in range(3)]  # fetch the three smallest entries
[-5, 0, 1]
@end example

@node Decimal Floating Point Arithmetic,,Tools for Working with Lists,Brief Tour of the Standard Library — Part II
@anchor{tutorial/stdlib2 decimal-floating-point-arithmetic}@anchor{15cf}@anchor{tutorial/stdlib2 tut-decimal-fp}@anchor{15d0}
@subsection Decimal Floating Point Arithmetic


The @ref{34,,decimal} module offers a @ref{248,,Decimal} datatype for
decimal floating point arithmetic.  Compared to the built-in @ref{256,,float}
implementation of binary floating point, the class is especially helpful for


@itemize *

@item 
financial applications and other uses which require exact decimal
representation,

@item 
control over precision,

@item 
control over rounding to meet legal or regulatory requirements,

@item 
tracking of significant decimal places, or

@item 
applications where the user expects the results to match calculations done by
hand.
@end itemize

For example, calculating a 5% tax on a 70 cent phone charge gives different
results in decimal floating point and binary floating point. The difference
becomes significant if the results are rounded to the nearest cent:

@example
>>> from decimal import *
>>> round(Decimal('0.70') * Decimal('1.05'), 2)
Decimal('0.74')
>>> round(.70 * 1.05, 2)
0.73
@end example

The @ref{248,,Decimal} result keeps a trailing zero, automatically
inferring four place significance from multiplicands with two place
significance.  Decimal reproduces mathematics as done by hand and avoids
issues that can arise when binary floating point cannot exactly represent
decimal quantities.

Exact representation enables the @ref{248,,Decimal} class to perform
modulo calculations and equality tests that are unsuitable for binary floating
point:

@example
>>> Decimal('1.00') % Decimal('.10')
Decimal('0.00')
>>> 1.00 % 0.10
0.09999999999999995

>>> sum([Decimal('0.1')]*10) == Decimal('1.0')
True
>>> sum([0.1]*10) == 1.0
False
@end example

The @ref{34,,decimal} module provides arithmetic with as much precision as needed:

@example
>>> getcontext().prec = 36
>>> Decimal(1) / Decimal(7)
Decimal('0.142857142857142857142857142857142857')
@end example

@node Virtual Environments and Packages,What Now?,Brief Tour of the Standard Library — Part II,The Python Tutorial
@anchor{tutorial/venv doc}@anchor{15d1}@anchor{tutorial/venv tut-venv}@anchor{15d2}@anchor{tutorial/venv virtual-environments-and-packages}@anchor{15d3}
@section Virtual Environments and Packages


@menu
* Introduction: Introduction<4>. 
* Creating Virtual Environments:: 
* Managing Packages with pip:: 

@end menu

@node Introduction<4>,Creating Virtual Environments,,Virtual Environments and Packages
@anchor{tutorial/venv introduction}@anchor{15d4}
@subsection Introduction


Python applications will often use packages and modules that don’t
come as part of the standard library.  Applications will sometimes
need a specific version of a library, because the application may
require that a particular bug has been fixed or the application may be
written using an obsolete version of the library’s interface.

This means it may not be possible for one Python installation to meet
the requirements of every application.  If application A needs version
1.0 of a particular module but application B needs version 2.0, then
the requirements are in conflict and installing either version 1.0 or 2.0
will leave one application unable to run.

The solution for this problem is to create a @ref{15d5,,virtual environment}, a
self-contained directory tree that contains a Python installation for a
particular version of Python, plus a number of additional packages.

Different applications can then use different virtual environments.
To resolve the earlier example of conflicting requirements,
application A can have its own virtual environment with version 1.0
installed while application B has another virtual environment with version 2.0.
If application B requires a library be upgraded to version 3.0, this will
not affect application A’s environment.

@node Creating Virtual Environments,Managing Packages with pip,Introduction<4>,Virtual Environments and Packages
@anchor{tutorial/venv creating-virtual-environments}@anchor{15d6}
@subsection Creating Virtual Environments


The module used to create and manage virtual environments is called
@ref{12c,,venv}.  @ref{12c,,venv} will usually install the most recent version of
Python that you have available. If you have multiple versions of Python on your
system, you can select a specific Python version by running @code{python3} or
whichever version you want.

To create a virtual environment, decide upon a directory where you want to
place it, and run the @ref{12c,,venv} module as a script with the directory path:

@example
python3 -m venv tutorial-env
@end example

This will create the @code{tutorial-env} directory if it doesn’t exist,
and also create directories inside it containing a copy of the Python
interpreter and various supporting files.

A common directory location for a virtual environment is @code{.venv}.
This name keeps the directory typically hidden in your shell and thus
out of the way while giving it a name that explains why the directory
exists. It also prevents clashing with @code{.env} environment variable
definition files that some tooling supports.

Once you’ve created a virtual environment, you may activate it.

On Windows, run:

@example
tutorial-env\Scripts\activate.bat
@end example

On Unix or MacOS, run:

@example
source tutorial-env/bin/activate
@end example

(This script is written for the bash shell.  If you use the
@code{csh} or @code{fish} shells, there are alternate
@code{activate.csh} and @code{activate.fish} scripts you should use
instead.)

Activating the virtual environment will change your shell’s prompt to show what
virtual environment you’re using, and modify the environment so that running
@code{python} will get you that particular version and installation of Python.
For example:

@example
$ source ~/envs/tutorial-env/bin/activate
(tutorial-env) $ python
Python 3.5.1 (default, May  6 2016, 10:59:36)
  ...
>>> import sys
>>> sys.path
['@w{'}, '/usr/local/lib/python35.zip', ...,
'~/envs/tutorial-env/lib/python3.5/site-packages']
>>>
@end example

To deactivate a virtual environment, type:

@example
deactivate
@end example

into the terminal.

@node Managing Packages with pip,,Creating Virtual Environments,Virtual Environments and Packages
@anchor{tutorial/venv managing-packages-with-pip}@anchor{15d7}
@subsection Managing Packages with pip


You can install, upgrade, and remove packages using a program called
@code{pip}.  By default @code{pip} will install packages from the Python
Package Index, <@indicateurl{https://pypi.org}>.  You can browse the Python
Package Index by going to it in your web browser.

@code{pip} has a number of subcommands: “install”, “uninstall”,
“freeze”, etc.  (Consult the @ref{a69,,Installing Python Modules} guide for
complete documentation for @code{pip}.)

You can install the latest version of a package by specifying a package’s name:

@example
(tutorial-env) $ python -m pip install novas
Collecting novas
  Downloading novas-3.1.1.3.tar.gz (136kB)
Installing collected packages: novas
  Running setup.py install for novas
Successfully installed novas-3.1.1.3
@end example

You can also install a specific version of a package by giving the
package name  followed by @code{==} and the version number:

@example
(tutorial-env) $ python -m pip install requests==2.6.0
Collecting requests==2.6.0
  Using cached requests-2.6.0-py2.py3-none-any.whl
Installing collected packages: requests
Successfully installed requests-2.6.0
@end example

If you re-run this command, @code{pip} will notice that the requested
version is already installed and do nothing.  You can supply a
different version number to get that version, or you can run @code{pip
install --upgrade} to upgrade the package to the latest version:

@example
(tutorial-env) $ python -m pip install --upgrade requests
Collecting requests
Installing collected packages: requests
  Found existing installation: requests 2.6.0
    Uninstalling requests-2.6.0:
      Successfully uninstalled requests-2.6.0
Successfully installed requests-2.7.0
@end example

@code{pip uninstall} followed by one or more package names will remove the
packages from the virtual environment.

@code{pip show} will display information about a particular package:

@example
(tutorial-env) $ pip show requests
---
Metadata-Version: 2.0
Name: requests
Version: 2.7.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: me@@kennethreitz.com
License: Apache 2.0
Location: /Users/akuchling/envs/tutorial-env/lib/python3.4/site-packages
Requires:
@end example

@code{pip list} will display all of the packages installed in the virtual
environment:

@example
(tutorial-env) $ pip list
novas (3.1.1.3)
numpy (1.9.2)
pip (7.0.3)
requests (2.7.0)
setuptools (16.0)
@end example

@code{pip freeze} will produce a similar list of the installed packages,
but the output uses the format that @code{pip install} expects.
A common convention is to put this list in a @code{requirements.txt} file:

@example
(tutorial-env) $ pip freeze > requirements.txt
(tutorial-env) $ cat requirements.txt
novas==3.1.1.3
numpy==1.9.2
requests==2.7.0
@end example

The @code{requirements.txt} can then be committed to version control and
shipped as part of an application.  Users can then install all the
necessary packages with @code{install -r}:

@example
(tutorial-env) $ python -m pip install -r requirements.txt
Collecting novas==3.1.1.3 (from -r requirements.txt (line 1))
  ...
Collecting numpy==1.9.2 (from -r requirements.txt (line 2))
  ...
Collecting requests==2.7.0 (from -r requirements.txt (line 3))
  ...
Installing collected packages: novas, numpy, requests
  Running setup.py install for novas
Successfully installed novas-3.1.1.3 numpy-1.9.2 requests-2.7.0
@end example

@code{pip} has many more options.  Consult the @ref{a69,,Installing Python Modules}
guide for complete documentation for @code{pip}.  When you’ve written
a package and want to make it available on the Python Package Index,
consult the @ref{a6a,,Distributing Python Modules} guide.

@node What Now?,Interactive Input Editing and History Substitution,Virtual Environments and Packages,The Python Tutorial
@anchor{tutorial/whatnow doc}@anchor{15d8}@anchor{tutorial/whatnow tut-whatnow}@anchor{15d9}@anchor{tutorial/whatnow what-now}@anchor{15da}
@section What Now?


Reading this tutorial has probably reinforced your interest in using Python —
you should be eager to apply Python to solving your real-world problems. Where
should you go to learn more?

This tutorial is part of Python’s documentation set.   Some other documents in
the set are:


@itemize *

@item 
@ref{14bc,,The Python Standard Library}:

You should browse through this manual, which gives complete (though terse)
reference material about types, functions, and the modules in the standard
library.  The standard Python distribution includes a @emph{lot} of additional code.
There are modules to read Unix mailboxes, retrieve documents via HTTP, generate
random numbers, parse command-line options, write CGI programs, compress data,
and many other tasks. Skimming through the Library Reference will give you an
idea of what’s available.

@item 
@ref{a69,,Installing Python Modules} explains how to install additional modules written
by other Python users.

@item 
@ref{14bd,,The Python Language Reference}: A detailed explanation of Python’s syntax and
semantics.  It’s heavy reading, but is useful as a complete guide to the
language itself.
@end itemize

More Python resources:


@itemize *

@item 
@indicateurl{https://www.python.org}:  The major Python web site.  It contains code,
documentation, and pointers to Python-related pages around the web.  This web
site is mirrored in various places around the world, such as Europe, Japan, and
Australia; a mirror may be faster than the main site, depending on your
geographical location.

@item 
@indicateurl{https://docs.python.org}:  Fast access to Python’s  documentation.

@item 
@indicateurl{https://pypi.org}: The Python Package Index, previously also nicknamed
the Cheese Shop @footnote{@w{(1)} 
“Cheese Shop” is a Monty Python’s sketch: a customer enters a cheese shop,
but whatever cheese he asks for, the clerk says it’s missing.
}, is an index of user-created Python modules that are available
for download.  Once you begin releasing code, you can register it here so that
others can find it.

@item 
@indicateurl{https://code.activestate.com/recipes/langs/python/}: The Python Cookbook is a
sizable collection of code examples, larger modules, and useful scripts.
Particularly notable contributions are collected in a book also titled Python
Cookbook (O’Reilly & Associates, ISBN 0-596-00797-3.)

@item 
@indicateurl{http://www.pyvideo.org} collects links to Python-related videos from
conferences and user-group meetings.

@item 
@indicateurl{https://scipy.org}: The Scientific Python project includes modules for fast
array computations and manipulations plus a host of packages for such
things as linear algebra, Fourier transforms, non-linear solvers,
random number distributions, statistical analysis and the like.
@end itemize

For Python-related questions and problem reports, you can post to the newsgroup
@code{comp.lang.python}, or send them to the mailing list at
@email{python-list@@python.org}.  The newsgroup and mailing list are gatewayed, so
messages posted to one will automatically be forwarded to the other.  There are
hundreds of postings a day, asking (and
answering) questions, suggesting new features, and announcing new modules.
Mailing list archives are available at @indicateurl{https://mail.python.org/pipermail/}.

Before posting, be sure to check the list of
@ref{15db,,Frequently Asked Questions} (also called the FAQ).  The
FAQ answers many of the questions that come up again and again, and may
already contain the solution for your problem.

@node Interactive Input Editing and History Substitution,Floating Point Arithmetic Issues and Limitations,What Now?,The Python Tutorial
@anchor{tutorial/interactive doc}@anchor{15dc}@anchor{tutorial/interactive interactive-input-editing-and-history-substitution}@anchor{15dd}@anchor{tutorial/interactive tut-interacting}@anchor{14cb}
@section Interactive Input Editing and History Substitution


Some versions of the Python interpreter support editing of the current input
line and history substitution, similar to facilities found in the Korn shell and
the GNU Bash shell.  This is implemented using the GNU Readline@footnote{https://tiswww.case.edu/php/chet/readline/rltop.html} library,
which supports various styles of editing.  This library has its own
documentation which we won’t duplicate here.

@menu
* Tab Completion and History Editing:: 
* Alternatives to the Interactive Interpreter:: 

@end menu

@node Tab Completion and History Editing,Alternatives to the Interactive Interpreter,,Interactive Input Editing and History Substitution
@anchor{tutorial/interactive tab-completion-and-history-editing}@anchor{15de}@anchor{tutorial/interactive tut-keybindings}@anchor{15df}
@subsection Tab Completion and History Editing


Completion of variable and module names is
@ref{b4e,,automatically enabled} at interpreter startup so
that the @code{Tab} key invokes the completion function; it looks at
Python statement names, the current local variables, and the available
module names.  For dotted expressions such as @code{string.a}, it will evaluate
the expression up to the final @code{'.'} and then suggest completions from
the attributes of the resulting object.  Note that this may execute
application-defined code if an object with a @code{__getattr__()} method
is part of the expression.  The default configuration also saves your
history into a file named @code{.python_history} in your user directory.
The history will be available again during the next interactive interpreter
session.

@node Alternatives to the Interactive Interpreter,,Tab Completion and History Editing,Interactive Input Editing and History Substitution
@anchor{tutorial/interactive alternatives-to-the-interactive-interpreter}@anchor{15e0}@anchor{tutorial/interactive tut-commentary}@anchor{15e1}
@subsection Alternatives to the Interactive Interpreter


This facility is an enormous step forward compared to earlier versions of the
interpreter; however, some wishes are left: It would be nice if the proper
indentation were suggested on continuation lines (the parser knows if an indent
token is required next).  The completion mechanism might use the interpreter’s
symbol table.  A command to check (or even suggest) matching parentheses,
quotes, etc., would also be useful.

One alternative enhanced interactive interpreter that has been around for quite
some time is IPython@footnote{https://ipython.org/}, which features tab completion, object exploration and
advanced history management.  It can also be thoroughly customized and embedded
into other applications.  Another similar enhanced interactive environment is
bpython@footnote{https://www.bpython-interpreter.org/}.

@c import math

@node Floating Point Arithmetic Issues and Limitations,Appendix,Interactive Input Editing and History Substitution,The Python Tutorial
@anchor{tutorial/floatingpoint doc}@anchor{15e2}@anchor{tutorial/floatingpoint bpython}@anchor{15e3}@anchor{tutorial/floatingpoint floating-point-arithmetic-issues-and-limitations}@anchor{15e4}@anchor{tutorial/floatingpoint tut-fp-issues}@anchor{15e5}
@section Floating Point Arithmetic:  Issues and Limitations


Floating-point numbers are represented in computer hardware as base 2 (binary)
fractions.  For example, the @strong{decimal} fraction @code{0.125}
has value 1/10 + 2/100 + 5/1000, and in the same way the @strong{binary} fraction @code{0.001}
has value 0/2 + 0/4 + 1/8. These two fractions have identical values, the only
real difference being that the first is written in base 10 fractional notation,
and the second in base 2.

Unfortunately, most decimal fractions cannot be represented exactly as binary
fractions.  A consequence is that, in general, the decimal floating-point
numbers you enter are only approximated by the binary floating-point numbers
actually stored in the machine.

The problem is easier to understand at first in base 10.  Consider the fraction
1/3.  You can approximate that as a base 10 fraction:

@example
0.3
@end example

or, better,

@example
0.33
@end example

or, better,

@example
0.333
@end example

and so on.  No matter how many digits you’re willing to write down, the result
will never be exactly 1/3, but will be an increasingly better approximation of
1/3.

In the same way, no matter how many base 2 digits you’re willing to use, the
decimal value 0.1 cannot be represented exactly as a base 2 fraction.  In base
2, 1/10 is the infinitely repeating fraction

@example
0.0001100110011001100110011001100110011001100110011...
@end example

Stop at any finite number of bits, and you get an approximation.  On most
machines today, floats are approximated using a binary fraction with
the numerator using the first 53 bits starting with the most significant bit and
with the denominator as a power of two.  In the case of 1/10, the binary fraction
is @code{3602879701896397 / 2 ** 55} which is close to but not exactly
equal to the true value of 1/10.

Many users are not aware of the approximation because of the way values are
displayed.  Python only prints a decimal approximation to the true decimal
value of the binary approximation stored by the machine.  On most machines, if
Python were to print the true decimal value of the binary approximation stored
for 0.1, it would have to display

@example
>>> 0.1
0.1000000000000000055511151231257827021181583404541015625
@end example

That is more digits than most people find useful, so Python keeps the number
of digits manageable by displaying a rounded value instead

@example
>>> 1 / 10
0.1
@end example

Just remember, even though the printed result looks like the exact value
of 1/10, the actual stored value is the nearest representable binary fraction.

Interestingly, there are many different decimal numbers that share the same
nearest approximate binary fraction.  For example, the numbers @code{0.1} and
@code{0.10000000000000001} and
@code{0.1000000000000000055511151231257827021181583404541015625} are all
approximated by @code{3602879701896397 / 2 ** 55}.  Since all of these decimal
values share the same approximation, any one of them could be displayed
while still preserving the invariant @code{eval(repr(x)) == x}.

Historically, the Python prompt and built-in @ref{27d,,repr()} function would choose
the one with 17 significant digits, @code{0.10000000000000001}.   Starting with
Python 3.1, Python (on most systems) is now able to choose the shortest of
these and simply display @code{0.1}.

Note that this is in the very nature of binary floating-point: this is not a bug
in Python, and it is not a bug in your code either.  You’ll see the same kind of
thing in all languages that support your hardware’s floating-point arithmetic
(although some languages may not @emph{display} the difference by default, or in all
output modes).

For more pleasant output, you may wish to use string formatting to produce a limited number of significant digits:

@example
>>> format(math.pi, '.12g')  # give 12 significant digits
'3.14159265359'

>>> format(math.pi, '.2f')   # give 2 digits after the point
'3.14'

>>> repr(math.pi)
'3.141592653589793'
@end example

It’s important to realize that this is, in a real sense, an illusion: you’re
simply rounding the @emph{display} of the true machine value.

One illusion may beget another.  For example, since 0.1 is not exactly 1/10,
summing three values of 0.1 may not yield exactly 0.3, either:

@example
>>> .1 + .1 + .1 == .3
False
@end example

Also, since the 0.1 cannot get any closer to the exact value of 1/10 and
0.3 cannot get any closer to the exact value of 3/10, then pre-rounding with
@ref{e75,,round()} function cannot help:

@example
>>> round(.1, 1) + round(.1, 1) + round(.1, 1) == round(.3, 1)
False
@end example

Though the numbers cannot be made closer to their intended exact values,
the @ref{e75,,round()} function can be useful for post-rounding so that results
with inexact values become comparable to one another:

@example
>>> round(.1 + .1 + .1, 10) == round(.3, 10)
True
@end example

Binary floating-point arithmetic holds many surprises like this.  The problem
with “0.1” is explained in precise detail below, in the “Representation Error”
section.  See The Perils of Floating Point@footnote{https://www.lahey.com/float.htm}
for a more complete account of other common surprises.

As that says near the end, “there are no easy answers.”  Still, don’t be unduly
wary of floating-point!  The errors in Python float operations are inherited
from the floating-point hardware, and on most machines are on the order of no
more than 1 part in 2**53 per operation.  That’s more than adequate for most
tasks, but you do need to keep in mind that it’s not decimal arithmetic and
that every float operation can suffer a new rounding error.

While pathological cases do exist, for most casual use of floating-point
arithmetic you’ll see the result you expect in the end if you simply round the
display of your final results to the number of decimal digits you expect.
@ref{2ef,,str()} usually suffices, and for finer control see the @ref{784,,str.format()}
method’s format specifiers in @ref{f1a,,Format String Syntax}.

For use cases which require exact decimal representation, try using the
@ref{34,,decimal} module which implements decimal arithmetic suitable for
accounting applications and high-precision applications.

Another form of exact arithmetic is supported by the @ref{7d,,fractions} module
which implements arithmetic based on rational numbers (so the numbers like
1/3 can be represented exactly).

If you are a heavy user of floating point operations you should take a look
at the NumPy package and many other packages for mathematical and
statistical operations supplied by the SciPy project. See <@indicateurl{https://scipy.org}>.

Python provides tools that may help on those rare occasions when you really
@emph{do} want to know the exact value of a float.  The
@ref{15e6,,float.as_integer_ratio()} method expresses the value of a float as a
fraction:

@example
>>> x = 3.14159
>>> x.as_integer_ratio()
(3537115888337719, 1125899906842624)
@end example

Since the ratio is exact, it can be used to losslessly recreate the
original value:

@example
>>> x == 3537115888337719 / 1125899906842624
True
@end example

The @ref{15e7,,float.hex()} method expresses a float in hexadecimal (base
16), again giving the exact value stored by your computer:

@example
>>> x.hex()
'0x1.921f9f01b866ep+1'
@end example

This precise hexadecimal representation can be used to reconstruct
the float value exactly:

@example
>>> x == float.fromhex('0x1.921f9f01b866ep+1')
True
@end example

Since the representation is exact, it is useful for reliably porting values
across different versions of Python (platform independence) and exchanging
data with other languages that support the same format (such as Java and C99).

Another helpful tool is the @ref{f3a,,math.fsum()} function which helps mitigate
loss-of-precision during summation.  It tracks “lost digits” as values are
added onto a running total.  That can make a difference in overall accuracy
so that the errors do not accumulate to the point where they affect the
final total:

@example
>>> sum([0.1] * 10) == 1.0
False
>>> math.fsum([0.1] * 10) == 1.0
True
@end example

@menu
* Representation Error:: 

@end menu

@node Representation Error,,,Floating Point Arithmetic Issues and Limitations
@anchor{tutorial/floatingpoint representation-error}@anchor{15e8}@anchor{tutorial/floatingpoint tut-fp-error}@anchor{15e9}
@subsection Representation Error


This section explains the “0.1” example in detail, and shows how you can perform
an exact analysis of cases like this yourself.  Basic familiarity with binary
floating-point representation is assumed.

@emph{Representation error} refers to the fact that some (most, actually)
decimal fractions cannot be represented exactly as binary (base 2) fractions.
This is the chief reason why Python (or Perl, C, C++, Java, Fortran, and many
others) often won’t display the exact decimal number you expect.

Why is that?  1/10 is not exactly representable as a binary fraction. Almost all
machines today (November 2000) use IEEE-754 floating point arithmetic, and
almost all platforms map Python floats to IEEE-754 “double precision”.  754
doubles contain 53 bits of precision, so on input the computer strives to
convert 0.1 to the closest fraction it can of the form @emph{J}/2**@emph{N} where @emph{J} is
an integer containing exactly 53 bits.  Rewriting

@example
1 / 10 ~= J / (2**N)
@end example

as

@example
J ~= 2**N / 10
@end example

and recalling that @emph{J} has exactly 53 bits (is @code{>= 2**52} but @code{< 2**53}),
the best value for @emph{N} is 56:

@example
>>> 2**52 <=  2**56 // 10  < 2**53
True
@end example

That is, 56 is the only value for @emph{N} that leaves @emph{J} with exactly 53 bits.  The
best possible value for @emph{J} is then that quotient rounded:

@example
>>> q, r = divmod(2**56, 10)
>>> r
6
@end example

Since the remainder is more than half of 10, the best approximation is obtained
by rounding up:

@example
>>> q+1
7205759403792794
@end example

Therefore the best possible approximation to 1/10 in 754 double precision is:

@example
7205759403792794 / 2 ** 56
@end example

Dividing both the numerator and denominator by two reduces the fraction to:

@example
3602879701896397 / 2 ** 55
@end example

Note that since we rounded up, this is actually a little bit larger than 1/10;
if we had not rounded up, the quotient would have been a little bit smaller than
1/10.  But in no case can it be @emph{exactly} 1/10!

So the computer never “sees” 1/10:  what it sees is the exact fraction given
above, the best 754 double approximation it can get:

@example
>>> 0.1 * 2 ** 55
3602879701896397.0
@end example

If we multiply that fraction by 10**55, we can see the value out to
55 decimal digits:

@example
>>> 3602879701896397 * 10 ** 55 // 2 ** 55
1000000000000000055511151231257827021181583404541015625
@end example

meaning that the exact number stored in the computer is equal to
the decimal value 0.1000000000000000055511151231257827021181583404541015625.
Instead of displaying the full decimal value, many languages (including
older versions of Python), round the result to 17 significant digits:

@example
>>> format(0.1, '.17f')
'0.10000000000000001'
@end example

The @ref{7d,,fractions} and @ref{34,,decimal} modules make these calculations
easy:

@example
>>> from decimal import Decimal
>>> from fractions import Fraction

>>> Fraction.from_float(0.1)
Fraction(3602879701896397, 36028797018963968)

>>> (0.1).as_integer_ratio()
(3602879701896397, 36028797018963968)

>>> Decimal.from_float(0.1)
Decimal('0.1000000000000000055511151231257827021181583404541015625')

>>> format(Decimal.from_float(0.1), '.17')
'0.10000000000000001'
@end example

@node Appendix,,Floating Point Arithmetic Issues and Limitations,The Python Tutorial
@anchor{tutorial/appendix doc}@anchor{15ea}@anchor{tutorial/appendix appendix}@anchor{15eb}@anchor{tutorial/appendix tut-appendix}@anchor{15ec}
@section Appendix


@menu
* Interactive Mode: Interactive Mode<2>. 

@end menu

@node Interactive Mode<2>,,,Appendix
@anchor{tutorial/appendix interactive-mode}@anchor{15ed}@anchor{tutorial/appendix tut-interac}@anchor{14cf}
@subsection Interactive Mode


@menu
* Error Handling:: 
* Executable Python Scripts:: 
* The Interactive Startup File:: 
* The Customization Modules:: 

@end menu

@node Error Handling,Executable Python Scripts,,Interactive Mode<2>
@anchor{tutorial/appendix error-handling}@anchor{15ee}@anchor{tutorial/appendix tut-error}@anchor{15ef}
@subsubsection Error Handling


When an error occurs, the interpreter prints an error message and a stack trace.
In interactive mode, it then returns to the primary prompt; when input came from
a file, it exits with a nonzero exit status after printing the stack trace.
(Exceptions handled by an @ref{16e,,except} clause in a @ref{f6d,,try} statement
are not errors in this context.)  Some errors are unconditionally fatal and
cause an exit with a nonzero exit; this applies to internal inconsistencies and
some cases of running out of memory.  All error messages are written to the
standard error stream; normal output from executed commands is written to
standard output.

Typing the interrupt character (usually @code{Control-C} or @code{Delete}) to the primary or
secondary prompt cancels the input and returns to the primary prompt. @footnote{@w{(1)} 
A problem with the GNU Readline package may prevent this.
}
Typing an interrupt while a command is executing raises the
@ref{4ca,,KeyboardInterrupt} exception, which may be handled by a @ref{f6d,,try}
statement.

@node Executable Python Scripts,The Interactive Startup File,Error Handling,Interactive Mode<2>
@anchor{tutorial/appendix executable-python-scripts}@anchor{15f0}@anchor{tutorial/appendix tut-scripts}@anchor{14d4}
@subsubsection Executable Python Scripts


On BSD’ish Unix systems, Python scripts can be made directly executable, like
shell scripts, by putting the line

@example
#!/usr/bin/env python3.5
@end example

(assuming that the interpreter is on the user’s 
@geindex PATH
@geindex environment variable; PATH
@code{PATH}) at the beginning
of the script and giving the file an executable mode.  The @code{#!} must be the
first two characters of the file.  On some platforms, this first line must end
with a Unix-style line ending (@code{'\n'}), not a Windows (@code{'\r\n'}) line
ending.  Note that the hash, or pound, character, @code{'#'}, is used to start a
comment in Python.

The script can be given an executable mode, or permission, using the
@code{chmod} command.

@example
$ chmod +x myscript.py
@end example

On Windows systems, there is no notion of an “executable mode”.  The Python
installer automatically associates @code{.py} files with @code{python.exe} so that
a double-click on a Python file will run it as a script.  The extension can
also be @code{.pyw}, in that case, the console window that normally appears is
suppressed.

@node The Interactive Startup File,The Customization Modules,Executable Python Scripts,Interactive Mode<2>
@anchor{tutorial/appendix the-interactive-startup-file}@anchor{15f1}@anchor{tutorial/appendix tut-startup}@anchor{15f2}
@subsubsection The Interactive Startup File


When you use Python interactively, it is frequently handy to have some standard
commands executed every time the interpreter is started.  You can do this by
setting an environment variable named 
@geindex PYTHONSTARTUP
@geindex environment variable; PYTHONSTARTUP
@ref{b4d,,PYTHONSTARTUP} to the name of a
file containing your start-up commands.  This is similar to the @code{.profile}
feature of the Unix shells.

This file is only read in interactive sessions, not when Python reads commands
from a script, and not when @code{/dev/tty} is given as the explicit source of
commands (which otherwise behaves like an interactive session).  It is executed
in the same namespace where interactive commands are executed, so that objects
that it defines or imports can be used without qualification in the interactive
session. You can also change the prompts @code{sys.ps1} and @code{sys.ps2} in this
file.

If you want to read an additional start-up file from the current directory, you
can program this in the global start-up file using code like @code{if
os.path.isfile('.pythonrc.py'): exec(open('.pythonrc.py').read())}.
If you want to use the startup file in a script, you must do this explicitly
in the script:

@example
import os
filename = os.environ.get('PYTHONSTARTUP')
if filename and os.path.isfile(filename):
    with open(filename) as fobj:
        startup_file = fobj.read()
    exec(startup_file)
@end example

@node The Customization Modules,,The Interactive Startup File,Interactive Mode<2>
@anchor{tutorial/appendix the-customization-modules}@anchor{15f3}@anchor{tutorial/appendix tut-customize}@anchor{15f4}
@subsubsection The Customization Modules


Python provides two hooks to let you customize it: @code{sitecustomize} and
@code{usercustomize}.  To see how it works, you need first to find the location
of your user site-packages directory.  Start Python and run this code:

@example
>>> import site
>>> site.getusersitepackages()
'/home/user/.local/lib/python3.5/site-packages'
@end example

Now you can create a file named @code{usercustomize.py} in that directory and
put anything you want in it.  It will affect every invocation of Python, unless
it is started with the @ref{f15,,-s} option to disable the automatic import.

@code{sitecustomize} works in the same way, but is typically created by an
administrator of the computer in the global site-packages directory, and is
imported before @code{usercustomize}.  See the documentation of the @ref{e6,,site}
module for more details.

@node Python Setup and Usage,The Python Language Reference,The Python Tutorial,Top
@anchor{using/index doc}@anchor{15f5}@anchor{using/index python-setup-and-usage}@anchor{15f6}@anchor{using/index using-index}@anchor{15f7}
@chapter Python Setup and Usage


This part of the documentation is devoted to general information on the setup
of the Python environment on different platforms, the invocation of the
interpreter and things that make working with Python easier.

@c ATTENTION: You probably should update Misc/python.man, too, if you modify
@c this file.

@menu
* Command line and environment:: 
* Using Python on Unix platforms:: 
* Configure Python:: 
* Using Python on Windows:: 
* Using Python on a Mac:: 
* Editors and IDEs:: 

@end menu

@node Command line and environment,Using Python on Unix platforms,,Python Setup and Usage
@anchor{using/cmdline doc}@anchor{15f8}@anchor{using/cmdline command-line-and-environment}@anchor{15f9}@anchor{using/cmdline using-on-general}@anchor{1259}
@section Command line and environment


The CPython interpreter scans the command line and the environment for various
settings.

@strong{CPython implementation detail:} Other implementations’ command line schemes may differ.  See
@ref{15fa,,Alternate Implementations} for further resources.

@menu
* Command line:: 
* Environment variables:: 

@end menu

@node Command line,Environment variables,,Command line and environment
@anchor{using/cmdline command-line}@anchor{15fb}@anchor{using/cmdline using-on-cmdline}@anchor{b8d}
@subsection Command line


When invoking Python, you may specify any of these options:

@example
python [-bBdEhiIOqsSuvVWx?] [-c command | -m module-name | script | - ] [args]
@end example

The most common use case is, of course, a simple invocation of a script:

@example
python myscript.py
@end example

@menu
* Interface options:: 
* Generic options:: 
* Miscellaneous options:: 
* Options you shouldn’t use:: 

@end menu

@node Interface options,Generic options,,Command line
@anchor{using/cmdline interface-options}@anchor{15fc}@anchor{using/cmdline using-on-interface-options}@anchor{15fd}
@subsubsection Interface options


The interpreter interface resembles that of the UNIX shell, but provides some
additional methods of invocation:


@itemize *

@item 
When called with standard input connected to a tty device, it prompts for
commands and executes them until an EOF (an end-of-file character, you can
produce that with @code{Ctrl-D} on UNIX or @code{Ctrl-Z, Enter} on Windows) is read.

@item 
When called with a file name argument or with a file as standard input, it
reads and executes a script from that file.

@item 
When called with a directory name argument, it reads and executes an
appropriately named script from that directory.

@item 
When called with @code{-c command}, it executes the Python statement(s) given as
@emph{command}.  Here @emph{command} may contain multiple statements separated by
newlines. Leading whitespace is significant in Python statements!

@item 
When called with @code{-m module-name}, the given module is located on the
Python module path and executed as a script.
@end itemize

In non-interactive mode, the entire input is parsed before it is executed.

An interface option terminates the list of options consumed by the interpreter,
all consecutive arguments will end up in @ref{e15,,sys.argv} – note that the first
element, subscript zero (@code{sys.argv[0]}), is a string reflecting the program’s
source.

@geindex command line option; -c <command>
@anchor{using/cmdline cmdoption-c}@anchor{1180}
@deffn {Program Option} @w{-}c <command>

Execute the Python code in @emph{command}.  @emph{command} can be one or more
statements separated by newlines, with significant leading whitespace as in
normal module code.

If this option is given, the first element of @ref{e15,,sys.argv} will be
@code{"-c"} and the current directory will be added to the start of
@ref{489,,sys.path} (allowing modules in that directory to be imported as top
level modules).

Raises an @ref{1130,,auditing event} @code{cpython.run_command} with argument @code{command}.
@end deffn

@geindex command line option; -m <module-name>
@anchor{using/cmdline cmdoption-m}@anchor{611}
@deffn {Program Option} @w{-}m <module@w{-}name>

Search @ref{489,,sys.path} for the named module and execute its contents as
the @ref{1,,__main__} module.

Since the argument is a @emph{module} name, you must not give a file extension
(@code{.py}).  The module name should be a valid absolute Python module name, but
the implementation may not always enforce this (e.g. it may allow you to
use a name that includes a hyphen).

Package names (including namespace packages) are also permitted. When a
package name is supplied instead
of a normal module, the interpreter will execute @code{<pkg>.__main__} as
the main module. This behaviour is deliberately similar to the handling
of directories and zipfiles that are passed to the interpreter as the
script argument.

@cartouche
@quotation Note 
This option cannot be used with built-in modules and extension modules
written in C, since they do not have Python module files. However, it
can still be used for precompiled modules, even if the original source
file is not available.
@end quotation
@end cartouche

If this option is given, the first element of @ref{e15,,sys.argv} will be the
full path to the module file (while the module file is being located, the
first element will be set to @code{"-m"}). As with the @ref{1180,,-c} option,
the current directory will be added to the start of @ref{489,,sys.path}.

@ref{461,,-I} option can  be used to run the script in isolated mode where
@ref{489,,sys.path} contains neither the current directory nor the user’s
site-packages directory. All 
@geindex PYTHON*
@geindex environment variable; PYTHON*
@code{PYTHON*} environment variables are
ignored, too.

Many standard library modules contain code that is invoked on their execution
as a script.  An example is the @ref{10b,,timeit} module:

@example
python -m timeit -s 'setup here' 'benchmarked code here'
python -m timeit -h # for details
@end example

Raises an @ref{1130,,auditing event} @code{cpython.run_module} with argument @code{module-name}.


@subsubheading See also



@table @asis

@item @ref{15fe,,runpy.run_module()}

Equivalent functionality directly available to Python code
@end table

@geindex Python Enhancement Proposals; PEP 338
PEP 338@footnote{https://www.python.org/dev/peps/pep-0338} – Executing modules as scripts



Changed in version 3.1: Supply the package name to run a @code{__main__} submodule.



Changed in version 3.4: namespace packages are also supported

@end deffn
@anchor{using/cmdline cmdarg-dash}@anchor{15ff}

@deffn {Describe} @w{-}

Read commands from standard input (@ref{5ee,,sys.stdin}).  If standard input is
a terminal, @ref{1018,,-i} is implied.

If this option is given, the first element of @ref{e15,,sys.argv} will be
@code{"-"} and the current directory will be added to the start of
@ref{489,,sys.path}.

Raises an @ref{1130,,auditing event} @code{cpython.run_stdin} with no arguments.
@end deffn
@anchor{using/cmdline cmdarg-script}@anchor{1600}

@deffn {Describe} <script>

Execute the Python code contained in @emph{script}, which must be a filesystem
path (absolute or relative) referring to either a Python file, a directory
containing a @code{__main__.py} file, or a zipfile containing a
@code{__main__.py} file.

If this option is given, the first element of @ref{e15,,sys.argv} will be the
script name as given on the command line.

If the script name refers directly to a Python file, the directory
containing that file is added to the start of @ref{489,,sys.path}, and the
file is executed as the @ref{1,,__main__} module.

If the script name refers to a directory or zipfile, the script name is
added to the start of @ref{489,,sys.path} and the @code{__main__.py} file in
that location is executed as the @ref{1,,__main__} module.

@ref{461,,-I} option can  be used to run the script in isolated mode where
@ref{489,,sys.path} contains neither the script’s directory nor the user’s
site-packages directory. All 
@geindex PYTHON*
@geindex environment variable; PYTHON*
@code{PYTHON*} environment variables are
ignored, too.

Raises an @ref{1130,,auditing event} @code{cpython.run_file} with argument @code{filename}.


@subsubheading See also



@table @asis

@item @ref{eb3,,runpy.run_path()}

Equivalent functionality directly available to Python code
@end table

@end deffn

If no interface option is given, @ref{1018,,-i} is implied, @code{sys.argv[0]} is
an empty string (@code{""}) and the current directory will be added to the
start of @ref{489,,sys.path}.  Also, tab-completion and history editing is
automatically enabled, if available on your platform (see
@ref{b4e,,Readline configuration}).


@subsubheading See also


@ref{14c8,,Invoking the Interpreter}



Changed in version 3.4: Automatic enabling of tab-completion and history editing.


@node Generic options,Miscellaneous options,Interface options,Command line
@anchor{using/cmdline generic-options}@anchor{1601}
@subsubsection Generic options


@geindex command line option; -?
@geindex command line option; -h
@geindex command line option; --help
@anchor{using/cmdline cmdoption-0}@anchor{f86}@anchor{using/cmdline cmdoption}@anchor{1602}
@deffn {Program Option} @w{-}?
@anchor{using/cmdline cmdoption-h}@anchor{1603}
@deffnx {Program Option} @w{-}h
@anchor{using/cmdline cmdoption-help}@anchor{f85}
@deffnx {Program Option} @w{-}@w{-}help

Print a short description of all command line options.
@end deffn

@geindex command line option; -V
@geindex command line option; --version
@anchor{using/cmdline cmdoption-V}@anchor{86b}@anchor{using/cmdline cmdoption-v}@anchor{1604}
@deffn {Program Option} @w{-}V
@anchor{using/cmdline cmdoption-version}@anchor{86a}
@deffnx {Program Option} @w{-}@w{-}version

Print the Python version number and exit.  Example output could be:

@example
Python 3.8.0b2+
@end example

When given twice, print more information about the build, like:

@example
Python 3.8.0b2+ (3.8:0c076caaa8, Apr 20 2019, 21:55:00)
[GCC 6.2.0 20161005]
@end example


New in version 3.6: The @code{-VV} option.

@end deffn

@node Miscellaneous options,Options you shouldn’t use,Generic options,Command line
@anchor{using/cmdline miscellaneous-options}@anchor{1605}@anchor{using/cmdline using-on-misc-options}@anchor{b8e}
@subsubsection Miscellaneous options


@geindex command line option; -b
@anchor{using/cmdline cmdoption-b}@anchor{6dc}
@deffn {Program Option} @w{-}b

Issue a warning when comparing @ref{2f0,,bytes} or @ref{2f1,,bytearray} with
@ref{2ef,,str} or @ref{2f0,,bytes} with @ref{1b1,,int}.  Issue an error when the
option is given twice (@code{-bb}).


Changed in version 3.5: Affects comparisons of @ref{2f0,,bytes} with @ref{1b1,,int}.

@end deffn

@geindex command line option; -B
@anchor{using/cmdline cmdoption-B}@anchor{f34}
@deffn {Program Option} @w{-}B

If given, Python won’t try to write @code{.pyc} files on the
import of source modules.  See also 
@geindex PYTHONDONTWRITEBYTECODE
@geindex environment variable; PYTHONDONTWRITEBYTECODE
@ref{f35,,PYTHONDONTWRITEBYTECODE}.
@end deffn

@geindex command line option; --check-hash-based-pycs default|always|never
@anchor{using/cmdline cmdoption-check-hash-based-pycs}@anchor{1606}
@deffn {Program Option} @w{-}@w{-}check@w{-}hash@w{-}based@w{-}pycs default|always|never

Control the validation behavior of hash-based @code{.pyc} files. See
@ref{607,,Cached bytecode invalidation}. When set to @code{default}, checked and unchecked
hash-based bytecode cache files are validated according to their default
semantics. When set to @code{always}, all hash-based @code{.pyc} files, whether
checked or unchecked, are validated against their corresponding source
file. When set to @code{never}, hash-based @code{.pyc} files are not validated
against their corresponding source files.

The semantics of timestamp-based @code{.pyc} files are unaffected by this
option.
@end deffn

@geindex command line option; -d
@anchor{using/cmdline cmdoption-d}@anchor{1607}
@deffn {Program Option} @w{-}d

Turn on parser debugging output (for expert only, depending on compilation
options).  See also 
@geindex PYTHONDEBUG
@geindex environment variable; PYTHONDEBUG
@ref{1608,,PYTHONDEBUG}.
@end deffn

@geindex command line option; -E
@anchor{using/cmdline cmdoption-E}@anchor{460}@anchor{using/cmdline cmdoption-e}@anchor{1609}
@deffn {Program Option} @w{-}E

Ignore all 
@geindex PYTHON*
@geindex environment variable; PYTHON*
@code{PYTHON*} environment variables, e.g.
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} and 
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME}, that might be set.
@end deffn

@geindex command line option; -i
@anchor{using/cmdline cmdoption-i}@anchor{1018}
@deffn {Program Option} @w{-}i

When a script is passed as first argument or the @ref{1180,,-c} option is used,
enter interactive mode after executing the script or the command, even when
@ref{5ee,,sys.stdin} does not appear to be a terminal.  The
@geindex PYTHONSTARTUP
@geindex environment variable; PYTHONSTARTUP
@ref{b4d,,PYTHONSTARTUP} file is not read.

This can be useful to inspect global variables or a stack trace when a script
raises an exception.  See also 
@geindex PYTHONINSPECT
@geindex environment variable; PYTHONINSPECT
@ref{1017,,PYTHONINSPECT}.
@end deffn

@geindex command line option; -I
@anchor{using/cmdline cmdoption-I}@anchor{461}
@deffn {Program Option} @w{-}I

Run Python in isolated mode. This also implies -E and -s.
In isolated mode @ref{489,,sys.path} contains neither the script’s directory nor
the user’s site-packages directory. All 
@geindex PYTHON*
@geindex environment variable; PYTHON*
@code{PYTHON*} environment
variables are ignored, too. Further restrictions may be imposed to prevent
the user from injecting malicious code.


New in version 3.4.

@end deffn

@geindex command line option; -O
@anchor{using/cmdline cmdoption-O}@anchor{912}@anchor{using/cmdline cmdoption-o}@anchor{160a}
@deffn {Program Option} @w{-}O

Remove assert statements and any code conditional on the value of
@ref{257,,__debug__}.  Augment the filename for compiled
(@ref{8a8,,bytecode}) files by adding @code{.opt-1} before the @code{.pyc}
extension (see 
@geindex Python Enhancement Proposals; PEP 488
PEP 488@footnote{https://www.python.org/dev/peps/pep-0488}).  See also 
@geindex PYTHONOPTIMIZE
@geindex environment variable; PYTHONOPTIMIZE
@ref{160b,,PYTHONOPTIMIZE}.


Changed in version 3.5: Modify @code{.pyc} filenames according to 
@geindex Python Enhancement Proposals; PEP 488
PEP 488@footnote{https://www.python.org/dev/peps/pep-0488}.

@end deffn

@geindex command line option; -OO
@anchor{using/cmdline cmdoption-OO}@anchor{913}@anchor{using/cmdline cmdoption-oo}@anchor{160c}
@deffn {Program Option} @w{-}OO

Do @ref{912,,-O} and also discard docstrings.  Augment the filename
for compiled (@ref{8a8,,bytecode}) files by adding @code{.opt-2} before the
@code{.pyc} extension (see 
@geindex Python Enhancement Proposals; PEP 488
PEP 488@footnote{https://www.python.org/dev/peps/pep-0488}).


Changed in version 3.5: Modify @code{.pyc} filenames according to 
@geindex Python Enhancement Proposals; PEP 488
PEP 488@footnote{https://www.python.org/dev/peps/pep-0488}.

@end deffn

@geindex command line option; -q
@anchor{using/cmdline cmdoption-q}@anchor{160d}
@deffn {Program Option} @w{-}q

Don’t display the copyright and version messages even in interactive mode.


New in version 3.2.

@end deffn

@geindex command line option; -R
@anchor{using/cmdline cmdoption-R}@anchor{1357}@anchor{using/cmdline cmdoption-r}@anchor{160e}
@deffn {Program Option} @w{-}R

Turn on hash randomization. This option only has an effect if the
@geindex PYTHONHASHSEED
@geindex environment variable; PYTHONHASHSEED
@ref{c15,,PYTHONHASHSEED} environment variable is set to @code{0}, since hash
randomization is enabled by default.

On previous versions of Python, this option turns on hash randomization,
so that the @code{__hash__()} values of str and bytes objects
are “salted” with an unpredictable random value.  Although they remain
constant within an individual Python process, they are not predictable
between repeated invocations of Python.

Hash randomization is intended to provide protection against a
denial-of-service caused by carefully-chosen inputs that exploit the worst
case performance of a dict construction, O(n@w{^2}) complexity.  See
@indicateurl{http://www.ocert.org/advisories/ocert-2011-003.html} for details.

@geindex PYTHONHASHSEED
@geindex environment variable; PYTHONHASHSEED
@ref{c15,,PYTHONHASHSEED} allows you to set a fixed value for the hash
seed secret.


Changed in version 3.7: The option is no longer ignored.



New in version 3.2.3.

@end deffn

@geindex command line option; -s
@anchor{using/cmdline cmdoption-s}@anchor{f15}
@deffn {Program Option} @w{-}s

Don’t add the @ref{160f,,user site-packages directory} to
@ref{489,,sys.path}.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 370
PEP 370@footnote{https://www.python.org/dev/peps/pep-0370} – Per user site-packages directory

@end deffn

@geindex command line option; -S
@anchor{using/cmdline cmdoption-S}@anchor{d4e}
@deffn {Program Option} @w{-}S

Disable the import of the module @ref{e6,,site} and the site-dependent
manipulations of @ref{489,,sys.path} that it entails.  Also disable these
manipulations if @ref{e6,,site} is explicitly imported later (call
@ref{1610,,site.main()} if you want them to be triggered).
@end deffn

@geindex command line option; -u
@anchor{using/cmdline cmdoption-u}@anchor{123a}
@deffn {Program Option} @w{-}u

Force the stdout and stderr streams to be unbuffered.  This option has no
effect on the stdin stream.

See also 
@geindex PYTHONUNBUFFERED
@geindex environment variable; PYTHONUNBUFFERED
@ref{123b,,PYTHONUNBUFFERED}.


Changed in version 3.7: The text layer of the stdout and stderr streams now is unbuffered.

@end deffn

@geindex command line option; -v
@anchor{using/cmdline cmdoption-1}@anchor{f82}
@deffn {Program Option} @w{-}v

Print a message each time a module is initialized, showing the place
(filename or built-in module) from which it is loaded.  When given twice
(@code{-vv}), print a message for each file that is checked for when
searching for a module.  Also provides information on module cleanup at exit.


Changed in version 3.10: The @ref{e6,,site} module reports the site-specific paths
and @code{.pth} files being processed.


See also 
@geindex PYTHONVERBOSE
@geindex environment variable; PYTHONVERBOSE
@ref{1611,,PYTHONVERBOSE}.
@end deffn
@anchor{using/cmdline using-on-warnings}@anchor{1612}
@geindex command line option; -W arg
@anchor{using/cmdline cmdoption-W}@anchor{3a8}@anchor{using/cmdline cmdoption-w}@anchor{1613}
@deffn {Program Option} @w{-}W arg

Warning control. Python’s warning machinery by default prints warning
messages to @ref{42d,,sys.stderr}.

The simplest settings apply a particular action unconditionally to all
warnings emitted by a process (even those that are otherwise ignored by
default):

@example
-Wdefault  # Warn once per call location
-Werror    # Convert to exceptions
-Walways   # Warn every time
-Wmodule   # Warn once per calling module
-Wonce     # Warn once per Python process
-Wignore   # Never warn
@end example

The action names can be abbreviated as desired and the interpreter will
resolve them to the appropriate action name. For example, @code{-Wi} is the
same as @code{-Wignore}.

The full form of argument is:

@example
action:message:category:module:lineno
@end example

Empty fields match all values; trailing empty fields may be omitted. For
example @code{-W ignore::DeprecationWarning} ignores all DeprecationWarning
warnings.

The @emph{action} field is as explained above but only applies to warnings that
match the remaining fields.

The @emph{message} field must match the whole warning message; this match is
case-insensitive.

The @emph{category} field matches the warning category
(ex: @code{DeprecationWarning}). This must be a class name; the match test
whether the actual warning category of the message is a subclass of the
specified warning category.

The @emph{module} field matches the (fully-qualified) module name; this match is
case-sensitive.

The @emph{lineno} field matches the line number, where zero matches all line
numbers and is thus equivalent to an omitted line number.

Multiple @ref{3a8,,-W} options can be given; when a warning matches more than
one option, the action for the last matching option is performed. Invalid
@ref{3a8,,-W} options are ignored (though, a warning message is printed about
invalid options when the first warning is issued).

Warnings can also be controlled using the 
@geindex PYTHONWARNINGS
@geindex environment variable; PYTHONWARNINGS
@ref{6de,,PYTHONWARNINGS}
environment variable and from within a Python program using the
@ref{12d,,warnings} module. For example, the @ref{ffd,,warnings.filterwarnings()}
function can be used to use a regular expression on the warning message.

See @ref{3aa,,The Warnings Filter} and @ref{1614,,Describing Warning Filters} for more
details.
@end deffn

@geindex command line option; -x
@anchor{using/cmdline cmdoption-x}@anchor{1615}
@deffn {Program Option} @w{-}x

Skip the first line of the source, allowing use of non-Unix forms of
@code{#!cmd}.  This is intended for a DOS specific hack only.
@end deffn

@geindex command line option; -X
@anchor{using/cmdline cmdoption-X}@anchor{159}
@deffn {Program Option} @w{-}X

Reserved for various implementation-specific options.  CPython currently
defines the following possible values:


@itemize *

@item 
@code{-X faulthandler} to enable @ref{78,,faulthandler};

@item 
@code{-X showrefcount} to output the total reference count and number of used
memory blocks when the program finishes or after each statement in the
interactive interpreter. This only works on @ref{338,,debug builds}.

@item 
@code{-X tracemalloc} to start tracing Python memory allocations using the
@ref{11b,,tracemalloc} module. By default, only the most recent frame is
stored in a traceback of a trace. Use @code{-X tracemalloc=NFRAME} to start
tracing with a traceback limit of @emph{NFRAME} frames. See the
@ref{1616,,tracemalloc.start()} for more information.

@item 
@code{-X importtime} to show how long each import takes. It shows module
name, cumulative time (including nested imports) and self time (excluding
nested imports).  Note that its output may be broken in multi-threaded
application.  Typical usage is @code{python3 -X importtime -c 'import
asyncio'}.  See also 
@geindex PYTHONPROFILEIMPORTTIME
@geindex environment variable; PYTHONPROFILEIMPORTTIME
@ref{612,,PYTHONPROFILEIMPORTTIME}.

@item 
@code{-X dev}: enable @ref{3ac,,Python Development Mode}, introducing
additional runtime checks that are too expensive to be enabled by
default.

@item 
@code{-X utf8} enables the @ref{5f2,,Python UTF-8 Mode}.
@code{-X utf8=0} explicitly disables @ref{5f2,,Python UTF-8 Mode}
(even when it would otherwise activate automatically).

@item 
@code{-X pycache_prefix=PATH} enables writing @code{.pyc} files to a parallel
tree rooted at the given directory instead of to the code tree. See also
@geindex PYTHONPYCACHEPREFIX
@geindex environment variable; PYTHONPYCACHEPREFIX
@ref{4aa,,PYTHONPYCACHEPREFIX}.

@item 
@code{-X warn_default_encoding} issues a @ref{1617,,EncodingWarning} when the
locale-specific default encoding is used for opening files.
See also 
@geindex PYTHONWARNDEFAULTENCODING
@geindex environment variable; PYTHONWARNDEFAULTENCODING
@ref{230,,PYTHONWARNDEFAULTENCODING}.

@item 
@code{-X no_debug_ranges} disables the inclusion of the tables mapping extra
location information (end line, start column offset and end column offset)
to every instruction in code objects. This is useful when smaller code
objects and pyc files are desired as well as suppressing the extra visual
location indicators when the interpreter displays tracebacks. See also
@geindex PYTHONNODEBUGRANGES
@geindex environment variable; PYTHONNODEBUGRANGES
@ref{15a,,PYTHONNODEBUGRANGES}.

@item 
@code{-X frozen_modules} determines whether or not frozen modules are
ignored by the import machinery.  A value of “on” means they get
imported and “off” means they are ignored.  The default is “on”
if this is an installed Python (the normal case).  If it’s under
development (running from the source tree) then the default is “off”.
Note that the “importlib_bootstrap” and “importlib_bootstrap_external”
frozen modules are always used, even if this flag is set to “off”.
@end itemize

It also allows passing arbitrary values and retrieving them through the
@ref{1618,,sys._xoptions} dictionary.


Changed in version 3.2: The @ref{159,,-X} option was added.



New in version 3.3: The @code{-X faulthandler} option.



New in version 3.4: The @code{-X showrefcount} and @code{-X tracemalloc} options.



New in version 3.6: The @code{-X showalloccount} option.



New in version 3.7: The @code{-X importtime}, @code{-X dev} and @code{-X utf8} options.



New in version 3.8: The @code{-X pycache_prefix} option. The @code{-X dev} option now logs
@code{close()} exceptions in @ref{503,,io.IOBase} destructor.



Changed in version 3.9: Using @code{-X dev} option, check @emph{encoding} and @emph{errors} arguments on
string encoding and decoding operations.

The @code{-X showalloccount} option has been removed.



New in version 3.10: The @code{-X warn_default_encoding} option.



Deprecated since version 3.9, removed in version 3.10: The @code{-X oldparser} option.



New in version 3.11: The @code{-X no_debug_ranges} option.



New in version 3.11: The @code{-X frozen_modules} option.

@end deffn

@node Options you shouldn’t use,,Miscellaneous options,Command line
@anchor{using/cmdline options-you-shouldn-t-use}@anchor{1619}
@subsubsection Options you shouldn’t use


@geindex command line option; -J
@anchor{using/cmdline cmdoption-J}@anchor{f33}@anchor{using/cmdline cmdoption-j}@anchor{161a}
@deffn {Program Option} @w{-}J

Reserved for use by Jython@footnote{http://www.jython.org/}.
@end deffn

@node Environment variables,,Command line,Command line and environment
@anchor{using/cmdline environment-variables}@anchor{161b}@anchor{using/cmdline using-on-envvars}@anchor{161c}
@subsection Environment variables


These environment variables influence Python’s behavior, they are processed
before the command-line switches other than -E or -I.  It is customary that
command-line switches override environmental variables where there is a
conflict.

@geindex environment variable; PYTHONHOME
@anchor{using/cmdline envvar-PYTHONHOME}@anchor{780}
@deffn {Environment Variable} PYTHONHOME

Change the location of the standard Python libraries.  By default, the
libraries are searched in @code{@emph{prefix}/lib/python@emph{version}} and
@code{@emph{exec_prefix}/lib/python@emph{version}}, where @code{@emph{prefix}} and
@code{@emph{exec_prefix}} are installation-dependent directories, both defaulting
to @code{/usr/local}.

When 
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME} is set to a single directory, its value replaces
both @code{@emph{prefix}} and @code{@emph{exec_prefix}}.  To specify different values
for these, set 
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME} to @code{@emph{prefix}:@emph{exec_prefix}}.
@end deffn

@geindex environment variable; PYTHONPATH
@anchor{using/cmdline envvar-PYTHONPATH}@anchor{bac}
@deffn {Environment Variable} PYTHONPATH

Augment the default search path for module files.  The format is the same as
the shell’s 
@geindex PATH
@geindex environment variable; PATH
@code{PATH}: one or more directory pathnames separated by
@ref{161d,,os.pathsep} (e.g. colons on Unix or semicolons on Windows).
Non-existent directories are silently ignored.

In addition to normal directories, individual 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} entries
may refer to zipfiles containing pure Python modules (in either source or
compiled form). Extension modules cannot be imported from zipfiles.

The default search path is installation dependent, but generally begins with
@code{@emph{prefix}/lib/python@emph{version}} (see 
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME} above).  It
is @emph{always} appended to 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH}.

An additional directory will be inserted in the search path in front of
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} as described above under
@ref{15fd,,Interface options}. The search path can be manipulated from
within a Python program as the variable @ref{489,,sys.path}.
@end deffn

@geindex environment variable; PYTHONPLATLIBDIR
@anchor{using/cmdline envvar-PYTHONPLATLIBDIR}@anchor{11bd}
@deffn {Environment Variable} PYTHONPLATLIBDIR

If this is set to a non-empty string, it overrides the @ref{42c,,sys.platlibdir}
value.


New in version 3.9.

@end deffn

@geindex environment variable; PYTHONSTARTUP
@anchor{using/cmdline envvar-PYTHONSTARTUP}@anchor{b4d}
@deffn {Environment Variable} PYTHONSTARTUP

If this is the name of a readable file, the Python commands in that file are
executed before the first prompt is displayed in interactive mode.  The file
is executed in the same namespace where interactive commands are executed so
that objects defined or imported in it can be used without qualification in
the interactive session.  You can also change the prompts @ref{161e,,sys.ps1} and
@ref{161f,,sys.ps2} and the hook @ref{b4c,,sys.__interactivehook__} in this file.


Raises an @ref{1130,,auditing event} @code{cpython.run_startup} with
the filename as the argument when called on startup.

@end deffn

@geindex environment variable; PYTHONOPTIMIZE
@anchor{using/cmdline envvar-PYTHONOPTIMIZE}@anchor{160b}
@deffn {Environment Variable} PYTHONOPTIMIZE

If this is set to a non-empty string it is equivalent to specifying the
@ref{912,,-O} option.  If set to an integer, it is equivalent to specifying
@ref{912,,-O} multiple times.
@end deffn

@geindex environment variable; PYTHONBREAKPOINT
@anchor{using/cmdline envvar-PYTHONBREAKPOINT}@anchor{5f5}
@deffn {Environment Variable} PYTHONBREAKPOINT

If this is set, it names a callable using dotted-path notation.  The module
containing the callable will be imported and then the callable will be run
by the default implementation of @ref{5f4,,sys.breakpointhook()} which itself is
called by built-in @ref{5db,,breakpoint()}.  If not set, or set to the empty
string, it is equivalent to the value “pdb.set_trace”.  Setting this to the
string “0” causes the default implementation of @ref{5f4,,sys.breakpointhook()}
to do nothing but return immediately.


New in version 3.7.

@end deffn

@geindex environment variable; PYTHONDEBUG
@anchor{using/cmdline envvar-PYTHONDEBUG}@anchor{1608}
@deffn {Environment Variable} PYTHONDEBUG

If this is set to a non-empty string it is equivalent to specifying the
@ref{1607,,-d} option.  If set to an integer, it is equivalent to specifying
@ref{1607,,-d} multiple times.
@end deffn

@geindex environment variable; PYTHONINSPECT
@anchor{using/cmdline envvar-PYTHONINSPECT}@anchor{1017}
@deffn {Environment Variable} PYTHONINSPECT

If this is set to a non-empty string it is equivalent to specifying the
@ref{1018,,-i} option.

This variable can also be modified by Python code using @ref{d60,,os.environ}
to force inspect mode on program termination.
@end deffn

@geindex environment variable; PYTHONUNBUFFERED
@anchor{using/cmdline envvar-PYTHONUNBUFFERED}@anchor{123b}
@deffn {Environment Variable} PYTHONUNBUFFERED

If this is set to a non-empty string it is equivalent to specifying the
@ref{123a,,-u} option.
@end deffn

@geindex environment variable; PYTHONVERBOSE
@anchor{using/cmdline envvar-PYTHONVERBOSE}@anchor{1611}
@deffn {Environment Variable} PYTHONVERBOSE

If this is set to a non-empty string it is equivalent to specifying the
@ref{f82,,-v} option.  If set to an integer, it is equivalent to specifying
@ref{f82,,-v} multiple times.
@end deffn

@geindex environment variable; PYTHONCASEOK
@anchor{using/cmdline envvar-PYTHONCASEOK}@anchor{45f}
@deffn {Environment Variable} PYTHONCASEOK

If this is set, Python ignores case in @ref{e35,,import} statements.  This
only works on Windows and macOS.
@end deffn

@geindex environment variable; PYTHONDONTWRITEBYTECODE
@anchor{using/cmdline envvar-PYTHONDONTWRITEBYTECODE}@anchor{f35}
@deffn {Environment Variable} PYTHONDONTWRITEBYTECODE

If this is set to a non-empty string, Python won’t try to write @code{.pyc}
files on the import of source modules.  This is equivalent to
specifying the @ref{f34,,-B} option.
@end deffn

@geindex environment variable; PYTHONPYCACHEPREFIX
@anchor{using/cmdline envvar-PYTHONPYCACHEPREFIX}@anchor{4aa}
@deffn {Environment Variable} PYTHONPYCACHEPREFIX

If this is set, Python will write @code{.pyc} files in a mirror directory tree
at this path, instead of in @code{__pycache__} directories within the source
tree. This is equivalent to specifying the @ref{159,,-X}
@code{pycache_prefix=PATH} option.


New in version 3.8.

@end deffn

@geindex environment variable; PYTHONHASHSEED
@anchor{using/cmdline envvar-PYTHONHASHSEED}@anchor{c15}
@deffn {Environment Variable} PYTHONHASHSEED

If this variable is not set or set to @code{random}, a random value is used
to seed the hashes of str and bytes objects.

If 
@geindex PYTHONHASHSEED
@geindex environment variable; PYTHONHASHSEED
@ref{c15,,PYTHONHASHSEED} is set to an integer value, it is used as a fixed
seed for generating the hash() of the types covered by the hash
randomization.

Its purpose is to allow repeatable hashing, such as for selftests for the
interpreter itself, or to allow a cluster of python processes to share hash
values.

The integer must be a decimal number in the range [0,4294967295].  Specifying
the value 0 will disable hash randomization.


New in version 3.2.3.

@end deffn

@geindex environment variable; PYTHONIOENCODING
@anchor{using/cmdline envvar-PYTHONIOENCODING}@anchor{b91}
@deffn {Environment Variable} PYTHONIOENCODING

If this is set before running the interpreter, it overrides the encoding used
for stdin/stdout/stderr, in the syntax @code{encodingname:errorhandler}.  Both
the @code{encodingname} and the @code{:errorhandler} parts are optional and have
the same meaning as in @ref{3ba,,str.encode()}.

For stderr, the @code{:errorhandler} part is ignored; the handler will always be
@code{'backslashreplace'}.


Changed in version 3.4: The @code{encodingname} part is now optional.



Changed in version 3.6: On Windows, the encoding specified by this variable is ignored for interactive
console buffers unless 
@geindex PYTHONLEGACYWINDOWSSTDIO
@geindex environment variable; PYTHONLEGACYWINDOWSSTDIO
@ref{7a2,,PYTHONLEGACYWINDOWSSTDIO} is also specified.
Files and pipes redirected through the standard streams are not affected.

@end deffn

@geindex environment variable; PYTHONNOUSERSITE
@anchor{using/cmdline envvar-PYTHONNOUSERSITE}@anchor{f16}
@deffn {Environment Variable} PYTHONNOUSERSITE

If this is set, Python won’t add the @ref{160f,,user site-packages directory} to @ref{489,,sys.path}.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 370
PEP 370@footnote{https://www.python.org/dev/peps/pep-0370} – Per user site-packages directory

@end deffn

@geindex environment variable; PYTHONUSERBASE
@anchor{using/cmdline envvar-PYTHONUSERBASE}@anchor{f14}
@deffn {Environment Variable} PYTHONUSERBASE

Defines the @ref{1620,,user base directory}, which is used to
compute the path of the @ref{160f,,user site-packages directory}
and @ref{1621,,Distutils installation paths} for
@code{python setup.py install --user}.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 370
PEP 370@footnote{https://www.python.org/dev/peps/pep-0370} – Per user site-packages directory

@end deffn

@geindex environment variable; PYTHONEXECUTABLE
@anchor{using/cmdline envvar-PYTHONEXECUTABLE}@anchor{1622}
@deffn {Environment Variable} PYTHONEXECUTABLE

If this environment variable is set, @code{sys.argv[0]} will be set to its
value instead of the value got through the C runtime.  Only works on
macOS.
@end deffn

@geindex environment variable; PYTHONWARNINGS
@anchor{using/cmdline envvar-PYTHONWARNINGS}@anchor{6de}
@deffn {Environment Variable} PYTHONWARNINGS

This is equivalent to the @ref{3a8,,-W} option. If set to a comma
separated string, it is equivalent to specifying @ref{3a8,,-W} multiple
times, with filters later in the list taking precedence over those earlier
in the list.

The simplest settings apply a particular action unconditionally to all
warnings emitted by a process (even those that are otherwise ignored by
default):

@example
PYTHONWARNINGS=default  # Warn once per call location
PYTHONWARNINGS=error    # Convert to exceptions
PYTHONWARNINGS=always   # Warn every time
PYTHONWARNINGS=module   # Warn once per calling module
PYTHONWARNINGS=once     # Warn once per Python process
PYTHONWARNINGS=ignore   # Never warn
@end example

See @ref{3aa,,The Warnings Filter} and @ref{1614,,Describing Warning Filters} for more
details.
@end deffn

@geindex environment variable; PYTHONFAULTHANDLER
@anchor{using/cmdline envvar-PYTHONFAULTHANDLER}@anchor{c18}
@deffn {Environment Variable} PYTHONFAULTHANDLER

If this environment variable is set to a non-empty string,
@ref{7ea,,faulthandler.enable()} is called at startup: install a handler for
@code{SIGSEGV}, @code{SIGFPE}, @code{SIGABRT}, @code{SIGBUS} and
@code{SIGILL} signals to dump the Python traceback.  This is equivalent to
@ref{159,,-X} @code{faulthandler} option.


New in version 3.3.

@end deffn

@geindex environment variable; PYTHONTRACEMALLOC
@anchor{using/cmdline envvar-PYTHONTRACEMALLOC}@anchor{1623}
@deffn {Environment Variable} PYTHONTRACEMALLOC

If this environment variable is set to a non-empty string, start tracing
Python memory allocations using the @ref{11b,,tracemalloc} module. The value of
the variable is the maximum number of frames stored in a traceback of a
trace. For example, @code{PYTHONTRACEMALLOC=1} stores only the most recent
frame. See the @ref{1616,,tracemalloc.start()} for more information.


New in version 3.4.

@end deffn

@geindex environment variable; PYTHONPROFILEIMPORTTIME
@anchor{using/cmdline envvar-PYTHONPROFILEIMPORTTIME}@anchor{612}
@deffn {Environment Variable} PYTHONPROFILEIMPORTTIME

If this environment variable is set to a non-empty string, Python will
show how long each import takes.  This is exactly equivalent to setting
@code{-X importtime} on the command line.


New in version 3.7.

@end deffn

@geindex environment variable; PYTHONASYNCIODEBUG
@anchor{using/cmdline envvar-PYTHONASYNCIODEBUG}@anchor{1624}
@deffn {Environment Variable} PYTHONASYNCIODEBUG

If this environment variable is set to a non-empty string, enable the
@ref{1625,,debug mode} of the @ref{9,,asyncio} module.


New in version 3.4.

@end deffn

@geindex environment variable; PYTHONMALLOC
@anchor{using/cmdline envvar-PYTHONMALLOC}@anchor{7ab}
@deffn {Environment Variable} PYTHONMALLOC

Set the Python memory allocators and/or install debug hooks.

Set the family of memory allocators used by Python:


@itemize *

@item 
@code{default}: use the @ref{1626,,default memory allocators}.

@item 
@code{malloc}: use the @code{malloc()} function of the C library
for all domains (@code{PYMEM_DOMAIN_RAW}, @code{PYMEM_DOMAIN_MEM},
@code{PYMEM_DOMAIN_OBJ}).

@item 
@code{pymalloc}: use the @ref{85c,,pymalloc allocator} for
@code{PYMEM_DOMAIN_MEM} and @code{PYMEM_DOMAIN_OBJ} domains and use
the @code{malloc()} function for the @code{PYMEM_DOMAIN_RAW} domain.
@end itemize

Install @ref{1627,,debug hooks}:


@itemize *

@item 
@code{debug}: install debug hooks on top of the @ref{1626,,default memory allocators}.

@item 
@code{malloc_debug}: same as @code{malloc} but also install debug hooks.

@item 
@code{pymalloc_debug}: same as @code{pymalloc} but also install debug hooks.
@end itemize


Changed in version 3.7: Added the @code{"default"} allocator.



New in version 3.6.

@end deffn

@geindex environment variable; PYTHONMALLOCSTATS
@anchor{using/cmdline envvar-PYTHONMALLOCSTATS}@anchor{1423}
@deffn {Environment Variable} PYTHONMALLOCSTATS

If set to a non-empty string, Python will print statistics of the
@ref{85c,,pymalloc memory allocator} every time a new pymalloc object
arena is created, and on shutdown.

This variable is ignored if the 
@geindex PYTHONMALLOC
@geindex environment variable; PYTHONMALLOC
@ref{7ab,,PYTHONMALLOC} environment variable
is used to force the @code{malloc()} allocator of the C library, or if
Python is configured without @code{pymalloc} support.


Changed in version 3.6: This variable can now also be used on Python compiled in release mode.
It now has no effect if set to an empty string.

@end deffn

@geindex environment variable; PYTHONLEGACYWINDOWSFSENCODING
@anchor{using/cmdline envvar-PYTHONLEGACYWINDOWSFSENCODING}@anchor{79f}
@deffn {Environment Variable} PYTHONLEGACYWINDOWSFSENCODING

If set to a non-empty string, the default @ref{1628,,filesystem encoding and error handler} mode will revert to their pre-3.6 values of ‘mbcs’ and
‘replace’, respectively.  Otherwise, the new defaults ‘utf-8’ and
‘surrogatepass’ are used.

This may also be enabled at runtime with
@ref{7a0,,sys._enablelegacywindowsfsencoding()}.

@ref{1629,,Availability}: Windows.


New in version 3.6: See 
@geindex Python Enhancement Proposals; PEP 529
PEP 529@footnote{https://www.python.org/dev/peps/pep-0529} for more details.

@end deffn

@geindex environment variable; PYTHONLEGACYWINDOWSSTDIO
@anchor{using/cmdline envvar-PYTHONLEGACYWINDOWSSTDIO}@anchor{7a2}
@deffn {Environment Variable} PYTHONLEGACYWINDOWSSTDIO

If set to a non-empty string, does not use the new console reader and
writer. This means that Unicode characters will be encoded according to
the active console code page, rather than using utf-8.

This variable is ignored if the standard streams are redirected (to files
or pipes) rather than referring to console buffers.

@ref{1629,,Availability}: Windows.


New in version 3.6.

@end deffn

@geindex environment variable; PYTHONCOERCECLOCALE
@anchor{using/cmdline envvar-PYTHONCOERCECLOCALE}@anchor{5ed}
@deffn {Environment Variable} PYTHONCOERCECLOCALE

If set to the value @code{0}, causes the main Python command line application
to skip coercing the legacy ASCII-based C and POSIX locales to a more
capable UTF-8 based alternative.

If this variable is @emph{not} set (or is set to a value other than @code{0}), the
@code{LC_ALL} locale override environment variable is also not set, and the
current locale reported for the @code{LC_CTYPE} category is either the default
@code{C} locale, or else the explicitly ASCII-based @code{POSIX} locale, then the
Python CLI will attempt to configure the following locales for the
@code{LC_CTYPE} category in the order listed before loading the interpreter
runtime:


@itemize *

@item 
@code{C.UTF-8}

@item 
@code{C.utf8}

@item 
@code{UTF-8}
@end itemize

If setting one of these locale categories succeeds, then the @code{LC_CTYPE}
environment variable will also be set accordingly in the current process
environment before the Python runtime is initialized. This ensures that in
addition to being seen by both the interpreter itself and other locale-aware
components running in the same process (such as the GNU @code{readline}
library), the updated setting is also seen in subprocesses (regardless of
whether or not those processes are running a Python interpreter), as well as
in operations that query the environment rather than the current C locale
(such as Python’s own @ref{1b3,,locale.getdefaultlocale()}).

Configuring one of these locales (either explicitly or via the above
implicit locale coercion) automatically enables the @code{surrogateescape}
@ref{162a,,error handler} for @ref{5ee,,sys.stdin} and
@ref{5ef,,sys.stdout} (@ref{42d,,sys.stderr} continues to use @code{backslashreplace}
as it does in any other locale). This stream handling behavior can be
overridden using 
@geindex PYTHONIOENCODING
@geindex environment variable; PYTHONIOENCODING
@ref{b91,,PYTHONIOENCODING} as usual.

For debugging purposes, setting @code{PYTHONCOERCECLOCALE=warn} will cause
Python to emit warning messages on @code{stderr} if either the locale coercion
activates, or else if a locale that @emph{would} have triggered coercion is
still active when the Python runtime is initialized.

Also note that even when locale coercion is disabled, or when it fails to
find a suitable target locale, 
@geindex PYTHONUTF8
@geindex environment variable; PYTHONUTF8
@ref{5f1,,PYTHONUTF8} will still activate by
default in legacy ASCII-based locales. Both features must be disabled in
order to force the interpreter to use @code{ASCII} instead of @code{UTF-8} for
system interfaces.

@ref{1629,,Availability}: *nix.


New in version 3.7: See 
@geindex Python Enhancement Proposals; PEP 538
PEP 538@footnote{https://www.python.org/dev/peps/pep-0538} for more details.

@end deffn

@geindex environment variable; PYTHONDEVMODE
@anchor{using/cmdline envvar-PYTHONDEVMODE}@anchor{60a}
@deffn {Environment Variable} PYTHONDEVMODE

If this environment variable is set to a non-empty string, enable
@ref{3ac,,Python Development Mode}, introducing additional runtime
checks that are too expensive to be enabled by default.


New in version 3.7.

@end deffn

@geindex environment variable; PYTHONUTF8
@anchor{using/cmdline envvar-PYTHONUTF8}@anchor{5f1}
@deffn {Environment Variable} PYTHONUTF8

If set to @code{1}, enable the @ref{5f2,,Python UTF-8 Mode}.

If set to @code{0}, disable the @ref{5f2,,Python UTF-8 Mode}.

Setting any other non-empty string causes an error during interpreter
initialisation.


New in version 3.7.

@end deffn

@geindex environment variable; PYTHONWARNDEFAULTENCODING
@anchor{using/cmdline envvar-PYTHONWARNDEFAULTENCODING}@anchor{230}
@deffn {Environment Variable} PYTHONWARNDEFAULTENCODING

If this environment variable is set to a non-empty string, issue a
@ref{1617,,EncodingWarning} when the locale-specific default encoding is used.

See @ref{162b,,Opt-in EncodingWarning} for details.


New in version 3.10.

@end deffn

@geindex environment variable; PYTHONNODEBUGRANGES
@anchor{using/cmdline envvar-PYTHONNODEBUGRANGES}@anchor{15a}
@deffn {Environment Variable} PYTHONNODEBUGRANGES

If this variable is set, it disables the inclusion of the tables mapping
extra location information (end line, start column offset and end column
offset) to every instruction in code objects. This is useful when smaller
code objects and pyc files are desired as well as suppressing the extra visual
location indicators when the interpreter displays tracebacks.


New in version 3.11.

@end deffn

@menu
* Debug-mode variables:: 

@end menu

@node Debug-mode variables,,,Environment variables
@anchor{using/cmdline debug-mode-variables}@anchor{162c}
@subsubsection Debug-mode variables


@geindex environment variable; PYTHONTHREADDEBUG
@anchor{using/cmdline envvar-PYTHONTHREADDEBUG}@anchor{337}
@deffn {Environment Variable} PYTHONTHREADDEBUG

If set, Python will print threading debug info into stdout.

Need a @ref{338,,debug build of Python}.


Deprecated since version 3.10, will be removed in version 3.12.

@end deffn

@geindex environment variable; PYTHONDUMPREFS
@anchor{using/cmdline envvar-PYTHONDUMPREFS}@anchor{4ad}
@deffn {Environment Variable} PYTHONDUMPREFS

If set, Python will dump objects and reference counts still alive after
shutting down the interpreter.

Need Python configured with the @ref{376,,--with-trace-refs} build option.
@end deffn

@geindex environment variable; PYTHONDUMPREFSFILE=FILENAME
@anchor{using/cmdline envvar-PYTHONDUMPREFSFILE-FILENAME}@anchor{162d}@anchor{using/cmdline envvar-PYTHONDUMPREFSFILE=FILENAME}@anchor{162e}
@deffn {Environment Variable} PYTHONDUMPREFSFILE=FILENAME

If set, Python will dump objects and reference counts still alive
after shutting down the interpreter into a file called @emph{FILENAME}.

Need Python configured with the @ref{376,,--with-trace-refs} build option.


New in version 3.11.

@end deffn

@node Using Python on Unix platforms,Configure Python,Command line and environment,Python Setup and Usage
@anchor{using/unix doc}@anchor{162f}@anchor{using/unix using-on-unix}@anchor{1630}@anchor{using/unix using-python-on-unix-platforms}@anchor{1631}
@section Using Python on Unix platforms


@menu
* Getting and installing the latest version of Python:: 
* Building Python:: 
* Python-related paths and files:: 
* Miscellaneous:: 
* Custom OpenSSL:: 

@end menu

@node Getting and installing the latest version of Python,Building Python,,Using Python on Unix platforms
@anchor{using/unix getting-and-installing-the-latest-version-of-python}@anchor{1632}
@subsection Getting and installing the latest version of Python


@menu
* On Linux:: 
* On FreeBSD and OpenBSD:: 
* On OpenSolaris:: 

@end menu

@node On Linux,On FreeBSD and OpenBSD,,Getting and installing the latest version of Python
@anchor{using/unix on-linux}@anchor{1633}
@subsubsection On Linux


Python comes preinstalled on most Linux distributions, and is available as a
package on all others.  However there are certain features you might want to use
that are not available on your distro’s package.  You can easily compile the
latest version of Python from source.

In the event that Python doesn’t come preinstalled and isn’t in the repositories as
well, you can easily make packages for your own distro.  Have a look at the
following links:


@subsubheading See also



@table @asis

@item @indicateurl{https://www.debian.org/doc/manuals/maint-guide/first.en.html}

for Debian users

@item @indicateurl{https://en.opensuse.org/Portal:Packaging}

for OpenSuse users

@item @indicateurl{https://docs-old.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch-creating-rpms.html}

for Fedora users

@item @indicateurl{http://www.slackbook.org/html/package-management-making-packages.html}

for Slackware users
@end table


@node On FreeBSD and OpenBSD,On OpenSolaris,On Linux,Getting and installing the latest version of Python
@anchor{using/unix on-freebsd-and-openbsd}@anchor{1634}
@subsubsection On FreeBSD and OpenBSD



@itemize *

@item 
FreeBSD users, to add the package use:

@example
pkg install python3
@end example

@item 
OpenBSD users, to add the package use:

@example
pkg_add -r python

pkg_add ftp://ftp.openbsd.org/pub/OpenBSD/4.2/packages/<insert your architecture here>/python-<version>.tgz
@end example

For example i386 users get the 2.5.1 version of Python using:

@example
pkg_add ftp://ftp.openbsd.org/pub/OpenBSD/4.2/packages/i386/python-2.5.1p2.tgz
@end example
@end itemize

@node On OpenSolaris,,On FreeBSD and OpenBSD,Getting and installing the latest version of Python
@anchor{using/unix on-opensolaris}@anchor{1635}
@subsubsection On OpenSolaris


You can get Python from OpenCSW@footnote{https://www.opencsw.org/}.  Various versions
of Python are available and can be installed with e.g. @code{pkgutil -i python27}.

@node Building Python,Python-related paths and files,Getting and installing the latest version of Python,Using Python on Unix platforms
@anchor{using/unix building-python}@anchor{1636}@anchor{using/unix building-python-on-unix}@anchor{a67}
@subsection Building Python


If you want to compile CPython yourself, first thing you should do is get the
source@footnote{https://www.python.org/downloads/source/}. You can download either the
latest release’s source or just grab a fresh clone@footnote{https://devguide.python.org/setup/#getting-the-source-code}.  (If you want
to contribute patches, you will need a clone.)

The build process consists of the usual commands:

@example
./configure
make
make install
@end example

@ref{1637,,Configuration options} and caveats for specific Unix
platforms are extensively documented in the README.rst@footnote{https://github.com/python/cpython/tree/main/README.rst} file in the
root of the Python source tree.

@cartouche
@quotation Warning 
@code{make install} can overwrite or masquerade the @code{python3} binary.
@code{make altinstall} is therefore recommended instead of @code{make install}
since it only installs @code{@emph{exec_prefix}/bin/python@emph{version}}.
@end quotation
@end cartouche

@node Python-related paths and files,Miscellaneous,Building Python,Using Python on Unix platforms
@anchor{using/unix python-related-paths-and-files}@anchor{1638}
@subsection Python-related paths and files


These are subject to difference depending on local installation conventions;
@geindex prefix
@geindex environment variable; prefix
@code{prefix} (@code{$@{prefix@}}) and 
@geindex exec_prefix
@geindex environment variable; exec_prefix
@code{exec_prefix} (@code{$@{exec_prefix@}})
are installation-dependent and should be interpreted as for GNU software; they
may be the same.

For example, on most Linux systems, the default for both is @code{/usr}.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

File/directory

@tab

Meaning

@item

@code{@emph{exec_prefix}/bin/python3}

@tab

Recommended location of the interpreter.

@item

@code{@emph{prefix}/lib/python@emph{version}},
@code{@emph{exec_prefix}/lib/python@emph{version}}

@tab

Recommended locations of the directories
containing the standard modules.

@item

@code{@emph{prefix}/include/python@emph{version}},
@code{@emph{exec_prefix}/include/python@emph{version}}

@tab

Recommended locations of the directories
containing the include files needed for
developing Python extensions and
embedding the interpreter.

@end multitable


@node Miscellaneous,Custom OpenSSL,Python-related paths and files,Using Python on Unix platforms
@anchor{using/unix miscellaneous}@anchor{1639}
@subsection Miscellaneous


To easily use Python scripts on Unix, you need to make them executable,
e.g. with

@example
$ chmod +x script
@end example

and put an appropriate Shebang line at the top of the script.  A good choice is
usually

@example
#!/usr/bin/env python3
@end example

which searches for the Python interpreter in the whole 
@geindex PATH
@geindex environment variable; PATH
@code{PATH}.  However,
some Unices may not have the @code{env} command, so you may need to hardcode
@code{/usr/bin/python3} as the interpreter path.

To use shell commands in your Python scripts, look at the @ref{f4,,subprocess} module.

@node Custom OpenSSL,,Miscellaneous,Using Python on Unix platforms
@anchor{using/unix custom-openssl}@anchor{163a}@anchor{using/unix unix-custom-openssl}@anchor{163b}
@subsection Custom OpenSSL



@enumerate 

@item 
To use your vendor’s OpenSSL configuration and system trust store, locate
the directory with @code{openssl.cnf} file or symlink in @code{/etc}. On most
distribution the file is either in @code{/etc/ssl} or @code{/etc/pki/tls}. The
directory should also contain a @code{cert.pem} file and/or a @code{certs}
directory.

@example
$ find /etc/ -name openssl.cnf -printf "%h\n"
/etc/ssl
@end example

@item 
Download, build, and install OpenSSL. Make sure you use @code{install_sw} and
not @code{install}. The @code{install_sw} target does not override
@code{openssl.cnf}.

@example
$ curl -O https://www.openssl.org/source/openssl-VERSION.tar.gz
   $ tar xzf openssl-VERSION
   $ pushd openssl-VERSION
   $ ./config \
        --prefix=/usr/local/custom-openssl \
        --libdir=lib \
        --openssldir=/etc/ssl
   $ make -j1 depend
   $ make -j8
   $ make install_sw
   $ popd
@end example

@item 
Build Python with custom OpenSSL
(see the configure @cite{–with-openssl} and @cite{–with-openssl-rpath} options)

@example
$ pushd python-3.x.x
$ ./configure -C \
    --with-openssl=/usr/local/custom-openssl \
    --with-openssl-rpath=auto \
    --prefix=/usr/local/python-3.x.x
$ make -j8
$ make altinstall
@end example
@end enumerate

@cartouche
@quotation Note 
Patch releases of OpenSSL have a backwards compatible ABI. You don’t need
to recompile Python to update OpenSSL. It’s sufficient to replace the
custom OpenSSL installation with a newer version.
@end quotation
@end cartouche

@node Configure Python,Using Python on Windows,Using Python on Unix platforms,Python Setup and Usage
@anchor{using/configure doc}@anchor{163c}@anchor{using/configure configure-python}@anchor{163d}
@section Configure Python


@menu
* Configure Options:: 
* Python Build System:: 
* Compiler and linker flags:: 

@end menu

@node Configure Options,Python Build System,,Configure Python
@anchor{using/configure configure-options}@anchor{1637}@anchor{using/configure id1}@anchor{163e}
@subsection Configure Options


List all @code{./configure} script options using:

@example
./configure --help
@end example

See also the @code{Misc/SpecialBuilds.txt} in the Python source distribution.

@menu
* General Options:: 
* Install Options:: 
* Performance options:: 
* Python Debug Build:: 
* Debug options:: 
* Linker options:: 
* Libraries options:: 
* Security Options:: 
* macOS Options:: 
* Cross Compiling Options:: 

@end menu

@node General Options,Install Options,,Configure Options
@anchor{using/configure general-options}@anchor{163f}
@subsubsection General Options


@geindex command line option; --enable-loadable-sqlite-extensions
@anchor{using/configure cmdoption-enable-loadable-sqlite-extensions}@anchor{1640}
@deffn {Program Option} @w{-}@w{-}enable@w{-}loadable@w{-}sqlite@w{-}extensions

Support loadable extensions in the @code{_sqlite} extension module (default
is no).

See the @ref{2c3,,sqlite3.Connection.enable_load_extension()} method of the
@ref{ed,,sqlite3} module.


New in version 3.6.

@end deffn

@geindex command line option; --disable-ipv6
@anchor{using/configure cmdoption-disable-ipv6}@anchor{1641}
@deffn {Program Option} @w{-}@w{-}disable@w{-}ipv6

Disable IPv6 support (enabled by default if supported), see the
@ref{ea,,socket} module.
@end deffn

@geindex command line option; --enable-big-digits=[15|30]
@anchor{using/configure cmdoption-enable-big-digits}@anchor{1642}
@deffn {Program Option} @w{-}@w{-}enable@w{-}big@w{-}digits=[15|30]

Define the size in bits of Python @ref{1b1,,int} digits: 15 or 30 bits.

By default, the digit size is 30.

Define the @code{PYLONG_BITS_IN_DIGIT} to @code{15} or @code{30}.

See @ref{e39,,sys.int_info.bits_per_digit}.
@end deffn

@geindex command line option; --with-cxx-main
@anchor{using/configure cmdoption-with-cxx-main}@anchor{1643}
@deffn {Program Option} @w{-}@w{-}with@w{-}cxx@w{-}main
@end deffn

@geindex command line option; --with-cxx-main=COMPILER
@anchor{using/configure cmdoption-0}@anchor{1644}
@deffn {Program Option} @w{-}@w{-}with@w{-}cxx@w{-}main=COMPILER

Compile the Python @code{main()} function and link Python executable with C++
compiler: @code{$CXX}, or @emph{COMPILER} if specified.
@end deffn

@geindex command line option; --with-suffix=SUFFIX
@anchor{using/configure cmdoption-with-suffix}@anchor{1645}
@deffn {Program Option} @w{-}@w{-}with@w{-}suffix=SUFFIX

Set the Python executable suffix to @emph{SUFFIX}.

The default suffix is @code{.exe} on Windows and macOS (@code{python.exe}
executable), @code{.wasm} on Emscripten (@code{python.wasm} executable), and
an empty string on other platforms (@code{python} executable).


Changed in version 3.11: The default suffix on Emscripten platform is @code{.wasm}.

@end deffn

@geindex command line option; --with-tzpath=<list of absolute paths separated by pathsep>
@anchor{using/configure cmdoption-with-tzpath}@anchor{1646}
@deffn {Program Option} @w{-}@w{-}with@w{-}tzpath=<list of absolute paths separated by pathsep>

Select the default time zone search path for @ref{1647,,zoneinfo.TZPATH}.
See the @ref{1648,,Compile-time configuration} of the @ref{14c,,zoneinfo} module.

Default: @code{/usr/share/zoneinfo:/usr/lib/zoneinfo:/usr/share/lib/zoneinfo:/etc/zoneinfo}.

See @ref{161d,,os.pathsep} path separator.


New in version 3.9.

@end deffn

@geindex command line option; --without-decimal-contextvar
@anchor{using/configure cmdoption-without-decimal-contextvar}@anchor{1649}
@deffn {Program Option} @w{-}@w{-}without@w{-}decimal@w{-}contextvar

Build the @code{_decimal} extension module using a thread-local context rather
than a coroutine-local context (default), see the @ref{34,,decimal} module.

See @ref{164a,,decimal.HAVE_CONTEXTVAR} and the @ref{23,,contextvars} module.


New in version 3.9.

@end deffn

@geindex command line option; --with-dbmliborder=<list of backend names>
@anchor{using/configure cmdoption-with-dbmliborder}@anchor{164b}
@deffn {Program Option} @w{-}@w{-}with@w{-}dbmliborder=<list of backend names>

Override order to check db backends for the @ref{30,,dbm} module

A valid value is a colon (@code{:}) separated string with the backend names:


@itemize *

@item 
@code{ndbm};

@item 
@code{gdbm};

@item 
@code{bdb}.
@end itemize
@end deffn

@geindex command line option; --without-c-locale-coercion
@anchor{using/configure cmdoption-without-c-locale-coercion}@anchor{164c}
@deffn {Program Option} @w{-}@w{-}without@w{-}c@w{-}locale@w{-}coercion

Disable C locale coercion to a UTF-8 based locale (enabled by default).

Don’t define the @code{PY_COERCE_C_LOCALE} macro.

See 
@geindex PYTHONCOERCECLOCALE
@geindex environment variable; PYTHONCOERCECLOCALE
@ref{5ed,,PYTHONCOERCECLOCALE} and the 
@geindex Python Enhancement Proposals; PEP 538
PEP 538@footnote{https://www.python.org/dev/peps/pep-0538}.
@end deffn

@geindex command line option; --with-platlibdir=DIRNAME
@anchor{using/configure cmdoption-with-platlibdir}@anchor{164d}
@deffn {Program Option} @w{-}@w{-}with@w{-}platlibdir=DIRNAME

Python library directory name (default is @code{lib}).

Fedora and SuSE use @code{lib64} on 64-bit platforms.

See @ref{42c,,sys.platlibdir}.


New in version 3.9.

@end deffn

@geindex command line option; --with-wheel-pkg-dir=PATH
@anchor{using/configure cmdoption-with-wheel-pkg-dir}@anchor{356}
@deffn {Program Option} @w{-}@w{-}with@w{-}wheel@w{-}pkg@w{-}dir=PATH

Directory of wheel packages used by the @ref{75,,ensurepip} module
(none by default).

Some Linux distribution packaging policies recommend against bundling
dependencies. For example, Fedora installs wheel packages in the
@code{/usr/share/python-wheels/} directory and don’t install the
@code{ensurepip._bundled} package.


New in version 3.10.

@end deffn

@geindex command line option; --with-pkg-config=[check|yes|no]
@anchor{using/configure cmdoption-with-pkg-config}@anchor{164e}
@deffn {Program Option} @w{-}@w{-}with@w{-}pkg@w{-}config=[check|yes|no]

Whether configure should use @code{pkg-config} to detect build
dependencies.


@itemize *

@item 
@code{check} (default): @code{pkg-config} is optional

@item 
@code{yes}: @code{pkg-config} is mandatory

@item 
@code{no}: configure does not use @code{pkg-config} even when present
@end itemize


New in version 3.11.

@end deffn

@geindex command line option; --enable-pystats
@anchor{using/configure cmdoption-enable-pystats}@anchor{164f}
@deffn {Program Option} @w{-}@w{-}enable@w{-}pystats

Turn on internal statistics gathering.

The statistics will be dumped to a arbitrary (probably unique) file in
@code{/tmp/py_stats/}, or @code{C:\temp\py_stats\} on Windows.

Use @code{Tools//summarize_stats.py} to read the stats.


New in version 3.11.

@end deffn

@node Install Options,Performance options,General Options,Configure Options
@anchor{using/configure install-options}@anchor{1650}
@subsubsection Install Options


@geindex command line option; --disable-test-modules
@anchor{using/configure cmdoption-disable-test-modules}@anchor{355}
@deffn {Program Option} @w{-}@w{-}disable@w{-}test@w{-}modules

Don’t build nor install test modules, like the @ref{ff,,test} package or the
@code{_testcapi} extension module (built and installed by default).


New in version 3.10.

@end deffn

@geindex command line option; --with-ensurepip=[upgrade|install|no]
@anchor{using/configure cmdoption-with-ensurepip}@anchor{1651}
@deffn {Program Option} @w{-}@w{-}with@w{-}ensurepip=[upgrade|install|no]

Select the @ref{75,,ensurepip} command run on Python installation:


@itemize *

@item 
@code{upgrade} (default): run @code{python -m ensurepip --altinstall --upgrade}
command.

@item 
@code{install}: run @code{python -m ensurepip --altinstall} command;

@item 
@code{no}: don’t run ensurepip;
@end itemize


New in version 3.6.

@end deffn

@node Performance options,Python Debug Build,Install Options,Configure Options
@anchor{using/configure performance-options}@anchor{1652}
@subsubsection Performance options


Configuring Python using @code{--enable-optimizations --with-lto} (PGO + LTO) is
recommended for best performance.

@geindex command line option; --enable-optimizations
@anchor{using/configure cmdoption-enable-optimizations}@anchor{2f2}
@deffn {Program Option} @w{-}@w{-}enable@w{-}optimizations

Enable Profile Guided Optimization (PGO) using 
@geindex PROFILE_TASK
@geindex environment variable; PROFILE_TASK
@ref{1653,,PROFILE_TASK}
(disabled by default).

The C compiler Clang requires @code{llvm-profdata} program for PGO. On
macOS, GCC also requires it: GCC is just an alias to Clang on macOS.

Disable also semantic interposition in libpython if @code{--enable-shared} and
GCC is used: add @code{-fno-semantic-interposition} to the compiler and linker
flags.


New in version 3.6.



Changed in version 3.10: Use @code{-fno-semantic-interposition} on GCC.

@end deffn

@geindex environment variable; PROFILE_TASK
@anchor{using/configure envvar-PROFILE_TASK}@anchor{1653}
@deffn {Environment Variable} PROFILE_TASK

Environment variable used in the Makefile: Python command line arguments for
the PGO generation task.

Default: @code{-m test --pgo --timeout=$(TESTTIMEOUT)}.


New in version 3.8.

@end deffn

@geindex command line option; --with-lto=[full|thin|no|yes]
@anchor{using/configure cmdoption-with-lto}@anchor{1654}
@deffn {Program Option} @w{-}@w{-}with@w{-}lto=[full|thin|no|yes]

Enable Link Time Optimization (LTO) in any build (disabled by default).

The C compiler Clang requires @code{llvm-ar} for LTO (@code{ar} on macOS), as well
as an LTO-aware linker (@code{ld.gold} or @code{lld}).


New in version 3.6.



New in version 3.11: To use ThinLTO feature, use @code{--with-lto=thin} on Clang.

@end deffn

@geindex command line option; --with-computed-gotos
@anchor{using/configure cmdoption-with-computed-gotos}@anchor{1655}
@deffn {Program Option} @w{-}@w{-}with@w{-}computed@w{-}gotos

Enable computed gotos in evaluation loop (enabled by default on supported
compilers).
@end deffn

@geindex command line option; --without-pymalloc
@anchor{using/configure cmdoption-without-pymalloc}@anchor{1656}
@deffn {Program Option} @w{-}@w{-}without@w{-}pymalloc

Disable the specialized Python memory allocator @ref{85c,,pymalloc}
(enabled by default).

See also 
@geindex PYTHONMALLOC
@geindex environment variable; PYTHONMALLOC
@ref{7ab,,PYTHONMALLOC} environment variable.
@end deffn

@geindex command line option; --without-doc-strings
@anchor{using/configure cmdoption-without-doc-strings}@anchor{1657}
@deffn {Program Option} @w{-}@w{-}without@w{-}doc@w{-}strings

Disable static documentation strings to reduce the memory footprint (enabled
by default). Documentation strings defined in Python are not affected.

Don’t define the @code{WITH_DOC_STRINGS} macro.

See the @code{PyDoc_STRVAR()} macro.
@end deffn

@geindex command line option; --enable-profiling
@anchor{using/configure cmdoption-enable-profiling}@anchor{1658}
@deffn {Program Option} @w{-}@w{-}enable@w{-}profiling

Enable C-level code profiling with @code{gprof} (disabled by default).
@end deffn

@node Python Debug Build,Debug options,Performance options,Configure Options
@anchor{using/configure debug-build}@anchor{338}@anchor{using/configure python-debug-build}@anchor{1659}
@subsubsection Python Debug Build


A debug build is Python built with the @ref{165a,,--with-pydebug} configure
option.

Effects of a debug build:


@itemize *

@item 
Display all warnings by default: the list of default warning filters is empty
in the @ref{12d,,warnings} module.

@item 
Add @code{d} to @ref{57a,,sys.abiflags}.

@item 
Add @code{sys.gettotalrefcount()} function.

@item 
Add @ref{159,,-X showrefcount} command line option.

@item 
Add 
@geindex PYTHONTHREADDEBUG
@geindex environment variable; PYTHONTHREADDEBUG
@ref{337,,PYTHONTHREADDEBUG} environment variable.

@item 
Add support for the @code{__ltrace__} variable: enable low-level tracing in the
bytecode evaluation loop if the variable is defined.

@item 
Install @ref{1626,,debug hooks on memory allocators}
to detect buffer overflow and other memory errors.

@item 
Define @code{Py_DEBUG} and @code{Py_REF_DEBUG} macros.

@item 
Add runtime checks: code surrounded by @code{#ifdef Py_DEBUG} and @code{#endif}.
Enable @code{assert(...)} and @code{_PyObject_ASSERT(...)} assertions: don’t set
the @code{NDEBUG} macro (see also the @ref{165b,,--with-assertions} configure
option). Main runtime checks:


@itemize *

@item 
Add sanity checks on the function arguments.

@item 
Unicode and int objects are created with their memory filled with a pattern
to detect usage of uninitialized objects.

@item 
Ensure that functions which can clear or replace the current exception are
not called with an exception raised.

@item 
The garbage collector (@ref{54b,,gc.collect()} function) runs some basic checks
on objects consistency.

@item 
The @code{Py_SAFE_DOWNCAST()} macro checks for integer underflow and
overflow when downcasting from wide types to narrow types.
@end itemize
@end itemize

See also the @ref{3ac,,Python Development Mode} and the
@ref{376,,--with-trace-refs} configure option.


Changed in version 3.8: Release builds and debug builds are now ABI compatible: defining the
@code{Py_DEBUG} macro no longer implies the @code{Py_TRACE_REFS} macro (see the
@ref{376,,--with-trace-refs} option), which introduces the only ABI
incompatibility.


@node Debug options,Linker options,Python Debug Build,Configure Options
@anchor{using/configure debug-options}@anchor{165c}
@subsubsection Debug options


@geindex command line option; --with-pydebug
@anchor{using/configure cmdoption-with-pydebug}@anchor{165a}
@deffn {Program Option} @w{-}@w{-}with@w{-}pydebug

@ref{338,,Build Python in debug mode}: define the @code{Py_DEBUG}
macro (disabled by default).
@end deffn

@geindex command line option; --with-trace-refs
@anchor{using/configure cmdoption-with-trace-refs}@anchor{376}
@deffn {Program Option} @w{-}@w{-}with@w{-}trace@w{-}refs

Enable tracing references for debugging purpose (disabled by default).

Effects:


@itemize *

@item 
Define the @code{Py_TRACE_REFS} macro.

@item 
Add @code{sys.getobjects()} function.

@item 
Add 
@geindex PYTHONDUMPREFS
@geindex environment variable; PYTHONDUMPREFS
@ref{4ad,,PYTHONDUMPREFS} environment variable.
@end itemize

This build is not ABI compatible with release build (default build) or debug
build (@code{Py_DEBUG} and @code{Py_REF_DEBUG} macros).


New in version 3.8.

@end deffn

@geindex command line option; --with-assertions
@anchor{using/configure cmdoption-with-assertions}@anchor{165b}
@deffn {Program Option} @w{-}@w{-}with@w{-}assertions

Build with C assertions enabled (default is no): @code{assert(...);} and
@code{_PyObject_ASSERT(...);}.

If set, the @code{NDEBUG} macro is not defined in the 
@geindex OPT
@geindex environment variable; OPT
@ref{165d,,OPT} compiler
variable.

See also the @ref{165a,,--with-pydebug} option (@ref{338,,debug build}) which also enables assertions.


New in version 3.6.

@end deffn

@geindex command line option; --with-valgrind
@anchor{using/configure cmdoption-with-valgrind}@anchor{165e}
@deffn {Program Option} @w{-}@w{-}with@w{-}valgrind

Enable Valgrind support (default is no).
@end deffn

@geindex command line option; --with-dtrace
@anchor{using/configure cmdoption-with-dtrace}@anchor{165f}
@deffn {Program Option} @w{-}@w{-}with@w{-}dtrace

Enable DTrace support (default is no).

See @ref{7b2,,Instrumenting CPython with DTrace and SystemTap}.


New in version 3.6.

@end deffn

@geindex command line option; --with-address-sanitizer
@anchor{using/configure cmdoption-with-address-sanitizer}@anchor{1660}
@deffn {Program Option} @w{-}@w{-}with@w{-}address@w{-}sanitizer

Enable AddressSanitizer memory error detector, @code{asan} (default is no).


New in version 3.6.

@end deffn

@geindex command line option; --with-memory-sanitizer
@anchor{using/configure cmdoption-with-memory-sanitizer}@anchor{1661}
@deffn {Program Option} @w{-}@w{-}with@w{-}memory@w{-}sanitizer

Enable MemorySanitizer allocation error detector, @code{msan} (default is no).


New in version 3.6.

@end deffn

@geindex command line option; --with-undefined-behavior-sanitizer
@anchor{using/configure cmdoption-with-undefined-behavior-sanitizer}@anchor{1662}
@deffn {Program Option} @w{-}@w{-}with@w{-}undefined@w{-}behavior@w{-}sanitizer

Enable UndefinedBehaviorSanitizer undefined behaviour detector, @code{ubsan}
(default is no).


New in version 3.6.

@end deffn

@node Linker options,Libraries options,Debug options,Configure Options
@anchor{using/configure linker-options}@anchor{1663}
@subsubsection Linker options


@geindex command line option; --enable-shared
@anchor{using/configure cmdoption-enable-shared}@anchor{2f3}
@deffn {Program Option} @w{-}@w{-}enable@w{-}shared

Enable building a shared Python library: @code{libpython} (default is no).
@end deffn

@geindex command line option; --without-static-libpython
@anchor{using/configure cmdoption-without-static-libpython}@anchor{357}
@deffn {Program Option} @w{-}@w{-}without@w{-}static@w{-}libpython

Do not build @code{libpythonMAJOR.MINOR.a} and do not install @code{python.o}
(built and enabled by default).


New in version 3.10.

@end deffn

@node Libraries options,Security Options,Linker options,Configure Options
@anchor{using/configure libraries-options}@anchor{1664}
@subsubsection Libraries options


@geindex command line option; --with-libs='lib1 ...'
@anchor{using/configure cmdoption-with-libs}@anchor{1665}
@deffn {Program Option} @w{-}@w{-}with@w{-}libs='lib1 ...'

Link against additional libraries (default is no).
@end deffn

@geindex command line option; --with-system-expat
@anchor{using/configure cmdoption-with-system-expat}@anchor{1666}
@deffn {Program Option} @w{-}@w{-}with@w{-}system@w{-}expat

Build the @code{pyexpat} module using an installed @code{expat} library
(default is no).
@end deffn

@geindex command line option; --with-system-ffi
@anchor{using/configure cmdoption-with-system-ffi}@anchor{1667}
@deffn {Program Option} @w{-}@w{-}with@w{-}system@w{-}ffi

Build the @code{_ctypes} extension module using an installed @code{ffi}
library, see the @ref{29,,ctypes} module (default is system-dependent).
@end deffn

@geindex command line option; --with-system-libmpdec
@anchor{using/configure cmdoption-with-system-libmpdec}@anchor{1668}
@deffn {Program Option} @w{-}@w{-}with@w{-}system@w{-}libmpdec

Build the @code{_decimal} extension module using an installed @code{mpdec}
library, see the @ref{34,,decimal} module (default is no).


New in version 3.3.

@end deffn

@geindex command line option; --with-readline=editline
@anchor{using/configure cmdoption-with-readline}@anchor{1669}
@deffn {Program Option} @w{-}@w{-}with@w{-}readline=editline

Use @code{editline} library for backend of the @ref{d9,,readline} module.

Define the @code{WITH_EDITLINE} macro.


New in version 3.10.

@end deffn

@geindex command line option; --without-readline
@anchor{using/configure cmdoption-without-readline}@anchor{166a}
@deffn {Program Option} @w{-}@w{-}without@w{-}readline

Don’t build the @ref{d9,,readline} module (built by default).

Don’t define the @code{HAVE_LIBREADLINE} macro.


New in version 3.10.

@end deffn

@geindex command line option; --with-tcltk-includes='-I...'
@anchor{using/configure cmdoption-with-tcltk-includes}@anchor{358}
@deffn {Program Option} @w{-}@w{-}with@w{-}tcltk@w{-}includes='@w{-}I...'

Override search for Tcl and Tk include files.
@end deffn

@geindex command line option; --with-tcltk-libs='-L...'
@anchor{using/configure cmdoption-with-tcltk-libs}@anchor{359}
@deffn {Program Option} @w{-}@w{-}with@w{-}tcltk@w{-}libs='@w{-}L...'

Override search for Tcl and Tk libraries.
@end deffn

@geindex command line option; --with-libm=STRING
@anchor{using/configure cmdoption-with-libm}@anchor{166b}
@deffn {Program Option} @w{-}@w{-}with@w{-}libm=STRING

Override @code{libm} math library to @emph{STRING} (default is system-dependent).
@end deffn

@geindex command line option; --with-libc=STRING
@anchor{using/configure cmdoption-with-libc}@anchor{166c}
@deffn {Program Option} @w{-}@w{-}with@w{-}libc=STRING

Override @code{libc} C library to @emph{STRING} (default is system-dependent).
@end deffn

@geindex command line option; --with-openssl=DIR
@anchor{using/configure cmdoption-with-openssl}@anchor{166d}
@deffn {Program Option} @w{-}@w{-}with@w{-}openssl=DIR

Root of the OpenSSL directory.


New in version 3.7.

@end deffn

@geindex command line option; --with-openssl-rpath=[no|auto|DIR]
@anchor{using/configure cmdoption-with-openssl-rpath}@anchor{35a}
@deffn {Program Option} @w{-}@w{-}with@w{-}openssl@w{-}rpath=[no|auto|DIR]

Set runtime library directory (rpath) for OpenSSL libraries:


@itemize *

@item 
@code{no} (default): don’t set rpath;

@item 
@code{auto}: auto-detect rpath from @ref{166d,,--with-openssl} and
@code{pkg-config};

@item 
@emph{DIR}: set an explicit rpath.
@end itemize


New in version 3.10.

@end deffn

@node Security Options,macOS Options,Libraries options,Configure Options
@anchor{using/configure security-options}@anchor{166e}
@subsubsection Security Options


@geindex command line option; --with-hash-algorithm=[fnv|siphash13|siphash24]
@anchor{using/configure cmdoption-with-hash-algorithm}@anchor{166f}
@deffn {Program Option} @w{-}@w{-}with@w{-}hash@w{-}algorithm=[fnv|siphash13|siphash24]

Select hash algorithm for use in @code{Python/pyhash.c}:


@itemize *

@item 
@code{siphash13} (default);

@item 
@code{siphash24};

@item 
@code{fnv}.
@end itemize


New in version 3.4.



New in version 3.11: @code{siphash13} is added and it is the new default.

@end deffn

@geindex command line option; --with-builtin-hashlib-hashes=md5@comma{}sha1@comma{}sha256@comma{}sha512@comma{}sha3@comma{}blake2
@anchor{using/configure cmdoption-with-builtin-hashlib-hashes}@anchor{1670}
@deffn {Program Option} @w{-}@w{-}with@w{-}builtin@w{-}hashlib@w{-}hashes=md5,sha1,sha256,sha512,sha3,blake2

Built-in hash modules:


@itemize *

@item 
@code{md5};

@item 
@code{sha1};

@item 
@code{sha256};

@item 
@code{sha512};

@item 
@code{sha3} (with shake);

@item 
@code{blake2}.
@end itemize


New in version 3.9.

@end deffn

@geindex command line option; --with-ssl-default-suites=[python|openssl|STRING]
@anchor{using/configure cmdoption-with-ssl-default-suites}@anchor{1671}
@deffn {Program Option} @w{-}@w{-}with@w{-}ssl@w{-}default@w{-}suites=[python|openssl|STRING]

Override the OpenSSL default cipher suites string:


@itemize *

@item 
@code{python} (default): use Python’s preferred selection;

@item 
@code{openssl}: leave OpenSSL’s defaults untouched;

@item 
@emph{STRING}: use a custom string
@end itemize

See the @ref{ee,,ssl} module.


New in version 3.7.



Changed in version 3.10: The settings @code{python} and @emph{STRING} also set TLS 1.2 as minimum
protocol version.

@end deffn

@node macOS Options,Cross Compiling Options,Security Options,Configure Options
@anchor{using/configure macos-options}@anchor{1672}
@subsubsection macOS Options


See @code{Mac/README.rst}.

@geindex command line option; --enable-universalsdk
@anchor{using/configure cmdoption-enable-universalsdk}@anchor{1673}
@deffn {Program Option} @w{-}@w{-}enable@w{-}universalsdk
@end deffn

@geindex command line option; --enable-universalsdk=SDKDIR
@anchor{using/configure cmdoption-1}@anchor{471}
@deffn {Program Option} @w{-}@w{-}enable@w{-}universalsdk=SDKDIR

Create a universal binary build. @emph{SDKDIR} specifies which macOS SDK should
be used to perform the build (default is no).
@end deffn

@geindex command line option; --enable-framework
@anchor{using/configure cmdoption-enable-framework}@anchor{1674}
@deffn {Program Option} @w{-}@w{-}enable@w{-}framework
@end deffn

@geindex command line option; --enable-framework=INSTALLDIR
@anchor{using/configure cmdoption-2}@anchor{1675}
@deffn {Program Option} @w{-}@w{-}enable@w{-}framework=INSTALLDIR

Create a Python.framework rather than a traditional Unix install. Optional
@emph{INSTALLDIR} specifies the installation path (default is no).
@end deffn

@geindex command line option; --with-universal-archs=ARCH
@anchor{using/configure cmdoption-with-universal-archs}@anchor{1676}
@deffn {Program Option} @w{-}@w{-}with@w{-}universal@w{-}archs=ARCH

Specify the kind of universal binary that should be created. This option is
only valid when @ref{471,,--enable-universalsdk} is set.

Options:


@itemize *

@item 
@code{universal2};

@item 
@code{32-bit};

@item 
@code{64-bit};

@item 
@code{3-way};

@item 
@code{intel};

@item 
@code{intel-32};

@item 
@code{intel-64};

@item 
@code{all}.
@end itemize
@end deffn

@geindex command line option; --with-framework-name=FRAMEWORK
@anchor{using/configure cmdoption-with-framework-name}@anchor{1677}
@deffn {Program Option} @w{-}@w{-}with@w{-}framework@w{-}name=FRAMEWORK

Specify the name for the python framework on macOS only valid when
@ref{1675,,--enable-framework} is set (default: @code{Python}).
@end deffn

@node Cross Compiling Options,,macOS Options,Configure Options
@anchor{using/configure cross-compiling-options}@anchor{1678}
@subsubsection Cross Compiling Options


Cross compiling, also known as cross building, can be used to build Python
for another CPU architecture or platform. Cross compiling requires a Python
interpreter for the build platform. The version of the build Python must match
the version of the cross compiled host Python.

@geindex command line option; --build=BUILD
@anchor{using/configure cmdoption-build}@anchor{1679}
@deffn {Program Option} @w{-}@w{-}build=BUILD

configure for building on BUILD, usually guessed by @code{config.guess}.
@end deffn

@geindex command line option; --host=HOST
@anchor{using/configure cmdoption-host}@anchor{167a}
@deffn {Program Option} @w{-}@w{-}host=HOST

cross-compile to build programs to run on HOST (target platform)
@end deffn

@geindex command line option; --with-build-python=path/to/python
@anchor{using/configure cmdoption-with-build-python}@anchor{167b}
@deffn {Program Option} @w{-}@w{-}with@w{-}build@w{-}python=path/to/python

path to build @code{python} binary for cross compiling


New in version 3.11.

@end deffn

@geindex command line option; CONFIG_SITE=file
@anchor{using/configure cmdoption-arg-CONFIG_SITE}@anchor{167c}@anchor{using/configure cmdoption-arg-config-site}@anchor{167d}
@deffn {Program Option} CONFIG_SITE=file

An environment variable that points to a file with configure overrides.

Example @emph{config.site} file:

@example
# config.site-aarch64
ac_cv_buggy_getaddrinfo=no
ac_cv_file__dev_ptmx=yes
ac_cv_file__dev_ptc=no
@end example
@end deffn

Cross compiling example:

@example
CONFIG_SITE=config.site-aarch64 ../configure \
    --build=x86_64-pc-linux-gnu \
    --host=aarch64-unknown-linux-gnu \
    --with-build-python=../x86_64/python
@end example

@node Python Build System,Compiler and linker flags,Configure Options,Configure Python
@anchor{using/configure python-build-system}@anchor{167e}
@subsection Python Build System


@menu
* Main files of the build system:: 
* Main build steps:: 
* Main Makefile targets:: 
* C extensions:: 

@end menu

@node Main files of the build system,Main build steps,,Python Build System
@anchor{using/configure main-files-of-the-build-system}@anchor{167f}
@subsubsection Main files of the build system



@itemize *

@item 
@code{configure.ac} => @code{configure};

@item 
@code{Makefile.pre.in} => @code{Makefile} (created by @code{configure});

@item 
@code{pyconfig.h} (created by @code{configure});

@item 
@code{Modules/Setup}: C extensions built by the Makefile using
@code{Module/makesetup} shell script;

@item 
@code{setup.py}: C extensions built using the @ref{37,,distutils} module.
@end itemize

@node Main build steps,Main Makefile targets,Main files of the build system,Python Build System
@anchor{using/configure main-build-steps}@anchor{1680}
@subsubsection Main build steps



@itemize *

@item 
C files (@code{.c}) are built as object files (@code{.o}).

@item 
A static @code{libpython} library (@code{.a}) is created from objects files.

@item 
@code{python.o} and the static @code{libpython} library are linked into the
final @code{python} program.

@item 
C extensions are built by the Makefile (see @code{Modules/Setup})
and @code{python setup.py build}.
@end itemize

@node Main Makefile targets,C extensions,Main build steps,Python Build System
@anchor{using/configure main-makefile-targets}@anchor{1681}
@subsubsection Main Makefile targets



@itemize *

@item 
@code{make}: Build Python with the standard library.

@item 
@code{make platform:}: build the @code{python} program, but don’t build the
standard library extension modules.

@item 
@code{make profile-opt}: build Python using Profile Guided Optimization (PGO).
You can use the configure @ref{2f2,,--enable-optimizations} option to make
this the default target of the @code{make} command (@code{make all} or just
@code{make}).

@item 
@code{make buildbottest}: Build Python and run the Python test suite, the same
way than buildbots test Python. Set @code{TESTTIMEOUT} variable (in seconds)
to change the test timeout (1200 by default: 20 minutes).

@item 
@code{make install}: Build and install Python.

@item 
@code{make regen-all}: Regenerate (almost) all generated files;
@code{make regen-stdlib-module-names} and @code{autoconf} must be run separately
for the remaining generated files.

@item 
@code{make clean}: Remove built files.

@item 
@code{make distclean}: Same than @code{make clean}, but remove also files created
by the configure script.
@end itemize

@node C extensions,,Main Makefile targets,Python Build System
@anchor{using/configure c-extensions}@anchor{1682}
@subsubsection C extensions


Some C extensions are built as built-in modules, like the @code{sys} module.
They are built with the @code{Py_BUILD_CORE_BUILTIN} macro defined.
Built-in modules have no @code{__file__} attribute:

@example
>>> import sys
>>> sys
<module 'sys' (built-in)>
>>> sys.__file__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'sys' has no attribute '__file__'
@end example

Other C extensions are built as dynamic libraries, like the @code{_asyncio} module.
They are built with the @code{Py_BUILD_CORE_MODULE} macro defined.
Example on Linux x86-64:

@example
>>> import _asyncio
>>> _asyncio
<module '_asyncio' from '/usr/lib64/python3.9/lib-dynload/_asyncio.cpython-39-x86_64-linux-gnu.so'>
>>> _asyncio.__file__
'/usr/lib64/python3.9/lib-dynload/_asyncio.cpython-39-x86_64-linux-gnu.so'
@end example

@code{Modules/Setup} is used to generate Makefile targets to build C extensions.
At the beginning of the files, C extensions are built as built-in modules.
Extensions defined after the @code{*shared*} marker are built as dynamic libraries.

The @code{setup.py} script only builds C extensions as shared libraries using
the @ref{37,,distutils} module.

The @code{PyAPI_FUNC()}, @code{PyAPI_API()} and
@code{PyMODINIT_FUNC()} macros of @code{Include/pyport.h} are defined
differently depending if the @code{Py_BUILD_CORE_MODULE} macro is defined:


@itemize *

@item 
Use @code{Py_EXPORTED_SYMBOL} if the @code{Py_BUILD_CORE_MODULE} is defined

@item 
Use @code{Py_IMPORTED_SYMBOL} otherwise.
@end itemize

If the @code{Py_BUILD_CORE_BUILTIN} macro is used by mistake on a C extension
built as a shared library, its @code{PyInit_xxx()} function is not exported,
causing an @ref{3b8,,ImportError} on import.

@node Compiler and linker flags,,Python Build System,Configure Python
@anchor{using/configure compiler-and-linker-flags}@anchor{1683}
@subsection Compiler and linker flags


Options set by the @code{./configure} script and environment variables and used by
@code{Makefile}.

@menu
* Preprocessor flags:: 
* Compiler flags:: 
* Linker flags:: 

@end menu

@node Preprocessor flags,Compiler flags,,Compiler and linker flags
@anchor{using/configure preprocessor-flags}@anchor{1684}
@subsubsection Preprocessor flags


@geindex environment variable; CONFIGURE_CPPFLAGS
@anchor{using/configure envvar-CONFIGURE_CPPFLAGS}@anchor{1685}
@deffn {Environment Variable} CONFIGURE_CPPFLAGS

Value of 
@geindex CPPFLAGS
@geindex environment variable; CPPFLAGS
@ref{1007,,CPPFLAGS} variable passed to the @code{./configure} script.


New in version 3.6.

@end deffn

@geindex environment variable; CPPFLAGS
@anchor{using/configure envvar-CPPFLAGS}@anchor{1007}
@deffn {Environment Variable} CPPFLAGS

(Objective) C/C++ preprocessor flags, e.g. @code{-I<include dir>} if you have
headers in a nonstandard directory @code{<include dir>}.

Both 
@geindex CPPFLAGS
@geindex environment variable; CPPFLAGS
@ref{1007,,CPPFLAGS} and 
@geindex LDFLAGS
@geindex environment variable; LDFLAGS
@ref{1006,,LDFLAGS} need to contain the shell’s
value for setup.py to be able to build extension modules using the
directories specified in the environment variables.
@end deffn

@geindex environment variable; BASECPPFLAGS
@anchor{using/configure envvar-BASECPPFLAGS}@anchor{1686}
@deffn {Environment Variable} BASECPPFLAGS


New in version 3.4.

@end deffn

@geindex environment variable; PY_CPPFLAGS
@anchor{using/configure envvar-PY_CPPFLAGS}@anchor{1687}
@deffn {Environment Variable} PY_CPPFLAGS

Extra preprocessor flags added for building the interpreter object files.

Default: @code{$(BASECPPFLAGS) -I. -I$(srcdir)/Include $(CONFIGURE_CPPFLAGS) $(CPPFLAGS)}.


New in version 3.2.

@end deffn

@node Compiler flags,Linker flags,Preprocessor flags,Compiler and linker flags
@anchor{using/configure compiler-flags}@anchor{1688}
@subsubsection Compiler flags


@geindex environment variable; CC
@anchor{using/configure envvar-CC}@anchor{1004}
@deffn {Environment Variable} CC

C compiler command.

Example: @code{gcc -pthread}.
@end deffn

@geindex environment variable; MAINCC
@anchor{using/configure envvar-MAINCC}@anchor{1689}
@deffn {Environment Variable} MAINCC

C compiler command used to build the @code{main()} function of programs like
@code{python}.

Variable set by the @ref{1644,,--with-cxx-main} option of the configure
script.

Default: @code{$(CC)}.
@end deffn

@geindex environment variable; CXX
@anchor{using/configure envvar-CXX}@anchor{168a}
@deffn {Environment Variable} CXX

C++ compiler command.

Used if the @ref{1644,,--with-cxx-main} option is used.

Example: @code{g++ -pthread}.
@end deffn

@geindex environment variable; CFLAGS
@anchor{using/configure envvar-CFLAGS}@anchor{1005}
@deffn {Environment Variable} CFLAGS

C compiler flags.
@end deffn

@geindex environment variable; CFLAGS_NODIST
@anchor{using/configure envvar-CFLAGS_NODIST}@anchor{168b}
@deffn {Environment Variable} CFLAGS_NODIST

@geindex CFLAGS_NODIST
@geindex environment variable; CFLAGS_NODIST
@ref{168b,,CFLAGS_NODIST} is used for building the interpreter and stdlib C
extensions.  Use it when a compiler flag should @emph{not} be part of the
distutils 
@geindex CFLAGS
@geindex environment variable; CFLAGS
@ref{1005,,CFLAGS} once Python is installed (bpo-21121@footnote{https://bugs.python.org/issue21121}).


New in version 3.5.

@end deffn

@geindex environment variable; EXTRA_CFLAGS
@anchor{using/configure envvar-EXTRA_CFLAGS}@anchor{168c}
@deffn {Environment Variable} EXTRA_CFLAGS

Extra C compiler flags.
@end deffn

@geindex environment variable; CONFIGURE_CFLAGS
@anchor{using/configure envvar-CONFIGURE_CFLAGS}@anchor{168d}
@deffn {Environment Variable} CONFIGURE_CFLAGS

Value of 
@geindex CFLAGS
@geindex environment variable; CFLAGS
@ref{1005,,CFLAGS} variable passed to the @code{./configure}
script.


New in version 3.2.

@end deffn

@geindex environment variable; CONFIGURE_CFLAGS_NODIST
@anchor{using/configure envvar-CONFIGURE_CFLAGS_NODIST}@anchor{168e}
@deffn {Environment Variable} CONFIGURE_CFLAGS_NODIST

Value of 
@geindex CFLAGS_NODIST
@geindex environment variable; CFLAGS_NODIST
@ref{168b,,CFLAGS_NODIST} variable passed to the @code{./configure}
script.


New in version 3.5.

@end deffn

@geindex environment variable; BASECFLAGS
@anchor{using/configure envvar-BASECFLAGS}@anchor{168f}
@deffn {Environment Variable} BASECFLAGS

Base compiler flags.
@end deffn

@geindex environment variable; OPT
@anchor{using/configure envvar-OPT}@anchor{165d}
@deffn {Environment Variable} OPT

Optimization flags.
@end deffn

@geindex environment variable; CFLAGS_ALIASING
@anchor{using/configure envvar-CFLAGS_ALIASING}@anchor{1690}
@deffn {Environment Variable} CFLAGS_ALIASING

Strict or non-strict aliasing flags used to compile @code{Python/dtoa.c}.


New in version 3.7.

@end deffn

@geindex environment variable; CCSHARED
@anchor{using/configure envvar-CCSHARED}@anchor{1691}
@deffn {Environment Variable} CCSHARED

Compiler flags used to build a shared library.

For example, @code{-fPIC} is used on Linux and on BSD.
@end deffn

@geindex environment variable; CFLAGSFORSHARED
@anchor{using/configure envvar-CFLAGSFORSHARED}@anchor{1692}
@deffn {Environment Variable} CFLAGSFORSHARED

Extra C flags added for building the interpreter object files.

Default: @code{$(CCSHARED)} when @ref{2f3,,--enable-shared} is used, or an empty
string otherwise.
@end deffn

@geindex environment variable; PY_CFLAGS
@anchor{using/configure envvar-PY_CFLAGS}@anchor{1693}
@deffn {Environment Variable} PY_CFLAGS

Default: @code{$(BASECFLAGS) $(OPT) $(CONFIGURE_CFLAGS) $(CFLAGS) $(EXTRA_CFLAGS)}.
@end deffn

@geindex environment variable; PY_CFLAGS_NODIST
@anchor{using/configure envvar-PY_CFLAGS_NODIST}@anchor{1694}
@deffn {Environment Variable} PY_CFLAGS_NODIST

Default: @code{$(CONFIGURE_CFLAGS_NODIST) $(CFLAGS_NODIST) -I$(srcdir)/Include/internal}.


New in version 3.5.

@end deffn

@geindex environment variable; PY_STDMODULE_CFLAGS
@anchor{using/configure envvar-PY_STDMODULE_CFLAGS}@anchor{1695}
@deffn {Environment Variable} PY_STDMODULE_CFLAGS

C flags used for building the interpreter object files.

Default: @code{$(PY_CFLAGS) $(PY_CFLAGS_NODIST) $(PY_CPPFLAGS) $(CFLAGSFORSHARED)}.


New in version 3.7.

@end deffn

@geindex environment variable; PY_CORE_CFLAGS
@anchor{using/configure envvar-PY_CORE_CFLAGS}@anchor{1696}
@deffn {Environment Variable} PY_CORE_CFLAGS

Default: @code{$(PY_STDMODULE_CFLAGS) -DPy_BUILD_CORE}.


New in version 3.2.

@end deffn

@geindex environment variable; PY_BUILTIN_MODULE_CFLAGS
@anchor{using/configure envvar-PY_BUILTIN_MODULE_CFLAGS}@anchor{1697}
@deffn {Environment Variable} PY_BUILTIN_MODULE_CFLAGS

Compiler flags to build a standard library extension module as a built-in
module, like the @ref{cc,,posix} module.

Default: @code{$(PY_STDMODULE_CFLAGS) -DPy_BUILD_CORE_BUILTIN}.


New in version 3.8.

@end deffn

@geindex environment variable; PURIFY
@anchor{using/configure envvar-PURIFY}@anchor{1698}
@deffn {Environment Variable} PURIFY

Purify command. Purify is a memory debugger program.

Default: empty string (not used).
@end deffn

@node Linker flags,,Compiler flags,Compiler and linker flags
@anchor{using/configure linker-flags}@anchor{1699}
@subsubsection Linker flags


@geindex environment variable; LINKCC
@anchor{using/configure envvar-LINKCC}@anchor{169a}
@deffn {Environment Variable} LINKCC

Linker command used to build programs like @code{python} and @code{_testembed}.

Default: @code{$(PURIFY) $(MAINCC)}.
@end deffn

@geindex environment variable; CONFIGURE_LDFLAGS
@anchor{using/configure envvar-CONFIGURE_LDFLAGS}@anchor{169b}
@deffn {Environment Variable} CONFIGURE_LDFLAGS

Value of 
@geindex LDFLAGS
@geindex environment variable; LDFLAGS
@ref{1006,,LDFLAGS} variable passed to the @code{./configure} script.

Avoid assigning 
@geindex CFLAGS
@geindex environment variable; CFLAGS
@ref{1005,,CFLAGS}, 
@geindex LDFLAGS
@geindex environment variable; LDFLAGS
@ref{1006,,LDFLAGS}, etc. so users can use
them on the command line to append to these values without stomping the
pre-set values.


New in version 3.2.

@end deffn

@geindex environment variable; LDFLAGS_NODIST
@anchor{using/configure envvar-LDFLAGS_NODIST}@anchor{169c}
@deffn {Environment Variable} LDFLAGS_NODIST

@geindex LDFLAGS_NODIST
@geindex environment variable; LDFLAGS_NODIST
@ref{169c,,LDFLAGS_NODIST} is used in the same manner as
@geindex CFLAGS_NODIST
@geindex environment variable; CFLAGS_NODIST
@ref{168b,,CFLAGS_NODIST}.  Use it when a linker flag should @emph{not} be part of
the distutils 
@geindex LDFLAGS
@geindex environment variable; LDFLAGS
@ref{1006,,LDFLAGS} once Python is installed (bpo-35257@footnote{https://bugs.python.org/issue35257}).
@end deffn

@geindex environment variable; CONFIGURE_LDFLAGS_NODIST
@anchor{using/configure envvar-CONFIGURE_LDFLAGS_NODIST}@anchor{169d}
@deffn {Environment Variable} CONFIGURE_LDFLAGS_NODIST

Value of 
@geindex LDFLAGS_NODIST
@geindex environment variable; LDFLAGS_NODIST
@ref{169c,,LDFLAGS_NODIST} variable passed to the @code{./configure}
script.


New in version 3.8.

@end deffn

@geindex environment variable; LDFLAGS
@anchor{using/configure envvar-LDFLAGS}@anchor{1006}
@deffn {Environment Variable} LDFLAGS

Linker flags, e.g. @code{-L<lib dir>} if you have libraries in a nonstandard
directory @code{<lib dir>}.

Both 
@geindex CPPFLAGS
@geindex environment variable; CPPFLAGS
@ref{1007,,CPPFLAGS} and 
@geindex LDFLAGS
@geindex environment variable; LDFLAGS
@ref{1006,,LDFLAGS} need to contain the shell’s
value for setup.py to be able to build extension modules using the
directories specified in the environment variables.
@end deffn

@geindex environment variable; LIBS
@anchor{using/configure envvar-LIBS}@anchor{169e}
@deffn {Environment Variable} LIBS

Linker flags to pass libraries to the linker when linking the Python
executable.

Example: @code{-lrt}.
@end deffn

@geindex environment variable; LDSHARED
@anchor{using/configure envvar-LDSHARED}@anchor{169f}
@deffn {Environment Variable} LDSHARED

Command to build a shared library.

Default: @code{@@LDSHARED@@ $(PY_LDFLAGS)}.
@end deffn

@geindex environment variable; BLDSHARED
@anchor{using/configure envvar-BLDSHARED}@anchor{16a0}
@deffn {Environment Variable} BLDSHARED

Command to build @code{libpython} shared library.

Default: @code{@@BLDSHARED@@ $(PY_CORE_LDFLAGS)}.
@end deffn

@geindex environment variable; PY_LDFLAGS
@anchor{using/configure envvar-PY_LDFLAGS}@anchor{16a1}
@deffn {Environment Variable} PY_LDFLAGS

Default: @code{$(CONFIGURE_LDFLAGS) $(LDFLAGS)}.
@end deffn

@geindex environment variable; PY_LDFLAGS_NODIST
@anchor{using/configure envvar-PY_LDFLAGS_NODIST}@anchor{16a2}
@deffn {Environment Variable} PY_LDFLAGS_NODIST

Default: @code{$(CONFIGURE_LDFLAGS_NODIST) $(LDFLAGS_NODIST)}.


New in version 3.8.

@end deffn

@geindex environment variable; PY_CORE_LDFLAGS
@anchor{using/configure envvar-PY_CORE_LDFLAGS}@anchor{16a3}
@deffn {Environment Variable} PY_CORE_LDFLAGS

Linker flags used for building the interpreter object files.


New in version 3.8.

@end deffn

@node Using Python on Windows,Using Python on a Mac,Configure Python,Python Setup and Usage
@anchor{using/windows doc}@anchor{16a4}@anchor{using/windows using-on-windows}@anchor{5c9}@anchor{using/windows using-python-on-windows}@anchor{16a5}
@section Using Python on Windows


This document aims to give an overview of Windows-specific behaviour you should
know about when using Python on Microsoft Windows.

Unlike most Unix systems and services, Windows does not include a system
supported installation of Python. To make Python available, the CPython team
has compiled Windows installers (MSI packages) with every release@footnote{https://www.python.org/download/releases/} for many years. These installers
are primarily intended to add a per-user installation of Python, with the
core interpreter and library being used by a single user. The installer is also
able to install for all users of a single machine, and a separate ZIP file is
available for application-local distributions.

As specified in 
@geindex Python Enhancement Proposals; PEP 11
PEP 11@footnote{https://www.python.org/dev/peps/pep-0011}, a Python release only supports a Windows platform
while Microsoft considers the platform under extended support. This means that
Python 3.11 supports Windows 8.1 and newer. If you require Windows 7
support, please install Python 3.8.

There are a number of different installers available for Windows, each with
certain benefits and downsides.

@ref{16a6,,The full installer} contains all components and is the best option for
developers using Python for any kind of project.

@ref{14c9,,The Microsoft Store package} is a simple installation of Python that is suitable for
running scripts and packages, and using IDLE or other development environments.
It requires Windows 10, but can be safely installed without corrupting other
programs. It also provides many convenient commands for launching Python and
its tools.

@ref{16a7,,The nuget.org packages} are lightweight installations intended for continuous
integration systems. It can be used to build Python packages or run scripts,
but is not updateable and has no user interface tools.

@ref{16a8,,The embeddable package} is a minimal package of Python suitable for
embedding into a larger application.

@menu
* The full installer:: 
* The Microsoft Store package:: 
* The nuget.org packages: The nuget org packages. 
* The embeddable package:: 
* Alternative bundles:: 
* Configuring Python:: 
* UTF-8 mode:: 
* Python Launcher for Windows:: 
* Finding modules:: 
* Additional modules:: 
* Compiling Python on Windows:: 
* Other Platforms:: 

@end menu

@node The full installer,The Microsoft Store package,,Using Python on Windows
@anchor{using/windows the-full-installer}@anchor{16a9}@anchor{using/windows windows-full}@anchor{16a6}
@subsection The full installer


@menu
* Installation steps:: 
* Removing the MAX_PATH Limitation:: 
* Installing Without UI:: 
* Installing Without Downloading:: 
* Modifying an install:: 

@end menu

@node Installation steps,Removing the MAX_PATH Limitation,,The full installer
@anchor{using/windows installation-steps}@anchor{16aa}
@subsubsection Installation steps


Four Python 3.11 installers are available for download - two each for the
32-bit and 64-bit versions of the interpreter. The @emph{web installer} is a small
initial download, and it will automatically download the required components as
necessary. The @emph{offline installer} includes the components necessary for a
default installation and only requires an internet connection for optional
features. See @ref{16ab,,Installing Without Downloading} for other ways to avoid downloading
during installation.

After starting the installer, one of two options may be selected:

@image{python-figures/win_installer,,,,png}

If you select “Install Now”:


@itemize *

@item 
You will @emph{not} need to be an administrator (unless a system update for the
C Runtime Library is required or you install the @ref{be5,,Python Launcher for Windows} for all
users)

@item 
Python will be installed into your user directory

@item 
The @ref{be5,,Python Launcher for Windows} will be installed according to the option at the bottom
of the first page

@item 
The standard library, test suite, launcher and pip will be installed

@item 
If selected, the install directory will be added to your 
@geindex PATH
@geindex environment variable; PATH
@code{PATH}

@item 
Shortcuts will only be visible for the current user
@end itemize

Selecting “Customize installation” will allow you to select the features to
install, the installation location and other options or post-install actions.
To install debugging symbols or binaries, you will need to use this option.

To perform an all-users installation, you should select “Customize
installation”. In this case:


@itemize *

@item 
You may be required to provide administrative credentials or approval

@item 
Python will be installed into the Program Files directory

@item 
The @ref{be5,,Python Launcher for Windows} will be installed into the Windows directory

@item 
Optional features may be selected during installation

@item 
The standard library can be pre-compiled to bytecode

@item 
If selected, the install directory will be added to the system 
@geindex PATH
@geindex environment variable; PATH
@code{PATH}

@item 
Shortcuts are available for all users
@end itemize

@node Removing the MAX_PATH Limitation,Installing Without UI,Installation steps,The full installer
@anchor{using/windows max-path}@anchor{77f}@anchor{using/windows removing-the-max-path-limitation}@anchor{16ac}
@subsubsection Removing the MAX_PATH Limitation


Windows historically has limited path lengths to 260 characters. This meant that
paths longer than this would not resolve and errors would result.

In the latest versions of Windows, this limitation can be expanded to
approximately 32,000 characters. Your administrator will need to activate the
“Enable Win32 long paths” group policy, or set @code{LongPathsEnabled} to @code{1}
in the registry key
@code{HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem}.

This allows the @ref{1c8,,open()} function, the @ref{c0,,os} module and most other
path functionality to accept and return paths longer than 260 characters.

After changing the above option, no further configuration is required.


Changed in version 3.6: Support for long paths was enabled in Python.


@node Installing Without UI,Installing Without Downloading,Removing the MAX_PATH Limitation,The full installer
@anchor{using/windows install-quiet-option}@anchor{16ad}@anchor{using/windows installing-without-ui}@anchor{16ae}
@subsubsection Installing Without UI


All of the options available in the installer UI can also be specified from the
command line, allowing scripted installers to replicate an installation on many
machines without user interaction.  These options may also be set without
suppressing the UI in order to change some of the defaults.

To completely hide the installer UI and install Python silently, pass the
@code{/quiet} option. To skip past the user interaction but still display
progress and errors, pass the @code{/passive} option. The @code{/uninstall}
option may be passed to immediately begin removing Python - no confirmation
prompt will be displayed.

All other options are passed as @code{name=value}, where the value is usually
@code{0} to disable a feature, @code{1} to enable a feature, or a path. The full list
of available options is shown below.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Name

@tab

Description

@tab

Default

@item

InstallAllUsers

@tab

Perform a system-wide installation.

@tab

0

@item

TargetDir

@tab

The installation directory

@tab

Selected based on
InstallAllUsers

@item

DefaultAllUsersTargetDir

@tab

The default installation directory
for all-user installs

@tab

@code{%ProgramFiles%\Python X.Y} or @code{%ProgramFiles(x86)%\Python X.Y}

@item

DefaultJustForMeTargetDir

@tab

The default install directory for
just-for-me installs

@tab

@code{%LocalAppData%\Programs\PythonXY} or
@code{%LocalAppData%\Programs\PythonXY-32} or
@code{%LocalAppData%\Programs\PythonXY-64}

@item

DefaultCustomTargetDir

@tab

The default custom install directory
displayed in the UI

@tab

(empty)

@item

AssociateFiles

@tab

Create file associations if the
launcher is also installed.

@tab

1

@item

CompileAll

@tab

Compile all @code{.py} files to
@code{.pyc}.

@tab

0

@item

PrependPath

@tab

Prepend install and Scripts
directories  to 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} and
add @code{.PY} to 
@geindex PATHEXT
@geindex environment variable; PATHEXT
@code{PATHEXT}

@tab

0

@item

AppendPath

@tab

Append install and Scripts
directories  to 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} and
add @code{.PY} to 
@geindex PATHEXT
@geindex environment variable; PATHEXT
@code{PATHEXT}

@tab

0

@item

Shortcuts

@tab

Create shortcuts for the interpreter,
documentation and IDLE if installed.

@tab

1

@item

Include_doc

@tab

Install Python manual

@tab

1

@item

Include_debug

@tab

Install debug binaries

@tab

0

@item

Include_dev

@tab

Install developer headers and
libraries

@tab

1

@item

Include_exe

@tab

Install @code{python.exe} and
related files

@tab

1

@item

Include_launcher

@tab

Install @ref{be5,,Python Launcher for Windows}.

@tab

1

@item

InstallLauncherAllUsers

@tab

Installs @ref{be5,,Python Launcher for Windows} for all
users.

@tab

1

@item

Include_lib

@tab

Install standard library and
extension modules

@tab

1

@item

Include_pip

@tab

Install bundled pip and setuptools

@tab

1

@item

Include_symbols

@tab

Install debugging symbols (@cite{*}.pdb)

@tab

0

@item

Include_tcltk

@tab

Install Tcl/Tk support and IDLE

@tab

1

@item

Include_test

@tab

Install standard library test suite

@tab

1

@item

Include_tools

@tab

Install utility scripts

@tab

1

@item

LauncherOnly

@tab

Only installs the launcher. This
will override most other options.

@tab

0

@item

SimpleInstall

@tab

Disable most install UI

@tab

0

@item

SimpleInstallDescription

@tab

A custom message to display when the
simplified install UI is used.

@tab

(empty)

@end multitable


For example, to silently install a default, system-wide Python installation,
you could use the following command (from an elevated command prompt):

@example
python-3.9.0.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0
@end example

To allow users to easily install a personal copy of Python without the test
suite, you could provide a shortcut with the following command. This will
display a simplified initial page and disallow customization:

@example
python-3.9.0.exe InstallAllUsers=0 Include_launcher=0 Include_test=0
    SimpleInstall=1 SimpleInstallDescription="Just for me, no test suite."
@end example

(Note that omitting the launcher also omits file associations, and is only
recommended for per-user installs when there is also a system-wide installation
that included the launcher.)

The options listed above can also be provided in a file named @code{unattend.xml}
alongside the executable. This file specifies a list of options and values.
When a value is provided as an attribute, it will be converted to a number if
possible. Values provided as element text are always left as strings. This
example file sets the same options as the previous example:

@example
<Options>
    <Option Name="InstallAllUsers" Value="no" />
    <Option Name="Include_launcher" Value="0" />
    <Option Name="Include_test" Value="no" />
    <Option Name="SimpleInstall" Value="yes" />
    <Option Name="SimpleInstallDescription">Just for me, no test suite</Option>
</Options>
@end example

@node Installing Without Downloading,Modifying an install,Installing Without UI,The full installer
@anchor{using/windows install-layout-option}@anchor{16ab}@anchor{using/windows installing-without-downloading}@anchor{16af}
@subsubsection Installing Without Downloading


As some features of Python are not included in the initial installer download,
selecting those features may require an internet connection.  To avoid this
need, all possible components may be downloaded on-demand to create a complete
@emph{layout} that will no longer require an internet connection regardless of the
selected features. Note that this download may be bigger than required, but
where a large number of installations are going to be performed it is very
useful to have a locally cached copy.

Execute the following command from Command Prompt to download all possible
required files.  Remember to substitute @code{python-3.9.0.exe} for the actual
name of your installer, and to create layouts in their own directories to
avoid collisions between files with the same name.

@example
python-3.9.0.exe /layout [optional target directory]
@end example

You may also specify the @code{/quiet} option to hide the progress display.

@node Modifying an install,,Installing Without Downloading,The full installer
@anchor{using/windows modifying-an-install}@anchor{16b0}
@subsubsection Modifying an install


Once Python has been installed, you can add or remove features through the
Programs and Features tool that is part of Windows. Select the Python entry and
choose “Uninstall/Change” to open the installer in maintenance mode.

“Modify” allows you to add or remove features by modifying the checkboxes -
unchanged checkboxes will not install or remove anything. Some options cannot be
changed in this mode, such as the install directory; to modify these, you will
need to remove and then reinstall Python completely.

“Repair” will verify all the files that should be installed using the current
settings and replace any that have been removed or modified.

“Uninstall” will remove Python entirely, with the exception of the
@ref{be5,,Python Launcher for Windows}, which has its own entry in Programs and Features.

@node The Microsoft Store package,The nuget org packages,The full installer,Using Python on Windows
@anchor{using/windows the-microsoft-store-package}@anchor{16b1}@anchor{using/windows windows-store}@anchor{14c9}
@subsection The Microsoft Store package



New in version 3.7.2.


The Microsoft Store package is an easily installable Python interpreter that
is intended mainly for interactive use, for example, by students.

To install the package, ensure you have the latest Windows 10 updates and
search the Microsoft Store app for “Python 3.11”. Ensure that the app
you select is published by the Python Software Foundation, and install it.

@cartouche
@quotation Warning 
Python will always be available for free on the Microsoft Store. If you
are asked to pay for it, you have not selected the correct package.
@end quotation
@end cartouche

After installation, Python may be launched by finding it in Start.
Alternatively, it will be available from any Command Prompt or PowerShell
session by typing @code{python}. Further, pip and IDLE may be used by typing
@code{pip} or @code{idle}. IDLE can also be found in Start.

All three commands are also available with version number suffixes, for
example, as @code{python3.exe} and @code{python3.x.exe} as well as
@code{python.exe} (where @code{3.x} is the specific version you want to launch,
such as 3.11). Open “Manage App Execution Aliases” through Start to
select which version of Python is associated with each command. It is
recommended to make sure that @code{pip} and @code{idle} are consistent with
whichever version of @code{python} is selected.

Virtual environments can be created with @code{python -m venv} and activated
and used as normal.

If you have installed another version of Python and added it to your
@code{PATH} variable, it will be available as @code{python.exe} rather than the
one from the Microsoft Store. To access the new installation, use
@code{python3.exe} or @code{python3.x.exe}.

The @code{py.exe} launcher will detect this Python installation, but will prefer
installations from the traditional installer.

To remove Python, open Settings and use Apps and Features, or else find
Python in Start and right-click to select Uninstall. Uninstalling will
remove all packages you installed directly into this Python installation, but
will not remove any virtual environments

@menu
* Known Issues:: 

@end menu

@node Known Issues,,,The Microsoft Store package
@anchor{using/windows known-issues}@anchor{16b2}
@subsubsection Known Issues


Because of restrictions on Microsoft Store apps, Python scripts may not have
full write access to shared locations such as @code{TEMP} and the registry.
Instead, it will write to a private copy. If your scripts must modify the
shared locations, you will need to install the full installer.

For more detail on the technical basis for these limitations, please consult
Microsoft’s documentation on packaged full-trust apps, currently available at
docs.microsoft.com/en-us/windows/msix/desktop/desktop-to-uwp-behind-the-scenes@footnote{https://docs.microsoft.com/en-us/windows/msix/desktop/desktop-to-uwp-behind-the-scenes}

@node The nuget org packages,The embeddable package,The Microsoft Store package,Using Python on Windows
@anchor{using/windows the-nuget-org-packages}@anchor{16b3}@anchor{using/windows windows-nuget}@anchor{16a7}
@subsection The nuget.org packages



New in version 3.5.2.


The nuget.org package is a reduced size Python environment intended for use on
continuous integration and build systems that do not have a system-wide
install of Python. While nuget is “the package manager for .NET”, it also works
perfectly fine for packages containing build-time tools.

Visit nuget.org@footnote{https://www.nuget.org/} for the most up-to-date information
on using nuget. What follows is a summary that is sufficient for Python
developers.

The @code{nuget.exe} command line tool may be downloaded directly from
@code{https://aka.ms/nugetclidl}, for example, using curl or PowerShell. With the
tool, the latest version of Python for 64-bit or 32-bit machines is installed
using:

@example
nuget.exe install python -ExcludeVersion -OutputDirectory .
nuget.exe install pythonx86 -ExcludeVersion -OutputDirectory .
@end example

To select a particular version, add a @code{-Version 3.x.y}. The output directory
may be changed from @code{.}, and the package will be installed into a
subdirectory. By default, the subdirectory is named the same as the package,
and without the @code{-ExcludeVersion} option this name will include the specific
version installed. Inside the subdirectory is a @code{tools} directory that
contains the Python installation:

@example
# Without -ExcludeVersion
> .\python.3.5.2\tools\python.exe -V
Python 3.5.2

# With -ExcludeVersion
> .\python\tools\python.exe -V
Python 3.5.2
@end example

In general, nuget packages are not upgradeable, and newer versions should be
installed side-by-side and referenced using the full path. Alternatively,
delete the package directory manually and install it again. Many CI systems
will do this automatically if they do not preserve files between builds.

Alongside the @code{tools} directory is a @code{build\native} directory. This
contains a MSBuild properties file @code{python.props} that can be used in a
C++ project to reference the Python install. Including the settings will
automatically use the headers and import libraries in your build.

The package information pages on nuget.org are
www.nuget.org/packages/python@footnote{https://www.nuget.org/packages/python}
for the 64-bit version and www.nuget.org/packages/pythonx86@footnote{https://www.nuget.org/packages/pythonx86} for the 32-bit version.

@node The embeddable package,Alternative bundles,The nuget org packages,Using Python on Windows
@anchor{using/windows the-embeddable-package}@anchor{16b4}@anchor{using/windows windows-embeddable}@anchor{16a8}
@subsection The embeddable package



New in version 3.5.


The embedded distribution is a ZIP file containing a minimal Python environment.
It is intended for acting as part of another application, rather than being
directly accessed by end-users.

When extracted, the embedded distribution is (almost) fully isolated from the
user’s system, including environment variables, system registry settings, and
installed packages. The standard library is included as pre-compiled and
optimized @code{.pyc} files in a ZIP, and @code{python3.dll}, @code{python37.dll},
@code{python.exe} and @code{pythonw.exe} are all provided. Tcl/tk (including all
dependants, such as Idle), pip and the Python documentation are not included.

@cartouche
@quotation Note 
The embedded distribution does not include the Microsoft C Runtime@footnote{https://www.microsoft.com/en-us/download/details.aspx?id=48145} and it is
the responsibility of the application installer to provide this. The
runtime may have already been installed on a user’s system previously or
automatically via Windows Update, and can be detected by finding
@code{ucrtbase.dll} in the system directory.
@end quotation
@end cartouche

Third-party packages should be installed by the application installer alongside
the embedded distribution. Using pip to manage dependencies as for a regular
Python installation is not supported with this distribution, though with some
care it may be possible to include and use pip for automatic updates. In
general, third-party packages should be treated as part of the application
(“vendoring”) so that the developer can ensure compatibility with newer
versions before providing updates to users.

The two recommended use cases for this distribution are described below.

@menu
* Python Application:: 
* Embedding Python:: 

@end menu

@node Python Application,Embedding Python,,The embeddable package
@anchor{using/windows python-application}@anchor{16b5}
@subsubsection Python Application


An application written in Python does not necessarily require users to be aware
of that fact. The embedded distribution may be used in this case to include a
private version of Python in an install package. Depending on how transparent it
should be (or conversely, how professional it should appear), there are two
options.

Using a specialized executable as a launcher requires some coding, but provides
the most transparent experience for users. With a customized launcher, there are
no obvious indications that the program is running on Python: icons can be
customized, company and version information can be specified, and file
associations behave properly. In most cases, a custom launcher should simply be
able to call @code{Py_Main} with a hard-coded command line.

The simpler approach is to provide a batch file or generated shortcut that
directly calls the @code{python.exe} or @code{pythonw.exe} with the required
command-line arguments. In this case, the application will appear to be Python
and not its actual name, and users may have trouble distinguishing it from other
running Python processes or file associations.

With the latter approach, packages should be installed as directories alongside
the Python executable to ensure they are available on the path. With the
specialized launcher, packages can be located in other locations as there is an
opportunity to specify the search path before launching the application.

@node Embedding Python,,Python Application,The embeddable package
@anchor{using/windows embedding-python}@anchor{16b6}
@subsubsection Embedding Python


Applications written in native code often require some form of scripting
language, and the embedded Python distribution can be used for this purpose. In
general, the majority of the application is in native code, and some part will
either invoke @code{python.exe} or directly use @code{python3.dll}. For either case,
extracting the embedded distribution to a subdirectory of the application
installation is sufficient to provide a loadable Python interpreter.

As with the application use, packages can be installed to any location as there
is an opportunity to specify search paths before initializing the interpreter.
Otherwise, there is no fundamental differences between using the embedded
distribution and a regular installation.

@node Alternative bundles,Configuring Python,The embeddable package,Using Python on Windows
@anchor{using/windows alternative-bundles}@anchor{16b7}
@subsection Alternative bundles


Besides the standard CPython distribution, there are modified packages including
additional functionality.  The following is a list of popular versions and their
key features:


@table @asis

@item ActivePython@footnote{https://www.activestate.com/activepython/}

Installer with multi-platform compatibility, documentation, PyWin32

@item Anaconda@footnote{https://www.anaconda.com/download/}

Popular scientific modules (such as numpy, scipy and pandas) and the
@code{conda} package manager.

@item Canopy@footnote{https://www.enthought.com/product/canopy/}

A “comprehensive Python analysis environment” with editors and other
development tools.

@item WinPython@footnote{https://winpython.github.io/}

Windows-specific distribution with prebuilt scientific packages and
tools for building packages.
@end table

Note that these packages may not include the latest versions of Python or
other libraries, and are not maintained or supported by the core Python team.

@node Configuring Python,UTF-8 mode,Alternative bundles,Using Python on Windows
@anchor{using/windows configuring-python}@anchor{16b8}
@subsection Configuring Python


To run Python conveniently from a command prompt, you might consider changing
some default environment variables in Windows.  While the installer provides an
option to configure the PATH and PATHEXT variables for you, this is only
reliable for a single, system-wide installation.  If you regularly use multiple
versions of Python, consider using the @ref{be5,,Python Launcher for Windows}.

@menu
* Excursus; Setting environment variables: Excursus Setting environment variables. 
* Finding the Python executable:: 

@end menu

@node Excursus Setting environment variables,Finding the Python executable,,Configuring Python
@anchor{using/windows excursus-setting-environment-variables}@anchor{16b9}@anchor{using/windows setting-envvars}@anchor{14ca}
@subsubsection Excursus: Setting environment variables


Windows allows environment variables to be configured permanently at both the
User level and the System level, or temporarily in a command prompt.

To temporarily set environment variables, open Command Prompt and use the
@code{set} command:

@example
C:\>set PATH=C:\Program Files\Python 3.9;%PATH%
C:\>set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib
C:\>python
@end example

These changes will apply to any further commands executed in that console, and
will be inherited by any applications started from the console.

Including the variable name within percent signs will expand to the existing
value, allowing you to add your new value at either the start or the end.
Modifying 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} by adding the directory containing
@code{python.exe} to the start is a common way to ensure the correct version
of Python is launched.

To permanently modify the default environment variables, click Start and search
for ‘edit environment variables’, or open System properties, Advanced
system settings and click the Environment Variables button.
In this dialog, you can add or modify User and System variables. To change
System variables, you need non-restricted access to your machine
(i.e. Administrator rights).

@cartouche
@quotation Note 
Windows will concatenate User variables @emph{after} System variables, which may
cause unexpected results when modifying 
@geindex PATH
@geindex environment variable; PATH
@code{PATH}.

The 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} variable is used by all versions of Python 2 and
Python 3, so you should not permanently configure this variable unless it
only includes code that is compatible with all of your installed Python
versions.
@end quotation
@end cartouche


@subsubheading See also



@table @asis

@item @indicateurl{https://www.microsoft.com/en-us/wdsi/help/folder-variables}

Environment variables in Windows NT

@item @indicateurl{https://technet.microsoft.com/en-us/library/cc754250.aspx}

The SET command, for temporarily modifying environment variables

@item @indicateurl{https://technet.microsoft.com/en-us/library/cc755104.aspx}

The SETX command, for permanently modifying environment variables

@item @indicateurl{https://support.microsoft.com/en-us/help/310519/how-to-manage-environment-variables-in-windows-xp}

How To Manage Environment Variables in Windows XP

@item @indicateurl{https://www.chem.gla.ac.uk/~louis/software/faq/q1.html}

Setting Environment variables, Louis J. Farrugia
@end table


@node Finding the Python executable,,Excursus Setting environment variables,Configuring Python
@anchor{using/windows finding-the-python-executable}@anchor{16ba}@anchor{using/windows windows-path-mod}@anchor{be6}
@subsubsection Finding the Python executable



Changed in version 3.5.


Besides using the automatically created start menu entry for the Python
interpreter, you might want to start Python in the command prompt. The
installer has an option to set that up for you.

On the first page of the installer, an option labelled “Add Python to PATH”
may be selected to have the installer add the install location into the
@geindex PATH
@geindex environment variable; PATH
@code{PATH}.  The location of the @code{Scripts\} folder is also added.
This allows you to type @code{python} to run the interpreter, and
@code{pip} for the package installer. Thus, you can also execute your
scripts with command line options, see @ref{b8d,,Command line} documentation.

If you don’t enable this option at install time, you can always re-run the
installer, select Modify, and enable it.  Alternatively, you can manually
modify the 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} using the directions in @ref{14ca,,Excursus; Setting environment variables}.  You
need to set your 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} environment variable to include the directory
of your Python installation, delimited by a semicolon from other entries.  An
example variable could look like this (assuming the first two entries already
existed):

@example
C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Python 3.9
@end example

@node UTF-8 mode,Python Launcher for Windows,Configuring Python,Using Python on Windows
@anchor{using/windows utf-8-mode}@anchor{16bb}@anchor{using/windows win-utf8-mode}@anchor{16bc}
@subsection UTF-8 mode



New in version 3.7.


Windows still uses legacy encodings for the system encoding (the ANSI Code
Page).  Python uses it for the default encoding of text files (e.g.
@ref{1b5,,locale.getpreferredencoding()}).

This may cause issues because UTF-8 is widely used on the internet
and most Unix systems, including WSL (Windows Subsystem for Linux).

You can use the @ref{5f2,,Python UTF-8 Mode} to change the default text
encoding to UTF-8. You can enable the @ref{5f2,,Python UTF-8 Mode} via
the @code{-X utf8} command line option, or the @code{PYTHONUTF8=1} environment
variable.  See 
@geindex PYTHONUTF8
@geindex environment variable; PYTHONUTF8
@ref{5f1,,PYTHONUTF8} for enabling UTF-8 mode, and
@ref{14ca,,Excursus; Setting environment variables} for how to modify environment variables.

When the @ref{5f2,,Python UTF-8 Mode} is enabled, you can still use the
system encoding (the ANSI Code Page) via the “mbcs” codec.

Note that adding @code{PYTHONUTF8=1} to the default environment variables
will affect all Python 3.7+ applications on your system.
If you have any Python 3.7+ applications which rely on the legacy
system encoding, it is recommended to set the environment variable
temporarily or use the @code{-X utf8} command line option.

@cartouche
@quotation Note 
Even when UTF-8 mode is disabled, Python uses UTF-8 by default
on Windows for:


@itemize *

@item 
Console I/O including standard I/O (see 
@geindex Python Enhancement Proposals; PEP 528
PEP 528@footnote{https://www.python.org/dev/peps/pep-0528} for details).

@item 
The @ref{1628,,filesystem encoding}
(see 
@geindex Python Enhancement Proposals; PEP 529
PEP 529@footnote{https://www.python.org/dev/peps/pep-0529} for details).
@end itemize
@end quotation
@end cartouche

@node Python Launcher for Windows,Finding modules,UTF-8 mode,Using Python on Windows
@anchor{using/windows launcher}@anchor{be5}@anchor{using/windows python-launcher-for-windows}@anchor{16bd}
@subsection Python Launcher for Windows



New in version 3.3.


The Python launcher for Windows is a utility which aids in locating and
executing of different Python versions.  It allows scripts (or the
command-line) to indicate a preference for a specific Python version, and
will locate and execute that version.

Unlike the 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} variable, the launcher will correctly select the most
appropriate version of Python. It will prefer per-user installations over
system-wide ones, and orders by language version rather than using the most
recently installed version.

The launcher was originally specified in 
@geindex Python Enhancement Proposals; PEP 397
PEP 397@footnote{https://www.python.org/dev/peps/pep-0397}.

@menu
* Getting started:: 
* Shebang Lines:: 
* Arguments in shebang lines:: 
* Customization:: 
* Diagnostics:: 

@end menu

@node Getting started,Shebang Lines,,Python Launcher for Windows
@anchor{using/windows getting-started}@anchor{16be}
@subsubsection Getting started


@menu
* From the command-line:: 
* Virtual environments:: 
* From a script:: 
* From file associations:: 

@end menu

@node From the command-line,Virtual environments,,Getting started
@anchor{using/windows from-the-command-line}@anchor{16bf}
@subsubsection From the command-line



Changed in version 3.6.


System-wide installations of Python 3.3 and later will put the launcher on your
@geindex PATH
@geindex environment variable; PATH
@code{PATH}. The launcher is compatible with all available versions of
Python, so it does not matter which version is installed. To check that the
launcher is available, execute the following command in Command Prompt:

@example
py
@end example

You should find that the latest version of Python you have installed is
started - it can be exited as normal, and any additional command-line
arguments specified will be sent directly to Python.

If you have multiple versions of Python installed (e.g., 2.7 and 3.11) you
will have noticed that Python 3.11 was started - to launch Python 2.7, try
the command:

@example
py -2.7
@end example

If you want the latest version of Python 2.x you have installed, try the
command:

@example
py -2
@end example

You should find the latest version of Python 2.x starts.

If you see the following error, you do not have the launcher installed:

@example
'py' is not recognized as an internal or external command,
operable program or batch file.
@end example

Per-user installations of Python do not add the launcher to 
@geindex PATH
@geindex environment variable; PATH
@code{PATH}
unless the option was selected on installation.

@node Virtual environments,From a script,From the command-line,Getting started
@anchor{using/windows virtual-environments}@anchor{16c0}
@subsubsection Virtual environments



New in version 3.5.


If the launcher is run with no explicit Python version specification, and a
virtual environment (created with the standard library @ref{12c,,venv} module or
the external @code{virtualenv} tool) active, the launcher will run the virtual
environment’s interpreter rather than the global one.  To run the global
interpreter, either deactivate the virtual environment, or explicitly specify
the global Python version.

@node From a script,From file associations,Virtual environments,Getting started
@anchor{using/windows from-a-script}@anchor{16c1}
@subsubsection From a script


Let’s create a test Python script - create a file called @code{hello.py} with the
following contents

@example
#! python
import sys
sys.stdout.write("hello from Python %s\n" % (sys.version,))
@end example

From the directory in which hello.py lives, execute the command:

@example
py hello.py
@end example

You should notice the version number of your latest Python 2.x installation
is printed.  Now try changing the first line to be:

@example
#! python3
@end example

Re-executing the command should now print the latest Python 3.x information.
As with the above command-line examples, you can specify a more explicit
version qualifier.  Assuming you have Python 2.6 installed, try changing the
first line to @code{#! python2.6} and you should find the 2.6 version
information printed.

Note that unlike interactive use, a bare “python” will use the latest
version of Python 2.x that you have installed.  This is for backward
compatibility and for compatibility with Unix, where the command @code{python}
typically refers to Python 2.

@node From file associations,,From a script,Getting started
@anchor{using/windows from-file-associations}@anchor{16c2}
@subsubsection From file associations


The launcher should have been associated with Python files (i.e. @code{.py},
@code{.pyw}, @code{.pyc} files) when it was installed.  This means that
when you double-click on one of these files from Windows explorer the launcher
will be used, and therefore you can use the same facilities described above to
have the script specify the version which should be used.

The key benefit of this is that a single launcher can support multiple Python
versions at the same time depending on the contents of the first line.

@node Shebang Lines,Arguments in shebang lines,Getting started,Python Launcher for Windows
@anchor{using/windows shebang-lines}@anchor{16c3}
@subsubsection Shebang Lines


If the first line of a script file starts with @code{#!}, it is known as a
“shebang” line.  Linux and other Unix like operating systems have native
support for such lines and they are commonly used on such systems to indicate
how a script should be executed.  This launcher allows the same facilities to
be used with Python scripts on Windows and the examples above demonstrate their
use.

To allow shebang lines in Python scripts to be portable between Unix and
Windows, this launcher supports a number of ‘virtual’ commands to specify
which interpreter to use.  The supported virtual commands are:


@itemize *

@item 
@code{/usr/bin/env python}

@item 
@code{/usr/bin/python}

@item 
@code{/usr/local/bin/python}

@item 
@code{python}
@end itemize

For example, if the first line of your script starts with

@example
#! /usr/bin/python
@end example

The default Python will be located and used.  As many Python scripts written
to work on Unix will already have this line, you should find these scripts can
be used by the launcher without modification.  If you are writing a new script
on Windows which you hope will be useful on Unix, you should use one of the
shebang lines starting with @code{/usr}.

Any of the above virtual commands can be suffixed with an explicit version
(either just the major version, or the major and minor version).
Furthermore the 32-bit version can be requested by adding “-32” after the
minor version. I.e. @code{/usr/bin/python2.7-32} will request usage of the
32-bit python 2.7.


New in version 3.7: Beginning with python launcher 3.7 it is possible to request 64-bit version
by the “-64” suffix. Furthermore it is possible to specify a major and
architecture without minor (i.e. @code{/usr/bin/python3-64}).


The @code{/usr/bin/env} form of shebang line has one further special property.
Before looking for installed Python interpreters, this form will search the
executable 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} for a Python executable. This corresponds to the
behaviour of the Unix @code{env} program, which performs a 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} search.

@node Arguments in shebang lines,Customization,Shebang Lines,Python Launcher for Windows
@anchor{using/windows arguments-in-shebang-lines}@anchor{16c4}
@subsubsection Arguments in shebang lines


The shebang lines can also specify additional options to be passed to the
Python interpreter.  For example, if you have a shebang line:

@example
#! /usr/bin/python -v
@end example

Then Python will be started with the @code{-v} option

@node Customization,Diagnostics,Arguments in shebang lines,Python Launcher for Windows
@anchor{using/windows customization}@anchor{16c5}
@subsubsection Customization


@menu
* Customization via INI files:: 
* Customizing default Python versions:: 

@end menu

@node Customization via INI files,Customizing default Python versions,,Customization
@anchor{using/windows customization-via-ini-files}@anchor{16c6}
@subsubsection Customization via INI files


Two .ini files will be searched by the launcher - @code{py.ini} in the current
user’s “application data” directory (i.e. the directory returned by calling the
Windows function @code{SHGetFolderPath} with @code{CSIDL_LOCAL_APPDATA}) and @code{py.ini} in the
same directory as the launcher. The same .ini files are used for both the
‘console’ version of the launcher (i.e. py.exe) and for the ‘windows’ version
(i.e. pyw.exe).

Customization specified in the “application directory” will have precedence over
the one next to the executable, so a user, who may not have write access to the
.ini file next to the launcher, can override commands in that global .ini file.

@node Customizing default Python versions,,Customization via INI files,Customization
@anchor{using/windows customizing-default-python-versions}@anchor{16c7}
@subsubsection Customizing default Python versions


In some cases, a version qualifier can be included in a command to dictate
which version of Python will be used by the command. A version qualifier
starts with a major version number and can optionally be followed by a period
(‘.’) and a minor version specifier. Furthermore it is possible to specify
if a 32 or 64 bit implementation shall be requested by adding “-32” or “-64”.

For example, a shebang line of @code{#!python} has no version qualifier, while
@code{#!python3} has a version qualifier which specifies only a major version.

If no version qualifiers are found in a command, the environment
variable 
@geindex PY_PYTHON
@geindex environment variable; PY_PYTHON
@code{PY_PYTHON} can be set to specify the default version
qualifier. If it is not set, the default is “3”. The variable can
specify any value that may be passed on the command line, such as “3”,
“3.7”, “3.7-32” or “3.7-64”. (Note that the “-64” option is only
available with the launcher included with Python 3.7 or newer.)

If no minor version qualifiers are found, the environment variable
@code{PY_PYTHON@{major@}} (where @code{@{major@}} is the current major version qualifier
as determined above) can be set to specify the full version. If no such option
is found, the launcher will enumerate the installed Python versions and use
the latest minor release found for the major version, which is likely,
although not guaranteed, to be the most recently installed version in that
family.

On 64-bit Windows with both 32-bit and 64-bit implementations of the same
(major.minor) Python version installed, the 64-bit version will always be
preferred. This will be true for both 32-bit and 64-bit implementations of the
launcher - a 32-bit launcher will prefer to execute a 64-bit Python installation
of the specified version if available. This is so the behavior of the launcher
can be predicted knowing only what versions are installed on the PC and
without regard to the order in which they were installed (i.e., without knowing
whether a 32 or 64-bit version of Python and corresponding launcher was
installed last). As noted above, an optional “-32” or “-64” suffix can be
used on a version specifier to change this behaviour.

Examples:


@itemize *

@item 
If no relevant options are set, the commands @code{python} and
@code{python2} will use the latest Python 2.x version installed and
the command @code{python3} will use the latest Python 3.x installed.

@item 
The commands @code{python3.1} and @code{python2.7} will not consult any
options at all as the versions are fully specified.

@item 
If @code{PY_PYTHON=3}, the commands @code{python} and @code{python3} will both use
the latest installed Python 3 version.

@item 
If @code{PY_PYTHON=3.1-32}, the command @code{python} will use the 32-bit
implementation of 3.1 whereas the command @code{python3} will use the latest
installed Python (PY_PYTHON was not considered at all as a major
version was specified.)

@item 
If @code{PY_PYTHON=3} and @code{PY_PYTHON3=3.1}, the commands
@code{python} and @code{python3} will both use specifically 3.1
@end itemize

In addition to environment variables, the same settings can be configured
in the .INI file used by the launcher.  The section in the INI file is
called @code{[defaults]} and the key name will be the same as the
environment variables without the leading @code{PY_} prefix (and note that
the key names in the INI file are case insensitive.)  The contents of
an environment variable will override things specified in the INI file.

For example:


@itemize *

@item 
Setting @code{PY_PYTHON=3.1} is equivalent to the INI file containing:
@end itemize

@example
[defaults]
python=3.1
@end example


@itemize *

@item 
Setting @code{PY_PYTHON=3} and @code{PY_PYTHON3=3.1} is equivalent to the INI file
containing:
@end itemize

@example
[defaults]
python=3
python3=3.1
@end example

@node Diagnostics,,Customization,Python Launcher for Windows
@anchor{using/windows diagnostics}@anchor{16c8}
@subsubsection Diagnostics


If an environment variable @code{PYLAUNCH_DEBUG} is set (to any value), the
launcher will print diagnostic information to stderr (i.e. to the console).
While this information manages to be simultaneously verbose @emph{and} terse, it
should allow you to see what versions of Python were located, why a
particular version was chosen and the exact command-line used to execute the
target Python.

@node Finding modules,Additional modules,Python Launcher for Windows,Using Python on Windows
@anchor{using/windows finding-modules}@anchor{75e}@anchor{using/windows id1}@anchor{16c9}
@subsection Finding modules


Python usually stores its library (and thereby your site-packages folder) in the
installation directory.  So, if you had installed Python to
@code{C:\Python\}, the default library would reside in
@code{C:\Python\Lib\} and third-party modules should be stored in
@code{C:\Python\Lib\site-packages\}.

To completely override @ref{489,,sys.path}, create a @code{._pth} file with the same
name as the DLL (@code{python37._pth}) or the executable (@code{python._pth}) and
specify one line for each path to add to @ref{489,,sys.path}. The file based on the
DLL name overrides the one based on the executable, which allows paths to be
restricted for any program loading the runtime if desired.

When the file exists, all registry and environment variables are ignored,
isolated mode is enabled, and @ref{e6,,site} is not imported unless one line in the
file specifies @code{import site}. Blank paths and lines starting with @code{#} are
ignored. Each path may be absolute or relative to the location of the file.
Import statements other than to @code{site} are not permitted, and arbitrary code
cannot be specified.

Note that @code{.pth} files (without leading underscore) will be processed normally
by the @ref{e6,,site} module when @code{import site} has been specified.

When no @code{._pth} file is found, this is how @ref{489,,sys.path} is populated on
Windows:


@itemize *

@item 
An empty entry is added at the start, which corresponds to the current
directory.

@item 
If the environment variable 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} exists, as described in
@ref{161c,,Environment variables}, its entries are added next.  Note that on Windows,
paths in this variable must be separated by semicolons, to distinguish them
from the colon used in drive identifiers (@code{C:\} etc.).

@item 
Additional “application paths” can be added in the registry as subkeys of
@code{\SOFTWARE\Python\PythonCore@{version@}\PythonPath} under both the
@code{HKEY_CURRENT_USER} and @code{HKEY_LOCAL_MACHINE} hives.  Subkeys which have
semicolon-delimited path strings as their default value will cause each path
to be added to @ref{489,,sys.path}.  (Note that all known installers only use
HKLM, so HKCU is typically empty.)

@item 
If the environment variable 
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME} is set, it is assumed as
“Python Home”.  Otherwise, the path of the main Python executable is used to
locate a “landmark file” (either @code{Lib\os.py} or @code{pythonXY.zip}) to deduce
the “Python Home”.  If a Python home is found, the relevant sub-directories
added to @ref{489,,sys.path} (@code{Lib}, @code{plat-win}, etc) are based on that
folder.  Otherwise, the core Python path is constructed from the PythonPath
stored in the registry.

@item 
If the Python Home cannot be located, no 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} is specified in
the environment, and no registry entries can be found, a default path with
relative entries is used (e.g. @code{.\Lib;.\plat-win}, etc).
@end itemize

If a @code{pyvenv.cfg} file is found alongside the main executable or in the
directory one level above the executable, the following variations apply:


@itemize *

@item 
If @code{home} is an absolute path and 
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME} is not set, this
path is used instead of the path to the main executable when deducing the
home location.
@end itemize

The end result of all this is:


@itemize *

@item 
When running @code{python.exe}, or any other .exe in the main Python
directory (either an installed version, or directly from the PCbuild
directory), the core path is deduced, and the core paths in the registry are
ignored.  Other “application paths” in the registry are always read.

@item 
When Python is hosted in another .exe (different directory, embedded via COM,
etc), the “Python Home” will not be deduced, so the core path from the
registry is used.  Other “application paths” in the registry are always read.

@item 
If Python can’t find its home and there are no registry value (frozen .exe,
some very strange installation setup) you get a path with some default, but
relative, paths.
@end itemize

For those who want to bundle Python into their application or distribution, the
following advice will prevent conflicts with other installations:


@itemize *

@item 
Include a @code{._pth} file alongside your executable containing the
directories to include. This will ignore paths listed in the registry and
environment variables, and also ignore @ref{e6,,site} unless @code{import site} is
listed.

@item 
If you are loading @code{python3.dll} or @code{python37.dll} in your own
executable, explicitly call @ref{202,,Py_SetPath()} or (at least)
@ref{203,,Py_SetProgramName()} before @ref{38d,,Py_Initialize()}.

@item 
Clear and/or overwrite 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} and set 
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME}
before launching @code{python.exe} from your application.

@item 
If you cannot use the previous suggestions (for example, you are a
distribution that allows people to run @code{python.exe} directly), ensure
that the landmark file (@code{Lib\os.py}) exists in your install directory.
(Note that it will not be detected inside a ZIP file, but a correctly named
ZIP file will be detected instead.)
@end itemize

These will ensure that the files in a system-wide installation will not take
precedence over the copy of the standard library bundled with your application.
Otherwise, your users may experience problems using your application. Note that
the first suggestion is the best, as the others may still be susceptible to
non-standard paths in the registry and user site-packages.


Changed in version 3.6: 

@quotation


@itemize *

@item 
Adds @code{._pth} file support and removes @code{applocal} option from
@code{pyvenv.cfg}.

@item 
Adds @code{pythonXX.zip} as a potential landmark when directly adjacent
to the executable.
@end itemize
@end quotation



Deprecated since version 3.6: 

@quotation

Modules specified in the registry under @code{Modules} (not @code{PythonPath})
may be imported by @ref{87b,,importlib.machinery.WindowsRegistryFinder}.
This finder is enabled on Windows in 3.6.0 and earlier, but may need to
be explicitly added to @ref{87c,,sys.meta_path} in the future.
@end quotation


@node Additional modules,Compiling Python on Windows,Finding modules,Using Python on Windows
@anchor{using/windows additional-modules}@anchor{16ca}
@subsection Additional modules


Even though Python aims to be portable among all platforms, there are features
that are unique to Windows.  A couple of modules, both in the standard library
and external, and snippets exist to use these features.

The Windows-specific standard modules are documented in
@ref{16cb,,MS Windows Specific Services}.

@menu
* PyWin32:: 
* cx_Freeze:: 
* WConio:: 

@end menu

@node PyWin32,cx_Freeze,,Additional modules
@anchor{using/windows pywin32}@anchor{16cc}
@subsubsection PyWin32


The PyWin32@footnote{https://pypi.org/project/pywin32} module by Mark Hammond
is a collection of modules for advanced Windows-specific support.  This includes
utilities for:


@itemize *

@item 
Component Object Model@footnote{https://docs.microsoft.com/en-us/windows/desktop/com/component-object-model--com--portal}
(COM)

@item 
Win32 API calls

@item 
Registry

@item 
Event log

@item 
Microsoft Foundation Classes@footnote{https://msdn.microsoft.com/en-us/library/fe1cf721%28VS.80%29.aspx} (MFC)
user interfaces
@end itemize

PythonWin@footnote{https://web.archive.org/web/20060524042422/https://www.python.org/windows/pythonwin/} is a sample MFC application
shipped with PyWin32.  It is an embeddable IDE with a built-in debugger.


@subsubheading See also



@table @asis

@item Win32 How Do I...?@footnote{http://timgolden.me.uk/python/win32_how_do_i.html}

by Tim Golden

@item Python and COM@footnote{http://www.boddie.org.uk/python/COM.html}

by David and Paul Boddie
@end table


@node cx_Freeze,WConio,PyWin32,Additional modules
@anchor{using/windows cx-freeze}@anchor{16cd}
@subsubsection cx_Freeze


cx_Freeze@footnote{https://cx-freeze.readthedocs.io/en/latest/} is a @ref{37,,distutils}
extension (see @ref{16ce,,Extending Distutils}) which wraps Python scripts into
executable Windows programs (@code{@emph{*}.exe} files).  When you have done this,
you can distribute your application without requiring your users to install
Python.

@node WConio,,cx_Freeze,Additional modules
@anchor{using/windows wconio}@anchor{16cf}
@subsubsection WConio


Since Python’s advanced terminal handling layer, @ref{2a,,curses}, is restricted to
Unix-like systems, there is a library exclusive to Windows as well: Windows
Console I/O for Python.

WConio@footnote{http://newcenturycomputers.net/projects/wconio.html} is a wrapper for
Turbo-C’s @code{CONIO.H}, used to create text user interfaces.

@node Compiling Python on Windows,Other Platforms,Additional modules,Using Python on Windows
@anchor{using/windows compiling-python-on-windows}@anchor{16d0}
@subsection Compiling Python on Windows


If you want to compile CPython yourself, first thing you should do is get the
source@footnote{https://www.python.org/downloads/source/}. You can download either the
latest release’s source or just grab a fresh checkout@footnote{https://devguide.python.org/setup/#getting-the-source-code}.

The source tree contains a build solution and project files for Microsoft
Visual Studio 2015, which is the compiler used to build the official Python
releases. These files are in the @code{PCbuild} directory.

Check @code{PCbuild/readme.txt} for general information on the build process.

For extension modules, consult @ref{16d1,,Building C and C++ Extensions on Windows}.


@subsubheading See also



@table @asis

@item Python + Windows + distutils + SWIG + gcc MinGW@footnote{http://sebsauvage.net/python/mingw.html}

or “Creating Python extensions in C/C++ with SWIG and compiling them with
MinGW gcc under Windows” or “Installing Python extension with distutils
and without Microsoft Visual C++” by Sébastien Sauvage, 2003
@end table


@node Other Platforms,,Compiling Python on Windows,Using Python on Windows
@anchor{using/windows other-platforms}@anchor{16d2}
@subsection Other Platforms


With ongoing development of Python, some platforms that used to be supported
earlier are no longer supported (due to the lack of users or developers).
Check 
@geindex Python Enhancement Proposals; PEP 11
PEP 11@footnote{https://www.python.org/dev/peps/pep-0011} for details on all unsupported platforms.


@itemize *

@item 
Windows CE@footnote{http://pythonce.sourceforge.net/} is still supported.

@item 
The Cygwin@footnote{https://cygwin.com/} installer offers to install the Python
interpreter as well (cf. Cygwin package source@footnote{ftp://ftp.uni-erlangen.de/pub/pc/gnuwin32/cygwin/mirrors/cygnus/release/python}, Maintainer releases@footnote{http://www.tishler.net/jason/software/python/})
@end itemize

See Python for Windows@footnote{https://www.python.org/downloads/windows/}
for detailed information about platforms with pre-compiled installers.

@node Using Python on a Mac,Editors and IDEs,Using Python on Windows,Python Setup and Usage
@anchor{using/mac doc}@anchor{16d3}@anchor{using/mac using-on-mac}@anchor{16d4}@anchor{using/mac using-python-on-a-mac}@anchor{16d5}
@section Using Python on a Mac


@*Author: 
Bob Savage <@email{bobsavage@@mac.com}>


Python on a Mac running macOS is in principle very similar to Python on
any other Unix platform, but there are a number of additional features such as
the IDE and the Package Manager that are worth pointing out.

@menu
* Getting and Installing MacPython:: 
* The IDE:: 
* Installing Additional Python Packages:: 
* GUI Programming on the Mac:: 
* Distributing Python Applications on the Mac:: 
* Other Resources:: 

@end menu

@node Getting and Installing MacPython,The IDE,,Using Python on a Mac
@anchor{using/mac getting-and-installing-macpython}@anchor{16d6}@anchor{using/mac getting-osx}@anchor{16d7}
@subsection Getting and Installing MacPython


macOS since version 10.8 comes with Python 2.7 pre-installed by Apple.  If you wish, you
are invited to install the most recent version of Python 3 from the Python
website (@indicateurl{https://www.python.org}).  A current “universal binary” build of Python,
which runs natively on the Mac’s new Intel and legacy PPC CPU’s, is available
there.

What you get after installing is a number of things:


@itemize *

@item 
A @code{Python 3.9} folder in your @code{Applications} folder. In here
you find IDLE, the development environment that is a standard part of official
Python distributions; and PythonLauncher, which handles double-clicking Python
scripts from the Finder.

@item 
A framework @code{/Library/Frameworks/Python.framework}, which includes the
Python executable and libraries. The installer adds this location to your shell
path. To uninstall MacPython, you can simply remove these three things. A
symlink to the Python executable is placed in /usr/local/bin/.
@end itemize

The Apple-provided build of Python is installed in
@code{/System/Library/Frameworks/Python.framework} and @code{/usr/bin/python},
respectively. You should never modify or delete these, as they are
Apple-controlled and are used by Apple- or third-party software.  Remember that
if you choose to install a newer Python version from python.org, you will have
two different but functional Python installations on your computer, so it will
be important that your paths and usages are consistent with what you want to do.

IDLE includes a help menu that allows you to access Python documentation. If you
are completely new to Python you should start reading the tutorial introduction
in that document.

If you are familiar with Python on other Unix platforms you should read the
section on running Python scripts from the Unix shell.

@menu
* How to run a Python script:: 
* Running scripts with a GUI:: 
* Configuration:: 

@end menu

@node How to run a Python script,Running scripts with a GUI,,Getting and Installing MacPython
@anchor{using/mac how-to-run-a-python-script}@anchor{16d8}
@subsubsection How to run a Python script


Your best way to get started with Python on macOS is through the IDLE
integrated development environment, see section @ref{16d9,,The IDE} and use the Help menu
when the IDE is running.

If you want to run Python scripts from the Terminal window command line or from
the Finder you first need an editor to create your script. macOS comes with a
number of standard Unix command line editors, @code{vim} and
@code{emacs} among them. If you want a more Mac-like editor,
@code{BBEdit} or @code{TextWrangler} from Bare Bones Software (see
@indicateurl{http://www.barebones.com/products/bbedit/index.html}) are good choices, as is
@code{TextMate} (see @indicateurl{https://macromates.com/}). Other editors include
@code{Gvim} (@indicateurl{http://macvim-dev.github.io/macvim/}) and @code{Aquamacs}
(@indicateurl{http://aquamacs.org/}).

To run your script from the Terminal window you must make sure that
@code{/usr/local/bin} is in your shell search path.

To run your script from the Finder you have two options:


@itemize *

@item 
Drag it to @code{PythonLauncher}

@item 
Select @code{PythonLauncher} as the default application to open your
script (or any .py script) through the finder Info window and double-click it.
@code{PythonLauncher} has various preferences to control how your script is
launched. Option-dragging allows you to change these for one invocation, or use
its Preferences menu to change things globally.
@end itemize

@node Running scripts with a GUI,Configuration,How to run a Python script,Getting and Installing MacPython
@anchor{using/mac osx-gui-scripts}@anchor{16da}@anchor{using/mac running-scripts-with-a-gui}@anchor{16db}
@subsubsection Running scripts with a GUI


With older versions of Python, there is one macOS quirk that you need to be
aware of: programs that talk to the Aqua window manager (in other words,
anything that has a GUI) need to be run in a special way. Use @code{pythonw}
instead of @code{python} to start such scripts.

With Python 3.9, you can use either @code{python} or @code{pythonw}.

@node Configuration,,Running scripts with a GUI,Getting and Installing MacPython
@anchor{using/mac configuration}@anchor{16dc}
@subsubsection Configuration


Python on macOS honors all standard Unix environment variables such as
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH}, but setting these variables for programs started from the
Finder is non-standard as the Finder does not read your @code{.profile} or
@code{.cshrc} at startup. You need to create a file
@code{~/.MacOSX/environment.plist}. See Apple’s Technical Document QA1067 for
details.

For more information on installation Python packages in MacPython, see section
@ref{16dd,,Installing Additional Python Packages}.

@node The IDE,Installing Additional Python Packages,Getting and Installing MacPython,Using Python on a Mac
@anchor{using/mac ide}@anchor{16d9}@anchor{using/mac the-ide}@anchor{16de}
@subsection The IDE


MacPython ships with the standard IDLE development environment. A good
introduction to using IDLE can be found at
@indicateurl{http://www.hashcollision.org/hkn/python/idle_intro/index.html}.

@node Installing Additional Python Packages,GUI Programming on the Mac,The IDE,Using Python on a Mac
@anchor{using/mac installing-additional-python-packages}@anchor{16df}@anchor{using/mac mac-package-manager}@anchor{16dd}
@subsection Installing Additional Python Packages


There are several methods to install additional Python packages:


@itemize *

@item 
Packages can be installed via the standard Python distutils mode (@code{python
setup.py install}).

@item 
Many packages can also be installed via the @code{setuptools} extension
or @code{pip} wrapper, see @indicateurl{https://pip.pypa.io/}.
@end itemize

@node GUI Programming on the Mac,Distributing Python Applications on the Mac,Installing Additional Python Packages,Using Python on a Mac
@anchor{using/mac gui-programming-on-the-mac}@anchor{16e0}
@subsection GUI Programming on the Mac


There are several options for building GUI applications on the Mac with Python.

@emph{PyObjC} is a Python binding to Apple’s Objective-C/Cocoa framework, which is
the foundation of most modern Mac development. Information on PyObjC is
available from @indicateurl{https://pypi.org/project/pyobjc/}.

The standard Python GUI toolkit is @ref{10c,,tkinter}, based on the cross-platform
Tk toolkit (@indicateurl{https://www.tcl.tk}). An Aqua-native version of Tk is bundled with OS
X by Apple, and the latest version can be downloaded and installed from
@indicateurl{https://www.activestate.com}; it can also be built from source.

@emph{wxPython} is another popular cross-platform GUI toolkit that runs natively on
macOS. Packages and documentation are available from @indicateurl{https://www.wxpython.org}.

@emph{PyQt} is another popular cross-platform GUI toolkit that runs natively on
macOS. More information can be found at
@indicateurl{https://riverbankcomputing.com/software/pyqt/intro}.

@node Distributing Python Applications on the Mac,Other Resources,GUI Programming on the Mac,Using Python on a Mac
@anchor{using/mac distributing-python-applications-on-the-mac}@anchor{16e1}
@subsection Distributing Python Applications on the Mac


The standard tool for deploying standalone Python applications on the Mac is
@code{py2app}. More information on installing and using py2app can be found
at @indicateurl{http://undefined.org/python/#py2app}.

@node Other Resources,,Distributing Python Applications on the Mac,Using Python on a Mac
@anchor{using/mac other-resources}@anchor{16e2}
@subsection Other Resources


The MacPython mailing list is an excellent support resource for Python users and
developers on the Mac:

@indicateurl{https://www.python.org/community/sigs/current/pythonmac-sig/}

Another useful resource is the MacPython wiki:

@indicateurl{https://wiki.python.org/moin/MacPython}

@node Editors and IDEs,,Using Python on a Mac,Python Setup and Usage
@anchor{using/editors doc}@anchor{16e3}@anchor{using/editors editors}@anchor{16e4}@anchor{using/editors editors-and-ides}@anchor{16e5}
@section Editors and IDEs


There are a number of IDEs that support Python programming language.
Many editors and IDEs provide syntax highlighting, debugging tools, and 
@geindex Python Enhancement Proposals; PEP 8
PEP 8@footnote{https://www.python.org/dev/peps/pep-0008} checks.

Please go to Python Editors@footnote{https://wiki.python.org/moin/PythonEditors} and
Integrated Development Environments@footnote{https://wiki.python.org/moin/IntegratedDevelopmentEnvironments}
for a comprehensive list.

@node The Python Language Reference,The Python Standard Library,Python Setup and Usage,Top
@anchor{reference/index doc}@anchor{16e6}@anchor{reference/index reference-index}@anchor{14bd}@anchor{reference/index the-python-language-reference}@anchor{16e7}
@chapter The Python Language Reference


This reference manual describes the syntax and “core semantics” of the
language. It is terse, but attempts to be exact and complete. The semantics of
non-essential built-in object types and of the built-in functions and modules
are described in @ref{14bc,,The Python Standard Library}. For an informal introduction to the
language, see @ref{14bb,,The Python Tutorial}. For C or C++ programmers, two additional
manuals exist: @ref{14be,,Extending and Embedding the Python Interpreter} describes the high-level picture of how to
write a Python extension module, and the @ref{14bf,,Python/C API Reference Manual} describes the
interfaces available to C/C++ programmers in detail.

@menu
* Introduction: Introduction<5>. 
* Lexical analysis:: 
* Data model:: 
* Execution model:: 
* The import system:: 
* Expressions:: 
* Simple statements:: 
* Compound statements:: 
* Top-level components:: 
* Full Grammar specification:: 

@end menu

@node Introduction<5>,Lexical analysis,,The Python Language Reference
@anchor{reference/introduction doc}@anchor{16e8}@anchor{reference/introduction id1}@anchor{16e9}@anchor{reference/introduction introduction}@anchor{16ea}
@section Introduction


This reference manual describes the Python programming language. It is not
intended as a tutorial.

While I am trying to be as precise as possible, I chose to use English rather
than formal specifications for everything except syntax and lexical analysis.
This should make the document more understandable to the average reader, but
will leave room for ambiguities. Consequently, if you were coming from Mars and
tried to re-implement Python from this document alone, you might have to guess
things and in fact you would probably end up implementing quite a different
language. On the other hand, if you are using Python and wonder what the precise
rules about a particular area of the language are, you should definitely be able
to find them here. If you would like to see a more formal definition of the
language, maybe you could volunteer your time — or invent a cloning machine
:-).

It is dangerous to add too many implementation details to a language reference
document — the implementation may change, and other implementations of the
same language may work differently.  On the other hand, CPython is the one
Python implementation in widespread use (although alternate implementations
continue to gain support), and its particular quirks are sometimes worth being
mentioned, especially where the implementation imposes additional limitations.
Therefore, you’ll find short “implementation notes” sprinkled throughout the
text.

Every Python implementation comes with a number of built-in and standard
modules.  These are documented in @ref{14bc,,The Python Standard Library}.  A few built-in modules
are mentioned when they interact in a significant way with the language
definition.

@menu
* Alternate Implementations:: 
* Notation:: 

@end menu

@node Alternate Implementations,Notation,,Introduction<5>
@anchor{reference/introduction alternate-implementations}@anchor{16eb}@anchor{reference/introduction implementations}@anchor{15fa}
@subsection Alternate Implementations


Though there is one Python implementation which is by far the most popular,
there are some alternate implementations which are of particular interest to
different audiences.

Known implementations include:


@table @asis

@item CPython

This is the original and most-maintained implementation of Python, written in C.
New language features generally appear here first.

@item Jython

Python implemented in Java.  This implementation can be used as a scripting
language for Java applications, or can be used to create applications using the
Java class libraries.  It is also often used to create tests for Java libraries.
More information can be found at the Jython website@footnote{http://www.jython.org/}.

@item Python for .NET

This implementation actually uses the CPython implementation, but is a managed
.NET application and makes .NET libraries available.  It was created by Brian
Lloyd.  For more information, see the Python for .NET home page@footnote{https://pythonnet.github.io/}.

@item IronPython

An alternate Python for .NET.  Unlike Python.NET, this is a complete Python
implementation that generates IL, and compiles Python code directly to .NET
assemblies.  It was created by Jim Hugunin, the original creator of Jython.  For
more information, see the IronPython website@footnote{http://ironpython.net/}.

@item PyPy

An implementation of Python written completely in Python. It supports several
advanced features not found in other implementations like stackless support
and a Just in Time compiler. One of the goals of the project is to encourage
experimentation with the language itself by making it easier to modify the
interpreter (since it is written in Python).  Additional information is
available on the PyPy project's home page@footnote{http://pypy.org/}.
@end table

Each of these implementations varies in some way from the language as documented
in this manual, or introduces specific information beyond what’s covered in the
standard Python documentation.  Please refer to the implementation-specific
documentation to determine what else you need to know about the specific
implementation you’re using.

@node Notation,,Alternate Implementations,Introduction<5>
@anchor{reference/introduction id2}@anchor{16ec}@anchor{reference/introduction notation}@anchor{16ed}
@subsection Notation


@geindex BNF
@geindex grammar
@geindex syntax
@geindex notation

The descriptions of lexical analysis and syntax use a modified BNF grammar
notation.  This uses the following style of definition:

@example
@anchor{reference/introduction grammar-token-notation-name}@anchor{16ee}@anchor{reference/introduction grammar-token-name}@anchor{16ef}name      ::= lc_letter (lc_letter | "_")*
@anchor{reference/introduction grammar-token-notation-lc_letter}@anchor{16f0}@anchor{reference/introduction grammar-token-lc-letter}@anchor{16f1}lc_letter ::= "a"..."z"
@end example

The first line says that a @code{name} is an @code{lc_letter} followed by a sequence
of zero or more @code{lc_letter}s and underscores.  An @code{lc_letter} in turn is
any of the single characters @code{'a'} through @code{'z'}.  (This rule is actually
adhered to for the names defined in lexical and grammar rules in this document.)

Each rule begins with a name (which is the name defined by the rule) and
@code{::=}.  A vertical bar (@code{|}) is used to separate alternatives; it is the
least binding operator in this notation.  A star (@code{*}) means zero or more
repetitions of the preceding item; likewise, a plus (@code{+}) means one or more
repetitions, and a phrase enclosed in square brackets (@code{[ ]}) means zero or
one occurrences (in other words, the enclosed phrase is optional).  The @code{*}
and @code{+} operators bind as tightly as possible; parentheses are used for
grouping.  Literal strings are enclosed in quotes.  White space is only
meaningful to separate tokens. Rules are normally contained on a single line;
rules with many alternatives may be formatted alternatively with each line after
the first beginning with a vertical bar.

@geindex lexical definitions
@geindex ASCII

In lexical definitions (as the example above), two more conventions are used:
Two literal characters separated by three dots mean a choice of any single
character in the given (inclusive) range of ASCII characters.  A phrase between
angular brackets (@code{<...>}) gives an informal description of the symbol
defined; e.g., this could be used to describe the notion of ‘control character’
if needed.

Even though the notation used is almost the same, there is a big difference
between the meaning of lexical and syntactic definitions: a lexical definition
operates on the individual characters of the input source, while a syntax
definition operates on the stream of tokens generated by the lexical analysis.
All uses of BNF in the next chapter (“Lexical Analysis”) are lexical
definitions; uses in subsequent chapters are syntactic definitions.

@node Lexical analysis,Data model,Introduction<5>,The Python Language Reference
@anchor{reference/lexical_analysis doc}@anchor{16f2}@anchor{reference/lexical_analysis lexical}@anchor{16f3}@anchor{reference/lexical_analysis lexical-analysis}@anchor{16f4}
@section Lexical analysis


@geindex lexical analysis
@geindex parser
@geindex token

A Python program is read by a @emph{parser}.  Input to the parser is a stream of
@emph{tokens}, generated by the @emph{lexical analyzer}.  This chapter describes how the
lexical analyzer breaks a file into tokens.

Python reads program text as Unicode code points; the encoding of a source file
can be given by an encoding declaration and defaults to UTF-8, see 
@geindex Python Enhancement Proposals; PEP 3120
PEP 3120@footnote{https://www.python.org/dev/peps/pep-3120}
for details.  If the source file cannot be decoded, a @ref{21a,,SyntaxError} is
raised.

@menu
* Line structure:: 
* Other tokens:: 
* Identifiers and keywords:: 
* Literals:: 
* Operators:: 
* Delimiters:: 

@end menu

@node Line structure,Other tokens,,Lexical analysis
@anchor{reference/lexical_analysis id1}@anchor{16f5}@anchor{reference/lexical_analysis line-structure}@anchor{16f6}
@subsection Line structure


@geindex line structure

A Python program is divided into a number of @emph{logical lines}.

@menu
* Logical lines:: 
* Physical lines:: 
* Comments:: 
* Encoding declarations:: 
* Explicit line joining:: 
* Implicit line joining:: 
* Blank lines:: 
* Indentation:: 
* Whitespace between tokens:: 

@end menu

@node Logical lines,Physical lines,,Line structure
@anchor{reference/lexical_analysis id2}@anchor{16f7}@anchor{reference/lexical_analysis logical-lines}@anchor{16f8}
@subsubsection Logical lines


@geindex logical line
@geindex physical line
@geindex line joining
@geindex NEWLINE token

The end of a logical line is represented by the token NEWLINE.  Statements
cannot cross logical line boundaries except where NEWLINE is allowed by the
syntax (e.g., between statements in compound statements). A logical line is
constructed from one or more @emph{physical lines} by following the explicit or
implicit @emph{line joining} rules.

@node Physical lines,Comments,Logical lines,Line structure
@anchor{reference/lexical_analysis id3}@anchor{16f9}@anchor{reference/lexical_analysis physical-lines}@anchor{16fa}
@subsubsection Physical lines


A physical line is a sequence of characters terminated by an end-of-line
sequence.  In source files and strings, any of the standard platform line
termination sequences can be used - the Unix form using ASCII LF (linefeed),
the Windows form using the ASCII sequence CR LF (return followed by linefeed),
or the old Macintosh form using the ASCII CR (return) character.  All of these
forms can be used equally, regardless of platform. The end of input also serves
as an implicit terminator for the final physical line.

When embedding Python, source code strings should be passed to Python APIs using
the standard C conventions for newline characters (the @code{\n} character,
representing ASCII LF, is the line terminator).

@node Comments,Encoding declarations,Physical lines,Line structure
@anchor{reference/lexical_analysis comments}@anchor{16fb}@anchor{reference/lexical_analysis id4}@anchor{16fc}
@subsubsection Comments


@geindex comment
@geindex hash character
@geindex # (hash); comment

A comment starts with a hash character (@code{#}) that is not part of a string
literal, and ends at the end of the physical line.  A comment signifies the end
of the logical line unless the implicit line joining rules are invoked. Comments
are ignored by the syntax.

@node Encoding declarations,Explicit line joining,Comments,Line structure
@anchor{reference/lexical_analysis encoding-declarations}@anchor{16fd}@anchor{reference/lexical_analysis encodings}@anchor{16fe}
@subsubsection Encoding declarations


@geindex source character set
@geindex encoding declarations (source file)
@geindex # (hash); source encoding declaration

If a comment in the first or second line of the Python script matches the
regular expression @code{coding[=:]\s*([-\w.]+)}, this comment is processed as an
encoding declaration; the first group of this expression names the encoding of
the source code file. The encoding declaration must appear on a line of its
own. If it is the second line, the first line must also be a comment-only line.
The recommended forms of an encoding expression are

@example
# -*- coding: <encoding-name> -*-
@end example

which is recognized also by GNU Emacs, and

@example
# vim:fileencoding=<encoding-name>
@end example

which is recognized by Bram Moolenaar’s VIM.

If no encoding declaration is found, the default encoding is UTF-8.  In
addition, if the first bytes of the file are the UTF-8 byte-order mark
(@code{b'\xef\xbb\xbf'}), the declared file encoding is UTF-8 (this is supported,
among others, by Microsoft’s @code{notepad}).

If an encoding is declared, the encoding name must be recognized by Python. The
encoding is used for all lexical analysis, including string literals, comments
and identifiers.

@c XXX there should be a list of supported encodings.

@node Explicit line joining,Implicit line joining,Encoding declarations,Line structure
@anchor{reference/lexical_analysis explicit-joining}@anchor{16ff}@anchor{reference/lexical_analysis explicit-line-joining}@anchor{1700}
@subsubsection Explicit line joining


@geindex physical line
@geindex line joining
@geindex line continuation
@geindex backslash character

Two or more physical lines may be joined into logical lines using backslash
characters (@code{\}), as follows: when a physical line ends in a backslash that is
not part of a string literal or comment, it is joined with the following forming
a single logical line, deleting the backslash and the following end-of-line
character.  For example:

@example
if 1900 < year < 2100 and 1 <= month <= 12 \
   and 1 <= day <= 31 and 0 <= hour < 24 \
   and 0 <= minute < 60 and 0 <= second < 60:   # Looks like a valid date
        return 1
@end example

A line ending in a backslash cannot carry a comment.  A backslash does not
continue a comment.  A backslash does not continue a token except for string
literals (i.e., tokens other than string literals cannot be split across
physical lines using a backslash).  A backslash is illegal elsewhere on a line
outside a string literal.

@node Implicit line joining,Blank lines,Explicit line joining,Line structure
@anchor{reference/lexical_analysis implicit-joining}@anchor{1701}@anchor{reference/lexical_analysis implicit-line-joining}@anchor{1702}
@subsubsection Implicit line joining


Expressions in parentheses, square brackets or curly braces can be split over
more than one physical line without using backslashes. For example:

@example
month_names = ['Januari', 'Februari', 'Maart',      # These are the
               'April',   'Mei',      'Juni',       # Dutch names
               'Juli',    'Augustus', 'September',  # for the months
               'Oktober', 'November', 'December']   # of the year
@end example

Implicitly continued lines can carry comments.  The indentation of the
continuation lines is not important.  Blank continuation lines are allowed.
There is no NEWLINE token between implicit continuation lines.  Implicitly
continued lines can also occur within triple-quoted strings (see below); in that
case they cannot carry comments.

@node Blank lines,Indentation,Implicit line joining,Line structure
@anchor{reference/lexical_analysis blank-lines}@anchor{1703}@anchor{reference/lexical_analysis id5}@anchor{1704}
@subsubsection Blank lines


@geindex blank line

A logical line that contains only spaces, tabs, formfeeds and possibly a
comment, is ignored (i.e., no NEWLINE token is generated).  During interactive
input of statements, handling of a blank line may differ depending on the
implementation of the read-eval-print loop.  In the standard interactive
interpreter, an entirely blank logical line (i.e. one containing not even
whitespace or a comment) terminates a multi-line statement.

@node Indentation,Whitespace between tokens,Blank lines,Line structure
@anchor{reference/lexical_analysis id6}@anchor{1705}@anchor{reference/lexical_analysis indentation}@anchor{1706}
@subsubsection Indentation


@geindex indentation
@geindex leading whitespace
@geindex space
@geindex tab
@geindex grouping
@geindex statement grouping

Leading whitespace (spaces and tabs) at the beginning of a logical line is used
to compute the indentation level of the line, which in turn is used to determine
the grouping of statements.

Tabs are replaced (from left to right) by one to eight spaces such that the
total number of characters up to and including the replacement is a multiple of
eight (this is intended to be the same rule as used by Unix).  The total number
of spaces preceding the first non-blank character then determines the line’s
indentation.  Indentation cannot be split over multiple physical lines using
backslashes; the whitespace up to the first backslash determines the
indentation.

Indentation is rejected as inconsistent if a source file mixes tabs and spaces
in a way that makes the meaning dependent on the worth of a tab in spaces; a
@ref{1068,,TabError} is raised in that case.

@strong{Cross-platform compatibility note:} because of the nature of text editors on
non-UNIX platforms, it is unwise to use a mixture of spaces and tabs for the
indentation in a single source file.  It should also be noted that different
platforms may explicitly limit the maximum indentation level.

A formfeed character may be present at the start of the line; it will be ignored
for the indentation calculations above.  Formfeed characters occurring elsewhere
in the leading whitespace have an undefined effect (for instance, they may reset
the space count to zero).

@geindex INDENT token
@geindex DEDENT token

The indentation levels of consecutive lines are used to generate INDENT and
DEDENT tokens, using a stack, as follows.

Before the first line of the file is read, a single zero is pushed on the stack;
this will never be popped off again.  The numbers pushed on the stack will
always be strictly increasing from bottom to top.  At the beginning of each
logical line, the line’s indentation level is compared to the top of the stack.
If it is equal, nothing happens. If it is larger, it is pushed on the stack, and
one INDENT token is generated.  If it is smaller, it @emph{must} be one of the
numbers occurring on the stack; all numbers on the stack that are larger are
popped off, and for each number popped off a DEDENT token is generated.  At the
end of the file, a DEDENT token is generated for each number remaining on the
stack that is larger than zero.

Here is an example of a correctly (though confusingly) indented piece of Python
code:

@example
def perm(l):
        # Compute the list of all permutations of l
    if len(l) <= 1:
                  return [l]
    r = []
    for i in range(len(l)):
             s = l[:i] + l[i+1:]
             p = perm(s)
             for x in p:
              r.append(l[i:i+1] + x)
    return r
@end example

The following example shows various indentation errors:

@example
 def perm(l):                       # error: first line indented
for i in range(len(l)):             # error: not indented
    s = l[:i] + l[i+1:]
        p = perm(l[:i] + l[i+1:])   # error: unexpected indent
        for x in p:
                r.append(l[i:i+1] + x)
            return r                # error: inconsistent dedent
@end example

(Actually, the first three errors are detected by the parser; only the last
error is found by the lexical analyzer — the indentation of @code{return r} does
not match a level popped off the stack.)

@node Whitespace between tokens,,Indentation,Line structure
@anchor{reference/lexical_analysis whitespace}@anchor{1707}@anchor{reference/lexical_analysis whitespace-between-tokens}@anchor{1708}
@subsubsection Whitespace between tokens


Except at the beginning of a logical line or in string literals, the whitespace
characters space, tab and formfeed can be used interchangeably to separate
tokens.  Whitespace is needed between two tokens only if their concatenation
could otherwise be interpreted as a different token (e.g., ab is one token, but
a b is two tokens).

@node Other tokens,Identifiers and keywords,Line structure,Lexical analysis
@anchor{reference/lexical_analysis id7}@anchor{1709}@anchor{reference/lexical_analysis other-tokens}@anchor{170a}
@subsection Other tokens


Besides NEWLINE, INDENT and DEDENT, the following categories of tokens exist:
@emph{identifiers}, @emph{keywords}, @emph{literals}, @emph{operators}, and @emph{delimiters}. Whitespace
characters (other than line terminators, discussed earlier) are not tokens, but
serve to delimit tokens. Where ambiguity exists, a token comprises the longest
possible string that forms a legal token, when read from left to right.

@node Identifiers and keywords,Literals,Other tokens,Lexical analysis
@anchor{reference/lexical_analysis identifiers}@anchor{170b}@anchor{reference/lexical_analysis identifiers-and-keywords}@anchor{170c}
@subsection Identifiers and keywords


@geindex identifier
@geindex name

Identifiers (also referred to as @emph{names}) are described by the following lexical
definitions.

The syntax of identifiers in Python is based on the Unicode standard annex
UAX-31, with elaboration and changes as defined below; see also 
@geindex Python Enhancement Proposals; PEP 3131
PEP 3131@footnote{https://www.python.org/dev/peps/pep-3131} for
further details.

Within the ASCII range (U+0001..U+007F), the valid characters for identifiers
are the same as in Python 2.x: the uppercase and lowercase letters @code{A} through
@code{Z}, the underscore @code{_} and, except for the first character, the digits
@code{0} through @code{9}.

Python 3.0 introduces additional characters from outside the ASCII range (see
@geindex Python Enhancement Proposals; PEP 3131
PEP 3131@footnote{https://www.python.org/dev/peps/pep-3131}).  For these characters, the classification uses the version of the
Unicode Character Database as included in the @ref{121,,unicodedata} module.

Identifiers are unlimited in length.  Case is significant.

@example
@anchor{reference/lexical_analysis grammar-token-python-grammar-identifier}@anchor{170d}@anchor{reference/lexical_analysis grammar-token-identifier}@anchor{170e}identifier   ::= xid_start xid_continue*
@anchor{reference/lexical_analysis grammar-token-python-grammar-id_start}@anchor{170f}@anchor{reference/lexical_analysis grammar-token-id-start}@anchor{1710}id_start     ::= <all characters in general categories Lu, Ll, Lt, Lm, Lo, Nl, the underscore, and characters with the Other_ID_Start property>
@anchor{reference/lexical_analysis grammar-token-python-grammar-id_continue}@anchor{1711}@anchor{reference/lexical_analysis grammar-token-id-continue}@anchor{1712}id_continue  ::= <all characters in id_start, plus characters in the categories Mn, Mc, Nd, Pc and others with the Other_ID_Continue property>
@anchor{reference/lexical_analysis grammar-token-python-grammar-xid_start}@anchor{1713}@anchor{reference/lexical_analysis grammar-token-xid-start}@anchor{1714}xid_start    ::= <all characters in id_start whose NFKC normalization is in "id_start xid_continue*">
@anchor{reference/lexical_analysis grammar-token-python-grammar-xid_continue}@anchor{1715}@anchor{reference/lexical_analysis grammar-token-xid-continue}@anchor{1716}xid_continue ::= <all characters in id_continue whose NFKC normalization is in "id_continue*">
@end example

The Unicode category codes mentioned above stand for:


@itemize *

@item 
@emph{Lu} - uppercase letters

@item 
@emph{Ll} - lowercase letters

@item 
@emph{Lt} - titlecase letters

@item 
@emph{Lm} - modifier letters

@item 
@emph{Lo} - other letters

@item 
@emph{Nl} - letter numbers

@item 
@emph{Mn} - nonspacing marks

@item 
@emph{Mc} - spacing combining marks

@item 
@emph{Nd} - decimal numbers

@item 
@emph{Pc} - connector punctuations

@item 
@emph{Other_ID_Start} - explicit list of characters in PropList.txt@footnote{https://www.unicode.org/Public/14.0.0/ucd/PropList.txt} to support backwards
compatibility

@item 
@emph{Other_ID_Continue} - likewise
@end itemize

All identifiers are converted into the normal form NFKC while parsing; comparison
of identifiers is based on NFKC.

A non-normative HTML file listing all valid identifier characters for Unicode
14.0.0 can be found at
@indicateurl{https://www.unicode.org/Public/14.0.0/ucd/DerivedCoreProperties.txt}

@menu
* Keywords:: 
* Soft Keywords:: 
* Reserved classes of identifiers:: 

@end menu

@node Keywords,Soft Keywords,,Identifiers and keywords
@anchor{reference/lexical_analysis id8}@anchor{1717}@anchor{reference/lexical_analysis keywords}@anchor{1718}
@subsubsection Keywords


@geindex keyword
@geindex reserved word

The following identifiers are used as reserved words, or @emph{keywords} of the
language, and cannot be used as ordinary identifiers.  They must be spelled
exactly as written here:

@example
False      await      else       import     pass
None       break      except     in         raise
True       class      finally    is         return
and        continue   for        lambda     try
as         def        from       nonlocal   while
assert     del        global     not        with
async      elif       if         or         yield
@end example

@node Soft Keywords,Reserved classes of identifiers,Keywords,Identifiers and keywords
@anchor{reference/lexical_analysis id9}@anchor{1719}@anchor{reference/lexical_analysis soft-keywords}@anchor{28f}
@subsubsection Soft Keywords


@geindex soft keyword
@geindex keyword


New in version 3.10.


Some identifiers are only reserved under specific contexts. These are known as
@emph{soft keywords}.  The identifiers @code{match}, @code{case} and @code{_} can
syntactically act as keywords in contexts related to the pattern matching
statement, but this distinction is done at the parser level, not when
tokenizing.

As soft keywords, their use with pattern matching is possible while still
preserving compatibility with existing code that uses @code{match}, @code{case} and @code{_} as
identifier names.

@geindex _@comma{} identifiers
@geindex __@comma{} identifiers

@node Reserved classes of identifiers,,Soft Keywords,Identifiers and keywords
@anchor{reference/lexical_analysis id-classes}@anchor{171a}@anchor{reference/lexical_analysis reserved-classes-of-identifiers}@anchor{171b}
@subsubsection Reserved classes of identifiers


Certain classes of identifiers (besides keywords) have special meanings.  These
classes are identified by the patterns of leading and trailing underscore
characters:


@table @asis

@item @code{_*}

Not imported by @code{from module import *}.

@item @code{_}

In a @code{case} pattern within a @ref{290,,match} statement, @code{_} is a
@ref{28f,,soft keyword} that denotes a
@ref{291,,wildcard}.

Separately, the interactive interpreter makes the result of the last evaluation
available in the variable @code{_}.
(It is stored in the @ref{11,,builtins} module, alongside built-in
functions like @code{print}.)

Elsewhere, @code{_} is a regular identifier. It is often used to name
“special” items, but it is not special to Python itself.

@cartouche
@quotation Note 
The name @code{_} is often used in conjunction with internationalization;
refer to the documentation for the @ref{83,,gettext} module for more
information on this convention.

It is also commonly used for unused variables.
@end quotation
@end cartouche

@item @code{__*__}

System-defined names, informally known as “dunder” names. These names are
defined by the interpreter and its implementation (including the standard library).
Current system names are discussed in the @ref{7b4,,Special method names} section and elsewhere.
More will likely be defined in future versions of Python.  @emph{Any} use of @code{__*__} names,
in any context, that does not follow explicitly documented use, is subject to
breakage without warning.

@item @code{__*}

Class-private names.  Names in this category, when used within the context of a
class definition, are re-written to use a mangled form to help avoid name
clashes between “private” attributes of base and derived classes. See section
@ref{171c,,Identifiers (Names)}.
@end table

@node Literals,Operators,Identifiers and keywords,Lexical analysis
@anchor{reference/lexical_analysis id10}@anchor{171d}@anchor{reference/lexical_analysis literals}@anchor{171e}
@subsection Literals


@geindex literal
@geindex constant

Literals are notations for constant values of some built-in types.

@geindex string literal
@geindex bytes literal
@geindex ASCII
@geindex ' (single quote); string literal
@geindex " (double quote); string literal
@geindex u'; string literal
@geindex u"; string literal

@menu
* String and Bytes literals:: 
* String literal concatenation:: 
* Formatted string literals:: 
* Numeric literals:: 
* Integer literals:: 
* Floating point literals:: 
* Imaginary literals:: 

@end menu

@node String and Bytes literals,String literal concatenation,,Literals
@anchor{reference/lexical_analysis string-and-bytes-literals}@anchor{171f}@anchor{reference/lexical_analysis strings}@anchor{1720}
@subsubsection String and Bytes literals


String literals are described by the following lexical definitions:

@example
@anchor{reference/lexical_analysis grammar-token-python-grammar-stringliteral}@anchor{1721}@anchor{reference/lexical_analysis grammar-token-stringliteral}@anchor{1722}stringliteral   ::= [stringprefix](shortstring | longstring)
@anchor{reference/lexical_analysis grammar-token-python-grammar-stringprefix}@anchor{1723}@anchor{reference/lexical_analysis grammar-token-stringprefix}@anchor{1724}stringprefix    ::= "r" | "u" | "R" | "U" | "f" | "F"
                    | "fr" | "Fr" | "fR" | "FR" | "rf" | "rF" | "Rf" | "RF"
@anchor{reference/lexical_analysis grammar-token-python-grammar-shortstring}@anchor{1725}@anchor{reference/lexical_analysis grammar-token-shortstring}@anchor{1726}shortstring     ::= "'" shortstringitem* "'" | '"' shortstringitem* '"'
@anchor{reference/lexical_analysis grammar-token-python-grammar-longstring}@anchor{1727}@anchor{reference/lexical_analysis grammar-token-longstring}@anchor{1728}longstring      ::= "'@w{'}'" longstringitem* "'@w{'}'" | '"""' longstringitem* '"""'
@anchor{reference/lexical_analysis grammar-token-python-grammar-shortstringitem}@anchor{1729}@anchor{reference/lexical_analysis grammar-token-shortstringitem}@anchor{172a}shortstringitem ::= shortstringchar | stringescapeseq
@anchor{reference/lexical_analysis grammar-token-python-grammar-longstringitem}@anchor{172b}@anchor{reference/lexical_analysis grammar-token-longstringitem}@anchor{172c}longstringitem  ::= longstringchar | stringescapeseq
@anchor{reference/lexical_analysis grammar-token-python-grammar-shortstringchar}@anchor{172d}@anchor{reference/lexical_analysis grammar-token-shortstringchar}@anchor{172e}shortstringchar ::= <any source character except "\" or newline or the quote>
@anchor{reference/lexical_analysis grammar-token-python-grammar-longstringchar}@anchor{172f}@anchor{reference/lexical_analysis grammar-token-longstringchar}@anchor{1730}longstringchar  ::= <any source character except "\">
@anchor{reference/lexical_analysis grammar-token-python-grammar-stringescapeseq}@anchor{1731}@anchor{reference/lexical_analysis grammar-token-stringescapeseq}@anchor{1732}stringescapeseq ::= "\" <any source character>
@end example

@example
@anchor{reference/lexical_analysis grammar-token-python-grammar-bytesliteral}@anchor{1733}@anchor{reference/lexical_analysis grammar-token-bytesliteral}@anchor{1734}bytesliteral   ::= bytesprefix(shortbytes | longbytes)
@anchor{reference/lexical_analysis grammar-token-python-grammar-bytesprefix}@anchor{1735}@anchor{reference/lexical_analysis grammar-token-bytesprefix}@anchor{1736}bytesprefix    ::= "b" | "B" | "br" | "Br" | "bR" | "BR" | "rb" | "rB" | "Rb" | "RB"
@anchor{reference/lexical_analysis grammar-token-python-grammar-shortbytes}@anchor{1737}@anchor{reference/lexical_analysis grammar-token-shortbytes}@anchor{1738}shortbytes     ::= "'" shortbytesitem* "'" | '"' shortbytesitem* '"'
@anchor{reference/lexical_analysis grammar-token-python-grammar-longbytes}@anchor{1739}@anchor{reference/lexical_analysis grammar-token-longbytes}@anchor{173a}longbytes      ::= "'@w{'}'" longbytesitem* "'@w{'}'" | '"""' longbytesitem* '"""'
@anchor{reference/lexical_analysis grammar-token-python-grammar-shortbytesitem}@anchor{173b}@anchor{reference/lexical_analysis grammar-token-shortbytesitem}@anchor{173c}shortbytesitem ::= shortbyteschar | bytesescapeseq
@anchor{reference/lexical_analysis grammar-token-python-grammar-longbytesitem}@anchor{173d}@anchor{reference/lexical_analysis grammar-token-longbytesitem}@anchor{173e}longbytesitem  ::= longbyteschar | bytesescapeseq
@anchor{reference/lexical_analysis grammar-token-python-grammar-shortbyteschar}@anchor{173f}@anchor{reference/lexical_analysis grammar-token-shortbyteschar}@anchor{1740}shortbyteschar ::= <any ASCII character except "\" or newline or the quote>
@anchor{reference/lexical_analysis grammar-token-python-grammar-longbyteschar}@anchor{1741}@anchor{reference/lexical_analysis grammar-token-longbyteschar}@anchor{1742}longbyteschar  ::= <any ASCII character except "\">
@anchor{reference/lexical_analysis grammar-token-python-grammar-bytesescapeseq}@anchor{1743}@anchor{reference/lexical_analysis grammar-token-bytesescapeseq}@anchor{1744}bytesescapeseq ::= "\" <any ASCII character>
@end example

One syntactic restriction not indicated by these productions is that whitespace
is not allowed between the @ref{1723,,stringprefix} or
@ref{1735,,bytesprefix} and the rest of the literal. The source
character set is defined by the encoding declaration; it is UTF-8 if no encoding
declaration is given in the source file; see section @ref{16fe,,Encoding declarations}.

@geindex triple-quoted string
@geindex Unicode Consortium
@geindex raw string
@geindex """; string literal
@geindex '@w{'}'; string literal

In plain English: Both types of literals can be enclosed in matching single quotes
(@code{'}) or double quotes (@code{"}).  They can also be enclosed in matching groups
of three single or double quotes (these are generally referred to as
@emph{triple-quoted strings}).  The backslash (@code{\}) character is used to escape
characters that otherwise have a special meaning, such as newline, backslash
itself, or the quote character.

@geindex b'; bytes literal
@geindex b"; bytes literal

Bytes literals are always prefixed with @code{'b'} or @code{'B'}; they produce an
instance of the @ref{2f0,,bytes} type instead of the @ref{2ef,,str} type.  They
may only contain ASCII characters; bytes with a numeric value of 128 or greater
must be expressed with escapes.

@geindex r'; raw string literal
@geindex r"; raw string literal

Both string and bytes literals may optionally be prefixed with a letter @code{'r'}
or @code{'R'}; such strings are called @emph{raw strings} and treat backslashes as
literal characters.  As a result, in string literals, @code{'\U'} and @code{'\u'}
escapes in raw strings are not treated specially. Given that Python 2.x’s raw
unicode literals behave differently than Python 3.x’s the @code{'ur'} syntax
is not supported.


New in version 3.3: The @code{'rb'} prefix of raw bytes literals has been added as a synonym
of @code{'br'}.



New in version 3.3: Support for the unicode legacy literal (@code{u'value'}) was reintroduced
to simplify the maintenance of dual Python 2.x and 3.x codebases.
See 
@geindex Python Enhancement Proposals; PEP 414
PEP 414@footnote{https://www.python.org/dev/peps/pep-0414} for more information.


@geindex f'; formatted string literal
@geindex f"; formatted string literal

A string literal with @code{'f'} or @code{'F'} in its prefix is a
@emph{formatted string literal}; see @ref{4b0,,Formatted string literals}.  The @code{'f'} may be
combined with @code{'r'}, but not with @code{'b'} or @code{'u'}, therefore raw
formatted strings are possible, but formatted bytes literals are not.

In triple-quoted literals, unescaped newlines and quotes are allowed (and are
retained), except that three unescaped quotes in a row terminate the literal.  (A
“quote” is the character used to open the literal, i.e. either @code{'} or @code{"}.)

@geindex physical line
@geindex escape sequence
@geindex Standard C
@geindex C
@geindex \ (backslash); escape sequence
@geindex \\; escape sequence
@geindex \a; escape sequence
@geindex \b; escape sequence
@geindex \f; escape sequence
@geindex \n; escape sequence
@geindex \r; escape sequence
@geindex \t; escape sequence
@geindex \v; escape sequence
@geindex \x; escape sequence
@geindex \N; escape sequence
@geindex \u; escape sequence
@geindex \U; escape sequence

Unless an @code{'r'} or @code{'R'} prefix is present, escape sequences in string and
bytes literals are interpreted according to rules similar to those used by
Standard C.  The recognized escape sequences are:


@multitable {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} 
@headitem

Escape Sequence

@tab

Meaning

@tab

Notes

@item

@code{\newline}

@tab

Backslash and newline ignored

@tab

@item

@code{\\}

@tab

Backslash (@code{\})

@tab

@item

@code{\'}

@tab

Single quote (@code{'})

@tab

@item

@code{\"}

@tab

Double quote (@code{"})

@tab

@item

@code{\a}

@tab

ASCII Bell (BEL)

@tab

@item

@code{\b}

@tab

ASCII Backspace (BS)

@tab

@item

@code{\f}

@tab

ASCII Formfeed (FF)

@tab

@item

@code{\n}

@tab

ASCII Linefeed (LF)

@tab

@item

@code{\r}

@tab

ASCII Carriage Return (CR)

@tab

@item

@code{\t}

@tab

ASCII Horizontal Tab (TAB)

@tab

@item

@code{\v}

@tab

ASCII Vertical Tab (VT)

@tab

@item

@code{\ooo}

@tab

Character with octal value
@emph{ooo}

@tab

(1,3)

@item

@code{\xhh}

@tab

Character with hex value @emph{hh}

@tab

(2,3)

@end multitable


Escape sequences only recognized in string literals are:


@multitable {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} 
@headitem

Escape Sequence

@tab

Meaning

@tab

Notes

@item

@code{\N@{name@}}

@tab

Character named @emph{name} in the
Unicode database

@tab

(4)

@item

@code{\uxxxx}

@tab

Character with 16-bit hex value
@emph{xxxx}

@tab

(5)

@item

@code{\Uxxxxxxxx}

@tab

Character with 32-bit hex value
@emph{xxxxxxxx}

@tab

(6)

@end multitable


Notes:


@enumerate 

@item 
As in Standard C, up to three octal digits are accepted.

@item 
Unlike in Standard C, exactly two hex digits are required.

@item 
In a bytes literal, hexadecimal and octal escapes denote the byte with the
given value. In a string literal, these escapes denote a Unicode character
with the given value.

@item 

Changed in version 3.3: Support for name aliases @footnote{@w{(1)} 
@indicateurl{https://www.unicode.org/Public/11.0.0/ucd/NameAliases.txt}
} has been added.


@item 
Exactly four hex digits are required.

@item 
Any Unicode character can be encoded this way.  Exactly eight hex digits
are required.
@end enumerate

@geindex unrecognized escape sequence

Unlike Standard C, all unrecognized escape sequences are left in the string
unchanged, i.e., @emph{the backslash is left in the result}.  (This behavior is
useful when debugging: if an escape sequence is mistyped, the resulting output
is more easily recognized as broken.)  It is also important to note that the
escape sequences only recognized in string literals fall into the category of
unrecognized escapes for bytes literals.

@quotation


Changed in version 3.6: Unrecognized escape sequences produce a @ref{1b2,,DeprecationWarning}.  In
a future Python version they will be a @ref{4c7,,SyntaxWarning} and
eventually a @ref{21a,,SyntaxError}.

@end quotation

Even in a raw literal, quotes can be escaped with a backslash, but the
backslash remains in the result; for example, @code{r"\""} is a valid string
literal consisting of two characters: a backslash and a double quote; @code{r"\"}
is not a valid string literal (even a raw string cannot end in an odd number of
backslashes).  Specifically, @emph{a raw literal cannot end in a single backslash}
(since the backslash would escape the following quote character).  Note also
that a single backslash followed by a newline is interpreted as those two
characters as part of the literal, @emph{not} as a line continuation.

@node String literal concatenation,Formatted string literals,String and Bytes literals,Literals
@anchor{reference/lexical_analysis string-concatenation}@anchor{1745}@anchor{reference/lexical_analysis string-literal-concatenation}@anchor{1746}
@subsubsection String literal concatenation


Multiple adjacent string or bytes literals (delimited by whitespace), possibly
using different quoting conventions, are allowed, and their meaning is the same
as their concatenation.  Thus, @code{"hello" 'world'} is equivalent to
@code{"helloworld"}.  This feature can be used to reduce the number of backslashes
needed, to split long strings conveniently across long lines, or even to add
comments to parts of strings, for example:

@example
re.compile("[A-Za-z_]"       # letter or underscore
           "[A-Za-z0-9_]*"   # letter, digit or underscore
          )
@end example

Note that this feature is defined at the syntactical level, but implemented at
compile time.  The ‘+’ operator must be used to concatenate string expressions
at run time.  Also note that literal concatenation can use different quoting
styles for each component (even mixing raw strings and triple quoted strings),
and formatted string literals may be concatenated with plain string literals.

@geindex formatted string literal
@geindex interpolated string literal
@geindex string; formatted literal
@geindex string; interpolated literal
@geindex f-string
@geindex fstring
@geindex @{@} (curly brackets); in formatted string literal
@geindex ! (exclamation); in formatted string literal
@geindex ; (colon); in formatted string literal
@geindex = (equals); for help in debugging using string literals

@node Formatted string literals,Numeric literals,String literal concatenation,Literals
@anchor{reference/lexical_analysis f-strings}@anchor{4b0}@anchor{reference/lexical_analysis formatted-string-literals}@anchor{1747}
@subsubsection Formatted string literals



New in version 3.6.


A @emph{formatted string literal} or @emph{f-string} is a string literal
that is prefixed with @code{'f'} or @code{'F'}.  These strings may contain
replacement fields, which are expressions delimited by curly braces @code{@{@}}.
While other string literals always have a constant value, formatted strings
are really expressions evaluated at run time.

Escape sequences are decoded like in ordinary string literals (except when
a literal is also marked as a raw string).  After decoding, the grammar
for the contents of the string is:

@example
@anchor{reference/lexical_analysis grammar-token-python-grammar-f_string}@anchor{1748}@anchor{reference/lexical_analysis grammar-token-f-string}@anchor{1749}f_string          ::= (literal_char | "@{@{" | "@}@}" | replacement_field)*
@anchor{reference/lexical_analysis grammar-token-python-grammar-replacement_field}@anchor{174a}@anchor{reference/lexical_analysis grammar-token-replacement-field}@anchor{174b}replacement_field ::= "@{" f_expression ["="] ["!" conversion] [":" format_spec] "@}"
@anchor{reference/lexical_analysis grammar-token-python-grammar-f_expression}@anchor{174c}@anchor{reference/lexical_analysis grammar-token-f-expression}@anchor{174d}f_expression      ::= (conditional_expression | "*" or_expr)
                        ("," conditional_expression | "," "*" or_expr)* [","]
                      | yield_expression
@anchor{reference/lexical_analysis grammar-token-python-grammar-conversion}@anchor{174e}@anchor{reference/lexical_analysis grammar-token-conversion}@anchor{174f}conversion        ::= "s" | "r" | "a"
@anchor{reference/lexical_analysis grammar-token-python-grammar-format_spec}@anchor{1750}@anchor{reference/lexical_analysis grammar-token-format-spec}@anchor{1751}format_spec       ::= (literal_char | NULL | replacement_field)*
@anchor{reference/lexical_analysis grammar-token-python-grammar-literal_char}@anchor{1752}@anchor{reference/lexical_analysis grammar-token-literal-char}@anchor{1753}literal_char      ::= <any code point except "@{", "@}" or NULL>
@end example

The parts of the string outside curly braces are treated literally,
except that any doubled curly braces @code{'@{@{'} or @code{'@}@}'} are replaced
with the corresponding single curly brace.  A single opening curly
bracket @code{'@{'} marks a replacement field, which starts with a
Python expression. To display both the expression text and its value after
evaluation, (useful in debugging), an equal sign @code{'='} may be added after the
expression. A conversion field, introduced by an exclamation point @code{'!'} may
follow.  A format specifier may also be appended, introduced by a colon @code{':'}.
A replacement field ends with a closing curly bracket @code{'@}'}.

Expressions in formatted string literals are treated like regular
Python expressions surrounded by parentheses, with a few exceptions.
An empty expression is not allowed, and both @ref{e42,,lambda}  and
assignment expressions @code{:=} must be surrounded by explicit parentheses.
Replacement expressions can contain line breaks (e.g. in triple-quoted
strings), but they cannot contain comments.  Each expression is evaluated
in the context where the formatted string literal appears, in order from
left to right.


Changed in version 3.7: Prior to Python 3.7, an @ref{4d5,,await} expression and comprehensions
containing an @ref{5cd,,async for} clause were illegal in the expressions
in formatted string literals due to a problem with the implementation.


When the equal sign @code{'='} is provided, the output will have the expression
text, the @code{'='} and the evaluated value. Spaces after the opening brace
@code{'@{'}, within the expression and after the @code{'='} are all retained in the
output. By default, the @code{'='} causes the @ref{27d,,repr()} of the expression to be
provided, unless there is a format specified. When a format is specified it
defaults to the @ref{2ef,,str()} of the expression unless a conversion @code{'!r'} is
declared.


New in version 3.8: The equal sign @code{'='}.


If a conversion is specified, the result of evaluating the expression
is converted before formatting.  Conversion @code{'!s'} calls @ref{2ef,,str()} on
the result, @code{'!r'} calls @ref{27d,,repr()}, and @code{'!a'} calls @ref{f47,,ascii()}.

The result is then formatted using the @ref{785,,format()} protocol.  The
format specifier is passed to the @code{__format__()} method of the
expression or conversion result.  An empty string is passed when the
format specifier is omitted.  The formatted result is then included in
the final value of the whole string.

Top-level format specifiers may include nested replacement fields. These nested
fields may include their own conversion fields and @ref{788,,format specifiers}, but may not include more deeply-nested replacement fields. The
@ref{788,,format specifier mini-language} is the same as that used by
the @ref{784,,str.format()} method.

Formatted string literals may be concatenated, but replacement fields
cannot be split across literals.

Some examples of formatted string literals:

@example
>>> name = "Fred"
>>> f"He said his name is @{name!r@}."
"He said his name is 'Fred'."
>>> f"He said his name is @{repr(name)@}."  # repr() is equivalent to !r
"He said his name is 'Fred'."
>>> width = 10
>>> precision = 4
>>> value = decimal.Decimal("12.34567")
>>> f"result: @{value:@{width@}.@{precision@}@}"  # nested fields
'result:      12.35'
>>> today = datetime(year=2017, month=1, day=27)
>>> f"@{today:%B %d, %Y@}"  # using date format specifier
'January 27, 2017'
>>> f"@{today=:%B %d, %Y@}" # using date format specifier and debugging
'today=January 27, 2017'
>>> number = 1024
>>> f"@{number:#0x@}"  # using integer format specifier
'0x400'
>>> foo = "bar"
>>> f"@{ foo = @}" # preserves whitespace
" foo = 'bar'"
>>> line = "The mill's closed"
>>> f"@{line = @}"
'line = "The mill\'s closed"'
>>> f"@{line = :20@}"
"line = The mill's closed   "
>>> f"@{line = !r:20@}"
'line = "The mill\'s closed" '
@end example

A consequence of sharing the same syntax as regular string literals is
that characters in the replacement fields must not conflict with the
quoting used in the outer formatted string literal:

@example
f"abc @{a["x"]@} def"    # error: outer string literal ended prematurely
f"abc @{a['x']@} def"    # workaround: use different quoting
@end example

Backslashes are not allowed in format expressions and will raise
an error:

@example
f"newline: @{ord('\n')@}"  # raises SyntaxError
@end example

To include a value in which a backslash escape is required, create
a temporary variable.

@example
>>> newline = ord('\n')
>>> f"newline: @{newline@}"
'newline: 10'
@end example

Formatted string literals cannot be used as docstrings, even if they do not
include expressions.

@example
>>> def foo():
...     f"Not a docstring"
...
>>> foo.__doc__ is None
True
@end example

See also 
@geindex Python Enhancement Proposals; PEP 498
PEP 498@footnote{https://www.python.org/dev/peps/pep-0498} for the proposal that added formatted string literals,
and @ref{784,,str.format()}, which uses a related format string mechanism.

@node Numeric literals,Integer literals,Formatted string literals,Literals
@anchor{reference/lexical_analysis numbers}@anchor{1754}@anchor{reference/lexical_analysis numeric-literals}@anchor{1755}
@subsubsection Numeric literals


@geindex number
@geindex numeric literal
@geindex integer literal
@geindex floating point literal
@geindex hexadecimal literal
@geindex octal literal
@geindex binary literal
@geindex decimal literal
@geindex imaginary literal
@geindex complex literal

There are three types of numeric literals: integers, floating point numbers, and
imaginary numbers.  There are no complex literals (complex numbers can be formed
by adding a real number and an imaginary number).

Note that numeric literals do not include a sign; a phrase like @code{-1} is
actually an expression composed of the unary operator ‘@code{-}’ and the literal
@code{1}.

@geindex 0b; integer literal
@geindex 0o; integer literal
@geindex 0x; integer literal
@geindex _ (underscore); in numeric literal

@node Integer literals,Floating point literals,Numeric literals,Literals
@anchor{reference/lexical_analysis integer-literals}@anchor{1756}@anchor{reference/lexical_analysis integers}@anchor{1757}
@subsubsection Integer literals


Integer literals are described by the following lexical definitions:

@example
@anchor{reference/lexical_analysis grammar-token-python-grammar-integer}@anchor{1758}@anchor{reference/lexical_analysis grammar-token-integer}@anchor{1759}integer      ::= decinteger | bininteger | octinteger | hexinteger
@anchor{reference/lexical_analysis grammar-token-python-grammar-decinteger}@anchor{175a}@anchor{reference/lexical_analysis grammar-token-decinteger}@anchor{175b}decinteger   ::= nonzerodigit (["_"] digit)* | "0"+ (["_"] "0")*
@anchor{reference/lexical_analysis grammar-token-python-grammar-bininteger}@anchor{175c}@anchor{reference/lexical_analysis grammar-token-bininteger}@anchor{175d}bininteger   ::= "0" ("b" | "B") (["_"] bindigit)+
@anchor{reference/lexical_analysis grammar-token-python-grammar-octinteger}@anchor{175e}@anchor{reference/lexical_analysis grammar-token-octinteger}@anchor{175f}octinteger   ::= "0" ("o" | "O") (["_"] octdigit)+
@anchor{reference/lexical_analysis grammar-token-python-grammar-hexinteger}@anchor{1760}@anchor{reference/lexical_analysis grammar-token-hexinteger}@anchor{1761}hexinteger   ::= "0" ("x" | "X") (["_"] hexdigit)+
@anchor{reference/lexical_analysis grammar-token-python-grammar-nonzerodigit}@anchor{1762}@anchor{reference/lexical_analysis grammar-token-nonzerodigit}@anchor{1763}nonzerodigit ::= "1"..."9"
@anchor{reference/lexical_analysis grammar-token-python-grammar-digit}@anchor{1764}@anchor{reference/lexical_analysis grammar-token-digit}@anchor{1765}digit        ::= "0"..."9"
@anchor{reference/lexical_analysis grammar-token-python-grammar-bindigit}@anchor{1766}@anchor{reference/lexical_analysis grammar-token-bindigit}@anchor{1767}bindigit     ::= "0" | "1"
@anchor{reference/lexical_analysis grammar-token-python-grammar-octdigit}@anchor{1768}@anchor{reference/lexical_analysis grammar-token-octdigit}@anchor{1769}octdigit     ::= "0"..."7"
@anchor{reference/lexical_analysis grammar-token-python-grammar-hexdigit}@anchor{176a}@anchor{reference/lexical_analysis grammar-token-hexdigit}@anchor{176b}hexdigit     ::= digit | "a"..."f" | "A"..."F"
@end example

There is no limit for the length of integer literals apart from what can be
stored in available memory.

Underscores are ignored for determining the numeric value of the literal.  They
can be used to group digits for enhanced readability.  One underscore can occur
between digits, and after base specifiers like @code{0x}.

Note that leading zeros in a non-zero decimal number are not allowed. This is
for disambiguation with C-style octal literals, which Python used before version
3.0.

Some examples of integer literals:

@example
7     2147483647                        0o177    0b100110111
3     79228162514264337593543950336     0o377    0xdeadbeef
      100_000_000_000                   0b_1110_0101
@end example


Changed in version 3.6: Underscores are now allowed for grouping purposes in literals.


@geindex . (dot); in numeric literal
@geindex e; in numeric literal
@geindex _ (underscore); in numeric literal

@node Floating point literals,Imaginary literals,Integer literals,Literals
@anchor{reference/lexical_analysis floating}@anchor{176c}@anchor{reference/lexical_analysis floating-point-literals}@anchor{176d}
@subsubsection Floating point literals


Floating point literals are described by the following lexical definitions:

@example
@anchor{reference/lexical_analysis grammar-token-python-grammar-floatnumber}@anchor{176e}@anchor{reference/lexical_analysis grammar-token-floatnumber}@anchor{176f}floatnumber   ::= pointfloat | exponentfloat
@anchor{reference/lexical_analysis grammar-token-python-grammar-pointfloat}@anchor{1770}@anchor{reference/lexical_analysis grammar-token-pointfloat}@anchor{1771}pointfloat    ::= [digitpart] fraction | digitpart "."
@anchor{reference/lexical_analysis grammar-token-python-grammar-exponentfloat}@anchor{1772}@anchor{reference/lexical_analysis grammar-token-exponentfloat}@anchor{1773}exponentfloat ::= (digitpart | pointfloat) exponent
@anchor{reference/lexical_analysis grammar-token-python-grammar-digitpart}@anchor{1774}@anchor{reference/lexical_analysis grammar-token-digitpart}@anchor{1775}digitpart     ::= digit (["_"] digit)*
@anchor{reference/lexical_analysis grammar-token-python-grammar-fraction}@anchor{1776}@anchor{reference/lexical_analysis grammar-token-fraction}@anchor{1777}fraction      ::= "." digitpart
@anchor{reference/lexical_analysis grammar-token-python-grammar-exponent}@anchor{1778}@anchor{reference/lexical_analysis grammar-token-exponent}@anchor{1779}exponent      ::= ("e" | "E") ["+" | "-"] digitpart
@end example

Note that the integer and exponent parts are always interpreted using radix 10.
For example, @code{077e010} is legal, and denotes the same number as @code{77e10}. The
allowed range of floating point literals is implementation-dependent.  As in
integer literals, underscores are supported for digit grouping.

Some examples of floating point literals:

@example
3.14    10.    .001    1e100    3.14e-10    0e0    3.14_15_93
@end example


Changed in version 3.6: Underscores are now allowed for grouping purposes in literals.


@geindex j; in numeric literal

@node Imaginary literals,,Floating point literals,Literals
@anchor{reference/lexical_analysis imaginary}@anchor{177a}@anchor{reference/lexical_analysis imaginary-literals}@anchor{177b}
@subsubsection Imaginary literals


Imaginary literals are described by the following lexical definitions:

@example
@anchor{reference/lexical_analysis grammar-token-python-grammar-imagnumber}@anchor{177c}@anchor{reference/lexical_analysis grammar-token-imagnumber}@anchor{177d}imagnumber ::= (floatnumber | digitpart) ("j" | "J")
@end example

An imaginary literal yields a complex number with a real part of 0.0.  Complex
numbers are represented as a pair of floating point numbers and have the same
restrictions on their range.  To create a complex number with a nonzero real
part, add a floating point number to it, e.g., @code{(3+4j)}.  Some examples of
imaginary literals:

@example
3.14j   10.j    10j     .001j   1e100j   3.14e-10j   3.14_15_93j
@end example

@node Operators,Delimiters,Literals,Lexical analysis
@anchor{reference/lexical_analysis id12}@anchor{177e}@anchor{reference/lexical_analysis operators}@anchor{177f}
@subsection Operators


@geindex operators

The following tokens are operators:

@example
+       -       *       **      /       //      %      @@
<<      >>      &       |       ^       ~       :=
<       >       <=      >=      ==      !=
@end example

@node Delimiters,,Operators,Lexical analysis
@anchor{reference/lexical_analysis delimiters}@anchor{1780}@anchor{reference/lexical_analysis id13}@anchor{1781}
@subsection Delimiters


@geindex delimiters

The following tokens serve as delimiters in the grammar:

@example
(       )       [       ]       @{       @}
,       :       .       ;       @@       =       ->
+=      -=      *=      /=      //=     %=      @@=
&=      |=      ^=      >>=     <<=     **=
@end example

The period can also occur in floating-point and imaginary literals.  A sequence
of three periods has a special meaning as an ellipsis literal. The second half
of the list, the augmented assignment operators, serve lexically as delimiters,
but also perform an operation.

The following printing ASCII characters have special meaning as part of other
tokens or are otherwise significant to the lexical analyzer:

@example
'       "       #       \
@end example

The following printing ASCII characters are not used in Python.  Their
occurrence outside string literals and comments is an unconditional error:

@example
$       ?       `
@end example

@node Data model,Execution model,Lexical analysis,The Python Language Reference
@anchor{reference/datamodel doc}@anchor{1782}@anchor{reference/datamodel data-model}@anchor{1783}@anchor{reference/datamodel datamodel}@anchor{11ff}
@section Data model


@menu
* Objects@comma{} values and types: Objects values and types. 
* The standard type hierarchy:: 
* Special method names:: 
* Coroutines:: 

@end menu

@node Objects values and types,The standard type hierarchy,,Data model
@anchor{reference/datamodel objects}@anchor{1784}@anchor{reference/datamodel objects-values-and-types}@anchor{1785}
@subsection Objects, values and types


@geindex object
@geindex data

@emph{Objects} are Python’s abstraction for data.  All data in a Python program
is represented by objects or by relations between objects. (In a sense, and in
conformance to Von Neumann’s model of a “stored program computer”, code is also
represented by objects.)

@geindex built-in function; id
@geindex built-in function; type
@geindex identity of an object
@geindex value of an object
@geindex type of an object
@geindex mutable object
@geindex immutable object

@c XXX it *is* now possible in some cases to change an object's
@c type, under certain controlled conditions

Every object has an identity, a type and a value.  An object’s @emph{identity} never
changes once it has been created; you may think of it as the object’s address in
memory.  The ‘@ref{2fd,,is}’ operator compares the identity of two objects; the
@ref{f80,,id()} function returns an integer representing its identity.

@strong{CPython implementation detail:} For CPython, @code{id(x)} is the memory address where @code{x} is stored.

An object’s type determines the operations that the object supports (e.g., “does
it have a length?”) and also defines the possible values for objects of that
type.  The @ref{89e,,type()} function returns an object’s type (which is an object
itself).  Like its identity, an object’s @emph{type} is also unchangeable.
@footnote{@w{(1)} 
It @emph{is} possible in some cases to change an object’s type, under certain
controlled conditions. It generally isn’t a good idea though, since it can
lead to some very strange behaviour if it is handled incorrectly.
}

The @emph{value} of some objects can change.  Objects whose value can
change are said to be @emph{mutable}; objects whose value is unchangeable once they
are created are called @emph{immutable}. (The value of an immutable container object
that contains a reference to a mutable object can change when the latter’s value
is changed; however the container is still considered immutable, because the
collection of objects it contains cannot be changed.  So, immutability is not
strictly the same as having an unchangeable value, it is more subtle.) An
object’s mutability is determined by its type; for instance, numbers, strings
and tuples are immutable, while dictionaries and lists are mutable.

@geindex garbage collection
@geindex reference counting
@geindex unreachable object

Objects are never explicitly destroyed; however, when they become unreachable
they may be garbage-collected.  An implementation is allowed to postpone garbage
collection or omit it altogether — it is a matter of implementation quality
how garbage collection is implemented, as long as no objects are collected that
are still reachable.

@strong{CPython implementation detail:} CPython currently uses a reference-counting scheme with (optional) delayed
detection of cyclically linked garbage, which collects most objects as soon
as they become unreachable, but is not guaranteed to collect garbage
containing circular references.  See the documentation of the @ref{80,,gc}
module for information on controlling the collection of cyclic garbage.
Other implementations act differently and CPython may change.
Do not depend on immediate finalization of objects when they become
unreachable (so you should always close files explicitly).

Note that the use of the implementation’s tracing or debugging facilities may
keep objects alive that would normally be collectable. Also note that catching
an exception with a ‘@ref{f6d,,try}…@ref{16e,,except}’ statement may keep
objects alive.

Some objects contain references to “external” resources such as open files or
windows.  It is understood that these resources are freed when the object is
garbage-collected, but since garbage collection is not guaranteed to happen,
such objects also provide an explicit way to release the external resource,
usually a @code{close()} method. Programs are strongly recommended to explicitly
close such objects.  The ‘@ref{f6d,,try}…@ref{4c1,,finally}’ statement
and the ‘@ref{168,,with}’ statement provide convenient ways to do this.

@geindex container

Some objects contain references to other objects; these are called @emph{containers}.
Examples of containers are tuples, lists and dictionaries.  The references are
part of a container’s value.  In most cases, when we talk about the value of a
container, we imply the values, not the identities of the contained objects;
however, when we talk about the mutability of a container, only the identities
of the immediately contained objects are implied.  So, if an immutable container
(like a tuple) contains a reference to a mutable object, its value changes if
that mutable object is changed.

Types affect almost all aspects of object behavior.  Even the importance of
object identity is affected in some sense: for immutable types, operations that
compute new values may actually return a reference to any existing object with
the same type and value, while for mutable objects this is not allowed.  E.g.,
after @code{a = 1; b = 1}, @code{a} and @code{b} may or may not refer to the same object
with the value one, depending on the implementation, but after @code{c = []; d =
[]}, @code{c} and @code{d} are guaranteed to refer to two different, unique, newly
created empty lists. (Note that @code{c = d = []} assigns the same object to both
@code{c} and @code{d}.)

@node The standard type hierarchy,Special method names,Objects values and types,Data model
@anchor{reference/datamodel the-standard-type-hierarchy}@anchor{1786}@anchor{reference/datamodel types}@anchor{1787}
@subsection The standard type hierarchy


@geindex type
@geindex data; type
@geindex type; hierarchy
@geindex extension; module
@geindex C; language

Below is a list of the types that are built into Python.  Extension modules
(written in C, Java, or other languages, depending on the implementation) can
define additional types.  Future versions of Python may add types to the type
hierarchy (e.g., rational numbers, efficiently stored arrays of integers, etc.),
although such additions will often be provided via the standard library instead.

@geindex attribute
@geindex special; attribute
@geindex generic; special; attribute

Some of the type descriptions below contain a paragraph listing ‘special
attributes.’  These are attributes that provide access to the implementation and
are not intended for general use.  Their definition may change in the future.


@table @asis

@item None

@geindex object; None

This type has a single value.  There is a single object with this value. This
object is accessed through the built-in name @code{None}. It is used to signify the
absence of a value in many situations, e.g., it is returned from functions that
don’t explicitly return anything. Its truth value is false.

@item NotImplemented

@geindex object; NotImplemented

This type has a single value.  There is a single object with this value. This
object is accessed through the built-in name @code{NotImplemented}. Numeric methods
and rich comparison methods should return this value if they do not implement the
operation for the operands provided.  (The interpreter will then try the
reflected operation, or some other fallback, depending on the operator.)  It
should not be evaluated in a boolean context.

See
@ref{1788,,Implementing the arithmetic operations}
for more details.


Changed in version 3.9: Evaluating @code{NotImplemented} in a boolean context is deprecated. While
it currently evaluates as true, it will emit a @ref{1b2,,DeprecationWarning}.
It will raise a @ref{162,,TypeError} in a future version of Python.


@item Ellipsis

@geindex object; Ellipsis
@geindex ...; ellipsis literal

This type has a single value.  There is a single object with this value. This
object is accessed through the literal @code{...} or the built-in name
@code{Ellipsis}.  Its truth value is true.

@item @ref{1789,,numbers.Number}

@geindex object; numeric

These are created by numeric literals and returned as results by arithmetic
operators and arithmetic built-in functions.  Numeric objects are immutable;
once created their value never changes.  Python numbers are of course strongly
related to mathematical numbers, but subject to the limitations of numerical
representation in computers.

The string representations of the numeric classes, computed by
@ref{617,,__repr__()} and @ref{178a,,__str__()}, have the following
properties:


@itemize *

@item 
They are valid numeric literals which, when passed to their
class constructor, produce an object having the value of the
original numeric.

@item 
The representation is in base 10, when possible.

@item 
Leading zeros, possibly excepting a single zero before a
decimal point, are not shown.

@item 
Trailing zeros, possibly excepting a single zero after a
decimal point, are not shown.

@item 
A sign is shown only when the number is negative.
@end itemize

Python distinguishes between integers, floating point numbers, and complex
numbers:


@table @asis

@item @ref{178b,,numbers.Integral}

@geindex object; integer

These represent elements from the mathematical set of integers (positive and
negative).

There are two types of integers:


@table @asis

@item Integers (@ref{1b1,,int})

These represent numbers in an unlimited range, subject to available (virtual)
memory only.  For the purpose of shift and mask operations, a binary
representation is assumed, and negative numbers are represented in a variant of
2’s complement which gives the illusion of an infinite string of sign bits
extending to the left.

@item Booleans (@ref{2f7,,bool})

@geindex object; Boolean
@geindex False
@geindex True

These represent the truth values False and True.  The two objects representing
the values @code{False} and @code{True} are the only Boolean objects. The Boolean type is a
subtype of the integer type, and Boolean values behave like the values 0 and 1,
respectively, in almost all contexts, the exception being that when converted to
a string, the strings @code{"False"} or @code{"True"} are returned, respectively.
@end table

@geindex integer; representation

The rules for integer representation are intended to give the most meaningful
interpretation of shift and mask operations involving negative integers.

@item @ref{178c,,numbers.Real} (@ref{256,,float})

@geindex object; floating point
@geindex floating point; number
@geindex C; language
@geindex Java; language

These represent machine-level double precision floating point numbers. You are
at the mercy of the underlying machine architecture (and C or Java
implementation) for the accepted range and handling of overflow. Python does not
support single-precision floating point numbers; the savings in processor and
memory usage that are usually the reason for using these are dwarfed by the
overhead of using objects in Python, so there is no reason to complicate the
language with two kinds of floating point numbers.

@item @ref{178d,,numbers.Complex} (@ref{33b,,complex})

@geindex object; complex
@geindex complex; number

These represent complex numbers as a pair of machine-level double precision
floating point numbers.  The same caveats apply as for floating point numbers.
The real and imaginary parts of a complex number @code{z} can be retrieved through
the read-only attributes @code{z.real} and @code{z.imag}.
@end table

@item Sequences

@geindex built-in function; len
@geindex object; sequence
@geindex index operation
@geindex item selection
@geindex subscription

These represent finite ordered sets indexed by non-negative numbers. The
built-in function @ref{4a6,,len()} returns the number of items of a sequence. When
the length of a sequence is @emph{n}, the index set contains the numbers 0, 1,
…, @emph{n}-1.  Item @emph{i} of sequence @emph{a} is selected by @code{a[i]}.

@geindex slicing

Sequences also support slicing: @code{a[i:j]} selects all items with index @emph{k} such
that @emph{i} @code{<=} @emph{k} @code{<} @emph{j}.  When used as an expression, a slice is a
sequence of the same type.  This implies that the index set is renumbered so
that it starts at 0.

Some sequences also support “extended slicing” with a third “step” parameter:
@code{a[i:j:k]} selects all items of @emph{a} with index @emph{x} where @code{x = i + n*k}, @emph{n}
@code{>=} @code{0} and @emph{i} @code{<=} @emph{x} @code{<} @emph{j}.

Sequences are distinguished according to their mutability:


@table @asis

@item Immutable sequences

@geindex object; immutable sequence
@geindex object; immutable

An object of an immutable sequence type cannot change once it is created.  (If
the object contains references to other objects, these other objects may be
mutable and may be changed; however, the collection of objects directly
referenced by an immutable object cannot change.)

The following types are immutable sequences:

@geindex string; immutable sequences


@table @asis

@item Strings

@geindex built-in function; chr
@geindex built-in function; ord
@geindex character
@geindex integer
@geindex Unicode

A string is a sequence of values that represent Unicode code points.
All the code points in the range @code{U+0000 - U+10FFFF} can be
represented in a string.  Python doesn’t have a char type;
instead, every code point in the string is represented as a string
object with length @code{1}.  The built-in function @ref{178e,,ord()}
converts a code point from its string form to an integer in the
range @code{0 - 10FFFF}; @ref{178f,,chr()} converts an integer in the range
@code{0 - 10FFFF} to the corresponding length @code{1} string object.
@ref{3ba,,str.encode()} can be used to convert a @ref{2ef,,str} to
@ref{2f0,,bytes} using the given text encoding, and
@ref{3bb,,bytes.decode()} can be used to achieve the opposite.

@item Tuples

@geindex object; tuple
@geindex singleton; tuple
@geindex empty; tuple

The items of a tuple are arbitrary Python objects. Tuples of two or
more items are formed by comma-separated lists of expressions.  A tuple
of one item (a ‘singleton’) can be formed by affixing a comma to an
expression (an expression by itself does not create a tuple, since
parentheses must be usable for grouping of expressions).  An empty
tuple can be formed by an empty pair of parentheses.

@item Bytes

@geindex bytes
@geindex byte

A bytes object is an immutable array.  The items are 8-bit bytes,
represented by integers in the range 0 <= x < 256.  Bytes literals
(like @code{b'abc'}) and the built-in @ref{2f0,,bytes()} constructor
can be used to create bytes objects.  Also, bytes objects can be
decoded to strings via the @ref{3bb,,decode()} method.
@end table

@item Mutable sequences

@geindex object; mutable sequence
@geindex object; mutable
@geindex assignment; statement
@geindex subscription
@geindex slicing

Mutable sequences can be changed after they are created.  The subscription and
slicing notations can be used as the target of assignment and @ref{152e,,del}
(delete) statements.

There are currently two intrinsic mutable sequence types:


@table @asis

@item Lists

@geindex object; list

The items of a list are arbitrary Python objects.  Lists are formed by
placing a comma-separated list of expressions in square brackets. (Note
that there are no special cases needed to form lists of length 0 or 1.)

@item Byte Arrays

@geindex bytearray

A bytearray object is a mutable array. They are created by the built-in
@ref{2f1,,bytearray()} constructor.  Aside from being mutable
(and hence unhashable), byte arrays otherwise provide the same interface
and functionality as immutable @ref{2f0,,bytes} objects.
@end table

@geindex module; array

The extension module @ref{6,,array} provides an additional example of a
mutable sequence type, as does the @ref{1c,,collections} module.
@end table

@item Set types

@geindex built-in function; len
@geindex object; set type

These represent unordered, finite sets of unique, immutable objects. As such,
they cannot be indexed by any subscript. However, they can be iterated over, and
the built-in function @ref{4a6,,len()} returns the number of items in a set. Common
uses for sets are fast membership testing, removing duplicates from a sequence,
and computing mathematical operations such as intersection, union, difference,
and symmetric difference.

For set elements, the same immutability rules apply as for dictionary keys. Note
that numeric types obey the normal rules for numeric comparison: if two numbers
compare equal (e.g., @code{1} and @code{1.0}), only one of them can be contained in a
set.

There are currently two intrinsic set types:


@table @asis

@item Sets

@geindex object; set

These represent a mutable set. They are created by the built-in @ref{370,,set()}
constructor and can be modified afterwards by several methods, such as
@code{add()}.

@item Frozen sets

@geindex object; frozenset

These represent an immutable set.  They are created by the built-in
@ref{439,,frozenset()} constructor.  As a frozenset is immutable and
@ref{2dc,,hashable}, it can be used again as an element of another set, or as
a dictionary key.
@end table

@item Mappings

@geindex built-in function; len
@geindex subscription
@geindex object; mapping

These represent finite sets of objects indexed by arbitrary index sets. The
subscript notation @code{a[k]} selects the item indexed by @code{k} from the mapping
@code{a}; this can be used in expressions and as the target of assignments or
@ref{152e,,del} statements. The built-in function @ref{4a6,,len()} returns the number
of items in a mapping.

There is currently a single intrinsic mapping type:


@table @asis

@item Dictionaries

@geindex object; dictionary

These represent finite sets of objects indexed by nearly arbitrary values.  The
only types of values not acceptable as keys are values containing lists or
dictionaries or other mutable types that are compared by value rather than by
object identity, the reason being that the efficient implementation of
dictionaries requires a key’s hash value to remain constant. Numeric types used
for keys obey the normal rules for numeric comparison: if two numbers compare
equal (e.g., @code{1} and @code{1.0}) then they can be used interchangeably to index
the same dictionary entry.

Dictionaries preserve insertion order, meaning that keys will be produced
in the same order they were added sequentially over the dictionary.
Replacing an existing key does not change the order, however removing a key
and re-inserting it will add it to the end instead of keeping its old place.

Dictionaries are mutable; they can be created by the @code{@{...@}} notation (see
section @ref{8d8,,Dictionary displays}).

@geindex module; dbm.ndbm
@geindex module; dbm.gnu

The extension modules @ref{33,,dbm.ndbm} and @ref{32,,dbm.gnu} provide
additional examples of mapping types, as does the @ref{1c,,collections}
module.


Changed in version 3.7: Dictionaries did not preserve insertion order in versions of Python before 3.6.
In CPython 3.6, insertion order was preserved, but it was considered
an implementation detail at that time rather than a language guarantee.

@end table

@item Callable types

@geindex object; callable
@geindex function; call
@geindex invocation
@geindex function; argument

These are the types to which the function call operation (see section
@ref{8d6,,Calls}) can be applied:


@table @asis

@item User-defined functions

@geindex user-defined; function
@geindex object; function
@geindex object; user-defined function

A user-defined function object is created by a function definition (see
section @ref{151c,,Function definitions}).  It should be called with an argument list
containing the same number of items as the function’s formal parameter
list.

Special attributes:

@geindex __doc__ (function attribute)
@geindex __name__ (function attribute)
@geindex __module__ (function attribute)
@geindex __dict__ (function attribute)
@geindex __defaults__ (function attribute)
@geindex __closure__ (function attribute)
@geindex __code__ (function attribute)
@geindex __globals__ (function attribute)
@geindex __annotations__ (function attribute)
@geindex __kwdefaults__ (function attribute)
@geindex global; namespace


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxx} 
@headitem

Attribute

@tab

Meaning

@tab

@item

@code{__doc__}

@tab

The function’s documentation
string, or @code{None} if
unavailable; not inherited by
subclasses.

@tab

Writable

@item

@ref{e70,,__name__}

@tab

The function’s name.

@tab

Writable

@item

@ref{1790,,__qualname__}

@tab

The function’s
@ref{11bf,,qualified name}.


New in version 3.3.


@tab

Writable

@item

@code{__module__}

@tab

The name of the module the
function was defined in, or
@code{None} if unavailable.

@tab

Writable

@item

@code{__defaults__}

@tab

A tuple containing default
argument values for those
arguments that have defaults,
or @code{None} if no arguments
have a default value.

@tab

Writable

@item

@code{__code__}

@tab

The code object representing
the compiled function body.

@tab

Writable

@item

@code{__globals__}

@tab

A reference to the dictionary
that holds the function’s
global variables — the
global namespace of the
module in which the function
was defined.

@tab

Read-only

@item

@ref{7a4,,__dict__}

@tab

The namespace supporting
arbitrary function
attributes.

@tab

Writable

@item

@code{__closure__}

@tab

@code{None} or a tuple of cells
that contain bindings for the
function’s free variables.
See below for information on
the @code{cell_contents}
attribute.

@tab

Read-only

@item

@code{__annotations__}

@tab

A dict containing annotations
of parameters.  The keys of
the dict are the parameter
names, and @code{'return'} for
the return annotation, if
provided.  For more
information on working with
this attribute, see
@ref{255,,Annotations Best Practices}.

@tab

Writable

@item

@code{__kwdefaults__}

@tab

A dict containing defaults
for keyword-only parameters.

@tab

Writable

@end multitable


Most of the attributes labelled “Writable” check the type of the assigned value.

Function objects also support getting and setting arbitrary attributes, which
can be used, for example, to attach metadata to functions.  Regular attribute
dot-notation is used to get and set such attributes. @emph{Note that the current
implementation only supports function attributes on user-defined functions.
Function attributes on built-in functions may be supported in the future.}

A cell object has the attribute @code{cell_contents}. This can be used to get
the value of the cell, as well as set the value.

Additional information about a function’s definition can be retrieved from its
code object; see the description of internal types below. The
@ref{1791,,cell} type can be accessed in the @ref{11f,,types}
module.

@item Instance methods

@geindex object; method
@geindex object; user-defined method
@geindex user-defined; method

An instance method object combines a class, a class instance and any
callable object (normally a user-defined function).

@geindex __func__ (method attribute)
@geindex __self__ (method attribute)
@geindex __doc__ (method attribute)
@geindex __name__ (method attribute)
@geindex __module__ (method attribute)

Special read-only attributes: @code{__self__} is the class instance object,
@code{__func__} is the function object; @code{__doc__} is the method’s
documentation (same as @code{__func__.__doc__}); @ref{e70,,__name__} is the
method name (same as @code{__func__.__name__}); @code{__module__} is the
name of the module the method was defined in, or @code{None} if unavailable.

Methods also support accessing (but not setting) the arbitrary function
attributes on the underlying function object.

User-defined method objects may be created when getting an attribute of a
class (perhaps via an instance of that class), if that attribute is a
user-defined function object or a class method object.

When an instance method object is created by retrieving a user-defined
function object from a class via one of its instances, its
@code{__self__} attribute is the instance, and the method object is said
to be bound.  The new method’s @code{__func__} attribute is the original
function object.

When an instance method object is created by retrieving a class method
object from a class or instance, its @code{__self__} attribute is the
class itself, and its @code{__func__} attribute is the function object
underlying the class method.

When an instance method object is called, the underlying function
(@code{__func__}) is called, inserting the class instance
(@code{__self__}) in front of the argument list.  For instance, when
@code{C} is a class which contains a definition for a function
@code{f()}, and @code{x} is an instance of @code{C}, calling @code{x.f(1)} is
equivalent to calling @code{C.f(x, 1)}.

When an instance method object is derived from a class method object, the
“class instance” stored in @code{__self__} will actually be the class
itself, so that calling either @code{x.f(1)} or @code{C.f(1)} is equivalent to
calling @code{f(C,1)} where @code{f} is the underlying function.

Note that the transformation from function object to instance method
object happens each time the attribute is retrieved from the instance.  In
some cases, a fruitful optimization is to assign the attribute to a local
variable and call that local variable. Also notice that this
transformation only happens for user-defined functions; other callable
objects (and all non-callable objects) are retrieved without
transformation.  It is also important to note that user-defined functions
which are attributes of a class instance are not converted to bound
methods; this @emph{only} happens when the function is an attribute of the
class.

@item Generator functions

@geindex generator; function
@geindex generator; iterator

A function or method which uses the @ref{4c5,,yield} statement (see section
@ref{4c5,,The yield statement}) is called a @emph{generator function}.  Such a function, when
called, always returns an @ref{11c3,,iterator} object which can be used to
execute the body of the function:  calling the iterator’s
@ref{e6d,,iterator.__next__()} method will cause the function to execute until
it provides a value using the @code{yield} statement.  When the
function executes a @ref{4c6,,return} statement or falls off the end, a
@ref{736,,StopIteration} exception is raised and the iterator will have
reached the end of the set of values to be returned.

@item Coroutine functions

@geindex coroutine; function

A function or method which is defined using @ref{1c0,,async def} is called
a @emph{coroutine function}.  Such a function, when called, returns a
@ref{3cf,,coroutine} object.  It may contain @ref{4d5,,await} expressions,
as well as @ref{169,,async with} and @ref{5cd,,async for} statements. See
also the @ref{1792,,Coroutine Objects} section.

@item Asynchronous generator functions

@geindex asynchronous generator; function
@geindex asynchronous generator; asynchronous iterator

A function or method which is defined using @ref{1c0,,async def} and
which uses the @ref{4c5,,yield} statement is called a
@emph{asynchronous generator function}.  Such a function, when called,
returns an @ref{8d2,,asynchronous iterator} object which can be used in an
@ref{5cd,,async for} statement to execute the body of the function.

Calling the asynchronous iterator’s
@ref{1793,,aiterator.__anext__} method
will return an @ref{7bd,,awaitable} which when awaited
will execute until it provides a value using the @ref{4c5,,yield}
expression.  When the function executes an empty @ref{4c6,,return}
statement or falls off the end, a @ref{12da,,StopAsyncIteration} exception
is raised and the asynchronous iterator will have reached the end of
the set of values to be yielded.

@item Built-in functions

@geindex object; built-in function
@geindex object; function
@geindex C; language

A built-in function object is a wrapper around a C function.  Examples of
built-in functions are @ref{4a6,,len()} and @ref{1008,,math.sin()} (@ref{ad,,math} is a
standard built-in module). The number and type of the arguments are
determined by the C function. Special read-only attributes:
@code{__doc__} is the function’s documentation string, or @code{None} if
unavailable; @ref{e70,,__name__} is the function’s name; @code{__self__} is
set to @code{None} (but see the next item); @code{__module__} is the name of
the module the function was defined in or @code{None} if unavailable.

@item Built-in methods

@geindex object; built-in method
@geindex object; method
@geindex built-in; method

This is really a different disguise of a built-in function, this time containing
an object passed to the C function as an implicit extra argument.  An example of
a built-in method is @code{alist.append()}, assuming @emph{alist} is a list object. In
this case, the special read-only attribute @code{__self__} is set to the object
denoted by @emph{alist}.

@item Classes

Classes are callable.  These objects normally act as factories for new
instances of themselves, but variations are possible for class types that
override @ref{1794,,__new__()}.  The arguments of the call are passed to
@code{__new__()} and, in the typical case, to @ref{f59,,__init__()} to
initialize the new instance.

@item Class Instances

Instances of arbitrary classes can be made callable by defining a
@ref{1795,,__call__()} method in their class.
@end table

@item Modules

@geindex statement; import
@geindex object; module

Modules are a basic organizational unit of Python code, and are created by
the @ref{1796,,import system} as invoked either by the
@ref{e35,,import} statement, or by calling
functions such as @ref{d3e,,importlib.import_module()} and built-in
@ref{3b7,,__import__()}.  A module object has a namespace implemented by a
dictionary object (this is the dictionary referenced by the @code{__globals__}
attribute of functions defined in the module).  Attribute references are
translated to lookups in this dictionary, e.g., @code{m.x} is equivalent to
@code{m.__dict__["x"]}. A module object does not contain the code object used
to initialize the module (since it isn’t needed once the initialization is
done).

Attribute assignment updates the module’s namespace dictionary, e.g.,
@code{m.x = 1} is equivalent to @code{m.__dict__["x"] = 1}.

@geindex __name__ (module attribute)
@geindex __doc__ (module attribute)
@geindex __file__ (module attribute)
@geindex __annotations__ (module attribute)
@geindex module; namespace

Predefined (writable) attributes:

@quotation


@table @asis

@item @ref{f12,,__name__}

The module’s name.

@item @code{__doc__}

The module’s documentation string, or @code{None} if
unavailable.

@item @ref{1797,,__file__}

The pathname of the file from which the
module was loaded, if it was loaded from a file.
The @ref{1797,,__file__}
attribute may be missing for certain types of modules, such as C modules
that are statically linked into the interpreter.  For extension modules
loaded dynamically from a shared library, it’s the pathname of the shared
library file.

@item @code{__annotations__}

A dictionary containing
@ref{8af,,variable annotations} collected during
module body execution.  For best practices on working
with @code{__annotations__}, please see @ref{255,,Annotations Best Practices}.
@end table
@end quotation

@geindex __dict__ (module attribute)

Special read-only attribute: @ref{7a4,,__dict__} is the module’s
namespace as a dictionary object.

@strong{CPython implementation detail:} Because of the way CPython clears module dictionaries, the module
dictionary will be cleared when the module falls out of scope even if the
dictionary still has live references.  To avoid this, copy the dictionary
or keep the module around while using its dictionary directly.

@item Custom classes

Custom class types are typically created by class definitions (see section
@ref{e72,,Class definitions}).  A class has a namespace implemented by a dictionary object.
Class attribute references are translated to lookups in this dictionary, e.g.,
@code{C.x} is translated to @code{C.__dict__["x"]} (although there are a number of
hooks which allow for other means of locating attributes). When the attribute
name is not found there, the attribute search continues in the base classes.
This search of the base classes uses the C3 method resolution order which
behaves correctly even in the presence of ‘diamond’ inheritance structures
where there are multiple inheritance paths leading back to a common ancestor.
Additional details on the C3 MRO used by Python can be found in the
documentation accompanying the 2.3 release at
@indicateurl{https://www.python.org/download/releases/2.3/mro/}.

@c XXX: Could we add that MRO doc as an appendix to the language ref?

@geindex object; class
@geindex object; class instance
@geindex object; instance
@geindex class object; call
@geindex container
@geindex object; dictionary
@geindex class; attribute

When a class attribute reference (for class @code{C}, say) would yield a
class method object, it is transformed into an instance method object whose
@code{__self__} attribute is @code{C}.  When it would yield a static
method object, it is transformed into the object wrapped by the static method
object. See section @ref{793,,Implementing Descriptors} for another way in which attributes
retrieved from a class may differ from those actually contained in its
@ref{7a4,,__dict__}.

@geindex class; attribute; assignment

Class attribute assignments update the class’s dictionary, never the dictionary
of a base class.

@geindex class object; call

A class object can be called (see above) to yield a class instance (see below).

@geindex __name__ (class attribute)
@geindex __module__ (class attribute)
@geindex __dict__ (class attribute)
@geindex __bases__ (class attribute)
@geindex __doc__ (class attribute)
@geindex __annotations__ (class attribute)

Special attributes:

@quotation


@table @asis

@item @ref{e70,,__name__}

The class name.

@item @code{__module__}

The name of the module in which the class was defined.

@item @ref{7a4,,__dict__}

The dictionary containing the class’s namespace.

@item @ref{ffe,,__bases__}

A tuple containing the base classes, in the order of
their occurrence in the base class list.

@item @code{__doc__}

The class’s documentation string, or @code{None} if undefined.

@item @code{__annotations__}

A dictionary containing
@ref{8af,,variable annotations}
collected during class body execution.  For best practices on
working with @code{__annotations__}, please see
@ref{255,,Annotations Best Practices}.
@end table
@end quotation

@item Class instances

@geindex object; class instance
@geindex object; instance
@geindex class; instance
@geindex class instance; attribute

A class instance is created by calling a class object (see above).  A class
instance has a namespace implemented as a dictionary which is the first place
in which attribute references are searched.  When an attribute is not found
there, and the instance’s class has an attribute by that name, the search
continues with the class attributes.  If a class attribute is found that is a
user-defined function object, it is transformed into an instance method
object whose @code{__self__} attribute is the instance.  Static method and
class method objects are also transformed; see above under “Classes”.  See
section @ref{793,,Implementing Descriptors} for another way in which attributes of a class
retrieved via its instances may differ from the objects actually stored in
the class’s @ref{7a4,,__dict__}.  If no class attribute is found, and the
object’s class has a @ref{709,,__getattr__()} method, that is called to satisfy
the lookup.

@geindex class instance; attribute; assignment

Attribute assignments and deletions update the instance’s dictionary, never a
class’s dictionary.  If the class has a @ref{1798,,__setattr__()} or
@ref{1799,,__delattr__()} method, this is called instead of updating the instance
dictionary directly.

@geindex object; numeric
@geindex object; sequence
@geindex object; mapping

Class instances can pretend to be numbers, sequences, or mappings if they have
methods with certain special names.  See section @ref{7b4,,Special method names}.

@geindex __dict__ (instance attribute)
@geindex __class__ (instance attribute)

Special attributes: @ref{7a4,,__dict__} is the attribute dictionary;
@ref{fff,,__class__} is the instance’s class.

@item I/O objects (also known as file objects)

@geindex built-in function; open
@geindex module; io
@geindex popen() (in module os)
@geindex makefile() (socket method)
@geindex sys.stdin
@geindex sys.stdout
@geindex sys.stderr
@geindex stdio
@geindex stdin (in module sys)
@geindex stdout (in module sys)
@geindex stderr (in module sys)

A @ref{d68,,file object} represents an open file.  Various shortcuts are
available to create file objects: the @ref{1c8,,open()} built-in function, and
also @ref{59f,,os.popen()}, @ref{179a,,os.fdopen()}, and the
@ref{d44,,makefile()} method of socket objects (and perhaps by
other functions or methods provided by extension modules).

The objects @code{sys.stdin}, @code{sys.stdout} and @code{sys.stderr} are
initialized to file objects corresponding to the interpreter’s standard
input, output and error streams; they are all open in text mode and
therefore follow the interface defined by the @ref{e47,,io.TextIOBase}
abstract class.

@item Internal types

@geindex internal type
@geindex types@comma{} internal

A few types used internally by the interpreter are exposed to the user. Their
definitions may change with future versions of the interpreter, but they are
mentioned here for completeness.

@geindex bytecode
@geindex object; code
@geindex code object


@table @asis

@item Code objects

Code objects represent @emph{byte-compiled} executable Python code, or @ref{8a8,,bytecode}.
The difference between a code object and a function object is that the function
object contains an explicit reference to the function’s globals (the module in
which it was defined), while a code object contains no context; also the default
argument values are stored in the function object, not in the code object
(because they represent values calculated at run-time).  Unlike function
objects, code objects are immutable and contain no references (directly or
indirectly) to mutable objects.

@geindex co_argcount (code object attribute)
@geindex co_posonlyargcount (code object attribute)
@geindex co_kwonlyargcount (code object attribute)
@geindex co_code (code object attribute)
@geindex co_consts (code object attribute)
@geindex co_filename (code object attribute)
@geindex co_firstlineno (code object attribute)
@geindex co_flags (code object attribute)
@geindex co_lnotab (code object attribute)
@geindex co_name (code object attribute)
@geindex co_names (code object attribute)
@geindex co_nlocals (code object attribute)
@geindex co_stacksize (code object attribute)
@geindex co_varnames (code object attribute)
@geindex co_cellvars (code object attribute)
@geindex co_freevars (code object attribute)
@geindex co_qualname (code object attribute)

Special read-only attributes: @code{co_name} gives the function name;
@code{co_qualname} gives the fully qualified function name;
@code{co_argcount} is the total number of positional arguments
(including positional-only arguments and arguments with default values);
@code{co_posonlyargcount} is the number of positional-only arguments
(including arguments with default values); @code{co_kwonlyargcount} is
the number of keyword-only arguments (including arguments with default
values); @code{co_nlocals} is the number of local variables used by the
function (including arguments); @code{co_varnames} is a tuple containing
the names of the local variables (starting with the argument names);
@code{co_cellvars} is a tuple containing the names of local variables
that are referenced by nested functions; @code{co_freevars} is a tuple
containing the names of free variables; @code{co_code} is a string
representing the sequence of bytecode instructions; @code{co_consts} is
a tuple containing the literals used by the bytecode; @code{co_names} is
a tuple containing the names used by the bytecode; @code{co_filename} is
the filename from which the code was compiled; @code{co_firstlineno} is
the first line number of the function; @code{co_lnotab} is a string
encoding the mapping from bytecode offsets to line numbers (for details
see the source code of the interpreter); @code{co_stacksize} is the
required stack size; @code{co_flags} is an integer encoding a number
of flags for the interpreter.

@geindex object; generator

The following flag bits are defined for @code{co_flags}: bit @code{0x04} is set if
the function uses the @code{*arguments} syntax to accept an arbitrary number of
positional arguments; bit @code{0x08} is set if the function uses the
@code{**keywords} syntax to accept arbitrary keyword arguments; bit @code{0x20} is set
if the function is a generator.

Future feature declarations (@code{from __future__ import division}) also use bits
in @code{co_flags} to indicate whether a code object was compiled with a
particular feature enabled: bit @code{0x2000} is set if the function was compiled
with future division enabled; bits @code{0x10} and @code{0x1000} were used in earlier
versions of Python.

Other bits in @code{co_flags} are reserved for internal use.

@geindex documentation string

If a code object represents a function, the first item in @code{co_consts} is
the documentation string of the function, or @code{None} if undefined.

@geindex co_positions() (codeobject method)
@anchor{reference/datamodel codeobject co_positions}@anchor{15c}
@deffn {Method} codeobject.co_positions ()

Returns an iterable over the source code positions of each bytecode
instruction in the code object.

The iterator returns tuples containing the @code{(start_line, end_line,
start_column, end_column)}. The @emph{i-th} tuple corresponds to the
position of the source code that compiled to the @emph{i-th} instruction.
Column information is 0-indexed utf-8 byte offsets on the given source
line.

This positional information can be missing. A non-exhaustive lists of
cases where this may happen:


@itemize -

@item 
Running the interpreter with @ref{159,,-X} @code{no_debug_ranges}.

@item 
Loading a pyc file compiled while using @ref{159,,-X} @code{no_debug_ranges}.

@item 
Position tuples corresponding to artificial instructions.

@item 
Line and column numbers that can’t be represented due to
implementation specific limitations.
@end itemize

When this occurs, some or all of the tuple elements can be
@ref{182,,None}.


New in version 3.11.


@cartouche
@quotation Note 
This feature requires storing column positions in code objects which may
result in a small increase of disk usage of compiled Python files or
interpreter memory usage. To avoid storing the extra information and/or
deactivate printing the extra traceback information, the
@ref{159,,-X} @code{no_debug_ranges} command line flag or the 
@geindex PYTHONNODEBUGRANGES
@geindex environment variable; PYTHONNODEBUGRANGES
@ref{15a,,PYTHONNODEBUGRANGES}
environment variable can be used.
@end quotation
@end cartouche
@end deffn
@end table
@anchor{reference/datamodel frame-objects}@anchor{179b}

@table @asis

@item Frame objects

@geindex object; frame

Frame objects represent execution frames.  They may occur in traceback objects
(see below), and are also passed to registered trace functions.

@geindex f_back (frame attribute)
@geindex f_code (frame attribute)
@geindex f_globals (frame attribute)
@geindex f_locals (frame attribute)
@geindex f_lasti (frame attribute)
@geindex f_builtins (frame attribute)

Special read-only attributes: @code{f_back} is to the previous stack frame
(towards the caller), or @code{None} if this is the bottom stack frame;
@code{f_code} is the code object being executed in this frame; @code{f_locals}
is the dictionary used to look up local variables; @code{f_globals} is used for
global variables; @code{f_builtins} is used for built-in (intrinsic) names;
@code{f_lasti} gives the precise instruction (this is an index into the
bytecode string of the code object).

Accessing @code{f_code} raises an @ref{1130,,auditing event}
@code{object.__getattr__} with arguments @code{obj} and @code{"f_code"}.

@geindex f_trace (frame attribute)
@geindex f_trace_lines (frame attribute)
@geindex f_trace_opcodes (frame attribute)
@geindex f_lineno (frame attribute)

Special writable attributes: @code{f_trace}, if not @code{None}, is a function
called for various events during code execution (this is used by the debugger).
Normally an event is triggered for each new source line - this can be
disabled by setting @code{f_trace_lines} to @ref{659,,False}.

Implementations @emph{may} allow per-opcode events to be requested by setting
@code{f_trace_opcodes} to @ref{748,,True}. Note that this may lead to
undefined interpreter behaviour if exceptions raised by the trace
function escape to the function being traced.

@code{f_lineno} is the current line number of the frame — writing to this
from within a trace function jumps to the given line (only for the bottom-most
frame).  A debugger can implement a Jump command (aka Set Next Statement)
by writing to f_lineno.

Frame objects support one method:

@geindex clear() (frame method)
@anchor{reference/datamodel frame clear}@anchor{a7f}
@deffn {Method} frame.clear ()

This method clears all references to local variables held by the
frame.  Also, if the frame belonged to a generator, the generator
is finalized.  This helps break reference cycles involving frame
objects (for example when catching an exception and storing its
traceback for later use).

@ref{5af,,RuntimeError} is raised if the frame is currently executing.


New in version 3.4.

@end deffn
@end table
@anchor{reference/datamodel traceback-objects}@anchor{610}

@table @asis

@item Traceback objects

@geindex object; traceback
@geindex stack; trace
@geindex exception; handler
@geindex execution; stack
@geindex exc_info (in module sys)
@geindex last_traceback (in module sys)
@geindex sys.exc_info
@geindex sys.last_traceback

Traceback objects represent a stack trace of an exception.  A traceback object
is implicitly created when an exception occurs, and may also be explicitly
created by calling @ref{60f,,types.TracebackType}.

For implicitly created tracebacks, when the search for an exception handler
unwinds the execution stack, at each unwound level a traceback object is
inserted in front of the current traceback.  When an exception handler is
entered, the stack trace is made available to the program. (See section
@ref{f6d,,The try statement}.) It is accessible as the third item of the
tuple returned by @code{sys.exc_info()}, and as the @code{__traceback__} attribute
of the caught exception.

When the program contains no suitable
handler, the stack trace is written (nicely formatted) to the standard error
stream; if the interpreter is interactive, it is also made available to the user
as @code{sys.last_traceback}.

For explicitly created tracebacks, it is up to the creator of the traceback
to determine how the @code{tb_next} attributes should be linked to form a
full stack trace.

@geindex tb_frame (traceback attribute)
@geindex tb_lineno (traceback attribute)
@geindex tb_lasti (traceback attribute)
@geindex statement; try

Special read-only attributes:
@code{tb_frame} points to the execution frame of the current level;
@code{tb_lineno} gives the line number where the exception occurred;
@code{tb_lasti} indicates the precise instruction.
The line number and last instruction in the traceback may differ from the
line number of its frame object if the exception occurred in a
@ref{f6d,,try} statement with no matching except clause or with a
finally clause.

Accessing @code{tb_frame} raises an @ref{1130,,auditing event}
@code{object.__getattr__} with arguments @code{obj} and @code{"tb_frame"}.

@geindex tb_next (traceback attribute)

Special writable attribute: @code{tb_next} is the next level in the stack
trace (towards the frame where the exception occurred), or @code{None} if
there is no next level.


Changed in version 3.7: Traceback objects can now be explicitly instantiated from Python code,
and the @code{tb_next} attribute of existing instances can be updated.


@item Slice objects

@geindex built-in function; slice

Slice objects are used to represent slices for
@ref{179c,,__getitem__()}
methods.  They are also created by the built-in @ref{ffb,,slice()} function.

@geindex start (slice object attribute)
@geindex stop (slice object attribute)
@geindex step (slice object attribute)

Special read-only attributes: @code{start} is the lower bound;
@code{stop} is the upper bound; @code{step} is the step
value; each is @code{None} if omitted.  These attributes can have any type.

Slice objects support one method:

@geindex indices() (slice method)
@anchor{reference/datamodel slice indices}@anchor{bb8}
@deffn {Method} slice.indices (self, length)

This method takes a single integer argument @emph{length} and computes
information about the slice that the slice object would describe if
applied to a sequence of @emph{length} items.  It returns a tuple of three
integers; respectively these are the @emph{start} and @emph{stop} indices and the
@emph{step} or stride length of the slice. Missing or out-of-bounds indices
are handled in a manner consistent with regular slices.
@end deffn

@item Static method objects

Static method objects provide a way of defeating the transformation of function
objects to method objects described above. A static method object is a wrapper
around any other object, usually a user-defined method object. When a static
method object is retrieved from a class or a class instance, the object actually
returned is the wrapped object, which is not subject to any further
transformation. Static method objects are also callable. Static method
objects are created by the built-in @ref{253,,staticmethod()} constructor.

@item Class method objects

A class method object, like a static method object, is a wrapper around another
object that alters the way in which that object is retrieved from classes and
class instances. The behaviour of class method objects upon such retrieval is
described above, under “User-defined methods”. Class method objects are created
by the built-in @ref{254,,classmethod()} constructor.
@end table
@end table

@node Special method names,Coroutines,The standard type hierarchy,Data model
@anchor{reference/datamodel special-method-names}@anchor{179d}@anchor{reference/datamodel specialnames}@anchor{7b4}
@subsection Special method names


@geindex operator; overloading
@geindex __getitem__() (mapping object method)

A class can implement certain operations that are invoked by special syntax
(such as arithmetic operations or subscripting and slicing) by defining methods
with special names. This is Python’s approach to @emph{operator overloading},
allowing classes to define their own behavior with respect to language
operators.  For instance, if a class defines a method named
@ref{179c,,__getitem__()},
and @code{x} is an instance of this class, then @code{x[i]} is roughly equivalent
to @code{type(x).__getitem__(x, i)}.  Except where mentioned, attempts to execute an
operation raise an exception when no appropriate method is defined (typically
@ref{163,,AttributeError} or @ref{162,,TypeError}).

Setting a special method to @code{None} indicates that the corresponding
operation is not available.  For example, if a class sets
@ref{179e,,__iter__()} to @code{None}, the class is not iterable, so calling
@ref{251,,iter()} on its instances will raise a @ref{162,,TypeError} (without
falling back to @ref{179c,,__getitem__()}). @footnote{@w{(2)} 
The @ref{619,,__hash__()}, @ref{179e,,__iter__()},
@ref{179f,,__reversed__()}, and @ref{17a0,,__contains__()} methods have
special handling for this; others
will still raise a @ref{162,,TypeError}, but may do so by relying on
the behavior that @code{None} is not callable.
}

When implementing a class that emulates any built-in type, it is important that
the emulation only be implemented to the degree that it makes sense for the
object being modelled.  For example, some sequences may work well with retrieval
of individual elements, but extracting a slice may not make sense.  (One example
of this is the @code{NodeList} interface in the W3C’s Document
Object Model.)

@menu
* Basic customization:: 
* Customizing attribute access:: 
* Customizing class creation:: 
* Customizing instance and subclass checks:: 
* Emulating generic types:: 
* Emulating callable objects:: 
* Emulating container types:: 
* Emulating numeric types:: 
* With Statement Context Managers:: 
* Customizing positional arguments in class pattern matching:: 
* Special method lookup:: 

@end menu

@node Basic customization,Customizing attribute access,,Special method names
@anchor{reference/datamodel basic-customization}@anchor{17a1}@anchor{reference/datamodel customization}@anchor{17a2}
@subsubsection Basic customization


@geindex __new__() (object method)
@anchor{reference/datamodel object __new__}@anchor{1794}
@deffn {Method} object.__new__ (cls[, ...])

@geindex subclassing; immutable types

Called to create a new instance of class @emph{cls}.  @ref{1794,,__new__()} is a static
method (special-cased so you need not declare it as such) that takes the class
of which an instance was requested as its first argument.  The remaining
arguments are those passed to the object constructor expression (the call to the
class).  The return value of @ref{1794,,__new__()} should be the new object instance
(usually an instance of @emph{cls}).

Typical implementations create a new instance of the class by invoking the
superclass’s @ref{1794,,__new__()} method using @code{super().__new__(cls[, ...])}
with appropriate arguments and then modifying the newly-created instance
as necessary before returning it.

If @ref{1794,,__new__()} is invoked during object construction and it returns an
instance of @emph{cls}, then the new instance’s @ref{f59,,__init__()} method
will be invoked like @code{__init__(self[, ...])}, where @emph{self} is the new instance
and the remaining arguments are the same as were passed to the object constructor.

If @ref{1794,,__new__()} does not return an instance of @emph{cls}, then the new instance’s
@ref{f59,,__init__()} method will not be invoked.

@ref{1794,,__new__()} is intended mainly to allow subclasses of immutable types (like
int, str, or tuple) to customize instance creation.  It is also commonly
overridden in custom metaclasses in order to customize class creation.
@end deffn

@geindex __init__() (object method)
@anchor{reference/datamodel object __init__}@anchor{f59}
@deffn {Method} object.__init__ (self[, ...])

@geindex class; constructor

Called after the instance has been created (by @ref{1794,,__new__()}), but before
it is returned to the caller.  The arguments are those passed to the
class constructor expression.  If a base class has an @ref{f59,,__init__()}
method, the derived class’s @ref{f59,,__init__()} method, if any, must explicitly
call it to ensure proper initialization of the base class part of the
instance; for example: @code{super().__init__([args...])}.

Because @ref{1794,,__new__()} and @ref{f59,,__init__()} work together in constructing
objects (@ref{1794,,__new__()} to create it, and @ref{f59,,__init__()} to customize it),
no non-@code{None} value may be returned by @ref{f59,,__init__()}; doing so will
cause a @ref{162,,TypeError} to be raised at runtime.
@end deffn

@geindex __del__() (object method)
@anchor{reference/datamodel object __del__}@anchor{17a3}
@deffn {Method} object.__del__ (self)

@geindex destructor
@geindex finalizer
@geindex statement; del

Called when the instance is about to be destroyed.  This is also called a
finalizer or (improperly) a destructor.  If a base class has a
@ref{17a3,,__del__()} method, the derived class’s @ref{17a3,,__del__()} method,
if any, must explicitly call it to ensure proper deletion of the base
class part of the instance.

It is possible (though not recommended!) for the @ref{17a3,,__del__()} method
to postpone destruction of the instance by creating a new reference to
it.  This is called object @emph{resurrection}.  It is implementation-dependent
whether @ref{17a3,,__del__()} is called a second time when a resurrected object
is about to be destroyed; the current @ref{17a4,,CPython} implementation
only calls it once.

It is not guaranteed that @ref{17a3,,__del__()} methods are called for objects
that still exist when the interpreter exits.

@cartouche
@quotation Note 
@code{del x} doesn’t directly call @code{x.__del__()} — the former decrements
the reference count for @code{x} by one, and the latter is only called when
@code{x}’s reference count reaches zero.
@end quotation
@end cartouche

@strong{CPython implementation detail:} It is possible for a reference cycle to prevent the reference count
of an object from going to zero.  In this case, the cycle will be
later detected and deleted by the @ref{1192,,cyclic garbage collector}.  A common cause of reference cycles is when
an exception has been caught in a local variable.  The frame’s
locals then reference the exception, which references its own
traceback, which references the locals of all frames caught in the
traceback.


@subsubheading See also


Documentation for the @ref{80,,gc} module.


@cartouche
@quotation Warning 
Due to the precarious circumstances under which @ref{17a3,,__del__()} methods are
invoked, exceptions that occur during their execution are ignored, and a warning
is printed to @code{sys.stderr} instead.  In particular:


@itemize *

@item 
@ref{17a3,,__del__()} can be invoked when arbitrary code is being executed,
including from any arbitrary thread.  If @ref{17a3,,__del__()} needs to take
a lock or invoke any other blocking resource, it may deadlock as
the resource may already be taken by the code that gets interrupted
to execute @ref{17a3,,__del__()}.

@item 
@ref{17a3,,__del__()} can be executed during interpreter shutdown.  As a
consequence, the global variables it needs to access (including other
modules) may already have been deleted or set to @code{None}. Python
guarantees that globals whose name begins with a single underscore
are deleted from their module before other globals are deleted; if
no other references to such globals exist, this may help in assuring
that imported modules are still available at the time when the
@ref{17a3,,__del__()} method is called.
@end itemize
@end quotation
@end cartouche

@geindex repr() (built-in function); __repr__() (object method)
@end deffn

@geindex __repr__() (object method)
@anchor{reference/datamodel object __repr__}@anchor{617}
@deffn {Method} object.__repr__ (self)

Called by the @ref{27d,,repr()} built-in function to compute the “official” string
representation of an object.  If at all possible, this should look like a
valid Python expression that could be used to recreate an object with the
same value (given an appropriate environment).  If this is not possible, a
string of the form @code{<...some useful description...>} should be returned.
The return value must be a string object. If a class defines @ref{617,,__repr__()}
but not @ref{178a,,__str__()}, then @ref{617,,__repr__()} is also used when an
“informal” string representation of instances of that class is required.

This is typically used for debugging, so it is important that the representation
is information-rich and unambiguous.

@geindex string; __str__() (object method)
@geindex format() (built-in function); __str__() (object method)
@geindex print() (built-in function); __str__() (object method)
@end deffn

@geindex __str__() (object method)
@anchor{reference/datamodel object __str__}@anchor{178a}
@deffn {Method} object.__str__ (self)

Called by @ref{2ef,,str(object)} and the built-in functions
@ref{785,,format()} and @ref{af6,,print()} to compute the “informal” or nicely
printable string representation of an object.  The return value must be a
@ref{14e2,,string} object.

This method differs from @ref{617,,object.__repr__()} in that there is no
expectation that @ref{178a,,__str__()} return a valid Python expression: a more
convenient or concise representation can be used.

The default implementation defined by the built-in type @ref{5a5,,object}
calls @ref{617,,object.__repr__()}.

@c XXX what about subclasses of string?
@end deffn

@geindex __bytes__() (object method)
@anchor{reference/datamodel object __bytes__}@anchor{17a5}
@deffn {Method} object.__bytes__ (self)

@geindex built-in function; bytes

Called by @ref{137e,,bytes} to compute a byte-string representation
of an object. This should return a @ref{2f0,,bytes} object.

@geindex string; __format__() (object method)
@geindex string; conversion
@geindex built-in function; print
@end deffn

@geindex __format__() (object method)
@anchor{reference/datamodel object __format__}@anchor{ba7}
@deffn {Method} object.__format__ (self, format_spec)

Called by the @ref{785,,format()} built-in function,
and by extension, evaluation of @ref{4b0,,formatted string literals} and the @ref{784,,str.format()} method, to produce a “formatted”
string representation of an object. The @emph{format_spec} argument is
a string that contains a description of the formatting options desired.
The interpretation of the @emph{format_spec} argument is up to the type
implementing @ref{ba7,,__format__()}, however most classes will either
delegate formatting to one of the built-in types, or use a similar
formatting option syntax.

See @ref{788,,Format Specification Mini-Language} for a description of the standard formatting syntax.

The return value must be a string object.


Changed in version 3.4: The __format__ method of @code{object} itself raises a @ref{162,,TypeError}
if passed any non-empty string.



Changed in version 3.7: @code{object.__format__(x, '@w{'})} is now equivalent to @code{str(x)} rather
than @code{format(str(x), '@w{'})}.

@end deffn
@anchor{reference/datamodel richcmpfuncs}@anchor{17a6}
@geindex __lt__() (object method)
@geindex __le__() (object method)
@geindex __eq__() (object method)
@geindex __ne__() (object method)
@geindex __gt__() (object method)
@geindex __ge__() (object method)
@anchor{reference/datamodel object __lt__}@anchor{17a7}
@deffn {Method} object.__lt__ (self, other)
@anchor{reference/datamodel object __le__}@anchor{17a8}
@deffnx {Method} object.__le__ (self, other)
@anchor{reference/datamodel object __eq__}@anchor{618}
@deffnx {Method} object.__eq__ (self, other)
@anchor{reference/datamodel object __ne__}@anchor{17a9}
@deffnx {Method} object.__ne__ (self, other)
@anchor{reference/datamodel object __gt__}@anchor{17aa}
@deffnx {Method} object.__gt__ (self, other)
@anchor{reference/datamodel object __ge__}@anchor{17ab}
@deffnx {Method} object.__ge__ (self, other)

@geindex comparisons

These are the so-called “rich comparison” methods. The correspondence between
operator symbols and method names is as follows: @code{x<y} calls @code{x.__lt__(y)},
@code{x<=y} calls @code{x.__le__(y)}, @code{x==y} calls @code{x.__eq__(y)}, @code{x!=y} calls
@code{x.__ne__(y)}, @code{x>y} calls @code{x.__gt__(y)}, and @code{x>=y} calls
@code{x.__ge__(y)}.

A rich comparison method may return the singleton @code{NotImplemented} if it does
not implement the operation for a given pair of arguments. By convention,
@code{False} and @code{True} are returned for a successful comparison. However, these
methods can return any value, so if the comparison operator is used in a Boolean
context (e.g., in the condition of an @code{if} statement), Python will call
@ref{2f7,,bool()} on the value to determine if the result is true or false.

By default, @code{object} implements @ref{618,,__eq__()} by using @code{is}, returning
@code{NotImplemented} in the case of a false comparison:
@code{True if x is y else NotImplemented}. For @ref{17a9,,__ne__()}, by default it
delegates to @ref{618,,__eq__()} and inverts the result unless it is
@code{NotImplemented}.  There are no other implied relationships among the
comparison operators or default implementations; for example, the truth of
@code{(x<y or x==y)} does not imply @code{x<=y}. To automatically generate ordering
operations from a single root operation, see @ref{abf,,functools.total_ordering()}.

See the paragraph on @ref{619,,__hash__()} for
some important notes on creating @ref{2dc,,hashable} objects which support
custom comparison operations and are usable as dictionary keys.

There are no swapped-argument versions of these methods (to be used when the
left argument does not support the operation but the right argument does);
rather, @ref{17a7,,__lt__()} and @ref{17aa,,__gt__()} are each other’s reflection,
@ref{17a8,,__le__()} and @ref{17ab,,__ge__()} are each other’s reflection, and
@ref{618,,__eq__()} and @ref{17a9,,__ne__()} are their own reflection.
If the operands are of different types, and right operand’s type is
a direct or indirect subclass of the left operand’s type,
the reflected method of the right operand has priority, otherwise
the left operand’s method has priority.  Virtual subclassing is
not considered.
@end deffn

@geindex __hash__() (object method)
@anchor{reference/datamodel object __hash__}@anchor{619}
@deffn {Method} object.__hash__ (self)

@geindex object; dictionary
@geindex built-in function; hash

Called by built-in function @ref{c14,,hash()} and for operations on members of
hashed collections including @ref{370,,set}, @ref{439,,frozenset}, and
@ref{3b0,,dict}.  @ref{619,,__hash__()} should return an integer. The only required
property is that objects which compare equal have the same hash value; it is
advised to mix together the hash values of the components of the object that
also play a part in comparison of objects by packing them into a tuple and
hashing the tuple. Example:

@example
def __hash__(self):
    return hash((self.name, self.nick, self.color))
@end example

@cartouche
@quotation Note 
@ref{c14,,hash()} truncates the value returned from an object’s custom
@ref{619,,__hash__()} method to the size of a @code{Py_ssize_t}.  This is
typically 8 bytes on 64-bit builds and 4 bytes on 32-bit builds.  If an
object’s   @ref{619,,__hash__()} must interoperate on builds of different bit
sizes, be sure to check the width on all supported builds.  An easy way
to do this is with
@code{python -c "import sys; print(sys.hash_info.width)"}.
@end quotation
@end cartouche

If a class does not define an @ref{618,,__eq__()} method it should not define a
@ref{619,,__hash__()} operation either; if it defines @ref{618,,__eq__()} but not
@ref{619,,__hash__()}, its instances will not be usable as items in hashable
collections.  If a class defines mutable objects and implements an
@ref{618,,__eq__()} method, it should not implement @ref{619,,__hash__()}, since the
implementation of hashable collections requires that a key’s hash value is
immutable (if the object’s hash value changes, it will be in the wrong hash
bucket).

User-defined classes have @ref{618,,__eq__()} and @ref{619,,__hash__()} methods
by default; with them, all objects compare unequal (except with themselves)
and @code{x.__hash__()} returns an appropriate value such that @code{x == y}
implies both that @code{x is y} and @code{hash(x) == hash(y)}.

A class that overrides @ref{618,,__eq__()} and does not define @ref{619,,__hash__()}
will have its @ref{619,,__hash__()} implicitly set to @code{None}.  When the
@ref{619,,__hash__()} method of a class is @code{None}, instances of the class will
raise an appropriate @ref{162,,TypeError} when a program attempts to retrieve
their hash value, and will also be correctly identified as unhashable when
checking @code{isinstance(obj, collections.abc.Hashable)}.

If a class that overrides @ref{618,,__eq__()} needs to retain the implementation
of @ref{619,,__hash__()} from a parent class, the interpreter must be told this
explicitly by setting @code{__hash__ = <ParentClass>.__hash__}.

If a class that does not override @ref{618,,__eq__()} wishes to suppress hash
support, it should include @code{__hash__ = None} in the class definition.
A class which defines its own @ref{619,,__hash__()} that explicitly raises
a @ref{162,,TypeError} would be incorrectly identified as hashable by
an @code{isinstance(obj, collections.abc.Hashable)} call.

@cartouche
@quotation Note 
By default, the @ref{619,,__hash__()} values of str and bytes objects are
“salted” with an unpredictable random value.  Although they
remain constant within an individual Python process, they are not
predictable between repeated invocations of Python.

This is intended to provide protection against a denial-of-service caused
by carefully-chosen inputs that exploit the worst case performance of a
dict insertion, O(n@w{^2}) complexity.  See
@indicateurl{http://www.ocert.org/advisories/ocert-2011-003.html} for details.

Changing hash values affects the iteration order of sets.
Python has never made guarantees about this ordering
(and it typically varies between 32-bit and 64-bit builds).

See also 
@geindex PYTHONHASHSEED
@geindex environment variable; PYTHONHASHSEED
@ref{c15,,PYTHONHASHSEED}.
@end quotation
@end cartouche


Changed in version 3.3: Hash randomization is enabled by default.

@end deffn

@geindex __bool__() (object method)
@anchor{reference/datamodel object __bool__}@anchor{17ac}
@deffn {Method} object.__bool__ (self)

@geindex __len__() (mapping object method)

Called to implement truth value testing and the built-in operation
@code{bool()}; should return @code{False} or @code{True}.  When this method is not
defined, @ref{17ad,,__len__()} is called, if it is defined, and the object is
considered true if its result is nonzero.  If a class defines neither
@ref{17ad,,__len__()} nor @ref{17ac,,__bool__()}, all its instances are considered
true.
@end deffn

@node Customizing attribute access,Customizing class creation,Basic customization,Special method names
@anchor{reference/datamodel attribute-access}@anchor{17ae}@anchor{reference/datamodel customizing-attribute-access}@anchor{17af}
@subsubsection Customizing attribute access


The following methods can be defined to customize the meaning of attribute
access (use of, assignment to, or deletion of @code{x.name}) for class instances.

@c XXX explain how descriptors interfere here!

@geindex __getattr__() (object method)
@anchor{reference/datamodel object __getattr__}@anchor{709}
@deffn {Method} object.__getattr__ (self, name)

Called when the default attribute access fails with an @ref{163,,AttributeError}
(either @ref{70a,,__getattribute__()} raises an @ref{163,,AttributeError} because
@emph{name} is not an instance attribute or an attribute in the class tree
for @code{self}; or @ref{17b0,,__get__()} of a @emph{name} property raises
@ref{163,,AttributeError}).  This method should either return the (computed)
attribute value or raise an @ref{163,,AttributeError} exception.

Note that if the attribute is found through the normal mechanism,
@ref{709,,__getattr__()} is not called.  (This is an intentional asymmetry between
@ref{709,,__getattr__()} and @ref{1798,,__setattr__()}.) This is done both for efficiency
reasons and because otherwise @ref{709,,__getattr__()} would have no way to access
other attributes of the instance.  Note that at least for instance variables,
you can fake total control by not inserting any values in the instance attribute
dictionary (but instead inserting them in another object).  See the
@ref{70a,,__getattribute__()} method below for a way to actually get total control
over attribute access.
@end deffn

@geindex __getattribute__() (object method)
@anchor{reference/datamodel object __getattribute__}@anchor{70a}
@deffn {Method} object.__getattribute__ (self, name)

Called unconditionally to implement attribute accesses for instances of the
class. If the class also defines @ref{709,,__getattr__()}, the latter will not be
called unless @ref{70a,,__getattribute__()} either calls it explicitly or raises an
@ref{163,,AttributeError}. This method should return the (computed) attribute value
or raise an @ref{163,,AttributeError} exception. In order to avoid infinite
recursion in this method, its implementation should always call the base class
method with the same name to access any attributes it needs, for example,
@code{object.__getattribute__(self, name)}.

@cartouche
@quotation Note 
This method may still be bypassed when looking up special methods as the
result of implicit invocation via language syntax or built-in functions.
See @ref{17b1,,Special method lookup}.
@end quotation
@end cartouche


For certain sensitive attribute accesses, raises an
@ref{1130,,auditing event} @code{object.__getattr__} with arguments
@code{obj} and @code{name}.

@end deffn

@geindex __setattr__() (object method)
@anchor{reference/datamodel object __setattr__}@anchor{1798}
@deffn {Method} object.__setattr__ (self, name, value)

Called when an attribute assignment is attempted.  This is called instead of
the normal mechanism (i.e. store the value in the instance dictionary).
@emph{name} is the attribute name, @emph{value} is the value to be assigned to it.

If @ref{1798,,__setattr__()} wants to assign to an instance attribute, it should
call the base class method with the same name, for example,
@code{object.__setattr__(self, name, value)}.


For certain sensitive attribute assignments, raises an
@ref{1130,,auditing event} @code{object.__setattr__} with arguments
@code{obj}, @code{name}, @code{value}.

@end deffn

@geindex __delattr__() (object method)
@anchor{reference/datamodel object __delattr__}@anchor{1799}
@deffn {Method} object.__delattr__ (self, name)

Like @ref{1798,,__setattr__()} but for attribute deletion instead of assignment.  This
should only be implemented if @code{del obj.name} is meaningful for the object.


For certain sensitive attribute deletions, raises an
@ref{1130,,auditing event} @code{object.__delattr__} with arguments
@code{obj} and @code{name}.

@end deffn

@geindex __dir__() (object method)
@anchor{reference/datamodel object __dir__}@anchor{17b2}
@deffn {Method} object.__dir__ (self)

Called when @ref{f2f,,dir()} is called on the object. A sequence must be
returned. @ref{f2f,,dir()} converts the returned sequence to a list and sorts it.
@end deffn

@menu
* Customizing module attribute access:: 
* Implementing Descriptors:: 
* Invoking Descriptors:: 
* __slots__: __slots__<2>. 

@end menu

@node Customizing module attribute access,Implementing Descriptors,,Customizing attribute access
@anchor{reference/datamodel customizing-module-attribute-access}@anchor{17b3}
@subsubsection Customizing module attribute access


@geindex __getattr__ (module attribute)
@geindex __dir__ (module attribute)
@geindex __class__ (module attribute)

Special names @code{__getattr__} and @code{__dir__} can be also used to customize
access to module attributes. The @code{__getattr__} function at the module level
should accept one argument which is the name of an attribute and return the
computed value or raise an @ref{163,,AttributeError}. If an attribute is
not found on a module object through the normal lookup, i.e.
@ref{70a,,object.__getattribute__()}, then @code{__getattr__} is searched in
the module @code{__dict__} before raising an @ref{163,,AttributeError}. If found,
it is called with the attribute name and the result is returned.

The @code{__dir__} function should accept no arguments, and return a sequence of
strings that represents the names accessible on module. If present, this
function overrides the standard @ref{f2f,,dir()} search on a module.

For a more fine grained customization of the module behavior (setting
attributes, properties, etc.), one can set the @code{__class__} attribute of
a module object to a subclass of @ref{973,,types.ModuleType}. For example:

@example
import sys
from types import ModuleType

class VerboseModule(ModuleType):
    def __repr__(self):
        return f'Verbose @{self.__name__@}'

    def __setattr__(self, attr, value):
        print(f'Setting @{attr@}...')
        super().__setattr__(attr, value)

sys.modules[__name__].__class__ = VerboseModule
@end example

@cartouche
@quotation Note 
Defining module @code{__getattr__} and setting module @code{__class__} only
affect lookups made using the attribute access syntax – directly accessing
the module globals (whether by code within the module, or via a reference
to the module’s globals dictionary) is unaffected.
@end quotation
@end cartouche


Changed in version 3.5: @code{__class__} module attribute is now writable.



New in version 3.7: @code{__getattr__} and @code{__dir__} module attributes.



@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 562

@item PEP 562@footnote{https://www.python.org/dev/peps/pep-0562} - Module __getattr__ and __dir__

Describes the @code{__getattr__} and @code{__dir__} functions on modules.
@end table


@node Implementing Descriptors,Invoking Descriptors,Customizing module attribute access,Customizing attribute access
@anchor{reference/datamodel descriptors}@anchor{793}@anchor{reference/datamodel implementing-descriptors}@anchor{17b4}
@subsubsection Implementing Descriptors


The following methods only apply when an instance of the class containing the
method (a so-called @emph{descriptor} class) appears in an @emph{owner} class (the
descriptor must be in either the owner’s class dictionary or in the class
dictionary for one of its parents).  In the examples below, “the attribute”
refers to the attribute whose name is the key of the property in the owner
class’ @ref{7a4,,__dict__}.

@geindex __get__() (object method)
@anchor{reference/datamodel object __get__}@anchor{17b0}
@deffn {Method} object.__get__ (self, instance, owner=None)

Called to get the attribute of the owner class (class attribute access) or
of an instance of that class (instance attribute access). The optional
@emph{owner} argument is the owner class, while @emph{instance} is the instance that
the attribute was accessed through, or @code{None} when the attribute is
accessed through the @emph{owner}.

This method should return the computed attribute value or raise an
@ref{163,,AttributeError} exception.

@geindex Python Enhancement Proposals; PEP 252
PEP 252@footnote{https://www.python.org/dev/peps/pep-0252} specifies that @ref{17b0,,__get__()} is callable with one or two
arguments.  Python’s own built-in descriptors support this specification;
however, it is likely that some third-party tools have descriptors
that require both arguments.  Python’s own @ref{70a,,__getattribute__()}
implementation always passes in both arguments whether they are required
or not.
@end deffn

@geindex __set__() (object method)
@anchor{reference/datamodel object __set__}@anchor{17b5}
@deffn {Method} object.__set__ (self, instance, value)

Called to set the attribute on an instance @emph{instance} of the owner class to a
new value, @emph{value}.

Note, adding @ref{17b5,,__set__()} or @ref{17b6,,__delete__()} changes the kind of
descriptor to a “data descriptor”.  See @ref{17b7,,Invoking Descriptors} for
more details.
@end deffn

@geindex __delete__() (object method)
@anchor{reference/datamodel object __delete__}@anchor{17b6}
@deffn {Method} object.__delete__ (self, instance)

Called to delete the attribute on an instance @emph{instance} of the owner class.
@end deffn

The attribute @code{__objclass__} is interpreted by the @ref{9c,,inspect} module
as specifying the class where this object was defined (setting this
appropriately can assist in runtime introspection of dynamic class attributes).
For callables, it may indicate that an instance of the given type (or a
subclass) is expected or required as the first positional argument (for example,
CPython sets this attribute for unbound methods that are implemented in C).

@node Invoking Descriptors,__slots__<2>,Implementing Descriptors,Customizing attribute access
@anchor{reference/datamodel descriptor-invocation}@anchor{17b7}@anchor{reference/datamodel invoking-descriptors}@anchor{17b8}
@subsubsection Invoking Descriptors


In general, a descriptor is an object attribute with “binding behavior”, one
whose attribute access has been overridden by methods in the descriptor
protocol:  @ref{17b0,,__get__()}, @ref{17b5,,__set__()}, and
@ref{17b6,,__delete__()}. If any of
those methods are defined for an object, it is said to be a descriptor.

The default behavior for attribute access is to get, set, or delete the
attribute from an object’s dictionary. For instance, @code{a.x} has a lookup chain
starting with @code{a.__dict__['x']}, then @code{type(a).__dict__['x']}, and
continuing through the base classes of @code{type(a)} excluding metaclasses.

However, if the looked-up value is an object defining one of the descriptor
methods, then Python may override the default behavior and invoke the descriptor
method instead.  Where this occurs in the precedence chain depends on which
descriptor methods were defined and how they were called.

The starting point for descriptor invocation is a binding, @code{a.x}. How the
arguments are assembled depends on @code{a}:


@table @asis

@item Direct Call

The simplest and least common call is when user code directly invokes a
descriptor method:    @code{x.__get__(a)}.

@item Instance Binding

If binding to an object instance, @code{a.x} is transformed into the call:
@code{type(a).__dict__['x'].__get__(a, type(a))}.

@item Class Binding

If binding to a class, @code{A.x} is transformed into the call:
@code{A.__dict__['x'].__get__(None, A)}.

@item Super Binding

A dotted lookup such as @code{super(A, a).x} searches
@code{a.__class__.__mro__} for a base class @code{B} following @code{A} and then
returns @code{B.__dict__['x'].__get__(a, A)}.  If not a descriptor, @code{x} is
returned unchanged.
@end table

@c class Desc:
@c     def __get__(*args):
@c         return args
@c 
@c class B:
@c 
@c     x = Desc()
@c 
@c class A(B):
@c 
@c     x = 999
@c 
@c     def m(self):
@c         'Demonstrate these two descriptor invocations are equivalent'
@c         result1 = super(A, self).x
@c         result2 = B.__dict__['x'].__get__(self, A)
@c         return result1 == result2

@c >>> a = A()
@c >>> a.__class__.__mro__.index(B) > a.__class__.__mro__.index(A)
@c True
@c >>> super(A, a).x == B.__dict__['x'].__get__(a, A)
@c True
@c >>> a.m()
@c True

For instance bindings, the precedence of descriptor invocation depends on
which descriptor methods are defined.  A descriptor can define any combination
of @ref{17b0,,__get__()}, @ref{17b5,,__set__()} and
@ref{17b6,,__delete__()}.  If it does not
define @code{__get__()}, then accessing the attribute will return the descriptor
object itself unless there is a value in the object’s instance dictionary.  If
the descriptor defines @code{__set__()} and/or @code{__delete__()}, it is a data
descriptor; if it defines neither, it is a non-data descriptor.  Normally, data
descriptors define both @code{__get__()} and @code{__set__()}, while non-data
descriptors have just the @code{__get__()} method.  Data descriptors with
@code{__get__()} and @code{__set__()} (and/or @code{__delete__()}) defined always override a redefinition in an
instance dictionary.  In contrast, non-data descriptors can be overridden by
instances.

Python methods (including those decorated with
@ref{253,,@@staticmethod} and @ref{254,,@@classmethod}) are
implemented as non-data descriptors.  Accordingly, instances can redefine and
override methods.  This allows individual instances to acquire behaviors that
differ from other instances of the same class.

The @ref{501,,property()} function is implemented as a data descriptor. Accordingly,
instances cannot override the behavior of a property.

@node __slots__<2>,,Invoking Descriptors,Customizing attribute access
@anchor{reference/datamodel id3}@anchor{17b9}@anchor{reference/datamodel slots}@anchor{17ba}
@subsubsection __slots__


@emph{__slots__} allow us to explicitly declare data members (like
properties) and deny the creation of @ref{7a4,,__dict__} and @emph{__weakref__}
(unless explicitly declared in @emph{__slots__} or available in a parent.)

The space saved over using @ref{7a4,,__dict__} can be significant.
Attribute lookup speed can be significantly improved as well.

@geindex object.__slots__ (built-in variable)
@anchor{reference/datamodel object __slots__}@anchor{1025}
@deffn {Data} object.__slots__

This class variable can be assigned a string, iterable, or sequence of
strings with variable names used by instances.  @emph{__slots__} reserves space
for the declared variables and prevents the automatic creation of
@ref{7a4,,__dict__}
and @emph{__weakref__} for each instance.
@end deffn

@menu
* Notes on using __slots__:: 

@end menu

@node Notes on using __slots__,,,__slots__<2>
@anchor{reference/datamodel notes-on-using-slots}@anchor{17bb}
@subsubsection Notes on using @emph{__slots__}



@itemize *

@item 
When inheriting from a class without @emph{__slots__}, the
@ref{7a4,,__dict__} and
@emph{__weakref__} attribute of the instances will always be accessible.

@item 
Without a @ref{7a4,,__dict__} variable, instances cannot be assigned new
variables not
listed in the @emph{__slots__} definition.  Attempts to assign to an unlisted
variable name raises @ref{163,,AttributeError}. If dynamic assignment of new
variables is desired, then add @code{'__dict__'} to the sequence of strings in
the @emph{__slots__} declaration.

@item 
Without a @emph{__weakref__} variable for each instance, classes defining
@emph{__slots__} do not support @ref{12f,,weak references} to its instances.
If weak reference
support is needed, then add @code{'__weakref__'} to the sequence of strings in the
@emph{__slots__} declaration.

@item 
@emph{__slots__} are implemented at the class level by creating @ref{793,,descriptors}
for each variable name.  As a result, class attributes
cannot be used to set default values for instance variables defined by
@emph{__slots__}; otherwise, the class attribute would overwrite the descriptor
assignment.

@item 
The action of a @emph{__slots__} declaration is not limited to the class
where it is defined.  @emph{__slots__} declared in parents are available in
child classes. However, child subclasses will get a @ref{7a4,,__dict__} and
@emph{__weakref__} unless they also define @emph{__slots__} (which should only
contain names of any @emph{additional} slots).

@item 
If a class defines a slot also defined in a base class, the instance variable
defined by the base class slot is inaccessible (except by retrieving its
descriptor directly from the base class). This renders the meaning of the
program undefined.  In the future, a check may be added to prevent this.

@item 
Nonempty @emph{__slots__} does not work for classes derived from “variable-length”
built-in types such as @ref{1b1,,int}, @ref{2f0,,bytes} and @ref{3e5,,tuple}.

@item 
Any non-string @ref{dcc,,iterable} may be assigned to @emph{__slots__}.

@item 
If a @ref{3b0,,dictionary} is used to assign @emph{__slots__}, the dictionary
keys will be used as the slot names. The values of the dictionary can be used
to provide per-attribute docstrings that will be recognised by
@ref{500,,inspect.getdoc()} and displayed in the output of @ref{3bc,,help()}.

@item 
@ref{fff,,__class__} assignment works only if both classes have the
same @emph{__slots__}.

@item 
@ref{1595,,Multiple inheritance} with multiple slotted parent
classes can be used,
but only one parent is allowed to have attributes created by slots
(the other bases must have empty slot layouts) - violations raise
@ref{162,,TypeError}.

@item 
If an @ref{11c3,,iterator} is used for @emph{__slots__} then a @ref{17bc,,descriptor} is
created for each
of the iterator’s values. However, the @emph{__slots__} attribute will be an empty
iterator.
@end itemize

@node Customizing class creation,Customizing instance and subclass checks,Customizing attribute access,Special method names
@anchor{reference/datamodel class-customization}@anchor{78f}@anchor{reference/datamodel customizing-class-creation}@anchor{17bd}
@subsubsection Customizing class creation


Whenever a class inherits from another class, @ref{78d,,__init_subclass__()} is
called on that class. This way, it is possible to write classes which
change the behavior of subclasses. This is closely related to class
decorators, but where class decorators only affect the specific class they’re
applied to, @code{__init_subclass__} solely applies to future subclasses of the
class defining the method.

@geindex __init_subclass__() (object class method)
@anchor{reference/datamodel object __init_subclass__}@anchor{78d}
@deffn {Method} classmethod object.__init_subclass__ (cls)

This method is called whenever the containing class is subclassed.
@emph{cls} is then the new subclass. If defined as a normal instance method,
this method is implicitly converted to a class method.

Keyword arguments which are given to a new class are passed to
the parent’s class @code{__init_subclass__}. For compatibility with
other classes using @code{__init_subclass__}, one should take out the
needed keyword arguments and pass the others over to the base
class, as in:

@example
class Philosopher:
    def __init_subclass__(cls, /, default_name, **kwargs):
        super().__init_subclass__(**kwargs)
        cls.default_name = default_name

class AustralianPhilosopher(Philosopher, default_name="Bruce"):
    pass
@end example

The default implementation @code{object.__init_subclass__} does
nothing, but raises an error if it is called with any arguments.

@cartouche
@quotation Note 
The metaclass hint @code{metaclass} is consumed by the rest of the type
machinery, and is never passed to @code{__init_subclass__} implementations.
The actual metaclass (rather than the explicit hint) can be accessed as
@code{type(cls)}.
@end quotation
@end cartouche


New in version 3.6.

@end deffn

When a class is created, @code{type.__new__()} scans the class variables
and makes callbacks to those with a @ref{792,,__set_name__()} hook.

@geindex __set_name__() (object method)
@anchor{reference/datamodel object __set_name__}@anchor{792}
@deffn {Method} object.__set_name__ (self, owner, name)

Automatically called at the time the owning class @emph{owner} is
created. The object has been assigned to @emph{name} in that class:

@example
class A:
    x = C()  # Automatically calls: x.__set_name__(A, 'x')
@end example

If the class variable is assigned after the class is created,
@ref{792,,__set_name__()} will not be called automatically.
If needed, @ref{792,,__set_name__()} can be called directly:

@example
class A:
   pass

c = C()
A.x = c                  # The hook is not called
c.__set_name__(A, 'x')   # Manually invoke the hook
@end example

See @ref{78e,,Creating the class object} for more details.


New in version 3.6.

@end deffn

@menu
* Metaclasses:: 
* Resolving MRO entries:: 
* Determining the appropriate metaclass:: 
* Preparing the class namespace:: 
* Executing the class body:: 
* Creating the class object:: 
* Uses for metaclasses:: 

@end menu

@node Metaclasses,Resolving MRO entries,,Customizing class creation
@anchor{reference/datamodel id4}@anchor{17be}@anchor{reference/datamodel metaclasses}@anchor{17bf}
@subsubsection Metaclasses


@geindex metaclass
@geindex built-in function; type
@geindex = (equals); class definition

By default, classes are constructed using @ref{89e,,type()}. The class body is
executed in a new namespace and the class name is bound locally to the
result of @code{type(name, bases, namespace)}.

The class creation process can be customized by passing the @code{metaclass}
keyword argument in the class definition line, or by inheriting from an
existing class that included such an argument. In the following example,
both @code{MyClass} and @code{MySubclass} are instances of @code{Meta}:

@example
class Meta(type):
    pass

class MyClass(metaclass=Meta):
    pass

class MySubclass(MyClass):
    pass
@end example

Any other keyword arguments that are specified in the class definition are
passed through to all metaclass operations described below.

When a class definition is executed, the following steps occur:


@itemize *

@item 
MRO entries are resolved;

@item 
the appropriate metaclass is determined;

@item 
the class namespace is prepared;

@item 
the class body is executed;

@item 
the class object is created.
@end itemize

@node Resolving MRO entries,Determining the appropriate metaclass,Metaclasses,Customizing class creation
@anchor{reference/datamodel resolving-mro-entries}@anchor{17c0}
@subsubsection Resolving MRO entries


If a base that appears in class definition is not an instance of @ref{89e,,type},
then an @code{__mro_entries__} method is searched on it. If found, it is called
with the original bases tuple. This method must return a tuple of classes that
will be used instead of this base. The tuple may be empty, in such case
the original base is ignored.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 560
PEP 560@footnote{https://www.python.org/dev/peps/pep-0560} - Core support for typing module and generic types


@node Determining the appropriate metaclass,Preparing the class namespace,Resolving MRO entries,Customizing class creation
@anchor{reference/datamodel determining-the-appropriate-metaclass}@anchor{17c1}
@subsubsection Determining the appropriate metaclass


@geindex metaclass hint

The appropriate metaclass for a class definition is determined as follows:


@itemize *

@item 
if no bases and no explicit metaclass are given, then @ref{89e,,type()} is used;

@item 
if an explicit metaclass is given and it is @emph{not} an instance of
@ref{89e,,type()}, then it is used directly as the metaclass;

@item 
if an instance of @ref{89e,,type()} is given as the explicit metaclass, or
bases are defined, then the most derived metaclass is used.
@end itemize

The most derived metaclass is selected from the explicitly specified
metaclass (if any) and the metaclasses (i.e. @code{type(cls)}) of all specified
base classes. The most derived metaclass is one which is a subtype of @emph{all}
of these candidate metaclasses. If none of the candidate metaclasses meets
that criterion, then the class definition will fail with @code{TypeError}.

@node Preparing the class namespace,Executing the class body,Determining the appropriate metaclass,Customizing class creation
@anchor{reference/datamodel prepare}@anchor{7a5}@anchor{reference/datamodel preparing-the-class-namespace}@anchor{17c2}
@subsubsection Preparing the class namespace


@geindex __prepare__ (metaclass method)

Once the appropriate metaclass has been identified, then the class namespace
is prepared. If the metaclass has a @code{__prepare__} attribute, it is called
as @code{namespace = metaclass.__prepare__(name, bases, **kwds)} (where the
additional keyword arguments, if any, come from the class definition). The
@code{__prepare__} method should be implemented as a
@ref{254,,classmethod}. The
namespace returned by @code{__prepare__} is passed in to @code{__new__}, but when
the final class object is created the namespace is copied into a new @code{dict}.

If the metaclass has no @code{__prepare__} attribute, then the class namespace
is initialised as an empty ordered mapping.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3115

@item PEP 3115@footnote{https://www.python.org/dev/peps/pep-3115} - Metaclasses in Python 3000

Introduced the @code{__prepare__} namespace hook
@end table


@node Executing the class body,Creating the class object,Preparing the class namespace,Customizing class creation
@anchor{reference/datamodel executing-the-class-body}@anchor{17c3}
@subsubsection Executing the class body


@geindex class; body

The class body is executed (approximately) as
@code{exec(body, globals(), namespace)}. The key difference from a normal
call to @ref{34b,,exec()} is that lexical scoping allows the class body (including
any methods) to reference names from the current and outer scopes when the
class definition occurs inside a function.

However, even when the class definition occurs inside the function, methods
defined inside the class still cannot see names defined at the class scope.
Class variables must be accessed through the first parameter of instance or
class methods, or through the implicit lexically scoped @code{__class__} reference
described in the next section.

@node Creating the class object,Uses for metaclasses,Executing the class body,Customizing class creation
@anchor{reference/datamodel class-object-creation}@anchor{78e}@anchor{reference/datamodel creating-the-class-object}@anchor{17c4}
@subsubsection Creating the class object


@geindex __class__ (method cell)
@geindex __classcell__ (class namespace entry)

Once the class namespace has been populated by executing the class body,
the class object is created by calling
@code{metaclass(name, bases, namespace, **kwds)} (the additional keywords
passed here are the same as those passed to @code{__prepare__}).

This class object is the one that will be referenced by the zero-argument
form of @ref{78c,,super()}. @code{__class__} is an implicit closure reference
created by the compiler if any methods in a class body refer to either
@code{__class__} or @code{super}. This allows the zero argument form of
@ref{78c,,super()} to correctly identify the class being defined based on
lexical scoping, while the class or instance that was used to make the
current call is identified based on the first argument passed to the method.

@strong{CPython implementation detail:} In CPython 3.6 and later, the @code{__class__} cell is passed to the metaclass
as a @code{__classcell__} entry in the class namespace. If present, this must
be propagated up to the @code{type.__new__} call in order for the class to be
initialised correctly.
Failing to do so will result in a @ref{5af,,RuntimeError} in Python 3.8.

When using the default metaclass @ref{89e,,type}, or any metaclass that ultimately
calls @code{type.__new__}, the following additional customization steps are
invoked after creating the class object:


@enumerate 

@item 
The @code{type.__new__} method collects all of the attributes in the class
namespace that define a @ref{792,,__set_name__()} method;

@item 
Those @code{__set_name__} methods are called with the class
being defined and the assigned name of that particular attribute;

@item 
The @ref{78d,,__init_subclass__()} hook is called on the
immediate parent of the new class in its method resolution order.
@end enumerate

After the class object is created, it is passed to the class decorators
included in the class definition (if any) and the resulting object is bound
in the local namespace as the defined class.

When a new class is created by @code{type.__new__}, the object provided as the
namespace parameter is copied to a new ordered mapping and the original
object is discarded. The new copy is wrapped in a read-only proxy, which
becomes the @ref{7a4,,__dict__} attribute of the class object.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3135

@item PEP 3135@footnote{https://www.python.org/dev/peps/pep-3135} - New super

Describes the implicit @code{__class__} closure reference
@end table


@node Uses for metaclasses,,Creating the class object,Customizing class creation
@anchor{reference/datamodel uses-for-metaclasses}@anchor{17c5}
@subsubsection Uses for metaclasses


The potential uses for metaclasses are boundless. Some ideas that have been
explored include enum, logging, interface checking, automatic delegation,
automatic property creation, proxies, frameworks, and automatic resource
locking/synchronization.

@node Customizing instance and subclass checks,Emulating generic types,Customizing class creation,Special method names
@anchor{reference/datamodel customizing-instance-and-subclass-checks}@anchor{17c6}
@subsubsection Customizing instance and subclass checks


The following methods are used to override the default behavior of the
@ref{236,,isinstance()} and @ref{237,,issubclass()} built-in functions.

In particular, the metaclass @ref{a8d,,abc.ABCMeta} implements these methods in
order to allow the addition of Abstract Base Classes (ABCs) as “virtual base
classes” to any class or type (including built-in types), including other
ABCs.

@geindex __instancecheck__() (class method)
@anchor{reference/datamodel class __instancecheck__}@anchor{17c7}
@deffn {Method} class.__instancecheck__ (self, instance)

Return true if @emph{instance} should be considered a (direct or indirect)
instance of @emph{class}. If defined, called to implement @code{isinstance(instance,
class)}.
@end deffn

@geindex __subclasscheck__() (class method)
@anchor{reference/datamodel class __subclasscheck__}@anchor{17c8}
@deffn {Method} class.__subclasscheck__ (self, subclass)

Return true if @emph{subclass} should be considered a (direct or indirect)
subclass of @emph{class}.  If defined, called to implement @code{issubclass(subclass,
class)}.
@end deffn

Note that these methods are looked up on the type (metaclass) of a class.  They
cannot be defined as class methods in the actual class.  This is consistent with
the lookup of special methods that are called on instances, only in this
case the instance is itself a class.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3119

@item PEP 3119@footnote{https://www.python.org/dev/peps/pep-3119} - Introducing Abstract Base Classes

Includes the specification for customizing @ref{236,,isinstance()} and
@ref{237,,issubclass()} behavior through @ref{17c7,,__instancecheck__()} and
@ref{17c8,,__subclasscheck__()}, with motivation for this functionality
in the context of adding Abstract Base Classes (see the @ref{3,,abc}
module) to the language.
@end table


@node Emulating generic types,Emulating callable objects,Customizing instance and subclass checks,Special method names
@anchor{reference/datamodel emulating-generic-types}@anchor{17c9}
@subsubsection Emulating generic types


When using @ref{17ca,,type annotations}, it is often useful to
@emph{parameterize} a @ref{11b0,,generic type} using Python’s square-brackets notation.
For example, the annotation @code{list[int]} might be used to signify a
@ref{43a,,list} in which all the elements are of type @ref{1b1,,int}.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 484

@item PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} - Type Hints

Introducing Python’s framework for type annotations

@item @ref{268,,Generic Alias Types}

Documentation for objects representing parameterized generic classes

@item @ref{17cb,,Generics}, @ref{17cc,,user-defined generics} and @ref{11c4,,typing.Generic}

Documentation on how to implement generic classes that can be
parameterized at runtime and understood by static type-checkers.
@end table


A class can @emph{generally} only be parameterized if it defines the special
class method @code{__class_getitem__()}.

@geindex __class_getitem__() (object class method)
@anchor{reference/datamodel object __class_getitem__}@anchor{10a5}
@deffn {Method} classmethod object.__class_getitem__ (cls, key)

Return an object representing the specialization of a generic class
by type arguments found in @emph{key}.

When defined on a class, @code{__class_getitem__()} is automatically a class
method. As such, there is no need for it to be decorated with
@ref{254,,@@classmethod} when it is defined.
@end deffn

@menu
* The purpose of __class_getitem__:: 
* __class_getitem__ versus __getitem__:: 

@end menu

@node The purpose of __class_getitem__,__class_getitem__ versus __getitem__,,Emulating generic types
@anchor{reference/datamodel the-purpose-of-class-getitem}@anchor{17cd}
@subsubsection The purpose of @emph{__class_getitem__}


The purpose of @ref{10a5,,__class_getitem__()} is to allow runtime
parameterization of standard-library generic classes in order to more easily
apply @ref{17ce,,type hints} to these classes.

To implement custom generic classes that can be parameterized at runtime and
understood by static type-checkers, users should either inherit from a standard
library class that already implements @ref{10a5,,__class_getitem__()}, or
inherit from @ref{11c4,,typing.Generic}, which has its own implementation of
@code{__class_getitem__()}.

Custom implementations of @ref{10a5,,__class_getitem__()} on classes defined
outside of the standard library may not be understood by third-party
type-checkers such as mypy. Using @code{__class_getitem__()} on any class for
purposes other than type hinting is discouraged.

@node __class_getitem__ versus __getitem__,,The purpose of __class_getitem__,Emulating generic types
@anchor{reference/datamodel class-getitem-versus-getitem}@anchor{17cf}@anchor{reference/datamodel classgetitem-versus-getitem}@anchor{17d0}
@subsubsection @emph{__class_getitem__} versus @emph{__getitem__}


Usually, the @ref{17d1,,subscription} of an object using square
brackets will call the @ref{179c,,__getitem__()} instance method defined on
the object’s class. However, if the object being subscribed is itself a class,
the class method @ref{10a5,,__class_getitem__()} may be called instead.
@code{__class_getitem__()} should return a @ref{268,,GenericAlias}
object if it is properly defined.

Presented with the @ref{17d2,,expression} @code{obj[x]}, the Python interpreter
follows something like the following process to decide whether
@ref{179c,,__getitem__()} or @ref{10a5,,__class_getitem__()} should be
called:

@example
from inspect import isclass

def subscribe(obj, x):
    """Return the result of the expression 'obj[x]'"""

    class_of_obj = type(obj)

    # If the class of obj defines __getitem__,
    # call class_of_obj.__getitem__(obj, x)
    if hasattr(class_of_obj, '__getitem__'):
        return class_of_obj.__getitem__(obj, x)

    # Else, if obj is a class and defines __class_getitem__,
    # call obj.__class_getitem__(x)
    elif isclass(obj) and hasattr(obj, '__class_getitem__'):
        return obj.__class_getitem__(x)

    # Else, raise an exception
    else:
        raise TypeError(
            f"'@{class_of_obj.__name__@}' object is not subscriptable"
        )
@end example

In Python, all classes are themselves instances of other classes. The class of
a class is known as that class’s @ref{17d3,,metaclass}, and most classes have the
@ref{89e,,type} class as their metaclass. @ref{89e,,type} does not define
@ref{179c,,__getitem__()}, meaning that expressions such as @code{list[int]},
@code{dict[str, float]} and @code{tuple[str, bytes]} all result in
@ref{10a5,,__class_getitem__()} being called:

@example
>>> # list has class "type" as its metaclass, like most classes:
>>> type(list)
<class 'type'>
>>> type(dict) == type(list) == type(tuple) == type(str) == type(bytes)
True
>>> # "list[int]" calls "list.__class_getitem__(int)"
>>> list[int]
list[int]
>>> # list.__class_getitem__ returns a GenericAlias object:
>>> type(list[int])
<class 'types.GenericAlias'>
@end example

However, if a class has a custom metaclass that defines
@ref{179c,,__getitem__()}, subscribing the class may result in different
behaviour. An example of this can be found in the @ref{76,,enum} module:

@example
>>> from enum import Enum
>>> class Menu(Enum):
...     """A breakfast menu"""
...     SPAM = 'spam'
...     BACON = 'bacon'
...
>>> # Enum classes have a custom metaclass:
>>> type(Menu)
<class 'enum.EnumMeta'>
>>> # EnumMeta defines __getitem__,
>>> # so __class_getitem__ is not called,
>>> # and the result is not a GenericAlias object:
>>> Menu['SPAM']
<Menu.SPAM: 'spam'>
>>> type(Menu['SPAM'])
<enum 'Menu'>
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 560

@item PEP 560@footnote{https://www.python.org/dev/peps/pep-0560} - Core Support for typing module and generic types

Introducing @ref{10a5,,__class_getitem__()}, and outlining when a
@ref{17d1,,subscription} results in @code{__class_getitem__()}
being called instead of @ref{179c,,__getitem__()}
@end table


@node Emulating callable objects,Emulating container types,Emulating generic types,Special method names
@anchor{reference/datamodel callable-types}@anchor{17d4}@anchor{reference/datamodel emulating-callable-objects}@anchor{17d5}
@subsubsection Emulating callable objects


@geindex __call__() (object method)
@anchor{reference/datamodel object __call__}@anchor{1795}
@deffn {Method} object.__call__ (self[, args...])

@geindex call; instance

Called when the instance is “called” as a function; if this method is defined,
@code{x(arg1, arg2, ...)} roughly translates to @code{type(x).__call__(x, arg1, ...)}.
@end deffn

@node Emulating container types,Emulating numeric types,Emulating callable objects,Special method names
@anchor{reference/datamodel emulating-container-types}@anchor{17d6}@anchor{reference/datamodel sequence-types}@anchor{17d7}
@subsubsection Emulating container types


The following methods can be defined to implement container objects.  Containers
usually are @ref{14e7,,sequences} (such as @ref{43a,,lists} or
@ref{3e5,,tuples}) or @ref{d62,,mappings} (like
@ref{3b0,,dictionaries}),
but can represent other containers as well.  The first set of methods is used
either to emulate a sequence or to emulate a mapping; the difference is that for
a sequence, the allowable keys should be the integers @emph{k} for which @code{0 <= k <
N} where @emph{N} is the length of the sequence, or @ref{ffb,,slice} objects, which define a
range of items.  It is also recommended that mappings provide the methods
@code{keys()}, @code{values()}, @code{items()}, @code{get()}, @code{clear()},
@code{setdefault()}, @code{pop()}, @code{popitem()}, @code{copy()}, and
@code{update()} behaving similar to those for Python’s standard @ref{3b0,,dictionary}
objects.  The @ref{1d,,collections.abc} module provides a
@ref{c3f,,MutableMapping}
@ref{d5c,,abstract base class} to help create those methods from a base set of
@ref{179c,,__getitem__()}, @ref{17d8,,__setitem__()}, @ref{17d9,,__delitem__()}, and @code{keys()}.
Mutable sequences should provide methods @code{append()}, @code{count()},
@code{index()}, @code{extend()}, @code{insert()}, @code{pop()}, @code{remove()},
@code{reverse()} and @code{sort()}, like Python standard @ref{43a,,list}
objects. Finally,
sequence types should implement addition (meaning concatenation) and
multiplication (meaning repetition) by defining the methods
@ref{17da,,__add__()}, @ref{17db,,__radd__()}, @ref{17dc,,__iadd__()},
@ref{17dd,,__mul__()}, @ref{17de,,__rmul__()} and @ref{17df,,__imul__()}
described below; they should not define other numerical
operators.  It is recommended that both mappings and sequences implement the
@ref{17a0,,__contains__()} method to allow efficient use of the @code{in}
operator; for
mappings, @code{in} should search the mapping’s keys; for sequences, it should
search through the values.  It is further recommended that both mappings and
sequences implement the @ref{179e,,__iter__()} method to allow efficient iteration
through the container; for mappings, @code{__iter__()} should iterate
through the object’s keys; for sequences, it should iterate through the values.

@geindex __len__() (object method)
@anchor{reference/datamodel object __len__}@anchor{17ad}
@deffn {Method} object.__len__ (self)

@geindex built-in function; len
@geindex __bool__() (object method)

Called to implement the built-in function @ref{4a6,,len()}.  Should return the length
of the object, an integer @code{>=} 0.  Also, an object that doesn’t define a
@ref{17ac,,__bool__()} method and whose @ref{17ad,,__len__()} method returns zero is
considered to be false in a Boolean context.

@strong{CPython implementation detail:} In CPython, the length is required to be at most @ref{d69,,sys.maxsize}.
If the length is larger than @code{sys.maxsize} some features (such as
@ref{4a6,,len()}) may raise @ref{347,,OverflowError}.  To prevent raising
@code{OverflowError} by truth value testing, an object must define a
@ref{17ac,,__bool__()} method.
@end deffn

@geindex __length_hint__() (object method)
@anchor{reference/datamodel object __length_hint__}@anchor{a80}
@deffn {Method} object.__length_hint__ (self)

Called to implement @ref{aee,,operator.length_hint()}. Should return an estimated
length for the object (which may be greater or less than the actual length).
The length must be an integer @code{>=} 0. The return value may also be
@ref{24c,,NotImplemented}, which is treated the same as if the
@code{__length_hint__} method didn’t exist at all. This method is purely an
optimization and is never required for correctness.


New in version 3.4.

@end deffn

@geindex object; slice

@cartouche
@quotation Note 
Slicing is done exclusively with the following three methods.  A call like

@example
a[1:2] = b
@end example

is translated to

@example
a[slice(1, 2, None)] = b
@end example

and so forth.  Missing slice items are always filled in with @code{None}.
@end quotation
@end cartouche

@geindex __getitem__() (object method)
@anchor{reference/datamodel object __getitem__}@anchor{179c}
@deffn {Method} object.__getitem__ (self, key)

Called to implement evaluation of @code{self[key]}. For @ref{14e7,,sequence} types,
the accepted keys should be integers and slice objects.  Note that the
special interpretation of negative indexes (if the class wishes to emulate a
@ref{14e7,,sequence} type) is up to the @ref{179c,,__getitem__()} method. If @emph{key} is
of an inappropriate type, @ref{162,,TypeError} may be raised; if of a value
outside the set of indexes for the sequence (after any special
interpretation of negative values), @ref{1066,,IndexError} should be raised. For
@ref{d62,,mapping} types, if @emph{key} is missing (not in the container),
@ref{5bb,,KeyError} should be raised.

@cartouche
@quotation Note 
@ref{161,,for} loops expect that an @ref{1066,,IndexError} will be raised for
illegal indexes to allow proper detection of the end of the sequence.
@end quotation
@end cartouche

@cartouche
@quotation Note 
When @ref{17d1,,subscripting} a @emph{class}, the special
class method @ref{10a5,,__class_getitem__()} may be called instead of
@code{__getitem__()}. See @ref{17d0,,__class_getitem__ versus __getitem__} for more
details.
@end quotation
@end cartouche
@end deffn

@geindex __setitem__() (object method)
@anchor{reference/datamodel object __setitem__}@anchor{17d8}
@deffn {Method} object.__setitem__ (self, key, value)

Called to implement assignment to @code{self[key]}.  Same note as for
@ref{179c,,__getitem__()}.  This should only be implemented for mappings if the
objects support changes to the values for keys, or if new keys can be added, or
for sequences if elements can be replaced.  The same exceptions should be raised
for improper @emph{key} values as for the @ref{179c,,__getitem__()} method.
@end deffn

@geindex __delitem__() (object method)
@anchor{reference/datamodel object __delitem__}@anchor{17d9}
@deffn {Method} object.__delitem__ (self, key)

Called to implement deletion of @code{self[key]}.  Same note as for
@ref{179c,,__getitem__()}.  This should only be implemented for mappings if the
objects support removal of keys, or for sequences if elements can be removed
from the sequence.  The same exceptions should be raised for improper @emph{key}
values as for the @ref{179c,,__getitem__()} method.
@end deffn

@geindex __missing__() (object method)
@anchor{reference/datamodel object __missing__}@anchor{17e0}
@deffn {Method} object.__missing__ (self, key)

Called by @ref{3b0,,dict}.@ref{179c,,__getitem__()} to implement @code{self[key]} for dict subclasses
when key is not in the dictionary.
@end deffn

@geindex __iter__() (object method)
@anchor{reference/datamodel object __iter__}@anchor{179e}
@deffn {Method} object.__iter__ (self)

This method is called when an @ref{11c3,,iterator} is required for a container.
This method should return a new iterator object that can iterate over all the
objects in the container.  For mappings, it should iterate over the keys of
the container.
@end deffn

@geindex __reversed__() (object method)
@anchor{reference/datamodel object __reversed__}@anchor{179f}
@deffn {Method} object.__reversed__ (self)

Called (if present) by the @ref{2f6,,reversed()} built-in to implement
reverse iteration.  It should return a new iterator object that iterates
over all the objects in the container in reverse order.

If the @ref{179f,,__reversed__()} method is not provided, the @ref{2f6,,reversed()}
built-in will fall back to using the sequence protocol (@ref{17ad,,__len__()} and
@ref{179c,,__getitem__()}).  Objects that support the sequence protocol should
only provide @ref{179f,,__reversed__()} if they can provide an implementation
that is more efficient than the one provided by @ref{2f6,,reversed()}.
@end deffn

The membership test operators (@ref{2fc,,in} and @ref{17e1,,not in}) are normally
implemented as an iteration through a container. However, container objects can
supply the following special method with a more efficient implementation, which
also does not require the object be iterable.

@geindex __contains__() (object method)
@anchor{reference/datamodel object __contains__}@anchor{17a0}
@deffn {Method} object.__contains__ (self, item)

Called to implement membership test operators.  Should return true if @emph{item}
is in @emph{self}, false otherwise.  For mapping objects, this should consider the
keys of the mapping rather than the values or the key-item pairs.

For objects that don’t define @ref{17a0,,__contains__()}, the membership test first
tries iteration via @ref{179e,,__iter__()}, then the old sequence iteration
protocol via @ref{179c,,__getitem__()}, see @ref{17e2,,this section in the language reference}.
@end deffn

@node Emulating numeric types,With Statement Context Managers,Emulating container types,Special method names
@anchor{reference/datamodel emulating-numeric-types}@anchor{17e3}@anchor{reference/datamodel numeric-types}@anchor{17e4}
@subsubsection Emulating numeric types


The following methods can be defined to emulate numeric objects. Methods
corresponding to operations that are not supported by the particular kind of
number implemented (e.g., bitwise operations for non-integral numbers) should be
left undefined.

@geindex __add__() (object method)
@geindex __sub__() (object method)
@geindex __mul__() (object method)
@geindex __matmul__() (object method)
@geindex __truediv__() (object method)
@geindex __floordiv__() (object method)
@geindex __mod__() (object method)
@geindex __divmod__() (object method)
@geindex __pow__() (object method)
@geindex __lshift__() (object method)
@geindex __rshift__() (object method)
@geindex __and__() (object method)
@geindex __xor__() (object method)
@geindex __or__() (object method)
@anchor{reference/datamodel object __add__}@anchor{17da}
@deffn {Method} object.__add__ (self, other)
@anchor{reference/datamodel object __sub__}@anchor{17e5}
@deffnx {Method} object.__sub__ (self, other)
@anchor{reference/datamodel object __mul__}@anchor{17dd}
@deffnx {Method} object.__mul__ (self, other)
@anchor{reference/datamodel object __matmul__}@anchor{17e6}
@deffnx {Method} object.__matmul__ (self, other)
@anchor{reference/datamodel object __truediv__}@anchor{17e7}
@deffnx {Method} object.__truediv__ (self, other)
@anchor{reference/datamodel object __floordiv__}@anchor{17e8}
@deffnx {Method} object.__floordiv__ (self, other)
@anchor{reference/datamodel object __mod__}@anchor{17e9}
@deffnx {Method} object.__mod__ (self, other)
@anchor{reference/datamodel object __divmod__}@anchor{17ea}
@deffnx {Method} object.__divmod__ (self, other)
@anchor{reference/datamodel object __pow__}@anchor{24d}
@deffnx {Method} object.__pow__ (self, other[, modulo])
@anchor{reference/datamodel object __lshift__}@anchor{17eb}
@deffnx {Method} object.__lshift__ (self, other)
@anchor{reference/datamodel object __rshift__}@anchor{17ec}
@deffnx {Method} object.__rshift__ (self, other)
@anchor{reference/datamodel object __and__}@anchor{17ed}
@deffnx {Method} object.__and__ (self, other)
@anchor{reference/datamodel object __xor__}@anchor{17ee}
@deffnx {Method} object.__xor__ (self, other)
@anchor{reference/datamodel object __or__}@anchor{17ef}
@deffnx {Method} object.__or__ (self, other)

@geindex built-in function; divmod
@geindex built-in function; pow
@geindex built-in function; pow

These methods are called to implement the binary arithmetic operations
(@code{+}, @code{-}, @code{*}, @code{@@}, @code{/}, @code{//}, @code{%}, @ref{4a8,,divmod()},
@ref{4cd,,pow()}, @code{**}, @code{<<}, @code{>>}, @code{&}, @code{^}, @code{|}).  For instance, to
evaluate the expression @code{x + y}, where @emph{x} is an instance of a class that
has an @ref{17da,,__add__()} method, @code{type(x).__add__(x, y)} is called.  The
@ref{17ea,,__divmod__()} method should be the equivalent to using
@ref{17e8,,__floordiv__()} and @ref{17e9,,__mod__()}; it should not be related to
@ref{17e7,,__truediv__()}.  Note that @ref{24d,,__pow__()} should be defined to accept
an optional third argument if the ternary version of the built-in @ref{4cd,,pow()}
function is to be supported.

If one of those methods does not support the operation with the supplied
arguments, it should return @code{NotImplemented}.
@end deffn

@geindex __radd__() (object method)
@geindex __rsub__() (object method)
@geindex __rmul__() (object method)
@geindex __rmatmul__() (object method)
@geindex __rtruediv__() (object method)
@geindex __rfloordiv__() (object method)
@geindex __rmod__() (object method)
@geindex __rdivmod__() (object method)
@geindex __rpow__() (object method)
@geindex __rlshift__() (object method)
@geindex __rrshift__() (object method)
@geindex __rand__() (object method)
@geindex __rxor__() (object method)
@geindex __ror__() (object method)
@anchor{reference/datamodel object __radd__}@anchor{17db}
@deffn {Method} object.__radd__ (self, other)
@anchor{reference/datamodel object __rsub__}@anchor{17f0}
@deffnx {Method} object.__rsub__ (self, other)
@anchor{reference/datamodel object __rmul__}@anchor{17de}
@deffnx {Method} object.__rmul__ (self, other)
@anchor{reference/datamodel object __rmatmul__}@anchor{17f1}
@deffnx {Method} object.__rmatmul__ (self, other)
@anchor{reference/datamodel object __rtruediv__}@anchor{17f2}
@deffnx {Method} object.__rtruediv__ (self, other)
@anchor{reference/datamodel object __rfloordiv__}@anchor{17f3}
@deffnx {Method} object.__rfloordiv__ (self, other)
@anchor{reference/datamodel object __rmod__}@anchor{17f4}
@deffnx {Method} object.__rmod__ (self, other)
@anchor{reference/datamodel object __rdivmod__}@anchor{17f5}
@deffnx {Method} object.__rdivmod__ (self, other)
@anchor{reference/datamodel object __rpow__}@anchor{24e}
@deffnx {Method} object.__rpow__ (self, other[, modulo])
@anchor{reference/datamodel object __rlshift__}@anchor{17f6}
@deffnx {Method} object.__rlshift__ (self, other)
@anchor{reference/datamodel object __rrshift__}@anchor{17f7}
@deffnx {Method} object.__rrshift__ (self, other)
@anchor{reference/datamodel object __rand__}@anchor{17f8}
@deffnx {Method} object.__rand__ (self, other)
@anchor{reference/datamodel object __rxor__}@anchor{17f9}
@deffnx {Method} object.__rxor__ (self, other)
@anchor{reference/datamodel object __ror__}@anchor{17fa}
@deffnx {Method} object.__ror__ (self, other)

@geindex built-in function; divmod
@geindex built-in function; pow

These methods are called to implement the binary arithmetic operations
(@code{+}, @code{-}, @code{*}, @code{@@}, @code{/}, @code{//}, @code{%}, @ref{4a8,,divmod()},
@ref{4cd,,pow()}, @code{**}, @code{<<}, @code{>>}, @code{&}, @code{^}, @code{|}) with reflected
(swapped) operands.  These functions are only called if the left operand does
not support the corresponding operation @footnote{@w{(3)} 
“Does not support” here means that the class has no such method, or
the method returns @code{NotImplemented}.  Do not set the method to
@code{None} if you want to force fallback to the right operand’s reflected
method—that will instead have the opposite effect of explicitly
@emph{blocking} such fallback.
} and the operands are of different
types. @footnote{@w{(4)} 
For operands of the same type, it is assumed that if the non-reflected
method – such as @ref{17da,,__add__()} – fails then the overall
operation is not
supported, which is why the reflected method is not called.
} For instance, to evaluate the expression @code{x - y}, where @emph{y} is
an instance of a class that has an @ref{17f0,,__rsub__()} method,
@code{type(y).__rsub__(y, x)} is called if @code{type(x).__sub__(x, y)} returns
@emph{NotImplemented}.

@geindex built-in function; pow

Note that ternary @ref{4cd,,pow()} will not try calling @ref{24e,,__rpow__()} (the
coercion rules would become too complicated).

@cartouche
@quotation Note 
If the right operand’s type is a subclass of the left operand’s type and
that subclass provides a different implementation of the reflected method
for the operation, this method will be called before the left operand’s
non-reflected method. This behavior allows subclasses to override their
ancestors’ operations.
@end quotation
@end cartouche
@end deffn

@geindex __iadd__() (object method)
@geindex __isub__() (object method)
@geindex __imul__() (object method)
@geindex __imatmul__() (object method)
@geindex __itruediv__() (object method)
@geindex __ifloordiv__() (object method)
@geindex __imod__() (object method)
@geindex __ipow__() (object method)
@geindex __ilshift__() (object method)
@geindex __irshift__() (object method)
@geindex __iand__() (object method)
@geindex __ixor__() (object method)
@geindex __ior__() (object method)
@anchor{reference/datamodel object __iadd__}@anchor{17dc}
@deffn {Method} object.__iadd__ (self, other)
@anchor{reference/datamodel object __isub__}@anchor{17fb}
@deffnx {Method} object.__isub__ (self, other)
@anchor{reference/datamodel object __imul__}@anchor{17df}
@deffnx {Method} object.__imul__ (self, other)
@anchor{reference/datamodel object __imatmul__}@anchor{17fc}
@deffnx {Method} object.__imatmul__ (self, other)
@anchor{reference/datamodel object __itruediv__}@anchor{17fd}
@deffnx {Method} object.__itruediv__ (self, other)
@anchor{reference/datamodel object __ifloordiv__}@anchor{17fe}
@deffnx {Method} object.__ifloordiv__ (self, other)
@anchor{reference/datamodel object __imod__}@anchor{17ff}
@deffnx {Method} object.__imod__ (self, other)
@anchor{reference/datamodel object __ipow__}@anchor{24b}
@deffnx {Method} object.__ipow__ (self, other[, modulo])
@anchor{reference/datamodel object __ilshift__}@anchor{1800}
@deffnx {Method} object.__ilshift__ (self, other)
@anchor{reference/datamodel object __irshift__}@anchor{1801}
@deffnx {Method} object.__irshift__ (self, other)
@anchor{reference/datamodel object __iand__}@anchor{1802}
@deffnx {Method} object.__iand__ (self, other)
@anchor{reference/datamodel object __ixor__}@anchor{1803}
@deffnx {Method} object.__ixor__ (self, other)
@anchor{reference/datamodel object __ior__}@anchor{1804}
@deffnx {Method} object.__ior__ (self, other)

These methods are called to implement the augmented arithmetic assignments
(@code{+=}, @code{-=}, @code{*=}, @code{@@=}, @code{/=}, @code{//=}, @code{%=}, @code{**=}, @code{<<=},
@code{>>=}, @code{&=}, @code{^=}, @code{|=}).  These methods should attempt to do the
operation in-place (modifying @emph{self}) and return the result (which could be,
but does not have to be, @emph{self}).  If a specific method is not defined, the
augmented assignment falls back to the normal methods.  For instance, if @emph{x}
is an instance of a class with an @ref{17dc,,__iadd__()} method, @code{x += y} is
equivalent to @code{x = x.__iadd__(y)} . Otherwise, @code{x.__add__(y)} and
@code{y.__radd__(x)} are considered, as with the evaluation of @code{x + y}. In
certain situations, augmented assignment can result in unexpected errors (see
@ref{1805,,Why does a_tuple[i] += [‘item’] raise an exception when the addition works?}), but this behavior is in fact
part of the data model.
@end deffn

@geindex __neg__() (object method)
@geindex __pos__() (object method)
@geindex __abs__() (object method)
@geindex __invert__() (object method)
@anchor{reference/datamodel object __neg__}@anchor{1806}
@deffn {Method} object.__neg__ (self)
@anchor{reference/datamodel object __pos__}@anchor{1807}
@deffnx {Method} object.__pos__ (self)
@anchor{reference/datamodel object __abs__}@anchor{1808}
@deffnx {Method} object.__abs__ (self)
@anchor{reference/datamodel object __invert__}@anchor{1809}
@deffnx {Method} object.__invert__ (self)

@geindex built-in function; abs

Called to implement the unary arithmetic operations (@code{-}, @code{+}, @ref{1582,,abs()}
and @code{~}).
@end deffn

@geindex __complex__() (object method)
@geindex __int__() (object method)
@geindex __float__() (object method)
@anchor{reference/datamodel object __complex__}@anchor{4c4}
@deffn {Method} object.__complex__ (self)
@anchor{reference/datamodel object __int__}@anchor{249}
@deffnx {Method} object.__int__ (self)
@anchor{reference/datamodel object __float__}@anchor{4c3}
@deffnx {Method} object.__float__ (self)

@geindex built-in function; complex
@geindex built-in function; int
@geindex built-in function; float

Called to implement the built-in functions @ref{33b,,complex()},
@ref{1b1,,int()} and @ref{256,,float()}.  Should return a value
of the appropriate type.
@end deffn

@geindex __index__() (object method)
@anchor{reference/datamodel object __index__}@anchor{24a}
@deffn {Method} object.__index__ (self)

Called to implement @ref{1187,,operator.index()}, and whenever Python needs to
losslessly convert the numeric object to an integer object (such as in
slicing, or in the built-in @ref{e4e,,bin()}, @ref{e6f,,hex()} and @ref{e6e,,oct()}
functions). Presence of this method indicates that the numeric object is
an integer type.  Must return an integer.

If @ref{249,,__int__()}, @ref{4c3,,__float__()} and @ref{4c4,,__complex__()} are not
defined then corresponding built-in functions @ref{1b1,,int()}, @ref{256,,float()}
and @ref{33b,,complex()} fall back to @ref{24a,,__index__()}.
@end deffn

@geindex __round__() (object method)
@geindex __trunc__() (object method)
@geindex __floor__() (object method)
@geindex __ceil__() (object method)
@anchor{reference/datamodel object __round__}@anchor{180a}
@deffn {Method} object.__round__ (self[, ndigits])
@anchor{reference/datamodel object __trunc__}@anchor{180b}
@deffnx {Method} object.__trunc__ (self)
@anchor{reference/datamodel object __floor__}@anchor{180c}
@deffnx {Method} object.__floor__ (self)
@anchor{reference/datamodel object __ceil__}@anchor{180d}
@deffnx {Method} object.__ceil__ (self)

@geindex built-in function; round

Called to implement the built-in function @ref{e75,,round()} and @ref{ad,,math}
functions @ref{f2a,,trunc()}, @ref{f28,,floor()} and @ref{f29,,ceil()}.
Unless @emph{ndigits} is passed to @code{__round__()} all these methods should
return the value of the object truncated to an @ref{178b,,Integral}
(typically an @ref{1b1,,int}).

The built-in function @ref{1b1,,int()} falls back to @ref{180b,,__trunc__()} if neither
@ref{249,,__int__()} nor @ref{24a,,__index__()} is defined.


Changed in version 3.11: The delegation of @ref{1b1,,int()} to @ref{180b,,__trunc__()} is deprecated.

@end deffn

@node With Statement Context Managers,Customizing positional arguments in class pattern matching,Emulating numeric types,Special method names
@anchor{reference/datamodel context-managers}@anchor{180e}@anchor{reference/datamodel with-statement-context-managers}@anchor{180f}
@subsubsection With Statement Context Managers


A @emph{context manager} is an object that defines the runtime context to be
established when executing a @ref{168,,with} statement. The context manager
handles the entry into, and the exit from, the desired runtime context for the
execution of the block of code.  Context managers are normally invoked using the
@code{with} statement (described in section @ref{168,,The with statement}), but can also be
used by directly invoking their methods.

@geindex statement; with
@geindex context manager

Typical uses of context managers include saving and restoring various kinds of
global state, locking and unlocking resources, closing opened files, etc.

For more information on context managers, see @ref{1810,,Context Manager Types}.

@geindex __enter__() (object method)
@anchor{reference/datamodel object __enter__}@anchor{1811}
@deffn {Method} object.__enter__ (self)

Enter the runtime context related to this object. The @ref{168,,with} statement
will bind this method’s return value to the target(s) specified in the
@code{as} clause of the statement, if any.
@end deffn

@geindex __exit__() (object method)
@anchor{reference/datamodel object __exit__}@anchor{1812}
@deffn {Method} object.__exit__ (self, exc_type, exc_value, traceback)

Exit the runtime context related to this object. The parameters describe the
exception that caused the context to be exited. If the context was exited
without an exception, all three arguments will be @ref{182,,None}.

If an exception is supplied, and the method wishes to suppress the exception
(i.e., prevent it from being propagated), it should return a true value.
Otherwise, the exception will be processed normally upon exit from this method.

Note that @ref{1812,,__exit__()} methods should not reraise the passed-in exception;
this is the caller’s responsibility.
@end deffn


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 343

@item PEP 343@footnote{https://www.python.org/dev/peps/pep-0343} - The “with” statement

The specification, background, and examples for the Python @ref{168,,with}
statement.
@end table


@node Customizing positional arguments in class pattern matching,Special method lookup,With Statement Context Managers,Special method names
@anchor{reference/datamodel class-pattern-matching}@anchor{1813}@anchor{reference/datamodel customizing-positional-arguments-in-class-pattern-matching}@anchor{1814}
@subsubsection Customizing positional arguments in class pattern matching


When using a class name in a pattern, positional arguments in the pattern are not
allowed by default, i.e. @code{case MyClass(x, y)} is typically invalid without special
support in @code{MyClass}. To be able to use that kind of patterns, the class needs to
define a @emph{__match_args__} attribute.

@geindex object.__match_args__ (built-in variable)
@anchor{reference/datamodel object __match_args__}@anchor{1135}
@deffn {Data} object.__match_args__

This class variable can be assigned a tuple of strings. When this class is
used in a class pattern with positional arguments, each positional argument will
be converted into a keyword argument, using the corresponding value in
@emph{__match_args__} as the keyword. The absence of this attribute is equivalent to
setting it to @code{()}.
@end deffn

For example, if @code{MyClass.__match_args__} is @code{("left", "center", "right")} that means
that @code{case MyClass(x, y)} is equivalent to @code{case MyClass(left=x, center=y)}. Note
that the number of arguments in the pattern must be smaller than or equal to the number
of elements in @emph{__match_args__}; if it is larger, the pattern match attempt will raise
a @ref{162,,TypeError}.


New in version 3.10.



@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 634

@item PEP 634@footnote{https://www.python.org/dev/peps/pep-0634} - Structural Pattern Matching

The specification for the Python @code{match} statement.
@end table


@node Special method lookup,,Customizing positional arguments in class pattern matching,Special method names
@anchor{reference/datamodel special-lookup}@anchor{17b1}@anchor{reference/datamodel special-method-lookup}@anchor{1815}
@subsubsection Special method lookup


For custom classes, implicit invocations of special methods are only guaranteed
to work correctly if defined on an object’s type, not in the object’s instance
dictionary.  That behaviour is the reason why the following code raises an
exception:

@example
>>> class C:
...     pass
...
>>> c = C()
>>> c.__len__ = lambda: 5
>>> len(c)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object of type 'C' has no len()
@end example

The rationale behind this behaviour lies with a number of special methods such
as @ref{619,,__hash__()} and @ref{617,,__repr__()} that are implemented
by all objects,
including type objects. If the implicit lookup of these methods used the
conventional lookup process, they would fail when invoked on the type object
itself:

@example
>>> 1 .__hash__() == hash(1)
True
>>> int.__hash__() == hash(int)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: descriptor '__hash__' of 'int' object needs an argument
@end example

Incorrectly attempting to invoke an unbound method of a class in this way is
sometimes referred to as ‘metaclass confusion’, and is avoided by bypassing
the instance when looking up special methods:

@example
>>> type(1).__hash__(1) == hash(1)
True
>>> type(int).__hash__(int) == hash(int)
True
@end example

In addition to bypassing any instance attributes in the interest of
correctness, implicit special method lookup generally also bypasses the
@ref{70a,,__getattribute__()} method even of the object’s metaclass:

@example
>>> class Meta(type):
...     def __getattribute__(*args):
...         print("Metaclass getattribute invoked")
...         return type.__getattribute__(*args)
...
>>> class C(object, metaclass=Meta):
...     def __len__(self):
...         return 10
...     def __getattribute__(*args):
...         print("Class getattribute invoked")
...         return object.__getattribute__(*args)
...
>>> c = C()
>>> c.__len__()                 # Explicit lookup via instance
Class getattribute invoked
10
>>> type(c).__len__(c)          # Explicit lookup via type
Metaclass getattribute invoked
10
>>> len(c)                      # Implicit lookup
10
@end example

Bypassing the @ref{70a,,__getattribute__()} machinery in this fashion
provides significant scope for speed optimisations within the
interpreter, at the cost of some flexibility in the handling of
special methods (the special method @emph{must} be set on the class
object itself in order to be consistently invoked by the interpreter).

@geindex coroutine

@node Coroutines,,Special method names,Data model
@anchor{reference/datamodel coroutines}@anchor{1816}
@subsection Coroutines


@menu
* Awaitable Objects:: 
* Coroutine Objects:: 
* Asynchronous Iterators:: 
* Asynchronous Context Managers:: 

@end menu

@node Awaitable Objects,Coroutine Objects,,Coroutines
@anchor{reference/datamodel awaitable-objects}@anchor{1817}
@subsubsection Awaitable Objects


An @ref{7bd,,awaitable} object generally implements an @ref{1818,,__await__()} method.
@ref{3cf,,Coroutine objects} returned from @ref{1c0,,async def} functions
are awaitable.

@cartouche
@quotation Note 
The @ref{715,,generator iterator} objects returned from generators
decorated with @ref{9fa,,types.coroutine()}
are also awaitable, but they do not implement @ref{1818,,__await__()}.
@end quotation
@end cartouche

@geindex __await__() (object method)
@anchor{reference/datamodel object __await__}@anchor{1818}
@deffn {Method} object.__await__ (self)

Must return an @ref{11c3,,iterator}.  Should be used to implement
@ref{7bd,,awaitable} objects.  For instance, @ref{316,,asyncio.Future} implements
this method to be compatible with the @ref{4d5,,await} expression.
@end deffn


New in version 3.5.



@subsubheading See also


@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} for additional information about awaitable objects.


@node Coroutine Objects,Asynchronous Iterators,Awaitable Objects,Coroutines
@anchor{reference/datamodel coroutine-objects}@anchor{1792}@anchor{reference/datamodel id7}@anchor{1819}
@subsubsection Coroutine Objects


@ref{3cf,,Coroutine objects} are @ref{7bd,,awaitable} objects.
A coroutine’s execution can be controlled by calling @ref{1818,,__await__()} and
iterating over the result.  When the coroutine has finished executing and
returns, the iterator raises @ref{736,,StopIteration}, and the exception’s
@code{value} attribute holds the return value.  If the
coroutine raises an exception, it is propagated by the iterator.  Coroutines
should not directly raise unhandled @ref{736,,StopIteration} exceptions.

Coroutines also have the methods listed below, which are analogous to
those of generators (see @ref{181a,,Generator-iterator methods}).  However, unlike
generators, coroutines do not directly support iteration.


Changed in version 3.5.2: It is a @ref{5af,,RuntimeError} to await on a coroutine more than once.


@geindex send() (coroutine method)
@anchor{reference/datamodel coroutine send}@anchor{181b}
@deffn {Method} coroutine.send (value)

Starts or resumes execution of the coroutine.  If @emph{value} is @code{None},
this is equivalent to advancing the iterator returned by
@ref{1818,,__await__()}.  If @emph{value} is not @code{None}, this method delegates
to the @ref{181c,,send()} method of the iterator that caused
the coroutine to suspend.  The result (return value,
@ref{736,,StopIteration}, or other exception) is the same as when
iterating over the @code{__await__()} return value, described above.
@end deffn

@geindex throw() (coroutine method)
@anchor{reference/datamodel coroutine throw}@anchor{181d}
@deffn {Method} coroutine.throw (type[, value[, traceback]])

Raises the specified exception in the coroutine.  This method delegates
to the @ref{11be,,throw()} method of the iterator that caused
the coroutine to suspend, if it has such a method.  Otherwise,
the exception is raised at the suspension point.  The result
(return value, @ref{736,,StopIteration}, or other exception) is the same as
when iterating over the @ref{1818,,__await__()} return value, described
above.  If the exception is not caught in the coroutine, it propagates
back to the caller.
@end deffn

@geindex close() (coroutine method)
@anchor{reference/datamodel coroutine close}@anchor{181e}
@deffn {Method} coroutine.close ()

Causes the coroutine to clean itself up and exit.  If the coroutine
is suspended, this method first delegates to the @ref{181f,,close()}
method of the iterator that caused the coroutine to suspend, if it
has such a method.  Then it raises @ref{f2e,,GeneratorExit} at the
suspension point, causing the coroutine to immediately clean itself up.
Finally, the coroutine is marked as having finished executing, even if
it was never started.

Coroutine objects are automatically closed using the above process when
they are about to be destroyed.
@end deffn

@node Asynchronous Iterators,Asynchronous Context Managers,Coroutine Objects,Coroutines
@anchor{reference/datamodel async-iterators}@anchor{8d3}@anchor{reference/datamodel asynchronous-iterators}@anchor{1820}
@subsubsection Asynchronous Iterators


An @emph{asynchronous iterator} can call asynchronous code in
its @code{__anext__} method.

Asynchronous iterators can be used in an @ref{5cd,,async for} statement.

@geindex __aiter__() (object method)
@anchor{reference/datamodel object __aiter__}@anchor{737}
@deffn {Method} object.__aiter__ (self)

Must return an @emph{asynchronous iterator} object.
@end deffn

@geindex __anext__() (object method)
@anchor{reference/datamodel object __anext__}@anchor{1793}
@deffn {Method} object.__anext__ (self)

Must return an @emph{awaitable} resulting in a next value of the iterator.  Should
raise a @ref{12da,,StopAsyncIteration} error when the iteration is over.
@end deffn

An example of an asynchronous iterable object:

@example
class Reader:
    async def readline(self):
        ...

    def __aiter__(self):
        return self

    async def __anext__(self):
        val = await self.readline()
        if val == b'@w{'}:
            raise StopAsyncIteration
        return val
@end example


New in version 3.5.



Changed in version 3.7: Prior to Python 3.7, @ref{737,,__aiter__()} could return an @emph{awaitable}
that would resolve to an
@ref{8d2,,asynchronous iterator}.

Starting with Python 3.7, @ref{737,,__aiter__()} must return an
asynchronous iterator object.  Returning anything else
will result in a @ref{162,,TypeError} error.


@node Asynchronous Context Managers,,Asynchronous Iterators,Coroutines
@anchor{reference/datamodel async-context-managers}@anchor{1821}@anchor{reference/datamodel asynchronous-context-managers}@anchor{1822}
@subsubsection Asynchronous Context Managers


An @emph{asynchronous context manager} is a @emph{context manager} that is able to
suspend execution in its @code{__aenter__} and @code{__aexit__} methods.

Asynchronous context managers can be used in an @ref{169,,async with} statement.

@geindex __aenter__() (object method)
@anchor{reference/datamodel object __aenter__}@anchor{1823}
@deffn {Method} object.__aenter__ (self)

Semantically similar to @ref{1811,,__enter__()}, the only
difference being that it must return an @emph{awaitable}.
@end deffn

@geindex __aexit__() (object method)
@anchor{reference/datamodel object __aexit__}@anchor{1824}
@deffn {Method} object.__aexit__ (self, exc_type, exc_value, traceback)

Semantically similar to @ref{1812,,__exit__()}, the only
difference being that it must return an @emph{awaitable}.
@end deffn

An example of an asynchronous context manager class:

@example
class AsyncContextManager:
    async def __aenter__(self):
        await log('entering context')

    async def __aexit__(self, exc_type, exc, tb):
        await log('exiting context')
@end example


New in version 3.5.


@node Execution model,The import system,Data model,The Python Language Reference
@anchor{reference/executionmodel doc}@anchor{1825}@anchor{reference/executionmodel execmodel}@anchor{1826}@anchor{reference/executionmodel execution-model}@anchor{1827}
@section Execution model


@geindex execution model
@geindex code; block

@menu
* Structure of a program:: 
* Naming and binding:: 
* Exceptions: Exceptions<2>. 

@end menu

@node Structure of a program,Naming and binding,,Execution model
@anchor{reference/executionmodel prog-structure}@anchor{1828}@anchor{reference/executionmodel structure-of-a-program}@anchor{1829}
@subsection Structure of a program


@geindex block

A Python program is constructed from code blocks.
A @emph{block} is a piece of Python program text that is executed as a unit.
The following are blocks: a module, a function body, and a class definition.
Each command typed interactively is a block.  A script file (a file given as
standard input to the interpreter or specified as a command line argument to the
interpreter) is a code block.  A script command (a command specified on the
interpreter command line with the @ref{1180,,-c} option) is a code block.
A module run as a top level script (as module @code{__main__}) from the command
line using a @ref{611,,-m} argument is also a code block. The string
argument passed to the built-in functions @ref{34a,,eval()} and @ref{34b,,exec()} is a
code block.

@geindex execution; frame

A code block is executed in an @emph{execution frame}.  A frame contains some
administrative information (used for debugging) and determines where and how
execution continues after the code block’s execution has completed.

@node Naming and binding,Exceptions<2>,Structure of a program,Execution model
@anchor{reference/executionmodel naming}@anchor{182a}@anchor{reference/executionmodel naming-and-binding}@anchor{182b}
@subsection Naming and binding


@geindex namespace
@geindex scope

@menu
* Binding of names:: 
* Resolution of names:: 
* Builtins and restricted execution:: 
* Interaction with dynamic features:: 

@end menu

@node Binding of names,Resolution of names,,Naming and binding
@anchor{reference/executionmodel bind-names}@anchor{182c}@anchor{reference/executionmodel binding-of-names}@anchor{182d}
@subsubsection Binding of names


@geindex name
@geindex binding; name

@emph{Names} refer to objects.  Names are introduced by name binding operations.

@geindex from; import statement

The following constructs bind names:


@itemize *

@item 
formal parameters to functions,

@item 
class definitions,

@item 
function definitions,

@item 
assignment expressions,

@item 
@ref{182e,,targets} that are identifiers if occurring in
an assignment:


@itemize +

@item 
@ref{161,,for} loop header,

@item 
after @code{as} in a @ref{168,,with} statement, @ref{16e,,except}
clause or in the as-pattern in structural pattern matching,

@item 
in a capture pattern in structural pattern matching
@end itemize

@item 
@ref{e35,,import} statements.
@end itemize

The @code{import} statement of the form @code{from ... import *} binds all
names defined in the imported module, except those beginning with an underscore.
This form may only be used at the module level.

A target occurring in a @ref{152e,,del} statement is also considered bound for
this purpose (though the actual semantics are to unbind the name).

Each assignment or import statement occurs within a block defined by a class or
function definition or at the module level (the top-level code block).

@geindex free; variable

If a name is bound in a block, it is a local variable of that block, unless
declared as @ref{e4d,,nonlocal} or @ref{1504,,global}.  If a name is bound at
the module level, it is a global variable.  (The variables of the module code
block are local and global.)  If a variable is used in a code block but not
defined there, it is a @emph{free variable}.

Each occurrence of a name in the program text refers to the @emph{binding} of
that name established by the following name resolution rules.

@node Resolution of names,Builtins and restricted execution,Binding of names,Naming and binding
@anchor{reference/executionmodel resolution-of-names}@anchor{182f}@anchor{reference/executionmodel resolve-names}@anchor{1830}
@subsubsection Resolution of names


@geindex scope

A @emph{scope} defines the visibility of a name within a block.  If a local
variable is defined in a block, its scope includes that block.  If the
definition occurs in a function block, the scope extends to any blocks contained
within the defining one, unless a contained block introduces a different binding
for the name.

@geindex environment

When a name is used in a code block, it is resolved using the nearest enclosing
scope.  The set of all such scopes visible to a code block is called the block’s
@emph{environment}.

@geindex NameError (built-in exception)
@geindex UnboundLocalError

When a name is not found at all, a @ref{21f,,NameError} exception is raised.
If the current scope is a function scope, and the name refers to a local
variable that has not yet been bound to a value at the point where the name is
used, an @ref{1067,,UnboundLocalError} exception is raised.
@ref{1067,,UnboundLocalError} is a subclass of @ref{21f,,NameError}.

If a name binding operation occurs anywhere within a code block, all uses of the
name within the block are treated as references to the current block.  This can
lead to errors when a name is used within a block before it is bound.  This rule
is subtle.  Python lacks declarations and allows name binding operations to
occur anywhere within a code block.  The local variables of a code block can be
determined by scanning the entire text of the block for name binding operations.

If the @ref{1504,,global} statement occurs within a block, all uses of the names
specified in the statement refer to the bindings of those names in the top-level
namespace.  Names are resolved in the top-level namespace by searching the
global namespace, i.e. the namespace of the module containing the code block,
and the builtins namespace, the namespace of the module @ref{11,,builtins}.  The
global namespace is searched first.  If the names are not found there, the
builtins namespace is searched.  The @code{global} statement must precede
all uses of the listed names.

The @ref{1504,,global} statement has the same scope as a name binding operation
in the same block.  If the nearest enclosing scope for a free variable contains
a global statement, the free variable is treated as a global.

@c XXX say more about "nonlocal" semantics here

The @ref{e4d,,nonlocal} statement causes corresponding names to refer
to previously bound variables in the nearest enclosing function scope.
@ref{21a,,SyntaxError} is raised at compile time if the given name does not
exist in any enclosing function scope.

@geindex module; __main__

The namespace for a module is automatically created the first time a module is
imported.  The main module for a script is always called @ref{1,,__main__}.

Class definition blocks and arguments to @ref{34b,,exec()} and @ref{34a,,eval()} are
special in the context of name resolution.
A class definition is an executable statement that may use and define names.
These references follow the normal rules for name resolution with an exception
that unbound local variables are looked up in the global namespace.
The namespace of the class definition becomes the attribute dictionary of
the class. The scope of names defined in a class block is limited to the
class block; it does not extend to the code blocks of methods – this includes
comprehensions and generator expressions since they are implemented using a
function scope.  This means that the following will fail:

@example
class A:
    a = 42
    b = list(a + i for i in range(10))
@end example

@node Builtins and restricted execution,Interaction with dynamic features,Resolution of names,Naming and binding
@anchor{reference/executionmodel builtins-and-restricted-execution}@anchor{1831}@anchor{reference/executionmodel restrict-exec}@anchor{1832}
@subsubsection Builtins and restricted execution


@geindex restricted; execution

@strong{CPython implementation detail:} Users should not touch @code{__builtins__}; it is strictly an implementation
detail.  Users wanting to override values in the builtins namespace should
@ref{e35,,import} the @ref{11,,builtins} module and modify its
attributes appropriately.

The builtins namespace associated with the execution of a code block
is actually found by looking up the name @code{__builtins__} in its
global namespace; this should be a dictionary or a module (in the
latter case the module’s dictionary is used).  By default, when in the
@ref{1,,__main__} module, @code{__builtins__} is the built-in module
@ref{11,,builtins}; when in any other module, @code{__builtins__} is an
alias for the dictionary of the @ref{11,,builtins} module itself.

@node Interaction with dynamic features,,Builtins and restricted execution,Naming and binding
@anchor{reference/executionmodel dynamic-features}@anchor{1833}@anchor{reference/executionmodel interaction-with-dynamic-features}@anchor{1834}
@subsubsection Interaction with dynamic features


Name resolution of free variables occurs at runtime, not at compile time.
This means that the following code will print 42:

@example
i = 10
def f():
    print(i)
i = 42
f()
@end example

@c XXX from * also invalid with relative imports (at least currently)

The @ref{34a,,eval()} and @ref{34b,,exec()} functions do not have access to the full
environment for resolving names.  Names may be resolved in the local and global
namespaces of the caller.  Free variables are not resolved in the nearest
enclosing namespace, but in the global namespace.  @footnote{@w{(1)} 
This limitation occurs because the code that is executed by these operations
is not available at the time the module is compiled.
} The @ref{34b,,exec()} and
@ref{34a,,eval()} functions have optional arguments to override the global and local
namespace.  If only one namespace is specified, it is used for both.

@node Exceptions<2>,,Naming and binding,Execution model
@anchor{reference/executionmodel exceptions}@anchor{1835}@anchor{reference/executionmodel id2}@anchor{1836}
@subsection Exceptions


@geindex exception

@geindex raise an exception
@geindex handle an exception
@geindex exception handler
@geindex errors
@geindex error handling

Exceptions are a means of breaking out of the normal flow of control of a code
block in order to handle errors or other exceptional conditions.  An exception
is @emph{raised} at the point where the error is detected; it may be @emph{handled} by the
surrounding code block or by any code block that directly or indirectly invoked
the code block where the error occurred.

The Python interpreter raises an exception when it detects a run-time error
(such as division by zero).  A Python program can also explicitly raise an
exception with the @ref{16d,,raise} statement. Exception handlers are specified
with the @ref{f6d,,try} … @ref{16e,,except} statement.  The @ref{4c1,,finally}
clause of such a statement can be used to specify cleanup code which does not
handle the exception, but is executed whether an exception occurred or not in
the preceding code.

@geindex termination model

Python uses the “termination” model of error handling: an exception handler can
find out what happened and continue execution at an outer level, but it cannot
repair the cause of the error and retry the failing operation (except by
re-entering the offending piece of code from the top).

@geindex SystemExit (built-in exception)

When an exception is not handled at all, the interpreter terminates execution of
the program, or returns to its interactive main loop.  In either case, it prints
a stack traceback, except when the exception is @ref{892,,SystemExit}.

Exceptions are identified by class instances.  The @ref{16e,,except} clause is
selected depending on the class of the instance: it must reference the class of
the instance or a base class thereof.  The instance can be received by the
handler and can carry additional information about the exceptional condition.

@cartouche
@quotation Note 
Exception messages are not part of the Python API.  Their contents may change
from one version of Python to the next without warning and should not be
relied on by code which will run under multiple versions of the interpreter.
@end quotation
@end cartouche

See also the description of the @ref{f6d,,try} statement in section @ref{f6d,,The try statement}
and @ref{16d,,raise} statement in section @ref{16d,,The raise statement}.

@node The import system,Expressions,Execution model,The Python Language Reference
@anchor{reference/import doc}@anchor{1837}@anchor{reference/import importsystem}@anchor{1796}@anchor{reference/import the-import-system}@anchor{1838}
@section The import system


@geindex import machinery

Python code in one @ref{1839,,module} gains access to the code in another module
by the process of @ref{183a,,importing} it.  The @ref{e35,,import} statement is
the most common way of invoking the import machinery, but it is not the only
way.  Functions such as @ref{d3e,,importlib.import_module()} and built-in
@ref{3b7,,__import__()} can also be used to invoke the import machinery.

The @ref{e35,,import} statement combines two operations; it searches for the
named module, then it binds the results of that search to a name in the local
scope.  The search operation of the @code{import} statement is defined as
a call to the @ref{3b7,,__import__()} function, with the appropriate arguments.
The return value of @ref{3b7,,__import__()} is used to perform the name
binding operation of the @code{import} statement.  See the
@code{import} statement for the exact details of that name binding
operation.

A direct call to @ref{3b7,,__import__()} performs only the module search and, if
found, the module creation operation.  While certain side-effects may occur,
such as the importing of parent packages, and the updating of various caches
(including @ref{11f7,,sys.modules}), only the @ref{e35,,import} statement performs
a name binding operation.

When an @ref{e35,,import} statement is executed, the standard builtin
@ref{3b7,,__import__()} function is called. Other mechanisms for invoking the
import system (such as @ref{d3e,,importlib.import_module()}) may choose to bypass
@ref{3b7,,__import__()} and use their own solutions to implement import semantics.

When a module is first imported, Python searches for the module and if found,
it creates a module object @footnote{@w{(1)} 
See @ref{973,,types.ModuleType}.
}, initializing it.  If the named module
cannot be found, a @ref{66b,,ModuleNotFoundError} is raised.  Python implements various
strategies to search for the named module when the import machinery is
invoked.  These strategies can be modified and extended by using various hooks
described in the sections below.


Changed in version 3.3: The import system has been updated to fully implement the second phase
of 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302}. There is no longer any implicit import machinery - the full
import system is exposed through @ref{87c,,sys.meta_path}. In addition,
native namespace package support has been implemented (see 
@geindex Python Enhancement Proposals; PEP 420
PEP 420@footnote{https://www.python.org/dev/peps/pep-0420}).


@menu
* importlib: importlib<8>. 
* Packages: Packages<2>. 
* Searching:: 
* Loading:: 
* The Path Based Finder:: 
* Replacing the standard import system:: 
* Package Relative Imports:: 
* Special considerations for __main__:: 
* Open issues:: 
* References:: 

@end menu

@node importlib<8>,Packages<2>,,The import system
@anchor{reference/import importlib}@anchor{183b}
@subsection @code{importlib}


The @ref{96,,importlib} module provides a rich API for interacting with the
import system.  For example @ref{d3e,,importlib.import_module()} provides a
recommended, simpler API than built-in @ref{3b7,,__import__()} for invoking the
import machinery.  Refer to the @ref{96,,importlib} library documentation for
additional detail.

@node Packages<2>,Searching,importlib<8>,The import system
@anchor{reference/import packages}@anchor{183c}
@subsection Packages


@geindex package

Python has only one type of module object, and all modules are of this type,
regardless of whether the module is implemented in Python, C, or something
else.  To help organize modules and provide a naming hierarchy, Python has a
concept of @ref{183d,,packages}.

You can think of packages as the directories on a file system and modules as
files within directories, but don’t take this analogy too literally since
packages and modules need not originate from the file system.  For the
purposes of this documentation, we’ll use this convenient analogy of
directories and files.  Like file system directories, packages are organized
hierarchically, and packages may themselves contain subpackages, as well as
regular modules.

It’s important to keep in mind that all packages are modules, but not all
modules are packages.  Or put another way, packages are just a special kind of
module.  Specifically, any module that contains a @code{__path__} attribute is
considered a package.

All modules have a name.  Subpackage names are separated from their parent
package name by a dot, akin to Python’s standard attribute access syntax.  Thus
you might have a module called @ref{f7,,sys} and a package called @ref{64,,email},
which in turn has a subpackage called @ref{6e,,email.mime} and a module within
that subpackage called @code{email.mime.text}.

@menu
* Regular packages:: 
* Namespace packages:: 

@end menu

@node Regular packages,Namespace packages,,Packages<2>
@anchor{reference/import regular-packages}@anchor{183e}
@subsubsection Regular packages


@geindex package; regular

Python defines two types of packages, @ref{183f,,regular packages} and @ref{1840,,namespace packages}.  Regular
packages are traditional packages as they existed in Python 3.2 and earlier.
A regular package is typically implemented as a directory containing an
@code{__init__.py} file.  When a regular package is imported, this
@code{__init__.py} file is implicitly executed, and the objects it defines are
bound to names in the package’s namespace.  The @code{__init__.py} file can
contain the same Python code that any other module can contain, and Python
will add some additional attributes to the module when it is imported.

For example, the following file system layout defines a top level @code{parent}
package with three subpackages:

@example
parent/
    __init__.py
    one/
        __init__.py
    two/
        __init__.py
    three/
        __init__.py
@end example

Importing @code{parent.one} will implicitly execute @code{parent/__init__.py} and
@code{parent/one/__init__.py}.  Subsequent imports of @code{parent.two} or
@code{parent.three} will execute @code{parent/two/__init__.py} and
@code{parent/three/__init__.py} respectively.

@node Namespace packages,,Regular packages,Packages<2>
@anchor{reference/import namespace-packages}@anchor{1841}
@subsubsection Namespace packages


@geindex package; namespace
@geindex package; portion

A namespace package is a composite of various @ref{1842,,portions},
where each portion contributes a subpackage to the parent package.  Portions
may reside in different locations on the file system.  Portions may also be
found in zip files, on the network, or anywhere else that Python searches
during import.  Namespace packages may or may not correspond directly to
objects on the file system; they may be virtual modules that have no concrete
representation.

Namespace packages do not use an ordinary list for their @code{__path__}
attribute. They instead use a custom iterable type which will automatically
perform a new search for package portions on the next import attempt within
that package if the path of their parent package (or @ref{489,,sys.path} for a
top level package) changes.

With namespace packages, there is no @code{parent/__init__.py} file.  In fact,
there may be multiple @code{parent} directories found during import search, where
each one is provided by a different portion.  Thus @code{parent/one} may not be
physically located next to @code{parent/two}.  In this case, Python will create a
namespace package for the top-level @code{parent} package whenever it or one of
its subpackages is imported.

See also 
@geindex Python Enhancement Proposals; PEP 420
PEP 420@footnote{https://www.python.org/dev/peps/pep-0420} for the namespace package specification.

@node Searching,Loading,Packages<2>,The import system
@anchor{reference/import searching}@anchor{1843}
@subsection Searching


To begin the search, Python needs the @ref{11bf,,fully qualified}
name of the module (or package, but for the purposes of this discussion, the
difference is immaterial) being imported.  This name may come from various
arguments to the @ref{e35,,import} statement, or from the parameters to the
@ref{d3e,,importlib.import_module()} or @ref{3b7,,__import__()} functions.

This name will be used in various phases of the import search, and it may be
the dotted path to a submodule, e.g. @code{foo.bar.baz}.  In this case, Python
first tries to import @code{foo}, then @code{foo.bar}, and finally @code{foo.bar.baz}.
If any of the intermediate imports fail, a @ref{66b,,ModuleNotFoundError} is raised.

@menu
* The module cache:: 
* Finders and loaders:: 
* Import hooks:: 
* The meta path:: 

@end menu

@node The module cache,Finders and loaders,,Searching
@anchor{reference/import the-module-cache}@anchor{1844}
@subsubsection The module cache


@geindex sys.modules

The first place checked during import search is @ref{11f7,,sys.modules}.  This
mapping serves as a cache of all modules that have been previously imported,
including the intermediate paths.  So if @code{foo.bar.baz} was previously
imported, @ref{11f7,,sys.modules} will contain entries for @code{foo}, @code{foo.bar},
and @code{foo.bar.baz}.  Each key will have as its value the corresponding module
object.

During import, the module name is looked up in @ref{11f7,,sys.modules} and if
present, the associated value is the module satisfying the import, and the
process completes.  However, if the value is @code{None}, then a
@ref{66b,,ModuleNotFoundError} is raised.  If the module name is missing, Python will
continue searching for the module.

@ref{11f7,,sys.modules} is writable.  Deleting a key may not destroy the
associated module (as other modules may hold references to it),
but it will invalidate the cache entry for the named module, causing
Python to search anew for the named module upon its next
import. The key can also be assigned to @code{None}, forcing the next import
of the module to result in a @ref{66b,,ModuleNotFoundError}.

Beware though, as if you keep a reference to the module object,
invalidate its cache entry in @ref{11f7,,sys.modules}, and then re-import the
named module, the two module objects will @emph{not} be the same. By contrast,
@ref{66a,,importlib.reload()} will reuse the @emph{same} module object, and simply
reinitialise the module contents by rerunning the module’s code.

@node Finders and loaders,Import hooks,The module cache,Searching
@anchor{reference/import finders-and-loaders}@anchor{1845}@anchor{reference/import id2}@anchor{1846}
@subsubsection Finders and loaders


@geindex finder
@geindex loader
@geindex module spec

If the named module is not found in @ref{11f7,,sys.modules}, then Python’s import
protocol is invoked to find and load the module.  This protocol consists of
two conceptual objects, @ref{1847,,finders} and @ref{1848,,loaders}.
A finder’s job is to determine whether it can find the named module using
whatever strategy it knows about. Objects that implement both of these
interfaces are referred to as @ref{1849,,importers} - they return
themselves when they find that they can load the requested module.

Python includes a number of default finders and importers.  The first one
knows how to locate built-in modules, and the second knows how to locate
frozen modules.  A third default finder searches an @ref{184a,,import path}
for modules.  The @ref{184a,,import path} is a list of locations that may
name file system paths or zip files.  It can also be extended to search
for any locatable resource, such as those identified by URLs.

The import machinery is extensible, so new finders can be added to extend the
range and scope of module searching.

Finders do not actually load modules.  If they can find the named module, they
return a @emph{module spec}, an encapsulation of the module’s import-related
information, which the import machinery then uses when loading the module.

The following sections describe the protocol for finders and loaders in more
detail, including how you can create and register new ones to extend the
import machinery.


Changed in version 3.4: In previous versions of Python, finders returned @ref{1848,,loaders}
directly, whereas now they return module specs which @emph{contain} loaders.
Loaders are still used during import but have fewer responsibilities.


@node Import hooks,The meta path,Finders and loaders,Searching
@anchor{reference/import import-hooks}@anchor{184b}
@subsubsection Import hooks


@geindex import hooks
@geindex meta hooks
@geindex path hooks
@geindex hooks; import
@geindex hooks; meta
@geindex hooks; path

The import machinery is designed to be extensible; the primary mechanism for
this are the @emph{import hooks}.  There are two types of import hooks: @emph{meta
hooks} and @emph{import path hooks}.

Meta hooks are called at the start of import processing, before any other
import processing has occurred, other than @ref{11f7,,sys.modules} cache look up.
This allows meta hooks to override @ref{489,,sys.path} processing, frozen
modules, or even built-in modules.  Meta hooks are registered by adding new
finder objects to @ref{87c,,sys.meta_path}, as described below.

Import path hooks are called as part of @ref{489,,sys.path} (or
@code{package.__path__}) processing, at the point where their associated path
item is encountered.  Import path hooks are registered by adding new callables
to @ref{bb5,,sys.path_hooks} as described below.

@node The meta path,,Import hooks,Searching
@anchor{reference/import the-meta-path}@anchor{184c}
@subsubsection The meta path


@geindex sys.meta_path
@geindex finder; find_spec

When the named module is not found in @ref{11f7,,sys.modules}, Python next
searches @ref{87c,,sys.meta_path}, which contains a list of meta path finder
objects.  These finders are queried in order to see if they know how to handle
the named module.  Meta path finders must implement a method called
@ref{308,,find_spec()} which takes three arguments:
a name, an import path, and (optionally) a target module.  The meta path
finder can use any strategy it wants to determine whether it can handle
the named module or not.

If the meta path finder knows how to handle the named module, it returns a
spec object.  If it cannot handle the named module, it returns @code{None}.  If
@ref{87c,,sys.meta_path} processing reaches the end of its list without returning
a spec, then a @ref{66b,,ModuleNotFoundError} is raised.  Any other exceptions
raised are simply propagated up, aborting the import process.

The @ref{308,,find_spec()} method of meta path
finders is called with two or three arguments.  The first is the fully
qualified name of the module being imported, for example @code{foo.bar.baz}.
The second argument is the path entries to use for the module search.  For
top-level modules, the second argument is @code{None}, but for submodules or
subpackages, the second argument is the value of the parent package’s
@code{__path__} attribute. If the appropriate @code{__path__} attribute cannot
be accessed, a @ref{66b,,ModuleNotFoundError} is raised.  The third argument
is an existing module object that will be the target of loading later.
The import system passes in a target module only during reload.

The meta path may be traversed multiple times for a single import request.
For example, assuming none of the modules involved has already been cached,
importing @code{foo.bar.baz} will first perform a top level import, calling
@code{mpf.find_spec("foo", None, None)} on each meta path finder (@code{mpf}). After
@code{foo} has been imported, @code{foo.bar} will be imported by traversing the
meta path a second time, calling
@code{mpf.find_spec("foo.bar", foo.__path__, None)}. Once @code{foo.bar} has been
imported, the final traversal will call
@code{mpf.find_spec("foo.bar.baz", foo.bar.__path__, None)}.

Some meta path finders only support top level imports. These importers will
always return @code{None} when anything other than @code{None} is passed as the
second argument.

Python’s default @ref{87c,,sys.meta_path} has three meta path finders, one that
knows how to import built-in modules, one that knows how to import frozen
modules, and one that knows how to import modules from an @ref{184a,,import path}
(i.e. the @ref{184d,,path based finder}).


Changed in version 3.4: The @ref{308,,find_spec()} method of meta path
finders replaced @ref{306,,find_module()}, which
is now deprecated.  While it will continue to work without change, the
import machinery will try it only if the finder does not implement
@code{find_spec()}.



Changed in version 3.10: Use of @ref{306,,find_module()} by the import system
now raises @ref{304,,ImportWarning}.


@node Loading,The Path Based Finder,Searching,The import system
@anchor{reference/import loading}@anchor{184e}
@subsection Loading


If and when a module spec is found, the import machinery will use it (and
the loader it contains) when loading the module.  Here is an approximation
of what happens during the loading portion of import:

@example
module = None
if spec.loader is not None and hasattr(spec.loader, 'create_module'):
    # It is assumed 'exec_module' will also be defined on the loader.
    module = spec.loader.create_module(spec)
if module is None:
    module = ModuleType(spec.name)
# The import-related module attributes get set here:
_init_module_attrs(spec, module)

if spec.loader is None:
    # unsupported
    raise ImportError
if spec.origin is None and spec.submodule_search_locations is not None:
    # namespace package
    sys.modules[spec.name] = module
elif not hasattr(spec.loader, 'exec_module'):
    module = spec.loader.load_module(spec.name)
    # Set __loader__ and __package__ if missing.
else:
    sys.modules[spec.name] = module
    try:
        spec.loader.exec_module(module)
    except BaseException:
        try:
            del sys.modules[spec.name]
        except KeyError:
            pass
        raise
return sys.modules[spec.name]
@end example

Note the following details:

@quotation


@itemize *

@item 
If there is an existing module object with the given name in
@ref{11f7,,sys.modules}, import will have already returned it.

@item 
The module will exist in @ref{11f7,,sys.modules} before the loader
executes the module code.  This is crucial because the module code may
(directly or indirectly) import itself; adding it to @ref{11f7,,sys.modules}
beforehand prevents unbounded recursion in the worst case and multiple
loading in the best.

@item 
If loading fails, the failing module – and only the failing module –
gets removed from @ref{11f7,,sys.modules}.  Any module already in the
@ref{11f7,,sys.modules} cache, and any module that was successfully loaded
as a side-effect, must remain in the cache.  This contrasts with
reloading where even the failing module is left in @ref{11f7,,sys.modules}.

@item 
After the module is created but before execution, the import machinery
sets the import-related module attributes (“_init_module_attrs” in
the pseudo-code example above), as summarized in a
@ref{184f,,later section}.

@item 
Module execution is the key moment of loading in which the module’s
namespace gets populated.  Execution is entirely delegated to the
loader, which gets to decide what gets populated and how.

@item 
The module created during loading and passed to exec_module() may
not be the one returned at the end of import @footnote{@w{(2)} 
The importlib implementation avoids using the return value
directly. Instead, it gets the module object by looking the module name up
in @ref{11f7,,sys.modules}.  The indirect effect of this is that an imported
module may replace itself in @ref{11f7,,sys.modules}.  This is
implementation-specific behavior that is not guaranteed to work in other
Python implementations.
}.
@end itemize
@end quotation


Changed in version 3.4: The import system has taken over the boilerplate responsibilities of
loaders.  These were previously performed by the
@ref{301,,importlib.abc.Loader.load_module()} method.


@menu
* Loaders:: 
* Submodules:: 
* Module spec:: 
* Import-related module attributes:: 
* module.__path__: module __path__. 
* Module reprs:: 
* Cached bytecode invalidation:: 

@end menu

@node Loaders,Submodules,,Loading
@anchor{reference/import loaders}@anchor{1850}
@subsubsection Loaders


Module loaders provide the critical function of loading: module execution.
The import machinery calls the @ref{302,,importlib.abc.Loader.exec_module()}
method with a single argument, the module object to execute.  Any value
returned from @ref{302,,exec_module()} is ignored.

Loaders must satisfy the following requirements:

@quotation


@itemize *

@item 
If the module is a Python module (as opposed to a built-in module or a
dynamically loaded extension), the loader should execute the module’s code
in the module’s global name space (@code{module.__dict__}).

@item 
If the loader cannot execute the module, it should raise an
@ref{3b8,,ImportError}, although any other exception raised during
@ref{302,,exec_module()} will be propagated.
@end itemize
@end quotation

In many cases, the finder and loader can be the same object; in such cases the
@ref{308,,find_spec()} method would just return a
spec with the loader set to @code{self}.

Module loaders may opt in to creating the module object during loading
by implementing a @ref{7f6,,create_module()} method.
It takes one argument, the module spec, and returns the new module object
to use during loading.  @code{create_module()} does not need to set any attributes
on the module object.  If the method returns @code{None}, the
import machinery will create the new module itself.


New in version 3.4: The @ref{7f6,,create_module()} method of loaders.



Changed in version 3.4: The @ref{301,,load_module()} method was replaced by
@ref{302,,exec_module()} and the import
machinery assumed all the boilerplate responsibilities of loading.

For compatibility with existing loaders, the import machinery will use
the @code{load_module()} method of loaders if it exists and the loader does
not also implement @code{exec_module()}.  However, @code{load_module()} has been
deprecated and loaders should implement @code{exec_module()} instead.

The @code{load_module()} method must implement all the boilerplate loading
functionality described above in addition to executing the module.  All
the same constraints apply, with some additional clarification:

@quotation


@itemize *

@item 
If there is an existing module object with the given name in
@ref{11f7,,sys.modules}, the loader must use that existing module.
(Otherwise, @ref{66a,,importlib.reload()} will not work correctly.)  If the
named module does not exist in @ref{11f7,,sys.modules}, the loader
must create a new module object and add it to @ref{11f7,,sys.modules}.

@item 
The module @emph{must} exist in @ref{11f7,,sys.modules} before the loader
executes the module code, to prevent unbounded recursion or multiple
loading.

@item 
If loading fails, the loader must remove any modules it has inserted
into @ref{11f7,,sys.modules}, but it must remove @strong{only} the failing
module(s), and only if the loader itself has loaded the module(s)
explicitly.
@end itemize
@end quotation



Changed in version 3.5: A @ref{1b2,,DeprecationWarning} is raised when @code{exec_module()} is defined but
@code{create_module()} is not.



Changed in version 3.6: An @ref{3b8,,ImportError} is raised when @code{exec_module()} is defined but
@code{create_module()} is not.



Changed in version 3.10: Use of @code{load_module()} will raise @ref{304,,ImportWarning}.


@node Submodules,Module spec,Loaders,Loading
@anchor{reference/import submodules}@anchor{1851}
@subsubsection Submodules


When a submodule is loaded using any mechanism (e.g. @code{importlib} APIs, the
@code{import} or @code{import-from} statements, or built-in @code{__import__()}) a
binding is placed in the parent module’s namespace to the submodule object.
For example, if package @code{spam} has a submodule @code{foo}, after importing
@code{spam.foo}, @code{spam} will have an attribute @code{foo} which is bound to the
submodule.  Let’s say you have the following directory structure:

@example
spam/
    __init__.py
    foo.py
    bar.py
@end example

and @code{spam/__init__.py} has the following lines in it:

@example
from .foo import Foo
from .bar import Bar
@end example

then executing the following puts a name binding to @code{foo} and @code{bar} in the
@code{spam} module:

@example
>>> import spam
>>> spam.foo
<module 'spam.foo' from '/tmp/imports/spam/foo.py'>
>>> spam.bar
<module 'spam.bar' from '/tmp/imports/spam/bar.py'>
@end example

Given Python’s familiar name binding rules this might seem surprising, but
it’s actually a fundamental feature of the import system.  The invariant
holding is that if you have @code{sys.modules['spam']} and
@code{sys.modules['spam.foo']} (as you would after the above import), the latter
must appear as the @code{foo} attribute of the former.

@node Module spec,Import-related module attributes,Submodules,Loading
@anchor{reference/import module-spec}@anchor{1852}
@subsubsection Module spec


The import machinery uses a variety of information about each module
during import, especially before loading.  Most of the information is
common to all modules.  The purpose of a module’s spec is to encapsulate
this import-related information on a per-module basis.

Using a spec during import allows state to be transferred between import
system components, e.g. between the finder that creates the module spec
and the loader that executes it.  Most importantly, it allows the
import machinery to perform the boilerplate operations of loading,
whereas without a module spec the loader had that responsibility.

The module’s spec is exposed as the @code{__spec__} attribute on a module object.
See @ref{1853,,ModuleSpec} for details on the contents of
the module spec.


New in version 3.4.


@node Import-related module attributes,module __path__,Module spec,Loading
@anchor{reference/import import-mod-attrs}@anchor{184f}@anchor{reference/import import-related-module-attributes}@anchor{1854}
@subsubsection Import-related module attributes


The import machinery fills in these attributes on each module object
during loading, based on the module’s spec, before the loader executes
the module.

@geindex __name__
@anchor{reference/import name__}@anchor{f12}@anchor{reference/import __name__}@anchor{1855}
@deffn {Attribute} __name__

The @code{__name__} attribute must be set to the fully-qualified name of
the module.  This name is used to uniquely identify the module in
the import system.
@end deffn

@geindex __loader__
@anchor{reference/import loader__}@anchor{baf}@anchor{reference/import __loader__}@anchor{1856}
@deffn {Attribute} __loader__

The @code{__loader__} attribute must be set to the loader object that
the import machinery used when loading the module.  This is mostly
for introspection, but can be used for additional loader-specific
functionality, for example getting data associated with a loader.
@end deffn

@geindex __package__
@anchor{reference/import package__}@anchor{bae}@anchor{reference/import __package__}@anchor{1857}
@deffn {Attribute} __package__

The module’s @code{__package__} attribute must be set.  Its value must
be a string, but it can be the same value as its @code{__name__}.  When
the module is a package, its @code{__package__} value should be set to
its @code{__name__}.  When the module is not a package, @code{__package__}
should be set to the empty string for top-level modules, or for
submodules, to the parent package’s name.  See 
@geindex Python Enhancement Proposals; PEP 366
PEP 366@footnote{https://www.python.org/dev/peps/pep-0366} for further
details.

This attribute is used instead of @code{__name__} to calculate explicit
relative imports for main modules, as defined in 
@geindex Python Enhancement Proposals; PEP 366
PEP 366@footnote{https://www.python.org/dev/peps/pep-0366}. It is
expected to have the same value as @code{__spec__.parent}.


Changed in version 3.6: The value of @code{__package__} is expected to be the same as
@code{__spec__.parent}.

@end deffn

@geindex __spec__
@anchor{reference/import spec__}@anchor{1858}@anchor{reference/import __spec__}@anchor{1859}
@deffn {Attribute} __spec__

The @code{__spec__} attribute must be set to the module spec that was
used when importing the module. Setting @code{__spec__}
appropriately applies equally to @ref{185a,,modules initialized during interpreter startup}.  The one exception is @code{__main__},
where @code{__spec__} is @ref{185b,,set to None in some cases}.

When @code{__package__} is not defined, @code{__spec__.parent} is used as
a fallback.


New in version 3.4.



Changed in version 3.6: @code{__spec__.parent} is used as a fallback when @code{__package__} is
not defined.

@end deffn

@geindex __path__
@anchor{reference/import path__}@anchor{1550}@anchor{reference/import __path__}@anchor{185c}
@deffn {Attribute} __path__

If the module is a package (either regular or namespace), the module
object’s @code{__path__} attribute must be set.  The value must be
iterable, but may be empty if @code{__path__} has no further significance.
If @code{__path__} is not empty, it must produce strings when iterated
over. More details on the semantics of @code{__path__} are given
@ref{185d,,below}.

Non-package modules should not have a @code{__path__} attribute.
@end deffn

@geindex __file__
@anchor{reference/import file__}@anchor{1797}@anchor{reference/import __file__}@anchor{185e}
@deffn {Attribute} __file__
@end deffn

@geindex __cached__
@anchor{reference/import cached__}@anchor{d5b}@anchor{reference/import __cached__}@anchor{185f}
@deffn {Attribute} __cached__

@code{__file__} is optional (if set, value must be a string). It indicates
the pathname of the file from which the module was loaded (if
loaded from a file), or the pathname of the shared library file
for extension modules loaded dynamically from a shared library.
It might be missing for certain types of modules, such as C
modules that are statically linked into the interpreter, and the
import system may opt to leave it unset if it has no semantic
meaning (e.g. a module loaded from a database).

If @code{__file__} is set, it may also be appropriate to set the
@code{__cached__} attribute which is the path to any compiled version of
the code (e.g. byte-compiled file). The file does not need to exist
to set this attribute; the path can simply point to where the
compiled file would exist (see 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147}).

It is also appropriate to set @code{__cached__} when @code{__file__} is not
set.  However, that scenario is quite atypical.  Ultimately, the
loader is what makes use of @code{__file__} and/or @code{__cached__}.  So
if a loader can load from a cached module but otherwise does not load
from a file, that atypical scenario may be appropriate.
@end deffn

@node module __path__,Module reprs,Import-related module attributes,Loading
@anchor{reference/import module-path}@anchor{1860}@anchor{reference/import package-path-rules}@anchor{185d}
@subsubsection module.__path__


By definition, if a module has a @code{__path__} attribute, it is a package.

A package’s @code{__path__} attribute is used during imports of its subpackages.
Within the import machinery, it functions much the same as @ref{489,,sys.path},
i.e. providing a list of locations to search for modules during import.
However, @code{__path__} is typically much more constrained than
@ref{489,,sys.path}.

@code{__path__} must be an iterable of strings, but it may be empty.
The same rules used for @ref{489,,sys.path} also apply to a package’s
@code{__path__}, and @ref{bb5,,sys.path_hooks} (described below) are
consulted when traversing a package’s @code{__path__}.

A package’s @code{__init__.py} file may set or alter the package’s @code{__path__}
attribute, and this was typically the way namespace packages were implemented
prior to 
@geindex Python Enhancement Proposals; PEP 420
PEP 420@footnote{https://www.python.org/dev/peps/pep-0420}.  With the adoption of 
@geindex Python Enhancement Proposals; PEP 420
PEP 420@footnote{https://www.python.org/dev/peps/pep-0420}, namespace packages no
longer need to supply @code{__init__.py} files containing only @code{__path__}
manipulation code; the import machinery automatically sets @code{__path__}
correctly for the namespace package.

@node Module reprs,Cached bytecode invalidation,module __path__,Loading
@anchor{reference/import module-reprs}@anchor{1861}
@subsubsection Module reprs


By default, all modules have a usable repr, however depending on the
attributes set above, and in the module’s spec, you can more explicitly
control the repr of module objects.

If the module has a spec (@code{__spec__}), the import machinery will try
to generate a repr from it.  If that fails or there is no spec, the import
system will craft a default repr using whatever information is available
on the module.  It will try to use the @code{module.__name__},
@code{module.__file__}, and @code{module.__loader__} as input into the repr,
with defaults for whatever information is missing.

Here are the exact rules used:

@quotation


@itemize *

@item 
If the module has a @code{__spec__} attribute, the information in the spec
is used to generate the repr.  The “name”, “loader”, “origin”, and
“has_location” attributes are consulted.

@item 
If the module has a @code{__file__} attribute, this is used as part of the
module’s repr.

@item 
If the module has no @code{__file__} but does have a @code{__loader__} that is not
@code{None}, then the loader’s repr is used as part of the module’s repr.

@item 
Otherwise, just use the module’s @code{__name__} in the repr.
@end itemize
@end quotation


Changed in version 3.4: Use of @ref{303,,loader.module_repr()}
has been deprecated and the module spec is now used by the import
machinery to generate a module repr.

For backward compatibility with Python 3.3, the module repr will be
generated by calling the loader’s
@ref{303,,module_repr()} method, if defined, before
trying either approach described above.  However, the method is deprecated.



Changed in version 3.10: Calling @ref{303,,module_repr()} now occurs after trying to
use a module’s @code{__spec__} attribute but before falling back on
@code{__file__}. Use of @ref{303,,module_repr()} is slated to
stop in Python 3.12.


@node Cached bytecode invalidation,,Module reprs,Loading
@anchor{reference/import cached-bytecode-invalidation}@anchor{1862}@anchor{reference/import pyc-invalidation}@anchor{607}
@subsubsection Cached bytecode invalidation


Before Python loads cached bytecode from a @code{.pyc} file, it checks whether the
cache is up-to-date with the source @code{.py} file. By default, Python does this
by storing the source’s last-modified timestamp and size in the cache file when
writing it. At runtime, the import system then validates the cache file by
checking the stored metadata in the cache file against the source’s
metadata.

Python also supports “hash-based” cache files, which store a hash of the source
file’s contents rather than its metadata. There are two variants of hash-based
@code{.pyc} files: checked and unchecked. For checked hash-based @code{.pyc} files,
Python validates the cache file by hashing the source file and comparing the
resulting hash with the hash in the cache file. If a checked hash-based cache
file is found to be invalid, Python regenerates it and writes a new checked
hash-based cache file. For unchecked hash-based @code{.pyc} files, Python simply
assumes the cache file is valid if it exists. Hash-based @code{.pyc} files
validation behavior may be overridden with the @ref{1606,,--check-hash-based-pycs}
flag.


Changed in version 3.7: Added hash-based @code{.pyc} files. Previously, Python only supported
timestamp-based invalidation of bytecode caches.


@node The Path Based Finder,Replacing the standard import system,Loading,The import system
@anchor{reference/import the-path-based-finder}@anchor{1863}
@subsection The Path Based Finder


@geindex path based finder

As mentioned previously, Python comes with several default meta path finders.
One of these, called the @ref{184d,,path based finder}
(@ref{bb4,,PathFinder}), searches an @ref{184a,,import path},
which contains a list of @ref{1864,,path entries}.  Each path
entry names a location to search for modules.

The path based finder itself doesn’t know how to import anything. Instead, it
traverses the individual path entries, associating each of them with a
path entry finder that knows how to handle that particular kind of path.

The default set of path entry finders implement all the semantics for finding
modules on the file system, handling special file types such as Python source
code (@code{.py} files), Python byte code (@code{.pyc} files) and
shared libraries (e.g. @code{.so} files). When supported by the @ref{14a,,zipimport}
module in the standard library, the default path entry finders also handle
loading all of these file types (other than shared libraries) from zipfiles.

Path entries need not be limited to file system locations.  They can refer to
URLs, database queries, or any other location that can be specified as a
string.

The path based finder provides additional hooks and protocols so that you
can extend and customize the types of searchable path entries.  For example,
if you wanted to support path entries as network URLs, you could write a hook
that implements HTTP semantics to find modules on the web.  This hook (a
callable) would return a @ref{c06,,path entry finder} supporting the protocol
described below, which was then used to get a loader for the module from the
web.

A word of warning: this section and the previous both use the term @emph{finder},
distinguishing between them by using the terms @ref{c05,,meta path finder} and
@ref{c06,,path entry finder}.  These two types of finders are very similar,
support similar protocols, and function in similar ways during the import
process, but it’s important to keep in mind that they are subtly different.
In particular, meta path finders operate at the beginning of the import
process, as keyed off the @ref{87c,,sys.meta_path} traversal.

By contrast, path entry finders are in a sense an implementation detail
of the path based finder, and in fact, if the path based finder were to be
removed from @ref{87c,,sys.meta_path}, none of the path entry finder semantics
would be invoked.

@menu
* Path entry finders:: 
* Path entry finder protocol:: 

@end menu

@node Path entry finders,Path entry finder protocol,,The Path Based Finder
@anchor{reference/import path-entry-finders}@anchor{1865}
@subsubsection Path entry finders


@geindex sys.path
@geindex sys.path_hooks
@geindex sys.path_importer_cache
@geindex PYTHONPATH

The @ref{184d,,path based finder} is responsible for finding and loading
Python modules and packages whose location is specified with a string
@ref{1864,,path entry}.  Most path entries name locations in the file system,
but they need not be limited to this.

As a meta path finder, the @ref{184d,,path based finder} implements the
@ref{308,,find_spec()} protocol previously
described, however it exposes additional hooks that can be used to
customize how modules are found and loaded from the @ref{184a,,import path}.

Three variables are used by the @ref{184d,,path based finder}, @ref{489,,sys.path},
@ref{bb5,,sys.path_hooks} and @ref{74c,,sys.path_importer_cache}.  The @code{__path__}
attributes on package objects are also used.  These provide additional ways
that the import machinery can be customized.

@ref{489,,sys.path} contains a list of strings providing search locations for
modules and packages.  It is initialized from the @code{PYTHONPATH}
environment variable and various other installation- and
implementation-specific defaults.  Entries in @ref{489,,sys.path} can name
directories on the file system, zip files, and potentially other “locations”
(see the @ref{e6,,site} module) that should be searched for modules, such as
URLs, or database queries.  Only strings and bytes should be present on
@ref{489,,sys.path}; all other data types are ignored.  The encoding of bytes
entries is determined by the individual @ref{c06,,path entry finders}.

The @ref{184d,,path based finder} is a @ref{c05,,meta path finder}, so the import
machinery begins the @ref{184a,,import path} search by calling the path
based finder’s @ref{b99,,find_spec()} method as
described previously.  When the @code{path} argument to
@ref{b99,,find_spec()} is given, it will be a
list of string paths to traverse - typically a package’s @code{__path__}
attribute for an import within that package.  If the @code{path} argument is
@code{None}, this indicates a top level import and @ref{489,,sys.path} is used.

The path based finder iterates over every entry in the search path, and
for each of these, looks for an appropriate @ref{c06,,path entry finder}
(@ref{30f,,PathEntryFinder}) for the
path entry.  Because this can be an expensive operation (e.g. there may be
@cite{stat()} call overheads for this search), the path based finder maintains
a cache mapping path entries to path entry finders.  This cache is maintained
in @ref{74c,,sys.path_importer_cache} (despite the name, this cache actually
stores finder objects rather than being limited to @ref{1849,,importer} objects).
In this way, the expensive search for a particular @ref{1864,,path entry}
location’s @ref{c06,,path entry finder} need only be done once.  User code is
free to remove cache entries from @ref{74c,,sys.path_importer_cache} forcing
the path based finder to perform the path entry search again @footnote{@w{(3)} 
In legacy code, it is possible to find instances of
@ref{c0c,,imp.NullImporter} in the @ref{74c,,sys.path_importer_cache}.  It
is recommended that code be changed to use @code{None} instead.  See
@ref{d3d,,Porting Python code} for more details.
}.

If the path entry is not present in the cache, the path based finder iterates
over every callable in @ref{bb5,,sys.path_hooks}.  Each of the @ref{1866,,path entry hooks} in this list is called with a single argument, the
path entry to be searched.  This callable may either return a @ref{c06,,path entry finder} that can handle the path entry, or it may raise
@ref{3b8,,ImportError}.  An @ref{3b8,,ImportError} is used by the path based finder to
signal that the hook cannot find a @ref{c06,,path entry finder}
for that @ref{1864,,path entry}.  The
exception is ignored and @ref{184a,,import path} iteration continues.  The hook
should expect either a string or bytes object; the encoding of bytes objects
is up to the hook (e.g. it may be a file system encoding, UTF-8, or something
else), and if the hook cannot decode the argument, it should raise
@ref{3b8,,ImportError}.

If @ref{bb5,,sys.path_hooks} iteration ends with no @ref{c06,,path entry finder}
being returned, then the path based finder’s
@ref{b99,,find_spec()} method will store @code{None}
in @ref{74c,,sys.path_importer_cache} (to indicate that there is no finder for
this path entry) and return @code{None}, indicating that this
@ref{c05,,meta path finder} could not find the module.

If a @ref{c06,,path entry finder} @emph{is} returned by one of the @ref{1866,,path entry hook} callables on @ref{bb5,,sys.path_hooks}, then the following protocol is used
to ask the finder for a module spec, which is then used when loading the
module.

The current working directory – denoted by an empty string – is handled
slightly differently from other entries on @ref{489,,sys.path}. First, if the
current working directory is found to not exist, no value is stored in
@ref{74c,,sys.path_importer_cache}. Second, the value for the current working
directory is looked up fresh for each module lookup. Third, the path used for
@ref{74c,,sys.path_importer_cache} and returned by
@ref{b99,,importlib.machinery.PathFinder.find_spec()} will be the actual current
working directory and not the empty string.

@node Path entry finder protocol,,Path entry finders,The Path Based Finder
@anchor{reference/import path-entry-finder-protocol}@anchor{1867}
@subsubsection Path entry finder protocol


In order to support imports of modules and initialized packages and also to
contribute portions to namespace packages, path entry finders must implement
the @ref{309,,find_spec()} method.

@ref{309,,find_spec()} takes two arguments: the
fully qualified name of the module being imported, and the (optional) target
module.  @code{find_spec()} returns a fully populated spec for the module.
This spec will always have “loader” set (with one exception).

To indicate to the import machinery that the spec represents a namespace
@ref{1842,,portion}, the path entry finder sets “submodule_search_locations” to
a list containing the portion.


Changed in version 3.4: @ref{309,,find_spec()} replaced
@ref{2ff,,find_loader()} and
@ref{307,,find_module()}, both of which
are now deprecated, but will be used if @code{find_spec()} is not defined.

Older path entry finders may implement one of these two deprecated methods
instead of @code{find_spec()}.  The methods are still respected for the
sake of backward compatibility.  However, if @code{find_spec()} is
implemented on the path entry finder, the legacy methods are ignored.

@ref{2ff,,find_loader()} takes one argument, the
fully qualified name of the module being imported.  @code{find_loader()}
returns a 2-tuple where the first item is the loader and the second item
is a namespace @ref{1842,,portion}.

For backwards compatibility with other implementations of the import
protocol, many path entry finders also support the same,
traditional @code{find_module()} method that meta path finders support.
However path entry finder @code{find_module()} methods are never called
with a @code{path} argument (they are expected to record the appropriate
path information from the initial call to the path hook).

The @code{find_module()} method on path entry finders is deprecated,
as it does not allow the path entry finder to contribute portions to
namespace packages.  If both @code{find_loader()} and @code{find_module()}
exist on a path entry finder, the import system will always call
@code{find_loader()} in preference to @code{find_module()}.



Changed in version 3.10: Calls to @ref{307,,find_module()} and
@ref{2ff,,find_loader()} by the import
system will raise @ref{304,,ImportWarning}.


@node Replacing the standard import system,Package Relative Imports,The Path Based Finder,The import system
@anchor{reference/import replacing-the-standard-import-system}@anchor{1868}
@subsection Replacing the standard import system


The most reliable mechanism for replacing the entire import system is to
delete the default contents of @ref{87c,,sys.meta_path}, replacing them
entirely with a custom meta path hook.

If it is acceptable to only alter the behaviour of import statements
without affecting other APIs that access the import system, then replacing
the builtin @ref{3b7,,__import__()} function may be sufficient. This technique
may also be employed at the module level to only alter the behaviour of
import statements within that module.

To selectively prevent the import of some modules from a hook early on the
meta path (rather than disabling the standard import system entirely),
it is sufficient to raise @ref{66b,,ModuleNotFoundError} directly from
@ref{308,,find_spec()} instead of returning
@code{None}. The latter indicates that the meta path search should continue,
while raising an exception terminates it immediately.

@node Package Relative Imports,Special considerations for __main__,Replacing the standard import system,The import system
@anchor{reference/import package-relative-imports}@anchor{1869}@anchor{reference/import relativeimports}@anchor{186a}
@subsection Package Relative Imports


Relative imports use leading dots. A single leading dot indicates a relative
import, starting with the current package. Two or more leading dots indicate a
relative import to the parent(s) of the current package, one level per dot
after the first. For example, given the following package layout:

@example
package/
    __init__.py
    subpackage1/
        __init__.py
        moduleX.py
        moduleY.py
    subpackage2/
        __init__.py
        moduleZ.py
    moduleA.py
@end example

In either @code{subpackage1/moduleX.py} or @code{subpackage1/__init__.py},
the following are valid relative imports:

@example
from .moduleY import spam
from .moduleY import spam as ham
from . import moduleY
from ..subpackage1 import moduleY
from ..subpackage2.moduleZ import eggs
from ..moduleA import foo
@end example

Absolute imports may use either the @code{import <>} or @code{from <> import <>}
syntax, but relative imports may only use the second form; the reason
for this is that:

@example
import XXX.YYY.ZZZ
@end example

should expose @code{XXX.YYY.ZZZ} as a usable expression, but .moduleY is
not a valid expression.

@node Special considerations for __main__,Open issues,Package Relative Imports,The import system
@anchor{reference/import import-dunder-main}@anchor{186b}@anchor{reference/import special-considerations-for-main}@anchor{186c}
@subsection Special considerations for __main__


The @ref{1,,__main__} module is a special case relative to Python’s import
system.  As noted @ref{185a,,elsewhere}, the @code{__main__} module
is directly initialized at interpreter startup, much like @ref{f7,,sys} and
@ref{11,,builtins}.  However, unlike those two, it doesn’t strictly
qualify as a built-in module.  This is because the manner in which
@code{__main__} is initialized depends on the flags and other options with
which the interpreter is invoked.

@menu
* __main__.__spec__: __main__ __spec__. 

@end menu

@node __main__ __spec__,,,Special considerations for __main__
@anchor{reference/import id5}@anchor{186d}@anchor{reference/import main-spec}@anchor{185b}
@subsubsection __main__.__spec__


Depending on how @ref{1,,__main__} is initialized, @code{__main__.__spec__}
gets set appropriately or to @code{None}.

When Python is started with the @ref{611,,-m} option, @code{__spec__} is set
to the module spec of the corresponding module or package. @code{__spec__} is
also populated when the @code{__main__} module is loaded as part of executing a
directory, zipfile or other @ref{489,,sys.path} entry.

In @ref{15fd,,the remaining cases}
@code{__main__.__spec__} is set to @code{None}, as the code used to populate the
@ref{1,,__main__} does not correspond directly with an importable module:


@itemize -

@item 
interactive prompt

@item 
@ref{1180,,-c} option

@item 
running from stdin

@item 
running directly from a source or bytecode file
@end itemize

Note that @code{__main__.__spec__} is always @code{None} in the last case,
@emph{even if} the file could technically be imported directly as a module
instead. Use the @ref{611,,-m} switch if valid module metadata is desired
in @ref{1,,__main__}.

Note also that even when @code{__main__} corresponds with an importable module
and @code{__main__.__spec__} is set accordingly, they’re still considered
@emph{distinct} modules. This is due to the fact that blocks guarded by
@code{if __name__ == "__main__":} checks only execute when the module is used
to populate the @code{__main__} namespace, and not during normal import.

@node Open issues,References,Special considerations for __main__,The import system
@anchor{reference/import open-issues}@anchor{186e}
@subsection Open issues


XXX It would be really nice to have a diagram.

XXX * (import_machinery.rst) how about a section devoted just to the
attributes of modules and packages, perhaps expanding upon or supplanting the
related entries in the data model reference page?

XXX runpy, pkgutil, et al in the library manual should all get “See Also”
links at the top pointing to the new import system section.

XXX Add more explanation regarding the different ways in which
@code{__main__} is initialized?

XXX Add more info on @code{__main__} quirks/pitfalls (i.e. copy from
@geindex Python Enhancement Proposals; PEP 395
PEP 395@footnote{https://www.python.org/dev/peps/pep-0395}).

@node References,,Open issues,The import system
@anchor{reference/import references}@anchor{186f}
@subsection References


The import machinery has evolved considerably since Python’s early days.  The
original specification for packages@footnote{https://www.python.org/doc/essays/packages/} is still available to read,
although some details have changed since the writing of that document.

The original specification for @ref{87c,,sys.meta_path} was 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302}, with
subsequent extension in 
@geindex Python Enhancement Proposals; PEP 420
PEP 420@footnote{https://www.python.org/dev/peps/pep-0420}.

@geindex Python Enhancement Proposals; PEP 420
PEP 420@footnote{https://www.python.org/dev/peps/pep-0420} introduced @ref{1840,,namespace packages} for
Python 3.3.  
@geindex Python Enhancement Proposals; PEP 420
PEP 420@footnote{https://www.python.org/dev/peps/pep-0420} also introduced the @code{find_loader()} protocol as an
alternative to @code{find_module()}.

@geindex Python Enhancement Proposals; PEP 366
PEP 366@footnote{https://www.python.org/dev/peps/pep-0366} describes the addition of the @code{__package__} attribute for
explicit relative imports in main modules.

@geindex Python Enhancement Proposals; PEP 328
PEP 328@footnote{https://www.python.org/dev/peps/pep-0328} introduced absolute and explicit relative imports and initially
proposed @code{__name__} for semantics 
@geindex Python Enhancement Proposals; PEP 366
PEP 366@footnote{https://www.python.org/dev/peps/pep-0366} would eventually specify for
@code{__package__}.

@geindex Python Enhancement Proposals; PEP 338
PEP 338@footnote{https://www.python.org/dev/peps/pep-0338} defines executing modules as scripts.

@geindex Python Enhancement Proposals; PEP 451
PEP 451@footnote{https://www.python.org/dev/peps/pep-0451} adds the encapsulation of per-module import state in spec
objects.  It also off-loads most of the boilerplate responsibilities of
loaders back onto the import machinery.  These changes allow the
deprecation of several APIs in the import system and also addition of new
methods to finders and loaders.

@node Expressions,Simple statements,The import system,The Python Language Reference
@anchor{reference/expressions doc}@anchor{1870}@anchor{reference/expressions expressions}@anchor{1871}@anchor{reference/expressions id1}@anchor{1872}
@section Expressions


@geindex expression
@geindex BNF

This chapter explains the meaning of the elements of expressions in Python.

@strong{Syntax Notes:} In this and the following chapters, extended BNF notation will
be used to describe syntax, not lexical analysis.  When (one alternative of) a
syntax rule has the form

@example
@anchor{reference/expressions grammar-token-python-grammar-name}@anchor{1873}@anchor{reference/expressions grammar-token-name}@anchor{1874}name ::= othername
@end example

and no semantics are given, the semantics of this form of @code{name} are the same
as for @code{othername}.

@menu
* Arithmetic conversions:: 
* Atoms:: 
* Primaries:: 
* Await expression:: 
* The power operator:: 
* Unary arithmetic and bitwise operations:: 
* Binary arithmetic operations:: 
* Shifting operations:: 
* Binary bitwise operations:: 
* Comparisons:: 
* Boolean operations:: 
* Assignment expressions: Assignment expressions<2>. 
* Conditional expressions:: 
* Lambdas:: 
* Expression lists:: 
* Evaluation order:: 
* Operator precedence:: 

@end menu

@node Arithmetic conversions,Atoms,,Expressions
@anchor{reference/expressions arithmetic-conversions}@anchor{1875}@anchor{reference/expressions conversions}@anchor{1876}
@subsection Arithmetic conversions


@geindex arithmetic; conversion

When a description of an arithmetic operator below uses the phrase “the numeric
arguments are converted to a common type”, this means that the operator
implementation for built-in types works as follows:


@itemize *

@item 
If either argument is a complex number, the other is converted to complex;

@item 
otherwise, if either argument is a floating point number, the other is
converted to floating point;

@item 
otherwise, both must be integers and no conversion is necessary.
@end itemize

Some additional rules apply for certain operators (e.g., a string as a left
argument to the ‘%’ operator).  Extensions must define their own conversion
behavior.

@node Atoms,Primaries,Arithmetic conversions,Expressions
@anchor{reference/expressions atoms}@anchor{1877}@anchor{reference/expressions id2}@anchor{1878}
@subsection Atoms


@geindex atom

Atoms are the most basic elements of expressions.  The simplest atoms are
identifiers or literals.  Forms enclosed in parentheses, brackets or braces are
also categorized syntactically as atoms.  The syntax for atoms is:

@example
@anchor{reference/expressions grammar-token-python-grammar-atom}@anchor{1879}@anchor{reference/expressions grammar-token-atom}@anchor{187a}atom      ::= identifier | literal | enclosure
@anchor{reference/expressions grammar-token-python-grammar-enclosure}@anchor{187b}@anchor{reference/expressions grammar-token-enclosure}@anchor{187c}enclosure ::= parenth_form | list_display | dict_display | set_display
              | generator_expression | yield_atom
@end example

@menu
* Identifiers (Names): Identifiers Names. 
* Literals: Literals<2>. 
* Parenthesized forms:: 
* Displays for lists@comma{} sets and dictionaries: Displays for lists sets and dictionaries. 
* List displays:: 
* Set displays:: 
* Dictionary displays:: 
* Generator expressions:: 
* Yield expressions:: 

@end menu

@node Identifiers Names,Literals<2>,,Atoms
@anchor{reference/expressions atom-identifiers}@anchor{171c}@anchor{reference/expressions identifiers-names}@anchor{187d}
@subsubsection Identifiers (Names)


@geindex name
@geindex identifier

An identifier occurring as an atom is a name.  See section @ref{170b,,Identifiers and keywords}
for lexical definition and section @ref{182a,,Naming and binding} for documentation of naming and
binding.

@geindex exception; NameError

When the name is bound to an object, evaluation of the atom yields that object.
When a name is not bound, an attempt to evaluate it raises a @ref{21f,,NameError}
exception.
@anchor{reference/expressions private-name-mangling}@anchor{187e}
@geindex name; mangling
@geindex private; names

@strong{Private name mangling:} When an identifier that textually occurs in a class
definition begins with two or more underscore characters and does not end in two
or more underscores, it is considered a @emph{private name} of that class.
Private names are transformed to a longer form before code is generated for
them.  The transformation inserts the class name, with leading underscores
removed and a single underscore inserted, in front of the name.  For example,
the identifier @code{__spam} occurring in a class named @code{Ham} will be transformed
to @code{_Ham__spam}.  This transformation is independent of the syntactical
context in which the identifier is used.  If the transformed name is extremely
long (longer than 255 characters), implementation defined truncation may happen.
If the class name consists only of underscores, no transformation is done.

@node Literals<2>,Parenthesized forms,Identifiers Names,Atoms
@anchor{reference/expressions atom-literals}@anchor{187f}@anchor{reference/expressions literals}@anchor{1880}
@subsubsection Literals


@geindex literal

Python supports string and bytes literals and various numeric literals:

@example
@anchor{reference/expressions grammar-token-python-grammar-literal}@anchor{1881}@anchor{reference/expressions grammar-token-literal}@anchor{1882}literal ::= stringliteral | bytesliteral
            | integer | floatnumber | imagnumber
@end example

Evaluation of a literal yields an object of the given type (string, bytes,
integer, floating point number, complex number) with the given value.  The value
may be approximated in the case of floating point and imaginary (complex)
literals.  See section @ref{171e,,Literals} for details.

@geindex immutable; data; type
@geindex immutable; object

All literals correspond to immutable data types, and hence the object’s identity
is less important than its value.  Multiple evaluations of literals with the
same value (either the same occurrence in the program text or a different
occurrence) may obtain the same object or a different object with the same
value.

@node Parenthesized forms,Displays for lists sets and dictionaries,Literals<2>,Atoms
@anchor{reference/expressions parenthesized}@anchor{1883}@anchor{reference/expressions parenthesized-forms}@anchor{1884}
@subsubsection Parenthesized forms


@geindex parenthesized form
@geindex () (parentheses); tuple display

A parenthesized form is an optional expression list enclosed in parentheses:

@example
@anchor{reference/expressions grammar-token-python-grammar-parenth_form}@anchor{1885}@anchor{reference/expressions grammar-token-parenth-form}@anchor{1886}parenth_form ::= "(" [starred_expression] ")"
@end example

A parenthesized expression list yields whatever that expression list yields: if
the list contains at least one comma, it yields a tuple; otherwise, it yields
the single expression that makes up the expression list.

@geindex empty; tuple

An empty pair of parentheses yields an empty tuple object.  Since tuples are
immutable, the same rules as for literals apply (i.e., two occurrences of the empty
tuple may or may not yield the same object).

@geindex comma
@geindex @comma{} (comma)

Note that tuples are not formed by the parentheses, but rather by use of the
comma operator.  The exception is the empty tuple, for which parentheses @emph{are}
required — allowing unparenthesized “nothing” in expressions would cause
ambiguities and allow common typos to pass uncaught.

@node Displays for lists sets and dictionaries,List displays,Parenthesized forms,Atoms
@anchor{reference/expressions comprehensions}@anchor{1887}@anchor{reference/expressions displays-for-lists-sets-and-dictionaries}@anchor{1888}
@subsubsection Displays for lists, sets and dictionaries


@geindex comprehensions

For constructing a list, a set or a dictionary Python provides special syntax
called “displays”, each of them in two flavors:


@itemize *

@item 
either the container contents are listed explicitly, or

@item 
they are computed via a set of looping and filtering instructions, called a
@emph{comprehension}.
@end itemize

@geindex for; in comprehensions
@geindex if; in comprehensions
@geindex async for; in comprehensions

Common syntax elements for comprehensions are:

@example
@anchor{reference/expressions grammar-token-python-grammar-comprehension}@anchor{1889}@anchor{reference/expressions grammar-token-comprehension}@anchor{188a}comprehension ::= assignment_expression comp_for
@anchor{reference/expressions grammar-token-python-grammar-comp_for}@anchor{188b}@anchor{reference/expressions grammar-token-comp-for}@anchor{188c}comp_for      ::= ["async"] "for" target_list "in" or_test [comp_iter]
@anchor{reference/expressions grammar-token-python-grammar-comp_iter}@anchor{188d}@anchor{reference/expressions grammar-token-comp-iter}@anchor{188e}comp_iter     ::= comp_for | comp_if
@anchor{reference/expressions grammar-token-python-grammar-comp_if}@anchor{188f}@anchor{reference/expressions grammar-token-comp-if}@anchor{1890}comp_if       ::= "if" or_test [comp_iter]
@end example

The comprehension consists of a single expression followed by at least one
@code{for} clause and zero or more @code{for} or @code{if} clauses.
In this case, the elements of the new container are those that would be produced
by considering each of the @code{for} or @code{if} clauses a block,
nesting from left to right, and evaluating the expression to produce an element
each time the innermost block is reached.

However, aside from the iterable expression in the leftmost @code{for} clause,
the comprehension is executed in a separate implicitly nested scope. This ensures
that names assigned to in the target list don’t “leak” into the enclosing scope.

The iterable expression in the leftmost @code{for} clause is evaluated
directly in the enclosing scope and then passed as an argument to the implicitly
nested scope. Subsequent @code{for} clauses and any filter condition in the
leftmost @code{for} clause cannot be evaluated in the enclosing scope as
they may depend on the values obtained from the leftmost iterable. For example:
@code{[x*y for x in range(10) for y in range(x, x+10)]}.

To ensure the comprehension always results in a container of the appropriate
type, @code{yield} and @code{yield from} expressions are prohibited in the implicitly
nested scope.

@geindex await; in comprehensions

Since Python 3.6, in an @ref{1c0,,async def} function, an @code{async for}
clause may be used to iterate over a @ref{8d2,,asynchronous iterator}.
A comprehension in an @code{async def} function may consist of either a
@code{for} or @code{async for} clause following the leading
expression, may contain additional @code{for} or @code{async for}
clauses, and may also use @ref{4d5,,await} expressions.
If a comprehension contains either @code{async for} clauses or
@code{await} expressions or other asynchronous comprehensions it is called
an @emph{asynchronous comprehension}.  An asynchronous comprehension may
suspend the execution of the coroutine function in which it appears.
See also 
@geindex Python Enhancement Proposals; PEP 530
PEP 530@footnote{https://www.python.org/dev/peps/pep-0530}.


New in version 3.6: Asynchronous comprehensions were introduced.



Changed in version 3.8: @code{yield} and @code{yield from} prohibited in the implicitly nested scope.



Changed in version 3.11: Asynchronous comprehensions are now allowed inside comprehensions in
asynchronous functions. Outer comprehensions implicitly become
asynchronous.


@node List displays,Set displays,Displays for lists sets and dictionaries,Atoms
@anchor{reference/expressions list-displays}@anchor{1891}@anchor{reference/expressions lists}@anchor{1892}
@subsubsection List displays


@geindex list; display
@geindex list; comprehensions
@geindex empty; list
@geindex object; list
@geindex [] (square brackets); list expression
@geindex @comma{} (comma); expression list

A list display is a possibly empty series of expressions enclosed in square
brackets:

@example
@anchor{reference/expressions grammar-token-python-grammar-list_display}@anchor{1893}@anchor{reference/expressions grammar-token-list-display}@anchor{1894}list_display ::= "[" [starred_list | comprehension] "]"
@end example

A list display yields a new list object, the contents being specified by either
a list of expressions or a comprehension.  When a comma-separated list of
expressions is supplied, its elements are evaluated from left to right and
placed into the list object in that order.  When a comprehension is supplied,
the list is constructed from the elements resulting from the comprehension.

@node Set displays,Dictionary displays,List displays,Atoms
@anchor{reference/expressions set}@anchor{1895}@anchor{reference/expressions set-displays}@anchor{1896}
@subsubsection Set displays


@geindex set; display
@geindex set; comprehensions
@geindex object; set
@geindex @{@} (curly brackets); set expression
@geindex @comma{} (comma); expression list

A set display is denoted by curly braces and distinguishable from dictionary
displays by the lack of colons separating keys and values:

@example
@anchor{reference/expressions grammar-token-python-grammar-set_display}@anchor{1897}@anchor{reference/expressions grammar-token-set-display}@anchor{1898}set_display ::= "@{" (starred_list | comprehension) "@}"
@end example

A set display yields a new mutable set object, the contents being specified by
either a sequence of expressions or a comprehension.  When a comma-separated
list of expressions is supplied, its elements are evaluated from left to right
and added to the set object.  When a comprehension is supplied, the set is
constructed from the elements resulting from the comprehension.

An empty set cannot be constructed with @code{@{@}}; this literal constructs an empty
dictionary.

@node Dictionary displays,Generator expressions,Set displays,Atoms
@anchor{reference/expressions dict}@anchor{8d8}@anchor{reference/expressions dictionary-displays}@anchor{1899}
@subsubsection Dictionary displays


@geindex dictionary; display
@geindex dictionary; comprehensions
@geindex key
@geindex datum
@geindex key/datum pair
@geindex object; dictionary
@geindex @{@} (curly brackets); dictionary expression
@geindex ; (colon); in dictionary expressions
@geindex @comma{} (comma); in dictionary displays

A dictionary display is a possibly empty series of key/datum pairs enclosed in
curly braces:

@example
@anchor{reference/expressions grammar-token-python-grammar-dict_display}@anchor{189a}@anchor{reference/expressions grammar-token-dict-display}@anchor{189b}dict_display       ::= "@{" [key_datum_list | dict_comprehension] "@}"
@anchor{reference/expressions grammar-token-python-grammar-key_datum_list}@anchor{189c}@anchor{reference/expressions grammar-token-key-datum-list}@anchor{189d}key_datum_list     ::= key_datum ("," key_datum)* [","]
@anchor{reference/expressions grammar-token-python-grammar-key_datum}@anchor{189e}@anchor{reference/expressions grammar-token-key-datum}@anchor{189f}key_datum          ::= expression ":" expression | "**" or_expr
@anchor{reference/expressions grammar-token-python-grammar-dict_comprehension}@anchor{18a0}@anchor{reference/expressions grammar-token-dict-comprehension}@anchor{18a1}dict_comprehension ::= expression ":" expression comp_for
@end example

A dictionary display yields a new dictionary object.

If a comma-separated sequence of key/datum pairs is given, they are evaluated
from left to right to define the entries of the dictionary: each key object is
used as a key into the dictionary to store the corresponding datum.  This means
that you can specify the same key multiple times in the key/datum list, and the
final dictionary’s value for that key will be the last one given.

@geindex unpacking; dictionary
@geindex **; in dictionary displays

A double asterisk @code{**} denotes @emph{dictionary unpacking}.
Its operand must be a @ref{d62,,mapping}.  Each mapping item is added
to the new dictionary.  Later values replace values already set by
earlier key/datum pairs and earlier dictionary unpackings.


New in version 3.5: Unpacking into dictionary displays, originally proposed by 
@geindex Python Enhancement Proposals; PEP 448
PEP 448@footnote{https://www.python.org/dev/peps/pep-0448}.


A dict comprehension, in contrast to list and set comprehensions, needs two
expressions separated with a colon followed by the usual “for” and “if” clauses.
When the comprehension is run, the resulting key and value elements are inserted
in the new dictionary in the order they are produced.

@geindex immutable; object
@geindex hashable

Restrictions on the types of the key values are listed earlier in section
@ref{1787,,The standard type hierarchy}.  (To summarize, the key type should be @ref{2dc,,hashable}, which excludes
all mutable objects.)  Clashes between duplicate keys are not detected; the last
datum (textually rightmost in the display) stored for a given key value
prevails.


Changed in version 3.8: Prior to Python 3.8, in dict comprehensions, the evaluation order of key
and value was not well-defined.  In CPython, the value was evaluated before
the key.  Starting with 3.8, the key is evaluated before the value, as
proposed by 
@geindex Python Enhancement Proposals; PEP 572
PEP 572@footnote{https://www.python.org/dev/peps/pep-0572}.


@node Generator expressions,Yield expressions,Dictionary displays,Atoms
@anchor{reference/expressions generator-expressions}@anchor{18a2}@anchor{reference/expressions genexpr}@anchor{18a3}
@subsubsection Generator expressions


@geindex generator; expression
@geindex object; generator
@geindex () (parentheses); generator expression

A generator expression is a compact generator notation in parentheses:

@example
@anchor{reference/expressions grammar-token-python-grammar-generator_expression}@anchor{18a4}@anchor{reference/expressions grammar-token-generator-expression}@anchor{18a5}generator_expression ::= "(" expression comp_for ")"
@end example

A generator expression yields a new generator object.  Its syntax is the same as
for comprehensions, except that it is enclosed in parentheses instead of
brackets or curly braces.

Variables used in the generator expression are evaluated lazily when the
@ref{159d,,__next__()} method is called for the generator object (in the same
fashion as normal generators).  However, the iterable expression in the
leftmost @code{for} clause is immediately evaluated, so that an error
produced by it will be emitted at the point where the generator expression
is defined, rather than at the point where the first value is retrieved.
Subsequent @code{for} clauses and any filter condition in the leftmost
@code{for} clause cannot be evaluated in the enclosing scope as they may
depend on the values obtained from the leftmost iterable. For example:
@code{(x*y for x in range(10) for y in range(x, x+10))}.

The parentheses can be omitted on calls with only one argument.  See section
@ref{8d6,,Calls} for details.

To avoid interfering with the expected operation of the generator expression
itself, @code{yield} and @code{yield from} expressions are prohibited in the
implicitly defined generator.

If a generator expression contains either @code{async for}
clauses or @ref{4d5,,await} expressions it is called an
@emph{asynchronous generator expression}.  An asynchronous generator
expression returns a new asynchronous generator object,
which is an asynchronous iterator (see @ref{8d3,,Asynchronous Iterators}).


New in version 3.6: Asynchronous generator expressions were introduced.



Changed in version 3.7: Prior to Python 3.7, asynchronous generator expressions could
only appear in @ref{1c0,,async def} coroutines.  Starting
with 3.7, any function can use asynchronous generator expressions.



Changed in version 3.8: @code{yield} and @code{yield from} prohibited in the implicitly nested scope.


@node Yield expressions,,Generator expressions,Atoms
@anchor{reference/expressions yield-expressions}@anchor{18a6}@anchor{reference/expressions yieldexpr}@anchor{18a7}
@subsubsection Yield expressions


@geindex keyword; yield
@geindex keyword; from
@geindex yield; expression
@geindex generator; function

@example
@anchor{reference/expressions grammar-token-python-grammar-yield_atom}@anchor{18a8}@anchor{reference/expressions grammar-token-yield-atom}@anchor{18a9}yield_atom       ::= "(" yield_expression ")"
@anchor{reference/expressions grammar-token-python-grammar-yield_expression}@anchor{18aa}@anchor{reference/expressions grammar-token-yield-expression}@anchor{18ab}yield_expression ::= "yield" [expression_list | "from" expression]
@end example

The yield expression is used when defining a @ref{bf7,,generator} function
or an @ref{18ac,,asynchronous generator} function and
thus can only be used in the body of a function definition.  Using a yield
expression in a function’s body causes that function to be a generator function,
and using it in an @ref{1c0,,async def} function’s body causes that
coroutine function to be an asynchronous generator function. For example:

@example
def gen():  # defines a generator function
    yield 123

async def agen(): # defines an asynchronous generator function
    yield 123
@end example

Due to their side effects on the containing scope, @code{yield} expressions
are not permitted as part of the implicitly defined scopes used to
implement comprehensions and generator expressions.


Changed in version 3.8: Yield expressions prohibited in the implicitly nested scopes used to
implement comprehensions and generator expressions.


Generator functions are described below, while asynchronous generator
functions are described separately in section
@ref{18ad,,Asynchronous generator functions}.

When a generator function is called, it returns an iterator known as a
generator.  That generator then controls the execution of the generator
function.  The execution starts when one of the generator’s methods is called.
At that time, the execution proceeds to the first yield expression, where it is
suspended again, returning the value of @ref{18ae,,expression_list}
to the generator’s caller.  By suspended, we mean that all local state is
retained, including the current bindings of local variables, the instruction
pointer, the internal evaluation stack, and the state of any exception handling.
When the execution is resumed by calling one of the generator’s methods, the
function can proceed exactly as if the yield expression were just another
external call.  The value of the yield expression after resuming depends on the
method which resumed the execution.  If @ref{159d,,__next__()} is used
(typically via either a @ref{161,,for} or the @ref{252,,next()} builtin) then the
result is @ref{182,,None}.  Otherwise, if @ref{181c,,send()} is used, then
the result will be the value passed in to that method.

@geindex coroutine

All of this makes generator functions quite similar to coroutines; they yield
multiple times, they have more than one entry point and their execution can be
suspended.  The only difference is that a generator function cannot control
where the execution should continue after it yields; the control is always
transferred to the generator’s caller.

Yield expressions are allowed anywhere in a @ref{f6d,,try} construct.  If the
generator is not resumed before it is
finalized (by reaching a zero reference count or by being garbage collected),
the generator-iterator’s @ref{181f,,close()} method will be called,
allowing any pending @ref{4c1,,finally} clauses to execute.

@geindex from; yield from expression

When @code{yield from <expr>} is used, the supplied expression must be an
iterable. The values produced by iterating that iterable are passed directly
to the caller of the current generator’s methods. Any values passed in with
@ref{181c,,send()} and any exceptions passed in with
@ref{11be,,throw()} are passed to the underlying iterator if it has the
appropriate methods.  If this is not the case, then @ref{181c,,send()}
will raise @ref{163,,AttributeError} or @ref{162,,TypeError}, while
@ref{11be,,throw()} will just raise the passed in exception immediately.

When the underlying iterator is complete, the @code{value}
attribute of the raised @ref{736,,StopIteration} instance becomes the value of
the yield expression. It can be either set explicitly when raising
@ref{736,,StopIteration}, or automatically when the subiterator is a generator
(by returning a value from the subgenerator).

@quotation


Changed in version 3.3: Added @code{yield from <expr>} to delegate control flow to a subiterator.

@end quotation

The parentheses may be omitted when the yield expression is the sole expression
on the right hand side of an assignment statement.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 255

@item PEP 255@footnote{https://www.python.org/dev/peps/pep-0255} - Simple Generators

The proposal for adding generators and the @ref{4c5,,yield} statement to Python.

@geindex Python Enhancement Proposals; PEP 342

@item PEP 342@footnote{https://www.python.org/dev/peps/pep-0342} - Coroutines via Enhanced Generators

The proposal to enhance the API and syntax of generators, making them
usable as simple coroutines.

@geindex Python Enhancement Proposals; PEP 380

@item PEP 380@footnote{https://www.python.org/dev/peps/pep-0380} - Syntax for Delegating to a Subgenerator

The proposal to introduce the @code{yield_from} syntax,
making delegation to subgenerators easy.

@geindex Python Enhancement Proposals; PEP 525

@item PEP 525@footnote{https://www.python.org/dev/peps/pep-0525} - Asynchronous Generators

The proposal that expanded on 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} by adding generator capabilities to
coroutine functions.
@end table


@geindex object; generator

@menu
* Generator-iterator methods:: 
* Examples:: 
* Asynchronous generator functions:: 
* Asynchronous generator-iterator methods:: 

@end menu

@node Generator-iterator methods,Examples,,Yield expressions
@anchor{reference/expressions generator-iterator-methods}@anchor{18af}@anchor{reference/expressions generator-methods}@anchor{181a}
@subsubsection Generator-iterator methods


This subsection describes the methods of a generator iterator.  They can
be used to control the execution of a generator function.

Note that calling any of the generator methods below when the generator
is already executing raises a @ref{17c,,ValueError} exception.

@geindex exception; StopIteration

@geindex __next__() (generator method)
@anchor{reference/expressions generator __next__}@anchor{159d}
@deffn {Method} generator.__next__ ()

Starts the execution of a generator function or resumes it at the last
executed yield expression.  When a generator function is resumed with a
@ref{159d,,__next__()} method, the current yield expression always
evaluates to @ref{182,,None}.  The execution then continues to the next yield
expression, where the generator is suspended again, and the value of the
@ref{18ae,,expression_list} is returned to @ref{159d,,__next__()}’s
caller.  If the generator exits without yielding another value, a
@ref{736,,StopIteration} exception is raised.

This method is normally called implicitly, e.g. by a @ref{161,,for} loop, or
by the built-in @ref{252,,next()} function.
@end deffn

@geindex send() (generator method)
@anchor{reference/expressions generator send}@anchor{181c}
@deffn {Method} generator.send (value)

Resumes the execution and “sends” a value into the generator function.  The
@emph{value} argument becomes the result of the current yield expression.  The
@ref{181c,,send()} method returns the next value yielded by the generator, or
raises @ref{736,,StopIteration} if the generator exits without yielding another
value.  When @ref{181c,,send()} is called to start the generator, it must be called
with @ref{182,,None} as the argument, because there is no yield expression that
could receive the value.
@end deffn

@geindex throw() (generator method)
@anchor{reference/expressions generator throw}@anchor{11be}
@deffn {Method} generator.throw (type[, value[, traceback]])

Raises an exception of type @code{type} at the point where the generator was paused,
and returns the next value yielded by the generator function.  If the generator
exits without yielding another value, a @ref{736,,StopIteration} exception is
raised.  If the generator function does not catch the passed-in exception, or
raises a different exception, then that exception propagates to the caller.
@end deffn

@geindex exception; GeneratorExit

@geindex close() (generator method)
@anchor{reference/expressions generator close}@anchor{181f}
@deffn {Method} generator.close ()

Raises a @ref{f2e,,GeneratorExit} at the point where the generator function was
paused.  If the generator function then exits gracefully, is already closed,
or raises @ref{f2e,,GeneratorExit} (by not catching the exception), close
returns to its caller.  If the generator yields a value, a
@ref{5af,,RuntimeError} is raised.  If the generator raises any other exception,
it is propagated to the caller.  @ref{181f,,close()} does nothing if the generator
has already exited due to an exception or normal exit.
@end deffn

@geindex yield; examples

@node Examples,Asynchronous generator functions,Generator-iterator methods,Yield expressions
@anchor{reference/expressions examples}@anchor{18b0}
@subsubsection Examples


Here is a simple example that demonstrates the behavior of generators and
generator functions:

@example
>>> def echo(value=None):
...     print("Execution starts when 'next()' is called for the first time.")
...     try:
...         while True:
...             try:
...                 value = (yield value)
...             except Exception as e:
...                 value = e
...     finally:
...         print("Don't forget to clean up when 'close()' is called.")
...
>>> generator = echo(1)
>>> print(next(generator))
Execution starts when 'next()' is called for the first time.
1
>>> print(next(generator))
None
>>> print(generator.send(2))
2
>>> generator.throw(TypeError, "spam")
TypeError('spam',)
>>> generator.close()
Don't forget to clean up when 'close()' is called.
@end example

For examples using @code{yield from}, see @ref{bce,,PEP 380; Syntax for Delegating to a Subgenerator} in “What’s New in
Python.”

@node Asynchronous generator functions,Asynchronous generator-iterator methods,Examples,Yield expressions
@anchor{reference/expressions asynchronous-generator-functions}@anchor{18ad}@anchor{reference/expressions id3}@anchor{18b1}
@subsubsection Asynchronous generator functions


The presence of a yield expression in a function or method defined using
@ref{1c0,,async def} further defines the function as an
@ref{18ac,,asynchronous generator} function.

When an asynchronous generator function is called, it returns an
asynchronous iterator known as an asynchronous generator object.
That object then controls the execution of the generator function.
An asynchronous generator object is typically used in an
@ref{5cd,,async for} statement in a coroutine function analogously to
how a generator object would be used in a @ref{161,,for} statement.

Calling one of the asynchronous generator’s methods returns an @ref{7bd,,awaitable}
object, and the execution starts when this object is awaited on. At that time,
the execution proceeds to the first yield expression, where it is suspended
again, returning the value of @ref{18ae,,expression_list} to the
awaiting coroutine. As with a generator, suspension means that all local state
is retained, including the current bindings of local variables, the instruction
pointer, the internal evaluation stack, and the state of any exception handling.
When the execution is resumed by awaiting on the next object returned by the
asynchronous generator’s methods, the function can proceed exactly as if the
yield expression were just another external call. The value of the yield
expression after resuming depends on the method which resumed the execution.  If
@ref{18b2,,__anext__()} is used then the result is @ref{182,,None}. Otherwise, if
@ref{3be,,asend()} is used, then the result will be the value passed in to that
method.

If an asynchronous generator happens to exit early by @ref{5cb,,break}, the caller
task being cancelled, or other exceptions, the generator’s async cleanup code
will run and possibly raise exceptions or access context variables in an
unexpected context–perhaps after the lifetime of tasks it depends, or
during the event loop shutdown when the async-generator garbage collection hook
is called.
To prevent this, the caller must explicitly close the async generator by calling
@ref{3bd,,aclose()} method to finalize the generator and ultimately detach it
from the event loop.

In an asynchronous generator function, yield expressions are allowed anywhere
in a @ref{f6d,,try} construct. However, if an asynchronous generator is not
resumed before it is finalized (by reaching a zero reference count or by
being garbage collected), then a yield expression within a @code{try}
construct could result in a failure to execute pending @ref{4c1,,finally}
clauses.  In this case, it is the responsibility of the event loop or
scheduler running the asynchronous generator to call the asynchronous
generator-iterator’s @ref{3bd,,aclose()} method and run the resulting
coroutine object, thus allowing any pending @code{finally} clauses
to execute.

To take care of finalization upon event loop termination, an event loop should
define a @emph{finalizer} function which takes an asynchronous generator-iterator and
presumably calls @ref{3bd,,aclose()} and executes the coroutine.
This  @emph{finalizer} may be registered by calling @ref{11fd,,sys.set_asyncgen_hooks()}.
When first iterated over, an asynchronous generator-iterator will store the
registered @emph{finalizer} to be called upon finalization. For a reference example
of a @emph{finalizer} method see the implementation of
@code{asyncio.Loop.shutdown_asyncgens} in Lib/asyncio/base_events.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/base_events.py}.

The expression @code{yield from <expr>} is a syntax error when used in an
asynchronous generator function.

@geindex object; asynchronous-generator

@node Asynchronous generator-iterator methods,,Asynchronous generator functions,Yield expressions
@anchor{reference/expressions asynchronous-generator-iterator-methods}@anchor{18b3}@anchor{reference/expressions asynchronous-generator-methods}@anchor{18b4}
@subsubsection Asynchronous generator-iterator methods


This subsection describes the methods of an asynchronous generator iterator,
which are used to control the execution of a generator function.

@geindex exception; StopAsyncIteration

@geindex __anext__() (agen method)
@anchor{reference/expressions agen __anext__}@anchor{18b2}
@deffn {Method} coroutine agen.__anext__ ()

Returns an awaitable which when run starts to execute the asynchronous
generator or resumes it at the last executed yield expression.  When an
asynchronous generator function is resumed with an @ref{18b2,,__anext__()}
method, the current yield expression always evaluates to @ref{182,,None} in the
returned awaitable, which when run will continue to the next yield
expression. The value of the @ref{18ae,,expression_list} of the
yield expression is the value of the @ref{736,,StopIteration} exception raised by
the completing coroutine.  If the asynchronous generator exits without
yielding another value, the awaitable instead raises a
@ref{12da,,StopAsyncIteration} exception, signalling that the asynchronous
iteration has completed.

This method is normally called implicitly by a @ref{5cd,,async for} loop.
@end deffn

@geindex asend() (agen method)
@anchor{reference/expressions agen asend}@anchor{3be}
@deffn {Method} coroutine agen.asend (value)

Returns an awaitable which when run resumes the execution of the
asynchronous generator. As with the @ref{181c,,send()} method for a
generator, this “sends” a value into the asynchronous generator function,
and the @emph{value} argument becomes the result of the current yield expression.
The awaitable returned by the @ref{3be,,asend()} method will return the next
value yielded by the generator as the value of the raised
@ref{736,,StopIteration}, or raises @ref{12da,,StopAsyncIteration} if the
asynchronous generator exits without yielding another value.  When
@ref{3be,,asend()} is called to start the asynchronous
generator, it must be called with @ref{182,,None} as the argument,
because there is no yield expression that could receive the value.
@end deffn

@geindex athrow() (agen method)
@anchor{reference/expressions agen athrow}@anchor{3bf}
@deffn {Method} coroutine agen.athrow (type[, value[, traceback]])

Returns an awaitable that raises an exception of type @code{type} at the point
where the asynchronous generator was paused, and returns the next value
yielded by the generator function as the value of the raised
@ref{736,,StopIteration} exception.  If the asynchronous generator exits
without yielding another value, a @ref{12da,,StopAsyncIteration} exception is
raised by the awaitable.
If the generator function does not catch the passed-in exception, or
raises a different exception, then when the awaitable is run that exception
propagates to the caller of the awaitable.
@end deffn

@geindex exception; GeneratorExit

@geindex aclose() (agen method)
@anchor{reference/expressions agen aclose}@anchor{3bd}
@deffn {Method} coroutine agen.aclose ()

Returns an awaitable that when run will throw a @ref{f2e,,GeneratorExit} into
the asynchronous generator function at the point where it was paused.
If the asynchronous generator function then exits gracefully, is already
closed, or raises @ref{f2e,,GeneratorExit} (by not catching the exception),
then the returned awaitable will raise a @ref{736,,StopIteration} exception.
Any further awaitables returned by subsequent calls to the asynchronous
generator will raise a @ref{12da,,StopAsyncIteration} exception.  If the
asynchronous generator yields a value, a @ref{5af,,RuntimeError} is raised
by the awaitable.  If the asynchronous generator raises any other exception,
it is propagated to the caller of the awaitable.  If the asynchronous
generator has already exited due to an exception or normal exit, then
further calls to @ref{3bd,,aclose()} will return an awaitable that does nothing.
@end deffn

@node Primaries,Await expression,Atoms,Expressions
@anchor{reference/expressions id4}@anchor{18b5}@anchor{reference/expressions primaries}@anchor{18b6}
@subsection Primaries


@geindex primary

Primaries represent the most tightly bound operations of the language. Their
syntax is:

@example
@anchor{reference/expressions grammar-token-python-grammar-primary}@anchor{18b7}@anchor{reference/expressions grammar-token-primary}@anchor{18b8}primary ::= atom | attributeref | subscription | slicing | call
@end example

@menu
* Attribute references:: 
* Subscriptions:: 
* Slicings:: 
* Calls:: 

@end menu

@node Attribute references,Subscriptions,,Primaries
@anchor{reference/expressions attribute-references}@anchor{18b9}@anchor{reference/expressions id5}@anchor{18ba}
@subsubsection Attribute references


@geindex attribute; reference
@geindex . (dot); attribute reference

An attribute reference is a primary followed by a period and a name:

@example
@anchor{reference/expressions grammar-token-python-grammar-attributeref}@anchor{18bb}@anchor{reference/expressions grammar-token-attributeref}@anchor{18bc}attributeref ::= primary "." identifier
@end example

@geindex exception; AttributeError
@geindex object; module
@geindex object; list

The primary must evaluate to an object of a type that supports attribute
references, which most objects do.  This object is then asked to produce the
attribute whose name is the identifier.  This production can be customized by
overriding the @code{__getattr__()} method.  If this attribute is not available,
the exception @ref{163,,AttributeError} is raised.  Otherwise, the type and value of
the object produced is determined by the object.  Multiple evaluations of the
same attribute reference may yield different objects.

@node Subscriptions,Slicings,Attribute references,Primaries
@anchor{reference/expressions id6}@anchor{18bd}@anchor{reference/expressions subscriptions}@anchor{17d1}
@subsubsection Subscriptions


@geindex subscription
@geindex [] (square brackets); subscription

@geindex object; sequence
@geindex object; mapping
@geindex object; string
@geindex object; tuple
@geindex object; list
@geindex object; dictionary
@geindex sequence; item

Subscription of a sequence (string, tuple or list) or mapping (dictionary)
object usually selects an item from the collection:

@example
@anchor{reference/expressions grammar-token-python-grammar-subscription}@anchor{18be}@anchor{reference/expressions grammar-token-subscription}@anchor{18bf}subscription ::= primary "[" expression_list "]"
@end example

The primary must evaluate to an object that supports subscription (lists or
dictionaries for example).  User-defined objects can support subscription by
defining a @code{__getitem__()} method.

For built-in objects, there are two types of objects that support subscription:

If the primary is a mapping, the expression list must evaluate to an object
whose value is one of the keys of the mapping, and the subscription selects the
value in the mapping that corresponds to that key.  (The expression list is a
tuple except if it has exactly one item.)

If the primary is a sequence, the expression list must evaluate to an integer
or a slice (as discussed in the following section).

The formal syntax makes no special provision for negative indices in
sequences; however, built-in sequences all provide a @code{__getitem__()}
method that interprets negative indices by adding the length of the sequence
to the index (so that @code{x[-1]} selects the last item of @code{x}).  The
resulting value must be a nonnegative integer less than the number of items in
the sequence, and the subscription selects the item whose index is that value
(counting from zero). Since the support for negative indices and slicing
occurs in the object’s @code{__getitem__()} method, subclasses overriding
this method will need to explicitly add that support.

@geindex character
@geindex string; item

A string’s items are characters.  A character is not a separate data type but a
string of exactly one character.

Subscription of certain @ref{11ae,,classes} or @ref{11af,,types}
creates a @ref{268,,generic alias}.
In this case, user-defined classes can support subscription by providing a
@code{__class_getitem__()} classmethod.

@node Slicings,Calls,Subscriptions,Primaries
@anchor{reference/expressions id7}@anchor{18c0}@anchor{reference/expressions slicings}@anchor{18c1}
@subsubsection Slicings


@geindex slicing
@geindex slice
@geindex ; (colon); slicing
@geindex @comma{} (comma); slicing

@geindex object; sequence
@geindex object; string
@geindex object; tuple
@geindex object; list

A slicing selects a range of items in a sequence object (e.g., a string, tuple
or list).  Slicings may be used as expressions or as targets in assignment or
@ref{152e,,del} statements.  The syntax for a slicing:

@example
@anchor{reference/expressions grammar-token-python-grammar-slicing}@anchor{18c2}@anchor{reference/expressions grammar-token-slicing}@anchor{18c3}slicing      ::= primary "[" slice_list "]"
@anchor{reference/expressions grammar-token-python-grammar-slice_list}@anchor{18c4}@anchor{reference/expressions grammar-token-slice-list}@anchor{18c5}slice_list   ::= slice_item ("," slice_item)* [","]
@anchor{reference/expressions grammar-token-python-grammar-slice_item}@anchor{18c6}@anchor{reference/expressions grammar-token-slice-item}@anchor{18c7}slice_item   ::= expression | proper_slice
@anchor{reference/expressions grammar-token-python-grammar-proper_slice}@anchor{18c8}@anchor{reference/expressions grammar-token-proper-slice}@anchor{18c9}proper_slice ::= [lower_bound] ":" [upper_bound] [ ":" [stride] ]
@anchor{reference/expressions grammar-token-python-grammar-lower_bound}@anchor{18ca}@anchor{reference/expressions grammar-token-lower-bound}@anchor{18cb}lower_bound  ::= expression
@anchor{reference/expressions grammar-token-python-grammar-upper_bound}@anchor{18cc}@anchor{reference/expressions grammar-token-upper-bound}@anchor{18cd}upper_bound  ::= expression
@anchor{reference/expressions grammar-token-python-grammar-stride}@anchor{18ce}@anchor{reference/expressions grammar-token-stride}@anchor{18cf}stride       ::= expression
@end example

There is ambiguity in the formal syntax here: anything that looks like an
expression list also looks like a slice list, so any subscription can be
interpreted as a slicing.  Rather than further complicating the syntax, this is
disambiguated by defining that in this case the interpretation as a subscription
takes priority over the interpretation as a slicing (this is the case if the
slice list contains no proper slice).

@geindex start (slice object attribute)
@geindex stop (slice object attribute)
@geindex step (slice object attribute)

The semantics for a slicing are as follows.  The primary is indexed (using the
same @code{__getitem__()} method as
normal subscription) with a key that is constructed from the slice list, as
follows.  If the slice list contains at least one comma, the key is a tuple
containing the conversion of the slice items; otherwise, the conversion of the
lone slice item is the key.  The conversion of a slice item that is an
expression is that expression.  The conversion of a proper slice is a slice
object (see section @ref{1787,,The standard type hierarchy}) whose @code{start},
@code{stop} and @code{step} attributes are the values of the
expressions given as lower bound, upper bound and stride, respectively,
substituting @code{None} for missing expressions.

@geindex object; callable
@geindex call
@geindex argument; call semantics
@geindex () (parentheses); call
@geindex @comma{} (comma); argument list
@geindex = (equals); in function calls

@node Calls,,Slicings,Primaries
@anchor{reference/expressions calls}@anchor{8d6}@anchor{reference/expressions id8}@anchor{18d0}
@subsubsection Calls


A call calls a callable object (e.g., a @ref{18d1,,function}) with a possibly empty
series of @ref{18d2,,arguments}:

@example
@anchor{reference/expressions grammar-token-python-grammar-call}@anchor{18d3}@anchor{reference/expressions grammar-token-call}@anchor{18d4}call                 ::= primary "(" [argument_list [","] | comprehension] ")"
@anchor{reference/expressions grammar-token-python-grammar-argument_list}@anchor{18d5}@anchor{reference/expressions grammar-token-argument-list}@anchor{18d6}argument_list        ::= positional_arguments ["," starred_and_keywords]
                           ["," keywords_arguments]
                         | starred_and_keywords ["," keywords_arguments]
                         | keywords_arguments
@anchor{reference/expressions grammar-token-python-grammar-positional_arguments}@anchor{18d7}@anchor{reference/expressions grammar-token-positional-arguments}@anchor{18d8}positional_arguments ::= positional_item ("," positional_item)*
@anchor{reference/expressions grammar-token-python-grammar-positional_item}@anchor{18d9}@anchor{reference/expressions grammar-token-positional-item}@anchor{18da}positional_item      ::= assignment_expression | "*" expression
@anchor{reference/expressions grammar-token-python-grammar-starred_and_keywords}@anchor{18db}@anchor{reference/expressions grammar-token-starred-and-keywords}@anchor{18dc}starred_and_keywords ::= ("*" expression | keyword_item)
                         ("," "*" expression | "," keyword_item)*
@anchor{reference/expressions grammar-token-python-grammar-keywords_arguments}@anchor{18dd}@anchor{reference/expressions grammar-token-keywords-arguments}@anchor{18de}keywords_arguments   ::= (keyword_item | "**" expression)
                         ("," keyword_item | "," "**" expression)*
@anchor{reference/expressions grammar-token-python-grammar-keyword_item}@anchor{18df}@anchor{reference/expressions grammar-token-keyword-item}@anchor{18e0}keyword_item         ::= identifier "=" expression
@end example

An optional trailing comma may be present after the positional and keyword arguments
but does not affect the semantics.

@geindex parameter; call semantics

The primary must evaluate to a callable object (user-defined functions, built-in
functions, methods of built-in objects, class objects, methods of class
instances, and all objects having a @code{__call__()} method are callable).  All
argument expressions are evaluated before the call is attempted.  Please refer
to section @ref{151c,,Function definitions} for the syntax of formal @ref{18e1,,parameter} lists.

@c XXX update with kwonly args PEP

If keyword arguments are present, they are first converted to positional
arguments, as follows.  First, a list of unfilled slots is created for the
formal parameters.  If there are N positional arguments, they are placed in the
first N slots.  Next, for each keyword argument, the identifier is used to
determine the corresponding slot (if the identifier is the same as the first
formal parameter name, the first slot is used, and so on).  If the slot is
already filled, a @ref{162,,TypeError} exception is raised. Otherwise, the
argument is placed in the slot, filling it (even if the expression is
@code{None}, it fills the slot).  When all arguments have been processed, the slots
that are still unfilled are filled with the corresponding default value from the
function definition.  (Default values are calculated, once, when the function is
defined; thus, a mutable object such as a list or dictionary used as default
value will be shared by all calls that don’t specify an argument value for the
corresponding slot; this should usually be avoided.)  If there are any unfilled
slots for which no default value is specified, a @ref{162,,TypeError} exception is
raised.  Otherwise, the list of filled slots is used as the argument list for
the call.

@strong{CPython implementation detail:} An implementation may provide built-in functions whose positional parameters
do not have names, even if they are ‘named’ for the purpose of documentation,
and which therefore cannot be supplied by keyword.  In CPython, this is the
case for functions implemented in C that use @ref{37f,,PyArg_ParseTuple()} to
parse their arguments.

If there are more positional arguments than there are formal parameter slots, a
@ref{162,,TypeError} exception is raised, unless a formal parameter using the syntax
@code{*identifier} is present; in this case, that formal parameter receives a tuple
containing the excess positional arguments (or an empty tuple if there were no
excess positional arguments).

If any keyword argument does not correspond to a formal parameter name, a
@ref{162,,TypeError} exception is raised, unless a formal parameter using the syntax
@code{**identifier} is present; in this case, that formal parameter receives a
dictionary containing the excess keyword arguments (using the keywords as keys
and the argument values as corresponding values), or a (new) empty dictionary if
there were no excess keyword arguments.

@geindex * (asterisk); in function calls
@geindex unpacking; in function calls

If the syntax @code{*expression} appears in the function call, @code{expression} must
evaluate to an @ref{dcc,,iterable}.  Elements from these iterables are
treated as if they were additional positional arguments.  For the call
@code{f(x1, x2, *y, x3, x4)}, if @emph{y} evaluates to a sequence @emph{y1}, …, @emph{yM},
this is equivalent to a call with M+4 positional arguments @emph{x1}, @emph{x2},
@emph{y1}, …, @emph{yM}, @emph{x3}, @emph{x4}.

A consequence of this is that although the @code{*expression} syntax may appear
@emph{after} explicit keyword arguments, it is processed @emph{before} the
keyword arguments (and any @code{**expression} arguments – see below).  So:

@example
>>> def f(a, b):
...     print(a, b)
...
>>> f(b=1, *(2,))
2 1
>>> f(a=1, *(2,))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() got multiple values for keyword argument 'a'
>>> f(1, *(2,))
1 2
@end example

It is unusual for both keyword arguments and the @code{*expression} syntax to be
used in the same call, so in practice this confusion does not often arise.

@geindex **; in function calls

If the syntax @code{**expression} appears in the function call, @code{expression} must
evaluate to a @ref{d62,,mapping}, the contents of which are treated as
additional keyword arguments.  If a keyword is already present
(as an explicit keyword argument, or from another unpacking),
a @ref{162,,TypeError} exception is raised.

Formal parameters using the syntax @code{*identifier} or @code{**identifier} cannot be
used as positional argument slots or as keyword argument names.


Changed in version 3.5: Function calls accept any number of @code{*} and @code{**} unpackings,
positional arguments may follow iterable unpackings (@code{*}),
and keyword arguments may follow dictionary unpackings (@code{**}).
Originally proposed by 
@geindex Python Enhancement Proposals; PEP 448
PEP 448@footnote{https://www.python.org/dev/peps/pep-0448}.


A call always returns some value, possibly @code{None}, unless it raises an
exception.  How this value is computed depends on the type of the callable
object.

If it is—


@table @asis

@item a user-defined function:

@geindex function; call
@geindex user-defined; function; call
@geindex object; user-defined function
@geindex object; function

The code block for the function is executed, passing it the argument list.  The
first thing the code block will do is bind the formal parameters to the
arguments; this is described in section @ref{151c,,Function definitions}.  When the code block
executes a @ref{4c6,,return} statement, this specifies the return value of the
function call.

@item a built-in function or method:

@geindex function; call
@geindex built-in function; call
@geindex method; call
@geindex built-in method; call
@geindex object; built-in method
@geindex object; built-in function
@geindex object; method
@geindex object; function

The result is up to the interpreter; see @ref{e0d,,Built-in Functions} for the
descriptions of built-in functions and methods.

@item a class object:

@geindex object; class
@geindex class object; call

A new instance of that class is returned.

@item a class instance method:

@geindex object; class instance
@geindex object; instance
@geindex class instance; call

The corresponding user-defined function is called, with an argument list that is
one longer than the argument list of the call: the instance becomes the first
argument.

@item a class instance:

@geindex instance; call
@geindex __call__() (object method)

The class must define a @code{__call__()} method; the effect is then the same as
if that method was called.
@end table

@geindex keyword; await

@node Await expression,The power operator,Primaries,Expressions
@anchor{reference/expressions await}@anchor{4d5}@anchor{reference/expressions await-expression}@anchor{18e2}
@subsection Await expression


Suspend the execution of @ref{3cf,,coroutine} on an @ref{7bd,,awaitable} object.
Can only be used inside a @ref{8cf,,coroutine function}.

@example
@anchor{reference/expressions grammar-token-python-grammar-await_expr}@anchor{18e3}@anchor{reference/expressions grammar-token-await-expr}@anchor{18e4}await_expr ::= "await" primary
@end example


New in version 3.5.


@node The power operator,Unary arithmetic and bitwise operations,Await expression,Expressions
@anchor{reference/expressions power}@anchor{18e5}@anchor{reference/expressions the-power-operator}@anchor{18e6}
@subsection The power operator


@geindex power; operation
@geindex operator; **

The power operator binds more tightly than unary operators on its left; it binds
less tightly than unary operators on its right.  The syntax is:

@example
@anchor{reference/expressions grammar-token-python-grammar-power}@anchor{18e7}@anchor{reference/expressions grammar-token-power}@anchor{18e8}power ::= (await_expr | primary) ["**" u_expr]
@end example

Thus, in an unparenthesized sequence of power and unary operators, the operators
are evaluated from right to left (this does not constrain the evaluation order
for the operands): @code{-1**2} results in @code{-1}.

The power operator has the same semantics as the built-in @ref{4cd,,pow()} function,
when called with two arguments: it yields its left argument raised to the power
of its right argument.  The numeric arguments are first converted to a common
type, and the result is of that type.

For int operands, the result has the same type as the operands unless the second
argument is negative; in that case, all arguments are converted to float and a
float result is delivered. For example, @code{10**2} returns @code{100}, but
@code{10**-2} returns @code{0.01}.

Raising @code{0.0} to a negative power results in a @ref{43c,,ZeroDivisionError}.
Raising a negative number to a fractional power results in a @ref{33b,,complex}
number. (In earlier versions it raised a @ref{17c,,ValueError}.)

This operation can be customized using the special @code{__pow__()} method.

@node Unary arithmetic and bitwise operations,Binary arithmetic operations,The power operator,Expressions
@anchor{reference/expressions unary}@anchor{18e9}@anchor{reference/expressions unary-arithmetic-and-bitwise-operations}@anchor{18ea}
@subsection Unary arithmetic and bitwise operations


@geindex unary; arithmetic; operation
@geindex unary; bitwise; operation

All unary arithmetic and bitwise operations have the same priority:

@example
@anchor{reference/expressions grammar-token-python-grammar-u_expr}@anchor{18eb}@anchor{reference/expressions grammar-token-u-expr}@anchor{18ec}u_expr ::= power | "-" u_expr | "+" u_expr | "~" u_expr
@end example

@geindex negation
@geindex minus
@geindex operator; - (minus)
@geindex - (minus); unary operator

The unary @code{-} (minus) operator yields the negation of its numeric argument; the
operation can be overridden with the @code{__neg__()} special method.

@geindex plus
@geindex operator; + (plus)
@geindex + (plus); unary operator

The unary @code{+} (plus) operator yields its numeric argument unchanged; the
operation can be overridden with the @code{__pos__()} special method.

@geindex inversion
@geindex operator; ~ (tilde)

The unary @code{~} (invert) operator yields the bitwise inversion of its integer
argument.  The bitwise inversion of @code{x} is defined as @code{-(x+1)}.  It only
applies to integral numbers or to custom objects that override the
@code{__invert__()} special method.

@geindex exception; TypeError

In all three cases, if the argument does not have the proper type, a
@ref{162,,TypeError} exception is raised.

@node Binary arithmetic operations,Shifting operations,Unary arithmetic and bitwise operations,Expressions
@anchor{reference/expressions binary}@anchor{18ed}@anchor{reference/expressions binary-arithmetic-operations}@anchor{18ee}
@subsection Binary arithmetic operations


@geindex binary; arithmetic; operation

The binary arithmetic operations have the conventional priority levels.  Note
that some of these operations also apply to certain non-numeric types.  Apart
from the power operator, there are only two levels, one for multiplicative
operators and one for additive operators:

@example
@anchor{reference/expressions grammar-token-python-grammar-m_expr}@anchor{18ef}@anchor{reference/expressions grammar-token-m-expr}@anchor{18f0}m_expr ::= u_expr | m_expr "*" u_expr | m_expr "@@" m_expr |
           m_expr "//" u_expr | m_expr "/" u_expr |
           m_expr "%" u_expr
@anchor{reference/expressions grammar-token-python-grammar-a_expr}@anchor{18f1}@anchor{reference/expressions grammar-token-a-expr}@anchor{18f2}a_expr ::= m_expr | a_expr "+" m_expr | a_expr "-" m_expr
@end example

@geindex multiplication
@geindex operator; * (asterisk)

The @code{*} (multiplication) operator yields the product of its arguments.  The
arguments must either both be numbers, or one argument must be an integer and
the other must be a sequence. In the former case, the numbers are converted to a
common type and then multiplied together.  In the latter case, sequence
repetition is performed; a negative repetition factor yields an empty sequence.

This operation can be customized using the special @code{__mul__()} and
@code{__rmul__()} methods.

@geindex matrix multiplication
@geindex operator; @@ (at)

The @code{@@} (at) operator is intended to be used for matrix multiplication.  No
builtin Python types implement this operator.


New in version 3.5.


@geindex exception; ZeroDivisionError
@geindex division
@geindex operator; / (slash)
@geindex operator; //

The @code{/} (division) and @code{//} (floor division) operators yield the quotient of
their arguments.  The numeric arguments are first converted to a common type.
Division of integers yields a float, while floor division of integers results in an
integer; the result is that of mathematical division with the ‘floor’ function
applied to the result.  Division by zero raises the @ref{43c,,ZeroDivisionError}
exception.

This operation can be customized using the special @code{__truediv__()} and
@code{__floordiv__()} methods.

@geindex modulo
@geindex operator; % (percent)

The @code{%} (modulo) operator yields the remainder from the division of the first
argument by the second.  The numeric arguments are first converted to a common
type.  A zero right argument raises the @ref{43c,,ZeroDivisionError} exception.  The
arguments may be floating point numbers, e.g., @code{3.14%0.7} equals @code{0.34}
(since @code{3.14} equals @code{4*0.7 + 0.34}.)  The modulo operator always yields a
result with the same sign as its second operand (or zero); the absolute value of
the result is strictly smaller than the absolute value of the second operand
@footnote{@w{(1)} 
While @code{abs(x%y) < abs(y)} is true mathematically, for floats it may not be
true numerically due to roundoff.  For example, and assuming a platform on which
a Python float is an IEEE 754 double-precision number, in order that @code{-1e-100 %
1e100} have the same sign as @code{1e100}, the computed result is @code{-1e-100 +
1e100}, which is numerically exactly equal to @code{1e100}.  The function
@ref{18f3,,math.fmod()} returns a result whose sign matches the sign of the
first argument instead, and so returns @code{-1e-100} in this case. Which approach
is more appropriate depends on the application.
}.

The floor division and modulo operators are connected by the following
identity: @code{x == (x//y)*y + (x%y)}.  Floor division and modulo are also
connected with the built-in function @ref{4a8,,divmod()}: @code{divmod(x, y) == (x//y,
x%y)}. @footnote{@w{(2)} 
If x is very close to an exact integer multiple of y, it’s possible for
@code{x//y} to be one larger than @code{(x-x%y)//y} due to rounding.  In such
cases, Python returns the latter result, in order to preserve that
@code{divmod(x,y)[0] * y + x % y} be very close to @code{x}.
}.

In addition to performing the modulo operation on numbers, the @code{%} operator is
also overloaded by string objects to perform old-style string formatting (also
known as interpolation).  The syntax for string formatting is described in the
Python Library Reference, section @ref{14e4,,printf-style String Formatting}.

The @emph{modulo} operation can be customized using the special @code{__mod__()} method.

The floor division operator, the modulo operator, and the @ref{4a8,,divmod()}
function are not defined for complex numbers.  Instead, convert to a floating
point number using the @ref{1582,,abs()} function if appropriate.

@geindex addition
@geindex operator; + (plus)
@geindex + (plus); binary operator

The @code{+} (addition) operator yields the sum of its arguments.  The arguments
must either both be numbers or both be sequences of the same type.  In the
former case, the numbers are converted to a common type and then added together.
In the latter case, the sequences are concatenated.

This operation can be customized using the special @code{__add__()} and
@code{__radd__()} methods.

@geindex subtraction
@geindex operator; - (minus)
@geindex - (minus); binary operator

The @code{-} (subtraction) operator yields the difference of its arguments.  The
numeric arguments are first converted to a common type.

This operation can be customized using the special @code{__sub__()} method.

@node Shifting operations,Binary bitwise operations,Binary arithmetic operations,Expressions
@anchor{reference/expressions shifting}@anchor{18f4}@anchor{reference/expressions shifting-operations}@anchor{18f5}
@subsection Shifting operations


@geindex shifting; operation
@geindex operator; <<
@geindex operator; >>

The shifting operations have lower priority than the arithmetic operations:

@example
@anchor{reference/expressions grammar-token-python-grammar-shift_expr}@anchor{18f6}@anchor{reference/expressions grammar-token-shift-expr}@anchor{18f7}shift_expr ::= a_expr | shift_expr ("<<" | ">>") a_expr
@end example

These operators accept integers as arguments.  They shift the first argument to
the left or right by the number of bits given by the second argument.

This operation can be customized using the special @code{__lshift__()} and
@code{__rshift__()} methods.

@geindex exception; ValueError

A right shift by @emph{n} bits is defined as floor division by @code{pow(2,n)}.  A left
shift by @emph{n} bits is defined as multiplication with @code{pow(2,n)}.

@node Binary bitwise operations,Comparisons,Shifting operations,Expressions
@anchor{reference/expressions binary-bitwise-operations}@anchor{18f8}@anchor{reference/expressions bitwise}@anchor{18f9}
@subsection Binary bitwise operations


@geindex binary; bitwise; operation

Each of the three bitwise operations has a different priority level:

@example
@anchor{reference/expressions grammar-token-python-grammar-and_expr}@anchor{18fa}@anchor{reference/expressions grammar-token-and-expr}@anchor{18fb}and_expr ::= shift_expr | and_expr "&" shift_expr
@anchor{reference/expressions grammar-token-python-grammar-xor_expr}@anchor{18fc}@anchor{reference/expressions grammar-token-xor-expr}@anchor{18fd}xor_expr ::= and_expr | xor_expr "^" and_expr
@anchor{reference/expressions grammar-token-python-grammar-or_expr}@anchor{18fe}@anchor{reference/expressions grammar-token-or-expr}@anchor{18ff}or_expr  ::= xor_expr | or_expr "|" xor_expr
@end example

@geindex bitwise; and
@geindex operator; & (ampersand)

The @code{&} operator yields the bitwise AND of its arguments, which must be
integers or one of them must be a custom object overriding @code{__and__()} or
@code{__rand__()} special methods.

@geindex bitwise; xor
@geindex exclusive; or
@geindex operator; ^ (caret)

The @code{^} operator yields the bitwise XOR (exclusive OR) of its arguments, which
must be integers or one of them must be a custom object overriding @code{__xor__()} or
@code{__rxor__()} special methods.

@geindex bitwise; or
@geindex inclusive; or
@geindex operator; | (vertical bar)

The @code{|} operator yields the bitwise (inclusive) OR of its arguments, which
must be integers or one of them must be a custom object overriding @code{__or__()} or
@code{__ror__()} special methods.

@node Comparisons,Boolean operations,Binary bitwise operations,Expressions
@anchor{reference/expressions comparisons}@anchor{1900}@anchor{reference/expressions id11}@anchor{1901}
@subsection Comparisons


@geindex comparison
@geindex C; language
@geindex operator; < (less)
@geindex operator; > (greater)
@geindex operator; <=
@geindex operator; >=
@geindex operator; ==
@geindex operator; !=

Unlike C, all comparison operations in Python have the same priority, which is
lower than that of any arithmetic, shifting or bitwise operation.  Also unlike
C, expressions like @code{a < b < c} have the interpretation that is conventional
in mathematics:

@example
@anchor{reference/expressions grammar-token-python-grammar-comparison}@anchor{1902}@anchor{reference/expressions grammar-token-comparison}@anchor{1903}comparison    ::= or_expr (comp_operator or_expr)*
@anchor{reference/expressions grammar-token-python-grammar-comp_operator}@anchor{1904}@anchor{reference/expressions grammar-token-comp-operator}@anchor{1905}comp_operator ::= "<" | ">" | "==" | ">=" | "<=" | "!="
                  | "is" ["not"] | ["not"] "in"
@end example

Comparisons yield boolean values: @code{True} or @code{False}. Custom
@emph{rich comparison methods} may return non-boolean values. In this case
Python will call @ref{2f7,,bool()} on such value in boolean contexts.

@geindex chaining; comparisons

Comparisons can be chained arbitrarily, e.g., @code{x < y <= z} is equivalent to
@code{x < y and y <= z}, except that @code{y} is evaluated only once (but in both
cases @code{z} is not evaluated at all when @code{x < y} is found to be false).

Formally, if @emph{a}, @emph{b}, @emph{c}, …, @emph{y}, @emph{z} are expressions and @emph{op1}, @emph{op2}, …,
@emph{opN} are comparison operators, then @code{a op1 b op2 c ... y opN z} is equivalent
to @code{a op1 b and b op2 c and ... y opN z}, except that each expression is
evaluated at most once.

Note that @code{a op1 b op2 c} doesn’t imply any kind of comparison between @emph{a} and
@emph{c}, so that, e.g., @code{x < y > z} is perfectly legal (though perhaps not
pretty).

@menu
* Value comparisons:: 
* Membership test operations:: 
* Identity comparisons:: 

@end menu

@node Value comparisons,Membership test operations,,Comparisons
@anchor{reference/expressions expressions-value-comparisons}@anchor{1cc}@anchor{reference/expressions value-comparisons}@anchor{1906}
@subsubsection Value comparisons


The operators @code{<}, @code{>}, @code{==}, @code{>=}, @code{<=}, and @code{!=} compare the
values of two objects.  The objects do not need to have the same type.

Chapter @ref{1784,,Objects@comma{} values and types} states that objects have a value (in addition to type
and identity).  The value of an object is a rather abstract notion in Python:
For example, there is no canonical access method for an object’s value.  Also,
there is no requirement that the value of an object should be constructed in a
particular way, e.g. comprised of all its data attributes. Comparison operators
implement a particular notion of what the value of an object is.  One can think
of them as defining the value of an object indirectly, by means of their
comparison implementation.

Because all types are (direct or indirect) subtypes of @ref{5a5,,object}, they
inherit the default comparison behavior from @ref{5a5,,object}.  Types can
customize their comparison behavior by implementing
@emph{rich comparison methods} like @code{__lt__()}, described in
@ref{17a2,,Basic customization}.

The default behavior for equality comparison (@code{==} and @code{!=}) is based on
the identity of the objects.  Hence, equality comparison of instances with the
same identity results in equality, and equality comparison of instances with
different identities results in inequality.  A motivation for this default
behavior is the desire that all objects should be reflexive (i.e. @code{x is y}
implies @code{x == y}).

A default order comparison (@code{<}, @code{>}, @code{<=}, and @code{>=}) is not provided;
an attempt raises @ref{162,,TypeError}.  A motivation for this default behavior is
the lack of a similar invariant as for equality.

The behavior of the default equality comparison, that instances with different
identities are always unequal, may be in contrast to what types will need that
have a sensible definition of object value and value-based equality.  Such
types will need to customize their comparison behavior, and in fact, a number
of built-in types have done that.

The following list describes the comparison behavior of the most important
built-in types.


@itemize *

@item 
Numbers of built-in numeric types (@ref{14de,,Numeric Types — int@comma{} float@comma{} complex}) and of the standard
library types @ref{172,,fractions.Fraction} and @ref{248,,decimal.Decimal} can be
compared within and across their types, with the restriction that complex
numbers do not support order comparison.  Within the limits of the types
involved, they compare mathematically (algorithmically) correct without loss
of precision.

The not-a-number values @code{float('NaN')} and @code{decimal.Decimal('NaN')} are
special.  Any ordered comparison of a number to a not-a-number value is false.
A counter-intuitive implication is that not-a-number values are not equal to
themselves.  For example, if @code{x = float('NaN')}, @code{3 < x}, @code{x < 3} and
@code{x == x} are all false, while @code{x != x} is true.  This behavior is
compliant with IEEE 754.

@item 
@code{None} and @code{NotImplemented} are singletons.  
@geindex Python Enhancement Proposals; PEP 8
PEP 8@footnote{https://www.python.org/dev/peps/pep-0008} advises that
comparisons for singletons should always be done with @code{is} or @code{is not},
never the equality operators.

@item 
Binary sequences (instances of @ref{2f0,,bytes} or @ref{2f1,,bytearray}) can be
compared within and across their types.  They compare lexicographically using
the numeric values of their elements.

@item 
Strings (instances of @ref{2ef,,str}) compare lexicographically using the
numerical Unicode code points (the result of the built-in function
@ref{178e,,ord()}) of their characters. @footnote{@w{(3)} 
The Unicode standard distinguishes between @emph{code points}
(e.g. U+0041) and @emph{abstract characters} (e.g. “LATIN CAPITAL LETTER A”).
While most abstract characters in Unicode are only represented using one
code point, there is a number of abstract characters that can in addition be
represented using a sequence of more than one code point.  For example, the
abstract character “LATIN CAPITAL LETTER C WITH CEDILLA” can be represented
as a single @emph{precomposed character} at code position U+00C7, or as a
sequence of a @emph{base character} at code position U+0043 (LATIN CAPITAL
LETTER C), followed by a @emph{combining character} at code position U+0327
(COMBINING CEDILLA).

The comparison operators on strings compare at the level of Unicode code
points. This may be counter-intuitive to humans.  For example,
@code{"\u00C7" == "\u0043\u0327"} is @code{False}, even though both strings
represent the same abstract character “LATIN CAPITAL LETTER C WITH CEDILLA”.

To compare strings at the level of abstract characters (that is, in a way
intuitive to humans), use @ref{19a,,unicodedata.normalize()}.
}

Strings and binary sequences cannot be directly compared.

@item 
Sequences (instances of @ref{3e5,,tuple}, @ref{43a,,list}, or @ref{438,,range}) can
be compared only within each of their types, with the restriction that ranges
do not support order comparison.  Equality comparison across these types
results in inequality, and ordering comparison across these types raises
@ref{162,,TypeError}.

Sequences compare lexicographically using comparison of corresponding
elements.  The built-in containers typically assume identical objects are
equal to themselves.  That lets them bypass equality tests for identical
objects to improve performance and to maintain their internal invariants.

Lexicographical comparison between built-in collections works as follows:


@itemize -

@item 
For two collections to compare equal, they must be of the same type, have
the same length, and each pair of corresponding elements must compare
equal (for example, @code{[1,2] == (1,2)} is false because the type is not the
same).

@item 
Collections that support order comparison are ordered the same as their
first unequal elements (for example, @code{[1,2,x] <= [1,2,y]} has the same
value as @code{x <= y}).  If a corresponding element does not exist, the
shorter collection is ordered first (for example, @code{[1,2] < [1,2,3]} is
true).
@end itemize

@item 
Mappings (instances of @ref{3b0,,dict}) compare equal if and only if they have
equal @cite{(key@comma{} value)} pairs. Equality comparison of the keys and values
enforces reflexivity.

Order comparisons (@code{<}, @code{>}, @code{<=}, and @code{>=}) raise @ref{162,,TypeError}.

@item 
Sets (instances of @ref{370,,set} or @ref{439,,frozenset}) can be compared within
and across their types.

They define order
comparison operators to mean subset and superset tests.  Those relations do
not define total orderings (for example, the two sets @code{@{1,2@}} and @code{@{2,3@}}
are not equal, nor subsets of one another, nor supersets of one
another).  Accordingly, sets are not appropriate arguments for functions
which depend on total ordering (for example, @ref{a7d,,min()}, @ref{a7e,,max()}, and
@ref{704,,sorted()} produce undefined results given a list of sets as inputs).

Comparison of sets enforces reflexivity of its elements.

@item 
Most other built-in types have no comparison methods implemented, so they
inherit the default comparison behavior.
@end itemize

User-defined classes that customize their comparison behavior should follow
some consistency rules, if possible:


@itemize *

@item 
Equality comparison should be reflexive.
In other words, identical objects should compare equal:

@quotation

@code{x is y} implies @code{x == y}
@end quotation

@item 
Comparison should be symmetric.
In other words, the following expressions should have the same result:

@quotation

@code{x == y} and @code{y == x}

@code{x != y} and @code{y != x}

@code{x < y} and @code{y > x}

@code{x <= y} and @code{y >= x}
@end quotation

@item 
Comparison should be transitive.
The following (non-exhaustive) examples illustrate that:

@quotation

@code{x > y and y > z} implies @code{x > z}

@code{x < y and y <= z} implies @code{x < z}
@end quotation

@item 
Inverse comparison should result in the boolean negation.
In other words, the following expressions should have the same result:

@quotation

@code{x == y} and @code{not x != y}

@code{x < y} and @code{not x >= y} (for total ordering)

@code{x > y} and @code{not x <= y} (for total ordering)
@end quotation

The last two expressions apply to totally ordered collections (e.g. to
sequences, but not to sets or mappings). See also the
@ref{abf,,total_ordering()} decorator.

@item 
The @ref{c14,,hash()} result should be consistent with equality.
Objects that are equal should either have the same hash value,
or be marked as unhashable.
@end itemize

Python does not enforce these consistency rules. In fact, the not-a-number
values are an example for not following these rules.
@anchor{reference/expressions in}@anchor{2fc}@anchor{reference/expressions not-in}@anchor{17e1}
@node Membership test operations,Identity comparisons,Value comparisons,Comparisons
@anchor{reference/expressions membership-test-details}@anchor{17e2}@anchor{reference/expressions membership-test-operations}@anchor{1907}
@subsubsection Membership test operations


The operators @ref{2fc,,in} and @ref{17e1,,not in} test for membership.  @code{x in
s} evaluates to @code{True} if @emph{x} is a member of @emph{s}, and @code{False} otherwise.
@code{x not in s} returns the negation of @code{x in s}.  All built-in sequences and
set types support this as well as dictionary, for which @code{in} tests
whether the dictionary has a given key. For container types such as list, tuple,
set, frozenset, dict, or collections.deque, the expression @code{x in y} is equivalent
to @code{any(x is e or x == e for e in y)}.

For the string and bytes types, @code{x in y} is @code{True} if and only if @emph{x} is a
substring of @emph{y}.  An equivalent test is @code{y.find(x) != -1}.  Empty strings are
always considered to be a substring of any other string, so @code{"" in "abc"} will
return @code{True}.

For user-defined classes which define the @code{__contains__()} method, @code{x in
y} returns @code{True} if @code{y.__contains__(x)} returns a true value, and
@code{False} otherwise.

For user-defined classes which do not define @code{__contains__()} but do define
@code{__iter__()}, @code{x in y} is @code{True} if some value @code{z}, for which the
expression @code{x is z or x == z} is true, is produced while iterating over @code{y}.
If an exception is raised during the iteration, it is as if @ref{2fc,,in} raised
that exception.

Lastly, the old-style iteration protocol is tried: if a class defines
@code{__getitem__()}, @code{x in y} is @code{True} if and only if there is a non-negative
integer index @emph{i} such that @code{x is y[i] or x == y[i]}, and no lower integer index
raises the @ref{1066,,IndexError} exception.  (If any other exception is raised, it is as
if @ref{2fc,,in} raised that exception).

@geindex operator; in
@geindex operator; not in
@geindex membership; test
@geindex object; sequence

The operator @ref{17e1,,not in} is defined to have the inverse truth value of
@ref{2fc,,in}.

@geindex operator; is
@geindex operator; is not
@geindex identity; test
@anchor{reference/expressions is}@anchor{2fd}
@node Identity comparisons,,Membership test operations,Comparisons
@anchor{reference/expressions identity-comparisons}@anchor{1908}@anchor{reference/expressions is-not}@anchor{1909}
@subsubsection Identity comparisons


The operators @ref{2fd,,is} and @ref{1909,,is not} test for an object’s identity: @code{x
is y} is true if and only if @emph{x} and @emph{y} are the same object.  An Object’s identity
is determined using the @ref{f80,,id()} function.  @code{x is not y} yields the inverse
truth value. @footnote{@w{(4)} 
Due to automatic garbage-collection, free lists, and the dynamic nature of
descriptors, you may notice seemingly unusual behaviour in certain uses of
the @ref{2fd,,is} operator, like those involving comparisons between instance
methods, or constants.  Check their documentation for more info.
}
@anchor{reference/expressions booleans}@anchor{190a}@anchor{reference/expressions and}@anchor{2f9}@anchor{reference/expressions or}@anchor{2fe}
@node Boolean operations,Assignment expressions<2>,Comparisons,Expressions
@anchor{reference/expressions boolean-operations}@anchor{190b}@anchor{reference/expressions not}@anchor{190c}
@subsection Boolean operations


@geindex Conditional; expression
@geindex Boolean; operation

@example
@anchor{reference/expressions grammar-token-python-grammar-or_test}@anchor{190d}@anchor{reference/expressions grammar-token-or-test}@anchor{190e}or_test  ::= and_test | or_test "or" and_test
@anchor{reference/expressions grammar-token-python-grammar-and_test}@anchor{190f}@anchor{reference/expressions grammar-token-and-test}@anchor{1910}and_test ::= not_test | and_test "and" not_test
@anchor{reference/expressions grammar-token-python-grammar-not_test}@anchor{1911}@anchor{reference/expressions grammar-token-not-test}@anchor{1912}not_test ::= comparison | "not" not_test
@end example

In the context of Boolean operations, and also when expressions are used by
control flow statements, the following values are interpreted as false:
@code{False}, @code{None}, numeric zero of all types, and empty strings and containers
(including strings, tuples, lists, dictionaries, sets and frozensets).  All
other values are interpreted as true.  User-defined objects can customize their
truth value by providing a @code{__bool__()} method.

@geindex operator; not

The operator @ref{190c,,not} yields @code{True} if its argument is false, @code{False}
otherwise.

@geindex operator; and

The expression @code{x and y} first evaluates @emph{x}; if @emph{x} is false, its value is
returned; otherwise, @emph{y} is evaluated and the resulting value is returned.

@geindex operator; or

The expression @code{x or y} first evaluates @emph{x}; if @emph{x} is true, its value is
returned; otherwise, @emph{y} is evaluated and the resulting value is returned.

Note that neither @ref{2f9,,and} nor @ref{2fe,,or} restrict the value and type
they return to @code{False} and @code{True}, but rather return the last evaluated
argument.  This is sometimes useful, e.g., if @code{s} is a string that should be
replaced by a default value if it is empty, the expression @code{s or 'foo'} yields
the desired value.  Because @ref{190c,,not} has to create a new value, it
returns a boolean value regardless of the type of its argument
(for example, @code{not 'foo'} produces @code{False} rather than @code{'@w{'}}.)

@node Assignment expressions<2>,Conditional expressions,Boolean operations,Expressions
@anchor{reference/expressions assignment-expressions}@anchor{1913}
@subsection Assignment expressions


@example
@anchor{reference/expressions grammar-token-python-grammar-assignment_expression}@anchor{1914}@anchor{reference/expressions grammar-token-assignment-expression}@anchor{1915}assignment_expression ::= [identifier ":="] expression
@end example

An assignment expression (sometimes also called a “named expression” or
“walrus”) assigns an @ref{1916,,expression} to an
@ref{170d,,identifier}, while also returning the value of the
@ref{1916,,expression}.

One common use case is when handling matched regular expressions:

@example
if matching := pattern.search(data):
    do_something(matching)
@end example

Or, when processing a file stream in chunks:

@example
while chunk := file.read(9000):
    process(chunk)
@end example


New in version 3.8: See 
@geindex Python Enhancement Proposals; PEP 572
PEP 572@footnote{https://www.python.org/dev/peps/pep-0572} for more details about assignment expressions.


@node Conditional expressions,Lambdas,Assignment expressions<2>,Expressions
@anchor{reference/expressions conditional-expressions}@anchor{1917}@anchor{reference/expressions if-expr}@anchor{1918}
@subsection Conditional expressions


@geindex conditional; expression
@geindex ternary; operator
@geindex if; conditional expression
@geindex else; conditional expression

@example
@anchor{reference/expressions grammar-token-python-grammar-conditional_expression}@anchor{1919}@anchor{reference/expressions grammar-token-conditional-expression}@anchor{191a}conditional_expression ::= or_test ["if" or_test "else" expression]
@anchor{reference/expressions grammar-token-python-grammar-expression}@anchor{1916}@anchor{reference/expressions grammar-token-expression}@anchor{191b}expression             ::= conditional_expression | lambda_expr
@end example

Conditional expressions (sometimes called a “ternary operator”) have the lowest
priority of all Python operations.

The expression @code{x if C else y} first evaluates the condition, @emph{C} rather than @emph{x}.
If @emph{C} is true, @emph{x} is evaluated and its value is returned; otherwise, @emph{y} is
evaluated and its value is returned.

See 
@geindex Python Enhancement Proposals; PEP 308
PEP 308@footnote{https://www.python.org/dev/peps/pep-0308} for more details about conditional expressions.
@anchor{reference/expressions lambdas}@anchor{191c}
@node Lambdas,Expression lists,Conditional expressions,Expressions
@anchor{reference/expressions id14}@anchor{191d}@anchor{reference/expressions lambda}@anchor{e42}
@subsection Lambdas


@geindex lambda; expression
@geindex lambda; form
@geindex anonymous; function
@geindex ; (colon); lambda expression

@example
@anchor{reference/expressions grammar-token-python-grammar-lambda_expr}@anchor{191e}@anchor{reference/expressions grammar-token-lambda-expr}@anchor{191f}lambda_expr ::= "lambda" [parameter_list] ":" expression
@end example

Lambda expressions (sometimes called lambda forms) are used to create anonymous
functions. The expression @code{lambda parameters: expression} yields a function
object.  The unnamed object behaves like a function object defined with:

@example
def <lambda>(parameters):
    return expression
@end example

See section @ref{151c,,Function definitions} for the syntax of parameter lists.  Note that
functions created with lambda expressions cannot contain statements or
annotations.

@node Expression lists,Evaluation order,Lambdas,Expressions
@anchor{reference/expressions expression-lists}@anchor{1920}@anchor{reference/expressions exprlists}@anchor{8d7}
@subsection Expression lists


@geindex expression; list
@geindex @comma{} (comma); expression list

@example
@anchor{reference/expressions grammar-token-python-grammar-expression_list}@anchor{18ae}@anchor{reference/expressions grammar-token-expression-list}@anchor{1921}expression_list    ::= expression ("," expression)* [","]
@anchor{reference/expressions grammar-token-python-grammar-starred_list}@anchor{1922}@anchor{reference/expressions grammar-token-starred-list}@anchor{1923}starred_list       ::= starred_item ("," starred_item)* [","]
@anchor{reference/expressions grammar-token-python-grammar-starred_expression}@anchor{1924}@anchor{reference/expressions grammar-token-starred-expression}@anchor{1925}starred_expression ::= expression | (starred_item ",")* [starred_item]
@anchor{reference/expressions grammar-token-python-grammar-starred_item}@anchor{1926}@anchor{reference/expressions grammar-token-starred-item}@anchor{1927}starred_item       ::= assignment_expression | "*" or_expr
@end example

@geindex object; tuple

Except when part of a list or set display, an expression list
containing at least one comma yields a tuple.  The length of
the tuple is the number of expressions in the list.  The expressions are
evaluated from left to right.

@geindex iterable; unpacking
@geindex * (asterisk); in expression lists

An asterisk @code{*} denotes @emph{iterable unpacking}.  Its operand must be
an @ref{dcc,,iterable}.  The iterable is expanded into a sequence of items,
which are included in the new tuple, list, or set, at the site of
the unpacking.


New in version 3.5: Iterable unpacking in expression lists, originally proposed by 
@geindex Python Enhancement Proposals; PEP 448
PEP 448@footnote{https://www.python.org/dev/peps/pep-0448}.


@geindex trailing; comma

The trailing comma is required only to create a single tuple (a.k.a. a
@emph{singleton}); it is optional in all other cases.  A single expression without a
trailing comma doesn’t create a tuple, but rather yields the value of that
expression. (To create an empty tuple, use an empty pair of parentheses:
@code{()}.)

@node Evaluation order,Operator precedence,Expression lists,Expressions
@anchor{reference/expressions evalorder}@anchor{1928}@anchor{reference/expressions evaluation-order}@anchor{1929}
@subsection Evaluation order


@geindex evaluation; order

Python evaluates expressions from left to right.  Notice that while evaluating
an assignment, the right-hand side is evaluated before the left-hand side.

In the following lines, expressions will be evaluated in the arithmetic order of
their suffixes:

@example
expr1, expr2, expr3, expr4
(expr1, expr2, expr3, expr4)
@{expr1: expr2, expr3: expr4@}
expr1 + expr2 * (expr3 - expr4)
expr1(expr2, expr3, *expr4, **expr5)
expr3, expr4 = expr1, expr2
@end example

@node Operator precedence,,Evaluation order,Expressions
@anchor{reference/expressions operator-precedence}@anchor{192a}@anchor{reference/expressions operator-summary}@anchor{192b}
@subsection Operator precedence


@geindex operator; precedence

The following table summarizes the operator precedence in Python, from highest
precedence (most binding) to lowest precedence (least binding).  Operators in
the same box have the same precedence.  Unless the syntax is explicitly given,
operators are binary.  Operators in the same box group left to right (except for
exponentiation, which groups from right to left).

Note that comparisons, membership tests, and identity tests, all have the same
precedence and have a left-to-right chaining feature as described in the
@ref{1900,,Comparisons} section.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Operator

@tab

Description

@item

@code{(expressions...)},

@code{[expressions...]},
@code{@{key: value...@}},
@code{@{expressions...@}}

@tab

Binding or parenthesized
expression,
list display,
dictionary display,
set display

@item

@code{x[index]}, @code{x[index:index]},
@code{x(arguments...)}, @code{x.attribute}

@tab

Subscription, slicing,
call, attribute reference

@item

@ref{4d5,,await} @code{x}

@tab

Await expression

@item

@code{**}

@tab

Exponentiation @footnote{@w{(5)} 
The power operator @code{**} binds less tightly than an arithmetic or
bitwise unary operator on its right, that is, @code{2**-1} is @code{0.5}.
}

@item

@code{+x}, @code{-x}, @code{~x}

@tab

Positive, negative, bitwise NOT

@item

@code{*}, @code{@@}, @code{/}, @code{//}, @code{%}

@tab

Multiplication, matrix
multiplication, division, floor
division, remainder @footnote{@w{(6)} 
The @code{%} operator is also used for string formatting; the same
precedence applies.
}

@item

@code{+}, @code{-}

@tab

Addition and subtraction

@item

@code{<<}, @code{>>}

@tab

Shifts

@item

@code{&}

@tab

Bitwise AND

@item

@code{^}

@tab

Bitwise XOR

@item

@code{|}

@tab

Bitwise OR

@item

@ref{2fc,,in}, @ref{17e1,,not in},
@ref{2fd,,is}, @ref{1909,,is not}, @code{<},
@code{<=}, @code{>}, @code{>=}, @code{!=}, @code{==}

@tab

Comparisons, including membership
tests and identity tests

@item

@ref{190c,,not} @code{x}

@tab

Boolean NOT

@item

@ref{2f9,,and}

@tab

Boolean AND

@item

@ref{2fe,,or}

@tab

Boolean OR

@item

@ref{1918,,if} – @code{else}

@tab

Conditional expression

@item

@ref{e42,,lambda}

@tab

Lambda expression

@item

@code{:=}

@tab

Assignment expression

@end multitable


@node Simple statements,Compound statements,Expressions,The Python Language Reference
@anchor{reference/simple_stmts doc}@anchor{192c}@anchor{reference/simple_stmts simple}@anchor{192d}@anchor{reference/simple_stmts simple-statements}@anchor{192e}
@section Simple statements


@geindex simple; statement

A simple statement is comprised within a single logical line. Several simple
statements may occur on a single line separated by semicolons.  The syntax for
simple statements is:

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-simple_stmt}@anchor{192f}@anchor{reference/simple_stmts grammar-token-simple-stmt}@anchor{1930}simple_stmt ::= expression_stmt
                | assert_stmt
                | assignment_stmt
                | augmented_assignment_stmt
                | annotated_assignment_stmt
                | pass_stmt
                | del_stmt
                | return_stmt
                | yield_stmt
                | raise_stmt
                | break_stmt
                | continue_stmt
                | import_stmt
                | future_stmt
                | global_stmt
                | nonlocal_stmt
@end example

@menu
* Expression statements:: 
* Assignment statements:: 
* The assert statement:: 
* The pass statement:: 
* The del statement: The del statement<2>. 
* The return statement:: 
* The yield statement:: 
* The raise statement:: 
* The break statement:: 
* The continue statement:: 
* The import statement:: 
* The global statement:: 
* The nonlocal statement:: 

@end menu

@node Expression statements,Assignment statements,,Simple statements
@anchor{reference/simple_stmts expression-statements}@anchor{1931}@anchor{reference/simple_stmts exprstmts}@anchor{1932}
@subsection Expression statements


@geindex expression; statement
@geindex expression; list

@geindex expression; list

Expression statements are used (mostly interactively) to compute and write a
value, or (usually) to call a procedure (a function that returns no meaningful
result; in Python, procedures return the value @code{None}).  Other uses of
expression statements are allowed and occasionally useful.  The syntax for an
expression statement is:

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-expression_stmt}@anchor{1933}@anchor{reference/simple_stmts grammar-token-expression-stmt}@anchor{1934}expression_stmt ::= starred_expression
@end example

An expression statement evaluates the expression list (which may be a single
expression).

@geindex built-in function; repr
@geindex object; None
@geindex string; conversion
@geindex output
@geindex standard; output
@geindex writing; values
@geindex procedure; call

In interactive mode, if the value is not @code{None}, it is converted to a string
using the built-in @ref{27d,,repr()} function and the resulting string is written to
standard output on a line by itself (except if the result is @code{None}, so that
procedure calls do not cause any output.)

@node Assignment statements,The assert statement,Expression statements,Simple statements
@anchor{reference/simple_stmts assignment}@anchor{182e}@anchor{reference/simple_stmts assignment-statements}@anchor{1935}
@subsection Assignment statements


@geindex = (equals); assignment statement
@geindex assignment; statement
@geindex binding; name
@geindex rebinding; name
@geindex object; mutable
@geindex attribute; assignment

Assignment statements are used to (re)bind names to values and to modify
attributes or items of mutable objects:

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-assignment_stmt}@anchor{1936}@anchor{reference/simple_stmts grammar-token-assignment-stmt}@anchor{1937}assignment_stmt ::= (target_list "=")+ (starred_expression | yield_expression)
@anchor{reference/simple_stmts grammar-token-python-grammar-target_list}@anchor{1938}@anchor{reference/simple_stmts grammar-token-target-list}@anchor{1939}target_list     ::= target ("," target)* [","]
@anchor{reference/simple_stmts grammar-token-python-grammar-target}@anchor{193a}@anchor{reference/simple_stmts grammar-token-target}@anchor{193b}target          ::= identifier
                    | "(" [target_list] ")"
                    | "[" [target_list] "]"
                    | attributeref
                    | subscription
                    | slicing
                    | "*" target
@end example

(See section @ref{18b6,,Primaries} for the syntax definitions for @emph{attributeref},
@emph{subscription}, and @emph{slicing}.)

An assignment statement evaluates the expression list (remember that this can be
a single expression or a comma-separated list, the latter yielding a tuple) and
assigns the single resulting object to each of the target lists, from left to
right.

@geindex target
@geindex target; list

Assignment is defined recursively depending on the form of the target (list).
When a target is part of a mutable object (an attribute reference, subscription
or slicing), the mutable object must ultimately perform the assignment and
decide about its validity, and may raise an exception if the assignment is
unacceptable.  The rules observed by various types and the exceptions raised are
given with the definition of the object types (see section @ref{1787,,The standard type hierarchy}).

@geindex target; list; assignment
@geindex @comma{} (comma); in target list
@geindex * (asterisk); in assignment target list
@geindex [] (square brackets); in assignment target list
@geindex () (parentheses); in assignment target list

Assignment of an object to a target list, optionally enclosed in parentheses or
square brackets, is recursively defined as follows.


@itemize *

@item 
If the target list is a single target with no trailing comma,
optionally in parentheses, the object is assigned to that target.

@item 
Else: The object must be an iterable with the same number of
items as there are targets in the target list, and the items are assigned,
from left to right, to the corresponding targets.


@itemize *

@item 
If the target list contains one target prefixed with an asterisk, called a
“starred” target: The object must be an iterable with at least as many items
as there are targets in the target list, minus one.  The first items of the
iterable are assigned, from left to right, to the targets before the starred
target.  The final items of the iterable are assigned to the targets after
the starred target.  A list of the remaining items in the iterable is then
assigned to the starred target (the list can be empty).

@item 
Else: The object must be an iterable with the same number of items as there
are targets in the target list, and the items are assigned, from left to
right, to the corresponding targets.
@end itemize
@end itemize

Assignment of an object to a single target is recursively defined as follows.


@itemize *

@item 
If the target is an identifier (name):


@itemize *

@item 
If the name does not occur in a @ref{1504,,global} or @ref{e4d,,nonlocal}
statement in the current code block: the name is bound to the object in the
current local namespace.

@item 
Otherwise: the name is bound to the object in the global namespace or the
outer namespace determined by @ref{e4d,,nonlocal}, respectively.
@end itemize

@geindex destructor

The name is rebound if it was already bound.  This may cause the reference
count for the object previously bound to the name to reach zero, causing the
object to be deallocated and its destructor (if it has one) to be called.

@geindex attribute; assignment

@item 
If the target is an attribute reference: The primary expression in the
reference is evaluated.  It should yield an object with assignable attributes;
if this is not the case, @ref{162,,TypeError} is raised.  That object is then
asked to assign the assigned object to the given attribute; if it cannot
perform the assignment, it raises an exception (usually but not necessarily
@ref{163,,AttributeError}).
@anchor{reference/simple_stmts attr-target-note}@anchor{193c}
Note: If the object is a class instance and the attribute reference occurs on
both sides of the assignment operator, the right-hand side expression, @code{a.x} can access
either an instance attribute or (if no instance attribute exists) a class
attribute.  The left-hand side target @code{a.x} is always set as an instance attribute,
creating it if necessary.  Thus, the two occurrences of @code{a.x} do not
necessarily refer to the same attribute: if the right-hand side expression refers to a
class attribute, the left-hand side creates a new instance attribute as the target of the
assignment:

@example
class Cls:
    x = 3             # class variable
inst = Cls()
inst.x = inst.x + 1   # writes inst.x as 4 leaving Cls.x as 3
@end example

This description does not necessarily apply to descriptor attributes, such as
properties created with @ref{501,,property()}.

@geindex subscription; assignment
@geindex object; mutable

@item 
If the target is a subscription: The primary expression in the reference is
evaluated.  It should yield either a mutable sequence object (such as a list)
or a mapping object (such as a dictionary).  Next, the subscript expression is
evaluated.

@geindex object; sequence
@geindex object; list

If the primary is a mutable sequence object (such as a list), the subscript
must yield an integer.  If it is negative, the sequence’s length is added to
it.  The resulting value must be a nonnegative integer less than the
sequence’s length, and the sequence is asked to assign the assigned object to
its item with that index.  If the index is out of range, @ref{1066,,IndexError} is
raised (assignment to a subscripted sequence cannot add new items to a list).

@geindex object; mapping
@geindex object; dictionary

If the primary is a mapping object (such as a dictionary), the subscript must
have a type compatible with the mapping’s key type, and the mapping is then
asked to create a key/datum pair which maps the subscript to the assigned
object.  This can either replace an existing key/value pair with the same key
value, or insert a new key/value pair (if no key with the same value existed).

For user-defined objects, the @code{__setitem__()} method is called with
appropriate arguments.

@geindex slicing; assignment

@item 
If the target is a slicing: The primary expression in the reference is
evaluated.  It should yield a mutable sequence object (such as a list).  The
assigned object should be a sequence object of the same type.  Next, the lower
and upper bound expressions are evaluated, insofar they are present; defaults
are zero and the sequence’s length.  The bounds should evaluate to integers.
If either bound is negative, the sequence’s length is added to it.  The
resulting bounds are clipped to lie between zero and the sequence’s length,
inclusive.  Finally, the sequence object is asked to replace the slice with
the items of the assigned sequence.  The length of the slice may be different
from the length of the assigned sequence, thus changing the length of the
target sequence, if the target sequence allows it.
@end itemize

@strong{CPython implementation detail:} In the current implementation, the syntax for targets is taken to be the same
as for expressions, and invalid syntax is rejected during the code generation
phase, causing less detailed error messages.

Although the definition of assignment implies that overlaps between the
left-hand side and the right-hand side are ‘simultaneous’ (for example @code{a, b =
b, a} swaps two variables), overlaps @emph{within} the collection of assigned-to
variables occur left-to-right, sometimes resulting in confusion.  For instance,
the following program prints @code{[0, 2]}:

@example
x = [0, 1]
i = 0
i, x[i] = 1, 2         # i is updated, then x[i] is updated
print(x)
@end example


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3132

@item PEP 3132@footnote{https://www.python.org/dev/peps/pep-3132} - Extended Iterable Unpacking

The specification for the @code{*target} feature.
@end table


@menu
* Augmented assignment statements:: 
* Annotated assignment statements:: 

@end menu

@node Augmented assignment statements,Annotated assignment statements,,Assignment statements
@anchor{reference/simple_stmts augassign}@anchor{193d}@anchor{reference/simple_stmts augmented-assignment-statements}@anchor{193e}
@subsubsection Augmented assignment statements


@geindex augmented; assignment
@geindex statement; assignment@comma{} augmented
@geindex +=; augmented assignment
@geindex -=; augmented assignment
@geindex *=; augmented assignment
@geindex /=; augmented assignment
@geindex %=; augmented assignment
@geindex &=; augmented assignment
@geindex ^=; augmented assignment
@geindex |=; augmented assignment
@geindex **=; augmented assignment
@geindex //=; augmented assignment
@geindex >>=; augmented assignment
@geindex <<=; augmented assignment

Augmented assignment is the combination, in a single statement, of a binary
operation and an assignment statement:

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-augmented_assignment_stmt}@anchor{193f}@anchor{reference/simple_stmts grammar-token-augmented-assignment-stmt}@anchor{1940}augmented_assignment_stmt ::= augtarget augop (expression_list | yield_expression)
@anchor{reference/simple_stmts grammar-token-python-grammar-augtarget}@anchor{1941}@anchor{reference/simple_stmts grammar-token-augtarget}@anchor{1942}augtarget                 ::= identifier | attributeref | subscription | slicing
@anchor{reference/simple_stmts grammar-token-python-grammar-augop}@anchor{1943}@anchor{reference/simple_stmts grammar-token-augop}@anchor{1944}augop                     ::= "+=" | "-=" | "*=" | "@@=" | "/=" | "//=" | "%=" | "**="
                              | ">>=" | "<<=" | "&=" | "^=" | "|="
@end example

(See section @ref{18b6,,Primaries} for the syntax definitions of the last three
symbols.)

An augmented assignment evaluates the target (which, unlike normal assignment
statements, cannot be an unpacking) and the expression list, performs the binary
operation specific to the type of assignment on the two operands, and assigns
the result to the original target.  The target is only evaluated once.

An augmented assignment expression like @code{x += 1} can be rewritten as @code{x = x +
1} to achieve a similar, but not exactly equal effect. In the augmented
version, @code{x} is only evaluated once. Also, when possible, the actual operation
is performed @emph{in-place}, meaning that rather than creating a new object and
assigning that to the target, the old object is modified instead.

Unlike normal assignments, augmented assignments evaluate the left-hand side
@emph{before} evaluating the right-hand side.  For example, @code{a[i] += f(x)} first
looks-up @code{a[i]}, then it evaluates @code{f(x)} and performs the addition, and
lastly, it writes the result back to @code{a[i]}.

With the exception of assigning to tuples and multiple targets in a single
statement, the assignment done by augmented assignment statements is handled the
same way as normal assignments. Similarly, with the exception of the possible
@emph{in-place} behavior, the binary operation performed by augmented assignment is
the same as the normal binary operations.

For targets which are attribute references, the same @ref{193c,,caveat about class and instance attributes} applies as for regular assignments.

@node Annotated assignment statements,,Augmented assignment statements,Assignment statements
@anchor{reference/simple_stmts annassign}@anchor{1945}@anchor{reference/simple_stmts annotated-assignment-statements}@anchor{1946}
@subsubsection Annotated assignment statements


@geindex annotated; assignment
@geindex statement; assignment@comma{} annotated
@geindex ; (colon); annotated variable

@ref{8af,,Annotation} assignment is the combination, in a single
statement, of a variable or attribute annotation and an optional assignment statement:

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-annotated_assignment_stmt}@anchor{1947}@anchor{reference/simple_stmts grammar-token-annotated-assignment-stmt}@anchor{1948}annotated_assignment_stmt ::= augtarget ":" expression
                              ["=" (starred_expression | yield_expression)]
@end example

The difference from normal @ref{182e,,Assignment statements} is that only single target is allowed.

For simple names as assignment targets, if in class or module scope,
the annotations are evaluated and stored in a special class or module
attribute @code{__annotations__}
that is a dictionary mapping from variable names (mangled if private) to
evaluated annotations. This attribute is writable and is automatically
created at the start of class or module body execution, if annotations
are found statically.

For expressions as assignment targets, the annotations are evaluated if
in class or module scope, but not stored.

If a name is annotated in a function scope, then this name is local for
that scope. Annotations are never evaluated and stored in function scopes.

If the right hand side is present, an annotated
assignment performs the actual assignment before evaluating annotations
(where applicable). If the right hand side is not present for an expression
target, then the interpreter evaluates the target except for the last
@code{__setitem__()} or @code{__setattr__()} call.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 526

@item PEP 526@footnote{https://www.python.org/dev/peps/pep-0526} - Syntax for Variable Annotations

The proposal that added syntax for annotating the types of variables
(including class variables and instance variables), instead of expressing
them through comments.

@geindex Python Enhancement Proposals; PEP 484

@item PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} - Type hints

The proposal that added the @ref{120,,typing} module to provide a standard
syntax for type annotations that can be used in static analysis tools and
IDEs.
@end table



Changed in version 3.8: Now annotated assignments allow same expressions in the right hand side as
the regular assignments. Previously, some expressions (like un-parenthesized
tuple expressions) caused a syntax error.


@node The assert statement,The pass statement,Assignment statements,Simple statements
@anchor{reference/simple_stmts assert}@anchor{46d}@anchor{reference/simple_stmts the-assert-statement}@anchor{1949}
@subsection The @code{assert} statement


@geindex statement; assert
@geindex debugging; assertions
@geindex @comma{} (comma); expression list

Assert statements are a convenient way to insert debugging assertions into a
program:

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-assert_stmt}@anchor{194a}@anchor{reference/simple_stmts grammar-token-assert-stmt}@anchor{194b}assert_stmt ::= "assert" expression ["," expression]
@end example

The simple form, @code{assert expression}, is equivalent to

@example
if __debug__:
    if not expression: raise AssertionError
@end example

The extended form, @code{assert expression1, expression2}, is equivalent to

@example
if __debug__:
    if not expression1: raise AssertionError(expression2)
@end example

@geindex __debug__
@geindex exception; AssertionError

These equivalences assume that @ref{257,,__debug__} and @ref{46e,,AssertionError} refer to
the built-in variables with those names.  In the current implementation, the
built-in variable @ref{257,,__debug__} is @code{True} under normal circumstances,
@code{False} when optimization is requested (command line option @ref{912,,-O}).  The current
code generator emits no code for an assert statement when optimization is
requested at compile time.  Note that it is unnecessary to include the source
code for the expression that failed in the error message; it will be displayed
as part of the stack trace.

Assignments to @ref{257,,__debug__} are illegal.  The value for the built-in variable
is determined when the interpreter starts.

@node The pass statement,The del statement<2>,The assert statement,Simple statements
@anchor{reference/simple_stmts pass}@anchor{14ff}@anchor{reference/simple_stmts the-pass-statement}@anchor{194c}
@subsection The @code{pass} statement


@geindex statement; pass
@geindex null; operation
@geindex null; operation

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-pass_stmt}@anchor{194d}@anchor{reference/simple_stmts grammar-token-pass-stmt}@anchor{194e}pass_stmt ::= "pass"
@end example

@ref{14ff,,pass} is a null operation — when it is executed, nothing happens.
It is useful as a placeholder when a statement is required syntactically, but no
code needs to be executed, for example:

@example
def f(arg): pass    # a function that does nothing (yet)

class C: pass       # a class with no methods (yet)
@end example

@node The del statement<2>,The return statement,The pass statement,Simple statements
@anchor{reference/simple_stmts del}@anchor{152e}@anchor{reference/simple_stmts the-del-statement}@anchor{194f}
@subsection The @code{del} statement


@geindex statement; del
@geindex deletion; target
@geindex deletion; target; list

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-del_stmt}@anchor{1950}@anchor{reference/simple_stmts grammar-token-del-stmt}@anchor{1951}del_stmt ::= "del" target_list
@end example

Deletion is recursively defined very similar to the way assignment is defined.
Rather than spelling it out in full details, here are some hints.

Deletion of a target list recursively deletes each target, from left to right.

@geindex statement; global
@geindex unbinding; name

Deletion of a name removes the binding of that name from the local or global
namespace, depending on whether the name occurs in a @ref{1504,,global} statement
in the same code block.  If the name is unbound, a @ref{21f,,NameError} exception
will be raised.

@geindex attribute; deletion

Deletion of attribute references, subscriptions and slicings is passed to the
primary object involved; deletion of a slicing is in general equivalent to
assignment of an empty slice of the right type (but even this is determined by
the sliced object).


Changed in version 3.2: Previously it was illegal to delete a name from the local namespace if it
occurs as a free variable in a nested block.


@node The return statement,The yield statement,The del statement<2>,Simple statements
@anchor{reference/simple_stmts return}@anchor{4c6}@anchor{reference/simple_stmts the-return-statement}@anchor{1952}
@subsection The @code{return} statement


@geindex statement; return
@geindex function; definition
@geindex class; definition

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-return_stmt}@anchor{1953}@anchor{reference/simple_stmts grammar-token-return-stmt}@anchor{1954}return_stmt ::= "return" [expression_list]
@end example

@ref{4c6,,return} may only occur syntactically nested in a function definition,
not within a nested class definition.

If an expression list is present, it is evaluated, else @code{None} is substituted.

@ref{4c6,,return} leaves the current function call with the expression list (or
@code{None}) as return value.

@geindex keyword; finally

When @ref{4c6,,return} passes control out of a @ref{f6d,,try} statement with a
@ref{4c1,,finally} clause, that @code{finally} clause is executed before
really leaving the function.

In a generator function, the @ref{4c6,,return} statement indicates that the
generator is done and will cause @ref{736,,StopIteration} to be raised. The returned
value (if any) is used as an argument to construct @ref{736,,StopIteration} and
becomes the @code{StopIteration.value} attribute.

In an asynchronous generator function, an empty @ref{4c6,,return} statement
indicates that the asynchronous generator is done and will cause
@ref{12da,,StopAsyncIteration} to be raised.  A non-empty @code{return}
statement is a syntax error in an asynchronous generator function.

@node The yield statement,The raise statement,The return statement,Simple statements
@anchor{reference/simple_stmts the-yield-statement}@anchor{1955}@anchor{reference/simple_stmts yield}@anchor{4c5}
@subsection The @code{yield} statement


@geindex statement; yield
@geindex generator; function
@geindex generator; iterator
@geindex function; generator
@geindex exception; StopIteration

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-yield_stmt}@anchor{1956}@anchor{reference/simple_stmts grammar-token-yield-stmt}@anchor{1957}yield_stmt ::= yield_expression
@end example

A @ref{4c5,,yield} statement is semantically equivalent to a @ref{18a7,,yield expression}. The yield statement can be used to omit the parentheses
that would otherwise be required in the equivalent yield expression
statement. For example, the yield statements

@example
yield <expr>
yield from <expr>
@end example

are equivalent to the yield expression statements

@example
(yield <expr>)
(yield from <expr>)
@end example

Yield expressions and statements are only used when defining a @ref{bf7,,generator}
function, and are only used in the body of the generator function.  Using yield
in a function definition is sufficient to cause that definition to create a
generator function instead of a normal function.

For full details of @ref{4c5,,yield} semantics, refer to the
@ref{18a7,,Yield expressions} section.

@node The raise statement,The break statement,The yield statement,Simple statements
@anchor{reference/simple_stmts raise}@anchor{16d}@anchor{reference/simple_stmts the-raise-statement}@anchor{1958}
@subsection The @code{raise} statement


@geindex statement; raise
@geindex exception
@geindex raising; exception
@geindex __traceback__ (exception attribute)

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-raise_stmt}@anchor{1959}@anchor{reference/simple_stmts grammar-token-raise-stmt}@anchor{195a}raise_stmt ::= "raise" [expression ["from" expression]]
@end example

If no expressions are present, @ref{16d,,raise} re-raises the
exception that is currently being handled, which is also known as the @emph{active exception}.
If there isn’t currently an active exception, a @ref{5af,,RuntimeError} exception is raised
indicating that this is an error.

Otherwise, @ref{16d,,raise} evaluates the first expression as the exception
object.  It must be either a subclass or an instance of @ref{15f,,BaseException}.
If it is a class, the exception instance will be obtained when needed by
instantiating the class with no arguments.

The @emph{type} of the exception is the exception instance’s class, the
@emph{value} is the instance itself.

@geindex object; traceback

A traceback object is normally created automatically when an exception is raised
and attached to it as the @code{__traceback__} attribute, which is writable.
You can create an exception and set your own traceback in one step using the
@ref{195b,,with_traceback()} exception method (which returns the
same exception instance, with its traceback set to its argument), like so:

@example
raise Exception("foo occurred").with_traceback(tracebackobj)
@end example

@geindex exception; chaining
@geindex __cause__ (exception attribute)
@geindex __context__ (exception attribute)

The @code{from} clause is used for exception chaining: if given, the second
@emph{expression} must be another exception class or instance. If the second
expression is an exception instance, it will be attached to the raised
exception as the @code{__cause__} attribute (which is writable). If the
expression is an exception class, the class will be instantiated and the
resulting exception instance will be attached to the raised exception as the
@code{__cause__} attribute. If the raised exception is not handled, both
exceptions will be printed:

@example
>>> try:
...     print(1 / 0)
... except Exception as exc:
...     raise RuntimeError("Something bad happened") from exc
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ZeroDivisionError: division by zero

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
RuntimeError: Something bad happened
@end example

A similar mechanism works implicitly if a new exception is raised when
an exception is already being handled.  An exception may be handled
when an @ref{16e,,except} or @ref{4c1,,finally} clause, or a
@ref{168,,with} statement, is used.  The previous exception is then
attached as the new exception’s @code{__context__} attribute:

@example
>>> try:
...     print(1 / 0)
... except:
...     raise RuntimeError("Something bad happened")
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
RuntimeError: Something bad happened
@end example

Exception chaining can be explicitly suppressed by specifying @ref{182,,None} in
the @code{from} clause:

@example
>>> try:
...     print(1 / 0)
... except:
...     raise RuntimeError("Something bad happened") from None
...
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
RuntimeError: Something bad happened
@end example

Additional information on exceptions can be found in section @ref{1835,,Exceptions},
and information about handling exceptions is in section @ref{f6d,,The try statement}.


Changed in version 3.3: @ref{182,,None} is now permitted as @code{Y} in @code{raise X from Y}.



New in version 3.3: The @code{__suppress_context__} attribute to suppress automatic display of the
exception context.



Changed in version 3.11: If the traceback of the active exception is modified in an @ref{16e,,except}
clause, a subsequent @code{raise} statement re-raises the exception with the
modified traceback. Previously, the exception was re-raised with the
traceback it had when it was caught.


@node The break statement,The continue statement,The raise statement,Simple statements
@anchor{reference/simple_stmts break}@anchor{5cb}@anchor{reference/simple_stmts the-break-statement}@anchor{195c}
@subsection The @code{break} statement


@geindex statement; break
@geindex statement; for
@geindex statement; while
@geindex loop; statement

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-break_stmt}@anchor{195d}@anchor{reference/simple_stmts grammar-token-break-stmt}@anchor{195e}break_stmt ::= "break"
@end example

@ref{5cb,,break} may only occur syntactically nested in a @ref{161,,for} or
@ref{14ec,,while} loop, but not nested in a function or class definition within
that loop.

@geindex keyword; else
@geindex loop control; target

It terminates the nearest enclosing loop, skipping the optional @code{else}
clause if the loop has one.

If a @ref{161,,for} loop is terminated by @ref{5cb,,break}, the loop control
target keeps its current value.

@geindex keyword; finally

When @ref{5cb,,break} passes control out of a @ref{f6d,,try} statement with a
@ref{4c1,,finally} clause, that @code{finally} clause is executed before
really leaving the loop.

@node The continue statement,The import statement,The break statement,Simple statements
@anchor{reference/simple_stmts continue}@anchor{4c0}@anchor{reference/simple_stmts the-continue-statement}@anchor{195f}
@subsection The @code{continue} statement


@geindex statement; continue
@geindex statement; for
@geindex statement; while
@geindex loop; statement
@geindex keyword; finally

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-continue_stmt}@anchor{1960}@anchor{reference/simple_stmts grammar-token-continue-stmt}@anchor{1961}continue_stmt ::= "continue"
@end example

@ref{4c0,,continue} may only occur syntactically nested in a @ref{161,,for} or
@ref{14ec,,while} loop, but not nested in a function or class definition within
that loop.  It continues with the next cycle of the nearest enclosing loop.

When @ref{4c0,,continue} passes control out of a @ref{f6d,,try} statement with a
@ref{4c1,,finally} clause, that @code{finally} clause is executed before
really starting the next loop cycle.
@anchor{reference/simple_stmts import}@anchor{e35}
@node The import statement,The global statement,The continue statement,Simple statements
@anchor{reference/simple_stmts from}@anchor{e51}@anchor{reference/simple_stmts the-import-statement}@anchor{1962}
@subsection The @code{import} statement


@geindex statement; import
@geindex module; importing
@geindex name; binding
@geindex keyword; from
@geindex keyword; as
@geindex exception; ImportError
@geindex @comma{} (comma); import statement

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-import_stmt}@anchor{1963}@anchor{reference/simple_stmts grammar-token-import-stmt}@anchor{1964}import_stmt     ::= "import" module ["as" identifier] ("," module ["as" identifier])*
                    | "from" relative_module "import" identifier ["as" identifier]
                    ("," identifier ["as" identifier])*
                    | "from" relative_module "import" "(" identifier ["as" identifier]
                    ("," identifier ["as" identifier])* [","] ")"
                    | "from" relative_module "import" "*"
@anchor{reference/simple_stmts grammar-token-python-grammar-module}@anchor{1965}@anchor{reference/simple_stmts grammar-token-module}@anchor{1966}module          ::= (identifier ".")* identifier
@anchor{reference/simple_stmts grammar-token-python-grammar-relative_module}@anchor{1967}@anchor{reference/simple_stmts grammar-token-relative-module}@anchor{1968}relative_module ::= "."* module | "."+
@end example

The basic import statement (no @ref{e51,,from} clause) is executed in two
steps:


@enumerate 

@item 
find a module, loading and initializing it if necessary

@item 
define a name or names in the local namespace for the scope where
the @ref{e35,,import} statement occurs.
@end enumerate

When the statement contains multiple clauses (separated by
commas) the two steps are carried out separately for each clause, just
as though the clauses had been separated out into individual import
statements.

The details of the first step, finding and loading modules are described in
greater detail in the section on the @ref{1796,,import system},
which also describes the various types of packages and modules that can
be imported, as well as all the hooks that can be used to customize
the import system. Note that failures in this step may indicate either
that the module could not be located, @emph{or} that an error occurred while
initializing the module, which includes execution of the module’s code.

If the requested module is retrieved successfully, it will be made
available in the local namespace in one of three ways:

@geindex as; import statement


@itemize *

@item 
If the module name is followed by @code{as}, then the name
following @code{as} is bound directly to the imported module.

@item 
If no other name is specified, and the module being imported is a top
level module, the module’s name is bound in the local namespace as a
reference to the imported module

@item 
If the module being imported is @emph{not} a top level module, then the name
of the top level package that contains the module is bound in the local
namespace as a reference to the top level package. The imported module
must be accessed using its full qualified name rather than directly
@end itemize

@geindex name; binding
@geindex from; import statement

The @ref{e51,,from} form uses a slightly more complex process:


@enumerate 

@item 
find the module specified in the @ref{e51,,from} clause, loading and
initializing it if necessary;

@item 
for each of the identifiers specified in the @ref{e35,,import} clauses:


@enumerate 

@item 
check if the imported module has an attribute by that name

@item 
if not, attempt to import a submodule with that name and then
check the imported module again for that attribute

@item 
if the attribute is not found, @ref{3b8,,ImportError} is raised.

@item 
otherwise, a reference to that value is stored in the local namespace,
using the name in the @code{as} clause if it is present,
otherwise using the attribute name
@end enumerate
@end enumerate

Examples:

@example
import foo                 # foo imported and bound locally
import foo.bar.baz         # foo.bar.baz imported, foo bound locally
import foo.bar.baz as fbb  # foo.bar.baz imported and bound as fbb
from foo.bar import baz    # foo.bar.baz imported and bound as baz
from foo import attr       # foo imported and foo.attr bound as attr
@end example

@geindex * (asterisk); import statement

If the list of identifiers is replaced by a star (@code{'*'}), all public
names defined in the module are bound in the local namespace for the scope
where the @ref{e35,,import} statement occurs.

@geindex __all__ (optional module attribute)

The @emph{public names} defined by a module are determined by checking the module’s
namespace for a variable named @code{__all__}; if defined, it must be a sequence
of strings which are names defined or imported by that module.  The names
given in @code{__all__} are all considered public and are required to exist.  If
@code{__all__} is not defined, the set of public names includes all names found
in the module’s namespace which do not begin with an underscore character
(@code{'_'}).  @code{__all__} should contain the entire public API. It is intended
to avoid accidentally exporting items that are not part of the API (such as
library modules which were imported and used within the module).

The wild card form of import — @code{from module import *} — is only allowed at
the module level.  Attempting to use it in class or function definitions will
raise a @ref{21a,,SyntaxError}.

@geindex relative; import

When specifying what module to import you do not have to specify the absolute
name of the module. When a module or package is contained within another
package it is possible to make a relative import within the same top package
without having to mention the package name. By using leading dots in the
specified module or package after @ref{e51,,from} you can specify how high to
traverse up the current package hierarchy without specifying exact names. One
leading dot means the current package where the module making the import
exists. Two dots means up one package level. Three dots is up two levels, etc.
So if you execute @code{from . import mod} from a module in the @code{pkg} package
then you will end up importing @code{pkg.mod}. If you execute @code{from ..subpkg2
import mod} from within @code{pkg.subpkg1} you will import @code{pkg.subpkg2.mod}.
The specification for relative imports is contained in
the @ref{186a,,Package Relative Imports} section.

@ref{d3e,,importlib.import_module()} is provided to support applications that
determine dynamically the modules to be loaded.

Raises an @ref{1130,,auditing event} @code{import} with arguments @code{module}, @code{filename}, @code{sys.path}, @code{sys.meta_path}, @code{sys.path_hooks}.

@menu
* Future statements:: 

@end menu

@node Future statements,,,The import statement
@anchor{reference/simple_stmts future}@anchor{1969}@anchor{reference/simple_stmts future-statements}@anchor{196a}
@subsubsection Future statements


@geindex future; statement
@geindex __future__; future statement

A @emph{future statement} is a directive to the compiler that a particular
module should be compiled using syntax or semantics that will be available in a
specified future release of Python where the feature becomes standard.

The future statement is intended to ease migration to future versions of Python
that introduce incompatible changes to the language.  It allows use of the new
features on a per-module basis before the release in which the feature becomes
standard.

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-future_stmt}@anchor{196b}@anchor{reference/simple_stmts grammar-token-future-stmt}@anchor{196c}future_stmt ::= "from" "__future__" "import" feature ["as" identifier]
                ("," feature ["as" identifier])*
                | "from" "__future__" "import" "(" feature ["as" identifier]
                ("," feature ["as" identifier])* [","] ")"
@anchor{reference/simple_stmts grammar-token-python-grammar-feature}@anchor{196d}@anchor{reference/simple_stmts grammar-token-feature}@anchor{196e}feature     ::= identifier
@end example

A future statement must appear near the top of the module.  The only lines that
can appear before a future statement are:


@itemize *

@item 
the module docstring (if any),

@item 
comments,

@item 
blank lines, and

@item 
other future statements.
@end itemize

The only feature that requires using the future statement is
@code{annotations} (see 
@geindex Python Enhancement Proposals; PEP 563
PEP 563@footnote{https://www.python.org/dev/peps/pep-0563}).

All historical features enabled by the future statement are still recognized
by Python 3.  The list includes @code{absolute_import}, @code{division},
@code{generators}, @code{generator_stop}, @code{unicode_literals},
@code{print_function}, @code{nested_scopes} and @code{with_statement}.  They are
all redundant because they are always enabled, and only kept for
backwards compatibility.

A future statement is recognized and treated specially at compile time: Changes
to the semantics of core constructs are often implemented by generating
different code.  It may even be the case that a new feature introduces new
incompatible syntax (such as a new reserved word), in which case the compiler
may need to parse the module differently.  Such decisions cannot be pushed off
until runtime.

For any given release, the compiler knows which feature names have been defined,
and raises a compile-time error if a future statement contains a feature not
known to it.

The direct runtime semantics are the same as for any import statement: there is
a standard module @ref{0,,__future__}, described later, and it will be imported in
the usual way at the time the future statement is executed.

The interesting runtime semantics depend on the specific feature enabled by the
future statement.

Note that there is nothing special about the statement:

@example
import __future__ [as name]
@end example

That is not a future statement; it’s an ordinary import statement with no
special semantics or syntax restrictions.

Code compiled by calls to the built-in functions @ref{34b,,exec()} and @ref{4e2,,compile()}
that occur in a module @code{M} containing a future statement will, by default,
use the new syntax or semantics associated with the future statement.  This can
be controlled by optional arguments to @ref{4e2,,compile()} — see the documentation
of that function for details.

A future statement typed at an interactive interpreter prompt will take effect
for the rest of the interpreter session.  If an interpreter is started with the
@ref{1018,,-i} option, is passed a script name to execute, and the script includes
a future statement, it will be in effect in the interactive session started
after the script is executed.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 236

@item PEP 236@footnote{https://www.python.org/dev/peps/pep-0236} - Back to the __future__

The original proposal for the __future__ mechanism.
@end table


@node The global statement,The nonlocal statement,The import statement,Simple statements
@anchor{reference/simple_stmts global}@anchor{1504}@anchor{reference/simple_stmts the-global-statement}@anchor{196f}
@subsection The @code{global} statement


@geindex statement; global
@geindex global; name; binding
@geindex @comma{} (comma); identifier list

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-global_stmt}@anchor{1970}@anchor{reference/simple_stmts grammar-token-global-stmt}@anchor{1971}global_stmt ::= "global" identifier ("," identifier)*
@end example

The @ref{1504,,global} statement is a declaration which holds for the entire
current code block.  It means that the listed identifiers are to be interpreted
as globals.  It would be impossible to assign to a global variable without
@code{global}, although free variables may refer to globals without being
declared global.

Names listed in a @ref{1504,,global} statement must not be used in the same code
block textually preceding that @code{global} statement.

Names listed in a @ref{1504,,global} statement must not be defined as formal
parameters, or as targets in @ref{168,,with} statements or @ref{16e,,except} clauses, or in a @ref{161,,for} target list, @ref{e72,,class}
definition, function definition, @ref{e35,,import} statement, or variable
annotation.

@strong{CPython implementation detail:} The current implementation does not enforce some of these restrictions, but
programs should not abuse this freedom, as future implementations may enforce
them or silently change the meaning of the program.

@geindex built-in function; exec
@geindex built-in function; eval
@geindex built-in function; compile

@strong{Programmer’s note:} @ref{1504,,global} is a directive to the parser.  It
applies only to code parsed at the same time as the @code{global} statement.
In particular, a @code{global} statement contained in a string or code
object supplied to the built-in @ref{34b,,exec()} function does not affect the code
block @emph{containing} the function call, and code contained in such a string is
unaffected by @code{global} statements in the code containing the function
call.  The same applies to the @ref{34a,,eval()} and @ref{4e2,,compile()} functions.

@node The nonlocal statement,,The global statement,Simple statements
@anchor{reference/simple_stmts nonlocal}@anchor{e4d}@anchor{reference/simple_stmts the-nonlocal-statement}@anchor{1972}
@subsection The @code{nonlocal} statement


@geindex statement; nonlocal
@geindex @comma{} (comma); identifier list

@example
@anchor{reference/simple_stmts grammar-token-python-grammar-nonlocal_stmt}@anchor{1973}@anchor{reference/simple_stmts grammar-token-nonlocal-stmt}@anchor{1974}nonlocal_stmt ::= "nonlocal" identifier ("," identifier)*
@end example

@c XXX add when implemented
@c : ["=" (`target_list` "=")+ starred_expression]
@c : | "nonlocal" identifier augop expression_list

The @ref{e4d,,nonlocal} statement causes the listed identifiers to refer to
previously bound variables in the nearest enclosing scope excluding globals.
This is important because the default behavior for binding is to search the
local namespace first.  The statement allows encapsulated code to rebind
variables outside of the local scope besides the global (module) scope.

@c XXX not implemented
@c The :keyword:`nonlocal` statement may prepend an assignment or augmented
@c assignment, but not an expression.

Names listed in a @ref{e4d,,nonlocal} statement, unlike those listed in a
@ref{1504,,global} statement, must refer to pre-existing bindings in an
enclosing scope (the scope in which a new binding should be created cannot
be determined unambiguously).

Names listed in a @ref{e4d,,nonlocal} statement must not collide with
pre-existing bindings in the local scope.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3104

@item PEP 3104@footnote{https://www.python.org/dev/peps/pep-3104} - Access to Names in Outer Scopes

The specification for the @ref{e4d,,nonlocal} statement.
@end table


@node Compound statements,Top-level components,Simple statements,The Python Language Reference
@anchor{reference/compound_stmts doc}@anchor{1975}@anchor{reference/compound_stmts compound}@anchor{1976}@anchor{reference/compound_stmts compound-statements}@anchor{1977}
@section Compound statements


@geindex compound; statement

Compound statements contain (groups of) other statements; they affect or control
the execution of those other statements in some way.  In general, compound
statements span multiple lines, although in simple incarnations a whole compound
statement may be contained in one line.

The @ref{2fb,,if}, @ref{14ec,,while} and @ref{161,,for} statements implement
traditional control flow constructs.  @ref{f6d,,try} specifies exception
handlers and/or cleanup code for a group of statements, while the
@ref{168,,with} statement allows the execution of initialization and
finalization code around a block of code.  Function and class definitions are
also syntactically compound statements.

@geindex clause
@geindex suite
@geindex ; (semicolon)

A compound statement consists of one or more ‘clauses.’  A clause consists of a
header and a ‘suite.’  The clause headers of a particular compound statement are
all at the same indentation level. Each clause header begins with a uniquely
identifying keyword and ends with a colon.  A suite is a group of statements
controlled by a clause.  A suite can be one or more semicolon-separated simple
statements on the same line as the header, following the header’s colon, or it
can be one or more indented statements on subsequent lines.  Only the latter
form of a suite can contain nested compound statements; the following is illegal,
mostly because it wouldn’t be clear to which @ref{2fb,,if} clause a following
@ref{2fa,,else} clause would belong:

@example
if test1: if test2: print(x)
@end example

Also note that the semicolon binds tighter than the colon in this context, so
that in the following example, either all or none of the @ref{af6,,print()} calls are
executed:

@example
if x < y < z: print(x); print(y); print(z)
@end example

Summarizing:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-compound_stmt}@anchor{1978}@anchor{reference/compound_stmts grammar-token-compound-stmt}@anchor{1979}compound_stmt ::= if_stmt
                  | while_stmt
                  | for_stmt
                  | try_stmt
                  | with_stmt
                  | match_stmt
                  | funcdef
                  | classdef
                  | async_with_stmt
                  | async_for_stmt
                  | async_funcdef
@anchor{reference/compound_stmts grammar-token-python-grammar-suite}@anchor{197a}@anchor{reference/compound_stmts grammar-token-suite}@anchor{197b}suite         ::= stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT
@anchor{reference/compound_stmts grammar-token-python-grammar-statement}@anchor{197c}@anchor{reference/compound_stmts grammar-token-statement}@anchor{197d}statement     ::= stmt_list NEWLINE | compound_stmt
@anchor{reference/compound_stmts grammar-token-python-grammar-stmt_list}@anchor{197e}@anchor{reference/compound_stmts grammar-token-stmt-list}@anchor{197f}stmt_list     ::= simple_stmt (";" simple_stmt)* [";"]
@end example

@geindex NEWLINE token
@geindex DEDENT token
@geindex dangling; else

Note that statements always end in a @code{NEWLINE} possibly followed by a
@code{DEDENT}.  Also note that optional continuation clauses always begin with a
keyword that cannot start a statement, thus there are no ambiguities (the
‘dangling @ref{2fa,,else}’ problem is solved in Python by requiring nested
@ref{2fb,,if} statements to be indented).

The formatting of the grammar rules in the following sections places each clause
on a separate line for clarity.
@anchor{reference/compound_stmts if}@anchor{2fb}@anchor{reference/compound_stmts elif}@anchor{14f2}
@menu
* The if statement:: 
* The while statement:: 
* The for statement:: 
* The try statement:: 
* The with statement:: 
* The match statement:: 
* Function definitions:: 
* Class definitions:: 
* Coroutines: Coroutines<2>. 

@end menu

@node The if statement,The while statement,,Compound statements
@anchor{reference/compound_stmts else}@anchor{2fa}@anchor{reference/compound_stmts the-if-statement}@anchor{1980}
@subsection The @code{if} statement


@geindex statement; if
@geindex keyword; elif
@geindex keyword; else
@geindex ; (colon); compound statement

The @ref{2fb,,if} statement is used for conditional execution:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-if_stmt}@anchor{1981}@anchor{reference/compound_stmts grammar-token-if-stmt}@anchor{1982}if_stmt ::= "if" assignment_expression ":" suite
            ("elif" assignment_expression ":" suite)*
            ["else" ":" suite]
@end example

It selects exactly one of the suites by evaluating the expressions one by one
until one is found to be true (see section @ref{190a,,Boolean operations} for the definition of
true and false); then that suite is executed (and no other part of the
@ref{2fb,,if} statement is executed or evaluated).  If all expressions are
false, the suite of the @ref{2fa,,else} clause, if present, is executed.

@node The while statement,The for statement,The if statement,Compound statements
@anchor{reference/compound_stmts the-while-statement}@anchor{1983}@anchor{reference/compound_stmts while}@anchor{14ec}
@subsection The @code{while} statement


@geindex statement; while
@geindex keyword; else
@geindex loop; statement
@geindex ; (colon); compound statement

The @ref{14ec,,while} statement is used for repeated execution as long as an
expression is true:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-while_stmt}@anchor{1984}@anchor{reference/compound_stmts grammar-token-while-stmt}@anchor{1985}while_stmt ::= "while" assignment_expression ":" suite
               ["else" ":" suite]
@end example

This repeatedly tests the expression and, if it is true, executes the first
suite; if the expression is false (which may be the first time it is tested) the
suite of the @code{else} clause, if present, is executed and the loop
terminates.

@geindex statement; break
@geindex statement; continue

A @ref{5cb,,break} statement executed in the first suite terminates the loop
without executing the @code{else} clause’s suite.  A @ref{4c0,,continue}
statement executed in the first suite skips the rest of the suite and goes back
to testing the expression.

@node The for statement,The try statement,The while statement,Compound statements
@anchor{reference/compound_stmts for}@anchor{161}@anchor{reference/compound_stmts the-for-statement}@anchor{1986}
@subsection The @code{for} statement


@geindex statement; for
@geindex keyword; in
@geindex keyword; else
@geindex target; list
@geindex loop; statement
@geindex object; sequence
@geindex ; (colon); compound statement

The @ref{161,,for} statement is used to iterate over the elements of a sequence
(such as a string, tuple or list) or other iterable object:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-for_stmt}@anchor{1987}@anchor{reference/compound_stmts grammar-token-for-stmt}@anchor{1988}for_stmt ::= "for" target_list "in" starred_list ":" suite
             ["else" ":" suite]
@end example

The expression list is evaluated once; it should yield an iterable object.  An
iterator is created for the result of the @code{starred_list}.  The expression
list can contain starred elements (@code{*x, *y}) that will be unpacked in the
final iterator (as when constructing a @code{tuple} or @code{list} literal). The
suite is then executed once for each item provided by the iterator, in the
order returned by the iterator.  Each item in turn is assigned to the target
list using the standard rules for assignments (see @ref{182e,,Assignment statements}), and then
the suite is executed.  When the items are exhausted (which is immediately when
the sequence is empty or an iterator raises a @ref{736,,StopIteration} exception),
the suite in the @code{else} clause, if present, is executed, and the loop
terminates.

@geindex statement; break
@geindex statement; continue

A @ref{5cb,,break} statement executed in the first suite terminates the loop
without executing the @code{else} clause’s suite.  A @ref{4c0,,continue}
statement executed in the first suite skips the rest of the suite and continues
with the next item, or with the @code{else} clause if there is no next
item.

The for-loop makes assignments to the variables in the target list.
This overwrites all previous assignments to those variables including
those made in the suite of the for-loop:

@example
for i in range(10):
    print(i)
    i = 5             # this will not affect the for-loop
                      # because i will be overwritten with the next
                      # index in the range
@end example

@geindex built-in function; range

Names in the target list are not deleted when the loop is finished, but if the
sequence is empty, they will not have been assigned to at all by the loop.  Hint:
the built-in function @ref{438,,range()} returns an iterator of integers suitable to
emulate the effect of Pascal’s @code{for i := a to b do}; e.g., @code{list(range(3))}
returns the list @code{[0, 1, 2]}.


Changed in version 3.11: Starred elements are now allowed in the expression list.

@anchor{reference/compound_stmts try}@anchor{f6d}@anchor{reference/compound_stmts except}@anchor{16e}@anchor{reference/compound_stmts except-star}@anchor{1989}
@node The try statement,The with statement,The for statement,Compound statements
@anchor{reference/compound_stmts finally}@anchor{4c1}@anchor{reference/compound_stmts the-try-statement}@anchor{198a}
@subsection The @code{try} statement


@geindex statement; try
@geindex keyword; except
@geindex keyword; finally
@geindex keyword; else
@geindex keyword; as
@geindex ; (colon); compound statement

The @ref{f6d,,try} statement specifies exception handlers and/or cleanup code
for a group of statements:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-try_stmt}@anchor{198b}@anchor{reference/compound_stmts grammar-token-try-stmt}@anchor{198c}try_stmt  ::= try1_stmt | try2_stmt | try3_stmt
@anchor{reference/compound_stmts grammar-token-python-grammar-try1_stmt}@anchor{198d}@anchor{reference/compound_stmts grammar-token-try1-stmt}@anchor{198e}try1_stmt ::= "try" ":" suite
              ("except" [expression ["as" identifier]] ":" suite)+
              ["else" ":" suite]
              ["finally" ":" suite]
@anchor{reference/compound_stmts grammar-token-python-grammar-try2_stmt}@anchor{198f}@anchor{reference/compound_stmts grammar-token-try2-stmt}@anchor{1990}try2_stmt ::= "try" ":" suite
              ("except" "*" expression ["as" identifier] ":" suite)+
              ["else" ":" suite]
              ["finally" ":" suite]
@anchor{reference/compound_stmts grammar-token-python-grammar-try3_stmt}@anchor{1991}@anchor{reference/compound_stmts grammar-token-try3-stmt}@anchor{1992}try3_stmt ::= "try" ":" suite
              "finally" ":" suite
@end example

The @ref{16e,,except} clause(s) specify one or more exception handlers. When no
exception occurs in the @ref{f6d,,try} clause, no exception handler is executed.
When an exception occurs in the @code{try} suite, a search for an exception
handler is started.  This search inspects the except clauses in turn until one
is found that matches the exception.  An expression-less except clause, if
present, must be last; it matches any exception.  For an except clause with an
expression, that expression is evaluated, and the clause matches the exception
if the resulting object is “compatible” with the exception.  An object is
compatible with an exception if the object is the class or a base class of the exception
object, or a tuple containing an item that is the class or a base class of
the exception object.

If no except clause matches the exception, the search for an exception handler
continues in the surrounding code and on the invocation stack.  @footnote{@w{(1)} 
The exception is propagated to the invocation stack unless
there is a @ref{4c1,,finally} clause which happens to raise another
exception. That new exception causes the old one to be lost.
}

If the evaluation of an expression in the header of an except clause raises an
exception, the original search for a handler is canceled and a search starts for
the new exception in the surrounding code and on the call stack (it is treated
as if the entire @ref{f6d,,try} statement raised the exception).

@geindex as; except clause

When a matching except clause is found, the exception is assigned to the target
specified after the @code{as} keyword in that except clause, if present, and
the except clause’s suite is executed.  All except clauses must have an
executable block.  When the end of this block is reached, execution continues
normally after the entire try statement.  (This means that if two nested
handlers exist for the same exception, and the exception occurs in the try
clause of the inner handler, the outer handler will not handle the exception.)

When an exception has been assigned using @code{as target}, it is cleared at the
end of the except clause.  This is as if

@example
except E as N:
    foo
@end example

was translated to

@example
except E as N:
    try:
        foo
    finally:
        del N
@end example

This means the exception must be assigned to a different name to be able to
refer to it after the except clause.  Exceptions are cleared because with the
traceback attached to them, they form a reference cycle with the stack frame,
keeping all locals in that frame alive until the next garbage collection occurs.

@geindex module; sys
@geindex object; traceback

Before an except clause’s suite is executed, details about the exception are
stored in the @ref{f7,,sys} module and can be accessed via @ref{18f,,sys.exc_info()}.
@ref{18f,,sys.exc_info()} returns a 3-tuple consisting of the exception class, the
exception instance and a traceback object (see section @ref{1787,,The standard type hierarchy}) identifying
the point in the program where the exception occurred.  The details about the
exception accessed via @ref{18f,,sys.exc_info()} are restored to their previous values
when leaving an exception handler:

@example
>>> print(sys.exc_info())
(None, None, None)
>>> try:
...     raise TypeError
... except:
...     print(sys.exc_info())
...     try:
...          raise ValueError
...     except:
...         print(sys.exc_info())
...     print(sys.exc_info())
...
(<class 'TypeError'>, TypeError(), <traceback object at 0x10efad080>)
(<class 'ValueError'>, ValueError(), <traceback object at 0x10efad040>)
(<class 'TypeError'>, TypeError(), <traceback object at 0x10efad080>)
>>> print(sys.exc_info())
(None, None, None)
@end example

@geindex keyword; except_star

The @ref{1989,,except*} clause(s) are used for handling
@ref{10db,,ExceptionGroup}s. The exception type for matching is interpreted as in
the case of @ref{16e,,except}, but in the case of exception groups we can have
partial matches when the type matches some of the exceptions in the group.
This means that multiple except* clauses can execute, each handling part of
the exception group. Each clause executes once and handles an exception group
of all matching exceptions.  Each exception in the group is handled by at most
one except* clause, the first that matches it.

@example
>>> try:
...     raise ExceptionGroup("eg",
...         [ValueError(1), TypeError(2), OSError(3), OSError(4)])
... except* TypeError as e:
...     print(f'caught @{type(e)@} with nested @{e.exceptions@}')
... except* OSError as e:
...     print(f'caught @{type(e)@} with nested @{e.exceptions@}')
...
caught <class 'ExceptionGroup'> with nested (TypeError(2),)
caught <class 'ExceptionGroup'> with nested (OSError(3), OSError(4))
  + Exception Group Traceback (most recent call last):
  |   File "<stdin>", line 2, in <module>
  | ExceptionGroup: eg
  +-+---------------- 1 ----------------
    | ValueError: 1
    +------------------------------------
>>>

Any remaining exceptions that were not handled by any except* clause
are re-raised at the end, combined into an exception group along with
all exceptions that were raised from within except* clauses.

An except* clause must have a matching type, and this type cannot be a
subclass of :exc:`BaseExceptionGroup`. It is not possible to mix except
and except* in the same :keyword:`try`. :keyword:`break`,
:keyword:`continue` and :keyword:`return` cannot appear in an except*
clause.
@end example

@geindex keyword; else
@geindex statement; return
@geindex statement; break
@geindex statement; continue

The optional @code{else} clause is executed if the control flow leaves the
@ref{f6d,,try} suite, no exception was raised, and no @ref{4c6,,return},
@ref{4c0,,continue}, or @ref{5cb,,break} statement was executed.  Exceptions in
the @code{else} clause are not handled by the preceding @ref{16e,,except}
clauses.

@geindex keyword; finally

If @ref{4c1,,finally} is present, it specifies a ‘cleanup’ handler.  The
@ref{f6d,,try} clause is executed, including any @ref{16e,,except} and
@code{else} clauses.  If an exception occurs in any of the clauses and is
not handled, the exception is temporarily saved. The @code{finally} clause
is executed.  If there is a saved exception it is re-raised at the end of the
@code{finally} clause.  If the @code{finally} clause raises another
exception, the saved exception is set as the context of the new exception.
If the @code{finally} clause executes a @ref{4c6,,return}, @ref{5cb,,break}
or @ref{4c0,,continue} statement, the saved exception is discarded:

@example
>>> def f():
...     try:
...         1/0
...     finally:
...         return 42
...
>>> f()
42
@end example

The exception information is not available to the program during execution of
the @ref{4c1,,finally} clause.

@geindex statement; return
@geindex statement; break
@geindex statement; continue

When a @ref{4c6,,return}, @ref{5cb,,break} or @ref{4c0,,continue} statement is
executed in the @ref{f6d,,try} suite of a @code{try}…@code{finally}
statement, the @ref{4c1,,finally} clause is also executed ‘on the way out.’

The return value of a function is determined by the last @ref{4c6,,return}
statement executed.  Since the @ref{4c1,,finally} clause always executes, a
@code{return} statement executed in the @code{finally} clause will
always be the last one executed:

@example
>>> def foo():
...     try:
...         return 'try'
...     finally:
...         return 'finally'
...
>>> foo()
'finally'
@end example

Additional information on exceptions can be found in section @ref{1835,,Exceptions},
and information on using the @ref{16d,,raise} statement to generate exceptions
may be found in section @ref{16d,,The raise statement}.


Changed in version 3.8: Prior to Python 3.8, a @ref{4c0,,continue} statement was illegal in the
@ref{4c1,,finally} clause due to a problem with the implementation.

@anchor{reference/compound_stmts with}@anchor{168}
@node The with statement,The match statement,The try statement,Compound statements
@anchor{reference/compound_stmts as}@anchor{1993}@anchor{reference/compound_stmts the-with-statement}@anchor{1994}
@subsection The @code{with} statement


@geindex statement; with
@geindex keyword; as
@geindex as; with statement
@geindex @comma{} (comma); with statement
@geindex ; (colon); compound statement

The @ref{168,,with} statement is used to wrap the execution of a block with
methods defined by a context manager (see section @ref{180e,,With Statement Context Managers}).
This allows common @ref{f6d,,try}…@ref{16e,,except}…@ref{4c1,,finally}
usage patterns to be encapsulated for convenient reuse.

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-with_stmt}@anchor{1995}@anchor{reference/compound_stmts grammar-token-with-stmt}@anchor{1996}with_stmt          ::= "with" ( "(" with_stmt_contents ","? ")" | with_stmt_contents ) ":" suite
@anchor{reference/compound_stmts grammar-token-python-grammar-with_stmt_contents}@anchor{1997}@anchor{reference/compound_stmts grammar-token-with-stmt-contents}@anchor{1998}with_stmt_contents ::= with_item ("," with_item)*
@anchor{reference/compound_stmts grammar-token-python-grammar-with_item}@anchor{1999}@anchor{reference/compound_stmts grammar-token-with-item}@anchor{199a}with_item          ::= expression ["as" target]
@end example

The execution of the @ref{168,,with} statement with one “item” proceeds as follows:


@enumerate 

@item 
The context expression (the expression given in the
@ref{1999,,with_item}) is evaluated to obtain a context manager.

@item 
The context manager’s @code{__enter__()} is loaded for later use.

@item 
The context manager’s @code{__exit__()} is loaded for later use.

@item 
The context manager’s @code{__enter__()} method is invoked.

@item 
If a target was included in the @ref{168,,with} statement, the return value
from @code{__enter__()} is assigned to it.

@cartouche
@quotation Note 
The @ref{168,,with} statement guarantees that if the @code{__enter__()}
method returns without an error, then @code{__exit__()} will always be
called. Thus, if an error occurs during the assignment to the target list,
it will be treated the same as an error occurring within the suite would
be. See step 6 below.
@end quotation
@end cartouche

@item 
The suite is executed.

@item 
The context manager’s @code{__exit__()} method is invoked.  If an exception
caused the suite to be exited, its type, value, and traceback are passed as
arguments to @code{__exit__()}. Otherwise, three @ref{182,,None} arguments are
supplied.

If the suite was exited due to an exception, and the return value from the
@code{__exit__()} method was false, the exception is reraised.  If the return
value was true, the exception is suppressed, and execution continues with the
statement following the @ref{168,,with} statement.

If the suite was exited for any reason other than an exception, the return
value from @code{__exit__()} is ignored, and execution proceeds at the normal
location for the kind of exit that was taken.
@end enumerate

The following code:

@example
with EXPRESSION as TARGET:
    SUITE
@end example

is semantically equivalent to:

@example
manager = (EXPRESSION)
enter = type(manager).__enter__
exit = type(manager).__exit__
value = enter(manager)
hit_except = False

try:
    TARGET = value
    SUITE
except:
    hit_except = True
    if not exit(manager, *sys.exc_info()):
        raise
finally:
    if not hit_except:
        exit(manager, None, None, None)
@end example

With more than one item, the context managers are processed as if multiple
@ref{168,,with} statements were nested:

@example
with A() as a, B() as b:
    SUITE
@end example

is semantically equivalent to:

@example
with A() as a:
    with B() as b:
        SUITE
@end example

You can also write multi-item context managers in multiple lines if
the items are surrounded by parentheses. For example:

@example
with (
    A() as a,
    B() as b,
):
    SUITE
@end example


Changed in version 3.1: Support for multiple context expressions.



Changed in version 3.10: Support for using grouping parentheses to break the statement in multiple lines.



@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 343

@item PEP 343@footnote{https://www.python.org/dev/peps/pep-0343} - The “with” statement

The specification, background, and examples for the Python @ref{168,,with}
statement.
@end table


@node The match statement,Function definitions,The with statement,Compound statements
@anchor{reference/compound_stmts match}@anchor{290}@anchor{reference/compound_stmts the-match-statement}@anchor{199b}
@subsection The @code{match} statement


@geindex statement; match
@geindex keyword; case
@geindex pattern matching
@geindex keyword; if
@geindex keyword; as
@geindex match; case
@geindex ; (colon); compound statement


New in version 3.10.


The match statement is used for pattern matching.  Syntax:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-match_stmt}@anchor{199c}@anchor{reference/compound_stmts grammar-token-match-stmt}@anchor{199d}match_stmt   ::= 'match' subject_expr ":" NEWLINE INDENT case_block+ DEDENT
@anchor{reference/compound_stmts grammar-token-python-grammar-subject_expr}@anchor{199e}@anchor{reference/compound_stmts grammar-token-subject-expr}@anchor{199f}subject_expr ::= star_named_expression "," star_named_expressions?
                 | named_expression
@anchor{reference/compound_stmts grammar-token-python-grammar-case_block}@anchor{19a0}@anchor{reference/compound_stmts grammar-token-case-block}@anchor{19a1}case_block   ::= 'case' patterns [guard] ":" block
@end example

@cartouche
@quotation Note 
This section uses single quotes to denote
@ref{28f,,soft keywords}.
@end quotation
@end cartouche

Pattern matching takes a pattern as input (following @code{case}) and a subject
value (following @code{match}).  The pattern (which may contain subpatterns) is
matched against the subject value.  The outcomes are:


@itemize *

@item 
A match success or failure (also termed a pattern success or failure).

@item 
Possible binding of matched values to a name.  The prerequisites for this are
further discussed below.
@end itemize

The @code{match} and @code{case} keywords are @ref{28f,,soft keywords}.


@subsubheading See also



@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 634
PEP 634@footnote{https://www.python.org/dev/peps/pep-0634} – Structural Pattern Matching: Specification

@item 
@geindex Python Enhancement Proposals; PEP 636
PEP 636@footnote{https://www.python.org/dev/peps/pep-0636} – Structural Pattern Matching: Tutorial
@end itemize


@menu
* Overview:: 
* Guards:: 
* Irrefutable Case Blocks:: 
* Patterns:: 

@end menu

@node Overview,Guards,,The match statement
@anchor{reference/compound_stmts overview}@anchor{19a2}
@subsubsection Overview


Here’s an overview of the logical flow of a match statement:


@enumerate 

@item 
The subject expression @code{subject_expr} is evaluated and a resulting subject
value obtained. If the subject expression contains a comma, a tuple is
constructed using @ref{19a3,,the standard rules}.

@item 
Each pattern in a @code{case_block} is attempted to match with the subject value. The
specific rules for success or failure are described below. The match attempt can also
bind some or all of the standalone names within the pattern. The precise
pattern binding rules vary per pattern type and are
specified below.  @strong{Name bindings made during a successful pattern match
outlive the executed block and can be used after the match statement}.

@quotation

@cartouche
@quotation Note 
During failed pattern matches, some subpatterns may succeed.  Do not
rely on bindings being made for a failed match.  Conversely, do not
rely on variables remaining unchanged after a failed match.  The exact
behavior is dependent on implementation and may vary.  This is an
intentional decision made to allow different implementations to add
optimizations.
@end quotation
@end cartouche
@end quotation

@item 
If the pattern succeeds, the corresponding guard (if present) is evaluated. In
this case all name bindings are guaranteed to have happened.


@itemize *

@item 
If the guard evaluates as true or is missing, the @code{block} inside
@code{case_block} is executed.

@item 
Otherwise, the next @code{case_block} is attempted as described above.

@item 
If there are no further case blocks, the match statement is completed.
@end itemize
@end enumerate

@cartouche
@quotation Note 
Users should generally never rely on a pattern being evaluated.  Depending on
implementation, the interpreter may cache values or use other optimizations
which skip repeated evaluations.
@end quotation
@end cartouche

A sample match statement:

@example
>>> flag = False
>>> match (100, 200):
...    case (100, 300):  # Mismatch: 200 != 300
...        print('Case 1')
...    case (100, 200) if flag:  # Successful match, but guard fails
...        print('Case 2')
...    case (100, y):  # Matches and binds y to 200
...        print(f'Case 3, y: @{y@}')
...    case _:  # Pattern not attempted
...        print('Case 4, I match anything!')
...
Case 3, y: 200
@end example

In this case, @code{if flag} is a guard.  Read more about that in the next section.

@node Guards,Irrefutable Case Blocks,Overview,The match statement
@anchor{reference/compound_stmts guards}@anchor{19a4}
@subsubsection Guards


@geindex guard

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-guard}@anchor{19a5}@anchor{reference/compound_stmts grammar-token-guard}@anchor{19a6}guard ::= "if" named_expression
@end example

A @code{guard} (which is part of the @code{case}) must succeed for code inside
the @code{case} block to execute.  It takes the form: @ref{2fb,,if} followed by an
expression.

The logical flow of a @code{case} block with a @code{guard} follows:


@enumerate 

@item 
Check that the pattern in the @code{case} block succeeded.  If the pattern
failed, the @code{guard} is not evaluated and the next @code{case} block is
checked.

@item 
If the pattern succeeded, evaluate the @code{guard}.


@itemize *

@item 
If the @code{guard} condition evaluates as true, the case block is
selected.

@item 
If the @code{guard} condition evaluates as false, the case block is not
selected.

@item 
If the @code{guard} raises an exception during evaluation, the exception
bubbles up.
@end itemize
@end enumerate

Guards are allowed to have side effects as they are expressions.  Guard
evaluation must proceed from the first to the last case block, one at a time,
skipping case blocks whose pattern(s) don’t all succeed. (I.e.,
guard evaluation must happen in order.) Guard evaluation must stop once a case
block is selected.

@node Irrefutable Case Blocks,Patterns,Guards,The match statement
@anchor{reference/compound_stmts irrefutable-case}@anchor{19a7}@anchor{reference/compound_stmts irrefutable-case-blocks}@anchor{19a8}
@subsubsection Irrefutable Case Blocks


@geindex irrefutable case block
@geindex case block

An irrefutable case block is a match-all case block.  A match statement may have
at most one irrefutable case block, and it must be last.

A case block is considered irrefutable if it has no guard and its pattern is
irrefutable.  A pattern is considered irrefutable if we can prove from its
syntax alone that it will always succeed.  Only the following patterns are
irrefutable:


@itemize *

@item 
@ref{19a9,,AS Patterns} whose left-hand side is irrefutable

@item 
@ref{19aa,,OR Patterns} containing at least one irrefutable pattern

@item 
@ref{19ab,,Capture Patterns}

@item 
@ref{291,,Wildcard Patterns}

@item 
parenthesized irrefutable patterns
@end itemize

@node Patterns,,Irrefutable Case Blocks,The match statement
@anchor{reference/compound_stmts patterns}@anchor{19ac}
@subsubsection Patterns


@geindex ! patterns
@geindex AS pattern@comma{} OR pattern@comma{} capture pattern@comma{} wildcard pattern

@cartouche
@quotation Note 
This section uses grammar notations beyond standard EBNF:


@itemize *

@item 
the notation @code{SEP.RULE+} is shorthand for @code{RULE (SEP RULE)*}

@item 
the notation @code{!RULE} is shorthand for a negative lookahead assertion
@end itemize
@end quotation
@end cartouche

The top-level syntax for @code{patterns} is:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-patterns}@anchor{19ad}@anchor{reference/compound_stmts grammar-token-patterns}@anchor{19ae}patterns       ::= open_sequence_pattern | pattern
@anchor{reference/compound_stmts grammar-token-python-grammar-pattern}@anchor{19af}@anchor{reference/compound_stmts grammar-token-pattern}@anchor{19b0}pattern        ::= as_pattern | or_pattern
@anchor{reference/compound_stmts grammar-token-python-grammar-closed_pattern}@anchor{19b1}@anchor{reference/compound_stmts grammar-token-closed-pattern}@anchor{19b2}closed_pattern ::= | literal_pattern
                   | capture_pattern
                   | wildcard_pattern
                   | value_pattern
                   | group_pattern
                   | sequence_pattern
                   | mapping_pattern
                   | class_pattern
@end example

The descriptions below will include a description “in simple terms” of what a pattern
does for illustration purposes (credits to Raymond Hettinger for a document that
inspired most of the descriptions). Note that these descriptions are purely for
illustration purposes and @strong{may not} reflect
the underlying implementation.  Furthermore, they do not cover all valid forms.

@menu
* OR Patterns:: 
* AS Patterns:: 
* Literal Patterns:: 
* Capture Patterns:: 
* Wildcard Patterns:: 
* Value Patterns:: 
* Group Patterns:: 
* Sequence Patterns:: 
* Mapping Patterns:: 
* Class Patterns:: 

@end menu

@node OR Patterns,AS Patterns,,Patterns
@anchor{reference/compound_stmts id2}@anchor{19b3}@anchor{reference/compound_stmts or-patterns}@anchor{19aa}
@subsubsection OR Patterns


An OR pattern is two or more patterns separated by vertical
bars @code{|}.  Syntax:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-or_pattern}@anchor{19b4}@anchor{reference/compound_stmts grammar-token-or-pattern}@anchor{19b5}or_pattern ::= "|".closed_pattern+
@end example

Only the final subpattern may be @ref{19a7,,irrefutable}, and each
subpattern must bind the same set of names to avoid ambiguity.

An OR pattern matches each of its subpatterns in turn to the subject value,
until one succeeds.  The OR pattern is then considered successful.  Otherwise,
if none of the subpatterns succeed, the OR pattern fails.

In simple terms, @code{P1 | P2 | ...} will try to match @code{P1}, if it fails it will try to
match @code{P2}, succeeding immediately if any succeeds, failing otherwise.

@node AS Patterns,Literal Patterns,OR Patterns,Patterns
@anchor{reference/compound_stmts as-patterns}@anchor{19a9}@anchor{reference/compound_stmts id3}@anchor{19b6}
@subsubsection AS Patterns


An AS pattern matches an OR pattern on the left of the @ref{1993,,as}
keyword against a subject.  Syntax:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-as_pattern}@anchor{19b7}@anchor{reference/compound_stmts grammar-token-as-pattern}@anchor{19b8}as_pattern ::= or_pattern "as" capture_pattern
@end example

If the OR pattern fails, the AS pattern fails.  Otherwise, the AS pattern binds
the subject to the name on the right of the as keyword and succeeds.
@code{capture_pattern} cannot be a a @code{_}.

In simple terms @code{P as NAME} will match with @code{P}, and on success it will
set @code{NAME = <subject>}.

@node Literal Patterns,Capture Patterns,AS Patterns,Patterns
@anchor{reference/compound_stmts id4}@anchor{19b9}@anchor{reference/compound_stmts literal-patterns}@anchor{19ba}
@subsubsection Literal Patterns


A literal pattern corresponds to most
@ref{171e,,literals} in Python.  Syntax:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-literal_pattern}@anchor{19bb}@anchor{reference/compound_stmts grammar-token-literal-pattern}@anchor{19bc}literal_pattern ::= signed_number
                    | signed_number "+" NUMBER
                    | signed_number "-" NUMBER
                    | strings
                    | "None"
                    | "True"
                    | "False"
                    | signed_number: NUMBER | "-" NUMBER
@end example

The rule @code{strings} and the token @code{NUMBER} are defined in the
@ref{19bd,,standard Python grammar}.  Triple-quoted strings are
supported.  Raw strings and byte strings are supported.  @ref{4b0,,Formatted string literals} are
not supported.

The forms @code{signed_number '+' NUMBER} and @code{signed_number '-' NUMBER} are
for expressing @ref{177a,,complex numbers}; they require a real number
on the left and an imaginary number on the right. E.g. @code{3 + 4j}.

In simple terms, @code{LITERAL} will succeed only if @code{<subject> == LITERAL}. For
the singletons @code{None}, @code{True} and @code{False}, the @ref{2fd,,is} operator is used.

@node Capture Patterns,Wildcard Patterns,Literal Patterns,Patterns
@anchor{reference/compound_stmts capture-patterns}@anchor{19ab}@anchor{reference/compound_stmts id5}@anchor{19be}
@subsubsection Capture Patterns


A capture pattern binds the subject value to a name.
Syntax:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-capture_pattern}@anchor{19bf}@anchor{reference/compound_stmts grammar-token-capture-pattern}@anchor{19c0}capture_pattern ::= !'_' NAME
@end example

A single underscore @code{_} is not a capture pattern (this is what @code{!'_'}
expresses). It is instead treated as a
@ref{19c1,,wildcard_pattern}.

In a given pattern, a given name can only be bound once.  E.g.
@code{case x, x: ...} is invalid while @code{case [x] | x: ...} is allowed.

Capture patterns always succeed.  The binding follows scoping rules
established by the assignment expression operator in 
@geindex Python Enhancement Proposals; PEP 572
PEP 572@footnote{https://www.python.org/dev/peps/pep-0572}; the
name becomes a local variable in the closest containing function scope unless
there’s an applicable @ref{1504,,global} or @ref{e4d,,nonlocal} statement.

In simple terms @code{NAME} will always succeed and it will set @code{NAME = <subject>}.

@node Wildcard Patterns,Value Patterns,Capture Patterns,Patterns
@anchor{reference/compound_stmts id6}@anchor{19c2}@anchor{reference/compound_stmts wildcard-patterns}@anchor{291}
@subsubsection Wildcard Patterns


A wildcard pattern always succeeds (matches anything)
and binds no name.  Syntax:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-wildcard_pattern}@anchor{19c1}@anchor{reference/compound_stmts grammar-token-wildcard-pattern}@anchor{19c3}wildcard_pattern ::= '_'
@end example

@code{_} is a @ref{28f,,soft keyword} within any pattern,
but only within patterns.  It is an identifier, as usual, even within
@code{match} subject expressions, @code{guard}s, and @code{case} blocks.

In simple terms, @code{_} will always succeed.

@node Value Patterns,Group Patterns,Wildcard Patterns,Patterns
@anchor{reference/compound_stmts id7}@anchor{19c4}@anchor{reference/compound_stmts value-patterns}@anchor{19c5}
@subsubsection Value Patterns


A value pattern represents a named value in Python.
Syntax:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-value_pattern}@anchor{19c6}@anchor{reference/compound_stmts grammar-token-value-pattern}@anchor{19c7}value_pattern ::= attr
@anchor{reference/compound_stmts grammar-token-python-grammar-attr}@anchor{19c8}@anchor{reference/compound_stmts grammar-token-attr}@anchor{19c9}attr          ::= name_or_attr "." NAME
@anchor{reference/compound_stmts grammar-token-python-grammar-name_or_attr}@anchor{19ca}@anchor{reference/compound_stmts grammar-token-name-or-attr}@anchor{19cb}name_or_attr  ::= attr | NAME
@end example

The dotted name in the pattern is looked up using standard Python
@ref{1830,,name resolution rules}.  The pattern succeeds if the
value found compares equal to the subject value (using the @code{==} equality
operator).

In simple terms @code{NAME1.NAME2} will succeed only if @code{<subject> == NAME1.NAME2}

@cartouche
@quotation Note 
If the same value occurs multiple times in the same match statement, the
interpreter may cache the first value found and reuse it rather than repeat
the same lookup.  This cache is strictly tied to a given execution of a
given match statement.
@end quotation
@end cartouche

@node Group Patterns,Sequence Patterns,Value Patterns,Patterns
@anchor{reference/compound_stmts group-patterns}@anchor{19cc}@anchor{reference/compound_stmts id8}@anchor{19cd}
@subsubsection Group Patterns


A group pattern allows users to add parentheses around patterns to
emphasize the intended grouping.  Otherwise, it has no additional syntax.
Syntax:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-group_pattern}@anchor{19ce}@anchor{reference/compound_stmts grammar-token-group-pattern}@anchor{19cf}group_pattern ::= "(" pattern ")"
@end example

In simple terms @code{(P)} has the same effect as @code{P}.

@node Sequence Patterns,Mapping Patterns,Group Patterns,Patterns
@anchor{reference/compound_stmts id9}@anchor{19d0}@anchor{reference/compound_stmts sequence-patterns}@anchor{19d1}
@subsubsection Sequence Patterns


A sequence pattern contains several subpatterns to be matched against sequence elements.
The syntax is similar to the unpacking of a list or tuple.

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-sequence_pattern}@anchor{19d2}@anchor{reference/compound_stmts grammar-token-sequence-pattern}@anchor{19d3}sequence_pattern       ::= "[" [maybe_sequence_pattern] "]"
                           | "(" [open_sequence_pattern] ")"
@anchor{reference/compound_stmts grammar-token-python-grammar-open_sequence_pattern}@anchor{19d4}@anchor{reference/compound_stmts grammar-token-open-sequence-pattern}@anchor{19d5}open_sequence_pattern  ::= maybe_star_pattern "," [maybe_sequence_pattern]
@anchor{reference/compound_stmts grammar-token-python-grammar-maybe_sequence_pattern}@anchor{19d6}@anchor{reference/compound_stmts grammar-token-maybe-sequence-pattern}@anchor{19d7}maybe_sequence_pattern ::= ",".maybe_star_pattern+ ","?
@anchor{reference/compound_stmts grammar-token-python-grammar-maybe_star_pattern}@anchor{19d8}@anchor{reference/compound_stmts grammar-token-maybe-star-pattern}@anchor{19d9}maybe_star_pattern     ::= star_pattern | pattern
@anchor{reference/compound_stmts grammar-token-python-grammar-star_pattern}@anchor{19da}@anchor{reference/compound_stmts grammar-token-star-pattern}@anchor{19db}star_pattern           ::= "*" (capture_pattern | wildcard_pattern)
@end example

There is no difference if parentheses  or square brackets
are used for sequence patterns (i.e. @code{(...)} vs @code{[...]} ).

@cartouche
@quotation Note 
A single pattern enclosed in parentheses without a trailing comma
(e.g. @code{(3 | 4)}) is a @ref{19cc,,group pattern}.
While a single pattern enclosed in square brackets (e.g. @code{[3 | 4]}) is
still a sequence pattern.
@end quotation
@end cartouche

At most one star subpattern may be in a sequence pattern.  The star subpattern
may occur in any position. If no star subpattern is present, the sequence
pattern is a fixed-length sequence pattern; otherwise it is a variable-length
sequence pattern.

The following is the logical flow for matching a sequence pattern against a
subject value:


@enumerate 

@item 
If the subject value is not a sequence @footnote{@w{(2)} 
In pattern matching, a sequence is defined as one of the following:

@quotation


@itemize *

@item 
a class that inherits from @ref{110c,,collections.abc.Sequence}

@item 
a Python class that has been registered as @ref{110c,,collections.abc.Sequence}

@item 
a builtin class that has its (CPython) @ref{110b,,Py_TPFLAGS_SEQUENCE} bit set

@item 
a class that inherits from any of the above
@end itemize
@end quotation

The following standard library classes are sequences:

@quotation


@itemize *

@item 
@ref{25e,,array.array}

@item 
@ref{7d3,,collections.deque}

@item 
@ref{43a,,list}

@item 
@ref{573,,memoryview}

@item 
@ref{438,,range}

@item 
@ref{3e5,,tuple}
@end itemize
@end quotation

@cartouche
@quotation Note 
Subject values of type @code{str}, @code{bytes}, and @code{bytearray}
do not match sequence patterns.
@end quotation
@end cartouche
}, the sequence pattern
fails.

@item 
If the subject value is an instance of @code{str}, @code{bytes} or @code{bytearray}
the sequence pattern fails.

@item 
The subsequent steps depend on whether the sequence pattern is fixed or
variable-length.

If the sequence pattern is fixed-length:


@enumerate 

@item 
If the length of the subject sequence is not equal to the number of
subpatterns, the sequence pattern fails

@item 
Subpatterns in the sequence pattern are matched to their corresponding
items in the subject sequence from left to right.  Matching stops as soon
as a subpattern fails.  If all subpatterns succeed in matching their
corresponding item, the sequence pattern succeeds.
@end enumerate

Otherwise, if the sequence pattern is variable-length:


@enumerate 

@item 
If the length of the subject sequence is less than the number of non-star
subpatterns, the sequence pattern fails.

@item 
The leading non-star subpatterns are matched to their corresponding items
as for fixed-length sequences.

@item 
If the previous step succeeds, the star subpattern matches a list formed
of the remaining subject items, excluding the remaining items
corresponding to non-star subpatterns following the star subpattern.

@item 
Remaining non-star subpatterns are matched to their corresponding subject
items, as for a fixed-length sequence.
@end enumerate

@cartouche
@quotation Note 
The length of the subject sequence is obtained via
@ref{4a6,,len()} (i.e. via the @code{__len__()} protocol).  This length may be
cached by the interpreter in a similar manner as
@ref{19c5,,value patterns}.
@end quotation
@end cartouche
@end enumerate

In simple terms @code{[P1, P2, P3,} … @code{, P<N>]} matches only if all the following
happens:


@itemize *

@item 
check @code{<subject>} is a sequence

@item 
@code{len(subject) == <N>}

@item 
@code{P1} matches @code{<subject>[0]} (note that this match can also bind names)

@item 
@code{P2} matches @code{<subject>[1]} (note that this match can also bind names)

@item 
… and so on for the corresponding pattern/element.
@end itemize

@node Mapping Patterns,Class Patterns,Sequence Patterns,Patterns
@anchor{reference/compound_stmts id11}@anchor{19dc}@anchor{reference/compound_stmts mapping-patterns}@anchor{19dd}
@subsubsection Mapping Patterns


A mapping pattern contains one or more key-value patterns.  The syntax is
similar to the construction of a dictionary.
Syntax:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-mapping_pattern}@anchor{19de}@anchor{reference/compound_stmts grammar-token-mapping-pattern}@anchor{19df}mapping_pattern     ::= "@{" [items_pattern] "@}"
@anchor{reference/compound_stmts grammar-token-python-grammar-items_pattern}@anchor{19e0}@anchor{reference/compound_stmts grammar-token-items-pattern}@anchor{19e1}items_pattern       ::= ",".key_value_pattern+ ","?
@anchor{reference/compound_stmts grammar-token-python-grammar-key_value_pattern}@anchor{19e2}@anchor{reference/compound_stmts grammar-token-key-value-pattern}@anchor{19e3}key_value_pattern   ::= (literal_pattern | value_pattern) ":" pattern
                        | double_star_pattern
@anchor{reference/compound_stmts grammar-token-python-grammar-double_star_pattern}@anchor{19e4}@anchor{reference/compound_stmts grammar-token-double-star-pattern}@anchor{19e5}double_star_pattern ::= "**" capture_pattern
@end example

At most one double star pattern may be in a mapping pattern.  The double star
pattern must be the last subpattern in the mapping pattern.

Duplicate keys in mapping patterns are disallowed. Duplicate literal keys will
raise a @ref{21a,,SyntaxError}. Two keys that otherwise have the same value will
raise a @ref{17c,,ValueError} at runtime.

The following is the logical flow for matching a mapping pattern against a
subject value:


@enumerate 

@item 
If the subject value is not a mapping @footnote{@w{(3)} 
In pattern matching, a mapping is defined as one of the following:

@quotation


@itemize *

@item 
a class that inherits from @ref{3ab,,collections.abc.Mapping}

@item 
a Python class that has been registered as @ref{3ab,,collections.abc.Mapping}

@item 
a builtin class that has its (CPython) @ref{110a,,Py_TPFLAGS_MAPPING} bit set

@item 
a class that inherits from any of the above
@end itemize
@end quotation

The standard library classes @ref{3b0,,dict} and @ref{246,,types.MappingProxyType}
are mappings.
},the mapping pattern fails.

@item 
If every key given in the mapping pattern is present in the subject mapping,
and the pattern for each key matches the corresponding item of the subject
mapping, the mapping pattern succeeds.

@item 
If duplicate keys are detected in the mapping pattern, the pattern is
considered invalid. A @ref{21a,,SyntaxError} is raised for duplicate literal
values; or a @ref{17c,,ValueError} for named keys of the same value.
@end enumerate

@cartouche
@quotation Note 
Key-value pairs are matched using the two-argument form of the mapping
subject’s @code{get()} method.  Matched key-value pairs must already be present
in the mapping, and not created on-the-fly via @code{__missing__()} or
@code{__getitem__()}.
@end quotation
@end cartouche

In simple terms @code{@{KEY1: P1, KEY2: P2, ... @}} matches only if all the following
happens:


@itemize *

@item 
check @code{<subject>} is a mapping

@item 
@code{KEY1 in <subject>}

@item 
@code{P1} matches @code{<subject>[KEY1]}

@item 
… and so on for the corresponding KEY/pattern pair.
@end itemize

@node Class Patterns,,Mapping Patterns,Patterns
@anchor{reference/compound_stmts class-patterns}@anchor{19e6}@anchor{reference/compound_stmts id13}@anchor{19e7}
@subsubsection Class Patterns


A class pattern represents a class and its positional and keyword arguments
(if any).  Syntax:

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-class_pattern}@anchor{19e8}@anchor{reference/compound_stmts grammar-token-class-pattern}@anchor{19e9}class_pattern       ::= name_or_attr "(" [pattern_arguments ","?] ")"
@anchor{reference/compound_stmts grammar-token-python-grammar-pattern_arguments}@anchor{19ea}@anchor{reference/compound_stmts grammar-token-pattern-arguments}@anchor{19eb}pattern_arguments   ::= positional_patterns ["," keyword_patterns]
                        | keyword_patterns
@anchor{reference/compound_stmts grammar-token-python-grammar-positional_patterns}@anchor{19ec}@anchor{reference/compound_stmts grammar-token-positional-patterns}@anchor{19ed}positional_patterns ::= ",".pattern+
@anchor{reference/compound_stmts grammar-token-python-grammar-keyword_patterns}@anchor{19ee}@anchor{reference/compound_stmts grammar-token-keyword-patterns}@anchor{19ef}keyword_patterns    ::= ",".keyword_pattern+
@anchor{reference/compound_stmts grammar-token-python-grammar-keyword_pattern}@anchor{19f0}@anchor{reference/compound_stmts grammar-token-keyword-pattern}@anchor{19f1}keyword_pattern     ::= NAME "=" pattern
@end example

The same keyword should not be repeated in class patterns.

The following is the logical flow for matching a class pattern against a
subject value:


@enumerate 

@item 
If @code{name_or_attr} is not an instance of the builtin @ref{89e,,type} , raise
@ref{162,,TypeError}.

@item 
If the subject value is not an instance of @code{name_or_attr} (tested via
@ref{236,,isinstance()}), the class pattern fails.

@item 
If no pattern arguments are present, the pattern succeeds.  Otherwise,
the subsequent steps depend on whether keyword or positional argument patterns
are present.

For a number of built-in types (specified below), a single positional
subpattern is accepted which will match the entire subject; for these types
keyword patterns also work as for other types.

If only keyword patterns are present, they are processed as follows,
one by one:

I. The keyword is looked up as an attribute on the subject.

@quotation


@itemize *

@item 
If this raises an exception other than @ref{163,,AttributeError}, the
exception bubbles up.

@item 
If this raises @ref{163,,AttributeError}, the class pattern has failed.

@item 
Else, the subpattern associated with the keyword pattern is matched
against the subject’s attribute value.  If this fails, the class
pattern fails; if this succeeds, the match proceeds to the next keyword.
@end itemize
@end quotation

II. If all keyword patterns succeed, the class pattern succeeds.

If any positional patterns are present, they are converted to keyword
patterns using the @ref{1135,,__match_args__} attribute on the class
@code{name_or_attr} before matching:

I. The equivalent of @code{getattr(cls, "__match_args__", ())} is called.

@quotation


@itemize *

@item 
If this raises an exception, the exception bubbles up.

@item 
If the returned value is not a tuple, the conversion fails and
@ref{162,,TypeError} is raised.

@item 
If there are more positional patterns than @code{len(cls.__match_args__)},
@ref{162,,TypeError} is raised.

@item 
Otherwise, positional pattern @code{i} is converted to a keyword pattern
using @code{__match_args__[i]} as the keyword.  @code{__match_args__[i]} must
be a string; if not @ref{162,,TypeError} is raised.

@item 
If there are duplicate keywords, @ref{162,,TypeError} is raised.
@end itemize


@subsubheading See also


@ref{1813,,Customizing positional arguments in class pattern matching}

@end quotation


@table @asis

@item II. Once all positional patterns have been converted to keyword patterns,

the match proceeds as if there were only keyword patterns.
@end table

For the following built-in types the handling of positional subpatterns is
different:


@itemize *

@item 
@ref{2f7,,bool}

@item 
@ref{2f1,,bytearray}

@item 
@ref{2f0,,bytes}

@item 
@ref{3b0,,dict}

@item 
@ref{256,,float}

@item 
@ref{439,,frozenset}

@item 
@ref{1b1,,int}

@item 
@ref{43a,,list}

@item 
@ref{370,,set}

@item 
@ref{2ef,,str}

@item 
@ref{3e5,,tuple}
@end itemize

These classes accept a single positional argument, and the pattern there is matched
against the whole object rather than an attribute. For example @code{int(0|1)} matches
the value @code{0}, but not the values @code{0.0} or @code{False}.
@end enumerate

In simple terms @code{CLS(P1, attr=P2)} matches only if the following happens:


@itemize *

@item 
@code{isinstance(<subject>, CLS)}

@item 
convert @code{P1} to a keyword pattern using @code{CLS.__match_args__}

@item 

@table @asis

@item For each keyword argument @code{attr=P2}:


@itemize *

@item 
@code{hasattr(<subject>, "attr")}

@item 
@code{P2} matches @code{<subject>.attr}
@end itemize
@end table

@item 
… and so on for the corresponding keyword argument/pattern pair.
@end itemize


@subsubheading See also



@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 634
PEP 634@footnote{https://www.python.org/dev/peps/pep-0634} – Structural Pattern Matching: Specification

@item 
@geindex Python Enhancement Proposals; PEP 636
PEP 636@footnote{https://www.python.org/dev/peps/pep-0636} – Structural Pattern Matching: Tutorial
@end itemize


@geindex parameter; function definition
@anchor{reference/compound_stmts function}@anchor{151c}
@node Function definitions,Class definitions,The match statement,Compound statements
@anchor{reference/compound_stmts def}@anchor{fb8}@anchor{reference/compound_stmts function-definitions}@anchor{19f2}
@subsection Function definitions


@geindex statement; def
@geindex function; definition
@geindex function; name
@geindex name; binding
@geindex object; user-defined function
@geindex object; function
@geindex function; name
@geindex name; binding
@geindex () (parentheses); function definition
@geindex @comma{} (comma); parameter list
@geindex ; (colon); compound statement

A function definition defines a user-defined function object (see section
@ref{1787,,The standard type hierarchy}):

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-funcdef}@anchor{19f3}@anchor{reference/compound_stmts grammar-token-funcdef}@anchor{19f4}funcdef                   ::= [decorators] "def" funcname "(" [parameter_list] ")"
                              ["->" expression] ":" suite
@anchor{reference/compound_stmts grammar-token-python-grammar-decorators}@anchor{19f5}@anchor{reference/compound_stmts grammar-token-decorators}@anchor{19f6}decorators                ::= decorator+
@anchor{reference/compound_stmts grammar-token-python-grammar-decorator}@anchor{19f7}@anchor{reference/compound_stmts grammar-token-decorator}@anchor{19f8}decorator                 ::= "@@" assignment_expression NEWLINE
@anchor{reference/compound_stmts grammar-token-python-grammar-parameter_list}@anchor{19f9}@anchor{reference/compound_stmts grammar-token-parameter-list}@anchor{19fa}parameter_list            ::= defparameter ("," defparameter)* "," "/" ["," [parameter_list_no_posonly]]
                                | parameter_list_no_posonly
@anchor{reference/compound_stmts grammar-token-python-grammar-parameter_list_no_posonly}@anchor{19fb}@anchor{reference/compound_stmts grammar-token-parameter-list-no-posonly}@anchor{19fc}parameter_list_no_posonly ::= defparameter ("," defparameter)* ["," [parameter_list_starargs]]
                              | parameter_list_starargs
@anchor{reference/compound_stmts grammar-token-python-grammar-parameter_list_starargs}@anchor{19fd}@anchor{reference/compound_stmts grammar-token-parameter-list-starargs}@anchor{19fe}parameter_list_starargs   ::= "*" [parameter] ("," defparameter)* ["," ["**" parameter [","]]]
                              | "**" parameter [","]
@anchor{reference/compound_stmts grammar-token-python-grammar-parameter}@anchor{19ff}@anchor{reference/compound_stmts grammar-token-parameter}@anchor{1a00}parameter                 ::= identifier [":" expression]
@anchor{reference/compound_stmts grammar-token-python-grammar-defparameter}@anchor{1a01}@anchor{reference/compound_stmts grammar-token-defparameter}@anchor{1a02}defparameter              ::= parameter ["=" expression]
@anchor{reference/compound_stmts grammar-token-python-grammar-funcname}@anchor{1a03}@anchor{reference/compound_stmts grammar-token-funcname}@anchor{1a04}funcname                  ::= identifier
@end example

A function definition is an executable statement.  Its execution binds the
function name in the current local namespace to a function object (a wrapper
around the executable code for the function).  This function object contains a
reference to the current global namespace as the global namespace to be used
when the function is called.

The function definition does not execute the function body; this gets executed
only when the function is called. @footnote{@w{(4)} 
A string literal appearing as the first statement in the function body is
transformed into the function’s @code{__doc__} attribute and therefore the
function’s @ref{1a05,,docstring}.
}

@geindex @@ (at); function definition

A function definition may be wrapped by one or more @ref{1bf,,decorator} expressions.
Decorator expressions are evaluated when the function is defined, in the scope
that contains the function definition.  The result must be a callable, which is
invoked with the function object as the only argument. The returned value is
bound to the function name instead of the function object.  Multiple decorators
are applied in nested fashion. For example, the following code

@example
@@f1(arg)
@@f2
def func(): pass
@end example

is roughly equivalent to

@example
def func(): pass
func = f1(arg)(f2(func))
@end example

except that the original function is not temporarily bound to the name @code{func}.


Changed in version 3.9: Functions may be decorated with any valid
@ref{1914,,assignment_expression}. Previously, the grammar was
much more restrictive; see 
@geindex Python Enhancement Proposals; PEP 614
PEP 614@footnote{https://www.python.org/dev/peps/pep-0614} for details.


@geindex default; parameter; value
@geindex argument; function definition
@geindex = (equals); function definition

When one or more @ref{18e1,,parameters} have the form @emph{parameter} @code{=}
@emph{expression}, the function is said to have “default parameter values.”  For a
parameter with a default value, the corresponding @ref{18d2,,argument} may be
omitted from a call, in which
case the parameter’s default value is substituted.  If a parameter has a default
value, all following parameters up until the “@code{*}” must also have a default
value — this is a syntactic restriction that is not expressed by the grammar.

@strong{Default parameter values are evaluated from left to right when the function
definition is executed.} This means that the expression is evaluated once, when
the function is defined, and that the same “pre-computed” value is used for each
call.  This is especially important to understand when a default parameter value is a
mutable object, such as a list or a dictionary: if the function modifies the
object (e.g. by appending an item to a list), the default parameter value is in effect
modified.  This is generally not what was intended.  A way around this is to use
@code{None} as the default, and explicitly test for it in the body of the function,
e.g.:

@example
def whats_on_the_telly(penguin=None):
    if penguin is None:
        penguin = []
    penguin.append("property of the zoo")
    return penguin
@end example

@geindex / (slash); function definition
@geindex * (asterisk); function definition
@geindex **; function definition

Function call semantics are described in more detail in section @ref{8d6,,Calls}. A
function call always assigns values to all parameters mentioned in the parameter
list, either from positional arguments, from keyword arguments, or from default
values.  If the form “@code{*identifier}” is present, it is initialized to a tuple
receiving any excess positional parameters, defaulting to the empty tuple.
If the form “@code{**identifier}” is present, it is initialized to a new
ordered mapping receiving any excess keyword arguments, defaulting to a
new empty mapping of the same type.  Parameters after “@code{*}” or
“@code{*identifier}” are keyword-only parameters and may only be passed
by keyword arguments.  Parameters before “@code{/}” are positional-only parameters
and may only be passed by positional arguments.


Changed in version 3.8: The @code{/} function parameter syntax may be used to indicate positional-only
parameters. See 
@geindex Python Enhancement Proposals; PEP 570
PEP 570@footnote{https://www.python.org/dev/peps/pep-0570} for details.


@geindex function; annotations
@geindex ->; function annotations
@geindex ; (colon); function annotations

Parameters may have an @ref{151d,,annotation} of the form “@code{: expression}”
following the parameter name.  Any parameter may have an annotation, even those of the form
@code{*identifier} or @code{**identifier}.  Functions may have “return” annotation of
the form “@code{-> expression}” after the parameter list.  These annotations can be
any valid Python expression.  The presence of annotations does not change the
semantics of a function.  The annotation values are available as values of
a dictionary keyed by the parameters’ names in the @code{__annotations__}
attribute of the function object.  If the @code{annotations} import from
@ref{0,,__future__} is used, annotations are preserved as strings at runtime which
enables postponed evaluation.  Otherwise, they are evaluated when the function
definition is executed.  In this case annotations may be evaluated in
a different order than they appear in the source code.

@geindex lambda; expression

It is also possible to create anonymous functions (functions not bound to a
name), for immediate use in expressions.  This uses lambda expressions, described in
section @ref{e42,,Lambdas}.  Note that the lambda expression is merely a shorthand for a
simplified function definition; a function defined in a “@ref{fb8,,def}”
statement can be passed around or assigned to another name just like a function
defined by a lambda expression.  The “@code{def}” form is actually more powerful
since it allows the execution of multiple statements and annotations.

@strong{Programmer’s note:} Functions are first-class objects.  A “@code{def}” statement
executed inside a function definition defines a local function that can be
returned or passed around.  Free variables used in the nested function can
access the local variables of the function containing the def.  See section
@ref{182a,,Naming and binding} for details.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3107

@item PEP 3107@footnote{https://www.python.org/dev/peps/pep-3107} - Function Annotations

The original specification for function annotations.

@geindex Python Enhancement Proposals; PEP 484

@item PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} - Type Hints

Definition of a standard meaning for annotations: type hints.

@geindex Python Enhancement Proposals; PEP 526

@item PEP 526@footnote{https://www.python.org/dev/peps/pep-0526} - Syntax for Variable Annotations

Ability to type hint variable declarations, including class
variables and instance variables

@geindex Python Enhancement Proposals; PEP 563

@item PEP 563@footnote{https://www.python.org/dev/peps/pep-0563} - Postponed Evaluation of Annotations

Support for forward references within annotations by preserving
annotations in a string form at runtime instead of eager evaluation.
@end table


@node Class definitions,Coroutines<2>,Function definitions,Compound statements
@anchor{reference/compound_stmts class}@anchor{e72}@anchor{reference/compound_stmts class-definitions}@anchor{1a06}
@subsection Class definitions


@geindex object; class
@geindex statement; class
@geindex class; definition
@geindex class; name
@geindex name; binding
@geindex execution; frame
@geindex inheritance
@geindex docstring
@geindex () (parentheses); class definition
@geindex @comma{} (comma); expression list
@geindex ; (colon); compound statement

A class definition defines a class object (see section @ref{1787,,The standard type hierarchy}):

@example
@anchor{reference/compound_stmts grammar-token-python-grammar-classdef}@anchor{1a07}@anchor{reference/compound_stmts grammar-token-classdef}@anchor{1a08}classdef    ::= [decorators] "class" classname [inheritance] ":" suite
@anchor{reference/compound_stmts grammar-token-python-grammar-inheritance}@anchor{1a09}@anchor{reference/compound_stmts grammar-token-inheritance}@anchor{1a0a}inheritance ::= "(" [argument_list] ")"
@anchor{reference/compound_stmts grammar-token-python-grammar-classname}@anchor{1a0b}@anchor{reference/compound_stmts grammar-token-classname}@anchor{1a0c}classname   ::= identifier
@end example

A class definition is an executable statement.  The inheritance list usually
gives a list of base classes (see @ref{17bf,,Metaclasses} for more advanced uses), so
each item in the list should evaluate to a class object which allows
subclassing.  Classes without an inheritance list inherit, by default, from the
base class @ref{5a5,,object}; hence,

@example
class Foo:
    pass
@end example

is equivalent to

@example
class Foo(object):
    pass
@end example

The class’s suite is then executed in a new execution frame (see @ref{182a,,Naming and binding}),
using a newly created local namespace and the original global namespace.
(Usually, the suite contains mostly function definitions.)  When the class’s
suite finishes execution, its execution frame is discarded but its local
namespace is saved. @footnote{@w{(5)} 
A string literal appearing as the first statement in the class body is
transformed into the namespace’s @code{__doc__} item and therefore the class’s
@ref{1a05,,docstring}.
} A class object is then created using the inheritance
list for the base classes and the saved local namespace for the attribute
dictionary.  The class name is bound to this class object in the original local
namespace.

The order in which attributes are defined in the class body is preserved
in the new class’s @code{__dict__}.  Note that this is reliable only right
after the class is created and only for classes that were defined using
the definition syntax.

Class creation can be customized heavily using @ref{17bf,,metaclasses}.

@geindex @@ (at); class definition

Classes can also be decorated: just like when decorating functions,

@example
@@f1(arg)
@@f2
class Foo: pass
@end example

is roughly equivalent to

@example
class Foo: pass
Foo = f1(arg)(f2(Foo))
@end example

The evaluation rules for the decorator expressions are the same as for function
decorators.  The result is then bound to the class name.


Changed in version 3.9: Classes may be decorated with any valid
@ref{1914,,assignment_expression}. Previously, the grammar was
much more restrictive; see 
@geindex Python Enhancement Proposals; PEP 614
PEP 614@footnote{https://www.python.org/dev/peps/pep-0614} for details.


@strong{Programmer’s note:} Variables defined in the class definition are class
attributes; they are shared by instances.  Instance attributes can be set in a
method with @code{self.name = value}.  Both class and instance attributes are
accessible through the notation “@code{self.name}”, and an instance attribute hides
a class attribute with the same name when accessed in this way.  Class
attributes can be used as defaults for instance attributes, but using mutable
values there can lead to unexpected results.  @ref{793,,Descriptors}
can be used to create instance variables with different implementation details.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3115

@item PEP 3115@footnote{https://www.python.org/dev/peps/pep-3115} - Metaclasses in Python 3000

The proposal that changed the declaration of metaclasses to the current
syntax, and the semantics for how classes with metaclasses are
constructed.

@geindex Python Enhancement Proposals; PEP 3129

@item PEP 3129@footnote{https://www.python.org/dev/peps/pep-3129} - Class Decorators

The proposal that added class decorators.  Function and method decorators
were introduced in 
@geindex Python Enhancement Proposals; PEP 318
PEP 318@footnote{https://www.python.org/dev/peps/pep-0318}.
@end table


@node Coroutines<2>,,Class definitions,Compound statements
@anchor{reference/compound_stmts async}@anchor{4d4}@anchor{reference/compound_stmts coroutines}@anchor{1a0d}
@subsection Coroutines



New in version 3.5.


@geindex statement; async def

@menu
* Coroutine function definition:: 
* The async for statement:: 
* The async with statement:: 

@end menu

@node Coroutine function definition,The async for statement,,Coroutines<2>
@anchor{reference/compound_stmts async-def}@anchor{1c0}@anchor{reference/compound_stmts coroutine-function-definition}@anchor{1a0e}
@subsubsection Coroutine function definition


@example
@anchor{reference/compound_stmts grammar-token-python-grammar-async_funcdef}@anchor{1a0f}@anchor{reference/compound_stmts grammar-token-async-funcdef}@anchor{1a10}async_funcdef ::= [decorators] "async" "def" funcname "(" [parameter_list] ")"
                  ["->" expression] ":" suite
@end example

@geindex keyword; async
@geindex keyword; await

Execution of Python coroutines can be suspended and resumed at many points
(see @ref{3cf,,coroutine}). @ref{4d5,,await} expressions, @ref{5cd,,async for} and
@ref{169,,async with} can only be used in the body of a coroutine function.

Functions defined with @code{async def} syntax are always coroutine functions,
even if they do not contain @code{await} or @code{async} keywords.

It is a @ref{21a,,SyntaxError} to use a @code{yield from} expression inside the body
of a coroutine function.

An example of a coroutine function:

@example
async def func(param1, param2):
    do_stuff()
    await some_coroutine()
@end example


Changed in version 3.7: @code{await} and @code{async} are now keywords; previously they were only
treated as such inside the body of a coroutine function.


@geindex statement; async for

@node The async for statement,The async with statement,Coroutine function definition,Coroutines<2>
@anchor{reference/compound_stmts async-for}@anchor{5cd}@anchor{reference/compound_stmts the-async-for-statement}@anchor{1a11}
@subsubsection The @code{async for} statement


@example
@anchor{reference/compound_stmts grammar-token-python-grammar-async_for_stmt}@anchor{1a12}@anchor{reference/compound_stmts grammar-token-async-for-stmt}@anchor{1a13}async_for_stmt ::= "async" for_stmt
@end example

An @ref{8d0,,asynchronous iterable} provides an @code{__aiter__} method that directly
returns an @ref{8d2,,asynchronous iterator}, which can call asynchronous code in
its @code{__anext__} method.

The @code{async for} statement allows convenient iteration over asynchronous
iterables.

The following code:

@example
async for TARGET in ITER:
    SUITE
else:
    SUITE2
@end example

Is semantically equivalent to:

@example
iter = (ITER)
iter = type(iter).__aiter__(iter)
running = True

while running:
    try:
        TARGET = await type(iter).__anext__(iter)
    except StopAsyncIteration:
        running = False
    else:
        SUITE
else:
    SUITE2
@end example

See also @code{__aiter__()} and @code{__anext__()} for details.

It is a @ref{21a,,SyntaxError} to use an @code{async for} statement outside the
body of a coroutine function.

@geindex statement; async with

@node The async with statement,,The async for statement,Coroutines<2>
@anchor{reference/compound_stmts async-with}@anchor{169}@anchor{reference/compound_stmts the-async-with-statement}@anchor{1a14}
@subsubsection The @code{async with} statement


@example
@anchor{reference/compound_stmts grammar-token-python-grammar-async_with_stmt}@anchor{1a15}@anchor{reference/compound_stmts grammar-token-async-with-stmt}@anchor{1a16}async_with_stmt ::= "async" with_stmt
@end example

An @ref{167,,asynchronous context manager} is a @ref{166,,context manager} that is
able to suspend execution in its @emph{enter} and @emph{exit} methods.

The following code:

@example
async with EXPRESSION as TARGET:
    SUITE
@end example

is semantically equivalent to:

@example
manager = (EXPRESSION)
aenter = type(manager).__aenter__
aexit = type(manager).__aexit__
value = await aenter(manager)
hit_except = False

try:
    TARGET = value
    SUITE
except:
    hit_except = True
    if not await aexit(manager, *sys.exc_info()):
        raise
finally:
    if not hit_except:
        await aexit(manager, None, None, None)
@end example

See also @code{__aenter__()} and @code{__aexit__()} for details.

It is a @ref{21a,,SyntaxError} to use an @code{async with} statement outside the
body of a coroutine function.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 492

@item PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} - Coroutines with async and await syntax

The proposal that made coroutines a proper standalone concept in Python,
and added supporting syntax.
@end table


@node Top-level components,Full Grammar specification,Compound statements,The Python Language Reference
@anchor{reference/toplevel_components doc}@anchor{1a17}@anchor{reference/toplevel_components top-level}@anchor{1a18}@anchor{reference/toplevel_components top-level-components}@anchor{1a19}
@section Top-level components


@geindex interpreter

The Python interpreter can get its input from a number of sources: from a script
passed to it as standard input or as program argument, typed in interactively,
from a module source file, etc.  This chapter gives the syntax used in these
cases.

@menu
* Complete Python programs:: 
* File input:: 
* Interactive input:: 
* Expression input:: 

@end menu

@node Complete Python programs,File input,,Top-level components
@anchor{reference/toplevel_components complete-python-programs}@anchor{1a1a}@anchor{reference/toplevel_components programs}@anchor{185a}
@subsection Complete Python programs


@geindex program

@geindex module; sys
@geindex module; __main__
@geindex module; builtins

While a language specification need not prescribe how the language interpreter
is invoked, it is useful to have a notion of a complete Python program.  A
complete Python program is executed in a minimally initialized environment: all
built-in and standard modules are available, but none have been initialized,
except for @ref{f7,,sys} (various system services), @ref{11,,builtins} (built-in
functions, exceptions and @code{None}) and @ref{1,,__main__}.  The latter is used to
provide the local and global namespace for execution of the complete program.

The syntax for a complete Python program is that for file input, described in
the next section.

@geindex interactive mode
@geindex module; __main__

The interpreter may also be invoked in interactive mode; in this case, it does
not read and execute a complete program but reads and executes one statement
(possibly compound) at a time.  The initial environment is identical to that of
a complete program; each statement is executed in the namespace of
@ref{1,,__main__}.

@geindex UNIX
@geindex Windows
@geindex command line
@geindex standard input

A complete program can be passed to the interpreter
in three forms: with the @ref{1180,,-c} @emph{string} command line option, as a file
passed as the first command line argument, or as standard input.  If the file
or standard input is a tty device, the interpreter enters interactive mode;
otherwise, it executes the file as a complete program.

@node File input,Interactive input,Complete Python programs,Top-level components
@anchor{reference/toplevel_components file-input}@anchor{1a1b}@anchor{reference/toplevel_components id1}@anchor{1a1c}
@subsection File input


All input read from non-interactive files has the same form:

@example
@anchor{reference/toplevel_components grammar-token-python-grammar-file_input}@anchor{1a1d}@anchor{reference/toplevel_components grammar-token-file-input}@anchor{1a1e}file_input ::= (NEWLINE | statement)*
@end example

This syntax is used in the following situations:


@itemize *

@item 
when parsing a complete Python program (from a file or from a string);

@item 
when parsing a module;

@item 
when parsing a string passed to the @ref{34b,,exec()} function;
@end itemize

@node Interactive input,Expression input,File input,Top-level components
@anchor{reference/toplevel_components interactive}@anchor{1a1f}@anchor{reference/toplevel_components interactive-input}@anchor{1a20}
@subsection Interactive input


Input in interactive mode is parsed using the following grammar:

@example
@anchor{reference/toplevel_components grammar-token-python-grammar-interactive_input}@anchor{1a21}@anchor{reference/toplevel_components grammar-token-interactive-input}@anchor{1a22}interactive_input ::= [stmt_list] NEWLINE | compound_stmt NEWLINE
@end example

Note that a (top-level) compound statement must be followed by a blank line in
interactive mode; this is needed to help the parser detect the end of the input.

@node Expression input,,Interactive input,Top-level components
@anchor{reference/toplevel_components expression-input}@anchor{1a23}@anchor{reference/toplevel_components id2}@anchor{1a24}
@subsection Expression input


@geindex input

@geindex built-in function; eval

@ref{34a,,eval()} is used for expression input.  It ignores leading whitespace. The
string argument to @ref{34a,,eval()} must have the following form:

@example
@anchor{reference/toplevel_components grammar-token-python-grammar-eval_input}@anchor{1a25}@anchor{reference/toplevel_components grammar-token-eval-input}@anchor{1a26}eval_input ::= expression_list NEWLINE*
@end example

@node Full Grammar specification,,Top-level components,The Python Language Reference
@anchor{reference/grammar doc}@anchor{19bd}@anchor{reference/grammar full-grammar-specification}@anchor{1a27}
@section Full Grammar specification


This is the full Python grammar, derived directly from the grammar
used to generate the CPython parser (see Grammar/python.gram@footnote{https://github.com/python/cpython/tree/main/Grammar/python.gram}).
The version here omits details related to code generation and
error recovery.

The notation is a mixture of EBNF@footnote{https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form}
and PEG@footnote{https://en.wikipedia.org/wiki/Parsing_expression_grammar}.
In particular, @code{&} followed by a symbol, token or parenthesized
group indicates a positive lookahead (i.e., is required to match but
not consumed), while @code{!} indicates a negative lookahead (i.e., is
required _not_ to match).  We use the @code{|} separator to mean PEG’s
“ordered choice” (written as @code{/} in traditional PEG grammars). See
@geindex Python Enhancement Proposals; PEP 617
PEP 617@footnote{https://www.python.org/dev/peps/pep-0617} for more details on the grammar’s syntax.

@example
# PEG grammar for Python

@@trailer '@w{'}'
void *
_PyPegen_parse(Parser *p)
@{
    // Initialize keywords
    p->keywords = reserved_keywords;
    p->n_keyword_lists = n_keyword_lists;
    p->soft_keywords = soft_keywords;

    // Run parser
    void *result = NULL;
    if (p->start_rule == Py_file_input) @{
        result = file_rule(p);
    @} else if (p->start_rule == Py_single_input) @{
        result = interactive_rule(p);
    @} else if (p->start_rule == Py_eval_input) @{
        result = eval_rule(p);
    @} else if (p->start_rule == Py_func_type_input) @{
        result = func_type_rule(p);
    @} else if (p->start_rule == Py_fstring_input) @{
        result = fstring_rule(p);
    @}

    return result;
@}
'@w{'}'

# ========================= START OF THE GRAMMAR =========================

# General grammatical elements and rules:
#
# * Strings with double quotes (") denote SOFT KEYWORDS
# * Strings with single quotes (') denote KEYWORDS
# * Upper case names (NAME) denote tokens in the Grammar/Tokens file
# * Rule names starting with "invalid_" are used for specialized syntax errors
#     - These rules are NOT used in the first pass of the parser.
#     - Only if the first pass fails to parse, a second pass including the invalid
#       rules will be executed.
#     - If the parser fails in the second phase with a generic syntax error, the
#       location of the generic failure of the first pass will be used (this avoids
#       reporting incorrect locations due to the invalid rules).
#     - The order of the alternatives involving invalid rules matter
#       (like any rule in PEG).
#
# Grammar Syntax (see PEP 617 for more information):
#
# rule_name: expression
#   Optionally, a type can be included right after the rule name, which
#   specifies the return type of the C or Python function corresponding to the
#   rule:
# rule_name[return_type]: expression
#   If the return type is omitted, then a void * is returned in C and an Any in
#   Python.
# e1 e2
#   Match e1, then match e2.
# e1 | e2
#   Match e1 or e2.
#   The first alternative can also appear on the line after the rule name for
#   formatting purposes. In that case, a | must be used before the first
#   alternative, like so:
#       rule_name[return_type]:
#            | first_alt
#            | second_alt
# ( e )
#   Match e (allows also to use other operators in the group like '(e)*')
# [ e ] or e?
#   Optionally match e.
# e*
#   Match zero or more occurrences of e.
# e+
#   Match one or more occurrences of e.
# s.e+
#   Match one or more occurrences of e, separated by s. The generated parse tree
#   does not include the separator. This is otherwise identical to (e (s e)*).
# &e
#   Succeed if e can be parsed, without consuming any input.
# !e
#   Fail if e can be parsed, without consuming any input.
# ~
#   Commit to the current alternative, even if it fails to parse.
#

# STARTING RULES
# ==============

file[mod_ty]: a=[statements] ENDMARKER @{ _PyPegen_make_module(p, a) @}
interactive[mod_ty]: a=statement_newline @{ _PyAST_Interactive(a, p->arena) @}
eval[mod_ty]: a=expressions NEWLINE* ENDMARKER @{ _PyAST_Expression(a, p->arena) @}
func_type[mod_ty]: '(' a=[type_expressions] ')' '->' b=expression NEWLINE* ENDMARKER @{ _PyAST_FunctionType(a, b, p->arena) @}
fstring[expr_ty]: star_expressions

# GENERAL STATEMENTS
# ==================

statements[asdl_stmt_seq*]: a=statement+ @{ (asdl_stmt_seq*)_PyPegen_seq_flatten(p, a) @}

statement[asdl_stmt_seq*]: a=compound_stmt @{ (asdl_stmt_seq*)_PyPegen_singleton_seq(p, a) @} | a[asdl_stmt_seq*]=simple_stmts @{ a @}

statement_newline[asdl_stmt_seq*]:
    | a=compound_stmt NEWLINE @{ (asdl_stmt_seq*)_PyPegen_singleton_seq(p, a) @}
    | simple_stmts
    | NEWLINE @{ (asdl_stmt_seq*)_PyPegen_singleton_seq(p, CHECK(stmt_ty, _PyAST_Pass(EXTRA))) @}
    | ENDMARKER @{ _PyPegen_interactive_exit(p) @}

simple_stmts[asdl_stmt_seq*]:
    | a=simple_stmt !';' NEWLINE @{ (asdl_stmt_seq*)_PyPegen_singleton_seq(p, a) @} # Not needed, there for speedup
    | a[asdl_stmt_seq*]=';'.simple_stmt+ [';'] NEWLINE @{ a @}

# NOTE: assignment MUST precede expression, else parsing a simple assignment
# will throw a SyntaxError.
simple_stmt[stmt_ty] (memo):
    | assignment
    | e=star_expressions @{ _PyAST_Expr(e, EXTRA) @}
    | &'return' return_stmt
    | &('import' | 'from') import_stmt
    | &'raise' raise_stmt
    | 'pass' @{ _PyAST_Pass(EXTRA) @}
    | &'del' del_stmt
    | &'yield' yield_stmt
    | &'assert' assert_stmt
    | 'break' @{ _PyAST_Break(EXTRA) @}
    | 'continue' @{ _PyAST_Continue(EXTRA) @}
    | &'global' global_stmt
    | &'nonlocal' nonlocal_stmt

compound_stmt[stmt_ty]:
    | &('def' | '@@' | ASYNC) function_def
    | &'if' if_stmt
    | &('class' | '@@') class_def
    | &('with' | ASYNC) with_stmt
    | &('for' | ASYNC) for_stmt
    | &'try' try_stmt
    | &'while' while_stmt
    | match_stmt

# SIMPLE STATEMENTS
# =================

# NOTE: annotated_rhs may start with 'yield'; yield_expr must start with 'yield'
assignment[stmt_ty]:
    | a=NAME ':' b=expression c=['=' d=annotated_rhs @{ d @}] @{
        CHECK_VERSION(
            stmt_ty,
            6,
            "Variable annotation syntax is",
            _PyAST_AnnAssign(CHECK(expr_ty, _PyPegen_set_expr_context(p, a, Store)), b, c, 1, EXTRA)
        ) @}
    | a=('(' b=single_target ')' @{ b @}
         | single_subscript_attribute_target) ':' b=expression c=['=' d=annotated_rhs @{ d @}] @{
        CHECK_VERSION(stmt_ty, 6, "Variable annotations syntax is", _PyAST_AnnAssign(a, b, c, 0, EXTRA)) @}
    | a[asdl_expr_seq*]=(z=star_targets '=' @{ z @})+ b=(yield_expr | star_expressions) !'=' tc=[TYPE_COMMENT] @{
         _PyAST_Assign(a, b, NEW_TYPE_COMMENT(p, tc), EXTRA) @}
    | a=single_target b=augassign ~ c=(yield_expr | star_expressions) @{
         _PyAST_AugAssign(a, b->kind, c, EXTRA) @}
    | invalid_assignment

annotated_rhs[expr_ty]: yield_expr | star_expressions

augassign[AugOperator*]:
    | '+=' @{ _PyPegen_augoperator(p, Add) @}
    | '-=' @{ _PyPegen_augoperator(p, Sub) @}
    | '*=' @{ _PyPegen_augoperator(p, Mult) @}
    | '@@=' @{ CHECK_VERSION(AugOperator*, 5, "The '@@' operator is", _PyPegen_augoperator(p, MatMult)) @}
    | '/=' @{ _PyPegen_augoperator(p, Div) @}
    | '%=' @{ _PyPegen_augoperator(p, Mod) @}
    | '&=' @{ _PyPegen_augoperator(p, BitAnd) @}
    | '|=' @{ _PyPegen_augoperator(p, BitOr) @}
    | '^=' @{ _PyPegen_augoperator(p, BitXor) @}
    | '<<=' @{ _PyPegen_augoperator(p, LShift) @}
    | '>>=' @{ _PyPegen_augoperator(p, RShift) @}
    | '**=' @{ _PyPegen_augoperator(p, Pow) @}
    | '//=' @{ _PyPegen_augoperator(p, FloorDiv) @}

return_stmt[stmt_ty]:
    | 'return' a=[star_expressions] @{ _PyAST_Return(a, EXTRA) @}

raise_stmt[stmt_ty]:
    | 'raise' a=expression b=['from' z=expression @{ z @}] @{ _PyAST_Raise(a, b, EXTRA) @}
    | 'raise' @{ _PyAST_Raise(NULL, NULL, EXTRA) @}

global_stmt[stmt_ty]: 'global' a[asdl_expr_seq*]=','.NAME+ @{
    _PyAST_Global(CHECK(asdl_identifier_seq*, _PyPegen_map_names_to_ids(p, a)), EXTRA) @}

nonlocal_stmt[stmt_ty]: 'nonlocal' a[asdl_expr_seq*]=','.NAME+ @{
    _PyAST_Nonlocal(CHECK(asdl_identifier_seq*, _PyPegen_map_names_to_ids(p, a)), EXTRA) @}

del_stmt[stmt_ty]:
    | 'del' a=del_targets &(';' | NEWLINE) @{ _PyAST_Delete(a, EXTRA) @}
    | invalid_del_stmt

yield_stmt[stmt_ty]: y=yield_expr @{ _PyAST_Expr(y, EXTRA) @}

assert_stmt[stmt_ty]: 'assert' a=expression b=[',' z=expression @{ z @}] @{ _PyAST_Assert(a, b, EXTRA) @}

import_stmt[stmt_ty]: import_name | import_from

# Import statements
# -----------------

import_name[stmt_ty]: 'import' a=dotted_as_names @{ _PyAST_Import(a, EXTRA) @}
# note below: the ('.' | '...') is necessary because '...' is tokenized as ELLIPSIS
import_from[stmt_ty]:
    | 'from' a=('.' | '...')* b=dotted_name 'import' c=import_from_targets @{
        _PyAST_ImportFrom(b->v.Name.id, c, _PyPegen_seq_count_dots(a), EXTRA) @}
    | 'from' a=('.' | '...')+ 'import' b=import_from_targets @{
        _PyAST_ImportFrom(NULL, b, _PyPegen_seq_count_dots(a), EXTRA) @}
import_from_targets[asdl_alias_seq*]:
    | '(' a=import_from_as_names [','] ')' @{ a @}
    | import_from_as_names !','
    | '*' @{ (asdl_alias_seq*)_PyPegen_singleton_seq(p, CHECK(alias_ty, _PyPegen_alias_for_star(p, EXTRA))) @}
    | invalid_import_from_targets
import_from_as_names[asdl_alias_seq*]:
    | a[asdl_alias_seq*]=','.import_from_as_name+ @{ a @}
import_from_as_name[alias_ty]:
    | a=NAME b=['as' z=NAME @{ z @}] @{ _PyAST_alias(a->v.Name.id,
                                               (b) ? ((expr_ty) b)->v.Name.id : NULL,
                                               EXTRA) @}
dotted_as_names[asdl_alias_seq*]:
    | a[asdl_alias_seq*]=','.dotted_as_name+ @{ a @}
dotted_as_name[alias_ty]:
    | a=dotted_name b=['as' z=NAME @{ z @}] @{ _PyAST_alias(a->v.Name.id,
                                                      (b) ? ((expr_ty) b)->v.Name.id : NULL,
                                                      EXTRA) @}
dotted_name[expr_ty]:
    | a=dotted_name '.' b=NAME @{ _PyPegen_join_names_with_dot(p, a, b) @}
    | NAME

# COMPOUND STATEMENTS
# ===================

# Common elements
# ---------------

block[asdl_stmt_seq*] (memo):
    | NEWLINE INDENT a=statements DEDENT @{ a @}
    | simple_stmts
    | invalid_block

decorators[asdl_expr_seq*]: a[asdl_expr_seq*]=('@@' f=named_expression NEWLINE @{ f @})+ @{ a @}

# Class definitions
# -----------------

class_def[stmt_ty]:
    | a=decorators b=class_def_raw @{ _PyPegen_class_def_decorators(p, a, b) @}
    | class_def_raw

class_def_raw[stmt_ty]:
    | invalid_class_def_raw
    | 'class' a=NAME b=['(' z=[arguments] ')' @{ z @}] &&':' c=block @{
        _PyAST_ClassDef(a->v.Name.id,
                     (b) ? ((expr_ty) b)->v.Call.args : NULL,
                     (b) ? ((expr_ty) b)->v.Call.keywords : NULL,
                     c, NULL, EXTRA) @}

# Function definitions
# --------------------

function_def[stmt_ty]:
    | d=decorators f=function_def_raw @{ _PyPegen_function_def_decorators(p, d, f) @}
    | function_def_raw

function_def_raw[stmt_ty]:
    | invalid_def_raw
    | 'def' n=NAME &&'(' params=[params] ')' a=['->' z=expression @{ z @}] &&':' tc=[func_type_comment] b=block @{
        _PyAST_FunctionDef(n->v.Name.id,
                        (params) ? params : CHECK(arguments_ty, _PyPegen_empty_arguments(p)),
                        b, NULL, a, NEW_TYPE_COMMENT(p, tc), EXTRA) @}
    | ASYNC 'def' n=NAME &&'(' params=[params] ')' a=['->' z=expression @{ z @}] &&':' tc=[func_type_comment] b=block @{
        CHECK_VERSION(
            stmt_ty,
            5,
            "Async functions are",
            _PyAST_AsyncFunctionDef(n->v.Name.id,
                            (params) ? params : CHECK(arguments_ty, _PyPegen_empty_arguments(p)),
                            b, NULL, a, NEW_TYPE_COMMENT(p, tc), EXTRA)
        ) @}

# Function parameters
# -------------------

params[arguments_ty]:
    | invalid_parameters
    | parameters

parameters[arguments_ty]:
    | a=slash_no_default b[asdl_arg_seq*]=param_no_default* c=param_with_default* d=[star_etc] @{
        _PyPegen_make_arguments(p, a, NULL, b, c, d) @}
    | a=slash_with_default b=param_with_default* c=[star_etc] @{
        _PyPegen_make_arguments(p, NULL, a, NULL, b, c) @}
    | a[asdl_arg_seq*]=param_no_default+ b=param_with_default* c=[star_etc] @{
        _PyPegen_make_arguments(p, NULL, NULL, a, b, c) @}
    | a=param_with_default+ b=[star_etc] @{ _PyPegen_make_arguments(p, NULL, NULL, NULL, a, b)@}
    | a=star_etc @{ _PyPegen_make_arguments(p, NULL, NULL, NULL, NULL, a) @}

# Some duplication here because we can't write (',' | &')'),
# which is because we don't support empty alternatives (yet).

slash_no_default[asdl_arg_seq*]:
    | a[asdl_arg_seq*]=param_no_default+ '/' ',' @{ a @}
    | a[asdl_arg_seq*]=param_no_default+ '/' &')' @{ a @}
slash_with_default[SlashWithDefault*]:
    | a=param_no_default* b=param_with_default+ '/' ',' @{ _PyPegen_slash_with_default(p, (asdl_arg_seq *)a, b) @}
    | a=param_no_default* b=param_with_default+ '/' &')' @{ _PyPegen_slash_with_default(p, (asdl_arg_seq *)a, b) @}

star_etc[StarEtc*]:
    | '*' a=param_no_default b=param_maybe_default* c=[kwds] @{
        _PyPegen_star_etc(p, a, b, c) @}
    | '*' ',' b=param_maybe_default+ c=[kwds] @{
        _PyPegen_star_etc(p, NULL, b, c) @}
    | a=kwds @{ _PyPegen_star_etc(p, NULL, NULL, a) @}
    | invalid_star_etc

kwds[arg_ty]: '**' a=param_no_default @{ a @}

# One parameter.  This *includes* a following comma and type comment.
#
# There are three styles:
# - No default
# - With default
# - Maybe with default
#
# There are two alternative forms of each, to deal with type comments:
# - Ends in a comma followed by an optional type comment
# - No comma, optional type comment, must be followed by close paren
# The latter form is for a final parameter without trailing comma.
#

param_no_default[arg_ty]:
    | a=param ',' tc=TYPE_COMMENT? @{ _PyPegen_add_type_comment_to_arg(p, a, tc) @}
    | a=param tc=TYPE_COMMENT? &')' @{ _PyPegen_add_type_comment_to_arg(p, a, tc) @}
param_with_default[NameDefaultPair*]:
    | a=param c=default ',' tc=TYPE_COMMENT? @{ _PyPegen_name_default_pair(p, a, c, tc) @}
    | a=param c=default tc=TYPE_COMMENT? &')' @{ _PyPegen_name_default_pair(p, a, c, tc) @}
param_maybe_default[NameDefaultPair*]:
    | a=param c=default? ',' tc=TYPE_COMMENT? @{ _PyPegen_name_default_pair(p, a, c, tc) @}
    | a=param c=default? tc=TYPE_COMMENT? &')' @{ _PyPegen_name_default_pair(p, a, c, tc) @}
param[arg_ty]: a=NAME b=annotation? @{ _PyAST_arg(a->v.Name.id, b, NULL, EXTRA) @}
annotation[expr_ty]: ':' a=expression @{ a @}
default[expr_ty]: '=' a=expression @{ a @}

# If statement
# ------------

if_stmt[stmt_ty]:
    | invalid_if_stmt
    | 'if' a=named_expression ':' b=block c=elif_stmt @{
        _PyAST_If(a, b, CHECK(asdl_stmt_seq*, _PyPegen_singleton_seq(p, c)), EXTRA) @}
    | 'if' a=named_expression ':' b=block c=[else_block] @{ _PyAST_If(a, b, c, EXTRA) @}
elif_stmt[stmt_ty]:
    | invalid_elif_stmt
    | 'elif' a=named_expression ':' b=block c=elif_stmt @{
        _PyAST_If(a, b, CHECK(asdl_stmt_seq*, _PyPegen_singleton_seq(p, c)), EXTRA) @}
    | 'elif' a=named_expression ':' b=block c=[else_block] @{ _PyAST_If(a, b, c, EXTRA) @}
else_block[asdl_stmt_seq*]:
    | invalid_else_stmt
    | 'else' &&':' b=block @{ b @}

# While statement
# ---------------

while_stmt[stmt_ty]:
    | invalid_while_stmt
    | 'while' a=named_expression ':' b=block c=[else_block] @{ _PyAST_While(a, b, c, EXTRA) @}

# For statement
# -------------

for_stmt[stmt_ty]:
    | invalid_for_stmt
    | 'for' t=star_targets 'in' ~ ex=star_expressions &&':' tc=[TYPE_COMMENT] b=block el=[else_block] @{
        _PyAST_For(t, ex, b, el, NEW_TYPE_COMMENT(p, tc), EXTRA) @}
    | ASYNC 'for' t=star_targets 'in' ~ ex=star_expressions &&':' tc=[TYPE_COMMENT] b=block el=[else_block] @{
        CHECK_VERSION(stmt_ty, 5, "Async for loops are", _PyAST_AsyncFor(t, ex, b, el, NEW_TYPE_COMMENT(p, tc), EXTRA)) @}
    | invalid_for_target

# With statement
# --------------

with_stmt[stmt_ty]:
    | invalid_with_stmt_indent
    | 'with' '(' a[asdl_withitem_seq*]=','.with_item+ ','? ')' ':' b=block @{
        _PyAST_With(a, b, NULL, EXTRA) @}
    | 'with' a[asdl_withitem_seq*]=','.with_item+ ':' tc=[TYPE_COMMENT] b=block @{
        _PyAST_With(a, b, NEW_TYPE_COMMENT(p, tc), EXTRA) @}
    | ASYNC 'with' '(' a[asdl_withitem_seq*]=','.with_item+ ','? ')' ':' b=block @{
       CHECK_VERSION(stmt_ty, 5, "Async with statements are", _PyAST_AsyncWith(a, b, NULL, EXTRA)) @}
    | ASYNC 'with' a[asdl_withitem_seq*]=','.with_item+ ':' tc=[TYPE_COMMENT] b=block @{
       CHECK_VERSION(stmt_ty, 5, "Async with statements are", _PyAST_AsyncWith(a, b, NEW_TYPE_COMMENT(p, tc), EXTRA)) @}
    | invalid_with_stmt

with_item[withitem_ty]:
    | e=expression 'as' t=star_target &(',' | ')' | ':') @{ _PyAST_withitem(e, t, p->arena) @}
    | invalid_with_item
    | e=expression @{ _PyAST_withitem(e, NULL, p->arena) @}

# Try statement
# -------------

try_stmt[stmt_ty]:
    | invalid_try_stmt
    | 'try' &&':' b=block f=finally_block @{ _PyAST_Try(b, NULL, NULL, f, EXTRA) @}
    | 'try' &&':' b=block ex[asdl_excepthandler_seq*]=except_block+ el=[else_block] f=[finally_block] @{ _PyAST_Try(b, ex, el, f, EXTRA) @}
    | 'try' &&':' b=block ex[asdl_excepthandler_seq*]=except_star_block+ el=[else_block] f=[finally_block] @{ _PyAST_TryStar(b, ex, el, f, EXTRA) @}


# Except statement
# ----------------

except_block[excepthandler_ty]:
    | invalid_except_stmt_indent
    | 'except' e=expression t=['as' z=NAME @{ z @}] ':' b=block @{
        _PyAST_ExceptHandler(e, (t) ? ((expr_ty) t)->v.Name.id : NULL, b, EXTRA) @}
    | 'except' ':' b=block @{ _PyAST_ExceptHandler(NULL, NULL, b, EXTRA) @}
    | invalid_except_stmt
except_star_block[excepthandler_ty]:
    | invalid_except_star_stmt_indent
    | 'except' '*' e=expression t=['as' z=NAME @{ z @}] ':' b=block @{
        _PyAST_ExceptHandler(e, (t) ? ((expr_ty) t)->v.Name.id : NULL, b, EXTRA) @}
    | invalid_except_stmt
finally_block[asdl_stmt_seq*]:
    | invalid_finally_stmt
    | 'finally' &&':' a=block @{ a @}

# Match statement
# ---------------

match_stmt[stmt_ty]:
    | "match" subject=subject_expr ':' NEWLINE INDENT cases[asdl_match_case_seq*]=case_block+ DEDENT @{
        CHECK_VERSION(stmt_ty, 10, "Pattern matching is", _PyAST_Match(subject, cases, EXTRA)) @}
    | invalid_match_stmt

subject_expr[expr_ty]:
    | value=star_named_expression ',' values=star_named_expressions? @{
        _PyAST_Tuple(CHECK(asdl_expr_seq*, _PyPegen_seq_insert_in_front(p, value, values)), Load, EXTRA) @}
    | named_expression

case_block[match_case_ty]:
    | invalid_case_block
    | "case" pattern=patterns guard=guard? ':' body=block @{
        _PyAST_match_case(pattern, guard, body, p->arena) @}

guard[expr_ty]: 'if' guard=named_expression @{ guard @}

patterns[pattern_ty]:
    | patterns[asdl_pattern_seq*]=open_sequence_pattern @{
        _PyAST_MatchSequence(patterns, EXTRA) @}
    | pattern

pattern[pattern_ty]:
    | as_pattern
    | or_pattern

as_pattern[pattern_ty]:
    | pattern=or_pattern 'as' target=pattern_capture_target @{
        _PyAST_MatchAs(pattern, target->v.Name.id, EXTRA) @}
    | invalid_as_pattern

or_pattern[pattern_ty]:
    | patterns[asdl_pattern_seq*]='|'.closed_pattern+ @{
        asdl_seq_LEN(patterns) == 1 ? asdl_seq_GET(patterns, 0) : _PyAST_MatchOr(patterns, EXTRA) @}

closed_pattern[pattern_ty]:
    | literal_pattern
    | capture_pattern
    | wildcard_pattern
    | value_pattern
    | group_pattern
    | sequence_pattern
    | mapping_pattern
    | class_pattern

# Literal patterns are used for equality and identity constraints
literal_pattern[pattern_ty]:
    | value=signed_number !('+' | '-') @{ _PyAST_MatchValue(value, EXTRA) @}
    | value=complex_number @{ _PyAST_MatchValue(value, EXTRA) @}
    | value=strings @{ _PyAST_MatchValue(value, EXTRA) @}
    | 'None' @{ _PyAST_MatchSingleton(Py_None, EXTRA) @}
    | 'True' @{ _PyAST_MatchSingleton(Py_True, EXTRA) @}
    | 'False' @{ _PyAST_MatchSingleton(Py_False, EXTRA) @}

# Literal expressions are used to restrict permitted mapping pattern keys
literal_expr[expr_ty]:
    | signed_number !('+' | '-')
    | complex_number
    | strings
    | 'None' @{ _PyAST_Constant(Py_None, NULL, EXTRA) @}
    | 'True' @{ _PyAST_Constant(Py_True, NULL, EXTRA) @}
    | 'False' @{ _PyAST_Constant(Py_False, NULL, EXTRA) @}

complex_number[expr_ty]:
    | real=signed_real_number '+' imag=imaginary_number @{
        _PyAST_BinOp(real, Add, imag, EXTRA) @}
    | real=signed_real_number '-' imag=imaginary_number  @{
        _PyAST_BinOp(real, Sub, imag, EXTRA) @}

signed_number[expr_ty]:
    | NUMBER
    | '-' number=NUMBER @{ _PyAST_UnaryOp(USub, number, EXTRA) @}

signed_real_number[expr_ty]:
    | real_number
    | '-' real=real_number @{ _PyAST_UnaryOp(USub, real, EXTRA) @}

real_number[expr_ty]:
    | real=NUMBER @{ _PyPegen_ensure_real(p, real) @}

imaginary_number[expr_ty]:
    | imag=NUMBER @{ _PyPegen_ensure_imaginary(p, imag) @}

capture_pattern[pattern_ty]:
    | target=pattern_capture_target @{ _PyAST_MatchAs(NULL, target->v.Name.id, EXTRA) @}

pattern_capture_target[expr_ty]:
    | !"_" name=NAME !('.' | '(' | '=') @{
        _PyPegen_set_expr_context(p, name, Store) @}

wildcard_pattern[pattern_ty]:
    | "_" @{ _PyAST_MatchAs(NULL, NULL, EXTRA) @}

value_pattern[pattern_ty]:
    | attr=attr !('.' | '(' | '=') @{ _PyAST_MatchValue(attr, EXTRA) @}

attr[expr_ty]:
    | value=name_or_attr '.' attr=NAME @{
        _PyAST_Attribute(value, attr->v.Name.id, Load, EXTRA) @}

name_or_attr[expr_ty]:
    | attr
    | NAME

group_pattern[pattern_ty]:
    | '(' pattern=pattern ')' @{ pattern @}

sequence_pattern[pattern_ty]:
    | '[' patterns=maybe_sequence_pattern? ']' @{ _PyAST_MatchSequence(patterns, EXTRA) @}
    | '(' patterns=open_sequence_pattern? ')' @{ _PyAST_MatchSequence(patterns, EXTRA) @}

open_sequence_pattern[asdl_seq*]:
    | pattern=maybe_star_pattern ',' patterns=maybe_sequence_pattern? @{
        _PyPegen_seq_insert_in_front(p, pattern, patterns) @}

maybe_sequence_pattern[asdl_seq*]:
    | patterns=','.maybe_star_pattern+ ','? @{ patterns @}

maybe_star_pattern[pattern_ty]:
    | star_pattern
    | pattern

star_pattern[pattern_ty]:
    | '*' target=pattern_capture_target @{
        _PyAST_MatchStar(target->v.Name.id, EXTRA) @}
    | '*' wildcard_pattern @{
        _PyAST_MatchStar(NULL, EXTRA) @}

mapping_pattern[pattern_ty]:
    | '@{' '@}' @{
        _PyAST_MatchMapping(NULL, NULL, NULL, EXTRA) @}
    | '@{' rest=double_star_pattern ','? '@}' @{
        _PyAST_MatchMapping(NULL, NULL, rest->v.Name.id, EXTRA) @}
    | '@{' items=items_pattern ',' rest=double_star_pattern ','? '@}' @{
        _PyAST_MatchMapping(
            CHECK(asdl_expr_seq*, _PyPegen_get_pattern_keys(p, items)),
            CHECK(asdl_pattern_seq*, _PyPegen_get_patterns(p, items)),
            rest->v.Name.id,
            EXTRA) @}
    | '@{' items=items_pattern ','? '@}' @{
        _PyAST_MatchMapping(
            CHECK(asdl_expr_seq*, _PyPegen_get_pattern_keys(p, items)),
            CHECK(asdl_pattern_seq*, _PyPegen_get_patterns(p, items)),
            NULL,
            EXTRA) @}

items_pattern[asdl_seq*]:
    | ','.key_value_pattern+

key_value_pattern[KeyPatternPair*]:
    | key=(literal_expr | attr) ':' pattern=pattern @{
        _PyPegen_key_pattern_pair(p, key, pattern) @}

double_star_pattern[expr_ty]:
    | '**' target=pattern_capture_target @{ target @}

class_pattern[pattern_ty]:
    | cls=name_or_attr '(' ')' @{
        _PyAST_MatchClass(cls, NULL, NULL, NULL, EXTRA) @}
    | cls=name_or_attr '(' patterns=positional_patterns ','? ')' @{
        _PyAST_MatchClass(cls, patterns, NULL, NULL, EXTRA) @}
    | cls=name_or_attr '(' keywords=keyword_patterns ','? ')' @{
        _PyAST_MatchClass(
            cls, NULL,
            CHECK(asdl_identifier_seq*, _PyPegen_map_names_to_ids(p,
                CHECK(asdl_expr_seq*, _PyPegen_get_pattern_keys(p, keywords)))),
            CHECK(asdl_pattern_seq*, _PyPegen_get_patterns(p, keywords)),
            EXTRA) @}
    | cls=name_or_attr '(' patterns=positional_patterns ',' keywords=keyword_patterns ','? ')' @{
        _PyAST_MatchClass(
            cls,
            patterns,
            CHECK(asdl_identifier_seq*, _PyPegen_map_names_to_ids(p,
                CHECK(asdl_expr_seq*, _PyPegen_get_pattern_keys(p, keywords)))),
            CHECK(asdl_pattern_seq*, _PyPegen_get_patterns(p, keywords)),
            EXTRA) @}
    | invalid_class_pattern

positional_patterns[asdl_pattern_seq*]:
    | args[asdl_pattern_seq*]=','.pattern+ @{ args @}

keyword_patterns[asdl_seq*]:
    | ','.keyword_pattern+

keyword_pattern[KeyPatternPair*]:
    | arg=NAME '=' value=pattern @{ _PyPegen_key_pattern_pair(p, arg, value) @}

# EXPRESSIONS
# -----------

expressions[expr_ty]:
    | a=expression b=(',' c=expression @{ c @})+ [','] @{
        _PyAST_Tuple(CHECK(asdl_expr_seq*, _PyPegen_seq_insert_in_front(p, a, b)), Load, EXTRA) @}
    | a=expression ',' @{ _PyAST_Tuple(CHECK(asdl_expr_seq*, _PyPegen_singleton_seq(p, a)), Load, EXTRA) @}
    | expression

expression[expr_ty] (memo):
    | invalid_expression
    | invalid_legacy_expression
    | a=disjunction 'if' b=disjunction 'else' c=expression @{ _PyAST_IfExp(b, a, c, EXTRA) @}
    | disjunction
    | lambdef

yield_expr[expr_ty]:
    | 'yield' 'from' a=expression @{ _PyAST_YieldFrom(a, EXTRA) @}
    | 'yield' a=[star_expressions] @{ _PyAST_Yield(a, EXTRA) @}

star_expressions[expr_ty]:
    | a=star_expression b=(',' c=star_expression @{ c @})+ [','] @{
        _PyAST_Tuple(CHECK(asdl_expr_seq*, _PyPegen_seq_insert_in_front(p, a, b)), Load, EXTRA) @}
    | a=star_expression ',' @{ _PyAST_Tuple(CHECK(asdl_expr_seq*, _PyPegen_singleton_seq(p, a)), Load, EXTRA) @}
    | star_expression

star_expression[expr_ty] (memo):
    | '*' a=bitwise_or @{ _PyAST_Starred(a, Load, EXTRA) @}
    | expression

star_named_expressions[asdl_expr_seq*]: a[asdl_expr_seq*]=','.star_named_expression+ [','] @{ a @}

star_named_expression[expr_ty]:
    | '*' a=bitwise_or @{ _PyAST_Starred(a, Load, EXTRA) @}
    | named_expression

assignment_expression[expr_ty]:
    | a=NAME ':=' ~ b=expression @{ _PyAST_NamedExpr(CHECK(expr_ty, _PyPegen_set_expr_context(p, a, Store)), b, EXTRA) @}

named_expression[expr_ty]:
    | assignment_expression
    | invalid_named_expression
    | expression !':='

disjunction[expr_ty] (memo):
    | a=conjunction b=('or' c=conjunction @{ c @})+ @{ _PyAST_BoolOp(
        Or,
        CHECK(asdl_expr_seq*, _PyPegen_seq_insert_in_front(p, a, b)),
        EXTRA) @}
    | conjunction

conjunction[expr_ty] (memo):
    | a=inversion b=('and' c=inversion @{ c @})+ @{ _PyAST_BoolOp(
        And,
        CHECK(asdl_expr_seq*, _PyPegen_seq_insert_in_front(p, a, b)),
        EXTRA) @}
    | inversion

inversion[expr_ty] (memo):
    | 'not' a=inversion @{ _PyAST_UnaryOp(Not, a, EXTRA) @}
    | comparison

# Comparison operators
# --------------------

comparison[expr_ty]:
    | a=bitwise_or b=compare_op_bitwise_or_pair+ @{
        _PyAST_Compare(
            a,
            CHECK(asdl_int_seq*, _PyPegen_get_cmpops(p, b)),
            CHECK(asdl_expr_seq*, _PyPegen_get_exprs(p, b)),
            EXTRA) @}
    | bitwise_or

compare_op_bitwise_or_pair[CmpopExprPair*]:
    | eq_bitwise_or
    | noteq_bitwise_or
    | lte_bitwise_or
    | lt_bitwise_or
    | gte_bitwise_or
    | gt_bitwise_or
    | notin_bitwise_or
    | in_bitwise_or
    | isnot_bitwise_or
    | is_bitwise_or

eq_bitwise_or[CmpopExprPair*]: '==' a=bitwise_or @{ _PyPegen_cmpop_expr_pair(p, Eq, a) @}
noteq_bitwise_or[CmpopExprPair*]:
    | (tok='!=' @{ _PyPegen_check_barry_as_flufl(p, tok) ? NULL : tok@}) a=bitwise_or @{_PyPegen_cmpop_expr_pair(p, NotEq, a) @}
lte_bitwise_or[CmpopExprPair*]: '<=' a=bitwise_or @{ _PyPegen_cmpop_expr_pair(p, LtE, a) @}
lt_bitwise_or[CmpopExprPair*]: '<' a=bitwise_or @{ _PyPegen_cmpop_expr_pair(p, Lt, a) @}
gte_bitwise_or[CmpopExprPair*]: '>=' a=bitwise_or @{ _PyPegen_cmpop_expr_pair(p, GtE, a) @}
gt_bitwise_or[CmpopExprPair*]: '>' a=bitwise_or @{ _PyPegen_cmpop_expr_pair(p, Gt, a) @}
notin_bitwise_or[CmpopExprPair*]: 'not' 'in' a=bitwise_or @{ _PyPegen_cmpop_expr_pair(p, NotIn, a) @}
in_bitwise_or[CmpopExprPair*]: 'in' a=bitwise_or @{ _PyPegen_cmpop_expr_pair(p, In, a) @}
isnot_bitwise_or[CmpopExprPair*]: 'is' 'not' a=bitwise_or @{ _PyPegen_cmpop_expr_pair(p, IsNot, a) @}
is_bitwise_or[CmpopExprPair*]: 'is' a=bitwise_or @{ _PyPegen_cmpop_expr_pair(p, Is, a) @}

# Bitwise operators
# -----------------

bitwise_or[expr_ty]:
    | a=bitwise_or '|' b=bitwise_xor @{ _PyAST_BinOp(a, BitOr, b, EXTRA) @}
    | bitwise_xor

bitwise_xor[expr_ty]:
    | a=bitwise_xor '^' b=bitwise_and @{ _PyAST_BinOp(a, BitXor, b, EXTRA) @}
    | bitwise_and

bitwise_and[expr_ty]:
    | a=bitwise_and '&' b=shift_expr @{ _PyAST_BinOp(a, BitAnd, b, EXTRA) @}
    | shift_expr

shift_expr[expr_ty]:
    | a=shift_expr '<<' b=sum @{ _PyAST_BinOp(a, LShift, b, EXTRA) @}
    | a=shift_expr '>>' b=sum @{ _PyAST_BinOp(a, RShift, b, EXTRA) @}
    | sum

# Arithmetic operators
# --------------------

sum[expr_ty]:
    | a=sum '+' b=term @{ _PyAST_BinOp(a, Add, b, EXTRA) @}
    | a=sum '-' b=term @{ _PyAST_BinOp(a, Sub, b, EXTRA) @}
    | term

term[expr_ty]:
    | a=term '*' b=factor @{ _PyAST_BinOp(a, Mult, b, EXTRA) @}
    | a=term '/' b=factor @{ _PyAST_BinOp(a, Div, b, EXTRA) @}
    | a=term '//' b=factor @{ _PyAST_BinOp(a, FloorDiv, b, EXTRA) @}
    | a=term '%' b=factor @{ _PyAST_BinOp(a, Mod, b, EXTRA) @}
    | a=term '@@' b=factor @{ CHECK_VERSION(expr_ty, 5, "The '@@' operator is", _PyAST_BinOp(a, MatMult, b, EXTRA)) @}
    | factor

factor[expr_ty] (memo):
    | '+' a=factor @{ _PyAST_UnaryOp(UAdd, a, EXTRA) @}
    | '-' a=factor @{ _PyAST_UnaryOp(USub, a, EXTRA) @}
    | '~' a=factor @{ _PyAST_UnaryOp(Invert, a, EXTRA) @}
    | power

power[expr_ty]:
    | a=await_primary '**' b=factor @{ _PyAST_BinOp(a, Pow, b, EXTRA) @}
    | await_primary

# Primary elements
# ----------------

# Primary elements are things like "obj.something.something", "obj[something]", "obj(something)", "obj" ...

await_primary[expr_ty] (memo):
    | AWAIT a=primary @{ CHECK_VERSION(expr_ty, 5, "Await expressions are", _PyAST_Await(a, EXTRA)) @}
    | primary

primary[expr_ty]:
    | a=primary '.' b=NAME @{ _PyAST_Attribute(a, b->v.Name.id, Load, EXTRA) @}
    | a=primary b=genexp @{ _PyAST_Call(a, CHECK(asdl_expr_seq*, (asdl_expr_seq*)_PyPegen_singleton_seq(p, b)), NULL, EXTRA) @}
    | a=primary '(' b=[arguments] ')' @{
        _PyAST_Call(a,
                 (b) ? ((expr_ty) b)->v.Call.args : NULL,
                 (b) ? ((expr_ty) b)->v.Call.keywords : NULL,
                 EXTRA) @}
    | a=primary '[' b=slices ']' @{ _PyAST_Subscript(a, b, Load, EXTRA) @}
    | atom

slices[expr_ty]:
    | a=slice !',' @{ a @}
    | a[asdl_expr_seq*]=','.slice+ [','] @{ _PyAST_Tuple(a, Load, EXTRA) @}

slice[expr_ty]:
    | a=[expression] ':' b=[expression] c=[':' d=[expression] @{ d @}] @{ _PyAST_Slice(a, b, c, EXTRA) @}
    | a=named_expression @{ a @}

atom[expr_ty]:
    | NAME
    | 'True' @{ _PyAST_Constant(Py_True, NULL, EXTRA) @}
    | 'False' @{ _PyAST_Constant(Py_False, NULL, EXTRA) @}
    | 'None' @{ _PyAST_Constant(Py_None, NULL, EXTRA) @}
    | &STRING strings
    | NUMBER
    | &'(' (tuple | group | genexp)
    | &'[' (list | listcomp)
    | &'@{' (dict | set | dictcomp | setcomp)
    | '...' @{ _PyAST_Constant(Py_Ellipsis, NULL, EXTRA) @}

group[expr_ty]:
    | '(' a=(yield_expr | named_expression) ')' @{ a @}
    | invalid_group

# Lambda functions
# ----------------

lambdef[expr_ty]:
    | 'lambda' a=[lambda_params] ':' b=expression @{
        _PyAST_Lambda((a) ? a : CHECK(arguments_ty, _PyPegen_empty_arguments(p)), b, EXTRA) @}

lambda_params[arguments_ty]:
    | invalid_lambda_parameters
    | lambda_parameters

# lambda_parameters etc. duplicates parameters but without annotations
# or type comments, and if there's no comma after a parameter, we expect
# a colon, not a close parenthesis.  (For more, see parameters above.)
#
lambda_parameters[arguments_ty]:
    | a=lambda_slash_no_default b[asdl_arg_seq*]=lambda_param_no_default* c=lambda_param_with_default* d=[lambda_star_etc] @{
        _PyPegen_make_arguments(p, a, NULL, b, c, d) @}
    | a=lambda_slash_with_default b=lambda_param_with_default* c=[lambda_star_etc] @{
        _PyPegen_make_arguments(p, NULL, a, NULL, b, c) @}
    | a[asdl_arg_seq*]=lambda_param_no_default+ b=lambda_param_with_default* c=[lambda_star_etc] @{
        _PyPegen_make_arguments(p, NULL, NULL, a, b, c) @}
    | a=lambda_param_with_default+ b=[lambda_star_etc] @{ _PyPegen_make_arguments(p, NULL, NULL, NULL, a, b)@}
    | a=lambda_star_etc @{ _PyPegen_make_arguments(p, NULL, NULL, NULL, NULL, a) @}

lambda_slash_no_default[asdl_arg_seq*]:
    | a[asdl_arg_seq*]=lambda_param_no_default+ '/' ',' @{ a @}
    | a[asdl_arg_seq*]=lambda_param_no_default+ '/' &':' @{ a @}

lambda_slash_with_default[SlashWithDefault*]:
    | a=lambda_param_no_default* b=lambda_param_with_default+ '/' ',' @{ _PyPegen_slash_with_default(p, (asdl_arg_seq *)a, b) @}
    | a=lambda_param_no_default* b=lambda_param_with_default+ '/' &':' @{ _PyPegen_slash_with_default(p, (asdl_arg_seq *)a, b) @}

lambda_star_etc[StarEtc*]:
    | '*' a=lambda_param_no_default b=lambda_param_maybe_default* c=[lambda_kwds] @{
        _PyPegen_star_etc(p, a, b, c) @}
    | '*' ',' b=lambda_param_maybe_default+ c=[lambda_kwds] @{
        _PyPegen_star_etc(p, NULL, b, c) @}
    | a=lambda_kwds @{ _PyPegen_star_etc(p, NULL, NULL, a) @}
    | invalid_lambda_star_etc

lambda_kwds[arg_ty]: '**' a=lambda_param_no_default @{ a @}

lambda_param_no_default[arg_ty]:
    | a=lambda_param ',' @{ a @}
    | a=lambda_param &':' @{ a @}
lambda_param_with_default[NameDefaultPair*]:
    | a=lambda_param c=default ',' @{ _PyPegen_name_default_pair(p, a, c, NULL) @}
    | a=lambda_param c=default &':' @{ _PyPegen_name_default_pair(p, a, c, NULL) @}
lambda_param_maybe_default[NameDefaultPair*]:
    | a=lambda_param c=default? ',' @{ _PyPegen_name_default_pair(p, a, c, NULL) @}
    | a=lambda_param c=default? &':' @{ _PyPegen_name_default_pair(p, a, c, NULL) @}
lambda_param[arg_ty]: a=NAME @{ _PyAST_arg(a->v.Name.id, NULL, NULL, EXTRA) @}

# LITERALS
# ========

strings[expr_ty] (memo): a=STRING+ @{ _PyPegen_concatenate_strings(p, a) @}

list[expr_ty]:
    | '[' a=[star_named_expressions] ']' @{ _PyAST_List(a, Load, EXTRA) @}

tuple[expr_ty]:
    | '(' a=[y=star_named_expression ',' z=[star_named_expressions] @{ _PyPegen_seq_insert_in_front(p, y, z) @} ] ')' @{
        _PyAST_Tuple(a, Load, EXTRA) @}

set[expr_ty]: '@{' a=star_named_expressions '@}' @{ _PyAST_Set(a, EXTRA) @}

# Dicts
# -----

dict[expr_ty]:
    | '@{' a=[double_starred_kvpairs] '@}' @{
        _PyAST_Dict(
            CHECK(asdl_expr_seq*, _PyPegen_get_keys(p, a)),
            CHECK(asdl_expr_seq*, _PyPegen_get_values(p, a)),
            EXTRA) @}
    | '@{' invalid_double_starred_kvpairs '@}'

double_starred_kvpairs[asdl_seq*]: a=','.double_starred_kvpair+ [','] @{ a @}

double_starred_kvpair[KeyValuePair*]:
    | '**' a=bitwise_or @{ _PyPegen_key_value_pair(p, NULL, a) @}
    | kvpair

kvpair[KeyValuePair*]: a=expression ':' b=expression @{ _PyPegen_key_value_pair(p, a, b) @}

# Comprehensions & Generators
# ---------------------------

for_if_clauses[asdl_comprehension_seq*]:
    | a[asdl_comprehension_seq*]=for_if_clause+ @{ a @}

for_if_clause[comprehension_ty]:
    | ASYNC 'for' a=star_targets 'in' ~ b=disjunction c[asdl_expr_seq*]=('if' z=disjunction @{ z @})* @{
        CHECK_VERSION(comprehension_ty, 6, "Async comprehensions are", _PyAST_comprehension(a, b, c, 1, p->arena)) @}
    | 'for' a=star_targets 'in' ~ b=disjunction c[asdl_expr_seq*]=('if' z=disjunction @{ z @})* @{
        _PyAST_comprehension(a, b, c, 0, p->arena) @}
    | invalid_for_target

listcomp[expr_ty]:
    | '[' a=named_expression b=for_if_clauses ']' @{ _PyAST_ListComp(a, b, EXTRA) @}
    | invalid_comprehension

setcomp[expr_ty]:
    | '@{' a=named_expression b=for_if_clauses '@}' @{ _PyAST_SetComp(a, b, EXTRA) @}
    | invalid_comprehension

genexp[expr_ty]:
    | '(' a=( assignment_expression | expression !':=') b=for_if_clauses ')' @{ _PyAST_GeneratorExp(a, b, EXTRA) @}
    | invalid_comprehension

dictcomp[expr_ty]:
    | '@{' a=kvpair b=for_if_clauses '@}' @{ _PyAST_DictComp(a->key, a->value, b, EXTRA) @}
    | invalid_dict_comprehension

# FUNCTION CALL ARGUMENTS
# =======================

arguments[expr_ty] (memo):
    | a=args [','] &')' @{ a @}
    | invalid_arguments

args[expr_ty]:
    | a[asdl_expr_seq*]=','.(starred_expression | ( assignment_expression | expression !':=') !'=')+ b=[',' k=kwargs @{k@}] @{
        _PyPegen_collect_call_seqs(p, a, b, EXTRA) @}
    | a=kwargs @{ _PyAST_Call(_PyPegen_dummy_name(p),
                          CHECK_NULL_ALLOWED(asdl_expr_seq*, _PyPegen_seq_extract_starred_exprs(p, a)),
                          CHECK_NULL_ALLOWED(asdl_keyword_seq*, _PyPegen_seq_delete_starred_exprs(p, a)),
                          EXTRA) @}

kwargs[asdl_seq*]:
    | a=','.kwarg_or_starred+ ',' b=','.kwarg_or_double_starred+ @{ _PyPegen_join_sequences(p, a, b) @}
    | ','.kwarg_or_starred+
    | ','.kwarg_or_double_starred+

starred_expression[expr_ty]:
    | '*' a=expression @{ _PyAST_Starred(a, Load, EXTRA) @}

kwarg_or_starred[KeywordOrStarred*]:
    | invalid_kwarg
    | a=NAME '=' b=expression @{
        _PyPegen_keyword_or_starred(p, CHECK(keyword_ty, _PyAST_keyword(a->v.Name.id, b, EXTRA)), 1) @}
    | a=starred_expression @{ _PyPegen_keyword_or_starred(p, a, 0) @}

kwarg_or_double_starred[KeywordOrStarred*]:
    | invalid_kwarg
    | a=NAME '=' b=expression @{
        _PyPegen_keyword_or_starred(p, CHECK(keyword_ty, _PyAST_keyword(a->v.Name.id, b, EXTRA)), 1) @}
    | '**' a=expression @{ _PyPegen_keyword_or_starred(p, CHECK(keyword_ty, _PyAST_keyword(NULL, a, EXTRA)), 1) @}

# ASSIGNMENT TARGETS
# ==================

# Generic targets
# ---------------

# NOTE: star_targets may contain *bitwise_or, targets may not.
star_targets[expr_ty]:
    | a=star_target !',' @{ a @}
    | a=star_target b=(',' c=star_target @{ c @})* [','] @{
        _PyAST_Tuple(CHECK(asdl_expr_seq*, _PyPegen_seq_insert_in_front(p, a, b)), Store, EXTRA) @}

star_targets_list_seq[asdl_expr_seq*]: a[asdl_expr_seq*]=','.star_target+ [','] @{ a @}

star_targets_tuple_seq[asdl_expr_seq*]:
    | a=star_target b=(',' c=star_target @{ c @})+ [','] @{ (asdl_expr_seq*) _PyPegen_seq_insert_in_front(p, a, b) @}
    | a=star_target ',' @{ (asdl_expr_seq*) _PyPegen_singleton_seq(p, a) @}

star_target[expr_ty] (memo):
    | '*' a=(!'*' star_target) @{
        _PyAST_Starred(CHECK(expr_ty, _PyPegen_set_expr_context(p, a, Store)), Store, EXTRA) @}
    | target_with_star_atom

target_with_star_atom[expr_ty] (memo):
    | a=t_primary '.' b=NAME !t_lookahead @{ _PyAST_Attribute(a, b->v.Name.id, Store, EXTRA) @}
    | a=t_primary '[' b=slices ']' !t_lookahead @{ _PyAST_Subscript(a, b, Store, EXTRA) @}
    | star_atom

star_atom[expr_ty]:
    | a=NAME @{ _PyPegen_set_expr_context(p, a, Store) @}
    | '(' a=target_with_star_atom ')' @{ _PyPegen_set_expr_context(p, a, Store) @}
    | '(' a=[star_targets_tuple_seq] ')' @{ _PyAST_Tuple(a, Store, EXTRA) @}
    | '[' a=[star_targets_list_seq] ']' @{ _PyAST_List(a, Store, EXTRA) @}

single_target[expr_ty]:
    | single_subscript_attribute_target
    | a=NAME @{ _PyPegen_set_expr_context(p, a, Store) @}
    | '(' a=single_target ')' @{ a @}

single_subscript_attribute_target[expr_ty]:
    | a=t_primary '.' b=NAME !t_lookahead @{ _PyAST_Attribute(a, b->v.Name.id, Store, EXTRA) @}
    | a=t_primary '[' b=slices ']' !t_lookahead @{ _PyAST_Subscript(a, b, Store, EXTRA) @}

t_primary[expr_ty]:
    | a=t_primary '.' b=NAME &t_lookahead @{ _PyAST_Attribute(a, b->v.Name.id, Load, EXTRA) @}
    | a=t_primary '[' b=slices ']' &t_lookahead @{ _PyAST_Subscript(a, b, Load, EXTRA) @}
    | a=t_primary b=genexp &t_lookahead @{
        _PyAST_Call(a, CHECK(asdl_expr_seq*, (asdl_expr_seq*)_PyPegen_singleton_seq(p, b)), NULL, EXTRA) @}
    | a=t_primary '(' b=[arguments] ')' &t_lookahead @{
        _PyAST_Call(a,
                 (b) ? ((expr_ty) b)->v.Call.args : NULL,
                 (b) ? ((expr_ty) b)->v.Call.keywords : NULL,
                 EXTRA) @}
    | a=atom &t_lookahead @{ a @}

t_lookahead: '(' | '[' | '.'

# Targets for del statements
# --------------------------

del_targets[asdl_expr_seq*]: a[asdl_expr_seq*]=','.del_target+ [','] @{ a @}

del_target[expr_ty] (memo):
    | a=t_primary '.' b=NAME !t_lookahead @{ _PyAST_Attribute(a, b->v.Name.id, Del, EXTRA) @}
    | a=t_primary '[' b=slices ']' !t_lookahead @{ _PyAST_Subscript(a, b, Del, EXTRA) @}
    | del_t_atom

del_t_atom[expr_ty]:
    | a=NAME @{ _PyPegen_set_expr_context(p, a, Del) @}
    | '(' a=del_target ')' @{ _PyPegen_set_expr_context(p, a, Del) @}
    | '(' a=[del_targets] ')' @{ _PyAST_Tuple(a, Del, EXTRA) @}
    | '[' a=[del_targets] ']' @{ _PyAST_List(a, Del, EXTRA) @}

# TYPING ELEMENTS
# ---------------

# type_expressions allow */** but ignore them
type_expressions[asdl_expr_seq*]:
    | a=','.expression+ ',' '*' b=expression ',' '**' c=expression @{
        (asdl_expr_seq*)_PyPegen_seq_append_to_end(
            p,
            CHECK(asdl_seq*, _PyPegen_seq_append_to_end(p, a, b)),
            c) @}
    | a=','.expression+ ',' '*' b=expression @{ (asdl_expr_seq*)_PyPegen_seq_append_to_end(p, a, b) @}
    | a=','.expression+ ',' '**' b=expression @{ (asdl_expr_seq*)_PyPegen_seq_append_to_end(p, a, b) @}
    | '*' a=expression ',' '**' b=expression @{
        (asdl_expr_seq*)_PyPegen_seq_append_to_end(
            p,
            CHECK(asdl_seq*, _PyPegen_singleton_seq(p, a)),
            b) @}
    | '*' a=expression @{ (asdl_expr_seq*)_PyPegen_singleton_seq(p, a) @}
    | '**' a=expression @{ (asdl_expr_seq*)_PyPegen_singleton_seq(p, a) @}
    | a[asdl_expr_seq*]=','.expression+ @{a@}

func_type_comment[Token*]:
    | NEWLINE t=TYPE_COMMENT &(NEWLINE INDENT) @{ t @}  # Must be followed by indented block
    | invalid_double_type_comments
    | TYPE_COMMENT

# ========================= END OF THE GRAMMAR ===========================



# ========================= START OF INVALID RULES =======================

# From here on, there are rules for invalid syntax with specialised error messages
invalid_arguments:
    | a=args ',' '*' @{ RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "iterable argument unpacking follows keyword argument unpacking") @}
    | a=expression b=for_if_clauses ',' [args | expression for_if_clauses] @{
        RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, PyPegen_last_item(b, comprehension_ty)->target, "Generator expression must be parenthesized") @}
    | a=NAME b='=' expression for_if_clauses @{
        RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "invalid syntax. Maybe you meant '==' or ':=' instead of '='?")@}
    | a=args b=for_if_clauses @{ _PyPegen_nonparen_genexp_in_call(p, a, b) @}
    | args ',' a=expression b=for_if_clauses @{
        RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, asdl_seq_GET(b, b->size-1)->target, "Generator expression must be parenthesized") @}
    | a=args ',' args @{ _PyPegen_arguments_parsing_error(p, a) @}
invalid_kwarg:
    | a[Token*]=('True'|'False'|'None') b='=' @{
        RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "cannot assign to %s", PyBytes_AS_STRING(a->bytes)) @}
    | a=NAME b='=' expression for_if_clauses @{
        RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "invalid syntax. Maybe you meant '==' or ':=' instead of '='?")@}
    | !(NAME '=') a=expression b='=' @{
        RAISE_SYNTAX_ERROR_KNOWN_RANGE(
            a, b, "expression cannot contain assignment, perhaps you meant \"==\"?") @}

# IMPORTANT: Note that the "_without_invalid" suffix causes the rule to not call invalid rules under it
expression_without_invalid[expr_ty]:
    | a=disjunction 'if' b=disjunction 'else' c=expression @{ _PyAST_IfExp(b, a, c, EXTRA) @}
    | disjunction
    | lambdef
invalid_legacy_expression:
    | a=NAME !'(' b=star_expressions @{
        _PyPegen_check_legacy_stmt(p, a) ? RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b,
            "Missing parentheses in call to '%U'. Did you mean %U(...)?", a->v.Name.id, a->v.Name.id) : NULL@}

invalid_expression:
    # !(NAME STRING) is not matched so we don't show this error with some invalid string prefixes like: kf"dsfsdf"
    # Soft keywords need to also be ignored because they can be parsed as NAME NAME
   | !(NAME STRING | SOFT_KEYWORD) a=disjunction b=expression_without_invalid @{
        _PyPegen_check_legacy_stmt(p, a) ? NULL : p->tokens[p->mark-1]->level == 0 ? NULL :
        RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "invalid syntax. Perhaps you forgot a comma?") @}
   | a=disjunction 'if' b=disjunction !('else'|':') @{ RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "expected 'else' after 'if' expression") @}

invalid_named_expression(memo):
    | a=expression ':=' expression @{
        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(
            a, "cannot use assignment expressions with %s", _PyPegen_get_expr_name(a)) @}
    | a=NAME '=' b=bitwise_or !('='|':=') @{
        RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "invalid syntax. Maybe you meant '==' or ':=' instead of '='?") @}
    | !(list|tuple|genexp|'True'|'None'|'False') a=bitwise_or b='=' bitwise_or !('='|':=') @{
        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "cannot assign to %s here. Maybe you meant '==' instead of '='?",
                                          _PyPegen_get_expr_name(a)) @}

invalid_assignment:
    | a=invalid_ann_assign_target ':' expression @{
        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(
            a,
            "only single target (not %s) can be annotated",
            _PyPegen_get_expr_name(a)
        )@}
    | a=star_named_expression ',' star_named_expressions* ':' expression @{
        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "only single target (not tuple) can be annotated") @}
    | a=expression ':' expression @{
        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "illegal target for annotation") @}
    | (star_targets '=')* a=star_expressions '=' @{
        RAISE_SYNTAX_ERROR_INVALID_TARGET(STAR_TARGETS, a) @}
    | (star_targets '=')* a=yield_expr '=' @{ RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "assignment to yield expression not possible") @}
    | a=star_expressions augassign (yield_expr | star_expressions) @{
        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(
            a,
            "'%s' is an illegal expression for augmented assignment",
            _PyPegen_get_expr_name(a)
        )@}
invalid_ann_assign_target[expr_ty]:
    | list
    | tuple
    | '(' a=invalid_ann_assign_target ')' @{ a @}
invalid_del_stmt:
    | 'del' a=star_expressions @{
        RAISE_SYNTAX_ERROR_INVALID_TARGET(DEL_TARGETS, a) @}
invalid_block:
    | NEWLINE !INDENT @{ RAISE_INDENTATION_ERROR("expected an indented block") @}
invalid_comprehension:
    | ('[' | '(' | '@{') a=starred_expression for_if_clauses @{
        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "iterable unpacking cannot be used in comprehension") @}
    | ('[' | '@{') a=star_named_expression ',' b=star_named_expressions for_if_clauses @{
        RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, PyPegen_last_item(b, expr_ty),
        "did you forget parentheses around the comprehension target?") @}
    | ('[' | '@{') a=star_named_expression b=',' for_if_clauses @{
        RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "did you forget parentheses around the comprehension target?") @}
invalid_dict_comprehension:
    | '@{' a='**' bitwise_or for_if_clauses '@}' @{
        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "dict unpacking cannot be used in dict comprehension") @}
invalid_parameters:
    | param_no_default* invalid_parameters_helper a=param_no_default @{
        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "non-default argument follows default argument") @}
    | param_no_default* a='(' param_no_default+ ','? b=')' @{
        RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "Function parameters cannot be parenthesized") @}
invalid_parameters_helper: # This is only there to avoid type errors
    | a=slash_with_default @{ _PyPegen_singleton_seq(p, a) @}
    | param_with_default+
invalid_lambda_parameters:
    | lambda_param_no_default* invalid_lambda_parameters_helper a=lambda_param_no_default @{
        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "non-default argument follows default argument") @}
    | lambda_param_no_default* a='(' ','.lambda_param+ ','? b=')' @{
        RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "Lambda expression parameters cannot be parenthesized") @}
invalid_lambda_parameters_helper:
    | a=lambda_slash_with_default @{ _PyPegen_singleton_seq(p, a) @}
    | lambda_param_with_default+
invalid_star_etc:
    | a='*' (')' | ',' (')' | '**')) @{ RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "named arguments must follow bare *") @}
    | '*' ',' TYPE_COMMENT @{ RAISE_SYNTAX_ERROR("bare * has associated type comment") @}
invalid_lambda_star_etc:
    | '*' (':' | ',' (':' | '**')) @{ RAISE_SYNTAX_ERROR("named arguments must follow bare *") @}
invalid_double_type_comments:
    | TYPE_COMMENT NEWLINE TYPE_COMMENT NEWLINE INDENT @{
        RAISE_SYNTAX_ERROR("Cannot have two type comments on def") @}
invalid_with_item:
    | expression 'as' a=expression &(',' | ')' | ':') @{
        RAISE_SYNTAX_ERROR_INVALID_TARGET(STAR_TARGETS, a) @}

invalid_for_target:
    | ASYNC? 'for' a=star_expressions @{
        RAISE_SYNTAX_ERROR_INVALID_TARGET(FOR_TARGETS, a) @}

invalid_group:
    | '(' a=starred_expression ')' @{
        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "cannot use starred expression here") @}
    | '(' a='**' expression ')' @{
        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "cannot use double starred expression here") @}
invalid_import_from_targets:
    | import_from_as_names ',' NEWLINE @{
        RAISE_SYNTAX_ERROR("trailing comma not allowed without surrounding parentheses") @}

invalid_with_stmt:
    | [ASYNC] 'with' ','.(expression ['as' star_target])+ &&':'
    | [ASYNC] 'with' '(' ','.(expressions ['as' star_target])+ ','? ')' &&':'
invalid_with_stmt_indent:
    | [ASYNC] a='with' ','.(expression ['as' star_target])+ ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after 'with' statement on line %d", a->lineno) @}
    | [ASYNC] a='with' '(' ','.(expressions ['as' star_target])+ ','? ')' ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after 'with' statement on line %d", a->lineno) @}

invalid_try_stmt:
    | a='try' ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after 'try' statement on line %d", a->lineno) @}
    | 'try' ':' block !('except' | 'finally') @{ RAISE_SYNTAX_ERROR("expected 'except' or 'finally' block") @}
    | 'try' ':' block* ((except_block+ except_star_block) | (except_star_block+ except_block)) block* @{
        RAISE_SYNTAX_ERROR("cannot have both 'except' and 'except*' on the same 'try'") @}
invalid_except_stmt:
    | 'except' '*'? a=expression ',' expressions ['as' NAME ] ':' @{
        RAISE_SYNTAX_ERROR_STARTING_FROM(a, "multiple exception types must be parenthesized") @}
    | a='except' '*'? expression ['as' NAME ] NEWLINE @{ RAISE_SYNTAX_ERROR("expected ':'") @}
    | a='except' NEWLINE @{ RAISE_SYNTAX_ERROR("expected ':'") @}
    | a='except' '*' (NEWLINE | ':') @{ RAISE_SYNTAX_ERROR("expected one or more exception types") @}
invalid_finally_stmt:
    | a='finally' ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after 'finally' statement on line %d", a->lineno) @}
invalid_except_stmt_indent:
    | a='except' expression ['as' NAME ] ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after 'except' statement on line %d", a->lineno) @}
    | a='except' ':' NEWLINE !INDENT @{ RAISE_SYNTAX_ERROR("expected an indented block after except statement on line %d", a->lineno) @}
invalid_except_star_stmt_indent:
    | a='except' '*' expression ['as' NAME ] ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after 'except*' statement on line %d", a->lineno) @}
invalid_match_stmt:
    | "match" subject_expr !':' @{ CHECK_VERSION(void*, 10, "Pattern matching is", RAISE_SYNTAX_ERROR("expected ':'") ) @}
    | a="match" subject=subject_expr ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after 'match' statement on line %d", a->lineno) @}
invalid_case_block:
    | "case" patterns guard? !':' @{ RAISE_SYNTAX_ERROR("expected ':'") @}
    | a="case" patterns guard? ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after 'case' statement on line %d", a->lineno) @}
invalid_as_pattern:
    | or_pattern 'as' a="_" @{ RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "cannot use '_' as a target") @}
    | or_pattern 'as' !NAME a=expression @{ RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "invalid pattern target") @}
invalid_class_pattern:
    | name_or_attr '(' a=invalid_class_argument_pattern  @{ RAISE_SYNTAX_ERROR_KNOWN_RANGE(
        PyPegen_first_item(a, pattern_ty),
        PyPegen_last_item(a, pattern_ty),
        "positional patterns follow keyword patterns") @}
invalid_class_argument_pattern[asdl_pattern_seq*]:
    | [positional_patterns ','] keyword_patterns ',' a=positional_patterns @{ a @}
invalid_if_stmt:
    | 'if' named_expression NEWLINE @{ RAISE_SYNTAX_ERROR("expected ':'") @}
    | a='if' a=named_expression ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after 'if' statement on line %d", a->lineno) @}
invalid_elif_stmt:
    | 'elif' named_expression NEWLINE @{ RAISE_SYNTAX_ERROR("expected ':'") @}
    | a='elif' named_expression ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after 'elif' statement on line %d", a->lineno) @}
invalid_else_stmt:
    | a='else' ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after 'else' statement on line %d", a->lineno) @}
invalid_while_stmt:
    | 'while' named_expression NEWLINE @{ RAISE_SYNTAX_ERROR("expected ':'") @}
    | a='while' named_expression ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after 'while' statement on line %d", a->lineno) @}
invalid_for_stmt:
    | [ASYNC] a='for' star_targets 'in' star_expressions ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after 'for' statement on line %d", a->lineno) @}
invalid_def_raw:
    | [ASYNC] a='def' NAME '(' [params] ')' ['->' expression] ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after function definition on line %d", a->lineno) @}
invalid_class_def_raw:
    | a='class' NAME ['('[arguments] ')'] ':' NEWLINE !INDENT @{
        RAISE_INDENTATION_ERROR("expected an indented block after class definition on line %d", a->lineno) @}

invalid_double_starred_kvpairs:
    | ','.double_starred_kvpair+ ',' invalid_kvpair
    | expression ':' a='*' bitwise_or @{ RAISE_SYNTAX_ERROR_STARTING_FROM(a, "cannot use a starred expression in a dictionary value") @}
    | expression a=':' &('@}'|',') @{ RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "expression expected after dictionary key and ':'") @}
invalid_kvpair:
    | a=expression !(':') @{
        RAISE_ERROR_KNOWN_LOCATION(p, PyExc_SyntaxError, a->lineno, a->end_col_offset - 1, a->end_lineno, -1, "':' expected after dictionary key") @}
    | expression ':' a='*' bitwise_or @{ RAISE_SYNTAX_ERROR_STARTING_FROM(a, "cannot use a starred expression in a dictionary value") @}
    | expression a=':' @{RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "expression expected after dictionary key and ':'") @}
@end example

@node The Python Standard Library,Extending and Embedding the Python Interpreter,The Python Language Reference,Top
@anchor{library/index doc}@anchor{1a28}@anchor{library/index library-index}@anchor{14bc}@anchor{library/index the-python-standard-library}@anchor{1a29}
@chapter The Python Standard Library


While @ref{14bd,,The Python Language Reference} describes the exact syntax and
semantics of the Python language, this library reference manual
describes the standard library that is distributed with Python. It also
describes some of the optional components that are commonly included
in Python distributions.

Python’s standard library is very extensive, offering a wide range of
facilities as indicated by the long table of contents listed below. The
library contains built-in modules (written in C) that provide access to
system functionality such as file I/O that would otherwise be
inaccessible to Python programmers, as well as modules written in Python
that provide standardized solutions for many problems that occur in
everyday programming. Some of these modules are explicitly designed to
encourage and enhance the portability of Python programs by abstracting
away platform-specifics into platform-neutral APIs.

The Python installers for the Windows platform usually include
the entire standard library and often also include many additional
components. For Unix-like operating systems Python is normally provided
as a collection of packages, so it may be necessary to use the packaging
tools provided with the operating system to obtain some or all of the
optional components.

In addition to the standard library, there is a growing collection of
several thousand components (from individual programs and modules to
packages and entire application development frameworks), available from
the Python Package Index@footnote{https://pypi.org}.

@c We don't use :numbered: option for the TOC below as it enforces
@c numbered sections for the entire stdlib docs.  If desired,
@c :numbered: can be enabled on a per-module basis.

@menu
* Introduction: Introduction<6>. 
* Built-in Functions:: 
* Built-in Constants:: 
* Built-in Types:: 
* Built-in Exceptions:: 
* Text Processing Services:: 
* Binary Data Services:: 
* Data Types:: 
* Numeric and Mathematical Modules:: 
* Functional Programming Modules:: 
* File and Directory Access:: 
* Data Persistence:: 
* Data Compression and Archiving:: 
* File Formats:: 
* Cryptographic Services:: 
* Generic Operating System Services:: 
* Concurrent Execution:: 
* Networking and Interprocess Communication:: 
* Internet Data Handling:: 
* Structured Markup Processing Tools:: 
* Internet Protocols and Support:: 
* Multimedia Services:: 
* Internationalization:: 
* Program Frameworks:: 
* Graphical User Interfaces with Tk:: 
* Development Tools:: 
* Debugging and Profiling:: 
* Software Packaging and Distribution:: 
* Python Runtime Services:: 
* Custom Python Interpreters:: 
* Importing Modules:: 
* Python Language Services:: 
* MS Windows Specific Services:: 
* Unix Specific Services:: 
* Superseded Modules:: 
* Undocumented Modules:: 
* Security Considerations: Security Considerations<3>. 

@end menu

@node Introduction<6>,Built-in Functions,,The Python Standard Library
@anchor{library/intro doc}@anchor{1a2a}@anchor{library/intro introduction}@anchor{1a2b}@anchor{library/intro library-intro}@anchor{1a2c}
@section Introduction


The “Python library” contains several different kinds of components.

It contains data types that would normally be considered part of the “core” of a
language, such as numbers and lists.  For these types, the Python language core
defines the form of literals and places some constraints on their semantics, but
does not fully define the semantics.  (On the other hand, the language core does
define syntactic properties like the spelling and priorities of operators.)

The library also contains built-in functions and exceptions — objects that can
be used by all Python code without the need of an @ref{e35,,import} statement.
Some of these are defined by the core language, but many are not essential for
the core semantics and are only described here.

The bulk of the library, however, consists of a collection of modules. There are
many ways to dissect this collection.  Some modules are written in C and built
in to the Python interpreter; others are written in Python and imported in
source form.  Some modules provide interfaces that are highly specific to
Python, like printing a stack trace; some provide interfaces that are specific
to particular operating systems, such as access to specific hardware; others
provide interfaces that are specific to a particular application domain, like
the World Wide Web. Some modules are available in all versions and ports of
Python; others are only available when the underlying system supports or
requires them; yet others are available only when a particular configuration
option was chosen at the time when Python was compiled and installed.

This manual is organized “from the inside out:” it first describes the built-in
functions, data types and exceptions, and finally the modules, grouped in
chapters of related modules.

This means that if you start reading this manual from the start, and skip to the
next chapter when you get bored, you will get a reasonable overview of the
available modules and application areas that are supported by the Python
library.  Of course, you don’t @emph{have} to read it like a novel — you can also
browse the table of contents (in front of the manual), or look for a specific
function, module or term in the index (in the back).  And finally, if you enjoy
learning about random subjects, you choose a random page number (see module
@ref{d7,,random}) and read a section or two.  Regardless of the order in which you
read the sections of this manual, it helps to start with chapter
@ref{e0d,,Built-in Functions}, as the remainder of the manual assumes familiarity with
this material.

Let the show begin!

@menu
* Notes on availability:: 

@end menu

@node Notes on availability,,,Introduction<6>
@anchor{library/intro availability}@anchor{1629}@anchor{library/intro notes-on-availability}@anchor{1a2d}
@subsection Notes on availability



@itemize *

@item 
An “Availability: Unix” note means that this function is commonly found on
Unix systems.  It does not make any claims about its existence on a specific
operating system.

@item 
If not separately noted, all functions that claim “Availability: Unix” are
supported on macOS, which builds on a Unix core.
@end itemize

@c XXX document all delegations to __special__ methods

@node Built-in Functions,Built-in Constants,Introduction<6>,The Python Standard Library
@anchor{library/functions doc}@anchor{1a2e}@anchor{library/functions built-in-funcs}@anchor{e0d}@anchor{library/functions built-in-functions}@anchor{1a2f}
@section Built-in Functions


The Python interpreter has a number of functions and types built into it that
are always available.  They are listed here in alphabetical order.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Built-in Functions

@tab

@tab

@tab

@item


@display
@strong{A}@w{ }
@ref{1582,,abs()}@w{ }
@ref{24f,,aiter()}@w{ }
@ref{f7f,,all()}@w{ }
@ref{f7e,,any()}@w{ }
@ref{250,,anext()}@w{ }
@ref{f47,,ascii()}@w{ }
@w{ }
@strong{B}@w{ }
@ref{e4e,,bin()}@w{ }
@ref{2f7,,bool()}@w{ }
@ref{5db,,breakpoint()}@w{ }
@ref{137f,,bytearray()}@w{ }
@ref{137e,,bytes()}@w{ }
@w{ }
@strong{C}@w{ }
@ref{d6a,,callable()}@w{ }
@ref{178f,,chr()}@w{ }
@ref{254,,classmethod()}@w{ }
@ref{4e2,,compile()}@w{ }
@ref{33b,,complex()}@w{ }
@w{ }
@strong{D}@w{ }
@ref{1a30,,delattr()}@w{ }
@ref{1a31,,dict()}@w{ }
@ref{f2f,,dir()}@w{ }
@ref{4a8,,divmod()}@w{ }
@w{ }
@end display



@tab


@display
@strong{E}@w{ }
@ref{fdc,,enumerate()}@w{ }
@ref{34a,,eval()}@w{ }
@ref{34b,,exec()}@w{ }
@w{ }
@strong{F}@w{ }
@ref{2f5,,filter()}@w{ }
@ref{256,,float()}@w{ }
@ref{785,,format()}@w{ }
@ref{1a32,,frozenset()}@w{ }
@w{ }
@strong{G}@w{ }
@ref{708,,getattr()}@w{ }
@ref{10c2,,globals()}@w{ }
@w{ }
@strong{H}@w{ }
@ref{707,,hasattr()}@w{ }
@ref{c14,,hash()}@w{ }
@ref{3bc,,help()}@w{ }
@ref{e6f,,hex()}@w{ }
@w{ }
@strong{I}@w{ }
@ref{f80,,id()}@w{ }
@ref{e73,,input()}@w{ }
@ref{1b1,,int()}@w{ }
@ref{236,,isinstance()}@w{ }
@ref{237,,issubclass()}@w{ }
@ref{251,,iter()}@w{ }
@end display



@tab


@display
@strong{L}@w{ }
@ref{4a6,,len()}@w{ }
@ref{1a33,,list()}@w{ }
@ref{713,,locals()}@w{ }
@w{ }
@strong{M}@w{ }
@ref{2f4,,map()}@w{ }
@ref{a7e,,max()}@w{ }
@ref{1a34,,memoryview()}@w{ }
@ref{a7d,,min()}@w{ }
@w{ }
@strong{N}@w{ }
@ref{252,,next()}@w{ }
@w{ }
@strong{O}@w{ }
@ref{5a5,,object()}@w{ }
@ref{e6e,,oct()}@w{ }
@ref{1c8,,open()}@w{ }
@ref{178e,,ord()}@w{ }
@w{ }
@strong{P}@w{ }
@ref{4cd,,pow()}@w{ }
@ref{af6,,print()}@w{ }
@ref{501,,property()}@w{ }
@w{ }
@w{ }
@w{ }
@w{ }
@end display



@tab


@display
@strong{R}@w{ }
@ref{1a35,,range()}@w{ }
@ref{27d,,repr()}@w{ }
@ref{2f6,,reversed()}@w{ }
@ref{e75,,round()}@w{ }
@w{ }
@strong{S}@w{ }
@ref{1a36,,set()}@w{ }
@ref{1a37,,setattr()}@w{ }
@ref{ffb,,slice()}@w{ }
@ref{704,,sorted()}@w{ }
@ref{253,,staticmethod()}@w{ }
@ref{1a38,,str()}@w{ }
@ref{50d,,sum()}@w{ }
@ref{78c,,super()}@w{ }
@w{ }
@strong{T}@w{ }
@ref{1a39,,tuple()}@w{ }
@ref{89e,,type()}@w{ }
@w{ }
@strong{V}@w{ }
@ref{12fc,,vars()}@w{ }
@w{ }
@strong{Z}@w{ }
@ref{247,,zip()}@w{ }
@w{ }
@strong{_}@w{ }
@ref{3b7,,__import__()}@w{ }
@end display



@end multitable


@c using :func:`dict` would create a link to another page, so local targets are
@c used, with replacement texts to make the output in the table consistent

@geindex built-in function; abs()
@anchor{library/functions abs}@anchor{1582}
@deffn {Function} abs (x)

Return the absolute value of a number.  The argument may be an
integer, a floating point number, or an object implementing @code{__abs__()}.
If the argument is a complex number, its magnitude is returned.
@end deffn

@geindex built-in function; aiter()
@anchor{library/functions aiter}@anchor{24f}
@deffn {Function} aiter (async_iterable)

Return an @ref{8d2,,asynchronous iterator} for an @ref{8d0,,asynchronous iterable}.
Equivalent to calling @code{x.__aiter__()}.

Note: Unlike @ref{251,,iter()}, @ref{24f,,aiter()} has no 2-argument variant.


New in version 3.10.

@end deffn

@geindex built-in function; all()
@anchor{library/functions all}@anchor{f7f}
@deffn {Function} all (iterable)

Return @code{True} if all elements of the @emph{iterable} are true (or if the iterable
is empty).  Equivalent to:

@example
def all(iterable):
    for element in iterable:
        if not element:
            return False
    return True
@end example
@end deffn

@geindex built-in function; anext()
@anchor{library/functions anext}@anchor{250}
@deffn {Function} awaitable anext (async_iterator[, default])

When awaited, return the next item from the given @ref{8d2,,asynchronous iterator}, or @emph{default} if given and the iterator is exhausted.

This is the async variant of the @ref{252,,next()} builtin, and behaves
similarly.

This calls the @ref{1793,,__anext__()} method of @emph{async_iterator},
returning an @ref{7bd,,awaitable}. Awaiting this returns the next value of the
iterator. If @emph{default} is given, it is returned if the iterator is exhausted,
otherwise @ref{12da,,StopAsyncIteration} is raised.


New in version 3.10.

@end deffn

@geindex built-in function; any()
@anchor{library/functions any}@anchor{f7e}
@deffn {Function} any (iterable)

Return @code{True} if any element of the @emph{iterable} is true.  If the iterable
is empty, return @code{False}.  Equivalent to:

@example
def any(iterable):
    for element in iterable:
        if element:
            return True
    return False
@end example
@end deffn

@geindex built-in function; ascii()
@anchor{library/functions ascii}@anchor{f47}
@deffn {Function} ascii (object)

As @ref{27d,,repr()}, return a string containing a printable representation of an
object, but escape the non-ASCII characters in the string returned by
@ref{27d,,repr()} using @code{\x}, @code{\u}, or @code{\U} escapes.  This generates a string
similar to that returned by @ref{27d,,repr()} in Python 2.
@end deffn

@geindex built-in function; bin()
@anchor{library/functions bin}@anchor{e4e}
@deffn {Function} bin (x)

Convert an integer number to a binary string prefixed with “0b”. The result
is a valid Python expression. If @emph{x} is not a Python @ref{1b1,,int} object, it
has to define an @code{__index__()} method that returns an integer. Some
examples:

@example
>>> bin(3)
'0b11'
>>> bin(-10)
'-0b1010'
@end example

If the prefix “0b” is desired or not, you can use either of the following ways.

@example
>>> format(14, '#b'), format(14, 'b')
('0b1110', '1110')
>>> f'@{14:#b@}', f'@{14:b@}'
('0b1110', '1110')
@end example

See also @ref{785,,format()} for more information.
@end deffn

@geindex bool (built-in class)
@anchor{library/functions bool}@anchor{2f7}
@deffn {Class} bool ([x])

Return a Boolean value, i.e. one of @code{True} or @code{False}.  @emph{x} is converted
using the standard @ref{1a3a,,truth testing procedure}.  If @emph{x} is false
or omitted, this returns @code{False}; otherwise, it returns @code{True}.  The
@ref{2f7,,bool} class is a subclass of @ref{1b1,,int} (see @ref{14de,,Numeric Types — int@comma{} float@comma{} complex}).
It cannot be subclassed further.  Its only instances are @code{False} and
@code{True} (see @ref{1a3b,,Boolean Values}).

@geindex Boolean; type


Changed in version 3.7: @emph{x} is now a positional-only parameter.

@end deffn

@geindex built-in function; breakpoint()
@anchor{library/functions breakpoint}@anchor{5db}
@deffn {Function} breakpoint (*args, **kws)

This function drops you into the debugger at the call site.  Specifically,
it calls @ref{5f4,,sys.breakpointhook()}, passing @code{args} and @code{kws} straight
through.  By default, @code{sys.breakpointhook()} calls
@ref{691,,pdb.set_trace()} expecting no arguments.  In this case, it is
purely a convenience function so you don’t have to explicitly import
@ref{c4,,pdb} or type as much code to enter the debugger.  However,
@ref{5f4,,sys.breakpointhook()} can be set to some other function and
@ref{5db,,breakpoint()} will automatically call that, allowing you to drop into
the debugger of choice.

Raises an @ref{1130,,auditing event} @code{builtins.breakpoint} with argument @code{breakpointhook}.


New in version 3.7.

@end deffn
@anchor{library/functions func-bytearray}@anchor{137f}

@deffn {Class} bytearray ([source[, encoding[, errors]]])

Return a new array of bytes.  The @ref{2f1,,bytearray} class is a mutable
sequence of integers in the range 0 <= x < 256.  It has most of the usual
methods of mutable sequences, described in @ref{1a3c,,Mutable Sequence Types}, as well
as most methods that the @ref{2f0,,bytes} type has, see @ref{1a3d,,Bytes and Bytearray Operations}.

The optional @emph{source} parameter can be used to initialize the array in a few
different ways:


@itemize *

@item 
If it is a @emph{string}, you must also give the @emph{encoding} (and optionally,
@emph{errors}) parameters; @ref{2f1,,bytearray()} then converts the string to
bytes using @ref{3ba,,str.encode()}.

@item 
If it is an @emph{integer}, the array will have that size and will be
initialized with null bytes.

@item 
If it is an object conforming to the @ref{1a3e,,buffer interface},
a read-only buffer of the object will be used to initialize the bytes array.

@item 
If it is an @emph{iterable}, it must be an iterable of integers in the range
@code{0 <= x < 256}, which are used as the initial contents of the array.
@end itemize

Without an argument, an array of size 0 is created.

See also @ref{1a3f,,Binary Sequence Types — bytes@comma{} bytearray@comma{} memoryview} and @ref{1a40,,Bytearray Objects}.
@end deffn
@anchor{library/functions func-bytes}@anchor{137e}

@deffn {Class} bytes ([source[, encoding[, errors]]])

Return a new “bytes” object which is an immutable sequence of integers in
the range @code{0 <= x < 256}.  @ref{2f0,,bytes} is an immutable version of
@ref{2f1,,bytearray} – it has the same non-mutating methods and the same
indexing and slicing behavior.

Accordingly, constructor arguments are interpreted as for @ref{2f1,,bytearray()}.

Bytes objects can also be created with literals, see @ref{1720,,String and Bytes literals}.

See also @ref{1a3f,,Binary Sequence Types — bytes@comma{} bytearray@comma{} memoryview}, @ref{1a41,,Bytes Objects}, and @ref{1a3d,,Bytes and Bytearray Operations}.
@end deffn

@geindex built-in function; callable()
@anchor{library/functions callable}@anchor{d6a}
@deffn {Function} callable (object)

Return @ref{748,,True} if the @emph{object} argument appears callable,
@ref{659,,False} if not.  If this returns @code{True}, it is still possible that a
call fails, but if it is @code{False}, calling @emph{object} will never succeed.
Note that classes are callable (calling a class returns a new instance);
instances are callable if their class has a @code{__call__()} method.


New in version 3.2: This function was first removed in Python 3.0 and then brought back
in Python 3.2.

@end deffn

@geindex built-in function; chr()
@anchor{library/functions chr}@anchor{178f}
@deffn {Function} chr (i)

Return the string representing a character whose Unicode code point is the
integer @emph{i}.  For example, @code{chr(97)} returns the string @code{'a'}, while
@code{chr(8364)} returns the string @code{'€'}. This is the inverse of @ref{178e,,ord()}.

The valid range for the argument is from 0 through 1,114,111 (0x10FFFF in
base 16).  @ref{17c,,ValueError} will be raised if @emph{i} is outside that range.
@end deffn

@geindex built-in function; classmethod()
@anchor{library/functions classmethod}@anchor{254}
@deffn {Function} @@classmethod

Transform a method into a class method.

A class method receives the class as an implicit first argument, just like an
instance method receives the instance. To declare a class method, use this
idiom:

@example
class C:
    @@classmethod
    def f(cls, arg1, arg2): ...
@end example

The @code{@@classmethod} form is a function @ref{1bf,,decorator} – see
@ref{151c,,Function definitions} for details.

A class method can be called either on the class (such as @code{C.f()}) or on an instance (such
as @code{C().f()}).  The instance is ignored except for its class. If a class
method is called for a derived class, the derived class object is passed as the
implied first argument.

Class methods are different than C++ or Java static methods. If you want those,
see @ref{253,,staticmethod()} in this section.
For more information on class methods, see @ref{1787,,The standard type hierarchy}.


Changed in version 3.9: Class methods can now wrap other @ref{17bc,,descriptors} such as
@ref{501,,property()}.



Changed in version 3.10: Class methods now inherit the method attributes (@code{__module__},
@code{__name__}, @code{__qualname__}, @code{__doc__} and @code{__annotations__}) and
have a new @code{__wrapped__} attribute.

@end deffn

@geindex built-in function; compile()
@anchor{library/functions compile}@anchor{4e2}
@deffn {Function} compile (source, filename, mode, flags=0, dont_inherit=False, optimize=- 1)

Compile the @emph{source} into a code or AST object.  Code objects can be executed
by @ref{34b,,exec()} or @ref{34a,,eval()}.  @emph{source} can either be a normal string, a
byte string, or an AST object.  Refer to the @ref{7,,ast} module documentation
for information on how to work with AST objects.

The @emph{filename} argument should give the file from which the code was read;
pass some recognizable value if it wasn’t read from a file (@code{'<string>'} is
commonly used).

The @emph{mode} argument specifies what kind of code must be compiled; it can be
@code{'exec'} if @emph{source} consists of a sequence of statements, @code{'eval'} if it
consists of a single expression, or @code{'single'} if it consists of a single
interactive statement (in the latter case, expression statements that
evaluate to something other than @code{None} will be printed).

The optional arguments @emph{flags} and @emph{dont_inherit} control which
@ref{1a42,,compiler options} should be activated
and which @ref{1969,,future features} should be allowed. If neither
is present (or both are zero) the code is compiled with the same flags that
affect the code that is calling @ref{4e2,,compile()}. If the @emph{flags}
argument is given and @emph{dont_inherit} is not (or is zero) then the compiler
options and the future statements specified by the @emph{flags} argument are used
in addition to those that would be used anyway. If @emph{dont_inherit} is a
non-zero integer then the @emph{flags} argument is it – the flags (future
features and compiler options) in the surrounding code are ignored.

Compiler options and future statements are specified by bits which can be
bitwise ORed together to specify multiple options. The bitfield required to
specify a given future feature can be found as the
@code{compiler_flag} attribute on the
@code{_Feature} instance in the @ref{0,,__future__} module.
@ref{1a42,,Compiler flags} can be found in @ref{7,,ast}
module, with @code{PyCF_} prefix.

The argument @emph{optimize} specifies the optimization level of the compiler; the
default value of @code{-1} selects the optimization level of the interpreter as
given by @ref{912,,-O} options.  Explicit levels are @code{0} (no optimization;
@code{__debug__} is true), @code{1} (asserts are removed, @code{__debug__} is false)
or @code{2} (docstrings are removed too).

This function raises @ref{21a,,SyntaxError} if the compiled source is invalid,
and @ref{17c,,ValueError} if the source contains null bytes.

If you want to parse Python code into its AST representation, see
@ref{4d3,,ast.parse()}.


Raises an @ref{1130,,auditing event} @code{compile} with arguments
@code{source} and @code{filename}. This event may also be raised by implicit
compilation.


@cartouche
@quotation Note 
When compiling a string with multi-line code in @code{'single'} or
@code{'eval'} mode, input must be terminated by at least one newline
character.  This is to facilitate detection of incomplete and complete
statements in the @ref{19,,code} module.
@end quotation
@end cartouche

@cartouche
@quotation Warning 
It is possible to crash the Python interpreter with a
sufficiently large/complex string when compiling to an AST
object due to stack depth limitations in Python’s AST compiler.
@end quotation
@end cartouche


Changed in version 3.2: Allowed use of Windows and Mac newlines.  Also, input in @code{'exec'} mode
does not have to end in a newline anymore.  Added the @emph{optimize} parameter.



Changed in version 3.5: Previously, @ref{162,,TypeError} was raised when null bytes were encountered
in @emph{source}.



New in version 3.8: @code{ast.PyCF_ALLOW_TOP_LEVEL_AWAIT} can now be passed in flags to enable
support for top-level @code{await}, @code{async for}, and @code{async with}.

@end deffn

@geindex complex (built-in class)
@anchor{library/functions complex}@anchor{33b}
@deffn {Class} complex ([real[, imag]])

Return a complex number with the value @emph{real} + @emph{imag}*1j or convert a string
or number to a complex number.  If the first parameter is a string, it will
be interpreted as a complex number and the function must be called without a
second parameter.  The second parameter can never be a string. Each argument
may be any numeric type (including complex).  If @emph{imag} is omitted, it
defaults to zero and the constructor serves as a numeric conversion like
@ref{1b1,,int} and @ref{256,,float}.  If both arguments are omitted, returns
@code{0j}.

For a general Python object @code{x}, @code{complex(x)} delegates to
@code{x.__complex__()}.  If @code{__complex__()} is not defined then it falls back
to @code{__float__()}.  If @code{__float__()} is not defined then it falls back
to @code{__index__()}.

@cartouche
@quotation Note 
When converting from a string, the string must not contain whitespace
around the central @code{+} or @code{-} operator.  For example,
@code{complex('1+2j')} is fine, but @code{complex('1 + 2j')} raises
@ref{17c,,ValueError}.
@end quotation
@end cartouche

The complex type is described in @ref{14de,,Numeric Types — int@comma{} float@comma{} complex}.


Changed in version 3.6: Grouping digits with underscores as in code literals is allowed.



Changed in version 3.8: Falls back to @code{__index__()} if @code{__complex__()} and
@code{__float__()} are not defined.

@end deffn

@geindex built-in function; delattr()
@anchor{library/functions delattr}@anchor{1a30}
@deffn {Function} delattr (object, name)

This is a relative of @ref{1a37,,setattr()}.  The arguments are an object and a
string.  The string must be the name of one of the object’s attributes.  The
function deletes the named attribute, provided the object allows it.  For
example, @code{delattr(x, 'foobar')} is equivalent to @code{del x.foobar}.
@end deffn
@anchor{library/functions func-dict}@anchor{1a31}

@deffn {Class} dict (**kwarg)

@deffnx {Class} dict (mapping, **kwarg)

@deffnx {Class} dict (iterable, **kwarg)

Create a new dictionary.  The @ref{3b0,,dict} object is the dictionary class.
See @ref{3b0,,dict} and @ref{5de,,Mapping Types — dict} for documentation about this class.

For other containers see the built-in @ref{43a,,list}, @ref{370,,set}, and
@ref{3e5,,tuple} classes, as well as the @ref{1c,,collections} module.
@end deffn

@geindex built-in function; dir()
@anchor{library/functions dir}@anchor{f2f}
@deffn {Function} dir ([object])

Without arguments, return the list of names in the current local scope.  With an
argument, attempt to return a list of valid attributes for that object.

If the object has a method named @code{__dir__()}, this method will be called and
must return the list of attributes. This allows objects that implement a custom
@code{__getattr__()} or @code{__getattribute__()} function to customize the way
@ref{f2f,,dir()} reports their attributes.

If the object does not provide @code{__dir__()}, the function tries its best to
gather information from the object’s @ref{7a4,,__dict__} attribute, if defined, and
from its type object.  The resulting list is not necessarily complete and may
be inaccurate when the object has a custom @code{__getattr__()}.

The default @ref{f2f,,dir()} mechanism behaves differently with different types of
objects, as it attempts to produce the most relevant, rather than complete,
information:


@itemize *

@item 
If the object is a module object, the list contains the names of the module’s
attributes.

@item 
If the object is a type or class object, the list contains the names of its
attributes, and recursively of the attributes of its bases.

@item 
Otherwise, the list contains the object’s attributes’ names, the names of its
class’s attributes, and recursively of the attributes of its class’s base
classes.
@end itemize

The resulting list is sorted alphabetically.  For example:

@example
>>> import struct
>>> dir()   # show the names in the module namespace  
['__builtins__', '__name__', 'struct']
>>> dir(struct)   # show the names in the struct module 
['Struct', '__all__', '__builtins__', '__cached__', '__doc__', '__file__',
 '__initializing__', '__loader__', '__name__', '__package__',
 '_clearcache', 'calcsize', 'error', 'pack', 'pack_into',
 'unpack', 'unpack_from']
>>> class Shape:
...     def __dir__(self):
...         return ['area', 'perimeter', 'location']
>>> s = Shape()
>>> dir(s)
['area', 'location', 'perimeter']
@end example

@cartouche
@quotation Note 
Because @ref{f2f,,dir()} is supplied primarily as a convenience for use at an
interactive prompt, it tries to supply an interesting set of names more
than it tries to supply a rigorously or consistently defined set of names,
and its detailed behavior may change across releases.  For example,
metaclass attributes are not in the result list when the argument is a
class.
@end quotation
@end cartouche
@end deffn

@geindex built-in function; divmod()
@anchor{library/functions divmod}@anchor{4a8}
@deffn {Function} divmod (a, b)

Take two (non-complex) numbers as arguments and return a pair of numbers
consisting of their quotient and remainder when using integer division.  With
mixed operand types, the rules for binary arithmetic operators apply.  For
integers, the result is the same as @code{(a // b, a % b)}. For floating point
numbers the result is @code{(q, a % b)}, where @emph{q} is usually @code{math.floor(a /
b)} but may be 1 less than that.  In any case @code{q * b + a % b} is very
close to @emph{a}, if @code{a % b} is non-zero it has the same sign as @emph{b}, and @code{0
<= abs(a % b) < abs(b)}.
@end deffn

@geindex built-in function; enumerate()
@anchor{library/functions enumerate}@anchor{fdc}
@deffn {Function} enumerate (iterable, start=0)

Return an enumerate object. @emph{iterable} must be a sequence, an
@ref{11c3,,iterator}, or some other object which supports iteration.
The @ref{e6d,,__next__()} method of the iterator returned by
@ref{fdc,,enumerate()} returns a tuple containing a count (from @emph{start} which
defaults to 0) and the values obtained from iterating over @emph{iterable}.

@example
>>> seasons = ['Spring', 'Summer', 'Fall', 'Winter']
>>> list(enumerate(seasons))
[(0, 'Spring'), (1, 'Summer'), (2, 'Fall'), (3, 'Winter')]
>>> list(enumerate(seasons, start=1))
[(1, 'Spring'), (2, 'Summer'), (3, 'Fall'), (4, 'Winter')]
@end example

Equivalent to:

@example
def enumerate(sequence, start=0):
    n = start
    for elem in sequence:
        yield n, elem
        n += 1
@end example
@end deffn

@geindex built-in function; eval()
@anchor{library/functions eval}@anchor{34a}
@deffn {Function} eval (expression[, globals[, locals]])

The arguments are a string and optional globals and locals.  If provided,
@emph{globals} must be a dictionary.  If provided, @emph{locals} can be any mapping
object.

The @emph{expression} argument is parsed and evaluated as a Python expression
(technically speaking, a condition list) using the @emph{globals} and @emph{locals}
dictionaries as global and local namespace.  If the @emph{globals} dictionary is
present and does not contain a value for the key @code{__builtins__}, a
reference to the dictionary of the built-in module @ref{11,,builtins} is
inserted under that key before @emph{expression} is parsed.  That way you can
control what builtins are available to the executed code by inserting your
own @code{__builtins__} dictionary into @emph{globals} before passing it to
@ref{34a,,eval()}.  If the @emph{locals} dictionary is omitted it defaults to the
@emph{globals} dictionary.  If both dictionaries are omitted, the expression is
executed with the @emph{globals} and @emph{locals} in the environment where
@ref{34a,,eval()} is called.  Note, @emph{eval()} does not have access to the
@ref{1a43,,nested scopes} (non-locals) in the enclosing
environment.

The return value is the result of
the evaluated expression. Syntax errors are reported as exceptions.  Example:

@example
>>> x = 1
>>> eval('x+1')
2
@end example

This function can also be used to execute arbitrary code objects (such as
those created by @ref{4e2,,compile()}).  In this case, pass a code object instead
of a string.  If the code object has been compiled with @code{'exec'} as the
@emph{mode} argument, @ref{34a,,eval()}'s return value will be @code{None}.

Hints: dynamic execution of statements is supported by the @ref{34b,,exec()}
function.  The @ref{10c2,,globals()} and @ref{713,,locals()} functions
return the current global and local dictionary, respectively, which may be
useful to pass around for use by @ref{34a,,eval()} or @ref{34b,,exec()}.

If the given source is a string, then leading and trailing spaces and tabs
are stripped.

See @ref{754,,ast.literal_eval()} for a function that can safely evaluate strings
with expressions containing only literals.


Raises an @ref{1130,,auditing event} @code{exec} with the code object
as the argument. Code compilation events may also be raised.

@end deffn

@geindex built-in function; exec

@geindex built-in function; exec()
@anchor{library/functions exec}@anchor{34b}
@deffn {Function} exec (object[, globals[, locals]])

This function supports dynamic execution of Python code. @emph{object} must be
either a string or a code object.  If it is a string, the string is parsed as
a suite of Python statements which is then executed (unless a syntax error
occurs). @footnote{@w{(1)} 
Note that the parser only accepts the Unix-style end of line convention.
If you are reading the code from a file, make sure to use newline conversion
mode to convert Windows or Mac-style newlines.
} If it is a code object, it is simply executed.  In all cases,
the code that’s executed is expected to be valid as file input (see the
section @ref{1a1b,,File input} in the Reference Manual). Be aware that the
@ref{e4d,,nonlocal}, @ref{4c5,,yield},  and @ref{4c6,,return}
statements may not be used outside of
function definitions even within the context of code passed to the
@ref{34b,,exec()} function. The return value is @code{None}.

In all cases, if the optional parts are omitted, the code is executed in the
current scope.  If only @emph{globals} is provided, it must be a dictionary
(and not a subclass of dictionary), which
will be used for both the global and the local variables.  If @emph{globals} and
@emph{locals} are given, they are used for the global and local variables,
respectively.  If provided, @emph{locals} can be any mapping object.  Remember
that at the module level, globals and locals are the same dictionary. If exec
gets two separate objects as @emph{globals} and @emph{locals}, the code will be
executed as if it were embedded in a class definition.

If the @emph{globals} dictionary does not contain a value for the key
@code{__builtins__}, a reference to the dictionary of the built-in module
@ref{11,,builtins} is inserted under that key.  That way you can control what
builtins are available to the executed code by inserting your own
@code{__builtins__} dictionary into @emph{globals} before passing it to @ref{34b,,exec()}.


Raises an @ref{1130,,auditing event} @code{exec} with the code object
as the argument. Code compilation events may also be raised.


@cartouche
@quotation Note 
The built-in functions @ref{10c2,,globals()} and @ref{713,,locals()} return the current
global and local dictionary, respectively, which may be useful to pass around
for use as the second and third argument to @ref{34b,,exec()}.
@end quotation
@end cartouche

@cartouche
@quotation Note 
The default @emph{locals} act as described for function @ref{713,,locals()} below:
modifications to the default @emph{locals} dictionary should not be attempted.
Pass an explicit @emph{locals} dictionary if you need to see effects of the
code on @emph{locals} after function @ref{34b,,exec()} returns.
@end quotation
@end cartouche
@end deffn

@geindex built-in function; filter()
@anchor{library/functions filter}@anchor{2f5}
@deffn {Function} filter (function, iterable)

Construct an iterator from those elements of @emph{iterable} for which @emph{function}
returns true.  @emph{iterable} may be either a sequence, a container which
supports iteration, or an iterator.  If @emph{function} is @code{None}, the identity
function is assumed, that is, all elements of @emph{iterable} that are false are
removed.

Note that @code{filter(function, iterable)} is equivalent to the generator
expression @code{(item for item in iterable if function(item))} if function is
not @code{None} and @code{(item for item in iterable if item)} if function is
@code{None}.

See @ref{1a44,,itertools.filterfalse()} for the complementary function that returns
elements of @emph{iterable} for which @emph{function} returns false.
@end deffn

@geindex float (built-in class)
@anchor{library/functions float}@anchor{256}
@deffn {Class} float ([x])

@geindex NaN
@geindex Infinity

Return a floating point number constructed from a number or string @emph{x}.

If the argument is a string, it should contain a decimal number, optionally
preceded by a sign, and optionally embedded in whitespace.  The optional
sign may be @code{'+'} or @code{'-'}; a @code{'+'} sign has no effect on the value
produced.  The argument may also be a string representing a NaN
(not-a-number), or positive or negative infinity.  More precisely, the
input must conform to the following grammar after leading and trailing
whitespace characters are removed:

@example
@anchor{library/functions grammar-token-float-sign}@anchor{1a45}@anchor{library/functions grammar-token-sign}@anchor{1a46}sign           ::= "+" | "-"
@anchor{library/functions grammar-token-float-infinity}@anchor{1a47}@anchor{library/functions grammar-token-infinity}@anchor{1a48}infinity       ::= "Infinity" | "inf"
@anchor{library/functions grammar-token-float-nan}@anchor{1a49}@anchor{library/functions grammar-token-nan}@anchor{1a4a}nan            ::= "nan"
@anchor{library/functions grammar-token-float-numeric_value}@anchor{1a4b}@anchor{library/functions grammar-token-numeric-value}@anchor{1a4c}numeric_value  ::= floatnumber | infinity | nan
@anchor{library/functions grammar-token-float-numeric_string}@anchor{1a4d}@anchor{library/functions grammar-token-numeric-string}@anchor{1a4e}numeric_string ::= [sign] numeric_value
@end example

Here @code{floatnumber} is the form of a Python floating-point literal,
described in @ref{176c,,Floating point literals}.  Case is not significant, so, for example,
“inf”, “Inf”, “INFINITY”, and “iNfINity” are all acceptable spellings for
positive infinity.

Otherwise, if the argument is an integer or a floating point number, a
floating point number with the same value (within Python’s floating point
precision) is returned.  If the argument is outside the range of a Python
float, an @ref{347,,OverflowError} will be raised.

For a general Python object @code{x}, @code{float(x)} delegates to
@code{x.__float__()}.  If @code{__float__()} is not defined then it falls back
to @code{__index__()}.

If no argument is given, @code{0.0} is returned.

Examples:

@example
>>> float('+1.23')
1.23
>>> float('   -12345\n')
-12345.0
>>> float('1e-003')
0.001
>>> float('+1E6')
1000000.0
>>> float('-Infinity')
-inf
@end example

The float type is described in @ref{14de,,Numeric Types — int@comma{} float@comma{} complex}.


Changed in version 3.6: Grouping digits with underscores as in code literals is allowed.



Changed in version 3.7: @emph{x} is now a positional-only parameter.



Changed in version 3.8: Falls back to @code{__index__()} if @code{__float__()} is not defined.

@end deffn

@geindex __format__
@geindex string; format() (built-in function)

@geindex built-in function; format()
@anchor{library/functions format}@anchor{785}
@deffn {Function} format (value[, format_spec])

Convert a @emph{value} to a “formatted” representation, as controlled by
@emph{format_spec}.  The interpretation of @emph{format_spec} will depend on the type
of the @emph{value} argument; however, there is a standard formatting syntax that
is used by most built-in types: @ref{788,,Format Specification Mini-Language}.

The default @emph{format_spec} is an empty string which usually gives the same
effect as calling @ref{2ef,,str(value)}.

A call to @code{format(value, format_spec)} is translated to
@code{type(value).__format__(value, format_spec)} which bypasses the instance
dictionary when searching for the value’s @code{__format__()} method.  A
@ref{162,,TypeError} exception is raised if the method search reaches
@ref{5a5,,object} and the @emph{format_spec} is non-empty, or if either the
@emph{format_spec} or the return value are not strings.


Changed in version 3.4: @code{object().__format__(format_spec)} raises @ref{162,,TypeError}
if @emph{format_spec} is not an empty string.

@end deffn
@anchor{library/functions func-frozenset}@anchor{1a32}

@deffn {Class} frozenset ([iterable])

Return a new @ref{439,,frozenset} object, optionally with elements taken from
@emph{iterable}.  @code{frozenset} is a built-in class.  See @ref{439,,frozenset} and
@ref{1a4f,,Set Types — set@comma{} frozenset} for documentation about this class.

For other containers see the built-in @ref{370,,set}, @ref{43a,,list},
@ref{3e5,,tuple}, and @ref{3b0,,dict} classes, as well as the @ref{1c,,collections}
module.
@end deffn

@geindex built-in function; getattr()
@anchor{library/functions getattr}@anchor{708}
@deffn {Function} getattr (object, name[, default])

Return the value of the named attribute of @emph{object}.  @emph{name} must be a string.
If the string is the name of one of the object’s attributes, the result is the
value of that attribute.  For example, @code{getattr(x, 'foobar')} is equivalent to
@code{x.foobar}.  If the named attribute does not exist, @emph{default} is returned if
provided, otherwise @ref{163,,AttributeError} is raised.

@cartouche
@quotation Note 
Since @ref{187e,,private name mangling} happens at
compilation time, one must manually mangle a private attribute’s
(attributes with two leading underscores) name in order to retrieve it with
@ref{708,,getattr()}.
@end quotation
@end cartouche
@end deffn

@geindex built-in function; globals()
@anchor{library/functions globals}@anchor{10c2}
@deffn {Function} globals ()

Return the dictionary implementing the current module namespace. For code within
functions, this is set when the function is defined and remains the same
regardless of where the function is called.
@end deffn

@geindex built-in function; hasattr()
@anchor{library/functions hasattr}@anchor{707}
@deffn {Function} hasattr (object, name)

The arguments are an object and a string.  The result is @code{True} if the
string is the name of one of the object’s attributes, @code{False} if not. (This
is implemented by calling @code{getattr(object, name)} and seeing whether it
raises an @ref{163,,AttributeError} or not.)
@end deffn

@geindex built-in function; hash()
@anchor{library/functions hash}@anchor{c14}
@deffn {Function} hash (object)

Return the hash value of the object (if it has one).  Hash values are
integers.  They are used to quickly compare dictionary keys during a
dictionary lookup.  Numeric values that compare equal have the same hash
value (even if they are of different types, as is the case for 1 and 1.0).

@cartouche
@quotation Note 
For objects with custom @code{__hash__()} methods, note that @ref{c14,,hash()}
truncates the return value based on the bit width of the host machine.
See @code{__hash__()} for details.
@end quotation
@end cartouche
@end deffn

@geindex built-in function; help()
@anchor{library/functions help}@anchor{3bc}
@deffn {Function} help ([object])

Invoke the built-in help system.  (This function is intended for interactive
use.)  If no argument is given, the interactive help system starts on the
interpreter console.  If the argument is a string, then the string is looked up
as the name of a module, function, class, method, keyword, or documentation
topic, and a help page is printed on the console.  If the argument is any other
kind of object, a help page on the object is generated.

Note that if a slash(/) appears in the parameter list of a function when
invoking @ref{3bc,,help()}, it means that the parameters prior to the slash are
positional-only. For more info, see
@ref{1a50,,the FAQ entry on positional-only parameters}.

This function is added to the built-in namespace by the @ref{e6,,site} module.


Changed in version 3.4: Changes to @ref{d4,,pydoc} and @ref{9c,,inspect} mean that the reported
signatures for callables are now more comprehensive and consistent.

@end deffn

@geindex built-in function; hex()
@anchor{library/functions hex}@anchor{e6f}
@deffn {Function} hex (x)

Convert an integer number to a lowercase hexadecimal string prefixed with
“0x”. If @emph{x} is not a Python @ref{1b1,,int} object, it has to define an
@code{__index__()} method that returns an integer. Some examples:

@example
>>> hex(255)
'0xff'
>>> hex(-42)
'-0x2a'
@end example

If you want to convert an integer number to an uppercase or lower hexadecimal
string with prefix or not, you can use either of the following ways:

@example
>>> '%#x' % 255, '%x' % 255, '%X' % 255
('0xff', 'ff', 'FF')
>>> format(255, '#x'), format(255, 'x'), format(255, 'X')
('0xff', 'ff', 'FF')
>>> f'@{255:#x@}', f'@{255:x@}', f'@{255:X@}'
('0xff', 'ff', 'FF')
@end example

See also @ref{785,,format()} for more information.

See also @ref{1b1,,int()} for converting a hexadecimal string to an
integer using a base of 16.

@cartouche
@quotation Note 
To obtain a hexadecimal string representation for a float, use the
@ref{15e7,,float.hex()} method.
@end quotation
@end cartouche
@end deffn

@geindex built-in function; id()
@anchor{library/functions id}@anchor{f80}
@deffn {Function} id (object)

Return the “identity” of an object.  This is an integer which
is guaranteed to be unique and constant for this object during its lifetime.
Two objects with non-overlapping lifetimes may have the same @ref{f80,,id()}
value.

@strong{CPython implementation detail:} This is the address of the object in memory.

Raises an @ref{1130,,auditing event} @code{builtins.id} with argument @code{id}.
@end deffn

@geindex built-in function; input()
@anchor{library/functions input}@anchor{e73}
@deffn {Function} input ([prompt])

If the @emph{prompt} argument is present, it is written to standard output without
a trailing newline.  The function then reads a line from input, converts it
to a string (stripping a trailing newline), and returns that.  When EOF is
read, @ref{e74,,EOFError} is raised.  Example:

@example
>>> s = input('--> ')  
--> Monty Python's Flying Circus
>>> s  
"Monty Python's Flying Circus"
@end example

If the @ref{d9,,readline} module was loaded, then @ref{e73,,input()} will use it
to provide elaborate line editing and history features.


Raises an @ref{1130,,auditing event} @code{builtins.input} with
argument @code{prompt} before reading input



Raises an auditing event @code{builtins.input/result} with the result after
successfully reading input.

@end deffn

@geindex int (built-in class)
@anchor{library/functions int}@anchor{1b1}
@deffn {Class} int ([x])

@deffnx {Class} int (x, base=10)

Return an integer object constructed from a number or string @emph{x}, or return
@code{0} if no arguments are given.  If @emph{x} defines @code{__int__()},
@code{int(x)} returns @code{x.__int__()}.  If @emph{x} defines @code{__index__()},
it returns @code{x.__index__()}.  If @emph{x} defines @code{__trunc__()},
it returns @code{x.__trunc__()}.
For floating point numbers, this truncates towards zero.

If @emph{x} is not a number or if @emph{base} is given, then @emph{x} must be a string,
@ref{2f0,,bytes}, or @ref{2f1,,bytearray} instance representing an @ref{1757,,integer literal} in radix @emph{base}.  Optionally, the literal can be
preceded by @code{+} or @code{-} (with no space in between) and surrounded by
whitespace.  A base-n literal consists of the digits 0 to n-1, with @code{a}
to @code{z} (or @code{A} to @code{Z}) having
values 10 to 35.  The default @emph{base} is 10. The allowed values are 0 and 2–36.
Base-2, -8, and -16 literals can be optionally prefixed with @code{0b}/@code{0B},
@code{0o}/@code{0O}, or @code{0x}/@code{0X}, as with integer literals in code.  Base 0
means to interpret exactly as a code literal, so that the actual base is 2,
8, 10, or 16, and so that @code{int('010', 0)} is not legal, while
@code{int('010')} is, as well as @code{int('010', 8)}.

The integer type is described in @ref{14de,,Numeric Types — int@comma{} float@comma{} complex}.


Changed in version 3.4: If @emph{base} is not an instance of @ref{1b1,,int} and the @emph{base} object has a
@ref{24a,,base.__index__} method, that method is called
to obtain an integer for the base.  Previous versions used
@ref{249,,base.__int__} instead of @ref{24a,,base.__index__}.



Changed in version 3.6: Grouping digits with underscores as in code literals is allowed.



Changed in version 3.7: @emph{x} is now a positional-only parameter.



Changed in version 3.8: Falls back to @code{__index__()} if @code{__int__()} is not defined.



Changed in version 3.11: The delegation to @code{__trunc__()} is deprecated.

@end deffn

@geindex built-in function; isinstance()
@anchor{library/functions isinstance}@anchor{236}
@deffn {Function} isinstance (object, classinfo)

Return @code{True} if the @emph{object} argument is an instance of the @emph{classinfo}
argument, or of a (direct, indirect, or @ref{d5c,,virtual}) subclass thereof.  If @emph{object} is not
an object of the given type, the function always returns @code{False}.
If @emph{classinfo} is a tuple of type objects (or recursively, other such
tuples) or a @ref{238,,Union Type} of multiple types, return @code{True} if
@emph{object} is an instance of any of the types.
If @emph{classinfo} is not a type or tuple of types and such tuples,
a @ref{162,,TypeError} exception is raised.


Changed in version 3.10: @emph{classinfo} can be a @ref{238,,Union Type}.

@end deffn

@geindex built-in function; issubclass()
@anchor{library/functions issubclass}@anchor{237}
@deffn {Function} issubclass (class, classinfo)

Return @code{True} if @emph{class} is a subclass (direct, indirect, or @ref{d5c,,virtual}) of @emph{classinfo}.  A
class is considered a subclass of itself. @emph{classinfo} may be a tuple of class
objects or a @ref{238,,Union Type}, in which case return @code{True} if @emph{class} is a
subclass of any entry in @emph{classinfo}.  In any other case, a @ref{162,,TypeError}
exception is raised.


Changed in version 3.10: @emph{classinfo} can be a @ref{238,,Union Type}.

@end deffn

@geindex built-in function; iter()
@anchor{library/functions iter}@anchor{251}
@deffn {Function} iter (object[, sentinel])

Return an @ref{11c3,,iterator} object.  The first argument is interpreted very
differently depending on the presence of the second argument. Without a
second argument, @emph{object} must be a collection object which supports the
@ref{dcc,,iterable} protocol (the @code{__iter__()} method), or it must support
the sequence protocol (the @code{__getitem__()} method with integer arguments
starting at @code{0}).  If it does not support either of those protocols,
@ref{162,,TypeError} is raised. If the second argument, @emph{sentinel}, is given,
then @emph{object} must be a callable object.  The iterator created in this case
will call @emph{object} with no arguments for each call to its
@ref{e6d,,__next__()} method; if the value returned is equal to
@emph{sentinel}, @ref{736,,StopIteration} will be raised, otherwise the value will
be returned.

See also @ref{1a51,,Iterator Types}.

One useful application of the second form of @ref{251,,iter()} is to build a
block-reader. For example, reading fixed-width blocks from a binary
database file until the end of file is reached:

@example
from functools import partial
with open('mydata.db', 'rb') as f:
    for block in iter(partial(f.read, 64), b'@w{'}):
        process_block(block)
@end example
@end deffn

@geindex built-in function; len()
@anchor{library/functions len}@anchor{4a6}
@deffn {Function} len (s)

Return the length (the number of items) of an object.  The argument may be a
sequence (such as a string, bytes, tuple, list, or range) or a collection
(such as a dictionary, set, or frozen set).

@strong{CPython implementation detail:} @code{len} raises @ref{347,,OverflowError} on lengths larger than
@ref{d69,,sys.maxsize}, such as @ref{438,,range(2 ** 100)}.
@end deffn
@anchor{library/functions func-list}@anchor{1a33}

@deffn {Class} list ([iterable])

Rather than being a function, @ref{43a,,list} is actually a mutable
sequence type, as documented in @ref{1a52,,Lists} and @ref{1530,,Sequence Types — list@comma{} tuple@comma{} range}.
@end deffn

@geindex built-in function; locals()
@anchor{library/functions locals}@anchor{713}
@deffn {Function} locals ()

Update and return a dictionary representing the current local symbol table.
Free variables are returned by @ref{713,,locals()} when it is called in function
blocks, but not in class blocks. Note that at the module level, @ref{713,,locals()}
and @ref{10c2,,globals()} are the same dictionary.

@cartouche
@quotation Note 
The contents of this dictionary should not be modified; changes may not
affect the values of local and free variables used by the interpreter.
@end quotation
@end cartouche
@end deffn

@geindex built-in function; map()
@anchor{library/functions map}@anchor{2f4}
@deffn {Function} map (function, iterable, ...)

Return an iterator that applies @emph{function} to every item of @emph{iterable},
yielding the results.  If additional @emph{iterable} arguments are passed,
@emph{function} must take that many arguments and is applied to the items from all
iterables in parallel.  With multiple iterables, the iterator stops when the
shortest iterable is exhausted.  For cases where the function inputs are
already arranged into argument tuples, see @ref{c76,,itertools.starmap()}.
@end deffn

@geindex built-in function; max()
@anchor{library/functions max}@anchor{a7e}
@deffn {Function} max (iterable, *[, key, default])

@deffnx {Function} max (arg1, arg2, *args[, key])

Return the largest item in an iterable or the largest of two or more
arguments.

If one positional argument is provided, it should be an @ref{dcc,,iterable}.
The largest item in the iterable is returned.  If two or more positional
arguments are provided, the largest of the positional arguments is
returned.

There are two optional keyword-only arguments. The @emph{key} argument specifies
a one-argument ordering function like that used for @ref{705,,list.sort()}. The
@emph{default} argument specifies an object to return if the provided iterable is
empty. If the iterable is empty and @emph{default} is not provided, a
@ref{17c,,ValueError} is raised.

If multiple items are maximal, the function returns the first one
encountered.  This is consistent with other sort-stability preserving tools
such as @code{sorted(iterable, key=keyfunc, reverse=True)[0]} and
@code{heapq.nlargest(1, iterable, key=keyfunc)}.


New in version 3.4: The @emph{default} keyword-only argument.



Changed in version 3.8: The @emph{key} can be @code{None}.

@end deffn
@anchor{library/functions func-memoryview}@anchor{1a34}

@deffn {Class} memoryview (object)

Return a “memory view” object created from the given argument.  See
@ref{1a53,,Memory Views} for more information.
@end deffn

@geindex built-in function; min()
@anchor{library/functions min}@anchor{a7d}
@deffn {Function} min (iterable, *[, key, default])

@deffnx {Function} min (arg1, arg2, *args[, key])

Return the smallest item in an iterable or the smallest of two or more
arguments.

If one positional argument is provided, it should be an @ref{dcc,,iterable}.
The smallest item in the iterable is returned.  If two or more positional
arguments are provided, the smallest of the positional arguments is
returned.

There are two optional keyword-only arguments. The @emph{key} argument specifies
a one-argument ordering function like that used for @ref{705,,list.sort()}. The
@emph{default} argument specifies an object to return if the provided iterable is
empty. If the iterable is empty and @emph{default} is not provided, a
@ref{17c,,ValueError} is raised.

If multiple items are minimal, the function returns the first one
encountered.  This is consistent with other sort-stability preserving tools
such as @code{sorted(iterable, key=keyfunc)[0]} and @code{heapq.nsmallest(1,
iterable, key=keyfunc)}.


New in version 3.4: The @emph{default} keyword-only argument.



Changed in version 3.8: The @emph{key} can be @code{None}.

@end deffn

@geindex built-in function; next()
@anchor{library/functions next}@anchor{252}
@deffn {Function} next (iterator[, default])

Retrieve the next item from the @ref{11c3,,iterator} by calling its
@ref{e6d,,__next__()} method.  If @emph{default} is given, it is returned
if the iterator is exhausted, otherwise @ref{736,,StopIteration} is raised.
@end deffn

@geindex object (built-in class)
@anchor{library/functions object}@anchor{5a5}
@deffn {Class} object

Return a new featureless object.  @ref{5a5,,object} is a base for all classes.
It has methods that are common to all instances of Python classes.  This
function does not accept any arguments.

@cartouche
@quotation Note 
@ref{5a5,,object} does @emph{not} have a @ref{7a4,,__dict__}, so you can’t
assign arbitrary attributes to an instance of the @ref{5a5,,object} class.
@end quotation
@end cartouche
@end deffn

@geindex built-in function; oct()
@anchor{library/functions oct}@anchor{e6e}
@deffn {Function} oct (x)

Convert an integer number to an octal string prefixed with “0o”.  The result
is a valid Python expression. If @emph{x} is not a Python @ref{1b1,,int} object, it
has to define an @code{__index__()} method that returns an integer. For
example:

@example
>>> oct(8)
'0o10'
>>> oct(-56)
'-0o70'
@end example

If you want to convert an integer number to an octal string either with the prefix
“0o” or not, you can use either of the following ways.

@example
>>> '%#o' % 10, '%o' % 10
('0o12', '12')
>>> format(10, '#o'), format(10, 'o')
('0o12', '12')
>>> f'@{10:#o@}', f'@{10:o@}'
('0o12', '12')
@end example

See also @ref{785,,format()} for more information.

@quotation

@geindex file object; open() built-in function
@end quotation
@end deffn

@geindex built-in function; open()
@anchor{library/functions open}@anchor{1c8}
@deffn {Function} open (file, mode='r', buffering=- 1, encoding=None, errors=None, newline=None, closefd=True, opener=None)

Open @emph{file} and return a corresponding @ref{d68,,file object}.  If the file
cannot be opened, an @ref{2a1,,OSError} is raised. See
@ref{1561,,Reading and Writing Files} for more examples of how to use this function.

@emph{file} is a @ref{63d,,path-like object} giving the pathname (absolute or
relative to the current working directory) of the file to be opened or an
integer file descriptor of the file to be wrapped.  (If a file descriptor is
given, it is closed when the returned I/O object is closed unless @emph{closefd}
is set to @code{False}.)

@emph{mode} is an optional string that specifies the mode in which the file is
opened.  It defaults to @code{'r'} which means open for reading in text mode.
Other common values are @code{'w'} for writing (truncating the file if it
already exists), @code{'x'} for exclusive creation, and @code{'a'} for appending
(which on @emph{some} Unix systems, means that @emph{all} writes append to the end of
the file regardless of the current seek position).  In text mode, if
@emph{encoding} is not specified the encoding used is platform-dependent:
@code{locale.getpreferredencoding(False)} is called to get the current locale
encoding. (For reading and writing raw bytes use binary mode and leave
@emph{encoding} unspecified.)  The available modes are:
@anchor{library/functions filemodes}@anchor{1a54}
@geindex file; modes


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Character

@tab

Meaning

@item

@code{'r'}

@tab

open for reading (default)

@item

@code{'w'}

@tab

open for writing, truncating the file first

@item

@code{'x'}

@tab

open for exclusive creation, failing if the file already exists

@item

@code{'a'}

@tab

open for writing, appending to the end of file if it exists

@item

@code{'b'}

@tab

binary mode

@item

@code{'t'}

@tab

text mode (default)

@item

@code{'+'}

@tab

open for updating (reading and writing)

@end multitable


The default mode is @code{'r'} (open for reading text, a synonym of @code{'rt'}).
Modes @code{'w+'} and @code{'w+b'} open and truncate the file.  Modes @code{'r+'}
and @code{'r+b'} open the file with no truncation.

As mentioned in the @ref{1a55,,Overview}, Python distinguishes between binary
and text I/O.  Files opened in binary mode (including @code{'b'} in the @emph{mode}
argument) return contents as @ref{2f0,,bytes} objects without any decoding.  In
text mode (the default, or when @code{'t'} is included in the @emph{mode} argument),
the contents of the file are returned as @ref{2ef,,str}, the bytes having been
first decoded using a platform-dependent encoding or using the specified
@emph{encoding} if given.

@cartouche
@quotation Note 
Python doesn’t depend on the underlying operating system’s notion of text
files; all the processing is done by Python itself, and is therefore
platform-independent.
@end quotation
@end cartouche

@emph{buffering} is an optional integer used to set the buffering policy.  Pass 0
to switch buffering off (only allowed in binary mode), 1 to select line
buffering (only usable in text mode), and an integer > 1 to indicate the size
in bytes of a fixed-size chunk buffer.  When no @emph{buffering} argument is
given, the default buffering policy works as follows:


@itemize *

@item 
Binary files are buffered in fixed-size chunks; the size of the buffer is
chosen using a heuristic trying to determine the underlying device’s “block
size” and falling back on @ref{1a56,,io.DEFAULT_BUFFER_SIZE}.  On many systems,
the buffer will typically be 4096 or 8192 bytes long.

@item 
“Interactive” text files (files for which @ref{1a57,,isatty()}
returns @code{True}) use line buffering.  Other text files use the policy
described above for binary files.
@end itemize

@emph{encoding} is the name of the encoding used to decode or encode the file.
This should only be used in text mode.  The default encoding is platform
dependent (whatever @ref{1b5,,locale.getpreferredencoding()} returns), but any
@ref{1a58,,text encoding} supported by Python
can be used.  See the @ref{1a,,codecs} module for
the list of supported encodings.

@emph{errors} is an optional string that specifies how encoding and decoding
errors are to be handled—this cannot be used in binary mode.
A variety of standard error handlers are available
(listed under @ref{162a,,Error Handlers}), though any
error handling name that has been registered with
@ref{fed,,codecs.register_error()} is also valid.  The standard names
include:


@itemize *

@item 
@code{'strict'} to raise a @ref{17c,,ValueError} exception if there is
an encoding error.  The default value of @code{None} has the same
effect.

@item 
@code{'ignore'} ignores errors.  Note that ignoring encoding errors
can lead to data loss.

@item 
@code{'replace'} causes a replacement marker (such as @code{'?'}) to be inserted
where there is malformed data.

@item 
@code{'surrogateescape'} will represent any incorrect bytes as low
surrogate code units ranging from U+DC80 to U+DCFF.
These surrogate code units will then be turned back into
the same bytes when the @code{surrogateescape} error handler is used
when writing data.  This is useful for processing files in an
unknown encoding.

@item 
@code{'xmlcharrefreplace'} is only supported when writing to a file.
Characters not supported by the encoding are replaced with the
appropriate XML character reference @code{&#nnn;}.

@item 
@code{'backslashreplace'} replaces malformed data by Python’s backslashed
escape sequences.

@item 
@code{'namereplace'} (also only supported when writing)
replaces unsupported characters with @code{\N@{...@}} escape sequences.
@end itemize

@geindex universal newlines; open() built-in function
@anchor{library/functions open-newline-parameter}@anchor{1cb}
@emph{newline} controls how @ref{88a,,universal newlines} mode works (it only
applies to text mode).  It can be @code{None}, @code{'@w{'}}, @code{'\n'}, @code{'\r'}, and
@code{'\r\n'}.  It works as follows:


@itemize *

@item 
When reading input from the stream, if @emph{newline} is @code{None}, universal
newlines mode is enabled.  Lines in the input can end in @code{'\n'},
@code{'\r'}, or @code{'\r\n'}, and these are translated into @code{'\n'} before
being returned to the caller.  If it is @code{'@w{'}}, universal newlines mode is
enabled, but line endings are returned to the caller untranslated.  If it
has any of the other legal values, input lines are only terminated by the
given string, and the line ending is returned to the caller untranslated.

@item 
When writing output to the stream, if @emph{newline} is @code{None}, any @code{'\n'}
characters written are translated to the system default line separator,
@ref{1a59,,os.linesep}.  If @emph{newline} is @code{'@w{'}} or @code{'\n'}, no translation
takes place.  If @emph{newline} is any of the other legal values, any @code{'\n'}
characters written are translated to the given string.
@end itemize

If @emph{closefd} is @code{False} and a file descriptor rather than a filename was
given, the underlying file descriptor will be kept open when the file is
closed.  If a filename is given @emph{closefd} must be @code{True} (the default);
otherwise, an error will be raised.

A custom opener can be used by passing a callable as @emph{opener}. The underlying
file descriptor for the file object is then obtained by calling @emph{opener} with
(@emph{file}, @emph{flags}). @emph{opener} must return an open file descriptor (passing
@ref{8ee,,os.open} as @emph{opener} results in functionality similar to passing
@code{None}).

The newly created file is @ref{a6e,,non-inheritable}.

The following example uses the @ref{c7c,,dir_fd} parameter of the
@ref{8ee,,os.open()} function to open a file relative to a given directory:

@example
>>> import os
>>> dir_fd = os.open('somedir', os.O_RDONLY)
>>> def opener(path, flags):
...     return os.open(path, flags, dir_fd=dir_fd)
...
>>> with open('spamspam.txt', 'w', opener=opener) as f:
...     print('This will be written to somedir/spamspam.txt', file=f)
...
>>> os.close(dir_fd)  # don't leak a file descriptor
@end example

The type of @ref{d68,,file object} returned by the @ref{1c8,,open()} function
depends on the mode.  When @ref{1c8,,open()} is used to open a file in a text
mode (@code{'w'}, @code{'r'}, @code{'wt'}, @code{'rt'}, etc.), it returns a subclass of
@ref{e47,,io.TextIOBase} (specifically @ref{889,,io.TextIOWrapper}).  When used
to open a file in a binary mode with buffering, the returned class is a
subclass of @ref{823,,io.BufferedIOBase}.  The exact class varies: in read
binary mode, it returns an @ref{daf,,io.BufferedReader}; in write binary and
append binary modes, it returns an @ref{1a5a,,io.BufferedWriter}, and in
read/write mode, it returns an @ref{1a5b,,io.BufferedRandom}.  When buffering is
disabled, the raw stream, a subclass of @ref{c62,,io.RawIOBase},
@ref{ea6,,io.FileIO}, is returned.

@geindex line-buffered I/O
@geindex unbuffered I/O
@geindex buffer size@comma{} I/O
@geindex I/O control; buffering
@geindex binary mode
@geindex text mode
@geindex module; sys

See also the file handling modules, such as @ref{7b,,fileinput}, @ref{9d,,io}
(where @ref{1c8,,open()} is declared), @ref{c0,,os}, @ref{c1,,os.path}, @ref{fd,,tempfile},
and @ref{e4,,shutil}.

Raises an @ref{1130,,auditing event} @code{open} with arguments @code{file}, @code{mode}, @code{flags}.

The @code{mode} and @code{flags} arguments may have been modified or inferred from
the original call.


Changed in version 3.3: 


@itemize *

@item 
The @emph{opener} parameter was added.

@item 
The @code{'x'} mode was added.

@item 
@ref{be8,,IOError} used to be raised, it is now an alias of @ref{2a1,,OSError}.

@item 
@ref{bb1,,FileExistsError} is now raised if the file opened in exclusive
creation mode (@code{'x'}) already exists.
@end itemize



Changed in version 3.4: 


@itemize *

@item 
The file is now non-inheritable.
@end itemize



Changed in version 3.5: 


@itemize *

@item 
If the system call is interrupted and the signal handler does not raise an
exception, the function now retries the system call instead of raising an
@ref{8e3,,InterruptedError} exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).

@item 
The @code{'namereplace'} error handler was added.
@end itemize



Changed in version 3.6: 


@itemize *

@item 
Support added to accept objects implementing @ref{795,,os.PathLike}.

@item 
On Windows, opening a console buffer may return a subclass of
@ref{c62,,io.RawIOBase} other than @ref{ea6,,io.FileIO}.
@end itemize



Changed in version 3.11: The @code{'U'} mode has been removed.

@end deffn

@geindex built-in function; ord()
@anchor{library/functions ord}@anchor{178e}
@deffn {Function} ord (c)

Given a string representing one Unicode character, return an integer
representing the Unicode code point of that character.  For example,
@code{ord('a')} returns the integer @code{97} and @code{ord('€')} (Euro sign)
returns @code{8364}.  This is the inverse of @ref{178f,,chr()}.
@end deffn

@geindex built-in function; pow()
@anchor{library/functions pow}@anchor{4cd}
@deffn {Function} pow (base, exp[, mod])

Return @emph{base} to the power @emph{exp}; if @emph{mod} is present, return @emph{base} to the
power @emph{exp}, modulo @emph{mod} (computed more efficiently than
@code{pow(base, exp) % mod}). The two-argument form @code{pow(base, exp)} is
equivalent to using the power operator: @code{base**exp}.

The arguments must have numeric types.  With mixed operand types, the
coercion rules for binary arithmetic operators apply.  For @ref{1b1,,int}
operands, the result has the same type as the operands (after coercion)
unless the second argument is negative; in that case, all arguments are
converted to float and a float result is delivered.  For example, @code{pow(10, 2)}
returns @code{100}, but @code{pow(10, -2)} returns @code{0.01}.  For a negative base of
type @ref{1b1,,int} or @ref{256,,float} and a non-integral exponent, a complex
result is delivered.  For example, @code{pow(-9, 0.5)} returns a value close
to @code{3j}.

For @ref{1b1,,int} operands @emph{base} and @emph{exp}, if @emph{mod} is present, @emph{mod} must
also be of integer type and @emph{mod} must be nonzero. If @emph{mod} is present and
@emph{exp} is negative, @emph{base} must be relatively prime to @emph{mod}. In that case,
@code{pow(inv_base, -exp, mod)} is returned, where @emph{inv_base} is an inverse to
@emph{base} modulo @emph{mod}.

Here’s an example of computing an inverse for @code{38} modulo @code{97}:

@example
>>> pow(38, -1, mod=97)
23
>>> 23 * 38 % 97 == 1
True
@end example


Changed in version 3.8: For @ref{1b1,,int} operands, the three-argument form of @code{pow} now allows
the second argument to be negative, permitting computation of modular
inverses.



Changed in version 3.8: Allow keyword arguments.  Formerly, only positional arguments were
supported.

@end deffn

@geindex built-in function; print()
@anchor{library/functions print}@anchor{af6}
@deffn {Function} print (*objects, sep=' ', end='\n', file=sys.stdout, flush=False)

Print @emph{objects} to the text stream @emph{file}, separated by @emph{sep} and followed
by @emph{end}.  @emph{sep}, @emph{end}, @emph{file}, and @emph{flush}, if present, must be given as keyword
arguments.

All non-keyword arguments are converted to strings like @ref{2ef,,str()} does and
written to the stream, separated by @emph{sep} and followed by @emph{end}.  Both @emph{sep}
and @emph{end} must be strings; they can also be @code{None}, which means to use the
default values.  If no @emph{objects} are given, @ref{af6,,print()} will just write
@emph{end}.

The @emph{file} argument must be an object with a @code{write(string)} method; if it
is not present or @code{None}, @ref{5ef,,sys.stdout} will be used.  Since printed
arguments are converted to text strings, @ref{af6,,print()} cannot be used with
binary mode file objects.  For these, use @code{file.write(...)} instead.

Whether the output is buffered is usually determined by @emph{file}, but if the
@emph{flush} keyword argument is true, the stream is forcibly flushed.


Changed in version 3.3: Added the @emph{flush} keyword argument.

@end deffn

@geindex property (built-in class)
@anchor{library/functions property}@anchor{501}
@deffn {Class} property (fget=None, fset=None, fdel=None, doc=None)

Return a property attribute.

@emph{fget} is a function for getting an attribute value.  @emph{fset} is a function
for setting an attribute value. @emph{fdel} is a function for deleting an attribute
value.  And @emph{doc} creates a docstring for the attribute.

A typical use is to define a managed attribute @code{x}:

@example
class C:
    def __init__(self):
        self._x = None

    def getx(self):
        return self._x

    def setx(self, value):
        self._x = value

    def delx(self):
        del self._x

    x = property(getx, setx, delx, "I'm the 'x' property.")
@end example

If @emph{c} is an instance of @emph{C}, @code{c.x} will invoke the getter,
@code{c.x = value} will invoke the setter, and @code{del c.x} the deleter.

If given, @emph{doc} will be the docstring of the property attribute. Otherwise, the
property will copy @emph{fget}’s docstring (if it exists).  This makes it possible to
create read-only properties easily using @ref{501,,property()} as a @ref{1bf,,decorator}:

@example
class Parrot:
    def __init__(self):
        self._voltage = 100000

    @@property
    def voltage(self):
        """Get the current voltage."""
        return self._voltage
@end example

The @code{@@property} decorator turns the @code{voltage()} method into a “getter”
for a read-only attribute with the same name, and it sets the docstring for
@emph{voltage} to “Get the current voltage.”

A property object has @code{getter}, @code{setter},
and @code{deleter} methods usable as decorators that create a
copy of the property with the corresponding accessor function set to the
decorated function.  This is best explained with an example:

@example
class C:
    def __init__(self):
        self._x = None

    @@property
    def x(self):
        """I'm the 'x' property."""
        return self._x

    @@x.setter
    def x(self, value):
        self._x = value

    @@x.deleter
    def x(self):
        del self._x
@end example

This code is exactly equivalent to the first example.  Be sure to give the
additional functions the same name as the original property (@code{x} in this
case.)

The returned property object also has the attributes @code{fget}, @code{fset}, and
@code{fdel} corresponding to the constructor arguments.


Changed in version 3.5: The docstrings of property objects are now writeable.

@end deffn
@anchor{library/functions func-range}@anchor{1a35}

@deffn {Class} range (stop)

@deffnx {Class} range (start, stop[, step])

Rather than being a function, @ref{438,,range} is actually an immutable
sequence type, as documented in @ref{1a5c,,Ranges} and @ref{1530,,Sequence Types — list@comma{} tuple@comma{} range}.
@end deffn

@geindex built-in function; repr()
@anchor{library/functions repr}@anchor{27d}
@deffn {Function} repr (object)

Return a string containing a printable representation of an object.  For many
types, this function makes an attempt to return a string that would yield an
object with the same value when passed to @ref{34a,,eval()}; otherwise, the
representation is a string enclosed in angle brackets that contains the name
of the type of the object together with additional information often
including the name and address of the object.  A class can control what this
function returns for its instances by defining a @code{__repr__()} method.
@end deffn

@geindex built-in function; reversed()
@anchor{library/functions reversed}@anchor{2f6}
@deffn {Function} reversed (seq)

Return a reverse @ref{11c3,,iterator}.  @emph{seq} must be an object which has
a @code{__reversed__()} method or supports the sequence protocol (the
@code{__len__()} method and the @code{__getitem__()} method with integer
arguments starting at @code{0}).
@end deffn

@geindex built-in function; round()
@anchor{library/functions round}@anchor{e75}
@deffn {Function} round (number[, ndigits])

Return @emph{number} rounded to @emph{ndigits} precision after the decimal
point.  If @emph{ndigits} is omitted or is @code{None}, it returns the
nearest integer to its input.

For the built-in types supporting @ref{e75,,round()}, values are rounded to the
closest multiple of 10 to the power minus @emph{ndigits}; if two multiples are
equally close, rounding is done toward the even choice (so, for example,
both @code{round(0.5)} and @code{round(-0.5)} are @code{0}, and @code{round(1.5)} is
@code{2}).  Any integer value is valid for @emph{ndigits} (positive, zero, or
negative).  The return value is an integer if @emph{ndigits} is omitted or
@code{None}.
Otherwise, the return value has the same type as @emph{number}.

For a general Python object @code{number}, @code{round} delegates to
@code{number.__round__}.

@cartouche
@quotation Note 
The behavior of @ref{e75,,round()} for floats can be surprising: for example,
@code{round(2.675, 2)} gives @code{2.67} instead of the expected @code{2.68}.
This is not a bug: it’s a result of the fact that most decimal fractions
can’t be represented exactly as a float.  See @ref{15e5,,Floating Point Arithmetic; Issues and Limitations} for
more information.
@end quotation
@end cartouche
@end deffn
@anchor{library/functions func-set}@anchor{1a36}

@deffn {Class} set ([iterable])

Return a new @ref{370,,set} object, optionally with elements taken from
@emph{iterable}.  @code{set} is a built-in class.  See @ref{370,,set} and
@ref{1a4f,,Set Types — set@comma{} frozenset} for documentation about this class.

For other containers see the built-in @ref{439,,frozenset}, @ref{43a,,list},
@ref{3e5,,tuple}, and @ref{3b0,,dict} classes, as well as the @ref{1c,,collections}
module.
@end deffn

@geindex built-in function; setattr()
@anchor{library/functions setattr}@anchor{1a37}
@deffn {Function} setattr (object, name, value)

This is the counterpart of @ref{708,,getattr()}.  The arguments are an object, a
string, and an arbitrary value.  The string may name an existing attribute or a
new attribute.  The function assigns the value to the attribute, provided the
object allows it.  For example, @code{setattr(x, 'foobar', 123)} is equivalent to
@code{x.foobar = 123}.

@cartouche
@quotation Note 
Since @ref{187e,,private name mangling} happens at
compilation time, one must manually mangle a private attribute’s
(attributes with two leading underscores) name in order to set it with
@ref{1a37,,setattr()}.
@end quotation
@end cartouche
@end deffn

@geindex slice (built-in class)
@anchor{library/functions slice}@anchor{ffb}
@deffn {Class} slice (stop)

@deffnx {Class} slice (start, stop[, step])

Return a @ref{1a5d,,slice} object representing the set of indices specified by
@code{range(start, stop, step)}.  The @emph{start} and @emph{step} arguments default to
@code{None}.  Slice objects have read-only data attributes @code{start},
@code{stop}, and @code{step} which merely return the argument
values (or their default).  They have no other explicit functionality;
however, they are used by NumPy and other third-party packages.
Slice objects are also generated when extended indexing syntax is used.  For
example: @code{a[start:stop:step]} or @code{a[start:stop, i]}.  See
@ref{672,,itertools.islice()} for an alternate version that returns an iterator.
@end deffn

@geindex built-in function; sorted()
@anchor{library/functions sorted}@anchor{704}
@deffn {Function} sorted (iterable, /, *, key=None, reverse=False)

Return a new sorted list from the items in @emph{iterable}.

Has two optional arguments which must be specified as keyword arguments.

@emph{key} specifies a function of one argument that is used to extract a comparison
key from each element in @emph{iterable} (for example, @code{key=str.lower}).  The
default value is @code{None} (compare the elements directly).

@emph{reverse} is a boolean value.  If set to @code{True}, then the list elements are
sorted as if each comparison were reversed.

Use @ref{d7c,,functools.cmp_to_key()} to convert an old-style @emph{cmp} function to a
@emph{key} function.

The built-in @ref{704,,sorted()} function is guaranteed to be stable. A sort is
stable if it guarantees not to change the relative order of elements that
compare equal — this is helpful for sorting in multiple passes (for
example, sort by department, then by salary grade).

The sort algorithm uses only @code{<} comparisons between items.  While
defining an @ref{17a7,,__lt__()} method will suffice for sorting,
@geindex Python Enhancement Proposals; PEP 8
PEP 8@footnote{https://www.python.org/dev/peps/pep-0008} recommends that all six @ref{1900,,rich comparisons} be implemented.  This will help avoid bugs when using
the same data with other ordering tools such as @ref{a7e,,max()} that rely
on a different underlying method.  Implementing all six comparisons
also helps avoid confusion for mixed type comparisons which can call
reflected the @ref{17aa,,__gt__()} method.

For sorting examples and a brief sorting tutorial, see @ref{1a5e,,Sorting HOW TO}.
@end deffn

@geindex built-in function; staticmethod()
@anchor{library/functions staticmethod}@anchor{253}
@deffn {Function} @@staticmethod

Transform a method into a static method.

A static method does not receive an implicit first argument. To declare a static
method, use this idiom:

@example
class C:
    @@staticmethod
    def f(arg1, arg2, ...): ...
@end example

The @code{@@staticmethod} form is a function @ref{1bf,,decorator} – see
@ref{151c,,Function definitions} for details.

A static method can be called either on the class (such as @code{C.f()}) or on
an instance (such as @code{C().f()}). Moreover, they can be called as regular
functions (such as @code{f()}).

Static methods in Python are similar to those found in Java or C++. Also, see
@ref{254,,classmethod()} for a variant that is useful for creating alternate class
constructors.

Like all decorators, it is also possible to call @code{staticmethod} as
a regular function and do something with its result.  This is needed
in some cases where you need a reference to a function from a class
body and you want to avoid the automatic transformation to instance
method.  For these cases, use this idiom:

@example
def regular_function():
    ...

class C:
    method = staticmethod(regular_function)
@end example

For more information on static methods, see @ref{1787,,The standard type hierarchy}.


Changed in version 3.10: Static methods now inherit the method attributes (@code{__module__},
@code{__name__}, @code{__qualname__}, @code{__doc__} and @code{__annotations__}),
have a new @code{__wrapped__} attribute, and are now callable as regular
functions.

@end deffn

@geindex string; str() (built-in function)
@anchor{library/functions func-str}@anchor{1a38}

@deffn {Class} str (object='@w{'})

@deffnx {Class} str (object=b'@w{'}, encoding='utf-8', errors='strict')

Return a @ref{2ef,,str} version of @emph{object}.  See @ref{2ef,,str()} for details.

@code{str} is the built-in string @ref{11ae,,class}.  For general information
about strings, see @ref{14e2,,Text Sequence Type — str}.
@end deffn

@geindex built-in function; sum()
@anchor{library/functions sum}@anchor{50d}
@deffn {Function} sum (iterable, /, start=0)

Sums @emph{start} and the items of an @emph{iterable} from left to right and returns the
total.  The @emph{iterable}’s items are normally numbers, and the start value is not
allowed to be a string.

For some use cases, there are good alternatives to @ref{50d,,sum()}.
The preferred, fast way to concatenate a sequence of strings is by calling
@code{'@w{'}.join(sequence)}.  To add floating point values with extended precision,
see @ref{f3a,,math.fsum()}.  To concatenate a series of iterables, consider using
@ref{1a5f,,itertools.chain()}.


Changed in version 3.8: The @emph{start} parameter can be specified as a keyword argument.

@end deffn

@geindex super (built-in class)
@anchor{library/functions super}@anchor{78c}
@deffn {Class} super ([type[, object-or-type]])

Return a proxy object that delegates method calls to a parent or sibling
class of @emph{type}.  This is useful for accessing inherited methods that have
been overridden in a class.

The @emph{object-or-type} determines the @ref{1a60,,method resolution order}
to be searched.  The search starts from the class right after the
@emph{type}.

For example, if @ref{1a61,,__mro__} of @emph{object-or-type} is
@code{D -> B -> C -> A -> object} and the value of @emph{type} is @code{B},
then @ref{78c,,super()} searches @code{C -> A -> object}.

The @ref{1a61,,__mro__} attribute of the @emph{object-or-type} lists the method
resolution search order used by both @ref{708,,getattr()} and @ref{78c,,super()}.  The
attribute is dynamic and can change whenever the inheritance hierarchy is
updated.

If the second argument is omitted, the super object returned is unbound.  If
the second argument is an object, @code{isinstance(obj, type)} must be true.  If
the second argument is a type, @code{issubclass(type2, type)} must be true (this
is useful for classmethods).

There are two typical use cases for @emph{super}.  In a class hierarchy with
single inheritance, @emph{super} can be used to refer to parent classes without
naming them explicitly, thus making the code more maintainable.  This use
closely parallels the use of @emph{super} in other programming languages.

The second use case is to support cooperative multiple inheritance in a
dynamic execution environment.  This use case is unique to Python and is
not found in statically compiled languages or languages that only support
single inheritance.  This makes it possible to implement “diamond diagrams”
where multiple base classes implement the same method.  Good design dictates
that such implementations have the same calling signature in every case (because the
order of calls is determined at runtime, because that order adapts
to changes in the class hierarchy, and because that order can include
sibling classes that are unknown prior to runtime).

For both use cases, a typical superclass call looks like this:

@example
class C(B):
    def method(self, arg):
        super().method(arg)    # This does the same thing as:
                               # super(C, self).method(arg)
@end example

In addition to method lookups, @ref{78c,,super()} also works for attribute
lookups.  One possible use case for this is calling @ref{17bc,,descriptors}
in a parent or sibling class.

Note that @ref{78c,,super()} is implemented as part of the binding process for
explicit dotted attribute lookups such as @code{super().__getitem__(name)}.
It does so by implementing its own @code{__getattribute__()} method for searching
classes in a predictable order that supports cooperative multiple inheritance.
Accordingly, @ref{78c,,super()} is undefined for implicit lookups using statements or
operators such as @code{super()[name]}.

Also note that, aside from the zero argument form, @ref{78c,,super()} is not
limited to use inside methods.  The two argument form specifies the
arguments exactly and makes the appropriate references.  The zero
argument form only works inside a class definition, as the compiler fills
in the necessary details to correctly retrieve the class being defined,
as well as accessing the current instance for ordinary methods.

For practical suggestions on how to design cooperative classes using
@ref{78c,,super()}, see guide to using super()@footnote{https://rhettinger.wordpress.com/2011/05/26/super-considered-super/}.
@end deffn
@anchor{library/functions func-tuple}@anchor{1a39}

@deffn {Class} tuple ([iterable])

Rather than being a function, @ref{3e5,,tuple} is actually an immutable
sequence type, as documented in @ref{19a3,,Tuples} and @ref{1530,,Sequence Types — list@comma{} tuple@comma{} range}.
@end deffn

@geindex type (built-in class)
@anchor{library/functions type}@anchor{89e}
@deffn {Class} type (object)

@deffnx {Class} type (name, bases, dict, **kwds)

@geindex object; type

With one argument, return the type of an @emph{object}.  The return value is a
type object and generally the same object as returned by
@ref{fff,,object.__class__}.

The @ref{236,,isinstance()} built-in function is recommended for testing the type
of an object, because it takes subclasses into account.

With three arguments, return a new type object.  This is essentially a
dynamic form of the @ref{e72,,class} statement. The @emph{name} string is
the class name and becomes the @ref{e70,,__name__} attribute.
The @emph{bases} tuple contains the base classes and becomes the
@ref{ffe,,__bases__} attribute; if empty, @ref{5a5,,object}, the
ultimate base of all classes, is added.  The @emph{dict} dictionary contains
attribute and method definitions for the class body; it may be copied
or wrapped before becoming the @ref{7a4,,__dict__} attribute.
The following two statements create identical @ref{89e,,type} objects:

@example
>>> class X:
...     a = 1
...
>>> X = type('X', (), dict(a=1))
@end example

See also @ref{1a62,,Type Objects}.

Keyword arguments provided to the three argument form are passed to the
appropriate metaclass machinery (usually @ref{78d,,__init_subclass__()})
in the same way that keywords in a class
definition (besides @emph{metaclass}) would.

See also @ref{78f,,Customizing class creation}.


Changed in version 3.6: Subclasses of @ref{89e,,type} which don’t override @code{type.__new__} may no
longer use the one-argument form to get the type of an object.

@end deffn

@geindex built-in function; vars()
@anchor{library/functions vars}@anchor{12fc}
@deffn {Function} vars ([object])

Return the @ref{7a4,,__dict__} attribute for a module, class, instance,
or any other object with a @ref{7a4,,__dict__} attribute.

Objects such as modules and instances have an updateable @ref{7a4,,__dict__}
attribute; however, other objects may have write restrictions on their
@ref{7a4,,__dict__} attributes (for example, classes use a
@ref{246,,types.MappingProxyType} to prevent direct dictionary updates).

Without an argument, @ref{12fc,,vars()} acts like @ref{713,,locals()}.  Note, the
locals dictionary is only useful for reads since updates to the locals
dictionary are ignored.

A @ref{162,,TypeError} exception is raised if an object is specified but
it doesn’t have a @ref{7a4,,__dict__} attribute (for example, if
its class defines the @ref{1025,,__slots__} attribute).
@end deffn

@geindex built-in function; zip()
@anchor{library/functions zip}@anchor{247}
@deffn {Function} zip (*iterables, strict=False)

Iterate over several iterables in parallel, producing tuples with an item
from each one.

Example:

@example
>>> for item in zip([1, 2, 3], ['sugar', 'spice', 'everything nice']):
...     print(item)
...
(1, 'sugar')
(2, 'spice')
(3, 'everything nice')
@end example

More formally: @ref{247,,zip()} returns an iterator of tuples, where the @emph{i}-th
tuple contains the @emph{i}-th element from each of the argument iterables.

Another way to think of @ref{247,,zip()} is that it turns rows into columns, and
columns into rows.  This is similar to transposing a matrix@footnote{https://en.wikipedia.org/wiki/Transpose}.

@ref{247,,zip()} is lazy: The elements won’t be processed until the iterable is
iterated on, e.g. by a @code{for} loop or by wrapping in a
@ref{43a,,list}.

One thing to consider is that the iterables passed to @ref{247,,zip()} could have
different lengths; sometimes by design, and sometimes because of a bug in
the code that prepared these iterables.  Python offers three different
approaches to dealing with this issue:


@itemize *

@item 
By default, @ref{247,,zip()} stops when the shortest iterable is exhausted.
It will ignore the remaining items in the longer iterables, cutting off
the result to the length of the shortest iterable:

@example
>>> list(zip(range(3), ['fee', 'fi', 'fo', 'fum']))
[(0, 'fee'), (1, 'fi'), (2, 'fo')]
@end example

@item 
@ref{247,,zip()} is often used in cases where the iterables are assumed to be
of equal length.  In such cases, it’s recommended to use the @code{strict=True}
option. Its output is the same as regular @ref{247,,zip()}:

@example
>>> list(zip(('a', 'b', 'c'), (1, 2, 3), strict=True))
[('a', 1), ('b', 2), ('c', 3)]
@end example

Unlike the default behavior, it checks that the lengths of iterables are
identical, raising a @ref{17c,,ValueError} if they aren’t:

@example
>>> list(zip(range(3), ['fee', 'fi', 'fo', 'fum'], strict=True))
Traceback (most recent call last):
  ...
ValueError: zip() argument 2 is longer than argument 1
@end example

Without the @code{strict=True} argument, any bug that results in iterables of
different lengths will be silenced, possibly manifesting as a hard-to-find
bug in another part of the program.

@item 
Shorter iterables can be padded with a constant value to make all the
iterables have the same length.  This is done by
@ref{e43,,itertools.zip_longest()}.
@end itemize

Edge cases: With a single iterable argument, @ref{247,,zip()} returns an
iterator of 1-tuples.  With no arguments, it returns an empty iterator.

Tips and tricks:


@itemize *

@item 
The left-to-right evaluation order of the iterables is guaranteed. This
makes possible an idiom for clustering a data series into n-length groups
using @code{zip(*[iter(s)]*n, strict=True)}.  This repeats the @emph{same} iterator
@code{n} times so that each output tuple has the result of @code{n} calls to the
iterator. This has the effect of dividing the input into n-length chunks.

@item 
@ref{247,,zip()} in conjunction with the @code{*} operator can be used to unzip a
list:

@example
>>> x = [1, 2, 3]
>>> y = [4, 5, 6]
>>> list(zip(x, y))
[(1, 4), (2, 5), (3, 6)]
>>> x2, y2 = zip(*zip(x, y))
>>> x == list(x2) and y == list(y2)
True
@end example
@end itemize


Changed in version 3.10: Added the @code{strict} argument.

@end deffn

@geindex built-in function; __import__()
@anchor{library/functions import__}@anchor{3b7}@anchor{library/functions __import__}@anchor{1a63}
@deffn {Function} __import__ (name, globals=None, locals=None, fromlist=(), level=0)

@geindex statement; import
@geindex module; imp

@cartouche
@quotation Note 
This is an advanced function that is not needed in everyday Python
programming, unlike @ref{d3e,,importlib.import_module()}.
@end quotation
@end cartouche

This function is invoked by the @ref{e35,,import} statement.  It can be
replaced (by importing the @ref{11,,builtins} module and assigning to
@code{builtins.__import__}) in order to change semantics of the
@code{import} statement, but doing so is @strong{strongly} discouraged as it
is usually simpler to use import hooks (see 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302}) to attain the same
goals and does not cause issues with code which assumes the default import
implementation is in use.  Direct use of @ref{3b7,,__import__()} is also
discouraged in favor of @ref{d3e,,importlib.import_module()}.

The function imports the module @emph{name}, potentially using the given @emph{globals}
and @emph{locals} to determine how to interpret the name in a package context.
The @emph{fromlist} gives the names of objects or submodules that should be
imported from the module given by @emph{name}.  The standard implementation does
not use its @emph{locals} argument at all and uses its @emph{globals} only to
determine the package context of the @ref{e35,,import} statement.

@emph{level} specifies whether to use absolute or relative imports. @code{0} (the
default) means only perform absolute imports.  Positive values for
@emph{level} indicate the number of parent directories to search relative to the
directory of the module calling @ref{3b7,,__import__()} (see 
@geindex Python Enhancement Proposals; PEP 328
PEP 328@footnote{https://www.python.org/dev/peps/pep-0328} for the
details).

When the @emph{name} variable is of the form @code{package.module}, normally, the
top-level package (the name up till the first dot) is returned, @emph{not} the
module named by @emph{name}.  However, when a non-empty @emph{fromlist} argument is
given, the module named by @emph{name} is returned.

For example, the statement @code{import spam} results in bytecode resembling the
following code:

@example
spam = __import__('spam', globals(), locals(), [], 0)
@end example

The statement @code{import spam.ham} results in this call:

@example
spam = __import__('spam.ham', globals(), locals(), [], 0)
@end example

Note how @ref{3b7,,__import__()} returns the toplevel module here because this is
the object that is bound to a name by the @ref{e35,,import} statement.

On the other hand, the statement @code{from spam.ham import eggs, sausage as
saus} results in

@example
_temp = __import__('spam.ham', globals(), locals(), ['eggs', 'sausage'], 0)
eggs = _temp.eggs
saus = _temp.sausage
@end example

Here, the @code{spam.ham} module is returned from @ref{3b7,,__import__()}.  From this
object, the names to import are retrieved and assigned to their respective
names.

If you simply want to import a module (potentially within a package) by name,
use @ref{d3e,,importlib.import_module()}.


Changed in version 3.3: Negative values for @emph{level} are no longer supported (which also changes
the default value to 0).



Changed in version 3.9: When the command line options @ref{460,,-E} or @ref{461,,-I} are being used,
the environment variable 
@geindex PYTHONCASEOK
@geindex environment variable; PYTHONCASEOK
@ref{45f,,PYTHONCASEOK} is now ignored.

@end deffn

@node Built-in Constants,Built-in Types,Built-in Functions,The Python Standard Library
@anchor{library/constants doc}@anchor{1a64}@anchor{library/constants built-in-constants}@anchor{1a65}@anchor{library/constants built-in-consts}@anchor{1a66}
@section Built-in Constants


A small number of constants live in the built-in namespace.  They are:

@geindex False (built-in variable)
@anchor{library/constants False}@anchor{659}
@deffn {Data} False

The false value of the @ref{2f7,,bool} type. Assignments to @code{False}
are illegal and raise a @ref{21a,,SyntaxError}.
@end deffn

@geindex True (built-in variable)
@anchor{library/constants True}@anchor{748}
@deffn {Data} True

The true value of the @ref{2f7,,bool} type. Assignments to @code{True}
are illegal and raise a @ref{21a,,SyntaxError}.
@end deffn

@geindex None (built-in variable)
@anchor{library/constants None}@anchor{182}
@deffn {Data} None

An object frequently used to represent the absence of a value, as when
default arguments are not passed to a function. Assignments to @code{None}
are illegal and raise a @ref{21a,,SyntaxError}.
@code{None} is the sole instance of the @code{NoneType} type.
@end deffn

@geindex NotImplemented (built-in variable)
@anchor{library/constants NotImplemented}@anchor{24c}
@deffn {Data} NotImplemented

A special value which should be returned by the binary special methods
(e.g. @code{__eq__()}, @code{__lt__()}, @code{__add__()}, @code{__rsub__()},
etc.) to indicate that the operation is not implemented with respect to
the other type; may be returned by the in-place binary special methods
(e.g. @code{__imul__()}, @code{__iand__()}, etc.) for the same purpose.
It should not be evaluated in a boolean context.
@code{NotImplemented} is the sole instance of the @ref{2d9,,types.NotImplementedType} type.

@cartouche
@quotation Note 
When a binary (or in-place) method returns @code{NotImplemented} the
interpreter will try the reflected operation on the other type (or some
other fallback, depending on the operator).  If all attempts return
@code{NotImplemented}, the interpreter will raise an appropriate exception.
Incorrectly returning @code{NotImplemented} will result in a misleading
error message or the @code{NotImplemented} value being returned to Python code.

See @ref{1788,,Implementing the arithmetic operations} for examples.
@end quotation
@end cartouche

@cartouche
@quotation Note 
@code{NotImplementedError} and @code{NotImplemented} are not interchangeable,
even though they have similar names and purposes.
See @ref{8a3,,NotImplementedError} for details on when to use it.
@end quotation
@end cartouche


Changed in version 3.9: Evaluating @code{NotImplemented} in a boolean context is deprecated. While
it currently evaluates as true, it will emit a @ref{1b2,,DeprecationWarning}.
It will raise a @ref{162,,TypeError} in a future version of Python.

@end deffn

@geindex ...; ellipsis literal

@geindex Ellipsis (built-in variable)
@anchor{library/constants Ellipsis}@anchor{1a67}
@deffn {Data} Ellipsis

The same as the ellipsis literal “@code{...}”. Special value used mostly in conjunction
with extended slicing syntax for user-defined container data types.
@code{Ellipsis} is the sole instance of the @ref{2d7,,types.EllipsisType} type.
@end deffn

@geindex __debug__ (built-in variable)
@anchor{library/constants debug__}@anchor{257}@anchor{library/constants __debug__}@anchor{1a68}
@deffn {Data} __debug__

This constant is true if Python was not started with an @ref{912,,-O} option.
See also the @ref{46d,,assert} statement.
@end deffn

@cartouche
@quotation Note 
The names @ref{182,,None}, @ref{659,,False}, @ref{748,,True} and @ref{257,,__debug__}
cannot be reassigned (assignments to them, even as an attribute name, raise
@ref{21a,,SyntaxError}), so they can be considered “true” constants.
@end quotation
@end cartouche

@menu
* Constants added by the site module:: 

@end menu

@node Constants added by the site module,,,Built-in Constants
@anchor{library/constants constants-added-by-the-site-module}@anchor{1a69}
@subsection Constants added by the @code{site} module


The @ref{e6,,site} module (which is imported automatically during startup, except
if the @ref{d4e,,-S} command-line option is given) adds several constants to the
built-in namespace.  They are useful for the interactive interpreter shell and
should not be used in programs.

@geindex quit (built-in variable)
@geindex exit (built-in variable)
@anchor{library/constants quit}@anchor{1a6a}
@deffn {Data} quit (code=None)
@anchor{library/constants exit}@anchor{1a6b}
@deffnx {Data} exit (code=None)

Objects that when printed, print a message like “Use quit() or Ctrl-D
(i.e. EOF) to exit”, and when called, raise @ref{892,,SystemExit} with the
specified exit code.
@end deffn

@geindex copyright (built-in variable)
@geindex credits (built-in variable)
@anchor{library/constants copyright}@anchor{1a6c}
@deffn {Data} copyright
@anchor{library/constants credits}@anchor{1a6d}
@deffnx {Data} credits

Objects that when printed or called, print the text of copyright or
credits, respectively.
@end deffn

@geindex license (built-in variable)
@anchor{library/constants license}@anchor{1a6e}
@deffn {Data} license

Object that when printed, prints the message “Type license() to see the
full license text”, and when called, displays the full license text in a
pager-like fashion (one screen at a time).
@end deffn

@c XXX: reference/datamodel and this have quite a few overlaps!

@node Built-in Types,Built-in Exceptions,Built-in Constants,The Python Standard Library
@anchor{library/stdtypes doc}@anchor{1a6f}@anchor{library/stdtypes bltin-types}@anchor{1a70}@anchor{library/stdtypes built-in-types}@anchor{1a71}
@section Built-in Types


The following sections describe the standard types that are built into the
interpreter.

@geindex built-in; types

The principal built-in types are numerics, sequences, mappings, classes,
instances and exceptions.

Some collection classes are mutable.  The methods that add, subtract, or
rearrange their members in place, and don’t return a specific item, never return
the collection instance itself but @code{None}.

Some operations are supported by several object types; in particular,
practically all objects can be compared for equality, tested for truth
value, and converted to a string (with the @ref{27d,,repr()} function or the
slightly different @ref{2ef,,str()} function).  The latter function is implicitly
used when an object is written by the @ref{af6,,print()} function.

@menu
* Truth Value Testing:: 
* Boolean Operations — and@comma{} or@comma{} not: Boolean Operations — and or not. 
* Comparisons: Comparisons<2>. 
* Numeric Types — int@comma{} float@comma{} complex: Numeric Types — int float complex. 
* Iterator Types:: 
* Sequence Types — list@comma{} tuple@comma{} range: Sequence Types — list tuple range. 
* Text Sequence Type — str:: 
* Binary Sequence Types — bytes@comma{} bytearray@comma{} memoryview: Binary Sequence Types — bytes bytearray memoryview. 
* Set Types — set@comma{} frozenset: Set Types — set frozenset. 
* Mapping Types — dict:: 
* Context Manager Types:: 
* Type Annotation Types — Generic Alias@comma{} Union: Type Annotation Types — Generic Alias Union. 
* Other Built-in Types:: 
* Special Attributes:: 

@end menu

@node Truth Value Testing,Boolean Operations — and or not,,Built-in Types
@anchor{library/stdtypes truth}@anchor{1a3a}@anchor{library/stdtypes truth-value-testing}@anchor{1a72}
@subsection Truth Value Testing


@geindex statement; if
@geindex statement; while
@geindex truth; value
@geindex Boolean; operations
@geindex false

Any object can be tested for truth value, for use in an @ref{2fb,,if} or
@ref{14ec,,while} condition or as operand of the Boolean operations below.

@geindex true

By default, an object is considered true unless its class defines either a
@code{__bool__()} method that returns @code{False} or a @code{__len__()} method that
returns zero, when called with the object. @footnote{@w{(1)} 
Additional information on these special methods may be found in the Python
Reference Manual (@ref{17a2,,Basic customization}).
}  Here are most of the built-in
objects considered false:

@quotation

@geindex None (Built-in object)
@geindex False (Built-in object)
@end quotation


@itemize *

@item 
constants defined to be false: @code{None} and @code{False}.

@item 
zero of any numeric type: @code{0}, @code{0.0}, @code{0j}, @code{Decimal(0)},
@code{Fraction(0, 1)}

@item 
empty sequences and collections: @code{'@w{'}}, @code{()}, @code{[]}, @code{@{@}}, @code{set()},
@code{range(0)}
@end itemize

@geindex operator; or
@geindex operator; and
@geindex False
@geindex True

Operations and built-in functions that have a Boolean result always return @code{0}
or @code{False} for false and @code{1} or @code{True} for true, unless otherwise stated.
(Important exception: the Boolean operations @code{or} and @code{and} always return
one of their operands.)

@node Boolean Operations — and or not,Comparisons<2>,Truth Value Testing,Built-in Types
@anchor{library/stdtypes boolean}@anchor{1a73}@anchor{library/stdtypes boolean-operations-and-or-not}@anchor{1a74}
@subsection Boolean Operations — @code{and}, @code{or}, @code{not}


@geindex Boolean; operations

These are the Boolean operations, ordered by ascending priority:


@multitable {xxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} 
@headitem

Operation

@tab

Result

@tab

Notes

@item

@code{x or y}

@tab

if @emph{x} is false, then @emph{y}, else
@emph{x}

@tab

(1)

@item

@code{x and y}

@tab

if @emph{x} is false, then @emph{x}, else
@emph{y}

@tab

(2)

@item

@code{not x}

@tab

if @emph{x} is false, then @code{True},
else @code{False}

@tab

(3)

@end multitable


@geindex operator; and
@geindex operator; or
@geindex operator; not

Notes:


@enumerate 

@item 
This is a short-circuit operator, so it only evaluates the second
argument if the first one is false.

@item 
This is a short-circuit operator, so it only evaluates the second
argument if the first one is true.

@item 
@code{not} has a lower priority than non-Boolean operators, so @code{not a == b} is
interpreted as @code{not (a == b)}, and @code{a == not b} is a syntax error.
@end enumerate

@node Comparisons<2>,Numeric Types — int float complex,Boolean Operations — and or not,Built-in Types
@anchor{library/stdtypes comparisons}@anchor{1a75}@anchor{library/stdtypes stdcomparisons}@anchor{1a76}
@subsection Comparisons


@geindex chaining; comparisons
@geindex operator; comparison
@geindex operator; ==
@geindex operator; < (less)
@geindex operator; <=
@geindex operator; > (greater)
@geindex operator; >=
@geindex operator; !=
@geindex operator; is
@geindex operator; is not

There are eight comparison operations in Python.  They all have the same
priority (which is higher than that of the Boolean operations).  Comparisons can
be chained arbitrarily; for example, @code{x < y <= z} is equivalent to @code{x < y and
y <= z}, except that @emph{y} is evaluated only once (but in both cases @emph{z} is not
evaluated at all when @code{x < y} is found to be false).

This table summarizes the comparison operations:


@multitable {xxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Operation

@tab

Meaning

@item

@code{<}

@tab

strictly less than

@item

@code{<=}

@tab

less than or equal

@item

@code{>}

@tab

strictly greater than

@item

@code{>=}

@tab

greater than or equal

@item

@code{==}

@tab

equal

@item

@code{!=}

@tab

not equal

@item

@code{is}

@tab

object identity

@item

@code{is not}

@tab

negated object identity

@end multitable


@geindex object; numeric
@geindex objects; comparing

Objects of different types, except different numeric types, never compare equal.
The @code{==} operator is always defined but for some object types (for example,
class objects) is equivalent to @ref{2fd,,is}. The @code{<}, @code{<=}, @code{>} and @code{>=}
operators are only defined where they make sense; for example, they raise a
@ref{162,,TypeError} exception when one of the arguments is a complex number.

@geindex __eq__() (instance method)
@geindex __ne__() (instance method)
@geindex __lt__() (instance method)
@geindex __le__() (instance method)
@geindex __gt__() (instance method)
@geindex __ge__() (instance method)

Non-identical instances of a class normally compare as non-equal unless the
class defines the @ref{618,,__eq__()} method.

Instances of a class cannot be ordered with respect to other instances of the
same class, or other types of object, unless the class defines enough of the
methods @ref{17a7,,__lt__()}, @ref{17a8,,__le__()}, @ref{17aa,,__gt__()}, and
@ref{17ab,,__ge__()} (in general, @ref{17a7,,__lt__()} and
@ref{618,,__eq__()} are sufficient, if you want the conventional meanings of the
comparison operators).

The behavior of the @ref{2fd,,is} and @ref{1909,,is not} operators cannot be
customized; also they can be applied to any two objects and never raise an
exception.

@geindex operator; in
@geindex operator; not in

Two more operations with the same syntactic priority, @ref{2fc,,in} and
@ref{17e1,,not in}, are supported by types that are @ref{dcc,,iterable} or
implement the @code{__contains__()} method.

@node Numeric Types — int float complex,Iterator Types,Comparisons<2>,Built-in Types
@anchor{library/stdtypes numeric-types-int-float-complex}@anchor{1a77}@anchor{library/stdtypes typesnumeric}@anchor{14de}
@subsection Numeric Types — @code{int}, @code{float}, @code{complex}


@geindex object; numeric
@geindex object; Boolean
@geindex object; integer
@geindex object; floating point
@geindex object; complex number
@geindex C; language

There are three distinct numeric types: @emph{integers}, @emph{floating
point numbers}, and @emph{complex numbers}.  In addition, Booleans are a
subtype of integers.  Integers have unlimited precision.  Floating point
numbers are usually implemented using double in C; information
about the precision and internal representation of floating point
numbers for the machine on which your program is running is available
in @ref{1290,,sys.float_info}.  Complex numbers have a real and imaginary
part, which are each a floating point number.  To extract these parts
from a complex number @emph{z}, use @code{z.real} and @code{z.imag}. (The standard
library includes the additional numeric types @ref{172,,fractions.Fraction}, for
rationals, and @ref{248,,decimal.Decimal}, for floating-point numbers with
user-definable precision.)

@geindex numeric; literals
@geindex integer; literals
@geindex floating point; literals
@geindex complex number; literals
@geindex hexadecimal; literals
@geindex octal; literals
@geindex binary; literals

Numbers are created by numeric literals or as the result of built-in functions
and operators.  Unadorned integer literals (including hex, octal and binary
numbers) yield integers.  Numeric literals containing a decimal point or an
exponent sign yield floating point numbers.  Appending @code{'j'} or @code{'J'} to a
numeric literal yields an imaginary number (a complex number with a zero real
part) which you can add to an integer or float to get a complex number with real
and imaginary parts.

@geindex arithmetic
@geindex built-in function; int
@geindex built-in function; float
@geindex built-in function; complex
@geindex operator; + (plus)
@geindex + (plus); unary operator
@geindex + (plus); binary operator
@geindex operator; - (minus)
@geindex - (minus); unary operator
@geindex - (minus); binary operator
@geindex operator; * (asterisk)
@geindex operator; / (slash)
@geindex operator; //
@geindex operator; % (percent)
@geindex operator; **

Python fully supports mixed arithmetic: when a binary arithmetic operator has
operands of different numeric types, the operand with the “narrower” type is
widened to that of the other, where integer is narrower than floating point,
which is narrower than complex. A comparison between numbers of different types
behaves as though the exact values of those numbers were being compared. @footnote{@w{(2)} 
As a consequence, the list @code{[1, 2]} is considered equal to @code{[1.0, 2.0]}, and
similarly for tuples.
}

The constructors @ref{1b1,,int()}, @ref{256,,float()}, and
@ref{33b,,complex()} can be used to produce numbers of a specific type.

All numeric types (except complex) support the following operations (for priorities of
the operations, see @ref{192b,,Operator precedence}):


@multitable {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Operation

@tab

Result

@tab

Notes

@tab

Full documentation

@item

@code{x + y}

@tab

sum of @emph{x} and @emph{y}

@tab

@tab

@item

@code{x - y}

@tab

difference of @emph{x} and @emph{y}

@tab

@tab

@item

@code{x * y}

@tab

product of @emph{x} and @emph{y}

@tab

@tab

@item

@code{x / y}

@tab

quotient of @emph{x} and @emph{y}

@tab

@tab

@item

@code{x // y}

@tab

floored quotient of @emph{x} and
@emph{y}

@tab

(1)

@tab

@item

@code{x % y}

@tab

remainder of @code{x / y}

@tab

(2)

@tab

@item

@code{-x}

@tab

@emph{x} negated

@tab

@tab

@item

@code{+x}

@tab

@emph{x} unchanged

@tab

@tab

@item

@code{abs(x)}

@tab

absolute value or magnitude of
@emph{x}

@tab

@tab

@ref{1582,,abs()}

@item

@code{int(x)}

@tab

@emph{x} converted to integer

@tab

(3)(6)

@tab

@ref{1b1,,int()}

@item

@code{float(x)}

@tab

@emph{x} converted to floating point

@tab

(4)(6)

@tab

@ref{256,,float()}

@item

@code{complex(re, im)}

@tab

a complex number with real part
@emph{re}, imaginary part @emph{im}.
@emph{im} defaults to zero.

@tab

(6)

@tab

@ref{33b,,complex()}

@item

@code{c.conjugate()}

@tab

conjugate of the complex number
@emph{c}

@tab

@tab

@item

@code{divmod(x, y)}

@tab

the pair @code{(x // y, x % y)}

@tab

(2)

@tab

@ref{4a8,,divmod()}

@item

@code{pow(x, y)}

@tab

@emph{x} to the power @emph{y}

@tab

(5)

@tab

@ref{4cd,,pow()}

@item

@code{x ** y}

@tab

@emph{x} to the power @emph{y}

@tab

(5)

@tab

@end multitable


@geindex operations on; numeric; types
@geindex conjugate() (complex number method)

Notes:


@enumerate 

@item 
Also referred to as integer division.  The resultant value is a whole
integer, though the result’s type is not necessarily int.  The result is
always rounded towards minus infinity: @code{1//2} is @code{0}, @code{(-1)//2} is
@code{-1}, @code{1//(-2)} is @code{-1}, and @code{(-1)//(-2)} is @code{0}.

@item 
Not for complex numbers.  Instead convert to floats using @ref{1582,,abs()} if
appropriate.

@item 
@geindex module; math
@geindex floor() (in module math)
@geindex ceil() (in module math)
@geindex trunc() (in module math)
@geindex numeric; conversions
@geindex C; language

Conversion from floating point to integer may round or truncate
as in C; see functions @ref{f28,,math.floor()} and @ref{f29,,math.ceil()} for
well-defined conversions.

@item 
float also accepts the strings “nan” and “inf” with an optional prefix “+”
or “-” for Not a Number (NaN) and positive or negative infinity.

@item 
Python defines @code{pow(0, 0)} and @code{0 ** 0} to be @code{1}, as is common for
programming languages.

@item 
The numeric literals accepted include the digits @code{0} to @code{9} or any
Unicode equivalent (code points with the @code{Nd} property).

See @indicateurl{https://www.unicode.org/Public/14.0.0/ucd/extracted/DerivedNumericType.txt}
for a complete list of code points with the @code{Nd} property.
@end enumerate

All @ref{178c,,numbers.Real} types (@ref{1b1,,int} and @ref{256,,float}) also include
the following operations:


@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Operation

@tab

Result

@item

@ref{f2a,,math.trunc(x)}

@tab

@emph{x} truncated to @ref{178b,,Integral}

@item

@ref{e75,,round(x[@comma{} n])}

@tab

@emph{x} rounded to @emph{n} digits,
rounding half to even. If @emph{n} is
omitted, it defaults to 0.

@item

@ref{f28,,math.floor(x)}

@tab

the greatest @ref{178b,,Integral}
<= @emph{x}

@item

@ref{f29,,math.ceil(x)}

@tab

the least @ref{178b,,Integral} >= @emph{x}

@end multitable


For additional numeric operations see the @ref{ad,,math} and @ref{17,,cmath}
modules.

@c XXXJH exceptions: overflow (when? what operations?) zerodivision

@menu
* Bitwise Operations on Integer Types:: 
* Additional Methods on Integer Types:: 
* Additional Methods on Float:: 
* Hashing of numeric types:: 

@end menu

@node Bitwise Operations on Integer Types,Additional Methods on Integer Types,,Numeric Types — int float complex
@anchor{library/stdtypes bitstring-ops}@anchor{1a78}@anchor{library/stdtypes bitwise-operations-on-integer-types}@anchor{1a79}
@subsubsection Bitwise Operations on Integer Types


@geindex operations on; integer; types
@geindex bitwise; operations
@geindex shifting; operations
@geindex masking; operations
@geindex operator; | (vertical bar)
@geindex operator; ^ (caret)
@geindex operator; & (ampersand)
@geindex operator; <<
@geindex operator; >>
@geindex operator; ~ (tilde)

Bitwise operations only make sense for integers. The result of bitwise
operations is calculated as though carried out in two’s complement with an
infinite number of sign bits.

The priorities of the binary bitwise operations are all lower than the numeric
operations and higher than the comparisons; the unary operation @code{~} has the
same priority as the other unary numeric operations (@code{+} and @code{-}).

This table lists the bitwise operations sorted in ascending priority:


@multitable {xxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxx} 
@headitem

Operation

@tab

Result

@tab

Notes

@item

@code{x | y}

@tab

bitwise @emph{or} of @emph{x} and
@emph{y}

@tab

(4)

@item

@code{x ^ y}

@tab

bitwise @emph{exclusive or} of
@emph{x} and @emph{y}

@tab

(4)

@item

@code{x & y}

@tab

bitwise @emph{and} of @emph{x} and
@emph{y}

@tab

(4)

@item

@code{x << n}

@tab

@emph{x} shifted left by @emph{n} bits

@tab

(1)(2)

@item

@code{x >> n}

@tab

@emph{x} shifted right by @emph{n} bits

@tab

(1)(3)

@item

@code{~x}

@tab

the bits of @emph{x} inverted

@tab

@end multitable


Notes:


@enumerate 

@item 
Negative shift counts are illegal and cause a @ref{17c,,ValueError} to be raised.

@item 
A left shift by @emph{n} bits is equivalent to multiplication by @code{pow(2, n)}.

@item 
A right shift by @emph{n} bits is equivalent to floor division by @code{pow(2, n)}.

@item 
Performing these calculations with at least one extra sign extension bit in
a finite two’s complement representation (a working bit-width of
@code{1 + max(x.bit_length(), y.bit_length())} or more) is sufficient to get the
same result as if there were an infinite number of sign bits.
@end enumerate

@node Additional Methods on Integer Types,Additional Methods on Float,Bitwise Operations on Integer Types,Numeric Types — int float complex
@anchor{library/stdtypes additional-methods-on-integer-types}@anchor{1a7a}
@subsubsection Additional Methods on Integer Types


The int type implements the @ref{178b,,numbers.Integral} @ref{d5c,,abstract base class}. In addition, it provides a few more methods:

@geindex bit_length() (int method)
@anchor{library/stdtypes int bit_length}@anchor{1a7b}
@deffn {Method} int.bit_length ()

Return the number of bits necessary to represent an integer in binary,
excluding the sign and leading zeros:

@example
>>> n = -37
>>> bin(n)
'-0b100101'
>>> n.bit_length()
6
@end example

More precisely, if @code{x} is nonzero, then @code{x.bit_length()} is the
unique positive integer @code{k} such that @code{2**(k-1) <= abs(x) < 2**k}.
Equivalently, when @code{abs(x)} is small enough to have a correctly
rounded logarithm, then @code{k = 1 + int(log(abs(x), 2))}.
If @code{x} is zero, then @code{x.bit_length()} returns @code{0}.

Equivalent to:

@example
def bit_length(self):
    s = bin(self)       # binary representation:  bin(-37) --> '-0b100101'
    s = s.lstrip('-0b') # remove leading zeros and minus sign
    return len(s)       # len('100101') --> 6
@end example


New in version 3.1.

@end deffn

@geindex bit_count() (int method)
@anchor{library/stdtypes int bit_count}@anchor{242}
@deffn {Method} int.bit_count ()

Return the number of ones in the binary representation of the absolute
value of the integer. This is also known as the population count.
Example:

@example
>>> n = 19
>>> bin(n)
'0b10011'
>>> n.bit_count()
3
>>> (-n).bit_count()
3
@end example

Equivalent to:

@example
def bit_count(self):
    return bin(self).count("1")
@end example


New in version 3.10.

@end deffn

@geindex to_bytes() (int method)
@anchor{library/stdtypes int to_bytes}@anchor{10f4}
@deffn {Method} int.to_bytes (length=1, byteorder='big', *, signed=False)

Return an array of bytes representing an integer.

@example
>>> (1024).to_bytes(2, byteorder='big')
b'\x04\x00'
>>> (1024).to_bytes(10, byteorder='big')
b'\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00'
>>> (-1024).to_bytes(10, byteorder='big', signed=True)
b'\xff\xff\xff\xff\xff\xff\xff\xff\xfc\x00'
>>> x = 1000
>>> x.to_bytes((x.bit_length() + 7) // 8, byteorder='little')
b'\xe8\x03'
@end example

The integer is represented using @emph{length} bytes, and defaults to 1.  An
@ref{347,,OverflowError} is raised if the integer is not representable with
the given number of bytes.

The @emph{byteorder} argument determines the byte order used to represent the
integer, and defaults to @code{"big"}.  If @emph{byteorder} is
@code{"big"}, the most significant byte is at the beginning of the byte
array.  If @emph{byteorder} is @code{"little"}, the most significant byte is at
the end of the byte array.

The @emph{signed} argument determines whether two’s complement is used to
represent the integer.  If @emph{signed} is @code{False} and a negative integer is
given, an @ref{347,,OverflowError} is raised. The default value for @emph{signed}
is @code{False}.

The default values can be used to conveniently turn an integer into a
single byte object.  However, when using the default arguments, don’t try
to convert a value greater than 255 or you’ll get an @ref{347,,OverflowError}:

@example
>>> (65).to_bytes()
b'A'
@end example

Equivalent to:

@example
def to_bytes(n, length=1, byteorder='big', signed=False):
    if byteorder == 'little':
        order = range(length)
    elif byteorder == 'big':
        order = reversed(range(length))
    else:
        raise ValueError("byteorder must be either 'little' or 'big'")

    return bytes((n >> i*8) & 0xff for i in order)
@end example


New in version 3.2.



Changed in version 3.11: Added default argument values for @code{length} and @code{byteorder}.

@end deffn

@geindex from_bytes() (int class method)
@anchor{library/stdtypes int from_bytes}@anchor{10a2}
@deffn {Method} classmethod int.from_bytes (bytes, byteorder='big', *, signed=False)

Return the integer represented by the given array of bytes.

@example
>>> int.from_bytes(b'\x00\x10', byteorder='big')
16
>>> int.from_bytes(b'\x00\x10', byteorder='little')
4096
>>> int.from_bytes(b'\xfc\x00', byteorder='big', signed=True)
-1024
>>> int.from_bytes(b'\xfc\x00', byteorder='big', signed=False)
64512
>>> int.from_bytes([255, 0, 0], byteorder='big')
16711680
@end example

The argument @emph{bytes} must either be a @ref{87e,,bytes-like object} or an
iterable producing bytes.

The @emph{byteorder} argument determines the byte order used to represent the
integer, and defaults to @code{"big"}.  If @emph{byteorder} is
@code{"big"}, the most significant byte is at the beginning of the byte
array.  If @emph{byteorder} is @code{"little"}, the most significant byte is at
the end of the byte array.  To request the native byte order of the host
system, use @ref{1a7c,,sys.byteorder} as the byte order value.

The @emph{signed} argument indicates whether two’s complement is used to
represent the integer.

Equivalent to:

@example
def from_bytes(bytes, byteorder='big', signed=False):
    if byteorder == 'little':
        little_ordered = list(bytes)
    elif byteorder == 'big':
        little_ordered = list(reversed(bytes))
    else:
        raise ValueError("byteorder must be either 'little' or 'big'")

    n = sum(b << i*8 for i, b in enumerate(little_ordered))
    if signed and little_ordered and (little_ordered[-1] & 0x80):
        n -= 1 << 8*len(little_ordered)

    return n
@end example


New in version 3.2.



Changed in version 3.11: Added default argument value for @code{byteorder}.

@end deffn

@geindex as_integer_ratio() (int method)
@anchor{library/stdtypes int as_integer_ratio}@anchor{4c2}
@deffn {Method} int.as_integer_ratio ()

Return a pair of integers whose ratio is exactly equal to the original
integer and with a positive denominator. The integer ratio of integers
(whole numbers) is always the integer as the numerator and @code{1} as the
denominator.


New in version 3.8.

@end deffn

@node Additional Methods on Float,Hashing of numeric types,Additional Methods on Integer Types,Numeric Types — int float complex
@anchor{library/stdtypes additional-methods-on-float}@anchor{1a7d}
@subsubsection Additional Methods on Float


The float type implements the @ref{178c,,numbers.Real} @ref{d5c,,abstract base class}. float also has the following additional methods.

@geindex as_integer_ratio() (float method)
@anchor{library/stdtypes float as_integer_ratio}@anchor{15e6}
@deffn {Method} float.as_integer_ratio ()

Return a pair of integers whose ratio is exactly equal to the
original float and with a positive denominator.  Raises
@ref{347,,OverflowError} on infinities and a @ref{17c,,ValueError} on
NaNs.
@end deffn

@geindex is_integer() (float method)
@anchor{library/stdtypes float is_integer}@anchor{1a7e}
@deffn {Method} float.is_integer ()

Return @code{True} if the float instance is finite with integral
value, and @code{False} otherwise:

@example
>>> (-2.0).is_integer()
True
>>> (3.2).is_integer()
False
@end example
@end deffn

Two methods support conversion to
and from hexadecimal strings.  Since Python’s floats are stored
internally as binary numbers, converting a float to or from a
@emph{decimal} string usually involves a small rounding error.  In
contrast, hexadecimal strings allow exact representation and
specification of floating-point numbers.  This can be useful when
debugging, and in numerical work.

@geindex hex() (float method)
@anchor{library/stdtypes float hex}@anchor{15e7}
@deffn {Method} float.hex ()

Return a representation of a floating-point number as a hexadecimal
string.  For finite floating-point numbers, this representation
will always include a leading @code{0x} and a trailing @code{p} and
exponent.
@end deffn

@geindex fromhex() (float class method)
@anchor{library/stdtypes float fromhex}@anchor{1a7f}
@deffn {Method} classmethod float.fromhex (s)

Class method to return the float represented by a hexadecimal
string @emph{s}.  The string @emph{s} may have leading and trailing
whitespace.
@end deffn

Note that @ref{15e7,,float.hex()} is an instance method, while
@ref{1a7f,,float.fromhex()} is a class method.

A hexadecimal string takes the form:

@example
[sign] ['0x'] integer ['.' fraction] ['p' exponent]
@end example

where the optional @code{sign} may by either @code{+} or @code{-}, @code{integer}
and @code{fraction} are strings of hexadecimal digits, and @code{exponent}
is a decimal integer with an optional leading sign.  Case is not
significant, and there must be at least one hexadecimal digit in
either the integer or the fraction.  This syntax is similar to the
syntax specified in section 6.4.4.2 of the C99 standard, and also to
the syntax used in Java 1.5 onwards.  In particular, the output of
@ref{15e7,,float.hex()} is usable as a hexadecimal floating-point literal in
C or Java code, and hexadecimal strings produced by C’s @code{%a} format
character or Java’s @code{Double.toHexString} are accepted by
@ref{1a7f,,float.fromhex()}.

Note that the exponent is written in decimal rather than hexadecimal,
and that it gives the power of 2 by which to multiply the coefficient.
For example, the hexadecimal string @code{0x3.a7p10} represents the
floating-point number @code{(3 + 10./16 + 7./16**2) * 2.0**10}, or
@code{3740.0}:

@example
>>> float.fromhex('0x3.a7p10')
3740.0
@end example

Applying the reverse conversion to @code{3740.0} gives a different
hexadecimal string representing the same number:

@example
>>> float.hex(3740.0)
'0x1.d380000000000p+11'
@end example

@node Hashing of numeric types,,Additional Methods on Float,Numeric Types — int float complex
@anchor{library/stdtypes hashing-of-numeric-types}@anchor{1a80}@anchor{library/stdtypes numeric-hash}@anchor{1a81}
@subsubsection Hashing of numeric types


For numbers @code{x} and @code{y}, possibly of different types, it’s a requirement
that @code{hash(x) == hash(y)} whenever @code{x == y} (see the @ref{619,,__hash__()}
method documentation for more details).  For ease of implementation and
efficiency across a variety of numeric types (including @ref{1b1,,int},
@ref{256,,float}, @ref{248,,decimal.Decimal} and @ref{172,,fractions.Fraction})
Python’s hash for numeric types is based on a single mathematical function
that’s defined for any rational number, and hence applies to all instances of
@ref{1b1,,int} and @ref{172,,fractions.Fraction}, and all finite instances of
@ref{256,,float} and @ref{248,,decimal.Decimal}.  Essentially, this function is
given by reduction modulo @code{P} for a fixed prime @code{P}.  The value of @code{P} is
made available to Python as the @code{modulus} attribute of
@ref{b85,,sys.hash_info}.

@strong{CPython implementation detail:} Currently, the prime used is @code{P = 2**31 - 1} on machines with 32-bit C
longs and @code{P = 2**61 - 1} on machines with 64-bit C longs.

Here are the rules in detail:


@itemize -

@item 
If @code{x = m / n} is a nonnegative rational number and @code{n} is not divisible
by @code{P}, define @code{hash(x)} as @code{m * invmod(n, P) % P}, where @code{invmod(n,
P)} gives the inverse of @code{n} modulo @code{P}.

@item 
If @code{x = m / n} is a nonnegative rational number and @code{n} is
divisible by @code{P} (but @code{m} is not) then @code{n} has no inverse
modulo @code{P} and the rule above doesn’t apply; in this case define
@code{hash(x)} to be the constant value @code{sys.hash_info.inf}.

@item 
If @code{x = m / n} is a negative rational number define @code{hash(x)}
as @code{-hash(-x)}.  If the resulting hash is @code{-1}, replace it with
@code{-2}.

@item 
The particular values @code{sys.hash_info.inf} and @code{-sys.hash_info.inf}
are used as hash values for positive
infinity or negative infinity (respectively).

@item 
For a @ref{33b,,complex} number @code{z}, the hash values of the real
and imaginary parts are combined by computing @code{hash(z.real) +
sys.hash_info.imag * hash(z.imag)}, reduced modulo
@code{2**sys.hash_info.width} so that it lies in
@code{range(-2**(sys.hash_info.width - 1), 2**(sys.hash_info.width -
1))}.  Again, if the result is @code{-1}, it’s replaced with @code{-2}.
@end itemize

To clarify the above rules, here’s some example Python code,
equivalent to the built-in hash, for computing the hash of a rational
number, @ref{256,,float}, or @ref{33b,,complex}:

@example
import sys, math

def hash_fraction(m, n):
    """Compute the hash of a rational number m / n.

    Assumes m and n are integers, with n positive.
    Equivalent to hash(fractions.Fraction(m, n)).

    """
    P = sys.hash_info.modulus
    # Remove common factors of P.  (Unnecessary if m and n already coprime.)
    while m % P == n % P == 0:
        m, n = m // P, n // P

    if n % P == 0:
        hash_value = sys.hash_info.inf
    else:
        # Fermat's Little Theorem: pow(n, P-1, P) is 1, so
        # pow(n, P-2, P) gives the inverse of n modulo P.
        hash_value = (abs(m) % P) * pow(n, P - 2, P) % P
    if m < 0:
        hash_value = -hash_value
    if hash_value == -1:
        hash_value = -2
    return hash_value

def hash_float(x):
    """Compute the hash of a float x."""

    if math.isnan(x):
        return object.__hash__(x)
    elif math.isinf(x):
        return sys.hash_info.inf if x > 0 else -sys.hash_info.inf
    else:
        return hash_fraction(*x.as_integer_ratio())

def hash_complex(z):
    """Compute the hash of a complex number z."""

    hash_value = hash_float(z.real) + sys.hash_info.imag * hash_float(z.imag)
    # do a signed reduction modulo 2**sys.hash_info.width
    M = 2**(sys.hash_info.width - 1)
    hash_value = (hash_value & (M - 1)) - (hash_value & M)
    if hash_value == -1:
        hash_value = -2
    return hash_value
@end example

@node Iterator Types,Sequence Types — list tuple range,Numeric Types — int float complex,Built-in Types
@anchor{library/stdtypes iterator-types}@anchor{1a82}@anchor{library/stdtypes typeiter}@anchor{1a51}
@subsection Iterator Types


@geindex iterator protocol
@geindex protocol; iterator
@geindex sequence; iteration
@geindex container; iteration over

Python supports a concept of iteration over containers.  This is implemented
using two distinct methods; these are used to allow user-defined classes to
support iteration.  Sequences, described below in more detail, always support
the iteration methods.

One method needs to be defined for container objects to provide @ref{dcc,,iterable}
support:

@c XXX duplicated in reference/datamodel!

@geindex __iter__() (container method)
@anchor{library/stdtypes container __iter__}@anchor{1a83}
@deffn {Method} container.__iter__ ()

Return an @ref{11c3,,iterator} object.  The object is required to support the
iterator protocol described below.  If a container supports different types
of iteration, additional methods can be provided to specifically request
iterators for those iteration types.  (An example of an object supporting
multiple forms of iteration would be a tree structure which supports both
breadth-first and depth-first traversal.)  This method corresponds to the
@ref{1028,,tp_iter} slot of the type structure for Python
objects in the Python/C API.
@end deffn

The iterator objects themselves are required to support the following two
methods, which together form the @emph{iterator protocol}:

@geindex __iter__() (iterator method)
@anchor{library/stdtypes iterator __iter__}@anchor{1a84}
@deffn {Method} iterator.__iter__ ()

Return the @ref{11c3,,iterator} object itself.  This is required to allow both
containers and iterators to be used with the @ref{161,,for} and
@ref{2fc,,in} statements.  This method corresponds to the
@ref{1028,,tp_iter} slot of the type structure for Python
objects in the Python/C API.
@end deffn

@geindex __next__() (iterator method)
@anchor{library/stdtypes iterator __next__}@anchor{e6d}
@deffn {Method} iterator.__next__ ()

Return the next item from the @ref{11c3,,iterator}.  If there are no further
items, raise the @ref{736,,StopIteration} exception.  This method corresponds to
the @ref{1029,,tp_iternext} slot of the type structure for
Python objects in the Python/C API.
@end deffn

Python defines several iterator objects to support iteration over general and
specific sequence types, dictionaries, and other more specialized forms.  The
specific types are not important beyond their implementation of the iterator
protocol.

Once an iterator’s @ref{e6d,,__next__()} method raises
@ref{736,,StopIteration}, it must continue to do so on subsequent calls.
Implementations that do not obey this property are deemed broken.

@menu
* Generator Types:: 

@end menu

@node Generator Types,,,Iterator Types
@anchor{library/stdtypes generator-types}@anchor{1a85}@anchor{library/stdtypes id3}@anchor{1a86}
@subsubsection Generator Types


Python’s @ref{bf7,,generator}s provide a convenient way to implement the iterator
protocol.  If a container object’s @code{__iter__()} method is implemented as a
generator, it will automatically return an iterator object (technically, a
generator object) supplying the @code{__iter__()} and @ref{159d,,__next__()}
methods.
More information about generators can be found in @ref{18a7,,the documentation for the yield expression}.

@node Sequence Types — list tuple range,Text Sequence Type — str,Iterator Types,Built-in Types
@anchor{library/stdtypes sequence-types-list-tuple-range}@anchor{1a87}@anchor{library/stdtypes typesseq}@anchor{1530}
@subsection Sequence Types — @code{list}, @code{tuple}, @code{range}


There are three basic sequence types: lists, tuples, and range objects.
Additional sequence types tailored for processing of
@ref{1a3f,,binary data} and @ref{14e2,,text strings} are
described in dedicated sections.

@menu
* Common Sequence Operations:: 
* Immutable Sequence Types:: 
* Mutable Sequence Types:: 
* Lists: Lists<2>. 
* Tuples:: 
* Ranges:: 

@end menu

@node Common Sequence Operations,Immutable Sequence Types,,Sequence Types — list tuple range
@anchor{library/stdtypes common-sequence-operations}@anchor{1a88}@anchor{library/stdtypes typesseq-common}@anchor{1a89}
@subsubsection Common Sequence Operations


@geindex object; sequence

The operations in the following table are supported by most sequence types,
both mutable and immutable. The @ref{110c,,collections.abc.Sequence} ABC is
provided to make it easier to correctly implement these operations on
custom sequence types.

This table lists the sequence operations sorted in ascending priority.  In the
table, @emph{s} and @emph{t} are sequences of the same type, @emph{n}, @emph{i}, @emph{j} and @emph{k} are
integers and @emph{x} is an arbitrary object that meets any type and value
restrictions imposed by @emph{s}.

The @code{in} and @code{not in} operations have the same priorities as the
comparison operations. The @code{+} (concatenation) and @code{*} (repetition)
operations have the same priority as the corresponding numeric operations. @footnote{@w{(3)} 
They must have since the parser can’t tell the type of the operands.
}

@geindex operations on; sequence; types
@geindex built-in function; len
@geindex built-in function; min
@geindex built-in function; max
@geindex concatenation; operation
@geindex repetition; operation
@geindex subscript; operation
@geindex slice; operation
@geindex operator; in
@geindex operator; not in
@geindex count() (sequence method)
@geindex index() (sequence method)


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxx} 
@headitem

Operation

@tab

Result

@tab

Notes

@item

@code{x in s}

@tab

@code{True} if an item of @emph{s} is
equal to @emph{x}, else @code{False}

@tab

(1)

@item

@code{x not in s}

@tab

@code{False} if an item of @emph{s} is
equal to @emph{x}, else @code{True}

@tab

(1)

@item

@code{s + t}

@tab

the concatenation of @emph{s} and
@emph{t}

@tab

(6)(7)

@item

@code{s * n} or
@code{n * s}

@tab

equivalent to adding @emph{s} to
itself @emph{n} times

@tab

(2)(7)

@item

@code{s[i]}

@tab

@emph{i}th item of @emph{s}, origin 0

@tab

(3)

@item

@code{s[i:j]}

@tab

slice of @emph{s} from @emph{i} to @emph{j}

@tab

(3)(4)

@item

@code{s[i:j:k]}

@tab

slice of @emph{s} from @emph{i} to @emph{j}
with step @emph{k}

@tab

(3)(5)

@item

@code{len(s)}

@tab

length of @emph{s}

@tab

@item

@code{min(s)}

@tab

smallest item of @emph{s}

@tab

@item

@code{max(s)}

@tab

largest item of @emph{s}

@tab

@item

@code{s.index(x[, i[, j]])}

@tab

index of the first occurrence
of @emph{x} in @emph{s} (at or after
index @emph{i} and before index @emph{j})

@tab

(8)

@item

@code{s.count(x)}

@tab

total number of occurrences of
@emph{x} in @emph{s}

@tab

@end multitable


Sequences of the same type also support comparisons.  In particular, tuples
and lists are compared lexicographically by comparing corresponding elements.
This means that to compare equal, every element must compare equal and the
two sequences must be of the same type and have the same length.  (For full
details see @ref{1900,,Comparisons} in the language reference.)

@geindex loop; over mutable sequence
@geindex mutable sequence; loop over

Forward and reversed iterators over mutable sequences access values using an
index.  That index will continue to march forward (or backward) even if the
underlying sequence is mutated.  The iterator terminates only when an
@ref{1066,,IndexError} or a @ref{736,,StopIteration} is encountered (or when the index
drops below zero).

Notes:


@enumerate 

@item 
While the @code{in} and @code{not in} operations are used only for simple
containment testing in the general case, some specialised sequences
(such as @ref{2ef,,str}, @ref{2f0,,bytes} and @ref{2f1,,bytearray}) also use
them for subsequence testing:

@example
>>> "gg" in "eggs"
True
@end example

@item 
Values of @emph{n} less than @code{0} are treated as @code{0} (which yields an empty
sequence of the same type as @emph{s}).  Note that items in the sequence @emph{s}
are not copied; they are referenced multiple times.  This often haunts
new Python programmers; consider:

@example
>>> lists = [[]] * 3
>>> lists
[[], [], []]
>>> lists[0].append(3)
>>> lists
[[3], [3], [3]]
@end example

What has happened is that @code{[[]]} is a one-element list containing an empty
list, so all three elements of @code{[[]] * 3} are references to this single empty
list.  Modifying any of the elements of @code{lists} modifies this single list.
You can create a list of different lists this way:

@example
>>> lists = [[] for i in range(3)]
>>> lists[0].append(3)
>>> lists[1].append(5)
>>> lists[2].append(7)
>>> lists
[[3], [5], [7]]
@end example

Further explanation is available in the FAQ entry
@ref{1a8a,,How do I create a multidimensional list?}.

@item 
If @emph{i} or @emph{j} is negative, the index is relative to the end of sequence @emph{s}:
@code{len(s) + i} or @code{len(s) + j} is substituted.  But note that @code{-0} is
still @code{0}.

@item 
The slice of @emph{s} from @emph{i} to @emph{j} is defined as the sequence of items with index
@emph{k} such that @code{i <= k < j}.  If @emph{i} or @emph{j} is greater than @code{len(s)}, use
@code{len(s)}.  If @emph{i} is omitted or @code{None}, use @code{0}.  If @emph{j} is omitted or
@code{None}, use @code{len(s)}.  If @emph{i} is greater than or equal to @emph{j}, the slice is
empty.

@item 
The slice of @emph{s} from @emph{i} to @emph{j} with step @emph{k} is defined as the sequence of
items with index  @code{x = i + n*k} such that @code{0 <= n < (j-i)/k}.  In other words,
the indices are @code{i}, @code{i+k}, @code{i+2*k}, @code{i+3*k} and so on, stopping when
@emph{j} is reached (but never including @emph{j}).  When @emph{k} is positive,
@emph{i} and @emph{j} are reduced to @code{len(s)} if they are greater.
When @emph{k} is negative, @emph{i} and @emph{j} are reduced to @code{len(s) - 1} if
they are greater.  If @emph{i} or @emph{j} are omitted or @code{None}, they become
“end” values (which end depends on the sign of @emph{k}).  Note, @emph{k} cannot be zero.
If @emph{k} is @code{None}, it is treated like @code{1}.

@item 
Concatenating immutable sequences always results in a new object.  This
means that building up a sequence by repeated concatenation will have a
quadratic runtime cost in the total sequence length.  To get a linear
runtime cost, you must switch to one of the alternatives below:


@itemize *

@item 
if concatenating @ref{2ef,,str} objects, you can build a list and use
@ref{1a8b,,str.join()} at the end or else write to an @ref{aa1,,io.StringIO}
instance and retrieve its value when complete

@item 
if concatenating @ref{2f0,,bytes} objects, you can similarly use
@ref{1a8c,,bytes.join()} or @ref{a15,,io.BytesIO}, or you can do in-place
concatenation with a @ref{2f1,,bytearray} object.  @ref{2f1,,bytearray}
objects are mutable and have an efficient overallocation mechanism

@item 
if concatenating @ref{3e5,,tuple} objects, extend a @ref{43a,,list} instead

@item 
for other types, investigate the relevant class documentation
@end itemize

@item 
Some sequence types (such as @ref{438,,range}) only support item sequences
that follow specific patterns, and hence don’t support sequence
concatenation or repetition.

@item 
@code{index} raises @ref{17c,,ValueError} when @emph{x} is not found in @emph{s}.
Not all implementations support passing the additional arguments @emph{i} and @emph{j}.
These arguments allow efficient searching of subsections of the sequence. Passing
the extra arguments is roughly equivalent to using @code{s[i:j].index(x)}, only
without copying any data and with the returned index being relative to
the start of the sequence rather than the start of the slice.
@end enumerate

@node Immutable Sequence Types,Mutable Sequence Types,Common Sequence Operations,Sequence Types — list tuple range
@anchor{library/stdtypes immutable-sequence-types}@anchor{1a8d}@anchor{library/stdtypes typesseq-immutable}@anchor{1a8e}
@subsubsection Immutable Sequence Types


@geindex immutable; sequence; types
@geindex object; tuple
@geindex built-in function; hash

The only operation that immutable sequence types generally implement that is
not also implemented by mutable sequence types is support for the @ref{c14,,hash()}
built-in.

This support allows immutable sequences, such as @ref{3e5,,tuple} instances, to
be used as @ref{3b0,,dict} keys and stored in @ref{370,,set} and @ref{439,,frozenset}
instances.

Attempting to hash an immutable sequence that contains unhashable values will
result in @ref{162,,TypeError}.

@node Mutable Sequence Types,Lists<2>,Immutable Sequence Types,Sequence Types — list tuple range
@anchor{library/stdtypes mutable-sequence-types}@anchor{1a8f}@anchor{library/stdtypes typesseq-mutable}@anchor{1a3c}
@subsubsection Mutable Sequence Types


@geindex mutable; sequence; types
@geindex object; list
@geindex object; bytearray

The operations in the following table are defined on mutable sequence types.
The @ref{933,,collections.abc.MutableSequence} ABC is provided to make it
easier to correctly implement these operations on custom sequence types.

In the table @emph{s} is an instance of a mutable sequence type, @emph{t} is any
iterable object and @emph{x} is an arbitrary object that meets any type
and value restrictions imposed by @emph{s} (for example, @ref{2f1,,bytearray} only
accepts integers that meet the value restriction @code{0 <= x <= 255}).

@geindex operations on; sequence; types
@geindex operations on; list; type
@geindex subscript; assignment
@geindex slice; assignment
@geindex statement; del
@geindex append() (sequence method)
@geindex clear() (sequence method)
@geindex copy() (sequence method)
@geindex extend() (sequence method)
@geindex insert() (sequence method)
@geindex pop() (sequence method)
@geindex remove() (sequence method)
@geindex reverse() (sequence method)


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Operation

@tab

Result

@tab

Notes

@item

@code{s[i] = x}

@tab

item @emph{i} of @emph{s} is replaced by
@emph{x}

@tab

@item

@code{s[i:j] = t}

@tab

slice of @emph{s} from @emph{i} to @emph{j}
is replaced by the contents of
the iterable @emph{t}

@tab

@item

@code{del s[i:j]}

@tab

same as @code{s[i:j] = []}

@tab

@item

@code{s[i:j:k] = t}

@tab

the elements of @code{s[i:j:k]}
are replaced by those of @emph{t}

@tab

(1)

@item

@code{del s[i:j:k]}

@tab

removes the elements of
@code{s[i:j:k]} from the list

@tab

@item

@code{s.append(x)}

@tab

appends @emph{x} to the end of the
sequence (same as
@code{s[len(s):len(s)] = [x]})

@tab

@item

@code{s.clear()}

@tab

removes all items from @emph{s}
(same as @code{del s[:]})

@tab

(5)

@item

@code{s.copy()}

@tab

creates a shallow copy of @emph{s}
(same as @code{s[:]})

@tab

(5)

@item

@code{s.extend(t)} or
@code{s += t}

@tab

extends @emph{s} with the
contents of @emph{t} (for the
most part the same as
@code{s[len(s):len(s)] = t})

@tab

@item

@code{s *= n}

@tab

updates @emph{s} with its contents
repeated @emph{n} times

@tab

(6)

@item

@code{s.insert(i, x)}

@tab

inserts @emph{x} into @emph{s} at the
index given by @emph{i}
(same as @code{s[i:i] = [x]})

@tab

@item

@code{s.pop()} or @code{s.pop(i)}

@tab

retrieves the item at @emph{i} and
also removes it from @emph{s}

@tab

(2)

@item

@code{s.remove(x)}

@tab

remove the first item from @emph{s}
where @code{s[i]} is equal to @emph{x}

@tab

(3)

@item

@code{s.reverse()}

@tab

reverses the items of @emph{s} in
place

@tab

(4)

@end multitable


Notes:


@enumerate 

@item 
@emph{t} must have the same length as the slice it is replacing.

@item 
The optional argument @emph{i} defaults to @code{-1}, so that by default the last
item is removed and returned.

@item 
@code{remove()} raises @ref{17c,,ValueError} when @emph{x} is not found in @emph{s}.

@item 
The @code{reverse()} method modifies the sequence in place for economy of
space when reversing a large sequence.  To remind users that it operates by
side effect, it does not return the reversed sequence.

@item 
@code{clear()} and @code{copy()} are included for consistency with the
interfaces of mutable containers that don’t support slicing operations
(such as @ref{3b0,,dict} and @ref{370,,set}). @code{copy()} is not part of the
@ref{933,,collections.abc.MutableSequence} ABC, but most concrete
mutable sequence classes provide it.


New in version 3.3: @code{clear()} and @code{copy()} methods.


@item 
The value @emph{n} is an integer, or an object implementing
@ref{24a,,__index__()}.  Zero and negative values of @emph{n} clear
the sequence.  Items in the sequence are not copied; they are referenced
multiple times, as explained for @code{s * n} under @ref{1a89,,Common Sequence Operations}.
@end enumerate

@node Lists<2>,Tuples,Mutable Sequence Types,Sequence Types — list tuple range
@anchor{library/stdtypes lists}@anchor{1a90}@anchor{library/stdtypes typesseq-list}@anchor{1a52}
@subsubsection Lists


@geindex object; list

Lists are mutable sequences, typically used to store collections of
homogeneous items (where the precise degree of similarity will vary by
application).

@geindex list (built-in class)
@anchor{library/stdtypes list}@anchor{43a}
@deffn {Class} list ([iterable])

Lists may be constructed in several ways:


@itemize *

@item 
Using a pair of square brackets to denote the empty list: @code{[]}

@item 
Using square brackets, separating items with commas: @code{[a]}, @code{[a, b, c]}

@item 
Using a list comprehension: @code{[x for x in iterable]}

@item 
Using the type constructor: @code{list()} or @code{list(iterable)}
@end itemize

The constructor builds a list whose items are the same and in the same
order as @emph{iterable}’s items.  @emph{iterable} may be either a sequence, a
container that supports iteration, or an iterator object.  If @emph{iterable}
is already a list, a copy is made and returned, similar to @code{iterable[:]}.
For example, @code{list('abc')} returns @code{['a', 'b', 'c']} and
@code{list( (1, 2, 3) )} returns @code{[1, 2, 3]}.
If no argument is given, the constructor creates a new empty list, @code{[]}.

Many other operations also produce lists, including the @ref{704,,sorted()}
built-in.

Lists implement all of the @ref{1a89,,common} and
@ref{1a3c,,mutable} sequence operations. Lists also provide the
following additional method:

@geindex sort() (list method)
@anchor{library/stdtypes list sort}@anchor{705}
@deffn {Method} sort (*, key=None, reverse=False)

This method sorts the list in place, using only @code{<} comparisons
between items. Exceptions are not suppressed - if any comparison operations
fail, the entire sort operation will fail (and the list will likely be left
in a partially modified state).

@ref{705,,sort()} accepts two arguments that can only be passed by keyword
(@ref{5a2,,keyword-only arguments}):

@emph{key} specifies a function of one argument that is used to extract a
comparison key from each list element (for example, @code{key=str.lower}).
The key corresponding to each item in the list is calculated once and
then used for the entire sorting process. The default value of @code{None}
means that list items are sorted directly without calculating a separate
key value.

The @ref{d7c,,functools.cmp_to_key()} utility is available to convert a 2.x
style @emph{cmp} function to a @emph{key} function.

@emph{reverse} is a boolean value.  If set to @code{True}, then the list elements
are sorted as if each comparison were reversed.

This method modifies the sequence in place for economy of space when
sorting a large sequence.  To remind users that it operates by side
effect, it does not return the sorted sequence (use @ref{704,,sorted()} to
explicitly request a new sorted list instance).

The @ref{705,,sort()} method is guaranteed to be stable.  A sort is stable if it
guarantees not to change the relative order of elements that compare equal
— this is helpful for sorting in multiple passes (for example, sort by
department, then by salary grade).

For sorting examples and a brief sorting tutorial, see @ref{1a5e,,Sorting HOW TO}.

@strong{CPython implementation detail:} While a list is being sorted, the effect of attempting to mutate, or even
inspect, the list is undefined.  The C implementation of Python makes the
list appear empty for the duration, and raises @ref{17c,,ValueError} if it can
detect that the list has been mutated during a sort.
@end deffn
@end deffn

@node Tuples,Ranges,Lists<2>,Sequence Types — list tuple range
@anchor{library/stdtypes tuples}@anchor{1a91}@anchor{library/stdtypes typesseq-tuple}@anchor{19a3}
@subsubsection Tuples


@geindex object; tuple

Tuples are immutable sequences, typically used to store collections of
heterogeneous data (such as the 2-tuples produced by the @ref{fdc,,enumerate()}
built-in). Tuples are also used for cases where an immutable sequence of
homogeneous data is needed (such as allowing storage in a @ref{370,,set} or
@ref{3b0,,dict} instance).

@geindex tuple (built-in class)
@anchor{library/stdtypes tuple}@anchor{3e5}
@deffn {Class} tuple ([iterable])

Tuples may be constructed in a number of ways:


@itemize *

@item 
Using a pair of parentheses to denote the empty tuple: @code{()}

@item 
Using a trailing comma for a singleton tuple: @code{a,} or @code{(a,)}

@item 
Separating items with commas: @code{a, b, c} or @code{(a, b, c)}

@item 
Using the @ref{3e5,,tuple()} built-in: @code{tuple()} or @code{tuple(iterable)}
@end itemize

The constructor builds a tuple whose items are the same and in the same
order as @emph{iterable}’s items.  @emph{iterable} may be either a sequence, a
container that supports iteration, or an iterator object.  If @emph{iterable}
is already a tuple, it is returned unchanged. For example,
@code{tuple('abc')} returns @code{('a', 'b', 'c')} and
@code{tuple( [1, 2, 3] )} returns @code{(1, 2, 3)}.
If no argument is given, the constructor creates a new empty tuple, @code{()}.

Note that it is actually the comma which makes a tuple, not the parentheses.
The parentheses are optional, except in the empty tuple case, or
when they are needed to avoid syntactic ambiguity. For example,
@code{f(a, b, c)} is a function call with three arguments, while
@code{f((a, b, c))} is a function call with a 3-tuple as the sole argument.

Tuples implement all of the @ref{1a89,,common} sequence
operations.
@end deffn

For heterogeneous collections of data where access by name is clearer than
access by index, @ref{3e4,,collections.namedtuple()} may be a more appropriate
choice than a simple tuple object.

@node Ranges,,Tuples,Sequence Types — list tuple range
@anchor{library/stdtypes ranges}@anchor{1a92}@anchor{library/stdtypes typesseq-range}@anchor{1a5c}
@subsubsection Ranges


@geindex object; range

The @ref{438,,range} type represents an immutable sequence of numbers and is
commonly used for looping a specific number of times in @ref{161,,for}
loops.

@geindex range (built-in class)
@anchor{library/stdtypes range}@anchor{438}
@deffn {Class} range (stop)

@deffnx {Class} range (start, stop[, step])

The arguments to the range constructor must be integers (either built-in
@ref{1b1,,int} or any object that implements the @ref{24a,,__index__()} special
method).  If the @emph{step} argument is omitted, it defaults to @code{1}.
If the @emph{start} argument is omitted, it defaults to @code{0}.
If @emph{step} is zero, @ref{17c,,ValueError} is raised.

For a positive @emph{step}, the contents of a range @code{r} are determined by the
formula @code{r[i] = start + step*i} where @code{i >= 0} and
@code{r[i] < stop}.

For a negative @emph{step}, the contents of the range are still determined by
the formula @code{r[i] = start + step*i}, but the constraints are @code{i >= 0}
and @code{r[i] > stop}.

A range object will be empty if @code{r[0]} does not meet the value
constraint. Ranges do support negative indices, but these are interpreted
as indexing from the end of the sequence determined by the positive
indices.

Ranges containing absolute values larger than @ref{d69,,sys.maxsize} are
permitted but some features (such as @ref{4a6,,len()}) may raise
@ref{347,,OverflowError}.

Range examples:

@example
>>> list(range(10))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> list(range(1, 11))
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> list(range(0, 30, 5))
[0, 5, 10, 15, 20, 25]
>>> list(range(0, 10, 3))
[0, 3, 6, 9]
>>> list(range(0, -10, -1))
[0, -1, -2, -3, -4, -5, -6, -7, -8, -9]
>>> list(range(0))
[]
>>> list(range(1, 0))
[]
@end example

Ranges implement all of the @ref{1a89,,common} sequence operations
except concatenation and repetition (due to the fact that range objects can
only represent sequences that follow a strict pattern and repetition and
concatenation will usually violate that pattern).

@geindex start (range attribute)
@anchor{library/stdtypes range start}@anchor{1a93}
@deffn {Attribute} start

The value of the @emph{start} parameter (or @code{0} if the parameter was
not supplied)
@end deffn

@geindex stop (range attribute)
@anchor{library/stdtypes range stop}@anchor{1a94}
@deffn {Attribute} stop

The value of the @emph{stop} parameter
@end deffn

@geindex step (range attribute)
@anchor{library/stdtypes range step}@anchor{1a95}
@deffn {Attribute} step

The value of the @emph{step} parameter (or @code{1} if the parameter was
not supplied)
@end deffn
@end deffn

The advantage of the @ref{438,,range} type over a regular @ref{43a,,list} or
@ref{3e5,,tuple} is that a @ref{438,,range} object will always take the same
(small) amount of memory, no matter the size of the range it represents (as it
only stores the @code{start}, @code{stop} and @code{step} values, calculating individual
items and subranges as needed).

Range objects implement the @ref{110c,,collections.abc.Sequence} ABC, and provide
features such as containment tests, element index lookup, slicing and
support for negative indices (see @ref{1530,,Sequence Types — list@comma{} tuple@comma{} range}):

@example
>>> r = range(0, 20, 2)
>>> r
range(0, 20, 2)
>>> 11 in r
False
>>> 10 in r
True
>>> r.index(10)
5
>>> r[5]
10
>>> r[:5]
range(0, 10, 2)
>>> r[-1]
18
@end example

Testing range objects for equality with @code{==} and @code{!=} compares
them as sequences.  That is, two range objects are considered equal if
they represent the same sequence of values.  (Note that two range
objects that compare equal might have different @ref{1a93,,start},
@ref{1a94,,stop} and @ref{1a95,,step} attributes, for example
@code{range(0) == range(2, 1, 3)} or @code{range(0, 3, 2) == range(0, 4, 2)}.)


Changed in version 3.2: Implement the Sequence ABC.
Support slicing and negative indices.
Test @ref{1b1,,int} objects for membership in constant time instead of
iterating through all items.



Changed in version 3.3: Define ‘==’ and ‘!=’ to compare range objects based on the
sequence of values they define (instead of comparing based on
object identity).



New in version 3.3: The @ref{1a93,,start}, @ref{1a94,,stop} and @ref{1a95,,step}
attributes.



@subsubheading See also



@itemize *

@item 
The linspace recipe@footnote{http://code.activestate.com/recipes/579000/}
shows how to implement a lazy version of range suitable for floating
point applications.
@end itemize


@geindex string; text sequence type
@geindex str (built-in class); (see also string)
@geindex object; string

@node Text Sequence Type — str,Binary Sequence Types — bytes bytearray memoryview,Sequence Types — list tuple range,Built-in Types
@anchor{library/stdtypes text-sequence-type-str}@anchor{1a96}@anchor{library/stdtypes textseq}@anchor{14e2}
@subsection Text Sequence Type — @code{str}


Textual data in Python is handled with @ref{2ef,,str} objects, or @emph{strings}.
Strings are immutable
@ref{1530,,sequences} of Unicode code points.  String literals are
written in a variety of ways:


@itemize *

@item 
Single quotes: @code{'allows embedded "double" quotes'}

@item 
Double quotes: @code{"allows embedded 'single' quotes"}

@item 
Triple quoted: @code{'@w{'}'Three single quotes'@w{'}'}, @code{"""Three double quotes"""}
@end itemize

Triple quoted strings may span multiple lines - all associated whitespace will
be included in the string literal.

String literals that are part of a single expression and have only whitespace
between them will be implicitly converted to a single string literal. That
is, @code{("spam " "eggs") == "spam eggs"}.

See @ref{1720,,String and Bytes literals} for more about the various forms of string literal,
including supported escape sequences, and the @code{r} (“raw”) prefix that
disables most escape sequence processing.

Strings may also be created from other objects using the @ref{2ef,,str}
constructor.

Since there is no separate “character” type, indexing a string produces
strings of length 1. That is, for a non-empty string @emph{s}, @code{s[0] == s[0:1]}.

@geindex object; io.StringIO

There is also no mutable string type, but @ref{1a8b,,str.join()} or
@ref{aa1,,io.StringIO} can be used to efficiently construct strings from
multiple fragments.


Changed in version 3.3: For backwards compatibility with the Python 2 series, the @code{u} prefix is
once again permitted on string literals. It has no effect on the meaning
of string literals and cannot be combined with the @code{r} prefix.


@geindex string; str (built-in class)

@geindex str (built-in class)
@anchor{library/stdtypes str}@anchor{2ef}
@deffn {Class} str (object='@w{'})

@deffnx {Class} str (object=b'@w{'}, encoding='utf-8', errors='strict')

Return a @ref{14e2,,string} version of @emph{object}.  If @emph{object} is not
provided, returns the empty string.  Otherwise, the behavior of @code{str()}
depends on whether @emph{encoding} or @emph{errors} is given, as follows.

If neither @emph{encoding} nor @emph{errors} is given, @code{str(object)} returns
@ref{178a,,object.__str__()}, which is the “informal” or nicely
printable string representation of @emph{object}.  For string objects, this is
the string itself.  If @emph{object} does not have a @ref{178a,,__str__()}
method, then @ref{2ef,,str()} falls back to returning
@ref{27d,,repr(object)}.

@geindex buffer protocol; str (built-in class)
@geindex bytes; str (built-in class)

If at least one of @emph{encoding} or @emph{errors} is given, @emph{object} should be a
@ref{87e,,bytes-like object} (e.g. @ref{2f0,,bytes} or @ref{2f1,,bytearray}).  In
this case, if @emph{object} is a @ref{2f0,,bytes} (or @ref{2f1,,bytearray}) object,
then @code{str(bytes, encoding, errors)} is equivalent to
@ref{3bb,,bytes.decode(encoding@comma{} errors)}.  Otherwise, the bytes
object underlying the buffer object is obtained before calling
@ref{3bb,,bytes.decode()}.  See @ref{1a3f,,Binary Sequence Types — bytes@comma{} bytearray@comma{} memoryview} and
@ref{1a3e,,Buffer Protocol} for information on buffer objects.

Passing a @ref{2f0,,bytes} object to @ref{2ef,,str()} without the @emph{encoding}
or @emph{errors} arguments falls under the first case of returning the informal
string representation (see also the @ref{6dc,,-b} command-line option to
Python).  For example:

@example
>>> str(b'Zoot!')
"b'Zoot!'"
@end example

For more information on the @code{str} class and its methods, see
@ref{14e2,,Text Sequence Type — str} and the @ref{14e3,,String Methods} section below.  To output
formatted strings, see the @ref{4b0,,Formatted string literals} and @ref{f1a,,Format String Syntax}
sections.  In addition, see the @ref{1a97,,Text Processing Services} section.
@end deffn

@geindex string; methods

@menu
* String Methods: String Methods<2>. 
* printf-style String Formatting:: 

@end menu

@node String Methods<2>,printf-style String Formatting,,Text Sequence Type — str
@anchor{library/stdtypes id5}@anchor{1a98}@anchor{library/stdtypes string-methods}@anchor{14e3}
@subsubsection String Methods


@geindex module; re

Strings implement all of the @ref{1a89,,common} sequence
operations, along with the additional methods described below.

Strings also support two styles of string formatting, one providing a large
degree of flexibility and customization (see @ref{784,,str.format()},
@ref{f1a,,Format String Syntax} and @ref{1a99,,Custom String Formatting}) and the other based on C
@code{printf} style formatting that handles a narrower range of types and is
slightly harder to use correctly, but is often faster for the cases it can
handle (@ref{14e4,,printf-style String Formatting}).

The @ref{1a9a,,Text Processing Services} section of the standard library covers a number of
other modules that provide various text related utilities (including regular
expression support in the @ref{d8,,re} module).

@geindex capitalize() (str method)
@anchor{library/stdtypes str capitalize}@anchor{1a9b}
@deffn {Method} str.capitalize ()

Return a copy of the string with its first character capitalized and the
rest lowercased.


Changed in version 3.8: The first character is now put into titlecase rather than uppercase.
This means that characters like digraphs will only have their first
letter capitalized, instead of the full character.

@end deffn

@geindex casefold() (str method)
@anchor{library/stdtypes str casefold}@anchor{935}
@deffn {Method} str.casefold ()

Return a casefolded copy of the string. Casefolded strings may be used for
caseless matching.

Casefolding is similar to lowercasing but more aggressive because it is
intended to remove all case distinctions in a string. For example, the German
lowercase letter @code{'ß'} is equivalent to @code{"ss"}. Since it is already
lowercase, @ref{1a9c,,lower()} would do nothing to @code{'ß'}; @ref{935,,casefold()}
converts it to @code{"ss"}.

The casefolding algorithm is described in section 3.13 of the Unicode
Standard.


New in version 3.3.

@end deffn

@geindex center() (str method)
@anchor{library/stdtypes str center}@anchor{155d}
@deffn {Method} str.center (width[, fillchar])

Return centered in a string of length @emph{width}. Padding is done using the
specified @emph{fillchar} (default is an ASCII space). The original string is
returned if @emph{width} is less than or equal to @code{len(s)}.
@end deffn

@geindex count() (str method)
@anchor{library/stdtypes str count}@anchor{1a9d}
@deffn {Method} str.count (sub[, start[, end]])

Return the number of non-overlapping occurrences of substring @emph{sub} in the
range [@emph{start}, @emph{end}].  Optional arguments @emph{start} and @emph{end} are
interpreted as in slice notation.
@end deffn

@geindex encode() (str method)
@anchor{library/stdtypes str encode}@anchor{3ba}
@deffn {Method} str.encode (encoding='utf-8', errors='strict')

Return an encoded version of the string as a bytes object. Default encoding
is @code{'utf-8'}. @emph{errors} may be given to set a different error handling scheme.
The default for @emph{errors} is @code{'strict'}, meaning that encoding errors raise
a @ref{e49,,UnicodeError}. Other possible
values are @code{'ignore'}, @code{'replace'}, @code{'xmlcharrefreplace'},
@code{'backslashreplace'} and any other name registered via
@ref{fed,,codecs.register_error()}, see section @ref{162a,,Error Handlers}. For a
list of possible encodings, see section @ref{916,,Standard Encodings}.

By default, the @emph{errors} argument is not checked for best performances, but
only used at the first encoding error. Enable the @ref{3ac,,Python Development Mode}, or use a @ref{338,,debug build} to check
@emph{errors}.


Changed in version 3.1: Support for keyword arguments added.



Changed in version 3.9: The @emph{errors} is now checked in development mode and
in @ref{338,,debug mode}.

@end deffn

@geindex endswith() (str method)
@anchor{library/stdtypes str endswith}@anchor{a3e}
@deffn {Method} str.endswith (suffix[, start[, end]])

Return @code{True} if the string ends with the specified @emph{suffix}, otherwise return
@code{False}.  @emph{suffix} can also be a tuple of suffixes to look for.  With optional
@emph{start}, test beginning at that position.  With optional @emph{end}, stop comparing
at that position.
@end deffn

@geindex expandtabs() (str method)
@anchor{library/stdtypes str expandtabs}@anchor{1a9e}
@deffn {Method} str.expandtabs (tabsize=8)

Return a copy of the string where all tab characters are replaced by one or
more spaces, depending on the current column and the given tab size.  Tab
positions occur every @emph{tabsize} characters (default is 8, giving tab
positions at columns 0, 8, 16 and so on).  To expand the string, the current
column is set to zero and the string is examined character by character.  If
the character is a tab (@code{\t}), one or more space characters are inserted
in the result until the current column is equal to the next tab position.
(The tab character itself is not copied.)  If the character is a newline
(@code{\n}) or return (@code{\r}), it is copied and the current column is reset to
zero.  Any other character is copied unchanged and the current column is
incremented by one regardless of how the character is represented when
printed.

@example
>>> '01\t012\t0123\t01234'.expandtabs()
'01      012     0123    01234'
>>> '01\t012\t0123\t01234'.expandtabs(4)
'01  012 0123    01234'
@end example
@end deffn

@geindex find() (str method)
@anchor{library/stdtypes str find}@anchor{a19}
@deffn {Method} str.find (sub[, start[, end]])

Return the lowest index in the string where substring @emph{sub} is found within
the slice @code{s[start:end]}.  Optional arguments @emph{start} and @emph{end} are
interpreted as in slice notation.  Return @code{-1} if @emph{sub} is not found.

@cartouche
@quotation Note 
The @ref{a19,,find()} method should be used only if you need to know the
position of @emph{sub}.  To check if @emph{sub} is a substring or not, use the
@ref{2fc,,in} operator:

@example
>>> 'Py' in 'Python'
True
@end example
@end quotation
@end cartouche
@end deffn

@geindex format() (str method)
@anchor{library/stdtypes str format}@anchor{784}
@deffn {Method} str.format (*args, **kwargs)

Perform a string formatting operation.  The string on which this method is
called can contain literal text or replacement fields delimited by braces
@code{@{@}}.  Each replacement field contains either the numeric index of a
positional argument, or the name of a keyword argument.  Returns a copy of
the string where each replacement field is replaced with the string value of
the corresponding argument.

@example
>>> "The sum of 1 + 2 is @{0@}".format(1+2)
'The sum of 1 + 2 is 3'
@end example

See @ref{f1a,,Format String Syntax} for a description of the various formatting options
that can be specified in format strings.

@cartouche
@quotation Note 
When formatting a number (@ref{1b1,,int}, @ref{256,,float}, @ref{33b,,complex},
@ref{248,,decimal.Decimal} and subclasses) with the @code{n} type
(ex: @code{'@{:n@}'.format(1234)}), the function temporarily sets the
@code{LC_CTYPE} locale to the @code{LC_NUMERIC} locale to decode
@code{decimal_point} and @code{thousands_sep} fields of @code{localeconv()} if
they are non-ASCII or longer than 1 byte, and the @code{LC_NUMERIC} locale is
different than the @code{LC_CTYPE} locale.  This temporary change affects
other threads.
@end quotation
@end cartouche


Changed in version 3.7: When formatting a number with the @code{n} type, the function sets
temporarily the @code{LC_CTYPE} locale to the @code{LC_NUMERIC} locale in some
cases.

@end deffn

@geindex format_map() (str method)
@anchor{library/stdtypes str format_map}@anchor{936}
@deffn {Method} str.format_map (mapping)

Similar to @code{str.format(**mapping)}, except that @code{mapping} is
used directly and not copied to a @ref{3b0,,dict}.  This is useful
if for example @code{mapping} is a dict subclass:

@example
>>> class Default(dict):
...     def __missing__(self, key):
...         return key
...
>>> '@{name@} was born in @{country@}'.format_map(Default(name='Guido'))
'Guido was born in country'
@end example


New in version 3.2.

@end deffn

@geindex index() (str method)
@anchor{library/stdtypes str index}@anchor{1a9f}
@deffn {Method} str.index (sub[, start[, end]])

Like @ref{a19,,find()}, but raise @ref{17c,,ValueError} when the substring is
not found.
@end deffn

@geindex isalnum() (str method)
@anchor{library/stdtypes str isalnum}@anchor{1aa0}
@deffn {Method} str.isalnum ()

Return @code{True} if all characters in the string are alphanumeric and there is at
least one character, @code{False} otherwise.  A character @code{c} is alphanumeric if one
of the following returns @code{True}: @code{c.isalpha()}, @code{c.isdecimal()},
@code{c.isdigit()}, or @code{c.isnumeric()}.
@end deffn

@geindex isalpha() (str method)
@anchor{library/stdtypes str isalpha}@anchor{1aa1}
@deffn {Method} str.isalpha ()

Return @code{True} if all characters in the string are alphabetic and there is at least
one character, @code{False} otherwise.  Alphabetic characters are those characters defined
in the Unicode character database as “Letter”, i.e., those with general category
property being one of “Lm”, “Lt”, “Lu”, “Ll”, or “Lo”.  Note that this is different
from the “Alphabetic” property defined in the Unicode Standard.
@end deffn

@geindex isascii() (str method)
@anchor{library/stdtypes str isascii}@anchor{60e}
@deffn {Method} str.isascii ()

Return @code{True} if the string is empty or all characters in the string are ASCII,
@code{False} otherwise.
ASCII characters have code points in the range U+0000-U+007F.


New in version 3.7.

@end deffn

@geindex isdecimal() (str method)
@anchor{library/stdtypes str isdecimal}@anchor{1aa2}
@deffn {Method} str.isdecimal ()

Return @code{True} if all characters in the string are decimal
characters and there is at least one character, @code{False}
otherwise. Decimal characters are those that can be used to form
numbers in base 10, e.g. U+0660, ARABIC-INDIC DIGIT
ZERO.  Formally a decimal character is a character in the Unicode
General Category “Nd”.
@end deffn

@geindex isdigit() (str method)
@anchor{library/stdtypes str isdigit}@anchor{1aa3}
@deffn {Method} str.isdigit ()

Return @code{True} if all characters in the string are digits and there is at least one
character, @code{False} otherwise.  Digits include decimal characters and digits that need
special handling, such as the compatibility superscript digits.
This covers digits which cannot be used to form numbers in base 10,
like the Kharosthi numbers.  Formally, a digit is a character that has the
property value Numeric_Type=Digit or Numeric_Type=Decimal.
@end deffn

@geindex isidentifier() (str method)
@anchor{library/stdtypes str isidentifier}@anchor{11e2}
@deffn {Method} str.isidentifier ()

Return @code{True} if the string is a valid identifier according to the language
definition, section @ref{170b,,Identifiers and keywords}.

Call @ref{1aa4,,keyword.iskeyword()} to test whether string @code{s} is a reserved
identifier, such as @ref{fb8,,def} and @ref{e72,,class}.

Example:

@example
>>> from keyword import iskeyword

>>> 'hello'.isidentifier(), iskeyword('hello')
(True, False)
>>> 'def'.isidentifier(), iskeyword('def')
(True, True)
@end example
@end deffn

@geindex islower() (str method)
@anchor{library/stdtypes str islower}@anchor{1aa5}
@deffn {Method} str.islower ()

Return @code{True} if all cased characters @footnote{@w{(4)} 
Cased characters are those with general category property being one of
“Lu” (Letter, uppercase), “Ll” (Letter, lowercase), or “Lt” (Letter, titlecase).
} in the string are lowercase and
there is at least one cased character, @code{False} otherwise.
@end deffn

@geindex isnumeric() (str method)
@anchor{library/stdtypes str isnumeric}@anchor{1aa6}
@deffn {Method} str.isnumeric ()

Return @code{True} if all characters in the string are numeric
characters, and there is at least one character, @code{False}
otherwise. Numeric characters include digit characters, and all characters
that have the Unicode numeric value property, e.g. U+2155,
VULGAR FRACTION ONE FIFTH.  Formally, numeric characters are those with the property
value Numeric_Type=Digit, Numeric_Type=Decimal or Numeric_Type=Numeric.
@end deffn

@geindex isprintable() (str method)
@anchor{library/stdtypes str isprintable}@anchor{937}
@deffn {Method} str.isprintable ()

Return @code{True} if all characters in the string are printable or the string is
empty, @code{False} otherwise.  Nonprintable characters are those characters defined
in the Unicode character database as “Other” or “Separator”, excepting the
ASCII space (0x20) which is considered printable.  (Note that printable
characters in this context are those which should not be escaped when
@ref{27d,,repr()} is invoked on a string.  It has no bearing on the handling of
strings written to @ref{5ef,,sys.stdout} or @ref{42d,,sys.stderr}.)
@end deffn

@geindex isspace() (str method)
@anchor{library/stdtypes str isspace}@anchor{1aa7}
@deffn {Method} str.isspace ()

Return @code{True} if there are only whitespace characters in the string and there is
at least one character, @code{False} otherwise.

A character is @emph{whitespace} if in the Unicode character database
(see @ref{121,,unicodedata}), either its general category is @code{Zs}
(“Separator, space”), or its bidirectional class is one of @code{WS},
@code{B}, or @code{S}.
@end deffn

@geindex istitle() (str method)
@anchor{library/stdtypes str istitle}@anchor{1aa8}
@deffn {Method} str.istitle ()

Return @code{True} if the string is a titlecased string and there is at least one
character, for example uppercase characters may only follow uncased characters
and lowercase characters only cased ones.  Return @code{False} otherwise.
@end deffn

@geindex isupper() (str method)
@anchor{library/stdtypes str isupper}@anchor{1aa9}
@deffn {Method} str.isupper ()

Return @code{True} if all cased characters @footnote{@w{(4)} 
Cased characters are those with general category property being one of
“Lu” (Letter, uppercase), “Ll” (Letter, lowercase), or “Lt” (Letter, titlecase).
} in the string are uppercase and
there is at least one cased character, @code{False} otherwise.

@example
>>> 'BANANA'.isupper()
True
>>> 'banana'.isupper()
False
>>> 'baNana'.isupper()
False
>>> ' '.isupper()
False
@end example
@end deffn

@geindex join() (str method)
@anchor{library/stdtypes str join}@anchor{1a8b}
@deffn {Method} str.join (iterable)

Return a string which is the concatenation of the strings in @emph{iterable}.
A @ref{162,,TypeError} will be raised if there are any non-string values in
@emph{iterable}, including @ref{2f0,,bytes} objects.  The separator between
elements is the string providing this method.
@end deffn

@geindex ljust() (str method)
@anchor{library/stdtypes str ljust}@anchor{155c}
@deffn {Method} str.ljust (width[, fillchar])

Return the string left justified in a string of length @emph{width}. Padding is
done using the specified @emph{fillchar} (default is an ASCII space). The
original string is returned if @emph{width} is less than or equal to @code{len(s)}.
@end deffn

@geindex lower() (str method)
@anchor{library/stdtypes str lower}@anchor{1a9c}
@deffn {Method} str.lower ()

Return a copy of the string with all the cased characters @footnote{@w{(4)} 
Cased characters are those with general category property being one of
“Lu” (Letter, uppercase), “Ll” (Letter, lowercase), or “Lt” (Letter, titlecase).
} converted to
lowercase.

The lowercasing algorithm used is described in section 3.13 of the Unicode
Standard.
@end deffn

@geindex lstrip() (str method)
@anchor{library/stdtypes str lstrip}@anchor{1aaa}
@deffn {Method} str.lstrip ([chars])

Return a copy of the string with leading characters removed.  The @emph{chars}
argument is a string specifying the set of characters to be removed.  If omitted
or @code{None}, the @emph{chars} argument defaults to removing whitespace.  The @emph{chars}
argument is not a prefix; rather, all combinations of its values are stripped:

@example
>>> '   spacious   '.lstrip()
'spacious   '
>>> 'www.example.com'.lstrip('cmowz.')
'example.com'
@end example

See @ref{3b2,,str.removeprefix()} for a method that will remove a single prefix
string rather than all of a set of characters.  For example:

@example
>>> 'Arthur: three!'.lstrip('Arthur: ')
'ee!'
>>> 'Arthur: three!'.removeprefix('Arthur: ')
'three!'
@end example
@end deffn

@geindex maketrans() (str static method)
@anchor{library/stdtypes str maketrans}@anchor{938}
@deffn {Method} static str.maketrans (x[, y[, z]])

This static method returns a translation table usable for @ref{1aab,,str.translate()}.

If there is only one argument, it must be a dictionary mapping Unicode
ordinals (integers) or characters (strings of length 1) to Unicode ordinals,
strings (of arbitrary lengths) or @code{None}.  Character keys will then be
converted to ordinals.

If there are two arguments, they must be strings of equal length, and in the
resulting dictionary, each character in x will be mapped to the character at
the same position in y.  If there is a third argument, it must be a string,
whose characters will be mapped to @code{None} in the result.
@end deffn

@geindex partition() (str method)
@anchor{library/stdtypes str partition}@anchor{a1c}
@deffn {Method} str.partition (sep)

Split the string at the first occurrence of @emph{sep}, and return a 3-tuple
containing the part before the separator, the separator itself, and the part
after the separator.  If the separator is not found, return a 3-tuple containing
the string itself, followed by two empty strings.
@end deffn

@geindex removeprefix() (str method)
@anchor{library/stdtypes str removeprefix}@anchor{3b2}
@deffn {Method} str.removeprefix (prefix, /)

If the string starts with the @emph{prefix} string, return
@code{string[len(prefix):]}. Otherwise, return a copy of the original
string:

@example
>>> 'TestHook'.removeprefix('Test')
'Hook'
>>> 'BaseTestCase'.removeprefix('Test')
'BaseTestCase'
@end example


New in version 3.9.

@end deffn

@geindex removesuffix() (str method)
@anchor{library/stdtypes str removesuffix}@anchor{3b3}
@deffn {Method} str.removesuffix (suffix, /)

If the string ends with the @emph{suffix} string and that @emph{suffix} is not empty,
return @code{string[:-len(suffix)]}. Otherwise, return a copy of the
original string:

@example
>>> 'MiscTests'.removesuffix('Tests')
'Misc'
>>> 'TmpDirMixin'.removesuffix('Tests')
'TmpDirMixin'
@end example


New in version 3.9.

@end deffn

@geindex replace() (str method)
@anchor{library/stdtypes str replace}@anchor{1aac}
@deffn {Method} str.replace (old, new[, count])

Return a copy of the string with all occurrences of substring @emph{old} replaced by
@emph{new}.  If the optional argument @emph{count} is given, only the first @emph{count}
occurrences are replaced.
@end deffn

@geindex rfind() (str method)
@anchor{library/stdtypes str rfind}@anchor{a1a}
@deffn {Method} str.rfind (sub[, start[, end]])

Return the highest index in the string where substring @emph{sub} is found, such
that @emph{sub} is contained within @code{s[start:end]}.  Optional arguments @emph{start}
and @emph{end} are interpreted as in slice notation.  Return @code{-1} on failure.
@end deffn

@geindex rindex() (str method)
@anchor{library/stdtypes str rindex}@anchor{1aad}
@deffn {Method} str.rindex (sub[, start[, end]])

Like @ref{a1a,,rfind()} but raises @ref{17c,,ValueError} when the substring @emph{sub} is not
found.
@end deffn

@geindex rjust() (str method)
@anchor{library/stdtypes str rjust}@anchor{155b}
@deffn {Method} str.rjust (width[, fillchar])

Return the string right justified in a string of length @emph{width}. Padding is
done using the specified @emph{fillchar} (default is an ASCII space). The
original string is returned if @emph{width} is less than or equal to @code{len(s)}.
@end deffn

@geindex rpartition() (str method)
@anchor{library/stdtypes str rpartition}@anchor{1aae}
@deffn {Method} str.rpartition (sep)

Split the string at the last occurrence of @emph{sep}, and return a 3-tuple
containing the part before the separator, the separator itself, and the part
after the separator.  If the separator is not found, return a 3-tuple containing
two empty strings, followed by the string itself.
@end deffn

@geindex rsplit() (str method)
@anchor{library/stdtypes str rsplit}@anchor{1aaf}
@deffn {Method} str.rsplit (sep=None, maxsplit=- 1)

Return a list of the words in the string, using @emph{sep} as the delimiter string.
If @emph{maxsplit} is given, at most @emph{maxsplit} splits are done, the @emph{rightmost}
ones.  If @emph{sep} is not specified or @code{None}, any whitespace string is a
separator.  Except for splitting from the right, @ref{1aaf,,rsplit()} behaves like
@ref{a1b,,split()} which is described in detail below.
@end deffn

@geindex rstrip() (str method)
@anchor{library/stdtypes str rstrip}@anchor{1ab0}
@deffn {Method} str.rstrip ([chars])

Return a copy of the string with trailing characters removed.  The @emph{chars}
argument is a string specifying the set of characters to be removed.  If omitted
or @code{None}, the @emph{chars} argument defaults to removing whitespace.  The @emph{chars}
argument is not a suffix; rather, all combinations of its values are stripped:

@example
>>> '   spacious   '.rstrip()
'   spacious'
>>> 'mississippi'.rstrip('ipz')
'mississ'
@end example

See @ref{3b3,,str.removesuffix()} for a method that will remove a single suffix
string rather than all of a set of characters.  For example:

@example
>>> 'Monty Python'.rstrip(' Python')
'M'
>>> 'Monty Python'.removesuffix(' Python')
'Monty'
@end example
@end deffn

@geindex split() (str method)
@anchor{library/stdtypes str split}@anchor{a1b}
@deffn {Method} str.split (sep=None, maxsplit=- 1)

Return a list of the words in the string, using @emph{sep} as the delimiter
string.  If @emph{maxsplit} is given, at most @emph{maxsplit} splits are done (thus,
the list will have at most @code{maxsplit+1} elements).  If @emph{maxsplit} is not
specified or @code{-1}, then there is no limit on the number of splits
(all possible splits are made).

If @emph{sep} is given, consecutive delimiters are not grouped together and are
deemed to delimit empty strings (for example, @code{'1,,2'.split(',')} returns
@code{['1', '@w{'}, '2']}).  The @emph{sep} argument may consist of multiple characters
(for example, @code{'1<>2<>3'.split('<>')} returns @code{['1', '2', '3']}).
Splitting an empty string with a specified separator returns @code{['@w{'}]}.

For example:

@example
>>> '1,2,3'.split(',')
['1', '2', '3']
>>> '1,2,3'.split(',', maxsplit=1)
['1', '2,3']
>>> '1,2,,3,'.split(',')
['1', '2', '@w{'}, '3', '@w{'}]
@end example

If @emph{sep} is not specified or is @code{None}, a different splitting algorithm is
applied: runs of consecutive whitespace are regarded as a single separator,
and the result will contain no empty strings at the start or end if the
string has leading or trailing whitespace.  Consequently, splitting an empty
string or a string consisting of just whitespace with a @code{None} separator
returns @code{[]}.

For example:

@example
>>> '1 2 3'.split()
['1', '2', '3']
>>> '1 2 3'.split(maxsplit=1)
['1', '2 3']
>>> '   1   2   3   '.split()
['1', '2', '3']
@end example
@end deffn

@geindex universal newlines; str.splitlines method

@geindex splitlines() (str method)
@anchor{library/stdtypes str splitlines}@anchor{1ab1}
@deffn {Method} str.splitlines (keepends=False)

Return a list of the lines in the string, breaking at line boundaries.  Line
breaks are not included in the resulting list unless @emph{keepends} is given and
true.

This method splits on the following line boundaries.  In particular, the
boundaries are a superset of @ref{88a,,universal newlines}.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Representation

@tab

Description

@item

@code{\n}

@tab

Line Feed

@item

@code{\r}

@tab

Carriage Return

@item

@code{\r\n}

@tab

Carriage Return + Line Feed

@item

@code{\v} or @code{\x0b}

@tab

Line Tabulation

@item

@code{\f} or @code{\x0c}

@tab

Form Feed

@item

@code{\x1c}

@tab

File Separator

@item

@code{\x1d}

@tab

Group Separator

@item

@code{\x1e}

@tab

Record Separator

@item

@code{\x85}

@tab

Next Line (C1 Control Code)

@item

@code{\u2028}

@tab

Line Separator

@item

@code{\u2029}

@tab

Paragraph Separator

@end multitable



Changed in version 3.2: @code{\v} and @code{\f} added to list of line boundaries.


For example:

@example
>>> 'ab c\n\nde fg\rkl\r\n'.splitlines()
['ab c', '@w{'}, 'de fg', 'kl']
>>> 'ab c\n\nde fg\rkl\r\n'.splitlines(keepends=True)
['ab c\n', '\n', 'de fg\r', 'kl\r\n']
@end example

Unlike @ref{a1b,,split()} when a delimiter string @emph{sep} is given, this
method returns an empty list for the empty string, and a terminal line
break does not result in an extra line:

@example
>>> "".splitlines()
[]
>>> "One line\n".splitlines()
['One line']
@end example

For comparison, @code{split('\n')} gives:

@example
>>> '@w{'}.split('\n')
['@w{'}]
>>> 'Two lines\n'.split('\n')
['Two lines', '@w{'}]
@end example
@end deffn

@geindex startswith() (str method)
@anchor{library/stdtypes str startswith}@anchor{a3d}
@deffn {Method} str.startswith (prefix[, start[, end]])

Return @code{True} if string starts with the @emph{prefix}, otherwise return @code{False}.
@emph{prefix} can also be a tuple of prefixes to look for.  With optional @emph{start},
test string beginning at that position.  With optional @emph{end}, stop comparing
string at that position.
@end deffn

@geindex strip() (str method)
@anchor{library/stdtypes str strip}@anchor{1ab2}
@deffn {Method} str.strip ([chars])

Return a copy of the string with the leading and trailing characters removed.
The @emph{chars} argument is a string specifying the set of characters to be removed.
If omitted or @code{None}, the @emph{chars} argument defaults to removing whitespace.
The @emph{chars} argument is not a prefix or suffix; rather, all combinations of its
values are stripped:

@example
>>> '   spacious   '.strip()
'spacious'
>>> 'www.example.com'.strip('cmowz.')
'example'
@end example

The outermost leading and trailing @emph{chars} argument values are stripped
from the string. Characters are removed from the leading end until
reaching a string character that is not contained in the set of
characters in @emph{chars}. A similar action takes place on the trailing end.
For example:

@example
>>> comment_string = '#....... Section 3.2.1 Issue #32 .......'
>>> comment_string.strip('.#! ')
'Section 3.2.1 Issue #32'
@end example
@end deffn

@geindex swapcase() (str method)
@anchor{library/stdtypes str swapcase}@anchor{1ab3}
@deffn {Method} str.swapcase ()

Return a copy of the string with uppercase characters converted to lowercase and
vice versa. Note that it is not necessarily true that
@code{s.swapcase().swapcase() == s}.
@end deffn

@geindex title() (str method)
@anchor{library/stdtypes str title}@anchor{1ab4}
@deffn {Method} str.title ()

Return a titlecased version of the string where words start with an uppercase
character and the remaining characters are lowercase.

For example:

@example
>>> 'Hello world'.title()
'Hello World'
@end example

The algorithm uses a simple language-independent definition of a word as
groups of consecutive letters.  The definition works in many contexts but
it means that apostrophes in contractions and possessives form word
boundaries, which may not be the desired result:

@example
>>> "they're bill's friends from the UK".title()
"They'Re Bill'S Friends From The Uk"
@end example

A workaround for apostrophes can be constructed using regular expressions:

@example
>>> import re
>>> def titlecase(s):
...     return re.sub(r"[A-Za-z]+('[A-Za-z]+)?",
...                   lambda mo: mo.group(0).capitalize(),
...                   s)
...
>>> titlecase("they're bill's friends.")
"They're Bill's Friends."
@end example
@end deffn

@geindex translate() (str method)
@anchor{library/stdtypes str translate}@anchor{1aab}
@deffn {Method} str.translate (table)

Return a copy of the string in which each character has been mapped through
the given translation table.  The table must be an object that implements
indexing via @code{__getitem__()}, typically a @ref{d62,,mapping} or
@ref{14e7,,sequence}.  When indexed by a Unicode ordinal (an integer), the
table object can do any of the following: return a Unicode ordinal or a
string, to map the character to one or more other characters; return
@code{None}, to delete the character from the return string; or raise a
@ref{1ab5,,LookupError} exception, to map the character to itself.

You can use @ref{938,,str.maketrans()} to create a translation map from
character-to-character mappings in different formats.

See also the @ref{1a,,codecs} module for a more flexible approach to custom
character mappings.
@end deffn

@geindex upper() (str method)
@anchor{library/stdtypes str upper}@anchor{1ab6}
@deffn {Method} str.upper ()

Return a copy of the string with all the cased characters @footnote{@w{(4)} 
Cased characters are those with general category property being one of
“Lu” (Letter, uppercase), “Ll” (Letter, lowercase), or “Lt” (Letter, titlecase).
} converted to
uppercase.  Note that @code{s.upper().isupper()} might be @code{False} if @code{s}
contains uncased characters or if the Unicode category of the resulting
character(s) is not “Lu” (Letter, uppercase), but e.g. “Lt” (Letter,
titlecase).

The uppercasing algorithm used is described in section 3.13 of the Unicode
Standard.
@end deffn

@geindex zfill() (str method)
@anchor{library/stdtypes str zfill}@anchor{155e}
@deffn {Method} str.zfill (width)

Return a copy of the string left filled with ASCII @code{'0'} digits to
make a string of length @emph{width}. A leading sign prefix (@code{'+'}/@code{'-'})
is handled by inserting the padding @emph{after} the sign character rather
than before. The original string is returned if @emph{width} is less than
or equal to @code{len(s)}.

For example:

@example
>>> "42".zfill(5)
'00042'
>>> "-42".zfill(5)
'-0042'
@end example
@end deffn

@node printf-style String Formatting,,String Methods<2>,Text Sequence Type — str
@anchor{library/stdtypes old-string-formatting}@anchor{14e4}@anchor{library/stdtypes printf-style-string-formatting}@anchor{1ab7}
@subsubsection @code{printf}-style String Formatting


@geindex formatting@comma{} string (%)
@geindex interpolation@comma{} string (%)
@geindex string; formatting@comma{} printf
@geindex string; interpolation@comma{} printf
@geindex printf-style formatting
@geindex sprintf-style formatting
@geindex % (percent); printf-style formatting

@cartouche
@quotation Note 
The formatting operations described here exhibit a variety of quirks that
lead to a number of common errors (such as failing to display tuples and
dictionaries correctly).  Using the newer @ref{4b0,,formatted string literals}, the @ref{784,,str.format()} interface, or @ref{1ab8,,template strings} may help avoid these errors.  Each of these
alternatives provides their own trade-offs and benefits of simplicity,
flexibility, and/or extensibility.
@end quotation
@end cartouche

String objects have one unique built-in operation: the @code{%} operator (modulo).
This is also known as the string @emph{formatting} or @emph{interpolation} operator.
Given @code{format % values} (where @emph{format} is a string), @code{%} conversion
specifications in @emph{format} are replaced with zero or more elements of @emph{values}.
The effect is similar to using the @code{sprintf()} in the C language.

If @emph{format} requires a single argument, @emph{values} may be a single non-tuple
object. @footnote{@w{(5)} 
To format only a tuple you should therefore provide a singleton tuple whose only
element is the tuple to be formatted.
}  Otherwise, @emph{values} must be a tuple with exactly the number of
items specified by the format string, or a single mapping object (for example, a
dictionary).

@geindex () (parentheses); in printf-style formatting
@geindex * (asterisk); in printf-style formatting
@geindex . (dot); in printf-style formatting

A conversion specifier contains two or more characters and has the following
components, which must occur in this order:


@enumerate 

@item 
The @code{'%'} character, which marks the start of the specifier.

@item 
Mapping key (optional), consisting of a parenthesised sequence of characters
(for example, @code{(somename)}).

@item 
Conversion flags (optional), which affect the result of some conversion
types.

@item 
Minimum field width (optional).  If specified as an @code{'*'} (asterisk), the
actual width is read from the next element of the tuple in @emph{values}, and the
object to convert comes after the minimum field width and optional precision.

@item 
Precision (optional), given as a @code{'.'} (dot) followed by the precision.  If
specified as @code{'*'} (an asterisk), the actual precision is read from the next
element of the tuple in @emph{values}, and the value to convert comes after the
precision.

@item 
Length modifier (optional).

@item 
Conversion type.
@end enumerate

When the right argument is a dictionary (or other mapping type), then the
formats in the string @emph{must} include a parenthesised mapping key into that
dictionary inserted immediately after the @code{'%'} character. The mapping key
selects the value to be formatted from the mapping.  For example:

@example
>>> print('%(language)s has %(number)03d quote types.' %
...       @{'language': "Python", "number": 2@})
Python has 002 quote types.
@end example

In this case no @code{*} specifiers may occur in a format (since they require a
sequential parameter list).

The conversion flag characters are:

@geindex # (hash); in printf-style formatting
@geindex - (minus); in printf-style formatting
@geindex + (plus); in printf-style formatting
@geindex space; in printf-style formatting


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Flag

@tab

Meaning

@item

@code{'#'}

@tab

The value conversion will use the “alternate form” (where defined
below).

@item

@code{'0'}

@tab

The conversion will be zero padded for numeric values.

@item

@code{'-'}

@tab

The converted value is left adjusted (overrides the @code{'0'}
conversion if both are given).

@item

@code{' '}

@tab

(a space) A blank should be left before a positive number (or empty
string) produced by a signed conversion.

@item

@code{'+'}

@tab

A sign character (@code{'+'} or @code{'-'}) will precede the conversion
(overrides a “space” flag).

@end multitable


A length modifier (@code{h}, @code{l}, or @code{L}) may be present, but is ignored as it
is not necessary for Python – so e.g. @code{%ld} is identical to @code{%d}.

The conversion types are:


@multitable {xxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} 
@headitem

Conversion

@tab

Meaning

@tab

Notes

@item

@code{'d'}

@tab

Signed integer decimal.

@tab

@item

@code{'i'}

@tab

Signed integer decimal.

@tab

@item

@code{'o'}

@tab

Signed octal value.

@tab

(1)

@item

@code{'u'}

@tab

Obsolete type – it is identical to @code{'d'}.

@tab

(6)

@item

@code{'x'}

@tab

Signed hexadecimal (lowercase).

@tab

(2)

@item

@code{'X'}

@tab

Signed hexadecimal (uppercase).

@tab

(2)

@item

@code{'e'}

@tab

Floating point exponential format (lowercase).

@tab

(3)

@item

@code{'E'}

@tab

Floating point exponential format (uppercase).

@tab

(3)

@item

@code{'f'}

@tab

Floating point decimal format.

@tab

(3)

@item

@code{'F'}

@tab

Floating point decimal format.

@tab

(3)

@item

@code{'g'}

@tab

Floating point format. Uses lowercase exponential
format if exponent is less than -4 or not less than
precision, decimal format otherwise.

@tab

(4)

@item

@code{'G'}

@tab

Floating point format. Uses uppercase exponential
format if exponent is less than -4 or not less than
precision, decimal format otherwise.

@tab

(4)

@item

@code{'c'}

@tab

Single character (accepts integer or single
character string).

@tab

@item

@code{'r'}

@tab

String (converts any Python object using
@ref{27d,,repr()}).

@tab

(5)

@item

@code{'s'}

@tab

String (converts any Python object using
@ref{2ef,,str()}).

@tab

(5)

@item

@code{'a'}

@tab

String (converts any Python object using
@ref{f47,,ascii()}).

@tab

(5)

@item

@code{'%'}

@tab

No argument is converted, results in a @code{'%'}
character in the result.

@tab

@end multitable


Notes:


@enumerate 

@item 
The alternate form causes a leading octal specifier (@code{'0o'}) to be
inserted before the first digit.

@item 
The alternate form causes a leading @code{'0x'} or @code{'0X'} (depending on whether
the @code{'x'} or @code{'X'} format was used) to be inserted before the first digit.

@item 
The alternate form causes the result to always contain a decimal point, even if
no digits follow it.

The precision determines the number of digits after the decimal point and
defaults to 6.

@item 
The alternate form causes the result to always contain a decimal point, and
trailing zeroes are not removed as they would otherwise be.

The precision determines the number of significant digits before and after the
decimal point and defaults to 6.

@item 
If precision is @code{N}, the output is truncated to @code{N} characters.

@item 
See 
@geindex Python Enhancement Proposals; PEP 237
PEP 237@footnote{https://www.python.org/dev/peps/pep-0237}.
@end enumerate

Since Python strings have an explicit length, @code{%s} conversions do not assume
that @code{'\0'} is the end of the string.

@c XXX Examples?


Changed in version 3.1: @code{%f} conversions for numbers whose absolute value is over 1e50 are no
longer replaced by @code{%g} conversions.


@geindex buffer protocol; binary sequence types

@node Binary Sequence Types — bytes bytearray memoryview,Set Types — set frozenset,Text Sequence Type — str,Built-in Types
@anchor{library/stdtypes binary-sequence-types-bytes-bytearray-memoryview}@anchor{1ab9}@anchor{library/stdtypes binaryseq}@anchor{1a3f}
@subsection Binary Sequence Types — @code{bytes}, @code{bytearray}, @code{memoryview}


@geindex object; bytes
@geindex object; bytearray
@geindex object; memoryview
@geindex module; array

The core built-in types for manipulating binary data are @ref{2f0,,bytes} and
@ref{2f1,,bytearray}. They are supported by @ref{573,,memoryview} which uses
the @ref{1a3e,,buffer protocol} to access the memory of other
binary objects without needing to make a copy.

The @ref{6,,array} module supports efficient storage of basic data types like
32-bit integers and IEEE754 double-precision floating values.

@menu
* Bytes Objects:: 
* Bytearray Objects:: 
* Bytes and Bytearray Operations:: 
* printf-style Bytes Formatting:: 
* Memory Views:: 

@end menu

@node Bytes Objects,Bytearray Objects,,Binary Sequence Types — bytes bytearray memoryview
@anchor{library/stdtypes bytes-objects}@anchor{1aba}@anchor{library/stdtypes typebytes}@anchor{1a41}
@subsubsection Bytes Objects


@geindex object; bytes

Bytes objects are immutable sequences of single bytes. Since many major
binary protocols are based on the ASCII text encoding, bytes objects offer
several methods that are only valid when working with ASCII compatible
data and are closely related to string objects in a variety of other ways.

@geindex bytes (built-in class)
@anchor{library/stdtypes bytes}@anchor{2f0}
@deffn {Class} bytes ([source[, encoding[, errors]]])

Firstly, the syntax for bytes literals is largely the same as that for string
literals, except that a @code{b} prefix is added:


@itemize *

@item 
Single quotes: @code{b'still allows embedded "double" quotes'}

@item 
Double quotes: @code{b"still allows embedded 'single' quotes"}

@item 
Triple quoted: @code{b'@w{'}'3 single quotes'@w{'}'}, @code{b"""3 double quotes"""}
@end itemize

Only ASCII characters are permitted in bytes literals (regardless of the
declared source code encoding). Any binary values over 127 must be entered
into bytes literals using the appropriate escape sequence.

As with string literals, bytes literals may also use a @code{r} prefix to disable
processing of escape sequences. See @ref{1720,,String and Bytes literals} for more about the various
forms of bytes literal, including supported escape sequences.

While bytes literals and representations are based on ASCII text, bytes
objects actually behave like immutable sequences of integers, with each
value in the sequence restricted such that @code{0 <= x < 256} (attempts to
violate this restriction will trigger @ref{17c,,ValueError}). This is done
deliberately to emphasise that while many binary formats include ASCII based
elements and can be usefully manipulated with some text-oriented algorithms,
this is not generally the case for arbitrary binary data (blindly applying
text processing algorithms to binary data formats that are not ASCII
compatible will usually lead to data corruption).

In addition to the literal forms, bytes objects can be created in a number of
other ways:


@itemize *

@item 
A zero-filled bytes object of a specified length: @code{bytes(10)}

@item 
From an iterable of integers: @code{bytes(range(20))}

@item 
Copying existing binary data via the buffer protocol:  @code{bytes(obj)}
@end itemize

Also see the @ref{137e,,bytes} built-in.

Since 2 hexadecimal digits correspond precisely to a single byte, hexadecimal
numbers are a commonly used format for describing binary data. Accordingly,
the bytes type has an additional class method to read data in that format:

@geindex fromhex() (bytes class method)
@anchor{library/stdtypes bytes fromhex}@anchor{60c}
@deffn {Method} classmethod fromhex (string)

This @ref{2f0,,bytes} class method returns a bytes object, decoding the
given string object.  The string must contain two hexadecimal digits per
byte, with ASCII whitespace being ignored.

@example
>>> bytes.fromhex('2Ef0 F1f2  ')
b'.\xf0\xf1\xf2'
@end example


Changed in version 3.7: @ref{60c,,bytes.fromhex()} now skips all ASCII whitespace in the string,
not just spaces.

@end deffn

A reverse conversion function exists to transform a bytes object into its
hexadecimal representation.

@geindex hex() (bytes method)
@anchor{library/stdtypes bytes hex}@anchor{8c1}
@deffn {Method} hex ([sep[, bytes_per_sep]])

Return a string object containing two hexadecimal digits for each
byte in the instance.

@example
>>> b'\xf0\xf1\xf2'.hex()
'f0f1f2'
@end example

If you want to make the hex string easier to read, you can specify a
single character separator @emph{sep} parameter to include in the output.
By default between each byte.  A second optional @emph{bytes_per_sep}
parameter controls the spacing.  Positive values calculate the
separator position from the right, negative values from the left.

@example
>>> value = b'\xf0\xf1\xf2'
>>> value.hex('-')
'f0-f1-f2'
>>> value.hex('_', 2)
'f0_f1f2'
>>> b'UUDDLRLRAB'.hex(' ', -4)
'55554444 4c524c52 4142'
@end example


New in version 3.5.



Changed in version 3.8: @ref{8c1,,bytes.hex()} now supports optional @emph{sep} and @emph{bytes_per_sep}
parameters to insert separators between bytes in the hex output.

@end deffn
@end deffn

Since bytes objects are sequences of integers (akin to a tuple), for a bytes
object @emph{b}, @code{b[0]} will be an integer, while @code{b[0:1]} will be a bytes
object of length 1.  (This contrasts with text strings, where both indexing
and slicing will produce a string of length 1)

The representation of bytes objects uses the literal format (@code{b'...'})
since it is often more useful than e.g. @code{bytes([46, 46, 46])}.  You can
always convert a bytes object into a list of integers using @code{list(b)}.

@cartouche
@quotation Note 
For Python 2.x users: In the Python 2.x series, a variety of implicit
conversions between 8-bit strings (the closest thing 2.x offers to a
built-in binary data type) and Unicode strings were permitted. This was a
backwards compatibility workaround to account for the fact that Python
originally only supported 8-bit text, and Unicode text was a later
addition. In Python 3.x, those implicit conversions are gone - conversions
between 8-bit binary data and Unicode text must be explicit, and bytes and
string objects will always compare unequal.
@end quotation
@end cartouche

@node Bytearray Objects,Bytes and Bytearray Operations,Bytes Objects,Binary Sequence Types — bytes bytearray memoryview
@anchor{library/stdtypes bytearray-objects}@anchor{1abb}@anchor{library/stdtypes typebytearray}@anchor{1a40}
@subsubsection Bytearray Objects


@geindex object; bytearray

@ref{2f1,,bytearray} objects are a mutable counterpart to @ref{2f0,,bytes}
objects.

@geindex bytearray (built-in class)
@anchor{library/stdtypes bytearray}@anchor{2f1}
@deffn {Class} bytearray ([source[, encoding[, errors]]])

There is no dedicated literal syntax for bytearray objects, instead
they are always created by calling the constructor:


@itemize *

@item 
Creating an empty instance: @code{bytearray()}

@item 
Creating a zero-filled instance with a given length: @code{bytearray(10)}

@item 
From an iterable of integers: @code{bytearray(range(20))}

@item 
Copying existing binary data via the buffer protocol:  @code{bytearray(b'Hi!')}
@end itemize

As bytearray objects are mutable, they support the
@ref{1a3c,,mutable} sequence operations in addition to the
common bytes and bytearray operations described in @ref{1a3d,,Bytes and Bytearray Operations}.

Also see the @ref{137f,,bytearray} built-in.

Since 2 hexadecimal digits correspond precisely to a single byte, hexadecimal
numbers are a commonly used format for describing binary data. Accordingly,
the bytearray type has an additional class method to read data in that format:

@geindex fromhex() (bytearray class method)
@anchor{library/stdtypes bytearray fromhex}@anchor{60d}
@deffn {Method} classmethod fromhex (string)

This @ref{2f1,,bytearray} class method returns bytearray object, decoding
the given string object.  The string must contain two hexadecimal digits
per byte, with ASCII whitespace being ignored.

@example
>>> bytearray.fromhex('2Ef0 F1f2  ')
bytearray(b'.\xf0\xf1\xf2')
@end example


Changed in version 3.7: @ref{60d,,bytearray.fromhex()} now skips all ASCII whitespace in the string,
not just spaces.

@end deffn

A reverse conversion function exists to transform a bytearray object into its
hexadecimal representation.

@geindex hex() (bytearray method)
@anchor{library/stdtypes bytearray hex}@anchor{8c2}
@deffn {Method} hex ([sep[, bytes_per_sep]])

Return a string object containing two hexadecimal digits for each
byte in the instance.

@example
>>> bytearray(b'\xf0\xf1\xf2').hex()
'f0f1f2'
@end example


New in version 3.5.



Changed in version 3.8: Similar to @ref{8c1,,bytes.hex()}, @ref{8c2,,bytearray.hex()} now supports
optional @emph{sep} and @emph{bytes_per_sep} parameters to insert separators
between bytes in the hex output.

@end deffn
@end deffn

Since bytearray objects are sequences of integers (akin to a list), for a
bytearray object @emph{b}, @code{b[0]} will be an integer, while @code{b[0:1]} will be
a bytearray object of length 1.  (This contrasts with text strings, where
both indexing and slicing will produce a string of length 1)

The representation of bytearray objects uses the bytes literal format
(@code{bytearray(b'...')}) since it is often more useful than e.g.
@code{bytearray([46, 46, 46])}.  You can always convert a bytearray object into
a list of integers using @code{list(b)}.

@node Bytes and Bytearray Operations,printf-style Bytes Formatting,Bytearray Objects,Binary Sequence Types — bytes bytearray memoryview
@anchor{library/stdtypes bytes-and-bytearray-operations}@anchor{1abc}@anchor{library/stdtypes bytes-methods}@anchor{1a3d}
@subsubsection Bytes and Bytearray Operations


@geindex bytes; methods
@geindex bytearray; methods

Both bytes and bytearray objects support the @ref{1a89,,common}
sequence operations. They interoperate not just with operands of the same
type, but with any @ref{87e,,bytes-like object}. Due to this flexibility, they can be
freely mixed in operations without causing errors. However, the return type
of the result may depend on the order of operands.

@cartouche
@quotation Note 
The methods on bytes and bytearray objects don’t accept strings as their
arguments, just as the methods on strings don’t accept bytes as their
arguments.  For example, you have to write:

@example
a = "abc"
b = a.replace("a", "f")
@end example

and:

@example
a = b"abc"
b = a.replace(b"a", b"f")
@end example
@end quotation
@end cartouche

Some bytes and bytearray operations assume the use of ASCII compatible
binary formats, and hence should be avoided when working with arbitrary
binary data. These restrictions are covered below.

@cartouche
@quotation Note 
Using these ASCII based operations to manipulate binary data that is not
stored in an ASCII based format may lead to data corruption.
@end quotation
@end cartouche

The following methods on bytes and bytearray objects can be used with
arbitrary binary data.

@geindex count() (bytes method)
@geindex count() (bytearray method)
@anchor{library/stdtypes bytes count}@anchor{1abd}
@deffn {Method} bytes.count (sub[, start[, end]])
@anchor{library/stdtypes bytearray count}@anchor{1abe}
@deffnx {Method} bytearray.count (sub[, start[, end]])

Return the number of non-overlapping occurrences of subsequence @emph{sub} in
the range [@emph{start}, @emph{end}].  Optional arguments @emph{start} and @emph{end} are
interpreted as in slice notation.

The subsequence to search for may be any @ref{87e,,bytes-like object} or an
integer in the range 0 to 255.


Changed in version 3.3: Also accept an integer in the range 0 to 255 as the subsequence.

@end deffn

@geindex removeprefix() (bytes method)
@geindex removeprefix() (bytearray method)
@anchor{library/stdtypes bytes removeprefix}@anchor{1abf}
@deffn {Method} bytes.removeprefix (prefix, /)
@anchor{library/stdtypes bytearray removeprefix}@anchor{1ac0}
@deffnx {Method} bytearray.removeprefix (prefix, /)

If the binary data starts with the @emph{prefix} string, return
@code{bytes[len(prefix):]}. Otherwise, return a copy of the original
binary data:

@example
>>> b'TestHook'.removeprefix(b'Test')
b'Hook'
>>> b'BaseTestCase'.removeprefix(b'Test')
b'BaseTestCase'
@end example

The @emph{prefix} may be any @ref{87e,,bytes-like object}.

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place -
it always produces a new object, even if no changes were made.
@end quotation
@end cartouche


New in version 3.9.

@end deffn

@geindex removesuffix() (bytes method)
@geindex removesuffix() (bytearray method)
@anchor{library/stdtypes bytes removesuffix}@anchor{1ac1}
@deffn {Method} bytes.removesuffix (suffix, /)
@anchor{library/stdtypes bytearray removesuffix}@anchor{1ac2}
@deffnx {Method} bytearray.removesuffix (suffix, /)

If the binary data ends with the @emph{suffix} string and that @emph{suffix} is
not empty, return @code{bytes[:-len(suffix)]}.  Otherwise, return a copy of
the original binary data:

@example
>>> b'MiscTests'.removesuffix(b'Tests')
b'Misc'
>>> b'TmpDirMixin'.removesuffix(b'Tests')
b'TmpDirMixin'
@end example

The @emph{suffix} may be any @ref{87e,,bytes-like object}.

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place -
it always produces a new object, even if no changes were made.
@end quotation
@end cartouche


New in version 3.9.

@end deffn

@geindex decode() (bytes method)
@geindex decode() (bytearray method)
@anchor{library/stdtypes bytes decode}@anchor{3bb}
@deffn {Method} bytes.decode (encoding='utf-8', errors='strict')
@anchor{library/stdtypes bytearray decode}@anchor{1ac3}
@deffnx {Method} bytearray.decode (encoding='utf-8', errors='strict')

Return a string decoded from the given bytes.  Default encoding is
@code{'utf-8'}. @emph{errors} may be given to set a different
error handling scheme.  The default for @emph{errors} is @code{'strict'}, meaning
that encoding errors raise a @ref{e49,,UnicodeError}.  Other possible values are
@code{'ignore'}, @code{'replace'} and any other name registered via
@ref{fed,,codecs.register_error()}, see section @ref{162a,,Error Handlers}. For a
list of possible encodings, see section @ref{916,,Standard Encodings}.

By default, the @emph{errors} argument is not checked for best performances, but
only used at the first decoding error. Enable the @ref{3ac,,Python Development Mode}, or use a @ref{338,,debug build} to check @emph{errors}.

@cartouche
@quotation Note 
Passing the @emph{encoding} argument to @ref{2ef,,str} allows decoding any
@ref{87e,,bytes-like object} directly, without needing to make a temporary
bytes or bytearray object.
@end quotation
@end cartouche


Changed in version 3.1: Added support for keyword arguments.



Changed in version 3.9: The @emph{errors} is now checked in development mode and
in @ref{338,,debug mode}.

@end deffn

@geindex endswith() (bytes method)
@geindex endswith() (bytearray method)
@anchor{library/stdtypes bytes endswith}@anchor{1ac4}
@deffn {Method} bytes.endswith (suffix[, start[, end]])
@anchor{library/stdtypes bytearray endswith}@anchor{1ac5}
@deffnx {Method} bytearray.endswith (suffix[, start[, end]])

Return @code{True} if the binary data ends with the specified @emph{suffix},
otherwise return @code{False}.  @emph{suffix} can also be a tuple of suffixes to
look for.  With optional @emph{start}, test beginning at that position.  With
optional @emph{end}, stop comparing at that position.

The suffix(es) to search for may be any @ref{87e,,bytes-like object}.
@end deffn

@geindex find() (bytes method)
@geindex find() (bytearray method)
@anchor{library/stdtypes bytes find}@anchor{1089}
@deffn {Method} bytes.find (sub[, start[, end]])
@anchor{library/stdtypes bytearray find}@anchor{1ac6}
@deffnx {Method} bytearray.find (sub[, start[, end]])

Return the lowest index in the data where the subsequence @emph{sub} is found,
such that @emph{sub} is contained in the slice @code{s[start:end]}.  Optional
arguments @emph{start} and @emph{end} are interpreted as in slice notation.  Return
@code{-1} if @emph{sub} is not found.

The subsequence to search for may be any @ref{87e,,bytes-like object} or an
integer in the range 0 to 255.

@cartouche
@quotation Note 
The @ref{1089,,find()} method should be used only if you need to know the
position of @emph{sub}.  To check if @emph{sub} is a substring or not, use the
@ref{2fc,,in} operator:

@example
>>> b'Py' in b'Python'
True
@end example
@end quotation
@end cartouche


Changed in version 3.3: Also accept an integer in the range 0 to 255 as the subsequence.

@end deffn

@geindex index() (bytes method)
@geindex index() (bytearray method)
@anchor{library/stdtypes bytes index}@anchor{1ac7}
@deffn {Method} bytes.index (sub[, start[, end]])
@anchor{library/stdtypes bytearray index}@anchor{1ac8}
@deffnx {Method} bytearray.index (sub[, start[, end]])

Like @ref{1089,,find()}, but raise @ref{17c,,ValueError} when the
subsequence is not found.

The subsequence to search for may be any @ref{87e,,bytes-like object} or an
integer in the range 0 to 255.


Changed in version 3.3: Also accept an integer in the range 0 to 255 as the subsequence.

@end deffn

@geindex join() (bytes method)
@geindex join() (bytearray method)
@anchor{library/stdtypes bytes join}@anchor{1a8c}
@deffn {Method} bytes.join (iterable)
@anchor{library/stdtypes bytearray join}@anchor{1ac9}
@deffnx {Method} bytearray.join (iterable)

Return a bytes or bytearray object which is the concatenation of the
binary data sequences in @emph{iterable}.  A @ref{162,,TypeError} will be raised
if there are any values in @emph{iterable} that are not @ref{87e,,bytes-like objects}, including @ref{2ef,,str} objects.  The
separator between elements is the contents of the bytes or
bytearray object providing this method.
@end deffn

@geindex maketrans() (bytes static method)
@geindex maketrans() (bytearray static method)
@anchor{library/stdtypes bytes maketrans}@anchor{e23}
@deffn {Method} static bytes.maketrans (from, to)
@anchor{library/stdtypes bytearray maketrans}@anchor{e24}
@deffnx {Method} static bytearray.maketrans (from, to)

This static method returns a translation table usable for
@ref{1aca,,bytes.translate()} that will map each character in @emph{from} into the
character at the same position in @emph{to}; @emph{from} and @emph{to} must both be
@ref{87e,,bytes-like objects} and have the same length.


New in version 3.1.

@end deffn

@geindex partition() (bytes method)
@geindex partition() (bytearray method)
@anchor{library/stdtypes bytes partition}@anchor{1acb}
@deffn {Method} bytes.partition (sep)
@anchor{library/stdtypes bytearray partition}@anchor{1acc}
@deffnx {Method} bytearray.partition (sep)

Split the sequence at the first occurrence of @emph{sep}, and return a 3-tuple
containing the part before the separator, the separator itself or its
bytearray copy, and the part after the separator.
If the separator is not found, return a 3-tuple
containing a copy of the original sequence, followed by two empty bytes or
bytearray objects.

The separator to search for may be any @ref{87e,,bytes-like object}.
@end deffn

@geindex replace() (bytes method)
@geindex replace() (bytearray method)
@anchor{library/stdtypes bytes replace}@anchor{1acd}
@deffn {Method} bytes.replace (old, new[, count])
@anchor{library/stdtypes bytearray replace}@anchor{1ace}
@deffnx {Method} bytearray.replace (old, new[, count])

Return a copy of the sequence with all occurrences of subsequence @emph{old}
replaced by @emph{new}.  If the optional argument @emph{count} is given, only the
first @emph{count} occurrences are replaced.

The subsequence to search for and its replacement may be any
@ref{87e,,bytes-like object}.

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place - it
always produces a new object, even if no changes were made.
@end quotation
@end cartouche
@end deffn

@geindex rfind() (bytes method)
@geindex rfind() (bytearray method)
@anchor{library/stdtypes bytes rfind}@anchor{108a}
@deffn {Method} bytes.rfind (sub[, start[, end]])
@anchor{library/stdtypes bytearray rfind}@anchor{1acf}
@deffnx {Method} bytearray.rfind (sub[, start[, end]])

Return the highest index in the sequence where the subsequence @emph{sub} is
found, such that @emph{sub} is contained within @code{s[start:end]}.  Optional
arguments @emph{start} and @emph{end} are interpreted as in slice notation. Return
@code{-1} on failure.

The subsequence to search for may be any @ref{87e,,bytes-like object} or an
integer in the range 0 to 255.


Changed in version 3.3: Also accept an integer in the range 0 to 255 as the subsequence.

@end deffn

@geindex rindex() (bytes method)
@geindex rindex() (bytearray method)
@anchor{library/stdtypes bytes rindex}@anchor{1ad0}
@deffn {Method} bytes.rindex (sub[, start[, end]])
@anchor{library/stdtypes bytearray rindex}@anchor{1ad1}
@deffnx {Method} bytearray.rindex (sub[, start[, end]])

Like @ref{108a,,rfind()} but raises @ref{17c,,ValueError} when the
subsequence @emph{sub} is not found.

The subsequence to search for may be any @ref{87e,,bytes-like object} or an
integer in the range 0 to 255.


Changed in version 3.3: Also accept an integer in the range 0 to 255 as the subsequence.

@end deffn

@geindex rpartition() (bytes method)
@geindex rpartition() (bytearray method)
@anchor{library/stdtypes bytes rpartition}@anchor{1ad2}
@deffn {Method} bytes.rpartition (sep)
@anchor{library/stdtypes bytearray rpartition}@anchor{1ad3}
@deffnx {Method} bytearray.rpartition (sep)

Split the sequence at the last occurrence of @emph{sep}, and return a 3-tuple
containing the part before the separator, the separator itself or its
bytearray copy, and the part after the separator.
If the separator is not found, return a 3-tuple
containing two empty bytes or bytearray objects, followed by a copy of the
original sequence.

The separator to search for may be any @ref{87e,,bytes-like object}.
@end deffn

@geindex startswith() (bytes method)
@geindex startswith() (bytearray method)
@anchor{library/stdtypes bytes startswith}@anchor{1ad4}
@deffn {Method} bytes.startswith (prefix[, start[, end]])
@anchor{library/stdtypes bytearray startswith}@anchor{1ad5}
@deffnx {Method} bytearray.startswith (prefix[, start[, end]])

Return @code{True} if the binary data starts with the specified @emph{prefix},
otherwise return @code{False}.  @emph{prefix} can also be a tuple of prefixes to
look for.  With optional @emph{start}, test beginning at that position.  With
optional @emph{end}, stop comparing at that position.

The prefix(es) to search for may be any @ref{87e,,bytes-like object}.
@end deffn

@geindex translate() (bytes method)
@geindex translate() (bytearray method)
@anchor{library/stdtypes bytes translate}@anchor{1aca}
@deffn {Method} bytes.translate (table, /, delete=b'@w{'})
@anchor{library/stdtypes bytearray translate}@anchor{e98}
@deffnx {Method} bytearray.translate (table, /, delete=b'@w{'})

Return a copy of the bytes or bytearray object where all bytes occurring in
the optional argument @emph{delete} are removed, and the remaining bytes have
been mapped through the given translation table, which must be a bytes
object of length 256.

You can use the @ref{e23,,bytes.maketrans()} method to create a translation
table.

Set the @emph{table} argument to @code{None} for translations that only delete
characters:

@example
>>> b'read this short text'.translate(None, b'aeiou')
b'rd ths shrt txt'
@end example


Changed in version 3.6: @emph{delete} is now supported as a keyword argument.

@end deffn

The following methods on bytes and bytearray objects have default behaviours
that assume the use of ASCII compatible binary formats, but can still be used
with arbitrary binary data by passing appropriate arguments. Note that all of
the bytearray methods in this section do @emph{not} operate in place, and instead
produce new objects.

@geindex center() (bytes method)
@geindex center() (bytearray method)
@anchor{library/stdtypes bytes center}@anchor{1ad6}
@deffn {Method} bytes.center (width[, fillbyte])
@anchor{library/stdtypes bytearray center}@anchor{1ad7}
@deffnx {Method} bytearray.center (width[, fillbyte])

Return a copy of the object centered in a sequence of length @emph{width}.
Padding is done using the specified @emph{fillbyte} (default is an ASCII
space). For @ref{2f0,,bytes} objects, the original sequence is returned if
@emph{width} is less than or equal to @code{len(s)}.

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place -
it always produces a new object, even if no changes were made.
@end quotation
@end cartouche
@end deffn

@geindex ljust() (bytes method)
@geindex ljust() (bytearray method)
@anchor{library/stdtypes bytes ljust}@anchor{1ad8}
@deffn {Method} bytes.ljust (width[, fillbyte])
@anchor{library/stdtypes bytearray ljust}@anchor{1ad9}
@deffnx {Method} bytearray.ljust (width[, fillbyte])

Return a copy of the object left justified in a sequence of length @emph{width}.
Padding is done using the specified @emph{fillbyte} (default is an ASCII
space). For @ref{2f0,,bytes} objects, the original sequence is returned if
@emph{width} is less than or equal to @code{len(s)}.

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place -
it always produces a new object, even if no changes were made.
@end quotation
@end cartouche
@end deffn

@geindex lstrip() (bytes method)
@geindex lstrip() (bytearray method)
@anchor{library/stdtypes bytes lstrip}@anchor{1ada}
@deffn {Method} bytes.lstrip ([chars])
@anchor{library/stdtypes bytearray lstrip}@anchor{1adb}
@deffnx {Method} bytearray.lstrip ([chars])

Return a copy of the sequence with specified leading bytes removed.  The
@emph{chars} argument is a binary sequence specifying the set of byte values to
be removed - the name refers to the fact this method is usually used with
ASCII characters.  If omitted or @code{None}, the @emph{chars} argument defaults
to removing ASCII whitespace.  The @emph{chars} argument is not a prefix;
rather, all combinations of its values are stripped:

@example
>>> b'   spacious   '.lstrip()
b'spacious   '
>>> b'www.example.com'.lstrip(b'cmowz.')
b'example.com'
@end example

The binary sequence of byte values to remove may be any
@ref{87e,,bytes-like object}. See @ref{1abf,,removeprefix()} for a method
that will remove a single prefix string rather than all of a set of
characters.  For example:

@example
>>> b'Arthur: three!'.lstrip(b'Arthur: ')
b'ee!'
>>> b'Arthur: three!'.removeprefix(b'Arthur: ')
b'three!'
@end example

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place -
it always produces a new object, even if no changes were made.
@end quotation
@end cartouche
@end deffn

@geindex rjust() (bytes method)
@geindex rjust() (bytearray method)
@anchor{library/stdtypes bytes rjust}@anchor{1adc}
@deffn {Method} bytes.rjust (width[, fillbyte])
@anchor{library/stdtypes bytearray rjust}@anchor{1add}
@deffnx {Method} bytearray.rjust (width[, fillbyte])

Return a copy of the object right justified in a sequence of length @emph{width}.
Padding is done using the specified @emph{fillbyte} (default is an ASCII
space). For @ref{2f0,,bytes} objects, the original sequence is returned if
@emph{width} is less than or equal to @code{len(s)}.

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place -
it always produces a new object, even if no changes were made.
@end quotation
@end cartouche
@end deffn

@geindex rsplit() (bytes method)
@geindex rsplit() (bytearray method)
@anchor{library/stdtypes bytes rsplit}@anchor{1ade}
@deffn {Method} bytes.rsplit (sep=None, maxsplit=- 1)
@anchor{library/stdtypes bytearray rsplit}@anchor{1adf}
@deffnx {Method} bytearray.rsplit (sep=None, maxsplit=- 1)

Split the binary sequence into subsequences of the same type, using @emph{sep}
as the delimiter string. If @emph{maxsplit} is given, at most @emph{maxsplit} splits
are done, the @emph{rightmost} ones.  If @emph{sep} is not specified or @code{None},
any subsequence consisting solely of ASCII whitespace is a separator.
Except for splitting from the right, @ref{1adf,,rsplit()} behaves like
@ref{1ae0,,split()} which is described in detail below.
@end deffn

@geindex rstrip() (bytes method)
@geindex rstrip() (bytearray method)
@anchor{library/stdtypes bytes rstrip}@anchor{1ae1}
@deffn {Method} bytes.rstrip ([chars])
@anchor{library/stdtypes bytearray rstrip}@anchor{1ae2}
@deffnx {Method} bytearray.rstrip ([chars])

Return a copy of the sequence with specified trailing bytes removed.  The
@emph{chars} argument is a binary sequence specifying the set of byte values to
be removed - the name refers to the fact this method is usually used with
ASCII characters.  If omitted or @code{None}, the @emph{chars} argument defaults to
removing ASCII whitespace.  The @emph{chars} argument is not a suffix; rather,
all combinations of its values are stripped:

@example
>>> b'   spacious   '.rstrip()
b'   spacious'
>>> b'mississippi'.rstrip(b'ipz')
b'mississ'
@end example

The binary sequence of byte values to remove may be any
@ref{87e,,bytes-like object}. See @ref{1ac1,,removesuffix()} for a method
that will remove a single suffix string rather than all of a set of
characters.  For example:

@example
>>> b'Monty Python'.rstrip(b' Python')
b'M'
>>> b'Monty Python'.removesuffix(b' Python')
b'Monty'
@end example

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place -
it always produces a new object, even if no changes were made.
@end quotation
@end cartouche
@end deffn

@geindex split() (bytes method)
@geindex split() (bytearray method)
@anchor{library/stdtypes bytes split}@anchor{1ae3}
@deffn {Method} bytes.split (sep=None, maxsplit=- 1)
@anchor{library/stdtypes bytearray split}@anchor{1ae0}
@deffnx {Method} bytearray.split (sep=None, maxsplit=- 1)

Split the binary sequence into subsequences of the same type, using @emph{sep}
as the delimiter string. If @emph{maxsplit} is given and non-negative, at most
@emph{maxsplit} splits are done (thus, the list will have at most @code{maxsplit+1}
elements).  If @emph{maxsplit} is not specified or is @code{-1}, then there is no
limit on the number of splits (all possible splits are made).

If @emph{sep} is given, consecutive delimiters are not grouped together and are
deemed to delimit empty subsequences (for example, @code{b'1,,2'.split(b',')}
returns @code{[b'1', b'@w{'}, b'2']}).  The @emph{sep} argument may consist of a
multibyte sequence (for example, @code{b'1<>2<>3'.split(b'<>')} returns
@code{[b'1', b'2', b'3']}). Splitting an empty sequence with a specified
separator returns @code{[b'@w{'}]} or @code{[bytearray(b'@w{'})]} depending on the type
of object being split.  The @emph{sep} argument may be any
@ref{87e,,bytes-like object}.

For example:

@example
>>> b'1,2,3'.split(b',')
[b'1', b'2', b'3']
>>> b'1,2,3'.split(b',', maxsplit=1)
[b'1', b'2,3']
>>> b'1,2,,3,'.split(b',')
[b'1', b'2', b'@w{'}, b'3', b'@w{'}]
@end example

If @emph{sep} is not specified or is @code{None}, a different splitting algorithm
is applied: runs of consecutive ASCII whitespace are regarded as a single
separator, and the result will contain no empty strings at the start or
end if the sequence has leading or trailing whitespace.  Consequently,
splitting an empty sequence or a sequence consisting solely of ASCII
whitespace without a specified separator returns @code{[]}.

For example:

@example
>>> b'1 2 3'.split()
[b'1', b'2', b'3']
>>> b'1 2 3'.split(maxsplit=1)
[b'1', b'2 3']
>>> b'   1   2   3   '.split()
[b'1', b'2', b'3']
@end example
@end deffn

@geindex strip() (bytes method)
@geindex strip() (bytearray method)
@anchor{library/stdtypes bytes strip}@anchor{1ae4}
@deffn {Method} bytes.strip ([chars])
@anchor{library/stdtypes bytearray strip}@anchor{1ae5}
@deffnx {Method} bytearray.strip ([chars])

Return a copy of the sequence with specified leading and trailing bytes
removed. The @emph{chars} argument is a binary sequence specifying the set of
byte values to be removed - the name refers to the fact this method is
usually used with ASCII characters.  If omitted or @code{None}, the @emph{chars}
argument defaults to removing ASCII whitespace. The @emph{chars} argument is
not a prefix or suffix; rather, all combinations of its values are
stripped:

@example
>>> b'   spacious   '.strip()
b'spacious'
>>> b'www.example.com'.strip(b'cmowz.')
b'example'
@end example

The binary sequence of byte values to remove may be any
@ref{87e,,bytes-like object}.

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place -
it always produces a new object, even if no changes were made.
@end quotation
@end cartouche
@end deffn

The following methods on bytes and bytearray objects assume the use of ASCII
compatible binary formats and should not be applied to arbitrary binary data.
Note that all of the bytearray methods in this section do @emph{not} operate in
place, and instead produce new objects.

@geindex capitalize() (bytes method)
@geindex capitalize() (bytearray method)
@anchor{library/stdtypes bytes capitalize}@anchor{1ae6}
@deffn {Method} bytes.capitalize ()
@anchor{library/stdtypes bytearray capitalize}@anchor{1ae7}
@deffnx {Method} bytearray.capitalize ()

Return a copy of the sequence with each byte interpreted as an ASCII
character, and the first byte capitalized and the rest lowercased.
Non-ASCII byte values are passed through unchanged.

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place - it
always produces a new object, even if no changes were made.
@end quotation
@end cartouche
@end deffn

@geindex expandtabs() (bytes method)
@geindex expandtabs() (bytearray method)
@anchor{library/stdtypes bytes expandtabs}@anchor{1ae8}
@deffn {Method} bytes.expandtabs (tabsize=8)
@anchor{library/stdtypes bytearray expandtabs}@anchor{1ae9}
@deffnx {Method} bytearray.expandtabs (tabsize=8)

Return a copy of the sequence where all ASCII tab characters are replaced
by one or more ASCII spaces, depending on the current column and the given
tab size.  Tab positions occur every @emph{tabsize} bytes (default is 8,
giving tab positions at columns 0, 8, 16 and so on).  To expand the
sequence, the current column is set to zero and the sequence is examined
byte by byte.  If the byte is an ASCII tab character (@code{b'\t'}), one or
more space characters are inserted in the result until the current column
is equal to the next tab position. (The tab character itself is not
copied.)  If the current byte is an ASCII newline (@code{b'\n'}) or
carriage return (@code{b'\r'}), it is copied and the current column is reset
to zero.  Any other byte value is copied unchanged and the current column
is incremented by one regardless of how the byte value is represented when
printed:

@example
>>> b'01\t012\t0123\t01234'.expandtabs()
b'01      012     0123    01234'
>>> b'01\t012\t0123\t01234'.expandtabs(4)
b'01  012 0123    01234'
@end example

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place - it
always produces a new object, even if no changes were made.
@end quotation
@end cartouche
@end deffn

@geindex isalnum() (bytes method)
@geindex isalnum() (bytearray method)
@anchor{library/stdtypes bytes isalnum}@anchor{1aea}
@deffn {Method} bytes.isalnum ()
@anchor{library/stdtypes bytearray isalnum}@anchor{1aeb}
@deffnx {Method} bytearray.isalnum ()

Return @code{True} if all bytes in the sequence are alphabetical ASCII characters
or ASCII decimal digits and the sequence is not empty, @code{False} otherwise.
Alphabetic ASCII characters are those byte values in the sequence
@code{b'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'}. ASCII decimal
digits are those byte values in the sequence @code{b'0123456789'}.

For example:

@example
>>> b'ABCabc1'.isalnum()
True
>>> b'ABC abc1'.isalnum()
False
@end example
@end deffn

@geindex isalpha() (bytes method)
@geindex isalpha() (bytearray method)
@anchor{library/stdtypes bytes isalpha}@anchor{1aec}
@deffn {Method} bytes.isalpha ()
@anchor{library/stdtypes bytearray isalpha}@anchor{1aed}
@deffnx {Method} bytearray.isalpha ()

Return @code{True} if all bytes in the sequence are alphabetic ASCII characters
and the sequence is not empty, @code{False} otherwise.  Alphabetic ASCII
characters are those byte values in the sequence
@code{b'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'}.

For example:

@example
>>> b'ABCabc'.isalpha()
True
>>> b'ABCabc1'.isalpha()
False
@end example
@end deffn

@geindex isascii() (bytes method)
@geindex isascii() (bytearray method)
@anchor{library/stdtypes bytes isascii}@anchor{1aee}
@deffn {Method} bytes.isascii ()
@anchor{library/stdtypes bytearray isascii}@anchor{1aef}
@deffnx {Method} bytearray.isascii ()

Return @code{True} if the sequence is empty or all bytes in the sequence are ASCII,
@code{False} otherwise.
ASCII bytes are in the range 0-0x7F.


New in version 3.7.

@end deffn

@geindex isdigit() (bytes method)
@geindex isdigit() (bytearray method)
@anchor{library/stdtypes bytes isdigit}@anchor{1af0}
@deffn {Method} bytes.isdigit ()
@anchor{library/stdtypes bytearray isdigit}@anchor{1af1}
@deffnx {Method} bytearray.isdigit ()

Return @code{True} if all bytes in the sequence are ASCII decimal digits
and the sequence is not empty, @code{False} otherwise. ASCII decimal digits are
those byte values in the sequence @code{b'0123456789'}.

For example:

@example
>>> b'1234'.isdigit()
True
>>> b'1.23'.isdigit()
False
@end example
@end deffn

@geindex islower() (bytes method)
@geindex islower() (bytearray method)
@anchor{library/stdtypes bytes islower}@anchor{1af2}
@deffn {Method} bytes.islower ()
@anchor{library/stdtypes bytearray islower}@anchor{1af3}
@deffnx {Method} bytearray.islower ()

Return @code{True} if there is at least one lowercase ASCII character
in the sequence and no uppercase ASCII characters, @code{False} otherwise.

For example:

@example
>>> b'hello world'.islower()
True
>>> b'Hello world'.islower()
False
@end example

Lowercase ASCII characters are those byte values in the sequence
@code{b'abcdefghijklmnopqrstuvwxyz'}. Uppercase ASCII characters
are those byte values in the sequence @code{b'ABCDEFGHIJKLMNOPQRSTUVWXYZ'}.
@end deffn

@geindex isspace() (bytes method)
@geindex isspace() (bytearray method)
@anchor{library/stdtypes bytes isspace}@anchor{1af4}
@deffn {Method} bytes.isspace ()
@anchor{library/stdtypes bytearray isspace}@anchor{1af5}
@deffnx {Method} bytearray.isspace ()

Return @code{True} if all bytes in the sequence are ASCII whitespace and the
sequence is not empty, @code{False} otherwise.  ASCII whitespace characters are
those byte values in the sequence @code{b' \t\n\r\x0b\f'} (space, tab, newline,
carriage return, vertical tab, form feed).
@end deffn

@geindex istitle() (bytes method)
@geindex istitle() (bytearray method)
@anchor{library/stdtypes bytes istitle}@anchor{1af6}
@deffn {Method} bytes.istitle ()
@anchor{library/stdtypes bytearray istitle}@anchor{1af7}
@deffnx {Method} bytearray.istitle ()

Return @code{True} if the sequence is ASCII titlecase and the sequence is not
empty, @code{False} otherwise. See @ref{1af8,,bytes.title()} for more details on the
definition of “titlecase”.

For example:

@example
>>> b'Hello World'.istitle()
True
>>> b'Hello world'.istitle()
False
@end example
@end deffn

@geindex isupper() (bytes method)
@geindex isupper() (bytearray method)
@anchor{library/stdtypes bytes isupper}@anchor{1af9}
@deffn {Method} bytes.isupper ()
@anchor{library/stdtypes bytearray isupper}@anchor{1afa}
@deffnx {Method} bytearray.isupper ()

Return @code{True} if there is at least one uppercase alphabetic ASCII character
in the sequence and no lowercase ASCII characters, @code{False} otherwise.

For example:

@example
>>> b'HELLO WORLD'.isupper()
True
>>> b'Hello world'.isupper()
False
@end example

Lowercase ASCII characters are those byte values in the sequence
@code{b'abcdefghijklmnopqrstuvwxyz'}. Uppercase ASCII characters
are those byte values in the sequence @code{b'ABCDEFGHIJKLMNOPQRSTUVWXYZ'}.
@end deffn

@geindex lower() (bytes method)
@geindex lower() (bytearray method)
@anchor{library/stdtypes bytes lower}@anchor{1afb}
@deffn {Method} bytes.lower ()
@anchor{library/stdtypes bytearray lower}@anchor{1afc}
@deffnx {Method} bytearray.lower ()

Return a copy of the sequence with all the uppercase ASCII characters
converted to their corresponding lowercase counterpart.

For example:

@example
>>> b'Hello World'.lower()
b'hello world'
@end example

Lowercase ASCII characters are those byte values in the sequence
@code{b'abcdefghijklmnopqrstuvwxyz'}. Uppercase ASCII characters
are those byte values in the sequence @code{b'ABCDEFGHIJKLMNOPQRSTUVWXYZ'}.

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place - it
always produces a new object, even if no changes were made.
@end quotation
@end cartouche
@end deffn

@geindex universal newlines; bytes.splitlines method
@geindex universal newlines; bytearray.splitlines method

@geindex splitlines() (bytes method)
@geindex splitlines() (bytearray method)
@anchor{library/stdtypes bytes splitlines}@anchor{1afd}
@deffn {Method} bytes.splitlines (keepends=False)
@anchor{library/stdtypes bytearray splitlines}@anchor{1afe}
@deffnx {Method} bytearray.splitlines (keepends=False)

Return a list of the lines in the binary sequence, breaking at ASCII
line boundaries. This method uses the @ref{88a,,universal newlines} approach
to splitting lines. Line breaks are not included in the resulting list
unless @emph{keepends} is given and true.

For example:

@example
>>> b'ab c\n\nde fg\rkl\r\n'.splitlines()
[b'ab c', b'@w{'}, b'de fg', b'kl']
>>> b'ab c\n\nde fg\rkl\r\n'.splitlines(keepends=True)
[b'ab c\n', b'\n', b'de fg\r', b'kl\r\n']
@end example

Unlike @ref{1ae3,,split()} when a delimiter string @emph{sep} is given, this
method returns an empty list for the empty string, and a terminal line
break does not result in an extra line:

@example
>>> b"".split(b'\n'), b"Two lines\n".split(b'\n')
([b'@w{'}], [b'Two lines', b'@w{'}])
>>> b"".splitlines(), b"One line\n".splitlines()
([], [b'One line'])
@end example
@end deffn

@geindex swapcase() (bytes method)
@geindex swapcase() (bytearray method)
@anchor{library/stdtypes bytes swapcase}@anchor{1aff}
@deffn {Method} bytes.swapcase ()
@anchor{library/stdtypes bytearray swapcase}@anchor{1b00}
@deffnx {Method} bytearray.swapcase ()

Return a copy of the sequence with all the lowercase ASCII characters
converted to their corresponding uppercase counterpart and vice-versa.

For example:

@example
>>> b'Hello World'.swapcase()
b'hELLO wORLD'
@end example

Lowercase ASCII characters are those byte values in the sequence
@code{b'abcdefghijklmnopqrstuvwxyz'}. Uppercase ASCII characters
are those byte values in the sequence @code{b'ABCDEFGHIJKLMNOPQRSTUVWXYZ'}.

Unlike @ref{1ab3,,str.swapcase()}, it is always the case that
@code{bin.swapcase().swapcase() == bin} for the binary versions. Case
conversions are symmetrical in ASCII, even though that is not generally
true for arbitrary Unicode code points.

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place - it
always produces a new object, even if no changes were made.
@end quotation
@end cartouche
@end deffn

@geindex title() (bytes method)
@geindex title() (bytearray method)
@anchor{library/stdtypes bytes title}@anchor{1af8}
@deffn {Method} bytes.title ()
@anchor{library/stdtypes bytearray title}@anchor{1b01}
@deffnx {Method} bytearray.title ()

Return a titlecased version of the binary sequence where words start with
an uppercase ASCII character and the remaining characters are lowercase.
Uncased byte values are left unmodified.

For example:

@example
>>> b'Hello world'.title()
b'Hello World'
@end example

Lowercase ASCII characters are those byte values in the sequence
@code{b'abcdefghijklmnopqrstuvwxyz'}. Uppercase ASCII characters
are those byte values in the sequence @code{b'ABCDEFGHIJKLMNOPQRSTUVWXYZ'}.
All other byte values are uncased.

The algorithm uses a simple language-independent definition of a word as
groups of consecutive letters.  The definition works in many contexts but
it means that apostrophes in contractions and possessives form word
boundaries, which may not be the desired result:

@example
>>> b"they're bill's friends from the UK".title()
b"They'Re Bill'S Friends From The Uk"
@end example

A workaround for apostrophes can be constructed using regular expressions:

@example
>>> import re
>>> def titlecase(s):
...     return re.sub(rb"[A-Za-z]+('[A-Za-z]+)?",
...                   lambda mo: mo.group(0)[0:1].upper() +
...                              mo.group(0)[1:].lower(),
...                   s)
...
>>> titlecase(b"they're bill's friends.")
b"They're Bill's Friends."
@end example

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place - it
always produces a new object, even if no changes were made.
@end quotation
@end cartouche
@end deffn

@geindex upper() (bytes method)
@geindex upper() (bytearray method)
@anchor{library/stdtypes bytes upper}@anchor{1b02}
@deffn {Method} bytes.upper ()
@anchor{library/stdtypes bytearray upper}@anchor{1b03}
@deffnx {Method} bytearray.upper ()

Return a copy of the sequence with all the lowercase ASCII characters
converted to their corresponding uppercase counterpart.

For example:

@example
>>> b'Hello World'.upper()
b'HELLO WORLD'
@end example

Lowercase ASCII characters are those byte values in the sequence
@code{b'abcdefghijklmnopqrstuvwxyz'}. Uppercase ASCII characters
are those byte values in the sequence @code{b'ABCDEFGHIJKLMNOPQRSTUVWXYZ'}.

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place - it
always produces a new object, even if no changes were made.
@end quotation
@end cartouche
@end deffn

@geindex zfill() (bytes method)
@geindex zfill() (bytearray method)
@anchor{library/stdtypes bytes zfill}@anchor{1b04}
@deffn {Method} bytes.zfill (width)
@anchor{library/stdtypes bytearray zfill}@anchor{1b05}
@deffnx {Method} bytearray.zfill (width)

Return a copy of the sequence left filled with ASCII @code{b'0'} digits to
make a sequence of length @emph{width}. A leading sign prefix (@code{b'+'}/
@code{b'-'}) is handled by inserting the padding @emph{after} the sign character
rather than before. For @ref{2f0,,bytes} objects, the original sequence is
returned if @emph{width} is less than or equal to @code{len(seq)}.

For example:

@example
>>> b"42".zfill(5)
b'00042'
>>> b"-42".zfill(5)
b'-0042'
@end example

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place - it
always produces a new object, even if no changes were made.
@end quotation
@end cartouche
@end deffn

@node printf-style Bytes Formatting,Memory Views,Bytes and Bytearray Operations,Binary Sequence Types — bytes bytearray memoryview
@anchor{library/stdtypes bytes-formatting}@anchor{8da}@anchor{library/stdtypes printf-style-bytes-formatting}@anchor{1b06}
@subsubsection @code{printf}-style Bytes Formatting


@geindex formatting; bytes (%)
@geindex formatting; bytearray (%)
@geindex interpolation; bytes (%)
@geindex interpolation; bytearray (%)
@geindex bytes; formatting
@geindex bytearray; formatting
@geindex bytes; interpolation
@geindex bytearray; interpolation
@geindex printf-style formatting
@geindex sprintf-style formatting
@geindex % (percent); printf-style formatting

@cartouche
@quotation Note 
The formatting operations described here exhibit a variety of quirks that
lead to a number of common errors (such as failing to display tuples and
dictionaries correctly).  If the value being printed may be a tuple or
dictionary, wrap it in a tuple.
@end quotation
@end cartouche

Bytes objects (@code{bytes}/@code{bytearray}) have one unique built-in operation:
the @code{%} operator (modulo).
This is also known as the bytes @emph{formatting} or @emph{interpolation} operator.
Given @code{format % values} (where @emph{format} is a bytes object), @code{%} conversion
specifications in @emph{format} are replaced with zero or more elements of @emph{values}.
The effect is similar to using the @code{sprintf()} in the C language.

If @emph{format} requires a single argument, @emph{values} may be a single non-tuple
object. @footnote{@w{(5)} 
To format only a tuple you should therefore provide a singleton tuple whose only
element is the tuple to be formatted.
}  Otherwise, @emph{values} must be a tuple with exactly the number of
items specified by the format bytes object, or a single mapping object (for
example, a dictionary).

@geindex () (parentheses); in printf-style formatting
@geindex * (asterisk); in printf-style formatting
@geindex . (dot); in printf-style formatting

A conversion specifier contains two or more characters and has the following
components, which must occur in this order:


@enumerate 

@item 
The @code{'%'} character, which marks the start of the specifier.

@item 
Mapping key (optional), consisting of a parenthesised sequence of characters
(for example, @code{(somename)}).

@item 
Conversion flags (optional), which affect the result of some conversion
types.

@item 
Minimum field width (optional).  If specified as an @code{'*'} (asterisk), the
actual width is read from the next element of the tuple in @emph{values}, and the
object to convert comes after the minimum field width and optional precision.

@item 
Precision (optional), given as a @code{'.'} (dot) followed by the precision.  If
specified as @code{'*'} (an asterisk), the actual precision is read from the next
element of the tuple in @emph{values}, and the value to convert comes after the
precision.

@item 
Length modifier (optional).

@item 
Conversion type.
@end enumerate

When the right argument is a dictionary (or other mapping type), then the
formats in the bytes object @emph{must} include a parenthesised mapping key into that
dictionary inserted immediately after the @code{'%'} character. The mapping key
selects the value to be formatted from the mapping.  For example:

@example
>>> print(b'%(language)s has %(number)03d quote types.' %
...       @{b'language': b"Python", b"number": 2@})
b'Python has 002 quote types.'
@end example

In this case no @code{*} specifiers may occur in a format (since they require a
sequential parameter list).

The conversion flag characters are:

@geindex # (hash); in printf-style formatting
@geindex - (minus); in printf-style formatting
@geindex + (plus); in printf-style formatting
@geindex space; in printf-style formatting


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Flag

@tab

Meaning

@item

@code{'#'}

@tab

The value conversion will use the “alternate form” (where defined
below).

@item

@code{'0'}

@tab

The conversion will be zero padded for numeric values.

@item

@code{'-'}

@tab

The converted value is left adjusted (overrides the @code{'0'}
conversion if both are given).

@item

@code{' '}

@tab

(a space) A blank should be left before a positive number (or empty
string) produced by a signed conversion.

@item

@code{'+'}

@tab

A sign character (@code{'+'} or @code{'-'}) will precede the conversion
(overrides a “space” flag).

@end multitable


A length modifier (@code{h}, @code{l}, or @code{L}) may be present, but is ignored as it
is not necessary for Python – so e.g. @code{%ld} is identical to @code{%d}.

The conversion types are:


@multitable {xxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} 
@headitem

Conversion

@tab

Meaning

@tab

Notes

@item

@code{'d'}

@tab

Signed integer decimal.

@tab

@item

@code{'i'}

@tab

Signed integer decimal.

@tab

@item

@code{'o'}

@tab

Signed octal value.

@tab

(1)

@item

@code{'u'}

@tab

Obsolete type – it is identical to @code{'d'}.

@tab

(8)

@item

@code{'x'}

@tab

Signed hexadecimal (lowercase).

@tab

(2)

@item

@code{'X'}

@tab

Signed hexadecimal (uppercase).

@tab

(2)

@item

@code{'e'}

@tab

Floating point exponential format (lowercase).

@tab

(3)

@item

@code{'E'}

@tab

Floating point exponential format (uppercase).

@tab

(3)

@item

@code{'f'}

@tab

Floating point decimal format.

@tab

(3)

@item

@code{'F'}

@tab

Floating point decimal format.

@tab

(3)

@item

@code{'g'}

@tab

Floating point format. Uses lowercase exponential
format if exponent is less than -4 or not less than
precision, decimal format otherwise.

@tab

(4)

@item

@code{'G'}

@tab

Floating point format. Uses uppercase exponential
format if exponent is less than -4 or not less than
precision, decimal format otherwise.

@tab

(4)

@item

@code{'c'}

@tab

Single byte (accepts integer or single
byte objects).

@tab

@item

@code{'b'}

@tab

Bytes (any object that follows the
@ref{1a3e,,buffer protocol} or has
@code{__bytes__()}).

@tab

(5)

@item

@code{'s'}

@tab

@code{'s'} is an alias for @code{'b'} and should only
be used for Python2/3 code bases.

@tab

(6)

@item

@code{'a'}

@tab

Bytes (converts any Python object using
@code{repr(obj).encode('ascii','backslashreplace)}).

@tab

(5)

@item

@code{'r'}

@tab

@code{'r'} is an alias for @code{'a'} and should only
be used for Python2/3 code bases.

@tab

(7)

@item

@code{'%'}

@tab

No argument is converted, results in a @code{'%'}
character in the result.

@tab

@end multitable


Notes:


@enumerate 

@item 
The alternate form causes a leading octal specifier (@code{'0o'}) to be
inserted before the first digit.

@item 
The alternate form causes a leading @code{'0x'} or @code{'0X'} (depending on whether
the @code{'x'} or @code{'X'} format was used) to be inserted before the first digit.

@item 
The alternate form causes the result to always contain a decimal point, even if
no digits follow it.

The precision determines the number of digits after the decimal point and
defaults to 6.

@item 
The alternate form causes the result to always contain a decimal point, and
trailing zeroes are not removed as they would otherwise be.

The precision determines the number of significant digits before and after the
decimal point and defaults to 6.

@item 
If precision is @code{N}, the output is truncated to @code{N} characters.

@item 
@code{b'%s'} is deprecated, but will not be removed during the 3.x series.

@item 
@code{b'%r'} is deprecated, but will not be removed during the 3.x series.

@item 
See 
@geindex Python Enhancement Proposals; PEP 237
PEP 237@footnote{https://www.python.org/dev/peps/pep-0237}.
@end enumerate

@cartouche
@quotation Note 
The bytearray version of this method does @emph{not} operate in place - it
always produces a new object, even if no changes were made.
@end quotation
@end cartouche


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 461
PEP 461@footnote{https://www.python.org/dev/peps/pep-0461} - Adding % formatting to bytes and bytearray



New in version 3.5.


@node Memory Views,,printf-style Bytes Formatting,Binary Sequence Types — bytes bytearray memoryview
@anchor{library/stdtypes memory-views}@anchor{1b07}@anchor{library/stdtypes typememoryview}@anchor{1a53}
@subsubsection Memory Views


@ref{573,,memoryview} objects allow Python code to access the internal data
of an object that supports the @ref{1a3e,,buffer protocol} without
copying.

@geindex memoryview (built-in class)
@anchor{library/stdtypes memoryview}@anchor{573}
@deffn {Class} memoryview (object)

Create a @ref{573,,memoryview} that references @emph{object}.  @emph{object} must
support the buffer protocol.  Built-in objects that support the buffer
protocol include @ref{2f0,,bytes} and @ref{2f1,,bytearray}.

A @ref{573,,memoryview} has the notion of an @emph{element}, which is the
atomic memory unit handled by the originating @emph{object}.  For many simple
types such as @ref{2f0,,bytes} and @ref{2f1,,bytearray}, an element is a single
byte, but other types such as @ref{25e,,array.array} may have bigger elements.

@code{len(view)} is equal to the length of @ref{1b08,,tolist}.
If @code{view.ndim = 0}, the length is 1. If @code{view.ndim = 1}, the length
is equal to the number of elements in the view. For higher dimensions,
the length is equal to the length of the nested list representation of
the view. The @ref{1b09,,itemsize} attribute will give you the
number of bytes in a single element.

A @ref{573,,memoryview} supports slicing and indexing to expose its data.
One-dimensional slicing will result in a subview:

@example
>>> v = memoryview(b'abcefg')
>>> v[1]
98
>>> v[-1]
103
>>> v[1:4]
<memory at 0x7f3ddc9f4350>
>>> bytes(v[1:4])
b'bce'
@end example

If @ref{1b0a,,format} is one of the native format specifiers
from the @ref{f3,,struct} module, indexing with an integer or a tuple of
integers is also supported and returns a single @emph{element} with
the correct type.  One-dimensional memoryviews can be indexed
with an integer or a one-integer tuple.  Multi-dimensional memoryviews
can be indexed with tuples of exactly @emph{ndim} integers where @emph{ndim} is
the number of dimensions.  Zero-dimensional memoryviews can be indexed
with the empty tuple.

Here is an example with a non-byte format:

@example
>>> import array
>>> a = array.array('l', [-11111111, 22222222, -33333333, 44444444])
>>> m = memoryview(a)
>>> m[0]
-11111111
>>> m[-1]
44444444
>>> m[::2].tolist()
[-11111111, -33333333]
@end example

If the underlying object is writable, the memoryview supports
one-dimensional slice assignment. Resizing is not allowed:

@example
>>> data = bytearray(b'abcefg')
>>> v = memoryview(data)
>>> v.readonly
False
>>> v[0] = ord(b'z')
>>> data
bytearray(b'zbcefg')
>>> v[1:4] = b'123'
>>> data
bytearray(b'z123fg')
>>> v[2:3] = b'spam'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview assignment: lvalue and rvalue have different structures
>>> v[2:6] = b'spam'
>>> data
bytearray(b'z1spam')
@end example

One-dimensional memoryviews of hashable (read-only) types with formats
‘B’, ‘b’ or ‘c’ are also hashable. The hash is defined as
@code{hash(m) == hash(m.tobytes())}:

@example
>>> v = memoryview(b'abcefg')
>>> hash(v) == hash(b'abcefg')
True
>>> hash(v[2:4]) == hash(b'ce')
True
>>> hash(v[::-2]) == hash(b'abcefg'[::-2])
True
@end example


Changed in version 3.3: One-dimensional memoryviews can now be sliced.
One-dimensional memoryviews with formats ‘B’, ‘b’ or ‘c’ are now hashable.



Changed in version 3.4: memoryview is now registered automatically with
@ref{110c,,collections.abc.Sequence}



Changed in version 3.5: memoryviews can now be indexed with tuple of integers.


@ref{573,,memoryview} has several methods:

@geindex __eq__() (memoryview method)
@anchor{library/stdtypes memoryview __eq__}@anchor{1b0b}
@deffn {Method} __eq__ (exporter)

A memoryview and a 
@geindex Python Enhancement Proposals; PEP 3118
PEP 3118@footnote{https://www.python.org/dev/peps/pep-3118} exporter are equal if their shapes are
equivalent and if all corresponding values are equal when the operands’
respective format codes are interpreted using @ref{f3,,struct} syntax.

For the subset of @ref{f3,,struct} format strings currently supported by
@ref{1b08,,tolist()}, @code{v} and @code{w} are equal if @code{v.tolist() == w.tolist()}:

@example
>>> import array
>>> a = array.array('I', [1, 2, 3, 4, 5])
>>> b = array.array('d', [1.0, 2.0, 3.0, 4.0, 5.0])
>>> c = array.array('b', [5, 3, 1])
>>> x = memoryview(a)
>>> y = memoryview(b)
>>> x == a == y == b
True
>>> x.tolist() == a.tolist() == y.tolist() == b.tolist()
True
>>> z = y[::-2]
>>> z == c
True
>>> z.tolist() == c.tolist()
True
@end example

If either format string is not supported by the @ref{f3,,struct} module,
then the objects will always compare as unequal (even if the format
strings and buffer contents are identical):

@example
>>> from ctypes import BigEndianStructure, c_long
>>> class BEPoint(BigEndianStructure):
...     _fields_ = [("x", c_long), ("y", c_long)]
...
>>> point = BEPoint(100, 200)
>>> a = memoryview(point)
>>> b = memoryview(point)
>>> a == point
False
>>> a == b
False
@end example

Note that, as with floating point numbers, @code{v is w} does @emph{not} imply
@code{v == w} for memoryview objects.


Changed in version 3.3: Previous versions compared the raw memory disregarding the item format
and the logical array structure.

@end deffn

@geindex tobytes() (memoryview method)
@anchor{library/stdtypes memoryview tobytes}@anchor{1b0c}
@deffn {Method} tobytes (order='C')

Return the data in the buffer as a bytestring.  This is equivalent to
calling the @ref{2f0,,bytes} constructor on the memoryview.

@example
>>> m = memoryview(b"abc")
>>> m.tobytes()
b'abc'
>>> bytes(m)
b'abc'
@end example

For non-contiguous arrays the result is equal to the flattened list
representation with all elements converted to bytes. @ref{1b0c,,tobytes()}
supports all format strings, including those that are not in
@ref{f3,,struct} module syntax.


New in version 3.8: @emph{order} can be @{‘C’, ‘F’, ‘A’@}.  When @emph{order} is ‘C’ or ‘F’, the data
of the original array is converted to C or Fortran order. For contiguous
views, ‘A’ returns an exact copy of the physical memory. In particular,
in-memory Fortran order is preserved. For non-contiguous views, the
data is converted to C first. @emph{order=None} is the same as @emph{order=’C’}.

@end deffn

@geindex hex() (memoryview method)
@anchor{library/stdtypes memoryview hex}@anchor{8c3}
@deffn {Method} hex ([sep[, bytes_per_sep]])

Return a string object containing two hexadecimal digits for each
byte in the buffer.

@example
>>> m = memoryview(b"abc")
>>> m.hex()
'616263'
@end example


New in version 3.5.



Changed in version 3.8: Similar to @ref{8c1,,bytes.hex()}, @ref{8c3,,memoryview.hex()} now supports
optional @emph{sep} and @emph{bytes_per_sep} parameters to insert separators
between bytes in the hex output.

@end deffn

@geindex tolist() (memoryview method)
@anchor{library/stdtypes memoryview tolist}@anchor{1b08}
@deffn {Method} tolist ()

Return the data in the buffer as a list of elements.

@example
>>> memoryview(b'abc').tolist()
[97, 98, 99]
>>> import array
>>> a = array.array('d', [1.1, 2.2, 3.3])
>>> m = memoryview(a)
>>> m.tolist()
[1.1, 2.2, 3.3]
@end example


Changed in version 3.3: @ref{1b08,,tolist()} now supports all single character native formats in
@ref{f3,,struct} module syntax as well as multi-dimensional
representations.

@end deffn

@geindex toreadonly() (memoryview method)
@anchor{library/stdtypes memoryview toreadonly}@anchor{1b0d}
@deffn {Method} toreadonly ()

Return a readonly version of the memoryview object.  The original
memoryview object is unchanged.

@example
>>> m = memoryview(bytearray(b'abc'))
>>> mm = m.toreadonly()
>>> mm.tolist()
[89, 98, 99]
>>> mm[0] = 42
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot modify read-only memory
>>> m[0] = 43
>>> mm.tolist()
[43, 98, 99]
@end example


New in version 3.8.

@end deffn

@geindex release() (memoryview method)
@anchor{library/stdtypes memoryview release}@anchor{d64}
@deffn {Method} release ()

Release the underlying buffer exposed by the memoryview object.  Many
objects take special actions when a view is held on them (for example,
a @ref{2f1,,bytearray} would temporarily forbid resizing); therefore,
calling release() is handy to remove these restrictions (and free any
dangling resources) as soon as possible.

After this method has been called, any further operation on the view
raises a @ref{17c,,ValueError} (except @ref{d64,,release()} itself which can
be called multiple times):

@example
>>> m = memoryview(b'abc')
>>> m.release()
>>> m[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: operation forbidden on released memoryview object
@end example

The context management protocol can be used for a similar effect,
using the @code{with} statement:

@example
>>> with memoryview(b'abc') as m:
...     m[0]
...
97
>>> m[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: operation forbidden on released memoryview object
@end example


New in version 3.2.

@end deffn

@geindex cast() (memoryview method)
@anchor{library/stdtypes memoryview cast}@anchor{1b0e}
@deffn {Method} cast (format[, shape])

Cast a memoryview to a new format or shape. @emph{shape} defaults to
@code{[byte_length//new_itemsize]}, which means that the result view
will be one-dimensional. The return value is a new memoryview, but
the buffer itself is not copied. Supported casts are 1D -> C-@ref{1b0f,,contiguous}
and C-contiguous -> 1D.

The destination format is restricted to a single element native format in
@ref{f3,,struct} syntax. One of the formats must be a byte format
(‘B’, ‘b’ or ‘c’). The byte length of the result must be the same
as the original length.

Cast 1D/long to 1D/unsigned bytes:

@example
>>> import array
>>> a = array.array('l', [1,2,3])
>>> x = memoryview(a)
>>> x.format
'l'
>>> x.itemsize
8
>>> len(x)
3
>>> x.nbytes
24
>>> y = x.cast('B')
>>> y.format
'B'
>>> y.itemsize
1
>>> len(y)
24
>>> y.nbytes
24
@end example

Cast 1D/unsigned bytes to 1D/char:

@example
>>> b = bytearray(b'zyz')
>>> x = memoryview(b)
>>> x[0] = b'a'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: invalid value for format "B"
>>> y = x.cast('c')
>>> y[0] = b'a'
>>> b
bytearray(b'ayz')
@end example

Cast 1D/bytes to 3D/ints to 1D/signed char:

@example
>>> import struct
>>> buf = struct.pack("i"*12, *list(range(12)))
>>> x = memoryview(buf)
>>> y = x.cast('i', shape=[2,2,3])
>>> y.tolist()
[[[0, 1, 2], [3, 4, 5]], [[6, 7, 8], [9, 10, 11]]]
>>> y.format
'i'
>>> y.itemsize
4
>>> len(y)
2
>>> y.nbytes
48
>>> z = y.cast('b')
>>> z.format
'b'
>>> z.itemsize
1
>>> len(z)
48
>>> z.nbytes
48
@end example

Cast 1D/unsigned long to 2D/unsigned long:

@example
>>> buf = struct.pack("L"*6, *list(range(6)))
>>> x = memoryview(buf)
>>> y = x.cast('L', shape=[2,3])
>>> len(y)
2
>>> y.nbytes
48
>>> y.tolist()
[[0, 1, 2], [3, 4, 5]]
@end example


New in version 3.3.



Changed in version 3.5: The source format is no longer restricted when casting to a byte view.

@end deffn

There are also several readonly attributes available:

@geindex obj (memoryview attribute)
@anchor{library/stdtypes memoryview obj}@anchor{1b10}
@deffn {Attribute} obj

The underlying object of the memoryview:

@example
>>> b  = bytearray(b'xyz')
>>> m = memoryview(b)
>>> m.obj is b
True
@end example


New in version 3.3.

@end deffn

@geindex nbytes (memoryview attribute)
@anchor{library/stdtypes memoryview nbytes}@anchor{1b11}
@deffn {Attribute} nbytes

@code{nbytes == product(shape) * itemsize == len(m.tobytes())}. This is
the amount of space in bytes that the array would use in a contiguous
representation. It is not necessarily equal to @code{len(m)}:

@example
>>> import array
>>> a = array.array('i', [1,2,3,4,5])
>>> m = memoryview(a)
>>> len(m)
5
>>> m.nbytes
20
>>> y = m[::2]
>>> len(y)
3
>>> y.nbytes
12
>>> len(y.tobytes())
12
@end example

Multi-dimensional arrays:

@example
>>> import struct
>>> buf = struct.pack("d"*12, *[1.5*x for x in range(12)])
>>> x = memoryview(buf)
>>> y = x.cast('d', shape=[3,4])
>>> y.tolist()
[[0.0, 1.5, 3.0, 4.5], [6.0, 7.5, 9.0, 10.5], [12.0, 13.5, 15.0, 16.5]]
>>> len(y)
3
>>> y.nbytes
96
@end example


New in version 3.3.

@end deffn

@geindex readonly (memoryview attribute)
@anchor{library/stdtypes memoryview readonly}@anchor{1b12}
@deffn {Attribute} readonly

A bool indicating whether the memory is read only.
@end deffn

@geindex format (memoryview attribute)
@anchor{library/stdtypes memoryview format}@anchor{1b0a}
@deffn {Attribute} format

A string containing the format (in @ref{f3,,struct} module style) for each
element in the view. A memoryview can be created from exporters with
arbitrary format strings, but some methods (e.g. @ref{1b08,,tolist()}) are
restricted to native single element formats.


Changed in version 3.3: format @code{'B'} is now handled according to the struct module syntax.
This means that @code{memoryview(b'abc')[0] == b'abc'[0] == 97}.

@end deffn

@geindex itemsize (memoryview attribute)
@anchor{library/stdtypes memoryview itemsize}@anchor{1b09}
@deffn {Attribute} itemsize

The size in bytes of each element of the memoryview:

@example
>>> import array, struct
>>> m = memoryview(array.array('H', [32000, 32001, 32002]))
>>> m.itemsize
2
>>> m[0]
32000
>>> struct.calcsize('H') == m.itemsize
True
@end example
@end deffn

@geindex ndim (memoryview attribute)
@anchor{library/stdtypes memoryview ndim}@anchor{1b13}
@deffn {Attribute} ndim

An integer indicating how many dimensions of a multi-dimensional array the
memory represents.
@end deffn

@geindex shape (memoryview attribute)
@anchor{library/stdtypes memoryview shape}@anchor{1b14}
@deffn {Attribute} shape

A tuple of integers the length of @ref{1b13,,ndim} giving the shape of the
memory as an N-dimensional array.


Changed in version 3.3: An empty tuple instead of @code{None} when ndim = 0.

@end deffn

@geindex strides (memoryview attribute)
@anchor{library/stdtypes memoryview strides}@anchor{1b15}
@deffn {Attribute} strides

A tuple of integers the length of @ref{1b13,,ndim} giving the size in bytes to
access each element for each dimension of the array.


Changed in version 3.3: An empty tuple instead of @code{None} when ndim = 0.

@end deffn

@geindex suboffsets (memoryview attribute)
@anchor{library/stdtypes memoryview suboffsets}@anchor{1b16}
@deffn {Attribute} suboffsets

Used internally for PIL-style arrays. The value is informational only.
@end deffn

@geindex c_contiguous (memoryview attribute)
@anchor{library/stdtypes memoryview c_contiguous}@anchor{1b17}
@deffn {Attribute} c_contiguous

A bool indicating whether the memory is C-@ref{1b0f,,contiguous}.


New in version 3.3.

@end deffn

@geindex f_contiguous (memoryview attribute)
@anchor{library/stdtypes memoryview f_contiguous}@anchor{1b18}
@deffn {Attribute} f_contiguous

A bool indicating whether the memory is Fortran @ref{1b0f,,contiguous}.


New in version 3.3.

@end deffn

@geindex contiguous (memoryview attribute)
@anchor{library/stdtypes memoryview contiguous}@anchor{1b19}
@deffn {Attribute} contiguous

A bool indicating whether the memory is @ref{1b0f,,contiguous}.


New in version 3.3.

@end deffn
@end deffn

@node Set Types — set frozenset,Mapping Types — dict,Binary Sequence Types — bytes bytearray memoryview,Built-in Types
@anchor{library/stdtypes set-types-set-frozenset}@anchor{1b1a}@anchor{library/stdtypes types-set}@anchor{1a4f}
@subsection Set Types — @code{set}, @code{frozenset}


@geindex object; set

A @emph{set} object is an unordered collection of distinct @ref{2dc,,hashable} objects.
Common uses include membership testing, removing duplicates from a sequence, and
computing mathematical operations such as intersection, union, difference, and
symmetric difference.
(For other containers see the built-in @ref{3b0,,dict}, @ref{43a,,list},
and @ref{3e5,,tuple} classes, and the @ref{1c,,collections} module.)

Like other collections, sets support @code{x in set}, @code{len(set)}, and @code{for x in
set}.  Being an unordered collection, sets do not record element position or
order of insertion.  Accordingly, sets do not support indexing, slicing, or
other sequence-like behavior.

There are currently two built-in set types, @ref{370,,set} and @ref{439,,frozenset}.
The @ref{370,,set} type is mutable — the contents can be changed using methods
like @code{add()} and @code{remove()}.  Since it is mutable, it has no
hash value and cannot be used as either a dictionary key or as an element of
another set.  The @ref{439,,frozenset} type is immutable and @ref{2dc,,hashable} —
its contents cannot be altered after it is created; it can therefore be used as
a dictionary key or as an element of another set.

Non-empty sets (not frozensets) can be created by placing a comma-separated list
of elements within braces, for example: @code{@{'jack', 'sjoerd'@}}, in addition to the
@ref{370,,set} constructor.

The constructors for both classes work the same:

@geindex set (built-in class)
@geindex frozenset (built-in class)
@anchor{library/stdtypes set}@anchor{370}
@deffn {Class} set ([iterable])
@anchor{library/stdtypes frozenset}@anchor{439}
@deffnx {Class} frozenset ([iterable])

Return a new set or frozenset object whose elements are taken from
@emph{iterable}.  The elements of a set must be @ref{2dc,,hashable}.  To
represent sets of sets, the inner sets must be @ref{439,,frozenset}
objects.  If @emph{iterable} is not specified, a new empty set is
returned.

Sets can be created by several means:


@itemize *

@item 
Use a comma-separated list of elements within braces: @code{@{'jack', 'sjoerd'@}}

@item 
Use a set comprehension: @code{@{c for c in 'abracadabra' if c not in 'abc'@}}

@item 
Use the type constructor: @code{set()}, @code{set('foobar')}, @code{set(['a', 'b', 'foo'])}
@end itemize

Instances of @ref{370,,set} and @ref{439,,frozenset} provide the following
operations:


@deffn {Describe} len(s)

Return the number of elements in set @emph{s} (cardinality of @emph{s}).
@end deffn


@deffn {Describe} x in s

Test @emph{x} for membership in @emph{s}.
@end deffn


@deffn {Describe} x not in s

Test @emph{x} for non-membership in @emph{s}.
@end deffn

@geindex isdisjoint() (frozenset method)
@anchor{library/stdtypes frozenset isdisjoint}@anchor{1b1b}
@deffn {Method} isdisjoint (other)

Return @code{True} if the set has no elements in common with @emph{other}.  Sets are
disjoint if and only if their intersection is the empty set.
@end deffn

@geindex issubset() (frozenset method)
@anchor{library/stdtypes frozenset issubset}@anchor{1b1c}
@deffn {Method} issubset (other)

@deffnx {Method} set <= other

Test whether every element in the set is in @emph{other}.
@end deffn


@deffn {Method} set < other

Test whether the set is a proper subset of @emph{other}, that is,
@code{set <= other and set != other}.
@end deffn

@geindex issuperset() (frozenset method)
@anchor{library/stdtypes frozenset issuperset}@anchor{1b1d}
@deffn {Method} issuperset (other)

@deffnx {Method} set >= other

Test whether every element in @emph{other} is in the set.
@end deffn


@deffn {Method} set > other

Test whether the set is a proper superset of @emph{other}, that is, @code{set >=
other and set != other}.
@end deffn

@geindex union() (frozenset method)
@anchor{library/stdtypes frozenset union}@anchor{1b1e}
@deffn {Method} union (*others)

@deffnx {Method} set | other | ...

Return a new set with elements from the set and all others.
@end deffn

@geindex intersection() (frozenset method)
@anchor{library/stdtypes frozenset intersection}@anchor{1b1f}
@deffn {Method} intersection (*others)

@deffnx {Method} set & other & ...

Return a new set with elements common to the set and all others.
@end deffn

@geindex difference() (frozenset method)
@anchor{library/stdtypes frozenset difference}@anchor{1b20}
@deffn {Method} difference (*others)

@deffnx {Method} set @w{-} other @w{-} ...

Return a new set with elements in the set that are not in the others.
@end deffn

@geindex symmetric_difference() (frozenset method)
@anchor{library/stdtypes frozenset symmetric_difference}@anchor{1b21}
@deffn {Method} symmetric_difference (other)

@deffnx {Method} set ^ other

Return a new set with elements in either the set or @emph{other} but not both.
@end deffn

@geindex copy() (frozenset method)
@anchor{library/stdtypes frozenset copy}@anchor{1b22}
@deffn {Method} copy ()

Return a shallow copy of the set.
@end deffn

Note, the non-operator versions of @ref{1b1e,,union()}, @ref{1b1f,,intersection()},
@ref{1b20,,difference()}, @ref{1b21,,symmetric_difference()}, @ref{1b1c,,issubset()}, and
@ref{1b1d,,issuperset()} methods will accept any iterable as an argument.  In
contrast, their operator based counterparts require their arguments to be
sets.  This precludes error-prone constructions like @code{set('abc') & 'cbs'}
in favor of the more readable @code{set('abc').intersection('cbs')}.

Both @ref{370,,set} and @ref{439,,frozenset} support set to set comparisons. Two
sets are equal if and only if every element of each set is contained in the
other (each is a subset of the other). A set is less than another set if and
only if the first set is a proper subset of the second set (is a subset, but
is not equal). A set is greater than another set if and only if the first set
is a proper superset of the second set (is a superset, but is not equal).

Instances of @ref{370,,set} are compared to instances of @ref{439,,frozenset}
based on their members.  For example, @code{set('abc') == frozenset('abc')}
returns @code{True} and so does @code{set('abc') in set([frozenset('abc')])}.

The subset and equality comparisons do not generalize to a total ordering
function.  For example, any two nonempty disjoint sets are not equal and are not
subsets of each other, so @emph{all} of the following return @code{False}: @code{a<b},
@code{a==b}, or @code{a>b}.

Since sets only define partial ordering (subset relationships), the output of
the @ref{705,,list.sort()} method is undefined for lists of sets.

Set elements, like dictionary keys, must be @ref{2dc,,hashable}.

Binary operations that mix @ref{370,,set} instances with @ref{439,,frozenset}
return the type of the first operand.  For example: @code{frozenset('ab') |
set('bc')} returns an instance of @ref{439,,frozenset}.

The following table lists operations available for @ref{370,,set} that do not
apply to immutable instances of @ref{439,,frozenset}:

@geindex update() (frozenset method)
@anchor{library/stdtypes frozenset update}@anchor{1b23}
@deffn {Method} update (*others)

@deffnx {Method} set |= other | ...

Update the set, adding elements from all others.
@end deffn

@geindex intersection_update() (frozenset method)
@anchor{library/stdtypes frozenset intersection_update}@anchor{1b24}
@deffn {Method} intersection_update (*others)

@deffnx {Method} set &= other & ...

Update the set, keeping only elements found in it and all others.
@end deffn

@geindex difference_update() (frozenset method)
@anchor{library/stdtypes frozenset difference_update}@anchor{1b25}
@deffn {Method} difference_update (*others)

@deffnx {Method} set @w{-}= other | ...

Update the set, removing elements found in others.
@end deffn

@geindex symmetric_difference_update() (frozenset method)
@anchor{library/stdtypes frozenset symmetric_difference_update}@anchor{1b26}
@deffn {Method} symmetric_difference_update (other)

@deffnx {Method} set ^= other

Update the set, keeping only elements found in either set, but not in both.
@end deffn

@geindex add() (frozenset method)
@anchor{library/stdtypes frozenset add}@anchor{1b27}
@deffn {Method} add (elem)

Add element @emph{elem} to the set.
@end deffn

@geindex remove() (frozenset method)
@anchor{library/stdtypes frozenset remove}@anchor{1b28}
@deffn {Method} remove (elem)

Remove element @emph{elem} from the set.  Raises @ref{5bb,,KeyError} if @emph{elem} is
not contained in the set.
@end deffn

@geindex discard() (frozenset method)
@anchor{library/stdtypes frozenset discard}@anchor{1b29}
@deffn {Method} discard (elem)

Remove element @emph{elem} from the set if it is present.
@end deffn

@geindex pop() (frozenset method)
@anchor{library/stdtypes frozenset pop}@anchor{1b2a}
@deffn {Method} pop ()

Remove and return an arbitrary element from the set.  Raises
@ref{5bb,,KeyError} if the set is empty.
@end deffn

@geindex clear() (frozenset method)
@anchor{library/stdtypes frozenset clear}@anchor{1b2b}
@deffn {Method} clear ()

Remove all elements from the set.
@end deffn

Note, the non-operator versions of the @ref{1b23,,update()},
@ref{1b24,,intersection_update()}, @ref{1b25,,difference_update()}, and
@ref{1b26,,symmetric_difference_update()} methods will accept any iterable as an
argument.

Note, the @emph{elem} argument to the @code{__contains__()}, @ref{1b28,,remove()}, and
@ref{1b29,,discard()} methods may be a set.  To support searching for an equivalent
frozenset, a temporary one is created from @emph{elem}.
@end deffn

@node Mapping Types — dict,Context Manager Types,Set Types — set frozenset,Built-in Types
@anchor{library/stdtypes mapping-types-dict}@anchor{1b2c}@anchor{library/stdtypes typesmapping}@anchor{5de}
@subsection Mapping Types — @code{dict}


@geindex object; mapping
@geindex object; dictionary
@geindex operations on; mapping; types
@geindex operations on; dictionary; type
@geindex statement; del
@geindex built-in function; len

A @ref{d62,,mapping} object maps @ref{2dc,,hashable} values to arbitrary objects.
Mappings are mutable objects.  There is currently only one standard mapping
type, the @emph{dictionary}.  (For other containers see the built-in
@ref{43a,,list}, @ref{370,,set}, and @ref{3e5,,tuple} classes, and the
@ref{1c,,collections} module.)

A dictionary’s keys are @emph{almost} arbitrary values.  Values that are not
@ref{2dc,,hashable}, that is, values containing lists, dictionaries or other
mutable types (that are compared by value rather than by object identity) may
not be used as keys.  Numeric types used for keys obey the normal rules for
numeric comparison: if two numbers compare equal (such as @code{1} and @code{1.0})
then they can be used interchangeably to index the same dictionary entry.  (Note
however, that since computers store floating-point numbers as approximations it
is usually unwise to use them as dictionary keys.)

Dictionaries can be created by placing a comma-separated list of @code{key: value}
pairs within braces, for example: @code{@{'jack': 4098, 'sjoerd': 4127@}} or @code{@{4098:
'jack', 4127: 'sjoerd'@}}, or by the @ref{3b0,,dict} constructor.

@geindex dict (built-in class)
@anchor{library/stdtypes dict}@anchor{3b0}
@deffn {Class} dict (**kwargs)

@deffnx {Class} dict (mapping, **kwargs)

@deffnx {Class} dict (iterable, **kwargs)

Return a new dictionary initialized from an optional positional argument
and a possibly empty set of keyword arguments.

Dictionaries can be created by several means:


@itemize *

@item 
Use a comma-separated list of @code{key: value} pairs within braces:
@code{@{'jack': 4098, 'sjoerd': 4127@}} or @code{@{4098: 'jack', 4127: 'sjoerd'@}}

@item 
Use a dict comprehension: @code{@{@}}, @code{@{x: x ** 2 for x in range(10)@}}

@item 
Use the type constructor: @code{dict()},
@code{dict([('foo', 100), ('bar', 200)])}, @code{dict(foo=100, bar=200)}
@end itemize

If no positional argument is given, an empty dictionary is created.
If a positional argument is given and it is a mapping object, a dictionary
is created with the same key-value pairs as the mapping object.  Otherwise,
the positional argument must be an @ref{dcc,,iterable} object.  Each item in
the iterable must itself be an iterable with exactly two objects.  The
first object of each item becomes a key in the new dictionary, and the
second object the corresponding value.  If a key occurs more than once, the
last value for that key becomes the corresponding value in the new
dictionary.

If keyword arguments are given, the keyword arguments and their values are
added to the dictionary created from the positional argument.  If a key
being added is already present, the value from the keyword argument
replaces the value from the positional argument.

To illustrate, the following examples all return a dictionary equal to
@code{@{"one": 1, "two": 2, "three": 3@}}:

@example
>>> a = dict(one=1, two=2, three=3)
>>> b = @{'one': 1, 'two': 2, 'three': 3@}
>>> c = dict(zip(['one', 'two', 'three'], [1, 2, 3]))
>>> d = dict([('two', 2), ('one', 1), ('three', 3)])
>>> e = dict(@{'three': 3, 'one': 1, 'two': 2@})
>>> f = dict(@{'one': 1, 'three': 3@}, two=2)
>>> a == b == c == d == e == f
True
@end example

Providing keyword arguments as in the first example only works for keys that
are valid Python identifiers.  Otherwise, any valid keys can be used.

These are the operations that dictionaries support (and therefore, custom
mapping types should support too):


@deffn {Describe} list(d)

Return a list of all the keys used in the dictionary @emph{d}.
@end deffn


@deffn {Describe} len(d)

Return the number of items in the dictionary @emph{d}.
@end deffn


@deffn {Describe} d[key]

Return the item of @emph{d} with key @emph{key}.  Raises a @ref{5bb,,KeyError} if @emph{key} is
not in the map.

@geindex __missing__()

If a subclass of dict defines a method @code{__missing__()} and @emph{key}
is not present, the @code{d[key]} operation calls that method with the key @emph{key}
as argument.  The @code{d[key]} operation then returns or raises whatever is
returned or raised by the @code{__missing__(key)} call.
No other operations or methods invoke @code{__missing__()}. If
@code{__missing__()} is not defined, @ref{5bb,,KeyError} is raised.
@code{__missing__()} must be a method; it cannot be an instance variable:

@example
>>> class Counter(dict):
...     def __missing__(self, key):
...         return 0
>>> c = Counter()
>>> c['red']
0
>>> c['red'] += 1
>>> c['red']
1
@end example

The example above shows part of the implementation of
@ref{c2a,,collections.Counter}.  A different @code{__missing__} method is used
by @ref{d63,,collections.defaultdict}.
@end deffn


@deffn {Describe} d[key] = value

Set @code{d[key]} to @emph{value}.
@end deffn


@deffn {Describe} del d[key]

Remove @code{d[key]} from @emph{d}.  Raises a @ref{5bb,,KeyError} if @emph{key} is not in the
map.
@end deffn


@deffn {Describe} key in d

Return @code{True} if @emph{d} has a key @emph{key}, else @code{False}.
@end deffn


@deffn {Describe} key not in d

Equivalent to @code{not key in d}.
@end deffn


@deffn {Describe} iter(d)

Return an iterator over the keys of the dictionary.  This is a shortcut
for @code{iter(d.keys())}.
@end deffn

@geindex clear() (dict method)
@anchor{library/stdtypes dict clear}@anchor{122c}
@deffn {Method} clear ()

Remove all items from the dictionary.
@end deffn

@geindex copy() (dict method)
@anchor{library/stdtypes dict copy}@anchor{706}
@deffn {Method} copy ()

Return a shallow copy of the dictionary.
@end deffn

@geindex fromkeys() (dict class method)
@anchor{library/stdtypes dict fromkeys}@anchor{1b2d}
@deffn {Method} classmethod fromkeys (iterable[, value])

Create a new dictionary with keys from @emph{iterable} and values set to @emph{value}.

@ref{1b2d,,fromkeys()} is a class method that returns a new dictionary. @emph{value}
defaults to @code{None}.  All of the values refer to just a single instance,
so it generally doesn’t make sense for @emph{value} to be a mutable object
such as an empty list.  To get distinct values, use a @ref{8d8,,dict comprehension} instead.
@end deffn

@geindex get() (dict method)
@anchor{library/stdtypes dict get}@anchor{1b2e}
@deffn {Method} get (key[, default])

Return the value for @emph{key} if @emph{key} is in the dictionary, else @emph{default}.
If @emph{default} is not given, it defaults to @code{None}, so that this method
never raises a @ref{5bb,,KeyError}.
@end deffn

@geindex items() (dict method)
@anchor{library/stdtypes dict items}@anchor{245}
@deffn {Method} items ()

Return a new view of the dictionary’s items (@code{(key, value)} pairs).
See the @ref{1b2f,,documentation of view objects}.
@end deffn

@geindex keys() (dict method)
@anchor{library/stdtypes dict keys}@anchor{243}
@deffn {Method} keys ()

Return a new view of the dictionary’s keys.  See the @ref{1b2f,,documentation of view objects}.
@end deffn

@geindex pop() (dict method)
@anchor{library/stdtypes dict pop}@anchor{1b30}
@deffn {Method} pop (key[, default])

If @emph{key} is in the dictionary, remove it and return its value, else return
@emph{default}.  If @emph{default} is not given and @emph{key} is not in the dictionary,
a @ref{5bb,,KeyError} is raised.
@end deffn

@geindex popitem() (dict method)
@anchor{library/stdtypes dict popitem}@anchor{1b31}
@deffn {Method} popitem ()

Remove and return a @code{(key, value)} pair from the dictionary.
Pairs are returned in @abbr{LIFO,last-in@comma{} first-out} order.

@ref{1b31,,popitem()} is useful to destructively iterate over a dictionary, as
often used in set algorithms.  If the dictionary is empty, calling
@ref{1b31,,popitem()} raises a @ref{5bb,,KeyError}.


Changed in version 3.7: LIFO order is now guaranteed. In prior versions, @ref{1b31,,popitem()} would
return an arbitrary key/value pair.

@end deffn


@deffn {Describe} reversed(d)

Return a reverse iterator over the keys of the dictionary. This is a
shortcut for @code{reversed(d.keys())}.


New in version 3.8.

@end deffn

@geindex setdefault() (dict method)
@anchor{library/stdtypes dict setdefault}@anchor{c0f}
@deffn {Method} setdefault (key[, default])

If @emph{key} is in the dictionary, return its value.  If not, insert @emph{key}
with a value of @emph{default} and return @emph{default}.  @emph{default} defaults to
@code{None}.
@end deffn

@geindex update() (dict method)
@anchor{library/stdtypes dict update}@anchor{fc3}
@deffn {Method} update ([other])

Update the dictionary with the key/value pairs from @emph{other}, overwriting
existing keys.  Return @code{None}.

@ref{fc3,,update()} accepts either another dictionary object or an iterable of
key/value pairs (as tuples or other iterables of length two).  If keyword
arguments are specified, the dictionary is then updated with those
key/value pairs: @code{d.update(red=1, blue=2)}.
@end deffn

@geindex values() (dict method)
@anchor{library/stdtypes dict values}@anchor{244}
@deffn {Method} values ()

Return a new view of the dictionary’s values.  See the
@ref{1b2f,,documentation of view objects}.

An equality comparison between one @code{dict.values()} view and another
will always return @code{False}. This also applies when comparing
@code{dict.values()} to itself:

@example
>>> d = @{'a': 1@}
>>> d.values() == d.values()
False
@end example
@end deffn


@deffn {Describe} d | other

Create a new dictionary with the merged keys and values of @emph{d} and
@emph{other}, which must both be dictionaries. The values of @emph{other} take
priority when @emph{d} and @emph{other} share keys.


New in version 3.9.

@end deffn


@deffn {Describe} d |= other

Update the dictionary @emph{d} with keys and values from @emph{other}, which may be
either a @ref{d62,,mapping} or an @ref{dcc,,iterable} of key/value pairs. The
values of @emph{other} take priority when @emph{d} and @emph{other} share keys.


New in version 3.9.

@end deffn

Dictionaries compare equal if and only if they have the same @code{(key,
value)} pairs (regardless of ordering). Order comparisons (‘<’, ‘<=’, ‘>=’, ‘>’) raise
@ref{162,,TypeError}.

Dictionaries preserve insertion order.  Note that updating a key does not
affect the order.  Keys added after deletion are inserted at the end.

@example
>>> d = @{"one": 1, "two": 2, "three": 3, "four": 4@}
>>> d
@{'one': 1, 'two': 2, 'three': 3, 'four': 4@}
>>> list(d)
['one', 'two', 'three', 'four']
>>> list(d.values())
[1, 2, 3, 4]
>>> d["one"] = 42
>>> d
@{'one': 42, 'two': 2, 'three': 3, 'four': 4@}
>>> del d["two"]
>>> d["two"] = None
>>> d
@{'one': 42, 'three': 3, 'four': 4, 'two': None@}
@end example


Changed in version 3.7: Dictionary order is guaranteed to be insertion order.  This behavior was
an implementation detail of CPython from 3.6.


Dictionaries and dictionary views are reversible.

@example
>>> d = @{"one": 1, "two": 2, "three": 3, "four": 4@}
>>> d
@{'one': 1, 'two': 2, 'three': 3, 'four': 4@}
>>> list(reversed(d))
['four', 'three', 'two', 'one']
>>> list(reversed(d.values()))
[4, 3, 2, 1]
>>> list(reversed(d.items()))
[('four', 4), ('three', 3), ('two', 2), ('one', 1)]
@end example


Changed in version 3.8: Dictionaries are now reversible.

@end deffn


@subsubheading See also


@ref{246,,types.MappingProxyType} can be used to create a read-only view
of a @ref{3b0,,dict}.


@menu
* Dictionary view objects:: 

@end menu

@node Dictionary view objects,,,Mapping Types — dict
@anchor{library/stdtypes dict-views}@anchor{1b2f}@anchor{library/stdtypes dictionary-view-objects}@anchor{1b32}
@subsubsection Dictionary view objects


The objects returned by @ref{243,,dict.keys()}, @ref{244,,dict.values()} and
@ref{245,,dict.items()} are @emph{view objects}.  They provide a dynamic view on the
dictionary’s entries, which means that when the dictionary changes, the view
reflects these changes.

Dictionary views can be iterated over to yield their respective data, and
support membership tests:


@deffn {Describe} len(dictview)

Return the number of entries in the dictionary.
@end deffn


@deffn {Describe} iter(dictview)

Return an iterator over the keys, values or items (represented as tuples of
@code{(key, value)}) in the dictionary.

Keys and values are iterated over in insertion order.
This allows the creation of @code{(value, key)} pairs
using @ref{247,,zip()}: @code{pairs = zip(d.values(), d.keys())}.  Another way to
create the same list is @code{pairs = [(v, k) for (k, v) in d.items()]}.

Iterating views while adding or deleting entries in the dictionary may raise
a @ref{5af,,RuntimeError} or fail to iterate over all entries.


Changed in version 3.7: Dictionary order is guaranteed to be insertion order.

@end deffn


@deffn {Describe} x in dictview

Return @code{True} if @emph{x} is in the underlying dictionary’s keys, values or
items (in the latter case, @emph{x} should be a @code{(key, value)} tuple).
@end deffn


@deffn {Describe} reversed(dictview)

Return a reverse iterator over the keys, values or items of the dictionary.
The view will be iterated in reverse order of the insertion.


Changed in version 3.8: Dictionary views are now reversible.

@end deffn


@deffn {Describe} dictview.mapping

Return a @ref{246,,types.MappingProxyType} that wraps the original
dictionary to which the view refers.


New in version 3.10.

@end deffn

Keys views are set-like since their entries are unique and hashable.  If all
values are hashable, so that @code{(key, value)} pairs are unique and hashable,
then the items view is also set-like.  (Values views are not treated as set-like
since the entries are generally not unique.)  For set-like views, all of the
operations defined for the abstract base class @ref{1b33,,collections.abc.Set} are
available (for example, @code{==}, @code{<}, or @code{^}).

An example of dictionary view usage:

@example
>>> dishes = @{'eggs': 2, 'sausage': 1, 'bacon': 1, 'spam': 500@}
>>> keys = dishes.keys()
>>> values = dishes.values()

>>> # iteration
>>> n = 0
>>> for val in values:
...     n += val
>>> print(n)
504

>>> # keys and values are iterated over in the same order (insertion order)
>>> list(keys)
['eggs', 'sausage', 'bacon', 'spam']
>>> list(values)
[2, 1, 1, 500]

>>> # view objects are dynamic and reflect dict changes
>>> del dishes['eggs']
>>> del dishes['sausage']
>>> list(keys)
['bacon', 'spam']

>>> # set operations
>>> keys & @{'eggs', 'bacon', 'salad'@}
@{'bacon'@}
>>> keys ^ @{'sausage', 'juice'@}
@{'juice', 'sausage', 'bacon', 'spam'@}

>>> # get back a read-only proxy for the original dictionary
>>> values.mapping
mappingproxy(@{'eggs': 2, 'sausage': 1, 'bacon': 1, 'spam': 500@})
>>> values.mapping['spam']
500
@end example

@node Context Manager Types,Type Annotation Types — Generic Alias Union,Mapping Types — dict,Built-in Types
@anchor{library/stdtypes context-manager-types}@anchor{1b34}@anchor{library/stdtypes typecontextmanager}@anchor{1810}
@subsection Context Manager Types


@geindex context manager
@geindex context management protocol
@geindex protocol; context management

Python’s @ref{168,,with} statement supports the concept of a runtime context
defined by a context manager.  This is implemented using a pair of methods
that allow user-defined classes to define a runtime context that is entered
before the statement body is executed and exited when the statement ends:

@geindex __enter__() (contextmanager method)
@anchor{library/stdtypes contextmanager __enter__}@anchor{1b35}
@deffn {Method} contextmanager.__enter__ ()

Enter the runtime context and return either this object or another object
related to the runtime context. The value returned by this method is bound to
the identifier in the @code{as} clause of @ref{168,,with} statements using
this context manager.

An example of a context manager that returns itself is a @ref{d68,,file object}.
File objects return themselves from __enter__() to allow @ref{1c8,,open()} to be
used as the context expression in a @ref{168,,with} statement.

An example of a context manager that returns a related object is the one
returned by @ref{1b36,,decimal.localcontext()}. These managers set the active
decimal context to a copy of the original decimal context and then return the
copy. This allows changes to be made to the current decimal context in the body
of the @ref{168,,with} statement without affecting code outside the
@code{with} statement.
@end deffn

@geindex __exit__() (contextmanager method)
@anchor{library/stdtypes contextmanager __exit__}@anchor{1b37}
@deffn {Method} contextmanager.__exit__ (exc_type, exc_val, exc_tb)

Exit the runtime context and return a Boolean flag indicating if any exception
that occurred should be suppressed. If an exception occurred while executing the
body of the @ref{168,,with} statement, the arguments contain the exception type,
value and traceback information. Otherwise, all three arguments are @code{None}.

Returning a true value from this method will cause the @ref{168,,with} statement
to suppress the exception and continue execution with the statement immediately
following the @code{with} statement. Otherwise the exception continues
propagating after this method has finished executing. Exceptions that occur
during execution of this method will replace any exception that occurred in the
body of the @code{with} statement.

The exception passed in should never be reraised explicitly - instead, this
method should return a false value to indicate that the method completed
successfully and does not want to suppress the raised exception. This allows
context management code to easily detect whether or not an @ref{1b37,,__exit__()}
method has actually failed.
@end deffn

Python defines several context managers to support easy thread synchronisation,
prompt closure of files or other objects, and simpler manipulation of the active
decimal arithmetic context. The specific types are not treated specially beyond
their implementation of the context management protocol. See the
@ref{22,,contextlib} module for some examples.

Python’s @ref{bf7,,generator}s and the @ref{da3,,contextlib.contextmanager} decorator
provide a convenient way to implement these protocols.  If a generator function is
decorated with the @ref{da3,,contextlib.contextmanager} decorator, it will return a
context manager implementing the necessary @ref{1b35,,__enter__()} and
@ref{1b37,,__exit__()} methods, rather than the iterator produced by an
undecorated generator function.

Note that there is no specific slot for any of these methods in the type
structure for Python objects in the Python/C API. Extension types wanting to
define these methods must provide them as a normal Python accessible method.
Compared to the overhead of setting up the runtime context, the overhead of a
single class dictionary lookup is negligible.

@node Type Annotation Types — Generic Alias Union,Other Built-in Types,Context Manager Types,Built-in Types
@anchor{library/stdtypes type-annotation-types-generic-alias-union}@anchor{1b38}
@subsection Type Annotation Types — Generic Alias, Union


@geindex annotation; type annotation; type hint

The core built-in types for @ref{17ca,,type annotations} are
@ref{268,,Generic Alias} and @ref{238,,Union}.

@menu
* Generic Alias Type:: 
* Union Type:: 

@end menu

@node Generic Alias Type,Union Type,,Type Annotation Types — Generic Alias Union
@anchor{library/stdtypes generic-alias-type}@anchor{1b39}@anchor{library/stdtypes types-genericalias}@anchor{268}
@subsubsection Generic Alias Type


@geindex object; GenericAlias
@geindex Generic; Alias

@code{GenericAlias} objects are generally created by
@ref{17d1,,subscripting} a class. They are most often used with
@ref{17d7,,container classes}, such as @ref{43a,,list} or
@ref{3b0,,dict}. For example, @code{list[int]} is a @code{GenericAlias} object created
by subscripting the @code{list} class with the argument @ref{1b1,,int}.
@code{GenericAlias} objects are intended primarily for use with
@ref{17ca,,type annotations}.

@cartouche
@quotation Note 
It is generally only possible to subscript a class if the class implements
the special method @ref{10a5,,__class_getitem__()}.
@end quotation
@end cartouche

A @code{GenericAlias} object acts as a proxy for a @ref{11b0,,generic type},
implementing @emph{parameterized generics}.

For a container class, the
argument(s) supplied to a @ref{17d1,,subscription} of the class may
indicate the type(s) of the elements an object contains. For example,
@code{set[bytes]} can be used in type annotations to signify a @ref{370,,set} in
which all the elements are of type @ref{2f0,,bytes}.

For a class which defines @ref{10a5,,__class_getitem__()} but is not a
container, the argument(s) supplied to a subscription of the class will often
indicate the return type(s) of one or more methods defined on an object. For
example, @ref{d8,,regular expressions} can be used on both the @ref{2ef,,str} data
type and the @ref{2f0,,bytes} data type:


@itemize *

@item 
If @code{x = re.search('foo', 'foo')}, @code{x} will be a
@ref{b08,,re.Match} object where the return values of
@code{x.group(0)} and @code{x[0]} will both be of type @ref{2ef,,str}. We can
represent this kind of object in type annotations with the @code{GenericAlias}
@code{re.Match[str]}.

@item 
If @code{y = re.search(b'bar', b'bar')}, (note the @code{b} for @ref{2f0,,bytes}),
@code{y} will also be an instance of @code{re.Match}, but the return
values of @code{y.group(0)} and @code{y[0]} will both be of type
@ref{2f0,,bytes}. In type annotations, we would represent this
variety of @ref{b08,,re.Match} objects with @code{re.Match[bytes]}.
@end itemize

@code{GenericAlias} objects are instances of the class
@ref{26a,,types.GenericAlias}, which can also be used to create @code{GenericAlias}
objects directly.


@deffn {Describe} T[X, Y, ...]

Creates a @code{GenericAlias} representing a type @code{T} parameterized by types
@emph{X}, @emph{Y}, and more depending on the @code{T} used.
For example, a function expecting a @ref{43a,,list} containing
@ref{256,,float} elements:

@example
def average(values: list[float]) -> float:
    return sum(values) / len(values)
@end example

Another example for @ref{d62,,mapping} objects, using a @ref{3b0,,dict}, which
is a generic type expecting two type parameters representing the key type
and the value type.  In this example, the function expects a @code{dict} with
keys of type @ref{2ef,,str} and values of type @ref{1b1,,int}:

@example
def send_post_request(url: str, body: dict[str, int]) -> None:
    ...
@end example
@end deffn

The builtin functions @ref{236,,isinstance()} and @ref{237,,issubclass()} do not accept
@code{GenericAlias} types for their second argument:

@example
>>> isinstance([1, 2], list[str])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: isinstance() argument 2 cannot be a parameterized generic
@end example

The Python runtime does not enforce @ref{17ca,,type annotations}.
This extends to generic types and their type parameters. When creating
a container object from a @code{GenericAlias}, the elements in the container are not checked
against their type. For example, the following code is discouraged, but will
run without errors:

@example
>>> t = list[str]
>>> t([1, 2, 3])
[1, 2, 3]
@end example

Furthermore, parameterized generics erase type parameters during object
creation:

@example
>>> t = list[str]
>>> type(t)
<class 'types.GenericAlias'>

>>> l = t()
>>> type(l)
<class 'list'>
@end example

Calling @ref{27d,,repr()} or @ref{2ef,,str()} on a generic shows the parameterized type:

@example
>>> repr(list[int])
'list[int]'

>>> str(list[int])
'list[int]'
@end example

The @ref{179c,,__getitem__()} method of generic containers will raise an
exception to disallow mistakes like @code{dict[str][str]}:

@example
>>> dict[str][str]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: There are no type variables left in dict[str]
@end example

However, such expressions are valid when @ref{17cb,,type variables} are
used.  The index must have as many elements as there are type variable items
in the @code{GenericAlias} object’s @ref{1b3a,,__args__}.

@example
>>> from typing import TypeVar
>>> Y = TypeVar('Y')
>>> dict[str, Y][int]
dict[str, int]
@end example

@menu
* Standard Generic Classes:: 
* Special Attributes of GenericAlias objects:: 

@end menu

@node Standard Generic Classes,Special Attributes of GenericAlias objects,,Generic Alias Type
@anchor{library/stdtypes standard-generic-classes}@anchor{1b3b}
@subsubsection Standard Generic Classes


The following standard library classes support parameterized generics. This
list is non-exhaustive.


@itemize *

@item 
@ref{3e5,,tuple}

@item 
@ref{43a,,list}

@item 
@ref{3b0,,dict}

@item 
@ref{370,,set}

@item 
@ref{439,,frozenset}

@item 
@ref{89e,,type}

@item 
@ref{7d3,,collections.deque}

@item 
@ref{d63,,collections.defaultdict}

@item 
@ref{4e5,,collections.OrderedDict}

@item 
@ref{c2a,,collections.Counter}

@item 
@ref{758,,collections.ChainMap}

@item 
@ref{93b,,collections.abc.Awaitable}

@item 
@ref{93c,,collections.abc.Coroutine}

@item 
@ref{93e,,collections.abc.AsyncIterable}

@item 
@ref{93d,,collections.abc.AsyncIterator}

@item 
@ref{7d2,,collections.abc.AsyncGenerator}

@item 
@ref{1b3c,,collections.abc.Iterable}

@item 
@ref{1b3d,,collections.abc.Iterator}

@item 
@ref{93a,,collections.abc.Generator}

@item 
@ref{7d1,,collections.abc.Reversible}

@item 
@ref{1b3e,,collections.abc.Container}

@item 
@ref{7d0,,collections.abc.Collection}

@item 
@ref{269,,collections.abc.Callable}

@item 
@ref{1b33,,collections.abc.Set}

@item 
@ref{1b3f,,collections.abc.MutableSet}

@item 
@ref{3ab,,collections.abc.Mapping}

@item 
@ref{c3f,,collections.abc.MutableMapping}

@item 
@ref{110c,,collections.abc.Sequence}

@item 
@ref{933,,collections.abc.MutableSequence}

@item 
@ref{1b40,,collections.abc.ByteString}

@item 
@ref{1b41,,collections.abc.MappingView}

@item 
@ref{1b42,,collections.abc.KeysView}

@item 
@ref{1b43,,collections.abc.ItemsView}

@item 
@ref{1b44,,collections.abc.ValuesView}

@item 
@ref{7d6,,contextlib.AbstractContextManager}

@item 
@ref{64a,,contextlib.AbstractAsyncContextManager}

@item 
@ref{1b45,,dataclasses.Field}

@item 
@ref{4f4,,functools.cached_property}

@item 
@ref{593,,functools.partialmethod}

@item 
@ref{795,,os.PathLike}

@item 
@ref{1b46,,queue.LifoQueue}

@item 
@ref{f18,,queue.Queue}

@item 
@ref{1b47,,queue.PriorityQueue}

@item 
@ref{695,,queue.SimpleQueue}

@item 
@ref{b07,,re.Pattern}

@item 
@ref{b08,,re.Match}

@item 
@ref{1b48,,shelve.BsdDbShelf}

@item 
@ref{1b49,,shelve.DbfilenameShelf}

@item 
@ref{b1b,,shelve.Shelf}

@item 
@ref{246,,types.MappingProxyType}

@item 
@ref{11fa,,weakref.WeakKeyDictionary}

@item 
@ref{b75,,weakref.WeakMethod}

@item 
@ref{eba,,weakref.WeakSet}

@item 
@ref{11f9,,weakref.WeakValueDictionary}
@end itemize

@node Special Attributes of GenericAlias objects,,Standard Generic Classes,Generic Alias Type
@anchor{library/stdtypes special-attributes-of-genericalias-objects}@anchor{1b4a}
@subsubsection Special Attributes of @code{GenericAlias} objects


All parameterized generics implement special read-only attributes.

@geindex __origin__ (genericalias attribute)
@anchor{library/stdtypes genericalias __origin__}@anchor{1b4b}
@deffn {Attribute} genericalias.__origin__

This attribute points at the non-parameterized generic class:

@example
>>> list[int].__origin__
<class 'list'>
@end example
@end deffn

@geindex __args__ (genericalias attribute)
@anchor{library/stdtypes genericalias __args__}@anchor{1b3a}
@deffn {Attribute} genericalias.__args__

This attribute is a @ref{3e5,,tuple} (possibly of length 1) of generic
types passed to the original @ref{10a5,,__class_getitem__()} of the
generic class:

@example
>>> dict[str, list[int]].__args__
(<class 'str'>, list[int])
@end example
@end deffn

@geindex __parameters__ (genericalias attribute)
@anchor{library/stdtypes genericalias __parameters__}@anchor{1b4c}
@deffn {Attribute} genericalias.__parameters__

This attribute is a lazily computed tuple (possibly empty) of unique type
variables found in @code{__args__}:

@example
>>> from typing import TypeVar

>>> T = TypeVar('T')
>>> list[T].__parameters__
(~T,)
@end example

@cartouche
@quotation Note 
A @code{GenericAlias} object with @ref{23a,,typing.ParamSpec} parameters may not
have correct @code{__parameters__} after substitution because
@ref{23a,,typing.ParamSpec} is intended primarily for static type checking.
@end quotation
@end cartouche
@end deffn


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 484

@item PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} - Type Hints

Introducing Python’s framework for type annotations.

@geindex Python Enhancement Proposals; PEP 585

@item PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} - Type Hinting Generics In Standard Collections

Introducing the ability to natively parameterize standard-library
classes, provided they implement the special class method
@ref{10a5,,__class_getitem__()}.

@item @ref{17cb,,Generics}, @ref{17cc,,user-defined generics} and @ref{11c4,,typing.Generic}

Documentation on how to implement generic classes that can be
parameterized at runtime and understood by static type-checkers.
@end table



New in version 3.9.


@node Union Type,,Generic Alias Type,Type Annotation Types — Generic Alias Union
@anchor{library/stdtypes types-union}@anchor{238}@anchor{library/stdtypes union-type}@anchor{1b4d}
@subsubsection Union Type


@geindex object; Union
@geindex union; type

A union object holds the value of the @code{|} (bitwise or) operation on
multiple @ref{1a62,,type objects}.  These types are intended
primarily for @ref{17ca,,type annotations}. The union type expression
enables cleaner type hinting syntax compared to @ref{235,,typing.Union}.


@deffn {Describe} X | Y | ...

Defines a union object which holds types @emph{X}, @emph{Y}, and so forth. @code{X | Y}
means either X or Y.  It is equivalent to @code{typing.Union[X, Y]}.
For example, the following function expects an argument of type
@ref{1b1,,int} or @ref{256,,float}:

@example
def square(number: int | float) -> int | float:
    return number ** 2
@end example
@end deffn


@deffn {Describe} union_object == other

Union objects can be tested for equality with other union objects.  Details:


@itemize *

@item 
Unions of unions are flattened:

@example
(int | str) | float == int | str | float
@end example

@item 
Redundant types are removed:

@example
int | str | int == int | str
@end example

@item 
When comparing unions, the order is ignored:

@example
int | str == str | int
@end example

@item 
It is compatible with @ref{235,,typing.Union}:

@example
int | str == typing.Union[int, str]
@end example

@item 
Optional types can be spelled as a union with @code{None}:

@example
str | None == typing.Optional[str]
@end example
@end itemize
@end deffn


@deffn {Describe} isinstance(obj, union_object)
@end deffn


@deffn {Describe} issubclass(obj, union_object)

Calls to @ref{236,,isinstance()} and @ref{237,,issubclass()} are also supported with a
union object:

@example
>>> isinstance("", int | str)
True
@end example

However, union objects containing @ref{268,,parameterized generics} cannot be used:

@example
>>> isinstance(1, int | list[int])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: isinstance() argument 2 cannot contain a parameterized generic
@end example
@end deffn

The user-exposed type for the union object can be accessed from
@ref{1b4e,,types.UnionType} and used for @ref{236,,isinstance()} checks.  An object cannot be
instantiated from the type:

@example
>>> import types
>>> isinstance(int | str, types.UnionType)
True
>>> types.UnionType()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot create 'types.UnionType' instances
@end example

@cartouche
@quotation Note 
The @code{__or__()} method for type objects was added to support the syntax
@code{X | Y}.  If a metaclass implements @code{__or__()}, the Union may
override it:

@example
>>> class M(type):
...     def __or__(self, other):
...         return "Hello"
...
>>> class C(metaclass=M):
...     pass
...
>>> C | int
'Hello'
>>> int | C
int | __main__.C
@end example
@end quotation
@end cartouche


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 604
PEP 604@footnote{https://www.python.org/dev/peps/pep-0604} – PEP proposing the @code{X | Y} syntax and the Union type.



New in version 3.10.


@node Other Built-in Types,Special Attributes,Type Annotation Types — Generic Alias Union,Built-in Types
@anchor{library/stdtypes other-built-in-types}@anchor{1b4f}@anchor{library/stdtypes typesother}@anchor{1b50}
@subsection Other Built-in Types


The interpreter supports several other kinds of objects. Most of these support
only one or two operations.

@menu
* Modules: Modules<2>. 
* Classes and Class Instances:: 
* Functions:: 
* Methods:: 
* Code Objects:: 
* Type Objects:: 
* The Null Object:: 
* The Ellipsis Object:: 
* The NotImplemented Object:: 
* Boolean Values:: 
* Internal Objects:: 

@end menu

@node Modules<2>,Classes and Class Instances,,Other Built-in Types
@anchor{library/stdtypes modules}@anchor{1b51}@anchor{library/stdtypes typesmodules}@anchor{1b52}
@subsubsection Modules


The only special operation on a module is attribute access: @code{m.name}, where
@emph{m} is a module and @emph{name} accesses a name defined in @emph{m}’s symbol table.
Module attributes can be assigned to.  (Note that the @ref{e35,,import}
statement is not, strictly speaking, an operation on a module object; @code{import
foo} does not require a module object named @emph{foo} to exist, rather it requires
an (external) @emph{definition} for a module named @emph{foo} somewhere.)

A special attribute of every module is @ref{7a4,,__dict__}. This is the
dictionary containing the module’s symbol table. Modifying this dictionary will
actually change the module’s symbol table, but direct assignment to the
@ref{7a4,,__dict__} attribute is not possible (you can write
@code{m.__dict__['a'] = 1}, which defines @code{m.a} to be @code{1}, but you can’t write
@code{m.__dict__ = @{@}}).  Modifying @ref{7a4,,__dict__} directly is
not recommended.

Modules built into the interpreter are written like this: @code{<module 'sys'
(built-in)>}.  If loaded from a file, they are written as @code{<module 'os' from
'/usr/local/lib/pythonX.Y/os.pyc'>}.

@node Classes and Class Instances,Functions,Modules<2>,Other Built-in Types
@anchor{library/stdtypes classes-and-class-instances}@anchor{1b53}@anchor{library/stdtypes typesobjects}@anchor{1b54}
@subsubsection Classes and Class Instances


See @ref{1784,,Objects@comma{} values and types} and @ref{e72,,Class definitions} for these.

@node Functions,Methods,Classes and Class Instances,Other Built-in Types
@anchor{library/stdtypes functions}@anchor{1b55}@anchor{library/stdtypes typesfunctions}@anchor{1b56}
@subsubsection Functions


Function objects are created by function definitions.  The only operation on a
function object is to call it: @code{func(argument-list)}.

There are really two flavors of function objects: built-in functions and
user-defined functions.  Both support the same operation (to call the function),
but the implementation is different, hence the different object types.

See @ref{151c,,Function definitions} for more information.

@node Methods,Code Objects,Functions,Other Built-in Types
@anchor{library/stdtypes methods}@anchor{1b57}@anchor{library/stdtypes typesmethods}@anchor{1b58}
@subsubsection Methods


@geindex object; method

Methods are functions that are called using the attribute notation. There are
two flavors: built-in methods (such as @code{append()} on lists) and class
instance methods.  Built-in methods are described with the types that support
them.

If you access a method (a function defined in a class namespace) through an
instance, you get a special object: a @emph{bound method} (also called
@emph{instance method}) object. When called, it will add the @code{self} argument
to the argument list.  Bound methods have two special read-only attributes:
@code{m.__self__} is the object on which the method operates, and @code{m.__func__} is
the function implementing the method.  Calling @code{m(arg-1, arg-2, ..., arg-n)}
is completely equivalent to calling @code{m.__func__(m.__self__, arg-1, arg-2, ...,
arg-n)}.

Like function objects, bound method objects support getting arbitrary
attributes.  However, since method attributes are actually stored on the
underlying function object (@code{meth.__func__}), setting method attributes on
bound methods is disallowed.  Attempting to set an attribute on a method
results in an @ref{163,,AttributeError} being raised.  In order to set a method
attribute, you need to explicitly set it on the underlying function object:

@example
>>> class C:
...     def method(self):
...         pass
...
>>> c = C()
>>> c.method.whoami = 'my name is method'  # can't set on the method
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'method' object has no attribute 'whoami'
>>> c.method.__func__.whoami = 'my name is method'
>>> c.method.whoami
'my name is method'
@end example

See @ref{1787,,The standard type hierarchy} for more information.

@geindex object; code
@geindex code object

@node Code Objects,Type Objects,Methods,Other Built-in Types
@anchor{library/stdtypes bltin-code-objects}@anchor{1b59}@anchor{library/stdtypes code-objects}@anchor{1b5a}
@subsubsection Code Objects


@geindex built-in function; compile
@geindex __code__ (function object attribute)

Code objects are used by the implementation to represent “pseudo-compiled”
executable Python code such as a function body. They differ from function
objects because they don’t contain a reference to their global execution
environment.  Code objects are returned by the built-in @ref{4e2,,compile()} function
and can be extracted from function objects through their @code{__code__}
attribute. See also the @ref{19,,code} module.

Accessing @code{__code__} raises an @ref{1130,,auditing event}
@code{object.__getattr__} with arguments @code{obj} and @code{"__code__"}.

@geindex built-in function; exec
@geindex built-in function; eval

A code object can be executed or evaluated by passing it (instead of a source
string) to the @ref{34b,,exec()} or @ref{34a,,eval()}  built-in functions.

See @ref{1787,,The standard type hierarchy} for more information.

@node Type Objects,The Null Object,Code Objects,Other Built-in Types
@anchor{library/stdtypes bltin-type-objects}@anchor{1a62}@anchor{library/stdtypes type-objects}@anchor{1b5b}
@subsubsection Type Objects


@geindex built-in function; type
@geindex module; types

Type objects represent the various object types.  An object’s type is accessed
by the built-in function @ref{89e,,type()}.  There are no special operations on
types.  The standard module @ref{11f,,types} defines names for all standard built-in
types.

Types are written like this: @code{<class 'int'>}.

@node The Null Object,The Ellipsis Object,Type Objects,Other Built-in Types
@anchor{library/stdtypes bltin-null-object}@anchor{1b5c}@anchor{library/stdtypes the-null-object}@anchor{1b5d}
@subsubsection The Null Object


This object is returned by functions that don’t explicitly return a value.  It
supports no special operations.  There is exactly one null object, named
@code{None} (a built-in name).  @code{type(None)()} produces the same singleton.

It is written as @code{None}.

@geindex ...; ellipsis literal

@node The Ellipsis Object,The NotImplemented Object,The Null Object,Other Built-in Types
@anchor{library/stdtypes bltin-ellipsis-object}@anchor{1b5e}@anchor{library/stdtypes the-ellipsis-object}@anchor{1b5f}
@subsubsection The Ellipsis Object


This object is commonly used by slicing (see @ref{18c1,,Slicings}).  It supports no
special operations.  There is exactly one ellipsis object, named
@ref{1a67,,Ellipsis} (a built-in name).  @code{type(Ellipsis)()} produces the
@ref{1a67,,Ellipsis} singleton.

It is written as @code{Ellipsis} or @code{...}.

@node The NotImplemented Object,Boolean Values,The Ellipsis Object,Other Built-in Types
@anchor{library/stdtypes bltin-notimplemented-object}@anchor{1b60}@anchor{library/stdtypes the-notimplemented-object}@anchor{1b61}
@subsubsection The NotImplemented Object


This object is returned from comparisons and binary operations when they are
asked to operate on types they don’t support. See @ref{1900,,Comparisons} for more
information.  There is exactly one @code{NotImplemented} object.
@code{type(NotImplemented)()} produces the singleton instance.

It is written as @code{NotImplemented}.

@node Boolean Values,Internal Objects,The NotImplemented Object,Other Built-in Types
@anchor{library/stdtypes bltin-boolean-values}@anchor{1a3b}@anchor{library/stdtypes boolean-values}@anchor{1b62}
@subsubsection Boolean Values


Boolean values are the two constant objects @code{False} and @code{True}.  They are
used to represent truth values (although other values can also be considered
false or true).  In numeric contexts (for example when used as the argument to
an arithmetic operator), they behave like the integers 0 and 1, respectively.
The built-in function @ref{2f7,,bool()} can be used to convert any value to a
Boolean, if the value can be interpreted as a truth value (see section
@ref{1a3a,,Truth Value Testing} above).

@geindex False
@geindex True
@geindex Boolean; values

They are written as @code{False} and @code{True}, respectively.

@node Internal Objects,,Boolean Values,Other Built-in Types
@anchor{library/stdtypes internal-objects}@anchor{1b63}@anchor{library/stdtypes typesinternal}@anchor{1b64}
@subsubsection Internal Objects


See @ref{1787,,The standard type hierarchy} for this information.  It describes stack frame objects,
traceback objects, and slice objects.

@node Special Attributes,,Other Built-in Types,Built-in Types
@anchor{library/stdtypes special-attributes}@anchor{1b65}@anchor{library/stdtypes specialattrs}@anchor{1b66}
@subsection Special Attributes


The implementation adds a few special read-only attributes to several object
types, where they are relevant.  Some of these are not reported by the
@ref{f2f,,dir()} built-in function.

@geindex __dict__ (object attribute)
@anchor{library/stdtypes object __dict__}@anchor{7a4}
@deffn {Attribute} object.__dict__

A dictionary or other mapping object used to store an object’s (writable)
attributes.
@end deffn

@geindex __class__ (instance attribute)
@anchor{library/stdtypes instance __class__}@anchor{fff}
@deffn {Attribute} instance.__class__

The class to which a class instance belongs.
@end deffn

@geindex __bases__ (class attribute)
@anchor{library/stdtypes class __bases__}@anchor{ffe}
@deffn {Attribute} class.__bases__

The tuple of base classes of a class object.
@end deffn

@geindex __name__ (definition attribute)
@anchor{library/stdtypes definition __name__}@anchor{e70}
@deffn {Attribute} definition.__name__

The name of the class, function, method, descriptor, or
generator instance.
@end deffn

@geindex __qualname__ (definition attribute)
@anchor{library/stdtypes definition __qualname__}@anchor{1790}
@deffn {Attribute} definition.__qualname__

The @ref{11bf,,qualified name} of the class, function, method, descriptor,
or generator instance.


New in version 3.3.

@end deffn

@geindex __mro__ (class attribute)
@anchor{library/stdtypes class __mro__}@anchor{1a61}
@deffn {Attribute} class.__mro__

This attribute is a tuple of classes that are considered when looking for
base classes during method resolution.
@end deffn

@geindex mro() (class method)
@anchor{library/stdtypes class mro}@anchor{1b67}
@deffn {Method} class.mro ()

This method can be overridden by a metaclass to customize the method
resolution order for its instances.  It is called at class instantiation, and
its result is stored in @ref{1a61,,__mro__}.
@end deffn

@geindex __subclasses__() (class method)
@anchor{library/stdtypes class __subclasses__}@anchor{1b68}
@deffn {Method} class.__subclasses__ ()

Each class keeps a list of weak references to its immediate subclasses.  This
method returns a list of all those references still alive.  The list is in
definition order.  Example:

@example
>>> int.__subclasses__()
[<class 'bool'>]
@end example
@end deffn

@node Built-in Exceptions,Text Processing Services,Built-in Types,The Python Standard Library
@anchor{library/exceptions doc}@anchor{1b69}@anchor{library/exceptions bltin-exceptions}@anchor{10e8}@anchor{library/exceptions built-in-exceptions}@anchor{1b6a}
@section Built-in Exceptions


@geindex statement; try
@geindex statement; except

In Python, all exceptions must be instances of a class that derives from
@ref{15f,,BaseException}.  In a @ref{f6d,,try} statement with an @ref{16e,,except}
clause that mentions a particular class, that clause also handles any exception
classes derived from that class (but not exception classes from which @emph{it} is
derived).  Two exception classes that are not related via subclassing are never
equivalent, even if they have the same name.

@geindex statement; raise

The built-in exceptions listed below can be generated by the interpreter or
built-in functions.  Except where mentioned, they have an “associated value”
indicating the detailed cause of the error.  This may be a string or a tuple of
several items of information (e.g., an error code and a string explaining the
code).  The associated value is usually passed as arguments to the exception
class’s constructor.

User code can raise built-in exceptions.  This can be used to test an exception
handler or to report an error condition “just like” the situation in which the
interpreter raises the same exception; but beware that there is nothing to
prevent user code from raising an inappropriate error.

The built-in exception classes can be subclassed to define new exceptions;
programmers are encouraged to derive new exceptions from the @ref{4d8,,Exception}
class or one of its subclasses, and not from @ref{15f,,BaseException}.  More
information on defining exceptions is available in the Python Tutorial under
@ref{1573,,User-defined Exceptions}.

@menu
* Exception context:: 
* Inheriting from built-in exceptions:: 
* Base classes:: 
* Concrete exceptions:: 
* Warnings:: 
* Exception groups:: 
* Exception hierarchy:: 

@end menu

@node Exception context,Inheriting from built-in exceptions,,Built-in Exceptions
@anchor{library/exceptions exception-context}@anchor{1b6b}
@subsection Exception context


When raising a new exception while another exception
is already being handled, the new exception’s
@code{__context__} attribute is automatically set to the handled
exception.  An exception may be handled when an @ref{16e,,except} or
@ref{4c1,,finally} clause, or a @ref{168,,with} statement, is used.

This implicit exception context can be
supplemented with an explicit cause by using @code{from} with
@ref{16d,,raise}:

@example
raise new_exc from original_exc
@end example

The expression following @ref{16d,,from} must be an exception or @code{None}. It
will be set as @code{__cause__} on the raised exception. Setting
@code{__cause__} also implicitly sets the @code{__suppress_context__}
attribute to @code{True}, so that using @code{raise new_exc from None}
effectively replaces the old exception with the new one for display
purposes (e.g. converting @ref{5bb,,KeyError} to @ref{163,,AttributeError}), while
leaving the old exception available in @code{__context__} for introspection
when debugging.

The default traceback display code shows these chained exceptions in
addition to the traceback for the exception itself. An explicitly chained
exception in @code{__cause__} is always shown when present. An implicitly
chained exception in @code{__context__} is shown only if @code{__cause__}
is @ref{182,,None} and @code{__suppress_context__} is false.

In either case, the exception itself is always shown after any chained
exceptions so that the final line of the traceback always shows the last
exception that was raised.

@node Inheriting from built-in exceptions,Base classes,Exception context,Built-in Exceptions
@anchor{library/exceptions inheriting-from-built-in-exceptions}@anchor{1b6c}
@subsection Inheriting from built-in exceptions


User code can create subclasses that inherit from an exception type.
It’s recommended to only subclass one exception type at a time to avoid
any possible conflicts between how the bases handle the @code{args}
attribute, as well as due to possible memory layout incompatibilities.

@strong{CPython implementation detail:} Most built-in exceptions are implemented in C for efficiency, see:
Objects/exceptions.c@footnote{https://github.com/python/cpython/tree/main/Objects/exceptions.c}.  Some have custom memory layouts
which makes it impossible to create a subclass that inherits from
multiple exception types. The memory layout of a type is an implementation
detail and might change between Python versions, leading to new
conflicts in the future.  Therefore, it’s recommended to avoid
subclassing multiple exception types altogether.

@node Base classes,Concrete exceptions,Inheriting from built-in exceptions,Built-in Exceptions
@anchor{library/exceptions base-classes}@anchor{1b6d}
@subsection Base classes


The following exceptions are used mostly as base classes for other exceptions.

@geindex BaseException
@anchor{library/exceptions BaseException}@anchor{15f}
@deffn {Exception} BaseException

The base class for all built-in exceptions.  It is not meant to be directly
inherited by user-defined classes (for that, use @ref{4d8,,Exception}).  If
@ref{2ef,,str()} is called on an instance of this class, the representation of
the argument(s) to the instance are returned, or the empty string when
there were no arguments.

@geindex args (BaseException attribute)
@anchor{library/exceptions BaseException args}@anchor{1b6e}
@deffn {Attribute} args

The tuple of arguments given to the exception constructor.  Some built-in
exceptions (like @ref{2a1,,OSError}) expect a certain number of arguments and
assign a special meaning to the elements of this tuple, while others are
usually called only with a single string giving an error message.
@end deffn

@geindex with_traceback() (BaseException method)
@anchor{library/exceptions BaseException with_traceback}@anchor{195b}
@deffn {Method} with_traceback (tb)

This method sets @emph{tb} as the new traceback for the exception and returns
the exception object.  It was more commonly used before the exception
chaining features of 
@geindex Python Enhancement Proposals; PEP 3134
PEP 3134@footnote{https://www.python.org/dev/peps/pep-3134} became available.  The following example
shows how we can convert an instance of @code{SomeException} into an
instance of @code{OtherException} while preserving the traceback.  Once
raised, the current frame is pushed onto the traceback of the
@code{OtherException}, as would have happened to the traceback of the
original @code{SomeException} had we allowed it to propagate to the caller.

@example
try:
    ...
except SomeException:
    tb = sys.exc_info()[2]
    raise OtherException(...).with_traceback(tb)
@end example
@end deffn

@geindex __note__ (BaseException attribute)
@anchor{library/exceptions BaseException __note__}@anchor{1b6f}
@deffn {Attribute} __note__

A mutable field which is @ref{182,,None} by default and can be set to a string.
If it is not @ref{182,,None}, it is included in the traceback. This field can
be used to enrich exceptions after they have been caught.
@end deffn


New in version 3.11.

@end deffn

@geindex Exception
@anchor{library/exceptions Exception}@anchor{4d8}
@deffn {Exception} Exception

All built-in, non-system-exiting exceptions are derived from this class.  All
user-defined exceptions should also be derived from this class.
@end deffn

@geindex ArithmeticError
@anchor{library/exceptions ArithmeticError}@anchor{1b70}
@deffn {Exception} ArithmeticError

The base class for those built-in exceptions that are raised for various
arithmetic errors: @ref{347,,OverflowError}, @ref{43c,,ZeroDivisionError},
@ref{1b71,,FloatingPointError}.
@end deffn

@geindex BufferError
@anchor{library/exceptions BufferError}@anchor{1b72}
@deffn {Exception} BufferError

Raised when a @ref{1a3e,,buffer} related operation cannot be
performed.
@end deffn

@geindex LookupError
@anchor{library/exceptions LookupError}@anchor{1ab5}
@deffn {Exception} LookupError

The base class for the exceptions that are raised when a key or index used on
a mapping or sequence is invalid: @ref{1066,,IndexError}, @ref{5bb,,KeyError}.  This
can be raised directly by @ref{466,,codecs.lookup()}.
@end deffn

@node Concrete exceptions,Warnings,Base classes,Built-in Exceptions
@anchor{library/exceptions concrete-exceptions}@anchor{1b73}
@subsection Concrete exceptions


The following exceptions are the exceptions that are usually raised.

@geindex AssertionError
@anchor{library/exceptions AssertionError}@anchor{46e}
@deffn {Exception} AssertionError

@geindex statement; assert

Raised when an @ref{46d,,assert} statement fails.
@end deffn

@geindex AttributeError
@anchor{library/exceptions AttributeError}@anchor{163}
@deffn {Exception} AttributeError

Raised when an attribute reference (see @ref{18b9,,Attribute references}) or
assignment fails.  (When an object does not support attribute references or
attribute assignments at all, @ref{162,,TypeError} is raised.)

The @code{name} and @code{obj} attributes can be set using keyword-only
arguments to the constructor. When set they represent the name of the attribute
that was attempted to be accessed and the object that was accessed for said
attribute, respectively.


Changed in version 3.10: Added the @code{name} and @code{obj} attributes.

@end deffn

@geindex EOFError
@anchor{library/exceptions EOFError}@anchor{e74}
@deffn {Exception} EOFError

Raised when the @ref{e73,,input()} function hits an end-of-file condition (EOF)
without reading any data. (N.B.: the @code{io.IOBase.read()} and
@ref{1b74,,io.IOBase.readline()} methods return an empty string when they hit EOF.)
@end deffn

@geindex FloatingPointError
@anchor{library/exceptions FloatingPointError}@anchor{1b71}
@deffn {Exception} FloatingPointError

Not currently used.
@end deffn

@geindex GeneratorExit
@anchor{library/exceptions GeneratorExit}@anchor{f2e}
@deffn {Exception} GeneratorExit

Raised when a @ref{bf7,,generator} or @ref{3cf,,coroutine} is closed;
see @ref{181f,,generator.close()} and @ref{181e,,coroutine.close()}.  It
directly inherits from @ref{15f,,BaseException} instead of @ref{4d8,,Exception} since
it is technically not an error.
@end deffn

@geindex ImportError
@anchor{library/exceptions ImportError}@anchor{3b8}
@deffn {Exception} ImportError

Raised when the @ref{e35,,import} statement has troubles trying to
load a module.  Also raised when the “from list” in @code{from ... import}
has a name that cannot be found.

The @code{name} and @code{path} attributes can be set using keyword-only
arguments to the constructor. When set they represent the name of the module
that was attempted to be imported and the path to any file which triggered
the exception, respectively.


Changed in version 3.3: Added the @code{name} and @code{path} attributes.

@end deffn

@geindex ModuleNotFoundError
@anchor{library/exceptions ModuleNotFoundError}@anchor{66b}
@deffn {Exception} ModuleNotFoundError

A subclass of @ref{3b8,,ImportError} which is raised by @ref{e35,,import}
when a module could not be located.  It is also raised when @code{None}
is found in @ref{11f7,,sys.modules}.


New in version 3.6.

@end deffn

@geindex IndexError
@anchor{library/exceptions IndexError}@anchor{1066}
@deffn {Exception} IndexError

Raised when a sequence subscript is out of range.  (Slice indices are
silently truncated to fall in the allowed range; if an index is not an
integer, @ref{162,,TypeError} is raised.)

@c XXX xref to sequences
@end deffn

@geindex KeyError
@anchor{library/exceptions KeyError}@anchor{5bb}
@deffn {Exception} KeyError

Raised when a mapping (dictionary) key is not found in the set of existing keys.

@c XXX xref to mapping objects?
@end deffn

@geindex KeyboardInterrupt
@anchor{library/exceptions KeyboardInterrupt}@anchor{4ca}
@deffn {Exception} KeyboardInterrupt

Raised when the user hits the interrupt key (normally @code{Control-C} or
@code{Delete}).  During execution, a check for interrupts is made
regularly. The exception inherits from @ref{15f,,BaseException} so as to not be
accidentally caught by code that catches @ref{4d8,,Exception} and thus prevent
the interpreter from exiting.
@end deffn

@geindex MemoryError
@anchor{library/exceptions MemoryError}@anchor{10de}
@deffn {Exception} MemoryError

Raised when an operation runs out of memory but the situation may still be
rescued (by deleting some objects).  The associated value is a string indicating
what kind of (internal) operation ran out of memory. Note that because of the
underlying memory management architecture (C’s @code{malloc()} function), the
interpreter may not always be able to completely recover from this situation; it
nevertheless raises an exception so that a stack traceback can be printed, in
case a run-away program was the cause.
@end deffn

@geindex NameError
@anchor{library/exceptions NameError}@anchor{21f}
@deffn {Exception} NameError

Raised when a local or global name is not found.  This applies only to
unqualified names.  The associated value is an error message that includes the
name that could not be found.

The @code{name} attribute can be set using a keyword-only argument to the
constructor. When set it represent the name of the variable that was attempted
to be accessed.


Changed in version 3.10: Added the @code{name} attribute.

@end deffn

@geindex NotImplementedError
@anchor{library/exceptions NotImplementedError}@anchor{8a3}
@deffn {Exception} NotImplementedError

This exception is derived from @ref{5af,,RuntimeError}.  In user defined base
classes, abstract methods should raise this exception when they require
derived classes to override the method, or while the class is being
developed to indicate that the real implementation still needs to be added.

@cartouche
@quotation Note 
It should not be used to indicate that an operator or method is not
meant to be supported at all – in that case either leave the operator /
method undefined or, if a subclass, set it to @ref{182,,None}.
@end quotation
@end cartouche

@cartouche
@quotation Note 
@code{NotImplementedError} and @code{NotImplemented} are not interchangeable,
even though they have similar names and purposes.  See
@ref{24c,,NotImplemented} for details on when to use it.
@end quotation
@end cartouche
@end deffn

@geindex OSError
@anchor{library/exceptions OSError}@anchor{2a1}
@deffn {Exception} OSError ([arg])

@deffnx {Exception} OSError (errno, strerror[, filename[, winerror[, filename2]]])

@geindex module; errno

This exception is raised when a system function returns a system-related
error, including I/O failures such as “file not found” or “disk full”
(not for illegal argument types or other incidental errors).

The second form of the constructor sets the corresponding attributes,
described below.  The attributes default to @ref{182,,None} if not
specified.  For backwards compatibility, if three arguments are passed,
the @ref{1b6e,,args} attribute contains only a 2-tuple
of the first two constructor arguments.

The constructor often actually returns a subclass of @ref{2a1,,OSError}, as
described in @ref{1b75,,OS exceptions} below.  The particular subclass depends on
the final @ref{1b76,,errno} value.  This behaviour only occurs when
constructing @ref{2a1,,OSError} directly or via an alias, and is not
inherited when subclassing.

@geindex errno (OSError attribute)
@anchor{library/exceptions OSError errno}@anchor{1b76}
@deffn {Attribute} errno

A numeric error code from the C variable @code{errno}.
@end deffn

@geindex winerror (OSError attribute)
@anchor{library/exceptions OSError winerror}@anchor{1b77}
@deffn {Attribute} winerror

Under Windows, this gives you the native
Windows error code.  The @ref{1b76,,errno} attribute is then an approximate
translation, in POSIX terms, of that native error code.

Under Windows, if the @emph{winerror} constructor argument is an integer,
the @ref{1b76,,errno} attribute is determined from the Windows error code,
and the @emph{errno} argument is ignored.  On other platforms, the
@emph{winerror} argument is ignored, and the @ref{1b77,,winerror} attribute
does not exist.
@end deffn

@geindex strerror (OSError attribute)
@anchor{library/exceptions OSError strerror}@anchor{1b78}
@deffn {Attribute} strerror

The corresponding error message, as provided by
the operating system.  It is formatted by the C
functions @code{perror()} under POSIX, and @code{FormatMessage()}
under Windows.
@end deffn

@geindex filename (OSError attribute)
@geindex filename2 (OSError attribute)
@anchor{library/exceptions OSError filename}@anchor{1b79}
@deffn {Attribute} filename
@anchor{library/exceptions OSError filename2}@anchor{1b7a}
@deffnx {Attribute} filename2

For exceptions that involve a file system path (such as @ref{1c8,,open()} or
@ref{c89,,os.unlink()}), @ref{1b79,,filename} is the file name passed to the function.
For functions that involve two file system paths (such as
@ref{c85,,os.rename()}), @ref{1b7a,,filename2} corresponds to the second
file name passed to the function.
@end deffn


Changed in version 3.3: @ref{be9,,EnvironmentError}, @ref{be8,,IOError}, @ref{bea,,WindowsError},
@ref{beb,,socket.error}, @ref{bec,,select.error} and
@code{mmap.error} have been merged into @ref{2a1,,OSError}, and the
constructor may return a subclass.



Changed in version 3.4: The @ref{1b79,,filename} attribute is now the original file name passed to
the function, instead of the name encoded to or decoded from the
@ref{1628,,filesystem encoding and error handler}. Also, the @emph{filename2}
constructor argument and attribute was added.

@end deffn

@geindex OverflowError
@anchor{library/exceptions OverflowError}@anchor{347}
@deffn {Exception} OverflowError

Raised when the result of an arithmetic operation is too large to be
represented.  This cannot occur for integers (which would rather raise
@ref{10de,,MemoryError} than give up).  However, for historical reasons,
OverflowError is sometimes raised for integers that are outside a required
range.   Because of the lack of standardization of floating point exception
handling in C, most floating point operations are not checked.
@end deffn

@geindex RecursionError
@anchor{library/exceptions RecursionError}@anchor{8c4}
@deffn {Exception} RecursionError

This exception is derived from @ref{5af,,RuntimeError}.  It is raised when the
interpreter detects that the maximum recursion depth (see
@ref{106a,,sys.getrecursionlimit()}) is exceeded.


New in version 3.5: Previously, a plain @ref{5af,,RuntimeError} was raised.

@end deffn

@geindex ReferenceError
@anchor{library/exceptions ReferenceError}@anchor{1043}
@deffn {Exception} ReferenceError

This exception is raised when a weak reference proxy, created by the
@ref{567,,weakref.proxy()} function, is used to access an attribute of the referent
after it has been garbage collected. For more information on weak references,
see the @ref{12f,,weakref} module.
@end deffn

@geindex RuntimeError
@anchor{library/exceptions RuntimeError}@anchor{5af}
@deffn {Exception} RuntimeError

Raised when an error is detected that doesn’t fall in any of the other
categories.  The associated value is a string indicating what precisely went
wrong.
@end deffn

@geindex StopIteration
@anchor{library/exceptions StopIteration}@anchor{736}
@deffn {Exception} StopIteration

Raised by built-in function @ref{252,,next()} and an @ref{11c3,,iterator}'s
@ref{e6d,,__next__()} method to signal that there are no further
items produced by the iterator.

The exception object has a single attribute @code{value}, which is
given as an argument when constructing the exception, and defaults
to @ref{182,,None}.

When a @ref{bf7,,generator} or @ref{3cf,,coroutine} function
returns, a new @ref{736,,StopIteration} instance is
raised, and the value returned by the function is used as the
@code{value} parameter to the constructor of the exception.

If a generator code directly or indirectly raises @ref{736,,StopIteration},
it is converted into a @ref{5af,,RuntimeError} (retaining the
@ref{736,,StopIteration} as the new exception’s cause).


Changed in version 3.3: Added @code{value} attribute and the ability for generator functions to
use it to return a value.



Changed in version 3.5: Introduced the RuntimeError transformation via
@code{from __future__ import generator_stop}, see 
@geindex Python Enhancement Proposals; PEP 479
PEP 479@footnote{https://www.python.org/dev/peps/pep-0479}.



Changed in version 3.7: Enable 
@geindex Python Enhancement Proposals; PEP 479
PEP 479@footnote{https://www.python.org/dev/peps/pep-0479} for all code by default: a @ref{736,,StopIteration}
error raised in a generator is transformed into a @ref{5af,,RuntimeError}.

@end deffn

@geindex StopAsyncIteration
@anchor{library/exceptions StopAsyncIteration}@anchor{12da}
@deffn {Exception} StopAsyncIteration

Must be raised by @code{__anext__()} method of an
@ref{8d2,,asynchronous iterator} object to stop the iteration.


New in version 3.5.

@end deffn

@geindex SyntaxError
@anchor{library/exceptions SyntaxError}@anchor{21a}
@deffn {Exception} SyntaxError (message, details)

Raised when the parser encounters a syntax error.  This may occur in an
@ref{e35,,import} statement, in a call to the built-in functions
@ref{4e2,,compile()}, @ref{34b,,exec()},
or @ref{34a,,eval()}, or when reading the initial script or standard input
(also interactively).

The @ref{2ef,,str()} of the exception instance returns only the error message.
Details is a tuple whose members are also available as separate attributes.

@geindex filename (SyntaxError attribute)
@anchor{library/exceptions SyntaxError filename}@anchor{1b7b}
@deffn {Attribute} filename

The name of the file the syntax error occurred in.
@end deffn

@geindex lineno (SyntaxError attribute)
@anchor{library/exceptions SyntaxError lineno}@anchor{1b7c}
@deffn {Attribute} lineno

Which line number in the file the error occurred in. This is
1-indexed: the first line in the file has a @code{lineno} of 1.
@end deffn

@geindex offset (SyntaxError attribute)
@anchor{library/exceptions SyntaxError offset}@anchor{1b7d}
@deffn {Attribute} offset

The column in the line where the error occurred. This is
1-indexed: the first character in the line has an @code{offset} of 1.
@end deffn

@geindex text (SyntaxError attribute)
@anchor{library/exceptions SyntaxError text}@anchor{1b7e}
@deffn {Attribute} text

The source code text involved in the error.
@end deffn

@geindex end_lineno (SyntaxError attribute)
@anchor{library/exceptions SyntaxError end_lineno}@anchor{1b7f}
@deffn {Attribute} end_lineno

Which line number in the file the error occurred ends in. This is
1-indexed: the first line in the file has a @code{lineno} of 1.
@end deffn

@geindex end_offset (SyntaxError attribute)
@anchor{library/exceptions SyntaxError end_offset}@anchor{1b80}
@deffn {Attribute} end_offset

The column in the end line where the error occurred finishes. This is
1-indexed: the first character in the line has an @code{offset} of 1.
@end deffn

For errors in f-string fields, the message is prefixed by “f-string: ”
and the offsets are offsets in a text constructed from the replacement
expression.  For example, compiling f’Bad @{a b@} field’ results in this
args attribute: (‘f-string: …’, (‘’, 1, 2, ‘(a b)n’, 1, 5)).


Changed in version 3.10: Added the @ref{1b7f,,end_lineno} and @ref{1b80,,end_offset} attributes.

@end deffn

@geindex IndentationError
@anchor{library/exceptions IndentationError}@anchor{21c}
@deffn {Exception} IndentationError

Base class for syntax errors related to incorrect indentation.  This is a
subclass of @ref{21a,,SyntaxError}.
@end deffn

@geindex TabError
@anchor{library/exceptions TabError}@anchor{1068}
@deffn {Exception} TabError

Raised when indentation contains an inconsistent use of tabs and spaces.
This is a subclass of @ref{21c,,IndentationError}.
@end deffn

@geindex SystemError
@anchor{library/exceptions SystemError}@anchor{891}
@deffn {Exception} SystemError

Raised when the interpreter finds an internal error, but the situation does not
look so serious to cause it to abandon all hope. The associated value is a
string indicating what went wrong (in low-level terms).

You should report this to the author or maintainer of your Python interpreter.
Be sure to report the version of the Python interpreter (@code{sys.version}; it is
also printed at the start of an interactive Python session), the exact error
message (the exception’s associated value) and if possible the source of the
program that triggered the error.
@end deffn

@geindex SystemExit
@anchor{library/exceptions SystemExit}@anchor{892}
@deffn {Exception} SystemExit

This exception is raised by the @ref{ee4,,sys.exit()} function.  It inherits from
@ref{15f,,BaseException} instead of @ref{4d8,,Exception} so that it is not accidentally
caught by code that catches @ref{4d8,,Exception}.  This allows the exception to
properly propagate up and cause the interpreter to exit.  When it is not
handled, the Python interpreter exits; no stack traceback is printed.  The
constructor accepts the same optional argument passed to @ref{ee4,,sys.exit()}.
If the value is an integer, it specifies the system exit status (passed to
C’s @code{exit()} function); if it is @code{None}, the exit status is zero; if
it has another type (such as a string), the object’s value is printed and
the exit status is one.

A call to @ref{ee4,,sys.exit()} is translated into an exception so that clean-up
handlers (@ref{4c1,,finally} clauses of @ref{f6d,,try} statements) can be
executed, and so that a debugger can execute a script without running the risk
of losing control.  The @ref{1b81,,os._exit()} function can be used if it is
absolutely positively necessary to exit immediately (for example, in the child
process after a call to @ref{bb9,,os.fork()}).

@geindex code (SystemExit attribute)
@anchor{library/exceptions SystemExit code}@anchor{1b82}
@deffn {Attribute} code

The exit status or error message that is passed to the constructor.
(Defaults to @code{None}.)
@end deffn
@end deffn

@geindex TypeError
@anchor{library/exceptions TypeError}@anchor{162}
@deffn {Exception} TypeError

Raised when an operation or function is applied to an object of inappropriate
type.  The associated value is a string giving details about the type mismatch.

This exception may be raised by user code to indicate that an attempted
operation on an object is not supported, and is not meant to be. If an object
is meant to support a given operation but has not yet provided an
implementation, @ref{8a3,,NotImplementedError} is the proper exception to raise.

Passing arguments of the wrong type (e.g. passing a @ref{43a,,list} when an
@ref{1b1,,int} is expected) should result in a @ref{162,,TypeError}, but passing
arguments with the wrong value (e.g. a number outside expected boundaries)
should result in a @ref{17c,,ValueError}.
@end deffn

@geindex UnboundLocalError
@anchor{library/exceptions UnboundLocalError}@anchor{1067}
@deffn {Exception} UnboundLocalError

Raised when a reference is made to a local variable in a function or method, but
no value has been bound to that variable.  This is a subclass of
@ref{21f,,NameError}.
@end deffn

@geindex UnicodeError
@anchor{library/exceptions UnicodeError}@anchor{e49}
@deffn {Exception} UnicodeError

Raised when a Unicode-related encoding or decoding error occurs.  It is a
subclass of @ref{17c,,ValueError}.

@ref{e49,,UnicodeError} has attributes that describe the encoding or decoding
error.  For example, @code{err.object[err.start:err.end]} gives the particular
invalid input that the codec failed on.

@geindex encoding (UnicodeError attribute)
@anchor{library/exceptions UnicodeError encoding}@anchor{1b83}
@deffn {Attribute} encoding

The name of the encoding that raised the error.
@end deffn

@geindex reason (UnicodeError attribute)
@anchor{library/exceptions UnicodeError reason}@anchor{1b84}
@deffn {Attribute} reason

A string describing the specific codec error.
@end deffn

@geindex object (UnicodeError attribute)
@anchor{library/exceptions UnicodeError object}@anchor{1b85}
@deffn {Attribute} object

The object the codec was attempting to encode or decode.
@end deffn

@geindex start (UnicodeError attribute)
@anchor{library/exceptions UnicodeError start}@anchor{1b86}
@deffn {Attribute} start

The first index of invalid data in @ref{5a5,,object}.
@end deffn

@geindex end (UnicodeError attribute)
@anchor{library/exceptions UnicodeError end}@anchor{1b87}
@deffn {Attribute} end

The index after the last invalid data in @ref{5a5,,object}.
@end deffn
@end deffn

@geindex UnicodeEncodeError
@anchor{library/exceptions UnicodeEncodeError}@anchor{185}
@deffn {Exception} UnicodeEncodeError

Raised when a Unicode-related error occurs during encoding.  It is a subclass of
@ref{e49,,UnicodeError}.
@end deffn

@geindex UnicodeDecodeError
@anchor{library/exceptions UnicodeDecodeError}@anchor{51f}
@deffn {Exception} UnicodeDecodeError

Raised when a Unicode-related error occurs during decoding.  It is a subclass of
@ref{e49,,UnicodeError}.
@end deffn

@geindex UnicodeTranslateError
@anchor{library/exceptions UnicodeTranslateError}@anchor{1b88}
@deffn {Exception} UnicodeTranslateError

Raised when a Unicode-related error occurs during translating.  It is a subclass
of @ref{e49,,UnicodeError}.
@end deffn

@geindex ValueError
@anchor{library/exceptions ValueError}@anchor{17c}
@deffn {Exception} ValueError

Raised when an operation or function receives an argument that has the
right type but an inappropriate value, and the situation is not described by a
more precise exception such as @ref{1066,,IndexError}.
@end deffn

@geindex ZeroDivisionError
@anchor{library/exceptions ZeroDivisionError}@anchor{43c}
@deffn {Exception} ZeroDivisionError

Raised when the second argument of a division or modulo operation is zero.  The
associated value is a string indicating the type of the operands and the
operation.
@end deffn

The following exceptions are kept for compatibility with previous versions;
starting from Python 3.3, they are aliases of @ref{2a1,,OSError}.

@geindex EnvironmentError
@anchor{library/exceptions EnvironmentError}@anchor{be9}
@deffn {Exception} EnvironmentError
@end deffn

@geindex IOError
@anchor{library/exceptions IOError}@anchor{be8}
@deffn {Exception} IOError
@end deffn

@geindex WindowsError
@anchor{library/exceptions WindowsError}@anchor{bea}
@deffn {Exception} WindowsError

Only available on Windows.
@end deffn

@menu
* OS exceptions:: 

@end menu

@node OS exceptions,,,Concrete exceptions
@anchor{library/exceptions os-exceptions}@anchor{1b75}
@subsubsection OS exceptions


The following exceptions are subclasses of @ref{2a1,,OSError}, they get raised
depending on the system error code.

@geindex BlockingIOError
@anchor{library/exceptions BlockingIOError}@anchor{bed}
@deffn {Exception} BlockingIOError

Raised when an operation would block on an object (e.g. socket) set
for non-blocking operation.
Corresponds to @code{errno} @code{EAGAIN}, @code{EALREADY},
@code{EWOULDBLOCK} and @code{EINPROGRESS}.

In addition to those of @ref{2a1,,OSError}, @ref{bed,,BlockingIOError} can have
one more attribute:

@geindex characters_written (BlockingIOError attribute)
@anchor{library/exceptions BlockingIOError characters_written}@anchor{1b89}
@deffn {Attribute} characters_written

An integer containing the number of characters written to the stream
before it blocked.  This attribute is available when using the
buffered I/O classes from the @ref{9d,,io} module.
@end deffn
@end deffn

@geindex ChildProcessError
@anchor{library/exceptions ChildProcessError}@anchor{bee}
@deffn {Exception} ChildProcessError

Raised when an operation on a child process failed.
Corresponds to @code{errno} @code{ECHILD}.
@end deffn

@geindex ConnectionError
@anchor{library/exceptions ConnectionError}@anchor{969}
@deffn {Exception} ConnectionError

A base class for connection-related issues.

Subclasses are @ref{bf2,,BrokenPipeError}, @ref{bf3,,ConnectionAbortedError},
@ref{bf4,,ConnectionRefusedError} and @ref{bf5,,ConnectionResetError}.
@end deffn

@geindex BrokenPipeError
@anchor{library/exceptions BrokenPipeError}@anchor{bf2}
@deffn {Exception} BrokenPipeError

A subclass of @ref{969,,ConnectionError}, raised when trying to write on a
pipe while the other end has been closed, or trying to write on a socket
which has been shutdown for writing.
Corresponds to @code{errno} @code{EPIPE} and @code{ESHUTDOWN}.
@end deffn

@geindex ConnectionAbortedError
@anchor{library/exceptions ConnectionAbortedError}@anchor{bf3}
@deffn {Exception} ConnectionAbortedError

A subclass of @ref{969,,ConnectionError}, raised when a connection attempt
is aborted by the peer.
Corresponds to @code{errno} @code{ECONNABORTED}.
@end deffn

@geindex ConnectionRefusedError
@anchor{library/exceptions ConnectionRefusedError}@anchor{bf4}
@deffn {Exception} ConnectionRefusedError

A subclass of @ref{969,,ConnectionError}, raised when a connection attempt
is refused by the peer.
Corresponds to @code{errno} @code{ECONNREFUSED}.
@end deffn

@geindex ConnectionResetError
@anchor{library/exceptions ConnectionResetError}@anchor{bf5}
@deffn {Exception} ConnectionResetError

A subclass of @ref{969,,ConnectionError}, raised when a connection is
reset by the peer.
Corresponds to @code{errno} @code{ECONNRESET}.
@end deffn

@geindex FileExistsError
@anchor{library/exceptions FileExistsError}@anchor{bb1}
@deffn {Exception} FileExistsError

Raised when trying to create a file or directory which already exists.
Corresponds to @code{errno} @code{EEXIST}.
@end deffn

@geindex FileNotFoundError
@anchor{library/exceptions FileNotFoundError}@anchor{a37}
@deffn {Exception} FileNotFoundError

Raised when a file or directory is requested but doesn’t exist.
Corresponds to @code{errno} @code{ENOENT}.
@end deffn

@geindex InterruptedError
@anchor{library/exceptions InterruptedError}@anchor{8e3}
@deffn {Exception} InterruptedError

Raised when a system call is interrupted by an incoming signal.
Corresponds to @code{errno} @ref{8e2,,EINTR}.


Changed in version 3.5: Python now retries system calls when a syscall is interrupted by a
signal, except if the signal handler raises an exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475}
for the rationale), instead of raising @ref{8e3,,InterruptedError}.

@end deffn

@geindex IsADirectoryError
@anchor{library/exceptions IsADirectoryError}@anchor{bef}
@deffn {Exception} IsADirectoryError

Raised when a file operation (such as @ref{c84,,os.remove()}) is requested
on a directory.
Corresponds to @code{errno} @code{EISDIR}.
@end deffn

@geindex NotADirectoryError
@anchor{library/exceptions NotADirectoryError}@anchor{bf0}
@deffn {Exception} NotADirectoryError

Raised when a directory operation (such as @ref{c8d,,os.listdir()}) is requested on
something which is not a directory.  On most POSIX platforms, it may also be
raised if an operation attempts to open or traverse a non-directory file as if
it were a directory.
Corresponds to @code{errno} @code{ENOTDIR}.
@end deffn

@geindex PermissionError
@anchor{library/exceptions PermissionError}@anchor{895}
@deffn {Exception} PermissionError

Raised when trying to run an operation without the adequate access
rights - for example filesystem permissions.
Corresponds to @code{errno} @code{EACCES} and @code{EPERM}.
@end deffn

@geindex ProcessLookupError
@anchor{library/exceptions ProcessLookupError}@anchor{bf1}
@deffn {Exception} ProcessLookupError

Raised when a given process doesn’t exist.
Corresponds to @code{errno} @code{ESRCH}.
@end deffn

@geindex TimeoutError
@anchor{library/exceptions TimeoutError}@anchor{2b8}
@deffn {Exception} TimeoutError

Raised when a system function timed out at the system level.
Corresponds to @code{errno} @code{ETIMEDOUT}.
@end deffn


New in version 3.3: All the above @ref{2a1,,OSError} subclasses were added.



@subsubheading See also


@geindex Python Enhancement Proposals; PEP 3151
PEP 3151@footnote{https://www.python.org/dev/peps/pep-3151} - Reworking the OS and IO exception hierarchy


@node Warnings,Exception groups,Concrete exceptions,Built-in Exceptions
@anchor{library/exceptions warning-categories-as-exceptions}@anchor{1b8a}@anchor{library/exceptions warnings}@anchor{1b8b}
@subsection Warnings


The following exceptions are used as warning categories; see the
@ref{1b8c,,Warning Categories} documentation for more details.

@geindex Warning
@anchor{library/exceptions Warning}@anchor{1b8d}
@deffn {Exception} Warning

Base class for warning categories.
@end deffn

@geindex UserWarning
@anchor{library/exceptions UserWarning}@anchor{1b8e}
@deffn {Exception} UserWarning

Base class for warnings generated by user code.
@end deffn

@geindex DeprecationWarning
@anchor{library/exceptions DeprecationWarning}@anchor{1b2}
@deffn {Exception} DeprecationWarning

Base class for warnings about deprecated features when those warnings are
intended for other Python developers.

Ignored by the default warning filters, except in the @code{__main__} module
(
@geindex Python Enhancement Proposals; PEP 565
PEP 565@footnote{https://www.python.org/dev/peps/pep-0565}). Enabling the @ref{3ac,,Python Development Mode} shows
this warning.

The deprecation policy is described in 
@geindex Python Enhancement Proposals; PEP 387
PEP 387@footnote{https://www.python.org/dev/peps/pep-0387}.
@end deffn

@geindex PendingDeprecationWarning
@anchor{library/exceptions PendingDeprecationWarning}@anchor{3a9}
@deffn {Exception} PendingDeprecationWarning

Base class for warnings about features which are obsolete and
expected to be deprecated in the future, but are not deprecated
at the moment.

This class is rarely used as emitting a warning about a possible
upcoming deprecation is unusual, and @ref{1b2,,DeprecationWarning}
is preferred for already active deprecations.

Ignored by the default warning filters. Enabling the @ref{3ac,,Python Development Mode} shows this warning.

The deprecation policy is described in 
@geindex Python Enhancement Proposals; PEP 387
PEP 387@footnote{https://www.python.org/dev/peps/pep-0387}.
@end deffn

@geindex SyntaxWarning
@anchor{library/exceptions SyntaxWarning}@anchor{4c7}
@deffn {Exception} SyntaxWarning

Base class for warnings about dubious syntax.
@end deffn

@geindex RuntimeWarning
@anchor{library/exceptions RuntimeWarning}@anchor{5b5}
@deffn {Exception} RuntimeWarning

Base class for warnings about dubious runtime behavior.
@end deffn

@geindex FutureWarning
@anchor{library/exceptions FutureWarning}@anchor{604}
@deffn {Exception} FutureWarning

Base class for warnings about deprecated features when those warnings are
intended for end users of applications that are written in Python.
@end deffn

@geindex ImportWarning
@anchor{library/exceptions ImportWarning}@anchor{304}
@deffn {Exception} ImportWarning

Base class for warnings about probable mistakes in module imports.

Ignored by the default warning filters. Enabling the @ref{3ac,,Python Development Mode} shows this warning.
@end deffn

@geindex UnicodeWarning
@anchor{library/exceptions UnicodeWarning}@anchor{f81}
@deffn {Exception} UnicodeWarning

Base class for warnings related to Unicode.
@end deffn

@geindex EncodingWarning
@anchor{library/exceptions EncodingWarning}@anchor{1617}
@deffn {Exception} EncodingWarning

Base class for warnings related to encodings.

See @ref{162b,,Opt-in EncodingWarning} for details.


New in version 3.10.

@end deffn

@geindex BytesWarning
@anchor{library/exceptions BytesWarning}@anchor{761}
@deffn {Exception} BytesWarning

Base class for warnings related to @ref{2f0,,bytes} and @ref{2f1,,bytearray}.
@end deffn

@geindex ResourceWarning
@anchor{library/exceptions ResourceWarning}@anchor{77b}
@deffn {Exception} ResourceWarning

Base class for warnings related to resource usage.

Ignored by the default warning filters. Enabling the @ref{3ac,,Python Development Mode} shows this warning.


New in version 3.2.

@end deffn

@node Exception groups,Exception hierarchy,Warnings,Built-in Exceptions
@anchor{library/exceptions exception-groups}@anchor{1b8f}
@subsection Exception groups


The following are used when it is necessary to raise multiple unrelated
exceptions. They are part of the exception hierarchy so they can be
handled with @ref{16e,,except} like all other exceptions. In addition,
they are recognised by @ref{1989,,except*}, which matches
their subgroups based on the types of the contained exceptions.

@geindex ExceptionGroup
@anchor{library/exceptions ExceptionGroup}@anchor{10db}
@deffn {Exception} ExceptionGroup (msg, excs)
@end deffn

@geindex BaseExceptionGroup
@anchor{library/exceptions BaseExceptionGroup}@anchor{10dc}
@deffn {Exception} BaseExceptionGroup (msg, excs)

Both of these exception types wrap the exceptions in the sequence @code{excs}.
The @code{msg} parameter must be a string. The difference between the two
classes is that @ref{10dc,,BaseExceptionGroup} extends @ref{15f,,BaseException} and
it can wrap any exception, while @ref{10db,,ExceptionGroup} extends @ref{4d8,,Exception}
and it can only wrap subclasses of @ref{4d8,,Exception}. This design is so that
@code{except Exception} catches an @ref{10db,,ExceptionGroup} but not
@ref{10dc,,BaseExceptionGroup}.

The @ref{10dc,,BaseExceptionGroup} constructor returns an @ref{10db,,ExceptionGroup}
rather than a @ref{10dc,,BaseExceptionGroup} if all contained exceptions are
@ref{4d8,,Exception} instances, so it can be used to make the selection
automatic. The @ref{10db,,ExceptionGroup} constructor, on the other hand,
raises a @ref{162,,TypeError} if any contained exception is not an
@ref{4d8,,Exception} subclass.

@geindex message (BaseExceptionGroup attribute)
@anchor{library/exceptions BaseExceptionGroup message}@anchor{1b90}
@deffn {Attribute} message

The @code{msg} argument to the constructor. This is a read-only attribute.
@end deffn

@geindex exceptions (BaseExceptionGroup attribute)
@anchor{library/exceptions BaseExceptionGroup exceptions}@anchor{1b91}
@deffn {Attribute} exceptions

A tuple of the exceptions in the @code{excs} sequence given to the
constructor. This is a read-only attribute.
@end deffn

@geindex subgroup() (BaseExceptionGroup method)
@anchor{library/exceptions BaseExceptionGroup subgroup}@anchor{1b92}
@deffn {Method} subgroup (condition)

Returns an exception group that contains only the exceptions from the
current group that match @emph{condition}, or @code{None} if the result is empty.

The condition can be either a function that accepts an exception and returns
true for those that should be in the subgroup, or it can be an exception type
or a tuple of exception types, which is used to check for a match using the
same check that is used in an @code{except} clause.

The nesting structure of the current exception is preserved in the result,
as are the values of its @ref{1b90,,message}, @code{__traceback__},
@code{__cause__}, @code{__context__} and @code{__note__} fields.
Empty nested groups are omitted from the result.

The condition is checked for all exceptions in the nested exception group,
including the top-level and any nested exception groups. If the condition is
true for such an exception group, it is included in the result in full.
@end deffn

@geindex split() (BaseExceptionGroup method)
@anchor{library/exceptions BaseExceptionGroup split}@anchor{1b93}
@deffn {Method} split (condition)

Like @ref{1b92,,subgroup()}, but returns the pair @code{(match, rest)} where @code{match}
is @code{subgroup(condition)} and @code{rest} is the remaining non-matching
part.
@end deffn

@geindex derive() (BaseExceptionGroup method)
@anchor{library/exceptions BaseExceptionGroup derive}@anchor{1b94}
@deffn {Method} derive (excs)

Returns an exception group with the same @ref{1b90,,message},
@code{__traceback__}, @code{__cause__}, @code{__context__}
and @code{__note__} but which wraps the exceptions in @code{excs}.

This method is used by @ref{1b92,,subgroup()} and @ref{1b93,,split()}. A
subclass needs to override it in order to make @ref{1b92,,subgroup()}
and @ref{1b93,,split()} return instances of the subclass rather
than @ref{10db,,ExceptionGroup}.

@example
>>> class MyGroup(ExceptionGroup):
...     def derive(self, exc):
...         return MyGroup(self.message, exc)
...
>>> MyGroup("eg", [ValueError(1), TypeError(2)]).split(TypeError)
(MyGroup('eg', [TypeError(2)]), MyGroup('eg', [ValueError(1)]))
@end example
@end deffn

Note that @ref{10dc,,BaseExceptionGroup} defines @code{__new__()}, so
subclasses that need a different constructor signature need to
override that rather than @code{__init__()}. For example, the following
defines an exception group subclass which accepts an exit_code and
and constructs the group’s message from it.

@example
class Errors(ExceptionGroup):
   def __new__(cls, errors, exit_code):
      self = super().__new__(Errors, f"exit code: @{exit_code@}", errors)
      self.exit_code = exit_code
      return self

   def derive(self, excs):
      return Errors(excs, self.exit_code)
@end example


New in version 3.11.

@end deffn

@node Exception hierarchy,,Exception groups,Built-in Exceptions
@anchor{library/exceptions exception-hierarchy}@anchor{1b95}
@subsection Exception hierarchy


The class hierarchy for built-in exceptions is:

@example
BaseException
 ├── BaseExceptionGroup
 ├── GeneratorExit
 ├── KeyboardInterrupt
 ├── SystemExit
 └── Exception
      ├── ArithmeticError
      │    ├── FloatingPointError
      │    ├── OverflowError
      │    └── ZeroDivisionError
      ├── AssertionError
      ├── AttributeError
      ├── BufferError
      ├── EOFError
      ├── ExceptionGroup [BaseExceptionGroup]
      ├── ImportError
      │    └── ModuleNotFoundError
      ├── LookupError
      │    ├── IndexError
      │    └── KeyError
      ├── MemoryError
      ├── NameError
      │    └── UnboundLocalError
      ├── OSError
      │    ├── BlockingIOError
      │    ├── ChildProcessError
      │    ├── ConnectionError
      │    │    ├── BrokenPipeError
      │    │    ├── ConnectionAbortedError
      │    │    ├── ConnectionRefusedError
      │    │    └── ConnectionResetError
      │    ├── FileExistsError
      │    ├── FileNotFoundError
      │    ├── InterruptedError
      │    ├── IsADirectoryError
      │    ├── NotADirectoryError
      │    ├── PermissionError
      │    ├── ProcessLookupError
      │    └── TimeoutError
      ├── ReferenceError
      ├── RuntimeError
      │    ├── NotImplementedError
      │    └── RecursionError
      ├── StopAsyncIteration
      ├── StopIteration
      ├── SyntaxError
      │    └── IndentationError
      │         └── TabError
      ├── SystemError
      ├── TypeError
      ├── ValueError
      │    └── UnicodeError
      │         ├── UnicodeDecodeError
      │         ├── UnicodeEncodeError
      │         └── UnicodeTranslateError
      └── Warning
           ├── BytesWarning
           ├── DeprecationWarning
           ├── EncodingWarning
           ├── FutureWarning
           ├── ImportWarning
           ├── PendingDeprecationWarning
           ├── ResourceWarning
           ├── RuntimeWarning
           ├── SyntaxWarning
           ├── UnicodeWarning
           └── UserWarning
@end example
@anchor{library/text stringservices}@anchor{1a97}
@node Text Processing Services,Binary Data Services,Built-in Exceptions,The Python Standard Library
@anchor{library/text doc}@anchor{1b96}@anchor{library/text text-processing-services}@anchor{1b97}@anchor{library/text textservices}@anchor{1a9a}
@section Text Processing Services


The modules described in this chapter provide a wide range of string
manipulation operations and other text processing services.

The @ref{1a,,codecs} module described under @ref{1b98,,Binary Data Services} is also
highly relevant to text processing. In addition, see the documentation for
Python’s built-in string type in @ref{14e2,,Text Sequence Type — str}.

@menu
* string — Common string operations:: 
* re — Regular expression operations:: 
* difflib — Helpers for computing deltas:: 
* textwrap — Text wrapping and filling:: 
* unicodedata — Unicode Database:: 
* stringprep — Internet String Preparation:: 
* readline — GNU readline interface:: 
* rlcompleter — Completion function for GNU readline:: 

@end menu

@node string — Common string operations,re — Regular expression operations,,Text Processing Services
@anchor{library/string doc}@anchor{1b99}@anchor{library/string module-string}@anchor{f1}@anchor{library/string string-common-string-operations}@anchor{1b9a}
@subsection @code{string} — Common string operations


@geindex module; string

@strong{Source code:} Lib/string.py@footnote{https://github.com/python/cpython/tree/main/Lib/string.py}


__________________________________________________________________



@subsubheading See also


@ref{14e2,,Text Sequence Type — str}

@ref{14e3,,String Methods}


@menu
* String constants:: 
* Custom String Formatting:: 
* Format String Syntax:: 
* Template strings:: 
* Helper functions:: 

@end menu

@node String constants,Custom String Formatting,,string — Common string operations
@anchor{library/string string-constants}@anchor{1b9b}
@subsubsection String constants


The constants defined in this module are:

@geindex ascii_letters (in module string)
@anchor{library/string string ascii_letters}@anchor{e68}
@deffn {Data} string.ascii_letters

The concatenation of the @ref{1b9c,,ascii_lowercase} and @ref{1b9d,,ascii_uppercase}
constants described below.  This value is not locale-dependent.
@end deffn

@geindex ascii_lowercase (in module string)
@anchor{library/string string ascii_lowercase}@anchor{1b9c}
@deffn {Data} string.ascii_lowercase

The lowercase letters @code{'abcdefghijklmnopqrstuvwxyz'}.  This value is not
locale-dependent and will not change.
@end deffn

@geindex ascii_uppercase (in module string)
@anchor{library/string string ascii_uppercase}@anchor{1b9d}
@deffn {Data} string.ascii_uppercase

The uppercase letters @code{'ABCDEFGHIJKLMNOPQRSTUVWXYZ'}.  This value is not
locale-dependent and will not change.
@end deffn

@geindex digits (in module string)
@anchor{library/string string digits}@anchor{1b9e}
@deffn {Data} string.digits

The string @code{'0123456789'}.
@end deffn

@geindex hexdigits (in module string)
@anchor{library/string string hexdigits}@anchor{1b9f}
@deffn {Data} string.hexdigits

The string @code{'0123456789abcdefABCDEF'}.
@end deffn

@geindex octdigits (in module string)
@anchor{library/string string octdigits}@anchor{1ba0}
@deffn {Data} string.octdigits

The string @code{'01234567'}.
@end deffn

@geindex punctuation (in module string)
@anchor{library/string string punctuation}@anchor{1ba1}
@deffn {Data} string.punctuation

String of ASCII characters which are considered punctuation characters
in the @code{C} locale: @code{!"#$%&'()*+,-./:;<=>?@@[\]^_`@{|@}~}.
@end deffn

@geindex printable (in module string)
@anchor{library/string string printable}@anchor{1ba2}
@deffn {Data} string.printable

String of ASCII characters which are considered printable.  This is a
combination of @ref{1b9e,,digits}, @ref{e68,,ascii_letters}, @ref{1ba1,,punctuation},
and @ref{1ba3,,whitespace}.
@end deffn

@geindex whitespace (in module string)
@anchor{library/string string whitespace}@anchor{1ba3}
@deffn {Data} string.whitespace

A string containing all ASCII characters that are considered whitespace.
This includes the characters space, tab, linefeed, return, formfeed, and
vertical tab.
@end deffn

@node Custom String Formatting,Format String Syntax,String constants,string — Common string operations
@anchor{library/string custom-string-formatting}@anchor{1ba4}@anchor{library/string string-formatting}@anchor{1a99}
@subsubsection Custom String Formatting


The built-in string class provides the ability to do complex variable
substitutions and value formatting via the @ref{784,,format()} method described in
@geindex Python Enhancement Proposals; PEP 3101
PEP 3101@footnote{https://www.python.org/dev/peps/pep-3101}.  The @ref{a2e,,Formatter} class in the @ref{f1,,string} module allows
you to create and customize your own string formatting behaviors using the same
implementation as the built-in @ref{784,,format()} method.

@geindex Formatter (class in string)
@anchor{library/string string Formatter}@anchor{a2e}
@deffn {Class} string.Formatter

The @ref{a2e,,Formatter} class has the following public methods:

@geindex format() (string.Formatter method)
@anchor{library/string string Formatter format}@anchor{73b}
@deffn {Method} format (format_string, /, *args, **kwargs)

The primary API method.  It takes a format string and
an arbitrary set of positional and keyword arguments.
It is just a wrapper that calls @ref{1ba5,,vformat()}.


Changed in version 3.7: A format string argument is now @ref{59d,,positional-only}.

@end deffn

@geindex vformat() (string.Formatter method)
@anchor{library/string string Formatter vformat}@anchor{1ba5}
@deffn {Method} vformat (format_string, args, kwargs)

This function does the actual work of formatting.  It is exposed as a
separate function for cases where you want to pass in a predefined
dictionary of arguments, rather than unpacking and repacking the
dictionary as individual arguments using the @code{*args} and @code{**kwargs}
syntax.  @ref{1ba5,,vformat()} does the work of breaking up the format string
into character data and replacement fields.  It calls the various
methods described below.
@end deffn

In addition, the @ref{a2e,,Formatter} defines a number of methods that are
intended to be replaced by subclasses:

@geindex parse() (string.Formatter method)
@anchor{library/string string Formatter parse}@anchor{1ba6}
@deffn {Method} parse (format_string)

Loop over the format_string and return an iterable of tuples
(@emph{literal_text}, @emph{field_name}, @emph{format_spec}, @emph{conversion}).  This is used
by @ref{1ba5,,vformat()} to break the string into either literal text, or
replacement fields.

The values in the tuple conceptually represent a span of literal text
followed by a single replacement field.  If there is no literal text
(which can happen if two replacement fields occur consecutively), then
@emph{literal_text} will be a zero-length string.  If there is no replacement
field, then the values of @emph{field_name}, @emph{format_spec} and @emph{conversion}
will be @code{None}.
@end deffn

@geindex get_field() (string.Formatter method)
@anchor{library/string string Formatter get_field}@anchor{1ba7}
@deffn {Method} get_field (field_name, args, kwargs)

Given @emph{field_name} as returned by @ref{1ba6,,parse()} (see above), convert it to
an object to be formatted.  Returns a tuple (obj, used_key).  The default
version takes strings of the form defined in 
@geindex Python Enhancement Proposals; PEP 3101
PEP 3101@footnote{https://www.python.org/dev/peps/pep-3101}, such as
“0[name]” or “label.title”.  @emph{args} and @emph{kwargs} are as passed in to
@ref{1ba5,,vformat()}.  The return value @emph{used_key} has the same meaning as the
@emph{key} parameter to @ref{1ba8,,get_value()}.
@end deffn

@geindex get_value() (string.Formatter method)
@anchor{library/string string Formatter get_value}@anchor{1ba8}
@deffn {Method} get_value (key, args, kwargs)

Retrieve a given field value.  The @emph{key} argument will be either an
integer or a string.  If it is an integer, it represents the index of the
positional argument in @emph{args}; if it is a string, then it represents a
named argument in @emph{kwargs}.

The @emph{args} parameter is set to the list of positional arguments to
@ref{1ba5,,vformat()}, and the @emph{kwargs} parameter is set to the dictionary of
keyword arguments.

For compound field names, these functions are only called for the first
component of the field name; subsequent components are handled through
normal attribute and indexing operations.

So for example, the field expression ‘0.name’ would cause
@ref{1ba8,,get_value()} to be called with a @emph{key} argument of 0.  The @code{name}
attribute will be looked up after @ref{1ba8,,get_value()} returns by calling the
built-in @ref{708,,getattr()} function.

If the index or keyword refers to an item that does not exist, then an
@ref{1066,,IndexError} or @ref{5bb,,KeyError} should be raised.
@end deffn

@geindex check_unused_args() (string.Formatter method)
@anchor{library/string string Formatter check_unused_args}@anchor{1ba9}
@deffn {Method} check_unused_args (used_args, args, kwargs)

Implement checking for unused arguments if desired.  The arguments to this
function is the set of all argument keys that were actually referred to in
the format string (integers for positional arguments, and strings for
named arguments), and a reference to the @emph{args} and @emph{kwargs} that was
passed to vformat.  The set of unused args can be calculated from these
parameters.  @ref{1ba9,,check_unused_args()} is assumed to raise an exception if
the check fails.
@end deffn

@geindex format_field() (string.Formatter method)
@anchor{library/string string Formatter format_field}@anchor{1baa}
@deffn {Method} format_field (value, format_spec)

@ref{1baa,,format_field()} simply calls the global @ref{785,,format()} built-in.  The
method is provided so that subclasses can override it.
@end deffn

@geindex convert_field() (string.Formatter method)
@anchor{library/string string Formatter convert_field}@anchor{1bab}
@deffn {Method} convert_field (value, conversion)

Converts the value (returned by @ref{1ba7,,get_field()}) given a conversion type
(as in the tuple returned by the @ref{1ba6,,parse()} method).  The default
version understands ‘s’ (str), ‘r’ (repr) and ‘a’ (ascii) conversion
types.
@end deffn
@end deffn

@node Format String Syntax,Template strings,Custom String Formatting,string — Common string operations
@anchor{library/string format-string-syntax}@anchor{1bac}@anchor{library/string formatstrings}@anchor{f1a}
@subsubsection Format String Syntax


The @ref{784,,str.format()} method and the @ref{a2e,,Formatter} class share the same
syntax for format strings (although in the case of @ref{a2e,,Formatter},
subclasses can define their own format string syntax).  The syntax is
related to that of @ref{4b0,,formatted string literals}, but it is
less sophisticated and, in particular, does not support arbitrary expressions.

@geindex @{@} (curly brackets); in string formatting
@geindex . (dot); in string formatting
@geindex [] (square brackets); in string formatting
@geindex ! (exclamation); in string formatting
@geindex ; (colon); in string formatting

Format strings contain “replacement fields” surrounded by curly braces @code{@{@}}.
Anything that is not contained in braces is considered literal text, which is
copied unchanged to the output.  If you need to include a brace character in the
literal text, it can be escaped by doubling: @code{@{@{} and @code{@}@}}.

The grammar for a replacement field is as follows:

@quotation

@example
@anchor{library/string grammar-token-format-string-replacement_field}@anchor{1bad}@anchor{library/string grammar-token-replacement-field}@anchor{1bae}replacement_field ::= "@{" [field_name] ["!" conversion] [":" format_spec] "@}"
@anchor{library/string grammar-token-format-string-field_name}@anchor{1baf}@anchor{library/string grammar-token-field-name}@anchor{1bb0}field_name        ::= arg_name ("." attribute_name | "[" element_index "]")*
@anchor{library/string grammar-token-format-string-arg_name}@anchor{1bb1}@anchor{library/string grammar-token-arg-name}@anchor{1bb2}arg_name          ::= [identifier | digit+]
@anchor{library/string grammar-token-format-string-attribute_name}@anchor{1bb3}@anchor{library/string grammar-token-attribute-name}@anchor{1bb4}attribute_name    ::= identifier
@anchor{library/string grammar-token-format-string-element_index}@anchor{1bb5}@anchor{library/string grammar-token-element-index}@anchor{1bb6}element_index     ::= digit+ | index_string
@anchor{library/string grammar-token-format-string-index_string}@anchor{1bb7}@anchor{library/string grammar-token-index-string}@anchor{1bb8}index_string      ::= <any source character except "]"> +
@anchor{library/string grammar-token-format-string-conversion}@anchor{1bb9}@anchor{library/string grammar-token-conversion}@anchor{1bba}conversion        ::= "r" | "s" | "a"
@anchor{library/string grammar-token-format-string-format_spec}@anchor{1bbb}@anchor{library/string grammar-token-format-spec}@anchor{1bbc}format_spec       ::= <described in the next section>
@end example
@end quotation

In less formal terms, the replacement field can start with a @emph{field_name} that specifies
the object whose value is to be formatted and inserted
into the output instead of the replacement field.
The @emph{field_name} is optionally followed by a  @emph{conversion} field, which is
preceded by an exclamation point @code{'!'}, and a @emph{format_spec}, which is preceded
by a colon @code{':'}.  These specify a non-default format for the replacement value.

See also the @ref{788,,Format Specification Mini-Language} section.

The @emph{field_name} itself begins with an @emph{arg_name} that is either a number or a
keyword.  If it’s a number, it refers to a positional argument, and if it’s a keyword,
it refers to a named keyword argument.  If the numerical arg_names in a format string
are 0, 1, 2, … in sequence, they can all be omitted (not just some)
and the numbers 0, 1, 2, … will be automatically inserted in that order.
Because @emph{arg_name} is not quote-delimited, it is not possible to specify arbitrary
dictionary keys (e.g., the strings @code{'10'} or @code{':-]'}) within a format string.
The @emph{arg_name} can be followed by any number of index or
attribute expressions. An expression of the form @code{'.name'} selects the named
attribute using @ref{708,,getattr()}, while an expression of the form @code{'[index]'}
does an index lookup using @code{__getitem__()}.


Changed in version 3.1: The positional argument specifiers can be omitted for @ref{784,,str.format()},
so @code{'@{@} @{@}'.format(a, b)} is equivalent to @code{'@{0@} @{1@}'.format(a, b)}.



Changed in version 3.4: The positional argument specifiers can be omitted for @ref{a2e,,Formatter}.


Some simple format string examples:

@example
"First, thou shalt count to @{0@}"  # References first positional argument
"Bring me a @{@}"                   # Implicitly references the first positional argument
"From @{@} to @{@}"                   # Same as "From @{0@} to @{1@}"
"My quest is @{name@}"              # References keyword argument 'name'
"Weight in tons @{0.weight@}"       # 'weight' attribute of first positional arg
"Units destroyed: @{players[0]@}"   # First element of keyword argument 'players'.
@end example

The @emph{conversion} field causes a type coercion before formatting.  Normally, the
job of formatting a value is done by the @code{__format__()} method of the value
itself.  However, in some cases it is desirable to force a type to be formatted
as a string, overriding its own definition of formatting.  By converting the
value to a string before calling @code{__format__()}, the normal formatting logic
is bypassed.

Three conversion flags are currently supported: @code{'!s'} which calls @ref{2ef,,str()}
on the value, @code{'!r'} which calls @ref{27d,,repr()} and @code{'!a'} which calls
@ref{f47,,ascii()}.

Some examples:

@example
"Harold's a clever @{0!s@}"        # Calls str() on the argument first
"Bring out the holy @{name!r@}"    # Calls repr() on the argument first
"More @{!a@}"                      # Calls ascii() on the argument first
@end example

The @emph{format_spec} field contains a specification of how the value should be
presented, including such details as field width, alignment, padding, decimal
precision and so on.  Each value type can define its own “formatting
mini-language” or interpretation of the @emph{format_spec}.

Most built-in types support a common formatting mini-language, which is
described in the next section.

A @emph{format_spec} field can also include nested replacement fields within it.
These nested replacement fields may contain a field name, conversion flag
and format specification, but deeper nesting is
not allowed.  The replacement fields within the
format_spec are substituted before the @emph{format_spec} string is interpreted.
This allows the formatting of a value to be dynamically specified.

See the @ref{1bbd,,Format examples} section for some examples.

@menu
* Format Specification Mini-Language:: 
* Format examples:: 

@end menu

@node Format Specification Mini-Language,Format examples,,Format String Syntax
@anchor{library/string format-specification-mini-language}@anchor{1bbe}@anchor{library/string formatspec}@anchor{788}
@subsubsection Format Specification Mini-Language


“Format specifications” are used within replacement fields contained within a
format string to define how individual values are presented (see
@ref{f1a,,Format String Syntax} and @ref{4b0,,Formatted string literals}).
They can also be passed directly to the built-in
@ref{785,,format()} function.  Each formattable type may define how the format
specification is to be interpreted.

Most built-in types implement the following options for format specifications,
although some of the formatting options are only supported by the numeric types.

A general convention is that an empty format specification produces
the same result as if you had called @ref{2ef,,str()} on the value. A
non-empty format specification typically modifies the result.

The general form of a @emph{standard format specifier} is:

@example
@anchor{library/string grammar-token-format-spec-format_spec}@anchor{1bbf}format_spec     ::= [[fill]align][sign][#][0][width][grouping_option][.precision][type]
@anchor{library/string grammar-token-format-spec-fill}@anchor{1bc0}@anchor{library/string grammar-token-fill}@anchor{1bc1}fill            ::= <any character>
@anchor{library/string grammar-token-format-spec-align}@anchor{1bc2}@anchor{library/string grammar-token-align}@anchor{1bc3}align           ::= "<" | ">" | "=" | "^"
@anchor{library/string grammar-token-format-spec-sign}@anchor{1bc4}@anchor{library/string grammar-token-sign}@anchor{1bc5}sign            ::= "+" | "-" | " "
@anchor{library/string grammar-token-format-spec-width}@anchor{1bc6}@anchor{library/string grammar-token-width}@anchor{1bc7}width           ::= digit+
@anchor{library/string grammar-token-format-spec-grouping_option}@anchor{1bc8}@anchor{library/string grammar-token-grouping-option}@anchor{1bc9}grouping_option ::= "_" | ","
@anchor{library/string grammar-token-format-spec-precision}@anchor{1bca}@anchor{library/string grammar-token-precision}@anchor{1bcb}precision       ::= digit+
@anchor{library/string grammar-token-format-spec-type}@anchor{1bcc}@anchor{library/string grammar-token-type}@anchor{1bcd}type            ::= "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"
@end example

If a valid @emph{align} value is specified, it can be preceded by a @emph{fill}
character that can be any character and defaults to a space if omitted.
It is not possible to use a literal curly brace (”@code{@{}” or “@code{@}}”) as
the @emph{fill} character in a @ref{4b0,,formatted string literal} or when using the @ref{784,,str.format()}
method.  However, it is possible to insert a curly brace
with a nested replacement field.  This limitation doesn’t
affect the @ref{785,,format()} function.

The meaning of the various alignment options is as follows:

@quotation

@geindex < (less); in string formatting
@geindex > (greater); in string formatting
@geindex = (equals); in string formatting
@geindex ^ (caret); in string formatting


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Meaning

@item

@code{'<'}

@tab

Forces the field to be left-aligned within the available
space (this is the default for most objects).

@item

@code{'>'}

@tab

Forces the field to be right-aligned within the
available space (this is the default for numbers).

@item

@code{'='}

@tab

Forces the padding to be placed after the sign (if any)
but before the digits.  This is used for printing fields
in the form ‘+000000120’. This alignment option is only
valid for numeric types.  It becomes the default for
numbers when ‘0’ immediately precedes the field width.

@item

@code{'^'}

@tab

Forces the field to be centered within the available
space.

@end multitable

@end quotation

Note that unless a minimum field width is defined, the field width will always
be the same size as the data to fill it, so that the alignment option has no
meaning in this case.

The @emph{sign} option is only valid for number types, and can be one of the
following:

@quotation

@geindex + (plus); in string formatting
@geindex - (minus); in string formatting
@geindex space; in string formatting


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Meaning

@item

@code{'+'}

@tab

indicates that a sign should be used for both
positive as well as negative numbers.

@item

@code{'-'}

@tab

indicates that a sign should be used only for negative
numbers (this is the default behavior).

@item

space

@tab

indicates that a leading space should be used on
positive numbers, and a minus sign on negative numbers.

@end multitable

@end quotation

@geindex # (hash); in string formatting

The @code{'#'} option causes the “alternate form” to be used for the
conversion.  The alternate form is defined differently for different
types.  This option is only valid for integer, float and complex
types. For integers, when binary, octal, or hexadecimal output
is used, this option adds the respective prefix @code{'0b'}, @code{'0o'},
@code{'0x'}, or @code{'0X'} to the output value. For float and complex the
alternate form causes the result of the conversion to always contain a
decimal-point character, even if no digits follow it. Normally, a
decimal-point character appears in the result of these conversions
only if a digit follows it. In addition, for @code{'g'} and @code{'G'}
conversions, trailing zeros are not removed from the result.

@geindex @comma{} (comma); in string formatting

The @code{','} option signals the use of a comma for a thousands separator.
For a locale aware separator, use the @code{'n'} integer presentation type
instead.


Changed in version 3.1: Added the @code{','} option (see also 
@geindex Python Enhancement Proposals; PEP 378
PEP 378@footnote{https://www.python.org/dev/peps/pep-0378}).


@geindex _ (underscore); in string formatting

The @code{'_'} option signals the use of an underscore for a thousands
separator for floating point presentation types and for integer
presentation type @code{'d'}.  For integer presentation types @code{'b'},
@code{'o'}, @code{'x'}, and @code{'X'}, underscores will be inserted every 4
digits.  For other presentation types, specifying this option is an
error.


Changed in version 3.6: Added the @code{'_'} option (see also 
@geindex Python Enhancement Proposals; PEP 515
PEP 515@footnote{https://www.python.org/dev/peps/pep-0515}).


@emph{width} is a decimal integer defining the minimum total field width,
including any prefixes, separators, and other formatting characters.
If not specified, then the field width will be determined by the content.

When no explicit alignment is given, preceding the @emph{width} field by a zero
(@code{'0'}) character enables
sign-aware zero-padding for numeric types.  This is equivalent to a @emph{fill}
character of @code{'0'} with an @emph{alignment} type of @code{'='}.


Changed in version 3.10: Preceding the @emph{width} field by @code{'0'} no longer affects the default
alignment for strings.


The @emph{precision} is a decimal integer indicating how many digits should be
displayed after the decimal point for presentation types
@code{'f'} and @code{'F'}, or before and after the decimal point for presentation
types @code{'g'} or @code{'G'}.  For string presentation types the field
indicates the maximum field size - in other words, how many characters will be
used from the field content.  The @emph{precision} is not allowed for integer
presentation types.

Finally, the @emph{type} determines how the data should be presented.

The available string presentation types are:

@quotation


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Type

@tab

Meaning

@item

@code{'s'}

@tab

String format. This is the default type for strings and
may be omitted.

@item

None

@tab

The same as @code{'s'}.

@end multitable

@end quotation

The available integer presentation types are:

@quotation


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Type

@tab

Meaning

@item

@code{'b'}

@tab

Binary format. Outputs the number in base 2.

@item

@code{'c'}

@tab

Character. Converts the integer to the corresponding
unicode character before printing.

@item

@code{'d'}

@tab

Decimal Integer. Outputs the number in base 10.

@item

@code{'o'}

@tab

Octal format. Outputs the number in base 8.

@item

@code{'x'}

@tab

Hex format. Outputs the number in base 16, using
lower-case letters for the digits above 9.

@item

@code{'X'}

@tab

Hex format. Outputs the number in base 16, using
upper-case letters for the digits above 9.
In case @code{'#'} is specified, the prefix @code{'0x'} will
be upper-cased to @code{'0X'} as well.

@item

@code{'n'}

@tab

Number. This is the same as @code{'d'}, except that it uses
the current locale setting to insert the appropriate
number separator characters.

@item

None

@tab

The same as @code{'d'}.

@end multitable

@end quotation

In addition to the above presentation types, integers can be formatted
with the floating point presentation types listed below (except
@code{'n'} and @code{None}). When doing so, @ref{256,,float()} is used to convert the
integer to a floating point number before formatting.

The available presentation types for @ref{256,,float} and
@ref{248,,Decimal} values are:

@quotation


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Type

@tab

Meaning

@item

@code{'e'}

@tab

Scientific notation. For a given precision @code{p},
formats the number in scientific notation with the
letter ‘e’ separating the coefficient from the exponent.
The coefficient has one digit before and @code{p} digits
after the decimal point, for a total of @code{p + 1}
significant digits. With no precision given, uses a
precision of @code{6} digits after the decimal point for
@ref{256,,float}, and shows all coefficient digits
for @ref{248,,Decimal}. If no digits follow the
decimal point, the decimal point is also removed unless
the @code{#} option is used.

@item

@code{'E'}

@tab

Scientific notation. Same as @code{'e'} except it uses
an upper case ‘E’ as the separator character.

@item

@code{'f'}

@tab

Fixed-point notation. For a given precision @code{p},
formats the number as a decimal number with exactly
@code{p} digits following the decimal point. With no
precision given, uses a precision of @code{6} digits after
the decimal point for @ref{256,,float}, and uses a
precision large enough to show all coefficient digits
for @ref{248,,Decimal}. If no digits follow the
decimal point, the decimal point is also removed unless
the @code{#} option is used.

@item

@code{'F'}

@tab

Fixed-point notation. Same as @code{'f'}, but converts
@code{nan} to  @code{NAN} and @code{inf} to @code{INF}.

@item

@code{'g'}

@tab

General format.  For a given precision @code{p >= 1},
this rounds the number to @code{p} significant digits and
then formats the result in either fixed-point format
or in scientific notation, depending on its magnitude.
A precision of @code{0} is treated as equivalent to a
precision of @code{1}.

The precise rules are as follows: suppose that the
result formatted with presentation type @code{'e'} and
precision @code{p-1} would have exponent @code{exp}.  Then,
if @code{m <= exp < p}, where @code{m} is -4 for floats and -6
for @ref{248,,Decimals}, the number is
formatted with presentation type @code{'f'} and precision
@code{p-1-exp}.  Otherwise, the number is formatted
with presentation type @code{'e'} and precision @code{p-1}.
In both cases insignificant trailing zeros are removed
from the significand, and the decimal point is also
removed if there are no remaining digits following it,
unless the @code{'#'} option is used.

With no precision given, uses a precision of @code{6}
significant digits for @ref{256,,float}. For
@ref{248,,Decimal}, the coefficient of the result
is formed from the coefficient digits of the value;
scientific notation is used for values smaller than
@code{1e-6} in absolute value and values where the place
value of the least significant digit is larger than 1,
and fixed-point notation is used otherwise.

Positive and negative infinity, positive and negative
zero, and nans, are formatted as @code{inf}, @code{-inf},
@code{0}, @code{-0} and @code{nan} respectively, regardless of
the precision.

@item

@code{'G'}

@tab

General format. Same as @code{'g'} except switches to
@code{'E'} if the number gets too large. The
representations of infinity and NaN are uppercased, too.

@item

@code{'n'}

@tab

Number. This is the same as @code{'g'}, except that it uses
the current locale setting to insert the appropriate
number separator characters.

@item

@code{'%'}

@tab

Percentage. Multiplies the number by 100 and displays
in fixed (@code{'f'}) format, followed by a percent sign.

@item

None

@tab

For @ref{256,,float} this is the same as @code{'g'}, except
that when fixed-point notation is used to format the
result, it always includes at least one digit past the
decimal point. The precision used is as large as needed
to represent the given value faithfully.

For @ref{248,,Decimal}, this is the same as
either @code{'g'} or @code{'G'} depending on the value of
@code{context.capitals} for the current decimal context.

The overall effect is to match the output of @ref{2ef,,str()}
as altered by the other format modifiers.

@end multitable

@end quotation

@node Format examples,,Format Specification Mini-Language,Format String Syntax
@anchor{library/string format-examples}@anchor{1bce}@anchor{library/string formatexamples}@anchor{1bbd}
@subsubsection Format examples


This section contains examples of the @ref{784,,str.format()} syntax and
comparison with the old @code{%}-formatting.

In most of the cases the syntax is similar to the old @code{%}-formatting, with the
addition of the @code{@{@}} and with @code{:} used instead of @code{%}.
For example, @code{'%03.2f'} can be translated to @code{'@{:03.2f@}'}.

The new format syntax also supports new and different options, shown in the
following examples.

Accessing arguments by position:

@example
>>> '@{0@}, @{1@}, @{2@}'.format('a', 'b', 'c')
'a, b, c'
>>> '@{@}, @{@}, @{@}'.format('a', 'b', 'c')  # 3.1+ only
'a, b, c'
>>> '@{2@}, @{1@}, @{0@}'.format('a', 'b', 'c')
'c, b, a'
>>> '@{2@}, @{1@}, @{0@}'.format(*'abc')      # unpacking argument sequence
'c, b, a'
>>> '@{0@}@{1@}@{0@}'.format('abra', 'cad')   # arguments' indices can be repeated
'abracadabra'
@end example

Accessing arguments by name:

@example
>>> 'Coordinates: @{latitude@}, @{longitude@}'.format(latitude='37.24N', longitude='-115.81W')
'Coordinates: 37.24N, -115.81W'
>>> coord = @{'latitude': '37.24N', 'longitude': '-115.81W'@}
>>> 'Coordinates: @{latitude@}, @{longitude@}'.format(**coord)
'Coordinates: 37.24N, -115.81W'
@end example

Accessing arguments’ attributes:

@example
>>> c = 3-5j
>>> ('The complex number @{0@} is formed from the real part @{0.real@} '
...  'and the imaginary part @{0.imag@}.').format(c)
'The complex number (3-5j) is formed from the real part 3.0 and the imaginary part -5.0.'
>>> class Point:
...     def __init__(self, x, y):
...         self.x, self.y = x, y
...     def __str__(self):
...         return 'Point(@{self.x@}, @{self.y@})'.format(self=self)
...
>>> str(Point(4, 2))
'Point(4, 2)'
@end example

Accessing arguments’ items:

@example
>>> coord = (3, 5)
>>> 'X: @{0[0]@};  Y: @{0[1]@}'.format(coord)
'X: 3;  Y: 5'
@end example

Replacing @code{%s} and @code{%r}:

@example
>>> "repr() shows quotes: @{!r@}; str() doesn't: @{!s@}".format('test1', 'test2')
"repr() shows quotes: 'test1'; str() doesn't: test2"
@end example

Aligning the text and specifying a width:

@example
>>> '@{:<30@}'.format('left aligned')
'left aligned                  '
>>> '@{:>30@}'.format('right aligned')
'                 right aligned'
>>> '@{:^30@}'.format('centered')
'           centered           '
>>> '@{:*^30@}'.format('centered')  # use '*' as a fill char
'***********centered***********'
@end example

Replacing @code{%+f}, @code{%-f}, and @code{% f} and specifying a sign:

@example
>>> '@{:+f@}; @{:+f@}'.format(3.14, -3.14)  # show it always
'+3.140000; -3.140000'
>>> '@{: f@}; @{: f@}'.format(3.14, -3.14)  # show a space for positive numbers
' 3.140000; -3.140000'
>>> '@{:-f@}; @{:-f@}'.format(3.14, -3.14)  # show only the minus -- same as '@{:f@}; @{:f@}'
'3.140000; -3.140000'
@end example

Replacing @code{%x} and @code{%o} and converting the value to different bases:

@example
>>> # format also supports binary numbers
>>> "int: @{0:d@};  hex: @{0:x@};  oct: @{0:o@};  bin: @{0:b@}".format(42)
'int: 42;  hex: 2a;  oct: 52;  bin: 101010'
>>> # with 0x, 0o, or 0b as prefix:
>>> "int: @{0:d@};  hex: @{0:#x@};  oct: @{0:#o@};  bin: @{0:#b@}".format(42)
'int: 42;  hex: 0x2a;  oct: 0o52;  bin: 0b101010'
@end example

Using the comma as a thousands separator:

@example
>>> '@{:,@}'.format(1234567890)
'1,234,567,890'
@end example

Expressing a percentage:

@example
>>> points = 19
>>> total = 22
>>> 'Correct answers: @{:.2%@}'.format(points/total)
'Correct answers: 86.36%'
@end example

Using type-specific formatting:

@example
>>> import datetime
>>> d = datetime.datetime(2010, 7, 4, 12, 15, 58)
>>> '@{:%Y-%m-%d %H:%M:%S@}'.format(d)
'2010-07-04 12:15:58'
@end example

Nesting arguments and more complex examples:

@example
>>> for align, text in zip('<^>', ['left', 'center', 'right']):
...     '@{0:@{fill@}@{align@}16@}'.format(text, fill=align, align=align)
...
'left<<<<<<<<<<<<'
'^^^^^center^^^^^'
'>>>>>>>>>>>right'
>>>
>>> octets = [192, 168, 0, 1]
>>> '@{:02X@}@{:02X@}@{:02X@}@{:02X@}'.format(*octets)
'C0A80001'
>>> int(_, 16)
3232235521
>>>
>>> width = 5
>>> for num in range(5,12): 
...     for base in 'dXob':
...         print('@{0:@{width@}@{base@}@}'.format(num, base=base, width=width), end=' ')
...     print()
...
    5     5     5   101
    6     6     6   110
    7     7     7   111
    8     8    10  1000
    9     9    11  1001
   10     A    12  1010
   11     B    13  1011
@end example

@node Template strings,Helper functions,Format String Syntax,string — Common string operations
@anchor{library/string id1}@anchor{1bcf}@anchor{library/string template-strings}@anchor{1ab8}
@subsubsection Template strings


Template strings provide simpler string substitutions as described in
@geindex Python Enhancement Proposals; PEP 292
PEP 292@footnote{https://www.python.org/dev/peps/pep-0292}.  A primary use case for template strings is for
internationalization (i18n) since in that context, the simpler syntax and
functionality makes it easier to translate than other built-in string
formatting facilities in Python.  As an example of a library built on template
strings for i18n, see the
flufl.i18n@footnote{http://flufli18n.readthedocs.io/en/latest/} package.

@geindex $ (dollar); in template strings

Template strings support @code{$}-based substitutions, using the following rules:


@itemize *

@item 
@code{$$} is an escape; it is replaced with a single @code{$}.

@item 
@code{$identifier} names a substitution placeholder matching a mapping key of
@code{"identifier"}.  By default, @code{"identifier"} is restricted to any
case-insensitive ASCII alphanumeric string (including underscores) that
starts with an underscore or ASCII letter.  The first non-identifier
character after the @code{$} character terminates this placeholder
specification.

@item 
@code{$@{identifier@}} is equivalent to @code{$identifier}.  It is required when
valid identifier characters follow the placeholder but are not part of the
placeholder, such as @code{"$@{noun@}ification"}.
@end itemize

Any other appearance of @code{$} in the string will result in a @ref{17c,,ValueError}
being raised.

The @ref{f1,,string} module provides a @ref{6b5,,Template} class that implements
these rules.  The methods of @ref{6b5,,Template} are:

@geindex Template (class in string)
@anchor{library/string string Template}@anchor{6b5}
@deffn {Class} string.Template (template)

The constructor takes a single argument which is the template string.

@geindex substitute() (string.Template method)
@anchor{library/string string Template substitute}@anchor{15c2}
@deffn {Method} substitute (mapping=@{@}, /, **kwds)

Performs the template substitution, returning a new string.  @emph{mapping} is
any dictionary-like object with keys that match the placeholders in the
template.  Alternatively, you can provide keyword arguments, where the
keywords are the placeholders.  When both @emph{mapping} and @emph{kwds} are given
and there are duplicates, the placeholders from @emph{kwds} take precedence.
@end deffn

@geindex safe_substitute() (string.Template method)
@anchor{library/string string Template safe_substitute}@anchor{15c3}
@deffn {Method} safe_substitute (mapping=@{@}, /, **kwds)

Like @ref{15c2,,substitute()}, except that if placeholders are missing from
@emph{mapping} and @emph{kwds}, instead of raising a @ref{5bb,,KeyError} exception, the
original placeholder will appear in the resulting string intact.  Also,
unlike with @ref{15c2,,substitute()}, any other appearances of the @code{$} will
simply return @code{$} instead of raising @ref{17c,,ValueError}.

While other exceptions may still occur, this method is called “safe”
because it always tries to return a usable string instead of
raising an exception.  In another sense, @ref{15c3,,safe_substitute()} may be
anything other than safe, since it will silently ignore malformed
templates containing dangling delimiters, unmatched braces, or
placeholders that are not valid Python identifiers.
@end deffn

@geindex is_valid() (string.Template method)
@anchor{library/string string Template is_valid}@anchor{10b8}
@deffn {Method} is_valid ()

Returns false if the template has invalid placeholders that will cause
@ref{15c2,,substitute()} to raise @ref{17c,,ValueError}.


New in version 3.11.

@end deffn

@geindex get_identifiers() (string.Template method)
@anchor{library/string string Template get_identifiers}@anchor{10b9}
@deffn {Method} get_identifiers ()

Returns a list of the valid identifiers in the template, in the order
they first appear, ignoring any invalid identifiers.


New in version 3.11.

@end deffn

@ref{6b5,,Template} instances also provide one public data attribute:

@geindex template (string.Template attribute)
@anchor{library/string string Template template}@anchor{1bd0}
@deffn {Attribute} template

This is the object passed to the constructor’s @emph{template} argument.  In
general, you shouldn’t change it, but read-only access is not enforced.
@end deffn
@end deffn

Here is an example of how to use a Template:

@example
>>> from string import Template
>>> s = Template('$who likes $what')
>>> s.substitute(who='tim', what='kung pao')
'tim likes kung pao'
>>> d = dict(who='tim')
>>> Template('Give $who $100').substitute(d)
Traceback (most recent call last):
...
ValueError: Invalid placeholder in string: line 1, col 11
>>> Template('$who likes $what').substitute(d)
Traceback (most recent call last):
...
KeyError: 'what'
>>> Template('$who likes $what').safe_substitute(d)
'tim likes $what'
@end example

Advanced usage: you can derive subclasses of @ref{6b5,,Template} to customize
the placeholder syntax, delimiter character, or the entire regular expression
used to parse template strings.  To do this, you can override these class
attributes:


@itemize *

@item 
@emph{delimiter} – This is the literal string describing a placeholder
introducing delimiter.  The default value is @code{$}.  Note that this should
@emph{not} be a regular expression, as the implementation will call
@ref{744,,re.escape()} on this string as needed.  Note further that you cannot
change the delimiter after class creation (i.e. a different delimiter must
be set in the subclass’s class namespace).

@item 
@emph{idpattern} – This is the regular expression describing the pattern for
non-braced placeholders.  The default value is the regular expression
@code{(?a:[_a-z][_a-z0-9]*)}.  If this is given and @emph{braceidpattern} is
@code{None} this pattern will also apply to braced placeholders.

@cartouche
@quotation Note 
Since default @emph{flags} is @code{re.IGNORECASE}, pattern @code{[a-z]} can match
with some non-ASCII characters. That’s why we use the local @code{a} flag
here.
@end quotation
@end cartouche


Changed in version 3.7: @emph{braceidpattern} can be used to define separate patterns used inside and
outside the braces.


@item 
@emph{braceidpattern} – This is like @emph{idpattern} but describes the pattern for
braced placeholders.  Defaults to @code{None} which means to fall back to
@emph{idpattern} (i.e. the same pattern is used both inside and outside braces).
If given, this allows you to define different patterns for braced and
unbraced placeholders.


New in version 3.7.


@item 
@emph{flags} – The regular expression flags that will be applied when compiling
the regular expression used for recognizing substitutions.  The default value
is @code{re.IGNORECASE}.  Note that @code{re.VERBOSE} will always be added to the
flags, so custom @emph{idpattern}s must follow conventions for verbose regular
expressions.


New in version 3.2.

@end itemize

Alternatively, you can provide the entire regular expression pattern by
overriding the class attribute @emph{pattern}.  If you do this, the value must be a
regular expression object with four named capturing groups.  The capturing
groups correspond to the rules given above, along with the invalid placeholder
rule:


@itemize *

@item 
@emph{escaped} – This group matches the escape sequence, e.g. @code{$$}, in the
default pattern.

@item 
@emph{named} – This group matches the unbraced placeholder name; it should not
include the delimiter in capturing group.

@item 
@emph{braced} – This group matches the brace enclosed placeholder name; it should
not include either the delimiter or braces in the capturing group.

@item 
@emph{invalid} – This group matches any other delimiter pattern (usually a single
delimiter), and it should appear last in the regular expression.
@end itemize

The methods on this class will raise @ref{17c,,ValueError} if the pattern matches
the template without one of these named groups matching.

@node Helper functions,,Template strings,string — Common string operations
@anchor{library/string helper-functions}@anchor{1bd1}
@subsubsection Helper functions


@geindex capwords() (in module string)
@anchor{library/string string capwords}@anchor{1bd2}
@deffn {Function} string.capwords (s, sep=None)

Split the argument into words using @ref{a1b,,str.split()}, capitalize each word
using @ref{1a9b,,str.capitalize()}, and join the capitalized words using
@ref{1a8b,,str.join()}.  If the optional second argument @emph{sep} is absent
or @code{None}, runs of whitespace characters are replaced by a single space
and leading and trailing whitespace are removed, otherwise @emph{sep} is used to
split and join the words.
@end deffn

@node re — Regular expression operations,difflib — Helpers for computing deltas,string — Common string operations,Text Processing Services
@anchor{library/re doc}@anchor{1bd3}@anchor{library/re module-re}@anchor{d8}@anchor{library/re re-regular-expression-operations}@anchor{1bd4}
@subsection @code{re} — Regular expression operations


@geindex module; re

@strong{Source code:} Lib/re.py@footnote{https://github.com/python/cpython/tree/main/Lib/re.py}


__________________________________________________________________


This module provides regular expression matching operations similar to
those found in Perl.

Both patterns and strings to be searched can be Unicode strings (@ref{2ef,,str})
as well as 8-bit strings (@ref{2f0,,bytes}).
However, Unicode strings and 8-bit strings cannot be mixed:
that is, you cannot match a Unicode string with a byte pattern or
vice-versa; similarly, when asking for a substitution, the replacement
string must be of the same type as both the pattern and the search string.

Regular expressions use the backslash character (@code{'\'}) to indicate
special forms or to allow special characters to be used without invoking
their special meaning.  This collides with Python’s usage of the same
character for the same purpose in string literals; for example, to match
a literal backslash, one might have to write @code{'\\\\'} as the pattern
string, because the regular expression must be @code{\\}, and each
backslash must be expressed as @code{\\} inside a regular Python string
literal. Also, please note that any invalid escape sequences in Python’s
usage of the backslash in string literals now generate a @ref{1b2,,DeprecationWarning}
and in the future this will become a @ref{21a,,SyntaxError}. This behaviour
will happen even if it is a valid escape sequence for a regular expression.

The solution is to use Python’s raw string notation for regular expression
patterns; backslashes are not handled in any special way in a string literal
prefixed with @code{'r'}.  So @code{r"\n"} is a two-character string containing
@code{'\'} and @code{'n'}, while @code{"\n"} is a one-character string containing a
newline.  Usually patterns will be expressed in Python code using this raw
string notation.

It is important to note that most regular expression operations are available as
module-level functions and methods on
@ref{b07,,compiled regular expressions}.  The functions are shortcuts
that don’t require you to compile a regex object first, but miss some
fine-tuning parameters.


@subsubheading See also


The third-party regex@footnote{https://pypi.org/project/regex/} module,
which has an API compatible with the standard library @ref{d8,,re} module,
but offers additional functionality and a more thorough Unicode support.


@menu
* Regular Expression Syntax:: 
* Module Contents:: 
* Regular Expression Objects:: 
* Match Objects:: 
* Regular Expression Examples:: 

@end menu

@node Regular Expression Syntax,Module Contents,,re — Regular expression operations
@anchor{library/re re-syntax}@anchor{1bd5}@anchor{library/re regular-expression-syntax}@anchor{1bd6}
@subsubsection Regular Expression Syntax


A regular expression (or RE) specifies a set of strings that matches it; the
functions in this module let you check if a particular string matches a given
regular expression (or if a given regular expression matches a particular
string, which comes down to the same thing).

Regular expressions can be concatenated to form new regular expressions; if @emph{A}
and @emph{B} are both regular expressions, then @emph{AB} is also a regular expression.
In general, if a string @emph{p} matches @emph{A} and another string @emph{q} matches @emph{B}, the
string @emph{pq} will match AB.  This holds unless @emph{A} or @emph{B} contain low precedence
operations; boundary conditions between @emph{A} and @emph{B}; or have numbered group
references.  Thus, complex expressions can easily be constructed from simpler
primitive expressions like the ones described here.  For details of the theory
and implementation of regular expressions, consult the Friedl book @ref{1bd7,,[Frie09]},
or almost any textbook about compiler construction.

A brief explanation of the format of regular expressions follows.  For further
information and a gentler presentation, consult the @ref{1bd8,,Regular Expression HOWTO}.

Regular expressions can contain both special and ordinary characters. Most
ordinary characters, like @code{'A'}, @code{'a'}, or @code{'0'}, are the simplest regular
expressions; they simply match themselves.  You can concatenate ordinary
characters, so @code{last} matches the string @code{'last'}.  (In the rest of this
section, we’ll write RE’s in @code{this special style}, usually without quotes, and
strings to be matched @code{'in single quotes'}.)

Some characters, like @code{'|'} or @code{'('}, are special. Special
characters either stand for classes of ordinary characters, or affect
how the regular expressions around them are interpreted.

Repetition qualifiers (@code{*}, @code{+}, @code{?}, @code{@{m,n@}}, etc) cannot be
directly nested. This avoids ambiguity with the non-greedy modifier suffix
@code{?}, and with other modifiers in other implementations. To apply a second
repetition to an inner repetition, parentheses may be used. For example,
the expression @code{(?:a@{6@})*} matches any multiple of six @code{'a'} characters.

The special characters are:

@geindex . (dot); in regular expressions


@table @asis

@item @code{.}

(Dot.)  In the default mode, this matches any character except a newline.  If
the @ref{1bd9,,DOTALL} flag has been specified, this matches any character
including a newline.
@end table

@geindex ^ (caret); in regular expressions


@table @asis

@item @code{^}

(Caret.)  Matches the start of the string, and in @ref{1bda,,MULTILINE} mode also
matches immediately after each newline.
@end table

@geindex $ (dollar); in regular expressions


@table @asis

@item @code{$}

Matches the end of the string or just before the newline at the end of the
string, and in @ref{1bda,,MULTILINE} mode also matches before a newline.  @code{foo}
matches both ‘foo’ and ‘foobar’, while the regular expression @code{foo$} matches
only ‘foo’.  More interestingly, searching for @code{foo.$} in @code{'foo1\nfoo2\n'}
matches ‘foo2’ normally, but ‘foo1’ in @ref{1bda,,MULTILINE} mode; searching for
a single @code{$} in @code{'foo\n'} will find two (empty) matches: one just before
the newline, and one at the end of the string.
@end table

@geindex * (asterisk); in regular expressions


@table @asis

@item @code{*}

Causes the resulting RE to match 0 or more repetitions of the preceding RE, as
many repetitions as are possible.  @code{ab*} will match ‘a’, ‘ab’, or ‘a’ followed
by any number of ‘b’s.
@end table

@geindex + (plus); in regular expressions


@table @asis

@item @code{+}

Causes the resulting RE to match 1 or more repetitions of the preceding RE.
@code{ab+} will match ‘a’ followed by any non-zero number of ‘b’s; it will not
match just ‘a’.
@end table

@geindex ? (question mark); in regular expressions


@table @asis

@item @code{?}

Causes the resulting RE to match 0 or 1 repetitions of the preceding RE.
@code{ab?} will match either ‘a’ or ‘ab’.
@end table

@geindex *?; in regular expressions
@geindex +?; in regular expressions
@geindex ??; in regular expressions


@table @asis

@item @code{*?}, @code{+?}, @code{??}

The @code{'*'}, @code{'+'}, and @code{'?'} qualifiers are all @emph{greedy}; they match
as much text as possible.  Sometimes this behaviour isn’t desired; if the RE
@code{<.*>} is matched against @code{'<a> b <c>'}, it will match the entire
string, and not just @code{'<a>'}.  Adding @code{?} after the qualifier makes it
perform the match in @emph{non-greedy} or @emph{minimal} fashion; as @emph{few}
characters as possible will be matched.  Using the RE @code{<.*?>} will match
only @code{'<a>'}.
@end table

@geindex @{@} (curly brackets); in regular expressions


@table @asis

@item @code{@{m@}}

Specifies that exactly @emph{m} copies of the previous RE should be matched; fewer
matches cause the entire RE not to match.  For example, @code{a@{6@}} will match
exactly six @code{'a'} characters, but not five.

@item @code{@{m,n@}}

Causes the resulting RE to match from @emph{m} to @emph{n} repetitions of the preceding
RE, attempting to match as many repetitions as possible.  For example,
@code{a@{3,5@}} will match from 3 to 5 @code{'a'} characters.  Omitting @emph{m} specifies a
lower bound of zero,  and omitting @emph{n} specifies an infinite upper bound.  As an
example, @code{a@{4,@}b} will match @code{'aaaab'} or a thousand @code{'a'} characters
followed by a @code{'b'}, but not @code{'aaab'}. The comma may not be omitted or the
modifier would be confused with the previously described form.

@item @code{@{m,n@}?}

Causes the resulting RE to match from @emph{m} to @emph{n} repetitions of the preceding
RE, attempting to match as @emph{few} repetitions as possible.  This is the
non-greedy version of the previous qualifier.  For example, on the
6-character string @code{'aaaaaa'}, @code{a@{3,5@}} will match 5 @code{'a'} characters,
while @code{a@{3,5@}?} will only match 3 characters.
@end table

@geindex \ (backslash); in regular expressions


@table @asis

@item @code{\}

Either escapes special characters (permitting you to match characters like
@code{'*'}, @code{'?'}, and so forth), or signals a special sequence; special
sequences are discussed below.

If you’re not using a raw string to express the pattern, remember that Python
also uses the backslash as an escape sequence in string literals; if the escape
sequence isn’t recognized by Python’s parser, the backslash and subsequent
character are included in the resulting string.  However, if Python would
recognize the resulting sequence, the backslash should be repeated twice.  This
is complicated and hard to understand, so it’s highly recommended that you use
raw strings for all but the simplest expressions.
@end table

@geindex [] (square brackets); in regular expressions


@table @asis

@item @code{[]}

Used to indicate a set of characters.  In a set:


@itemize *

@item 
Characters can be listed individually, e.g. @code{[amk]} will match @code{'a'},
@code{'m'}, or @code{'k'}.
@end itemize

@geindex - (minus); in regular expressions


@itemize *

@item 
Ranges of characters can be indicated by giving two characters and separating
them by a @code{'-'}, for example @code{[a-z]} will match any lowercase ASCII letter,
@code{[0-5][0-9]} will match all the two-digits numbers from @code{00} to @code{59}, and
@code{[0-9A-Fa-f]} will match any hexadecimal digit.  If @code{-} is escaped (e.g.
@code{[a\-z]}) or if it’s placed as the first or last character
(e.g. @code{[-a]} or @code{[a-]}), it will match a literal @code{'-'}.

@item 
Special characters lose their special meaning inside sets.  For example,
@code{[(+*)]} will match any of the literal characters @code{'('}, @code{'+'},
@code{'*'}, or @code{')'}.
@end itemize

@geindex \ (backslash); in regular expressions


@itemize *

@item 
Character classes such as @code{\w} or @code{\S} (defined below) are also accepted
inside a set, although the characters they match depends on whether
@ref{697,,ASCII} or @ref{698,,LOCALE} mode is in force.
@end itemize

@geindex ^ (caret); in regular expressions


@itemize *

@item 
Characters that are not within a range can be matched by @emph{complementing}
the set.  If the first character of the set is @code{'^'}, all the characters
that are @emph{not} in the set will be matched.  For example, @code{[^5]} will match
any character except @code{'5'}, and @code{[^^]} will match any character except
@code{'^'}.  @code{^} has no special meaning if it’s not the first character in
the set.

@item 
To match a literal @code{']'} inside a set, precede it with a backslash, or
place it at the beginning of the set.  For example, both @code{[()[\]@{@}]} and
@code{[]()[@{@}]} will both match a parenthesis.
@end itemize

@c .. index:: single: --; in regular expressions

@c .. index:: single: &&; in regular expressions

@c .. index:: single: ~~; in regular expressions

@c .. index:: single: ||; in regular expressions


@itemize *

@item 
Support of nested sets and set operations as in Unicode Technical Standard #18@footnote{https://unicode.org/reports/tr18/} might be added in the future.  This would change the
syntax, so to facilitate this change a @ref{604,,FutureWarning} will be raised
in ambiguous cases for the time being.
That includes sets starting with a literal @code{'['} or containing literal
character sequences @code{'--'}, @code{'&&'}, @code{'~~'}, and @code{'||'}.  To
avoid a warning escape them with a backslash.
@end itemize


Changed in version 3.7: @ref{604,,FutureWarning} is raised if a character set contains constructs
that will change semantically in the future.

@end table

@geindex | (vertical bar); in regular expressions


@table @asis

@item @code{|}

@code{A|B}, where @emph{A} and @emph{B} can be arbitrary REs, creates a regular expression that
will match either @emph{A} or @emph{B}.  An arbitrary number of REs can be separated by the
@code{'|'} in this way.  This can be used inside groups (see below) as well.  As
the target string is scanned, REs separated by @code{'|'} are tried from left to
right. When one pattern completely matches, that branch is accepted. This means
that once @emph{A} matches, @emph{B} will not be tested further, even if it would
produce a longer overall match.  In other words, the @code{'|'} operator is never
greedy.  To match a literal @code{'|'}, use @code{\|}, or enclose it inside a
character class, as in @code{[|]}.
@end table

@geindex () (parentheses); in regular expressions


@table @asis

@item @code{(...)}

Matches whatever regular expression is inside the parentheses, and indicates the
start and end of a group; the contents of a group can be retrieved after a match
has been performed, and can be matched later in the string with the @code{\number}
special sequence, described below.  To match the literals @code{'('} or @code{')'},
use @code{\(} or @code{\)}, or enclose them inside a character class: @code{[(]}, @code{[)]}.
@end table

@geindex (?; in regular expressions


@table @asis

@item @code{(?...)}

This is an extension notation (a @code{'?'} following a @code{'('} is not meaningful
otherwise).  The first character after the @code{'?'} determines what the meaning
and further syntax of the construct is. Extensions usually do not create a new
group; @code{(?P<name>...)} is the only exception to this rule. Following are the
currently supported extensions.

@item @code{(?aiLmsux)}

(One or more letters from the set @code{'a'}, @code{'i'}, @code{'L'}, @code{'m'},
@code{'s'}, @code{'u'}, @code{'x'}.)  The group matches the empty string; the
letters set the corresponding flags: @ref{1bdb,,re.A} (ASCII-only matching),
@ref{1bdc,,re.I} (ignore case), @ref{1bdd,,re.L} (locale dependent),
@ref{1bde,,re.M} (multi-line), @ref{1bdf,,re.S} (dot matches all),
@code{re.U} (Unicode matching), and @ref{1be0,,re.X} (verbose),
for the entire regular expression.
(The flags are described in @ref{1be1,,Module Contents}.)
This is useful if you wish to include the flags as part of the
regular expression, instead of passing a @emph{flag} argument to the
@ref{70b,,re.compile()} function.  Flags should be used first in the
expression string.
@end table

@geindex (?;; in regular expressions


@table @asis

@item @code{(?:...)}

A non-capturing version of regular parentheses.  Matches whatever regular
expression is inside the parentheses, but the substring matched by the group
@emph{cannot} be retrieved after performing a match or referenced later in the
pattern.

@item @code{(?aiLmsux-imsx:...)}

(Zero or more letters from the set @code{'a'}, @code{'i'}, @code{'L'}, @code{'m'},
@code{'s'}, @code{'u'}, @code{'x'}, optionally followed by @code{'-'} followed by
one or more letters from the @code{'i'}, @code{'m'}, @code{'s'}, @code{'x'}.)
The letters set or remove the corresponding flags:
@ref{1bdb,,re.A} (ASCII-only matching), @ref{1bdc,,re.I} (ignore case),
@ref{1bdd,,re.L} (locale dependent), @ref{1bde,,re.M} (multi-line),
@ref{1bdf,,re.S} (dot matches all), @code{re.U} (Unicode matching),
and @ref{1be0,,re.X} (verbose), for the part of the expression.
(The flags are described in @ref{1be1,,Module Contents}.)

The letters @code{'a'}, @code{'L'} and @code{'u'} are mutually exclusive when used
as inline flags, so they can’t be combined or follow @code{'-'}.  Instead,
when one of them appears in an inline group, it overrides the matching mode
in the enclosing group.  In Unicode patterns @code{(?a:...)} switches to
ASCII-only matching, and @code{(?u:...)} switches to Unicode matching
(default).  In byte pattern @code{(?L:...)} switches to locale depending
matching, and @code{(?a:...)} switches to ASCII-only matching (default).
This override is only in effect for the narrow inline group, and the
original matching mode is restored outside of the group.


New in version 3.6.



Changed in version 3.7: The letters @code{'a'}, @code{'L'} and @code{'u'} also can be used in a group.

@end table

@geindex (?P<; in regular expressions


@table @asis

@item @code{(?P<name>...)}

Similar to regular parentheses, but the substring matched by the group is
accessible via the symbolic group name @emph{name}.  Group names must be valid
Python identifiers, and each group name must be defined only once within a
regular expression.  A symbolic group is also a numbered group, just as if
the group were not named.

Named groups can be referenced in three contexts.  If the pattern is
@code{(?P<quote>['"]).*?(?P=quote)} (i.e. matching a string quoted with either
single or double quotes):


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Context of reference to group “quote”

@tab

Ways to reference it

@item

in the same pattern itself

@tab


@itemize *

@item 
@code{(?P=quote)} (as shown)

@item 
@code{\1}
@end itemize

@item

when processing match object @emph{m}

@tab


@itemize *

@item 
@code{m.group('quote')}

@item 
@code{m.end('quote')} (etc.)
@end itemize

@item

in a string passed to the @emph{repl}
argument of @code{re.sub()}

@tab


@itemize *

@item 
@code{\g<quote>}

@item 
@code{\g<1>}

@item 
@code{\1}
@end itemize

@end multitable

@end table

@geindex (?P=; in regular expressions


@table @asis

@item @code{(?P=name)}

A backreference to a named group; it matches whatever text was matched by the
earlier group named @emph{name}.
@end table

@geindex (?#; in regular expressions


@table @asis

@item @code{(?#...)}

A comment; the contents of the parentheses are simply ignored.
@end table

@geindex (?=; in regular expressions


@table @asis

@item @code{(?=...)}

Matches if @code{...} matches next, but doesn’t consume any of the string.  This is
called a @emph{lookahead assertion}.  For example, @code{Isaac (?=Asimov)} will match
@code{'Isaac '} only if it’s followed by @code{'Asimov'}.
@end table

@geindex (?!; in regular expressions


@table @asis

@item @code{(?!...)}

Matches if @code{...} doesn’t match next.  This is a @emph{negative lookahead assertion}.
For example, @code{Isaac (?!Asimov)} will match @code{'Isaac '} only if it’s @emph{not}
followed by @code{'Asimov'}.
@end table

@geindex (?<=; in regular expressions


@table @asis

@item @code{(?<=...)}

Matches if the current position in the string is preceded by a match for @code{...}
that ends at the current position.  This is called a @emph{positive lookbehind
assertion}. @code{(?<=abc)def} will find a match in @code{'abcdef'}, since the
lookbehind will back up 3 characters and check if the contained pattern matches.
The contained pattern must only match strings of some fixed length, meaning that
@code{abc} or @code{a|b} are allowed, but @code{a*} and @code{a@{3,4@}} are not.  Note that
patterns which start with positive lookbehind assertions will not match at the
beginning of the string being searched; you will most likely want to use the
@ref{dd2,,search()} function rather than the @ref{dd3,,match()} function:

@example
>>> import re
>>> m = re.search('(?<=abc)def', 'abcdef')
>>> m.group(0)
'def'
@end example

This example looks for a word following a hyphen:

@example
>>> m = re.search(r'(?<=-)\w+', 'spam-egg')
>>> m.group(0)
'egg'
@end example


Changed in version 3.5: Added support for group references of fixed length.

@end table

@geindex (?<!; in regular expressions


@table @asis

@item @code{(?<!...)}

Matches if the current position in the string is not preceded by a match for
@code{...}.  This is called a @emph{negative lookbehind assertion}.  Similar to
positive lookbehind assertions, the contained pattern must only match strings of
some fixed length.  Patterns which start with negative lookbehind assertions may
match at the beginning of the string being searched.

@item @code{(?(id/name)yes-pattern|no-pattern)}

Will try to match with @code{yes-pattern} if the group with given @emph{id} or
@emph{name} exists, and with @code{no-pattern} if it doesn’t. @code{no-pattern} is
optional and can be omitted. For example,
@code{(<)?(\w+@@\w+(?:\.\w+)+)(?(1)>|$)} is a poor email matching pattern, which
will match with @code{'<user@@host.com>'} as well as @code{'user@@host.com'}, but
not with @code{'<user@@host.com'} nor @code{'user@@host.com>'}.
@end table

The special sequences consist of @code{'\'} and a character from the list below.
If the ordinary character is not an ASCII digit or an ASCII letter, then the
resulting RE will match the second character.  For example, @code{\$} matches the
character @code{'$'}.

@geindex \ (backslash); in regular expressions


@table @asis

@item @code{\number}

Matches the contents of the group of the same number.  Groups are numbered
starting from 1.  For example, @code{(.+) \1} matches @code{'the the'} or @code{'55 55'},
but not @code{'thethe'} (note the space after the group).  This special sequence
can only be used to match one of the first 99 groups.  If the first digit of
@emph{number} is 0, or @emph{number} is 3 octal digits long, it will not be interpreted as
a group match, but as the character with octal value @emph{number}. Inside the
@code{'['} and @code{']'} of a character class, all numeric escapes are treated as
characters.
@end table

@geindex \A; in regular expressions


@table @asis

@item @code{\A}

Matches only at the start of the string.
@end table

@geindex \b; in regular expressions


@table @asis

@item @code{\b}

Matches the empty string, but only at the beginning or end of a word.
A word is defined as a sequence of word characters.  Note that formally,
@code{\b} is defined as the boundary between a @code{\w} and a @code{\W} character
(or vice versa), or between @code{\w} and the beginning/end of the string.
This means that @code{r'\bfoo\b'} matches @code{'foo'}, @code{'foo.'}, @code{'(foo)'},
@code{'bar foo baz'} but not @code{'foobar'} or @code{'foo3'}.

By default Unicode alphanumerics are the ones used in Unicode patterns, but
this can be changed by using the @ref{697,,ASCII} flag.  Word boundaries are
determined by the current locale if the @ref{698,,LOCALE} flag is used.
Inside a character range, @code{\b} represents the backspace character, for
compatibility with Python’s string literals.
@end table

@geindex \B; in regular expressions


@table @asis

@item @code{\B}

Matches the empty string, but only when it is @emph{not} at the beginning or end
of a word.  This means that @code{r'py\B'} matches @code{'python'}, @code{'py3'},
@code{'py2'}, but not @code{'py'}, @code{'py.'}, or @code{'py!'}.
@code{\B} is just the opposite of @code{\b}, so word characters in Unicode
patterns are Unicode alphanumerics or the underscore, although this can
be changed by using the @ref{697,,ASCII} flag.  Word boundaries are
determined by the current locale if the @ref{698,,LOCALE} flag is used.
@end table

@geindex \d; in regular expressions


@table @asis

@item @code{\d}


@table @asis

@item For Unicode (str) patterns:

Matches any Unicode decimal digit (that is, any character in
Unicode character category [Nd]).  This includes @code{[0-9]}, and
also many other digit characters.  If the @ref{697,,ASCII} flag is
used only @code{[0-9]} is matched.

@item For 8-bit (bytes) patterns:

Matches any decimal digit; this is equivalent to @code{[0-9]}.
@end table
@end table

@geindex \D; in regular expressions


@table @asis

@item @code{\D}

Matches any character which is not a decimal digit. This is
the opposite of @code{\d}. If the @ref{697,,ASCII} flag is used this
becomes the equivalent of @code{[^0-9]}.
@end table

@geindex \s; in regular expressions


@table @asis

@item @code{\s}


@table @asis

@item For Unicode (str) patterns:

Matches Unicode whitespace characters (which includes
@code{[ \t\n\r\f\v]}, and also many other characters, for example the
non-breaking spaces mandated by typography rules in many
languages). If the @ref{697,,ASCII} flag is used, only
@code{[ \t\n\r\f\v]} is matched.

@item For 8-bit (bytes) patterns:

Matches characters considered whitespace in the ASCII character set;
this is equivalent to @code{[ \t\n\r\f\v]}.
@end table
@end table

@geindex \S; in regular expressions


@table @asis

@item @code{\S}

Matches any character which is not a whitespace character. This is
the opposite of @code{\s}. If the @ref{697,,ASCII} flag is used this
becomes the equivalent of @code{[^ \t\n\r\f\v]}.
@end table

@geindex \w; in regular expressions


@table @asis

@item @code{\w}


@table @asis

@item For Unicode (str) patterns:

Matches Unicode word characters; this includes most characters
that can be part of a word in any language, as well as numbers and
the underscore. If the @ref{697,,ASCII} flag is used, only
@code{[a-zA-Z0-9_]} is matched.

@item For 8-bit (bytes) patterns:

Matches characters considered alphanumeric in the ASCII character set;
this is equivalent to @code{[a-zA-Z0-9_]}.  If the @ref{698,,LOCALE} flag is
used, matches characters considered alphanumeric in the current locale
and the underscore.
@end table
@end table

@geindex \W; in regular expressions


@table @asis

@item @code{\W}

Matches any character which is not a word character. This is
the opposite of @code{\w}. If the @ref{697,,ASCII} flag is used this
becomes the equivalent of @code{[^a-zA-Z0-9_]}.  If the @ref{698,,LOCALE} flag is
used, matches characters which are neither alphanumeric in the current locale
nor the underscore.
@end table

@geindex \Z; in regular expressions


@table @asis

@item @code{\Z}

Matches only at the end of the string.
@end table

@geindex \a; in regular expressions
@geindex \b; in regular expressions
@geindex \f; in regular expressions
@geindex \n; in regular expressions
@geindex \N; in regular expressions
@geindex \r; in regular expressions
@geindex \t; in regular expressions
@geindex \u; in regular expressions
@geindex \U; in regular expressions
@geindex \v; in regular expressions
@geindex \x; in regular expressions
@geindex \\; in regular expressions

Most of the standard escapes supported by Python string literals are also
accepted by the regular expression parser:

@example
\a      \b      \f      \n
\N      \r      \t      \u
\U      \v      \x      \\
@end example

(Note that @code{\b} is used to represent word boundaries, and means “backspace”
only inside character classes.)

@code{'\u'}, @code{'\U'}, and @code{'\N'} escape sequences are only recognized in Unicode
patterns.  In bytes patterns they are errors.  Unknown escapes of ASCII
letters are reserved for future use and treated as errors.

Octal escapes are included in a limited form.  If the first digit is a 0, or if
there are three octal digits, it is considered an octal escape. Otherwise, it is
a group reference.  As for string literals, octal escapes are always at most
three digits in length.


Changed in version 3.3: The @code{'\u'} and @code{'\U'} escape sequences have been added.



Changed in version 3.6: Unknown escapes consisting of @code{'\'} and an ASCII letter now are errors.



Changed in version 3.8: The @code{'\N@{name@}'} escape sequence has been added. As in string literals,
it expands to the named Unicode character (e.g. @code{'\N@{EM DASH@}'}).


@node Module Contents,Regular Expression Objects,Regular Expression Syntax,re — Regular expression operations
@anchor{library/re contents-of-module-re}@anchor{1be1}@anchor{library/re module-contents}@anchor{1be2}
@subsubsection Module Contents


The module defines several functions, constants, and an exception. Some of the
functions are simplified versions of the full featured methods for compiled
regular expressions.  Most non-trivial applications always use the compiled
form.


Changed in version 3.6: Flag constants are now instances of @ref{1095,,RegexFlag}, which is a subclass of
@ref{11c2,,enum.IntFlag}.


@geindex compile() (in module re)
@anchor{library/re re compile}@anchor{70b}
@deffn {Function} re.compile (pattern, flags=0)

Compile a regular expression pattern into a @ref{b07,,regular expression object}, which can be used for matching using its
@ref{1be3,,match()}, @ref{1be4,,search()} and other methods, described
below.

The expression’s behaviour can be modified by specifying a @emph{flags} value.
Values can be any of the following variables, combined using bitwise OR (the
@code{|} operator).

The sequence

@example
prog = re.compile(pattern)
result = prog.match(string)
@end example

is equivalent to

@example
result = re.match(pattern, string)
@end example

but using @ref{70b,,re.compile()} and saving the resulting regular expression
object for reuse is more efficient when the expression will be used several
times in a single program.

@cartouche
@quotation Note 
The compiled versions of the most recent patterns passed to
@ref{70b,,re.compile()} and the module-level matching functions are cached, so
programs that use only a few regular expressions at a time needn’t worry
about compiling regular expressions.
@end quotation
@end cartouche
@end deffn

@geindex RegexFlag (class in re)
@anchor{library/re re RegexFlag}@anchor{1095}
@deffn {Class} re.RegexFlag

An @ref{11c2,,enum.IntFlag} class containing the regex options listed below.


New in version 3.11: - added to @code{__all__}

@end deffn

@geindex A (in module re)
@geindex ASCII (in module re)
@anchor{library/re re A}@anchor{1bdb}
@deffn {Data} re.A
@anchor{library/re re ASCII}@anchor{697}
@deffnx {Data} re.ASCII

Make @code{\w}, @code{\W}, @code{\b}, @code{\B}, @code{\d}, @code{\D}, @code{\s} and @code{\S}
perform ASCII-only matching instead of full Unicode matching.  This is only
meaningful for Unicode patterns, and is ignored for byte patterns.
Corresponds to the inline flag @code{(?a)}.

Note that for backward compatibility, the @code{re.U} flag still
exists (as well as its synonym @code{re.UNICODE} and its embedded
counterpart @code{(?u)}), but these are redundant in Python 3 since
matches are Unicode by default for strings (and Unicode matching
isn’t allowed for bytes).
@end deffn

@geindex DEBUG (in module re)
@anchor{library/re re DEBUG}@anchor{1be5}
@deffn {Data} re.DEBUG

Display debug information about compiled expression.
No corresponding inline flag.
@end deffn

@geindex I (in module re)
@geindex IGNORECASE (in module re)
@anchor{library/re re I}@anchor{1bdc}
@deffn {Data} re.I
@anchor{library/re re IGNORECASE}@anchor{1151}
@deffnx {Data} re.IGNORECASE

Perform case-insensitive matching; expressions like @code{[A-Z]} will also
match lowercase letters.  Full Unicode matching (such as @code{Ü} matching
@code{ü}) also works unless the @ref{697,,re.ASCII} flag is used to disable
non-ASCII matches.  The current locale does not change the effect of this
flag unless the @ref{698,,re.LOCALE} flag is also used.
Corresponds to the inline flag @code{(?i)}.

Note that when the Unicode patterns @code{[a-z]} or @code{[A-Z]} are used in
combination with the @ref{1151,,IGNORECASE} flag, they will match the 52 ASCII
letters and 4 additional non-ASCII letters: ‘İ’ (U+0130, Latin capital
letter I with dot above), ‘ı’ (U+0131, Latin small letter dotless i),
‘ſ’ (U+017F, Latin small letter long s) and ‘K’ (U+212A, Kelvin sign).
If the @ref{697,,ASCII} flag is used, only letters ‘a’ to ‘z’
and ‘A’ to ‘Z’ are matched.
@end deffn

@geindex L (in module re)
@geindex LOCALE (in module re)
@anchor{library/re re L}@anchor{1bdd}
@deffn {Data} re.L
@anchor{library/re re LOCALE}@anchor{698}
@deffnx {Data} re.LOCALE

Make @code{\w}, @code{\W}, @code{\b}, @code{\B} and case-insensitive matching
dependent on the current locale.  This flag can be used only with bytes
patterns.  The use of this flag is discouraged as the locale mechanism
is very unreliable, it only handles one “culture” at a time, and it only
works with 8-bit locales.  Unicode matching is already enabled by default
in Python 3 for Unicode (str) patterns, and it is able to handle different
locales/languages.
Corresponds to the inline flag @code{(?L)}.


Changed in version 3.6: @ref{698,,re.LOCALE} can be used only with bytes patterns and is
not compatible with @ref{697,,re.ASCII}.



Changed in version 3.7: Compiled regular expression objects with the @ref{698,,re.LOCALE} flag no
longer depend on the locale at compile time.  Only the locale at
matching time affects the result of matching.

@end deffn

@geindex M (in module re)
@geindex MULTILINE (in module re)
@anchor{library/re re M}@anchor{1bde}
@deffn {Data} re.M
@anchor{library/re re MULTILINE}@anchor{1bda}
@deffnx {Data} re.MULTILINE

When specified, the pattern character @code{'^'} matches at the beginning of the
string and at the beginning of each line (immediately following each newline);
and the pattern character @code{'$'} matches at the end of the string and at the
end of each line (immediately preceding each newline).  By default, @code{'^'}
matches only at the beginning of the string, and @code{'$'} only at the end of the
string and immediately before the newline (if any) at the end of the string.
Corresponds to the inline flag @code{(?m)}.
@end deffn

@geindex NOFLAG (in module re)
@anchor{library/re re NOFLAG}@anchor{1be6}
@deffn {Data} re.NOFLAG

Indicates no flag being applied, the value is @code{0}.  This flag may be used
as a default value for a function keyword argument or as a base value that
will be conditionally ORed with other flags.  Example of use as a default
value:

@example
def myfunc(text, flag=re.NOFLAG):
    return re.match(text, flag)
@end example


New in version 3.11.

@end deffn

@geindex S (in module re)
@geindex DOTALL (in module re)
@anchor{library/re re S}@anchor{1bdf}
@deffn {Data} re.S
@anchor{library/re re DOTALL}@anchor{1bd9}
@deffnx {Data} re.DOTALL

Make the @code{'.'} special character match any character at all, including a
newline; without this flag, @code{'.'} will match anything @emph{except} a newline.
Corresponds to the inline flag @code{(?s)}.
@end deffn

@geindex X (in module re)
@geindex VERBOSE (in module re)
@anchor{library/re re X}@anchor{1be0}
@deffn {Data} re.X
@anchor{library/re re VERBOSE}@anchor{1be7}
@deffnx {Data} re.VERBOSE

@geindex # (hash); in regular expressions

This flag allows you to write regular expressions that look nicer and are
more readable by allowing you to visually separate logical sections of the
pattern and add comments. Whitespace within the pattern is ignored, except
when in a character class, or when preceded by an unescaped backslash,
or within tokens like @code{*?}, @code{(?:} or @code{(?P<...>}.
When a line contains a @code{#} that is not in a character class and is not
preceded by an unescaped backslash, all characters from the leftmost such
@code{#} through the end of the line are ignored.

This means that the two following regular expression objects that match a
decimal number are functionally equal:

@example
a = re.compile(r"""\d +  # the integral part
                   \.    # the decimal point
                   \d *  # some fractional digits""", re.X)
b = re.compile(r"\d+\.\d*")
@end example

Corresponds to the inline flag @code{(?x)}.
@end deffn

@geindex search() (in module re)
@anchor{library/re re search}@anchor{dd2}
@deffn {Function} re.search (pattern, string, flags=0)

Scan through @emph{string} looking for the first location where the regular expression
@emph{pattern} produces a match, and return a corresponding @ref{b08,,match object}.  Return @code{None} if no position in the string matches the
pattern; note that this is different from finding a zero-length match at some
point in the string.
@end deffn

@geindex match() (in module re)
@anchor{library/re re match}@anchor{dd3}
@deffn {Function} re.match (pattern, string, flags=0)

If zero or more characters at the beginning of @emph{string} match the regular
expression @emph{pattern}, return a corresponding @ref{b08,,match object}.  Return @code{None} if the string does not match the pattern;
note that this is different from a zero-length match.

Note that even in @ref{1bda,,MULTILINE} mode, @ref{dd3,,re.match()} will only match
at the beginning of the string and not at the beginning of each line.

If you want to locate a match anywhere in @emph{string}, use @ref{dd2,,search()}
instead (see also @ref{1be8,,search() vs. match()}).
@end deffn

@geindex fullmatch() (in module re)
@anchor{library/re re fullmatch}@anchor{b06}
@deffn {Function} re.fullmatch (pattern, string, flags=0)

If the whole @emph{string} matches the regular expression @emph{pattern}, return a
corresponding @ref{b08,,match object}.  Return @code{None} if the
string does not match the pattern; note that this is different from a
zero-length match.


New in version 3.4.

@end deffn

@geindex split() (in module re)
@anchor{library/re re split}@anchor{699}
@deffn {Function} re.split (pattern, string, maxsplit=0, flags=0)

Split @emph{string} by the occurrences of @emph{pattern}.  If capturing parentheses are
used in @emph{pattern}, then the text of all groups in the pattern are also returned
as part of the resulting list. If @emph{maxsplit} is nonzero, at most @emph{maxsplit}
splits occur, and the remainder of the string is returned as the final element
of the list.

@example
>>> re.split(r'\W+', 'Words, words, words.')
['Words', 'words', 'words', '@w{'}]
>>> re.split(r'(\W+)', 'Words, words, words.')
['Words', ', ', 'words', ', ', 'words', '.', '@w{'}]
>>> re.split(r'\W+', 'Words, words, words.', 1)
['Words', 'words, words.']
>>> re.split('[a-f]+', '0a3B9', flags=re.IGNORECASE)
['0', '3', '9']
@end example

If there are capturing groups in the separator and it matches at the start of
the string, the result will start with an empty string.  The same holds for
the end of the string:

@example
>>> re.split(r'(\W+)', '...words, words...')
['@w{'}, '...', 'words', ', ', 'words', '...', '@w{'}]
@end example

That way, separator components are always found at the same relative
indices within the result list.

Empty matches for the pattern split the string only when not adjacent
to a previous empty match.

@example
>>> re.split(r'\b', 'Words, words, words.')
['@w{'}, 'Words', ', ', 'words', ', ', 'words', '.']
>>> re.split(r'\W*', '...words...')
['@w{'}, '@w{'}, 'w', 'o', 'r', 'd', 's', '@w{'}, '@w{'}]
>>> re.split(r'(\W*)', '...words...')
['@w{'}, '...', '@w{'}, '@w{'}, 'w', '@w{'}, 'o', '@w{'}, 'r', '@w{'}, 'd', '@w{'}, 's', '...', '@w{'}, '@w{'}, '@w{'}]
@end example


Changed in version 3.1: Added the optional flags argument.



Changed in version 3.7: Added support of splitting on a pattern that could match an empty string.

@end deffn

@geindex findall() (in module re)
@anchor{library/re re findall}@anchor{bbb}
@deffn {Function} re.findall (pattern, string, flags=0)

Return all non-overlapping matches of @emph{pattern} in @emph{string}, as a list of
strings or tuples.  The @emph{string} is scanned left-to-right, and matches
are returned in the order found.  Empty matches are included in the result.

The result depends on the number of capturing groups in the pattern.
If there are no groups, return a list of strings matching the whole
pattern.  If there is exactly one group, return a list of strings
matching that group.  If multiple groups are present, return a list
of tuples of strings matching the groups.  Non-capturing groups do not
affect the form of the result.

@example
>>> re.findall(r'\bf[a-z]*', 'which foot or hand fell fastest')
['foot', 'fell', 'fastest']
>>> re.findall(r'(\w+)=(\d+)', 'set width=20 and height=10')
[('width', '20'), ('height', '10')]
@end example


Changed in version 3.7: Non-empty matches can now start just after a previous empty match.

@end deffn

@geindex finditer() (in module re)
@anchor{library/re re finditer}@anchor{1be9}
@deffn {Function} re.finditer (pattern, string, flags=0)

Return an @ref{11c3,,iterator} yielding @ref{b08,,match objects} over
all non-overlapping matches for the RE @emph{pattern} in @emph{string}.  The @emph{string}
is scanned left-to-right, and matches are returned in the order found.  Empty
matches are included in the result.


Changed in version 3.7: Non-empty matches can now start just after a previous empty match.

@end deffn

@geindex sub() (in module re)
@anchor{library/re re sub}@anchor{72e}
@deffn {Function} re.sub (pattern, repl, string, count=0, flags=0)

Return the string obtained by replacing the leftmost non-overlapping occurrences
of @emph{pattern} in @emph{string} by the replacement @emph{repl}.  If the pattern isn’t found,
@emph{string} is returned unchanged.  @emph{repl} can be a string or a function; if it is
a string, any backslash escapes in it are processed.  That is, @code{\n} is
converted to a single newline character, @code{\r} is converted to a carriage return, and
so forth.  Unknown escapes of ASCII letters are reserved for future use and
treated as errors.  Other unknown escapes such as @code{\&} are left alone.
Backreferences, such
as @code{\6}, are replaced with the substring matched by group 6 in the pattern.
For example:

@example
>>> re.sub(r'def\s+([a-zA-Z_][a-zA-Z_0-9]*)\s*\(\s*\):',
...        r'static PyObject*\npy_\1(void)\n@{',
...        'def myfunc():')
'static PyObject*\npy_myfunc(void)\n@{'
@end example

If @emph{repl} is a function, it is called for every non-overlapping occurrence of
@emph{pattern}.  The function takes a single @ref{b08,,match object}
argument, and returns the replacement string.  For example:

@example
>>> def dashrepl(matchobj):
...     if matchobj.group(0) == '-': return ' '
...     else: return '-'
>>> re.sub('-@{1,2@}', dashrepl, 'pro----gram-files')
'pro--gram files'
>>> re.sub(r'\sAND\s', ' & ', 'Baked Beans And Spam', flags=re.IGNORECASE)
'Baked Beans & Spam'
@end example

The pattern may be a string or a @ref{b07,,pattern object}.

The optional argument @emph{count} is the maximum number of pattern occurrences to be
replaced; @emph{count} must be a non-negative integer.  If omitted or zero, all
occurrences will be replaced. Empty matches for the pattern are replaced only
when not adjacent to a previous empty match, so @code{sub('x*', '-', 'abxd')} returns
@code{'-a-b--d-'}.

@geindex \g; in regular expressions

In string-type @emph{repl} arguments, in addition to the character escapes and
backreferences described above,
@code{\g<name>} will use the substring matched by the group named @code{name}, as
defined by the @code{(?P<name>...)} syntax. @code{\g<number>} uses the corresponding
group number; @code{\g<2>} is therefore equivalent to @code{\2}, but isn’t ambiguous
in a replacement such as @code{\g<2>0}.  @code{\20} would be interpreted as a
reference to group 20, not a reference to group 2 followed by the literal
character @code{'0'}.  The backreference @code{\g<0>} substitutes in the entire
substring matched by the RE.


Changed in version 3.1: Added the optional flags argument.



Changed in version 3.5: Unmatched groups are replaced with an empty string.



Changed in version 3.6: Unknown escapes in @emph{pattern} consisting of @code{'\'} and an ASCII letter
now are errors.



Changed in version 3.7: Unknown escapes in @emph{repl} consisting of @code{'\'} and an ASCII letter
now are errors.



Changed in version 3.7: Empty matches for the pattern are replaced when adjacent to a previous
non-empty match.

@end deffn

@geindex subn() (in module re)
@anchor{library/re re subn}@anchor{9b3}
@deffn {Function} re.subn (pattern, repl, string, count=0, flags=0)

Perform the same operation as @ref{72e,,sub()}, but return a tuple @code{(new_string,
number_of_subs_made)}.


Changed in version 3.1: Added the optional flags argument.



Changed in version 3.5: Unmatched groups are replaced with an empty string.

@end deffn

@geindex escape() (in module re)
@anchor{library/re re escape}@anchor{744}
@deffn {Function} re.escape (pattern)

Escape special characters in @emph{pattern}.
This is useful if you want to match an arbitrary literal string that may
have regular expression metacharacters in it.  For example:

@example
>>> print(re.escape('https://www.python.org'))
https://www\.python\.org

>>> legal_chars = string.ascii_lowercase + string.digits + "!#$%&'*+-.^_`|~:"
>>> print('[%s]+' % re.escape(legal_chars))
[abcdefghijklmnopqrstuvwxyz0123456789!\#\$%\&'\*\+\-\.\^_`\|\~:]+

>>> operators = ['+', '-', '*', '/', '**']
>>> print('|'.join(map(re.escape, sorted(operators, reverse=True))))
/|\-|\+|\*\*|\*
@end example

This function must not be used for the replacement string in @ref{72e,,sub()}
and @ref{9b3,,subn()}, only backslashes should be escaped.  For example:

@example
>>> digits_re = r'\d+'
>>> sample = '/usr/sbin/sendmail - 0 errors, 12 warnings'
>>> print(re.sub(digits_re, digits_re.replace('\\', r'\\'), sample))
/usr/sbin/sendmail - \d+ errors, \d+ warnings
@end example


Changed in version 3.3: The @code{'_'} character is no longer escaped.



Changed in version 3.7: Only characters that can have special meaning in a regular expression
are escaped. As a result, @code{'!'}, @code{'"'}, @code{'%'}, @code{"'"}, @code{','},
@code{'/'}, @code{':'}, @code{';'}, @code{'<'}, @code{'='}, @code{'>'}, @code{'@@'}, and
@code{"`"} are no longer escaped.

@end deffn

@geindex purge() (in module re)
@anchor{library/re re purge}@anchor{1bea}
@deffn {Function} re.purge ()

Clear the regular expression cache.
@end deffn

@geindex error
@anchor{library/re re error}@anchor{9b4}
@deffn {Exception} re.error (msg, pattern=None, pos=None)

Exception raised when a string passed to one of the functions here is not a
valid regular expression (for example, it might contain unmatched parentheses)
or when some other error occurs during compilation or matching.  It is never an
error if a string contains no match for a pattern.  The error instance has
the following additional attributes:

@geindex msg (re.error attribute)
@anchor{library/re re error msg}@anchor{9b5}
@deffn {Attribute} msg

The unformatted error message.
@end deffn

@geindex pattern (re.error attribute)
@anchor{library/re re error pattern}@anchor{9b6}
@deffn {Attribute} pattern

The regular expression pattern.
@end deffn

@geindex pos (re.error attribute)
@anchor{library/re re error pos}@anchor{9b7}
@deffn {Attribute} pos

The index in @emph{pattern} where compilation failed (may be @code{None}).
@end deffn

@geindex lineno (re.error attribute)
@anchor{library/re re error lineno}@anchor{9b8}
@deffn {Attribute} lineno

The line corresponding to @emph{pos} (may be @code{None}).
@end deffn

@geindex colno (re.error attribute)
@anchor{library/re re error colno}@anchor{9b9}
@deffn {Attribute} colno

The column corresponding to @emph{pos} (may be @code{None}).
@end deffn


Changed in version 3.5: Added additional attributes.

@end deffn

@node Regular Expression Objects,Match Objects,Module Contents,re — Regular expression operations
@anchor{library/re re-objects}@anchor{b07}@anchor{library/re regular-expression-objects}@anchor{1beb}
@subsubsection Regular Expression Objects


Compiled regular expression objects support the following methods and
attributes:

@geindex search() (re.Pattern method)
@anchor{library/re re Pattern search}@anchor{1be4}
@deffn {Method} Pattern.search (string[, pos[, endpos]])

Scan through @emph{string} looking for the first location where this regular
expression produces a match, and return a corresponding @ref{b08,,match object}.  Return @code{None} if no position in the string matches the
pattern; note that this is different from finding a zero-length match at some
point in the string.

The optional second parameter @emph{pos} gives an index in the string where the
search is to start; it defaults to @code{0}.  This is not completely equivalent to
slicing the string; the @code{'^'} pattern character matches at the real beginning
of the string and at positions just after a newline, but not necessarily at the
index where the search is to start.

The optional parameter @emph{endpos} limits how far the string will be searched; it
will be as if the string is @emph{endpos} characters long, so only the characters
from @emph{pos} to @code{endpos - 1} will be searched for a match.  If @emph{endpos} is less
than @emph{pos}, no match will be found; otherwise, if @emph{rx} is a compiled regular
expression object, @code{rx.search(string, 0, 50)} is equivalent to
@code{rx.search(string[:50], 0)}.

@example
>>> pattern = re.compile("d")
>>> pattern.search("dog")     # Match at index 0
<re.Match object; span=(0, 1), match='d'>
>>> pattern.search("dog", 1)  # No match; search doesn't include the "d"
@end example
@end deffn

@geindex match() (re.Pattern method)
@anchor{library/re re Pattern match}@anchor{1be3}
@deffn {Method} Pattern.match (string[, pos[, endpos]])

If zero or more characters at the @emph{beginning} of @emph{string} match this regular
expression, return a corresponding @ref{b08,,match object}.
Return @code{None} if the string does not match the pattern; note that this is
different from a zero-length match.

The optional @emph{pos} and @emph{endpos} parameters have the same meaning as for the
@ref{1be4,,search()} method.

@example
>>> pattern = re.compile("o")
>>> pattern.match("dog")      # No match as "o" is not at the start of "dog".
>>> pattern.match("dog", 1)   # Match as "o" is the 2nd character of "dog".
<re.Match object; span=(1, 2), match='o'>
@end example

If you want to locate a match anywhere in @emph{string}, use
@ref{1be4,,search()} instead (see also @ref{1be8,,search() vs. match()}).
@end deffn

@geindex fullmatch() (re.Pattern method)
@anchor{library/re re Pattern fullmatch}@anchor{1bec}
@deffn {Method} Pattern.fullmatch (string[, pos[, endpos]])

If the whole @emph{string} matches this regular expression, return a corresponding
@ref{b08,,match object}.  Return @code{None} if the string does not
match the pattern; note that this is different from a zero-length match.

The optional @emph{pos} and @emph{endpos} parameters have the same meaning as for the
@ref{1be4,,search()} method.

@example
>>> pattern = re.compile("o[gh]")
>>> pattern.fullmatch("dog")      # No match as "o" is not at the start of "dog".
>>> pattern.fullmatch("ogre")     # No match as not the full string matches.
>>> pattern.fullmatch("doggie", 1, 3)   # Matches within given limits.
<re.Match object; span=(1, 3), match='og'>
@end example


New in version 3.4.

@end deffn

@geindex split() (re.Pattern method)
@anchor{library/re re Pattern split}@anchor{1bed}
@deffn {Method} Pattern.split (string, maxsplit=0)

Identical to the @ref{699,,split()} function, using the compiled pattern.
@end deffn

@geindex findall() (re.Pattern method)
@anchor{library/re re Pattern findall}@anchor{1bee}
@deffn {Method} Pattern.findall (string[, pos[, endpos]])

Similar to the @ref{bbb,,findall()} function, using the compiled pattern, but
also accepts optional @emph{pos} and @emph{endpos} parameters that limit the search
region like for @ref{dd2,,search()}.
@end deffn

@geindex finditer() (re.Pattern method)
@anchor{library/re re Pattern finditer}@anchor{1bef}
@deffn {Method} Pattern.finditer (string[, pos[, endpos]])

Similar to the @ref{1be9,,finditer()} function, using the compiled pattern, but
also accepts optional @emph{pos} and @emph{endpos} parameters that limit the search
region like for @ref{dd2,,search()}.
@end deffn

@geindex sub() (re.Pattern method)
@anchor{library/re re Pattern sub}@anchor{1bf0}
@deffn {Method} Pattern.sub (repl, string, count=0)

Identical to the @ref{72e,,sub()} function, using the compiled pattern.
@end deffn

@geindex subn() (re.Pattern method)
@anchor{library/re re Pattern subn}@anchor{1bf1}
@deffn {Method} Pattern.subn (repl, string, count=0)

Identical to the @ref{9b3,,subn()} function, using the compiled pattern.
@end deffn

@geindex flags (re.Pattern attribute)
@anchor{library/re re Pattern flags}@anchor{1bf2}
@deffn {Attribute} Pattern.flags

The regex matching flags.  This is a combination of the flags given to
@ref{70b,,compile()}, any @code{(?...)} inline flags in the pattern, and implicit
flags such as @code{UNICODE} if the pattern is a Unicode string.
@end deffn

@geindex groups (re.Pattern attribute)
@anchor{library/re re Pattern groups}@anchor{1bf3}
@deffn {Attribute} Pattern.groups

The number of capturing groups in the pattern.
@end deffn

@geindex groupindex (re.Pattern attribute)
@anchor{library/re re Pattern groupindex}@anchor{1bf4}
@deffn {Attribute} Pattern.groupindex

A dictionary mapping any symbolic group names defined by @code{(?P<id>)} to group
numbers.  The dictionary is empty if no symbolic groups were used in the
pattern.
@end deffn

@geindex pattern (re.Pattern attribute)
@anchor{library/re re Pattern pattern}@anchor{1bf5}
@deffn {Attribute} Pattern.pattern

The pattern string from which the pattern object was compiled.
@end deffn


Changed in version 3.7: Added support of @ref{69a,,copy.copy()} and @ref{69b,,copy.deepcopy()}.  Compiled
regular expression objects are considered atomic.


@node Match Objects,Regular Expression Examples,Regular Expression Objects,re — Regular expression operations
@anchor{library/re id2}@anchor{1bf6}@anchor{library/re match-objects}@anchor{b08}
@subsubsection Match Objects


Match objects always have a boolean value of @code{True}.
Since @ref{1be3,,match()} and @ref{1be4,,search()} return @code{None}
when there is no match, you can test whether there was a match with a simple
@code{if} statement:

@example
match = re.search(pattern, string)
if match:
    process(match)
@end example

Match objects support the following methods and attributes:

@geindex expand() (re.Match method)
@anchor{library/re re Match expand}@anchor{1bf7}
@deffn {Method} Match.expand (template)

Return the string obtained by doing backslash substitution on the template
string @emph{template}, as done by the @ref{1bf0,,sub()} method.
Escapes such as @code{\n} are converted to the appropriate characters,
and numeric backreferences (@code{\1}, @code{\2}) and named backreferences
(@code{\g<1>}, @code{\g<name>}) are replaced by the contents of the
corresponding group.


Changed in version 3.5: Unmatched groups are replaced with an empty string.

@end deffn

@geindex group() (re.Match method)
@anchor{library/re re Match group}@anchor{1bf8}
@deffn {Method} Match.group ([group1, ...])

Returns one or more subgroups of the match.  If there is a single argument, the
result is a single string; if there are multiple arguments, the result is a
tuple with one item per argument. Without arguments, @emph{group1} defaults to zero
(the whole match is returned). If a @emph{groupN} argument is zero, the corresponding
return value is the entire matching string; if it is in the inclusive range
[1..99], it is the string matching the corresponding parenthesized group.  If a
group number is negative or larger than the number of groups defined in the
pattern, an @ref{1066,,IndexError} exception is raised. If a group is contained in a
part of the pattern that did not match, the corresponding result is @code{None}.
If a group is contained in a part of the pattern that matched multiple times,
the last match is returned.

@example
>>> m = re.match(r"(\w+) (\w+)", "Isaac Newton, physicist")
>>> m.group(0)       # The entire match
'Isaac Newton'
>>> m.group(1)       # The first parenthesized subgroup.
'Isaac'
>>> m.group(2)       # The second parenthesized subgroup.
'Newton'
>>> m.group(1, 2)    # Multiple arguments give us a tuple.
('Isaac', 'Newton')
@end example

If the regular expression uses the @code{(?P<name>...)} syntax, the @emph{groupN}
arguments may also be strings identifying groups by their group name.  If a
string argument is not used as a group name in the pattern, an @ref{1066,,IndexError}
exception is raised.

A moderately complicated example:

@example
>>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds")
>>> m.group('first_name')
'Malcolm'
>>> m.group('last_name')
'Reynolds'
@end example

Named groups can also be referred to by their index:

@example
>>> m.group(1)
'Malcolm'
>>> m.group(2)
'Reynolds'
@end example

If a group matches multiple times, only the last match is accessible:

@example
>>> m = re.match(r"(..)+", "a1b2c3")  # Matches 3 times.
>>> m.group(1)                        # Returns only the last match.
'c3'
@end example
@end deffn

@geindex __getitem__() (re.Match method)
@anchor{library/re re Match __getitem__}@anchor{1bf9}
@deffn {Method} Match.__getitem__ (g)

This is identical to @code{m.group(g)}.  This allows easier access to
an individual group from a match:

@example
>>> m = re.match(r"(\w+) (\w+)", "Isaac Newton, physicist")
>>> m[0]       # The entire match
'Isaac Newton'
>>> m[1]       # The first parenthesized subgroup.
'Isaac'
>>> m[2]       # The second parenthesized subgroup.
'Newton'
@end example


New in version 3.6.

@end deffn

@geindex groups() (re.Match method)
@anchor{library/re re Match groups}@anchor{1bfa}
@deffn {Method} Match.groups (default=None)

Return a tuple containing all the subgroups of the match, from 1 up to however
many groups are in the pattern.  The @emph{default} argument is used for groups that
did not participate in the match; it defaults to @code{None}.

For example:

@example
>>> m = re.match(r"(\d+)\.(\d+)", "24.1632")
>>> m.groups()
('24', '1632')
@end example

If we make the decimal place and everything after it optional, not all groups
might participate in the match.  These groups will default to @code{None} unless
the @emph{default} argument is given:

@example
>>> m = re.match(r"(\d+)\.?(\d+)?", "24")
>>> m.groups()      # Second group defaults to None.
('24', None)
>>> m.groups('0')   # Now, the second group defaults to '0'.
('24', '0')
@end example
@end deffn

@geindex groupdict() (re.Match method)
@anchor{library/re re Match groupdict}@anchor{1bfb}
@deffn {Method} Match.groupdict (default=None)

Return a dictionary containing all the @emph{named} subgroups of the match, keyed by
the subgroup name.  The @emph{default} argument is used for groups that did not
participate in the match; it defaults to @code{None}.  For example:

@example
>>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds")
>>> m.groupdict()
@{'first_name': 'Malcolm', 'last_name': 'Reynolds'@}
@end example
@end deffn

@geindex start() (re.Match method)
@geindex end() (re.Match method)
@anchor{library/re re Match start}@anchor{1bfc}
@deffn {Method} Match.start ([group])
@anchor{library/re re Match end}@anchor{1bfd}
@deffnx {Method} Match.end ([group])

Return the indices of the start and end of the substring matched by @emph{group};
@emph{group} defaults to zero (meaning the whole matched substring). Return @code{-1} if
@emph{group} exists but did not contribute to the match.  For a match object @emph{m}, and
a group @emph{g} that did contribute to the match, the substring matched by group @emph{g}
(equivalent to @code{m.group(g)}) is

@example
m.string[m.start(g):m.end(g)]
@end example

Note that @code{m.start(group)} will equal @code{m.end(group)} if @emph{group} matched a
null string.  For example, after @code{m = re.search('b(c?)', 'cba')},
@code{m.start(0)} is 1, @code{m.end(0)} is 2, @code{m.start(1)} and @code{m.end(1)} are both
2, and @code{m.start(2)} raises an @ref{1066,,IndexError} exception.

An example that will remove @emph{remove_this} from email addresses:

@example
>>> email = "tony@@tiremove_thisger.net"
>>> m = re.search("remove_this", email)
>>> email[:m.start()] + email[m.end():]
'tony@@tiger.net'
@end example
@end deffn

@geindex span() (re.Match method)
@anchor{library/re re Match span}@anchor{1bfe}
@deffn {Method} Match.span ([group])

For a match @emph{m}, return the 2-tuple @code{(m.start(group), m.end(group))}. Note
that if @emph{group} did not contribute to the match, this is @code{(-1, -1)}.
@emph{group} defaults to zero, the entire match.
@end deffn

@geindex pos (re.Match attribute)
@anchor{library/re re Match pos}@anchor{1bff}
@deffn {Attribute} Match.pos

The value of @emph{pos} which was passed to the @ref{1be4,,search()} or
@ref{1be3,,match()} method of a @ref{b07,,regex object}.  This is
the index into the string at which the RE engine started looking for a match.
@end deffn

@geindex endpos (re.Match attribute)
@anchor{library/re re Match endpos}@anchor{1c00}
@deffn {Attribute} Match.endpos

The value of @emph{endpos} which was passed to the @ref{1be4,,search()} or
@ref{1be3,,match()} method of a @ref{b07,,regex object}.  This is
the index into the string beyond which the RE engine will not go.
@end deffn

@geindex lastindex (re.Match attribute)
@anchor{library/re re Match lastindex}@anchor{1c01}
@deffn {Attribute} Match.lastindex

The integer index of the last matched capturing group, or @code{None} if no group
was matched at all. For example, the expressions @code{(a)b}, @code{((a)(b))}, and
@code{((ab))} will have @code{lastindex == 1} if applied to the string @code{'ab'}, while
the expression @code{(a)(b)} will have @code{lastindex == 2}, if applied to the same
string.
@end deffn

@geindex lastgroup (re.Match attribute)
@anchor{library/re re Match lastgroup}@anchor{1c02}
@deffn {Attribute} Match.lastgroup

The name of the last matched capturing group, or @code{None} if the group didn’t
have a name, or if no group was matched at all.
@end deffn

@geindex re (re.Match attribute)
@anchor{library/re re Match re}@anchor{1c03}
@deffn {Attribute} Match.re

The @ref{b07,,regular expression object} whose @ref{1be3,,match()} or
@ref{1be4,,search()} method produced this match instance.
@end deffn

@geindex string (re.Match attribute)
@anchor{library/re re Match string}@anchor{1c04}
@deffn {Attribute} Match.string

The string passed to @ref{1be3,,match()} or @ref{1be4,,search()}.
@end deffn


Changed in version 3.7: Added support of @ref{69a,,copy.copy()} and @ref{69b,,copy.deepcopy()}.  Match objects
are considered atomic.


@node Regular Expression Examples,,Match Objects,re — Regular expression operations
@anchor{library/re re-examples}@anchor{e0e}@anchor{library/re regular-expression-examples}@anchor{1c05}
@subsubsection Regular Expression Examples


@menu
* Checking for a Pair:: 
* Simulating scanf(): Simulating scanf. 
* search() vs. match(): search vs match. 
* Making a Phonebook:: 
* Text Munging:: 
* Finding all Adverbs:: 
* Finding all Adverbs and their Positions:: 
* Raw String Notation:: 
* Writing a Tokenizer:: 

@end menu

@node Checking for a Pair,Simulating scanf,,Regular Expression Examples
@anchor{library/re checking-for-a-pair}@anchor{1c06}
@subsubsection Checking for a Pair


In this example, we’ll use the following helper function to display match
objects a little more gracefully:

@example
def displaymatch(match):
    if match is None:
        return None
    return '<Match: %r, groups=%r>' % (match.group(), match.groups())
@end example

Suppose you are writing a poker program where a player’s hand is represented as
a 5-character string with each character representing a card, “a” for ace, “k”
for king, “q” for queen, “j” for jack, “t” for 10, and “2” through “9”
representing the card with that value.

To see if a given string is a valid hand, one could do the following:

@example
>>> valid = re.compile(r"^[a2-9tjqk]@{5@}$")
>>> displaymatch(valid.match("akt5q"))  # Valid.
"<Match: 'akt5q', groups=()>"
>>> displaymatch(valid.match("akt5e"))  # Invalid.
>>> displaymatch(valid.match("akt"))    # Invalid.
>>> displaymatch(valid.match("727ak"))  # Valid.
"<Match: '727ak', groups=()>"
@end example

That last hand, @code{"727ak"}, contained a pair, or two of the same valued cards.
To match this with a regular expression, one could use backreferences as such:

@example
>>> pair = re.compile(r".*(.).*\1")
>>> displaymatch(pair.match("717ak"))     # Pair of 7s.
"<Match: '717', groups=('7',)>"
>>> displaymatch(pair.match("718ak"))     # No pairs.
>>> displaymatch(pair.match("354aa"))     # Pair of aces.
"<Match: '354aa', groups=('a',)>"
@end example

To find out what card the pair consists of, one could use the
@ref{1bf8,,group()} method of the match object in the following manner:

@example
>>> pair = re.compile(r".*(.).*\1")
>>> pair.match("717ak").group(1)
'7'

# Error because re.match() returns None, which doesn't have a group() method:
>>> pair.match("718ak").group(1)
Traceback (most recent call last):
  File "<pyshell#23>", line 1, in <module>
    re.match(r".*(.).*\1", "718ak").group(1)
AttributeError: 'NoneType' object has no attribute 'group'

>>> pair.match("354aa").group(1)
'a'
@end example

@node Simulating scanf,search vs match,Checking for a Pair,Regular Expression Examples
@anchor{library/re simulating-scanf}@anchor{1c07}
@subsubsection Simulating scanf()


@geindex scanf()

Python does not currently have an equivalent to @code{scanf()}.  Regular
expressions are generally more powerful, though also more verbose, than
@code{scanf()} format strings.  The table below offers some more-or-less
equivalent mappings between @code{scanf()} format tokens and regular
expressions.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

@code{scanf()} Token

@tab

Regular Expression

@item

@code{%c}

@tab

@code{.}

@item

@code{%5c}

@tab

@code{.@{5@}}

@item

@code{%d}

@tab

@code{[-+]?\d+}

@item

@code{%e}, @code{%E}, @code{%f}, @code{%g}

@tab

@code{[-+]?(\d+(\.\d*)?|\.\d+)([eE][-+]?\d+)?}

@item

@code{%i}

@tab

@code{[-+]?(0[xX][\dA-Fa-f]+|0[0-7]*|\d+)}

@item

@code{%o}

@tab

@code{[-+]?[0-7]+}

@item

@code{%s}

@tab

@code{\S+}

@item

@code{%u}

@tab

@code{\d+}

@item

@code{%x}, @code{%X}

@tab

@code{[-+]?(0[xX])?[\dA-Fa-f]+}

@end multitable


To extract the filename and numbers from a string like

@example
/usr/sbin/sendmail - 0 errors, 4 warnings
@end example

you would use a @code{scanf()} format like

@example
%s - %d errors, %d warnings
@end example

The equivalent regular expression would be

@example
(\S+) - (\d+) errors, (\d+) warnings
@end example

@node search vs match,Making a Phonebook,Simulating scanf,Regular Expression Examples
@anchor{library/re id3}@anchor{1c08}@anchor{library/re search-vs-match}@anchor{1be8}
@subsubsection search() vs. match()


Python offers two different primitive operations based on regular expressions:
@ref{dd3,,re.match()} checks for a match only at the beginning of the string, while
@ref{dd2,,re.search()} checks for a match anywhere in the string (this is what Perl
does by default).

For example:

@example
>>> re.match("c", "abcdef")    # No match
>>> re.search("c", "abcdef")   # Match
<re.Match object; span=(2, 3), match='c'>
@end example

Regular expressions beginning with @code{'^'} can be used with @ref{dd2,,search()} to
restrict the match at the beginning of the string:

@example
>>> re.match("c", "abcdef")    # No match
>>> re.search("^c", "abcdef")  # No match
>>> re.search("^a", "abcdef")  # Match
<re.Match object; span=(0, 1), match='a'>
@end example

Note however that in @ref{1bda,,MULTILINE} mode @ref{dd3,,match()} only matches at the
beginning of the string, whereas using @ref{dd2,,search()} with a regular expression
beginning with @code{'^'} will match at the beginning of each line.

@example
>>> re.match('X', 'A\nB\nX', re.MULTILINE)  # No match
>>> re.search('^X', 'A\nB\nX', re.MULTILINE)  # Match
<re.Match object; span=(4, 5), match='X'>
@end example

@node Making a Phonebook,Text Munging,search vs match,Regular Expression Examples
@anchor{library/re making-a-phonebook}@anchor{1c09}
@subsubsection Making a Phonebook


@ref{699,,split()} splits a string into a list delimited by the passed pattern.  The
method is invaluable for converting textual data into data structures that can be
easily read and modified by Python as demonstrated in the following example that
creates a phonebook.

First, here is the input.  Normally it may come from a file, here we are using
triple-quoted string syntax

@example
>>> text = """Ross McFluff: 834.345.1254 155 Elm Street
...
... Ronald Heathmore: 892.345.3428 436 Finley Avenue
... Frank Burger: 925.541.7625 662 South Dogwood Way
...
...
... Heather Albrecht: 548.326.4584 919 Park Place"""
@end example

The entries are separated by one or more newlines. Now we convert the string
into a list with each nonempty line having its own entry:

@example
>>> entries = re.split("\n+", text)
>>> entries
['Ross McFluff: 834.345.1254 155 Elm Street',
'Ronald Heathmore: 892.345.3428 436 Finley Avenue',
'Frank Burger: 925.541.7625 662 South Dogwood Way',
'Heather Albrecht: 548.326.4584 919 Park Place']
@end example

Finally, split each entry into a list with first name, last name, telephone
number, and address.  We use the @code{maxsplit} parameter of @ref{699,,split()}
because the address has spaces, our splitting pattern, in it:

@example
>>> [re.split(":? ", entry, 3) for entry in entries]
[['Ross', 'McFluff', '834.345.1254', '155 Elm Street'],
['Ronald', 'Heathmore', '892.345.3428', '436 Finley Avenue'],
['Frank', 'Burger', '925.541.7625', '662 South Dogwood Way'],
['Heather', 'Albrecht', '548.326.4584', '919 Park Place']]
@end example

The @code{:?} pattern matches the colon after the last name, so that it does not
occur in the result list.  With a @code{maxsplit} of @code{4}, we could separate the
house number from the street name:

@example
>>> [re.split(":? ", entry, 4) for entry in entries]
[['Ross', 'McFluff', '834.345.1254', '155', 'Elm Street'],
['Ronald', 'Heathmore', '892.345.3428', '436', 'Finley Avenue'],
['Frank', 'Burger', '925.541.7625', '662', 'South Dogwood Way'],
['Heather', 'Albrecht', '548.326.4584', '919', 'Park Place']]
@end example

@node Text Munging,Finding all Adverbs,Making a Phonebook,Regular Expression Examples
@anchor{library/re text-munging}@anchor{1c0a}
@subsubsection Text Munging


@ref{72e,,sub()} replaces every occurrence of a pattern with a string or the
result of a function.  This example demonstrates using @ref{72e,,sub()} with
a function to “munge” text, or randomize the order of all the characters
in each word of a sentence except for the first and last characters:

@example
>>> def repl(m):
...     inner_word = list(m.group(2))
...     random.shuffle(inner_word)
...     return m.group(1) + "".join(inner_word) + m.group(3)
>>> text = "Professor Abdolmalek, please report your absences promptly."
>>> re.sub(r"(\w)(\w+)(\w)", repl, text)
'Poefsrosr Aealmlobdk, pslaee reorpt your abnseces plmrptoy.'
>>> re.sub(r"(\w)(\w+)(\w)", repl, text)
'Pofsroser Aodlambelk, plasee reoprt yuor asnebces potlmrpy.'
@end example

@node Finding all Adverbs,Finding all Adverbs and their Positions,Text Munging,Regular Expression Examples
@anchor{library/re finding-all-adverbs}@anchor{1c0b}
@subsubsection Finding all Adverbs


@ref{bbb,,findall()} matches @emph{all} occurrences of a pattern, not just the first
one as @ref{dd2,,search()} does.  For example, if a writer wanted to
find all of the adverbs in some text, they might use @ref{bbb,,findall()} in
the following manner:

@example
>>> text = "He was carefully disguised but captured quickly by police."
>>> re.findall(r"\w+ly\b", text)
['carefully', 'quickly']
@end example

@node Finding all Adverbs and their Positions,Raw String Notation,Finding all Adverbs,Regular Expression Examples
@anchor{library/re finding-all-adverbs-and-their-positions}@anchor{1c0c}
@subsubsection Finding all Adverbs and their Positions


If one wants more information about all matches of a pattern than the matched
text, @ref{1be9,,finditer()} is useful as it provides @ref{b08,,match objects} instead of strings.  Continuing with the previous example, if
a writer wanted to find all of the adverbs @emph{and their positions} in
some text, they would use @ref{1be9,,finditer()} in the following manner:

@example
>>> text = "He was carefully disguised but captured quickly by police."
>>> for m in re.finditer(r"\w+ly\b", text):
...     print('%02d-%02d: %s' % (m.start(), m.end(), m.group(0)))
07-16: carefully
40-47: quickly
@end example

@node Raw String Notation,Writing a Tokenizer,Finding all Adverbs and their Positions,Regular Expression Examples
@anchor{library/re raw-string-notation}@anchor{1c0d}
@subsubsection Raw String Notation


Raw string notation (@code{r"text"}) keeps regular expressions sane.  Without it,
every backslash (@code{'\'}) in a regular expression would have to be prefixed with
another one to escape it.  For example, the two following lines of code are
functionally identical:

@example
>>> re.match(r"\W(.)\1\W", " ff ")
<re.Match object; span=(0, 4), match=' ff '>
>>> re.match("\\W(.)\\1\\W", " ff ")
<re.Match object; span=(0, 4), match=' ff '>
@end example

When one wants to match a literal backslash, it must be escaped in the regular
expression.  With raw string notation, this means @code{r"\\"}.  Without raw string
notation, one must use @code{"\\\\"}, making the following lines of code
functionally identical:

@example
>>> re.match(r"\\", r"\\")
<re.Match object; span=(0, 1), match='\\'>
>>> re.match("\\\\", r"\\")
<re.Match object; span=(0, 1), match='\\'>
@end example

@node Writing a Tokenizer,,Raw String Notation,Regular Expression Examples
@anchor{library/re writing-a-tokenizer}@anchor{1c0e}
@subsubsection Writing a Tokenizer


A tokenizer or scanner@footnote{https://en.wikipedia.org/wiki/Lexical_analysis}
analyzes a string to categorize groups of characters.  This is a useful first
step in writing a compiler or interpreter.

The text categories are specified with regular expressions.  The technique is
to combine those into a single master regular expression and to loop over
successive matches:

@example
from typing import NamedTuple
import re

class Token(NamedTuple):
    type: str
    value: str
    line: int
    column: int

def tokenize(code):
    keywords = @{'IF', 'THEN', 'ENDIF', 'FOR', 'NEXT', 'GOSUB', 'RETURN'@}
    token_specification = [
        ('NUMBER',   r'\d+(\.\d*)?'),  # Integer or decimal number
        ('ASSIGN',   r':='),           # Assignment operator
        ('END',      r';'),            # Statement terminator
        ('ID',       r'[A-Za-z]+'),    # Identifiers
        ('OP',       r'[+\-*/]'),      # Arithmetic operators
        ('NEWLINE',  r'\n'),           # Line endings
        ('SKIP',     r'[ \t]+'),       # Skip over spaces and tabs
        ('MISMATCH', r'.'),            # Any other character
    ]
    tok_regex = '|'.join('(?P<%s>%s)' % pair for pair in token_specification)
    line_num = 1
    line_start = 0
    for mo in re.finditer(tok_regex, code):
        kind = mo.lastgroup
        value = mo.group()
        column = mo.start() - line_start
        if kind == 'NUMBER':
            value = float(value) if '.' in value else int(value)
        elif kind == 'ID' and value in keywords:
            kind = value
        elif kind == 'NEWLINE':
            line_start = mo.end()
            line_num += 1
            continue
        elif kind == 'SKIP':
            continue
        elif kind == 'MISMATCH':
            raise RuntimeError(f'@{value!r@} unexpected on line @{line_num@}')
        yield Token(kind, value, line_num, column)

statements = '@w{'}'
    IF quantity THEN
        total := total + price * quantity;
        tax := price * 0.05;
    ENDIF;
'@w{'}'

for token in tokenize(statements):
    print(token)
@end example

The tokenizer produces the following output:

@example
Token(type='IF', value='IF', line=2, column=4)
Token(type='ID', value='quantity', line=2, column=7)
Token(type='THEN', value='THEN', line=2, column=16)
Token(type='ID', value='total', line=3, column=8)
Token(type='ASSIGN', value=':=', line=3, column=14)
Token(type='ID', value='total', line=3, column=17)
Token(type='OP', value='+', line=3, column=23)
Token(type='ID', value='price', line=3, column=25)
Token(type='OP', value='*', line=3, column=31)
Token(type='ID', value='quantity', line=3, column=33)
Token(type='END', value=';', line=3, column=41)
Token(type='ID', value='tax', line=4, column=8)
Token(type='ASSIGN', value=':=', line=4, column=12)
Token(type='ID', value='price', line=4, column=15)
Token(type='OP', value='*', line=4, column=21)
Token(type='NUMBER', value=0.05, line=4, column=23)
Token(type='END', value=';', line=4, column=27)
Token(type='ENDIF', value='ENDIF', line=5, column=4)
Token(type='END', value=';', line=5, column=9)
@end example

@anchor{library/re frie09}@anchor{1bd7}@w{(Frie09)} 
Friedl, Jeffrey. Mastering Regular Expressions. 3rd ed., O’Reilly Media, 2009. The third edition of the book no longer covers Python at all, but the first edition covered writing good regular expression patterns in great detail.

@node difflib — Helpers for computing deltas,textwrap — Text wrapping and filling,re — Regular expression operations,Text Processing Services
@anchor{library/difflib doc}@anchor{1c0f}@anchor{library/difflib difflib-helpers-for-computing-deltas}@anchor{1c10}@anchor{library/difflib module-difflib}@anchor{35}
@subsection @code{difflib} — Helpers for computing deltas


@geindex module; difflib

@c Markup by Fred L. Drake, Jr. <fdrake@acm.org>

@strong{Source code:} Lib/difflib.py@footnote{https://github.com/python/cpython/tree/main/Lib/difflib.py}

@c import sys
@c from difflib import *


__________________________________________________________________


This module provides classes and functions for comparing sequences. It
can be used for example, for comparing files, and can produce information
about file differences in various formats, including HTML and context and unified
diffs. For comparing directories and files, see also, the @ref{7a,,filecmp} module.


@deffn {Class} difflib.SequenceMatcher

This is a flexible class for comparing pairs of sequences of any type, so long
as the sequence elements are @ref{2dc,,hashable}.  The basic algorithm predates, and is a
little fancier than, an algorithm published in the late 1980’s by Ratcliff and
Obershelp under the hyperbolic name “gestalt pattern matching.”  The idea is to
find the longest contiguous matching subsequence that contains no “junk”
elements; these “junk” elements are ones that are uninteresting in some
sense, such as blank lines or whitespace.  (Handling junk is an
extension to the Ratcliff and Obershelp algorithm.) The same
idea is then applied recursively to the pieces of the sequences to the left and
to the right of the matching subsequence.  This does not yield minimal edit
sequences, but does tend to yield matches that “look right” to people.

@strong{Timing:} The basic Ratcliff-Obershelp algorithm is cubic time in the worst
case and quadratic time in the expected case. @ref{ba8,,SequenceMatcher} is
quadratic time for the worst case and has expected-case behavior dependent in a
complicated way on how many elements the sequences have in common; best case
time is linear.

@strong{Automatic junk heuristic:} @ref{ba8,,SequenceMatcher} supports a heuristic that
automatically treats certain sequence items as junk. The heuristic counts how many
times each individual item appears in the sequence. If an item’s duplicates (after
the first one) account for more than 1% of the sequence and the sequence is at least
200 items long, this item is marked as “popular” and is treated as junk for
the purpose of sequence matching. This heuristic can be turned off by setting
the @code{autojunk} argument to @code{False} when creating the @ref{ba8,,SequenceMatcher}.


New in version 3.2: The @emph{autojunk} parameter.

@end deffn

@geindex Differ (class in difflib)
@anchor{library/difflib difflib Differ}@anchor{1c11}
@deffn {Class} difflib.Differ

This is a class for comparing sequences of lines of text, and producing
human-readable differences or deltas.  Differ uses @ref{ba8,,SequenceMatcher}
both to compare sequences of lines, and to compare sequences of characters
within similar (near-matching) lines.

Each line of a @ref{1c11,,Differ} delta begins with a two-letter code:


@multitable {xxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Code

@tab

Meaning

@item

@code{'- '}

@tab

line unique to sequence 1

@item

@code{'+ '}

@tab

line unique to sequence 2

@item

@code{'  '}

@tab

line common to both sequences

@item

@code{'? '}

@tab

line not present in either input sequence

@end multitable


Lines beginning with ‘@code{?}’ attempt to guide the eye to intraline differences,
and were not present in either input sequence. These lines can be confusing if
the sequences contain tab characters.
@end deffn

@geindex HtmlDiff (class in difflib)
@anchor{library/difflib difflib HtmlDiff}@anchor{1c12}
@deffn {Class} difflib.HtmlDiff

This class can be used to create an HTML table (or a complete HTML file
containing the table) showing a side by side, line by line comparison of text
with inter-line and intra-line change highlights.  The table can be generated in
either full or contextual difference mode.

The constructor for this class is:

@geindex __init__() (difflib.HtmlDiff method)
@anchor{library/difflib difflib HtmlDiff __init__}@anchor{1c13}
@deffn {Method} __init__ (tabsize=8, wrapcolumn=None, linejunk=None, charjunk=IS_CHARACTER_JUNK)

Initializes instance of @ref{1c12,,HtmlDiff}.

@emph{tabsize} is an optional keyword argument to specify tab stop spacing and
defaults to @code{8}.

@emph{wrapcolumn} is an optional keyword to specify column number where lines are
broken and wrapped, defaults to @code{None} where lines are not wrapped.

@emph{linejunk} and @emph{charjunk} are optional keyword arguments passed into @ref{1c14,,ndiff()}
(used by @ref{1c12,,HtmlDiff} to generate the side by side HTML differences).  See
@ref{1c14,,ndiff()} documentation for argument default values and descriptions.
@end deffn

The following methods are public:

@geindex make_file() (difflib.HtmlDiff method)
@anchor{library/difflib difflib HtmlDiff make_file}@anchor{94e}
@deffn {Method} make_file (fromlines, tolines, fromdesc='@w{'}, todesc='@w{'}, context=False, numlines=5, *, charset='utf-8')

Compares @emph{fromlines} and @emph{tolines} (lists of strings) and returns a string which
is a complete HTML file containing a table showing line by line differences with
inter-line and intra-line changes highlighted.

@emph{fromdesc} and @emph{todesc} are optional keyword arguments to specify from/to file
column header strings (both default to an empty string).

@emph{context} and @emph{numlines} are both optional keyword arguments. Set @emph{context} to
@code{True} when contextual differences are to be shown, else the default is
@code{False} to show the full files. @emph{numlines} defaults to @code{5}.  When @emph{context}
is @code{True} @emph{numlines} controls the number of context lines which surround the
difference highlights.  When @emph{context} is @code{False} @emph{numlines} controls the
number of lines which are shown before a difference highlight when using the
“next” hyperlinks (setting to zero would cause the “next” hyperlinks to place
the next difference highlight at the top of the browser without any leading
context).

@cartouche
@quotation Note 
@emph{fromdesc} and @emph{todesc} are interpreted as unescaped HTML and should be
properly escaped while receiving input from untrusted sources.
@end quotation
@end cartouche


Changed in version 3.5: @emph{charset} keyword-only argument was added.  The default charset of
HTML document changed from @code{'ISO-8859-1'} to @code{'utf-8'}.

@end deffn

@geindex make_table() (difflib.HtmlDiff method)
@anchor{library/difflib difflib HtmlDiff make_table}@anchor{1c15}
@deffn {Method} make_table (fromlines, tolines, fromdesc='@w{'}, todesc='@w{'}, context=False, numlines=5)

Compares @emph{fromlines} and @emph{tolines} (lists of strings) and returns a string which
is a complete HTML table showing line by line differences with inter-line and
intra-line changes highlighted.

The arguments for this method are the same as those for the @ref{94e,,make_file()}
method.
@end deffn

@code{Tools/scripts/diff.py} is a command-line front-end to this class and
contains a good example of its use.
@end deffn

@geindex context_diff() (in module difflib)
@anchor{library/difflib difflib context_diff}@anchor{1c16}
@deffn {Function} difflib.context_diff (a, b, fromfile='@w{'}, tofile='@w{'}, fromfiledate='@w{'}, tofiledate='@w{'}, n=3, lineterm='\n')

Compare @emph{a} and @emph{b} (lists of strings); return a delta (a @ref{bf7,,generator}
generating the delta lines) in context diff format.

Context diffs are a compact way of showing just the lines that have changed plus
a few lines of context.  The changes are shown in a before/after style.  The
number of context lines is set by @emph{n} which defaults to three.

By default, the diff control lines (those with @code{***} or @code{---}) are created
with a trailing newline.  This is helpful so that inputs created from
@ref{1c17,,io.IOBase.readlines()} result in diffs that are suitable for use with
@ref{1c18,,io.IOBase.writelines()} since both the inputs and outputs have trailing
newlines.

For inputs that do not have trailing newlines, set the @emph{lineterm} argument to
@code{""} so that the output will be uniformly newline free.

The context diff format normally has a header for filenames and modification
times.  Any or all of these may be specified using strings for @emph{fromfile},
@emph{tofile}, @emph{fromfiledate}, and @emph{tofiledate}.  The modification times are normally
expressed in the ISO 8601 format. If not specified, the
strings default to blanks.

@example
>>> s1 = ['bacon\n', 'eggs\n', 'ham\n', 'guido\n']
>>> s2 = ['python\n', 'eggy\n', 'hamster\n', 'guido\n']
>>> sys.stdout.writelines(context_diff(s1, s2, fromfile='before.py', tofile='after.py'))
*** before.py
--- after.py
***************
*** 1,4 ****
! bacon
! eggs
! ham
  guido
--- 1,4 ----
! python
! eggy
! hamster
  guido
@end example

See @ref{1c19,,A command-line interface to difflib} for a more detailed example.
@end deffn

@geindex get_close_matches() (in module difflib)
@anchor{library/difflib difflib get_close_matches}@anchor{1c1a}
@deffn {Function} difflib.get_close_matches (word, possibilities, n=3, cutoff=0.6)

Return a list of the best “good enough” matches.  @emph{word} is a sequence for which
close matches are desired (typically a string), and @emph{possibilities} is a list of
sequences against which to match @emph{word} (typically a list of strings).

Optional argument @emph{n} (default @code{3}) is the maximum number of close matches to
return; @emph{n} must be greater than @code{0}.

Optional argument @emph{cutoff} (default @code{0.6}) is a float in the range [0, 1].
Possibilities that don’t score at least that similar to @emph{word} are ignored.

The best (no more than @emph{n}) matches among the possibilities are returned in a
list, sorted by similarity score, most similar first.

@example
>>> get_close_matches('appel', ['ape', 'apple', 'peach', 'puppy'])
['apple', 'ape']
>>> import keyword
>>> get_close_matches('wheel', keyword.kwlist)
['while']
>>> get_close_matches('pineapple', keyword.kwlist)
[]
>>> get_close_matches('accept', keyword.kwlist)
['except']
@end example
@end deffn

@geindex ndiff() (in module difflib)
@anchor{library/difflib difflib ndiff}@anchor{1c14}
@deffn {Function} difflib.ndiff (a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK)

Compare @emph{a} and @emph{b} (lists of strings); return a @ref{1c11,,Differ}-style
delta (a @ref{bf7,,generator} generating the delta lines).

Optional keyword parameters @emph{linejunk} and @emph{charjunk} are filtering functions
(or @code{None}):

@emph{linejunk}: A function that accepts a single string argument, and returns
true if the string is junk, or false if not. The default is @code{None}. There
is also a module-level function @ref{1c1b,,IS_LINE_JUNK()}, which filters out lines
without visible characters, except for at most one pound character (@code{'#'})
– however the underlying @ref{ba8,,SequenceMatcher} class does a dynamic
analysis of which lines are so frequent as to constitute noise, and this
usually works better than using this function.

@emph{charjunk}: A function that accepts a character (a string of length 1), and
returns if the character is junk, or false if not. The default is module-level
function @ref{1c1c,,IS_CHARACTER_JUNK()}, which filters out whitespace characters (a
blank or tab; it’s a bad idea to include newline in this!).

@code{Tools/scripts/ndiff.py} is a command-line front-end to this function.

@example
>>> diff = ndiff('one\ntwo\nthree\n'.splitlines(keepends=True),
...              'ore\ntree\nemu\n'.splitlines(keepends=True))
>>> print('@w{'}.join(diff), end="")
- one
?  ^
+ ore
?  ^
- two
- three
?  -
+ tree
+ emu
@end example
@end deffn

@geindex restore() (in module difflib)
@anchor{library/difflib difflib restore}@anchor{1c1d}
@deffn {Function} difflib.restore (sequence, which)

Return one of the two sequences that generated a delta.

Given a @emph{sequence} produced by @ref{1c1e,,Differ.compare()} or @ref{1c14,,ndiff()}, extract
lines originating from file 1 or 2 (parameter @emph{which}), stripping off line
prefixes.

Example:

@example
>>> diff = ndiff('one\ntwo\nthree\n'.splitlines(keepends=True),
...              'ore\ntree\nemu\n'.splitlines(keepends=True))
>>> diff = list(diff) # materialize the generated delta into a list
>>> print('@w{'}.join(restore(diff, 1)), end="")
one
two
three
>>> print('@w{'}.join(restore(diff, 2)), end="")
ore
tree
emu
@end example
@end deffn

@geindex unified_diff() (in module difflib)
@anchor{library/difflib difflib unified_diff}@anchor{1c1f}
@deffn {Function} difflib.unified_diff (a, b, fromfile='@w{'}, tofile='@w{'}, fromfiledate='@w{'}, tofiledate='@w{'}, n=3, lineterm='\n')

Compare @emph{a} and @emph{b} (lists of strings); return a delta (a @ref{bf7,,generator}
generating the delta lines) in unified diff format.

Unified diffs are a compact way of showing just the lines that have changed plus
a few lines of context.  The changes are shown in an inline style (instead of
separate before/after blocks).  The number of context lines is set by @emph{n} which
defaults to three.

By default, the diff control lines (those with @code{---}, @code{+++}, or @code{@@@@}) are
created with a trailing newline.  This is helpful so that inputs created from
@ref{1c17,,io.IOBase.readlines()} result in diffs that are suitable for use with
@ref{1c18,,io.IOBase.writelines()} since both the inputs and outputs have trailing
newlines.

For inputs that do not have trailing newlines, set the @emph{lineterm} argument to
@code{""} so that the output will be uniformly newline free.

The context diff format normally has a header for filenames and modification
times.  Any or all of these may be specified using strings for @emph{fromfile},
@emph{tofile}, @emph{fromfiledate}, and @emph{tofiledate}.  The modification times are normally
expressed in the ISO 8601 format. If not specified, the
strings default to blanks.

@example
>>> s1 = ['bacon\n', 'eggs\n', 'ham\n', 'guido\n']
>>> s2 = ['python\n', 'eggy\n', 'hamster\n', 'guido\n']
>>> sys.stdout.writelines(unified_diff(s1, s2, fromfile='before.py', tofile='after.py'))
--- before.py
+++ after.py
@@@@ -1,4 +1,4 @@@@
-bacon
-eggs
-ham
+python
+eggy
+hamster
 guido
@end example

See @ref{1c19,,A command-line interface to difflib} for a more detailed example.
@end deffn

@geindex diff_bytes() (in module difflib)
@anchor{library/difflib difflib diff_bytes}@anchor{94f}
@deffn {Function} difflib.diff_bytes (dfunc, a, b, fromfile=b'@w{'}, tofile=b'@w{'}, fromfiledate=b'@w{'}, tofiledate=b'@w{'}, n=3, lineterm=b'\n')

Compare @emph{a} and @emph{b} (lists of bytes objects) using @emph{dfunc}; yield a
sequence of delta lines (also bytes) in the format returned by @emph{dfunc}.
@emph{dfunc} must be a callable, typically either @ref{1c1f,,unified_diff()} or
@ref{1c16,,context_diff()}.

Allows you to compare data with unknown or inconsistent encoding. All
inputs except @emph{n} must be bytes objects, not str. Works by losslessly
converting all inputs (except @emph{n}) to str, and calling @code{dfunc(a, b,
fromfile, tofile, fromfiledate, tofiledate, n, lineterm)}. The output of
@emph{dfunc} is then converted back to bytes, so the delta lines that you
receive have the same unknown/inconsistent encodings as @emph{a} and @emph{b}.


New in version 3.5.

@end deffn

@geindex IS_LINE_JUNK() (in module difflib)
@anchor{library/difflib difflib IS_LINE_JUNK}@anchor{1c1b}
@deffn {Function} difflib.IS_LINE_JUNK (line)

Return @code{True} for ignorable lines.  The line @emph{line} is ignorable if @emph{line} is
blank or contains a single @code{'#'}, otherwise it is not ignorable.  Used as a
default for parameter @emph{linejunk} in @ref{1c14,,ndiff()} in older versions.
@end deffn

@geindex IS_CHARACTER_JUNK() (in module difflib)
@anchor{library/difflib difflib IS_CHARACTER_JUNK}@anchor{1c1c}
@deffn {Function} difflib.IS_CHARACTER_JUNK (ch)

Return @code{True} for ignorable characters.  The character @emph{ch} is ignorable if @emph{ch}
is a space or tab, otherwise it is not ignorable.  Used as a default for
parameter @emph{charjunk} in @ref{1c14,,ndiff()}.
@end deffn


@subsubheading See also



@table @asis

@item Pattern Matching: The Gestalt Approach@footnote{http://www.drdobbs.com/database/pattern-matching-the-gestalt-approach/184407970}

Discussion of a similar algorithm by John W. Ratcliff and D. E. Metzener. This
was published in Dr. Dobb's Journal@footnote{http://www.drdobbs.com/} in July, 1988.
@end table


@menu
* SequenceMatcher Objects:: 
* SequenceMatcher Examples:: 
* Differ Objects:: 
* Differ Example:: 
* A command-line interface to difflib:: 

@end menu

@node SequenceMatcher Objects,SequenceMatcher Examples,,difflib — Helpers for computing deltas
@anchor{library/difflib sequence-matcher}@anchor{1c20}@anchor{library/difflib sequencematcher-objects}@anchor{1c21}
@subsubsection SequenceMatcher Objects


The @ref{ba8,,SequenceMatcher} class has this constructor:

@geindex SequenceMatcher (class in difflib)
@anchor{library/difflib difflib SequenceMatcher}@anchor{ba8}
@deffn {Class} difflib.SequenceMatcher (isjunk=None, a='@w{'}, b='@w{'}, autojunk=True)

Optional argument @emph{isjunk} must be @code{None} (the default) or a one-argument
function that takes a sequence element and returns true if and only if the
element is “junk” and should be ignored. Passing @code{None} for @emph{isjunk} is
equivalent to passing @code{lambda x: False}; in other words, no elements are ignored.
For example, pass:

@example
lambda x: x in " \t"
@end example

if you’re comparing lines as sequences of characters, and don’t want to synch up
on blanks or hard tabs.

The optional arguments @emph{a} and @emph{b} are sequences to be compared; both default to
empty strings.  The elements of both sequences must be @ref{2dc,,hashable}.

The optional argument @emph{autojunk} can be used to disable the automatic junk
heuristic.


New in version 3.2: The @emph{autojunk} parameter.


SequenceMatcher objects get three data attributes: @emph{bjunk} is the
set of elements of @emph{b} for which @emph{isjunk} is @code{True}; @emph{bpopular} is the set of
non-junk elements considered popular by the heuristic (if it is not
disabled); @emph{b2j} is a dict mapping the remaining elements of @emph{b} to a list
of positions where they occur. All three are reset whenever @emph{b} is reset
with @ref{1c22,,set_seqs()} or @ref{1c23,,set_seq2()}.


New in version 3.2: The @emph{bjunk} and @emph{bpopular} attributes.


@ref{ba8,,SequenceMatcher} objects have the following methods:

@geindex set_seqs() (difflib.SequenceMatcher method)
@anchor{library/difflib difflib SequenceMatcher set_seqs}@anchor{1c22}
@deffn {Method} set_seqs (a, b)

Set the two sequences to be compared.
@end deffn

@ref{ba8,,SequenceMatcher} computes and caches detailed information about the
second sequence, so if you want to compare one sequence against many
sequences, use @ref{1c23,,set_seq2()} to set the commonly used sequence once and
call @ref{1c24,,set_seq1()} repeatedly, once for each of the other sequences.

@geindex set_seq1() (difflib.SequenceMatcher method)
@anchor{library/difflib difflib SequenceMatcher set_seq1}@anchor{1c24}
@deffn {Method} set_seq1 (a)

Set the first sequence to be compared.  The second sequence to be compared
is not changed.
@end deffn

@geindex set_seq2() (difflib.SequenceMatcher method)
@anchor{library/difflib difflib SequenceMatcher set_seq2}@anchor{1c23}
@deffn {Method} set_seq2 (b)

Set the second sequence to be compared.  The first sequence to be compared
is not changed.
@end deffn

@geindex find_longest_match() (difflib.SequenceMatcher method)
@anchor{library/difflib difflib SequenceMatcher find_longest_match}@anchor{11e8}
@deffn {Method} find_longest_match (alo=0, ahi=None, blo=0, bhi=None)

Find longest matching block in @code{a[alo:ahi]} and @code{b[blo:bhi]}.

If @emph{isjunk} was omitted or @code{None}, @ref{11e8,,find_longest_match()} returns
@code{(i, j, k)} such that @code{a[i:i+k]} is equal to @code{b[j:j+k]}, where @code{alo
<= i <= i+k <= ahi} and @code{blo <= j <= j+k <= bhi}. For all @code{(i', j',
k')} meeting those conditions, the additional conditions @code{k >= k'}, @code{i
<= i'}, and if @code{i == i'}, @code{j <= j'} are also met. In other words, of
all maximal matching blocks, return one that starts earliest in @emph{a}, and
of all those maximal matching blocks that start earliest in @emph{a}, return
the one that starts earliest in @emph{b}.

@example
>>> s = SequenceMatcher(None, " abcd", "abcd abcd")
>>> s.find_longest_match(0, 5, 0, 9)
Match(a=0, b=4, size=5)
@end example

If @emph{isjunk} was provided, first the longest matching block is determined
as above, but with the additional restriction that no junk element appears
in the block.  Then that block is extended as far as possible by matching
(only) junk elements on both sides. So the resulting block never matches
on junk except as identical junk happens to be adjacent to an interesting
match.

Here’s the same example as before, but considering blanks to be junk. That
prevents @code{' abcd'} from matching the @code{' abcd'} at the tail end of the
second sequence directly.  Instead only the @code{'abcd'} can match, and
matches the leftmost @code{'abcd'} in the second sequence:

@example
>>> s = SequenceMatcher(lambda x: x==" ", " abcd", "abcd abcd")
>>> s.find_longest_match(0, 5, 0, 9)
Match(a=1, b=0, size=4)
@end example

If no blocks match, this returns @code{(alo, blo, 0)}.

This method returns a @ref{ce9,,named tuple} @code{Match(a, b, size)}.


Changed in version 3.9: Added default arguments.

@end deffn

@geindex get_matching_blocks() (difflib.SequenceMatcher method)
@anchor{library/difflib difflib SequenceMatcher get_matching_blocks}@anchor{1c25}
@deffn {Method} get_matching_blocks ()

Return list of triples describing non-overlapping matching subsequences.
Each triple is of the form @code{(i, j, n)},
and means that @code{a[i:i+n] == b[j:j+n]}.  The
triples are monotonically increasing in @emph{i} and @emph{j}.

The last triple is a dummy, and has the value @code{(len(a), len(b), 0)}.  It
is the only triple with @code{n == 0}.  If @code{(i, j, n)} and @code{(i', j', n')}
are adjacent triples in the list, and the second is not the last triple in
the list, then @code{i+n < i'} or @code{j+n < j'}; in other words, adjacent
triples always describe non-adjacent equal blocks.

@c XXX Explain why a dummy is used!

@example
>>> s = SequenceMatcher(None, "abxcd", "abcd")
>>> s.get_matching_blocks()
[Match(a=0, b=0, size=2), Match(a=3, b=2, size=2), Match(a=5, b=4, size=0)]
@end example
@end deffn

@geindex get_opcodes() (difflib.SequenceMatcher method)
@anchor{library/difflib difflib SequenceMatcher get_opcodes}@anchor{1c26}
@deffn {Method} get_opcodes ()

Return list of 5-tuples describing how to turn @emph{a} into @emph{b}. Each tuple is
of the form @code{(tag, i1, i2, j1, j2)}.  The first tuple has @code{i1 == j1 ==
0}, and remaining tuples have @emph{i1} equal to the @emph{i2} from the preceding
tuple, and, likewise, @emph{j1} equal to the previous @emph{j2}.

The @emph{tag} values are strings, with these meanings:


@multitable {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Value

@tab

Meaning

@item

@code{'replace'}

@tab

@code{a[i1:i2]} should be replaced by
@code{b[j1:j2]}.

@item

@code{'delete'}

@tab

@code{a[i1:i2]} should be deleted.  Note that
@code{j1 == j2} in this case.

@item

@code{'insert'}

@tab

@code{b[j1:j2]} should be inserted at
@code{a[i1:i1]}. Note that @code{i1 == i2} in
this case.

@item

@code{'equal'}

@tab

@code{a[i1:i2] == b[j1:j2]} (the sub-sequences
are equal).

@end multitable


For example:

@example
>>> a = "qabxcd"
>>> b = "abycdf"
>>> s = SequenceMatcher(None, a, b)
>>> for tag, i1, i2, j1, j2 in s.get_opcodes():
...     print('@{:7@}   a[@{@}:@{@}] --> b[@{@}:@{@}] @{!r:>8@} --> @{!r@}'.format(
...         tag, i1, i2, j1, j2, a[i1:i2], b[j1:j2]))
delete    a[0:1] --> b[0:0]      'q' --> '@w{'}
equal     a[1:3] --> b[0:2]     'ab' --> 'ab'
replace   a[3:4] --> b[2:3]      'x' --> 'y'
equal     a[4:6] --> b[3:5]     'cd' --> 'cd'
insert    a[6:6] --> b[5:6]       '@w{'} --> 'f'
@end example
@end deffn

@geindex get_grouped_opcodes() (difflib.SequenceMatcher method)
@anchor{library/difflib difflib SequenceMatcher get_grouped_opcodes}@anchor{1c27}
@deffn {Method} get_grouped_opcodes (n=3)

Return a @ref{bf7,,generator} of groups with up to @emph{n} lines of context.

Starting with the groups returned by @ref{1c26,,get_opcodes()}, this method
splits out smaller change clusters and eliminates intervening ranges which
have no changes.

The groups are returned in the same format as @ref{1c26,,get_opcodes()}.
@end deffn

@geindex ratio() (difflib.SequenceMatcher method)
@anchor{library/difflib difflib SequenceMatcher ratio}@anchor{1c28}
@deffn {Method} ratio ()

Return a measure of the sequences’ similarity as a float in the range [0,
1].

Where T is the total number of elements in both sequences, and M is the
number of matches, this is 2.0*M / T. Note that this is @code{1.0} if the
sequences are identical, and @code{0.0} if they have nothing in common.

This is expensive to compute if @ref{1c25,,get_matching_blocks()} or
@ref{1c26,,get_opcodes()} hasn’t already been called, in which case you may want
to try @ref{1c29,,quick_ratio()} or @ref{1c2a,,real_quick_ratio()} first to get an
upper bound.

@cartouche
@quotation Note 
Caution: The result of a @ref{1c28,,ratio()} call may depend on the order of
the arguments. For instance:

@example
>>> SequenceMatcher(None, 'tide', 'diet').ratio()
0.25
>>> SequenceMatcher(None, 'diet', 'tide').ratio()
0.5
@end example
@end quotation
@end cartouche
@end deffn

@geindex quick_ratio() (difflib.SequenceMatcher method)
@anchor{library/difflib difflib SequenceMatcher quick_ratio}@anchor{1c29}
@deffn {Method} quick_ratio ()

Return an upper bound on @ref{1c28,,ratio()} relatively quickly.
@end deffn

@geindex real_quick_ratio() (difflib.SequenceMatcher method)
@anchor{library/difflib difflib SequenceMatcher real_quick_ratio}@anchor{1c2a}
@deffn {Method} real_quick_ratio ()

Return an upper bound on @ref{1c28,,ratio()} very quickly.
@end deffn
@end deffn

The three methods that return the ratio of matching to total characters can give
different results due to differing levels of approximation, although
@code{quick_ratio()} and @code{real_quick_ratio()} are always at least as large as
@code{ratio()}:

@example
>>> s = SequenceMatcher(None, "abcd", "bcde")
>>> s.ratio()
0.75
>>> s.quick_ratio()
0.75
>>> s.real_quick_ratio()
1.0
@end example

@node SequenceMatcher Examples,Differ Objects,SequenceMatcher Objects,difflib — Helpers for computing deltas
@anchor{library/difflib id1}@anchor{1c2b}@anchor{library/difflib sequencematcher-examples}@anchor{1c2c}
@subsubsection SequenceMatcher Examples


This example compares two strings, considering blanks to be “junk”:

@example
>>> s = SequenceMatcher(lambda x: x == " ",
...                     "private Thread currentThread;",
...                     "private volatile Thread currentThread;")
@end example

@code{ratio()} returns a float in [0, 1], measuring the similarity of the
sequences.  As a rule of thumb, a @code{ratio()} value over 0.6 means the
sequences are close matches:

@example
>>> print(round(s.ratio(), 3))
0.866
@end example

If you’re only interested in where the sequences match,
@code{get_matching_blocks()} is handy:

@example
>>> for block in s.get_matching_blocks():
...     print("a[%d] and b[%d] match for %d elements" % block)
a[0] and b[0] match for 8 elements
a[8] and b[17] match for 21 elements
a[29] and b[38] match for 0 elements
@end example

Note that the last tuple returned by @code{get_matching_blocks()} is always a
dummy, @code{(len(a), len(b), 0)}, and this is the only case in which the last
tuple element (number of elements matched) is @code{0}.

If you want to know how to change the first sequence into the second, use
@code{get_opcodes()}:

@example
>>> for opcode in s.get_opcodes():
...     print("%6s a[%d:%d] b[%d:%d]" % opcode)
 equal a[0:8] b[0:8]
insert a[8:8] b[8:17]
 equal a[8:29] b[17:38]
@end example


@subsubheading See also



@itemize *

@item 
The @ref{1c1a,,get_close_matches()} function in this module which shows how
simple code building on @ref{ba8,,SequenceMatcher} can be used to do useful
work.

@item 
Simple version control recipe@footnote{https://code.activestate.com/recipes/576729/} for a small application
built with @ref{ba8,,SequenceMatcher}.
@end itemize


@node Differ Objects,Differ Example,SequenceMatcher Examples,difflib — Helpers for computing deltas
@anchor{library/difflib differ-objects}@anchor{1c2d}@anchor{library/difflib id2}@anchor{1c2e}
@subsubsection Differ Objects


Note that @ref{1c11,,Differ}-generated deltas make no claim to be @strong{minimal}
diffs. To the contrary, minimal diffs are often counter-intuitive, because they
synch up anywhere possible, sometimes accidental matches 100 pages apart.
Restricting synch points to contiguous matches preserves some notion of
locality, at the occasional cost of producing a longer diff.

The @ref{1c11,,Differ} class has this constructor:


@deffn {Class} difflib.Differ (linejunk=None, charjunk=None)

Optional keyword parameters @emph{linejunk} and @emph{charjunk} are for filter functions
(or @code{None}):

@emph{linejunk}: A function that accepts a single string argument, and returns true
if the string is junk.  The default is @code{None}, meaning that no line is
considered junk.

@emph{charjunk}: A function that accepts a single character argument (a string of
length 1), and returns true if the character is junk. The default is @code{None},
meaning that no character is considered junk.

These junk-filtering functions speed up matching to find
differences and do not cause any differing lines or characters to
be ignored.  Read the description of the
@ref{11e8,,find_longest_match()} method’s @emph{isjunk}
parameter for an explanation.

@ref{1c11,,Differ} objects are used (deltas generated) via a single method:

@geindex compare() (difflib.Differ method)
@anchor{library/difflib difflib Differ compare}@anchor{1c1e}
@deffn {Method} compare (a, b)

Compare two sequences of lines, and generate the delta (a sequence of lines).

Each sequence must contain individual single-line strings ending with
newlines.  Such sequences can be obtained from the
@ref{1c17,,readlines()} method of file-like objects.  The delta
generated also consists of newline-terminated strings, ready to be
printed as-is via the @ref{1c18,,writelines()} method of a
file-like object.
@end deffn
@end deffn

@node Differ Example,A command-line interface to difflib,Differ Objects,difflib — Helpers for computing deltas
@anchor{library/difflib differ-example}@anchor{1c2f}@anchor{library/difflib differ-examples}@anchor{1c30}
@subsubsection Differ Example


This example compares two texts. First we set up the texts, sequences of
individual single-line strings ending with newlines (such sequences can also be
obtained from the @code{readlines()} method of file-like objects):

@example
>>> text1 = '@w{'}'  1. Beautiful is better than ugly.
...   2. Explicit is better than implicit.
...   3. Simple is better than complex.
...   4. Complex is better than complicated.
... '@w{'}'.splitlines(keepends=True)
>>> len(text1)
4
>>> text1[0][-1]
'\n'
>>> text2 = '@w{'}'  1. Beautiful is better than ugly.
...   3.   Simple is better than complex.
...   4. Complicated is better than complex.
...   5. Flat is better than nested.
... '@w{'}'.splitlines(keepends=True)
@end example

Next we instantiate a Differ object:

@example
>>> d = Differ()
@end example

Note that when instantiating a @ref{1c11,,Differ} object we may pass functions to
filter out line and character “junk.”  See the @ref{1c11,,Differ()} constructor for
details.

Finally, we compare the two:

@example
>>> result = list(d.compare(text1, text2))
@end example

@code{result} is a list of strings, so let’s pretty-print it:

@example
>>> from pprint import pprint
>>> pprint(result)
['    1. Beautiful is better than ugly.\n',
 '-   2. Explicit is better than implicit.\n',
 '-   3. Simple is better than complex.\n',
 '+   3.   Simple is better than complex.\n',
 '?     ++\n',
 '-   4. Complex is better than complicated.\n',
 '?            ^                     ---- ^\n',
 '+   4. Complicated is better than complex.\n',
 '?           ++++ ^                      ^\n',
 '+   5. Flat is better than nested.\n']
@end example

As a single multi-line string it looks like this:

@example
>>> import sys
>>> sys.stdout.writelines(result)
    1. Beautiful is better than ugly.
-   2. Explicit is better than implicit.
-   3. Simple is better than complex.
+   3.   Simple is better than complex.
?     ++
-   4. Complex is better than complicated.
?            ^                     ---- ^
+   4. Complicated is better than complex.
?           ++++ ^                      ^
+   5. Flat is better than nested.
@end example

@node A command-line interface to difflib,,Differ Example,difflib — Helpers for computing deltas
@anchor{library/difflib a-command-line-interface-to-difflib}@anchor{1c31}@anchor{library/difflib difflib-interface}@anchor{1c19}
@subsubsection A command-line interface to difflib


This example shows how to use difflib to create a @code{diff}-like utility.
It is also contained in the Python source distribution, as
@code{Tools/scripts/diff.py}.

@example
#!/usr/bin/env python3
""" Command line interface to difflib.py providing diffs in four formats:

* ndiff:    lists every line and highlights interline changes.
* context:  highlights clusters of changes in a before/after format.
* unified:  highlights clusters of changes in an inline format.
* html:     generates side by side comparison with change highlights.

"""

import sys, os, difflib, argparse
from datetime import datetime, timezone

def file_mtime(path):
    t = datetime.fromtimestamp(os.stat(path).st_mtime,
                               timezone.utc)
    return t.astimezone().isoformat()

def main():

    parser = argparse.ArgumentParser()
    parser.add_argument('-c', action='store_true', default=False,
                        help='Produce a context format diff (default)')
    parser.add_argument('-u', action='store_true', default=False,
                        help='Produce a unified format diff')
    parser.add_argument('-m', action='store_true', default=False,
                        help='Produce HTML side by side diff '
                             '(can use -c and -l in conjunction)')
    parser.add_argument('-n', action='store_true', default=False,
                        help='Produce a ndiff format diff')
    parser.add_argument('-l', '--lines', type=int, default=3,
                        help='Set number of context lines (default 3)')
    parser.add_argument('fromfile')
    parser.add_argument('tofile')
    options = parser.parse_args()

    n = options.lines
    fromfile = options.fromfile
    tofile = options.tofile

    fromdate = file_mtime(fromfile)
    todate = file_mtime(tofile)
    with open(fromfile) as ff:
        fromlines = ff.readlines()
    with open(tofile) as tf:
        tolines = tf.readlines()

    if options.u:
        diff = difflib.unified_diff(fromlines, tolines, fromfile, tofile, fromdate, todate, n=n)
    elif options.n:
        diff = difflib.ndiff(fromlines, tolines)
    elif options.m:
        diff = difflib.HtmlDiff().make_file(fromlines,tolines,fromfile,tofile,context=options.c,numlines=n)
    else:
        diff = difflib.context_diff(fromlines, tolines, fromfile, tofile, fromdate, todate, n=n)

    sys.stdout.writelines(diff)

if __name__ == '__main__':
    main()
@end example

@node textwrap — Text wrapping and filling,unicodedata — Unicode Database,difflib — Helpers for computing deltas,Text Processing Services
@anchor{library/textwrap doc}@anchor{1c32}@anchor{library/textwrap module-textwrap}@anchor{108}@anchor{library/textwrap textwrap-text-wrapping-and-filling}@anchor{1c33}
@subsection @code{textwrap} — Text wrapping and filling


@geindex module; textwrap

@strong{Source code:} Lib/textwrap.py@footnote{https://github.com/python/cpython/tree/main/Lib/textwrap.py}


__________________________________________________________________


The @ref{108,,textwrap} module provides some convenience functions,
as well as @ref{b52,,TextWrapper}, the class that does all the work.
If you’re just wrapping or filling one or two text strings, the convenience
functions should be good enough; otherwise, you should use an instance of
@ref{b52,,TextWrapper} for efficiency.

@geindex wrap() (in module textwrap)
@anchor{library/textwrap textwrap wrap}@anchor{111c}
@deffn {Function} textwrap.wrap (text, width=70, *, initial_indent='@w{'}, subsequent_indent='@w{'}, expand_tabs=True, replace_whitespace=True, fix_sentence_endings=False, break_long_words=True, drop_whitespace=True, break_on_hyphens=True, tabsize=8, max_lines=None)

Wraps the single paragraph in @emph{text} (a string) so every line is at most
@emph{width} characters long.  Returns a list of output lines, without final
newlines.

Optional keyword arguments correspond to the instance attributes of
@ref{b52,,TextWrapper}, documented below.

See the @ref{1c34,,TextWrapper.wrap()} method for additional details on how
@ref{111c,,wrap()} behaves.
@end deffn

@geindex fill() (in module textwrap)
@anchor{library/textwrap textwrap fill}@anchor{111d}
@deffn {Function} textwrap.fill (text, width=70, *, initial_indent='@w{'}, subsequent_indent='@w{'}, expand_tabs=True, replace_whitespace=True, fix_sentence_endings=False, break_long_words=True, drop_whitespace=True, break_on_hyphens=True, tabsize=8, max_lines=None)

Wraps the single paragraph in @emph{text}, and returns a single string containing the
wrapped paragraph.  @ref{111d,,fill()} is shorthand for

@example
"\n".join(wrap(text, ...))
@end example

In particular, @ref{111d,,fill()} accepts exactly the same keyword arguments as
@ref{111c,,wrap()}.
@end deffn

@geindex shorten() (in module textwrap)
@anchor{library/textwrap textwrap shorten}@anchor{b55}
@deffn {Function} textwrap.shorten (text, width, *, fix_sentence_endings=False, break_long_words=True, break_on_hyphens=True, placeholder=' [...]')

Collapse and truncate the given @emph{text} to fit in the given @emph{width}.

First the whitespace in @emph{text} is collapsed (all whitespace is replaced by
single spaces).  If the result fits in the @emph{width}, it is returned.
Otherwise, enough words are dropped from the end so that the remaining words
plus the @code{placeholder} fit within @code{width}:

@example
>>> textwrap.shorten("Hello  world!", width=12)
'Hello world!'
>>> textwrap.shorten("Hello  world!", width=11)
'Hello [...]'
>>> textwrap.shorten("Hello world", width=10, placeholder="...")
'Hello...'
@end example

Optional keyword arguments correspond to the instance attributes of
@ref{b52,,TextWrapper}, documented below.  Note that the whitespace is
collapsed before the text is passed to the @ref{b52,,TextWrapper} @ref{111d,,fill()}
function, so changing the value of @ref{1c35,,tabsize}, @ref{1c36,,expand_tabs},
@ref{1c37,,drop_whitespace}, and @ref{1c38,,replace_whitespace} will have no effect.


New in version 3.4.

@end deffn

@geindex dedent() (in module textwrap)
@anchor{library/textwrap textwrap dedent}@anchor{1c39}
@deffn {Function} textwrap.dedent (text)

Remove any common leading whitespace from every line in @emph{text}.

This can be used to make triple-quoted strings line up with the left edge of the
display, while still presenting them in the source code in indented form.

Note that tabs and spaces are both treated as whitespace, but they are not
equal: the lines @code{"  hello"} and @code{"\thello"} are considered to have no
common leading whitespace.

Lines containing only whitespace are ignored in the input and normalized to a
single newline character in the output.

For example:

@example
def test():
    # end first line with \ to avoid the empty line!
    s = '@w{'}'\
    hello
      world
    '@w{'}'
    print(repr(s))          # prints '    hello\n      world\n    '
    print(repr(dedent(s)))  # prints 'hello\n  world\n'
@end example
@end deffn

@geindex indent() (in module textwrap)
@anchor{library/textwrap textwrap indent}@anchor{cee}
@deffn {Function} textwrap.indent (text, prefix, predicate=None)

Add @emph{prefix} to the beginning of selected lines in @emph{text}.

Lines are separated by calling @code{text.splitlines(True)}.

By default, @emph{prefix} is added to all lines that do not consist
solely of whitespace (including any line endings).

For example:

@example
>>> s = 'hello\n\n \nworld'
>>> indent(s, '  ')
'  hello\n\n \n  world'
@end example

The optional @emph{predicate} argument can be used to control which lines
are indented. For example, it is easy to add @emph{prefix} to even empty
and whitespace-only lines:

@example
>>> print(indent(s, '+ ', lambda line: True))
+ hello
+
+
+ world
@end example


New in version 3.3.

@end deffn

@ref{111c,,wrap()}, @ref{111d,,fill()} and @ref{b55,,shorten()} work by creating a
@ref{b52,,TextWrapper} instance and calling a single method on it.  That
instance is not reused, so for applications that process many text
strings using @ref{111c,,wrap()} and/or @ref{111d,,fill()}, it may be more efficient to
create your own @ref{b52,,TextWrapper} object.

Text is preferably wrapped on whitespaces and right after the hyphens in
hyphenated words; only then will long words be broken if necessary, unless
@ref{1c3a,,TextWrapper.break_long_words} is set to false.

@geindex TextWrapper (class in textwrap)
@anchor{library/textwrap textwrap TextWrapper}@anchor{b52}
@deffn {Class} textwrap.TextWrapper (**kwargs)

The @ref{b52,,TextWrapper} constructor accepts a number of optional keyword
arguments.  Each keyword argument corresponds to an instance attribute, so
for example

@example
wrapper = TextWrapper(initial_indent="* ")
@end example

is the same as

@example
wrapper = TextWrapper()
wrapper.initial_indent = "* "
@end example

You can re-use the same @ref{b52,,TextWrapper} object many times, and you can
change any of its options through direct assignment to instance attributes
between uses.

The @ref{b52,,TextWrapper} instance attributes (and keyword arguments to the
constructor) are as follows:

@geindex width (textwrap.TextWrapper attribute)
@anchor{library/textwrap textwrap TextWrapper width}@anchor{1c3b}
@deffn {Attribute} width

(default: @code{70}) The maximum length of wrapped lines.  As long as there
are no individual words in the input text longer than @ref{1c3b,,width},
@ref{b52,,TextWrapper} guarantees that no output line will be longer than
@ref{1c3b,,width} characters.
@end deffn

@geindex expand_tabs (textwrap.TextWrapper attribute)
@anchor{library/textwrap textwrap TextWrapper expand_tabs}@anchor{1c36}
@deffn {Attribute} expand_tabs

(default: @code{True}) If true, then all tab characters in @emph{text} will be
expanded to spaces using the @code{expandtabs()} method of @emph{text}.
@end deffn

@geindex tabsize (textwrap.TextWrapper attribute)
@anchor{library/textwrap textwrap TextWrapper tabsize}@anchor{1c35}
@deffn {Attribute} tabsize

(default: @code{8}) If @ref{1c36,,expand_tabs} is true, then all tab characters
in @emph{text} will be expanded to zero or more spaces, depending on the
current column and the given tab size.


New in version 3.3.

@end deffn

@geindex replace_whitespace (textwrap.TextWrapper attribute)
@anchor{library/textwrap textwrap TextWrapper replace_whitespace}@anchor{1c38}
@deffn {Attribute} replace_whitespace

(default: @code{True}) If true, after tab expansion but before wrapping,
the @ref{111c,,wrap()} method will replace each whitespace character
with a single space.  The whitespace characters replaced are
as follows: tab, newline, vertical tab, formfeed, and carriage
return (@code{'\t\n\v\f\r'}).

@cartouche
@quotation Note 
If @ref{1c36,,expand_tabs} is false and @ref{1c38,,replace_whitespace} is true,
each tab character will be replaced by a single space, which is @emph{not}
the same as tab expansion.
@end quotation
@end cartouche

@cartouche
@quotation Note 
If @ref{1c38,,replace_whitespace} is false, newlines may appear in the
middle of a line and cause strange output. For this reason, text should
be split into paragraphs (using @ref{1ab1,,str.splitlines()} or similar)
which are wrapped separately.
@end quotation
@end cartouche
@end deffn

@geindex drop_whitespace (textwrap.TextWrapper attribute)
@anchor{library/textwrap textwrap TextWrapper drop_whitespace}@anchor{1c37}
@deffn {Attribute} drop_whitespace

(default: @code{True}) If true, whitespace at the beginning and ending of
every line (after wrapping but before indenting) is dropped.
Whitespace at the beginning of the paragraph, however, is not dropped
if non-whitespace follows it.  If whitespace being dropped takes up an
entire line, the whole line is dropped.
@end deffn

@geindex initial_indent (textwrap.TextWrapper attribute)
@anchor{library/textwrap textwrap TextWrapper initial_indent}@anchor{1c3c}
@deffn {Attribute} initial_indent

(default: @code{'@w{'}}) String that will be prepended to the first line of
wrapped output.  Counts towards the length of the first line.  The empty
string is not indented.
@end deffn

@geindex subsequent_indent (textwrap.TextWrapper attribute)
@anchor{library/textwrap textwrap TextWrapper subsequent_indent}@anchor{1c3d}
@deffn {Attribute} subsequent_indent

(default: @code{'@w{'}}) String that will be prepended to all lines of wrapped
output except the first.  Counts towards the length of each line except
the first.
@end deffn

@geindex fix_sentence_endings (textwrap.TextWrapper attribute)
@anchor{library/textwrap textwrap TextWrapper fix_sentence_endings}@anchor{1c3e}
@deffn {Attribute} fix_sentence_endings

(default: @code{False}) If true, @ref{b52,,TextWrapper} attempts to detect
sentence endings and ensure that sentences are always separated by exactly
two spaces.  This is generally desired for text in a monospaced font.
However, the sentence detection algorithm is imperfect: it assumes that a
sentence ending consists of a lowercase letter followed by one of @code{'.'},
@code{'!'}, or @code{'?'}, possibly followed by one of @code{'"'} or @code{"'"},
followed by a space.  One problem with this is algorithm is that it is
unable to detect the difference between “Dr.” in

@example
[...] Dr. Frankenstein's monster [...]
@end example

and “Spot.” in

@example
[...] See Spot. See Spot run [...]
@end example

@ref{1c3e,,fix_sentence_endings} is false by default.

Since the sentence detection algorithm relies on @code{string.lowercase} for
the definition of “lowercase letter”, and a convention of using two spaces
after a period to separate sentences on the same line, it is specific to
English-language texts.
@end deffn

@geindex break_long_words (textwrap.TextWrapper attribute)
@anchor{library/textwrap textwrap TextWrapper break_long_words}@anchor{1c3a}
@deffn {Attribute} break_long_words

(default: @code{True}) If true, then words longer than @ref{1c3b,,width} will be
broken in order to ensure that no lines are longer than @ref{1c3b,,width}.  If
it is false, long words will not be broken, and some lines may be longer
than @ref{1c3b,,width}.  (Long words will be put on a line by themselves, in
order to minimize the amount by which @ref{1c3b,,width} is exceeded.)
@end deffn

@geindex break_on_hyphens (textwrap.TextWrapper attribute)
@anchor{library/textwrap textwrap TextWrapper break_on_hyphens}@anchor{1c3f}
@deffn {Attribute} break_on_hyphens

(default: @code{True}) If true, wrapping will occur preferably on whitespaces
and right after hyphens in compound words, as it is customary in English.
If false, only whitespaces will be considered as potentially good places
for line breaks, but you need to set @ref{1c3a,,break_long_words} to false if
you want truly insecable words.  Default behaviour in previous versions
was to always allow breaking hyphenated words.
@end deffn

@geindex max_lines (textwrap.TextWrapper attribute)
@anchor{library/textwrap textwrap TextWrapper max_lines}@anchor{b53}
@deffn {Attribute} max_lines

(default: @code{None}) If not @code{None}, then the output will contain at most
@emph{max_lines} lines, with @emph{placeholder} appearing at the end of the output.


New in version 3.4.

@end deffn

@geindex ...; placeholder

@geindex placeholder (textwrap.TextWrapper attribute)
@anchor{library/textwrap textwrap TextWrapper placeholder}@anchor{b54}
@deffn {Attribute} placeholder

(default: @code{' [...]'}) String that will appear at the end of the output
text if it has been truncated.


New in version 3.4.

@end deffn

@ref{b52,,TextWrapper} also provides some public methods, analogous to the
module-level convenience functions:

@geindex wrap() (textwrap.TextWrapper method)
@anchor{library/textwrap textwrap TextWrapper wrap}@anchor{1c34}
@deffn {Method} wrap (text)

Wraps the single paragraph in @emph{text} (a string) so every line is at most
@ref{1c3b,,width} characters long.  All wrapping options are taken from
instance attributes of the @ref{b52,,TextWrapper} instance.  Returns a list
of output lines, without final newlines.  If the wrapped output has no
content, the returned list is empty.
@end deffn

@geindex fill() (textwrap.TextWrapper method)
@anchor{library/textwrap textwrap TextWrapper fill}@anchor{1c40}
@deffn {Method} fill (text)

Wraps the single paragraph in @emph{text}, and returns a single string
containing the wrapped paragraph.
@end deffn
@end deffn

@node unicodedata — Unicode Database,stringprep — Internet String Preparation,textwrap — Text wrapping and filling,Text Processing Services
@anchor{library/unicodedata doc}@anchor{1c41}@anchor{library/unicodedata module-unicodedata}@anchor{121}@anchor{library/unicodedata unicodedata-unicode-database}@anchor{1c42}
@subsection @code{unicodedata} — Unicode Database


@geindex module; unicodedata

@geindex Unicode
@geindex character
@geindex Unicode; database


__________________________________________________________________


This module provides access to the Unicode Character Database (UCD) which
defines character properties for all Unicode characters. The data contained in
this database is compiled from the UCD version 14.0.0@footnote{https://www.unicode.org/Public/14.0.0/ucd}.

The module uses the same names and symbols as defined by Unicode
Standard Annex #44, "Unicode Character Database"@footnote{https://www.unicode.org/reports/tr44/}.  It defines the
following functions:

@geindex lookup() (in module unicodedata)
@anchor{library/unicodedata unicodedata lookup}@anchor{c0e}
@deffn {Function} unicodedata.lookup (name)

Look up character by name.  If a character with the given name is found, return
the corresponding character.  If not found, @ref{5bb,,KeyError} is raised.


Changed in version 3.3: Support for name aliases @footnote{@w{(1)} 
@indicateurl{https://www.unicode.org/Public/14.0.0/ucd/NameAliases.txt}
} and named sequences @footnote{@w{(2)} 
@indicateurl{https://www.unicode.org/Public/14.0.0/ucd/NamedSequences.txt}
} has been added.

@end deffn

@geindex name() (in module unicodedata)
@anchor{library/unicodedata unicodedata name}@anchor{1c43}
@deffn {Function} unicodedata.name (chr[, default])

Returns the name assigned to the character @emph{chr} as a string. If no
name is defined, @emph{default} is returned, or, if not given, @ref{17c,,ValueError} is
raised.
@end deffn

@geindex decimal() (in module unicodedata)
@anchor{library/unicodedata unicodedata decimal}@anchor{1c44}
@deffn {Function} unicodedata.decimal (chr[, default])

Returns the decimal value assigned to the character @emph{chr} as integer.
If no such value is defined, @emph{default} is returned, or, if not given,
@ref{17c,,ValueError} is raised.
@end deffn

@geindex digit() (in module unicodedata)
@anchor{library/unicodedata unicodedata digit}@anchor{1c45}
@deffn {Function} unicodedata.digit (chr[, default])

Returns the digit value assigned to the character @emph{chr} as integer.
If no such value is defined, @emph{default} is returned, or, if not given,
@ref{17c,,ValueError} is raised.
@end deffn

@geindex numeric() (in module unicodedata)
@anchor{library/unicodedata unicodedata numeric}@anchor{e13}
@deffn {Function} unicodedata.numeric (chr[, default])

Returns the numeric value assigned to the character @emph{chr} as float.
If no such value is defined, @emph{default} is returned, or, if not given,
@ref{17c,,ValueError} is raised.
@end deffn

@geindex category() (in module unicodedata)
@anchor{library/unicodedata unicodedata category}@anchor{1c46}
@deffn {Function} unicodedata.category (chr)

Returns the general category assigned to the character @emph{chr} as
string.
@end deffn

@geindex bidirectional() (in module unicodedata)
@anchor{library/unicodedata unicodedata bidirectional}@anchor{1c47}
@deffn {Function} unicodedata.bidirectional (chr)

Returns the bidirectional class assigned to the character @emph{chr} as
string. If no such value is defined, an empty string is returned.
@end deffn

@geindex combining() (in module unicodedata)
@anchor{library/unicodedata unicodedata combining}@anchor{1c48}
@deffn {Function} unicodedata.combining (chr)

Returns the canonical combining class assigned to the character @emph{chr}
as integer. Returns @code{0} if no combining class is defined.
@end deffn

@geindex east_asian_width() (in module unicodedata)
@anchor{library/unicodedata unicodedata east_asian_width}@anchor{1c49}
@deffn {Function} unicodedata.east_asian_width (chr)

Returns the east asian width assigned to the character @emph{chr} as
string.
@end deffn

@geindex mirrored() (in module unicodedata)
@anchor{library/unicodedata unicodedata mirrored}@anchor{1c4a}
@deffn {Function} unicodedata.mirrored (chr)

Returns the mirrored property assigned to the character @emph{chr} as
integer. Returns @code{1} if the character has been identified as a “mirrored”
character in bidirectional text, @code{0} otherwise.
@end deffn

@geindex decomposition() (in module unicodedata)
@anchor{library/unicodedata unicodedata decomposition}@anchor{1c4b}
@deffn {Function} unicodedata.decomposition (chr)

Returns the character decomposition mapping assigned to the character
@emph{chr} as string. An empty string is returned in case no such mapping is
defined.
@end deffn

@geindex normalize() (in module unicodedata)
@anchor{library/unicodedata unicodedata normalize}@anchor{19a}
@deffn {Function} unicodedata.normalize (form, unistr)

Return the normal form @emph{form} for the Unicode string @emph{unistr}. Valid values for
@emph{form} are ‘NFC’, ‘NFKC’, ‘NFD’, and ‘NFKD’.

The Unicode standard defines various normalization forms of a Unicode string,
based on the definition of canonical equivalence and compatibility equivalence.
In Unicode, several characters can be expressed in various way. For example, the
character U+00C7 (LATIN CAPITAL LETTER C WITH CEDILLA) can also be expressed as
the sequence U+0043 (LATIN CAPITAL LETTER C) U+0327 (COMBINING CEDILLA).

For each character, there are two normal forms: normal form C and normal form D.
Normal form D (NFD) is also known as canonical decomposition, and translates
each character into its decomposed form. Normal form C (NFC) first applies a
canonical decomposition, then composes pre-combined characters again.

In addition to these two forms, there are two additional normal forms based on
compatibility equivalence. In Unicode, certain characters are supported which
normally would be unified with other characters. For example, U+2160 (ROMAN
NUMERAL ONE) is really the same thing as U+0049 (LATIN CAPITAL LETTER I).
However, it is supported in Unicode for compatibility with existing character
sets (e.g. gb2312).

The normal form KD (NFKD) will apply the compatibility decomposition, i.e.
replace all compatibility characters with their equivalents. The normal form KC
(NFKC) first applies the compatibility decomposition, followed by the canonical
composition.

Even if two unicode strings are normalized and look the same to
a human reader, if one has combining characters and the other
doesn’t, they may not compare equal.
@end deffn

@geindex is_normalized() (in module unicodedata)
@anchor{library/unicodedata unicodedata is_normalized}@anchor{55e}
@deffn {Function} unicodedata.is_normalized (form, unistr)

Return whether the Unicode string @emph{unistr} is in the normal form @emph{form}. Valid
values for @emph{form} are ‘NFC’, ‘NFKC’, ‘NFD’, and ‘NFKD’.


New in version 3.8.

@end deffn

In addition, the module exposes the following constant:

@geindex unidata_version (in module unicodedata)
@anchor{library/unicodedata unicodedata unidata_version}@anchor{1c4c}
@deffn {Data} unicodedata.unidata_version

The version of the Unicode database used in this module.
@end deffn

@geindex ucd_3_2_0 (in module unicodedata)
@anchor{library/unicodedata unicodedata ucd_3_2_0}@anchor{f96}
@deffn {Data} unicodedata.ucd_3_2_0

This is an object that has the same methods as the entire module, but uses the
Unicode database version 3.2 instead, for applications that require this
specific version of the Unicode database (such as IDNA).
@end deffn

Examples:

@example
>>> import unicodedata
>>> unicodedata.lookup('LEFT CURLY BRACKET')
'@{'
>>> unicodedata.name('/')
'SOLIDUS'
>>> unicodedata.decimal('9')
9
>>> unicodedata.decimal('a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: not a decimal
>>> unicodedata.category('A')  # 'L'etter, 'u'ppercase
'Lu'
>>> unicodedata.bidirectional('\u0660') # 'A'rabic, 'N'umber
'AN'
@end example

@node stringprep — Internet String Preparation,readline — GNU readline interface,unicodedata — Unicode Database,Text Processing Services
@anchor{library/stringprep doc}@anchor{1c4d}@anchor{library/stringprep module-stringprep}@anchor{f2}@anchor{library/stringprep stringprep-internet-string-preparation}@anchor{1c4e}
@subsection @code{stringprep} — Internet String Preparation


@geindex module; stringprep

@strong{Source code:} Lib/stringprep.py@footnote{https://github.com/python/cpython/tree/main/Lib/stringprep.py}


__________________________________________________________________


When identifying things (such as host names) in the internet, it is often
necessary to compare such identifications for “equality”. Exactly how this
comparison is executed may depend on the application domain, e.g. whether it
should be case-insensitive or not. It may be also necessary to restrict the
possible identifications, to allow only identifications consisting of
“printable” characters.

@geindex RFC; RFC 3454
RFC 3454@footnote{https://tools.ietf.org/html/rfc3454.html} defines a procedure for “preparing” Unicode strings in internet
protocols. Before passing strings onto the wire, they are processed with the
preparation procedure, after which they have a certain normalized form. The RFC
defines a set of tables, which can be combined into profiles. Each profile must
define which tables it uses, and what other optional parts of the @code{stringprep}
procedure are part of the profile. One example of a @code{stringprep} profile is
@code{nameprep}, which is used for internationalized domain names.

The module @ref{f2,,stringprep} only exposes the tables from 
@geindex RFC; RFC 3454
RFC 3454@footnote{https://tools.ietf.org/html/rfc3454.html}. As these
tables would be very large to represent them as dictionaries or lists, the
module uses the Unicode character database internally. The module source code
itself was generated using the @code{mkstringprep.py} utility.

As a result, these tables are exposed as functions, not as data structures.
There are two kinds of tables in the RFC: sets and mappings. For a set,
@ref{f2,,stringprep} provides the “characteristic function”, i.e. a function that
returns @code{True} if the parameter is part of the set. For mappings, it provides the
mapping function: given the key, it returns the associated value. Below is a
list of all functions available in the module.

@geindex in_table_a1() (in module stringprep)
@anchor{library/stringprep stringprep in_table_a1}@anchor{1c4f}
@deffn {Function} stringprep.in_table_a1 (code)

Determine whether @emph{code} is in tableA.1 (Unassigned code points in Unicode 3.2).
@end deffn

@geindex in_table_b1() (in module stringprep)
@anchor{library/stringprep stringprep in_table_b1}@anchor{1c50}
@deffn {Function} stringprep.in_table_b1 (code)

Determine whether @emph{code} is in tableB.1 (Commonly mapped to nothing).
@end deffn

@geindex map_table_b2() (in module stringprep)
@anchor{library/stringprep stringprep map_table_b2}@anchor{1c51}
@deffn {Function} stringprep.map_table_b2 (code)

Return the mapped value for @emph{code} according to tableB.2 (Mapping for
case-folding used with NFKC).
@end deffn

@geindex map_table_b3() (in module stringprep)
@anchor{library/stringprep stringprep map_table_b3}@anchor{1c52}
@deffn {Function} stringprep.map_table_b3 (code)

Return the mapped value for @emph{code} according to tableB.3 (Mapping for
case-folding used with no normalization).
@end deffn

@geindex in_table_c11() (in module stringprep)
@anchor{library/stringprep stringprep in_table_c11}@anchor{1c53}
@deffn {Function} stringprep.in_table_c11 (code)

Determine whether @emph{code} is in tableC.1.1  (ASCII space characters).
@end deffn

@geindex in_table_c12() (in module stringprep)
@anchor{library/stringprep stringprep in_table_c12}@anchor{1c54}
@deffn {Function} stringprep.in_table_c12 (code)

Determine whether @emph{code} is in tableC.1.2  (Non-ASCII space characters).
@end deffn

@geindex in_table_c11_c12() (in module stringprep)
@anchor{library/stringprep stringprep in_table_c11_c12}@anchor{1c55}
@deffn {Function} stringprep.in_table_c11_c12 (code)

Determine whether @emph{code} is in tableC.1  (Space characters, union of C.1.1 and
C.1.2).
@end deffn

@geindex in_table_c21() (in module stringprep)
@anchor{library/stringprep stringprep in_table_c21}@anchor{1c56}
@deffn {Function} stringprep.in_table_c21 (code)

Determine whether @emph{code} is in tableC.2.1  (ASCII control characters).
@end deffn

@geindex in_table_c22() (in module stringprep)
@anchor{library/stringprep stringprep in_table_c22}@anchor{1c57}
@deffn {Function} stringprep.in_table_c22 (code)

Determine whether @emph{code} is in tableC.2.2  (Non-ASCII control characters).
@end deffn

@geindex in_table_c21_c22() (in module stringprep)
@anchor{library/stringprep stringprep in_table_c21_c22}@anchor{1c58}
@deffn {Function} stringprep.in_table_c21_c22 (code)

Determine whether @emph{code} is in tableC.2  (Control characters, union of C.2.1 and
C.2.2).
@end deffn

@geindex in_table_c3() (in module stringprep)
@anchor{library/stringprep stringprep in_table_c3}@anchor{1c59}
@deffn {Function} stringprep.in_table_c3 (code)

Determine whether @emph{code} is in tableC.3  (Private use).
@end deffn

@geindex in_table_c4() (in module stringprep)
@anchor{library/stringprep stringprep in_table_c4}@anchor{1c5a}
@deffn {Function} stringprep.in_table_c4 (code)

Determine whether @emph{code} is in tableC.4  (Non-character code points).
@end deffn

@geindex in_table_c5() (in module stringprep)
@anchor{library/stringprep stringprep in_table_c5}@anchor{1c5b}
@deffn {Function} stringprep.in_table_c5 (code)

Determine whether @emph{code} is in tableC.5  (Surrogate codes).
@end deffn

@geindex in_table_c6() (in module stringprep)
@anchor{library/stringprep stringprep in_table_c6}@anchor{1c5c}
@deffn {Function} stringprep.in_table_c6 (code)

Determine whether @emph{code} is in tableC.6  (Inappropriate for plain text).
@end deffn

@geindex in_table_c7() (in module stringprep)
@anchor{library/stringprep stringprep in_table_c7}@anchor{1c5d}
@deffn {Function} stringprep.in_table_c7 (code)

Determine whether @emph{code} is in tableC.7  (Inappropriate for canonical
representation).
@end deffn

@geindex in_table_c8() (in module stringprep)
@anchor{library/stringprep stringprep in_table_c8}@anchor{1c5e}
@deffn {Function} stringprep.in_table_c8 (code)

Determine whether @emph{code} is in tableC.8  (Change display properties or are
deprecated).
@end deffn

@geindex in_table_c9() (in module stringprep)
@anchor{library/stringprep stringprep in_table_c9}@anchor{1c5f}
@deffn {Function} stringprep.in_table_c9 (code)

Determine whether @emph{code} is in tableC.9  (Tagging characters).
@end deffn

@geindex in_table_d1() (in module stringprep)
@anchor{library/stringprep stringprep in_table_d1}@anchor{1c60}
@deffn {Function} stringprep.in_table_d1 (code)

Determine whether @emph{code} is in tableD.1  (Characters with bidirectional property
“R” or “AL”).
@end deffn

@geindex in_table_d2() (in module stringprep)
@anchor{library/stringprep stringprep in_table_d2}@anchor{1c61}
@deffn {Function} stringprep.in_table_d2 (code)

Determine whether @emph{code} is in tableD.2  (Characters with bidirectional property
“L”).
@end deffn

@node readline — GNU readline interface,rlcompleter — Completion function for GNU readline,stringprep — Internet String Preparation,Text Processing Services
@anchor{library/readline doc}@anchor{1c62}@anchor{library/readline module-readline}@anchor{d9}@anchor{library/readline readline-gnu-readline-interface}@anchor{1c63}
@subsection @code{readline} — GNU readline interface


@geindex module; readline


__________________________________________________________________


The @ref{d9,,readline} module defines a number of functions to facilitate
completion and reading/writing of history files from the Python interpreter.
This module can be used directly, or via the @ref{dc,,rlcompleter} module, which
supports completion of Python identifiers at the interactive prompt.  Settings
made using  this module affect the behaviour of both the interpreter’s
interactive prompt  and the prompts offered by the built-in @ref{e73,,input()}
function.

Readline keybindings may be configured via an initialization file, typically
@code{.inputrc} in your home directory.  See Readline Init File@footnote{https://tiswww.cwru.edu/php/chet/readline/rluserman.html#SEC9}
in the GNU Readline manual for information about the format and
allowable constructs of that file, and the capabilities of the
Readline library in general.

@cartouche
@quotation Note 
The underlying Readline library API may be implemented by
the @code{libedit} library instead of GNU readline.
On macOS the @ref{d9,,readline} module detects which library is being used
at run time.

The configuration file for @code{libedit} is different from that
of GNU readline. If you programmatically load configuration strings
you can check for the text “libedit” in @code{readline.__doc__}
to differentiate between GNU readline and libedit.

If you use @emph{editline}/@code{libedit} readline emulation on macOS, the
initialization file located in your home directory is named
@code{.editrc}. For example, the following content in @code{~/.editrc} will
turn ON @emph{vi} keybindings and TAB completion:

@example
python:bind -v
python:bind ^I rl_complete
@end example
@end quotation
@end cartouche

@menu
* Init file:: 
* Line buffer:: 
* History file:: 
* History list:: 
* Startup hooks:: 
* Completion:: 
* Example:: 

@end menu

@node Init file,Line buffer,,readline — GNU readline interface
@anchor{library/readline init-file}@anchor{1c64}
@subsubsection Init file


The following functions relate to the init file and user configuration:

@geindex parse_and_bind() (in module readline)
@anchor{library/readline readline parse_and_bind}@anchor{1c65}
@deffn {Function} readline.parse_and_bind (string)

Execute the init line provided in the @emph{string} argument. This calls
@code{rl_parse_and_bind()} in the underlying library.
@end deffn

@geindex read_init_file() (in module readline)
@anchor{library/readline readline read_init_file}@anchor{1c66}
@deffn {Function} readline.read_init_file ([filename])

Execute a readline initialization file. The default filename is the last filename
used. This calls @code{rl_read_init_file()} in the underlying library.
@end deffn

@node Line buffer,History file,Init file,readline — GNU readline interface
@anchor{library/readline line-buffer}@anchor{1c67}
@subsubsection Line buffer


The following functions operate on the line buffer:

@geindex get_line_buffer() (in module readline)
@anchor{library/readline readline get_line_buffer}@anchor{1c68}
@deffn {Function} readline.get_line_buffer ()

Return the current contents of the line buffer (@code{rl_line_buffer}
in the underlying library).
@end deffn

@geindex insert_text() (in module readline)
@anchor{library/readline readline insert_text}@anchor{1c69}
@deffn {Function} readline.insert_text (string)

Insert text into the line buffer at the cursor position.  This calls
@code{rl_insert_text()} in the underlying library, but ignores
the return value.
@end deffn

@geindex redisplay() (in module readline)
@anchor{library/readline readline redisplay}@anchor{1c6a}
@deffn {Function} readline.redisplay ()

Change what’s displayed on the screen to reflect the current contents of the
line buffer.  This calls @code{rl_redisplay()} in the underlying library.
@end deffn

@node History file,History list,Line buffer,readline — GNU readline interface
@anchor{library/readline history-file}@anchor{1c6b}
@subsubsection History file


The following functions operate on a history file:

@geindex read_history_file() (in module readline)
@anchor{library/readline readline read_history_file}@anchor{1c6c}
@deffn {Function} readline.read_history_file ([filename])

Load a readline history file, and append it to the history list.
The default filename is @code{~/.history}.  This calls
@code{read_history()} in the underlying library.
@end deffn

@geindex write_history_file() (in module readline)
@anchor{library/readline readline write_history_file}@anchor{1c6d}
@deffn {Function} readline.write_history_file ([filename])

Save the history list to a readline history file, overwriting any
existing file.  The default filename is @code{~/.history}.  This calls
@code{write_history()} in the underlying library.
@end deffn

@geindex append_history_file() (in module readline)
@anchor{library/readline readline append_history_file}@anchor{9bb}
@deffn {Function} readline.append_history_file (nelements[, filename])

Append the last @emph{nelements} items of history to a file.  The default filename is
@code{~/.history}.  The file must already exist.  This calls
@code{append_history()} in the underlying library.  This function
only exists if Python was compiled for a version of the library
that supports it.


New in version 3.5.

@end deffn

@geindex get_history_length() (in module readline)
@geindex set_history_length() (in module readline)
@anchor{library/readline readline get_history_length}@anchor{1c6e}
@deffn {Function} readline.get_history_length ()
@anchor{library/readline readline set_history_length}@anchor{1c6f}
@deffnx {Function} readline.set_history_length (length)

Set or return the desired number of lines to save in the history file.
The @ref{1c6d,,write_history_file()} function uses this value to truncate
the history file, by calling @code{history_truncate_file()} in
the underlying library.  Negative values imply
unlimited history file size.
@end deffn

@node History list,Startup hooks,History file,readline — GNU readline interface
@anchor{library/readline history-list}@anchor{1c70}
@subsubsection History list


The following functions operate on a global history list:

@geindex clear_history() (in module readline)
@anchor{library/readline readline clear_history}@anchor{1c71}
@deffn {Function} readline.clear_history ()

Clear the current history.  This calls @code{clear_history()} in the
underlying library.  The Python function only exists if Python was
compiled for a version of the library that supports it.
@end deffn

@geindex get_current_history_length() (in module readline)
@anchor{library/readline readline get_current_history_length}@anchor{1c72}
@deffn {Function} readline.get_current_history_length ()

Return the number of items currently in the history.  (This is different from
@ref{1c6e,,get_history_length()}, which returns the maximum number of lines that will
be written to a history file.)
@end deffn

@geindex get_history_item() (in module readline)
@anchor{library/readline readline get_history_item}@anchor{1c73}
@deffn {Function} readline.get_history_item (index)

Return the current contents of history item at @emph{index}.  The item index
is one-based.  This calls @code{history_get()} in the underlying library.
@end deffn

@geindex remove_history_item() (in module readline)
@anchor{library/readline readline remove_history_item}@anchor{1c74}
@deffn {Function} readline.remove_history_item (pos)

Remove history item specified by its position from the history.
The position is zero-based.  This calls @code{remove_history()} in
the underlying library.
@end deffn

@geindex replace_history_item() (in module readline)
@anchor{library/readline readline replace_history_item}@anchor{1c75}
@deffn {Function} readline.replace_history_item (pos, line)

Replace history item specified by its position with @emph{line}.
The position is zero-based.  This calls @code{replace_history_entry()}
in the underlying library.
@end deffn

@geindex add_history() (in module readline)
@anchor{library/readline readline add_history}@anchor{1c76}
@deffn {Function} readline.add_history (line)

Append @emph{line} to the history buffer, as if it was the last line typed.
This calls @code{add_history()} in the underlying library.
@end deffn

@geindex set_auto_history() (in module readline)
@anchor{library/readline readline set_auto_history}@anchor{812}
@deffn {Function} readline.set_auto_history (enabled)

Enable or disable automatic calls to @code{add_history()} when reading
input via readline.  The @emph{enabled} argument should be a Boolean value
that when true, enables auto history, and that when false, disables
auto history.


New in version 3.6.


@strong{CPython implementation detail:} Auto history is enabled by default, and changes to this do not persist
across multiple sessions.
@end deffn

@node Startup hooks,Completion,History list,readline — GNU readline interface
@anchor{library/readline startup-hooks}@anchor{1c77}
@subsubsection Startup hooks


@geindex set_startup_hook() (in module readline)
@anchor{library/readline readline set_startup_hook}@anchor{1c78}
@deffn {Function} readline.set_startup_hook ([function])

Set or remove the function invoked by the @code{rl_startup_hook}
callback of the underlying library.  If @emph{function} is specified, it will
be used as the new hook function; if omitted or @code{None}, any function
already installed is removed.  The hook is called with no
arguments just before readline prints the first prompt.
@end deffn

@geindex set_pre_input_hook() (in module readline)
@anchor{library/readline readline set_pre_input_hook}@anchor{1c79}
@deffn {Function} readline.set_pre_input_hook ([function])

Set or remove the function invoked by the @code{rl_pre_input_hook}
callback of the underlying library.  If @emph{function} is specified, it will
be used as the new hook function; if omitted or @code{None}, any
function already installed is removed.  The hook is called
with no arguments after the first prompt has been printed and just before
readline starts reading input characters.  This function only exists
if Python was compiled for a version of the library that supports it.
@end deffn

@node Completion,Example,Startup hooks,readline — GNU readline interface
@anchor{library/readline completion}@anchor{1c7a}
@subsubsection Completion


The following functions relate to implementing a custom word completion
function.  This is typically operated by the Tab key, and can suggest and
automatically complete a word being typed.  By default, Readline is set up
to be used by @ref{dc,,rlcompleter} to complete Python identifiers for
the interactive interpreter.  If the @ref{d9,,readline} module is to be used
with a custom completer, a different set of word delimiters should be set.

@geindex set_completer() (in module readline)
@anchor{library/readline readline set_completer}@anchor{1c7b}
@deffn {Function} readline.set_completer ([function])

Set or remove the completer function.  If @emph{function} is specified, it will be
used as the new completer function; if omitted or @code{None}, any completer
function already installed is removed.  The completer function is called as
@code{function(text, state)}, for @emph{state} in @code{0}, @code{1}, @code{2}, …, until it
returns a non-string value.  It should return the next possible completion
starting with @emph{text}.

The installed completer function is invoked by the @emph{entry_func} callback
passed to @code{rl_completion_matches()} in the underlying library.
The @emph{text} string comes from the first parameter to the
@code{rl_attempted_completion_function} callback of the
underlying library.
@end deffn

@geindex get_completer() (in module readline)
@anchor{library/readline readline get_completer}@anchor{1c7c}
@deffn {Function} readline.get_completer ()

Get the completer function, or @code{None} if no completer function has been set.
@end deffn

@geindex get_completion_type() (in module readline)
@anchor{library/readline readline get_completion_type}@anchor{1c7d}
@deffn {Function} readline.get_completion_type ()

Get the type of completion being attempted.  This returns the
@code{rl_completion_type} variable in the underlying library as
an integer.
@end deffn

@geindex get_begidx() (in module readline)
@geindex get_endidx() (in module readline)
@anchor{library/readline readline get_begidx}@anchor{1160}
@deffn {Function} readline.get_begidx ()
@anchor{library/readline readline get_endidx}@anchor{1161}
@deffnx {Function} readline.get_endidx ()

Get the beginning or ending index of the completion scope.
These indexes are the @emph{start} and @emph{end} arguments passed to the
@code{rl_attempted_completion_function} callback of the
underlying library.  The values may be different in the same
input editing scenario based on the underlying C readline implementation.
Ex: libedit is known to behave differently than libreadline.
@end deffn

@geindex set_completer_delims() (in module readline)
@geindex get_completer_delims() (in module readline)
@anchor{library/readline readline set_completer_delims}@anchor{1c7e}
@deffn {Function} readline.set_completer_delims (string)
@anchor{library/readline readline get_completer_delims}@anchor{1c7f}
@deffnx {Function} readline.get_completer_delims ()

Set or get the word delimiters for completion.  These determine the
start of the word to be considered for completion (the completion scope).
These functions access the @code{rl_completer_word_break_characters}
variable in the underlying library.
@end deffn

@geindex set_completion_display_matches_hook() (in module readline)
@anchor{library/readline readline set_completion_display_matches_hook}@anchor{1c80}
@deffn {Function} readline.set_completion_display_matches_hook ([function])

Set or remove the completion display function.  If @emph{function} is
specified, it will be used as the new completion display function;
if omitted or @code{None}, any completion display function already
installed is removed.  This sets or clears the
@code{rl_completion_display_matches_hook} callback in the
underlying library.  The completion display function is called as
@code{function(substitution, [matches], longest_match_length)} once
each time matches need to be displayed.
@end deffn

@node Example,,Completion,readline — GNU readline interface
@anchor{library/readline example}@anchor{1c81}@anchor{library/readline readline-example}@anchor{1c82}
@subsubsection Example


The following example demonstrates how to use the @ref{d9,,readline} module’s
history reading and writing functions to automatically load and save a history
file named @code{.python_history} from the user’s home directory.  The code
below would normally be executed automatically during interactive sessions
from the user’s 
@geindex PYTHONSTARTUP
@geindex environment variable; PYTHONSTARTUP
@ref{b4d,,PYTHONSTARTUP} file.

@example
import atexit
import os
import readline

histfile = os.path.join(os.path.expanduser("~"), ".python_history")
try:
    readline.read_history_file(histfile)
    # default history len is -1 (infinite), which may grow unruly
    readline.set_history_length(1000)
except FileNotFoundError:
    pass

atexit.register(readline.write_history_file, histfile)
@end example

This code is actually automatically run when Python is run in
@ref{b4b,,interactive mode} (see @ref{b4e,,Readline configuration}).

The following example achieves the same goal but supports concurrent interactive
sessions, by only appending the new history.

@example
import atexit
import os
import readline
histfile = os.path.join(os.path.expanduser("~"), ".python_history")

try:
    readline.read_history_file(histfile)
    h_len = readline.get_current_history_length()
except FileNotFoundError:
    open(histfile, 'wb').close()
    h_len = 0

def save(prev_h_len, histfile):
    new_h_len = readline.get_current_history_length()
    readline.set_history_length(1000)
    readline.append_history_file(new_h_len - prev_h_len, histfile)
atexit.register(save, h_len, histfile)
@end example

The following example extends the @ref{1c83,,code.InteractiveConsole} class to
support history save/restore.

@example
import atexit
import code
import os
import readline

class HistoryConsole(code.InteractiveConsole):
    def __init__(self, locals=None, filename="<console>",
                 histfile=os.path.expanduser("~/.console-history")):
        code.InteractiveConsole.__init__(self, locals, filename)
        self.init_history(histfile)

    def init_history(self, histfile):
        readline.parse_and_bind("tab: complete")
        if hasattr(readline, "read_history_file"):
            try:
                readline.read_history_file(histfile)
            except FileNotFoundError:
                pass
            atexit.register(self.save_history, histfile)

    def save_history(self, histfile):
        readline.set_history_length(1000)
        readline.write_history_file(histfile)
@end example

@node rlcompleter — Completion function for GNU readline,,readline — GNU readline interface,Text Processing Services
@anchor{library/rlcompleter doc}@anchor{1c84}@anchor{library/rlcompleter module-rlcompleter}@anchor{dc}@anchor{library/rlcompleter rlcompleter-completion-function-for-gnu-readline}@anchor{1c85}
@subsection @code{rlcompleter} — Completion function for GNU readline


@geindex module; rlcompleter

@strong{Source code:} Lib/rlcompleter.py@footnote{https://github.com/python/cpython/tree/main/Lib/rlcompleter.py}


__________________________________________________________________


The @ref{dc,,rlcompleter} module defines a completion function suitable for the
@ref{d9,,readline} module by completing valid Python identifiers and keywords.

When this module is imported on a Unix platform with the @ref{d9,,readline} module
available, an instance of the @code{Completer} class is automatically created
and its @code{complete()} method is set as the @ref{d9,,readline} completer.

Example:

@example
>>> import rlcompleter
>>> import readline
>>> readline.parse_and_bind("tab: complete")
>>> readline. <TAB PRESSED>
readline.__doc__          readline.get_line_buffer(  readline.read_init_file(
readline.__file__         readline.insert_text(      readline.set_completer(
readline.__name__         readline.parse_and_bind(
>>> readline.
@end example

The @ref{dc,,rlcompleter} module is designed for use with Python’s
@ref{b4b,,interactive mode}.  Unless Python is run with the
@ref{d4e,,-S} option, the module is automatically imported and configured
(see @ref{b4e,,Readline configuration}).

On platforms without @ref{d9,,readline}, the @code{Completer} class defined by
this module can still be used for custom purposes.

@menu
* Completer Objects:: 

@end menu

@node Completer Objects,,,rlcompleter — Completion function for GNU readline
@anchor{library/rlcompleter completer-objects}@anchor{1c86}@anchor{library/rlcompleter id1}@anchor{1c87}
@subsubsection Completer Objects


Completer objects have the following method:

@geindex complete() (rlcompleter.Completer method)
@anchor{library/rlcompleter rlcompleter Completer complete}@anchor{1c88}
@deffn {Method} Completer.complete (text, state)

Return the @emph{state}th completion for @emph{text}.

If called for @emph{text} that doesn’t include a period character (@code{'.'}), it will
complete from names currently defined in @ref{1,,__main__}, @ref{11,,builtins} and
keywords (as defined by the @ref{a2,,keyword} module).

If called for a dotted name, it will try to evaluate anything without obvious
side-effects (functions will not be evaluated, but it can generate calls to
@code{__getattr__()}) up to the last part, and find matches for the rest via the
@ref{f2f,,dir()} function.  Any exception raised during the evaluation of the
expression is caught, silenced and @ref{182,,None} is returned.
@end deffn

@node Binary Data Services,Data Types,Text Processing Services,The Python Standard Library
@anchor{library/binary doc}@anchor{1c89}@anchor{library/binary binary-data-services}@anchor{1c8a}@anchor{library/binary binaryservices}@anchor{1b98}
@section Binary Data Services


The modules described in this chapter provide some basic services operations
for manipulation of binary data. Other operations on binary data, specifically
in relation to file formats and network protocols, are described in the
relevant sections.

Some libraries described under @ref{1a9a,,Text Processing Services} also work with either
ASCII-compatible binary formats (for example, @ref{d8,,re}) or all binary data
(for example, @ref{35,,difflib}).

In addition, see the documentation for Python’s built-in binary data types in
@ref{1a3f,,Binary Sequence Types — bytes@comma{} bytearray@comma{} memoryview}.

@menu
* struct — Interpret bytes as packed binary data:: 
* codecs — Codec registry and base classes:: 

@end menu

@node struct — Interpret bytes as packed binary data,codecs — Codec registry and base classes,,Binary Data Services
@anchor{library/struct doc}@anchor{1c8b}@anchor{library/struct module-struct}@anchor{f3}@anchor{library/struct struct-interpret-bytes-as-packed-binary-data}@anchor{1c8c}
@subsection @code{struct} — Interpret bytes as packed binary data


@geindex module; struct

@strong{Source code:} Lib/struct.py@footnote{https://github.com/python/cpython/tree/main/Lib/struct.py}

@geindex C; structures
@geindex packing; binary; data


__________________________________________________________________


This module performs conversions between Python values and C structs represented
as Python @ref{2f0,,bytes} objects.  This can be used in handling binary data
stored in files or from network connections, among other sources.  It uses
@ref{1c8d,,Format Strings} as compact descriptions of the layout of the C
structs and the intended conversion to/from Python values.

@cartouche
@quotation Note 
By default, the result of packing a given C struct includes pad bytes in
order to maintain proper alignment for the C types involved; similarly,
alignment is taken into account when unpacking.  This behavior is chosen so
that the bytes of a packed struct correspond exactly to the layout in memory
of the corresponding C struct.  To handle platform-independent data formats
or omit implicit pad bytes, use @code{standard} size and alignment instead of
@code{native} size and alignment: see @ref{1c8e,,Byte Order@comma{} Size@comma{} and Alignment} for details.
@end quotation
@end cartouche

Several @ref{f3,,struct} functions (and methods of @ref{1c8f,,Struct}) take a @emph{buffer}
argument.  This refers to objects that implement the @ref{1a3e,,Buffer Protocol} and
provide either a readable or read-writable buffer.  The most common types used
for that purpose are @ref{2f0,,bytes} and @ref{2f1,,bytearray}, but many other types
that can be viewed as an array of bytes implement the buffer protocol, so that
they can be read/filled without additional copying from a @ref{2f0,,bytes} object.

@menu
* Functions and Exceptions:: 
* Format Strings:: 
* Classes: Classes<2>. 

@end menu

@node Functions and Exceptions,Format Strings,,struct — Interpret bytes as packed binary data
@anchor{library/struct functions-and-exceptions}@anchor{1c90}
@subsubsection Functions and Exceptions


The module defines the following exception and functions:

@geindex error
@anchor{library/struct struct error}@anchor{eb6}
@deffn {Exception} struct.error

Exception raised on various occasions; argument is a string describing what
is wrong.
@end deffn

@geindex pack() (in module struct)
@anchor{library/struct struct pack}@anchor{e25}
@deffn {Function} struct.pack (format, v1, v2, ...)

Return a bytes object containing the values @emph{v1}, @emph{v2}, … packed according
to the format string @emph{format}.  The arguments must match the values required by
the format exactly.
@end deffn

@geindex pack_into() (in module struct)
@anchor{library/struct struct pack_into}@anchor{1c91}
@deffn {Function} struct.pack_into (format, buffer, offset, v1, v2, ...)

Pack the values @emph{v1}, @emph{v2}, … according to the format string @emph{format} and
write the packed bytes into the writable buffer @emph{buffer} starting at
position @emph{offset}.  Note that @emph{offset} is a required argument.
@end deffn

@geindex unpack() (in module struct)
@anchor{library/struct struct unpack}@anchor{15c6}
@deffn {Function} struct.unpack (format, buffer)

Unpack from the buffer @emph{buffer} (presumably packed by @code{pack(format, ...)})
according to the format string @emph{format}.  The result is a tuple even if it
contains exactly one item.  The buffer’s size in bytes must match the
size required by the format, as reflected by @ref{125e,,calcsize()}.
@end deffn

@geindex unpack_from() (in module struct)
@anchor{library/struct struct unpack_from}@anchor{1c92}
@deffn {Function} struct.unpack_from (format, /, buffer, offset=0)

Unpack from @emph{buffer} starting at position @emph{offset}, according to the format
string @emph{format}.  The result is a tuple even if it contains exactly one
item.  The buffer’s size in bytes, starting at position @emph{offset}, must be at
least the size required by the format, as reflected by @ref{125e,,calcsize()}.
@end deffn

@geindex iter_unpack() (in module struct)
@anchor{library/struct struct iter_unpack}@anchor{b40}
@deffn {Function} struct.iter_unpack (format, buffer)

Iteratively unpack from the buffer @emph{buffer} according to the format
string @emph{format}.  This function returns an iterator which will read
equally-sized chunks from the buffer until all its contents have been
consumed.  The buffer’s size in bytes must be a multiple of the size
required by the format, as reflected by @ref{125e,,calcsize()}.

Each iteration yields a tuple as specified by the format string.


New in version 3.4.

@end deffn

@geindex calcsize() (in module struct)
@anchor{library/struct struct calcsize}@anchor{125e}
@deffn {Function} struct.calcsize (format)

Return the size of the struct (and hence of the bytes object produced by
@code{pack(format, ...)}) corresponding to the format string @emph{format}.
@end deffn

@node Format Strings,Classes<2>,Functions and Exceptions,struct — Interpret bytes as packed binary data
@anchor{library/struct format-strings}@anchor{1c93}@anchor{library/struct struct-format-strings}@anchor{1c8d}
@subsubsection Format Strings


Format strings are the mechanism used to specify the expected layout when
packing and unpacking data.  They are built up from @ref{1c94,,Format Characters},
which specify the type of data being packed/unpacked.  In addition, there are
special characters for controlling the @ref{1c8e,,Byte Order@comma{} Size@comma{} and Alignment}.

@menu
* Byte Order@comma{} Size@comma{} and Alignment: Byte Order Size and Alignment. 
* Format Characters:: 
* Examples: Examples<2>. 

@end menu

@node Byte Order Size and Alignment,Format Characters,,Format Strings
@anchor{library/struct byte-order-size-and-alignment}@anchor{1c95}@anchor{library/struct struct-alignment}@anchor{1c8e}
@subsubsection Byte Order, Size, and Alignment


By default, C types are represented in the machine’s native format and byte
order, and properly aligned by skipping pad bytes if necessary (according to the
rules used by the C compiler).

@geindex @@ (at); in struct format strings
@geindex = (equals); in struct format strings
@geindex < (less); in struct format strings
@geindex > (greater); in struct format strings
@geindex ! (exclamation); in struct format strings

Alternatively, the first character of the format string can be used to indicate
the byte order, size and alignment of the packed data, according to the
following table:


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxx} {xxxxxxxxxxxxx} 
@headitem

Character

@tab

Byte order

@tab

Size

@tab

Alignment

@item

@code{@@}

@tab

native

@tab

native

@tab

native

@item

@code{=}

@tab

native

@tab

standard

@tab

none

@item

@code{<}

@tab

little-endian

@tab

standard

@tab

none

@item

@code{>}

@tab

big-endian

@tab

standard

@tab

none

@item

@code{!}

@tab

network (= big-endian)

@tab

standard

@tab

none

@end multitable


If the first character is not one of these, @code{'@@'} is assumed.

Native byte order is big-endian or little-endian, depending on the host
system. For example, Intel x86 and AMD64 (x86-64) are little-endian;
Motorola 68000 and PowerPC G5 are big-endian; ARM and Intel Itanium feature
switchable endianness (bi-endian). Use @code{sys.byteorder} to check the
endianness of your system.

Native size and alignment are determined using the C compiler’s
@code{sizeof} expression.  This is always combined with native byte order.

Standard size depends only on the format character;  see the table in
the @ref{1c94,,Format Characters} section.

Note the difference between @code{'@@'} and @code{'='}: both use native byte order, but
the size and alignment of the latter is standardized.

The form @code{'!'} represents the network byte order which is always big-endian
as defined in IETF RFC 1700@footnote{https://tools.ietf.org/html/rfc1700}.

There is no way to indicate non-native byte order (force byte-swapping); use the
appropriate choice of @code{'<'} or @code{'>'}.

Notes:


@enumerate 

@item 
Padding is only automatically added between successive structure members.
No padding is added at the beginning or the end of the encoded struct.

@item 
No padding is added when using non-native size and alignment, e.g.
with ‘<’, ‘>’, ‘=’, and ‘!’.

@item 
To align the end of a structure to the alignment requirement of a
particular type, end the format with the code for that type with a repeat
count of zero.  See @ref{1c96,,Examples}.
@end enumerate

@node Format Characters,Examples<2>,Byte Order Size and Alignment,Format Strings
@anchor{library/struct format-characters}@anchor{1c94}@anchor{library/struct id1}@anchor{1c97}
@subsubsection Format Characters


Format characters have the following meaning; the conversion between C and
Python values should be obvious given their types.  The ‘Standard size’ column
refers to the size of the packed value in bytes when using standard size; that
is, when the format string starts with one of @code{'<'}, @code{'>'}, @code{'!'} or
@code{'='}.  When using native size, the size of the packed value is
platform-dependent.


@multitable {xxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxx} 
@headitem

Format

@tab

C Type

@tab

Python type

@tab

Standard size

@tab

Notes

@item

@code{x}

@tab

pad byte

@tab

no value

@tab

@tab

@item

@code{c}

@tab

char

@tab

bytes of length 1

@tab

1

@tab

@item

@code{b}

@tab

signed  char

@tab

integer

@tab

1

@tab

(1), (2)

@item

@code{B}

@tab

unsigned  char

@tab

integer

@tab

1

@tab

(2)

@item

@code{?}

@tab

_Bool

@tab

bool

@tab

1

@tab

(1)

@item

@code{h}

@tab

short

@tab

integer

@tab

2

@tab

(2)

@item

@code{H}

@tab

unsigned  short

@tab

integer

@tab

2

@tab

(2)

@item

@code{i}

@tab

int

@tab

integer

@tab

4

@tab

(2)

@item

@code{I}

@tab

unsigned  int

@tab

integer

@tab

4

@tab

(2)

@item

@code{l}

@tab

long

@tab

integer

@tab

4

@tab

(2)

@item

@code{L}

@tab

unsigned  long

@tab

integer

@tab

4

@tab

(2)

@item

@code{q}

@tab

long  long

@tab

integer

@tab

8

@tab

(2)

@item

@code{Q}

@tab

unsigned  long  long

@tab

integer

@tab

8

@tab

(2)

@item

@code{n}

@tab

@code{ssize_t}

@tab

integer

@tab

@tab

(3)

@item

@code{N}

@tab

@code{size_t}

@tab

integer

@tab

@tab

(3)

@item

@code{e}

@tab

(6)

@tab

float

@tab

2

@tab

(4)

@item

@code{f}

@tab

float

@tab

float

@tab

4

@tab

(4)

@item

@code{d}

@tab

double

@tab

float

@tab

8

@tab

(4)

@item

@code{s}

@tab

char[]

@tab

bytes

@tab

@tab

@item

@code{p}

@tab

char[]

@tab

bytes

@tab

@tab

@item

@code{P}

@tab

void*

@tab

integer

@tab

@tab

(5)

@end multitable



Changed in version 3.3: Added support for the @code{'n'} and @code{'N'} formats.



Changed in version 3.6: Added support for the @code{'e'} format.


Notes:


@enumerate 

@item 
@geindex ? (question mark); in struct format strings

The @code{'?'} conversion code corresponds to the _Bool type defined by
C99. If this type is not available, it is simulated using a char. In
standard mode, it is always represented by one byte.

@item 
When attempting to pack a non-integer using any of the integer conversion
codes, if the non-integer has a @code{__index__()} method then that method is
called to convert the argument to an integer before packing.


Changed in version 3.2: Added use of the @code{__index__()} method for non-integers.


@item 
The @code{'n'} and @code{'N'} conversion codes are only available for the native
size (selected as the default or with the @code{'@@'} byte order character).
For the standard size, you can use whichever of the other integer formats
fits your application.

@item 
For the @code{'f'}, @code{'d'} and @code{'e'} conversion codes, the packed
representation uses the IEEE 754 binary32, binary64 or binary16 format (for
@code{'f'}, @code{'d'} or @code{'e'} respectively), regardless of the floating-point
format used by the platform.

@item 
The @code{'P'} format character is only available for the native byte ordering
(selected as the default or with the @code{'@@'} byte order character). The byte
order character @code{'='} chooses to use little- or big-endian ordering based
on the host system. The struct module does not interpret this as native
ordering, so the @code{'P'} format is not available.

@item 
The IEEE 754 binary16 “half precision” type was introduced in the 2008
revision of the IEEE 754 standard@footnote{https://en.wikipedia.org/wiki/IEEE_floating_point#IEEE_754-2008}. It has a sign
bit, a 5-bit exponent and 11-bit precision (with 10 bits explicitly stored),
and can represent numbers between approximately @code{6.1e-05} and @code{6.5e+04}
at full precision. This type is not widely supported by C compilers: on a
typical machine, an unsigned short can be used for storage, but not for math
operations. See the Wikipedia page on the half-precision floating-point format@footnote{https://en.wikipedia.org/wiki/Half-precision_floating-point_format} for more information.
@end enumerate

A format character may be preceded by an integral repeat count.  For example,
the format string @code{'4h'} means exactly the same as @code{'hhhh'}.

Whitespace characters between formats are ignored; a count and its format must
not contain whitespace though.

For the @code{'s'} format character, the count is interpreted as the length of the
bytes, not a repeat count like for the other format characters; for example,
@code{'10s'} means a single 10-byte string, while @code{'10c'} means 10 characters.
If a count is not given, it defaults to 1.  For packing, the string is
truncated or padded with null bytes as appropriate to make it fit. For
unpacking, the resulting bytes object always has exactly the specified number
of bytes.  As a special case, @code{'0s'} means a single, empty string (while
@code{'0c'} means 0 characters).

When packing a value @code{x} using one of the integer formats (@code{'b'},
@code{'B'}, @code{'h'}, @code{'H'}, @code{'i'}, @code{'I'}, @code{'l'}, @code{'L'},
@code{'q'}, @code{'Q'}), if @code{x} is outside the valid range for that format
then @ref{eb6,,struct.error} is raised.


Changed in version 3.1: Previously, some of the integer formats wrapped out-of-range values and
raised @ref{1b2,,DeprecationWarning} instead of @ref{eb6,,struct.error}.


The @code{'p'} format character encodes a “Pascal string”, meaning a short
variable-length string stored in a @emph{fixed number of bytes}, given by the count.
The first byte stored is the length of the string, or 255, whichever is
smaller.  The bytes of the string follow.  If the string passed in to
@ref{e25,,pack()} is too long (longer than the count minus 1), only the leading
@code{count-1} bytes of the string are stored.  If the string is shorter than
@code{count-1}, it is padded with null bytes so that exactly count bytes in all
are used.  Note that for @ref{15c6,,unpack()}, the @code{'p'} format character consumes
@code{count} bytes, but that the string returned can never contain more than 255
bytes.

@geindex ? (question mark); in struct format strings

For the @code{'?'} format character, the return value is either @ref{748,,True} or
@ref{659,,False}. When packing, the truth value of the argument object is used.
Either 0 or 1 in the native or standard bool representation will be packed, and
any non-zero value will be @code{True} when unpacking.

@node Examples<2>,,Format Characters,Format Strings
@anchor{library/struct examples}@anchor{1c98}@anchor{library/struct struct-examples}@anchor{1c96}
@subsubsection Examples


@cartouche
@quotation Note 
All examples assume a native byte order, size, and alignment with a
big-endian machine.
@end quotation
@end cartouche

A basic example of packing/unpacking three integers:

@example
>>> from struct import *
>>> pack('hhl', 1, 2, 3)
b'\x00\x01\x00\x02\x00\x00\x00\x03'
>>> unpack('hhl', b'\x00\x01\x00\x02\x00\x00\x00\x03')
(1, 2, 3)
>>> calcsize('hhl')
8
@end example

Unpacked fields can be named by assigning them to variables or by wrapping
the result in a named tuple:

@example
>>> record = b'raymond   \x32\x12\x08\x01\x08'
>>> name, serialnum, school, gradelevel = unpack('<10sHHb', record)

>>> from collections import namedtuple
>>> Student = namedtuple('Student', 'name serialnum school gradelevel')
>>> Student._make(unpack('<10sHHb', record))
Student(name=b'raymond   ', serialnum=4658, school=264, gradelevel=8)
@end example

The ordering of format characters may have an impact on size since the padding
needed to satisfy alignment requirements is different:

@example
>>> pack('ci', b'*', 0x12131415)
b'*\x00\x00\x00\x12\x13\x14\x15'
>>> pack('ic', 0x12131415, b'*')
b'\x12\x13\x14\x15*'
>>> calcsize('ci')
8
>>> calcsize('ic')
5
@end example

The following format @code{'llh0l'} specifies two pad bytes at the end, assuming
longs are aligned on 4-byte boundaries:

@example
>>> pack('llh0l', 1, 2, 3)
b'\x00\x00\x00\x01\x00\x00\x00\x02\x00\x03\x00\x00'
@end example

This only works when native size and alignment are in effect; standard size and
alignment does not enforce any alignment.


@subsubheading See also



@table @asis

@item Module @ref{6,,array}

Packed binary storage of homogeneous data.

@item Module @ref{139,,xdrlib}

Packing and unpacking of XDR data.
@end table


@node Classes<2>,,Format Strings,struct — Interpret bytes as packed binary data
@anchor{library/struct classes}@anchor{1c99}@anchor{library/struct struct-objects}@anchor{1c9a}
@subsubsection Classes


The @ref{f3,,struct} module also defines the following type:

@geindex Struct (class in struct)
@anchor{library/struct struct Struct}@anchor{1c8f}
@deffn {Class} struct.Struct (format)

Return a new Struct object which writes and reads binary data according to
the format string @emph{format}.  Creating a Struct object once and calling its
methods is more efficient than calling the @ref{f3,,struct} functions with the
same format since the format string only needs to be compiled once.

@cartouche
@quotation Note 
The compiled versions of the most recent format strings passed to
@ref{1c8f,,Struct} and the module-level functions are cached, so programs
that use only a few format strings needn’t worry about reusing a single
@ref{1c8f,,Struct} instance.
@end quotation
@end cartouche

Compiled Struct objects support the following methods and attributes:

@geindex pack() (struct.Struct method)
@anchor{library/struct struct Struct pack}@anchor{1c9b}
@deffn {Method} pack (v1, v2, ...)

Identical to the @ref{e25,,pack()} function, using the compiled format.
(@code{len(result)} will equal @ref{1c9c,,size}.)
@end deffn

@geindex pack_into() (struct.Struct method)
@anchor{library/struct struct Struct pack_into}@anchor{1c9d}
@deffn {Method} pack_into (buffer, offset, v1, v2, ...)

Identical to the @ref{1c91,,pack_into()} function, using the compiled format.
@end deffn

@geindex unpack() (struct.Struct method)
@anchor{library/struct struct Struct unpack}@anchor{1c9e}
@deffn {Method} unpack (buffer)

Identical to the @ref{15c6,,unpack()} function, using the compiled format.
The buffer’s size in bytes must equal @ref{1c9c,,size}.
@end deffn

@geindex unpack_from() (struct.Struct method)
@anchor{library/struct struct Struct unpack_from}@anchor{1c9f}
@deffn {Method} unpack_from (buffer, offset=0)

Identical to the @ref{1c92,,unpack_from()} function, using the compiled format.
The buffer’s size in bytes, starting at position @emph{offset}, must be at least
@ref{1c9c,,size}.
@end deffn

@geindex iter_unpack() (struct.Struct method)
@anchor{library/struct struct Struct iter_unpack}@anchor{b41}
@deffn {Method} iter_unpack (buffer)

Identical to the @ref{b40,,iter_unpack()} function, using the compiled format.
The buffer’s size in bytes must be a multiple of @ref{1c9c,,size}.


New in version 3.4.

@end deffn

@geindex format (struct.Struct attribute)
@anchor{library/struct struct Struct format}@anchor{741}
@deffn {Attribute} format

The format string used to construct this Struct object.


Changed in version 3.7: The format string type is now @ref{2ef,,str} instead of @ref{2f0,,bytes}.

@end deffn

@geindex size (struct.Struct attribute)
@anchor{library/struct struct Struct size}@anchor{1c9c}
@deffn {Attribute} size

The calculated size of the struct (and hence of the bytes object produced
by the @ref{e25,,pack()} method) corresponding to @ref{785,,format}.
@end deffn
@end deffn

@node codecs — Codec registry and base classes,,struct — Interpret bytes as packed binary data,Binary Data Services
@anchor{library/codecs doc}@anchor{1ca0}@anchor{library/codecs codecs-codec-registry-and-base-classes}@anchor{1ca1}@anchor{library/codecs ietf-rfc-1700}@anchor{1ca2}@anchor{library/codecs module-codecs}@anchor{1a}
@subsection @code{codecs} — Codec registry and base classes


@geindex module; codecs

@strong{Source code:} Lib/codecs.py@footnote{https://github.com/python/cpython/tree/main/Lib/codecs.py}

@geindex Unicode
@geindex Codecs
@geindex Codecs; encode
@geindex Codecs; decode
@geindex streams
@geindex stackable; streams


__________________________________________________________________


This module defines base classes for standard Python codecs (encoders and
decoders) and provides access to the internal Python codec registry, which
manages the codec and error handling lookup process. Most standard codecs
are @ref{1a58,,text encodings}, which encode text to bytes,
but there are also codecs provided that encode text to text, and bytes to
bytes. Custom codecs may encode and decode between arbitrary types, but some
module features are restricted to use specifically with
@ref{1a58,,text encodings}, or with codecs that encode to
@ref{2f0,,bytes}.

The module defines the following functions for encoding and decoding with
any codec:

@geindex encode() (in module codecs)
@anchor{library/codecs codecs encode}@anchor{a76}
@deffn {Function} codecs.encode (obj, encoding='utf-8', errors='strict')

Encodes @emph{obj} using the codec registered for @emph{encoding}.

@emph{Errors} may be given to set the desired error handling scheme. The
default error handler is @code{'strict'} meaning that encoding errors raise
@ref{17c,,ValueError} (or a more codec specific subclass, such as
@ref{185,,UnicodeEncodeError}). Refer to @ref{1ca3,,Codec Base Classes} for more
information on codec error handling.
@end deffn

@geindex decode() (in module codecs)
@anchor{library/codecs codecs decode}@anchor{a77}
@deffn {Function} codecs.decode (obj, encoding='utf-8', errors='strict')

Decodes @emph{obj} using the codec registered for @emph{encoding}.

@emph{Errors} may be given to set the desired error handling scheme. The
default error handler is @code{'strict'} meaning that decoding errors raise
@ref{17c,,ValueError} (or a more codec specific subclass, such as
@ref{51f,,UnicodeDecodeError}). Refer to @ref{1ca3,,Codec Base Classes} for more
information on codec error handling.
@end deffn

The full details for each codec can also be looked up directly:

@geindex lookup() (in module codecs)
@anchor{library/codecs codecs lookup}@anchor{466}
@deffn {Function} codecs.lookup (encoding)

Looks up the codec info in the Python codec registry and returns a
@ref{1ca4,,CodecInfo} object as defined below.

Encodings are first looked up in the registry’s cache. If not found, the list of
registered search functions is scanned. If no @ref{1ca4,,CodecInfo} object is
found, a @ref{1ab5,,LookupError} is raised. Otherwise, the @ref{1ca4,,CodecInfo} object
is stored in the cache and returned to the caller.
@end deffn

@geindex CodecInfo (class in codecs)
@anchor{library/codecs codecs CodecInfo}@anchor{1ca4}
@deffn {Class} codecs.CodecInfo (encode, decode, streamreader=None, streamwriter=None, incrementalencoder=None, incrementaldecoder=None, name=None)

Codec details when looking up the codec registry. The constructor
arguments are stored in attributes of the same name:

@geindex name (codecs.CodecInfo attribute)
@anchor{library/codecs codecs CodecInfo name}@anchor{1ca5}
@deffn {Attribute} name

The name of the encoding.
@end deffn

@geindex encode (codecs.CodecInfo attribute)
@geindex decode (codecs.CodecInfo attribute)
@anchor{library/codecs codecs CodecInfo encode}@anchor{1ca6}
@deffn {Attribute} encode
@anchor{library/codecs codecs CodecInfo decode}@anchor{1ca7}
@deffnx {Attribute} decode

The stateless encoding and decoding functions. These must be
functions or methods which have the same interface as
the @ref{1ca8,,encode()} and @ref{1ca9,,decode()} methods of Codec
instances (see @ref{1caa,,Codec Interface}).
The functions or methods are expected to work in a stateless mode.
@end deffn

@geindex incrementalencoder (codecs.CodecInfo attribute)
@geindex incrementaldecoder (codecs.CodecInfo attribute)
@anchor{library/codecs codecs CodecInfo incrementalencoder}@anchor{1cab}
@deffn {Attribute} incrementalencoder
@anchor{library/codecs codecs CodecInfo incrementaldecoder}@anchor{1cac}
@deffnx {Attribute} incrementaldecoder

Incremental encoder and decoder classes or factory functions.
These have to provide the interface defined by the base classes
@ref{1cad,,IncrementalEncoder} and @ref{1cae,,IncrementalDecoder},
respectively. Incremental codecs can maintain state.
@end deffn

@geindex streamwriter (codecs.CodecInfo attribute)
@geindex streamreader (codecs.CodecInfo attribute)
@anchor{library/codecs codecs CodecInfo streamwriter}@anchor{1caf}
@deffn {Attribute} streamwriter
@anchor{library/codecs codecs CodecInfo streamreader}@anchor{1cb0}
@deffnx {Attribute} streamreader

Stream writer and reader classes or factory functions. These have to
provide the interface defined by the base classes
@ref{1cb1,,StreamWriter} and @ref{1cb2,,StreamReader}, respectively.
Stream codecs can maintain state.
@end deffn
@end deffn

To simplify access to the various codec components, the module provides
these additional functions which use @ref{466,,lookup()} for the codec lookup:

@geindex getencoder() (in module codecs)
@anchor{library/codecs codecs getencoder}@anchor{1cb3}
@deffn {Function} codecs.getencoder (encoding)

Look up the codec for the given encoding and return its encoder function.

Raises a @ref{1ab5,,LookupError} in case the encoding cannot be found.
@end deffn

@geindex getdecoder() (in module codecs)
@anchor{library/codecs codecs getdecoder}@anchor{1cb4}
@deffn {Function} codecs.getdecoder (encoding)

Look up the codec for the given encoding and return its decoder function.

Raises a @ref{1ab5,,LookupError} in case the encoding cannot be found.
@end deffn

@geindex getincrementalencoder() (in module codecs)
@anchor{library/codecs codecs getincrementalencoder}@anchor{1cb5}
@deffn {Function} codecs.getincrementalencoder (encoding)

Look up the codec for the given encoding and return its incremental encoder
class or factory function.

Raises a @ref{1ab5,,LookupError} in case the encoding cannot be found or the codec
doesn’t support an incremental encoder.
@end deffn

@geindex getincrementaldecoder() (in module codecs)
@anchor{library/codecs codecs getincrementaldecoder}@anchor{1cb6}
@deffn {Function} codecs.getincrementaldecoder (encoding)

Look up the codec for the given encoding and return its incremental decoder
class or factory function.

Raises a @ref{1ab5,,LookupError} in case the encoding cannot be found or the codec
doesn’t support an incremental decoder.
@end deffn

@geindex getreader() (in module codecs)
@anchor{library/codecs codecs getreader}@anchor{1cb7}
@deffn {Function} codecs.getreader (encoding)

Look up the codec for the given encoding and return its @ref{1cb2,,StreamReader}
class or factory function.

Raises a @ref{1ab5,,LookupError} in case the encoding cannot be found.
@end deffn

@geindex getwriter() (in module codecs)
@anchor{library/codecs codecs getwriter}@anchor{1cb8}
@deffn {Function} codecs.getwriter (encoding)

Look up the codec for the given encoding and return its @ref{1cb1,,StreamWriter}
class or factory function.

Raises a @ref{1ab5,,LookupError} in case the encoding cannot be found.
@end deffn

Custom codecs are made available by registering a suitable codec search
function:

@geindex register() (in module codecs)
@anchor{library/codecs codecs register}@anchor{1cb9}
@deffn {Function} codecs.register (search_function)

Register a codec search function. Search functions are expected to take one
argument, being the encoding name in all lower case letters with hyphens
and spaces converted to underscores, and return a @ref{1ca4,,CodecInfo} object.
In case a search function cannot find a given encoding, it should return
@code{None}.


Changed in version 3.9: Hyphens and spaces are converted to underscore.

@end deffn

@geindex unregister() (in module codecs)
@anchor{library/codecs codecs unregister}@anchor{266}
@deffn {Function} codecs.unregister (search_function)

Unregister a codec search function and clear the registry’s cache.
If the search function is not registered, do nothing.


New in version 3.10.

@end deffn

While the builtin @ref{1c8,,open()} and the associated @ref{9d,,io} module are the
recommended approach for working with encoded text files, this module
provides additional utility functions and classes that allow the use of a
wider range of codecs when working with binary files:

@geindex open() (in module codecs)
@anchor{library/codecs codecs open}@anchor{1ca}
@deffn {Function} codecs.open (filename, mode='r', encoding=None, errors='strict', buffering=- 1)

Open an encoded file using the given @emph{mode} and return an instance of
@ref{1cba,,StreamReaderWriter}, providing transparent encoding/decoding.
The default file mode is @code{'r'}, meaning to open the file in read mode.

@cartouche
@quotation Note 
Underlying encoded files are always opened in binary mode.
No automatic conversion of @code{'\n'} is done on reading and writing.
The @emph{mode} argument may be any binary mode acceptable to the built-in
@ref{1c8,,open()} function; the @code{'b'} is automatically added.
@end quotation
@end cartouche

@emph{encoding} specifies the encoding which is to be used for the file.
Any encoding that encodes to and decodes from bytes is allowed, and
the data types supported by the file methods depend on the codec used.

@emph{errors} may be given to define the error handling. It defaults to @code{'strict'}
which causes a @ref{17c,,ValueError} to be raised in case an encoding error occurs.

@emph{buffering} has the same meaning as for the built-in @ref{1c8,,open()} function.
It defaults to -1 which means that the default buffer size will be used.


Changed in version 3.11: The @code{'U'} mode has been removed.

@end deffn

@geindex EncodedFile() (in module codecs)
@anchor{library/codecs codecs EncodedFile}@anchor{1cbb}
@deffn {Function} codecs.EncodedFile (file, data_encoding, file_encoding=None, errors='strict')

Return a @ref{129b,,StreamRecoder} instance, a wrapped version of @emph{file}
which provides transparent transcoding. The original file is closed
when the wrapped version is closed.

Data written to the wrapped file is decoded according to the given
@emph{data_encoding} and then written to the original file as bytes using
@emph{file_encoding}. Bytes read from the original file are decoded
according to @emph{file_encoding}, and the result is encoded
using @emph{data_encoding}.

If @emph{file_encoding} is not given, it defaults to @emph{data_encoding}.

@emph{errors} may be given to define the error handling. It defaults to
@code{'strict'}, which causes @ref{17c,,ValueError} to be raised in case an encoding
error occurs.
@end deffn

@geindex iterencode() (in module codecs)
@anchor{library/codecs codecs iterencode}@anchor{1cbc}
@deffn {Function} codecs.iterencode (iterator, encoding, errors='strict', **kwargs)

Uses an incremental encoder to iteratively encode the input provided by
@emph{iterator}. This function is a @ref{bf7,,generator}.
The @emph{errors} argument (as well as any
other keyword argument) is passed through to the incremental encoder.

This function requires that the codec accept text @ref{2ef,,str} objects
to encode. Therefore it does not support bytes-to-bytes encoders such as
@code{base64_codec}.
@end deffn

@geindex iterdecode() (in module codecs)
@anchor{library/codecs codecs iterdecode}@anchor{1cbd}
@deffn {Function} codecs.iterdecode (iterator, encoding, errors='strict', **kwargs)

Uses an incremental decoder to iteratively decode the input provided by
@emph{iterator}. This function is a @ref{bf7,,generator}.
The @emph{errors} argument (as well as any
other keyword argument) is passed through to the incremental decoder.

This function requires that the codec accept @ref{2f0,,bytes} objects
to decode. Therefore it does not support text-to-text encoders such as
@code{rot_13}, although @code{rot_13} may be used equivalently with
@ref{1cbc,,iterencode()}.
@end deffn

The module also provides the following constants which are useful for reading
and writing to platform dependent files:

@geindex BOM (in module codecs)
@geindex BOM_BE (in module codecs)
@geindex BOM_LE (in module codecs)
@geindex BOM_UTF8 (in module codecs)
@geindex BOM_UTF16 (in module codecs)
@geindex BOM_UTF16_BE (in module codecs)
@geindex BOM_UTF16_LE (in module codecs)
@geindex BOM_UTF32 (in module codecs)
@geindex BOM_UTF32_BE (in module codecs)
@geindex BOM_UTF32_LE (in module codecs)
@anchor{library/codecs codecs BOM}@anchor{1cbe}
@deffn {Data} codecs.BOM
@anchor{library/codecs codecs BOM_BE}@anchor{1cbf}
@deffnx {Data} codecs.BOM_BE
@anchor{library/codecs codecs BOM_LE}@anchor{1cc0}
@deffnx {Data} codecs.BOM_LE
@anchor{library/codecs codecs BOM_UTF8}@anchor{1cc1}
@deffnx {Data} codecs.BOM_UTF8
@anchor{library/codecs codecs BOM_UTF16}@anchor{1cc2}
@deffnx {Data} codecs.BOM_UTF16
@anchor{library/codecs codecs BOM_UTF16_BE}@anchor{1cc3}
@deffnx {Data} codecs.BOM_UTF16_BE
@anchor{library/codecs codecs BOM_UTF16_LE}@anchor{1cc4}
@deffnx {Data} codecs.BOM_UTF16_LE
@anchor{library/codecs codecs BOM_UTF32}@anchor{1cc5}
@deffnx {Data} codecs.BOM_UTF32
@anchor{library/codecs codecs BOM_UTF32_BE}@anchor{1cc6}
@deffnx {Data} codecs.BOM_UTF32_BE
@anchor{library/codecs codecs BOM_UTF32_LE}@anchor{1cc7}
@deffnx {Data} codecs.BOM_UTF32_LE

These constants define various byte sequences,
being Unicode byte order marks (BOMs) for several encodings. They are
used in UTF-16 and UTF-32 data streams to indicate the byte order used,
and in UTF-8 as a Unicode signature. @ref{1cc2,,BOM_UTF16} is either
@ref{1cc3,,BOM_UTF16_BE} or @ref{1cc4,,BOM_UTF16_LE} depending on the platform’s
native byte order, @ref{1cbe,,BOM} is an alias for @ref{1cc2,,BOM_UTF16},
@ref{1cc0,,BOM_LE} for @ref{1cc4,,BOM_UTF16_LE} and @ref{1cbf,,BOM_BE} for
@ref{1cc3,,BOM_UTF16_BE}. The others represent the BOM in UTF-8 and UTF-32
encodings.
@end deffn

@menu
* Codec Base Classes:: 
* Encodings and Unicode:: 
* Standard Encodings:: 
* Python Specific Encodings:: 
* encodings.idna — Internationalized Domain Names in Applications: encodings idna — Internationalized Domain Names in Applications. 
* encodings.mbcs — Windows ANSI codepage: encodings mbcs — Windows ANSI codepage. 
* encodings.utf_8_sig — UTF-8 codec with BOM signature: encodings utf_8_sig — UTF-8 codec with BOM signature. 

@end menu

@node Codec Base Classes,Encodings and Unicode,,codecs — Codec registry and base classes
@anchor{library/codecs codec-base-classes}@anchor{1ca3}@anchor{library/codecs id1}@anchor{1cc8}
@subsubsection Codec Base Classes


The @ref{1a,,codecs} module defines a set of base classes which define the
interfaces for working with codec objects, and can also be used as the basis
for custom codec implementations.

Each codec has to define four interfaces to make it usable as codec in Python:
stateless encoder, stateless decoder, stream reader and stream writer. The
stream reader and writers typically reuse the stateless encoder/decoder to
implement the file protocols. Codec authors also need to define how the
codec will handle encoding and decoding errors.
@anchor{library/codecs surrogateescape}@anchor{1cc9}
@menu
* Error Handlers:: 
* Stateless Encoding and Decoding:: 
* Incremental Encoding and Decoding:: 
* Stream Encoding and Decoding:: 

@end menu

@node Error Handlers,Stateless Encoding and Decoding,,Codec Base Classes
@anchor{library/codecs error-handlers}@anchor{162a}@anchor{library/codecs id2}@anchor{1cca}
@subsubsection Error Handlers


To simplify and standardize error handling,
codecs may implement different error handling schemes by
accepting the @emph{errors} string argument. The following string values are
defined and implemented by all standard Python codecs:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Value

@tab

Meaning

@item

@code{'strict'}

@tab

Raise @ref{e49,,UnicodeError} (or a subclass);
this is the default. Implemented in
@ref{1ccb,,strict_errors()}.

@item

@code{'ignore'}

@tab

Ignore the malformed data and continue
without further notice. Implemented in
@ref{1ccc,,ignore_errors()}.

@end multitable


The following error handlers are only applicable to
@ref{1a58,,text encodings}:

@geindex ? (question mark); replacement character
@geindex \ (backslash); escape sequence
@geindex \x; escape sequence
@geindex \u; escape sequence
@geindex \U; escape sequence
@geindex \N; escape sequence


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Value

@tab

Meaning

@item

@code{'replace'}

@tab

Replace with a suitable replacement
marker; Python will use the official
@code{U+FFFD} REPLACEMENT CHARACTER for the
built-in codecs on decoding, and ‘?’ on
encoding. Implemented in
@ref{1ccd,,replace_errors()}.

@item

@code{'xmlcharrefreplace'}

@tab

Replace with the appropriate XML character
reference (only for encoding). Implemented
in @ref{1cce,,xmlcharrefreplace_errors()}.

@item

@code{'backslashreplace'}

@tab

Replace with backslashed escape sequences.
Implemented in
@ref{1ccf,,backslashreplace_errors()}.

@item

@code{'namereplace'}

@tab

Replace with @code{\N@{...@}} escape sequences
(only for encoding). Implemented in
@ref{1cd0,,namereplace_errors()}.

@item

@code{'surrogateescape'}

@tab

On decoding, replace byte with individual
surrogate code ranging from @code{U+DC80} to
@code{U+DCFF}. This code will then be turned
back into the same byte when the
@code{'surrogateescape'} error handler is used
when encoding the data. (See 
@geindex Python Enhancement Proposals; PEP 383
PEP 383@footnote{https://www.python.org/dev/peps/pep-0383} for
more.)

@end multitable


In addition, the following error handler is specific to the given codecs:


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Value

@tab

Codecs

@tab

Meaning

@item

@code{'surrogatepass'}

@tab

utf-8, utf-16, utf-32,
utf-16-be, utf-16-le,
utf-32-be, utf-32-le

@tab

Allow encoding and decoding of surrogate
codes. These codecs normally treat the
presence of surrogates as an error.

@end multitable



New in version 3.1: The @code{'surrogateescape'} and @code{'surrogatepass'} error handlers.



Changed in version 3.4: The @code{'surrogatepass'} error handlers now works with utf-16* and utf-32* codecs.



New in version 3.5: The @code{'namereplace'} error handler.



Changed in version 3.5: The @code{'backslashreplace'} error handlers now works with decoding and
translating.


The set of allowed values can be extended by registering a new named error
handler:

@geindex register_error() (in module codecs)
@anchor{library/codecs codecs register_error}@anchor{fed}
@deffn {Function} codecs.register_error (name, error_handler)

Register the error handling function @emph{error_handler} under the name @emph{name}.
The @emph{error_handler} argument will be called during encoding and decoding
in case of an error, when @emph{name} is specified as the errors parameter.

For encoding, @emph{error_handler} will be called with a @ref{185,,UnicodeEncodeError}
instance, which contains information about the location of the error. The
error handler must either raise this or a different exception, or return a
tuple with a replacement for the unencodable part of the input and a position
where encoding should continue. The replacement may be either @ref{2ef,,str} or
@ref{2f0,,bytes}. If the replacement is bytes, the encoder will simply copy
them into the output buffer. If the replacement is a string, the encoder will
encode the replacement. Encoding continues on original input at the
specified position. Negative position values will be treated as being
relative to the end of the input string. If the resulting position is out of
bound an @ref{1066,,IndexError} will be raised.

Decoding and translating works similarly, except @ref{51f,,UnicodeDecodeError} or
@ref{1b88,,UnicodeTranslateError} will be passed to the handler and that the
replacement from the error handler will be put into the output directly.
@end deffn

Previously registered error handlers (including the standard error handlers)
can be looked up by name:

@geindex lookup_error() (in module codecs)
@anchor{library/codecs codecs lookup_error}@anchor{fee}
@deffn {Function} codecs.lookup_error (name)

Return the error handler previously registered under the name @emph{name}.

Raises a @ref{1ab5,,LookupError} in case the handler cannot be found.
@end deffn

The following standard error handlers are also made available as module level
functions:

@geindex strict_errors() (in module codecs)
@anchor{library/codecs codecs strict_errors}@anchor{1ccb}
@deffn {Function} codecs.strict_errors (exception)

Implements the @code{'strict'} error handling: each encoding or
decoding error raises a @ref{e49,,UnicodeError}.
@end deffn

@geindex replace_errors() (in module codecs)
@anchor{library/codecs codecs replace_errors}@anchor{1ccd}
@deffn {Function} codecs.replace_errors (exception)

Implements the @code{'replace'} error handling (for @ref{1a58,,text encodings} only): substitutes @code{'?'} for encoding errors
(to be encoded by the codec), and @code{'\ufffd'} (the Unicode replacement
character) for decoding errors.
@end deffn

@geindex ignore_errors() (in module codecs)
@anchor{library/codecs codecs ignore_errors}@anchor{1ccc}
@deffn {Function} codecs.ignore_errors (exception)

Implements the @code{'ignore'} error handling: malformed data is ignored and
encoding or decoding is continued without further notice.
@end deffn

@geindex xmlcharrefreplace_errors() (in module codecs)
@anchor{library/codecs codecs xmlcharrefreplace_errors}@anchor{1cce}
@deffn {Function} codecs.xmlcharrefreplace_errors (exception)

Implements the @code{'xmlcharrefreplace'} error handling (for encoding with
@ref{1a58,,text encodings} only): the
unencodable character is replaced by an appropriate XML character reference.
@end deffn

@geindex backslashreplace_errors() (in module codecs)
@anchor{library/codecs codecs backslashreplace_errors}@anchor{1ccf}
@deffn {Function} codecs.backslashreplace_errors (exception)

Implements the @code{'backslashreplace'} error handling (for
@ref{1a58,,text encodings} only): malformed data is
replaced by a backslashed escape sequence.
@end deffn

@geindex namereplace_errors() (in module codecs)
@anchor{library/codecs codecs namereplace_errors}@anchor{1cd0}
@deffn {Function} codecs.namereplace_errors (exception)

Implements the @code{'namereplace'} error handling (for encoding with
@ref{1a58,,text encodings} only): the
unencodable character is replaced by a @code{\N@{...@}} escape sequence.


New in version 3.5.

@end deffn

@node Stateless Encoding and Decoding,Incremental Encoding and Decoding,Error Handlers,Codec Base Classes
@anchor{library/codecs codec-objects}@anchor{1caa}@anchor{library/codecs stateless-encoding-and-decoding}@anchor{1cd1}
@subsubsection Stateless Encoding and Decoding


The base @code{Codec} class defines these methods which also define the
function interfaces of the stateless encoder and decoder:

@geindex encode() (codecs.Codec method)
@anchor{library/codecs codecs Codec encode}@anchor{1ca8}
@deffn {Method} Codec.encode (input[, errors])

Encodes the object @emph{input} and returns a tuple (output object, length consumed).
For instance, @ref{1a58,,text encoding} converts
a string object to a bytes object using a particular
character set encoding (e.g., @code{cp1252} or @code{iso-8859-1}).

The @emph{errors} argument defines the error handling to apply.
It defaults to @code{'strict'} handling.

The method may not store state in the @code{Codec} instance. Use
@ref{1cb1,,StreamWriter} for codecs which have to keep state in order to make
encoding efficient.

The encoder must be able to handle zero length input and return an empty object
of the output object type in this situation.
@end deffn

@geindex decode() (codecs.Codec method)
@anchor{library/codecs codecs Codec decode}@anchor{1ca9}
@deffn {Method} Codec.decode (input[, errors])

Decodes the object @emph{input} and returns a tuple (output object, length
consumed). For instance, for a @ref{1a58,,text encoding}, decoding converts
a bytes object encoded using a particular
character set encoding to a string object.

For text encodings and bytes-to-bytes codecs,
@emph{input} must be a bytes object or one which provides the read-only
buffer interface – for example, buffer objects and memory mapped files.

The @emph{errors} argument defines the error handling to apply.
It defaults to @code{'strict'} handling.

The method may not store state in the @code{Codec} instance. Use
@ref{1cb2,,StreamReader} for codecs which have to keep state in order to make
decoding efficient.

The decoder must be able to handle zero length input and return an empty object
of the output object type in this situation.
@end deffn

@node Incremental Encoding and Decoding,Stream Encoding and Decoding,Stateless Encoding and Decoding,Codec Base Classes
@anchor{library/codecs incremental-encoding-and-decoding}@anchor{1cd2}
@subsubsection Incremental Encoding and Decoding


The @ref{1cad,,IncrementalEncoder} and @ref{1cae,,IncrementalDecoder} classes provide
the basic interface for incremental encoding and decoding. Encoding/decoding the
input isn’t done with one call to the stateless encoder/decoder function, but
with multiple calls to the
@ref{1cd3,,encode()}/@ref{1cd4,,decode()} method of
the incremental encoder/decoder. The incremental encoder/decoder keeps track of
the encoding/decoding process during method calls.

The joined output of calls to the
@ref{1cd3,,encode()}/@ref{1cd4,,decode()} method is
the same as if all the single inputs were joined into one, and this input was
encoded/decoded with the stateless encoder/decoder.

@menu
* IncrementalEncoder Objects:: 
* IncrementalDecoder Objects:: 

@end menu

@node IncrementalEncoder Objects,IncrementalDecoder Objects,,Incremental Encoding and Decoding
@anchor{library/codecs incremental-encoder-objects}@anchor{1cd5}@anchor{library/codecs incrementalencoder-objects}@anchor{1cd6}
@subsubsection IncrementalEncoder Objects


The @ref{1cad,,IncrementalEncoder} class is used for encoding an input in multiple
steps. It defines the following methods which every incremental encoder must
define in order to be compatible with the Python codec registry.

@geindex IncrementalEncoder (class in codecs)
@anchor{library/codecs codecs IncrementalEncoder}@anchor{1cad}
@deffn {Class} codecs.IncrementalEncoder (errors='strict')

Constructor for an @ref{1cad,,IncrementalEncoder} instance.

All incremental encoders must provide this constructor interface. They are free
to add additional keyword arguments, but only the ones defined here are used by
the Python codec registry.

The @ref{1cad,,IncrementalEncoder} may implement different error handling schemes
by providing the @emph{errors} keyword argument. See @ref{162a,,Error Handlers} for
possible values.

The @emph{errors} argument will be assigned to an attribute of the same name.
Assigning to this attribute makes it possible to switch between different error
handling strategies during the lifetime of the @ref{1cad,,IncrementalEncoder}
object.

@geindex encode() (codecs.IncrementalEncoder method)
@anchor{library/codecs codecs IncrementalEncoder encode}@anchor{1cd3}
@deffn {Method} encode (object[, final])

Encodes @emph{object} (taking the current state of the encoder into account)
and returns the resulting encoded object. If this is the last call to
@ref{a76,,encode()} @emph{final} must be true (the default is false).
@end deffn

@geindex reset() (codecs.IncrementalEncoder method)
@anchor{library/codecs codecs IncrementalEncoder reset}@anchor{1cd7}
@deffn {Method} reset ()

Reset the encoder to the initial state. The output is discarded: call
@code{.encode(object, final=True)}, passing an empty byte or text string
if necessary, to reset the encoder and to get the output.
@end deffn

@geindex getstate() (codecs.IncrementalEncoder method)
@anchor{library/codecs codecs IncrementalEncoder getstate}@anchor{1cd8}
@deffn {Method} getstate ()

Return the current state of the encoder which must be an integer. The
implementation should make sure that @code{0} is the most common
state. (States that are more complicated than integers can be converted
into an integer by marshaling/pickling the state and encoding the bytes
of the resulting string into an integer.)
@end deffn

@geindex setstate() (codecs.IncrementalEncoder method)
@anchor{library/codecs codecs IncrementalEncoder setstate}@anchor{1cd9}
@deffn {Method} setstate (state)

Set the state of the encoder to @emph{state}. @emph{state} must be an encoder state
returned by @ref{1cd8,,getstate()}.
@end deffn
@end deffn

@node IncrementalDecoder Objects,,IncrementalEncoder Objects,Incremental Encoding and Decoding
@anchor{library/codecs incremental-decoder-objects}@anchor{1cda}@anchor{library/codecs incrementaldecoder-objects}@anchor{1cdb}
@subsubsection IncrementalDecoder Objects


The @ref{1cae,,IncrementalDecoder} class is used for decoding an input in multiple
steps. It defines the following methods which every incremental decoder must
define in order to be compatible with the Python codec registry.

@geindex IncrementalDecoder (class in codecs)
@anchor{library/codecs codecs IncrementalDecoder}@anchor{1cae}
@deffn {Class} codecs.IncrementalDecoder (errors='strict')

Constructor for an @ref{1cae,,IncrementalDecoder} instance.

All incremental decoders must provide this constructor interface. They are free
to add additional keyword arguments, but only the ones defined here are used by
the Python codec registry.

The @ref{1cae,,IncrementalDecoder} may implement different error handling schemes
by providing the @emph{errors} keyword argument. See @ref{162a,,Error Handlers} for
possible values.

The @emph{errors} argument will be assigned to an attribute of the same name.
Assigning to this attribute makes it possible to switch between different error
handling strategies during the lifetime of the @ref{1cae,,IncrementalDecoder}
object.

@geindex decode() (codecs.IncrementalDecoder method)
@anchor{library/codecs codecs IncrementalDecoder decode}@anchor{1cd4}
@deffn {Method} decode (object[, final])

Decodes @emph{object} (taking the current state of the decoder into account)
and returns the resulting decoded object. If this is the last call to
@ref{a77,,decode()} @emph{final} must be true (the default is false). If @emph{final} is
true the decoder must decode the input completely and must flush all
buffers. If this isn’t possible (e.g. because of incomplete byte sequences
at the end of the input) it must initiate error handling just like in the
stateless case (which might raise an exception).
@end deffn

@geindex reset() (codecs.IncrementalDecoder method)
@anchor{library/codecs codecs IncrementalDecoder reset}@anchor{1cdc}
@deffn {Method} reset ()

Reset the decoder to the initial state.
@end deffn

@geindex getstate() (codecs.IncrementalDecoder method)
@anchor{library/codecs codecs IncrementalDecoder getstate}@anchor{1cdd}
@deffn {Method} getstate ()

Return the current state of the decoder. This must be a tuple with two
items, the first must be the buffer containing the still undecoded
input. The second must be an integer and can be additional state
info. (The implementation should make sure that @code{0} is the most common
additional state info.) If this additional state info is @code{0} it must be
possible to set the decoder to the state which has no input buffered and
@code{0} as the additional state info, so that feeding the previously
buffered input to the decoder returns it to the previous state without
producing any output. (Additional state info that is more complicated than
integers can be converted into an integer by marshaling/pickling the info
and encoding the bytes of the resulting string into an integer.)
@end deffn

@geindex setstate() (codecs.IncrementalDecoder method)
@anchor{library/codecs codecs IncrementalDecoder setstate}@anchor{1cde}
@deffn {Method} setstate (state)

Set the state of the decoder to @emph{state}. @emph{state} must be a decoder state
returned by @ref{1cdd,,getstate()}.
@end deffn
@end deffn

@node Stream Encoding and Decoding,,Incremental Encoding and Decoding,Codec Base Classes
@anchor{library/codecs stream-encoding-and-decoding}@anchor{1cdf}
@subsubsection Stream Encoding and Decoding


The @ref{1cb1,,StreamWriter} and @ref{1cb2,,StreamReader} classes provide generic
working interfaces which can be used to implement new encoding submodules very
easily. See @code{encodings.utf_8} for an example of how this is done.

@menu
* StreamWriter Objects:: 
* StreamReader Objects:: 
* StreamReaderWriter Objects:: 
* StreamRecoder Objects:: 

@end menu

@node StreamWriter Objects,StreamReader Objects,,Stream Encoding and Decoding
@anchor{library/codecs stream-writer-objects}@anchor{1ce0}@anchor{library/codecs streamwriter-objects}@anchor{1ce1}
@subsubsection StreamWriter Objects


The @ref{1cb1,,StreamWriter} class is a subclass of @code{Codec} and defines the
following methods which every stream writer must define in order to be
compatible with the Python codec registry.

@geindex StreamWriter (class in codecs)
@anchor{library/codecs codecs StreamWriter}@anchor{1cb1}
@deffn {Class} codecs.StreamWriter (stream, errors='strict')

Constructor for a @ref{1cb1,,StreamWriter} instance.

All stream writers must provide this constructor interface. They are free to add
additional keyword arguments, but only the ones defined here are used by the
Python codec registry.

The @emph{stream} argument must be a file-like object open for writing
text or binary data, as appropriate for the specific codec.

The @ref{1cb1,,StreamWriter} may implement different error handling schemes by
providing the @emph{errors} keyword argument. See @ref{162a,,Error Handlers} for
the standard error handlers the underlying stream codec may support.

The @emph{errors} argument will be assigned to an attribute of the same name.
Assigning to this attribute makes it possible to switch between different error
handling strategies during the lifetime of the @ref{1cb1,,StreamWriter} object.

@geindex write() (codecs.StreamWriter method)
@anchor{library/codecs codecs StreamWriter write}@anchor{1ce2}
@deffn {Method} write (object)

Writes the object’s contents encoded to the stream.
@end deffn

@geindex writelines() (codecs.StreamWriter method)
@anchor{library/codecs codecs StreamWriter writelines}@anchor{1ce3}
@deffn {Method} writelines (list)

Writes the concatenated iterable of strings to the stream (possibly by reusing
the @ref{1ce2,,write()} method). Infinite or
very large iterables are not supported. The standard bytes-to-bytes codecs
do not support this method.
@end deffn

@geindex reset() (codecs.StreamWriter method)
@anchor{library/codecs codecs StreamWriter reset}@anchor{1ce4}
@deffn {Method} reset ()

Resets the codec buffers used for keeping internal state.

Calling this method should ensure that the data on the output is put into
a clean state that allows appending of new fresh data without having to
rescan the whole stream to recover state.
@end deffn
@end deffn

In addition to the above methods, the @ref{1cb1,,StreamWriter} must also inherit
all other methods and attributes from the underlying stream.

@node StreamReader Objects,StreamReaderWriter Objects,StreamWriter Objects,Stream Encoding and Decoding
@anchor{library/codecs stream-reader-objects}@anchor{1ce5}@anchor{library/codecs streamreader-objects}@anchor{1ce6}
@subsubsection StreamReader Objects


The @ref{1cb2,,StreamReader} class is a subclass of @code{Codec} and defines the
following methods which every stream reader must define in order to be
compatible with the Python codec registry.

@geindex StreamReader (class in codecs)
@anchor{library/codecs codecs StreamReader}@anchor{1cb2}
@deffn {Class} codecs.StreamReader (stream, errors='strict')

Constructor for a @ref{1cb2,,StreamReader} instance.

All stream readers must provide this constructor interface. They are free to add
additional keyword arguments, but only the ones defined here are used by the
Python codec registry.

The @emph{stream} argument must be a file-like object open for reading
text or binary data, as appropriate for the specific codec.

The @ref{1cb2,,StreamReader} may implement different error handling schemes by
providing the @emph{errors} keyword argument. See @ref{162a,,Error Handlers} for
the standard error handlers the underlying stream codec may support.

The @emph{errors} argument will be assigned to an attribute of the same name.
Assigning to this attribute makes it possible to switch between different error
handling strategies during the lifetime of the @ref{1cb2,,StreamReader} object.

The set of allowed values for the @emph{errors} argument can be extended with
@ref{fed,,register_error()}.

@geindex read() (codecs.StreamReader method)
@anchor{library/codecs codecs StreamReader read}@anchor{1ce7}
@deffn {Method} read ([size[, chars[, firstline]]])

Decodes data from the stream and returns the resulting object.

The @emph{chars} argument indicates the number of decoded
code points or bytes to return. The @ref{1ce7,,read()} method will
never return more data than requested, but it might return less,
if there is not enough available.

The @emph{size} argument indicates the approximate maximum
number of encoded bytes or code points to read
for decoding. The decoder can modify this setting as
appropriate. The default value -1 indicates to read and decode as much as
possible. This parameter is intended to
prevent having to decode huge files in one step.

The @emph{firstline} flag indicates that
it would be sufficient to only return the first
line, if there are decoding errors on later lines.

The method should use a greedy read strategy meaning that it should read
as much data as is allowed within the definition of the encoding and the
given size, e.g.  if optional encoding endings or state markers are
available on the stream, these should be read too.
@end deffn

@geindex readline() (codecs.StreamReader method)
@anchor{library/codecs codecs StreamReader readline}@anchor{1ce8}
@deffn {Method} readline ([size[, keepends]])

Read one line from the input stream and return the decoded data.

@emph{size}, if given, is passed as size argument to the stream’s
@ref{1ce7,,read()} method.

If @emph{keepends} is false line-endings will be stripped from the lines
returned.
@end deffn

@geindex readlines() (codecs.StreamReader method)
@anchor{library/codecs codecs StreamReader readlines}@anchor{1ce9}
@deffn {Method} readlines ([sizehint[, keepends]])

Read all lines available on the input stream and return them as a list of
lines.

Line-endings are implemented using the codec’s @ref{a77,,decode()} method and
are included in the list entries if @emph{keepends} is true.

@emph{sizehint}, if given, is passed as the @emph{size} argument to the stream’s
@ref{1ce7,,read()} method.
@end deffn

@geindex reset() (codecs.StreamReader method)
@anchor{library/codecs codecs StreamReader reset}@anchor{1cea}
@deffn {Method} reset ()

Resets the codec buffers used for keeping internal state.

Note that no stream repositioning should take place. This method is
primarily intended to be able to recover from decoding errors.
@end deffn
@end deffn

In addition to the above methods, the @ref{1cb2,,StreamReader} must also inherit
all other methods and attributes from the underlying stream.

@node StreamReaderWriter Objects,StreamRecoder Objects,StreamReader Objects,Stream Encoding and Decoding
@anchor{library/codecs stream-reader-writer}@anchor{1ceb}@anchor{library/codecs streamreaderwriter-objects}@anchor{1cec}
@subsubsection StreamReaderWriter Objects


The @ref{1cba,,StreamReaderWriter} is a convenience class that allows wrapping
streams which work in both read and write modes.

The design is such that one can use the factory functions returned by the
@ref{466,,lookup()} function to construct the instance.

@geindex StreamReaderWriter (class in codecs)
@anchor{library/codecs codecs StreamReaderWriter}@anchor{1cba}
@deffn {Class} codecs.StreamReaderWriter (stream, Reader, Writer, errors='strict')

Creates a @ref{1cba,,StreamReaderWriter} instance. @emph{stream} must be a file-like
object. @emph{Reader} and @emph{Writer} must be factory functions or classes providing the
@ref{1cb2,,StreamReader} and @ref{1cb1,,StreamWriter} interface resp. Error handling
is done in the same way as defined for the stream readers and writers.
@end deffn

@ref{1cba,,StreamReaderWriter} instances define the combined interfaces of
@ref{1cb2,,StreamReader} and @ref{1cb1,,StreamWriter} classes. They inherit all other
methods and attributes from the underlying stream.

@node StreamRecoder Objects,,StreamReaderWriter Objects,Stream Encoding and Decoding
@anchor{library/codecs stream-recoder-objects}@anchor{1ced}@anchor{library/codecs streamrecoder-objects}@anchor{1cee}
@subsubsection StreamRecoder Objects


The @ref{129b,,StreamRecoder} translates data from one encoding to another,
which is sometimes useful when dealing with different encoding environments.

The design is such that one can use the factory functions returned by the
@ref{466,,lookup()} function to construct the instance.

@geindex StreamRecoder (class in codecs)
@anchor{library/codecs codecs StreamRecoder}@anchor{129b}
@deffn {Class} codecs.StreamRecoder (stream, encode, decode, Reader, Writer, errors='strict')

Creates a @ref{129b,,StreamRecoder} instance which implements a two-way conversion:
@emph{encode} and @emph{decode} work on the frontend — the data visible to
code calling @code{read()} and @code{write()}, while @emph{Reader} and @emph{Writer}
work on the backend — the data in @emph{stream}.

You can use these objects to do transparent transcodings, e.g., from Latin-1
to UTF-8 and back.

The @emph{stream} argument must be a file-like object.

The @emph{encode} and @emph{decode} arguments must
adhere to the @code{Codec} interface. @emph{Reader} and
@emph{Writer} must be factory functions or classes providing objects of the
@ref{1cb2,,StreamReader} and @ref{1cb1,,StreamWriter} interface respectively.

Error handling is done in the same way as defined for the stream readers and
writers.
@end deffn

@ref{129b,,StreamRecoder} instances define the combined interfaces of
@ref{1cb2,,StreamReader} and @ref{1cb1,,StreamWriter} classes. They inherit all other
methods and attributes from the underlying stream.

@node Encodings and Unicode,Standard Encodings,Codec Base Classes,codecs — Codec registry and base classes
@anchor{library/codecs encodings-and-unicode}@anchor{1cef}@anchor{library/codecs encodings-overview}@anchor{1cf0}
@subsubsection Encodings and Unicode


Strings are stored internally as sequences of code points in
range @code{0x0}–@code{0x10FFFF}. (See 
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393} for
more details about the implementation.)
Once a string object is used outside of CPU and memory, endianness
and how these arrays are stored as bytes become an issue. As with other
codecs, serialising a string into a sequence of bytes is known as @emph{encoding},
and recreating the string from the sequence of bytes is known as @emph{decoding}.
There are a variety of different text serialisation codecs, which are
collectivity referred to as @ref{1a58,,text encodings}.

The simplest text encoding (called @code{'latin-1'} or @code{'iso-8859-1'}) maps
the code points 0–255 to the bytes @code{0x0}–@code{0xff}, which means that a string
object that contains code points above @code{U+00FF} can’t be encoded with this
codec. Doing so will raise a @ref{185,,UnicodeEncodeError} that looks
like the following (although the details of the error message may differ):
@code{UnicodeEncodeError: 'latin-1' codec can't encode character '\u1234' in
position 3: ordinal not in range(256)}.

There’s another group of encodings (the so called charmap encodings) that choose
a different subset of all Unicode code points and how these code points are
mapped to the bytes @code{0x0}–@code{0xff}. To see how this is done simply open
e.g. @code{encodings/cp1252.py} (which is an encoding that is used primarily on
Windows). There’s a string constant with 256 characters that shows you which
character is mapped to which byte value.

All of these encodings can only encode 256 of the 1114112 code points
defined in Unicode. A simple and straightforward way that can store each Unicode
code point, is to store each code point as four consecutive bytes. There are two
possibilities: store the bytes in big endian or in little endian order. These
two encodings are called @code{UTF-32-BE} and @code{UTF-32-LE} respectively. Their
disadvantage is that if e.g. you use @code{UTF-32-BE} on a little endian machine you
will always have to swap bytes on encoding and decoding. @code{UTF-32} avoids this
problem: bytes will always be in natural endianness. When these bytes are read
by a CPU with a different endianness, then bytes have to be swapped though. To
be able to detect the endianness of a @code{UTF-16} or @code{UTF-32} byte sequence,
there’s the so called BOM (“Byte Order Mark”). This is the Unicode character
@code{U+FEFF}. This character can be prepended to every @code{UTF-16} or @code{UTF-32}
byte sequence. The byte swapped version of this character (@code{0xFFFE}) is an
illegal character that may not appear in a Unicode text. So when the
first character in an @code{UTF-16} or @code{UTF-32} byte sequence
appears to be a @code{U+FFFE} the bytes have to be swapped on decoding.
Unfortunately the character @code{U+FEFF} had a second purpose as
a @code{ZERO WIDTH NO-BREAK SPACE}: a character that has no width and doesn’t allow
a word to be split. It can e.g. be used to give hints to a ligature algorithm.
With Unicode 4.0 using @code{U+FEFF} as a @code{ZERO WIDTH NO-BREAK SPACE} has been
deprecated (with @code{U+2060} (@code{WORD JOINER}) assuming this role). Nevertheless
Unicode software still must be able to handle @code{U+FEFF} in both roles: as a BOM
it’s a device to determine the storage layout of the encoded bytes, and vanishes
once the byte sequence has been decoded into a string; as a @code{ZERO WIDTH
NO-BREAK SPACE} it’s a normal character that will be decoded like any other.

There’s another encoding that is able to encode the full range of Unicode
characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues
with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two
parts: marker bits (the most significant bits) and payload bits. The marker bits
are a sequence of zero to four @code{1} bits followed by a @code{0} bit. Unicode characters are
encoded like this (with x being payload bits, which when concatenated give the
Unicode character):


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Range

@tab

Encoding

@item

@code{U-00000000} … @code{U-0000007F}

@tab

0xxxxxxx

@item

@code{U-00000080} … @code{U-000007FF}

@tab

110xxxxx 10xxxxxx

@item

@code{U-00000800} … @code{U-0000FFFF}

@tab

1110xxxx 10xxxxxx 10xxxxxx

@item

@code{U-00010000} … @code{U-0010FFFF}

@tab

11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

@end multitable


The least significant bit of the Unicode character is the rightmost x bit.

As UTF-8 is an 8-bit encoding no BOM is required and any @code{U+FEFF} character in
the decoded string (even if it’s the first character) is treated as a @code{ZERO
WIDTH NO-BREAK SPACE}.

Without external information it’s impossible to reliably determine which
encoding was used for encoding a string. Each charmap encoding can
decode any random byte sequence. However that’s not possible with UTF-8, as
UTF-8 byte sequences have a structure that doesn’t allow arbitrary byte
sequences. To increase the reliability with which a UTF-8 encoding can be
detected, Microsoft invented a variant of UTF-8 (that Python 2.5 calls
@code{"utf-8-sig"}) for its Notepad program: Before any of the Unicode characters
is written to the file, a UTF-8 encoded BOM (which looks like this as a byte
sequence: @code{0xef}, @code{0xbb}, @code{0xbf}) is written. As it’s rather improbable
that any charmap encoded file starts with these byte values (which would e.g.
map to

@quotation


@display
LATIN SMALL LETTER I WITH DIAERESIS@w{ }
RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK@w{ }
INVERTED QUESTION MARK@w{ }
@end display


@end quotation

in iso-8859-1), this increases the probability that a @code{utf-8-sig} encoding can be
correctly guessed from the byte sequence. So here the BOM is not used to be able
to determine the byte order used for generating the byte sequence, but as a
signature that helps in guessing the encoding. On encoding the utf-8-sig codec
will write @code{0xef}, @code{0xbb}, @code{0xbf} as the first three bytes to the file. On
decoding @code{utf-8-sig} will skip those three bytes if they appear as the first
three bytes in the file. In UTF-8, the use of the BOM is discouraged and
should generally be avoided.

@node Standard Encodings,Python Specific Encodings,Encodings and Unicode,codecs — Codec registry and base classes
@anchor{library/codecs id3}@anchor{1cf1}@anchor{library/codecs standard-encodings}@anchor{916}
@subsubsection Standard Encodings


Python comes with a number of codecs built-in, either implemented as C functions
or with dictionaries as mapping tables. The following table lists the codecs by
name, together with a few common aliases, and the languages for which the
encoding is likely used. Neither the list of aliases nor the list of languages
is meant to be exhaustive. Notice that spelling alternatives that only differ in
case or use a hyphen instead of an underscore are also valid aliases; therefore,
e.g. @code{'utf-8'} is a valid alias for the @code{'utf_8'} codec.

@strong{CPython implementation detail:} Some common encodings can bypass the codecs lookup machinery to
improve performance. These optimization opportunities are only
recognized by CPython for a limited set of (case insensitive)
aliases: utf-8, utf8, latin-1, latin1, iso-8859-1, iso8859-1, mbcs
(Windows only), ascii, us-ascii, utf-16, utf16, utf-32, utf32, and
the same using underscores instead of dashes. Using alternative
aliases for these encodings may result in slower execution.


Changed in version 3.6: Optimization opportunity recognized for us-ascii.


Many of the character sets support the same languages. They vary in individual
characters (e.g. whether the EURO SIGN is supported or not), and in the
assignment of characters to code positions. For the European languages in
particular, the following variants typically exist:


@itemize *

@item 
an ISO 8859 codeset

@item 
a Microsoft Windows code page, which is typically derived from an 8859 codeset,
but replaces control characters with additional graphic characters

@item 
an IBM EBCDIC code page

@item 
an IBM PC code page, which is ASCII compatible
@end itemize


@multitable {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Codec

@tab

Aliases

@tab

Languages

@item

ascii

@tab

646, us-ascii

@tab

English

@item

big5

@tab

big5-tw, csbig5

@tab

Traditional Chinese

@item

big5hkscs

@tab

big5-hkscs, hkscs

@tab

Traditional Chinese

@item

cp037

@tab

IBM037, IBM039

@tab

English

@item

cp273

@tab

273, IBM273, csIBM273

@tab

German


New in version 3.4.


@item

cp424

@tab

EBCDIC-CP-HE, IBM424

@tab

Hebrew

@item

cp437

@tab

437, IBM437

@tab

English

@item

cp500

@tab

EBCDIC-CP-BE, EBCDIC-CP-CH,
IBM500

@tab

Western Europe

@item

cp720

@tab

@tab

Arabic

@item

cp737

@tab

@tab

Greek

@item

cp775

@tab

IBM775

@tab

Baltic languages

@item

cp850

@tab

850, IBM850

@tab

Western Europe

@item

cp852

@tab

852, IBM852

@tab

Central and Eastern Europe

@item

cp855

@tab

855, IBM855

@tab

Bulgarian, Byelorussian,
Macedonian, Russian, Serbian

@item

cp856

@tab

@tab

Hebrew

@item

cp857

@tab

857, IBM857

@tab

Turkish

@item

cp858

@tab

858, IBM858

@tab

Western Europe

@item

cp860

@tab

860, IBM860

@tab

Portuguese

@item

cp861

@tab

861, CP-IS, IBM861

@tab

Icelandic

@item

cp862

@tab

862, IBM862

@tab

Hebrew

@item

cp863

@tab

863, IBM863

@tab

Canadian

@item

cp864

@tab

IBM864

@tab

Arabic

@item

cp865

@tab

865, IBM865

@tab

Danish, Norwegian

@item

cp866

@tab

866, IBM866

@tab

Russian

@item

cp869

@tab

869, CP-GR, IBM869

@tab

Greek

@item

cp874

@tab

@tab

Thai

@item

cp875

@tab

@tab

Greek

@item

cp932

@tab

932, ms932, mskanji, ms-kanji

@tab

Japanese

@item

cp949

@tab

949, ms949, uhc

@tab

Korean

@item

cp950

@tab

950, ms950

@tab

Traditional Chinese

@item

cp1006

@tab

@tab

Urdu

@item

cp1026

@tab

ibm1026

@tab

Turkish

@item

cp1125

@tab

1125, ibm1125, cp866u, ruscii

@tab

Ukrainian


New in version 3.4.


@item

cp1140

@tab

ibm1140

@tab

Western Europe

@item

cp1250

@tab

windows-1250

@tab

Central and Eastern Europe

@item

cp1251

@tab

windows-1251

@tab

Bulgarian, Byelorussian,
Macedonian, Russian, Serbian

@item

cp1252

@tab

windows-1252

@tab

Western Europe

@item

cp1253

@tab

windows-1253

@tab

Greek

@item

cp1254

@tab

windows-1254

@tab

Turkish

@item

cp1255

@tab

windows-1255

@tab

Hebrew

@item

cp1256

@tab

windows-1256

@tab

Arabic

@item

cp1257

@tab

windows-1257

@tab

Baltic languages

@item

cp1258

@tab

windows-1258

@tab

Vietnamese

@item

euc_jp

@tab

eucjp, ujis, u-jis

@tab

Japanese

@item

euc_jis_2004

@tab

jisx0213, eucjis2004

@tab

Japanese

@item

euc_jisx0213

@tab

eucjisx0213

@tab

Japanese

@item

euc_kr

@tab

euckr, korean, ksc5601,
ks_c-5601, ks_c-5601-1987,
ksx1001, ks_x-1001

@tab

Korean

@item

gb2312

@tab

chinese, csiso58gb231280,
euc-cn, euccn, eucgb2312-cn,
gb2312-1980, gb2312-80,
iso-ir-58

@tab

Simplified Chinese

@item

gbk

@tab

936, cp936, ms936

@tab

Unified Chinese

@item

gb18030

@tab

gb18030-2000

@tab

Unified Chinese

@item

hz

@tab

hzgb, hz-gb, hz-gb-2312

@tab

Simplified Chinese

@item

iso2022_jp

@tab

csiso2022jp, iso2022jp,
iso-2022-jp

@tab

Japanese

@item

iso2022_jp_1

@tab

iso2022jp-1, iso-2022-jp-1

@tab

Japanese

@item

iso2022_jp_2

@tab

iso2022jp-2, iso-2022-jp-2

@tab

Japanese, Korean, Simplified
Chinese, Western Europe, Greek

@item

iso2022_jp_2004

@tab

iso2022jp-2004,
iso-2022-jp-2004

@tab

Japanese

@item

iso2022_jp_3

@tab

iso2022jp-3, iso-2022-jp-3

@tab

Japanese

@item

iso2022_jp_ext

@tab

iso2022jp-ext, iso-2022-jp-ext

@tab

Japanese

@item

iso2022_kr

@tab

csiso2022kr, iso2022kr,
iso-2022-kr

@tab

Korean

@item

latin_1

@tab

iso-8859-1, iso8859-1, 8859,
cp819, latin, latin1, L1

@tab

Western Europe

@item

iso8859_2

@tab

iso-8859-2, latin2, L2

@tab

Central and Eastern Europe

@item

iso8859_3

@tab

iso-8859-3, latin3, L3

@tab

Esperanto, Maltese

@item

iso8859_4

@tab

iso-8859-4, latin4, L4

@tab

Baltic languages

@item

iso8859_5

@tab

iso-8859-5, cyrillic

@tab

Bulgarian, Byelorussian,
Macedonian, Russian, Serbian

@item

iso8859_6

@tab

iso-8859-6, arabic

@tab

Arabic

@item

iso8859_7

@tab

iso-8859-7, greek, greek8

@tab

Greek

@item

iso8859_8

@tab

iso-8859-8, hebrew

@tab

Hebrew

@item

iso8859_9

@tab

iso-8859-9, latin5, L5

@tab

Turkish

@item

iso8859_10

@tab

iso-8859-10, latin6, L6

@tab

Nordic languages

@item

iso8859_11

@tab

iso-8859-11, thai

@tab

Thai languages

@item

iso8859_13

@tab

iso-8859-13, latin7, L7

@tab

Baltic languages

@item

iso8859_14

@tab

iso-8859-14, latin8, L8

@tab

Celtic languages

@item

iso8859_15

@tab

iso-8859-15, latin9, L9

@tab

Western Europe

@item

iso8859_16

@tab

iso-8859-16, latin10, L10

@tab

South-Eastern Europe

@item

johab

@tab

cp1361, ms1361

@tab

Korean

@item

koi8_r

@tab

@tab

Russian

@item

koi8_t

@tab

@tab

Tajik


New in version 3.5.


@item

koi8_u

@tab

@tab

Ukrainian

@item

kz1048

@tab

kz_1048, strk1048_2002, rk1048

@tab

Kazakh


New in version 3.5.


@item

mac_cyrillic

@tab

maccyrillic

@tab

Bulgarian, Byelorussian,
Macedonian, Russian, Serbian

@item

mac_greek

@tab

macgreek

@tab

Greek

@item

mac_iceland

@tab

maciceland

@tab

Icelandic

@item

mac_latin2

@tab

maclatin2, maccentraleurope,
mac_centeuro

@tab

Central and Eastern Europe

@item

mac_roman

@tab

macroman, macintosh

@tab

Western Europe

@item

mac_turkish

@tab

macturkish

@tab

Turkish

@item

ptcp154

@tab

csptcp154, pt154, cp154,
cyrillic-asian

@tab

Kazakh

@item

shift_jis

@tab

csshiftjis, shiftjis, sjis,
s_jis

@tab

Japanese

@item

shift_jis_2004

@tab

shiftjis2004, sjis_2004,
sjis2004

@tab

Japanese

@item

shift_jisx0213

@tab

shiftjisx0213, sjisx0213,
s_jisx0213

@tab

Japanese

@item

utf_32

@tab

U32, utf32

@tab

all languages

@item

utf_32_be

@tab

UTF-32BE

@tab

all languages

@item

utf_32_le

@tab

UTF-32LE

@tab

all languages

@item

utf_16

@tab

U16, utf16

@tab

all languages

@item

utf_16_be

@tab

UTF-16BE

@tab

all languages

@item

utf_16_le

@tab

UTF-16LE

@tab

all languages

@item

utf_7

@tab

U7, unicode-1-1-utf-7

@tab

all languages

@item

utf_8

@tab

U8, UTF, utf8, cp65001

@tab

all languages

@item

utf_8_sig

@tab

@tab

all languages

@end multitable



Changed in version 3.4: The utf-16* and utf-32* encoders no longer allow surrogate code points
(@code{U+D800}–@code{U+DFFF}) to be encoded.
The utf-32* decoders no longer decode
byte sequences that correspond to surrogate code points.



Changed in version 3.8: @code{cp65001} is now an alias to @code{utf_8}.


@node Python Specific Encodings,encodings idna — Internationalized Domain Names in Applications,Standard Encodings,codecs — Codec registry and base classes
@anchor{library/codecs python-specific-encodings}@anchor{1cf2}
@subsubsection Python Specific Encodings


A number of predefined codecs are specific to Python, so their codec names have
no meaning outside Python. These are listed in the tables below based on the
expected input and output types (note that while text encodings are the most
common use case for codecs, the underlying codec infrastructure supports
arbitrary data transforms rather than just text encodings). For asymmetric
codecs, the stated meaning describes the encoding direction.

@menu
* Text Encodings:: 
* Binary Transforms:: 
* Text Transforms:: 

@end menu

@node Text Encodings,Binary Transforms,,Python Specific Encodings
@anchor{library/codecs text-encodings}@anchor{1cf3}
@subsubsection Text Encodings


The following codecs provide @ref{2ef,,str} to @ref{2f0,,bytes} encoding and
@ref{87e,,bytes-like object} to @ref{2ef,,str} decoding, similar to the Unicode text
encodings.


@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Codec

@tab

Aliases

@tab

Meaning

@item

idna

@tab

@tab

Implement 
@geindex RFC; RFC 3490
RFC 3490@footnote{https://tools.ietf.org/html/rfc3490.html},
see also
@ref{72,,encodings.idna}.
Only @code{errors='strict'}
is supported.

@item

mbcs

@tab

ansi,
dbcs

@tab

Windows only: Encode the
operand according to the
ANSI codepage (CP_ACP).

@item

oem

@tab

@tab

Windows only: Encode the
operand according to the
OEM codepage (CP_OEMCP).


New in version 3.6.


@item

palmos

@tab

@tab

Encoding of PalmOS 3.5.

@item

punycode

@tab

@tab

Implement 
@geindex RFC; RFC 3492
RFC 3492@footnote{https://tools.ietf.org/html/rfc3492.html}.
Stateful codecs are not
supported.

@item

raw_unicode_escape

@tab

@tab

Latin-1 encoding with
@code{\uXXXX} and
@code{\UXXXXXXXX} for other
code points. Existing
backslashes are not
escaped in any way.
It is used in the Python
pickle protocol.

@item

undefined

@tab

@tab

Raise an exception for
all conversions, even
empty strings. The error
handler is ignored.

@item

unicode_escape

@tab

@tab

Encoding suitable as the
contents of a Unicode
literal in ASCII-encoded
Python source code,
except that quotes are
not escaped. Decode
from Latin-1 source code.
Beware that Python source
code actually uses UTF-8
by default.

@end multitable



Changed in version 3.8: “unicode_internal” codec is removed.


@node Binary Transforms,Text Transforms,Text Encodings,Python Specific Encodings
@anchor{library/codecs binary-transforms}@anchor{a78}@anchor{library/codecs id4}@anchor{1cf4}
@subsubsection Binary Transforms


The following codecs provide binary transforms: @ref{87e,,bytes-like object}
to @ref{2f0,,bytes} mappings. They are not supported by @ref{3bb,,bytes.decode()}
(which only produces @ref{2ef,,str} output).


@multitable {xxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Codec

@tab

Aliases

@tab

Meaning

@tab

Encoder / decoder

@item

base64_codec @footnote{@w{(1)} 
In addition to @ref{87e,,bytes-like objects},
@code{'base64_codec'} also accepts ASCII-only instances of @ref{2ef,,str} for
decoding
}

@tab

base64, base_64

@tab

Convert the operand to
multiline MIME base64 (the
result always includes a
trailing @code{'\n'}).


Changed in version 3.4: accepts any
@ref{87e,,bytes-like object}
as input for encoding and
decoding


@tab

@ref{453,,base64.encodebytes()} /
@ref{454,,base64.decodebytes()}

@item

bz2_codec

@tab

bz2

@tab

Compress the operand using
bz2.

@tab

@ref{1cf5,,bz2.compress()} /
@ref{c27,,bz2.decompress()}

@item

hex_codec

@tab

hex

@tab

Convert the operand to
hexadecimal
representation, with two
digits per byte.

@tab

@ref{1cf6,,binascii.b2a_hex()} /
@ref{1cf7,,binascii.a2b_hex()}

@item

quopri_codec

@tab

quopri,
quotedprintable,
quoted_printable

@tab

Convert the operand to MIME
quoted printable.

@tab

@ref{1cf8,,quopri.encode()} with
@code{quotetabs=True} /
@ref{1cf9,,quopri.decode()}

@item

uu_codec

@tab

uu

@tab

Convert the operand using
uuencode.

@tab

@ref{6d6,,uu.encode()} /
@ref{1cfa,,uu.decode()}

@item

zlib_codec

@tab

zip, zlib

@tab

Compress the operand using
gzip.

@tab

@ref{859,,zlib.compress()} /
@ref{85a,,zlib.decompress()}

@end multitable



New in version 3.2: Restoration of the binary transforms.



Changed in version 3.4: Restoration of the aliases for the binary transforms.


@node Text Transforms,,Binary Transforms,Python Specific Encodings
@anchor{library/codecs id6}@anchor{1cfb}@anchor{library/codecs text-transforms}@anchor{a79}
@subsubsection Text Transforms


The following codec provides a text transform: a @ref{2ef,,str} to @ref{2ef,,str}
mapping. It is not supported by @ref{3ba,,str.encode()} (which only produces
@ref{2f0,,bytes} output).


@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Codec

@tab

Aliases

@tab

Meaning

@item

rot_13

@tab

rot13

@tab

Return the Caesar-cypher
encryption of the
operand.

@end multitable



New in version 3.2: Restoration of the @code{rot_13} text transform.



Changed in version 3.4: Restoration of the @code{rot13} alias.


@node encodings idna — Internationalized Domain Names in Applications,encodings mbcs — Windows ANSI codepage,Python Specific Encodings,codecs — Codec registry and base classes
@anchor{library/codecs encodings-idna-internationalized-domain-names-in-applications}@anchor{1cfc}@anchor{library/codecs module-encodings idna}@anchor{72}
@subsubsection @code{encodings.idna} — Internationalized Domain Names in Applications


@geindex module; encodings.idna

This module implements 
@geindex RFC; RFC 3490
RFC 3490@footnote{https://tools.ietf.org/html/rfc3490.html} (Internationalized Domain Names in
Applications) and 
@geindex RFC; RFC 3492
RFC 3492@footnote{https://tools.ietf.org/html/rfc3492.html} (Nameprep: A Stringprep Profile for
Internationalized Domain Names (IDN)). It builds upon the @code{punycode} encoding
and @ref{f2,,stringprep}.

If you need the IDNA 2008 standard from 
@geindex RFC; RFC 5891
RFC 5891@footnote{https://tools.ietf.org/html/rfc5891.html} and 
@geindex RFC; RFC 5895
RFC 5895@footnote{https://tools.ietf.org/html/rfc5895.html}, use the
third-party @cite{idna module <https://pypi.org/project/idna/>_}.

These RFCs together define a protocol to support non-ASCII characters in domain
names. A domain name containing non-ASCII characters (such as
@code{www.Alliancefrançaise.nu}) is converted into an ASCII-compatible encoding
(ACE, such as @code{www.xn--alliancefranaise-npb.nu}). The ACE form of the domain
name is then used in all places where arbitrary characters are not allowed by
the protocol, such as DNS queries, HTTP @code{Host} fields, and so
on. This conversion is carried out in the application; if possible invisible to
the user: The application should transparently convert Unicode domain labels to
IDNA on the wire, and convert back ACE labels to Unicode before presenting them
to the user.

Python supports this conversion in several ways:  the @code{idna} codec performs
conversion between Unicode and ACE, separating an input string into labels
based on the separator characters defined in 
@geindex RFC; RFC 3490#section-3.1
section 3.1 of RFC 3490@footnote{https://tools.ietf.org/html/rfc3490.html#section-3.1}
and converting each label to ACE as required, and conversely separating an input
byte string into labels based on the @code{.} separator and converting any ACE
labels found into unicode. Furthermore, the @ref{ea,,socket} module
transparently converts Unicode host names to ACE, so that applications need not
be concerned about converting host names themselves when they pass them to the
socket module. On top of that, modules that have host names as function
parameters, such as @ref{8f,,http.client} and @ref{7e,,ftplib}, accept Unicode host
names (@ref{8f,,http.client} then also transparently sends an IDNA hostname in the
@code{Host} field if it sends that field at all).

When receiving host names from the wire (such as in reverse name lookup), no
automatic conversion to Unicode is performed: applications wishing to present
such host names to the user should decode them to Unicode.

The module @ref{72,,encodings.idna} also implements the nameprep procedure, which
performs certain normalizations on host names, to achieve case-insensitivity of
international domain names, and to unify similar characters. The nameprep
functions can be used directly if desired.

@geindex nameprep() (in module encodings.idna)
@anchor{library/codecs encodings idna nameprep}@anchor{1cfd}
@deffn {Function} encodings.idna.nameprep (label)

Return the nameprepped version of @emph{label}. The implementation currently assumes
query strings, so @code{AllowUnassigned} is true.
@end deffn

@geindex ToASCII() (in module encodings.idna)
@anchor{library/codecs encodings idna ToASCII}@anchor{1cfe}
@deffn {Function} encodings.idna.ToASCII (label)

Convert a label to ASCII, as specified in 
@geindex RFC; RFC 3490
RFC 3490@footnote{https://tools.ietf.org/html/rfc3490.html}. @code{UseSTD3ASCIIRules} is
assumed to be false.
@end deffn

@geindex ToUnicode() (in module encodings.idna)
@anchor{library/codecs encodings idna ToUnicode}@anchor{1cff}
@deffn {Function} encodings.idna.ToUnicode (label)

Convert a label to Unicode, as specified in 
@geindex RFC; RFC 3490
RFC 3490@footnote{https://tools.ietf.org/html/rfc3490.html}.
@end deffn

@node encodings mbcs — Windows ANSI codepage,encodings utf_8_sig — UTF-8 codec with BOM signature,encodings idna — Internationalized Domain Names in Applications,codecs — Codec registry and base classes
@anchor{library/codecs encodings-mbcs-windows-ansi-codepage}@anchor{1d00}@anchor{library/codecs module-encodings mbcs}@anchor{73}
@subsubsection @code{encodings.mbcs} — Windows ANSI codepage


@geindex module; encodings.mbcs

This module implements the ANSI codepage (CP_ACP).

@ref{1629,,Availability}: Windows only.


Changed in version 3.3: Support any error handler.



Changed in version 3.2: Before 3.2, the @emph{errors} argument was ignored; @code{'replace'} was always used
to encode, and @code{'ignore'} to decode.


@node encodings utf_8_sig — UTF-8 codec with BOM signature,,encodings mbcs — Windows ANSI codepage,codecs — Codec registry and base classes
@anchor{library/codecs encodings-utf-8-sig-utf-8-codec-with-bom-signature}@anchor{1d01}@anchor{library/codecs module-encodings utf_8_sig}@anchor{74}
@subsubsection @code{encodings.utf_8_sig} — UTF-8 codec with BOM signature


@geindex module; encodings.utf_8_sig

This module implements a variant of the UTF-8 codec. On encoding, a UTF-8 encoded
BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this
is only done once (on the first write to the byte stream). On decoding, an
optional UTF-8 encoded BOM at the start of the data will be skipped.

@node Data Types,Numeric and Mathematical Modules,Binary Data Services,The Python Standard Library
@anchor{library/datatypes doc}@anchor{1d02}@anchor{library/datatypes data-types}@anchor{1d03}@anchor{library/datatypes datatypes}@anchor{1d04}
@section Data Types


The modules described in this chapter provide a variety of specialized data
types such as dates and times, fixed-type arrays, heap queues, double-ended
queues, and enumerations.

Python also provides some built-in data types, in particular,
@ref{3b0,,dict}, @ref{43a,,list}, @ref{370,,set} and @ref{439,,frozenset}, and
@ref{3e5,,tuple}.  The @ref{2ef,,str} class is used to hold
Unicode strings, and the @ref{2f0,,bytes} and @ref{2f1,,bytearray} classes are used
to hold binary data.

The following modules are documented in this chapter:

@menu
* datetime — Basic date and time types:: 
* zoneinfo — IANA time zone support:: 
* calendar — General calendar-related functions:: 
* collections — Container datatypes:: 
* collections.abc — Abstract Base Classes for Containers: collections abc — Abstract Base Classes for Containers. 
* heapq — Heap queue algorithm:: 
* bisect — Array bisection algorithm:: 
* array — Efficient arrays of numeric values:: 
* weakref — Weak references:: 
* types — Dynamic type creation and names for built-in types:: 
* copy — Shallow and deep copy operations:: 
* pprint — Data pretty printer:: 
* reprlib — Alternate repr() implementation: reprlib — Alternate repr implementation. 
* enum — Support for enumerations:: 
* graphlib — Functionality to operate with graph-like structures:: 

@end menu

@node datetime — Basic date and time types,zoneinfo — IANA time zone support,,Data Types
@anchor{library/datetime doc}@anchor{1d05}@anchor{library/datetime datetime-basic-date-and-time-types}@anchor{1d06}@anchor{library/datetime module-datetime}@anchor{2f}
@subsection @code{datetime} — Basic date and time types


@geindex module; datetime

@strong{Source code:} Lib/datetime.py@footnote{https://github.com/python/cpython/tree/main/Lib/datetime.py}


__________________________________________________________________


@c XXX what order should the types be discussed in?

The @ref{2f,,datetime} module supplies classes for manipulating dates and times.

While date and time arithmetic is supported, the focus of the implementation is
on efficient attribute extraction for output formatting and manipulation.


@subsubheading See also



@table @asis

@item Module @ref{13,,calendar}

General calendar related functions.

@item Module @ref{10a,,time}

Time access and conversions.

@item Module @ref{14c,,zoneinfo}

Concrete time zones representing the IANA time zone database.

@item Package dateutil@footnote{https://dateutil.readthedocs.io/en/stable/}

Third-party library with expanded time zone and parsing support.
@end table


@menu
* Aware and Naive Objects:: 
* Constants:: 
* Available Types:: 
* timedelta Objects:: 
* date Objects:: 
* datetime Objects:: 
* time Objects:: 
* tzinfo Objects:: 
* timezone Objects:: 
* strftime() and strptime() Behavior: strftime and strptime Behavior. 

@end menu

@node Aware and Naive Objects,Constants,,datetime — Basic date and time types
@anchor{library/datetime aware-and-naive-objects}@anchor{1d07}@anchor{library/datetime datetime-naive-aware}@anchor{1d08}
@subsubsection Aware and Naive Objects


Date and time objects may be categorized as “aware” or “naive” depending on
whether or not they include timezone information.

With sufficient knowledge of applicable algorithmic and political time
adjustments, such as time zone and daylight saving time information,
an @strong{aware} object can locate itself relative to other aware objects.
An aware object represents a specific moment in time that is not open to
interpretation. @footnote{@w{(1)} 
If, that is, we ignore the effects of Relativity
}

A @strong{naive} object does not contain enough information to unambiguously locate
itself relative to other date/time objects. Whether a naive object represents
Coordinated Universal Time (UTC), local time, or time in some other timezone is
purely up to the program, just like it is up to the program whether a
particular number represents metres, miles, or mass. Naive objects are easy to
understand and to work with, at the cost of ignoring some aspects of reality.

For applications requiring aware objects, @ref{363,,datetime} and @ref{364,,time}
objects have an optional time zone information attribute, @code{tzinfo}, that
can be set to an instance of a subclass of the abstract @ref{3c6,,tzinfo} class.
These @ref{3c6,,tzinfo} objects capture information about the offset from UTC
time, the time zone name, and whether daylight saving time is in effect.

Only one concrete @ref{3c6,,tzinfo} class, the @ref{c51,,timezone} class, is
supplied by the @ref{2f,,datetime} module. The @ref{c51,,timezone} class can
represent simple timezones with fixed offsets from UTC, such as UTC itself or
North American EST and EDT timezones. Supporting timezones at deeper levels of
detail is up to the application. The rules for time adjustment across the
world are more political than rational, change frequently, and there is no
standard suitable for every application aside from UTC.

@node Constants,Available Types,Aware and Naive Objects,datetime — Basic date and time types
@anchor{library/datetime constants}@anchor{1d09}
@subsubsection Constants


The @ref{2f,,datetime} module exports the following constants:

@geindex MINYEAR (in module datetime)
@anchor{library/datetime datetime MINYEAR}@anchor{1d0a}
@deffn {Data} datetime.MINYEAR

The smallest year number allowed in a @ref{3e2,,date} or @ref{363,,datetime} object.
@ref{1d0a,,MINYEAR} is @code{1}.
@end deffn

@geindex MAXYEAR (in module datetime)
@anchor{library/datetime datetime MAXYEAR}@anchor{1d0b}
@deffn {Data} datetime.MAXYEAR

The largest year number allowed in a @ref{3e2,,date} or @ref{363,,datetime} object.
@ref{1d0b,,MAXYEAR} is @code{9999}.
@end deffn

@node Available Types,timedelta Objects,Constants,datetime — Basic date and time types
@anchor{library/datetime available-types}@anchor{1d0c}
@subsubsection Available Types



@deffn {Class} datetime.date

An idealized naive date, assuming the current Gregorian calendar always was, and
always will be, in effect. Attributes: @ref{1d0d,,year}, @ref{1d0e,,month}, and
@ref{1d0f,,day}.
@end deffn


@deffn {Class} datetime.time

An idealized time, independent of any particular day, assuming that every day
has exactly 24*60*60 seconds.  (There is no notion of “leap seconds” here.)
Attributes: @ref{1d10,,hour}, @ref{1d11,,minute}, @ref{1d12,,second}, @ref{1d13,,microsecond},
and @ref{1d14,,tzinfo}.
@end deffn


@deffn {Class} datetime.datetime

A combination of a date and a time. Attributes: @ref{1d15,,year}, @ref{1d16,,month},
@ref{1d17,,day}, @ref{1d18,,hour}, @ref{1d19,,minute}, @ref{1d1a,,second}, @ref{1d1b,,microsecond},
and @ref{1d1c,,tzinfo}.
@end deffn


@deffn {Class} datetime.timedelta

A duration expressing the difference between two @ref{3e2,,date}, @ref{364,,time},
or @ref{363,,datetime} instances to microsecond resolution.
@end deffn


@deffn {Class} datetime.tzinfo

An abstract base class for time zone information objects. These are used by the
@ref{363,,datetime} and @ref{364,,time} classes to provide a customizable notion of
time adjustment (for example, to account for time zone and/or daylight saving
time).
@end deffn


@deffn {Class} datetime.timezone

A class that implements the @ref{3c6,,tzinfo} abstract base class as a
fixed offset from the UTC.


New in version 3.2.

@end deffn

Objects of these types are immutable.

Subclass relationships:

@example
object
    timedelta
    tzinfo
        timezone
    time
    date
        datetime
@end example

@menu
* Common Properties:: 
* Determining if an Object is Aware or Naive:: 

@end menu

@node Common Properties,Determining if an Object is Aware or Naive,,Available Types
@anchor{library/datetime common-properties}@anchor{1d1d}
@subsubsection Common Properties


The @ref{3e2,,date}, @ref{363,,datetime}, @ref{364,,time}, and @ref{c51,,timezone} types
share these common features:


@itemize -

@item 
Objects of these types are immutable.

@item 
Objects of these types are hashable, meaning that they can be used as
dictionary keys.

@item 
Objects of these types support efficient pickling via the @ref{c5,,pickle} module.
@end itemize

@node Determining if an Object is Aware or Naive,,Common Properties,Available Types
@anchor{library/datetime determining-if-an-object-is-aware-or-naive}@anchor{1d1e}
@subsubsection Determining if an Object is Aware or Naive


Objects of the @ref{3e2,,date} type are always naive.

An object of type @ref{364,,time} or @ref{363,,datetime} may be aware or naive.

A @ref{363,,datetime} object @emph{d} is aware if both of the following hold:


@enumerate 

@item 
@code{d.tzinfo} is not @code{None}

@item 
@code{d.tzinfo.utcoffset(d)} does not return @code{None}
@end enumerate

Otherwise, @emph{d} is naive.

A @ref{364,,time} object @emph{t} is aware if both of the following hold:


@enumerate 

@item 
@code{t.tzinfo} is not @code{None}

@item 
@code{t.tzinfo.utcoffset(None)} does not return @code{None}.
@end enumerate

Otherwise, @emph{t} is naive.

The distinction between aware and naive doesn’t apply to @ref{4c8,,timedelta}
objects.

@node timedelta Objects,date Objects,Available Types,datetime — Basic date and time types
@anchor{library/datetime datetime-timedelta}@anchor{1d1f}@anchor{library/datetime timedelta-objects}@anchor{1d20}
@subsubsection @code{timedelta} Objects


A @ref{4c8,,timedelta} object represents a duration, the difference between two
dates or times.

@geindex timedelta (class in datetime)
@anchor{library/datetime datetime timedelta}@anchor{4c8}
@deffn {Class} datetime.timedelta (days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0)

All arguments are optional and default to @code{0}. Arguments may be integers
or floats, and may be positive or negative.

Only @emph{days}, @emph{seconds} and @emph{microseconds} are stored internally.
Arguments are converted to those units:


@itemize *

@item 
A millisecond is converted to 1000 microseconds.

@item 
A minute is converted to 60 seconds.

@item 
An hour is converted to 3600 seconds.

@item 
A week is converted to 7 days.
@end itemize

and days, seconds and microseconds are then normalized so that the
representation is unique, with


@itemize *

@item 
@code{0 <= microseconds < 1000000}

@item 
@code{0 <= seconds < 3600*24} (the number of seconds in one day)

@item 
@code{-999999999 <= days <= 999999999}
@end itemize

The following example illustrates how any arguments besides
@emph{days}, @emph{seconds} and @emph{microseconds} are “merged” and normalized into those
three resulting attributes:

@example
>>> from datetime import timedelta
>>> delta = timedelta(
...     days=50,
...     seconds=27,
...     microseconds=10,
...     milliseconds=29000,
...     minutes=5,
...     hours=8,
...     weeks=2
... )
>>> # Only days, seconds, and microseconds remain
>>> delta
datetime.timedelta(days=64, seconds=29156, microseconds=10)
@end example

If any argument is a float and there are fractional microseconds,
the fractional microseconds left over from all arguments are
combined and their sum is rounded to the nearest microsecond using
round-half-to-even tiebreaker. If no argument is a float, the
conversion and normalization processes are exact (no information is
lost).

If the normalized value of days lies outside the indicated range,
@ref{347,,OverflowError} is raised.

Note that normalization of negative values may be surprising at first. For
example:

@example
>>> from datetime import timedelta
>>> d = timedelta(microseconds=-1)
>>> (d.days, d.seconds, d.microseconds)
(-1, 86399, 999999)
@end example
@end deffn

Class attributes:

@geindex min (datetime.timedelta attribute)
@anchor{library/datetime datetime timedelta min}@anchor{1d21}
@deffn {Attribute} timedelta.min

The most negative @ref{4c8,,timedelta} object, @code{timedelta(-999999999)}.
@end deffn

@geindex max (datetime.timedelta attribute)
@anchor{library/datetime datetime timedelta max}@anchor{1d22}
@deffn {Attribute} timedelta.max

The most positive @ref{4c8,,timedelta} object, @code{timedelta(days=999999999,
hours=23, minutes=59, seconds=59, microseconds=999999)}.
@end deffn

@geindex resolution (datetime.timedelta attribute)
@anchor{library/datetime datetime timedelta resolution}@anchor{1d23}
@deffn {Attribute} timedelta.resolution

The smallest possible difference between non-equal @ref{4c8,,timedelta} objects,
@code{timedelta(microseconds=1)}.
@end deffn

Note that, because of normalization, @code{timedelta.max} > @code{-timedelta.min}.
@code{-timedelta.max} is not representable as a @ref{4c8,,timedelta} object.

Instance attributes (read-only):


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Attribute

@tab

Value

@item

@code{days}

@tab

Between -999999999 and 999999999 inclusive

@item

@code{seconds}

@tab

Between 0 and 86399 inclusive

@item

@code{microseconds}

@tab

Between 0 and 999999 inclusive

@end multitable


Supported operations:

@c XXX this table is too wide!


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Operation

@tab

Result

@item

@code{t1 = t2 + t3}

@tab

Sum of @emph{t2} and @emph{t3}. Afterwards @emph{t1}-@emph{t2} ==
@emph{t3} and @emph{t1}-@emph{t3} == @emph{t2} are true. (1)

@item

@code{t1 = t2 - t3}

@tab

Difference of @emph{t2} and @emph{t3}. Afterwards @emph{t1}
== @emph{t2} - @emph{t3} and @emph{t2} == @emph{t1} + @emph{t3} are
true. (1)(6)

@item

@code{t1 = t2 * i or t1 = i * t2}

@tab

Delta multiplied by an integer.
Afterwards @emph{t1} // i == @emph{t2} is true,
provided @code{i != 0}.

@item

@tab

In general, @emph{t1} * i == @emph{t1} * (i-1) + @emph{t1}
is true. (1)

@item

@code{t1 = t2 * f or t1 = f * t2}

@tab

Delta multiplied by a float. The result is
rounded to the nearest multiple of
timedelta.resolution using round-half-to-even.

@item

@code{f = t2 / t3}

@tab

Division (3) of overall duration @emph{t2} by
interval unit @emph{t3}. Returns a @ref{256,,float}
object.

@item

@code{t1 = t2 / f or t1 = t2 / i}

@tab

Delta divided by a float or an int. The result
is rounded to the nearest multiple of
timedelta.resolution using round-half-to-even.

@item

@code{t1 = t2 // i} or
@code{t1 = t2 // t3}

@tab

The floor is computed and the remainder (if
any) is thrown away. In the second case, an
integer is returned. (3)

@item

@code{t1 = t2 % t3}

@tab

The remainder is computed as a
@ref{4c8,,timedelta} object. (3)

@item

@code{q, r = divmod(t1, t2)}

@tab

Computes the quotient and the remainder:
@code{q = t1 // t2} (3) and @code{r = t1 % t2}.
q is an integer and r is a @ref{4c8,,timedelta}
object.

@item

@code{+t1}

@tab

Returns a @ref{4c8,,timedelta} object with the
same value. (2)

@item

@code{-t1}

@tab

equivalent to
@ref{4c8,,timedelta}(-@emph{t1.days},
-@emph{t1.seconds}, -@emph{t1.microseconds}),
and to @emph{t1}* -1. (1)(4)

@item

@code{abs(t)}

@tab

equivalent to +@emph{t} when @code{t.days >= 0},
and to -@emph{t} when @code{t.days < 0}. (2)

@item

@code{str(t)}

@tab

Returns a string in the form
@code{[D day[s], ][H]H:MM:SS[.UUUUUU]}, where D
is negative for negative @code{t}. (5)

@item

@code{repr(t)}

@tab

Returns a string representation of the
@ref{4c8,,timedelta} object as a constructor
call with canonical attribute values.

@end multitable


Notes:


@enumerate 

@item 
This is exact but may overflow.

@item 
This is exact and cannot overflow.

@item 
Division by 0 raises @ref{43c,,ZeroDivisionError}.

@item 
-@emph{timedelta.max} is not representable as a @ref{4c8,,timedelta} object.

@item 
String representations of @ref{4c8,,timedelta} objects are normalized
similarly to their internal representation. This leads to somewhat
unusual results for negative timedeltas. For example:

@example
>>> timedelta(hours=-5)
datetime.timedelta(days=-1, seconds=68400)
>>> print(_)
-1 day, 19:00:00
@end example

@item 
The expression @code{t2 - t3} will always be equal to the expression @code{t2 + (-t3)} except
when t3 is equal to @code{timedelta.max}; in that case the former will produce a result
while the latter will overflow.
@end enumerate

In addition to the operations listed above, @ref{4c8,,timedelta} objects support
certain additions and subtractions with @ref{3e2,,date} and @ref{363,,datetime}
objects (see below).


Changed in version 3.2: Floor division and true division of a @ref{4c8,,timedelta} object by another
@ref{4c8,,timedelta} object are now supported, as are remainder operations and
the @ref{4a8,,divmod()} function. True division and multiplication of a
@ref{4c8,,timedelta} object by a @ref{256,,float} object are now supported.


Comparisons of @ref{4c8,,timedelta} objects are supported, with some caveats.

The comparisons @code{==} or @code{!=} @emph{always} return a @ref{2f7,,bool}, no matter
the type of the compared object:

@example
>>> from datetime import timedelta
>>> delta1 = timedelta(seconds=57)
>>> delta2 = timedelta(hours=25, seconds=2)
>>> delta2 != delta1
True
>>> delta2 == 5
False
@end example

For all other comparisons (such as @code{<} and @code{>}), when a @ref{4c8,,timedelta}
object is compared to an object of a different type, @ref{162,,TypeError}
is raised:

@example
>>> delta2 > delta1
True
>>> delta2 > 5
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '>' not supported between instances of 'datetime.timedelta' and 'int'
@end example

In Boolean contexts, a @ref{4c8,,timedelta} object is
considered to be true if and only if it isn’t equal to @code{timedelta(0)}.

Instance methods:

@geindex total_seconds() (datetime.timedelta method)
@anchor{library/datetime datetime timedelta total_seconds}@anchor{ea1}
@deffn {Method} timedelta.total_seconds ()

Return the total number of seconds contained in the duration. Equivalent to
@code{td / timedelta(seconds=1)}. For interval units other than seconds, use the
division form directly (e.g. @code{td / timedelta(microseconds=1)}).

Note that for very large time intervals (greater than 270 years on
most platforms) this method will lose microsecond accuracy.


New in version 3.2.

@end deffn

@menu
* Examples of usage; timedelta: Examples of usage timedelta. 

@end menu

@node Examples of usage timedelta,,,timedelta Objects
@anchor{library/datetime examples-of-usage-timedelta}@anchor{1d24}
@subsubsection Examples of usage: @code{timedelta}


An additional example of normalization:

@example
>>> # Components of another_year add up to exactly 365 days
>>> from datetime import timedelta
>>> year = timedelta(days=365)
>>> another_year = timedelta(weeks=40, days=84, hours=23,
...                          minutes=50, seconds=600)
>>> year == another_year
True
>>> year.total_seconds()
31536000.0
@end example

Examples of @ref{4c8,,timedelta} arithmetic:

@example
>>> from datetime import timedelta
>>> year = timedelta(days=365)
>>> ten_years = 10 * year
>>> ten_years
datetime.timedelta(days=3650)
>>> ten_years.days // 365
10
>>> nine_years = ten_years - year
>>> nine_years
datetime.timedelta(days=3285)
>>> three_years = nine_years // 3
>>> three_years, three_years.days // 365
(datetime.timedelta(days=1095), 3)
@end example

@node date Objects,datetime Objects,timedelta Objects,datetime — Basic date and time types
@anchor{library/datetime date-objects}@anchor{1d25}@anchor{library/datetime datetime-date}@anchor{1d26}
@subsubsection @code{date} Objects


A @ref{3e2,,date} object represents a date (year, month and day) in an idealized
calendar, the current Gregorian calendar indefinitely extended in both
directions.

January 1 of year 1 is called day number 1, January 2 of year 1 is
called day number 2, and so on. @footnote{@w{(2)} 
This matches the definition of the “proleptic Gregorian” calendar in
Dershowitz and Reingold’s book @emph{Calendrical Calculations},
where it’s the base calendar for all computations. See the book for
algorithms for converting between proleptic Gregorian ordinals and
many other calendar systems.
}

@geindex date (class in datetime)
@anchor{library/datetime datetime date}@anchor{3e2}
@deffn {Class} datetime.date (year, month, day)

All arguments are required. Arguments must be integers, in the following
ranges:


@itemize *

@item 
@code{MINYEAR <= year <= MAXYEAR}

@item 
@code{1 <= month <= 12}

@item 
@code{1 <= day <= number of days in the given month and year}
@end itemize

If an argument outside those ranges is given, @ref{17c,,ValueError} is raised.
@end deffn

Other constructors, all class methods:

@geindex today() (datetime.date class method)
@anchor{library/datetime datetime date today}@anchor{1d27}
@deffn {Method} classmethod date.today ()

Return the current local date.

This is equivalent to @code{date.fromtimestamp(time.time())}.
@end deffn

@geindex fromtimestamp() (datetime.date class method)
@anchor{library/datetime datetime date fromtimestamp}@anchor{1d28}
@deffn {Method} classmethod date.fromtimestamp (timestamp)

Return the local date corresponding to the POSIX timestamp, such as is
returned by @ref{5fc,,time.time()}.

This may raise @ref{347,,OverflowError}, if the timestamp is out
of the range of values supported by the platform C @code{localtime()}
function, and @ref{2a1,,OSError} on @code{localtime()} failure.
It’s common for this to be restricted to years from 1970 through 2038. Note
that on non-POSIX systems that include leap seconds in their notion of a
timestamp, leap seconds are ignored by @ref{1d28,,fromtimestamp()}.


Changed in version 3.3: Raise @ref{347,,OverflowError} instead of @ref{17c,,ValueError} if the timestamp
is out of the range of values supported by the platform C
@code{localtime()} function. Raise @ref{2a1,,OSError} instead of
@ref{17c,,ValueError} on @code{localtime()} failure.

@end deffn

@geindex fromordinal() (datetime.date class method)
@anchor{library/datetime datetime date fromordinal}@anchor{1d29}
@deffn {Method} classmethod date.fromordinal (ordinal)

Return the date corresponding to the proleptic Gregorian ordinal, where
January 1 of year 1 has ordinal 1.

@ref{17c,,ValueError} is raised unless @code{1 <= ordinal <=
date.max.toordinal()}. For any date @emph{d},
@code{date.fromordinal(d.toordinal()) == d}.
@end deffn

@geindex fromisoformat() (datetime.date class method)
@anchor{library/datetime datetime date fromisoformat}@anchor{135a}
@deffn {Method} classmethod date.fromisoformat (date_string)

Return a @ref{3e2,,date} corresponding to a @emph{date_string} given in the format
@code{YYYY-MM-DD}:

@example
>>> from datetime import date
>>> date.fromisoformat('2019-12-04')
datetime.date(2019, 12, 4)
@end example

This is the inverse of @ref{1d2a,,date.isoformat()}. It only supports the format
@code{YYYY-MM-DD}.


New in version 3.7.

@end deffn

@geindex fromisocalendar() (datetime.date class method)
@anchor{library/datetime datetime date fromisocalendar}@anchor{4f0}
@deffn {Method} classmethod date.fromisocalendar (year, week, day)

Return a @ref{3e2,,date} corresponding to the ISO calendar date specified by
year, week and day. This is the inverse of the function @ref{3e1,,date.isocalendar()}.


New in version 3.8.

@end deffn

Class attributes:

@geindex min (datetime.date attribute)
@anchor{library/datetime datetime date min}@anchor{1d2b}
@deffn {Attribute} date.min

The earliest representable date, @code{date(MINYEAR, 1, 1)}.
@end deffn

@geindex max (datetime.date attribute)
@anchor{library/datetime datetime date max}@anchor{1d2c}
@deffn {Attribute} date.max

The latest representable date, @code{date(MAXYEAR, 12, 31)}.
@end deffn

@geindex resolution (datetime.date attribute)
@anchor{library/datetime datetime date resolution}@anchor{1d2d}
@deffn {Attribute} date.resolution

The smallest possible difference between non-equal date objects,
@code{timedelta(days=1)}.
@end deffn

Instance attributes (read-only):

@geindex year (datetime.date attribute)
@anchor{library/datetime datetime date year}@anchor{1d0d}
@deffn {Attribute} date.year

Between @ref{1d0a,,MINYEAR} and @ref{1d0b,,MAXYEAR} inclusive.
@end deffn

@geindex month (datetime.date attribute)
@anchor{library/datetime datetime date month}@anchor{1d0e}
@deffn {Attribute} date.month

Between 1 and 12 inclusive.
@end deffn

@geindex day (datetime.date attribute)
@anchor{library/datetime datetime date day}@anchor{1d0f}
@deffn {Attribute} date.day

Between 1 and the number of days in the given month of the given year.
@end deffn

Supported operations:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Operation

@tab

Result

@item

@code{date2 = date1 + timedelta}

@tab

@emph{date2} is @code{timedelta.days} days removed
from @emph{date1}. (1)

@item

@code{date2 = date1 - timedelta}

@tab

Computes @emph{date2} such that @code{date2 +
timedelta == date1}. (2)

@item

@code{timedelta = date1 - date2}

@tab

(3)

@item

@code{date1 < date2}

@tab

@emph{date1} is considered less than @emph{date2} when
@emph{date1} precedes @emph{date2} in time. (4)

@end multitable


Notes:


@enumerate 

@item 
@emph{date2} is moved forward in time if @code{timedelta.days > 0}, or backward if
@code{timedelta.days < 0}. Afterward @code{date2 - date1 == timedelta.days}.
@code{timedelta.seconds} and @code{timedelta.microseconds} are ignored.
@ref{347,,OverflowError} is raised if @code{date2.year} would be smaller than
@ref{1d0a,,MINYEAR} or larger than @ref{1d0b,,MAXYEAR}.

@item 
@code{timedelta.seconds} and @code{timedelta.microseconds} are ignored.

@item 
This is exact, and cannot overflow. timedelta.seconds and
timedelta.microseconds are 0, and date2 + timedelta == date1 after.

@item 
In other words, @code{date1 < date2} if and only if @code{date1.toordinal() <
date2.toordinal()}. Date comparison raises @ref{162,,TypeError} if
the other comparand isn’t also a @ref{3e2,,date} object. However,
@code{NotImplemented} is returned instead if the other comparand has a
@code{timetuple()} attribute. This hook gives other kinds of date objects a
chance at implementing mixed-type comparison. If not, when a @ref{3e2,,date}
object is compared to an object of a different type, @ref{162,,TypeError} is raised
unless the comparison is @code{==} or @code{!=}. The latter cases return
@ref{659,,False} or @ref{748,,True}, respectively.
@end enumerate

In Boolean contexts, all @ref{3e2,,date} objects are considered to be true.

Instance methods:

@geindex replace() (datetime.date method)
@anchor{library/datetime datetime date replace}@anchor{1d2e}
@deffn {Method} date.replace (year=self.year, month=self.month, day=self.day)

Return a date with the same value, except for those parameters given new
values by whichever keyword arguments are specified.

Example:

@example
>>> from datetime import date
>>> d = date(2002, 12, 31)
>>> d.replace(day=26)
datetime.date(2002, 12, 26)
@end example
@end deffn

@geindex timetuple() (datetime.date method)
@anchor{library/datetime datetime date timetuple}@anchor{1d2f}
@deffn {Method} date.timetuple ()

Return a @ref{834,,time.struct_time} such as returned by @ref{1224,,time.localtime()}.

The hours, minutes and seconds are 0, and the DST flag is -1.

@code{d.timetuple()} is equivalent to:

@example
time.struct_time((d.year, d.month, d.day, 0, 0, 0, d.weekday(), yday, -1))
@end example

where @code{yday = d.toordinal() - date(d.year, 1, 1).toordinal() + 1}
is the day number within the current year starting with @code{1} for January 1st.
@end deffn

@geindex toordinal() (datetime.date method)
@anchor{library/datetime datetime date toordinal}@anchor{1d30}
@deffn {Method} date.toordinal ()

Return the proleptic Gregorian ordinal of the date, where January 1 of year 1
has ordinal 1. For any @ref{3e2,,date} object @emph{d},
@code{date.fromordinal(d.toordinal()) == d}.
@end deffn

@geindex weekday() (datetime.date method)
@anchor{library/datetime datetime date weekday}@anchor{1d31}
@deffn {Method} date.weekday ()

Return the day of the week as an integer, where Monday is 0 and Sunday is 6.
For example, @code{date(2002, 12, 4).weekday() == 2}, a Wednesday. See also
@ref{1d32,,isoweekday()}.
@end deffn

@geindex isoweekday() (datetime.date method)
@anchor{library/datetime datetime date isoweekday}@anchor{1d32}
@deffn {Method} date.isoweekday ()

Return the day of the week as an integer, where Monday is 1 and Sunday is 7.
For example, @code{date(2002, 12, 4).isoweekday() == 3}, a Wednesday. See also
@ref{1d31,,weekday()}, @ref{3e1,,isocalendar()}.
@end deffn

@geindex isocalendar() (datetime.date method)
@anchor{library/datetime datetime date isocalendar}@anchor{3e1}
@deffn {Method} date.isocalendar ()

Return a @ref{ce9,,named tuple} object with three components: @code{year},
@code{week} and @code{weekday}.

The ISO calendar is a widely used variant of the Gregorian calendar. @footnote{@w{(3)} 
See R. H. van Gent’s @uref{https://www.staff.science.uu.nl/~gent0113/calendar/isocalendar.htm,guide to the mathematics of the ISO 8601 calendar}
for a good explanation.
}

The ISO year consists of 52 or 53 full weeks, and where a week starts on a
Monday and ends on a Sunday. The first week of an ISO year is the first
(Gregorian) calendar week of a year containing a Thursday. This is called week
number 1, and the ISO year of that Thursday is the same as its Gregorian year.

For example, 2004 begins on a Thursday, so the first week of ISO year 2004
begins on Monday, 29 Dec 2003 and ends on Sunday, 4 Jan 2004:

@example
>>> from datetime import date
>>> date(2003, 12, 29).isocalendar()
datetime.IsoCalendarDate(year=2004, week=1, weekday=1)
>>> date(2004, 1, 4).isocalendar()
datetime.IsoCalendarDate(year=2004, week=1, weekday=7)
@end example


Changed in version 3.9: Result changed from a tuple to a @ref{ce9,,named tuple}.

@end deffn

@geindex isoformat() (datetime.date method)
@anchor{library/datetime datetime date isoformat}@anchor{1d2a}
@deffn {Method} date.isoformat ()

Return a string representing the date in ISO 8601 format, @code{YYYY-MM-DD}:

@example
>>> from datetime import date
>>> date(2002, 12, 4).isoformat()
'2002-12-04'
@end example

This is the inverse of @ref{135a,,date.fromisoformat()}.
@end deffn

@geindex __str__() (datetime.date method)
@anchor{library/datetime datetime date __str__}@anchor{1d33}
@deffn {Method} date.__str__ ()

For a date @emph{d}, @code{str(d)} is equivalent to @code{d.isoformat()}.
@end deffn

@geindex ctime() (datetime.date method)
@anchor{library/datetime datetime date ctime}@anchor{1d34}
@deffn {Method} date.ctime ()

Return a string representing the date:

@example
>>> from datetime import date
>>> date(2002, 12, 4).ctime()
'Wed Dec  4 00:00:00 2002'
@end example

@code{d.ctime()} is equivalent to:

@example
time.ctime(time.mktime(d.timetuple()))
@end example

on platforms where the native C
@code{ctime()} function (which @ref{1d35,,time.ctime()} invokes, but which
@ref{1d34,,date.ctime()} does not invoke) conforms to the C standard.
@end deffn

@geindex strftime() (datetime.date method)
@anchor{library/datetime datetime date strftime}@anchor{7da}
@deffn {Method} date.strftime (format)

Return a string representing the date, controlled by an explicit format string.
Format codes referring to hours, minutes or seconds will see 0 values. For a
complete list of formatting directives, see
@ref{1d36,,strftime() and strptime() Behavior}.
@end deffn

@geindex __format__() (datetime.date method)
@anchor{library/datetime datetime date __format__}@anchor{1d37}
@deffn {Method} date.__format__ (format)

Same as @ref{7da,,date.strftime()}. This makes it possible to specify a format
string for a @ref{3e2,,date} object in @ref{4b0,,formatted string literals} and when using @ref{784,,str.format()}. For a
complete list of formatting directives, see
@ref{1d36,,strftime() and strptime() Behavior}.
@end deffn

@menu
* Examples of Usage; date: Examples of Usage date. 

@end menu

@node Examples of Usage date,,,date Objects
@anchor{library/datetime examples-of-usage-date}@anchor{1d38}
@subsubsection Examples of Usage: @code{date}


Example of counting days to an event:

@example
>>> import time
>>> from datetime import date
>>> today = date.today()
>>> today
datetime.date(2007, 12, 5)
>>> today == date.fromtimestamp(time.time())
True
>>> my_birthday = date(today.year, 6, 24)
>>> if my_birthday < today:
...     my_birthday = my_birthday.replace(year=today.year + 1)
>>> my_birthday
datetime.date(2008, 6, 24)
>>> time_to_birthday = abs(my_birthday - today)
>>> time_to_birthday.days
202
@end example

More examples of working with @ref{3e2,,date}:

@example
>>> from datetime import date
>>> d = date.fromordinal(730920) # 730920th day after 1. 1. 0001
>>> d
datetime.date(2002, 3, 11)

>>> # Methods related to formatting string output
>>> d.isoformat()
'2002-03-11'
>>> d.strftime("%d/%m/%y")
'11/03/02'
>>> d.strftime("%A %d. %B %Y")
'Monday 11. March 2002'
>>> d.ctime()
'Mon Mar 11 00:00:00 2002'
>>> 'The @{1@} is @{0:%d@}, the @{2@} is @{0:%B@}.'.format(d, "day", "month")
'The day is 11, the month is March.'

>>> # Methods for to extracting 'components' under different calendars
>>> t = d.timetuple()
>>> for i in t:     
...     print(i)
2002                # year
3                   # month
11                  # day
0
0
0
0                   # weekday (0 = Monday)
70                  # 70th day in the year
-1
>>> ic = d.isocalendar()
>>> for i in ic:    
...     print(i)
2002                # ISO year
11                  # ISO week number
1                   # ISO day number ( 1 = Monday )

>>> # A date object is immutable; all operations produce a new object
>>> d.replace(year=2005)
datetime.date(2005, 3, 11)
@end example

@node datetime Objects,time Objects,date Objects,datetime — Basic date and time types
@anchor{library/datetime datetime-datetime}@anchor{1d39}@anchor{library/datetime datetime-objects}@anchor{1d3a}
@subsubsection @code{datetime} Objects


A @ref{363,,datetime} object is a single object containing all the information
from a @ref{3e2,,date} object and a @ref{364,,time} object.

Like a @ref{3e2,,date} object, @ref{363,,datetime} assumes the current Gregorian
calendar extended in both directions; like a @ref{364,,time} object,
@ref{363,,datetime} assumes there are exactly 3600*24 seconds in every day.

Constructor:

@geindex datetime (class in datetime)
@anchor{library/datetime datetime datetime}@anchor{363}
@deffn {Class} datetime.datetime (year, month, day, hour=0, minute=0, second=0, microsecond=0, tzinfo=None, *, fold=0)

The @emph{year}, @emph{month} and @emph{day} arguments are required. @emph{tzinfo} may be @code{None}, or an
instance of a @ref{3c6,,tzinfo} subclass. The remaining arguments must be integers
in the following ranges:


@itemize *

@item 
@code{MINYEAR <= year <= MAXYEAR},

@item 
@code{1 <= month <= 12},

@item 
@code{1 <= day <= number of days in the given month and year},

@item 
@code{0 <= hour < 24},

@item 
@code{0 <= minute < 60},

@item 
@code{0 <= second < 60},

@item 
@code{0 <= microsecond < 1000000},

@item 
@code{fold in [0, 1]}.
@end itemize

If an argument outside those ranges is given, @ref{17c,,ValueError} is raised.


New in version 3.6: Added the @code{fold} argument.

@end deffn

Other constructors, all class methods:

@geindex today() (datetime.datetime class method)
@anchor{library/datetime datetime datetime today}@anchor{1d3b}
@deffn {Method} classmethod datetime.today ()

Return the current local datetime, with @ref{1d1c,,tzinfo} @code{None}.

Equivalent to:

@example
datetime.fromtimestamp(time.time())
@end example

See also @ref{12cd,,now()}, @ref{134e,,fromtimestamp()}.

This method is functionally equivalent to @ref{12cd,,now()}, but without a
@code{tz} parameter.
@end deffn

@geindex now() (datetime.datetime class method)
@anchor{library/datetime datetime datetime now}@anchor{12cd}
@deffn {Method} classmethod datetime.now (tz=None)

Return the current local date and time.

If optional argument @emph{tz} is @code{None}
or not specified, this is like @ref{1d3b,,today()}, but, if possible, supplies more
precision than can be gotten from going through a @ref{5fc,,time.time()} timestamp
(for example, this may be possible on platforms supplying the C
@code{gettimeofday()} function).

If @emph{tz} is not @code{None}, it must be an instance of a @ref{3c6,,tzinfo} subclass,
and the current date and time are converted to @emph{tz}’s time zone.

This function is preferred over @ref{1d3b,,today()} and @ref{1d3c,,utcnow()}.
@end deffn

@geindex utcnow() (datetime.datetime class method)
@anchor{library/datetime datetime datetime utcnow}@anchor{1d3c}
@deffn {Method} classmethod datetime.utcnow ()

Return the current UTC date and time, with @ref{1d1c,,tzinfo} @code{None}.

This is like @ref{12cd,,now()}, but returns the current UTC date and time, as a naive
@ref{363,,datetime} object. An aware current UTC datetime can be obtained by
calling @code{datetime.now(timezone.utc)}. See also @ref{12cd,,now()}.

@cartouche
@quotation Warning 
Because naive @code{datetime} objects are treated by many @code{datetime} methods
as local times, it is preferred to use aware datetimes to represent times
in UTC. As such, the recommended way to create an object representing the
current time in UTC is by calling @code{datetime.now(timezone.utc)}.
@end quotation
@end cartouche
@end deffn

@geindex fromtimestamp() (datetime.datetime class method)
@anchor{library/datetime datetime datetime fromtimestamp}@anchor{134e}
@deffn {Method} classmethod datetime.fromtimestamp (timestamp, tz=None)

Return the local date and time corresponding to the POSIX timestamp, such as is
returned by @ref{5fc,,time.time()}. If optional argument @emph{tz} is @code{None} or not
specified, the timestamp is converted to the platform’s local date and time, and
the returned @ref{363,,datetime} object is naive.

If @emph{tz} is not @code{None}, it must be an instance of a @ref{3c6,,tzinfo} subclass, and the
timestamp is converted to @emph{tz}’s time zone.

@ref{134e,,fromtimestamp()} may raise @ref{347,,OverflowError}, if the timestamp is out of
the range of values supported by the platform C @code{localtime()} or
@code{gmtime()} functions, and @ref{2a1,,OSError} on @code{localtime()} or
@code{gmtime()} failure.
It’s common for this to be restricted to years in
1970 through 2038. Note that on non-POSIX systems that include leap seconds in
their notion of a timestamp, leap seconds are ignored by @ref{134e,,fromtimestamp()},
and then it’s possible to have two timestamps differing by a second that yield
identical @ref{363,,datetime} objects. This method is preferred over
@ref{1d3d,,utcfromtimestamp()}.


Changed in version 3.3: Raise @ref{347,,OverflowError} instead of @ref{17c,,ValueError} if the timestamp
is out of the range of values supported by the platform C
@code{localtime()} or @code{gmtime()} functions. Raise @ref{2a1,,OSError}
instead of @ref{17c,,ValueError} on @code{localtime()} or @code{gmtime()}
failure.



Changed in version 3.6: @ref{134e,,fromtimestamp()} may return instances with @ref{79c,,fold} set to 1.

@end deffn

@geindex utcfromtimestamp() (datetime.datetime class method)
@anchor{library/datetime datetime datetime utcfromtimestamp}@anchor{1d3d}
@deffn {Method} classmethod datetime.utcfromtimestamp (timestamp)

Return the UTC @ref{363,,datetime} corresponding to the POSIX timestamp, with
@ref{1d1c,,tzinfo} @code{None}.  (The resulting object is naive.)

This may raise @ref{347,,OverflowError}, if the timestamp is
out of the range of values supported by the platform C @code{gmtime()} function,
and @ref{2a1,,OSError} on @code{gmtime()} failure.
It’s common for this to be restricted to years in 1970 through 2038.

To get an aware @ref{363,,datetime} object, call @ref{134e,,fromtimestamp()}:

@example
datetime.fromtimestamp(timestamp, timezone.utc)
@end example

On the POSIX compliant platforms, it is equivalent to the following
expression:

@example
datetime(1970, 1, 1, tzinfo=timezone.utc) + timedelta(seconds=timestamp)
@end example

except the latter formula always supports the full years range: between
@ref{1d0a,,MINYEAR} and @ref{1d0b,,MAXYEAR} inclusive.

@cartouche
@quotation Warning 
Because naive @code{datetime} objects are treated by many @code{datetime} methods
as local times, it is preferred to use aware datetimes to represent times
in UTC. As such, the recommended way to create an object representing a
specific timestamp in UTC is by calling
@code{datetime.fromtimestamp(timestamp, tz=timezone.utc)}.
@end quotation
@end cartouche


Changed in version 3.3: Raise @ref{347,,OverflowError} instead of @ref{17c,,ValueError} if the timestamp
is out of the range of values supported by the platform C
@code{gmtime()} function. Raise @ref{2a1,,OSError} instead of
@ref{17c,,ValueError} on @code{gmtime()} failure.

@end deffn

@geindex fromordinal() (datetime.datetime class method)
@anchor{library/datetime datetime datetime fromordinal}@anchor{1d3e}
@deffn {Method} classmethod datetime.fromordinal (ordinal)

Return the @ref{363,,datetime} corresponding to the proleptic Gregorian ordinal,
where January 1 of year 1 has ordinal 1. @ref{17c,,ValueError} is raised unless @code{1
<= ordinal <= datetime.max.toordinal()}. The hour, minute, second and
microsecond of the result are all 0, and @ref{1d1c,,tzinfo} is @code{None}.
@end deffn

@geindex combine() (datetime.datetime class method)
@anchor{library/datetime datetime datetime combine}@anchor{7db}
@deffn {Method} classmethod datetime.combine (date, time, tzinfo=self.tzinfo)

Return a new @ref{363,,datetime} object whose date components are equal to the
given @ref{3e2,,date} object’s, and whose time components
are equal to the given @ref{364,,time} object’s. If the @emph{tzinfo}
argument is provided, its value is used to set the @ref{1d1c,,tzinfo} attribute
of the result, otherwise the @ref{1d14,,tzinfo} attribute of the @emph{time} argument
is used.

For any @ref{363,,datetime} object @emph{d},
@code{d == datetime.combine(d.date(), d.time(), d.tzinfo)}. If date is a
@ref{363,,datetime} object, its time components and @ref{1d1c,,tzinfo} attributes
are ignored.


Changed in version 3.6: Added the @emph{tzinfo} argument.

@end deffn

@geindex fromisoformat() (datetime.datetime class method)
@anchor{library/datetime datetime datetime fromisoformat}@anchor{650}
@deffn {Method} classmethod datetime.fromisoformat (date_string)

Return a @ref{363,,datetime} corresponding to a @emph{date_string} in one of the
formats emitted by @ref{1d2a,,date.isoformat()} and @ref{651,,datetime.isoformat()}.

Specifically, this function supports strings in the format:

@example
YYYY-MM-DD[*HH[:MM[:SS[.fff[fff]]]][+HH:MM[:SS[.ffffff]]]]
@end example

where @code{*} can match any single character.

@cartouche
@quotation Caution 
This does @emph{not} support parsing arbitrary ISO 8601 strings - it is only intended
as the inverse operation of @ref{651,,datetime.isoformat()}. A more full-featured
ISO 8601 parser, @code{dateutil.parser.isoparse} is available in the third-party package
dateutil@footnote{https://dateutil.readthedocs.io/en/stable/parser.html#dateutil.parser.isoparse}.
@end quotation
@end cartouche

Examples:

@example
>>> from datetime import datetime
>>> datetime.fromisoformat('2011-11-04')
datetime.datetime(2011, 11, 4, 0, 0)
>>> datetime.fromisoformat('2011-11-04T00:05:23')
datetime.datetime(2011, 11, 4, 0, 5, 23)
>>> datetime.fromisoformat('2011-11-04 00:05:23.283')
datetime.datetime(2011, 11, 4, 0, 5, 23, 283000)
>>> datetime.fromisoformat('2011-11-04 00:05:23.283+00:00')
datetime.datetime(2011, 11, 4, 0, 5, 23, 283000, tzinfo=datetime.timezone.utc)
>>> datetime.fromisoformat('2011-11-04T00:05:23+04:00')   
datetime.datetime(2011, 11, 4, 0, 5, 23,
    tzinfo=datetime.timezone(datetime.timedelta(seconds=14400)))
@end example


New in version 3.7.

@end deffn

@geindex fromisocalendar() (datetime.datetime class method)
@anchor{library/datetime datetime datetime fromisocalendar}@anchor{4f1}
@deffn {Method} classmethod datetime.fromisocalendar (year, week, day)

Return a @ref{363,,datetime} corresponding to the ISO calendar date specified
by year, week and day. The non-date components of the datetime are populated
with their normal default values. This is the inverse of the function
@ref{3e3,,datetime.isocalendar()}.


New in version 3.8.

@end deffn

@geindex strptime() (datetime.datetime class method)
@anchor{library/datetime datetime datetime strptime}@anchor{114f}
@deffn {Method} classmethod datetime.strptime (date_string, format)

Return a @ref{363,,datetime} corresponding to @emph{date_string}, parsed according to
@emph{format}.

This is equivalent to:

@example
datetime(*(time.strptime(date_string, format)[0:6]))
@end example

@ref{17c,,ValueError} is raised if the date_string and format
can’t be parsed by @ref{f8b,,time.strptime()} or if it returns a value which isn’t a
time tuple. For a complete list of formatting directives, see
@ref{1d36,,strftime() and strptime() Behavior}.
@end deffn

Class attributes:

@geindex min (datetime.datetime attribute)
@anchor{library/datetime datetime datetime min}@anchor{1d3f}
@deffn {Attribute} datetime.min

The earliest representable @ref{363,,datetime}, @code{datetime(MINYEAR, 1, 1,
tzinfo=None)}.
@end deffn

@geindex max (datetime.datetime attribute)
@anchor{library/datetime datetime datetime max}@anchor{1d40}
@deffn {Attribute} datetime.max

The latest representable @ref{363,,datetime}, @code{datetime(MAXYEAR, 12, 31, 23, 59,
59, 999999, tzinfo=None)}.
@end deffn

@geindex resolution (datetime.datetime attribute)
@anchor{library/datetime datetime datetime resolution}@anchor{1d41}
@deffn {Attribute} datetime.resolution

The smallest possible difference between non-equal @ref{363,,datetime} objects,
@code{timedelta(microseconds=1)}.
@end deffn

Instance attributes (read-only):

@geindex year (datetime.datetime attribute)
@anchor{library/datetime datetime datetime year}@anchor{1d15}
@deffn {Attribute} datetime.year

Between @ref{1d0a,,MINYEAR} and @ref{1d0b,,MAXYEAR} inclusive.
@end deffn

@geindex month (datetime.datetime attribute)
@anchor{library/datetime datetime datetime month}@anchor{1d16}
@deffn {Attribute} datetime.month

Between 1 and 12 inclusive.
@end deffn

@geindex day (datetime.datetime attribute)
@anchor{library/datetime datetime datetime day}@anchor{1d17}
@deffn {Attribute} datetime.day

Between 1 and the number of days in the given month of the given year.
@end deffn

@geindex hour (datetime.datetime attribute)
@anchor{library/datetime datetime datetime hour}@anchor{1d18}
@deffn {Attribute} datetime.hour

In @code{range(24)}.
@end deffn

@geindex minute (datetime.datetime attribute)
@anchor{library/datetime datetime datetime minute}@anchor{1d19}
@deffn {Attribute} datetime.minute

In @code{range(60)}.
@end deffn

@geindex second (datetime.datetime attribute)
@anchor{library/datetime datetime datetime second}@anchor{1d1a}
@deffn {Attribute} datetime.second

In @code{range(60)}.
@end deffn

@geindex microsecond (datetime.datetime attribute)
@anchor{library/datetime datetime datetime microsecond}@anchor{1d1b}
@deffn {Attribute} datetime.microsecond

In @code{range(1000000)}.
@end deffn

@geindex tzinfo (datetime.datetime attribute)
@anchor{library/datetime datetime datetime tzinfo}@anchor{1d1c}
@deffn {Attribute} datetime.tzinfo

The object passed as the @emph{tzinfo} argument to the @ref{363,,datetime} constructor,
or @code{None} if none was passed.
@end deffn

@geindex fold (datetime.datetime attribute)
@anchor{library/datetime datetime datetime fold}@anchor{79c}
@deffn {Attribute} datetime.fold

In @code{[0, 1]}. Used to disambiguate wall times during a repeated interval. (A
repeated interval occurs when clocks are rolled back at the end of daylight saving
time or when the UTC offset for the current zone is decreased for political reasons.)
The value 0 (1) represents the earlier (later) of the two moments with the same wall
time representation.


New in version 3.6.

@end deffn

Supported operations:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Operation

@tab

Result

@item

@code{datetime2 = datetime1 + timedelta}

@tab

(1)

@item

@code{datetime2 = datetime1 - timedelta}

@tab

(2)

@item

@code{timedelta = datetime1 - datetime2}

@tab

(3)

@item

@code{datetime1 < datetime2}

@tab

Compares @ref{363,,datetime} to
@ref{363,,datetime}. (4)

@end multitable



@enumerate 

@item 
datetime2 is a duration of timedelta removed from datetime1, moving forward in
time if @code{timedelta.days} > 0, or backward if @code{timedelta.days} < 0. The
result has the same @ref{1d1c,,tzinfo} attribute as the input datetime, and
datetime2 - datetime1 == timedelta after. @ref{347,,OverflowError} is raised if
datetime2.year would be smaller than @ref{1d0a,,MINYEAR} or larger than
@ref{1d0b,,MAXYEAR}. Note that no time zone adjustments are done even if the
input is an aware object.

@item 
Computes the datetime2 such that datetime2 + timedelta == datetime1. As for
addition, the result has the same @ref{1d1c,,tzinfo} attribute as the input
datetime, and no time zone adjustments are done even if the input is aware.

@item 
Subtraction of a @ref{363,,datetime} from a @ref{363,,datetime} is defined only if
both operands are naive, or if both are aware. If one is aware and the other is
naive, @ref{162,,TypeError} is raised.

If both are naive, or both are aware and have the same @ref{1d1c,,tzinfo} attribute,
the @ref{1d1c,,tzinfo} attributes are ignored, and the result is a @ref{4c8,,timedelta}
object @emph{t} such that @code{datetime2 + t == datetime1}. No time zone adjustments
are done in this case.

If both are aware and have different @ref{1d1c,,tzinfo} attributes, @code{a-b} acts
as if @emph{a} and @emph{b} were first converted to naive UTC datetimes first. The
result is @code{(a.replace(tzinfo=None) - a.utcoffset()) - (b.replace(tzinfo=None)
- b.utcoffset())} except that the implementation never overflows.

@item 
@emph{datetime1} is considered less than @emph{datetime2} when @emph{datetime1} precedes
@emph{datetime2} in time.

If one comparand is naive and the other is aware, @ref{162,,TypeError}
is raised if an order comparison is attempted. For equality
comparisons, naive instances are never equal to aware instances.

If both comparands are aware, and have the same @ref{1d1c,,tzinfo} attribute, the
common @ref{1d1c,,tzinfo} attribute is ignored and the base datetimes are
compared. If both comparands are aware and have different @ref{1d1c,,tzinfo}
attributes, the comparands are first adjusted by subtracting their UTC
offsets (obtained from @code{self.utcoffset()}).


Changed in version 3.3: Equality comparisons between aware and naive @ref{363,,datetime}
instances don’t raise @ref{162,,TypeError}.


@cartouche
@quotation Note 
In order to stop comparison from falling back to the default scheme of comparing
object addresses, datetime comparison normally raises @ref{162,,TypeError} if the
other comparand isn’t also a @ref{363,,datetime} object. However,
@code{NotImplemented} is returned instead if the other comparand has a
@code{timetuple()} attribute. This hook gives other kinds of date objects a
chance at implementing mixed-type comparison. If not, when a @ref{363,,datetime}
object is compared to an object of a different type, @ref{162,,TypeError} is raised
unless the comparison is @code{==} or @code{!=}. The latter cases return
@ref{659,,False} or @ref{748,,True}, respectively.
@end quotation
@end cartouche
@end enumerate

Instance methods:

@geindex date() (datetime.datetime method)
@anchor{library/datetime datetime datetime date}@anchor{1d42}
@deffn {Method} datetime.date ()

Return @ref{3e2,,date} object with same year, month and day.
@end deffn

@geindex time() (datetime.datetime method)
@anchor{library/datetime datetime datetime time}@anchor{1d43}
@deffn {Method} datetime.time ()

Return @ref{364,,time} object with same hour, minute, second, microsecond and fold.
@ref{1d1c,,tzinfo} is @code{None}. See also method @ref{1d44,,timetz()}.


Changed in version 3.6: The fold value is copied to the returned @ref{364,,time} object.

@end deffn

@geindex timetz() (datetime.datetime method)
@anchor{library/datetime datetime datetime timetz}@anchor{1d44}
@deffn {Method} datetime.timetz ()

Return @ref{364,,time} object with same hour, minute, second, microsecond, fold, and
tzinfo attributes. See also method @ref{10a,,time()}.


Changed in version 3.6: The fold value is copied to the returned @ref{364,,time} object.

@end deffn

@geindex replace() (datetime.datetime method)
@anchor{library/datetime datetime datetime replace}@anchor{1d45}
@deffn {Method} datetime.replace (year=self.year, month=self.month, day=self.day, hour=self.hour, minute=self.minute, second=self.second, microsecond=self.microsecond, tzinfo=self.tzinfo, *, fold=0)

Return a datetime with the same attributes, except for those attributes given
new values by whichever keyword arguments are specified. Note that
@code{tzinfo=None} can be specified to create a naive datetime from an aware
datetime with no conversion of date and time data.


New in version 3.6: Added the @code{fold} argument.

@end deffn

@geindex astimezone() (datetime.datetime method)
@anchor{library/datetime datetime datetime astimezone}@anchor{4c9}
@deffn {Method} datetime.astimezone (tz=None)

Return a @ref{363,,datetime} object with new @ref{1d1c,,tzinfo} attribute @emph{tz},
adjusting the date and time data so the result is the same UTC time as
@emph{self}, but in @emph{tz}’s local time.

If provided, @emph{tz} must be an instance of a @ref{3c6,,tzinfo} subclass, and its
@ref{1d46,,utcoffset()} and @ref{1d47,,dst()} methods must not return @code{None}. If @emph{self}
is naive, it is presumed to represent time in the system timezone.

If called without arguments (or with @code{tz=None}) the system local
timezone is assumed for the target timezone. The @code{.tzinfo} attribute of the converted
datetime instance will be set to an instance of @ref{c51,,timezone}
with the zone name and offset obtained from the OS.

If @code{self.tzinfo} is @emph{tz}, @code{self.astimezone(tz)} is equal to @emph{self}:  no
adjustment of date or time data is performed. Else the result is local
time in the timezone @emph{tz}, representing the same UTC time as @emph{self}:  after
@code{astz = dt.astimezone(tz)}, @code{astz - astz.utcoffset()} will have
the same date and time data as @code{dt - dt.utcoffset()}.

If you merely want to attach a time zone object @emph{tz} to a datetime @emph{dt} without
adjustment of date and time data, use @code{dt.replace(tzinfo=tz)}. If you
merely want to remove the time zone object from an aware datetime @emph{dt} without
conversion of date and time data, use @code{dt.replace(tzinfo=None)}.

Note that the default @ref{1d48,,tzinfo.fromutc()} method can be overridden in a
@ref{3c6,,tzinfo} subclass to affect the result returned by @ref{4c9,,astimezone()}.
Ignoring error cases, @ref{4c9,,astimezone()} acts like:

@example
def astimezone(self, tz):
    if self.tzinfo is tz:
        return self
    # Convert self to UTC, and attach the new time zone object.
    utc = (self - self.utcoffset()).replace(tzinfo=tz)
    # Convert from UTC to tz's local time.
    return tz.fromutc(utc)
@end example


Changed in version 3.3: @emph{tz} now can be omitted.



Changed in version 3.6: The @ref{4c9,,astimezone()} method can now be called on naive instances that
are presumed to represent system local time.

@end deffn

@geindex utcoffset() (datetime.datetime method)
@anchor{library/datetime datetime datetime utcoffset}@anchor{1d46}
@deffn {Method} datetime.utcoffset ()

If @ref{1d1c,,tzinfo} is @code{None}, returns @code{None}, else returns
@code{self.tzinfo.utcoffset(self)}, and raises an exception if the latter doesn’t
return @code{None} or a @ref{4c8,,timedelta} object with magnitude less than one day.


Changed in version 3.7: The UTC offset is not restricted to a whole number of minutes.

@end deffn

@geindex dst() (datetime.datetime method)
@anchor{library/datetime datetime datetime dst}@anchor{1d47}
@deffn {Method} datetime.dst ()

If @ref{1d1c,,tzinfo} is @code{None}, returns @code{None}, else returns
@code{self.tzinfo.dst(self)}, and raises an exception if the latter doesn’t return
@code{None} or a @ref{4c8,,timedelta} object with magnitude less than one day.


Changed in version 3.7: The DST offset is not restricted to a whole number of minutes.

@end deffn

@geindex tzname() (datetime.datetime method)
@anchor{library/datetime datetime datetime tzname}@anchor{1d49}
@deffn {Method} datetime.tzname ()

If @ref{1d1c,,tzinfo} is @code{None}, returns @code{None}, else returns
@code{self.tzinfo.tzname(self)}, raises an exception if the latter doesn’t return
@code{None} or a string object,
@end deffn

@geindex timetuple() (datetime.datetime method)
@anchor{library/datetime datetime datetime timetuple}@anchor{1d4a}
@deffn {Method} datetime.timetuple ()

Return a @ref{834,,time.struct_time} such as returned by @ref{1224,,time.localtime()}.

@code{d.timetuple()} is equivalent to:

@example
time.struct_time((d.year, d.month, d.day,
                  d.hour, d.minute, d.second,
                  d.weekday(), yday, dst))
@end example

where @code{yday = d.toordinal() - date(d.year, 1, 1).toordinal() + 1}
is the day number within the current year starting with @code{1} for January
1st. The @code{tm_isdst} flag of the result is set according to the
@ref{1d47,,dst()} method: @ref{1d1c,,tzinfo} is @code{None} or @ref{1d47,,dst()} returns
@code{None}, @code{tm_isdst} is set to @code{-1}; else if @ref{1d47,,dst()} returns a
non-zero value, @code{tm_isdst} is set to @code{1}; else @code{tm_isdst} is
set to @code{0}.
@end deffn

@geindex utctimetuple() (datetime.datetime method)
@anchor{library/datetime datetime datetime utctimetuple}@anchor{1d4b}
@deffn {Method} datetime.utctimetuple ()

If @ref{363,,datetime} instance @emph{d} is naive, this is the same as
@code{d.timetuple()} except that @code{tm_isdst} is forced to 0 regardless of what
@code{d.dst()} returns. DST is never in effect for a UTC time.

If @emph{d} is aware, @emph{d} is normalized to UTC time, by subtracting
@code{d.utcoffset()}, and a @ref{834,,time.struct_time} for the
normalized time is returned. @code{tm_isdst} is forced to 0. Note
that an @ref{347,,OverflowError} may be raised if @emph{d}.year was
@code{MINYEAR} or @code{MAXYEAR} and UTC adjustment spills over a year
boundary.

@cartouche
@quotation Warning 
Because naive @code{datetime} objects are treated by many @code{datetime} methods
as local times, it is preferred to use aware datetimes to represent times
in UTC; as a result, using @code{utcfromtimetuple} may give misleading
results. If you have a naive @code{datetime} representing UTC, use
@code{datetime.replace(tzinfo=timezone.utc)} to make it aware, at which point
you can use @ref{1d4a,,datetime.timetuple()}.
@end quotation
@end cartouche
@end deffn

@geindex toordinal() (datetime.datetime method)
@anchor{library/datetime datetime datetime toordinal}@anchor{1d4c}
@deffn {Method} datetime.toordinal ()

Return the proleptic Gregorian ordinal of the date. The same as
@code{self.date().toordinal()}.
@end deffn

@geindex timestamp() (datetime.datetime method)
@anchor{library/datetime datetime datetime timestamp}@anchor{c32}
@deffn {Method} datetime.timestamp ()

Return POSIX timestamp corresponding to the @ref{363,,datetime}
instance. The return value is a @ref{256,,float} similar to that
returned by @ref{5fc,,time.time()}.

Naive @ref{363,,datetime} instances are assumed to represent local
time and this method relies on the platform C @code{mktime()}
function to perform the conversion. Since @ref{363,,datetime}
supports wider range of values than @code{mktime()} on many
platforms, this method may raise @ref{347,,OverflowError} for times far
in the past or far in the future.

For aware @ref{363,,datetime} instances, the return value is computed
as:

@example
(dt - datetime(1970, 1, 1, tzinfo=timezone.utc)).total_seconds()
@end example


New in version 3.3.



Changed in version 3.6: The @ref{c32,,timestamp()} method uses the @ref{79c,,fold} attribute to
disambiguate the times during a repeated interval.


@cartouche
@quotation Note 
There is no method to obtain the POSIX timestamp directly from a
naive @ref{363,,datetime} instance representing UTC time. If your
application uses this convention and your system timezone is not
set to UTC, you can obtain the POSIX timestamp by supplying
@code{tzinfo=timezone.utc}:

@example
timestamp = dt.replace(tzinfo=timezone.utc).timestamp()
@end example

or by calculating the timestamp directly:

@example
timestamp = (dt - datetime(1970, 1, 1)) / timedelta(seconds=1)
@end example
@end quotation
@end cartouche
@end deffn

@geindex weekday() (datetime.datetime method)
@anchor{library/datetime datetime datetime weekday}@anchor{1d4d}
@deffn {Method} datetime.weekday ()

Return the day of the week as an integer, where Monday is 0 and Sunday is 6.
The same as @code{self.date().weekday()}. See also @ref{1d4e,,isoweekday()}.
@end deffn

@geindex isoweekday() (datetime.datetime method)
@anchor{library/datetime datetime datetime isoweekday}@anchor{1d4e}
@deffn {Method} datetime.isoweekday ()

Return the day of the week as an integer, where Monday is 1 and Sunday is 7.
The same as @code{self.date().isoweekday()}. See also @ref{1d4d,,weekday()},
@ref{3e3,,isocalendar()}.
@end deffn

@geindex isocalendar() (datetime.datetime method)
@anchor{library/datetime datetime datetime isocalendar}@anchor{3e3}
@deffn {Method} datetime.isocalendar ()

Return a @ref{ce9,,named tuple} with three components: @code{year}, @code{week}
and @code{weekday}. The same as @code{self.date().isocalendar()}.
@end deffn

@geindex isoformat() (datetime.datetime method)
@anchor{library/datetime datetime datetime isoformat}@anchor{651}
@deffn {Method} datetime.isoformat (sep='T', timespec='auto')

Return a string representing the date and time in ISO 8601 format:


@itemize -

@item 
@code{YYYY-MM-DDTHH:MM:SS.ffffff}, if @ref{1d1b,,microsecond} is not 0

@item 
@code{YYYY-MM-DDTHH:MM:SS}, if @ref{1d1b,,microsecond} is 0
@end itemize

If @ref{1d46,,utcoffset()} does not return @code{None}, a string is
appended, giving the UTC offset:


@itemize -

@item 
@code{YYYY-MM-DDTHH:MM:SS.ffffff+HH:MM[:SS[.ffffff]]}, if @ref{1d1b,,microsecond}
is not 0

@item 
@code{YYYY-MM-DDTHH:MM:SS+HH:MM[:SS[.ffffff]]},  if @ref{1d1b,,microsecond} is 0
@end itemize

Examples:

@example
>>> from datetime import datetime, timezone
>>> datetime(2019, 5, 18, 15, 17, 8, 132263).isoformat()
'2019-05-18T15:17:08.132263'
>>> datetime(2019, 5, 18, 15, 17, tzinfo=timezone.utc).isoformat()
'2019-05-18T15:17:00+00:00'
@end example

The optional argument @emph{sep} (default @code{'T'}) is a one-character separator,
placed between the date and time portions of the result. For example:

@example
>>> from datetime import tzinfo, timedelta, datetime
>>> class TZ(tzinfo):
...     """A time zone with an arbitrary, constant -06:39 offset."""
...     def utcoffset(self, dt):
...         return timedelta(hours=-6, minutes=-39)
...
>>> datetime(2002, 12, 25, tzinfo=TZ()).isoformat(' ')
'2002-12-25 00:00:00-06:39'
>>> datetime(2009, 11, 27, microsecond=100, tzinfo=TZ()).isoformat()
'2009-11-27T00:00:00.000100-06:39'
@end example

The optional argument @emph{timespec} specifies the number of additional
components of the time to include (the default is @code{'auto'}).
It can be one of the following:


@itemize -

@item 
@code{'auto'}: Same as @code{'seconds'} if @ref{1d1b,,microsecond} is 0,
same as @code{'microseconds'} otherwise.

@item 
@code{'hours'}: Include the @ref{1d18,,hour} in the two-digit @code{HH} format.

@item 
@code{'minutes'}: Include @ref{1d18,,hour} and @ref{1d19,,minute} in @code{HH:MM} format.

@item 
@code{'seconds'}: Include @ref{1d18,,hour}, @ref{1d19,,minute}, and @ref{1d1a,,second}
in @code{HH:MM:SS} format.

@item 
@code{'milliseconds'}: Include full time, but truncate fractional second
part to milliseconds. @code{HH:MM:SS.sss} format.

@item 
@code{'microseconds'}: Include full time in @code{HH:MM:SS.ffffff} format.
@end itemize

@cartouche
@quotation Note 
Excluded time components are truncated, not rounded.
@end quotation
@end cartouche

@ref{17c,,ValueError} will be raised on an invalid @emph{timespec} argument:

@example
>>> from datetime import datetime
>>> datetime.now().isoformat(timespec='minutes')   
'2002-12-25T00:00'
>>> dt = datetime(2015, 1, 1, 12, 30, 59, 0)
>>> dt.isoformat(timespec='microseconds')
'2015-01-01T12:30:59.000000'
@end example


New in version 3.6: Added the @emph{timespec} argument.

@end deffn

@geindex __str__() (datetime.datetime method)
@anchor{library/datetime datetime datetime __str__}@anchor{1d4f}
@deffn {Method} datetime.__str__ ()

For a @ref{363,,datetime} instance @emph{d}, @code{str(d)} is equivalent to
@code{d.isoformat(' ')}.
@end deffn

@geindex ctime() (datetime.datetime method)
@anchor{library/datetime datetime datetime ctime}@anchor{1d50}
@deffn {Method} datetime.ctime ()

Return a string representing the date and time:

@example
>>> from datetime import datetime
>>> datetime(2002, 12, 4, 20, 30, 40).ctime()
'Wed Dec  4 20:30:40 2002'
@end example

The output string will @emph{not} include time zone information, regardless
of whether the input is aware or naive.

@code{d.ctime()} is equivalent to:

@example
time.ctime(time.mktime(d.timetuple()))
@end example

on platforms where the native C @code{ctime()} function
(which @ref{1d35,,time.ctime()} invokes, but which
@ref{1d50,,datetime.ctime()} does not invoke) conforms to the C standard.
@end deffn

@geindex strftime() (datetime.datetime method)
@anchor{library/datetime datetime datetime strftime}@anchor{7d9}
@deffn {Method} datetime.strftime (format)

Return a string representing the date and time, controlled by an explicit format
string. For a complete list of formatting directives, see
@ref{1d36,,strftime() and strptime() Behavior}.
@end deffn

@geindex __format__() (datetime.datetime method)
@anchor{library/datetime datetime datetime __format__}@anchor{1d51}
@deffn {Method} datetime.__format__ (format)

Same as @ref{7d9,,datetime.strftime()}. This makes it possible to specify a format
string for a @ref{363,,datetime} object in @ref{4b0,,formatted string literals} and when using @ref{784,,str.format()}. For a
complete list of formatting directives, see
@ref{1d36,,strftime() and strptime() Behavior}.
@end deffn

@menu
* Examples of Usage; datetime: Examples of Usage datetime. 

@end menu

@node Examples of Usage datetime,,,datetime Objects
@anchor{library/datetime examples-of-usage-datetime}@anchor{1d52}
@subsubsection Examples of Usage: @code{datetime}


Examples of working with @ref{363,,datetime} objects:

@example
>>> from datetime import datetime, date, time, timezone

>>> # Using datetime.combine()
>>> d = date(2005, 7, 14)
>>> t = time(12, 30)
>>> datetime.combine(d, t)
datetime.datetime(2005, 7, 14, 12, 30)

>>> # Using datetime.now()
>>> datetime.now()   
datetime.datetime(2007, 12, 6, 16, 29, 43, 79043)   # GMT +1
>>> datetime.now(timezone.utc)   
datetime.datetime(2007, 12, 6, 15, 29, 43, 79060, tzinfo=datetime.timezone.utc)

>>> # Using datetime.strptime()
>>> dt = datetime.strptime("21/11/06 16:30", "%d/%m/%y %H:%M")
>>> dt
datetime.datetime(2006, 11, 21, 16, 30)

>>> # Using datetime.timetuple() to get tuple of all attributes
>>> tt = dt.timetuple()
>>> for it in tt:   
...     print(it)
...
2006    # year
11      # month
21      # day
16      # hour
30      # minute
0       # second
1       # weekday (0 = Monday)
325     # number of days since 1st January
-1      # dst - method tzinfo.dst() returned None

>>> # Date in ISO format
>>> ic = dt.isocalendar()
>>> for it in ic:   
...     print(it)
...
2006    # ISO year
47      # ISO week
2       # ISO weekday

>>> # Formatting a datetime
>>> dt.strftime("%A, %d. %B %Y %I:%M%p")
'Tuesday, 21. November 2006 04:30PM'
>>> 'The @{1@} is @{0:%d@}, the @{2@} is @{0:%B@}, the @{3@} is @{0:%I:%M%p@}.'.format(dt, "day", "month", "time")
'The day is 21, the month is November, the time is 04:30PM.'
@end example

The example below defines a @ref{3c6,,tzinfo} subclass capturing time zone
information for Kabul, Afghanistan, which used +4 UTC until 1945
and then +4:30 UTC thereafter:

@example
from datetime import timedelta, datetime, tzinfo, timezone

class KabulTz(tzinfo):
    # Kabul used +4 until 1945, when they moved to +4:30
    UTC_MOVE_DATE = datetime(1944, 12, 31, 20, tzinfo=timezone.utc)

    def utcoffset(self, dt):
        if dt.year < 1945:
            return timedelta(hours=4)
        elif (1945, 1, 1, 0, 0) <= dt.timetuple()[:5] < (1945, 1, 1, 0, 30):
            # An ambiguous ("imaginary") half-hour range representing
            # a 'fold' in time due to the shift from +4 to +4:30.
            # If dt falls in the imaginary range, use fold to decide how
            # to resolve. See PEP495.
            return timedelta(hours=4, minutes=(30 if dt.fold else 0))
        else:
            return timedelta(hours=4, minutes=30)

    def fromutc(self, dt):
        # Follow same validations as in datetime.tzinfo
        if not isinstance(dt, datetime):
            raise TypeError("fromutc() requires a datetime argument")
        if dt.tzinfo is not self:
            raise ValueError("dt.tzinfo is not self")

        # A custom implementation is required for fromutc as
        # the input to this function is a datetime with utc values
        # but with a tzinfo set to self.
        # See datetime.astimezone or fromtimestamp.
        if dt.replace(tzinfo=timezone.utc) >= self.UTC_MOVE_DATE:
            return dt + timedelta(hours=4, minutes=30)
        else:
            return dt + timedelta(hours=4)

    def dst(self, dt):
        # Kabul does not observe daylight saving time.
        return timedelta(0)

    def tzname(self, dt):
        if dt >= self.UTC_MOVE_DATE:
            return "+04:30"
        return "+04"
@end example

Usage of @code{KabulTz} from above:

@example
>>> tz1 = KabulTz()

>>> # Datetime before the change
>>> dt1 = datetime(1900, 11, 21, 16, 30, tzinfo=tz1)
>>> print(dt1.utcoffset())
4:00:00

>>> # Datetime after the change
>>> dt2 = datetime(2006, 6, 14, 13, 0, tzinfo=tz1)
>>> print(dt2.utcoffset())
4:30:00

>>> # Convert datetime to another time zone
>>> dt3 = dt2.astimezone(timezone.utc)
>>> dt3
datetime.datetime(2006, 6, 14, 8, 30, tzinfo=datetime.timezone.utc)
>>> dt2
datetime.datetime(2006, 6, 14, 13, 0, tzinfo=KabulTz())
>>> dt2 == dt3
True
@end example

@node time Objects,tzinfo Objects,datetime Objects,datetime — Basic date and time types
@anchor{library/datetime datetime-time}@anchor{1d53}@anchor{library/datetime time-objects}@anchor{1d54}
@subsubsection @code{time} Objects


A @ref{10a,,time} object represents a (local) time of day, independent of any particular
day, and subject to adjustment via a @ref{3c6,,tzinfo} object.

@geindex time (class in datetime)
@anchor{library/datetime datetime time}@anchor{364}
@deffn {Class} datetime.time (hour=0, minute=0, second=0, microsecond=0, tzinfo=None, *, fold=0)

All arguments are optional. @emph{tzinfo} may be @code{None}, or an instance of a
@ref{3c6,,tzinfo} subclass. The remaining arguments must be integers in the
following ranges:


@itemize *

@item 
@code{0 <= hour < 24},

@item 
@code{0 <= minute < 60},

@item 
@code{0 <= second < 60},

@item 
@code{0 <= microsecond < 1000000},

@item 
@code{fold in [0, 1]}.
@end itemize

If an argument outside those ranges is given, @ref{17c,,ValueError} is raised. All
default to @code{0} except @emph{tzinfo}, which defaults to @ref{182,,None}.
@end deffn

Class attributes:

@geindex min (datetime.time attribute)
@anchor{library/datetime datetime time min}@anchor{1d55}
@deffn {Attribute} time.min

The earliest representable @ref{364,,time}, @code{time(0, 0, 0, 0)}.
@end deffn

@geindex max (datetime.time attribute)
@anchor{library/datetime datetime time max}@anchor{1d56}
@deffn {Attribute} time.max

The latest representable @ref{364,,time}, @code{time(23, 59, 59, 999999)}.
@end deffn

@geindex resolution (datetime.time attribute)
@anchor{library/datetime datetime time resolution}@anchor{1d57}
@deffn {Attribute} time.resolution

The smallest possible difference between non-equal @ref{364,,time} objects,
@code{timedelta(microseconds=1)}, although note that arithmetic on
@ref{364,,time} objects is not supported.
@end deffn

Instance attributes (read-only):

@geindex hour (datetime.time attribute)
@anchor{library/datetime datetime time hour}@anchor{1d10}
@deffn {Attribute} time.hour

In @code{range(24)}.
@end deffn

@geindex minute (datetime.time attribute)
@anchor{library/datetime datetime time minute}@anchor{1d11}
@deffn {Attribute} time.minute

In @code{range(60)}.
@end deffn

@geindex second (datetime.time attribute)
@anchor{library/datetime datetime time second}@anchor{1d12}
@deffn {Attribute} time.second

In @code{range(60)}.
@end deffn

@geindex microsecond (datetime.time attribute)
@anchor{library/datetime datetime time microsecond}@anchor{1d13}
@deffn {Attribute} time.microsecond

In @code{range(1000000)}.
@end deffn

@geindex tzinfo (datetime.time attribute)
@anchor{library/datetime datetime time tzinfo}@anchor{1d14}
@deffn {Attribute} time.tzinfo

The object passed as the tzinfo argument to the @ref{364,,time} constructor, or
@code{None} if none was passed.
@end deffn

@geindex fold (datetime.time attribute)
@anchor{library/datetime datetime time fold}@anchor{1d58}
@deffn {Attribute} time.fold

In @code{[0, 1]}. Used to disambiguate wall times during a repeated interval. (A
repeated interval occurs when clocks are rolled back at the end of daylight saving
time or when the UTC offset for the current zone is decreased for political reasons.)
The value 0 (1) represents the earlier (later) of the two moments with the same wall
time representation.


New in version 3.6.

@end deffn

@ref{364,,time} objects support comparison of @ref{364,,time} to @ref{364,,time},
where @emph{a} is considered less
than @emph{b} when @emph{a} precedes @emph{b} in time. If one comparand is naive and the other
is aware, @ref{162,,TypeError} is raised if an order comparison is attempted. For equality
comparisons, naive instances are never equal to aware instances.

If both comparands are aware, and have
the same @ref{1d14,,tzinfo} attribute, the common @ref{1d14,,tzinfo} attribute is
ignored and the base times are compared. If both comparands are aware and
have different @ref{1d14,,tzinfo} attributes, the comparands are first adjusted by
subtracting their UTC offsets (obtained from @code{self.utcoffset()}). In order
to stop mixed-type comparisons from falling back to the default comparison by
object address, when a @ref{364,,time} object is compared to an object of a
different type, @ref{162,,TypeError} is raised unless the comparison is @code{==} or
@code{!=}. The latter cases return @ref{659,,False} or @ref{748,,True}, respectively.


Changed in version 3.3: Equality comparisons between aware and naive @ref{364,,time} instances
don’t raise @ref{162,,TypeError}.


In Boolean contexts, a @ref{364,,time} object is always considered to be true.


Changed in version 3.5: Before Python 3.5, a @ref{364,,time} object was considered to be false if it
represented midnight in UTC. This behavior was considered obscure and
error-prone and has been removed in Python 3.5. See bpo-13936@footnote{https://bugs.python.org/issue13936} for full
details.


Other constructor:

@geindex fromisoformat() (datetime.time class method)
@anchor{library/datetime datetime time fromisoformat}@anchor{1359}
@deffn {Method} classmethod time.fromisoformat (time_string)

Return a @ref{364,,time} corresponding to a @emph{time_string} in one of the
formats emitted by @ref{1d59,,time.isoformat()}. Specifically, this function supports
strings in the format:

@example
HH[:MM[:SS[.fff[fff]]]][+HH:MM[:SS[.ffffff]]]
@end example

@cartouche
@quotation Caution 
This does @emph{not} support parsing arbitrary ISO 8601 strings. It is only
intended as the inverse operation of @ref{1d59,,time.isoformat()}.
@end quotation
@end cartouche

Examples:

@example
>>> from datetime import time
>>> time.fromisoformat('04:23:01')
datetime.time(4, 23, 1)
>>> time.fromisoformat('04:23:01.000384')
datetime.time(4, 23, 1, 384)
>>> time.fromisoformat('04:23:01+04:00')
datetime.time(4, 23, 1, tzinfo=datetime.timezone(datetime.timedelta(seconds=14400)))
@end example


New in version 3.7.

@end deffn

Instance methods:

@geindex replace() (datetime.time method)
@anchor{library/datetime datetime time replace}@anchor{1d5a}
@deffn {Method} time.replace (hour=self.hour, minute=self.minute, second=self.second, microsecond=self.microsecond, tzinfo=self.tzinfo, *, fold=0)

Return a @ref{364,,time} with the same value, except for those attributes given
new values by whichever keyword arguments are specified. Note that
@code{tzinfo=None} can be specified to create a naive @ref{364,,time} from an
aware @ref{364,,time}, without conversion of the time data.


New in version 3.6: Added the @code{fold} argument.

@end deffn

@geindex isoformat() (datetime.time method)
@anchor{library/datetime datetime time isoformat}@anchor{1d59}
@deffn {Method} time.isoformat (timespec='auto')

Return a string representing the time in ISO 8601 format, one of:


@itemize -

@item 
@code{HH:MM:SS.ffffff}, if @ref{1d13,,microsecond} is not 0

@item 
@code{HH:MM:SS}, if @ref{1d13,,microsecond} is 0

@item 
@code{HH:MM:SS.ffffff+HH:MM[:SS[.ffffff]]}, if @ref{1d5b,,utcoffset()} does not return @code{None}

@item 
@code{HH:MM:SS+HH:MM[:SS[.ffffff]]}, if @ref{1d13,,microsecond} is 0 and @ref{1d5b,,utcoffset()} does not return @code{None}
@end itemize

The optional argument @emph{timespec} specifies the number of additional
components of the time to include (the default is @code{'auto'}).
It can be one of the following:


@itemize -

@item 
@code{'auto'}: Same as @code{'seconds'} if @ref{1d13,,microsecond} is 0,
same as @code{'microseconds'} otherwise.

@item 
@code{'hours'}: Include the @ref{1d10,,hour} in the two-digit @code{HH} format.

@item 
@code{'minutes'}: Include @ref{1d10,,hour} and @ref{1d11,,minute} in @code{HH:MM} format.

@item 
@code{'seconds'}: Include @ref{1d10,,hour}, @ref{1d11,,minute}, and @ref{1d12,,second}
in @code{HH:MM:SS} format.

@item 
@code{'milliseconds'}: Include full time, but truncate fractional second
part to milliseconds. @code{HH:MM:SS.sss} format.

@item 
@code{'microseconds'}: Include full time in @code{HH:MM:SS.ffffff} format.
@end itemize

@cartouche
@quotation Note 
Excluded time components are truncated, not rounded.
@end quotation
@end cartouche

@ref{17c,,ValueError} will be raised on an invalid @emph{timespec} argument.

Example:

@example
>>> from datetime import time
>>> time(hour=12, minute=34, second=56, microsecond=123456).isoformat(timespec='minutes')
'12:34'
>>> dt = time(hour=12, minute=34, second=56, microsecond=0)
>>> dt.isoformat(timespec='microseconds')
'12:34:56.000000'
>>> dt.isoformat(timespec='auto')
'12:34:56'
@end example


New in version 3.6: Added the @emph{timespec} argument.

@end deffn

@geindex __str__() (datetime.time method)
@anchor{library/datetime datetime time __str__}@anchor{1d5c}
@deffn {Method} time.__str__ ()

For a time @emph{t}, @code{str(t)} is equivalent to @code{t.isoformat()}.
@end deffn

@geindex strftime() (datetime.time method)
@anchor{library/datetime datetime time strftime}@anchor{1d5d}
@deffn {Method} time.strftime (format)

Return a string representing the time, controlled by an explicit format
string. For a complete list of formatting directives, see
@ref{1d36,,strftime() and strptime() Behavior}.
@end deffn

@geindex __format__() (datetime.time method)
@anchor{library/datetime datetime time __format__}@anchor{1d5e}
@deffn {Method} time.__format__ (format)

Same as @ref{1d5d,,time.strftime()}. This makes it possible to specify a format string
for a @ref{364,,time} object in @ref{4b0,,formatted string literals} and when using @ref{784,,str.format()}. For a
complete list of formatting directives, see
@ref{1d36,,strftime() and strptime() Behavior}.
@end deffn

@geindex utcoffset() (datetime.time method)
@anchor{library/datetime datetime time utcoffset}@anchor{1d5b}
@deffn {Method} time.utcoffset ()

If @ref{1d14,,tzinfo} is @code{None}, returns @code{None}, else returns
@code{self.tzinfo.utcoffset(None)}, and raises an exception if the latter doesn’t
return @code{None} or a @ref{4c8,,timedelta} object with magnitude less than one day.


Changed in version 3.7: The UTC offset is not restricted to a whole number of minutes.

@end deffn

@geindex dst() (datetime.time method)
@anchor{library/datetime datetime time dst}@anchor{1d5f}
@deffn {Method} time.dst ()

If @ref{1d14,,tzinfo} is @code{None}, returns @code{None}, else returns
@code{self.tzinfo.dst(None)}, and raises an exception if the latter doesn’t return
@code{None}, or a @ref{4c8,,timedelta} object with magnitude less than one day.


Changed in version 3.7: The DST offset is not restricted to a whole number of minutes.

@end deffn

@geindex tzname() (datetime.time method)
@anchor{library/datetime datetime time tzname}@anchor{1d60}
@deffn {Method} time.tzname ()

If @ref{1d14,,tzinfo} is @code{None}, returns @code{None}, else returns
@code{self.tzinfo.tzname(None)}, or raises an exception if the latter doesn’t
return @code{None} or a string object.
@end deffn

@menu
* Examples of Usage; time: Examples of Usage time. 

@end menu

@node Examples of Usage time,,,time Objects
@anchor{library/datetime examples-of-usage-time}@anchor{1d61}
@subsubsection Examples of Usage: @code{time}


Examples of working with a @ref{364,,time} object:

@example
>>> from datetime import time, tzinfo, timedelta
>>> class TZ1(tzinfo):
...     def utcoffset(self, dt):
...         return timedelta(hours=1)
...     def dst(self, dt):
...         return timedelta(0)
...     def tzname(self,dt):
...         return "+01:00"
...     def  __repr__(self):
...         return f"@{self.__class__.__name__@}()"
...
>>> t = time(12, 10, 30, tzinfo=TZ1())
>>> t
datetime.time(12, 10, 30, tzinfo=TZ1())
>>> t.isoformat()
'12:10:30+01:00'
>>> t.dst()
datetime.timedelta(0)
>>> t.tzname()
'+01:00'
>>> t.strftime("%H:%M:%S %Z")
'12:10:30 +01:00'
>>> 'The @{@} is @{:%H:%M@}.'.format("time", t)
'The time is 12:10.'
@end example

@node tzinfo Objects,timezone Objects,time Objects,datetime — Basic date and time types
@anchor{library/datetime datetime-tzinfo}@anchor{1d62}@anchor{library/datetime tzinfo-objects}@anchor{1d63}
@subsubsection @code{tzinfo} Objects


@geindex tzinfo (class in datetime)
@anchor{library/datetime datetime tzinfo}@anchor{3c6}
@deffn {Class} datetime.tzinfo

This is an abstract base class, meaning that this class should not be
instantiated directly.  Define a subclass of @ref{3c6,,tzinfo} to capture
information about a particular time zone.

An instance of (a concrete subclass of) @ref{3c6,,tzinfo} can be passed to the
constructors for @ref{363,,datetime} and @ref{364,,time} objects. The latter objects
view their attributes as being in local time, and the @ref{3c6,,tzinfo} object
supports methods revealing offset of local time from UTC, the name of the time
zone, and DST offset, all relative to a date or time object passed to them.

You need to derive a concrete subclass, and (at least)
supply implementations of the standard @ref{3c6,,tzinfo} methods needed by the
@ref{363,,datetime} methods you use. The @ref{2f,,datetime} module provides
@ref{c51,,timezone}, a simple concrete subclass of @ref{3c6,,tzinfo} which can
represent timezones with fixed offset from UTC such as UTC itself or North
American EST and EDT.

Special requirement for pickling:  A @ref{3c6,,tzinfo} subclass must have an
@code{__init__()} method that can be called with no arguments, otherwise it can be
pickled but possibly not unpickled again. This is a technical requirement that
may be relaxed in the future.

A concrete subclass of @ref{3c6,,tzinfo} may need to implement the following
methods. Exactly which methods are needed depends on the uses made of aware
@ref{2f,,datetime} objects. If in doubt, simply implement all of them.
@end deffn

@geindex utcoffset() (datetime.tzinfo method)
@anchor{library/datetime datetime tzinfo utcoffset}@anchor{1d64}
@deffn {Method} tzinfo.utcoffset (dt)

Return offset of local time from UTC, as a @ref{4c8,,timedelta} object that is
positive east of UTC. If local time is west of UTC, this should be negative.

This represents the @emph{total} offset from UTC; for example, if a
@ref{3c6,,tzinfo} object represents both time zone and DST adjustments,
@ref{1d64,,utcoffset()} should return their sum. If the UTC offset isn’t known,
return @code{None}. Else the value returned must be a @ref{4c8,,timedelta} object
strictly between @code{-timedelta(hours=24)} and @code{timedelta(hours=24)}
(the magnitude of the offset must be less than one day). Most implementations
of @ref{1d64,,utcoffset()} will probably look like one of these two:

@example
return CONSTANT                 # fixed-offset class
return CONSTANT + self.dst(dt)  # daylight-aware class
@end example

If @ref{1d64,,utcoffset()} does not return @code{None}, @ref{1d65,,dst()} should not return
@code{None} either.

The default implementation of @ref{1d64,,utcoffset()} raises
@ref{8a3,,NotImplementedError}.


Changed in version 3.7: The UTC offset is not restricted to a whole number of minutes.

@end deffn

@geindex dst() (datetime.tzinfo method)
@anchor{library/datetime datetime tzinfo dst}@anchor{1d65}
@deffn {Method} tzinfo.dst (dt)

Return the daylight saving time (DST) adjustment, as a @ref{4c8,,timedelta}
object or
@code{None} if DST information isn’t known.

Return @code{timedelta(0)} if DST is not in effect.
If DST is in effect, return the offset as a @ref{4c8,,timedelta} object
(see @ref{1d64,,utcoffset()} for details). Note that DST offset, if applicable, has
already been added to the UTC offset returned by @ref{1d64,,utcoffset()}, so there’s
no need to consult @ref{1d65,,dst()} unless you’re interested in obtaining DST info
separately. For example, @ref{1d4a,,datetime.timetuple()} calls its @ref{1d1c,,tzinfo}
attribute’s @ref{1d65,,dst()} method to determine how the @code{tm_isdst} flag
should be set, and @ref{1d48,,tzinfo.fromutc()} calls @ref{1d65,,dst()} to account for
DST changes when crossing time zones.

An instance @emph{tz} of a @ref{3c6,,tzinfo} subclass that models both standard and
daylight times must be consistent in this sense:

@code{tz.utcoffset(dt) - tz.dst(dt)}

must return the same result for every @ref{363,,datetime} @emph{dt} with @code{dt.tzinfo ==
tz}  For sane @ref{3c6,,tzinfo} subclasses, this expression yields the time
zone’s “standard offset”, which should not depend on the date or the time, but
only on geographic location. The implementation of @ref{4c9,,datetime.astimezone()}
relies on this, but cannot detect violations; it’s the programmer’s
responsibility to ensure it. If a @ref{3c6,,tzinfo} subclass cannot guarantee
this, it may be able to override the default implementation of
@ref{1d48,,tzinfo.fromutc()} to work correctly with @code{astimezone()} regardless.

Most implementations of @ref{1d65,,dst()} will probably look like one of these two:

@example
def dst(self, dt):
    # a fixed-offset class:  doesn't account for DST
    return timedelta(0)
@end example

or:

@example
def dst(self, dt):
    # Code to set dston and dstoff to the time zone's DST
    # transition times based on the input dt.year, and expressed
    # in standard local time.

    if dston <= dt.replace(tzinfo=None) < dstoff:
        return timedelta(hours=1)
    else:
        return timedelta(0)
@end example

The default implementation of @ref{1d65,,dst()} raises @ref{8a3,,NotImplementedError}.


Changed in version 3.7: The DST offset is not restricted to a whole number of minutes.

@end deffn

@geindex tzname() (datetime.tzinfo method)
@anchor{library/datetime datetime tzinfo tzname}@anchor{1d66}
@deffn {Method} tzinfo.tzname (dt)

Return the time zone name corresponding to the @ref{363,,datetime} object @emph{dt}, as
a string. Nothing about string names is defined by the @ref{2f,,datetime} module,
and there’s no requirement that it mean anything in particular. For example,
“GMT”, “UTC”, “-500”, “-5:00”, “EDT”, “US/Eastern”, “America/New York” are all
valid replies. Return @code{None} if a string name isn’t known. Note that this is
a method rather than a fixed string primarily because some @ref{3c6,,tzinfo}
subclasses will wish to return different names depending on the specific value
of @emph{dt} passed, especially if the @ref{3c6,,tzinfo} class is accounting for
daylight time.

The default implementation of @ref{1d66,,tzname()} raises @ref{8a3,,NotImplementedError}.
@end deffn

These methods are called by a @ref{363,,datetime} or @ref{364,,time} object, in
response to their methods of the same names. A @ref{363,,datetime} object passes
itself as the argument, and a @ref{364,,time} object passes @code{None} as the
argument. A @ref{3c6,,tzinfo} subclass’s methods should therefore be prepared to
accept a @emph{dt} argument of @code{None}, or of class @ref{363,,datetime}.

When @code{None} is passed, it’s up to the class designer to decide the best
response. For example, returning @code{None} is appropriate if the class wishes to
say that time objects don’t participate in the @ref{3c6,,tzinfo} protocols. It
may be more useful for @code{utcoffset(None)} to return the standard UTC offset, as
there is no other convention for discovering the standard offset.

When a @ref{363,,datetime} object is passed in response to a @ref{363,,datetime}
method, @code{dt.tzinfo} is the same object as @emph{self}. @ref{3c6,,tzinfo} methods can
rely on this, unless user code calls @ref{3c6,,tzinfo} methods directly. The
intent is that the @ref{3c6,,tzinfo} methods interpret @emph{dt} as being in local
time, and not need worry about objects in other timezones.

There is one more @ref{3c6,,tzinfo} method that a subclass may wish to override:

@geindex fromutc() (datetime.tzinfo method)
@anchor{library/datetime datetime tzinfo fromutc}@anchor{1d48}
@deffn {Method} tzinfo.fromutc (dt)

This is called from the default @ref{4c9,,datetime.astimezone()}
implementation. When called from that, @code{dt.tzinfo} is @emph{self}, and @emph{dt}’s
date and time data are to be viewed as expressing a UTC time. The purpose
of @ref{1d48,,fromutc()} is to adjust the date and time data, returning an
equivalent datetime in @emph{self}’s local time.

Most @ref{3c6,,tzinfo} subclasses should be able to inherit the default
@ref{1d48,,fromutc()} implementation without problems. It’s strong enough to handle
fixed-offset time zones, and time zones accounting for both standard and
daylight time, and the latter even if the DST transition times differ in
different years. An example of a time zone the default @ref{1d48,,fromutc()}
implementation may not handle correctly in all cases is one where the standard
offset (from UTC) depends on the specific date and time passed, which can happen
for political reasons. The default implementations of @code{astimezone()} and
@ref{1d48,,fromutc()} may not produce the result you want if the result is one of the
hours straddling the moment the standard offset changes.

Skipping code for error cases, the default @ref{1d48,,fromutc()} implementation acts
like:

@example
def fromutc(self, dt):
    # raise ValueError error if dt.tzinfo is not self
    dtoff = dt.utcoffset()
    dtdst = dt.dst()
    # raise ValueError if dtoff is None or dtdst is None
    delta = dtoff - dtdst  # this is self's standard offset
    if delta:
        dt += delta   # convert to standard local time
        dtdst = dt.dst()
        # raise ValueError if dtdst is None
    if dtdst:
        return dt + dtdst
    else:
        return dt
@end example
@end deffn

In the following @code{tzinfo_examples.py} file there are some examples of
@ref{3c6,,tzinfo} classes:

@example
from datetime import tzinfo, timedelta, datetime

ZERO = timedelta(0)
HOUR = timedelta(hours=1)
SECOND = timedelta(seconds=1)

# A class capturing the platform's idea of local time.
# (May result in wrong values on historical times in
#  timezones where UTC offset and/or the DST rules had
#  changed in the past.)
import time as _time

STDOFFSET = timedelta(seconds = -_time.timezone)
if _time.daylight:
    DSTOFFSET = timedelta(seconds = -_time.altzone)
else:
    DSTOFFSET = STDOFFSET

DSTDIFF = DSTOFFSET - STDOFFSET

class LocalTimezone(tzinfo):

    def fromutc(self, dt):
        assert dt.tzinfo is self
        stamp = (dt - datetime(1970, 1, 1, tzinfo=self)) // SECOND
        args = _time.localtime(stamp)[:6]
        dst_diff = DSTDIFF // SECOND
        # Detect fold
        fold = (args == _time.localtime(stamp - dst_diff))
        return datetime(*args, microsecond=dt.microsecond,
                        tzinfo=self, fold=fold)

    def utcoffset(self, dt):
        if self._isdst(dt):
            return DSTOFFSET
        else:
            return STDOFFSET

    def dst(self, dt):
        if self._isdst(dt):
            return DSTDIFF
        else:
            return ZERO

    def tzname(self, dt):
        return _time.tzname[self._isdst(dt)]

    def _isdst(self, dt):
        tt = (dt.year, dt.month, dt.day,
              dt.hour, dt.minute, dt.second,
              dt.weekday(), 0, 0)
        stamp = _time.mktime(tt)
        tt = _time.localtime(stamp)
        return tt.tm_isdst > 0

Local = LocalTimezone()


# A complete implementation of current DST rules for major US time zones.

def first_sunday_on_or_after(dt):
    days_to_go = 6 - dt.weekday()
    if days_to_go:
        dt += timedelta(days_to_go)
    return dt


# US DST Rules
#
# This is a simplified (i.e., wrong for a few cases) set of rules for US
# DST start and end times. For a complete and up-to-date set of DST rules
# and timezone definitions, visit the Olson Database (or try pytz):
# http://www.twinsun.com/tz/tz-link.htm
# http://sourceforge.net/projects/pytz/ (might not be up-to-date)
#
# In the US, since 2007, DST starts at 2am (standard time) on the second
# Sunday in March, which is the first Sunday on or after Mar 8.
DSTSTART_2007 = datetime(1, 3, 8, 2)
# and ends at 2am (DST time) on the first Sunday of Nov.
DSTEND_2007 = datetime(1, 11, 1, 2)
# From 1987 to 2006, DST used to start at 2am (standard time) on the first
# Sunday in April and to end at 2am (DST time) on the last
# Sunday of October, which is the first Sunday on or after Oct 25.
DSTSTART_1987_2006 = datetime(1, 4, 1, 2)
DSTEND_1987_2006 = datetime(1, 10, 25, 2)
# From 1967 to 1986, DST used to start at 2am (standard time) on the last
# Sunday in April (the one on or after April 24) and to end at 2am (DST time)
# on the last Sunday of October, which is the first Sunday
# on or after Oct 25.
DSTSTART_1967_1986 = datetime(1, 4, 24, 2)
DSTEND_1967_1986 = DSTEND_1987_2006

def us_dst_range(year):
    # Find start and end times for US DST. For years before 1967, return
    # start = end for no DST.
    if 2006 < year:
        dststart, dstend = DSTSTART_2007, DSTEND_2007
    elif 1986 < year < 2007:
        dststart, dstend = DSTSTART_1987_2006, DSTEND_1987_2006
    elif 1966 < year < 1987:
        dststart, dstend = DSTSTART_1967_1986, DSTEND_1967_1986
    else:
        return (datetime(year, 1, 1), ) * 2

    start = first_sunday_on_or_after(dststart.replace(year=year))
    end = first_sunday_on_or_after(dstend.replace(year=year))
    return start, end


class USTimeZone(tzinfo):

    def __init__(self, hours, reprname, stdname, dstname):
        self.stdoffset = timedelta(hours=hours)
        self.reprname = reprname
        self.stdname = stdname
        self.dstname = dstname

    def __repr__(self):
        return self.reprname

    def tzname(self, dt):
        if self.dst(dt):
            return self.dstname
        else:
            return self.stdname

    def utcoffset(self, dt):
        return self.stdoffset + self.dst(dt)

    def dst(self, dt):
        if dt is None or dt.tzinfo is None:
            # An exception may be sensible here, in one or both cases.
            # It depends on how you want to treat them.  The default
            # fromutc() implementation (called by the default astimezone()
            # implementation) passes a datetime with dt.tzinfo is self.
            return ZERO
        assert dt.tzinfo is self
        start, end = us_dst_range(dt.year)
        # Can't compare naive to aware objects, so strip the timezone from
        # dt first.
        dt = dt.replace(tzinfo=None)
        if start + HOUR <= dt < end - HOUR:
            # DST is in effect.
            return HOUR
        if end - HOUR <= dt < end:
            # Fold (an ambiguous hour): use dt.fold to disambiguate.
            return ZERO if dt.fold else HOUR
        if start <= dt < start + HOUR:
            # Gap (a non-existent hour): reverse the fold rule.
            return HOUR if dt.fold else ZERO
        # DST is off.
        return ZERO

    def fromutc(self, dt):
        assert dt.tzinfo is self
        start, end = us_dst_range(dt.year)
        start = start.replace(tzinfo=self)
        end = end.replace(tzinfo=self)
        std_time = dt + self.stdoffset
        dst_time = std_time + HOUR
        if end <= dst_time < end + HOUR:
            # Repeated hour
            return std_time.replace(fold=1)
        if std_time < start or dst_time >= end:
            # Standard time
            return std_time
        if start <= std_time < end - HOUR:
            # Daylight saving time
            return dst_time


Eastern  = USTimeZone(-5, "Eastern",  "EST", "EDT")
Central  = USTimeZone(-6, "Central",  "CST", "CDT")
Mountain = USTimeZone(-7, "Mountain", "MST", "MDT")
Pacific  = USTimeZone(-8, "Pacific",  "PST", "PDT")
@end example

Note that there are unavoidable subtleties twice per year in a @ref{3c6,,tzinfo}
subclass accounting for both standard and daylight time, at the DST transition
points. For concreteness, consider US Eastern (UTC -0500), where EDT begins the
minute after 1:59 (EST) on the second Sunday in March, and ends the minute after
1:59 (EDT) on the first Sunday in November:

@example
  UTC   3:MM  4:MM  5:MM  6:MM  7:MM  8:MM
  EST  22:MM 23:MM  0:MM  1:MM  2:MM  3:MM
  EDT  23:MM  0:MM  1:MM  2:MM  3:MM  4:MM

start  22:MM 23:MM  0:MM  1:MM  3:MM  4:MM

  end  23:MM  0:MM  1:MM  1:MM  2:MM  3:MM
@end example

When DST starts (the “start” line), the local wall clock leaps from 1:59 to
3:00. A wall time of the form 2:MM doesn’t really make sense on that day, so
@code{astimezone(Eastern)} won’t deliver a result with @code{hour == 2} on the day DST
begins. For example, at the Spring forward transition of 2016, we get:

@example
>>> from datetime import datetime, timezone
>>> from tzinfo_examples import HOUR, Eastern
>>> u0 = datetime(2016, 3, 13, 5, tzinfo=timezone.utc)
>>> for i in range(4):
...     u = u0 + i*HOUR
...     t = u.astimezone(Eastern)
...     print(u.time(), 'UTC =', t.time(), t.tzname())
...
05:00:00 UTC = 00:00:00 EST
06:00:00 UTC = 01:00:00 EST
07:00:00 UTC = 03:00:00 EDT
08:00:00 UTC = 04:00:00 EDT
@end example

When DST ends (the “end” line), there’s a potentially worse problem: there’s an
hour that can’t be spelled unambiguously in local wall time: the last hour of
daylight time. In Eastern, that’s times of the form 5:MM UTC on the day
daylight time ends. The local wall clock leaps from 1:59 (daylight time) back
to 1:00 (standard time) again. Local times of the form 1:MM are ambiguous.
@code{astimezone()} mimics the local clock’s behavior by mapping two adjacent UTC
hours into the same local hour then. In the Eastern example, UTC times of the
form 5:MM and 6:MM both map to 1:MM when converted to Eastern, but earlier times
have the @ref{79c,,fold} attribute set to 0 and the later times have it set to 1.
For example, at the Fall back transition of 2016, we get:

@example
>>> u0 = datetime(2016, 11, 6, 4, tzinfo=timezone.utc)
>>> for i in range(4):
...     u = u0 + i*HOUR
...     t = u.astimezone(Eastern)
...     print(u.time(), 'UTC =', t.time(), t.tzname(), t.fold)
...
04:00:00 UTC = 00:00:00 EDT 0
05:00:00 UTC = 01:00:00 EDT 0
06:00:00 UTC = 01:00:00 EST 1
07:00:00 UTC = 02:00:00 EST 0
@end example

Note that the @ref{363,,datetime} instances that differ only by the value of the
@ref{79c,,fold} attribute are considered equal in comparisons.

Applications that can’t bear wall-time ambiguities should explicitly check the
value of the @ref{79c,,fold} attribute or avoid using hybrid
@ref{3c6,,tzinfo} subclasses; there are no ambiguities when using @ref{c51,,timezone},
or any other fixed-offset @ref{3c6,,tzinfo} subclass (such as a class representing
only EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)).


@subsubheading See also


@quotation


@table @asis

@item @ref{14c,,zoneinfo}

The @ref{2f,,datetime} module has a basic @ref{c51,,timezone} class (for
handling arbitrary fixed offsets from UTC) and its @ref{1d67,,timezone.utc}
attribute (a UTC timezone instance).

@code{zoneinfo} brings the @emph{IANA timezone database} (also known as the Olson
database) to Python, and its usage is recommended.
@end table
@end quotation


@table @asis

@item IANA timezone database@footnote{https://www.iana.org/time-zones}

The Time Zone Database (often called tz, tzdata or zoneinfo) contains code
and data that represent the history of local time for many representative
locations around the globe. It is updated periodically to reflect changes
made by political bodies to time zone boundaries, UTC offsets, and
daylight-saving rules.
@end table


@node timezone Objects,strftime and strptime Behavior,tzinfo Objects,datetime — Basic date and time types
@anchor{library/datetime datetime-timezone}@anchor{1d68}@anchor{library/datetime timezone-objects}@anchor{1d69}
@subsubsection @code{timezone} Objects


The @ref{c51,,timezone} class is a subclass of @ref{3c6,,tzinfo}, each
instance of which represents a timezone defined by a fixed offset from
UTC.

Objects of this class cannot be used to represent timezone information in the
locations where different offsets are used in different days of the year or
where historical changes have been made to civil time.

@geindex timezone (class in datetime)
@anchor{library/datetime datetime timezone}@anchor{c51}
@deffn {Class} datetime.timezone (offset, name=None)

The @emph{offset} argument must be specified as a @ref{4c8,,timedelta}
object representing the difference between the local time and UTC. It must
be strictly between @code{-timedelta(hours=24)} and
@code{timedelta(hours=24)}, otherwise @ref{17c,,ValueError} is raised.

The @emph{name} argument is optional. If specified it must be a string that
will be used as the value returned by the @ref{1d49,,datetime.tzname()} method.


New in version 3.2.



Changed in version 3.7: The UTC offset is not restricted to a whole number of minutes.

@end deffn

@geindex utcoffset() (datetime.timezone method)
@anchor{library/datetime datetime timezone utcoffset}@anchor{1d6a}
@deffn {Method} timezone.utcoffset (dt)

Return the fixed value specified when the @ref{c51,,timezone} instance is
constructed.

The @emph{dt} argument is ignored. The return value is a @ref{4c8,,timedelta}
instance equal to the difference between the local time and UTC.


Changed in version 3.7: The UTC offset is not restricted to a whole number of minutes.

@end deffn

@geindex tzname() (datetime.timezone method)
@anchor{library/datetime datetime timezone tzname}@anchor{1d6b}
@deffn {Method} timezone.tzname (dt)

Return the fixed value specified when the @ref{c51,,timezone} instance
is constructed.

If @emph{name} is not provided in the constructor, the name returned by
@code{tzname(dt)} is generated from the value of the @code{offset} as follows. If
@emph{offset} is @code{timedelta(0)}, the name is “UTC”, otherwise it is a string in
the format @code{UTC±HH:MM}, where ± is the sign of @code{offset}, HH and MM are
two digits of @code{offset.hours} and @code{offset.minutes} respectively.


Changed in version 3.6: Name generated from @code{offset=timedelta(0)} is now plain @cite{‘UTC’}, not
@code{'UTC+00:00'}.

@end deffn

@geindex dst() (datetime.timezone method)
@anchor{library/datetime datetime timezone dst}@anchor{1d6c}
@deffn {Method} timezone.dst (dt)

Always returns @code{None}.
@end deffn

@geindex fromutc() (datetime.timezone method)
@anchor{library/datetime datetime timezone fromutc}@anchor{1d6d}
@deffn {Method} timezone.fromutc (dt)

Return @code{dt + offset}. The @emph{dt} argument must be an aware
@ref{363,,datetime} instance, with @code{tzinfo} set to @code{self}.
@end deffn

Class attributes:

@geindex utc (datetime.timezone attribute)
@anchor{library/datetime datetime timezone utc}@anchor{1d67}
@deffn {Attribute} timezone.utc

The UTC timezone, @code{timezone(timedelta(0))}.
@end deffn

@geindex % (percent); datetime format

@node strftime and strptime Behavior,,timezone Objects,datetime — Basic date and time types
@anchor{library/datetime strftime-and-strptime-behavior}@anchor{1d6e}@anchor{library/datetime strftime-strptime-behavior}@anchor{1d36}
@subsubsection @code{strftime()} and @code{strptime()} Behavior


@ref{3e2,,date}, @ref{363,,datetime}, and @ref{364,,time} objects all support a
@code{strftime(format)} method, to create a string representing the time under the
control of an explicit format string.

Conversely, the @ref{114f,,datetime.strptime()} class method creates a
@ref{363,,datetime} object from a string representing a date and time and a
corresponding format string.

The table below provides a high-level comparison of @code{strftime()}
versus @code{strptime()}:


@multitable {xxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

@tab

@code{strftime}

@tab

@code{strptime}

@item

Usage

@tab

Convert object to a string according to a given format

@tab

Parse a string into a @ref{363,,datetime} object given a corresponding format

@item

Type of method

@tab

Instance method

@tab

Class method

@item

Method of

@tab

@ref{3e2,,date}; @ref{363,,datetime}; @ref{364,,time}

@tab

@ref{363,,datetime}

@item

Signature

@tab

@code{strftime(format)}

@tab

@code{strptime(date_string, format)}

@end multitable


@menu
* strftime() and strptime() Format Codes: strftime and strptime Format Codes. 
* Technical Detail:: 

@end menu

@node strftime and strptime Format Codes,Technical Detail,,strftime and strptime Behavior
@anchor{library/datetime strftime-and-strptime-format-codes}@anchor{1d6f}
@subsubsection @code{strftime()} and @code{strptime()} Format Codes


The following is a list of all the format codes that the 1989 C standard
requires, and these work on all platforms with a standard C implementation.


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} 
@headitem

Directive

@tab

Meaning

@tab

Example

@tab

Notes

@item

@code{%a}

@tab

Weekday as locale’s
abbreviated name.

@tab


@display
Sun, Mon, …, Sat (en_US);@w{ }
So, Mo, …, Sa (de_DE)@w{ }
@end display



@tab

(1)

@item

@code{%A}

@tab

Weekday as locale’s full name.

@tab


@display
Sunday, Monday, …, Saturday (en_US);@w{ }
Sonntag, Montag, …, Samstag (de_DE)@w{ }
@end display



@tab

(1)

@item

@code{%w}

@tab

Weekday as a decimal number,
where 0 is Sunday and 6 is
Saturday.

@tab

0, 1, …, 6

@tab

@item

@code{%d}

@tab

Day of the month as a
zero-padded decimal number.

@tab

01, 02, …, 31

@tab

(9)

@item

@code{%b}

@tab

Month as locale’s abbreviated
name.

@tab


@display
Jan, Feb, …, Dec (en_US);@w{ }
Jan, Feb, …, Dez (de_DE)@w{ }
@end display



@tab

(1)

@item

@code{%B}

@tab

Month as locale’s full name.

@tab


@display
January, February, …, December (en_US);@w{ }
Januar, Februar, …, Dezember (de_DE)@w{ }
@end display



@tab

(1)

@item

@code{%m}

@tab

Month as a zero-padded
decimal number.

@tab

01, 02, …, 12

@tab

(9)

@item

@code{%y}

@tab

Year without century as a
zero-padded decimal number.

@tab

00, 01, …, 99

@tab

(9)

@item

@code{%Y}

@tab

Year with century as a decimal
number.

@tab

0001, 0002, …, 2013,
2014, …, 9998, 9999

@tab

(2)

@item

@code{%H}

@tab

Hour (24-hour clock) as a
zero-padded decimal number.

@tab

00, 01, …, 23

@tab

(9)

@item

@code{%I}

@tab

Hour (12-hour clock) as a
zero-padded decimal number.

@tab

01, 02, …, 12

@tab

(9)

@item

@code{%p}

@tab

Locale’s equivalent of either
AM or PM.

@tab


@display
AM, PM (en_US);@w{ }
am, pm (de_DE)@w{ }
@end display



@tab

(1),
(3)

@item

@code{%M}

@tab

Minute as a zero-padded
decimal number.

@tab

00, 01, …, 59

@tab

(9)

@item

@code{%S}

@tab

Second as a zero-padded
decimal number.

@tab

00, 01, …, 59

@tab

(4),
(9)

@item

@code{%f}

@tab

Microsecond as a decimal
number, zero-padded to 6
digits.

@tab

000000, 000001, …,
999999

@tab

(5)

@item

@code{%z}

@tab

UTC offset in the form
@code{±HHMM[SS[.ffffff]]} (empty
string if the object is
naive).

@tab

(empty), +0000,
-0400, +1030,
+063415,
-030712.345216

@tab

(6)

@item

@code{%Z}

@tab

Time zone name (empty string
if the object is naive).

@tab

(empty), UTC, GMT

@tab

(6)

@item

@code{%j}

@tab

Day of the year as a
zero-padded decimal number.

@tab

001, 002, …, 366

@tab

(9)

@item

@code{%U}

@tab

Week number of the year
(Sunday as the first day of
the week) as a zero-padded
decimal number. All days in a
new year preceding the first
Sunday are considered to be in
week 0.

@tab

00, 01, …, 53

@tab

(7),
(9)

@item

@code{%W}

@tab

Week number of the year
(Monday as the first day of
the week) as a zero-padded
decimal number. All days in a
new year preceding the first
Monday are considered to be in
week 0.

@tab

00, 01, …, 53

@tab

(7),
(9)

@item

@code{%c}

@tab

Locale’s appropriate date and
time representation.

@tab


@display
Tue Aug 16 21:30:00 1988 (en_US);@w{ }
Di 16 Aug 21:30:00 1988 (de_DE)@w{ }
@end display



@tab

(1)

@item

@code{%x}

@tab

Locale’s appropriate date
representation.

@tab


@display
08/16/88 (None);@w{ }
08/16/1988 (en_US);@w{ }
16.08.1988 (de_DE)@w{ }
@end display



@tab

(1)

@item

@code{%X}

@tab

Locale’s appropriate time
representation.

@tab


@display
21:30:00 (en_US);@w{ }
21:30:00 (de_DE)@w{ }
@end display



@tab

(1)

@item

@code{%%}

@tab

A literal @code{'%'} character.

@tab

%

@tab

@end multitable


Several additional directives not required by the C89 standard are included for
convenience. These parameters all correspond to ISO 8601 date values.


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} 
@headitem

Directive

@tab

Meaning

@tab

Example

@tab

Notes

@item

@code{%G}

@tab

ISO 8601 year with century
representing the year that
contains the greater part of
the ISO week (@code{%V}).

@tab

0001, 0002, …, 2013,
2014, …, 9998, 9999

@tab

(8)

@item

@code{%u}

@tab

ISO 8601 weekday as a decimal
number where 1 is Monday.

@tab

1, 2, …, 7

@tab

@item

@code{%V}

@tab

ISO 8601 week as a decimal
number with Monday as
the first day of the week.
Week 01 is the week containing
Jan 4.

@tab

01, 02, …, 53

@tab

(8),
(9)

@end multitable


These may not be available on all platforms when used with the @code{strftime()}
method. The ISO 8601 year and ISO 8601 week directives are not interchangeable
with the year and week number directives above. Calling @code{strptime()} with
incomplete or ambiguous ISO 8601 directives will raise a @ref{17c,,ValueError}.

The full set of format codes supported varies across platforms, because Python
calls the platform C library’s @code{strftime()} function, and platform
variations are common. To see the full set of format codes supported on your
platform, consult the @code{strftime(3)} documentation. There are also
differences between platforms in handling of unsupported format specifiers.


New in version 3.6: @code{%G}, @code{%u} and @code{%V} were added.


@node Technical Detail,,strftime and strptime Format Codes,strftime and strptime Behavior
@anchor{library/datetime technical-detail}@anchor{1d70}
@subsubsection Technical Detail


Broadly speaking, @code{d.strftime(fmt)} acts like the @ref{10a,,time} module’s
@code{time.strftime(fmt, d.timetuple())} although not all objects support a
@code{timetuple()} method.

For the @ref{114f,,datetime.strptime()} class method, the default value is
@code{1900-01-01T00:00:00.000}: any components not specified in the format string
will be pulled from the default value. @footnote{@w{(4)} 
Passing @code{datetime.strptime('Feb 29', '%b %d')} will fail since @code{1900} is not a leap year.
}

Using @code{datetime.strptime(date_string, format)} is equivalent to:

@example
datetime(*(time.strptime(date_string, format)[0:6]))
@end example

except when the format includes sub-second components or timezone offset
information, which are supported in @code{datetime.strptime} but are discarded by
@code{time.strptime}.

For @ref{364,,time} objects, the format codes for year, month, and day should not
be used, as @ref{10a,,time} objects have no such values. If they’re used anyway,
@code{1900} is substituted for the year, and @code{1} for the month and day.

For @ref{3e2,,date} objects, the format codes for hours, minutes, seconds, and
microseconds should not be used, as @ref{3e2,,date} objects have no such
values. If they’re used anyway, @code{0} is substituted for them.

For the same reason, handling of format strings containing Unicode code points
that can’t be represented in the charset of the current locale is also
platform-dependent. On some platforms such code points are preserved intact in
the output, while on others @code{strftime} may raise @ref{e49,,UnicodeError} or return
an empty string instead.

Notes:


@enumerate 

@item 
Because the format depends on the current locale, care should be taken when
making assumptions about the output value. Field orderings will vary (for
example, “month/day/year” versus “day/month/year”), and the output may
contain Unicode characters encoded using the locale’s default encoding (for
example, if the current locale is @code{ja_JP}, the default encoding could be
any one of @code{eucJP}, @code{SJIS}, or @code{utf-8}; use @ref{1b6,,locale.getlocale()}
to determine the current locale’s encoding).

@item 
The @code{strptime()} method can parse years in the full [1, 9999] range, but
years < 1000 must be zero-filled to 4-digit width.


Changed in version 3.2: In previous versions, @code{strftime()} method was restricted to
years >= 1900.



Changed in version 3.3: In version 3.2, @code{strftime()} method was restricted to
years >= 1000.


@item 
When used with the @code{strptime()} method, the @code{%p} directive only affects
the output hour field if the @code{%I} directive is used to parse the hour.

@item 
Unlike the @ref{10a,,time} module, the @ref{2f,,datetime} module does not support
leap seconds.

@item 
When used with the @code{strptime()} method, the @code{%f} directive
accepts from one to six digits and zero pads on the right. @code{%f} is
an extension to the set of format characters in the C standard (but
implemented separately in datetime objects, and therefore always
available).

@item 
For a naive object, the @code{%z} and @code{%Z} format codes are replaced by empty
strings.

For an aware object:


@table @asis

@item @code{%z}

@code{utcoffset()} is transformed into a string of the form
@code{±HHMM[SS[.ffffff]]}, where @code{HH} is a 2-digit string giving the number
of UTC offset hours, @code{MM} is a 2-digit string giving the number of UTC
offset minutes, @code{SS} is a 2-digit string giving the number of UTC offset
seconds and @code{ffffff} is a 6-digit string giving the number of UTC
offset microseconds. The @code{ffffff} part is omitted when the offset is a
whole number of seconds and both the @code{ffffff} and the @code{SS} part is
omitted when the offset is a whole number of minutes. For example, if
@code{utcoffset()} returns @code{timedelta(hours=-3, minutes=-30)}, @code{%z} is
replaced with the string @code{'-0330'}.
@end table


Changed in version 3.7: The UTC offset is not restricted to a whole number of minutes.



Changed in version 3.7: When the @code{%z} directive is provided to the  @code{strptime()} method,
the UTC offsets can have a colon as a separator between hours, minutes
and seconds.
For example, @code{'+01:00:00'} will be parsed as an offset of one hour.
In addition, providing @code{'Z'} is identical to @code{'+00:00'}.



@table @asis

@item @code{%Z}

In @code{strftime()}, @code{%Z} is replaced by an empty string if
@code{tzname()} returns @code{None}; otherwise @code{%Z} is replaced by the
returned value, which must be a string.

@code{strptime()} only accepts certain values for @code{%Z}:


@enumerate 

@item 
any value in @code{time.tzname} for your machine’s locale

@item 
the hard-coded values @code{UTC} and @code{GMT}
@end enumerate

So someone living in Japan may have @code{JST}, @code{UTC}, and @code{GMT} as
valid values, but probably not @code{EST}. It will raise @code{ValueError} for
invalid values.
@end table


Changed in version 3.2: When the @code{%z} directive is provided to the @code{strptime()} method, an
aware @ref{363,,datetime} object will be produced. The @code{tzinfo} of the
result will be set to a @ref{c51,,timezone} instance.


@item 
When used with the @code{strptime()} method, @code{%U} and @code{%W} are only used
in calculations when the day of the week and the calendar year (@code{%Y})
are specified.

@item 
Similar to @code{%U} and @code{%W}, @code{%V} is only used in calculations when the
day of the week and the ISO year (@code{%G}) are specified in a
@code{strptime()} format string. Also note that @code{%G} and @code{%Y} are not
interchangeable.

@item 
When used with the @code{strptime()} method, the leading zero is optional
for  formats @code{%d}, @code{%m}, @code{%H}, @code{%I}, @code{%M}, @code{%S}, @code{%J}, @code{%U},
@code{%W}, and @code{%V}. Format @code{%y} does require a leading zero.
@end enumerate

@node zoneinfo — IANA time zone support,calendar — General calendar-related functions,datetime — Basic date and time types,Data Types
@anchor{library/zoneinfo doc}@anchor{1d71}@anchor{library/zoneinfo module-zoneinfo}@anchor{14c}@anchor{library/zoneinfo zoneinfo-iana-time-zone-support}@anchor{1d72}
@subsection @code{zoneinfo} — IANA time zone support


@geindex module; zoneinfo


New in version 3.9.



__________________________________________________________________


The @ref{14c,,zoneinfo} module provides a concrete time zone implementation to
support the IANA time zone database as originally specified in 
@geindex Python Enhancement Proposals; PEP 615
PEP 615@footnote{https://www.python.org/dev/peps/pep-0615}. By
default, @ref{14c,,zoneinfo} uses the system’s time zone data if available; if no
system time zone data is available, the library will fall back to using the
first-party tzdata@footnote{https://pypi.org/project/tzdata/} package available on PyPI.


@subsubheading See also



@table @asis

@item Module: @ref{2f,,datetime}

Provides the @ref{364,,time} and @ref{363,,datetime}
types with which the @ref{3c5,,ZoneInfo} class is designed to be used.

@item Package tzdata@footnote{https://pypi.org/project/tzdata/}

First-party package maintained by the CPython core developers to supply
time zone data via PyPI.
@end table


@menu
* Using ZoneInfo:: 
* Data sources:: 
* The ZoneInfo class:: 
* Functions: Functions<2>. 
* Globals:: 
* Exceptions and warnings:: 

@end menu

@node Using ZoneInfo,Data sources,,zoneinfo — IANA time zone support
@anchor{library/zoneinfo using-zoneinfo}@anchor{1d73}
@subsubsection Using @code{ZoneInfo}


@ref{3c5,,ZoneInfo} is a concrete implementation of the @ref{3c6,,datetime.tzinfo}
abstract base class, and is intended to be attached to @code{tzinfo}, either via
the constructor, the @ref{1d45,,datetime.replace}
method or @ref{4c9,,datetime.astimezone}:

@example
>>> from zoneinfo import ZoneInfo
>>> from datetime import datetime, timedelta

>>> dt = datetime(2020, 10, 31, 12, tzinfo=ZoneInfo("America/Los_Angeles"))
>>> print(dt)
2020-10-31 12:00:00-07:00

>>> dt.tzname()
'PDT'
@end example

Datetimes constructed in this way are compatible with datetime arithmetic and
handle daylight saving time transitions with no further intervention:

@example
>>> dt_add = dt + timedelta(days=1)

>>> print(dt_add)
2020-11-01 12:00:00-08:00

>>> dt_add.tzname()
'PST'
@end example

These time zones also support the @ref{79c,,fold} attribute
introduced in 
@geindex Python Enhancement Proposals; PEP 495
PEP 495@footnote{https://www.python.org/dev/peps/pep-0495}.  During offset transitions which induce ambiguous
times (such as a daylight saving time to standard time transition), the offset
from @emph{before} the transition is used when @code{fold=0}, and the offset @emph{after}
the transition is used when @code{fold=1}, for example:

@example
>>> dt = datetime(2020, 11, 1, 1, tzinfo=ZoneInfo("America/Los_Angeles"))
>>> print(dt)
2020-11-01 01:00:00-07:00

>>> print(dt.replace(fold=1))
2020-11-01 01:00:00-08:00
@end example

When converting from another time zone, the fold will be set to the correct
value:

@example
>>> from datetime import timezone
>>> LOS_ANGELES = ZoneInfo("America/Los_Angeles")
>>> dt_utc = datetime(2020, 11, 1, 8, tzinfo=timezone.utc)

>>> # Before the PDT -> PST transition
>>> print(dt_utc.astimezone(LOS_ANGELES))
2020-11-01 01:00:00-07:00

>>> # After the PDT -> PST transition
>>> print((dt_utc + timedelta(hours=1)).astimezone(LOS_ANGELES))
2020-11-01 01:00:00-08:00
@end example

@node Data sources,The ZoneInfo class,Using ZoneInfo,zoneinfo — IANA time zone support
@anchor{library/zoneinfo data-sources}@anchor{1d74}
@subsubsection Data sources


The @code{zoneinfo} module does not directly provide time zone data, and instead
pulls time zone information from the system time zone database or the
first-party PyPI package tzdata@footnote{https://pypi.org/project/tzdata/}, if available. Some systems, including
notably Windows systems, do not have an IANA database available, and so for
projects targeting cross-platform compatibility that require time zone data, it
is recommended to declare a dependency on tzdata. If neither system data nor
tzdata are available, all calls to @ref{3c5,,ZoneInfo} will raise
@ref{1d75,,ZoneInfoNotFoundError}.

@menu
* Configuring the data sources:: 

@end menu

@node Configuring the data sources,,,Data sources
@anchor{library/zoneinfo configuring-the-data-sources}@anchor{1d76}@anchor{library/zoneinfo zoneinfo-data-configuration}@anchor{1d77}
@subsubsection Configuring the data sources


When @code{ZoneInfo(key)} is called, the constructor first searches the
directories specified in @ref{1647,,TZPATH} for a file matching @code{key}, and on
failure looks for a match in the tzdata package. This behavior can be
configured in three ways:


@enumerate 

@item 
The default @ref{1647,,TZPATH} when not otherwise specified can be configured at
@ref{1648,,compile time}.

@item 
@ref{1647,,TZPATH} can be configured using @ref{1d78,,an environment variable}.

@item 
At @ref{1d79,,runtime}, the search path can be
manipulated using the @ref{1d7a,,reset_tzpath()} function.
@end enumerate

@menu
* Compile-time configuration:: 
* Environment configuration:: 
* Runtime configuration:: 

@end menu

@node Compile-time configuration,Environment configuration,,Configuring the data sources
@anchor{library/zoneinfo compile-time-configuration}@anchor{1d7b}@anchor{library/zoneinfo zoneinfo-data-compile-time-config}@anchor{1648}
@subsubsection Compile-time configuration


The default @ref{1647,,TZPATH} includes several common deployment locations for the
time zone database (except on Windows, where there are no “well-known”
locations for time zone data). On POSIX systems, downstream distributors and
those building Python from source who know where their system
time zone data is deployed may change the default time zone path by specifying
the compile-time option @code{TZPATH} (or, more likely, the @ref{1646,,configure flag --with-tzpath}), which should be a string delimited by
@ref{161d,,os.pathsep}.

On all platforms, the configured value is available as the @code{TZPATH} key in
@ref{bbc,,sysconfig.get_config_var()}.

@node Environment configuration,Runtime configuration,Compile-time configuration,Configuring the data sources
@anchor{library/zoneinfo environment-configuration}@anchor{1d7c}@anchor{library/zoneinfo zoneinfo-data-environment-var}@anchor{1d78}
@subsubsection Environment configuration


When initializing @ref{1647,,TZPATH} (either at import time or whenever
@ref{1d7a,,reset_tzpath()} is called with no arguments), the @code{zoneinfo} module will
use the environment variable @code{PYTHONTZPATH}, if it exists, to set the search
path.

@geindex environment variable; PYTHONTZPATH
@anchor{library/zoneinfo envvar-PYTHONTZPATH}@anchor{1d7d}
@deffn {Environment Variable} PYTHONTZPATH

This is an @ref{161d,,os.pathsep}-separated string containing the time zone
search path to use. It must consist of only absolute rather than relative
paths. Relative components specified in @code{PYTHONTZPATH} will not be used,
but otherwise the behavior when a relative path is specified is
implementation-defined; CPython will raise @ref{1d7e,,InvalidTZPathWarning}, but
other implementations are free to silently ignore the erroneous component
or raise an exception.
@end deffn

To set the system to ignore the system data and use the tzdata package
instead, set @code{PYTHONTZPATH=""}.

@node Runtime configuration,,Environment configuration,Configuring the data sources
@anchor{library/zoneinfo runtime-configuration}@anchor{1d7f}@anchor{library/zoneinfo zoneinfo-data-runtime-config}@anchor{1d79}
@subsubsection Runtime configuration


The TZ search path can also be configured at runtime using the
@ref{1d7a,,reset_tzpath()} function. This is generally not an advisable operation,
though it is reasonable to use it in test functions that require the use of a
specific time zone path (or require disabling access to the system time zones).

@node The ZoneInfo class,Functions<2>,Data sources,zoneinfo — IANA time zone support
@anchor{library/zoneinfo the-zoneinfo-class}@anchor{1d80}
@subsubsection The @code{ZoneInfo} class


@geindex ZoneInfo (class in zoneinfo)
@anchor{library/zoneinfo zoneinfo ZoneInfo}@anchor{3c5}
@deffn {Class} zoneinfo.ZoneInfo (key)

A concrete @ref{3c6,,datetime.tzinfo} subclass that represents an IANA time
zone specified by the string @code{key}. Calls to the primary constructor will
always return objects that compare identically; put another way, barring
cache invalidation via @ref{1d81,,ZoneInfo.clear_cache()}, for all values of
@code{key}, the following assertion will always be true:

@example
a = ZoneInfo(key)
b = ZoneInfo(key)
assert a is b
@end example

@code{key} must be in the form of a relative, normalized POSIX path, with no
up-level references. The constructor will raise @ref{17c,,ValueError} if a
non-conforming key is passed.

If no file matching @code{key} is found, the constructor will raise
@ref{1d75,,ZoneInfoNotFoundError}.
@end deffn

The @code{ZoneInfo} class has two alternate constructors:

@geindex from_file() (zoneinfo.ZoneInfo class method)
@anchor{library/zoneinfo zoneinfo ZoneInfo from_file}@anchor{1d82}
@deffn {Method} classmethod ZoneInfo.from_file (fobj, /, key=None)

Constructs a @code{ZoneInfo} object from a file-like object returning bytes
(e.g. a file opened in binary mode or an @ref{a15,,io.BytesIO} object).
Unlike the primary constructor, this always constructs a new object.

The @code{key} parameter sets the name of the zone for the purposes of
@ref{178a,,__str__()} and @ref{617,,__repr__()}.

Objects created via this constructor cannot be pickled (see @ref{1d83,,pickling}).
@end deffn

@geindex no_cache() (zoneinfo.ZoneInfo class method)
@anchor{library/zoneinfo zoneinfo ZoneInfo no_cache}@anchor{1d84}
@deffn {Method} classmethod ZoneInfo.no_cache (key)

An alternate constructor that bypasses the constructor’s cache. It is
identical to the primary constructor, but returns a new object on each
call. This is most likely to be useful for testing or demonstration
purposes, but it can also be used to create a system with a different cache
invalidation strategy.

Objects created via this constructor will also bypass the cache of a
deserializing process when unpickled.

@c TODO: Add "See `cache_behavior`_" reference when that section is ready.

@cartouche
@quotation Caution 
Using this constructor may change the semantics of your datetimes in
surprising ways, only use it if you know that you need to.
@end quotation
@end cartouche
@end deffn

The following class methods are also available:

@geindex clear_cache() (zoneinfo.ZoneInfo class method)
@anchor{library/zoneinfo zoneinfo ZoneInfo clear_cache}@anchor{1d81}
@deffn {Method} classmethod ZoneInfo.clear_cache (*, only_keys=None)

A method for invalidating the cache on the @code{ZoneInfo} class. If no
arguments are passed, all caches are invalidated and the next call to
the primary constructor for each key will return a new instance.

If an iterable of key names is passed to the @code{only_keys} parameter, only
the specified keys will be removed from the cache. Keys passed to
@code{only_keys} but not found in the cache are ignored.

@c TODO: Add "See `cache_behavior`_" reference when that section is ready.

@cartouche
@quotation Warning 
Invoking this function may change the semantics of datetimes using
@code{ZoneInfo} in surprising ways; this modifies process-wide global state
and thus may have wide-ranging effects. Only use it if you know that you
need to.
@end quotation
@end cartouche
@end deffn

The class has one attribute:

@geindex key (zoneinfo.ZoneInfo attribute)
@anchor{library/zoneinfo zoneinfo ZoneInfo key}@anchor{1d85}
@deffn {Attribute} ZoneInfo.key

This is a read-only @ref{1d86,,attribute} that returns the value of @code{key}
passed to the constructor, which should be a lookup key in the IANA time
zone database (e.g. @code{America/New_York}, @code{Europe/Paris} or
@code{Asia/Tokyo}).

For zones constructed from file without specifying a @code{key} parameter,
this will be set to @code{None}.

@cartouche
@quotation Note 
Although it is a somewhat common practice to expose these to end users,
these values are designed to be primary keys for representing the
relevant zones and not necessarily user-facing elements.  Projects like
CLDR (the Unicode Common Locale Data Repository) can be used to get
more user-friendly strings from these keys.
@end quotation
@end cartouche
@end deffn

@menu
* String representations:: 
* Pickle serialization:: 

@end menu

@node String representations,Pickle serialization,,The ZoneInfo class
@anchor{library/zoneinfo string-representations}@anchor{1d87}
@subsubsection String representations


The string representation returned when calling @ref{2ef,,str} on a
@ref{3c5,,ZoneInfo} object defaults to using the @ref{1d85,,ZoneInfo.key} attribute (see
the note on usage in the attribute documentation):

@example
>>> zone = ZoneInfo("Pacific/Kwajalein")
>>> str(zone)
'Pacific/Kwajalein'

>>> dt = datetime(2020, 4, 1, 3, 15, tzinfo=zone)
>>> f"@{dt.isoformat()@} [@{dt.tzinfo@}]"
'2020-04-01T03:15:00+12:00 [Pacific/Kwajalein]'
@end example

For objects constructed from a file without specifying a @code{key} parameter,
@code{str} falls back to calling @ref{27d,,repr()}. @code{ZoneInfo}’s @code{repr} is
implementation-defined and not necessarily stable between versions, but it is
guaranteed not to be a valid @code{ZoneInfo} key.

@node Pickle serialization,,String representations,The ZoneInfo class
@anchor{library/zoneinfo pickle-serialization}@anchor{1d88}@anchor{library/zoneinfo pickling}@anchor{1d83}
@subsubsection Pickle serialization


Rather than serializing all transition data, @code{ZoneInfo} objects are
serialized by key, and @code{ZoneInfo} objects constructed from files (even those
with a value for @code{key} specified) cannot be pickled.

The behavior of a @code{ZoneInfo} file depends on how it was constructed:


@enumerate 

@item 
@code{ZoneInfo(key)}: When constructed with the primary constructor, a
@code{ZoneInfo} object is serialized by key, and when deserialized, the
deserializing process uses the primary and thus it is expected that these
are expected to be the same object as other references to the same time
zone.  For example, if @code{europe_berlin_pkl} is a string containing a pickle
constructed from @code{ZoneInfo("Europe/Berlin")}, one would expect the
following behavior:

@example
>>> a = ZoneInfo("Europe/Berlin")
>>> b = pickle.loads(europe_berlin_pkl)
>>> a is b
True
@end example

@item 
@code{ZoneInfo.no_cache(key)}: When constructed from the cache-bypassing
constructor, the @code{ZoneInfo} object is also serialized by key, but when
deserialized, the deserializing process uses the cache bypassing
constructor. If @code{europe_berlin_pkl_nc} is a string containing a pickle
constructed from @code{ZoneInfo.no_cache("Europe/Berlin")}, one would expect
the following behavior:

@example
>>> a = ZoneInfo("Europe/Berlin")
>>> b = pickle.loads(europe_berlin_pkl_nc)
>>> a is b
False
@end example

@item 
@code{ZoneInfo.from_file(fobj, /, key=None)}: When constructed from a file, the
@code{ZoneInfo} object raises an exception on pickling. If an end user wants to
pickle a @code{ZoneInfo} constructed from a file, it is recommended that they
use a wrapper type or a custom serialization function: either serializing by
key or storing the contents of the file object and serializing that.
@end enumerate

This method of serialization requires that the time zone data for the required
key be available on both the serializing and deserializing side, similar to the
way that references to classes and functions are expected to exist in both the
serializing and deserializing environments. It also means that no guarantees
are made about the consistency of results when unpickling a @code{ZoneInfo}
pickled in an environment with a different version of the time zone data.

@node Functions<2>,Globals,The ZoneInfo class,zoneinfo — IANA time zone support
@anchor{library/zoneinfo functions}@anchor{1d89}
@subsubsection Functions


@geindex available_timezones() (in module zoneinfo)
@anchor{library/zoneinfo zoneinfo available_timezones}@anchor{11e5}
@deffn {Function} zoneinfo.available_timezones ()

Get a set containing all the valid keys for IANA time zones available
anywhere on the time zone path. This is recalculated on every call to the
function.

This function only includes canonical zone names and does not include
“special” zones such as those under the @code{posix/} and @code{right/}
directories, or the @code{posixrules} zone.

@cartouche
@quotation Caution 
This function may open a large number of files, as the best way to
determine if a file on the time zone path is a valid time zone is to
read the “magic string” at the beginning.
@end quotation
@end cartouche

@cartouche
@quotation Note 
These values are not designed to be exposed to end-users; for user
facing elements, applications should use something like CLDR (the
Unicode Common Locale Data Repository) to get more user-friendly
strings. See also the cautionary note on @ref{1d85,,ZoneInfo.key}.
@end quotation
@end cartouche
@end deffn

@geindex reset_tzpath() (in module zoneinfo)
@anchor{library/zoneinfo zoneinfo reset_tzpath}@anchor{1d7a}
@deffn {Function} zoneinfo.reset_tzpath (to=None)

Sets or resets the time zone search path (@ref{1647,,TZPATH}) for the module.
When called with no arguments, @ref{1647,,TZPATH} is set to the default value.

Calling @code{reset_tzpath} will not invalidate the @ref{3c5,,ZoneInfo} cache,
and so calls to the primary @code{ZoneInfo} constructor will only use the new
@code{TZPATH} in the case of a cache miss.

The @code{to} parameter must be a @ref{14e7,,sequence} of strings or
@ref{795,,os.PathLike} and not a string, all of which must be absolute paths.
@ref{17c,,ValueError} will be raised if something other than an absolute path
is passed.
@end deffn

@node Globals,Exceptions and warnings,Functions<2>,zoneinfo — IANA time zone support
@anchor{library/zoneinfo globals}@anchor{1d8a}
@subsubsection Globals


@geindex TZPATH (in module zoneinfo)
@anchor{library/zoneinfo zoneinfo TZPATH}@anchor{1647}
@deffn {Data} zoneinfo.TZPATH

A read-only sequence representing the time zone search path – when
constructing a @code{ZoneInfo} from a key, the key is joined to each entry in
the @code{TZPATH}, and the first file found is used.

@code{TZPATH} may contain only absolute paths, never relative paths,
regardless of how it is configured.

The object that @code{zoneinfo.TZPATH} points to may change in response to a
call to @ref{1d7a,,reset_tzpath()}, so it is recommended to use
@code{zoneinfo.TZPATH} rather than importing @code{TZPATH} from @code{zoneinfo} or
assigning a long-lived variable to @code{zoneinfo.TZPATH}.

For more information on configuring the time zone search path, see
@ref{1d77,,Configuring the data sources}.
@end deffn

@node Exceptions and warnings,,Globals,zoneinfo — IANA time zone support
@anchor{library/zoneinfo exceptions-and-warnings}@anchor{1d8b}
@subsubsection Exceptions and warnings


@geindex ZoneInfoNotFoundError
@anchor{library/zoneinfo zoneinfo ZoneInfoNotFoundError}@anchor{1d75}
@deffn {Exception} zoneinfo.ZoneInfoNotFoundError

Raised when construction of a @ref{3c5,,ZoneInfo} object fails because the
specified key could not be found on the system. This is a subclass of
@ref{5bb,,KeyError}.
@end deffn

@geindex InvalidTZPathWarning
@anchor{library/zoneinfo zoneinfo InvalidTZPathWarning}@anchor{1d7e}
@deffn {Exception} zoneinfo.InvalidTZPathWarning

Raised when 
@geindex PYTHONTZPATH
@geindex environment variable; PYTHONTZPATH
@ref{1d7d,,PYTHONTZPATH} contains an invalid component that will
be filtered out, such as a relative path.
@end deffn

@c Links and references:

@node calendar — General calendar-related functions,collections — Container datatypes,zoneinfo — IANA time zone support,Data Types
@anchor{library/calendar doc}@anchor{1d8c}@anchor{library/calendar calendar-general-calendar-related-functions}@anchor{1d8d}@anchor{library/calendar module-calendar}@anchor{13}@anchor{library/calendar tzdata}@anchor{1d8e}
@subsection @code{calendar} — General calendar-related functions


@geindex module; calendar

@strong{Source code:} Lib/calendar.py@footnote{https://github.com/python/cpython/tree/main/Lib/calendar.py}


__________________________________________________________________


This module allows you to output calendars like the Unix @code{cal} program,
and provides additional useful functions related to the calendar. By default,
these calendars have Monday as the first day of the week, and Sunday as the last
(the European convention). Use @ref{1d8f,,setfirstweekday()} to set the first day of
the week to Sunday (6) or to any other weekday.  Parameters that specify dates
are given as integers. For related
functionality, see also the @ref{2f,,datetime} and @ref{10a,,time} modules.

The functions and classes defined in this module
use an idealized calendar, the current Gregorian calendar extended indefinitely
in both directions.  This matches the definition of the “proleptic Gregorian”
calendar in Dershowitz and Reingold’s book “Calendrical Calculations”, where
it’s the base calendar for all computations.  Zero and negative years are
interpreted as prescribed by the ISO 8601 standard.  Year 0 is 1 BC, year -1 is
2 BC, and so on.

@geindex Calendar (class in calendar)
@anchor{library/calendar calendar Calendar}@anchor{1d90}
@deffn {Class} calendar.Calendar (firstweekday=0)

Creates a @ref{1d90,,Calendar} object. @emph{firstweekday} is an integer specifying the
first day of the week. @ref{1d91,,MONDAY} is @code{0} (the default), @ref{1d92,,SUNDAY} is @code{6}.

A @ref{1d90,,Calendar} object provides several methods that can be used for
preparing the calendar data for formatting. This class doesn’t do any formatting
itself. This is the job of subclasses.

@ref{1d90,,Calendar} instances have the following methods:

@geindex iterweekdays() (calendar.Calendar method)
@anchor{library/calendar calendar Calendar iterweekdays}@anchor{1d93}
@deffn {Method} iterweekdays ()

Return an iterator for the week day numbers that will be used for one
week.  The first value from the iterator will be the same as the value of
the @ref{1d94,,firstweekday} property.
@end deffn

@geindex itermonthdates() (calendar.Calendar method)
@anchor{library/calendar calendar Calendar itermonthdates}@anchor{755}
@deffn {Method} itermonthdates (year, month)

Return an iterator for the month @emph{month} (1–12) in the year @emph{year}. This
iterator will return all days (as @ref{3e2,,datetime.date} objects) for the
month and all days before the start of the month or after the end of the
month that are required to get a complete week.
@end deffn

@geindex itermonthdays() (calendar.Calendar method)
@anchor{library/calendar calendar Calendar itermonthdays}@anchor{1d95}
@deffn {Method} itermonthdays (year, month)

Return an iterator for the month @emph{month} in the year @emph{year} similar to
@ref{755,,itermonthdates()}, but not restricted by the @ref{3e2,,datetime.date}
range. Days returned will simply be day of the month numbers.  For the
days outside of the specified month, the day number is @code{0}.
@end deffn

@geindex itermonthdays2() (calendar.Calendar method)
@anchor{library/calendar calendar Calendar itermonthdays2}@anchor{1d96}
@deffn {Method} itermonthdays2 (year, month)

Return an iterator for the month @emph{month} in the year @emph{year} similar to
@ref{755,,itermonthdates()}, but not restricted by the @ref{3e2,,datetime.date}
range. Days returned will be tuples consisting of a day of the month
number and a week day number.
@end deffn

@geindex itermonthdays3() (calendar.Calendar method)
@anchor{library/calendar calendar Calendar itermonthdays3}@anchor{756}
@deffn {Method} itermonthdays3 (year, month)

Return an iterator for the month @emph{month} in the year @emph{year} similar to
@ref{755,,itermonthdates()}, but not restricted by the @ref{3e2,,datetime.date}
range. Days returned will be tuples consisting of a year, a month and a day
of the month numbers.


New in version 3.7.

@end deffn

@geindex itermonthdays4() (calendar.Calendar method)
@anchor{library/calendar calendar Calendar itermonthdays4}@anchor{757}
@deffn {Method} itermonthdays4 (year, month)

Return an iterator for the month @emph{month} in the year @emph{year} similar to
@ref{755,,itermonthdates()}, but not restricted by the @ref{3e2,,datetime.date}
range. Days returned will be tuples consisting of a year, a month, a day
of the month, and a day of the week numbers.


New in version 3.7.

@end deffn

@geindex monthdatescalendar() (calendar.Calendar method)
@anchor{library/calendar calendar Calendar monthdatescalendar}@anchor{1d97}
@deffn {Method} monthdatescalendar (year, month)

Return a list of the weeks in the month @emph{month} of the @emph{year} as full
weeks.  Weeks are lists of seven @ref{3e2,,datetime.date} objects.
@end deffn

@geindex monthdays2calendar() (calendar.Calendar method)
@anchor{library/calendar calendar Calendar monthdays2calendar}@anchor{1d98}
@deffn {Method} monthdays2calendar (year, month)

Return a list of the weeks in the month @emph{month} of the @emph{year} as full
weeks.  Weeks are lists of seven tuples of day numbers and weekday
numbers.
@end deffn

@geindex monthdayscalendar() (calendar.Calendar method)
@anchor{library/calendar calendar Calendar monthdayscalendar}@anchor{1d99}
@deffn {Method} monthdayscalendar (year, month)

Return a list of the weeks in the month @emph{month} of the @emph{year} as full
weeks.  Weeks are lists of seven day numbers.
@end deffn

@geindex yeardatescalendar() (calendar.Calendar method)
@anchor{library/calendar calendar Calendar yeardatescalendar}@anchor{1d9a}
@deffn {Method} yeardatescalendar (year, width=3)

Return the data for the specified year ready for formatting. The return
value is a list of month rows. Each month row contains up to @emph{width}
months (defaulting to 3). Each month contains between 4 and 6 weeks and
each week contains 1–7 days. Days are @ref{3e2,,datetime.date} objects.
@end deffn

@geindex yeardays2calendar() (calendar.Calendar method)
@anchor{library/calendar calendar Calendar yeardays2calendar}@anchor{1d9b}
@deffn {Method} yeardays2calendar (year, width=3)

Return the data for the specified year ready for formatting (similar to
@ref{1d9a,,yeardatescalendar()}). Entries in the week lists are tuples of day
numbers and weekday numbers. Day numbers outside this month are zero.
@end deffn

@geindex yeardayscalendar() (calendar.Calendar method)
@anchor{library/calendar calendar Calendar yeardayscalendar}@anchor{1d9c}
@deffn {Method} yeardayscalendar (year, width=3)

Return the data for the specified year ready for formatting (similar to
@ref{1d9a,,yeardatescalendar()}). Entries in the week lists are day numbers. Day
numbers outside this month are zero.
@end deffn
@end deffn

@geindex TextCalendar (class in calendar)
@anchor{library/calendar calendar TextCalendar}@anchor{1d9d}
@deffn {Class} calendar.TextCalendar (firstweekday=0)

This class can be used to generate plain text calendars.

@ref{1d9d,,TextCalendar} instances have the following methods:

@geindex formatmonth() (calendar.TextCalendar method)
@anchor{library/calendar calendar TextCalendar formatmonth}@anchor{1d9e}
@deffn {Method} formatmonth (theyear, themonth, w=0, l=0)

Return a month’s calendar in a multi-line string. If @emph{w} is provided, it
specifies the width of the date columns, which are centered. If @emph{l} is
given, it specifies the number of lines that each week will use. Depends
on the first weekday as specified in the constructor or set by the
@ref{1d8f,,setfirstweekday()} method.
@end deffn

@geindex prmonth() (calendar.TextCalendar method)
@anchor{library/calendar calendar TextCalendar prmonth}@anchor{1d9f}
@deffn {Method} prmonth (theyear, themonth, w=0, l=0)

Print a month’s calendar as returned by @ref{1d9e,,formatmonth()}.
@end deffn

@geindex formatyear() (calendar.TextCalendar method)
@anchor{library/calendar calendar TextCalendar formatyear}@anchor{1da0}
@deffn {Method} formatyear (theyear, w=2, l=1, c=6, m=3)

Return a @emph{m}-column calendar for an entire year as a multi-line string.
Optional parameters @emph{w}, @emph{l}, and @emph{c} are for date column width, lines per
week, and number of spaces between month columns, respectively. Depends on
the first weekday as specified in the constructor or set by the
@ref{1d8f,,setfirstweekday()} method.  The earliest year for which a calendar
can be generated is platform-dependent.
@end deffn

@geindex pryear() (calendar.TextCalendar method)
@anchor{library/calendar calendar TextCalendar pryear}@anchor{1da1}
@deffn {Method} pryear (theyear, w=2, l=1, c=6, m=3)

Print the calendar for an entire year as returned by @ref{1da0,,formatyear()}.
@end deffn
@end deffn

@geindex HTMLCalendar (class in calendar)
@anchor{library/calendar calendar HTMLCalendar}@anchor{642}
@deffn {Class} calendar.HTMLCalendar (firstweekday=0)

This class can be used to generate HTML calendars.

@code{HTMLCalendar} instances have the following methods:

@geindex formatmonth() (calendar.HTMLCalendar method)
@anchor{library/calendar calendar HTMLCalendar formatmonth}@anchor{1da2}
@deffn {Method} formatmonth (theyear, themonth, withyear=True)

Return a month’s calendar as an HTML table. If @emph{withyear} is true the year
will be included in the header, otherwise just the month name will be
used.
@end deffn

@geindex formatyear() (calendar.HTMLCalendar method)
@anchor{library/calendar calendar HTMLCalendar formatyear}@anchor{1da3}
@deffn {Method} formatyear (theyear, width=3)

Return a year’s calendar as an HTML table. @emph{width} (defaulting to 3)
specifies the number of months per row.
@end deffn

@geindex formatyearpage() (calendar.HTMLCalendar method)
@anchor{library/calendar calendar HTMLCalendar formatyearpage}@anchor{1da4}
@deffn {Method} formatyearpage (theyear, width=3, css='calendar.css', encoding=None)

Return a year’s calendar as a complete HTML page. @emph{width} (defaulting to
3) specifies the number of months per row. @emph{css} is the name for the
cascading style sheet to be used. @ref{182,,None} can be passed if no style
sheet should be used. @emph{encoding} specifies the encoding to be used for the
output (defaulting to the system default encoding).
@end deffn

@code{HTMLCalendar} has the following attributes you can override to
customize the CSS classes used by the calendar:

@geindex cssclasses (calendar.HTMLCalendar attribute)
@anchor{library/calendar calendar HTMLCalendar cssclasses}@anchor{1da5}
@deffn {Attribute} cssclasses

A list of CSS classes used for each weekday. The default class list is:

@example
cssclasses = ["mon", "tue", "wed", "thu", "fri", "sat", "sun"]
@end example

more styles can be added for each day:

@example
cssclasses = ["mon text-bold", "tue", "wed", "thu", "fri", "sat", "sun red"]
@end example

Note that the length of this list must be seven items.
@end deffn

@geindex cssclass_noday (calendar.HTMLCalendar attribute)
@anchor{library/calendar calendar HTMLCalendar cssclass_noday}@anchor{1da6}
@deffn {Attribute} cssclass_noday

The CSS class for a weekday occurring in the previous or coming month.


New in version 3.7.

@end deffn

@geindex cssclasses_weekday_head (calendar.HTMLCalendar attribute)
@anchor{library/calendar calendar HTMLCalendar cssclasses_weekday_head}@anchor{1da7}
@deffn {Attribute} cssclasses_weekday_head

A list of CSS classes used for weekday names in the header row.
The default is the same as @ref{1da5,,cssclasses}.


New in version 3.7.

@end deffn

@geindex cssclass_month_head (calendar.HTMLCalendar attribute)
@anchor{library/calendar calendar HTMLCalendar cssclass_month_head}@anchor{1da8}
@deffn {Attribute} cssclass_month_head

The month’s head CSS class (used by @code{formatmonthname()}).
The default value is @code{"month"}.


New in version 3.7.

@end deffn

@geindex cssclass_month (calendar.HTMLCalendar attribute)
@anchor{library/calendar calendar HTMLCalendar cssclass_month}@anchor{1da9}
@deffn {Attribute} cssclass_month

The CSS class for the whole month’s table (used by @ref{1da2,,formatmonth()}).
The default value is @code{"month"}.


New in version 3.7.

@end deffn

@geindex cssclass_year (calendar.HTMLCalendar attribute)
@anchor{library/calendar calendar HTMLCalendar cssclass_year}@anchor{1daa}
@deffn {Attribute} cssclass_year

The CSS class for the whole year’s table of tables (used by
@ref{1da3,,formatyear()}). The default value is @code{"year"}.


New in version 3.7.

@end deffn

@geindex cssclass_year_head (calendar.HTMLCalendar attribute)
@anchor{library/calendar calendar HTMLCalendar cssclass_year_head}@anchor{1dab}
@deffn {Attribute} cssclass_year_head

The CSS class for the table head for the whole year (used by
@ref{1da3,,formatyear()}). The default value is @code{"year"}.


New in version 3.7.

@end deffn

Note that although the naming for the above described class attributes is
singular (e.g. @code{cssclass_month} @code{cssclass_noday}), one can replace the
single CSS class with a space separated list of CSS classes, for example:

@example
"text-bold text-red"
@end example

Here is an example how @code{HTMLCalendar} can be customized:

@example
class CustomHTMLCal(calendar.HTMLCalendar):
    cssclasses = [style + " text-nowrap" for style in
                  calendar.HTMLCalendar.cssclasses]
    cssclass_month_head = "text-center month-head"
    cssclass_month = "text-center month"
    cssclass_year = "text-italic lead"
@end example
@end deffn

@geindex LocaleTextCalendar (class in calendar)
@anchor{library/calendar calendar LocaleTextCalendar}@anchor{1cd}
@deffn {Class} calendar.LocaleTextCalendar (firstweekday=0, locale=None)

This subclass of @ref{1d9d,,TextCalendar} can be passed a locale name in the
constructor and will return month and weekday names in the specified locale.
@end deffn

@geindex LocaleHTMLCalendar (class in calendar)
@anchor{library/calendar calendar LocaleHTMLCalendar}@anchor{1ce}
@deffn {Class} calendar.LocaleHTMLCalendar (firstweekday=0, locale=None)

This subclass of @ref{642,,HTMLCalendar} can be passed a locale name in the
constructor and will return month and weekday names in the specified
locale.
@end deffn

@cartouche
@quotation Note 
The constructor, @code{formatweekday()} and @code{formatmonthname()} methods
of these two classes temporarily change the @code{LC_TIME} locale to the given
@emph{locale}. Because the current locale is a process-wide setting, they are
not thread-safe.
@end quotation
@end cartouche

For simple text calendars this module provides the following functions.

@geindex setfirstweekday() (in module calendar)
@anchor{library/calendar calendar setfirstweekday}@anchor{1d8f}
@deffn {Function} calendar.setfirstweekday (weekday)

Sets the weekday (@code{0} is Monday, @code{6} is Sunday) to start each week. The
values @ref{1d91,,MONDAY}, @ref{1dac,,TUESDAY}, @ref{1dad,,WEDNESDAY}, @ref{1dae,,THURSDAY},
@ref{1daf,,FRIDAY}, @ref{1db0,,SATURDAY}, and @ref{1d92,,SUNDAY} are provided for
convenience. For example, to set the first weekday to Sunday:

@example
import calendar
calendar.setfirstweekday(calendar.SUNDAY)
@end example
@end deffn

@geindex firstweekday() (in module calendar)
@anchor{library/calendar calendar firstweekday}@anchor{1d94}
@deffn {Function} calendar.firstweekday ()

Returns the current setting for the weekday to start each week.
@end deffn

@geindex isleap() (in module calendar)
@anchor{library/calendar calendar isleap}@anchor{1db1}
@deffn {Function} calendar.isleap (year)

Returns @ref{748,,True} if @emph{year} is a leap year, otherwise @ref{659,,False}.
@end deffn

@geindex leapdays() (in module calendar)
@anchor{library/calendar calendar leapdays}@anchor{1db2}
@deffn {Function} calendar.leapdays (y1, y2)

Returns the number of leap years in the range from @emph{y1} to @emph{y2} (exclusive),
where @emph{y1} and @emph{y2} are years.

This function works for ranges spanning a century change.
@end deffn

@geindex weekday() (in module calendar)
@anchor{library/calendar calendar weekday}@anchor{1db3}
@deffn {Function} calendar.weekday (year, month, day)

Returns the day of the week (@code{0} is Monday) for @emph{year} (@code{1970}–…),
@emph{month} (@code{1}–@code{12}), @emph{day} (@code{1}–@code{31}).
@end deffn

@geindex weekheader() (in module calendar)
@anchor{library/calendar calendar weekheader}@anchor{1db4}
@deffn {Function} calendar.weekheader (n)

Return a header containing abbreviated weekday names. @emph{n} specifies the width in
characters for one weekday.
@end deffn

@geindex monthrange() (in module calendar)
@anchor{library/calendar calendar monthrange}@anchor{1db5}
@deffn {Function} calendar.monthrange (year, month)

Returns weekday of first day of the month and number of days in month,  for the
specified @emph{year} and @emph{month}.
@end deffn

@geindex monthcalendar() (in module calendar)
@anchor{library/calendar calendar monthcalendar}@anchor{1db6}
@deffn {Function} calendar.monthcalendar (year, month)

Returns a matrix representing a month’s calendar.  Each row represents a week;
days outside of the month are represented by zeros. Each week begins with Monday
unless set by @ref{1d8f,,setfirstweekday()}.
@end deffn

@geindex prmonth() (in module calendar)
@anchor{library/calendar calendar prmonth}@anchor{1db7}
@deffn {Function} calendar.prmonth (theyear, themonth, w=0, l=0)

Prints a month’s calendar as returned by @ref{1db8,,month()}.
@end deffn

@geindex month() (in module calendar)
@anchor{library/calendar calendar month}@anchor{1db8}
@deffn {Function} calendar.month (theyear, themonth, w=0, l=0)

Returns a month’s calendar in a multi-line string using the @code{formatmonth()}
of the @ref{1d9d,,TextCalendar} class.
@end deffn

@geindex prcal() (in module calendar)
@anchor{library/calendar calendar prcal}@anchor{1db9}
@deffn {Function} calendar.prcal (year, w=0, l=0, c=6, m=3)

Prints the calendar for an entire year as returned by  @ref{13,,calendar()}.
@end deffn

@geindex calendar() (in module calendar)
@anchor{library/calendar calendar calendar}@anchor{1dba}
@deffn {Function} calendar.calendar (year, w=2, l=1, c=6, m=3)

Returns a 3-column calendar for an entire year as a multi-line string using
the @code{formatyear()} of the @ref{1d9d,,TextCalendar} class.
@end deffn

@geindex timegm() (in module calendar)
@anchor{library/calendar calendar timegm}@anchor{1dbb}
@deffn {Function} calendar.timegm (tuple)

An unrelated but handy function that takes a time tuple such as returned by
the @ref{d65,,gmtime()} function in the @ref{10a,,time} module, and returns the
corresponding Unix timestamp value, assuming an epoch of 1970, and the POSIX
encoding.  In fact, @ref{d65,,time.gmtime()} and @ref{1dbb,,timegm()} are each others’
inverse.
@end deffn

The @ref{13,,calendar} module exports the following data attributes:

@geindex day_name (in module calendar)
@anchor{library/calendar calendar day_name}@anchor{1dbc}
@deffn {Data} calendar.day_name

An array that represents the days of the week in the current locale.
@end deffn

@geindex day_abbr (in module calendar)
@anchor{library/calendar calendar day_abbr}@anchor{1dbd}
@deffn {Data} calendar.day_abbr

An array that represents the abbreviated days of the week in the current locale.
@end deffn

@geindex month_name (in module calendar)
@anchor{library/calendar calendar month_name}@anchor{1dbe}
@deffn {Data} calendar.month_name

An array that represents the months of the year in the current locale.  This
follows normal convention of January being month number 1, so it has a length of
13 and  @code{month_name[0]} is the empty string.
@end deffn

@geindex month_abbr (in module calendar)
@anchor{library/calendar calendar month_abbr}@anchor{1dbf}
@deffn {Data} calendar.month_abbr

An array that represents the abbreviated months of the year in the current
locale.  This follows normal convention of January being month number 1, so it
has a length of 13 and  @code{month_abbr[0]} is the empty string.
@end deffn

@geindex MONDAY (in module calendar)
@geindex TUESDAY (in module calendar)
@geindex WEDNESDAY (in module calendar)
@geindex THURSDAY (in module calendar)
@geindex FRIDAY (in module calendar)
@geindex SATURDAY (in module calendar)
@geindex SUNDAY (in module calendar)
@anchor{library/calendar calendar MONDAY}@anchor{1d91}
@deffn {Data} calendar.MONDAY
@anchor{library/calendar calendar TUESDAY}@anchor{1dac}
@deffnx {Data} calendar.TUESDAY
@anchor{library/calendar calendar WEDNESDAY}@anchor{1dad}
@deffnx {Data} calendar.WEDNESDAY
@anchor{library/calendar calendar THURSDAY}@anchor{1dae}
@deffnx {Data} calendar.THURSDAY
@anchor{library/calendar calendar FRIDAY}@anchor{1daf}
@deffnx {Data} calendar.FRIDAY
@anchor{library/calendar calendar SATURDAY}@anchor{1db0}
@deffnx {Data} calendar.SATURDAY
@anchor{library/calendar calendar SUNDAY}@anchor{1d92}
@deffnx {Data} calendar.SUNDAY

Aliases for day numbers, where @code{MONDAY} is @code{0} and @code{SUNDAY} is @code{6}.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{2f,,datetime}

Object-oriented interface to dates and times with similar functionality to the
@ref{10a,,time} module.

@item Module @ref{10a,,time}

Low-level time related functions.
@end table


@node collections — Container datatypes,collections abc — Abstract Base Classes for Containers,calendar — General calendar-related functions,Data Types
@anchor{library/collections doc}@anchor{1dc0}@anchor{library/collections collections-container-datatypes}@anchor{1dc1}@anchor{library/collections module-collections}@anchor{1c}
@subsection @code{collections} — Container datatypes


@geindex module; collections

@strong{Source code:} Lib/collections/__init__.py@footnote{https://github.com/python/cpython/tree/main/Lib/collections/__init__.py}

@c from collections import *
@c import itertools
@c __name__ = '<doctest>'


__________________________________________________________________


This module implements specialized container datatypes providing alternatives to
Python’s general purpose built-in containers, @ref{3b0,,dict}, @ref{43a,,list},
@ref{370,,set}, and @ref{3e5,,tuple}.


@multitable {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{3e4,,namedtuple()}

@tab

factory function for creating tuple subclasses with named fields

@item

@ref{7d3,,deque}

@tab

list-like container with fast appends and pops on either end

@item

@ref{758,,ChainMap}

@tab

dict-like class for creating a single view of multiple mappings

@item

@ref{c2a,,Counter}

@tab

dict subclass for counting hashable objects

@item

@ref{4e5,,OrderedDict}

@tab

dict subclass that remembers the order entries were added

@item

@ref{d63,,defaultdict}

@tab

dict subclass that calls a factory function to supply missing values

@item

@ref{1225,,UserDict}

@tab

wrapper around dictionary objects for easier dict subclassing

@item

@ref{1dc2,,UserList}

@tab

wrapper around list objects for easier list subclassing

@item

@ref{934,,UserString}

@tab

wrapper around string objects for easier string subclassing

@end multitable


@menu
* ChainMap objects:: 
* Counter objects:: 
* deque objects:: 
* defaultdict objects:: 
* namedtuple() Factory Function for Tuples with Named Fields: namedtuple Factory Function for Tuples with Named Fields. 
* OrderedDict objects:: 
* UserDict objects:: 
* UserList objects:: 
* UserString objects:: 

@end menu

@node ChainMap objects,Counter objects,,collections — Container datatypes
@anchor{library/collections chainmap-objects}@anchor{1dc3}
@subsubsection @code{ChainMap} objects



New in version 3.3.


A @ref{758,,ChainMap} class is provided for quickly linking a number of mappings
so they can be treated as a single unit.  It is often much faster than creating
a new dictionary and running multiple @ref{fc3,,update()} calls.

The class can be used to simulate nested scopes and is useful in templating.

@geindex ChainMap (class in collections)
@anchor{library/collections collections ChainMap}@anchor{758}
@deffn {Class} collections.ChainMap (*maps)

A @ref{758,,ChainMap} groups multiple dicts or other mappings together to
create a single, updateable view.  If no @emph{maps} are specified, a single empty
dictionary is provided so that a new chain always has at least one mapping.

The underlying mappings are stored in a list.  That list is public and can
be accessed or updated using the @emph{maps} attribute.  There is no other state.

Lookups search the underlying mappings successively until a key is found.  In
contrast, writes, updates, and deletions only operate on the first mapping.

A @ref{758,,ChainMap} incorporates the underlying mappings by reference.  So, if
one of the underlying mappings gets updated, those changes will be reflected
in @ref{758,,ChainMap}.

All of the usual dictionary methods are supported.  In addition, there is a
@emph{maps} attribute, a method for creating new subcontexts, and a property for
accessing all but the first mapping:

@geindex maps (collections.ChainMap attribute)
@anchor{library/collections collections ChainMap maps}@anchor{1dc4}
@deffn {Attribute} maps

A user updateable list of mappings.  The list is ordered from
first-searched to last-searched.  It is the only stored state and can
be modified to change which mappings are searched.  The list should
always contain at least one mapping.
@end deffn

@geindex new_child() (collections.ChainMap method)
@anchor{library/collections collections ChainMap new_child}@anchor{a9d}
@deffn {Method} new_child (m=None, **kwargs)

Returns a new @ref{758,,ChainMap} containing a new map followed by
all of the maps in the current instance.  If @code{m} is specified,
it becomes the new map at the front of the list of mappings; if not
specified, an empty dict is used, so that a call to @code{d.new_child()}
is equivalent to: @code{ChainMap(@{@}, *d.maps)}. If any keyword arguments
are specified, they update passed map or new empty dict. This method
is used for creating subcontexts that can be updated without altering
values in any of the parent mappings.


Changed in version 3.4: The optional @code{m} parameter was added.



Changed in version 3.10: Keyword arguments support was added.

@end deffn

@geindex parents (collections.ChainMap attribute)
@anchor{library/collections collections ChainMap parents}@anchor{1dc5}
@deffn {Attribute} parents

Property returning a new @ref{758,,ChainMap} containing all of the maps in
the current instance except the first one.  This is useful for skipping
the first map in the search.  Use cases are similar to those for the
@ref{e4d,,nonlocal} keyword used in @ref{1a43,,nested scopes}.  The use cases also parallel those for the built-in
@ref{78c,,super()} function.  A reference to @code{d.parents} is equivalent to:
@code{ChainMap(*d.maps[1:])}.
@end deffn

Note, the iteration order of a @ref{758,,ChainMap()} is determined by
scanning the mappings last to first:

@example
>>> baseline = @{'music': 'bach', 'art': 'rembrandt'@}
>>> adjustments = @{'art': 'van gogh', 'opera': 'carmen'@}
>>> list(ChainMap(adjustments, baseline))
['music', 'art', 'opera']
@end example

This gives the same ordering as a series of @ref{fc3,,dict.update()} calls
starting with the last mapping:

@example
>>> combined = baseline.copy()
>>> combined.update(adjustments)
>>> list(combined)
['music', 'art', 'opera']
@end example


Changed in version 3.9: Added support for @code{|} and @code{|=} operators, specified in 
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584}.

@end deffn


@subsubheading See also



@itemize *

@item 
The MultiContext class@footnote{https://github.com/enthought/codetools/blob/4.0.0/codetools/contexts/multi_context.py}
in the Enthought CodeTools package@footnote{https://github.com/enthought/codetools} has options to support
writing to any mapping in the chain.

@item 
Django’s Context class@footnote{https://github.com/django/django/blob/main/django/template/context.py}
for templating is a read-only chain of mappings.  It also features
pushing and popping of contexts similar to the
@ref{a9d,,new_child()} method and the
@ref{1dc5,,parents} property.

@item 
The Nested Contexts recipe@footnote{https://code.activestate.com/recipes/577434/} has options to control
whether writes and other mutations apply only to the first mapping or to
any mapping in the chain.

@item 
A greatly simplified read-only version of Chainmap@footnote{https://code.activestate.com/recipes/305268/}.
@end itemize


@menu
* ChainMap Examples and Recipes:: 

@end menu

@node ChainMap Examples and Recipes,,,ChainMap objects
@anchor{library/collections chainmap-examples-and-recipes}@anchor{1dc6}
@subsubsection @code{ChainMap} Examples and Recipes


This section shows various approaches to working with chained maps.

Example of simulating Python’s internal lookup chain:

@example
import builtins
pylookup = ChainMap(locals(), globals(), vars(builtins))
@end example

Example of letting user specified command-line arguments take precedence over
environment variables which in turn take precedence over default values:

@example
import os, argparse

defaults = @{'color': 'red', 'user': 'guest'@}

parser = argparse.ArgumentParser()
parser.add_argument('-u', '--user')
parser.add_argument('-c', '--color')
namespace = parser.parse_args()
command_line_args = @{k: v for k, v in vars(namespace).items() if v is not None@}

combined = ChainMap(command_line_args, os.environ, defaults)
print(combined['color'])
print(combined['user'])
@end example

Example patterns for using the @ref{758,,ChainMap} class to simulate nested
contexts:

@example
c = ChainMap()        # Create root context
d = c.new_child()     # Create nested child context
e = c.new_child()     # Child of c, independent from d
e.maps[0]             # Current context dictionary -- like Python's locals()
e.maps[-1]            # Root context -- like Python's globals()
e.parents             # Enclosing context chain -- like Python's nonlocals

d['x'] = 1            # Set value in current context
d['x']                # Get first key in the chain of contexts
del d['x']            # Delete from current context
list(d)               # All nested values
k in d                # Check all nested values
len(d)                # Number of nested values
d.items()             # All nested items
dict(d)               # Flatten into a regular dictionary
@end example

The @ref{758,,ChainMap} class only makes updates (writes and deletions) to the
first mapping in the chain while lookups will search the full chain.  However,
if deep writes and deletions are desired, it is easy to make a subclass that
updates keys found deeper in the chain:

@example
class DeepChainMap(ChainMap):
    'Variant of ChainMap that allows direct updates to inner scopes'

    def __setitem__(self, key, value):
        for mapping in self.maps:
            if key in mapping:
                mapping[key] = value
                return
        self.maps[0][key] = value

    def __delitem__(self, key):
        for mapping in self.maps:
            if key in mapping:
                del mapping[key]
                return
        raise KeyError(key)

>>> d = DeepChainMap(@{'zebra': 'black'@}, @{'elephant': 'blue'@}, @{'lion': 'yellow'@})
>>> d['lion'] = 'orange'         # update an existing key two levels down
>>> d['snake'] = 'red'           # new keys get added to the topmost dict
>>> del d['elephant']            # remove an existing key one level down
>>> d                            # display result
DeepChainMap(@{'zebra': 'black', 'snake': 'red'@}, @{@}, @{'lion': 'orange'@})
@end example

@node Counter objects,deque objects,ChainMap objects,collections — Container datatypes
@anchor{library/collections counter-objects}@anchor{1dc7}
@subsubsection @code{Counter} objects


A counter tool is provided to support convenient and rapid tallies.
For example:

@example
>>> # Tally occurrences of words in a list
>>> cnt = Counter()
>>> for word in ['red', 'blue', 'red', 'green', 'blue', 'blue']:
...     cnt[word] += 1
>>> cnt
Counter(@{'blue': 3, 'red': 2, 'green': 1@})

>>> # Find the ten most common words in Hamlet
>>> import re
>>> words = re.findall(r'\w+', open('hamlet.txt').read().lower())
>>> Counter(words).most_common(10)
[('the', 1143), ('and', 966), ('to', 762), ('of', 669), ('i', 631),
 ('you', 554),  ('a', 546), ('my', 514), ('hamlet', 471), ('in', 451)]
@end example

@geindex Counter (class in collections)
@anchor{library/collections collections Counter}@anchor{c2a}
@deffn {Class} collections.Counter ([iterable-or-mapping])

A @ref{c2a,,Counter} is a @ref{3b0,,dict} subclass for counting hashable objects.
It is a collection where elements are stored as dictionary keys
and their counts are stored as dictionary values.  Counts are allowed to be
any integer value including zero or negative counts.  The @ref{c2a,,Counter}
class is similar to bags or multisets in other languages.

Elements are counted from an @emph{iterable} or initialized from another
@emph{mapping} (or counter):

@example
>>> c = Counter()                           # a new, empty counter
>>> c = Counter('gallahad')                 # a new counter from an iterable
>>> c = Counter(@{'red': 4, 'blue': 2@})      # a new counter from a mapping
>>> c = Counter(cats=4, dogs=8)             # a new counter from keyword args
@end example

Counter objects have a dictionary interface except that they return a zero
count for missing items instead of raising a @ref{5bb,,KeyError}:

@example
>>> c = Counter(['eggs', 'ham'])
>>> c['bacon']                              # count of a missing element is zero
0
@end example

Setting a count to zero does not remove an element from a counter.
Use @code{del} to remove it entirely:

@example
>>> c['sausage'] = 0                        # counter entry with a zero count
>>> del c['sausage']                        # del actually removes the entry
@end example


New in version 3.1.



Changed in version 3.7: As a @ref{3b0,,dict} subclass, @ref{c2a,,Counter}
inherited the capability to remember insertion order.  Math operations
on @emph{Counter} objects also preserve order.  Results are ordered
according to when an element is first encountered in the left operand
and then by the order encountered in the right operand.


Counter objects support additional methods beyond those available for all
dictionaries:

@geindex elements() (collections.Counter method)
@anchor{library/collections collections Counter elements}@anchor{e9f}
@deffn {Method} elements ()

Return an iterator over elements repeating each as many times as its
count.  Elements are returned in the order first encountered. If an
element’s count is less than one, @ref{e9f,,elements()} will ignore it.

@example
>>> c = Counter(a=4, b=2, c=0, d=-2)
>>> sorted(c.elements())
['a', 'a', 'a', 'a', 'b', 'b']
@end example
@end deffn

@geindex most_common() (collections.Counter method)
@anchor{library/collections collections Counter most_common}@anchor{e9e}
@deffn {Method} most_common ([n])

Return a list of the @emph{n} most common elements and their counts from the
most common to the least.  If @emph{n} is omitted or @code{None},
@ref{e9e,,most_common()} returns @emph{all} elements in the counter.
Elements with equal counts are ordered in the order first encountered:

@example
>>> Counter('abracadabra').most_common(3)
[('a', 5), ('b', 2), ('r', 2)]
@end example
@end deffn

@geindex subtract() (collections.Counter method)
@anchor{library/collections collections Counter subtract}@anchor{d80}
@deffn {Method} subtract ([iterable-or-mapping])

Elements are subtracted from an @emph{iterable} or from another @emph{mapping}
(or counter).  Like @ref{fc3,,dict.update()} but subtracts counts instead
of replacing them.  Both inputs and outputs may be zero or negative.

@example
>>> c = Counter(a=4, b=2, c=0, d=-2)
>>> d = Counter(a=1, b=2, c=3, d=4)
>>> c.subtract(d)
>>> c
Counter(@{'a': 3, 'b': 0, 'c': -3, 'd': -6@})
@end example


New in version 3.2.

@end deffn

@geindex total() (collections.Counter method)
@anchor{library/collections collections Counter total}@anchor{1dc8}
@deffn {Method} total ()

Compute the sum of the counts.

@example
>>> c = Counter(a=10, b=5, c=0)
>>> c.total()
15
@end example


New in version 3.10.

@end deffn

The usual dictionary methods are available for @ref{c2a,,Counter} objects
except for two which work differently for counters.

@geindex fromkeys() (collections.Counter method)
@anchor{library/collections collections Counter fromkeys}@anchor{1dc9}
@deffn {Method} fromkeys (iterable)

This class method is not implemented for @ref{c2a,,Counter} objects.
@end deffn

@geindex update() (collections.Counter method)
@anchor{library/collections collections Counter update}@anchor{1dca}
@deffn {Method} update ([iterable-or-mapping])

Elements are counted from an @emph{iterable} or added-in from another
@emph{mapping} (or counter).  Like @ref{fc3,,dict.update()} but adds counts
instead of replacing them.  Also, the @emph{iterable} is expected to be a
sequence of elements, not a sequence of @code{(key, value)} pairs.
@end deffn
@end deffn

Counters support rich comparison operators for equality, subset, and
superset relationships: @code{==}, @code{!=}, @code{<}, @code{<=}, @code{>}, @code{>=}.
All of those tests treat missing elements as having zero counts so that
@code{Counter(a=1) == Counter(a=1, b=0)} returns true.


New in version 3.10: Rich comparison operations were added.



Changed in version 3.10: In equality tests, missing elements are treated as having zero counts.
Formerly, @code{Counter(a=3)} and @code{Counter(a=3, b=0)} were considered
distinct.


Common patterns for working with @ref{c2a,,Counter} objects:

@example
c.total()                       # total of all counts
c.clear()                       # reset all counts
list(c)                         # list unique elements
set(c)                          # convert to a set
dict(c)                         # convert to a regular dictionary
c.items()                       # convert to a list of (elem, cnt) pairs
Counter(dict(list_of_pairs))    # convert from a list of (elem, cnt) pairs
c.most_common()[:-n-1:-1]       # n least common elements
+c                              # remove zero and negative counts
@end example

Several mathematical operations are provided for combining @ref{c2a,,Counter}
objects to produce multisets (counters that have counts greater than zero).
Addition and subtraction combine counters by adding or subtracting the counts
of corresponding elements.  Intersection and union return the minimum and
maximum of corresponding counts.  Equality and inclusion compare
corresponding counts.  Each operation can accept inputs with signed
counts, but the output will exclude results with counts of zero or less.

@example
>>> c = Counter(a=3, b=1)
>>> d = Counter(a=1, b=2)
>>> c + d                       # add two counters together:  c[x] + d[x]
Counter(@{'a': 4, 'b': 3@})
>>> c - d                       # subtract (keeping only positive counts)
Counter(@{'a': 2@})
>>> c & d                       # intersection:  min(c[x], d[x])
Counter(@{'a': 1, 'b': 1@})
>>> c | d                       # union:  max(c[x], d[x])
Counter(@{'a': 3, 'b': 2@})
>>> c == d                      # equality:  c[x] == d[x]
False
>>> c <= d                      # inclusion:  c[x] <= d[x]
False
@end example

Unary addition and subtraction are shortcuts for adding an empty counter
or subtracting from an empty counter.

@example
>>> c = Counter(a=2, b=-4)
>>> +c
Counter(@{'a': 2@})
>>> -c
Counter(@{'b': 4@})
@end example


New in version 3.3: Added support for unary plus, unary minus, and in-place multiset operations.


@cartouche
@quotation Note 
Counters were primarily designed to work with positive integers to represent
running counts; however, care was taken to not unnecessarily preclude use
cases needing other types or negative values.  To help with those use cases,
this section documents the minimum range and type restrictions.


@itemize *

@item 
The @ref{c2a,,Counter} class itself is a dictionary subclass with no
restrictions on its keys and values.  The values are intended to be numbers
representing counts, but you @emph{could} store anything in the value field.

@item 
The @ref{e9e,,most_common()} method requires only that the values be orderable.

@item 
For in-place operations such as @code{c[key] += 1}, the value type need only
support addition and subtraction.  So fractions, floats, and decimals would
work and negative values are supported.  The same is also true for
@ref{1dca,,update()} and @ref{d80,,subtract()} which allow negative and zero values
for both inputs and outputs.

@item 
The multiset methods are designed only for use cases with positive values.
The inputs may be negative or zero, but only outputs with positive values
are created.  There are no type restrictions, but the value type needs to
support addition, subtraction, and comparison.

@item 
The @ref{e9f,,elements()} method requires integer counts.  It ignores zero and
negative counts.
@end itemize
@end quotation
@end cartouche


@subsubheading See also



@itemize *

@item 
Bag class@footnote{https://www.gnu.org/software/smalltalk/manual-base/html_node/Bag.html}
in Smalltalk.

@item 
Wikipedia entry for Multisets@footnote{https://en.wikipedia.org/wiki/Multiset}.

@item 
C++ multisets@footnote{http://www.java2s.com/Tutorial/Cpp/0380__set-multiset/Catalog0380__set-multiset.htm}
tutorial with examples.

@item 
For mathematical operations on multisets and their use cases, see
@emph{Knuth, Donald. The Art of Computer Programming Volume II,
Section 4.6.3, Exercise 19}.

@item 
To enumerate all distinct multisets of a given size over a given set of
elements, see @ref{e31,,itertools.combinations_with_replacement()}:

@example
map(Counter, combinations_with_replacement('ABC', 2)) # --> AA AB AC BB BC CC
@end example
@end itemize


@node deque objects,defaultdict objects,Counter objects,collections — Container datatypes
@anchor{library/collections deque-objects}@anchor{1dcb}
@subsubsection @code{deque} objects


@geindex deque (class in collections)
@anchor{library/collections collections deque}@anchor{7d3}
@deffn {Class} collections.deque ([iterable[, maxlen]])

Returns a new deque object initialized left-to-right (using @ref{1dcc,,append()}) with
data from @emph{iterable}.  If @emph{iterable} is not specified, the new deque is empty.

Deques are a generalization of stacks and queues (the name is pronounced “deck”
and is short for “double-ended queue”).  Deques support thread-safe, memory
efficient appends and pops from either side of the deque with approximately the
same O(1) performance in either direction.

Though @ref{43a,,list} objects support similar operations, they are optimized for
fast fixed-length operations and incur O(n) memory movement costs for
@code{pop(0)} and @code{insert(0, v)} operations which change both the size and
position of the underlying data representation.

If @emph{maxlen} is not specified or is @code{None}, deques may grow to an
arbitrary length.  Otherwise, the deque is bounded to the specified maximum
length.  Once a bounded length deque is full, when new items are added, a
corresponding number of items are discarded from the opposite end.  Bounded
length deques provide functionality similar to the @code{tail} filter in
Unix. They are also useful for tracking transactions and other pools of data
where only the most recent activity is of interest.

Deque objects support the following methods:

@geindex append() (collections.deque method)
@anchor{library/collections collections deque append}@anchor{1dcc}
@deffn {Method} append (x)

Add @emph{x} to the right side of the deque.
@end deffn

@geindex appendleft() (collections.deque method)
@anchor{library/collections collections deque appendleft}@anchor{1dcd}
@deffn {Method} appendleft (x)

Add @emph{x} to the left side of the deque.
@end deffn

@geindex clear() (collections.deque method)
@anchor{library/collections collections deque clear}@anchor{1dce}
@deffn {Method} clear ()

Remove all elements from the deque leaving it with length 0.
@end deffn

@geindex copy() (collections.deque method)
@anchor{library/collections collections deque copy}@anchor{932}
@deffn {Method} copy ()

Create a shallow copy of the deque.


New in version 3.5.

@end deffn

@geindex count() (collections.deque method)
@anchor{library/collections collections deque count}@anchor{d82}
@deffn {Method} count (x)

Count the number of deque elements equal to @emph{x}.


New in version 3.2.

@end deffn

@geindex extend() (collections.deque method)
@anchor{library/collections collections deque extend}@anchor{1dcf}
@deffn {Method} extend (iterable)

Extend the right side of the deque by appending elements from the iterable
argument.
@end deffn

@geindex extendleft() (collections.deque method)
@anchor{library/collections collections deque extendleft}@anchor{1dd0}
@deffn {Method} extendleft (iterable)

Extend the left side of the deque by appending elements from @emph{iterable}.
Note, the series of left appends results in reversing the order of
elements in the iterable argument.
@end deffn

@geindex index() (collections.deque method)
@anchor{library/collections collections deque index}@anchor{930}
@deffn {Method} index (x[, start[, stop]])

Return the position of @emph{x} in the deque (at or after index @emph{start}
and before index @emph{stop}).  Returns the first match or raises
@ref{17c,,ValueError} if not found.


New in version 3.5.

@end deffn

@geindex insert() (collections.deque method)
@anchor{library/collections collections deque insert}@anchor{931}
@deffn {Method} insert (i, x)

Insert @emph{x} into the deque at position @emph{i}.

If the insertion would cause a bounded deque to grow beyond @emph{maxlen},
an @ref{1066,,IndexError} is raised.


New in version 3.5.

@end deffn

@geindex pop() (collections.deque method)
@anchor{library/collections collections deque pop}@anchor{1dd1}
@deffn {Method} pop ()

Remove and return an element from the right side of the deque. If no
elements are present, raises an @ref{1066,,IndexError}.
@end deffn

@geindex popleft() (collections.deque method)
@anchor{library/collections collections deque popleft}@anchor{1dd2}
@deffn {Method} popleft ()

Remove and return an element from the left side of the deque. If no
elements are present, raises an @ref{1066,,IndexError}.
@end deffn

@geindex remove() (collections.deque method)
@anchor{library/collections collections deque remove}@anchor{1184}
@deffn {Method} remove (value)

Remove the first occurrence of @emph{value}.  If not found, raises a
@ref{17c,,ValueError}.
@end deffn

@geindex reverse() (collections.deque method)
@anchor{library/collections collections deque reverse}@anchor{d83}
@deffn {Method} reverse ()

Reverse the elements of the deque in-place and then return @code{None}.


New in version 3.2.

@end deffn

@geindex rotate() (collections.deque method)
@anchor{library/collections collections deque rotate}@anchor{114e}
@deffn {Method} rotate (n=1)

Rotate the deque @emph{n} steps to the right.  If @emph{n} is negative, rotate
to the left.

When the deque is not empty, rotating one step to the right is equivalent
to @code{d.appendleft(d.pop())}, and rotating one step to the left is
equivalent to @code{d.append(d.popleft())}.
@end deffn

Deque objects also provide one read-only attribute:

@geindex maxlen (collections.deque attribute)
@anchor{library/collections collections deque maxlen}@anchor{ea0}
@deffn {Attribute} maxlen

Maximum size of a deque or @code{None} if unbounded.


New in version 3.1.

@end deffn
@end deffn

In addition to the above, deques support iteration, pickling, @code{len(d)},
@code{reversed(d)}, @code{copy.copy(d)}, @code{copy.deepcopy(d)}, membership testing with
the @ref{2fc,,in} operator, and subscript references such as @code{d[0]} to access
the first element.  Indexed access is O(1) at both ends but slows to O(n) in
the middle.  For fast random access, use lists instead.

Starting in version 3.5, deques support @code{__add__()}, @code{__mul__()},
and @code{__imul__()}.

Example:

@example
>>> from collections import deque
>>> d = deque('ghi')                 # make a new deque with three items
>>> for elem in d:                   # iterate over the deque's elements
...     print(elem.upper())
G
H
I

>>> d.append('j')                    # add a new entry to the right side
>>> d.appendleft('f')                # add a new entry to the left side
>>> d                                # show the representation of the deque
deque(['f', 'g', 'h', 'i', 'j'])

>>> d.pop()                          # return and remove the rightmost item
'j'
>>> d.popleft()                      # return and remove the leftmost item
'f'
>>> list(d)                          # list the contents of the deque
['g', 'h', 'i']
>>> d[0]                             # peek at leftmost item
'g'
>>> d[-1]                            # peek at rightmost item
'i'

>>> list(reversed(d))                # list the contents of a deque in reverse
['i', 'h', 'g']
>>> 'h' in d                         # search the deque
True
>>> d.extend('jkl')                  # add multiple elements at once
>>> d
deque(['g', 'h', 'i', 'j', 'k', 'l'])
>>> d.rotate(1)                      # right rotation
>>> d
deque(['l', 'g', 'h', 'i', 'j', 'k'])
>>> d.rotate(-1)                     # left rotation
>>> d
deque(['g', 'h', 'i', 'j', 'k', 'l'])

>>> deque(reversed(d))               # make a new deque in reverse order
deque(['l', 'k', 'j', 'i', 'h', 'g'])
>>> d.clear()                        # empty the deque
>>> d.pop()                          # cannot pop from an empty deque
Traceback (most recent call last):
    File "<pyshell#6>", line 1, in -toplevel-
        d.pop()
IndexError: pop from an empty deque

>>> d.extendleft('abc')              # extendleft() reverses the input order
>>> d
deque(['c', 'b', 'a'])
@end example

@menu
* deque Recipes:: 

@end menu

@node deque Recipes,,,deque objects
@anchor{library/collections deque-recipes}@anchor{1dd3}
@subsubsection @code{deque} Recipes


This section shows various approaches to working with deques.

Bounded length deques provide functionality similar to the @code{tail} filter
in Unix:

@example
def tail(filename, n=10):
    'Return the last n lines of a file'
    with open(filename) as f:
        return deque(f, n)
@end example

Another approach to using deques is to maintain a sequence of recently
added elements by appending to the right and popping to the left:

@example
def moving_average(iterable, n=3):
    # moving_average([40, 30, 50, 46, 39, 44]) --> 40.0 42.0 45.0 43.0
    # http://en.wikipedia.org/wiki/Moving_average
    it = iter(iterable)
    d = deque(itertools.islice(it, n-1))
    d.appendleft(0)
    s = sum(d)
    for elem in it:
        s += elem - d.popleft()
        d.append(elem)
        yield s / n
@end example

A round-robin scheduler@footnote{https://en.wikipedia.org/wiki/Round-robin_scheduling} can be implemented with
input iterators stored in a @ref{7d3,,deque}.  Values are yielded from the active
iterator in position zero.  If that iterator is exhausted, it can be removed
with @ref{1dd2,,popleft()}; otherwise, it can be cycled back to the end with
the @ref{114e,,rotate()} method:

@example
def roundrobin(*iterables):
    "roundrobin('ABC', 'D', 'EF') --> A D E B F C"
    iterators = deque(map(iter, iterables))
    while iterators:
        try:
            while True:
                yield next(iterators[0])
                iterators.rotate(-1)
        except StopIteration:
            # Remove an exhausted iterator.
            iterators.popleft()
@end example

The @ref{114e,,rotate()} method provides a way to implement @ref{7d3,,deque} slicing and
deletion.  For example, a pure Python implementation of @code{del d[n]} relies on
the @code{rotate()} method to position elements to be popped:

@example
def delete_nth(d, n):
    d.rotate(-n)
    d.popleft()
    d.rotate(n)
@end example

To implement @ref{7d3,,deque} slicing, use a similar approach applying
@ref{114e,,rotate()} to bring a target element to the left side of the deque. Remove
old entries with @ref{1dd2,,popleft()}, add new entries with @ref{1dcf,,extend()}, and then
reverse the rotation.
With minor variations on that approach, it is easy to implement Forth style
stack manipulations such as @code{dup}, @code{drop}, @code{swap}, @code{over}, @code{pick},
@code{rot}, and @code{roll}.

@node defaultdict objects,namedtuple Factory Function for Tuples with Named Fields,deque objects,collections — Container datatypes
@anchor{library/collections defaultdict-objects}@anchor{1dd4}
@subsubsection @code{defaultdict} objects


@geindex defaultdict (class in collections)
@anchor{library/collections collections defaultdict}@anchor{d63}
@deffn {Class} collections.defaultdict (default_factory=None, /[, ...])

Return a new dictionary-like object.  @ref{d63,,defaultdict} is a subclass of the
built-in @ref{3b0,,dict} class.  It overrides one method and adds one writable
instance variable.  The remaining functionality is the same as for the
@ref{3b0,,dict} class and is not documented here.

The first argument provides the initial value for the @ref{1dd5,,default_factory}
attribute; it defaults to @code{None}. All remaining arguments are treated the same
as if they were passed to the @ref{3b0,,dict} constructor, including keyword
arguments.

@ref{d63,,defaultdict} objects support the following method in addition to the
standard @ref{3b0,,dict} operations:

@geindex __missing__() (collections.defaultdict method)
@anchor{library/collections collections defaultdict __missing__}@anchor{1dd6}
@deffn {Method} __missing__ (key)

If the @ref{1dd5,,default_factory} attribute is @code{None}, this raises a
@ref{5bb,,KeyError} exception with the @emph{key} as argument.

If @ref{1dd5,,default_factory} is not @code{None}, it is called without arguments
to provide a default value for the given @emph{key}, this value is inserted in
the dictionary for the @emph{key}, and returned.

If calling @ref{1dd5,,default_factory} raises an exception this exception is
propagated unchanged.

This method is called by the @code{__getitem__()} method of the
@ref{3b0,,dict} class when the requested key is not found; whatever it
returns or raises is then returned or raised by @code{__getitem__()}.

Note that @ref{1dd6,,__missing__()} is @emph{not} called for any operations besides
@code{__getitem__()}. This means that @code{get()} will, like normal
dictionaries, return @code{None} as a default rather than using
@ref{1dd5,,default_factory}.
@end deffn

@ref{d63,,defaultdict} objects support the following instance variable:

@geindex default_factory (collections.defaultdict attribute)
@anchor{library/collections collections defaultdict default_factory}@anchor{1dd5}
@deffn {Attribute} default_factory

This attribute is used by the @ref{1dd6,,__missing__()} method; it is
initialized from the first argument to the constructor, if present, or to
@code{None}, if absent.
@end deffn


Changed in version 3.9: Added merge (@code{|}) and update (@code{|=}) operators, specified in
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584}.

@end deffn

@menu
* defaultdict Examples:: 

@end menu

@node defaultdict Examples,,,defaultdict objects
@anchor{library/collections defaultdict-examples}@anchor{1dd7}
@subsubsection @code{defaultdict} Examples


Using @ref{43a,,list} as the @ref{1dd5,,default_factory}, it is easy to group a
sequence of key-value pairs into a dictionary of lists:

@example
>>> s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
>>> d = defaultdict(list)
>>> for k, v in s:
...     d[k].append(v)
...
>>> sorted(d.items())
[('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]
@end example

When each key is encountered for the first time, it is not already in the
mapping; so an entry is automatically created using the @ref{1dd5,,default_factory}
function which returns an empty @ref{43a,,list}.  The @code{list.append()}
operation then attaches the value to the new list.  When keys are encountered
again, the look-up proceeds normally (returning the list for that key) and the
@code{list.append()} operation adds another value to the list. This technique is
simpler and faster than an equivalent technique using @ref{c0f,,dict.setdefault()}:

@example
>>> d = @{@}
>>> for k, v in s:
...     d.setdefault(k, []).append(v)
...
>>> sorted(d.items())
[('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]
@end example

Setting the @ref{1dd5,,default_factory} to @ref{1b1,,int} makes the
@ref{d63,,defaultdict} useful for counting (like a bag or multiset in other
languages):

@example
>>> s = 'mississippi'
>>> d = defaultdict(int)
>>> for k in s:
...     d[k] += 1
...
>>> sorted(d.items())
[('i', 4), ('m', 1), ('p', 2), ('s', 4)]
@end example

When a letter is first encountered, it is missing from the mapping, so the
@ref{1dd5,,default_factory} function calls @ref{1b1,,int()} to supply a default count of
zero.  The increment operation then builds up the count for each letter.

The function @ref{1b1,,int()} which always returns zero is just a special case of
constant functions.  A faster and more flexible way to create constant functions
is to use a lambda function which can supply any constant value (not just
zero):

@example
>>> def constant_factory(value):
...     return lambda: value
>>> d = defaultdict(constant_factory('<missing>'))
>>> d.update(name='John', action='ran')
>>> '%(name)s %(action)s to %(object)s' % d
'John ran to <missing>'
@end example

Setting the @ref{1dd5,,default_factory} to @ref{370,,set} makes the
@ref{d63,,defaultdict} useful for building a dictionary of sets:

@example
>>> s = [('red', 1), ('blue', 2), ('red', 3), ('blue', 4), ('red', 1), ('blue', 4)]
>>> d = defaultdict(set)
>>> for k, v in s:
...     d[k].add(v)
...
>>> sorted(d.items())
[('blue', @{2, 4@}), ('red', @{1, 3@})]
@end example

@node namedtuple Factory Function for Tuples with Named Fields,OrderedDict objects,defaultdict objects,collections — Container datatypes
@anchor{library/collections namedtuple-factory-function-for-tuples-with-named-fields}@anchor{1dd8}
@subsubsection @code{namedtuple()} Factory Function for Tuples with Named Fields


Named tuples assign meaning to each position in a tuple and allow for more readable,
self-documenting code.  They can be used wherever regular tuples are used, and
they add the ability to access fields by name instead of position index.

@geindex namedtuple() (in module collections)
@anchor{library/collections collections namedtuple}@anchor{3e4}
@deffn {Function} collections.namedtuple (typename, field_names, *, rename=False, defaults=None, module=None)

Returns a new tuple subclass named @emph{typename}.  The new subclass is used to
create tuple-like objects that have fields accessible by attribute lookup as
well as being indexable and iterable.  Instances of the subclass also have a
helpful docstring (with typename and field_names) and a helpful @code{__repr__()}
method which lists the tuple contents in a @code{name=value} format.

The @emph{field_names} are a sequence of strings such as @code{['x', 'y']}.
Alternatively, @emph{field_names} can be a single string with each fieldname
separated by whitespace and/or commas, for example @code{'x y'} or @code{'x, y'}.

Any valid Python identifier may be used for a fieldname except for names
starting with an underscore.  Valid identifiers consist of letters, digits,
and underscores but do not start with a digit or underscore and cannot be
a @ref{a2,,keyword} such as @emph{class}, @emph{for}, @emph{return}, @emph{global}, @emph{pass},
or @emph{raise}.

If @emph{rename} is true, invalid fieldnames are automatically replaced
with positional names.  For example, @code{['abc', 'def', 'ghi', 'abc']} is
converted to @code{['abc', '_1', 'ghi', '_3']}, eliminating the keyword
@code{def} and the duplicate fieldname @code{abc}.

@emph{defaults} can be @code{None} or an @ref{dcc,,iterable} of default values.
Since fields with a default value must come after any fields without a
default, the @emph{defaults} are applied to the rightmost parameters.  For
example, if the fieldnames are @code{['x', 'y', 'z']} and the defaults are
@code{(1, 2)}, then @code{x} will be a required argument, @code{y} will default to
@code{1}, and @code{z} will default to @code{2}.

If @emph{module} is defined, the @code{__module__} attribute of the named tuple is
set to that value.

Named tuple instances do not have per-instance dictionaries, so they are
lightweight and require no more memory than regular tuples.

To support pickling, the named tuple class should be assigned to a variable
that matches @emph{typename}.


Changed in version 3.1: Added support for @emph{rename}.



Changed in version 3.6: The @emph{verbose} and @emph{rename} parameters became
@ref{5a2,,keyword-only arguments}.



Changed in version 3.6: Added the @emph{module} parameter.



Changed in version 3.7: Removed the @emph{verbose} parameter and the @code{_source} attribute.



Changed in version 3.7: Added the @emph{defaults} parameter and the @code{_field_defaults}
attribute.

@end deffn

@example
>>> # Basic example
>>> Point = namedtuple('Point', ['x', 'y'])
>>> p = Point(11, y=22)     # instantiate with positional or keyword arguments
>>> p[0] + p[1]             # indexable like the plain tuple (11, 22)
33
>>> x, y = p                # unpack like a regular tuple
>>> x, y
(11, 22)
>>> p.x + p.y               # fields also accessible by name
33
>>> p                       # readable __repr__ with a name=value style
Point(x=11, y=22)
@end example

Named tuples are especially useful for assigning field names to result tuples returned
by the @ref{28,,csv} or @ref{ed,,sqlite3} modules:

@example
EmployeeRecord = namedtuple('EmployeeRecord', 'name, age, title, department, paygrade')

import csv
for emp in map(EmployeeRecord._make, csv.reader(open("employees.csv", "rb"))):
    print(emp.name, emp.title)

import sqlite3
conn = sqlite3.connect('/companydata')
cursor = conn.cursor()
cursor.execute('SELECT name, age, title, department, paygrade FROM employees')
for emp in map(EmployeeRecord._make, cursor.fetchall()):
    print(emp.name, emp.title)
@end example

In addition to the methods inherited from tuples, named tuples support
three additional methods and two attributes.  To prevent conflicts with
field names, the method and attribute names start with an underscore.

@geindex _make() (collections.somenamedtuple class method)
@anchor{library/collections collections somenamedtuple _make}@anchor{1dd9}
@deffn {Method} classmethod somenamedtuple._make (iterable)

Class method that makes a new instance from an existing sequence or iterable.

@example
>>> t = [11, 22]
>>> Point._make(t)
Point(x=11, y=22)
@end example
@end deffn

@geindex _asdict() (collections.somenamedtuple method)
@anchor{library/collections collections somenamedtuple _asdict}@anchor{4e4}
@deffn {Method} somenamedtuple._asdict ()

Return a new @ref{3b0,,dict} which maps field names to their corresponding
values:

@example
>>> p = Point(x=11, y=22)
>>> p._asdict()
@{'x': 11, 'y': 22@}
@end example


Changed in version 3.1: Returns an @ref{4e5,,OrderedDict} instead of a regular @ref{3b0,,dict}.



Changed in version 3.8: Returns a regular @ref{3b0,,dict} instead of an @ref{4e5,,OrderedDict}.
As of Python 3.7, regular dicts are guaranteed to be ordered.  If the
extra features of @ref{4e5,,OrderedDict} are required, the suggested
remediation is to cast the result to the desired type:
@code{OrderedDict(nt._asdict())}.

@end deffn

@geindex _replace() (collections.somenamedtuple method)
@anchor{library/collections collections somenamedtuple _replace}@anchor{1dda}
@deffn {Method} somenamedtuple._replace (**kwargs)

Return a new instance of the named tuple replacing specified fields with new
values:

@example
>>> p = Point(x=11, y=22)
>>> p._replace(x=33)
Point(x=33, y=22)

>>> for partnum, record in inventory.items():
...     inventory[partnum] = record._replace(price=newprices[partnum], timestamp=time.now())
@end example
@end deffn

@geindex _fields (collections.somenamedtuple attribute)
@anchor{library/collections collections somenamedtuple _fields}@anchor{1ddb}
@deffn {Attribute} somenamedtuple._fields

Tuple of strings listing the field names.  Useful for introspection
and for creating new named tuple types from existing named tuples.

@example
>>> p._fields            # view the field names
('x', 'y')

>>> Color = namedtuple('Color', 'red green blue')
>>> Pixel = namedtuple('Pixel', Point._fields + Color._fields)
>>> Pixel(11, 22, 128, 255, 0)
Pixel(x=11, y=22, red=128, green=255, blue=0)
@end example
@end deffn

@geindex _field_defaults (collections.somenamedtuple attribute)
@anchor{library/collections collections somenamedtuple _field_defaults}@anchor{1ddc}
@deffn {Attribute} somenamedtuple._field_defaults

Dictionary mapping field names to default values.

@example
>>> Account = namedtuple('Account', ['type', 'balance'], defaults=[0])
>>> Account._field_defaults
@{'balance': 0@}
>>> Account('premium')
Account(type='premium', balance=0)
@end example
@end deffn

To retrieve a field whose name is stored in a string, use the @ref{708,,getattr()}
function:

@example
>>> getattr(p, 'x')
11
@end example

To convert a dictionary to a named tuple, use the double-star-operator
(as described in @ref{1515,,Unpacking Argument Lists}):

@example
>>> d = @{'x': 11, 'y': 22@}
>>> Point(**d)
Point(x=11, y=22)
@end example

Since a named tuple is a regular Python class, it is easy to add or change
functionality with a subclass.  Here is how to add a calculated field and
a fixed-width print format:

@example
>>> class Point(namedtuple('Point', ['x', 'y'])):
...     __slots__ = ()
...     @@property
...     def hypot(self):
...         return (self.x ** 2 + self.y ** 2) ** 0.5
...     def __str__(self):
...         return 'Point: x=%6.3f  y=%6.3f  hypot=%6.3f' % (self.x, self.y, self.hypot)

>>> for p in Point(3, 4), Point(14, 5/7):
...     print(p)
Point: x= 3.000  y= 4.000  hypot= 5.000
Point: x=14.000  y= 0.714  hypot=14.018
@end example

The subclass shown above sets @code{__slots__} to an empty tuple.  This helps
keep memory requirements low by preventing the creation of instance dictionaries.

Subclassing is not useful for adding new, stored fields.  Instead, simply
create a new named tuple type from the @ref{1ddb,,_fields} attribute:

@example
>>> Point3D = namedtuple('Point3D', Point._fields + ('z',))
@end example

Docstrings can be customized by making direct assignments to the @code{__doc__}
fields:

@example
>>> Book = namedtuple('Book', ['id', 'title', 'authors'])
>>> Book.__doc__ += ': Hardcover book in active collection'
>>> Book.id.__doc__ = '13-digit ISBN'
>>> Book.title.__doc__ = 'Title of first printing'
>>> Book.authors.__doc__ = 'List of authors sorted by last name'
@end example


Changed in version 3.5: Property docstrings became writeable.



@subsubheading See also



@itemize *

@item 
See @ref{457,,typing.NamedTuple} for a way to add type hints for named
tuples.  It also provides an elegant notation using the @ref{e72,,class}
keyword:

@example
class Component(NamedTuple):
    part_number: int
    weight: float
    description: Optional[str] = None
@end example

@item 
See @ref{41b,,types.SimpleNamespace()} for a mutable namespace based on an
underlying dictionary instead of a tuple.

@item 
The @ref{2e,,dataclasses} module provides a decorator and functions for
automatically adding generated special methods to user-defined classes.
@end itemize


@node OrderedDict objects,UserDict objects,namedtuple Factory Function for Tuples with Named Fields,collections — Container datatypes
@anchor{library/collections ordereddict-objects}@anchor{1ddd}
@subsubsection @code{OrderedDict} objects


Ordered dictionaries are just like regular dictionaries but have some extra
capabilities relating to ordering operations.  They have become less
important now that the built-in @ref{3b0,,dict} class gained the ability
to remember insertion order (this new behavior became guaranteed in
Python 3.7).

Some differences from @ref{3b0,,dict} still remain:


@itemize *

@item 
The regular @ref{3b0,,dict} was designed to be very good at mapping
operations.  Tracking insertion order was secondary.

@item 
The @ref{4e5,,OrderedDict} was designed to be good at reordering operations.
Space efficiency, iteration speed, and the performance of update
operations were secondary.

@item 
The @ref{4e5,,OrderedDict} algorithm can handle frequent reordering operations
better than @ref{3b0,,dict}.  As shown in the recipes below, this makes it
suitable for implementing various kinds of LRU caches.

@item 
The equality operation for @ref{4e5,,OrderedDict} checks for matching order.

A regular @ref{3b0,,dict} can emulate the order sensitive equality test with
@code{p == q and all(k1 == k2 for k1, k2 in zip(p, q))}.

@item 
The @code{popitem()} method of @ref{4e5,,OrderedDict} has a different
signature.  It accepts an optional argument to specify which item is popped.

A regular @ref{3b0,,dict} can emulate OrderedDict’s @code{od.popitem(last=True)}
with @code{d.popitem()} which is guaranteed to pop the rightmost (last) item.

A regular @ref{3b0,,dict} can emulate OrderedDict’s @code{od.popitem(last=False)}
with @code{(k := next(iter(d)), d.pop(k))} which will return and remove the
leftmost (first) item if it exists.

@item 
@ref{4e5,,OrderedDict} has a @code{move_to_end()} method to efficiently
reposition an element to an endpoint.

A regular @ref{3b0,,dict} can emulate OrderedDict’s @code{od.move_to_end(k,
last=True)} with @code{d[k] = d.pop(k)} which will move the key and its
associated value to the rightmost (last) position.

A regular @ref{3b0,,dict} does not have an efficient equivalent for
OrderedDict’s @code{od.move_to_end(k, last=False)} which moves the key
and its associated value to the leftmost (first) position.

@item 
Until Python 3.8, @ref{3b0,,dict} lacked a @code{__reversed__()} method.
@end itemize

@geindex OrderedDict (class in collections)
@anchor{library/collections collections OrderedDict}@anchor{4e5}
@deffn {Class} collections.OrderedDict ([items])

Return an instance of a @ref{3b0,,dict} subclass that has methods
specialized for rearranging dictionary order.


New in version 3.1.


@geindex popitem() (collections.OrderedDict method)
@anchor{library/collections collections OrderedDict popitem}@anchor{e8b}
@deffn {Method} popitem (last=True)

The @ref{e8b,,popitem()} method for ordered dictionaries returns and removes a
(key, value) pair.  The pairs are returned in
@abbr{LIFO} order if @emph{last} is true
or @abbr{FIFO,first-in@comma{} first-out} order if false.
@end deffn

@geindex move_to_end() (collections.OrderedDict method)
@anchor{library/collections collections OrderedDict move_to_end}@anchor{d81}
@deffn {Method} move_to_end (key, last=True)

Move an existing @emph{key} to either end of an ordered dictionary.  The item
is moved to the right end if @emph{last} is true (the default) or to the
beginning if @emph{last} is false.  Raises @ref{5bb,,KeyError} if the @emph{key} does
not exist:

@example
>>> d = OrderedDict.fromkeys('abcde')
>>> d.move_to_end('b')
>>> '@w{'}.join(d)
'acdeb'
>>> d.move_to_end('b', last=False)
>>> '@w{'}.join(d)
'bacde'
@end example


New in version 3.2.

@end deffn
@end deffn

In addition to the usual mapping methods, ordered dictionaries also support
reverse iteration using @ref{2f6,,reversed()}.

Equality tests between @ref{4e5,,OrderedDict} objects are order-sensitive
and are implemented as @code{list(od1.items())==list(od2.items())}.
Equality tests between @ref{4e5,,OrderedDict} objects and other
@ref{3ab,,Mapping} objects are order-insensitive like regular
dictionaries.  This allows @ref{4e5,,OrderedDict} objects to be substituted
anywhere a regular dictionary is used.


Changed in version 3.5: The items, keys, and values @ref{1dde,,views}
of @ref{4e5,,OrderedDict} now support reverse iteration using @ref{2f6,,reversed()}.



Changed in version 3.6: With the acceptance of 
@geindex Python Enhancement Proposals; PEP 468
PEP 468@footnote{https://www.python.org/dev/peps/pep-0468}, order is retained for keyword arguments
passed to the @ref{4e5,,OrderedDict} constructor and its @code{update()}
method.



Changed in version 3.9: Added merge (@code{|}) and update (@code{|=}) operators, specified in 
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584}.


@menu
* OrderedDict Examples and Recipes:: 

@end menu

@node OrderedDict Examples and Recipes,,,OrderedDict objects
@anchor{library/collections ordereddict-examples-and-recipes}@anchor{1ddf}
@subsubsection @code{OrderedDict} Examples and Recipes


It is straightforward to create an ordered dictionary variant
that remembers the order the keys were @emph{last} inserted.
If a new entry overwrites an existing entry, the
original insertion position is changed and moved to the end:

@example
class LastUpdatedOrderedDict(OrderedDict):
    'Store items in the order the keys were last added'

    def __setitem__(self, key, value):
        super().__setitem__(key, value)
        self.move_to_end(key)
@end example

An @ref{4e5,,OrderedDict} would also be useful for implementing
variants of @ref{4f3,,functools.lru_cache()}:

@example
from time import time

class TimeBoundedLRU:
    "LRU Cache that invalidates and refreshes old entries."

    def __init__(self, func, maxsize=128, maxage=30):
        self.cache = OrderedDict()      # @{ args : (timestamp, result)@}
        self.func = func
        self.maxsize = maxsize
        self.maxage = maxage

    def __call__(self, *args):
        if args in self.cache:
            self.cache.move_to_end(args)
            timestamp, result = self.cache[args]
            if time() - timestamp <= self.maxage:
                return result
        result = self.func(*args)
        self.cache[args] = time(), result
        if len(self.cache) > self.maxsize:
            self.cache.popitem(0)
        return result
@end example

@example
class MultiHitLRUCache:
    """ LRU cache that defers caching a result until
        it has been requested multiple times.

        To avoid flushing the LRU cache with one-time requests,
        we don't cache until a request has been made more than once.

    """

    def __init__(self, func, maxsize=128, maxrequests=4096, cache_after=1):
        self.requests = OrderedDict()   # @{ uncached_key : request_count @}
        self.cache = OrderedDict()      # @{ cached_key : function_result @}
        self.func = func
        self.maxrequests = maxrequests  # max number of uncached requests
        self.maxsize = maxsize          # max number of stored return values
        self.cache_after = cache_after

    def __call__(self, *args):
        if args in self.cache:
            self.cache.move_to_end(args)
            return self.cache[args]
        result = self.func(*args)
        self.requests[args] = self.requests.get(args, 0) + 1
        if self.requests[args] <= self.cache_after:
            self.requests.move_to_end(args)
            if len(self.requests) > self.maxrequests:
                self.requests.popitem(0)
        else:
            self.requests.pop(args, None)
            self.cache[args] = result
            if len(self.cache) > self.maxsize:
                self.cache.popitem(0)
        return result
@end example

@c >>> def square(x):
@c ...     return x * x
@c ...
@c >>> f = MultiHitLRUCache(square, maxsize=4, maxrequests=6)
@c >>> list(map(f, range(10)))  # First requests, don't cache
@c [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
@c >>> f(4)  # Cache the second request
@c 16
@c >>> f(6)  # Cache the second request
@c 36
@c >>> f(2)  # The first request aged out, so don't cache
@c 4
@c >>> f(6)  # Cache hit
@c 36
@c >>> f(4)  # Cache hit and move to front
@c 16
@c >>> list(f.cache.values())
@c [36, 16]
@c >>> set(f.requests).isdisjoint(f.cache)
@c True
@c >>> list(map(f, [9, 8, 7]))   # Cache these second requests
@c [81, 64, 49]
@c >>> list(map(f, [7, 9]))  # Cache hits
@c [49, 81]
@c >>> list(f.cache.values())
@c [16, 64, 49, 81]
@c >>> set(f.requests).isdisjoint(f.cache)
@c True

@node UserDict objects,UserList objects,OrderedDict objects,collections — Container datatypes
@anchor{library/collections userdict-objects}@anchor{1de0}
@subsubsection @code{UserDict} objects


The class, @ref{1225,,UserDict} acts as a wrapper around dictionary objects.
The need for this class has been partially supplanted by the ability to
subclass directly from @ref{3b0,,dict}; however, this class can be easier
to work with because the underlying dictionary is accessible as an
attribute.

@geindex UserDict (class in collections)
@anchor{library/collections collections UserDict}@anchor{1225}
@deffn {Class} collections.UserDict ([initialdata])

Class that simulates a dictionary.  The instance’s contents are kept in a
regular dictionary, which is accessible via the @ref{1de1,,data} attribute of
@ref{1225,,UserDict} instances.  If @emph{initialdata} is provided, @ref{1de1,,data} is
initialized with its contents; note that a reference to @emph{initialdata} will not
be kept, allowing it to be used for other purposes.

In addition to supporting the methods and operations of mappings,
@ref{1225,,UserDict} instances provide the following attribute:

@geindex data (collections.UserDict attribute)
@anchor{library/collections collections UserDict data}@anchor{1de1}
@deffn {Attribute} data

A real dictionary used to store the contents of the @ref{1225,,UserDict}
class.
@end deffn
@end deffn

@node UserList objects,UserString objects,UserDict objects,collections — Container datatypes
@anchor{library/collections userlist-objects}@anchor{1de2}
@subsubsection @code{UserList} objects


This class acts as a wrapper around list objects.  It is a useful base class
for your own list-like classes which can inherit from them and override
existing methods or add new ones.  In this way, one can add new behaviors to
lists.

The need for this class has been partially supplanted by the ability to
subclass directly from @ref{43a,,list}; however, this class can be easier
to work with because the underlying list is accessible as an attribute.

@geindex UserList (class in collections)
@anchor{library/collections collections UserList}@anchor{1dc2}
@deffn {Class} collections.UserList ([list])

Class that simulates a list.  The instance’s contents are kept in a regular
list, which is accessible via the @ref{1de3,,data} attribute of @ref{1dc2,,UserList}
instances.  The instance’s contents are initially set to a copy of @emph{list},
defaulting to the empty list @code{[]}.  @emph{list} can be any iterable, for
example a real Python list or a @ref{1dc2,,UserList} object.

In addition to supporting the methods and operations of mutable sequences,
@ref{1dc2,,UserList} instances provide the following attribute:

@geindex data (collections.UserList attribute)
@anchor{library/collections collections UserList data}@anchor{1de3}
@deffn {Attribute} data

A real @ref{43a,,list} object used to store the contents of the
@ref{1dc2,,UserList} class.
@end deffn
@end deffn

@strong{Subclassing requirements:} Subclasses of @ref{1dc2,,UserList} are expected to
offer a constructor which can be called with either no arguments or one
argument.  List operations which return a new sequence attempt to create an
instance of the actual implementation class.  To do so, it assumes that the
constructor can be called with a single parameter, which is a sequence object
used as a data source.

If a derived class does not wish to comply with this requirement, all of the
special methods supported by this class will need to be overridden; please
consult the sources for information about the methods which need to be provided
in that case.

@node UserString objects,,UserList objects,collections — Container datatypes
@anchor{library/collections userstring-objects}@anchor{1de4}
@subsubsection @code{UserString} objects


The class, @ref{934,,UserString} acts as a wrapper around string objects.
The need for this class has been partially supplanted by the ability to
subclass directly from @ref{2ef,,str}; however, this class can be easier
to work with because the underlying string is accessible as an
attribute.

@geindex UserString (class in collections)
@anchor{library/collections collections UserString}@anchor{934}
@deffn {Class} collections.UserString (seq)

Class that simulates a string object.  The instance’s
content is kept in a regular string object, which is accessible via the
@ref{1de5,,data} attribute of @ref{934,,UserString} instances.  The instance’s
contents are initially set to a copy of @emph{seq}.  The @emph{seq} argument can
be any object which can be converted into a string using the built-in
@ref{2ef,,str()} function.

In addition to supporting the methods and operations of strings,
@ref{934,,UserString} instances provide the following attribute:

@geindex data (collections.UserString attribute)
@anchor{library/collections collections UserString data}@anchor{1de5}
@deffn {Attribute} data

A real @ref{2ef,,str} object used to store the contents of the
@ref{934,,UserString} class.
@end deffn


Changed in version 3.5: New methods @code{__getnewargs__}, @code{__rmod__}, @code{casefold},
@code{format_map}, @code{isprintable}, and @code{maketrans}.

@end deffn

@node collections abc — Abstract Base Classes for Containers,heapq — Heap queue algorithm,collections — Container datatypes,Data Types
@anchor{library/collections abc doc}@anchor{1de6}@anchor{library/collections abc collections-abc-abstract-base-classes-for-containers}@anchor{1de7}@anchor{library/collections abc module-collections abc}@anchor{1d}
@subsection @code{collections.abc} — Abstract Base Classes for Containers


@geindex module; collections.abc


New in version 3.3: Formerly, this module was part of the @ref{1c,,collections} module.


@strong{Source code:} Lib/_collections_abc.py@footnote{https://github.com/python/cpython/tree/main/Lib/_collections_abc.py}

@c from collections.abc import *
@c import itertools
@c __name__ = '<doctest>'


__________________________________________________________________


This module provides @ref{d5c,,abstract base classes} that
can be used to test whether a class provides a particular interface; for
example, whether it is hashable or whether it is a mapping.

An @ref{237,,issubclass()} or @ref{236,,isinstance()} test for an interface works in one
of three ways.

1) A newly written class can inherit directly from one of the
abstract base classes.  The class must supply the required abstract
methods.  The remaining mixin methods come from inheritance and can be
overridden if desired.  Other methods may be added as needed:

@example
class C(Sequence):                      # Direct inheritance
    def __init__(self): ...             # Extra method not required by the ABC
    def __getitem__(self, index):  ...  # Required abstract method
    def __len__(self):  ...             # Required abstract method
    def count(self, value): ...         # Optionally override a mixin method
@end example

@example
>>> issubclass(C, Sequence)
True
>>> isinstance(C(), Sequence)
True
@end example

2) Existing classes and built-in classes can be registered as “virtual
subclasses” of the ABCs.  Those classes should define the full API
including all of the abstract methods and all of the mixin methods.
This lets users rely on @ref{237,,issubclass()} or @ref{236,,isinstance()} tests
to determine whether the full interface is supported.  The exception to
this rule is for methods that are automatically inferred from the rest
of the API:

@example
class D:                                 # No inheritance
    def __init__(self): ...              # Extra method not required by the ABC
    def __getitem__(self, index):  ...   # Abstract method
    def __len__(self):  ...              # Abstract method
    def count(self, value): ...          # Mixin method
    def index(self, value): ...          # Mixin method

Sequence.register(D)                     # Register instead of inherit
@end example

@example
>>> issubclass(D, Sequence)
True
>>> isinstance(D(), Sequence)
True
@end example

In this example, class @code{D} does not need to define
@code{__contains__}, @code{__iter__}, and @code{__reversed__} because the
@ref{1900,,in-operator}, the @ref{dcc,,iteration}
logic, and the @ref{2f6,,reversed()} function automatically fall back to
using @code{__getitem__} and @code{__len__}.

3) Some simple interfaces are directly recognizable by the presence of
the required methods (unless those methods have been set to
@ref{182,,None}):

@example
class E:
    def __iter__(self): ...
    def __next__(next): ...
@end example

@example
>>> issubclass(E, Iterable)
True
>>> isinstance(E(), Iterable)
True
@end example

Complex interfaces do not support this last technique because an
interface is more than just the presence of method names.  Interfaces
specify semantics and relationships between methods that cannot be
inferred solely from the presence of specific method names.  For
example, knowing that a class supplies @code{__getitem__}, @code{__len__}, and
@code{__iter__} is insufficient for distinguishing a @ref{110c,,Sequence} from
a @ref{3ab,,Mapping}.


New in version 3.9: These abstract classes now support @code{[]}. See @ref{268,,Generic Alias Type}
and 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}.


@menu
* Collections Abstract Base Classes:: 
* Collections Abstract Base Classes – Detailed Descriptions:: 
* Examples and Recipes:: 

@end menu

@node Collections Abstract Base Classes,Collections Abstract Base Classes – Detailed Descriptions,,collections abc — Abstract Base Classes for Containers
@anchor{library/collections abc collections-abstract-base-classes}@anchor{33d}@anchor{library/collections abc id1}@anchor{1de8}
@subsubsection Collections Abstract Base Classes


The collections module offers the following @ref{d5c,,ABCs}:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

ABC

@tab

Inherits from

@tab

Abstract Methods

@tab

Mixin Methods

@item

@ref{1b3e,,Container} @footnote{@w{(1)} 
These ABCs override @code{object.__subclasshook__()} to support
testing an interface by verifying the required methods are present
and have not been set to @ref{182,,None}.  This only works for simple
interfaces.  More complex interfaces require registration or direct
subclassing.
}

@tab

@tab

@code{__contains__}

@tab

@item

@ref{1de9,,Hashable} @footnote{@w{(1)} 
These ABCs override @code{object.__subclasshook__()} to support
testing an interface by verifying the required methods are present
and have not been set to @ref{182,,None}.  This only works for simple
interfaces.  More complex interfaces require registration or direct
subclassing.
}

@tab

@tab

@code{__hash__}

@tab

@item

@ref{1b3c,,Iterable} @footnote{@w{(1)} 
These ABCs override @code{object.__subclasshook__()} to support
testing an interface by verifying the required methods are present
and have not been set to @ref{182,,None}.  This only works for simple
interfaces.  More complex interfaces require registration or direct
subclassing.
} @footnote{@w{(2)} 
Checking @code{isinstance(obj, Iterable)} detects classes that are
registered as @ref{1b3c,,Iterable} or that have an @code{__iter__()}
method, but it does not detect classes that iterate with the
@code{__getitem__()} method.  The only reliable way to determine
whether an object is @ref{dcc,,iterable} is to call @code{iter(obj)}.
}

@tab

@tab

@code{__iter__}

@tab

@item

@ref{1b3d,,Iterator} @footnote{@w{(1)} 
These ABCs override @code{object.__subclasshook__()} to support
testing an interface by verifying the required methods are present
and have not been set to @ref{182,,None}.  This only works for simple
interfaces.  More complex interfaces require registration or direct
subclassing.
}

@tab

@ref{1b3c,,Iterable}

@tab

@code{__next__}

@tab

@code{__iter__}

@item

@ref{7d1,,Reversible} @footnote{@w{(1)} 
These ABCs override @code{object.__subclasshook__()} to support
testing an interface by verifying the required methods are present
and have not been set to @ref{182,,None}.  This only works for simple
interfaces.  More complex interfaces require registration or direct
subclassing.
}

@tab

@ref{1b3c,,Iterable}

@tab

@code{__reversed__}

@tab

@item

@ref{93a,,Generator}  @footnote{@w{(1)} 
These ABCs override @code{object.__subclasshook__()} to support
testing an interface by verifying the required methods are present
and have not been set to @ref{182,,None}.  This only works for simple
interfaces.  More complex interfaces require registration or direct
subclassing.
}

@tab

@ref{1b3d,,Iterator}

@tab

@code{send}, @code{throw}

@tab

@code{close}, @code{__iter__}, @code{__next__}

@item

@ref{1dea,,Sized}  @footnote{@w{(1)} 
These ABCs override @code{object.__subclasshook__()} to support
testing an interface by verifying the required methods are present
and have not been set to @ref{182,,None}.  This only works for simple
interfaces.  More complex interfaces require registration or direct
subclassing.
}

@tab

@tab

@code{__len__}

@tab

@item

@ref{269,,Callable}  @footnote{@w{(1)} 
These ABCs override @code{object.__subclasshook__()} to support
testing an interface by verifying the required methods are present
and have not been set to @ref{182,,None}.  This only works for simple
interfaces.  More complex interfaces require registration or direct
subclassing.
}

@tab

@tab

@code{__call__}

@tab

@item

@ref{7d0,,Collection}  @footnote{@w{(1)} 
These ABCs override @code{object.__subclasshook__()} to support
testing an interface by verifying the required methods are present
and have not been set to @ref{182,,None}.  This only works for simple
interfaces.  More complex interfaces require registration or direct
subclassing.
}

@tab

@ref{1dea,,Sized},
@ref{1b3c,,Iterable},
@ref{1b3e,,Container}

@tab

@code{__contains__},
@code{__iter__},
@code{__len__}

@tab

@item

@ref{110c,,Sequence}

@tab

@ref{7d1,,Reversible},
@ref{7d0,,Collection}

@tab

@code{__getitem__},
@code{__len__}

@tab

@code{__contains__}, @code{__iter__}, @code{__reversed__},
@code{index}, and @code{count}

@item

@ref{933,,MutableSequence}

@tab

@ref{110c,,Sequence}

@tab

@code{__getitem__},
@code{__setitem__},
@code{__delitem__},
@code{__len__},
@code{insert}

@tab

Inherited @ref{110c,,Sequence} methods and
@code{append}, @code{reverse}, @code{extend}, @code{pop},
@code{remove}, and @code{__iadd__}

@item

@ref{1b40,,ByteString}

@tab

@ref{110c,,Sequence}

@tab

@code{__getitem__},
@code{__len__}

@tab

Inherited @ref{110c,,Sequence} methods

@item

@ref{1b33,,Set}

@tab

@ref{7d0,,Collection}

@tab

@code{__contains__},
@code{__iter__},
@code{__len__}

@tab

@code{__le__}, @code{__lt__}, @code{__eq__}, @code{__ne__},
@code{__gt__}, @code{__ge__}, @code{__and__}, @code{__or__},
@code{__sub__}, @code{__xor__}, and @code{isdisjoint}

@item

@ref{1b3f,,MutableSet}

@tab

@ref{1b33,,Set}

@tab

@code{__contains__},
@code{__iter__},
@code{__len__},
@code{add},
@code{discard}

@tab

Inherited @ref{1b33,,Set} methods and
@code{clear}, @code{pop}, @code{remove}, @code{__ior__},
@code{__iand__}, @code{__ixor__}, and @code{__isub__}

@item

@ref{3ab,,Mapping}

@tab

@ref{7d0,,Collection}

@tab

@code{__getitem__},
@code{__iter__},
@code{__len__}

@tab

@code{__contains__}, @code{keys}, @code{items}, @code{values},
@code{get}, @code{__eq__}, and @code{__ne__}

@item

@ref{c3f,,MutableMapping}

@tab

@ref{3ab,,Mapping}

@tab

@code{__getitem__},
@code{__setitem__},
@code{__delitem__},
@code{__iter__},
@code{__len__}

@tab

Inherited @ref{3ab,,Mapping} methods and
@code{pop}, @code{popitem}, @code{clear}, @code{update},
and @code{setdefault}

@item

@ref{1b41,,MappingView}

@tab

@ref{1dea,,Sized}

@tab

@tab

@code{__len__}

@item

@ref{1b43,,ItemsView}

@tab

@ref{1b41,,MappingView},
@ref{1b33,,Set}

@tab

@tab

@code{__contains__},
@code{__iter__}

@item

@ref{1b42,,KeysView}

@tab

@ref{1b41,,MappingView},
@ref{1b33,,Set}

@tab

@tab

@code{__contains__},
@code{__iter__}

@item

@ref{1b44,,ValuesView}

@tab

@ref{1b41,,MappingView},
@ref{7d0,,Collection}

@tab

@tab

@code{__contains__}, @code{__iter__}

@item

@ref{93b,,Awaitable} @footnote{@w{(1)} 
These ABCs override @code{object.__subclasshook__()} to support
testing an interface by verifying the required methods are present
and have not been set to @ref{182,,None}.  This only works for simple
interfaces.  More complex interfaces require registration or direct
subclassing.
}

@tab

@tab

@code{__await__}

@tab

@item

@ref{93c,,Coroutine} @footnote{@w{(1)} 
These ABCs override @code{object.__subclasshook__()} to support
testing an interface by verifying the required methods are present
and have not been set to @ref{182,,None}.  This only works for simple
interfaces.  More complex interfaces require registration or direct
subclassing.
}

@tab

@ref{93b,,Awaitable}

@tab

@code{send}, @code{throw}

@tab

@code{close}

@item

@ref{93e,,AsyncIterable} @footnote{@w{(1)} 
These ABCs override @code{object.__subclasshook__()} to support
testing an interface by verifying the required methods are present
and have not been set to @ref{182,,None}.  This only works for simple
interfaces.  More complex interfaces require registration or direct
subclassing.
}

@tab

@tab

@code{__aiter__}

@tab

@item

@ref{93d,,AsyncIterator} @footnote{@w{(1)} 
These ABCs override @code{object.__subclasshook__()} to support
testing an interface by verifying the required methods are present
and have not been set to @ref{182,,None}.  This only works for simple
interfaces.  More complex interfaces require registration or direct
subclassing.
}

@tab

@ref{93e,,AsyncIterable}

@tab

@code{__anext__}

@tab

@code{__aiter__}

@item

@ref{7d2,,AsyncGenerator} @footnote{@w{(1)} 
These ABCs override @code{object.__subclasshook__()} to support
testing an interface by verifying the required methods are present
and have not been set to @ref{182,,None}.  This only works for simple
interfaces.  More complex interfaces require registration or direct
subclassing.
}

@tab

@ref{93d,,AsyncIterator}

@tab

@code{asend}, @code{athrow}

@tab

@code{aclose}, @code{__aiter__}, @code{__anext__}

@end multitable


@node Collections Abstract Base Classes – Detailed Descriptions,Examples and Recipes,Collections Abstract Base Classes,collections abc — Abstract Base Classes for Containers
@anchor{library/collections abc collections-abstract-base-classes-detailed-descriptions}@anchor{1deb}
@subsubsection Collections Abstract Base Classes – Detailed Descriptions


@geindex Container (class in collections.abc)
@anchor{library/collections abc collections abc Container}@anchor{1b3e}
@deffn {Class} collections.abc.Container

ABC for classes that provide the @code{__contains__()} method.
@end deffn

@geindex Hashable (class in collections.abc)
@anchor{library/collections abc collections abc Hashable}@anchor{1de9}
@deffn {Class} collections.abc.Hashable

ABC for classes that provide the @code{__hash__()} method.
@end deffn

@geindex Sized (class in collections.abc)
@anchor{library/collections abc collections abc Sized}@anchor{1dea}
@deffn {Class} collections.abc.Sized

ABC for classes that provide the @code{__len__()} method.
@end deffn

@geindex Callable (class in collections.abc)
@anchor{library/collections abc collections abc Callable}@anchor{269}
@deffn {Class} collections.abc.Callable

ABC for classes that provide the @code{__call__()} method.
@end deffn

@geindex Iterable (class in collections.abc)
@anchor{library/collections abc collections abc Iterable}@anchor{1b3c}
@deffn {Class} collections.abc.Iterable

ABC for classes that provide the @code{__iter__()} method.

Checking @code{isinstance(obj, Iterable)} detects classes that are registered
as @ref{1b3c,,Iterable} or that have an @code{__iter__()} method, but it does
not detect classes that iterate with the @code{__getitem__()} method.
The only reliable way to determine whether an object is @ref{dcc,,iterable}
is to call @code{iter(obj)}.
@end deffn

@geindex Collection (class in collections.abc)
@anchor{library/collections abc collections abc Collection}@anchor{7d0}
@deffn {Class} collections.abc.Collection

ABC for sized iterable container classes.


New in version 3.6.

@end deffn

@geindex Iterator (class in collections.abc)
@anchor{library/collections abc collections abc Iterator}@anchor{1b3d}
@deffn {Class} collections.abc.Iterator

ABC for classes that provide the @ref{1a84,,__iter__()} and
@ref{e6d,,__next__()} methods.  See also the definition of
@ref{11c3,,iterator}.
@end deffn

@geindex Reversible (class in collections.abc)
@anchor{library/collections abc collections abc Reversible}@anchor{7d1}
@deffn {Class} collections.abc.Reversible

ABC for iterable classes that also provide the @code{__reversed__()}
method.


New in version 3.6.

@end deffn

@geindex Generator (class in collections.abc)
@anchor{library/collections abc collections abc Generator}@anchor{93a}
@deffn {Class} collections.abc.Generator

ABC for generator classes that implement the protocol defined in
@geindex Python Enhancement Proposals; PEP 342
PEP 342@footnote{https://www.python.org/dev/peps/pep-0342} that extends iterators with the @ref{181c,,send()},
@ref{11be,,throw()} and @ref{181f,,close()} methods.
See also the definition of @ref{bf7,,generator}.


New in version 3.5.

@end deffn

@geindex Sequence (class in collections.abc)
@geindex MutableSequence (class in collections.abc)
@geindex ByteString (class in collections.abc)
@anchor{library/collections abc collections abc Sequence}@anchor{110c}
@deffn {Class} collections.abc.Sequence
@anchor{library/collections abc collections abc MutableSequence}@anchor{933}
@deffnx {Class} collections.abc.MutableSequence
@anchor{library/collections abc collections abc ByteString}@anchor{1b40}
@deffnx {Class} collections.abc.ByteString

ABCs for read-only and mutable @ref{14e7,,sequences}.

Implementation note: Some of the mixin methods, such as
@code{__iter__()}, @code{__reversed__()} and @code{index()}, make
repeated calls to the underlying @code{__getitem__()} method.
Consequently, if @code{__getitem__()} is implemented with constant
access speed, the mixin methods will have linear performance;
however, if the underlying method is linear (as it would be with a
linked list), the mixins will have quadratic performance and will
likely need to be overridden.


Changed in version 3.5: The index() method added support for @emph{stop} and @emph{start}
arguments.

@end deffn

@geindex Set (class in collections.abc)
@geindex MutableSet (class in collections.abc)
@anchor{library/collections abc collections abc Set}@anchor{1b33}
@deffn {Class} collections.abc.Set
@anchor{library/collections abc collections abc MutableSet}@anchor{1b3f}
@deffnx {Class} collections.abc.MutableSet

ABCs for read-only and mutable sets.
@end deffn

@geindex Mapping (class in collections.abc)
@geindex MutableMapping (class in collections.abc)
@anchor{library/collections abc collections abc Mapping}@anchor{3ab}
@deffn {Class} collections.abc.Mapping
@anchor{library/collections abc collections abc MutableMapping}@anchor{c3f}
@deffnx {Class} collections.abc.MutableMapping

ABCs for read-only and mutable @ref{d62,,mappings}.
@end deffn

@geindex MappingView (class in collections.abc)
@geindex ItemsView (class in collections.abc)
@geindex KeysView (class in collections.abc)
@geindex ValuesView (class in collections.abc)
@anchor{library/collections abc collections abc MappingView}@anchor{1b41}
@deffn {Class} collections.abc.MappingView
@anchor{library/collections abc collections abc ItemsView}@anchor{1b43}
@deffnx {Class} collections.abc.ItemsView
@anchor{library/collections abc collections abc KeysView}@anchor{1b42}
@deffnx {Class} collections.abc.KeysView
@anchor{library/collections abc collections abc ValuesView}@anchor{1b44}
@deffnx {Class} collections.abc.ValuesView

ABCs for mapping, items, keys, and values @ref{1dde,,views}.
@end deffn

@geindex Awaitable (class in collections.abc)
@anchor{library/collections abc collections abc Awaitable}@anchor{93b}
@deffn {Class} collections.abc.Awaitable

ABC for @ref{7bd,,awaitable} objects, which can be used in @ref{4d5,,await}
expressions.  Custom implementations must provide the @code{__await__()}
method.

@ref{3cf,,Coroutine} objects and instances of the
@ref{93c,,Coroutine} ABC are all instances of this ABC.

@cartouche
@quotation Note 
In CPython, generator-based coroutines (generators decorated with
@ref{9fa,,types.coroutine()}) are
@emph{awaitables}, even though they do not have an @code{__await__()} method.
Using @code{isinstance(gencoro, Awaitable)} for them will return @code{False}.
Use @ref{97a,,inspect.isawaitable()} to detect them.
@end quotation
@end cartouche


New in version 3.5.

@end deffn

@geindex Coroutine (class in collections.abc)
@anchor{library/collections abc collections abc Coroutine}@anchor{93c}
@deffn {Class} collections.abc.Coroutine

ABC for coroutine compatible classes.  These implement the
following methods, defined in @ref{1792,,Coroutine Objects}:
@ref{181b,,send()}, @ref{181d,,throw()}, and
@ref{181e,,close()}.  Custom implementations must also implement
@code{__await__()}.  All @ref{93c,,Coroutine} instances are also instances of
@ref{93b,,Awaitable}.  See also the definition of @ref{3cf,,coroutine}.

@cartouche
@quotation Note 
In CPython, generator-based coroutines (generators decorated with
@ref{9fa,,types.coroutine()}) are
@emph{awaitables}, even though they do not have an @code{__await__()} method.
Using @code{isinstance(gencoro, Coroutine)} for them will return @code{False}.
Use @ref{97a,,inspect.isawaitable()} to detect them.
@end quotation
@end cartouche


New in version 3.5.

@end deffn

@geindex AsyncIterable (class in collections.abc)
@anchor{library/collections abc collections abc AsyncIterable}@anchor{93e}
@deffn {Class} collections.abc.AsyncIterable

ABC for classes that provide @code{__aiter__} method.  See also the
definition of @ref{8d0,,asynchronous iterable}.


New in version 3.5.

@end deffn

@geindex AsyncIterator (class in collections.abc)
@anchor{library/collections abc collections abc AsyncIterator}@anchor{93d}
@deffn {Class} collections.abc.AsyncIterator

ABC for classes that provide @code{__aiter__} and @code{__anext__}
methods.  See also the definition of @ref{8d2,,asynchronous iterator}.


New in version 3.5.

@end deffn

@geindex AsyncGenerator (class in collections.abc)
@anchor{library/collections abc collections abc AsyncGenerator}@anchor{7d2}
@deffn {Class} collections.abc.AsyncGenerator

ABC for asynchronous generator classes that implement the protocol
defined in 
@geindex Python Enhancement Proposals; PEP 525
PEP 525@footnote{https://www.python.org/dev/peps/pep-0525} and 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492}.


New in version 3.6.

@end deffn

@node Examples and Recipes,,Collections Abstract Base Classes – Detailed Descriptions,collections abc — Abstract Base Classes for Containers
@anchor{library/collections abc examples-and-recipes}@anchor{1dec}
@subsubsection Examples and Recipes


ABCs allow us to ask classes or instances if they provide
particular functionality, for example:

@example
size = None
if isinstance(myvar, collections.abc.Sized):
    size = len(myvar)
@end example

Several of the ABCs are also useful as mixins that make it easier to develop
classes supporting container APIs.  For example, to write a class supporting
the full @ref{1b33,,Set} API, it is only necessary to supply the three underlying
abstract methods: @code{__contains__()}, @code{__iter__()}, and @code{__len__()}.
The ABC supplies the remaining methods such as @code{__and__()} and
@code{isdisjoint()}:

@example
class ListBasedSet(collections.abc.Set):
    '@w{'}' Alternate set implementation favoring space over speed
        and not requiring the set elements to be hashable. '@w{'}'
    def __init__(self, iterable):
        self.elements = lst = []
        for value in iterable:
            if value not in lst:
                lst.append(value)

    def __iter__(self):
        return iter(self.elements)

    def __contains__(self, value):
        return value in self.elements

    def __len__(self):
        return len(self.elements)

s1 = ListBasedSet('abcdef')
s2 = ListBasedSet('defghi')
overlap = s1 & s2            # The __and__() method is supported automatically
@end example

Notes on using @ref{1b33,,Set} and @ref{1b3f,,MutableSet} as a mixin:


@enumerate 

@item 
Since some set operations create new sets, the default mixin methods need
a way to create new instances from an iterable. The class constructor is
assumed to have a signature in the form @code{ClassName(iterable)}.
That assumption is factored-out to an internal classmethod called
@code{_from_iterable()} which calls @code{cls(iterable)} to produce a new set.
If the @ref{1b33,,Set} mixin is being used in a class with a different
constructor signature, you will need to override @code{_from_iterable()}
with a classmethod or regular method that can construct new instances from
an iterable argument.

@item 
To override the comparisons (presumably for speed, as the
semantics are fixed), redefine @code{__le__()} and @code{__ge__()},
then the other operations will automatically follow suit.

@item 
The @ref{1b33,,Set} mixin provides a @code{_hash()} method to compute a hash value
for the set; however, @code{__hash__()} is not defined because not all sets
are hashable or immutable.  To add set hashability using mixins,
inherit from both @ref{1b33,,Set()} and @ref{1de9,,Hashable()}, then define
@code{__hash__ = Set._hash}.
@end enumerate


@subsubheading See also



@itemize *

@item 
OrderedSet recipe@footnote{https://code.activestate.com/recipes/576694/} for an
example built on @ref{1b3f,,MutableSet}.

@item 
For more about ABCs, see the @ref{3,,abc} module and 
@geindex Python Enhancement Proposals; PEP 3119
PEP 3119@footnote{https://www.python.org/dev/peps/pep-3119}.
@end itemize


@node heapq — Heap queue algorithm,bisect — Array bisection algorithm,collections abc — Abstract Base Classes for Containers,Data Types
@anchor{library/heapq doc}@anchor{1ded}@anchor{library/heapq heapq-heap-queue-algorithm}@anchor{1dee}@anchor{library/heapq module-heapq}@anchor{89}
@subsection @code{heapq} — Heap queue algorithm


@geindex module; heapq

@strong{Source code:} Lib/heapq.py@footnote{https://github.com/python/cpython/tree/main/Lib/heapq.py}


__________________________________________________________________


This module provides an implementation of the heap queue algorithm, also known
as the priority queue algorithm.

Heaps are binary trees for which every parent node has a value less than or
equal to any of its children.  This implementation uses arrays for which
@code{heap[k] <= heap[2*k+1]} and @code{heap[k] <= heap[2*k+2]} for all @emph{k}, counting
elements from zero.  For the sake of comparison, non-existing elements are
considered to be infinite.  The interesting property of a heap is that its
smallest element is always the root, @code{heap[0]}.

The API below differs from textbook heap algorithms in two aspects: (a) We use
zero-based indexing.  This makes the relationship between the index for a node
and the indexes for its children slightly less obvious, but is more suitable
since Python uses zero-based indexing. (b) Our pop method returns the smallest
item, not the largest (called a “min heap” in textbooks; a “max heap” is more
common in texts because of its suitability for in-place sorting).

These two make it possible to view the heap as a regular Python list without
surprises: @code{heap[0]} is the smallest item, and @code{heap.sort()} maintains the
heap invariant!

To create a heap, use a list initialized to @code{[]}, or you can transform a
populated list into a heap via function @ref{1def,,heapify()}.

The following functions are provided:

@geindex heappush() (in module heapq)
@anchor{library/heapq heapq heappush}@anchor{1df0}
@deffn {Function} heapq.heappush (heap, item)

Push the value @emph{item} onto the @emph{heap}, maintaining the heap invariant.
@end deffn

@geindex heappop() (in module heapq)
@anchor{library/heapq heapq heappop}@anchor{1df1}
@deffn {Function} heapq.heappop (heap)

Pop and return the smallest item from the @emph{heap}, maintaining the heap
invariant.  If the heap is empty, @ref{1066,,IndexError} is raised.  To access the
smallest item without popping it, use @code{heap[0]}.
@end deffn

@geindex heappushpop() (in module heapq)
@anchor{library/heapq heapq heappushpop}@anchor{1df2}
@deffn {Function} heapq.heappushpop (heap, item)

Push @emph{item} on the heap, then pop and return the smallest item from the
@emph{heap}.  The combined action runs more efficiently than @ref{1df0,,heappush()}
followed by a separate call to @ref{1df1,,heappop()}.
@end deffn

@geindex heapify() (in module heapq)
@anchor{library/heapq heapq heapify}@anchor{1def}
@deffn {Function} heapq.heapify (x)

Transform list @emph{x} into a heap, in-place, in linear time.
@end deffn

@geindex heapreplace() (in module heapq)
@anchor{library/heapq heapq heapreplace}@anchor{1df3}
@deffn {Function} heapq.heapreplace (heap, item)

Pop and return the smallest item from the @emph{heap}, and also push the new @emph{item}.
The heap size doesn’t change. If the heap is empty, @ref{1066,,IndexError} is raised.

This one step operation is more efficient than a @ref{1df1,,heappop()} followed by
@ref{1df0,,heappush()} and can be more appropriate when using a fixed-size heap.
The pop/push combination always returns an element from the heap and replaces
it with @emph{item}.

The value returned may be larger than the @emph{item} added.  If that isn’t
desired, consider using @ref{1df2,,heappushpop()} instead.  Its push/pop
combination returns the smaller of the two values, leaving the larger value
on the heap.
@end deffn

The module also offers three general purpose functions based on heaps.

@geindex merge() (in module heapq)
@anchor{library/heapq heapq merge}@anchor{963}
@deffn {Function} heapq.merge (*iterables, key=None, reverse=False)

Merge multiple sorted inputs into a single sorted output (for example, merge
timestamped entries from multiple log files).  Returns an @ref{11c3,,iterator}
over the sorted values.

Similar to @code{sorted(itertools.chain(*iterables))} but returns an iterable, does
not pull the data into memory all at once, and assumes that each of the input
streams is already sorted (smallest to largest).

Has two optional arguments which must be specified as keyword arguments.

@emph{key} specifies a @ref{964,,key function} of one argument that is used to
extract a comparison key from each input element.  The default value is
@code{None} (compare the elements directly).

@emph{reverse} is a boolean value.  If set to @code{True}, then the input elements
are merged as if each comparison were reversed. To achieve behavior similar
to @code{sorted(itertools.chain(*iterables), reverse=True)}, all iterables must
be sorted from largest to smallest.


Changed in version 3.5: Added the optional @emph{key} and @emph{reverse} parameters.

@end deffn

@geindex nlargest() (in module heapq)
@anchor{library/heapq heapq nlargest}@anchor{1df4}
@deffn {Function} heapq.nlargest (n, iterable, key=None)

Return a list with the @emph{n} largest elements from the dataset defined by
@emph{iterable}.  @emph{key}, if provided, specifies a function of one argument that is
used to extract a comparison key from each element in @emph{iterable} (for example,
@code{key=str.lower}).  Equivalent to:  @code{sorted(iterable, key=key,
reverse=True)[:n]}.
@end deffn

@geindex nsmallest() (in module heapq)
@anchor{library/heapq heapq nsmallest}@anchor{1df5}
@deffn {Function} heapq.nsmallest (n, iterable, key=None)

Return a list with the @emph{n} smallest elements from the dataset defined by
@emph{iterable}.  @emph{key}, if provided, specifies a function of one argument that is
used to extract a comparison key from each element in @emph{iterable} (for example,
@code{key=str.lower}).  Equivalent to:  @code{sorted(iterable, key=key)[:n]}.
@end deffn

The latter two functions perform best for smaller values of @emph{n}.  For larger
values, it is more efficient to use the @ref{704,,sorted()} function.  Also, when
@code{n==1}, it is more efficient to use the built-in @ref{a7d,,min()} and @ref{a7e,,max()}
functions.  If repeated usage of these functions is required, consider turning
the iterable into an actual heap.

@menu
* Basic Examples:: 
* Priority Queue Implementation Notes:: 
* Theory:: 

@end menu

@node Basic Examples,Priority Queue Implementation Notes,,heapq — Heap queue algorithm
@anchor{library/heapq basic-examples}@anchor{1df6}
@subsubsection Basic Examples


A heapsort@footnote{https://en.wikipedia.org/wiki/Heapsort} can be implemented by
pushing all values onto a heap and then popping off the smallest values one at a
time:

@example
>>> def heapsort(iterable):
...     h = []
...     for value in iterable:
...         heappush(h, value)
...     return [heappop(h) for i in range(len(h))]
...
>>> heapsort([1, 3, 5, 7, 9, 2, 4, 6, 8, 0])
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
@end example

This is similar to @code{sorted(iterable)}, but unlike @ref{704,,sorted()}, this
implementation is not stable.

Heap elements can be tuples.  This is useful for assigning comparison values
(such as task priorities) alongside the main record being tracked:

@example
>>> h = []
>>> heappush(h, (5, 'write code'))
>>> heappush(h, (7, 'release product'))
>>> heappush(h, (1, 'write spec'))
>>> heappush(h, (3, 'create tests'))
>>> heappop(h)
(1, 'write spec')
@end example

@node Priority Queue Implementation Notes,Theory,Basic Examples,heapq — Heap queue algorithm
@anchor{library/heapq priority-queue-implementation-notes}@anchor{1df7}
@subsubsection Priority Queue Implementation Notes


A priority queue@footnote{https://en.wikipedia.org/wiki/Priority_queue} is common use
for a heap, and it presents several implementation challenges:


@itemize *

@item 
Sort stability:  how do you get two tasks with equal priorities to be returned
in the order they were originally added?

@item 
Tuple comparison breaks for (priority, task) pairs if the priorities are equal
and the tasks do not have a default comparison order.

@item 
If the priority of a task changes, how do you move it to a new position in
the heap?

@item 
Or if a pending task needs to be deleted, how do you find it and remove it
from the queue?
@end itemize

A solution to the first two challenges is to store entries as 3-element list
including the priority, an entry count, and the task.  The entry count serves as
a tie-breaker so that two tasks with the same priority are returned in the order
they were added. And since no two entry counts are the same, the tuple
comparison will never attempt to directly compare two tasks.

Another solution to the problem of non-comparable tasks is to create a wrapper
class that ignores the task item and only compares the priority field:

@example
from dataclasses import dataclass, field
from typing import Any

@@dataclass(order=True)
class PrioritizedItem:
    priority: int
    item: Any=field(compare=False)
@end example

The remaining challenges revolve around finding a pending task and making
changes to its priority or removing it entirely.  Finding a task can be done
with a dictionary pointing to an entry in the queue.

Removing the entry or changing its priority is more difficult because it would
break the heap structure invariants.  So, a possible solution is to mark the
entry as removed and add a new entry with the revised priority:

@example
pq = []                         # list of entries arranged in a heap
entry_finder = @{@}               # mapping of tasks to entries
REMOVED = '<removed-task>'      # placeholder for a removed task
counter = itertools.count()     # unique sequence count

def add_task(task, priority=0):
    'Add a new task or update the priority of an existing task'
    if task in entry_finder:
        remove_task(task)
    count = next(counter)
    entry = [priority, count, task]
    entry_finder[task] = entry
    heappush(pq, entry)

def remove_task(task):
    'Mark an existing task as REMOVED.  Raise KeyError if not found.'
    entry = entry_finder.pop(task)
    entry[-1] = REMOVED

def pop_task():
    'Remove and return the lowest priority task. Raise KeyError if empty.'
    while pq:
        priority, count, task = heappop(pq)
        if task is not REMOVED:
            del entry_finder[task]
            return task
    raise KeyError('pop from an empty priority queue')
@end example

@node Theory,,Priority Queue Implementation Notes,heapq — Heap queue algorithm
@anchor{library/heapq theory}@anchor{1df8}
@subsubsection Theory


Heaps are arrays for which @code{a[k] <= a[2*k+1]} and @code{a[k] <= a[2*k+2]} for all
@emph{k}, counting elements from 0.  For the sake of comparison, non-existing
elements are considered to be infinite.  The interesting property of a heap is
that @code{a[0]} is always its smallest element.

The strange invariant above is meant to be an efficient memory representation
for a tournament.  The numbers below are @emph{k}, not @code{a[k]}:

@example
                               0

              1                                 2

      3               4                5               6

  7       8       9       10      11      12      13      14

15 16   17 18   19 20   21 22   23 24   25 26   27 28   29 30
@end example

In the tree above, each cell @emph{k} is topping @code{2*k+1} and @code{2*k+2}. In a usual
binary tournament we see in sports, each cell is the winner over the two cells
it tops, and we can trace the winner down the tree to see all opponents s/he
had.  However, in many computer applications of such tournaments, we do not need
to trace the history of a winner. To be more memory efficient, when a winner is
promoted, we try to replace it by something else at a lower level, and the rule
becomes that a cell and the two cells it tops contain three different items, but
the top cell “wins” over the two topped cells.

If this heap invariant is protected at all time, index 0 is clearly the overall
winner.  The simplest algorithmic way to remove it and find the “next” winner is
to move some loser (let’s say cell 30 in the diagram above) into the 0 position,
and then percolate this new 0 down the tree, exchanging values, until the
invariant is re-established. This is clearly logarithmic on the total number of
items in the tree. By iterating over all items, you get an O(n log n) sort.

A nice feature of this sort is that you can efficiently insert new items while
the sort is going on, provided that the inserted items are not “better” than the
last 0’th element you extracted.  This is especially useful in simulation
contexts, where the tree holds all incoming events, and the “win” condition
means the smallest scheduled time.  When an event schedules other events for
execution, they are scheduled into the future, so they can easily go into the
heap.  So, a heap is a good structure for implementing schedulers (this is what
I used for my MIDI sequencer :-).

Various structures for implementing schedulers have been extensively studied,
and heaps are good for this, as they are reasonably speedy, the speed is almost
constant, and the worst case is not much different than the average case.
However, there are other representations which are more efficient overall, yet
the worst cases might be terrible.

Heaps are also very useful in big disk sorts.  You most probably all know that a
big sort implies producing “runs” (which are pre-sorted sequences, whose size is
usually related to the amount of CPU memory), followed by a merging passes for
these runs, which merging is often very cleverly organised @footnote{@w{(1)} 
The disk balancing algorithms which are current, nowadays, are more annoying
than clever, and this is a consequence of the seeking capabilities of the disks.
On devices which cannot seek, like big tape drives, the story was quite
different, and one had to be very clever to ensure (far in advance) that each
tape movement will be the most effective possible (that is, will best
participate at “progressing” the merge).  Some tapes were even able to read
backwards, and this was also used to avoid the rewinding time. Believe me, real
good tape sorts were quite spectacular to watch! From all times, sorting has
always been a Great Art! :-)
}. It is very
important that the initial sort produces the longest runs possible.  Tournaments
are a good way to achieve that.  If, using all the memory available to hold a
tournament, you replace and percolate items that happen to fit the current run,
you’ll produce runs which are twice the size of the memory for random input, and
much better for input fuzzily ordered.

Moreover, if you output the 0’th item on disk and get an input which may not fit
in the current tournament (because the value “wins” over the last output value),
it cannot fit in the heap, so the size of the heap decreases.  The freed memory
could be cleverly reused immediately for progressively building a second heap,
which grows at exactly the same rate the first heap is melting.  When the first
heap completely vanishes, you switch heaps and start a new run.  Clever and
quite effective!

In a word, heaps are useful memory structures to know.  I use them in a few
applications, and I think it is good to keep a ‘heap’ module around. :-)

@node bisect — Array bisection algorithm,array — Efficient arrays of numeric values,heapq — Heap queue algorithm,Data Types
@anchor{library/bisect doc}@anchor{1df9}@anchor{library/bisect bisect-array-bisection-algorithm}@anchor{1dfa}@anchor{library/bisect module-bisect}@anchor{10}
@subsection @code{bisect} — Array bisection algorithm


@geindex module; bisect

@c example based on the PyModules FAQ entry by Aaron Watters <arw@pythonpros.com>

@strong{Source code:} Lib/bisect.py@footnote{https://github.com/python/cpython/tree/main/Lib/bisect.py}


__________________________________________________________________


This module provides support for maintaining a list in sorted order without
having to sort the list after each insertion.  For long lists of items with
expensive comparison operations, this can be an improvement over the more common
approach.  The module is called @ref{10,,bisect} because it uses a basic bisection
algorithm to do its work.  The source code may be most useful as a working
example of the algorithm (the boundary conditions are already right!).

The following functions are provided:

@geindex bisect_left() (in module bisect)
@anchor{library/bisect bisect bisect_left}@anchor{1dfb}
@deffn {Function} bisect.bisect_left (a, x, lo=0, hi=len(a), *, key=None)

Locate the insertion point for @emph{x} in @emph{a} to maintain sorted order.
The parameters @emph{lo} and @emph{hi} may be used to specify a subset of the list
which should be considered; by default the entire list is used.  If @emph{x} is
already present in @emph{a}, the insertion point will be before (to the left of)
any existing entries.  The return value is suitable for use as the first
parameter to @code{list.insert()} assuming that @emph{a} is already sorted.

The returned insertion point @emph{i} partitions the array @emph{a} into two halves so
that @code{all(val < x for val in a[lo : i])} for the left side and
@code{all(val >= x for val in a[i : hi])} for the right side.

@emph{key} specifies a @ref{964,,key function} of one argument that is used to
extract a comparison key from each input element.  The default value is
@code{None} (compare the elements directly).


Changed in version 3.10: Added the @emph{key} parameter.

@end deffn

@geindex bisect_right() (in module bisect)
@geindex bisect() (in module bisect)
@anchor{library/bisect bisect bisect_right}@anchor{1dfc}
@deffn {Function} bisect.bisect_right (a, x, lo=0, hi=len(a), *, key=None)
@anchor{library/bisect bisect bisect}@anchor{1dfd}
@deffnx {Function} bisect.bisect (a, x, lo=0, hi=len(a), *, key=None)

Similar to @ref{1dfb,,bisect_left()}, but returns an insertion point which comes
after (to the right of) any existing entries of @emph{x} in @emph{a}.

The returned insertion point @emph{i} partitions the array @emph{a} into two halves so
that @code{all(val <= x for val in a[lo : i])} for the left side and
@code{all(val > x for val in a[i : hi])} for the right side.

@emph{key} specifies a @ref{964,,key function} of one argument that is used to
extract a comparison key from each input element.  The default value is
@code{None} (compare the elements directly).


Changed in version 3.10: Added the @emph{key} parameter.

@end deffn

@geindex insort_left() (in module bisect)
@anchor{library/bisect bisect insort_left}@anchor{1dfe}
@deffn {Function} bisect.insort_left (a, x, lo=0, hi=len(a), *, key=None)

Insert @emph{x} in @emph{a} in sorted order.

@emph{key} specifies a @ref{964,,key function} of one argument that is used to
extract a comparison key from each input element.  The default value is
@code{None} (compare the elements directly).

This function first runs @ref{1dfb,,bisect_left()} to locate an insertion point.
Next, it runs the @code{insert()} method on @emph{a} to insert @emph{x} at the
appropriate position to maintain sort order.

Keep in mind that the @code{O(log n)} search is dominated by the slow O(n)
insertion step.


Changed in version 3.10: Added the @emph{key} parameter.

@end deffn

@geindex insort_right() (in module bisect)
@geindex insort() (in module bisect)
@anchor{library/bisect bisect insort_right}@anchor{1dff}
@deffn {Function} bisect.insort_right (a, x, lo=0, hi=len(a), *, key=None)
@anchor{library/bisect bisect insort}@anchor{1e00}
@deffnx {Function} bisect.insort (a, x, lo=0, hi=len(a), *, key=None)

Similar to @ref{1dfe,,insort_left()}, but inserting @emph{x} in @emph{a} after any existing
entries of @emph{x}.

@emph{key} specifies a @ref{964,,key function} of one argument that is used to
extract a comparison key from each input element.  The default value is
@code{None} (compare the elements directly).

This function first runs @ref{1dfc,,bisect_right()} to locate an insertion point.
Next, it runs the @code{insert()} method on @emph{a} to insert @emph{x} at the
appropriate position to maintain sort order.

Keep in mind that the @code{O(log n)} search is dominated by the slow O(n)
insertion step.


Changed in version 3.10: Added the @emph{key} parameter.

@end deffn

@menu
* Performance Notes:: 
* Searching Sorted Lists:: 
* Examples: Examples<3>. 

@end menu

@node Performance Notes,Searching Sorted Lists,,bisect — Array bisection algorithm
@anchor{library/bisect performance-notes}@anchor{1e01}
@subsubsection Performance Notes


When writing time sensitive code using @emph{bisect()} and @emph{insort()}, keep these
thoughts in mind:


@itemize *

@item 
Bisection is effective for searching ranges of values.
For locating specific values, dictionaries are more performant.

@item 
The @emph{insort()} functions are @code{O(n)} because the logarithmic search step
is dominated by the linear time insertion step.

@item 
The search functions are stateless and discard key function results after
they are used.  Consequently, if the search functions are used in a loop,
the key function may be called again and again on the same array elements.
If the key function isn’t fast, consider wrapping it with
@ref{1e02,,functools.cache()} to avoid duplicate computations.  Alternatively,
consider searching an array of precomputed keys to locate the insertion
point (as shown in the examples section below).
@end itemize


@subsubheading See also



@itemize *

@item 
Sorted Collections@footnote{http://www.grantjenks.com/docs/sortedcollections/} is a high performance
module that uses @emph{bisect} to managed sorted collections of data.

@item 
The SortedCollection recipe@footnote{https://code.activestate.com/recipes/577197-sortedcollection/} uses
bisect to build a full-featured collection class with straight-forward search
methods and support for a key-function.  The keys are precomputed to save
unnecessary calls to the key function during searches.
@end itemize


@node Searching Sorted Lists,Examples<3>,Performance Notes,bisect — Array bisection algorithm
@anchor{library/bisect searching-sorted-lists}@anchor{1e03}
@subsubsection Searching Sorted Lists


The above @ref{10,,bisect()} functions are useful for finding insertion points but
can be tricky or awkward to use for common searching tasks. The following five
functions show how to transform them into the standard lookups for sorted
lists:

@example
def index(a, x):
    'Locate the leftmost value exactly equal to x'
    i = bisect_left(a, x)
    if i != len(a) and a[i] == x:
        return i
    raise ValueError

def find_lt(a, x):
    'Find rightmost value less than x'
    i = bisect_left(a, x)
    if i:
        return a[i-1]
    raise ValueError

def find_le(a, x):
    'Find rightmost value less than or equal to x'
    i = bisect_right(a, x)
    if i:
        return a[i-1]
    raise ValueError

def find_gt(a, x):
    'Find leftmost value greater than x'
    i = bisect_right(a, x)
    if i != len(a):
        return a[i]
    raise ValueError

def find_ge(a, x):
    'Find leftmost item greater than or equal to x'
    i = bisect_left(a, x)
    if i != len(a):
        return a[i]
    raise ValueError
@end example

@node Examples<3>,,Searching Sorted Lists,bisect — Array bisection algorithm
@anchor{library/bisect examples}@anchor{1e04}
@subsubsection Examples

@anchor{library/bisect bisect-example}@anchor{1e05}
The @ref{10,,bisect()} function can be useful for numeric table lookups. This
example uses @ref{10,,bisect()} to look up a letter grade for an exam score (say)
based on a set of ordered numeric breakpoints: 90 and up is an ‘A’, 80 to 89 is
a ‘B’, and so on:

@example
>>> def grade(score, breakpoints=[60, 70, 80, 90], grades='FDCBA'):
...     i = bisect(breakpoints, score)
...     return grades[i]
...
>>> [grade(score) for score in [33, 99, 77, 70, 89, 90, 100]]
['F', 'A', 'C', 'C', 'B', 'A', 'A']
@end example

One technique to avoid repeated calls to a key function is to search a list of
precomputed keys to find the index of a record:

@example
>>> data = [('red', 5), ('blue', 1), ('yellow', 8), ('black', 0)]
>>> data.sort(key=lambda r: r[1])       # Or use operator.itemgetter(1).
>>> keys = [r[1] for r in data]         # Precompute a list of keys.
>>> data[bisect_left(keys, 0)]
('black', 0)
>>> data[bisect_left(keys, 1)]
('blue', 1)
>>> data[bisect_left(keys, 5)]
('red', 5)
>>> data[bisect_left(keys, 8)]
('yellow', 8)
@end example

@node array — Efficient arrays of numeric values,weakref — Weak references,bisect — Array bisection algorithm,Data Types
@anchor{library/array doc}@anchor{1e06}@anchor{library/array array-efficient-arrays-of-numeric-values}@anchor{1e07}@anchor{library/array module-array}@anchor{6}
@subsection @code{array} — Efficient arrays of numeric values


@geindex module; array

@geindex arrays


__________________________________________________________________


This module defines an object type which can compactly represent an array of
basic values: characters, integers, floating point numbers.  Arrays are sequence
types and behave very much like lists, except that the type of objects stored in
them is constrained.  The type is specified at object creation time by using a
@emph{type code}, which is a single character.  The following type codes are
defined:


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} 
@headitem

Type code

@tab

C Type

@tab

Python Type

@tab

Minimum size in bytes

@tab

Notes

@item

@code{'b'}

@tab

signed char

@tab

int

@tab

1

@tab

@item

@code{'B'}

@tab

unsigned char

@tab

int

@tab

1

@tab

@item

@code{'u'}

@tab

wchar_t

@tab

Unicode character

@tab

2

@tab

(1)

@item

@code{'h'}

@tab

signed short

@tab

int

@tab

2

@tab

@item

@code{'H'}

@tab

unsigned short

@tab

int

@tab

2

@tab

@item

@code{'i'}

@tab

signed int

@tab

int

@tab

2

@tab

@item

@code{'I'}

@tab

unsigned int

@tab

int

@tab

2

@tab

@item

@code{'l'}

@tab

signed long

@tab

int

@tab

4

@tab

@item

@code{'L'}

@tab

unsigned long

@tab

int

@tab

4

@tab

@item

@code{'q'}

@tab

signed long long

@tab

int

@tab

8

@tab

@item

@code{'Q'}

@tab

unsigned long long

@tab

int

@tab

8

@tab

@item

@code{'f'}

@tab

float

@tab

float

@tab

4

@tab

@item

@code{'d'}

@tab

double

@tab

float

@tab

8

@tab

@end multitable


Notes:


@enumerate 

@item 
It can be 16 bits or 32 bits depending on the platform.


Changed in version 3.9: @code{array('u')} now uses @code{wchar_t} as C type instead of deprecated
@code{Py_UNICODE}. This change doesn’t affect to its behavior because
@code{Py_UNICODE} is alias of @code{wchar_t} since Python 3.3.



Deprecated since version 3.3, will be removed in version 4.0.

@end enumerate

The actual representation of values is determined by the machine architecture
(strictly speaking, by the C implementation).  The actual size can be accessed
through the @code{itemsize} attribute.

The module defines the following type:

@geindex array (class in array)
@anchor{library/array array array}@anchor{25e}
@deffn {Class} array.array (typecode[, initializer])

A new array whose items are restricted by @emph{typecode}, and initialized
from the optional @emph{initializer} value, which must be a list, a
@ref{87e,,bytes-like object}, or iterable over elements of the
appropriate type.

If given a list or string, the initializer is passed to the new array’s
@ref{1e08,,fromlist()}, @ref{1e09,,frombytes()}, or @ref{1e0a,,fromunicode()} method (see below)
to add initial items to the array.  Otherwise, the iterable initializer is
passed to the @ref{1e0b,,extend()} method.

Raises an @ref{1130,,auditing event} @code{array.__new__} with arguments @code{typecode}, @code{initializer}.
@end deffn

@geindex typecodes (in module array)
@anchor{library/array array typecodes}@anchor{1e0c}
@deffn {Data} array.typecodes

A string with all available type codes.
@end deffn

Array objects support the ordinary sequence operations of indexing, slicing,
concatenation, and multiplication.  When using slice assignment, the assigned
value must be an array object with the same type code; in all other cases,
@ref{162,,TypeError} is raised. Array objects also implement the buffer interface,
and may be used wherever @ref{87e,,bytes-like objects} are supported.

The following data items and methods are also supported:

@geindex typecode (array.array attribute)
@anchor{library/array array array typecode}@anchor{1e0d}
@deffn {Attribute} array.typecode

The typecode character used to create the array.
@end deffn

@geindex itemsize (array.array attribute)
@anchor{library/array array array itemsize}@anchor{1e0e}
@deffn {Attribute} array.itemsize

The length in bytes of one array item in the internal representation.
@end deffn

@geindex append() (array.array method)
@anchor{library/array array array append}@anchor{1e0f}
@deffn {Method} array.append (x)

Append a new item with value @emph{x} to the end of the array.
@end deffn

@geindex buffer_info() (array.array method)
@anchor{library/array array array buffer_info}@anchor{1e10}
@deffn {Method} array.buffer_info ()

Return a tuple @code{(address, length)} giving the current memory address and the
length in elements of the buffer used to hold array’s contents.  The size of the
memory buffer in bytes can be computed as @code{array.buffer_info()[1] *
array.itemsize}.  This is occasionally useful when working with low-level (and
inherently unsafe) I/O interfaces that require memory addresses, such as certain
@code{ioctl()} operations.  The returned numbers are valid as long as the array
exists and no length-changing operations are applied to it.

@cartouche
@quotation Note 
When using array objects from code written in C or C++ (the only way to
effectively make use of this information), it makes more sense to use the buffer
interface supported by array objects.  This method is maintained for backward
compatibility and should be avoided in new code.  The buffer interface is
documented in @ref{1a3e,,Buffer Protocol}.
@end quotation
@end cartouche
@end deffn

@geindex byteswap() (array.array method)
@anchor{library/array array array byteswap}@anchor{1e11}
@deffn {Method} array.byteswap ()

“Byteswap” all items of the array.  This is only supported for values which are
1, 2, 4, or 8 bytes in size; for other types of values, @ref{5af,,RuntimeError} is
raised.  It is useful when reading data from a file written on a machine with a
different byte order.
@end deffn

@geindex count() (array.array method)
@anchor{library/array array array count}@anchor{1e12}
@deffn {Method} array.count (x)

Return the number of occurrences of @emph{x} in the array.
@end deffn

@geindex extend() (array.array method)
@anchor{library/array array array extend}@anchor{1e0b}
@deffn {Method} array.extend (iterable)

Append items from @emph{iterable} to the end of the array.  If @emph{iterable} is another
array, it must have @emph{exactly} the same type code; if not, @ref{162,,TypeError} will
be raised.  If @emph{iterable} is not an array, it must be iterable and its elements
must be the right type to be appended to the array.
@end deffn

@geindex frombytes() (array.array method)
@anchor{library/array array array frombytes}@anchor{1e09}
@deffn {Method} array.frombytes (s)

Appends items from the string, interpreting the string as an array of machine
values (as if it had been read from a file using the @ref{1e13,,fromfile()} method).


New in version 3.2: @code{fromstring()} is renamed to @ref{1e09,,frombytes()} for clarity.

@end deffn

@geindex fromfile() (array.array method)
@anchor{library/array array array fromfile}@anchor{1e13}
@deffn {Method} array.fromfile (f, n)

Read @emph{n} items (as machine values) from the @ref{d68,,file object} @emph{f} and append
them to the end of the array.  If less than @emph{n} items are available,
@ref{e74,,EOFError} is raised, but the items that were available are still
inserted into the array.
@end deffn

@geindex fromlist() (array.array method)
@anchor{library/array array array fromlist}@anchor{1e08}
@deffn {Method} array.fromlist (list)

Append items from the list.  This is equivalent to @code{for x in list:
a.append(x)} except that if there is a type error, the array is unchanged.
@end deffn

@geindex fromunicode() (array.array method)
@anchor{library/array array array fromunicode}@anchor{1e0a}
@deffn {Method} array.fromunicode (s)

Extends this array with data from the given unicode string.  The array must
be a type @code{'u'} array; otherwise a @ref{17c,,ValueError} is raised.  Use
@code{array.frombytes(unicodestring.encode(enc))} to append Unicode data to an
array of some other type.
@end deffn

@geindex index() (array.array method)
@anchor{library/array array array index}@anchor{25d}
@deffn {Method} array.index (x[, start[, stop]])

Return the smallest @emph{i} such that @emph{i} is the index of the first occurrence of
@emph{x} in the array.  The optional arguments @emph{start} and @emph{stop} can be
specified to search for @emph{x} within a subsection of the array.  Raise
@ref{17c,,ValueError} if @emph{x} is not found.


Changed in version 3.10: Added optional @emph{start} and @emph{stop} parameters.

@end deffn

@geindex insert() (array.array method)
@anchor{library/array array array insert}@anchor{1e14}
@deffn {Method} array.insert (i, x)

Insert a new item with value @emph{x} in the array before position @emph{i}. Negative
values are treated as being relative to the end of the array.
@end deffn

@geindex pop() (array.array method)
@anchor{library/array array array pop}@anchor{1e15}
@deffn {Method} array.pop ([i])

Removes the item with the index @emph{i} from the array and returns it. The optional
argument defaults to @code{-1}, so that by default the last item is removed and
returned.
@end deffn

@geindex remove() (array.array method)
@anchor{library/array array array remove}@anchor{1e16}
@deffn {Method} array.remove (x)

Remove the first occurrence of @emph{x} from the array.
@end deffn

@geindex reverse() (array.array method)
@anchor{library/array array array reverse}@anchor{1e17}
@deffn {Method} array.reverse ()

Reverse the order of the items in the array.
@end deffn

@geindex tobytes() (array.array method)
@anchor{library/array array array tobytes}@anchor{1e18}
@deffn {Method} array.tobytes ()

Convert the array to an array of machine values and return the bytes
representation (the same sequence of bytes that would be written to a file by
the @ref{1e19,,tofile()} method.)


New in version 3.2: @code{tostring()} is renamed to @ref{1e18,,tobytes()} for clarity.

@end deffn

@geindex tofile() (array.array method)
@anchor{library/array array array tofile}@anchor{1e19}
@deffn {Method} array.tofile (f)

Write all items (as machine values) to the @ref{d68,,file object} @emph{f}.
@end deffn

@geindex tolist() (array.array method)
@anchor{library/array array array tolist}@anchor{1e1a}
@deffn {Method} array.tolist ()

Convert the array to an ordinary list with the same items.
@end deffn

@geindex tounicode() (array.array method)
@anchor{library/array array array tounicode}@anchor{1e1b}
@deffn {Method} array.tounicode ()

Convert the array to a unicode string.  The array must be a type @code{'u'} array;
otherwise a @ref{17c,,ValueError} is raised. Use @code{array.tobytes().decode(enc)} to
obtain a unicode string from an array of some other type.
@end deffn

When an array object is printed or converted to a string, it is represented as
@code{array(typecode, initializer)}.  The @emph{initializer} is omitted if the array is
empty, otherwise it is a string if the @emph{typecode} is @code{'u'}, otherwise it is a
list of numbers.  The string is guaranteed to be able to be converted back to an
array with the same type and value using @ref{34a,,eval()}, so long as the
@ref{25e,,array} class has been imported using @code{from array import array}.
Examples:

@example
array('l')
array('u', 'hello \u2641')
array('l', [1, 2, 3, 4, 5])
array('d', [1.0, 2.0, 3.14])
@end example


@subsubheading See also



@table @asis

@item Module @ref{f3,,struct}

Packing and unpacking of heterogeneous binary data.

@item Module @ref{139,,xdrlib}

Packing and unpacking of External Data Representation (XDR) data as used in some
remote procedure call systems.

@item NumPy@footnote{https://numpy.org/}

The NumPy package defines another array type.
@end table


@node weakref — Weak references,types — Dynamic type creation and names for built-in types,array — Efficient arrays of numeric values,Data Types
@anchor{library/weakref doc}@anchor{1e1c}@anchor{library/weakref module-weakref}@anchor{12f}@anchor{library/weakref weakref-weak-references}@anchor{1e1d}
@subsection @code{weakref} — Weak references


@geindex module; weakref

@strong{Source code:} Lib/weakref.py@footnote{https://github.com/python/cpython/tree/main/Lib/weakref.py}


__________________________________________________________________


The @ref{12f,,weakref} module allows the Python programmer to create @emph{weak
references} to objects.

@c When making changes to the examples in this file, be sure to update
@c Lib/test/test_weakref.py::libreftest too!

In the following, the term @emph{referent} means the object which is referred to
by a weak reference.

A weak reference to an object is not enough to keep the object alive: when the
only remaining references to a referent are weak references,
@ref{1192,,garbage collection} is free to destroy the referent and reuse its memory
for something else.  However, until the object is actually destroyed the weak
reference may return the object even if there are no strong references to it.

A primary use for weak references is to implement caches or
mappings holding large objects, where it’s desired that a large object not be
kept alive solely because it appears in a cache or mapping.

For example, if you have a number of large binary image objects, you may wish to
associate a name with each.  If you used a Python dictionary to map names to
images, or images to names, the image objects would remain alive just because
they appeared as values or keys in the dictionaries.  The
@ref{11fa,,WeakKeyDictionary} and @ref{11f9,,WeakValueDictionary} classes supplied by
the @ref{12f,,weakref} module are an alternative, using weak references to construct
mappings that don’t keep objects alive solely because they appear in the mapping
objects.  If, for example, an image object is a value in a
@ref{11f9,,WeakValueDictionary}, then when the last remaining references to that
image object are the weak references held by weak mappings, garbage collection
can reclaim the object, and its corresponding entries in weak mappings are
simply deleted.

@ref{11fa,,WeakKeyDictionary} and @ref{11f9,,WeakValueDictionary} use weak references
in their implementation, setting up callback functions on the weak references
that notify the weak dictionaries when a key or value has been reclaimed by
garbage collection.  @ref{eba,,WeakSet} implements the @ref{370,,set} interface,
but keeps weak references to its elements, just like a
@ref{11fa,,WeakKeyDictionary} does.

@ref{594,,finalize} provides a straight forward way to register a
cleanup function to be called when an object is garbage collected.
This is simpler to use than setting up a callback function on a raw
weak reference, since the module automatically ensures that the finalizer
remains alive until the object is collected.

Most programs should find that using one of these weak container types
or @ref{594,,finalize} is all they need – it’s not usually necessary to
create your own weak references directly.  The low-level machinery is
exposed by the @ref{12f,,weakref} module for the benefit of advanced uses.

Not all objects can be weakly referenced. Objects which support weak references
include class instances, functions written in Python (but not in C), instance methods,
sets, frozensets, some @ref{d68,,file objects}, @ref{bf7,,generators},
type objects, sockets, arrays, deques, regular expression pattern objects, and code
objects.


Changed in version 3.2: Added support for thread.lock, threading.Lock, and code objects.


Several built-in types such as @ref{43a,,list} and @ref{3b0,,dict} do not directly
support weak references but can add support through subclassing:

@example
class Dict(dict):
    pass

obj = Dict(red=1, green=2, blue=3)   # this object is weak referenceable
@end example

@strong{CPython implementation detail:} Other built-in types such as @ref{3e5,,tuple} and @ref{1b1,,int} do not support weak
references even when subclassed.

Extension types can easily be made to support weak references; see
@ref{1e1e,,Weak Reference Support}.

When @code{__slots__} are defined for a given type, weak reference support is
disabled unless a @code{'__weakref__'} string is also present in the sequence of
strings in the @code{__slots__} declaration.
See @ref{17ba,,__slots__ documentation} for details.

@geindex ref (class in weakref)
@anchor{library/weakref weakref ref}@anchor{b76}
@deffn {Class} weakref.ref (object[, callback])

Return a weak reference to @emph{object}.  The original object can be retrieved by
calling the reference object if the referent is still alive; if the referent is
no longer alive, calling the reference object will cause @ref{182,,None} to be
returned.  If @emph{callback} is provided and not @ref{182,,None}, and the returned
weakref object is still alive, the callback will be called when the object is
about to be finalized; the weak reference object will be passed as the only
parameter to the callback; the referent will no longer be available.

It is allowable for many weak references to be constructed for the same object.
Callbacks registered for each weak reference will be called from the most
recently registered callback to the oldest registered callback.

Exceptions raised by the callback will be noted on the standard error output,
but cannot be propagated; they are handled in exactly the same way as exceptions
raised from an object’s @code{__del__()} method.

Weak references are @ref{2dc,,hashable} if the @emph{object} is hashable.  They will
maintain their hash value even after the @emph{object} was deleted.  If
@ref{c14,,hash()} is called the first time only after the @emph{object} was deleted,
the call will raise @ref{162,,TypeError}.

Weak references support tests for equality, but not ordering.  If the referents
are still alive, two references have the same equality relationship as their
referents (regardless of the @emph{callback}).  If either referent has been deleted,
the references are equal only if the reference objects are the same object.

This is a subclassable type rather than a factory function.

@geindex __callback__ (weakref.ref attribute)
@anchor{library/weakref weakref ref __callback__}@anchor{b77}
@deffn {Attribute} __callback__

This read-only attribute returns the callback currently associated to the
weakref.  If there is no callback or if the referent of the weakref is
no longer alive then this attribute will have value @code{None}.
@end deffn


Changed in version 3.4: Added the @ref{b77,,__callback__} attribute.

@end deffn

@geindex proxy() (in module weakref)
@anchor{library/weakref weakref proxy}@anchor{567}
@deffn {Function} weakref.proxy (object[, callback])

Return a proxy to @emph{object} which uses a weak reference.  This supports use of
the proxy in most contexts instead of requiring the explicit dereferencing used
with weak reference objects.  The returned object will have a type of either
@code{ProxyType} or @code{CallableProxyType}, depending on whether @emph{object} is
callable.  Proxy objects are not @ref{2dc,,hashable} regardless of the referent; this
avoids a number of problems related to their fundamentally mutable nature, and
prevent their use as dictionary keys.  @emph{callback} is the same as the parameter
of the same name to the @ref{b76,,ref()} function.


Changed in version 3.8: Extended the operator support on proxy objects to include the matrix
multiplication operators @code{@@} and @code{@@=}.

@end deffn

@geindex getweakrefcount() (in module weakref)
@anchor{library/weakref weakref getweakrefcount}@anchor{1e1f}
@deffn {Function} weakref.getweakrefcount (object)

Return the number of weak references and proxies which refer to @emph{object}.
@end deffn

@geindex getweakrefs() (in module weakref)
@anchor{library/weakref weakref getweakrefs}@anchor{1e20}
@deffn {Function} weakref.getweakrefs (object)

Return a list of all weak reference and proxy objects which refer to @emph{object}.
@end deffn

@geindex WeakKeyDictionary (class in weakref)
@anchor{library/weakref weakref WeakKeyDictionary}@anchor{11fa}
@deffn {Class} weakref.WeakKeyDictionary ([dict])

Mapping class that references keys weakly.  Entries in the dictionary will be
discarded when there is no longer a strong reference to the key.  This can be
used to associate additional data with an object owned by other parts of an
application without adding attributes to those objects.  This can be especially
useful with objects that override attribute accesses.


Changed in version 3.9: Added support for @code{|} and @code{|=} operators, specified in 
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584}.

@end deffn

@ref{11fa,,WeakKeyDictionary} objects have an additional method that
exposes the internal references directly.  The references are not guaranteed to
be “live” at the time they are used, so the result of calling the references
needs to be checked before being used.  This can be used to avoid creating
references that will cause the garbage collector to keep the keys around longer
than needed.

@geindex keyrefs() (weakref.WeakKeyDictionary method)
@anchor{library/weakref weakref WeakKeyDictionary keyrefs}@anchor{1e21}
@deffn {Method} WeakKeyDictionary.keyrefs ()

Return an iterable of the weak references to the keys.
@end deffn

@geindex WeakValueDictionary (class in weakref)
@anchor{library/weakref weakref WeakValueDictionary}@anchor{11f9}
@deffn {Class} weakref.WeakValueDictionary ([dict])

Mapping class that references values weakly.  Entries in the dictionary will be
discarded when no strong reference to the value exists any more.


Changed in version 3.9: Added support for @code{|} and @code{|=} operators, as specified in 
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584}.

@end deffn

@ref{11f9,,WeakValueDictionary} objects have an additional method that has the
same issues as the @code{keyrefs()} method of @ref{11fa,,WeakKeyDictionary}
objects.

@geindex valuerefs() (weakref.WeakValueDictionary method)
@anchor{library/weakref weakref WeakValueDictionary valuerefs}@anchor{1e22}
@deffn {Method} WeakValueDictionary.valuerefs ()

Return an iterable of the weak references to the values.
@end deffn

@geindex WeakSet (class in weakref)
@anchor{library/weakref weakref WeakSet}@anchor{eba}
@deffn {Class} weakref.WeakSet ([elements])

Set class that keeps weak references to its elements.  An element will be
discarded when no strong reference to it exists any more.
@end deffn

@geindex WeakMethod (class in weakref)
@anchor{library/weakref weakref WeakMethod}@anchor{b75}
@deffn {Class} weakref.WeakMethod (method)

A custom @ref{b76,,ref} subclass which simulates a weak reference to a bound
method (i.e., a method defined on a class and looked up on an instance).
Since a bound method is ephemeral, a standard weak reference cannot keep
hold of it.  @ref{b75,,WeakMethod} has special code to recreate the bound
method until either the object or the original function dies:

@example
>>> class C:
...     def method(self):
...         print("method called!")
...
>>> c = C()
>>> r = weakref.ref(c.method)
>>> r()
>>> r = weakref.WeakMethod(c.method)
>>> r()
<bound method C.method of <__main__.C object at 0x7fc859830220>>
>>> r()()
method called!
>>> del c
>>> gc.collect()
0
>>> r()
>>>
@end example


New in version 3.4.

@end deffn

@geindex finalize (class in weakref)
@anchor{library/weakref weakref finalize}@anchor{594}
@deffn {Class} weakref.finalize (obj, func, /, *args, **kwargs)

Return a callable finalizer object which will be called when @emph{obj}
is garbage collected. Unlike an ordinary weak reference, a finalizer
will always survive until the reference object is collected, greatly
simplifying lifecycle management.

A finalizer is considered @emph{alive} until it is called (either explicitly
or at garbage collection), and after that it is @emph{dead}.  Calling a live
finalizer returns the result of evaluating @code{func(*arg, **kwargs)},
whereas calling a dead finalizer returns @ref{182,,None}.

Exceptions raised by finalizer callbacks during garbage collection
will be shown on the standard error output, but cannot be
propagated.  They are handled in the same way as exceptions raised
from an object’s @code{__del__()} method or a weak reference’s
callback.

When the program exits, each remaining live finalizer is called
unless its @ref{b,,atexit} attribute has been set to false.  They
are called in reverse order of creation.

A finalizer will never invoke its callback during the later part of
the @ref{9e2,,interpreter shutdown} when module globals are liable to have
been replaced by @ref{182,,None}.

@geindex __call__() (weakref.finalize method)
@anchor{library/weakref weakref finalize __call__}@anchor{1e23}
@deffn {Method} __call__ ()

If @emph{self} is alive then mark it as dead and return the result of
calling @code{func(*args, **kwargs)}.  If @emph{self} is dead then return
@ref{182,,None}.
@end deffn

@geindex detach() (weakref.finalize method)
@anchor{library/weakref weakref finalize detach}@anchor{1e24}
@deffn {Method} detach ()

If @emph{self} is alive then mark it as dead and return the tuple
@code{(obj, func, args, kwargs)}.  If @emph{self} is dead then return
@ref{182,,None}.
@end deffn

@geindex peek() (weakref.finalize method)
@anchor{library/weakref weakref finalize peek}@anchor{1e25}
@deffn {Method} peek ()

If @emph{self} is alive then return the tuple @code{(obj, func, args,
kwargs)}.  If @emph{self} is dead then return @ref{182,,None}.
@end deffn

@geindex alive (weakref.finalize attribute)
@anchor{library/weakref weakref finalize alive}@anchor{1e26}
@deffn {Attribute} alive

Property which is true if the finalizer is alive, false otherwise.
@end deffn

@geindex atexit (weakref.finalize attribute)
@anchor{library/weakref weakref finalize atexit}@anchor{1e27}
@deffn {Attribute} atexit

A writable boolean property which by default is true.  When the
program exits, it calls all remaining live finalizers for which
@ref{1e27,,atexit} is true.  They are called in reverse order of
creation.
@end deffn

@cartouche
@quotation Note 
It is important to ensure that @emph{func}, @emph{args} and @emph{kwargs} do
not own any references to @emph{obj}, either directly or indirectly,
since otherwise @emph{obj} will never be garbage collected.  In
particular, @emph{func} should not be a bound method of @emph{obj}.
@end quotation
@end cartouche


New in version 3.4.

@end deffn

@geindex ReferenceType (in module weakref)
@anchor{library/weakref weakref ReferenceType}@anchor{1e28}
@deffn {Data} weakref.ReferenceType

The type object for weak references objects.
@end deffn

@geindex ProxyType (in module weakref)
@anchor{library/weakref weakref ProxyType}@anchor{1e29}
@deffn {Data} weakref.ProxyType

The type object for proxies of objects which are not callable.
@end deffn

@geindex CallableProxyType (in module weakref)
@anchor{library/weakref weakref CallableProxyType}@anchor{1e2a}
@deffn {Data} weakref.CallableProxyType

The type object for proxies of callable objects.
@end deffn

@geindex ProxyTypes (in module weakref)
@anchor{library/weakref weakref ProxyTypes}@anchor{1e2b}
@deffn {Data} weakref.ProxyTypes

Sequence containing all the type objects for proxies.  This can make it simpler
to test if an object is a proxy without being dependent on naming both proxy
types.
@end deffn


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 205

@item PEP 205@footnote{https://www.python.org/dev/peps/pep-0205} - Weak References

The proposal and rationale for this feature, including links to earlier
implementations and information about similar features in other languages.
@end table


@menu
* Weak Reference Objects:: 
* Example: Example<2>. 
* Finalizer Objects:: 
* Comparing finalizers with __del__() methods: Comparing finalizers with __del__ methods. 

@end menu

@node Weak Reference Objects,Example<2>,,weakref — Weak references
@anchor{library/weakref weak-reference-objects}@anchor{1e2c}@anchor{library/weakref weakref-objects}@anchor{1e2d}
@subsubsection Weak Reference Objects


Weak reference objects have no methods and no attributes besides
@ref{b77,,ref.__callback__}. A weak reference object allows the referent to be
obtained, if it still exists, by calling it:

@example
>>> import weakref
>>> class Object:
...     pass
...
>>> o = Object()
>>> r = weakref.ref(o)
>>> o2 = r()
>>> o is o2
True
@end example

If the referent no longer exists, calling the reference object returns
@ref{182,,None}:

@example
>>> del o, o2
>>> print(r())
None
@end example

Testing that a weak reference object is still live should be done using the
expression @code{ref() is not None}.  Normally, application code that needs to use
a reference object should follow this pattern:

@example
# r is a weak reference object
o = r()
if o is None:
    # referent has been garbage collected
    print("Object has been deallocated; can't frobnicate.")
else:
    print("Object is still live!")
    o.do_something_useful()
@end example

Using a separate test for “liveness” creates race conditions in threaded
applications; another thread can cause a weak reference to become invalidated
before the weak reference is called; the idiom shown above is safe in threaded
applications as well as single-threaded applications.

Specialized versions of @ref{b76,,ref} objects can be created through subclassing.
This is used in the implementation of the @ref{11f9,,WeakValueDictionary} to reduce
the memory overhead for each entry in the mapping.  This may be most useful to
associate additional information with a reference, but could also be used to
insert additional processing on calls to retrieve the referent.

This example shows how a subclass of @ref{b76,,ref} can be used to store
additional information about an object and affect the value that’s returned when
the referent is accessed:

@example
import weakref

class ExtendedRef(weakref.ref):
    def __init__(self, ob, callback=None, /, **annotations):
        super().__init__(ob, callback)
        self.__counter = 0
        for k, v in annotations.items():
            setattr(self, k, v)

    def __call__(self):
        """Return a pair containing the referent and the number of
        times the reference has been called.
        """
        ob = super().__call__()
        if ob is not None:
            self.__counter += 1
            ob = (ob, self.__counter)
        return ob
@end example

@node Example<2>,Finalizer Objects,Weak Reference Objects,weakref — Weak references
@anchor{library/weakref example}@anchor{1e2e}@anchor{library/weakref weakref-example}@anchor{1e2f}
@subsubsection Example


This simple example shows how an application can use object IDs to retrieve
objects that it has seen before.  The IDs of the objects can then be used in
other data structures without forcing the objects to remain alive, but the
objects can still be retrieved by ID if they do.

@c Example contributed by Tim Peters.

@example
import weakref

_id2obj_dict = weakref.WeakValueDictionary()

def remember(obj):
    oid = id(obj)
    _id2obj_dict[oid] = obj
    return oid

def id2obj(oid):
    return _id2obj_dict[oid]
@end example

@node Finalizer Objects,Comparing finalizers with __del__ methods,Example<2>,weakref — Weak references
@anchor{library/weakref finalize-examples}@anchor{1e30}@anchor{library/weakref finalizer-objects}@anchor{1e31}
@subsubsection Finalizer Objects


The main benefit of using @ref{594,,finalize} is that it makes it simple
to register a callback without needing to preserve the returned finalizer
object.  For instance

@example
>>> import weakref
>>> class Object:
...     pass
...
>>> kenny = Object()
>>> weakref.finalize(kenny, print, "You killed Kenny!")  
<finalize object at ...; for 'Object' at ...>
>>> del kenny
You killed Kenny!
@end example

The finalizer can be called directly as well.  However the finalizer
will invoke the callback at most once.

@example
>>> def callback(x, y, z):
...     print("CALLBACK")
...     return x + y + z
...
>>> obj = Object()
>>> f = weakref.finalize(obj, callback, 1, 2, z=3)
>>> assert f.alive
>>> assert f() == 6
CALLBACK
>>> assert not f.alive
>>> f()                     # callback not called because finalizer dead
>>> del obj                 # callback not called because finalizer dead
@end example

You can unregister a finalizer using its @ref{1e24,,detach()}
method.  This kills the finalizer and returns the arguments passed to
the constructor when it was created.

@example
>>> obj = Object()
>>> f = weakref.finalize(obj, callback, 1, 2, z=3)
>>> f.detach()                                           
(<...Object object ...>, <function callback ...>, (1, 2), @{'z': 3@})
>>> newobj, func, args, kwargs = _
>>> assert not f.alive
>>> assert newobj is obj
>>> assert func(*args, **kwargs) == 6
CALLBACK
@end example

Unless you set the @ref{1e27,,atexit} attribute to
@ref{659,,False}, a finalizer will be called when the program exits if it
is still alive.  For instance

@example
>>> obj = Object()
>>> weakref.finalize(obj, print, "obj dead or exiting")
<finalize object at ...; for 'Object' at ...>
>>> exit()
obj dead or exiting
@end example

@node Comparing finalizers with __del__ methods,,Finalizer Objects,weakref — Weak references
@anchor{library/weakref comparing-finalizers-with-del-methods}@anchor{1e32}
@subsubsection Comparing finalizers with @code{__del__()} methods


Suppose we want to create a class whose instances represent temporary
directories.  The directories should be deleted with their contents
when the first of the following events occurs:


@itemize *

@item 
the object is garbage collected,

@item 
the object’s @code{remove()} method is called, or

@item 
the program exits.
@end itemize

We might try to implement the class using a @code{__del__()} method as
follows:

@example
class TempDir:
    def __init__(self):
        self.name = tempfile.mkdtemp()

    def remove(self):
        if self.name is not None:
            shutil.rmtree(self.name)
            self.name = None

    @@property
    def removed(self):
        return self.name is None

    def __del__(self):
        self.remove()
@end example

Starting with Python 3.4, @code{__del__()} methods no longer prevent
reference cycles from being garbage collected, and module globals are
no longer forced to @ref{182,,None} during @ref{9e2,,interpreter shutdown}.
So this code should work without any issues on CPython.

However, handling of @code{__del__()} methods is notoriously implementation
specific, since it depends on internal details of the interpreter’s garbage
collector implementation.

A more robust alternative can be to define a finalizer which only references
the specific functions and objects that it needs, rather than having access
to the full state of the object:

@example
class TempDir:
    def __init__(self):
        self.name = tempfile.mkdtemp()
        self._finalizer = weakref.finalize(self, shutil.rmtree, self.name)

    def remove(self):
        self._finalizer()

    @@property
    def removed(self):
        return not self._finalizer.alive
@end example

Defined like this, our finalizer only receives a reference to the details
it needs to clean up the directory appropriately. If the object never gets
garbage collected the finalizer will still be called at exit.

The other advantage of weakref based finalizers is that they can be used to
register finalizers for classes where the definition is controlled by a
third party, such as running code when a module is unloaded:

@example
import weakref, sys
def unloading_module():
    # implicit reference to the module globals from the function body
weakref.finalize(sys.modules[__name__], unloading_module)
@end example

@cartouche
@quotation Note 
If you create a finalizer object in a daemonic thread just as the program
exits then there is the possibility that the finalizer
does not get called at exit.  However, in a daemonic thread
@ref{343,,atexit.register()}, @code{try: ... finally: ...} and @code{with: ...}
do not guarantee that cleanup occurs either.
@end quotation
@end cartouche

@node types — Dynamic type creation and names for built-in types,copy — Shallow and deep copy operations,weakref — Weak references,Data Types
@anchor{library/types doc}@anchor{1e33}@anchor{library/types module-types}@anchor{11f}@anchor{library/types types-dynamic-type-creation-and-names-for-built-in-types}@anchor{1e34}
@subsection @code{types} — Dynamic type creation and names for built-in types


@geindex module; types

@strong{Source code:} Lib/types.py@footnote{https://github.com/python/cpython/tree/main/Lib/types.py}


__________________________________________________________________


This module defines utility functions to assist in dynamic creation of
new types.

It also defines names for some object types that are used by the standard
Python interpreter, but not exposed as builtins like @ref{1b1,,int} or
@ref{2ef,,str} are.

Finally, it provides some additional type-related utility classes and functions
that are not fundamental enough to be builtins.

@menu
* Dynamic Type Creation:: 
* Standard Interpreter Types:: 
* Additional Utility Classes and Functions:: 
* Coroutine Utility Functions:: 

@end menu

@node Dynamic Type Creation,Standard Interpreter Types,,types — Dynamic type creation and names for built-in types
@anchor{library/types dynamic-type-creation}@anchor{1e35}
@subsubsection Dynamic Type Creation


@geindex new_class() (in module types)
@anchor{library/types types new_class}@anchor{cfc}
@deffn {Function} types.new_class (name, bases=(), kwds=None, exec_body=None)

Creates a class object dynamically using the appropriate metaclass.

The first three arguments are the components that make up a class
definition header: the class name, the base classes (in order), the
keyword arguments (such as @code{metaclass}).

The @emph{exec_body} argument is a callback that is used to populate the
freshly created class namespace. It should accept the class namespace
as its sole argument and update the namespace directly with the class
contents. If no callback is provided, it has the same effect as passing
in @code{lambda ns: None}.


New in version 3.3.

@end deffn

@geindex prepare_class() (in module types)
@anchor{library/types types prepare_class}@anchor{cfd}
@deffn {Function} types.prepare_class (name, bases=(), kwds=None)

Calculates the appropriate metaclass and creates the class namespace.

The arguments are the components that make up a class definition header:
the class name, the base classes (in order) and the keyword arguments
(such as @code{metaclass}).

The return value is a 3-tuple: @code{metaclass, namespace, kwds}

@emph{metaclass} is the appropriate metaclass, @emph{namespace} is the
prepared class namespace and @emph{kwds} is an updated copy of the passed
in @emph{kwds} argument with any @code{'metaclass'} entry removed. If no @emph{kwds}
argument is passed in, this will be an empty dict.


New in version 3.3.



Changed in version 3.6: The default value for the @code{namespace} element of the returned
tuple has changed.  Now an insertion-order-preserving mapping is
used when the metaclass does not have a @code{__prepare__} method.

@end deffn


@subsubheading See also



@table @asis

@item @ref{17bf,,Metaclasses}

Full details of the class creation process supported by these functions

@geindex Python Enhancement Proposals; PEP 3115

@item PEP 3115@footnote{https://www.python.org/dev/peps/pep-3115} - Metaclasses in Python 3000

Introduced the @code{__prepare__} namespace hook
@end table


@geindex resolve_bases() (in module types)
@anchor{library/types types resolve_bases}@anchor{6cd}
@deffn {Function} types.resolve_bases (bases)

Resolve MRO entries dynamically as specified by 
@geindex Python Enhancement Proposals; PEP 560
PEP 560@footnote{https://www.python.org/dev/peps/pep-0560}.

This function looks for items in @emph{bases} that are not instances of
@ref{89e,,type}, and returns a tuple where each such object that has
an @code{__mro_entries__} method is replaced with an unpacked result of
calling this method.  If a @emph{bases} item is an instance of @ref{89e,,type},
or it doesn’t have an @code{__mro_entries__} method, then it is included in
the return tuple unchanged.


New in version 3.7.

@end deffn


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 560
PEP 560@footnote{https://www.python.org/dev/peps/pep-0560} - Core support for typing module and generic types


@node Standard Interpreter Types,Additional Utility Classes and Functions,Dynamic Type Creation,types — Dynamic type creation and names for built-in types
@anchor{library/types standard-interpreter-types}@anchor{1e36}
@subsubsection Standard Interpreter Types


This module provides names for many of the types that are required to
implement a Python interpreter. It deliberately avoids including some of
the types that arise only incidentally during processing such as the
@code{listiterator} type.

Typical use of these names is for @ref{236,,isinstance()} or
@ref{237,,issubclass()} checks.

If you instantiate any of these types, note that signatures may vary between Python versions.

Standard names are defined for the following types:

@geindex NoneType (in module types)
@anchor{library/types types NoneType}@anchor{2d8}
@deffn {Data} types.NoneType

The type of @ref{182,,None}.


New in version 3.10.

@end deffn

@geindex FunctionType (in module types)
@geindex LambdaType (in module types)
@anchor{library/types types FunctionType}@anchor{349}
@deffn {Data} types.FunctionType
@anchor{library/types types LambdaType}@anchor{1e37}
@deffnx {Data} types.LambdaType

The type of user-defined functions and functions created by
@ref{e42,,lambda}  expressions.

Raises an @ref{1130,,auditing event} @code{function.__new__} with argument @code{code}.

The audit event only occurs for direct instantiation of function objects,
and is not raised for normal compilation.
@end deffn

@geindex GeneratorType (in module types)
@anchor{library/types types GeneratorType}@anchor{1e38}
@deffn {Data} types.GeneratorType

The type of @ref{bf7,,generator}-iterator objects, created by
generator functions.
@end deffn

@geindex CoroutineType (in module types)
@anchor{library/types types CoroutineType}@anchor{9fb}
@deffn {Data} types.CoroutineType

The type of @ref{3cf,,coroutine} objects, created by
@ref{1c0,,async def} functions.


New in version 3.5.

@end deffn

@geindex AsyncGeneratorType (in module types)
@anchor{library/types types AsyncGeneratorType}@anchor{1e39}
@deffn {Data} types.AsyncGeneratorType

The type of @ref{18ac,,asynchronous generator}-iterator objects, created by
asynchronous generator functions.


New in version 3.6.

@end deffn

@geindex CodeType (class in types)
@anchor{library/types types CodeType}@anchor{4cb}
@deffn {Class} types.CodeType (**kwargs)

@geindex built-in function; compile

The type for code objects such as returned by @ref{4e2,,compile()}.

Raises an @ref{1130,,auditing event} @code{code.__new__} with arguments @code{code}, @code{filename}, @code{name}, @code{argcount}, @code{posonlyargcount}, @code{kwonlyargcount}, @code{nlocals}, @code{stacksize}, @code{flags}.

Note that the audited arguments may not match the names or positions
required by the initializer.  The audit event only occurs for direct
instantiation of code objects, and is not raised for normal compilation.

@geindex replace() (types.CodeType method)
@anchor{library/types types CodeType replace}@anchor{1e3a}
@deffn {Method} replace (**kwargs)

Return a copy of the code object with new values for the specified fields.


New in version 3.8.

@end deffn
@end deffn

@geindex CellType (in module types)
@anchor{library/types types CellType}@anchor{1791}
@deffn {Data} types.CellType

The type for cell objects: such objects are used as containers for
a function’s free variables.


New in version 3.8.

@end deffn

@geindex MethodType (in module types)
@anchor{library/types types MethodType}@anchor{1e3b}
@deffn {Data} types.MethodType

The type of methods of user-defined class instances.
@end deffn

@geindex BuiltinFunctionType (in module types)
@geindex BuiltinMethodType (in module types)
@anchor{library/types types BuiltinFunctionType}@anchor{1e3c}
@deffn {Data} types.BuiltinFunctionType
@anchor{library/types types BuiltinMethodType}@anchor{1e3d}
@deffnx {Data} types.BuiltinMethodType

The type of built-in functions like @ref{4a6,,len()} or @ref{ee4,,sys.exit()}, and
methods of built-in classes.  (Here, the term “built-in” means “written in
C”.)
@end deffn

@geindex WrapperDescriptorType (in module types)
@anchor{library/types types WrapperDescriptorType}@anchor{6ca}
@deffn {Data} types.WrapperDescriptorType

The type of methods of some built-in data types and base classes such as
@ref{f59,,object.__init__()} or @ref{17a7,,object.__lt__()}.


New in version 3.7.

@end deffn

@geindex MethodWrapperType (in module types)
@anchor{library/types types MethodWrapperType}@anchor{177}
@deffn {Data} types.MethodWrapperType

The type of @emph{bound} methods of some built-in data types and base classes.
For example it is the type of @code{object().__str__}.


New in version 3.7.

@end deffn

@geindex NotImplementedType (in module types)
@anchor{library/types types NotImplementedType}@anchor{2d9}
@deffn {Data} types.NotImplementedType

The type of @ref{24c,,NotImplemented}.


New in version 3.10.

@end deffn

@geindex MethodDescriptorType (in module types)
@anchor{library/types types MethodDescriptorType}@anchor{6cb}
@deffn {Data} types.MethodDescriptorType

The type of methods of some built-in data types such as @ref{1a8b,,str.join()}.


New in version 3.7.

@end deffn

@geindex ClassMethodDescriptorType (in module types)
@anchor{library/types types ClassMethodDescriptorType}@anchor{6cc}
@deffn {Data} types.ClassMethodDescriptorType

The type of @emph{unbound} class methods of some built-in data types such as
@code{dict.__dict__['fromkeys']}.


New in version 3.7.

@end deffn

@geindex ModuleType (class in types)
@anchor{library/types types ModuleType}@anchor{973}
@deffn {Class} types.ModuleType (name, doc=None)

The type of @ref{1839,,modules}. The constructor takes the name of the
module to be created and optionally its @ref{1a05,,docstring}.

@cartouche
@quotation Note 
Use @ref{972,,importlib.util.module_from_spec()} to create a new module if you
wish to set the various import-controlled attributes.
@end quotation
@end cartouche

@geindex __doc__ (types.ModuleType attribute)
@anchor{library/types types ModuleType __doc__}@anchor{1e3e}
@deffn {Attribute} __doc__

The @ref{1a05,,docstring} of the module. Defaults to @code{None}.
@end deffn

@geindex __loader__ (types.ModuleType attribute)
@anchor{library/types types ModuleType __loader__}@anchor{1e3f}
@deffn {Attribute} __loader__

The @ref{1848,,loader} which loaded the module. Defaults to @code{None}.

This attribute is to match @ref{1e40,,importlib.machinery.ModuleSpec.loader}
as stored in the @ref{1858,,__spec__} object.

@cartouche
@quotation Note 
A future version of Python may stop setting this attribute by default.
To guard against this potential change, preferably read from the
@ref{1858,,__spec__} attribute instead or use
@code{getattr(module, "__loader__", None)} if you explicitly need to use
this attribute.
@end quotation
@end cartouche


Changed in version 3.4: Defaults to @code{None}. Previously the attribute was optional.

@end deffn

@geindex __name__ (types.ModuleType attribute)
@anchor{library/types types ModuleType __name__}@anchor{1e41}
@deffn {Attribute} __name__

The name of the module. Expected to match
@ref{1e42,,importlib.machinery.ModuleSpec.name}.
@end deffn

@geindex __package__ (types.ModuleType attribute)
@anchor{library/types types ModuleType __package__}@anchor{1e43}
@deffn {Attribute} __package__

Which @ref{183d,,package} a module belongs to. If the module is top-level
(i.e. not a part of any specific package) then the attribute should be set
to @code{'@w{'}}, else it should be set to the name of the package (which can be
@ref{f12,,__name__} if the module is a package itself). Defaults to @code{None}.

This attribute is to match @ref{1e44,,importlib.machinery.ModuleSpec.parent}
as stored in the @ref{1858,,__spec__} object.

@cartouche
@quotation Note 
A future version of Python may stop setting this attribute by default.
To guard against this potential change, preferably read from the
@ref{1858,,__spec__} attribute instead or use
@code{getattr(module, "__package__", None)} if you explicitly need to use
this attribute.
@end quotation
@end cartouche


Changed in version 3.4: Defaults to @code{None}. Previously the attribute was optional.

@end deffn

@geindex __spec__ (types.ModuleType attribute)
@anchor{library/types types ModuleType __spec__}@anchor{1e45}
@deffn {Attribute} __spec__

A record of the module’s import-system-related state. Expected to be an
instance of @ref{1853,,importlib.machinery.ModuleSpec}.


New in version 3.4.

@end deffn
@end deffn

@geindex EllipsisType (in module types)
@anchor{library/types types EllipsisType}@anchor{2d7}
@deffn {Data} types.EllipsisType

The type of @ref{1a67,,Ellipsis}.


New in version 3.10.

@end deffn

@geindex GenericAlias (class in types)
@anchor{library/types types GenericAlias}@anchor{26a}
@deffn {Class} types.GenericAlias (t_origin, t_args)

The type of @ref{268,,parameterized generics} such as
@code{list[int]}.

@code{t_origin} should be a non-parameterized generic class, such as @code{list},
@code{tuple} or @code{dict}.  @code{t_args} should be a @ref{3e5,,tuple} (possibly of
length 1) of types which parameterize @code{t_origin}:

@example
>>> from types import GenericAlias

>>> list[int] == GenericAlias(list, (int,))
True
>>> dict[str, int] == GenericAlias(dict, (str, int))
True
@end example


New in version 3.9.



Changed in version 3.9.2: This type can now be subclassed.

@end deffn

@geindex UnionType (class in types)
@anchor{library/types types UnionType}@anchor{1b4e}
@deffn {Class} types.UnionType

The type of @ref{238,,union type expressions}.


New in version 3.10.

@end deffn

@geindex TracebackType (class in types)
@anchor{library/types types TracebackType}@anchor{60f}
@deffn {Class} types.TracebackType (tb_next, tb_frame, tb_lasti, tb_lineno)

The type of traceback objects such as found in @code{sys.exc_info()[2]}.

See @ref{610,,the language reference} for details of the
available attributes and operations, and guidance on creating tracebacks
dynamically.
@end deffn

@geindex FrameType (in module types)
@anchor{library/types types FrameType}@anchor{10a4}
@deffn {Data} types.FrameType

The type of frame objects such as found in @code{tb.tb_frame} if @code{tb} is a
traceback object.

See @ref{179b,,the language reference} for details of the
available attributes and operations.
@end deffn

@geindex GetSetDescriptorType (in module types)
@anchor{library/types types GetSetDescriptorType}@anchor{1e46}
@deffn {Data} types.GetSetDescriptorType

The type of objects defined in extension modules with @code{PyGetSetDef}, such
as @code{FrameType.f_locals} or @code{array.array.typecode}.  This type is used as
descriptor for object attributes; it has the same purpose as the
@ref{501,,property} type, but for classes defined in extension modules.
@end deffn

@geindex MemberDescriptorType (in module types)
@anchor{library/types types MemberDescriptorType}@anchor{1e47}
@deffn {Data} types.MemberDescriptorType

The type of objects defined in extension modules with @code{PyMemberDef}, such
as @code{datetime.timedelta.days}.  This type is used as descriptor for simple C
data members which use standard conversion functions; it has the same purpose
as the @ref{501,,property} type, but for classes defined in extension modules.

@strong{CPython implementation detail:} In other implementations of Python, this type may be identical to
@code{GetSetDescriptorType}.
@end deffn

@geindex MappingProxyType (class in types)
@anchor{library/types types MappingProxyType}@anchor{246}
@deffn {Class} types.MappingProxyType (mapping)

Read-only proxy of a mapping. It provides a dynamic view on the mapping’s
entries, which means that when the mapping changes, the view reflects these
changes.


New in version 3.3.



Changed in version 3.9: Updated to support the new union (@code{|}) operator from 
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584}, which
simply delegates to the underlying mapping.



@deffn {Describe} key in proxy

Return @code{True} if the underlying mapping has a key @emph{key}, else
@code{False}.
@end deffn


@deffn {Describe} proxy[key]

Return the item of the underlying mapping with key @emph{key}.  Raises a
@ref{5bb,,KeyError} if @emph{key} is not in the underlying mapping.
@end deffn


@deffn {Describe} iter(proxy)

Return an iterator over the keys of the underlying mapping.  This is a
shortcut for @code{iter(proxy.keys())}.
@end deffn


@deffn {Describe} len(proxy)

Return the number of items in the underlying mapping.
@end deffn

@geindex copy() (types.MappingProxyType method)
@anchor{library/types types MappingProxyType copy}@anchor{1e48}
@deffn {Method} copy ()

Return a shallow copy of the underlying mapping.
@end deffn

@geindex get() (types.MappingProxyType method)
@anchor{library/types types MappingProxyType get}@anchor{1e49}
@deffn {Method} get (key[, default])

Return the value for @emph{key} if @emph{key} is in the underlying mapping, else
@emph{default}.  If @emph{default} is not given, it defaults to @code{None}, so that
this method never raises a @ref{5bb,,KeyError}.
@end deffn

@geindex items() (types.MappingProxyType method)
@anchor{library/types types MappingProxyType items}@anchor{1e4a}
@deffn {Method} items ()

Return a new view of the underlying mapping’s items (@code{(key, value)}
pairs).
@end deffn

@geindex keys() (types.MappingProxyType method)
@anchor{library/types types MappingProxyType keys}@anchor{1e4b}
@deffn {Method} keys ()

Return a new view of the underlying mapping’s keys.
@end deffn

@geindex values() (types.MappingProxyType method)
@anchor{library/types types MappingProxyType values}@anchor{1e4c}
@deffn {Method} values ()

Return a new view of the underlying mapping’s values.
@end deffn


@deffn {Describe} reversed(proxy)

Return a reverse iterator over the keys of the underlying mapping.


New in version 3.9.

@end deffn
@end deffn

@node Additional Utility Classes and Functions,Coroutine Utility Functions,Standard Interpreter Types,types — Dynamic type creation and names for built-in types
@anchor{library/types additional-utility-classes-and-functions}@anchor{1e4d}
@subsubsection Additional Utility Classes and Functions


@geindex SimpleNamespace (class in types)
@anchor{library/types types SimpleNamespace}@anchor{41b}
@deffn {Class} types.SimpleNamespace

A simple @ref{5a5,,object} subclass that provides attribute access to its
namespace, as well as a meaningful repr.

Unlike @ref{5a5,,object}, with @code{SimpleNamespace} you can add and remove
attributes.  If a @code{SimpleNamespace} object is initialized with keyword
arguments, those are directly added to the underlying namespace.

The type is roughly equivalent to the following code:

@example
class SimpleNamespace:
    def __init__(self, /, **kwargs):
        self.__dict__.update(kwargs)

    def __repr__(self):
        items = (f"@{k@}=@{v!r@}" for k, v in self.__dict__.items())
        return "@{@}(@{@})".format(type(self).__name__, ", ".join(items))

    def __eq__(self, other):
        if isinstance(self, SimpleNamespace) and isinstance(other, SimpleNamespace):
           return self.__dict__ == other.__dict__
        return NotImplemented
@end example

@code{SimpleNamespace} may be useful as a replacement for @code{class NS: pass}.
However, for a structured record type use @ref{3e4,,namedtuple()}
instead.


New in version 3.3.



Changed in version 3.9: Attribute order in the repr changed from alphabetical to insertion (like
@code{dict}).

@end deffn

@geindex DynamicClassAttribute() (in module types)
@anchor{library/types types DynamicClassAttribute}@anchor{b5b}
@deffn {Function} types.DynamicClassAttribute (fget=None, fset=None, fdel=None, doc=None)

Route attribute access on a class to __getattr__.

This is a descriptor, used to define attributes that act differently when
accessed through an instance and through a class.  Instance access remains
normal, but access to an attribute through a class will be routed to the
class’s __getattr__ method; this is done by raising AttributeError.

This allows one to have properties active on an instance, and have virtual
attributes on the class with the same name (see @ref{658,,enum.Enum} for an example).


New in version 3.4.

@end deffn

@node Coroutine Utility Functions,,Additional Utility Classes and Functions,types — Dynamic type creation and names for built-in types
@anchor{library/types coroutine-utility-functions}@anchor{1e4e}
@subsubsection Coroutine Utility Functions


@geindex coroutine() (in module types)
@anchor{library/types types coroutine}@anchor{9fa}
@deffn {Function} types.coroutine (gen_func)

This function transforms a @ref{bf7,,generator} function into a
@ref{8cf,,coroutine function} which returns a generator-based coroutine.
The generator-based coroutine is still a @ref{715,,generator iterator},
but is also considered to be a @ref{3cf,,coroutine} object and is
@ref{7bd,,awaitable}.  However, it may not necessarily implement
the @code{__await__()} method.

If @emph{gen_func} is a generator function, it will be modified in-place.

If @emph{gen_func} is not a generator function, it will be wrapped. If it
returns an instance of @ref{93a,,collections.abc.Generator}, the instance
will be wrapped in an @emph{awaitable} proxy object.  All other types
of objects will be returned as is.


New in version 3.5.

@end deffn

@node copy — Shallow and deep copy operations,pprint — Data pretty printer,types — Dynamic type creation and names for built-in types,Data Types
@anchor{library/copy doc}@anchor{1e4f}@anchor{library/copy copy-shallow-and-deep-copy-operations}@anchor{1e50}@anchor{library/copy module-copy}@anchor{24}
@subsection @code{copy} — Shallow and deep copy operations


@geindex module; copy

@strong{Source code:} Lib/copy.py@footnote{https://github.com/python/cpython/tree/main/Lib/copy.py}


__________________________________________________________________


Assignment statements in Python do not copy objects, they create bindings
between a target and an object. For collections that are mutable or contain
mutable items, a copy is sometimes needed so one can change one copy without
changing the other. This module provides generic shallow and deep copy
operations (explained below).

Interface summary:

@geindex copy() (in module copy)
@anchor{library/copy copy copy}@anchor{69a}
@deffn {Function} copy.copy (x)

Return a shallow copy of @emph{x}.
@end deffn

@geindex deepcopy() (in module copy)
@anchor{library/copy copy deepcopy}@anchor{69b}
@deffn {Function} copy.deepcopy (x[, memo])

Return a deep copy of @emph{x}.
@end deffn

@geindex Error
@anchor{library/copy copy Error}@anchor{1e51}
@deffn {Exception} copy.Error

Raised for module specific errors.
@end deffn
@anchor{library/copy shallow-vs-deep-copy}@anchor{14e8}
The difference between shallow and deep copying is only relevant for compound
objects (objects that contain other objects, like lists or class instances):


@itemize *

@item 
A @emph{shallow copy} constructs a new compound object and then (to the extent
possible) inserts @emph{references} into it to the objects found in the original.

@item 
A @emph{deep copy} constructs a new compound object and then, recursively, inserts
@emph{copies} into it of the objects found in the original.
@end itemize

Two problems often exist with deep copy operations that don’t exist with shallow
copy operations:


@itemize *

@item 
Recursive objects (compound objects that, directly or indirectly, contain a
reference to themselves) may cause a recursive loop.

@item 
Because deep copy copies everything it may copy too much, such as data
which is intended to be shared between copies.
@end itemize

The @ref{69b,,deepcopy()} function avoids these problems by:


@itemize *

@item 
keeping a @code{memo} dictionary of objects already copied during the current
copying pass; and

@item 
letting user-defined classes override the copying operation or the set of
components copied.
@end itemize

This module does not copy types like module, method, stack trace, stack frame,
file, socket, window, or any similar types.  It does “copy” functions and
classes (shallow and deeply), by returning the original object unchanged; this
is compatible with the way these are treated by the @ref{c5,,pickle} module.

Shallow copies of dictionaries can be made using @ref{706,,dict.copy()}, and
of lists by assigning a slice of the entire list, for example,
@code{copied_list = original_list[:]}.

@geindex module; pickle

Classes can use the same interfaces to control copying that they use to control
pickling.  See the description of module @ref{c5,,pickle} for information on these
methods.  In fact, the @ref{24,,copy} module uses the registered
pickle functions from the @ref{25,,copyreg} module.

@geindex __copy__() (copy protocol)
@geindex __deepcopy__() (copy protocol)

In order for a class to define its own copy implementation, it can define
special methods @code{__copy__()} and @code{__deepcopy__()}.  The former is called
to implement the shallow copy operation; no additional arguments are passed.
The latter is called to implement the deep copy operation; it is passed one
argument, the @code{memo} dictionary.  If the @code{__deepcopy__()} implementation needs
to make a deep copy of a component, it should call the @ref{69b,,deepcopy()} function
with the component as first argument and the memo dictionary as second argument.
The memo dictionary should be treated as an opaque object.


@subsubheading See also



@table @asis

@item Module @ref{c5,,pickle}

Discussion of the special methods used to support object state retrieval and
restoration.
@end table


@node pprint — Data pretty printer,reprlib — Alternate repr implementation,copy — Shallow and deep copy operations,Data Types
@anchor{library/pprint doc}@anchor{1e52}@anchor{library/pprint module-pprint}@anchor{cd}@anchor{library/pprint pprint-data-pretty-printer}@anchor{1e53}
@subsection @code{pprint} — Data pretty printer


@geindex module; pprint

@strong{Source code:} Lib/pprint.py@footnote{https://github.com/python/cpython/tree/main/Lib/pprint.py}


__________________________________________________________________


The @ref{cd,,pprint} module provides a capability to “pretty-print” arbitrary
Python data structures in a form which can be used as input to the interpreter.
If the formatted structures include objects which are not fundamental Python
types, the representation may not be loadable.  This may be the case if objects
such as files, sockets or classes are included, as well as many other
objects which are not representable as Python literals.

The formatted representation keeps objects on a single line if it can, and
breaks them onto multiple lines if they don’t fit within the allowed width.
Construct @ref{b00,,PrettyPrinter} objects explicitly if you need to adjust the
width constraint.

Dictionaries are sorted by key before the display is computed.


Changed in version 3.9: Added support for pretty-printing @ref{41b,,types.SimpleNamespace}.



Changed in version 3.10: Added support for pretty-printing @ref{276,,dataclasses.dataclass}.


The @ref{cd,,pprint} module defines one class:

@c First the implementation class:

@geindex ...; placeholder

@geindex PrettyPrinter (class in pprint)
@anchor{library/pprint pprint PrettyPrinter}@anchor{b00}
@deffn {Class} pprint.PrettyPrinter (indent=1, width=80, depth=None, stream=None, *, compact=False, sort_dicts=True, underscore_numbers=False)

Construct a @ref{b00,,PrettyPrinter} instance.  This constructor understands
several keyword parameters.

@emph{stream} (default @code{sys.stdout}) is a @ref{1e54,,file-like object} to
which the output will be written by calling its @code{write()} method.
If both @emph{stream} and @code{sys.stdout} are @code{None}, then
@ref{1e55,,pprint()} silently returns.

Other values configure the manner in which nesting of complex data
structures is displayed.

@emph{indent} (default 1) specifies the amount of indentation added for
each nesting level.

@emph{depth} controls the number of nesting levels which may be printed; if
the data structure being printed is too deep, the next contained level
is replaced by @code{...}.  By default, there is no constraint on the
depth of the objects being formatted.

@emph{width} (default 80) specifies the desired maximum number of characters per
line in the output. If a structure cannot be formatted within the width
constraint, a best effort will be made.

@emph{compact} impacts the way that long sequences (lists, tuples, sets, etc)
are formatted. If @emph{compact} is false (the default) then each item of a
sequence will be formatted on a separate line.  If @emph{compact} is true, as
many items as will fit within the @emph{width} will be formatted on each output
line.

If @emph{sort_dicts} is true (the default), dictionaries will be formatted with
their keys sorted, otherwise they will display in insertion order.

If @emph{underscore_numbers} is true, integers will be formatted with the
@code{_} character for a thousands separator, otherwise underscores are not
displayed (the default).


Changed in version 3.4: Added the @emph{compact} parameter.



Changed in version 3.8: Added the @emph{sort_dicts} parameter.



Changed in version 3.10: Added the @emph{underscore_numbers} parameter.



Changed in version 3.11: No longer attempts to write to @code{sys.stdout} if it is @code{None}.

@example
>>> import pprint
>>> stuff = ['spam', 'eggs', 'lumberjack', 'knights', 'ni']
>>> stuff.insert(0, stuff[:])
>>> pp = pprint.PrettyPrinter(indent=4)
>>> pp.pprint(stuff)
[   ['spam', 'eggs', 'lumberjack', 'knights', 'ni'],
    'spam',
    'eggs',
    'lumberjack',
    'knights',
    'ni']
>>> pp = pprint.PrettyPrinter(width=41, compact=True)
>>> pp.pprint(stuff)
[['spam', 'eggs', 'lumberjack',
  'knights', 'ni'],
 'spam', 'eggs', 'lumberjack', 'knights',
 'ni']
>>> tup = ('spam', ('eggs', ('lumberjack', ('knights', ('ni', ('dead',
... ('parrot', ('fresh fruit',))))))))
>>> pp = pprint.PrettyPrinter(depth=6)
>>> pp.pprint(tup)
('spam', ('eggs', ('lumberjack', ('knights', ('ni', ('dead', (...)))))))
@end example

@end deffn

@geindex pformat() (in module pprint)
@anchor{library/pprint pprint pformat}@anchor{b01}
@deffn {Function} pprint.pformat (object, indent=1, width=80, depth=None, *, compact=False, sort_dicts=True, underscore_numbers=False)

Return the formatted representation of @emph{object} as a string.  @emph{indent},
@emph{width}, @emph{depth}, @emph{compact}, @emph{sort_dicts} and @emph{underscore_numbers} are
passed to the @ref{b00,,PrettyPrinter} constructor as formatting parameters
and their meanings are as described in its documentation above.
@end deffn

@geindex pp() (in module pprint)
@anchor{library/pprint pprint pp}@anchor{532}
@deffn {Function} pprint.pp (object, *args, sort_dicts=False, **kwargs)

Prints the formatted representation of @emph{object} followed by a newline.
If @emph{sort_dicts} is false (the default), dictionaries will be displayed with
their keys in insertion order, otherwise the dict keys will be sorted.
@emph{args} and @emph{kwargs} will be passed to @ref{cd,,pprint()} as formatting
parameters.


New in version 3.8.

@end deffn

@geindex pprint() (in module pprint)
@anchor{library/pprint pprint pprint}@anchor{2ac}
@deffn {Function} pprint.pprint (object, stream=None, indent=1, width=80, depth=None, *, compact=False, sort_dicts=True, underscore_numbers=False)

Prints the formatted representation of @emph{object} on @emph{stream}, followed by a
newline.  If @emph{stream} is @code{None}, @code{sys.stdout} is used. This may be used
in the interactive interpreter instead of the @ref{af6,,print()} function for
inspecting values (you can even reassign @code{print = pprint.pprint} for use
within a scope).

The configuration parameters @emph{stream}, @emph{indent}, @emph{width}, @emph{depth},
@emph{compact}, @emph{sort_dicts} and @emph{underscore_numbers} are passed to the
@ref{b00,,PrettyPrinter} constructor and their meanings are as
described in its documentation above.

@example
>>> import pprint
>>> stuff = ['spam', 'eggs', 'lumberjack', 'knights', 'ni']
>>> stuff.insert(0, stuff)
>>> pprint.pprint(stuff)
[<Recursion on list with id=...>,
 'spam',
 'eggs',
 'lumberjack',
 'knights',
 'ni']
@end example
@end deffn

@geindex isreadable() (in module pprint)
@anchor{library/pprint pprint isreadable}@anchor{1e56}
@deffn {Function} pprint.isreadable (object)

@geindex built-in function; eval

Determine if the formatted representation of @emph{object} is “readable”, or can be
used to reconstruct the value using @ref{34a,,eval()}.  This always returns @code{False}
for recursive objects.

@example
>>> pprint.isreadable(stuff)
False
@end example
@end deffn

@geindex isrecursive() (in module pprint)
@anchor{library/pprint pprint isrecursive}@anchor{1e57}
@deffn {Function} pprint.isrecursive (object)

Determine if @emph{object} requires a recursive representation.
@end deffn

One more support function is also defined:

@geindex saferepr() (in module pprint)
@anchor{library/pprint pprint saferepr}@anchor{1e58}
@deffn {Function} pprint.saferepr (object)

Return a string representation of @emph{object}, protected against recursive data
structures.  If the representation of @emph{object} exposes a recursive entry, the
recursive reference will be represented as @code{<Recursion on typename with
id=number>}.  The representation is not otherwise formatted.

@example
>>> pprint.saferepr(stuff)
"[<Recursion on list with id=...>, 'spam', 'eggs', 'lumberjack', 'knights', 'ni']"
@end example
@end deffn

@menu
* PrettyPrinter Objects:: 
* Example: Example<3>. 

@end menu

@node PrettyPrinter Objects,Example<3>,,pprint — Data pretty printer
@anchor{library/pprint id1}@anchor{1e59}@anchor{library/pprint prettyprinter-objects}@anchor{1e5a}
@subsubsection PrettyPrinter Objects


@ref{b00,,PrettyPrinter} instances have the following methods:

@geindex pformat() (pprint.PrettyPrinter method)
@anchor{library/pprint pprint PrettyPrinter pformat}@anchor{1e5b}
@deffn {Method} PrettyPrinter.pformat (object)

Return the formatted representation of @emph{object}.  This takes into account the
options passed to the @ref{b00,,PrettyPrinter} constructor.
@end deffn

@geindex pprint() (pprint.PrettyPrinter method)
@anchor{library/pprint pprint PrettyPrinter pprint}@anchor{1e55}
@deffn {Method} PrettyPrinter.pprint (object)

Print the formatted representation of @emph{object} on the configured stream,
followed by a newline.
@end deffn

The following methods provide the implementations for the corresponding
functions of the same names.  Using these methods on an instance is slightly
more efficient since new @ref{b00,,PrettyPrinter} objects don’t need to be
created.

@geindex isreadable() (pprint.PrettyPrinter method)
@anchor{library/pprint pprint PrettyPrinter isreadable}@anchor{1e5c}
@deffn {Method} PrettyPrinter.isreadable (object)

@geindex built-in function; eval

Determine if the formatted representation of the object is “readable,” or can be
used to reconstruct the value using @ref{34a,,eval()}.  Note that this returns
@code{False} for recursive objects.  If the @emph{depth} parameter of the
@ref{b00,,PrettyPrinter} is set and the object is deeper than allowed, this
returns @code{False}.
@end deffn

@geindex isrecursive() (pprint.PrettyPrinter method)
@anchor{library/pprint pprint PrettyPrinter isrecursive}@anchor{1e5d}
@deffn {Method} PrettyPrinter.isrecursive (object)

Determine if the object requires a recursive representation.
@end deffn

This method is provided as a hook to allow subclasses to modify the way objects
are converted to strings.  The default implementation uses the internals of the
@ref{1e58,,saferepr()} implementation.

@geindex format() (pprint.PrettyPrinter method)
@anchor{library/pprint pprint PrettyPrinter format}@anchor{1197}
@deffn {Method} PrettyPrinter.format (object, context, maxlevels, level)

Returns three values: the formatted version of @emph{object} as a string, a flag
indicating whether the result is readable, and a flag indicating whether
recursion was detected.  The first argument is the object to be presented.  The
second is a dictionary which contains the @ref{f80,,id()} of objects that are part of
the current presentation context (direct and indirect containers for @emph{object}
that are affecting the presentation) as the keys; if an object needs to be
presented which is already represented in @emph{context}, the third return value
should be @code{True}.  Recursive calls to the @ref{1197,,format()} method should add
additional entries for containers to this dictionary.  The third argument,
@emph{maxlevels}, gives the requested limit to recursion; this will be @code{0} if there
is no requested limit.  This argument should be passed unmodified to recursive
calls. The fourth argument, @emph{level}, gives the current level; recursive calls
should be passed a value less than that of the current call.
@end deffn

@node Example<3>,,PrettyPrinter Objects,pprint — Data pretty printer
@anchor{library/pprint example}@anchor{1e5e}@anchor{library/pprint pprint-example}@anchor{1e5f}
@subsubsection Example


To demonstrate several uses of the @ref{cd,,pprint()} function and its parameters,
let’s fetch information about a project from PyPI@footnote{https://pypi.org}:

@example
>>> import json
>>> import pprint
>>> from urllib.request import urlopen
>>> with urlopen('https://pypi.org/pypi/sampleproject/json') as resp:
...     project_info = json.load(resp)['info']
@end example

In its basic form, @ref{cd,,pprint()} shows the whole object:

@example
>>> pprint.pprint(project_info)
@{'author': 'The Python Packaging Authority',
 'author_email': 'pypa-dev@@googlegroups.com',
 'bugtrack_url': None,
 'classifiers': ['Development Status :: 3 - Alpha',
                 'Intended Audience :: Developers',
                 'License :: OSI Approved :: MIT License',
                 'Programming Language :: Python :: 2',
                 'Programming Language :: Python :: 2.6',
                 'Programming Language :: Python :: 2.7',
                 'Programming Language :: Python :: 3',
                 'Programming Language :: Python :: 3.2',
                 'Programming Language :: Python :: 3.3',
                 'Programming Language :: Python :: 3.4',
                 'Topic :: Software Development :: Build Tools'],
 'description': 'A sample Python project\n'
                '=======================\n'
                '\n'
                'This is the description file for the project.\n'
                '\n'
                'The file should use UTF-8 encoding and be written using '
                'ReStructured Text. It\n'
                'will be used to generate the project webpage on PyPI, and '
                'should be written for\n'
                'that purpose.\n'
                '\n'
                'Typical contents for this file would include an overview of '
                'the project, basic\n'
                'usage examples, etc. Generally, including the project '
                'changelog in here is not\n'
                'a good idea, although a simple "What\'s New" section for the '
                'most recent version\n'
                'may be appropriate.',
 'description_content_type': None,
 'docs_url': None,
 'download_url': 'UNKNOWN',
 'downloads': @{'last_day': -1, 'last_month': -1, 'last_week': -1@},
 'home_page': 'https://github.com/pypa/sampleproject',
 'keywords': 'sample setuptools development',
 'license': 'MIT',
 'maintainer': None,
 'maintainer_email': None,
 'name': 'sampleproject',
 'package_url': 'https://pypi.org/project/sampleproject/',
 'platform': 'UNKNOWN',
 'project_url': 'https://pypi.org/project/sampleproject/',
 'project_urls': @{'Download': 'UNKNOWN',
                  'Homepage': 'https://github.com/pypa/sampleproject'@},
 'release_url': 'https://pypi.org/project/sampleproject/1.2.0/',
 'requires_dist': None,
 'requires_python': None,
 'summary': 'A sample Python project',
 'version': '1.2.0'@}
@end example

The result can be limited to a certain @emph{depth} (ellipsis is used for deeper
contents):

@example
>>> pprint.pprint(project_info, depth=1)
@{'author': 'The Python Packaging Authority',
 'author_email': 'pypa-dev@@googlegroups.com',
 'bugtrack_url': None,
 'classifiers': [...],
 'description': 'A sample Python project\n'
                '=======================\n'
                '\n'
                'This is the description file for the project.\n'
                '\n'
                'The file should use UTF-8 encoding and be written using '
                'ReStructured Text. It\n'
                'will be used to generate the project webpage on PyPI, and '
                'should be written for\n'
                'that purpose.\n'
                '\n'
                'Typical contents for this file would include an overview of '
                'the project, basic\n'
                'usage examples, etc. Generally, including the project '
                'changelog in here is not\n'
                'a good idea, although a simple "What\'s New" section for the '
                'most recent version\n'
                'may be appropriate.',
 'description_content_type': None,
 'docs_url': None,
 'download_url': 'UNKNOWN',
 'downloads': @{...@},
 'home_page': 'https://github.com/pypa/sampleproject',
 'keywords': 'sample setuptools development',
 'license': 'MIT',
 'maintainer': None,
 'maintainer_email': None,
 'name': 'sampleproject',
 'package_url': 'https://pypi.org/project/sampleproject/',
 'platform': 'UNKNOWN',
 'project_url': 'https://pypi.org/project/sampleproject/',
 'project_urls': @{...@},
 'release_url': 'https://pypi.org/project/sampleproject/1.2.0/',
 'requires_dist': None,
 'requires_python': None,
 'summary': 'A sample Python project',
 'version': '1.2.0'@}
@end example

Additionally, maximum character @emph{width} can be suggested. If a long object
cannot be split, the specified width will be exceeded:

@example
>>> pprint.pprint(project_info, depth=1, width=60)
@{'author': 'The Python Packaging Authority',
 'author_email': 'pypa-dev@@googlegroups.com',
 'bugtrack_url': None,
 'classifiers': [...],
 'description': 'A sample Python project\n'
                '=======================\n'
                '\n'
                'This is the description file for the '
                'project.\n'
                '\n'
                'The file should use UTF-8 encoding and be '
                'written using ReStructured Text. It\n'
                'will be used to generate the project '
                'webpage on PyPI, and should be written '
                'for\n'
                'that purpose.\n'
                '\n'
                'Typical contents for this file would '
                'include an overview of the project, '
                'basic\n'
                'usage examples, etc. Generally, including '
                'the project changelog in here is not\n'
                'a good idea, although a simple "What\'s '
                'New" section for the most recent version\n'
                'may be appropriate.',
 'description_content_type': None,
 'docs_url': None,
 'download_url': 'UNKNOWN',
 'downloads': @{...@},
 'home_page': 'https://github.com/pypa/sampleproject',
 'keywords': 'sample setuptools development',
 'license': 'MIT',
 'maintainer': None,
 'maintainer_email': None,
 'name': 'sampleproject',
 'package_url': 'https://pypi.org/project/sampleproject/',
 'platform': 'UNKNOWN',
 'project_url': 'https://pypi.org/project/sampleproject/',
 'project_urls': @{...@},
 'release_url': 'https://pypi.org/project/sampleproject/1.2.0/',
 'requires_dist': None,
 'requires_python': None,
 'summary': 'A sample Python project',
 'version': '1.2.0'@}
@end example

@node reprlib — Alternate repr implementation,enum — Support for enumerations,pprint — Data pretty printer,Data Types
@anchor{library/reprlib doc}@anchor{1e60}@anchor{library/reprlib module-reprlib}@anchor{da}@anchor{library/reprlib reprlib-alternate-repr-implementation}@anchor{1e61}
@subsection @code{reprlib} — Alternate @code{repr()} implementation


@geindex module; reprlib

@strong{Source code:} Lib/reprlib.py@footnote{https://github.com/python/cpython/tree/main/Lib/reprlib.py}


__________________________________________________________________


The @ref{da,,reprlib} module provides a means for producing object representations
with limits on the size of the resulting strings. This is used in the Python
debugger and may be useful in other contexts as well.

This module provides a class, an instance, and a function:

@geindex Repr (class in reprlib)
@anchor{library/reprlib reprlib Repr}@anchor{1e62}
@deffn {Class} reprlib.Repr

Class which provides formatting services useful in implementing functions
similar to the built-in @ref{27d,,repr()}; size limits for  different object types
are added to avoid the generation of representations which are excessively long.
@end deffn

@geindex aRepr (in module reprlib)
@anchor{library/reprlib reprlib aRepr}@anchor{1e63}
@deffn {Data} reprlib.aRepr

This is an instance of @ref{1e62,,Repr} which is used to provide the
@ref{1e64,,repr()} function described below.  Changing the attributes of this
object will affect the size limits used by @ref{1e64,,repr()} and the Python
debugger.
@end deffn

@geindex repr() (in module reprlib)
@anchor{library/reprlib reprlib repr}@anchor{1e64}
@deffn {Function} reprlib.repr (obj)

This is the @ref{1e65,,repr()} method of @code{aRepr}.  It returns a string
similar to that returned by the built-in function of the same name, but with
limits on most sizes.
@end deffn

In addition to size-limiting tools, the module also provides a decorator for
detecting recursive calls to @code{__repr__()} and substituting a placeholder
string instead.

@geindex ...; placeholder

@geindex recursive_repr() (in module reprlib)
@anchor{library/reprlib reprlib recursive_repr}@anchor{d95}
@deffn {Function} @@reprlib.recursive_repr (fillvalue='...')

Decorator for @code{__repr__()} methods to detect recursive calls within the
same thread.  If a recursive call is made, the @emph{fillvalue} is returned,
otherwise, the usual @code{__repr__()} call is made.  For example:

@example
>>> from reprlib import recursive_repr
>>> class MyList(list):
...     @@recursive_repr()
...     def __repr__(self):
...         return '<' + '|'.join(map(repr, self)) + '>'
...
>>> m = MyList('abc')
>>> m.append(m)
>>> m.append('x')
>>> print(m)
<'a'|'b'|'c'|...|'x'>
@end example


New in version 3.2.

@end deffn

@menu
* Repr Objects:: 
* Subclassing Repr Objects:: 

@end menu

@node Repr Objects,Subclassing Repr Objects,,reprlib — Alternate repr implementation
@anchor{library/reprlib id1}@anchor{1e66}@anchor{library/reprlib repr-objects}@anchor{1e67}
@subsubsection Repr Objects


@ref{1e62,,Repr} instances provide several attributes which can be used to provide
size limits for the representations of different object types,  and methods
which format specific object types.

@geindex fillvalue (reprlib.Repr attribute)
@anchor{library/reprlib reprlib Repr fillvalue}@anchor{1e68}
@deffn {Attribute} Repr.fillvalue

This string is displayed for recursive references. It defaults to
@code{...}.


New in version 3.11.

@end deffn

@geindex maxlevel (reprlib.Repr attribute)
@anchor{library/reprlib reprlib Repr maxlevel}@anchor{1e69}
@deffn {Attribute} Repr.maxlevel

Depth limit on the creation of recursive representations.  The default is @code{6}.
@end deffn

@geindex maxdict (reprlib.Repr attribute)
@geindex maxlist (reprlib.Repr attribute)
@geindex maxtuple (reprlib.Repr attribute)
@geindex maxset (reprlib.Repr attribute)
@geindex maxfrozenset (reprlib.Repr attribute)
@geindex maxdeque (reprlib.Repr attribute)
@geindex maxarray (reprlib.Repr attribute)
@anchor{library/reprlib reprlib Repr maxdict}@anchor{1e6a}
@deffn {Attribute} Repr.maxdict
@anchor{library/reprlib reprlib Repr maxlist}@anchor{1e6b}
@deffnx {Attribute} Repr.maxlist
@anchor{library/reprlib reprlib Repr maxtuple}@anchor{1e6c}
@deffnx {Attribute} Repr.maxtuple
@anchor{library/reprlib reprlib Repr maxset}@anchor{1e6d}
@deffnx {Attribute} Repr.maxset
@anchor{library/reprlib reprlib Repr maxfrozenset}@anchor{1e6e}
@deffnx {Attribute} Repr.maxfrozenset
@anchor{library/reprlib reprlib Repr maxdeque}@anchor{1e6f}
@deffnx {Attribute} Repr.maxdeque
@anchor{library/reprlib reprlib Repr maxarray}@anchor{1e70}
@deffnx {Attribute} Repr.maxarray

Limits on the number of entries represented for the named object type.  The
default is @code{4} for @ref{1e6a,,maxdict}, @code{5} for @ref{1e70,,maxarray}, and  @code{6} for
the others.
@end deffn

@geindex maxlong (reprlib.Repr attribute)
@anchor{library/reprlib reprlib Repr maxlong}@anchor{1e71}
@deffn {Attribute} Repr.maxlong

Maximum number of characters in the representation for an integer.  Digits
are dropped from the middle.  The default is @code{40}.
@end deffn

@geindex maxstring (reprlib.Repr attribute)
@anchor{library/reprlib reprlib Repr maxstring}@anchor{1e72}
@deffn {Attribute} Repr.maxstring

Limit on the number of characters in the representation of the string.  Note
that the “normal” representation of the string is used as the character source:
if escape sequences are needed in the representation, these may be mangled when
the representation is shortened.  The default is @code{30}.
@end deffn

@geindex maxother (reprlib.Repr attribute)
@anchor{library/reprlib reprlib Repr maxother}@anchor{1e73}
@deffn {Attribute} Repr.maxother

This limit is used to control the size of object types for which no specific
formatting method is available on the @ref{1e62,,Repr} object. It is applied in a
similar manner as @ref{1e72,,maxstring}.  The default is @code{20}.
@end deffn

@geindex repr() (reprlib.Repr method)
@anchor{library/reprlib reprlib Repr repr}@anchor{1e65}
@deffn {Method} Repr.repr (obj)

The equivalent to the built-in @ref{27d,,repr()} that uses the formatting imposed by
the instance.
@end deffn

@geindex repr1() (reprlib.Repr method)
@anchor{library/reprlib reprlib Repr repr1}@anchor{1e74}
@deffn {Method} Repr.repr1 (obj, level)

Recursive implementation used by @ref{1e65,,repr()}.  This uses the type of @emph{obj} to
determine which formatting method to call, passing it @emph{obj} and @emph{level}.  The
type-specific methods should call @ref{1e74,,repr1()} to perform recursive formatting,
with @code{level - 1} for the value of @emph{level} in the recursive  call.
@end deffn


@deffn {Method} Repr.repr_TYPE (obj, level)

Formatting methods for specific types are implemented as methods with a name
based on the type name.  In the method name, @strong{TYPE} is replaced by
@code{'_'.join(type(obj).__name__.split())}. Dispatch to these methods is
handled by @ref{1e74,,repr1()}. Type-specific methods which need to recursively
format a value should call @code{self.repr1(subobj, level - 1)}.
@end deffn

@node Subclassing Repr Objects,,Repr Objects,reprlib — Alternate repr implementation
@anchor{library/reprlib subclassing-repr-objects}@anchor{1e75}@anchor{library/reprlib subclassing-reprs}@anchor{1e76}
@subsubsection Subclassing Repr Objects


The use of dynamic dispatching by @ref{1e74,,Repr.repr1()} allows subclasses of
@ref{1e62,,Repr} to add support for additional built-in object types or to modify
the handling of types already supported. This example shows how special support
for file objects could be added:

@example
import reprlib
import sys

class MyRepr(reprlib.Repr):

    def repr_TextIOWrapper(self, obj, level):
        if obj.name in @{'<stdin>', '<stdout>', '<stderr>'@}:
            return obj.name
        return repr(obj)

aRepr = MyRepr()
print(aRepr.repr(sys.stdin))         # prints '<stdin>'
@end example

@node enum — Support for enumerations,graphlib — Functionality to operate with graph-like structures,reprlib — Alternate repr implementation,Data Types
@anchor{library/enum doc}@anchor{1e77}@anchor{library/enum enum-support-for-enumerations}@anchor{1e78}@anchor{library/enum module-enum}@anchor{76}
@subsection @code{enum} — Support for enumerations


@geindex module; enum


New in version 3.4.


@strong{Source code:} Lib/enum.py@footnote{https://github.com/python/cpython/tree/main/Lib/enum.py}

@subsubheading Important



This page contains the API reference information. For tutorial
information and discussion of more advanced topics, see


@itemize *

@item 
@ref{1e79,,Basic Tutorial}

@item 
@ref{1e7a,,Advanced Tutorial}

@item 
@ref{1e7b,,Enum Cookbook}
@end itemize


__________________________________________________________________


An enumeration:


@itemize *

@item 
is a set of symbolic names (members) bound to unique values

@item 
can be iterated over to return its members in definition order

@item 
uses @emph{call} syntax to return members by value

@item 
uses @emph{index} syntax to return members by name
@end itemize

Enumerations are created either by using @ref{e72,,class} syntax, or by
using function-call syntax:

@example
>>> from enum import Enum

>>> # class syntax
>>> class Color(Enum):
...     RED = 1
...     GREEN = 2
...     BLUE = 3

>>> # functional syntax
>>> Color = Enum('Color', ['RED', 'GREEN', 'BLUE'])
@end example

Even though we can use @ref{e72,,class} syntax to create Enums, Enums
are not normal Python classes.  See
@ref{1e7c,,How are Enums different?} for more details.

@cartouche
@quotation Note 
Nomenclature


@itemize -

@item 
The class @code{Color} is an @emph{enumeration} (or @emph{enum})

@item 
The attributes @code{Color.RED}, @code{Color.GREEN}, etc., are
@emph{enumeration members} (or @emph{members}) and are functionally constants.

@item 
The enum members have @emph{names} and @emph{values} (the name of
@code{Color.RED} is @code{RED}, the value of @code{Color.BLUE} is
@code{3}, etc.)
@end itemize
@end quotation
@end cartouche


__________________________________________________________________


@menu
* Module Contents: Module Contents<2>. 
* Data Types: Data Types<2>. 
* Utilities and Decorators:: 
* Notes:: 

@end menu

@node Module Contents<2>,Data Types<2>,,enum — Support for enumerations
@anchor{library/enum module-contents}@anchor{1e7d}
@subsubsection Module Contents


@quotation

@ref{1e7e,,EnumType}

@quotation

The @code{type} for Enum and its subclasses.
@end quotation

@ref{658,,Enum}

@quotation

Base class for creating enumerated constants.
@end quotation

@ref{828,,IntEnum}

@quotation

Base class for creating enumerated constants that are also
subclasses of @ref{1b1,,int}. (@ref{1e7f,,Notes})
@end quotation

@ref{27e,,StrEnum}

@quotation

Base class for creating enumerated constants that are also
subclasses of @ref{2ef,,str}. (@ref{1e7f,,Notes})
@end quotation

@ref{7e7,,Flag}

@quotation

Base class for creating enumerated constants that can be combined using
the bitwise operations without losing their @ref{7e7,,Flag} membership.
@end quotation

@ref{11c2,,IntFlag}

@quotation

Base class for creating enumerated constants that can be combined using
the bitwise operators without losing their @ref{11c2,,IntFlag} membership.
@ref{11c2,,IntFlag} members are also subclasses of @ref{1b1,,int}. (@ref{1e7f,,Notes})
@end quotation

@ref{1e80,,EnumCheck}

@quotation

An enumeration with the values @code{CONTINUOUS}, @code{NAMED_FLAGS}, and
@code{UNIQUE}, for use with @ref{1e81,,verify()} to ensure various constraints
are met by a given enumeration.
@end quotation

@ref{1e82,,FlagBoundary}

@quotation

An enumeration with the values @code{STRICT}, @code{CONFORM}, @code{EJECT}, and
@code{KEEP} which allows for more fine-grained control over how invalid values
are dealt with in an enumeration.
@end quotation

@ref{7e8,,auto}

@quotation

Instances are replaced with an appropriate value for Enum members.
@ref{27e,,StrEnum} defaults to the lower-cased version of the member name,
while other Enums default to 1 and increase from there.
@end quotation

@ref{1e83,,property()}

@quotation

Allows @ref{658,,Enum} members to have attributes without conflicting with
member names.
@end quotation

@ref{1e84,,unique()}

@quotation

Enum class decorator that ensures only one name is bound to any one value.
@end quotation

@ref{1e81,,verify()}

@quotation

Enum class decorator that checks user-selectable constraints on an
enumeration.
@end quotation
@end quotation


New in version 3.6: @code{Flag}, @code{IntFlag}, @code{auto}



New in version 3.11: @code{StrEnum}, @code{EnumCheck}, @code{FlagBoundary}, @code{property}



__________________________________________________________________


@node Data Types<2>,Utilities and Decorators,Module Contents<2>,enum — Support for enumerations
@anchor{library/enum data-types}@anchor{1e85}
@subsubsection Data Types


@geindex EnumType (class in enum)
@anchor{library/enum enum EnumType}@anchor{1e7e}
@deffn {Class} enum.EnumType

@emph{EnumType} is the @ref{17d3,,metaclass} for @emph{enum} enumerations.  It is possible
to subclass @emph{EnumType} – see @ref{1e86,,Subclassing EnumType}
for details.

@emph{EnumType} is responsible for setting the correct @code{__repr__()},
@code{__str__()}, @code{__format__()}, and @code{__reduce__()} methods on the
final @emph{enum}, as well as creating the enum members, properly handling
duplicates, providing iteration over the enum class, etc.

@geindex __contains__() (enum.EnumType method)
@anchor{library/enum enum EnumType __contains__}@anchor{1e87}
@deffn {Method} __contains__ (cls, member)

Returns @code{True} if member belongs to the @code{cls}:

@example
>>> some_var = Color.RED
>>> some_var in Color
True
@end example

@cartouche
@quotation Note 
In Python 3.12 it will be possible to check for member values and not
just members; until then, a @code{TypeError} will be raised if a
non-Enum-member is used in a containment check.
@end quotation
@end cartouche
@end deffn

@geindex __dir__() (enum.EnumType method)
@anchor{library/enum enum EnumType __dir__}@anchor{1e88}
@deffn {Method} __dir__ (cls)

Returns @code{['__class__', '__doc__', '__members__', '__module__']} and the
names of the members in @emph{cls}:

@example
>>> dir(Color)
['BLUE', 'GREEN', 'RED', '__class__', '__contains__', '__doc__', '__getitem__', '__init_subclass__', '__iter__', '__len__', '__members__', '__module__', '__name__', '__qualname__']
@end example
@end deffn

@geindex __getattr__() (enum.EnumType method)
@anchor{library/enum enum EnumType __getattr__}@anchor{1e89}
@deffn {Method} __getattr__ (cls, name)

Returns the Enum member in @emph{cls} matching @emph{name}, or raises an @ref{163,,AttributeError}:

@example
>>> Color.GREEN
<Color.GREEN: 2>
@end example
@end deffn

@geindex __getitem__() (enum.EnumType method)
@anchor{library/enum enum EnumType __getitem__}@anchor{1e8a}
@deffn {Method} __getitem__ (cls, name)

Returns the Enum member in @emph{cls} matching @emph{name}, or raises an @ref{5bb,,KeyError}:

@example
>>> Color['BLUE']
<Color.BLUE: 3>
@end example
@end deffn

@geindex __iter__() (enum.EnumType method)
@anchor{library/enum enum EnumType __iter__}@anchor{1e8b}
@deffn {Method} __iter__ (cls)

Returns each member in @emph{cls} in definition order:

@example
>>> list(Color)
[<Color.RED: 1>, <Color.GREEN: 2>, <Color.BLUE: 3>]
@end example
@end deffn

@geindex __len__() (enum.EnumType method)
@anchor{library/enum enum EnumType __len__}@anchor{1e8c}
@deffn {Method} __len__ (cls)

Returns the number of member in @emph{cls}:

@example
>>> len(Color)
3
@end example
@end deffn

@geindex __reversed__() (enum.EnumType method)
@anchor{library/enum enum EnumType __reversed__}@anchor{1e8d}
@deffn {Method} __reversed__ (cls)

Returns each member in @emph{cls} in reverse definition order:

@example
>>> list(reversed(Color))
[<Color.BLUE: 3>, <Color.GREEN: 2>, <Color.RED: 1>]
@end example
@end deffn
@end deffn

@geindex Enum (class in enum)
@anchor{library/enum enum Enum}@anchor{658}
@deffn {Class} enum.Enum

@emph{Enum} is the base class for all @emph{enum} enumerations.

@geindex name (enum.Enum attribute)
@anchor{library/enum enum Enum name}@anchor{1e8e}
@deffn {Attribute} name

The name used to define the @code{Enum} member:

@example
>>> Color.BLUE.name
'BLUE'
@end example
@end deffn

@geindex value (enum.Enum attribute)
@anchor{library/enum enum Enum value}@anchor{1e8f}
@deffn {Attribute} value

The value given to the @code{Enum} member:

@example
>>> Color.RED.value
1
@end example

@cartouche
@quotation Note 
Enum member values

Member values can be anything: @ref{1b1,,int}, @ref{2ef,,str}, etc..  If
the exact value is unimportant you may use @ref{7e8,,auto} instances and an
appropriate value will be chosen for you.  Care must be taken if you mix
@ref{7e8,,auto} with other values.
@end quotation
@end cartouche
@end deffn

@geindex _ignore_ (enum.Enum attribute)
@anchor{library/enum enum Enum _ignore_}@anchor{1e90}
@deffn {Attribute} _ignore_

@code{_ignore_} is only used during creation and is removed from the
enumeration once creation is complete.

@code{_ignore_} is a list of names that will not become members, and whose
names will also be removed from the completed enumeration.  See
@ref{1e91,,TimePeriod} for an example.
@end deffn

@geindex __call__() (enum.Enum method)
@anchor{library/enum enum Enum __call__}@anchor{10a9}
@deffn {Method} __call__ (cls, value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

This method is called in two different ways:


@itemize *

@item 
to look up an existing member:

@quotation

@*cls: 
The enum class being called.


@*value: 
The value to lookup.

@end quotation

@item 
to use the @code{cls} enum to create a new enum:

@quotation

@*cls: 
The enum class being called.


@*value: 
The name of the new Enum to create.


@*names: 
The names/values of the members for the new Enum.


@*module: 
The name of the module the new Enum is created in.


@*qualname: 
The actual location in the module where this Enum can be found.


@*type: 
A mix-in type for the new Enum.


@*start: 
The first integer value for the Enum (used by @ref{7e8,,auto})


@*boundary: 
How to handle out-of-range values from bit operations (@ref{7e7,,Flag} only)

@end quotation
@end itemize
@end deffn

@geindex __dir__() (enum.Enum method)
@anchor{library/enum enum Enum __dir__}@anchor{10bb}
@deffn {Method} __dir__ (self)

Returns @code{['__class__', '__doc__', '__module__', 'name', 'value']} and
any public methods defined on @emph{self.__class__}:

@example
>>> from datetime import date
>>> class Weekday(Enum):
...     MONDAY = 1
...     TUESDAY = 2
...     WEDNESDAY = 3
...     THURSDAY = 4
...     FRIDAY = 5
...     SATURDAY = 6
...     SUNDAY = 7
...     @@classmethod
...     def today(cls):
...         print('today is %s' % cls(date.today().isoweekday()).name)
>>> dir(Weekday.SATURDAY)
['__class__', '__doc__', '__eq__', '__hash__', '__module__', 'name', 'today', 'value']
@end example
@end deffn

@geindex _generate_next_value_() (enum.Enum method)
@anchor{library/enum enum Enum _generate_next_value_}@anchor{1e92}
@deffn {Method} _generate_next_value_ (name, start, count, last_values)

@quotation

@*name: 
The name of the member being defined (e.g. ‘RED’).


@*start: 
The start value for the Enum; the default is 1.


@*count: 
The number of members currently defined, not including this one.


@*last_values: 
A list of the previous values.

@end quotation

A @emph{staticmethod} that is used to determine the next value returned by
@ref{7e8,,auto}:

@example
>>> from enum import auto
>>> class PowersOfThree(Enum):
...     @@staticmethod
...     def _generate_next_value_(name, start, count, last_values):
...         return (count + 1) * 3
...     FIRST = auto()
...     SECOND = auto()
>>> PowersOfThree.SECOND.value
6
@end example
@end deffn

@geindex __init_subclass__() (enum.Enum method)
@anchor{library/enum enum Enum __init_subclass__}@anchor{1e93}
@deffn {Method} __init_subclass__ (cls, **kwds)

A @emph{classmethod} that is used to further configure subsequent subclasses.
By default, does nothing.
@end deffn

@geindex _missing_() (enum.Enum method)
@anchor{library/enum enum Enum _missing_}@anchor{1e94}
@deffn {Method} _missing_ (cls, value)

A @emph{classmethod} for looking up values not found in @emph{cls}.  By default it
does nothing, but can be overridden to implement custom search behavior:

@example
>>> from enum import StrEnum
>>> class Build(StrEnum):
...     DEBUG = auto()
...     OPTIMIZED = auto()
...     @@classmethod
...     def _missing_(cls, value):
...         value = value.lower()
...         for member in cls:
...             if member.value == value:
...                 return member
...         return None
>>> Build.DEBUG.value
'debug'
>>> Build('deBUG')
<Build.DEBUG: 'debug'>
@end example
@end deffn

@geindex __repr__() (enum.Enum method)
@anchor{library/enum enum Enum __repr__}@anchor{1e95}
@deffn {Method} __repr__ (self)

Returns the string used for @emph{repr()} calls.  By default, returns the
@emph{Enum} name, member name, and value, but can be overridden:

@example
>>> class OtherStyle(Enum):
...     ALTERNATE = auto()
...     OTHER = auto()
...     SOMETHING_ELSE = auto()
...     def __repr__(self):
...         cls_name = self.__class__.__name__
...         return f'@{cls_name@}.@{self.name@}'
>>> OtherStyle.ALTERNATE, str(OtherStyle.ALTERNATE), f"@{OtherStyle.ALTERNATE@}"
(OtherStyle.ALTERNATE, 'OtherStyle.ALTERNATE', 'OtherStyle.ALTERNATE')
@end example
@end deffn

@geindex __str__() (enum.Enum method)
@anchor{library/enum enum Enum __str__}@anchor{1e96}
@deffn {Method} __str__ (self)

Returns the string used for @emph{str()} calls.  By default, returns the
@emph{Enum} name and member name, but can be overridden:

@example
>>> class OtherStyle(Enum):
...     ALTERNATE = auto()
...     OTHER = auto()
...     SOMETHING_ELSE = auto()
...     def __str__(self):
...         return f'@{self.name@}'
>>> OtherStyle.ALTERNATE, str(OtherStyle.ALTERNATE), f"@{OtherStyle.ALTERNATE@}"
(<OtherStyle.ALTERNATE: 1>, 'ALTERNATE', 'ALTERNATE')
@end example
@end deffn

@geindex __format__() (enum.Enum method)
@anchor{library/enum enum Enum __format__}@anchor{1e97}
@deffn {Method} __format__ (self)

Returns the string used for @emph{format()} and @emph{f-string} calls.  By default,
returns @ref{1e96,,__str__()} returns, but can be overridden:

@example
>>> class OtherStyle(Enum):
...     ALTERNATE = auto()
...     OTHER = auto()
...     SOMETHING_ELSE = auto()
...     def __format__(self, spec):
...         return f'@{self.name@}'
>>> OtherStyle.ALTERNATE, str(OtherStyle.ALTERNATE), f"@{OtherStyle.ALTERNATE@}"
(<OtherStyle.ALTERNATE: 1>, 'OtherStyle.ALTERNATE', 'ALTERNATE')
@end example
@end deffn

@cartouche
@quotation Note 
Using @ref{7e8,,auto} with @ref{658,,Enum} results in integers of increasing value,
starting with @code{1}.
@end quotation
@end cartouche
@end deffn

@geindex IntEnum (class in enum)
@anchor{library/enum enum IntEnum}@anchor{828}
@deffn {Class} enum.IntEnum

@emph{IntEnum} is the same as @emph{Enum}, but its members are also integers and can be
used anywhere that an integer can be used.  If any integer operation is performed
with an @emph{IntEnum} member, the resulting value loses its enumeration status.

@example
>>> from enum import IntEnum
>>> class Numbers(IntEnum):
...     ONE = 1
...     TWO = 2
...     THREE = 3
>>> Numbers.THREE
<Numbers.THREE: 3>
>>> Numbers.ONE + Numbers.TWO
3
>>> Numbers.THREE + 5
8
>>> Numbers.THREE == 3
True
@end example

@cartouche
@quotation Note 
Using @ref{7e8,,auto} with @ref{828,,IntEnum} results in integers of increasing
value, starting with @code{1}.
@end quotation
@end cartouche


Changed in version 3.11: @code{__str__()} is now @code{int.__str__()} to
better support the @emph{replacement of existing constants} use-case.
@code{__format__()} was already @code{int.__format__()} for that same reason.

@end deffn

@geindex StrEnum (class in enum)
@anchor{library/enum enum StrEnum}@anchor{27e}
@deffn {Class} enum.StrEnum

@emph{StrEnum} is the same as @emph{Enum}, but its members are also strings and can be used
in most of the same places that a string can be used.  The result of any string
operation performed on or with a @emph{StrEnum} member is not part of the enumeration.

@cartouche
@quotation Note 
There are places in the stdlib that check for an exact @ref{2ef,,str}
instead of a @ref{2ef,,str} subclass (i.e. @code{type(unknown) == str}
instead of @code{isinstance(str, unknown)}), and in those locations you
will need to use @code{str(StrEnum.member)}.
@end quotation
@end cartouche

@cartouche
@quotation Note 
Using @ref{7e8,,auto} with @ref{27e,,StrEnum} results in the lower-cased member
name as the value.
@end quotation
@end cartouche

@cartouche
@quotation Note 
@code{__str__()} is @code{str.__str__()} to better support the
@emph{replacement of existing constants} use-case.  @code{__format__()} is likewise
@code{str.__format__()} for that same reason.
@end quotation
@end cartouche


New in version 3.11.

@end deffn

@geindex Flag (class in enum)
@anchor{library/enum enum Flag}@anchor{7e7}
@deffn {Class} enum.Flag

@emph{Flag} members support the bitwise operators @code{&} (@emph{AND}), @code{|} (@emph{OR}),
@code{^} (@emph{XOR}), and @code{~} (@emph{INVERT}); the results of those operators are members
of the enumeration.

@geindex __contains__() (enum.Flag method)
@anchor{library/enum enum Flag __contains__}@anchor{1e98}
@deffn {Method} __contains__ (self, value)

Returns @emph{True} if value is in self:

@example
>>> from enum import Flag, auto
>>> class Color(Flag):
...     RED = auto()
...     GREEN = auto()
...     BLUE = auto()
>>> purple = Color.RED | Color.BLUE
>>> white = Color.RED | Color.GREEN | Color.BLUE
>>> Color.GREEN in purple
False
>>> Color.GREEN in white
True
>>> purple in white
True
>>> white in purple
False
@end example
@end deffn


@deffn {Method} __iter__(self):

Returns all contained members:

@example
>>> list(Color.RED)
[<Color.RED: 1>]
>>> list(purple)
[<Color.RED: 1>, <Color.BLUE: 4>]
@end example
@end deffn


@deffn {Method} __len__(self):

Returns number of members in flag:

@example
>>> len(Color.GREEN)
1
>>> len(white)
3
@end example
@end deffn


@deffn {Method} __bool__(self):

Returns @emph{True} if any members in flag, @emph{False} otherwise:

@example
>>> bool(Color.GREEN)
True
>>> bool(white)
True
>>> black = Color(0)
>>> bool(black)
False
@end example
@end deffn

@geindex __or__() (enum.Flag method)
@anchor{library/enum enum Flag __or__}@anchor{1e99}
@deffn {Method} __or__ (self, other)

Returns current flag binary or’ed with other:

@example
>>> Color.RED | Color.GREEN
<Color.RED|GREEN: 3>
@end example
@end deffn

@geindex __and__() (enum.Flag method)
@anchor{library/enum enum Flag __and__}@anchor{1e9a}
@deffn {Method} __and__ (self, other)

Returns current flag binary and’ed with other:

@example
>>> purple & white
<Color.RED|BLUE: 5>
>>> purple & Color.GREEN
<Color: 0>
@end example
@end deffn

@geindex __xor__() (enum.Flag method)
@anchor{library/enum enum Flag __xor__}@anchor{1e9b}
@deffn {Method} __xor__ (self, other)

Returns current flag binary xor’ed with other:

@example
>>> purple ^ white
<Color.GREEN: 2>
>>> purple ^ Color.GREEN
<Color.RED|GREEN|BLUE: 7>
@end example
@end deffn


@deffn {Method} __invert__(self):

Returns all the flags in @emph{type(self)} that are not in self:

@example
>>> ~white
<Color: 0>
>>> ~purple
<Color.GREEN: 2>
>>> ~Color.RED
<Color.GREEN|BLUE: 6>
@end example
@end deffn

@geindex _numeric_repr_() (enum.Flag method)
@anchor{library/enum enum Flag _numeric_repr_}@anchor{1e9c}
@deffn {Method} _numeric_repr_ ()

Function used to format any remaining unnamed numeric values.  Default is
the value’s repr; common choices are @ref{e6f,,hex()} and @ref{e6e,,oct()}.
@end deffn

@cartouche
@quotation Note 
Using @ref{7e8,,auto} with @ref{7e7,,Flag} results in integers that are powers
of two, starting with @code{1}.
@end quotation
@end cartouche


Changed in version 3.11: The @emph{repr()} of zero-valued flags has changed.  It
is now::

@example
>>> Color(0) 
<Color: 0>
@end example

@end deffn

@geindex IntFlag (class in enum)
@anchor{library/enum enum IntFlag}@anchor{11c2}
@deffn {Class} enum.IntFlag

@emph{IntFlag} is the same as @emph{Flag}, but its members are also integers and can be
used anywhere that an integer can be used.

@example
>>> from enum import IntFlag, auto
>>> class Color(IntFlag):
...     RED = auto()
...     GREEN = auto()
...     BLUE = auto()
>>> Color.RED & 2
<Color: 0>
>>> Color.RED | 2
<Color.RED|GREEN: 3>
@end example

If any integer operation is performed with an @emph{IntFlag} member, the result is
not an @emph{IntFlag}:

@example
>>> Color.RED + 2
3
@end example

If a @emph{Flag} operation is performed with an @emph{IntFlag} member and:

@quotation


@itemize *

@item 
the result is a valid @emph{IntFlag}: an @emph{IntFlag} is returned

@item 
the result is not a valid @emph{IntFlag}: the result depends on the @emph{FlagBoundary} setting
@end itemize
@end quotation

The @emph{repr()} of unnamed zero-valued flags has changed.  It is now:

@example
>>> Color(0)
<Color: 0>
@end example

@cartouche
@quotation Note 
Using @ref{7e8,,auto} with @ref{11c2,,IntFlag} results in integers that are powers
of two, starting with @code{1}.
@end quotation
@end cartouche


Changed in version 3.11: @code{__str__()} is now @code{int.__str__()} to
better support the @emph{replacement of existing constants} use-case.
@code{__format__()} was already @code{int.__format__()} for that same reason.

@end deffn

@geindex EnumCheck (class in enum)
@anchor{library/enum enum EnumCheck}@anchor{1e80}
@deffn {Class} enum.EnumCheck

@emph{EnumCheck} contains the options used by the @ref{1e81,,verify()} decorator to ensure
various constraints; failed constraints result in a @ref{17c,,ValueError}.

@geindex UNIQUE (enum.EnumCheck attribute)
@anchor{library/enum enum EnumCheck UNIQUE}@anchor{1e9d}
@deffn {Attribute} UNIQUE

Ensure that each value has only one name:

@example
>>> from enum import Enum, verify, UNIQUE
>>> @@verify(UNIQUE)
... class Color(Enum):
...     RED = 1
...     GREEN = 2
...     BLUE = 3
...     CRIMSON = 1
Traceback (most recent call last):
...
ValueError: aliases found in <enum 'Color'>: CRIMSON -> RED
@end example
@end deffn

@geindex CONTINUOUS (enum.EnumCheck attribute)
@anchor{library/enum enum EnumCheck CONTINUOUS}@anchor{1e9e}
@deffn {Attribute} CONTINUOUS

Ensure that there are no missing values between the lowest-valued member
and the highest-valued member:

@example
>>> from enum import Enum, verify, CONTINUOUS
>>> @@verify(CONTINUOUS)
... class Color(Enum):
...     RED = 1
...     GREEN = 2
...     BLUE = 5
Traceback (most recent call last):
...
ValueError: invalid enum 'Color': missing values 3, 4
@end example
@end deffn

@geindex NAMED_FLAGS (enum.EnumCheck attribute)
@anchor{library/enum enum EnumCheck NAMED_FLAGS}@anchor{1e9f}
@deffn {Attribute} NAMED_FLAGS

Ensure that any flag groups/masks contain only named flags – useful when
values are specified instead of being generated by @ref{7e8,,auto()}

@example
>>> from enum import Flag, verify, NAMED_FLAGS
>>> @@verify(NAMED_FLAGS)
... class Color(Flag):
...     RED = 1
...     GREEN = 2
...     BLUE = 4
...     WHITE = 15
...     NEON = 31
Traceback (most recent call last):
...
ValueError: invalid Flag 'Color': aliases WHITE and NEON are missing combined values of 0x18 [use enum.show_flag_values(value) for details]
@end example
@end deffn

@cartouche
@quotation Note 
CONTINUOUS and NAMED_FLAGS are designed to work with integer-valued members.
@end quotation
@end cartouche


New in version 3.11.

@end deffn

@geindex FlagBoundary (class in enum)
@anchor{library/enum enum FlagBoundary}@anchor{1e82}
@deffn {Class} enum.FlagBoundary

@emph{FlagBoundary} controls how out-of-range values are handled in @emph{Flag} and its
subclasses.

@geindex STRICT (enum.FlagBoundary attribute)
@anchor{library/enum enum FlagBoundary STRICT}@anchor{1ea0}
@deffn {Attribute} STRICT

Out-of-range values cause a @ref{17c,,ValueError} to be raised.  This is the
default for @ref{7e7,,Flag}:

@example
>>> from enum import Flag, STRICT
>>> class StrictFlag(Flag, boundary=STRICT):
...     RED = auto()
...     GREEN = auto()
...     BLUE = auto()
>>> StrictFlag(2**2 + 2**4)
Traceback (most recent call last):
...
ValueError: <flag 'StrictFlag'> invalid value 20
    given 0b0 10100
  allowed 0b0 00111
@end example
@end deffn

@geindex CONFORM (enum.FlagBoundary attribute)
@anchor{library/enum enum FlagBoundary CONFORM}@anchor{1ea1}
@deffn {Attribute} CONFORM

Out-of-range values have invalid values removed, leaving a valid @emph{Flag}
value:

@example
>>> from enum import Flag, CONFORM
>>> class ConformFlag(Flag, boundary=CONFORM):
...     RED = auto()
...     GREEN = auto()
...     BLUE = auto()
>>> ConformFlag(2**2 + 2**4)
<ConformFlag.BLUE: 4>
@end example
@end deffn

@geindex EJECT (enum.FlagBoundary attribute)
@anchor{library/enum enum FlagBoundary EJECT}@anchor{1ea2}
@deffn {Attribute} EJECT

Out-of-range values lose their @emph{Flag} membership and revert to @ref{1b1,,int}.
This is the default for @ref{11c2,,IntFlag}:

@example
>>> from enum import Flag, EJECT
>>> class EjectFlag(Flag, boundary=EJECT):
...     RED = auto()
...     GREEN = auto()
...     BLUE = auto()
>>> EjectFlag(2**2 + 2**4)
20
@end example
@end deffn

@geindex KEEP (enum.FlagBoundary attribute)
@anchor{library/enum enum FlagBoundary KEEP}@anchor{1ea3}
@deffn {Attribute} KEEP

Out-of-range values are kept, and the @emph{Flag} membership is kept.  This is
used for some stdlib flags:

@example
>>> from enum import Flag, KEEP
>>> class KeepFlag(Flag, boundary=KEEP):
...     RED = auto()
...     GREEN = auto()
...     BLUE = auto()
>>> KeepFlag(2**2 + 2**4)
<KeepFlag.BLUE|16: 20>
@end example
@end deffn
@end deffn


New in version 3.11.



__________________________________________________________________


@menu
* Supported __dunder__ names:: 
* Supported _sunder_ names:: 

@end menu

@node Supported __dunder__ names,Supported _sunder_ names,,Data Types<2>
@anchor{library/enum supported-dunder-names}@anchor{1ea4}
@subsubsection Supported @code{__dunder__} names


@code{__members__} is a read-only ordered mapping of @code{member_name}:@code{member}
items.  It is only available on the class.

@code{__new__()}, if specified, must create and return the enum members; it is
also a very good idea to set the member’s @code{_value_} appropriately.  Once
all the members are created it is no longer used.

@node Supported _sunder_ names,,Supported __dunder__ names,Data Types<2>
@anchor{library/enum supported-sunder-names}@anchor{1ea5}
@subsubsection Supported @code{_sunder_} names



@itemize -

@item 
@code{_name_} – name of the member

@item 
@code{_value_} – value of the member; can be set / modified in @code{__new__}

@item 
@code{_missing_} – a lookup function used when a value is not found; may be
overridden

@item 
@code{_ignore_} – a list of names, either as a @ref{43a,,list} or a @ref{2ef,,str},
that will not be transformed into members, and will be removed from the final
class

@item 
@code{_order_} – used in Python 2/3 code to ensure member order is consistent
(class attribute, removed during class creation)

@item 
@code{_generate_next_value_} – used to get an appropriate value for an enum
member; may be overridden

@quotation

@cartouche
@quotation Note 
For standard @ref{658,,Enum} classes the next value chosen is the last value seen
incremented by one.

For @ref{7e7,,Flag} classes the next value chosen will be the next highest
power-of-two, regardless of the last value seen.
@end quotation
@end cartouche
@end quotation
@end itemize


New in version 3.6: @code{_missing_}, @code{_order_}, @code{_generate_next_value_}



New in version 3.7: @code{_ignore_}



__________________________________________________________________


@node Utilities and Decorators,Notes,Data Types<2>,enum — Support for enumerations
@anchor{library/enum utilities-and-decorators}@anchor{1ea6}
@subsubsection Utilities and Decorators


@geindex auto (class in enum)
@anchor{library/enum enum auto}@anchor{7e8}
@deffn {Class} enum.auto

@emph{auto} can be used in place of a value.  If used, the @emph{Enum} machinery will
call an @emph{Enum}’s @code{_generate_next_value_()} to get an appropriate value.
For @emph{Enum} and @emph{IntEnum} that appropriate value will be the last value plus
one; for @emph{Flag} and @emph{IntFlag} it will be the first power-of-two greater
than the last value; for @emph{StrEnum} it will be the lower-cased version of the
member’s name.

@code{_generate_next_value_} can be overridden to customize the values used by
@emph{auto}.
@end deffn

@geindex property() (in module enum)
@anchor{library/enum enum property}@anchor{1e83}
@deffn {Function} @@enum.property

A decorator similar to the built-in @emph{property}, but specifically for
enumerations.  It allows member attributes to have the same names as members
themselves.

@cartouche
@quotation Note 
the @emph{property} and the member must be defined in separate classes;
for example, the @emph{value} and @emph{name} attributes are defined in the
@emph{Enum} class, and @emph{Enum} subclasses can define members with the
names @code{value} and @code{name}.
@end quotation
@end cartouche


New in version 3.11.

@end deffn

@geindex unique() (in module enum)
@anchor{library/enum enum unique}@anchor{1e84}
@deffn {Function} @@enum.unique

A @ref{e72,,class} decorator specifically for enumerations.  It searches an
enumeration’s @code{__members__}, gathering any aliases it finds; if any are
found @ref{17c,,ValueError} is raised with the details:

@example
>>> from enum import Enum, unique
>>> @@unique
... class Mistake(Enum):
...     ONE = 1
...     TWO = 2
...     THREE = 3
...     FOUR = 3
...
Traceback (most recent call last):
...
ValueError: duplicate values found in <enum 'Mistake'>: FOUR -> THREE
@end example
@end deffn

@geindex verify() (in module enum)
@anchor{library/enum enum verify}@anchor{1e81}
@deffn {Function} @@enum.verify

A @ref{e72,,class} decorator specifically for enumerations.  Members from
@ref{1e80,,EnumCheck} are used to specify which constraints should be checked
on the decorated enumeration.


New in version 3.11.

@end deffn


__________________________________________________________________


@node Notes,,Utilities and Decorators,enum — Support for enumerations
@anchor{library/enum notes}@anchor{1e7f}
@subsubsection Notes


@ref{828,,IntEnum}, @ref{27e,,StrEnum}, and @ref{11c2,,IntFlag}

@quotation

@quotation

These three enum types are designed to be drop-in replacements for existing
integer- and string-based values; as such, they have extra limitations:


@itemize -

@item 
@code{__str__} uses the value and not the name of the enum member

@item 
@code{__format__}, because it uses @code{__str__}, will also use the value of
the enum member instead of its name
@end itemize

If you do not need/want those limitations, you can either create your own
base class by mixing in the @code{int} or @code{str} type yourself:

@example
>>> from enum import Enum
>>> class MyIntEnum(int, Enum):
...     pass
@end example
@end quotation

or you can reassign the appropriate @ref{2ef,,str()}, etc., in your enum:

@example
>>> from enum import IntEnum
>>> class MyIntEnum(IntEnum):
...     __str__ = IntEnum.__str__
@end example
@end quotation

@node graphlib — Functionality to operate with graph-like structures,,enum — Support for enumerations,Data Types
@anchor{library/graphlib doc}@anchor{1ea7}@anchor{library/graphlib graphlib-functionality-to-operate-with-graph-like-structures}@anchor{1ea8}@anchor{library/graphlib module-graphlib}@anchor{85}
@subsection @code{graphlib} — Functionality to operate with graph-like structures


@geindex module; graphlib

@strong{Source code:} Lib/graphlib.py@footnote{https://github.com/python/cpython/tree/main/Lib/graphlib.py}

@c import graphlib
@c from graphlib import *


__________________________________________________________________


@geindex TopologicalSorter (class in graphlib)
@anchor{library/graphlib graphlib TopologicalSorter}@anchor{3c8}
@deffn {Class} graphlib.TopologicalSorter (graph=None)

Provides functionality to topologically sort a graph of hashable nodes.

A topological order is a linear ordering of the vertices in a graph such that
for every directed edge u -> v from vertex u to vertex v, vertex u comes
before vertex v in the ordering. For instance, the vertices of the graph may
represent tasks to be performed, and the edges may represent constraints that
one task must be performed before another; in this example, a topological
ordering is just a valid sequence for the tasks. A complete topological
ordering is possible if and only if the graph has no directed cycles, that
is, if it is a directed acyclic graph.

If the optional @emph{graph} argument is provided it must be a dictionary
representing a directed acyclic graph where the keys are nodes and the values
are iterables of all predecessors of that node in the graph (the nodes that
have edges that point to the value in the key). Additional nodes can be added
to the graph using the @ref{1ea9,,add()} method.

In the general case, the steps required to perform the sorting of a given
graph are as follows:

@quotation


@itemize *

@item 
Create an instance of the @ref{3c8,,TopologicalSorter} with an optional
initial graph.

@item 
Add additional nodes to the graph.

@item 
Call @ref{1eaa,,prepare()} on the graph.

@item 
While @ref{1eab,,is_active()} is @code{True}, iterate over
the nodes returned by @ref{1eac,,get_ready()} and
process them. Call @ref{1ead,,done()} on each node as it
finishes processing.
@end itemize
@end quotation

In case just an immediate sorting of the nodes in the graph is required and
no parallelism is involved, the convenience method
@ref{1eae,,TopologicalSorter.static_order()} can be used directly:

@example
>>> graph = @{"D": @{"B", "C"@}, "C": @{"A"@}, "B": @{"A"@}@}
>>> ts = TopologicalSorter(graph)
>>> tuple(ts.static_order())
('A', 'C', 'B', 'D')
@end example

The class is designed to easily support parallel processing of the nodes as
they become ready. For instance:

@example
topological_sorter = TopologicalSorter()

# Add nodes to 'topological_sorter'...

topological_sorter.prepare()
while topological_sorter.is_active():
    for node in topological_sorter.get_ready():
        # Worker threads or processes take nodes to work on off the
        # 'task_queue' queue.
        task_queue.put(node)

    # When the work for a node is done, workers put the node in
    # 'finalized_tasks_queue' so we can get more nodes to work on.
    # The definition of 'is_active()' guarantees that, at this point, at
    # least one node has been placed on 'task_queue' that hasn't yet
    # been passed to 'done()', so this blocking 'get()' must (eventually)
    # succeed.  After calling 'done()', we loop back to call 'get_ready()'
    # again, so put newly freed nodes on 'task_queue' as soon as
    # logically possible.
    node = finalized_tasks_queue.get()
    topological_sorter.done(node)
@end example

@geindex add() (graphlib.TopologicalSorter method)
@anchor{library/graphlib graphlib TopologicalSorter add}@anchor{1ea9}
@deffn {Method} add (node, *predecessors)

Add a new node and its predecessors to the graph. Both the @emph{node} and all
elements in @emph{predecessors} must be hashable.

If called multiple times with the same node argument, the set of
dependencies will be the union of all dependencies passed in.

It is possible to add a node with no dependencies (@emph{predecessors} is not
provided) or to provide a dependency twice. If a node that has not been
provided before is included among @emph{predecessors} it will be automatically
added to the graph with no predecessors of its own.

Raises @ref{17c,,ValueError} if called after @ref{1eaa,,prepare()}.
@end deffn

@geindex prepare() (graphlib.TopologicalSorter method)
@anchor{library/graphlib graphlib TopologicalSorter prepare}@anchor{1eaa}
@deffn {Method} prepare ()

Mark the graph as finished and check for cycles in the graph. If any cycle
is detected, @ref{1eaf,,CycleError} will be raised, but
@ref{1eac,,get_ready()} can still be used to obtain as many
nodes as possible until cycles block more progress. After a call to this
function, the graph cannot be modified, and therefore no more nodes can be
added using @ref{1ea9,,add()}.
@end deffn

@geindex is_active() (graphlib.TopologicalSorter method)
@anchor{library/graphlib graphlib TopologicalSorter is_active}@anchor{1eab}
@deffn {Method} is_active ()

Returns @code{True} if more progress can be made and @code{False} otherwise.
Progress can be made if cycles do not block the resolution and either
there are still nodes ready that haven’t yet been returned by
@ref{1eac,,TopologicalSorter.get_ready()} or the number of nodes marked
@ref{1ead,,TopologicalSorter.done()} is less than the number that have been
returned by @ref{1eac,,TopologicalSorter.get_ready()}.

The @code{__bool__()} method of this class defers to
this function, so instead of:

@example
if ts.is_active():
    ...
@end example

it is possible to simply do:

@example
if ts:
    ...
@end example

Raises @ref{17c,,ValueError} if called without calling
@ref{1eaa,,prepare()} previously.
@end deffn

@geindex done() (graphlib.TopologicalSorter method)
@anchor{library/graphlib graphlib TopologicalSorter done}@anchor{1ead}
@deffn {Method} done (*nodes)

Marks a set of nodes returned by @ref{1eac,,TopologicalSorter.get_ready()} as
processed, unblocking any successor of each node in @emph{nodes} for being
returned in the future by a call to @ref{1eac,,TopologicalSorter.get_ready()}.

Raises @ref{17c,,ValueError} if any node in @emph{nodes} has already been marked as
processed by a previous call to this method or if a node was not added to
the graph by using @ref{1ea9,,TopologicalSorter.add()}, if called without
calling @ref{1eaa,,prepare()} or if node has not yet been
returned by @ref{1eac,,get_ready()}.
@end deffn

@geindex get_ready() (graphlib.TopologicalSorter method)
@anchor{library/graphlib graphlib TopologicalSorter get_ready}@anchor{1eac}
@deffn {Method} get_ready ()

Returns a @code{tuple} with all the nodes that are ready. Initially it
returns all nodes with no predecessors, and once those are marked as
processed by calling @ref{1ead,,TopologicalSorter.done()}, further calls will
return all new nodes that have all their predecessors already processed.
Once no more progress can be made, empty tuples are returned.

Raises @ref{17c,,ValueError} if called without calling
@ref{1eaa,,prepare()} previously.
@end deffn

@geindex static_order() (graphlib.TopologicalSorter method)
@anchor{library/graphlib graphlib TopologicalSorter static_order}@anchor{1eae}
@deffn {Method} static_order ()

Returns an iterator object which will iterate over nodes in a topological
order. When using this method, @ref{1eaa,,prepare()} and
@ref{1ead,,done()} should not be called. This method is
equivalent to:

@example
def static_order(self):
    self.prepare()
    while self.is_active():
        node_group = self.get_ready()
        yield from node_group
        self.done(*node_group)
@end example

The particular order that is returned may depend on the specific order in
which the items were inserted in the graph. For example:

@example
>>> ts = TopologicalSorter()
>>> ts.add(3, 2, 1)
>>> ts.add(1, 0)
>>> print([*ts.static_order()])
[2, 0, 1, 3]

>>> ts2 = TopologicalSorter()
>>> ts2.add(1, 0)
>>> ts2.add(3, 2, 1)
>>> print([*ts2.static_order()])
[0, 2, 1, 3]
@end example

This is due to the fact that “0” and “2” are in the same level in the
graph (they would have been returned in the same call to
@ref{1eac,,get_ready()}) and the order between them is
determined by the order of insertion.

If any cycle is detected, @ref{1eaf,,CycleError} will be raised.
@end deffn


New in version 3.9.

@end deffn

@menu
* Exceptions: Exceptions<3>. 

@end menu

@node Exceptions<3>,,,graphlib — Functionality to operate with graph-like structures
@anchor{library/graphlib exceptions}@anchor{1eb0}
@subsubsection Exceptions


The @ref{85,,graphlib} module defines the following exception classes:

@geindex CycleError
@anchor{library/graphlib graphlib CycleError}@anchor{1eaf}
@deffn {Exception} graphlib.CycleError

Subclass of @ref{17c,,ValueError} raised by @ref{1eaa,,TopologicalSorter.prepare()} if cycles exist
in the working graph. If multiple cycles exist, only one undefined choice among them will
be reported and included in the exception.

The detected cycle can be accessed via the second element in the @code{args}
attribute of the exception instance and consists in a list of nodes, such that each node is,
in the graph, an immediate predecessor of the next node in the list. In the reported list,
the first and the last node will be the same, to make it clear that it is cyclic.
@end deffn

@node Numeric and Mathematical Modules,Functional Programming Modules,Data Types,The Python Standard Library
@anchor{library/numeric doc}@anchor{1eb1}@anchor{library/numeric numeric}@anchor{1eb2}@anchor{library/numeric numeric-and-mathematical-modules}@anchor{1eb3}
@section Numeric and Mathematical Modules


The modules described in this chapter provide numeric and math-related functions
and data types. The @ref{bd,,numbers} module defines an abstract hierarchy of
numeric types. The @ref{ad,,math} and @ref{17,,cmath} modules contain various
mathematical functions for floating-point and complex numbers. The @ref{34,,decimal}
module supports exact representations of decimal numbers, using arbitrary precision
arithmetic.

The following modules are documented in this chapter:

@menu
* numbers — Numeric abstract base classes:: 
* math — Mathematical functions:: 
* cmath — Mathematical functions for complex numbers:: 
* decimal — Decimal fixed point and floating point arithmetic:: 
* fractions — Rational numbers:: 
* random — Generate pseudo-random numbers:: 
* statistics — Mathematical statistics functions:: 

@end menu

@node numbers — Numeric abstract base classes,math — Mathematical functions,,Numeric and Mathematical Modules
@anchor{library/numbers doc}@anchor{1eb4}@anchor{library/numbers module-numbers}@anchor{bd}@anchor{library/numbers numbers-numeric-abstract-base-classes}@anchor{1eb5}
@subsection @code{numbers} — Numeric abstract base classes


@geindex module; numbers

@strong{Source code:} Lib/numbers.py@footnote{https://github.com/python/cpython/tree/main/Lib/numbers.py}


__________________________________________________________________


The @ref{bd,,numbers} module (
@geindex Python Enhancement Proposals; PEP 3141
PEP 3141@footnote{https://www.python.org/dev/peps/pep-3141}) defines a hierarchy of numeric
@ref{d5c,,abstract base classes} which progressively define
more operations.  None of the types defined in this module are intended to be instantiated.

@geindex Number (class in numbers)
@anchor{library/numbers numbers Number}@anchor{1789}
@deffn {Class} numbers.Number

The root of the numeric hierarchy. If you just want to check if an argument
@emph{x} is a number, without caring what kind, use @code{isinstance(x, Number)}.
@end deffn

@menu
* The numeric tower:: 
* Notes for type implementors:: 

@end menu

@node The numeric tower,Notes for type implementors,,numbers — Numeric abstract base classes
@anchor{library/numbers the-numeric-tower}@anchor{1eb6}
@subsubsection The numeric tower


@geindex Complex (class in numbers)
@anchor{library/numbers numbers Complex}@anchor{178d}
@deffn {Class} numbers.Complex

Subclasses of this type describe complex numbers and include the operations
that work on the built-in @ref{33b,,complex} type. These are: conversions to
@ref{33b,,complex} and @ref{2f7,,bool}, @ref{1eb7,,real}, @ref{1eb8,,imag}, @code{+},
@code{-}, @code{*}, @code{/}, @code{**}, @ref{1582,,abs()}, @ref{1eb9,,conjugate()}, @code{==}, and
@code{!=}. All except @code{-} and @code{!=} are abstract.

@geindex real (numbers.Complex attribute)
@anchor{library/numbers numbers Complex real}@anchor{1eb7}
@deffn {Attribute} real

Abstract. Retrieves the real component of this number.
@end deffn

@geindex imag (numbers.Complex attribute)
@anchor{library/numbers numbers Complex imag}@anchor{1eb8}
@deffn {Attribute} imag

Abstract. Retrieves the imaginary component of this number.
@end deffn

@geindex conjugate() (numbers.Complex method)
@anchor{library/numbers numbers Complex conjugate}@anchor{1eb9}
@deffn {Method} abstractmethod conjugate ()

Abstract. Returns the complex conjugate. For example, @code{(1+3j).conjugate()
== (1-3j)}.
@end deffn
@end deffn

@geindex Real (class in numbers)
@anchor{library/numbers numbers Real}@anchor{178c}
@deffn {Class} numbers.Real

To @ref{178d,,Complex}, @ref{178c,,Real} adds the operations that work on real
numbers.

In short, those are: a conversion to @ref{256,,float}, @ref{f2a,,math.trunc()},
@ref{e75,,round()}, @ref{f28,,math.floor()}, @ref{f29,,math.ceil()}, @ref{4a8,,divmod()}, @code{//},
@code{%}, @code{<}, @code{<=}, @code{>}, and @code{>=}.

Real also provides defaults for @ref{33b,,complex()}, @ref{1eb7,,real},
@ref{1eb8,,imag}, and @ref{1eb9,,conjugate()}.
@end deffn

@geindex Rational (class in numbers)
@anchor{library/numbers numbers Rational}@anchor{e64}
@deffn {Class} numbers.Rational

Subtypes @ref{178c,,Real} and adds
@ref{1eba,,numerator} and @ref{1ebb,,denominator} properties, which
should be in lowest terms. With these, it provides a default for
@ref{256,,float()}.

@geindex numerator (numbers.Rational attribute)
@anchor{library/numbers numbers Rational numerator}@anchor{1eba}
@deffn {Attribute} numerator

Abstract.
@end deffn

@geindex denominator (numbers.Rational attribute)
@anchor{library/numbers numbers Rational denominator}@anchor{1ebb}
@deffn {Attribute} denominator

Abstract.
@end deffn
@end deffn

@geindex Integral (class in numbers)
@anchor{library/numbers numbers Integral}@anchor{178b}
@deffn {Class} numbers.Integral

Subtypes @ref{e64,,Rational} and adds a conversion to @ref{1b1,,int}.  Provides
defaults for @ref{256,,float()}, @ref{1eba,,numerator}, and
@ref{1ebb,,denominator}.  Adds abstract methods for @ref{4cd,,pow()} with
modulus and bit-string operations: @code{<<}, @code{>>}, @code{&}, @code{^}, @code{|},
@code{~}.
@end deffn

@node Notes for type implementors,,The numeric tower,numbers — Numeric abstract base classes
@anchor{library/numbers notes-for-type-implementors}@anchor{1ebc}
@subsubsection Notes for type implementors


Implementors should be careful to make equal numbers equal and hash
them to the same values. This may be subtle if there are two different
extensions of the real numbers. For example, @ref{172,,fractions.Fraction}
implements @ref{c14,,hash()} as follows:

@example
def __hash__(self):
    if self.denominator == 1:
        # Get integers right.
        return hash(self.numerator)
    # Expensive check, but definitely correct.
    if self == float(self):
        return hash(float(self))
    else:
        # Use tuple's hash to avoid a high collision rate on
        # simple fractions.
        return hash((self.numerator, self.denominator))
@end example

@menu
* Adding More Numeric ABCs:: 
* Implementing the arithmetic operations:: 

@end menu

@node Adding More Numeric ABCs,Implementing the arithmetic operations,,Notes for type implementors
@anchor{library/numbers adding-more-numeric-abcs}@anchor{1ebd}
@subsubsection Adding More Numeric ABCs


There are, of course, more possible ABCs for numbers, and this would
be a poor hierarchy if it precluded the possibility of adding
those. You can add @code{MyFoo} between @ref{178d,,Complex} and
@ref{178c,,Real} with:

@example
class MyFoo(Complex): ...
MyFoo.register(Real)
@end example

@node Implementing the arithmetic operations,,Adding More Numeric ABCs,Notes for type implementors
@anchor{library/numbers id1}@anchor{1ebe}@anchor{library/numbers implementing-the-arithmetic-operations}@anchor{1788}
@subsubsection Implementing the arithmetic operations


We want to implement the arithmetic operations so that mixed-mode
operations either call an implementation whose author knew about the
types of both arguments, or convert both to the nearest built in type
and do the operation there. For subtypes of @ref{178b,,Integral}, this
means that @code{__add__()} and @code{__radd__()} should be defined as:

@example
class MyIntegral(Integral):

    def __add__(self, other):
        if isinstance(other, MyIntegral):
            return do_my_adding_stuff(self, other)
        elif isinstance(other, OtherTypeIKnowAbout):
            return do_my_other_adding_stuff(self, other)
        else:
            return NotImplemented

    def __radd__(self, other):
        if isinstance(other, MyIntegral):
            return do_my_adding_stuff(other, self)
        elif isinstance(other, OtherTypeIKnowAbout):
            return do_my_other_adding_stuff(other, self)
        elif isinstance(other, Integral):
            return int(other) + int(self)
        elif isinstance(other, Real):
            return float(other) + float(self)
        elif isinstance(other, Complex):
            return complex(other) + complex(self)
        else:
            return NotImplemented
@end example

There are 5 different cases for a mixed-type operation on subclasses
of @ref{178d,,Complex}. I’ll refer to all of the above code that doesn’t
refer to @code{MyIntegral} and @code{OtherTypeIKnowAbout} as
“boilerplate”. @code{a} will be an instance of @code{A}, which is a subtype
of @ref{178d,,Complex} (@code{a : A <: Complex}), and @code{b : B <:
Complex}. I’ll consider @code{a + b}:

@quotation


@enumerate 

@item 
If @code{A} defines an @code{__add__()} which accepts @code{b}, all is
well.

@item 
If @code{A} falls back to the boilerplate code, and it were to
return a value from @code{__add__()}, we’d miss the possibility
that @code{B} defines a more intelligent @code{__radd__()}, so the
boilerplate should return @ref{24c,,NotImplemented} from
@code{__add__()}. (Or @code{A} may not implement @code{__add__()} at
all.)

@item 
Then @code{B}’s @code{__radd__()} gets a chance. If it accepts
@code{a}, all is well.

@item 
If it falls back to the boilerplate, there are no more possible
methods to try, so this is where the default implementation
should live.

@item 
If @code{B <: A}, Python tries @code{B.__radd__} before
@code{A.__add__}. This is ok, because it was implemented with
knowledge of @code{A}, so it can handle those instances before
delegating to @ref{178d,,Complex}.
@end enumerate
@end quotation

If @code{A <: Complex} and @code{B <: Real} without sharing any other knowledge,
then the appropriate shared operation is the one involving the built
in @ref{33b,,complex}, and both @code{__radd__()} s land there, so @code{a+b
== b+a}.

Because most of the operations on any given type will be very similar,
it can be useful to define a helper function which generates the
forward and reverse instances of any given operator. For example,
@ref{172,,fractions.Fraction} uses:

@example
def _operator_fallbacks(monomorphic_operator, fallback_operator):
    def forward(a, b):
        if isinstance(b, (int, Fraction)):
            return monomorphic_operator(a, b)
        elif isinstance(b, float):
            return fallback_operator(float(a), b)
        elif isinstance(b, complex):
            return fallback_operator(complex(a), b)
        else:
            return NotImplemented
    forward.__name__ = '__' + fallback_operator.__name__ + '__'
    forward.__doc__ = monomorphic_operator.__doc__

    def reverse(b, a):
        if isinstance(a, Rational):
            # Includes ints.
            return monomorphic_operator(a, b)
        elif isinstance(a, numbers.Real):
            return fallback_operator(float(a), float(b))
        elif isinstance(a, numbers.Complex):
            return fallback_operator(complex(a), complex(b))
        else:
            return NotImplemented
    reverse.__name__ = '__r' + fallback_operator.__name__ + '__'
    reverse.__doc__ = monomorphic_operator.__doc__

    return forward, reverse

def _add(a, b):
    """a + b"""
    return Fraction(a.numerator * b.denominator +
                    b.numerator * a.denominator,
                    a.denominator * b.denominator)

__add__, __radd__ = _operator_fallbacks(_add, operator.add)

# ...
@end example

@node math — Mathematical functions,cmath — Mathematical functions for complex numbers,numbers — Numeric abstract base classes,Numeric and Mathematical Modules
@anchor{library/math doc}@anchor{1ebf}@anchor{library/math math-mathematical-functions}@anchor{1ec0}@anchor{library/math module-math}@anchor{ad}
@subsection @code{math} — Mathematical functions


@geindex module; math

@c from math import fsum


__________________________________________________________________


This module provides access to the mathematical functions defined by the C
standard.

These functions cannot be used with complex numbers; use the functions of the
same name from the @ref{17,,cmath} module if you require support for complex
numbers.  The distinction between functions which support complex numbers and
those which don’t is made since most users do not want to learn quite as much
mathematics as required to understand complex numbers.  Receiving an exception
instead of a complex result allows earlier detection of the unexpected complex
number used as a parameter, so that the programmer can determine how and why it
was generated in the first place.

The following functions are provided by this module.  Except when explicitly
noted otherwise, all return values are floats.

@menu
* Number-theoretic and representation functions:: 
* Power and logarithmic functions:: 
* Trigonometric functions:: 
* Angular conversion:: 
* Hyperbolic functions:: 
* Special functions:: 
* Constants: Constants<2>. 

@end menu

@node Number-theoretic and representation functions,Power and logarithmic functions,,math — Mathematical functions
@anchor{library/math number-theoretic-and-representation-functions}@anchor{1ec1}
@subsubsection Number-theoretic and representation functions


@geindex ceil() (in module math)
@anchor{library/math math ceil}@anchor{f29}
@deffn {Function} math.ceil (x)

Return the ceiling of @emph{x}, the smallest integer greater than or equal to @emph{x}.
If @emph{x} is not a float, delegates to @code{x.__ceil__()}, which should return an
@ref{178b,,Integral} value.
@end deffn

@geindex comb() (in module math)
@anchor{library/math math comb}@anchor{19b}
@deffn {Function} math.comb (n, k)

Return the number of ways to choose @emph{k} items from @emph{n} items without repetition
and without order.

Evaluates to @code{n! / (k! * (n - k)!)} when @code{k <= n} and evaluates
to zero when @code{k > n}.

Also called the binomial coefficient because it is equivalent
to the coefficient of k-th term in polynomial expansion of the
expression @code{(1 + x) ** n}.

Raises @ref{162,,TypeError} if either of the arguments are not integers.
Raises @ref{17c,,ValueError} if either of the arguments are negative.


New in version 3.8.

@end deffn

@geindex copysign() (in module math)
@anchor{library/math math copysign}@anchor{f39}
@deffn {Function} math.copysign (x, y)

Return a float with the magnitude (absolute value) of @emph{x} but the sign of
@emph{y}.  On platforms that support signed zeros, @code{copysign(1.0, -0.0)}
returns @emph{-1.0}.
@end deffn

@geindex fabs() (in module math)
@anchor{library/math math fabs}@anchor{1ec2}
@deffn {Function} math.fabs (x)

Return the absolute value of @emph{x}.
@end deffn

@geindex factorial() (in module math)
@anchor{library/math math factorial}@anchor{43e}
@deffn {Function} math.factorial (x)

Return @emph{x} factorial as an integer.  Raises @ref{17c,,ValueError} if @emph{x} is not integral or
is negative.


Deprecated since version 3.9: Accepting floats with integral values (like @code{5.0}) is deprecated.

@end deffn

@geindex floor() (in module math)
@anchor{library/math math floor}@anchor{f28}
@deffn {Function} math.floor (x)

Return the floor of @emph{x}, the largest integer less than or equal to @emph{x}.
If @emph{x} is not a float, delegates to @code{x.__floor__()}, which should return an
@ref{178b,,Integral} value.
@end deffn

@geindex fmod() (in module math)
@anchor{library/math math fmod}@anchor{18f3}
@deffn {Function} math.fmod (x, y)

Return @code{fmod(x, y)}, as defined by the platform C library. Note that the
Python expression @code{x % y} may not return the same result.  The intent of the C
standard is that @code{fmod(x, y)} be exactly (mathematically; to infinite
precision) equal to @code{x - n*y} for some integer @emph{n} such that the result has
the same sign as @emph{x} and magnitude less than @code{abs(y)}.  Python’s @code{x % y}
returns a result with the sign of @emph{y} instead, and may not be exactly computable
for float arguments. For example, @code{fmod(-1e-100, 1e100)} is @code{-1e-100}, but
the result of Python’s @code{-1e-100 % 1e100} is @code{1e100-1e-100}, which cannot be
represented exactly as a float, and rounds to the surprising @code{1e100}.  For
this reason, function @ref{18f3,,fmod()} is generally preferred when working with
floats, while Python’s @code{x % y} is preferred when working with integers.
@end deffn

@geindex frexp() (in module math)
@anchor{library/math math frexp}@anchor{1ec3}
@deffn {Function} math.frexp (x)

Return the mantissa and exponent of @emph{x} as the pair @code{(m, e)}.  @emph{m} is a float
and @emph{e} is an integer such that @code{x == m * 2**e} exactly. If @emph{x} is zero,
returns @code{(0.0, 0)}, otherwise @code{0.5 <= abs(m) < 1}.  This is used to “pick
apart” the internal representation of a float in a portable way.
@end deffn

@geindex fsum() (in module math)
@anchor{library/math math fsum}@anchor{f3a}
@deffn {Function} math.fsum (iterable)

Return an accurate floating point sum of values in the iterable.  Avoids
loss of precision by tracking multiple intermediate partial sums:

@example
>>> sum([.1, .1, .1, .1, .1, .1, .1, .1, .1, .1])
0.9999999999999999
>>> fsum([.1, .1, .1, .1, .1, .1, .1, .1, .1, .1])
1.0
@end example

The algorithm’s accuracy depends on IEEE-754 arithmetic guarantees and the
typical case where the rounding mode is half-even.  On some non-Windows
builds, the underlying C library uses extended precision addition and may
occasionally double-round an intermediate sum causing it to be off in its
least significant bit.

For further discussion and two alternative approaches, see the ASPN cookbook recipes for accurate floating point summation@footnote{https://code.activestate.com/recipes/393090/}.
@end deffn

@geindex gcd() (in module math)
@anchor{library/math math gcd}@anchor{401}
@deffn {Function} math.gcd (*integers)

Return the greatest common divisor of the specified integer arguments.
If any of the arguments is nonzero, then the returned value is the largest
positive integer that is a divisor of all arguments.  If all arguments
are zero, then the returned value is @code{0}.  @code{gcd()} without arguments
returns @code{0}.


New in version 3.5.



Changed in version 3.9: Added support for an arbitrary number of arguments. Formerly, only two
arguments were supported.

@end deffn

@geindex isclose() (in module math)
@anchor{library/math math isclose}@anchor{90d}
@deffn {Function} math.isclose (a, b, *, rel_tol=1e-09, abs_tol=0.0)

Return @code{True} if the values @emph{a} and @emph{b} are close to each other and
@code{False} otherwise.

Whether or not two values are considered close is determined according to
given absolute and relative tolerances.

@emph{rel_tol} is the relative tolerance – it is the maximum allowed difference
between @emph{a} and @emph{b}, relative to the larger absolute value of @emph{a} or @emph{b}.
For example, to set a tolerance of 5%, pass @code{rel_tol=0.05}.  The default
tolerance is @code{1e-09}, which assures that the two values are the same
within about 9 decimal digits.  @emph{rel_tol} must be greater than zero.

@emph{abs_tol} is the minimum absolute tolerance – useful for comparisons near
zero. @emph{abs_tol} must be at least zero.

If no errors occur, the result will be:
@code{abs(a-b) <= max(rel_tol * max(abs(a), abs(b)), abs_tol)}.

The IEEE 754 special values of @code{NaN}, @code{inf}, and @code{-inf} will be
handled according to IEEE rules.  Specifically, @code{NaN} is not considered
close to any other value, including @code{NaN}.  @code{inf} and @code{-inf} are only
considered close to themselves.


New in version 3.5.



@subsubheading See also


@geindex Python Enhancement Proposals; PEP 485
PEP 485@footnote{https://www.python.org/dev/peps/pep-0485} – A function for testing approximate equality

@end deffn

@geindex isfinite() (in module math)
@anchor{library/math math isfinite}@anchor{d8d}
@deffn {Function} math.isfinite (x)

Return @code{True} if @emph{x} is neither an infinity nor a NaN, and
@code{False} otherwise.  (Note that @code{0.0} @emph{is} considered finite.)


New in version 3.2.

@end deffn

@geindex isinf() (in module math)
@anchor{library/math math isinf}@anchor{f37}
@deffn {Function} math.isinf (x)

Return @code{True} if @emph{x} is a positive or negative infinity, and
@code{False} otherwise.
@end deffn

@geindex isnan() (in module math)
@anchor{library/math math isnan}@anchor{f38}
@deffn {Function} math.isnan (x)

Return @code{True} if @emph{x} is a NaN (not a number), and @code{False} otherwise.
@end deffn

@geindex isqrt() (in module math)
@anchor{library/math math isqrt}@anchor{50e}
@deffn {Function} math.isqrt (n)

Return the integer square root of the nonnegative integer @emph{n}. This is the
floor of the exact square root of @emph{n}, or equivalently the greatest integer
@emph{a} such that @emph{a}² ≤ @emph{n}.

For some applications, it may be more convenient to have the least integer
@emph{a} such that @emph{n} ≤ @emph{a}², or in other words the ceiling of
the exact square root of @emph{n}. For positive @emph{n}, this can be computed using
@code{a = 1 + isqrt(n - 1)}.


New in version 3.8.

@end deffn

@geindex lcm() (in module math)
@anchor{library/math math lcm}@anchor{402}
@deffn {Function} math.lcm (*integers)

Return the least common multiple of the specified integer arguments.
If all arguments are nonzero, then the returned value is the smallest
positive integer that is a multiple of all arguments.  If any of the arguments
is zero, then the returned value is @code{0}.  @code{lcm()} without arguments
returns @code{1}.


New in version 3.9.

@end deffn

@geindex ldexp() (in module math)
@anchor{library/math math ldexp}@anchor{1ec4}
@deffn {Function} math.ldexp (x, i)

Return @code{x * (2**i)}.  This is essentially the inverse of function
@ref{1ec3,,frexp()}.
@end deffn

@geindex modf() (in module math)
@anchor{library/math math modf}@anchor{1ec5}
@deffn {Function} math.modf (x)

Return the fractional and integer parts of @emph{x}.  Both results carry the sign
of @emph{x} and are floats.
@end deffn

@geindex nextafter() (in module math)
@anchor{library/math math nextafter}@anchor{403}
@deffn {Function} math.nextafter (x, y)

Return the next floating-point value after @emph{x} towards @emph{y}.

If @emph{x} is equal to @emph{y}, return @emph{y}.

Examples:


@itemize *

@item 
@code{math.nextafter(x, math.inf)} goes up: towards positive infinity.

@item 
@code{math.nextafter(x, -math.inf)} goes down: towards minus infinity.

@item 
@code{math.nextafter(x, 0.0)} goes towards zero.

@item 
@code{math.nextafter(x, math.copysign(math.inf, x))} goes away from zero.
@end itemize

See also @ref{404,,math.ulp()}.


New in version 3.9.

@end deffn

@geindex perm() (in module math)
@anchor{library/math math perm}@anchor{19c}
@deffn {Function} math.perm (n, k=None)

Return the number of ways to choose @emph{k} items from @emph{n} items
without repetition and with order.

Evaluates to @code{n! / (n - k)!} when @code{k <= n} and evaluates
to zero when @code{k > n}.

If @emph{k} is not specified or is None, then @emph{k} defaults to @emph{n}
and the function returns @code{n!}.

Raises @ref{162,,TypeError} if either of the arguments are not integers.
Raises @ref{17c,,ValueError} if either of the arguments are negative.


New in version 3.8.

@end deffn

@geindex prod() (in module math)
@anchor{library/math math prod}@anchor{50c}
@deffn {Function} math.prod (iterable, *, start=1)

Calculate the product of all the elements in the input @emph{iterable}.
The default @emph{start} value for the product is @code{1}.

When the iterable is empty, return the start value.  This function is
intended specifically for use with numeric values and may reject
non-numeric types.


New in version 3.8.

@end deffn

@geindex remainder() (in module math)
@anchor{library/math math remainder}@anchor{67a}
@deffn {Function} math.remainder (x, y)

Return the IEEE 754-style remainder of @emph{x} with respect to @emph{y}.  For
finite @emph{x} and finite nonzero @emph{y}, this is the difference @code{x - n*y},
where @code{n} is the closest integer to the exact value of the quotient @code{x /
y}.  If @code{x / y} is exactly halfway between two consecutive integers, the
nearest @emph{even} integer is used for @code{n}.  The remainder @code{r = remainder(x,
y)} thus always satisfies @code{abs(r) <= 0.5 * abs(y)}.

Special cases follow IEEE 754: in particular, @code{remainder(x, math.inf)} is
@emph{x} for any finite @emph{x}, and @code{remainder(x, 0)} and
@code{remainder(math.inf, x)} raise @ref{17c,,ValueError} for any non-NaN @emph{x}.
If the result of the remainder operation is zero, that zero will have
the same sign as @emph{x}.

On platforms using IEEE 754 binary floating-point, the result of this
operation is always exactly representable: no rounding error is introduced.


New in version 3.7.

@end deffn

@geindex trunc() (in module math)
@anchor{library/math math trunc}@anchor{f2a}
@deffn {Function} math.trunc (x)

Return the @ref{178c,,Real} value @emph{x} truncated to an
@ref{178b,,Integral} (usually an integer). Delegates to
@ref{180b,,x.__trunc__()}.
@end deffn

@geindex ulp() (in module math)
@anchor{library/math math ulp}@anchor{404}
@deffn {Function} math.ulp (x)

Return the value of the least significant bit of the float @emph{x}:


@itemize *

@item 
If @emph{x} is a NaN (not a number), return @emph{x}.

@item 
If @emph{x} is negative, return @code{ulp(-x)}.

@item 
If @emph{x} is a positive infinity, return @emph{x}.

@item 
If @emph{x} is equal to zero, return the smallest positive
@emph{denormalized} representable float (smaller than the minimum positive
@emph{normalized} float, @ref{1290,,sys.float_info.min}).

@item 
If @emph{x} is equal to the largest positive representable float,
return the value of the least significant bit of @emph{x}, such that the first
float smaller than @emph{x} is @code{x - ulp(x)}.

@item 
Otherwise (@emph{x} is a positive finite number), return the value of the least
significant bit of @emph{x}, such that the first float bigger than @emph{x}
is @code{x + ulp(x)}.
@end itemize

ULP stands for “Unit in the Last Place”.

See also @ref{403,,math.nextafter()} and @ref{1290,,sys.float_info.epsilon}.


New in version 3.9.

@end deffn

Note that @ref{1ec3,,frexp()} and @ref{1ec5,,modf()} have a different call/return pattern
than their C equivalents: they take a single argument and return a pair of
values, rather than returning their second return value through an ‘output
parameter’ (there is no such thing in Python).

For the @ref{f29,,ceil()}, @ref{f28,,floor()}, and @ref{1ec5,,modf()} functions, note that @emph{all}
floating-point numbers of sufficiently large magnitude are exact integers.
Python floats typically carry no more than 53 bits of precision (the same as the
platform C double type), in which case any float @emph{x} with @code{abs(x) >= 2**52}
necessarily has no fractional bits.

@node Power and logarithmic functions,Trigonometric functions,Number-theoretic and representation functions,math — Mathematical functions
@anchor{library/math power-and-logarithmic-functions}@anchor{1ec6}
@subsubsection Power and logarithmic functions


@geindex cbrt() (in module math)
@anchor{library/math math cbrt}@anchor{17a}
@deffn {Function} math.cbrt (x)

Return the cube root of @emph{x}.


New in version 3.11.

@end deffn

@geindex exp() (in module math)
@anchor{library/math math exp}@anchor{eab}
@deffn {Function} math.exp (x)

Return @emph{e} raised to the power @emph{x}, where @emph{e} = 2.718281… is the base
of natural logarithms.  This is usually more accurate than @code{math.e ** x}
or @code{pow(math.e, x)}.
@end deffn

@geindex exp2() (in module math)
@anchor{library/math math exp2}@anchor{179}
@deffn {Function} math.exp2 (x)

Return @emph{2} raised to the power @emph{x}.


New in version 3.11.

@end deffn

@geindex expm1() (in module math)
@anchor{library/math math expm1}@anchor{d8e}
@deffn {Function} math.expm1 (x)

Return @emph{e} raised to the power @emph{x}, minus 1.  Here @emph{e} is the base of natural
logarithms.  For small floats @emph{x}, the subtraction in @code{exp(x) - 1}
can result in a significant loss of precision@footnote{https://en.wikipedia.org/wiki/Loss_of_significance}; the @ref{d8e,,expm1()}
function provides a way to compute this quantity to full precision:

@example
>>> from math import exp, expm1
>>> exp(1e-5) - 1  # gives result accurate to 11 places
1.0000050000069649e-05
>>> expm1(1e-5)    # result accurate to full precision
1.0000050000166668e-05
@end example


New in version 3.2.

@end deffn

@geindex log() (in module math)
@anchor{library/math math log}@anchor{100a}
@deffn {Function} math.log (x[, base])

With one argument, return the natural logarithm of @emph{x} (to base @emph{e}).

With two arguments, return the logarithm of @emph{x} to the given @emph{base},
calculated as @code{log(x)/log(base)}.
@end deffn

@geindex log1p() (in module math)
@anchor{library/math math log1p}@anchor{f3e}
@deffn {Function} math.log1p (x)

Return the natural logarithm of @emph{1+x} (base @emph{e}). The
result is calculated in a way which is accurate for @emph{x} near zero.
@end deffn

@geindex log2() (in module math)
@anchor{library/math math log2}@anchor{c6e}
@deffn {Function} math.log2 (x)

Return the base-2 logarithm of @emph{x}. This is usually more accurate than
@code{log(x, 2)}.


New in version 3.3.



@subsubheading See also


@ref{1a7b,,int.bit_length()} returns the number of bits necessary to represent
an integer in binary, excluding the sign and leading zeros.

@end deffn

@geindex log10() (in module math)
@anchor{library/math math log10}@anchor{1ec7}
@deffn {Function} math.log10 (x)

Return the base-10 logarithm of @emph{x}.  This is usually more accurate
than @code{log(x, 10)}.
@end deffn

@geindex pow() (in module math)
@anchor{library/math math pow}@anchor{17b}
@deffn {Function} math.pow (x, y)

Return @code{x} raised to the power @code{y}.  Exceptional cases follow
the IEEE 754 standard as far as possible.  In particular,
@code{pow(1.0, x)} and @code{pow(x, 0.0)} always return @code{1.0}, even
when @code{x} is a zero or a NaN.  If both @code{x} and @code{y} are finite,
@code{x} is negative, and @code{y} is not an integer then @code{pow(x, y)}
is undefined, and raises @ref{17c,,ValueError}.

Unlike the built-in @code{**} operator, @ref{17b,,math.pow()} converts both
its arguments to type @ref{256,,float}.  Use @code{**} or the built-in
@ref{4cd,,pow()} function for computing exact integer powers.


Changed in version 3.11: The special cases @code{pow(0.0, -inf)} and @code{pow(-0.0, -inf)} were
changed to return @code{inf} instead of raising @ref{17c,,ValueError},
for consistency with IEEE 754.

@end deffn

@geindex sqrt() (in module math)
@anchor{library/math math sqrt}@anchor{50f}
@deffn {Function} math.sqrt (x)

Return the square root of @emph{x}.
@end deffn

@node Trigonometric functions,Angular conversion,Power and logarithmic functions,math — Mathematical functions
@anchor{library/math trigonometric-functions}@anchor{1ec8}
@subsubsection Trigonometric functions


@geindex acos() (in module math)
@anchor{library/math math acos}@anchor{1ec9}
@deffn {Function} math.acos (x)

Return the arc cosine of @emph{x}, in radians. The result is between @code{0} and
@code{pi}.
@end deffn

@geindex asin() (in module math)
@anchor{library/math math asin}@anchor{1eca}
@deffn {Function} math.asin (x)

Return the arc sine of @emph{x}, in radians. The result is between @code{-pi/2} and
@code{pi/2}.
@end deffn

@geindex atan() (in module math)
@anchor{library/math math atan}@anchor{1ecb}
@deffn {Function} math.atan (x)

Return the arc tangent of @emph{x}, in radians. The result is between @code{-pi/2} and
@code{pi/2}.
@end deffn

@geindex atan2() (in module math)
@anchor{library/math math atan2}@anchor{1ecc}
@deffn {Function} math.atan2 (y, x)

Return @code{atan(y / x)}, in radians. The result is between @code{-pi} and @code{pi}.
The vector in the plane from the origin to point @code{(x, y)} makes this angle
with the positive X axis. The point of @ref{1ecc,,atan2()} is that the signs of both
inputs are known to it, so it can compute the correct quadrant for the angle.
For example, @code{atan(1)} and @code{atan2(1, 1)} are both @code{pi/4}, but @code{atan2(-1,
-1)} is @code{-3*pi/4}.
@end deffn

@geindex cos() (in module math)
@anchor{library/math math cos}@anchor{1009}
@deffn {Function} math.cos (x)

Return the cosine of @emph{x} radians.
@end deffn

@geindex dist() (in module math)
@anchor{library/math math dist}@anchor{50a}
@deffn {Function} math.dist (p, q)

Return the Euclidean distance between two points @emph{p} and @emph{q}, each
given as a sequence (or iterable) of coordinates.  The two points
must have the same dimension.

Roughly equivalent to:

@example
sqrt(sum((px - qx) ** 2.0 for px, qx in zip(p, q)))
@end example


New in version 3.8.

@end deffn

@geindex hypot() (in module math)
@anchor{library/math math hypot}@anchor{50b}
@deffn {Function} math.hypot (*coordinates)

Return the Euclidean norm, @code{sqrt(sum(x**2 for x in coordinates))}.
This is the length of the vector from the origin to the point
given by the coordinates.

For a two dimensional point @code{(x, y)}, this is equivalent to computing
the hypotenuse of a right triangle using the Pythagorean theorem,
@code{sqrt(x*x + y*y)}.


Changed in version 3.8: Added support for n-dimensional points. Formerly, only the two
dimensional case was supported.



Changed in version 3.10: Improved the algorithm’s accuracy so that the maximum error is
under 1 ulp (unit in the last place).  More typically, the result
is almost always correctly rounded to within 1/2 ulp.

@end deffn

@geindex sin() (in module math)
@anchor{library/math math sin}@anchor{1008}
@deffn {Function} math.sin (x)

Return the sine of @emph{x} radians.
@end deffn

@geindex tan() (in module math)
@anchor{library/math math tan}@anchor{1ecd}
@deffn {Function} math.tan (x)

Return the tangent of @emph{x} radians.
@end deffn

@node Angular conversion,Hyperbolic functions,Trigonometric functions,math — Mathematical functions
@anchor{library/math angular-conversion}@anchor{1ece}
@subsubsection Angular conversion


@geindex degrees() (in module math)
@anchor{library/math math degrees}@anchor{1ecf}
@deffn {Function} math.degrees (x)

Convert angle @emph{x} from radians to degrees.
@end deffn

@geindex radians() (in module math)
@anchor{library/math math radians}@anchor{1ed0}
@deffn {Function} math.radians (x)

Convert angle @emph{x} from degrees to radians.
@end deffn

@node Hyperbolic functions,Special functions,Angular conversion,math — Mathematical functions
@anchor{library/math hyperbolic-functions}@anchor{1ed1}
@subsubsection Hyperbolic functions


Hyperbolic functions@footnote{https://en.wikipedia.org/wiki/Hyperbolic_function}
are analogs of trigonometric functions that are based on hyperbolas
instead of circles.

@geindex acosh() (in module math)
@anchor{library/math math acosh}@anchor{f3b}
@deffn {Function} math.acosh (x)

Return the inverse hyperbolic cosine of @emph{x}.
@end deffn

@geindex asinh() (in module math)
@anchor{library/math math asinh}@anchor{f3c}
@deffn {Function} math.asinh (x)

Return the inverse hyperbolic sine of @emph{x}.
@end deffn

@geindex atanh() (in module math)
@anchor{library/math math atanh}@anchor{f3d}
@deffn {Function} math.atanh (x)

Return the inverse hyperbolic tangent of @emph{x}.
@end deffn

@geindex cosh() (in module math)
@anchor{library/math math cosh}@anchor{1ed2}
@deffn {Function} math.cosh (x)

Return the hyperbolic cosine of @emph{x}.
@end deffn

@geindex sinh() (in module math)
@anchor{library/math math sinh}@anchor{1ed3}
@deffn {Function} math.sinh (x)

Return the hyperbolic sine of @emph{x}.
@end deffn

@geindex tanh() (in module math)
@anchor{library/math math tanh}@anchor{1ed4}
@deffn {Function} math.tanh (x)

Return the hyperbolic tangent of @emph{x}.
@end deffn

@node Special functions,Constants<2>,Hyperbolic functions,math — Mathematical functions
@anchor{library/math special-functions}@anchor{1ed5}
@subsubsection Special functions


@geindex erf() (in module math)
@anchor{library/math math erf}@anchor{710}
@deffn {Function} math.erf (x)

Return the error function@footnote{https://en.wikipedia.org/wiki/Error_function} at
@emph{x}.

The @ref{710,,erf()} function can be used to compute traditional statistical
functions such as the cumulative standard normal distribution@footnote{https://en.wikipedia.org/wiki/Normal_distribution#Cumulative_distribution_function}:

@example
def phi(x):
    'Cumulative distribution function for the standard normal distribution'
    return (1.0 + erf(x / sqrt(2.0))) / 2.0
@end example


New in version 3.2.

@end deffn

@geindex erfc() (in module math)
@anchor{library/math math erfc}@anchor{711}
@deffn {Function} math.erfc (x)

Return the complementary error function at @emph{x}.  The complementary error function@footnote{https://en.wikipedia.org/wiki/Error_function} is defined as
@code{1.0 - erf(x)}.  It is used for large values of @emph{x} where a subtraction
from one would cause a loss of significance@footnote{https://en.wikipedia.org/wiki/Loss_of_significance}.


New in version 3.2.

@end deffn

@geindex gamma() (in module math)
@anchor{library/math math gamma}@anchor{d8f}
@deffn {Function} math.gamma (x)

Return the Gamma function@footnote{https://en.wikipedia.org/wiki/Gamma_function} at
@emph{x}.


New in version 3.2.

@end deffn

@geindex lgamma() (in module math)
@anchor{library/math math lgamma}@anchor{d90}
@deffn {Function} math.lgamma (x)

Return the natural logarithm of the absolute value of the Gamma
function at @emph{x}.


New in version 3.2.

@end deffn

@node Constants<2>,,Special functions,math — Mathematical functions
@anchor{library/math constants}@anchor{1ed6}
@subsubsection Constants


@geindex pi (in module math)
@anchor{library/math math pi}@anchor{1ed7}
@deffn {Data} math.pi

The mathematical constant @emph{π} = 3.141592…, to available precision.
@end deffn

@geindex e (in module math)
@anchor{library/math math e}@anchor{1ed8}
@deffn {Data} math.e

The mathematical constant @emph{e} = 2.718281…, to available precision.
@end deffn

@geindex tau (in module math)
@anchor{library/math math tau}@anchor{1ed9}
@deffn {Data} math.tau

The mathematical constant @emph{τ} = 6.283185…, to available precision.
Tau is a circle constant equal to 2@emph{π}, the ratio of a circle’s circumference to
its radius. To learn more about Tau, check out Vi Hart’s video Pi is (still) Wrong@footnote{https://www.youtube.com/watch?v=jG7vhMMXagQ}, and start celebrating
Tau day@footnote{https://tauday.com/} by eating twice as much pie!


New in version 3.6.

@end deffn

@geindex inf (in module math)
@anchor{library/math math inf}@anchor{7cb}
@deffn {Data} math.inf

A floating-point positive infinity.  (For negative infinity, use
@code{-math.inf}.)  Equivalent to the output of @code{float('inf')}.


New in version 3.5.

@end deffn

@geindex nan (in module math)
@anchor{library/math math nan}@anchor{7cc}
@deffn {Data} math.nan

A floating-point “not a number” (NaN) value.  Equivalent to the output of
@code{float('nan')}.


New in version 3.5.

@end deffn

@strong{CPython implementation detail:} The @ref{ad,,math} module consists mostly of thin wrappers around the platform C
math library functions.  Behavior in exceptional cases follows Annex F of
the C99 standard where appropriate.  The current implementation will raise
@ref{17c,,ValueError} for invalid operations like @code{sqrt(-1.0)} or @code{log(0.0)}
(where C99 Annex F recommends signaling invalid operation or divide-by-zero),
and @ref{347,,OverflowError} for results that overflow (for example,
@code{exp(1000.0)}).  A NaN will not be returned from any of the functions
above unless one or more of the input arguments was a NaN; in that case,
most functions will return a NaN, but (again following C99 Annex F) there
are some exceptions to this rule, for example @code{pow(float('nan'), 0.0)} or
@code{hypot(float('nan'), float('inf'))}.

Note that Python makes no effort to distinguish signaling NaNs from
quiet NaNs, and behavior for signaling NaNs remains unspecified.
Typical behavior is to treat all NaNs as though they were quiet.


@subsubheading See also



@table @asis

@item Module @ref{17,,cmath}

Complex number versions of many of these functions.
@end table


@node cmath — Mathematical functions for complex numbers,decimal — Decimal fixed point and floating point arithmetic,math — Mathematical functions,Numeric and Mathematical Modules
@anchor{library/cmath doc}@anchor{1eda}@anchor{library/cmath cmath-mathematical-functions-for-complex-numbers}@anchor{1edb}@anchor{library/cmath module-cmath}@anchor{17}
@subsection @code{cmath} — Mathematical functions for complex numbers


@geindex module; cmath


__________________________________________________________________


This module provides access to mathematical functions for complex numbers.  The
functions in this module accept integers, floating-point numbers or complex
numbers as arguments. They will also accept any Python object that has either a
@code{__complex__()} or a @code{__float__()} method: these methods are used to
convert the object to a complex or floating-point number, respectively, and
the function is then applied to the result of the conversion.

@cartouche
@quotation Note 
On platforms with hardware and system-level support for signed
zeros, functions involving branch cuts are continuous on @emph{both}
sides of the branch cut: the sign of the zero distinguishes one
side of the branch cut from the other.  On platforms that do not
support signed zeros the continuity is as specified below.
@end quotation
@end cartouche

@menu
* Conversions to and from polar coordinates:: 
* Power and logarithmic functions: Power and logarithmic functions<2>. 
* Trigonometric functions: Trigonometric functions<2>. 
* Hyperbolic functions: Hyperbolic functions<2>. 
* Classification functions:: 
* Constants: Constants<3>. 

@end menu

@node Conversions to and from polar coordinates,Power and logarithmic functions<2>,,cmath — Mathematical functions for complex numbers
@anchor{library/cmath conversions-to-and-from-polar-coordinates}@anchor{1edc}
@subsubsection Conversions to and from polar coordinates


A Python complex number @code{z} is stored internally using @emph{rectangular}
or @emph{Cartesian} coordinates.  It is completely determined by its @emph{real
part} @code{z.real} and its @emph{imaginary part} @code{z.imag}.  In other
words:

@example
z == z.real + z.imag*1j
@end example

@emph{Polar coordinates} give an alternative way to represent a complex
number.  In polar coordinates, a complex number @emph{z} is defined by the
modulus @emph{r} and the phase angle @emph{phi}. The modulus @emph{r} is the distance
from @emph{z} to the origin, while the phase @emph{phi} is the counterclockwise
angle, measured in radians, from the positive x-axis to the line
segment that joins the origin to @emph{z}.

The following functions can be used to convert from the native
rectangular coordinates to polar coordinates and back.

@geindex phase() (in module cmath)
@anchor{library/cmath cmath phase}@anchor{1edd}
@deffn {Function} cmath.phase (x)

Return the phase of @emph{x} (also known as the @emph{argument} of @emph{x}), as a
float.  @code{phase(x)} is equivalent to @code{math.atan2(x.imag,
x.real)}.  The result lies in the range [-@emph{π}, @emph{π}], and the branch
cut for this operation lies along the negative real axis,
continuous from above.  On systems with support for signed zeros
(which includes most systems in current use), this means that the
sign of the result is the same as the sign of @code{x.imag}, even when
@code{x.imag} is zero:

@example
>>> phase(complex(-1.0, 0.0))
3.141592653589793
>>> phase(complex(-1.0, -0.0))
-3.141592653589793
@end example
@end deffn

@cartouche
@quotation Note 
The modulus (absolute value) of a complex number @emph{x} can be
computed using the built-in @ref{1582,,abs()} function.  There is no
separate @ref{17,,cmath} module function for this operation.
@end quotation
@end cartouche

@geindex polar() (in module cmath)
@anchor{library/cmath cmath polar}@anchor{1ede}
@deffn {Function} cmath.polar (x)

Return the representation of @emph{x} in polar coordinates.  Returns a
pair @code{(r, phi)} where @emph{r} is the modulus of @emph{x} and phi is the
phase of @emph{x}.  @code{polar(x)} is equivalent to @code{(abs(x),
phase(x))}.
@end deffn

@geindex rect() (in module cmath)
@anchor{library/cmath cmath rect}@anchor{1edf}
@deffn {Function} cmath.rect (r, phi)

Return the complex number @emph{x} with polar coordinates @emph{r} and @emph{phi}.
Equivalent to @code{r * (math.cos(phi) + math.sin(phi)*1j)}.
@end deffn

@node Power and logarithmic functions<2>,Trigonometric functions<2>,Conversions to and from polar coordinates,cmath — Mathematical functions for complex numbers
@anchor{library/cmath power-and-logarithmic-functions}@anchor{1ee0}
@subsubsection Power and logarithmic functions


@geindex exp() (in module cmath)
@anchor{library/cmath cmath exp}@anchor{1ee1}
@deffn {Function} cmath.exp (x)

Return @emph{e} raised to the power @emph{x}, where @emph{e} is the base of natural
logarithms.
@end deffn

@geindex log() (in module cmath)
@anchor{library/cmath cmath log}@anchor{1ee2}
@deffn {Function} cmath.log (x[, base])

Returns the logarithm of @emph{x} to the given @emph{base}. If the @emph{base} is not
specified, returns the natural logarithm of @emph{x}. There is one branch cut, from 0
along the negative real axis to -∞, continuous from above.
@end deffn

@geindex log10() (in module cmath)
@anchor{library/cmath cmath log10}@anchor{1ee3}
@deffn {Function} cmath.log10 (x)

Return the base-10 logarithm of @emph{x}. This has the same branch cut as
@ref{1ee2,,log()}.
@end deffn

@geindex sqrt() (in module cmath)
@anchor{library/cmath cmath sqrt}@anchor{1ee4}
@deffn {Function} cmath.sqrt (x)

Return the square root of @emph{x}. This has the same branch cut as @ref{1ee2,,log()}.
@end deffn

@node Trigonometric functions<2>,Hyperbolic functions<2>,Power and logarithmic functions<2>,cmath — Mathematical functions for complex numbers
@anchor{library/cmath trigonometric-functions}@anchor{1ee5}
@subsubsection Trigonometric functions


@geindex acos() (in module cmath)
@anchor{library/cmath cmath acos}@anchor{1ee6}
@deffn {Function} cmath.acos (x)

Return the arc cosine of @emph{x}. There are two branch cuts: One extends right from
1 along the real axis to ∞, continuous from below. The other extends left from
-1 along the real axis to -∞, continuous from above.
@end deffn

@geindex asin() (in module cmath)
@anchor{library/cmath cmath asin}@anchor{1ee7}
@deffn {Function} cmath.asin (x)

Return the arc sine of @emph{x}. This has the same branch cuts as @ref{1ee6,,acos()}.
@end deffn

@geindex atan() (in module cmath)
@anchor{library/cmath cmath atan}@anchor{1ee8}
@deffn {Function} cmath.atan (x)

Return the arc tangent of @emph{x}. There are two branch cuts: One extends from
@code{1j} along the imaginary axis to @code{∞j}, continuous from the right. The
other extends from @code{-1j} along the imaginary axis to @code{-∞j}, continuous
from the left.
@end deffn

@geindex cos() (in module cmath)
@anchor{library/cmath cmath cos}@anchor{1ee9}
@deffn {Function} cmath.cos (x)

Return the cosine of @emph{x}.
@end deffn

@geindex sin() (in module cmath)
@anchor{library/cmath cmath sin}@anchor{1eea}
@deffn {Function} cmath.sin (x)

Return the sine of @emph{x}.
@end deffn

@geindex tan() (in module cmath)
@anchor{library/cmath cmath tan}@anchor{1eeb}
@deffn {Function} cmath.tan (x)

Return the tangent of @emph{x}.
@end deffn

@node Hyperbolic functions<2>,Classification functions,Trigonometric functions<2>,cmath — Mathematical functions for complex numbers
@anchor{library/cmath hyperbolic-functions}@anchor{1eec}
@subsubsection Hyperbolic functions


@geindex acosh() (in module cmath)
@anchor{library/cmath cmath acosh}@anchor{1eed}
@deffn {Function} cmath.acosh (x)

Return the inverse hyperbolic cosine of @emph{x}. There is one branch cut,
extending left from 1 along the real axis to -∞, continuous from above.
@end deffn

@geindex asinh() (in module cmath)
@anchor{library/cmath cmath asinh}@anchor{1eee}
@deffn {Function} cmath.asinh (x)

Return the inverse hyperbolic sine of @emph{x}. There are two branch cuts:
One extends from @code{1j} along the imaginary axis to @code{∞j},
continuous from the right.  The other extends from @code{-1j} along
the imaginary axis to @code{-∞j}, continuous from the left.
@end deffn

@geindex atanh() (in module cmath)
@anchor{library/cmath cmath atanh}@anchor{1eef}
@deffn {Function} cmath.atanh (x)

Return the inverse hyperbolic tangent of @emph{x}. There are two branch cuts: One
extends from @code{1} along the real axis to @code{∞}, continuous from below. The
other extends from @code{-1} along the real axis to @code{-∞}, continuous from
above.
@end deffn

@geindex cosh() (in module cmath)
@anchor{library/cmath cmath cosh}@anchor{1ef0}
@deffn {Function} cmath.cosh (x)

Return the hyperbolic cosine of @emph{x}.
@end deffn

@geindex sinh() (in module cmath)
@anchor{library/cmath cmath sinh}@anchor{1ef1}
@deffn {Function} cmath.sinh (x)

Return the hyperbolic sine of @emph{x}.
@end deffn

@geindex tanh() (in module cmath)
@anchor{library/cmath cmath tanh}@anchor{1ef2}
@deffn {Function} cmath.tanh (x)

Return the hyperbolic tangent of @emph{x}.
@end deffn

@node Classification functions,Constants<3>,Hyperbolic functions<2>,cmath — Mathematical functions for complex numbers
@anchor{library/cmath classification-functions}@anchor{1ef3}
@subsubsection Classification functions


@geindex isfinite() (in module cmath)
@anchor{library/cmath cmath isfinite}@anchor{1ef4}
@deffn {Function} cmath.isfinite (x)

Return @code{True} if both the real and imaginary parts of @emph{x} are finite, and
@code{False} otherwise.


New in version 3.2.

@end deffn

@geindex isinf() (in module cmath)
@anchor{library/cmath cmath isinf}@anchor{1ef5}
@deffn {Function} cmath.isinf (x)

Return @code{True} if either the real or the imaginary part of @emph{x} is an
infinity, and @code{False} otherwise.
@end deffn

@geindex isnan() (in module cmath)
@anchor{library/cmath cmath isnan}@anchor{1ef6}
@deffn {Function} cmath.isnan (x)

Return @code{True} if either the real or the imaginary part of @emph{x} is a NaN,
and @code{False} otherwise.
@end deffn

@geindex isclose() (in module cmath)
@anchor{library/cmath cmath isclose}@anchor{90e}
@deffn {Function} cmath.isclose (a, b, *, rel_tol=1e-09, abs_tol=0.0)

Return @code{True} if the values @emph{a} and @emph{b} are close to each other and
@code{False} otherwise.

Whether or not two values are considered close is determined according to
given absolute and relative tolerances.

@emph{rel_tol} is the relative tolerance – it is the maximum allowed difference
between @emph{a} and @emph{b}, relative to the larger absolute value of @emph{a} or @emph{b}.
For example, to set a tolerance of 5%, pass @code{rel_tol=0.05}.  The default
tolerance is @code{1e-09}, which assures that the two values are the same
within about 9 decimal digits.  @emph{rel_tol} must be greater than zero.

@emph{abs_tol} is the minimum absolute tolerance – useful for comparisons near
zero. @emph{abs_tol} must be at least zero.

If no errors occur, the result will be:
@code{abs(a-b) <= max(rel_tol * max(abs(a), abs(b)), abs_tol)}.

The IEEE 754 special values of @code{NaN}, @code{inf}, and @code{-inf} will be
handled according to IEEE rules.  Specifically, @code{NaN} is not considered
close to any other value, including @code{NaN}.  @code{inf} and @code{-inf} are only
considered close to themselves.


New in version 3.5.



@subsubheading See also


@geindex Python Enhancement Proposals; PEP 485
PEP 485@footnote{https://www.python.org/dev/peps/pep-0485} – A function for testing approximate equality

@end deffn

@node Constants<3>,,Classification functions,cmath — Mathematical functions for complex numbers
@anchor{library/cmath constants}@anchor{1ef7}
@subsubsection Constants


@geindex pi (in module cmath)
@anchor{library/cmath cmath pi}@anchor{1ef8}
@deffn {Data} cmath.pi

The mathematical constant @emph{π}, as a float.
@end deffn

@geindex e (in module cmath)
@anchor{library/cmath cmath e}@anchor{1ef9}
@deffn {Data} cmath.e

The mathematical constant @emph{e}, as a float.
@end deffn

@geindex tau (in module cmath)
@anchor{library/cmath cmath tau}@anchor{7c8}
@deffn {Data} cmath.tau

The mathematical constant @emph{τ}, as a float.


New in version 3.6.

@end deffn

@geindex inf (in module cmath)
@anchor{library/cmath cmath inf}@anchor{7c9}
@deffn {Data} cmath.inf

Floating-point positive infinity. Equivalent to @code{float('inf')}.


New in version 3.6.

@end deffn

@geindex infj (in module cmath)
@anchor{library/cmath cmath infj}@anchor{7cd}
@deffn {Data} cmath.infj

Complex number with zero real part and positive infinity imaginary
part. Equivalent to @code{complex(0.0, float('inf'))}.


New in version 3.6.

@end deffn

@geindex nan (in module cmath)
@anchor{library/cmath cmath nan}@anchor{7ca}
@deffn {Data} cmath.nan

A floating-point “not a number” (NaN) value.  Equivalent to
@code{float('nan')}.


New in version 3.6.

@end deffn

@geindex nanj (in module cmath)
@anchor{library/cmath cmath nanj}@anchor{7ce}
@deffn {Data} cmath.nanj

Complex number with zero real part and NaN imaginary part. Equivalent to
@code{complex(0.0, float('nan'))}.


New in version 3.6.

@end deffn

@geindex module; math

Note that the selection of functions is similar, but not identical, to that in
module @ref{ad,,math}.  The reason for having two modules is that some users aren’t
interested in complex numbers, and perhaps don’t even know what they are.  They
would rather have @code{math.sqrt(-1)} raise an exception than return a complex
number. Also note that the functions defined in @ref{17,,cmath} always return a
complex number, even if the answer can be expressed as a real number (in which
case the complex number has an imaginary part of zero).

A note on branch cuts: They are curves along which the given function fails to
be continuous.  They are a necessary feature of many complex functions.  It is
assumed that if you need to compute with complex functions, you will understand
about branch cuts.  Consult almost any (not too elementary) book on complex
variables for enlightenment.  For information of the proper choice of branch
cuts for numerical purposes, a good reference should be the following:


@subsubheading See also


Kahan, W:  Branch cuts for complex elementary functions; or, Much ado about
nothing’s sign bit.  In Iserles, A., and Powell, M. (eds.), The state of the art
in numerical analysis. Clarendon Press (1987) pp165–211.


@node decimal — Decimal fixed point and floating point arithmetic,fractions — Rational numbers,cmath — Mathematical functions for complex numbers,Numeric and Mathematical Modules
@anchor{library/decimal doc}@anchor{1efa}@anchor{library/decimal decimal-decimal-fixed-point-and-floating-point-arithmetic}@anchor{1efb}@anchor{library/decimal module-decimal}@anchor{34}
@subsection @code{decimal} — Decimal fixed point and floating point arithmetic


@geindex module; decimal

@strong{Source code:} Lib/decimal.py@footnote{https://github.com/python/cpython/tree/main/Lib/decimal.py}

@c import modules for testing inline doctests with the Sphinx doctest builder

@c import decimal
@c import math
@c from decimal import *
@c # make sure each group gets a fresh context
@c setcontext(Context())

@c # make sure other tests (outside this file) get a fresh context
@c setcontext(Context())


__________________________________________________________________


The @ref{34,,decimal} module provides support for fast correctly-rounded
decimal floating point arithmetic. It offers several advantages over the
@ref{256,,float} datatype:


@itemize *

@item 
Decimal “is based on a floating-point model which was designed with people
in mind, and necessarily has a paramount guiding principle – computers must
provide an arithmetic that works in the same way as the arithmetic that
people learn at school.” – excerpt from the decimal arithmetic specification.

@item 
Decimal numbers can be represented exactly.  In contrast, numbers like
@code{1.1} and @code{2.2} do not have exact representations in binary
floating point. End users typically would not expect @code{1.1 + 2.2} to display
as @code{3.3000000000000003} as it does with binary floating point.

@item 
The exactness carries over into arithmetic.  In decimal floating point, @code{0.1
+ 0.1 + 0.1 - 0.3} is exactly equal to zero.  In binary floating point, the result
is @code{5.5511151231257827e-017}.  While near to zero, the differences
prevent reliable equality testing and differences can accumulate. For this
reason, decimal is preferred in accounting applications which have strict
equality invariants.

@item 
The decimal module incorporates a notion of significant places so that @code{1.30
+ 1.20} is @code{2.50}.  The trailing zero is kept to indicate significance.
This is the customary presentation for monetary applications. For
multiplication, the “schoolbook” approach uses all the figures in the
multiplicands.  For instance, @code{1.3 * 1.2} gives @code{1.56} while @code{1.30 *
1.20} gives @code{1.5600}.

@item 
Unlike hardware based binary floating point, the decimal module has a user
alterable precision (defaulting to 28 places) which can be as large as needed for
a given problem:

@example
>>> from decimal import *
>>> getcontext().prec = 6
>>> Decimal(1) / Decimal(7)
Decimal('0.142857')
>>> getcontext().prec = 28
>>> Decimal(1) / Decimal(7)
Decimal('0.1428571428571428571428571429')
@end example

@item 
Both binary and decimal floating point are implemented in terms of published
standards.  While the built-in float type exposes only a modest portion of its
capabilities, the decimal module exposes all required parts of the standard.
When needed, the programmer has full control over rounding and signal handling.
This includes an option to enforce exact arithmetic by using exceptions
to block any inexact operations.

@item 
The decimal module was designed to support “without prejudice, both exact
unrounded decimal arithmetic (sometimes called fixed-point arithmetic)
and rounded floating-point arithmetic.”  – excerpt from the decimal
arithmetic specification.
@end itemize

The module design is centered around three concepts:  the decimal number, the
context for arithmetic, and signals.

A decimal number is immutable.  It has a sign, coefficient digits, and an
exponent.  To preserve significance, the coefficient digits do not truncate
trailing zeros.  Decimals also include special values such as
@code{Infinity}, @code{-Infinity}, and @code{NaN}.  The standard also
differentiates @code{-0} from @code{+0}.

The context for arithmetic is an environment specifying precision, rounding
rules, limits on exponents, flags indicating the results of operations, and trap
enablers which determine whether signals are treated as exceptions.  Rounding
options include @ref{1efc,,ROUND_CEILING}, @ref{1efd,,ROUND_DOWN},
@ref{1efe,,ROUND_FLOOR}, @ref{1eff,,ROUND_HALF_DOWN}, @ref{1f00,,ROUND_HALF_EVEN},
@ref{1f01,,ROUND_HALF_UP}, @ref{1f02,,ROUND_UP}, and @ref{1f03,,ROUND_05UP}.

Signals are groups of exceptional conditions arising during the course of
computation.  Depending on the needs of the application, signals may be ignored,
considered as informational, or treated as exceptions. The signals in the
decimal module are: @ref{1f04,,Clamped}, @ref{c3b,,InvalidOperation},
@ref{1f05,,DivisionByZero}, @ref{1f06,,Inexact}, @ref{1f07,,Rounded}, @ref{1f08,,Subnormal},
@ref{1f09,,Overflow}, @ref{1f0a,,Underflow} and @ref{c35,,FloatOperation}.

For each signal there is a flag and a trap enabler.  When a signal is
encountered, its flag is set to one, then, if the trap enabler is
set to one, an exception is raised.  Flags are sticky, so the user needs to
reset them before monitoring a calculation.


@subsubheading See also



@itemize *

@item 
IBM’s General Decimal Arithmetic Specification, The General Decimal Arithmetic Specification@footnote{http://speleotrove.com/decimal/decarith.html}.
@end itemize


@c %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

@menu
* Quick-start Tutorial:: 
* Decimal objects:: 
* Context objects:: 
* Constants: Constants<4>. 
* Rounding modes:: 
* Signals:: 
* Floating Point Notes:: 
* Working with threads:: 
* Recipes:: 
* Decimal FAQ:: 

@end menu

@node Quick-start Tutorial,Decimal objects,,decimal — Decimal fixed point and floating point arithmetic
@anchor{library/decimal decimal-tutorial}@anchor{1f0b}@anchor{library/decimal quick-start-tutorial}@anchor{1f0c}
@subsubsection Quick-start Tutorial


The usual start to using decimals is importing the module, viewing the current
context with @ref{1f0d,,getcontext()} and, if necessary, setting new values for
precision, rounding, or enabled traps:

@example
>>> from decimal import *
>>> getcontext()
Context(prec=28, rounding=ROUND_HALF_EVEN, Emin=-999999, Emax=999999,
        capitals=1, clamp=0, flags=[], traps=[Overflow, DivisionByZero,
        InvalidOperation])

>>> getcontext().prec = 7       # Set a new precision
@end example

Decimal instances can be constructed from integers, strings, floats, or tuples.
Construction from an integer or a float performs an exact conversion of the
value of that integer or float.  Decimal numbers include special values such as
@code{NaN} which stands for “Not a number”, positive and negative
@code{Infinity}, and @code{-0}:

@example
>>> getcontext().prec = 28
>>> Decimal(10)
Decimal('10')
>>> Decimal('3.14')
Decimal('3.14')
>>> Decimal(3.14)
Decimal('3.140000000000000124344978758017532527446746826171875')
>>> Decimal((0, (3, 1, 4), -2))
Decimal('3.14')
>>> Decimal(str(2.0 ** 0.5))
Decimal('1.4142135623730951')
>>> Decimal(2) ** Decimal('0.5')
Decimal('1.414213562373095048801688724')
>>> Decimal('NaN')
Decimal('NaN')
>>> Decimal('-Infinity')
Decimal('-Infinity')
@end example

If the @ref{c35,,FloatOperation} signal is trapped, accidental mixing of
decimals and floats in constructors or ordering comparisons raises
an exception:

@example
>>> c = getcontext()
>>> c.traps[FloatOperation] = True
>>> Decimal(3.14)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
decimal.FloatOperation: [<class 'decimal.FloatOperation'>]
>>> Decimal('3.5') < 3.7
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
decimal.FloatOperation: [<class 'decimal.FloatOperation'>]
>>> Decimal('3.5') == 3.5
True
@end example


New in version 3.3.


The significance of a new Decimal is determined solely by the number of digits
input.  Context precision and rounding only come into play during arithmetic
operations.

@example
>>> getcontext().prec = 6
>>> Decimal('3.0')
Decimal('3.0')
>>> Decimal('3.1415926535')
Decimal('3.1415926535')
>>> Decimal('3.1415926535') + Decimal('2.7182818285')
Decimal('5.85987')
>>> getcontext().rounding = ROUND_UP
>>> Decimal('3.1415926535') + Decimal('2.7182818285')
Decimal('5.85988')
@end example

If the internal limits of the C version are exceeded, constructing
a decimal raises @ref{c3b,,InvalidOperation}:

@example
>>> Decimal("1e9999999999999999999")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
decimal.InvalidOperation: [<class 'decimal.InvalidOperation'>]
@end example


Changed in version 3.3.


Decimals interact well with much of the rest of Python.  Here is a small decimal
floating point flying circus:

@example
>>> data = list(map(Decimal, '1.34 1.87 3.45 2.35 1.00 0.03 9.25'.split()))
>>> max(data)
Decimal('9.25')
>>> min(data)
Decimal('0.03')
>>> sorted(data)
[Decimal('0.03'), Decimal('1.00'), Decimal('1.34'), Decimal('1.87'),
 Decimal('2.35'), Decimal('3.45'), Decimal('9.25')]
>>> sum(data)
Decimal('19.29')
>>> a,b,c = data[:3]
>>> str(a)
'1.34'
>>> float(a)
1.34
>>> round(a, 1)
Decimal('1.3')
>>> int(a)
1
>>> a * 5
Decimal('6.70')
>>> a * b
Decimal('2.5058')
>>> c % a
Decimal('0.77')
@end example

And some mathematical functions are also available to Decimal:

@example
>>> getcontext().prec = 28
>>> Decimal(2).sqrt()
Decimal('1.414213562373095048801688724')
>>> Decimal(1).exp()
Decimal('2.718281828459045235360287471')
>>> Decimal('10').ln()
Decimal('2.302585092994045684017991455')
>>> Decimal('10').log10()
Decimal('1')
@end example

The @code{quantize()} method rounds a number to a fixed exponent.  This method is
useful for monetary applications that often round results to a fixed number of
places:

@example
>>> Decimal('7.325').quantize(Decimal('.01'), rounding=ROUND_DOWN)
Decimal('7.32')
>>> Decimal('7.325').quantize(Decimal('1.'), rounding=ROUND_UP)
Decimal('8')
@end example

As shown above, the @ref{1f0d,,getcontext()} function accesses the current context and
allows the settings to be changed.  This approach meets the needs of most
applications.

For more advanced work, it may be useful to create alternate contexts using the
Context() constructor.  To make an alternate active, use the @ref{1f0e,,setcontext()}
function.

In accordance with the standard, the @ref{34,,decimal} module provides two ready to
use standard contexts, @ref{c39,,BasicContext} and @ref{c3a,,ExtendedContext}. The
former is especially useful for debugging because many of the traps are
enabled:

@example
>>> myothercontext = Context(prec=60, rounding=ROUND_HALF_DOWN)
>>> setcontext(myothercontext)
>>> Decimal(1) / Decimal(7)
Decimal('0.142857142857142857142857142857142857142857142857142857142857')

>>> ExtendedContext
Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999, Emax=999999,
        capitals=1, clamp=0, flags=[], traps=[])
>>> setcontext(ExtendedContext)
>>> Decimal(1) / Decimal(7)
Decimal('0.142857143')
>>> Decimal(42) / Decimal(0)
Decimal('Infinity')

>>> setcontext(BasicContext)
>>> Decimal(42) / Decimal(0)
Traceback (most recent call last):
  File "<pyshell#143>", line 1, in -toplevel-
    Decimal(42) / Decimal(0)
DivisionByZero: x / 0
@end example

Contexts also have signal flags for monitoring exceptional conditions
encountered during computations.  The flags remain set until explicitly cleared,
so it is best to clear the flags before each set of monitored computations by
using the @code{clear_flags()} method.

@example
>>> setcontext(ExtendedContext)
>>> getcontext().clear_flags()
>>> Decimal(355) / Decimal(113)
Decimal('3.14159292')
>>> getcontext()
Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999, Emax=999999,
        capitals=1, clamp=0, flags=[Inexact, Rounded], traps=[])
@end example

The @emph{flags} entry shows that the rational approximation to @code{Pi} was
rounded (digits beyond the context precision were thrown away) and that the
result is inexact (some of the discarded digits were non-zero).

Individual traps are set using the dictionary in the @code{traps} field of a
context:

@example
>>> setcontext(ExtendedContext)
>>> Decimal(1) / Decimal(0)
Decimal('Infinity')
>>> getcontext().traps[DivisionByZero] = 1
>>> Decimal(1) / Decimal(0)
Traceback (most recent call last):
  File "<pyshell#112>", line 1, in -toplevel-
    Decimal(1) / Decimal(0)
DivisionByZero: x / 0
@end example

Most programs adjust the current context only once, at the beginning of the
program.  And, in many applications, data is converted to @ref{248,,Decimal} with
a single cast inside a loop.  With context set and decimals created, the bulk of
the program manipulates the data no differently than with other Python numeric
types.

@c %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

@node Decimal objects,Context objects,Quick-start Tutorial,decimal — Decimal fixed point and floating point arithmetic
@anchor{library/decimal decimal-decimal}@anchor{1f0f}@anchor{library/decimal decimal-objects}@anchor{1f10}
@subsubsection Decimal objects


@geindex Decimal (class in decimal)
@anchor{library/decimal decimal Decimal}@anchor{248}
@deffn {Class} decimal.Decimal (value='0', context=None)

Construct a new @ref{248,,Decimal} object based from @emph{value}.

@emph{value} can be an integer, string, tuple, @ref{256,,float}, or another @ref{248,,Decimal}
object. If no @emph{value} is given, returns @code{Decimal('0')}.  If @emph{value} is a
string, it should conform to the decimal numeric string syntax after leading
and trailing whitespace characters, as well as underscores throughout, are removed:

@example
sign           ::=  '+' | '-'
digit          ::=  '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9'
indicator      ::=  'e' | 'E'
digits         ::=  digit [digit]...
decimal-part   ::=  digits '.' [digits] | ['.'] digits
exponent-part  ::=  indicator [sign] digits
infinity       ::=  'Infinity' | 'Inf'
nan            ::=  'NaN' [digits] | 'sNaN' [digits]
numeric-value  ::=  decimal-part [exponent-part] | infinity
numeric-string ::=  [sign] numeric-value | [sign] nan
@end example

Other Unicode decimal digits are also permitted where @code{digit}
appears above.  These include decimal digits from various other
alphabets (for example, Arabic-Indic and Devanāgarī digits) along
with the fullwidth digits @code{'\uff10'} through @code{'\uff19'}.

If @emph{value} is a @ref{3e5,,tuple}, it should have three components, a sign
(@code{0} for positive or @code{1} for negative), a @ref{3e5,,tuple} of
digits, and an integer exponent. For example, @code{Decimal((0, (1, 4, 1, 4), -3))}
returns @code{Decimal('1.414')}.

If @emph{value} is a @ref{256,,float}, the binary floating point value is losslessly
converted to its exact decimal equivalent.  This conversion can often require
53 or more digits of precision.  For example, @code{Decimal(float('1.1'))}
converts to
@code{Decimal('1.100000000000000088817841970012523233890533447265625')}.

The @emph{context} precision does not affect how many digits are stored. That is
determined exclusively by the number of digits in @emph{value}. For example,
@code{Decimal('3.00000')} records all five zeros even if the context precision is
only three.

The purpose of the @emph{context} argument is determining what to do if @emph{value} is a
malformed string.  If the context traps @ref{c3b,,InvalidOperation}, an exception
is raised; otherwise, the constructor returns a new Decimal with the value of
@code{NaN}.

Once constructed, @ref{248,,Decimal} objects are immutable.


Changed in version 3.2: The argument to the constructor is now permitted to be a @ref{256,,float}
instance.



Changed in version 3.3: @ref{256,,float} arguments raise an exception if the @ref{c35,,FloatOperation}
trap is set. By default the trap is off.



Changed in version 3.6: Underscores are allowed for grouping, as with integral and floating-point
literals in code.


Decimal floating point objects share many properties with the other built-in
numeric types such as @ref{256,,float} and @ref{1b1,,int}.  All of the usual math
operations and special methods apply.  Likewise, decimal objects can be
copied, pickled, printed, used as dictionary keys, used as set elements,
compared, sorted, and coerced to another type (such as @ref{256,,float} or
@ref{1b1,,int}).

There are some small differences between arithmetic on Decimal objects and
arithmetic on integers and floats.  When the remainder operator @code{%} is
applied to Decimal objects, the sign of the result is the sign of the
@emph{dividend} rather than the sign of the divisor:

@example
>>> (-7) % 4
1
>>> Decimal(-7) % Decimal(4)
Decimal('-3')
@end example

The integer division operator @code{//} behaves analogously, returning the
integer part of the true quotient (truncating towards zero) rather than its
floor, so as to preserve the usual identity @code{x == (x // y) * y + x % y}:

@example
>>> -7 // 4
-2
>>> Decimal(-7) // Decimal(4)
Decimal('-1')
@end example

The @code{%} and @code{//} operators implement the @code{remainder} and
@code{divide-integer} operations (respectively) as described in the
specification.

Decimal objects cannot generally be combined with floats or
instances of @ref{172,,fractions.Fraction} in arithmetic operations:
an attempt to add a @ref{248,,Decimal} to a @ref{256,,float}, for
example, will raise a @ref{162,,TypeError}.  However, it is possible to
use Python’s comparison operators to compare a @ref{248,,Decimal}
instance @code{x} with another number @code{y}.  This avoids confusing results
when doing equality comparisons between numbers of different types.


Changed in version 3.2: Mixed-type comparisons between @ref{248,,Decimal} instances and other
numeric types are now fully supported.


In addition to the standard numeric properties, decimal floating point
objects also have a number of specialized methods:

@geindex adjusted() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal adjusted}@anchor{1f11}
@deffn {Method} adjusted ()

Return the adjusted exponent after shifting out the coefficient’s
rightmost digits until only the lead digit remains:
@code{Decimal('321e+5').adjusted()} returns seven.  Used for determining the
position of the most significant digit with respect to the decimal point.
@end deffn

@geindex as_integer_ratio() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal as_integer_ratio}@anchor{7dd}
@deffn {Method} as_integer_ratio ()

Return a pair @code{(n, d)} of integers that represent the given
@ref{248,,Decimal} instance as a fraction, in lowest terms and
with a positive denominator:

@example
>>> Decimal('-3.14').as_integer_ratio()
(-157, 50)
@end example

The conversion is exact.  Raise OverflowError on infinities and ValueError
on NaNs.
@end deffn


New in version 3.6.


@geindex as_tuple() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal as_tuple}@anchor{1f12}
@deffn {Method} as_tuple ()

Return a @ref{ce9,,named tuple} representation of the number:
@code{DecimalTuple(sign, digits, exponent)}.
@end deffn

@geindex canonical() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal canonical}@anchor{1f13}
@deffn {Method} canonical ()

Return the canonical encoding of the argument.  Currently, the encoding of
a @ref{248,,Decimal} instance is always canonical, so this operation returns
its argument unchanged.
@end deffn

@geindex compare() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal compare}@anchor{1f14}
@deffn {Method} compare (other, context=None)

Compare the values of two Decimal instances.  @ref{1f14,,compare()} returns a
Decimal instance, and if either operand is a NaN then the result is a
NaN:

@example
a or b is a NaN  ==> Decimal('NaN')
a < b            ==> Decimal('-1')
a == b           ==> Decimal('0')
a > b            ==> Decimal('1')
@end example
@end deffn

@geindex compare_signal() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal compare_signal}@anchor{1f15}
@deffn {Method} compare_signal (other, context=None)

This operation is identical to the @ref{1f14,,compare()} method, except that all
NaNs signal.  That is, if neither operand is a signaling NaN then any
quiet NaN operand is treated as though it were a signaling NaN.
@end deffn

@geindex compare_total() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal compare_total}@anchor{1f16}
@deffn {Method} compare_total (other, context=None)

Compare two operands using their abstract representation rather than their
numerical value.  Similar to the @ref{1f14,,compare()} method, but the result
gives a total ordering on @ref{248,,Decimal} instances.  Two
@ref{248,,Decimal} instances with the same numeric value but different
representations compare unequal in this ordering:

@example
>>> Decimal('12.0').compare_total(Decimal('12'))
Decimal('-1')
@end example

Quiet and signaling NaNs are also included in the total ordering.  The
result of this function is @code{Decimal('0')} if both operands have the same
representation, @code{Decimal('-1')} if the first operand is lower in the
total order than the second, and @code{Decimal('1')} if the first operand is
higher in the total order than the second operand.  See the specification
for details of the total order.

This operation is unaffected by context and is quiet: no flags are changed
and no rounding is performed.  As an exception, the C version may raise
InvalidOperation if the second operand cannot be converted exactly.
@end deffn

@geindex compare_total_mag() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal compare_total_mag}@anchor{1f17}
@deffn {Method} compare_total_mag (other, context=None)

Compare two operands using their abstract representation rather than their
value as in @ref{1f16,,compare_total()}, but ignoring the sign of each operand.
@code{x.compare_total_mag(y)} is equivalent to
@code{x.copy_abs().compare_total(y.copy_abs())}.

This operation is unaffected by context and is quiet: no flags are changed
and no rounding is performed.  As an exception, the C version may raise
InvalidOperation if the second operand cannot be converted exactly.
@end deffn

@geindex conjugate() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal conjugate}@anchor{1f18}
@deffn {Method} conjugate ()

Just returns self, this method is only to comply with the Decimal
Specification.
@end deffn

@geindex copy_abs() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal copy_abs}@anchor{1f19}
@deffn {Method} copy_abs ()

Return the absolute value of the argument.  This operation is unaffected
by the context and is quiet: no flags are changed and no rounding is
performed.
@end deffn

@geindex copy_negate() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal copy_negate}@anchor{1f1a}
@deffn {Method} copy_negate ()

Return the negation of the argument.  This operation is unaffected by the
context and is quiet: no flags are changed and no rounding is performed.
@end deffn

@geindex copy_sign() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal copy_sign}@anchor{1f1b}
@deffn {Method} copy_sign (other, context=None)

Return a copy of the first operand with the sign set to be the same as the
sign of the second operand.  For example:

@example
>>> Decimal('2.3').copy_sign(Decimal('-1.5'))
Decimal('-2.3')
@end example

This operation is unaffected by context and is quiet: no flags are changed
and no rounding is performed.  As an exception, the C version may raise
InvalidOperation if the second operand cannot be converted exactly.
@end deffn

@geindex exp() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal exp}@anchor{c3d}
@deffn {Method} exp (context=None)

Return the value of the (natural) exponential function @code{e**x} at the
given number.  The result is correctly rounded using the
@ref{1f00,,ROUND_HALF_EVEN} rounding mode.

@example
>>> Decimal(1).exp()
Decimal('2.718281828459045235360287471')
>>> Decimal(321).exp()
Decimal('2.561702493119680037517373933E+139')
@end example
@end deffn

@geindex from_float() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal from_float}@anchor{da5}
@deffn {Method} from_float (f)

Classmethod that converts a float to a decimal number, exactly.

Note @cite{Decimal.from_float(0.1)} is not the same as @cite{Decimal(‘0.1’)}.
Since 0.1 is not exactly representable in binary floating point, the
value is stored as the nearest representable value which is
@cite{0x1.999999999999ap-4}.  That equivalent value in decimal is
@cite{0.1000000000000000055511151231257827021181583404541015625}.

@cartouche
@quotation Note 
From Python 3.2 onwards, a @ref{248,,Decimal} instance
can also be constructed directly from a @ref{256,,float}.
@end quotation
@end cartouche

@example
>>> Decimal.from_float(0.1)
Decimal('0.1000000000000000055511151231257827021181583404541015625')
>>> Decimal.from_float(float('nan'))
Decimal('NaN')
>>> Decimal.from_float(float('inf'))
Decimal('Infinity')
>>> Decimal.from_float(float('-inf'))
Decimal('-Infinity')
@end example


New in version 3.1.

@end deffn

@geindex fma() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal fma}@anchor{1f1c}
@deffn {Method} fma (other, third, context=None)

Fused multiply-add.  Return self*other+third with no rounding of the
intermediate product self*other.

@example
>>> Decimal(2).fma(3, 5)
Decimal('11')
@end example
@end deffn

@geindex is_canonical() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal is_canonical}@anchor{1f1d}
@deffn {Method} is_canonical ()

Return @ref{748,,True} if the argument is canonical and @ref{659,,False}
otherwise.  Currently, a @ref{248,,Decimal} instance is always canonical, so
this operation always returns @ref{748,,True}.
@end deffn

@geindex is_finite() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal is_finite}@anchor{1f1e}
@deffn {Method} is_finite ()

Return @ref{748,,True} if the argument is a finite number, and
@ref{659,,False} if the argument is an infinity or a NaN.
@end deffn

@geindex is_infinite() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal is_infinite}@anchor{1f1f}
@deffn {Method} is_infinite ()

Return @ref{748,,True} if the argument is either positive or negative
infinity and @ref{659,,False} otherwise.
@end deffn

@geindex is_nan() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal is_nan}@anchor{1f20}
@deffn {Method} is_nan ()

Return @ref{748,,True} if the argument is a (quiet or signaling) NaN and
@ref{659,,False} otherwise.
@end deffn

@geindex is_normal() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal is_normal}@anchor{1f21}
@deffn {Method} is_normal (context=None)

Return @ref{748,,True} if the argument is a @emph{normal} finite number.  Return
@ref{659,,False} if the argument is zero, subnormal, infinite or a NaN.
@end deffn

@geindex is_qnan() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal is_qnan}@anchor{1f22}
@deffn {Method} is_qnan ()

Return @ref{748,,True} if the argument is a quiet NaN, and
@ref{659,,False} otherwise.
@end deffn

@geindex is_signed() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal is_signed}@anchor{1f23}
@deffn {Method} is_signed ()

Return @ref{748,,True} if the argument has a negative sign and
@ref{659,,False} otherwise.  Note that zeros and NaNs can both carry signs.
@end deffn

@geindex is_snan() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal is_snan}@anchor{1f24}
@deffn {Method} is_snan ()

Return @ref{748,,True} if the argument is a signaling NaN and @ref{659,,False}
otherwise.
@end deffn

@geindex is_subnormal() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal is_subnormal}@anchor{1f25}
@deffn {Method} is_subnormal (context=None)

Return @ref{748,,True} if the argument is subnormal, and @ref{659,,False}
otherwise.
@end deffn

@geindex is_zero() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal is_zero}@anchor{1f26}
@deffn {Method} is_zero ()

Return @ref{748,,True} if the argument is a (positive or negative) zero and
@ref{659,,False} otherwise.
@end deffn

@geindex ln() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal ln}@anchor{c3e}
@deffn {Method} ln (context=None)

Return the natural (base e) logarithm of the operand.  The result is
correctly rounded using the @ref{1f00,,ROUND_HALF_EVEN} rounding mode.
@end deffn

@geindex log10() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal log10}@anchor{1f27}
@deffn {Method} log10 (context=None)

Return the base ten logarithm of the operand.  The result is correctly
rounded using the @ref{1f00,,ROUND_HALF_EVEN} rounding mode.
@end deffn

@geindex logb() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal logb}@anchor{1f28}
@deffn {Method} logb (context=None)

For a nonzero number, return the adjusted exponent of its operand as a
@ref{248,,Decimal} instance.  If the operand is a zero then
@code{Decimal('-Infinity')} is returned and the @ref{1f05,,DivisionByZero} flag
is raised.  If the operand is an infinity then @code{Decimal('Infinity')} is
returned.
@end deffn

@geindex logical_and() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal logical_and}@anchor{1f29}
@deffn {Method} logical_and (other, context=None)

@ref{1f29,,logical_and()} is a logical operation which takes two @emph{logical
operands} (see @ref{1f2a,,Logical operands}).  The result is the
digit-wise @code{and} of the two operands.
@end deffn

@geindex logical_invert() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal logical_invert}@anchor{1f2b}
@deffn {Method} logical_invert (context=None)

@ref{1f2b,,logical_invert()} is a logical operation.  The
result is the digit-wise inversion of the operand.
@end deffn

@geindex logical_or() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal logical_or}@anchor{1f2c}
@deffn {Method} logical_or (other, context=None)

@ref{1f2c,,logical_or()} is a logical operation which takes two @emph{logical
operands} (see @ref{1f2a,,Logical operands}).  The result is the
digit-wise @code{or} of the two operands.
@end deffn

@geindex logical_xor() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal logical_xor}@anchor{1f2d}
@deffn {Method} logical_xor (other, context=None)

@ref{1f2d,,logical_xor()} is a logical operation which takes two @emph{logical
operands} (see @ref{1f2a,,Logical operands}).  The result is the
digit-wise exclusive or of the two operands.
@end deffn

@geindex max() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal max}@anchor{1f2e}
@deffn {Method} max (other, context=None)

Like @code{max(self, other)} except that the context rounding rule is applied
before returning and that @code{NaN} values are either signaled or
ignored (depending on the context and whether they are signaling or
quiet).
@end deffn

@geindex max_mag() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal max_mag}@anchor{1f2f}
@deffn {Method} max_mag (other, context=None)

Similar to the @ref{1f2e,,max()} method, but the comparison is done using the
absolute values of the operands.
@end deffn

@geindex min() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal min}@anchor{1f30}
@deffn {Method} min (other, context=None)

Like @code{min(self, other)} except that the context rounding rule is applied
before returning and that @code{NaN} values are either signaled or
ignored (depending on the context and whether they are signaling or
quiet).
@end deffn

@geindex min_mag() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal min_mag}@anchor{1f31}
@deffn {Method} min_mag (other, context=None)

Similar to the @ref{1f30,,min()} method, but the comparison is done using the
absolute values of the operands.
@end deffn

@geindex next_minus() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal next_minus}@anchor{1f32}
@deffn {Method} next_minus (context=None)

Return the largest number representable in the given context (or in the
current thread’s context if no context is given) that is smaller than the
given operand.
@end deffn

@geindex next_plus() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal next_plus}@anchor{1f33}
@deffn {Method} next_plus (context=None)

Return the smallest number representable in the given context (or in the
current thread’s context if no context is given) that is larger than the
given operand.
@end deffn

@geindex next_toward() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal next_toward}@anchor{1f34}
@deffn {Method} next_toward (other, context=None)

If the two operands are unequal, return the number closest to the first
operand in the direction of the second operand.  If both operands are
numerically equal, return a copy of the first operand with the sign set to
be the same as the sign of the second operand.
@end deffn

@geindex normalize() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal normalize}@anchor{1f35}
@deffn {Method} normalize (context=None)

Normalize the number by stripping the rightmost trailing zeros and
converting any result equal to @code{Decimal('0')} to
@code{Decimal('0e0')}. Used for producing canonical values for attributes
of an equivalence class. For example, @code{Decimal('32.100')} and
@code{Decimal('0.321000e+2')} both normalize to the equivalent value
@code{Decimal('32.1')}.
@end deffn

@geindex number_class() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal number_class}@anchor{1f36}
@deffn {Method} number_class (context=None)

Return a string describing the @emph{class} of the operand.  The returned value
is one of the following ten strings.


@itemize *

@item 
@code{"-Infinity"}, indicating that the operand is negative infinity.

@item 
@code{"-Normal"}, indicating that the operand is a negative normal number.

@item 
@code{"-Subnormal"}, indicating that the operand is negative and subnormal.

@item 
@code{"-Zero"}, indicating that the operand is a negative zero.

@item 
@code{"+Zero"}, indicating that the operand is a positive zero.

@item 
@code{"+Subnormal"}, indicating that the operand is positive and subnormal.

@item 
@code{"+Normal"}, indicating that the operand is a positive normal number.

@item 
@code{"+Infinity"}, indicating that the operand is positive infinity.

@item 
@code{"NaN"}, indicating that the operand is a quiet NaN (Not a Number).

@item 
@code{"sNaN"}, indicating that the operand is a signaling NaN.
@end itemize
@end deffn

@geindex quantize() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal quantize}@anchor{c41}
@deffn {Method} quantize (exp, rounding=None, context=None)

Return a value equal to the first operand after rounding and having the
exponent of the second operand.

@example
>>> Decimal('1.41421356').quantize(Decimal('1.000'))
Decimal('1.414')
@end example

Unlike other operations, if the length of the coefficient after the
quantize operation would be greater than precision, then an
@ref{c3b,,InvalidOperation} is signaled. This guarantees that, unless there
is an error condition, the quantized exponent is always equal to that of
the right-hand operand.

Also unlike other operations, quantize never signals Underflow, even if
the result is subnormal and inexact.

If the exponent of the second operand is larger than that of the first
then rounding may be necessary.  In this case, the rounding mode is
determined by the @code{rounding} argument if given, else by the given
@code{context} argument; if neither argument is given the rounding mode of
the current thread’s context is used.

An error is returned whenever the resulting exponent is greater than
@code{Emax} or less than @code{Etiny}.
@end deffn

@geindex radix() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal radix}@anchor{1f37}
@deffn {Method} radix ()

Return @code{Decimal(10)}, the radix (base) in which the @ref{248,,Decimal}
class does all its arithmetic.  Included for compatibility with the
specification.
@end deffn

@geindex remainder_near() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal remainder_near}@anchor{1f38}
@deffn {Method} remainder_near (other, context=None)

Return the remainder from dividing @emph{self} by @emph{other}.  This differs from
@code{self % other} in that the sign of the remainder is chosen so as to
minimize its absolute value.  More precisely, the return value is
@code{self - n * other} where @code{n} is the integer nearest to the exact
value of @code{self / other}, and if two integers are equally near then the
even one is chosen.

If the result is zero then its sign will be the sign of @emph{self}.

@example
>>> Decimal(18).remainder_near(Decimal(10))
Decimal('-2')
>>> Decimal(25).remainder_near(Decimal(10))
Decimal('5')
>>> Decimal(35).remainder_near(Decimal(10))
Decimal('-5')
@end example
@end deffn

@geindex rotate() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal rotate}@anchor{1f39}
@deffn {Method} rotate (other, context=None)

Return the result of rotating the digits of the first operand by an amount
specified by the second operand.  The second operand must be an integer in
the range -precision through precision.  The absolute value of the second
operand gives the number of places to rotate.  If the second operand is
positive then rotation is to the left; otherwise rotation is to the right.
The coefficient of the first operand is padded on the left with zeros to
length precision if necessary.  The sign and exponent of the first operand
are unchanged.
@end deffn

@geindex same_quantum() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal same_quantum}@anchor{1f3a}
@deffn {Method} same_quantum (other, context=None)

Test whether self and other have the same exponent or whether both are
@code{NaN}.

This operation is unaffected by context and is quiet: no flags are changed
and no rounding is performed.  As an exception, the C version may raise
InvalidOperation if the second operand cannot be converted exactly.
@end deffn

@geindex scaleb() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal scaleb}@anchor{1f3b}
@deffn {Method} scaleb (other, context=None)

Return the first operand with exponent adjusted by the second.
Equivalently, return the first operand multiplied by @code{10**other}.  The
second operand must be an integer.
@end deffn

@geindex shift() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal shift}@anchor{1f3c}
@deffn {Method} shift (other, context=None)

Return the result of shifting the digits of the first operand by an amount
specified by the second operand.  The second operand must be an integer in
the range -precision through precision.  The absolute value of the second
operand gives the number of places to shift.  If the second operand is
positive then the shift is to the left; otherwise the shift is to the
right.  Digits shifted into the coefficient are zeros.  The sign and
exponent of the first operand are unchanged.
@end deffn

@geindex sqrt() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal sqrt}@anchor{1f3d}
@deffn {Method} sqrt (context=None)

Return the square root of the argument to full precision.
@end deffn

@geindex to_eng_string() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal to_eng_string}@anchor{1f3e}
@deffn {Method} to_eng_string (context=None)

Convert to a string, using engineering notation if an exponent is needed.

Engineering notation has an exponent which is a multiple of 3.  This
can leave up to 3 digits to the left of the decimal place and may
require the addition of either one or two trailing zeros.

For example, this converts @code{Decimal('123E+1')} to @code{Decimal('1.23E+3')}.
@end deffn

@geindex to_integral() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal to_integral}@anchor{1f3f}
@deffn {Method} to_integral (rounding=None, context=None)

Identical to the @ref{1f40,,to_integral_value()} method.  The @code{to_integral}
name has been kept for compatibility with older versions.
@end deffn

@geindex to_integral_exact() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal to_integral_exact}@anchor{1f41}
@deffn {Method} to_integral_exact (rounding=None, context=None)

Round to the nearest integer, signaling @ref{1f06,,Inexact} or
@ref{1f07,,Rounded} as appropriate if rounding occurs.  The rounding mode is
determined by the @code{rounding} parameter if given, else by the given
@code{context}.  If neither parameter is given then the rounding mode of the
current context is used.
@end deffn

@geindex to_integral_value() (decimal.Decimal method)
@anchor{library/decimal decimal Decimal to_integral_value}@anchor{1f40}
@deffn {Method} to_integral_value (rounding=None, context=None)

Round to the nearest integer without signaling @ref{1f06,,Inexact} or
@ref{1f07,,Rounded}.  If given, applies @emph{rounding}; otherwise, uses the
rounding method in either the supplied @emph{context} or the current context.
@end deffn
@end deffn

@menu
* Logical operands:: 

@end menu

@node Logical operands,,,Decimal objects
@anchor{library/decimal logical-operands}@anchor{1f42}@anchor{library/decimal logical-operands-label}@anchor{1f2a}
@subsubsection Logical operands


The @code{logical_and()}, @code{logical_invert()}, @code{logical_or()},
and @code{logical_xor()} methods expect their arguments to be @emph{logical
operands}.  A @emph{logical operand} is a @ref{248,,Decimal} instance whose
exponent and sign are both zero, and whose digits are all either
@code{0} or @code{1}.

@c %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

@node Context objects,Constants<4>,Decimal objects,decimal — Decimal fixed point and floating point arithmetic
@anchor{library/decimal context-objects}@anchor{1f43}@anchor{library/decimal decimal-context}@anchor{1f44}
@subsubsection Context objects


Contexts are environments for arithmetic operations.  They govern precision, set
rules for rounding, determine which signals are treated as exceptions, and limit
the range for exponents.

Each thread has its own current context which is accessed or changed using the
@ref{1f0d,,getcontext()} and @ref{1f0e,,setcontext()} functions:

@geindex getcontext() (in module decimal)
@anchor{library/decimal decimal getcontext}@anchor{1f0d}
@deffn {Function} decimal.getcontext ()

Return the current context for the active thread.
@end deffn

@geindex setcontext() (in module decimal)
@anchor{library/decimal decimal setcontext}@anchor{1f0e}
@deffn {Function} decimal.setcontext (c)

Set the current context for the active thread to @emph{c}.
@end deffn

You can also use the @ref{168,,with} statement and the @ref{1b36,,localcontext()}
function to temporarily change the active context.

@geindex localcontext() (in module decimal)
@anchor{library/decimal decimal localcontext}@anchor{1b36}
@deffn {Function} decimal.localcontext (ctx=None)

Return a context manager that will set the current context for the active thread
to a copy of @emph{ctx} on entry to the with-statement and restore the previous context
when exiting the with-statement. If no context is specified, a copy of the
current context is used.

For example, the following code sets the current decimal precision to 42 places,
performs a calculation, and then automatically restores the previous context:

@example
from decimal import localcontext

with localcontext() as ctx:
    ctx.prec = 42   # Perform a high precision calculation
    s = calculate_something()
s = +s  # Round the final result back to the default precision
@end example
@end deffn

New contexts can also be created using the @ref{c40,,Context} constructor
described below. In addition, the module provides three pre-made contexts:

@geindex BasicContext (class in decimal)
@anchor{library/decimal decimal BasicContext}@anchor{c39}
@deffn {Class} decimal.BasicContext

This is a standard context defined by the General Decimal Arithmetic
Specification.  Precision is set to nine.  Rounding is set to
@ref{1f01,,ROUND_HALF_UP}.  All flags are cleared.  All traps are enabled (treated
as exceptions) except @ref{1f06,,Inexact}, @ref{1f07,,Rounded}, and
@ref{1f08,,Subnormal}.

Because many of the traps are enabled, this context is useful for debugging.
@end deffn

@geindex ExtendedContext (class in decimal)
@anchor{library/decimal decimal ExtendedContext}@anchor{c3a}
@deffn {Class} decimal.ExtendedContext

This is a standard context defined by the General Decimal Arithmetic
Specification.  Precision is set to nine.  Rounding is set to
@ref{1f00,,ROUND_HALF_EVEN}.  All flags are cleared.  No traps are enabled (so that
exceptions are not raised during computations).

Because the traps are disabled, this context is useful for applications that
prefer to have result value of @code{NaN} or @code{Infinity} instead of
raising exceptions.  This allows an application to complete a run in the
presence of conditions that would otherwise halt the program.
@end deffn

@geindex DefaultContext (class in decimal)
@anchor{library/decimal decimal DefaultContext}@anchor{c38}
@deffn {Class} decimal.DefaultContext

This context is used by the @ref{c40,,Context} constructor as a prototype for new
contexts.  Changing a field (such a precision) has the effect of changing the
default for new contexts created by the @ref{c40,,Context} constructor.

This context is most useful in multi-threaded environments.  Changing one of the
fields before threads are started has the effect of setting system-wide
defaults.  Changing the fields after threads have started is not recommended as
it would require thread synchronization to prevent race conditions.

In single threaded environments, it is preferable to not use this context at
all.  Instead, simply create contexts explicitly as described below.

The default values are @code{prec}=@code{28},
@code{rounding}=@ref{1f00,,ROUND_HALF_EVEN},
and enabled traps for @ref{1f09,,Overflow}, @ref{c3b,,InvalidOperation}, and
@ref{1f05,,DivisionByZero}.
@end deffn

In addition to the three supplied contexts, new contexts can be created with the
@ref{c40,,Context} constructor.

@geindex Context (class in decimal)
@anchor{library/decimal decimal Context}@anchor{c40}
@deffn {Class} decimal.Context (prec=None, rounding=None, Emin=None, Emax=None, capitals=None, clamp=None, flags=None, traps=None)

Creates a new context.  If a field is not specified or is @ref{182,,None}, the
default values are copied from the @ref{c38,,DefaultContext}.  If the @emph{flags}
field is not specified or is @ref{182,,None}, all flags are cleared.

@emph{prec} is an integer in the range [@code{1}, @ref{1f45,,MAX_PREC}] that sets
the precision for arithmetic operations in the context.

The @emph{rounding} option is one of the constants listed in the section
@ref{1f46,,Rounding Modes}.

The @emph{traps} and @emph{flags} fields list any signals to be set. Generally, new
contexts should only set traps and leave the flags clear.

The @emph{Emin} and @emph{Emax} fields are integers specifying the outer limits allowable
for exponents. @emph{Emin} must be in the range [@ref{1f47,,MIN_EMIN}, @code{0}],
@emph{Emax} in the range [@code{0}, @ref{1f48,,MAX_EMAX}].

The @emph{capitals} field is either @code{0} or @code{1} (the default). If set to
@code{1}, exponents are printed with a capital @code{E}; otherwise, a
lowercase @code{e} is used: @code{Decimal('6.02e+23')}.

The @emph{clamp} field is either @code{0} (the default) or @code{1}.
If set to @code{1}, the exponent @code{e} of a @ref{248,,Decimal}
instance representable in this context is strictly limited to the
range @code{Emin - prec + 1 <= e <= Emax - prec + 1}.  If @emph{clamp} is
@code{0} then a weaker condition holds: the adjusted exponent of
the @ref{248,,Decimal} instance is at most @code{Emax}.  When @emph{clamp} is
@code{1}, a large normal number will, where possible, have its
exponent reduced and a corresponding number of zeros added to its
coefficient, in order to fit the exponent constraints; this
preserves the value of the number but loses information about
significant trailing zeros.  For example:

@example
>>> Context(prec=6, Emax=999, clamp=1).create_decimal('1.23e999')
Decimal('1.23000E+999')
@end example

A @emph{clamp} value of @code{1} allows compatibility with the
fixed-width decimal interchange formats specified in IEEE 754.

The @ref{c40,,Context} class defines several general purpose methods as well as
a large number of methods for doing arithmetic directly in a given context.
In addition, for each of the @ref{248,,Decimal} methods described above (with
the exception of the @code{adjusted()} and @code{as_tuple()} methods) there is
a corresponding @ref{c40,,Context} method.  For example, for a @ref{c40,,Context}
instance @code{C} and @ref{248,,Decimal} instance @code{x}, @code{C.exp(x)} is
equivalent to @code{x.exp(context=C)}.  Each @ref{c40,,Context} method accepts a
Python integer (an instance of @ref{1b1,,int}) anywhere that a
Decimal instance is accepted.

@geindex clear_flags() (decimal.Context method)
@anchor{library/decimal decimal Context clear_flags}@anchor{1f49}
@deffn {Method} clear_flags ()

Resets all of the flags to @code{0}.
@end deffn

@geindex clear_traps() (decimal.Context method)
@anchor{library/decimal decimal Context clear_traps}@anchor{1f4a}
@deffn {Method} clear_traps ()

Resets all of the traps to @code{0}.


New in version 3.3.

@end deffn

@geindex copy() (decimal.Context method)
@anchor{library/decimal decimal Context copy}@anchor{1f4b}
@deffn {Method} copy ()

Return a duplicate of the context.
@end deffn

@geindex copy_decimal() (decimal.Context method)
@anchor{library/decimal decimal Context copy_decimal}@anchor{1f4c}
@deffn {Method} copy_decimal (num)

Return a copy of the Decimal instance num.
@end deffn

@geindex create_decimal() (decimal.Context method)
@anchor{library/decimal decimal Context create_decimal}@anchor{c3c}
@deffn {Method} create_decimal (num)

Creates a new Decimal instance from @emph{num} but using @emph{self} as
context. Unlike the @ref{248,,Decimal} constructor, the context precision,
rounding method, flags, and traps are applied to the conversion.

This is useful because constants are often given to a greater precision
than is needed by the application.  Another benefit is that rounding
immediately eliminates unintended effects from digits beyond the current
precision. In the following example, using unrounded inputs means that
adding zero to a sum can change the result:

@example
>>> getcontext().prec = 3
>>> Decimal('3.4445') + Decimal('1.0023')
Decimal('4.45')
>>> Decimal('3.4445') + Decimal(0) + Decimal('1.0023')
Decimal('4.44')
@end example

This method implements the to-number operation of the IBM specification.
If the argument is a string, no leading or trailing whitespace or
underscores are permitted.
@end deffn

@geindex create_decimal_from_float() (decimal.Context method)
@anchor{library/decimal decimal Context create_decimal_from_float}@anchor{1f4d}
@deffn {Method} create_decimal_from_float (f)

Creates a new Decimal instance from a float @emph{f} but rounding using @emph{self}
as the context.  Unlike the @ref{da5,,Decimal.from_float()} class method,
the context precision, rounding method, flags, and traps are applied to
the conversion.

@example
>>> context = Context(prec=5, rounding=ROUND_DOWN)
>>> context.create_decimal_from_float(math.pi)
Decimal('3.1415')
>>> context = Context(prec=5, traps=[Inexact])
>>> context.create_decimal_from_float(math.pi)
Traceback (most recent call last):
    ...
decimal.Inexact: None
@end example


New in version 3.1.

@end deffn

@geindex Etiny() (decimal.Context method)
@anchor{library/decimal decimal Context Etiny}@anchor{1f4e}
@deffn {Method} Etiny ()

Returns a value equal to @code{Emin - prec + 1} which is the minimum exponent
value for subnormal results.  When underflow occurs, the exponent is set
to @ref{1f4e,,Etiny}.
@end deffn

@geindex Etop() (decimal.Context method)
@anchor{library/decimal decimal Context Etop}@anchor{1f4f}
@deffn {Method} Etop ()

Returns a value equal to @code{Emax - prec + 1}.
@end deffn

The usual approach to working with decimals is to create @ref{248,,Decimal}
instances and then apply arithmetic operations which take place within the
current context for the active thread.  An alternative approach is to use
context methods for calculating within a specific context.  The methods are
similar to those for the @ref{248,,Decimal} class and are only briefly
recounted here.

@geindex abs() (decimal.Context method)
@anchor{library/decimal decimal Context abs}@anchor{1f50}
@deffn {Method} abs (x)

Returns the absolute value of @emph{x}.
@end deffn

@geindex add() (decimal.Context method)
@anchor{library/decimal decimal Context add}@anchor{1f51}
@deffn {Method} add (x, y)

Return the sum of @emph{x} and @emph{y}.
@end deffn

@geindex canonical() (decimal.Context method)
@anchor{library/decimal decimal Context canonical}@anchor{ea2}
@deffn {Method} canonical (x)

Returns the same Decimal object @emph{x}.
@end deffn

@geindex compare() (decimal.Context method)
@anchor{library/decimal decimal Context compare}@anchor{1f52}
@deffn {Method} compare (x, y)

Compares @emph{x} and @emph{y} numerically.
@end deffn

@geindex compare_signal() (decimal.Context method)
@anchor{library/decimal decimal Context compare_signal}@anchor{1f53}
@deffn {Method} compare_signal (x, y)

Compares the values of the two operands numerically.
@end deffn

@geindex compare_total() (decimal.Context method)
@anchor{library/decimal decimal Context compare_total}@anchor{1f54}
@deffn {Method} compare_total (x, y)

Compares two operands using their abstract representation.
@end deffn

@geindex compare_total_mag() (decimal.Context method)
@anchor{library/decimal decimal Context compare_total_mag}@anchor{1f55}
@deffn {Method} compare_total_mag (x, y)

Compares two operands using their abstract representation, ignoring sign.
@end deffn

@geindex copy_abs() (decimal.Context method)
@anchor{library/decimal decimal Context copy_abs}@anchor{1f56}
@deffn {Method} copy_abs (x)

Returns a copy of @emph{x} with the sign set to 0.
@end deffn

@geindex copy_negate() (decimal.Context method)
@anchor{library/decimal decimal Context copy_negate}@anchor{1f57}
@deffn {Method} copy_negate (x)

Returns a copy of @emph{x} with the sign inverted.
@end deffn

@geindex copy_sign() (decimal.Context method)
@anchor{library/decimal decimal Context copy_sign}@anchor{1f58}
@deffn {Method} copy_sign (x, y)

Copies the sign from @emph{y} to @emph{x}.
@end deffn

@geindex divide() (decimal.Context method)
@anchor{library/decimal decimal Context divide}@anchor{1f59}
@deffn {Method} divide (x, y)

Return @emph{x} divided by @emph{y}.
@end deffn

@geindex divide_int() (decimal.Context method)
@anchor{library/decimal decimal Context divide_int}@anchor{1f5a}
@deffn {Method} divide_int (x, y)

Return @emph{x} divided by @emph{y}, truncated to an integer.
@end deffn

@geindex divmod() (decimal.Context method)
@anchor{library/decimal decimal Context divmod}@anchor{1f5b}
@deffn {Method} divmod (x, y)

Divides two numbers and returns the integer part of the result.
@end deffn

@geindex exp() (decimal.Context method)
@anchor{library/decimal decimal Context exp}@anchor{1f5c}
@deffn {Method} exp (x)

Returns @cite{e ** x}.
@end deffn

@geindex fma() (decimal.Context method)
@anchor{library/decimal decimal Context fma}@anchor{1f5d}
@deffn {Method} fma (x, y, z)

Returns @emph{x} multiplied by @emph{y}, plus @emph{z}.
@end deffn

@geindex is_canonical() (decimal.Context method)
@anchor{library/decimal decimal Context is_canonical}@anchor{ea3}
@deffn {Method} is_canonical (x)

Returns @code{True} if @emph{x} is canonical; otherwise returns @code{False}.
@end deffn

@geindex is_finite() (decimal.Context method)
@anchor{library/decimal decimal Context is_finite}@anchor{1f5e}
@deffn {Method} is_finite (x)

Returns @code{True} if @emph{x} is finite; otherwise returns @code{False}.
@end deffn

@geindex is_infinite() (decimal.Context method)
@anchor{library/decimal decimal Context is_infinite}@anchor{1f5f}
@deffn {Method} is_infinite (x)

Returns @code{True} if @emph{x} is infinite; otherwise returns @code{False}.
@end deffn

@geindex is_nan() (decimal.Context method)
@anchor{library/decimal decimal Context is_nan}@anchor{1f60}
@deffn {Method} is_nan (x)

Returns @code{True} if @emph{x} is a qNaN or sNaN; otherwise returns @code{False}.
@end deffn

@geindex is_normal() (decimal.Context method)
@anchor{library/decimal decimal Context is_normal}@anchor{1f61}
@deffn {Method} is_normal (x)

Returns @code{True} if @emph{x} is a normal number; otherwise returns @code{False}.
@end deffn

@geindex is_qnan() (decimal.Context method)
@anchor{library/decimal decimal Context is_qnan}@anchor{1f62}
@deffn {Method} is_qnan (x)

Returns @code{True} if @emph{x} is a quiet NaN; otherwise returns @code{False}.
@end deffn

@geindex is_signed() (decimal.Context method)
@anchor{library/decimal decimal Context is_signed}@anchor{1f63}
@deffn {Method} is_signed (x)

Returns @code{True} if @emph{x} is negative; otherwise returns @code{False}.
@end deffn

@geindex is_snan() (decimal.Context method)
@anchor{library/decimal decimal Context is_snan}@anchor{1f64}
@deffn {Method} is_snan (x)

Returns @code{True} if @emph{x} is a signaling NaN; otherwise returns @code{False}.
@end deffn

@geindex is_subnormal() (decimal.Context method)
@anchor{library/decimal decimal Context is_subnormal}@anchor{1f65}
@deffn {Method} is_subnormal (x)

Returns @code{True} if @emph{x} is subnormal; otherwise returns @code{False}.
@end deffn

@geindex is_zero() (decimal.Context method)
@anchor{library/decimal decimal Context is_zero}@anchor{1f66}
@deffn {Method} is_zero (x)

Returns @code{True} if @emph{x} is a zero; otherwise returns @code{False}.
@end deffn

@geindex ln() (decimal.Context method)
@anchor{library/decimal decimal Context ln}@anchor{1f67}
@deffn {Method} ln (x)

Returns the natural (base e) logarithm of @emph{x}.
@end deffn

@geindex log10() (decimal.Context method)
@anchor{library/decimal decimal Context log10}@anchor{1f68}
@deffn {Method} log10 (x)

Returns the base 10 logarithm of @emph{x}.
@end deffn

@geindex logb() (decimal.Context method)
@anchor{library/decimal decimal Context logb}@anchor{1f69}
@deffn {Method} logb (x)

Returns the exponent of the magnitude of the operand’s MSD.
@end deffn

@geindex logical_and() (decimal.Context method)
@anchor{library/decimal decimal Context logical_and}@anchor{1f6a}
@deffn {Method} logical_and (x, y)

Applies the logical operation @emph{and} between each operand’s digits.
@end deffn

@geindex logical_invert() (decimal.Context method)
@anchor{library/decimal decimal Context logical_invert}@anchor{1f6b}
@deffn {Method} logical_invert (x)

Invert all the digits in @emph{x}.
@end deffn

@geindex logical_or() (decimal.Context method)
@anchor{library/decimal decimal Context logical_or}@anchor{1f6c}
@deffn {Method} logical_or (x, y)

Applies the logical operation @emph{or} between each operand’s digits.
@end deffn

@geindex logical_xor() (decimal.Context method)
@anchor{library/decimal decimal Context logical_xor}@anchor{1f6d}
@deffn {Method} logical_xor (x, y)

Applies the logical operation @emph{xor} between each operand’s digits.
@end deffn

@geindex max() (decimal.Context method)
@anchor{library/decimal decimal Context max}@anchor{1f6e}
@deffn {Method} max (x, y)

Compares two values numerically and returns the maximum.
@end deffn

@geindex max_mag() (decimal.Context method)
@anchor{library/decimal decimal Context max_mag}@anchor{1f6f}
@deffn {Method} max_mag (x, y)

Compares the values numerically with their sign ignored.
@end deffn

@geindex min() (decimal.Context method)
@anchor{library/decimal decimal Context min}@anchor{1f70}
@deffn {Method} min (x, y)

Compares two values numerically and returns the minimum.
@end deffn

@geindex min_mag() (decimal.Context method)
@anchor{library/decimal decimal Context min_mag}@anchor{1f71}
@deffn {Method} min_mag (x, y)

Compares the values numerically with their sign ignored.
@end deffn

@geindex minus() (decimal.Context method)
@anchor{library/decimal decimal Context minus}@anchor{1f72}
@deffn {Method} minus (x)

Minus corresponds to the unary prefix minus operator in Python.
@end deffn

@geindex multiply() (decimal.Context method)
@anchor{library/decimal decimal Context multiply}@anchor{1f73}
@deffn {Method} multiply (x, y)

Return the product of @emph{x} and @emph{y}.
@end deffn

@geindex next_minus() (decimal.Context method)
@anchor{library/decimal decimal Context next_minus}@anchor{1f74}
@deffn {Method} next_minus (x)

Returns the largest representable number smaller than @emph{x}.
@end deffn

@geindex next_plus() (decimal.Context method)
@anchor{library/decimal decimal Context next_plus}@anchor{1f75}
@deffn {Method} next_plus (x)

Returns the smallest representable number larger than @emph{x}.
@end deffn

@geindex next_toward() (decimal.Context method)
@anchor{library/decimal decimal Context next_toward}@anchor{1f76}
@deffn {Method} next_toward (x, y)

Returns the number closest to @emph{x}, in direction towards @emph{y}.
@end deffn

@geindex normalize() (decimal.Context method)
@anchor{library/decimal decimal Context normalize}@anchor{1f77}
@deffn {Method} normalize (x)

Reduces @emph{x} to its simplest form.
@end deffn

@geindex number_class() (decimal.Context method)
@anchor{library/decimal decimal Context number_class}@anchor{1f78}
@deffn {Method} number_class (x)

Returns an indication of the class of @emph{x}.
@end deffn

@geindex plus() (decimal.Context method)
@anchor{library/decimal decimal Context plus}@anchor{1f79}
@deffn {Method} plus (x)

Plus corresponds to the unary prefix plus operator in Python.  This
operation applies the context precision and rounding, so it is @emph{not} an
identity operation.
@end deffn

@geindex power() (decimal.Context method)
@anchor{library/decimal decimal Context power}@anchor{1f7a}
@deffn {Method} power (x, y, modulo=None)

Return @code{x} to the power of @code{y}, reduced modulo @code{modulo} if given.

With two arguments, compute @code{x**y}.  If @code{x} is negative then @code{y}
must be integral.  The result will be inexact unless @code{y} is integral and
the result is finite and can be expressed exactly in ‘precision’ digits.
The rounding mode of the context is used. Results are always correctly-rounded
in the Python version.

@code{Decimal(0) ** Decimal(0)} results in @code{InvalidOperation}, and if @code{InvalidOperation}
is not trapped, then results in @code{Decimal('NaN')}.


Changed in version 3.3: The C module computes @ref{1f7a,,power()} in terms of the correctly-rounded
@ref{1f5c,,exp()} and @ref{1f67,,ln()} functions. The result is well-defined but
only “almost always correctly-rounded”.


With three arguments, compute @code{(x**y) % modulo}.  For the three argument
form, the following restrictions on the arguments hold:

@quotation


@itemize -

@item 
all three arguments must be integral

@item 
@code{y} must be nonnegative

@item 
at least one of @code{x} or @code{y} must be nonzero

@item 
@code{modulo} must be nonzero and have at most ‘precision’ digits
@end itemize
@end quotation

The value resulting from @code{Context.power(x, y, modulo)} is
equal to the value that would be obtained by computing @code{(x**y)
% modulo} with unbounded precision, but is computed more
efficiently.  The exponent of the result is zero, regardless of
the exponents of @code{x}, @code{y} and @code{modulo}.  The result is
always exact.
@end deffn

@geindex quantize() (decimal.Context method)
@anchor{library/decimal decimal Context quantize}@anchor{1f7b}
@deffn {Method} quantize (x, y)

Returns a value equal to @emph{x} (rounded), having the exponent of @emph{y}.
@end deffn

@geindex radix() (decimal.Context method)
@anchor{library/decimal decimal Context radix}@anchor{1f7c}
@deffn {Method} radix ()

Just returns 10, as this is Decimal, :)
@end deffn

@geindex remainder() (decimal.Context method)
@anchor{library/decimal decimal Context remainder}@anchor{1f7d}
@deffn {Method} remainder (x, y)

Returns the remainder from integer division.

The sign of the result, if non-zero, is the same as that of the original
dividend.
@end deffn

@geindex remainder_near() (decimal.Context method)
@anchor{library/decimal decimal Context remainder_near}@anchor{1f7e}
@deffn {Method} remainder_near (x, y)

Returns @code{x - y * n}, where @emph{n} is the integer nearest the exact value
of @code{x / y} (if the result is 0 then its sign will be the sign of @emph{x}).
@end deffn

@geindex rotate() (decimal.Context method)
@anchor{library/decimal decimal Context rotate}@anchor{1f7f}
@deffn {Method} rotate (x, y)

Returns a rotated copy of @emph{x}, @emph{y} times.
@end deffn

@geindex same_quantum() (decimal.Context method)
@anchor{library/decimal decimal Context same_quantum}@anchor{1f80}
@deffn {Method} same_quantum (x, y)

Returns @code{True} if the two operands have the same exponent.
@end deffn

@geindex scaleb() (decimal.Context method)
@anchor{library/decimal decimal Context scaleb}@anchor{1f81}
@deffn {Method} scaleb (x, y)

Returns the first operand after adding the second value its exp.
@end deffn

@geindex shift() (decimal.Context method)
@anchor{library/decimal decimal Context shift}@anchor{1f82}
@deffn {Method} shift (x, y)

Returns a shifted copy of @emph{x}, @emph{y} times.
@end deffn

@geindex sqrt() (decimal.Context method)
@anchor{library/decimal decimal Context sqrt}@anchor{1f83}
@deffn {Method} sqrt (x)

Square root of a non-negative number to context precision.
@end deffn

@geindex subtract() (decimal.Context method)
@anchor{library/decimal decimal Context subtract}@anchor{1f84}
@deffn {Method} subtract (x, y)

Return the difference between @emph{x} and @emph{y}.
@end deffn

@geindex to_eng_string() (decimal.Context method)
@anchor{library/decimal decimal Context to_eng_string}@anchor{1f85}
@deffn {Method} to_eng_string (x)

Convert to a string, using engineering notation if an exponent is needed.

Engineering notation has an exponent which is a multiple of 3.  This
can leave up to 3 digits to the left of the decimal place and may
require the addition of either one or two trailing zeros.
@end deffn

@geindex to_integral_exact() (decimal.Context method)
@anchor{library/decimal decimal Context to_integral_exact}@anchor{1f86}
@deffn {Method} to_integral_exact (x)

Rounds to an integer.
@end deffn

@geindex to_sci_string() (decimal.Context method)
@anchor{library/decimal decimal Context to_sci_string}@anchor{1f87}
@deffn {Method} to_sci_string (x)

Converts a number to a string using scientific notation.
@end deffn
@end deffn

@c %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

@node Constants<4>,Rounding modes,Context objects,decimal — Decimal fixed point and floating point arithmetic
@anchor{library/decimal constants}@anchor{1f88}@anchor{library/decimal decimal-rounding-modes}@anchor{1f89}
@subsubsection Constants


The constants in this section are only relevant for the C module. They
are also included in the pure Python version for compatibility.


@multitable {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

@tab

32-bit

@tab

64-bit

@item

@geindex MAX_PREC (in module decimal)
@anchor{library/decimal decimal MAX_PREC}@anchor{1f45}
@deffn {Data} decimal.MAX_PREC
@end deffn

@tab

@code{425000000}

@tab

@code{999999999999999999}

@item

@geindex MAX_EMAX (in module decimal)
@anchor{library/decimal decimal MAX_EMAX}@anchor{1f48}
@deffn {Data} decimal.MAX_EMAX
@end deffn

@tab

@code{425000000}

@tab

@code{999999999999999999}

@item

@geindex MIN_EMIN (in module decimal)
@anchor{library/decimal decimal MIN_EMIN}@anchor{1f47}
@deffn {Data} decimal.MIN_EMIN
@end deffn

@tab

@code{-425000000}

@tab

@code{-999999999999999999}

@item

@geindex MIN_ETINY (in module decimal)
@anchor{library/decimal decimal MIN_ETINY}@anchor{1f8a}
@deffn {Data} decimal.MIN_ETINY
@end deffn

@tab

@code{-849999999}

@tab

@code{-1999999999999999997}

@end multitable


@geindex HAVE_THREADS (in module decimal)
@anchor{library/decimal decimal HAVE_THREADS}@anchor{c36}
@deffn {Data} decimal.HAVE_THREADS

The value is @code{True}.  Deprecated, because Python now always has threads.
@end deffn


Deprecated since version 3.9.


@geindex HAVE_CONTEXTVAR (in module decimal)
@anchor{library/decimal decimal HAVE_CONTEXTVAR}@anchor{164a}
@deffn {Data} decimal.HAVE_CONTEXTVAR

The default value is @code{True}. If Python is @ref{1649,,configured using the --without-decimal-contextvar option},
the C version uses a thread-local rather than a coroutine-local context and the value
is @code{False}.  This is slightly faster in some nested context scenarios.
@end deffn


New in version 3.9: backported to 3.7 and 3.8.


@node Rounding modes,Signals,Constants<4>,decimal — Decimal fixed point and floating point arithmetic
@anchor{library/decimal rounding-modes}@anchor{1f46}
@subsubsection Rounding modes


@geindex ROUND_CEILING (in module decimal)
@anchor{library/decimal decimal ROUND_CEILING}@anchor{1efc}
@deffn {Data} decimal.ROUND_CEILING

Round towards @code{Infinity}.
@end deffn

@geindex ROUND_DOWN (in module decimal)
@anchor{library/decimal decimal ROUND_DOWN}@anchor{1efd}
@deffn {Data} decimal.ROUND_DOWN

Round towards zero.
@end deffn

@geindex ROUND_FLOOR (in module decimal)
@anchor{library/decimal decimal ROUND_FLOOR}@anchor{1efe}
@deffn {Data} decimal.ROUND_FLOOR

Round towards @code{-Infinity}.
@end deffn

@geindex ROUND_HALF_DOWN (in module decimal)
@anchor{library/decimal decimal ROUND_HALF_DOWN}@anchor{1eff}
@deffn {Data} decimal.ROUND_HALF_DOWN

Round to nearest with ties going towards zero.
@end deffn

@geindex ROUND_HALF_EVEN (in module decimal)
@anchor{library/decimal decimal ROUND_HALF_EVEN}@anchor{1f00}
@deffn {Data} decimal.ROUND_HALF_EVEN

Round to nearest with ties going to nearest even integer.
@end deffn

@geindex ROUND_HALF_UP (in module decimal)
@anchor{library/decimal decimal ROUND_HALF_UP}@anchor{1f01}
@deffn {Data} decimal.ROUND_HALF_UP

Round to nearest with ties going away from zero.
@end deffn

@geindex ROUND_UP (in module decimal)
@anchor{library/decimal decimal ROUND_UP}@anchor{1f02}
@deffn {Data} decimal.ROUND_UP

Round away from zero.
@end deffn

@geindex ROUND_05UP (in module decimal)
@anchor{library/decimal decimal ROUND_05UP}@anchor{1f03}
@deffn {Data} decimal.ROUND_05UP

Round away from zero if last digit after rounding towards zero would have
been 0 or 5; otherwise round towards zero.
@end deffn

@node Signals,Floating Point Notes,Rounding modes,decimal — Decimal fixed point and floating point arithmetic
@anchor{library/decimal decimal-signals}@anchor{1f8b}@anchor{library/decimal signals}@anchor{1f8c}
@subsubsection Signals


Signals represent conditions that arise during computation. Each corresponds to
one context flag and one context trap enabler.

The context flag is set whenever the condition is encountered. After the
computation, flags may be checked for informational purposes (for instance, to
determine whether a computation was exact). After checking the flags, be sure to
clear all flags before starting the next computation.

If the context’s trap enabler is set for the signal, then the condition causes a
Python exception to be raised.  For example, if the @ref{1f05,,DivisionByZero} trap
is set, then a @ref{1f05,,DivisionByZero} exception is raised upon encountering the
condition.

@geindex Clamped (class in decimal)
@anchor{library/decimal decimal Clamped}@anchor{1f04}
@deffn {Class} decimal.Clamped

Altered an exponent to fit representation constraints.

Typically, clamping occurs when an exponent falls outside the context’s
@code{Emin} and @code{Emax} limits.  If possible, the exponent is reduced to
fit by adding zeros to the coefficient.
@end deffn

@geindex DecimalException (class in decimal)
@anchor{library/decimal decimal DecimalException}@anchor{1f8d}
@deffn {Class} decimal.DecimalException

Base class for other signals and a subclass of @ref{1b70,,ArithmeticError}.
@end deffn

@geindex DivisionByZero (class in decimal)
@anchor{library/decimal decimal DivisionByZero}@anchor{1f05}
@deffn {Class} decimal.DivisionByZero

Signals the division of a non-infinite number by zero.

Can occur with division, modulo division, or when raising a number to a negative
power.  If this signal is not trapped, returns @code{Infinity} or
@code{-Infinity} with the sign determined by the inputs to the calculation.
@end deffn

@geindex Inexact (class in decimal)
@anchor{library/decimal decimal Inexact}@anchor{1f06}
@deffn {Class} decimal.Inexact

Indicates that rounding occurred and the result is not exact.

Signals when non-zero digits were discarded during rounding. The rounded result
is returned.  The signal flag or trap is used to detect when results are
inexact.
@end deffn

@geindex InvalidOperation (class in decimal)
@anchor{library/decimal decimal InvalidOperation}@anchor{c3b}
@deffn {Class} decimal.InvalidOperation

An invalid operation was performed.

Indicates that an operation was requested that does not make sense. If not
trapped, returns @code{NaN}.  Possible causes include:

@example
Infinity - Infinity
0 * Infinity
Infinity / Infinity
x % 0
Infinity % x
sqrt(-x) and x > 0
0 ** 0
x ** (non-integer)
x ** Infinity
@end example
@end deffn

@geindex Overflow (class in decimal)
@anchor{library/decimal decimal Overflow}@anchor{1f09}
@deffn {Class} decimal.Overflow

Numerical overflow.

Indicates the exponent is larger than @code{Emax} after rounding has
occurred.  If not trapped, the result depends on the rounding mode, either
pulling inward to the largest representable finite number or rounding outward
to @code{Infinity}.  In either case, @ref{1f06,,Inexact} and @ref{1f07,,Rounded}
are also signaled.
@end deffn

@geindex Rounded (class in decimal)
@anchor{library/decimal decimal Rounded}@anchor{1f07}
@deffn {Class} decimal.Rounded

Rounding occurred though possibly no information was lost.

Signaled whenever rounding discards digits; even if those digits are zero
(such as rounding @code{5.00} to @code{5.0}).  If not trapped, returns
the result unchanged.  This signal is used to detect loss of significant
digits.
@end deffn

@geindex Subnormal (class in decimal)
@anchor{library/decimal decimal Subnormal}@anchor{1f08}
@deffn {Class} decimal.Subnormal

Exponent was lower than @code{Emin} prior to rounding.

Occurs when an operation result is subnormal (the exponent is too small). If
not trapped, returns the result unchanged.
@end deffn

@geindex Underflow (class in decimal)
@anchor{library/decimal decimal Underflow}@anchor{1f0a}
@deffn {Class} decimal.Underflow

Numerical underflow with result rounded to zero.

Occurs when a subnormal result is pushed to zero by rounding. @ref{1f06,,Inexact}
and @ref{1f08,,Subnormal} are also signaled.
@end deffn

@geindex FloatOperation (class in decimal)
@anchor{library/decimal decimal FloatOperation}@anchor{c35}
@deffn {Class} decimal.FloatOperation

Enable stricter semantics for mixing floats and Decimals.

If the signal is not trapped (default), mixing floats and Decimals is
permitted in the @ref{248,,Decimal} constructor,
@ref{c3c,,create_decimal()} and all comparison operators.
Both conversion and comparisons are exact. Any occurrence of a mixed
operation is silently recorded by setting @ref{c35,,FloatOperation} in the
context flags. Explicit conversions with @ref{da5,,from_float()}
or @ref{1f4d,,create_decimal_from_float()} do not set the flag.

Otherwise (the signal is trapped), only equality comparisons and explicit
conversions are silent. All other mixed operations raise @ref{c35,,FloatOperation}.
@end deffn

The following table summarizes the hierarchy of signals:

@example
exceptions.ArithmeticError(exceptions.Exception)
    DecimalException
        Clamped
        DivisionByZero(DecimalException, exceptions.ZeroDivisionError)
        Inexact
            Overflow(Inexact, Rounded)
            Underflow(Inexact, Rounded, Subnormal)
        InvalidOperation
        Rounded
        Subnormal
        FloatOperation(DecimalException, exceptions.TypeError)
@end example

@c %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

@node Floating Point Notes,Working with threads,Signals,decimal — Decimal fixed point and floating point arithmetic
@anchor{library/decimal decimal-notes}@anchor{1f8e}@anchor{library/decimal floating-point-notes}@anchor{1f8f}
@subsubsection Floating Point Notes


@menu
* Mitigating round-off error with increased precision:: 
* Special values:: 

@end menu

@node Mitigating round-off error with increased precision,Special values,,Floating Point Notes
@anchor{library/decimal mitigating-round-off-error-with-increased-precision}@anchor{1f90}
@subsubsection Mitigating round-off error with increased precision


The use of decimal floating point eliminates decimal representation error
(making it possible to represent @code{0.1} exactly); however, some operations
can still incur round-off error when non-zero digits exceed the fixed precision.

The effects of round-off error can be amplified by the addition or subtraction
of nearly offsetting quantities resulting in loss of significance.  Knuth
provides two instructive examples where rounded floating point arithmetic with
insufficient precision causes the breakdown of the associative and distributive
properties of addition:

@example
# Examples from Seminumerical Algorithms, Section 4.2.2.
>>> from decimal import Decimal, getcontext
>>> getcontext().prec = 8

>>> u, v, w = Decimal(11111113), Decimal(-11111111), Decimal('7.51111111')
>>> (u + v) + w
Decimal('9.5111111')
>>> u + (v + w)
Decimal('10')

>>> u, v, w = Decimal(20000), Decimal(-6), Decimal('6.0000003')
>>> (u*v) + (u*w)
Decimal('0.01')
>>> u * (v+w)
Decimal('0.0060000')
@end example

The @ref{34,,decimal} module makes it possible to restore the identities by
expanding the precision sufficiently to avoid loss of significance:

@example
>>> getcontext().prec = 20
>>> u, v, w = Decimal(11111113), Decimal(-11111111), Decimal('7.51111111')
>>> (u + v) + w
Decimal('9.51111111')
>>> u + (v + w)
Decimal('9.51111111')
>>>
>>> u, v, w = Decimal(20000), Decimal(-6), Decimal('6.0000003')
>>> (u*v) + (u*w)
Decimal('0.0060000')
>>> u * (v+w)
Decimal('0.0060000')
@end example

@node Special values,,Mitigating round-off error with increased precision,Floating Point Notes
@anchor{library/decimal special-values}@anchor{1f91}
@subsubsection Special values


The number system for the @ref{34,,decimal} module provides special values
including @code{NaN}, @code{sNaN}, @code{-Infinity}, @code{Infinity},
and two zeros, @code{+0} and @code{-0}.

Infinities can be constructed directly with:  @code{Decimal('Infinity')}. Also,
they can arise from dividing by zero when the @ref{1f05,,DivisionByZero} signal is
not trapped.  Likewise, when the @ref{1f09,,Overflow} signal is not trapped, infinity
can result from rounding beyond the limits of the largest representable number.

The infinities are signed (affine) and can be used in arithmetic operations
where they get treated as very large, indeterminate numbers.  For instance,
adding a constant to infinity gives another infinite result.

Some operations are indeterminate and return @code{NaN}, or if the
@ref{c3b,,InvalidOperation} signal is trapped, raise an exception.  For example,
@code{0/0} returns @code{NaN} which means “not a number”.  This variety of
@code{NaN} is quiet and, once created, will flow through other computations
always resulting in another @code{NaN}.  This behavior can be useful for a
series of computations that occasionally have missing inputs — it allows the
calculation to proceed while flagging specific results as invalid.

A variant is @code{sNaN} which signals rather than remaining quiet after every
operation.  This is a useful return value when an invalid result needs to
interrupt a calculation for special handling.

The behavior of Python’s comparison operators can be a little surprising where a
@code{NaN} is involved.  A test for equality where one of the operands is a
quiet or signaling @code{NaN} always returns @ref{659,,False} (even when doing
@code{Decimal('NaN')==Decimal('NaN')}), while a test for inequality always returns
@ref{748,,True}.  An attempt to compare two Decimals using any of the @code{<},
@code{<=}, @code{>} or @code{>=} operators will raise the @ref{c3b,,InvalidOperation} signal
if either operand is a @code{NaN}, and return @ref{659,,False} if this signal is
not trapped.  Note that the General Decimal Arithmetic specification does not
specify the behavior of direct comparisons; these rules for comparisons
involving a @code{NaN} were taken from the IEEE 854 standard (see Table 3 in
section 5.7).  To ensure strict standards-compliance, use the @code{compare()}
and @code{compare-signal()} methods instead.

The signed zeros can result from calculations that underflow. They keep the sign
that would have resulted if the calculation had been carried out to greater
precision.  Since their magnitude is zero, both positive and negative zeros are
treated as equal and their sign is informational.

In addition to the two signed zeros which are distinct yet equal, there are
various representations of zero with differing precisions yet equivalent in
value.  This takes a bit of getting used to.  For an eye accustomed to
normalized floating point representations, it is not immediately obvious that
the following calculation returns a value equal to zero:

@example
>>> 1 / Decimal('Infinity')
Decimal('0E-1000026')
@end example

@c %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

@node Working with threads,Recipes,Floating Point Notes,decimal — Decimal fixed point and floating point arithmetic
@anchor{library/decimal decimal-threads}@anchor{1f92}@anchor{library/decimal working-with-threads}@anchor{1f93}
@subsubsection Working with threads


The @ref{1f0d,,getcontext()} function accesses a different @ref{c40,,Context} object for
each thread.  Having separate thread contexts means that threads may make
changes (such as @code{getcontext().prec=10}) without interfering with other threads.

Likewise, the @ref{1f0e,,setcontext()} function automatically assigns its target to
the current thread.

If @ref{1f0e,,setcontext()} has not been called before @ref{1f0d,,getcontext()}, then
@ref{1f0d,,getcontext()} will automatically create a new context for use in the
current thread.

The new context is copied from a prototype context called @emph{DefaultContext}. To
control the defaults so that each thread will use the same values throughout the
application, directly modify the @emph{DefaultContext} object. This should be done
@emph{before} any threads are started so that there won’t be a race condition between
threads calling @ref{1f0d,,getcontext()}. For example:

@example
# Set applicationwide defaults for all threads about to be launched
DefaultContext.prec = 12
DefaultContext.rounding = ROUND_DOWN
DefaultContext.traps = ExtendedContext.traps.copy()
DefaultContext.traps[InvalidOperation] = 1
setcontext(DefaultContext)

# Afterwards, the threads can be started
t1.start()
t2.start()
t3.start()
 . . .
@end example

@c %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

@node Recipes,Decimal FAQ,Working with threads,decimal — Decimal fixed point and floating point arithmetic
@anchor{library/decimal decimal-recipes}@anchor{1f94}@anchor{library/decimal recipes}@anchor{1f95}
@subsubsection Recipes


Here are a few recipes that serve as utility functions and that demonstrate ways
to work with the @ref{248,,Decimal} class:

@example
def moneyfmt(value, places=2, curr='@w{'}, sep=',', dp='.',
             pos='@w{'}, neg='-', trailneg='@w{'}):
    """Convert Decimal to a money formatted string.

    places:  required number of places after the decimal point
    curr:    optional currency symbol before the sign (may be blank)
    sep:     optional grouping separator (comma, period, space, or blank)
    dp:      decimal point indicator (comma or period)
             only specify as blank when places is zero
    pos:     optional sign for positive numbers: '+', space or blank
    neg:     optional sign for negative numbers: '-', '(', space or blank
    trailneg:optional trailing minus indicator:  '-', ')', space or blank

    >>> d = Decimal('-1234567.8901')
    >>> moneyfmt(d, curr='$')
    '-$1,234,567.89'
    >>> moneyfmt(d, places=0, sep='.', dp='@w{'}, neg='@w{'}, trailneg='-')
    '1.234.568-'
    >>> moneyfmt(d, curr='$', neg='(', trailneg=')')
    '($1,234,567.89)'
    >>> moneyfmt(Decimal(123456789), sep=' ')
    '123 456 789.00'
    >>> moneyfmt(Decimal('-0.02'), neg='<', trailneg='>')
    '<0.02>'

    """
    q = Decimal(10) ** -places      # 2 places --> '0.01'
    sign, digits, exp = value.quantize(q).as_tuple()
    result = []
    digits = list(map(str, digits))
    build, next = result.append, digits.pop
    if sign:
        build(trailneg)
    for i in range(places):
        build(next() if digits else '0')
    if places:
        build(dp)
    if not digits:
        build('0')
    i = 0
    while digits:
        build(next())
        i += 1
        if i == 3 and digits:
            i = 0
            build(sep)
    build(curr)
    build(neg if sign else pos)
    return '@w{'}.join(reversed(result))

def pi():
    """Compute Pi to the current precision.

    >>> print(pi())
    3.141592653589793238462643383

    """
    getcontext().prec += 2  # extra digits for intermediate steps
    three = Decimal(3)      # substitute "three=3.0" for regular floats
    lasts, t, s, n, na, d, da = 0, three, 3, 1, 0, 0, 24
    while s != lasts:
        lasts = s
        n, na = n+na, na+8
        d, da = d+da, da+32
        t = (t * n) / d
        s += t
    getcontext().prec -= 2
    return +s               # unary plus applies the new precision

def exp(x):
    """Return e raised to the power of x.  Result type matches input type.

    >>> print(exp(Decimal(1)))
    2.718281828459045235360287471
    >>> print(exp(Decimal(2)))
    7.389056098930650227230427461
    >>> print(exp(2.0))
    7.38905609893
    >>> print(exp(2+0j))
    (7.38905609893+0j)

    """
    getcontext().prec += 2
    i, lasts, s, fact, num = 0, 0, 1, 1, 1
    while s != lasts:
        lasts = s
        i += 1
        fact *= i
        num *= x
        s += num / fact
    getcontext().prec -= 2
    return +s

def cos(x):
    """Return the cosine of x as measured in radians.

    The Taylor series approximation works best for a small value of x.
    For larger values, first compute x = x % (2 * pi).

    >>> print(cos(Decimal('0.5')))
    0.8775825618903727161162815826
    >>> print(cos(0.5))
    0.87758256189
    >>> print(cos(0.5+0j))
    (0.87758256189+0j)

    """
    getcontext().prec += 2
    i, lasts, s, fact, num, sign = 0, 0, 1, 1, 1, 1
    while s != lasts:
        lasts = s
        i += 2
        fact *= i * (i-1)
        num *= x * x
        sign *= -1
        s += num / fact * sign
    getcontext().prec -= 2
    return +s

def sin(x):
    """Return the sine of x as measured in radians.

    The Taylor series approximation works best for a small value of x.
    For larger values, first compute x = x % (2 * pi).

    >>> print(sin(Decimal('0.5')))
    0.4794255386042030002732879352
    >>> print(sin(0.5))
    0.479425538604
    >>> print(sin(0.5+0j))
    (0.479425538604+0j)

    """
    getcontext().prec += 2
    i, lasts, s, fact, num, sign = 1, 0, x, 1, x, 1
    while s != lasts:
        lasts = s
        i += 2
        fact *= i * (i-1)
        num *= x * x
        sign *= -1
        s += num / fact * sign
    getcontext().prec -= 2
    return +s
@end example

@c %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

@node Decimal FAQ,,Recipes,decimal — Decimal fixed point and floating point arithmetic
@anchor{library/decimal decimal-faq}@anchor{1f96}@anchor{library/decimal id1}@anchor{1f97}
@subsubsection Decimal FAQ


Q. It is cumbersome to type @code{decimal.Decimal('1234.5')}.  Is there a way to
minimize typing when using the interactive interpreter?

A. Some users abbreviate the constructor to just a single letter:

@example
>>> D = decimal.Decimal
>>> D('1.23') + D('3.45')
Decimal('4.68')
@end example

Q. In a fixed-point application with two decimal places, some inputs have many
places and need to be rounded.  Others are not supposed to have excess digits
and need to be validated.  What methods should be used?

A. The @code{quantize()} method rounds to a fixed number of decimal places. If
the @ref{1f06,,Inexact} trap is set, it is also useful for validation:

@example
>>> TWOPLACES = Decimal(10) ** -2       # same as Decimal('0.01')
@end example

@example
>>> # Round to two places
>>> Decimal('3.214').quantize(TWOPLACES)
Decimal('3.21')
@end example

@example
>>> # Validate that a number does not exceed two places
>>> Decimal('3.21').quantize(TWOPLACES, context=Context(traps=[Inexact]))
Decimal('3.21')
@end example

@example
>>> Decimal('3.214').quantize(TWOPLACES, context=Context(traps=[Inexact]))
Traceback (most recent call last):
   ...
Inexact: None
@end example

Q. Once I have valid two place inputs, how do I maintain that invariant
throughout an application?

A. Some operations like addition, subtraction, and multiplication by an integer
will automatically preserve fixed point.  Others operations, like division and
non-integer multiplication, will change the number of decimal places and need to
be followed-up with a @code{quantize()} step:

@example
>>> a = Decimal('102.72')           # Initial fixed-point values
>>> b = Decimal('3.17')
>>> a + b                           # Addition preserves fixed-point
Decimal('105.89')
>>> a - b
Decimal('99.55')
>>> a * 42                          # So does integer multiplication
Decimal('4314.24')
>>> (a * b).quantize(TWOPLACES)     # Must quantize non-integer multiplication
Decimal('325.62')
>>> (b / a).quantize(TWOPLACES)     # And quantize division
Decimal('0.03')
@end example

In developing fixed-point applications, it is convenient to define functions
to handle the @code{quantize()} step:

@example
>>> def mul(x, y, fp=TWOPLACES):
...     return (x * y).quantize(fp)
>>> def div(x, y, fp=TWOPLACES):
...     return (x / y).quantize(fp)
@end example

@example
>>> mul(a, b)                       # Automatically preserve fixed-point
Decimal('325.62')
>>> div(b, a)
Decimal('0.03')
@end example

Q. There are many ways to express the same value.  The numbers @code{200},
@code{200.000}, @code{2E2}, and @code{02E+4} all have the same value at
various precisions. Is there a way to transform them to a single recognizable
canonical value?

A. The @code{normalize()} method maps all equivalent values to a single
representative:

@example
>>> values = map(Decimal, '200 200.000 2E2 .02E+4'.split())
>>> [v.normalize() for v in values]
[Decimal('2E+2'), Decimal('2E+2'), Decimal('2E+2'), Decimal('2E+2')]
@end example

Q. Some decimal values always print with exponential notation.  Is there a way
to get a non-exponential representation?

A. For some values, exponential notation is the only way to express the number
of significant places in the coefficient.  For example, expressing
@code{5.0E+3} as @code{5000} keeps the value constant but cannot show the
original’s two-place significance.

If an application does not care about tracking significance, it is easy to
remove the exponent and trailing zeroes, losing significance, but keeping the
value unchanged:

@example
>>> def remove_exponent(d):
...     return d.quantize(Decimal(1)) if d == d.to_integral() else d.normalize()
@end example

@example
>>> remove_exponent(Decimal('5E+3'))
Decimal('5000')
@end example

Q. Is there a way to convert a regular float to a @ref{248,,Decimal}?

A. Yes, any binary floating point number can be exactly expressed as a
Decimal though an exact conversion may take more precision than intuition would
suggest:

@example
>>> Decimal(math.pi)
Decimal('3.141592653589793115997963468544185161590576171875')
@end example

Q. Within a complex calculation, how can I make sure that I haven’t gotten a
spurious result because of insufficient precision or rounding anomalies.

A. The decimal module makes it easy to test results.  A best practice is to
re-run calculations using greater precision and with various rounding modes.
Widely differing results indicate insufficient precision, rounding mode issues,
ill-conditioned inputs, or a numerically unstable algorithm.

Q. I noticed that context precision is applied to the results of operations but
not to the inputs.  Is there anything to watch out for when mixing values of
different precisions?

A. Yes.  The principle is that all values are considered to be exact and so is
the arithmetic on those values.  Only the results are rounded.  The advantage
for inputs is that “what you type is what you get”.  A disadvantage is that the
results can look odd if you forget that the inputs haven’t been rounded:

@example
>>> getcontext().prec = 3
>>> Decimal('3.104') + Decimal('2.104')
Decimal('5.21')
>>> Decimal('3.104') + Decimal('0.000') + Decimal('2.104')
Decimal('5.20')
@end example

The solution is either to increase precision or to force rounding of inputs
using the unary plus operation:

@example
>>> getcontext().prec = 3
>>> +Decimal('1.23456789')      # unary plus triggers rounding
Decimal('1.23')
@end example

Alternatively, inputs can be rounded upon creation using the
@ref{c3c,,Context.create_decimal()} method:

@example
>>> Context(prec=5, rounding=ROUND_DOWN).create_decimal('1.2345678')
Decimal('1.2345')
@end example

Q. Is the CPython implementation fast for large numbers?

A. Yes.  In the CPython and PyPy3 implementations, the C/CFFI versions of
the decimal module integrate the high speed libmpdec@footnote{https://www.bytereef.org/mpdecimal/doc/libmpdec/index.html} library for
arbitrary precision correctly-rounded decimal floating point arithmetic @footnote{@w{(1)} 

New in version 3.3.

}.
@code{libmpdec} uses Karatsuba multiplication@footnote{https://en.wikipedia.org/wiki/Karatsuba_algorithm}
for medium-sized numbers and the Number Theoretic Transform@footnote{https://en.wikipedia.org/wiki/Discrete_Fourier_transform_(general)#Number-theoretic_transform}
for very large numbers.

The context must be adapted for exact arbitrary precision arithmetic. @code{Emin}
and @code{Emax} should always be set to the maximum values, @code{clamp}
should always be 0 (the default).  Setting @code{prec} requires some care.

The easiest approach for trying out bignum arithmetic is to use the maximum
value for @code{prec} as well @footnote{@w{(2)} 

Changed in version 3.9: This approach now works for all exact results except for non-integer powers.

}:

@example
>>> setcontext(Context(prec=MAX_PREC, Emax=MAX_EMAX, Emin=MIN_EMIN))
>>> x = Decimal(2) ** 256
>>> x / 128
Decimal('904625697166532776746648320380374280103671755200316906558262375061821325312')
@end example

For inexact results, @ref{1f45,,MAX_PREC} is far too large on 64-bit platforms and
the available memory will be insufficient:

@example
>>> Decimal(1) / 3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError
@end example

On systems with overallocation (e.g. Linux), a more sophisticated approach is to
adjust @code{prec} to the amount of available RAM.  Suppose that you have 8GB of
RAM and expect 10 simultaneous operands using a maximum of 500MB each:

@example
>>> import sys
>>>
>>> # Maximum number of digits for a single operand using 500MB in 8-byte words
>>> # with 19 digits per word (4-byte and 9 digits for the 32-bit build):
>>> maxdigits = 19 * ((500 * 1024**2) // 8)
>>>
>>> # Check that this works:
>>> c = Context(prec=maxdigits, Emax=MAX_EMAX, Emin=MIN_EMIN)
>>> c.traps[Inexact] = True
>>> setcontext(c)
>>>
>>> # Fill the available precision with nines:
>>> x = Decimal(0).logical_invert() * 9
>>> sys.getsizeof(x)
524288112
>>> x + 2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  decimal.Inexact: [<class 'decimal.Inexact'>]
@end example

In general (and especially on systems without overallocation), it is recommended
to estimate even tighter bounds and set the @ref{1f06,,Inexact} trap if all calculations
are expected to be exact.

@node fractions — Rational numbers,random — Generate pseudo-random numbers,decimal — Decimal fixed point and floating point arithmetic,Numeric and Mathematical Modules
@anchor{library/fractions doc}@anchor{1f98}@anchor{library/fractions fractions-rational-numbers}@anchor{1f99}@anchor{library/fractions module-fractions}@anchor{7d}
@subsection @code{fractions} — Rational numbers


@geindex module; fractions

@strong{Source code:} Lib/fractions.py@footnote{https://github.com/python/cpython/tree/main/Lib/fractions.py}


__________________________________________________________________


The @ref{7d,,fractions} module provides support for rational number arithmetic.

A Fraction instance can be constructed from a pair of integers, from
another rational number, or from a string.

@geindex Fraction (class in fractions)
@anchor{library/fractions fractions Fraction}@anchor{172}
@deffn {Class} fractions.Fraction (numerator=0, denominator=1)

@deffnx {Class} fractions.Fraction (other_fraction)

@deffnx {Class} fractions.Fraction (float)

@deffnx {Class} fractions.Fraction (decimal)

@deffnx {Class} fractions.Fraction (string)

The first version requires that @emph{numerator} and @emph{denominator} are instances
of @ref{e64,,numbers.Rational} and returns a new @ref{172,,Fraction} instance
with value @code{numerator/denominator}. If @emph{denominator} is @code{0}, it
raises a @ref{43c,,ZeroDivisionError}. The second version requires that
@emph{other_fraction} is an instance of @ref{e64,,numbers.Rational} and returns a
@ref{172,,Fraction} instance with the same value.  The next two versions accept
either a @ref{256,,float} or a @ref{248,,decimal.Decimal} instance, and return a
@ref{172,,Fraction} instance with exactly the same value.  Note that due to the
usual issues with binary floating-point (see @ref{15e5,,Floating Point Arithmetic; Issues and Limitations}), the
argument to @code{Fraction(1.1)} is not exactly equal to 11/10, and so
@code{Fraction(1.1)} does @emph{not} return @code{Fraction(11, 10)} as one might expect.
(But see the documentation for the @ref{1f9a,,limit_denominator()} method below.)
The last version of the constructor expects a string or unicode instance.
The usual form for this instance is:

@example
[sign] numerator ['/' denominator]
@end example

where the optional @code{sign} may be either ‘+’ or ‘-’ and
@code{numerator} and @code{denominator} (if present) are strings of
decimal digits (underscores may be used to delimit digits as with
integral literals in code).  In addition, any string that represents a finite
value and is accepted by the @ref{256,,float} constructor is also
accepted by the @ref{172,,Fraction} constructor.  In either form the
input string may also have leading and/or trailing whitespace.
Here are some examples:

@example
>>> from fractions import Fraction
>>> Fraction(16, -10)
Fraction(-8, 5)
>>> Fraction(123)
Fraction(123, 1)
>>> Fraction()
Fraction(0, 1)
>>> Fraction('3/7')
Fraction(3, 7)
>>> Fraction(' -3/7 ')
Fraction(-3, 7)
>>> Fraction('1.414213 \t\n')
Fraction(1414213, 1000000)
>>> Fraction('-.125')
Fraction(-1, 8)
>>> Fraction('7e-6')
Fraction(7, 1000000)
>>> Fraction(2.25)
Fraction(9, 4)
>>> Fraction(1.1)
Fraction(2476979795053773, 2251799813685248)
>>> from decimal import Decimal
>>> Fraction(Decimal('1.1'))
Fraction(11, 10)
@end example

The @ref{172,,Fraction} class inherits from the abstract base class
@ref{e64,,numbers.Rational}, and implements all of the methods and
operations from that class.  @ref{172,,Fraction} instances are hashable,
and should be treated as immutable.  In addition,
@ref{172,,Fraction} has the following properties and methods:


Changed in version 3.2: The @ref{172,,Fraction} constructor now accepts @ref{256,,float} and
@ref{248,,decimal.Decimal} instances.



Changed in version 3.9: The @ref{401,,math.gcd()} function is now used to normalize the @emph{numerator}
and @emph{denominator}. @ref{401,,math.gcd()} always return a @ref{1b1,,int} type.
Previously, the GCD type depended on @emph{numerator} and @emph{denominator}.



Changed in version 3.11: Underscores are now permitted when creating a @ref{172,,Fraction} instance
from a string, following 
@geindex Python Enhancement Proposals; PEP 515
PEP 515@footnote{https://www.python.org/dev/peps/pep-0515} rules.



Changed in version 3.11: @ref{172,,Fraction} implements @code{__int__} now to satisfy
@code{typing.SupportsInt} instance checks.


@geindex numerator (fractions.Fraction attribute)
@anchor{library/fractions fractions Fraction numerator}@anchor{1f9b}
@deffn {Attribute} numerator

Numerator of the Fraction in lowest term.
@end deffn

@geindex denominator (fractions.Fraction attribute)
@anchor{library/fractions fractions Fraction denominator}@anchor{1f9c}
@deffn {Attribute} denominator

Denominator of the Fraction in lowest term.
@end deffn

@geindex as_integer_ratio() (fractions.Fraction method)
@anchor{library/fractions fractions Fraction as_integer_ratio}@anchor{1f9d}
@deffn {Method} as_integer_ratio ()

Return a tuple of two integers, whose ratio is equal
to the Fraction and with a positive denominator.


New in version 3.8.

@end deffn

@geindex from_float() (fractions.Fraction method)
@anchor{library/fractions fractions Fraction from_float}@anchor{da6}
@deffn {Method} from_float (flt)

This class method constructs a @ref{172,,Fraction} representing the exact
value of @emph{flt}, which must be a @ref{256,,float}. Beware that
@code{Fraction.from_float(0.3)} is not the same value as @code{Fraction(3, 10)}.

@cartouche
@quotation Note 
From Python 3.2 onwards, you can also construct a
@ref{172,,Fraction} instance directly from a @ref{256,,float}.
@end quotation
@end cartouche
@end deffn

@geindex from_decimal() (fractions.Fraction method)
@anchor{library/fractions fractions Fraction from_decimal}@anchor{da7}
@deffn {Method} from_decimal (dec)

This class method constructs a @ref{172,,Fraction} representing the exact
value of @emph{dec}, which must be a @ref{248,,decimal.Decimal} instance.

@cartouche
@quotation Note 
From Python 3.2 onwards, you can also construct a
@ref{172,,Fraction} instance directly from a @ref{248,,decimal.Decimal}
instance.
@end quotation
@end cartouche
@end deffn

@geindex limit_denominator() (fractions.Fraction method)
@anchor{library/fractions fractions Fraction limit_denominator}@anchor{1f9a}
@deffn {Method} limit_denominator (max_denominator=1000000)

Finds and returns the closest @ref{172,,Fraction} to @code{self} that has
denominator at most max_denominator.  This method is useful for finding
rational approximations to a given floating-point number:

@example
>>> from fractions import Fraction
>>> Fraction('3.1415926535897932').limit_denominator(1000)
Fraction(355, 113)
@end example

or for recovering a rational number that’s represented as a float:

@example
>>> from math import pi, cos
>>> Fraction(cos(pi/3))
Fraction(4503599627370497, 9007199254740992)
>>> Fraction(cos(pi/3)).limit_denominator()
Fraction(1, 2)
>>> Fraction(1.1).limit_denominator()
Fraction(11, 10)
@end example
@end deffn

@geindex __floor__() (fractions.Fraction method)
@anchor{library/fractions fractions Fraction __floor__}@anchor{1f9e}
@deffn {Method} __floor__ ()

Returns the greatest @ref{1b1,,int} @code{<= self}.  This method can
also be accessed through the @ref{f28,,math.floor()} function:

@example
>>> from math import floor
>>> floor(Fraction(355, 113))
3
@end example
@end deffn

@geindex __ceil__() (fractions.Fraction method)
@anchor{library/fractions fractions Fraction __ceil__}@anchor{1f9f}
@deffn {Method} __ceil__ ()

Returns the least @ref{1b1,,int} @code{>= self}.  This method can
also be accessed through the @ref{f29,,math.ceil()} function.
@end deffn

@geindex __round__() (fractions.Fraction method)
@anchor{library/fractions fractions Fraction __round__}@anchor{1fa0}
@deffn {Method} __round__ ()

@deffnx {Method} __round__ (ndigits)

The first version returns the nearest @ref{1b1,,int} to @code{self},
rounding half to even. The second version rounds @code{self} to the
nearest multiple of @code{Fraction(1, 10**ndigits)} (logically, if
@code{ndigits} is negative), again rounding half toward even.  This
method can also be accessed through the @ref{e75,,round()} function.
@end deffn
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{bd,,numbers}

The abstract base classes making up the numeric tower.
@end table


@node random — Generate pseudo-random numbers,statistics — Mathematical statistics functions,fractions — Rational numbers,Numeric and Mathematical Modules
@anchor{library/random doc}@anchor{1fa1}@anchor{library/random module-random}@anchor{d7}@anchor{library/random random-generate-pseudo-random-numbers}@anchor{1fa2}
@subsection @code{random} — Generate pseudo-random numbers


@geindex module; random

@strong{Source code:} Lib/random.py@footnote{https://github.com/python/cpython/tree/main/Lib/random.py}


__________________________________________________________________


This module implements pseudo-random number generators for various
distributions.

For integers, there is uniform selection from a range. For sequences, there is
uniform selection of a random element, a function to generate a random
permutation of a list in-place, and a function for random sampling without
replacement.

On the real line, there are functions to compute uniform, normal (Gaussian),
lognormal, negative exponential, gamma, and beta distributions. For generating
distributions of angles, the von Mises distribution is available.

Almost all module functions depend on the basic function @ref{1fa3,,random()}, which
generates a random float uniformly in the semi-open range [0.0, 1.0).  Python
uses the Mersenne Twister as the core generator.  It produces 53-bit precision
floats and has a period of 2**19937-1.  The underlying implementation in C is
both fast and threadsafe.  The Mersenne Twister is one of the most extensively
tested random number generators in existence.  However, being completely
deterministic, it is not suitable for all purposes, and is completely unsuitable
for cryptographic purposes.

The functions supplied by this module are actually bound methods of a hidden
instance of the @ref{110f,,random.Random} class.  You can instantiate your own
instances of @ref{110f,,Random} to get generators that don’t share state.

Class @ref{110f,,Random} can also be subclassed if you want to use a different
basic generator of your own devising: in that case, override the @code{random()},
@code{seed()}, @code{getstate()}, and @code{setstate()} methods.
Optionally, a new generator can supply a @code{getrandbits()} method — this
allows @ref{305,,randrange()} to produce selections over an arbitrarily large range.

The @ref{d7,,random} module also provides the @ref{1fa4,,SystemRandom} class which
uses the system function @ref{17f,,os.urandom()} to generate random numbers
from sources provided by the operating system.

@cartouche
@quotation Warning 
The pseudo-random generators of this module should not be used for
security purposes.  For security or cryptographic uses, see the
@ref{df,,secrets} module.
@end quotation
@end cartouche


@subsubheading See also


M. Matsumoto and T. Nishimura, “Mersenne Twister: A 623-dimensionally
equidistributed uniform pseudorandom number generator”, ACM Transactions on
Modeling and Computer Simulation Vol. 8, No. 1, January pp.3–30 1998.

Complementary-Multiply-with-Carry recipe@footnote{https://code.activestate.com/recipes/576707/} for a compatible alternative
random number generator with a long period and comparatively simple update
operations.


@menu
* Bookkeeping functions:: 
* Functions for bytes:: 
* Functions for integers:: 
* Functions for sequences:: 
* Real-valued distributions:: 
* Alternative Generator:: 
* Notes on Reproducibility:: 
* Examples: Examples<4>. 
* Recipes: Recipes<2>. 

@end menu

@node Bookkeeping functions,Functions for bytes,,random — Generate pseudo-random numbers
@anchor{library/random bookkeeping-functions}@anchor{1fa5}
@subsubsection Bookkeeping functions


@geindex seed() (in module random)
@anchor{library/random random seed}@anchor{e22}
@deffn {Function} random.seed (a=None, version=2)

Initialize the random number generator.

If @emph{a} is omitted or @code{None}, the current system time is used.  If
randomness sources are provided by the operating system, they are used
instead of the system time (see the @ref{17f,,os.urandom()} function for details
on availability).

If @emph{a} is an int, it is used directly.

With version 2 (the default), a @ref{2ef,,str}, @ref{2f0,,bytes}, or @ref{2f1,,bytearray}
object gets converted to an @ref{1b1,,int} and all of its bits are used.

With version 1 (provided for reproducing random sequences from older versions
of Python), the algorithm for @ref{2ef,,str} and @ref{2f0,,bytes} generates a
narrower range of seeds.


Changed in version 3.2: Moved to the version 2 scheme which uses all of the bits in a string seed.



Changed in version 3.11: The @emph{seed} must be one of the following types:
@emph{NoneType}, @ref{1b1,,int}, @ref{256,,float}, @ref{2ef,,str},
@ref{2f0,,bytes}, or @ref{2f1,,bytearray}.

@end deffn

@geindex getstate() (in module random)
@anchor{library/random random getstate}@anchor{1fa6}
@deffn {Function} random.getstate ()

Return an object capturing the current internal state of the generator.  This
object can be passed to @ref{1fa7,,setstate()} to restore the state.
@end deffn

@geindex setstate() (in module random)
@anchor{library/random random setstate}@anchor{1fa7}
@deffn {Function} random.setstate (state)

@emph{state} should have been obtained from a previous call to @ref{1fa6,,getstate()}, and
@ref{1fa7,,setstate()} restores the internal state of the generator to what it was at
the time @ref{1fa6,,getstate()} was called.
@end deffn

@node Functions for bytes,Functions for integers,Bookkeeping functions,random — Generate pseudo-random numbers
@anchor{library/random functions-for-bytes}@anchor{1fa8}
@subsubsection Functions for bytes


@geindex randbytes() (in module random)
@anchor{library/random random randbytes}@anchor{11f5}
@deffn {Function} random.randbytes (n)

Generate @emph{n} random bytes.

This method should not be used for generating security tokens.
Use @ref{1fa9,,secrets.token_bytes()} instead.


New in version 3.9.

@end deffn

@node Functions for integers,Functions for sequences,Functions for bytes,random — Generate pseudo-random numbers
@anchor{library/random functions-for-integers}@anchor{1faa}
@subsubsection Functions for integers


@geindex randrange() (in module random)
@anchor{library/random random randrange}@anchor{305}
@deffn {Function} random.randrange (stop)

@deffnx {Function} random.randrange (start, stop[, step])

Return a randomly selected element from @code{range(start, stop, step)}.  This is
equivalent to @code{choice(range(start, stop, step))}, but doesn’t actually build a
range object.

The positional argument pattern matches that of @ref{438,,range()}.  Keyword arguments
should not be used because the function may use them in unexpected ways.


Changed in version 3.2: @ref{305,,randrange()} is more sophisticated about producing equally distributed
values.  Formerly it used a style like @code{int(random()*n)} which could produce
slightly uneven distributions.



Deprecated since version 3.10: The automatic conversion of non-integer types to equivalent integers is
deprecated.  Currently @code{randrange(10.0)} is losslessly converted to
@code{randrange(10)}.  In the future, this will raise a @ref{162,,TypeError}.



Deprecated since version 3.10: The exception raised for non-integral values such as @code{randrange(10.5)}
or @code{randrange('10')} will be changed from @ref{17c,,ValueError} to
@ref{162,,TypeError}.

@end deffn

@geindex randint() (in module random)
@anchor{library/random random randint}@anchor{ddb}
@deffn {Function} random.randint (a, b)

Return a random integer @emph{N} such that @code{a <= N <= b}.  Alias for
@code{randrange(a, b+1)}.
@end deffn

@geindex getrandbits() (in module random)
@anchor{library/random random getrandbits}@anchor{b94}
@deffn {Function} random.getrandbits (k)

Returns a non-negative Python integer with @emph{k} random bits. This method
is supplied with the MersenneTwister generator and some other generators
may also provide it as an optional part of the API. When available,
@ref{b94,,getrandbits()} enables @ref{305,,randrange()} to handle arbitrarily large
ranges.


Changed in version 3.9: This method now accepts zero for @emph{k}.

@end deffn

@node Functions for sequences,Real-valued distributions,Functions for integers,random — Generate pseudo-random numbers
@anchor{library/random functions-for-sequences}@anchor{1fab}
@subsubsection Functions for sequences


@geindex choice() (in module random)
@anchor{library/random random choice}@anchor{ddc}
@deffn {Function} random.choice (seq)

Return a random element from the non-empty sequence @emph{seq}. If @emph{seq} is empty,
raises @ref{1066,,IndexError}.
@end deffn

@geindex choices() (in module random)
@anchor{library/random random choices}@anchor{80f}
@deffn {Function} random.choices (population, weights=None, *, cum_weights=None, k=1)

Return a @emph{k} sized list of elements chosen from the @emph{population} with replacement.
If the @emph{population} is empty, raises @ref{1066,,IndexError}.

If a @emph{weights} sequence is specified, selections are made according to the
relative weights.  Alternatively, if a @emph{cum_weights} sequence is given, the
selections are made according to the cumulative weights (perhaps computed
using @ref{505,,itertools.accumulate()}).  For example, the relative weights
@code{[10, 5, 30, 5]} are equivalent to the cumulative weights
@code{[10, 15, 45, 50]}.  Internally, the relative weights are converted to
cumulative weights before making selections, so supplying the cumulative
weights saves work.

If neither @emph{weights} nor @emph{cum_weights} are specified, selections are made
with equal probability.  If a weights sequence is supplied, it must be
the same length as the @emph{population} sequence.  It is a @ref{162,,TypeError}
to specify both @emph{weights} and @emph{cum_weights}.

The @emph{weights} or @emph{cum_weights} can use any numeric type that interoperates
with the @ref{256,,float} values returned by @ref{d7,,random()} (that includes
integers, floats, and fractions but excludes decimals).  Weights are assumed
to be non-negative and finite.  A @ref{17c,,ValueError} is raised if all
weights are zero.

For a given seed, the @ref{80f,,choices()} function with equal weighting
typically produces a different sequence than repeated calls to
@ref{ddc,,choice()}.  The algorithm used by @ref{80f,,choices()} uses floating
point arithmetic for internal consistency and speed.  The algorithm used
by @ref{ddc,,choice()} defaults to integer arithmetic with repeated selections
to avoid small biases from round-off error.


New in version 3.6.



Changed in version 3.9: Raises a @ref{17c,,ValueError} if all weights are zero.

@end deffn

@geindex shuffle() (in module random)
@anchor{library/random random shuffle}@anchor{444}
@deffn {Function} random.shuffle (x)

Shuffle the sequence @emph{x} in place.

To shuffle an immutable sequence and return a new shuffled list, use
@code{sample(x, k=len(x))} instead.

Note that even for small @code{len(x)}, the total number of permutations of @emph{x}
can quickly grow larger than the period of most random number generators.
This implies that most permutations of a long sequence can never be
generated.  For example, a sequence of length 2080 is the largest that
can fit within the period of the Mersenne Twister random number generator.


Deprecated since version 3.9, removed in version 3.11: The optional parameter @emph{random}.

@end deffn

@geindex sample() (in module random)
@anchor{library/random random sample}@anchor{ddd}
@deffn {Function} random.sample (population, k, *, counts=None)

Return a @emph{k} length list of unique elements chosen from the population
sequence.  Used for random sampling without replacement.

Returns a new list containing elements from the population while leaving the
original population unchanged.  The resulting list is in selection order so that
all sub-slices will also be valid random samples.  This allows raffle winners
(the sample) to be partitioned into grand prize and second place winners (the
subslices).

Members of the population need not be @ref{2dc,,hashable} or unique.  If the population
contains repeats, then each occurrence is a possible selection in the sample.

Repeated elements can be specified one at a time or with the optional
keyword-only @emph{counts} parameter.  For example, @code{sample(['red', 'blue'],
counts=[4, 2], k=5)} is equivalent to @code{sample(['red', 'red', 'red', 'red',
'blue', 'blue'], k=5)}.

To choose a sample from a range of integers, use a @ref{438,,range()} object as an
argument.  This is especially fast and space efficient for sampling from a large
population:  @code{sample(range(10000000), k=60)}.

If the sample size is larger than the population size, a @ref{17c,,ValueError}
is raised.


Changed in version 3.9: Added the @emph{counts} parameter.



Changed in version 3.11: The @emph{population} must be a sequence.  Automatic conversion of sets
to lists is longer supported.

@end deffn

@node Real-valued distributions,Alternative Generator,Functions for sequences,random — Generate pseudo-random numbers
@anchor{library/random id1}@anchor{1fac}@anchor{library/random real-valued-distributions}@anchor{1fad}
@subsubsection Real-valued distributions


The following functions generate specific real-valued distributions. Function
parameters are named after the corresponding variables in the distribution’s
equation, as used in common mathematical practice; most of these equations can
be found in any statistics text.

@geindex random() (in module random)
@anchor{library/random random random}@anchor{1fa3}
@deffn {Function} random.random ()

Return the next random floating point number in the range [0.0, 1.0).
@end deffn

@geindex uniform() (in module random)
@anchor{library/random random uniform}@anchor{1fae}
@deffn {Function} random.uniform (a, b)

Return a random floating point number @emph{N} such that @code{a <= N <= b} for
@code{a <= b} and @code{b <= N <= a} for @code{b < a}.

The end-point value @code{b} may or may not be included in the range
depending on floating-point rounding in the equation @code{a + (b-a) * random()}.
@end deffn

@geindex triangular() (in module random)
@anchor{library/random random triangular}@anchor{1faf}
@deffn {Function} random.triangular (low, high, mode)

Return a random floating point number @emph{N} such that @code{low <= N <= high} and
with the specified @emph{mode} between those bounds.  The @emph{low} and @emph{high} bounds
default to zero and one.  The @emph{mode} argument defaults to the midpoint
between the bounds, giving a symmetric distribution.
@end deffn

@geindex betavariate() (in module random)
@anchor{library/random random betavariate}@anchor{1fb0}
@deffn {Function} random.betavariate (alpha, beta)

Beta distribution.  Conditions on the parameters are @code{alpha > 0} and
@code{beta > 0}. Returned values range between 0 and 1.
@end deffn

@geindex expovariate() (in module random)
@anchor{library/random random expovariate}@anchor{1fb1}
@deffn {Function} random.expovariate (lambd)

Exponential distribution.  @emph{lambd} is 1.0 divided by the desired
mean.  It should be nonzero.  (The parameter would be called
“lambda”, but that is a reserved word in Python.)  Returned values
range from 0 to positive infinity if @emph{lambd} is positive, and from
negative infinity to 0 if @emph{lambd} is negative.
@end deffn

@geindex gammavariate() (in module random)
@anchor{library/random random gammavariate}@anchor{1fb2}
@deffn {Function} random.gammavariate (alpha, beta)

Gamma distribution.  (@emph{Not} the gamma function!)  Conditions on the
parameters are @code{alpha > 0} and @code{beta > 0}.

The probability distribution function is:

@example
          x ** (alpha - 1) * math.exp(-x / beta)
pdf(x) =  --------------------------------------
            math.gamma(alpha) * beta ** alpha
@end example
@end deffn

@geindex gauss() (in module random)
@anchor{library/random random gauss}@anchor{108e}
@deffn {Function} random.gauss (mu=0.0, sigma=1.0)

Normal distribution, also called the Gaussian distribution.  @emph{mu} is the mean,
and @emph{sigma} is the standard deviation.  This is slightly faster than
the @ref{108f,,normalvariate()} function defined below.

Multithreading note:  When two threads call this function
simultaneously, it is possible that they will receive the
same return value.  This can be avoided in three ways.
1) Have each thread use a different instance of the random
number generator. 2) Put locks around all calls. 3) Use the
slower, but thread-safe @ref{108f,,normalvariate()} function instead.


Changed in version 3.11: @emph{mu} and @emph{sigma} now have default arguments.

@end deffn

@geindex lognormvariate() (in module random)
@anchor{library/random random lognormvariate}@anchor{1fb3}
@deffn {Function} random.lognormvariate (mu, sigma)

Log normal distribution.  If you take the natural logarithm of this
distribution, you’ll get a normal distribution with mean @emph{mu} and standard
deviation @emph{sigma}.  @emph{mu} can have any value, and @emph{sigma} must be greater than
zero.
@end deffn

@geindex normalvariate() (in module random)
@anchor{library/random random normalvariate}@anchor{108f}
@deffn {Function} random.normalvariate (mu=0.0, sigma=1.0)

Normal distribution.  @emph{mu} is the mean, and @emph{sigma} is the standard deviation.


Changed in version 3.11: @emph{mu} and @emph{sigma} now have default arguments.

@end deffn

@geindex vonmisesvariate() (in module random)
@anchor{library/random random vonmisesvariate}@anchor{1fb4}
@deffn {Function} random.vonmisesvariate (mu, kappa)

@emph{mu} is the mean angle, expressed in radians between 0 and 2*@emph{pi}, and @emph{kappa}
is the concentration parameter, which must be greater than or equal to zero.  If
@emph{kappa} is equal to zero, this distribution reduces to a uniform random angle
over the range 0 to 2*@emph{pi}.
@end deffn

@geindex paretovariate() (in module random)
@anchor{library/random random paretovariate}@anchor{1fb5}
@deffn {Function} random.paretovariate (alpha)

Pareto distribution.  @emph{alpha} is the shape parameter.
@end deffn

@geindex weibullvariate() (in module random)
@anchor{library/random random weibullvariate}@anchor{1fb6}
@deffn {Function} random.weibullvariate (alpha, beta)

Weibull distribution.  @emph{alpha} is the scale parameter and @emph{beta} is the shape
parameter.
@end deffn

@node Alternative Generator,Notes on Reproducibility,Real-valued distributions,random — Generate pseudo-random numbers
@anchor{library/random alternative-generator}@anchor{1fb7}
@subsubsection Alternative Generator


@geindex Random (class in random)
@anchor{library/random random Random}@anchor{110f}
@deffn {Class} random.Random ([seed])

Class that implements the default pseudo-random number generator used by the
@ref{d7,,random} module.


Deprecated since version 3.9: In the future, the @emph{seed} must be one of the following types:
@code{NoneType}, @ref{1b1,,int}, @ref{256,,float}, @ref{2ef,,str},
@ref{2f0,,bytes}, or @ref{2f1,,bytearray}.

@end deffn

@geindex SystemRandom (class in random)
@anchor{library/random random SystemRandom}@anchor{1fa4}
@deffn {Class} random.SystemRandom ([seed])

Class that uses the @ref{17f,,os.urandom()} function for generating random numbers
from sources provided by the operating system. Not available on all systems.
Does not rely on software state, and sequences are not reproducible. Accordingly,
the @ref{e22,,seed()} method has no effect and is ignored.
The @ref{1fa6,,getstate()} and @ref{1fa7,,setstate()} methods raise
@ref{8a3,,NotImplementedError} if called.
@end deffn

@node Notes on Reproducibility,Examples<4>,Alternative Generator,random — Generate pseudo-random numbers
@anchor{library/random notes-on-reproducibility}@anchor{1fb8}
@subsubsection Notes on Reproducibility


Sometimes it is useful to be able to reproduce the sequences given by a
pseudo-random number generator.  By re-using a seed value, the same sequence should be
reproducible from run to run as long as multiple threads are not running.

Most of the random module’s algorithms and seeding functions are subject to
change across Python versions, but two aspects are guaranteed not to change:


@itemize *

@item 
If a new seeding method is added, then a backward compatible seeder will be
offered.

@item 
The generator’s @code{random()} method will continue to produce the same
sequence when the compatible seeder is given the same seed.
@end itemize

@node Examples<4>,Recipes<2>,Notes on Reproducibility,random — Generate pseudo-random numbers
@anchor{library/random examples}@anchor{1fb9}@anchor{library/random random-examples}@anchor{d7e}
@subsubsection Examples


Basic examples:

@example
>>> random()                             # Random float:  0.0 <= x < 1.0
0.37444887175646646

>>> uniform(2.5, 10.0)                   # Random float:  2.5 <= x <= 10.0
3.1800146073117523

>>> expovariate(1 / 5)                   # Interval between arrivals averaging 5 seconds
5.148957571865031

>>> randrange(10)                        # Integer from 0 to 9 inclusive
7

>>> randrange(0, 101, 2)                 # Even integer from 0 to 100 inclusive
26

>>> choice(['win', 'lose', 'draw'])      # Single random element from a sequence
'draw'

>>> deck = 'ace two three four'.split()
>>> shuffle(deck)                        # Shuffle a list
>>> deck
['four', 'two', 'ace', 'three']

>>> sample([10, 20, 30, 40, 50], k=4)    # Four samples without replacement
[40, 10, 50, 30]
@end example

Simulations:

@example
>>> # Six roulette wheel spins (weighted sampling with replacement)
>>> choices(['red', 'black', 'green'], [18, 18, 2], k=6)
['red', 'green', 'black', 'black', 'red', 'black']

>>> # Deal 20 cards without replacement from a deck
>>> # of 52 playing cards, and determine the proportion of cards
>>> # with a ten-value:  ten, jack, queen, or king.
>>> dealt = sample(['tens', 'low cards'], counts=[16, 36], k=20)
>>> dealt.count('tens') / 20
0.15

>>> # Estimate the probability of getting 5 or more heads from 7 spins
>>> # of a biased coin that settles on heads 60% of the time.
>>> def trial():
...     return choices('HT', cum_weights=(0.60, 1.00), k=7).count('H') >= 5
...
>>> sum(trial() for i in range(10_000)) / 10_000
0.4169

>>> # Probability of the median of 5 samples being in middle two quartiles
>>> def trial():
...     return 2_500 <= sorted(choices(range(10_000), k=5))[2] < 7_500
...
>>> sum(trial() for i in range(10_000)) / 10_000
0.7958
@end example

Example of statistical bootstrapping@footnote{https://en.wikipedia.org/wiki/Bootstrapping_(statistics)} using resampling
with replacement to estimate a confidence interval for the mean of a sample:

@example
# http://statistics.about.com/od/Applications/a/Example-Of-Bootstrapping.htm
from statistics import fmean as mean
from random import choices

data = [41, 50, 29, 37, 81, 30, 73, 63, 20, 35, 68, 22, 60, 31, 95]
means = sorted(mean(choices(data, k=len(data))) for i in range(100))
print(f'The sample mean of @{mean(data):.1f@} has a 90% confidence '
      f'interval from @{means[5]:.1f@} to @{means[94]:.1f@}')
@end example

Example of a resampling permutation test@footnote{https://en.wikipedia.org/wiki/Resampling_(statistics)#Permutation_tests}
to determine the statistical significance or p-value@footnote{https://en.wikipedia.org/wiki/P-value} of an observed difference
between the effects of a drug versus a placebo:

@example
# Example from "Statistics is Easy" by Dennis Shasha and Manda Wilson
from statistics import fmean as mean
from random import shuffle

drug = [54, 73, 53, 70, 73, 68, 52, 65, 65]
placebo = [54, 51, 58, 44, 55, 52, 42, 47, 58, 46]
observed_diff = mean(drug) - mean(placebo)

n = 10_000
count = 0
combined = drug + placebo
for i in range(n):
    shuffle(combined)
    new_diff = mean(combined[:len(drug)]) - mean(combined[len(drug):])
    count += (new_diff >= observed_diff)

print(f'@{n@} label reshufflings produced only @{count@} instances with a difference')
print(f'at least as extreme as the observed difference of @{observed_diff:.1f@}.')
print(f'The one-sided p-value of @{count / n:.4f@} leads us to reject the null')
print(f'hypothesis that there is no difference between the drug and the placebo.')
@end example

Simulation of arrival times and service deliveries for a multiserver queue:

@example
from heapq import heapify, heapreplace
from random import expovariate, gauss
from statistics import mean, quantiles

average_arrival_interval = 5.6
average_service_time = 15.0
stdev_service_time = 3.5
num_servers = 3

waits = []
arrival_time = 0.0
servers = [0.0] * num_servers  # time when each server becomes available
heapify(servers)
for i in range(1_000_000):
    arrival_time += expovariate(1.0 / average_arrival_interval)
    next_server_available = servers[0]
    wait = max(0.0, next_server_available - arrival_time)
    waits.append(wait)
    service_duration = max(0.0, gauss(average_service_time, stdev_service_time))
    service_completed = arrival_time + wait + service_duration
    heapreplace(servers, service_completed)

print(f'Mean wait: @{mean(waits):.1f@}   Max wait: @{max(waits):.1f@}')
print('Quartiles:', [round(q, 1) for q in quantiles(waits)])
@end example


@subsubheading See also


Statistics for Hackers@footnote{https://www.youtube.com/watch?v=Iq9DzN6mvYA}
a video tutorial by
Jake Vanderplas@footnote{https://us.pycon.org/2016/speaker/profile/295/}
on statistical analysis using just a few fundamental concepts
including simulation, sampling, shuffling, and cross-validation.

Economics Simulation@footnote{http://nbviewer.jupyter.org/url/norvig.com/ipython/Economics.ipynb}
a simulation of a marketplace by
Peter Norvig@footnote{http://norvig.com/bio.html} that shows effective
use of many of the tools and distributions provided by this module
(gauss, uniform, sample, betavariate, choice, triangular, and randrange).

A Concrete Introduction to Probability (using Python)@footnote{http://nbviewer.jupyter.org/url/norvig.com/ipython/Probability.ipynb}
a tutorial by Peter Norvig@footnote{http://norvig.com/bio.html} covering
the basics of probability theory, how to write simulations, and
how to perform data analysis using Python.


@node Recipes<2>,,Examples<4>,random — Generate pseudo-random numbers
@anchor{library/random recipes}@anchor{1fba}
@subsubsection Recipes


The default @ref{1fa3,,random()} returns multiples of 2⁻⁵³ in the range
@emph{0.0 ≤ x < 1.0}.  All such numbers are evenly spaced and are exactly
representable as Python floats.  However, many other representable
floats in that interval are not possible selections.  For example,
@code{0.05954861408025609} isn’t an integer multiple of 2⁻⁵³.

The following recipe takes a different approach.  All floats in the
interval are possible selections.  The mantissa comes from a uniform
distribution of integers in the range @emph{2⁵² ≤ mantissa < 2⁵³}.  The
exponent comes from a geometric distribution where exponents smaller
than @emph{-53} occur half as often as the next larger exponent.

@example
from random import Random
from math import ldexp

class FullRandom(Random):

    def random(self):
        mantissa = 0x10_0000_0000_0000 | self.getrandbits(52)
        exponent = -53
        x = 0
        while not x:
            x = self.getrandbits(32)
            exponent += x.bit_length() - 32
        return ldexp(mantissa, exponent)
@end example

All @ref{1fad,,real valued distributions}
in the class will use the new method:

@example
>>> fr = FullRandom()
>>> fr.random()
0.05954861408025609
>>> fr.expovariate(0.25)
8.87925541791544
@end example

The recipe is conceptually equivalent to an algorithm that chooses from
all the multiples of 2⁻¹⁰⁷⁴ in the range @emph{0.0 ≤ x < 1.0}.  All such
numbers are evenly spaced, but most have to be rounded down to the
nearest representable Python float.  (The value 2⁻¹⁰⁷⁴ is the smallest
positive unnormalized float and is equal to @code{math.ulp(0.0)}.)


@subsubheading See also


Generating Pseudo-random Floating-Point Values@footnote{https://allendowney.com/research/rand/downey07randfloat.pdf} a
paper by Allen B. Downey describing ways to generate more
fine-grained floats than normally generated by @ref{1fa3,,random()}.


@node statistics — Mathematical statistics functions,,random — Generate pseudo-random numbers,Numeric and Mathematical Modules
@anchor{library/statistics doc}@anchor{1fbb}@anchor{library/statistics module-statistics}@anchor{f0}@anchor{library/statistics statistics-mathematical-statistics-functions}@anchor{1fbc}
@subsection @code{statistics} — Mathematical statistics functions


@geindex module; statistics


New in version 3.4.


@strong{Source code:} Lib/statistics.py@footnote{https://github.com/python/cpython/tree/main/Lib/statistics.py}

@c from statistics import *
@c __name__ = '<doctest>'


__________________________________________________________________


This module provides functions for calculating mathematical statistics of
numeric (@ref{178c,,Real}-valued) data.

The module is not intended to be a competitor to third-party libraries such
as NumPy@footnote{https://numpy.org}, SciPy@footnote{https://www.scipy.org/}, or
proprietary full-featured statistics packages aimed at professional
statisticians such as Minitab, SAS and Matlab. It is aimed at the level of
graphing and scientific calculators.

Unless explicitly noted, these functions support @ref{1b1,,int},
@ref{256,,float}, @ref{248,,Decimal} and @ref{172,,Fraction}.
Behaviour with other types (whether in the numeric tower or not) is
currently unsupported.  Collections with a mix of types are also undefined
and implementation-dependent.  If your input data consists of mixed types,
you may be able to use @ref{2f4,,map()} to ensure a consistent result, for
example: @code{map(float, input_data)}.

@menu
* Averages and measures of central location:: 
* Measures of spread:: 
* Statistics for relations between two inputs:: 
* Function details:: 
* Exceptions: Exceptions<4>. 
* NormalDist objects:: 

@end menu

@node Averages and measures of central location,Measures of spread,,statistics — Mathematical statistics functions
@anchor{library/statistics averages-and-measures-of-central-location}@anchor{1fbd}
@subsubsection Averages and measures of central location


These functions calculate an average or typical value from a population
or sample.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{4cc,,mean()}

@tab

Arithmetic mean (“average”) of data.

@item

@ref{545,,fmean()}

@tab

Fast, floating point arithmetic mean, with optional weighting.

@item

@ref{546,,geometric_mean()}

@tab

Geometric mean of data.

@item

@ref{82a,,harmonic_mean()}

@tab

Harmonic mean of data.

@item

@ref{1fbe,,median()}

@tab

Median (middle value) of data.

@item

@ref{1fbf,,median_low()}

@tab

Low median of data.

@item

@ref{1fc0,,median_high()}

@tab

High median of data.

@item

@ref{1fc1,,median_grouped()}

@tab

Median, or 50th percentile, of grouped data.

@item

@ref{5b0,,mode()}

@tab

Single mode (most common value) of discrete or nominal data.

@item

@ref{547,,multimode()}

@tab

List of modes (most common values) of discrete or nominal data.

@item

@ref{548,,quantiles()}

@tab

Divide data into intervals with equal probability.

@end multitable


@node Measures of spread,Statistics for relations between two inputs,Averages and measures of central location,statistics — Mathematical statistics functions
@anchor{library/statistics measures-of-spread}@anchor{1fc2}
@subsubsection Measures of spread


These functions calculate a measure of how much the population or sample
tends to deviate from the typical or average values.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{1fc3,,pstdev()}

@tab

Population standard deviation of data.

@item

@ref{1fc4,,pvariance()}

@tab

Population variance of data.

@item

@ref{1fc5,,stdev()}

@tab

Sample standard deviation of data.

@item

@ref{1fc6,,variance()}

@tab

Sample variance of data.

@end multitable


@node Statistics for relations between two inputs,Function details,Measures of spread,statistics — Mathematical statistics functions
@anchor{library/statistics statistics-for-relations-between-two-inputs}@anchor{1fc7}
@subsubsection Statistics for relations between two inputs


These functions calculate statistics regarding relations between two inputs.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{2b2,,covariance()}

@tab

Sample covariance for two variables.

@item

@ref{2b3,,correlation()}

@tab

Pearson’s correlation coefficient for two variables.

@item

@ref{2b4,,linear_regression()}

@tab

Slope and intercept for simple linear regression.

@end multitable


@node Function details,Exceptions<4>,Statistics for relations between two inputs,statistics — Mathematical statistics functions
@anchor{library/statistics function-details}@anchor{1fc8}
@subsubsection Function details


Note: The functions do not require the data given to them to be sorted.
However, for reading convenience, most of the examples show sorted sequences.

@geindex mean() (in module statistics)
@anchor{library/statistics statistics mean}@anchor{4cc}
@deffn {Function} statistics.mean (data)

Return the sample arithmetic mean of @emph{data} which can be a sequence or iterable.

The arithmetic mean is the sum of the data divided by the number of data
points.  It is commonly called “the average”, although it is only one of many
different mathematical averages.  It is a measure of the central location of
the data.

If @emph{data} is empty, @ref{1fc9,,StatisticsError} will be raised.

Some examples of use:

@example
>>> mean([1, 2, 3, 4, 4])
2.8
>>> mean([-1.0, 2.5, 3.25, 5.75])
2.625

>>> from fractions import Fraction as F
>>> mean([F(3, 7), F(1, 21), F(5, 3), F(1, 3)])
Fraction(13, 21)

>>> from decimal import Decimal as D
>>> mean([D("0.5"), D("0.75"), D("0.625"), D("0.375")])
Decimal('0.5625')
@end example

@cartouche
@quotation Note 
The mean is strongly affected by outliers@footnote{https://en.wikipedia.org/wiki/Outlier} and is not necessarily a
typical example of the data points. For a more robust, although less
efficient, measure of central tendency@footnote{https://en.wikipedia.org/wiki/Central_tendency}, see @ref{1fbe,,median()}.

The sample mean gives an unbiased estimate of the true population mean,
so that when taken on average over all the possible samples,
@code{mean(sample)} converges on the true mean of the entire population.  If
@emph{data} represents the entire population rather than a sample, then
@code{mean(data)} is equivalent to calculating the true population mean μ.
@end quotation
@end cartouche
@end deffn

@geindex fmean() (in module statistics)
@anchor{library/statistics statistics fmean}@anchor{545}
@deffn {Function} statistics.fmean (data, weights=None)

Convert @emph{data} to floats and compute the arithmetic mean.

This runs faster than the @ref{4cc,,mean()} function and it always returns a
@ref{256,,float}.  The @emph{data} may be a sequence or iterable.  If the input
dataset is empty, raises a @ref{1fc9,,StatisticsError}.

@example
>>> fmean([3.5, 4.0, 5.25])
4.25
@end example

Optional weighting is supported.  For example, a professor assigns a
grade for a course by weighting quizzes at 20%, homework at 20%, a
midterm exam at 30%, and a final exam at 30%:

@example
>>> grades = [85, 92, 83, 91]
>>> weights = [0.20, 0.20, 0.30, 0.30]
>>> fmean(grades, weights)
87.6
@end example

If @emph{weights} is supplied, it must be the same length as the @emph{data} or
a @ref{17c,,ValueError} will be raised.


New in version 3.8.



Changed in version 3.11: Added support for @emph{weights}.

@end deffn

@geindex geometric_mean() (in module statistics)
@anchor{library/statistics statistics geometric_mean}@anchor{546}
@deffn {Function} statistics.geometric_mean (data)

Convert @emph{data} to floats and compute the geometric mean.

The geometric mean indicates the central tendency or typical value of the
@emph{data} using the product of the values (as opposed to the arithmetic mean
which uses their sum).

Raises a @ref{1fc9,,StatisticsError} if the input dataset is empty,
if it contains a zero, or if it contains a negative value.
The @emph{data} may be a sequence or iterable.

No special efforts are made to achieve exact results.
(However, this may change in the future.)

@example
>>> round(geometric_mean([54, 24, 36]), 1)
36.0
@end example


New in version 3.8.

@end deffn

@geindex harmonic_mean() (in module statistics)
@anchor{library/statistics statistics harmonic_mean}@anchor{82a}
@deffn {Function} statistics.harmonic_mean (data, weights=None)

Return the harmonic mean of @emph{data}, a sequence or iterable of
real-valued numbers.  If @emph{weights} is omitted or @emph{None}, then
equal weighting is assumed.

The harmonic mean is the reciprocal of the arithmetic @ref{4cc,,mean()} of the
reciprocals of the data. For example, the harmonic mean of three values @emph{a},
@emph{b} and @emph{c} will be equivalent to @code{3/(1/a + 1/b + 1/c)}.  If one of the
values is zero, the result will be zero.

The harmonic mean is a type of average, a measure of the central
location of the data.  It is often appropriate when averaging
ratios or rates, for example speeds.

Suppose a car travels 10 km at 40 km/hr, then another 10 km at 60 km/hr.
What is the average speed?

@example
>>> harmonic_mean([40, 60])
48.0
@end example

Suppose a car travels 40 km/hr for 5 km, and when traffic clears,
speeds-up to 60 km/hr for the remaining 30 km of the journey. What
is the average speed?

@example
>>> harmonic_mean([40, 60], weights=[5, 30])
56.0
@end example

@ref{1fc9,,StatisticsError} is raised if @emph{data} is empty, any element
is less than zero, or if the weighted sum isn’t positive.

The current algorithm has an early-out when it encounters a zero
in the input.  This means that the subsequent inputs are not tested
for validity.  (This behavior may change in the future.)


New in version 3.6.



Changed in version 3.10: Added support for @emph{weights}.

@end deffn

@geindex median() (in module statistics)
@anchor{library/statistics statistics median}@anchor{1fbe}
@deffn {Function} statistics.median (data)

Return the median (middle value) of numeric data, using the common “mean of
middle two” method.  If @emph{data} is empty, @ref{1fc9,,StatisticsError} is raised.
@emph{data} can be a sequence or iterable.

The median is a robust measure of central location and is less affected by
the presence of outliers.  When the number of data points is odd, the
middle data point is returned:

@example
>>> median([1, 3, 5])
3
@end example

When the number of data points is even, the median is interpolated by taking
the average of the two middle values:

@example
>>> median([1, 3, 5, 7])
4.0
@end example

This is suited for when your data is discrete, and you don’t mind that the
median may not be an actual data point.

If the data is ordinal (supports order operations) but not numeric (doesn’t
support addition), consider using @ref{1fbf,,median_low()} or @ref{1fc0,,median_high()}
instead.
@end deffn

@geindex median_low() (in module statistics)
@anchor{library/statistics statistics median_low}@anchor{1fbf}
@deffn {Function} statistics.median_low (data)

Return the low median of numeric data.  If @emph{data} is empty,
@ref{1fc9,,StatisticsError} is raised.  @emph{data} can be a sequence or iterable.

The low median is always a member of the data set.  When the number of data
points is odd, the middle value is returned.  When it is even, the smaller of
the two middle values is returned.

@example
>>> median_low([1, 3, 5])
3
>>> median_low([1, 3, 5, 7])
3
@end example

Use the low median when your data are discrete and you prefer the median to
be an actual data point rather than interpolated.
@end deffn

@geindex median_high() (in module statistics)
@anchor{library/statistics statistics median_high}@anchor{1fc0}
@deffn {Function} statistics.median_high (data)

Return the high median of data.  If @emph{data} is empty, @ref{1fc9,,StatisticsError}
is raised.  @emph{data} can be a sequence or iterable.

The high median is always a member of the data set.  When the number of data
points is odd, the middle value is returned.  When it is even, the larger of
the two middle values is returned.

@example
>>> median_high([1, 3, 5])
3
>>> median_high([1, 3, 5, 7])
5
@end example

Use the high median when your data are discrete and you prefer the median to
be an actual data point rather than interpolated.
@end deffn

@geindex median_grouped() (in module statistics)
@anchor{library/statistics statistics median_grouped}@anchor{1fc1}
@deffn {Function} statistics.median_grouped (data, interval=1)

Return the median of grouped continuous data, calculated as the 50th
percentile, using interpolation.  If @emph{data} is empty, @ref{1fc9,,StatisticsError}
is raised.  @emph{data} can be a sequence or iterable.

@example
>>> median_grouped([52, 52, 53, 54])
52.5
@end example

In the following example, the data are rounded, so that each value represents
the midpoint of data classes, e.g. 1 is the midpoint of the class 0.5–1.5, 2
is the midpoint of 1.5–2.5, 3 is the midpoint of 2.5–3.5, etc.  With the data
given, the middle value falls somewhere in the class 3.5–4.5, and
interpolation is used to estimate it:

@example
>>> median_grouped([1, 2, 2, 3, 4, 4, 4, 4, 4, 5])
3.7
@end example

Optional argument @emph{interval} represents the class interval, and defaults
to 1.  Changing the class interval naturally will change the interpolation:

@example
>>> median_grouped([1, 3, 3, 5, 7], interval=1)
3.25
>>> median_grouped([1, 3, 3, 5, 7], interval=2)
3.5
@end example

This function does not check whether the data points are at least
@emph{interval} apart.

@strong{CPython implementation detail:} Under some circumstances, @ref{1fc1,,median_grouped()} may coerce data points to
floats.  This behaviour is likely to change in the future.


@subsubheading See also



@itemize *

@item 
“Statistics for the Behavioral Sciences”, Frederick J Gravetter and
Larry B Wallnau (8th Edition).

@item 
The SSMEDIAN@footnote{https://help.gnome.org/users/gnumeric/stable/gnumeric.html#gnumeric-function-SSMEDIAN}
function in the Gnome Gnumeric spreadsheet, including this discussion@footnote{https://mail.gnome.org/archives/gnumeric-list/2011-April/msg00018.html}.
@end itemize

@end deffn

@geindex mode() (in module statistics)
@anchor{library/statistics statistics mode}@anchor{5b0}
@deffn {Function} statistics.mode (data)

Return the single most common data point from discrete or nominal @emph{data}.
The mode (when it exists) is the most typical value and serves as a
measure of central location.

If there are multiple modes with the same frequency, returns the first one
encountered in the @emph{data}.  If the smallest or largest of those is
desired instead, use @code{min(multimode(data))} or @code{max(multimode(data))}.
If the input @emph{data} is empty, @ref{1fc9,,StatisticsError} is raised.

@code{mode} assumes discrete data and returns a single value. This is the
standard treatment of the mode as commonly taught in schools:

@example
>>> mode([1, 1, 2, 3, 3, 3, 3, 4])
3
@end example

The mode is unique in that it is the only statistic in this package that
also applies to nominal (non-numeric) data:

@example
>>> mode(["red", "blue", "blue", "red", "green", "red", "red"])
'red'
@end example


Changed in version 3.8: Now handles multimodal datasets by returning the first mode encountered.
Formerly, it raised @ref{1fc9,,StatisticsError} when more than one mode was
found.

@end deffn

@geindex multimode() (in module statistics)
@anchor{library/statistics statistics multimode}@anchor{547}
@deffn {Function} statistics.multimode (data)

Return a list of the most frequently occurring values in the order they
were first encountered in the @emph{data}.  Will return more than one result if
there are multiple modes or an empty list if the @emph{data} is empty:

@example
>>> multimode('aabbbbccddddeeffffgg')
['b', 'd', 'f']
>>> multimode('@w{'})
[]
@end example


New in version 3.8.

@end deffn

@geindex pstdev() (in module statistics)
@anchor{library/statistics statistics pstdev}@anchor{1fc3}
@deffn {Function} statistics.pstdev (data, mu=None)

Return the population standard deviation (the square root of the population
variance).  See @ref{1fc4,,pvariance()} for arguments and other details.

@example
>>> pstdev([1.5, 2.5, 2.5, 2.75, 3.25, 4.75])
0.986893273527251
@end example
@end deffn

@geindex pvariance() (in module statistics)
@anchor{library/statistics statistics pvariance}@anchor{1fc4}
@deffn {Function} statistics.pvariance (data, mu=None)

Return the population variance of @emph{data}, a non-empty sequence or iterable
of real-valued numbers.  Variance, or second moment about the mean, is a
measure of the variability (spread or dispersion) of data.  A large
variance indicates that the data is spread out; a small variance indicates
it is clustered closely around the mean.

If the optional second argument @emph{mu} is given, it is typically the mean of
the @emph{data}.  It can also be used to compute the second moment around a
point that is not the mean.  If it is missing or @code{None} (the default),
the arithmetic mean is automatically calculated.

Use this function to calculate the variance from the entire population.  To
estimate the variance from a sample, the @ref{1fc6,,variance()} function is usually
a better choice.

Raises @ref{1fc9,,StatisticsError} if @emph{data} is empty.

Examples:

@example
>>> data = [0.0, 0.25, 0.25, 1.25, 1.5, 1.75, 2.75, 3.25]
>>> pvariance(data)
1.25
@end example

If you have already calculated the mean of your data, you can pass it as the
optional second argument @emph{mu} to avoid recalculation:

@example
>>> mu = mean(data)
>>> pvariance(data, mu)
1.25
@end example

Decimals and Fractions are supported:

@example
>>> from decimal import Decimal as D
>>> pvariance([D("27.5"), D("30.25"), D("30.25"), D("34.5"), D("41.75")])
Decimal('24.815')

>>> from fractions import Fraction as F
>>> pvariance([F(1, 4), F(5, 4), F(1, 2)])
Fraction(13, 72)
@end example

@cartouche
@quotation Note 
When called with the entire population, this gives the population variance
σ².  When called on a sample instead, this is the biased sample variance
s², also known as variance with N degrees of freedom.

If you somehow know the true population mean μ, you may use this
function to calculate the variance of a sample, giving the known
population mean as the second argument.  Provided the data points are a
random sample of the population, the result will be an unbiased estimate
of the population variance.
@end quotation
@end cartouche
@end deffn

@geindex stdev() (in module statistics)
@anchor{library/statistics statistics stdev}@anchor{1fc5}
@deffn {Function} statistics.stdev (data, xbar=None)

Return the sample standard deviation (the square root of the sample
variance).  See @ref{1fc6,,variance()} for arguments and other details.

@example
>>> stdev([1.5, 2.5, 2.5, 2.75, 3.25, 4.75])
1.0810874155219827
@end example
@end deffn

@geindex variance() (in module statistics)
@anchor{library/statistics statistics variance}@anchor{1fc6}
@deffn {Function} statistics.variance (data, xbar=None)

Return the sample variance of @emph{data}, an iterable of at least two real-valued
numbers.  Variance, or second moment about the mean, is a measure of the
variability (spread or dispersion) of data.  A large variance indicates that
the data is spread out; a small variance indicates it is clustered closely
around the mean.

If the optional second argument @emph{xbar} is given, it should be the mean of
@emph{data}.  If it is missing or @code{None} (the default), the mean is
automatically calculated.

Use this function when your data is a sample from a population. To calculate
the variance from the entire population, see @ref{1fc4,,pvariance()}.

Raises @ref{1fc9,,StatisticsError} if @emph{data} has fewer than two values.

Examples:

@example
>>> data = [2.75, 1.75, 1.25, 0.25, 0.5, 1.25, 3.5]
>>> variance(data)
1.3720238095238095
@end example

If you have already calculated the mean of your data, you can pass it as the
optional second argument @emph{xbar} to avoid recalculation:

@example
>>> m = mean(data)
>>> variance(data, m)
1.3720238095238095
@end example

This function does not attempt to verify that you have passed the actual mean
as @emph{xbar}.  Using arbitrary values for @emph{xbar} can lead to invalid or
impossible results.

Decimal and Fraction values are supported:

@example
>>> from decimal import Decimal as D
>>> variance([D("27.5"), D("30.25"), D("30.25"), D("34.5"), D("41.75")])
Decimal('31.01875')

>>> from fractions import Fraction as F
>>> variance([F(1, 6), F(1, 2), F(5, 3)])
Fraction(67, 108)
@end example

@cartouche
@quotation Note 
This is the sample variance s² with Bessel’s correction, also known as
variance with N-1 degrees of freedom.  Provided that the data points are
representative (e.g. independent and identically distributed), the result
should be an unbiased estimate of the true population variance.

If you somehow know the actual population mean μ you should pass it to the
@ref{1fc4,,pvariance()} function as the @emph{mu} parameter to get the variance of a
sample.
@end quotation
@end cartouche
@end deffn

@geindex quantiles() (in module statistics)
@anchor{library/statistics statistics quantiles}@anchor{548}
@deffn {Function} statistics.quantiles (data, *, n=4, method='exclusive')

Divide @emph{data} into @emph{n} continuous intervals with equal probability.
Returns a list of @code{n - 1} cut points separating the intervals.

Set @emph{n} to 4 for quartiles (the default).  Set @emph{n} to 10 for deciles.  Set
@emph{n} to 100 for percentiles which gives the 99 cuts points that separate
@emph{data} into 100 equal sized groups.  Raises @ref{1fc9,,StatisticsError} if @emph{n}
is not least 1.

The @emph{data} can be any iterable containing sample data.  For meaningful
results, the number of data points in @emph{data} should be larger than @emph{n}.
Raises @ref{1fc9,,StatisticsError} if there are not at least two data points.

The cut points are linearly interpolated from the
two nearest data points.  For example, if a cut point falls one-third
of the distance between two sample values, @code{100} and @code{112}, the
cut-point will evaluate to @code{104}.

The @emph{method} for computing quantiles can be varied depending on
whether the @emph{data} includes or excludes the lowest and
highest possible values from the population.

The default @emph{method} is “exclusive” and is used for data sampled from
a population that can have more extreme values than found in the
samples.  The portion of the population falling below the @emph{i-th} of
@emph{m} sorted data points is computed as @code{i / (m + 1)}.  Given nine
sample values, the method sorts them and assigns the following
percentiles: 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%.

Setting the @emph{method} to “inclusive” is used for describing population
data or for samples that are known to include the most extreme values
from the population.  The minimum value in @emph{data} is treated as the 0th
percentile and the maximum value is treated as the 100th percentile.
The portion of the population falling below the @emph{i-th} of @emph{m} sorted
data points is computed as @code{(i - 1) / (m - 1)}.  Given 11 sample
values, the method sorts them and assigns the following percentiles:
0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%.

@example
# Decile cut points for empirically sampled data
>>> data = [105, 129, 87, 86, 111, 111, 89, 81, 108, 92, 110,
...         100, 75, 105, 103, 109, 76, 119, 99, 91, 103, 129,
...         106, 101, 84, 111, 74, 87, 86, 103, 103, 106, 86,
...         111, 75, 87, 102, 121, 111, 88, 89, 101, 106, 95,
...         103, 107, 101, 81, 109, 104]
>>> [round(q, 1) for q in quantiles(data, n=10)]
[81.0, 86.2, 89.0, 99.4, 102.5, 103.6, 106.0, 109.8, 111.0]
@end example


New in version 3.8.

@end deffn

@geindex covariance() (in module statistics)
@anchor{library/statistics statistics covariance}@anchor{2b2}
@deffn {Function} statistics.covariance (x, y, /)

Return the sample covariance of two inputs @emph{x} and @emph{y}. Covariance
is a measure of the joint variability of two inputs.

Both inputs must be of the same length (no less than two), otherwise
@ref{1fc9,,StatisticsError} is raised.

Examples:

@example
>>> x = [1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> y = [1, 2, 3, 1, 2, 3, 1, 2, 3]
>>> covariance(x, y)
0.75
>>> z = [9, 8, 7, 6, 5, 4, 3, 2, 1]
>>> covariance(x, z)
-7.5
>>> covariance(z, x)
-7.5
@end example


New in version 3.10.

@end deffn

@geindex correlation() (in module statistics)
@anchor{library/statistics statistics correlation}@anchor{2b3}
@deffn {Function} statistics.correlation (x, y, /)

Return the Pearson's correlation coefficient@footnote{https://en.wikipedia.org/wiki/Pearson_correlation_coefficient}
for two inputs. Pearson’s correlation coefficient @emph{r} takes values
between -1 and +1. It measures the strength and direction of the linear
relationship, where +1 means very strong, positive linear relationship,
-1 very strong, negative linear relationship, and 0 no linear relationship.

Both inputs must be of the same length (no less than two), and need
not to be constant, otherwise @ref{1fc9,,StatisticsError} is raised.

Examples:

@example
>>> x = [1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> y = [9, 8, 7, 6, 5, 4, 3, 2, 1]
>>> correlation(x, x)
1.0
>>> correlation(x, y)
-1.0
@end example


New in version 3.10.

@end deffn

@geindex linear_regression() (in module statistics)
@anchor{library/statistics statistics linear_regression}@anchor{2b4}
@deffn {Function} statistics.linear_regression (x, y, /, *, proportional=False)

Return the slope and intercept of simple linear regression@footnote{https://en.wikipedia.org/wiki/Simple_linear_regression}
parameters estimated using ordinary least squares. Simple linear
regression describes the relationship between an independent variable @emph{x} and
a dependent variable @emph{y} in terms of this linear function:

@quotation

@emph{y = slope * x + intercept + noise}
@end quotation

where @code{slope} and @code{intercept} are the regression parameters that are
estimated, and @code{noise} represents the
variability of the data that was not explained by the linear regression
(it is equal to the difference between predicted and actual values
of the dependent variable).

Both inputs must be of the same length (no less than two), and
the independent variable @emph{x} cannot be constant;
otherwise a @ref{1fc9,,StatisticsError} is raised.

For example, we can use the release dates of the Monty Python films@footnote{https://en.wikipedia.org/wiki/Monty_Python#Films}
to predict the cumulative number of Monty Python films
that would have been produced by 2019
assuming that they had kept the pace.

@example
>>> year = [1971, 1975, 1979, 1982, 1983]
>>> films_total = [1, 2, 3, 4, 5]
>>> slope, intercept = linear_regression(year, films_total)
>>> round(slope * 2019 + intercept)
16
@end example

If @emph{proportional} is true, the independent variable @emph{x} and the
dependent variable @emph{y} are assumed to be directly proportional.
The data is fit to a line passing through the origin.
Since the @emph{intercept} will always be 0.0, the underlying linear
function simplifies to:

@quotation

@emph{y = slope * x + noise}
@end quotation


New in version 3.10.



Changed in version 3.11: Added support for @emph{proportional}.

@end deffn

@node Exceptions<4>,NormalDist objects,Function details,statistics — Mathematical statistics functions
@anchor{library/statistics exceptions}@anchor{1fca}
@subsubsection Exceptions


A single exception is defined:

@geindex StatisticsError
@anchor{library/statistics statistics StatisticsError}@anchor{1fc9}
@deffn {Exception} statistics.StatisticsError

Subclass of @ref{17c,,ValueError} for statistics-related exceptions.
@end deffn

@node NormalDist objects,,Exceptions<4>,statistics — Mathematical statistics functions
@anchor{library/statistics normaldist-objects}@anchor{1fcb}
@subsubsection @code{NormalDist} objects


@ref{549,,NormalDist} is a tool for creating and manipulating normal
distributions of a random variable@footnote{http://www.stat.yale.edu/Courses/1997-98/101/ranvar.htm}.  It is a
class that treats the mean and standard deviation of data
measurements as a single entity.

Normal distributions arise from the Central Limit Theorem@footnote{https://en.wikipedia.org/wiki/Central_limit_theorem} and have a wide range
of applications in statistics.

@geindex NormalDist (class in statistics)
@anchor{library/statistics statistics NormalDist}@anchor{549}
@deffn {Class} statistics.NormalDist (mu=0.0, sigma=1.0)

Returns a new @emph{NormalDist} object where @emph{mu} represents the arithmetic mean@footnote{https://en.wikipedia.org/wiki/Arithmetic_mean} and @emph{sigma}
represents the standard deviation@footnote{https://en.wikipedia.org/wiki/Standard_deviation}.

If @emph{sigma} is negative, raises @ref{1fc9,,StatisticsError}.

@geindex mean (statistics.NormalDist attribute)
@anchor{library/statistics statistics NormalDist mean}@anchor{1fcc}
@deffn {Attribute} mean

A read-only property for the arithmetic mean@footnote{https://en.wikipedia.org/wiki/Arithmetic_mean} of a normal
distribution.
@end deffn

@geindex median (statistics.NormalDist attribute)
@anchor{library/statistics statistics NormalDist median}@anchor{1fcd}
@deffn {Attribute} median

A read-only property for the median@footnote{https://en.wikipedia.org/wiki/Median} of a normal
distribution.
@end deffn

@geindex mode (statistics.NormalDist attribute)
@anchor{library/statistics statistics NormalDist mode}@anchor{1fce}
@deffn {Attribute} mode

A read-only property for the mode@footnote{https://en.wikipedia.org/wiki/Mode_(statistics)} of a normal
distribution.
@end deffn

@geindex stdev (statistics.NormalDist attribute)
@anchor{library/statistics statistics NormalDist stdev}@anchor{1fcf}
@deffn {Attribute} stdev

A read-only property for the standard deviation@footnote{https://en.wikipedia.org/wiki/Standard_deviation} of a normal
distribution.
@end deffn

@geindex variance (statistics.NormalDist attribute)
@anchor{library/statistics statistics NormalDist variance}@anchor{1fd0}
@deffn {Attribute} variance

A read-only property for the variance@footnote{https://en.wikipedia.org/wiki/Variance} of a normal
distribution. Equal to the square of the standard deviation.
@end deffn

@geindex from_samples() (statistics.NormalDist class method)
@anchor{library/statistics statistics NormalDist from_samples}@anchor{1fd1}
@deffn {Method} classmethod from_samples (data)

Makes a normal distribution instance with @emph{mu} and @emph{sigma} parameters
estimated from the @emph{data} using @ref{545,,fmean()} and @ref{1fc5,,stdev()}.

The @emph{data} can be any @ref{dcc,,iterable} and should consist of values
that can be converted to type @ref{256,,float}.  If @emph{data} does not
contain at least two elements, raises @ref{1fc9,,StatisticsError} because it
takes at least one point to estimate a central value and at least two
points to estimate dispersion.
@end deffn

@geindex samples() (statistics.NormalDist method)
@anchor{library/statistics statistics NormalDist samples}@anchor{1fd2}
@deffn {Method} samples (n, *, seed=None)

Generates @emph{n} random samples for a given mean and standard deviation.
Returns a @ref{43a,,list} of @ref{256,,float} values.

If @emph{seed} is given, creates a new instance of the underlying random
number generator.  This is useful for creating reproducible results,
even in a multi-threading context.
@end deffn

@geindex pdf() (statistics.NormalDist method)
@anchor{library/statistics statistics NormalDist pdf}@anchor{1fd3}
@deffn {Method} pdf (x)

Using a probability density function (pdf)@footnote{https://en.wikipedia.org/wiki/Probability_density_function}, compute
the relative likelihood that a random variable @emph{X} will be near the
given value @emph{x}.  Mathematically, it is the limit of the ratio @code{P(x <=
X < x+dx) / dx} as @emph{dx} approaches zero.

The relative likelihood is computed as the probability of a sample
occurring in a narrow range divided by the width of the range (hence
the word “density”).  Since the likelihood is relative to other points,
its value can be greater than @cite{1.0}.
@end deffn

@geindex cdf() (statistics.NormalDist method)
@anchor{library/statistics statistics NormalDist cdf}@anchor{1fd4}
@deffn {Method} cdf (x)

Using a cumulative distribution function (cdf)@footnote{https://en.wikipedia.org/wiki/Cumulative_distribution_function},
compute the probability that a random variable @emph{X} will be less than or
equal to @emph{x}.  Mathematically, it is written @code{P(X <= x)}.
@end deffn

@geindex inv_cdf() (statistics.NormalDist method)
@anchor{library/statistics statistics NormalDist inv_cdf}@anchor{1fd5}
@deffn {Method} inv_cdf (p)

Compute the inverse cumulative distribution function, also known as the
quantile function@footnote{https://en.wikipedia.org/wiki/Quantile_function}
or the percent-point@footnote{https://www.statisticshowto.datasciencecentral.com/inverse-distribution-function/}
function.  Mathematically, it is written @code{x : P(X <= x) = p}.

Finds the value @emph{x} of the random variable @emph{X} such that the
probability of the variable being less than or equal to that value
equals the given probability @emph{p}.
@end deffn

@geindex overlap() (statistics.NormalDist method)
@anchor{library/statistics statistics NormalDist overlap}@anchor{1fd6}
@deffn {Method} overlap (other)

Measures the agreement between two normal probability distributions.
Returns a value between 0.0 and 1.0 giving the overlapping area for the two probability density functions@footnote{https://www.rasch.org/rmt/rmt101r.htm}.
@end deffn

@geindex quantiles() (statistics.NormalDist method)
@anchor{library/statistics statistics NormalDist quantiles}@anchor{1fd7}
@deffn {Method} quantiles (n=4)

Divide the normal distribution into @emph{n} continuous intervals with
equal probability.  Returns a list of (n - 1) cut points separating
the intervals.

Set @emph{n} to 4 for quartiles (the default).  Set @emph{n} to 10 for deciles.
Set @emph{n} to 100 for percentiles which gives the 99 cuts points that
separate the normal distribution into 100 equal sized groups.
@end deffn

@geindex zscore() (statistics.NormalDist method)
@anchor{library/statistics statistics NormalDist zscore}@anchor{1fd8}
@deffn {Method} zscore (x)

Compute the
Standard Score@footnote{https://www.statisticshowto.com/probability-and-statistics/z-score/}
describing @emph{x} in terms of the number of standard deviations
above or below the mean of the normal distribution:
@code{(x - mean) / stdev}.


New in version 3.9.

@end deffn

Instances of @ref{549,,NormalDist} support addition, subtraction,
multiplication and division by a constant.  These operations
are used for translation and scaling.  For example:

@example
>>> temperature_february = NormalDist(5, 2.5)             # Celsius
>>> temperature_february * (9/5) + 32                     # Fahrenheit
NormalDist(mu=41.0, sigma=4.5)
@end example

Dividing a constant by an instance of @ref{549,,NormalDist} is not supported
because the result wouldn’t be normally distributed.

Since normal distributions arise from additive effects of independent
variables, it is possible to add and subtract two independent normally distributed random variables@footnote{https://en.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables}
represented as instances of @ref{549,,NormalDist}.  For example:

@example
>>> birth_weights = NormalDist.from_samples([2.5, 3.1, 2.1, 2.4, 2.7, 3.5])
>>> drug_effects = NormalDist(0.4, 0.15)
>>> combined = birth_weights + drug_effects
>>> round(combined.mean, 1)
3.1
>>> round(combined.stdev, 1)
0.5
@end example


New in version 3.8.

@end deffn

@menu
* NormalDist Examples and Recipes:: 

@end menu

@node NormalDist Examples and Recipes,,,NormalDist objects
@anchor{library/statistics normaldist-examples-and-recipes}@anchor{1fd9}
@subsubsection @code{NormalDist} Examples and Recipes


@ref{549,,NormalDist} readily solves classic probability problems.

For example, given historical data for SAT exams@footnote{https://nces.ed.gov/programs/digest/d17/tables/dt17_226.40.asp} showing
that scores are normally distributed with a mean of 1060 and a standard
deviation of 195, determine the percentage of students with test scores
between 1100 and 1200, after rounding to the nearest whole number:

@example
>>> sat = NormalDist(1060, 195)
>>> fraction = sat.cdf(1200 + 0.5) - sat.cdf(1100 - 0.5)
>>> round(fraction * 100.0, 1)
18.4
@end example

Find the quartiles@footnote{https://en.wikipedia.org/wiki/Quartile} and deciles@footnote{https://en.wikipedia.org/wiki/Decile} for the SAT scores:

@example
>>> list(map(round, sat.quantiles()))
[928, 1060, 1192]
>>> list(map(round, sat.quantiles(n=10)))
[810, 896, 958, 1011, 1060, 1109, 1162, 1224, 1310]
@end example

To estimate the distribution for a model than isn’t easy to solve
analytically, @ref{549,,NormalDist} can generate input samples for a Monte Carlo simulation@footnote{https://en.wikipedia.org/wiki/Monte_Carlo_method}:

@example
>>> def model(x, y, z):
...     return (3*x + 7*x*y - 5*y) / (11 * z)
...
>>> n = 100_000
>>> X = NormalDist(10, 2.5).samples(n, seed=3652260728)
>>> Y = NormalDist(15, 1.75).samples(n, seed=4582495471)
>>> Z = NormalDist(50, 1.25).samples(n, seed=6582483453)
>>> quantiles(map(model, X, Y, Z))       
[1.4591308524824727, 1.8035946855390597, 2.175091447274739]
@end example

Normal distributions can be used to approximate Binomial distributions@footnote{http://mathworld.wolfram.com/BinomialDistribution.html}
when the sample size is large and when the probability of a successful
trial is near 50%.

For example, an open source conference has 750 attendees and two rooms with a
500 person capacity.  There is a talk about Python and another about Ruby.
In previous conferences, 65% of the attendees preferred to listen to Python
talks.  Assuming the population preferences haven’t changed, what is the
probability that the Python room will stay within its capacity limits?

@example
>>> n = 750             # Sample size
>>> p = 0.65            # Preference for Python
>>> q = 1.0 - p         # Preference for Ruby
>>> k = 500             # Room capacity

>>> # Approximation using the cumulative normal distribution
>>> from math import sqrt
>>> round(NormalDist(mu=n*p, sigma=sqrt(n*p*q)).cdf(k + 0.5), 4)
0.8402

>>> # Solution using the cumulative binomial distribution
>>> from math import comb, fsum
>>> round(fsum(comb(n, r) * p**r * q**(n-r) for r in range(k+1)), 4)
0.8402

>>> # Approximation using a simulation
>>> from random import seed, choices
>>> seed(8675309)
>>> def trial():
...     return choices(('Python', 'Ruby'), (p, q), k=n).count('Python')
>>> mean(trial() <= k for i in range(10_000))
0.8398
@end example

Normal distributions commonly arise in machine learning problems.

Wikipedia has a nice example of a Naive Bayesian Classifier@footnote{https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Sex_classification}.
The challenge is to predict a person’s gender from measurements of normally
distributed features including height, weight, and foot size.

We’re given a training dataset with measurements for eight people.  The
measurements are assumed to be normally distributed, so we summarize the data
with @ref{549,,NormalDist}:

@example
>>> height_male = NormalDist.from_samples([6, 5.92, 5.58, 5.92])
>>> height_female = NormalDist.from_samples([5, 5.5, 5.42, 5.75])
>>> weight_male = NormalDist.from_samples([180, 190, 170, 165])
>>> weight_female = NormalDist.from_samples([100, 150, 130, 150])
>>> foot_size_male = NormalDist.from_samples([12, 11, 12, 10])
>>> foot_size_female = NormalDist.from_samples([6, 8, 7, 9])
@end example

Next, we encounter a new person whose feature measurements are known but whose
gender is unknown:

@example
>>> ht = 6.0        # height
>>> wt = 130        # weight
>>> fs = 8          # foot size
@end example

Starting with a 50% prior probability@footnote{https://en.wikipedia.org/wiki/Prior_probability} of being male or female,
we compute the posterior as the prior times the product of likelihoods for the
feature measurements given the gender:

@example
>>> prior_male = 0.5
>>> prior_female = 0.5
>>> posterior_male = (prior_male * height_male.pdf(ht) *
...                   weight_male.pdf(wt) * foot_size_male.pdf(fs))

>>> posterior_female = (prior_female * height_female.pdf(ht) *
...                     weight_female.pdf(wt) * foot_size_female.pdf(fs))
@end example

The final prediction goes to the largest posterior. This is known as the
maximum a posteriori@footnote{https://en.wikipedia.org/wiki/Maximum_a_posteriori_estimation} or MAP:

@example
>>> 'male' if posterior_male > posterior_female else 'female'
'female'
@end example

@c # This modelines must appear within the last ten lines of the file.
@c kate: indent-width 3; remove-trailing-space on; replace-tabs on; encoding utf-8;

@node Functional Programming Modules,File and Directory Access,Numeric and Mathematical Modules,The Python Standard Library
@anchor{library/functional doc}@anchor{1fda}@anchor{library/functional functional-programming-modules}@anchor{1fdb}
@section Functional Programming Modules


The modules described in this chapter provide functions and classes that support
a functional programming style, and general operations on callables.

The following modules are documented in this chapter:

@menu
* itertools — Functions creating iterators for efficient looping:: 
* functools — Higher-order functions and operations on callable objects:: 
* operator — Standard operators as functions:: 

@end menu

@node itertools — Functions creating iterators for efficient looping,functools — Higher-order functions and operations on callable objects,,Functional Programming Modules
@anchor{library/itertools doc}@anchor{1fdc}@anchor{library/itertools itertools-functions-creating-iterators-for-efficient-looping}@anchor{1fdd}@anchor{library/itertools module-itertools}@anchor{9f}
@subsection @code{itertools} — Functions creating iterators for efficient looping


@geindex module; itertools

@c from itertools import *


__________________________________________________________________


This module implements a number of @ref{11c3,,iterator} building blocks inspired
by constructs from APL, Haskell, and SML.  Each has been recast in a form
suitable for Python.

The module standardizes a core set of fast, memory efficient tools that are
useful by themselves or in combination.  Together, they form an “iterator
algebra” making it possible to construct specialized tools succinctly and
efficiently in pure Python.

For instance, SML provides a tabulation tool: @code{tabulate(f)} which produces a
sequence @code{f(0), f(1), ...}.  The same effect can be achieved in Python
by combining @ref{2f4,,map()} and @ref{e33,,count()} to form @code{map(f, count())}.

These tools and their built-in counterparts also work well with the high-speed
functions in the @ref{be,,operator} module.  For example, the multiplication
operator can be mapped across two vectors to form an efficient dot-product:
@code{sum(map(operator.mul, vector1, vector2))}.

@strong{Infinite iterators:}


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Iterator

@tab

Arguments

@tab

Results

@tab

Example

@item

@ref{e33,,count()}

@tab

start, [step]

@tab

start, start+step, start+2*step, …

@tab

@code{count(10) --> 10 11 12 13 14 ...}

@item

@ref{1fde,,cycle()}

@tab

p

@tab

p0, p1, … plast, p0, p1, …

@tab

@code{cycle('ABCD') --> A B C D A B C D ...}

@item

@ref{1373,,repeat()}

@tab

elem [,n]

@tab

elem, elem, elem, … endlessly or up to n times

@tab

@code{repeat(10, 3) --> 10 10 10}

@end multitable


@strong{Iterators terminating on the shortest input sequence:}


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Iterator

@tab

Arguments

@tab

Results

@tab

Example

@item

@ref{505,,accumulate()}

@tab

p [,func]

@tab

p0, p0+p1, p0+p1+p2, …

@tab

@code{accumulate([1,2,3,4,5]) --> 1 3 6 10 15}

@item

@ref{1a5f,,chain()}

@tab

p, q, …

@tab

p0, p1, … plast, q0, q1, …

@tab

@code{chain('ABC', 'DEF') --> A B C D E F}

@item

@ref{1fdf,,chain.from_iterable()}

@tab

iterable

@tab

p0, p1, … plast, q0, q1, …

@tab

@code{chain.from_iterable(['ABC', 'DEF']) --> A B C D E F}

@item

@ref{e32,,compress()}

@tab

data, selectors

@tab

(d[0] if s[0]), (d[1] if s[1]), …

@tab

@code{compress('ABCDEF', [1,0,1,0,1,1]) --> A C E F}

@item

@ref{1fe0,,dropwhile()}

@tab

pred, seq

@tab

seq[n], seq[n+1], starting when pred fails

@tab

@code{dropwhile(lambda x: x<5, [1,4,6,4,1]) --> 6 4 1}

@item

@ref{1a44,,filterfalse()}

@tab

pred, seq

@tab

elements of seq where pred(elem) is false

@tab

@code{filterfalse(lambda x: x%2, range(10)) --> 0 2 4 6 8}

@item

@ref{1fe1,,groupby()}

@tab

iterable[, key]

@tab

sub-iterators grouped by value of key(v)

@tab

@item

@ref{672,,islice()}

@tab

seq, [start,] stop [, step]

@tab

elements from seq[start:stop:step]

@tab

@code{islice('ABCDEFG', 2, None) --> C D E F G}

@item

@ref{1fe2,,pairwise()}

@tab

iterable

@tab

(p[0], p[1]), (p[1], p[2])

@tab

@code{pairwise('ABCDEFG') --> AB BC CD DE EF FG}

@item

@ref{c76,,starmap()}

@tab

func, seq

@tab

func(*seq[0]), func(*seq[1]), …

@tab

@code{starmap(pow, [(2,5), (3,2), (10,3)]) --> 32 9 1000}

@item

@ref{1fe3,,takewhile()}

@tab

pred, seq

@tab

seq[0], seq[1], until pred fails

@tab

@code{takewhile(lambda x: x<5, [1,4,6,4,1]) --> 1 4}

@item

@ref{1fe4,,tee()}

@tab

it, n

@tab

it1, it2, … itn  splits one iterator into n

@tab

@item

@ref{e43,,zip_longest()}

@tab

p, q, …

@tab

(p[0], q[0]), (p[1], q[1]), …

@tab

@code{zip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D-}

@end multitable


@strong{Combinatoric iterators:}


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Iterator

@tab

Arguments

@tab

Results

@item

@ref{ea9,,product()}

@tab

p, q, … [repeat=1]

@tab

cartesian product, equivalent to a nested for-loop

@item

@ref{1193,,permutations()}

@tab

p[, r]

@tab

r-length tuples, all possible orderings, no repeated elements

@item

@ref{ea8,,combinations()}

@tab

p, r

@tab

r-length tuples, in sorted order, no repeated elements

@item

@ref{e31,,combinations_with_replacement()}

@tab

p, r

@tab

r-length tuples, in sorted order, with repeated elements

@end multitable



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Examples

@tab

Results

@item

@code{product('ABCD', repeat=2)}

@tab

@code{AA AB AC AD BA BB BC BD CA CB CC CD DA DB DC DD}

@item

@code{permutations('ABCD', 2)}

@tab

@code{AB AC AD BA BC BD CA CB CD DA DB DC}

@item

@code{combinations('ABCD', 2)}

@tab

@code{AB AC AD BC BD CD}

@item

@code{combinations_with_replacement('ABCD', 2)}

@tab

@code{AA AB AC AD BB BC BD CC CD DD}

@end multitable


@menu
* Itertool functions:: 
* Itertools Recipes:: 

@end menu

@node Itertool functions,Itertools Recipes,,itertools — Functions creating iterators for efficient looping
@anchor{library/itertools itertool-functions}@anchor{1fe5}@anchor{library/itertools itertools-functions}@anchor{1fe6}
@subsubsection Itertool functions


The following module functions all construct and return iterators. Some provide
streams of infinite length, so they should only be accessed by functions or
loops that truncate the stream.

@geindex accumulate() (in module itertools)
@anchor{library/itertools itertools accumulate}@anchor{505}
@deffn {Function} itertools.accumulate (iterable[, func, *, initial=None])

Make an iterator that returns accumulated sums, or accumulated
results of other binary functions (specified via the optional
@emph{func} argument).

If @emph{func} is supplied, it should be a function
of two arguments. Elements of the input @emph{iterable} may be any type
that can be accepted as arguments to @emph{func}. (For example, with
the default operation of addition, elements may be any addable
type including @ref{248,,Decimal} or
@ref{172,,Fraction}.)

Usually, the number of elements output matches the input iterable.
However, if the keyword argument @emph{initial} is provided, the
accumulation leads off with the @emph{initial} value so that the output
has one more element than the input iterable.

Roughly equivalent to:

@example
def accumulate(iterable, func=operator.add, *, initial=None):
    'Return running totals'
    # accumulate([1,2,3,4,5]) --> 1 3 6 10 15
    # accumulate([1,2,3,4,5], initial=100) --> 100 101 103 106 110 115
    # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120
    it = iter(iterable)
    total = initial
    if initial is None:
        try:
            total = next(it)
        except StopIteration:
            return
    yield total
    for element in it:
        total = func(total, element)
        yield total
@end example

There are a number of uses for the @emph{func} argument.  It can be set to
@ref{a7d,,min()} for a running minimum, @ref{a7e,,max()} for a running maximum, or
@ref{1fe7,,operator.mul()} for a running product.  Amortization tables can be
built by accumulating interest and applying payments.  First-order
recurrence relations@footnote{https://en.wikipedia.org/wiki/Recurrence_relation}
can be modeled by supplying the initial value in the iterable and using only
the accumulated total in @emph{func} argument:

@example
>>> data = [3, 4, 6, 2, 1, 9, 0, 7, 5, 8]
>>> list(accumulate(data, operator.mul))     # running product
[3, 12, 72, 144, 144, 1296, 0, 0, 0, 0]
>>> list(accumulate(data, max))              # running maximum
[3, 4, 6, 6, 6, 9, 9, 9, 9, 9]

# Amortize a 5% loan of 1000 with 4 annual payments of 90
>>> cashflows = [1000, -90, -90, -90, -90]
>>> list(accumulate(cashflows, lambda bal, pmt: bal*1.05 + pmt))
[1000, 960.0, 918.0, 873.9000000000001, 827.5950000000001]

# Chaotic recurrence relation https://en.wikipedia.org/wiki/Logistic_map
>>> logistic_map = lambda x, _:  r * x * (1 - x)
>>> r = 3.8
>>> x0 = 0.4
>>> inputs = repeat(x0, 36)     # only the initial value is used
>>> [format(x, '.2f') for x in accumulate(inputs, logistic_map)]
['0.40', '0.91', '0.30', '0.81', '0.60', '0.92', '0.29', '0.79', '0.63',
 '0.88', '0.39', '0.90', '0.33', '0.84', '0.52', '0.95', '0.18', '0.57',
 '0.93', '0.25', '0.71', '0.79', '0.63', '0.88', '0.39', '0.91', '0.32',
 '0.83', '0.54', '0.95', '0.20', '0.60', '0.91', '0.30', '0.80', '0.60']
@end example

See @ref{e77,,functools.reduce()} for a similar function that returns only the
final accumulated value.


New in version 3.2.



Changed in version 3.3: Added the optional @emph{func} parameter.



Changed in version 3.8: Added the optional @emph{initial} parameter.

@end deffn

@geindex chain() (in module itertools)
@anchor{library/itertools itertools chain}@anchor{1a5f}
@deffn {Function} itertools.chain (*iterables)

Make an iterator that returns elements from the first iterable until it is
exhausted, then proceeds to the next iterable, until all of the iterables are
exhausted.  Used for treating consecutive sequences as a single sequence.
Roughly equivalent to:

@example
def chain(*iterables):
    # chain('ABC', 'DEF') --> A B C D E F
    for it in iterables:
        for element in it:
            yield element
@end example
@end deffn

@geindex from_iterable() (itertools.chain class method)
@anchor{library/itertools itertools chain from_iterable}@anchor{1fdf}
@deffn {Method} classmethod chain.from_iterable (iterable)

Alternate constructor for @ref{1a5f,,chain()}.  Gets chained inputs from a
single iterable argument that is evaluated lazily.  Roughly equivalent to:

@example
def from_iterable(iterables):
    # chain.from_iterable(['ABC', 'DEF']) --> A B C D E F
    for it in iterables:
        for element in it:
            yield element
@end example
@end deffn

@geindex combinations() (in module itertools)
@anchor{library/itertools itertools combinations}@anchor{ea8}
@deffn {Function} itertools.combinations (iterable, r)

Return @emph{r} length subsequences of elements from the input @emph{iterable}.

The combination tuples are emitted in lexicographic ordering according to
the order of the input @emph{iterable}. So, if the input @emph{iterable} is sorted,
the combination tuples will be produced in sorted order.

Elements are treated as unique based on their position, not on their
value.  So if the input elements are unique, there will be no repeat
values in each combination.

Roughly equivalent to:

@example
def combinations(iterable, r):
    # combinations('ABCD', 2) --> AB AC AD BC BD CD
    # combinations(range(4), 3) --> 012 013 023 123
    pool = tuple(iterable)
    n = len(pool)
    if r > n:
        return
    indices = list(range(r))
    yield tuple(pool[i] for i in indices)
    while True:
        for i in reversed(range(r)):
            if indices[i] != i + n - r:
                break
        else:
            return
        indices[i] += 1
        for j in range(i+1, r):
            indices[j] = indices[j-1] + 1
        yield tuple(pool[i] for i in indices)
@end example

The code for @ref{ea8,,combinations()} can be also expressed as a subsequence
of @ref{1193,,permutations()} after filtering entries where the elements are not
in sorted order (according to their position in the input pool):

@example
def combinations(iterable, r):
    pool = tuple(iterable)
    n = len(pool)
    for indices in permutations(range(n), r):
        if sorted(indices) == list(indices):
            yield tuple(pool[i] for i in indices)
@end example

The number of items returned is @code{n! / r! / (n-r)!} when @code{0 <= r <= n}
or zero when @code{r > n}.
@end deffn

@geindex combinations_with_replacement() (in module itertools)
@anchor{library/itertools itertools combinations_with_replacement}@anchor{e31}
@deffn {Function} itertools.combinations_with_replacement (iterable, r)

Return @emph{r} length subsequences of elements from the input @emph{iterable}
allowing individual elements to be repeated more than once.

The combination tuples are emitted in lexicographic ordering according to
the order of the input @emph{iterable}. So, if the input @emph{iterable} is sorted,
the combination tuples will be produced in sorted order.

Elements are treated as unique based on their position, not on their
value.  So if the input elements are unique, the generated combinations
will also be unique.

Roughly equivalent to:

@example
def combinations_with_replacement(iterable, r):
    # combinations_with_replacement('ABC', 2) --> AA AB AC BB BC CC
    pool = tuple(iterable)
    n = len(pool)
    if not n and r:
        return
    indices = [0] * r
    yield tuple(pool[i] for i in indices)
    while True:
        for i in reversed(range(r)):
            if indices[i] != n - 1:
                break
        else:
            return
        indices[i:] = [indices[i] + 1] * (r - i)
        yield tuple(pool[i] for i in indices)
@end example

The code for @ref{e31,,combinations_with_replacement()} can be also expressed as
a subsequence of @ref{ea9,,product()} after filtering entries where the elements
are not in sorted order (according to their position in the input pool):

@example
def combinations_with_replacement(iterable, r):
    pool = tuple(iterable)
    n = len(pool)
    for indices in product(range(n), repeat=r):
        if sorted(indices) == list(indices):
            yield tuple(pool[i] for i in indices)
@end example

The number of items returned is @code{(n+r-1)! / r! / (n-1)!} when @code{n > 0}.


New in version 3.1.

@end deffn

@geindex compress() (in module itertools)
@anchor{library/itertools itertools compress}@anchor{e32}
@deffn {Function} itertools.compress (data, selectors)

Make an iterator that filters elements from @emph{data} returning only those that
have a corresponding element in @emph{selectors} that evaluates to @code{True}.
Stops when either the @emph{data} or @emph{selectors} iterables has been exhausted.
Roughly equivalent to:

@example
def compress(data, selectors):
    # compress('ABCDEF', [1,0,1,0,1,1]) --> A C E F
    return (d for d, s in zip(data, selectors) if s)
@end example


New in version 3.1.

@end deffn

@geindex count() (in module itertools)
@anchor{library/itertools itertools count}@anchor{e33}
@deffn {Function} itertools.count (start=0, step=1)

Make an iterator that returns evenly spaced values starting with number @emph{start}. Often
used as an argument to @ref{2f4,,map()} to generate consecutive data points.
Also, used with @ref{247,,zip()} to add sequence numbers.  Roughly equivalent to:

@example
def count(start=0, step=1):
    # count(10) --> 10 11 12 13 14 ...
    # count(2.5, 0.5) -> 2.5 3.0 3.5 ...
    n = start
    while True:
        yield n
        n += step
@end example

When counting with floating point numbers, better accuracy can sometimes be
achieved by substituting multiplicative code such as: @code{(start + step * i
for i in count())}.


Changed in version 3.1: Added @emph{step} argument and allowed non-integer arguments.

@end deffn

@geindex cycle() (in module itertools)
@anchor{library/itertools itertools cycle}@anchor{1fde}
@deffn {Function} itertools.cycle (iterable)

Make an iterator returning elements from the iterable and saving a copy of each.
When the iterable is exhausted, return elements from the saved copy.  Repeats
indefinitely.  Roughly equivalent to:

@example
def cycle(iterable):
    # cycle('ABCD') --> A B C D A B C D A B C D ...
    saved = []
    for element in iterable:
        yield element
        saved.append(element)
    while saved:
        for element in saved:
              yield element
@end example

Note, this member of the toolkit may require significant auxiliary storage
(depending on the length of the iterable).
@end deffn

@geindex dropwhile() (in module itertools)
@anchor{library/itertools itertools dropwhile}@anchor{1fe0}
@deffn {Function} itertools.dropwhile (predicate, iterable)

Make an iterator that drops elements from the iterable as long as the predicate
is true; afterwards, returns every element.  Note, the iterator does not produce
@emph{any} output until the predicate first becomes false, so it may have a lengthy
start-up time.  Roughly equivalent to:

@example
def dropwhile(predicate, iterable):
    # dropwhile(lambda x: x<5, [1,4,6,4,1]) --> 6 4 1
    iterable = iter(iterable)
    for x in iterable:
        if not predicate(x):
            yield x
            break
    for x in iterable:
        yield x
@end example
@end deffn

@geindex filterfalse() (in module itertools)
@anchor{library/itertools itertools filterfalse}@anchor{1a44}
@deffn {Function} itertools.filterfalse (predicate, iterable)

Make an iterator that filters elements from iterable returning only those for
which the predicate is @code{False}. If @emph{predicate} is @code{None}, return the items
that are false. Roughly equivalent to:

@example
def filterfalse(predicate, iterable):
    # filterfalse(lambda x: x%2, range(10)) --> 0 2 4 6 8
    if predicate is None:
        predicate = bool
    for x in iterable:
        if not predicate(x):
            yield x
@end example
@end deffn

@geindex groupby() (in module itertools)
@anchor{library/itertools itertools groupby}@anchor{1fe1}
@deffn {Function} itertools.groupby (iterable, key=None)

Make an iterator that returns consecutive keys and groups from the @emph{iterable}.
The @emph{key} is a function computing a key value for each element.  If not
specified or is @code{None}, @emph{key} defaults to an identity function and returns
the element unchanged.  Generally, the iterable needs to already be sorted on
the same key function.

The operation of @ref{1fe1,,groupby()} is similar to the @code{uniq} filter in Unix.  It
generates a break or new group every time the value of the key function changes
(which is why it is usually necessary to have sorted the data using the same key
function).  That behavior differs from SQL’s GROUP BY which aggregates common
elements regardless of their input order.

The returned group is itself an iterator that shares the underlying iterable
with @ref{1fe1,,groupby()}.  Because the source is shared, when the @ref{1fe1,,groupby()}
object is advanced, the previous group is no longer visible.  So, if that data
is needed later, it should be stored as a list:

@example
groups = []
uniquekeys = []
data = sorted(data, key=keyfunc)
for k, g in groupby(data, keyfunc):
    groups.append(list(g))      # Store group iterator as a list
    uniquekeys.append(k)
@end example

@ref{1fe1,,groupby()} is roughly equivalent to:

@example
class groupby:
    # [k for k, g in groupby('AAAABBBCCDAABBB')] --> A B C D A B
    # [list(g) for k, g in groupby('AAAABBBCCD')] --> AAAA BBB CC D
    def __init__(self, iterable, key=None):
        if key is None:
            key = lambda x: x
        self.keyfunc = key
        self.it = iter(iterable)
        self.tgtkey = self.currkey = self.currvalue = object()
    def __iter__(self):
        return self
    def __next__(self):
        self.id = object()
        while self.currkey == self.tgtkey:
            self.currvalue = next(self.it)    # Exit on StopIteration
            self.currkey = self.keyfunc(self.currvalue)
        self.tgtkey = self.currkey
        return (self.currkey, self._grouper(self.tgtkey, self.id))
    def _grouper(self, tgtkey, id):
        while self.id is id and self.currkey == tgtkey:
            yield self.currvalue
            try:
                self.currvalue = next(self.it)
            except StopIteration:
                return
            self.currkey = self.keyfunc(self.currvalue)
@end example
@end deffn

@geindex islice() (in module itertools)
@anchor{library/itertools itertools islice}@anchor{672}
@deffn {Function} itertools.islice (iterable, stop)

@deffnx {Function} itertools.islice (iterable, start, stop[, step])

Make an iterator that returns selected elements from the iterable. If @emph{start} is
non-zero, then elements from the iterable are skipped until start is reached.
Afterward, elements are returned consecutively unless @emph{step} is set higher than
one which results in items being skipped.  If @emph{stop} is @code{None}, then iteration
continues until the iterator is exhausted, if at all; otherwise, it stops at the
specified position.  Unlike regular slicing, @ref{672,,islice()} does not support
negative values for @emph{start}, @emph{stop}, or @emph{step}.  Can be used to extract related
fields from data where the internal structure has been flattened (for example, a
multi-line report may list a name field on every third line).  Roughly equivalent to:

@example
def islice(iterable, *args):
    # islice('ABCDEFG', 2) --> A B
    # islice('ABCDEFG', 2, 4) --> C D
    # islice('ABCDEFG', 2, None) --> C D E F G
    # islice('ABCDEFG', 0, None, 2) --> A C E G
    s = slice(*args)
    start, stop, step = s.start or 0, s.stop or sys.maxsize, s.step or 1
    it = iter(range(start, stop, step))
    try:
        nexti = next(it)
    except StopIteration:
        # Consume *iterable* up to the *start* position.
        for i, element in zip(range(start), iterable):
            pass
        return
    try:
        for i, element in enumerate(iterable):
            if i == nexti:
                yield element
                nexti = next(it)
    except StopIteration:
        # Consume to *stop*.
        for i, element in zip(range(i + 1, stop), iterable):
            pass
@end example

If @emph{start} is @code{None}, then iteration starts at zero. If @emph{step} is @code{None},
then the step defaults to one.
@end deffn

@geindex pairwise() (in module itertools)
@anchor{library/itertools itertools pairwise}@anchor{1fe2}
@deffn {Function} itertools.pairwise (iterable)

Return successive overlapping pairs taken from the input @emph{iterable}.

The number of 2-tuples in the output iterator will be one fewer than the
number of inputs.  It will be empty if the input iterable has fewer than
two values.

Roughly equivalent to:

@example
def pairwise(iterable):
    # pairwise('ABCDEFG') --> AB BC CD DE EF FG
    a, b = tee(iterable)
    next(b, None)
    return zip(a, b)
@end example


New in version 3.10.

@end deffn

@geindex permutations() (in module itertools)
@anchor{library/itertools itertools permutations}@anchor{1193}
@deffn {Function} itertools.permutations (iterable, r=None)

Return successive @emph{r} length permutations of elements in the @emph{iterable}.

If @emph{r} is not specified or is @code{None}, then @emph{r} defaults to the length
of the @emph{iterable} and all possible full-length permutations
are generated.

The permutation tuples are emitted in lexicographic ordering according to
the order of the input @emph{iterable}. So, if the input @emph{iterable} is sorted,
the combination tuples will be produced in sorted order.

Elements are treated as unique based on their position, not on their
value.  So if the input elements are unique, there will be no repeat
values in each permutation.

Roughly equivalent to:

@example
def permutations(iterable, r=None):
    # permutations('ABCD', 2) --> AB AC AD BA BC BD CA CB CD DA DB DC
    # permutations(range(3)) --> 012 021 102 120 201 210
    pool = tuple(iterable)
    n = len(pool)
    r = n if r is None else r
    if r > n:
        return
    indices = list(range(n))
    cycles = list(range(n, n-r, -1))
    yield tuple(pool[i] for i in indices[:r])
    while n:
        for i in reversed(range(r)):
            cycles[i] -= 1
            if cycles[i] == 0:
                indices[i:] = indices[i+1:] + indices[i:i+1]
                cycles[i] = n - i
            else:
                j = cycles[i]
                indices[i], indices[-j] = indices[-j], indices[i]
                yield tuple(pool[i] for i in indices[:r])
                break
        else:
            return
@end example

The code for @ref{1193,,permutations()} can be also expressed as a subsequence of
@ref{ea9,,product()}, filtered to exclude entries with repeated elements (those
from the same position in the input pool):

@example
def permutations(iterable, r=None):
    pool = tuple(iterable)
    n = len(pool)
    r = n if r is None else r
    for indices in product(range(n), repeat=r):
        if len(set(indices)) == r:
            yield tuple(pool[i] for i in indices)
@end example

The number of items returned is @code{n! / (n-r)!} when @code{0 <= r <= n}
or zero when @code{r > n}.
@end deffn

@geindex product() (in module itertools)
@anchor{library/itertools itertools product}@anchor{ea9}
@deffn {Function} itertools.product (*iterables, repeat=1)

Cartesian product of input iterables.

Roughly equivalent to nested for-loops in a generator expression. For example,
@code{product(A, B)} returns the same as @code{((x,y) for x in A for y in B)}.

The nested loops cycle like an odometer with the rightmost element advancing
on every iteration.  This pattern creates a lexicographic ordering so that if
the input’s iterables are sorted, the product tuples are emitted in sorted
order.

To compute the product of an iterable with itself, specify the number of
repetitions with the optional @emph{repeat} keyword argument.  For example,
@code{product(A, repeat=4)} means the same as @code{product(A, A, A, A)}.

This function is roughly equivalent to the following code, except that the
actual implementation does not build up intermediate results in memory:

@example
def product(*args, repeat=1):
    # product('ABCD', 'xy') --> Ax Ay Bx By Cx Cy Dx Dy
    # product(range(2), repeat=3) --> 000 001 010 011 100 101 110 111
    pools = [tuple(pool) for pool in args] * repeat
    result = [[]]
    for pool in pools:
        result = [x+[y] for x in result for y in pool]
    for prod in result:
        yield tuple(prod)
@end example

Before @ref{ea9,,product()} runs, it completely consumes the input iterables,
keeping pools of values in memory to generate the products.  Accordingly,
it is only useful with finite inputs.
@end deffn

@geindex repeat() (in module itertools)
@anchor{library/itertools itertools repeat}@anchor{1373}
@deffn {Function} itertools.repeat (object[, times])

Make an iterator that returns @emph{object} over and over again. Runs indefinitely
unless the @emph{times} argument is specified. Used as argument to @ref{2f4,,map()} for
invariant parameters to the called function.  Also used with @ref{247,,zip()} to
create an invariant part of a tuple record.

Roughly equivalent to:

@example
def repeat(object, times=None):
    # repeat(10, 3) --> 10 10 10
    if times is None:
        while True:
            yield object
    else:
        for i in range(times):
            yield object
@end example

A common use for @emph{repeat} is to supply a stream of constant values to @emph{map}
or @emph{zip}:

@example
>>> list(map(pow, range(10), repeat(2)))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
@end example
@end deffn

@geindex starmap() (in module itertools)
@anchor{library/itertools itertools starmap}@anchor{c76}
@deffn {Function} itertools.starmap (function, iterable)

Make an iterator that computes the function using arguments obtained from
the iterable.  Used instead of @ref{2f4,,map()} when argument parameters are already
grouped in tuples from a single iterable (the data has been “pre-zipped”).  The
difference between @ref{2f4,,map()} and @ref{c76,,starmap()} parallels the distinction
between @code{function(a,b)} and @code{function(*c)}. Roughly equivalent to:

@example
def starmap(function, iterable):
    # starmap(pow, [(2,5), (3,2), (10,3)]) --> 32 9 1000
    for args in iterable:
        yield function(*args)
@end example
@end deffn

@geindex takewhile() (in module itertools)
@anchor{library/itertools itertools takewhile}@anchor{1fe3}
@deffn {Function} itertools.takewhile (predicate, iterable)

Make an iterator that returns elements from the iterable as long as the
predicate is true.  Roughly equivalent to:

@example
def takewhile(predicate, iterable):
    # takewhile(lambda x: x<5, [1,4,6,4,1]) --> 1 4
    for x in iterable:
        if predicate(x):
            yield x
        else:
            break
@end example
@end deffn

@geindex tee() (in module itertools)
@anchor{library/itertools itertools tee}@anchor{1fe4}
@deffn {Function} itertools.tee (iterable, n=2)

Return @emph{n} independent iterators from a single iterable.

The following Python code helps explain what @emph{tee} does (although the actual
implementation is more complex and uses only a single underlying
@abbr{FIFO} queue).

Roughly equivalent to:

@example
def tee(iterable, n=2):
    it = iter(iterable)
    deques = [collections.deque() for i in range(n)]
    def gen(mydeque):
        while True:
            if not mydeque:             # when the local deque is empty
                try:
                    newval = next(it)   # fetch a new value and
                except StopIteration:
                    return
                for d in deques:        # load it to all the deques
                    d.append(newval)
            yield mydeque.popleft()
    return tuple(gen(d) for d in deques)
@end example

Once @ref{1fe4,,tee()} has made a split, the original @emph{iterable} should not be
used anywhere else; otherwise, the @emph{iterable} could get advanced without
the tee objects being informed.

@code{tee} iterators are not threadsafe. A @ref{5af,,RuntimeError} may be
raised when using simultaneously iterators returned by the same @ref{1fe4,,tee()}
call, even if the original @emph{iterable} is threadsafe.

This itertool may require significant auxiliary storage (depending on how
much temporary data needs to be stored). In general, if one iterator uses
most or all of the data before another iterator starts, it is faster to use
@ref{43a,,list()} instead of @ref{1fe4,,tee()}.
@end deffn

@geindex zip_longest() (in module itertools)
@anchor{library/itertools itertools zip_longest}@anchor{e43}
@deffn {Function} itertools.zip_longest (*iterables, fillvalue=None)

Make an iterator that aggregates elements from each of the iterables. If the
iterables are of uneven length, missing values are filled-in with @emph{fillvalue}.
Iteration continues until the longest iterable is exhausted.  Roughly equivalent to:

@example
def zip_longest(*args, fillvalue=None):
    # zip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D-
    iterators = [iter(it) for it in args]
    num_active = len(iterators)
    if not num_active:
        return
    while True:
        values = []
        for i, it in enumerate(iterators):
            try:
                value = next(it)
            except StopIteration:
                num_active -= 1
                if not num_active:
                    return
                iterators[i] = repeat(fillvalue)
                value = fillvalue
            values.append(value)
        yield tuple(values)
@end example

If one of the iterables is potentially infinite, then the @ref{e43,,zip_longest()}
function should be wrapped with something that limits the number of calls
(for example @ref{672,,islice()} or @ref{1fe3,,takewhile()}).  If not specified,
@emph{fillvalue} defaults to @code{None}.
@end deffn

@node Itertools Recipes,,Itertool functions,itertools — Functions creating iterators for efficient looping
@anchor{library/itertools id1}@anchor{1fe8}@anchor{library/itertools itertools-recipes}@anchor{e0f}
@subsubsection Itertools Recipes


This section shows recipes for creating an extended toolset using the existing
itertools as building blocks.

Substantially all of these recipes and many, many others can be installed from
the more-itertools project@footnote{https://pypi.org/project/more-itertools/} found
on the Python Package Index:

@example
pip install more-itertools
@end example

The extended tools offer the same high performance as the underlying toolset.
The superior memory performance is kept by processing elements one at a time
rather than bringing the whole iterable into memory all at once. Code volume is
kept small by linking the tools together in a functional style which helps
eliminate temporary variables.  High speed is retained by preferring
“vectorized” building blocks over the use of for-loops and @ref{bf7,,generator}s
which incur interpreter overhead.

@example
def take(n, iterable):
    "Return first n items of the iterable as a list"
    return list(islice(iterable, n))

def prepend(value, iterator):
    "Prepend a single value in front of an iterator"
    # prepend(1, [2, 3, 4]) -> 1 2 3 4
    return chain([value], iterator)

def tabulate(function, start=0):
    "Return function(0), function(1), ..."
    return map(function, count(start))

def tail(n, iterable):
    "Return an iterator over the last n items"
    # tail(3, 'ABCDEFG') --> E F G
    return iter(collections.deque(iterable, maxlen=n))

def consume(iterator, n=None):
    "Advance the iterator n-steps ahead. If n is None, consume entirely."
    # Use functions that consume iterators at C speed.
    if n is None:
        # feed the entire iterator into a zero-length deque
        collections.deque(iterator, maxlen=0)
    else:
        # advance to the empty slice starting at position n
        next(islice(iterator, n, n), None)

def nth(iterable, n, default=None):
    "Returns the nth item or a default value"
    return next(islice(iterable, n, None), default)

def all_equal(iterable):
    "Returns True if all the elements are equal to each other"
    g = groupby(iterable)
    return next(g, True) and not next(g, False)

def quantify(iterable, pred=bool):
    "Count how many times the predicate is true"
    return sum(map(pred, iterable))

def pad_none(iterable):
    """Returns the sequence elements and then returns None indefinitely.

    Useful for emulating the behavior of the built-in map() function.
    """
    return chain(iterable, repeat(None))

def ncycles(iterable, n):
    "Returns the sequence elements n times"
    return chain.from_iterable(repeat(tuple(iterable), n))

def dotproduct(vec1, vec2):
    return sum(map(operator.mul, vec1, vec2))

def convolve(signal, kernel):
    # See:  https://betterexplained.com/articles/intuitive-convolution/
    # convolve(data, [0.25, 0.25, 0.25, 0.25]) --> Moving average (blur)
    # convolve(data, [1, -1]) --> 1st finite difference (1st derivative)
    # convolve(data, [1, -2, 1]) --> 2nd finite difference (2nd derivative)
    kernel = tuple(kernel)[::-1]
    n = len(kernel)
    window = collections.deque([0], maxlen=n) * n
    for x in chain(signal, repeat(0, n-1)):
        window.append(x)
        yield sum(map(operator.mul, kernel, window))

def flatten(list_of_lists):
    "Flatten one level of nesting"
    return chain.from_iterable(list_of_lists)

def repeatfunc(func, times=None, *args):
    """Repeat calls to func with specified arguments.

    Example:  repeatfunc(random.random)
    """
    if times is None:
        return starmap(func, repeat(args))
    return starmap(func, repeat(args, times))

def grouper(iterable, n, *, incomplete='fill', fillvalue=None):
    "Collect data into non-overlapping fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, fillvalue='x') --> ABC DEF Gxx
    # grouper('ABCDEFG', 3, incomplete='strict') --> ABC DEF ValueError
    # grouper('ABCDEFG', 3, incomplete='ignore') --> ABC DEF
    args = [iter(iterable)] * n
    if incomplete == 'fill':
        return zip_longest(*args, fillvalue=fillvalue)
    if incomplete == 'strict':
        return zip(*args, strict=True)
    if incomplete == 'ignore':
        return zip(*args)
    else:
        raise ValueError('Expected fill, strict, or ignore')

def triplewise(iterable):
    "Return overlapping triplets from an iterable"
    # triplewise('ABCDEFG') -> ABC BCD CDE DEF EFG
    for (a, _), (b, c) in pairwise(pairwise(iterable)):
        yield a, b, c

def sliding_window(iterable, n):
    # sliding_window('ABCDEFG', 4) -> ABCD BCDE CDEF DEFG
    it = iter(iterable)
    window = collections.deque(islice(it, n), maxlen=n)
    if len(window) == n:
        yield tuple(window)
    for x in it:
        window.append(x)
        yield tuple(window)

def roundrobin(*iterables):
    "roundrobin('ABC', 'D', 'EF') --> A D E B F C"
    # Recipe credited to George Sakkis
    num_active = len(iterables)
    nexts = cycle(iter(it).__next__ for it in iterables)
    while num_active:
        try:
            for next in nexts:
                yield next()
        except StopIteration:
            # Remove the iterator we just exhausted from the cycle.
            num_active -= 1
            nexts = cycle(islice(nexts, num_active))

def partition(pred, iterable):
    "Use a predicate to partition entries into false entries and true entries"
    # partition(is_odd, range(10)) --> 0 2 4 6 8   and  1 3 5 7 9
    t1, t2 = tee(iterable)
    return filterfalse(pred, t1), filter(pred, t2)

def before_and_after(predicate, it):
    """ Variant of takewhile() that allows complete
        access to the remainder of the iterator.

        >>> it = iter('ABCdEfGhI')
        >>> all_upper, remainder = before_and_after(str.isupper, it)
        >>> '@w{'}.join(all_upper)
        'ABC'
        >>> '@w{'}.join(remainder)     # takewhile() would lose the 'd'
        'dEfGhI'

        Note that the first iterator must be fully
        consumed before the second iterator can
        generate valid results.
    """
    it = iter(it)
    transition = []
    def true_iterator():
        for elem in it:
            if predicate(elem):
                yield elem
            else:
                transition.append(elem)
                return
    def remainder_iterator():
        yield from transition
        yield from it
    return true_iterator(), remainder_iterator()

def subslices(seq):
    "Return all contiguous non-empty subslices of a sequence"
    # subslices('ABCD') --> A AB ABC ABCD B BC BCD C CD D
    slices = starmap(slice, combinations(range(len(seq) + 1), 2))
    return map(operator.getitem, repeat(seq), slices)

def powerset(iterable):
    "powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)"
    s = list(iterable)
    return chain.from_iterable(combinations(s, r) for r in range(len(s)+1))

def unique_everseen(iterable, key=None):
    "List unique elements, preserving order. Remember all elements ever seen."
    # unique_everseen('AAAABBBCCDAABBB') --> A B C D
    # unique_everseen('ABBCcAD', str.lower) --> A B C D
    seen = set()
    seen_add = seen.add
    if key is None:
        for element in filterfalse(seen.__contains__, iterable):
            seen_add(element)
            yield element
    else:
        for element in iterable:
            k = key(element)
            if k not in seen:
                seen_add(k)
                yield element

def unique_justseen(iterable, key=None):
    "List unique elements, preserving order. Remember only the element just seen."
    # unique_justseen('AAAABBBCCDAABBB') --> A B C D A B
    # unique_justseen('ABBCcAD', str.lower) --> A B C A D
    return map(next, map(operator.itemgetter(1), groupby(iterable, key)))

def iter_except(func, exception, first=None):
    """ Call a function repeatedly until an exception is raised.

    Converts a call-until-exception interface to an iterator interface.
    Like builtins.iter(func, sentinel) but uses an exception instead
    of a sentinel to end the loop.

    Examples:
        iter_except(functools.partial(heappop, h), IndexError)   # priority queue iterator
        iter_except(d.popitem, KeyError)                         # non-blocking dict iterator
        iter_except(d.popleft, IndexError)                       # non-blocking deque iterator
        iter_except(q.get_nowait, Queue.Empty)                   # loop over a producer Queue
        iter_except(s.pop, KeyError)                             # non-blocking set iterator

    """
    try:
        if first is not None:
            yield first()            # For database APIs needing an initial cast to db.first()
        while True:
            yield func()
    except exception:
        pass

def first_true(iterable, default=False, pred=None):
    """Returns the first true value in the iterable.

    If no true value is found, returns *default*

    If *pred* is not None, returns the first item
    for which pred(item) is true.

    """
    # first_true([a,b,c], x) --> a or b or c or x
    # first_true([a,b], x, f) --> a if f(a) else b if f(b) else x
    return next(filter(pred, iterable), default)

def random_product(*args, repeat=1):
    "Random selection from itertools.product(*args, **kwds)"
    pools = [tuple(pool) for pool in args] * repeat
    return tuple(map(random.choice, pools))

def random_permutation(iterable, r=None):
    "Random selection from itertools.permutations(iterable, r)"
    pool = tuple(iterable)
    r = len(pool) if r is None else r
    return tuple(random.sample(pool, r))

def random_combination(iterable, r):
    "Random selection from itertools.combinations(iterable, r)"
    pool = tuple(iterable)
    n = len(pool)
    indices = sorted(random.sample(range(n), r))
    return tuple(pool[i] for i in indices)

def random_combination_with_replacement(iterable, r):
    "Random selection from itertools.combinations_with_replacement(iterable, r)"
    pool = tuple(iterable)
    n = len(pool)
    indices = sorted(random.choices(range(n), k=r))
    return tuple(pool[i] for i in indices)

def nth_combination(iterable, r, index):
    "Equivalent to list(combinations(iterable, r))[index]"
    pool = tuple(iterable)
    n = len(pool)
    if r < 0 or r > n:
        raise ValueError
    c = 1
    k = min(r, n-r)
    for i in range(1, k+1):
        c = c * (n - k + i) // i
    if index < 0:
        index += c
    if index < 0 or index >= c:
        raise IndexError
    result = []
    while r:
        c, n, r = c*r//n, n-1, r-1
        while index >= c:
            index -= c
            c, n = c*(n-r)//n, n-1
        result.append(pool[-1-n])
    return tuple(result)
@end example

@c These examples no longer appear in the docs but are guaranteed
@c to keep working.
@c 
@c >>> amounts = [120.15, 764.05, 823.14]
@c >>> for checknum, amount in zip(count(1200), amounts):
@c ...     print('Check %d is for $%.2f' % (checknum, amount))
@c ...
@c Check 1200 is for $120.15
@c Check 1201 is for $764.05
@c Check 1202 is for $823.14
@c 
@c >>> import operator
@c >>> for cube in map(operator.pow, range(1,4), repeat(3)):
@c ...    print(cube)
@c ...
@c 1
@c 8
@c 27
@c 
@c >>> reportlines = ['EuroPython', 'Roster', '', 'alex', '', 'laura', '', 'martin', '', 'walter', '', 'samuele']
@c >>> for name in islice(reportlines, 3, None, 2):
@c ...    print(name.title())
@c ...
@c Alex
@c Laura
@c Martin
@c Walter
@c Samuele
@c 
@c >>> from operator import itemgetter
@c >>> d = dict(a=1, b=2, c=1, d=2, e=1, f=2, g=3)
@c >>> di = sorted(sorted(d.items()), key=itemgetter(1))
@c >>> for k, g in groupby(di, itemgetter(1)):
@c ...     print(k, list(map(itemgetter(0), g)))
@c ...
@c 1 ['a', 'c', 'e']
@c 2 ['b', 'd', 'f']
@c 3 ['g']
@c 
@c # Find runs of consecutive numbers using groupby.  The key to the solution
@c # is differencing with a range so that consecutive numbers all appear in
@c # same group.
@c >>> data = [ 1,  4,5,6, 10, 15,16,17,18, 22, 25,26,27,28]
@c >>> for k, g in groupby(enumerate(data), lambda t:t[0]-t[1]):
@c ...     print(list(map(operator.itemgetter(1), g)))
@c ...
@c [1]
@c [4, 5, 6]
@c [10]
@c [15, 16, 17, 18]
@c [22]
@c [25, 26, 27, 28]
@c 
@c Now, we test all of the itertool recipes
@c 
@c >>> import operator
@c >>> import collections
@c 
@c >>> take(10, count())
@c [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
@c 
@c >>> list(prepend(1, [2, 3, 4]))
@c [1, 2, 3, 4]
@c 
@c >>> list(enumerate('abc'))
@c [(0, 'a'), (1, 'b'), (2, 'c')]
@c 
@c >>> list(islice(tabulate(lambda x: 2*x), 4))
@c [0, 2, 4, 6]
@c 
@c >>> list(tail(3, 'ABCDEFG'))
@c ['E', 'F', 'G']
@c 
@c >>> it = iter(range(10))
@c >>> consume(it, 3)
@c >>> next(it)
@c 3
@c >>> consume(it)
@c >>> next(it, 'Done')
@c 'Done'
@c 
@c >>> nth('abcde', 3)
@c 'd'
@c 
@c >>> nth('abcde', 9) is None
@c True
@c 
@c >>> [all_equal(s) for s in ('', 'A', 'AAAA', 'AAAB', 'AAABA')]
@c [True, True, True, False, False]
@c 
@c >>> quantify(range(99), lambda x: x%2==0)
@c 50
@c 
@c >>> quantify([True, False, False, True, True])
@c 3
@c 
@c >>> quantify(range(12), pred=lambda x: x%2==1)
@c 6
@c 
@c >>> a = [[1, 2, 3], [4, 5, 6]]
@c >>> list(flatten(a))
@c [1, 2, 3, 4, 5, 6]
@c 
@c >>> list(repeatfunc(pow, 5, 2, 3))
@c [8, 8, 8, 8, 8]
@c 
@c >>> import random
@c >>> take(5, map(int, repeatfunc(random.random)))
@c [0, 0, 0, 0, 0]
@c 
@c >>> list(islice(pad_none('abc'), 0, 6))
@c ['a', 'b', 'c', None, None, None]
@c 
@c >>> list(ncycles('abc', 3))
@c ['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c']
@c 
@c >>> dotproduct([1,2,3], [4,5,6])
@c 32
@c 
@c >>> data = [20, 40, 24, 32, 20, 28, 16]
@c >>> list(convolve(data, [0.25, 0.25, 0.25, 0.25]))
@c [5.0, 15.0, 21.0, 29.0, 29.0, 26.0, 24.0, 16.0, 11.0, 4.0]
@c >>> list(convolve(data, [1, -1]))
@c [20, 20, -16, 8, -12, 8, -12, -16]
@c >>> list(convolve(data, [1, -2, 1]))
@c [20, 0, -36, 24, -20, 20, -20, -4, 16]
@c 
@c >>> list(flatten([('a', 'b'), (), ('c', 'd', 'e'), ('f',), ('g', 'h', 'i')]))
@c ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i']
@c 
@c >>> import random
@c >>> random.seed(85753098575309)
@c >>> list(repeatfunc(random.random, 3))
@c [0.16370491282496968, 0.45889608687313455, 0.3747076837820118]
@c >>> list(repeatfunc(chr, 3, 65))
@c ['A', 'A', 'A']
@c >>> list(repeatfunc(pow, 3, 2, 5))
@c [32, 32, 32]
@c 
@c >>> list(grouper('abcdefg', 3, fillvalue='x'))
@c [('a', 'b', 'c'), ('d', 'e', 'f'), ('g', 'x', 'x')]
@c 
@c >>> it = grouper('abcdefg', 3, incomplete='strict')
@c >>> next(it)
@c ('a', 'b', 'c')
@c >>> next(it)
@c ('d', 'e', 'f')
@c >>> next(it)
@c Traceback (most recent call last):
@c   ...
@c ValueError: zip() argument 2 is shorter than argument 1
@c 
@c >>> list(grouper('abcdefg', n=3, incomplete='ignore'))
@c [('a', 'b', 'c'), ('d', 'e', 'f')]
@c 
@c >>> list(triplewise('ABCDEFG'))
@c [('A', 'B', 'C'), ('B', 'C', 'D'), ('C', 'D', 'E'), ('D', 'E', 'F'), ('E', 'F', 'G')]
@c 
@c >>> list(sliding_window('ABCDEFG', 4))
@c [('A', 'B', 'C', 'D'), ('B', 'C', 'D', 'E'), ('C', 'D', 'E', 'F'), ('D', 'E', 'F', 'G')]
@c 
@c >>> list(roundrobin('abc', 'd', 'ef'))
@c ['a', 'd', 'e', 'b', 'f', 'c']
@c 
@c >>> def is_odd(x):
@c ...     return x % 2 == 1
@c 
@c >>> evens, odds = partition(is_odd, range(10))
@c >>> list(evens)
@c [0, 2, 4, 6, 8]
@c >>> list(odds)
@c [1, 3, 5, 7, 9]
@c 
@c >>> it = iter('ABCdEfGhI')
@c >>> all_upper, remainder = before_and_after(str.isupper, it)
@c >>> ''.join(all_upper)
@c 'ABC'
@c >>> ''.join(remainder)
@c 'dEfGhI'
@c 
@c >>> list(subslices('ABCD'))
@c ['A', 'AB', 'ABC', 'ABCD', 'B', 'BC', 'BCD', 'C', 'CD', 'D']
@c 
@c >>> list(powerset([1,2,3]))
@c [(), (1,), (2,), (3,), (1, 2), (1, 3), (2, 3), (1, 2, 3)]
@c 
@c >>> all(len(list(powerset(range(n)))) == 2**n for n in range(18))
@c True
@c 
@c >>> list(powerset('abcde')) == sorted(sorted(set(powerset('abcde'))), key=len)
@c True
@c 
@c >>> list(unique_everseen('AAAABBBCCDAABBB'))
@c ['A', 'B', 'C', 'D']
@c 
@c >>> list(unique_everseen('ABBCcAD', str.lower))
@c ['A', 'B', 'C', 'D']
@c 
@c >>> list(unique_justseen('AAAABBBCCDAABBB'))
@c ['A', 'B', 'C', 'D', 'A', 'B']
@c 
@c >>> list(unique_justseen('ABBCcAD', str.lower))
@c ['A', 'B', 'C', 'A', 'D']
@c 
@c >>> d = dict(a=1, b=2, c=3)
@c >>> it = iter_except(d.popitem, KeyError)
@c >>> d['d'] = 4
@c >>> next(it)
@c ('d', 4)
@c >>> next(it)
@c ('c', 3)
@c >>> next(it)
@c ('b', 2)
@c >>> d['e'] = 5
@c >>> next(it)
@c ('e', 5)
@c >>> next(it)
@c ('a', 1)
@c >>> next(it, 'empty')
@c 'empty'
@c 
@c >>> first_true('ABC0DEF1', '9', str.isdigit)
@c '0'
@c 
@c >>> population = 'ABCDEFGH'
@c >>> for r in range(len(population) + 1):
@c ...     seq = list(combinations(population, r))
@c ...     for i in range(len(seq)):
@c ...         assert nth_combination(population, r, i) == seq[i]
@c ...     for i in range(-len(seq), 0):
@c ...         assert nth_combination(population, r, i) == seq[i]
@c 
@c >>> iterable = 'abcde'
@c >>> r = 3
@c >>> combos = list(combinations(iterable, r))
@c >>> all(nth_combination(iterable, r, i) == comb for i, comb in enumerate(combos))
@c True

@node functools — Higher-order functions and operations on callable objects,operator — Standard operators as functions,itertools — Functions creating iterators for efficient looping,Functional Programming Modules
@anchor{library/functools doc}@anchor{1fe9}@anchor{library/functools functools-higher-order-functions-and-operations-on-callable-objects}@anchor{1fea}@anchor{library/functools module-functools}@anchor{7f}
@subsection @code{functools} — Higher-order functions and operations on callable objects


@geindex module; functools

@strong{Source code:} Lib/functools.py@footnote{https://github.com/python/cpython/tree/main/Lib/functools.py}

@c import functools
@c from functools import *


__________________________________________________________________


The @ref{7f,,functools} module is for higher-order functions: functions that act on
or return other functions. In general, any callable object can be treated as a
function for the purposes of this module.

The @ref{7f,,functools} module defines the following functions:

@geindex cache() (in module functools)
@anchor{library/functools functools cache}@anchor{1e02}
@deffn {Function} @@functools.cache (user_function)

Simple lightweight unbounded function cache.  Sometimes called
"memoize"@footnote{https://en.wikipedia.org/wiki/Memoization}.

Returns the same as @code{lru_cache(maxsize=None)}, creating a thin
wrapper around a dictionary lookup for the function arguments.  Because it
never needs to evict old values, this is smaller and faster than
@ref{4f3,,lru_cache()} with a size limit.

For example:

@example
@@cache
def factorial(n):
    return n * factorial(n-1) if n else 1

>>> factorial(10)      # no previously cached result, makes 11 recursive calls
3628800
>>> factorial(5)       # just looks up cached value result
120
>>> factorial(12)      # makes two new recursive calls, the other 10 are cached
479001600
@end example


New in version 3.9.

@end deffn

@geindex cached_property() (in module functools)
@anchor{library/functools functools cached_property}@anchor{4f4}
@deffn {Function} @@functools.cached_property (func)

Transform a method of a class into a property whose value is computed once
and then cached as a normal attribute for the life of the instance. Similar
to @ref{501,,property()}, with the addition of caching. Useful for expensive
computed properties of instances that are otherwise effectively immutable.

Example:

@example
class DataSet:

    def __init__(self, sequence_of_numbers):
        self._data = tuple(sequence_of_numbers)

    @@cached_property
    def stdev(self):
        return statistics.stdev(self._data)
@end example

The mechanics of @ref{4f4,,cached_property()} are somewhat different from
@ref{501,,property()}.  A regular property blocks attribute writes unless a
setter is defined. In contrast, a @emph{cached_property} allows writes.

The @emph{cached_property} decorator only runs on lookups and only when an
attribute of the same name doesn’t exist.  When it does run, the
@emph{cached_property} writes to the attribute with the same name. Subsequent
attribute reads and writes take precedence over the @emph{cached_property}
method and it works like a normal attribute.

The cached value can be cleared by deleting the attribute.  This
allows the @emph{cached_property} method to run again.

Note, this decorator interferes with the operation of 
@geindex Python Enhancement Proposals; PEP 412
PEP 412@footnote{https://www.python.org/dev/peps/pep-0412}
key-sharing dictionaries.  This means that instance dictionaries
can take more space than usual.

Also, this decorator requires that the @code{__dict__} attribute on each instance
be a mutable mapping. This means it will not work with some types, such as
metaclasses (since the @code{__dict__} attributes on type instances are
read-only proxies for the class namespace), and those that specify
@code{__slots__} without including @code{__dict__} as one of the defined slots
(as such classes don’t provide a @code{__dict__} attribute at all).

If a mutable mapping is not available or if space-efficient key sharing
is desired, an effect similar to @ref{4f4,,cached_property()} can be achieved
by a stacking @ref{501,,property()} on top of @ref{1e02,,cache()}:

@example
class DataSet:
    def __init__(self, sequence_of_numbers):
        self._data = sequence_of_numbers

    @@property
    @@cache
    def stdev(self):
        return statistics.stdev(self._data)
@end example


New in version 3.8.

@end deffn

@geindex cmp_to_key() (in module functools)
@anchor{library/functools functools cmp_to_key}@anchor{d7c}
@deffn {Function} functools.cmp_to_key (func)

Transform an old-style comparison function to a @ref{964,,key function}.  Used
with tools that accept key functions (such as @ref{704,,sorted()}, @ref{a7d,,min()},
@ref{a7e,,max()}, @ref{1df4,,heapq.nlargest()}, @ref{1df5,,heapq.nsmallest()},
@ref{1fe1,,itertools.groupby()}).  This function is primarily used as a transition
tool for programs being converted from Python 2 which supported the use of
comparison functions.

A comparison function is any callable that accept two arguments, compares them,
and returns a negative number for less-than, zero for equality, or a positive
number for greater-than.  A key function is a callable that accepts one
argument and returns another value to be used as the sort key.

Example:

@example
sorted(iterable, key=cmp_to_key(locale.strcoll))  # locale-aware sort order
@end example

For sorting examples and a brief sorting tutorial, see @ref{1a5e,,Sorting HOW TO}.


New in version 3.2.

@end deffn

@geindex lru_cache() (in module functools)
@anchor{library/functools functools lru_cache}@anchor{4f3}
@deffn {Function} @@functools.lru_cache (user_function)

@deffnx {Function} @@functools.lru_cache (maxsize=128, typed=False)

Decorator to wrap a function with a memoizing callable that saves up to the
@emph{maxsize} most recent calls.  It can save time when an expensive or I/O bound
function is periodically called with the same arguments.

Since a dictionary is used to cache results, the positional and keyword
arguments to the function must be hashable.

Distinct argument patterns may be considered to be distinct calls with
separate cache entries.  For example, @cite{f(a=1@comma{} b=2)} and @cite{f(b=2@comma{} a=1)}
differ in their keyword argument order and may have two separate cache
entries.

If @emph{user_function} is specified, it must be a callable. This allows the
@emph{lru_cache} decorator to be applied directly to a user function, leaving
the @emph{maxsize} at its default value of 128:

@example
@@lru_cache
def count_vowels(sentence):
    return sum(sentence.count(vowel) for vowel in 'AEIOUaeiou')
@end example

If @emph{maxsize} is set to @code{None}, the LRU feature is disabled and the cache can
grow without bound.

If @emph{typed} is set to true, function arguments of different types will be
cached separately.  If @emph{typed} is false, the implementation will usually
regard them as equivalent calls and only cache a single result. (Some
types such as @emph{str} and @emph{int} may be cached separately even when @emph{typed}
is false.)

Note, type specificity applies only to the function’s immediate arguments
rather than their contents.  The scalar arguments, @code{Decimal(42)} and
@code{Fraction(42)} are be treated as distinct calls with distinct results.
In contrast, the tuple arguments @code{('answer', Decimal(42))} and
@code{('answer', Fraction(42))} are treated as equivalent.

The wrapped function is instrumented with a @code{cache_parameters()}
function that returns a new @ref{3b0,,dict} showing the values for @emph{maxsize}
and @emph{typed}.  This is for information purposes only.  Mutating the values
has no effect.

To help measure the effectiveness of the cache and tune the @emph{maxsize}
parameter, the wrapped function is instrumented with a @code{cache_info()}
function that returns a @ref{ce9,,named tuple} showing @emph{hits}, @emph{misses},
@emph{maxsize} and @emph{currsize}.

The decorator also provides a @code{cache_clear()} function for clearing or
invalidating the cache.

The original underlying function is accessible through the
@code{__wrapped__} attribute.  This is useful for introspection, for
bypassing the cache, or for rewrapping the function with a different cache.

The cache keeps references to the arguments and return values until they age
out of the cache or until the cache is cleared.

An LRU (least recently used) cache@footnote{https://en.wikipedia.org/wiki/Cache_replacement_policies#Least_recently_used_(LRU)}
works best when the most recent calls are the best predictors of upcoming
calls (for example, the most popular articles on a news server tend to
change each day).  The cache’s size limit assures that the cache does not
grow without bound on long-running processes such as web servers.

In general, the LRU cache should only be used when you want to reuse
previously computed values.  Accordingly, it doesn’t make sense to cache
functions with side-effects, functions that need to create distinct mutable
objects on each call, or impure functions such as time() or random().

Example of an LRU cache for static web content:

@example
@@lru_cache(maxsize=32)
def get_pep(num):
    'Retrieve text of a Python Enhancement Proposal'
    resource = 'https://www.python.org/dev/peps/pep-%04d/' % num
    try:
        with urllib.request.urlopen(resource) as s:
            return s.read()
    except urllib.error.HTTPError:
        return 'Not Found'

>>> for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991:
...     pep = get_pep(n)
...     print(n, len(pep))

>>> get_pep.cache_info()
CacheInfo(hits=3, misses=8, maxsize=32, currsize=8)
@end example

Example of efficiently computing
Fibonacci numbers@footnote{https://en.wikipedia.org/wiki/Fibonacci_number}
using a cache to implement a
dynamic programming@footnote{https://en.wikipedia.org/wiki/Dynamic_programming}
technique:

@example
@@lru_cache(maxsize=None)
def fib(n):
    if n < 2:
        return n
    return fib(n-1) + fib(n-2)

>>> [fib(n) for n in range(16)]
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610]

>>> fib.cache_info()
CacheInfo(hits=28, misses=16, maxsize=None, currsize=16)
@end example


New in version 3.2.



Changed in version 3.3: Added the @emph{typed} option.



Changed in version 3.8: Added the @emph{user_function} option.



New in version 3.9: Added the function @code{cache_parameters()}

@end deffn

@geindex total_ordering() (in module functools)
@anchor{library/functools functools total_ordering}@anchor{abf}
@deffn {Function} @@functools.total_ordering

Given a class defining one or more rich comparison ordering methods, this
class decorator supplies the rest.  This simplifies the effort involved
in specifying all of the possible rich comparison operations:

The class must define one of @code{__lt__()}, @code{__le__()},
@code{__gt__()}, or @code{__ge__()}.
In addition, the class should supply an @code{__eq__()} method.

For example:

@example
@@total_ordering
class Student:
    def _is_valid_operand(self, other):
        return (hasattr(other, "lastname") and
                hasattr(other, "firstname"))
    def __eq__(self, other):
        if not self._is_valid_operand(other):
            return NotImplemented
        return ((self.lastname.lower(), self.firstname.lower()) ==
                (other.lastname.lower(), other.firstname.lower()))
    def __lt__(self, other):
        if not self._is_valid_operand(other):
            return NotImplemented
        return ((self.lastname.lower(), self.firstname.lower()) <
                (other.lastname.lower(), other.firstname.lower()))
@end example

@cartouche
@quotation Note 
While this decorator makes it easy to create well behaved totally
ordered types, it @emph{does} come at the cost of slower execution and
more complex stack traces for the derived comparison methods. If
performance benchmarking indicates this is a bottleneck for a given
application, implementing all six rich comparison methods instead is
likely to provide an easy speed boost.
@end quotation
@end cartouche

@cartouche
@quotation Note 
This decorator makes no attempt to override methods that have been
declared in the class @emph{or its superclasses}. Meaning that if a
superclass defines a comparison operator, @emph{total_ordering} will not
implement it again, even if the original method is abstract.
@end quotation
@end cartouche


New in version 3.2.



Changed in version 3.4: Returning NotImplemented from the underlying comparison function for
unrecognised types is now supported.

@end deffn

@geindex partial() (in module functools)
@anchor{library/functools functools partial}@anchor{a3f}
@deffn {Function} functools.partial (func, /, *args, **keywords)

Return a new @ref{1feb,,partial object} which when called
will behave like @emph{func} called with the positional arguments @emph{args}
and keyword arguments @emph{keywords}. If more arguments are supplied to the
call, they are appended to @emph{args}. If additional keyword arguments are
supplied, they extend and override @emph{keywords}.
Roughly equivalent to:

@example
def partial(func, /, *args, **keywords):
    def newfunc(*fargs, **fkeywords):
        newkeywords = @{**keywords, **fkeywords@}
        return func(*args, *fargs, **newkeywords)
    newfunc.func = func
    newfunc.args = args
    newfunc.keywords = keywords
    return newfunc
@end example

The @ref{a3f,,partial()} is used for partial function application which “freezes”
some portion of a function’s arguments and/or keywords resulting in a new object
with a simplified signature.  For example, @ref{a3f,,partial()} can be used to create
a callable that behaves like the @ref{1b1,,int()} function where the @emph{base} argument
defaults to two:

@example
>>> from functools import partial
>>> basetwo = partial(int, base=2)
>>> basetwo.__doc__ = 'Convert base 2 string to an int.'
>>> basetwo('10010')
18
@end example
@end deffn

@geindex partialmethod (class in functools)
@anchor{library/functools functools partialmethod}@anchor{593}
@deffn {Class} functools.partialmethod (func, /, *args, **keywords)

Return a new @ref{593,,partialmethod} descriptor which behaves
like @ref{a3f,,partial} except that it is designed to be used as a method
definition rather than being directly callable.

@emph{func} must be a @ref{17bc,,descriptor} or a callable (objects which are both,
like normal functions, are handled as descriptors).

When @emph{func} is a descriptor (such as a normal Python function,
@ref{254,,classmethod()}, @ref{253,,staticmethod()}, @code{abstractmethod()} or
another instance of @ref{593,,partialmethod}), calls to @code{__get__} are
delegated to the underlying descriptor, and an appropriate
@ref{1feb,,partial object} returned as the result.

When @emph{func} is a non-descriptor callable, an appropriate bound method is
created dynamically. This behaves like a normal Python function when
used as a method: the @emph{self} argument will be inserted as the first
positional argument, even before the @emph{args} and @emph{keywords} supplied to
the @ref{593,,partialmethod} constructor.

Example:

@example
>>> class Cell:
...     def __init__(self):
...         self._alive = False
...     @@property
...     def alive(self):
...         return self._alive
...     def set_state(self, state):
...         self._alive = bool(state)
...     set_alive = partialmethod(set_state, True)
...     set_dead = partialmethod(set_state, False)
...
>>> c = Cell()
>>> c.alive
False
>>> c.set_alive()
>>> c.alive
True
@end example


New in version 3.4.

@end deffn

@geindex reduce() (in module functools)
@anchor{library/functools functools reduce}@anchor{e77}
@deffn {Function} functools.reduce (function, iterable[, initializer])

Apply @emph{function} of two arguments cumulatively to the items of @emph{iterable}, from
left to right, so as to reduce the iterable to a single value.  For example,
@code{reduce(lambda x, y: x+y, [1, 2, 3, 4, 5])} calculates @code{((((1+2)+3)+4)+5)}.
The left argument, @emph{x}, is the accumulated value and the right argument, @emph{y}, is
the update value from the @emph{iterable}.  If the optional @emph{initializer} is present,
it is placed before the items of the iterable in the calculation, and serves as
a default when the iterable is empty.  If @emph{initializer} is not given and
@emph{iterable} contains only one item, the first item is returned.

Roughly equivalent to:

@example
def reduce(function, iterable, initializer=None):
    it = iter(iterable)
    if initializer is None:
        value = next(it)
    else:
        value = initializer
    for element in it:
        value = function(value, element)
    return value
@end example

See @ref{505,,itertools.accumulate()} for an iterator that yields all intermediate
values.
@end deffn

@geindex singledispatch() (in module functools)
@anchor{library/functools functools singledispatch}@anchor{65b}
@deffn {Function} @@functools.singledispatch

Transform a function into a @ref{4f7,,single-dispatch} @ref{4f6,,generic function}.

To define a generic function, decorate it with the @code{@@singledispatch}
decorator. When defining a function using @code{@@singledispatch}, note that the
dispatch happens on the type of the first argument:

@example
>>> from functools import singledispatch
>>> @@singledispatch
... def fun(arg, verbose=False):
...     if verbose:
...         print("Let me just say,", end=" ")
...     print(arg)
@end example

To add overloaded implementations to the function, use the @code{register()}
attribute of the generic function, which can be used as a decorator.  For
functions annotated with types, the decorator will infer the type of the
first argument automatically:

@example
>>> @@fun.register
... def _(arg: int, verbose=False):
...     if verbose:
...         print("Strength in numbers, eh?", end=" ")
...     print(arg)
...
>>> @@fun.register
... def _(arg: list, verbose=False):
...     if verbose:
...         print("Enumerate this:")
...     for i, elem in enumerate(arg):
...         print(i, elem)
@end example

For code which doesn’t use type annotations, the appropriate type
argument can be passed explicitly to the decorator itself:

@example
>>> @@fun.register(complex)
... def _(arg, verbose=False):
...     if verbose:
...         print("Better than complicated.", end=" ")
...     print(arg.real, arg.imag)
...
@end example

To enable registering @ref{1fec,,lambdas} and pre-existing functions,
the @code{register()} attribute can also be used in a functional form:

@example
>>> def nothing(arg, verbose=False):
...     print("Nothing.")
...
>>> fun.register(type(None), nothing)
@end example

The @code{register()} attribute returns the undecorated function. This
enables decorator stacking, @ref{c5,,pickling}, and the creation
of unit tests for each variant independently:

@example
>>> @@fun.register(float)
... @@fun.register(Decimal)
... def fun_num(arg, verbose=False):
...     if verbose:
...         print("Half of your number:", end=" ")
...     print(arg / 2)
...
>>> fun_num is fun
False
@end example

When called, the generic function dispatches on the type of the first
argument:

@example
>>> fun("Hello, world.")
Hello, world.
>>> fun("test.", verbose=True)
Let me just say, test.
>>> fun(42, verbose=True)
Strength in numbers, eh? 42
>>> fun(['spam', 'spam', 'eggs', 'spam'], verbose=True)
Enumerate this:
0 spam
1 spam
2 eggs
3 spam
>>> fun(None)
Nothing.
>>> fun(1.23)
0.615
@end example

Where there is no registered implementation for a specific type, its
method resolution order is used to find a more generic implementation.
The original function decorated with @code{@@singledispatch} is registered
for the base @ref{5a5,,object} type, which means it is used if no better
implementation is found.

If an implementation is registered to an @ref{d5c,,abstract base class},
virtual subclasses of the base class will be dispatched to that
implementation:

@example
>>> from collections.abc import Mapping
>>> @@fun.register
... def _(arg: Mapping, verbose=False):
...     if verbose:
...         print("Keys & Values")
...     for key, value in arg.items():
...         print(key, "=>", value)
...
>>> fun(@{"a": "b"@})
a => b
@end example

To check which implementation the generic function will choose for
a given type, use the @code{dispatch()} attribute:

@example
>>> fun.dispatch(float)
<function fun_num at 0x1035a2840>
>>> fun.dispatch(dict)    # note: default implementation
<function fun at 0x103fe0000>
@end example

To access all registered implementations, use the read-only @code{registry}
attribute:

@example
>>> fun.registry.keys()
dict_keys([<class 'NoneType'>, <class 'int'>, <class 'object'>,
          <class 'decimal.Decimal'>, <class 'list'>,
          <class 'float'>])
>>> fun.registry[float]
<function fun_num at 0x1035a2840>
>>> fun.registry[object]
<function fun at 0x103fe0000>
@end example


New in version 3.4.



Changed in version 3.7: The @code{register()} attribute now supports using type annotations.

@end deffn

@geindex singledispatchmethod (class in functools)
@anchor{library/functools functools singledispatchmethod}@anchor{4f5}
@deffn {Class} functools.singledispatchmethod (func)

Transform a method into a @ref{4f7,,single-dispatch} @ref{4f6,,generic function}.

To define a generic method, decorate it with the @code{@@singledispatchmethod}
decorator. When defining a function using @code{@@singledispatchmethod}, note
that the dispatch happens on the type of the first non-@emph{self} or non-@emph{cls}
argument:

@example
class Negator:
    @@singledispatchmethod
    def neg(self, arg):
        raise NotImplementedError("Cannot negate a")

    @@neg.register
    def _(self, arg: int):
        return -arg

    @@neg.register
    def _(self, arg: bool):
        return not arg
@end example

@code{@@singledispatchmethod} supports nesting with other decorators such as
@ref{254,,@@classmethod}. Note that to allow for
@code{dispatcher.register}, @code{singledispatchmethod} must be the @emph{outer most}
decorator. Here is the @code{Negator} class with the @code{neg} methods bound to
the class, rather than an instance of the class:

@example
class Negator:
    @@singledispatchmethod
    @@classmethod
    def neg(cls, arg):
        raise NotImplementedError("Cannot negate a")

    @@neg.register
    @@classmethod
    def _(cls, arg: int):
        return -arg

    @@neg.register
    @@classmethod
    def _(cls, arg: bool):
        return not arg
@end example

The same pattern can be used for other similar decorators:
@ref{253,,@@staticmethod},
@ref{c1e,,@@abstractmethod}, and others.


New in version 3.8.

@end deffn

@geindex update_wrapper() (in module functools)
@anchor{library/functools functools update_wrapper}@anchor{bb3}
@deffn {Function} functools.update_wrapper (wrapper, wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES)

Update a @emph{wrapper} function to look like the @emph{wrapped} function. The optional
arguments are tuples to specify which attributes of the original function are
assigned directly to the matching attributes on the wrapper function and which
attributes of the wrapper function are updated with the corresponding attributes
from the original function. The default values for these arguments are the
module level constants @code{WRAPPER_ASSIGNMENTS} (which assigns to the wrapper
function’s @code{__module__}, @code{__name__}, @code{__qualname__}, @code{__annotations__}
and @code{__doc__}, the documentation string) and @code{WRAPPER_UPDATES} (which
updates the wrapper function’s @code{__dict__}, i.e. the instance dictionary).

To allow access to the original function for introspection and other purposes
(e.g. bypassing a caching decorator such as @ref{4f3,,lru_cache()}), this function
automatically adds a @code{__wrapped__} attribute to the wrapper that refers to
the function being wrapped.

The main intended use for this function is in @ref{1bf,,decorator} functions which
wrap the decorated function and return the wrapper. If the wrapper function is
not updated, the metadata of the returned function will reflect the wrapper
definition rather than the original function definition, which is typically less
than helpful.

@ref{bb3,,update_wrapper()} may be used with callables other than functions. Any
attributes named in @emph{assigned} or @emph{updated} that are missing from the object
being wrapped are ignored (i.e. this function will not attempt to set them
on the wrapper function). @ref{163,,AttributeError} is still raised if the
wrapper function itself is missing any attributes named in @emph{updated}.


New in version 3.2: Automatic addition of the @code{__wrapped__} attribute.



New in version 3.2: Copying of the @code{__annotations__} attribute by default.



Changed in version 3.2: Missing attributes no longer trigger an @ref{163,,AttributeError}.



Changed in version 3.4: The @code{__wrapped__} attribute now always refers to the wrapped
function, even if that function defined a @code{__wrapped__} attribute.
(see bpo-17482@footnote{https://bugs.python.org/issue17482})

@end deffn

@geindex wraps() (in module functools)
@anchor{library/functools functools wraps}@anchor{adb}
@deffn {Function} @@functools.wraps (wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES)

This is a convenience function for invoking @ref{bb3,,update_wrapper()} as a
function decorator when defining a wrapper function.  It is equivalent to
@code{partial(update_wrapper, wrapped=wrapped, assigned=assigned, updated=updated)}.
For example:

@example
>>> from functools import wraps
>>> def my_decorator(f):
...     @@wraps(f)
...     def wrapper(*args, **kwds):
...         print('Calling decorated function')
...         return f(*args, **kwds)
...     return wrapper
...
>>> @@my_decorator
... def example():
...     """Docstring"""
...     print('Called example function')
...
>>> example()
Calling decorated function
Called example function
>>> example.__name__
'example'
>>> example.__doc__
'Docstring'
@end example

Without the use of this decorator factory, the name of the example function
would have been @code{'wrapper'}, and the docstring of the original @code{example()}
would have been lost.
@end deffn

@menu
* partial Objects:: 

@end menu

@node partial Objects,,,functools — Higher-order functions and operations on callable objects
@anchor{library/functools id1}@anchor{1fed}@anchor{library/functools partial-objects}@anchor{1feb}
@subsubsection @code{partial} Objects


@ref{a3f,,partial} objects are callable objects created by @ref{a3f,,partial()}. They
have three read-only attributes:

@geindex func (functools.partial attribute)
@anchor{library/functools functools partial func}@anchor{1fee}
@deffn {Attribute} partial.func

A callable object or function.  Calls to the @ref{a3f,,partial} object will be
forwarded to @ref{1fee,,func} with new arguments and keywords.
@end deffn

@geindex args (functools.partial attribute)
@anchor{library/functools functools partial args}@anchor{1fef}
@deffn {Attribute} partial.args

The leftmost positional arguments that will be prepended to the positional
arguments provided to a @ref{a3f,,partial} object call.
@end deffn

@geindex keywords (functools.partial attribute)
@anchor{library/functools functools partial keywords}@anchor{1ff0}
@deffn {Attribute} partial.keywords

The keyword arguments that will be supplied when the @ref{a3f,,partial} object is
called.
@end deffn

@ref{a3f,,partial} objects are like @code{function} objects in that they are
callable, weak referencable, and can have attributes.  There are some important
differences.  For instance, the @ref{e70,,__name__} and @code{__doc__} attributes
are not created automatically.  Also, @ref{a3f,,partial} objects defined in
classes behave like static methods and do not transform into bound methods
during instance attribute look-up.

@node operator — Standard operators as functions,,functools — Higher-order functions and operations on callable objects,Functional Programming Modules
@anchor{library/operator doc}@anchor{1ff1}@anchor{library/operator module-operator}@anchor{be}@anchor{library/operator operator-standard-operators-as-functions}@anchor{1ff2}
@subsection @code{operator} — Standard operators as functions


@geindex module; operator

@strong{Source code:} Lib/operator.py@footnote{https://github.com/python/cpython/tree/main/Lib/operator.py}

@c import operator
@c from operator import itemgetter, iadd


__________________________________________________________________


The @ref{be,,operator} module exports a set of efficient functions corresponding to
the intrinsic operators of Python.  For example, @code{operator.add(x, y)} is
equivalent to the expression @code{x+y}. Many function names are those used for
special methods, without the double underscores.  For backward compatibility,
many of these have a variant with the double underscores kept. The variants
without the double underscores are preferred for clarity.

The functions fall into categories that perform object comparisons, logical
operations, mathematical operations and sequence operations.

The object comparison functions are useful for all objects, and are named after
the rich comparison operators they support:

@geindex lt() (in module operator)
@geindex le() (in module operator)
@geindex eq() (in module operator)
@geindex ne() (in module operator)
@geindex ge() (in module operator)
@geindex gt() (in module operator)
@geindex __lt__() (in module operator)
@geindex __le__() (in module operator)
@geindex __eq__() (in module operator)
@geindex __ne__() (in module operator)
@geindex __ge__() (in module operator)
@geindex __gt__() (in module operator)
@anchor{library/operator operator lt}@anchor{1ff3}
@deffn {Function} operator.lt (a, b)
@anchor{library/operator operator le}@anchor{1ff4}
@deffnx {Function} operator.le (a, b)
@anchor{library/operator operator eq}@anchor{1ff5}
@deffnx {Function} operator.eq (a, b)
@anchor{library/operator operator ne}@anchor{1ff6}
@deffnx {Function} operator.ne (a, b)
@anchor{library/operator operator ge}@anchor{1ff7}
@deffnx {Function} operator.ge (a, b)
@anchor{library/operator operator gt}@anchor{1ff8}
@deffnx {Function} operator.gt (a, b)
@anchor{library/operator operator __lt__}@anchor{1ff9}
@deffnx {Function} operator.__lt__ (a, b)
@anchor{library/operator operator __le__}@anchor{1ffa}
@deffnx {Function} operator.__le__ (a, b)
@anchor{library/operator operator __eq__}@anchor{1ffb}
@deffnx {Function} operator.__eq__ (a, b)
@anchor{library/operator operator __ne__}@anchor{1ffc}
@deffnx {Function} operator.__ne__ (a, b)
@anchor{library/operator operator __ge__}@anchor{1ffd}
@deffnx {Function} operator.__ge__ (a, b)
@anchor{library/operator operator __gt__}@anchor{1ffe}
@deffnx {Function} operator.__gt__ (a, b)

Perform “rich comparisons” between @emph{a} and @emph{b}. Specifically, @code{lt(a, b)} is
equivalent to @code{a < b}, @code{le(a, b)} is equivalent to @code{a <= b}, @code{eq(a,
b)} is equivalent to @code{a == b}, @code{ne(a, b)} is equivalent to @code{a != b},
@code{gt(a, b)} is equivalent to @code{a > b} and @code{ge(a, b)} is equivalent to @code{a
>= b}.  Note that these functions can return any value, which may
or may not be interpretable as a Boolean value.  See
@ref{1900,,Comparisons} for more information about rich comparisons.
@end deffn

The logical operations are also generally applicable to all objects, and support
truth tests, identity tests, and boolean operations:

@geindex not_() (in module operator)
@geindex __not__() (in module operator)
@anchor{library/operator operator not_}@anchor{1fff}
@deffn {Function} operator.not_ (obj)
@anchor{library/operator operator __not__}@anchor{2000}
@deffnx {Function} operator.__not__ (obj)

Return the outcome of @ref{190c,,not} @emph{obj}.  (Note that there is no
@ref{2000,,__not__()} method for object instances; only the interpreter core defines
this operation.  The result is affected by the @code{__bool__()} and
@code{__len__()} methods.)
@end deffn

@geindex truth() (in module operator)
@anchor{library/operator operator truth}@anchor{2001}
@deffn {Function} operator.truth (obj)

Return @ref{748,,True} if @emph{obj} is true, and @ref{659,,False} otherwise.  This is
equivalent to using the @ref{2f7,,bool} constructor.
@end deffn

@geindex is_() (in module operator)
@anchor{library/operator operator is_}@anchor{2002}
@deffn {Function} operator.is_ (a, b)

Return @code{a is b}.  Tests object identity.
@end deffn

@geindex is_not() (in module operator)
@anchor{library/operator operator is_not}@anchor{2003}
@deffn {Function} operator.is_not (a, b)

Return @code{a is not b}.  Tests object identity.
@end deffn

The mathematical and bitwise operations are the most numerous:

@geindex abs() (in module operator)
@geindex __abs__() (in module operator)
@anchor{library/operator operator abs}@anchor{2004}
@deffn {Function} operator.abs (obj)
@anchor{library/operator operator __abs__}@anchor{2005}
@deffnx {Function} operator.__abs__ (obj)

Return the absolute value of @emph{obj}.
@end deffn

@geindex add() (in module operator)
@geindex __add__() (in module operator)
@anchor{library/operator operator add}@anchor{2006}
@deffn {Function} operator.add (a, b)
@anchor{library/operator operator __add__}@anchor{2007}
@deffnx {Function} operator.__add__ (a, b)

Return @code{a + b}, for @emph{a} and @emph{b} numbers.
@end deffn

@geindex and_() (in module operator)
@geindex __and__() (in module operator)
@anchor{library/operator operator and_}@anchor{2008}
@deffn {Function} operator.and_ (a, b)
@anchor{library/operator operator __and__}@anchor{2009}
@deffnx {Function} operator.__and__ (a, b)

Return the bitwise and of @emph{a} and @emph{b}.
@end deffn

@geindex floordiv() (in module operator)
@geindex __floordiv__() (in module operator)
@anchor{library/operator operator floordiv}@anchor{200a}
@deffn {Function} operator.floordiv (a, b)
@anchor{library/operator operator __floordiv__}@anchor{200b}
@deffnx {Function} operator.__floordiv__ (a, b)

Return @code{a // b}.
@end deffn

@geindex index() (in module operator)
@geindex __index__() (in module operator)
@anchor{library/operator operator index}@anchor{1187}
@deffn {Function} operator.index (a)
@anchor{library/operator operator __index__}@anchor{200c}
@deffnx {Function} operator.__index__ (a)

Return @emph{a} converted to an integer.  Equivalent to @code{a.__index__()}.


Changed in version 3.10: The result always has exact type @ref{1b1,,int}.  Previously, the result
could have been an instance of a subclass of @code{int}.

@end deffn

@geindex inv() (in module operator)
@geindex invert() (in module operator)
@geindex __inv__() (in module operator)
@geindex __invert__() (in module operator)
@anchor{library/operator operator inv}@anchor{200d}
@deffn {Function} operator.inv (obj)
@anchor{library/operator operator invert}@anchor{200e}
@deffnx {Function} operator.invert (obj)
@anchor{library/operator operator __inv__}@anchor{200f}
@deffnx {Function} operator.__inv__ (obj)
@anchor{library/operator operator __invert__}@anchor{2010}
@deffnx {Function} operator.__invert__ (obj)

Return the bitwise inverse of the number @emph{obj}.  This is equivalent to @code{~obj}.
@end deffn

@geindex lshift() (in module operator)
@geindex __lshift__() (in module operator)
@anchor{library/operator operator lshift}@anchor{2011}
@deffn {Function} operator.lshift (a, b)
@anchor{library/operator operator __lshift__}@anchor{2012}
@deffnx {Function} operator.__lshift__ (a, b)

Return @emph{a} shifted left by @emph{b}.
@end deffn

@geindex mod() (in module operator)
@geindex __mod__() (in module operator)
@anchor{library/operator operator mod}@anchor{2013}
@deffn {Function} operator.mod (a, b)
@anchor{library/operator operator __mod__}@anchor{2014}
@deffnx {Function} operator.__mod__ (a, b)

Return @code{a % b}.
@end deffn

@geindex mul() (in module operator)
@geindex __mul__() (in module operator)
@anchor{library/operator operator mul}@anchor{1fe7}
@deffn {Function} operator.mul (a, b)
@anchor{library/operator operator __mul__}@anchor{2015}
@deffnx {Function} operator.__mul__ (a, b)

Return @code{a * b}, for @emph{a} and @emph{b} numbers.
@end deffn

@geindex matmul() (in module operator)
@geindex __matmul__() (in module operator)
@anchor{library/operator operator matmul}@anchor{99c}
@deffn {Function} operator.matmul (a, b)
@anchor{library/operator operator __matmul__}@anchor{2016}
@deffnx {Function} operator.__matmul__ (a, b)

Return @code{a @@ b}.


New in version 3.5.

@end deffn

@geindex neg() (in module operator)
@geindex __neg__() (in module operator)
@anchor{library/operator operator neg}@anchor{2017}
@deffn {Function} operator.neg (obj)
@anchor{library/operator operator __neg__}@anchor{2018}
@deffnx {Function} operator.__neg__ (obj)

Return @emph{obj} negated (@code{-obj}).
@end deffn

@geindex or_() (in module operator)
@geindex __or__() (in module operator)
@anchor{library/operator operator or_}@anchor{2019}
@deffn {Function} operator.or_ (a, b)
@anchor{library/operator operator __or__}@anchor{201a}
@deffnx {Function} operator.__or__ (a, b)

Return the bitwise or of @emph{a} and @emph{b}.
@end deffn

@geindex pos() (in module operator)
@geindex __pos__() (in module operator)
@anchor{library/operator operator pos}@anchor{201b}
@deffn {Function} operator.pos (obj)
@anchor{library/operator operator __pos__}@anchor{201c}
@deffnx {Function} operator.__pos__ (obj)

Return @emph{obj} positive (@code{+obj}).
@end deffn

@geindex pow() (in module operator)
@geindex __pow__() (in module operator)
@anchor{library/operator operator pow}@anchor{201d}
@deffn {Function} operator.pow (a, b)
@anchor{library/operator operator __pow__}@anchor{201e}
@deffnx {Function} operator.__pow__ (a, b)

Return @code{a ** b}, for @emph{a} and @emph{b} numbers.
@end deffn

@geindex rshift() (in module operator)
@geindex __rshift__() (in module operator)
@anchor{library/operator operator rshift}@anchor{201f}
@deffn {Function} operator.rshift (a, b)
@anchor{library/operator operator __rshift__}@anchor{2020}
@deffnx {Function} operator.__rshift__ (a, b)

Return @emph{a} shifted right by @emph{b}.
@end deffn

@geindex sub() (in module operator)
@geindex __sub__() (in module operator)
@anchor{library/operator operator sub}@anchor{2021}
@deffn {Function} operator.sub (a, b)
@anchor{library/operator operator __sub__}@anchor{2022}
@deffnx {Function} operator.__sub__ (a, b)

Return @code{a - b}.
@end deffn

@geindex truediv() (in module operator)
@geindex __truediv__() (in module operator)
@anchor{library/operator operator truediv}@anchor{2023}
@deffn {Function} operator.truediv (a, b)
@anchor{library/operator operator __truediv__}@anchor{2024}
@deffnx {Function} operator.__truediv__ (a, b)

Return @code{a / b} where 2/3 is .66 rather than 0.  This is also known as
“true” division.
@end deffn

@geindex xor() (in module operator)
@geindex __xor__() (in module operator)
@anchor{library/operator operator xor}@anchor{2025}
@deffn {Function} operator.xor (a, b)
@anchor{library/operator operator __xor__}@anchor{2026}
@deffnx {Function} operator.__xor__ (a, b)

Return the bitwise exclusive or of @emph{a} and @emph{b}.
@end deffn

Operations which work with sequences (some of them with mappings too) include:

@geindex concat() (in module operator)
@geindex __concat__() (in module operator)
@anchor{library/operator operator concat}@anchor{2027}
@deffn {Function} operator.concat (a, b)
@anchor{library/operator operator __concat__}@anchor{2028}
@deffnx {Function} operator.__concat__ (a, b)

Return @code{a + b} for @emph{a} and @emph{b} sequences.
@end deffn

@geindex contains() (in module operator)
@geindex __contains__() (in module operator)
@anchor{library/operator operator contains}@anchor{3c0}
@deffn {Function} operator.contains (a, b)
@anchor{library/operator operator __contains__}@anchor{2029}
@deffnx {Function} operator.__contains__ (a, b)

Return the outcome of the test @code{b in a}. Note the reversed operands.
@end deffn

@geindex countOf() (in module operator)
@anchor{library/operator operator countOf}@anchor{3c2}
@deffn {Function} operator.countOf (a, b)

Return the number of occurrences of @emph{b} in @emph{a}.
@end deffn

@geindex delitem() (in module operator)
@geindex __delitem__() (in module operator)
@anchor{library/operator operator delitem}@anchor{202a}
@deffn {Function} operator.delitem (a, b)
@anchor{library/operator operator __delitem__}@anchor{202b}
@deffnx {Function} operator.__delitem__ (a, b)

Remove the value of @emph{a} at index @emph{b}.
@end deffn

@geindex getitem() (in module operator)
@geindex __getitem__() (in module operator)
@anchor{library/operator operator getitem}@anchor{202c}
@deffn {Function} operator.getitem (a, b)
@anchor{library/operator operator __getitem__}@anchor{202d}
@deffnx {Function} operator.__getitem__ (a, b)

Return the value of @emph{a} at index @emph{b}.
@end deffn

@geindex indexOf() (in module operator)
@anchor{library/operator operator indexOf}@anchor{3c1}
@deffn {Function} operator.indexOf (a, b)

Return the index of the first of occurrence of @emph{b} in @emph{a}.
@end deffn

@geindex setitem() (in module operator)
@geindex __setitem__() (in module operator)
@anchor{library/operator operator setitem}@anchor{202e}
@deffn {Function} operator.setitem (a, b, c)
@anchor{library/operator operator __setitem__}@anchor{202f}
@deffnx {Function} operator.__setitem__ (a, b, c)

Set the value of @emph{a} at index @emph{b} to @emph{c}.
@end deffn

@geindex length_hint() (in module operator)
@anchor{library/operator operator length_hint}@anchor{aee}
@deffn {Function} operator.length_hint (obj, default=0)

Return an estimated length for the object @emph{o}. First try to return its
actual length, then an estimate using @ref{a80,,object.__length_hint__()}, and
finally return the default value.


New in version 3.4.

@end deffn

The following operation works with callables:

@geindex call() (in module operator)
@geindex __call__() (in module operator)
@anchor{library/operator operator call}@anchor{10df}
@deffn {Function} operator.call (obj, /, *args, **kwargs)
@anchor{library/operator operator __call__}@anchor{2030}
@deffnx {Function} operator.__call__ (obj, /, *args, **kwargs)

Return @code{obj(*args, **kwargs)}.


New in version 3.11.

@end deffn

The @ref{be,,operator} module also defines tools for generalized attribute and item
lookups.  These are useful for making fast field extractors as arguments for
@ref{2f4,,map()}, @ref{704,,sorted()}, @ref{1fe1,,itertools.groupby()}, or other functions that
expect a function argument.

@geindex attrgetter() (in module operator)
@anchor{library/operator operator attrgetter}@anchor{99a}
@deffn {Function} operator.attrgetter (attr)

@deffnx {Function} operator.attrgetter (*attrs)

Return a callable object that fetches @emph{attr} from its operand.
If more than one attribute is requested, returns a tuple of attributes.
The attribute names can also contain dots. For example:


@itemize *

@item 
After @code{f = attrgetter('name')}, the call @code{f(b)} returns @code{b.name}.

@item 
After @code{f = attrgetter('name', 'date')}, the call @code{f(b)} returns
@code{(b.name, b.date)}.

@item 
After @code{f = attrgetter('name.first', 'name.last')}, the call @code{f(b)}
returns @code{(b.name.first, b.name.last)}.
@end itemize

Equivalent to:

@example
def attrgetter(*items):
    if any(not isinstance(item, str) for item in items):
        raise TypeError('attribute name must be a string')
    if len(items) == 1:
        attr = items[0]
        def g(obj):
            return resolve_attr(obj, attr)
    else:
        def g(obj):
            return tuple(resolve_attr(obj, attr) for attr in items)
    return g

def resolve_attr(obj, attr):
    for name in attr.split("."):
        obj = getattr(obj, name)
    return obj
@end example
@end deffn

@geindex itemgetter() (in module operator)
@anchor{library/operator operator itemgetter}@anchor{578}
@deffn {Function} operator.itemgetter (item)

@deffnx {Function} operator.itemgetter (*items)

Return a callable object that fetches @emph{item} from its operand using the
operand’s @ref{202d,,__getitem__()} method.  If multiple items are specified,
returns a tuple of lookup values.  For example:


@itemize *

@item 
After @code{f = itemgetter(2)}, the call @code{f(r)} returns @code{r[2]}.

@item 
After @code{g = itemgetter(2, 5, 3)}, the call @code{g(r)} returns
@code{(r[2], r[5], r[3])}.
@end itemize

Equivalent to:

@example
def itemgetter(*items):
    if len(items) == 1:
        item = items[0]
        def g(obj):
            return obj[item]
    else:
        def g(obj):
            return tuple(obj[item] for item in items)
    return g
@end example

The items can be any type accepted by the operand’s @ref{202d,,__getitem__()}
method.  Dictionaries accept any hashable value.  Lists, tuples, and
strings accept an index or a slice:

@example
>>> itemgetter(1)('ABCDEFG')
'B'
>>> itemgetter(1, 3, 5)('ABCDEFG')
('B', 'D', 'F')
>>> itemgetter(slice(2, None))('ABCDEFG')
'CDEFG'
>>> soldier = dict(rank='captain', name='dotterbart')
>>> itemgetter('rank')(soldier)
'captain'
@end example

Example of using @ref{578,,itemgetter()} to retrieve specific fields from a
tuple record:

@example
>>> inventory = [('apple', 3), ('banana', 2), ('pear', 5), ('orange', 1)]
>>> getcount = itemgetter(1)
>>> list(map(getcount, inventory))
[3, 2, 5, 1]
>>> sorted(inventory, key=getcount)
[('orange', 1), ('banana', 2), ('apple', 3), ('pear', 5)]
@end example
@end deffn

@geindex methodcaller() (in module operator)
@anchor{library/operator operator methodcaller}@anchor{99b}
@deffn {Function} operator.methodcaller (name, /, *args, **kwargs)

Return a callable object that calls the method @emph{name} on its operand.  If
additional arguments and/or keyword arguments are given, they will be given
to the method as well.  For example:


@itemize *

@item 
After @code{f = methodcaller('name')}, the call @code{f(b)} returns @code{b.name()}.

@item 
After @code{f = methodcaller('name', 'foo', bar=1)}, the call @code{f(b)}
returns @code{b.name('foo', bar=1)}.
@end itemize

Equivalent to:

@example
def methodcaller(name, /, *args, **kwargs):
    def caller(obj):
        return getattr(obj, name)(*args, **kwargs)
    return caller
@end example
@end deffn

@menu
* Mapping Operators to Functions:: 
* In-place Operators:: 

@end menu

@node Mapping Operators to Functions,In-place Operators,,operator — Standard operators as functions
@anchor{library/operator mapping-operators-to-functions}@anchor{2031}@anchor{library/operator operator-map}@anchor{2032}
@subsubsection Mapping Operators to Functions


This table shows how abstract operations correspond to operator symbols in the
Python syntax and the functions in the @ref{be,,operator} module.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Operation

@tab

Syntax

@tab

Function

@item

Addition

@tab

@code{a + b}

@tab

@code{add(a, b)}

@item

Concatenation

@tab

@code{seq1 + seq2}

@tab

@code{concat(seq1, seq2)}

@item

Containment Test

@tab

@code{obj in seq}

@tab

@code{contains(seq, obj)}

@item

Division

@tab

@code{a / b}

@tab

@code{truediv(a, b)}

@item

Division

@tab

@code{a // b}

@tab

@code{floordiv(a, b)}

@item

Bitwise And

@tab

@code{a & b}

@tab

@code{and_(a, b)}

@item

Bitwise Exclusive Or

@tab

@code{a ^ b}

@tab

@code{xor(a, b)}

@item

Bitwise Inversion

@tab

@code{~ a}

@tab

@code{invert(a)}

@item

Bitwise Or

@tab

@code{a | b}

@tab

@code{or_(a, b)}

@item

Exponentiation

@tab

@code{a ** b}

@tab

@code{pow(a, b)}

@item

Identity

@tab

@code{a is b}

@tab

@code{is_(a, b)}

@item

Identity

@tab

@code{a is not b}

@tab

@code{is_not(a, b)}

@item

Indexed Assignment

@tab

@code{obj[k] = v}

@tab

@code{setitem(obj, k, v)}

@item

Indexed Deletion

@tab

@code{del obj[k]}

@tab

@code{delitem(obj, k)}

@item

Indexing

@tab

@code{obj[k]}

@tab

@code{getitem(obj, k)}

@item

Left Shift

@tab

@code{a << b}

@tab

@code{lshift(a, b)}

@item

Modulo

@tab

@code{a % b}

@tab

@code{mod(a, b)}

@item

Multiplication

@tab

@code{a * b}

@tab

@code{mul(a, b)}

@item

Matrix Multiplication

@tab

@code{a @@ b}

@tab

@code{matmul(a, b)}

@item

Negation (Arithmetic)

@tab

@code{- a}

@tab

@code{neg(a)}

@item

Negation (Logical)

@tab

@code{not a}

@tab

@code{not_(a)}

@item

Positive

@tab

@code{+ a}

@tab

@code{pos(a)}

@item

Right Shift

@tab

@code{a >> b}

@tab

@code{rshift(a, b)}

@item

Slice Assignment

@tab

@code{seq[i:j] = values}

@tab

@code{setitem(seq, slice(i, j), values)}

@item

Slice Deletion

@tab

@code{del seq[i:j]}

@tab

@code{delitem(seq, slice(i, j))}

@item

Slicing

@tab

@code{seq[i:j]}

@tab

@code{getitem(seq, slice(i, j))}

@item

String Formatting

@tab

@code{s % obj}

@tab

@code{mod(s, obj)}

@item

Subtraction

@tab

@code{a - b}

@tab

@code{sub(a, b)}

@item

Truth Test

@tab

@code{obj}

@tab

@code{truth(obj)}

@item

Ordering

@tab

@code{a < b}

@tab

@code{lt(a, b)}

@item

Ordering

@tab

@code{a <= b}

@tab

@code{le(a, b)}

@item

Equality

@tab

@code{a == b}

@tab

@code{eq(a, b)}

@item

Difference

@tab

@code{a != b}

@tab

@code{ne(a, b)}

@item

Ordering

@tab

@code{a >= b}

@tab

@code{ge(a, b)}

@item

Ordering

@tab

@code{a > b}

@tab

@code{gt(a, b)}

@end multitable


@node In-place Operators,,Mapping Operators to Functions,operator — Standard operators as functions
@anchor{library/operator in-place-operators}@anchor{2033}
@subsubsection In-place Operators


Many operations have an “in-place” version.  Listed below are functions
providing a more primitive access to in-place operators than the usual syntax
does; for example, the @ref{2034,,statement} @code{x += y} is equivalent to
@code{x = operator.iadd(x, y)}.  Another way to put it is to say that
@code{z = operator.iadd(x, y)} is equivalent to the compound statement
@code{z = x; z += y}.

In those examples, note that when an in-place method is called, the computation
and assignment are performed in two separate steps.  The in-place functions
listed below only do the first step, calling the in-place method.  The second
step, assignment, is not handled.

For immutable targets such as strings, numbers, and tuples, the updated
value is computed, but not assigned back to the input variable:

@example
>>> a = 'hello'
>>> iadd(a, ' world')
'hello world'
>>> a
'hello'
@end example

For mutable targets such as lists and dictionaries, the in-place method
will perform the update, so no subsequent assignment is necessary:

@example
>>> s = ['h', 'e', 'l', 'l', 'o']
>>> iadd(s, [' ', 'w', 'o', 'r', 'l', 'd'])
['h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd']
>>> s
['h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd']
@end example

@geindex iadd() (in module operator)
@geindex __iadd__() (in module operator)
@anchor{library/operator operator iadd}@anchor{2035}
@deffn {Function} operator.iadd (a, b)
@anchor{library/operator operator __iadd__}@anchor{2036}
@deffnx {Function} operator.__iadd__ (a, b)

@code{a = iadd(a, b)} is equivalent to @code{a += b}.
@end deffn

@geindex iand() (in module operator)
@geindex __iand__() (in module operator)
@anchor{library/operator operator iand}@anchor{2037}
@deffn {Function} operator.iand (a, b)
@anchor{library/operator operator __iand__}@anchor{2038}
@deffnx {Function} operator.__iand__ (a, b)

@code{a = iand(a, b)} is equivalent to @code{a &= b}.
@end deffn

@geindex iconcat() (in module operator)
@geindex __iconcat__() (in module operator)
@anchor{library/operator operator iconcat}@anchor{2039}
@deffn {Function} operator.iconcat (a, b)
@anchor{library/operator operator __iconcat__}@anchor{203a}
@deffnx {Function} operator.__iconcat__ (a, b)

@code{a = iconcat(a, b)} is equivalent to @code{a += b} for @emph{a} and @emph{b} sequences.
@end deffn

@geindex ifloordiv() (in module operator)
@geindex __ifloordiv__() (in module operator)
@anchor{library/operator operator ifloordiv}@anchor{203b}
@deffn {Function} operator.ifloordiv (a, b)
@anchor{library/operator operator __ifloordiv__}@anchor{203c}
@deffnx {Function} operator.__ifloordiv__ (a, b)

@code{a = ifloordiv(a, b)} is equivalent to @code{a //= b}.
@end deffn

@geindex ilshift() (in module operator)
@geindex __ilshift__() (in module operator)
@anchor{library/operator operator ilshift}@anchor{203d}
@deffn {Function} operator.ilshift (a, b)
@anchor{library/operator operator __ilshift__}@anchor{203e}
@deffnx {Function} operator.__ilshift__ (a, b)

@code{a = ilshift(a, b)} is equivalent to @code{a <<= b}.
@end deffn

@geindex imod() (in module operator)
@geindex __imod__() (in module operator)
@anchor{library/operator operator imod}@anchor{203f}
@deffn {Function} operator.imod (a, b)
@anchor{library/operator operator __imod__}@anchor{2040}
@deffnx {Function} operator.__imod__ (a, b)

@code{a = imod(a, b)} is equivalent to @code{a %= b}.
@end deffn

@geindex imul() (in module operator)
@geindex __imul__() (in module operator)
@anchor{library/operator operator imul}@anchor{2041}
@deffn {Function} operator.imul (a, b)
@anchor{library/operator operator __imul__}@anchor{2042}
@deffnx {Function} operator.__imul__ (a, b)

@code{a = imul(a, b)} is equivalent to @code{a *= b}.
@end deffn

@geindex imatmul() (in module operator)
@geindex __imatmul__() (in module operator)
@anchor{library/operator operator imatmul}@anchor{99d}
@deffn {Function} operator.imatmul (a, b)
@anchor{library/operator operator __imatmul__}@anchor{2043}
@deffnx {Function} operator.__imatmul__ (a, b)

@code{a = imatmul(a, b)} is equivalent to @code{a @@= b}.


New in version 3.5.

@end deffn

@geindex ior() (in module operator)
@geindex __ior__() (in module operator)
@anchor{library/operator operator ior}@anchor{2044}
@deffn {Function} operator.ior (a, b)
@anchor{library/operator operator __ior__}@anchor{2045}
@deffnx {Function} operator.__ior__ (a, b)

@code{a = ior(a, b)} is equivalent to @code{a |= b}.
@end deffn

@geindex ipow() (in module operator)
@geindex __ipow__() (in module operator)
@anchor{library/operator operator ipow}@anchor{2046}
@deffn {Function} operator.ipow (a, b)
@anchor{library/operator operator __ipow__}@anchor{2047}
@deffnx {Function} operator.__ipow__ (a, b)

@code{a = ipow(a, b)} is equivalent to @code{a **= b}.
@end deffn

@geindex irshift() (in module operator)
@geindex __irshift__() (in module operator)
@anchor{library/operator operator irshift}@anchor{2048}
@deffn {Function} operator.irshift (a, b)
@anchor{library/operator operator __irshift__}@anchor{2049}
@deffnx {Function} operator.__irshift__ (a, b)

@code{a = irshift(a, b)} is equivalent to @code{a >>= b}.
@end deffn

@geindex isub() (in module operator)
@geindex __isub__() (in module operator)
@anchor{library/operator operator isub}@anchor{204a}
@deffn {Function} operator.isub (a, b)
@anchor{library/operator operator __isub__}@anchor{204b}
@deffnx {Function} operator.__isub__ (a, b)

@code{a = isub(a, b)} is equivalent to @code{a -= b}.
@end deffn

@geindex itruediv() (in module operator)
@geindex __itruediv__() (in module operator)
@anchor{library/operator operator itruediv}@anchor{204c}
@deffn {Function} operator.itruediv (a, b)
@anchor{library/operator operator __itruediv__}@anchor{204d}
@deffnx {Function} operator.__itruediv__ (a, b)

@code{a = itruediv(a, b)} is equivalent to @code{a /= b}.
@end deffn

@geindex ixor() (in module operator)
@geindex __ixor__() (in module operator)
@anchor{library/operator operator ixor}@anchor{204e}
@deffn {Function} operator.ixor (a, b)
@anchor{library/operator operator __ixor__}@anchor{204f}
@deffnx {Function} operator.__ixor__ (a, b)

@code{a = ixor(a, b)} is equivalent to @code{a ^= b}.
@end deffn

@node File and Directory Access,Data Persistence,Functional Programming Modules,The Python Standard Library
@anchor{library/filesys doc}@anchor{2050}@anchor{library/filesys file-and-directory-access}@anchor{2051}@anchor{library/filesys filesys}@anchor{2052}
@section File and Directory Access


The modules described in this chapter deal with disk files and directories.  For
example, there are modules for reading the properties of files, manipulating
paths in a portable way, and creating temporary files.  The full list of modules
in this chapter is:

@menu
* pathlib — Object-oriented filesystem paths:: 
* os.path — Common pathname manipulations: os path — Common pathname manipulations. 
* fileinput — Iterate over lines from multiple input streams:: 
* stat — Interpreting stat() results: stat — Interpreting stat results. 
* filecmp — File and Directory Comparisons:: 
* tempfile — Generate temporary files and directories:: 
* glob — Unix style pathname pattern expansion:: 
* fnmatch — Unix filename pattern matching:: 
* linecache — Random access to text lines:: 
* shutil — High-level file operations:: 

@end menu

@node pathlib — Object-oriented filesystem paths,os path — Common pathname manipulations,,File and Directory Access
@anchor{library/pathlib doc}@anchor{2053}@anchor{library/pathlib module-pathlib}@anchor{c3}@anchor{library/pathlib pathlib-object-oriented-filesystem-paths}@anchor{2054}
@subsection @code{pathlib} — Object-oriented filesystem paths


@geindex module; pathlib


New in version 3.4.


@strong{Source code:} Lib/pathlib.py@footnote{https://github.com/python/cpython/tree/main/Lib/pathlib.py}

@geindex path; operations


__________________________________________________________________


This module offers classes representing filesystem paths with semantics
appropriate for different operating systems.  Path classes are divided
between @ref{2055,,pure paths}, which provide purely computational
operations without I/O, and @ref{2056,,concrete paths}, which
inherit from pure paths but also provide I/O operations.

@image{python-figures/pathlib-inheritance,,,,png}

If you’ve never used this module before or just aren’t sure which class is
right for your task, @ref{522,,Path} is most likely what you need. It instantiates
a @ref{2056,,concrete path} for the platform the code is running on.

Pure paths are useful in some special cases; for example:


@enumerate 

@item 
If you want to manipulate Windows paths on a Unix machine (or vice versa).
You cannot instantiate a @ref{2057,,WindowsPath} when running on Unix, but you
can instantiate @ref{2058,,PureWindowsPath}.

@item 
You want to make sure that your code only manipulates paths without actually
accessing the OS. In this case, instantiating one of the pure classes may be
useful since those simply don’t have any OS-accessing operations.
@end enumerate


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 428
PEP 428@footnote{https://www.python.org/dev/peps/pep-0428}: The pathlib module – object-oriented filesystem paths.



@subsubheading See also


For low-level path manipulation on strings, you can also use the
@ref{c1,,os.path} module.


@menu
* Basic use:: 
* Pure paths:: 
* Concrete paths:: 
* Correspondence to tools in the os module:: 

@end menu

@node Basic use,Pure paths,,pathlib — Object-oriented filesystem paths
@anchor{library/pathlib basic-use}@anchor{2059}
@subsubsection Basic use


Importing the main class:

@example
>>> from pathlib import Path
@end example

Listing subdirectories:

@example
>>> p = Path('.')
>>> [x for x in p.iterdir() if x.is_dir()]
[PosixPath('.hg'), PosixPath('docs'), PosixPath('dist'),
 PosixPath('__pycache__'), PosixPath('build')]
@end example

Listing Python source files in this directory tree:

@example
>>> list(p.glob('**/*.py'))
[PosixPath('test_pathlib.py'), PosixPath('setup.py'),
 PosixPath('pathlib.py'), PosixPath('docs/conf.py'),
 PosixPath('build/lib/pathlib.py')]
@end example

Navigating inside a directory tree:

@example
>>> p = Path('/etc')
>>> q = p / 'init.d' / 'reboot'
>>> q
PosixPath('/etc/init.d/reboot')
>>> q.resolve()
PosixPath('/etc/rc.d/init.d/halt')
@end example

Querying path properties:

@example
>>> q.exists()
True
>>> q.is_dir()
False
@end example

Opening a file:

@example
>>> with q.open() as f: f.readline()
...
'#!/bin/bash\n'
@end example

@node Pure paths,Concrete paths,Basic use,pathlib — Object-oriented filesystem paths
@anchor{library/pathlib id1}@anchor{205a}@anchor{library/pathlib pure-paths}@anchor{2055}
@subsubsection Pure paths


Pure path objects provide path-handling operations which don’t actually
access a filesystem.  There are three ways to access these classes, which
we also call @emph{flavours}:

@geindex PurePath (class in pathlib)
@anchor{library/pathlib pathlib PurePath}@anchor{1c4}
@deffn {Class} pathlib.PurePath (*pathsegments)

A generic class that represents the system’s path flavour (instantiating
it creates either a @ref{205b,,PurePosixPath} or a @ref{2058,,PureWindowsPath}):

@example
>>> PurePath('setup.py')      # Running on a Unix machine
PurePosixPath('setup.py')
@end example

Each element of @emph{pathsegments} can be either a string representing a
path segment, an object implementing the @ref{795,,os.PathLike} interface
which returns a string, or another path object:

@example
>>> PurePath('foo', 'some/path', 'bar')
PurePosixPath('foo/some/path/bar')
>>> PurePath(Path('foo'), Path('bar'))
PurePosixPath('foo/bar')
@end example

When @emph{pathsegments} is empty, the current directory is assumed:

@example
>>> PurePath()
PurePosixPath('.')
@end example

When several absolute paths are given, the last is taken as an anchor
(mimicking @ref{1260,,os.path.join()}’s behaviour):

@example
>>> PurePath('/etc', '/usr', 'lib64')
PurePosixPath('/usr/lib64')
>>> PureWindowsPath('c:/Windows', 'd:bar')
PureWindowsPath('d:bar')
@end example

However, in a Windows path, changing the local root doesn’t discard the
previous drive setting:

@example
>>> PureWindowsPath('c:/Windows', '/Program Files')
PureWindowsPath('c:/Program Files')
@end example

Spurious slashes and single dots are collapsed, but double dots (@code{'..'})
are not, since this would change the meaning of a path in the face of
symbolic links:

@example
>>> PurePath('foo//bar')
PurePosixPath('foo/bar')
>>> PurePath('foo/./bar')
PurePosixPath('foo/bar')
>>> PurePath('foo/../bar')
PurePosixPath('foo/../bar')
@end example

(a naïve approach would make @code{PurePosixPath('foo/../bar')} equivalent
to @code{PurePosixPath('bar')}, which is wrong if @code{foo} is a symbolic link
to another directory)

Pure path objects implement the @ref{795,,os.PathLike} interface, allowing them
to be used anywhere the interface is accepted.


Changed in version 3.6: Added support for the @ref{795,,os.PathLike} interface.

@end deffn

@geindex PurePosixPath (class in pathlib)
@anchor{library/pathlib pathlib PurePosixPath}@anchor{205b}
@deffn {Class} pathlib.PurePosixPath (*pathsegments)

A subclass of @ref{1c4,,PurePath}, this path flavour represents non-Windows
filesystem paths:

@example
>>> PurePosixPath('/etc')
PurePosixPath('/etc')
@end example

@emph{pathsegments} is specified similarly to @ref{1c4,,PurePath}.
@end deffn

@geindex PureWindowsPath (class in pathlib)
@anchor{library/pathlib pathlib PureWindowsPath}@anchor{2058}
@deffn {Class} pathlib.PureWindowsPath (*pathsegments)

A subclass of @ref{1c4,,PurePath}, this path flavour represents Windows
filesystem paths:

@example
>>> PureWindowsPath('c:/Program Files/')
PureWindowsPath('c:/Program Files')
@end example

@emph{pathsegments} is specified similarly to @ref{1c4,,PurePath}.
@end deffn

Regardless of the system you’re running on, you can instantiate all of
these classes, since they don’t provide any operation that does system calls.

@menu
* General properties:: 
* Operators: Operators<2>. 
* Accessing individual parts:: 
* Methods and properties:: 

@end menu

@node General properties,Operators<2>,,Pure paths
@anchor{library/pathlib general-properties}@anchor{205c}
@subsubsection General properties


Paths are immutable and hashable.  Paths of a same flavour are comparable
and orderable.  These properties respect the flavour’s case-folding
semantics:

@example
>>> PurePosixPath('foo') == PurePosixPath('FOO')
False
>>> PureWindowsPath('foo') == PureWindowsPath('FOO')
True
>>> PureWindowsPath('FOO') in @{ PureWindowsPath('foo') @}
True
>>> PureWindowsPath('C:') < PureWindowsPath('d:')
True
@end example

Paths of a different flavour compare unequal and cannot be ordered:

@example
>>> PureWindowsPath('foo') == PurePosixPath('foo')
False
>>> PureWindowsPath('foo') < PurePosixPath('foo')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'PureWindowsPath' and 'PurePosixPath'
@end example

@node Operators<2>,Accessing individual parts,General properties,Pure paths
@anchor{library/pathlib operators}@anchor{205d}
@subsubsection Operators


The slash operator helps create child paths, similarly to @ref{1260,,os.path.join()}:

@example
>>> p = PurePath('/etc')
>>> p
PurePosixPath('/etc')
>>> p / 'init.d' / 'apache2'
PurePosixPath('/etc/init.d/apache2')
>>> q = PurePath('bin')
>>> '/usr' / q
PurePosixPath('/usr/bin')
@end example

A path object can be used anywhere an object implementing @ref{795,,os.PathLike}
is accepted:

@example
>>> import os
>>> p = PurePath('/etc')
>>> os.fspath(p)
'/etc'
@end example

The string representation of a path is the raw filesystem path itself
(in native form, e.g. with backslashes under Windows), which you can
pass to any function taking a file path as a string:

@example
>>> p = PurePath('/etc')
>>> str(p)
'/etc'
>>> p = PureWindowsPath('c:/Program Files')
>>> str(p)
'c:\\Program Files'
@end example

Similarly, calling @ref{2f0,,bytes} on a path gives the raw filesystem path as a
bytes object, as encoded by @ref{799,,os.fsencode()}:

@example
>>> bytes(p)
b'/etc'
@end example

@cartouche
@quotation Note 
Calling @ref{2f0,,bytes} is only recommended under Unix.  Under Windows,
the unicode form is the canonical representation of filesystem paths.
@end quotation
@end cartouche

@node Accessing individual parts,Methods and properties,Operators<2>,Pure paths
@anchor{library/pathlib accessing-individual-parts}@anchor{205e}
@subsubsection Accessing individual parts


To access the individual “parts” (components) of a path, use the following
property:

@geindex PurePath.parts (in module pathlib)
@anchor{library/pathlib pathlib PurePath parts}@anchor{205f}
@deffn {Data} PurePath.parts

A tuple giving access to the path’s various components:

@example
>>> p = PurePath('/usr/bin/python3')
>>> p.parts
('/', 'usr', 'bin', 'python3')

>>> p = PureWindowsPath('c:/Program Files/PSF')
>>> p.parts
('c:\\', 'Program Files', 'PSF')
@end example

(note how the drive and local root are regrouped in a single part)
@end deffn

@node Methods and properties,,Accessing individual parts,Pure paths
@anchor{library/pathlib methods-and-properties}@anchor{2060}
@subsubsection Methods and properties


@c from pathlib import PurePath, PurePosixPath, PureWindowsPath

Pure paths provide the following methods and properties:

@geindex PurePath.drive (in module pathlib)
@anchor{library/pathlib pathlib PurePath drive}@anchor{2061}
@deffn {Data} PurePath.drive

A string representing the drive letter or name, if any:

@example
>>> PureWindowsPath('c:/Program Files/').drive
'c:'
>>> PureWindowsPath('/Program Files/').drive
'@w{'}
>>> PurePosixPath('/etc').drive
'@w{'}
@end example

UNC shares are also considered drives:

@example
>>> PureWindowsPath('//host/share/foo.txt').drive
'\\\\host\\share'
@end example
@end deffn

@geindex PurePath.root (in module pathlib)
@anchor{library/pathlib pathlib PurePath root}@anchor{2062}
@deffn {Data} PurePath.root

A string representing the (local or global) root, if any:

@example
>>> PureWindowsPath('c:/Program Files/').root
'\\'
>>> PureWindowsPath('c:Program Files/').root
'@w{'}
>>> PurePosixPath('/etc').root
'/'
@end example

UNC shares always have a root:

@example
>>> PureWindowsPath('//host/share').root
'\\'
@end example
@end deffn

@geindex PurePath.anchor (in module pathlib)
@anchor{library/pathlib pathlib PurePath anchor}@anchor{2063}
@deffn {Data} PurePath.anchor

The concatenation of the drive and root:

@example
>>> PureWindowsPath('c:/Program Files/').anchor
'c:\\'
>>> PureWindowsPath('c:Program Files/').anchor
'c:'
>>> PurePosixPath('/etc').anchor
'/'
>>> PureWindowsPath('//host/share').anchor
'\\\\host\\share\\'
@end example
@end deffn

@geindex PurePath.parents (in module pathlib)
@anchor{library/pathlib pathlib PurePath parents}@anchor{2a3}
@deffn {Data} PurePath.parents

An immutable sequence providing access to the logical ancestors of
the path:

@example
>>> p = PureWindowsPath('c:/foo/bar/setup.py')
>>> p.parents[0]
PureWindowsPath('c:/foo/bar')
>>> p.parents[1]
PureWindowsPath('c:/foo')
>>> p.parents[2]
PureWindowsPath('c:/')
@end example


Changed in version 3.10: The parents sequence now supports @ref{1a5d,,slices} and negative index values.

@end deffn

@geindex PurePath.parent (in module pathlib)
@anchor{library/pathlib pathlib PurePath parent}@anchor{2064}
@deffn {Data} PurePath.parent

The logical parent of the path:

@example
>>> p = PurePosixPath('/a/b/c/d')
>>> p.parent
PurePosixPath('/a/b/c')
@end example

You cannot go past an anchor, or empty path:

@example
>>> p = PurePosixPath('/')
>>> p.parent
PurePosixPath('/')
>>> p = PurePosixPath('.')
>>> p.parent
PurePosixPath('.')
@end example

@cartouche
@quotation Note 
This is a purely lexical operation, hence the following behaviour:

@example
>>> p = PurePosixPath('foo/..')
>>> p.parent
PurePosixPath('foo')
@end example

If you want to walk an arbitrary filesystem path upwards, it is
recommended to first call @ref{2065,,Path.resolve()} so as to resolve
symlinks and eliminate @cite{“..”} components.
@end quotation
@end cartouche
@end deffn

@geindex PurePath.name (in module pathlib)
@anchor{library/pathlib pathlib PurePath name}@anchor{2066}
@deffn {Data} PurePath.name

A string representing the final path component, excluding the drive and
root, if any:

@example
>>> PurePosixPath('my/library/setup.py').name
'setup.py'
@end example

UNC drive names are not considered:

@example
>>> PureWindowsPath('//some/share/setup.py').name
'setup.py'
>>> PureWindowsPath('//some/share').name
'@w{'}
@end example
@end deffn

@geindex PurePath.suffix (in module pathlib)
@anchor{library/pathlib pathlib PurePath suffix}@anchor{2067}
@deffn {Data} PurePath.suffix

The file extension of the final component, if any:

@example
>>> PurePosixPath('my/library/setup.py').suffix
'.py'
>>> PurePosixPath('my/library.tar.gz').suffix
'.gz'
>>> PurePosixPath('my/library').suffix
'@w{'}
@end example
@end deffn

@geindex PurePath.suffixes (in module pathlib)
@anchor{library/pathlib pathlib PurePath suffixes}@anchor{2068}
@deffn {Data} PurePath.suffixes

A list of the path’s file extensions:

@example
>>> PurePosixPath('my/library.tar.gar').suffixes
['.tar', '.gar']
>>> PurePosixPath('my/library.tar.gz').suffixes
['.tar', '.gz']
>>> PurePosixPath('my/library').suffixes
[]
@end example
@end deffn

@geindex PurePath.stem (in module pathlib)
@anchor{library/pathlib pathlib PurePath stem}@anchor{2069}
@deffn {Data} PurePath.stem

The final path component, without its suffix:

@example
>>> PurePosixPath('my/library.tar.gz').stem
'library.tar'
>>> PurePosixPath('my/library.tar').stem
'library'
>>> PurePosixPath('my/library').stem
'library'
@end example
@end deffn

@geindex as_posix() (pathlib.PurePath method)
@anchor{library/pathlib pathlib PurePath as_posix}@anchor{206a}
@deffn {Method} PurePath.as_posix ()

Return a string representation of the path with forward slashes (@code{/}):

@example
>>> p = PureWindowsPath('c:\\windows')
>>> str(p)
'c:\\windows'
>>> p.as_posix()
'c:/windows'
@end example
@end deffn

@geindex as_uri() (pathlib.PurePath method)
@anchor{library/pathlib pathlib PurePath as_uri}@anchor{206b}
@deffn {Method} PurePath.as_uri ()

Represent the path as a @code{file} URI.  @ref{17c,,ValueError} is raised if
the path isn’t absolute.

@example
>>> p = PurePosixPath('/etc/passwd')
>>> p.as_uri()
'file:///etc/passwd'
>>> p = PureWindowsPath('c:/Windows')
>>> p.as_uri()
'file:///c:/Windows'
@end example
@end deffn

@geindex is_absolute() (pathlib.PurePath method)
@anchor{library/pathlib pathlib PurePath is_absolute}@anchor{206c}
@deffn {Method} PurePath.is_absolute ()

Return whether the path is absolute or not.  A path is considered absolute
if it has both a root and (if the flavour allows) a drive:

@example
>>> PurePosixPath('/a/b').is_absolute()
True
>>> PurePosixPath('a/b').is_absolute()
False

>>> PureWindowsPath('c:/a/b').is_absolute()
True
>>> PureWindowsPath('/a/b').is_absolute()
False
>>> PureWindowsPath('c:').is_absolute()
False
>>> PureWindowsPath('//some/share').is_absolute()
True
@end example
@end deffn

@geindex is_relative_to() (pathlib.PurePath method)
@anchor{library/pathlib pathlib PurePath is_relative_to}@anchor{206d}
@deffn {Method} PurePath.is_relative_to (*other)

Return whether or not this path is relative to the @emph{other} path.

@example
>>> p = PurePath('/etc/passwd')
>>> p.is_relative_to('/etc')
True
>>> p.is_relative_to('/usr')
False
@end example


New in version 3.9.

@end deffn

@geindex is_reserved() (pathlib.PurePath method)
@anchor{library/pathlib pathlib PurePath is_reserved}@anchor{206e}
@deffn {Method} PurePath.is_reserved ()

With @ref{2058,,PureWindowsPath}, return @code{True} if the path is considered
reserved under Windows, @code{False} otherwise.  With @ref{205b,,PurePosixPath},
@code{False} is always returned.

@example
>>> PureWindowsPath('nul').is_reserved()
True
>>> PurePosixPath('nul').is_reserved()
False
@end example

File system calls on reserved paths can fail mysteriously or have
unintended effects.
@end deffn

@geindex joinpath() (pathlib.PurePath method)
@anchor{library/pathlib pathlib PurePath joinpath}@anchor{206f}
@deffn {Method} PurePath.joinpath (*other)

Calling this method is equivalent to combining the path with each of
the @emph{other} arguments in turn:

@example
>>> PurePosixPath('/etc').joinpath('passwd')
PurePosixPath('/etc/passwd')
>>> PurePosixPath('/etc').joinpath(PurePosixPath('passwd'))
PurePosixPath('/etc/passwd')
>>> PurePosixPath('/etc').joinpath('init.d', 'apache2')
PurePosixPath('/etc/init.d/apache2')
>>> PureWindowsPath('c:').joinpath('/Program Files')
PureWindowsPath('c:/Program Files')
@end example
@end deffn

@geindex match() (pathlib.PurePath method)
@anchor{library/pathlib pathlib PurePath match}@anchor{2070}
@deffn {Method} PurePath.match (pattern)

Match this path against the provided glob-style pattern.  Return @code{True}
if matching is successful, @code{False} otherwise.

If @emph{pattern} is relative, the path can be either relative or absolute,
and matching is done from the right:

@example
>>> PurePath('a/b.py').match('*.py')
True
>>> PurePath('/a/b/c.py').match('b/*.py')
True
>>> PurePath('/a/b/c.py').match('a/*.py')
False
@end example

If @emph{pattern} is absolute, the path must be absolute, and the whole path
must match:

@example
>>> PurePath('/a.py').match('/*.py')
True
>>> PurePath('a/b.py').match('/*.py')
False
@end example

As with other methods, case-sensitivity follows platform defaults:

@example
>>> PurePosixPath('b.py').match('*.PY')
False
>>> PureWindowsPath('b.py').match('*.PY')
True
@end example
@end deffn

@geindex relative_to() (pathlib.PurePath method)
@anchor{library/pathlib pathlib PurePath relative_to}@anchor{2071}
@deffn {Method} PurePath.relative_to (*other)

Compute a version of this path relative to the path represented by
@emph{other}.  If it’s impossible, ValueError is raised:

@example
>>> p = PurePosixPath('/etc/passwd')
>>> p.relative_to('/')
PurePosixPath('etc/passwd')
>>> p.relative_to('/etc')
PurePosixPath('passwd')
>>> p.relative_to('/usr')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pathlib.py", line 694, in relative_to
    .format(str(self), str(formatted)))
ValueError: '/etc/passwd' is not in the subpath of '/usr' OR one path is relative and the other absolute.
@end example

NOTE: This function is part of @ref{1c4,,PurePath} and works with strings. It does not check or access the underlying file structure.
@end deffn

@geindex with_name() (pathlib.PurePath method)
@anchor{library/pathlib pathlib PurePath with_name}@anchor{2072}
@deffn {Method} PurePath.with_name (name)

Return a new path with the @ref{2066,,name} changed.  If the original path
doesn’t have a name, ValueError is raised:

@example
>>> p = PureWindowsPath('c:/Downloads/pathlib.tar.gz')
>>> p.with_name('setup.py')
PureWindowsPath('c:/Downloads/setup.py')
>>> p = PureWindowsPath('c:/')
>>> p.with_name('setup.py')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/antoine/cpython/default/Lib/pathlib.py", line 751, in with_name
    raise ValueError("%r has an empty name" % (self,))
ValueError: PureWindowsPath('c:/') has an empty name
@end example
@end deffn

@geindex with_stem() (pathlib.PurePath method)
@anchor{library/pathlib pathlib PurePath with_stem}@anchor{2073}
@deffn {Method} PurePath.with_stem (stem)

Return a new path with the @ref{2069,,stem} changed.  If the original path
doesn’t have a name, ValueError is raised:

@example
>>> p = PureWindowsPath('c:/Downloads/draft.txt')
>>> p.with_stem('final')
PureWindowsPath('c:/Downloads/final.txt')
>>> p = PureWindowsPath('c:/Downloads/pathlib.tar.gz')
>>> p.with_stem('lib')
PureWindowsPath('c:/Downloads/lib.gz')
>>> p = PureWindowsPath('c:/')
>>> p.with_stem('@w{'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/antoine/cpython/default/Lib/pathlib.py", line 861, in with_stem
    return self.with_name(stem + self.suffix)
  File "/home/antoine/cpython/default/Lib/pathlib.py", line 851, in with_name
    raise ValueError("%r has an empty name" % (self,))
ValueError: PureWindowsPath('c:/') has an empty name
@end example


New in version 3.9.

@end deffn

@geindex with_suffix() (pathlib.PurePath method)
@anchor{library/pathlib pathlib PurePath with_suffix}@anchor{12f6}
@deffn {Method} PurePath.with_suffix (suffix)

Return a new path with the @ref{2067,,suffix} changed.  If the original path
doesn’t have a suffix, the new @emph{suffix} is appended instead.  If the
@emph{suffix} is an empty string, the original suffix is removed:

@example
>>> p = PureWindowsPath('c:/Downloads/pathlib.tar.gz')
>>> p.with_suffix('.bz2')
PureWindowsPath('c:/Downloads/pathlib.tar.bz2')
>>> p = PureWindowsPath('README')
>>> p.with_suffix('.txt')
PureWindowsPath('README.txt')
>>> p = PureWindowsPath('README.txt')
>>> p.with_suffix('@w{'})
PureWindowsPath('README')
@end example
@end deffn

@node Concrete paths,Correspondence to tools in the os module,Pure paths,pathlib — Object-oriented filesystem paths
@anchor{library/pathlib concrete-paths}@anchor{2056}@anchor{library/pathlib id2}@anchor{2074}
@subsubsection Concrete paths


Concrete paths are subclasses of the pure path classes.  In addition to
operations provided by the latter, they also provide methods to do system
calls on path objects.  There are three ways to instantiate concrete paths:

@geindex Path (class in pathlib)
@anchor{library/pathlib pathlib Path}@anchor{522}
@deffn {Class} pathlib.Path (*pathsegments)

A subclass of @ref{1c4,,PurePath}, this class represents concrete paths of
the system’s path flavour (instantiating it creates either a
@ref{2075,,PosixPath} or a @ref{2057,,WindowsPath}):

@example
>>> Path('setup.py')
PosixPath('setup.py')
@end example

@emph{pathsegments} is specified similarly to @ref{1c4,,PurePath}.
@end deffn

@geindex PosixPath (class in pathlib)
@anchor{library/pathlib pathlib PosixPath}@anchor{2075}
@deffn {Class} pathlib.PosixPath (*pathsegments)

A subclass of @ref{522,,Path} and @ref{205b,,PurePosixPath}, this class
represents concrete non-Windows filesystem paths:

@example
>>> PosixPath('/etc')
PosixPath('/etc')
@end example

@emph{pathsegments} is specified similarly to @ref{1c4,,PurePath}.
@end deffn

@geindex WindowsPath (class in pathlib)
@anchor{library/pathlib pathlib WindowsPath}@anchor{2057}
@deffn {Class} pathlib.WindowsPath (*pathsegments)

A subclass of @ref{522,,Path} and @ref{2058,,PureWindowsPath}, this class
represents concrete Windows filesystem paths:

@example
>>> WindowsPath('c:/Program Files/')
WindowsPath('c:/Program Files')
@end example

@emph{pathsegments} is specified similarly to @ref{1c4,,PurePath}.
@end deffn

You can only instantiate the class flavour that corresponds to your system
(allowing system calls on non-compatible path flavours could lead to
bugs or failures in your application):

@example
>>> import os
>>> os.name
'posix'
>>> Path('setup.py')
PosixPath('setup.py')
>>> PosixPath('setup.py')
PosixPath('setup.py')
>>> WindowsPath('setup.py')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pathlib.py", line 798, in __new__
    % (cls.__name__,))
NotImplementedError: cannot instantiate 'WindowsPath' on your system
@end example

@menu
* Methods: Methods<2>. 

@end menu

@node Methods<2>,,,Concrete paths
@anchor{library/pathlib methods}@anchor{2076}
@subsubsection Methods


Concrete paths provide the following methods in addition to pure paths
methods.  Many of these methods can raise an @ref{2a1,,OSError} if a system
call fails (for example because the path doesn’t exist).


Changed in version 3.8: @ref{523,,exists()}, @ref{524,,is_dir()}, @ref{525,,is_file()},
@ref{526,,is_mount()}, @ref{527,,is_symlink()},
@ref{528,,is_block_device()}, @ref{529,,is_char_device()},
@ref{52a,,is_fifo()}, @ref{52b,,is_socket()} now return @code{False}
instead of raising an exception for paths that contain characters
unrepresentable at the OS level.


@geindex cwd() (pathlib.Path class method)
@anchor{library/pathlib pathlib Path cwd}@anchor{2077}
@deffn {Method} classmethod Path.cwd ()

Return a new path object representing the current directory (as returned
by @ref{12ac,,os.getcwd()}):

@example
>>> Path.cwd()
PosixPath('/home/antoine/pathlib')
@end example
@end deffn

@geindex home() (pathlib.Path class method)
@anchor{library/pathlib pathlib Path home}@anchor{9aa}
@deffn {Method} classmethod Path.home ()

Return a new path object representing the user’s home directory (as
returned by @ref{520,,os.path.expanduser()} with @code{~} construct). If the home
directory can’t be resolved, @ref{5af,,RuntimeError} is raised.

@example
>>> Path.home()
PosixPath('/home/antoine')
@end example


New in version 3.5.

@end deffn

@geindex stat() (pathlib.Path method)
@anchor{library/pathlib pathlib Path stat}@anchor{2a7}
@deffn {Method} Path.stat (*, follow_symlinks=True)

Return a @ref{134d,,os.stat_result} object containing information about this path, like @ref{516,,os.stat()}.
The result is looked up at each call to this method.

This method normally follows symlinks; to stat a symlink add the argument
@code{follow_symlinks=False}, or use @ref{2078,,lstat()}.

@example
>>> p = Path('setup.py')
>>> p.stat().st_size
956
>>> p.stat().st_mtime
1327883547.852554
@end example


Changed in version 3.10: The @emph{follow_symlinks} parameter was added.

@end deffn

@geindex chmod() (pathlib.Path method)
@anchor{library/pathlib pathlib Path chmod}@anchor{2a8}
@deffn {Method} Path.chmod (mode, *, follow_symlinks=True)

Change the file mode and permissions, like @ref{c80,,os.chmod()}.

This method normally follows symlinks. Some Unix flavours support changing
permissions on the symlink itself; on these platforms you may add the
argument @code{follow_symlinks=False}, or use @ref{2079,,lchmod()}.

@example
>>> p = Path('setup.py')
>>> p.stat().st_mode
33277
>>> p.chmod(0o444)
>>> p.stat().st_mode
33060
@end example


Changed in version 3.10: The @emph{follow_symlinks} parameter was added.

@end deffn

@geindex exists() (pathlib.Path method)
@anchor{library/pathlib pathlib Path exists}@anchor{523}
@deffn {Method} Path.exists ()

Whether the path points to an existing file or directory:

@example
>>> Path('.').exists()
True
>>> Path('setup.py').exists()
True
>>> Path('/etc').exists()
True
>>> Path('nonexistentfile').exists()
False
@end example

@cartouche
@quotation Note 
If the path points to a symlink, @ref{523,,exists()} returns whether the
symlink @emph{points to} an existing file or directory.
@end quotation
@end cartouche
@end deffn

@geindex expanduser() (pathlib.Path method)
@anchor{library/pathlib pathlib Path expanduser}@anchor{9a9}
@deffn {Method} Path.expanduser ()

Return a new path with expanded @code{~} and @code{~user} constructs,
as returned by @ref{520,,os.path.expanduser()}. If a home directory can’t be
resolved, @ref{5af,,RuntimeError} is raised.

@example
>>> p = PosixPath('~/films/Monty Python')
>>> p.expanduser()
PosixPath('/home/eric/films/Monty Python')
@end example


New in version 3.5.

@end deffn

@geindex glob() (pathlib.Path method)
@anchor{library/pathlib pathlib Path glob}@anchor{1228}
@deffn {Method} Path.glob (pattern)

Glob the given relative @emph{pattern} in the directory represented by this path,
yielding all matching files (of any kind):

@example
>>> sorted(Path('.').glob('*.py'))
[PosixPath('pathlib.py'), PosixPath('setup.py'), PosixPath('test_pathlib.py')]
>>> sorted(Path('.').glob('*/*.py'))
[PosixPath('docs/conf.py')]
@end example

Patterns are the same as for @ref{7c,,fnmatch}, with the addition of “@code{**}”
which means “this directory and all subdirectories, recursively”.  In other
words, it enables recursive globbing:

@example
>>> sorted(Path('.').glob('**/*.py'))
[PosixPath('build/lib/pathlib.py'),
 PosixPath('docs/conf.py'),
 PosixPath('pathlib.py'),
 PosixPath('setup.py'),
 PosixPath('test_pathlib.py')]
@end example

@cartouche
@quotation Note 
Using the “@code{**}” pattern in large directory trees may consume
an inordinate amount of time.
@end quotation
@end cartouche

Raises an @ref{1130,,auditing event} @code{pathlib.Path.glob} with arguments @code{self}, @code{pattern}.
@end deffn

@geindex group() (pathlib.Path method)
@anchor{library/pathlib pathlib Path group}@anchor{207a}
@deffn {Method} Path.group ()

Return the name of the group owning the file.  @ref{5bb,,KeyError} is raised
if the file’s gid isn’t found in the system database.
@end deffn

@geindex is_dir() (pathlib.Path method)
@anchor{library/pathlib pathlib Path is_dir}@anchor{524}
@deffn {Method} Path.is_dir ()

Return @code{True} if the path points to a directory (or a symbolic link
pointing to a directory), @code{False} if it points to another kind of file.

@code{False} is also returned if the path doesn’t exist or is a broken symlink;
other errors (such as permission errors) are propagated.
@end deffn

@geindex is_file() (pathlib.Path method)
@anchor{library/pathlib pathlib Path is_file}@anchor{525}
@deffn {Method} Path.is_file ()

Return @code{True} if the path points to a regular file (or a symbolic link
pointing to a regular file), @code{False} if it points to another kind of file.

@code{False} is also returned if the path doesn’t exist or is a broken symlink;
other errors (such as permission errors) are propagated.
@end deffn

@geindex is_mount() (pathlib.Path method)
@anchor{library/pathlib pathlib Path is_mount}@anchor{526}
@deffn {Method} Path.is_mount ()

Return @code{True} if the path is a @emph{mount point}: a point in a
file system where a different file system has been mounted.  On POSIX, the
function checks whether @emph{path}’s parent, @code{path/..}, is on a different
device than @emph{path}, or whether @code{path/..} and @emph{path} point to the same
i-node on the same device — this should detect mount points for all Unix
and POSIX variants.  Not implemented on Windows.


New in version 3.7.

@end deffn

@geindex is_symlink() (pathlib.Path method)
@anchor{library/pathlib pathlib Path is_symlink}@anchor{527}
@deffn {Method} Path.is_symlink ()

Return @code{True} if the path points to a symbolic link, @code{False} otherwise.

@code{False} is also returned if the path doesn’t exist; other errors (such
as permission errors) are propagated.
@end deffn

@geindex is_socket() (pathlib.Path method)
@anchor{library/pathlib pathlib Path is_socket}@anchor{52b}
@deffn {Method} Path.is_socket ()

Return @code{True} if the path points to a Unix socket (or a symbolic link
pointing to a Unix socket), @code{False} if it points to another kind of file.

@code{False} is also returned if the path doesn’t exist or is a broken symlink;
other errors (such as permission errors) are propagated.
@end deffn

@geindex is_fifo() (pathlib.Path method)
@anchor{library/pathlib pathlib Path is_fifo}@anchor{52a}
@deffn {Method} Path.is_fifo ()

Return @code{True} if the path points to a FIFO (or a symbolic link
pointing to a FIFO), @code{False} if it points to another kind of file.

@code{False} is also returned if the path doesn’t exist or is a broken symlink;
other errors (such as permission errors) are propagated.
@end deffn

@geindex is_block_device() (pathlib.Path method)
@anchor{library/pathlib pathlib Path is_block_device}@anchor{528}
@deffn {Method} Path.is_block_device ()

Return @code{True} if the path points to a block device (or a symbolic link
pointing to a block device), @code{False} if it points to another kind of file.

@code{False} is also returned if the path doesn’t exist or is a broken symlink;
other errors (such as permission errors) are propagated.
@end deffn

@geindex is_char_device() (pathlib.Path method)
@anchor{library/pathlib pathlib Path is_char_device}@anchor{529}
@deffn {Method} Path.is_char_device ()

Return @code{True} if the path points to a character device (or a symbolic link
pointing to a character device), @code{False} if it points to another kind of file.

@code{False} is also returned if the path doesn’t exist or is a broken symlink;
other errors (such as permission errors) are propagated.
@end deffn

@geindex iterdir() (pathlib.Path method)
@anchor{library/pathlib pathlib Path iterdir}@anchor{10a7}
@deffn {Method} Path.iterdir ()

When the path points to a directory, yield path objects of the directory
contents:

@example
>>> p = Path('docs')
>>> for child in p.iterdir(): child
...
PosixPath('docs/conf.py')
PosixPath('docs/_templates')
PosixPath('docs/make.bat')
PosixPath('docs/index.rst')
PosixPath('docs/_build')
PosixPath('docs/_static')
PosixPath('docs/Makefile')
@end example

The children are yielded in arbitrary order, and the special entries
@code{'.'} and @code{'..'} are not included.  If a file is removed from or added
to the directory after creating the iterator, whether an path object for
that file be included is unspecified.
@end deffn

@geindex lchmod() (pathlib.Path method)
@anchor{library/pathlib pathlib Path lchmod}@anchor{2079}
@deffn {Method} Path.lchmod (mode)

Like @ref{2a8,,Path.chmod()} but, if the path points to a symbolic link, the
symbolic link’s mode is changed rather than its target’s.
@end deffn

@geindex lstat() (pathlib.Path method)
@anchor{library/pathlib pathlib Path lstat}@anchor{2078}
@deffn {Method} Path.lstat ()

Like @ref{2a7,,Path.stat()} but, if the path points to a symbolic link, return
the symbolic link’s information rather than its target’s.
@end deffn

@geindex mkdir() (pathlib.Path method)
@anchor{library/pathlib pathlib Path mkdir}@anchor{9a8}
@deffn {Method} Path.mkdir (mode=511, parents=False, exist_ok=False)

Create a new directory at this given path.  If @emph{mode} is given, it is
combined with the process’ @code{umask} value to determine the file mode
and access flags.  If the path already exists, @ref{bb1,,FileExistsError}
is raised.

If @emph{parents} is true, any missing parents of this path are created
as needed; they are created with the default permissions without taking
@emph{mode} into account (mimicking the POSIX @code{mkdir -p} command).

If @emph{parents} is false (the default), a missing parent raises
@ref{a37,,FileNotFoundError}.

If @emph{exist_ok} is false (the default), @ref{bb1,,FileExistsError} is
raised if the target directory already exists.

If @emph{exist_ok} is true, @ref{bb1,,FileExistsError} exceptions will be
ignored (same behavior as the POSIX @code{mkdir -p} command), but only if the
last path component is not an existing non-directory file.


Changed in version 3.5: The @emph{exist_ok} parameter was added.

@end deffn

@geindex open() (pathlib.Path method)
@anchor{library/pathlib pathlib Path open}@anchor{207b}
@deffn {Method} Path.open (mode='r', buffering=- 1, encoding=None, errors=None, newline=None)

Open the file pointed to by the path, like the built-in @ref{1c8,,open()}
function does:

@example
>>> p = Path('setup.py')
>>> with p.open() as f:
...     f.readline()
...
'#!/usr/bin/env python3\n'
@end example
@end deffn

@geindex owner() (pathlib.Path method)
@anchor{library/pathlib pathlib Path owner}@anchor{207c}
@deffn {Method} Path.owner ()

Return the name of the user owning the file.  @ref{5bb,,KeyError} is raised
if the file’s uid isn’t found in the system database.
@end deffn

@geindex read_bytes() (pathlib.Path method)
@anchor{library/pathlib pathlib Path read_bytes}@anchor{9ae}
@deffn {Method} Path.read_bytes ()

Return the binary contents of the pointed-to file as a bytes object:

@example
>>> p = Path('my_binary_file')
>>> p.write_bytes(b'Binary file contents')
20
>>> p.read_bytes()
b'Binary file contents'
@end example


New in version 3.5.

@end deffn

@geindex read_text() (pathlib.Path method)
@anchor{library/pathlib pathlib Path read_text}@anchor{9ac}
@deffn {Method} Path.read_text (encoding=None, errors=None)

Return the decoded contents of the pointed-to file as a string:

@example
>>> p = Path('my_text_file')
>>> p.write_text('Text file contents')
18
>>> p.read_text()
'Text file contents'
@end example

The file is opened and then closed. The optional parameters have the same
meaning as in @ref{1c8,,open()}.


New in version 3.5.

@end deffn

@geindex readlink() (pathlib.Path method)
@anchor{library/pathlib pathlib Path readlink}@anchor{413}
@deffn {Method} Path.readlink ()

Return the path to which the symbolic link points (as returned by
@ref{414,,os.readlink()}):

@example
>>> p = Path('mylink')
>>> p.symlink_to('setup.py')
>>> p.readlink()
PosixPath('setup.py')
@end example


New in version 3.9.

@end deffn

@geindex rename() (pathlib.Path method)
@anchor{library/pathlib pathlib Path rename}@anchor{207d}
@deffn {Method} Path.rename (target)

Rename this file or directory to the given @emph{target}, and return a new Path
instance pointing to @emph{target}.  On Unix, if @emph{target} exists and is a file,
it will be replaced silently if the user has permission.  @emph{target} can be
either a string or another path object:

@example
>>> p = Path('foo')
>>> p.open('w').write('some text')
9
>>> target = Path('bar')
>>> p.rename(target)
PosixPath('bar')
>>> target.open().read()
'some text'
@end example

The target path may be absolute or relative. Relative paths are interpreted
relative to the current working directory, @emph{not} the directory of the Path
object.


Changed in version 3.8: Added return value, return the new Path instance.

@end deffn

@geindex replace() (pathlib.Path method)
@anchor{library/pathlib pathlib Path replace}@anchor{207e}
@deffn {Method} Path.replace (target)

Rename this file or directory to the given @emph{target}, and return a new Path
instance pointing to @emph{target}.  If @emph{target} points to an existing file or
directory, it will be unconditionally replaced.

The target path may be absolute or relative. Relative paths are interpreted
relative to the current working directory, @emph{not} the directory of the Path
object.


Changed in version 3.8: Added return value, return the new Path instance.

@end deffn

@geindex absolute() (pathlib.Path method)
@anchor{library/pathlib pathlib Path absolute}@anchor{10a8}
@deffn {Method} Path.absolute ()

Make the path absolute, without normalization or resolving symlinks.
Returns a new path object:

@example
>>> p = Path('tests')
>>> p
PosixPath('tests')
>>> p.absolute()
PosixPath('/home/antoine/pathlib/tests')
@end example
@end deffn

@geindex resolve() (pathlib.Path method)
@anchor{library/pathlib pathlib Path resolve}@anchor{2065}
@deffn {Method} Path.resolve (strict=False)

Make the path absolute, resolving any symlinks.  A new path object is
returned:

@example
>>> p = Path()
>>> p
PosixPath('.')
>>> p.resolve()
PosixPath('/home/antoine/pathlib')
@end example

“@code{..}” components are also eliminated (this is the only method to do so):

@example
>>> p = Path('docs/../setup.py')
>>> p.resolve()
PosixPath('/home/antoine/pathlib/setup.py')
@end example

If the path doesn’t exist and @emph{strict} is @code{True}, @ref{a37,,FileNotFoundError}
is raised.  If @emph{strict} is @code{False}, the path is resolved as far as possible
and any remainder is appended without checking whether it exists.  If an
infinite loop is encountered along the resolution path, @ref{5af,,RuntimeError}
is raised.


New in version 3.6: The @emph{strict} argument (pre-3.6 behavior is strict).

@end deffn

@geindex rglob() (pathlib.Path method)
@anchor{library/pathlib pathlib Path rglob}@anchor{1229}
@deffn {Method} Path.rglob (pattern)

This is like calling @ref{1228,,Path.glob()} with “@code{**/}” added in front of the
given relative @emph{pattern}:

@example
>>> sorted(Path().rglob("*.py"))
[PosixPath('build/lib/pathlib.py'),
 PosixPath('docs/conf.py'),
 PosixPath('pathlib.py'),
 PosixPath('setup.py'),
 PosixPath('test_pathlib.py')]
@end example

Raises an @ref{1130,,auditing event} @code{pathlib.Path.rglob} with arguments @code{self}, @code{pattern}.
@end deffn

@geindex rmdir() (pathlib.Path method)
@anchor{library/pathlib pathlib Path rmdir}@anchor{207f}
@deffn {Method} Path.rmdir ()

Remove this directory.  The directory must be empty.
@end deffn

@geindex samefile() (pathlib.Path method)
@anchor{library/pathlib pathlib Path samefile}@anchor{9a7}
@deffn {Method} Path.samefile (other_path)

Return whether this path points to the same file as @emph{other_path}, which
can be either a Path object, or a string.  The semantics are similar
to @ref{af2,,os.path.samefile()} and @ref{af1,,os.path.samestat()}.

An @ref{2a1,,OSError} can be raised if either file cannot be accessed for some
reason.

@example
>>> p = Path('spam')
>>> q = Path('eggs')
>>> p.samefile(q)
False
>>> p.samefile('spam')
True
@end example


New in version 3.5.

@end deffn

@geindex symlink_to() (pathlib.Path method)
@anchor{library/pathlib pathlib Path symlink_to}@anchor{2a6}
@deffn {Method} Path.symlink_to (target, target_is_directory=False)

Make this path a symbolic link to @emph{target}.  Under Windows,
@emph{target_is_directory} must be true (default @code{False}) if the link’s target
is a directory.  Under POSIX, @emph{target_is_directory}’s value is ignored.

@example
>>> p = Path('mylink')
>>> p.symlink_to('setup.py')
>>> p.resolve()
PosixPath('/home/antoine/pathlib/setup.py')
>>> p.stat().st_size
956
>>> p.lstat().st_size
8
@end example

@cartouche
@quotation Note 
The order of arguments (link, target) is the reverse
of @ref{c88,,os.symlink()}’s.
@end quotation
@end cartouche
@end deffn

@geindex hardlink_to() (pathlib.Path method)
@anchor{library/pathlib pathlib Path hardlink_to}@anchor{2a4}
@deffn {Method} Path.hardlink_to (target)

Make this path a hard link to the same file as @emph{target}.

@cartouche
@quotation Note 
The order of arguments (link, target) is the reverse
of @ref{c82,,os.link()}’s.
@end quotation
@end cartouche


New in version 3.10.

@end deffn

@geindex link_to() (pathlib.Path method)
@anchor{library/pathlib pathlib Path link_to}@anchor{2a5}
@deffn {Method} Path.link_to (target)

Make @emph{target} a hard link to this path.

@cartouche
@quotation Warning 
This function does not make this path a hard link to @emph{target}, despite
the implication of the function and argument names. The argument order
(target, link) is the reverse of @ref{2a6,,Path.symlink_to()} and
@ref{2a4,,Path.hardlink_to()}, but matches that of @ref{c82,,os.link()}.
@end quotation
@end cartouche


New in version 3.8.



Deprecated since version 3.10: This method is deprecated in favor of @ref{2a4,,Path.hardlink_to()}, as the
argument order of @ref{2a5,,Path.link_to()}  does not match that of
@ref{2a6,,Path.symlink_to()}.

@end deffn

@geindex touch() (pathlib.Path method)
@anchor{library/pathlib pathlib Path touch}@anchor{2080}
@deffn {Method} Path.touch (mode=438, exist_ok=True)

Create a file at this given path.  If @emph{mode} is given, it is combined
with the process’ @code{umask} value to determine the file mode and access
flags.  If the file already exists, the function succeeds if @emph{exist_ok}
is true (and its modification time is updated to the current time),
otherwise @ref{bb1,,FileExistsError} is raised.
@end deffn

@geindex unlink() (pathlib.Path method)
@anchor{library/pathlib pathlib Path unlink}@anchor{129c}
@deffn {Method} Path.unlink (missing_ok=False)

Remove this file or symbolic link.  If the path points to a directory,
use @ref{207f,,Path.rmdir()} instead.

If @emph{missing_ok} is false (the default), @ref{a37,,FileNotFoundError} is
raised if the path does not exist.

If @emph{missing_ok} is true, @ref{a37,,FileNotFoundError} exceptions will be
ignored (same behavior as the POSIX @code{rm -f} command).


Changed in version 3.8: The @emph{missing_ok} parameter was added.

@end deffn

@geindex write_bytes() (pathlib.Path method)
@anchor{library/pathlib pathlib Path write_bytes}@anchor{9ad}
@deffn {Method} Path.write_bytes (data)

Open the file pointed to in bytes mode, write @emph{data} to it, and close the
file:

@example
>>> p = Path('my_binary_file')
>>> p.write_bytes(b'Binary file contents')
20
>>> p.read_bytes()
b'Binary file contents'
@end example

An existing file of the same name is overwritten.


New in version 3.5.

@end deffn

@geindex write_text() (pathlib.Path method)
@anchor{library/pathlib pathlib Path write_text}@anchor{9ab}
@deffn {Method} Path.write_text (data, encoding=None, errors=None, newline=None)

Open the file pointed to in text mode, write @emph{data} to it, and close the
file:

@example
>>> p = Path('my_text_file')
>>> p.write_text('Text file contents')
18
>>> p.read_text()
'Text file contents'
@end example

An existing file of the same name is overwritten. The optional parameters
have the same meaning as in @ref{1c8,,open()}.


New in version 3.5.



Changed in version 3.10: The @emph{newline} parameter was added.

@end deffn

@node Correspondence to tools in the os module,,Concrete paths,pathlib — Object-oriented filesystem paths
@anchor{library/pathlib correspondence-to-tools-in-the-os-module}@anchor{2081}
@subsubsection Correspondence to tools in the @code{os} module


Below is a table mapping various @ref{c0,,os} functions to their corresponding
@ref{1c4,,PurePath}/@ref{522,,Path} equivalent.

@cartouche
@quotation Note 
Not all pairs of functions/methods below are equivalent. Some of them,
despite having some overlapping use-cases, have different semantics. They
include @ref{eb2,,os.path.abspath()} and @ref{10a8,,Path.absolute()},
@ref{2082,,os.path.relpath()} and @ref{2071,,PurePath.relative_to()}.
@end quotation
@end cartouche


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

@ref{c0,,os} and @ref{c1,,os.path}

@tab

@ref{c3,,pathlib}

@item

@ref{eb2,,os.path.abspath()}

@tab

@ref{10a8,,Path.absolute()} @footnote{@w{(1)} 
@ref{eb2,,os.path.abspath()} normalizes the resulting path, which may change its meaning in the presence of symlinks, while @ref{10a8,,Path.absolute()} does not.
}

@item

@ref{2a0,,os.path.realpath()}

@tab

@ref{2065,,Path.resolve()}

@item

@ref{c80,,os.chmod()}

@tab

@ref{2a8,,Path.chmod()}

@item

@ref{c83,,os.mkdir()}

@tab

@ref{9a8,,Path.mkdir()}

@item

@ref{68c,,os.makedirs()}

@tab

@ref{9a8,,Path.mkdir()}

@item

@ref{c85,,os.rename()}

@tab

@ref{207d,,Path.rename()}

@item

@ref{c86,,os.replace()}

@tab

@ref{207e,,Path.replace()}

@item

@ref{c87,,os.rmdir()}

@tab

@ref{207f,,Path.rmdir()}

@item

@ref{c84,,os.remove()}, @ref{c89,,os.unlink()}

@tab

@ref{129c,,Path.unlink()}

@item

@ref{12ac,,os.getcwd()}

@tab

@ref{2077,,Path.cwd()}

@item

@ref{51a,,os.path.exists()}

@tab

@ref{523,,Path.exists()}

@item

@ref{520,,os.path.expanduser()}

@tab

@ref{9a9,,Path.expanduser()} and
@ref{9aa,,Path.home()}

@item

@ref{c8d,,os.listdir()}

@tab

@ref{10a7,,Path.iterdir()}

@item

@ref{51c,,os.path.isdir()}

@tab

@ref{524,,Path.is_dir()}

@item

@ref{51d,,os.path.isfile()}

@tab

@ref{525,,Path.is_file()}

@item

@ref{518,,os.path.islink()}

@tab

@ref{527,,Path.is_symlink()}

@item

@ref{c82,,os.link()}

@tab

@ref{2a4,,Path.hardlink_to()}

@item

@ref{c88,,os.symlink()}

@tab

@ref{2a6,,Path.symlink_to()}

@item

@ref{414,,os.readlink()}

@tab

@ref{413,,Path.readlink()}

@item

@ref{2082,,os.path.relpath()}

@tab

@code{Path.relative_to()} @footnote{@w{(2)} 
@code{Path.relative_to()} requires @code{self} to be the subpath of the argument, but @ref{2082,,os.path.relpath()} does not.
}

@item

@ref{516,,os.stat()}

@tab

@ref{2a7,,Path.stat()},
@ref{207c,,Path.owner()},
@ref{207a,,Path.group()}

@item

@ref{2083,,os.path.isabs()}

@tab

@ref{206c,,PurePath.is_absolute()}

@item

@ref{1260,,os.path.join()}

@tab

@ref{206f,,PurePath.joinpath()}

@item

@ref{2084,,os.path.basename()}

@tab

@ref{2066,,PurePath.name}

@item

@ref{2085,,os.path.dirname()}

@tab

@ref{2064,,PurePath.parent}

@item

@ref{af2,,os.path.samefile()}

@tab

@ref{9a7,,Path.samefile()}

@item

@ref{2086,,os.path.splitext()}

@tab

@ref{2067,,PurePath.suffix}

@end multitable


@node os path — Common pathname manipulations,fileinput — Iterate over lines from multiple input streams,pathlib — Object-oriented filesystem paths,File and Directory Access
@anchor{library/os path doc}@anchor{2087}@anchor{library/os path module-os path}@anchor{c1}@anchor{library/os path os-path-common-pathname-manipulations}@anchor{2088}
@subsection @code{os.path} — Common pathname manipulations


@geindex module; os.path

@strong{Source code:} Lib/posixpath.py@footnote{https://github.com/python/cpython/tree/main/Lib/posixpath.py} (for POSIX) and
Lib/ntpath.py@footnote{https://github.com/python/cpython/tree/main/Lib/ntpath.py} (for Windows NT).

@geindex path; operations


__________________________________________________________________


This module implements some useful functions on pathnames. To read or
write files see @ref{1c8,,open()}, and for accessing the filesystem see the
@ref{c0,,os} module. The path parameters can be passed as either strings,
or bytes. Applications are encouraged to represent file names as
(Unicode) character strings. Unfortunately, some file names may not be
representable as strings on Unix, so applications that need to support
arbitrary file names on Unix should use bytes objects to represent
path names. Vice versa, using bytes objects cannot represent all file
names on Windows (in the standard @code{mbcs} encoding), hence Windows
applications should use string objects to access all files.

Unlike a unix shell, Python does not do any @emph{automatic} path expansions.
Functions such as @ref{520,,expanduser()} and @ref{f3f,,expandvars()} can be invoked
explicitly when an application desires shell-like path expansion.  (See also
the @ref{84,,glob} module.)


@subsubheading See also


The @ref{c3,,pathlib} module offers high-level path objects.


@cartouche
@quotation Note 
All of these functions accept either only bytes or only string objects as
their parameters.  The result is an object of the same type, if a path or
file name is returned.
@end quotation
@end cartouche

@cartouche
@quotation Note 
Since different operating systems have different path name conventions, there
are several versions of this module in the standard library.  The
@ref{c1,,os.path} module is always the path module suitable for the operating
system Python is running on, and therefore usable for local paths.  However,
you can also import and use the individual modules if you want to manipulate
a path that is @emph{always} in one of the different formats.  They all have the
same interface:


@itemize *

@item 
@code{posixpath} for UNIX-style paths

@item 
@code{ntpath} for Windows paths
@end itemize
@end quotation
@end cartouche


Changed in version 3.8: @ref{51a,,exists()}, @ref{51b,,lexists()}, @ref{51c,,isdir()}, @ref{51d,,isfile()},
@ref{518,,islink()}, and @ref{51e,,ismount()} now return @code{False} instead of
raising an exception for paths that contain characters or bytes
unrepresentable at the OS level.


@geindex abspath() (in module os.path)
@anchor{library/os path os path abspath}@anchor{eb2}
@deffn {Function} os.path.abspath (path)

Return a normalized absolutized version of the pathname @emph{path}. On most
platforms, this is equivalent to calling the function @ref{eb1,,normpath()} as
follows: @code{normpath(join(os.getcwd(), path))}.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex basename() (in module os.path)
@anchor{library/os path os path basename}@anchor{2084}
@deffn {Function} os.path.basename (path)

Return the base name of pathname @emph{path}.  This is the second element of the
pair returned by passing @emph{path} to the function @ref{2089,,split()}.  Note that
the result of this function is different
from the Unix @code{basename} program; where @code{basename} for
@code{'/foo/bar/'} returns @code{'bar'}, the @ref{2084,,basename()} function returns an
empty string (@code{'@w{'}}).


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex commonpath() (in module os.path)
@anchor{library/os path os path commonpath}@anchor{9a4}
@deffn {Function} os.path.commonpath (paths)

Return the longest common sub-path of each pathname in the sequence
@emph{paths}.  Raise @ref{17c,,ValueError} if @emph{paths} contain both absolute
and relative pathnames, the @emph{paths} are on the different drives or
if @emph{paths} is empty.  Unlike @ref{9a5,,commonprefix()}, this returns a
valid path.

@ref{1629,,Availability}: Unix, Windows.


New in version 3.5.



Changed in version 3.6: Accepts a sequence of @ref{63d,,path-like objects}.

@end deffn

@geindex commonprefix() (in module os.path)
@anchor{library/os path os path commonprefix}@anchor{9a5}
@deffn {Function} os.path.commonprefix (list)

Return the longest path prefix (taken character-by-character) that is a
prefix of all paths in  @emph{list}.  If @emph{list} is empty, return the empty string
(@code{'@w{'}}).

@cartouche
@quotation Note 
This function may return invalid paths because it works a
character at a time.  To obtain a valid path, see
@ref{9a4,,commonpath()}.

@example
>>> os.path.commonprefix(['/usr/lib', '/usr/local/lib'])
'/usr/l'

>>> os.path.commonpath(['/usr/lib', '/usr/local/lib'])
'/usr'
@end example
@end quotation
@end cartouche


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex dirname() (in module os.path)
@anchor{library/os path os path dirname}@anchor{2085}
@deffn {Function} os.path.dirname (path)

Return the directory name of pathname @emph{path}.  This is the first element of
the pair returned by passing @emph{path} to the function @ref{2089,,split()}.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex exists() (in module os.path)
@anchor{library/os path os path exists}@anchor{51a}
@deffn {Function} os.path.exists (path)

Return @code{True} if @emph{path} refers to an existing path or an open
file descriptor.  Returns @code{False} for broken symbolic links.  On
some platforms, this function may return @code{False} if permission is
not granted to execute @ref{516,,os.stat()} on the requested file, even
if the @emph{path} physically exists.


Changed in version 3.3: @emph{path} can now be an integer: @code{True} is returned if it is an
 open file descriptor, @code{False} otherwise.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex lexists() (in module os.path)
@anchor{library/os path os path lexists}@anchor{51b}
@deffn {Function} os.path.lexists (path)

Return @code{True} if @emph{path} refers to an existing path. Returns @code{True} for
broken symbolic links.   Equivalent to @ref{51a,,exists()} on platforms lacking
@ref{517,,os.lstat()}.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex ~ (tilde); home directory expansion

@geindex expanduser() (in module os.path)
@anchor{library/os path os path expanduser}@anchor{520}
@deffn {Function} os.path.expanduser (path)

On Unix and Windows, return the argument with an initial component of @code{~} or
@code{~user} replaced by that @emph{user}’s home directory.

@geindex module; pwd

On Unix, an initial @code{~} is replaced by the environment variable 
@geindex HOME
@geindex environment variable; HOME
@code{HOME}
if it is set; otherwise the current user’s home directory is looked up in the
password directory through the built-in module @ref{d1,,pwd}. An initial @code{~user}
is looked up directly in the password directory.

On Windows, 
@geindex USERPROFILE
@geindex environment variable; USERPROFILE
@code{USERPROFILE} will be used if set, otherwise a combination
of 
@geindex HOMEPATH
@geindex environment variable; HOMEPATH
@code{HOMEPATH} and 
@geindex HOMEDRIVE
@geindex environment variable; HOMEDRIVE
@code{HOMEDRIVE} will be used.  An initial
@code{~user} is handled by checking that the last directory component of the current
user’s home directory matches 
@geindex USERNAME
@geindex environment variable; USERNAME
@code{USERNAME}, and replacing it if so.

If the expansion fails or if the path does not begin with a tilde, the path is
returned unchanged.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.



Changed in version 3.8: No longer uses 
@geindex HOME
@geindex environment variable; HOME
@code{HOME} on Windows.

@end deffn

@geindex $ (dollar); environment variables expansion
@geindex % (percent); environment variables expansion (Windows)

@geindex expandvars() (in module os.path)
@anchor{library/os path os path expandvars}@anchor{f3f}
@deffn {Function} os.path.expandvars (path)

Return the argument with environment variables expanded.  Substrings of the form
@code{$name} or @code{$@{name@}} are replaced by the value of environment variable
@emph{name}.  Malformed variable names and references to non-existing variables are
left unchanged.

On Windows, @code{%name%} expansions are supported in addition to @code{$name} and
@code{$@{name@}}.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex getatime() (in module os.path)
@anchor{library/os path os path getatime}@anchor{208a}
@deffn {Function} os.path.getatime (path)

Return the time of last access of @emph{path}.  The return value is a floating point number giving
the number of seconds since the epoch (see the  @ref{10a,,time} module).  Raise
@ref{2a1,,OSError} if the file does not exist or is inaccessible.
@end deffn

@geindex getmtime() (in module os.path)
@anchor{library/os path os path getmtime}@anchor{208b}
@deffn {Function} os.path.getmtime (path)

Return the time of last modification of @emph{path}.  The return value is a floating point number
giving the number of seconds since the epoch (see the  @ref{10a,,time} module).
Raise @ref{2a1,,OSError} if the file does not exist or is inaccessible.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex getctime() (in module os.path)
@anchor{library/os path os path getctime}@anchor{208c}
@deffn {Function} os.path.getctime (path)

Return the system’s ctime which, on some systems (like Unix) is the time of the
last metadata change, and, on others (like Windows), is the creation time for @emph{path}.
The return value is a number giving the number of seconds since the epoch (see
the  @ref{10a,,time} module).  Raise @ref{2a1,,OSError} if the file does not exist or
is inaccessible.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex getsize() (in module os.path)
@anchor{library/os path os path getsize}@anchor{208d}
@deffn {Function} os.path.getsize (path)

Return the size, in bytes, of @emph{path}.  Raise @ref{2a1,,OSError} if the file does
not exist or is inaccessible.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex isabs() (in module os.path)
@anchor{library/os path os path isabs}@anchor{2083}
@deffn {Function} os.path.isabs (path)

Return @code{True} if @emph{path} is an absolute pathname.  On Unix, that means it
begins with a slash, on Windows that it begins with a (back)slash after chopping
off a potential drive letter.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex isfile() (in module os.path)
@anchor{library/os path os path isfile}@anchor{51d}
@deffn {Function} os.path.isfile (path)

Return @code{True} if @emph{path} is an @ref{51a,,existing} regular file.
This follows symbolic links, so both @ref{518,,islink()} and @ref{51d,,isfile()} can
be true for the same path.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex isdir() (in module os.path)
@anchor{library/os path os path isdir}@anchor{51c}
@deffn {Function} os.path.isdir (path)

Return @code{True} if @emph{path} is an @ref{51a,,existing} directory.  This
follows symbolic links, so both @ref{518,,islink()} and @ref{51c,,isdir()} can be true
for the same path.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex islink() (in module os.path)
@anchor{library/os path os path islink}@anchor{518}
@deffn {Function} os.path.islink (path)

Return @code{True} if @emph{path} refers to an @ref{51a,,existing} directory
entry that is a symbolic link.  Always @code{False} if symbolic links are not
supported by the Python runtime.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex ismount() (in module os.path)
@anchor{library/os path os path ismount}@anchor{51e}
@deffn {Function} os.path.ismount (path)

Return @code{True} if pathname @emph{path} is a @emph{mount point}: a point in a
file system where a different file system has been mounted.  On POSIX, the
function checks whether @emph{path}’s parent, @code{@emph{path}/..}, is on a different
device than @emph{path}, or whether @code{@emph{path}/..} and @emph{path} point to the same
i-node on the same device — this should detect mount points for all Unix
and POSIX variants.  It is not able to reliably detect bind mounts on the
same filesystem.  On Windows, a drive letter root and a share UNC are
always mount points, and for any other path @code{GetVolumePathName} is called
to see if it is different from the input path.


New in version 3.4: Support for detecting non-root mount points on Windows.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex join() (in module os.path)
@anchor{library/os path os path join}@anchor{1260}
@deffn {Function} os.path.join (path, *paths)

Join one or more path components intelligently.  The return value is the
concatenation of @emph{path} and any members of @emph{*paths} with exactly one
directory separator following each non-empty part except the last, meaning
that the result will only end in a separator if the last part is empty.  If
a component is an absolute path, all previous components are thrown away
and joining continues from the absolute path component.

On Windows, the drive letter is not reset when an absolute path component
(e.g., @code{r'\foo'}) is encountered.  If a component contains a drive
letter, all previous components are thrown away and the drive letter is
reset.  Note that since there is a current directory for each drive,
@code{os.path.join("c:", "foo")} represents a path relative to the current
directory on drive @code{C:} (@code{c:foo}), not @code{c:\foo}.


Changed in version 3.6: Accepts a @ref{63d,,path-like object} for @emph{path} and @emph{paths}.

@end deffn

@geindex normcase() (in module os.path)
@anchor{library/os path os path normcase}@anchor{208e}
@deffn {Function} os.path.normcase (path)

Normalize the case of a pathname.  On Windows, convert all characters in the
pathname to lowercase, and also convert forward slashes to backward slashes.
On other operating systems, return the path unchanged.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex normpath() (in module os.path)
@anchor{library/os path os path normpath}@anchor{eb1}
@deffn {Function} os.path.normpath (path)

@quotation

Normalize a pathname by collapsing redundant separators and up-level
references so that @code{A//B}, @code{A/B/}, @code{A/./B} and @code{A/foo/../B} all
become @code{A/B}.  This string manipulation may change the meaning of a path
that contains symbolic links.  On Windows, it converts forward slashes to
backward slashes. To normalize case, use @ref{208e,,normcase()}.
@end quotation

@cartouche
@quotation Note 
@quotation

On POSIX systems, in accordance with IEEE Std 1003.1 2013 Edition; 4.13 Pathname Resolution@footnote{http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_13},
if a pathname begins with exactly two slashes, the first component
following the leading characters may be interpreted in an implementation-defined
manner, although more than two leading characters shall be treated as a
single character.
@end quotation


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end quotation
@end cartouche
@end deffn

@geindex realpath() (in module os.path)
@anchor{library/os path os path realpath}@anchor{2a0}
@deffn {Function} os.path.realpath (path, *, strict=False)

Return the canonical path of the specified filename, eliminating any symbolic
links encountered in the path (if they are supported by the operating
system).

If a path doesn’t exist or a symlink loop is encountered, and @emph{strict} is
@code{True}, @ref{2a1,,OSError} is raised. If @emph{strict} is @code{False}, the path is
resolved as far as possible and any remainder is appended without checking
whether it exists.

@cartouche
@quotation Note 
This function emulates the operating system’s procedure for making a path
canonical, which differs slightly between Windows and UNIX with respect
to how links and subsequent path components interact.

Operating system APIs make paths canonical as needed, so it’s not
normally necessary to call this function.
@end quotation
@end cartouche


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.



Changed in version 3.8: Symbolic links and junctions are now resolved on Windows.



Changed in version 3.10: The @emph{strict} parameter was added.

@end deffn

@geindex relpath() (in module os.path)
@anchor{library/os path os path relpath}@anchor{2082}
@deffn {Function} os.path.relpath (path, start=os.curdir)

Return a relative filepath to @emph{path} either from the current directory or
from an optional @emph{start} directory.  This is a path computation:  the
filesystem is not accessed to confirm the existence or nature of @emph{path} or
@emph{start}.  On Windows, @ref{17c,,ValueError} is raised when @emph{path} and @emph{start}
are on different drives.

@emph{start} defaults to @ref{208f,,os.curdir}.

@ref{1629,,Availability}: Unix, Windows.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex samefile() (in module os.path)
@anchor{library/os path os path samefile}@anchor{af2}
@deffn {Function} os.path.samefile (path1, path2)

Return @code{True} if both pathname arguments refer to the same file or directory.
This is determined by the device number and i-node number and raises an
exception if an @ref{516,,os.stat()} call on either pathname fails.

@ref{1629,,Availability}: Unix, Windows.


Changed in version 3.2: Added Windows support.



Changed in version 3.4: Windows now uses the same implementation as all other platforms.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex sameopenfile() (in module os.path)
@anchor{library/os path os path sameopenfile}@anchor{2090}
@deffn {Function} os.path.sameopenfile (fp1, fp2)

Return @code{True} if the file descriptors @emph{fp1} and @emph{fp2} refer to the same file.

@ref{1629,,Availability}: Unix, Windows.


Changed in version 3.2: Added Windows support.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex samestat() (in module os.path)
@anchor{library/os path os path samestat}@anchor{af1}
@deffn {Function} os.path.samestat (stat1, stat2)

Return @code{True} if the stat tuples @emph{stat1} and @emph{stat2} refer to the same file.
These structures may have been returned by @ref{8e8,,os.fstat()},
@ref{517,,os.lstat()}, or @ref{516,,os.stat()}.  This function implements the
underlying comparison used by @ref{af2,,samefile()} and @ref{2090,,sameopenfile()}.

@ref{1629,,Availability}: Unix, Windows.


Changed in version 3.4: Added Windows support.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex split() (in module os.path)
@anchor{library/os path os path split}@anchor{2089}
@deffn {Function} os.path.split (path)

Split the pathname @emph{path} into a pair, @code{(head, tail)} where @emph{tail} is the
last pathname component and @emph{head} is everything leading up to that.  The
@emph{tail} part will never contain a slash; if @emph{path} ends in a slash, @emph{tail}
will be empty.  If there is no slash in @emph{path}, @emph{head} will be empty.  If
@emph{path} is empty, both @emph{head} and @emph{tail} are empty.  Trailing slashes are
stripped from @emph{head} unless it is the root (one or more slashes only).  In
all cases, @code{join(head, tail)} returns a path to the same location as @emph{path}
(but the strings may differ).  Also see the functions @ref{2085,,dirname()} and
@ref{2084,,basename()}.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex splitdrive() (in module os.path)
@anchor{library/os path os path splitdrive}@anchor{730}
@deffn {Function} os.path.splitdrive (path)

Split the pathname @emph{path} into a pair @code{(drive, tail)} where @emph{drive} is either
a mount point or the empty string.  On systems which do not use drive
specifications, @emph{drive} will always be the empty string.  In all cases, @code{drive
+ tail} will be the same as @emph{path}.

On Windows, splits a pathname into drive/UNC sharepoint and relative path.

If the path contains a drive letter, drive will contain everything
up to and including the colon:

@example
>>> splitdrive("c:/dir")
("c:", "/dir")
@end example

If the path contains a UNC path, drive will contain the host name
and share, up to but not including the fourth separator:

@example
>>> splitdrive("//host/computer/dir")
("//host/computer", "/dir")
@end example


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex splitext() (in module os.path)
@anchor{library/os path os path splitext}@anchor{2086}
@deffn {Function} os.path.splitext (path)

Split the pathname @emph{path} into a pair @code{(root, ext)}  such that @code{root + ext ==
path}, and the extension, @emph{ext}, is empty or begins with a period and contains at
most one period.

If the path contains no extension, @emph{ext} will be @code{'@w{'}}:

@example
>>> splitext('bar')
('bar', '@w{'})
@end example

If the path contains an extension, then @emph{ext} will be set to this extension,
including the leading period. Note that previous periods will be ignored:

@example
>>> splitext('foo.bar.exe')
('foo.bar', '.exe')
>>> splitext('/foo/bar.exe')
('/foo/bar', '.exe')
@end example

Leading periods of the last component of the path are considered to
be part of the root:

@example
>>> splitext('.cshrc')
('.cshrc', '@w{'})
>>> splitext('/foo/....jpg')
('/foo/....jpg', '@w{'})
@end example


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex supports_unicode_filenames (in module os.path)
@anchor{library/os path os path supports_unicode_filenames}@anchor{fe5}
@deffn {Data} os.path.supports_unicode_filenames

@code{True} if arbitrary Unicode strings can be used as file names (within limitations
imposed by the file system).
@end deffn

@node fileinput — Iterate over lines from multiple input streams,stat — Interpreting stat results,os path — Common pathname manipulations,File and Directory Access
@anchor{library/fileinput doc}@anchor{2091}@anchor{library/fileinput fileinput-iterate-over-lines-from-multiple-input-streams}@anchor{2092}@anchor{library/fileinput module-fileinput}@anchor{7b}
@subsection @code{fileinput} — Iterate over lines from multiple input streams


@geindex module; fileinput

@strong{Source code:} Lib/fileinput.py@footnote{https://github.com/python/cpython/tree/main/Lib/fileinput.py}


__________________________________________________________________


This module implements a helper class and functions to quickly write a
loop over standard input or a list of files. If you just want to read or
write one file see @ref{1c8,,open()}.

The typical use is:

@example
import fileinput
for line in fileinput.input(encoding="utf-8"):
    process(line)
@end example

This iterates over the lines of all files listed in @code{sys.argv[1:]}, defaulting
to @code{sys.stdin} if the list is empty.  If a filename is @code{'-'}, it is also
replaced by @code{sys.stdin} and the optional arguments @emph{mode} and @emph{openhook}
are ignored.  To specify an alternative list of filenames, pass it as the
first argument to @ref{280,,input()}.  A single file name is also allowed.

All files are opened in text mode by default, but you can override this by
specifying the @emph{mode} parameter in the call to @ref{280,,input()} or
@ref{1bc,,FileInput}.  If an I/O error occurs during opening or reading a file,
@ref{2a1,,OSError} is raised.


Changed in version 3.3: @ref{be8,,IOError} used to be raised; it is now an alias of @ref{2a1,,OSError}.


If @code{sys.stdin} is used more than once, the second and further use will return
no lines, except perhaps for interactive use, or if it has been explicitly reset
(e.g. using @code{sys.stdin.seek(0)}).

Empty files are opened and immediately closed; the only time their presence in
the list of filenames is noticeable at all is when the last file opened is
empty.

Lines are returned with any newlines intact, which means that the last line in
a file may not have one.

You can control how files are opened by providing an opening hook via the
@emph{openhook} parameter to @ref{280,,fileinput.input()} or @ref{1bc,,FileInput()}. The
hook must be a function that takes two arguments, @emph{filename} and @emph{mode}, and
returns an accordingly opened file-like object. If @emph{encoding} and/or @emph{errors}
are specified, they will be passed to the hook as additional keyword arguments.
This module provides a @ref{281,,hook_compressed()} to support compressed files.

The following function is the primary interface of this module:

@geindex input() (in module fileinput)
@anchor{library/fileinput fileinput input}@anchor{280}
@deffn {Function} fileinput.input (files=None, inplace=False, backup='@w{'}, *, mode='r', openhook=None, encoding=None, errors=None)

Create an instance of the @ref{1bc,,FileInput} class.  The instance will be used
as global state for the functions of this module, and is also returned to use
during iteration.  The parameters to this function will be passed along to the
constructor of the @ref{1bc,,FileInput} class.

The @ref{1bc,,FileInput} instance can be used as a context manager in the
@ref{168,,with} statement.  In this example, @emph{input} is closed after the
@code{with} statement is exited, even if an exception occurs:

@example
with fileinput.input(files=('spam.txt', 'eggs.txt'), encoding="utf-8") as f:
    for line in f:
        process(line)
@end example


Changed in version 3.2: Can be used as a context manager.



Changed in version 3.8: The keyword parameters @emph{mode} and @emph{openhook} are now keyword-only.



Changed in version 3.10: The keyword-only parameter @emph{encoding} and @emph{errors} are added.

@end deffn

The following functions use the global state created by @ref{280,,fileinput.input()};
if there is no active state, @ref{5af,,RuntimeError} is raised.

@geindex filename() (in module fileinput)
@anchor{library/fileinput fileinput filename}@anchor{2093}
@deffn {Function} fileinput.filename ()

Return the name of the file currently being read.  Before the first line has
been read, returns @code{None}.
@end deffn

@geindex fileno() (in module fileinput)
@anchor{library/fileinput fileinput fileno}@anchor{2094}
@deffn {Function} fileinput.fileno ()

Return the integer “file descriptor” for the current file. When no file is
opened (before the first line and between files), returns @code{-1}.
@end deffn

@geindex lineno() (in module fileinput)
@anchor{library/fileinput fileinput lineno}@anchor{2095}
@deffn {Function} fileinput.lineno ()

Return the cumulative line number of the line that has just been read.  Before
the first line has been read, returns @code{0}.  After the last line of the last
file has been read, returns the line number of that line.
@end deffn

@geindex filelineno() (in module fileinput)
@anchor{library/fileinput fileinput filelineno}@anchor{2096}
@deffn {Function} fileinput.filelineno ()

Return the line number in the current file.  Before the first line has been
read, returns @code{0}.  After the last line of the last file has been read,
returns the line number of that line within the file.
@end deffn

@geindex isfirstline() (in module fileinput)
@anchor{library/fileinput fileinput isfirstline}@anchor{2097}
@deffn {Function} fileinput.isfirstline ()

Return @code{True} if the line just read is the first line of its file, otherwise
return @code{False}.
@end deffn

@geindex isstdin() (in module fileinput)
@anchor{library/fileinput fileinput isstdin}@anchor{2098}
@deffn {Function} fileinput.isstdin ()

Return @code{True} if the last line was read from @code{sys.stdin}, otherwise return
@code{False}.
@end deffn

@geindex nextfile() (in module fileinput)
@anchor{library/fileinput fileinput nextfile}@anchor{2099}
@deffn {Function} fileinput.nextfile ()

Close the current file so that the next iteration will read the first line from
the next file (if any); lines not read from the file will not count towards the
cumulative line count.  The filename is not changed until after the first line
of the next file has been read.  Before the first line has been read, this
function has no effect; it cannot be used to skip the first file.  After the
last line of the last file has been read, this function has no effect.
@end deffn

@geindex close() (in module fileinput)
@anchor{library/fileinput fileinput close}@anchor{209a}
@deffn {Function} fileinput.close ()

Close the sequence.
@end deffn

The class which implements the sequence behavior provided by the module is
available for subclassing as well:

@geindex FileInput (class in fileinput)
@anchor{library/fileinput fileinput FileInput}@anchor{1bc}
@deffn {Class} fileinput.FileInput (files=None, inplace=False, backup='@w{'}, *, mode='r', openhook=None, encoding=None, errors=None)

Class @ref{1bc,,FileInput} is the implementation; its methods @ref{2093,,filename()},
@ref{2094,,fileno()}, @ref{2095,,lineno()}, @ref{2096,,filelineno()}, @ref{2097,,isfirstline()},
@ref{2098,,isstdin()}, @ref{2099,,nextfile()} and @ref{209a,,close()} correspond to the
functions of the same name in the module. In addition it is @ref{dcc,,iterable}
and has a @ref{209b,,readline()} method which returns the next
input line. The sequence must be accessed in strictly sequential order;
random access and @ref{209b,,readline()} cannot be mixed.

With @emph{mode} you can specify which file mode will be passed to @ref{1c8,,open()}. It
must be one of @code{'r'} and @code{'rb'}.

The @emph{openhook}, when given, must be a function that takes two arguments,
@emph{filename} and @emph{mode}, and returns an accordingly opened file-like object. You
cannot use @emph{inplace} and @emph{openhook} together.

You can specify @emph{encoding} and @emph{errors} that is passed to @ref{1c8,,open()} or @emph{openhook}.

A @ref{1bc,,FileInput} instance can be used as a context manager in the
@ref{168,,with} statement.  In this example, @emph{input} is closed after the
@code{with} statement is exited, even if an exception occurs:

@example
with FileInput(files=('spam.txt', 'eggs.txt')) as input:
    process(input)
@end example


Changed in version 3.2: Can be used as a context manager.



Changed in version 3.8: The keyword parameter @emph{mode} and @emph{openhook} are now keyword-only.



Changed in version 3.10: The keyword-only parameter @emph{encoding} and @emph{errors} are added.



Changed in version 3.11: The @code{'rU'} and @code{'U'} modes and the @code{__getitem__()} method have
been removed.

@end deffn

@strong{Optional in-place filtering:} if the keyword argument @code{inplace=True} is
passed to @ref{280,,fileinput.input()} or to the @ref{1bc,,FileInput} constructor, the
file is moved to a backup file and standard output is directed to the input file
(if a file of the same name as the backup file already exists, it will be
replaced silently).  This makes it possible to write a filter that rewrites its
input file in place.  If the @emph{backup} parameter is given (typically as
@code{backup='.<some extension>'}), it specifies the extension for the backup file,
and the backup file remains around; by default, the extension is @code{'.bak'} and
it is deleted when the output file is closed.  In-place filtering is disabled
when standard input is read.

The two following opening hooks are provided by this module:

@geindex hook_compressed() (in module fileinput)
@anchor{library/fileinput fileinput hook_compressed}@anchor{281}
@deffn {Function} fileinput.hook_compressed (filename, mode, *, encoding=None, errors=None)

Transparently opens files compressed with gzip and bzip2 (recognized by the
extensions @code{'.gz'} and @code{'.bz2'}) using the @ref{87,,gzip} and @ref{12,,bz2}
modules.  If the filename extension is not @code{'.gz'} or @code{'.bz2'}, the file is
opened normally (ie, using @ref{1c8,,open()} without any decompression).

The @emph{encoding} and @emph{errors} values are passed to @ref{889,,io.TextIOWrapper}
for compressed files and open for normal files.

Usage example:  @code{fi = fileinput.FileInput(openhook=fileinput.hook_compressed, encoding="utf-8")}


Changed in version 3.10: The keyword-only parameter @emph{encoding} and @emph{errors} are added.

@end deffn

@geindex hook_encoded() (in module fileinput)
@anchor{library/fileinput fileinput hook_encoded}@anchor{7ec}
@deffn {Function} fileinput.hook_encoded (encoding, errors=None)

Returns a hook which opens each file with @ref{1c8,,open()}, using the given
@emph{encoding} and @emph{errors} to read the file.

Usage example: @code{fi =
fileinput.FileInput(openhook=fileinput.hook_encoded("utf-8",
"surrogateescape"))}


Changed in version 3.6: Added the optional @emph{errors} parameter.



Deprecated since version 3.10: This function is deprecated since @ref{e73,,input()} and @ref{1bc,,FileInput}
now have @emph{encoding} and @emph{errors} parameters.

@end deffn

@node stat — Interpreting stat results,filecmp — File and Directory Comparisons,fileinput — Iterate over lines from multiple input streams,File and Directory Access
@anchor{library/stat doc}@anchor{209c}@anchor{library/stat module-stat}@anchor{ef}@anchor{library/stat stat-interpreting-stat-results}@anchor{209d}
@subsection @code{stat} — Interpreting @code{stat()} results


@geindex module; stat

@strong{Source code:} Lib/stat.py@footnote{https://github.com/python/cpython/tree/main/Lib/stat.py}


__________________________________________________________________


The @ref{ef,,stat} module defines constants and functions for interpreting the
results of @ref{516,,os.stat()}, @ref{8e8,,os.fstat()} and @ref{517,,os.lstat()} (if they
exist).  For complete details about the @code{stat()}, @code{fstat()} and
@code{lstat()} calls, consult the documentation for your system.


Changed in version 3.4: The stat module is backed by a C implementation.


The @ref{ef,,stat} module defines the following functions to test for specific file
types:

@geindex S_ISDIR() (in module stat)
@anchor{library/stat stat S_ISDIR}@anchor{209e}
@deffn {Function} stat.S_ISDIR (mode)

Return non-zero if the mode is from a directory.
@end deffn

@geindex S_ISCHR() (in module stat)
@anchor{library/stat stat S_ISCHR}@anchor{209f}
@deffn {Function} stat.S_ISCHR (mode)

Return non-zero if the mode is from a character special device file.
@end deffn

@geindex S_ISBLK() (in module stat)
@anchor{library/stat stat S_ISBLK}@anchor{20a0}
@deffn {Function} stat.S_ISBLK (mode)

Return non-zero if the mode is from a block special device file.
@end deffn

@geindex S_ISREG() (in module stat)
@anchor{library/stat stat S_ISREG}@anchor{20a1}
@deffn {Function} stat.S_ISREG (mode)

Return non-zero if the mode is from a regular file.
@end deffn

@geindex S_ISFIFO() (in module stat)
@anchor{library/stat stat S_ISFIFO}@anchor{20a2}
@deffn {Function} stat.S_ISFIFO (mode)

Return non-zero if the mode is from a FIFO (named pipe).
@end deffn

@geindex S_ISLNK() (in module stat)
@anchor{library/stat stat S_ISLNK}@anchor{20a3}
@deffn {Function} stat.S_ISLNK (mode)

Return non-zero if the mode is from a symbolic link.
@end deffn

@geindex S_ISSOCK() (in module stat)
@anchor{library/stat stat S_ISSOCK}@anchor{20a4}
@deffn {Function} stat.S_ISSOCK (mode)

Return non-zero if the mode is from a socket.
@end deffn

@geindex S_ISDOOR() (in module stat)
@anchor{library/stat stat S_ISDOOR}@anchor{1261}
@deffn {Function} stat.S_ISDOOR (mode)

Return non-zero if the mode is from a door.


New in version 3.4.

@end deffn

@geindex S_ISPORT() (in module stat)
@anchor{library/stat stat S_ISPORT}@anchor{1262}
@deffn {Function} stat.S_ISPORT (mode)

Return non-zero if the mode is from an event port.


New in version 3.4.

@end deffn

@geindex S_ISWHT() (in module stat)
@anchor{library/stat stat S_ISWHT}@anchor{1263}
@deffn {Function} stat.S_ISWHT (mode)

Return non-zero if the mode is from a whiteout.


New in version 3.4.

@end deffn

Two additional functions are defined for more general manipulation of the file’s
mode:

@geindex S_IMODE() (in module stat)
@anchor{library/stat stat S_IMODE}@anchor{20a5}
@deffn {Function} stat.S_IMODE (mode)

Return the portion of the file’s mode that can be set by
@ref{c80,,os.chmod()}—that is, the file’s permission bits, plus the sticky
bit, set-group-id, and set-user-id bits (on systems that support them).
@end deffn

@geindex S_IFMT() (in module stat)
@anchor{library/stat stat S_IFMT}@anchor{20a6}
@deffn {Function} stat.S_IFMT (mode)

Return the portion of the file’s mode that describes the file type (used by the
@code{S_IS*()} functions above).
@end deffn

Normally, you would use the @code{os.path.is*()} functions for testing the type
of a file; the functions here are useful when you are doing multiple tests of
the same file and wish to avoid the overhead of the @code{stat()} system call
for each test.  These are also useful when checking for information about a file
that isn’t handled by @ref{c1,,os.path}, like the tests for block and character
devices.

Example:

@example
import os, sys
from stat import *

def walktree(top, callback):
    '@w{'}'recursively descend the directory tree rooted at top,
       calling the callback function for each regular file'@w{'}'

    for f in os.listdir(top):
        pathname = os.path.join(top, f)
        mode = os.stat(pathname).st_mode
        if S_ISDIR(mode):
            # It's a directory, recurse into it
            walktree(pathname, callback)
        elif S_ISREG(mode):
            # It's a file, call the callback function
            callback(pathname)
        else:
            # Unknown file type, print a message
            print('Skipping %s' % pathname)

def visitfile(file):
    print('visiting', file)

if __name__ == '__main__':
    walktree(sys.argv[1], visitfile)
@end example

An additional utility function is provided to convert a file’s mode in a human
readable string:

@geindex filemode() (in module stat)
@anchor{library/stat stat filemode}@anchor{ce3}
@deffn {Function} stat.filemode (mode)

Convert a file’s mode to a string of the form ‘-rwxrwxrwx’.


New in version 3.3.



Changed in version 3.4: The function supports @ref{b3c,,S_IFDOOR}, @ref{b3d,,S_IFPORT} and
@ref{b3e,,S_IFWHT}.

@end deffn

All the variables below are simply symbolic indexes into the 10-tuple returned
by @ref{516,,os.stat()}, @ref{8e8,,os.fstat()} or @ref{517,,os.lstat()}.

@geindex ST_MODE (in module stat)
@anchor{library/stat stat ST_MODE}@anchor{b3b}
@deffn {Data} stat.ST_MODE

Inode protection mode.
@end deffn

@geindex ST_INO (in module stat)
@anchor{library/stat stat ST_INO}@anchor{20a7}
@deffn {Data} stat.ST_INO

Inode number.
@end deffn

@geindex ST_DEV (in module stat)
@anchor{library/stat stat ST_DEV}@anchor{20a8}
@deffn {Data} stat.ST_DEV

Device inode resides on.
@end deffn

@geindex ST_NLINK (in module stat)
@anchor{library/stat stat ST_NLINK}@anchor{20a9}
@deffn {Data} stat.ST_NLINK

Number of links to the inode.
@end deffn

@geindex ST_UID (in module stat)
@anchor{library/stat stat ST_UID}@anchor{20aa}
@deffn {Data} stat.ST_UID

User id of the owner.
@end deffn

@geindex ST_GID (in module stat)
@anchor{library/stat stat ST_GID}@anchor{20ab}
@deffn {Data} stat.ST_GID

Group id of the owner.
@end deffn

@geindex ST_SIZE (in module stat)
@anchor{library/stat stat ST_SIZE}@anchor{20ac}
@deffn {Data} stat.ST_SIZE

Size in bytes of a plain file; amount of data waiting on some special files.
@end deffn

@geindex ST_ATIME (in module stat)
@anchor{library/stat stat ST_ATIME}@anchor{20ad}
@deffn {Data} stat.ST_ATIME

Time of last access.
@end deffn

@geindex ST_MTIME (in module stat)
@anchor{library/stat stat ST_MTIME}@anchor{20ae}
@deffn {Data} stat.ST_MTIME

Time of last modification.
@end deffn

@geindex ST_CTIME (in module stat)
@anchor{library/stat stat ST_CTIME}@anchor{20af}
@deffn {Data} stat.ST_CTIME

The “ctime” as reported by the operating system.  On some systems (like Unix) is
the time of the last metadata change, and, on others (like Windows), is the
creation time (see platform documentation for details).
@end deffn

The interpretation of “file size” changes according to the file type.  For plain
files this is the size of the file in bytes.  For FIFOs and sockets under most
flavors of Unix (including Linux in particular), the “size” is the number of
bytes waiting to be read at the time of the call to @ref{516,,os.stat()},
@ref{8e8,,os.fstat()}, or @ref{517,,os.lstat()}; this can sometimes be useful, especially
for polling one of these special files after a non-blocking open.  The meaning
of the size field for other character and block devices varies more, depending
on the implementation of the underlying system call.

The variables below define the flags used in the @ref{b3b,,ST_MODE} field.

Use of the functions above is more portable than use of the first set of flags:

@geindex S_IFSOCK (in module stat)
@anchor{library/stat stat S_IFSOCK}@anchor{20b0}
@deffn {Data} stat.S_IFSOCK

Socket.
@end deffn

@geindex S_IFLNK (in module stat)
@anchor{library/stat stat S_IFLNK}@anchor{20b1}
@deffn {Data} stat.S_IFLNK

Symbolic link.
@end deffn

@geindex S_IFREG (in module stat)
@anchor{library/stat stat S_IFREG}@anchor{20b2}
@deffn {Data} stat.S_IFREG

Regular file.
@end deffn

@geindex S_IFBLK (in module stat)
@anchor{library/stat stat S_IFBLK}@anchor{20b3}
@deffn {Data} stat.S_IFBLK

Block device.
@end deffn

@geindex S_IFDIR (in module stat)
@anchor{library/stat stat S_IFDIR}@anchor{20b4}
@deffn {Data} stat.S_IFDIR

Directory.
@end deffn

@geindex S_IFCHR (in module stat)
@anchor{library/stat stat S_IFCHR}@anchor{20b5}
@deffn {Data} stat.S_IFCHR

Character device.
@end deffn

@geindex S_IFIFO (in module stat)
@anchor{library/stat stat S_IFIFO}@anchor{20b6}
@deffn {Data} stat.S_IFIFO

FIFO.
@end deffn

@geindex S_IFDOOR (in module stat)
@anchor{library/stat stat S_IFDOOR}@anchor{b3c}
@deffn {Data} stat.S_IFDOOR

Door.


New in version 3.4.

@end deffn

@geindex S_IFPORT (in module stat)
@anchor{library/stat stat S_IFPORT}@anchor{b3d}
@deffn {Data} stat.S_IFPORT

Event port.


New in version 3.4.

@end deffn

@geindex S_IFWHT (in module stat)
@anchor{library/stat stat S_IFWHT}@anchor{b3e}
@deffn {Data} stat.S_IFWHT

Whiteout.


New in version 3.4.

@end deffn

@cartouche
@quotation Note 
@ref{b3c,,S_IFDOOR}, @ref{b3d,,S_IFPORT} or @ref{b3e,,S_IFWHT} are defined as
0 when the platform does not have support for the file types.
@end quotation
@end cartouche

The following flags can also be used in the @emph{mode} argument of @ref{c80,,os.chmod()}:

@geindex S_ISUID (in module stat)
@anchor{library/stat stat S_ISUID}@anchor{20b7}
@deffn {Data} stat.S_ISUID

Set UID bit.
@end deffn

@geindex S_ISGID (in module stat)
@anchor{library/stat stat S_ISGID}@anchor{20b8}
@deffn {Data} stat.S_ISGID

Set-group-ID bit.  This bit has several special uses.  For a directory
it indicates that BSD semantics is to be used for that directory:
files created there inherit their group ID from the directory, not
from the effective group ID of the creating process, and directories
created there will also get the @ref{20b8,,S_ISGID} bit set.  For a
file that does not have the group execution bit (@ref{20b9,,S_IXGRP})
set, the set-group-ID bit indicates mandatory file/record locking
(see also @ref{20ba,,S_ENFMT}).
@end deffn

@geindex S_ISVTX (in module stat)
@anchor{library/stat stat S_ISVTX}@anchor{20bb}
@deffn {Data} stat.S_ISVTX

Sticky bit.  When this bit is set on a directory it means that a file
in that directory can be renamed or deleted only by the owner of the
file, by the owner of the directory, or by a privileged process.
@end deffn

@geindex S_IRWXU (in module stat)
@anchor{library/stat stat S_IRWXU}@anchor{20bc}
@deffn {Data} stat.S_IRWXU

Mask for file owner permissions.
@end deffn

@geindex S_IRUSR (in module stat)
@anchor{library/stat stat S_IRUSR}@anchor{20bd}
@deffn {Data} stat.S_IRUSR

Owner has read permission.
@end deffn

@geindex S_IWUSR (in module stat)
@anchor{library/stat stat S_IWUSR}@anchor{20be}
@deffn {Data} stat.S_IWUSR

Owner has write permission.
@end deffn

@geindex S_IXUSR (in module stat)
@anchor{library/stat stat S_IXUSR}@anchor{20bf}
@deffn {Data} stat.S_IXUSR

Owner has execute permission.
@end deffn

@geindex S_IRWXG (in module stat)
@anchor{library/stat stat S_IRWXG}@anchor{20c0}
@deffn {Data} stat.S_IRWXG

Mask for group permissions.
@end deffn

@geindex S_IRGRP (in module stat)
@anchor{library/stat stat S_IRGRP}@anchor{20c1}
@deffn {Data} stat.S_IRGRP

Group has read permission.
@end deffn

@geindex S_IWGRP (in module stat)
@anchor{library/stat stat S_IWGRP}@anchor{20c2}
@deffn {Data} stat.S_IWGRP

Group has write permission.
@end deffn

@geindex S_IXGRP (in module stat)
@anchor{library/stat stat S_IXGRP}@anchor{20b9}
@deffn {Data} stat.S_IXGRP

Group has execute permission.
@end deffn

@geindex S_IRWXO (in module stat)
@anchor{library/stat stat S_IRWXO}@anchor{20c3}
@deffn {Data} stat.S_IRWXO

Mask for permissions for others (not in group).
@end deffn

@geindex S_IROTH (in module stat)
@anchor{library/stat stat S_IROTH}@anchor{20c4}
@deffn {Data} stat.S_IROTH

Others have read permission.
@end deffn

@geindex S_IWOTH (in module stat)
@anchor{library/stat stat S_IWOTH}@anchor{20c5}
@deffn {Data} stat.S_IWOTH

Others have write permission.
@end deffn

@geindex S_IXOTH (in module stat)
@anchor{library/stat stat S_IXOTH}@anchor{20c6}
@deffn {Data} stat.S_IXOTH

Others have execute permission.
@end deffn

@geindex S_ENFMT (in module stat)
@anchor{library/stat stat S_ENFMT}@anchor{20ba}
@deffn {Data} stat.S_ENFMT

System V file locking enforcement.  This flag is shared with @ref{20b8,,S_ISGID}:
file/record locking is enforced on files that do not have the group
execution bit (@ref{20b9,,S_IXGRP}) set.
@end deffn

@geindex S_IREAD (in module stat)
@anchor{library/stat stat S_IREAD}@anchor{20c7}
@deffn {Data} stat.S_IREAD

Unix V7 synonym for @ref{20bd,,S_IRUSR}.
@end deffn

@geindex S_IWRITE (in module stat)
@anchor{library/stat stat S_IWRITE}@anchor{20c8}
@deffn {Data} stat.S_IWRITE

Unix V7 synonym for @ref{20be,,S_IWUSR}.
@end deffn

@geindex S_IEXEC (in module stat)
@anchor{library/stat stat S_IEXEC}@anchor{20c9}
@deffn {Data} stat.S_IEXEC

Unix V7 synonym for @ref{20bf,,S_IXUSR}.
@end deffn

The following flags can be used in the @emph{flags} argument of @ref{c7f,,os.chflags()}:

@geindex UF_NODUMP (in module stat)
@anchor{library/stat stat UF_NODUMP}@anchor{20ca}
@deffn {Data} stat.UF_NODUMP

Do not dump the file.
@end deffn

@geindex UF_IMMUTABLE (in module stat)
@anchor{library/stat stat UF_IMMUTABLE}@anchor{20cb}
@deffn {Data} stat.UF_IMMUTABLE

The file may not be changed.
@end deffn

@geindex UF_APPEND (in module stat)
@anchor{library/stat stat UF_APPEND}@anchor{20cc}
@deffn {Data} stat.UF_APPEND

The file may only be appended to.
@end deffn

@geindex UF_OPAQUE (in module stat)
@anchor{library/stat stat UF_OPAQUE}@anchor{20cd}
@deffn {Data} stat.UF_OPAQUE

The directory is opaque when viewed through a union stack.
@end deffn

@geindex UF_NOUNLINK (in module stat)
@anchor{library/stat stat UF_NOUNLINK}@anchor{20ce}
@deffn {Data} stat.UF_NOUNLINK

The file may not be renamed or deleted.
@end deffn

@geindex UF_COMPRESSED (in module stat)
@anchor{library/stat stat UF_COMPRESSED}@anchor{20cf}
@deffn {Data} stat.UF_COMPRESSED

The file is stored compressed (macOS 10.6+).
@end deffn

@geindex UF_HIDDEN (in module stat)
@anchor{library/stat stat UF_HIDDEN}@anchor{20d0}
@deffn {Data} stat.UF_HIDDEN

The file should not be displayed in a GUI (macOS 10.5+).
@end deffn

@geindex SF_ARCHIVED (in module stat)
@anchor{library/stat stat SF_ARCHIVED}@anchor{20d1}
@deffn {Data} stat.SF_ARCHIVED

The file may be archived.
@end deffn

@geindex SF_IMMUTABLE (in module stat)
@anchor{library/stat stat SF_IMMUTABLE}@anchor{20d2}
@deffn {Data} stat.SF_IMMUTABLE

The file may not be changed.
@end deffn

@geindex SF_APPEND (in module stat)
@anchor{library/stat stat SF_APPEND}@anchor{20d3}
@deffn {Data} stat.SF_APPEND

The file may only be appended to.
@end deffn

@geindex SF_NOUNLINK (in module stat)
@anchor{library/stat stat SF_NOUNLINK}@anchor{20d4}
@deffn {Data} stat.SF_NOUNLINK

The file may not be renamed or deleted.
@end deffn

@geindex SF_SNAPSHOT (in module stat)
@anchor{library/stat stat SF_SNAPSHOT}@anchor{20d5}
@deffn {Data} stat.SF_SNAPSHOT

The file is a snapshot file.
@end deffn

See the *BSD or macOS systems man page @code{chflags(2)} for more information.

On Windows, the following file attribute constants are available for use when
testing bits in the @code{st_file_attributes} member returned by @ref{516,,os.stat()}.
See the Windows API documentation@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/gg258117.aspx}
for more detail on the meaning of these constants.

@geindex FILE_ATTRIBUTE_ARCHIVE (in module stat)
@geindex FILE_ATTRIBUTE_COMPRESSED (in module stat)
@geindex FILE_ATTRIBUTE_DEVICE (in module stat)
@geindex FILE_ATTRIBUTE_DIRECTORY (in module stat)
@geindex FILE_ATTRIBUTE_ENCRYPTED (in module stat)
@geindex FILE_ATTRIBUTE_HIDDEN (in module stat)
@geindex FILE_ATTRIBUTE_INTEGRITY_STREAM (in module stat)
@geindex FILE_ATTRIBUTE_NORMAL (in module stat)
@geindex FILE_ATTRIBUTE_NOT_CONTENT_INDEXED (in module stat)
@geindex FILE_ATTRIBUTE_NO_SCRUB_DATA (in module stat)
@geindex FILE_ATTRIBUTE_OFFLINE (in module stat)
@geindex FILE_ATTRIBUTE_READONLY (in module stat)
@geindex FILE_ATTRIBUTE_REPARSE_POINT (in module stat)
@geindex FILE_ATTRIBUTE_SPARSE_FILE (in module stat)
@geindex FILE_ATTRIBUTE_SYSTEM (in module stat)
@geindex FILE_ATTRIBUTE_TEMPORARY (in module stat)
@geindex FILE_ATTRIBUTE_VIRTUAL (in module stat)
@anchor{library/stat stat FILE_ATTRIBUTE_ARCHIVE}@anchor{20d6}
@deffn {Data} stat.FILE_ATTRIBUTE_ARCHIVE
@anchor{library/stat stat FILE_ATTRIBUTE_COMPRESSED}@anchor{20d7}
@deffnx {Data} stat.FILE_ATTRIBUTE_COMPRESSED
@anchor{library/stat stat FILE_ATTRIBUTE_DEVICE}@anchor{20d8}
@deffnx {Data} stat.FILE_ATTRIBUTE_DEVICE
@anchor{library/stat stat FILE_ATTRIBUTE_DIRECTORY}@anchor{20d9}
@deffnx {Data} stat.FILE_ATTRIBUTE_DIRECTORY
@anchor{library/stat stat FILE_ATTRIBUTE_ENCRYPTED}@anchor{20da}
@deffnx {Data} stat.FILE_ATTRIBUTE_ENCRYPTED
@anchor{library/stat stat FILE_ATTRIBUTE_HIDDEN}@anchor{20db}
@deffnx {Data} stat.FILE_ATTRIBUTE_HIDDEN
@anchor{library/stat stat FILE_ATTRIBUTE_INTEGRITY_STREAM}@anchor{20dc}
@deffnx {Data} stat.FILE_ATTRIBUTE_INTEGRITY_STREAM
@anchor{library/stat stat FILE_ATTRIBUTE_NORMAL}@anchor{20dd}
@deffnx {Data} stat.FILE_ATTRIBUTE_NORMAL
@anchor{library/stat stat FILE_ATTRIBUTE_NOT_CONTENT_INDEXED}@anchor{20de}
@deffnx {Data} stat.FILE_ATTRIBUTE_NOT_CONTENT_INDEXED
@anchor{library/stat stat FILE_ATTRIBUTE_NO_SCRUB_DATA}@anchor{20df}
@deffnx {Data} stat.FILE_ATTRIBUTE_NO_SCRUB_DATA
@anchor{library/stat stat FILE_ATTRIBUTE_OFFLINE}@anchor{20e0}
@deffnx {Data} stat.FILE_ATTRIBUTE_OFFLINE
@anchor{library/stat stat FILE_ATTRIBUTE_READONLY}@anchor{20e1}
@deffnx {Data} stat.FILE_ATTRIBUTE_READONLY
@anchor{library/stat stat FILE_ATTRIBUTE_REPARSE_POINT}@anchor{20e2}
@deffnx {Data} stat.FILE_ATTRIBUTE_REPARSE_POINT
@anchor{library/stat stat FILE_ATTRIBUTE_SPARSE_FILE}@anchor{20e3}
@deffnx {Data} stat.FILE_ATTRIBUTE_SPARSE_FILE
@anchor{library/stat stat FILE_ATTRIBUTE_SYSTEM}@anchor{20e4}
@deffnx {Data} stat.FILE_ATTRIBUTE_SYSTEM
@anchor{library/stat stat FILE_ATTRIBUTE_TEMPORARY}@anchor{20e5}
@deffnx {Data} stat.FILE_ATTRIBUTE_TEMPORARY
@anchor{library/stat stat FILE_ATTRIBUTE_VIRTUAL}@anchor{20e6}
@deffnx {Data} stat.FILE_ATTRIBUTE_VIRTUAL


New in version 3.5.

@end deffn

On Windows, the following constants are available for comparing against the
@code{st_reparse_tag} member returned by @ref{517,,os.lstat()}. These are well-known
constants, but are not an exhaustive list.

@geindex IO_REPARSE_TAG_SYMLINK (in module stat)
@geindex IO_REPARSE_TAG_MOUNT_POINT (in module stat)
@geindex IO_REPARSE_TAG_APPEXECLINK (in module stat)
@anchor{library/stat stat IO_REPARSE_TAG_SYMLINK}@anchor{20e7}
@deffn {Data} stat.IO_REPARSE_TAG_SYMLINK
@anchor{library/stat stat IO_REPARSE_TAG_MOUNT_POINT}@anchor{20e8}
@deffnx {Data} stat.IO_REPARSE_TAG_MOUNT_POINT
@anchor{library/stat stat IO_REPARSE_TAG_APPEXECLINK}@anchor{20e9}
@deffnx {Data} stat.IO_REPARSE_TAG_APPEXECLINK


New in version 3.8.

@end deffn

@node filecmp — File and Directory Comparisons,tempfile — Generate temporary files and directories,stat — Interpreting stat results,File and Directory Access
@anchor{library/filecmp doc}@anchor{20ea}@anchor{library/filecmp filecmp-file-and-directory-comparisons}@anchor{20eb}@anchor{library/filecmp module-filecmp}@anchor{7a}
@subsection @code{filecmp} — File and Directory Comparisons


@geindex module; filecmp

@strong{Source code:} Lib/filecmp.py@footnote{https://github.com/python/cpython/tree/main/Lib/filecmp.py}


__________________________________________________________________


The @ref{7a,,filecmp} module defines functions to compare files and directories,
with various optional time/correctness trade-offs. For comparing files,
see also the @ref{35,,difflib} module.

The @ref{7a,,filecmp} module defines the following functions:

@geindex cmp() (in module filecmp)
@anchor{library/filecmp filecmp cmp}@anchor{111b}
@deffn {Function} filecmp.cmp (f1, f2, shallow=True)

Compare the files named @emph{f1} and @emph{f2}, returning @code{True} if they seem equal,
@code{False} otherwise.

If @emph{shallow} is true and the @ref{516,,os.stat()} signatures (file type, size, and
modification time) of both files are identical, the files are taken to be
equal.

Otherwise, the files are treated as different if their sizes or contents differ.

Note that no external programs are called from this function, giving it
portability and efficiency.

This function uses a cache for past comparisons and the results,
with cache entries invalidated if the @ref{516,,os.stat()} information for the
file changes.  The entire cache may be cleared using @ref{abb,,clear_cache()}.
@end deffn

@geindex cmpfiles() (in module filecmp)
@anchor{library/filecmp filecmp cmpfiles}@anchor{20ec}
@deffn {Function} filecmp.cmpfiles (dir1, dir2, common, shallow=True)

Compare the files in the two directories @emph{dir1} and @emph{dir2} whose names are
given by @emph{common}.

Returns three lists of file names: @emph{match}, @emph{mismatch},
@emph{errors}.  @emph{match} contains the list of files that match, @emph{mismatch} contains
the names of those that don’t, and @emph{errors} lists the names of files which
could not be compared.  Files are listed in @emph{errors} if they don’t exist in
one of the directories, the user lacks permission to read them or if the
comparison could not be done for some other reason.

The @emph{shallow} parameter has the same meaning and default value as for
@ref{111b,,filecmp.cmp()}.

For example, @code{cmpfiles('a', 'b', ['c', 'd/e'])} will compare @code{a/c} with
@code{b/c} and @code{a/d/e} with @code{b/d/e}.  @code{'c'} and @code{'d/e'} will each be in
one of the three returned lists.
@end deffn

@geindex clear_cache() (in module filecmp)
@anchor{library/filecmp filecmp clear_cache}@anchor{abb}
@deffn {Function} filecmp.clear_cache ()

Clear the filecmp cache. This may be useful if a file is compared so quickly
after it is modified that it is within the mtime resolution of
the underlying filesystem.


New in version 3.4.

@end deffn

@menu
* The dircmp class:: 

@end menu

@node The dircmp class,,,filecmp — File and Directory Comparisons
@anchor{library/filecmp dircmp-objects}@anchor{20ed}@anchor{library/filecmp the-dircmp-class}@anchor{20ee}
@subsubsection The @code{dircmp} class


@geindex dircmp (class in filecmp)
@anchor{library/filecmp filecmp dircmp}@anchor{abd}
@deffn {Class} filecmp.dircmp (a, b, ignore=None, hide=None)

Construct a new directory comparison object, to compare the directories @emph{a}
and @emph{b}.  @emph{ignore} is a list of names to ignore, and defaults to
@ref{abc,,filecmp.DEFAULT_IGNORES}.  @emph{hide} is a list of names to hide, and
defaults to @code{[os.curdir, os.pardir]}.

The @ref{abd,,dircmp} class compares files by doing @emph{shallow} comparisons
as described for @ref{111b,,filecmp.cmp()}.

The @ref{abd,,dircmp} class provides the following methods:

@geindex report() (filecmp.dircmp method)
@anchor{library/filecmp filecmp dircmp report}@anchor{20ef}
@deffn {Method} report ()

Print (to @ref{5ef,,sys.stdout}) a comparison between @emph{a} and @emph{b}.
@end deffn

@geindex report_partial_closure() (filecmp.dircmp method)
@anchor{library/filecmp filecmp dircmp report_partial_closure}@anchor{20f0}
@deffn {Method} report_partial_closure ()

Print a comparison between @emph{a} and @emph{b} and common immediate
subdirectories.
@end deffn

@geindex report_full_closure() (filecmp.dircmp method)
@anchor{library/filecmp filecmp dircmp report_full_closure}@anchor{20f1}
@deffn {Method} report_full_closure ()

Print a comparison between @emph{a} and @emph{b} and common subdirectories
(recursively).
@end deffn

The @ref{abd,,dircmp} class offers a number of interesting attributes that may be
used to get various bits of information about the directory trees being
compared.

Note that via @code{__getattr__()} hooks, all attributes are computed lazily,
so there is no speed penalty if only those attributes which are lightweight
to compute are used.

@geindex left (filecmp.dircmp attribute)
@anchor{library/filecmp filecmp dircmp left}@anchor{20f2}
@deffn {Attribute} left

The directory @emph{a}.
@end deffn

@geindex right (filecmp.dircmp attribute)
@anchor{library/filecmp filecmp dircmp right}@anchor{20f3}
@deffn {Attribute} right

The directory @emph{b}.
@end deffn

@geindex left_list (filecmp.dircmp attribute)
@anchor{library/filecmp filecmp dircmp left_list}@anchor{20f4}
@deffn {Attribute} left_list

Files and subdirectories in @emph{a}, filtered by @emph{hide} and @emph{ignore}.
@end deffn

@geindex right_list (filecmp.dircmp attribute)
@anchor{library/filecmp filecmp dircmp right_list}@anchor{20f5}
@deffn {Attribute} right_list

Files and subdirectories in @emph{b}, filtered by @emph{hide} and @emph{ignore}.
@end deffn

@geindex common (filecmp.dircmp attribute)
@anchor{library/filecmp filecmp dircmp common}@anchor{20f6}
@deffn {Attribute} common

Files and subdirectories in both @emph{a} and @emph{b}.
@end deffn

@geindex left_only (filecmp.dircmp attribute)
@anchor{library/filecmp filecmp dircmp left_only}@anchor{20f7}
@deffn {Attribute} left_only

Files and subdirectories only in @emph{a}.
@end deffn

@geindex right_only (filecmp.dircmp attribute)
@anchor{library/filecmp filecmp dircmp right_only}@anchor{20f8}
@deffn {Attribute} right_only

Files and subdirectories only in @emph{b}.
@end deffn

@geindex common_dirs (filecmp.dircmp attribute)
@anchor{library/filecmp filecmp dircmp common_dirs}@anchor{20f9}
@deffn {Attribute} common_dirs

Subdirectories in both @emph{a} and @emph{b}.
@end deffn

@geindex common_files (filecmp.dircmp attribute)
@anchor{library/filecmp filecmp dircmp common_files}@anchor{20fa}
@deffn {Attribute} common_files

Files in both @emph{a} and @emph{b}.
@end deffn

@geindex common_funny (filecmp.dircmp attribute)
@anchor{library/filecmp filecmp dircmp common_funny}@anchor{20fb}
@deffn {Attribute} common_funny

Names in both @emph{a} and @emph{b}, such that the type differs between the
directories, or names for which @ref{516,,os.stat()} reports an error.
@end deffn

@geindex same_files (filecmp.dircmp attribute)
@anchor{library/filecmp filecmp dircmp same_files}@anchor{20fc}
@deffn {Attribute} same_files

Files which are identical in both @emph{a} and @emph{b}, using the class’s
file comparison operator.
@end deffn

@geindex diff_files (filecmp.dircmp attribute)
@anchor{library/filecmp filecmp dircmp diff_files}@anchor{20fd}
@deffn {Attribute} diff_files

Files which are in both @emph{a} and @emph{b}, whose contents differ according
to the class’s file comparison operator.
@end deffn

@geindex funny_files (filecmp.dircmp attribute)
@anchor{library/filecmp filecmp dircmp funny_files}@anchor{20fe}
@deffn {Attribute} funny_files

Files which are in both @emph{a} and @emph{b}, but could not be compared.
@end deffn

@geindex subdirs (filecmp.dircmp attribute)
@anchor{library/filecmp filecmp dircmp subdirs}@anchor{1195}
@deffn {Attribute} subdirs

A dictionary mapping names in @ref{20f9,,common_dirs} to @ref{abd,,dircmp}
instances (or MyDirCmp instances if this instance is of type MyDirCmp, a
subclass of @ref{abd,,dircmp}).


Changed in version 3.10: Previously entries were always @ref{abd,,dircmp} instances. Now entries
are the same type as @emph{self}, if @emph{self} is a subclass of
@ref{abd,,dircmp}.

@end deffn
@end deffn

@geindex DEFAULT_IGNORES (in module filecmp)
@anchor{library/filecmp filecmp DEFAULT_IGNORES}
@deffn {Attribute} filecmp.DEFAULT_IGNORES


New in version 3.4.


List of directories ignored by @ref{abd,,dircmp} by default.
@end deffn

Here is a simplified example of using the @code{subdirs} attribute to search
recursively through two directories to show common different files:

@example
>>> from filecmp import dircmp
>>> def print_diff_files(dcmp):
...     for name in dcmp.diff_files:
...         print("diff_file %s found in %s and %s" % (name, dcmp.left,
...               dcmp.right))
...     for sub_dcmp in dcmp.subdirs.values():
...         print_diff_files(sub_dcmp)
...
>>> dcmp = dircmp('dir1', 'dir2') 
>>> print_diff_files(dcmp) 
@end example

@node tempfile — Generate temporary files and directories,glob — Unix style pathname pattern expansion,filecmp — File and Directory Comparisons,File and Directory Access
@anchor{library/tempfile doc}@anchor{20ff}@anchor{library/tempfile module-tempfile}@anchor{fd}@anchor{library/tempfile tempfile-generate-temporary-files-and-directories}@anchor{2100}
@subsection @code{tempfile} — Generate temporary files and directories


@geindex module; tempfile

@strong{Source code:} Lib/tempfile.py@footnote{https://github.com/python/cpython/tree/main/Lib/tempfile.py}

@geindex temporary; file name
@geindex temporary; file


__________________________________________________________________


This module creates temporary files and directories.  It works on all
supported platforms. @ref{2101,,TemporaryFile}, @ref{f44,,NamedTemporaryFile},
@ref{de5,,TemporaryDirectory}, and @ref{cec,,SpooledTemporaryFile} are high-level
interfaces which provide automatic cleanup and can be used as
context managers. @ref{2102,,mkstemp()} and
@ref{2103,,mkdtemp()} are lower-level functions which require manual cleanup.

All the user-callable functions and constructors take additional arguments which
allow direct control over the location and name of temporary files and
directories. Files names used by this module include a string of
random characters which allows those files to be securely created in
shared temporary directories.
To maintain backward compatibility, the argument order is somewhat odd; it
is recommended to use keyword arguments for clarity.

The module defines the following user-callable items:

@geindex TemporaryFile() (in module tempfile)
@anchor{library/tempfile tempfile TemporaryFile}@anchor{2101}
@deffn {Function} tempfile.TemporaryFile (mode='w+b', buffering=- 1, encoding=None, newline=None, suffix=None, prefix=None, dir=None, *, errors=None)

Return a @ref{1e54,,file-like object} that can be used as a temporary storage area.
The file is created securely, using the same rules as @ref{2102,,mkstemp()}. It will be destroyed as soon
as it is closed (including an implicit close when the object is garbage
collected).  Under Unix, the directory entry for the file is either not created at all or is removed
immediately after the file is created.  Other platforms do not support
this; your code should not rely on a temporary file created using this
function having or not having a visible name in the file system.

The resulting object can be used as a context manager (see
@ref{2104,,Examples}).  On completion of the context or
destruction of the file object the temporary file will be removed
from the filesystem.

The @emph{mode} parameter defaults to @code{'w+b'} so that the file created can
be read and written without being closed.  Binary mode is used so that it
behaves consistently on all platforms without regard for the data that is
stored.  @emph{buffering}, @emph{encoding}, @emph{errors} and @emph{newline} are interpreted as for
@ref{1c8,,open()}.

The @emph{dir}, @emph{prefix} and @emph{suffix} parameters have the same meaning and
defaults as with @ref{2102,,mkstemp()}.

The returned object is a true file object on POSIX platforms.  On other
platforms, it is a file-like object whose @code{file} attribute is the
underlying true file object.

The @ref{af4,,os.O_TMPFILE} flag is used if it is available and works
(Linux-specific, requires Linux kernel 3.11 or later).

On platforms that are neither Posix nor Cygwin, TemporaryFile is an alias
for NamedTemporaryFile.

Raises an @ref{1130,,auditing event} @code{tempfile.mkstemp} with argument @code{fullpath}.


Changed in version 3.5: The @ref{af4,,os.O_TMPFILE} flag is now used if available.



Changed in version 3.8: Added @emph{errors} parameter.

@end deffn

@geindex NamedTemporaryFile() (in module tempfile)
@anchor{library/tempfile tempfile NamedTemporaryFile}@anchor{f44}
@deffn {Function} tempfile.NamedTemporaryFile (mode='w+b', buffering=- 1, encoding=None, newline=None, suffix=None, prefix=None, dir=None, delete=True, *, errors=None)

This function operates exactly as @ref{2101,,TemporaryFile()} does, except that
the file is guaranteed to have a visible name in the file system (on
Unix, the directory entry is not unlinked).  That name can be retrieved
from the @code{name} attribute of the returned
file-like object.  Whether the name can be
used to open the file a second time, while the named temporary file is
still open, varies across platforms (it can be so used on Unix; it cannot
on Windows NT or later).  If @emph{delete} is true (the default), the file is
deleted as soon as it is closed.
The returned object is always a file-like object whose @code{file}
attribute is the underlying true file object. This file-like object can
be used in a @ref{168,,with} statement, just like a normal file.

On POSIX (only), a process that is terminated abruptly with SIGKILL
cannot automatically delete any NamedTemporaryFiles it created.

Raises an @ref{1130,,auditing event} @code{tempfile.mkstemp} with argument @code{fullpath}.


Changed in version 3.8: Added @emph{errors} parameter.

@end deffn

@geindex SpooledTemporaryFile() (in module tempfile)
@anchor{library/tempfile tempfile SpooledTemporaryFile}@anchor{cec}
@deffn {Function} tempfile.SpooledTemporaryFile (max_size=0, mode='w+b', buffering=- 1, encoding=None, newline=None, suffix=None, prefix=None, dir=None, *, errors=None)

This function operates exactly as @ref{2101,,TemporaryFile()} does, except that
data is spooled in memory until the file size exceeds @emph{max_size}, or
until the file’s @code{fileno()} method is called, at which point the
contents are written to disk and operation proceeds as with
@ref{2101,,TemporaryFile()}.

The resulting file has one additional method, @code{rollover()}, which
causes the file to roll over to an on-disk file regardless of its size.

The returned object is a file-like object whose @code{_file} attribute
is either an @ref{a15,,io.BytesIO} or @ref{889,,io.TextIOWrapper} object
(depending on whether binary or text @emph{mode} was specified) or a true file
object, depending on whether @code{rollover()} has been called.  This
file-like object can be used in a @ref{168,,with} statement, just like
a normal file.


Changed in version 3.3: the truncate method now accepts a @code{size} argument.



Changed in version 3.8: Added @emph{errors} parameter.

@end deffn

@geindex TemporaryDirectory() (in module tempfile)
@anchor{library/tempfile tempfile TemporaryDirectory}@anchor{de5}
@deffn {Function} tempfile.TemporaryDirectory (suffix=None, prefix=None, dir=None, ignore_cleanup_errors=False)

This function securely creates a temporary directory using the same rules as @ref{2103,,mkdtemp()}.
The resulting object can be used as a context manager (see
@ref{2104,,Examples}).  On completion of the context or destruction
of the temporary directory object, the newly created temporary directory
and all its contents are removed from the filesystem.

The directory name can be retrieved from the @code{name} attribute of the
returned object.  When the returned object is used as a context manager, the
@code{name} will be assigned to the target of the @code{as} clause in
the @ref{168,,with} statement, if there is one.

The directory can be explicitly cleaned up by calling the
@code{cleanup()} method. If @emph{ignore_cleanup_errors} is true, any unhandled
exceptions during explicit or implicit cleanup (such as a
@ref{895,,PermissionError} removing open files on Windows) will be ignored,
and the remaining removable items deleted on a “best-effort” basis.
Otherwise, errors will be raised in whatever context cleanup occurs
(the @code{cleanup()} call, exiting the context manager, when the object
is garbage-collected or during interpreter shutdown).

Raises an @ref{1130,,auditing event} @code{tempfile.mkdtemp} with argument @code{fullpath}.


New in version 3.2.



Changed in version 3.10: Added @emph{ignore_cleanup_errors} parameter.

@end deffn

@geindex mkstemp() (in module tempfile)
@anchor{library/tempfile tempfile mkstemp}@anchor{2102}
@deffn {Function} tempfile.mkstemp (suffix=None, prefix=None, dir=None, text=False)

Creates a temporary file in the most secure manner possible.  There are
no race conditions in the file’s creation, assuming that the platform
properly implements the @ref{2105,,os.O_EXCL} flag for @ref{8ee,,os.open()}.  The
file is readable and writable only by the creating user ID.  If the
platform uses permission bits to indicate whether a file is executable,
the file is executable by no one.  The file descriptor is not inherited
by child processes.

Unlike @ref{2101,,TemporaryFile()}, the user of @ref{2102,,mkstemp()} is responsible
for deleting the temporary file when done with it.

If @emph{suffix} is not @code{None}, the file name will end with that suffix,
otherwise there will be no suffix.  @ref{2102,,mkstemp()} does not put a dot
between the file name and the suffix; if you need one, put it at the
beginning of @emph{suffix}.

If @emph{prefix} is not @code{None}, the file name will begin with that prefix;
otherwise, a default prefix is used.  The default is the return value of
@ref{2106,,gettempprefix()} or @ref{2107,,gettempprefixb()}, as appropriate.

If @emph{dir} is not @code{None}, the file will be created in that directory;
otherwise, a default directory is used.  The default directory is chosen
from a platform-dependent list, but the user of the application can
control the directory location by setting the @emph{TMPDIR}, @emph{TEMP} or @emph{TMP}
environment variables.  There is thus no guarantee that the generated
filename will have any nice properties, such as not requiring quoting
when passed to external commands via @code{os.popen()}.

If any of @emph{suffix}, @emph{prefix}, and @emph{dir} are not
@code{None}, they must be the same type.
If they are bytes, the returned name will be bytes instead of str.
If you want to force a bytes return value with otherwise default behavior,
pass @code{suffix=b'@w{'}}.

If @emph{text} is specified and true, the file is opened in text mode.
Otherwise, (the default) the file is opened in binary mode.

@ref{2102,,mkstemp()} returns a tuple containing an OS-level handle to an open
file (as would be returned by @ref{8ee,,os.open()}) and the absolute pathname
of that file, in that order.

Raises an @ref{1130,,auditing event} @code{tempfile.mkstemp} with argument @code{fullpath}.


Changed in version 3.5: @emph{suffix}, @emph{prefix}, and @emph{dir} may now be supplied in bytes in order to
obtain a bytes return value.  Prior to this, only str was allowed.
@emph{suffix} and @emph{prefix} now accept and default to @code{None} to cause
an appropriate default value to be used.



Changed in version 3.6: The @emph{dir} parameter now accepts a @ref{63d,,path-like object}.

@end deffn

@geindex mkdtemp() (in module tempfile)
@anchor{library/tempfile tempfile mkdtemp}@anchor{2103}
@deffn {Function} tempfile.mkdtemp (suffix=None, prefix=None, dir=None)

Creates a temporary directory in the most secure manner possible. There
are no race conditions in the directory’s creation.  The directory is
readable, writable, and searchable only by the creating user ID.

The user of @ref{2103,,mkdtemp()} is responsible for deleting the temporary
directory and its contents when done with it.

The @emph{prefix}, @emph{suffix}, and @emph{dir} arguments are the same as for
@ref{2102,,mkstemp()}.

@ref{2103,,mkdtemp()} returns the absolute pathname of the new directory.

Raises an @ref{1130,,auditing event} @code{tempfile.mkdtemp} with argument @code{fullpath}.


Changed in version 3.5: @emph{suffix}, @emph{prefix}, and @emph{dir} may now be supplied in bytes in order to
obtain a bytes return value.  Prior to this, only str was allowed.
@emph{suffix} and @emph{prefix} now accept and default to @code{None} to cause
an appropriate default value to be used.



Changed in version 3.6: The @emph{dir} parameter now accepts a @ref{63d,,path-like object}.

@end deffn

@geindex gettempdir() (in module tempfile)
@anchor{library/tempfile tempfile gettempdir}@anchor{1153}
@deffn {Function} tempfile.gettempdir ()

Return the name of the directory used for temporary files. This
defines the default value for the @emph{dir} argument to all functions
in this module.

Python searches a standard list of directories to find one which
the calling user can create files in.  The list is:


@enumerate 

@item 
The directory named by the 
@geindex TMPDIR
@geindex environment variable; TMPDIR
@code{TMPDIR} environment variable.

@item 
The directory named by the 
@geindex TEMP
@geindex environment variable; TEMP
@code{TEMP} environment variable.

@item 
The directory named by the 
@geindex TMP
@geindex environment variable; TMP
@code{TMP} environment variable.

@item 
A platform-specific location:


@itemize *

@item 
On Windows, the directories @code{C:\TEMP}, @code{C:\TMP},
@code{\TEMP}, and @code{\TMP}, in that order.

@item 
On all other platforms, the directories @code{/tmp}, @code{/var/tmp}, and
@code{/usr/tmp}, in that order.
@end itemize

@item 
As a last resort, the current working directory.
@end enumerate

The result of this search is cached, see the description of
@ref{1152,,tempdir} below.


Changed in version 3.10: Always returns a str.  Previously it would return any @ref{1152,,tempdir}
value regardless of type so long as it was not @code{None}.

@end deffn

@geindex gettempdirb() (in module tempfile)
@anchor{library/tempfile tempfile gettempdirb}@anchor{1154}
@deffn {Function} tempfile.gettempdirb ()

Same as @ref{1153,,gettempdir()} but the return value is in bytes.


New in version 3.5.

@end deffn

@geindex gettempprefix() (in module tempfile)
@anchor{library/tempfile tempfile gettempprefix}@anchor{2106}
@deffn {Function} tempfile.gettempprefix ()

Return the filename prefix used to create temporary files.  This does not
contain the directory component.
@end deffn

@geindex gettempprefixb() (in module tempfile)
@anchor{library/tempfile tempfile gettempprefixb}@anchor{2107}
@deffn {Function} tempfile.gettempprefixb ()

Same as @ref{2106,,gettempprefix()} but the return value is in bytes.


New in version 3.5.

@end deffn

The module uses a global variable to store the name of the directory
used for temporary files returned by @ref{1153,,gettempdir()}.  It can be
set directly to override the selection process, but this is discouraged.
All functions in this module take a @emph{dir} argument which can be used
to specify the directory. This is the recommended approach that does
not surprise other unsuspecting code by changing global API behavior.

@geindex tempdir (in module tempfile)
@anchor{library/tempfile tempfile tempdir}@anchor{1152}
@deffn {Data} tempfile.tempdir

When set to a value other than @code{None}, this variable defines the
default value for the @emph{dir} argument to the functions defined in this
module, including its type, bytes or str.  It cannot be a
@ref{63d,,path-like object}.

If @code{tempdir} is @code{None} (the default) at any call to any of the above
functions except @ref{2106,,gettempprefix()} it is initialized following the
algorithm described in @ref{1153,,gettempdir()}.

@cartouche
@quotation Note 
Beware that if you set @code{tempdir} to a bytes value, there is a
nasty side effect: The global default return type of
@ref{2102,,mkstemp()} and @ref{2103,,mkdtemp()} changes to bytes when no
explicit @code{prefix}, @code{suffix}, or @code{dir} arguments of type
str are supplied. Please do not write code expecting or
depending on this. This awkward behavior is maintained for
compatibility with the historical implementation.
@end quotation
@end cartouche
@end deffn

@menu
* Examples: Examples<5>. 
* Deprecated functions and variables:: 

@end menu

@node Examples<5>,Deprecated functions and variables,,tempfile — Generate temporary files and directories
@anchor{library/tempfile examples}@anchor{2108}@anchor{library/tempfile tempfile-examples}@anchor{2104}
@subsubsection Examples


Here are some examples of typical usage of the @ref{fd,,tempfile} module:

@example
>>> import tempfile

# create a temporary file and write some data to it
>>> fp = tempfile.TemporaryFile()
>>> fp.write(b'Hello world!')
# read data from file
>>> fp.seek(0)
>>> fp.read()
b'Hello world!'
# close the file, it will be removed
>>> fp.close()

# create a temporary file using a context manager
>>> with tempfile.TemporaryFile() as fp:
...     fp.write(b'Hello world!')
...     fp.seek(0)
...     fp.read()
b'Hello world!'
>>>
# file is now closed and removed

# create a temporary directory using the context manager
>>> with tempfile.TemporaryDirectory() as tmpdirname:
...     print('created temporary directory', tmpdirname)
>>>
# directory and contents have been removed
@end example

@node Deprecated functions and variables,,Examples<5>,tempfile — Generate temporary files and directories
@anchor{library/tempfile deprecated-functions-and-variables}@anchor{2109}@anchor{library/tempfile tempfile-mktemp-deprecated}@anchor{210a}
@subsubsection Deprecated functions and variables


A historical way to create temporary files was to first generate a
file name with the @ref{10f2,,mktemp()} function and then create a file
using this name. Unfortunately this is not secure, because a different
process may create a file with this name in the time between the call
to @ref{10f2,,mktemp()} and the subsequent attempt to create the file by the
first process. The solution is to combine the two steps and create the
file immediately. This approach is used by @ref{2102,,mkstemp()} and the
other functions described above.

@geindex mktemp() (in module tempfile)
@anchor{library/tempfile tempfile mktemp}@anchor{10f2}
@deffn {Function} tempfile.mktemp (suffix='@w{'}, prefix='tmp', dir=None)


Deprecated since version 2.3: Use @ref{2102,,mkstemp()} instead.


Return an absolute pathname of a file that did not exist at the time the
call is made.  The @emph{prefix}, @emph{suffix}, and @emph{dir} arguments are similar
to those of @ref{2102,,mkstemp()}, except that bytes file names, @code{suffix=None}
and @code{prefix=None} are not supported.

@cartouche
@quotation Warning 
Use of this function may introduce a security hole in your program.  By
the time you get around to doing anything with the file name it returns,
someone else may have beaten you to the punch.  @ref{10f2,,mktemp()} usage can
be replaced easily with @ref{f44,,NamedTemporaryFile()}, passing it the
@code{delete=False} parameter:

@example
>>> f = NamedTemporaryFile(delete=False)
>>> f.name
'/tmp/tmptjujjt'
>>> f.write(b"Hello World!\n")
13
>>> f.close()
>>> os.unlink(f.name)
>>> os.path.exists(f.name)
False
@end example
@end quotation
@end cartouche
@end deffn

@node glob — Unix style pathname pattern expansion,fnmatch — Unix filename pattern matching,tempfile — Generate temporary files and directories,File and Directory Access
@anchor{library/glob doc}@anchor{210b}@anchor{library/glob glob-unix-style-pathname-pattern-expansion}@anchor{210c}@anchor{library/glob module-glob}@anchor{84}
@subsection @code{glob} — Unix style pathname pattern expansion


@geindex module; glob

@strong{Source code:} Lib/glob.py@footnote{https://github.com/python/cpython/tree/main/Lib/glob.py}

@geindex filenames; pathname expansion


__________________________________________________________________


@geindex * (asterisk); in glob-style wildcards
@geindex ? (question mark); in glob-style wildcards
@geindex [] (square brackets); in glob-style wildcards
@geindex ! (exclamation); in glob-style wildcards
@geindex - (minus); in glob-style wildcards
@geindex . (dot); in glob-style wildcards

The @ref{84,,glob} module finds all the pathnames matching a specified pattern
according to the rules used by the Unix shell, although results are returned in
arbitrary order.  No tilde expansion is done, but @code{*}, @code{?}, and character
ranges expressed with @code{[]} will be correctly matched.  This is done by using
the @ref{576,,os.scandir()} and @ref{1106,,fnmatch.fnmatch()} functions in concert, and
not by actually invoking a subshell.  Note that unlike @ref{1106,,fnmatch.fnmatch()},
@ref{84,,glob} treats filenames beginning with a dot (@code{.}) as special cases.
(For tilde and shell variable expansion, use @ref{520,,os.path.expanduser()} and
@ref{f3f,,os.path.expandvars()}.)

For a literal match, wrap the meta-characters in brackets.
For example, @code{'[?]'} matches the character @code{'?'}.


@subsubheading See also


The @ref{c3,,pathlib} module offers high-level path objects.


@geindex glob() (in module glob)
@anchor{library/glob glob glob}@anchor{288}
@deffn {Function} glob.glob (pathname, *, root_dir=None, dir_fd=None, recursive=False, include_hidden=False)

Return a possibly-empty list of path names that match @emph{pathname}, which must be
a string containing a path specification. @emph{pathname} can be either absolute
(like @code{/usr/src/Python-1.5/Makefile}) or relative (like
@code{../../Tools/*/*.gif}), and can contain shell-style wildcards. Broken
symlinks are included in the results (as in the shell). Whether or not the
results are sorted depends on the file system.  If a file that satisfies
conditions is removed or added during the call of this function, whether
a path name for that file be included is unspecified.

If @emph{root_dir} is not @code{None}, it should be a @ref{63d,,path-like object}
specifying the root directory for searching.  It has the same effect on
@ref{84,,glob()} as changing the current directory before calling it.  If
@emph{pathname} is relative, the result will contain paths relative to
@emph{root_dir}.

This function can support @ref{c7c,,paths relative to directory descriptors} with the @emph{dir_fd} parameter.

@geindex **; in glob-style wildcards

If @emph{recursive} is true, the pattern “@code{**}” will match any files and zero or
more directories, subdirectories and symbolic links to directories. If the
pattern is followed by an @ref{210d,,os.sep} or @ref{210e,,os.altsep} then files will not
match.

If @emph{include_hidden} is true, “@code{**}” pattern will match hidden directories.

Raises an @ref{1130,,auditing event} @code{glob.glob} with arguments @code{pathname}, @code{recursive}.

Raises an @ref{1130,,auditing event} @code{glob.glob/2} with arguments @code{pathname}, @code{recursive}, @code{root_dir}, @code{dir_fd}.

@cartouche
@quotation Note 
Using the “@code{**}” pattern in large directory trees may consume
an inordinate amount of time.
@end quotation
@end cartouche


Changed in version 3.5: Support for recursive globs using “@code{**}”.



Changed in version 3.10: Added the @emph{root_dir} and @emph{dir_fd} parameters.



Changed in version 3.11: Added the @emph{include_hidden} parameter.

@end deffn

@geindex iglob() (in module glob)
@anchor{library/glob glob iglob}@anchor{289}
@deffn {Function} glob.iglob (pathname, *, root_dir=None, dir_fd=None, recursive=False, include_hidden=False)

Return an @ref{11c3,,iterator} which yields the same values as @ref{84,,glob()}
without actually storing them all simultaneously.

Raises an @ref{1130,,auditing event} @code{glob.glob} with arguments @code{pathname}, @code{recursive}.

Raises an @ref{1130,,auditing event} @code{glob.glob/2} with arguments @code{pathname}, @code{recursive}, @code{root_dir}, @code{dir_fd}.


Changed in version 3.5: Support for recursive globs using “@code{**}”.



Changed in version 3.10: Added the @emph{root_dir} and @emph{dir_fd} parameters.



Changed in version 3.11: Added the @emph{include_hidden} parameter.

@end deffn

@geindex escape() (in module glob)
@anchor{library/glob glob escape}@anchor{ac3}
@deffn {Function} glob.escape (pathname)

Escape all special characters (@code{'?'}, @code{'*'} and @code{'['}).
This is useful if you want to match an arbitrary literal string that may
have special characters in it.  Special characters in drive/UNC
sharepoints are not escaped, e.g. on Windows
@code{escape('//?/c:/Quo vadis?.txt')} returns @code{'//?/c:/Quo vadis[?].txt'}.


New in version 3.4.

@end deffn

For example, consider a directory containing the following files:
@code{1.gif}, @code{2.txt}, @code{card.gif} and a subdirectory @code{sub}
which contains only the file @code{3.txt}.  @ref{84,,glob()} will produce
the following results.  Notice how any leading components of the path are
preserved.

@example
>>> import glob
>>> glob.glob('./[0-9].*')
['./1.gif', './2.txt']
>>> glob.glob('*.gif')
['1.gif', 'card.gif']
>>> glob.glob('?.gif')
['1.gif']
>>> glob.glob('**/*.txt', recursive=True)
['2.txt', 'sub/3.txt']
>>> glob.glob('./**/', recursive=True)
['./', './sub/']
@end example

If the directory contains files starting with @code{.} they won’t be matched by
default. For example, consider a directory containing @code{card.gif} and
@code{.card.gif}:

@example
>>> import glob
>>> glob.glob('*.gif')
['card.gif']
>>> glob.glob('.c*')
['.card.gif']
@end example


@subsubheading See also



@table @asis

@item Module @ref{7c,,fnmatch}

Shell-style filename (not path) expansion
@end table


@node fnmatch — Unix filename pattern matching,linecache — Random access to text lines,glob — Unix style pathname pattern expansion,File and Directory Access
@anchor{library/fnmatch doc}@anchor{210f}@anchor{library/fnmatch fnmatch-unix-filename-pattern-matching}@anchor{2110}@anchor{library/fnmatch module-fnmatch}@anchor{7c}
@subsection @code{fnmatch} — Unix filename pattern matching


@geindex module; fnmatch

@strong{Source code:} Lib/fnmatch.py@footnote{https://github.com/python/cpython/tree/main/Lib/fnmatch.py}

@geindex filenames; wildcard expansion

@geindex module; re


__________________________________________________________________


This module provides support for Unix shell-style wildcards, which are @emph{not} the
same as regular expressions (which are documented in the @ref{d8,,re} module).  The
special characters used in shell-style wildcards are:

@geindex * (asterisk); in glob-style wildcards
@geindex ? (question mark); in glob-style wildcards
@geindex [] (square brackets); in glob-style wildcards
@geindex ! (exclamation); in glob-style wildcards
@geindex - (minus); in glob-style wildcards


@multitable {xxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Pattern

@tab

Meaning

@item

@code{*}

@tab

matches everything

@item

@code{?}

@tab

matches any single character

@item

@code{[seq]}

@tab

matches any character in @emph{seq}

@item

@code{[!seq]}

@tab

matches any character not in @emph{seq}

@end multitable


For a literal match, wrap the meta-characters in brackets.
For example, @code{'[?]'} matches the character @code{'?'}.

@geindex module; glob

Note that the filename separator (@code{'/'} on Unix) is @emph{not} special to this
module.  See module @ref{84,,glob} for pathname expansion (@ref{84,,glob} uses
@ref{1108,,filter()} to match pathname segments).  Similarly, filenames starting with
a period are not special for this module, and are matched by the @code{*} and @code{?}
patterns.

Also note that @ref{4f3,,functools.lru_cache()} with the @emph{maxsize} of 32768 is used to
cache the compiled regex patterns in the following functions: @ref{7c,,fnmatch()},
@ref{1107,,fnmatchcase()}, @ref{2f5,,filter()}.

@geindex fnmatch() (in module fnmatch)
@anchor{library/fnmatch fnmatch fnmatch}@anchor{1106}
@deffn {Function} fnmatch.fnmatch (filename, pattern)

Test whether the @emph{filename} string matches the @emph{pattern} string, returning
@ref{748,,True} or @ref{659,,False}.  Both parameters are case-normalized
using @ref{208e,,os.path.normcase()}. @ref{1107,,fnmatchcase()} can be used to perform a
case-sensitive comparison, regardless of whether that’s standard for the
operating system.

This example will print all file names in the current directory with the
extension @code{.txt}:

@example
import fnmatch
import os

for file in os.listdir('.'):
    if fnmatch.fnmatch(file, '*.txt'):
        print(file)
@end example
@end deffn

@geindex fnmatchcase() (in module fnmatch)
@anchor{library/fnmatch fnmatch fnmatchcase}@anchor{1107}
@deffn {Function} fnmatch.fnmatchcase (filename, pattern)

Test whether @emph{filename} matches @emph{pattern}, returning @ref{748,,True} or
@ref{659,,False}; the comparison is case-sensitive and does not apply
@ref{208e,,os.path.normcase()}.
@end deffn

@geindex filter() (in module fnmatch)
@anchor{library/fnmatch fnmatch filter}@anchor{1108}
@deffn {Function} fnmatch.filter (names, pattern)

Construct a list from those elements of the iterable @emph{names} that match @emph{pattern}. It is the same as
@code{[n for n in names if fnmatch(n, pattern)]}, but implemented more efficiently.
@end deffn

@geindex translate() (in module fnmatch)
@anchor{library/fnmatch fnmatch translate}@anchor{1301}
@deffn {Function} fnmatch.translate (pattern)

Return the shell-style @emph{pattern} converted to a regular expression for
using with @ref{dd3,,re.match()}.

Example:

@example
>>> import fnmatch, re
>>>
>>> regex = fnmatch.translate('*.txt')
>>> regex
'(?s:.*\\.txt)\\Z'
>>> reobj = re.compile(regex)
>>> reobj.match('foobar.txt')
<re.Match object; span=(0, 10), match='foobar.txt'>
@end example
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{84,,glob}

Unix shell-style path expansion.
@end table


@node linecache — Random access to text lines,shutil — High-level file operations,fnmatch — Unix filename pattern matching,File and Directory Access
@anchor{library/linecache doc}@anchor{2111}@anchor{library/linecache linecache-random-access-to-text-lines}@anchor{2112}@anchor{library/linecache module-linecache}@anchor{a4}
@subsection @code{linecache} — Random access to text lines


@geindex module; linecache

@strong{Source code:} Lib/linecache.py@footnote{https://github.com/python/cpython/tree/main/Lib/linecache.py}


__________________________________________________________________


The @ref{a4,,linecache} module allows one to get any line from a Python source file, while
attempting to optimize internally, using a cache, the common case where many
lines are read from a single file.  This is used by the @ref{11a,,traceback} module
to retrieve source lines for inclusion in  the formatted traceback.

The @ref{2113,,tokenize.open()} function is used to open files. This
function uses @ref{2114,,tokenize.detect_encoding()} to get the encoding of the
file; in the absence of an encoding token, the file encoding defaults to UTF-8.

The @ref{a4,,linecache} module defines the following functions:

@geindex getline() (in module linecache)
@anchor{library/linecache linecache getline}@anchor{98a}
@deffn {Function} linecache.getline (filename, lineno, module_globals=None)

Get line @emph{lineno} from file named @emph{filename}. This function will never raise an
exception — it will return @code{'@w{'}} on errors (the terminating newline character
will be included for lines that are found).

@geindex module; search; path

If a file named @emph{filename} is not found, the function first checks
for a 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} @code{__loader__} in @emph{module_globals}.
If there is such a loader and it defines a @code{get_source} method,
then that determines the source lines
(if @code{get_source()} returns @code{None}, then @code{'@w{'}} is returned).
Finally, if @emph{filename} is a relative filename,
it is looked up relative to the entries in the module search path, @code{sys.path}.
@end deffn

@geindex clearcache() (in module linecache)
@anchor{library/linecache linecache clearcache}@anchor{2115}
@deffn {Function} linecache.clearcache ()

Clear the cache.  Use this function if you no longer need lines from files
previously read using @ref{98a,,getline()}.
@end deffn

@geindex checkcache() (in module linecache)
@anchor{library/linecache linecache checkcache}@anchor{2116}
@deffn {Function} linecache.checkcache (filename=None)

Check the cache for validity.  Use this function if files in the cache  may have
changed on disk, and you require the updated version.  If @emph{filename} is omitted,
it will check all the entries in the cache.
@end deffn

@geindex lazycache() (in module linecache)
@anchor{library/linecache linecache lazycache}@anchor{989}
@deffn {Function} linecache.lazycache (filename, module_globals)

Capture enough detail about a non-file-based module to permit getting its
lines later via @ref{98a,,getline()} even if @emph{module_globals} is @code{None} in the later
call. This avoids doing I/O until a line is actually needed, without having
to carry the module globals around indefinitely.


New in version 3.5.

@end deffn

Example:

@example
>>> import linecache
>>> linecache.getline(linecache.__file__, 8)
'import sys\n'
@end example

@node shutil — High-level file operations,,linecache — Random access to text lines,File and Directory Access
@anchor{library/shutil doc}@anchor{2117}@anchor{library/shutil module-shutil}@anchor{e4}@anchor{library/shutil shutil-high-level-file-operations}@anchor{2118}
@subsection @code{shutil} — High-level file operations


@geindex module; shutil

@c partly based on the docstrings

@strong{Source code:} Lib/shutil.py@footnote{https://github.com/python/cpython/tree/main/Lib/shutil.py}

@geindex file; copying
@geindex copying files


__________________________________________________________________


The @ref{e4,,shutil} module offers a number of high-level operations on files and
collections of files.  In particular, functions are provided  which support file
copying and removal. For operations on individual files, see also the
@ref{c0,,os} module.

@cartouche
@quotation Warning 
Even the higher-level file copying functions (@ref{570,,shutil.copy()},
@ref{571,,shutil.copy2()}) cannot copy all file metadata.

On POSIX platforms, this means that file owner and group are lost as well
as ACLs.  On Mac OS, the resource fork and other metadata are not used.
This means that resources will be lost and file type and creator codes will
not be correct. On Windows, file owners, ACLs and alternate data streams
are not copied.
@end quotation
@end cartouche

@menu
* Directory and files operations:: 
* Archiving operations:: 
* Querying the size of the output terminal:: 

@end menu

@node Directory and files operations,Archiving operations,,shutil — High-level file operations
@anchor{library/shutil directory-and-files-operations}@anchor{2119}@anchor{library/shutil file-operations}@anchor{211a}
@subsubsection Directory and files operations


@geindex copyfileobj() (in module shutil)
@anchor{library/shutil shutil copyfileobj}@anchor{574}
@deffn {Function} shutil.copyfileobj (fsrc, fdst[, length])

Copy the contents of the file-like object @emph{fsrc} to the file-like object @emph{fdst}.
The integer @emph{length}, if given, is the buffer size. In particular, a negative
@emph{length} value means to copy the data without looping over the source data in
chunks; by default the data is read in chunks to avoid uncontrolled memory
consumption. Note that if the current file position of the @emph{fsrc} object is not
0, only the contents from the current file position to the end of the file will
be copied.
@end deffn

@geindex copyfile() (in module shutil)
@anchor{library/shutil shutil copyfile}@anchor{56f}
@deffn {Function} shutil.copyfile (src, dst, *, follow_symlinks=True)

Copy the contents (no metadata) of the file named @emph{src} to a file named
@emph{dst} and return @emph{dst} in the most efficient way possible.
@emph{src} and @emph{dst} are path-like objects or path names given as strings.

@emph{dst} must be the complete target file name; look at @ref{570,,copy()}
for a copy that accepts a target directory path.  If @emph{src} and @emph{dst}
specify the same file, @ref{b1e,,SameFileError} is raised.

The destination location must be writable; otherwise, an @ref{2a1,,OSError}
exception will be raised. If @emph{dst} already exists, it will be replaced.
Special files such as character or block devices and pipes cannot be
copied with this function.

If @emph{follow_symlinks} is false and @emph{src} is a symbolic link,
a new symbolic link will be created instead of copying the
file @emph{src} points to.

Raises an @ref{1130,,auditing event} @code{shutil.copyfile} with arguments @code{src}, @code{dst}.


Changed in version 3.3: @ref{be8,,IOError} used to be raised instead of @ref{2a1,,OSError}.
Added @emph{follow_symlinks} argument.
Now returns @emph{dst}.



Changed in version 3.4: Raise @ref{b1e,,SameFileError} instead of @ref{b1d,,Error}.  Since the former is
a subclass of the latter, this change is backward compatible.



Changed in version 3.8: Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
@ref{575,,Platform-dependent efficient copy operations} section.

@end deffn

@geindex SameFileError
@anchor{library/shutil shutil SameFileError}@anchor{b1e}
@deffn {Exception} shutil.SameFileError

This exception is raised if source and destination in @ref{56f,,copyfile()}
are the same file.


New in version 3.4.

@end deffn

@geindex copymode() (in module shutil)
@anchor{library/shutil shutil copymode}@anchor{211b}
@deffn {Function} shutil.copymode (src, dst, *, follow_symlinks=True)

Copy the permission bits from @emph{src} to @emph{dst}.  The file contents, owner, and
group are unaffected.  @emph{src} and @emph{dst} are path-like objects or path names
given as strings.
If @emph{follow_symlinks} is false, and both @emph{src} and @emph{dst} are symbolic links,
@ref{211b,,copymode()} will attempt to modify the mode of @emph{dst} itself (rather
than the file it points to).  This functionality is not available on every
platform; please see @ref{cc3,,copystat()} for more information.  If
@ref{211b,,copymode()} cannot modify symbolic links on the local platform, and it
is asked to do so, it will do nothing and return.

Raises an @ref{1130,,auditing event} @code{shutil.copymode} with arguments @code{src}, @code{dst}.


Changed in version 3.3: Added @emph{follow_symlinks} argument.

@end deffn

@geindex copystat() (in module shutil)
@anchor{library/shutil shutil copystat}@anchor{cc3}
@deffn {Function} shutil.copystat (src, dst, *, follow_symlinks=True)

Copy the permission bits, last access time, last modification time, and
flags from @emph{src} to @emph{dst}.  On Linux, @ref{cc3,,copystat()} also copies the
“extended attributes” where possible.  The file contents, owner, and
group are unaffected.  @emph{src} and @emph{dst} are path-like objects or path
names given as strings.

If @emph{follow_symlinks} is false, and @emph{src} and @emph{dst} both
refer to symbolic links, @ref{cc3,,copystat()} will operate on
the symbolic links themselves rather than the files the
symbolic links refer to—reading the information from the
@emph{src} symbolic link, and writing the information to the
@emph{dst} symbolic link.

@cartouche
@quotation Note 
Not all platforms provide the ability to examine and
modify symbolic links.  Python itself can tell you what
functionality is locally available.


@itemize *

@item 
If @code{os.chmod in os.supports_follow_symlinks} is
@code{True}, @ref{cc3,,copystat()} can modify the permission
bits of a symbolic link.

@item 
If @code{os.utime in os.supports_follow_symlinks} is
@code{True}, @ref{cc3,,copystat()} can modify the last access
and modification times of a symbolic link.

@item 
If @code{os.chflags in os.supports_follow_symlinks} is
@code{True}, @ref{cc3,,copystat()} can modify the flags of
a symbolic link.  (@code{os.chflags} is not available on
all platforms.)
@end itemize

On platforms where some or all of this functionality
is unavailable, when asked to modify a symbolic link,
@ref{cc3,,copystat()} will copy everything it can.
@ref{cc3,,copystat()} never returns failure.

Please see @ref{211c,,os.supports_follow_symlinks}
for more information.
@end quotation
@end cartouche

Raises an @ref{1130,,auditing event} @code{shutil.copystat} with arguments @code{src}, @code{dst}.


Changed in version 3.3: Added @emph{follow_symlinks} argument and support for Linux extended attributes.

@end deffn

@geindex copy() (in module shutil)
@anchor{library/shutil shutil copy}@anchor{570}
@deffn {Function} shutil.copy (src, dst, *, follow_symlinks=True)

Copies the file @emph{src} to the file or directory @emph{dst}.  @emph{src} and @emph{dst}
should be @ref{63d,,path-like objects} or strings.  If
@emph{dst} specifies a directory, the file will be copied into @emph{dst} using the
base filename from @emph{src}.  Returns the path to the newly created file.

If @emph{follow_symlinks} is false, and @emph{src} is a symbolic link,
@emph{dst} will be created as a symbolic link.  If @emph{follow_symlinks}
is true and @emph{src} is a symbolic link, @emph{dst} will be a copy of
the file @emph{src} refers to.

@ref{570,,copy()} copies the file data and the file’s permission
mode (see @ref{c80,,os.chmod()}).  Other metadata, like the
file’s creation and modification times, is not preserved.
To preserve all file metadata from the original, use
@ref{571,,copy2()} instead.

Raises an @ref{1130,,auditing event} @code{shutil.copyfile} with arguments @code{src}, @code{dst}.

Raises an @ref{1130,,auditing event} @code{shutil.copymode} with arguments @code{src}, @code{dst}.


Changed in version 3.3: Added @emph{follow_symlinks} argument.
Now returns path to the newly created file.



Changed in version 3.8: Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
@ref{575,,Platform-dependent efficient copy operations} section.

@end deffn

@geindex copy2() (in module shutil)
@anchor{library/shutil shutil copy2}@anchor{571}
@deffn {Function} shutil.copy2 (src, dst, *, follow_symlinks=True)

Identical to @ref{570,,copy()} except that @ref{571,,copy2()}
also attempts to preserve file metadata.

When @emph{follow_symlinks} is false, and @emph{src} is a symbolic
link, @ref{571,,copy2()} attempts to copy all metadata from the
@emph{src} symbolic link to the newly-created @emph{dst} symbolic link.
However, this functionality is not available on all platforms.
On platforms where some or all of this functionality is
unavailable, @ref{571,,copy2()} will preserve all the metadata
it can; @ref{571,,copy2()} never raises an exception because it
cannot preserve file metadata.

@ref{571,,copy2()} uses @ref{cc3,,copystat()} to copy the file metadata.
Please see @ref{cc3,,copystat()} for more information
about platform support for modifying symbolic link metadata.

Raises an @ref{1130,,auditing event} @code{shutil.copyfile} with arguments @code{src}, @code{dst}.

Raises an @ref{1130,,auditing event} @code{shutil.copystat} with arguments @code{src}, @code{dst}.


Changed in version 3.3: Added @emph{follow_symlinks} argument, try to copy extended
file system attributes too (currently Linux only).
Now returns path to the newly created file.



Changed in version 3.8: Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
@ref{575,,Platform-dependent efficient copy operations} section.

@end deffn

@geindex ignore_patterns() (in module shutil)
@anchor{library/shutil shutil ignore_patterns}@anchor{211d}
@deffn {Function} shutil.ignore_patterns (*patterns)

This factory function creates a function that can be used as a callable for
@ref{538,,copytree()}'s @emph{ignore} argument, ignoring files and directories that
match one of the glob-style @emph{patterns} provided.  See the example below.
@end deffn

@geindex copytree() (in module shutil)
@anchor{library/shutil shutil copytree}@anchor{538}
@deffn {Function} shutil.copytree (src, dst, symlinks=False, ignore=None, copy_function=copy2, ignore_dangling_symlinks=False, dirs_exist_ok=False)

Recursively copy an entire directory tree rooted at @emph{src} to a directory
named @emph{dst} and return the destination directory. @emph{dirs_exist_ok} dictates
whether to raise an exception in case @emph{dst} or any missing parent directory
already exists.

Permissions and times of directories are copied with @ref{cc3,,copystat()},
individual files are copied using @ref{571,,copy2()}.

If @emph{symlinks} is true, symbolic links in the source tree are represented as
symbolic links in the new tree and the metadata of the original links will
be copied as far as the platform allows; if false or omitted, the contents
and metadata of the linked files are copied to the new tree.

When @emph{symlinks} is false, if the file pointed by the symlink doesn’t
exist, an exception will be added in the list of errors raised in
an @ref{b1d,,Error} exception at the end of the copy process.
You can set the optional @emph{ignore_dangling_symlinks} flag to true if you
want to silence this exception. Notice that this option has no effect
on platforms that don’t support @ref{c88,,os.symlink()}.

If @emph{ignore} is given, it must be a callable that will receive as its
arguments the directory being visited by @ref{538,,copytree()}, and a list of its
contents, as returned by @ref{c8d,,os.listdir()}.  Since @ref{538,,copytree()} is
called recursively, the @emph{ignore} callable will be called once for each
directory that is copied.  The callable must return a sequence of directory
and file names relative to the current directory (i.e. a subset of the items
in its second argument); these names will then be ignored in the copy
process.  @ref{211d,,ignore_patterns()} can be used to create such a callable that
ignores names based on glob-style patterns.

If exception(s) occur, an @ref{b1d,,Error} is raised with a list of reasons.

If @emph{copy_function} is given, it must be a callable that will be used to copy
each file. It will be called with the source path and the destination path
as arguments. By default, @ref{571,,copy2()} is used, but any function
that supports the same signature (like @ref{570,,copy()}) can be used.

Raises an @ref{1130,,auditing event} @code{shutil.copytree} with arguments @code{src}, @code{dst}.


Changed in version 3.3: Copy metadata when @emph{symlinks} is false.
Now returns @emph{dst}.



Changed in version 3.2: Added the @emph{copy_function} argument to be able to provide a custom copy
function.
Added the @emph{ignore_dangling_symlinks} argument to silent dangling symlinks
errors when @emph{symlinks} is false.



Changed in version 3.8: Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
@ref{575,,Platform-dependent efficient copy operations} section.



New in version 3.8: The @emph{dirs_exist_ok} parameter.

@end deffn

@geindex rmtree() (in module shutil)
@anchor{library/shutil shutil rmtree}@anchor{53a}
@deffn {Function} shutil.rmtree (path, ignore_errors=False, onerror=None)

@geindex directory; deleting

Delete an entire directory tree; @emph{path} must point to a directory (but not a
symbolic link to a directory).  If @emph{ignore_errors} is true, errors resulting
from failed removals will be ignored; if false or omitted, such errors are
handled by calling a handler specified by @emph{onerror} or, if that is omitted,
they raise an exception.

@cartouche
@quotation Note 
On platforms that support the necessary fd-based functions a symlink
attack resistant version of @ref{53a,,rmtree()} is used by default.  On other
platforms, the @ref{53a,,rmtree()} implementation is susceptible to a symlink
attack: given proper timing and circumstances, attackers can manipulate
symlinks on the filesystem to delete files they wouldn’t be able to access
otherwise.  Applications can use the @ref{211e,,rmtree.avoids_symlink_attacks}
function attribute to determine which case applies.
@end quotation
@end cartouche

If @emph{onerror} is provided, it must be a callable that accepts three
parameters: @emph{function}, @emph{path}, and @emph{excinfo}.

The first parameter, @emph{function}, is the function which raised the exception;
it depends on the platform and implementation.  The second parameter,
@emph{path}, will be the path name passed to @emph{function}.  The third parameter,
@emph{excinfo}, will be the exception information returned by
@ref{18f,,sys.exc_info()}.  Exceptions raised by @emph{onerror} will not be caught.

Raises an @ref{1130,,auditing event} @code{shutil.rmtree} with argument @code{path}.


Changed in version 3.3: Added a symlink attack resistant version that is used automatically
if platform supports fd-based functions.



Changed in version 3.8: On Windows, will no longer delete the contents of a directory junction
before removing the junction.


@geindex avoids_symlink_attacks (shutil.rmtree attribute)
@anchor{library/shutil shutil rmtree avoids_symlink_attacks}@anchor{211e}
@deffn {Attribute} rmtree.avoids_symlink_attacks

Indicates whether the current platform and implementation provides a
symlink attack resistant version of @ref{53a,,rmtree()}.  Currently this is
only true for platforms supporting fd-based directory access functions.


New in version 3.3.

@end deffn
@end deffn

@geindex move() (in module shutil)
@anchor{library/shutil shutil move}@anchor{572}
@deffn {Function} shutil.move (src, dst, copy_function=copy2)

Recursively move a file or directory (@emph{src}) to another location (@emph{dst})
and return the destination.

If the destination is an existing directory, then @emph{src} is moved inside that
directory. If the destination already exists but is not a directory, it may
be overwritten depending on @ref{c85,,os.rename()} semantics.

If the destination is on the current filesystem, then @ref{c85,,os.rename()} is
used. Otherwise, @emph{src} is copied to @emph{dst} using @emph{copy_function} and then
removed.  In case of symlinks, a new symlink pointing to the target of @emph{src}
will be created in or as @emph{dst} and @emph{src} will be removed.

If @emph{copy_function} is given, it must be a callable that takes two arguments
@emph{src} and @emph{dst}, and will be used to copy @emph{src} to @emph{dst} if
@ref{c85,,os.rename()} cannot be used.  If the source is a directory,
@ref{538,,copytree()} is called, passing it the @code{copy_function()}. The
default @emph{copy_function} is @ref{571,,copy2()}.  Using @ref{570,,copy()} as the
@emph{copy_function} allows the move to succeed when it is not possible to also
copy the metadata, at the expense of not copying any of the metadata.

Raises an @ref{1130,,auditing event} @code{shutil.move} with arguments @code{src}, @code{dst}.


Changed in version 3.3: Added explicit symlink handling for foreign filesystems, thus adapting
it to the behavior of GNU’s @code{mv}.
Now returns @emph{dst}.



Changed in version 3.5: Added the @emph{copy_function} keyword argument.



Changed in version 3.8: Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
@ref{575,,Platform-dependent efficient copy operations} section.



Changed in version 3.9: Accepts a @ref{63d,,path-like object} for both @emph{src} and @emph{dst}.

@end deffn

@geindex disk_usage() (in module shutil)
@anchor{library/shutil shutil disk_usage}@anchor{cc1}
@deffn {Function} shutil.disk_usage (path)

Return disk usage statistics about the given path as a @ref{ce9,,named tuple}
with the attributes @emph{total}, @emph{used} and @emph{free}, which are the amount of
total, used and free space, in bytes. @emph{path} may be a file or a
directory.


New in version 3.3.



Changed in version 3.8: On Windows, @emph{path} can now be a file or directory.


@ref{1629,,Availability}: Unix, Windows.
@end deffn

@geindex chown() (in module shutil)
@anchor{library/shutil shutil chown}@anchor{cc2}
@deffn {Function} shutil.chown (path, user=None, group=None)

Change owner @emph{user} and/or @emph{group} of the given @emph{path}.

@emph{user} can be a system user name or a uid; the same applies to @emph{group}. At
least one argument is required.

See also @ref{c81,,os.chown()}, the underlying function.

Raises an @ref{1130,,auditing event} @code{shutil.chown} with arguments @code{path}, @code{user}, @code{group}.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex which() (in module shutil)
@anchor{library/shutil shutil which}@anchor{11ac}
@deffn {Function} shutil.which (cmd, mode=os.F_OK | os.X_OK, path=None)

Return the path to an executable which would be run if the given @emph{cmd} was
called.  If no @emph{cmd} would be called, return @code{None}.

@emph{mode} is a permission mask passed to @ref{c7e,,os.access()}, by default
determining if the file exists and executable.

When no @emph{path} is specified, the results of @ref{d60,,os.environ()} are used,
returning either the “PATH” value or a fallback of @ref{12ad,,os.defpath}.

On Windows, the current directory is always prepended to the @emph{path} whether
or not you use the default or provide your own, which is the behavior the
command shell uses when finding executables.  Additionally, when finding the
@emph{cmd} in the @emph{path}, the @code{PATHEXT} environment variable is checked.  For
example, if you call @code{shutil.which("python")}, @ref{11ac,,which()} will search
@code{PATHEXT} to know that it should look for @code{python.exe} within the @emph{path}
directories.  For example, on Windows:

@example
>>> shutil.which("python")
'C:\\Python33\\python.EXE'
@end example


New in version 3.3.



Changed in version 3.8: The @ref{2f0,,bytes} type is now accepted.  If @emph{cmd} type is
@ref{2f0,,bytes}, the result type is also @ref{2f0,,bytes}.

@end deffn

@geindex Error
@anchor{library/shutil shutil Error}@anchor{b1d}
@deffn {Exception} shutil.Error

This exception collects exceptions that are raised during a multi-file
operation. For @ref{538,,copytree()}, the exception argument is a list of 3-tuples
(@emph{srcname}, @emph{dstname}, @emph{exception}).
@end deffn

@menu
* Platform-dependent efficient copy operations:: 
* copytree example:: 
* rmtree example:: 

@end menu

@node Platform-dependent efficient copy operations,copytree example,,Directory and files operations
@anchor{library/shutil platform-dependent-efficient-copy-operations}@anchor{211f}@anchor{library/shutil shutil-platform-dependent-efficient-copy-operations}@anchor{575}
@subsubsection Platform-dependent efficient copy operations


Starting from Python 3.8, all functions involving a file copy
(@ref{56f,,copyfile()}, @ref{570,,copy()}, @ref{571,,copy2()},
@ref{538,,copytree()}, and @ref{572,,move()}) may use
platform-specific “fast-copy” syscalls in order to copy the file more
efficiently (see bpo-33671@footnote{https://bugs.python.org/issue33671}).
“fast-copy” means that the copying operation occurs within the kernel, avoiding
the use of userspace buffers in Python as in “@code{outfd.write(infd.read())}”.

On macOS fcopyfile@footnote{http://www.manpagez.com/man/3/copyfile/} is used to copy the file content (not metadata).

On Linux @ref{62c,,os.sendfile()} is used.

On Windows @ref{56f,,shutil.copyfile()} uses a bigger default buffer size (1 MiB
instead of 64 KiB) and a @ref{573,,memoryview()}-based variant of
@ref{574,,shutil.copyfileobj()} is used.

If the fast-copy operation fails and no data was written in the destination
file then shutil will silently fallback on using less efficient
@ref{574,,copyfileobj()} function internally.


Changed in version 3.8.


@node copytree example,rmtree example,Platform-dependent efficient copy operations,Directory and files operations
@anchor{library/shutil copytree-example}@anchor{2120}@anchor{library/shutil shutil-copytree-example}@anchor{2121}
@subsubsection copytree example


This example is the implementation of the @ref{538,,copytree()} function, described
above, with the docstring omitted.  It demonstrates many of the other functions
provided by this module.

@example
def copytree(src, dst, symlinks=False):
    names = os.listdir(src)
    os.makedirs(dst)
    errors = []
    for name in names:
        srcname = os.path.join(src, name)
        dstname = os.path.join(dst, name)
        try:
            if symlinks and os.path.islink(srcname):
                linkto = os.readlink(srcname)
                os.symlink(linkto, dstname)
            elif os.path.isdir(srcname):
                copytree(srcname, dstname, symlinks)
            else:
                copy2(srcname, dstname)
            # XXX What about devices, sockets etc.?
        except OSError as why:
            errors.append((srcname, dstname, str(why)))
        # catch the Error from the recursive copytree so that we can
        # continue with other files
        except Error as err:
            errors.extend(err.args[0])
    try:
        copystat(src, dst)
    except OSError as why:
        # can't copy file access times on Windows
        if why.winerror is None:
            errors.extend((src, dst, str(why)))
    if errors:
        raise Error(errors)
@end example

Another example that uses the @ref{211d,,ignore_patterns()} helper:

@example
from shutil import copytree, ignore_patterns

copytree(source, destination, ignore=ignore_patterns('*.pyc', 'tmp*'))
@end example

This will copy everything except @code{.pyc} files and files or directories whose
name starts with @code{tmp}.

Another example that uses the @emph{ignore} argument to add a logging call:

@example
from shutil import copytree
import logging

def _logpath(path, names):
    logging.info('Working in %s', path)
    return []   # nothing will be ignored

copytree(source, destination, ignore=_logpath)
@end example

@node rmtree example,,copytree example,Directory and files operations
@anchor{library/shutil rmtree-example}@anchor{2122}@anchor{library/shutil shutil-rmtree-example}@anchor{2123}
@subsubsection rmtree example


This example shows how to remove a directory tree on Windows where some
of the files have their read-only bit set. It uses the onerror callback
to clear the readonly bit and reattempt the remove. Any subsequent failure
will propagate.

@example
import os, stat
import shutil

def remove_readonly(func, path, _):
    "Clear the readonly bit and reattempt the removal"
    os.chmod(path, stat.S_IWRITE)
    func(path)

shutil.rmtree(directory, onerror=remove_readonly)
@end example

@node Archiving operations,Querying the size of the output terminal,Directory and files operations,shutil — High-level file operations
@anchor{library/shutil archiving-operations}@anchor{dba}@anchor{library/shutil id1}@anchor{2124}
@subsubsection Archiving operations



New in version 3.2.



Changed in version 3.5: Added support for the @emph{xztar} format.


High-level utilities to create and read compressed and archived files are also
provided.  They rely on the @ref{149,,zipfile} and @ref{fb,,tarfile} modules.

@geindex make_archive() (in module shutil)
@anchor{library/shutil shutil make_archive}@anchor{539}
@deffn {Function} shutil.make_archive (base_name, format[, root_dir[, base_dir[, verbose[, dry_run[, owner[, group[, logger]]]]]]])

Create an archive file (such as zip or tar) and return its name.

@emph{base_name} is the name of the file to create, including the path, minus
any format-specific extension. @emph{format} is the archive format: one of
“zip” (if the @ref{14b,,zlib} module is available), “tar”, “gztar” (if the
@ref{14b,,zlib} module is available), “bztar” (if the @ref{12,,bz2} module is
available), or “xztar” (if the @ref{a9,,lzma} module is available).

@emph{root_dir} is a directory that will be the root directory of the
archive, all paths in the archive will be relative to it; for example,
we typically chdir into @emph{root_dir} before creating the archive.

@emph{base_dir} is the directory where we start archiving from;
i.e. @emph{base_dir} will be the common prefix of all files and
directories in the archive.  @emph{base_dir} must be given relative
to @emph{root_dir}.  See @ref{2125,,Archiving example with base_dir} for how to
use @emph{base_dir} and @emph{root_dir} together.

@emph{root_dir} and @emph{base_dir} both default to the current directory.

If @emph{dry_run} is true, no archive is created, but the operations that would be
executed are logged to @emph{logger}.

@emph{owner} and @emph{group} are used when creating a tar archive. By default,
uses the current owner and group.

@emph{logger} must be an object compatible with 
@geindex Python Enhancement Proposals; PEP 282
PEP 282@footnote{https://www.python.org/dev/peps/pep-0282}, usually an instance of
@ref{676,,logging.Logger}.

The @emph{verbose} argument is unused and deprecated.

Raises an @ref{1130,,auditing event} @code{shutil.make_archive} with arguments @code{base_name}, @code{format}, @code{root_dir}, @code{base_dir}.

@cartouche
@quotation Note 
This function is not thread-safe.
@end quotation
@end cartouche


Changed in version 3.8: The modern pax (POSIX.1-2001) format is now used instead of
the legacy GNU format for archives created with @code{format="tar"}.

@end deffn

@geindex get_archive_formats() (in module shutil)
@anchor{library/shutil shutil get_archive_formats}@anchor{2126}
@deffn {Function} shutil.get_archive_formats ()

Return a list of supported formats for archiving.
Each element of the returned sequence is a tuple @code{(name, description)}.

By default @ref{e4,,shutil} provides these formats:


@itemize -

@item 
@emph{zip}: ZIP file (if the @ref{14b,,zlib} module is available).

@item 
@emph{tar}: Uncompressed tar file. Uses POSIX.1-2001 pax format for new archives.

@item 
@emph{gztar}: gzip’ed tar-file (if the @ref{14b,,zlib} module is available).

@item 
@emph{bztar}: bzip2’ed tar-file (if the @ref{12,,bz2} module is available).

@item 
@emph{xztar}: xz’ed tar-file (if the @ref{a9,,lzma} module is available).
@end itemize

You can register new formats or provide your own archiver for any existing
formats, by using @ref{2127,,register_archive_format()}.
@end deffn

@geindex register_archive_format() (in module shutil)
@anchor{library/shutil shutil register_archive_format}@anchor{2127}
@deffn {Function} shutil.register_archive_format (name, function[, extra_args[, description]])

Register an archiver for the format @emph{name}.

@emph{function} is the callable that will be used to unpack archives. The callable
will receive the @emph{base_name} of the file to create, followed by the
@emph{base_dir} (which defaults to @ref{208f,,os.curdir}) to start archiving from.
Further arguments are passed as keyword arguments: @emph{owner}, @emph{group},
@emph{dry_run} and @emph{logger} (as passed in @ref{539,,make_archive()}).

If given, @emph{extra_args} is a sequence of @code{(name, value)} pairs that will be
used as extra keywords arguments when the archiver callable is used.

@emph{description} is used by @ref{2126,,get_archive_formats()} which returns the
list of archivers.  Defaults to an empty string.
@end deffn

@geindex unregister_archive_format() (in module shutil)
@anchor{library/shutil shutil unregister_archive_format}@anchor{2128}
@deffn {Function} shutil.unregister_archive_format (name)

Remove the archive format @emph{name} from the list of supported formats.
@end deffn

@geindex unpack_archive() (in module shutil)
@anchor{library/shutil shutil unpack_archive}@anchor{dbb}
@deffn {Function} shutil.unpack_archive (filename[, extract_dir[, format]])

Unpack an archive. @emph{filename} is the full path of the archive.

@emph{extract_dir} is the name of the target directory where the archive is
unpacked. If not provided, the current working directory is used.

@emph{format} is the archive format: one of “zip”, “tar”, “gztar”, “bztar”, or
“xztar”.  Or any other format registered with
@ref{2129,,register_unpack_format()}.  If not provided, @ref{dbb,,unpack_archive()}
will use the archive file name extension and see if an unpacker was
registered for that extension.  In case none is found,
a @ref{17c,,ValueError} is raised.

Raises an @ref{1130,,auditing event} @code{shutil.unpack_archive} with arguments @code{filename}, @code{extract_dir}, @code{format}.


Changed in version 3.7: Accepts a @ref{63d,,path-like object} for @emph{filename} and @emph{extract_dir}.

@end deffn

@geindex register_unpack_format() (in module shutil)
@anchor{library/shutil shutil register_unpack_format}@anchor{2129}
@deffn {Function} shutil.register_unpack_format (name, extensions, function[, extra_args[, description]])

Registers an unpack format. @emph{name} is the name of the format and
@emph{extensions} is a list of extensions corresponding to the format, like
@code{.zip} for Zip files.

@emph{function} is the callable that will be used to unpack archives. The
callable will receive the path of the archive, followed by the directory
the archive must be extracted to.

When provided, @emph{extra_args} is a sequence of @code{(name, value)} tuples that
will be passed as keywords arguments to the callable.

@emph{description} can be provided to describe the format, and will be returned
by the @ref{212a,,get_unpack_formats()} function.
@end deffn

@geindex unregister_unpack_format() (in module shutil)
@anchor{library/shutil shutil unregister_unpack_format}@anchor{212b}
@deffn {Function} shutil.unregister_unpack_format (name)

Unregister an unpack format. @emph{name} is the name of the format.
@end deffn

@geindex get_unpack_formats() (in module shutil)
@anchor{library/shutil shutil get_unpack_formats}@anchor{212a}
@deffn {Function} shutil.get_unpack_formats ()

Return a list of all registered formats for unpacking.
Each element of the returned sequence is a tuple
@code{(name, extensions, description)}.

By default @ref{e4,,shutil} provides these formats:


@itemize -

@item 
@emph{zip}: ZIP file (unpacking compressed files works only if the corresponding
module is available).

@item 
@emph{tar}: uncompressed tar file.

@item 
@emph{gztar}: gzip’ed tar-file (if the @ref{14b,,zlib} module is available).

@item 
@emph{bztar}: bzip2’ed tar-file (if the @ref{12,,bz2} module is available).

@item 
@emph{xztar}: xz’ed tar-file (if the @ref{a9,,lzma} module is available).
@end itemize

You can register new formats or provide your own unpacker for any existing
formats, by using @ref{2129,,register_unpack_format()}.
@end deffn

@menu
* Archiving example:: 
* Archiving example with base_dir:: 

@end menu

@node Archiving example,Archiving example with base_dir,,Archiving operations
@anchor{library/shutil archiving-example}@anchor{212c}@anchor{library/shutil shutil-archiving-example}@anchor{212d}
@subsubsection Archiving example


In this example, we create a gzip’ed tar-file archive containing all files
found in the @code{.ssh} directory of the user:

@example
>>> from shutil import make_archive
>>> import os
>>> archive_name = os.path.expanduser(os.path.join('~', 'myarchive'))
>>> root_dir = os.path.expanduser(os.path.join('~', '.ssh'))
>>> make_archive(archive_name, 'gztar', root_dir)
'/Users/tarek/myarchive.tar.gz'
@end example

The resulting archive contains:

@example
$ tar -tzvf /Users/tarek/myarchive.tar.gz
drwx------ tarek/staff       0 2010-02-01 16:23:40 ./
-rw-r--r-- tarek/staff     609 2008-06-09 13:26:54 ./authorized_keys
-rwxr-xr-x tarek/staff      65 2008-06-09 13:26:54 ./config
-rwx------ tarek/staff     668 2008-06-09 13:26:54 ./id_dsa
-rwxr-xr-x tarek/staff     609 2008-06-09 13:26:54 ./id_dsa.pub
-rw------- tarek/staff    1675 2008-06-09 13:26:54 ./id_rsa
-rw-r--r-- tarek/staff     397 2008-06-09 13:26:54 ./id_rsa.pub
-rw-r--r-- tarek/staff   37192 2010-02-06 18:23:10 ./known_hosts
@end example

@node Archiving example with base_dir,,Archiving example,Archiving operations
@anchor{library/shutil archiving-example-with-base-dir}@anchor{212e}@anchor{library/shutil shutil-archiving-example-with-basedir}@anchor{2125}
@subsubsection Archiving example with @emph{base_dir}


In this example, similar to the @ref{212d,,one above},
we show how to use @ref{539,,make_archive()}, but this time with the usage of
@emph{base_dir}.  We now have the following directory structure:

@example
$ tree tmp
tmp
└── root
    └── structure
        ├── content
            └── please_add.txt
        └── do_not_add.txt
@end example

In the final archive, @code{please_add.txt} should be included, but
@code{do_not_add.txt} should not.  Therefore we use the following:

@example
>>> from shutil import make_archive
>>> import os
>>> archive_name = os.path.expanduser(os.path.join('~', 'myarchive'))
>>> make_archive(
...     archive_name,
...     'tar',
...     root_dir='tmp/root',
...     base_dir='structure/content',
... )
'/Users/tarek/my_archive.tar'
@end example

Listing the files in the resulting archive gives us:

@example
$ python -m tarfile -l /Users/tarek/myarchive.tar
structure/content/
structure/content/please_add.txt
@end example

@node Querying the size of the output terminal,,Archiving operations,shutil — High-level file operations
@anchor{library/shutil querying-the-size-of-the-output-terminal}@anchor{212f}
@subsubsection Querying the size of the output terminal


@geindex get_terminal_size() (in module shutil)
@anchor{library/shutil shutil get_terminal_size}@anchor{c96}
@deffn {Function} shutil.get_terminal_size (fallback=(columns, lines))

Get the size of the terminal window.

For each of the two dimensions, the environment variable, @code{COLUMNS}
and @code{LINES} respectively, is checked. If the variable is defined and
the value is a positive integer, it is used.

When @code{COLUMNS} or @code{LINES} is not defined, which is the common case,
the terminal connected to @ref{2130,,sys.__stdout__} is queried
by invoking @ref{c95,,os.get_terminal_size()}.

If the terminal size cannot be successfully queried, either because
the system doesn’t support querying, or because we are not
connected to a terminal, the value given in @code{fallback} parameter
is used. @code{fallback} defaults to @code{(80, 24)} which is the default
size used by many terminal emulators.

The value returned is a named tuple of type @ref{2131,,os.terminal_size}.

See also: The Single UNIX Specification, Version 2,
Other Environment Variables@footnote{http://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html#tag_002_003}.


New in version 3.3.



Changed in version 3.11: The @code{fallback} values are also used if @ref{c95,,os.get_terminal_size()}
returns zeroes.

@end deffn


@subsubheading See also



@table @asis

@item Module @ref{c0,,os}

Operating system interfaces, including functions to work with files at a
lower level than Python @ref{d68,,file objects}.

@item Module @ref{9d,,io}

Python’s built-in I/O library, including both abstract classes and
some concrete classes such as file I/O.

@item Built-in function @ref{1c8,,open()}

The standard way to open files for reading and writing with Python.
@end table


@node Data Persistence,Data Compression and Archiving,File and Directory Access,The Python Standard Library
@anchor{library/persistence doc}@anchor{2132}@anchor{library/persistence data-persistence}@anchor{2133}@anchor{library/persistence other-environment-variables}@anchor{2134}@anchor{library/persistence persistence}@anchor{2135}
@section Data Persistence


The modules described in this chapter support storing Python data in a
persistent form on disk.  The @ref{c5,,pickle} and @ref{ac,,marshal} modules can turn
many Python data types into a stream of bytes and then recreate the objects from
the bytes.  The various DBM-related modules support a family of hash-based file
formats that store a mapping of strings to other strings.

The list of modules described in this chapter is:

@menu
* pickle — Python object serialization:: 
* copyreg — Register pickle support functions:: 
* shelve — Python object persistence:: 
* marshal — Internal Python object serialization:: 
* dbm — Interfaces to Unix “databases”:: 
* sqlite3 — DB-API 2.0 interface for SQLite databases: sqlite3 — DB-API 2 0 interface for SQLite databases. 

@end menu

@node pickle — Python object serialization,copyreg — Register pickle support functions,,Data Persistence
@anchor{library/pickle doc}@anchor{2136}@anchor{library/pickle module-pickle}@anchor{c5}@anchor{library/pickle pickle-python-object-serialization}@anchor{2137}
@subsection @code{pickle} — Python object serialization


@geindex module; pickle

@strong{Source code:} Lib/pickle.py@footnote{https://github.com/python/cpython/tree/main/Lib/pickle.py}

@geindex persistence
@geindex persistent; objects
@geindex serializing; objects
@geindex marshalling; objects
@geindex flattening; objects
@geindex pickling; objects


__________________________________________________________________


The @ref{c5,,pickle} module implements binary protocols for serializing and
de-serializing a Python object structure.  @emph{“Pickling”} is the process
whereby a Python object hierarchy is converted into a byte stream, and
@emph{“unpickling”} is the inverse operation, whereby a byte stream
(from a @ref{2138,,binary file} or @ref{87e,,bytes-like object}) is converted
back into an object hierarchy.  Pickling (and unpickling) is alternatively
known as “serialization”, “marshalling,” @footnote{@w{(1)} 
Don’t confuse this with the @ref{ac,,marshal} module
} or “flattening”; however, to
avoid confusion, the terms used here are “pickling” and “unpickling”.

@cartouche
@quotation Warning 
The @code{pickle} module @strong{is not secure}. Only unpickle data you trust.

It is possible to construct malicious pickle data which will @strong{execute
arbitrary code during unpickling}. Never unpickle data that could have come
from an untrusted source, or that could have been tampered with.

Consider signing data with @ref{8a,,hmac} if you need to ensure that it has not
been tampered with.

Safer serialization formats such as @ref{a0,,json} may be more appropriate if
you are processing untrusted data. See @ref{2139,,Comparison with json}.
@end quotation
@end cartouche

@menu
* Relationship to other Python modules:: 
* Data stream format:: 
* Module Interface:: 
* What can be pickled and unpickled?:: 
* Pickling Class Instances:: 
* Custom Reduction for Types@comma{} Functions@comma{} and Other Objects: Custom Reduction for Types Functions and Other Objects. 
* Out-of-band Buffers:: 
* Restricting Globals:: 
* Performance: Performance<2>. 
* Examples: Examples<6>. 

@end menu

@node Relationship to other Python modules,Data stream format,,pickle — Python object serialization
@anchor{library/pickle relationship-to-other-python-modules}@anchor{213a}
@subsubsection Relationship to other Python modules


@menu
* Comparison with marshal:: 
* Comparison with json:: 

@end menu

@node Comparison with marshal,Comparison with json,,Relationship to other Python modules
@anchor{library/pickle comparison-with-marshal}@anchor{213b}
@subsubsection Comparison with @code{marshal}


Python has a more primitive serialization module called @ref{ac,,marshal}, but in
general @ref{c5,,pickle} should always be the preferred way to serialize Python
objects.  @ref{ac,,marshal} exists primarily to support Python’s @code{.pyc}
files.

The @ref{c5,,pickle} module differs from @ref{ac,,marshal} in several significant ways:


@itemize *

@item 
The @ref{c5,,pickle} module keeps track of the objects it has already serialized,
so that later references to the same object won’t be serialized again.
@ref{ac,,marshal} doesn’t do this.

This has implications both for recursive objects and object sharing.  Recursive
objects are objects that contain references to themselves.  These are not
handled by marshal, and in fact, attempting to marshal recursive objects will
crash your Python interpreter.  Object sharing happens when there are multiple
references to the same object in different places in the object hierarchy being
serialized.  @ref{c5,,pickle} stores such objects only once, and ensures that all
other references point to the master copy.  Shared objects remain shared, which
can be very important for mutable objects.

@item 
@ref{ac,,marshal} cannot be used to serialize user-defined classes and their
instances.  @ref{c5,,pickle} can save and restore class instances transparently,
however the class definition must be importable and live in the same module as
when the object was stored.

@item 
The @ref{ac,,marshal} serialization format is not guaranteed to be portable
across Python versions.  Because its primary job in life is to support
@code{.pyc} files, the Python implementers reserve the right to change the
serialization format in non-backwards compatible ways should the need arise.
The @ref{c5,,pickle} serialization format is guaranteed to be backwards compatible
across Python releases provided a compatible pickle protocol is chosen and
pickling and unpickling code deals with Python 2 to Python 3 type differences
if your data is crossing that unique breaking change language boundary.
@end itemize

@node Comparison with json,,Comparison with marshal,Relationship to other Python modules
@anchor{library/pickle comparison-with-json}@anchor{2139}@anchor{library/pickle id2}@anchor{213c}
@subsubsection Comparison with @code{json}


There are fundamental differences between the pickle protocols and
JSON (JavaScript Object Notation)@footnote{http://json.org}:


@itemize *

@item 
JSON is a text serialization format (it outputs unicode text, although
most of the time it is then encoded to @code{utf-8}), while pickle is
a binary serialization format;

@item 
JSON is human-readable, while pickle is not;

@item 
JSON is interoperable and widely used outside of the Python ecosystem,
while pickle is Python-specific;

@item 
JSON, by default, can only represent a subset of the Python built-in
types, and no custom classes; pickle can represent an extremely large
number of Python types (many of them automatically, by clever usage
of Python’s introspection facilities; complex cases can be tackled by
implementing @ref{213d,,specific object APIs});

@item 
Unlike pickle, deserializing untrusted JSON does not in itself create an
arbitrary code execution vulnerability.
@end itemize


@subsubheading See also


The @ref{a0,,json} module: a standard library module allowing JSON
serialization and deserialization.


@node Data stream format,Module Interface,Relationship to other Python modules,pickle — Python object serialization
@anchor{library/pickle data-stream-format}@anchor{213e}@anchor{library/pickle pickle-protocols}@anchor{80a}
@subsubsection Data stream format


@geindex External Data Representation

The data format used by @ref{c5,,pickle} is Python-specific.  This has the
advantage that there are no restrictions imposed by external standards such as
JSON or XDR (which can’t represent pointer sharing); however it means that
non-Python programs may not be able to reconstruct pickled Python objects.

By default, the @ref{c5,,pickle} data format uses a relatively compact binary
representation.  If you need optimal size characteristics, you can efficiently
@ref{213f,,compress} pickled data.

The module @ref{c6,,pickletools} contains tools for analyzing data streams
generated by @ref{c5,,pickle}.  @ref{c6,,pickletools} source code has extensive
comments about opcodes used by pickle protocols.

There are currently 6 different protocols which can be used for pickling.
The higher the protocol used, the more recent the version of Python needed
to read the pickle produced.


@itemize *

@item 
Protocol version 0 is the original “human-readable” protocol and is
backwards compatible with earlier versions of Python.

@item 
Protocol version 1 is an old binary format which is also compatible with
earlier versions of Python.

@item 
Protocol version 2 was introduced in Python 2.3.  It provides much more
efficient pickling of @ref{2140,,new-style class}es.  Refer to 
@geindex Python Enhancement Proposals; PEP 307
PEP 307@footnote{https://www.python.org/dev/peps/pep-0307} for
information about improvements brought by protocol 2.

@item 
Protocol version 3 was added in Python 3.0.  It has explicit support for
@ref{2f0,,bytes} objects and cannot be unpickled by Python 2.x.  This was
the default protocol in Python 3.0–3.7.

@item 
Protocol version 4 was added in Python 3.4.  It adds support for very large
objects, pickling more kinds of objects, and some data format
optimizations.  It is the default protocol starting with Python 3.8.
Refer to 
@geindex Python Enhancement Proposals; PEP 3154
PEP 3154@footnote{https://www.python.org/dev/peps/pep-3154} for information about improvements brought by
protocol 4.

@item 
Protocol version 5 was added in Python 3.8.  It adds support for out-of-band
data and speedup for in-band data.  Refer to 
@geindex Python Enhancement Proposals; PEP 574
PEP 574@footnote{https://www.python.org/dev/peps/pep-0574} for information about
improvements brought by protocol 5.
@end itemize

@cartouche
@quotation Note 
Serialization is a more primitive notion than persistence; although
@ref{c5,,pickle} reads and writes file objects, it does not handle the issue of
naming persistent objects, nor the (even more complicated) issue of concurrent
access to persistent objects.  The @ref{c5,,pickle} module can transform a complex
object into a byte stream and it can transform the byte stream into an object
with the same internal structure.  Perhaps the most obvious thing to do with
these byte streams is to write them onto a file, but it is also conceivable to
send them across a network or store them in a database.  The @ref{e2,,shelve}
module provides a simple interface to pickle and unpickle objects on
DBM-style database files.
@end quotation
@end cartouche

@node Module Interface,What can be pickled and unpickled?,Data stream format,pickle — Python object serialization
@anchor{library/pickle module-interface}@anchor{2141}
@subsubsection Module Interface


To serialize an object hierarchy, you simply call the @ref{ff7,,dumps()} function.
Similarly, to de-serialize a data stream, you call the @ref{85e,,loads()} function.
However, if you want more control over serialization and de-serialization,
you can create a @ref{52d,,Pickler} or an @ref{2142,,Unpickler} object, respectively.

The @ref{c5,,pickle} module provides the following constants:

@geindex HIGHEST_PROTOCOL (in module pickle)
@anchor{library/pickle pickle HIGHEST_PROTOCOL}@anchor{ff8}
@deffn {Data} pickle.HIGHEST_PROTOCOL

An integer, the highest @ref{80a,,protocol version}
available.  This value can be passed as a @emph{protocol} value to functions
@ref{2143,,dump()} and @ref{ff7,,dumps()} as well as the @ref{52d,,Pickler}
constructor.
@end deffn

@geindex DEFAULT_PROTOCOL (in module pickle)
@anchor{library/pickle pickle DEFAULT_PROTOCOL}@anchor{2b0}
@deffn {Data} pickle.DEFAULT_PROTOCOL

An integer, the default @ref{80a,,protocol version} used
for pickling.  May be less than @ref{ff8,,HIGHEST_PROTOCOL}.  Currently the
default protocol is 4, first introduced in Python 3.4 and incompatible
with previous versions.


Changed in version 3.0: The default protocol is 3.



Changed in version 3.8: The default protocol is 4.

@end deffn

The @ref{c5,,pickle} module provides the following functions to make the pickling
process more convenient:

@geindex dump() (in module pickle)
@anchor{library/pickle pickle dump}@anchor{2143}
@deffn {Function} pickle.dump (obj, file, protocol=None, *, fix_imports=True, buffer_callback=None)

Write the pickled representation of the object @emph{obj} to the open
@ref{d68,,file object} @emph{file}.  This is equivalent to
@code{Pickler(file, protocol).dump(obj)}.

Arguments @emph{file}, @emph{protocol}, @emph{fix_imports} and @emph{buffer_callback} have
the same meaning as in the @ref{52d,,Pickler} constructor.


Changed in version 3.8: The @emph{buffer_callback} argument was added.

@end deffn

@geindex dumps() (in module pickle)
@anchor{library/pickle pickle dumps}@anchor{ff7}
@deffn {Function} pickle.dumps (obj, protocol=None, *, fix_imports=True, buffer_callback=None)

Return the pickled representation of the object @emph{obj} as a @ref{2f0,,bytes} object,
instead of writing it to a file.

Arguments @emph{protocol}, @emph{fix_imports} and @emph{buffer_callback} have the same
meaning as in the @ref{52d,,Pickler} constructor.


Changed in version 3.8: The @emph{buffer_callback} argument was added.

@end deffn

@geindex load() (in module pickle)
@anchor{library/pickle pickle load}@anchor{85d}
@deffn {Function} pickle.load (file, *, fix_imports=True, encoding='ASCII', errors='strict', buffers=None)

Read the pickled representation of an object from the open @ref{d68,,file object}
@emph{file} and return the reconstituted object hierarchy specified therein.
This is equivalent to @code{Unpickler(file).load()}.

The protocol version of the pickle is detected automatically, so no
protocol argument is needed.  Bytes past the pickled representation
of the object are ignored.

Arguments @emph{file}, @emph{fix_imports}, @emph{encoding}, @emph{errors}, @emph{strict} and @emph{buffers}
have the same meaning as in the @ref{2142,,Unpickler} constructor.


Changed in version 3.8: The @emph{buffers} argument was added.

@end deffn

@geindex loads() (in module pickle)
@anchor{library/pickle pickle loads}@anchor{85e}
@deffn {Function} pickle.loads (data, /, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None)

Return the reconstituted object hierarchy of the pickled representation
@emph{data} of an object. @emph{data} must be a @ref{87e,,bytes-like object}.

The protocol version of the pickle is detected automatically, so no
protocol argument is needed.  Bytes past the pickled representation
of the object are ignored.

Arguments @emph{file}, @emph{fix_imports}, @emph{encoding}, @emph{errors}, @emph{strict} and @emph{buffers}
have the same meaning as in the @ref{2142,,Unpickler} constructor.


Changed in version 3.8: The @emph{buffers} argument was added.

@end deffn

The @ref{c5,,pickle} module defines three exceptions:

@geindex PickleError
@anchor{library/pickle pickle PickleError}@anchor{2144}
@deffn {Exception} pickle.PickleError

Common base class for the other pickling exceptions.  It inherits
@ref{4d8,,Exception}.
@end deffn

@geindex PicklingError
@anchor{library/pickle pickle PicklingError}@anchor{2145}
@deffn {Exception} pickle.PicklingError

Error raised when an unpicklable object is encountered by @ref{52d,,Pickler}.
It inherits @ref{2144,,PickleError}.

Refer to @ref{2146,,What can be pickled and unpickled?} to learn what kinds of objects can be
pickled.
@end deffn

@geindex UnpicklingError
@anchor{library/pickle pickle UnpicklingError}@anchor{2147}
@deffn {Exception} pickle.UnpicklingError

Error raised when there is a problem unpickling an object, such as a data
corruption or a security violation.  It inherits @ref{2144,,PickleError}.

Note that other exceptions may also be raised during unpickling, including
(but not necessarily limited to) AttributeError, EOFError, ImportError, and
IndexError.
@end deffn

The @ref{c5,,pickle} module exports three classes, @ref{52d,,Pickler},
@ref{2142,,Unpickler} and @ref{2148,,PickleBuffer}:

@geindex Pickler (class in pickle)
@anchor{library/pickle pickle Pickler}@anchor{52d}
@deffn {Class} pickle.Pickler (file, protocol=None, *, fix_imports=True, buffer_callback=None)

This takes a binary file for writing a pickle data stream.

The optional @emph{protocol} argument, an integer, tells the pickler to use
the given protocol; supported protocols are 0 to @ref{ff8,,HIGHEST_PROTOCOL}.
If not specified, the default is @ref{2b0,,DEFAULT_PROTOCOL}.  If a negative
number is specified, @ref{ff8,,HIGHEST_PROTOCOL} is selected.

The @emph{file} argument must have a write() method that accepts a single bytes
argument.  It can thus be an on-disk file opened for binary writing, an
@ref{a15,,io.BytesIO} instance, or any other custom object that meets this
interface.

If @emph{fix_imports} is true and @emph{protocol} is less than 3, pickle will try to
map the new Python 3 names to the old module names used in Python 2, so
that the pickle data stream is readable with Python 2.

If @emph{buffer_callback} is None (the default), buffer views are
serialized into @emph{file} as part of the pickle stream.

If @emph{buffer_callback} is not None, then it can be called any number
of times with a buffer view.  If the callback returns a false value
(such as None), the given buffer is @ref{2149,,out-of-band};
otherwise the buffer is serialized in-band, i.e. inside the pickle stream.

It is an error if @emph{buffer_callback} is not None and @emph{protocol} is
None or smaller than 5.


Changed in version 3.8: The @emph{buffer_callback} argument was added.


@geindex dump() (pickle.Pickler method)
@anchor{library/pickle pickle Pickler dump}@anchor{1300}
@deffn {Method} dump (obj)

Write the pickled representation of @emph{obj} to the open file object given in
the constructor.
@end deffn

@geindex persistent_id() (pickle.Pickler method)
@anchor{library/pickle pickle Pickler persistent_id}@anchor{214a}
@deffn {Method} persistent_id (obj)

Do nothing by default.  This exists so a subclass can override it.

If @ref{214a,,persistent_id()} returns @code{None}, @emph{obj} is pickled as usual.  Any
other value causes @ref{52d,,Pickler} to emit the returned value as a
persistent ID for @emph{obj}.  The meaning of this persistent ID should be
defined by @ref{214b,,Unpickler.persistent_load()}.  Note that the value
returned by @ref{214a,,persistent_id()} cannot itself have a persistent ID.

See @ref{214c,,Persistence of External Objects} for details and examples of uses.
@end deffn

@geindex dispatch_table (pickle.Pickler attribute)
@anchor{library/pickle pickle Pickler dispatch_table}@anchor{cb5}
@deffn {Attribute} dispatch_table

A pickler object’s dispatch table is a registry of @emph{reduction
functions} of the kind which can be declared using
@ref{214d,,copyreg.pickle()}.  It is a mapping whose keys are classes
and whose values are reduction functions.  A reduction function
takes a single argument of the associated class and should
conform to the same interface as a @code{__reduce__()}
method.

By default, a pickler object will not have a
@ref{cb5,,dispatch_table} attribute, and it will instead use the
global dispatch table managed by the @ref{25,,copyreg} module.
However, to customize the pickling for a specific pickler object
one can set the @ref{cb5,,dispatch_table} attribute to a dict-like
object.  Alternatively, if a subclass of @ref{52d,,Pickler} has a
@ref{cb5,,dispatch_table} attribute then this will be used as the
default dispatch table for instances of that class.

See @ref{214e,,Dispatch Tables} for usage examples.


New in version 3.3.

@end deffn

@geindex reducer_override() (pickle.Pickler method)
@anchor{library/pickle pickle Pickler reducer_override}@anchor{52e}
@deffn {Method} reducer_override (self, obj)

Special reducer that can be defined in @ref{52d,,Pickler} subclasses. This
method has priority over any reducer in the @ref{cb5,,dispatch_table}.  It
should conform to the same interface as a @code{__reduce__()} method, and
can optionally return @code{NotImplemented} to fallback on
@ref{cb5,,dispatch_table}-registered reducers to pickle @code{obj}.

For a detailed example, see @ref{214f,,Custom Reduction for Types@comma{} Functions@comma{} and Other Objects}.


New in version 3.8.

@end deffn

@geindex fast (pickle.Pickler attribute)
@anchor{library/pickle pickle Pickler fast}@anchor{2150}
@deffn {Attribute} fast

Deprecated. Enable fast mode if set to a true value.  The fast mode
disables the usage of memo, therefore speeding the pickling process by not
generating superfluous PUT opcodes.  It should not be used with
self-referential objects, doing otherwise will cause @ref{52d,,Pickler} to
recurse infinitely.

Use @ref{2151,,pickletools.optimize()} if you need more compact pickles.
@end deffn
@end deffn

@geindex Unpickler (class in pickle)
@anchor{library/pickle pickle Unpickler}@anchor{2142}
@deffn {Class} pickle.Unpickler (file, *, fix_imports=True, encoding='ASCII', errors='strict', buffers=None)

This takes a binary file for reading a pickle data stream.

The protocol version of the pickle is detected automatically, so no
protocol argument is needed.

The argument @emph{file} must have three methods, a read() method that takes an
integer argument, a readinto() method that takes a buffer argument
and a readline() method that requires no arguments, as in the
@ref{823,,io.BufferedIOBase} interface.  Thus @emph{file} can be an on-disk file
opened for binary reading, an @ref{a15,,io.BytesIO} object, or any other
custom object that meets this interface.

The optional arguments @emph{fix_imports}, @emph{encoding} and @emph{errors} are used
to control compatibility support for pickle stream generated by Python 2.
If @emph{fix_imports} is true, pickle will try to map the old Python 2 names
to the new names used in Python 3.  The @emph{encoding} and @emph{errors} tell
pickle how to decode 8-bit string instances pickled by Python 2;
these default to ‘ASCII’ and ‘strict’, respectively.  The @emph{encoding} can
be ‘bytes’ to read these 8-bit string instances as bytes objects.
Using @code{encoding='latin1'} is required for unpickling NumPy arrays and
instances of @ref{363,,datetime}, @ref{3e2,,date} and
@ref{364,,time} pickled by Python 2.

If @emph{buffers} is None (the default), then all data necessary for
deserialization must be contained in the pickle stream.  This means
that the @emph{buffer_callback} argument was None when a @ref{52d,,Pickler}
was instantiated (or when @ref{2143,,dump()} or @ref{ff7,,dumps()} was called).

If @emph{buffers} is not None, it should be an iterable of buffer-enabled
objects that is consumed each time the pickle stream references
an @ref{2149,,out-of-band} buffer view.  Such buffers have been
given in order to the @emph{buffer_callback} of a Pickler object.


Changed in version 3.8: The @emph{buffers} argument was added.


@geindex load() (pickle.Unpickler method)
@anchor{library/pickle pickle Unpickler load}@anchor{2152}
@deffn {Method} load ()

Read the pickled representation of an object from the open file object
given in the constructor, and return the reconstituted object hierarchy
specified therein.  Bytes past the pickled representation of the object
are ignored.
@end deffn

@geindex persistent_load() (pickle.Unpickler method)
@anchor{library/pickle pickle Unpickler persistent_load}@anchor{214b}
@deffn {Method} persistent_load (pid)

Raise an @ref{2147,,UnpicklingError} by default.

If defined, @ref{214b,,persistent_load()} should return the object specified by
the persistent ID @emph{pid}.  If an invalid persistent ID is encountered, an
@ref{2147,,UnpicklingError} should be raised.

See @ref{214c,,Persistence of External Objects} for details and examples of uses.
@end deffn

@geindex find_class() (pickle.Unpickler method)
@anchor{library/pickle pickle Unpickler find_class}@anchor{2153}
@deffn {Method} find_class (module, name)

Import @emph{module} if necessary and return the object called @emph{name} from it,
where the @emph{module} and @emph{name} arguments are @ref{2ef,,str} objects.  Note,
unlike its name suggests, @ref{2153,,find_class()} is also used for finding
functions.

Subclasses may override this to gain control over what type of objects and
how they can be loaded, potentially reducing security risks. Refer to
@ref{2154,,Restricting Globals} for details.

Raises an @ref{1130,,auditing event} @code{pickle.find_class} with arguments @code{module}, @code{name}.
@end deffn
@end deffn

@geindex PickleBuffer (class in pickle)
@anchor{library/pickle pickle PickleBuffer}@anchor{2148}
@deffn {Class} pickle.PickleBuffer (buffer)

A wrapper for a buffer representing picklable data.  @emph{buffer} must be a
@ref{1a3e,,buffer-providing} object, such as a
@ref{87e,,bytes-like object} or a N-dimensional array.

@ref{2148,,PickleBuffer} is itself a buffer provider, therefore it is
possible to pass it to other APIs expecting a buffer-providing object,
such as @ref{573,,memoryview}.

@ref{2148,,PickleBuffer} objects can only be serialized using pickle
protocol 5 or higher.  They are eligible for
@ref{2149,,out-of-band serialization}.


New in version 3.8.


@geindex raw() (pickle.PickleBuffer method)
@anchor{library/pickle pickle PickleBuffer raw}@anchor{2155}
@deffn {Method} raw ()

Return a @ref{573,,memoryview} of the memory area underlying this buffer.
The returned object is a one-dimensional, C-contiguous memoryview
with format @code{B} (unsigned bytes).  @ref{1b72,,BufferError} is raised if
the buffer is neither C- nor Fortran-contiguous.
@end deffn

@geindex release() (pickle.PickleBuffer method)
@anchor{library/pickle pickle PickleBuffer release}@anchor{2156}
@deffn {Method} release ()

Release the underlying buffer exposed by the PickleBuffer object.
@end deffn
@end deffn

@node What can be pickled and unpickled?,Pickling Class Instances,Module Interface,pickle — Python object serialization
@anchor{library/pickle pickle-picklable}@anchor{2146}@anchor{library/pickle what-can-be-pickled-and-unpickled}@anchor{2157}
@subsubsection What can be pickled and unpickled?


The following types can be pickled:


@itemize *

@item 
@code{None}, @code{True}, and @code{False}

@item 
integers, floating point numbers, complex numbers

@item 
strings, bytes, bytearrays

@item 
tuples, lists, sets, and dictionaries containing only picklable objects

@item 
functions defined at the top level of a module (using @ref{fb8,,def}, not
@ref{e42,,lambda})

@item 
built-in functions defined at the top level of a module

@item 
classes that are defined at the top level of a module

@item 
instances of such classes whose @ref{7a4,,__dict__} or the result of
calling @code{__getstate__()} is picklable  (see section @ref{213d,,Pickling Class Instances} for
details).
@end itemize

Attempts to pickle unpicklable objects will raise the @ref{2145,,PicklingError}
exception; when this happens, an unspecified number of bytes may have already
been written to the underlying file.  Trying to pickle a highly recursive data
structure may exceed the maximum recursion depth, a @ref{8c4,,RecursionError} will be
raised in this case.  You can carefully raise this limit with
@ref{106b,,sys.setrecursionlimit()}.

Note that functions (built-in and user-defined) are pickled by “fully qualified”
name reference, not by value. @footnote{@w{(2)} 
This is why @ref{e42,,lambda} functions cannot be pickled:  all
@code{lambda} functions share the same name:  @code{<lambda>}.
}  This means that only the function name is
pickled, along with the name of the module the function is defined in.  Neither
the function’s code, nor any of its function attributes are pickled.  Thus the
defining module must be importable in the unpickling environment, and the module
must contain the named object, otherwise an exception will be raised. @footnote{@w{(3)} 
The exception raised will likely be an @ref{3b8,,ImportError} or an
@ref{163,,AttributeError} but it could be something else.
}

Similarly, classes are pickled by named reference, so the same restrictions in
the unpickling environment apply.  Note that none of the class’s code or data is
pickled, so in the following example the class attribute @code{attr} is not
restored in the unpickling environment:

@example
class Foo:
    attr = 'A class attribute'

picklestring = pickle.dumps(Foo)
@end example

These restrictions are why picklable functions and classes must be defined in
the top level of a module.

Similarly, when class instances are pickled, their class’s code and data are not
pickled along with them.  Only the instance data are pickled.  This is done on
purpose, so you can fix bugs in a class or add methods to the class and still
load objects that were created with an earlier version of the class.  If you
plan to have long-lived objects that will see many versions of a class, it may
be worthwhile to put a version number in the objects so that suitable
conversions can be made by the class’s @code{__setstate__()} method.

@node Pickling Class Instances,Custom Reduction for Types Functions and Other Objects,What can be pickled and unpickled?,pickle — Python object serialization
@anchor{library/pickle pickle-inst}@anchor{213d}@anchor{library/pickle pickling-class-instances}@anchor{2158}
@subsubsection Pickling Class Instances


In this section, we describe the general mechanisms available to you to define,
customize, and control how class instances are pickled and unpickled.

In most cases, no additional code is needed to make instances picklable.  By
default, pickle will retrieve the class and the attributes of an instance via
introspection. When a class instance is unpickled, its @code{__init__()} method
is usually @emph{not} invoked.  The default behaviour first creates an uninitialized
instance and then restores the saved attributes.  The following code shows an
implementation of this behaviour:

@example
def save(obj):
    return (obj.__class__, obj.__dict__)

def load(cls, attributes):
    obj = cls.__new__(cls)
    obj.__dict__.update(attributes)
    return obj
@end example

Classes can alter the default behaviour by providing one or several special
methods:

@geindex __getnewargs_ex__() (object method)
@anchor{library/pickle object __getnewargs_ex__}@anchor{2159}
@deffn {Method} object.__getnewargs_ex__ ()

In protocols 2 and newer, classes that implements the
@ref{2159,,__getnewargs_ex__()} method can dictate the values passed to the
@ref{1794,,__new__()} method upon unpickling.  The method must return a pair
@code{(args, kwargs)} where @emph{args} is a tuple of positional arguments
and @emph{kwargs} a dictionary of named arguments for constructing the
object.  Those will be passed to the @ref{1794,,__new__()} method upon
unpickling.

You should implement this method if the @ref{1794,,__new__()} method of your
class requires keyword-only arguments.  Otherwise, it is recommended for
compatibility to implement @ref{215a,,__getnewargs__()}.


Changed in version 3.6: @ref{2159,,__getnewargs_ex__()} is now used in protocols 2 and 3.

@end deffn

@geindex __getnewargs__() (object method)
@anchor{library/pickle object __getnewargs__}@anchor{215a}
@deffn {Method} object.__getnewargs__ ()

This method serves a similar purpose as @ref{2159,,__getnewargs_ex__()}, but
supports only positional arguments.  It must return a tuple of arguments
@code{args} which will be passed to the @ref{1794,,__new__()} method upon unpickling.

@ref{215a,,__getnewargs__()} will not be called if @ref{2159,,__getnewargs_ex__()} is
defined.


Changed in version 3.6: Before Python 3.6, @ref{215a,,__getnewargs__()} was called instead of
@ref{2159,,__getnewargs_ex__()} in protocols 2 and 3.

@end deffn

@geindex __getstate__() (object method)
@anchor{library/pickle object __getstate__}@anchor{215b}
@deffn {Method} object.__getstate__ ()

Classes can further influence how their instances are pickled; if the class
defines the method @ref{215b,,__getstate__()}, it is called and the returned object
is pickled as the contents for the instance, instead of the contents of the
instance’s dictionary.  If the @ref{215b,,__getstate__()} method is absent, the
instance’s @ref{7a4,,__dict__} is pickled as usual.
@end deffn

@geindex __setstate__() (object method)
@anchor{library/pickle object __setstate__}@anchor{215c}
@deffn {Method} object.__setstate__ (state)

Upon unpickling, if the class defines @ref{215c,,__setstate__()}, it is called with
the unpickled state.  In that case, there is no requirement for the state
object to be a dictionary.  Otherwise, the pickled state must be a dictionary
and its items are assigned to the new instance’s dictionary.

@cartouche
@quotation Note 
If @ref{215b,,__getstate__()} returns a false value, the @ref{215c,,__setstate__()}
method will not be called upon unpickling.
@end quotation
@end cartouche
@end deffn

Refer to the section @ref{215d,,Handling Stateful Objects} for more information about how to use
the methods @code{__getstate__()} and @code{__setstate__()}.

@cartouche
@quotation Note 
At unpickling time, some methods like @code{__getattr__()},
@code{__getattribute__()}, or @code{__setattr__()} may be called upon the
instance.  In case those methods rely on some internal invariant being
true, the type should implement @code{__new__()} to establish such an
invariant, as @code{__init__()} is not called when unpickling an
instance.
@end quotation
@end cartouche

@geindex copy; protocol

As we shall see, pickle does not use directly the methods described above.  In
fact, these methods are part of the copy protocol which implements the
@code{__reduce__()} special method.  The copy protocol provides a unified
interface for retrieving the data necessary for pickling and copying
objects. @footnote{@w{(4)} 
The @ref{24,,copy} module uses this protocol for shallow and deep copying
operations.
}

Although powerful, implementing @code{__reduce__()} directly in your classes is
error prone.  For this reason, class designers should use the high-level
interface (i.e., @code{__getnewargs_ex__()}, @code{__getstate__()} and
@code{__setstate__()}) whenever possible.  We will show, however, cases where
using @code{__reduce__()} is the only option or leads to more efficient pickling
or both.

@geindex __reduce__() (object method)
@anchor{library/pickle object __reduce__}@anchor{4ce}
@deffn {Method} object.__reduce__ ()

The interface is currently defined as follows.  The @ref{4ce,,__reduce__()} method
takes no argument and shall return either a string or preferably a tuple (the
returned object is often referred to as the “reduce value”).

If a string is returned, the string should be interpreted as the name of a
global variable.  It should be the object’s local name relative to its
module; the pickle module searches the module namespace to determine the
object’s module.  This behaviour is typically useful for singletons.

When a tuple is returned, it must be between two and six items long.
Optional items can either be omitted, or @code{None} can be provided as their
value.  The semantics of each item are in order:

@c XXX Mention __newobj__ special-case?


@itemize *

@item 
A callable object that will be called to create the initial version of the
object.

@item 
A tuple of arguments for the callable object.  An empty tuple must be given
if the callable does not accept any argument.

@item 
Optionally, the object’s state, which will be passed to the object’s
@ref{215c,,__setstate__()} method as previously described.  If the object has no
such method then, the value must be a dictionary and it will be added to
the object’s @ref{7a4,,__dict__} attribute.

@item 
Optionally, an iterator (and not a sequence) yielding successive items.
These items will be appended to the object either using
@code{obj.append(item)} or, in batch, using @code{obj.extend(list_of_items)}.
This is primarily used for list subclasses, but may be used by other
classes as long as they have @code{append()} and @code{extend()} methods with
the appropriate signature.  (Whether @code{append()} or @code{extend()} is
used depends on which pickle protocol version is used as well as the number
of items to append, so both must be supported.)

@item 
Optionally, an iterator (not a sequence) yielding successive key-value
pairs.  These items will be stored to the object using @code{obj[key] =
value}.  This is primarily used for dictionary subclasses, but may be used
by other classes as long as they implement @ref{17d8,,__setitem__()}.

@item 
Optionally, a callable with a @code{(obj, state)} signature. This
callable allows the user to programmatically control the state-updating
behavior of a specific object, instead of using @code{obj}’s static
@ref{215c,,__setstate__()} method. If not @code{None}, this callable will have
priority over @code{obj}’s @ref{215c,,__setstate__()}.


New in version 3.8: The optional sixth tuple item, @code{(obj, state)}, was added.

@end itemize
@end deffn

@geindex __reduce_ex__() (object method)
@anchor{library/pickle object __reduce_ex__}@anchor{215e}
@deffn {Method} object.__reduce_ex__ (protocol)

Alternatively, a @ref{215e,,__reduce_ex__()} method may be defined.  The only
difference is this method should take a single integer argument, the protocol
version.  When defined, pickle will prefer it over the @ref{4ce,,__reduce__()}
method.  In addition, @ref{4ce,,__reduce__()} automatically becomes a synonym for
the extended version.  The main use for this method is to provide
backwards-compatible reduce values for older Python releases.
@end deffn

@menu
* Persistence of External Objects:: 
* Dispatch Tables:: 
* Handling Stateful Objects:: 

@end menu

@node Persistence of External Objects,Dispatch Tables,,Pickling Class Instances
@anchor{library/pickle persistence-of-external-objects}@anchor{215f}@anchor{library/pickle pickle-persistent}@anchor{214c}
@subsubsection Persistence of External Objects


@geindex persistent_id (pickle protocol)
@geindex persistent_load (pickle protocol)

For the benefit of object persistence, the @ref{c5,,pickle} module supports the
notion of a reference to an object outside the pickled data stream.  Such
objects are referenced by a persistent ID, which should be either a string of
alphanumeric characters (for protocol 0) @footnote{@w{(5)} 
The limitation on alphanumeric characters is due to the fact
the persistent IDs, in protocol 0, are delimited by the newline
character.  Therefore if any kind of newline characters occurs in
persistent IDs, the resulting pickle will become unreadable.
} or just an arbitrary object (for
any newer protocol).

The resolution of such persistent IDs is not defined by the @ref{c5,,pickle}
module; it will delegate this resolution to the user-defined methods on the
pickler and unpickler, @ref{214a,,persistent_id()} and
@ref{214b,,persistent_load()} respectively.

To pickle objects that have an external persistent ID, the pickler must have a
custom @ref{214a,,persistent_id()} method that takes an object as an
argument and returns either @code{None} or the persistent ID for that object.
When @code{None} is returned, the pickler simply pickles the object as normal.
When a persistent ID string is returned, the pickler will pickle that object,
along with a marker so that the unpickler will recognize it as a persistent ID.

To unpickle external objects, the unpickler must have a custom
@ref{214b,,persistent_load()} method that takes a persistent ID object and
returns the referenced object.

Here is a comprehensive example presenting how persistent ID can be used to
pickle external objects by reference.

@example
# Simple example presenting how persistent ID can be used to pickle
# external objects by reference.

import pickle
import sqlite3
from collections import namedtuple

# Simple class representing a record in our database.
MemoRecord = namedtuple("MemoRecord", "key, task")

class DBPickler(pickle.Pickler):

    def persistent_id(self, obj):
        # Instead of pickling MemoRecord as a regular class instance, we emit a
        # persistent ID.
        if isinstance(obj, MemoRecord):
            # Here, our persistent ID is simply a tuple, containing a tag and a
            # key, which refers to a specific record in the database.
            return ("MemoRecord", obj.key)
        else:
            # If obj does not have a persistent ID, return None. This means obj
            # needs to be pickled as usual.
            return None


class DBUnpickler(pickle.Unpickler):

    def __init__(self, file, connection):
        super().__init__(file)
        self.connection = connection

    def persistent_load(self, pid):
        # This method is invoked whenever a persistent ID is encountered.
        # Here, pid is the tuple returned by DBPickler.
        cursor = self.connection.cursor()
        type_tag, key_id = pid
        if type_tag == "MemoRecord":
            # Fetch the referenced record from the database and return it.
            cursor.execute("SELECT * FROM memos WHERE key=?", (str(key_id),))
            key, task = cursor.fetchone()
            return MemoRecord(key, task)
        else:
            # Always raises an error if you cannot return the correct object.
            # Otherwise, the unpickler will think None is the object referenced
            # by the persistent ID.
            raise pickle.UnpicklingError("unsupported persistent object")


def main():
    import io
    import pprint

    # Initialize and populate our database.
    conn = sqlite3.connect(":memory:")
    cursor = conn.cursor()
    cursor.execute("CREATE TABLE memos(key INTEGER PRIMARY KEY, task TEXT)")
    tasks = (
        'give food to fish',
        'prepare group meeting',
        'fight with a zebra',
        )
    for task in tasks:
        cursor.execute("INSERT INTO memos VALUES(NULL, ?)", (task,))

    # Fetch the records to be pickled.
    cursor.execute("SELECT * FROM memos")
    memos = [MemoRecord(key, task) for key, task in cursor]
    # Save the records using our custom DBPickler.
    file = io.BytesIO()
    DBPickler(file).dump(memos)

    print("Pickled records:")
    pprint.pprint(memos)

    # Update a record, just for good measure.
    cursor.execute("UPDATE memos SET task='learn italian' WHERE key=1")

    # Load the records from the pickle data stream.
    file.seek(0)
    memos = DBUnpickler(file, conn).load()

    print("Unpickled records:")
    pprint.pprint(memos)


if __name__ == '__main__':
    main()
@end example

@node Dispatch Tables,Handling Stateful Objects,Persistence of External Objects,Pickling Class Instances
@anchor{library/pickle dispatch-tables}@anchor{2160}@anchor{library/pickle pickle-dispatch}@anchor{214e}
@subsubsection Dispatch Tables


If one wants to customize pickling of some classes without disturbing
any other code which depends on pickling, then one can create a
pickler with a private dispatch table.

The global dispatch table managed by the @ref{25,,copyreg} module is
available as @code{copyreg.dispatch_table}.  Therefore, one may
choose to use a modified copy of @code{copyreg.dispatch_table} as a
private dispatch table.

For example

@example
f = io.BytesIO()
p = pickle.Pickler(f)
p.dispatch_table = copyreg.dispatch_table.copy()
p.dispatch_table[SomeClass] = reduce_SomeClass
@end example

creates an instance of @ref{52d,,pickle.Pickler} with a private dispatch
table which handles the @code{SomeClass} class specially.  Alternatively,
the code

@example
class MyPickler(pickle.Pickler):
    dispatch_table = copyreg.dispatch_table.copy()
    dispatch_table[SomeClass] = reduce_SomeClass
f = io.BytesIO()
p = MyPickler(f)
@end example

does the same, but all instances of @code{MyPickler} will by default
share the same dispatch table.  The equivalent code using the
@ref{25,,copyreg} module is

@example
copyreg.pickle(SomeClass, reduce_SomeClass)
f = io.BytesIO()
p = pickle.Pickler(f)
@end example

@node Handling Stateful Objects,,Dispatch Tables,Pickling Class Instances
@anchor{library/pickle handling-stateful-objects}@anchor{2161}@anchor{library/pickle pickle-state}@anchor{215d}
@subsubsection Handling Stateful Objects


@geindex __getstate__() (copy protocol)
@geindex __setstate__() (copy protocol)

Here’s an example that shows how to modify pickling behavior for a class.
The @code{TextReader} class opens a text file, and returns the line number and
line contents each time its @code{readline()} method is called. If a
@code{TextReader} instance is pickled, all attributes @emph{except} the file object
member are saved. When the instance is unpickled, the file is reopened, and
reading resumes from the last location. The @code{__setstate__()} and
@code{__getstate__()} methods are used to implement this behavior.

@example
class TextReader:
    """Print and number lines in a text file."""

    def __init__(self, filename):
        self.filename = filename
        self.file = open(filename)
        self.lineno = 0

    def readline(self):
        self.lineno += 1
        line = self.file.readline()
        if not line:
            return None
        if line.endswith('\n'):
            line = line[:-1]
        return "%i: %s" % (self.lineno, line)

    def __getstate__(self):
        # Copy the object's state from self.__dict__ which contains
        # all our instance attributes. Always use the dict.copy()
        # method to avoid modifying the original state.
        state = self.__dict__.copy()
        # Remove the unpicklable entries.
        del state['file']
        return state

    def __setstate__(self, state):
        # Restore instance attributes (i.e., filename and lineno).
        self.__dict__.update(state)
        # Restore the previously opened file's state. To do so, we need to
        # reopen it and read from it until the line count is restored.
        file = open(self.filename)
        for _ in range(self.lineno):
            file.readline()
        # Finally, save the file.
        self.file = file
@end example

A sample usage might be something like this:

@example
>>> reader = TextReader("hello.txt")
>>> reader.readline()
'1: Hello world!'
>>> reader.readline()
'2: I am line number two.'
>>> new_reader = pickle.loads(pickle.dumps(reader))
>>> new_reader.readline()
'3: Goodbye!'
@end example

@node Custom Reduction for Types Functions and Other Objects,Out-of-band Buffers,Pickling Class Instances,pickle — Python object serialization
@anchor{library/pickle custom-reduction-for-types-functions-and-other-objects}@anchor{2162}@anchor{library/pickle reducer-override}@anchor{214f}
@subsubsection Custom Reduction for Types, Functions, and Other Objects



New in version 3.8.


Sometimes, @ref{cb5,,dispatch_table} may not be flexible enough.
In particular we may want to customize pickling based on another criterion
than the object’s type, or we may want to customize the pickling of
functions and classes.

For those cases, it is possible to subclass from the @ref{52d,,Pickler} class and
implement a @ref{52e,,reducer_override()} method. This method can return an
arbitrary reduction tuple (see @code{__reduce__()}). It can alternatively return
@code{NotImplemented} to fallback to the traditional behavior.

If both the @ref{cb5,,dispatch_table} and
@ref{52e,,reducer_override()} are defined, then
@ref{52e,,reducer_override()} method takes priority.

@cartouche
@quotation Note 
For performance reasons, @ref{52e,,reducer_override()} may not be
called for the following objects: @code{None}, @code{True}, @code{False}, and
exact instances of @ref{1b1,,int}, @ref{256,,float}, @ref{2f0,,bytes},
@ref{2ef,,str}, @ref{3b0,,dict}, @ref{370,,set}, @ref{439,,frozenset}, @ref{43a,,list}
and @ref{3e5,,tuple}.
@end quotation
@end cartouche

Here is a simple example where we allow pickling and reconstructing
a given class:

@example
import io
import pickle

class MyClass:
    my_attribute = 1

class MyPickler(pickle.Pickler):
    def reducer_override(self, obj):
        """Custom reducer for MyClass."""
        if getattr(obj, "__name__", None) == "MyClass":
            return type, (obj.__name__, obj.__bases__,
                          @{'my_attribute': obj.my_attribute@})
        else:
            # For any other object, fallback to usual reduction
            return NotImplemented

f = io.BytesIO()
p = MyPickler(f)
p.dump(MyClass)

del MyClass

unpickled_class = pickle.loads(f.getvalue())

assert isinstance(unpickled_class, type)
assert unpickled_class.__name__ == "MyClass"
assert unpickled_class.my_attribute == 1
@end example

@node Out-of-band Buffers,Restricting Globals,Custom Reduction for Types Functions and Other Objects,pickle — Python object serialization
@anchor{library/pickle out-of-band-buffers}@anchor{2163}@anchor{library/pickle pickle-oob}@anchor{2149}
@subsubsection Out-of-band Buffers



New in version 3.8.


In some contexts, the @ref{c5,,pickle} module is used to transfer massive amounts
of data.  Therefore, it can be important to minimize the number of memory
copies, to preserve performance and resource consumption.  However, normal
operation of the @ref{c5,,pickle} module, as it transforms a graph-like structure
of objects into a sequential stream of bytes, intrinsically involves copying
data to and from the pickle stream.

This constraint can be eschewed if both the @emph{provider} (the implementation
of the object types to be transferred) and the @emph{consumer} (the implementation
of the communications system) support the out-of-band transfer facilities
provided by pickle protocol 5 and higher.

@menu
* Provider API:: 
* Consumer API:: 
* Example: Example<4>. 

@end menu

@node Provider API,Consumer API,,Out-of-band Buffers
@anchor{library/pickle provider-api}@anchor{2164}
@subsubsection Provider API


The large data objects to be pickled must implement a @code{__reduce_ex__()}
method specialized for protocol 5 and higher, which returns a
@ref{2148,,PickleBuffer} instance (instead of e.g. a @ref{2f0,,bytes} object)
for any large data.

A @ref{2148,,PickleBuffer} object @emph{signals} that the underlying buffer is
eligible for out-of-band data transfer.  Those objects remain compatible
with normal usage of the @ref{c5,,pickle} module.  However, consumers can also
opt-in to tell @ref{c5,,pickle} that they will handle those buffers by
themselves.

@node Consumer API,Example<4>,Provider API,Out-of-band Buffers
@anchor{library/pickle consumer-api}@anchor{2165}
@subsubsection Consumer API


A communications system can enable custom handling of the @ref{2148,,PickleBuffer}
objects generated when serializing an object graph.

On the sending side, it needs to pass a @emph{buffer_callback} argument to
@ref{52d,,Pickler} (or to the @ref{2143,,dump()} or @ref{ff7,,dumps()} function), which
will be called with each @ref{2148,,PickleBuffer} generated while pickling
the object graph.  Buffers accumulated by the @emph{buffer_callback} will not
see their data copied into the pickle stream, only a cheap marker will be
inserted.

On the receiving side, it needs to pass a @emph{buffers} argument to
@ref{2142,,Unpickler} (or to the @ref{85d,,load()} or @ref{85e,,loads()} function),
which is an iterable of the buffers which were passed to @emph{buffer_callback}.
That iterable should produce buffers in the same order as they were passed
to @emph{buffer_callback}.  Those buffers will provide the data expected by the
reconstructors of the objects whose pickling produced the original
@ref{2148,,PickleBuffer} objects.

Between the sending side and the receiving side, the communications system
is free to implement its own transfer mechanism for out-of-band buffers.
Potential optimizations include the use of shared memory or datatype-dependent
compression.

@node Example<4>,,Consumer API,Out-of-band Buffers
@anchor{library/pickle example}@anchor{2166}
@subsubsection Example


Here is a trivial example where we implement a @ref{2f1,,bytearray} subclass
able to participate in out-of-band buffer pickling:

@example
class ZeroCopyByteArray(bytearray):

    def __reduce_ex__(self, protocol):
        if protocol >= 5:
            return type(self)._reconstruct, (PickleBuffer(self),), None
        else:
            # PickleBuffer is forbidden with pickle protocols <= 4.
            return type(self)._reconstruct, (bytearray(self),)

    @@classmethod
    def _reconstruct(cls, obj):
        with memoryview(obj) as m:
            # Get a handle over the original buffer object
            obj = m.obj
            if type(obj) is cls:
                # Original buffer object is a ZeroCopyByteArray, return it
                # as-is.
                return obj
            else:
                return cls(obj)
@end example

The reconstructor (the @code{_reconstruct} class method) returns the buffer’s
providing object if it has the right type.  This is an easy way to simulate
zero-copy behaviour on this toy example.

On the consumer side, we can pickle those objects the usual way, which
when unserialized will give us a copy of the original object:

@example
b = ZeroCopyByteArray(b"abc")
data = pickle.dumps(b, protocol=5)
new_b = pickle.loads(data)
print(b == new_b)  # True
print(b is new_b)  # False: a copy was made
@end example

But if we pass a @emph{buffer_callback} and then give back the accumulated
buffers when unserializing, we are able to get back the original object:

@example
b = ZeroCopyByteArray(b"abc")
buffers = []
data = pickle.dumps(b, protocol=5, buffer_callback=buffers.append)
new_b = pickle.loads(data, buffers=buffers)
print(b == new_b)  # True
print(b is new_b)  # True: no copy was made
@end example

This example is limited by the fact that @ref{2f1,,bytearray} allocates its
own memory: you cannot create a @ref{2f1,,bytearray} instance that is backed
by another object’s memory.  However, third-party datatypes such as NumPy
arrays do not have this limitation, and allow use of zero-copy pickling
(or making as few copies as possible) when transferring between distinct
processes or systems.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 574
PEP 574@footnote{https://www.python.org/dev/peps/pep-0574} – Pickle protocol 5 with out-of-band data


@node Restricting Globals,Performance<2>,Out-of-band Buffers,pickle — Python object serialization
@anchor{library/pickle pickle-restrict}@anchor{2154}@anchor{library/pickle restricting-globals}@anchor{2167}
@subsubsection Restricting Globals


@geindex find_class() (pickle protocol)

By default, unpickling will import any class or function that it finds in the
pickle data.  For many applications, this behaviour is unacceptable as it
permits the unpickler to import and invoke arbitrary code.  Just consider what
this hand-crafted pickle data stream does when loaded:

@example
>>> import pickle
>>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
hello world
0
@end example

In this example, the unpickler imports the @ref{fbb,,os.system()} function and then
apply the string argument “echo hello world”.  Although this example is
inoffensive, it is not difficult to imagine one that could damage your system.

For this reason, you may want to control what gets unpickled by customizing
@ref{2153,,Unpickler.find_class()}.  Unlike its name suggests,
@ref{2153,,Unpickler.find_class()} is called whenever a global (i.e., a class or
a function) is requested.  Thus it is possible to either completely forbid
globals or restrict them to a safe subset.

Here is an example of an unpickler allowing only few safe classes from the
@ref{11,,builtins} module to be loaded:

@example
import builtins
import io
import pickle

safe_builtins = @{
    'range',
    'complex',
    'set',
    'frozenset',
    'slice',
@}

class RestrictedUnpickler(pickle.Unpickler):

    def find_class(self, module, name):
        # Only allow safe classes from builtins.
        if module == "builtins" and name in safe_builtins:
            return getattr(builtins, name)
        # Forbid everything else.
        raise pickle.UnpicklingError("global '%s.%s' is forbidden" %
                                     (module, name))

def restricted_loads(s):
    """Helper function analogous to pickle.loads()."""
    return RestrictedUnpickler(io.BytesIO(s)).load()
@end example

A sample usage of our unpickler working has intended:

@example
>>> restricted_loads(pickle.dumps([1, 2, range(15)]))
[1, 2, range(0, 15)]
>>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
Traceback (most recent call last):
  ...
pickle.UnpicklingError: global 'os.system' is forbidden
>>> restricted_loads(b'cbuiltins\neval\n'
...                  b'(S\'getattr(__import__("os"), "system")'
...                  b'("echo hello world")\'\ntR.')
Traceback (most recent call last):
  ...
pickle.UnpicklingError: global 'builtins.eval' is forbidden
@end example

@c XXX Add note about how extension codes could evade our protection
@c mechanism (e.g. cached classes do not invokes find_class()).

As our examples shows, you have to be careful with what you allow to be
unpickled.  Therefore if security is a concern, you may want to consider
alternatives such as the marshalling API in @ref{146,,xmlrpc.client} or
third-party solutions.

@node Performance<2>,Examples<6>,Restricting Globals,pickle — Python object serialization
@anchor{library/pickle performance}@anchor{2168}
@subsubsection Performance


Recent versions of the pickle protocol (from protocol 2 and upwards) feature
efficient binary encodings for several common features and built-in types.
Also, the @ref{c5,,pickle} module has a transparent optimizer written in C.

@node Examples<6>,,Performance<2>,pickle — Python object serialization
@anchor{library/pickle examples}@anchor{2169}@anchor{library/pickle pickle-example}@anchor{216a}
@subsubsection Examples


For the simplest code, use the @ref{2143,,dump()} and @ref{85d,,load()} functions.

@example
import pickle

# An arbitrary collection of objects supported by pickle.
data = @{
    'a': [1, 2.0, 3, 4+6j],
    'b': ("character string", b"byte string"),
    'c': @{None, True, False@}
@}

with open('data.pickle', 'wb') as f:
    # Pickle the 'data' dictionary using the highest protocol available.
    pickle.dump(data, f, pickle.HIGHEST_PROTOCOL)
@end example

The following example reads the resulting pickled data.

@example
import pickle

with open('data.pickle', 'rb') as f:
    # The protocol version used is detected automatically, so we do not
    # have to specify it.
    data = pickle.load(f)
@end example

@c XXX: Add examples showing how to optimize pickles for size (like using

@c pickletools.optimize() or the gzip module).


@subsubheading See also



@table @asis

@item Module @ref{25,,copyreg}

Pickle interface constructor registration for extension types.

@item Module @ref{c6,,pickletools}

Tools for working with and analyzing pickled data.

@item Module @ref{e2,,shelve}

Indexed databases of objects; uses @ref{c5,,pickle}.

@item Module @ref{24,,copy}

Shallow and deep object copying.

@item Module @ref{ac,,marshal}

High-performance serialization of built-in types.
@end table


@node copyreg — Register pickle support functions,shelve — Python object persistence,pickle — Python object serialization,Data Persistence
@anchor{library/copyreg doc}@anchor{216b}@anchor{library/copyreg copyreg-register-pickle-support-functions}@anchor{216c}@anchor{library/copyreg module-copyreg}@anchor{25}
@subsection @code{copyreg} — Register @code{pickle} support functions


@geindex module; copyreg

@strong{Source code:} Lib/copyreg.py@footnote{https://github.com/python/cpython/tree/main/Lib/copyreg.py}

@geindex module; pickle
@geindex module; copy


__________________________________________________________________


The @ref{25,,copyreg} module offers a way to define functions used while pickling
specific objects.  The @ref{c5,,pickle} and @ref{24,,copy} modules use those functions
when pickling/copying those objects.  The module provides configuration
information about object constructors which are not classes.
Such constructors may be factory functions or class instances.

@geindex constructor() (in module copyreg)
@anchor{library/copyreg copyreg constructor}@anchor{216d}
@deffn {Function} copyreg.constructor (object)

Declares @emph{object} to be a valid constructor.  If @emph{object} is not callable (and
hence not valid as a constructor), raises @ref{162,,TypeError}.
@end deffn

@geindex pickle() (in module copyreg)
@anchor{library/copyreg copyreg pickle}@anchor{214d}
@deffn {Function} copyreg.pickle (type, function, constructor=None)

Declares that @emph{function} should be used as a “reduction” function for objects
of type @emph{type}.  @emph{function} should return either a string or a tuple
containing two or three elements.

The optional @emph{constructor} parameter, if provided, is a callable object which
can be used to reconstruct the object when called with the tuple of arguments
returned by @emph{function} at pickling time.  A @ref{162,,TypeError} is raised if the
@emph{constructor} is not callable.

See the @ref{c5,,pickle} module for more details on the interface
expected of @emph{function} and @emph{constructor}.  Note that the
@ref{cb5,,dispatch_table} attribute of a pickler
object or subclass of @ref{52d,,pickle.Pickler} can also be used for
declaring reduction functions.
@end deffn

@menu
* Example: Example<5>. 

@end menu

@node Example<5>,,,copyreg — Register pickle support functions
@anchor{library/copyreg example}@anchor{216e}
@subsubsection Example


The example below would like to show how to register a pickle function and how
it will be used:

@example
>>> import copyreg, copy, pickle
>>> class C:
...     def __init__(self, a):
...         self.a = a
...
>>> def pickle_c(c):
...     print("pickling a C instance...")
...     return C, (c.a,)
...
>>> copyreg.pickle(C, pickle_c)
>>> c = C(1)
>>> d = copy.copy(c)  
pickling a C instance...
>>> p = pickle.dumps(c)  
pickling a C instance...
@end example

@node shelve — Python object persistence,marshal — Internal Python object serialization,copyreg — Register pickle support functions,Data Persistence
@anchor{library/shelve doc}@anchor{216f}@anchor{library/shelve module-shelve}@anchor{e2}@anchor{library/shelve shelve-python-object-persistence}@anchor{2170}
@subsection @code{shelve} — Python object persistence


@geindex module; shelve

@strong{Source code:} Lib/shelve.py@footnote{https://github.com/python/cpython/tree/main/Lib/shelve.py}

@geindex module; pickle


__________________________________________________________________


A “shelf” is a persistent, dictionary-like object.  The difference with “dbm”
databases is that the values (not the keys!) in a shelf can be essentially
arbitrary Python objects — anything that the @ref{c5,,pickle} module can handle.
This includes most class instances, recursive data types, and objects containing
lots of shared  sub-objects.  The keys are ordinary strings.

@geindex open() (in module shelve)
@anchor{library/shelve shelve open}@anchor{2171}
@deffn {Function} shelve.open (filename, flag='c', protocol=None, writeback=False)

Open a persistent dictionary.  The filename specified is the base filename for
the underlying database.  As a side-effect, an extension may be added to the
filename and more than one file may be created.  By default, the underlying
database file is opened for reading and writing.  The optional @emph{flag} parameter
has the same interpretation as the @emph{flag} parameter of @ref{aa4,,dbm.open()}.

By default, pickles created with @ref{2b0,,pickle.DEFAULT_PROTOCOL} are used
to serialize values.  The version of the pickle protocol can be specified
with the @emph{protocol} parameter.

Because of Python semantics, a shelf cannot know when a mutable
persistent-dictionary entry is modified.  By default modified objects are
written @emph{only} when assigned to the shelf (see @ref{2172,,Example}).  If the
optional @emph{writeback} parameter is set to @code{True}, all entries accessed are also
cached in memory, and written back on @ref{2173,,sync()} and
@ref{2174,,close()}; this can make it handier to mutate mutable entries in
the persistent dictionary, but, if many entries are accessed, it can consume
vast amounts of memory for the cache, and it can make the close operation
very slow since all accessed entries are written back (there is no way to
determine which accessed entries are mutable, nor which ones were actually
mutated).


Changed in version 3.10: @ref{2b0,,pickle.DEFAULT_PROTOCOL} is now used as the default pickle
protocol.



Changed in version 3.11: Accepts @ref{63d,,path-like object} for filename.


@cartouche
@quotation Note 
Do not rely on the shelf being closed automatically; always call
@ref{2174,,close()} explicitly when you don’t need it any more, or
use @ref{2171,,shelve.open()} as a context manager:

@example
with shelve.open('spam') as db:
    db['eggs'] = 'eggs'
@end example
@end quotation
@end cartouche
@end deffn
@anchor{library/shelve shelve-security}@anchor{2175}
@cartouche
@quotation Warning 
Because the @ref{e2,,shelve} module is backed by @ref{c5,,pickle}, it is insecure
to load a shelf from an untrusted source.  Like with pickle, loading a shelf
can execute arbitrary code.
@end quotation
@end cartouche

Shelf objects support most of methods and operations supported by dictionaries
(except copying, constructors and operators @code{|} and @code{|=}).  This eases the
transition from dictionary based scripts to those requiring persistent storage.

Two additional methods are supported:

@geindex sync() (shelve.Shelf method)
@anchor{library/shelve shelve Shelf sync}@anchor{2173}
@deffn {Method} Shelf.sync ()

Write back all entries in the cache if the shelf was opened with @emph{writeback}
set to @ref{748,,True}.  Also empty the cache and synchronize the persistent
dictionary on disk, if feasible.  This is called automatically when the shelf
is closed with @ref{2174,,close()}.
@end deffn

@geindex close() (shelve.Shelf method)
@anchor{library/shelve shelve Shelf close}@anchor{2174}
@deffn {Method} Shelf.close ()

Synchronize and close the persistent @emph{dict} object.  Operations on a closed
shelf will fail with a @ref{17c,,ValueError}.
@end deffn


@subsubheading See also


Persistent dictionary recipe@footnote{https://code.activestate.com/recipes/576642/}
with widely supported storage formats and having the speed of native
dictionaries.


@menu
* Restrictions:: 
* Example: Example<6>. 

@end menu

@node Restrictions,Example<6>,,shelve — Python object persistence
@anchor{library/shelve restrictions}@anchor{2176}
@subsubsection Restrictions


@quotation

@geindex module; dbm.ndbm
@geindex module; dbm.gnu
@end quotation


@itemize *

@item 
The choice of which database package will be used (such as @ref{33,,dbm.ndbm} or
@ref{32,,dbm.gnu}) depends on which interface is available.  Therefore it is not
safe to open the database directly using @ref{30,,dbm}.  The database is also
(unfortunately) subject to the limitations of @ref{30,,dbm}, if it is used —
this means that (the pickled representation of) the objects stored in the
database should be fairly small, and in rare cases key collisions may cause
the database to refuse updates.

@item 
The @ref{e2,,shelve} module does not support @emph{concurrent} read/write access to
shelved objects.  (Multiple simultaneous read accesses are safe.)  When a
program has a shelf open for writing, no other program should have it open for
reading or writing.  Unix file locking can be used to solve this, but this
differs across Unix versions and requires knowledge about the database
implementation used.
@end itemize

@geindex Shelf (class in shelve)
@anchor{library/shelve shelve Shelf}@anchor{b1b}
@deffn {Class} shelve.Shelf (dict, protocol=None, writeback=False, keyencoding='utf-8')

A subclass of @ref{c3f,,collections.abc.MutableMapping} which stores pickled
values in the @emph{dict} object.

By default, pickles created with @ref{2b0,,pickle.DEFAULT_PROTOCOL} are used
to serialize values.  The version of the pickle protocol can be specified
with the @emph{protocol} parameter.  See the @ref{c5,,pickle} documentation for a
discussion of the pickle protocols.

If the @emph{writeback} parameter is @code{True}, the object will hold a cache of all
entries accessed and write them back to the @emph{dict} at sync and close times.
This allows natural operations on mutable entries, but can consume much more
memory and make sync and close take a long time.

The @emph{keyencoding} parameter is the encoding used to encode keys before they
are used with the underlying dict.

A @ref{b1b,,Shelf} object can also be used as a context manager, in which
case it will be automatically closed when the @ref{168,,with} block ends.


Changed in version 3.2: Added the @emph{keyencoding} parameter; previously, keys were always encoded in
UTF-8.



Changed in version 3.4: Added context manager support.



Changed in version 3.10: @ref{2b0,,pickle.DEFAULT_PROTOCOL} is now used as the default pickle
protocol.

@end deffn

@geindex BsdDbShelf (class in shelve)
@anchor{library/shelve shelve BsdDbShelf}@anchor{1b48}
@deffn {Class} shelve.BsdDbShelf (dict, protocol=None, writeback=False, keyencoding='utf-8')

A subclass of @ref{b1b,,Shelf} which exposes @code{first()}, @code{next()},
@code{previous()}, @code{last()} and @code{set_location()} which are available
in the third-party @code{bsddb} module from pybsddb@footnote{https://www.jcea.es/programacion/pybsddb.htm} but not in other database
modules.  The @emph{dict} object passed to the constructor must support those
methods.  This is generally accomplished by calling one of
@code{bsddb.hashopen()}, @code{bsddb.btopen()} or @code{bsddb.rnopen()}.  The
optional @emph{protocol}, @emph{writeback}, and @emph{keyencoding} parameters have the same
interpretation as for the @ref{b1b,,Shelf} class.
@end deffn

@geindex DbfilenameShelf (class in shelve)
@anchor{library/shelve shelve DbfilenameShelf}@anchor{1b49}
@deffn {Class} shelve.DbfilenameShelf (filename, flag='c', protocol=None, writeback=False)

A subclass of @ref{b1b,,Shelf} which accepts a @emph{filename} instead of a dict-like
object.  The underlying file will be opened using @ref{aa4,,dbm.open()}.  By
default, the file will be created and opened for both read and write.  The
optional @emph{flag} parameter has the same interpretation as for the @ref{2171,,open()}
function.  The optional @emph{protocol} and @emph{writeback} parameters have the same
interpretation as for the @ref{b1b,,Shelf} class.
@end deffn

@node Example<6>,,Restrictions,shelve — Python object persistence
@anchor{library/shelve example}@anchor{2177}@anchor{library/shelve shelve-example}@anchor{2172}
@subsubsection Example


To summarize the interface (@code{key} is a string, @code{data} is an arbitrary
object):

@example
import shelve

d = shelve.open(filename)  # open -- file may get suffix added by low-level
                           # library

d[key] = data              # store data at key (overwrites old data if
                           # using an existing key)
data = d[key]              # retrieve a COPY of data at key (raise KeyError
                           # if no such key)
del d[key]                 # delete data stored at key (raises KeyError
                           # if no such key)

flag = key in d            # true if the key exists
klist = list(d.keys())     # a list of all existing keys (slow!)

# as d was opened WITHOUT writeback=True, beware:
d['xx'] = [0, 1, 2]        # this works as expected, but...
d['xx'].append(3)          # *this doesn't!* -- d['xx'] is STILL [0, 1, 2]!

# having opened d without writeback=True, you need to code carefully:
temp = d['xx']             # extracts the copy
temp.append(5)             # mutates the copy
d['xx'] = temp             # stores the copy right back, to persist it

# or, d=shelve.open(filename,writeback=True) would let you just code
# d['xx'].append(5) and have it work as expected, BUT it would also
# consume more memory and make the d.close() operation slower.

d.close()                  # close it
@end example


@subsubheading See also



@table @asis

@item Module @ref{30,,dbm}

Generic interface to @code{dbm}-style databases.

@item Module @ref{c5,,pickle}

Object serialization used by @ref{e2,,shelve}.
@end table


@node marshal — Internal Python object serialization,dbm — Interfaces to Unix “databases”,shelve — Python object persistence,Data Persistence
@anchor{library/marshal doc}@anchor{2178}@anchor{library/marshal marshal-internal-python-object-serialization}@anchor{2179}@anchor{library/marshal module-marshal}@anchor{ac}
@subsection @code{marshal} — Internal Python object serialization


@geindex module; marshal


__________________________________________________________________


This module contains functions that can read and write Python values in a binary
format.  The format is specific to Python, but independent of machine
architecture issues (e.g., you can write a Python value to a file on a PC,
transport the file to a Sun, and read it back there).  Details of the format are
undocumented on purpose; it may change between Python versions (although it
rarely does). @footnote{@w{(1)} 
The name of this module stems from a bit of terminology used by the designers of
Modula-3 (amongst others), who use the term “marshalling” for shipping of data
around in a self-contained form. Strictly speaking, “to marshal” means to
convert some data from internal to external form (in an RPC buffer for instance)
and “unmarshalling” for the reverse process.
}

@geindex module; pickle
@geindex module; shelve

This is not a general “persistence” module.  For general persistence and
transfer of Python objects through RPC calls, see the modules @ref{c5,,pickle} and
@ref{e2,,shelve}.  The @ref{ac,,marshal} module exists mainly to support reading and
writing the “pseudo-compiled” code for Python modules of @code{.pyc} files.
Therefore, the Python maintainers reserve the right to modify the marshal format
in backward incompatible ways should the need arise.  If you’re serializing and
de-serializing Python objects, use the @ref{c5,,pickle} module instead – the
performance is comparable, version independence is guaranteed, and pickle
supports a substantially wider range of objects than marshal.

@cartouche
@quotation Warning 
The @ref{ac,,marshal} module is not intended to be secure against erroneous or
maliciously constructed data.  Never unmarshal data received from an
untrusted or unauthenticated source.
@end quotation
@end cartouche

@geindex object; code
@geindex code object

Not all Python object types are supported; in general, only objects whose value
is independent from a particular invocation of Python can be written and read by
this module.  The following types are supported: booleans, integers, floating
point numbers, complex numbers, strings, bytes, bytearrays, tuples, lists, sets,
frozensets, dictionaries, and code objects, where it should be understood that
tuples, lists, sets, frozensets and dictionaries are only supported as long as
the values contained therein are themselves supported.  The
singletons @ref{182,,None}, @ref{1a67,,Ellipsis} and @ref{736,,StopIteration} can also be
marshalled and unmarshalled.
For format @emph{version} lower than 3, recursive lists, sets and dictionaries cannot
be written (see below).

There are functions that read/write files as well as functions operating on
bytes-like objects.

The module defines these functions:

@geindex dump() (in module marshal)
@anchor{library/marshal marshal dump}@anchor{217a}
@deffn {Function} marshal.dump (value, file[, version])

Write the value on the open file.  The value must be a supported type.  The
file must be a writeable @ref{2138,,binary file}.

If the value has (or contains an object that has) an unsupported type, a
@ref{17c,,ValueError} exception is raised — but garbage data will also be written
to the file.  The object will not be properly read back by @ref{12be,,load()}.

The @emph{version} argument indicates the data format that @code{dump} should use
(see below).

Raises an @ref{1130,,auditing event} @code{marshal.dumps} with arguments @code{value}, @code{version}.
@end deffn

@geindex load() (in module marshal)
@anchor{library/marshal marshal load}@anchor{12be}
@deffn {Function} marshal.load (file)

Read one value from the open file and return it.  If no valid value is read
(e.g. because the data has a different Python version’s incompatible marshal
format), raise @ref{e74,,EOFError}, @ref{17c,,ValueError} or @ref{162,,TypeError}.  The
file must be a readable @ref{2138,,binary file}.

Raises an @ref{1130,,auditing event} @code{marshal.load} with no arguments.

@cartouche
@quotation Note 
If an object containing an unsupported type was marshalled with @ref{217a,,dump()},
@ref{12be,,load()} will substitute @code{None} for the unmarshallable type.
@end quotation
@end cartouche


Changed in version 3.10: This call used to raise a @code{code.__new__} audit event for each code object. Now
it raises a single @code{marshal.load} event for the entire load operation.

@end deffn

@geindex dumps() (in module marshal)
@anchor{library/marshal marshal dumps}@anchor{a16}
@deffn {Function} marshal.dumps (value[, version])

Return the bytes object that would be written to a file by @code{dump(value, file)}.  The
value must be a supported type.  Raise a @ref{17c,,ValueError} exception if value
has (or contains an object that has) an unsupported type.

The @emph{version} argument indicates the data format that @code{dumps} should use
(see below).

Raises an @ref{1130,,auditing event} @code{marshal.dumps} with arguments @code{value}, @code{version}.
@end deffn

@geindex loads() (in module marshal)
@anchor{library/marshal marshal loads}@anchor{217b}
@deffn {Function} marshal.loads (bytes)

Convert the @ref{87e,,bytes-like object} to a value.  If no valid value is found, raise
@ref{e74,,EOFError}, @ref{17c,,ValueError} or @ref{162,,TypeError}.  Extra bytes in the
input are ignored.

Raises an @ref{1130,,auditing event} @code{marshal.loads} with argument @code{bytes}.


Changed in version 3.10: This call used to raise a @code{code.__new__} audit event for each code object. Now
it raises a single @code{marshal.loads} event for the entire load operation.

@end deffn

In addition, the following constants are defined:

@geindex version (in module marshal)
@anchor{library/marshal marshal version}@anchor{217c}
@deffn {Data} marshal.version

Indicates the format that the module uses. Version 0 is the historical
format, version 1 shares interned strings and version 2 uses a binary format
for floating point numbers.
Version 3 adds support for object instancing and recursion.
The current version is 4.
@end deffn

@node dbm — Interfaces to Unix “databases”,sqlite3 — DB-API 2 0 interface for SQLite databases,marshal — Internal Python object serialization,Data Persistence
@anchor{library/dbm doc}@anchor{217d}@anchor{library/dbm dbm-interfaces-to-unix-databases}@anchor{217e}@anchor{library/dbm module-dbm}@anchor{30}
@subsection @code{dbm} — Interfaces to Unix “databases”


@geindex module; dbm

@strong{Source code:} Lib/dbm/__init__.py@footnote{https://github.com/python/cpython/tree/main/Lib/dbm/__init__.py}


__________________________________________________________________


@ref{30,,dbm} is a generic interface to variants of the DBM database —
@ref{32,,dbm.gnu} or @ref{33,,dbm.ndbm}.  If none of these modules is installed, the
slow-but-simple implementation in module @ref{31,,dbm.dumb} will be used.  There
is a third party interface@footnote{https://www.jcea.es/programacion/pybsddb.htm} to
the Oracle Berkeley DB.

@geindex error
@anchor{library/dbm dbm error}@anchor{217f}
@deffn {Exception} dbm.error

A tuple containing the exceptions that can be raised by each of the supported
modules, with a unique exception also named @ref{217f,,dbm.error} as the first
item — the latter is used when @ref{217f,,dbm.error} is raised.
@end deffn

@geindex whichdb() (in module dbm)
@anchor{library/dbm dbm whichdb}@anchor{2180}
@deffn {Function} dbm.whichdb (filename)

This function attempts to guess which of the several simple database modules
available — @ref{32,,dbm.gnu}, @ref{33,,dbm.ndbm} or @ref{31,,dbm.dumb} — should
be used to open a given file.

Returns one of the following values: @code{None} if the file can’t be opened
because it’s unreadable or doesn’t exist; the empty string (@code{'@w{'}}) if the
file’s format can’t be guessed; or a string containing the required module
name, such as @code{'dbm.ndbm'} or @code{'dbm.gnu'}.
@end deffn


Changed in version 3.11: Accepts @ref{63d,,path-like object} for filename.


@geindex open() (in module dbm)
@anchor{library/dbm dbm open}@anchor{aa4}
@deffn {Function} dbm.open (file, flag='r', mode=438)

Open the database file @emph{file} and return a corresponding object.

If the database file already exists, the @ref{2180,,whichdb()} function is used to
determine its type and the appropriate module is used; if it does not exist,
the first module listed above that can be imported is used.

The optional @emph{flag} argument can be:


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Value

@tab

Meaning

@item

@code{'r'}

@tab

Open existing database for reading only
(default)

@item

@code{'w'}

@tab

Open existing database for reading and
writing

@item

@code{'c'}

@tab

Open database for reading and writing,
creating it if it doesn’t exist

@item

@code{'n'}

@tab

Always create a new, empty database, open
for reading and writing

@end multitable


The optional @emph{mode} argument is the Unix mode of the file, used only when the
database has to be created.  It defaults to octal @code{0o666} (and will be
modified by the prevailing umask).
@end deffn

The object returned by @ref{aa4,,open()} supports the same basic functionality as
dictionaries; keys and their corresponding values can be stored, retrieved, and
deleted, and the @ref{2fc,,in} operator and the @code{keys()} method are
available, as well as @code{get()} and @code{setdefault()}.


Changed in version 3.2: @code{get()} and @code{setdefault()} are now available in all database modules.



Changed in version 3.8: Deleting a key from a read-only database raises database module specific error
instead of @ref{5bb,,KeyError}.



Changed in version 3.11: Accepts @ref{63d,,path-like object} for file.


Key and values are always stored as bytes. This means that when
strings are used they are implicitly converted to the default encoding before
being stored.

These objects also support being used in a @ref{168,,with} statement, which
will automatically close them when done.


Changed in version 3.4: Added native support for the context management protocol to the objects
returned by @ref{aa4,,open()}.


The following example records some hostnames and a corresponding title,  and
then prints out the contents of the database:

@example
import dbm

# Open database, creating it if necessary.
with dbm.open('cache', 'c') as db:

    # Record some values
    db[b'hello'] = b'there'
    db['www.python.org'] = 'Python Website'
    db['www.cnn.com'] = 'Cable News Network'

    # Note that the keys are considered bytes now.
    assert db[b'www.python.org'] == b'Python Website'
    # Notice how the value is now in bytes.
    assert db['www.cnn.com'] == b'Cable News Network'

    # Often-used methods of the dict interface work too.
    print(db.get('python.org', b'not present'))

    # Storing a non-string key or value will raise an exception (most
    # likely a TypeError).
    db['www.yahoo.com'] = 4

# db is automatically closed when leaving the with statement.
@end example


@subsubheading See also



@table @asis

@item Module @ref{e2,,shelve}

Persistence module which stores non-string data.
@end table


The individual submodules are described in the following sections.

@menu
* dbm.gnu — GNU’s reinterpretation of dbm: dbm gnu — GNU’s reinterpretation of dbm. 
* dbm.ndbm — Interface based on ndbm: dbm ndbm — Interface based on ndbm. 
* dbm.dumb — Portable DBM implementation: dbm dumb — Portable DBM implementation. 

@end menu

@node dbm gnu — GNU’s reinterpretation of dbm,dbm ndbm — Interface based on ndbm,,dbm — Interfaces to Unix “databases”
@anchor{library/dbm dbm-gnu-gnu-s-reinterpretation-of-dbm}@anchor{2181}@anchor{library/dbm module-dbm gnu}@anchor{32}
@subsubsection @code{dbm.gnu} — GNU’s reinterpretation of dbm


@geindex module; dbm.gnu

@strong{Source code:} Lib/dbm/gnu.py@footnote{https://github.com/python/cpython/tree/main/Lib/dbm/gnu.py}


__________________________________________________________________


This module is quite similar to the @ref{30,,dbm} module, but uses the GNU library
@code{gdbm} instead to provide some additional functionality.  Please note that the
file formats created by @ref{32,,dbm.gnu} and @ref{33,,dbm.ndbm} are incompatible.

The @ref{32,,dbm.gnu} module provides an interface to the GNU DBM library.
@code{dbm.gnu.gdbm} objects behave like mappings (dictionaries), except that keys and
values are always converted to bytes before storing.  Printing a @code{gdbm}
object doesn’t print the
keys and values, and the @code{items()} and @code{values()} methods are not
supported.

@geindex error
@anchor{library/dbm dbm gnu error}@anchor{5b9}
@deffn {Exception} dbm.gnu.error

Raised on @ref{32,,dbm.gnu}-specific errors, such as I/O errors. @ref{5bb,,KeyError} is
raised for general mapping errors like specifying an incorrect key.
@end deffn

@geindex open() (in module dbm.gnu)
@anchor{library/dbm dbm gnu open}@anchor{12f7}
@deffn {Function} dbm.gnu.open (filename[, flag[, mode]])

Open a @code{gdbm} database and return a @code{gdbm} object.  The @emph{filename}
argument is the name of the database file.

The optional @emph{flag} argument can be:


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Value

@tab

Meaning

@item

@code{'r'}

@tab

Open existing database for reading only
(default)

@item

@code{'w'}

@tab

Open existing database for reading and
writing

@item

@code{'c'}

@tab

Open database for reading and writing,
creating it if it doesn’t exist

@item

@code{'n'}

@tab

Always create a new, empty database, open
for reading and writing

@end multitable


The following additional characters may be appended to the flag to control
how the database is opened:


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Value

@tab

Meaning

@item

@code{'f'}

@tab

Open the database in fast mode.  Writes
to the database will not be synchronized.

@item

@code{'s'}

@tab

Synchronized mode. This will cause changes
to the database to be immediately written
to the file.

@item

@code{'u'}

@tab

Do not lock database.

@end multitable


Not all flags are valid for all versions of @code{gdbm}.  The module constant
@code{open_flags} is a string of supported flag characters.  The exception
@ref{5b9,,error} is raised if an invalid flag is specified.

The optional @emph{mode} argument is the Unix mode of the file, used only when the
database has to be created.  It defaults to octal @code{0o666}.

In addition to the dictionary-like methods, @code{gdbm} objects have the
following methods:


Changed in version 3.11: Accepts @ref{63d,,path-like object} for filename.


@geindex firstkey() (dbm.gnu.gdbm method)
@anchor{library/dbm dbm gnu gdbm firstkey}@anchor{2182}
@deffn {Method} gdbm.firstkey ()

It’s possible to loop over every key in the database using this method  and the
@ref{2183,,nextkey()} method.  The traversal is ordered by @code{gdbm}’s internal
hash values, and won’t be sorted by the key values.  This method returns
the starting key.
@end deffn

@geindex nextkey() (dbm.gnu.gdbm method)
@anchor{library/dbm dbm gnu gdbm nextkey}@anchor{2183}
@deffn {Method} gdbm.nextkey (key)

Returns the key that follows @emph{key} in the traversal.  The following code prints
every key in the database @code{db}, without having to create a list in memory that
contains them all:

@example
k = db.firstkey()
while k is not None:
    print(k)
    k = db.nextkey(k)
@end example
@end deffn

@geindex reorganize() (dbm.gnu.gdbm method)
@anchor{library/dbm dbm gnu gdbm reorganize}@anchor{2184}
@deffn {Method} gdbm.reorganize ()

If you have carried out a lot of deletions and would like to shrink the space
used by the @code{gdbm} file, this routine will reorganize the database.  @code{gdbm}
objects will not shorten the length of a database file except by using this
reorganization; otherwise, deleted file space will be kept and reused as new
(key, value) pairs are added.
@end deffn

@geindex sync() (dbm.gnu.gdbm method)
@anchor{library/dbm dbm gnu gdbm sync}@anchor{2185}
@deffn {Method} gdbm.sync ()

When the database has been opened in fast mode, this method forces any
unwritten data to be written to the disk.
@end deffn

@geindex close() (dbm.gnu.gdbm method)
@anchor{library/dbm dbm gnu gdbm close}@anchor{2186}
@deffn {Method} gdbm.close ()

Close the @code{gdbm} database.
@end deffn
@end deffn

@node dbm ndbm — Interface based on ndbm,dbm dumb — Portable DBM implementation,dbm gnu — GNU’s reinterpretation of dbm,dbm — Interfaces to Unix “databases”
@anchor{library/dbm dbm-ndbm-interface-based-on-ndbm}@anchor{2187}@anchor{library/dbm module-dbm ndbm}@anchor{33}
@subsubsection @code{dbm.ndbm} — Interface based on ndbm


@geindex module; dbm.ndbm

@strong{Source code:} Lib/dbm/ndbm.py@footnote{https://github.com/python/cpython/tree/main/Lib/dbm/ndbm.py}


__________________________________________________________________


The @ref{33,,dbm.ndbm} module provides an interface to the Unix “(n)dbm” library.
Dbm objects behave like mappings (dictionaries), except that keys and values are
always stored as bytes. Printing a @code{dbm} object doesn’t print the keys and
values, and the @code{items()} and @code{values()} methods are not supported.

This module can be used with the “classic” ndbm interface or the GNU GDBM
compatibility interface. On Unix, the @code{configure} script will attempt
to locate the appropriate header file to simplify building this module.

@geindex error
@anchor{library/dbm dbm ndbm error}@anchor{5ba}
@deffn {Exception} dbm.ndbm.error

Raised on @ref{33,,dbm.ndbm}-specific errors, such as I/O errors. @ref{5bb,,KeyError} is raised
for general mapping errors like specifying an incorrect key.
@end deffn

@geindex library (in module dbm.ndbm)
@anchor{library/dbm dbm ndbm library}@anchor{2188}
@deffn {Data} dbm.ndbm.library

Name of the @code{ndbm} implementation library used.
@end deffn

@geindex open() (in module dbm.ndbm)
@anchor{library/dbm dbm ndbm open}@anchor{12f8}
@deffn {Function} dbm.ndbm.open (filename[, flag[, mode]])

Open a dbm database and return a @code{ndbm} object.  The @emph{filename} argument is the
name of the database file (without the @code{.dir} or @code{.pag} extensions).

The optional @emph{flag} argument must be one of these values:


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Value

@tab

Meaning

@item

@code{'r'}

@tab

Open existing database for reading only
(default)

@item

@code{'w'}

@tab

Open existing database for reading and
writing

@item

@code{'c'}

@tab

Open database for reading and writing,
creating it if it doesn’t exist

@item

@code{'n'}

@tab

Always create a new, empty database, open
for reading and writing

@end multitable


The optional @emph{mode} argument is the Unix mode of the file, used only when the
database has to be created.  It defaults to octal @code{0o666} (and will be
modified by the prevailing umask).

In addition to the dictionary-like methods, @code{ndbm} objects
provide the following method:


Changed in version 3.11: Accepts @ref{63d,,path-like object} for filename.


@geindex close() (dbm.ndbm.ndbm method)
@anchor{library/dbm dbm ndbm ndbm close}@anchor{2189}
@deffn {Method} ndbm.close ()

Close the @code{ndbm} database.
@end deffn
@end deffn

@node dbm dumb — Portable DBM implementation,,dbm ndbm — Interface based on ndbm,dbm — Interfaces to Unix “databases”
@anchor{library/dbm dbm-dumb-portable-dbm-implementation}@anchor{218a}@anchor{library/dbm module-dbm dumb}@anchor{31}
@subsubsection @code{dbm.dumb} — Portable DBM implementation


@geindex module; dbm.dumb

@strong{Source code:} Lib/dbm/dumb.py@footnote{https://github.com/python/cpython/tree/main/Lib/dbm/dumb.py}

@geindex databases

@cartouche
@quotation Note 
The @ref{31,,dbm.dumb} module is intended as a last resort fallback for the
@ref{30,,dbm} module when a more robust module is not available. The @ref{31,,dbm.dumb}
module is not written for speed and is not nearly as heavily used as the other
database modules.
@end quotation
@end cartouche


__________________________________________________________________


The @ref{31,,dbm.dumb} module provides a persistent dictionary-like interface which
is written entirely in Python.  Unlike other modules such as @ref{32,,dbm.gnu} no
external library is required.  As with other persistent mappings, the keys and
values are always stored as bytes.

The module defines the following:

@geindex error
@anchor{library/dbm dbm dumb error}@anchor{5b8}
@deffn {Exception} dbm.dumb.error

Raised on @ref{31,,dbm.dumb}-specific errors, such as I/O errors.  @ref{5bb,,KeyError} is
raised for general mapping errors like specifying an incorrect key.
@end deffn

@geindex open() (in module dbm.dumb)
@anchor{library/dbm dbm dumb open}@anchor{5b4}
@deffn {Function} dbm.dumb.open (filename[, flag[, mode]])

Open a @code{dumbdbm} database and return a dumbdbm object.  The @emph{filename} argument is
the basename of the database file (without any specific extensions).  When a
dumbdbm database is created, files with @code{.dat} and @code{.dir} extensions
are created.

The optional @emph{flag} argument can be:


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Value

@tab

Meaning

@item

@code{'r'}

@tab

Open existing database for reading only
(default)

@item

@code{'w'}

@tab

Open existing database for reading and
writing

@item

@code{'c'}

@tab

Open database for reading and writing,
creating it if it doesn’t exist

@item

@code{'n'}

@tab

Always create a new, empty database, open
for reading and writing

@end multitable


The optional @emph{mode} argument is the Unix mode of the file, used only when the
database has to be created.  It defaults to octal @code{0o666} (and will be modified
by the prevailing umask).

@cartouche
@quotation Warning 
It is possible to crash the Python interpreter when loading a database
with a sufficiently large/complex entry due to stack depth limitations in
Python’s AST compiler.
@end quotation
@end cartouche


Changed in version 3.5: @ref{5b4,,open()} always creates a new database when the flag has the value
@code{'n'}.



Changed in version 3.8: A database opened with flags @code{'r'} is now read-only.  Opening with
flags @code{'r'} and @code{'w'} no longer creates a database if it does not
exist.



Changed in version 3.11: Accepts @ref{63d,,path-like object} for filename.


In addition to the methods provided by the
@ref{c3f,,collections.abc.MutableMapping} class, @code{dumbdbm} objects
provide the following methods:

@geindex sync() (dbm.dumb.dumbdbm method)
@anchor{library/dbm dbm dumb dumbdbm sync}@anchor{218b}
@deffn {Method} dumbdbm.sync ()

Synchronize the on-disk directory and data files.  This method is called
by the @code{Shelve.sync()} method.
@end deffn

@geindex close() (dbm.dumb.dumbdbm method)
@anchor{library/dbm dbm dumb dumbdbm close}@anchor{218c}
@deffn {Method} dumbdbm.close ()

Close the @code{dumbdbm} database.
@end deffn
@end deffn

@node sqlite3 — DB-API 2 0 interface for SQLite databases,,dbm — Interfaces to Unix “databases”,Data Persistence
@anchor{library/sqlite3 doc}@anchor{218d}@anchor{library/sqlite3 module-sqlite3}@anchor{ed}@anchor{library/sqlite3 sqlite3-db-api-2-0-interface-for-sqlite-databases}@anchor{218e}
@subsection @code{sqlite3} — DB-API 2.0 interface for SQLite databases


@geindex module; sqlite3

@strong{Source code:} Lib/sqlite3/@footnote{https://github.com/python/cpython/tree/main/Lib/sqlite3/}


__________________________________________________________________


SQLite is a C library that provides a lightweight disk-based database that
doesn’t require a separate server process and allows accessing the database
using a nonstandard variant of the SQL query language. Some applications can use
SQLite for internal data storage.  It’s also possible to prototype an
application using SQLite and then port the code to a larger database such as
PostgreSQL or Oracle.

The sqlite3 module was written by Gerhard Häring.  It provides a SQL interface
compliant with the DB-API 2.0 specification described by 
@geindex Python Enhancement Proposals; PEP 249
PEP 249@footnote{https://www.python.org/dev/peps/pep-0249}, and
requires SQLite 3.7.15 or newer.

To use the module, start by creating a @ref{18b,,Connection} object that
represents the database.  Here the data will be stored in the
@code{example.db} file:

@example
import sqlite3
con = sqlite3.connect('example.db')
@end example

The special path name @code{:memory:} can be provided to create a temporary
database in RAM.

Once a @ref{18b,,Connection} has been established, create a @ref{218f,,Cursor} object
and call its @ref{2190,,execute()} method to perform SQL commands:

@example
cur = con.cursor()

# Create table
cur.execute('@w{'}'CREATE TABLE stocks
               (date text, trans text, symbol text, qty real, price real)'@w{'}')

# Insert a row of data
cur.execute("INSERT INTO stocks VALUES ('2006-01-05','BUY','RHAT',100,35.14)")

# Save (commit) the changes
con.commit()

# We can also close the connection if we are done with it.
# Just be sure any changes have been committed or they will be lost.
con.close()
@end example

The saved data is persistent: it can be reloaded in a subsequent session even
after restarting the Python interpreter:

@example
import sqlite3
con = sqlite3.connect('example.db')
cur = con.cursor()
@end example

To retrieve data after executing a SELECT statement, either treat the cursor as
an @ref{11c3,,iterator}, call the cursor’s @ref{2191,,fetchone()} method to
retrieve a single matching row, or call @ref{2192,,fetchall()} to get a list
of the matching rows.

This example uses the iterator form:

@example
>>> for row in cur.execute('SELECT * FROM stocks ORDER BY price'):
        print(row)

('2006-01-05', 'BUY', 'RHAT', 100, 35.14)
('2006-03-28', 'BUY', 'IBM', 1000, 45.0)
('2006-04-06', 'SELL', 'IBM', 500, 53.0)
('2006-04-05', 'BUY', 'MSFT', 1000, 72.0)
@end example
@anchor{library/sqlite3 sqlite3-placeholders}@anchor{2193}
SQL operations usually need to use values from Python variables. However,
beware of using Python’s string operations to assemble queries, as they
are vulnerable to SQL injection attacks (see the xkcd webcomic@footnote{https://xkcd.com/327/} for a humorous example of what can go wrong):

@example
# Never do this -- insecure!
symbol = 'RHAT'
cur.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol)
@end example

Instead, use the DB-API’s parameter substitution. To insert a variable into a
query string, use a placeholder in the string, and substitute the actual values
into the query by providing them as a @ref{3e5,,tuple} of values to the second
argument of the cursor’s @ref{2190,,execute()} method. An SQL statement may
use one of two kinds of placeholders: question marks (qmark style) or named
placeholders (named style). For the qmark style, @code{parameters} must be a
@ref{14e7,,sequence}. For the named style, it can be either a
@ref{14e7,,sequence} or @ref{3b0,,dict} instance. The length of the
@ref{14e7,,sequence} must match the number of placeholders, or a
@ref{186,,ProgrammingError} is raised. If a @ref{3b0,,dict} is given, it must contain
keys for all named parameters. Any extra items are ignored. Here’s an example of
both styles:

@example
import sqlite3

con = sqlite3.connect(":memory:")
cur = con.cursor()
cur.execute("create table lang (name, first_appeared)")

# This is the qmark style:
cur.execute("insert into lang values (?, ?)", ("C", 1972))

# The qmark style used with executemany():
lang_list = [
    ("Fortran", 1957),
    ("Python", 1991),
    ("Go", 2009),
]
cur.executemany("insert into lang values (?, ?)", lang_list)

# And this is the named style:
cur.execute("select * from lang where first_appeared=:year", @{"year": 1972@})
print(cur.fetchall())

con.close()
@end example


@subsubheading See also



@table @asis

@item @indicateurl{https://www.sqlite.org}

The SQLite web page; the documentation describes the syntax and the
available data types for the supported SQL dialect.

@item @indicateurl{https://www.w3schools.com/sql/}

Tutorial, reference and examples for learning SQL syntax.

@geindex Python Enhancement Proposals; PEP 249

@item PEP 249@footnote{https://www.python.org/dev/peps/pep-0249} - Database API Specification 2.0

PEP written by Marc-André Lemburg.
@end table


@menu
* Module functions and constants:: 
* Connection Objects:: 
* Cursor Objects:: 
* Row Objects:: 
* Exceptions: Exceptions<5>. 
* SQLite and Python types:: 
* Controlling Transactions:: 
* Using sqlite3 efficiently:: 

@end menu

@node Module functions and constants,Connection Objects,,sqlite3 — DB-API 2 0 interface for SQLite databases
@anchor{library/sqlite3 module-functions-and-constants}@anchor{2194}@anchor{library/sqlite3 sqlite3-module-contents}@anchor{2195}
@subsubsection Module functions and constants


@geindex apilevel (in module sqlite3)
@anchor{library/sqlite3 sqlite3 apilevel}@anchor{2196}
@deffn {Data} sqlite3.apilevel

String constant stating the supported DB-API level. Required by the DB-API.
Hard-coded to @code{"2.0"}.
@end deffn

@geindex paramstyle (in module sqlite3)
@anchor{library/sqlite3 sqlite3 paramstyle}@anchor{2197}
@deffn {Data} sqlite3.paramstyle

String constant stating the type of parameter marker formatting expected by
the @ref{ed,,sqlite3} module. Required by the DB-API. Hard-coded to
@code{"qmark"}.

@cartouche
@quotation Note 
The @ref{ed,,sqlite3} module supports both @code{qmark} and @code{numeric} DB-API
parameter styles, because that is what the underlying SQLite library
supports. However, the DB-API does not allow multiple values for
the @code{paramstyle} attribute.
@end quotation
@end cartouche
@end deffn

@geindex version (in module sqlite3)
@anchor{library/sqlite3 sqlite3 version}@anchor{2198}
@deffn {Data} sqlite3.version

The version number of this module, as a string. This is not the version of
the SQLite library.
@end deffn

@geindex version_info (in module sqlite3)
@anchor{library/sqlite3 sqlite3 version_info}@anchor{2199}
@deffn {Data} sqlite3.version_info

The version number of this module, as a tuple of integers. This is not the
version of the SQLite library.
@end deffn

@geindex sqlite_version (in module sqlite3)
@anchor{library/sqlite3 sqlite3 sqlite_version}@anchor{219a}
@deffn {Data} sqlite3.sqlite_version

The version number of the run-time SQLite library, as a string.
@end deffn

@geindex sqlite_version_info (in module sqlite3)
@anchor{library/sqlite3 sqlite3 sqlite_version_info}@anchor{219b}
@deffn {Data} sqlite3.sqlite_version_info

The version number of the run-time SQLite library, as a tuple of integers.
@end deffn

@geindex threadsafety (in module sqlite3)
@anchor{library/sqlite3 sqlite3 threadsafety}@anchor{18c}
@deffn {Data} sqlite3.threadsafety

Integer constant required by the DB-API 2.0, stating the level of thread
safety the @ref{ed,,sqlite3} module supports. This attribute is set based on
the default threading mode@footnote{https://sqlite.org/threadsafe.html} the
underlying SQLite library is compiled with. The SQLite threading modes are:

@quotation


@enumerate 

@item 
@strong{Single-thread}: In this mode, all mutexes are disabled and SQLite is
unsafe to use in more than a single thread at once.

@item 
@strong{Multi-thread}: In this mode, SQLite can be safely used by multiple
threads provided that no single database connection is used
simultaneously in two or more threads.

@item 
@strong{Serialized}: In serialized mode, SQLite can be safely used by
multiple threads with no restriction.
@end enumerate
@end quotation

The mappings from SQLite threading modes to DB-API 2.0 threadsafety levels
are as follows:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

SQLite threading
mode

@tab

threadsafety@footnote{https://www.python.org/dev/peps/pep-0249/#threadsafety}

@tab

SQLITE_THREADSAFE@footnote{https://sqlite.org/compile.html#threadsafe}

@tab

DB-API 2.0 meaning

@item

single-thread

@tab

0

@tab

0

@tab

Threads may not share the
module

@item

multi-thread

@tab

1

@tab

2

@tab

Threads may share the module,
but not connections

@item

serialized

@tab

3

@tab

1

@tab

Threads may share the module,
connections and cursors

@end multitable



Changed in version 3.11: Set @emph{threadsafety} dynamically instead of hard-coding it to @code{1}.

@end deffn

@geindex PARSE_DECLTYPES (in module sqlite3)
@anchor{library/sqlite3 sqlite3 PARSE_DECLTYPES}@anchor{219c}
@deffn {Data} sqlite3.PARSE_DECLTYPES

This constant is meant to be used with the @emph{detect_types} parameter of the
@ref{6a8,,connect()} function.

Setting it makes the @ref{ed,,sqlite3} module parse the declared type for each
column it returns.  It will parse out the first word of the declared type,
i. e.  for “integer primary key”, it will parse out “integer”, or for
“number(10)” it will parse out “number”. Then for that column, it will look
into the converters dictionary and use the converter function registered for
that type there.
@end deffn

@geindex PARSE_COLNAMES (in module sqlite3)
@anchor{library/sqlite3 sqlite3 PARSE_COLNAMES}@anchor{219d}
@deffn {Data} sqlite3.PARSE_COLNAMES

This constant is meant to be used with the @emph{detect_types} parameter of the
@ref{6a8,,connect()} function.

Setting this makes the SQLite interface parse the column name for each column it
returns.  It will look for a string formed [mytype] in there, and then decide
that ‘mytype’ is the type of the column. It will try to find an entry of
‘mytype’ in the converters dictionary and then use the converter function found
there to return the value. The column name found in @ref{219e,,Cursor.description}
does not include the type, i. e. if you use something like
@code{'as "Expiration date [datetime]"'} in your SQL, then we will parse out
everything until the first @code{'['} for the column name and strip
the preceding space: the column name would simply be “Expiration date”.
@end deffn

@geindex connect() (in module sqlite3)
@anchor{library/sqlite3 sqlite3 connect}@anchor{6a8}
@deffn {Function} sqlite3.connect (database[, timeout, detect_types, isolation_level, check_same_thread, factory, cached_statements, uri])

Opens a connection to the SQLite database file @emph{database}. By default returns a
@ref{18b,,Connection} object, unless a custom @emph{factory} is given.

@emph{database} is a @ref{63d,,path-like object} giving the pathname (absolute or
relative to the current  working directory) of the database file to be opened.
You can use @code{":memory:"} to open a database connection to a database that
resides in RAM instead of on disk.

When a database is accessed by multiple connections, and one of the processes
modifies the database, the SQLite database is locked until that transaction is
committed. The @emph{timeout} parameter specifies how long the connection should wait
for the lock to go away until raising an exception. The default for the timeout
parameter is 5.0 (five seconds).

For the @emph{isolation_level} parameter, please see the
@ref{219f,,isolation_level} property of @ref{18b,,Connection} objects.

SQLite natively supports only the types TEXT, INTEGER, REAL, BLOB and NULL. If
you want to use other types you must add support for them yourself. The
@emph{detect_types} parameter and the using custom @strong{converters} registered with the
module-level @ref{21a0,,register_converter()} function allow you to easily do that.

@emph{detect_types} defaults to 0 (i. e. off, no type detection), you can set it to
any combination of @ref{219c,,PARSE_DECLTYPES} and @ref{219d,,PARSE_COLNAMES} to turn
type detection on. Due to SQLite behaviour, types can’t be detected for generated
fields (for example @code{max(data)}), even when @emph{detect_types} parameter is set. In
such case, the returned type is @ref{2ef,,str}.

By default, @emph{check_same_thread} is @ref{748,,True} and only the creating thread may
use the connection. If set @ref{659,,False}, the returned connection may be shared
across multiple threads. When using multiple threads with the same connection
writing operations should be serialized by the user to avoid data corruption.

By default, the @ref{ed,,sqlite3} module uses its @ref{18b,,Connection} class for the
connect call.  You can, however, subclass the @ref{18b,,Connection} class and make
@ref{6a8,,connect()} use your class instead by providing your class for the @emph{factory}
parameter.

Consult the section @ref{21a1,,SQLite and Python types} of this manual for details.

The @ref{ed,,sqlite3} module internally uses a statement cache to avoid SQL parsing
overhead. If you want to explicitly set the number of statements that are cached
for the connection, you can set the @emph{cached_statements} parameter. The currently
implemented default is to cache 128 statements.

If @emph{uri} is @ref{748,,True}, @emph{database} is interpreted as a
@abbr{URI,Uniform Resource Identifier} with a file path and an optional
query string.  The scheme part @emph{must} be @code{"file:"}.  The path can be a
relative or absolute file path.  The query string allows us to pass
parameters to SQLite. Some useful URI tricks include:

@example
# Open a database in read-only mode.
con = sqlite3.connect("file:template.db?mode=ro", uri=True)

# Don't implicitly create a new database file if it does not already exist.
# Will raise sqlite3.OperationalError if unable to open a database file.
con = sqlite3.connect("file:nosuchdb.db?mode=rw", uri=True)

# Create a shared named in-memory database.
con1 = sqlite3.connect("file:mem1?mode=memory&cache=shared", uri=True)
con2 = sqlite3.connect("file:mem1?mode=memory&cache=shared", uri=True)
con1.executescript("create table t(t); insert into t values(28);")
rows = con2.execute("select * from t").fetchall()
@end example

More information about this feature, including a list of recognized
parameters, can be found in the
SQLite URI documentation@footnote{https://www.sqlite.org/uri.html}.

Raises an @ref{1130,,auditing event} @code{sqlite3.connect} with argument @code{database}.

Raises an @ref{1130,,auditing event} @code{sqlite3.connect/handle} with argument @code{connection_handle}.


Changed in version 3.4: Added the @emph{uri} parameter.



Changed in version 3.7: @emph{database} can now also be a @ref{63d,,path-like object}, not only a string.



Changed in version 3.10: Added the @code{sqlite3.connect/handle} auditing event.

@end deffn

@geindex register_converter() (in module sqlite3)
@anchor{library/sqlite3 sqlite3 register_converter}@anchor{21a0}
@deffn {Function} sqlite3.register_converter (typename, callable)

Registers a callable to convert a bytestring from the database into a custom
Python type. The callable will be invoked for all database values that are of
the type @emph{typename}. Confer the parameter @emph{detect_types} of the @ref{6a8,,connect()}
function for how the type detection works. Note that @emph{typename} and the name of
the type in your query are matched in case-insensitive manner.
@end deffn

@geindex register_adapter() (in module sqlite3)
@anchor{library/sqlite3 sqlite3 register_adapter}@anchor{21a2}
@deffn {Function} sqlite3.register_adapter (type, callable)

Registers a callable to convert the custom Python type @emph{type} into one of
SQLite’s supported types. The callable @emph{callable} accepts as single parameter
the Python value, and must return a value of the following types: int,
float, str or bytes.
@end deffn

@geindex complete_statement() (in module sqlite3)
@anchor{library/sqlite3 sqlite3 complete_statement}@anchor{21a3}
@deffn {Function} sqlite3.complete_statement (sql)

Returns @ref{748,,True} if the string @emph{sql} contains one or more complete SQL
statements terminated by semicolons. It does not verify that the SQL is
syntactically correct, only that there are no unclosed string literals and the
statement is terminated by a semicolon.

This can be used to build a shell for SQLite, as in the following example:

@example
# A minimal SQLite shell for experiments

import sqlite3

con = sqlite3.connect(":memory:")
con.isolation_level = None
cur = con.cursor()

buffer = ""

print("Enter your SQL commands to execute in sqlite3.")
print("Enter a blank line to exit.")

while True:
    line = input()
    if line == "":
        break
    buffer += line
    if sqlite3.complete_statement(buffer):
        try:
            buffer = buffer.strip()
            cur.execute(buffer)

            if buffer.lstrip().upper().startswith("SELECT"):
                print(cur.fetchall())
        except sqlite3.Error as e:
            err_msg = str(e)
            err_code = e.sqlite_errorcode
            err_name = e.sqlite_errorname
            print(f"@{err_name@} (@{err_code@}): @{err_msg@}")
        buffer = ""

con.close()
@end example
@end deffn

@geindex enable_callback_tracebacks() (in module sqlite3)
@anchor{library/sqlite3 sqlite3 enable_callback_tracebacks}@anchor{21a4}
@deffn {Function} sqlite3.enable_callback_tracebacks (flag)

By default you will not get any tracebacks in user-defined functions,
aggregates, converters, authorizer callbacks etc. If you want to debug them,
you can call this function with @emph{flag} set to @ref{748,,True}. Afterwards, you
will get tracebacks from callbacks on @ref{42d,,sys.stderr}. Use @ref{659,,False}
to disable the feature again.

Register an @ref{18d,,unraisable hook handler} for an
improved debug experience:

@example
>>> import sqlite3
>>> sqlite3.enable_callback_tracebacks(True)
>>> cx = sqlite3.connect(":memory:")
>>> cx.set_trace_callback(lambda stmt: 5/0)
>>> cx.execute("select 1")
Exception ignored in: <function <lambda> at 0x10b4e3ee0>
Traceback (most recent call last):
  File "<stdin>", line 1, in <lambda>
ZeroDivisionError: division by zero
>>> import sys
>>> sys.unraisablehook = lambda unraisable: print(unraisable)
>>> cx.execute("select 1")
UnraisableHookArgs(exc_type=<class 'ZeroDivisionError'>, exc_value=ZeroDivisionError('division by zero'), exc_traceback=<traceback object at 0x10b559900>, err_msg=None, object=<function <lambda> at 0x10b4e3ee0>)
<sqlite3.Cursor object at 0x10b1fe840>
@end example
@end deffn

@node Connection Objects,Cursor Objects,Module functions and constants,sqlite3 — DB-API 2 0 interface for SQLite databases
@anchor{library/sqlite3 connection-objects}@anchor{21a5}@anchor{library/sqlite3 sqlite3-connection-objects}@anchor{21a6}
@subsubsection Connection Objects


@geindex Connection (class in sqlite3)
@anchor{library/sqlite3 sqlite3 Connection}@anchor{18b}
@deffn {Class} sqlite3.Connection

A SQLite database connection has the following attributes and methods:

@geindex isolation_level (sqlite3.Connection attribute)
@anchor{library/sqlite3 sqlite3 Connection isolation_level}@anchor{219f}
@deffn {Attribute} isolation_level

Get or set the current default isolation level. @ref{182,,None} for autocommit mode or
one of “DEFERRED”, “IMMEDIATE” or “EXCLUSIVE”. See section
@ref{21a7,,Controlling Transactions} for a more detailed explanation.
@end deffn

@geindex in_transaction (sqlite3.Connection attribute)
@anchor{library/sqlite3 sqlite3 Connection in_transaction}@anchor{21a8}
@deffn {Attribute} in_transaction

@ref{748,,True} if a transaction is active (there are uncommitted changes),
@ref{659,,False} otherwise.  Read-only attribute.


New in version 3.2.

@end deffn

@geindex cursor() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection cursor}@anchor{21a9}
@deffn {Method} cursor (factory=Cursor)

The cursor method accepts a single optional parameter @emph{factory}. If
supplied, this must be a callable returning an instance of @ref{218f,,Cursor}
or its subclasses.
@end deffn

@geindex commit() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection commit}@anchor{21aa}
@deffn {Method} commit ()

This method commits the current transaction. If you don’t call this method,
anything you did since the last call to @code{commit()} is not visible from
other database connections. If you wonder why you don’t see the data you’ve
written to the database, please check you didn’t forget to call this method.
@end deffn

@geindex rollback() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection rollback}@anchor{21ab}
@deffn {Method} rollback ()

This method rolls back any changes to the database since the last call to
@ref{21aa,,commit()}.
@end deffn

@geindex close() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection close}@anchor{21ac}
@deffn {Method} close ()

This closes the database connection. Note that this does not automatically
call @ref{21aa,,commit()}. If you just close your database connection without
calling @ref{21aa,,commit()} first, your changes will be lost!
@end deffn

@geindex execute() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection execute}@anchor{21ad}
@deffn {Method} execute (sql[, parameters])

This is a nonstandard shortcut that creates a cursor object by calling
the @ref{21a9,,cursor()} method, calls the cursor’s
@ref{2190,,execute()} method with the @emph{parameters} given, and returns
the cursor.
@end deffn

@geindex executemany() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection executemany}@anchor{21ae}
@deffn {Method} executemany (sql[, parameters])

This is a nonstandard shortcut that creates a cursor object by
calling the @ref{21a9,,cursor()} method, calls the cursor’s
@ref{21af,,executemany()} method with the @emph{parameters} given, and
returns the cursor.
@end deffn

@geindex executescript() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection executescript}@anchor{21b0}
@deffn {Method} executescript (sql_script)

This is a nonstandard shortcut that creates a cursor object by
calling the @ref{21a9,,cursor()} method, calls the cursor’s
@ref{21b1,,executescript()} method with the given @emph{sql_script}, and
returns the cursor.
@end deffn

@geindex create_function() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection create_function}@anchor{126d}
@deffn {Method} create_function (name, num_params, func, *, deterministic=False)

Creates a user-defined function that you can later use from within SQL
statements under the function name @emph{name}. @emph{num_params} is the number of
parameters the function accepts (if @emph{num_params} is -1, the function may
take any number of arguments), and @emph{func} is a Python callable that is
called as the SQL function. If @emph{deterministic} is true, the created function
is marked as deterministic@footnote{https://sqlite.org/deterministic.html}, which
allows SQLite to perform additional optimizations. This flag is supported by
SQLite 3.8.3 or higher, @ref{21b2,,NotSupportedError} will be raised if used
with older versions.

The function can return any of the types supported by SQLite: bytes, str, int,
float and @code{None}.


Changed in version 3.8: The @emph{deterministic} parameter was added.


Example:

@example
import sqlite3
import hashlib

def md5sum(t):
    return hashlib.md5(t).hexdigest()

con = sqlite3.connect(":memory:")
con.create_function("md5", 1, md5sum)
cur = con.cursor()
cur.execute("select md5(?)", (b"foo",))
print(cur.fetchone()[0])

con.close()
@end example
@end deffn

@geindex create_aggregate() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection create_aggregate}@anchor{126c}
@deffn {Method} create_aggregate (name, num_params, aggregate_class)

Creates a user-defined aggregate function.

The aggregate class must implement a @code{step} method, which accepts the number
of parameters @emph{num_params} (if @emph{num_params} is -1, the function may take
any number of arguments), and a @code{finalize} method which will return the
final result of the aggregate.

The @code{finalize} method can return any of the types supported by SQLite:
bytes, str, int, float and @code{None}.

Example:

@example
import sqlite3

class MySum:
    def __init__(self):
        self.count = 0

    def step(self, value):
        self.count += value

    def finalize(self):
        return self.count

con = sqlite3.connect(":memory:")
con.create_aggregate("mysum", 1, MySum)
cur = con.cursor()
cur.execute("create table test(i)")
cur.execute("insert into test(i) values (1)")
cur.execute("insert into test(i) values (2)")
cur.execute("select mysum(i) from test")
print(cur.fetchone()[0])

con.close()
@end example
@end deffn

@geindex create_collation() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection create_collation}@anchor{184}
@deffn {Method} create_collation (name, callable)

Creates a collation with the specified @emph{name} and @emph{callable}. The callable will
be passed two string arguments. It should return -1 if the first is ordered
lower than the second, 0 if they are ordered equal and 1 if the first is ordered
higher than the second.  Note that this controls sorting (ORDER BY in SQL) so
your comparisons don’t affect other SQL operations.

Note that the callable will get its parameters as Python bytestrings, which will
normally be encoded in UTF-8.

The following example shows a custom collation that sorts “the wrong way”:

@example
import sqlite3

def collate_reverse(string1, string2):
    if string1 == string2:
        return 0
    elif string1 < string2:
        return 1
    else:
        return -1

con = sqlite3.connect(":memory:")
con.create_collation("reverse", collate_reverse)

cur = con.cursor()
cur.execute("create table test(x)")
cur.executemany("insert into test(x) values (?)", [("a",), ("b",)])
cur.execute("select x from test order by x collate reverse")
for row in cur:
    print(row)
con.close()
@end example

To remove a collation, call @code{create_collation} with @code{None} as callable:

@example
con.create_collation("reverse", None)
@end example


Changed in version 3.11: The collation name can contain any Unicode character.  Earlier, only
ASCII characters were allowed.

@end deffn

@geindex interrupt() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection interrupt}@anchor{21b3}
@deffn {Method} interrupt ()

You can call this method from a different thread to abort any queries that might
be executing on the connection. The query will then abort and the caller will
get an exception.
@end deffn

@geindex set_authorizer() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection set_authorizer}@anchor{183}
@deffn {Method} set_authorizer (authorizer_callback)

This routine registers a callback. The callback is invoked for each attempt to
access a column of a table in the database. The callback should return
@code{SQLITE_OK} if access is allowed, @code{SQLITE_DENY} if the entire SQL
statement should be aborted with an error and @code{SQLITE_IGNORE} if the
column should be treated as a NULL value. These constants are available in the
@ref{ed,,sqlite3} module.

The first argument to the callback signifies what kind of operation is to be
authorized. The second and third argument will be arguments or @ref{182,,None}
depending on the first argument. The 4th argument is the name of the database
(“main”, “temp”, etc.) if applicable. The 5th argument is the name of the
inner-most trigger or view that is responsible for the access attempt or
@ref{182,,None} if this access attempt is directly from input SQL code.

Please consult the SQLite documentation about the possible values for the first
argument and the meaning of the second and third argument depending on the first
one. All necessary constants are available in the @ref{ed,,sqlite3} module.

Passing @ref{182,,None} as @emph{authorizer_callback} will disable the authorizer.


Changed in version 3.11: Added support for disabling the authorizer using @ref{182,,None}.

@end deffn

@geindex set_progress_handler() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection set_progress_handler}@anchor{126e}
@deffn {Method} set_progress_handler (handler, n)

This routine registers a callback. The callback is invoked for every @emph{n}
instructions of the SQLite virtual machine. This is useful if you want to
get called from SQLite during long-running operations, for example to update
a GUI.

If you want to clear any previously installed progress handler, call the
method with @ref{182,,None} for @emph{handler}.

Returning a non-zero value from the handler function will terminate the
currently executing query and cause it to raise an @ref{1103,,OperationalError}
exception.
@end deffn

@geindex set_trace_callback() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection set_trace_callback}@anchor{cd6}
@deffn {Method} set_trace_callback (trace_callback)

Registers @emph{trace_callback} to be called for each SQL statement that is
actually executed by the SQLite backend.

The only argument passed to the callback is the statement (as
@ref{2ef,,str}) that is being executed. The return value of the callback is
ignored. Note that the backend does not only run statements passed to the
@ref{2190,,Cursor.execute()} methods.  Other sources include the
@ref{21a7,,transaction management} of the
sqlite3 module and the execution of triggers defined in the current
database.

Passing @ref{182,,None} as @emph{trace_callback} will disable the trace callback.

@cartouche
@quotation Note 
Exceptions raised in the trace callback are not propagated. As a
development and debugging aid, use
@ref{21a4,,enable_callback_tracebacks()} to enable printing
tracebacks from exceptions raised in the trace callback.
@end quotation
@end cartouche


New in version 3.3.

@end deffn

@geindex enable_load_extension() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection enable_load_extension}@anchor{2c3}
@deffn {Method} enable_load_extension (enabled)

This routine allows/disallows the SQLite engine to load SQLite extensions
from shared libraries.  SQLite extensions can define new functions,
aggregates or whole new virtual table implementations.  One well-known
extension is the fulltext-search extension distributed with SQLite.

Loadable extensions are disabled by default. See @footnote{@w{(1)} 
The sqlite3 module is not built with loadable extension support by
default, because some platforms (notably macOS) have SQLite
libraries which are compiled without this feature. To get loadable
extension support, you must pass the
@ref{1640,,--enable-loadable-sqlite-extensions} option to configure.
}.

Raises an @ref{1130,,auditing event} @code{sqlite3.enable_load_extension} with arguments @code{connection}, @code{enabled}.


New in version 3.2.



Changed in version 3.10: Added the @code{sqlite3.enable_load_extension} auditing event.


@example
import sqlite3

con = sqlite3.connect(":memory:")

# enable extension loading
con.enable_load_extension(True)

# Load the fulltext search extension
con.execute("select load_extension('./fts3.so')")

# alternatively you can load the extension using an API call:
# con.load_extension("./fts3.so")

# disable extension loading again
con.enable_load_extension(False)

# example from SQLite wiki
con.execute("create virtual table recipe using fts3(name, ingredients)")
con.executescript("""
    insert into recipe (name, ingredients) values ('broccoli stew', 'broccoli peppers cheese tomatoes');
    insert into recipe (name, ingredients) values ('pumpkin stew', 'pumpkin onions garlic celery');
    insert into recipe (name, ingredients) values ('broccoli pie', 'broccoli cheese onions flour');
    insert into recipe (name, ingredients) values ('pumpkin pie', 'pumpkin sugar flour butter');
    """)
for row in con.execute("select rowid, name, ingredients from recipe where name match 'pie'"):
    print(row)

con.close()
@end example
@end deffn

@geindex load_extension() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection load_extension}@anchor{2c4}
@deffn {Method} load_extension (path)

This routine loads a SQLite extension from a shared library.  You have to
enable extension loading with @ref{2c3,,enable_load_extension()} before you can
use this routine.

Loadable extensions are disabled by default. See @footnote{@w{(1)} 
The sqlite3 module is not built with loadable extension support by
default, because some platforms (notably macOS) have SQLite
libraries which are compiled without this feature. To get loadable
extension support, you must pass the
@ref{1640,,--enable-loadable-sqlite-extensions} option to configure.
}.

Raises an @ref{1130,,auditing event} @code{sqlite3.load_extension} with arguments @code{connection}, @code{path}.


New in version 3.2.



Changed in version 3.10: Added the @code{sqlite3.load_extension} auditing event.

@end deffn

@geindex row_factory (sqlite3.Connection attribute)
@anchor{library/sqlite3 sqlite3 Connection row_factory}@anchor{21b4}
@deffn {Attribute} row_factory

You can change this attribute to a callable that accepts the cursor and the
original row as a tuple and will return the real result row.  This way, you can
implement more advanced ways of returning results, such  as returning an object
that can also access columns by name.

Example:

@example
import sqlite3

def dict_factory(cursor, row):
    d = @{@}
    for idx, col in enumerate(cursor.description):
        d[col[0]] = row[idx]
    return d

con = sqlite3.connect(":memory:")
con.row_factory = dict_factory
cur = con.cursor()
cur.execute("select 1 as a")
print(cur.fetchone()["a"])

con.close()
@end example

If returning a tuple doesn’t suffice and you want name-based access to
columns, you should consider setting @ref{21b4,,row_factory} to the
highly-optimized @ref{9dd,,sqlite3.Row} type. @ref{9dd,,Row} provides both
index-based and case-insensitive name-based access to columns with almost no
memory overhead. It will probably be better than your own custom
dictionary-based approach or even a db_row based solution.

@c XXX what's a db_row-based solution?
@end deffn

@geindex text_factory (sqlite3.Connection attribute)
@anchor{library/sqlite3 sqlite3 Connection text_factory}@anchor{21b5}
@deffn {Attribute} text_factory

Using this attribute you can control what objects are returned for the @code{TEXT}
data type. By default, this attribute is set to @ref{2ef,,str} and the
@ref{ed,,sqlite3} module will return @ref{2ef,,str} objects for @code{TEXT}.
If you want to return @ref{2f0,,bytes} instead, you can set it to @ref{2f0,,bytes}.

You can also set it to any other callable that accepts a single bytestring
parameter and returns the resulting object.

See the following example code for illustration:

@example
import sqlite3

con = sqlite3.connect(":memory:")
cur = con.cursor()

AUSTRIA = "Österreich"

# by default, rows are returned as str
cur.execute("select ?", (AUSTRIA,))
row = cur.fetchone()
assert row[0] == AUSTRIA

# but we can make sqlite3 always return bytestrings ...
con.text_factory = bytes
cur.execute("select ?", (AUSTRIA,))
row = cur.fetchone()
assert type(row[0]) is bytes
# the bytestrings will be encoded in UTF-8, unless you stored garbage in the
# database ...
assert row[0] == AUSTRIA.encode("utf-8")

# we can also implement a custom text_factory ...
# here we implement one that appends "foo" to all strings
con.text_factory = lambda x: x.decode("utf-8") + "foo"
cur.execute("select ?", ("bar",))
row = cur.fetchone()
assert row[0] == "barfoo"

con.close()
@end example
@end deffn

@geindex total_changes (sqlite3.Connection attribute)
@anchor{library/sqlite3 sqlite3 Connection total_changes}@anchor{21b6}
@deffn {Attribute} total_changes

Returns the total number of database rows that have been modified, inserted, or
deleted since the database connection was opened.
@end deffn

@geindex iterdump() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection iterdump}@anchor{21b7}
@deffn {Method} iterdump ()

Returns an iterator to dump the database in an SQL text format.  Useful when
saving an in-memory database for later restoration.  This function provides
the same capabilities as the @code{.dump} command in the @code{sqlite3}
shell.

Example:

@example
# Convert file existing_db.db to SQL dump file dump.sql
import sqlite3

con = sqlite3.connect('existing_db.db')
with open('dump.sql', 'w') as f:
    for line in con.iterdump():
        f.write('%s\n' % line)
con.close()
@end example
@end deffn

@geindex backup() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection backup}@anchor{6a7}
@deffn {Method} backup (target, *, pages=- 1, progress=None, name='main', sleep=0.25)

This method makes a backup of a SQLite database even while it’s being accessed
by other clients, or concurrently by the same connection.  The copy will be
written into the mandatory argument @emph{target}, that must be another
@ref{18b,,Connection} instance.

By default, or when @emph{pages} is either @code{0} or a negative integer, the entire
database is copied in a single step; otherwise the method performs a loop
copying up to @emph{pages} pages at a time.

If @emph{progress} is specified, it must either be @code{None} or a callable object that
will be executed at each iteration with three integer arguments, respectively
the @emph{status} of the last iteration, the @emph{remaining} number of pages still to be
copied and the @emph{total} number of pages.

The @emph{name} argument specifies the database name that will be copied: it must be
a string containing either @code{"main"}, the default, to indicate the main
database, @code{"temp"} to indicate the temporary database or the name specified
after the @code{AS} keyword in an @code{ATTACH DATABASE} statement for an attached
database.

The @emph{sleep} argument specifies the number of seconds to sleep by between
successive attempts to backup remaining pages, can be specified either as an
integer or a floating point value.

Example 1, copy an existing database into another:

@example
import sqlite3

def progress(status, remaining, total):
    print(f'Copied @{total-remaining@} of @{total@} pages...')

con = sqlite3.connect('existing_db.db')
bck = sqlite3.connect('backup.db')
with bck:
    con.backup(bck, pages=1, progress=progress)
bck.close()
con.close()
@end example

Example 2, copy an existing database into a transient copy:

@example
import sqlite3

source = sqlite3.connect('existing_db.db')
dest = sqlite3.connect(':memory:')
source.backup(dest)
@end example


New in version 3.7.

@end deffn

@geindex getlimit() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection getlimit}@anchor{18a}
@deffn {Method} getlimit (category, /)

Get a connection run-time limit. @emph{category} is the limit category to be
queried.

Example, query the maximum length of an SQL statement:

@example
import sqlite3
con = sqlite3.connect(":memory:")
lim = con.getlimit(sqlite3.SQLITE_LIMIT_SQL_LENGTH)
print(f"SQLITE_LIMIT_SQL_LENGTH=@{lim@}")
@end example


New in version 3.11.

@end deffn

@geindex setlimit() (sqlite3.Connection method)
@anchor{library/sqlite3 sqlite3 Connection setlimit}@anchor{189}
@deffn {Method} setlimit (category, limit, /)

Set a connection run-time limit. @emph{category} is the limit category to be
set. @emph{limit} is the new limit. If the new limit is a negative number, the
limit is unchanged.

Attempts to increase a limit above its hard upper bound are silently
truncated to the hard upper bound. Regardless of whether or not the limit
was changed, the prior value of the limit is returned.

Example, limit the number of attached databases to 1:

@example
import sqlite3
con = sqlite3.connect(":memory:")
con.setlimit(sqlite3.SQLITE_LIMIT_ATTACHED, 1)
@end example


New in version 3.11.

@end deffn
@end deffn

@node Cursor Objects,Row Objects,Connection Objects,sqlite3 — DB-API 2 0 interface for SQLite databases
@anchor{library/sqlite3 cursor-objects}@anchor{21b8}@anchor{library/sqlite3 sqlite3-cursor-objects}@anchor{21b9}
@subsubsection Cursor Objects


@geindex Cursor (class in sqlite3)
@anchor{library/sqlite3 sqlite3 Cursor}@anchor{218f}
@deffn {Class} sqlite3.Cursor

A @ref{218f,,Cursor} instance has the following attributes and methods.

@geindex ? (question mark); in SQL statements

@geindex ; (colon); in SQL statements

@geindex execute() (sqlite3.Cursor method)
@anchor{library/sqlite3 sqlite3 Cursor execute}@anchor{2190}
@deffn {Method} execute (sql[, parameters])

Executes an SQL statement. Values may be bound to the statement using
@ref{2193,,placeholders}.

@ref{2190,,execute()} will only execute a single SQL statement. If you try to execute
more than one statement with it, it will raise a @ref{1102,,Warning}. Use
@ref{21b1,,executescript()} if you want to execute multiple SQL statements with one
call.
@end deffn

@geindex executemany() (sqlite3.Cursor method)
@anchor{library/sqlite3 sqlite3 Cursor executemany}@anchor{21af}
@deffn {Method} executemany (sql, seq_of_parameters)

Executes a @ref{2193,,parameterized} SQL command
against all parameter sequences or mappings found in the sequence
@emph{seq_of_parameters}. The @ref{ed,,sqlite3} module also allows using an
@ref{11c3,,iterator} yielding parameters instead of a sequence.

@example
import sqlite3

class IterChars:
    def __init__(self):
        self.count = ord('a')

    def __iter__(self):
        return self

    def __next__(self):
        if self.count > ord('z'):
            raise StopIteration
        self.count += 1
        return (chr(self.count - 1),) # this is a 1-tuple

con = sqlite3.connect(":memory:")
cur = con.cursor()
cur.execute("create table characters(c)")

theIter = IterChars()
cur.executemany("insert into characters(c) values (?)", theIter)

cur.execute("select c from characters")
print(cur.fetchall())

con.close()
@end example

Here’s a shorter example using a @ref{bf7,,generator}:

@example
import sqlite3
import string

def char_generator():
    for c in string.ascii_lowercase:
        yield (c,)

con = sqlite3.connect(":memory:")
cur = con.cursor()
cur.execute("create table characters(c)")

cur.executemany("insert into characters(c) values (?)", char_generator())

cur.execute("select c from characters")
print(cur.fetchall())

con.close()
@end example
@end deffn

@geindex executescript() (sqlite3.Cursor method)
@anchor{library/sqlite3 sqlite3 Cursor executescript}@anchor{21b1}
@deffn {Method} executescript (sql_script)

This is a nonstandard convenience method for executing multiple SQL statements
at once. It issues a @code{COMMIT} statement first, then executes the SQL script it
gets as a parameter.  This method disregards @code{isolation_level}; any
transaction control must be added to @emph{sql_script}.

@emph{sql_script} can be an instance of @ref{2ef,,str}.

Example:

@example
import sqlite3

con = sqlite3.connect(":memory:")
cur = con.cursor()
cur.executescript("""
    create table person(
        firstname,
        lastname,
        age
    );

    create table book(
        title,
        author,
        published
    );

    insert into book(title, author, published)
    values (
        'Dirk Gently'@w{'}s Holistic Detective Agency',
        'Douglas Adams',
        1987
    );
    """)
con.close()
@end example
@end deffn

@geindex fetchone() (sqlite3.Cursor method)
@anchor{library/sqlite3 sqlite3 Cursor fetchone}@anchor{2191}
@deffn {Method} fetchone ()

Fetches the next row of a query result set, returning a single sequence,
or @ref{182,,None} when no more data is available.
@end deffn

@geindex fetchmany() (sqlite3.Cursor method)
@anchor{library/sqlite3 sqlite3 Cursor fetchmany}@anchor{21ba}
@deffn {Method} fetchmany (size=cursor.arraysize)

Fetches the next set of rows of a query result, returning a list.  An empty
list is returned when no more rows are available.

The number of rows to fetch per call is specified by the @emph{size} parameter.
If it is not given, the cursor’s arraysize determines the number of rows
to be fetched. The method should try to fetch as many rows as indicated by
the size parameter. If this is not possible due to the specified number of
rows not being available, fewer rows may be returned.

Note there are performance considerations involved with the @emph{size} parameter.
For optimal performance, it is usually best to use the arraysize attribute.
If the @emph{size} parameter is used, then it is best for it to retain the same
value from one @ref{21ba,,fetchmany()} call to the next.
@end deffn

@geindex fetchall() (sqlite3.Cursor method)
@anchor{library/sqlite3 sqlite3 Cursor fetchall}@anchor{2192}
@deffn {Method} fetchall ()

Fetches all (remaining) rows of a query result, returning a list.  Note that
the cursor’s arraysize attribute can affect the performance of this operation.
An empty list is returned when no rows are available.
@end deffn

@geindex close() (sqlite3.Cursor method)
@anchor{library/sqlite3 sqlite3 Cursor close}@anchor{21bb}
@deffn {Method} close ()

Close the cursor now (rather than whenever @code{__del__} is called).

The cursor will be unusable from this point forward; a @ref{186,,ProgrammingError}
exception will be raised if any operation is attempted with the cursor.
@end deffn

@geindex setinputsizes() (sqlite3.Cursor method)
@anchor{library/sqlite3 sqlite3 Cursor setinputsizes}@anchor{21bc}
@deffn {Method} setinputsizes (sizes)

Required by the DB-API. Is a no-op in @ref{ed,,sqlite3}.
@end deffn

@geindex setoutputsize() (sqlite3.Cursor method)
@anchor{library/sqlite3 sqlite3 Cursor setoutputsize}@anchor{21bd}
@deffn {Method} setoutputsize (size[, column])

Required by the DB-API. Is a no-op in @ref{ed,,sqlite3}.
@end deffn

@geindex rowcount (sqlite3.Cursor attribute)
@anchor{library/sqlite3 sqlite3 Cursor rowcount}@anchor{21be}
@deffn {Attribute} rowcount

Although the @ref{218f,,Cursor} class of the @ref{ed,,sqlite3} module implements this
attribute, the database engine’s own support for the determination of “rows
affected”/”rows selected” is quirky.

For @ref{21af,,executemany()} statements, the number of modifications are summed up
into @ref{21be,,rowcount}.

As required by the Python DB API Spec, the @ref{21be,,rowcount} attribute “is -1 in
case no @code{executeXX()} has been performed on the cursor or the rowcount of the
last operation is not determinable by the interface”. This includes @code{SELECT}
statements because we cannot determine the number of rows a query produced
until all rows were fetched.
@end deffn

@geindex lastrowid (sqlite3.Cursor attribute)
@anchor{library/sqlite3 sqlite3 Cursor lastrowid}@anchor{819}
@deffn {Attribute} lastrowid

This read-only attribute provides the row id of the last inserted row. It
is only updated after successful @code{INSERT} or @code{REPLACE} statements
using the @ref{2190,,execute()} method.  For other statements, after
@ref{21af,,executemany()} or @ref{21b1,,executescript()}, or if the insertion failed,
the value of @code{lastrowid} is left unchanged.  The initial value of
@code{lastrowid} is @ref{182,,None}.

@cartouche
@quotation Note 
Inserts into @code{WITHOUT ROWID} tables are not recorded.
@end quotation
@end cartouche


Changed in version 3.6: Added support for the @code{REPLACE} statement.

@end deffn

@geindex arraysize (sqlite3.Cursor attribute)
@anchor{library/sqlite3 sqlite3 Cursor arraysize}@anchor{21bf}
@deffn {Attribute} arraysize

Read/write attribute that controls the number of rows returned by @ref{21ba,,fetchmany()}.
The default value is 1 which means a single row would be fetched per call.
@end deffn

@geindex description (sqlite3.Cursor attribute)
@anchor{library/sqlite3 sqlite3 Cursor description}@anchor{219e}
@deffn {Attribute} description

This read-only attribute provides the column names of the last query. To
remain compatible with the Python DB API, it returns a 7-tuple for each
column where the last six items of each tuple are @ref{182,,None}.

It is set for @code{SELECT} statements without any matching rows as well.
@end deffn

@geindex connection (sqlite3.Cursor attribute)
@anchor{library/sqlite3 sqlite3 Cursor connection}@anchor{21c0}
@deffn {Attribute} connection

This read-only attribute provides the SQLite database @ref{18b,,Connection}
used by the @ref{218f,,Cursor} object.  A @ref{218f,,Cursor} object created by
calling @ref{21a9,,con.cursor()} will have a
@ref{21c0,,connection} attribute that refers to @emph{con}:

@example
>>> con = sqlite3.connect(":memory:")
>>> cur = con.cursor()
>>> cur.connection == con
True
@end example
@end deffn
@end deffn

@node Row Objects,Exceptions<5>,Cursor Objects,sqlite3 — DB-API 2 0 interface for SQLite databases
@anchor{library/sqlite3 row-objects}@anchor{21c1}@anchor{library/sqlite3 sqlite3-row-objects}@anchor{21c2}
@subsubsection Row Objects


@geindex Row (class in sqlite3)
@anchor{library/sqlite3 sqlite3 Row}@anchor{9dd}
@deffn {Class} sqlite3.Row

A @ref{9dd,,Row} instance serves as a highly optimized
@ref{21b4,,row_factory} for @ref{18b,,Connection} objects.
It tries to mimic a tuple in most of its features.

It supports mapping access by column name and index, iteration,
representation, equality testing and @ref{4a6,,len()}.

If two @ref{9dd,,Row} objects have exactly the same columns and their
members are equal, they compare equal.

@geindex keys() (sqlite3.Row method)
@anchor{library/sqlite3 sqlite3 Row keys}@anchor{21c3}
@deffn {Method} keys ()

This method returns a list of column names. Immediately after a query,
it is the first member of each tuple in @ref{219e,,Cursor.description}.
@end deffn


Changed in version 3.5: Added support of slicing.

@end deffn

Let’s assume we initialize a table as in the example given above:

@example
con = sqlite3.connect(":memory:")
cur = con.cursor()
cur.execute('@w{'}'create table stocks
(date text, trans text, symbol text,
 qty real, price real)'@w{'}')
cur.execute("""insert into stocks
            values ('2006-01-05','BUY','RHAT',100,35.14)""")
con.commit()
cur.close()
@end example

Now we plug @ref{9dd,,Row} in:

@example
>>> con.row_factory = sqlite3.Row
>>> cur = con.cursor()
>>> cur.execute('select * from stocks')
<sqlite3.Cursor object at 0x7f4e7dd8fa80>
>>> r = cur.fetchone()
>>> type(r)
<class 'sqlite3.Row'>
>>> tuple(r)
('2006-01-05', 'BUY', 'RHAT', 100.0, 35.14)
>>> len(r)
5
>>> r[2]
'RHAT'
>>> r.keys()
['date', 'trans', 'symbol', 'qty', 'price']
>>> r['qty']
100.0
>>> for member in r:
...     print(member)
...
2006-01-05
BUY
RHAT
100.0
35.14
@end example

@node Exceptions<5>,SQLite and Python types,Row Objects,sqlite3 — DB-API 2 0 interface for SQLite databases
@anchor{library/sqlite3 exceptions}@anchor{21c4}@anchor{library/sqlite3 sqlite3-exceptions}@anchor{21c5}
@subsubsection Exceptions


@geindex Warning
@anchor{library/sqlite3 sqlite3 Warning}@anchor{1102}
@deffn {Exception} sqlite3.Warning

A subclass of @ref{4d8,,Exception}.
@end deffn

@geindex Error
@anchor{library/sqlite3 sqlite3 Error}@anchor{10e5}
@deffn {Exception} sqlite3.Error

The base class of the other exceptions in this module.  It is a subclass
of @ref{4d8,,Exception}.

@geindex sqlite_errorcode (sqlite3.Error attribute)
@anchor{library/sqlite3 sqlite3 Error sqlite_errorcode}@anchor{187}
@deffn {Attribute} sqlite_errorcode

The numeric error code from the
SQLite API@footnote{https://sqlite.org/rescode.html}


New in version 3.11.

@end deffn

@geindex sqlite_errorname (sqlite3.Error attribute)
@anchor{library/sqlite3 sqlite3 Error sqlite_errorname}@anchor{188}
@deffn {Attribute} sqlite_errorname

The symbolic name of the numeric error code
from the SQLite API@footnote{https://sqlite.org/rescode.html}


New in version 3.11.

@end deffn
@end deffn

@geindex DatabaseError
@anchor{library/sqlite3 sqlite3 DatabaseError}@anchor{21c6}
@deffn {Exception} sqlite3.DatabaseError

Exception raised for errors that are related to the database.
@end deffn

@geindex IntegrityError
@anchor{library/sqlite3 sqlite3 IntegrityError}@anchor{21c7}
@deffn {Exception} sqlite3.IntegrityError

Exception raised when the relational integrity of the database is affected,
e.g. a foreign key check fails.  It is a subclass of @ref{21c6,,DatabaseError}.
@end deffn

@geindex ProgrammingError
@anchor{library/sqlite3 sqlite3 ProgrammingError}@anchor{186}
@deffn {Exception} sqlite3.ProgrammingError

Exception raised for programming errors, e.g. table not found or already
exists, syntax error in the SQL statement, wrong number of parameters
specified, etc.  It is a subclass of @ref{21c6,,DatabaseError}.
@end deffn

@geindex OperationalError
@anchor{library/sqlite3 sqlite3 OperationalError}@anchor{1103}
@deffn {Exception} sqlite3.OperationalError

Exception raised for errors that are related to the database’s operation
and not necessarily under the control of the programmer, e.g. an unexpected
disconnect occurs, the data source name is not found, a transaction could
not be processed, etc.  It is a subclass of @ref{21c6,,DatabaseError}.
@end deffn

@geindex NotSupportedError
@anchor{library/sqlite3 sqlite3 NotSupportedError}@anchor{21b2}
@deffn {Exception} sqlite3.NotSupportedError

Exception raised in case a method or database API was used which is not
supported by the database, e.g. calling the @ref{21ab,,rollback()}
method on a connection that does not support transaction or has
transactions turned off.  It is a subclass of @ref{21c6,,DatabaseError}.
@end deffn

@node SQLite and Python types,Controlling Transactions,Exceptions<5>,sqlite3 — DB-API 2 0 interface for SQLite databases
@anchor{library/sqlite3 sqlite-and-python-types}@anchor{21c8}@anchor{library/sqlite3 sqlite3-types}@anchor{21a1}
@subsubsection SQLite and Python types


@menu
* Introduction: Introduction<7>. 
* Using adapters to store additional Python types in SQLite databases:: 
* Converting SQLite values to custom Python types:: 
* Default adapters and converters:: 

@end menu

@node Introduction<7>,Using adapters to store additional Python types in SQLite databases,,SQLite and Python types
@anchor{library/sqlite3 introduction}@anchor{21c9}
@subsubsection Introduction


SQLite natively supports the following types: @code{NULL}, @code{INTEGER},
@code{REAL}, @code{TEXT}, @code{BLOB}.

The following Python types can thus be sent to SQLite without any problem:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxx} 
@headitem

Python type

@tab

SQLite type

@item

@ref{182,,None}

@tab

@code{NULL}

@item

@ref{1b1,,int}

@tab

@code{INTEGER}

@item

@ref{256,,float}

@tab

@code{REAL}

@item

@ref{2ef,,str}

@tab

@code{TEXT}

@item

@ref{2f0,,bytes}

@tab

@code{BLOB}

@end multitable


This is how SQLite types are converted to Python types by default:


@multitable {xxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

SQLite type

@tab

Python type

@item

@code{NULL}

@tab

@ref{182,,None}

@item

@code{INTEGER}

@tab

@ref{1b1,,int}

@item

@code{REAL}

@tab

@ref{256,,float}

@item

@code{TEXT}

@tab

depends on @ref{21b5,,text_factory},
@ref{2ef,,str} by default

@item

@code{BLOB}

@tab

@ref{2f0,,bytes}

@end multitable


The type system of the @ref{ed,,sqlite3} module is extensible in two ways: you can
store additional Python types in a SQLite database via object adaptation, and
you can let the @ref{ed,,sqlite3} module convert SQLite types to different Python
types via converters.

@node Using adapters to store additional Python types in SQLite databases,Converting SQLite values to custom Python types,Introduction<7>,SQLite and Python types
@anchor{library/sqlite3 using-adapters-to-store-additional-python-types-in-sqlite-databases}@anchor{21ca}
@subsubsection Using adapters to store additional Python types in SQLite databases


As described before, SQLite supports only a limited set of types natively. To
use other Python types with SQLite, you must @strong{adapt} them to one of the
sqlite3 module’s supported types for SQLite: one of NoneType, int, float,
str, bytes.

There are two ways to enable the @ref{ed,,sqlite3} module to adapt a custom Python
type to one of the supported ones.

@menu
* Letting your object adapt itself:: 
* Registering an adapter callable:: 

@end menu

@node Letting your object adapt itself,Registering an adapter callable,,Using adapters to store additional Python types in SQLite databases
@anchor{library/sqlite3 letting-your-object-adapt-itself}@anchor{21cb}
@subsubsection Letting your object adapt itself


This is a good approach if you write the class yourself. Let’s suppose you have
a class like this:

@example
class Point:
    def __init__(self, x, y):
        self.x, self.y = x, y
@end example

Now you want to store the point in a single SQLite column.  First you’ll have to
choose one of the supported types to be used for representing the point.
Let’s just use str and separate the coordinates using a semicolon. Then you need
to give your class a method @code{__conform__(self, protocol)} which must return
the converted value. The parameter @emph{protocol} will be @code{PrepareProtocol}.

@example
import sqlite3

class Point:
    def __init__(self, x, y):
        self.x, self.y = x, y

    def __conform__(self, protocol):
        if protocol is sqlite3.PrepareProtocol:
            return "%f;%f" % (self.x, self.y)

con = sqlite3.connect(":memory:")
cur = con.cursor()

p = Point(4.0, -3.2)
cur.execute("select ?", (p,))
print(cur.fetchone()[0])

con.close()
@end example

@node Registering an adapter callable,,Letting your object adapt itself,Using adapters to store additional Python types in SQLite databases
@anchor{library/sqlite3 registering-an-adapter-callable}@anchor{21cc}
@subsubsection Registering an adapter callable


The other possibility is to create a function that converts the type to the
string representation and register the function with @ref{21a2,,register_adapter()}.

@example
import sqlite3

class Point:
    def __init__(self, x, y):
        self.x, self.y = x, y

def adapt_point(point):
    return "%f;%f" % (point.x, point.y)

sqlite3.register_adapter(Point, adapt_point)

con = sqlite3.connect(":memory:")
cur = con.cursor()

p = Point(4.0, -3.2)
cur.execute("select ?", (p,))
print(cur.fetchone()[0])

con.close()
@end example

The @ref{ed,,sqlite3} module has two default adapters for Python’s built-in
@ref{3e2,,datetime.date} and @ref{363,,datetime.datetime} types.  Now let’s suppose
we want to store @ref{363,,datetime.datetime} objects not in ISO representation,
but as a Unix timestamp.

@example
import sqlite3
import datetime
import time

def adapt_datetime(ts):
    return time.mktime(ts.timetuple())

sqlite3.register_adapter(datetime.datetime, adapt_datetime)

con = sqlite3.connect(":memory:")
cur = con.cursor()

now = datetime.datetime.now()
cur.execute("select ?", (now,))
print(cur.fetchone()[0])

con.close()
@end example

@node Converting SQLite values to custom Python types,Default adapters and converters,Using adapters to store additional Python types in SQLite databases,SQLite and Python types
@anchor{library/sqlite3 converting-sqlite-values-to-custom-python-types}@anchor{21cd}
@subsubsection Converting SQLite values to custom Python types


Writing an adapter lets you send custom Python types to SQLite. But to make it
really useful we need to make the Python to SQLite to Python roundtrip work.

Enter converters.

Let’s go back to the @code{Point} class. We stored the x and y coordinates
separated via semicolons as strings in SQLite.

First, we’ll define a converter function that accepts the string as a parameter
and constructs a @code{Point} object from it.

@cartouche
@quotation Note 
Converter functions @strong{always} get called with a @ref{2f0,,bytes} object, no
matter under which data type you sent the value to SQLite.
@end quotation
@end cartouche

@example
def convert_point(s):
    x, y = map(float, s.split(b";"))
    return Point(x, y)
@end example

Now you need to make the @ref{ed,,sqlite3} module know that what you select from
the database is actually a point. There are two ways of doing this:


@itemize *

@item 
Implicitly via the declared type

@item 
Explicitly via the column name
@end itemize

Both ways are described in section @ref{2195,,Module functions and constants}, in the entries
for the constants @ref{219c,,PARSE_DECLTYPES} and @ref{219d,,PARSE_COLNAMES}.

The following example illustrates both approaches.

@example
import sqlite3

class Point:
    def __init__(self, x, y):
        self.x, self.y = x, y

    def __repr__(self):
        return "(%f;%f)" % (self.x, self.y)

def adapt_point(point):
    return ("%f;%f" % (point.x, point.y)).encode('ascii')

def convert_point(s):
    x, y = list(map(float, s.split(b";")))
    return Point(x, y)

# Register the adapter
sqlite3.register_adapter(Point, adapt_point)

# Register the converter
sqlite3.register_converter("point", convert_point)

p = Point(4.0, -3.2)

#########################
# 1) Using declared types
con = sqlite3.connect(":memory:", detect_types=sqlite3.PARSE_DECLTYPES)
cur = con.cursor()
cur.execute("create table test(p point)")

cur.execute("insert into test(p) values (?)", (p,))
cur.execute("select p from test")
print("with declared types:", cur.fetchone()[0])
cur.close()
con.close()

#######################
# 1) Using column names
con = sqlite3.connect(":memory:", detect_types=sqlite3.PARSE_COLNAMES)
cur = con.cursor()
cur.execute("create table test(p)")

cur.execute("insert into test(p) values (?)", (p,))
cur.execute('select p as "p [point]" from test')
print("with column names:", cur.fetchone()[0])
cur.close()
con.close()
@end example

@node Default adapters and converters,,Converting SQLite values to custom Python types,SQLite and Python types
@anchor{library/sqlite3 default-adapters-and-converters}@anchor{21ce}
@subsubsection Default adapters and converters


There are default adapters for the date and datetime types in the datetime
module. They will be sent as ISO dates/ISO timestamps to SQLite.

The default converters are registered under the name “date” for
@ref{3e2,,datetime.date} and under the name “timestamp” for
@ref{363,,datetime.datetime}.

This way, you can use date/timestamps from Python without any additional
fiddling in most cases. The format of the adapters is also compatible with the
experimental SQLite date/time functions.

The following example demonstrates this.

@example
import sqlite3
import datetime

con = sqlite3.connect(":memory:", detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
cur = con.cursor()
cur.execute("create table test(d date, ts timestamp)")

today = datetime.date.today()
now = datetime.datetime.now()

cur.execute("insert into test(d, ts) values (?, ?)", (today, now))
cur.execute("select d, ts from test")
row = cur.fetchone()
print(today, "=>", row[0], type(row[0]))
print(now, "=>", row[1], type(row[1]))

cur.execute('select current_date as "d [date]", current_timestamp as "ts [timestamp]"')
row = cur.fetchone()
print("current_date", row[0], type(row[0]))
print("current_timestamp", row[1], type(row[1]))

con.close()
@end example

If a timestamp stored in SQLite has a fractional part longer than 6
numbers, its value will be truncated to microsecond precision by the
timestamp converter.

@cartouche
@quotation Note 
The default “timestamp” converter ignores UTC offsets in the database and
always returns a naive @ref{363,,datetime.datetime} object. To preserve UTC
offsets in timestamps, either leave converters disabled, or register an
offset-aware converter with @ref{21a0,,register_converter()}.
@end quotation
@end cartouche

@node Controlling Transactions,Using sqlite3 efficiently,SQLite and Python types,sqlite3 — DB-API 2 0 interface for SQLite databases
@anchor{library/sqlite3 controlling-transactions}@anchor{21cf}@anchor{library/sqlite3 sqlite3-controlling-transactions}@anchor{21a7}
@subsubsection Controlling Transactions


The underlying @code{sqlite3} library operates in @code{autocommit} mode by default,
but the Python @ref{ed,,sqlite3} module by default does not.

@code{autocommit} mode means that statements that modify the database take effect
immediately.  A @code{BEGIN} or @code{SAVEPOINT} statement disables @code{autocommit}
mode, and a @code{COMMIT}, a @code{ROLLBACK}, or a @code{RELEASE} that ends the
outermost transaction, turns @code{autocommit} mode back on.

The Python @ref{ed,,sqlite3} module by default issues a @code{BEGIN} statement
implicitly before a Data Modification Language (DML) statement (i.e.
@code{INSERT}/@code{UPDATE}/@code{DELETE}/@code{REPLACE}).

You can control which kind of @code{BEGIN} statements @ref{ed,,sqlite3} implicitly
executes via the @emph{isolation_level} parameter to the @ref{6a8,,connect()}
call, or via the @code{isolation_level} property of connections.
If you specify no @emph{isolation_level}, a plain @code{BEGIN} is used, which is
equivalent to specifying @code{DEFERRED}.  Other possible values are @code{IMMEDIATE}
and @code{EXCLUSIVE}.

You can disable the @ref{ed,,sqlite3} module’s implicit transaction management by
setting @code{isolation_level} to @code{None}.  This will leave the underlying
@code{sqlite3} library operating in @code{autocommit} mode.  You can then completely
control the transaction state by explicitly issuing @code{BEGIN}, @code{ROLLBACK},
@code{SAVEPOINT}, and @code{RELEASE} statements in your code.

Note that @ref{21b1,,executescript()} disregards
@code{isolation_level}; any transaction control must be added explicitly.


Changed in version 3.6: @ref{ed,,sqlite3} used to implicitly commit an open transaction before DDL
statements.  This is no longer the case.


@node Using sqlite3 efficiently,,Controlling Transactions,sqlite3 — DB-API 2 0 interface for SQLite databases
@anchor{library/sqlite3 using-sqlite3-efficiently}@anchor{21d0}
@subsubsection Using @code{sqlite3} efficiently


@menu
* Using shortcut methods:: 
* Accessing columns by name instead of by index:: 
* Using the connection as a context manager:: 

@end menu

@node Using shortcut methods,Accessing columns by name instead of by index,,Using sqlite3 efficiently
@anchor{library/sqlite3 using-shortcut-methods}@anchor{21d1}
@subsubsection Using shortcut methods


Using the nonstandard @code{execute()}, @code{executemany()} and
@code{executescript()} methods of the @ref{18b,,Connection} object, your code can
be written more concisely because you don’t have to create the (often
superfluous) @ref{218f,,Cursor} objects explicitly. Instead, the @ref{218f,,Cursor}
objects are created implicitly and these shortcut methods return the cursor
objects. This way, you can execute a @code{SELECT} statement and iterate over it
directly using only a single call on the @ref{18b,,Connection} object.

@example
import sqlite3

langs = [
    ("C++", 1985),
    ("Objective-C", 1984),
]

con = sqlite3.connect(":memory:")

# Create the table
con.execute("create table lang(name, first_appeared)")

# Fill the table
con.executemany("insert into lang(name, first_appeared) values (?, ?)", langs)

# Print the table contents
for row in con.execute("select name, first_appeared from lang"):
    print(row)

print("I just deleted", con.execute("delete from lang").rowcount, "rows")

# close is not a shortcut method and it's not called automatically,
# so the connection object should be closed manually
con.close()
@end example

@node Accessing columns by name instead of by index,Using the connection as a context manager,Using shortcut methods,Using sqlite3 efficiently
@anchor{library/sqlite3 accessing-columns-by-name-instead-of-by-index}@anchor{21d2}
@subsubsection Accessing columns by name instead of by index


One useful feature of the @ref{ed,,sqlite3} module is the built-in
@ref{9dd,,sqlite3.Row} class designed to be used as a row factory.

Rows wrapped with this class can be accessed both by index (like tuples) and
case-insensitively by name:

@example
import sqlite3

con = sqlite3.connect(":memory:")
con.row_factory = sqlite3.Row

cur = con.cursor()
cur.execute("select 'John' as name, 42 as age")
for row in cur:
    assert row[0] == row["name"]
    assert row["name"] == row["nAmE"]
    assert row[1] == row["age"]
    assert row[1] == row["AgE"]

con.close()
@end example

@node Using the connection as a context manager,,Accessing columns by name instead of by index,Using sqlite3 efficiently
@anchor{library/sqlite3 using-the-connection-as-a-context-manager}@anchor{21d3}
@subsubsection Using the connection as a context manager


Connection objects can be used as context managers
that automatically commit or rollback transactions.  In the event of an
exception, the transaction is rolled back; otherwise, the transaction is
committed:

@example
import sqlite3

con = sqlite3.connect(":memory:")
con.execute("create table lang (id integer primary key, name varchar unique)")

# Successful, con.commit() is called automatically afterwards
with con:
    con.execute("insert into lang(name) values (?)", ("Python",))

# con.rollback() is called after the with block finishes with an exception, the
# exception is still raised and must be caught
try:
    with con:
        con.execute("insert into lang(name) values (?)", ("Python",))
except sqlite3.IntegrityError:
    print("couldn't add Python twice")

# Connection object used as context manager only commits or rollbacks transactions,
# so the connection object should be closed manually
con.close()
@end example

@node Data Compression and Archiving,File Formats,Data Persistence,The Python Standard Library
@anchor{library/archiving doc}@anchor{213f}@anchor{library/archiving archiving}@anchor{21d4}@anchor{library/archiving data-compression-and-archiving}@anchor{21d5}
@section Data Compression and Archiving


The modules described in this chapter support data compression with the zlib,
gzip, bzip2 and lzma algorithms, and the creation of ZIP- and tar-format
archives.  See also @ref{dba,,Archiving operations} provided by the @ref{e4,,shutil}
module.

@menu
* zlib — Compression compatible with gzip:: 
* gzip — Support for gzip files:: 
* bz2 — Support for bzip2 compression:: 
* lzma — Compression using the LZMA algorithm:: 
* zipfile — Work with ZIP archives:: 
* tarfile — Read and write tar archive files:: 

@end menu

@node zlib — Compression compatible with gzip,gzip — Support for gzip files,,Data Compression and Archiving
@anchor{library/zlib doc}@anchor{21d6}@anchor{library/zlib module-zlib}@anchor{14b}@anchor{library/zlib zlib-compression-compatible-with-gzip}@anchor{21d7}
@subsection @code{zlib} — Compression compatible with @code{gzip}


@geindex module; zlib


__________________________________________________________________


For applications that require data compression, the functions in this module
allow compression and decompression, using the zlib library. The zlib library
has its own home page at @indicateurl{https://www.zlib.net}.   There are known
incompatibilities between the Python module and versions of the zlib library
earlier than 1.1.3; 1.1.3 has a security vulnerability@footnote{https://zlib.net/zlib_faq.html#faq33}, so we recommend using
1.1.4 or later.

zlib’s functions have many options and often need to be used in a particular
order.  This documentation doesn’t attempt to cover all of the permutations;
consult the zlib manual at @indicateurl{http://www.zlib.net/manual.html} for authoritative
information.

For reading and writing @code{.gz} files see the @ref{87,,gzip} module.

The available exception and functions in this module are:

@geindex error
@anchor{library/zlib zlib error}@anchor{21d8}
@deffn {Exception} zlib.error

Exception raised on compression and decompression errors.
@end deffn

@geindex adler32() (in module zlib)
@anchor{library/zlib zlib adler32}@anchor{21d9}
@deffn {Function} zlib.adler32 (data[, value])

Computes an Adler-32 checksum of @emph{data}.  (An Adler-32 checksum is almost as
reliable as a CRC32 but can be computed much more quickly.)  The result
is an unsigned 32-bit integer.  If @emph{value} is present, it is used as
the starting value of the checksum; otherwise, a default value of 1
is used.  Passing in @emph{value} allows computing a running checksum over the
concatenation of several inputs.  The algorithm is not cryptographically
strong, and should not be used for authentication or digital signatures.  Since
the algorithm is designed for use as a checksum algorithm, it is not suitable
for use as a general hash algorithm.


Changed in version 3.0: Always returns an unsigned value.
To generate the same numeric value across all Python versions and
platforms, use @code{adler32(data) & 0xffffffff}.

@end deffn

@geindex compress() (in module zlib)
@anchor{library/zlib zlib compress}@anchor{859}
@deffn {Function} zlib.compress (data, /, level=-1, wbits=MAX_WBITS)

Compresses the bytes in @emph{data}, returning a bytes object containing compressed data.
@emph{level} is an integer from @code{0} to @code{9} or @code{-1} controlling the level of compression;
@code{1} (Z_BEST_SPEED) is fastest and produces the least compression, @code{9} (Z_BEST_COMPRESSION)
is slowest and produces the most.  @code{0} (Z_NO_COMPRESSION) is no compression.
The default value is @code{-1} (Z_DEFAULT_COMPRESSION).  Z_DEFAULT_COMPRESSION represents a default
compromise between speed and compression (currently equivalent to level 6).
@anchor{library/zlib compress-wbits}@anchor{21da}
The @emph{wbits} argument controls the size of the history buffer (or the
“window size”) used when compressing data, and whether a header and
trailer is included in the output.  It can take several ranges of values,
defaulting to @code{15} (MAX_WBITS):


@itemize *

@item 
+9 to +15: The base-two logarithm of the window size, which
therefore ranges between 512 and 32768.  Larger values produce
better compression at the expense of greater memory usage.  The
resulting output will include a zlib-specific header and trailer.

@item 
−9 to −15: Uses the absolute value of @emph{wbits} as the
window size logarithm, while producing a raw output stream with no
header or trailing checksum.

@item 
+25 to +31 = 16 + (9 to 15): Uses the low 4 bits of the value as the
window size logarithm, while including a basic @code{gzip} header
and trailing checksum in the output.
@end itemize

Raises the @ref{21d8,,error} exception if any error occurs.


Changed in version 3.6: @emph{level} can now be used as a keyword parameter.



Changed in version 3.11: The @emph{wbits} parameter is now available to set window bits and
compression type.

@end deffn

@geindex compressobj() (in module zlib)
@anchor{library/zlib zlib compressobj}@anchor{21db}
@deffn {Function} zlib.compressobj (level=-1, method=DEFLATED, wbits=MAX_WBITS, memLevel=DEF_MEM_LEVEL, strategy=Z_DEFAULT_STRATEGY[, zdict])

Returns a compression object, to be used for compressing data streams that won’t
fit into memory at once.

@emph{level} is the compression level – an integer from @code{0} to @code{9} or @code{-1}.
A value of @code{1} (Z_BEST_SPEED) is fastest and produces the least compression,
while a value of @code{9} (Z_BEST_COMPRESSION) is slowest and produces the most.
@code{0} (Z_NO_COMPRESSION) is no compression.  The default value is @code{-1} (Z_DEFAULT_COMPRESSION).
Z_DEFAULT_COMPRESSION represents a default compromise between speed and compression
(currently equivalent to level 6).

@emph{method} is the compression algorithm. Currently, the only supported value is
@code{DEFLATED}.

The @emph{wbits} parameter controls the size of the history buffer (or the
“window size”), and what header and trailer format will be used. It has
the same meaning as @ref{21da,,described for compress()}.

The @emph{memLevel} argument controls the amount of memory used for the
internal compression state. Valid values range from @code{1} to @code{9}.
Higher values use more memory, but are faster and produce smaller output.

@emph{strategy} is used to tune the compression algorithm. Possible values are
@code{Z_DEFAULT_STRATEGY}, @code{Z_FILTERED}, @code{Z_HUFFMAN_ONLY},
@code{Z_RLE} (zlib 1.2.0.1) and @code{Z_FIXED} (zlib 1.2.2.2).

@emph{zdict} is a predefined compression dictionary. This is a sequence of bytes
(such as a @ref{2f0,,bytes} object) containing subsequences that are expected
to occur frequently in the data that is to be compressed. Those subsequences
that are expected to be most common should come at the end of the dictionary.


Changed in version 3.3: Added the @emph{zdict} parameter and keyword argument support.

@end deffn

@geindex crc32() (in module zlib)
@anchor{library/zlib zlib crc32}@anchor{21dc}
@deffn {Function} zlib.crc32 (data[, value])

@geindex Cyclic Redundancy Check
@geindex checksum; Cyclic Redundancy Check

Computes a CRC (Cyclic Redundancy Check) checksum of @emph{data}. The
result is an unsigned 32-bit integer. If @emph{value} is present, it is used
as the starting value of the checksum; otherwise, a default value of 0
is used.  Passing in @emph{value} allows computing a running checksum over the
concatenation of several inputs.  The algorithm is not cryptographically
strong, and should not be used for authentication or digital signatures.  Since
the algorithm is designed for use as a checksum algorithm, it is not suitable
for use as a general hash algorithm.


Changed in version 3.0: Always returns an unsigned value.
To generate the same numeric value across all Python versions and
platforms, use @code{crc32(data) & 0xffffffff}.

@end deffn

@geindex decompress() (in module zlib)
@anchor{library/zlib zlib decompress}@anchor{85a}
@deffn {Function} zlib.decompress (data, /, wbits=MAX_WBITS, bufsize=DEF_BUF_SIZE)

Decompresses the bytes in @emph{data}, returning a bytes object containing the
uncompressed data.  The @emph{wbits} parameter depends on
the format of @emph{data}, and is discussed further below.
If @emph{bufsize} is given, it is used as the initial size of the output
buffer.  Raises the @ref{21d8,,error} exception if any error occurs.
@anchor{library/zlib decompress-wbits}@anchor{21dd}
The @emph{wbits} parameter controls the size of the history buffer
(or “window size”), and what header and trailer format is expected.
It is similar to the parameter for @ref{21db,,compressobj()}, but accepts
more ranges of values:


@itemize *

@item 
+8 to +15: The base-two logarithm of the window size.  The input
must include a zlib header and trailer.

@item 
0: Automatically determine the window size from the zlib header.
Only supported since zlib 1.2.3.5.

@item 
−8 to −15: Uses the absolute value of @emph{wbits} as the window size
logarithm.  The input must be a raw stream with no header or trailer.

@item 
+24 to +31 = 16 + (8 to 15): Uses the low 4 bits of the value as
the window size logarithm.  The input must include a gzip header and
trailer.

@item 
+40 to +47 = 32 + (8 to 15): Uses the low 4 bits of the value as
the window size logarithm, and automatically accepts either
the zlib or gzip format.
@end itemize

When decompressing a stream, the window size must not be smaller
than the size originally used to compress the stream; using a too-small
value may result in an @ref{21d8,,error} exception. The default @emph{wbits} value
corresponds to the largest window size and requires a zlib header and
trailer to be included.

@emph{bufsize} is the initial size of the buffer used to hold decompressed data.  If
more space is required, the buffer size will be increased as needed, so you
don’t have to get this value exactly right; tuning it will only save a few calls
to @code{malloc()}.


Changed in version 3.6: @emph{wbits} and @emph{bufsize} can be used as keyword arguments.

@end deffn

@geindex decompressobj() (in module zlib)
@anchor{library/zlib zlib decompressobj}@anchor{21de}
@deffn {Function} zlib.decompressobj (wbits=MAX_WBITS[, zdict])

Returns a decompression object, to be used for decompressing data streams that
won’t fit into memory at once.

The @emph{wbits} parameter controls the size of the history buffer (or the
“window size”), and what header and trailer format is expected.  It has
the same meaning as @ref{21dd,,described for decompress()}.

The @emph{zdict} parameter specifies a predefined compression dictionary. If
provided, this must be the same dictionary as was used by the compressor that
produced the data that is to be decompressed.

@cartouche
@quotation Note 
If @emph{zdict} is a mutable object (such as a @ref{2f1,,bytearray}), you must not
modify its contents between the call to @ref{21de,,decompressobj()} and the first
call to the decompressor’s @code{decompress()} method.
@end quotation
@end cartouche


Changed in version 3.3: Added the @emph{zdict} parameter.

@end deffn

Compression objects support the following methods:

@geindex compress() (zlib.Compress method)
@anchor{library/zlib zlib Compress compress}@anchor{21df}
@deffn {Method} Compress.compress (data)

Compress @emph{data}, returning a bytes object containing compressed data for at least
part of the data in @emph{data}.  This data should be concatenated to the output
produced by any preceding calls to the @ref{859,,compress()} method.  Some input may
be kept in internal buffers for later processing.
@end deffn

@geindex flush() (zlib.Compress method)
@anchor{library/zlib zlib Compress flush}@anchor{21e0}
@deffn {Method} Compress.flush ([mode])

All pending input is processed, and a bytes object containing the remaining compressed
output is returned.  @emph{mode} can be selected from the constants
@code{Z_NO_FLUSH}, @code{Z_PARTIAL_FLUSH}, @code{Z_SYNC_FLUSH},
@code{Z_FULL_FLUSH}, @code{Z_BLOCK} (zlib 1.2.3.4), or @code{Z_FINISH},
defaulting to @code{Z_FINISH}.  Except @code{Z_FINISH}, all constants
allow compressing further bytestrings of data, while @code{Z_FINISH} finishes the
compressed stream and prevents compressing any more data.  After calling @ref{21e0,,flush()}
with @emph{mode} set to @code{Z_FINISH}, the @ref{859,,compress()} method cannot be called again;
the only realistic action is to delete the object.
@end deffn

@geindex copy() (zlib.Compress method)
@anchor{library/zlib zlib Compress copy}@anchor{21e1}
@deffn {Method} Compress.copy ()

Returns a copy of the compression object.  This can be used to efficiently
compress a set of data that share a common initial prefix.
@end deffn


Changed in version 3.8: Added @ref{69a,,copy.copy()} and @ref{69b,,copy.deepcopy()} support to compression
objects.


Decompression objects support the following methods and attributes:

@geindex unused_data (zlib.Decompress attribute)
@anchor{library/zlib zlib Decompress unused_data}@anchor{21e2}
@deffn {Attribute} Decompress.unused_data

A bytes object which contains any bytes past the end of the compressed data. That is,
this remains @code{b""} until the last byte that contains compression data is
available.  If the whole bytestring turned out to contain compressed data, this is
@code{b""}, an empty bytes object.
@end deffn

@geindex unconsumed_tail (zlib.Decompress attribute)
@anchor{library/zlib zlib Decompress unconsumed_tail}@anchor{21e3}
@deffn {Attribute} Decompress.unconsumed_tail

A bytes object that contains any data that was not consumed by the last
@ref{85a,,decompress()} call because it exceeded the limit for the uncompressed data
buffer.  This data has not yet been seen by the zlib machinery, so you must feed
it (possibly with further data concatenated to it) back to a subsequent
@ref{85a,,decompress()} method call in order to get correct output.
@end deffn

@geindex eof (zlib.Decompress attribute)
@anchor{library/zlib zlib Decompress eof}@anchor{d0a}
@deffn {Attribute} Decompress.eof

A boolean indicating whether the end of the compressed data stream has been
reached.

This makes it possible to distinguish between a properly-formed compressed
stream, and an incomplete or truncated one.


New in version 3.3.

@end deffn

@geindex decompress() (zlib.Decompress method)
@anchor{library/zlib zlib Decompress decompress}@anchor{21e4}
@deffn {Method} Decompress.decompress (data, max_length=0)

Decompress @emph{data}, returning a bytes object containing the uncompressed data
corresponding to at least part of the data in @emph{string}.  This data should be
concatenated to the output produced by any preceding calls to the
@ref{85a,,decompress()} method.  Some of the input data may be preserved in internal
buffers for later processing.

If the optional parameter @emph{max_length} is non-zero then the return value will be
no longer than @emph{max_length}. This may mean that not all of the compressed input
can be processed; and unconsumed data will be stored in the attribute
@ref{21e3,,unconsumed_tail}. This bytestring must be passed to a subsequent call to
@ref{85a,,decompress()} if decompression is to continue.  If @emph{max_length} is zero
then the whole input is decompressed, and @ref{21e3,,unconsumed_tail} is empty.


Changed in version 3.6: @emph{max_length} can be used as a keyword argument.

@end deffn

@geindex flush() (zlib.Decompress method)
@anchor{library/zlib zlib Decompress flush}@anchor{21e5}
@deffn {Method} Decompress.flush ([length])

All pending input is processed, and a bytes object containing the remaining
uncompressed output is returned.  After calling @ref{21e5,,flush()}, the
@ref{85a,,decompress()} method cannot be called again; the only realistic action is
to delete the object.

The optional parameter @emph{length} sets the initial size of the output buffer.
@end deffn

@geindex copy() (zlib.Decompress method)
@anchor{library/zlib zlib Decompress copy}@anchor{21e6}
@deffn {Method} Decompress.copy ()

Returns a copy of the decompression object.  This can be used to save the state
of the decompressor midway through the data stream in order to speed up random
seeks into the stream at a future point.
@end deffn


Changed in version 3.8: Added @ref{69a,,copy.copy()} and @ref{69b,,copy.deepcopy()} support to decompression
objects.


Information about the version of the zlib library in use is available through
the following constants:

@geindex ZLIB_VERSION (in module zlib)
@anchor{library/zlib zlib ZLIB_VERSION}@anchor{21e7}
@deffn {Data} zlib.ZLIB_VERSION

The version string of the zlib library that was used for building the module.
This may be different from the zlib library actually used at runtime, which
is available as @ref{d0b,,ZLIB_RUNTIME_VERSION}.
@end deffn

@geindex ZLIB_RUNTIME_VERSION (in module zlib)
@anchor{library/zlib zlib ZLIB_RUNTIME_VERSION}@anchor{d0b}
@deffn {Data} zlib.ZLIB_RUNTIME_VERSION

The version string of the zlib library actually loaded by the interpreter.


New in version 3.3.

@end deffn


@subsubheading See also



@table @asis

@item Module @ref{87,,gzip}

Reading and writing @code{gzip}-format files.

@item @indicateurl{http://www.zlib.net}

The zlib library home page.

@item @indicateurl{http://www.zlib.net/manual.html}

The zlib manual explains  the semantics and usage of the library’s many
functions.
@end table


@node gzip — Support for gzip files,bz2 — Support for bzip2 compression,zlib — Compression compatible with gzip,Data Compression and Archiving
@anchor{library/gzip doc}@anchor{21e8}@anchor{library/gzip gzip-support-for-gzip-files}@anchor{21e9}@anchor{library/gzip module-gzip}@anchor{87}
@subsection @code{gzip} — Support for @code{gzip} files


@geindex module; gzip

@strong{Source code:} Lib/gzip.py@footnote{https://github.com/python/cpython/tree/main/Lib/gzip.py}


__________________________________________________________________


This module provides a simple interface to compress and decompress files just
like the GNU programs @code{gzip} and @code{gunzip} would.

The data compression is provided by the @ref{14b,,zlib} module.

The @ref{87,,gzip} module provides the @ref{43f,,GzipFile} class, as well as the
@ref{21ea,,open()}, @ref{4fc,,compress()} and @ref{dae,,decompress()} convenience functions.
The @ref{43f,,GzipFile} class reads and writes @code{gzip}-format files,
automatically compressing or decompressing the data so that it looks like an
ordinary @ref{d68,,file object}.

Note that additional file formats which can be decompressed by the
@code{gzip} and @code{gunzip} programs, such  as those produced by
@code{compress} and @code{pack}, are not supported by this module.

The module defines the following items:

@geindex open() (in module gzip)
@anchor{library/gzip gzip open}@anchor{21ea}
@deffn {Function} gzip.open (filename, mode='rb', compresslevel=9, encoding=None, errors=None, newline=None)

Open a gzip-compressed file in binary or text mode, returning a @ref{d68,,file object}.

The @emph{filename} argument can be an actual filename (a @ref{2ef,,str} or
@ref{2f0,,bytes} object), or an existing file object to read from or write to.

The @emph{mode} argument can be any of @code{'r'}, @code{'rb'}, @code{'a'}, @code{'ab'},
@code{'w'}, @code{'wb'}, @code{'x'} or @code{'xb'} for binary mode, or @code{'rt'},
@code{'at'}, @code{'wt'}, or @code{'xt'} for text mode. The default is @code{'rb'}.

The @emph{compresslevel} argument is an integer from 0 to 9, as for the
@ref{43f,,GzipFile} constructor.

For binary mode, this function is equivalent to the @ref{43f,,GzipFile}
constructor: @code{GzipFile(filename, mode, compresslevel)}. In this case, the
@emph{encoding}, @emph{errors} and @emph{newline} arguments must not be provided.

For text mode, a @ref{43f,,GzipFile} object is created, and wrapped in an
@ref{889,,io.TextIOWrapper} instance with the specified encoding, error
handling behavior, and line ending(s).


Changed in version 3.3: Added support for @emph{filename} being a file object, support for text mode,
and the @emph{encoding}, @emph{errors} and @emph{newline} arguments.



Changed in version 3.4: Added support for the @code{'x'}, @code{'xb'} and @code{'xt'} modes.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex BadGzipFile
@anchor{library/gzip gzip BadGzipFile}@anchor{4fd}
@deffn {Exception} gzip.BadGzipFile

An exception raised for invalid gzip files.  It inherits @ref{2a1,,OSError}.
@ref{e74,,EOFError} and @ref{21d8,,zlib.error} can also be raised for invalid gzip
files.


New in version 3.8.

@end deffn

@geindex GzipFile (class in gzip)
@anchor{library/gzip gzip GzipFile}@anchor{43f}
@deffn {Class} gzip.GzipFile (filename=None, mode=None, compresslevel=9, fileobj=None, mtime=None)

Constructor for the @ref{43f,,GzipFile} class, which simulates most of the
methods of a @ref{d68,,file object}, with the exception of the @code{truncate()}
method.  At least one of @emph{fileobj} and @emph{filename} must be given a non-trivial
value.

The new class instance is based on @emph{fileobj}, which can be a regular file, an
@ref{a15,,io.BytesIO} object, or any other object which simulates a file.  It
defaults to @code{None}, in which case @emph{filename} is opened to provide a file
object.

When @emph{fileobj} is not @code{None}, the @emph{filename} argument is only used to be
included in the @code{gzip} file header, which may include the original
filename of the uncompressed file.  It defaults to the filename of @emph{fileobj}, if
discernible; otherwise, it defaults to the empty string, and in this case the
original filename is not included in the header.

The @emph{mode} argument can be any of @code{'r'}, @code{'rb'}, @code{'a'}, @code{'ab'}, @code{'w'},
@code{'wb'}, @code{'x'}, or @code{'xb'}, depending on whether the file will be read or
written.  The default is the mode of @emph{fileobj} if discernible; otherwise, the
default is @code{'rb'}.  In future Python releases the mode of @emph{fileobj} will
not be used.  It is better to always specify @emph{mode} for writing.

Note that the file is always opened in binary mode. To open a compressed file
in text mode, use @ref{21ea,,open()} (or wrap your @ref{43f,,GzipFile} with an
@ref{889,,io.TextIOWrapper}).

The @emph{compresslevel} argument is an integer from @code{0} to @code{9} controlling
the level of compression; @code{1} is fastest and produces the least
compression, and @code{9} is slowest and produces the most compression. @code{0}
is no compression. The default is @code{9}.

The @emph{mtime} argument is an optional numeric timestamp to be written to
the last modification time field in the stream when compressing.  It
should only be provided in compression mode.  If omitted or @code{None}, the
current time is used.  See the @ref{21eb,,mtime} attribute for more details.

Calling a @ref{43f,,GzipFile} object’s @code{close()} method does not close
@emph{fileobj}, since you might wish to append more material after the compressed
data.  This also allows you to pass an @ref{a15,,io.BytesIO} object opened for
writing as @emph{fileobj}, and retrieve the resulting memory buffer using the
@ref{a15,,io.BytesIO} object’s @ref{21ec,,getvalue()} method.

@ref{43f,,GzipFile} supports the @ref{823,,io.BufferedIOBase} interface,
including iteration and the @ref{168,,with} statement.  Only the
@code{truncate()} method isn’t implemented.

@ref{43f,,GzipFile} also provides the following method and attribute:

@geindex peek() (gzip.GzipFile method)
@anchor{library/gzip gzip GzipFile peek}@anchor{dad}
@deffn {Method} peek (n)

Read @emph{n} uncompressed bytes without advancing the file position.
At most one single read on the compressed stream is done to satisfy
the call.  The number of bytes returned may be more or less than
requested.

@cartouche
@quotation Note 
While calling @ref{dad,,peek()} does not change the file position of
the @ref{43f,,GzipFile}, it may change the position of the underlying
file object (e.g. if the @ref{43f,,GzipFile} was constructed with the
@emph{fileobj} parameter).
@end quotation
@end cartouche


New in version 3.2.

@end deffn

@geindex mtime (gzip.GzipFile attribute)
@anchor{library/gzip gzip GzipFile mtime}@anchor{21eb}
@deffn {Attribute} mtime

When decompressing, the value of the last modification time field in
the most recently read header may be read from this attribute, as an
integer.  The initial value before reading any headers is @code{None}.

All @code{gzip} compressed streams are required to contain this
timestamp field.  Some programs, such as @code{gunzip}, make use
of the timestamp.  The format is the same as the return value of
@ref{5fc,,time.time()} and the @ref{21ed,,st_mtime} attribute of
the object returned by @ref{516,,os.stat()}.
@end deffn


Changed in version 3.1: Support for the @ref{168,,with} statement was added, along with the
@emph{mtime} constructor argument and @ref{21eb,,mtime} attribute.



Changed in version 3.2: Support for zero-padded and unseekable files was added.



Changed in version 3.3: The @ref{21ee,,io.BufferedIOBase.read1()} method is now implemented.



Changed in version 3.4: Added support for the @code{'x'} and @code{'xb'} modes.



Changed in version 3.5: Added support for writing arbitrary
@ref{87e,,bytes-like objects}.
The @ref{21ef,,read()} method now accepts an argument of
@code{None}.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.



Deprecated since version 3.9: Opening @ref{43f,,GzipFile} for writing without specifying the @emph{mode}
argument is deprecated.

@end deffn

@geindex compress() (in module gzip)
@anchor{library/gzip gzip compress}@anchor{4fc}
@deffn {Function} gzip.compress (data, compresslevel=9, *, mtime=None)

Compress the @emph{data}, returning a @ref{2f0,,bytes} object containing
the compressed data.  @emph{compresslevel} and @emph{mtime} have the same meaning as in
the @ref{43f,,GzipFile} constructor above. When @emph{mtime} is set to @code{0}, this
function is equivalent to @ref{859,,zlib.compress()} with @emph{wbits} set to @code{31}.
The zlib function is faster.


New in version 3.2.



Changed in version 3.8: Added the @emph{mtime} parameter for reproducible output.



Changed in version 3.11: Speed is improved by compressing all data at once instead of in a
streamed fashion. Calls with @emph{mtime} set to @code{0} are delegated to
@ref{859,,zlib.compress()} for better speed.

@end deffn

@geindex decompress() (in module gzip)
@anchor{library/gzip gzip decompress}@anchor{dae}
@deffn {Function} gzip.decompress (data)

Decompress the @emph{data}, returning a @ref{2f0,,bytes} object containing the
uncompressed data. This function is capable of decompressing multi-member
gzip data (multiple gzip blocks concatenated together). When the data is
certain to contain only one member the @ref{85a,,zlib.decompress()} function with
@emph{wbits} set to 31 is faster.


New in version 3.2.



Changed in version 3.11: Speed is improved by decompressing members at once in memory instead of in
a streamed fashion.

@end deffn

@menu
* Examples of usage:: 
* Command Line Interface:: 

@end menu

@node Examples of usage,Command Line Interface,,gzip — Support for gzip files
@anchor{library/gzip examples-of-usage}@anchor{21f0}@anchor{library/gzip gzip-usage-examples}@anchor{21f1}
@subsubsection Examples of usage


Example of how to read a compressed file:

@example
import gzip
with gzip.open('/home/joe/file.txt.gz', 'rb') as f:
    file_content = f.read()
@end example

Example of how to create a compressed GZIP file:

@example
import gzip
content = b"Lots of content here"
with gzip.open('/home/joe/file.txt.gz', 'wb') as f:
    f.write(content)
@end example

Example of how to GZIP compress an existing file:

@example
import gzip
import shutil
with open('/home/joe/file.txt', 'rb') as f_in:
    with gzip.open('/home/joe/file.txt.gz', 'wb') as f_out:
        shutil.copyfileobj(f_in, f_out)
@end example

Example of how to GZIP compress a binary string:

@example
import gzip
s_in = b"Lots of content here"
s_out = gzip.compress(s_in)
@end example


@subsubheading See also



@table @asis

@item Module @ref{14b,,zlib}

The basic data compression module needed to support the @code{gzip} file
format.
@end table


@node Command Line Interface,,Examples of usage,gzip — Support for gzip files
@anchor{library/gzip command-line-interface}@anchor{21f2}
@subsubsection Command Line Interface


The @ref{87,,gzip} module provides a simple command line interface to compress or
decompress files.

Once executed the @ref{87,,gzip} module keeps the input file(s).


Changed in version 3.8: Add a new command line interface with a usage.
By default, when you will execute the CLI, the default compression level is 6.


@menu
* Command line options:: 

@end menu

@node Command line options,,,Command Line Interface
@anchor{library/gzip command-line-options}@anchor{21f3}
@subsubsection Command line options


@geindex gzip command line option; file
@anchor{library/gzip cmdoption-gzip-arg-file}@anchor{21f4}
@deffn {Program Option} file

If @emph{file} is not specified, read from @ref{5ee,,sys.stdin}.
@end deffn

@geindex gzip command line option; --fast
@anchor{library/gzip cmdoption-gzip-fast}@anchor{21f5}
@deffn {Program Option} @w{-}@w{-}fast

Indicates the fastest compression method (less compression).
@end deffn

@geindex gzip command line option; --best
@anchor{library/gzip cmdoption-gzip-best}@anchor{21f6}
@deffn {Program Option} @w{-}@w{-}best

Indicates the slowest compression method (best compression).
@end deffn

@geindex gzip command line option; -d
@geindex gzip command line option; --decompress
@anchor{library/gzip cmdoption-gzip-d}@anchor{21f7}@anchor{library/gzip cmdoption-gzip-decompress}@anchor{21f8}
@deffn {Program Option} @w{-}d, @w{-}@w{-}decompress

Decompress the given file.
@end deffn

@geindex gzip command line option; -h
@geindex gzip command line option; --help
@anchor{library/gzip cmdoption-gzip-h}@anchor{21f9}@anchor{library/gzip cmdoption-gzip-help}@anchor{21fa}
@deffn {Program Option} @w{-}h, @w{-}@w{-}help

Show the help message.
@end deffn

@node bz2 — Support for bzip2 compression,lzma — Compression using the LZMA algorithm,gzip — Support for gzip files,Data Compression and Archiving
@anchor{library/bz2 doc}@anchor{21fb}@anchor{library/bz2 bz2-support-for-bzip2-compression}@anchor{21fc}@anchor{library/bz2 module-bz2}@anchor{12}
@subsection @code{bz2} — Support for @code{bzip2} compression


@geindex module; bz2

@strong{Source code:} Lib/bz2.py@footnote{https://github.com/python/cpython/tree/main/Lib/bz2.py}


__________________________________________________________________


This module provides a comprehensive interface for compressing and
decompressing data using the bzip2 compression algorithm.

The @ref{12,,bz2} module contains:


@itemize *

@item 
The @ref{c26,,open()} function and @ref{455,,BZ2File} class for reading and
writing compressed files.

@item 
The @ref{21fd,,BZ2Compressor} and @ref{21fe,,BZ2Decompressor} classes for
incremental (de)compression.

@item 
The @ref{1cf5,,compress()} and @ref{c27,,decompress()} functions for one-shot
(de)compression.
@end itemize

@menu
* (De)compression of files: De compression of files. 
* Incremental (de)compression: Incremental de compression. 
* One-shot (de)compression: One-shot de compression. 
* Examples of usage: Examples of usage<2>. 

@end menu

@node De compression of files,Incremental de compression,,bz2 — Support for bzip2 compression
@anchor{library/bz2 de-compression-of-files}@anchor{21ff}
@subsubsection (De)compression of files


@geindex open() (in module bz2)
@anchor{library/bz2 bz2 open}@anchor{c26}
@deffn {Function} bz2.open (filename, mode='rb', compresslevel=9, encoding=None, errors=None, newline=None)

Open a bzip2-compressed file in binary or text mode, returning a @ref{d68,,file object}.

As with the constructor for @ref{455,,BZ2File}, the @emph{filename} argument can be
an actual filename (a @ref{2ef,,str} or @ref{2f0,,bytes} object), or an existing
file object to read from or write to.

The @emph{mode} argument can be any of @code{'r'}, @code{'rb'}, @code{'w'}, @code{'wb'},
@code{'x'}, @code{'xb'}, @code{'a'} or @code{'ab'} for binary mode, or @code{'rt'},
@code{'wt'}, @code{'xt'}, or @code{'at'} for text mode. The default is @code{'rb'}.

The @emph{compresslevel} argument is an integer from 1 to 9, as for the
@ref{455,,BZ2File} constructor.

For binary mode, this function is equivalent to the @ref{455,,BZ2File}
constructor: @code{BZ2File(filename, mode, compresslevel=compresslevel)}. In
this case, the @emph{encoding}, @emph{errors} and @emph{newline} arguments must not be
provided.

For text mode, a @ref{455,,BZ2File} object is created, and wrapped in an
@ref{889,,io.TextIOWrapper} instance with the specified encoding, error
handling behavior, and line ending(s).


New in version 3.3.



Changed in version 3.4: The @code{'x'} (exclusive creation) mode was added.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex BZ2File (class in bz2)
@anchor{library/bz2 bz2 BZ2File}@anchor{455}
@deffn {Class} bz2.BZ2File (filename, mode='r', *, compresslevel=9)

Open a bzip2-compressed file in binary mode.

If @emph{filename} is a @ref{2ef,,str} or @ref{2f0,,bytes} object, open the named file
directly. Otherwise, @emph{filename} should be a @ref{d68,,file object}, which will
be used to read or write the compressed data.

The @emph{mode} argument can be either @code{'r'} for reading (default), @code{'w'} for
overwriting, @code{'x'} for exclusive creation, or @code{'a'} for appending. These
can equivalently be given as @code{'rb'}, @code{'wb'}, @code{'xb'} and @code{'ab'}
respectively.

If @emph{filename} is a file object (rather than an actual file name), a mode of
@code{'w'} does not truncate the file, and is instead equivalent to @code{'a'}.

If @emph{mode} is @code{'w'} or @code{'a'}, @emph{compresslevel} can be an integer between
@code{1} and @code{9} specifying the level of compression: @code{1} produces the
least compression, and @code{9} (default) produces the most compression.

If @emph{mode} is @code{'r'}, the input file may be the concatenation of multiple
compressed streams.

@ref{455,,BZ2File} provides all of the members specified by the
@ref{823,,io.BufferedIOBase}, except for @code{detach()} and @code{truncate()}.
Iteration and the @ref{168,,with} statement are supported.

@ref{455,,BZ2File} also provides the following method:

@geindex peek() (bz2.BZ2File method)
@anchor{library/bz2 bz2 BZ2File peek}@anchor{2200}
@deffn {Method} peek ([n])

Return buffered data without advancing the file position. At least one
byte of data will be returned (unless at EOF). The exact number of bytes
returned is unspecified.

@cartouche
@quotation Note 
While calling @ref{2200,,peek()} does not change the file position of
the @ref{455,,BZ2File}, it may change the position of the underlying file
object (e.g. if the @ref{455,,BZ2File} was constructed by passing a file
object for @emph{filename}).
@end quotation
@end cartouche


New in version 3.3.

@end deffn


Changed in version 3.1: Support for the @ref{168,,with} statement was added.



Changed in version 3.3: The @code{fileno()}, @code{readable()}, @code{seekable()}, @code{writable()},
@code{read1()} and @code{readinto()} methods were added.



Changed in version 3.3: Support was added for @emph{filename} being a @ref{d68,,file object} instead of an
actual filename.



Changed in version 3.3: The @code{'a'} (append) mode was added, along with support for reading
multi-stream files.



Changed in version 3.4: The @code{'x'} (exclusive creation) mode was added.



Changed in version 3.5: The @ref{21ef,,read()} method now accepts an argument of
@code{None}.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.



Changed in version 3.9: The @emph{buffering} parameter has been removed. It was ignored and deprecated
since Python 3.0. Pass an open file object to control how the file is
opened.

The @emph{compresslevel} parameter became keyword-only.



Changed in version 3.10: This class is thread unsafe in the face of multiple simultaneous
readers or writers, just like its equivalent classes in @ref{87,,gzip} and
@ref{a9,,lzma} have always been.

@end deffn

@node Incremental de compression,One-shot de compression,De compression of files,bz2 — Support for bzip2 compression
@anchor{library/bz2 incremental-de-compression}@anchor{2201}
@subsubsection Incremental (de)compression


@geindex BZ2Compressor (class in bz2)
@anchor{library/bz2 bz2 BZ2Compressor}@anchor{21fd}
@deffn {Class} bz2.BZ2Compressor (compresslevel=9)

Create a new compressor object. This object may be used to compress data
incrementally. For one-shot compression, use the @ref{1cf5,,compress()} function
instead.

@emph{compresslevel}, if given, must be an integer between @code{1} and @code{9}. The
default is @code{9}.

@geindex compress() (bz2.BZ2Compressor method)
@anchor{library/bz2 bz2 BZ2Compressor compress}@anchor{2202}
@deffn {Method} compress (data)

Provide data to the compressor object. Returns a chunk of compressed data
if possible, or an empty byte string otherwise.

When you have finished providing data to the compressor, call the
@ref{2203,,flush()} method to finish the compression process.
@end deffn

@geindex flush() (bz2.BZ2Compressor method)
@anchor{library/bz2 bz2 BZ2Compressor flush}@anchor{2203}
@deffn {Method} flush ()

Finish the compression process. Returns the compressed data left in
internal buffers.

The compressor object may not be used after this method has been called.
@end deffn
@end deffn

@geindex BZ2Decompressor (class in bz2)
@anchor{library/bz2 bz2 BZ2Decompressor}@anchor{21fe}
@deffn {Class} bz2.BZ2Decompressor

Create a new decompressor object. This object may be used to decompress data
incrementally. For one-shot compression, use the @ref{c27,,decompress()} function
instead.

@cartouche
@quotation Note 
This class does not transparently handle inputs containing multiple
compressed streams, unlike @ref{c27,,decompress()} and @ref{455,,BZ2File}. If
you need to decompress a multi-stream input with @ref{21fe,,BZ2Decompressor},
you must use a new decompressor for each stream.
@end quotation
@end cartouche

@geindex decompress() (bz2.BZ2Decompressor method)
@anchor{library/bz2 bz2 BZ2Decompressor decompress}@anchor{92a}
@deffn {Method} decompress (data, max_length=- 1)

Decompress @emph{data} (a @ref{87e,,bytes-like object}), returning
uncompressed data as bytes. Some of @emph{data} may be buffered
internally, for use in later calls to @ref{c27,,decompress()}. The
returned data should be concatenated with the output of any
previous calls to @ref{c27,,decompress()}.

If @emph{max_length} is nonnegative, returns at most @emph{max_length}
bytes of decompressed data. If this limit is reached and further
output can be produced, the @ref{2204,,needs_input} attribute will
be set to @code{False}. In this case, the next call to
@ref{92a,,decompress()} may provide @emph{data} as @code{b'@w{'}} to obtain
more of the output.

If all of the input data was decompressed and returned (either
because this was less than @emph{max_length} bytes, or because
@emph{max_length} was negative), the @ref{2204,,needs_input} attribute
will be set to @code{True}.

Attempting to decompress data after the end of stream is reached
raises an @cite{EOFError}.  Any data found after the end of the
stream is ignored and saved in the @ref{2205,,unused_data} attribute.


Changed in version 3.5: Added the @emph{max_length} parameter.

@end deffn

@geindex eof (bz2.BZ2Decompressor attribute)
@anchor{library/bz2 bz2 BZ2Decompressor eof}@anchor{2206}
@deffn {Attribute} eof

@code{True} if the end-of-stream marker has been reached.


New in version 3.3.

@end deffn

@geindex unused_data (bz2.BZ2Decompressor attribute)
@anchor{library/bz2 bz2 BZ2Decompressor unused_data}@anchor{2205}
@deffn {Attribute} unused_data

Data found after the end of the compressed stream.

If this attribute is accessed before the end of the stream has been
reached, its value will be @code{b'@w{'}}.
@end deffn

@geindex needs_input (bz2.BZ2Decompressor attribute)
@anchor{library/bz2 bz2 BZ2Decompressor needs_input}@anchor{2204}
@deffn {Attribute} needs_input

@code{False} if the @ref{92a,,decompress()} method can provide more
decompressed data before requiring new uncompressed input.


New in version 3.5.

@end deffn
@end deffn

@node One-shot de compression,Examples of usage<2>,Incremental de compression,bz2 — Support for bzip2 compression
@anchor{library/bz2 one-shot-de-compression}@anchor{2207}
@subsubsection One-shot (de)compression


@geindex compress() (in module bz2)
@anchor{library/bz2 bz2 compress}@anchor{1cf5}
@deffn {Function} bz2.compress (data, compresslevel=9)

Compress @emph{data}, a @ref{87e,,bytes-like object}.

@emph{compresslevel}, if given, must be an integer between @code{1} and @code{9}. The
default is @code{9}.

For incremental compression, use a @ref{21fd,,BZ2Compressor} instead.
@end deffn

@geindex decompress() (in module bz2)
@anchor{library/bz2 bz2 decompress}@anchor{c27}
@deffn {Function} bz2.decompress (data)

Decompress @emph{data}, a @ref{87e,,bytes-like object}.

If @emph{data} is the concatenation of multiple compressed streams, decompress
all of the streams.

For incremental decompression, use a @ref{21fe,,BZ2Decompressor} instead.


Changed in version 3.3: Support for multi-stream inputs was added.

@end deffn

@node Examples of usage<2>,,One-shot de compression,bz2 — Support for bzip2 compression
@anchor{library/bz2 bz2-usage-examples}@anchor{2208}@anchor{library/bz2 examples-of-usage}@anchor{2209}
@subsubsection Examples of usage


Below are some examples of typical usage of the @ref{12,,bz2} module.

Using @ref{1cf5,,compress()} and @ref{c27,,decompress()} to demonstrate round-trip compression:

@example
>>> import bz2
>>> data = b"""\
... Donec rhoncus quis sapien sit amet molestie. Fusce scelerisque vel augue
... nec ullamcorper. Nam rutrum pretium placerat. Aliquam vel tristique lorem,
... sit amet cursus ante. In interdum laoreet mi, sit amet ultrices purus
... pulvinar a. Nam gravida euismod magna, non varius justo tincidunt feugiat.
... Aliquam pharetra lacus non risus vehicula rutrum. Maecenas aliquam leo
... felis. Pellentesque semper nunc sit amet nibh ullamcorper, ac elementum
... dolor luctus. Curabitur lacinia mi ornare consectetur vestibulum."""
>>> c = bz2.compress(data)
>>> len(data) / len(c)  # Data compression ratio
1.513595166163142
>>> d = bz2.decompress(c)
>>> data == d  # Check equality to original object after round-trip
True
@end example

Using @ref{21fd,,BZ2Compressor} for incremental compression:

@example
>>> import bz2
>>> def gen_data(chunks=10, chunksize=1000):
...     """Yield incremental blocks of chunksize bytes."""
...     for _ in range(chunks):
...         yield b"z" * chunksize
...
>>> comp = bz2.BZ2Compressor()
>>> out = b""
>>> for chunk in gen_data():
...     # Provide data to the compressor object
...     out = out + comp.compress(chunk)
...
>>> # Finish the compression process.  Call this once you have
>>> # finished providing data to the compressor.
>>> out = out + comp.flush()
@end example

The example above uses a very “nonrandom” stream of data
(a stream of @cite{b”z”} chunks).  Random data tends to compress poorly,
while ordered, repetitive data usually yields a high compression ratio.

Writing and reading a bzip2-compressed file in binary mode:

@example
>>> import bz2
>>> data = b"""\
... Donec rhoncus quis sapien sit amet molestie. Fusce scelerisque vel augue
... nec ullamcorper. Nam rutrum pretium placerat. Aliquam vel tristique lorem,
... sit amet cursus ante. In interdum laoreet mi, sit amet ultrices purus
... pulvinar a. Nam gravida euismod magna, non varius justo tincidunt feugiat.
... Aliquam pharetra lacus non risus vehicula rutrum. Maecenas aliquam leo
... felis. Pellentesque semper nunc sit amet nibh ullamcorper, ac elementum
... dolor luctus. Curabitur lacinia mi ornare consectetur vestibulum."""
>>> with bz2.open("myfile.bz2", "wb") as f:
...     # Write compressed data to file
...     unused = f.write(data)
>>> with bz2.open("myfile.bz2", "rb") as f:
...     # Decompress data from file
...     content = f.read()
>>> content == data  # Check equality to original object after round-trip
True
@end example

@c import os
@c os.remove("myfile.bz2")

@node lzma — Compression using the LZMA algorithm,zipfile — Work with ZIP archives,bz2 — Support for bzip2 compression,Data Compression and Archiving
@anchor{library/lzma doc}@anchor{220a}@anchor{library/lzma lzma-compression-using-the-lzma-algorithm}@anchor{220b}@anchor{library/lzma module-lzma}@anchor{a9}
@subsection @code{lzma} — Compression using the LZMA algorithm


@geindex module; lzma


New in version 3.3.


@strong{Source code:} Lib/lzma.py@footnote{https://github.com/python/cpython/tree/main/Lib/lzma.py}


__________________________________________________________________


This module provides classes and convenience functions for compressing and
decompressing data using the LZMA compression algorithm. Also included is a file
interface supporting the @code{.xz} and legacy @code{.lzma} file formats used by the
@code{xz} utility, as well as raw compressed streams.

The interface provided by this module is very similar to that of the @ref{12,,bz2}
module. Note that @ref{b93,,LZMAFile} and @ref{455,,bz2.BZ2File} are @emph{not}
thread-safe, so if you need to use a single @ref{b93,,LZMAFile} instance
from multiple threads, it is necessary to protect it with a lock.

@geindex LZMAError
@anchor{library/lzma lzma LZMAError}@anchor{220c}
@deffn {Exception} lzma.LZMAError

This exception is raised when an error occurs during compression or
decompression, or while initializing the compressor/decompressor state.
@end deffn

@menu
* Reading and writing compressed files:: 
* Compressing and decompressing data in memory:: 
* Miscellaneous: Miscellaneous<2>. 
* Specifying custom filter chains:: 
* Examples: Examples<7>. 

@end menu

@node Reading and writing compressed files,Compressing and decompressing data in memory,,lzma — Compression using the LZMA algorithm
@anchor{library/lzma reading-and-writing-compressed-files}@anchor{220d}
@subsubsection Reading and writing compressed files


@geindex open() (in module lzma)
@anchor{library/lzma lzma open}@anchor{220e}
@deffn {Function} lzma.open (filename, mode='rb', *, format=None, check=- 1, preset=None, filters=None, encoding=None, errors=None, newline=None)

Open an LZMA-compressed file in binary or text mode, returning a @ref{d68,,file object}.

The @emph{filename} argument can be either an actual file name (given as a
@ref{2ef,,str}, @ref{2f0,,bytes} or @ref{63d,,path-like} object), in
which case the named file is opened, or it can be an existing file object
to read from or write to.

The @emph{mode} argument can be any of @code{"r"}, @code{"rb"}, @code{"w"}, @code{"wb"},
@code{"x"}, @code{"xb"}, @code{"a"} or @code{"ab"} for binary mode, or @code{"rt"},
@code{"wt"}, @code{"xt"}, or @code{"at"} for text mode. The default is @code{"rb"}.

When opening a file for reading, the @emph{format} and @emph{filters} arguments have
the same meanings as for @ref{220f,,LZMADecompressor}. In this case, the @emph{check}
and @emph{preset} arguments should not be used.

When opening a file for writing, the @emph{format}, @emph{check}, @emph{preset} and
@emph{filters} arguments have the same meanings as for @ref{2210,,LZMACompressor}.

For binary mode, this function is equivalent to the @ref{b93,,LZMAFile}
constructor: @code{LZMAFile(filename, mode, ...)}. In this case, the @emph{encoding},
@emph{errors} and @emph{newline} arguments must not be provided.

For text mode, a @ref{b93,,LZMAFile} object is created, and wrapped in an
@ref{889,,io.TextIOWrapper} instance with the specified encoding, error
handling behavior, and line ending(s).


Changed in version 3.4: Added support for the @code{"x"}, @code{"xb"} and @code{"xt"} modes.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex LZMAFile (class in lzma)
@anchor{library/lzma lzma LZMAFile}@anchor{b93}
@deffn {Class} lzma.LZMAFile (filename=None, mode='r', *, format=None, check=- 1, preset=None, filters=None)

Open an LZMA-compressed file in binary mode.

An @ref{b93,,LZMAFile} can wrap an already-open @ref{d68,,file object}, or operate
directly on a named file. The @emph{filename} argument specifies either the file
object to wrap, or the name of the file to open (as a @ref{2ef,,str},
@ref{2f0,,bytes} or @ref{63d,,path-like} object). When wrapping an
existing file object, the wrapped file will not be closed when the
@ref{b93,,LZMAFile} is closed.

The @emph{mode} argument can be either @code{"r"} for reading (default), @code{"w"} for
overwriting, @code{"x"} for exclusive creation, or @code{"a"} for appending. These
can equivalently be given as @code{"rb"}, @code{"wb"}, @code{"xb"} and @code{"ab"}
respectively.

If @emph{filename} is a file object (rather than an actual file name), a mode of
@code{"w"} does not truncate the file, and is instead equivalent to @code{"a"}.

When opening a file for reading, the input file may be the concatenation of
multiple separate compressed streams. These are transparently decoded as a
single logical stream.

When opening a file for reading, the @emph{format} and @emph{filters} arguments have
the same meanings as for @ref{220f,,LZMADecompressor}. In this case, the @emph{check}
and @emph{preset} arguments should not be used.

When opening a file for writing, the @emph{format}, @emph{check}, @emph{preset} and
@emph{filters} arguments have the same meanings as for @ref{2210,,LZMACompressor}.

@ref{b93,,LZMAFile} supports all the members specified by
@ref{823,,io.BufferedIOBase}, except for @code{detach()} and @code{truncate()}.
Iteration and the @ref{168,,with} statement are supported.

The following method is also provided:

@geindex peek() (lzma.LZMAFile method)
@anchor{library/lzma lzma LZMAFile peek}@anchor{2211}
@deffn {Method} peek (size=- 1)

Return buffered data without advancing the file position. At least one
byte of data will be returned, unless EOF has been reached. The exact
number of bytes returned is unspecified (the @emph{size} argument is ignored).

@cartouche
@quotation Note 
While calling @ref{2211,,peek()} does not change the file position of
the @ref{b93,,LZMAFile}, it may change the position of the underlying
file object (e.g. if the @ref{b93,,LZMAFile} was constructed by passing a
file object for @emph{filename}).
@end quotation
@end cartouche
@end deffn


Changed in version 3.4: Added support for the @code{"x"} and @code{"xb"} modes.



Changed in version 3.5: The @ref{21ef,,read()} method now accepts an argument of
@code{None}.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@node Compressing and decompressing data in memory,Miscellaneous<2>,Reading and writing compressed files,lzma — Compression using the LZMA algorithm
@anchor{library/lzma compressing-and-decompressing-data-in-memory}@anchor{2212}
@subsubsection Compressing and decompressing data in memory


@geindex LZMACompressor (class in lzma)
@anchor{library/lzma lzma LZMACompressor}@anchor{2210}
@deffn {Class} lzma.LZMACompressor (format=FORMAT_XZ, check=- 1, preset=None, filters=None)

Create a compressor object, which can be used to compress data incrementally.

For a more convenient way of compressing a single chunk of data, see
@ref{2213,,compress()}.

The @emph{format} argument specifies what container format should be used.
Possible values are:


@itemize *

@item 

@table @asis

@item @code{FORMAT_XZ}: The @code{.xz} container format.

This is the default format.
@end table

@item 

@table @asis

@item @code{FORMAT_ALONE}: The legacy @code{.lzma} container format.

This format is more limited than @code{.xz} – it does not support integrity
checks or multiple filters.
@end table

@item 

@table @asis

@item @code{FORMAT_RAW}: A raw data stream, not using any container format.

This format specifier does not support integrity checks, and requires that
you always specify a custom filter chain (for both compression and
decompression). Additionally, data compressed in this manner cannot be
decompressed using @code{FORMAT_AUTO} (see @ref{220f,,LZMADecompressor}).
@end table
@end itemize

The @emph{check} argument specifies the type of integrity check to include in the
compressed data. This check is used when decompressing, to ensure that the
data has not been corrupted. Possible values are:


@itemize *

@item 
@code{CHECK_NONE}: No integrity check.
This is the default (and the only acceptable value) for
@code{FORMAT_ALONE} and @code{FORMAT_RAW}.

@item 
@code{CHECK_CRC32}: 32-bit Cyclic Redundancy Check.

@item 
@code{CHECK_CRC64}: 64-bit Cyclic Redundancy Check.
This is the default for @code{FORMAT_XZ}.

@item 
@code{CHECK_SHA256}: 256-bit Secure Hash Algorithm.
@end itemize

If the specified check is not supported, an @ref{220c,,LZMAError} is raised.

The compression settings can be specified either as a preset compression
level (with the @emph{preset} argument), or in detail as a custom filter chain
(with the @emph{filters} argument).

The @emph{preset} argument (if provided) should be an integer between @code{0} and
@code{9} (inclusive), optionally OR-ed with the constant
@code{PRESET_EXTREME}. If neither @emph{preset} nor @emph{filters} are given, the
default behavior is to use @code{PRESET_DEFAULT} (preset level @code{6}).
Higher presets produce smaller output, but make the compression process
slower.

@cartouche
@quotation Note 
In addition to being more CPU-intensive, compression with higher presets
also requires much more memory (and produces output that needs more memory
to decompress). With preset @code{9} for example, the overhead for an
@ref{2210,,LZMACompressor} object can be as high as 800 MiB. For this reason,
it is generally best to stick with the default preset.
@end quotation
@end cartouche

The @emph{filters} argument (if provided) should be a filter chain specifier.
See @ref{2214,,Specifying custom filter chains} for details.

@geindex compress() (lzma.LZMACompressor method)
@anchor{library/lzma lzma LZMACompressor compress}@anchor{2215}
@deffn {Method} compress (data)

Compress @emph{data} (a @ref{2f0,,bytes} object), returning a @ref{2f0,,bytes}
object containing compressed data for at least part of the input. Some of
@emph{data} may be buffered internally, for use in later calls to
@ref{2213,,compress()} and @ref{2216,,flush()}. The returned data should be
concatenated with the output of any previous calls to @ref{2213,,compress()}.
@end deffn

@geindex flush() (lzma.LZMACompressor method)
@anchor{library/lzma lzma LZMACompressor flush}@anchor{2216}
@deffn {Method} flush ()

Finish the compression process, returning a @ref{2f0,,bytes} object
containing any data stored in the compressor’s internal buffers.

The compressor cannot be used after this method has been called.
@end deffn
@end deffn

@geindex LZMADecompressor (class in lzma)
@anchor{library/lzma lzma LZMADecompressor}@anchor{220f}
@deffn {Class} lzma.LZMADecompressor (format=FORMAT_AUTO, memlimit=None, filters=None)

Create a decompressor object, which can be used to decompress data
incrementally.

For a more convenient way of decompressing an entire compressed stream at
once, see @ref{2217,,decompress()}.

The @emph{format} argument specifies the container format that should be used. The
default is @code{FORMAT_AUTO}, which can decompress both @code{.xz} and
@code{.lzma} files. Other possible values are @code{FORMAT_XZ},
@code{FORMAT_ALONE}, and @code{FORMAT_RAW}.

The @emph{memlimit} argument specifies a limit (in bytes) on the amount of memory
that the decompressor can use. When this argument is used, decompression will
fail with an @ref{220c,,LZMAError} if it is not possible to decompress the input
within the given memory limit.

The @emph{filters} argument specifies the filter chain that was used to create
the stream being decompressed. This argument is required if @emph{format} is
@code{FORMAT_RAW}, but should not be used for other formats.
See @ref{2214,,Specifying custom filter chains} for more information about filter chains.

@cartouche
@quotation Note 
This class does not transparently handle inputs containing multiple
compressed streams, unlike @ref{2217,,decompress()} and @ref{b93,,LZMAFile}. To
decompress a multi-stream input with @ref{220f,,LZMADecompressor}, you must
create a new decompressor for each stream.
@end quotation
@end cartouche

@geindex decompress() (lzma.LZMADecompressor method)
@anchor{library/lzma lzma LZMADecompressor decompress}@anchor{995}
@deffn {Method} decompress (data, max_length=- 1)

Decompress @emph{data} (a @ref{87e,,bytes-like object}), returning
uncompressed data as bytes. Some of @emph{data} may be buffered
internally, for use in later calls to @ref{2217,,decompress()}. The
returned data should be concatenated with the output of any
previous calls to @ref{2217,,decompress()}.

If @emph{max_length} is nonnegative, returns at most @emph{max_length}
bytes of decompressed data. If this limit is reached and further
output can be produced, the @ref{2218,,needs_input} attribute will
be set to @code{False}. In this case, the next call to
@ref{995,,decompress()} may provide @emph{data} as @code{b'@w{'}} to obtain
more of the output.

If all of the input data was decompressed and returned (either
because this was less than @emph{max_length} bytes, or because
@emph{max_length} was negative), the @ref{2218,,needs_input} attribute
will be set to @code{True}.

Attempting to decompress data after the end of stream is reached
raises an @cite{EOFError}.  Any data found after the end of the
stream is ignored and saved in the @ref{2219,,unused_data} attribute.


Changed in version 3.5: Added the @emph{max_length} parameter.

@end deffn

@geindex check (lzma.LZMADecompressor attribute)
@anchor{library/lzma lzma LZMADecompressor check}@anchor{221a}
@deffn {Attribute} check

The ID of the integrity check used by the input stream. This may be
@code{CHECK_UNKNOWN} until enough of the input has been decoded to
determine what integrity check it uses.
@end deffn

@geindex eof (lzma.LZMADecompressor attribute)
@anchor{library/lzma lzma LZMADecompressor eof}@anchor{221b}
@deffn {Attribute} eof

@code{True} if the end-of-stream marker has been reached.
@end deffn

@geindex unused_data (lzma.LZMADecompressor attribute)
@anchor{library/lzma lzma LZMADecompressor unused_data}@anchor{2219}
@deffn {Attribute} unused_data

Data found after the end of the compressed stream.

Before the end of the stream is reached, this will be @code{b""}.
@end deffn

@geindex needs_input (lzma.LZMADecompressor attribute)
@anchor{library/lzma lzma LZMADecompressor needs_input}@anchor{2218}
@deffn {Attribute} needs_input

@code{False} if the @ref{995,,decompress()} method can provide more
decompressed data before requiring new uncompressed input.


New in version 3.5.

@end deffn
@end deffn

@geindex compress() (in module lzma)
@anchor{library/lzma lzma compress}@anchor{2213}
@deffn {Function} lzma.compress (data, format=FORMAT_XZ, check=- 1, preset=None, filters=None)

Compress @emph{data} (a @ref{2f0,,bytes} object), returning the compressed data as a
@ref{2f0,,bytes} object.

See @ref{2210,,LZMACompressor} above for a description of the @emph{format}, @emph{check},
@emph{preset} and @emph{filters} arguments.
@end deffn

@geindex decompress() (in module lzma)
@anchor{library/lzma lzma decompress}@anchor{2217}
@deffn {Function} lzma.decompress (data, format=FORMAT_AUTO, memlimit=None, filters=None)

Decompress @emph{data} (a @ref{2f0,,bytes} object), returning the uncompressed data
as a @ref{2f0,,bytes} object.

If @emph{data} is the concatenation of multiple distinct compressed streams,
decompress all of these streams, and return the concatenation of the results.

See @ref{220f,,LZMADecompressor} above for a description of the @emph{format},
@emph{memlimit} and @emph{filters} arguments.
@end deffn

@node Miscellaneous<2>,Specifying custom filter chains,Compressing and decompressing data in memory,lzma — Compression using the LZMA algorithm
@anchor{library/lzma miscellaneous}@anchor{221c}
@subsubsection Miscellaneous


@geindex is_check_supported() (in module lzma)
@anchor{library/lzma lzma is_check_supported}@anchor{221d}
@deffn {Function} lzma.is_check_supported (check)

Return @code{True} if the given integrity check is supported on this system.

@code{CHECK_NONE} and @code{CHECK_CRC32} are always supported.
@code{CHECK_CRC64} and @code{CHECK_SHA256} may be unavailable if you are
using a version of @code{liblzma} that was compiled with a limited
feature set.
@end deffn

@node Specifying custom filter chains,Examples<7>,Miscellaneous<2>,lzma — Compression using the LZMA algorithm
@anchor{library/lzma filter-chain-specs}@anchor{2214}@anchor{library/lzma specifying-custom-filter-chains}@anchor{221e}
@subsubsection Specifying custom filter chains


A filter chain specifier is a sequence of dictionaries, where each dictionary
contains the ID and options for a single filter. Each dictionary must contain
the key @code{"id"}, and may contain additional keys to specify filter-dependent
options. Valid filter IDs are as follows:


@itemize *

@item 

@table @asis

@item Compression filters:


@itemize *

@item 
@code{FILTER_LZMA1} (for use with @code{FORMAT_ALONE})

@item 
@code{FILTER_LZMA2} (for use with @code{FORMAT_XZ} and @code{FORMAT_RAW})
@end itemize
@end table

@item 

@table @asis

@item Delta filter:


@itemize *

@item 
@code{FILTER_DELTA}
@end itemize
@end table

@item 

@table @asis

@item Branch-Call-Jump (BCJ) filters:


@itemize *

@item 
@code{FILTER_X86}

@item 
@code{FILTER_IA64}

@item 
@code{FILTER_ARM}

@item 
@code{FILTER_ARMTHUMB}

@item 
@code{FILTER_POWERPC}

@item 
@code{FILTER_SPARC}
@end itemize
@end table
@end itemize

A filter chain can consist of up to 4 filters, and cannot be empty. The last
filter in the chain must be a compression filter, and any other filters must be
delta or BCJ filters.

Compression filters support the following options (specified as additional
entries in the dictionary representing the filter):

@quotation


@itemize *

@item 
@code{preset}: A compression preset to use as a source of default values for
options that are not specified explicitly.

@item 
@code{dict_size}: Dictionary size in bytes. This should be between 4 KiB and
1.5 GiB (inclusive).

@item 
@code{lc}: Number of literal context bits.

@item 
@code{lp}: Number of literal position bits. The sum @code{lc + lp} must be at
most 4.

@item 
@code{pb}: Number of position bits; must be at most 4.

@item 
@code{mode}: @code{MODE_FAST} or @code{MODE_NORMAL}.

@item 
@code{nice_len}: What should be considered a “nice length” for a match.
This should be 273 or less.

@item 
@code{mf}: What match finder to use – @code{MF_HC3}, @code{MF_HC4},
@code{MF_BT2}, @code{MF_BT3}, or @code{MF_BT4}.

@item 
@code{depth}: Maximum search depth used by match finder. 0 (default) means to
select automatically based on other filter options.
@end itemize
@end quotation

The delta filter stores the differences between bytes, producing more repetitive
input for the compressor in certain circumstances. It supports one option,
@code{dist}. This indicates the distance between bytes to be subtracted. The
default is 1, i.e. take the differences between adjacent bytes.

The BCJ filters are intended to be applied to machine code. They convert
relative branches, calls and jumps in the code to use absolute addressing, with
the aim of increasing the redundancy that can be exploited by the compressor.
These filters support one option, @code{start_offset}. This specifies the address
that should be mapped to the beginning of the input data. The default is 0.

@node Examples<7>,,Specifying custom filter chains,lzma — Compression using the LZMA algorithm
@anchor{library/lzma examples}@anchor{221f}
@subsubsection Examples


Reading in a compressed file:

@example
import lzma
with lzma.open("file.xz") as f:
    file_content = f.read()
@end example

Creating a compressed file:

@example
import lzma
data = b"Insert Data Here"
with lzma.open("file.xz", "w") as f:
    f.write(data)
@end example

Compressing data in memory:

@example
import lzma
data_in = b"Insert Data Here"
data_out = lzma.compress(data_in)
@end example

Incremental compression:

@example
import lzma
lzc = lzma.LZMACompressor()
out1 = lzc.compress(b"Some data\n")
out2 = lzc.compress(b"Another piece of data\n")
out3 = lzc.compress(b"Even more data\n")
out4 = lzc.flush()
# Concatenate all the partial results:
result = b"".join([out1, out2, out3, out4])
@end example

Writing compressed data to an already-open file:

@example
import lzma
with open("file.xz", "wb") as f:
    f.write(b"This data will not be compressed\n")
    with lzma.open(f, "w") as lzf:
        lzf.write(b"This *will* be compressed\n")
    f.write(b"Not compressed\n")
@end example

Creating a compressed file using a custom filter chain:

@example
import lzma
my_filters = [
    @{"id": lzma.FILTER_DELTA, "dist": 5@},
    @{"id": lzma.FILTER_LZMA2, "preset": 7 | lzma.PRESET_EXTREME@},
]
with lzma.open("file.xz", "w", filters=my_filters) as f:
    f.write(b"blah blah blah")
@end example

@node zipfile — Work with ZIP archives,tarfile — Read and write tar archive files,lzma — Compression using the LZMA algorithm,Data Compression and Archiving
@anchor{library/zipfile doc}@anchor{2220}@anchor{library/zipfile module-zipfile}@anchor{149}@anchor{library/zipfile zipfile-work-with-zip-archives}@anchor{2221}
@subsection @code{zipfile} — Work with ZIP archives


@geindex module; zipfile

@strong{Source code:} Lib/zipfile.py@footnote{https://github.com/python/cpython/tree/main/Lib/zipfile.py}


__________________________________________________________________


The ZIP file format is a common archive and compression standard. This module
provides tools to create, read, write, append, and list a ZIP file.  Any
advanced use of this module will require an understanding of the format, as
defined in PKZIP Application Note@footnote{https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT}.

This module does not currently handle multi-disk ZIP files.
It can handle ZIP files that use the ZIP64 extensions
(that is ZIP files that are more than 4 GiB in size).  It supports
decryption of encrypted files in ZIP archives, but it currently cannot
create an encrypted file.  Decryption is extremely slow as it is
implemented in native Python rather than C.

The module defines the following items:

@geindex BadZipFile
@anchor{library/zipfile zipfile BadZipFile}@anchor{2222}
@deffn {Exception} zipfile.BadZipFile

The error raised for bad ZIP files.


New in version 3.2.

@end deffn

@geindex BadZipfile
@anchor{library/zipfile zipfile BadZipfile}@anchor{2223}
@deffn {Exception} zipfile.BadZipfile

Alias of @ref{2222,,BadZipFile}, for compatibility with older Python versions.


Deprecated since version 3.2.

@end deffn

@geindex LargeZipFile
@anchor{library/zipfile zipfile LargeZipFile}@anchor{2224}
@deffn {Exception} zipfile.LargeZipFile

The error raised when a ZIP file would require ZIP64 functionality but that has
not been enabled.
@end deffn


@deffn {Class} zipfile.ZipFile

The class for reading and writing ZIP files.  See section
@ref{2225,,ZipFile Objects} for constructor details.
@end deffn


@deffn {Class} zipfile.Path

A pathlib-compatible wrapper for zip files. See section
@ref{2226,,Path Objects} for details.


New in version 3.8.

@end deffn


@deffn {Class} zipfile.PyZipFile

Class for creating ZIP archives containing Python libraries.
@end deffn

@geindex ZipInfo (class in zipfile)
@anchor{library/zipfile zipfile ZipInfo}@anchor{855}
@deffn {Class} zipfile.ZipInfo (filename='NoName', date_time=(1980, 1, 1, 0, 0, 0))

Class used to represent information about a member of an archive. Instances
of this class are returned by the @ref{2227,,getinfo()} and @ref{2228,,infolist()}
methods of @ref{6e5,,ZipFile} objects.  Most users of the @ref{149,,zipfile} module
will not need to create these, but only use those created by this
module. @emph{filename} should be the full name of the archive member, and
@emph{date_time} should be a tuple containing six fields which describe the time
of the last modification to the file; the fields are described in section
@ref{2229,,ZipInfo Objects}.
@end deffn

@geindex is_zipfile() (in module zipfile)
@anchor{library/zipfile zipfile is_zipfile}@anchor{ebc}
@deffn {Function} zipfile.is_zipfile (filename)

Returns @code{True} if @emph{filename} is a valid ZIP file based on its magic number,
otherwise returns @code{False}.  @emph{filename} may be a file or file-like object too.


Changed in version 3.1: Support for file and file-like objects.

@end deffn

@geindex ZIP_STORED (in module zipfile)
@anchor{library/zipfile zipfile ZIP_STORED}@anchor{222a}
@deffn {Data} zipfile.ZIP_STORED

The numeric constant for an uncompressed archive member.
@end deffn

@geindex ZIP_DEFLATED (in module zipfile)
@anchor{library/zipfile zipfile ZIP_DEFLATED}@anchor{222b}
@deffn {Data} zipfile.ZIP_DEFLATED

The numeric constant for the usual ZIP compression method.  This requires the
@ref{14b,,zlib} module.
@end deffn

@geindex ZIP_BZIP2 (in module zipfile)
@anchor{library/zipfile zipfile ZIP_BZIP2}@anchor{222c}
@deffn {Data} zipfile.ZIP_BZIP2

The numeric constant for the BZIP2 compression method.  This requires the
@ref{12,,bz2} module.


New in version 3.3.

@end deffn

@geindex ZIP_LZMA (in module zipfile)
@anchor{library/zipfile zipfile ZIP_LZMA}@anchor{222d}
@deffn {Data} zipfile.ZIP_LZMA

The numeric constant for the LZMA compression method.  This requires the
@ref{a9,,lzma} module.


New in version 3.3.


@cartouche
@quotation Note 
The ZIP file format specification has included support for bzip2 compression
since 2001, and for LZMA compression since 2006. However, some tools
(including older Python releases) do not support these compression
methods, and may either refuse to process the ZIP file altogether,
or fail to extract individual files.
@end quotation
@end cartouche
@end deffn


@subsubheading See also



@table @asis

@item PKZIP Application Note@footnote{https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT}

Documentation on the ZIP file format by Phil Katz, the creator of the format and
algorithms used.

@item Info-ZIP Home Page@footnote{http://www.info-zip.org/}

Information about the Info-ZIP project’s ZIP archive programs and development
libraries.
@end table


@menu
* ZipFile Objects:: 
* Path Objects:: 
* PyZipFile Objects:: 
* ZipInfo Objects:: 
* Command-Line Interface:: 
* Decompression pitfalls:: 

@end menu

@node ZipFile Objects,Path Objects,,zipfile — Work with ZIP archives
@anchor{library/zipfile id1}@anchor{222e}@anchor{library/zipfile zipfile-objects}@anchor{2225}
@subsubsection ZipFile Objects


@geindex ZipFile (class in zipfile)
@anchor{library/zipfile zipfile ZipFile}@anchor{6e5}
@deffn {Class} zipfile.ZipFile (file, mode='r', compression=ZIP_STORED, allowZip64=True, compresslevel=None, *, strict_timestamps=True)

Open a ZIP file, where @emph{file} can be a path to a file (a string), a
file-like object or a @ref{63d,,path-like object}.

The @emph{mode} parameter should be @code{'r'} to read an existing
file, @code{'w'} to truncate and write a new file, @code{'a'} to append to an
existing file, or @code{'x'} to exclusively create and write a new file.
If @emph{mode} is @code{'x'} and @emph{file} refers to an existing file,
a @ref{bb1,,FileExistsError} will be raised.
If @emph{mode} is @code{'a'} and @emph{file} refers to an existing ZIP
file, then additional files are added to it.  If @emph{file} does not refer to a
ZIP file, then a new ZIP archive is appended to the file.  This is meant for
adding a ZIP archive to another file (such as @code{python.exe}).  If
@emph{mode} is @code{'a'} and the file does not exist at all, it is created.
If @emph{mode} is @code{'r'} or @code{'a'}, the file should be seekable.

@emph{compression} is the ZIP compression method to use when writing the archive,
and should be @ref{222a,,ZIP_STORED}, @ref{222b,,ZIP_DEFLATED},
@ref{222c,,ZIP_BZIP2} or @ref{222d,,ZIP_LZMA}; unrecognized
values will cause @ref{8a3,,NotImplementedError} to be raised.  If
@ref{222b,,ZIP_DEFLATED}, @ref{222c,,ZIP_BZIP2} or @ref{222d,,ZIP_LZMA} is specified
but the corresponding module (@ref{14b,,zlib}, @ref{12,,bz2} or @ref{a9,,lzma}) is not
available, @ref{5af,,RuntimeError} is raised. The default is @ref{222a,,ZIP_STORED}.

If @emph{allowZip64} is @code{True} (the default) zipfile will create ZIP files that
use the ZIP64 extensions when the zipfile is larger than 4 GiB. If it is
@code{false} @ref{149,,zipfile} will raise an exception when the ZIP file would
require ZIP64 extensions.

The @emph{compresslevel} parameter controls the compression level to use when
writing files to the archive.
When using @ref{222a,,ZIP_STORED} or @ref{222d,,ZIP_LZMA} it has no effect.
When using @ref{222b,,ZIP_DEFLATED} integers @code{0} through @code{9} are accepted
(see @ref{21db,,zlib} for more information).
When using @ref{222c,,ZIP_BZIP2} integers @code{1} through @code{9} are accepted
(see @ref{455,,bz2} for more information).

The @emph{strict_timestamps} argument, when set to @code{False}, allows to
zip files older than 1980-01-01 at the cost of setting the
timestamp to 1980-01-01.
Similar behavior occurs with files newer than 2107-12-31,
the timestamp is also set to the limit.

If the file is created with mode @code{'w'}, @code{'x'} or @code{'a'} and then
@ref{222f,,closed} without adding any files to the archive, the appropriate
ZIP structures for an empty archive will be written to the file.

ZipFile is also a context manager and therefore supports the
@ref{168,,with} statement.  In the example, @emph{myzip} is closed after the
@code{with} statement’s suite is finished—even if an exception occurs:

@example
with ZipFile('spam.zip', 'w') as myzip:
    myzip.write('eggs.txt')
@end example


New in version 3.2: Added the ability to use @ref{6e5,,ZipFile} as a context manager.



Changed in version 3.3: Added support for @ref{12,,bzip2} and @ref{a9,,lzma} compression.



Changed in version 3.4: ZIP64 extensions are enabled by default.



Changed in version 3.5: Added support for writing to unseekable streams.
Added support for the @code{'x'} mode.



Changed in version 3.6: Previously, a plain @ref{5af,,RuntimeError} was raised for unrecognized
compression values.



Changed in version 3.6.2: The @emph{file} parameter accepts a @ref{63d,,path-like object}.



Changed in version 3.7: Add the @emph{compresslevel} parameter.



New in version 3.8: The @emph{strict_timestamps} keyword-only argument

@end deffn

@geindex close() (zipfile.ZipFile method)
@anchor{library/zipfile zipfile ZipFile close}@anchor{222f}
@deffn {Method} ZipFile.close ()

Close the archive file.  You must call @ref{222f,,close()} before exiting your program
or essential records will not be written.
@end deffn

@geindex getinfo() (zipfile.ZipFile method)
@anchor{library/zipfile zipfile ZipFile getinfo}@anchor{2227}
@deffn {Method} ZipFile.getinfo (name)

Return a @ref{855,,ZipInfo} object with information about the archive member
@emph{name}.  Calling @ref{2227,,getinfo()} for a name not currently contained in the
archive will raise a @ref{5bb,,KeyError}.
@end deffn

@geindex infolist() (zipfile.ZipFile method)
@anchor{library/zipfile zipfile ZipFile infolist}@anchor{2228}
@deffn {Method} ZipFile.infolist ()

Return a list containing a @ref{855,,ZipInfo} object for each member of the
archive.  The objects are in the same order as their entries in the actual ZIP
file on disk if an existing archive was opened.
@end deffn

@geindex namelist() (zipfile.ZipFile method)
@anchor{library/zipfile zipfile ZipFile namelist}@anchor{2230}
@deffn {Method} ZipFile.namelist ()

Return a list of archive members by name.
@end deffn

@geindex open() (zipfile.ZipFile method)
@anchor{library/zipfile zipfile ZipFile open}@anchor{857}
@deffn {Method} ZipFile.open (name, mode='r', pwd=None, *, force_zip64=False)

Access a member of the archive as a binary file-like object.  @emph{name}
can be either the name of a file within the archive or a @ref{855,,ZipInfo}
object.  The @emph{mode} parameter, if included, must be @code{'r'} (the default)
or @code{'w'}.  @emph{pwd} is the password used to decrypt encrypted ZIP files.

@ref{857,,open()} is also a context manager and therefore supports the
@ref{168,,with} statement:

@example
with ZipFile('spam.zip') as myzip:
    with myzip.open('eggs.txt') as myfile:
        print(myfile.read())
@end example

With @emph{mode} @code{'r'} the file-like object
(@code{ZipExtFile}) is read-only and provides the following methods:
@ref{21ef,,read()}, @ref{1b74,,readline()},
@ref{1c17,,readlines()}, @ref{2231,,seek()},
@ref{2232,,tell()}, @code{__iter__()}, @ref{e6d,,__next__()}.
These objects can operate independently of the ZipFile.

With @code{mode='w'}, a writable file handle is returned, which supports the
@ref{824,,write()} method.  While a writable file handle is open,
attempting to read or write other files in the ZIP file will raise a
@ref{17c,,ValueError}.

When writing a file, if the file size is not known in advance but may exceed
2 GiB, pass @code{force_zip64=True} to ensure that the header format is
capable of supporting large files.  If the file size is known in advance,
construct a @ref{855,,ZipInfo} object with @ref{2233,,file_size} set, and
use that as the @emph{name} parameter.

@cartouche
@quotation Note 
The @ref{857,,open()}, @ref{ebb,,read()} and @ref{2234,,extract()} methods can take a filename
or a @ref{855,,ZipInfo} object.  You will appreciate this when trying to read a
ZIP file that contains members with duplicate names.
@end quotation
@end cartouche


Changed in version 3.6: Removed support of @code{mode='U'}.  Use @ref{889,,io.TextIOWrapper} for reading
compressed text files in @ref{88a,,universal newlines} mode.



Changed in version 3.6: @ref{1c8,,open()} can now be used to write files into the archive with the
@code{mode='w'} option.



Changed in version 3.6: Calling @ref{857,,open()} on a closed ZipFile will raise a @ref{17c,,ValueError}.
Previously, a @ref{5af,,RuntimeError} was raised.

@end deffn

@geindex extract() (zipfile.ZipFile method)
@anchor{library/zipfile zipfile ZipFile extract}@anchor{2234}
@deffn {Method} ZipFile.extract (member, path=None, pwd=None)

Extract a member from the archive to the current working directory; @emph{member}
must be its full name or a @ref{855,,ZipInfo} object.  Its file information is
extracted as accurately as possible.  @emph{path} specifies a different directory
to extract to.  @emph{member} can be a filename or a @ref{855,,ZipInfo} object.
@emph{pwd} is the password used for encrypted files.

Returns the normalized path created (a directory or new file).

@cartouche
@quotation Note 
If a member filename is an absolute path, a drive/UNC sharepoint and
leading (back)slashes will be stripped, e.g.: @code{///foo/bar} becomes
@code{foo/bar} on Unix, and @code{C:\foo\bar} becomes @code{foo\bar} on Windows.
And all @code{".."} components in a member filename will be removed, e.g.:
@code{../../foo../../ba..r} becomes @code{foo../ba..r}.  On Windows illegal
characters (@code{:}, @code{<}, @code{>}, @code{|}, @code{"}, @code{?}, and @code{*})
replaced by underscore (@code{_}).
@end quotation
@end cartouche


Changed in version 3.6: Calling @ref{2234,,extract()} on a closed ZipFile will raise a
@ref{17c,,ValueError}.  Previously, a @ref{5af,,RuntimeError} was raised.



Changed in version 3.6.2: The @emph{path} parameter accepts a @ref{63d,,path-like object}.

@end deffn

@geindex extractall() (zipfile.ZipFile method)
@anchor{library/zipfile zipfile ZipFile extractall}@anchor{2235}
@deffn {Method} ZipFile.extractall (path=None, members=None, pwd=None)

Extract all members from the archive to the current working directory.  @emph{path}
specifies a different directory to extract to.  @emph{members} is optional and must
be a subset of the list returned by @ref{2230,,namelist()}.  @emph{pwd} is the password
used for encrypted files.

@cartouche
@quotation Warning 
Never extract archives from untrusted sources without prior inspection.
It is possible that files are created outside of @emph{path}, e.g. members
that have absolute filenames starting with @code{"/"} or filenames with two
dots @code{".."}.  This module attempts to prevent that.
See @ref{2234,,extract()} note.
@end quotation
@end cartouche


Changed in version 3.6: Calling @ref{2235,,extractall()} on a closed ZipFile will raise a
@ref{17c,,ValueError}.  Previously, a @ref{5af,,RuntimeError} was raised.



Changed in version 3.6.2: The @emph{path} parameter accepts a @ref{63d,,path-like object}.

@end deffn

@geindex printdir() (zipfile.ZipFile method)
@anchor{library/zipfile zipfile ZipFile printdir}@anchor{2236}
@deffn {Method} ZipFile.printdir ()

Print a table of contents for the archive to @code{sys.stdout}.
@end deffn

@geindex setpassword() (zipfile.ZipFile method)
@anchor{library/zipfile zipfile ZipFile setpassword}@anchor{2237}
@deffn {Method} ZipFile.setpassword (pwd)

Set @emph{pwd} as default password to extract encrypted files.
@end deffn

@geindex read() (zipfile.ZipFile method)
@anchor{library/zipfile zipfile ZipFile read}@anchor{ebb}
@deffn {Method} ZipFile.read (name, pwd=None)

Return the bytes of the file @emph{name} in the archive.  @emph{name} is the name of the
file in the archive, or a @ref{855,,ZipInfo} object.  The archive must be open for
read or append. @emph{pwd} is the password used for encrypted  files and, if specified,
it will override the default password set with @ref{2237,,setpassword()}.  Calling
@ref{ebb,,read()} on a ZipFile that uses a compression method other than
@ref{222a,,ZIP_STORED}, @ref{222b,,ZIP_DEFLATED}, @ref{222c,,ZIP_BZIP2} or
@ref{222d,,ZIP_LZMA} will raise a @ref{8a3,,NotImplementedError}. An error will also
be raised if the corresponding compression module is not available.


Changed in version 3.6: Calling @ref{ebb,,read()} on a closed ZipFile will raise a @ref{17c,,ValueError}.
Previously, a @ref{5af,,RuntimeError} was raised.

@end deffn

@geindex testzip() (zipfile.ZipFile method)
@anchor{library/zipfile zipfile ZipFile testzip}@anchor{2238}
@deffn {Method} ZipFile.testzip ()

Read all the files in the archive and check their CRC’s and file headers.
Return the name of the first bad file, or else return @code{None}.


Changed in version 3.6: Calling @ref{2238,,testzip()} on a closed ZipFile will raise a
@ref{17c,,ValueError}.  Previously, a @ref{5af,,RuntimeError} was raised.

@end deffn

@geindex write() (zipfile.ZipFile method)
@anchor{library/zipfile zipfile ZipFile write}@anchor{8a4}
@deffn {Method} ZipFile.write (filename, arcname=None, compress_type=None, compresslevel=None)

Write the file named @emph{filename} to the archive, giving it the archive name
@emph{arcname} (by default, this will be the same as @emph{filename}, but without a drive
letter and with leading path separators removed).  If given, @emph{compress_type}
overrides the value given for the @emph{compression} parameter to the constructor for
the new entry. Similarly, @emph{compresslevel} will override the constructor if
given.
The archive must be open with mode @code{'w'}, @code{'x'} or @code{'a'}.

@cartouche
@quotation Note 
Archive names should be relative to the archive root, that is, they should not
start with a path separator.
@end quotation
@end cartouche

@cartouche
@quotation Note 
If @code{arcname} (or @code{filename}, if @code{arcname} is  not given) contains a null
byte, the name of the file in the archive will be truncated at the null byte.
@end quotation
@end cartouche

@cartouche
@quotation Note 
A leading slash in the filename may lead to the archive being impossible to
open in some zip programs on Windows systems.
@end quotation
@end cartouche


Changed in version 3.6: Calling @ref{8a4,,write()} on a ZipFile created with mode @code{'r'} or
a closed ZipFile will raise a @ref{17c,,ValueError}.  Previously,
a @ref{5af,,RuntimeError} was raised.

@end deffn

@geindex writestr() (zipfile.ZipFile method)
@anchor{library/zipfile zipfile ZipFile writestr}@anchor{ebd}
@deffn {Method} ZipFile.writestr (zinfo_or_arcname, data, compress_type=None, compresslevel=None)

Write a file into the archive.  The contents is @emph{data}, which may be either
a @ref{2ef,,str} or a @ref{2f0,,bytes} instance; if it is a @ref{2ef,,str},
it is encoded as UTF-8 first.  @emph{zinfo_or_arcname} is either the file
name it will be given in the archive, or a @ref{855,,ZipInfo} instance.  If it’s
an instance, at least the filename, date, and time must be given.  If it’s a
name, the date and time is set to the current date and time.
The archive must be opened with mode @code{'w'}, @code{'x'} or @code{'a'}.

If given, @emph{compress_type} overrides the value given for the @emph{compression}
parameter to the constructor for the new entry, or in the @emph{zinfo_or_arcname}
(if that is a @ref{855,,ZipInfo} instance). Similarly, @emph{compresslevel} will
override the constructor if given.

@cartouche
@quotation Note 
When passing a @ref{855,,ZipInfo} instance as the @emph{zinfo_or_arcname} parameter,
the compression method used will be that specified in the @emph{compress_type}
member of the given @ref{855,,ZipInfo} instance.  By default, the
@ref{855,,ZipInfo} constructor sets this member to @ref{222a,,ZIP_STORED}.
@end quotation
@end cartouche


Changed in version 3.2: The @emph{compress_type} argument.



Changed in version 3.6: Calling @ref{ebd,,writestr()} on a ZipFile created with mode @code{'r'} or
a closed ZipFile will raise a @ref{17c,,ValueError}.  Previously,
a @ref{5af,,RuntimeError} was raised.

@end deffn

The following data attributes are also available:

@geindex filename (zipfile.ZipFile attribute)
@anchor{library/zipfile zipfile ZipFile filename}@anchor{2239}
@deffn {Attribute} ZipFile.filename

Name of the ZIP file.
@end deffn

@geindex debug (zipfile.ZipFile attribute)
@anchor{library/zipfile zipfile ZipFile debug}@anchor{223a}
@deffn {Attribute} ZipFile.debug

The level of debug output to use.  This may be set from @code{0} (the default, no
output) to @code{3} (the most output).  Debugging information is written to
@code{sys.stdout}.
@end deffn

@geindex comment (zipfile.ZipFile attribute)
@anchor{library/zipfile zipfile ZipFile comment}@anchor{223b}
@deffn {Attribute} ZipFile.comment

The comment associated with the ZIP file as a @ref{2f0,,bytes} object.
If assigning a comment to a
@ref{6e5,,ZipFile} instance created with mode @code{'w'}, @code{'x'} or @code{'a'},
it should be no longer than 65535 bytes.  Comments longer than this will be
truncated.
@end deffn

@node Path Objects,PyZipFile Objects,ZipFile Objects,zipfile — Work with ZIP archives
@anchor{library/zipfile id2}@anchor{223c}@anchor{library/zipfile path-objects}@anchor{2226}
@subsubsection Path Objects


@geindex Path (class in zipfile)
@anchor{library/zipfile zipfile Path}@anchor{1111}
@deffn {Class} zipfile.Path (root, at='@w{'})

Construct a Path object from a @code{root} zipfile (which may be a
@ref{6e5,,ZipFile} instance or @code{file} suitable for passing to
the @ref{6e5,,ZipFile} constructor).

@code{at} specifies the location of this Path within the zipfile,
e.g. ‘dir/file.txt’, ‘dir/’, or ‘’. Defaults to the empty string,
indicating the root.
@end deffn

Path objects expose the following features of @ref{522,,pathlib.Path}
objects:

Path objects are traversable using the @code{/} operator or @code{joinpath}.

@geindex name (zipfile.Path attribute)
@anchor{library/zipfile zipfile Path name}@anchor{223d}
@deffn {Attribute} Path.name

The final path component.
@end deffn

@geindex open() (zipfile.Path method)
@anchor{library/zipfile zipfile Path open}@anchor{223e}
@deffn {Method} Path.open (mode='r', *, pwd, **)

Invoke @ref{857,,ZipFile.open()} on the current path.
Allows opening for read or write, text or binary
through supported modes: ‘r’, ‘w’, ‘rb’, ‘wb’.
Positional and keyword arguments are passed through to
@ref{889,,io.TextIOWrapper} when opened as text and
ignored otherwise.
@code{pwd} is the @code{pwd} parameter to
@ref{857,,ZipFile.open()}.


Changed in version 3.9: Added support for text and binary modes for open. Default
mode is now text.

@end deffn

@geindex iterdir() (zipfile.Path method)
@anchor{library/zipfile zipfile Path iterdir}@anchor{223f}
@deffn {Method} Path.iterdir ()

Enumerate the children of the current directory.
@end deffn

@geindex is_dir() (zipfile.Path method)
@anchor{library/zipfile zipfile Path is_dir}@anchor{2240}
@deffn {Method} Path.is_dir ()

Return @code{True} if the current context references a directory.
@end deffn

@geindex is_file() (zipfile.Path method)
@anchor{library/zipfile zipfile Path is_file}@anchor{2241}
@deffn {Method} Path.is_file ()

Return @code{True} if the current context references a file.
@end deffn

@geindex exists() (zipfile.Path method)
@anchor{library/zipfile zipfile Path exists}@anchor{2242}
@deffn {Method} Path.exists ()

Return @code{True} if the current context references a file or
directory in the zip file.
@end deffn

@geindex Path.suffix (in module zipfile)
@anchor{library/zipfile zipfile Path suffix}@anchor{1114}
@deffn {Data} Path.suffix

The file extension of the final component.


New in version 3.11: Added @ref{1114,,Path.suffix} property.

@end deffn

@geindex Path.stem (in module zipfile)
@anchor{library/zipfile zipfile Path stem}@anchor{1112}
@deffn {Data} Path.stem

The final path component, without its suffix.


New in version 3.11: Added @ref{1112,,Path.stem} property.

@end deffn

@geindex Path.suffixes (in module zipfile)
@anchor{library/zipfile zipfile Path suffixes}@anchor{1113}
@deffn {Data} Path.suffixes

A list of the path’s file extensions.


New in version 3.11: Added @ref{1113,,Path.suffixes} property.

@end deffn

@geindex read_text() (zipfile.Path method)
@anchor{library/zipfile zipfile Path read_text}@anchor{2243}
@deffn {Method} Path.read_text (*, **)

Read the current file as unicode text. Positional and
keyword arguments are passed through to
@ref{889,,io.TextIOWrapper} (except @code{buffer}, which is
implied by the context).
@end deffn

@geindex read_bytes() (zipfile.Path method)
@anchor{library/zipfile zipfile Path read_bytes}@anchor{2244}
@deffn {Method} Path.read_bytes ()

Read the current file as bytes.
@end deffn

@geindex joinpath() (zipfile.Path method)
@anchor{library/zipfile zipfile Path joinpath}@anchor{2245}
@deffn {Method} Path.joinpath (*other)

Return a new Path object with each of the @emph{other} arguments
joined. The following are equivalent:

@example
>>> Path(...).joinpath('child').joinpath('grandchild')
>>> Path(...).joinpath('child', 'grandchild')
>>> Path(...) / 'child' / 'grandchild'
@end example


Changed in version 3.10: Prior to 3.10, @code{joinpath} was undocumented and accepted
exactly one parameter.

@end deffn

The zipp@footnote{https://pypi.org/project/zipp} project provides backports
of the latest path object functionality to older Pythons. Use
@code{zipp.Path} in place of @code{zipfile.Path} for early access to
changes.

@node PyZipFile Objects,ZipInfo Objects,Path Objects,zipfile — Work with ZIP archives
@anchor{library/zipfile id3}@anchor{2246}@anchor{library/zipfile pyzipfile-objects}@anchor{2247}
@subsubsection PyZipFile Objects


The @ref{b80,,PyZipFile} constructor takes the same parameters as the
@ref{6e5,,ZipFile} constructor, and one additional parameter, @emph{optimize}.

@geindex PyZipFile (class in zipfile)
@anchor{library/zipfile zipfile PyZipFile}@anchor{b80}
@deffn {Class} zipfile.PyZipFile (file, mode='r', compression=ZIP_STORED, allowZip64=True, optimize=- 1)


New in version 3.2: The @emph{optimize} parameter.



Changed in version 3.4: ZIP64 extensions are enabled by default.


Instances have one method in addition to those of @ref{6e5,,ZipFile} objects:

@geindex writepy() (zipfile.PyZipFile method)
@anchor{library/zipfile zipfile PyZipFile writepy}@anchor{b7f}
@deffn {Method} writepy (pathname, basename='@w{'}, filterfunc=None)

Search for files @code{*.py} and add the corresponding file to the
archive.

If the @emph{optimize} parameter to @ref{b80,,PyZipFile} was not given or @code{-1},
the corresponding file is a @code{*.pyc} file, compiling if necessary.

If the @emph{optimize} parameter to @ref{b80,,PyZipFile} was @code{0}, @code{1} or
@code{2}, only files with that optimization level (see @ref{4e2,,compile()}) are
added to the archive, compiling if necessary.

If @emph{pathname} is a file, the filename must end with @code{.py}, and
just the (corresponding @code{*.pyc}) file is added at the top level
(no path information).  If @emph{pathname} is a file that does not end with
@code{.py}, a @ref{5af,,RuntimeError} will be raised.  If it is a directory,
and the directory is not a package directory, then all the files
@code{*.pyc} are added at the top level.  If the directory is a
package directory, then all @code{*.pyc} are added under the package
name as a file path, and if any subdirectories are package directories,
all of these are added recursively in sorted order.

@emph{basename} is intended for internal use only.

@emph{filterfunc}, if given, must be a function taking a single string
argument.  It will be passed each path (including each individual full
file path) before it is added to the archive.  If @emph{filterfunc} returns a
false value, the path will not be added, and if it is a directory its
contents will be ignored.  For example, if our test files are all either
in @code{test} directories or start with the string @code{test_}, we can use a
@emph{filterfunc} to exclude them:

@example
>>> zf = PyZipFile('myprog.zip')
>>> def notests(s):
...     fn = os.path.basename(s)
...     return (not (fn == 'test' or fn.startswith('test_')))
>>> zf.writepy('myprog', filterfunc=notests)
@end example

The @ref{b7f,,writepy()} method makes archives with file names like
this:

@example
string.pyc                   # Top level name
test/__init__.pyc            # Package directory
test/testall.pyc             # Module test.testall
test/bogus/__init__.pyc      # Subpackage directory
test/bogus/myfile.pyc        # Submodule test.bogus.myfile
@end example


New in version 3.4: The @emph{filterfunc} parameter.



Changed in version 3.6.2: The @emph{pathname} parameter accepts a @ref{63d,,path-like object}.



Changed in version 3.7: Recursion sorts directory entries.

@end deffn
@end deffn

@node ZipInfo Objects,Command-Line Interface,PyZipFile Objects,zipfile — Work with ZIP archives
@anchor{library/zipfile id4}@anchor{2248}@anchor{library/zipfile zipinfo-objects}@anchor{2229}
@subsubsection ZipInfo Objects


Instances of the @ref{855,,ZipInfo} class are returned by the @ref{2227,,getinfo()} and
@ref{2228,,infolist()} methods of @ref{6e5,,ZipFile} objects.  Each object stores
information about a single member of the ZIP archive.

There is one classmethod to make a @ref{855,,ZipInfo} instance for a filesystem
file:

@geindex from_file() (zipfile.ZipInfo class method)
@anchor{library/zipfile zipfile ZipInfo from_file}@anchor{854}
@deffn {Method} classmethod ZipInfo.from_file (filename, arcname=None, *, strict_timestamps=True)

Construct a @ref{855,,ZipInfo} instance for a file on the filesystem, in
preparation for adding it to a zip file.

@emph{filename} should be the path to a file or directory on the filesystem.

If @emph{arcname} is specified, it is used as the name within the archive.
If @emph{arcname} is not specified, the name will be the same as @emph{filename}, but
with any drive letter and leading path separators removed.

The @emph{strict_timestamps} argument, when set to @code{False}, allows to
zip files older than 1980-01-01 at the cost of setting the
timestamp to 1980-01-01.
Similar behavior occurs with files newer than 2107-12-31,
the timestamp is also set to the limit.


New in version 3.6.



Changed in version 3.6.2: The @emph{filename} parameter accepts a @ref{63d,,path-like object}.



New in version 3.8: The @emph{strict_timestamps} keyword-only argument

@end deffn

Instances have the following methods and attributes:

@geindex is_dir() (zipfile.ZipInfo method)
@anchor{library/zipfile zipfile ZipInfo is_dir}@anchor{856}
@deffn {Method} ZipInfo.is_dir ()

Return @code{True} if this archive member is a directory.

This uses the entry’s name: directories should always end with @code{/}.


New in version 3.6.

@end deffn

@geindex filename (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo filename}@anchor{2249}
@deffn {Attribute} ZipInfo.filename

Name of the file in the archive.
@end deffn

@geindex date_time (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo date_time}@anchor{224a}
@deffn {Attribute} ZipInfo.date_time

The time and date of the last modification to the archive member.  This is a
tuple of six values:


@multitable {xxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Index

@tab

Value

@item

@code{0}

@tab

Year (>= 1980)

@item

@code{1}

@tab

Month (one-based)

@item

@code{2}

@tab

Day of month (one-based)

@item

@code{3}

@tab

Hours (zero-based)

@item

@code{4}

@tab

Minutes (zero-based)

@item

@code{5}

@tab

Seconds (zero-based)

@end multitable


@cartouche
@quotation Note 
The ZIP file format does not support timestamps before 1980.
@end quotation
@end cartouche
@end deffn

@geindex compress_type (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo compress_type}@anchor{224b}
@deffn {Attribute} ZipInfo.compress_type

Type of compression for the archive member.
@end deffn

@geindex comment (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo comment}@anchor{224c}
@deffn {Attribute} ZipInfo.comment

Comment for the individual archive member as a @ref{2f0,,bytes} object.
@end deffn

@geindex extra (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo extra}@anchor{224d}
@deffn {Attribute} ZipInfo.extra

Expansion field data.  The PKZIP Application Note@footnote{https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT} contains
some comments on the internal structure of the data contained in this
@ref{2f0,,bytes} object.
@end deffn

@geindex create_system (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo create_system}@anchor{224e}
@deffn {Attribute} ZipInfo.create_system

System which created ZIP archive.
@end deffn

@geindex create_version (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo create_version}@anchor{224f}
@deffn {Attribute} ZipInfo.create_version

PKZIP version which created ZIP archive.
@end deffn

@geindex extract_version (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo extract_version}@anchor{2250}
@deffn {Attribute} ZipInfo.extract_version

PKZIP version needed to extract archive.
@end deffn

@geindex reserved (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo reserved}@anchor{2251}
@deffn {Attribute} ZipInfo.reserved

Must be zero.
@end deffn

@geindex flag_bits (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo flag_bits}@anchor{2252}
@deffn {Attribute} ZipInfo.flag_bits

ZIP flag bits.
@end deffn

@geindex volume (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo volume}@anchor{2253}
@deffn {Attribute} ZipInfo.volume

Volume number of file header.
@end deffn

@geindex internal_attr (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo internal_attr}@anchor{2254}
@deffn {Attribute} ZipInfo.internal_attr

Internal attributes.
@end deffn

@geindex external_attr (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo external_attr}@anchor{2255}
@deffn {Attribute} ZipInfo.external_attr

External file attributes.
@end deffn

@geindex header_offset (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo header_offset}@anchor{2256}
@deffn {Attribute} ZipInfo.header_offset

Byte offset to the file header.
@end deffn

@geindex CRC (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo CRC}@anchor{2257}
@deffn {Attribute} ZipInfo.CRC

CRC-32 of the uncompressed file.
@end deffn

@geindex compress_size (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo compress_size}@anchor{2258}
@deffn {Attribute} ZipInfo.compress_size

Size of the compressed data.
@end deffn

@geindex file_size (zipfile.ZipInfo attribute)
@anchor{library/zipfile zipfile ZipInfo file_size}@anchor{2233}
@deffn {Attribute} ZipInfo.file_size

Size of the uncompressed file.
@end deffn

@node Command-Line Interface,Decompression pitfalls,ZipInfo Objects,zipfile — Work with ZIP archives
@anchor{library/zipfile command-line-interface}@anchor{2259}@anchor{library/zipfile zipfile-commandline}@anchor{225a}
@subsubsection Command-Line Interface


The @ref{149,,zipfile} module provides a simple command-line interface to interact
with ZIP archives.

If you want to create a new ZIP archive, specify its name after the @ref{225b,,-c}
option and then list the filename(s) that should be included:

@example
$ python -m zipfile -c monty.zip spam.txt eggs.txt
@end example

Passing a directory is also acceptable:

@example
$ python -m zipfile -c monty.zip life-of-brian_1979/
@end example

If you want to extract a ZIP archive into the specified directory, use
the @ref{225c,,-e} option:

@example
$ python -m zipfile -e monty.zip target-dir/
@end example

For a list of the files in a ZIP archive, use the @ref{225d,,-l} option:

@example
$ python -m zipfile -l monty.zip
@end example

@menu
* Command-line options:: 

@end menu

@node Command-line options,,,Command-Line Interface
@anchor{library/zipfile command-line-options}@anchor{225e}
@subsubsection Command-line options


@geindex zipfile command line option; -l <zipfile>
@geindex zipfile command line option; --list <zipfile>
@anchor{library/zipfile cmdoption-zipfile-l}@anchor{225d}
@deffn {Program Option} @w{-}l <zipfile>
@anchor{library/zipfile cmdoption-zipfile-list}@anchor{225f}
@deffnx {Program Option} @w{-}@w{-}list <zipfile>

List files in a zipfile.
@end deffn

@geindex zipfile command line option; -c <zipfile> <source1> ... <sourceN>
@geindex zipfile command line option; --create <zipfile> <source1> ... <sourceN>
@anchor{library/zipfile cmdoption-zipfile-c}@anchor{225b}
@deffn {Program Option} @w{-}c <zipfile> <source1> ... <sourceN>
@anchor{library/zipfile cmdoption-zipfile-create}@anchor{2260}
@deffnx {Program Option} @w{-}@w{-}create <zipfile> <source1> ... <sourceN>

Create zipfile from source files.
@end deffn

@geindex zipfile command line option; -e <zipfile> <output_dir>
@geindex zipfile command line option; --extract <zipfile> <output_dir>
@anchor{library/zipfile cmdoption-zipfile-e}@anchor{225c}
@deffn {Program Option} @w{-}e <zipfile> <output_dir>
@anchor{library/zipfile cmdoption-zipfile-extract}@anchor{2261}
@deffnx {Program Option} @w{-}@w{-}extract <zipfile> <output_dir>

Extract zipfile into target directory.
@end deffn

@geindex zipfile command line option; -t <zipfile>
@geindex zipfile command line option; --test <zipfile>
@anchor{library/zipfile cmdoption-zipfile-t}@anchor{2262}
@deffn {Program Option} @w{-}t <zipfile>
@anchor{library/zipfile cmdoption-zipfile-test}@anchor{2263}
@deffnx {Program Option} @w{-}@w{-}test <zipfile>

Test whether the zipfile is valid or not.
@end deffn

@node Decompression pitfalls,,Command-Line Interface,zipfile — Work with ZIP archives
@anchor{library/zipfile decompression-pitfalls}@anchor{2264}
@subsubsection Decompression pitfalls


The extraction in zipfile module might fail due to some pitfalls listed below.

@menu
* From file itself:: 
* File System limitations:: 
* Resources limitations:: 
* Interruption:: 
* Default behaviors of extraction:: 

@end menu

@node From file itself,File System limitations,,Decompression pitfalls
@anchor{library/zipfile from-file-itself}@anchor{2265}
@subsubsection From file itself


Decompression may fail due to incorrect password / CRC checksum / ZIP format or
unsupported compression method / decryption.

@node File System limitations,Resources limitations,From file itself,Decompression pitfalls
@anchor{library/zipfile file-system-limitations}@anchor{2266}
@subsubsection File System limitations


Exceeding limitations on different file systems can cause decompression failed.
Such as allowable characters in the directory entries, length of the file name,
length of the pathname, size of a single file, and number of files, etc.

@node Resources limitations,Interruption,File System limitations,Decompression pitfalls
@anchor{library/zipfile resources-limitations}@anchor{2267}@anchor{library/zipfile zipfile-resources-limitations}@anchor{2268}
@subsubsection Resources limitations


The lack of memory or disk volume would lead to decompression
failed. For example, decompression bombs (aka ZIP bomb@footnote{https://en.wikipedia.org/wiki/Zip_bomb})
apply to zipfile library that can cause disk volume exhaustion.

@node Interruption,Default behaviors of extraction,Resources limitations,Decompression pitfalls
@anchor{library/zipfile interruption}@anchor{2269}
@subsubsection Interruption


Interruption during the decompression, such as pressing control-C or killing the
decompression process may result in incomplete decompression of the archive.

@node Default behaviors of extraction,,Interruption,Decompression pitfalls
@anchor{library/zipfile default-behaviors-of-extraction}@anchor{226a}
@subsubsection Default behaviors of extraction


Not knowing the default extraction behaviors
can cause unexpected decompression results.
For example, when extracting the same archive twice,
it overwrites files without asking.

@node tarfile — Read and write tar archive files,,zipfile — Work with ZIP archives,Data Compression and Archiving
@anchor{library/tarfile doc}@anchor{226b}@anchor{library/tarfile module-tarfile}@anchor{fb}@anchor{library/tarfile pkzip-application-note}@anchor{226c}@anchor{library/tarfile tarfile-read-and-write-tar-archive-files}@anchor{226d}
@subsection @code{tarfile} — Read and write tar archive files


@geindex module; tarfile

@strong{Source code:} Lib/tarfile.py@footnote{https://github.com/python/cpython/tree/main/Lib/tarfile.py}


__________________________________________________________________


The @ref{fb,,tarfile} module makes it possible to read and write tar
archives, including those using gzip, bz2 and lzma compression.
Use the @ref{149,,zipfile} module to read or write @code{.zip} files, or the
higher-level functions in @ref{dba,,shutil}.

Some facts and figures:


@itemize *

@item 
reads and writes @ref{87,,gzip}, @ref{12,,bz2} and @ref{a9,,lzma} compressed archives
if the respective modules are available.

@item 
read/write support for the POSIX.1-1988 (ustar) format.

@item 
read/write support for the GNU tar format including @emph{longname} and @emph{longlink}
extensions, read-only support for all variants of the @emph{sparse} extension
including restoration of sparse files.

@item 
read/write support for the POSIX.1-2001 (pax) format.

@item 
handles directories, regular files, hardlinks, symbolic links, fifos,
character devices and block devices and is able to acquire and restore file
information like timestamp, access permissions and owner.
@end itemize


Changed in version 3.3: Added support for @ref{a9,,lzma} compression.


@geindex open() (in module tarfile)
@anchor{library/tarfile tarfile open}@anchor{9e5}
@deffn {Function} tarfile.open (name=None, mode='r', fileobj=None, bufsize=10240, **kwargs)

Return a @ref{db1,,TarFile} object for the pathname @emph{name}. For detailed
information on @ref{db1,,TarFile} objects and the keyword arguments that are
allowed, see @ref{226e,,TarFile Objects}.

@emph{mode} has to be a string of the form @code{'filemode[:compression]'}, it defaults
to @code{'r'}. Here is a full list of mode combinations:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

mode

@tab

action

@item

@code{'r' or 'r:*'}

@tab

Open for reading with transparent
compression (recommended).

@item

@code{'r:'}

@tab

Open for reading exclusively without
compression.

@item

@code{'r:gz'}

@tab

Open for reading with gzip compression.

@item

@code{'r:bz2'}

@tab

Open for reading with bzip2 compression.

@item

@code{'r:xz'}

@tab

Open for reading with lzma compression.

@item

@code{'x'} or
@code{'x:'}

@tab

Create a tarfile exclusively without
compression.
Raise a @ref{bb1,,FileExistsError} exception
if it already exists.

@item

@code{'x:gz'}

@tab

Create a tarfile with gzip compression.
Raise a @ref{bb1,,FileExistsError} exception
if it already exists.

@item

@code{'x:bz2'}

@tab

Create a tarfile with bzip2 compression.
Raise a @ref{bb1,,FileExistsError} exception
if it already exists.

@item

@code{'x:xz'}

@tab

Create a tarfile with lzma compression.
Raise a @ref{bb1,,FileExistsError} exception
if it already exists.

@item

@code{'a' or 'a:'}

@tab

Open for appending with no compression. The
file is created if it does not exist.

@item

@code{'w' or 'w:'}

@tab

Open for uncompressed writing.

@item

@code{'w:gz'}

@tab

Open for gzip compressed writing.

@item

@code{'w:bz2'}

@tab

Open for bzip2 compressed writing.

@item

@code{'w:xz'}

@tab

Open for lzma compressed writing.

@end multitable


Note that @code{'a:gz'}, @code{'a:bz2'} or @code{'a:xz'} is not possible. If @emph{mode}
is not suitable to open a certain (compressed) file for reading,
@ref{1100,,ReadError} is raised. Use @emph{mode} @code{'r'} to avoid this.  If a
compression method is not supported, @ref{226f,,CompressionError} is raised.

If @emph{fileobj} is specified, it is used as an alternative to a @ref{d68,,file object}
opened in binary mode for @emph{name}. It is supposed to be at position 0.

For modes @code{'w:gz'}, @code{'r:gz'}, @code{'w:bz2'}, @code{'r:bz2'}, @code{'x:gz'},
@code{'x:bz2'}, @ref{9e5,,tarfile.open()} accepts the keyword argument
@emph{compresslevel} (default @code{9}) to specify the compression level of the file.

For modes @code{'w:xz'} and @code{'x:xz'}, @ref{9e5,,tarfile.open()} accepts the
keyword argument @emph{preset} to specify the compression level of the file.

For special purposes, there is a second format for @emph{mode}:
@code{'filemode|[compression]'}.  @ref{9e5,,tarfile.open()} will return a @ref{db1,,TarFile}
object that processes its data as a stream of blocks.  No random seeking will
be done on the file. If given, @emph{fileobj} may be any object that has a
@code{read()} or @code{write()} method (depending on the @emph{mode}). @emph{bufsize}
specifies the blocksize and defaults to @code{20 * 512} bytes. Use this variant
in combination with e.g. @code{sys.stdin}, a socket @ref{d68,,file object} or a tape
device. However, such a @ref{db1,,TarFile} object is limited in that it does
not allow random access, see @ref{2270,,Examples}.  The currently
possible modes:


@multitable {xxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Mode

@tab

Action

@item

@code{'r|*'}

@tab

Open a @emph{stream} of tar blocks for reading
with transparent compression.

@item

@code{'r|'}

@tab

Open a @emph{stream} of uncompressed tar blocks
for reading.

@item

@code{'r|gz'}

@tab

Open a gzip compressed @emph{stream} for
reading.

@item

@code{'r|bz2'}

@tab

Open a bzip2 compressed @emph{stream} for
reading.

@item

@code{'r|xz'}

@tab

Open an lzma compressed @emph{stream} for
reading.

@item

@code{'w|'}

@tab

Open an uncompressed @emph{stream} for writing.

@item

@code{'w|gz'}

@tab

Open a gzip compressed @emph{stream} for
writing.

@item

@code{'w|bz2'}

@tab

Open a bzip2 compressed @emph{stream} for
writing.

@item

@code{'w|xz'}

@tab

Open an lzma compressed @emph{stream} for
writing.

@end multitable



Changed in version 3.5: The @code{'x'} (exclusive creation) mode was added.



Changed in version 3.6: The @emph{name} parameter accepts a @ref{63d,,path-like object}.

@end deffn


@deffn {Class} tarfile.TarFile

Class for reading and writing tar archives. Do not use this class directly:
use @ref{9e5,,tarfile.open()} instead. See @ref{226e,,TarFile Objects}.
@end deffn

@geindex is_tarfile() (in module tarfile)
@anchor{library/tarfile tarfile is_tarfile}@anchor{1096}
@deffn {Function} tarfile.is_tarfile (name)

Return @ref{748,,True} if @emph{name} is a tar archive file, that the @ref{fb,,tarfile}
module can read. @emph{name} may be a @ref{2ef,,str}, file, or file-like object.


Changed in version 3.9: Support for file and file-like objects.

@end deffn

The @ref{fb,,tarfile} module defines the following exceptions:

@geindex TarError
@anchor{library/tarfile tarfile TarError}@anchor{2271}
@deffn {Exception} tarfile.TarError

Base class for all @ref{fb,,tarfile} exceptions.
@end deffn

@geindex ReadError
@anchor{library/tarfile tarfile ReadError}@anchor{1100}
@deffn {Exception} tarfile.ReadError

Is raised when a tar archive is opened, that either cannot be handled by the
@ref{fb,,tarfile} module or is somehow invalid.
@end deffn

@geindex CompressionError
@anchor{library/tarfile tarfile CompressionError}@anchor{226f}
@deffn {Exception} tarfile.CompressionError

Is raised when a compression method is not supported or when the data cannot be
decoded properly.
@end deffn

@geindex StreamError
@anchor{library/tarfile tarfile StreamError}@anchor{2272}
@deffn {Exception} tarfile.StreamError

Is raised for the limitations that are typical for stream-like @ref{db1,,TarFile}
objects.
@end deffn

@geindex ExtractError
@anchor{library/tarfile tarfile ExtractError}@anchor{2273}
@deffn {Exception} tarfile.ExtractError

Is raised for @emph{non-fatal} errors when using @ref{9e7,,TarFile.extract()}, but only if
@code{TarFile.errorlevel}@code{== 2}.
@end deffn

@geindex HeaderError
@anchor{library/tarfile tarfile HeaderError}@anchor{2274}
@deffn {Exception} tarfile.HeaderError

Is raised by @ref{2275,,TarInfo.frombuf()} if the buffer it gets is invalid.
@end deffn

The following constants are available at the module level:

@geindex ENCODING (in module tarfile)
@anchor{library/tarfile tarfile ENCODING}@anchor{2276}
@deffn {Data} tarfile.ENCODING

The default character encoding: @code{'utf-8'} on Windows, the value returned by
@ref{79e,,sys.getfilesystemencoding()} otherwise.
@end deffn

Each of the following constants defines a tar archive format that the
@ref{fb,,tarfile} module is able to create. See section @ref{2277,,Supported tar formats} for
details.

@geindex USTAR_FORMAT (in module tarfile)
@anchor{library/tarfile tarfile USTAR_FORMAT}@anchor{2278}
@deffn {Data} tarfile.USTAR_FORMAT

POSIX.1-1988 (ustar) format.
@end deffn

@geindex GNU_FORMAT (in module tarfile)
@anchor{library/tarfile tarfile GNU_FORMAT}@anchor{2279}
@deffn {Data} tarfile.GNU_FORMAT

GNU tar format.
@end deffn

@geindex PAX_FORMAT (in module tarfile)
@anchor{library/tarfile tarfile PAX_FORMAT}@anchor{227a}
@deffn {Data} tarfile.PAX_FORMAT

POSIX.1-2001 (pax) format.
@end deffn

@geindex DEFAULT_FORMAT (in module tarfile)
@anchor{library/tarfile tarfile DEFAULT_FORMAT}@anchor{227b}
@deffn {Data} tarfile.DEFAULT_FORMAT

The default format for creating archives. This is currently @ref{227a,,PAX_FORMAT}.


Changed in version 3.8: The default format for new archives was changed to
@ref{227a,,PAX_FORMAT} from @ref{2279,,GNU_FORMAT}.

@end deffn


@subsubheading See also



@table @asis

@item Module @ref{149,,zipfile}

Documentation of the @ref{149,,zipfile} standard module.

@item @ref{dba,,Archiving operations}

Documentation of the higher-level archiving facilities provided by the
standard @ref{e4,,shutil} module.

@item GNU tar manual@comma{} Basic Tar Format@footnote{https://www.gnu.org/software/tar/manual/html_node/Standard.html}

Documentation for tar archive files, including GNU tar extensions.
@end table


@menu
* TarFile Objects:: 
* TarInfo Objects:: 
* Command-Line Interface: Command-Line Interface<2>. 
* Examples: Examples<8>. 
* Supported tar formats:: 
* Unicode issues:: 

@end menu

@node TarFile Objects,TarInfo Objects,,tarfile — Read and write tar archive files
@anchor{library/tarfile id1}@anchor{227c}@anchor{library/tarfile tarfile-objects}@anchor{226e}
@subsubsection TarFile Objects


The @ref{db1,,TarFile} object provides an interface to a tar archive. A tar
archive is a sequence of blocks. An archive member (a stored file) is made up of
a header block followed by data blocks. It is possible to store a file in a tar
archive several times. Each archive member is represented by a @ref{db2,,TarInfo}
object, see @ref{227d,,TarInfo Objects} for details.

A @ref{db1,,TarFile} object can be used as a context manager in a @ref{168,,with}
statement. It will automatically be closed when the block is completed. Please
note that in the event of an exception an archive opened for writing will not
be finalized; only the internally used file object will be closed. See the
@ref{2270,,Examples} section for a use case.


New in version 3.2: Added support for the context management protocol.


@geindex TarFile (class in tarfile)
@anchor{library/tarfile tarfile TarFile}@anchor{db1}
@deffn {Class} tarfile.TarFile (name=None, mode='r', fileobj=None, format=DEFAULT_FORMAT, tarinfo=TarInfo, dereference=False, ignore_zeros=False, encoding=ENCODING, errors='surrogateescape', pax_headers=None, debug=0, errorlevel=0)

All following arguments are optional and can be accessed as instance attributes
as well.

@emph{name} is the pathname of the archive. @emph{name} may be a @ref{63d,,path-like object}.
It can be omitted if @emph{fileobj} is given.
In this case, the file object’s @code{name} attribute is used if it exists.

@emph{mode} is either @code{'r'} to read from an existing archive, @code{'a'} to append
data to an existing file, @code{'w'} to create a new file overwriting an existing
one, or @code{'x'} to create a new file only if it does not already exist.

If @emph{fileobj} is given, it is used for reading or writing data. If it can be
determined, @emph{mode} is overridden by @emph{fileobj}’s mode. @emph{fileobj} will be used
from position 0.

@cartouche
@quotation Note 
@emph{fileobj} is not closed, when @ref{db1,,TarFile} is closed.
@end quotation
@end cartouche

@emph{format} controls the archive format for writing. It must be one of the constants
@ref{2278,,USTAR_FORMAT}, @ref{2279,,GNU_FORMAT} or @ref{227a,,PAX_FORMAT} that are
defined at module level. When reading, format will be automatically detected, even
if different formats are present in a single archive.

The @emph{tarinfo} argument can be used to replace the default @ref{db2,,TarInfo} class
with a different one.

If @emph{dereference} is @ref{659,,False}, add symbolic and hard links to the archive. If it
is @ref{748,,True}, add the content of the target files to the archive. This has no
effect on systems that do not support symbolic links.

If @emph{ignore_zeros} is @ref{659,,False}, treat an empty block as the end of the archive.
If it is @ref{748,,True}, skip empty (and invalid) blocks and try to get as many members
as possible. This is only useful for reading concatenated or damaged archives.

@emph{debug} can be set from @code{0} (no debug messages) up to @code{3} (all debug
messages). The messages are written to @code{sys.stderr}.

If @emph{errorlevel} is @code{0}, all errors are ignored when using @ref{9e7,,TarFile.extract()}.
Nevertheless, they appear as error messages in the debug output, when debugging
is enabled.  If @code{1}, all @emph{fatal} errors are raised as @ref{2a1,,OSError}
exceptions. If @code{2}, all @emph{non-fatal} errors are raised as @ref{2271,,TarError}
exceptions as well.

The @emph{encoding} and @emph{errors} arguments define the character encoding to be
used for reading or writing the archive and how conversion errors are going
to be handled. The default settings will work for most users.
See section @ref{227e,,Unicode issues} for in-depth information.

The @emph{pax_headers} argument is an optional dictionary of strings which
will be added as a pax global header if @emph{format} is @ref{227a,,PAX_FORMAT}.


Changed in version 3.2: Use @code{'surrogateescape'} as the default for the @emph{errors} argument.



Changed in version 3.5: The @code{'x'} (exclusive creation) mode was added.



Changed in version 3.6: The @emph{name} parameter accepts a @ref{63d,,path-like object}.

@end deffn

@geindex open() (tarfile.TarFile class method)
@anchor{library/tarfile tarfile TarFile open}@anchor{1097}
@deffn {Method} classmethod TarFile.open (...)

Alternative constructor. The @ref{9e5,,tarfile.open()} function is actually a
shortcut to this classmethod.
@end deffn

@geindex getmember() (tarfile.TarFile method)
@anchor{library/tarfile tarfile TarFile getmember}@anchor{10aa}
@deffn {Method} TarFile.getmember (name)

Return a @ref{db2,,TarInfo} object for member @emph{name}. If @emph{name} can not be found
in the archive, @ref{5bb,,KeyError} is raised.

@cartouche
@quotation Note 
If a member occurs more than once in the archive, its last occurrence is assumed
to be the most up-to-date version.
@end quotation
@end cartouche
@end deffn

@geindex getmembers() (tarfile.TarFile method)
@anchor{library/tarfile tarfile TarFile getmembers}@anchor{9e9}
@deffn {Method} TarFile.getmembers ()

Return the members of the archive as a list of @ref{db2,,TarInfo} objects. The
list has the same order as the members in the archive.
@end deffn

@geindex getnames() (tarfile.TarFile method)
@anchor{library/tarfile tarfile TarFile getnames}@anchor{227f}
@deffn {Method} TarFile.getnames ()

Return the members as a list of their names. It has the same order as the list
returned by @ref{9e9,,getmembers()}.
@end deffn

@geindex list() (tarfile.TarFile method)
@anchor{library/tarfile tarfile TarFile list}@anchor{9e8}
@deffn {Method} TarFile.list (verbose=True, *, members=None)

Print a table of contents to @code{sys.stdout}. If @emph{verbose} is @ref{659,,False},
only the names of the members are printed. If it is @ref{748,,True}, output
similar to that of @code{ls -l} is produced. If optional @emph{members} is
given, it must be a subset of the list returned by @ref{9e9,,getmembers()}.


Changed in version 3.5: Added the @emph{members} parameter.

@end deffn

@geindex next() (tarfile.TarFile method)
@anchor{library/tarfile tarfile TarFile next}@anchor{2280}
@deffn {Method} TarFile.next ()

Return the next member of the archive as a @ref{db2,,TarInfo} object, when
@ref{db1,,TarFile} is opened for reading. Return @ref{182,,None} if there is no more
available.
@end deffn

@geindex extractall() (tarfile.TarFile method)
@anchor{library/tarfile tarfile TarFile extractall}@anchor{9e6}
@deffn {Method} TarFile.extractall (path='.', members=None, *, numeric_owner=False)

Extract all members from the archive to the current working directory or
directory @emph{path}. If optional @emph{members} is given, it must be a subset of the
list returned by @ref{9e9,,getmembers()}. Directory information like owner,
modification time and permissions are set after all members have been extracted.
This is done to work around two problems: A directory’s modification time is
reset each time a file is created in it. And, if a directory’s permissions do
not allow writing, extracting files to it will fail.

If @emph{numeric_owner} is @ref{748,,True}, the uid and gid numbers from the tarfile
are used to set the owner/group for the extracted files. Otherwise, the named
values from the tarfile are used.

@cartouche
@quotation Warning 
Never extract archives from untrusted sources without prior inspection.
It is possible that files are created outside of @emph{path}, e.g. members
that have absolute filenames starting with @code{"/"} or filenames with two
dots @code{".."}.
@end quotation
@end cartouche


Changed in version 3.5: Added the @emph{numeric_owner} parameter.



Changed in version 3.6: The @emph{path} parameter accepts a @ref{63d,,path-like object}.

@end deffn

@geindex extract() (tarfile.TarFile method)
@anchor{library/tarfile tarfile TarFile extract}@anchor{9e7}
@deffn {Method} TarFile.extract (member, path='@w{'}, set_attrs=True, *, numeric_owner=False)

Extract a member from the archive to the current working directory, using its
full name. Its file information is extracted as accurately as possible. @emph{member}
may be a filename or a @ref{db2,,TarInfo} object. You can specify a different
directory using @emph{path}. @emph{path} may be a @ref{63d,,path-like object}.
File attributes (owner, mtime, mode) are set unless @emph{set_attrs} is false.

If @emph{numeric_owner} is @ref{748,,True}, the uid and gid numbers from the tarfile
are used to set the owner/group for the extracted files. Otherwise, the named
values from the tarfile are used.

@cartouche
@quotation Note 
The @ref{9e7,,extract()} method does not take care of several extraction issues.
In most cases you should consider using the @ref{9e6,,extractall()} method.
@end quotation
@end cartouche

@cartouche
@quotation Warning 
See the warning for @ref{9e6,,extractall()}.
@end quotation
@end cartouche


Changed in version 3.2: Added the @emph{set_attrs} parameter.



Changed in version 3.5: Added the @emph{numeric_owner} parameter.



Changed in version 3.6: The @emph{path} parameter accepts a @ref{63d,,path-like object}.

@end deffn

@geindex extractfile() (tarfile.TarFile method)
@anchor{library/tarfile tarfile TarFile extractfile}@anchor{2281}
@deffn {Method} TarFile.extractfile (member)

Extract a member from the archive as a file object. @emph{member} may be
a filename or a @ref{db2,,TarInfo} object. If @emph{member} is a regular file or
a link, an @ref{daf,,io.BufferedReader} object is returned. For all other
existing members, @ref{182,,None} is returned. If @emph{member} does not appear
in the archive, @ref{5bb,,KeyError} is raised.


Changed in version 3.3: Return an @ref{daf,,io.BufferedReader} object.

@end deffn

@geindex add() (tarfile.TarFile method)
@anchor{library/tarfile tarfile TarFile add}@anchor{72f}
@deffn {Method} TarFile.add (name, arcname=None, recursive=True, *, filter=None)

Add the file @emph{name} to the archive. @emph{name} may be any type of file
(directory, fifo, symbolic link, etc.). If given, @emph{arcname} specifies an
alternative name for the file in the archive. Directories are added
recursively by default. This can be avoided by setting @emph{recursive} to
@ref{659,,False}. Recursion adds entries in sorted order.
If @emph{filter} is given, it
should be a function that takes a @ref{db2,,TarInfo} object argument and
returns the changed @ref{db2,,TarInfo} object. If it instead returns
@ref{182,,None} the @ref{db2,,TarInfo} object will be excluded from the
archive. See @ref{2270,,Examples} for an example.


Changed in version 3.2: Added the @emph{filter} parameter.



Changed in version 3.7: Recursion adds entries in sorted order.

@end deffn

@geindex addfile() (tarfile.TarFile method)
@anchor{library/tarfile tarfile TarFile addfile}@anchor{2282}
@deffn {Method} TarFile.addfile (tarinfo, fileobj=None)

Add the @ref{db2,,TarInfo} object @emph{tarinfo} to the archive. If @emph{fileobj} is given,
it should be a @ref{2138,,binary file}, and
@code{tarinfo.size} bytes are read from it and added to the archive.  You can
create @ref{db2,,TarInfo} objects directly, or by using @ref{2283,,gettarinfo()}.
@end deffn

@geindex gettarinfo() (tarfile.TarFile method)
@anchor{library/tarfile tarfile TarFile gettarinfo}@anchor{2283}
@deffn {Method} TarFile.gettarinfo (name=None, arcname=None, fileobj=None)

Create a @ref{db2,,TarInfo} object from the result of @ref{516,,os.stat()} or
equivalent on an existing file.  The file is either named by @emph{name}, or
specified as a @ref{d68,,file object} @emph{fileobj} with a file descriptor.
@emph{name} may be a @ref{63d,,path-like object}.  If
given, @emph{arcname} specifies an alternative name for the file in the
archive, otherwise, the name is taken from @emph{fileobj}’s
@ref{2284,,name} attribute, or the @emph{name} argument.  The name
should be a text string.

You can modify
some of the @ref{db2,,TarInfo}’s attributes before you add it using @ref{2282,,addfile()}.
If the file object is not an ordinary file object positioned at the
beginning of the file, attributes such as @ref{2285,,size} may need
modifying.  This is the case for objects such as @ref{43f,,GzipFile}.
The @ref{2286,,name} may also be modified, in which case @emph{arcname}
could be a dummy string.


Changed in version 3.6: The @emph{name} parameter accepts a @ref{63d,,path-like object}.

@end deffn

@geindex close() (tarfile.TarFile method)
@anchor{library/tarfile tarfile TarFile close}@anchor{2287}
@deffn {Method} TarFile.close ()

Close the @ref{db1,,TarFile}. In write mode, two finishing zero blocks are
appended to the archive.
@end deffn

@geindex pax_headers (tarfile.TarFile attribute)
@anchor{library/tarfile tarfile TarFile pax_headers}@anchor{2288}
@deffn {Attribute} TarFile.pax_headers

A dictionary containing key-value pairs of pax global headers.
@end deffn

@node TarInfo Objects,Command-Line Interface<2>,TarFile Objects,tarfile — Read and write tar archive files
@anchor{library/tarfile id2}@anchor{2289}@anchor{library/tarfile tarinfo-objects}@anchor{227d}
@subsubsection TarInfo Objects


A @ref{db2,,TarInfo} object represents one member in a @ref{db1,,TarFile}. Aside
from storing all required attributes of a file (like file type, size, time,
permissions, owner etc.), it provides some useful methods to determine its type.
It does @emph{not} contain the file’s data itself.

@ref{db2,,TarInfo} objects are returned by @ref{db1,,TarFile}’s methods
@code{getmember()}, @code{getmembers()} and @code{gettarinfo()}.

@geindex TarInfo (class in tarfile)
@anchor{library/tarfile tarfile TarInfo}@anchor{db2}
@deffn {Class} tarfile.TarInfo (name='@w{'})

Create a @ref{db2,,TarInfo} object.
@end deffn

@geindex frombuf() (tarfile.TarInfo class method)
@anchor{library/tarfile tarfile TarInfo frombuf}@anchor{2275}
@deffn {Method} classmethod TarInfo.frombuf (buf, encoding, errors)

Create and return a @ref{db2,,TarInfo} object from string buffer @emph{buf}.

Raises @ref{2274,,HeaderError} if the buffer is invalid.
@end deffn

@geindex fromtarfile() (tarfile.TarInfo class method)
@anchor{library/tarfile tarfile TarInfo fromtarfile}@anchor{228a}
@deffn {Method} classmethod TarInfo.fromtarfile (tarfile)

Read the next member from the @ref{db1,,TarFile} object @emph{tarfile} and return it as
a @ref{db2,,TarInfo} object.
@end deffn

@geindex tobuf() (tarfile.TarInfo method)
@anchor{library/tarfile tarfile TarInfo tobuf}@anchor{228b}
@deffn {Method} TarInfo.tobuf (format=DEFAULT_FORMAT, encoding=ENCODING, errors='surrogateescape')

Create a string buffer from a @ref{db2,,TarInfo} object. For information on the
arguments see the constructor of the @ref{db1,,TarFile} class.


Changed in version 3.2: Use @code{'surrogateescape'} as the default for the @emph{errors} argument.

@end deffn

A @code{TarInfo} object has the following public data attributes:

@geindex name (tarfile.TarInfo attribute)
@anchor{library/tarfile tarfile TarInfo name}@anchor{2286}
@deffn {Attribute} TarInfo.name

Name of the archive member.
@end deffn

@geindex size (tarfile.TarInfo attribute)
@anchor{library/tarfile tarfile TarInfo size}@anchor{2285}
@deffn {Attribute} TarInfo.size

Size in bytes.
@end deffn

@geindex mtime (tarfile.TarInfo attribute)
@anchor{library/tarfile tarfile TarInfo mtime}@anchor{228c}
@deffn {Attribute} TarInfo.mtime

Time of last modification.
@end deffn

@geindex mode (tarfile.TarInfo attribute)
@anchor{library/tarfile tarfile TarInfo mode}@anchor{228d}
@deffn {Attribute} TarInfo.mode

Permission bits.
@end deffn

@geindex type (tarfile.TarInfo attribute)
@anchor{library/tarfile tarfile TarInfo type}@anchor{228e}
@deffn {Attribute} TarInfo.type

File type.  @emph{type} is usually one of these constants: @code{REGTYPE},
@code{AREGTYPE}, @code{LNKTYPE}, @code{SYMTYPE}, @code{DIRTYPE},
@code{FIFOTYPE}, @code{CONTTYPE}, @code{CHRTYPE}, @code{BLKTYPE},
@code{GNUTYPE_SPARSE}.  To determine the type of a @ref{db2,,TarInfo} object
more conveniently, use the @code{is*()} methods below.
@end deffn

@geindex linkname (tarfile.TarInfo attribute)
@anchor{library/tarfile tarfile TarInfo linkname}@anchor{228f}
@deffn {Attribute} TarInfo.linkname

Name of the target file name, which is only present in @ref{db2,,TarInfo} objects
of type @code{LNKTYPE} and @code{SYMTYPE}.
@end deffn

@geindex uid (tarfile.TarInfo attribute)
@anchor{library/tarfile tarfile TarInfo uid}@anchor{2290}
@deffn {Attribute} TarInfo.uid

User ID of the user who originally stored this member.
@end deffn

@geindex gid (tarfile.TarInfo attribute)
@anchor{library/tarfile tarfile TarInfo gid}@anchor{2291}
@deffn {Attribute} TarInfo.gid

Group ID of the user who originally stored this member.
@end deffn

@geindex uname (tarfile.TarInfo attribute)
@anchor{library/tarfile tarfile TarInfo uname}@anchor{2292}
@deffn {Attribute} TarInfo.uname

User name.
@end deffn

@geindex gname (tarfile.TarInfo attribute)
@anchor{library/tarfile tarfile TarInfo gname}@anchor{2293}
@deffn {Attribute} TarInfo.gname

Group name.
@end deffn

@geindex pax_headers (tarfile.TarInfo attribute)
@anchor{library/tarfile tarfile TarInfo pax_headers}@anchor{2294}
@deffn {Attribute} TarInfo.pax_headers

A dictionary containing key-value pairs of an associated pax extended header.
@end deffn

A @ref{db2,,TarInfo} object also provides some convenient query methods:

@geindex isfile() (tarfile.TarInfo method)
@anchor{library/tarfile tarfile TarInfo isfile}@anchor{2295}
@deffn {Method} TarInfo.isfile ()

Return @ref{748,,True} if the @code{Tarinfo} object is a regular file.
@end deffn

@geindex isreg() (tarfile.TarInfo method)
@anchor{library/tarfile tarfile TarInfo isreg}@anchor{2296}
@deffn {Method} TarInfo.isreg ()

Same as @ref{2295,,isfile()}.
@end deffn

@geindex isdir() (tarfile.TarInfo method)
@anchor{library/tarfile tarfile TarInfo isdir}@anchor{2297}
@deffn {Method} TarInfo.isdir ()

Return @ref{748,,True} if it is a directory.
@end deffn

@geindex issym() (tarfile.TarInfo method)
@anchor{library/tarfile tarfile TarInfo issym}@anchor{2298}
@deffn {Method} TarInfo.issym ()

Return @ref{748,,True} if it is a symbolic link.
@end deffn

@geindex islnk() (tarfile.TarInfo method)
@anchor{library/tarfile tarfile TarInfo islnk}@anchor{2299}
@deffn {Method} TarInfo.islnk ()

Return @ref{748,,True} if it is a hard link.
@end deffn

@geindex ischr() (tarfile.TarInfo method)
@anchor{library/tarfile tarfile TarInfo ischr}@anchor{229a}
@deffn {Method} TarInfo.ischr ()

Return @ref{748,,True} if it is a character device.
@end deffn

@geindex isblk() (tarfile.TarInfo method)
@anchor{library/tarfile tarfile TarInfo isblk}@anchor{229b}
@deffn {Method} TarInfo.isblk ()

Return @ref{748,,True} if it is a block device.
@end deffn

@geindex isfifo() (tarfile.TarInfo method)
@anchor{library/tarfile tarfile TarInfo isfifo}@anchor{229c}
@deffn {Method} TarInfo.isfifo ()

Return @ref{748,,True} if it is a FIFO.
@end deffn

@geindex isdev() (tarfile.TarInfo method)
@anchor{library/tarfile tarfile TarInfo isdev}@anchor{229d}
@deffn {Method} TarInfo.isdev ()

Return @ref{748,,True} if it is one of character device, block device or FIFO.
@end deffn

@node Command-Line Interface<2>,Examples<8>,TarInfo Objects,tarfile — Read and write tar archive files
@anchor{library/tarfile command-line-interface}@anchor{229e}@anchor{library/tarfile tarfile-commandline}@anchor{b50}
@subsubsection Command-Line Interface



New in version 3.4.


The @ref{fb,,tarfile} module provides a simple command-line interface to interact
with tar archives.

If you want to create a new tar archive, specify its name after the @ref{229f,,-c}
option and then list the filename(s) that should be included:

@example
$ python -m tarfile -c monty.tar  spam.txt eggs.txt
@end example

Passing a directory is also acceptable:

@example
$ python -m tarfile -c monty.tar life-of-brian_1979/
@end example

If you want to extract a tar archive into the current directory, use
the @ref{22a0,,-e} option:

@example
$ python -m tarfile -e monty.tar
@end example

You can also extract a tar archive into a different directory by passing the
directory’s name:

@example
$ python -m tarfile -e monty.tar  other-dir/
@end example

For a list of the files in a tar archive, use the @ref{22a1,,-l} option:

@example
$ python -m tarfile -l monty.tar
@end example

@menu
* Command-line options: Command-line options<2>. 

@end menu

@node Command-line options<2>,,,Command-Line Interface<2>
@anchor{library/tarfile command-line-options}@anchor{22a2}
@subsubsection Command-line options


@geindex tarfile command line option; -l <tarfile>
@geindex tarfile command line option; --list <tarfile>
@anchor{library/tarfile cmdoption-tarfile-l}@anchor{22a1}
@deffn {Program Option} @w{-}l <tarfile>
@anchor{library/tarfile cmdoption-tarfile-list}@anchor{22a3}
@deffnx {Program Option} @w{-}@w{-}list <tarfile>

List files in a tarfile.
@end deffn

@geindex tarfile command line option; -c <tarfile> <source1> ... <sourceN>
@geindex tarfile command line option; --create <tarfile> <source1> ... <sourceN>
@anchor{library/tarfile cmdoption-tarfile-c}@anchor{229f}
@deffn {Program Option} @w{-}c <tarfile> <source1> ... <sourceN>
@anchor{library/tarfile cmdoption-tarfile-create}@anchor{22a4}
@deffnx {Program Option} @w{-}@w{-}create <tarfile> <source1> ... <sourceN>

Create tarfile from source files.
@end deffn

@geindex tarfile command line option; -e <tarfile> [<output_dir>]
@geindex tarfile command line option; --extract <tarfile> [<output_dir>]
@anchor{library/tarfile cmdoption-tarfile-e}@anchor{22a0}
@deffn {Program Option} @w{-}e <tarfile> [<output_dir>]
@anchor{library/tarfile cmdoption-tarfile-extract}@anchor{22a5}
@deffnx {Program Option} @w{-}@w{-}extract <tarfile> [<output_dir>]

Extract tarfile into the current directory if @emph{output_dir} is not specified.
@end deffn

@geindex tarfile command line option; -t <tarfile>
@geindex tarfile command line option; --test <tarfile>
@anchor{library/tarfile cmdoption-tarfile-t}@anchor{22a6}
@deffn {Program Option} @w{-}t <tarfile>
@anchor{library/tarfile cmdoption-tarfile-test}@anchor{22a7}
@deffnx {Program Option} @w{-}@w{-}test <tarfile>

Test whether the tarfile is valid or not.
@end deffn

@geindex tarfile command line option; -v
@geindex tarfile command line option; --verbose
@anchor{library/tarfile cmdoption-tarfile-v}@anchor{22a8}@anchor{library/tarfile cmdoption-tarfile-verbose}@anchor{22a9}
@deffn {Program Option} @w{-}v, @w{-}@w{-}verbose

Verbose output.
@end deffn

@node Examples<8>,Supported tar formats,Command-Line Interface<2>,tarfile — Read and write tar archive files
@anchor{library/tarfile examples}@anchor{22aa}@anchor{library/tarfile tar-examples}@anchor{2270}
@subsubsection Examples


How to extract an entire tar archive to the current working directory:

@example
import tarfile
tar = tarfile.open("sample.tar.gz")
tar.extractall()
tar.close()
@end example

How to extract a subset of a tar archive with @ref{9e6,,TarFile.extractall()} using
a generator function instead of a list:

@example
import os
import tarfile

def py_files(members):
    for tarinfo in members:
        if os.path.splitext(tarinfo.name)[1] == ".py":
            yield tarinfo

tar = tarfile.open("sample.tar.gz")
tar.extractall(members=py_files(tar))
tar.close()
@end example

How to create an uncompressed tar archive from a list of filenames:

@example
import tarfile
tar = tarfile.open("sample.tar", "w")
for name in ["foo", "bar", "quux"]:
    tar.add(name)
tar.close()
@end example

The same example using the @ref{168,,with} statement:

@example
import tarfile
with tarfile.open("sample.tar", "w") as tar:
    for name in ["foo", "bar", "quux"]:
        tar.add(name)
@end example

How to read a gzip compressed tar archive and display some member information:

@example
import tarfile
tar = tarfile.open("sample.tar.gz", "r:gz")
for tarinfo in tar:
    print(tarinfo.name, "is", tarinfo.size, "bytes in size and is ", end="")
    if tarinfo.isreg():
        print("a regular file.")
    elif tarinfo.isdir():
        print("a directory.")
    else:
        print("something else.")
tar.close()
@end example

How to create an archive and reset the user information using the @emph{filter}
parameter in @ref{72f,,TarFile.add()}:

@example
import tarfile
def reset(tarinfo):
    tarinfo.uid = tarinfo.gid = 0
    tarinfo.uname = tarinfo.gname = "root"
    return tarinfo
tar = tarfile.open("sample.tar.gz", "w:gz")
tar.add("foo", filter=reset)
tar.close()
@end example

@node Supported tar formats,Unicode issues,Examples<8>,tarfile — Read and write tar archive files
@anchor{library/tarfile supported-tar-formats}@anchor{22ab}@anchor{library/tarfile tar-formats}@anchor{2277}
@subsubsection Supported tar formats


There are three tar formats that can be created with the @ref{fb,,tarfile} module:


@itemize *

@item 
The POSIX.1-1988 ustar format (@ref{2278,,USTAR_FORMAT}). It supports filenames
up to a length of at best 256 characters and linknames up to 100 characters.
The maximum file size is 8 GiB. This is an old and limited but widely
supported format.

@item 
The GNU tar format (@ref{2279,,GNU_FORMAT}). It supports long filenames and
linknames, files bigger than 8 GiB and sparse files. It is the de facto
standard on GNU/Linux systems. @ref{fb,,tarfile} fully supports the GNU tar
extensions for long names, sparse file support is read-only.

@item 
The POSIX.1-2001 pax format (@ref{227a,,PAX_FORMAT}). It is the most flexible
format with virtually no limits. It supports long filenames and linknames, large
files and stores pathnames in a portable way. Modern tar implementations,
including GNU tar, bsdtar/libarchive and star, fully support extended @emph{pax}
features; some old or unmaintained libraries may not, but should treat
@emph{pax} archives as if they were in the universally-supported @emph{ustar} format.
It is the current default format for new archives.

It extends the existing @emph{ustar} format with extra headers for information
that cannot be stored otherwise. There are two flavours of pax headers:
Extended headers only affect the subsequent file header, global
headers are valid for the complete archive and affect all following files.
All the data in a pax header is encoded in @emph{UTF-8} for portability reasons.
@end itemize

There are some more variants of the tar format which can be read, but not
created:


@itemize *

@item 
The ancient V7 format. This is the first tar format from Unix Seventh Edition,
storing only regular files and directories. Names must not be longer than 100
characters, there is no user/group name information. Some archives have
miscalculated header checksums in case of fields with non-ASCII characters.

@item 
The SunOS tar extended format. This format is a variant of the POSIX.1-2001
pax format, but is not compatible.
@end itemize

@node Unicode issues,,Supported tar formats,tarfile — Read and write tar archive files
@anchor{library/tarfile tar-unicode}@anchor{227e}@anchor{library/tarfile unicode-issues}@anchor{22ac}
@subsubsection Unicode issues


The tar format was originally conceived to make backups on tape drives with the
main focus on preserving file system information. Nowadays tar archives are
commonly used for file distribution and exchanging archives over networks. One
problem of the original format (which is the basis of all other formats) is
that there is no concept of supporting different character encodings. For
example, an ordinary tar archive created on a @emph{UTF-8} system cannot be read
correctly on a @emph{Latin-1} system if it contains non-@emph{ASCII} characters. Textual
metadata (like filenames, linknames, user/group names) will appear damaged.
Unfortunately, there is no way to autodetect the encoding of an archive. The
pax format was designed to solve this problem. It stores non-ASCII metadata
using the universal character encoding @emph{UTF-8}.

The details of character conversion in @ref{fb,,tarfile} are controlled by the
@emph{encoding} and @emph{errors} keyword arguments of the @ref{db1,,TarFile} class.

@emph{encoding} defines the character encoding to use for the metadata in the
archive. The default value is @ref{79e,,sys.getfilesystemencoding()} or @code{'ascii'}
as a fallback. Depending on whether the archive is read or written, the
metadata must be either decoded or encoded. If @emph{encoding} is not set
appropriately, this conversion may fail.

The @emph{errors} argument defines how characters are treated that cannot be
converted. Possible values are listed in section @ref{162a,,Error Handlers}.
The default scheme is @code{'surrogateescape'} which Python also uses for its
file system calls, see @ref{22ad,,File Names@comma{} Command Line Arguments@comma{} and Environment Variables}.

For @ref{227a,,PAX_FORMAT} archives (the default), @emph{encoding} is generally not needed
because all the metadata is stored using @emph{UTF-8}. @emph{encoding} is only used in
the rare cases when binary pax headers are decoded or when strings with
surrogate characters are stored.

@node File Formats,Cryptographic Services,Data Compression and Archiving,The Python Standard Library
@anchor{library/fileformats doc}@anchor{22ae}@anchor{library/fileformats file-formats}@anchor{22af}@anchor{library/fileformats fileformats}@anchor{22b0}
@section File Formats


The modules described in this chapter parse various miscellaneous file formats
that aren’t markup languages and are not related to e-mail.

@menu
* csv — CSV File Reading and Writing:: 
* configparser — Configuration file parser:: 
* netrc — netrc file processing:: 
* xdrlib — Encode and decode XDR data:: 
* plistlib — Generate and parse Apple .plist files: plistlib — Generate and parse Apple plist files. 

@end menu

@node csv — CSV File Reading and Writing,configparser — Configuration file parser,,File Formats
@anchor{library/csv doc}@anchor{22b1}@anchor{library/csv csv-csv-file-reading-and-writing}@anchor{22b2}@anchor{library/csv module-csv}@anchor{28}
@subsection @code{csv} — CSV File Reading and Writing


@geindex module; csv

@strong{Source code:} Lib/csv.py@footnote{https://github.com/python/cpython/tree/main/Lib/csv.py}

@geindex csv
@geindex data; tabular


__________________________________________________________________


The so-called CSV (Comma Separated Values) format is the most common import and
export format for spreadsheets and databases.  CSV format was used for many
years prior to attempts to describe the format in a standardized way in
@geindex RFC; RFC 4180
RFC 4180@footnote{https://tools.ietf.org/html/rfc4180.html}.  The lack of a well-defined standard means that subtle differences
often exist in the data produced and consumed by different applications.  These
differences can make it annoying to process CSV files from multiple sources.
Still, while the delimiters and quoting characters vary, the overall format is
similar enough that it is possible to write a single module which can
efficiently manipulate such data, hiding the details of reading and writing the
data from the programmer.

The @ref{28,,csv} module implements classes to read and write tabular data in CSV
format.  It allows programmers to say, “write this data in the format preferred
by Excel,” or “read data from this file which was generated by Excel,” without
knowing the precise details of the CSV format used by Excel.  Programmers can
also describe the CSV formats understood by other applications or define their
own special-purpose CSV formats.

The @ref{28,,csv} module’s @ref{11c7,,reader} and @ref{ff4,,writer} objects read and
write sequences.  Programmers can also read and write data in dictionary form
using the @ref{4e9,,DictReader} and @ref{d9f,,DictWriter} classes.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 305

@item PEP 305@footnote{https://www.python.org/dev/peps/pep-0305} - CSV File API

The Python Enhancement Proposal which proposed this addition to Python.
@end table


@menu
* Module Contents: Module Contents<3>. 
* Dialects and Formatting Parameters:: 
* Reader Objects:: 
* Writer Objects:: 
* Examples: Examples<9>. 

@end menu

@node Module Contents<3>,Dialects and Formatting Parameters,,csv — CSV File Reading and Writing
@anchor{library/csv csv-contents}@anchor{22b3}@anchor{library/csv module-contents}@anchor{22b4}
@subsubsection Module Contents


The @ref{28,,csv} module defines the following functions:

@geindex universal newlines; csv.reader function

@geindex reader() (in module csv)
@anchor{library/csv csv reader}@anchor{11c7}
@deffn {Function} csv.reader (csvfile, dialect='excel', **fmtparams)

Return a reader object which will iterate over lines in the given @emph{csvfile}.
@emph{csvfile} can be any object which supports the @ref{11c3,,iterator} protocol and returns a
string each time its @code{__next__()} method is called — @ref{d68,,file objects} and list objects are both suitable.   If @emph{csvfile} is a file object,
it should be opened with @code{newline='@w{'}}. @footnote{@w{(1)} 
If @code{newline='@w{'}} is not specified, newlines embedded inside quoted fields
will not be interpreted correctly, and on platforms that use @code{\r\n} linendings
on write an extra @code{\r} will be added.  It should always be safe to specify
@code{newline='@w{'}}, since the csv module does its own
(@ref{88a,,universal}) newline handling.
}  An optional
@emph{dialect} parameter can be given which is used to define a set of parameters
specific to a particular CSV dialect.  It may be an instance of a subclass of
the @ref{10e2,,Dialect} class or one of the strings returned by the
@ref{22b5,,list_dialects()} function.  The other optional @emph{fmtparams} keyword arguments
can be given to override individual formatting parameters in the current
dialect.  For full details about the dialect and formatting parameters, see
section @ref{22b6,,Dialects and Formatting Parameters}.

Each row read from the csv file is returned as a list of strings.  No
automatic data type conversion is performed unless the @code{QUOTE_NONNUMERIC} format
option is specified (in which case unquoted fields are transformed into floats).

A short usage example:

@example
>>> import csv
>>> with open('eggs.csv', newline='@w{'}) as csvfile:
...     spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
...     for row in spamreader:
...         print(', '.join(row))
Spam, Spam, Spam, Spam, Spam, Baked Beans
Spam, Lovely Spam, Wonderful Spam
@end example
@end deffn

@geindex writer() (in module csv)
@anchor{library/csv csv writer}@anchor{ff4}
@deffn {Function} csv.writer (csvfile, dialect='excel', **fmtparams)

Return a writer object responsible for converting the user’s data into delimited
strings on the given file-like object.  @emph{csvfile} can be any object with a
@code{write()} method.  If @emph{csvfile} is a file object, it should be opened with
@code{newline='@w{'}} @footnote{@w{(1)} 
If @code{newline='@w{'}} is not specified, newlines embedded inside quoted fields
will not be interpreted correctly, and on platforms that use @code{\r\n} linendings
on write an extra @code{\r} will be added.  It should always be safe to specify
@code{newline='@w{'}}, since the csv module does its own
(@ref{88a,,universal}) newline handling.
}.  An optional @emph{dialect}
parameter can be given which is used to define a set of parameters specific to a
particular CSV dialect.  It may be an instance of a subclass of the
@ref{10e2,,Dialect} class or one of the strings returned by the
@ref{22b5,,list_dialects()} function.  The other optional @emph{fmtparams} keyword arguments
can be given to override individual formatting parameters in the current
dialect.  For full details about dialects and formatting parameters, see
the @ref{22b6,,Dialects and Formatting Parameters} section. To make it
as easy as possible to interface with modules which implement the DB API, the
value @ref{182,,None} is written as the empty string.  While this isn’t a
reversible transformation, it makes it easier to dump SQL NULL data values to
CSV files without preprocessing the data returned from a @code{cursor.fetch*} call.
All other non-string data are stringified with @ref{2ef,,str()} before being written.

A short usage example:

@example
import csv
with open('eggs.csv', 'w', newline='@w{'}) as csvfile:
    spamwriter = csv.writer(csvfile, delimiter=' ',
                            quotechar='|', quoting=csv.QUOTE_MINIMAL)
    spamwriter.writerow(['Spam'] * 5 + ['Baked Beans'])
    spamwriter.writerow(['Spam', 'Lovely Spam', 'Wonderful Spam'])
@end example
@end deffn

@geindex register_dialect() (in module csv)
@anchor{library/csv csv register_dialect}@anchor{22b7}
@deffn {Function} csv.register_dialect (name[, dialect[, **fmtparams]])

Associate @emph{dialect} with @emph{name}.  @emph{name} must be a string. The
dialect can be specified either by passing a sub-class of @ref{10e2,,Dialect}, or
by @emph{fmtparams} keyword arguments, or both, with keyword arguments overriding
parameters of the dialect. For full details about dialects and formatting
parameters, see section @ref{22b6,,Dialects and Formatting Parameters}.
@end deffn

@geindex unregister_dialect() (in module csv)
@anchor{library/csv csv unregister_dialect}@anchor{22b8}
@deffn {Function} csv.unregister_dialect (name)

Delete the dialect associated with @emph{name} from the dialect registry.  An
@ref{22b9,,Error} is raised if @emph{name} is not a registered dialect name.
@end deffn

@geindex get_dialect() (in module csv)
@anchor{library/csv csv get_dialect}@anchor{22ba}
@deffn {Function} csv.get_dialect (name)

Return the dialect associated with @emph{name}.  An @ref{22b9,,Error} is raised if
@emph{name} is not a registered dialect name.  This function returns an immutable
@ref{10e2,,Dialect}.
@end deffn

@geindex list_dialects() (in module csv)
@anchor{library/csv csv list_dialects}@anchor{22b5}
@deffn {Function} csv.list_dialects ()

Return the names of all registered dialects.
@end deffn

@geindex field_size_limit() (in module csv)
@anchor{library/csv csv field_size_limit}@anchor{22bb}
@deffn {Function} csv.field_size_limit ([new_limit])

Returns the current maximum field size allowed by the parser. If @emph{new_limit} is
given, this becomes the new limit.
@end deffn

The @ref{28,,csv} module defines the following classes:

@geindex DictReader (class in csv)
@anchor{library/csv csv DictReader}@anchor{4e9}
@deffn {Class} csv.DictReader (f, fieldnames=None, restkey=None, restval=None, dialect='excel', *args, **kwds)

Create an object that operates like a regular reader but maps the
information in each row to a @ref{3b0,,dict} whose keys are given by the
optional @emph{fieldnames} parameter.

The @emph{fieldnames} parameter is a @ref{14e7,,sequence}.  If @emph{fieldnames} is
omitted, the values in the first row of file @emph{f} will be used as the
fieldnames.  Regardless of how the fieldnames are determined, the
dictionary preserves their original ordering.

If a row has more fields than fieldnames, the remaining data is put in a
list and stored with the fieldname specified by @emph{restkey} (which defaults
to @code{None}).  If a non-blank row has fewer fields than fieldnames, the
missing values are filled-in with the value of @emph{restval} (which defaults
to @code{None}).

All other optional or keyword arguments are passed to the underlying
@ref{11c7,,reader} instance.


Changed in version 3.6: Returned rows are now of type @code{OrderedDict}.



Changed in version 3.8: Returned rows are now of type @ref{3b0,,dict}.


A short usage example:

@example
>>> import csv
>>> with open('names.csv', newline='@w{'}) as csvfile:
...     reader = csv.DictReader(csvfile)
...     for row in reader:
...         print(row['first_name'], row['last_name'])
...
Eric Idle
John Cleese

>>> print(row)
@{'first_name': 'John', 'last_name': 'Cleese'@}
@end example
@end deffn

@geindex DictWriter (class in csv)
@anchor{library/csv csv DictWriter}@anchor{d9f}
@deffn {Class} csv.DictWriter (f, fieldnames, restval='@w{'}, extrasaction='raise', dialect='excel', *args, **kwds)

Create an object which operates like a regular writer but maps dictionaries
onto output rows.  The @emph{fieldnames} parameter is a @ref{1d,,sequence} of keys that identify the order in which values in the
dictionary passed to the @code{writerow()} method are written to file
@emph{f}.  The optional @emph{restval} parameter specifies the value to be
written if the dictionary is missing a key in @emph{fieldnames}.  If the
dictionary passed to the @code{writerow()} method contains a key not found in
@emph{fieldnames}, the optional @emph{extrasaction} parameter indicates what action to
take.
If it is set to @code{'raise'}, the default value, a @ref{17c,,ValueError}
is raised.
If it is set to @code{'ignore'}, extra values in the dictionary are ignored.
Any other optional or keyword arguments are passed to the underlying
@ref{ff4,,writer} instance.

Note that unlike the @ref{4e9,,DictReader} class, the @emph{fieldnames} parameter
of the @ref{d9f,,DictWriter} class is not optional.

A short usage example:

@example
import csv

with open('names.csv', 'w', newline='@w{'}) as csvfile:
    fieldnames = ['first_name', 'last_name']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)

    writer.writeheader()
    writer.writerow(@{'first_name': 'Baked', 'last_name': 'Beans'@})
    writer.writerow(@{'first_name': 'Lovely', 'last_name': 'Spam'@})
    writer.writerow(@{'first_name': 'Wonderful', 'last_name': 'Spam'@})
@end example
@end deffn

@geindex Dialect (class in csv)
@anchor{library/csv csv Dialect}@anchor{10e2}
@deffn {Class} csv.Dialect

The @ref{10e2,,Dialect} class is a container class whose attributes contain
information for how to handle doublequotes, whitespace, delimiters, etc.
Due to the lack of a strict CSV specification, different applications
produce subtly different CSV data.  @ref{10e2,,Dialect} instances define how
@ref{11c7,,reader} and @ref{ff4,,writer} instances behave.

All available @ref{10e2,,Dialect} names are returned by @ref{22b5,,list_dialects()},
and they can be registered with specific @ref{11c7,,reader} and @ref{ff4,,writer}
classes through their initializer (@code{__init__}) functions like this:

@example
import csv

with open('students.csv', 'w', newline='@w{'}) as csvfile:
    writer = csv.writer(csvfile, dialect='unix')
                                 ^^^^^^^^^^^^^^
@end example
@end deffn

@geindex excel (class in csv)
@anchor{library/csv csv excel}@anchor{22bc}
@deffn {Class} csv.excel

The @ref{22bc,,excel} class defines the usual properties of an Excel-generated CSV
file.  It is registered with the dialect name @code{'excel'}.
@end deffn

@geindex excel_tab (class in csv)
@anchor{library/csv csv excel_tab}@anchor{22bd}
@deffn {Class} csv.excel_tab

The @ref{22bd,,excel_tab} class defines the usual properties of an Excel-generated
TAB-delimited file.  It is registered with the dialect name @code{'excel-tab'}.
@end deffn

@geindex unix_dialect (class in csv)
@anchor{library/csv csv unix_dialect}@anchor{d9e}
@deffn {Class} csv.unix_dialect

The @ref{d9e,,unix_dialect} class defines the usual properties of a CSV file
generated on UNIX systems, i.e. using @code{'\n'} as line terminator and quoting
all fields.  It is registered with the dialect name @code{'unix'}.


New in version 3.2.

@end deffn

@geindex Sniffer (class in csv)
@anchor{library/csv csv Sniffer}@anchor{22be}
@deffn {Class} csv.Sniffer

The @ref{22be,,Sniffer} class is used to deduce the format of a CSV file.

The @ref{22be,,Sniffer} class provides two methods:

@geindex sniff() (csv.Sniffer method)
@anchor{library/csv csv Sniffer sniff}@anchor{22bf}
@deffn {Method} sniff (sample, delimiters=None)

Analyze the given @emph{sample} and return a @ref{10e2,,Dialect} subclass
reflecting the parameters found.  If the optional @emph{delimiters} parameter
is given, it is interpreted as a string containing possible valid
delimiter characters.
@end deffn

@geindex has_header() (csv.Sniffer method)
@anchor{library/csv csv Sniffer has_header}@anchor{1109}
@deffn {Method} has_header (sample)

Analyze the sample text (presumed to be in CSV format) and return
@ref{748,,True} if the first row appears to be a series of column headers.
Inspecting each column, one of two key criteria will be considered to
estimate if the sample contains a header:

@quotation


@itemize -

@item 
the second through n-th rows contain numeric values

@item 
the second through n-th rows contain strings where at least one value’s
length differs from that of the putative header of that column.
@end itemize
@end quotation

Twenty rows after the first row are sampled; if more than half of columns +
rows meet the criteria, @ref{748,,True} is returned.
@end deffn

@cartouche
@quotation Note 
This method is a rough heuristic and may produce both false positives and
negatives.
@end quotation
@end cartouche
@end deffn

An example for @ref{22be,,Sniffer} use:

@example
with open('example.csv', newline='@w{'}) as csvfile:
    dialect = csv.Sniffer().sniff(csvfile.read(1024))
    csvfile.seek(0)
    reader = csv.reader(csvfile, dialect)
    # ... process CSV file contents here ...
@end example

The @ref{28,,csv} module defines the following constants:

@geindex QUOTE_ALL (in module csv)
@anchor{library/csv csv QUOTE_ALL}@anchor{22c0}
@deffn {Data} csv.QUOTE_ALL

Instructs @ref{ff4,,writer} objects to quote all fields.
@end deffn

@geindex QUOTE_MINIMAL (in module csv)
@anchor{library/csv csv QUOTE_MINIMAL}@anchor{22c1}
@deffn {Data} csv.QUOTE_MINIMAL

Instructs @ref{ff4,,writer} objects to only quote those fields which contain
special characters such as @emph{delimiter}, @emph{quotechar} or any of the characters in
@emph{lineterminator}.
@end deffn

@geindex QUOTE_NONNUMERIC (in module csv)
@anchor{library/csv csv QUOTE_NONNUMERIC}@anchor{22c2}
@deffn {Data} csv.QUOTE_NONNUMERIC

Instructs @ref{ff4,,writer} objects to quote all non-numeric fields.

Instructs the reader to convert all non-quoted fields to type @emph{float}.
@end deffn

@geindex QUOTE_NONE (in module csv)
@anchor{library/csv csv QUOTE_NONE}@anchor{22c3}
@deffn {Data} csv.QUOTE_NONE

Instructs @ref{ff4,,writer} objects to never quote fields.  When the current
@emph{delimiter} occurs in output data it is preceded by the current @emph{escapechar}
character.  If @emph{escapechar} is not set, the writer will raise @ref{22b9,,Error} if
any characters that require escaping are encountered.

Instructs @ref{11c7,,reader} to perform no special processing of quote characters.
@end deffn

The @ref{28,,csv} module defines the following exception:

@geindex Error
@anchor{library/csv csv Error}@anchor{22b9}
@deffn {Exception} csv.Error

Raised by any of the functions when an error is detected.
@end deffn

@node Dialects and Formatting Parameters,Reader Objects,Module Contents<3>,csv — CSV File Reading and Writing
@anchor{library/csv csv-fmt-params}@anchor{22b6}@anchor{library/csv dialects-and-formatting-parameters}@anchor{22c4}
@subsubsection Dialects and Formatting Parameters


To make it easier to specify the format of input and output records, specific
formatting parameters are grouped together into dialects.  A dialect is a
subclass of the @ref{10e2,,Dialect} class having a set of specific methods and a
single @code{validate()} method.  When creating @ref{11c7,,reader} or
@ref{ff4,,writer} objects, the programmer can specify a string or a subclass of
the @ref{10e2,,Dialect} class as the dialect parameter.  In addition to, or instead
of, the @emph{dialect} parameter, the programmer can also specify individual
formatting parameters, which have the same names as the attributes defined below
for the @ref{10e2,,Dialect} class.

Dialects support the following attributes:

@geindex delimiter (csv.Dialect attribute)
@anchor{library/csv csv Dialect delimiter}@anchor{22c5}
@deffn {Attribute} Dialect.delimiter

A one-character string used to separate fields.  It defaults to @code{','}.
@end deffn

@geindex doublequote (csv.Dialect attribute)
@anchor{library/csv csv Dialect doublequote}@anchor{22c6}
@deffn {Attribute} Dialect.doublequote

Controls how instances of @emph{quotechar} appearing inside a field should
themselves be quoted.  When @ref{748,,True}, the character is doubled. When
@ref{659,,False}, the @emph{escapechar} is used as a prefix to the @emph{quotechar}.  It
defaults to @ref{748,,True}.

On output, if @emph{doublequote} is @ref{659,,False} and no @emph{escapechar} is set,
@ref{22b9,,Error} is raised if a @emph{quotechar} is found in a field.
@end deffn

@geindex escapechar (csv.Dialect attribute)
@anchor{library/csv csv Dialect escapechar}@anchor{22c7}
@deffn {Attribute} Dialect.escapechar

A one-character string used by the writer to escape the @emph{delimiter} if @emph{quoting}
is set to @ref{22c3,,QUOTE_NONE} and the @emph{quotechar} if @emph{doublequote} is
@ref{659,,False}. On reading, the @emph{escapechar} removes any special meaning from
the following character. It defaults to @ref{182,,None}, which disables escaping.


Changed in version 3.11: An empty @emph{escapechar} is not allowed.

@end deffn

@geindex lineterminator (csv.Dialect attribute)
@anchor{library/csv csv Dialect lineterminator}@anchor{22c8}
@deffn {Attribute} Dialect.lineterminator

The string used to terminate lines produced by the @ref{ff4,,writer}. It defaults
to @code{'\r\n'}.

@cartouche
@quotation Note 
The @ref{11c7,,reader} is hard-coded to recognise either @code{'\r'} or @code{'\n'} as
end-of-line, and ignores @emph{lineterminator}. This behavior may change in the
future.
@end quotation
@end cartouche
@end deffn

@geindex quotechar (csv.Dialect attribute)
@anchor{library/csv csv Dialect quotechar}@anchor{22c9}
@deffn {Attribute} Dialect.quotechar

A one-character string used to quote fields containing special characters, such
as the @emph{delimiter} or @emph{quotechar}, or which contain new-line characters.  It
defaults to @code{'"'}.


Changed in version 3.11: An empty @emph{quotechar} is not allowed.

@end deffn

@geindex quoting (csv.Dialect attribute)
@anchor{library/csv csv Dialect quoting}@anchor{22ca}
@deffn {Attribute} Dialect.quoting

Controls when quotes should be generated by the writer and recognised by the
reader.  It can take on any of the @code{QUOTE_*} constants (see section
@ref{22b3,,Module Contents}) and defaults to @ref{22c1,,QUOTE_MINIMAL}.
@end deffn

@geindex skipinitialspace (csv.Dialect attribute)
@anchor{library/csv csv Dialect skipinitialspace}@anchor{22cb}
@deffn {Attribute} Dialect.skipinitialspace

When @ref{748,,True}, whitespace immediately following the @emph{delimiter} is ignored.
The default is @ref{659,,False}.
@end deffn

@geindex strict (csv.Dialect attribute)
@anchor{library/csv csv Dialect strict}@anchor{22cc}
@deffn {Attribute} Dialect.strict

When @code{True}, raise exception @ref{22b9,,Error} on bad CSV input.
The default is @code{False}.
@end deffn

@node Reader Objects,Writer Objects,Dialects and Formatting Parameters,csv — CSV File Reading and Writing
@anchor{library/csv reader-objects}@anchor{22cd}
@subsubsection Reader Objects


Reader objects (@ref{4e9,,DictReader} instances and objects returned by the
@ref{11c7,,reader()} function) have the following public methods:

@geindex __next__() (csv.csvreader method)
@anchor{library/csv csv csvreader __next__}@anchor{22ce}
@deffn {Method} csvreader.__next__ ()

Return the next row of the reader’s iterable object as a list (if the object
was returned from @ref{11c7,,reader()}) or a dict (if it is a @ref{4e9,,DictReader}
instance), parsed according to the current @ref{10e2,,Dialect}.  Usually you
should call this as @code{next(reader)}.
@end deffn

Reader objects have the following public attributes:

@geindex dialect (csv.csvreader attribute)
@anchor{library/csv csv csvreader dialect}@anchor{22cf}
@deffn {Attribute} csvreader.dialect

A read-only description of the dialect in use by the parser.
@end deffn

@geindex line_num (csv.csvreader attribute)
@anchor{library/csv csv csvreader line_num}@anchor{22d0}
@deffn {Attribute} csvreader.line_num

The number of lines read from the source iterator. This is not the same as the
number of records returned, as records can span multiple lines.
@end deffn

DictReader objects have the following public attribute:

@geindex fieldnames (csv.csvreader attribute)
@anchor{library/csv csv csvreader fieldnames}@anchor{22d1}
@deffn {Attribute} csvreader.fieldnames

If not passed as a parameter when creating the object, this attribute is
initialized upon first access or when the first record is read from the
file.
@end deffn

@node Writer Objects,Examples<9>,Reader Objects,csv — CSV File Reading and Writing
@anchor{library/csv writer-objects}@anchor{22d2}
@subsubsection Writer Objects


@code{Writer} objects (@ref{d9f,,DictWriter} instances and objects returned by
the @ref{ff4,,writer()} function) have the following public methods.  A @emph{row} must be
an iterable of strings or numbers for @code{Writer} objects and a dictionary
mapping fieldnames to strings or numbers (by passing them through @ref{2ef,,str()}
first) for @ref{d9f,,DictWriter} objects.  Note that complex numbers are written
out surrounded by parens. This may cause some problems for other programs which
read CSV files (assuming they support complex numbers at all).

@geindex writerow() (csv.csvwriter method)
@anchor{library/csv csv csvwriter writerow}@anchor{949}
@deffn {Method} csvwriter.writerow (row)

Write the @emph{row} parameter to the writer’s file object, formatted according
to the current @ref{10e2,,Dialect}. Return the return value of the call to the
@emph{write} method of the underlying file object.


Changed in version 3.5: Added support of arbitrary iterables.

@end deffn

@geindex writerows() (csv.csvwriter method)
@anchor{library/csv csv csvwriter writerows}@anchor{22d3}
@deffn {Method} csvwriter.writerows (rows)

Write all elements in @emph{rows} (an iterable of @emph{row} objects as described
above) to the writer’s file object, formatted according to the current
dialect.
@end deffn

Writer objects have the following public attribute:

@geindex dialect (csv.csvwriter attribute)
@anchor{library/csv csv csvwriter dialect}@anchor{22d4}
@deffn {Attribute} csvwriter.dialect

A read-only description of the dialect in use by the writer.
@end deffn

DictWriter objects have the following public method:

@geindex writeheader() (csv.DictWriter method)
@anchor{library/csv csv DictWriter writeheader}@anchor{da0}
@deffn {Method} DictWriter.writeheader ()

Write a row with the field names (as specified in the constructor) to
the writer’s file object, formatted according to the current dialect. Return
the return value of the @ref{949,,csvwriter.writerow()} call used internally.


New in version 3.2.



Changed in version 3.8: @ref{da0,,writeheader()} now also returns the value returned by
the @ref{949,,csvwriter.writerow()} method it uses internally.

@end deffn

@node Examples<9>,,Writer Objects,csv — CSV File Reading and Writing
@anchor{library/csv csv-examples}@anchor{22d5}@anchor{library/csv examples}@anchor{22d6}
@subsubsection Examples


The simplest example of reading a CSV file:

@example
import csv
with open('some.csv', newline='@w{'}) as f:
    reader = csv.reader(f)
    for row in reader:
        print(row)
@end example

Reading a file with an alternate format:

@example
import csv
with open('passwd', newline='@w{'}) as f:
    reader = csv.reader(f, delimiter=':', quoting=csv.QUOTE_NONE)
    for row in reader:
        print(row)
@end example

The corresponding simplest possible writing example is:

@example
import csv
with open('some.csv', 'w', newline='@w{'}) as f:
    writer = csv.writer(f)
    writer.writerows(someiterable)
@end example

Since @ref{1c8,,open()} is used to open a CSV file for reading, the file
will by default be decoded into unicode using the system default
encoding (see @ref{1b5,,locale.getpreferredencoding()}).  To decode a file
using a different encoding, use the @code{encoding} argument of open:

@example
import csv
with open('some.csv', newline='@w{'}, encoding='utf-8') as f:
    reader = csv.reader(f)
    for row in reader:
        print(row)
@end example

The same applies to writing in something other than the system default
encoding: specify the encoding argument when opening the output file.

Registering a new dialect:

@example
import csv
csv.register_dialect('unixpwd', delimiter=':', quoting=csv.QUOTE_NONE)
with open('passwd', newline='@w{'}) as f:
    reader = csv.reader(f, 'unixpwd')
@end example

A slightly more advanced use of the reader — catching and reporting errors:

@example
import csv, sys
filename = 'some.csv'
with open(filename, newline='@w{'}) as f:
    reader = csv.reader(f)
    try:
        for row in reader:
            print(row)
    except csv.Error as e:
        sys.exit('file @{@}, line @{@}: @{@}'.format(filename, reader.line_num, e))
@end example

And while the module doesn’t directly support parsing strings, it can easily be
done:

@example
import csv
for row in csv.reader(['one,two,three']):
    print(row)
@end example

@node configparser — Configuration file parser,netrc — netrc file processing,csv — CSV File Reading and Writing,File Formats
@anchor{library/configparser doc}@anchor{22d7}@anchor{library/configparser configparser-configuration-file-parser}@anchor{22d8}@anchor{library/configparser module-configparser}@anchor{21}
@subsection @code{configparser} — Configuration file parser


@geindex module; configparser

@strong{Source code:} Lib/configparser.py@footnote{https://github.com/python/cpython/tree/main/Lib/configparser.py}

@geindex .ini; file
@geindex configuration; file
@geindex ini file
@geindex Windows ini file


__________________________________________________________________


This module provides the @ref{759,,ConfigParser} class which implements a basic
configuration language which provides a structure similar to what’s found in
Microsoft Windows INI files.  You can use this to write Python programs which
can be customized by end users easily.

@cartouche
@quotation Note 
This library does @emph{not} interpret or write the value-type prefixes used in
the Windows Registry extended version of INI syntax.
@end quotation
@end cartouche


@subsubheading See also



@table @asis

@item Module @ref{e3,,shlex}

Support for creating Unix shell-like mini-languages which can be used as
an alternate format for application configuration files.

@item Module @ref{a0,,json}

The json module implements a subset of JavaScript syntax which can also
be used for this purpose.
@end table


@c import configparser

@c import os
@c os.remove("example.ini")

@menu
* Quick Start:: 
* Supported Datatypes:: 
* Fallback Values:: 
* Supported INI File Structure:: 
* Interpolation of values:: 
* Mapping Protocol Access:: 
* Customizing Parser Behaviour:: 
* Legacy API Examples:: 
* ConfigParser Objects:: 
* RawConfigParser Objects:: 
* Exceptions: Exceptions<6>. 

@end menu

@node Quick Start,Supported Datatypes,,configparser — Configuration file parser
@anchor{library/configparser quick-start}@anchor{22d9}
@subsubsection Quick Start


Let’s take a very basic configuration file that looks like this:

@example
[DEFAULT]
ServerAliveInterval = 45
Compression = yes
CompressionLevel = 9
ForwardX11 = yes

[bitbucket.org]
User = hg

[topsecret.server.com]
Port = 50022
ForwardX11 = no
@end example

The structure of INI files is described @ref{22da,,in the following section}.  Essentially, the file
consists of sections, each of which contains keys with values.
@ref{21,,configparser} classes can read and write such files.  Let’s start by
creating the above configuration file programmatically.

@example
>>> import configparser
>>> config = configparser.ConfigParser()
>>> config['DEFAULT'] = @{'ServerAliveInterval': '45',
...                      'Compression': 'yes',
...                      'CompressionLevel': '9'@}
>>> config['bitbucket.org'] = @{@}
>>> config['bitbucket.org']['User'] = 'hg'
>>> config['topsecret.server.com'] = @{@}
>>> topsecret = config['topsecret.server.com']
>>> topsecret['Port'] = '50022'     # mutates the parser
>>> topsecret['ForwardX11'] = 'no'  # same here
>>> config['DEFAULT']['ForwardX11'] = 'yes'
>>> with open('example.ini', 'w') as configfile:
...   config.write(configfile)
...
@end example

As you can see, we can treat a config parser much like a dictionary.
There are differences, @ref{22db,,outlined later}, but
the behavior is very close to what you would expect from a dictionary.

Now that we have created and saved a configuration file, let’s read it
back and explore the data it holds.

@example
>>> config = configparser.ConfigParser()
>>> config.sections()
[]
>>> config.read('example.ini')
['example.ini']
>>> config.sections()
['bitbucket.org', 'topsecret.server.com']
>>> 'bitbucket.org' in config
True
>>> 'bytebong.com' in config
False
>>> config['bitbucket.org']['User']
'hg'
>>> config['DEFAULT']['Compression']
'yes'
>>> topsecret = config['topsecret.server.com']
>>> topsecret['ForwardX11']
'no'
>>> topsecret['Port']
'50022'
>>> for key in config['bitbucket.org']:  
...     print(key)
user
compressionlevel
serveraliveinterval
compression
forwardx11
>>> config['bitbucket.org']['ForwardX11']
'yes'
@end example

As we can see above, the API is pretty straightforward.  The only bit of magic
involves the @code{DEFAULT} section which provides default values for all other
sections @footnote{@w{(1)} 
Config parsers allow for heavy customization.  If you are interested in
changing the behaviour outlined by the footnote reference, consult the
@ref{22dc,,Customizing Parser Behaviour} section.
}.  Note also that keys in sections are
case-insensitive and stored in lowercase @footnote{@w{(1)} 
Config parsers allow for heavy customization.  If you are interested in
changing the behaviour outlined by the footnote reference, consult the
@ref{22dc,,Customizing Parser Behaviour} section.
}.

It is possible to read several configurations into a single
@ref{759,,ConfigParser}, where the most recently added configuration has the
highest priority. Any conflicting keys are taken from the more recent
configuration while the previously existing keys are retained.

@example
>>> another_config = configparser.ConfigParser()
>>> another_config.read('example.ini')
['example.ini']
>>> another_config['topsecret.server.com']['Port']
'50022'
>>> another_config.read_string("[topsecret.server.com]\nPort=48484")
>>> another_config['topsecret.server.com']['Port']
'48484'
>>> another_config.read_dict(@{"topsecret.server.com": @{"Port": 21212@}@})
>>> another_config['topsecret.server.com']['Port']
'21212'
>>> another_config['topsecret.server.com']['ForwardX11']
'no'
@end example

This behaviour is equivalent to a @ref{1366,,ConfigParser.read()} call with several
files passed to the @emph{filenames} parameter.

@node Supported Datatypes,Fallback Values,Quick Start,configparser — Configuration file parser
@anchor{library/configparser supported-datatypes}@anchor{22dd}
@subsubsection Supported Datatypes


Config parsers do not guess datatypes of values in configuration files, always
storing them internally as strings.  This means that if you need other
datatypes, you should convert on your own:

@example
>>> int(topsecret['Port'])
50022
>>> float(topsecret['CompressionLevel'])
9.0
@end example

Since this task is so common, config parsers provide a range of handy getter
methods to handle integers, floats and booleans.  The last one is the most
interesting because simply passing the value to @code{bool()} would do no good
since @code{bool('False')} is still @code{True}.  This is why config parsers also
provide @ref{22de,,getboolean()}.  This method is case-insensitive and
recognizes Boolean values from @code{'yes'}/@code{'no'}, @code{'on'}/@code{'off'},
@code{'true'}/@code{'false'} and @code{'1'}/@code{'0'} @footnote{@w{(1)} 
Config parsers allow for heavy customization.  If you are interested in
changing the behaviour outlined by the footnote reference, consult the
@ref{22dc,,Customizing Parser Behaviour} section.
}.  For example:

@example
>>> topsecret.getboolean('ForwardX11')
False
>>> config['bitbucket.org'].getboolean('ForwardX11')
True
>>> config.getboolean('bitbucket.org', 'Compression')
True
@end example

Apart from @ref{22de,,getboolean()}, config parsers also
provide equivalent @ref{22df,,getint()} and
@ref{22e0,,getfloat()} methods.  You can register your own
converters and customize the provided ones. @footnote{@w{(1)} 
Config parsers allow for heavy customization.  If you are interested in
changing the behaviour outlined by the footnote reference, consult the
@ref{22dc,,Customizing Parser Behaviour} section.
}

@node Fallback Values,Supported INI File Structure,Supported Datatypes,configparser — Configuration file parser
@anchor{library/configparser fallback-values}@anchor{22e1}
@subsubsection Fallback Values


As with a dictionary, you can use a section’s @code{get()} method to
provide fallback values:

@example
>>> topsecret.get('Port')
'50022'
>>> topsecret.get('CompressionLevel')
'9'
>>> topsecret.get('Cipher')
>>> topsecret.get('Cipher', '3des-cbc')
'3des-cbc'
@end example

Please note that default values have precedence over fallback values.
For instance, in our example the @code{'CompressionLevel'} key was
specified only in the @code{'DEFAULT'} section.  If we try to get it from
the section @code{'topsecret.server.com'}, we will always get the default,
even if we specify a fallback:

@example
>>> topsecret.get('CompressionLevel', '3')
'9'
@end example

One more thing to be aware of is that the parser-level @code{get()} method
provides a custom, more complex interface, maintained for backwards
compatibility.  When using this method, a fallback value can be provided via
the @code{fallback} keyword-only argument:

@example
>>> config.get('bitbucket.org', 'monster',
...            fallback='No such things as monsters')
'No such things as monsters'
@end example

The same @code{fallback} argument can be used with the
@ref{22df,,getint()}, @ref{22e0,,getfloat()} and
@ref{22de,,getboolean()} methods, for example:

@example
>>> 'BatchMode' in topsecret
False
>>> topsecret.getboolean('BatchMode', fallback=True)
True
>>> config['DEFAULT']['BatchMode'] = 'no'
>>> topsecret.getboolean('BatchMode', fallback=True)
False
@end example

@node Supported INI File Structure,Interpolation of values,Fallback Values,configparser — Configuration file parser
@anchor{library/configparser supported-ini-file-structure}@anchor{22da}
@subsubsection Supported INI File Structure


A configuration file consists of sections, each led by a @code{[section]} header,
followed by key/value entries separated by a specific string (@code{=} or @code{:} by
default @footnote{@w{(1)} 
Config parsers allow for heavy customization.  If you are interested in
changing the behaviour outlined by the footnote reference, consult the
@ref{22dc,,Customizing Parser Behaviour} section.
}).  By default, section names are case sensitive but keys are not
@footnote{@w{(1)} 
Config parsers allow for heavy customization.  If you are interested in
changing the behaviour outlined by the footnote reference, consult the
@ref{22dc,,Customizing Parser Behaviour} section.
}.  Leading and trailing whitespace is removed from keys and values.
Values can be omitted if the parser is configured to allow it @footnote{@w{(1)} 
Config parsers allow for heavy customization.  If you are interested in
changing the behaviour outlined by the footnote reference, consult the
@ref{22dc,,Customizing Parser Behaviour} section.
},
in which case the key/value delimiter may also be left
out.  Values can also span multiple lines, as long as they are indented deeper
than the first line of the value.  Depending on the parser’s mode, blank lines
may be treated as parts of multiline values or ignored.

By default,  a valid section name can be any string that does not contain ‘\n’ or ‘]’.
To change this, see @ref{22e2,,ConfigParser.SECTCRE}.

Configuration files may include comments, prefixed by specific
characters (@code{#} and @code{;} by default @footnote{@w{(1)} 
Config parsers allow for heavy customization.  If you are interested in
changing the behaviour outlined by the footnote reference, consult the
@ref{22dc,,Customizing Parser Behaviour} section.
}).  Comments may appear on
their own on an otherwise empty line, possibly indented. @footnote{@w{(1)} 
Config parsers allow for heavy customization.  If you are interested in
changing the behaviour outlined by the footnote reference, consult the
@ref{22dc,,Customizing Parser Behaviour} section.
}

For example:

@example
[Simple Values]
key=value
spaces in keys=allowed
spaces in values=allowed as well
spaces around the delimiter = obviously
you can also use : to delimit keys from values

[All Values Are Strings]
values like this: 1000000
or this: 3.14159265359
are they treated as numbers? : no
integers, floats and booleans are held as: strings
can use the API to get converted values directly: true

[Multiline Values]
chorus: I'm a lumberjack, and I'm okay
    I sleep all night and I work all day

[No Values]
key_without_value
empty string value here =

[You can use comments]
# like this
; or this

# By default only in an empty line.
# Inline comments can be harmful because they prevent users
# from using the delimiting characters as parts of values.
# That being said, this can be customized.

    [Sections Can Be Indented]
        can_values_be_as_well = True
        does_that_mean_anything_special = False
        purpose = formatting for readability
        multiline_values = are
            handled just fine as
            long as they are indented
            deeper than the first line
            of a value
        # Did I mention we can indent comments, too?
@end example

@node Interpolation of values,Mapping Protocol Access,Supported INI File Structure,configparser — Configuration file parser
@anchor{library/configparser interpolation-of-values}@anchor{22e3}
@subsubsection Interpolation of values


On top of the core functionality, @ref{759,,ConfigParser} supports
interpolation.  This means values can be preprocessed before returning them
from @code{get()} calls.

@geindex % (percent); interpolation in configuration files

@geindex BasicInterpolation (class in configparser)
@anchor{library/configparser configparser BasicInterpolation}@anchor{22e4}
@deffn {Class} configparser.BasicInterpolation

The default implementation used by @ref{759,,ConfigParser}.  It enables
values to contain format strings which refer to other values in the same
section, or values in the special default section @footnote{@w{(1)} 
Config parsers allow for heavy customization.  If you are interested in
changing the behaviour outlined by the footnote reference, consult the
@ref{22dc,,Customizing Parser Behaviour} section.
}.  Additional default
values can be provided on initialization.

For example:

@example
[Paths]
home_dir: /Users
my_dir: %(home_dir)s/lumberjack
my_pictures: %(my_dir)s/Pictures

[Escape]
gain: 80%%  # use a %% to escape the % sign (% is the only character that needs to be escaped)
@end example

In the example above, @ref{759,,ConfigParser} with @emph{interpolation} set to
@code{BasicInterpolation()} would resolve @code{%(home_dir)s} to the value of
@code{home_dir} (@code{/Users} in this case).  @code{%(my_dir)s} in effect would
resolve to @code{/Users/lumberjack}.  All interpolations are done on demand so
keys used in the chain of references do not have to be specified in any
specific order in the configuration file.

With @code{interpolation} set to @code{None}, the parser would simply return
@code{%(my_dir)s/Pictures} as the value of @code{my_pictures} and
@code{%(home_dir)s/lumberjack} as the value of @code{my_dir}.
@end deffn

@geindex $ (dollar); interpolation in configuration files

@geindex ExtendedInterpolation (class in configparser)
@anchor{library/configparser configparser ExtendedInterpolation}@anchor{df9}
@deffn {Class} configparser.ExtendedInterpolation

An alternative handler for interpolation which implements a more advanced
syntax, used for instance in @code{zc.buildout}.  Extended interpolation is
using @code{$@{section:option@}} to denote a value from a foreign section.
Interpolation can span multiple levels.  For convenience, if the
@code{section:} part is omitted, interpolation defaults to the current section
(and possibly the default values from the special section).

For example, the configuration specified above with basic interpolation,
would look like this with extended interpolation:

@example
[Paths]
home_dir: /Users
my_dir: $@{home_dir@}/lumberjack
my_pictures: $@{my_dir@}/Pictures

[Escape]
cost: $$80  # use a $$ to escape the $ sign ($ is the only character that needs to be escaped)
@end example

Values from other sections can be fetched as well:

@example
[Common]
home_dir: /Users
library_dir: /Library
system_dir: /System
macports_dir: /opt/local

[Frameworks]
Python: 3.2
path: $@{Common:system_dir@}/Library/Frameworks/

[Arthur]
nickname: Two Sheds
last_name: Jackson
my_dir: $@{Common:home_dir@}/twosheds
my_pictures: $@{my_dir@}/Pictures
python_dir: $@{Frameworks:path@}/Python/Versions/$@{Frameworks:Python@}
@end example
@end deffn

@node Mapping Protocol Access,Customizing Parser Behaviour,Interpolation of values,configparser — Configuration file parser
@anchor{library/configparser mapping-protocol-access}@anchor{22db}
@subsubsection Mapping Protocol Access



New in version 3.2.


Mapping protocol access is a generic name for functionality that enables using
custom objects as if they were dictionaries.  In case of @ref{21,,configparser},
the mapping interface implementation is using the
@code{parser['section']['option']} notation.

@code{parser['section']} in particular returns a proxy for the section’s data in
the parser.  This means that the values are not copied but they are taken from
the original parser on demand.  What’s even more important is that when values
are changed on a section proxy, they are actually mutated in the original
parser.

@ref{21,,configparser} objects behave as close to actual dictionaries as possible.
The mapping interface is complete and adheres to the
@ref{c3f,,MutableMapping} ABC.
However, there are a few differences that should be taken into account:


@itemize *

@item 
By default, all keys in sections are accessible in a case-insensitive manner
@footnote{@w{(1)} 
Config parsers allow for heavy customization.  If you are interested in
changing the behaviour outlined by the footnote reference, consult the
@ref{22dc,,Customizing Parser Behaviour} section.
}.  E.g. @code{for option in parser["section"]} yields only @code{optionxform}’ed
option key names.  This means lowercased keys by default.  At the same time,
for a section that holds the key @code{'a'}, both expressions return @code{True}:

@example
"a" in parser["section"]
"A" in parser["section"]
@end example

@item 
All sections include @code{DEFAULTSECT} values as well which means that
@code{.clear()} on a section may not leave the section visibly empty.  This is
because default values cannot be deleted from the section (because technically
they are not there).  If they are overridden in the section, deleting causes
the default value to be visible again.  Trying to delete a default value
causes a @ref{5bb,,KeyError}.

@item 
@code{DEFAULTSECT} cannot be removed from the parser:


@itemize *

@item 
trying to delete it raises @ref{17c,,ValueError},

@item 
@code{parser.clear()} leaves it intact,

@item 
@code{parser.popitem()} never returns it.
@end itemize

@item 
@code{parser.get(section, option, **kwargs)} - the second argument is @strong{not}
a fallback value.  Note however that the section-level @code{get()} methods are
compatible both with the mapping protocol and the classic configparser API.

@item 
@code{parser.items()} is compatible with the mapping protocol (returns a list of
@emph{section_name}, @emph{section_proxy} pairs including the DEFAULTSECT).  However,
this method can also be invoked with arguments: @code{parser.items(section, raw,
vars)}.  The latter call returns a list of @emph{option}, @emph{value} pairs for
a specified @code{section}, with all interpolations expanded (unless
@code{raw=True} is provided).
@end itemize

The mapping protocol is implemented on top of the existing legacy API so that
subclasses overriding the original interface still should have mappings working
as expected.

@node Customizing Parser Behaviour,Legacy API Examples,Mapping Protocol Access,configparser — Configuration file parser
@anchor{library/configparser customizing-parser-behaviour}@anchor{22dc}
@subsubsection Customizing Parser Behaviour


There are nearly as many INI format variants as there are applications using it.
@ref{21,,configparser} goes a long way to provide support for the largest sensible
set of INI styles available.  The default functionality is mainly dictated by
historical background and it’s very likely that you will want to customize some
of the features.

The most common way to change the way a specific config parser works is to use
the @code{__init__()} options:


@itemize *

@item 
@emph{defaults}, default value: @code{None}

This option accepts a dictionary of key-value pairs which will be initially
put in the @code{DEFAULT} section.  This makes for an elegant way to support
concise configuration files that don’t specify values which are the same as
the documented default.

Hint: if you want to specify default values for a specific section, use
@code{read_dict()} before you read the actual file.

@item 
@emph{dict_type}, default value: @ref{3b0,,dict}

This option has a major impact on how the mapping protocol will behave and how
the written configuration files look.  With the standard dictionary, every
section is stored in the order they were added to the parser.  Same goes for
options within sections.

An alternative dictionary type can be used for example to sort sections and
options on write-back.

Please note: there are ways to add a set of key-value pairs in a single
operation.  When you use a regular dictionary in those operations, the order
of the keys will be ordered.  For example:

@example
>>> parser = configparser.ConfigParser()
>>> parser.read_dict(@{'section1': @{'key1': 'value1',
...                                'key2': 'value2',
...                                'key3': 'value3'@},
...                   'section2': @{'keyA': 'valueA',
...                                'keyB': 'valueB',
...                                'keyC': 'valueC'@},
...                   'section3': @{'foo': 'x',
...                                'bar': 'y',
...                                'baz': 'z'@}
... @})
>>> parser.sections()
['section1', 'section2', 'section3']
>>> [option for option in parser['section3']]
['foo', 'bar', 'baz']
@end example

@item 
@emph{allow_no_value}, default value: @code{False}

Some configuration files are known to include settings without values, but
which otherwise conform to the syntax supported by @ref{21,,configparser}.  The
@emph{allow_no_value} parameter to the constructor can be used to
indicate that such values should be accepted:

@example
>>> import configparser

>>> sample_config = """
... [mysqld]
...   user = mysql
...   pid-file = /var/run/mysqld/mysqld.pid
...   skip-external-locking
...   old_passwords = 1
...   skip-bdb
...   # we don't need ACID today
...   skip-innodb
... """
>>> config = configparser.ConfigParser(allow_no_value=True)
>>> config.read_string(sample_config)

>>> # Settings with values are treated as before:
>>> config["mysqld"]["user"]
'mysql'

>>> # Settings without values provide None:
>>> config["mysqld"]["skip-bdb"]

>>> # Settings which aren't specified still raise an error:
>>> config["mysqld"]["does-not-exist"]
Traceback (most recent call last):
  ...
KeyError: 'does-not-exist'
@end example

@item 
@emph{delimiters}, default value: @code{('=', ':')}

Delimiters are substrings that delimit keys from values within a section.
The first occurrence of a delimiting substring on a line is considered
a delimiter.  This means values (but not keys) can contain the delimiters.

See also the @emph{space_around_delimiters} argument to
@ref{22e5,,ConfigParser.write()}.

@item 
@emph{comment_prefixes}, default value: @code{('#', ';')}

@item 
@emph{inline_comment_prefixes}, default value: @code{None}

Comment prefixes are strings that indicate the start of a valid comment within
a config file. @emph{comment_prefixes} are used only on otherwise empty lines
(optionally indented) whereas @emph{inline_comment_prefixes} can be used after
every valid value (e.g. section names, options and empty lines as well).  By
default inline comments are disabled and @code{'#'} and @code{';'} are used as
prefixes for whole line comments.


Changed in version 3.2: In previous versions of @ref{21,,configparser} behaviour matched
@code{comment_prefixes=('#',';')} and @code{inline_comment_prefixes=(';',)}.


Please note that config parsers don’t support escaping of comment prefixes so
using @emph{inline_comment_prefixes} may prevent users from specifying option
values with characters used as comment prefixes.  When in doubt, avoid
setting @emph{inline_comment_prefixes}.  In any circumstances, the only way of
storing comment prefix characters at the beginning of a line in multiline
values is to interpolate the prefix, for example:

@example
>>> from configparser import ConfigParser, ExtendedInterpolation
>>> parser = ConfigParser(interpolation=ExtendedInterpolation())
>>> # the default BasicInterpolation could be used as well
>>> parser.read_string("""
... [DEFAULT]
... hash = #
...
... [hashes]
... shebang =
...   $@{hash@}!/usr/bin/env python
...   $@{hash@} -*- coding: utf-8 -*-
...
... extensions =
...   enabled_extension
...   another_extension
...   #disabled_by_comment
...   yet_another_extension
...
... interpolation not necessary = if # is not at line start
... even in multiline values = line #1
...   line #2
...   line #3
... """)
>>> print(parser['hashes']['shebang'])

#!/usr/bin/env python
# -*- coding: utf-8 -*-
>>> print(parser['hashes']['extensions'])

enabled_extension
another_extension
yet_another_extension
>>> print(parser['hashes']['interpolation not necessary'])
if # is not at line start
>>> print(parser['hashes']['even in multiline values'])
line #1
line #2
line #3
@end example

@item 
@emph{strict}, default value: @code{True}

When set to @code{True}, the parser will not allow for any section or option
duplicates while reading from a single source (using @code{read_file()},
@code{read_string()} or @code{read_dict()}).  It is recommended to use strict
parsers in new applications.


Changed in version 3.2: In previous versions of @ref{21,,configparser} behaviour matched
@code{strict=False}.


@item 
@emph{empty_lines_in_values}, default value: @code{True}

In config parsers, values can span multiple lines as long as they are
indented more than the key that holds them.  By default parsers also let
empty lines to be parts of values.  At the same time, keys can be arbitrarily
indented themselves to improve readability.  In consequence, when
configuration files get big and complex, it is easy for the user to lose
track of the file structure.  Take for instance:

@example
[Section]
key = multiline
  value with a gotcha

 this = is still a part of the multiline value of 'key'
@end example

This can be especially problematic for the user to see if she’s using a
proportional font to edit the file.  That is why when your application does
not need values with empty lines, you should consider disallowing them.  This
will make empty lines split keys every time.  In the example above, it would
produce two keys, @code{key} and @code{this}.

@item 
@emph{default_section}, default value: @code{configparser.DEFAULTSECT} (that is:
@code{"DEFAULT"})

The convention of allowing a special section of default values for other
sections or interpolation purposes is a powerful concept of this library,
letting users create complex declarative configurations.  This section is
normally called @code{"DEFAULT"} but this can be customized to point to any
other valid section name.  Some typical values include: @code{"general"} or
@code{"common"}.  The name provided is used for recognizing default sections
when reading from any source and is used when writing configuration back to
a file.  Its current value can be retrieved using the
@code{parser_instance.default_section} attribute and may be modified at runtime
(i.e. to convert files from one format to another).

@item 
@emph{interpolation}, default value: @code{configparser.BasicInterpolation}

Interpolation behaviour may be customized by providing a custom handler
through the @emph{interpolation} argument. @code{None} can be used to turn off
interpolation completely, @code{ExtendedInterpolation()} provides a more
advanced variant inspired by @code{zc.buildout}.  More on the subject in the
@ref{22e3,,dedicated documentation section}.
@ref{ae2,,RawConfigParser} has a default value of @code{None}.

@item 
@emph{converters}, default value: not set

Config parsers provide option value getters that perform type conversion.  By
default @ref{22df,,getint()}, @ref{22e0,,getfloat()}, and
@ref{22de,,getboolean()} are implemented.  Should other getters be
desirable, users may define them in a subclass or pass a dictionary where each
key is a name of the converter and each value is a callable implementing said
conversion.  For instance, passing @code{@{'decimal': decimal.Decimal@}} would add
@code{getdecimal()} on both the parser object and all section proxies.  In
other words, it will be possible to write both
@code{parser_instance.getdecimal('section', 'key', fallback=0)} and
@code{parser_instance['section'].getdecimal('key', 0)}.

If the converter needs to access the state of the parser, it can be
implemented as a method on a config parser subclass.  If the name of this
method starts with @code{get}, it will be available on all section proxies, in
the dict-compatible form (see the @code{getdecimal()} example above).
@end itemize

More advanced customization may be achieved by overriding default values of
these parser attributes.  The defaults are defined on the classes, so they may
be overridden by subclasses or by attribute assignment.

@geindex BOOLEAN_STATES (configparser.ConfigParser attribute)
@anchor{library/configparser configparser ConfigParser BOOLEAN_STATES}@anchor{22e6}
@deffn {Attribute} ConfigParser.BOOLEAN_STATES

By default when using @ref{22de,,getboolean()}, config parsers
consider the following values @code{True}: @code{'1'}, @code{'yes'}, @code{'true'},
@code{'on'} and the following values @code{False}: @code{'0'}, @code{'no'}, @code{'false'},
@code{'off'}.  You can override this by specifying a custom dictionary of strings
and their Boolean outcomes. For example:

@example
>>> custom = configparser.ConfigParser()
>>> custom['section1'] = @{'funky': 'nope'@}
>>> custom['section1'].getboolean('funky')
Traceback (most recent call last):
...
ValueError: Not a boolean: nope
>>> custom.BOOLEAN_STATES = @{'sure': True, 'nope': False@}
>>> custom['section1'].getboolean('funky')
False
@end example

Other typical Boolean pairs include @code{accept}/@code{reject} or
@code{enabled}/@code{disabled}.
@end deffn


@deffn {Method} ConfigParser.optionxform (option)

This method transforms option names on every read, get, or set
operation.  The default converts the name to lowercase.  This also
means that when a configuration file gets written, all keys will be
lowercase.  Override this method if that’s unsuitable.
For example:

@example
>>> config = """
... [Section1]
... Key = Value
...
... [Section2]
... AnotherKey = Value
... """
>>> typical = configparser.ConfigParser()
>>> typical.read_string(config)
>>> list(typical['Section1'].keys())
['key']
>>> list(typical['Section2'].keys())
['anotherkey']
>>> custom = configparser.RawConfigParser()
>>> custom.optionxform = lambda option: option
>>> custom.read_string(config)
>>> list(custom['Section1'].keys())
['Key']
>>> list(custom['Section2'].keys())
['AnotherKey']
@end example

@cartouche
@quotation Note 
The optionxform function transforms option names to a canonical form.
This should be an idempotent function: if the name is already in
canonical form, it should be returned unchanged.
@end quotation
@end cartouche
@end deffn

@geindex SECTCRE (configparser.ConfigParser attribute)
@anchor{library/configparser configparser ConfigParser SECTCRE}@anchor{22e2}
@deffn {Attribute} ConfigParser.SECTCRE

A compiled regular expression used to parse section headers.  The default
matches @code{[section]} to the name @code{"section"}.  Whitespace is considered
part of the section name, thus @code{[  larch  ]} will be read as a section of
name @code{"  larch  "}.  Override this attribute if that’s unsuitable.  For
example:

@example
>>> import re
>>> config = """
... [Section 1]
... option = value
...
... [  Section 2  ]
... another = val
... """
>>> typical = configparser.ConfigParser()
>>> typical.read_string(config)
>>> typical.sections()
['Section 1', '  Section 2  ']
>>> custom = configparser.ConfigParser()
>>> custom.SECTCRE = re.compile(r"\[ *(?P<header>[^]]+?) *\]")
>>> custom.read_string(config)
>>> custom.sections()
['Section 1', 'Section 2']
@end example

@cartouche
@quotation Note 
While ConfigParser objects also use an @code{OPTCRE} attribute for recognizing
option lines, it’s not recommended to override it because that would
interfere with constructor options @emph{allow_no_value} and @emph{delimiters}.
@end quotation
@end cartouche
@end deffn

@node Legacy API Examples,ConfigParser Objects,Customizing Parser Behaviour,configparser — Configuration file parser
@anchor{library/configparser legacy-api-examples}@anchor{22e7}
@subsubsection Legacy API Examples


Mainly because of backwards compatibility concerns, @ref{21,,configparser}
provides also a legacy API with explicit @code{get}/@code{set} methods.  While there
are valid use cases for the methods outlined below, mapping protocol access is
preferred for new projects.  The legacy API is at times more advanced,
low-level and downright counterintuitive.

An example of writing to a configuration file:

@example
import configparser

config = configparser.RawConfigParser()

# Please note that using RawConfigParser's set functions, you can assign
# non-string values to keys internally, but will receive an error when
# attempting to write to a file or when you get it in non-raw mode. Setting
# values using the mapping protocol or ConfigParser's set() does not allow
# such assignments to take place.
config.add_section('Section1')
config.set('Section1', 'an_int', '15')
config.set('Section1', 'a_bool', 'true')
config.set('Section1', 'a_float', '3.1415')
config.set('Section1', 'baz', 'fun')
config.set('Section1', 'bar', 'Python')
config.set('Section1', 'foo', '%(bar)s is %(baz)s!')

# Writing our configuration file to 'example.cfg'
with open('example.cfg', 'w') as configfile:
    config.write(configfile)
@end example

An example of reading the configuration file again:

@example
import configparser

config = configparser.RawConfigParser()
config.read('example.cfg')

# getfloat() raises an exception if the value is not a float
# getint() and getboolean() also do this for their respective types
a_float = config.getfloat('Section1', 'a_float')
an_int = config.getint('Section1', 'an_int')
print(a_float + an_int)

# Notice that the next output does not interpolate '%(bar)s' or '%(baz)s'.
# This is because we are using a RawConfigParser().
if config.getboolean('Section1', 'a_bool'):
    print(config.get('Section1', 'foo'))
@end example

To get interpolation, use @ref{759,,ConfigParser}:

@example
import configparser

cfg = configparser.ConfigParser()
cfg.read('example.cfg')

# Set the optional *raw* argument of get() to True if you wish to disable
# interpolation in a single get operation.
print(cfg.get('Section1', 'foo', raw=False))  # -> "Python is fun!"
print(cfg.get('Section1', 'foo', raw=True))   # -> "%(bar)s is %(baz)s!"

# The optional *vars* argument is a dict with members that will take
# precedence in interpolation.
print(cfg.get('Section1', 'foo', vars=@{'bar': 'Documentation',
                                       'baz': 'evil'@}))

# The optional *fallback* argument can be used to provide a fallback value
print(cfg.get('Section1', 'foo'))
      # -> "Python is fun!"

print(cfg.get('Section1', 'foo', fallback='Monty is not.'))
      # -> "Python is fun!"

print(cfg.get('Section1', 'monster', fallback='No such things as monsters.'))
      # -> "No such things as monsters."

# A bare print(cfg.get('Section1', 'monster')) would raise NoOptionError
# but we can also use:

print(cfg.get('Section1', 'monster', fallback=None))
      # -> None
@end example

Default values are available in both types of ConfigParsers.  They are used in
interpolation if an option used is not defined elsewhere.

@example
import configparser

# New instance with 'bar' and 'baz' defaulting to 'Life' and 'hard' each
config = configparser.ConfigParser(@{'bar': 'Life', 'baz': 'hard'@})
config.read('example.cfg')

print(config.get('Section1', 'foo'))     # -> "Python is fun!"
config.remove_option('Section1', 'bar')
config.remove_option('Section1', 'baz')
print(config.get('Section1', 'foo'))     # -> "Life is hard!"
@end example

@node ConfigParser Objects,RawConfigParser Objects,Legacy API Examples,configparser — Configuration file parser
@anchor{library/configparser configparser-objects}@anchor{22e8}@anchor{library/configparser id12}@anchor{22e9}
@subsubsection ConfigParser Objects


@geindex ConfigParser (class in configparser)
@anchor{library/configparser configparser ConfigParser}@anchor{759}
@deffn {Class} configparser.ConfigParser (defaults=None, dict_type=dict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation(), converters=@{@})

The main configuration parser.  When @emph{defaults} is given, it is initialized
into the dictionary of intrinsic defaults.  When @emph{dict_type} is given, it
will be used to create the dictionary objects for the list of sections, for
the options within a section, and for the default values.

When @emph{delimiters} is given, it is used as the set of substrings that
divide keys from values.  When @emph{comment_prefixes} is given, it will be used
as the set of substrings that prefix comments in otherwise empty lines.
Comments can be indented.  When @emph{inline_comment_prefixes} is given, it will
be used as the set of substrings that prefix comments in non-empty lines.

When @emph{strict} is @code{True} (the default), the parser won’t allow for
any section or option duplicates while reading from a single source (file,
string or dictionary), raising @ref{e1f,,DuplicateSectionError} or
@ref{e20,,DuplicateOptionError}.  When @emph{empty_lines_in_values} is @code{False}
(default: @code{True}), each empty line marks the end of an option.  Otherwise,
internal empty lines of a multiline option are kept as part of the value.
When @emph{allow_no_value} is @code{True} (default: @code{False}), options without
values are accepted; the value held for these is @code{None} and they are
serialized without the trailing delimiter.

When @emph{default_section} is given, it specifies the name for the special
section holding default values for other sections and interpolation purposes
(normally named @code{"DEFAULT"}).  This value can be retrieved and changed on
runtime using the @code{default_section} instance attribute.

Interpolation behaviour may be customized by providing a custom handler
through the @emph{interpolation} argument. @code{None} can be used to turn off
interpolation completely, @code{ExtendedInterpolation()} provides a more
advanced variant inspired by @code{zc.buildout}.  More on the subject in the
@ref{22e3,,dedicated documentation section}.

All option names used in interpolation will be passed through the
@ref{22ea,,optionxform()} method just like any other option name reference.  For
example, using the default implementation of @ref{22ea,,optionxform()} (which
converts option names to lower case), the values @code{foo %(bar)s} and @code{foo
%(BAR)s} are equivalent.

When @emph{converters} is given, it should be a dictionary where each key
represents the name of a type converter and each value is a callable
implementing the conversion from string to the desired datatype.  Every
converter gets its own corresponding @code{get*()} method on the parser
object and section proxies.


Changed in version 3.1: The default @emph{dict_type} is @ref{4e5,,collections.OrderedDict}.



Changed in version 3.2: @emph{allow_no_value}, @emph{delimiters}, @emph{comment_prefixes}, @emph{strict},
@emph{empty_lines_in_values}, @emph{default_section} and @emph{interpolation} were
added.



Changed in version 3.5: The @emph{converters} argument was added.



Changed in version 3.7: The @emph{defaults} argument is read with @ref{22eb,,read_dict()},
providing consistent behavior across the parser: non-string
keys and values are implicitly converted to strings.



Changed in version 3.8: The default @emph{dict_type} is @ref{3b0,,dict}, since it now preserves
insertion order.


@geindex defaults() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser defaults}@anchor{22ec}
@deffn {Method} defaults ()

Return a dictionary containing the instance-wide defaults.
@end deffn

@geindex sections() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser sections}@anchor{22ed}
@deffn {Method} sections ()

Return a list of the sections available; the @emph{default section} is not
included in the list.
@end deffn

@geindex add_section() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser add_section}@anchor{e1e}
@deffn {Method} add_section (section)

Add a section named @emph{section} to the instance.  If a section by the given
name already exists, @ref{e1f,,DuplicateSectionError} is raised.  If the
@emph{default section} name is passed, @ref{17c,,ValueError} is raised.  The name
of the section must be a string; if not, @ref{162,,TypeError} is raised.


Changed in version 3.2: Non-string section names raise @ref{162,,TypeError}.

@end deffn

@geindex has_section() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser has_section}@anchor{22ee}
@deffn {Method} has_section (section)

Indicates whether the named @emph{section} is present in the configuration.
The @emph{default section} is not acknowledged.
@end deffn

@geindex options() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser options}@anchor{22ef}
@deffn {Method} options (section)

Return a list of options available in the specified @emph{section}.
@end deffn

@geindex has_option() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser has_option}@anchor{22f0}
@deffn {Method} has_option (section, option)

If the given @emph{section} exists, and contains the given @emph{option}, return
@ref{748,,True}; otherwise return @ref{659,,False}.  If the specified
@emph{section} is @ref{182,,None} or an empty string, DEFAULT is assumed.
@end deffn

@geindex read() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser read}@anchor{1366}
@deffn {Method} read (filenames, encoding=None)

Attempt to read and parse an iterable of filenames, returning a list of
filenames which were successfully parsed.

If @emph{filenames} is a string, a @ref{2f0,,bytes} object or a
@ref{63d,,path-like object}, it is treated as
a single filename.  If a file named in @emph{filenames} cannot be opened, that
file will be ignored.  This is designed so that you can specify an
iterable of potential configuration file locations (for example, the
current directory, the user’s home directory, and some system-wide
directory), and all existing configuration files in the iterable will be
read.

If none of the named files exist, the @ref{759,,ConfigParser}
instance will contain an empty dataset.  An application which requires
initial values to be loaded from a file should load the required file or
files using @ref{22f1,,read_file()} before calling @ref{1366,,read()} for any
optional files:

@example
import configparser, os

config = configparser.ConfigParser()
config.read_file(open('defaults.cfg'))
config.read(['site.cfg', os.path.expanduser('~/.myapp.cfg')],
            encoding='cp1250')
@end example


New in version 3.2: The @emph{encoding} parameter.  Previously, all files were read using the
default encoding for @ref{1c8,,open()}.



New in version 3.6.1: The @emph{filenames} parameter accepts a @ref{63d,,path-like object}.



New in version 3.7: The @emph{filenames} parameter accepts a @ref{2f0,,bytes} object.

@end deffn

@geindex read_file() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser read_file}@anchor{22f1}
@deffn {Method} read_file (f, source=None)

Read and parse configuration data from @emph{f} which must be an iterable
yielding Unicode strings (for example files opened in text mode).

Optional argument @emph{source} specifies the name of the file being read.  If
not given and @emph{f} has a @code{name} attribute, that is used for
@emph{source}; the default is @code{'<???>'}.


New in version 3.2: Replaces @ref{22f2,,readfp()}.

@end deffn

@geindex read_string() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser read_string}@anchor{22f3}
@deffn {Method} read_string (string, source='<string>')

Parse configuration data from a string.

Optional argument @emph{source} specifies a context-specific name of the
string passed.  If not given, @code{'<string>'} is used.  This should
commonly be a filesystem path or a URL.


New in version 3.2.

@end deffn

@geindex read_dict() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser read_dict}@anchor{22eb}
@deffn {Method} read_dict (dictionary, source='<dict>')

Load configuration from any object that provides a dict-like @code{items()}
method.  Keys are section names, values are dictionaries with keys and
values that should be present in the section.  If the used dictionary
type preserves order, sections and their keys will be added in order.
Values are automatically converted to strings.

Optional argument @emph{source} specifies a context-specific name of the
dictionary passed.  If not given, @code{<dict>} is used.

This method can be used to copy state between parsers.


New in version 3.2.

@end deffn

@geindex get() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser get}@anchor{e1c}
@deffn {Method} get (section, option, *, raw=False, vars=None[, fallback])

Get an @emph{option} value for the named @emph{section}.  If @emph{vars} is provided, it
must be a dictionary.  The @emph{option} is looked up in @emph{vars} (if provided),
@emph{section}, and in @emph{DEFAULTSECT} in that order.  If the key is not found
and @emph{fallback} is provided, it is used as a fallback value.  @code{None} can
be provided as a @emph{fallback} value.

All the @code{'%'} interpolations are expanded in the return values, unless
the @emph{raw} argument is true.  Values for interpolation keys are looked up
in the same manner as the option.


Changed in version 3.2: Arguments @emph{raw}, @emph{vars} and @emph{fallback} are keyword only to protect
users from trying to use the third argument as the @emph{fallback} fallback
(especially when using the mapping protocol).

@end deffn

@geindex getint() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser getint}@anchor{22df}
@deffn {Method} getint (section, option, *, raw=False, vars=None[, fallback])

A convenience method which coerces the @emph{option} in the specified @emph{section}
to an integer.  See @ref{e1c,,get()} for explanation of @emph{raw}, @emph{vars} and
@emph{fallback}.
@end deffn

@geindex getfloat() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser getfloat}@anchor{22e0}
@deffn {Method} getfloat (section, option, *, raw=False, vars=None[, fallback])

A convenience method which coerces the @emph{option} in the specified @emph{section}
to a floating point number.  See @ref{e1c,,get()} for explanation of @emph{raw},
@emph{vars} and @emph{fallback}.
@end deffn

@geindex getboolean() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser getboolean}@anchor{22de}
@deffn {Method} getboolean (section, option, *, raw=False, vars=None[, fallback])

A convenience method which coerces the @emph{option} in the specified @emph{section}
to a Boolean value.  Note that the accepted values for the option are
@code{'1'}, @code{'yes'}, @code{'true'}, and @code{'on'}, which cause this method to
return @code{True}, and @code{'0'}, @code{'no'}, @code{'false'}, and @code{'off'}, which
cause it to return @code{False}.  These string values are checked in a
case-insensitive manner.  Any other value will cause it to raise
@ref{17c,,ValueError}.  See @ref{e1c,,get()} for explanation of @emph{raw}, @emph{vars} and
@emph{fallback}.
@end deffn

@geindex items() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser items}@anchor{22f4}
@deffn {Method} items (raw=False, vars=None)

@deffnx {Method} items (section, raw=False, vars=None)

When @emph{section} is not given, return a list of @emph{section_name},
@emph{section_proxy} pairs, including DEFAULTSECT.

Otherwise, return a list of @emph{name}, @emph{value} pairs for the options in the
given @emph{section}.  Optional arguments have the same meaning as for the
@ref{e1c,,get()} method.


Changed in version 3.8: Items present in @emph{vars} no longer appear in the result.  The previous
behaviour mixed actual parser options with variables provided for
interpolation.

@end deffn

@geindex set() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser set}@anchor{e1d}
@deffn {Method} set (section, option, value)

If the given section exists, set the given option to the specified value;
otherwise raise @ref{22f5,,NoSectionError}.  @emph{option} and @emph{value} must be
strings; if not, @ref{162,,TypeError} is raised.
@end deffn

@geindex write() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser write}@anchor{22e5}
@deffn {Method} write (fileobject, space_around_delimiters=True)

Write a representation of the configuration to the specified @ref{d68,,file object}, which must be opened in text mode (accepting strings).  This
representation can be parsed by a future @ref{1366,,read()} call.  If
@emph{space_around_delimiters} is true, delimiters between
keys and values are surrounded by spaces.
@end deffn

@cartouche
@quotation Note 
Comments in the original configuration file are not preserved when
writing the configuration back.
What is considered a comment, depends on the given values for
@emph{comment_prefix} and @emph{inline_comment_prefix}.
@end quotation
@end cartouche

@geindex remove_option() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser remove_option}@anchor{22f6}
@deffn {Method} remove_option (section, option)

Remove the specified @emph{option} from the specified @emph{section}.  If the
section does not exist, raise @ref{22f5,,NoSectionError}.  If the option
existed to be removed, return @ref{748,,True}; otherwise return
@ref{659,,False}.
@end deffn

@geindex remove_section() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser remove_section}@anchor{22f7}
@deffn {Method} remove_section (section)

Remove the specified @emph{section} from the configuration.  If the section in
fact existed, return @code{True}.  Otherwise return @code{False}.
@end deffn

@geindex optionxform() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser optionxform}@anchor{22ea}
@deffn {Method} optionxform (option)

Transforms the option name @emph{option} as found in an input file or as passed
in by client code to the form that should be used in the internal
structures.  The default implementation returns a lower-case version of
@emph{option}; subclasses may override this or client code can set an attribute
of this name on instances to affect this behavior.

You don’t need to subclass the parser to use this method, you can also
set it on an instance, to a function that takes a string argument and
returns a string.  Setting it to @code{str}, for example, would make option
names case sensitive:

@example
cfgparser = ConfigParser()
cfgparser.optionxform = str
@end example

Note that when reading configuration files, whitespace around the option
names is stripped before @ref{22ea,,optionxform()} is called.
@end deffn

@geindex readfp() (configparser.ConfigParser method)
@anchor{library/configparser configparser ConfigParser readfp}@anchor{22f2}
@deffn {Method} readfp (fp, filename=None)


Deprecated since version 3.2: Use @ref{22f1,,read_file()} instead.



Changed in version 3.2: @ref{22f2,,readfp()} now iterates on @emph{fp} instead of calling @code{fp.readline()}.


For existing code calling @ref{22f2,,readfp()} with arguments which don’t
support iteration, the following generator may be used as a wrapper
around the file-like object:

@example
def readline_generator(fp):
    line = fp.readline()
    while line:
        yield line
        line = fp.readline()
@end example

Instead of @code{parser.readfp(fp)} use
@code{parser.read_file(readline_generator(fp))}.
@end deffn
@end deffn

@geindex MAX_INTERPOLATION_DEPTH (in module configparser)
@anchor{library/configparser configparser MAX_INTERPOLATION_DEPTH}@anchor{22f8}
@deffn {Data} configparser.MAX_INTERPOLATION_DEPTH

The maximum depth for recursive interpolation for @code{get()} when the @emph{raw}
parameter is false.  This is relevant only when the default @emph{interpolation}
is used.
@end deffn

@node RawConfigParser Objects,Exceptions<6>,ConfigParser Objects,configparser — Configuration file parser
@anchor{library/configparser id14}@anchor{22f9}@anchor{library/configparser rawconfigparser-objects}@anchor{22fa}
@subsubsection RawConfigParser Objects


@geindex RawConfigParser (class in configparser)
@anchor{library/configparser configparser RawConfigParser}@anchor{ae2}
@deffn {Class} configparser.RawConfigParser (defaults=None, dict_type=dict, allow_no_value=False, *, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT[, interpolation])

Legacy variant of the @ref{759,,ConfigParser}.  It has interpolation
disabled by default and allows for non-string section names, option
names, and values via its unsafe @code{add_section} and @code{set} methods,
as well as the legacy @code{defaults=} keyword argument handling.


Changed in version 3.8: The default @emph{dict_type} is @ref{3b0,,dict}, since it now preserves
insertion order.


@cartouche
@quotation Note 
Consider using @ref{759,,ConfigParser} instead which checks types of
the values to be stored internally.  If you don’t want interpolation, you
can use @code{ConfigParser(interpolation=None)}.
@end quotation
@end cartouche

@geindex add_section() (configparser.RawConfigParser method)
@anchor{library/configparser configparser RawConfigParser add_section}@anchor{22fb}
@deffn {Method} add_section (section)

Add a section named @emph{section} to the instance.  If a section by the given
name already exists, @ref{e1f,,DuplicateSectionError} is raised.  If the
@emph{default section} name is passed, @ref{17c,,ValueError} is raised.

Type of @emph{section} is not checked which lets users create non-string named
sections.  This behaviour is unsupported and may cause internal errors.
@end deffn

@geindex set() (configparser.RawConfigParser method)
@anchor{library/configparser configparser RawConfigParser set}@anchor{22fc}
@deffn {Method} set (section, option, value)

If the given section exists, set the given option to the specified value;
otherwise raise @ref{22f5,,NoSectionError}.  While it is possible to use
@ref{ae2,,RawConfigParser} (or @ref{759,,ConfigParser} with @emph{raw} parameters
set to true) for @emph{internal} storage of non-string values, full
functionality (including interpolation and output to files) can only be
achieved using string values.

This method lets users assign non-string values to keys internally.  This
behaviour is unsupported and will cause errors when attempting to write
to a file or get it in non-raw mode.  @strong{Use the mapping protocol API}
which does not allow such assignments to take place.
@end deffn
@end deffn

@node Exceptions<6>,,RawConfigParser Objects,configparser — Configuration file parser
@anchor{library/configparser exceptions}@anchor{22fd}
@subsubsection Exceptions


@geindex Error
@anchor{library/configparser configparser Error}@anchor{22fe}
@deffn {Exception} configparser.Error

Base class for all other @ref{21,,configparser} exceptions.
@end deffn

@geindex NoSectionError
@anchor{library/configparser configparser NoSectionError}@anchor{22f5}
@deffn {Exception} configparser.NoSectionError

Exception raised when a specified section is not found.
@end deffn

@geindex DuplicateSectionError
@anchor{library/configparser configparser DuplicateSectionError}@anchor{e1f}
@deffn {Exception} configparser.DuplicateSectionError

Exception raised if @code{add_section()} is called with the name of a section
that is already present or in strict parsers when a section if found more
than once in a single input file, string or dictionary.


New in version 3.2: Optional @code{source} and @code{lineno} attributes and arguments to
@code{__init__()} were added.

@end deffn

@geindex DuplicateOptionError
@anchor{library/configparser configparser DuplicateOptionError}@anchor{e20}
@deffn {Exception} configparser.DuplicateOptionError

Exception raised by strict parsers if a single option appears twice during
reading from a single file, string or dictionary. This catches misspellings
and case sensitivity-related errors, e.g. a dictionary may have two keys
representing the same case-insensitive configuration key.
@end deffn

@geindex NoOptionError
@anchor{library/configparser configparser NoOptionError}@anchor{22ff}
@deffn {Exception} configparser.NoOptionError

Exception raised when a specified option is not found in the specified
section.
@end deffn

@geindex InterpolationError
@anchor{library/configparser configparser InterpolationError}@anchor{2300}
@deffn {Exception} configparser.InterpolationError

Base class for exceptions raised when problems occur performing string
interpolation.
@end deffn

@geindex InterpolationDepthError
@anchor{library/configparser configparser InterpolationDepthError}@anchor{2301}
@deffn {Exception} configparser.InterpolationDepthError

Exception raised when string interpolation cannot be completed because the
number of iterations exceeds @ref{22f8,,MAX_INTERPOLATION_DEPTH}.  Subclass of
@ref{2300,,InterpolationError}.
@end deffn

@geindex InterpolationMissingOptionError
@anchor{library/configparser configparser InterpolationMissingOptionError}@anchor{2302}
@deffn {Exception} configparser.InterpolationMissingOptionError

Exception raised when an option referenced from a value does not exist.
Subclass of @ref{2300,,InterpolationError}.
@end deffn

@geindex InterpolationSyntaxError
@anchor{library/configparser configparser InterpolationSyntaxError}@anchor{2303}
@deffn {Exception} configparser.InterpolationSyntaxError

Exception raised when the source text into which substitutions are made does
not conform to the required syntax.  Subclass of @ref{2300,,InterpolationError}.
@end deffn

@geindex MissingSectionHeaderError
@anchor{library/configparser configparser MissingSectionHeaderError}@anchor{2304}
@deffn {Exception} configparser.MissingSectionHeaderError

Exception raised when attempting to parse a file which has no section
headers.
@end deffn

@geindex ParsingError
@anchor{library/configparser configparser ParsingError}@anchor{2305}
@deffn {Exception} configparser.ParsingError

Exception raised when errors occur attempting to parse a file.


Changed in version 3.2: The @code{filename} attribute and @code{__init__()} argument were renamed to
@code{source} for consistency.

@end deffn

@node netrc — netrc file processing,xdrlib — Encode and decode XDR data,configparser — Configuration file parser,File Formats
@anchor{library/netrc doc}@anchor{2306}@anchor{library/netrc module-netrc}@anchor{ba}@anchor{library/netrc netrc-netrc-file-processing}@anchor{2307}
@subsection @code{netrc} — netrc file processing


@geindex module; netrc

@strong{Source code:} Lib/netrc.py@footnote{https://github.com/python/cpython/tree/main/Lib/netrc.py}


__________________________________________________________________


The @ref{1137,,netrc} class parses and encapsulates the netrc file format used by
the Unix @code{ftp} program and other FTP clients.

@geindex netrc (class in netrc)
@anchor{library/netrc netrc netrc}@anchor{1137}
@deffn {Class} netrc.netrc ([file])

A @ref{1137,,netrc} instance or subclass instance encapsulates data from  a netrc
file.  The initialization argument, if present, specifies the file to parse.  If
no argument is given, the file @code{.netrc} in the user’s home directory –
as determined by @ref{520,,os.path.expanduser()} – will be read.  Otherwise,
a @ref{a37,,FileNotFoundError} exception will be raised.
Parse errors will raise @ref{2308,,NetrcParseError} with diagnostic
information including the file name, line number, and terminating token.
If no argument is specified on a POSIX system, the presence of passwords in
the @code{.netrc} file will raise a @ref{2308,,NetrcParseError} if the file
ownership or permissions are insecure (owned by a user other than the user
running the process, or accessible for read or write by any other user).
This implements security behavior equivalent to that of ftp and other
programs that use @code{.netrc}.


Changed in version 3.4: Added the POSIX permission check.



Changed in version 3.7: @ref{520,,os.path.expanduser()} is used to find the location of the
@code{.netrc} file when @emph{file} is not passed as argument.



Changed in version 3.10: @ref{ba,,netrc} try UTF-8 encoding before using locale specific
encoding.
The entry in the netrc file no longer needs to contain all tokens.  The missing
tokens’ value default to an empty string.  All the tokens and their values now
can contain arbitrary characters, like whitespace and non-ASCII characters.
If the login name is anonymous, it won’t trigger the security check.

@end deffn

@geindex NetrcParseError
@anchor{library/netrc netrc NetrcParseError}@anchor{2308}
@deffn {Exception} netrc.NetrcParseError

Exception raised by the @ref{1137,,netrc} class when syntactical errors are
encountered in source text.  Instances of this exception provide three
interesting attributes:  @code{msg} is a textual explanation of the error,
@code{filename} is the name of the source file, and @code{lineno} gives the
line number on which the error was found.
@end deffn

@menu
* netrc Objects:: 

@end menu

@node netrc Objects,,,netrc — netrc file processing
@anchor{library/netrc id1}@anchor{2309}@anchor{library/netrc netrc-objects}@anchor{230a}
@subsubsection netrc Objects


A @ref{1137,,netrc} instance has the following methods:

@geindex authenticators() (netrc.netrc method)
@anchor{library/netrc netrc netrc authenticators}@anchor{230b}
@deffn {Method} netrc.authenticators (host)

Return a 3-tuple @code{(login, account, password)} of authenticators for @emph{host}.
If the netrc file did not contain an entry for the given host, return the tuple
associated with the ‘default’ entry.  If neither matching host nor default entry
is available, return @code{None}.
@end deffn

@geindex __repr__() (netrc.netrc method)
@anchor{library/netrc netrc netrc __repr__}@anchor{230c}
@deffn {Method} netrc.__repr__ ()

Dump the class data as a string in the format of a netrc file. (This discards
comments and may reorder the entries.)
@end deffn

Instances of @ref{1137,,netrc} have public instance variables:

@geindex hosts (netrc.netrc attribute)
@anchor{library/netrc netrc netrc hosts}@anchor{230d}
@deffn {Attribute} netrc.hosts

Dictionary mapping host names to @code{(login, account, password)} tuples.  The
‘default’ entry, if any, is represented as a pseudo-host by that name.
@end deffn

@geindex macros (netrc.netrc attribute)
@anchor{library/netrc netrc netrc macros}@anchor{230e}
@deffn {Attribute} netrc.macros

Dictionary mapping macro names to string lists.
@end deffn

@node xdrlib — Encode and decode XDR data,plistlib — Generate and parse Apple plist files,netrc — netrc file processing,File Formats
@anchor{library/xdrlib doc}@anchor{230f}@anchor{library/xdrlib module-xdrlib}@anchor{139}@anchor{library/xdrlib xdrlib-encode-and-decode-xdr-data}@anchor{2310}
@subsection @code{xdrlib} — Encode and decode XDR data


@geindex module; xdrlib

@strong{Source code:} Lib/xdrlib.py@footnote{https://github.com/python/cpython/tree/main/Lib/xdrlib.py}

@geindex XDR
@geindex External Data Representation


__________________________________________________________________


The @ref{139,,xdrlib} module supports the External Data Representation Standard as
described in 
@geindex RFC; RFC 1014
RFC 1014@footnote{https://tools.ietf.org/html/rfc1014.html}, written by Sun Microsystems, Inc. June 1987.  It
supports most of the data types described in the RFC.

The @ref{139,,xdrlib} module defines two classes, one for packing variables into XDR
representation, and another for unpacking from XDR representation.  There are
also two exception classes.

@geindex Packer (class in xdrlib)
@anchor{library/xdrlib xdrlib Packer}@anchor{2311}
@deffn {Class} xdrlib.Packer

@ref{2311,,Packer} is the class for packing data into XDR representation. The
@ref{2311,,Packer} class is instantiated with no arguments.
@end deffn

@geindex Unpacker (class in xdrlib)
@anchor{library/xdrlib xdrlib Unpacker}@anchor{2312}
@deffn {Class} xdrlib.Unpacker (data)

@code{Unpacker} is the complementary class which unpacks XDR data values from a
string buffer.  The input buffer is given as @emph{data}.
@end deffn


@subsubheading See also



@table @asis

@geindex RFC; RFC 1014

@item RFC 1014@footnote{https://tools.ietf.org/html/rfc1014.html} - XDR: External Data Representation Standard

This RFC defined the encoding of data which was XDR at the time this module was
originally written.  It has apparently been obsoleted by 
@geindex RFC; RFC 1832
RFC 1832@footnote{https://tools.ietf.org/html/rfc1832.html}.

@geindex RFC; RFC 1832

@item RFC 1832@footnote{https://tools.ietf.org/html/rfc1832.html} - XDR: External Data Representation Standard

Newer RFC that provides a revised definition of XDR.
@end table


@menu
* Packer Objects:: 
* Unpacker Objects:: 
* Exceptions: Exceptions<7>. 

@end menu

@node Packer Objects,Unpacker Objects,,xdrlib — Encode and decode XDR data
@anchor{library/xdrlib packer-objects}@anchor{2313}@anchor{library/xdrlib xdr-packer-objects}@anchor{2314}
@subsubsection Packer Objects


@ref{2311,,Packer} instances have the following methods:

@geindex get_buffer() (xdrlib.Packer method)
@anchor{library/xdrlib xdrlib Packer get_buffer}@anchor{2315}
@deffn {Method} Packer.get_buffer ()

Returns the current pack buffer as a string.
@end deffn

@geindex reset() (xdrlib.Packer method)
@anchor{library/xdrlib xdrlib Packer reset}@anchor{2316}
@deffn {Method} Packer.reset ()

Resets the pack buffer to the empty string.
@end deffn

In general, you can pack any of the most common XDR data types by calling the
appropriate @code{pack_type()} method.  Each method takes a single argument, the
value to pack.  The following simple data type packing methods are supported:
@code{pack_uint()}, @code{pack_int()}, @code{pack_enum()}, @code{pack_bool()},
@code{pack_uhyper()}, and @code{pack_hyper()}.

@geindex pack_float() (xdrlib.Packer method)
@anchor{library/xdrlib xdrlib Packer pack_float}@anchor{2317}
@deffn {Method} Packer.pack_float (value)

Packs the single-precision floating point number @emph{value}.
@end deffn

@geindex pack_double() (xdrlib.Packer method)
@anchor{library/xdrlib xdrlib Packer pack_double}@anchor{2318}
@deffn {Method} Packer.pack_double (value)

Packs the double-precision floating point number @emph{value}.
@end deffn

The following methods support packing strings, bytes, and opaque data:

@geindex pack_fstring() (xdrlib.Packer method)
@anchor{library/xdrlib xdrlib Packer pack_fstring}@anchor{2319}
@deffn {Method} Packer.pack_fstring (n, s)

Packs a fixed length string, @emph{s}.  @emph{n} is the length of the string but it is
@emph{not} packed into the data buffer.  The string is padded with null bytes if
necessary to guaranteed 4 byte alignment.
@end deffn

@geindex pack_fopaque() (xdrlib.Packer method)
@anchor{library/xdrlib xdrlib Packer pack_fopaque}@anchor{231a}
@deffn {Method} Packer.pack_fopaque (n, data)

Packs a fixed length opaque data stream, similarly to @ref{2319,,pack_fstring()}.
@end deffn

@geindex pack_string() (xdrlib.Packer method)
@anchor{library/xdrlib xdrlib Packer pack_string}@anchor{231b}
@deffn {Method} Packer.pack_string (s)

Packs a variable length string, @emph{s}.  The length of the string is first packed
as an unsigned integer, then the string data is packed with
@ref{2319,,pack_fstring()}.
@end deffn

@geindex pack_opaque() (xdrlib.Packer method)
@anchor{library/xdrlib xdrlib Packer pack_opaque}@anchor{231c}
@deffn {Method} Packer.pack_opaque (data)

Packs a variable length opaque data string, similarly to @ref{231b,,pack_string()}.
@end deffn

@geindex pack_bytes() (xdrlib.Packer method)
@anchor{library/xdrlib xdrlib Packer pack_bytes}@anchor{231d}
@deffn {Method} Packer.pack_bytes (bytes)

Packs a variable length byte stream, similarly to @ref{231b,,pack_string()}.
@end deffn

The following methods support packing arrays and lists:

@geindex pack_list() (xdrlib.Packer method)
@anchor{library/xdrlib xdrlib Packer pack_list}@anchor{231e}
@deffn {Method} Packer.pack_list (list, pack_item)

Packs a @emph{list} of homogeneous items.  This method is useful for lists with an
indeterminate size; i.e. the size is not available until the entire list has
been walked.  For each item in the list, an unsigned integer @code{1} is packed
first, followed by the data value from the list.  @emph{pack_item} is the function
that is called to pack the individual item.  At the end of the list, an unsigned
integer @code{0} is packed.

For example, to pack a list of integers, the code might appear like this:

@example
import xdrlib
p = xdrlib.Packer()
p.pack_list([1, 2, 3], p.pack_int)
@end example
@end deffn

@geindex pack_farray() (xdrlib.Packer method)
@anchor{library/xdrlib xdrlib Packer pack_farray}@anchor{231f}
@deffn {Method} Packer.pack_farray (n, array, pack_item)

Packs a fixed length list (@emph{array}) of homogeneous items.  @emph{n} is the length of
the list; it is @emph{not} packed into the buffer, but a @ref{17c,,ValueError} exception
is raised if @code{len(array)} is not equal to @emph{n}.  As above, @emph{pack_item} is the
function used to pack each element.
@end deffn

@geindex pack_array() (xdrlib.Packer method)
@anchor{library/xdrlib xdrlib Packer pack_array}@anchor{2320}
@deffn {Method} Packer.pack_array (list, pack_item)

Packs a variable length @emph{list} of homogeneous items.  First, the length of the
list is packed as an unsigned integer, then each element is packed as in
@ref{231f,,pack_farray()} above.
@end deffn

@node Unpacker Objects,Exceptions<7>,Packer Objects,xdrlib — Encode and decode XDR data
@anchor{library/xdrlib unpacker-objects}@anchor{2321}@anchor{library/xdrlib xdr-unpacker-objects}@anchor{2322}
@subsubsection Unpacker Objects


The @ref{2312,,Unpacker} class offers the following methods:

@geindex reset() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker reset}@anchor{2323}
@deffn {Method} Unpacker.reset (data)

Resets the string buffer with the given @emph{data}.
@end deffn

@geindex get_position() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker get_position}@anchor{2324}
@deffn {Method} Unpacker.get_position ()

Returns the current unpack position in the data buffer.
@end deffn

@geindex set_position() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker set_position}@anchor{2325}
@deffn {Method} Unpacker.set_position (position)

Sets the data buffer unpack position to @emph{position}.  You should be careful about
using @ref{2324,,get_position()} and @ref{2325,,set_position()}.
@end deffn

@geindex get_buffer() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker get_buffer}@anchor{2326}
@deffn {Method} Unpacker.get_buffer ()

Returns the current unpack data buffer as a string.
@end deffn

@geindex done() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker done}@anchor{2327}
@deffn {Method} Unpacker.done ()

Indicates unpack completion.  Raises an @ref{2328,,Error} exception if all of the
data has not been unpacked.
@end deffn

In addition, every data type that can be packed with a @ref{2311,,Packer}, can be
unpacked with an @ref{2312,,Unpacker}.  Unpacking methods are of the form
@code{unpack_type()}, and take no arguments.  They return the unpacked object.

@geindex unpack_float() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker unpack_float}@anchor{2329}
@deffn {Method} Unpacker.unpack_float ()

Unpacks a single-precision floating point number.
@end deffn

@geindex unpack_double() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker unpack_double}@anchor{232a}
@deffn {Method} Unpacker.unpack_double ()

Unpacks a double-precision floating point number, similarly to
@ref{2329,,unpack_float()}.
@end deffn

In addition, the following methods unpack strings, bytes, and opaque data:

@geindex unpack_fstring() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker unpack_fstring}@anchor{232b}
@deffn {Method} Unpacker.unpack_fstring (n)

Unpacks and returns a fixed length string.  @emph{n} is the number of characters
expected.  Padding with null bytes to guaranteed 4 byte alignment is assumed.
@end deffn

@geindex unpack_fopaque() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker unpack_fopaque}@anchor{232c}
@deffn {Method} Unpacker.unpack_fopaque (n)

Unpacks and returns a fixed length opaque data stream, similarly to
@ref{232b,,unpack_fstring()}.
@end deffn

@geindex unpack_string() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker unpack_string}@anchor{232d}
@deffn {Method} Unpacker.unpack_string ()

Unpacks and returns a variable length string.  The length of the string is first
unpacked as an unsigned integer, then the string data is unpacked with
@ref{232b,,unpack_fstring()}.
@end deffn

@geindex unpack_opaque() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker unpack_opaque}@anchor{232e}
@deffn {Method} Unpacker.unpack_opaque ()

Unpacks and returns a variable length opaque data string, similarly to
@ref{232d,,unpack_string()}.
@end deffn

@geindex unpack_bytes() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker unpack_bytes}@anchor{232f}
@deffn {Method} Unpacker.unpack_bytes ()

Unpacks and returns a variable length byte stream, similarly to
@ref{232d,,unpack_string()}.
@end deffn

The following methods support unpacking arrays and lists:

@geindex unpack_list() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker unpack_list}@anchor{2330}
@deffn {Method} Unpacker.unpack_list (unpack_item)

Unpacks and returns a list of homogeneous items.  The list is unpacked one
element at a time by first unpacking an unsigned integer flag.  If the flag is
@code{1}, then the item is unpacked and appended to the list.  A flag of @code{0}
indicates the end of the list.  @emph{unpack_item} is the function that is called to
unpack the items.
@end deffn

@geindex unpack_farray() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker unpack_farray}@anchor{2331}
@deffn {Method} Unpacker.unpack_farray (n, unpack_item)

Unpacks and returns (as a list) a fixed length array of homogeneous items.  @emph{n}
is number of list elements to expect in the buffer. As above, @emph{unpack_item} is
the function used to unpack each element.
@end deffn

@geindex unpack_array() (xdrlib.Unpacker method)
@anchor{library/xdrlib xdrlib Unpacker unpack_array}@anchor{2332}
@deffn {Method} Unpacker.unpack_array (unpack_item)

Unpacks and returns a variable length @emph{list} of homogeneous items. First, the
length of the list is unpacked as an unsigned integer, then each element is
unpacked as in @ref{2331,,unpack_farray()} above.
@end deffn

@node Exceptions<7>,,Unpacker Objects,xdrlib — Encode and decode XDR data
@anchor{library/xdrlib exceptions}@anchor{2333}@anchor{library/xdrlib xdr-exceptions}@anchor{2334}
@subsubsection Exceptions


Exceptions in this module are coded as class instances:

@geindex Error
@anchor{library/xdrlib xdrlib Error}@anchor{2328}
@deffn {Exception} xdrlib.Error

The base exception class.  @ref{2328,,Error} has a single public attribute
@code{msg} containing the description of the error.
@end deffn

@geindex ConversionError
@anchor{library/xdrlib xdrlib ConversionError}@anchor{2335}
@deffn {Exception} xdrlib.ConversionError

Class derived from @ref{2328,,Error}.  Contains no additional instance variables.
@end deffn

Here is an example of how you would catch one of these exceptions:

@example
import xdrlib
p = xdrlib.Packer()
try:
    p.pack_double(8.01)
except xdrlib.ConversionError as instance:
    print('packing the double failed:', instance.msg)
@end example

@node plistlib — Generate and parse Apple plist files,,xdrlib — Encode and decode XDR data,File Formats
@anchor{library/plistlib doc}@anchor{2336}@anchor{library/plistlib module-plistlib}@anchor{ca}@anchor{library/plistlib plistlib-generate-and-parse-apple-plist-files}@anchor{2337}
@subsection @code{plistlib} — Generate and parse Apple @code{.plist} files


@geindex module; plistlib

@c (harvested from docstrings in the original file)

@strong{Source code:} Lib/plistlib.py@footnote{https://github.com/python/cpython/tree/main/Lib/plistlib.py}

@geindex plist; file
@geindex property list


__________________________________________________________________


This module provides an interface for reading and writing the “property list”
files used by Apple, primarily on macOS and iOS. This module supports both binary
and XML plist files.

The property list (@code{.plist}) file format is a simple serialization supporting
basic object types, like dictionaries, lists, numbers and strings.  Usually the
top level object is a dictionary.

To write out and to parse a plist file, use the @ref{451,,dump()} and
@ref{44f,,load()} functions.

To work with plist data in bytes objects, use @ref{452,,dumps()}
and @ref{450,,loads()}.

Values can be strings, integers, floats, booleans, tuples, lists, dictionaries
(but only with string keys), @ref{2f0,,bytes}, @ref{2f1,,bytearray}
or @ref{363,,datetime.datetime} objects.


Changed in version 3.4: New API, old API deprecated.  Support for binary format plists added.



Changed in version 3.8: Support added for reading and writing @ref{530,,UID} tokens in binary plists as used
by NSKeyedArchiver and NSKeyedUnarchiver.



Changed in version 3.9: Old API removed.



@subsubheading See also



@table @asis

@item PList manual page@footnote{https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/PropertyLists/}

Apple’s documentation of the file format.
@end table


This module defines the following functions:

@geindex load() (in module plistlib)
@anchor{library/plistlib plistlib load}@anchor{44f}
@deffn {Function} plistlib.load (fp, *, fmt=None, dict_type=dict)

Read a plist file. @emph{fp} should be a readable and binary file object.
Return the unpacked root object (which usually is a
dictionary).

The @emph{fmt} is the format of the file and the following values are valid:


@itemize *

@item 
@ref{182,,None}: Autodetect the file format

@item 
@ref{afa,,FMT_XML}: XML file format

@item 
@ref{afb,,FMT_BINARY}: Binary plist format
@end itemize

The @emph{dict_type} is the type used for dictionaries that are read from the
plist file.

XML data for the @ref{afa,,FMT_XML} format is parsed using the Expat parser
from @ref{13f,,xml.parsers.expat} – see its documentation for possible
exceptions on ill-formed XML.  Unknown elements will simply be ignored
by the plist parser.

The parser for the binary format raises @code{InvalidFileException}
when the file cannot be parsed.


New in version 3.4.

@end deffn

@geindex loads() (in module plistlib)
@anchor{library/plistlib plistlib loads}@anchor{450}
@deffn {Function} plistlib.loads (data, *, fmt=None, dict_type=dict)

Load a plist from a bytes object. See @ref{44f,,load()} for an explanation of
the keyword arguments.


New in version 3.4.

@end deffn

@geindex dump() (in module plistlib)
@anchor{library/plistlib plistlib dump}@anchor{451}
@deffn {Function} plistlib.dump (value, fp, *, fmt=FMT_XML, sort_keys=True, skipkeys=False)

Write @emph{value} to a plist file. @emph{Fp} should be a writable, binary
file object.

The @emph{fmt} argument specifies the format of the plist file and can be
one of the following values:


@itemize *

@item 
@ref{afa,,FMT_XML}: XML formatted plist file

@item 
@ref{afb,,FMT_BINARY}: Binary formatted plist file
@end itemize

When @emph{sort_keys} is true (the default) the keys for dictionaries will be
written to the plist in sorted order, otherwise they will be written in
the iteration order of the dictionary.

When @emph{skipkeys} is false (the default) the function raises @ref{162,,TypeError}
when a key of a dictionary is not a string, otherwise such keys are skipped.

A @ref{162,,TypeError} will be raised if the object is of an unsupported type or
a container that contains objects of unsupported types.

An @ref{347,,OverflowError} will be raised for integer values that cannot
be represented in (binary) plist files.


New in version 3.4.

@end deffn

@geindex dumps() (in module plistlib)
@anchor{library/plistlib plistlib dumps}@anchor{452}
@deffn {Function} plistlib.dumps (value, *, fmt=FMT_XML, sort_keys=True, skipkeys=False)

Return @emph{value} as a plist-formatted bytes object. See
the documentation for @ref{451,,dump()} for an explanation of the keyword
arguments of this function.


New in version 3.4.

@end deffn

The following classes are available:

@geindex UID (class in plistlib)
@anchor{library/plistlib plistlib UID}@anchor{530}
@deffn {Class} plistlib.UID (data)

Wraps an @ref{1b1,,int}.  This is used when reading or writing NSKeyedArchiver
encoded data, which contains UID (see PList manual).

It has one attribute, @code{data}, which can be used to retrieve the int value
of the UID.  @code{data} must be in the range @code{0 <= data < 2**64}.


New in version 3.8.

@end deffn

The following constants are available:

@geindex FMT_XML (in module plistlib)
@anchor{library/plistlib plistlib FMT_XML}@anchor{afa}
@deffn {Data} plistlib.FMT_XML

The XML format for plist files.


New in version 3.4.

@end deffn

@geindex FMT_BINARY (in module plistlib)
@anchor{library/plistlib plistlib FMT_BINARY}@anchor{afb}
@deffn {Data} plistlib.FMT_BINARY

The binary format for plist files


New in version 3.4.

@end deffn

@menu
* Examples: Examples<10>. 

@end menu

@node Examples<10>,,,plistlib — Generate and parse Apple plist files
@anchor{library/plistlib examples}@anchor{2338}
@subsubsection Examples


Generating a plist:

@example
pl = dict(
    aString = "Doodah",
    aList = ["A", "B", 12, 32.1, [1, 2, 3]],
    aFloat = 0.1,
    anInt = 728,
    aDict = dict(
        anotherString = "<hello & hi there!>",
        aThirdString = "M\xe4ssig, Ma\xdf",
        aTrueValue = True,
        aFalseValue = False,
    ),
    someData = b"<binary gunk>",
    someMoreData = b"<lots of binary gunk>" * 10,
    aDate = datetime.datetime.fromtimestamp(time.mktime(time.gmtime())),
)
with open(fileName, 'wb') as fp:
    dump(pl, fp)
@end example

Parsing a plist:

@example
with open(fileName, 'rb') as fp:
    pl = load(fp)
print(pl["aKey"])
@end example

@node Cryptographic Services,Generic Operating System Services,File Formats,The Python Standard Library
@anchor{library/crypto doc}@anchor{2339}@anchor{library/crypto crypto}@anchor{233a}@anchor{library/crypto cryptographic-services}@anchor{233b}
@section Cryptographic Services


@geindex cryptography

The modules described in this chapter implement various algorithms of a
cryptographic nature.  They are available at the discretion of the installation.
On Unix systems, the @ref{27,,crypt} module may also be available.
Here’s an overview:

@menu
* hashlib — Secure hashes and message digests:: 
* hmac — Keyed-Hashing for Message Authentication:: 
* secrets — Generate secure random numbers for managing secrets:: 

@end menu

@node hashlib — Secure hashes and message digests,hmac — Keyed-Hashing for Message Authentication,,Cryptographic Services
@anchor{library/hashlib doc}@anchor{233c}@anchor{library/hashlib hashlib-secure-hashes-and-message-digests}@anchor{233d}@anchor{library/hashlib module-hashlib}@anchor{88}
@subsection @code{hashlib} — Secure hashes and message digests


@geindex module; hashlib

@strong{Source code:} Lib/hashlib.py@footnote{https://github.com/python/cpython/tree/main/Lib/hashlib.py}

@geindex message digest@comma{} MD5
@geindex secure hash algorithm@comma{} SHA1@comma{} SHA224@comma{} SHA256@comma{} SHA384@comma{} SHA512

@c import hashlib


__________________________________________________________________


This module implements a common interface to many different secure hash and
message digest algorithms.  Included are the FIPS secure hash algorithms SHA1,
SHA224, SHA256, SHA384, and SHA512 (defined in FIPS 180-2) as well as RSA’s MD5
algorithm (defined in internet 
@geindex RFC; RFC 1321
RFC 1321@footnote{https://tools.ietf.org/html/rfc1321.html}).  The terms “secure hash” and
“message digest” are interchangeable.  Older algorithms were called message
digests.  The modern term is secure hash.

@cartouche
@quotation Note 
If you want the adler32 or crc32 hash functions, they are available in
the @ref{14b,,zlib} module.
@end quotation
@end cartouche

@cartouche
@quotation Warning 
Some algorithms have known hash collision weaknesses, refer to the “See
also” section at the end.
@end quotation
@end cartouche

@menu
* Hash algorithms:: 
* SHAKE variable length digests:: 
* Key derivation:: 
* BLAKE2:: 

@end menu

@node Hash algorithms,SHAKE variable length digests,,hashlib — Secure hashes and message digests
@anchor{library/hashlib hash-algorithms}@anchor{233e}@anchor{library/hashlib id1}@anchor{233f}
@subsubsection Hash algorithms


There is one constructor method named for each type of @emph{hash}.  All return
a hash object with the same simple interface. For example: use @code{sha256()} to
create a SHA-256 hash object. You can now feed this object with @ref{87e,,bytes-like objects} (normally @ref{2f0,,bytes}) using the @code{update()} method.
At any point you can ask it for the @emph{digest} of the
concatenation of the data fed to it so far using the @code{digest()} or
@code{hexdigest()} methods.

@cartouche
@quotation Note 
For better multithreading performance, the Python @ref{385,,GIL} is released for
data larger than 2047 bytes at object creation or on update.
@end quotation
@end cartouche

@cartouche
@quotation Note 
Feeding string objects into @code{update()} is not supported, as hashes work
on bytes, not on characters.
@end quotation
@end cartouche

@geindex OpenSSL; (use in module hashlib)

Constructors for hash algorithms that are always present in this module are
@code{sha1()}, @code{sha224()}, @code{sha256()}, @code{sha384()},
@code{sha512()}, @ref{7ee,,blake2b()}, and @ref{7ef,,blake2s()}.
@code{md5()} is normally available as well, though it
may be missing or blocked if you are using a rare “FIPS compliant” build of Python.
Additional algorithms may also be available depending upon the OpenSSL
library that Python uses on your platform. On most platforms the
@code{sha3_224()}, @code{sha3_256()}, @code{sha3_384()}, @code{sha3_512()},
@code{shake_128()}, @code{shake_256()} are also available.


New in version 3.6: SHA3 (Keccak) and SHAKE constructors @code{sha3_224()}, @code{sha3_256()},
@code{sha3_384()}, @code{sha3_512()}, @code{shake_128()}, @code{shake_256()}.



New in version 3.6: @ref{7ee,,blake2b()} and @ref{7ef,,blake2s()} were added.

@anchor{library/hashlib hashlib-usedforsecurity}@anchor{2340}

Changed in version 3.9: All hashlib constructors take a keyword-only argument @emph{usedforsecurity}
with default value @code{True}. A false value allows the use of insecure and
blocked hashing algorithms in restricted environments. @code{False} indicates
that the hashing algorithm is not used in a security context, e.g. as a
non-cryptographic one-way compression function.

Hashlib now uses SHA3 and SHAKE from OpenSSL 1.1.1 and newer.


For example, to obtain the digest of the byte string @code{b'Nobody inspects the
spammish repetition'}:

@example
>>> import hashlib
>>> m = hashlib.sha256()
>>> m.update(b"Nobody inspects")
>>> m.update(b" the spammish repetition")
>>> m.digest()
b'\x03\x1e\xdd@}Ae\x15\x93\xc5\xfe\\\x00o\xa5u+7\xfd\xdf\xf7\xbcN\x84:\xa6\xaf\x0c\x95\x0fK\x94\x06'
>>> m.digest_size
32
>>> m.block_size
64
@end example

More condensed:

@example
>>> hashlib.sha224(b"Nobody inspects the spammish repetition").hexdigest()
'a4337bc45a8fc544c03f52dc550cd6e1e87021bc896588bd79e901e2'
@end example

@geindex new() (in module hashlib)
@anchor{library/hashlib hashlib new}@anchor{11cb}
@deffn {Function} hashlib.new (name[, data], *, usedforsecurity=True)

Is a generic constructor that takes the string @emph{name} of the desired
algorithm as its first parameter.  It also exists to allow access to the
above listed hashes as well as any other algorithms that your OpenSSL
library may offer.  The named constructors are much faster than @ref{11cb,,new()}
and should be preferred.
@end deffn

Using @ref{11cb,,new()} with an algorithm provided by OpenSSL:

@example
>>> h = hashlib.new('sha256')
>>> h.update(b"Nobody inspects the spammish repetition")
>>> h.hexdigest()
'031edd7d41651593c5fe5c006fa5752b37fddff7bc4e843aa6af0c950f4b9406'
@end example

Hashlib provides the following constant attributes:

@geindex algorithms_guaranteed (in module hashlib)
@anchor{library/hashlib hashlib algorithms_guaranteed}@anchor{efc}
@deffn {Data} hashlib.algorithms_guaranteed

A set containing the names of the hash algorithms guaranteed to be supported
by this module on all platforms.  Note that ‘md5’ is in this list despite
some upstream vendors offering an odd “FIPS compliant” Python build that
excludes it.


New in version 3.2.

@end deffn

@geindex algorithms_available (in module hashlib)
@anchor{library/hashlib hashlib algorithms_available}@anchor{efd}
@deffn {Data} hashlib.algorithms_available

A set containing the names of the hash algorithms that are available in the
running Python interpreter.  These names will be recognized when passed to
@ref{11cb,,new()}.  @ref{efc,,algorithms_guaranteed} will always be a subset.  The
same algorithm may appear multiple times in this set under different names
(thanks to OpenSSL).


New in version 3.2.

@end deffn

The following values are provided as constant attributes of the hash objects
returned by the constructors:

@geindex hash.digest_size (in module hashlib)
@anchor{library/hashlib hashlib hash digest_size}@anchor{2341}
@deffn {Data} hash.digest_size

The size of the resulting hash in bytes.
@end deffn

@geindex hash.block_size (in module hashlib)
@anchor{library/hashlib hashlib hash block_size}@anchor{2342}
@deffn {Data} hash.block_size

The internal block size of the hash algorithm in bytes.
@end deffn

A hash object has the following attributes:

@geindex name (hashlib.hash attribute)
@anchor{library/hashlib hashlib hash name}@anchor{ac5}
@deffn {Attribute} hash.name

The canonical name of this hash, always lowercase and always suitable as a
parameter to @ref{11cb,,new()} to create another hash of this type.


Changed in version 3.4: The name attribute has been present in CPython since its inception, but
until Python 3.4 was not formally specified, so may not exist on some
platforms.

@end deffn

A hash object has the following methods:

@geindex update() (hashlib.hash method)
@anchor{library/hashlib hashlib hash update}@anchor{2343}
@deffn {Method} hash.update (data)

Update the hash object with the @ref{87e,,bytes-like object}.
Repeated calls are equivalent to a single call with the
concatenation of all the arguments: @code{m.update(a); m.update(b)} is
equivalent to @code{m.update(a+b)}.


Changed in version 3.1: The Python GIL is released to allow other threads to run while hash
updates on data larger than 2047 bytes is taking place when using hash
algorithms supplied by OpenSSL.

@end deffn

@geindex digest() (hashlib.hash method)
@anchor{library/hashlib hashlib hash digest}@anchor{2344}
@deffn {Method} hash.digest ()

Return the digest of the data passed to the @ref{2343,,update()} method so far.
This is a bytes object of size @ref{2341,,digest_size} which may contain bytes in
the whole range from 0 to 255.
@end deffn

@geindex hexdigest() (hashlib.hash method)
@anchor{library/hashlib hashlib hash hexdigest}@anchor{2345}
@deffn {Method} hash.hexdigest ()

Like @ref{2344,,digest()} except the digest is returned as a string object of
double length, containing only hexadecimal digits.  This may be used to
exchange the value safely in email or other non-binary environments.
@end deffn

@geindex copy() (hashlib.hash method)
@anchor{library/hashlib hashlib hash copy}@anchor{2346}
@deffn {Method} hash.copy ()

Return a copy (“clone”) of the hash object.  This can be used to efficiently
compute the digests of data sharing a common initial substring.
@end deffn

@node SHAKE variable length digests,Key derivation,Hash algorithms,hashlib — Secure hashes and message digests
@anchor{library/hashlib shake-variable-length-digests}@anchor{2347}
@subsubsection SHAKE variable length digests


The @code{shake_128()} and @code{shake_256()} algorithms provide variable
length digests with length_in_bits//2 up to 128 or 256 bits of security.
As such, their digest methods require a length. Maximum length is not limited
by the SHAKE algorithm.

@geindex digest() (hashlib.shake method)
@anchor{library/hashlib hashlib shake digest}@anchor{12ed}
@deffn {Method} shake.digest (length)

Return the digest of the data passed to the @code{update()} method so far.
This is a bytes object of size @emph{length} which may contain bytes in
the whole range from 0 to 255.
@end deffn

@geindex hexdigest() (hashlib.shake method)
@anchor{library/hashlib hashlib shake hexdigest}@anchor{12ee}
@deffn {Method} shake.hexdigest (length)

Like @ref{12ed,,digest()} except the digest is returned as a string object of
double length, containing only hexadecimal digits.  This may be used to
exchange the value safely in email or other non-binary environments.
@end deffn

@node Key derivation,BLAKE2,SHAKE variable length digests,hashlib — Secure hashes and message digests
@anchor{library/hashlib key-derivation}@anchor{2348}
@subsubsection Key derivation


Key derivation and key stretching algorithms are designed for secure password
hashing. Naive algorithms such as @code{sha1(password)} are not resistant against
brute-force attacks. A good password hashing function must be tunable, slow, and
include a salt@footnote{https://en.wikipedia.org/wiki/Salt_%28cryptography%29}.

@geindex pbkdf2_hmac() (in module hashlib)
@anchor{library/hashlib hashlib pbkdf2_hmac}@anchor{28b}
@deffn {Function} hashlib.pbkdf2_hmac (hash_name, password, salt, iterations, dklen=None)

The function provides PKCS#5 password-based key derivation function 2. It
uses HMAC as pseudorandom function.

The string @emph{hash_name} is the desired name of the hash digest algorithm for
HMAC, e.g. ‘sha1’ or ‘sha256’. @emph{password} and @emph{salt} are interpreted as
buffers of bytes. Applications and libraries should limit @emph{password} to
a sensible length (e.g. 1024). @emph{salt} should be about 16 or more bytes from
a proper source, e.g. @ref{17f,,os.urandom()}.

The number of @emph{iterations} should be chosen based on the hash algorithm and
computing power. As of 2022, hundreds of thousands of iterations of SHA-256
are suggested. For rationale as to why and how to choose what is best for
your application, read @emph{Appendix A.2.2} of NIST-SP-800-132@footnote{https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-132.pdf}. The answers
on the stackexchange pbkdf2 iterations question@footnote{https://security.stackexchange.com/questions/3959/recommended-of-iterations-when-using-pbkdf2-sha256/} explain in detail.

@emph{dklen} is the length of the derived key. If @emph{dklen} is @code{None} then the
digest size of the hash algorithm @emph{hash_name} is used, e.g. 64 for SHA-512.

@example
>>> from hashlib import pbkdf2_hmac
>>> our_app_iters = 500_000  # Application specific, read above.
>>> dk = pbkdf2_hmac('sha256', b'password', b'bad salt'*2, our_app_iters)
>>> dk.hex()
'15530bba69924174860db778f2c6f8104d3aaf9d26241840c8c4a641c8d000a9'
@end example


New in version 3.4.


@cartouche
@quotation Note 
A fast implementation of @emph{pbkdf2_hmac} is available with OpenSSL.  The
Python implementation uses an inline version of @ref{8a,,hmac}. It is about
three times slower and doesn’t release the GIL.
@end quotation
@end cartouche


Deprecated since version 3.10: Slow Python implementation of @emph{pbkdf2_hmac} is deprecated. In the
future the function will only be available when Python is compiled
with OpenSSL.

@end deffn

@geindex scrypt() (in module hashlib)
@anchor{library/hashlib hashlib scrypt}@anchor{77c}
@deffn {Function} hashlib.scrypt (password, *, salt, n, r, p, maxmem=0, dklen=64)

The function provides scrypt password-based key derivation function as
defined in 
@geindex RFC; RFC 7914
RFC 7914@footnote{https://tools.ietf.org/html/rfc7914.html}.

@emph{password} and @emph{salt} must be @ref{87e,,bytes-like objects}.  Applications and libraries should limit @emph{password}
to a sensible length (e.g. 1024).  @emph{salt} should be about 16 or more
bytes from a proper source, e.g. @ref{17f,,os.urandom()}.

@emph{n} is the CPU/Memory cost factor, @emph{r} the block size, @emph{p} parallelization
factor and @emph{maxmem} limits memory (OpenSSL 1.1.0 defaults to 32 MiB).
@emph{dklen} is the length of the derived key.


New in version 3.6.

@end deffn

@node BLAKE2,,Key derivation,hashlib — Secure hashes and message digests
@anchor{library/hashlib blake2}@anchor{2349}
@subsubsection BLAKE2


@geindex blake2b@comma{} blake2s

BLAKE2@footnote{https://blake2.net} is a cryptographic hash function defined in 
@geindex RFC; RFC 7693
RFC 7693@footnote{https://tools.ietf.org/html/rfc7693.html} that comes in two
flavors:


@itemize *

@item 
@strong{BLAKE2b}, optimized for 64-bit platforms and produces digests of any size
between 1 and 64 bytes,

@item 
@strong{BLAKE2s}, optimized for 8- to 32-bit platforms and produces digests of any
size between 1 and 32 bytes.
@end itemize

BLAKE2 supports @strong{keyed mode} (a faster and simpler replacement for HMAC@footnote{https://en.wikipedia.org/wiki/Hash-based_message_authentication_code}),
@strong{salted hashing}, @strong{personalization}, and @strong{tree hashing}.

Hash objects from this module follow the API of standard library’s
@ref{88,,hashlib} objects.

@menu
* Creating hash objects:: 
* Constants: Constants<5>. 
* Examples: Examples<11>. 
* Credits:: 

@end menu

@node Creating hash objects,Constants<5>,,BLAKE2
@anchor{library/hashlib creating-hash-objects}@anchor{234a}
@subsubsection Creating hash objects


New hash objects are created by calling constructor functions:

@geindex blake2b() (in module hashlib)
@anchor{library/hashlib hashlib blake2b}@anchor{7ee}
@deffn {Function} hashlib.blake2b (data=b'@w{'}, *, digest_size=64, key=b'@w{'}, salt=b'@w{'}, person=b'@w{'}, fanout=1, depth=1, leaf_size=0, node_offset=0, node_depth=0, inner_size=0, last_node=False, usedforsecurity=True)
@end deffn

@geindex blake2s() (in module hashlib)
@anchor{library/hashlib hashlib blake2s}@anchor{7ef}
@deffn {Function} hashlib.blake2s (data=b'@w{'}, *, digest_size=32, key=b'@w{'}, salt=b'@w{'}, person=b'@w{'}, fanout=1, depth=1, leaf_size=0, node_offset=0, node_depth=0, inner_size=0, last_node=False, usedforsecurity=True)
@end deffn

These functions return the corresponding hash objects for calculating
BLAKE2b or BLAKE2s. They optionally take these general parameters:


@itemize *

@item 
@emph{data}: initial chunk of data to hash, which must be
@ref{87e,,bytes-like object}.  It can be passed only as positional argument.

@item 
@emph{digest_size}: size of output digest in bytes.

@item 
@emph{key}: key for keyed hashing (up to 64 bytes for BLAKE2b, up to 32 bytes for
BLAKE2s).

@item 
@emph{salt}: salt for randomized hashing (up to 16 bytes for BLAKE2b, up to 8
bytes for BLAKE2s).

@item 
@emph{person}: personalization string (up to 16 bytes for BLAKE2b, up to 8 bytes
for BLAKE2s).
@end itemize

The following table shows limits for general parameters (in bytes):


@multitable {xxxxxxxxx} {xxxxxxxxxxxxx} {xxxxxxxxxx} {xxxxxxxxxxx} {xxxxxxxxxxxxx} 
@headitem

Hash

@tab

digest_size

@tab

len(key)

@tab

len(salt)

@tab

len(person)

@item

BLAKE2b

@tab

64

@tab

64

@tab

16

@tab

16

@item

BLAKE2s

@tab

32

@tab

32

@tab

8

@tab

8

@end multitable


@cartouche
@quotation Note 
BLAKE2 specification defines constant lengths for salt and personalization
parameters, however, for convenience, this implementation accepts byte
strings of any size up to the specified length. If the length of the
parameter is less than specified, it is padded with zeros, thus, for
example, @code{b'salt'} and @code{b'salt\x00'} is the same value. (This is not
the case for @emph{key}.)
@end quotation
@end cartouche

These sizes are available as module @ref{234b,,constants} described below.

Constructor functions also accept the following tree hashing parameters:


@itemize *

@item 
@emph{fanout}: fanout (0 to 255, 0 if unlimited, 1 in sequential mode).

@item 
@emph{depth}: maximal depth of tree (1 to 255, 255 if unlimited, 1 in
sequential mode).

@item 
@emph{leaf_size}: maximal byte length of leaf (0 to @code{2**32-1}, 0 if unlimited or in
sequential mode).

@item 
@emph{node_offset}: node offset (0 to @code{2**64-1} for BLAKE2b, 0 to @code{2**48-1} for
BLAKE2s, 0 for the first, leftmost, leaf, or in sequential mode).

@item 
@emph{node_depth}: node depth (0 to 255, 0 for leaves, or in sequential mode).

@item 
@emph{inner_size}: inner digest size (0 to 64 for BLAKE2b, 0 to 32 for
BLAKE2s, 0 in sequential mode).

@item 
@emph{last_node}: boolean indicating whether the processed node is the last
one (@cite{False} for sequential mode).
@end itemize


@float Figure

@image{python-figures/hashlib-blake2-tree,,,Explanation of tree mode parameters.,png}

@end float


See section 2.10 in BLAKE2 specification@footnote{https://blake2.net/blake2_20130129.pdf} for comprehensive review of tree
hashing.

@node Constants<5>,Examples<11>,Creating hash objects,BLAKE2
@anchor{library/hashlib constants}@anchor{234b}
@subsubsection Constants


@geindex blake2b.SALT_SIZE (in module hashlib)
@anchor{library/hashlib hashlib blake2b SALT_SIZE}@anchor{234c}
@deffn {Data} blake2b.SALT_SIZE
@end deffn

@geindex blake2s.SALT_SIZE (in module hashlib)
@anchor{library/hashlib hashlib blake2s SALT_SIZE}@anchor{234d}
@deffn {Data} blake2s.SALT_SIZE
@end deffn

Salt length (maximum length accepted by constructors).

@geindex blake2b.PERSON_SIZE (in module hashlib)
@anchor{library/hashlib hashlib blake2b PERSON_SIZE}@anchor{234e}
@deffn {Data} blake2b.PERSON_SIZE
@end deffn

@geindex blake2s.PERSON_SIZE (in module hashlib)
@anchor{library/hashlib hashlib blake2s PERSON_SIZE}@anchor{234f}
@deffn {Data} blake2s.PERSON_SIZE
@end deffn

Personalization string length (maximum length accepted by constructors).

@geindex blake2b.MAX_KEY_SIZE (in module hashlib)
@anchor{library/hashlib hashlib blake2b MAX_KEY_SIZE}@anchor{2350}
@deffn {Data} blake2b.MAX_KEY_SIZE
@end deffn

@geindex blake2s.MAX_KEY_SIZE (in module hashlib)
@anchor{library/hashlib hashlib blake2s MAX_KEY_SIZE}@anchor{2351}
@deffn {Data} blake2s.MAX_KEY_SIZE
@end deffn

Maximum key size.

@geindex blake2b.MAX_DIGEST_SIZE (in module hashlib)
@anchor{library/hashlib hashlib blake2b MAX_DIGEST_SIZE}@anchor{2352}
@deffn {Data} blake2b.MAX_DIGEST_SIZE
@end deffn

@geindex blake2s.MAX_DIGEST_SIZE (in module hashlib)
@anchor{library/hashlib hashlib blake2s MAX_DIGEST_SIZE}@anchor{2353}
@deffn {Data} blake2s.MAX_DIGEST_SIZE
@end deffn

Maximum digest size that the hash function can output.

@node Examples<11>,Credits,Constants<5>,BLAKE2
@anchor{library/hashlib examples}@anchor{2354}
@subsubsection Examples


@menu
* Simple hashing:: 
* Using different digest sizes:: 
* Keyed hashing:: 
* Randomized hashing:: 
* Personalization:: 
* Tree mode:: 

@end menu

@node Simple hashing,Using different digest sizes,,Examples<11>
@anchor{library/hashlib simple-hashing}@anchor{2355}
@subsubsection Simple hashing


To calculate hash of some data, you should first construct a hash object by
calling the appropriate constructor function (@ref{7ee,,blake2b()} or
@ref{7ef,,blake2s()}), then update it with the data by calling @code{update()} on the
object, and, finally, get the digest out of the object by calling
@code{digest()} (or @code{hexdigest()} for hex-encoded string).

@example
>>> from hashlib import blake2b
>>> h = blake2b()
>>> h.update(b'Hello world')
>>> h.hexdigest()
'6ff843ba685842aa82031d3f53c48b66326df7639a63d128974c5c14f31a0f33343a8c65551134ed1ae0f2b0dd2bb495dc81039e3eeb0aa1bb0388bbeac29183'
@end example

As a shortcut, you can pass the first chunk of data to update directly to the
constructor as the positional argument:

@example
>>> from hashlib import blake2b
>>> blake2b(b'Hello world').hexdigest()
'6ff843ba685842aa82031d3f53c48b66326df7639a63d128974c5c14f31a0f33343a8c65551134ed1ae0f2b0dd2bb495dc81039e3eeb0aa1bb0388bbeac29183'
@end example

You can call @ref{2343,,hash.update()} as many times as you need to iteratively
update the hash:

@example
>>> from hashlib import blake2b
>>> items = [b'Hello', b' ', b'world']
>>> h = blake2b()
>>> for item in items:
...     h.update(item)
>>> h.hexdigest()
'6ff843ba685842aa82031d3f53c48b66326df7639a63d128974c5c14f31a0f33343a8c65551134ed1ae0f2b0dd2bb495dc81039e3eeb0aa1bb0388bbeac29183'
@end example

@node Using different digest sizes,Keyed hashing,Simple hashing,Examples<11>
@anchor{library/hashlib using-different-digest-sizes}@anchor{2356}
@subsubsection Using different digest sizes


BLAKE2 has configurable size of digests up to 64 bytes for BLAKE2b and up to 32
bytes for BLAKE2s. For example, to replace SHA-1 with BLAKE2b without changing
the size of output, we can tell BLAKE2b to produce 20-byte digests:

@example
>>> from hashlib import blake2b
>>> h = blake2b(digest_size=20)
>>> h.update(b'Replacing SHA1 with the more secure function')
>>> h.hexdigest()
'd24f26cf8de66472d58d4e1b1774b4c9158b1f4c'
>>> h.digest_size
20
>>> len(h.digest())
20
@end example

Hash objects with different digest sizes have completely different outputs
(shorter hashes are @emph{not} prefixes of longer hashes); BLAKE2b and BLAKE2s
produce different outputs even if the output length is the same:

@example
>>> from hashlib import blake2b, blake2s
>>> blake2b(digest_size=10).hexdigest()
'6fa1d8fcfd719046d762'
>>> blake2b(digest_size=11).hexdigest()
'eb6ec15daf9546254f0809'
>>> blake2s(digest_size=10).hexdigest()
'1bf21a98c78a1c376ae9'
>>> blake2s(digest_size=11).hexdigest()
'567004bf96e4a25773ebf4'
@end example

@node Keyed hashing,Randomized hashing,Using different digest sizes,Examples<11>
@anchor{library/hashlib keyed-hashing}@anchor{2357}
@subsubsection Keyed hashing


Keyed hashing can be used for authentication as a faster and simpler
replacement for Hash-based message authentication code@footnote{https://en.wikipedia.org/wiki/HMAC} (HMAC).
BLAKE2 can be securely used in prefix-MAC mode thanks to the
indifferentiability property inherited from BLAKE.

This example shows how to get a (hex-encoded) 128-bit authentication code for
message @code{b'message data'} with key @code{b'pseudorandom key'}:

@example
>>> from hashlib import blake2b
>>> h = blake2b(key=b'pseudorandom key', digest_size=16)
>>> h.update(b'message data')
>>> h.hexdigest()
'3d363ff7401e02026f4a4687d4863ced'
@end example

As a practical example, a web application can symmetrically sign cookies sent
to users and later verify them to make sure they weren’t tampered with:

@example
>>> from hashlib import blake2b
>>> from hmac import compare_digest
>>>
>>> SECRET_KEY = b'pseudorandomly generated server secret key'
>>> AUTH_SIZE = 16
>>>
>>> def sign(cookie):
...     h = blake2b(digest_size=AUTH_SIZE, key=SECRET_KEY)
...     h.update(cookie)
...     return h.hexdigest().encode('utf-8')
>>>
>>> def verify(cookie, sig):
...     good_sig = sign(cookie)
...     return compare_digest(good_sig, sig)
>>>
>>> cookie = b'user-alice'
>>> sig = sign(cookie)
>>> print("@{0@},@{1@}".format(cookie.decode('utf-8'), sig))
user-alice,b'43b3c982cf697e0c5ab22172d1ca7421'
>>> verify(cookie, sig)
True
>>> verify(b'user-bob', sig)
False
>>> verify(cookie, b'0102030405060708090a0b0c0d0e0f00')
False
@end example

Even though there’s a native keyed hashing mode, BLAKE2 can, of course, be used
in HMAC construction with @ref{8a,,hmac} module:

@example
>>> import hmac, hashlib
>>> m = hmac.new(b'secret key', digestmod=hashlib.blake2s)
>>> m.update(b'message')
>>> m.hexdigest()
'e3c8102868d28b5ff85fc35dda07329970d1a01e273c37481326fe0c861c8142'
@end example

@node Randomized hashing,Personalization,Keyed hashing,Examples<11>
@anchor{library/hashlib randomized-hashing}@anchor{2358}
@subsubsection Randomized hashing


By setting @emph{salt} parameter users can introduce randomization to the hash
function. Randomized hashing is useful for protecting against collision attacks
on the hash function used in digital signatures.

@quotation

Randomized hashing is designed for situations where one party, the message
preparer, generates all or part of a message to be signed by a second
party, the message signer. If the message preparer is able to find
cryptographic hash function collisions (i.e., two messages producing the
same hash value), then they might prepare meaningful versions of the message
that would produce the same hash value and digital signature, but with
different results (e.g., transferring $1,000,000 to an account, rather than
$10). Cryptographic hash functions have been designed with collision
resistance as a major goal, but the current concentration on attacking
cryptographic hash functions may result in a given cryptographic hash
function providing less collision resistance than expected. Randomized
hashing offers the signer additional protection by reducing the likelihood
that a preparer can generate two or more messages that ultimately yield the
same hash value during the digital signature generation process — even if
it is practical to find collisions for the hash function. However, the use
of randomized hashing may reduce the amount of security provided by a
digital signature when all portions of the message are prepared
by the signer.

(NIST SP-800-106 "Randomized Hashing for Digital Signatures"@footnote{https://csrc.nist.gov/publications/detail/sp/800-106/final})
@end quotation

In BLAKE2 the salt is processed as a one-time input to the hash function during
initialization, rather than as an input to each compression function.

@cartouche
@quotation Warning 
@emph{Salted hashing} (or just hashing) with BLAKE2 or any other general-purpose
cryptographic hash function, such as SHA-256, is not suitable for hashing
passwords.  See BLAKE2 FAQ@footnote{https://blake2.net/#qa} for more
information.
@end quotation
@end cartouche


@example
>>> import os
>>> from hashlib import blake2b
>>> msg = b'some message'
>>> # Calculate the first hash with a random salt.
>>> salt1 = os.urandom(blake2b.SALT_SIZE)
>>> h1 = blake2b(salt=salt1)
>>> h1.update(msg)
>>> # Calculate the second hash with a different random salt.
>>> salt2 = os.urandom(blake2b.SALT_SIZE)
>>> h2 = blake2b(salt=salt2)
>>> h2.update(msg)
>>> # The digests are different.
>>> h1.digest() != h2.digest()
True
@end example

@node Personalization,Tree mode,Randomized hashing,Examples<11>
@anchor{library/hashlib personalization}@anchor{2359}
@subsubsection Personalization


Sometimes it is useful to force hash function to produce different digests for
the same input for different purposes. Quoting the authors of the Skein hash
function:

@quotation

We recommend that all application designers seriously consider doing this;
we have seen many protocols where a hash that is computed in one part of
the protocol can be used in an entirely different part because two hash
computations were done on similar or related data, and the attacker can
force the application to make the hash inputs the same. Personalizing each
hash function used in the protocol summarily stops this type of attack.

(The Skein Hash Function Family@footnote{http://www.skein-hash.info/sites/default/files/skein1.3.pdf},
p. 21)
@end quotation

BLAKE2 can be personalized by passing bytes to the @emph{person} argument:

@example
>>> from hashlib import blake2b
>>> FILES_HASH_PERSON = b'MyApp Files Hash'
>>> BLOCK_HASH_PERSON = b'MyApp Block Hash'
>>> h = blake2b(digest_size=32, person=FILES_HASH_PERSON)
>>> h.update(b'the same content')
>>> h.hexdigest()
'20d9cd024d4fb086aae819a1432dd2466de12947831b75c5a30cf2676095d3b4'
>>> h = blake2b(digest_size=32, person=BLOCK_HASH_PERSON)
>>> h.update(b'the same content')
>>> h.hexdigest()
'cf68fb5761b9c44e7878bfb2c4c9aea52264a80b75005e65619778de59f383a3'
@end example

Personalization together with the keyed mode can also be used to derive different
keys from a single one.

@example
>>> from hashlib import blake2s
>>> from base64 import b64decode, b64encode
>>> orig_key = b64decode(b'Rm5EPJai72qcK3RGBpW3vPNfZy5OZothY+kHY6h21KM=')
>>> enc_key = blake2s(key=orig_key, person=b'kEncrypt').digest()
>>> mac_key = blake2s(key=orig_key, person=b'kMAC').digest()
>>> print(b64encode(enc_key).decode('utf-8'))
rbPb15S/Z9t+agffno5wuhB77VbRi6F9Iv2qIxU7WHw=
>>> print(b64encode(mac_key).decode('utf-8'))
G9GtHFE1YluXY1zWPlYk1e/nWfu0WSEb0KRcjhDeP/o=
@end example

@node Tree mode,,Personalization,Examples<11>
@anchor{library/hashlib tree-mode}@anchor{235a}
@subsubsection Tree mode


Here’s an example of hashing a minimal tree with two leaf nodes:

@example
  10
 /  \
00  01
@end example

This example uses 64-byte internal digests, and returns the 32-byte final
digest:

@example
>>> from hashlib import blake2b
>>>
>>> FANOUT = 2
>>> DEPTH = 2
>>> LEAF_SIZE = 4096
>>> INNER_SIZE = 64
>>>
>>> buf = bytearray(6000)
>>>
>>> # Left leaf
... h00 = blake2b(buf[0:LEAF_SIZE], fanout=FANOUT, depth=DEPTH,
...               leaf_size=LEAF_SIZE, inner_size=INNER_SIZE,
...               node_offset=0, node_depth=0, last_node=False)
>>> # Right leaf
... h01 = blake2b(buf[LEAF_SIZE:], fanout=FANOUT, depth=DEPTH,
...               leaf_size=LEAF_SIZE, inner_size=INNER_SIZE,
...               node_offset=1, node_depth=0, last_node=True)
>>> # Root node
... h10 = blake2b(digest_size=32, fanout=FANOUT, depth=DEPTH,
...               leaf_size=LEAF_SIZE, inner_size=INNER_SIZE,
...               node_offset=0, node_depth=1, last_node=True)
>>> h10.update(h00.digest())
>>> h10.update(h01.digest())
>>> h10.hexdigest()
'3ad2a9b37c6070e374c7a8c508fe20ca86b6ed54e286e93a0318e95e881db5aa'
@end example

@node Credits,,Examples<11>,BLAKE2
@anchor{library/hashlib credits}@anchor{235b}
@subsubsection Credits


BLAKE2@footnote{https://blake2.net} was designed by @emph{Jean-Philippe Aumasson}, @emph{Samuel Neves}, @emph{Zooko
Wilcox-O’Hearn}, and @emph{Christian Winnerlein} based on SHA-3@footnote{https://en.wikipedia.org/wiki/NIST_hash_function_competition} finalist BLAKE@footnote{https://131002.net/blake/}
created by @emph{Jean-Philippe Aumasson}, @emph{Luca Henzen}, @emph{Willi Meier}, and
@emph{Raphael C.-W. Phan}.

It uses core algorithm from ChaCha@footnote{https://cr.yp.to/chacha.html} cipher designed by @emph{Daniel J.  Bernstein}.

The stdlib implementation is based on pyblake2@footnote{https://pythonhosted.org/pyblake2/} module. It was written by
@emph{Dmitry Chestnykh} based on C implementation written by @emph{Samuel Neves}. The
documentation was copied from pyblake2@footnote{https://pythonhosted.org/pyblake2/} and written by @emph{Dmitry Chestnykh}.

The C code was partly rewritten for Python by @emph{Christian Heimes}.

The following public domain dedication applies for both C hash function
implementation, extension code, and this documentation:

@quotation

To the extent possible under law, the author(s) have dedicated all copyright
and related and neighboring rights to this software to the public domain
worldwide. This software is distributed without any warranty.

You should have received a copy of the CC0 Public Domain Dedication along
with this software. If not, see
@indicateurl{https://creativecommons.org/publicdomain/zero/1.0/}.
@end quotation

The following people have helped with development or contributed their changes
to the project and the public domain according to the Creative Commons Public
Domain Dedication 1.0 Universal:


@itemize *

@item 
@emph{Alexandr Sokolovskiy}
@end itemize


@subsubheading See also



@table @asis

@item Module @ref{8a,,hmac}

A module to generate message authentication codes using hashes.

@item Module @ref{d,,base64}

Another way to encode binary hashes for non-binary environments.

@item @indicateurl{https://blake2.net}

Official BLAKE2 website.

@item @indicateurl{https://csrc.nist.gov/csrc/media/publications/fips/180/2/archive/2002-08-01/documents/fips180-2.pdf}

The FIPS 180-2 publication on Secure Hash Algorithms.

@item @indicateurl{https://en.wikipedia.org/wiki/Cryptographic_hash_function#Cryptographic_hash_algorithms}

Wikipedia article with information on which algorithms have known issues and
what that means regarding their use.

@item @indicateurl{https://www.ietf.org/rfc/rfc8018.txt}

PKCS #5: Password-Based Cryptography Specification Version 2.1

@item @indicateurl{https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-132.pdf}

NIST Recommendation for Password-Based Key Derivation.
@end table


@node hmac — Keyed-Hashing for Message Authentication,secrets — Generate secure random numbers for managing secrets,hashlib — Secure hashes and message digests,Cryptographic Services
@anchor{library/hmac doc}@anchor{235c}@anchor{library/hmac hmac-keyed-hashing-for-message-authentication}@anchor{235d}@anchor{library/hmac module-hmac}@anchor{8a}
@subsection @code{hmac} — Keyed-Hashing for Message Authentication


@geindex module; hmac

@strong{Source code:} Lib/hmac.py@footnote{https://github.com/python/cpython/tree/main/Lib/hmac.py}


__________________________________________________________________


This module implements the HMAC algorithm as described by 
@geindex RFC; RFC 2104
RFC 2104@footnote{https://tools.ietf.org/html/rfc2104.html}.

@geindex new() (in module hmac)
@anchor{library/hmac hmac new}@anchor{ac7}
@deffn {Function} hmac.new (key, msg=None, digestmod='@w{'})

Return a new hmac object.  @emph{key} is a bytes or bytearray object giving the
secret key.  If @emph{msg} is present, the method call @code{update(msg)} is made.
@emph{digestmod} is the digest name, digest constructor or module for the HMAC
object to use.  It may be any name suitable to @ref{11cb,,hashlib.new()}.
Despite its argument position, it is required.


Changed in version 3.4: Parameter @emph{key} can be a bytes or bytearray object.
Parameter @emph{msg} can be of any type supported by @ref{88,,hashlib}.
Parameter @emph{digestmod} can be the name of a hash algorithm.



Deprecated since version 3.4, removed in version 3.8: MD5 as implicit default digest for @emph{digestmod} is deprecated.
The digestmod parameter is now required.  Pass it as a keyword
argument to avoid awkwardness when you do not have an initial msg.

@end deffn

@geindex digest() (in module hmac)
@anchor{library/hmac hmac digest}@anchor{661}
@deffn {Function} hmac.digest (key, msg, digest)

Return digest of @emph{msg} for given secret @emph{key} and @emph{digest}. The
function is equivalent to @code{HMAC(key, msg, digest).digest()}, but
uses an optimized C or inline implementation, which is faster for messages
that fit into memory. The parameters @emph{key}, @emph{msg}, and @emph{digest} have
the same meaning as in @ref{ac7,,new()}.

CPython implementation detail, the optimized C implementation is only used
when @emph{digest} is a string and name of a digest algorithm, which is
supported by OpenSSL.


New in version 3.7.

@end deffn

An HMAC object has the following methods:

@geindex update() (hmac.HMAC method)
@anchor{library/hmac hmac HMAC update}@anchor{ac8}
@deffn {Method} HMAC.update (msg)

Update the hmac object with @emph{msg}.  Repeated calls are equivalent to a
single call with the concatenation of all the arguments:
@code{m.update(a); m.update(b)} is equivalent to @code{m.update(a + b)}.


Changed in version 3.4: Parameter @emph{msg} can be of any type supported by @ref{88,,hashlib}.

@end deffn

@geindex digest() (hmac.HMAC method)
@anchor{library/hmac hmac HMAC digest}@anchor{235e}
@deffn {Method} HMAC.digest ()

Return the digest of the bytes passed to the @ref{ac8,,update()} method so far.
This bytes object will be the same length as the @emph{digest_size} of the digest
given to the constructor.  It may contain non-ASCII bytes, including NUL
bytes.

@cartouche
@quotation Warning 
When comparing the output of @ref{661,,digest()} to an externally-supplied
digest during a verification routine, it is recommended to use the
@ref{c5b,,compare_digest()} function instead of the @code{==} operator
to reduce the vulnerability to timing attacks.
@end quotation
@end cartouche
@end deffn

@geindex hexdigest() (hmac.HMAC method)
@anchor{library/hmac hmac HMAC hexdigest}@anchor{235f}
@deffn {Method} HMAC.hexdigest ()

Like @ref{661,,digest()} except the digest is returned as a string twice the
length containing only hexadecimal digits.  This may be used to exchange the
value safely in email or other non-binary environments.

@cartouche
@quotation Warning 
When comparing the output of @ref{235f,,hexdigest()} to an externally-supplied
digest during a verification routine, it is recommended to use the
@ref{c5b,,compare_digest()} function instead of the @code{==} operator
to reduce the vulnerability to timing attacks.
@end quotation
@end cartouche
@end deffn

@geindex copy() (hmac.HMAC method)
@anchor{library/hmac hmac HMAC copy}@anchor{2360}
@deffn {Method} HMAC.copy ()

Return a copy (“clone”) of the hmac object.  This can be used to efficiently
compute the digests of strings that share a common initial substring.
@end deffn

A hash object has the following attributes:

@geindex digest_size (hmac.HMAC attribute)
@anchor{library/hmac hmac HMAC digest_size}@anchor{acb}
@deffn {Attribute} HMAC.digest_size

The size of the resulting HMAC digest in bytes.
@end deffn

@geindex block_size (hmac.HMAC attribute)
@anchor{library/hmac hmac HMAC block_size}@anchor{ac9}
@deffn {Attribute} HMAC.block_size

The internal block size of the hash algorithm in bytes.


New in version 3.4.

@end deffn

@geindex name (hmac.HMAC attribute)
@anchor{library/hmac hmac HMAC name}@anchor{aca}
@deffn {Attribute} HMAC.name

The canonical name of this HMAC, always lowercase, e.g. @code{hmac-md5}.


New in version 3.4.

@end deffn


Deprecated since version 3.9: The undocumented attributes @code{HMAC.digest_cons}, @code{HMAC.inner}, and
@code{HMAC.outer} are internal implementation details and will be removed in
Python 3.10.


This module also provides the following helper function:

@geindex compare_digest() (in module hmac)
@anchor{library/hmac hmac compare_digest}@anchor{c5b}
@deffn {Function} hmac.compare_digest (a, b)

Return @code{a == b}.  This function uses an approach designed to prevent
timing analysis by avoiding content-based short circuiting behaviour,
making it appropriate for cryptography.  @emph{a} and @emph{b} must both be of the
same type: either @ref{2ef,,str} (ASCII only, as e.g. returned by
@ref{235f,,HMAC.hexdigest()}), or a @ref{87e,,bytes-like object}.

@cartouche
@quotation Note 
If @emph{a} and @emph{b} are of different lengths, or if an error occurs,
a timing attack could theoretically reveal information about the
types and lengths of @emph{a} and @emph{b}—but not their values.
@end quotation
@end cartouche


New in version 3.3.



Changed in version 3.10: The function uses OpenSSL’s @code{CRYPTO_memcmp()} internally when
available.

@end deffn


@subsubheading See also



@table @asis

@item Module @ref{88,,hashlib}

The Python module providing secure hash functions.
@end table


@node secrets — Generate secure random numbers for managing secrets,,hmac — Keyed-Hashing for Message Authentication,Cryptographic Services
@anchor{library/secrets doc}@anchor{2361}@anchor{library/secrets module-secrets}@anchor{df}@anchor{library/secrets secrets-generate-secure-random-numbers-for-managing-secrets}@anchor{2362}
@subsection @code{secrets} — Generate secure random numbers for managing secrets


@geindex module; secrets


New in version 3.6.


@c from secrets import *
@c __name__ = '<doctest>'

@strong{Source code:} Lib/secrets.py@footnote{https://github.com/python/cpython/tree/main/Lib/secrets.py}


__________________________________________________________________


The @ref{df,,secrets} module is used for generating cryptographically strong
random numbers suitable for managing data such as passwords, account
authentication, security tokens, and related secrets.

In particular, @ref{df,,secrets} should be used in preference to the
default pseudo-random number generator in the @ref{d7,,random} module, which
is designed for modelling and simulation, not security or cryptography.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 506
PEP 506@footnote{https://www.python.org/dev/peps/pep-0506}


@menu
* Random numbers:: 
* Generating tokens:: 
* Other functions:: 
* Recipes and best practices:: 

@end menu

@node Random numbers,Generating tokens,,secrets — Generate secure random numbers for managing secrets
@anchor{library/secrets random-numbers}@anchor{2363}
@subsubsection Random numbers


The @ref{df,,secrets} module provides access to the most secure source of
randomness that your operating system provides.

@geindex SystemRandom (class in secrets)
@anchor{library/secrets secrets SystemRandom}@anchor{2364}
@deffn {Class} secrets.SystemRandom

A class for generating random numbers using the highest-quality
sources provided by the operating system.  See
@ref{1fa4,,random.SystemRandom} for additional details.
@end deffn

@geindex choice() (in module secrets)
@anchor{library/secrets secrets choice}@anchor{2365}
@deffn {Function} secrets.choice (sequence)

Return a randomly-chosen element from a non-empty sequence.
@end deffn

@geindex randbelow() (in module secrets)
@anchor{library/secrets secrets randbelow}@anchor{2366}
@deffn {Function} secrets.randbelow (n)

Return a random int in the range [0, @emph{n}).
@end deffn

@geindex randbits() (in module secrets)
@anchor{library/secrets secrets randbits}@anchor{2367}
@deffn {Function} secrets.randbits (k)

Return an int with @emph{k} random bits.
@end deffn

@node Generating tokens,Other functions,Random numbers,secrets — Generate secure random numbers for managing secrets
@anchor{library/secrets generating-tokens}@anchor{2368}
@subsubsection Generating tokens


The @ref{df,,secrets} module provides functions for generating secure
tokens, suitable for applications such as password resets,
hard-to-guess URLs, and similar.

@geindex token_bytes() (in module secrets)
@anchor{library/secrets secrets token_bytes}@anchor{1fa9}
@deffn {Function} secrets.token_bytes ([nbytes=None])

Return a random byte string containing @emph{nbytes} number of bytes.
If @emph{nbytes} is @code{None} or not supplied, a reasonable default is
used.

@example
>>> token_bytes(16)  
b'\xebr\x17D*t\xae\xd4\xe3S\xb6\xe2\xebP1\x8b'
@end example
@end deffn

@geindex token_hex() (in module secrets)
@anchor{library/secrets secrets token_hex}@anchor{2369}
@deffn {Function} secrets.token_hex ([nbytes=None])

Return a random text string, in hexadecimal.  The string has @emph{nbytes}
random bytes, each byte converted to two hex digits.  If @emph{nbytes} is
@code{None} or not supplied, a reasonable default is used.

@example
>>> token_hex(16)  
'f9bf78b9a18ce6d46a0cd2b0b86df9da'
@end example
@end deffn

@geindex token_urlsafe() (in module secrets)
@anchor{library/secrets secrets token_urlsafe}@anchor{236a}
@deffn {Function} secrets.token_urlsafe ([nbytes=None])

Return a random URL-safe text string, containing @emph{nbytes} random
bytes.  The text is Base64 encoded, so on average each byte results
in approximately 1.3 characters.  If @emph{nbytes} is @code{None} or not
supplied, a reasonable default is used.

@example
>>> token_urlsafe(16)  
'Drmhze6EPcv0fN_81Bj-nA'
@end example
@end deffn

@menu
* How many bytes should tokens use?:: 

@end menu

@node How many bytes should tokens use?,,,Generating tokens
@anchor{library/secrets how-many-bytes-should-tokens-use}@anchor{236b}
@subsubsection How many bytes should tokens use?


To be secure against
brute-force attacks@footnote{https://en.wikipedia.org/wiki/Brute-force_attack},
tokens need to have sufficient randomness.  Unfortunately, what is
considered sufficient will necessarily increase as computers get more
powerful and able to make more guesses in a shorter period.  As of 2015,
it is believed that 32 bytes (256 bits) of randomness is sufficient for
the typical use-case expected for the @ref{df,,secrets} module.

For those who want to manage their own token length, you can explicitly
specify how much randomness is used for tokens by giving an @ref{1b1,,int}
argument to the various @code{token_*} functions.  That argument is taken
as the number of bytes of randomness to use.

Otherwise, if no argument is provided, or if the argument is @code{None},
the @code{token_*} functions will use a reasonable default instead.

@cartouche
@quotation Note 
That default is subject to change at any time, including during
maintenance releases.
@end quotation
@end cartouche

@node Other functions,Recipes and best practices,Generating tokens,secrets — Generate secure random numbers for managing secrets
@anchor{library/secrets other-functions}@anchor{236c}
@subsubsection Other functions


@geindex compare_digest() (in module secrets)
@anchor{library/secrets secrets compare_digest}@anchor{236d}
@deffn {Function} secrets.compare_digest (a, b)

Return @code{True} if strings @emph{a} and @emph{b} are equal, otherwise @code{False},
in such a way as to reduce the risk of
timing attacks@footnote{https://codahale.com/a-lesson-in-timing-attacks/}.
See @ref{c5b,,hmac.compare_digest()} for additional details.
@end deffn

@node Recipes and best practices,,Other functions,secrets — Generate secure random numbers for managing secrets
@anchor{library/secrets recipes-and-best-practices}@anchor{236e}
@subsubsection Recipes and best practices


This section shows recipes and best practices for using @ref{df,,secrets}
to manage a basic level of security.

Generate an eight-character alphanumeric password:

@example
import string
import secrets
alphabet = string.ascii_letters + string.digits
password = '@w{'}.join(secrets.choice(alphabet) for i in range(8))
@end example

@cartouche
@quotation Note 
Applications should not
store passwords in a recoverable format@footnote{http://cwe.mitre.org/data/definitions/257.html},
whether plain text or encrypted.  They should be salted and hashed
using a cryptographically-strong one-way (irreversible) hash function.
@end quotation
@end cartouche

Generate a ten-character alphanumeric password with at least one
lowercase character, at least one uppercase character, and at least
three digits:

@example
import string
import secrets
alphabet = string.ascii_letters + string.digits
while True:
    password = '@w{'}.join(secrets.choice(alphabet) for i in range(10))
    if (any(c.islower() for c in password)
            and any(c.isupper() for c in password)
            and sum(c.isdigit() for c in password) >= 3):
        break
@end example

Generate an XKCD-style passphrase@footnote{https://xkcd.com/936/}:

@example
import secrets
# On standard Linux systems, use a convenient dictionary file.
# Other platforms may need to provide their own word-list.
with open('/usr/share/dict/words') as f:
    words = [word.strip() for word in f]
    password = ' '.join(secrets.choice(words) for i in range(4))
@end example

Generate a hard-to-guess temporary URL containing a security token
suitable for password recovery applications:

@example
import secrets
url = 'https://mydomain.com/reset=' + secrets.token_urlsafe()
@end example

@c # This modeline must appear within the last ten lines of the file.
@c kate: indent-width 3; remove-trailing-space on; replace-tabs on; encoding utf-8;

@node Generic Operating System Services,Concurrent Execution,Cryptographic Services,The Python Standard Library
@anchor{library/allos doc}@anchor{236f}@anchor{library/allos allos}@anchor{2370}@anchor{library/allos generic-operating-system-services}@anchor{2371}
@section Generic Operating System Services


The modules described in this chapter provide interfaces to operating system
features that are available on (almost) all operating systems, such as files and
a clock.  The interfaces are generally modeled after the Unix or C interfaces,
but they are available on most other systems as well.  Here’s an overview:

@menu
* os — Miscellaneous operating system interfaces:: 
* io — Core tools for working with streams:: 
* time — Time access and conversions:: 
* argparse — Parser for command-line options@comma{} arguments and sub-commands: argparse — Parser for command-line options arguments and sub-commands. 
* getopt — C-style parser for command line options:: 
* logging — Logging facility for Python:: 
* logging.config — Logging configuration: logging config — Logging configuration. 
* logging.handlers — Logging handlers: logging handlers — Logging handlers. 
* getpass — Portable password input:: 
* curses — Terminal handling for character-cell displays:: 
* curses.textpad — Text input widget for curses programs: curses textpad — Text input widget for curses programs. 
* curses.ascii — Utilities for ASCII characters: curses ascii — Utilities for ASCII characters. 
* curses.panel — A panel stack extension for curses: curses panel — A panel stack extension for curses. 
* platform — Access to underlying platform’s identifying data:: 
* errno — Standard errno system symbols:: 
* ctypes — A foreign function library for Python:: 

@end menu

@node os — Miscellaneous operating system interfaces,io — Core tools for working with streams,,Generic Operating System Services
@anchor{library/os doc}@anchor{2372}@anchor{library/os module-os}@anchor{c0}@anchor{library/os os-miscellaneous-operating-system-interfaces}@anchor{2373}
@subsection @code{os} — Miscellaneous operating system interfaces


@geindex module; os

@strong{Source code:} Lib/os.py@footnote{https://github.com/python/cpython/tree/main/Lib/os.py}


__________________________________________________________________


This module provides a portable way of using operating system dependent
functionality.  If you just want to read or write a file see @ref{1c8,,open()}, if
you want to manipulate paths, see the @ref{c1,,os.path} module, and if you want to
read all the lines in all the files on the command line see the @ref{7b,,fileinput}
module.  For creating temporary files and directories see the @ref{fd,,tempfile}
module, and for high-level file and directory handling see the @ref{e4,,shutil}
module.

Notes on the availability of these functions:


@itemize *

@item 
The design of all built-in operating system dependent modules of Python is
such that as long as the same functionality is available, it uses the same
interface; for example, the function @code{os.stat(path)} returns stat
information about @emph{path} in the same format (which happens to have originated
with the POSIX interface).

@item 
Extensions peculiar to a particular operating system are also available
through the @ref{c0,,os} module, but using them is of course a threat to
portability.

@item 
All functions accepting path or file names accept both bytes and string
objects, and result in an object of the same type, if a path or file name is
returned.

@item 
On VxWorks, os.popen, os.fork, os.execv and os.spawn*p* are not supported.
@end itemize

@cartouche
@quotation Note 
All functions in this module raise @ref{2a1,,OSError} (or subclasses thereof) in
the case of invalid or inaccessible file names and paths, or other arguments
that have the correct type, but are not accepted by the operating system.
@end quotation
@end cartouche

@geindex error
@anchor{library/os os error}@anchor{2374}
@deffn {Exception} os.error

An alias for the built-in @ref{2a1,,OSError} exception.
@end deffn

@geindex name (in module os)
@anchor{library/os os name}@anchor{2375}
@deffn {Data} os.name

The name of the operating system dependent module imported.  The following
names have currently been registered: @code{'posix'}, @code{'nt'},
@code{'java'}.


@subsubheading See also


@ref{5a6,,sys.platform} has a finer granularity.  @ref{ca9,,os.uname()} gives
system-dependent version information.

The @ref{c9,,platform} module provides detailed checks for the
system’s identity.

@end deffn
@anchor{library/os os-filenames}@anchor{22ad}
@menu
* File Names@comma{} Command Line Arguments@comma{} and Environment Variables: File Names Command Line Arguments and Environment Variables. 
* Python UTF-8 Mode:: 
* Process Parameters:: 
* File Object Creation:: 
* File Descriptor Operations:: 
* Files and Directories:: 
* Process Management:: 
* Interface to the scheduler:: 
* Miscellaneous System Information:: 
* Random numbers: Random numbers<2>. 

@end menu

@node File Names Command Line Arguments and Environment Variables,Python UTF-8 Mode,,os — Miscellaneous operating system interfaces
@anchor{library/os file-names-command-line-arguments-and-environment-variables}@anchor{2376}@anchor{library/os filesystem-encoding}@anchor{2377}
@subsubsection File Names, Command Line Arguments, and Environment Variables


In Python, file names, command line arguments, and environment variables are
represented using the string type. On some systems, decoding these strings to
and from bytes is necessary before passing them to the operating system. Python
uses the @ref{1628,,filesystem encoding and error handler} to perform this
conversion (see @ref{79e,,sys.getfilesystemencoding()}).

The @ref{1628,,filesystem encoding and error handler} are configured at Python
startup by the @code{PyConfig_Read()} function: see
@ref{2378,,filesystem_encoding} and
@ref{2379,,filesystem_errors} members of @ref{206,,PyConfig}.


Changed in version 3.1: On some systems, conversion using the file system encoding may fail. In this
case, Python uses the @ref{1cc9,,surrogateescape encoding error handler}, which means that undecodable bytes are replaced by a
Unicode character U+DCxx on decoding, and these are again translated to the
original byte on encoding.


The @ref{1628,,file system encoding} must
guarantee to successfully decode all bytes below 128. If the file system
encoding fails to provide this guarantee, API functions can raise
@ref{e49,,UnicodeError}.

See also the @ref{237a,,locale encoding}.

@node Python UTF-8 Mode,Process Parameters,File Names Command Line Arguments and Environment Variables,os — Miscellaneous operating system interfaces
@anchor{library/os python-utf-8-mode}@anchor{237b}@anchor{library/os utf8-mode}@anchor{5f2}
@subsubsection Python UTF-8 Mode



New in version 3.7: See 
@geindex Python Enhancement Proposals; PEP 540
PEP 540@footnote{https://www.python.org/dev/peps/pep-0540} for more details.


The Python UTF-8 Mode ignores the @ref{237a,,locale encoding} and forces the usage
of the UTF-8 encoding:


@itemize *

@item 
Use UTF-8 as the @ref{1628,,filesystem encoding}.

@item 
@ref{79e,,sys.getfilesystemencoding()} returns @code{'UTF-8'}.

@item 
@ref{1b5,,locale.getpreferredencoding()} returns @code{'UTF-8'} (the @emph{do_setlocale}
argument has no effect).

@item 
@ref{5ee,,sys.stdin}, @ref{5ef,,sys.stdout}, and @ref{42d,,sys.stderr} all use
UTF-8 as their text encoding, with the @code{surrogateescape}
@ref{162a,,error handler} being enabled for @ref{5ee,,sys.stdin}
and @ref{5ef,,sys.stdout} (@ref{42d,,sys.stderr} continues to use
@code{backslashreplace} as it does in the default locale-aware mode)

@item 
On Unix, @ref{10f6,,os.device_encoding()} returns @code{'UTF-8'}. rather than the
device encoding.
@end itemize

Note that the standard stream settings in UTF-8 mode can be overridden by
@geindex PYTHONIOENCODING
@geindex environment variable; PYTHONIOENCODING
@ref{b91,,PYTHONIOENCODING} (just as they can be in the default locale-aware
mode).

As a consequence of the changes in those lower level APIs, other higher
level APIs also exhibit different default behaviours:


@itemize *

@item 
Command line arguments, environment variables and filenames are decoded
to text using the UTF-8 encoding.

@item 
@ref{798,,os.fsdecode()} and @ref{799,,os.fsencode()} use the UTF-8 encoding.

@item 
@ref{1c8,,open()}, @ref{1c9,,io.open()}, and @ref{1ca,,codecs.open()} use the UTF-8
encoding by default. However, they still use the strict error handler by
default so that attempting to open a binary file in text mode is likely
to raise an exception rather than producing nonsense data.
@end itemize

The @ref{5f2,,Python UTF-8 Mode} is enabled if the LC_CTYPE locale is
@code{C} or @code{POSIX} at Python startup (see the @code{PyConfig_Read()}
function).

It can be enabled or disabled using the @ref{159,,-X utf8} command line
option and the 
@geindex PYTHONUTF8
@geindex environment variable; PYTHONUTF8
@ref{5f1,,PYTHONUTF8} environment variable.

If the 
@geindex PYTHONUTF8
@geindex environment variable; PYTHONUTF8
@ref{5f1,,PYTHONUTF8} environment variable is not set at all, then the
interpreter defaults to using the current locale settings, @emph{unless} the current
locale is identified as a legacy ASCII-based locale (as described for
@geindex PYTHONCOERCECLOCALE
@geindex environment variable; PYTHONCOERCECLOCALE
@ref{5ed,,PYTHONCOERCECLOCALE}), and locale coercion is either disabled or
fails. In such legacy locales, the interpreter will default to enabling UTF-8
mode unless explicitly instructed not to do so.

The Python UTF-8 Mode can only be enabled at the Python startup. Its value
can be read from @ref{22f,,sys.flags.utf8_mode}.

See also the @ref{16bc,,UTF-8 mode on Windows}
and the @ref{1628,,filesystem encoding and error handler}.

@node Process Parameters,File Object Creation,Python UTF-8 Mode,os — Miscellaneous operating system interfaces
@anchor{library/os os-procinfo}@anchor{237c}@anchor{library/os process-parameters}@anchor{237d}
@subsubsection Process Parameters


These functions and data items provide information and operate on the current
process and user.

@geindex ctermid() (in module os)
@anchor{library/os os ctermid}@anchor{237e}
@deffn {Function} os.ctermid ()

Return the filename corresponding to the controlling terminal of the process.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex environ (in module os)
@anchor{library/os os environ}@anchor{d60}
@deffn {Data} os.environ

A @ref{d62,,mapping} object where keys and values are strings that represent
the process environment.  For example, @code{environ['HOME']} is the pathname
of your home directory (on some platforms), and is equivalent to
@code{getenv("HOME")} in C.

This mapping is captured the first time the @ref{c0,,os} module is imported,
typically during Python startup as part of processing @code{site.py}.  Changes
to the environment made after this time are not reflected in @code{os.environ},
except for changes made by modifying @code{os.environ} directly.

This mapping may be used to modify the environment as well as query the
environment.  @ref{410,,putenv()} will be called automatically when the mapping
is modified.

On Unix, keys and values use @ref{79e,,sys.getfilesystemencoding()} and
@code{'surrogateescape'} error handler. Use @ref{db8,,environb} if you would like
to use a different encoding.

@cartouche
@quotation Note 
Calling @ref{410,,putenv()} directly does not change @code{os.environ}, so it’s better
to modify @code{os.environ}.
@end quotation
@end cartouche

@cartouche
@quotation Note 
On some platforms, including FreeBSD and macOS, setting @code{environ} may
cause memory leaks.  Refer to the system documentation for
@code{putenv()}.
@end quotation
@end cartouche

You can delete items in this mapping to unset environment variables.
@ref{40f,,unsetenv()} will be called automatically when an item is deleted from
@code{os.environ}, and when one of the @code{pop()} or @code{clear()} methods is
called.


Changed in version 3.9: Updated to support 
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584}’s merge (@code{|}) and update (@code{|=}) operators.

@end deffn

@geindex environb (in module os)
@anchor{library/os os environb}@anchor{db8}
@deffn {Data} os.environb

Bytes version of @ref{d60,,environ}: a @ref{d62,,mapping} object where both keys
and values are @ref{2f0,,bytes} objects representing the process environment.
@ref{d60,,environ} and @ref{db8,,environb} are synchronized (modifying
@ref{db8,,environb} updates @ref{d60,,environ}, and vice versa).

@ref{db8,,environb} is only available if @ref{db6,,supports_bytes_environ} is
@code{True}.


New in version 3.2.



Changed in version 3.9: Updated to support 
@geindex Python Enhancement Proposals; PEP 584
PEP 584@footnote{https://www.python.org/dev/peps/pep-0584}’s merge (@code{|}) and update (@code{|=}) operators.

@end deffn


@deffn {Function} os.chdir (path)

@deffnx {Function} os.fchdir (fd)

@deffnx {Function} os.getcwd ()

These functions are described in @ref{237f,,Files and Directories}.
@end deffn

@geindex fsencode() (in module os)
@anchor{library/os os fsencode}@anchor{799}
@deffn {Function} os.fsencode (filename)

Encode @ref{63d,,path-like} @emph{filename} to the
@ref{1628,,filesystem encoding and error handler}; return @ref{2f0,,bytes}
unchanged.

@ref{798,,fsdecode()} is the reverse function.


New in version 3.2.



Changed in version 3.6: Support added to accept objects implementing the @ref{795,,os.PathLike}
interface.

@end deffn

@geindex fsdecode() (in module os)
@anchor{library/os os fsdecode}@anchor{798}
@deffn {Function} os.fsdecode (filename)

Decode the @ref{63d,,path-like} @emph{filename} from the
@ref{1628,,filesystem encoding and error handler}; return @ref{2ef,,str}
unchanged.

@ref{799,,fsencode()} is the reverse function.


New in version 3.2.



Changed in version 3.6: Support added to accept objects implementing the @ref{795,,os.PathLike}
interface.

@end deffn

@geindex fspath() (in module os)
@anchor{library/os os fspath}@anchor{797}
@deffn {Function} os.fspath (path)

Return the file system representation of the path.

If @ref{2ef,,str} or @ref{2f0,,bytes} is passed in, it is returned unchanged.
Otherwise @ref{796,,__fspath__()} is called and its value is
returned as long as it is a @ref{2ef,,str} or @ref{2f0,,bytes} object.
In all other cases, @ref{162,,TypeError} is raised.


New in version 3.6.

@end deffn

@geindex PathLike (class in os)
@anchor{library/os os PathLike}@anchor{795}
@deffn {Class} os.PathLike

An @ref{d5c,,abstract base class} for objects representing a file system path,
e.g. @ref{1c4,,pathlib.PurePath}.


New in version 3.6.


@geindex __fspath__() (os.PathLike method)
@anchor{library/os os PathLike __fspath__}@anchor{796}
@deffn {Method} abstractmethod __fspath__ ()

Return the file system path representation of the object.

The method should only return a @ref{2ef,,str} or @ref{2f0,,bytes} object,
with the preference being for @ref{2ef,,str}.
@end deffn
@end deffn

@geindex getenv() (in module os)
@anchor{library/os os getenv}@anchor{2380}
@deffn {Function} os.getenv (key, default=None)

Return the value of the environment variable @emph{key} if it exists, or
@emph{default} if it doesn’t. @emph{key}, @emph{default} and the result are str.

On Unix, keys and values are decoded with @ref{79e,,sys.getfilesystemencoding()}
and @code{'surrogateescape'} error handler. Use @ref{db7,,os.getenvb()} if you
would like to use a different encoding.

@ref{1629,,Availability}: most flavors of Unix, Windows.
@end deffn

@geindex getenvb() (in module os)
@anchor{library/os os getenvb}@anchor{db7}
@deffn {Function} os.getenvb (key, default=None)

Return the value of the environment variable @emph{key} if it exists, or
@emph{default} if it doesn’t. @emph{key}, @emph{default} and the result are bytes.

@ref{db7,,getenvb()} is only available if @ref{db6,,supports_bytes_environ}
is @code{True}.

@ref{1629,,Availability}: most flavors of Unix.


New in version 3.2.

@end deffn

@geindex get_exec_path() (in module os)
@anchor{library/os os get_exec_path}@anchor{2381}
@deffn {Function} os.get_exec_path (env=None)

Returns the list of directories that will be searched for a named
executable, similar to a shell, when launching a process.
@emph{env}, when specified, should be an environment variable dictionary
to lookup the PATH in.
By default, when @emph{env} is @code{None}, @ref{d60,,environ} is used.


New in version 3.2.

@end deffn

@geindex getegid() (in module os)
@anchor{library/os os getegid}@anchor{2382}
@deffn {Function} os.getegid ()

Return the effective group id of the current process.  This corresponds to the
“set id” bit on the file being executed in the current process.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex geteuid() (in module os)
@anchor{library/os os geteuid}@anchor{2383}
@deffn {Function} os.geteuid ()

@geindex user; effective id

Return the current process’s effective user id.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex getgid() (in module os)
@anchor{library/os os getgid}@anchor{2384}
@deffn {Function} os.getgid ()

@geindex process; group

Return the real group id of the current process.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex getgrouplist() (in module os)
@anchor{library/os os getgrouplist}@anchor{ca7}
@deffn {Function} os.getgrouplist (user, group)

Return list of group ids that @emph{user} belongs to. If @emph{group} is not in the
list, it is included; typically, @emph{group} is specified as the group ID
field from the password record for @emph{user}.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex getgroups() (in module os)
@anchor{library/os os getgroups}@anchor{2385}
@deffn {Function} os.getgroups ()

Return list of supplemental group ids associated with the current process.

@ref{1629,,Availability}: Unix.

@cartouche
@quotation Note 
On macOS, @ref{2385,,getgroups()} behavior differs somewhat from
other Unix platforms. If the Python interpreter was built with a
deployment target of @code{10.5} or earlier, @ref{2385,,getgroups()} returns
the list of effective group ids associated with the current user process;
this list is limited to a system-defined number of entries, typically 16,
and may be modified by calls to @ref{2386,,setgroups()} if suitably privileged.
If built with a deployment target greater than @code{10.5},
@ref{2385,,getgroups()} returns the current group access list for the user
associated with the effective user id of the process; the group access
list may change over the lifetime of the process, it is not affected by
calls to @ref{2386,,setgroups()}, and its length is not limited to 16.  The
deployment target value, @code{MACOSX_DEPLOYMENT_TARGET}, can be
obtained with @ref{bbc,,sysconfig.get_config_var()}.
@end quotation
@end cartouche
@end deffn

@geindex getlogin() (in module os)
@anchor{library/os os getlogin}@anchor{2387}
@deffn {Function} os.getlogin ()

Return the name of the user logged in on the controlling terminal of the
process.  For most purposes, it is more useful to use
@ref{2388,,getpass.getuser()} since the latter checks the environment variables
@geindex LOGNAME
@geindex environment variable; LOGNAME
@code{LOGNAME} or 
@geindex USERNAME
@geindex environment variable; USERNAME
@code{USERNAME} to find out who the user is, and
falls back to @code{pwd.getpwuid(os.getuid())[0]} to get the login name of the
current real user id.

@ref{1629,,Availability}: Unix, Windows.
@end deffn

@geindex getpgid() (in module os)
@anchor{library/os os getpgid}@anchor{2389}
@deffn {Function} os.getpgid (pid)

Return the process group id of the process with process id @emph{pid}. If @emph{pid} is 0,
the process group id of the current process is returned.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex getpgrp() (in module os)
@anchor{library/os os getpgrp}@anchor{238a}
@deffn {Function} os.getpgrp ()

@geindex process; group

Return the id of the current process group.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex getpid() (in module os)
@anchor{library/os os getpid}@anchor{238b}
@deffn {Function} os.getpid ()

@geindex process; id

Return the current process id.
@end deffn

@geindex getppid() (in module os)
@anchor{library/os os getppid}@anchor{238c}
@deffn {Function} os.getppid ()

@geindex process; id of parent

Return the parent’s process id.  When the parent process has exited, on Unix
the id returned is the one of the init process (1), on Windows it is still
the same id, which may be already reused by another process.

@ref{1629,,Availability}: Unix, Windows.


Changed in version 3.2: Added support for Windows.

@end deffn

@geindex getpriority() (in module os)
@anchor{library/os os getpriority}@anchor{c92}
@deffn {Function} os.getpriority (which, who)

@geindex process; scheduling priority

Get program scheduling priority.  The value @emph{which} is one of
@ref{238d,,PRIO_PROCESS}, @ref{238e,,PRIO_PGRP}, or @ref{238f,,PRIO_USER}, and @emph{who}
is interpreted relative to @emph{which} (a process identifier for
@ref{238d,,PRIO_PROCESS}, process group identifier for @ref{238e,,PRIO_PGRP}, and a
user ID for @ref{238f,,PRIO_USER}).  A zero value for @emph{who} denotes
(respectively) the calling process, the process group of the calling process,
or the real user ID of the calling process.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex PRIO_PROCESS (in module os)
@geindex PRIO_PGRP (in module os)
@geindex PRIO_USER (in module os)
@anchor{library/os os PRIO_PROCESS}@anchor{238d}
@deffn {Data} os.PRIO_PROCESS
@anchor{library/os os PRIO_PGRP}@anchor{238e}
@deffnx {Data} os.PRIO_PGRP
@anchor{library/os os PRIO_USER}@anchor{238f}
@deffnx {Data} os.PRIO_USER

Parameters for the @ref{c92,,getpriority()} and @ref{c93,,setpriority()} functions.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex getresuid() (in module os)
@anchor{library/os os getresuid}@anchor{ead}
@deffn {Function} os.getresuid ()

Return a tuple (ruid, euid, suid) denoting the current process’s
real, effective, and saved user ids.

@ref{1629,,Availability}: Unix.


New in version 3.2.

@end deffn

@geindex getresgid() (in module os)
@anchor{library/os os getresgid}@anchor{eac}
@deffn {Function} os.getresgid ()

Return a tuple (rgid, egid, sgid) denoting the current process’s
real, effective, and saved group ids.

@ref{1629,,Availability}: Unix.


New in version 3.2.

@end deffn

@geindex getuid() (in module os)
@anchor{library/os os getuid}@anchor{2390}
@deffn {Function} os.getuid ()

@geindex user; id

Return the current process’s real user id.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex initgroups() (in module os)
@anchor{library/os os initgroups}@anchor{eb0}
@deffn {Function} os.initgroups (username, gid)

Call the system initgroups() to initialize the group access list with all of
the groups of which the specified username is a member, plus the specified
group id.

@ref{1629,,Availability}: Unix.


New in version 3.2.

@end deffn

@geindex putenv() (in module os)
@anchor{library/os os putenv}@anchor{410}
@deffn {Function} os.putenv (key, value)

@geindex environment variables; setting

Set the environment variable named @emph{key} to the string @emph{value}.  Such
changes to the environment affect subprocesses started with @ref{fbb,,os.system()},
@ref{59f,,popen()} or @ref{bb9,,fork()} and @ref{2391,,execv()}.

Assignments to items in @code{os.environ} are automatically translated into
corresponding calls to @ref{410,,putenv()}; however, calls to @ref{410,,putenv()}
don’t update @code{os.environ}, so it is actually preferable to assign to items
of @code{os.environ}.

@cartouche
@quotation Note 
On some platforms, including FreeBSD and macOS, setting @code{environ} may
cause memory leaks. Refer to the system documentation for @code{putenv()}.
@end quotation
@end cartouche

Raises an @ref{1130,,auditing event} @code{os.putenv} with arguments @code{key}, @code{value}.


Changed in version 3.9: The function is now always available.

@end deffn

@geindex setegid() (in module os)
@anchor{library/os os setegid}@anchor{2392}
@deffn {Function} os.setegid (egid)

Set the current process’s effective group id.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex seteuid() (in module os)
@anchor{library/os os seteuid}@anchor{2393}
@deffn {Function} os.seteuid (euid)

Set the current process’s effective user id.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex setgid() (in module os)
@anchor{library/os os setgid}@anchor{2394}
@deffn {Function} os.setgid (gid)

Set the current process’ group id.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex setgroups() (in module os)
@anchor{library/os os setgroups}@anchor{2386}
@deffn {Function} os.setgroups (groups)

Set the list of supplemental group ids associated with the current process to
@emph{groups}. @emph{groups} must be a sequence, and each element must be an integer
identifying a group. This operation is typically available only to the superuser.

@ref{1629,,Availability}: Unix.

@cartouche
@quotation Note 
On macOS, the length of @emph{groups} may not exceed the
system-defined maximum number of effective group ids, typically 16.
See the documentation for @ref{2385,,getgroups()} for cases where it may not
return the same group list set by calling setgroups().
@end quotation
@end cartouche
@end deffn

@geindex setpgrp() (in module os)
@anchor{library/os os setpgrp}@anchor{2395}
@deffn {Function} os.setpgrp ()

Call the system call @code{setpgrp()} or @code{setpgrp(0, 0)} depending on
which version is implemented (if any).  See the Unix manual for the semantics.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex setpgid() (in module os)
@anchor{library/os os setpgid}@anchor{2396}
@deffn {Function} os.setpgid (pid, pgrp)

Call the system call @code{setpgid()} to set the process group id of the
process with id @emph{pid} to the process group with id @emph{pgrp}.  See the Unix manual
for the semantics.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex setpriority() (in module os)
@anchor{library/os os setpriority}@anchor{c93}
@deffn {Function} os.setpriority (which, who, priority)

@geindex process; scheduling priority

Set program scheduling priority. The value @emph{which} is one of
@ref{238d,,PRIO_PROCESS}, @ref{238e,,PRIO_PGRP}, or @ref{238f,,PRIO_USER}, and @emph{who}
is interpreted relative to @emph{which} (a process identifier for
@ref{238d,,PRIO_PROCESS}, process group identifier for @ref{238e,,PRIO_PGRP}, and a
user ID for @ref{238f,,PRIO_USER}). A zero value for @emph{who} denotes
(respectively) the calling process, the process group of the calling process,
or the real user ID of the calling process.
@emph{priority} is a value in the range -20 to 19. The default priority is 0;
lower priorities cause more favorable scheduling.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex setregid() (in module os)
@anchor{library/os os setregid}@anchor{2397}
@deffn {Function} os.setregid (rgid, egid)

Set the current process’s real and effective group ids.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex setresgid() (in module os)
@anchor{library/os os setresgid}@anchor{eae}
@deffn {Function} os.setresgid (rgid, egid, sgid)

Set the current process’s real, effective, and saved group ids.

@ref{1629,,Availability}: Unix.


New in version 3.2.

@end deffn

@geindex setresuid() (in module os)
@anchor{library/os os setresuid}@anchor{eaf}
@deffn {Function} os.setresuid (ruid, euid, suid)

Set the current process’s real, effective, and saved user ids.

@ref{1629,,Availability}: Unix.


New in version 3.2.

@end deffn

@geindex setreuid() (in module os)
@anchor{library/os os setreuid}@anchor{2398}
@deffn {Function} os.setreuid (ruid, euid)

Set the current process’s real and effective user ids.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex getsid() (in module os)
@anchor{library/os os getsid}@anchor{2399}
@deffn {Function} os.getsid (pid)

Call the system call @code{getsid()}.  See the Unix manual for the semantics.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex setsid() (in module os)
@anchor{library/os os setsid}@anchor{1277}
@deffn {Function} os.setsid ()

Call the system call @code{setsid()}.  See the Unix manual for the semantics.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex setuid() (in module os)
@anchor{library/os os setuid}@anchor{239a}
@deffn {Function} os.setuid (uid)

@geindex user; id@comma{} setting

Set the current process’s user id.

@ref{1629,,Availability}: Unix.
@end deffn

@c placed in this section since it relates to errno.... a little weak

@geindex strerror() (in module os)
@anchor{library/os os strerror}@anchor{239b}
@deffn {Function} os.strerror (code)

Return the error message corresponding to the error code in @emph{code}.
On platforms where @code{strerror()} returns @code{NULL} when given an unknown
error number, @ref{17c,,ValueError} is raised.
@end deffn

@geindex supports_bytes_environ (in module os)
@anchor{library/os os supports_bytes_environ}@anchor{db6}
@deffn {Data} os.supports_bytes_environ

@code{True} if the native OS type of the environment is bytes (eg. @code{False} on
Windows).


New in version 3.2.

@end deffn

@geindex umask() (in module os)
@anchor{library/os os umask}@anchor{239c}
@deffn {Function} os.umask (mask)

Set the current numeric umask and return the previous umask.
@end deffn

@geindex uname() (in module os)
@anchor{library/os os uname}@anchor{ca9}
@deffn {Function} os.uname ()

@geindex gethostname() (in module socket)
@geindex gethostbyaddr() (in module socket)

Returns information identifying the current operating system.
The return value is an object with five attributes:


@itemize *

@item 
@code{sysname} - operating system name

@item 
@code{nodename} - name of machine on network (implementation-defined)

@item 
@code{release} - operating system release

@item 
@code{version} - operating system version

@item 
@code{machine} - hardware identifier
@end itemize

For backwards compatibility, this object is also iterable, behaving
like a five-tuple containing @code{sysname}, @code{nodename},
@code{release}, @code{version}, and @code{machine}
in that order.

Some systems truncate @code{nodename} to 8 characters or to the
leading component; a better way to get the hostname is
@ref{239d,,socket.gethostname()}  or even
@code{socket.gethostbyaddr(socket.gethostname())}.

@ref{1629,,Availability}: recent flavors of Unix.


Changed in version 3.3: Return type changed from a tuple to a tuple-like object
with named attributes.

@end deffn

@geindex unsetenv() (in module os)
@anchor{library/os os unsetenv}@anchor{40f}
@deffn {Function} os.unsetenv (key)

@geindex environment variables; deleting

Unset (delete) the environment variable named @emph{key}. Such changes to the
environment affect subprocesses started with @ref{fbb,,os.system()}, @ref{59f,,popen()} or
@ref{bb9,,fork()} and @ref{2391,,execv()}.

Deletion of items in @code{os.environ} is automatically translated into a
corresponding call to @ref{40f,,unsetenv()}; however, calls to @ref{40f,,unsetenv()}
don’t update @code{os.environ}, so it is actually preferable to delete items of
@code{os.environ}.

Raises an @ref{1130,,auditing event} @code{os.unsetenv} with argument @code{key}.


Changed in version 3.9: The function is now always available and is also available on Windows.

@end deffn

@node File Object Creation,File Descriptor Operations,Process Parameters,os — Miscellaneous operating system interfaces
@anchor{library/os file-object-creation}@anchor{239e}@anchor{library/os os-newstreams}@anchor{239f}
@subsubsection File Object Creation


These functions create new @ref{d68,,file objects}.  (See also
@ref{8ee,,open()} for opening file descriptors.)

@geindex fdopen() (in module os)
@anchor{library/os os fdopen}@anchor{179a}
@deffn {Function} os.fdopen (fd, *args, **kwargs)

Return an open file object connected to the file descriptor @emph{fd}.  This is an
alias of the @ref{1c8,,open()} built-in function and accepts the same arguments.
The only difference is that the first argument of @ref{179a,,fdopen()} must always
be an integer.
@end deffn

@node File Descriptor Operations,Files and Directories,File Object Creation,os — Miscellaneous operating system interfaces
@anchor{library/os file-descriptor-operations}@anchor{23a0}@anchor{library/os os-fd-ops}@anchor{23a1}
@subsubsection File Descriptor Operations


These functions operate on I/O streams referenced using file descriptors.

File descriptors are small integers corresponding to a file that has been opened
by the current process.  For example, standard input is usually file descriptor
0, standard output is 1, and standard error is 2.  Further files opened by a
process will then be assigned 3, 4, 5, and so forth.  The name “file descriptor”
is slightly deceptive; on Unix platforms, sockets and pipes are also referenced
by file descriptors.

The @ref{23a2,,fileno()} method can be used to obtain the file descriptor
associated with a @ref{d68,,file object} when required.  Note that using the file
descriptor directly will bypass the file object methods, ignoring aspects such
as internal buffering of data.

@geindex close() (in module os)
@anchor{library/os os close}@anchor{6a1}
@deffn {Function} os.close (fd)

Close file descriptor @emph{fd}.

@cartouche
@quotation Note 
This function is intended for low-level I/O and must be applied to a file
descriptor as returned by @ref{8ee,,os.open()} or @ref{23a3,,pipe()}.  To close a “file
object” returned by the built-in function @ref{1c8,,open()} or by @ref{59f,,popen()} or
@ref{179a,,fdopen()}, use its @ref{12d8,,close()} method.
@end quotation
@end cartouche
@end deffn

@geindex closerange() (in module os)
@anchor{library/os os closerange}@anchor{23a4}
@deffn {Function} os.closerange (fd_low, fd_high)

Close all file descriptors from @emph{fd_low} (inclusive) to @emph{fd_high} (exclusive),
ignoring errors. Equivalent to (but much faster than):

@example
for fd in range(fd_low, fd_high):
    try:
        os.close(fd)
    except OSError:
        pass
@end example
@end deffn

@geindex copy_file_range() (in module os)
@anchor{library/os os copy_file_range}@anchor{23a5}
@deffn {Function} os.copy_file_range (src, dst, count, offset_src=None, offset_dst=None)

Copy @emph{count} bytes from file descriptor @emph{src}, starting from offset
@emph{offset_src}, to file descriptor @emph{dst}, starting from offset @emph{offset_dst}.
If @emph{offset_src} is None, then @emph{src} is read from the current position;
respectively for @emph{offset_dst}. The files pointed by @emph{src} and @emph{dst}
must reside in the same filesystem, otherwise an @ref{2a1,,OSError} is
raised with @ref{1b76,,errno} set to @ref{23a6,,errno.EXDEV}.

This copy is done without the additional cost of transferring data
from the kernel to user space and then back into the kernel. Additionally,
some filesystems could implement extra optimizations. The copy is done as if
both files are opened as binary.

The return value is the amount of bytes copied. This could be less than the
amount requested.

@ref{1629,,Availability}: Linux kernel >= 4.5 or glibc >= 2.27.


New in version 3.8.

@end deffn

@geindex device_encoding() (in module os)
@anchor{library/os os device_encoding}@anchor{10f6}
@deffn {Function} os.device_encoding (fd)

Return a string describing the encoding of the device associated with @emph{fd}
if it is connected to a terminal; else return @ref{182,,None}.

On Unix, if the @ref{5f2,,Python UTF-8 Mode} is enabled, return
@code{'UTF-8'} rather than the device encoding.


Changed in version 3.10: On Unix, the function now implements the Python UTF-8 Mode.

@end deffn

@geindex dup() (in module os)
@anchor{library/os os dup}@anchor{127f}
@deffn {Function} os.dup (fd)

Return a duplicate of file descriptor @emph{fd}. The new file descriptor is
@ref{a6e,,non-inheritable}.

On Windows, when duplicating a standard stream (0: stdin, 1: stdout,
2: stderr), the new file descriptor is @ref{a6e,,inheritable}.


Changed in version 3.4: The new file descriptor is now non-inheritable.

@end deffn

@geindex dup2() (in module os)
@anchor{library/os os dup2}@anchor{68d}
@deffn {Function} os.dup2 (fd, fd2, inheritable=True)

Duplicate file descriptor @emph{fd} to @emph{fd2}, closing the latter first if
necessary. Return @emph{fd2}. The new file descriptor is @ref{a6e,,inheritable} by default or non-inheritable if @emph{inheritable}
is @code{False}.


Changed in version 3.4: Add the optional @emph{inheritable} parameter.



Changed in version 3.7: Return @emph{fd2} on success. Previously, @code{None} was always returned.

@end deffn

@geindex fchmod() (in module os)
@anchor{library/os os fchmod}@anchor{8e5}
@deffn {Function} os.fchmod (fd, mode)

Change the mode of the file given by @emph{fd} to the numeric @emph{mode}.  See the
docs for @ref{c80,,chmod()} for possible values of @emph{mode}.  As of Python 3.3, this
is equivalent to @code{os.chmod(fd, mode)}.

Raises an @ref{1130,,auditing event} @code{os.chmod} with arguments @code{path}, @code{mode}, @code{dir_fd}.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex fchown() (in module os)
@anchor{library/os os fchown}@anchor{8e6}
@deffn {Function} os.fchown (fd, uid, gid)

Change the owner and group id of the file given by @emph{fd} to the numeric @emph{uid}
and @emph{gid}.  To leave one of the ids unchanged, set it to -1.  See
@ref{c81,,chown()}.  As of Python 3.3, this is equivalent to @code{os.chown(fd, uid,
gid)}.

Raises an @ref{1130,,auditing event} @code{os.chown} with arguments @code{path}, @code{uid}, @code{gid}, @code{dir_fd}.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex fdatasync() (in module os)
@anchor{library/os os fdatasync}@anchor{8e7}
@deffn {Function} os.fdatasync (fd)

Force write of file with filedescriptor @emph{fd} to disk. Does not force update of
metadata.

@ref{1629,,Availability}: Unix.

@cartouche
@quotation Note 
This function is not available on MacOS.
@end quotation
@end cartouche
@end deffn

@geindex fpathconf() (in module os)
@anchor{library/os os fpathconf}@anchor{23a7}
@deffn {Function} os.fpathconf (fd, name)

Return system configuration information relevant to an open file. @emph{name}
specifies the configuration value to retrieve; it may be a string which is the
name of a defined system value; these names are specified in a number of
standards (POSIX.1, Unix 95, Unix 98, and others).  Some platforms define
additional names as well.  The names known to the host operating system are
given in the @code{pathconf_names} dictionary.  For configuration variables not
included in that mapping, passing an integer for @emph{name} is also accepted.

If @emph{name} is a string and is not known, @ref{17c,,ValueError} is raised.  If a
specific value for @emph{name} is not supported by the host system, even if it is
included in @code{pathconf_names}, an @ref{2a1,,OSError} is raised with
@ref{1298,,errno.EINVAL} for the error number.

As of Python 3.3, this is equivalent to @code{os.pathconf(fd, name)}.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex fstat() (in module os)
@anchor{library/os os fstat}@anchor{8e8}
@deffn {Function} os.fstat (fd)

Get the status of the file descriptor @emph{fd}. Return a @ref{134d,,stat_result}
object.

As of Python 3.3, this is equivalent to @code{os.stat(fd)}.


@subsubheading See also


The @ref{516,,stat()} function.

@end deffn

@geindex fstatvfs() (in module os)
@anchor{library/os os fstatvfs}@anchor{8e9}
@deffn {Function} os.fstatvfs (fd)

Return information about the filesystem containing the file associated with
file descriptor @emph{fd}, like @ref{c8f,,statvfs()}.  As of Python 3.3, this is
equivalent to @code{os.statvfs(fd)}.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex fsync() (in module os)
@anchor{library/os os fsync}@anchor{8ea}
@deffn {Function} os.fsync (fd)

Force write of file with filedescriptor @emph{fd} to disk.  On Unix, this calls the
native @code{fsync()} function; on Windows, the MS @code{_commit()} function.

If you’re starting with a buffered Python @ref{d68,,file object} @emph{f}, first do
@code{f.flush()}, and then do @code{os.fsync(f.fileno())}, to ensure that all internal
buffers associated with @emph{f} are written to disk.

@ref{1629,,Availability}: Unix, Windows.
@end deffn

@geindex ftruncate() (in module os)
@anchor{library/os os ftruncate}@anchor{8eb}
@deffn {Function} os.ftruncate (fd, length)

Truncate the file corresponding to file descriptor @emph{fd}, so that it is at
most @emph{length} bytes in size.  As of Python 3.3, this is equivalent to
@code{os.truncate(fd, length)}.

Raises an @ref{1130,,auditing event} @code{os.truncate} with arguments @code{fd}, @code{length}.

@ref{1629,,Availability}: Unix, Windows.


Changed in version 3.5: Added support for Windows

@end deffn

@geindex get_blocking() (in module os)
@anchor{library/os os get_blocking}@anchor{9a0}
@deffn {Function} os.get_blocking (fd)

Get the blocking mode of the file descriptor: @code{False} if the
@ref{9a2,,O_NONBLOCK} flag is set, @code{True} if the flag is cleared.

See also @ref{9a1,,set_blocking()} and @ref{23a8,,socket.socket.setblocking()}.

@ref{1629,,Availability}: Unix.


New in version 3.5.

@end deffn

@geindex isatty() (in module os)
@anchor{library/os os isatty}@anchor{10f5}
@deffn {Function} os.isatty (fd)

Return @code{True} if the file descriptor @emph{fd} is open and connected to a
tty(-like) device, else @code{False}.
@end deffn

@geindex lockf() (in module os)
@anchor{library/os os lockf}@anchor{ca6}
@deffn {Function} os.lockf (fd, cmd, len)

Apply, test or remove a POSIX lock on an open file descriptor.
@emph{fd} is an open file descriptor.
@emph{cmd} specifies the command to use - one of @ref{23a9,,F_LOCK}, @ref{23aa,,F_TLOCK},
@ref{23ab,,F_ULOCK} or @ref{23ac,,F_TEST}.
@emph{len} specifies the section of the file to lock.

Raises an @ref{1130,,auditing event} @code{os.lockf} with arguments @code{fd}, @code{cmd}, @code{len}.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex F_LOCK (in module os)
@geindex F_TLOCK (in module os)
@geindex F_ULOCK (in module os)
@geindex F_TEST (in module os)
@anchor{library/os os F_LOCK}@anchor{23a9}
@deffn {Data} os.F_LOCK
@anchor{library/os os F_TLOCK}@anchor{23aa}
@deffnx {Data} os.F_TLOCK
@anchor{library/os os F_ULOCK}@anchor{23ab}
@deffnx {Data} os.F_ULOCK
@anchor{library/os os F_TEST}@anchor{23ac}
@deffnx {Data} os.F_TEST

Flags that specify what action @ref{ca6,,lockf()} will take.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex lseek() (in module os)
@anchor{library/os os lseek}@anchor{caa}
@deffn {Function} os.lseek (fd, pos, how)

Set the current position of file descriptor @emph{fd} to position @emph{pos}, modified
by @emph{how}: @ref{f8e,,SEEK_SET} or @code{0} to set the position relative to the
beginning of the file; @ref{f8f,,SEEK_CUR} or @code{1} to set it relative to the
current position; @ref{f90,,SEEK_END} or @code{2} to set it relative to the end of
the file. Return the new cursor position in bytes, starting from the beginning.
@end deffn

@geindex SEEK_SET (in module os)
@geindex SEEK_CUR (in module os)
@geindex SEEK_END (in module os)
@anchor{library/os os SEEK_SET}@anchor{f8e}
@deffn {Data} os.SEEK_SET
@anchor{library/os os SEEK_CUR}@anchor{f8f}
@deffnx {Data} os.SEEK_CUR
@anchor{library/os os SEEK_END}@anchor{f90}
@deffnx {Data} os.SEEK_END

Parameters to the @ref{caa,,lseek()} function. Their values are 0, 1, and 2,
respectively.


New in version 3.3: Some operating systems could support additional values, like
@code{os.SEEK_HOLE} or @code{os.SEEK_DATA}.

@end deffn

@geindex open() (in module os)
@anchor{library/os os open}@anchor{8ee}
@deffn {Function} os.open (path, flags, mode=511, *, dir_fd=None)

Open the file @emph{path} and set various flags according to @emph{flags} and possibly
its mode according to @emph{mode}.  When computing @emph{mode}, the current umask value
is first masked out.  Return the file descriptor for the newly opened file.
The new file descriptor is @ref{a6e,,non-inheritable}.

For a description of the flag and mode values, see the C run-time documentation;
flag constants (like @ref{23ad,,O_RDONLY} and @ref{23ae,,O_WRONLY}) are defined in
the @ref{c0,,os} module.  In particular, on Windows adding
@ref{23af,,O_BINARY} is needed to open files in binary mode.

This function can support @ref{c7c,,paths relative to directory descriptors} with the @emph{dir_fd} parameter.

Raises an @ref{1130,,auditing event} @code{open} with arguments @code{path}, @code{mode}, @code{flags}.


Changed in version 3.4: The new file descriptor is now non-inheritable.


@cartouche
@quotation Note 
This function is intended for low-level I/O.  For normal usage, use the
built-in function @ref{1c8,,open()}, which returns a @ref{d68,,file object} with
@code{read()} and @code{write()} methods (and many more).  To
wrap a file descriptor in a file object, use @ref{179a,,fdopen()}.
@end quotation
@end cartouche


New in version 3.3: The @emph{dir_fd} argument.



Changed in version 3.5: If the system call is interrupted and the signal handler does not raise an
exception, the function now retries the system call instead of raising an
@ref{8e3,,InterruptedError} exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

The following constants are options for the @emph{flags} parameter to the
@ref{8ee,,open()} function.  They can be combined using the bitwise OR operator
@code{|}.  Some of them are not available on all platforms.  For descriptions of
their availability and use, consult the @code{open(2)} manual page on Unix
or the MSDN@footnote{https://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx} on Windows.

@geindex O_RDONLY (in module os)
@geindex O_WRONLY (in module os)
@geindex O_RDWR (in module os)
@geindex O_APPEND (in module os)
@geindex O_CREAT (in module os)
@geindex O_EXCL (in module os)
@geindex O_TRUNC (in module os)
@anchor{library/os os O_RDONLY}@anchor{23ad}
@deffn {Data} os.O_RDONLY
@anchor{library/os os O_WRONLY}@anchor{23ae}
@deffnx {Data} os.O_WRONLY
@anchor{library/os os O_RDWR}@anchor{23b0}
@deffnx {Data} os.O_RDWR
@anchor{library/os os O_APPEND}@anchor{23b1}
@deffnx {Data} os.O_APPEND
@anchor{library/os os O_CREAT}@anchor{23b2}
@deffnx {Data} os.O_CREAT
@anchor{library/os os O_EXCL}@anchor{2105}
@deffnx {Data} os.O_EXCL
@anchor{library/os os O_TRUNC}@anchor{23b3}
@deffnx {Data} os.O_TRUNC

The above constants are available on Unix and Windows.
@end deffn

@geindex O_DSYNC (in module os)
@geindex O_RSYNC (in module os)
@geindex O_SYNC (in module os)
@geindex O_NDELAY (in module os)
@geindex O_NONBLOCK (in module os)
@geindex O_NOCTTY (in module os)
@geindex O_CLOEXEC (in module os)
@anchor{library/os os O_DSYNC}@anchor{23b4}
@deffn {Data} os.O_DSYNC
@anchor{library/os os O_RSYNC}@anchor{23b5}
@deffnx {Data} os.O_RSYNC
@anchor{library/os os O_SYNC}@anchor{23b6}
@deffnx {Data} os.O_SYNC
@anchor{library/os os O_NDELAY}@anchor{23b7}
@deffnx {Data} os.O_NDELAY
@anchor{library/os os O_NONBLOCK}@anchor{9a2}
@deffnx {Data} os.O_NONBLOCK
@anchor{library/os os O_NOCTTY}@anchor{23b8}
@deffnx {Data} os.O_NOCTTY
@anchor{library/os os O_CLOEXEC}@anchor{c13}
@deffnx {Data} os.O_CLOEXEC

The above constants are only available on Unix.


Changed in version 3.3: Add @ref{c13,,O_CLOEXEC} constant.

@end deffn

@geindex O_BINARY (in module os)
@geindex O_NOINHERIT (in module os)
@geindex O_SHORT_LIVED (in module os)
@geindex O_TEMPORARY (in module os)
@geindex O_RANDOM (in module os)
@geindex O_SEQUENTIAL (in module os)
@geindex O_TEXT (in module os)
@anchor{library/os os O_BINARY}@anchor{23af}
@deffn {Data} os.O_BINARY
@anchor{library/os os O_NOINHERIT}@anchor{23b9}
@deffnx {Data} os.O_NOINHERIT
@anchor{library/os os O_SHORT_LIVED}@anchor{23ba}
@deffnx {Data} os.O_SHORT_LIVED
@anchor{library/os os O_TEMPORARY}@anchor{23bb}
@deffnx {Data} os.O_TEMPORARY
@anchor{library/os os O_RANDOM}@anchor{23bc}
@deffnx {Data} os.O_RANDOM
@anchor{library/os os O_SEQUENTIAL}@anchor{23bd}
@deffnx {Data} os.O_SEQUENTIAL
@anchor{library/os os O_TEXT}@anchor{23be}
@deffnx {Data} os.O_TEXT

The above constants are only available on Windows.
@end deffn

@geindex O_EVTONLY (in module os)
@geindex O_FSYNC (in module os)
@geindex O_SYMLINK (in module os)
@geindex O_NOFOLLOW_ANY (in module os)
@anchor{library/os os O_EVTONLY}@anchor{29b}
@deffn {Data} os.O_EVTONLY
@anchor{library/os os O_FSYNC}@anchor{29c}
@deffnx {Data} os.O_FSYNC
@anchor{library/os os O_SYMLINK}@anchor{29d}
@deffnx {Data} os.O_SYMLINK
@anchor{library/os os O_NOFOLLOW_ANY}@anchor{29e}
@deffnx {Data} os.O_NOFOLLOW_ANY

The above constants are only available on macOS.


Changed in version 3.10: Add @ref{29b,,O_EVTONLY}, @ref{29c,,O_FSYNC}, @ref{29d,,O_SYMLINK}
and @ref{29e,,O_NOFOLLOW_ANY} constants.

@end deffn

@geindex O_ASYNC (in module os)
@geindex O_DIRECT (in module os)
@geindex O_DIRECTORY (in module os)
@geindex O_NOFOLLOW (in module os)
@geindex O_NOATIME (in module os)
@geindex O_PATH (in module os)
@geindex O_TMPFILE (in module os)
@geindex O_SHLOCK (in module os)
@geindex O_EXLOCK (in module os)
@anchor{library/os os O_ASYNC}@anchor{23bf}
@deffn {Data} os.O_ASYNC
@anchor{library/os os O_DIRECT}@anchor{23c0}
@deffnx {Data} os.O_DIRECT
@anchor{library/os os O_DIRECTORY}@anchor{23c1}
@deffnx {Data} os.O_DIRECTORY
@anchor{library/os os O_NOFOLLOW}@anchor{23c2}
@deffnx {Data} os.O_NOFOLLOW
@anchor{library/os os O_NOATIME}@anchor{23c3}
@deffnx {Data} os.O_NOATIME
@anchor{library/os os O_PATH}@anchor{af3}
@deffnx {Data} os.O_PATH
@anchor{library/os os O_TMPFILE}@anchor{af4}
@deffnx {Data} os.O_TMPFILE
@anchor{library/os os O_SHLOCK}@anchor{f91}
@deffnx {Data} os.O_SHLOCK
@anchor{library/os os O_EXLOCK}@anchor{f92}
@deffnx {Data} os.O_EXLOCK

The above constants are extensions and not present if they are not defined by
the C library.


Changed in version 3.4: Add @ref{af3,,O_PATH} on systems that support it.
Add @ref{af4,,O_TMPFILE}, only available on Linux Kernel 3.11
  or newer.

@end deffn

@geindex openpty() (in module os)
@anchor{library/os os openpty}@anchor{23c4}
@deffn {Function} os.openpty ()

@geindex module; pty

Open a new pseudo-terminal pair. Return a pair of file descriptors
@code{(master, slave)} for the pty and the tty, respectively. The new file
descriptors are @ref{a6e,,non-inheritable}. For a (slightly) more
portable approach, use the @ref{d0,,pty} module.

@ref{1629,,Availability}: some flavors of Unix.


Changed in version 3.4: The new file descriptors are now non-inheritable.

@end deffn

@geindex pipe() (in module os)
@anchor{library/os os pipe}@anchor{23a3}
@deffn {Function} os.pipe ()

Create a pipe.  Return a pair of file descriptors @code{(r, w)} usable for
reading and writing, respectively. The new file descriptor is
@ref{a6e,,non-inheritable}.

@ref{1629,,Availability}: Unix, Windows.


Changed in version 3.4: The new file descriptors are now non-inheritable.

@end deffn

@geindex pipe2() (in module os)
@anchor{library/os os pipe2}@anchor{c7b}
@deffn {Function} os.pipe2 (flags)

Create a pipe with @emph{flags} set atomically.
@emph{flags} can be constructed by ORing together one or more of these values:
@ref{9a2,,O_NONBLOCK}, @ref{c13,,O_CLOEXEC}.
Return a pair of file descriptors @code{(r, w)} usable for reading and writing,
respectively.

@ref{1629,,Availability}: some flavors of Unix.


New in version 3.3.

@end deffn

@geindex posix_fallocate() (in module os)
@anchor{library/os os posix_fallocate}@anchor{8f0}
@deffn {Function} os.posix_fallocate (fd, offset, len)

Ensures that enough disk space is allocated for the file specified by @emph{fd}
starting from @emph{offset} and continuing for @emph{len} bytes.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex posix_fadvise() (in module os)
@anchor{library/os os posix_fadvise}@anchor{8ef}
@deffn {Function} os.posix_fadvise (fd, offset, len, advice)

Announces an intention to access data in a specific pattern thus allowing
the kernel to make optimizations.
The advice applies to the region of the file specified by @emph{fd} starting at
@emph{offset} and continuing for @emph{len} bytes.
@emph{advice} is one of @ref{23c5,,POSIX_FADV_NORMAL}, @ref{23c6,,POSIX_FADV_SEQUENTIAL},
@ref{23c7,,POSIX_FADV_RANDOM}, @ref{23c8,,POSIX_FADV_NOREUSE},
@ref{23c9,,POSIX_FADV_WILLNEED} or @ref{23ca,,POSIX_FADV_DONTNEED}.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex POSIX_FADV_NORMAL (in module os)
@geindex POSIX_FADV_SEQUENTIAL (in module os)
@geindex POSIX_FADV_RANDOM (in module os)
@geindex POSIX_FADV_NOREUSE (in module os)
@geindex POSIX_FADV_WILLNEED (in module os)
@geindex POSIX_FADV_DONTNEED (in module os)
@anchor{library/os os POSIX_FADV_NORMAL}@anchor{23c5}
@deffn {Data} os.POSIX_FADV_NORMAL
@anchor{library/os os POSIX_FADV_SEQUENTIAL}@anchor{23c6}
@deffnx {Data} os.POSIX_FADV_SEQUENTIAL
@anchor{library/os os POSIX_FADV_RANDOM}@anchor{23c7}
@deffnx {Data} os.POSIX_FADV_RANDOM
@anchor{library/os os POSIX_FADV_NOREUSE}@anchor{23c8}
@deffnx {Data} os.POSIX_FADV_NOREUSE
@anchor{library/os os POSIX_FADV_WILLNEED}@anchor{23c9}
@deffnx {Data} os.POSIX_FADV_WILLNEED
@anchor{library/os os POSIX_FADV_DONTNEED}@anchor{23ca}
@deffnx {Data} os.POSIX_FADV_DONTNEED

Flags that can be used in @emph{advice} in @ref{8ef,,posix_fadvise()} that specify
the access pattern that is likely to be used.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex pread() (in module os)
@anchor{library/os os pread}@anchor{688}
@deffn {Function} os.pread (fd, n, offset)

Read at most @emph{n} bytes from file descriptor @emph{fd} at a position of @emph{offset},
leaving the file offset unchanged.

Return a bytestring containing the bytes read. If the end of the file
referred to by @emph{fd} has been reached, an empty bytes object is returned.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex preadv() (in module os)
@anchor{library/os os preadv}@anchor{686}
@deffn {Function} os.preadv (fd, buffers, offset, flags=0)

Read from a file descriptor @emph{fd} at a position of @emph{offset} into mutable
@ref{87e,,bytes-like objects} @emph{buffers}, leaving the file
offset unchanged.  Transfer data into each buffer until it is full and then
move on to the next buffer in the sequence to hold the rest of the data.

The flags argument contains a bitwise OR of zero or more of the following
flags:


@itemize -

@item 
@ref{23cb,,RWF_HIPRI}

@item 
@ref{23cc,,RWF_NOWAIT}
@end itemize

Return the total number of bytes actually read which can be less than the
total capacity of all the objects.

The operating system may set a limit (@ref{23cd,,sysconf()} value
@code{'SC_IOV_MAX'}) on the number of buffers that can be used.

Combine the functionality of @ref{687,,os.readv()} and @ref{688,,os.pread()}.

@ref{1629,,Availability}: Linux 2.6.30 and newer, FreeBSD 6.0 and newer,
OpenBSD 2.7 and newer, AIX 7.1 and newer. Using flags requires
Linux 4.6 or newer.


New in version 3.7.

@end deffn

@geindex RWF_NOWAIT (in module os)
@anchor{library/os os RWF_NOWAIT}@anchor{23cc}
@deffn {Data} os.RWF_NOWAIT

Do not wait for data which is not immediately available. If this flag is
specified, the system call will return instantly if it would have to read
data from the backing storage or wait for a lock.

If some data was successfully read, it will return the number of bytes read.
If no bytes were read, it will return @code{-1} and set errno to
@ref{23ce,,errno.EAGAIN}.

@ref{1629,,Availability}: Linux 4.14 and newer.


New in version 3.7.

@end deffn

@geindex RWF_HIPRI (in module os)
@anchor{library/os os RWF_HIPRI}@anchor{23cb}
@deffn {Data} os.RWF_HIPRI

High priority read/write. Allows block-based filesystems to use polling
of the device, which provides lower latency, but may use additional
resources.

Currently, on Linux, this feature is usable only on a file descriptor opened
using the @ref{23c0,,O_DIRECT} flag.

@ref{1629,,Availability}: Linux 4.6 and newer.


New in version 3.7.

@end deffn

@geindex pwrite() (in module os)
@anchor{library/os os pwrite}@anchor{68b}
@deffn {Function} os.pwrite (fd, str, offset)

Write the bytestring in @emph{str} to file descriptor @emph{fd} at position of
@emph{offset}, leaving the file offset unchanged.

Return the number of bytes actually written.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex pwritev() (in module os)
@anchor{library/os os pwritev}@anchor{689}
@deffn {Function} os.pwritev (fd, buffers, offset, flags=0)

Write the @emph{buffers} contents to file descriptor @emph{fd} at a offset @emph{offset},
leaving the file offset unchanged.  @emph{buffers} must be a sequence of
@ref{87e,,bytes-like objects}. Buffers are processed in
array order. Entire contents of the first buffer is written before
proceeding to the second, and so on.

The flags argument contains a bitwise OR of zero or more of the following
flags:


@itemize -

@item 
@ref{23cf,,RWF_DSYNC}

@item 
@ref{23d0,,RWF_SYNC}

@item 
@ref{11c9,,RWF_APPEND}
@end itemize

Return the total number of bytes actually written.

The operating system may set a limit (@ref{23cd,,sysconf()} value
@code{'SC_IOV_MAX'}) on the number of buffers that can be used.

Combine the functionality of @ref{68a,,os.writev()} and @ref{68b,,os.pwrite()}.

@ref{1629,,Availability}: Linux 2.6.30 and newer, FreeBSD 6.0 and newer,
OpenBSD 2.7 and newer, AIX 7.1 and newer. Using flags requires
Linux 4.7 or newer.


New in version 3.7.

@end deffn

@geindex RWF_DSYNC (in module os)
@anchor{library/os os RWF_DSYNC}@anchor{23cf}
@deffn {Data} os.RWF_DSYNC

Provide a per-write equivalent of the @ref{23b4,,O_DSYNC} @ref{8ee,,os.open()} flag.
This flag effect applies only to the data range written by the system call.

@ref{1629,,Availability}: Linux 4.7 and newer.


New in version 3.7.

@end deffn

@geindex RWF_SYNC (in module os)
@anchor{library/os os RWF_SYNC}@anchor{23d0}
@deffn {Data} os.RWF_SYNC

Provide a per-write equivalent of the @ref{23b6,,O_SYNC} @ref{8ee,,os.open()} flag.
This flag effect applies only to the data range written by the system call.

@ref{1629,,Availability}: Linux 4.7 and newer.


New in version 3.7.

@end deffn

@geindex RWF_APPEND (in module os)
@anchor{library/os os RWF_APPEND}@anchor{11c9}
@deffn {Data} os.RWF_APPEND

Provide a per-write equivalent of the @ref{23b1,,O_APPEND} @ref{8ee,,os.open()}
flag. This flag is meaningful only for @ref{689,,os.pwritev()}, and its
effect applies only to the data range written by the system call. The
@emph{offset} argument does not affect the write operation; the data is always
appended to the end of the file. However, if the @emph{offset} argument is
@code{-1}, the current file @emph{offset} is updated.

@ref{1629,,Availability}: Linux 4.16 and newer.


New in version 3.10.

@end deffn

@geindex read() (in module os)
@anchor{library/os os read}@anchor{8f1}
@deffn {Function} os.read (fd, n)

Read at most @emph{n} bytes from file descriptor @emph{fd}.

Return a bytestring containing the bytes read. If the end of the file
referred to by @emph{fd} has been reached, an empty bytes object is returned.

@cartouche
@quotation Note 
This function is intended for low-level I/O and must be applied to a file
descriptor as returned by @ref{8ee,,os.open()} or @ref{23a3,,pipe()}.  To read a
“file object” returned by the built-in function @ref{1c8,,open()} or by
@ref{59f,,popen()} or @ref{179a,,fdopen()}, or @ref{5ee,,sys.stdin}, use its
@code{read()} or @code{readline()} methods.
@end quotation
@end cartouche


Changed in version 3.5: If the system call is interrupted and the signal handler does not raise an
exception, the function now retries the system call instead of raising an
@ref{8e3,,InterruptedError} exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).

@end deffn

@geindex sendfile() (in module os)
@anchor{library/os os sendfile}@anchor{62c}
@deffn {Function} os.sendfile (out_fd, in_fd, offset, count)

@deffnx {Function} os.sendfile (out_fd, in_fd, offset, count, headers=(), trailers=(), flags=0)

Copy @emph{count} bytes from file descriptor @emph{in_fd} to file descriptor @emph{out_fd}
starting at @emph{offset}.
Return the number of bytes sent. When EOF is reached return @code{0}.

The first function notation is supported by all platforms that define
@ref{62c,,sendfile()}.

On Linux, if @emph{offset} is given as @code{None}, the bytes are read from the
current position of @emph{in_fd} and the position of @emph{in_fd} is updated.

The second case may be used on macOS and FreeBSD where @emph{headers} and
@emph{trailers} are arbitrary sequences of buffers that are written before and
after the data from @emph{in_fd} is written. It returns the same as the first case.

On macOS and FreeBSD, a value of @code{0} for @emph{count} specifies to send until
the end of @emph{in_fd} is reached.

All platforms support sockets as @emph{out_fd} file descriptor, and some platforms
allow other types (e.g. regular file, pipe) as well.

Cross-platform applications should not use @emph{headers}, @emph{trailers} and @emph{flags}
arguments.

@ref{1629,,Availability}: Unix.

@cartouche
@quotation Note 
For a higher-level wrapper of @ref{62c,,sendfile()}, see
@ref{9c9,,socket.socket.sendfile()}.
@end quotation
@end cartouche


New in version 3.3.



Changed in version 3.9: Parameters @emph{out} and @emph{in} was renamed to @emph{out_fd} and @emph{in_fd}.

@end deffn

@geindex set_blocking() (in module os)
@anchor{library/os os set_blocking}@anchor{9a1}
@deffn {Function} os.set_blocking (fd, blocking)

Set the blocking mode of the specified file descriptor. Set the
@ref{9a2,,O_NONBLOCK} flag if blocking is @code{False}, clear the flag otherwise.

See also @ref{9a0,,get_blocking()} and @ref{23a8,,socket.socket.setblocking()}.

@ref{1629,,Availability}: Unix.


New in version 3.5.

@end deffn

@geindex SF_NODISKIO (in module os)
@geindex SF_MNOWAIT (in module os)
@geindex SF_SYNC (in module os)
@anchor{library/os os SF_NODISKIO}@anchor{23d1}
@deffn {Data} os.SF_NODISKIO
@anchor{library/os os SF_MNOWAIT}@anchor{23d2}
@deffnx {Data} os.SF_MNOWAIT
@anchor{library/os os SF_SYNC}@anchor{23d3}
@deffnx {Data} os.SF_SYNC

Parameters to the @ref{62c,,sendfile()} function, if the implementation supports
them.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex SF_NOCACHE (in module os)
@anchor{library/os os SF_NOCACHE}@anchor{23d4}
@deffn {Data} os.SF_NOCACHE

Parameter to the @ref{62c,,sendfile()} function, if the implementation supports
it. The data won’t be cached in the virtual memory and will be freed afterwards.

@ref{1629,,Availability}: Unix.


New in version 3.11.

@end deffn

@geindex splice() (in module os)
@anchor{library/os os splice}@anchor{29a}
@deffn {Function} os.splice (src, dst, count, offset_src=None, offset_dst=None)

Transfer @emph{count} bytes from file descriptor @emph{src}, starting from offset
@emph{offset_src}, to file descriptor @emph{dst}, starting from offset @emph{offset_dst}.
At least one of the file descriptors must refer to a pipe. If @emph{offset_src}
is None, then @emph{src} is read from the current position; respectively for
@emph{offset_dst}. The offset associated to the file descriptor that refers to a
pipe must be @code{None}. The files pointed by @emph{src} and @emph{dst} must reside in
the same filesystem, otherwise an @ref{2a1,,OSError} is raised with
@ref{1b76,,errno} set to @ref{23a6,,errno.EXDEV}.

This copy is done without the additional cost of transferring data
from the kernel to user space and then back into the kernel. Additionally,
some filesystems could implement extra optimizations. The copy is done as if
both files are opened as binary.

Upon successful completion, returns the number of bytes spliced to or from
the pipe. A return value of 0 means end of input. If @emph{src} refers to a
pipe, then this means that there was no data to transfer, and it would not
make sense to block because there are no writers connected to the write end
of the pipe.

@ref{1629,,Availability}: Linux kernel >= 2.6.17 and glibc >= 2.5


New in version 3.10.

@end deffn

@geindex SPLICE_F_MOVE (in module os)
@geindex SPLICE_F_NONBLOCK (in module os)
@geindex SPLICE_F_MORE (in module os)
@anchor{library/os os SPLICE_F_MOVE}@anchor{23d5}
@deffn {Data} os.SPLICE_F_MOVE
@anchor{library/os os SPLICE_F_NONBLOCK}@anchor{23d6}
@deffnx {Data} os.SPLICE_F_NONBLOCK
@anchor{library/os os SPLICE_F_MORE}@anchor{23d7}
@deffnx {Data} os.SPLICE_F_MORE


New in version 3.10.

@end deffn

@geindex readv() (in module os)
@anchor{library/os os readv}@anchor{687}
@deffn {Function} os.readv (fd, buffers)

Read from a file descriptor @emph{fd} into a number of mutable @ref{87e,,bytes-like objects} @emph{buffers}. Transfer data into each buffer until
it is full and then move on to the next buffer in the sequence to hold the
rest of the data.

Return the total number of bytes actually read which can be less than the
total capacity of all the objects.

The operating system may set a limit (@ref{23cd,,sysconf()} value
@code{'SC_IOV_MAX'}) on the number of buffers that can be used.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex tcgetpgrp() (in module os)
@anchor{library/os os tcgetpgrp}@anchor{23d8}
@deffn {Function} os.tcgetpgrp (fd)

Return the process group associated with the terminal given by @emph{fd} (an open
file descriptor as returned by @ref{8ee,,os.open()}).

@ref{1629,,Availability}: Unix.
@end deffn

@geindex tcsetpgrp() (in module os)
@anchor{library/os os tcsetpgrp}@anchor{23d9}
@deffn {Function} os.tcsetpgrp (fd, pg)

Set the process group associated with the terminal given by @emph{fd} (an open file
descriptor as returned by @ref{8ee,,os.open()}) to @emph{pg}.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex ttyname() (in module os)
@anchor{library/os os ttyname}@anchor{23da}
@deffn {Function} os.ttyname (fd)

Return a string which specifies the terminal device associated with
file descriptor @emph{fd}.  If @emph{fd} is not associated with a terminal device, an
exception is raised.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex write() (in module os)
@anchor{library/os os write}@anchor{8f7}
@deffn {Function} os.write (fd, str)

Write the bytestring in @emph{str} to file descriptor @emph{fd}.

Return the number of bytes actually written.

@cartouche
@quotation Note 
This function is intended for low-level I/O and must be applied to a file
descriptor as returned by @ref{8ee,,os.open()} or @ref{23a3,,pipe()}.  To write a “file
object” returned by the built-in function @ref{1c8,,open()} or by @ref{59f,,popen()} or
@ref{179a,,fdopen()}, or @ref{5ef,,sys.stdout} or @ref{42d,,sys.stderr}, use its
@code{write()} method.
@end quotation
@end cartouche


Changed in version 3.5: If the system call is interrupted and the signal handler does not raise an
exception, the function now retries the system call instead of raising an
@ref{8e3,,InterruptedError} exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).

@end deffn

@geindex writev() (in module os)
@anchor{library/os os writev}@anchor{68a}
@deffn {Function} os.writev (fd, buffers)

Write the contents of @emph{buffers} to file descriptor @emph{fd}. @emph{buffers} must be
a sequence of @ref{87e,,bytes-like objects}. Buffers are
processed in array order. Entire contents of the first buffer is written
before proceeding to the second, and so on.

Returns the total number of bytes actually written.

The operating system may set a limit (@ref{23cd,,sysconf()} value
@code{'SC_IOV_MAX'}) on the number of buffers that can be used.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@menu
* Querying the size of a terminal:: 
* Inheritance of File Descriptors:: 

@end menu

@node Querying the size of a terminal,Inheritance of File Descriptors,,File Descriptor Operations
@anchor{library/os querying-the-size-of-a-terminal}@anchor{23db}@anchor{library/os terminal-size}@anchor{23dc}
@subsubsection Querying the size of a terminal



New in version 3.3.


@geindex get_terminal_size() (in module os)
@anchor{library/os os get_terminal_size}@anchor{c95}
@deffn {Function} os.get_terminal_size (fd=STDOUT_FILENO)

Return the size of the terminal window as @code{(columns, lines)},
tuple of type @ref{2131,,terminal_size}.

The optional argument @code{fd} (default @code{STDOUT_FILENO}, or standard
output) specifies which file descriptor should be queried.

If the file descriptor is not connected to a terminal, an @ref{2a1,,OSError}
is raised.

@ref{c96,,shutil.get_terminal_size()} is the high-level function which
should normally be used, @code{os.get_terminal_size} is the low-level
implementation.

@ref{1629,,Availability}: Unix, Windows.
@end deffn

@geindex terminal_size (class in os)
@anchor{library/os os terminal_size}@anchor{2131}
@deffn {Class} os.terminal_size

A subclass of tuple, holding @code{(columns, lines)} of the terminal window size.

@geindex columns (os.terminal_size attribute)
@anchor{library/os os terminal_size columns}@anchor{23dd}
@deffn {Attribute} columns

Width of the terminal window in characters.
@end deffn

@geindex lines (os.terminal_size attribute)
@anchor{library/os os terminal_size lines}@anchor{23de}
@deffn {Attribute} lines

Height of the terminal window in characters.
@end deffn
@end deffn

@node Inheritance of File Descriptors,,Querying the size of a terminal,File Descriptor Operations
@anchor{library/os fd-inheritance}@anchor{a6e}@anchor{library/os inheritance-of-file-descriptors}@anchor{23df}
@subsubsection Inheritance of File Descriptors



New in version 3.4.


A file descriptor has an “inheritable” flag which indicates if the file descriptor
can be inherited by child processes.  Since Python 3.4, file descriptors
created by Python are non-inheritable by default.

On UNIX, non-inheritable file descriptors are closed in child processes at the
execution of a new program, other file descriptors are inherited.

On Windows, non-inheritable handles and file descriptors are closed in child
processes, except for standard streams (file descriptors 0, 1 and 2: stdin, stdout
and stderr), which are always inherited.  Using @ref{23e0,,spawn*} functions,
all inheritable handles and all inheritable file descriptors are inherited.
Using the @ref{f4,,subprocess} module, all file descriptors except standard
streams are closed, and inheritable handles are only inherited if the
@emph{close_fds} parameter is @code{False}.

@geindex get_inheritable() (in module os)
@anchor{library/os os get_inheritable}@anchor{a6f}
@deffn {Function} os.get_inheritable (fd)

Get the “inheritable” flag of the specified file descriptor (a boolean).
@end deffn

@geindex set_inheritable() (in module os)
@anchor{library/os os set_inheritable}@anchor{a70}
@deffn {Function} os.set_inheritable (fd, inheritable)

Set the “inheritable” flag of the specified file descriptor.
@end deffn

@geindex get_handle_inheritable() (in module os)
@anchor{library/os os get_handle_inheritable}@anchor{a71}
@deffn {Function} os.get_handle_inheritable (handle)

Get the “inheritable” flag of the specified handle (a boolean).

@ref{1629,,Availability}: Windows.
@end deffn

@geindex set_handle_inheritable() (in module os)
@anchor{library/os os set_handle_inheritable}@anchor{a72}
@deffn {Function} os.set_handle_inheritable (handle, inheritable)

Set the “inheritable” flag of the specified handle.

@ref{1629,,Availability}: Windows.
@end deffn

@node Files and Directories,Process Management,File Descriptor Operations,os — Miscellaneous operating system interfaces
@anchor{library/os files-and-directories}@anchor{23e1}@anchor{library/os os-file-dir}@anchor{237f}
@subsubsection Files and Directories


On some Unix platforms, many of these functions support one or more of these
features:
@anchor{library/os path-fd}@anchor{684}

@itemize *

@item 
@strong{specifying a file descriptor:}
Normally the @emph{path} argument provided to functions in the @ref{c0,,os} module
must be a string specifying a file path.  However, some functions now
alternatively accept an open file descriptor for their @emph{path} argument.
The function will then operate on the file referred to by the descriptor.
(For POSIX systems, Python will call the variant of the function prefixed
with @code{f} (e.g. call @code{fchdir} instead of @code{chdir}).)

You can check whether or not @emph{path} can be specified as a file descriptor
for a particular function on your platform using @ref{c90,,os.supports_fd}.
If this functionality is unavailable, using it will raise a
@ref{8a3,,NotImplementedError}.

If the function also supports @emph{dir_fd} or @emph{follow_symlinks} arguments, it’s
an error to specify one of those when supplying @emph{path} as a file descriptor.
@end itemize
@anchor{library/os dir-fd}@anchor{c7c}

@itemize *

@item 
@strong{paths relative to directory descriptors:} If @emph{dir_fd} is not @code{None}, it
should be a file descriptor referring to a directory, and the path to operate
on should be relative; path will then be relative to that directory.  If the
path is absolute, @emph{dir_fd} is ignored.  (For POSIX systems, Python will call
the variant of the function with an @code{at} suffix and possibly prefixed with
@code{f} (e.g. call @code{faccessat} instead of @code{access}).

You can check whether or not @emph{dir_fd} is supported for a particular function
on your platform using @ref{c8b,,os.supports_dir_fd}.  If it’s unavailable,
using it will raise a @ref{8a3,,NotImplementedError}.
@end itemize
@anchor{library/os follow-symlinks}@anchor{c7d}

@itemize *

@item 
@strong{not following symlinks:} If @emph{follow_symlinks} is
@code{False}, and the last element of the path to operate on is a symbolic link,
the function will operate on the symbolic link itself rather than the file
pointed to by the link.  (For POSIX systems, Python will call the @code{l...}
variant of the function.)

You can check whether or not @emph{follow_symlinks} is supported for a particular
function on your platform using @ref{211c,,os.supports_follow_symlinks}.
If it’s unavailable, using it will raise a @ref{8a3,,NotImplementedError}.
@end itemize

@geindex access() (in module os)
@anchor{library/os os access}@anchor{c7e}
@deffn {Function} os.access (path, mode, *, dir_fd=None, effective_ids=False, follow_symlinks=True)

Use the real uid/gid to test for access to @emph{path}.  Note that most operations
will use the effective uid/gid, therefore this routine can be used in a
suid/sgid environment to test if the invoking user has the specified access to
@emph{path}.  @emph{mode} should be @ref{23e2,,F_OK} to test the existence of @emph{path}, or it
can be the inclusive OR of one or more of @ref{23e3,,R_OK}, @ref{23e4,,W_OK}, and
@ref{23e5,,X_OK} to test permissions.  Return @ref{748,,True} if access is allowed,
@ref{659,,False} if not. See the Unix man page @code{access(2)} for more
information.

This function can support specifying @ref{c7c,,paths relative to directory descriptors} and @ref{c7d,,not following symlinks}.

If @emph{effective_ids} is @code{True}, @ref{c7e,,access()} will perform its access
checks using the effective uid/gid instead of the real uid/gid.
@emph{effective_ids} may not be supported on your platform; you can check whether
or not it is available using @ref{c91,,os.supports_effective_ids}.  If it is
unavailable, using it will raise a @ref{8a3,,NotImplementedError}.

@cartouche
@quotation Note 
Using @ref{c7e,,access()} to check if a user is authorized to e.g. open a file
before actually doing so using @ref{1c8,,open()} creates a security hole,
because the user might exploit the short time interval between checking
and opening the file to manipulate it. It’s preferable to use @ref{23e6,,EAFP}
techniques. For example:

@example
if os.access("myfile", os.R_OK):
    with open("myfile") as fp:
        return fp.read()
return "some default data"
@end example

is better written as:

@example
try:
    fp = open("myfile")
except PermissionError:
    return "some default data"
else:
    with fp:
        return fp.read()
@end example
@end quotation
@end cartouche

@cartouche
@quotation Note 
I/O operations may fail even when @ref{c7e,,access()} indicates that they would
succeed, particularly for operations on network filesystems which may have
permissions semantics beyond the usual POSIX permission-bit model.
@end quotation
@end cartouche


Changed in version 3.3: Added the @emph{dir_fd}, @emph{effective_ids}, and @emph{follow_symlinks} parameters.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex F_OK (in module os)
@geindex R_OK (in module os)
@geindex W_OK (in module os)
@geindex X_OK (in module os)
@anchor{library/os os F_OK}@anchor{23e2}
@deffn {Data} os.F_OK
@anchor{library/os os R_OK}@anchor{23e3}
@deffnx {Data} os.R_OK
@anchor{library/os os W_OK}@anchor{23e4}
@deffnx {Data} os.W_OK
@anchor{library/os os X_OK}@anchor{23e5}
@deffnx {Data} os.X_OK

Values to pass as the @emph{mode} parameter of @ref{c7e,,access()} to test the
existence, readability, writability and executability of @emph{path},
respectively.
@end deffn

@geindex chdir() (in module os)
@anchor{library/os os chdir}@anchor{3b9}
@deffn {Function} os.chdir (path)

@geindex directory; changing

Change the current working directory to @emph{path}.

This function can support @ref{684,,specifying a file descriptor}.  The
descriptor must refer to an opened directory, not an open file.

This function can raise @ref{2a1,,OSError} and subclasses such as
@ref{a37,,FileNotFoundError}, @ref{895,,PermissionError}, and @ref{bf0,,NotADirectoryError}.

Raises an @ref{1130,,auditing event} @code{os.chdir} with argument @code{path}.


New in version 3.3: Added support for specifying @emph{path} as a file descriptor
on some platforms.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex chflags() (in module os)
@anchor{library/os os chflags}@anchor{c7f}
@deffn {Function} os.chflags (path, flags, *, follow_symlinks=True)

Set the flags of @emph{path} to the numeric @emph{flags}. @emph{flags} may take a combination
(bitwise OR) of the following values (as defined in the @ref{ef,,stat} module):


@itemize *

@item 
@ref{20ca,,stat.UF_NODUMP}

@item 
@ref{20cb,,stat.UF_IMMUTABLE}

@item 
@ref{20cc,,stat.UF_APPEND}

@item 
@ref{20cd,,stat.UF_OPAQUE}

@item 
@ref{20ce,,stat.UF_NOUNLINK}

@item 
@ref{20cf,,stat.UF_COMPRESSED}

@item 
@ref{20d0,,stat.UF_HIDDEN}

@item 
@ref{20d1,,stat.SF_ARCHIVED}

@item 
@ref{20d2,,stat.SF_IMMUTABLE}

@item 
@ref{20d3,,stat.SF_APPEND}

@item 
@ref{20d4,,stat.SF_NOUNLINK}

@item 
@ref{20d5,,stat.SF_SNAPSHOT}
@end itemize

This function can support @ref{c7d,,not following symlinks}.

Raises an @ref{1130,,auditing event} @code{os.chflags} with arguments @code{path}, @code{flags}.

@ref{1629,,Availability}: Unix.


New in version 3.3: The @emph{follow_symlinks} argument.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex chmod() (in module os)
@anchor{library/os os chmod}@anchor{c80}
@deffn {Function} os.chmod (path, mode, *, dir_fd=None, follow_symlinks=True)

Change the mode of @emph{path} to the numeric @emph{mode}. @emph{mode} may take one of the
following values (as defined in the @ref{ef,,stat} module) or bitwise ORed
combinations of them:


@itemize *

@item 
@ref{20b7,,stat.S_ISUID}

@item 
@ref{20b8,,stat.S_ISGID}

@item 
@ref{20ba,,stat.S_ENFMT}

@item 
@ref{20bb,,stat.S_ISVTX}

@item 
@ref{20c7,,stat.S_IREAD}

@item 
@ref{20c8,,stat.S_IWRITE}

@item 
@ref{20c9,,stat.S_IEXEC}

@item 
@ref{20bc,,stat.S_IRWXU}

@item 
@ref{20bd,,stat.S_IRUSR}

@item 
@ref{20be,,stat.S_IWUSR}

@item 
@ref{20bf,,stat.S_IXUSR}

@item 
@ref{20c0,,stat.S_IRWXG}

@item 
@ref{20c1,,stat.S_IRGRP}

@item 
@ref{20c2,,stat.S_IWGRP}

@item 
@ref{20b9,,stat.S_IXGRP}

@item 
@ref{20c3,,stat.S_IRWXO}

@item 
@ref{20c4,,stat.S_IROTH}

@item 
@ref{20c5,,stat.S_IWOTH}

@item 
@ref{20c6,,stat.S_IXOTH}
@end itemize

This function can support @ref{684,,specifying a file descriptor},
@ref{c7c,,paths relative to directory descriptors} and @ref{c7d,,not following symlinks}.

@cartouche
@quotation Note 
Although Windows supports @ref{c80,,chmod()}, you can only set the file’s
read-only flag with it (via the @code{stat.S_IWRITE} and @code{stat.S_IREAD}
constants or a corresponding integer value).  All other bits are ignored.
@end quotation
@end cartouche

Raises an @ref{1130,,auditing event} @code{os.chmod} with arguments @code{path}, @code{mode}, @code{dir_fd}.


New in version 3.3: Added support for specifying @emph{path} as an open file descriptor,
and the @emph{dir_fd} and @emph{follow_symlinks} arguments.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex chown() (in module os)
@anchor{library/os os chown}@anchor{c81}
@deffn {Function} os.chown (path, uid, gid, *, dir_fd=None, follow_symlinks=True)

Change the owner and group id of @emph{path} to the numeric @emph{uid} and @emph{gid}.  To
leave one of the ids unchanged, set it to -1.

This function can support @ref{684,,specifying a file descriptor},
@ref{c7c,,paths relative to directory descriptors} and @ref{c7d,,not following symlinks}.

See @ref{cc2,,shutil.chown()} for a higher-level function that accepts names in
addition to numeric ids.

Raises an @ref{1130,,auditing event} @code{os.chown} with arguments @code{path}, @code{uid}, @code{gid}, @code{dir_fd}.

@ref{1629,,Availability}: Unix.


New in version 3.3: Added support for specifying @emph{path} as an open file descriptor,
and the @emph{dir_fd} and @emph{follow_symlinks} arguments.



Changed in version 3.6: Supports a @ref{63d,,path-like object}.

@end deffn

@geindex chroot() (in module os)
@anchor{library/os os chroot}@anchor{1223}
@deffn {Function} os.chroot (path)

Change the root directory of the current process to @emph{path}.

@ref{1629,,Availability}: Unix.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex fchdir() (in module os)
@anchor{library/os os fchdir}@anchor{8e4}
@deffn {Function} os.fchdir (fd)

Change the current working directory to the directory represented by the file
descriptor @emph{fd}.  The descriptor must refer to an opened directory, not an
open file.  As of Python 3.3, this is equivalent to @code{os.chdir(fd)}.

Raises an @ref{1130,,auditing event} @code{os.chdir} with argument @code{path}.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex getcwd() (in module os)
@anchor{library/os os getcwd}@anchor{12ac}
@deffn {Function} os.getcwd ()

Return a string representing the current working directory.
@end deffn

@geindex getcwdb() (in module os)
@anchor{library/os os getcwdb}@anchor{5ad}
@deffn {Function} os.getcwdb ()

Return a bytestring representing the current working directory.


Changed in version 3.8: The function now uses the UTF-8 encoding on Windows, rather than the ANSI
code page: see 
@geindex Python Enhancement Proposals; PEP 529
PEP 529@footnote{https://www.python.org/dev/peps/pep-0529} for the rationale. The function is no longer
deprecated on Windows.

@end deffn

@geindex lchflags() (in module os)
@anchor{library/os os lchflags}@anchor{23e7}
@deffn {Function} os.lchflags (path, flags)

Set the flags of @emph{path} to the numeric @emph{flags}, like @ref{c7f,,chflags()}, but do
not follow symbolic links.  As of Python 3.3, this is equivalent to
@code{os.chflags(path, flags, follow_symlinks=False)}.

Raises an @ref{1130,,auditing event} @code{os.chflags} with arguments @code{path}, @code{flags}.

@ref{1629,,Availability}: Unix.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex lchmod() (in module os)
@anchor{library/os os lchmod}@anchor{12f0}
@deffn {Function} os.lchmod (path, mode)

Change the mode of @emph{path} to the numeric @emph{mode}. If path is a symlink, this
affects the symlink rather than the target.  See the docs for @ref{c80,,chmod()}
for possible values of @emph{mode}.  As of Python 3.3, this is equivalent to
@code{os.chmod(path, mode, follow_symlinks=False)}.

Raises an @ref{1130,,auditing event} @code{os.chmod} with arguments @code{path}, @code{mode}, @code{dir_fd}.

@ref{1629,,Availability}: Unix.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex lchown() (in module os)
@anchor{library/os os lchown}@anchor{23e8}
@deffn {Function} os.lchown (path, uid, gid)

Change the owner and group id of @emph{path} to the numeric @emph{uid} and @emph{gid}.  This
function will not follow symbolic links.  As of Python 3.3, this is equivalent
to @code{os.chown(path, uid, gid, follow_symlinks=False)}.

Raises an @ref{1130,,auditing event} @code{os.chown} with arguments @code{path}, @code{uid}, @code{gid}, @code{dir_fd}.

@ref{1629,,Availability}: Unix.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex link() (in module os)
@anchor{library/os os link}@anchor{c82}
@deffn {Function} os.link (src, dst, *, src_dir_fd=None, dst_dir_fd=None, follow_symlinks=True)

Create a hard link pointing to @emph{src} named @emph{dst}.

This function can support specifying @emph{src_dir_fd} and/or @emph{dst_dir_fd} to
supply @ref{c7c,,paths relative to directory descriptors}, and @ref{c7d,,not following symlinks}.

Raises an @ref{1130,,auditing event} @code{os.link} with arguments @code{src}, @code{dst}, @code{src_dir_fd}, @code{dst_dir_fd}.

@ref{1629,,Availability}: Unix, Windows.


Changed in version 3.2: Added Windows support.



New in version 3.3: Added the @emph{src_dir_fd}, @emph{dst_dir_fd}, and @emph{follow_symlinks} arguments.



Changed in version 3.6: Accepts a @ref{63d,,path-like object} for @emph{src} and @emph{dst}.

@end deffn

@geindex listdir() (in module os)
@anchor{library/os os listdir}@anchor{c8d}
@deffn {Function} os.listdir (path='.')

Return a list containing the names of the entries in the directory given by
@emph{path}.  The list is in arbitrary order, and does not include the special
entries @code{'.'} and @code{'..'} even if they are present in the directory.
If a file is removed from or added to the directory during the call of
this function, whether a name for that file be included is unspecified.

@emph{path} may be a @ref{63d,,path-like object}.  If @emph{path} is of type @code{bytes}
(directly or indirectly through the @ref{795,,PathLike} interface),
the filenames returned will also be of type @code{bytes};
in all other circumstances, they will be of type @code{str}.

This function can also support @ref{684,,specifying a file descriptor}; the file descriptor must refer to a directory.

Raises an @ref{1130,,auditing event} @code{os.listdir} with argument @code{path}.

@cartouche
@quotation Note 
To encode @code{str} filenames to @code{bytes}, use @ref{799,,fsencode()}.
@end quotation
@end cartouche


@subsubheading See also


The @ref{576,,scandir()} function returns directory entries along with
file attribute information, giving better performance for many
common use cases.



Changed in version 3.2: The @emph{path} parameter became optional.



New in version 3.3: Added support for specifying @emph{path} as an open file descriptor.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex lstat() (in module os)
@anchor{library/os os lstat}@anchor{517}
@deffn {Function} os.lstat (path, *, dir_fd=None)

Perform the equivalent of an @code{lstat()} system call on the given path.
Similar to @ref{516,,stat()}, but does not follow symbolic links. Return a
@ref{134d,,stat_result} object.

On platforms that do not support symbolic links, this is an alias for
@ref{516,,stat()}.

As of Python 3.3, this is equivalent to @code{os.stat(path, dir_fd=dir_fd,
follow_symlinks=False)}.

This function can also support @ref{c7c,,paths relative to directory descriptors}.


@subsubheading See also


The @ref{516,,stat()} function.



Changed in version 3.2: Added support for Windows 6.0 (Vista) symbolic links.



Changed in version 3.3: Added the @emph{dir_fd} parameter.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.



Changed in version 3.8: On Windows, now opens reparse points that represent another path
(name surrogates), including symbolic links and directory junctions.
Other kinds of reparse points are resolved by the operating system as
for @ref{516,,stat()}.

@end deffn

@geindex mkdir() (in module os)
@anchor{library/os os mkdir}@anchor{c83}
@deffn {Function} os.mkdir (path, mode=511, *, dir_fd=None)

Create a directory named @emph{path} with numeric mode @emph{mode}.

If the directory already exists, @ref{bb1,,FileExistsError} is raised.
@anchor{library/os mkdir-modebits}@anchor{23e9}
On some systems, @emph{mode} is ignored.  Where it is used, the current umask
value is first masked out.  If bits other than the last 9 (i.e. the last 3
digits of the octal representation of the @emph{mode}) are set, their meaning is
platform-dependent.  On some platforms, they are ignored and you should call
@ref{c80,,chmod()} explicitly to set them.

This function can also support @ref{c7c,,paths relative to directory descriptors}.

It is also possible to create temporary directories; see the
@ref{fd,,tempfile} module’s @ref{2103,,tempfile.mkdtemp()} function.

Raises an @ref{1130,,auditing event} @code{os.mkdir} with arguments @code{path}, @code{mode}, @code{dir_fd}.


New in version 3.3: The @emph{dir_fd} argument.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex makedirs() (in module os)
@anchor{library/os os makedirs}@anchor{68c}
@deffn {Function} os.makedirs (name, mode=511, exist_ok=False)

@geindex directory; creating
@geindex UNC paths; and os.makedirs()

Recursive directory creation function.  Like @ref{c83,,mkdir()}, but makes all
intermediate-level directories needed to contain the leaf directory.

The @emph{mode} parameter is passed to @ref{c83,,mkdir()} for creating the leaf
directory; see @ref{23e9,,the mkdir() description} for how it
is interpreted.  To set the file permission bits of any newly-created parent
directories you can set the umask before invoking @ref{68c,,makedirs()}.  The
file permission bits of existing parent directories are not changed.

If @emph{exist_ok} is @code{False} (the default), a @ref{bb1,,FileExistsError} is
raised if the target directory already exists.

@cartouche
@quotation Note 
@ref{68c,,makedirs()} will become confused if the path elements to create
include @ref{23ea,,pardir} (eg. “..” on UNIX systems).
@end quotation
@end cartouche

This function handles UNC paths correctly.

Raises an @ref{1130,,auditing event} @code{os.mkdir} with arguments @code{path}, @code{mode}, @code{dir_fd}.


New in version 3.2: The @emph{exist_ok} parameter.



Changed in version 3.4.1: Before Python 3.4.1, if @emph{exist_ok} was @code{True} and the directory existed,
@ref{68c,,makedirs()} would still raise an error if @emph{mode} did not match the
mode of the existing directory. Since this behavior was impossible to
implement safely, it was removed in Python 3.4.1. See bpo-21082@footnote{https://bugs.python.org/issue21082}.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.



Changed in version 3.7: The @emph{mode} argument no longer affects the file permission bits of
newly-created intermediate-level directories.

@end deffn

@geindex mkfifo() (in module os)
@anchor{library/os os mkfifo}@anchor{8ec}
@deffn {Function} os.mkfifo (path, mode=438, *, dir_fd=None)

Create a FIFO (a named pipe) named @emph{path} with numeric mode @emph{mode}.
The current umask value is first masked out from the mode.

This function can also support @ref{c7c,,paths relative to directory descriptors}.

FIFOs are pipes that can be accessed like regular files.  FIFOs exist until they
are deleted (for example with @ref{c89,,os.unlink()}). Generally, FIFOs are used as
rendezvous between “client” and “server” type processes: the server opens the
FIFO for reading, and the client opens it for writing.  Note that @ref{8ec,,mkfifo()}
doesn’t open the FIFO — it just creates the rendezvous point.

@ref{1629,,Availability}: Unix.


New in version 3.3: The @emph{dir_fd} argument.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex mknod() (in module os)
@anchor{library/os os mknod}@anchor{8ed}
@deffn {Function} os.mknod (path, mode=384, device=0, *, dir_fd=None)

Create a filesystem node (file, device special file or named pipe) named
@emph{path}. @emph{mode} specifies both the permissions to use and the type of node
to be created, being combined (bitwise OR) with one of @code{stat.S_IFREG},
@code{stat.S_IFCHR}, @code{stat.S_IFBLK}, and @code{stat.S_IFIFO} (those constants are
available in @ref{ef,,stat}).  For @code{stat.S_IFCHR} and @code{stat.S_IFBLK},
@emph{device} defines the newly created device special file (probably using
@ref{23eb,,os.makedev()}), otherwise it is ignored.

This function can also support @ref{c7c,,paths relative to directory descriptors}.

@ref{1629,,Availability}: Unix.


New in version 3.3: The @emph{dir_fd} argument.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex major() (in module os)
@anchor{library/os os major}@anchor{23ec}
@deffn {Function} os.major (device)

Extract the device major number from a raw device number (usually the
@code{st_dev} or @code{st_rdev} field from @code{stat}).
@end deffn

@geindex minor() (in module os)
@anchor{library/os os minor}@anchor{23ed}
@deffn {Function} os.minor (device)

Extract the device minor number from a raw device number (usually the
@code{st_dev} or @code{st_rdev} field from @code{stat}).
@end deffn

@geindex makedev() (in module os)
@anchor{library/os os makedev}@anchor{23eb}
@deffn {Function} os.makedev (major, minor)

Compose a raw device number from the major and minor device numbers.
@end deffn

@geindex pathconf() (in module os)
@anchor{library/os os pathconf}@anchor{c8e}
@deffn {Function} os.pathconf (path, name)

Return system configuration information relevant to a named file. @emph{name}
specifies the configuration value to retrieve; it may be a string which is the
name of a defined system value; these names are specified in a number of
standards (POSIX.1, Unix 95, Unix 98, and others).  Some platforms define
additional names as well.  The names known to the host operating system are
given in the @code{pathconf_names} dictionary.  For configuration variables not
included in that mapping, passing an integer for @emph{name} is also accepted.

If @emph{name} is a string and is not known, @ref{17c,,ValueError} is raised.  If a
specific value for @emph{name} is not supported by the host system, even if it is
included in @code{pathconf_names}, an @ref{2a1,,OSError} is raised with
@ref{1298,,errno.EINVAL} for the error number.

This function can support @ref{684,,specifying a file descriptor}.

@ref{1629,,Availability}: Unix.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex pathconf_names (in module os)
@anchor{library/os os pathconf_names}@anchor{23ee}
@deffn {Data} os.pathconf_names

Dictionary mapping names accepted by @ref{c8e,,pathconf()} and @ref{23a7,,fpathconf()} to
the integer values defined for those names by the host operating system.  This
can be used to determine the set of names known to the system.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex readlink() (in module os)
@anchor{library/os os readlink}@anchor{414}
@deffn {Function} os.readlink (path, *, dir_fd=None)

Return a string representing the path to which the symbolic link points.  The
result may be either an absolute or relative pathname; if it is relative, it
may be converted to an absolute pathname using
@code{os.path.join(os.path.dirname(path), result)}.

If the @emph{path} is a string object (directly or indirectly through a
@ref{795,,PathLike} interface), the result will also be a string object,
and the call may raise a UnicodeDecodeError. If the @emph{path} is a bytes
object (direct or indirectly), the result will be a bytes object.

This function can also support @ref{c7c,,paths relative to directory descriptors}.

When trying to resolve a path that may contain links, use
@ref{2a0,,realpath()} to properly handle recursion and platform
differences.

@ref{1629,,Availability}: Unix, Windows.


Changed in version 3.2: Added support for Windows 6.0 (Vista) symbolic links.



New in version 3.3: The @emph{dir_fd} argument.



Changed in version 3.6: Accepts a @ref{63d,,path-like object} on Unix.



Changed in version 3.8: Accepts a @ref{63d,,path-like object} and a bytes object on Windows.



Changed in version 3.8: Added support for directory junctions, and changed to return the
substitution path (which typically includes @code{\\?\} prefix) rather
than the optional “print name” field that was previously returned.

@end deffn

@geindex remove() (in module os)
@anchor{library/os os remove}@anchor{c84}
@deffn {Function} os.remove (path, *, dir_fd=None)

Remove (delete) the file @emph{path}.  If @emph{path} is a directory, an
@ref{bef,,IsADirectoryError} is raised.  Use @ref{c87,,rmdir()} to remove directories.
If the file does not exist, a @ref{a37,,FileNotFoundError} is raised.

This function can support @ref{c7c,,paths relative to directory descriptors}.

On Windows, attempting to remove a file that is in use causes an exception to
be raised; on Unix, the directory entry is removed but the storage allocated
to the file is not made available until the original file is no longer in use.

This function is semantically identical to @ref{c89,,unlink()}.

Raises an @ref{1130,,auditing event} @code{os.remove} with arguments @code{path}, @code{dir_fd}.


New in version 3.3: The @emph{dir_fd} argument.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex removedirs() (in module os)
@anchor{library/os os removedirs}@anchor{23ef}
@deffn {Function} os.removedirs (name)

@geindex directory; deleting

Remove directories recursively.  Works like @ref{c87,,rmdir()} except that, if the
leaf directory is successfully removed, @ref{23ef,,removedirs()}  tries to
successively remove every parent directory mentioned in  @emph{path} until an error
is raised (which is ignored, because it generally means that a parent directory
is not empty). For example, @code{os.removedirs('foo/bar/baz')} will first remove
the directory @code{'foo/bar/baz'}, and then remove @code{'foo/bar'} and @code{'foo'} if
they are empty. Raises @ref{2a1,,OSError} if the leaf directory could not be
successfully removed.

Raises an @ref{1130,,auditing event} @code{os.remove} with arguments @code{path}, @code{dir_fd}.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex rename() (in module os)
@anchor{library/os os rename}@anchor{c85}
@deffn {Function} os.rename (src, dst, *, src_dir_fd=None, dst_dir_fd=None)

Rename the file or directory @emph{src} to @emph{dst}. If @emph{dst} exists, the operation
will fail with an @ref{2a1,,OSError} subclass in a number of cases:

On Windows, if @emph{dst} exists a @ref{bb1,,FileExistsError} is always raised.

On Unix, if @emph{src} is a file and @emph{dst} is a directory or vice-versa, an
@ref{bef,,IsADirectoryError} or a @ref{bf0,,NotADirectoryError} will be raised
respectively.  If both are directories and @emph{dst} is empty, @emph{dst} will be
silently replaced.  If @emph{dst} is a non-empty directory, an @ref{2a1,,OSError}
is raised. If both are files, @emph{dst} it will be replaced silently if the user
has permission.  The operation may fail on some Unix flavors if @emph{src} and
@emph{dst} are on different filesystems.  If successful, the renaming will be an
atomic operation (this is a POSIX requirement).

This function can support specifying @emph{src_dir_fd} and/or @emph{dst_dir_fd} to
supply @ref{c7c,,paths relative to directory descriptors}.

If you want cross-platform overwriting of the destination, use @ref{c86,,replace()}.

Raises an @ref{1130,,auditing event} @code{os.rename} with arguments @code{src}, @code{dst}, @code{src_dir_fd}, @code{dst_dir_fd}.


New in version 3.3: The @emph{src_dir_fd} and @emph{dst_dir_fd} arguments.



Changed in version 3.6: Accepts a @ref{63d,,path-like object} for @emph{src} and @emph{dst}.

@end deffn

@geindex renames() (in module os)
@anchor{library/os os renames}@anchor{23f0}
@deffn {Function} os.renames (old, new)

Recursive directory or file renaming function. Works like @ref{c85,,rename()}, except
creation of any intermediate directories needed to make the new pathname good is
attempted first. After the rename, directories corresponding to rightmost path
segments of the old name will be pruned away using @ref{23ef,,removedirs()}.

@cartouche
@quotation Note 
This function can fail with the new directory structure made if you lack
permissions needed to remove the leaf directory or file.
@end quotation
@end cartouche

Raises an @ref{1130,,auditing event} @code{os.rename} with arguments @code{src}, @code{dst}, @code{src_dir_fd}, @code{dst_dir_fd}.


Changed in version 3.6: Accepts a @ref{63d,,path-like object} for @emph{old} and @emph{new}.

@end deffn

@geindex replace() (in module os)
@anchor{library/os os replace}@anchor{c86}
@deffn {Function} os.replace (src, dst, *, src_dir_fd=None, dst_dir_fd=None)

Rename the file or directory @emph{src} to @emph{dst}.  If @emph{dst} is a directory,
@ref{2a1,,OSError} will be raised.  If @emph{dst} exists and is a file, it will
be replaced silently if the user has permission.  The operation may fail
if @emph{src} and @emph{dst} are on different filesystems.  If successful,
the renaming will be an atomic operation (this is a POSIX requirement).

This function can support specifying @emph{src_dir_fd} and/or @emph{dst_dir_fd} to
supply @ref{c7c,,paths relative to directory descriptors}.

Raises an @ref{1130,,auditing event} @code{os.rename} with arguments @code{src}, @code{dst}, @code{src_dir_fd}, @code{dst_dir_fd}.


New in version 3.3.



Changed in version 3.6: Accepts a @ref{63d,,path-like object} for @emph{src} and @emph{dst}.

@end deffn

@geindex rmdir() (in module os)
@anchor{library/os os rmdir}@anchor{c87}
@deffn {Function} os.rmdir (path, *, dir_fd=None)

Remove (delete) the directory @emph{path}.  If the directory does not exist or is
not empty, a @ref{a37,,FileNotFoundError} or an @ref{2a1,,OSError} is raised
respectively.  In order to remove whole directory trees,
@ref{53a,,shutil.rmtree()} can be used.

This function can support @ref{c7c,,paths relative to directory descriptors}.

Raises an @ref{1130,,auditing event} @code{os.rmdir} with arguments @code{path}, @code{dir_fd}.


New in version 3.3: The @emph{dir_fd} parameter.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex scandir() (in module os)
@anchor{library/os os scandir}@anchor{576}
@deffn {Function} os.scandir (path='.')

Return an iterator of @ref{79a,,os.DirEntry} objects corresponding to the
entries in the directory given by @emph{path}. The entries are yielded in
arbitrary order, and the special entries @code{'.'} and @code{'..'} are not
included.  If a file is removed from or added to the directory after
creating the iterator, whether an entry for that file be included is
unspecified.

Using @ref{576,,scandir()} instead of @ref{c8d,,listdir()} can significantly
increase the performance of code that also needs file type or file
attribute information, because @ref{79a,,os.DirEntry} objects expose this
information if the operating system provides it when scanning a directory.
All @ref{79a,,os.DirEntry} methods may perform a system call, but
@ref{23f1,,is_dir()} and @ref{8df,,is_file()} usually only
require a system call for symbolic links; @ref{10f7,,os.DirEntry.stat()}
always requires a system call on Unix but only requires one for
symbolic links on Windows.

@emph{path} may be a @ref{63d,,path-like object}.  If @emph{path} is of type @code{bytes}
(directly or indirectly through the @ref{795,,PathLike} interface),
the type of the @ref{23f2,,name} and @ref{23f3,,path}
attributes of each @ref{79a,,os.DirEntry} will be @code{bytes}; in all other
circumstances, they will be of type @code{str}.

This function can also support @ref{684,,specifying a file descriptor}; the file descriptor must refer to a directory.

Raises an @ref{1130,,auditing event} @code{os.scandir} with argument @code{path}.

The @ref{576,,scandir()} iterator supports the @ref{166,,context manager} protocol
and has the following method:

@geindex close() (os.scandir method)
@anchor{library/os os scandir close}@anchor{805}
@deffn {Method} scandir.close ()

Close the iterator and free acquired resources.

This is called automatically when the iterator is exhausted or garbage
collected, or when an error happens during iterating.  However it
is advisable to call it explicitly or use the @ref{168,,with}
statement.


New in version 3.6.

@end deffn

The following example shows a simple use of @ref{576,,scandir()} to display all
the files (excluding directories) in the given @emph{path} that don’t start with
@code{'.'}. The @code{entry.is_file()} call will generally not make an additional
system call:

@example
with os.scandir(path) as it:
    for entry in it:
        if not entry.name.startswith('.') and entry.is_file():
            print(entry.name)
@end example

@cartouche
@quotation Note 
On Unix-based systems, @ref{576,,scandir()} uses the system’s
opendir()@footnote{http://pubs.opengroup.org/onlinepubs/009695399/functions/opendir.html}
and
readdir()@footnote{http://pubs.opengroup.org/onlinepubs/009695399/functions/readdir_r.html}
functions. On Windows, it uses the Win32
FindFirstFileW@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa364418(v=vs.85).aspx}
and
FindNextFileW@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa364428(v=vs.85).aspx}
functions.
@end quotation
@end cartouche


New in version 3.5.



New in version 3.6: Added support for the @ref{166,,context manager} protocol and the
@ref{805,,close()} method.  If a @ref{576,,scandir()} iterator is neither
exhausted nor explicitly closed a @ref{77b,,ResourceWarning} will be emitted
in its destructor.

The function accepts a @ref{63d,,path-like object}.



Changed in version 3.7: Added support for @ref{684,,file descriptors} on Unix.

@end deffn

@geindex DirEntry (class in os)
@anchor{library/os os DirEntry}@anchor{79a}
@deffn {Class} os.DirEntry

Object yielded by @ref{576,,scandir()} to expose the file path and other file
attributes of a directory entry.

@ref{576,,scandir()} will provide as much of this information as possible without
making additional system calls. When a @code{stat()} or @code{lstat()} system call
is made, the @code{os.DirEntry} object will cache the result.

@code{os.DirEntry} instances are not intended to be stored in long-lived data
structures; if you know the file metadata has changed or if a long time has
elapsed since calling @ref{576,,scandir()}, call @code{os.stat(entry.path)} to fetch
up-to-date information.

Because the @code{os.DirEntry} methods can make operating system calls, they may
also raise @ref{2a1,,OSError}. If you need very fine-grained
control over errors, you can catch @ref{2a1,,OSError} when calling one of the
@code{os.DirEntry} methods and handle as appropriate.

To be directly usable as a @ref{63d,,path-like object}, @code{os.DirEntry}
implements the @ref{795,,PathLike} interface.

Attributes and methods on a @code{os.DirEntry} instance are as follows:

@geindex name (os.DirEntry attribute)
@anchor{library/os os DirEntry name}@anchor{23f2}
@deffn {Attribute} name

The entry’s base filename, relative to the @ref{576,,scandir()} @emph{path}
argument.

The @ref{2375,,name} attribute will be @code{bytes} if the @ref{576,,scandir()}
@emph{path} argument is of type @code{bytes} and @code{str} otherwise.  Use
@ref{798,,fsdecode()} to decode byte filenames.
@end deffn

@geindex path (os.DirEntry attribute)
@anchor{library/os os DirEntry path}@anchor{23f3}
@deffn {Attribute} path

The entry’s full path name: equivalent to @code{os.path.join(scandir_path,
entry.name)} where @emph{scandir_path} is the @ref{576,,scandir()} @emph{path}
argument.  The path is only absolute if the @ref{576,,scandir()} @emph{path}
argument was absolute.  If the @ref{576,,scandir()} @emph{path}
argument was a @ref{684,,file descriptor}, the @ref{c1,,path}
attribute is the same as the @ref{2375,,name} attribute.

The @ref{c1,,path} attribute will be @code{bytes} if the @ref{576,,scandir()}
@emph{path} argument is of type @code{bytes} and @code{str} otherwise.  Use
@ref{798,,fsdecode()} to decode byte filenames.
@end deffn

@geindex inode() (os.DirEntry method)
@anchor{library/os os DirEntry inode}@anchor{23f4}
@deffn {Method} inode ()

Return the inode number of the entry.

The result is cached on the @code{os.DirEntry} object. Use
@code{os.stat(entry.path, follow_symlinks=False).st_ino} to fetch up-to-date
information.

On the first, uncached call, a system call is required on Windows but
not on Unix.
@end deffn

@geindex is_dir() (os.DirEntry method)
@anchor{library/os os DirEntry is_dir}@anchor{23f1}
@deffn {Method} is_dir (*, follow_symlinks=True)

Return @code{True} if this entry is a directory or a symbolic link pointing
to a directory; return @code{False} if the entry is or points to any other
kind of file, or if it doesn’t exist anymore.

If @emph{follow_symlinks} is @code{False}, return @code{True} only if this entry
is a directory (without following symlinks); return @code{False} if the
entry is any other kind of file or if it doesn’t exist anymore.

The result is cached on the @code{os.DirEntry} object, with a separate cache
for @emph{follow_symlinks} @code{True} and @code{False}. Call @ref{516,,os.stat()} along
with @ref{209e,,stat.S_ISDIR()} to fetch up-to-date information.

On the first, uncached call, no system call is required in most cases.
Specifically, for non-symlinks, neither Windows or Unix require a system
call, except on certain Unix file systems, such as network file systems,
that return @code{dirent.d_type == DT_UNKNOWN}. If the entry is a symlink,
a system call will be required to follow the symlink unless
@emph{follow_symlinks} is @code{False}.

This method can raise @ref{2a1,,OSError}, such as @ref{895,,PermissionError},
but @ref{a37,,FileNotFoundError} is caught and not raised.
@end deffn

@geindex is_file() (os.DirEntry method)
@anchor{library/os os DirEntry is_file}@anchor{8df}
@deffn {Method} is_file (*, follow_symlinks=True)

Return @code{True} if this entry is a file or a symbolic link pointing to a
file; return @code{False} if the entry is or points to a directory or other
non-file entry, or if it doesn’t exist anymore.

If @emph{follow_symlinks} is @code{False}, return @code{True} only if this entry
is a file (without following symlinks); return @code{False} if the entry is
a directory or other non-file entry, or if it doesn’t exist anymore.

The result is cached on the @code{os.DirEntry} object. Caching, system calls
made, and exceptions raised are as per @ref{23f1,,is_dir()}.
@end deffn

@geindex is_symlink() (os.DirEntry method)
@anchor{library/os os DirEntry is_symlink}@anchor{23f5}
@deffn {Method} is_symlink ()

Return @code{True} if this entry is a symbolic link (even if broken);
return @code{False} if the entry points to a directory or any kind of file,
or if it doesn’t exist anymore.

The result is cached on the @code{os.DirEntry} object. Call
@ref{518,,os.path.islink()} to fetch up-to-date information.

On the first, uncached call, no system call is required in most cases.
Specifically, neither Windows or Unix require a system call, except on
certain Unix file systems, such as network file systems, that return
@code{dirent.d_type == DT_UNKNOWN}.

This method can raise @ref{2a1,,OSError}, such as @ref{895,,PermissionError},
but @ref{a37,,FileNotFoundError} is caught and not raised.
@end deffn

@geindex stat() (os.DirEntry method)
@anchor{library/os os DirEntry stat}@anchor{10f7}
@deffn {Method} stat (*, follow_symlinks=True)

Return a @ref{134d,,stat_result} object for this entry. This method
follows symbolic links by default; to stat a symbolic link add the
@code{follow_symlinks=False} argument.

On Unix, this method always requires a system call. On Windows, it
only requires a system call if @emph{follow_symlinks} is @code{True} and the
entry is a reparse point (for example, a symbolic link or directory
junction).

On Windows, the @code{st_ino}, @code{st_dev} and @code{st_nlink} attributes of the
@ref{134d,,stat_result} are always set to zero. Call @ref{516,,os.stat()} to
get these attributes.

The result is cached on the @code{os.DirEntry} object, with a separate cache
for @emph{follow_symlinks} @code{True} and @code{False}. Call @ref{516,,os.stat()} to
fetch up-to-date information.
@end deffn

Note that there is a nice correspondence between several attributes
and methods of @code{os.DirEntry} and of @ref{522,,pathlib.Path}.  In
particular, the @code{name} attribute has the same
meaning, as do the @code{is_dir()}, @code{is_file()}, @code{is_symlink()}
and @code{stat()} methods.


New in version 3.5.



Changed in version 3.6: Added support for the @ref{795,,PathLike} interface.  Added support
for @ref{2f0,,bytes} paths on Windows.

@end deffn

@geindex stat() (in module os)
@anchor{library/os os stat}@anchor{516}
@deffn {Function} os.stat (path, *, dir_fd=None, follow_symlinks=True)

Get the status of a file or a file descriptor. Perform the equivalent of a
@code{stat()} system call on the given path. @emph{path} may be specified as
either a string or bytes – directly or indirectly through the @ref{795,,PathLike}
interface – or as an open file descriptor. Return a @ref{134d,,stat_result}
object.

This function normally follows symlinks; to stat a symlink add the argument
@code{follow_symlinks=False}, or use @ref{517,,lstat()}.

This function can support @ref{684,,specifying a file descriptor} and
@ref{c7d,,not following symlinks}.

On Windows, passing @code{follow_symlinks=False} will disable following all
name-surrogate reparse points, which includes symlinks and directory
junctions. Other types of reparse points that do not resemble links or that
the operating system is unable to follow will be opened directly. When
following a chain of multiple links, this may result in the original link
being returned instead of the non-link that prevented full traversal. To
obtain stat results for the final path in this case, use the
@ref{2a0,,os.path.realpath()} function to resolve the path name as far as
possible and call @ref{517,,lstat()} on the result. This does not apply to
dangling symlinks or junction points, which will raise the usual exceptions.

@geindex module; stat

Example:

@example
>>> import os
>>> statinfo = os.stat('somefile.txt')
>>> statinfo
os.stat_result(st_mode=33188, st_ino=7876932, st_dev=234881026,
st_nlink=1, st_uid=501, st_gid=501, st_size=264, st_atime=1297230295,
st_mtime=1297230027, st_ctime=1297230027)
>>> statinfo.st_size
264
@end example


@subsubheading See also


@ref{8e8,,fstat()} and @ref{517,,lstat()} functions.



New in version 3.3: Added the @emph{dir_fd} and @emph{follow_symlinks} arguments, specifying a file
descriptor instead of a path.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.



Changed in version 3.8: On Windows, all reparse points that can be resolved by the operating
system are now followed, and passing @code{follow_symlinks=False}
disables following all name surrogate reparse points. If the operating
system reaches a reparse point that it is not able to follow, @emph{stat} now
returns the information for the original path as if
@code{follow_symlinks=False} had been specified instead of raising an error.

@end deffn

@geindex stat_result (class in os)
@anchor{library/os os stat_result}@anchor{134d}
@deffn {Class} os.stat_result

Object whose attributes correspond roughly to the members of the
@code{stat} structure. It is used for the result of @ref{516,,os.stat()},
@ref{8e8,,os.fstat()} and @ref{517,,os.lstat()}.

Attributes:

@geindex st_mode (os.stat_result attribute)
@anchor{library/os os stat_result st_mode}@anchor{23f6}
@deffn {Attribute} st_mode

File mode: file type and file mode bits (permissions).
@end deffn

@geindex st_ino (os.stat_result attribute)
@anchor{library/os os stat_result st_ino}@anchor{23f7}
@deffn {Attribute} st_ino

Platform dependent, but if non-zero, uniquely identifies the
file for a given value of @code{st_dev}. Typically:


@itemize *

@item 
the inode number on Unix,

@item 
the file index@footnote{https://msdn.microsoft.com/en-us/library/aa363788} on
Windows
@end itemize
@end deffn

@geindex st_dev (os.stat_result attribute)
@anchor{library/os os stat_result st_dev}@anchor{23f8}
@deffn {Attribute} st_dev

Identifier of the device on which this file resides.
@end deffn

@geindex st_nlink (os.stat_result attribute)
@anchor{library/os os stat_result st_nlink}@anchor{23f9}
@deffn {Attribute} st_nlink

Number of hard links.
@end deffn

@geindex st_uid (os.stat_result attribute)
@anchor{library/os os stat_result st_uid}@anchor{23fa}
@deffn {Attribute} st_uid

User identifier of the file owner.
@end deffn

@geindex st_gid (os.stat_result attribute)
@anchor{library/os os stat_result st_gid}@anchor{23fb}
@deffn {Attribute} st_gid

Group identifier of the file owner.
@end deffn

@geindex st_size (os.stat_result attribute)
@anchor{library/os os stat_result st_size}@anchor{23fc}
@deffn {Attribute} st_size

Size of the file in bytes, if it is a regular file or a symbolic link.
The size of a symbolic link is the length of the pathname it contains,
without a terminating null byte.
@end deffn

Timestamps:

@geindex st_atime (os.stat_result attribute)
@anchor{library/os os stat_result st_atime}@anchor{23fd}
@deffn {Attribute} st_atime

Time of most recent access expressed in seconds.
@end deffn

@geindex st_mtime (os.stat_result attribute)
@anchor{library/os os stat_result st_mtime}@anchor{21ed}
@deffn {Attribute} st_mtime

Time of most recent content modification expressed in seconds.
@end deffn

@geindex st_ctime (os.stat_result attribute)
@anchor{library/os os stat_result st_ctime}@anchor{23fe}
@deffn {Attribute} st_ctime

Platform dependent:


@itemize *

@item 
the time of most recent metadata change on Unix,

@item 
the time of creation on Windows, expressed in seconds.
@end itemize
@end deffn

@geindex st_atime_ns (os.stat_result attribute)
@anchor{library/os os stat_result st_atime_ns}@anchor{23ff}
@deffn {Attribute} st_atime_ns

Time of most recent access expressed in nanoseconds as an integer.
@end deffn

@geindex st_mtime_ns (os.stat_result attribute)
@anchor{library/os os stat_result st_mtime_ns}@anchor{2400}
@deffn {Attribute} st_mtime_ns

Time of most recent content modification expressed in nanoseconds as an
integer.
@end deffn

@geindex st_ctime_ns (os.stat_result attribute)
@anchor{library/os os stat_result st_ctime_ns}@anchor{2401}
@deffn {Attribute} st_ctime_ns

Platform dependent:


@itemize *

@item 
the time of most recent metadata change on Unix,

@item 
the time of creation on Windows, expressed in nanoseconds as an
integer.
@end itemize
@end deffn

@cartouche
@quotation Note 
The exact meaning and resolution of the @ref{23fd,,st_atime},
@ref{21ed,,st_mtime}, and @ref{23fe,,st_ctime} attributes depend on the operating
system and the file system. For example, on Windows systems using the FAT
or FAT32 file systems, @ref{21ed,,st_mtime} has 2-second resolution, and
@ref{23fd,,st_atime} has only 1-day resolution.  See your operating system
documentation for details.

Similarly, although @ref{23ff,,st_atime_ns}, @ref{2400,,st_mtime_ns},
and @ref{2401,,st_ctime_ns} are always expressed in nanoseconds, many
systems do not provide nanosecond precision.  On systems that do
provide nanosecond precision, the floating-point object used to
store @ref{23fd,,st_atime}, @ref{21ed,,st_mtime}, and @ref{23fe,,st_ctime}
cannot preserve all of it, and as such will be slightly inexact.
If you need the exact timestamps you should always use
@ref{23ff,,st_atime_ns}, @ref{2400,,st_mtime_ns}, and @ref{2401,,st_ctime_ns}.
@end quotation
@end cartouche

On some Unix systems (such as Linux), the following attributes may also be
available:

@geindex st_blocks (os.stat_result attribute)
@anchor{library/os os stat_result st_blocks}@anchor{2402}
@deffn {Attribute} st_blocks

Number of 512-byte blocks allocated for file.
This may be smaller than @ref{23fc,,st_size}/512 when the file has holes.
@end deffn

@geindex st_blksize (os.stat_result attribute)
@anchor{library/os os stat_result st_blksize}@anchor{2403}
@deffn {Attribute} st_blksize

“Preferred” blocksize for efficient file system I/O. Writing to a file in
smaller chunks may cause an inefficient read-modify-rewrite.
@end deffn

@geindex st_rdev (os.stat_result attribute)
@anchor{library/os os stat_result st_rdev}@anchor{2404}
@deffn {Attribute} st_rdev

Type of device if an inode device.
@end deffn

@geindex st_flags (os.stat_result attribute)
@anchor{library/os os stat_result st_flags}@anchor{2405}
@deffn {Attribute} st_flags

User defined flags for file.
@end deffn

On other Unix systems (such as FreeBSD), the following attributes may be
available (but may be only filled out if root tries to use them):

@geindex st_gen (os.stat_result attribute)
@anchor{library/os os stat_result st_gen}@anchor{2406}
@deffn {Attribute} st_gen

File generation number.
@end deffn

@geindex st_birthtime (os.stat_result attribute)
@anchor{library/os os stat_result st_birthtime}@anchor{2407}
@deffn {Attribute} st_birthtime

Time of file creation.
@end deffn

On Solaris and derivatives, the following attributes may also be
available:

@geindex st_fstype (os.stat_result attribute)
@anchor{library/os os stat_result st_fstype}@anchor{68e}
@deffn {Attribute} st_fstype

String that uniquely identifies the type of the filesystem that
contains the file.
@end deffn

On macOS systems, the following attributes may also be available:

@geindex st_rsize (os.stat_result attribute)
@anchor{library/os os stat_result st_rsize}@anchor{2408}
@deffn {Attribute} st_rsize

Real size of the file.
@end deffn

@geindex st_creator (os.stat_result attribute)
@anchor{library/os os stat_result st_creator}@anchor{2409}
@deffn {Attribute} st_creator

Creator of the file.
@end deffn

@geindex st_type (os.stat_result attribute)
@anchor{library/os os stat_result st_type}@anchor{240a}
@deffn {Attribute} st_type

File type.
@end deffn

On Windows systems, the following attributes are also available:

@geindex st_file_attributes (os.stat_result attribute)
@anchor{library/os os stat_result st_file_attributes}@anchor{99f}
@deffn {Attribute} st_file_attributes

Windows file attributes: @code{dwFileAttributes} member of the
@code{BY_HANDLE_FILE_INFORMATION} structure returned by
@code{GetFileInformationByHandle()}. See the @code{FILE_ATTRIBUTE_*}
constants in the @ref{ef,,stat} module.
@end deffn

@geindex st_reparse_tag (os.stat_result attribute)
@anchor{library/os os stat_result st_reparse_tag}@anchor{240b}
@deffn {Attribute} st_reparse_tag

When @ref{99f,,st_file_attributes} has the @code{FILE_ATTRIBUTE_REPARSE_POINT}
set, this field contains the tag identifying the type of reparse point.
See the @code{IO_REPARSE_TAG_*} constants in the @ref{ef,,stat} module.
@end deffn

The standard module @ref{ef,,stat} defines functions and constants that are
useful for extracting information from a @code{stat} structure. (On
Windows, some items are filled with dummy values.)

For backward compatibility, a @ref{134d,,stat_result} instance is also
accessible as a tuple of at least 10 integers giving the most important (and
portable) members of the @code{stat} structure, in the order
@ref{23f6,,st_mode}, @ref{23f7,,st_ino}, @ref{23f8,,st_dev}, @ref{23f9,,st_nlink},
@ref{23fa,,st_uid}, @ref{23fb,,st_gid}, @ref{23fc,,st_size}, @ref{23fd,,st_atime},
@ref{21ed,,st_mtime}, @ref{23fe,,st_ctime}. More items may be added at the end by
some implementations. For compatibility with older Python versions,
accessing @ref{134d,,stat_result} as a tuple always returns integers.


New in version 3.3: Added the @ref{23ff,,st_atime_ns}, @ref{2400,,st_mtime_ns}, and
@ref{2401,,st_ctime_ns} members.



New in version 3.5: Added the @ref{99f,,st_file_attributes} member on Windows.



Changed in version 3.5: Windows now returns the file index as @ref{23f7,,st_ino} when
available.



New in version 3.7: Added the @ref{68e,,st_fstype} member to Solaris/derivatives.



New in version 3.8: Added the @ref{240b,,st_reparse_tag} member on Windows.



Changed in version 3.8: On Windows, the @ref{23f6,,st_mode} member now identifies special
files as @code{S_IFCHR}, @code{S_IFIFO} or @code{S_IFBLK}
as appropriate.

@end deffn

@geindex statvfs() (in module os)
@anchor{library/os os statvfs}@anchor{c8f}
@deffn {Function} os.statvfs (path)

Perform a @code{statvfs()} system call on the given path.  The return value is
an object whose attributes describe the filesystem on the given path, and
correspond to the members of the @code{statvfs} structure, namely:
@code{f_bsize}, @code{f_frsize}, @code{f_blocks}, @code{f_bfree},
@code{f_bavail}, @code{f_files}, @code{f_ffree}, @code{f_favail},
@code{f_flag}, @code{f_namemax}, @code{f_fsid}.

Two module-level constants are defined for the @code{f_flag} attribute’s
bit-flags: if @code{ST_RDONLY} is set, the filesystem is mounted
read-only, and if @code{ST_NOSUID} is set, the semantics of
setuid/setgid bits are disabled or not supported.

Additional module-level constants are defined for GNU/glibc based systems.
These are @code{ST_NODEV} (disallow access to device special files),
@code{ST_NOEXEC} (disallow program execution), @code{ST_SYNCHRONOUS}
(writes are synced at once), @code{ST_MANDLOCK} (allow mandatory locks on an FS),
@code{ST_WRITE} (write on file/directory/symlink), @code{ST_APPEND}
(append-only file), @code{ST_IMMUTABLE} (immutable file), @code{ST_NOATIME}
(do not update access times), @code{ST_NODIRATIME} (do not update directory access
times), @code{ST_RELATIME} (update atime relative to mtime/ctime).

This function can support @ref{684,,specifying a file descriptor}.

@ref{1629,,Availability}: Unix.


Changed in version 3.2: The @code{ST_RDONLY} and @code{ST_NOSUID} constants were added.



New in version 3.3: Added support for specifying @emph{path} as an open file descriptor.



Changed in version 3.4: The @code{ST_NODEV}, @code{ST_NOEXEC}, @code{ST_SYNCHRONOUS},
@code{ST_MANDLOCK}, @code{ST_WRITE}, @code{ST_APPEND},
@code{ST_IMMUTABLE}, @code{ST_NOATIME}, @code{ST_NODIRATIME},
and @code{ST_RELATIME} constants were added.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.



New in version 3.7: Added @code{f_fsid}.

@end deffn

@geindex supports_dir_fd (in module os)
@anchor{library/os os supports_dir_fd}@anchor{c8b}
@deffn {Data} os.supports_dir_fd

A @ref{370,,set} object indicating which functions in the @ref{c0,,os}
module accept an open file descriptor for their @emph{dir_fd} parameter.
Different platforms provide different features, and the underlying
functionality Python uses to implement the @emph{dir_fd} parameter is not
available on all platforms Python supports.  For consistency’s sake,
functions that may support @emph{dir_fd} always allow specifying the
parameter, but will throw an exception if the functionality is used
when it’s not locally available. (Specifying @code{None} for @emph{dir_fd}
is always supported on all platforms.)

To check whether a particular function accepts an open file descriptor
for its @emph{dir_fd} parameter, use the @code{in} operator on @code{supports_dir_fd}.
As an example, this expression evaluates to @code{True} if @ref{516,,os.stat()}
accepts open file descriptors for @emph{dir_fd} on the local platform:

@example
os.stat in os.supports_dir_fd
@end example

Currently @emph{dir_fd} parameters only work on Unix platforms;
none of them work on Windows.


New in version 3.3.

@end deffn

@geindex supports_effective_ids (in module os)
@anchor{library/os os supports_effective_ids}@anchor{c91}
@deffn {Data} os.supports_effective_ids

A @ref{370,,set} object indicating whether @ref{c7e,,os.access()} permits
specifying @code{True} for its @emph{effective_ids} parameter on the local platform.
(Specifying @code{False} for @emph{effective_ids} is always supported on all
platforms.)  If the local platform supports it, the collection will contain
@ref{c7e,,os.access()}; otherwise it will be empty.

This expression evaluates to @code{True} if @ref{c7e,,os.access()} supports
@code{effective_ids=True} on the local platform:

@example
os.access in os.supports_effective_ids
@end example

Currently @emph{effective_ids} is only supported on Unix platforms;
it does not work on Windows.


New in version 3.3.

@end deffn

@geindex supports_fd (in module os)
@anchor{library/os os supports_fd}@anchor{c90}
@deffn {Data} os.supports_fd

A @ref{370,,set} object indicating which functions in the
@ref{c0,,os} module permit specifying their @emph{path} parameter as an open file
descriptor on the local platform.  Different platforms provide different
features, and the underlying functionality Python uses to accept open file
descriptors as @emph{path} arguments is not available on all platforms Python
supports.

To determine whether a particular function permits specifying an open file
descriptor for its @emph{path} parameter, use the @code{in} operator on
@code{supports_fd}. As an example, this expression evaluates to @code{True} if
@ref{3b9,,os.chdir()} accepts open file descriptors for @emph{path} on your local
platform:

@example
os.chdir in os.supports_fd
@end example


New in version 3.3.

@end deffn

@geindex supports_follow_symlinks (in module os)
@anchor{library/os os supports_follow_symlinks}@anchor{211c}
@deffn {Data} os.supports_follow_symlinks

A @ref{370,,set} object indicating which functions in the @ref{c0,,os} module
accept @code{False} for their @emph{follow_symlinks} parameter on the local platform.
Different platforms provide different features, and the underlying
functionality Python uses to implement @emph{follow_symlinks} is not available
on all platforms Python supports.  For consistency’s sake, functions that
may support @emph{follow_symlinks} always allow specifying the parameter, but
will throw an exception if the functionality is used when it’s not locally
available.  (Specifying @code{True} for @emph{follow_symlinks} is always supported
on all platforms.)

To check whether a particular function accepts @code{False} for its
@emph{follow_symlinks} parameter, use the @code{in} operator on
@code{supports_follow_symlinks}.  As an example, this expression evaluates
to @code{True} if you may specify @code{follow_symlinks=False} when calling
@ref{516,,os.stat()} on the local platform:

@example
os.stat in os.supports_follow_symlinks
@end example


New in version 3.3.

@end deffn

@geindex symlink() (in module os)
@anchor{library/os os symlink}@anchor{c88}
@deffn {Function} os.symlink (src, dst, target_is_directory=False, *, dir_fd=None)

Create a symbolic link pointing to @emph{src} named @emph{dst}.

On Windows, a symlink represents either a file or a directory, and does not
morph to the target dynamically.  If the target is present, the type of the
symlink will be created to match. Otherwise, the symlink will be created
as a directory if @emph{target_is_directory} is @code{True} or a file symlink (the
default) otherwise.  On non-Windows platforms, @emph{target_is_directory} is ignored.

This function can support @ref{c7c,,paths relative to directory descriptors}.

@cartouche
@quotation Note 
On newer versions of Windows 10, unprivileged accounts can create symlinks
if Developer Mode is enabled. When Developer Mode is not available/enabled,
the @emph{SeCreateSymbolicLinkPrivilege} privilege is required, or the process
must be run as an administrator.

@ref{2a1,,OSError} is raised when the function is called by an unprivileged
user.
@end quotation
@end cartouche

Raises an @ref{1130,,auditing event} @code{os.symlink} with arguments @code{src}, @code{dst}, @code{dir_fd}.

@ref{1629,,Availability}: Unix, Windows.


Changed in version 3.2: Added support for Windows 6.0 (Vista) symbolic links.



New in version 3.3: Added the @emph{dir_fd} argument, and now allow @emph{target_is_directory}
on non-Windows platforms.



Changed in version 3.6: Accepts a @ref{63d,,path-like object} for @emph{src} and @emph{dst}.



Changed in version 3.8: Added support for unelevated symlinks on Windows with Developer Mode.

@end deffn

@geindex sync() (in module os)
@anchor{library/os os sync}@anchor{ca5}
@deffn {Function} os.sync ()

Force write of everything to disk.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex truncate() (in module os)
@anchor{library/os os truncate}@anchor{9a3}
@deffn {Function} os.truncate (path, length)

Truncate the file corresponding to @emph{path}, so that it is at most
@emph{length} bytes in size.

This function can support @ref{684,,specifying a file descriptor}.

Raises an @ref{1130,,auditing event} @code{os.truncate} with arguments @code{path}, @code{length}.

@ref{1629,,Availability}: Unix, Windows.


New in version 3.3.



Changed in version 3.5: Added support for Windows



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex unlink() (in module os)
@anchor{library/os os unlink}@anchor{c89}
@deffn {Function} os.unlink (path, *, dir_fd=None)

Remove (delete) the file @emph{path}.  This function is semantically
identical to @ref{c84,,remove()}; the @code{unlink} name is its
traditional Unix name.  Please see the documentation for
@ref{c84,,remove()} for further information.

Raises an @ref{1130,,auditing event} @code{os.remove} with arguments @code{path}, @code{dir_fd}.


New in version 3.3: The @emph{dir_fd} parameter.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex utime() (in module os)
@anchor{library/os os utime}@anchor{c8a}
@deffn {Function} os.utime (path, times=None, *[, ns], dir_fd=None, follow_symlinks=True)

Set the access and modified times of the file specified by @emph{path}.

@ref{c8a,,utime()} takes two optional parameters, @emph{times} and @emph{ns}.
These specify the times set on @emph{path} and are used as follows:


@itemize -

@item 
If @emph{ns} is specified,
it must be a 2-tuple of the form @code{(atime_ns, mtime_ns)}
where each member is an int expressing nanoseconds.

@item 
If @emph{times} is not @code{None},
it must be a 2-tuple of the form @code{(atime, mtime)}
where each member is an int or float expressing seconds.

@item 
If @emph{times} is @code{None} and @emph{ns} is unspecified,
this is equivalent to specifying @code{ns=(atime_ns, mtime_ns)}
where both times are the current time.
@end itemize

It is an error to specify tuples for both @emph{times} and @emph{ns}.

Note that the exact times you set here may not be returned by a subsequent
@ref{516,,stat()} call, depending on the resolution with which your operating
system records access and modification times; see @ref{516,,stat()}. The best
way to preserve exact times is to use the @emph{st_atime_ns} and @emph{st_mtime_ns}
fields from the @ref{516,,os.stat()} result object with the @emph{ns} parameter to
@cite{utime}.

This function can support @ref{684,,specifying a file descriptor},
@ref{c7c,,paths relative to directory descriptors} and @ref{c7d,,not following symlinks}.

Raises an @ref{1130,,auditing event} @code{os.utime} with arguments @code{path}, @code{times}, @code{ns}, @code{dir_fd}.


New in version 3.3: Added support for specifying @emph{path} as an open file descriptor,
and the @emph{dir_fd}, @emph{follow_symlinks}, and @emph{ns} parameters.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex walk() (in module os)
@anchor{library/os os walk}@anchor{8de}
@deffn {Function} os.walk (top, topdown=True, onerror=None, followlinks=False)

@geindex directory; walking
@geindex directory; traversal

Generate the file names in a directory tree by walking the tree
either top-down or bottom-up. For each directory in the tree rooted at directory
@emph{top} (including @emph{top} itself), it yields a 3-tuple @code{(dirpath, dirnames,
filenames)}.

@emph{dirpath} is a string, the path to the directory.  @emph{dirnames} is a list of the
names of the subdirectories in @emph{dirpath} (excluding @code{'.'} and @code{'..'}).
@emph{filenames} is a list of the names of the non-directory files in @emph{dirpath}.
Note that the names in the lists contain no path components.  To get a full path
(which begins with @emph{top}) to a file or directory in @emph{dirpath}, do
@code{os.path.join(dirpath, name)}.  Whether or not the lists are sorted
depends on the file system.  If a file is removed from or added to the
@emph{dirpath} directory during generating the lists, whether a name for that
file be included is unspecified.

If optional argument @emph{topdown} is @code{True} or not specified, the triple for a
directory is generated before the triples for any of its subdirectories
(directories are generated top-down).  If @emph{topdown} is @code{False}, the triple
for a directory is generated after the triples for all of its subdirectories
(directories are generated bottom-up). No matter the value of @emph{topdown}, the
list of subdirectories is retrieved before the tuples for the directory and
its subdirectories are generated.

When @emph{topdown} is @code{True}, the caller can modify the @emph{dirnames} list in-place
(perhaps using @ref{152e,,del} or slice assignment), and @ref{8de,,walk()} will only
recurse into the subdirectories whose names remain in @emph{dirnames}; this can be
used to prune the search, impose a specific order of visiting, or even to inform
@ref{8de,,walk()} about directories the caller creates or renames before it resumes
@ref{8de,,walk()} again.  Modifying @emph{dirnames} when @emph{topdown} is @code{False} has
no effect on the behavior of the walk, because in bottom-up mode the directories
in @emph{dirnames} are generated before @emph{dirpath} itself is generated.

By default, errors from the @ref{576,,scandir()} call are ignored.  If optional
argument @emph{onerror} is specified, it should be a function; it will be called with
one argument, an @ref{2a1,,OSError} instance.  It can report the error to continue
with the walk, or raise the exception to abort the walk.  Note that the filename
is available as the @code{filename} attribute of the exception object.

By default, @ref{8de,,walk()} will not walk down into symbolic links that resolve to
directories. Set @emph{followlinks} to @code{True} to visit directories pointed to by
symlinks, on systems that support them.

@cartouche
@quotation Note 
Be aware that setting @emph{followlinks} to @code{True} can lead to infinite
recursion if a link points to a parent directory of itself. @ref{8de,,walk()}
does not keep track of the directories it visited already.
@end quotation
@end cartouche

@cartouche
@quotation Note 
If you pass a relative pathname, don’t change the current working directory
between resumptions of @ref{8de,,walk()}.  @ref{8de,,walk()} never changes the current
directory, and assumes that its caller doesn’t either.
@end quotation
@end cartouche

This example displays the number of bytes taken by non-directory files in each
directory under the starting directory, except that it doesn’t look under any
CVS subdirectory:

@example
import os
from os.path import join, getsize
for root, dirs, files in os.walk('python/Lib/email'):
    print(root, "consumes", end=" ")
    print(sum(getsize(join(root, name)) for name in files), end=" ")
    print("bytes in", len(files), "non-directory files")
    if 'CVS' in dirs:
        dirs.remove('CVS')  # don't visit CVS directories
@end example

In the next example (simple implementation of @ref{53a,,shutil.rmtree()}),
walking the tree bottom-up is essential, @ref{c87,,rmdir()} doesn’t allow
deleting a directory before the directory is empty:

@example
# Delete everything reachable from the directory named in "top",
# assuming there are no symbolic links.
# CAUTION:  This is dangerous!  For example, if top == '/', it
# could delete all your disk files.
import os
for root, dirs, files in os.walk(top, topdown=False):
    for name in files:
        os.remove(os.path.join(root, name))
    for name in dirs:
        os.rmdir(os.path.join(root, name))
@end example

Raises an @ref{1130,,auditing event} @code{os.walk} with arguments @code{top}, @code{topdown}, @code{onerror}, @code{followlinks}.


Changed in version 3.5: This function now calls @ref{576,,os.scandir()} instead of @ref{c8d,,os.listdir()},
making it faster by reducing the number of calls to @ref{516,,os.stat()}.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex fwalk() (in module os)
@anchor{library/os os fwalk}@anchor{683}
@deffn {Function} os.fwalk (top='.', topdown=True, onerror=None, *, follow_symlinks=False, dir_fd=None)

@geindex directory; walking
@geindex directory; traversal

This behaves exactly like @ref{8de,,walk()}, except that it yields a 4-tuple
@code{(dirpath, dirnames, filenames, dirfd)}, and it supports @code{dir_fd}.

@emph{dirpath}, @emph{dirnames} and @emph{filenames} are identical to @ref{8de,,walk()} output,
and @emph{dirfd} is a file descriptor referring to the directory @emph{dirpath}.

This function always supports @ref{c7c,,paths relative to directory descriptors} and @ref{c7d,,not following symlinks}.  Note however
that, unlike other functions, the @ref{683,,fwalk()} default value for
@emph{follow_symlinks} is @code{False}.

@cartouche
@quotation Note 
Since @ref{683,,fwalk()} yields file descriptors, those are only valid until
the next iteration step, so you should duplicate them (e.g. with
@ref{127f,,dup()}) if you want to keep them longer.
@end quotation
@end cartouche

This example displays the number of bytes taken by non-directory files in each
directory under the starting directory, except that it doesn’t look under any
CVS subdirectory:

@example
import os
for root, dirs, files, rootfd in os.fwalk('python/Lib/email'):
    print(root, "consumes", end="")
    print(sum([os.stat(name, dir_fd=rootfd).st_size for name in files]),
          end="")
    print("bytes in", len(files), "non-directory files")
    if 'CVS' in dirs:
        dirs.remove('CVS')  # don't visit CVS directories
@end example

In the next example, walking the tree bottom-up is essential:
@ref{c87,,rmdir()} doesn’t allow deleting a directory before the directory is
empty:

@example
# Delete everything reachable from the directory named in "top",
# assuming there are no symbolic links.
# CAUTION:  This is dangerous!  For example, if top == '/', it
# could delete all your disk files.
import os
for root, dirs, files, rootfd in os.fwalk(top, topdown=False):
    for name in files:
        os.unlink(name, dir_fd=rootfd)
    for name in dirs:
        os.rmdir(name, dir_fd=rootfd)
@end example

Raises an @ref{1130,,auditing event} @code{os.fwalk} with arguments @code{top}, @code{topdown}, @code{onerror}, @code{follow_symlinks}, @code{dir_fd}.

@ref{1629,,Availability}: Unix.


New in version 3.3.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.



Changed in version 3.7: Added support for @ref{2f0,,bytes} paths.

@end deffn

@geindex memfd_create() (in module os)
@anchor{library/os os memfd_create}@anchor{515}
@deffn {Function} os.memfd_create (name[, flags=os.MFD_CLOEXEC])

Create an anonymous file and return a file descriptor that refers to it.
@emph{flags} must be one of the @code{os.MFD_*} constants available on the system
(or a bitwise ORed combination of them).  By default, the new file
descriptor is @ref{a6e,,non-inheritable}.

The name supplied in @emph{name} is used as a filename and will be displayed as
the target of the corresponding symbolic link in the directory
@code{/proc/self/fd/}. The displayed name is always prefixed with @code{memfd:}
and serves only for debugging purposes. Names do not affect the behavior of
the file descriptor, and as such multiple files can have the same name
without any side effects.

@ref{1629,,Availability}: Linux 3.17 or newer with glibc 2.27 or newer.


New in version 3.8.

@end deffn

@geindex MFD_CLOEXEC (in module os)
@geindex MFD_ALLOW_SEALING (in module os)
@geindex MFD_HUGETLB (in module os)
@geindex MFD_HUGE_SHIFT (in module os)
@geindex MFD_HUGE_MASK (in module os)
@geindex MFD_HUGE_64KB (in module os)
@geindex MFD_HUGE_512KB (in module os)
@geindex MFD_HUGE_1MB (in module os)
@geindex MFD_HUGE_2MB (in module os)
@geindex MFD_HUGE_8MB (in module os)
@geindex MFD_HUGE_16MB (in module os)
@geindex MFD_HUGE_32MB (in module os)
@geindex MFD_HUGE_256MB (in module os)
@geindex MFD_HUGE_512MB (in module os)
@geindex MFD_HUGE_1GB (in module os)
@geindex MFD_HUGE_2GB (in module os)
@geindex MFD_HUGE_16GB (in module os)
@anchor{library/os os MFD_CLOEXEC}@anchor{240c}
@deffn {Data} os.MFD_CLOEXEC
@anchor{library/os os MFD_ALLOW_SEALING}@anchor{240d}
@deffnx {Data} os.MFD_ALLOW_SEALING
@anchor{library/os os MFD_HUGETLB}@anchor{240e}
@deffnx {Data} os.MFD_HUGETLB
@anchor{library/os os MFD_HUGE_SHIFT}@anchor{240f}
@deffnx {Data} os.MFD_HUGE_SHIFT
@anchor{library/os os MFD_HUGE_MASK}@anchor{2410}
@deffnx {Data} os.MFD_HUGE_MASK
@anchor{library/os os MFD_HUGE_64KB}@anchor{2411}
@deffnx {Data} os.MFD_HUGE_64KB
@anchor{library/os os MFD_HUGE_512KB}@anchor{2412}
@deffnx {Data} os.MFD_HUGE_512KB
@anchor{library/os os MFD_HUGE_1MB}@anchor{2413}
@deffnx {Data} os.MFD_HUGE_1MB
@anchor{library/os os MFD_HUGE_2MB}@anchor{2414}
@deffnx {Data} os.MFD_HUGE_2MB
@anchor{library/os os MFD_HUGE_8MB}@anchor{2415}
@deffnx {Data} os.MFD_HUGE_8MB
@anchor{library/os os MFD_HUGE_16MB}@anchor{2416}
@deffnx {Data} os.MFD_HUGE_16MB
@anchor{library/os os MFD_HUGE_32MB}@anchor{2417}
@deffnx {Data} os.MFD_HUGE_32MB
@anchor{library/os os MFD_HUGE_256MB}@anchor{2418}
@deffnx {Data} os.MFD_HUGE_256MB
@anchor{library/os os MFD_HUGE_512MB}@anchor{2419}
@deffnx {Data} os.MFD_HUGE_512MB
@anchor{library/os os MFD_HUGE_1GB}@anchor{241a}
@deffnx {Data} os.MFD_HUGE_1GB
@anchor{library/os os MFD_HUGE_2GB}@anchor{241b}
@deffnx {Data} os.MFD_HUGE_2GB
@anchor{library/os os MFD_HUGE_16GB}@anchor{241c}
@deffnx {Data} os.MFD_HUGE_16GB

These flags can be passed to @ref{515,,memfd_create()}.

@ref{1629,,Availability}: Linux 3.17 or newer with glibc 2.27 or newer.  The
@code{MFD_HUGE*} flags are only available since Linux 4.14.


New in version 3.8.

@end deffn

@geindex eventfd() (in module os)
@anchor{library/os os eventfd}@anchor{299}
@deffn {Function} os.eventfd (initval[, flags=os.EFD_CLOEXEC])

Create and return an event file descriptor. The file descriptors supports
raw @ref{8f1,,read()} and @ref{8f7,,write()} with a buffer size of 8,
@ref{8fc,,select()}, @ref{241d,,poll()} and similar. See man page
@code{eventfd(2)} for more information.  By default, the
new file descriptor is @ref{a6e,,non-inheritable}.

@emph{initval} is the initial value of the event counter. The initial value
must be an 32 bit unsigned integer. Please note that the initial value is
limited to a 32 bit unsigned int although the event counter is an unsigned
64 bit integer with a maximum value of 2@w{^64}-2.

@emph{flags} can be constructed from @ref{241e,,EFD_CLOEXEC},
@ref{241f,,EFD_NONBLOCK}, and @ref{2420,,EFD_SEMAPHORE}.

If @ref{2420,,EFD_SEMAPHORE} is specified and the event counter is non-zero,
@ref{2421,,eventfd_read()} returns 1 and decrements the counter by one.

If @ref{2420,,EFD_SEMAPHORE} is not specified and the event counter is
non-zero, @ref{2421,,eventfd_read()} returns the current event counter value and
resets the counter to zero.

If the event counter is zero and @ref{241f,,EFD_NONBLOCK} is not
specified, @ref{2421,,eventfd_read()} blocks.

@ref{2422,,eventfd_write()} increments the event counter. Write blocks if the
write operation would increment the counter to a value larger than
2@w{^64}-2.

Example:

@example
import os

# semaphore with start value '1'
fd = os.eventfd(1, os.EFD_SEMAPHORE | os.EFC_CLOEXEC)
try:
    # acquire semaphore
    v = os.eventfd_read(fd)
    try:
        do_work()
    finally:
        # release semaphore
        os.eventfd_write(fd, v)
finally:
    os.close(fd)
@end example

@ref{1629,,Availability}: Linux 2.6.27 or newer with glibc 2.8 or newer.


New in version 3.10.

@end deffn

@geindex eventfd_read() (in module os)
@anchor{library/os os eventfd_read}@anchor{2421}
@deffn {Function} os.eventfd_read (fd)

Read value from an @ref{299,,eventfd()} file descriptor and return a 64 bit
unsigned int. The function does not verify that @emph{fd} is an @ref{299,,eventfd()}.

@ref{1629,,Availability}: See @ref{299,,eventfd()}


New in version 3.10.

@end deffn

@geindex eventfd_write() (in module os)
@anchor{library/os os eventfd_write}@anchor{2422}
@deffn {Function} os.eventfd_write (fd, value)

Add value to an @ref{299,,eventfd()} file descriptor. @emph{value} must be a 64 bit
unsigned int. The function does not verify that @emph{fd} is an @ref{299,,eventfd()}.

@ref{1629,,Availability}: See @ref{299,,eventfd()}


New in version 3.10.

@end deffn

@geindex EFD_CLOEXEC (in module os)
@anchor{library/os os EFD_CLOEXEC}@anchor{241e}
@deffn {Data} os.EFD_CLOEXEC

Set close-on-exec flag for new @ref{299,,eventfd()} file descriptor.

@ref{1629,,Availability}: See @ref{299,,eventfd()}


New in version 3.10.

@end deffn

@geindex EFD_NONBLOCK (in module os)
@anchor{library/os os EFD_NONBLOCK}@anchor{241f}
@deffn {Data} os.EFD_NONBLOCK

Set @ref{9a2,,O_NONBLOCK} status flag for new @ref{299,,eventfd()} file
descriptor.

@ref{1629,,Availability}: See @ref{299,,eventfd()}


New in version 3.10.

@end deffn

@geindex EFD_SEMAPHORE (in module os)
@anchor{library/os os EFD_SEMAPHORE}@anchor{2420}
@deffn {Data} os.EFD_SEMAPHORE

Provide semaphore-like semantics for reads from a @ref{299,,eventfd()} file
descriptor. On read the internal counter is decremented by one.

@ref{1629,,Availability}: Linux 2.6.30 or newer with glibc 2.8 or newer.


New in version 3.10.

@end deffn

@menu
* Linux extended attributes:: 

@end menu

@node Linux extended attributes,,,Files and Directories
@anchor{library/os linux-extended-attributes}@anchor{2423}
@subsubsection Linux extended attributes



New in version 3.3.


These functions are all available on Linux only.

@geindex getxattr() (in module os)
@anchor{library/os os getxattr}@anchor{c97}
@deffn {Function} os.getxattr (path, attribute, *, follow_symlinks=True)

Return the value of the extended filesystem attribute @emph{attribute} for
@emph{path}. @emph{attribute} can be bytes or str (directly or indirectly through the
@ref{795,,PathLike} interface). If it is str, it is encoded with the filesystem
encoding.

This function can support @ref{684,,specifying a file descriptor} and
@ref{c7d,,not following symlinks}.

Raises an @ref{1130,,auditing event} @code{os.getxattr} with arguments @code{path}, @code{attribute}.


Changed in version 3.6: Accepts a @ref{63d,,path-like object} for @emph{path} and @emph{attribute}.

@end deffn

@geindex listxattr() (in module os)
@anchor{library/os os listxattr}@anchor{c98}
@deffn {Function} os.listxattr (path=None, *, follow_symlinks=True)

Return a list of the extended filesystem attributes on @emph{path}.  The
attributes in the list are represented as strings decoded with the filesystem
encoding.  If @emph{path} is @code{None}, @ref{c98,,listxattr()} will examine the current
directory.

This function can support @ref{684,,specifying a file descriptor} and
@ref{c7d,,not following symlinks}.

Raises an @ref{1130,,auditing event} @code{os.listxattr} with argument @code{path}.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex removexattr() (in module os)
@anchor{library/os os removexattr}@anchor{c99}
@deffn {Function} os.removexattr (path, attribute, *, follow_symlinks=True)

Removes the extended filesystem attribute @emph{attribute} from @emph{path}.
@emph{attribute} should be bytes or str (directly or indirectly through the
@ref{795,,PathLike} interface). If it is a string, it is encoded
with the @ref{1628,,filesystem encoding and error handler}.

This function can support @ref{684,,specifying a file descriptor} and
@ref{c7d,,not following symlinks}.

Raises an @ref{1130,,auditing event} @code{os.removexattr} with arguments @code{path}, @code{attribute}.


Changed in version 3.6: Accepts a @ref{63d,,path-like object} for @emph{path} and @emph{attribute}.

@end deffn

@geindex setxattr() (in module os)
@anchor{library/os os setxattr}@anchor{c9a}
@deffn {Function} os.setxattr (path, attribute, value, flags=0, *, follow_symlinks=True)

Set the extended filesystem attribute @emph{attribute} on @emph{path} to @emph{value}.
@emph{attribute} must be a bytes or str with no embedded NULs (directly or
indirectly through the @ref{795,,PathLike} interface). If it is a str,
it is encoded with the @ref{1628,,filesystem encoding and error handler}.  @emph{flags} may be
@ref{2424,,XATTR_REPLACE} or @ref{2425,,XATTR_CREATE}. If @ref{2424,,XATTR_REPLACE} is
given and the attribute does not exist, @code{ENODATA} will be raised.
If @ref{2425,,XATTR_CREATE} is given and the attribute already exists, the
attribute will not be created and @code{EEXISTS} will be raised.

This function can support @ref{684,,specifying a file descriptor} and
@ref{c7d,,not following symlinks}.

@cartouche
@quotation Note 
A bug in Linux kernel versions less than 2.6.39 caused the flags argument
to be ignored on some filesystems.
@end quotation
@end cartouche

Raises an @ref{1130,,auditing event} @code{os.setxattr} with arguments @code{path}, @code{attribute}, @code{value}, @code{flags}.


Changed in version 3.6: Accepts a @ref{63d,,path-like object} for @emph{path} and @emph{attribute}.

@end deffn

@geindex XATTR_SIZE_MAX (in module os)
@anchor{library/os os XATTR_SIZE_MAX}@anchor{2426}
@deffn {Data} os.XATTR_SIZE_MAX

The maximum size the value of an extended attribute can be. Currently, this
is 64 KiB on Linux.
@end deffn

@geindex XATTR_CREATE (in module os)
@anchor{library/os os XATTR_CREATE}@anchor{2425}
@deffn {Data} os.XATTR_CREATE

This is a possible value for the flags argument in @ref{c9a,,setxattr()}. It
indicates the operation must create an attribute.
@end deffn

@geindex XATTR_REPLACE (in module os)
@anchor{library/os os XATTR_REPLACE}@anchor{2424}
@deffn {Data} os.XATTR_REPLACE

This is a possible value for the flags argument in @ref{c9a,,setxattr()}. It
indicates the operation must replace an existing attribute.
@end deffn

@node Process Management,Interface to the scheduler,Files and Directories,os — Miscellaneous operating system interfaces
@anchor{library/os os-process}@anchor{2427}@anchor{library/os process-management}@anchor{2428}
@subsubsection Process Management


These functions may be used to create and manage processes.

The various @ref{2429,,exec*} functions take a list of arguments for the new
program loaded into the process.  In each case, the first of these arguments is
passed to the new program as its own name rather than as an argument a user may
have typed on a command line.  For the C programmer, this is the @code{argv[0]}
passed to a program’s @code{main()}.  For example, @code{os.execv('/bin/echo',
['foo', 'bar'])} will only print @code{bar} on standard output; @code{foo} will seem
to be ignored.

@geindex abort() (in module os)
@anchor{library/os os abort}@anchor{242a}
@deffn {Function} os.abort ()

Generate a @code{SIGABRT} signal to the current process.  On Unix, the default
behavior is to produce a core dump; on Windows, the process immediately returns
an exit code of @code{3}.  Be aware that calling this function will not call the
Python signal handler registered for @code{SIGABRT} with
@ref{cc9,,signal.signal()}.
@end deffn

@geindex add_dll_directory() (in module os)
@anchor{library/os os add_dll_directory}@anchor{4ee}
@deffn {Function} os.add_dll_directory (path)

Add a path to the DLL search path.

This search path is used when resolving dependencies for imported
extension modules (the module itself is resolved through sys.path),
and also by @ref{29,,ctypes}.

Remove the directory by calling @strong{close()} on the returned object
or using it in a @ref{168,,with} statement.

See the Microsoft documentation@footnote{https://msdn.microsoft.com/44228cf2-6306-466c-8f16-f513cd3ba8b5}
for more information about how DLLs are loaded.

Raises an @ref{1130,,auditing event} @code{os.add_dll_directory} with argument @code{path}.

@ref{1629,,Availability}: Windows.


New in version 3.8: Previous versions of CPython would resolve DLLs using the default
behavior for the current process. This led to inconsistencies,
such as only sometimes searching 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} or the current
working directory, and OS functions such as @code{AddDllDirectory}
having no effect.

In 3.8, the two primary ways DLLs are loaded now explicitly
override the process-wide behavior to ensure consistency. See the
@ref{5be,,porting notes} for information on
updating libraries.

@end deffn

@geindex execl() (in module os)
@geindex execle() (in module os)
@geindex execlp() (in module os)
@geindex execlpe() (in module os)
@geindex execv() (in module os)
@geindex execve() (in module os)
@geindex execvp() (in module os)
@geindex execvpe() (in module os)
@anchor{library/os os execl}@anchor{2429}
@deffn {Function} os.execl (path, arg0, arg1, ...)
@anchor{library/os os execle}@anchor{242b}
@deffnx {Function} os.execle (path, arg0, arg1, ..., env)
@anchor{library/os os execlp}@anchor{242c}
@deffnx {Function} os.execlp (file, arg0, arg1, ...)
@anchor{library/os os execlpe}@anchor{242d}
@deffnx {Function} os.execlpe (file, arg0, arg1, ..., env)
@anchor{library/os os execv}@anchor{2391}
@deffnx {Function} os.execv (path, args)
@anchor{library/os os execve}@anchor{c8c}
@deffnx {Function} os.execve (path, args, env)
@anchor{library/os os execvp}@anchor{242e}
@deffnx {Function} os.execvp (file, args)
@anchor{library/os os execvpe}@anchor{242f}
@deffnx {Function} os.execvpe (file, args, env)

These functions all execute a new program, replacing the current process; they
do not return.  On Unix, the new executable is loaded into the current process,
and will have the same process id as the caller.  Errors will be reported as
@ref{2a1,,OSError} exceptions.

The current process is replaced immediately. Open file objects and
descriptors are not flushed, so if there may be data buffered
on these open files, you should flush them using
@code{sys.stdout.flush()} or @ref{8ea,,os.fsync()} before calling an
@ref{2429,,exec*} function.

The “l” and “v” variants of the @ref{2429,,exec*} functions differ in how
command-line arguments are passed.  The “l” variants are perhaps the easiest
to work with if the number of parameters is fixed when the code is written; the
individual parameters simply become additional parameters to the @code{execl*()}
functions.  The “v” variants are good when the number of parameters is
variable, with the arguments being passed in a list or tuple as the @emph{args}
parameter.  In either case, the arguments to the child process should start with
the name of the command being run, but this is not enforced.

The variants which include a “p” near the end (@ref{242c,,execlp()},
@ref{242d,,execlpe()}, @ref{242e,,execvp()}, and @ref{242f,,execvpe()}) will use the
@geindex PATH
@geindex environment variable; PATH
@code{PATH} environment variable to locate the program @emph{file}.  When the
environment is being replaced (using one of the @ref{2429,,exec*e} variants,
discussed in the next paragraph), the new environment is used as the source of
the 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} variable. The other variants, @ref{2429,,execl()}, @ref{242b,,execle()},
@ref{2391,,execv()}, and @ref{c8c,,execve()}, will not use the 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} variable to
locate the executable; @emph{path} must contain an appropriate absolute or relative
path.

For @ref{242b,,execle()}, @ref{242d,,execlpe()}, @ref{c8c,,execve()}, and @ref{242f,,execvpe()} (note
that these all end in “e”), the @emph{env} parameter must be a mapping which is
used to define the environment variables for the new process (these are used
instead of the current process’ environment); the functions @ref{2429,,execl()},
@ref{242c,,execlp()}, @ref{2391,,execv()}, and @ref{242e,,execvp()} all cause the new process to
inherit the environment of the current process.

For @ref{c8c,,execve()} on some platforms, @emph{path} may also be specified as an open
file descriptor.  This functionality may not be supported on your platform;
you can check whether or not it is available using @ref{c90,,os.supports_fd}.
If it is unavailable, using it will raise a @ref{8a3,,NotImplementedError}.

Raises an @ref{1130,,auditing event} @code{os.exec} with arguments @code{path}, @code{args}, @code{env}.

@ref{1629,,Availability}: Unix, Windows.


New in version 3.3: Added support for specifying @emph{path} as an open file descriptor
for @ref{c8c,,execve()}.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex _exit() (in module os)
@anchor{library/os os _exit}@anchor{1b81}
@deffn {Function} os._exit (n)

Exit the process with status @emph{n}, without calling cleanup handlers, flushing
stdio buffers, etc.

@cartouche
@quotation Note 
The standard way to exit is @code{sys.exit(n)}.  @ref{1b81,,_exit()} should
normally only be used in the child process after a @ref{bb9,,fork()}.
@end quotation
@end cartouche
@end deffn

The following exit codes are defined and can be used with @ref{1b81,,_exit()},
although they are not required.  These are typically used for system programs
written in Python, such as a mail server’s external command delivery program.

@cartouche
@quotation Note 
Some of these may not be available on all Unix platforms, since there is some
variation.  These constants are defined where they are defined by the underlying
platform.
@end quotation
@end cartouche

@geindex EX_OK (in module os)
@anchor{library/os os EX_OK}@anchor{2430}
@deffn {Data} os.EX_OK

Exit code that means no error occurred. May be taken from the defined value of
@code{EXIT_SUCCESS} on some platforms. Generally has a value of zero.

@ref{1629,,Availability}: Unix, Windows.
@end deffn

@geindex EX_USAGE (in module os)
@anchor{library/os os EX_USAGE}@anchor{2431}
@deffn {Data} os.EX_USAGE

Exit code that means the command was used incorrectly, such as when the wrong
number of arguments are given.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_DATAERR (in module os)
@anchor{library/os os EX_DATAERR}@anchor{2432}
@deffn {Data} os.EX_DATAERR

Exit code that means the input data was incorrect.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_NOINPUT (in module os)
@anchor{library/os os EX_NOINPUT}@anchor{2433}
@deffn {Data} os.EX_NOINPUT

Exit code that means an input file did not exist or was not readable.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_NOUSER (in module os)
@anchor{library/os os EX_NOUSER}@anchor{2434}
@deffn {Data} os.EX_NOUSER

Exit code that means a specified user did not exist.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_NOHOST (in module os)
@anchor{library/os os EX_NOHOST}@anchor{2435}
@deffn {Data} os.EX_NOHOST

Exit code that means a specified host did not exist.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_UNAVAILABLE (in module os)
@anchor{library/os os EX_UNAVAILABLE}@anchor{2436}
@deffn {Data} os.EX_UNAVAILABLE

Exit code that means that a required service is unavailable.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_SOFTWARE (in module os)
@anchor{library/os os EX_SOFTWARE}@anchor{2437}
@deffn {Data} os.EX_SOFTWARE

Exit code that means an internal software error was detected.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_OSERR (in module os)
@anchor{library/os os EX_OSERR}@anchor{2438}
@deffn {Data} os.EX_OSERR

Exit code that means an operating system error was detected, such as the
inability to fork or create a pipe.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_OSFILE (in module os)
@anchor{library/os os EX_OSFILE}@anchor{2439}
@deffn {Data} os.EX_OSFILE

Exit code that means some system file did not exist, could not be opened, or had
some other kind of error.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_CANTCREAT (in module os)
@anchor{library/os os EX_CANTCREAT}@anchor{243a}
@deffn {Data} os.EX_CANTCREAT

Exit code that means a user specified output file could not be created.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_IOERR (in module os)
@anchor{library/os os EX_IOERR}@anchor{243b}
@deffn {Data} os.EX_IOERR

Exit code that means that an error occurred while doing I/O on some file.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_TEMPFAIL (in module os)
@anchor{library/os os EX_TEMPFAIL}@anchor{243c}
@deffn {Data} os.EX_TEMPFAIL

Exit code that means a temporary failure occurred.  This indicates something
that may not really be an error, such as a network connection that couldn’t be
made during a retryable operation.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_PROTOCOL (in module os)
@anchor{library/os os EX_PROTOCOL}@anchor{243d}
@deffn {Data} os.EX_PROTOCOL

Exit code that means that a protocol exchange was illegal, invalid, or not
understood.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_NOPERM (in module os)
@anchor{library/os os EX_NOPERM}@anchor{243e}
@deffn {Data} os.EX_NOPERM

Exit code that means that there were insufficient permissions to perform the
operation (but not intended for file system problems).

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_CONFIG (in module os)
@anchor{library/os os EX_CONFIG}@anchor{243f}
@deffn {Data} os.EX_CONFIG

Exit code that means that some kind of configuration error occurred.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex EX_NOTFOUND (in module os)
@anchor{library/os os EX_NOTFOUND}@anchor{2440}
@deffn {Data} os.EX_NOTFOUND

Exit code that means something like “an entry was not found”.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex fork() (in module os)
@anchor{library/os os fork}@anchor{bb9}
@deffn {Function} os.fork ()

Fork a child process.  Return @code{0} in the child and the child’s process id in the
parent.  If an error occurs @ref{2a1,,OSError} is raised.

Note that some platforms including FreeBSD <= 6.3 and Cygwin have
known issues when using @code{fork()} from a thread.

Raises an @ref{1130,,auditing event} @code{os.fork} with no arguments.


Changed in version 3.8: Calling @code{fork()} in a subinterpreter is no longer supported
(@ref{5af,,RuntimeError} is raised).


@cartouche
@quotation Warning 
See @ref{ee,,ssl} for applications that use the SSL module with fork().
@end quotation
@end cartouche

@ref{1629,,Availability}: Unix.
@end deffn

@geindex forkpty() (in module os)
@anchor{library/os os forkpty}@anchor{2441}
@deffn {Function} os.forkpty ()

Fork a child process, using a new pseudo-terminal as the child’s controlling
terminal. Return a pair of @code{(pid, fd)}, where @emph{pid} is @code{0} in the child, the
new child’s process id in the parent, and @emph{fd} is the file descriptor of the
master end of the pseudo-terminal.  For a more portable approach, use the
@ref{d0,,pty} module.  If an error occurs @ref{2a1,,OSError} is raised.

Raises an @ref{1130,,auditing event} @code{os.forkpty} with no arguments.


Changed in version 3.8: Calling @code{forkpty()} in a subinterpreter is no longer supported
(@ref{5af,,RuntimeError} is raised).


@ref{1629,,Availability}: some flavors of Unix.
@end deffn

@geindex kill() (in module os)
@anchor{library/os os kill}@anchor{ef3}
@deffn {Function} os.kill (pid, sig)

@geindex process; killing
@geindex process; signalling

Send signal @emph{sig} to the process @emph{pid}.  Constants for the specific signals
available on the host platform are defined in the @ref{e5,,signal} module.

Windows: The @ref{2442,,signal.CTRL_C_EVENT} and
@ref{2443,,signal.CTRL_BREAK_EVENT} signals are special signals which can
only be sent to console processes which share a common console window,
e.g., some subprocesses. Any other value for @emph{sig} will cause the process
to be unconditionally killed by the TerminateProcess API, and the exit code
will be set to @emph{sig}. The Windows version of @ref{ef3,,kill()} additionally takes
process handles to be killed.

See also @ref{cc6,,signal.pthread_kill()}.

Raises an @ref{1130,,auditing event} @code{os.kill} with arguments @code{pid}, @code{sig}.


New in version 3.2: Windows support.

@end deffn

@geindex killpg() (in module os)
@anchor{library/os os killpg}@anchor{1278}
@deffn {Function} os.killpg (pgid, sig)

@geindex process; killing
@geindex process; signalling

Send the signal @emph{sig} to the process group @emph{pgid}.

Raises an @ref{1130,,auditing event} @code{os.killpg} with arguments @code{pgid}, @code{sig}.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex nice() (in module os)
@anchor{library/os os nice}@anchor{c94}
@deffn {Function} os.nice (increment)

Add @emph{increment} to the process’s “niceness”.  Return the new niceness.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex pidfd_open() (in module os)
@anchor{library/os os pidfd_open}@anchor{3a6}
@deffn {Function} os.pidfd_open (pid, flags=0)

Return a file descriptor referring to the process @emph{pid}.  This descriptor can
be used to perform process management without races and signals.  The @emph{flags}
argument is provided for future extensions; no flag values are currently
defined.

See the @code{pidfd_open(2)} man page for more details.

@ref{1629,,Availability}: Linux 5.3+


New in version 3.9.

@end deffn

@geindex plock() (in module os)
@anchor{library/os os plock}@anchor{2444}
@deffn {Function} os.plock (op)

Lock program segments into memory.  The value of @emph{op} (defined in
@code{<sys/lock.h>}) determines which segments are locked.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex popen() (in module os)
@anchor{library/os os popen}@anchor{59f}
@deffn {Function} os.popen (cmd, mode='r', buffering=- 1)

Open a pipe to or from command @emph{cmd}.
The return value is an open file object
connected to the pipe, which can be read or written depending on whether @emph{mode}
is @code{'r'} (default) or @code{'w'}. The @emph{buffering} argument has the same meaning as
the corresponding argument to the built-in @ref{1c8,,open()} function. The
returned file object reads or writes text strings rather than bytes.

The @code{close} method returns @ref{182,,None} if the subprocess exited
successfully, or the subprocess’s return code if there was an
error. On POSIX systems, if the return code is positive it
represents the return value of the process left-shifted by one
byte.  If the return code is negative, the process was terminated
by the signal given by the negated value of the return code.  (For
example, the return value might be @code{- signal.SIGKILL} if the
subprocess was killed.)  On Windows systems, the return value
contains the signed integer return code from the child process.

On Unix, @ref{411,,waitstatus_to_exitcode()} can be used to convert the @code{close}
method result (exit status) into an exit code if it is not @code{None}. On
Windows, the @code{close} method result is directly the exit code
(or @code{None}).

This is implemented using @ref{5ae,,subprocess.Popen}; see that class’s
documentation for more powerful ways to manage and communicate with
subprocesses.
@end deffn

@geindex posix_spawn() (in module os)
@anchor{library/os os posix_spawn}@anchor{56e}
@deffn {Function} os.posix_spawn (path, argv, env, *, file_actions=None, setpgroup=None, resetids=False, setsid=False, setsigmask=(), setsigdef=(), scheduler=None)

Wraps the @code{posix_spawn()} C library API for use from Python.

Most users should use @ref{6b7,,subprocess.run()} instead of @ref{56e,,posix_spawn()}.

The positional-only arguments @emph{path}, @emph{args}, and @emph{env} are similar to
@ref{c8c,,execve()}.

The @emph{path} parameter is the path to the executable file.  The @emph{path} should
contain a directory.  Use @ref{12dd,,posix_spawnp()} to pass an executable file
without directory.

The @emph{file_actions} argument may be a sequence of tuples describing actions
to take on specific file descriptors in the child process between the C
library implementation’s @code{fork()} and @code{exec()} steps.
The first item in each tuple must be one of the three type indicator
listed below describing the remaining tuple elements:

@geindex POSIX_SPAWN_OPEN (in module os)
@anchor{library/os os POSIX_SPAWN_OPEN}@anchor{2445}
@deffn {Data} os.POSIX_SPAWN_OPEN

(@code{os.POSIX_SPAWN_OPEN}, @emph{fd}, @emph{path}, @emph{flags}, @emph{mode})

Performs @code{os.dup2(os.open(path, flags, mode), fd)}.
@end deffn

@geindex POSIX_SPAWN_CLOSE (in module os)
@anchor{library/os os POSIX_SPAWN_CLOSE}@anchor{2446}
@deffn {Data} os.POSIX_SPAWN_CLOSE

(@code{os.POSIX_SPAWN_CLOSE}, @emph{fd})

Performs @code{os.close(fd)}.
@end deffn

@geindex POSIX_SPAWN_DUP2 (in module os)
@anchor{library/os os POSIX_SPAWN_DUP2}@anchor{2447}
@deffn {Data} os.POSIX_SPAWN_DUP2

(@code{os.POSIX_SPAWN_DUP2}, @emph{fd}, @emph{new_fd})

Performs @code{os.dup2(fd, new_fd)}.
@end deffn

These tuples correspond to the C library
@code{posix_spawn_file_actions_addopen()},
@code{posix_spawn_file_actions_addclose()}, and
@code{posix_spawn_file_actions_adddup2()} API calls used to prepare
for the @code{posix_spawn()} call itself.

The @emph{setpgroup} argument will set the process group of the child to the value
specified. If the value specified is 0, the child’s process group ID will be
made the same as its process ID. If the value of @emph{setpgroup} is not set, the
child will inherit the parent’s process group ID. This argument corresponds
to the C library @code{POSIX_SPAWN_SETPGROUP} flag.

If the @emph{resetids} argument is @code{True} it will reset the effective UID and
GID of the child to the real UID and GID of the parent process. If the
argument is @code{False}, then the child retains the effective UID and GID of
the parent. In either case, if the set-user-ID and set-group-ID permission
bits are enabled on the executable file, their effect will override the
setting of the effective UID and GID. This argument corresponds to the C
library @code{POSIX_SPAWN_RESETIDS} flag.

If the @emph{setsid} argument is @code{True}, it will create a new session ID
for @cite{posix_spawn}. @emph{setsid} requires @code{POSIX_SPAWN_SETSID}
or @code{POSIX_SPAWN_SETSID_NP} flag. Otherwise, @ref{8a3,,NotImplementedError}
is raised.

The @emph{setsigmask} argument will set the signal mask to the signal set
specified. If the parameter is not used, then the child inherits the
parent’s signal mask. This argument corresponds to the C library
@code{POSIX_SPAWN_SETSIGMASK} flag.

The @emph{sigdef} argument will reset the disposition of all signals in the set
specified. This argument corresponds to the C library
@code{POSIX_SPAWN_SETSIGDEF} flag.

The @emph{scheduler} argument must be a tuple containing the (optional) scheduler
policy and an instance of @ref{2448,,sched_param} with the scheduler parameters.
A value of @code{None} in the place of the scheduler policy indicates that is
not being provided. This argument is a combination of the C library
@code{POSIX_SPAWN_SETSCHEDPARAM} and @code{POSIX_SPAWN_SETSCHEDULER}
flags.

Raises an @ref{1130,,auditing event} @code{os.posix_spawn} with arguments @code{path}, @code{argv}, @code{env}.


New in version 3.8.


@ref{1629,,Availability}: Unix.
@end deffn

@geindex posix_spawnp() (in module os)
@anchor{library/os os posix_spawnp}@anchor{12dd}
@deffn {Function} os.posix_spawnp (path, argv, env, *, file_actions=None, setpgroup=None, resetids=False, setsid=False, setsigmask=(), setsigdef=(), scheduler=None)

Wraps the @code{posix_spawnp()} C library API for use from Python.

Similar to @ref{56e,,posix_spawn()} except that the system searches
for the @emph{executable} file in the list of directories specified by the
@geindex PATH
@geindex environment variable; PATH
@code{PATH} environment variable (in the same way as for @code{execvp(3)}).

Raises an @ref{1130,,auditing event} @code{os.posix_spawn} with arguments @code{path}, @code{argv}, @code{env}.


New in version 3.8.


@ref{1629,,Availability}: See @ref{56e,,posix_spawn()} documentation.
@end deffn

@geindex register_at_fork() (in module os)
@anchor{library/os os register_at_fork}@anchor{685}
@deffn {Function} os.register_at_fork (*, before=None, after_in_parent=None, after_in_child=None)

Register callables to be executed when a new child process is forked
using @ref{bb9,,os.fork()} or similar process cloning APIs.
The parameters are optional and keyword-only.
Each specifies a different call point.


@itemize *

@item 
@emph{before} is a function called before forking a child process.

@item 
@emph{after_in_parent} is a function called from the parent process
after forking a child process.

@item 
@emph{after_in_child} is a function called from the child process.
@end itemize

These calls are only made if control is expected to return to the
Python interpreter.  A typical @ref{f4,,subprocess} launch will not
trigger them as the child is not going to re-enter the interpreter.

Functions registered for execution before forking are called in
reverse registration order.  Functions registered for execution
after forking (either in the parent or in the child) are called
in registration order.

Note that @code{fork()} calls made by third-party C code may not
call those functions, unless it explicitly calls @ref{6f6,,PyOS_BeforeFork()},
@ref{6f7,,PyOS_AfterFork_Parent()} and @ref{5c1,,PyOS_AfterFork_Child()}.

There is no way to unregister a function.

@ref{1629,,Availability}: Unix.


New in version 3.7.

@end deffn

@geindex spawnl() (in module os)
@geindex spawnle() (in module os)
@geindex spawnlp() (in module os)
@geindex spawnlpe() (in module os)
@geindex spawnv() (in module os)
@geindex spawnve() (in module os)
@geindex spawnvp() (in module os)
@geindex spawnvpe() (in module os)
@anchor{library/os os spawnl}@anchor{23e0}
@deffn {Function} os.spawnl (mode, path, ...)
@anchor{library/os os spawnle}@anchor{2449}
@deffnx {Function} os.spawnle (mode, path, ..., env)
@anchor{library/os os spawnlp}@anchor{244a}
@deffnx {Function} os.spawnlp (mode, file, ...)
@anchor{library/os os spawnlpe}@anchor{244b}
@deffnx {Function} os.spawnlpe (mode, file, ..., env)
@anchor{library/os os spawnv}@anchor{244c}
@deffnx {Function} os.spawnv (mode, path, args)
@anchor{library/os os spawnve}@anchor{244d}
@deffnx {Function} os.spawnve (mode, path, args, env)
@anchor{library/os os spawnvp}@anchor{244e}
@deffnx {Function} os.spawnvp (mode, file, args)
@anchor{library/os os spawnvpe}@anchor{244f}
@deffnx {Function} os.spawnvpe (mode, file, args, env)

Execute the program @emph{path} in a new process.

(Note that the @ref{f4,,subprocess} module provides more powerful facilities for
spawning new processes and retrieving their results; using that module is
preferable to using these functions.  Check especially the
@ref{d28,,Replacing Older Functions with the subprocess Module} section.)

If @emph{mode} is @ref{2450,,P_NOWAIT}, this function returns the process id of the new
process; if @emph{mode} is @ref{2451,,P_WAIT}, returns the process’s exit code if it
exits normally, or @code{-signal}, where @emph{signal} is the signal that killed the
process.  On Windows, the process id will actually be the process handle, so can
be used with the @ref{8f6,,waitpid()} function.

Note on VxWorks, this function doesn’t return @code{-signal} when the new process is
killed. Instead it raises OSError exception.

The “l” and “v” variants of the @ref{23e0,,spawn*} functions differ in how
command-line arguments are passed.  The “l” variants are perhaps the easiest
to work with if the number of parameters is fixed when the code is written; the
individual parameters simply become additional parameters to the
@code{spawnl*()} functions.  The “v” variants are good when the number of
parameters is variable, with the arguments being passed in a list or tuple as
the @emph{args} parameter.  In either case, the arguments to the child process must
start with the name of the command being run.

The variants which include a second “p” near the end (@ref{244a,,spawnlp()},
@ref{244b,,spawnlpe()}, @ref{244e,,spawnvp()}, and @ref{244f,,spawnvpe()}) will use the
@geindex PATH
@geindex environment variable; PATH
@code{PATH} environment variable to locate the program @emph{file}.  When the
environment is being replaced (using one of the @ref{23e0,,spawn*e} variants,
discussed in the next paragraph), the new environment is used as the source of
the 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} variable.  The other variants, @ref{23e0,,spawnl()},
@ref{2449,,spawnle()}, @ref{244c,,spawnv()}, and @ref{244d,,spawnve()}, will not use the
@geindex PATH
@geindex environment variable; PATH
@code{PATH} variable to locate the executable; @emph{path} must contain an
appropriate absolute or relative path.

For @ref{2449,,spawnle()}, @ref{244b,,spawnlpe()}, @ref{244d,,spawnve()}, and @ref{244f,,spawnvpe()}
(note that these all end in “e”), the @emph{env} parameter must be a mapping
which is used to define the environment variables for the new process (they are
used instead of the current process’ environment); the functions
@ref{23e0,,spawnl()}, @ref{244a,,spawnlp()}, @ref{244c,,spawnv()}, and @ref{244e,,spawnvp()} all cause
the new process to inherit the environment of the current process.  Note that
keys and values in the @emph{env} dictionary must be strings; invalid keys or
values will cause the function to fail, with a return value of @code{127}.

As an example, the following calls to @ref{244a,,spawnlp()} and @ref{244f,,spawnvpe()} are
equivalent:

@example
import os
os.spawnlp(os.P_WAIT, 'cp', 'cp', 'index.html', '/dev/null')

L = ['cp', 'index.html', '/dev/null']
os.spawnvpe(os.P_WAIT, 'cp', L, os.environ)
@end example

Raises an @ref{1130,,auditing event} @code{os.spawn} with arguments @code{mode}, @code{path}, @code{args}, @code{env}.

@ref{1629,,Availability}: Unix, Windows.  @ref{244a,,spawnlp()}, @ref{244b,,spawnlpe()}, @ref{244e,,spawnvp()}
and @ref{244f,,spawnvpe()} are not available on Windows.  @ref{2449,,spawnle()} and
@ref{244d,,spawnve()} are not thread-safe on Windows; we advise you to use the
@ref{f4,,subprocess} module instead.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex P_NOWAIT (in module os)
@geindex P_NOWAITO (in module os)
@anchor{library/os os P_NOWAIT}@anchor{2450}
@deffn {Data} os.P_NOWAIT
@anchor{library/os os P_NOWAITO}@anchor{2452}
@deffnx {Data} os.P_NOWAITO

Possible values for the @emph{mode} parameter to the @ref{23e0,,spawn*} family of
functions.  If either of these values is given, the @code{spawn*()} functions
will return as soon as the new process has been created, with the process id as
the return value.

@ref{1629,,Availability}: Unix, Windows.
@end deffn

@geindex P_WAIT (in module os)
@anchor{library/os os P_WAIT}@anchor{2451}
@deffn {Data} os.P_WAIT

Possible value for the @emph{mode} parameter to the @ref{23e0,,spawn*} family of
functions.  If this is given as @emph{mode}, the @code{spawn*()} functions will not
return until the new process has run to completion and will return the exit code
of the process the run is successful, or @code{-signal} if a signal kills the
process.

@ref{1629,,Availability}: Unix, Windows.
@end deffn

@geindex P_DETACH (in module os)
@geindex P_OVERLAY (in module os)
@anchor{library/os os P_DETACH}@anchor{2453}
@deffn {Data} os.P_DETACH
@anchor{library/os os P_OVERLAY}@anchor{2454}
@deffnx {Data} os.P_OVERLAY

Possible values for the @emph{mode} parameter to the @ref{23e0,,spawn*} family of
functions.  These are less portable than those listed above. @ref{2453,,P_DETACH}
is similar to @ref{2450,,P_NOWAIT}, but the new process is detached from the
console of the calling process. If @ref{2454,,P_OVERLAY} is used, the current
process will be replaced; the @ref{23e0,,spawn*} function will not return.

@ref{1629,,Availability}: Windows.
@end deffn

@geindex startfile() (in module os)
@anchor{library/os os startfile}@anchor{1143}
@deffn {Function} os.startfile (path[, operation][, arguments][, cwd][, show_cmd])

Start a file with its associated application.

When @emph{operation} is not specified or @code{'open'}, this acts like double-clicking
the file in Windows Explorer, or giving the file name as an argument to the
@code{start} command from the interactive command shell: the file is opened
with whatever application (if any) its extension is associated.

When another @emph{operation} is given, it must be a “command verb” that specifies
what should be done with the file. Common verbs documented by Microsoft are
@code{'print'} and  @code{'edit'} (to be used on files) as well as @code{'explore'} and
@code{'find'} (to be used on directories).

When launching an application, specify @emph{arguments} to be passed as a single
string. This argument may have no effect when using this function to launch a
document.

The default working directory is inherited, but may be overridden by the @emph{cwd}
argument. This should be an absolute path. A relative @emph{path} will be resolved
against this argument.

Use @emph{show_cmd} to override the default window style. Whether this has any
effect will depend on the application being launched. Values are integers as
supported by the Win32 @code{ShellExecute()} function.

@ref{1143,,startfile()} returns as soon as the associated application is launched.
There is no option to wait for the application to close, and no way to retrieve
the application’s exit status.  The @emph{path} parameter is relative to the current
directory or @emph{cwd}.  If you want to use an absolute path, make sure the first
character is not a slash (@code{'/'})  Use @ref{c3,,pathlib} or the
@ref{eb1,,os.path.normpath()} function to ensure that paths are properly encoded for
Win32.

To reduce interpreter startup overhead, the Win32 @code{ShellExecute()}
function is not resolved until this function is first called.  If the function
cannot be resolved, @ref{8a3,,NotImplementedError} will be raised.

Raises an @ref{1130,,auditing event} @code{os.startfile} with arguments @code{path}, @code{operation}.

Raises an @ref{1130,,auditing event} @code{os.startfile/2} with arguments @code{path}, @code{operation}, @code{arguments}, @code{cwd}, @code{show_cmd}.

@ref{1629,,Availability}: Windows.


Changed in version 3.10: Added the @emph{arguments}, @emph{cwd} and @emph{show_cmd} arguments, and the
@code{os.startfile/2} audit event.

@end deffn

@geindex system() (in module os)
@anchor{library/os os system}@anchor{fbb}
@deffn {Function} os.system (command)

Execute the command (a string) in a subshell.  This is implemented by calling
the Standard C function @code{system()}, and has the same limitations.
Changes to @ref{5ee,,sys.stdin}, etc. are not reflected in the environment of
the executed command. If @emph{command} generates any output, it will be sent to
the interpreter standard output stream. The C standard does not
specify the meaning of the return value of the C function, so the return
value of the Python function is system-dependent.

On Unix, the return value is the exit status of the process encoded in the
format specified for @ref{8f4,,wait()}.

On Windows, the return value is that returned by the system shell after
running @emph{command}.  The shell is given by the Windows environment variable
@geindex COMSPEC
@geindex environment variable; COMSPEC
@code{COMSPEC}: it is usually @code{cmd.exe}, which returns the exit
status of the command run; on systems using a non-native shell, consult your
shell documentation.

The @ref{f4,,subprocess} module provides more powerful facilities for spawning
new processes and retrieving their results; using that module is preferable
to using this function.  See the @ref{d28,,Replacing Older Functions with the subprocess Module} section in
the @ref{f4,,subprocess} documentation for some helpful recipes.

On Unix, @ref{411,,waitstatus_to_exitcode()} can be used to convert the result
(exit status) into an exit code. On Windows, the result is directly the exit
code.

Raises an @ref{1130,,auditing event} @code{os.system} with argument @code{command}.

@ref{1629,,Availability}: Unix, Windows.
@end deffn

@geindex times() (in module os)
@anchor{library/os os times}@anchor{ca8}
@deffn {Function} os.times ()

Returns the current global process times.
The return value is an object with five attributes:


@itemize *

@item 
@code{user} - user time

@item 
@code{system} - system time

@item 
@code{children_user} - user time of all child processes

@item 
@code{children_system} - system time of all child processes

@item 
@code{elapsed} - elapsed real time since a fixed point in the past
@end itemize

For backwards compatibility, this object also behaves like a five-tuple
containing @code{user}, @code{system}, @code{children_user},
@code{children_system}, and @code{elapsed} in that order.

See the Unix manual page
@code{times(2)} and @code{times(3)} manual page on Unix or the GetProcessTimes MSDN@footnote{https://docs.microsoft.com/windows/win32/api/processthreadsapi/nf-processthreadsapi-getprocesstimes}
on Windows. On Windows, only @code{user} and @code{system} are known; the other attributes are zero.

@ref{1629,,Availability}: Unix, Windows.


Changed in version 3.3: Return type changed from a tuple to a tuple-like object
with named attributes.

@end deffn

@geindex wait() (in module os)
@anchor{library/os os wait}@anchor{8f4}
@deffn {Function} os.wait ()

Wait for completion of a child process, and return a tuple containing its pid
and exit status indication: a 16-bit number, whose low byte is the signal number
that killed the process, and whose high byte is the exit status (if the signal
number is zero); the high bit of the low byte is set if a core file was
produced.

@ref{411,,waitstatus_to_exitcode()} can be used to convert the exit status into an
exit code.

@ref{1629,,Availability}: Unix.


@subsubheading See also


@ref{8f6,,waitpid()} can be used to wait for the completion of a specific
child process and has more options.

@end deffn

@geindex waitid() (in module os)
@anchor{library/os os waitid}@anchor{8f5}
@deffn {Function} os.waitid (idtype, id, options)

Wait for the completion of one or more child processes.
@emph{idtype} can be @ref{2455,,P_PID}, @ref{2456,,P_PGID}, @ref{2457,,P_ALL}, or
@ref{40e,,P_PIDFD} on Linux.
@emph{id} specifies the pid to wait on.
@emph{options} is constructed from the ORing of one or more of @ref{2458,,WEXITED},
@ref{2459,,WSTOPPED} or @ref{245a,,WCONTINUED} and additionally may be ORed with
@ref{245b,,WNOHANG} or @ref{245c,,WNOWAIT}. The return value is an object
representing the data contained in the @code{siginfo_t} structure, namely:
@code{si_pid}, @code{si_uid}, @code{si_signo}, @code{si_status},
@code{si_code} or @code{None} if @ref{245b,,WNOHANG} is specified and there are no
children in a waitable state.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex P_PID (in module os)
@geindex P_PGID (in module os)
@geindex P_ALL (in module os)
@anchor{library/os os P_PID}@anchor{2455}
@deffn {Data} os.P_PID
@anchor{library/os os P_PGID}@anchor{2456}
@deffnx {Data} os.P_PGID
@anchor{library/os os P_ALL}@anchor{2457}
@deffnx {Data} os.P_ALL

These are the possible values for @emph{idtype} in @ref{8f5,,waitid()}. They affect
how @emph{id} is interpreted.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex P_PIDFD (in module os)
@anchor{library/os os P_PIDFD}@anchor{40e}
@deffn {Data} os.P_PIDFD

This is a Linux-specific @emph{idtype} that indicates that @emph{id} is a file
descriptor that refers to a process.

@ref{1629,,Availability}: Linux 5.4+


New in version 3.9.

@end deffn

@geindex WEXITED (in module os)
@geindex WSTOPPED (in module os)
@geindex WNOWAIT (in module os)
@anchor{library/os os WEXITED}@anchor{2458}
@deffn {Data} os.WEXITED
@anchor{library/os os WSTOPPED}@anchor{2459}
@deffnx {Data} os.WSTOPPED
@anchor{library/os os WNOWAIT}@anchor{245c}
@deffnx {Data} os.WNOWAIT

Flags that can be used in @emph{options} in @ref{8f5,,waitid()} that specify what
child signal to wait for.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex CLD_EXITED (in module os)
@geindex CLD_KILLED (in module os)
@geindex CLD_DUMPED (in module os)
@geindex CLD_TRAPPED (in module os)
@geindex CLD_STOPPED (in module os)
@geindex CLD_CONTINUED (in module os)
@anchor{library/os os CLD_EXITED}@anchor{245d}
@deffn {Data} os.CLD_EXITED
@anchor{library/os os CLD_KILLED}@anchor{40c}
@deffnx {Data} os.CLD_KILLED
@anchor{library/os os CLD_DUMPED}@anchor{245e}
@deffnx {Data} os.CLD_DUMPED
@anchor{library/os os CLD_TRAPPED}@anchor{245f}
@deffnx {Data} os.CLD_TRAPPED
@anchor{library/os os CLD_STOPPED}@anchor{40d}
@deffnx {Data} os.CLD_STOPPED
@anchor{library/os os CLD_CONTINUED}@anchor{2460}
@deffnx {Data} os.CLD_CONTINUED

These are the possible values for @code{si_code} in the result returned by
@ref{8f5,,waitid()}.

@ref{1629,,Availability}: Unix.


New in version 3.3.



Changed in version 3.9: Added @ref{40c,,CLD_KILLED} and @ref{40d,,CLD_STOPPED} values.

@end deffn

@geindex waitpid() (in module os)
@anchor{library/os os waitpid}@anchor{8f6}
@deffn {Function} os.waitpid (pid, options)

The details of this function differ on Unix and Windows.

On Unix: Wait for completion of a child process given by process id @emph{pid}, and
return a tuple containing its process id and exit status indication (encoded as
for @ref{8f4,,wait()}).  The semantics of the call are affected by the value of the
integer @emph{options}, which should be @code{0} for normal operation.

If @emph{pid} is greater than @code{0}, @ref{8f6,,waitpid()} requests status information for
that specific process.  If @emph{pid} is @code{0}, the request is for the status of any
child in the process group of the current process.  If @emph{pid} is @code{-1}, the
request pertains to any child of the current process.  If @emph{pid} is less than
@code{-1}, status is requested for any process in the process group @code{-pid} (the
absolute value of @emph{pid}).

An @ref{2a1,,OSError} is raised with the value of errno when the syscall
returns -1.

On Windows: Wait for completion of a process given by process handle @emph{pid}, and
return a tuple containing @emph{pid}, and its exit status shifted left by 8 bits
(shifting makes cross-platform use of the function easier). A @emph{pid} less than or
equal to @code{0} has no special meaning on Windows, and raises an exception. The
value of integer @emph{options} has no effect. @emph{pid} can refer to any process whose
id is known, not necessarily a child process. The @ref{23e0,,spawn*}
functions called with @ref{2450,,P_NOWAIT} return suitable process handles.

@ref{411,,waitstatus_to_exitcode()} can be used to convert the exit status into an
exit code.


Changed in version 3.5: If the system call is interrupted and the signal handler does not raise an
exception, the function now retries the system call instead of raising an
@ref{8e3,,InterruptedError} exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).

@end deffn

@geindex wait3() (in module os)
@anchor{library/os os wait3}@anchor{8f2}
@deffn {Function} os.wait3 (options)

Similar to @ref{8f6,,waitpid()}, except no process id argument is given and a
3-element tuple containing the child’s process id, exit status indication,
and resource usage information is returned.  Refer to
@ref{db,,resource}.@ref{f93,,getrusage()} for details on resource usage
information.  The option argument is the same as that provided to
@ref{8f6,,waitpid()} and @ref{8f3,,wait4()}.

@ref{411,,waitstatus_to_exitcode()} can be used to convert the exit status into an
exitcode.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex wait4() (in module os)
@anchor{library/os os wait4}@anchor{8f3}
@deffn {Function} os.wait4 (pid, options)

Similar to @ref{8f6,,waitpid()}, except a 3-element tuple, containing the child’s
process id, exit status indication, and resource usage information is returned.
Refer to @ref{db,,resource}.@ref{f93,,getrusage()} for details on
resource usage information.  The arguments to @ref{8f3,,wait4()} are the same
as those provided to @ref{8f6,,waitpid()}.

@ref{411,,waitstatus_to_exitcode()} can be used to convert the exit status into an
exitcode.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex waitstatus_to_exitcode() (in module os)
@anchor{library/os os waitstatus_to_exitcode}@anchor{411}
@deffn {Function} os.waitstatus_to_exitcode (status)

Convert a wait status to an exit code.

On Unix:


@itemize *

@item 
If the process exited normally (if @code{WIFEXITED(status)} is true),
return the process exit status (return @code{WEXITSTATUS(status)}):
result greater than or equal to 0.

@item 
If the process was terminated by a signal (if @code{WIFSIGNALED(status)} is
true), return @code{-signum} where @emph{signum} is the number of the signal that
caused the process to terminate (return @code{-WTERMSIG(status)}):
result less than 0.

@item 
Otherwise, raise a @ref{17c,,ValueError}.
@end itemize

On Windows, return @emph{status} shifted right by 8 bits.

On Unix, if the process is being traced or if @ref{8f6,,waitpid()} was called
with @ref{2461,,WUNTRACED} option, the caller must first check if
@code{WIFSTOPPED(status)} is true. This function must not be called if
@code{WIFSTOPPED(status)} is true.


@subsubheading See also


@ref{2462,,WIFEXITED()}, @ref{2463,,WEXITSTATUS()}, @ref{2464,,WIFSIGNALED()},
@ref{2465,,WTERMSIG()}, @ref{2466,,WIFSTOPPED()}, @ref{2467,,WSTOPSIG()} functions.



New in version 3.9.

@end deffn

@geindex WNOHANG (in module os)
@anchor{library/os os WNOHANG}@anchor{245b}
@deffn {Data} os.WNOHANG

The option for @ref{8f6,,waitpid()} to return immediately if no child process status
is available immediately. The function returns @code{(0, 0)} in this case.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex WCONTINUED (in module os)
@anchor{library/os os WCONTINUED}@anchor{245a}
@deffn {Data} os.WCONTINUED

This option causes child processes to be reported if they have been continued
from a job control stop since their status was last reported.

@ref{1629,,Availability}: some Unix systems.
@end deffn

@geindex WUNTRACED (in module os)
@anchor{library/os os WUNTRACED}@anchor{2461}
@deffn {Data} os.WUNTRACED

This option causes child processes to be reported if they have been stopped but
their current state has not been reported since they were stopped.

@ref{1629,,Availability}: Unix.
@end deffn

The following functions take a process status code as returned by
@ref{fbb,,system()}, @ref{8f4,,wait()}, or @ref{8f6,,waitpid()} as a parameter.  They may be
used to determine the disposition of a process.

@geindex WCOREDUMP() (in module os)
@anchor{library/os os WCOREDUMP}@anchor{2468}
@deffn {Function} os.WCOREDUMP (status)

Return @code{True} if a core dump was generated for the process, otherwise
return @code{False}.

This function should be employed only if @ref{2464,,WIFSIGNALED()} is true.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex WIFCONTINUED() (in module os)
@anchor{library/os os WIFCONTINUED}@anchor{2469}
@deffn {Function} os.WIFCONTINUED (status)

Return @code{True} if a stopped child has been resumed by delivery of
@ref{246a,,SIGCONT} (if the process has been continued from a job
control stop), otherwise return @code{False}.

See @ref{245a,,WCONTINUED} option.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex WIFSTOPPED() (in module os)
@anchor{library/os os WIFSTOPPED}@anchor{2466}
@deffn {Function} os.WIFSTOPPED (status)

Return @code{True} if the process was stopped by delivery of a signal,
otherwise return @code{False}.

@ref{2466,,WIFSTOPPED()} only returns @code{True} if the @ref{8f6,,waitpid()} call was
done using @ref{2461,,WUNTRACED} option or when the process is being traced (see
@code{ptrace(2)}).

@ref{1629,,Availability}: Unix.
@end deffn

@geindex WIFSIGNALED() (in module os)
@anchor{library/os os WIFSIGNALED}@anchor{2464}
@deffn {Function} os.WIFSIGNALED (status)

Return @code{True} if the process was terminated by a signal, otherwise return
@code{False}.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex WIFEXITED() (in module os)
@anchor{library/os os WIFEXITED}@anchor{2462}
@deffn {Function} os.WIFEXITED (status)

Return @code{True} if the process exited terminated normally, that is,
by calling @code{exit()} or @code{_exit()}, or by returning from @code{main()};
otherwise return @code{False}.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex WEXITSTATUS() (in module os)
@anchor{library/os os WEXITSTATUS}@anchor{2463}
@deffn {Function} os.WEXITSTATUS (status)

Return the process exit status.

This function should be employed only if @ref{2462,,WIFEXITED()} is true.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex WSTOPSIG() (in module os)
@anchor{library/os os WSTOPSIG}@anchor{2467}
@deffn {Function} os.WSTOPSIG (status)

Return the signal which caused the process to stop.

This function should be employed only if @ref{2466,,WIFSTOPPED()} is true.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex WTERMSIG() (in module os)
@anchor{library/os os WTERMSIG}@anchor{2465}
@deffn {Function} os.WTERMSIG (status)

Return the number of the signal that caused the process to terminate.

This function should be employed only if @ref{2464,,WIFSIGNALED()} is true.

@ref{1629,,Availability}: Unix.
@end deffn

@node Interface to the scheduler,Miscellaneous System Information,Process Management,os — Miscellaneous operating system interfaces
@anchor{library/os interface-to-the-scheduler}@anchor{246b}
@subsubsection Interface to the scheduler


These functions control how a process is allocated CPU time by the operating
system. They are only available on some Unix platforms. For more detailed
information, consult your Unix manpages.


New in version 3.3.


The following scheduling policies are exposed if they are supported by the
operating system.

@geindex SCHED_OTHER (in module os)
@anchor{library/os os SCHED_OTHER}@anchor{246c}
@deffn {Data} os.SCHED_OTHER

The default scheduling policy.
@end deffn

@geindex SCHED_BATCH (in module os)
@anchor{library/os os SCHED_BATCH}@anchor{246d}
@deffn {Data} os.SCHED_BATCH

Scheduling policy for CPU-intensive processes that tries to preserve
interactivity on the rest of the computer.
@end deffn

@geindex SCHED_IDLE (in module os)
@anchor{library/os os SCHED_IDLE}@anchor{246e}
@deffn {Data} os.SCHED_IDLE

Scheduling policy for extremely low priority background tasks.
@end deffn

@geindex SCHED_SPORADIC (in module os)
@anchor{library/os os SCHED_SPORADIC}@anchor{246f}
@deffn {Data} os.SCHED_SPORADIC

Scheduling policy for sporadic server programs.
@end deffn

@geindex SCHED_FIFO (in module os)
@anchor{library/os os SCHED_FIFO}@anchor{2470}
@deffn {Data} os.SCHED_FIFO

A First In First Out scheduling policy.
@end deffn

@geindex SCHED_RR (in module os)
@anchor{library/os os SCHED_RR}@anchor{2471}
@deffn {Data} os.SCHED_RR

A round-robin scheduling policy.
@end deffn

@geindex SCHED_RESET_ON_FORK (in module os)
@anchor{library/os os SCHED_RESET_ON_FORK}@anchor{2472}
@deffn {Data} os.SCHED_RESET_ON_FORK

This flag can be OR’ed with any other scheduling policy. When a process with
this flag set forks, its child’s scheduling policy and priority are reset to
the default.
@end deffn

@geindex sched_param (class in os)
@anchor{library/os os sched_param}@anchor{2448}
@deffn {Class} os.sched_param (sched_priority)

This class represents tunable scheduling parameters used in
@ref{ca2,,sched_setparam()}, @ref{ca3,,sched_setscheduler()}, and
@ref{c9e,,sched_getparam()}. It is immutable.

At the moment, there is only one possible parameter:

@geindex sched_priority (os.sched_param attribute)
@anchor{library/os os sched_param sched_priority}@anchor{2473}
@deffn {Attribute} sched_priority

The scheduling priority for a scheduling policy.
@end deffn
@end deffn

@geindex sched_get_priority_min() (in module os)
@anchor{library/os os sched_get_priority_min}@anchor{c9c}
@deffn {Function} os.sched_get_priority_min (policy)

Get the minimum priority value for @emph{policy}. @emph{policy} is one of the
scheduling policy constants above.
@end deffn

@geindex sched_get_priority_max() (in module os)
@anchor{library/os os sched_get_priority_max}@anchor{c9b}
@deffn {Function} os.sched_get_priority_max (policy)

Get the maximum priority value for @emph{policy}. @emph{policy} is one of the
scheduling policy constants above.
@end deffn

@geindex sched_setscheduler() (in module os)
@anchor{library/os os sched_setscheduler}@anchor{ca3}
@deffn {Function} os.sched_setscheduler (pid, policy, param)

Set the scheduling policy for the process with PID @emph{pid}. A @emph{pid} of 0 means
the calling process. @emph{policy} is one of the scheduling policy constants
above. @emph{param} is a @ref{2448,,sched_param} instance.
@end deffn

@geindex sched_getscheduler() (in module os)
@anchor{library/os os sched_getscheduler}@anchor{c9f}
@deffn {Function} os.sched_getscheduler (pid)

Return the scheduling policy for the process with PID @emph{pid}. A @emph{pid} of 0
means the calling process. The result is one of the scheduling policy
constants above.
@end deffn

@geindex sched_setparam() (in module os)
@anchor{library/os os sched_setparam}@anchor{ca2}
@deffn {Function} os.sched_setparam (pid, param)

Set the scheduling parameters for the process with PID @emph{pid}. A @emph{pid} of 0 means
the calling process. @emph{param} is a @ref{2448,,sched_param} instance.
@end deffn

@geindex sched_getparam() (in module os)
@anchor{library/os os sched_getparam}@anchor{c9e}
@deffn {Function} os.sched_getparam (pid)

Return the scheduling parameters as a @ref{2448,,sched_param} instance for the
process with PID @emph{pid}. A @emph{pid} of 0 means the calling process.
@end deffn

@geindex sched_rr_get_interval() (in module os)
@anchor{library/os os sched_rr_get_interval}@anchor{ca0}
@deffn {Function} os.sched_rr_get_interval (pid)

Return the round-robin quantum in seconds for the process with PID @emph{pid}. A
@emph{pid} of 0 means the calling process.
@end deffn

@geindex sched_yield() (in module os)
@anchor{library/os os sched_yield}@anchor{ca4}
@deffn {Function} os.sched_yield ()

Voluntarily relinquish the CPU.
@end deffn

@geindex sched_setaffinity() (in module os)
@anchor{library/os os sched_setaffinity}@anchor{ca1}
@deffn {Function} os.sched_setaffinity (pid, mask)

Restrict the process with PID @emph{pid} (or the current process if zero) to a
set of CPUs.  @emph{mask} is an iterable of integers representing the set of
CPUs to which the process should be restricted.
@end deffn

@geindex sched_getaffinity() (in module os)
@anchor{library/os os sched_getaffinity}@anchor{c9d}
@deffn {Function} os.sched_getaffinity (pid)

Return the set of CPUs the process with PID @emph{pid} (or the current process
if zero) is restricted to.
@end deffn

@node Miscellaneous System Information,Random numbers<2>,Interface to the scheduler,os — Miscellaneous operating system interfaces
@anchor{library/os miscellaneous-system-information}@anchor{2474}@anchor{library/os os-path}@anchor{2475}
@subsubsection Miscellaneous System Information


@geindex confstr() (in module os)
@anchor{library/os os confstr}@anchor{2476}
@deffn {Function} os.confstr (name)

Return string-valued system configuration values. @emph{name} specifies the
configuration value to retrieve; it may be a string which is the name of a
defined system value; these names are specified in a number of standards (POSIX,
Unix 95, Unix 98, and others).  Some platforms define additional names as well.
The names known to the host operating system are given as the keys of the
@code{confstr_names} dictionary.  For configuration variables not included in that
mapping, passing an integer for @emph{name} is also accepted.

If the configuration value specified by @emph{name} isn’t defined, @code{None} is
returned.

If @emph{name} is a string and is not known, @ref{17c,,ValueError} is raised.  If a
specific value for @emph{name} is not supported by the host system, even if it is
included in @code{confstr_names}, an @ref{2a1,,OSError} is raised with
@ref{1298,,errno.EINVAL} for the error number.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex confstr_names (in module os)
@anchor{library/os os confstr_names}@anchor{2477}
@deffn {Data} os.confstr_names

Dictionary mapping names accepted by @ref{2476,,confstr()} to the integer values
defined for those names by the host operating system. This can be used to
determine the set of names known to the system.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex cpu_count() (in module os)
@anchor{library/os os cpu_count}@anchor{298}
@deffn {Function} os.cpu_count ()

Return the number of CPUs in the system. Returns @code{None} if undetermined.

This number is not equivalent to the number of CPUs the current process can
use.  The number of usable CPUs can be obtained with
@code{len(os.sched_getaffinity(0))}


New in version 3.4.

@end deffn

@geindex getloadavg() (in module os)
@anchor{library/os os getloadavg}@anchor{2478}
@deffn {Function} os.getloadavg ()

Return the number of processes in the system run queue averaged over the last
1, 5, and 15 minutes or raises @ref{2a1,,OSError} if the load average was
unobtainable.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex sysconf() (in module os)
@anchor{library/os os sysconf}@anchor{23cd}
@deffn {Function} os.sysconf (name)

Return integer-valued system configuration values. If the configuration value
specified by @emph{name} isn’t defined, @code{-1} is returned.  The comments regarding
the @emph{name} parameter for @ref{2476,,confstr()} apply here as well; the dictionary that
provides information on the known names is given by @code{sysconf_names}.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex sysconf_names (in module os)
@anchor{library/os os sysconf_names}@anchor{2479}
@deffn {Data} os.sysconf_names

Dictionary mapping names accepted by @ref{23cd,,sysconf()} to the integer values
defined for those names by the host operating system. This can be used to
determine the set of names known to the system.

@ref{1629,,Availability}: Unix.
@end deffn

The following data values are used to support path manipulation operations.  These
are defined for all platforms.

Higher-level operations on pathnames are defined in the @ref{c1,,os.path} module.

@geindex . (dot); in pathnames

@geindex curdir (in module os)
@anchor{library/os os curdir}@anchor{208f}
@deffn {Data} os.curdir

The constant string used by the operating system to refer to the current
directory. This is @code{'.'} for Windows and POSIX. Also available via
@ref{c1,,os.path}.
@end deffn

@geindex ..; in pathnames

@geindex pardir (in module os)
@anchor{library/os os pardir}@anchor{23ea}
@deffn {Data} os.pardir

The constant string used by the operating system to refer to the parent
directory. This is @code{'..'} for Windows and POSIX. Also available via
@ref{c1,,os.path}.
@end deffn

@geindex / (slash); in pathnames

@geindex \ (backslash); in pathnames (Windows)

@geindex sep (in module os)
@anchor{library/os os sep}@anchor{210d}
@deffn {Data} os.sep

The character used by the operating system to separate pathname components.
This is @code{'/'} for POSIX and @code{'\\'} for Windows.  Note that knowing this
is not sufficient to be able to parse or concatenate pathnames — use
@ref{2089,,os.path.split()} and @ref{1260,,os.path.join()} — but it is occasionally
useful. Also available via @ref{c1,,os.path}.
@end deffn

@geindex / (slash); in pathnames

@geindex altsep (in module os)
@anchor{library/os os altsep}@anchor{210e}
@deffn {Data} os.altsep

An alternative character used by the operating system to separate pathname
components, or @code{None} if only one separator character exists.  This is set to
@code{'/'} on Windows systems where @code{sep} is a backslash. Also available via
@ref{c1,,os.path}.
@end deffn

@geindex . (dot); in pathnames

@geindex extsep (in module os)
@anchor{library/os os extsep}@anchor{247a}
@deffn {Data} os.extsep

The character which separates the base filename from the extension; for example,
the @code{'.'} in @code{os.py}. Also available via @ref{c1,,os.path}.
@end deffn

@geindex ; (colon); path separator (POSIX)
@geindex ; (semicolon)

@geindex pathsep (in module os)
@anchor{library/os os pathsep}@anchor{161d}
@deffn {Data} os.pathsep

The character conventionally used by the operating system to separate search
path components (as in 
@geindex PATH
@geindex environment variable; PATH
@code{PATH}), such as @code{':'} for POSIX or @code{';'} for
Windows. Also available via @ref{c1,,os.path}.
@end deffn

@geindex defpath (in module os)
@anchor{library/os os defpath}@anchor{12ad}
@deffn {Data} os.defpath

The default search path used by @ref{2429,,exec*p*} and
@ref{23e0,,spawn*p*} if the environment doesn’t have a @code{'PATH'}
key. Also available via @ref{c1,,os.path}.
@end deffn

@geindex linesep (in module os)
@anchor{library/os os linesep}@anchor{1a59}
@deffn {Data} os.linesep

The string used to separate (or, rather, terminate) lines on the current
platform.  This may be a single character, such as @code{'\n'} for POSIX, or
multiple characters, for example, @code{'\r\n'} for Windows. Do not use
@emph{os.linesep} as a line terminator when writing files opened in text mode (the
default); use a single @code{'\n'} instead, on all platforms.
@end deffn

@geindex devnull (in module os)
@anchor{library/os os devnull}@anchor{12e2}
@deffn {Data} os.devnull

The file path of the null device. For example: @code{'/dev/null'} for
POSIX, @code{'nul'} for Windows.  Also available via @ref{c1,,os.path}.
@end deffn

@geindex RTLD_LAZY (in module os)
@geindex RTLD_NOW (in module os)
@geindex RTLD_GLOBAL (in module os)
@geindex RTLD_LOCAL (in module os)
@geindex RTLD_NODELETE (in module os)
@geindex RTLD_NOLOAD (in module os)
@geindex RTLD_DEEPBIND (in module os)
@anchor{library/os os RTLD_LAZY}@anchor{cab}
@deffn {Data} os.RTLD_LAZY
@anchor{library/os os RTLD_NOW}@anchor{cac}
@deffnx {Data} os.RTLD_NOW
@anchor{library/os os RTLD_GLOBAL}@anchor{cad}
@deffnx {Data} os.RTLD_GLOBAL
@anchor{library/os os RTLD_LOCAL}@anchor{cae}
@deffnx {Data} os.RTLD_LOCAL
@anchor{library/os os RTLD_NODELETE}@anchor{caf}
@deffnx {Data} os.RTLD_NODELETE
@anchor{library/os os RTLD_NOLOAD}@anchor{cb0}
@deffnx {Data} os.RTLD_NOLOAD
@anchor{library/os os RTLD_DEEPBIND}@anchor{cb1}
@deffnx {Data} os.RTLD_DEEPBIND

Flags for use with the @ref{cb2,,setdlopenflags()} and
@ref{1044,,getdlopenflags()} functions.  See the Unix manual page
@code{dlopen(3)} for what the different flags mean.


New in version 3.3.

@end deffn

@node Random numbers<2>,,Miscellaneous System Information,os — Miscellaneous operating system interfaces
@anchor{library/os random-numbers}@anchor{247b}
@subsubsection Random numbers


@geindex getrandom() (in module os)
@anchor{library/os os getrandom}@anchor{806}
@deffn {Function} os.getrandom (size, flags=0)

Get up to @emph{size} random bytes. The function can return less bytes than
requested.

These bytes can be used to seed user-space random number generators or for
cryptographic purposes.

@code{getrandom()} relies on entropy gathered from device drivers and other
sources of environmental noise. Unnecessarily reading large quantities of
data will have a negative impact on  other users  of the @code{/dev/random} and
@code{/dev/urandom} devices.

The flags argument is a bit mask that can contain zero or more of the
following values ORed together: @ref{247c,,os.GRND_RANDOM} and
@ref{247d,,GRND_NONBLOCK}.

See also the Linux getrandom() manual page@footnote{http://man7.org/linux/man-pages/man2/getrandom.2.html}.

@ref{1629,,Availability}: Linux 3.17 and newer.


New in version 3.6.

@end deffn

@geindex urandom() (in module os)
@anchor{library/os os urandom}@anchor{17f}
@deffn {Function} os.urandom (size)

Return a bytestring of @emph{size} random bytes suitable for cryptographic use.

This function returns random bytes from an OS-specific randomness source.  The
returned data should be unpredictable enough for cryptographic applications,
though its exact quality depends on the OS implementation.

On Linux, if the @code{getrandom()} syscall is available, it is used in
blocking mode: block until the system urandom entropy pool is initialized
(128 bits of entropy are collected by the kernel). See the 
@geindex Python Enhancement Proposals; PEP 524
PEP 524@footnote{https://www.python.org/dev/peps/pep-0524} for
the rationale. On Linux, the @ref{806,,getrandom()} function can be used to get
random bytes in non-blocking mode (using the @ref{247d,,GRND_NONBLOCK} flag) or
to poll until the system urandom entropy pool is initialized.

On a Unix-like system, random bytes are read from the @code{/dev/urandom}
device. If the @code{/dev/urandom} device is not available or not readable, the
@ref{8a3,,NotImplementedError} exception is raised.

On Windows, it will use @code{BCryptGenRandom()}.


@subsubheading See also


The @ref{df,,secrets} module provides higher level functions. For an
easy-to-use interface to the random number generator provided by your
platform, please see @ref{1fa4,,random.SystemRandom}.



Changed in version 3.6.0: On Linux, @code{getrandom()} is now used in blocking mode to increase the
security.



Changed in version 3.5.2: On Linux, if the @code{getrandom()} syscall blocks (the urandom entropy pool
is not initialized yet), fall back on reading @code{/dev/urandom}.



Changed in version 3.5: On Linux 3.17 and newer, the @code{getrandom()} syscall is now used
when available.  On OpenBSD 5.6 and newer, the C @code{getentropy()}
function is now used. These functions avoid the usage of an internal file
descriptor.



Changed in version 3.11: On Windows, @code{BCryptGenRandom()} is used instead of @code{CryptGenRandom()}
which is deprecated.

@end deffn

@geindex GRND_NONBLOCK (in module os)
@anchor{library/os os GRND_NONBLOCK}@anchor{247d}
@deffn {Data} os.GRND_NONBLOCK

By  default, when reading from @code{/dev/random}, @ref{806,,getrandom()} blocks if
no random bytes are available, and when reading from @code{/dev/urandom}, it blocks
if the entropy pool has not yet been initialized.

If the @ref{247d,,GRND_NONBLOCK} flag is set, then @ref{806,,getrandom()} does not
block in these cases, but instead immediately raises @ref{bed,,BlockingIOError}.


New in version 3.6.

@end deffn

@geindex GRND_RANDOM (in module os)
@anchor{library/os os GRND_RANDOM}@anchor{247c}
@deffn {Data} os.GRND_RANDOM

If  this  bit  is  set,  then  random bytes are drawn from the
@code{/dev/random} pool instead of the @code{/dev/urandom} pool.


New in version 3.6.

@end deffn

@node io — Core tools for working with streams,time — Time access and conversions,os — Miscellaneous operating system interfaces,Generic Operating System Services
@anchor{library/io doc}@anchor{247e}@anchor{library/io io-core-tools-for-working-with-streams}@anchor{247f}@anchor{library/io module-io}@anchor{9d}
@subsection @code{io} — Core tools for working with streams


@geindex module; io

@strong{Source code:} Lib/io.py@footnote{https://github.com/python/cpython/tree/main/Lib/io.py}


__________________________________________________________________


@menu
* Overview: Overview<2>. 
* Text Encoding:: 
* High-level Module Interface:: 
* Class hierarchy:: 
* Performance: Performance<3>. 

@end menu

@node Overview<2>,Text Encoding,,io — Core tools for working with streams
@anchor{library/io io-overview}@anchor{1a55}@anchor{library/io overview}@anchor{2480}
@subsubsection Overview


@geindex file object; io module

The @ref{9d,,io} module provides Python’s main facilities for dealing with various
types of I/O.  There are three main types of I/O: @emph{text I/O}, @emph{binary I/O}
and @emph{raw I/O}.  These are generic categories, and various backing stores can
be used for each of them.  A concrete object belonging to any of these
categories is called a @ref{d68,,file object}.  Other common terms are @emph{stream}
and @emph{file-like object}.

Independent of its category, each concrete stream object will also have
various capabilities: it can be read-only, write-only, or read-write. It can
also allow arbitrary random access (seeking forwards or backwards to any
location), or only sequential access (for example in the case of a socket or
pipe).

All streams are careful about the type of data you give to them.  For example
giving a @ref{2ef,,str} object to the @code{write()} method of a binary stream
will raise a @ref{162,,TypeError}.  So will giving a @ref{2f0,,bytes} object to the
@code{write()} method of a text stream.


Changed in version 3.3: Operations that used to raise @ref{be8,,IOError} now raise @ref{2a1,,OSError}, since
@ref{be8,,IOError} is now an alias of @ref{2a1,,OSError}.


@menu
* Text I/O:: 
* Binary I/O:: 
* Raw I/O:: 

@end menu

@node Text I/O,Binary I/O,,Overview<2>
@anchor{library/io text-i-o}@anchor{2481}
@subsubsection Text I/O


Text I/O expects and produces @ref{2ef,,str} objects.  This means that whenever
the backing store is natively made of bytes (such as in the case of a file),
encoding and decoding of data is made transparently as well as optional
translation of platform-specific newline characters.

The easiest way to create a text stream is with @ref{1c8,,open()}, optionally
specifying an encoding:

@example
f = open("myfile.txt", "r", encoding="utf-8")
@end example

In-memory text streams are also available as @ref{aa1,,StringIO} objects:

@example
f = io.StringIO("some initial text data")
@end example

The text stream API is described in detail in the documentation of
@ref{e47,,TextIOBase}.

@node Binary I/O,Raw I/O,Text I/O,Overview<2>
@anchor{library/io binary-i-o}@anchor{2482}
@subsubsection Binary I/O


Binary I/O (also called @emph{buffered I/O}) expects
@ref{87e,,bytes-like objects} and produces @ref{2f0,,bytes}
objects.  No encoding, decoding, or newline translation is performed.  This
category of streams can be used for all kinds of non-text data, and also when
manual control over the handling of text data is desired.

The easiest way to create a binary stream is with @ref{1c8,,open()} with @code{'b'} in
the mode string:

@example
f = open("myfile.jpg", "rb")
@end example

In-memory binary streams are also available as @ref{a15,,BytesIO} objects:

@example
f = io.BytesIO(b"some initial binary data: \x00\x01")
@end example

The binary stream API is described in detail in the docs of
@ref{823,,BufferedIOBase}.

Other library modules may provide additional ways to create text or binary
streams.  See @ref{d44,,socket.socket.makefile()} for example.

@node Raw I/O,,Binary I/O,Overview<2>
@anchor{library/io raw-i-o}@anchor{2483}
@subsubsection Raw I/O


Raw I/O (also called @emph{unbuffered I/O}) is generally used as a low-level
building-block for binary and text streams; it is rarely useful to directly
manipulate a raw stream from user code.  Nevertheless, you can create a raw
stream by opening a file in binary mode with buffering disabled:

@example
f = open("myfile.jpg", "rb", buffering=0)
@end example

The raw stream API is described in detail in the docs of @ref{c62,,RawIOBase}.

@node Text Encoding,High-level Module Interface,Overview<2>,io — Core tools for working with streams
@anchor{library/io io-text-encoding}@anchor{231}@anchor{library/io text-encoding}@anchor{2484}
@subsubsection Text Encoding


The default encoding of @ref{889,,TextIOWrapper} and @ref{1c8,,open()} is
locale-specific (@ref{1b5,,locale.getpreferredencoding(False)}).

However, many developers forget to specify the encoding when opening text files
encoded in UTF-8 (e.g. JSON, TOML, Markdown, etc…) since most Unix
platforms use UTF-8 locale by default. This causes bugs because the locale
encoding is not UTF-8 for most Windows users. For example:

@example
# May not work on Windows when non-ASCII characters in the file.
with open("README.md") as f:
    long_description = f.read()
@end example

Additionally, while there is no concrete plan as of yet, Python may change
the default text file encoding to UTF-8 in the future.

Accordingly, it is highly recommended that you specify the encoding
explicitly when opening text files. If you want to use UTF-8, pass
@code{encoding="utf-8"}. To use the current locale encoding,
@code{encoding="locale"} is supported in Python 3.10.

When you need to run existing code on Windows that attempts to open
UTF-8 files using the default locale encoding, you can enable the UTF-8
mode. See @ref{16bc,,UTF-8 mode on Windows}.

@menu
* Opt-in EncodingWarning:: 

@end menu

@node Opt-in EncodingWarning,,,Text Encoding
@anchor{library/io io-encoding-warning}@anchor{162b}@anchor{library/io opt-in-encodingwarning}@anchor{2485}
@subsubsection Opt-in EncodingWarning



New in version 3.10: See 
@geindex Python Enhancement Proposals; PEP 597
PEP 597@footnote{https://www.python.org/dev/peps/pep-0597} for more details.


To find where the default locale encoding is used, you can enable
the @code{-X warn_default_encoding} command line option or set the
@geindex PYTHONWARNDEFAULTENCODING
@geindex environment variable; PYTHONWARNDEFAULTENCODING
@ref{230,,PYTHONWARNDEFAULTENCODING} environment variable, which will
emit an @ref{1617,,EncodingWarning} when the default encoding is used.

If you are providing an API that uses @ref{1c8,,open()} or
@ref{889,,TextIOWrapper} and passes @code{encoding=None} as a parameter, you
can use @ref{2486,,text_encoding()} so that callers of the API will emit an
@ref{1617,,EncodingWarning} if they don’t pass an @code{encoding}. However,
please consider using UTF-8 by default (i.e. @code{encoding="utf-8"}) for
new APIs.

@node High-level Module Interface,Class hierarchy,Text Encoding,io — Core tools for working with streams
@anchor{library/io high-level-module-interface}@anchor{2487}
@subsubsection High-level Module Interface


@geindex DEFAULT_BUFFER_SIZE (in module io)
@anchor{library/io io DEFAULT_BUFFER_SIZE}@anchor{1a56}
@deffn {Data} io.DEFAULT_BUFFER_SIZE

An int containing the default buffer size used by the module’s buffered I/O
classes.  @ref{1c8,,open()} uses the file’s blksize (as obtained by
@ref{516,,os.stat()}) if possible.
@end deffn

@geindex open() (in module io)
@anchor{library/io io open}@anchor{1c9}
@deffn {Function} io.open (file, mode='r', buffering=- 1, encoding=None, errors=None, newline=None, closefd=True, opener=None)

This is an alias for the builtin @ref{1c8,,open()} function.


This function raises an @ref{1130,,auditing event} @code{open} with
arguments @code{path}, @code{mode} and @code{flags}. The @code{mode} and @code{flags}
arguments may have been modified or inferred from the original call.

@end deffn

@geindex open_code() (in module io)
@anchor{library/io io open_code}@anchor{11f6}
@deffn {Function} io.open_code (path)

Opens the provided file with mode @code{'rb'}. This function should be used
when the intent is to treat the contents as executable code.

@code{path} should be a @ref{2ef,,str} and an absolute path.

The behavior of this function may be overridden by an earlier call to the
@ref{2488,,PyFile_SetOpenCodeHook()}. However, assuming that @code{path} is a
@ref{2ef,,str} and an absolute path, @code{open_code(path)} should always behave
the same as @code{open(path, 'rb')}. Overriding the behavior is intended for
additional validation or preprocessing of the file.


New in version 3.8.

@end deffn

@geindex text_encoding() (in module io)
@anchor{library/io io text_encoding}@anchor{2486}
@deffn {Function} io.text_encoding (encoding, stacklevel=2)

This is a helper function for callables that use @ref{1c8,,open()} or
@ref{889,,TextIOWrapper} and have an @code{encoding=None} parameter.

This function returns @emph{encoding} if it is not @code{None} and @code{"locale"} if
@emph{encoding} is @code{None}.

This function emits an @ref{1617,,EncodingWarning} if
@ref{22f,,sys.flags.warn_default_encoding} is true and @emph{encoding}
is None. @emph{stacklevel} specifies where the warning is emitted.
For example:

@example
def read_text(path, encoding=None):
    encoding = io.text_encoding(encoding)  # stacklevel=2
    with open(path, encoding) as f:
        return f.read()
@end example

In this example, an @ref{1617,,EncodingWarning} is emitted for the caller of
@code{read_text()}.

See @ref{231,,Text Encoding} for more information.


New in version 3.10.

@end deffn

@geindex BlockingIOError
@anchor{library/io io BlockingIOError}@anchor{2489}
@deffn {Exception} io.BlockingIOError

This is a compatibility alias for the builtin @ref{bed,,BlockingIOError}
exception.
@end deffn

@geindex UnsupportedOperation
@anchor{library/io io UnsupportedOperation}@anchor{1227}
@deffn {Exception} io.UnsupportedOperation

An exception inheriting @ref{2a1,,OSError} and @ref{17c,,ValueError} that is raised
when an unsupported operation is called on a stream.
@end deffn


@subsubheading See also



@table @asis

@item @ref{f7,,sys}

contains the standard IO streams: @ref{5ee,,sys.stdin}, @ref{5ef,,sys.stdout},
and @ref{42d,,sys.stderr}.
@end table


@node Class hierarchy,Performance<3>,High-level Module Interface,io — Core tools for working with streams
@anchor{library/io class-hierarchy}@anchor{248a}
@subsubsection Class hierarchy


The implementation of I/O streams is organized as a hierarchy of classes.  First
@ref{d5c,,abstract base classes} (ABCs), which are used to
specify the various categories of streams, then concrete classes providing the
standard stream implementations.

@quotation

@cartouche
@quotation Note 
The abstract base classes also provide default implementations of some
methods in order to help implementation of concrete stream classes.  For
example, @ref{823,,BufferedIOBase} provides unoptimized implementations of
@code{readinto()} and @ref{1b74,,readline()}.
@end quotation
@end cartouche
@end quotation

At the top of the I/O hierarchy is the abstract base class @ref{503,,IOBase}.  It
defines the basic interface to a stream.  Note, however, that there is no
separation between reading and writing to streams; implementations are allowed
to raise @ref{1227,,UnsupportedOperation} if they do not support a given operation.

The @ref{c62,,RawIOBase} ABC extends @ref{503,,IOBase}.  It deals with the reading
and writing of bytes to a stream.  @ref{ea6,,FileIO} subclasses @ref{c62,,RawIOBase}
to provide an interface to files in the machine’s file system.

The @ref{823,,BufferedIOBase} ABC extends @ref{503,,IOBase}.  It deals with
buffering on a raw binary stream (@ref{c62,,RawIOBase}).  Its subclasses,
@ref{1a5a,,BufferedWriter}, @ref{daf,,BufferedReader}, and @ref{248b,,BufferedRWPair}
buffer raw binary streams that are readable, writable, and both readable and writable,
respectively. @ref{1a5b,,BufferedRandom} provides a buffered interface to seekable streams.
Another @ref{823,,BufferedIOBase} subclass, @ref{a15,,BytesIO}, is a stream of
in-memory bytes.

The @ref{e47,,TextIOBase} ABC extends @ref{503,,IOBase}.  It deals with
streams whose bytes represent text, and handles encoding and decoding to and
from strings.  @ref{889,,TextIOWrapper}, which extends @ref{e47,,TextIOBase}, is a buffered text
interface to a buffered raw stream (@ref{823,,BufferedIOBase}).  Finally,
@ref{aa1,,StringIO} is an in-memory stream for text.

Argument names are not part of the specification, and only the arguments of
@ref{1c8,,open()} are intended to be used as keyword arguments.

The following table summarizes the ABCs provided by the @ref{9d,,io} module:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

ABC

@tab

Inherits

@tab

Stub Methods

@tab

Mixin Methods and Properties

@item

@ref{503,,IOBase}

@tab

@tab

@code{fileno}, @code{seek},
and @code{truncate}

@tab

@code{close}, @code{closed}, @code{__enter__},
@code{__exit__}, @code{flush}, @code{isatty}, @code{__iter__},
@code{__next__}, @code{readable}, @code{readline},
@code{readlines}, @code{seekable}, @code{tell},
@code{writable}, and @code{writelines}

@item

@ref{c62,,RawIOBase}

@tab

@ref{503,,IOBase}

@tab

@code{readinto} and
@code{write}

@tab

Inherited @ref{503,,IOBase} methods, @code{read},
and @code{readall}

@item

@ref{823,,BufferedIOBase}

@tab

@ref{503,,IOBase}

@tab

@code{detach}, @code{read},
@code{read1}, and @code{write}

@tab

Inherited @ref{503,,IOBase} methods, @code{readinto},
and @code{readinto1}

@item

@ref{e47,,TextIOBase}

@tab

@ref{503,,IOBase}

@tab

@code{detach}, @code{read},
@code{readline}, and
@code{write}

@tab

Inherited @ref{503,,IOBase} methods, @code{encoding},
@code{errors}, and @code{newlines}

@end multitable


@menu
* I/O Base Classes:: 
* Raw File I/O:: 
* Buffered Streams:: 
* Text I/O: Text I/O<2>. 

@end menu

@node I/O Base Classes,Raw File I/O,,Class hierarchy
@anchor{library/io i-o-base-classes}@anchor{248c}
@subsubsection I/O Base Classes


@geindex IOBase (class in io)
@anchor{library/io io IOBase}@anchor{503}
@deffn {Class} io.IOBase

The abstract base class for all I/O classes.

This class provides empty abstract implementations for many methods
that derived classes can override selectively; the default
implementations represent a file that cannot be read, written or
seeked.

Even though @ref{503,,IOBase} does not declare @code{read()}
or @code{write()} because their signatures will vary, implementations and
clients should consider those methods part of the interface.  Also,
implementations may raise a @ref{17c,,ValueError} (or @ref{1227,,UnsupportedOperation})
when operations they do not support are called.

The basic type used for binary data read from or written to a file is
@ref{2f0,,bytes}.  Other @ref{87e,,bytes-like objects} are
accepted as method arguments too.  Text I/O classes work with @ref{2ef,,str} data.

Note that calling any method (even inquiries) on a closed stream is
undefined.  Implementations may raise @ref{17c,,ValueError} in this case.

@ref{503,,IOBase} (and its subclasses) supports the iterator protocol, meaning
that an @ref{503,,IOBase} object can be iterated over yielding the lines in a
stream.  Lines are defined slightly differently depending on whether the
stream is a binary stream (yielding bytes), or a text stream (yielding
character strings).  See @ref{1b74,,readline()} below.

@ref{503,,IOBase} is also a context manager and therefore supports the
@ref{168,,with} statement.  In this example, @emph{file} is closed after the
@code{with} statement’s suite is finished—even if an exception occurs:

@example
with open('spam.txt', 'w') as file:
    file.write('Spam and eggs!')
@end example

@ref{503,,IOBase} provides these data attributes and methods:

@geindex close() (io.IOBase method)
@anchor{library/io io IOBase close}@anchor{12d8}
@deffn {Method} close ()

Flush and close this stream. This method has no effect if the file is
already closed. Once the file is closed, any operation on the file
(e.g. reading or writing) will raise a @ref{17c,,ValueError}.

As a convenience, it is allowed to call this method more than once;
only the first call, however, will have an effect.
@end deffn

@geindex closed (io.IOBase attribute)
@anchor{library/io io IOBase closed}@anchor{12d9}
@deffn {Attribute} closed

@code{True} if the stream is closed.
@end deffn

@geindex fileno() (io.IOBase method)
@anchor{library/io io IOBase fileno}@anchor{23a2}
@deffn {Method} fileno ()

Return the underlying file descriptor (an integer) of the stream if it
exists.  An @ref{2a1,,OSError} is raised if the IO object does not use a file
descriptor.
@end deffn

@geindex flush() (io.IOBase method)
@anchor{library/io io IOBase flush}@anchor{248d}
@deffn {Method} flush ()

Flush the write buffers of the stream if applicable.  This does nothing
for read-only and non-blocking streams.
@end deffn

@geindex isatty() (io.IOBase method)
@anchor{library/io io IOBase isatty}@anchor{1a57}
@deffn {Method} isatty ()

Return @code{True} if the stream is interactive (i.e., connected to
a terminal/tty device).
@end deffn

@geindex readable() (io.IOBase method)
@anchor{library/io io IOBase readable}@anchor{248e}
@deffn {Method} readable ()

Return @code{True} if the stream can be read from.  If @code{False}, @code{read()}
will raise @ref{2a1,,OSError}.
@end deffn

@geindex readline() (io.IOBase method)
@anchor{library/io io IOBase readline}@anchor{1b74}
@deffn {Method} readline (size=- 1)

Read and return one line from the stream.  If @emph{size} is specified, at
most @emph{size} bytes will be read.

The line terminator is always @code{b'\n'} for binary files; for text files,
the @emph{newline} argument to @ref{1c8,,open()} can be used to select the line
terminator(s) recognized.
@end deffn

@geindex readlines() (io.IOBase method)
@anchor{library/io io IOBase readlines}@anchor{1c17}
@deffn {Method} readlines (hint=- 1)

Read and return a list of lines from the stream.  @emph{hint} can be specified
to control the number of lines read: no more lines will be read if the
total size (in bytes/characters) of all lines so far exceeds @emph{hint}.

@emph{hint} values of @code{0} or less, as well as @code{None}, are treated as no
hint.

Note that it’s already possible to iterate on file objects using @code{for
line in file: ...} without calling @code{file.readlines()}.
@end deffn

@geindex seek() (io.IOBase method)
@anchor{library/io io IOBase seek}@anchor{2231}
@deffn {Method} seek (offset, whence=SEEK_SET)

Change the stream position to the given byte @emph{offset}.  @emph{offset} is
interpreted relative to the position indicated by @emph{whence}.  The default
value for @emph{whence} is @code{SEEK_SET}.  Values for @emph{whence} are:


@itemize *

@item 
@code{SEEK_SET} or @code{0} – start of the stream (the default);
@emph{offset} should be zero or positive

@item 
@code{SEEK_CUR} or @code{1} – current stream position; @emph{offset} may
be negative

@item 
@code{SEEK_END} or @code{2} – end of the stream; @emph{offset} is usually
negative
@end itemize

Return the new absolute position.


New in version 3.1: The @code{SEEK_*} constants.



New in version 3.3: Some operating systems could support additional values, like
@code{os.SEEK_HOLE} or @code{os.SEEK_DATA}. The valid values
for a file could depend on it being open in text or binary mode.

@end deffn

@geindex seekable() (io.IOBase method)
@anchor{library/io io IOBase seekable}@anchor{248f}
@deffn {Method} seekable ()

Return @code{True} if the stream supports random access.  If @code{False},
@ref{2231,,seek()}, @ref{2232,,tell()} and @ref{ea7,,truncate()} will raise @ref{2a1,,OSError}.
@end deffn

@geindex tell() (io.IOBase method)
@anchor{library/io io IOBase tell}@anchor{2232}
@deffn {Method} tell ()

Return the current stream position.
@end deffn

@geindex truncate() (io.IOBase method)
@anchor{library/io io IOBase truncate}@anchor{ea7}
@deffn {Method} truncate (size=None)

Resize the stream to the given @emph{size} in bytes (or the current position
if @emph{size} is not specified).  The current stream position isn’t changed.
This resizing can extend or reduce the current file size.  In case of
extension, the contents of the new file area depend on the platform
(on most systems, additional bytes are zero-filled).  The new file size
is returned.


Changed in version 3.5: Windows will now zero-fill files when extending.

@end deffn

@geindex writable() (io.IOBase method)
@anchor{library/io io IOBase writable}@anchor{2490}
@deffn {Method} writable ()

Return @code{True} if the stream supports writing.  If @code{False},
@code{write()} and @ref{ea7,,truncate()} will raise @ref{2a1,,OSError}.
@end deffn

@geindex writelines() (io.IOBase method)
@anchor{library/io io IOBase writelines}@anchor{1c18}
@deffn {Method} writelines (lines)

Write a list of lines to the stream.  Line separators are not added, so it
is usual for each of the lines provided to have a line separator at the
end.
@end deffn

@geindex __del__() (io.IOBase method)
@anchor{library/io io IOBase __del__}@anchor{2491}
@deffn {Method} __del__ ()

Prepare for object destruction. @ref{503,,IOBase} provides a default
implementation of this method that calls the instance’s
@ref{12d8,,close()} method.
@end deffn
@end deffn

@geindex RawIOBase (class in io)
@anchor{library/io io RawIOBase}@anchor{c62}
@deffn {Class} io.RawIOBase

Base class for raw binary streams.  It inherits @ref{503,,IOBase}.

Raw binary streams typically provide low-level access to an underlying OS
device or API, and do not try to encapsulate it in high-level primitives
(this functionality is done at a higher-level in buffered binary streams and text streams, described later
in this page).

@ref{c62,,RawIOBase} provides these methods in addition to those from
@ref{503,,IOBase}:

@geindex read() (io.RawIOBase method)
@anchor{library/io io RawIOBase read}@anchor{983}
@deffn {Method} read (size=- 1)

Read up to @emph{size} bytes from the object and return them.  As a convenience,
if @emph{size} is unspecified or -1, all bytes until EOF are returned.
Otherwise, only one system call is ever made.  Fewer than @emph{size} bytes may
be returned if the operating system call returns fewer than @emph{size} bytes.

If 0 bytes are returned, and @emph{size} was not 0, this indicates end of file.
If the object is in non-blocking mode and no bytes are available,
@code{None} is returned.

The default implementation defers to @ref{2492,,readall()} and
@ref{984,,readinto()}.
@end deffn

@geindex readall() (io.RawIOBase method)
@anchor{library/io io RawIOBase readall}@anchor{2492}
@deffn {Method} readall ()

Read and return all the bytes from the stream until EOF, using multiple
calls to the stream if necessary.
@end deffn

@geindex readinto() (io.RawIOBase method)
@anchor{library/io io RawIOBase readinto}@anchor{984}
@deffn {Method} readinto (b)

Read bytes into a pre-allocated, writable
@ref{87e,,bytes-like object} @emph{b}, and return the
number of bytes read.  For example, @emph{b} might be a @ref{2f1,,bytearray}.
If the object is in non-blocking mode and no bytes
are available, @code{None} is returned.
@end deffn

@geindex write() (io.RawIOBase method)
@anchor{library/io io RawIOBase write}@anchor{2493}
@deffn {Method} write (b)

Write the given @ref{87e,,bytes-like object}, @emph{b}, to the
underlying raw stream, and return the number of
bytes written.  This can be less than the length of @emph{b} in
bytes, depending on specifics of the underlying raw
stream, and especially if it is in non-blocking mode.  @code{None} is
returned if the raw stream is set not to block and no single byte could
be readily written to it.  The caller may release or mutate @emph{b} after
this method returns, so the implementation should only access @emph{b}
during the method call.
@end deffn
@end deffn

@geindex BufferedIOBase (class in io)
@anchor{library/io io BufferedIOBase}@anchor{823}
@deffn {Class} io.BufferedIOBase

Base class for binary streams that support some kind of buffering.
It inherits @ref{503,,IOBase}.

The main difference with @ref{c62,,RawIOBase} is that methods @ref{21ef,,read()},
@ref{2494,,readinto()} and @ref{824,,write()} will try (respectively) to read as much
input as requested or to consume all given output, at the expense of
making perhaps more than one system call.

In addition, those methods can raise @ref{bed,,BlockingIOError} if the
underlying raw stream is in non-blocking mode and cannot take or give
enough data; unlike their @ref{c62,,RawIOBase} counterparts, they will
never return @code{None}.

Besides, the @ref{21ef,,read()} method does not have a default
implementation that defers to @ref{2494,,readinto()}.

A typical @ref{823,,BufferedIOBase} implementation should not inherit from a
@ref{c62,,RawIOBase} implementation, but wrap one, like
@ref{1a5a,,BufferedWriter} and @ref{daf,,BufferedReader} do.

@ref{823,,BufferedIOBase} provides or overrides these data attributes and
methods in addition to those from @ref{503,,IOBase}:

@geindex raw (io.BufferedIOBase attribute)
@anchor{library/io io BufferedIOBase raw}@anchor{2495}
@deffn {Attribute} raw

The underlying raw stream (a @ref{c62,,RawIOBase} instance) that
@ref{823,,BufferedIOBase} deals with.  This is not part of the
@ref{823,,BufferedIOBase} API and may not exist on some implementations.
@end deffn

@geindex detach() (io.BufferedIOBase method)
@anchor{library/io io BufferedIOBase detach}@anchor{2496}
@deffn {Method} detach ()

Separate the underlying raw stream from the buffer and return it.

After the raw stream has been detached, the buffer is in an unusable
state.

Some buffers, like @ref{a15,,BytesIO}, do not have the concept of a single
raw stream to return from this method.  They raise
@ref{1227,,UnsupportedOperation}.


New in version 3.1.

@end deffn

@geindex read() (io.BufferedIOBase method)
@anchor{library/io io BufferedIOBase read}@anchor{21ef}
@deffn {Method} read (size=- 1)

Read and return up to @emph{size} bytes.  If the argument is omitted, @code{None},
or negative, data is read and returned until EOF is reached.  An empty
@ref{2f0,,bytes} object is returned if the stream is already at EOF.

If the argument is positive, and the underlying raw stream is not
interactive, multiple raw reads may be issued to satisfy the byte count
(unless EOF is reached first).  But for interactive raw streams, at most
one raw read will be issued, and a short result does not imply that EOF is
imminent.

A @ref{bed,,BlockingIOError} is raised if the underlying raw stream is in
non blocking-mode, and has no data available at the moment.
@end deffn

@geindex read1() (io.BufferedIOBase method)
@anchor{library/io io BufferedIOBase read1}@anchor{21ee}
@deffn {Method} read1 ([size])

Read and return up to @emph{size} bytes, with at most one call to the
underlying raw stream’s @ref{983,,read()} (or
@ref{984,,readinto()}) method.  This can be useful if you are
implementing your own buffering on top of a @ref{823,,BufferedIOBase}
object.

If @emph{size} is @code{-1} (the default), an arbitrary number of bytes are
returned (more than zero unless EOF is reached).
@end deffn

@geindex readinto() (io.BufferedIOBase method)
@anchor{library/io io BufferedIOBase readinto}@anchor{2494}
@deffn {Method} readinto (b)

Read bytes into a pre-allocated, writable
@ref{87e,,bytes-like object} @emph{b} and return the number of bytes read.
For example, @emph{b} might be a @ref{2f1,,bytearray}.

Like @ref{21ef,,read()}, multiple reads may be issued to the underlying raw
stream, unless the latter is interactive.

A @ref{bed,,BlockingIOError} is raised if the underlying raw stream is in non
blocking-mode, and has no data available at the moment.
@end deffn

@geindex readinto1() (io.BufferedIOBase method)
@anchor{library/io io BufferedIOBase readinto1}@anchor{982}
@deffn {Method} readinto1 (b)

Read bytes into a pre-allocated, writable
@ref{87e,,bytes-like object} @emph{b}, using at most one call to
the underlying raw stream’s @ref{983,,read()} (or
@ref{984,,readinto()}) method. Return the number of bytes read.

A @ref{bed,,BlockingIOError} is raised if the underlying raw stream is in non
blocking-mode, and has no data available at the moment.


New in version 3.5.

@end deffn

@geindex write() (io.BufferedIOBase method)
@anchor{library/io io BufferedIOBase write}@anchor{824}
@deffn {Method} write (b)

Write the given @ref{87e,,bytes-like object}, @emph{b}, and return the number
of bytes written (always equal to the length of @emph{b} in bytes, since if
the write fails an @ref{2a1,,OSError} will be raised).  Depending on the
actual implementation, these bytes may be readily written to the
underlying stream, or held in a buffer for performance and latency
reasons.

When in non-blocking mode, a @ref{bed,,BlockingIOError} is raised if the
data needed to be written to the raw stream but it couldn’t accept
all the data without blocking.

The caller may release or mutate @emph{b} after this method returns,
so the implementation should only access @emph{b} during the method call.
@end deffn
@end deffn

@node Raw File I/O,Buffered Streams,I/O Base Classes,Class hierarchy
@anchor{library/io raw-file-i-o}@anchor{2497}
@subsubsection Raw File I/O


@geindex FileIO (class in io)
@anchor{library/io io FileIO}@anchor{ea6}
@deffn {Class} io.FileIO (name, mode='r', closefd=True, opener=None)

A raw binary stream representing an OS-level file containing bytes data.  It
inherits @ref{c62,,RawIOBase}.

The @emph{name} can be one of two things:


@itemize *

@item 
a character string or @ref{2f0,,bytes} object representing the path to the
file which will be opened. In this case closefd must be @code{True} (the default)
otherwise an error will be raised.

@item 
an integer representing the number of an existing OS-level file descriptor
to which the resulting @ref{ea6,,FileIO} object will give access. When the
FileIO object is closed this fd will be closed as well, unless @emph{closefd}
is set to @code{False}.
@end itemize

The @emph{mode} can be @code{'r'}, @code{'w'}, @code{'x'} or @code{'a'} for reading
(default), writing, exclusive creation or appending. The file will be
created if it doesn’t exist when opened for writing or appending; it will be
truncated when opened for writing. @ref{bb1,,FileExistsError} will be raised if
it already exists when opened for creating. Opening a file for creating
implies writing, so this mode behaves in a similar way to @code{'w'}. Add a
@code{'+'} to the mode to allow simultaneous reading and writing.

The @code{read()} (when called with a positive argument), @code{readinto()}
and @code{write()} methods on this class will only make one system call.

A custom opener can be used by passing a callable as @emph{opener}. The underlying
file descriptor for the file object is then obtained by calling @emph{opener} with
(@emph{name}, @emph{flags}). @emph{opener} must return an open file descriptor (passing
@ref{8ee,,os.open} as @emph{opener} results in functionality similar to passing
@code{None}).

The newly created file is @ref{a6e,,non-inheritable}.

See the @ref{1c8,,open()} built-in function for examples on using the @emph{opener}
parameter.


Changed in version 3.3: The @emph{opener} parameter was added.
The @code{'x'} mode was added.



Changed in version 3.4: The file is now non-inheritable.


@ref{ea6,,FileIO} provides these data attributes in addition to those from
@ref{c62,,RawIOBase} and @ref{503,,IOBase}:

@geindex mode (io.FileIO attribute)
@anchor{library/io io FileIO mode}@anchor{2498}
@deffn {Attribute} mode

The mode as given in the constructor.
@end deffn

@geindex name (io.FileIO attribute)
@anchor{library/io io FileIO name}@anchor{2284}
@deffn {Attribute} name

The file name.  This is the file descriptor of the file when no name is
given in the constructor.
@end deffn
@end deffn

@node Buffered Streams,Text I/O<2>,Raw File I/O,Class hierarchy
@anchor{library/io buffered-streams}@anchor{2499}
@subsubsection Buffered Streams


Buffered I/O streams provide a higher-level interface to an I/O device
than raw I/O does.

@geindex BytesIO (class in io)
@anchor{library/io io BytesIO}@anchor{a15}
@deffn {Class} io.BytesIO ([initial_bytes])

A binary stream using an in-memory bytes buffer.  It inherits
@ref{823,,BufferedIOBase}.  The buffer is discarded when the
@ref{12d8,,close()} method is called.

The optional argument @emph{initial_bytes} is a @ref{87e,,bytes-like object} that
contains initial data.

@ref{a15,,BytesIO} provides or overrides these methods in addition to those
from @ref{823,,BufferedIOBase} and @ref{503,,IOBase}:

@geindex getbuffer() (io.BytesIO method)
@anchor{library/io io BytesIO getbuffer}@anchor{d93}
@deffn {Method} getbuffer ()

Return a readable and writable view over the contents of the buffer
without copying them.  Also, mutating the view will transparently
update the contents of the buffer:

@example
>>> b = io.BytesIO(b"abcdef")
>>> view = b.getbuffer()
>>> view[2:4] = b"56"
>>> b.getvalue()
b'ab56ef'
@end example

@cartouche
@quotation Note 
As long as the view exists, the @ref{a15,,BytesIO} object cannot be
resized or closed.
@end quotation
@end cartouche


New in version 3.2.

@end deffn

@geindex getvalue() (io.BytesIO method)
@anchor{library/io io BytesIO getvalue}@anchor{21ec}
@deffn {Method} getvalue ()

Return @ref{2f0,,bytes} containing the entire contents of the buffer.
@end deffn

@geindex read1() (io.BytesIO method)
@anchor{library/io io BytesIO read1}@anchor{249a}
@deffn {Method} read1 ([size])

In @ref{a15,,BytesIO}, this is the same as @ref{21ef,,read()}.


Changed in version 3.7: The @emph{size} argument is now optional.

@end deffn

@geindex readinto1() (io.BytesIO method)
@anchor{library/io io BytesIO readinto1}@anchor{249b}
@deffn {Method} readinto1 (b)

In @ref{a15,,BytesIO}, this is the same as @ref{2494,,readinto()}.


New in version 3.5.

@end deffn
@end deffn

@geindex BufferedReader (class in io)
@anchor{library/io io BufferedReader}@anchor{daf}
@deffn {Class} io.BufferedReader (raw, buffer_size=DEFAULT_BUFFER_SIZE)

A buffered binary stream providing higher-level access to a readable, non
seekable @ref{c62,,RawIOBase} raw binary stream.  It inherits
@ref{823,,BufferedIOBase}.

When reading data from this object, a larger amount of data may be
requested from the underlying raw stream, and kept in an internal buffer.
The buffered data can then be returned directly on subsequent reads.

The constructor creates a @ref{daf,,BufferedReader} for the given readable
@emph{raw} stream and @emph{buffer_size}.  If @emph{buffer_size} is omitted,
@ref{1a56,,DEFAULT_BUFFER_SIZE} is used.

@ref{daf,,BufferedReader} provides or overrides these methods in addition to
those from @ref{823,,BufferedIOBase} and @ref{503,,IOBase}:

@geindex peek() (io.BufferedReader method)
@anchor{library/io io BufferedReader peek}@anchor{249c}
@deffn {Method} peek ([size])

Return bytes from the stream without advancing the position.  At most one
single read on the raw stream is done to satisfy the call. The number of
bytes returned may be less or more than requested.
@end deffn

@geindex read() (io.BufferedReader method)
@anchor{library/io io BufferedReader read}@anchor{249d}
@deffn {Method} read ([size])

Read and return @emph{size} bytes, or if @emph{size} is not given or negative, until
EOF or if the read call would block in non-blocking mode.
@end deffn

@geindex read1() (io.BufferedReader method)
@anchor{library/io io BufferedReader read1}@anchor{249e}
@deffn {Method} read1 ([size])

Read and return up to @emph{size} bytes with only one call on the raw stream.
If at least one byte is buffered, only buffered bytes are returned.
Otherwise, one raw stream read call is made.


Changed in version 3.7: The @emph{size} argument is now optional.

@end deffn
@end deffn

@geindex BufferedWriter (class in io)
@anchor{library/io io BufferedWriter}@anchor{1a5a}
@deffn {Class} io.BufferedWriter (raw, buffer_size=DEFAULT_BUFFER_SIZE)

A buffered binary stream providing higher-level access to a writeable, non
seekable @ref{c62,,RawIOBase} raw binary stream.  It inherits
@ref{823,,BufferedIOBase}.

When writing to this object, data is normally placed into an internal
buffer.  The buffer will be written out to the underlying @ref{c62,,RawIOBase}
object under various conditions, including:


@itemize *

@item 
when the buffer gets too small for all pending data;

@item 
when @ref{249f,,flush()} is called;

@item 
when a @code{seek()} is requested (for @ref{1a5b,,BufferedRandom} objects);

@item 
when the @ref{1a5a,,BufferedWriter} object is closed or destroyed.
@end itemize

The constructor creates a @ref{1a5a,,BufferedWriter} for the given writeable
@emph{raw} stream.  If the @emph{buffer_size} is not given, it defaults to
@ref{1a56,,DEFAULT_BUFFER_SIZE}.

@ref{1a5a,,BufferedWriter} provides or overrides these methods in addition to
those from @ref{823,,BufferedIOBase} and @ref{503,,IOBase}:

@geindex flush() (io.BufferedWriter method)
@anchor{library/io io BufferedWriter flush}@anchor{249f}
@deffn {Method} flush ()

Force bytes held in the buffer into the raw stream.  A
@ref{bed,,BlockingIOError} should be raised if the raw stream blocks.
@end deffn

@geindex write() (io.BufferedWriter method)
@anchor{library/io io BufferedWriter write}@anchor{24a0}
@deffn {Method} write (b)

Write the @ref{87e,,bytes-like object}, @emph{b}, and return the
number of bytes written.  When in non-blocking mode, a
@ref{bed,,BlockingIOError} is raised if the buffer needs to be written out but
the raw stream blocks.
@end deffn
@end deffn

@geindex BufferedRandom (class in io)
@anchor{library/io io BufferedRandom}@anchor{1a5b}
@deffn {Class} io.BufferedRandom (raw, buffer_size=DEFAULT_BUFFER_SIZE)

A buffered binary stream providing higher-level access to a seekable
@ref{c62,,RawIOBase} raw binary stream.  It inherits @ref{daf,,BufferedReader}
and @ref{1a5a,,BufferedWriter}.

The constructor creates a reader and writer for a seekable raw stream, given
in the first argument.  If the @emph{buffer_size} is omitted it defaults to
@ref{1a56,,DEFAULT_BUFFER_SIZE}.

@ref{1a5b,,BufferedRandom} is capable of anything @ref{daf,,BufferedReader} or
@ref{1a5a,,BufferedWriter} can do.  In addition, @code{seek()} and @code{tell()}
are guaranteed to be implemented.
@end deffn

@geindex BufferedRWPair (class in io)
@anchor{library/io io BufferedRWPair}@anchor{248b}
@deffn {Class} io.BufferedRWPair (reader, writer, buffer_size=DEFAULT_BUFFER_SIZE)

A buffered binary stream providing higher-level access to two non seekable
@ref{c62,,RawIOBase} raw binary streams—one readable, the other writeable.
It inherits @ref{823,,BufferedIOBase}.

@emph{reader} and @emph{writer} are @ref{c62,,RawIOBase} objects that are readable and
writeable respectively.  If the @emph{buffer_size} is omitted it defaults to
@ref{1a56,,DEFAULT_BUFFER_SIZE}.

@ref{248b,,BufferedRWPair} implements all of @ref{823,,BufferedIOBase}'s methods
except for @ref{2496,,detach()}, which raises
@ref{1227,,UnsupportedOperation}.

@cartouche
@quotation Warning 
@ref{248b,,BufferedRWPair} does not attempt to synchronize accesses to
its underlying raw streams.  You should not pass it the same object
as reader and writer; use @ref{1a5b,,BufferedRandom} instead.
@end quotation
@end cartouche
@end deffn

@node Text I/O<2>,,Buffered Streams,Class hierarchy
@anchor{library/io id1}@anchor{24a1}
@subsubsection Text I/O


@geindex TextIOBase (class in io)
@anchor{library/io io TextIOBase}@anchor{e47}
@deffn {Class} io.TextIOBase

Base class for text streams.  This class provides a character and line based
interface to stream I/O.  It inherits @ref{503,,IOBase}.

@ref{e47,,TextIOBase} provides or overrides these data attributes and
methods in addition to those from @ref{503,,IOBase}:

@geindex encoding (io.TextIOBase attribute)
@anchor{library/io io TextIOBase encoding}@anchor{24a2}
@deffn {Attribute} encoding

The name of the encoding used to decode the stream’s bytes into
strings, and to encode strings into bytes.
@end deffn

@geindex errors (io.TextIOBase attribute)
@anchor{library/io io TextIOBase errors}@anchor{24a3}
@deffn {Attribute} errors

The error setting of the decoder or encoder.
@end deffn

@geindex newlines (io.TextIOBase attribute)
@anchor{library/io io TextIOBase newlines}@anchor{24a4}
@deffn {Attribute} newlines

A string, a tuple of strings, or @code{None}, indicating the newlines
translated so far.  Depending on the implementation and the initial
constructor flags, this may not be available.
@end deffn

@geindex buffer (io.TextIOBase attribute)
@anchor{library/io io TextIOBase buffer}@anchor{e48}
@deffn {Attribute} buffer

The underlying binary buffer (a @ref{823,,BufferedIOBase} instance) that
@ref{e47,,TextIOBase} deals with.  This is not part of the
@ref{e47,,TextIOBase} API and may not exist in some implementations.
@end deffn

@geindex detach() (io.TextIOBase method)
@anchor{library/io io TextIOBase detach}@anchor{24a5}
@deffn {Method} detach ()

Separate the underlying binary buffer from the @ref{e47,,TextIOBase} and
return it.

After the underlying buffer has been detached, the @ref{e47,,TextIOBase} is
in an unusable state.

Some @ref{e47,,TextIOBase} implementations, like @ref{aa1,,StringIO}, may not
have the concept of an underlying buffer and calling this method will
raise @ref{1227,,UnsupportedOperation}.


New in version 3.1.

@end deffn

@geindex read() (io.TextIOBase method)
@anchor{library/io io TextIOBase read}@anchor{24a6}
@deffn {Method} read (size=- 1)

Read and return at most @emph{size} characters from the stream as a single
@ref{2ef,,str}.  If @emph{size} is negative or @code{None}, reads until EOF.
@end deffn

@geindex readline() (io.TextIOBase method)
@anchor{library/io io TextIOBase readline}@anchor{209b}
@deffn {Method} readline (size=- 1)

Read until newline or EOF and return a single @code{str}.  If the stream is
already at EOF, an empty string is returned.

If @emph{size} is specified, at most @emph{size} characters will be read.
@end deffn

@geindex seek() (io.TextIOBase method)
@anchor{library/io io TextIOBase seek}@anchor{24a7}
@deffn {Method} seek (offset, whence=SEEK_SET)

Change the stream position to the given @emph{offset}.  Behaviour depends on
the @emph{whence} parameter.  The default value for @emph{whence} is
@code{SEEK_SET}.


@itemize *

@item 
@code{SEEK_SET} or @code{0}: seek from the start of the stream
(the default); @emph{offset} must either be a number returned by
@ref{24a8,,TextIOBase.tell()}, or zero.  Any other @emph{offset} value
produces undefined behaviour.

@item 
@code{SEEK_CUR} or @code{1}: “seek” to the current position;
@emph{offset} must be zero, which is a no-operation (all other values
are unsupported).

@item 
@code{SEEK_END} or @code{2}: seek to the end of the stream;
@emph{offset} must be zero (all other values are unsupported).
@end itemize

Return the new absolute position as an opaque number.


New in version 3.1: The @code{SEEK_*} constants.

@end deffn

@geindex tell() (io.TextIOBase method)
@anchor{library/io io TextIOBase tell}@anchor{24a8}
@deffn {Method} tell ()

Return the current stream position as an opaque number.  The number
does not usually represent a number of bytes in the underlying
binary storage.
@end deffn

@geindex write() (io.TextIOBase method)
@anchor{library/io io TextIOBase write}@anchor{24a9}
@deffn {Method} write (s)

Write the string @emph{s} to the stream and return the number of characters
written.
@end deffn
@end deffn

@geindex TextIOWrapper (class in io)
@anchor{library/io io TextIOWrapper}@anchor{889}
@deffn {Class} io.TextIOWrapper (buffer, encoding=None, errors=None, newline=None, line_buffering=False, write_through=False)

A buffered text stream providing higher-level access to a
@ref{823,,BufferedIOBase} buffered binary stream.  It inherits
@ref{e47,,TextIOBase}.

@emph{encoding} gives the name of the encoding that the stream will be decoded or
encoded with.  It defaults to
@ref{1b5,,locale.getpreferredencoding(False)}.
@code{encoding="locale"} can be used to specify the current locale’s encoding
explicitly. See @ref{231,,Text Encoding} for more information.

@emph{errors} is an optional string that specifies how encoding and decoding
errors are to be handled.  Pass @code{'strict'} to raise a @ref{17c,,ValueError}
exception if there is an encoding error (the default of @code{None} has the same
effect), or pass @code{'ignore'} to ignore errors.  (Note that ignoring encoding
errors can lead to data loss.)  @code{'replace'} causes a replacement marker
(such as @code{'?'}) to be inserted where there is malformed data.
@code{'backslashreplace'} causes malformed data to be replaced by a
backslashed escape sequence.  When writing, @code{'xmlcharrefreplace'}
(replace with the appropriate XML character reference)  or @code{'namereplace'}
(replace with @code{\N@{...@}} escape sequences) can be used.  Any other error
handling name that has been registered with
@ref{fed,,codecs.register_error()} is also valid.

@geindex universal newlines; io.TextIOWrapper class

@emph{newline} controls how line endings are handled.  It can be @code{None},
@code{'@w{'}}, @code{'\n'}, @code{'\r'}, and @code{'\r\n'}.  It works as follows:


@itemize *

@item 
When reading input from the stream, if @emph{newline} is @code{None},
@ref{88a,,universal newlines} mode is enabled.  Lines in the input can end in
@code{'\n'}, @code{'\r'}, or @code{'\r\n'}, and these are translated into @code{'\n'}
before being returned to the caller.  If @emph{newline} is @code{'@w{'}}, universal
newlines mode is enabled, but line endings are returned to the caller
untranslated.  If @emph{newline} has any of the other legal values, input lines
are only terminated by the given string, and the line ending is returned to
the caller untranslated.

@item 
When writing output to the stream, if @emph{newline} is @code{None}, any @code{'\n'}
characters written are translated to the system default line separator,
@ref{1a59,,os.linesep}.  If @emph{newline} is @code{'@w{'}} or @code{'\n'}, no translation
takes place.  If @emph{newline} is any of the other legal values, any @code{'\n'}
characters written are translated to the given string.
@end itemize

If @emph{line_buffering} is @code{True}, @code{flush()} is implied when a call to
write contains a newline character or a carriage return.

If @emph{write_through} is @code{True}, calls to @code{write()} are guaranteed
not to be buffered: any data written on the @ref{889,,TextIOWrapper}
object is immediately handled to its underlying binary @emph{buffer}.


Changed in version 3.3: The @emph{write_through} argument has been added.



Changed in version 3.3: The default @emph{encoding} is now @code{locale.getpreferredencoding(False)}
instead of @code{locale.getpreferredencoding()}. Don’t change temporary the
locale encoding using @ref{1b4,,locale.setlocale()}, use the current locale
encoding instead of the user preferred encoding.



Changed in version 3.10: The @emph{encoding} argument now supports the @code{"locale"} dummy encoding name.


@ref{889,,TextIOWrapper} provides these data attributes and methods in
addition to those from @ref{e47,,TextIOBase} and @ref{503,,IOBase}:

@geindex line_buffering (io.TextIOWrapper attribute)
@anchor{library/io io TextIOWrapper line_buffering}@anchor{24aa}
@deffn {Attribute} line_buffering

Whether line buffering is enabled.
@end deffn

@geindex write_through (io.TextIOWrapper attribute)
@anchor{library/io io TextIOWrapper write_through}@anchor{24ab}
@deffn {Attribute} write_through

Whether writes are passed immediately to the underlying binary
buffer.


New in version 3.7.

@end deffn

@geindex reconfigure() (io.TextIOWrapper method)
@anchor{library/io io TextIOWrapper reconfigure}@anchor{66d}
@deffn {Method} reconfigure (*[, encoding][, errors][, newline][,                      line_buffering][, write_through])

Reconfigure this text stream using new settings for @emph{encoding},
@emph{errors}, @emph{newline}, @emph{line_buffering} and @emph{write_through}.

Parameters not specified keep current settings, except
@code{errors='strict'} is used when @emph{encoding} is specified but
@emph{errors} is not specified.

It is not possible to change the encoding or newline if some data
has already been read from the stream. On the other hand, changing
encoding after write is possible.

This method does an implicit stream flush before setting the
new parameters.


New in version 3.7.

@end deffn
@end deffn

@geindex StringIO (class in io)
@anchor{library/io io StringIO}@anchor{aa1}
@deffn {Class} io.StringIO (initial_value='@w{'}, newline='\n')

A text stream using an in-memory text buffer.  It inherits
@ref{e47,,TextIOBase}.

The text buffer is discarded when the @ref{12d8,,close()} method is
called.

The initial value of the buffer can be set by providing @emph{initial_value}.
If newline translation is enabled, newlines will be encoded as if by
@ref{24a9,,write()}.  The stream is positioned at the start of
the buffer.

The @emph{newline} argument works like that of @ref{889,,TextIOWrapper},
except that when writing output to the stream, if @emph{newline} is @code{None},
newlines are written as @code{\n} on all platforms.

@ref{aa1,,StringIO} provides this method in addition to those from
@ref{e47,,TextIOBase} and @ref{503,,IOBase}:

@geindex getvalue() (io.StringIO method)
@anchor{library/io io StringIO getvalue}@anchor{24ac}
@deffn {Method} getvalue ()

Return a @code{str} containing the entire contents of the buffer.
Newlines are decoded as if by @ref{24a6,,read()}, although
the stream position is not changed.
@end deffn

Example usage:

@example
import io

output = io.StringIO()
output.write('First line.\n')
print('Second line.', file=output)

# Retrieve file contents -- this will be
# 'First line.\nSecond line.\n'
contents = output.getvalue()

# Close object and discard memory buffer --
# .getvalue() will now raise an exception.
output.close()
@end example
@end deffn

@geindex universal newlines; io.IncrementalNewlineDecoder class

@geindex IncrementalNewlineDecoder (class in io)
@anchor{library/io io IncrementalNewlineDecoder}@anchor{24ad}
@deffn {Class} io.IncrementalNewlineDecoder

A helper codec that decodes newlines for @ref{88a,,universal newlines} mode.
It inherits @ref{1cae,,codecs.IncrementalDecoder}.
@end deffn

@node Performance<3>,,Class hierarchy,io — Core tools for working with streams
@anchor{library/io performance}@anchor{24ae}
@subsubsection Performance


This section discusses the performance of the provided concrete I/O
implementations.

@menu
* Binary I/O: Binary I/O<2>. 
* Text I/O: Text I/O<3>. 
* Multi-threading: Multi-threading<3>. 
* Reentrancy:: 

@end menu

@node Binary I/O<2>,Text I/O<3>,,Performance<3>
@anchor{library/io id2}@anchor{24af}
@subsubsection Binary I/O


By reading and writing only large chunks of data even when the user asks for a
single byte, buffered I/O hides any inefficiency in calling and executing the
operating system’s unbuffered I/O routines.  The gain depends on the OS and the
kind of I/O which is performed.  For example, on some modern OSes such as Linux,
unbuffered disk I/O can be as fast as buffered I/O.  The bottom line, however,
is that buffered I/O offers predictable performance regardless of the platform
and the backing device.  Therefore, it is almost always preferable to use
buffered I/O rather than unbuffered I/O for binary data.

@node Text I/O<3>,Multi-threading<3>,Binary I/O<2>,Performance<3>
@anchor{library/io id3}@anchor{24b0}
@subsubsection Text I/O


Text I/O over a binary storage (such as a file) is significantly slower than
binary I/O over the same storage, because it requires conversions between
unicode and binary data using a character codec.  This can become noticeable
handling huge amounts of text data like large log files.  Also,
@code{TextIOWrapper.tell()} and @code{TextIOWrapper.seek()} are both quite slow
due to the reconstruction algorithm used.

@ref{aa1,,StringIO}, however, is a native in-memory unicode container and will
exhibit similar speed to @ref{a15,,BytesIO}.

@node Multi-threading<3>,Reentrancy,Text I/O<3>,Performance<3>
@anchor{library/io multi-threading}@anchor{24b1}
@subsubsection Multi-threading


@ref{ea6,,FileIO} objects are thread-safe to the extent that the operating system
calls (such as @code{read(2)} under Unix) they wrap are thread-safe too.

Binary buffered objects (instances of @ref{daf,,BufferedReader},
@ref{1a5a,,BufferedWriter}, @ref{1a5b,,BufferedRandom} and @ref{248b,,BufferedRWPair})
protect their internal structures using a lock; it is therefore safe to call
them from multiple threads at once.

@ref{889,,TextIOWrapper} objects are not thread-safe.

@node Reentrancy,,Multi-threading<3>,Performance<3>
@anchor{library/io reentrancy}@anchor{24b2}
@subsubsection Reentrancy


Binary buffered objects (instances of @ref{daf,,BufferedReader},
@ref{1a5a,,BufferedWriter}, @ref{1a5b,,BufferedRandom} and @ref{248b,,BufferedRWPair})
are not reentrant.  While reentrant calls will not happen in normal situations,
they can arise from doing I/O in a @ref{e5,,signal} handler.  If a thread tries to
re-enter a buffered object which it is already accessing, a @ref{5af,,RuntimeError}
is raised.  Note this doesn’t prohibit a different thread from entering the
buffered object.

The above implicitly extends to text files, since the @ref{1c8,,open()} function
will wrap a buffered object inside a @ref{889,,TextIOWrapper}.  This includes
standard streams and therefore affects the built-in @ref{af6,,print()} function as
well.

@node time — Time access and conversions,argparse — Parser for command-line options arguments and sub-commands,io — Core tools for working with streams,Generic Operating System Services
@anchor{library/time doc}@anchor{24b3}@anchor{library/time module-time}@anchor{10a}@anchor{library/time time-time-access-and-conversions}@anchor{24b4}
@subsection @code{time} — Time access and conversions


@geindex module; time


__________________________________________________________________


This module provides various time-related functions. For related
functionality, see also the @ref{2f,,datetime} and @ref{13,,calendar} modules.

Although this module is always available,
not all functions are available on all platforms.  Most of the functions
defined in this module call platform C library functions with the same name.  It
may sometimes be helpful to consult the platform documentation, because the
semantics of these functions varies among platforms.

An explanation of some terminology and conventions is in order.
@anchor{library/time epoch}@anchor{24b5}
@geindex epoch


@itemize *

@item 
The @emph{epoch} is the point where the time starts, the return value of
@code{time.gmtime(0)}. It is January 1, 1970, 00:00:00 (UTC) on all platforms.
@end itemize

@geindex seconds since the epoch


@itemize *

@item 
The term @emph{seconds since the epoch} refers to the total number
of elapsed seconds since the epoch, typically excluding
leap seconds@footnote{https://en.wikipedia.org/wiki/Leap_second}.  Leap seconds are excluded from this total on all
POSIX-compliant platforms.
@end itemize

@geindex Year 2038


@itemize *

@item 
The functions in this module may not handle dates and times before the @ref{24b5,,epoch} or
far in the future.  The cut-off point in the future is determined by the C
library; for 32-bit systems, it is typically in 2038.
@end itemize

@geindex 2-digit years


@itemize *

@item 
Function @ref{f8b,,strptime()} can parse 2-digit years when given @code{%y} format
code. When 2-digit years are parsed, they are converted according to the POSIX
and ISO C standards: values 69–99 are mapped to 1969–1999, and values 0–68
are mapped to 2000–2068.
@end itemize

@geindex UTC
@geindex Coordinated Universal Time
@geindex Greenwich Mean Time


@itemize *

@item 
UTC is Coordinated Universal Time (formerly known as Greenwich Mean Time, or
GMT).  The acronym UTC is not a mistake but a compromise between English and
French.
@end itemize

@geindex Daylight Saving Time


@itemize *

@item 
DST is Daylight Saving Time, an adjustment of the timezone by (usually) one
hour during part of the year.  DST rules are magic (determined by local law) and
can change from year to year.  The C library has a table containing the local
rules (often it is read from a system file for flexibility) and is the only
source of True Wisdom in this respect.

@item 
The precision of the various real-time functions may be less than suggested by
the units in which their value or argument is expressed. E.g. on most Unix
systems, the clock “ticks” only 50 or 100 times a second.

@item 
On the other hand, the precision of @ref{5fc,,time()} and @ref{196,,sleep()} is better
than their Unix equivalents: times are expressed as floating point numbers,
@ref{5fc,,time()} returns the most accurate time available (using Unix
@code{gettimeofday()} where available), and @ref{196,,sleep()} will accept a time
with a nonzero fraction (Unix @code{select()} is used to implement this, where
available).

@item 
The time value as returned by @ref{d65,,gmtime()}, @ref{1224,,localtime()}, and
@ref{f8b,,strptime()}, and accepted by @ref{d89,,asctime()}, @ref{d8a,,mktime()} and
@ref{d8b,,strftime()}, is a sequence of 9 integers.  The return values of
@ref{d65,,gmtime()}, @ref{1224,,localtime()}, and @ref{f8b,,strptime()} also offer attribute
names for individual fields.

See @ref{834,,struct_time} for a description of these objects.


Changed in version 3.3: The @ref{834,,struct_time} type was extended to provide the @code{tm_gmtoff}
and @code{tm_zone} attributes when platform supports corresponding
@code{struct tm} members.



Changed in version 3.6: The @ref{834,,struct_time} attributes @code{tm_gmtoff} and @code{tm_zone}
are now available on all platforms.


@item 
Use the following functions to convert between time representations:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

From

@tab

To

@tab

Use

@item

seconds since the epoch

@tab

@ref{834,,struct_time} in
UTC

@tab

@ref{d65,,gmtime()}

@item

seconds since the epoch

@tab

@ref{834,,struct_time} in
local time

@tab

@ref{1224,,localtime()}

@item

@ref{834,,struct_time} in
UTC

@tab

seconds since the epoch

@tab

@ref{1dbb,,calendar.timegm()}

@item

@ref{834,,struct_time} in
local time

@tab

seconds since the epoch

@tab

@ref{d8a,,mktime()}

@end multitable

@end itemize

@menu
* Functions: Functions<3>. 
* Clock ID Constants:: 
* Timezone Constants:: 

@end menu

@node Functions<3>,Clock ID Constants,,time — Time access and conversions
@anchor{library/time functions}@anchor{24b6}@anchor{library/time time-functions}@anchor{24b7}
@subsubsection Functions


@geindex asctime() (in module time)
@anchor{library/time time asctime}@anchor{d89}
@deffn {Function} time.asctime ([t])

Convert a tuple or @ref{834,,struct_time} representing a time as returned by
@ref{d65,,gmtime()} or @ref{1224,,localtime()} to a string of the following
form: @code{'Sun Jun 20 23:21:05 1993'}. The day field is two characters long
and is space padded if the day is a single digit,
e.g.: @code{'Wed Jun  9 04:26:40 1993'}.

If @emph{t} is not provided, the current time as returned by @ref{1224,,localtime()}
is used. Locale information is not used by @ref{d89,,asctime()}.

@cartouche
@quotation Note 
Unlike the C function of the same name, @ref{d89,,asctime()} does not add a
trailing newline.
@end quotation
@end cartouche
@end deffn

@geindex pthread_getcpuclockid() (in module time)
@anchor{library/time time pthread_getcpuclockid}@anchor{6c3}
@deffn {Function} time.pthread_getcpuclockid (thread_id)

Return the @emph{clk_id} of the thread-specific CPU-time clock for the specified @emph{thread_id}.

Use @ref{cf5,,threading.get_ident()} or the @ref{24b8,,ident}
attribute of @ref{44b,,threading.Thread} objects to get a suitable value
for @emph{thread_id}.

@cartouche
@quotation Warning 
Passing an invalid or expired @emph{thread_id} may result in
undefined behavior, such as segmentation fault.
@end quotation
@end cartouche

@ref{1629,,Availability}: Unix (see the man page for @code{pthread_getcpuclockid(3)} for
further information).


New in version 3.7.

@end deffn

@geindex clock_getres() (in module time)
@anchor{library/time time clock_getres}@anchor{cf8}
@deffn {Function} time.clock_getres (clk_id)

Return the resolution (precision) of the specified clock @emph{clk_id}.  Refer to
@ref{24b9,,Clock ID Constants} for a list of accepted values for @emph{clk_id}.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex clock_gettime() (in module time)
@anchor{library/time time clock_gettime}@anchor{cf9}
@deffn {Function} time.clock_gettime (clk_id) -> float

Return the time of the specified clock @emph{clk_id}.  Refer to
@ref{24b9,,Clock ID Constants} for a list of accepted values for @emph{clk_id}.

Use @ref{5fd,,clock_gettime_ns()} to avoid the precision loss caused by the
@ref{256,,float} type.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex clock_gettime_ns() (in module time)
@anchor{library/time time clock_gettime_ns}@anchor{5fd}
@deffn {Function} time.clock_gettime_ns (clk_id) -> int

Similar to @ref{cf9,,clock_gettime()} but return time as nanoseconds.

@ref{1629,,Availability}: Unix.


New in version 3.7.

@end deffn

@geindex clock_settime() (in module time)
@anchor{library/time time clock_settime}@anchor{cfa}
@deffn {Function} time.clock_settime (clk_id, time: float)

Set the time of the specified clock @emph{clk_id}.  Currently,
@ref{194,,CLOCK_REALTIME} is the only accepted value for @emph{clk_id}.

Use @ref{5fe,,clock_settime_ns()} to avoid the precision loss caused by the
@ref{256,,float} type.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex clock_settime_ns() (in module time)
@anchor{library/time time clock_settime_ns}@anchor{5fe}
@deffn {Function} time.clock_settime_ns (clk_id, time: int)

Similar to @ref{cfa,,clock_settime()} but set time with nanoseconds.

@ref{1629,,Availability}: Unix.


New in version 3.7.

@end deffn

@geindex ctime() (in module time)
@anchor{library/time time ctime}@anchor{1d35}
@deffn {Function} time.ctime ([secs])

Convert a time expressed in seconds since the @ref{24b5,,epoch} to a string of a form:
@code{'Sun Jun 20 23:21:05 1993'} representing local time. The day field
is two characters long and is space padded if the day is a single digit,
e.g.: @code{'Wed Jun  9 04:26:40 1993'}.

If @emph{secs} is not provided or @ref{182,,None}, the current time as
returned by @ref{5fc,,time()} is used. @code{ctime(secs)} is equivalent to
@code{asctime(localtime(secs))}. Locale information is not used by
@ref{1d35,,ctime()}.
@end deffn

@geindex get_clock_info() (in module time)
@anchor{library/time time get_clock_info}@anchor{cf7}
@deffn {Function} time.get_clock_info (name)

Get information on the specified clock as a namespace object.
Supported clock names and the corresponding functions to read their value
are:


@itemize *

@item 
@code{'monotonic'}: @ref{9ed,,time.monotonic()}

@item 
@code{'perf_counter'}: @ref{5a0,,time.perf_counter()}

@item 
@code{'process_time'}: @ref{5a1,,time.process_time()}

@item 
@code{'thread_time'}: @ref{42a,,time.thread_time()}

@item 
@code{'time'}: @ref{5fc,,time.time()}
@end itemize

The result has the following attributes:


@itemize -

@item 
@emph{adjustable}: @code{True} if the clock can be changed automatically (e.g. by
a NTP daemon) or manually by the system administrator, @code{False} otherwise

@item 
@emph{implementation}: The name of the underlying C function used to get
the clock value.  Refer to @ref{24b9,,Clock ID Constants} for possible values.

@item 
@emph{monotonic}: @code{True} if the clock cannot go backward,
@code{False} otherwise

@item 
@emph{resolution}: The resolution of the clock in seconds (@ref{256,,float})
@end itemize


New in version 3.3.

@end deffn

@geindex gmtime() (in module time)
@anchor{library/time time gmtime}@anchor{d65}
@deffn {Function} time.gmtime ([secs])

Convert a time expressed in seconds since the @ref{24b5,,epoch} to a @ref{834,,struct_time} in
UTC in which the dst flag is always zero.  If @emph{secs} is not provided or
@ref{182,,None}, the current time as returned by @ref{5fc,,time()} is used.  Fractions
of a second are ignored.  See above for a description of the
@ref{834,,struct_time} object. See @ref{1dbb,,calendar.timegm()} for the inverse of this
function.
@end deffn

@geindex localtime() (in module time)
@anchor{library/time time localtime}@anchor{1224}
@deffn {Function} time.localtime ([secs])

Like @ref{d65,,gmtime()} but converts to local time.  If @emph{secs} is not provided or
@ref{182,,None}, the current time as returned by @ref{5fc,,time()} is used.  The dst
flag is set to @code{1} when DST applies to the given time.
@end deffn

@geindex mktime() (in module time)
@anchor{library/time time mktime}@anchor{d8a}
@deffn {Function} time.mktime (t)

This is the inverse function of @ref{1224,,localtime()}.  Its argument is the
@ref{834,,struct_time} or full 9-tuple (since the dst flag is needed; use @code{-1}
as the dst flag if it is unknown) which expresses the time in @emph{local} time, not
UTC.  It returns a floating point number, for compatibility with @ref{5fc,,time()}.
If the input value cannot be represented as a valid time, either
@ref{347,,OverflowError} or @ref{17c,,ValueError} will be raised (which depends on
whether the invalid value is caught by Python or the underlying C libraries).
The earliest date for which it can generate a time is platform-dependent.
@end deffn

@geindex monotonic() (in module time)
@anchor{library/time time monotonic}@anchor{9ed}
@deffn {Function} time.monotonic () -> float

Return the value (in fractional seconds) of a monotonic clock, i.e. a clock
that cannot go backwards.  The clock is not affected by system clock updates.
The reference point of the returned value is undefined, so that only the
difference between the results of two calls is valid.

Use @ref{5ff,,monotonic_ns()} to avoid the precision loss caused by the
@ref{256,,float} type.


New in version 3.3.



Changed in version 3.5: The function is now always available and always system-wide.



Changed in version 3.10: On macOS, the function is now system-wide.

@end deffn

@geindex monotonic_ns() (in module time)
@anchor{library/time time monotonic_ns}@anchor{5ff}
@deffn {Function} time.monotonic_ns () -> int

Similar to @ref{9ed,,monotonic()}, but return time as nanoseconds.


New in version 3.7.

@end deffn

@geindex perf_counter() (in module time)
@anchor{library/time time perf_counter}@anchor{5a0}
@deffn {Function} time.perf_counter () -> float

@geindex benchmarking

Return the value (in fractional seconds) of a performance counter, i.e. a
clock with the highest available resolution to measure a short duration.  It
does include time elapsed during sleep and is system-wide.  The reference
point of the returned value is undefined, so that only the difference between
the results of two calls is valid.

Use @ref{600,,perf_counter_ns()} to avoid the precision loss caused by the
@ref{256,,float} type.


New in version 3.3.



Changed in version 3.10: On Windows, the function is now system-wide.

@end deffn

@geindex perf_counter_ns() (in module time)
@anchor{library/time time perf_counter_ns}@anchor{600}
@deffn {Function} time.perf_counter_ns () -> int

Similar to @ref{5a0,,perf_counter()}, but return time as nanoseconds.


New in version 3.7.

@end deffn

@geindex process_time() (in module time)
@anchor{library/time time process_time}@anchor{5a1}
@deffn {Function} time.process_time () -> float

@geindex CPU time
@geindex processor time
@geindex benchmarking

Return the value (in fractional seconds) of the sum of the system and user
CPU time of the current process.  It does not include time elapsed during
sleep.  It is process-wide by definition.  The reference point of the
returned value is undefined, so that only the difference between the results
of two calls is valid.

Use @ref{601,,process_time_ns()} to avoid the precision loss caused by the
@ref{256,,float} type.


New in version 3.3.

@end deffn

@geindex process_time_ns() (in module time)
@anchor{library/time time process_time_ns}@anchor{601}
@deffn {Function} time.process_time_ns () -> int

Similar to @ref{5a1,,process_time()} but return time as nanoseconds.


New in version 3.7.

@end deffn

@geindex sleep() (in module time)
@anchor{library/time time sleep}@anchor{196}
@deffn {Function} time.sleep (secs)

Suspend execution of the calling thread for the given number of seconds.
The argument may be a floating point number to indicate a more precise sleep
time.

If the sleep is interrupted by a signal and no exception is raised by the
signal handler, the sleep is restarted with a recomputed timeout.

The suspension time may be longer than requested by an arbitrary amount,
because of the scheduling of other activity in the system.

On Windows, if @emph{secs} is zero, the thread relinquishes the remainder of its
time slice to any other thread that is ready to run. If there are no other
threads ready to run, the function returns immediately, and the thread
continues execution.  On Windows 8.1 and newer the implementation uses
a high-resolution timer@footnote{https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/high-resolution-timers}
which provides resolution of 100 nanoseconds. If @emph{secs} is zero, @code{Sleep(0)} is used.

Unix implementation:


@itemize *

@item 
Use @code{clock_nanosleep()} if available (resolution: 1 nanosecond);

@item 
Or use @code{nanosleep()} if available (resolution: 1 nanosecond);

@item 
Or use @code{select()} (resolution: 1 microsecond).
@end itemize


Changed in version 3.11: On Unix, the @code{clock_nanosleep()} and @code{nanosleep()} functions are now
used if available. On Windows, a waitable timer is now used.



Changed in version 3.5: The function now sleeps at least @emph{secs} even if the sleep is interrupted
by a signal, except if the signal handler raises an exception (see
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).

@end deffn

@geindex % (percent); datetime format

@geindex strftime() (in module time)
@anchor{library/time time strftime}@anchor{d8b}
@deffn {Function} time.strftime (format[, t])

Convert a tuple or @ref{834,,struct_time} representing a time as returned by
@ref{d65,,gmtime()} or @ref{1224,,localtime()} to a string as specified by the @emph{format}
argument.  If @emph{t} is not provided, the current time as returned by
@ref{1224,,localtime()} is used.  @emph{format} must be a string.  @ref{17c,,ValueError} is
raised if any field in @emph{t} is outside of the allowed range.

0 is a legal argument for any position in the time tuple; if it is normally
illegal the value is forced to a correct one.

The following directives can be embedded in the @emph{format} string. They are shown
without the optional field width and precision specification, and are replaced
by the indicated characters in the @ref{d8b,,strftime()} result:


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} 
@headitem

Directive

@tab

Meaning

@tab

Notes

@item

@code{%a}

@tab

Locale’s abbreviated weekday name.

@tab

@item

@code{%A}

@tab

Locale’s full weekday name.

@tab

@item

@code{%b}

@tab

Locale’s abbreviated month name.

@tab

@item

@code{%B}

@tab

Locale’s full month name.

@tab

@item

@code{%c}

@tab

Locale’s appropriate date and time
representation.

@tab

@item

@code{%d}

@tab

Day of the month as a decimal number [01,31].

@tab

@item

@code{%H}

@tab

Hour (24-hour clock) as a decimal number
[00,23].

@tab

@item

@code{%I}

@tab

Hour (12-hour clock) as a decimal number
[01,12].

@tab

@item

@code{%j}

@tab

Day of the year as a decimal number [001,366].

@tab

@item

@code{%m}

@tab

Month as a decimal number [01,12].

@tab

@item

@code{%M}

@tab

Minute as a decimal number [00,59].

@tab

@item

@code{%p}

@tab

Locale’s equivalent of either AM or PM.

@tab

(1)

@item

@code{%S}

@tab

Second as a decimal number [00,61].

@tab

(2)

@item

@code{%U}

@tab

Week number of the year (Sunday as the first
day of the week) as a decimal number [00,53].
All days in a new year preceding the first
Sunday are considered to be in week 0.

@tab

(3)

@item

@code{%w}

@tab

Weekday as a decimal number [0(Sunday),6].

@tab

@item

@code{%W}

@tab

Week number of the year (Monday as the first
day of the week) as a decimal number [00,53].
All days in a new year preceding the first
Monday are considered to be in week 0.

@tab

(3)

@item

@code{%x}

@tab

Locale’s appropriate date representation.

@tab

@item

@code{%X}

@tab

Locale’s appropriate time representation.

@tab

@item

@code{%y}

@tab

Year without century as a decimal number
[00,99].

@tab

@item

@code{%Y}

@tab

Year with century as a decimal number.

@tab

@item

@code{%z}

@tab

Time zone offset indicating a positive or
negative time difference from UTC/GMT of the
form +HHMM or -HHMM, where H represents decimal
hour digits and M represents decimal minute
digits [-23:59, +23:59]. @footnote{@w{(1)} 
The use of @code{%Z} is now deprecated, but the @code{%z} escape that expands to the
preferred hour/minute offset is not supported by all ANSI C libraries. Also, a
strict reading of the original 1982 
@geindex RFC; RFC 822
@uref{https://tools.ietf.org/html/rfc822.html,RFC 822} standard calls for a two-digit
year (@code{%y} rather than @code{%Y}), but practice moved to 4-digit years long before the
year 2000.  After that, 
@geindex RFC; RFC 822
@uref{https://tools.ietf.org/html/rfc822.html,RFC 822} became obsolete and the 4-digit year has
been first recommended by 
@geindex RFC; RFC 1123
@uref{https://tools.ietf.org/html/rfc1123.html,RFC 1123} and then mandated by 
@geindex RFC; RFC 2822
@uref{https://tools.ietf.org/html/rfc2822.html,RFC 2822}.
}

@tab

@item

@code{%Z}

@tab

Time zone name (no characters if no time zone
exists). Deprecated. @footnote{@w{(1)} 
The use of @code{%Z} is now deprecated, but the @code{%z} escape that expands to the
preferred hour/minute offset is not supported by all ANSI C libraries. Also, a
strict reading of the original 1982 
@geindex RFC; RFC 822
@uref{https://tools.ietf.org/html/rfc822.html,RFC 822} standard calls for a two-digit
year (@code{%y} rather than @code{%Y}), but practice moved to 4-digit years long before the
year 2000.  After that, 
@geindex RFC; RFC 822
@uref{https://tools.ietf.org/html/rfc822.html,RFC 822} became obsolete and the 4-digit year has
been first recommended by 
@geindex RFC; RFC 1123
@uref{https://tools.ietf.org/html/rfc1123.html,RFC 1123} and then mandated by 
@geindex RFC; RFC 2822
@uref{https://tools.ietf.org/html/rfc2822.html,RFC 2822}.
}

@tab

@item

@code{%%}

@tab

A literal @code{'%'} character.

@tab

@end multitable


Notes:


@enumerate 

@item 
When used with the @ref{f8b,,strptime()} function, the @code{%p} directive only affects
the output hour field if the @code{%I} directive is used to parse the hour.

@item 
The range really is @code{0} to @code{61}; value @code{60} is valid in
timestamps representing leap seconds@footnote{https://en.wikipedia.org/wiki/Leap_second} and value @code{61} is supported
for historical reasons.

@item 
When used with the @ref{f8b,,strptime()} function, @code{%U} and @code{%W} are only used in
calculations when the day of the week and the year are specified.
@end enumerate

Here is an example, a format for dates compatible with that specified  in the
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html} Internet email standard.  @footnote{@w{(1)} 
The use of @code{%Z} is now deprecated, but the @code{%z} escape that expands to the
preferred hour/minute offset is not supported by all ANSI C libraries. Also, a
strict reading of the original 1982 
@geindex RFC; RFC 822
@uref{https://tools.ietf.org/html/rfc822.html,RFC 822} standard calls for a two-digit
year (@code{%y} rather than @code{%Y}), but practice moved to 4-digit years long before the
year 2000.  After that, 
@geindex RFC; RFC 822
@uref{https://tools.ietf.org/html/rfc822.html,RFC 822} became obsolete and the 4-digit year has
been first recommended by 
@geindex RFC; RFC 1123
@uref{https://tools.ietf.org/html/rfc1123.html,RFC 1123} and then mandated by 
@geindex RFC; RFC 2822
@uref{https://tools.ietf.org/html/rfc2822.html,RFC 2822}.
}

@example
>>> from time import gmtime, strftime
>>> strftime("%a, %d %b %Y %H:%M:%S +0000", gmtime())
'Thu, 28 Jun 2001 14:17:15 +0000'
@end example

Additional directives may be supported on certain platforms, but only the
ones listed here have a meaning standardized by ANSI C.  To see the full set
of format codes supported on your platform, consult the @code{strftime(3)}
documentation.

On some platforms, an optional field width and precision specification can
immediately follow the initial @code{'%'} of a directive in the following order;
this is also not portable. The field width is normally 2 except for @code{%j} where
it is 3.
@end deffn

@geindex % (percent); datetime format

@geindex strptime() (in module time)
@anchor{library/time time strptime}@anchor{f8b}
@deffn {Function} time.strptime (string[, format])

Parse a string representing a time according to a format.  The return value
is a @ref{834,,struct_time} as returned by @ref{d65,,gmtime()} or
@ref{1224,,localtime()}.

The @emph{format} parameter uses the same directives as those used by
@ref{d8b,,strftime()}; it defaults to @code{"%a %b %d %H:%M:%S %Y"} which matches the
formatting returned by @ref{1d35,,ctime()}. If @emph{string} cannot be parsed according
to @emph{format}, or if it has excess data after parsing, @ref{17c,,ValueError} is
raised. The default values used to fill in any missing data when more
accurate values cannot be inferred are @code{(1900, 1, 1, 0, 0, 0, 0, 1, -1)}.
Both @emph{string} and @emph{format} must be strings.

For example:

@example
>>> import time
>>> time.strptime("30 Nov 00", "%d %b %y")   
time.struct_time(tm_year=2000, tm_mon=11, tm_mday=30, tm_hour=0, tm_min=0,
                 tm_sec=0, tm_wday=3, tm_yday=335, tm_isdst=-1)
@end example

Support for the @code{%Z} directive is based on the values contained in @code{tzname}
and whether @code{daylight} is true.  Because of this, it is platform-specific
except for recognizing UTC and GMT which are always known (and are considered to
be non-daylight savings timezones).

Only the directives specified in the documentation are supported.  Because
@code{strftime()} is implemented per platform it can sometimes offer more
directives than those listed.  But @code{strptime()} is independent of any platform
and thus does not necessarily support all directives available that are not
documented as supported.
@end deffn

@geindex struct_time (class in time)
@anchor{library/time time struct_time}@anchor{834}
@deffn {Class} time.struct_time

The type of the time value sequence returned by @ref{d65,,gmtime()},
@ref{1224,,localtime()}, and @ref{f8b,,strptime()}.  It is an object with a @ref{ce9,,named tuple} interface: values can be accessed by index and by attribute name.  The
following values are present:


@multitable {xxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Index

@tab

Attribute

@tab

Values

@item

0

@tab

@code{tm_year}

@tab

(for example, 1993)

@item

1

@tab

@code{tm_mon}

@tab

range [1, 12]

@item

2

@tab

@code{tm_mday}

@tab

range [1, 31]

@item

3

@tab

@code{tm_hour}

@tab

range [0, 23]

@item

4

@tab

@code{tm_min}

@tab

range [0, 59]

@item

5

@tab

@code{tm_sec}

@tab

range [0, 61]; see @strong{(2)} in
@ref{d8b,,strftime()} description

@item

6

@tab

@code{tm_wday}

@tab

range [0, 6], Monday is 0

@item

7

@tab

@code{tm_yday}

@tab

range [1, 366]

@item

8

@tab

@code{tm_isdst}

@tab

0, 1 or -1; see below

@item

N/A

@tab

@code{tm_zone}

@tab

abbreviation of timezone name

@item

N/A

@tab

@code{tm_gmtoff}

@tab

offset east of UTC in seconds

@end multitable


Note that unlike the C structure, the month value is a range of [1, 12], not
[0, 11].

In calls to @ref{d8a,,mktime()}, @code{tm_isdst} may be set to 1 when daylight
savings time is in effect, and 0 when it is not.  A value of -1 indicates that
this is not known, and will usually result in the correct state being filled in.

When a tuple with an incorrect length is passed to a function expecting a
@ref{834,,struct_time}, or having elements of the wrong type, a
@ref{162,,TypeError} is raised.
@end deffn

@geindex time() (in module time)
@anchor{library/time time time}@anchor{5fc}
@deffn {Function} time.time () -> float

Return the time in seconds since the @ref{24b5,,epoch} as a floating point
number. The handling of leap seconds@footnote{https://en.wikipedia.org/wiki/Leap_second} is platform dependent.
On Windows and most Unix systems, the leap seconds are not counted towards
the time in seconds since the @ref{24b5,,epoch}. This is commonly referred to as Unix time@footnote{https://en.wikipedia.org/wiki/Unix_time}.

Note that even though the time is always returned as a floating point
number, not all systems provide time with a better precision than 1 second.
While this function normally returns non-decreasing values, it can return a
lower value than a previous call if the system clock has been set back
between the two calls.

The number returned by @ref{5fc,,time()} may be converted into a more common
time format (i.e. year, month, day, hour, etc…) in UTC by passing it to
@ref{d65,,gmtime()} function or in local time by passing it to the
@ref{1224,,localtime()} function. In both cases a
@ref{834,,struct_time} object is returned, from which the components
of the calendar date may be accessed as attributes.

Use @ref{602,,time_ns()} to avoid the precision loss caused by the @ref{256,,float}
type.
@end deffn

@geindex time_ns() (in module time)
@anchor{library/time time time_ns}@anchor{602}
@deffn {Function} time.time_ns () -> int

Similar to @ref{5fc,,time()} but returns time as an integer number of
nanoseconds since the @ref{24b5,,epoch}.


New in version 3.7.

@end deffn

@geindex thread_time() (in module time)
@anchor{library/time time thread_time}@anchor{42a}
@deffn {Function} time.thread_time () -> float

@geindex CPU time
@geindex processor time
@geindex benchmarking

Return the value (in fractional seconds) of the sum of the system and user
CPU time of the current thread.  It does not include time elapsed during
sleep.  It is thread-specific by definition.  The reference point of the
returned value is undefined, so that only the difference between the results
of two calls in the same thread is valid.

Use @ref{6c2,,thread_time_ns()} to avoid the precision loss caused by the
@ref{256,,float} type.

@ref{1629,,Availability}: Windows, Linux, Unix systems supporting
@code{CLOCK_THREAD_CPUTIME_ID}.


New in version 3.7.

@end deffn

@geindex thread_time_ns() (in module time)
@anchor{library/time time thread_time_ns}@anchor{6c2}
@deffn {Function} time.thread_time_ns () -> int

Similar to @ref{42a,,thread_time()} but return time as nanoseconds.


New in version 3.7.

@end deffn

@geindex tzset() (in module time)
@anchor{library/time time tzset}@anchor{24ba}
@deffn {Function} time.tzset ()

Reset the time conversion rules used by the library routines. The environment
variable 
@geindex TZ
@geindex environment variable; TZ
@code{TZ} specifies how this is done. It will also set the variables
@code{tzname} (from the 
@geindex TZ
@geindex environment variable; TZ
@code{TZ} environment variable), @code{timezone} (non-DST
seconds West of UTC), @code{altzone} (DST seconds west of UTC) and @code{daylight}
(to 0 if this timezone does not have any daylight saving time rules, or to
nonzero if there is a time, past, present or future when daylight saving time
applies).

@ref{1629,,Availability}: Unix.

@cartouche
@quotation Note 
Although in many cases, changing the 
@geindex TZ
@geindex environment variable; TZ
@code{TZ} environment variable may
affect the output of functions like @ref{1224,,localtime()} without calling
@ref{24ba,,tzset()}, this behavior should not be relied on.

The 
@geindex TZ
@geindex environment variable; TZ
@code{TZ} environment variable should contain no whitespace.
@end quotation
@end cartouche

The standard format of the 
@geindex TZ
@geindex environment variable; TZ
@code{TZ} environment variable is (whitespace
added for clarity):

@example
std offset [dst [offset [,start[/time], end[/time]]]]
@end example

Where the components are:


@table @asis

@item @code{std} and @code{dst}

Three or more alphanumerics giving the timezone abbreviations. These will be
propagated into time.tzname

@item @code{offset}

The offset has the form: @code{± hh[:mm[:ss]]}. This indicates the value
added the local time to arrive at UTC.  If preceded by a ‘-’, the timezone
is east of the Prime Meridian; otherwise, it is west. If no offset follows
dst, summer time is assumed to be one hour ahead of standard time.

@item @code{start[/time], end[/time]}

Indicates when to change to and back from DST. The format of the
start and end dates are one of the following:


@table @asis

@item @code{J@emph{n}}

The Julian day @emph{n} (1 <= @emph{n} <= 365). Leap days are not counted, so in
all years February 28 is day 59 and March 1 is day 60.

@item @code{@emph{n}}

The zero-based Julian day (0 <= @emph{n} <= 365). Leap days are counted, and
it is possible to refer to February 29.

@item @code{M@emph{m}.@emph{n}.@emph{d}}

The @emph{d}’th day (0 <= @emph{d} <= 6) of week @emph{n} of month @emph{m} of the year (1
<= @emph{n} <= 5, 1 <= @emph{m} <= 12, where week 5 means “the last @emph{d} day in
month @emph{m}” which may occur in either the fourth or the fifth
week). Week 1 is the first week in which the @emph{d}’th day occurs. Day
zero is a Sunday.
@end table

@code{time} has the same format as @code{offset} except that no leading sign
(‘-’ or ‘+’) is allowed. The default, if time is not given, is 02:00:00.
@end table

@example
>>> os.environ['TZ'] = 'EST+05EDT,M4.1.0,M10.5.0'
>>> time.tzset()
>>> time.strftime('%X %x %Z')
'02:07:36 05/08/03 EDT'
>>> os.environ['TZ'] = 'AEST-10AEDT-11,M10.5.0,M3.5.0'
>>> time.tzset()
>>> time.strftime('%X %x %Z')
'16:08:12 05/08/03 AEST'
@end example

On many Unix systems (including *BSD, Linux, Solaris, and Darwin), it is more
convenient to use the system’s zoneinfo (@code{tzfile(5)})  database to
specify the timezone rules. To do this, set the  
@geindex TZ
@geindex environment variable; TZ
@code{TZ} environment
variable to the path of the required timezone  datafile, relative to the root of
the systems ‘zoneinfo’ timezone database, usually located at
@code{/usr/share/zoneinfo}. For example,  @code{'US/Eastern'},
@code{'Australia/Melbourne'}, @code{'Egypt'} or  @code{'Europe/Amsterdam'}.

@example
>>> os.environ['TZ'] = 'US/Eastern'
>>> time.tzset()
>>> time.tzname
('EST', 'EDT')
>>> os.environ['TZ'] = 'Egypt'
>>> time.tzset()
>>> time.tzname
('EET', 'EEST')
@end example
@end deffn

@node Clock ID Constants,Timezone Constants,Functions<3>,time — Time access and conversions
@anchor{library/time clock-id-constants}@anchor{24bb}@anchor{library/time time-clock-id-constants}@anchor{24b9}
@subsubsection Clock ID Constants


These constants are used as parameters for @ref{cf8,,clock_getres()} and
@ref{cf9,,clock_gettime()}.

@geindex CLOCK_BOOTTIME (in module time)
@anchor{library/time time CLOCK_BOOTTIME}@anchor{6bf}
@deffn {Data} time.CLOCK_BOOTTIME

Identical to @ref{193,,CLOCK_MONOTONIC}, except it also includes any time that
the system is suspended.

This allows applications to get a suspend-aware monotonic  clock  without
having to deal with the complications of @ref{194,,CLOCK_REALTIME}, which may
have  discontinuities if the time is changed using @code{settimeofday()} or
similar.

@ref{1629,,Availability}: Linux 2.6.39 or later.


New in version 3.7.

@end deffn

@geindex CLOCK_HIGHRES (in module time)
@anchor{library/time time CLOCK_HIGHRES}@anchor{24bc}
@deffn {Data} time.CLOCK_HIGHRES

The Solaris OS has a @code{CLOCK_HIGHRES} timer that attempts to use an optimal
hardware source, and may give close to nanosecond resolution.
@code{CLOCK_HIGHRES} is the nonadjustable, high-resolution clock.

@ref{1629,,Availability}: Solaris.


New in version 3.3.

@end deffn

@geindex CLOCK_MONOTONIC (in module time)
@anchor{library/time time CLOCK_MONOTONIC}@anchor{193}
@deffn {Data} time.CLOCK_MONOTONIC

Clock that cannot be set and represents monotonic time since some unspecified
starting point.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex CLOCK_MONOTONIC_RAW (in module time)
@anchor{library/time time CLOCK_MONOTONIC_RAW}@anchor{24bd}
@deffn {Data} time.CLOCK_MONOTONIC_RAW

Similar to @ref{193,,CLOCK_MONOTONIC}, but provides access to a raw
hardware-based time that is not subject to NTP adjustments.

@ref{1629,,Availability}: Linux 2.6.28 and newer, macOS 10.12 and newer.


New in version 3.3.

@end deffn

@geindex CLOCK_PROCESS_CPUTIME_ID (in module time)
@anchor{library/time time CLOCK_PROCESS_CPUTIME_ID}@anchor{24be}
@deffn {Data} time.CLOCK_PROCESS_CPUTIME_ID

High-resolution per-process timer from the CPU.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex CLOCK_PROF (in module time)
@anchor{library/time time CLOCK_PROF}@anchor{6c0}
@deffn {Data} time.CLOCK_PROF

High-resolution per-process timer from the CPU.

@ref{1629,,Availability}: FreeBSD, NetBSD 7 or later, OpenBSD.


New in version 3.7.

@end deffn

@geindex CLOCK_TAI (in module time)
@anchor{library/time time CLOCK_TAI}@anchor{11f8}
@deffn {Data} time.CLOCK_TAI

International Atomic Time@footnote{https://www.nist.gov/pml/time-and-frequency-division/nist-time-frequently-asked-questions-faq#tai}

The system must have a current leap second table in order for this to give
the correct answer.  PTP or NTP software can maintain a leap second table.

@ref{1629,,Availability}: Linux.


New in version 3.9.

@end deffn

@geindex CLOCK_THREAD_CPUTIME_ID (in module time)
@anchor{library/time time CLOCK_THREAD_CPUTIME_ID}@anchor{24bf}
@deffn {Data} time.CLOCK_THREAD_CPUTIME_ID

Thread-specific CPU-time clock.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex CLOCK_UPTIME (in module time)
@anchor{library/time time CLOCK_UPTIME}@anchor{6c1}
@deffn {Data} time.CLOCK_UPTIME

Time whose absolute value is the time the system has been running and not
suspended, providing accurate uptime measurement, both absolute and
interval.

@ref{1629,,Availability}: FreeBSD, OpenBSD 5.5 or later.


New in version 3.7.

@end deffn

@geindex CLOCK_UPTIME_RAW (in module time)
@anchor{library/time time CLOCK_UPTIME_RAW}@anchor{554}
@deffn {Data} time.CLOCK_UPTIME_RAW

Clock that increments monotonically, tracking the time since an arbitrary
point, unaffected by frequency or time adjustments and not incremented while
the system is asleep.

@ref{1629,,Availability}: macOS 10.12 and newer.


New in version 3.8.

@end deffn

The following constant is the only parameter that can be sent to
@ref{cfa,,clock_settime()}.

@geindex CLOCK_REALTIME (in module time)
@anchor{library/time time CLOCK_REALTIME}@anchor{194}
@deffn {Data} time.CLOCK_REALTIME

System-wide real-time clock.  Setting this clock requires appropriate
privileges.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@node Timezone Constants,,Clock ID Constants,time — Time access and conversions
@anchor{library/time time-timezone-constants}@anchor{24c0}@anchor{library/time timezone-constants}@anchor{24c1}
@subsubsection Timezone Constants


@geindex altzone (in module time)
@anchor{library/time time altzone}@anchor{24c2}
@deffn {Data} time.altzone

The offset of the local DST timezone, in seconds west of UTC, if one is defined.
This is negative if the local DST timezone is east of UTC (as in Western Europe,
including the UK).  Only use this if @code{daylight} is nonzero.  See note below.
@end deffn

@geindex daylight (in module time)
@anchor{library/time time daylight}@anchor{24c3}
@deffn {Data} time.daylight

Nonzero if a DST timezone is defined.  See note below.
@end deffn

@geindex timezone (in module time)
@anchor{library/time time timezone}@anchor{24c4}
@deffn {Data} time.timezone

The offset of the local (non-DST) timezone, in seconds west of UTC (negative in
most of Western Europe, positive in the US, zero in the UK).  See note below.
@end deffn

@geindex tzname (in module time)
@anchor{library/time time tzname}@anchor{24c5}
@deffn {Data} time.tzname

A tuple of two strings: the first is the name of the local non-DST timezone, the
second is the name of the local DST timezone.  If no DST timezone is defined,
the second string should not be used.  See note below.
@end deffn

@cartouche
@quotation Note 
For the above Timezone constants (@ref{24c2,,altzone}, @ref{24c3,,daylight}, @ref{24c4,,timezone},
and @ref{24c5,,tzname}), the value is determined by the timezone rules in effect
at module load time or the last time @ref{24ba,,tzset()} is called and may be incorrect
for times in the past.  It is recommended to use the @code{tm_gmtoff} and
@code{tm_zone} results from @ref{1224,,localtime()} to obtain timezone information.
@end quotation
@end cartouche


@subsubheading See also



@table @asis

@item Module @ref{2f,,datetime}

More object-oriented interface to dates and times.

@item Module @ref{a5,,locale}

Internationalization services.  The locale setting affects the interpretation
of many format specifiers in @ref{d8b,,strftime()} and @ref{f8b,,strptime()}.

@item Module @ref{13,,calendar}

General calendar-related functions.   @ref{1dbb,,timegm()} is the
inverse of @ref{d65,,gmtime()} from this module.
@end table


@node argparse — Parser for command-line options arguments and sub-commands,getopt — C-style parser for command line options,time — Time access and conversions,Generic Operating System Services
@anchor{library/argparse doc}@anchor{24c6}@anchor{library/argparse argparse-parser-for-command-line-options-arguments-and-sub-commands}@anchor{24c7}@anchor{library/argparse module-argparse}@anchor{5}
@subsection @code{argparse} — Parser for command-line options, arguments and sub-commands


@geindex module; argparse


New in version 3.2.


@strong{Source code:} Lib/argparse.py@footnote{https://github.com/python/cpython/tree/main/Lib/argparse.py}


__________________________________________________________________


@subsubheading Tutorial



This page contains the API reference information. For a more gentle
introduction to Python command-line parsing, have a look at the
@ref{24c8,,argparse tutorial}.

The @ref{5,,argparse} module makes it easy to write user-friendly command-line
interfaces. The program defines what arguments it requires, and @ref{5,,argparse}
will figure out how to parse those out of @ref{e15,,sys.argv}.  The @ref{5,,argparse}
module also automatically generates help and usage messages and issues errors
when users give the program invalid arguments.

@menu
* Example: Example<7>. 
* ArgumentParser objects:: 
* The add_argument() method: The add_argument method. 
* The parse_args() method: The parse_args method. 
* Other utilities:: 
* Upgrading optparse code:: 

@end menu

@node Example<7>,ArgumentParser objects,,argparse — Parser for command-line options arguments and sub-commands
@anchor{library/argparse example}@anchor{24c9}
@subsubsection Example


The following code is a Python program that takes a list of integers and
produces either the sum or the max:

@example
import argparse

parser = argparse.ArgumentParser(description='Process some integers.')
parser.add_argument('integers', metavar='N', type=int, nargs='+',
                    help='an integer for the accumulator')
parser.add_argument('--sum', dest='accumulate', action='store_const',
                    const=sum, default=max,
                    help='sum the integers (default: find the max)')

args = parser.parse_args()
print(args.accumulate(args.integers))
@end example

Assuming the Python code above is saved into a file called @code{prog.py}, it can
be run at the command line and provides useful help messages:

@example
$ python prog.py -h
usage: prog.py [-h] [--sum] N [N ...]

Process some integers.

positional arguments:
 N           an integer for the accumulator

options:
 -h, --help  show this help message and exit
 --sum       sum the integers (default: find the max)
@end example

When run with the appropriate arguments, it prints either the sum or the max of
the command-line integers:

@example
$ python prog.py 1 2 3 4
4

$ python prog.py 1 2 3 4 --sum
10
@end example

If invalid arguments are passed in, it will issue an error:

@example
$ python prog.py a b c
usage: prog.py [-h] [--sum] N [N ...]
prog.py: error: argument N: invalid int value: 'a'
@end example

The following sections walk you through this example.

@menu
* Creating a parser:: 
* Adding arguments:: 
* Parsing arguments:: 

@end menu

@node Creating a parser,Adding arguments,,Example<7>
@anchor{library/argparse creating-a-parser}@anchor{24ca}
@subsubsection Creating a parser


The first step in using the @ref{5,,argparse} is creating an
@ref{91c,,ArgumentParser} object:

@example
>>> parser = argparse.ArgumentParser(description='Process some integers.')
@end example

The @ref{91c,,ArgumentParser} object will hold all the information necessary to
parse the command line into Python data types.

@node Adding arguments,Parsing arguments,Creating a parser,Example<7>
@anchor{library/argparse adding-arguments}@anchor{24cb}
@subsubsection Adding arguments


Filling an @ref{91c,,ArgumentParser} with information about program arguments is
done by making calls to the @ref{24cc,,add_argument()} method.
Generally, these calls tell the @ref{91c,,ArgumentParser} how to take the strings
on the command line and turn them into objects.  This information is stored and
used when @ref{24cd,,parse_args()} is called. For example:

@example
>>> parser.add_argument('integers', metavar='N', type=int, nargs='+',
...                     help='an integer for the accumulator')
>>> parser.add_argument('--sum', dest='accumulate', action='store_const',
...                     const=sum, default=max,
...                     help='sum the integers (default: find the max)')
@end example

Later, calling @ref{24cd,,parse_args()} will return an object with
two attributes, @code{integers} and @code{accumulate}.  The @code{integers} attribute
will be a list of one or more ints, and the @code{accumulate} attribute will be
either the @ref{50d,,sum()} function, if @code{--sum} was specified at the command line,
or the @ref{a7e,,max()} function if it was not.

@node Parsing arguments,,Adding arguments,Example<7>
@anchor{library/argparse parsing-arguments}@anchor{24ce}
@subsubsection Parsing arguments


@ref{91c,,ArgumentParser} parses arguments through the
@ref{24cd,,parse_args()} method.  This will inspect the command line,
convert each argument to the appropriate type and then invoke the appropriate action.
In most cases, this means a simple @ref{24cf,,Namespace} object will be built up from
attributes parsed out of the command line:

@example
>>> parser.parse_args(['--sum', '7', '-1', '42'])
Namespace(accumulate=<built-in function sum>, integers=[7, -1, 42])
@end example

In a script, @ref{24cd,,parse_args()} will typically be called with no
arguments, and the @ref{91c,,ArgumentParser} will automatically determine the
command-line arguments from @ref{e15,,sys.argv}.

@node ArgumentParser objects,The add_argument method,Example<7>,argparse — Parser for command-line options arguments and sub-commands
@anchor{library/argparse argumentparser-objects}@anchor{24d0}
@subsubsection ArgumentParser objects


@geindex ArgumentParser (class in argparse)
@anchor{library/argparse argparse ArgumentParser}@anchor{91c}
@deffn {Class} argparse.ArgumentParser (prog=None, usage=None, description=None, epilog=None, parents=[], formatter_class=argparse.HelpFormatter, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True, allow_abbrev=True, exit_on_error=True)

Create a new @ref{91c,,ArgumentParser} object. All parameters should be passed
as keyword arguments. Each parameter has its own more detailed description
below, but in short they are:


@itemize *

@item 
@ref{24d1,,prog} - The name of the program (default:
@code{os.path.basename(sys.argv[0])})

@item 
@ref{24d2,,usage} - The string describing the program usage (default: generated from
arguments added to parser)

@item 
@ref{24d3,,description} - Text to display before the argument help (default: none)

@item 
@ref{24d4,,epilog} - Text to display after the argument help (default: none)

@item 
@ref{24d5,,parents} - A list of @ref{91c,,ArgumentParser} objects whose arguments should
also be included

@item 
@ref{24d6,,formatter_class} - A class for customizing the help output

@item 
@ref{24d7,,prefix_chars} - The set of characters that prefix optional arguments
(default: ‘-‘)

@item 
@ref{24d8,,fromfile_prefix_chars} - The set of characters that prefix files from
which additional arguments should be read (default: @code{None})

@item 
@ref{24d9,,argument_default} - The global default value for arguments
(default: @code{None})

@item 
@ref{24da,,conflict_handler} - The strategy for resolving conflicting optionals
(usually unnecessary)

@item 
@ref{24db,,add_help} - Add a @code{-h/--help} option to the parser (default: @code{True})

@item 
@ref{91e,,allow_abbrev} - Allows long options to be abbreviated if the
abbreviation is unambiguous. (default: @code{True})

@item 
@ref{24dc,,exit_on_error} - Determines whether or not ArgumentParser exits with
error info when an error occurs. (default: @code{True})
@end itemize


Changed in version 3.5: @emph{allow_abbrev} parameter was added.



Changed in version 3.8: In previous versions, @emph{allow_abbrev} also disabled grouping of short
flags such as @code{-vv} to mean @code{-v -v}.



Changed in version 3.9: @emph{exit_on_error} parameter was added.

@end deffn

The following sections describe how each of these are used.

@menu
* prog:: 
* usage:: 
* description:: 
* epilog:: 
* parents:: 
* formatter_class:: 
* prefix_chars:: 
* fromfile_prefix_chars:: 
* argument_default:: 
* allow_abbrev:: 
* conflict_handler:: 
* add_help:: 
* exit_on_error:: 

@end menu

@node prog,usage,,ArgumentParser objects
@anchor{library/argparse prog}@anchor{24d1}
@subsubsection prog


By default, @ref{91c,,ArgumentParser} objects use @code{sys.argv[0]} to determine
how to display the name of the program in help messages.  This default is almost
always desirable because it will make the help messages match how the program was
invoked on the command line.  For example, consider a file named
@code{myprogram.py} with the following code:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo', help='foo help')
args = parser.parse_args()
@end example

The help for this program will display @code{myprogram.py} as the program name
(regardless of where the program was invoked from):

@example
$ python myprogram.py --help
usage: myprogram.py [-h] [--foo FOO]

options:
 -h, --help  show this help message and exit
 --foo FOO   foo help
$ cd ..
$ python subdir/myprogram.py --help
usage: myprogram.py [-h] [--foo FOO]

options:
 -h, --help  show this help message and exit
 --foo FOO   foo help
@end example

To change this default behavior, another value can be supplied using the
@code{prog=} argument to @ref{91c,,ArgumentParser}:

@example
>>> parser = argparse.ArgumentParser(prog='myprogram')
>>> parser.print_help()
usage: myprogram [-h]

options:
 -h, --help  show this help message and exit
@end example

Note that the program name, whether determined from @code{sys.argv[0]} or from the
@code{prog=} argument, is available to help messages using the @code{%(prog)s} format
specifier.

@example
>>> parser = argparse.ArgumentParser(prog='myprogram')
>>> parser.add_argument('--foo', help='foo of the %(prog)s program')
>>> parser.print_help()
usage: myprogram [-h] [--foo FOO]

options:
 -h, --help  show this help message and exit
 --foo FOO   foo of the myprogram program
@end example

@node usage,description,prog,ArgumentParser objects
@anchor{library/argparse usage}@anchor{24d2}
@subsubsection usage


By default, @ref{91c,,ArgumentParser} calculates the usage message from the
arguments it contains:

@example
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--foo', nargs='?', help='foo help')
>>> parser.add_argument('bar', nargs='+', help='bar help')
>>> parser.print_help()
usage: PROG [-h] [--foo [FOO]] bar [bar ...]

positional arguments:
 bar          bar help

options:
 -h, --help   show this help message and exit
 --foo [FOO]  foo help
@end example

The default message can be overridden with the @code{usage=} keyword argument:

@example
>>> parser = argparse.ArgumentParser(prog='PROG', usage='%(prog)s [options]')
>>> parser.add_argument('--foo', nargs='?', help='foo help')
>>> parser.add_argument('bar', nargs='+', help='bar help')
>>> parser.print_help()
usage: PROG [options]

positional arguments:
 bar          bar help

options:
 -h, --help   show this help message and exit
 --foo [FOO]  foo help
@end example

The @code{%(prog)s} format specifier is available to fill in the program name in
your usage messages.

@node description,epilog,usage,ArgumentParser objects
@anchor{library/argparse description}@anchor{24d3}
@subsubsection description


Most calls to the @ref{91c,,ArgumentParser} constructor will use the
@code{description=} keyword argument.  This argument gives a brief description of
what the program does and how it works.  In help messages, the description is
displayed between the command-line usage string and the help messages for the
various arguments:

@example
>>> parser = argparse.ArgumentParser(description='A foo that bars')
>>> parser.print_help()
usage: argparse.py [-h]

A foo that bars

options:
 -h, --help  show this help message and exit
@end example

By default, the description will be line-wrapped so that it fits within the
given space.  To change this behavior, see the @ref{24d6,,formatter_class} argument.

@node epilog,parents,description,ArgumentParser objects
@anchor{library/argparse epilog}@anchor{24d4}
@subsubsection epilog


Some programs like to display additional description of the program after the
description of the arguments.  Such text can be specified using the @code{epilog=}
argument to @ref{91c,,ArgumentParser}:

@example
>>> parser = argparse.ArgumentParser(
...     description='A foo that bars',
...     epilog="And that's how you'd foo a bar")
>>> parser.print_help()
usage: argparse.py [-h]

A foo that bars

options:
 -h, --help  show this help message and exit

And that's how you'd foo a bar
@end example

As with the @ref{24d3,,description} argument, the @code{epilog=} text is by default
line-wrapped, but this behavior can be adjusted with the @ref{24d6,,formatter_class}
argument to @ref{91c,,ArgumentParser}.

@node parents,formatter_class,epilog,ArgumentParser objects
@anchor{library/argparse parents}@anchor{24d5}
@subsubsection parents


Sometimes, several parsers share a common set of arguments. Rather than
repeating the definitions of these arguments, a single parser with all the
shared arguments and passed to @code{parents=} argument to @ref{91c,,ArgumentParser}
can be used.  The @code{parents=} argument takes a list of @ref{91c,,ArgumentParser}
objects, collects all the positional and optional actions from them, and adds
these actions to the @ref{91c,,ArgumentParser} object being constructed:

@example
>>> parent_parser = argparse.ArgumentParser(add_help=False)
>>> parent_parser.add_argument('--parent', type=int)

>>> foo_parser = argparse.ArgumentParser(parents=[parent_parser])
>>> foo_parser.add_argument('foo')
>>> foo_parser.parse_args(['--parent', '2', 'XXX'])
Namespace(foo='XXX', parent=2)

>>> bar_parser = argparse.ArgumentParser(parents=[parent_parser])
>>> bar_parser.add_argument('--bar')
>>> bar_parser.parse_args(['--bar', 'YYY'])
Namespace(bar='YYY', parent=None)
@end example

Note that most parent parsers will specify @code{add_help=False}.  Otherwise, the
@ref{91c,,ArgumentParser} will see two @code{-h/--help} options (one in the parent
and one in the child) and raise an error.

@cartouche
@quotation Note 
You must fully initialize the parsers before passing them via @code{parents=}.
If you change the parent parsers after the child parser, those changes will
not be reflected in the child.
@end quotation
@end cartouche

@node formatter_class,prefix_chars,parents,ArgumentParser objects
@anchor{library/argparse formatter-class}@anchor{24d6}
@subsubsection formatter_class


@ref{91c,,ArgumentParser} objects allow the help formatting to be customized by
specifying an alternate formatting class.  Currently, there are four such
classes:

@geindex RawDescriptionHelpFormatter (class in argparse)
@geindex RawTextHelpFormatter (class in argparse)
@geindex ArgumentDefaultsHelpFormatter (class in argparse)
@geindex MetavarTypeHelpFormatter (class in argparse)
@anchor{library/argparse argparse RawDescriptionHelpFormatter}@anchor{24dd}
@deffn {Class} argparse.RawDescriptionHelpFormatter
@anchor{library/argparse argparse RawTextHelpFormatter}@anchor{24de}
@deffnx {Class} argparse.RawTextHelpFormatter
@anchor{library/argparse argparse ArgumentDefaultsHelpFormatter}@anchor{1101}
@deffnx {Class} argparse.ArgumentDefaultsHelpFormatter
@anchor{library/argparse argparse MetavarTypeHelpFormatter}@anchor{24df}
@deffnx {Class} argparse.MetavarTypeHelpFormatter
@end deffn

@ref{24dd,,RawDescriptionHelpFormatter} and @ref{24de,,RawTextHelpFormatter} give
more control over how textual descriptions are displayed.
By default, @ref{91c,,ArgumentParser} objects line-wrap the @ref{24d3,,description} and
@ref{24d4,,epilog} texts in command-line help messages:

@example
>>> parser = argparse.ArgumentParser(
...     prog='PROG',
...     description='@w{'}'this description
...         was indented weird
...             but that is okay'@w{'}',
...     epilog='@w{'}'
...             likewise for this epilog whose whitespace will
...         be cleaned up and whose words will be wrapped
...         across a couple lines'@w{'}')
>>> parser.print_help()
usage: PROG [-h]

this description was indented weird but that is okay

options:
 -h, --help  show this help message and exit

likewise for this epilog whose whitespace will be cleaned up and whose words
will be wrapped across a couple lines
@end example

Passing @ref{24dd,,RawDescriptionHelpFormatter} as @code{formatter_class=}
indicates that @ref{24d3,,description} and @ref{24d4,,epilog} are already correctly formatted and
should not be line-wrapped:

@example
>>> parser = argparse.ArgumentParser(
...     prog='PROG',
...     formatter_class=argparse.RawDescriptionHelpFormatter,
...     description=textwrap.dedent('@w{'}'\
...         Please do not mess up this text!
...         --------------------------------
...             I have indented it
...             exactly the way
...             I want it
...         '@w{'}'))
>>> parser.print_help()
usage: PROG [-h]

Please do not mess up this text!
--------------------------------
   I have indented it
   exactly the way
   I want it

options:
 -h, --help  show this help message and exit
@end example

@ref{24de,,RawTextHelpFormatter} maintains whitespace for all sorts of help text,
including argument descriptions. However, multiple new lines are replaced with
one. If you wish to preserve multiple blank lines, add spaces between the
newlines.

@ref{1101,,ArgumentDefaultsHelpFormatter} automatically adds information about
default values to each of the argument help messages:

@example
>>> parser = argparse.ArgumentParser(
...     prog='PROG',
...     formatter_class=argparse.ArgumentDefaultsHelpFormatter)
>>> parser.add_argument('--foo', type=int, default=42, help='FOO!')
>>> parser.add_argument('bar', nargs='*', default=[1, 2, 3], help='BAR!')
>>> parser.print_help()
usage: PROG [-h] [--foo FOO] [bar ...]

positional arguments:
 bar         BAR! (default: [1, 2, 3])

options:
 -h, --help  show this help message and exit
 --foo FOO   FOO! (default: 42)
@end example

@ref{24df,,MetavarTypeHelpFormatter} uses the name of the @ref{24e0,,type} argument for each
argument as the display name for its values (rather than using the @ref{24e1,,dest}
as the regular formatter does):

@example
>>> parser = argparse.ArgumentParser(
...     prog='PROG',
...     formatter_class=argparse.MetavarTypeHelpFormatter)
>>> parser.add_argument('--foo', type=int)
>>> parser.add_argument('bar', type=float)
>>> parser.print_help()
usage: PROG [-h] [--foo int] float

positional arguments:
  float

options:
  -h, --help  show this help message and exit
  --foo int
@end example

@node prefix_chars,fromfile_prefix_chars,formatter_class,ArgumentParser objects
@anchor{library/argparse prefix-chars}@anchor{24d7}
@subsubsection prefix_chars


Most command-line options will use @code{-} as the prefix, e.g. @code{-f/--foo}.
Parsers that need to support different or additional prefix
characters, e.g. for options
like @code{+f} or @code{/foo}, may specify them using the @code{prefix_chars=} argument
to the ArgumentParser constructor:

@example
>>> parser = argparse.ArgumentParser(prog='PROG', prefix_chars='-+')
>>> parser.add_argument('+f')
>>> parser.add_argument('++bar')
>>> parser.parse_args('+f X ++bar Y'.split())
Namespace(bar='Y', f='X')
@end example

The @code{prefix_chars=} argument defaults to @code{'-'}. Supplying a set of
characters that does not include @code{-} will cause @code{-f/--foo} options to be
disallowed.

@node fromfile_prefix_chars,argument_default,prefix_chars,ArgumentParser objects
@anchor{library/argparse fromfile-prefix-chars}@anchor{24d8}
@subsubsection fromfile_prefix_chars


Sometimes, for example when dealing with a particularly long argument lists, it
may make sense to keep the list of arguments in a file rather than typing it out
at the command line.  If the @code{fromfile_prefix_chars=} argument is given to the
@ref{91c,,ArgumentParser} constructor, then arguments that start with any of the
specified characters will be treated as files, and will be replaced by the
arguments they contain.  For example:

@example
>>> with open('args.txt', 'w') as fp:
...     fp.write('-f\nbar')
>>> parser = argparse.ArgumentParser(fromfile_prefix_chars='@@')
>>> parser.add_argument('-f')
>>> parser.parse_args(['-f', 'foo', '@@args.txt'])
Namespace(f='bar')
@end example

Arguments read from a file must by default be one per line (but see also
@ref{24e2,,convert_arg_line_to_args()}) and are treated as if they
were in the same place as the original file referencing argument on the command
line.  So in the example above, the expression @code{['-f', 'foo', '@@args.txt']}
is considered equivalent to the expression @code{['-f', 'foo', '-f', 'bar']}.

The @code{fromfile_prefix_chars=} argument defaults to @code{None}, meaning that
arguments will never be treated as file references.

@node argument_default,allow_abbrev,fromfile_prefix_chars,ArgumentParser objects
@anchor{library/argparse argument-default}@anchor{24d9}
@subsubsection argument_default


Generally, argument defaults are specified either by passing a default to
@ref{24cc,,add_argument()} or by calling the
@ref{24e3,,set_defaults()} methods with a specific set of name-value
pairs.  Sometimes however, it may be useful to specify a single parser-wide
default for arguments.  This can be accomplished by passing the
@code{argument_default=} keyword argument to @ref{91c,,ArgumentParser}.  For example,
to globally suppress attribute creation on @ref{24cd,,parse_args()}
calls, we supply @code{argument_default=SUPPRESS}:

@example
>>> parser = argparse.ArgumentParser(argument_default=argparse.SUPPRESS)
>>> parser.add_argument('--foo')
>>> parser.add_argument('bar', nargs='?')
>>> parser.parse_args(['--foo', '1', 'BAR'])
Namespace(bar='BAR', foo='1')
>>> parser.parse_args([])
Namespace()
@end example

@node allow_abbrev,conflict_handler,argument_default,ArgumentParser objects
@anchor{library/argparse allow-abbrev}@anchor{91e}@anchor{library/argparse id1}@anchor{24e4}
@subsubsection allow_abbrev


Normally, when you pass an argument list to the
@ref{24cd,,parse_args()} method of an @ref{91c,,ArgumentParser},
it @ref{91d,,recognizes abbreviations} of long options.

This feature can be disabled by setting @code{allow_abbrev} to @code{False}:

@example
>>> parser = argparse.ArgumentParser(prog='PROG', allow_abbrev=False)
>>> parser.add_argument('--foobar', action='store_true')
>>> parser.add_argument('--foonley', action='store_false')
>>> parser.parse_args(['--foon'])
usage: PROG [-h] [--foobar] [--foonley]
PROG: error: unrecognized arguments: --foon
@end example


New in version 3.5.


@node conflict_handler,add_help,allow_abbrev,ArgumentParser objects
@anchor{library/argparse conflict-handler}@anchor{24da}
@subsubsection conflict_handler


@ref{91c,,ArgumentParser} objects do not allow two actions with the same option
string.  By default, @ref{91c,,ArgumentParser} objects raise an exception if an
attempt is made to create an argument with an option string that is already in
use:

@example
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('-f', '--foo', help='old foo help')
>>> parser.add_argument('--foo', help='new foo help')
Traceback (most recent call last):
 ..
ArgumentError: argument --foo: conflicting option string(s): --foo
@end example

Sometimes (e.g. when using @ref{24d5,,parents}) it may be useful to simply override any
older arguments with the same option string.  To get this behavior, the value
@code{'resolve'} can be supplied to the @code{conflict_handler=} argument of
@ref{91c,,ArgumentParser}:

@example
>>> parser = argparse.ArgumentParser(prog='PROG', conflict_handler='resolve')
>>> parser.add_argument('-f', '--foo', help='old foo help')
>>> parser.add_argument('--foo', help='new foo help')
>>> parser.print_help()
usage: PROG [-h] [-f FOO] [--foo FOO]

options:
 -h, --help  show this help message and exit
 -f FOO      old foo help
 --foo FOO   new foo help
@end example

Note that @ref{91c,,ArgumentParser} objects only remove an action if all of its
option strings are overridden.  So, in the example above, the old @code{-f/--foo}
action is retained as the @code{-f} action, because only the @code{--foo} option
string was overridden.

@node add_help,exit_on_error,conflict_handler,ArgumentParser objects
@anchor{library/argparse add-help}@anchor{24db}
@subsubsection add_help


By default, ArgumentParser objects add an option which simply displays
the parser’s help message. For example, consider a file named
@code{myprogram.py} containing the following code:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo', help='foo help')
args = parser.parse_args()
@end example

If @code{-h} or @code{--help} is supplied at the command line, the ArgumentParser
help will be printed:

@example
$ python myprogram.py --help
usage: myprogram.py [-h] [--foo FOO]

options:
 -h, --help  show this help message and exit
 --foo FOO   foo help
@end example

Occasionally, it may be useful to disable the addition of this help option.
This can be achieved by passing @code{False} as the @code{add_help=} argument to
@ref{91c,,ArgumentParser}:

@example
>>> parser = argparse.ArgumentParser(prog='PROG', add_help=False)
>>> parser.add_argument('--foo', help='foo help')
>>> parser.print_help()
usage: PROG [--foo FOO]

options:
 --foo FOO  foo help
@end example

The help option is typically @code{-h/--help}. The exception to this is
if the @code{prefix_chars=} is specified and does not include @code{-}, in
which case @code{-h} and @code{--help} are not valid options.  In
this case, the first character in @code{prefix_chars} is used to prefix
the help options:

@example
>>> parser = argparse.ArgumentParser(prog='PROG', prefix_chars='+/')
>>> parser.print_help()
usage: PROG [+h]

options:
  +h, ++help  show this help message and exit
@end example

@node exit_on_error,,add_help,ArgumentParser objects
@anchor{library/argparse exit-on-error}@anchor{24dc}
@subsubsection exit_on_error


Normally, when you pass an invalid argument list to the @ref{24cd,,parse_args()}
method of an @ref{91c,,ArgumentParser}, it will exit with error info.

If the user would like to catch errors manually, the feature can be enabled by setting
@code{exit_on_error} to @code{False}:

@example
>>> parser = argparse.ArgumentParser(exit_on_error=False)
>>> parser.add_argument('--integers', type=int)
_StoreAction(option_strings=['--integers'], dest='integers', nargs=None, const=None, default=None, type=<class 'int'>, choices=None, help=None, metavar=None)
>>> try:
...     parser.parse_args('--integers a'.split())
... except argparse.ArgumentError:
...     print('Catching an argumentError')
...
Catching an argumentError
@end example


New in version 3.9.


@node The add_argument method,The parse_args method,ArgumentParser objects,argparse — Parser for command-line options arguments and sub-commands
@anchor{library/argparse the-add-argument-method}@anchor{24e5}
@subsubsection The add_argument() method


@geindex add_argument() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser add_argument}@anchor{24cc}
@deffn {Method} ArgumentParser.add_argument (name or flags...[, action][, nargs][, const][, default][, type][, choices][, required][, help][, metavar][, dest])

Define how a single command-line argument should be parsed.  Each parameter
has its own more detailed description below, but in short they are:


@itemize *

@item 
@ref{24e6,,name or flags} - Either a name or a list of option strings, e.g. @code{foo}
or @code{-f, --foo}.

@item 
@ref{24e7,,action} - The basic type of action to be taken when this argument is
encountered at the command line.

@item 
@ref{24e8,,nargs} - The number of command-line arguments that should be consumed.

@item 
@ref{24e9,,const} - A constant value required by some @ref{24e7,,action} and @ref{24e8,,nargs} selections.

@item 
@ref{24ea,,default} - The value produced if the argument is absent from the
command line and if it is absent from the namespace object.

@item 
@ref{24e0,,type} - The type to which the command-line argument should be converted.

@item 
@ref{24eb,,choices} - A container of the allowable values for the argument.

@item 
@ref{24ec,,required} - Whether or not the command-line option may be omitted
(optionals only).

@item 
@ref{24ed,,help} - A brief description of what the argument does.

@item 
@ref{24ee,,metavar} - A name for the argument in usage messages.

@item 
@ref{24e1,,dest} - The name of the attribute to be added to the object returned by
@ref{24cd,,parse_args()}.
@end itemize
@end deffn

The following sections describe how each of these are used.

@menu
* name or flags:: 
* action:: 
* nargs:: 
* const:: 
* default:: 
* type:: 
* choices:: 
* required:: 
* help:: 
* metavar:: 
* dest:: 
* Action classes:: 

@end menu

@node name or flags,action,,The add_argument method
@anchor{library/argparse name-or-flags}@anchor{24e6}
@subsubsection name or flags


The @ref{24cc,,add_argument()} method must know whether an optional
argument, like @code{-f} or @code{--foo}, or a positional argument, like a list of
filenames, is expected.  The first arguments passed to
@ref{24cc,,add_argument()} must therefore be either a series of
flags, or a simple argument name.

For example, an optional argument could be created like:

@example
>>> parser.add_argument('-f', '--foo')
@end example

while a positional argument could be created like:

@example
>>> parser.add_argument('bar')
@end example

When @ref{24cd,,parse_args()} is called, optional arguments will be
identified by the @code{-} prefix, and the remaining arguments will be assumed to
be positional:

@example
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('-f', '--foo')
>>> parser.add_argument('bar')
>>> parser.parse_args(['BAR'])
Namespace(bar='BAR', foo=None)
>>> parser.parse_args(['BAR', '--foo', 'FOO'])
Namespace(bar='BAR', foo='FOO')
>>> parser.parse_args(['--foo', 'FOO'])
usage: PROG [-h] [-f FOO] bar
PROG: error: the following arguments are required: bar
@end example

@node action,nargs,name or flags,The add_argument method
@anchor{library/argparse action}@anchor{24e7}
@subsubsection action


@ref{91c,,ArgumentParser} objects associate command-line arguments with actions.  These
actions can do just about anything with the command-line arguments associated with
them, though most actions simply add an attribute to the object returned by
@ref{24cd,,parse_args()}.  The @code{action} keyword argument specifies
how the command-line arguments should be handled. The supplied actions are:


@itemize *

@item 
@code{'store'} - This just stores the argument’s value.  This is the default
action. For example:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo')
>>> parser.parse_args('--foo 1'.split())
Namespace(foo='1')
@end example

@item 
@code{'store_const'} - This stores the value specified by the @ref{24e9,,const} keyword
argument; note that the @ref{24e9,,const} keyword argument defaults to @code{None}.  The
@code{'store_const'} action is most commonly used with optional arguments that
specify some sort of flag.  For example:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', action='store_const', const=42)
>>> parser.parse_args(['--foo'])
Namespace(foo=42)
@end example

@item 
@code{'store_true'} and @code{'store_false'} - These are special cases of
@code{'store_const'} used for storing the values @code{True} and @code{False}
respectively.  In addition, they create default values of @code{False} and
@code{True} respectively.  For example:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', action='store_true')
>>> parser.add_argument('--bar', action='store_false')
>>> parser.add_argument('--baz', action='store_false')
>>> parser.parse_args('--foo --bar'.split())
Namespace(foo=True, bar=False, baz=True)
@end example

@item 
@code{'append'} - This stores a list, and appends each argument value to the
list.  This is useful to allow an option to be specified multiple times.
Example usage:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', action='append')
>>> parser.parse_args('--foo 1 --foo 2'.split())
Namespace(foo=['1', '2'])
@end example

@item 
@code{'append_const'} - This stores a list, and appends the value specified by
the @ref{24e9,,const} keyword argument to the list; note that the @ref{24e9,,const} keyword
argument defaults to @code{None}. The @code{'append_const'} action is typically
useful when multiple arguments need to store constants to the same list. For
example:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--str', dest='types', action='append_const', const=str)
>>> parser.add_argument('--int', dest='types', action='append_const', const=int)
>>> parser.parse_args('--str --int'.split())
Namespace(types=[<class 'str'>, <class 'int'>])
@end example

@item 
@code{'count'} - This counts the number of times a keyword argument occurs. For
example, this is useful for increasing verbosity levels:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--verbose', '-v', action='count', default=0)
>>> parser.parse_args(['-vvv'])
Namespace(verbose=3)
@end example

Note, the @emph{default} will be @code{None} unless explicitly set to @emph{0}.

@item 
@code{'help'} - This prints a complete help message for all the options in the
current parser and then exits. By default a help action is automatically
added to the parser. See @ref{91c,,ArgumentParser} for details of how the
output is created.

@item 
@code{'version'} - This expects a @code{version=} keyword argument in the
@ref{24cc,,add_argument()} call, and prints version information
and exits when invoked:

@example
>>> import argparse
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--version', action='version', version='%(prog)s 2.0')
>>> parser.parse_args(['--version'])
PROG 2.0
@end example

@item 
@code{'extend'} - This stores a list, and extends each argument value to the
list.
Example usage:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument("--foo", action="extend", nargs="+", type=str)
>>> parser.parse_args(["--foo", "f1", "--foo", "f2", "f3", "f4"])
Namespace(foo=['f1', 'f2', 'f3', 'f4'])
@end example


New in version 3.8.

@end itemize

You may also specify an arbitrary action by passing an Action subclass or
other object that implements the same interface. The @code{BooleanOptionalAction}
is available in @code{argparse} and adds support for boolean actions such as
@code{--foo} and @code{--no-foo}:

@example
>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', action=argparse.BooleanOptionalAction)
>>> parser.parse_args(['--no-foo'])
Namespace(foo=False)
@end example


New in version 3.9.


The recommended way to create a custom action is to extend @ref{24ef,,Action},
overriding the @code{__call__} method and optionally the @code{__init__} and
@code{format_usage} methods.

An example of a custom action:

@example
>>> class FooAction(argparse.Action):
...     def __init__(self, option_strings, dest, nargs=None, **kwargs):
...         if nargs is not None:
...             raise ValueError("nargs not allowed")
...         super().__init__(option_strings, dest, **kwargs)
...     def __call__(self, parser, namespace, values, option_string=None):
...         print('%r %r %r' % (namespace, values, option_string))
...         setattr(namespace, self.dest, values)
...
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', action=FooAction)
>>> parser.add_argument('bar', action=FooAction)
>>> args = parser.parse_args('1 --foo 2'.split())
Namespace(bar=None, foo=None) '1' None
Namespace(bar='1', foo=None) '2' '--foo'
>>> args
Namespace(bar='1', foo='2')
@end example

For more details, see @ref{24ef,,Action}.

@node nargs,const,action,The add_argument method
@anchor{library/argparse nargs}@anchor{24e8}
@subsubsection nargs


ArgumentParser objects usually associate a single command-line argument with a
single action to be taken.  The @code{nargs} keyword argument associates a
different number of command-line arguments with a single action.  The supported
values are:


@itemize *

@item 
@code{N} (an integer).  @code{N} arguments from the command line will be gathered
together into a list.  For example:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', nargs=2)
>>> parser.add_argument('bar', nargs=1)
>>> parser.parse_args('c --foo a b'.split())
Namespace(bar=['c'], foo=['a', 'b'])
@end example

Note that @code{nargs=1} produces a list of one item.  This is different from
the default, in which the item is produced by itself.
@end itemize

@geindex ? (question mark); in argparse module


@itemize *

@item 
@code{'?'}. One argument will be consumed from the command line if possible, and
produced as a single item.  If no command-line argument is present, the value from
@ref{24ea,,default} will be produced.  Note that for optional arguments, there is an
additional case - the option string is present but not followed by a
command-line argument.  In this case the value from @ref{24e9,,const} will be produced.  Some
examples to illustrate this:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', nargs='?', const='c', default='d')
>>> parser.add_argument('bar', nargs='?', default='d')
>>> parser.parse_args(['XX', '--foo', 'YY'])
Namespace(bar='XX', foo='YY')
>>> parser.parse_args(['XX', '--foo'])
Namespace(bar='XX', foo='c')
>>> parser.parse_args([])
Namespace(bar='d', foo='d')
@end example

One of the more common uses of @code{nargs='?'} is to allow optional input and
output files:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('infile', nargs='?', type=argparse.FileType('r'),
...                     default=sys.stdin)
>>> parser.add_argument('outfile', nargs='?', type=argparse.FileType('w'),
...                     default=sys.stdout)
>>> parser.parse_args(['input.txt', 'output.txt'])
Namespace(infile=<_io.TextIOWrapper name='input.txt' encoding='UTF-8'>,
          outfile=<_io.TextIOWrapper name='output.txt' encoding='UTF-8'>)
>>> parser.parse_args([])
Namespace(infile=<_io.TextIOWrapper name='<stdin>' encoding='UTF-8'>,
          outfile=<_io.TextIOWrapper name='<stdout>' encoding='UTF-8'>)
@end example
@end itemize

@geindex * (asterisk); in argparse module


@itemize *

@item 
@code{'*'}.  All command-line arguments present are gathered into a list.  Note that
it generally doesn’t make much sense to have more than one positional argument
with @code{nargs='*'}, but multiple optional arguments with @code{nargs='*'} is
possible.  For example:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', nargs='*')
>>> parser.add_argument('--bar', nargs='*')
>>> parser.add_argument('baz', nargs='*')
>>> parser.parse_args('a b --foo x y --bar 1 2'.split())
Namespace(bar=['1', '2'], baz=['a', 'b'], foo=['x', 'y'])
@end example
@end itemize

@geindex + (plus); in argparse module


@itemize *

@item 
@code{'+'}. Just like @code{'*'}, all command-line args present are gathered into a
list.  Additionally, an error message will be generated if there wasn’t at
least one command-line argument present.  For example:

@example
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('foo', nargs='+')
>>> parser.parse_args(['a', 'b'])
Namespace(foo=['a', 'b'])
>>> parser.parse_args([])
usage: PROG [-h] foo [foo ...]
PROG: error: the following arguments are required: foo
@end example
@end itemize

If the @code{nargs} keyword argument is not provided, the number of arguments consumed
is determined by the @ref{24e7,,action}.  Generally this means a single command-line argument
will be consumed and a single item (not a list) will be produced.

@node const,default,nargs,The add_argument method
@anchor{library/argparse const}@anchor{24e9}
@subsubsection const


The @code{const} argument of @ref{24cc,,add_argument()} is used to hold
constant values that are not read from the command line but are required for
the various @ref{91c,,ArgumentParser} actions.  The two most common uses of it are:


@itemize *

@item 
When @ref{24cc,,add_argument()} is called with
@code{action='store_const'} or @code{action='append_const'}.  These actions add the
@code{const} value to one of the attributes of the object returned by
@ref{24cd,,parse_args()}. See the @ref{24e7,,action} description for examples.
If @code{const} is not provided to @ref{24cc,,add_argument()}, it will
receive a default value of @code{None}.

@item 
When @ref{24cc,,add_argument()} is called with option strings
(like @code{-f} or @code{--foo}) and @code{nargs='?'}.  This creates an optional
argument that can be followed by zero or one command-line arguments.
When parsing the command line, if the option string is encountered with no
command-line argument following it, the value of @code{const} will be assumed to
be @code{None} instead.  See the @ref{24e8,,nargs} description for examples.
@end itemize


Changed in version 3.11: @code{const=None} by default, including when @code{action='append_const'} or
@code{action='store_const'}.


@node default,type,const,The add_argument method
@anchor{library/argparse default}@anchor{24ea}
@subsubsection default


All optional arguments and some positional arguments may be omitted at the
command line.  The @code{default} keyword argument of
@ref{24cc,,add_argument()}, whose value defaults to @code{None},
specifies what value should be used if the command-line argument is not present.
For optional arguments, the @code{default} value is used when the option string
was not present at the command line:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', default=42)
>>> parser.parse_args(['--foo', '2'])
Namespace(foo='2')
>>> parser.parse_args([])
Namespace(foo=42)
@end example

If the target namespace already has an attribute set, the action @emph{default}
will not over write it:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', default=42)
>>> parser.parse_args([], namespace=argparse.Namespace(foo=101))
Namespace(foo=101)
@end example

If the @code{default} value is a string, the parser parses the value as if it
were a command-line argument.  In particular, the parser applies any @ref{24e0,,type}
conversion argument, if provided, before setting the attribute on the
@ref{24cf,,Namespace} return value.  Otherwise, the parser uses the value as is:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--length', default='10', type=int)
>>> parser.add_argument('--width', default=10.5, type=int)
>>> parser.parse_args()
Namespace(length=10, width=10.5)
@end example

For positional arguments with @ref{24e8,,nargs} equal to @code{?} or @code{*}, the @code{default} value
is used when no command-line argument was present:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('foo', nargs='?', default=42)
>>> parser.parse_args(['a'])
Namespace(foo='a')
>>> parser.parse_args([])
Namespace(foo=42)
@end example

Providing @code{default=argparse.SUPPRESS} causes no attribute to be added if the
command-line argument was not present:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', default=argparse.SUPPRESS)
>>> parser.parse_args([])
Namespace()
>>> parser.parse_args(['--foo', '1'])
Namespace(foo='1')
@end example

@node type,choices,default,The add_argument method
@anchor{library/argparse type}@anchor{24e0}
@subsubsection type


By default, the parser reads command-line arguments in as simple
strings. However, quite often the command-line string should instead be
interpreted as another type, such as a @ref{256,,float} or @ref{1b1,,int}.  The
@code{type} keyword for @ref{24cc,,add_argument()} allows any
necessary type-checking and type conversions to be performed.

If the @ref{24e0,,type} keyword is used with the @ref{24ea,,default} keyword, the type converter
is only applied if the default is a string.

The argument to @code{type} can be any callable that accepts a single string.
If the function raises @code{ArgumentTypeError}, @ref{162,,TypeError}, or
@ref{17c,,ValueError}, the exception is caught and a nicely formatted error
message is displayed.  No other exception types are handled.

Common built-in types and functions can be used as type converters:

@example
import argparse
import pathlib

parser = argparse.ArgumentParser()
parser.add_argument('count', type=int)
parser.add_argument('distance', type=float)
parser.add_argument('street', type=ascii)
parser.add_argument('code_point', type=ord)
parser.add_argument('source_file', type=open)
parser.add_argument('dest_file', type=argparse.FileType('w', encoding='latin-1'))
parser.add_argument('datapath', type=pathlib.Path)
@end example

User defined functions can be used as well:

@example
>>> def hyphenated(string):
...     return '-'.join([word[:4] for word in string.casefold().split()])
...
>>> parser = argparse.ArgumentParser()
>>> _ = parser.add_argument('short_title', type=hyphenated)
>>> parser.parse_args(['"The Tale of Two Cities"'])
Namespace(short_title='"the-tale-of-two-citi')
@end example

The @ref{2f7,,bool()} function is not recommended as a type converter.  All it does
is convert empty strings to @code{False} and non-empty strings to @code{True}.
This is usually not what is desired.

In general, the @code{type} keyword is a convenience that should only be used for
simple conversions that can only raise one of the three supported exceptions.
Anything with more interesting error-handling or resource management should be
done downstream after the arguments are parsed.

For example, JSON or YAML conversions have complex error cases that require
better reporting than can be given by the @code{type} keyword.  A
@ref{987,,JSONDecodeError} would not be well formatted and a
@code{FileNotFound} exception would not be handled at all.

Even @ref{a94,,FileType} has its limitations for use with the @code{type}
keyword.  If one argument uses @emph{FileType} and then a subsequent argument fails,
an error is reported but the file is not automatically closed.  In this case, it
would be better to wait until after the parser has run and then use the
@ref{168,,with}-statement to manage the files.

For type checkers that simply check against a fixed set of values, consider
using the @ref{24eb,,choices} keyword instead.

@node choices,required,type,The add_argument method
@anchor{library/argparse choices}@anchor{24eb}
@subsubsection choices


Some command-line arguments should be selected from a restricted set of values.
These can be handled by passing a container object as the @emph{choices} keyword
argument to @ref{24cc,,add_argument()}.  When the command line is
parsed, argument values will be checked, and an error message will be displayed
if the argument was not one of the acceptable values:

@example
>>> parser = argparse.ArgumentParser(prog='game.py')
>>> parser.add_argument('move', choices=['rock', 'paper', 'scissors'])
>>> parser.parse_args(['rock'])
Namespace(move='rock')
>>> parser.parse_args(['fire'])
usage: game.py [-h] @{rock,paper,scissors@}
game.py: error: argument move: invalid choice: 'fire' (choose from 'rock',
'paper', 'scissors')
@end example

Note that inclusion in the @emph{choices} container is checked after any @ref{24e0,,type}
conversions have been performed, so the type of the objects in the @emph{choices}
container should match the @ref{24e0,,type} specified:

@example
>>> parser = argparse.ArgumentParser(prog='doors.py')
>>> parser.add_argument('door', type=int, choices=range(1, 4))
>>> print(parser.parse_args(['3']))
Namespace(door=3)
>>> parser.parse_args(['4'])
usage: doors.py [-h] @{1,2,3@}
doors.py: error: argument door: invalid choice: 4 (choose from 1, 2, 3)
@end example

Any container can be passed as the @emph{choices} value, so @ref{43a,,list} objects,
@ref{370,,set} objects, and custom containers are all supported.

Use of @ref{658,,enum.Enum} is not recommended because it is difficult to
control its appearance in usage, help, and error messages.

Formatted choices overrides the default @emph{metavar} which is normally derived
from @emph{dest}.  This is usually what you want because the user never sees the
@emph{dest} parameter.  If this display isn’t desirable (perhaps because there are
many choices), just specify an explicit @ref{24ee,,metavar}.

@node required,help,choices,The add_argument method
@anchor{library/argparse required}@anchor{24ec}
@subsubsection required


In general, the @ref{5,,argparse} module assumes that flags like @code{-f} and @code{--bar}
indicate @emph{optional} arguments, which can always be omitted at the command line.
To make an option @emph{required}, @code{True} can be specified for the @code{required=}
keyword argument to @ref{24cc,,add_argument()}:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', required=True)
>>> parser.parse_args(['--foo', 'BAR'])
Namespace(foo='BAR')
>>> parser.parse_args([])
usage: [-h] --foo FOO
: error: the following arguments are required: --foo
@end example

As the example shows, if an option is marked as @code{required},
@ref{24cd,,parse_args()} will report an error if that option is not
present at the command line.

@cartouche
@quotation Note 
Required options are generally considered bad form because users expect
@emph{options} to be @emph{optional}, and thus they should be avoided when possible.
@end quotation
@end cartouche

@node help,metavar,required,The add_argument method
@anchor{library/argparse help}@anchor{24ed}
@subsubsection help


The @code{help} value is a string containing a brief description of the argument.
When a user requests help (usually by using @code{-h} or @code{--help} at the
command line), these @code{help} descriptions will be displayed with each
argument:

@example
>>> parser = argparse.ArgumentParser(prog='frobble')
>>> parser.add_argument('--foo', action='store_true',
...                     help='foo the bars before frobbling')
>>> parser.add_argument('bar', nargs='+',
...                     help='one of the bars to be frobbled')
>>> parser.parse_args(['-h'])
usage: frobble [-h] [--foo] bar [bar ...]

positional arguments:
 bar     one of the bars to be frobbled

options:
 -h, --help  show this help message and exit
 --foo   foo the bars before frobbling
@end example

The @code{help} strings can include various format specifiers to avoid repetition
of things like the program name or the argument @ref{24ea,,default}.  The available
specifiers include the program name, @code{%(prog)s} and most keyword arguments to
@ref{24cc,,add_argument()}, e.g. @code{%(default)s}, @code{%(type)s}, etc.:

@example
>>> parser = argparse.ArgumentParser(prog='frobble')
>>> parser.add_argument('bar', nargs='?', type=int, default=42,
...                     help='the bar to %(prog)s (default: %(default)s)')
>>> parser.print_help()
usage: frobble [-h] [bar]

positional arguments:
 bar     the bar to frobble (default: 42)

options:
 -h, --help  show this help message and exit
@end example

As the help string supports %-formatting, if you want a literal @code{%} to appear
in the help string, you must escape it as @code{%%}.

@ref{5,,argparse} supports silencing the help entry for certain options, by
setting the @code{help} value to @code{argparse.SUPPRESS}:

@example
>>> parser = argparse.ArgumentParser(prog='frobble')
>>> parser.add_argument('--foo', help=argparse.SUPPRESS)
>>> parser.print_help()
usage: frobble [-h]

options:
  -h, --help  show this help message and exit
@end example

@node metavar,dest,help,The add_argument method
@anchor{library/argparse metavar}@anchor{24ee}
@subsubsection metavar


When @ref{91c,,ArgumentParser} generates help messages, it needs some way to refer
to each expected argument.  By default, ArgumentParser objects use the @ref{24e1,,dest}
value as the “name” of each object.  By default, for positional argument
actions, the @ref{24e1,,dest} value is used directly, and for optional argument actions,
the @ref{24e1,,dest} value is uppercased.  So, a single positional argument with
@code{dest='bar'} will be referred to as @code{bar}. A single
optional argument @code{--foo} that should be followed by a single command-line argument
will be referred to as @code{FOO}.  An example:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo')
>>> parser.add_argument('bar')
>>> parser.parse_args('X --foo Y'.split())
Namespace(bar='X', foo='Y')
>>> parser.print_help()
usage:  [-h] [--foo FOO] bar

positional arguments:
 bar

options:
 -h, --help  show this help message and exit
 --foo FOO
@end example

An alternative name can be specified with @code{metavar}:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', metavar='YYY')
>>> parser.add_argument('bar', metavar='XXX')
>>> parser.parse_args('X --foo Y'.split())
Namespace(bar='X', foo='Y')
>>> parser.print_help()
usage:  [-h] [--foo YYY] XXX

positional arguments:
 XXX

options:
 -h, --help  show this help message and exit
 --foo YYY
@end example

Note that @code{metavar} only changes the @emph{displayed} name - the name of the
attribute on the @ref{24cd,,parse_args()} object is still determined
by the @ref{24e1,,dest} value.

Different values of @code{nargs} may cause the metavar to be used multiple times.
Providing a tuple to @code{metavar} specifies a different display for each of the
arguments:

@example
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('-x', nargs=2)
>>> parser.add_argument('--foo', nargs=2, metavar=('bar', 'baz'))
>>> parser.print_help()
usage: PROG [-h] [-x X X] [--foo bar baz]

options:
 -h, --help     show this help message and exit
 -x X X
 --foo bar baz
@end example

@node dest,Action classes,metavar,The add_argument method
@anchor{library/argparse dest}@anchor{24e1}
@subsubsection dest


Most @ref{91c,,ArgumentParser} actions add some value as an attribute of the
object returned by @ref{24cd,,parse_args()}.  The name of this
attribute is determined by the @code{dest} keyword argument of
@ref{24cc,,add_argument()}.  For positional argument actions,
@code{dest} is normally supplied as the first argument to
@ref{24cc,,add_argument()}:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('bar')
>>> parser.parse_args(['XXX'])
Namespace(bar='XXX')
@end example

For optional argument actions, the value of @code{dest} is normally inferred from
the option strings.  @ref{91c,,ArgumentParser} generates the value of @code{dest} by
taking the first long option string and stripping away the initial @code{--}
string.  If no long option strings were supplied, @code{dest} will be derived from
the first short option string by stripping the initial @code{-} character.  Any
internal @code{-} characters will be converted to @code{_} characters to make sure
the string is a valid attribute name.  The examples below illustrate this
behavior:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('-f', '--foo-bar', '--foo')
>>> parser.add_argument('-x', '-y')
>>> parser.parse_args('-f 1 -x 2'.split())
Namespace(foo_bar='1', x='2')
>>> parser.parse_args('--foo 1 -y 2'.split())
Namespace(foo_bar='1', x='2')
@end example

@code{dest} allows a custom attribute name to be provided:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', dest='bar')
>>> parser.parse_args('--foo XXX'.split())
Namespace(bar='XXX')
@end example

@node Action classes,,dest,The add_argument method
@anchor{library/argparse action-classes}@anchor{24f0}
@subsubsection Action classes


Action classes implement the Action API, a callable which returns a callable
which processes arguments from the command-line. Any object which follows
this API may be passed as the @code{action} parameter to
@code{add_argument()}.

@geindex Action (class in argparse)
@anchor{library/argparse argparse Action}@anchor{24ef}
@deffn {Class} argparse.Action (option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)
@end deffn

Action objects are used by an ArgumentParser to represent the information
needed to parse a single argument from one or more strings from the
command line. The Action class must accept the two positional arguments
plus any keyword arguments passed to @ref{24cc,,ArgumentParser.add_argument()}
except for the @code{action} itself.

Instances of Action (or return value of any callable to the @code{action}
parameter) should have attributes “dest”, “option_strings”, “default”, “type”,
“required”, “help”, etc. defined. The easiest way to ensure these attributes
are defined is to call @code{Action.__init__}.

Action instances should be callable, so subclasses must override the
@code{__call__} method, which should accept four parameters:


@itemize *

@item 
@code{parser} - The ArgumentParser object which contains this action.

@item 
@code{namespace} - The @ref{24cf,,Namespace} object that will be returned by
@ref{24cd,,parse_args()}.  Most actions add an attribute to this
object using @ref{1a37,,setattr()}.

@item 
@code{values} - The associated command-line arguments, with any type conversions
applied.  Type conversions are specified with the @ref{24e0,,type} keyword argument to
@ref{24cc,,add_argument()}.

@item 
@code{option_string} - The option string that was used to invoke this action.
The @code{option_string} argument is optional, and will be absent if the action
is associated with a positional argument.
@end itemize

The @code{__call__} method may perform arbitrary actions, but will typically set
attributes on the @code{namespace} based on @code{dest} and @code{values}.

Action subclasses can define a @code{format_usage} method that takes no argument
and return a string which will be used when printing the usage of the program.
If such method is not provided, a sensible default will be used.

@node The parse_args method,Other utilities,The add_argument method,argparse — Parser for command-line options arguments and sub-commands
@anchor{library/argparse the-parse-args-method}@anchor{24f1}
@subsubsection The parse_args() method


@geindex parse_args() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser parse_args}@anchor{24cd}
@deffn {Method} ArgumentParser.parse_args (args=None, namespace=None)

Convert argument strings to objects and assign them as attributes of the
namespace.  Return the populated namespace.

Previous calls to @ref{24cc,,add_argument()} determine exactly what objects are
created and how they are assigned. See the documentation for
@ref{24cc,,add_argument()} for details.


@itemize *

@item 
@ref{24f2,,args} - List of strings to parse.  The default is taken from
@ref{e15,,sys.argv}.

@item 
@ref{24f3,,namespace} - An object to take the attributes.  The default is a new empty
@ref{24cf,,Namespace} object.
@end itemize
@end deffn

@menu
* Option value syntax:: 
* Invalid arguments:: 
* Arguments containing -:: 
* Argument abbreviations (prefix matching): Argument abbreviations prefix matching. 
* Beyond sys.argv: Beyond sys argv. 
* The Namespace object:: 

@end menu

@node Option value syntax,Invalid arguments,,The parse_args method
@anchor{library/argparse option-value-syntax}@anchor{24f4}
@subsubsection Option value syntax


The @ref{24cd,,parse_args()} method supports several ways of
specifying the value of an option (if it takes one).  In the simplest case, the
option and its value are passed as two separate arguments:

@example
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('-x')
>>> parser.add_argument('--foo')
>>> parser.parse_args(['-x', 'X'])
Namespace(foo=None, x='X')
>>> parser.parse_args(['--foo', 'FOO'])
Namespace(foo='FOO', x=None)
@end example

For long options (options with names longer than a single character), the option
and value can also be passed as a single command-line argument, using @code{=} to
separate them:

@example
>>> parser.parse_args(['--foo=FOO'])
Namespace(foo='FOO', x=None)
@end example

For short options (options only one character long), the option and its value
can be concatenated:

@example
>>> parser.parse_args(['-xX'])
Namespace(foo=None, x='X')
@end example

Several short options can be joined together, using only a single @code{-} prefix,
as long as only the last option (or none of them) requires a value:

@example
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('-x', action='store_true')
>>> parser.add_argument('-y', action='store_true')
>>> parser.add_argument('-z')
>>> parser.parse_args(['-xyzZ'])
Namespace(x=True, y=True, z='Z')
@end example

@node Invalid arguments,Arguments containing -,Option value syntax,The parse_args method
@anchor{library/argparse invalid-arguments}@anchor{24f5}
@subsubsection Invalid arguments


While parsing the command line, @ref{24cd,,parse_args()} checks for a
variety of errors, including ambiguous options, invalid types, invalid options,
wrong number of positional arguments, etc.  When it encounters such an error,
it exits and prints the error along with a usage message:

@example
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--foo', type=int)
>>> parser.add_argument('bar', nargs='?')

>>> # invalid type
>>> parser.parse_args(['--foo', 'spam'])
usage: PROG [-h] [--foo FOO] [bar]
PROG: error: argument --foo: invalid int value: 'spam'

>>> # invalid option
>>> parser.parse_args(['--bar'])
usage: PROG [-h] [--foo FOO] [bar]
PROG: error: no such option: --bar

>>> # wrong number of arguments
>>> parser.parse_args(['spam', 'badger'])
usage: PROG [-h] [--foo FOO] [bar]
PROG: error: extra arguments found: badger
@end example

@node Arguments containing -,Argument abbreviations prefix matching,Invalid arguments,The parse_args method
@anchor{library/argparse arguments-containing}@anchor{24f6}
@subsubsection Arguments containing @code{-}


The @ref{24cd,,parse_args()} method attempts to give errors whenever
the user has clearly made a mistake, but some situations are inherently
ambiguous.  For example, the command-line argument @code{-1} could either be an
attempt to specify an option or an attempt to provide a positional argument.
The @ref{24cd,,parse_args()} method is cautious here: positional
arguments may only begin with @code{-} if they look like negative numbers and
there are no options in the parser that look like negative numbers:

@example
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('-x')
>>> parser.add_argument('foo', nargs='?')

>>> # no negative number options, so -1 is a positional argument
>>> parser.parse_args(['-x', '-1'])
Namespace(foo=None, x='-1')

>>> # no negative number options, so -1 and -5 are positional arguments
>>> parser.parse_args(['-x', '-1', '-5'])
Namespace(foo='-5', x='-1')

>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('-1', dest='one')
>>> parser.add_argument('foo', nargs='?')

>>> # negative number options present, so -1 is an option
>>> parser.parse_args(['-1', 'X'])
Namespace(foo=None, one='X')

>>> # negative number options present, so -2 is an option
>>> parser.parse_args(['-2'])
usage: PROG [-h] [-1 ONE] [foo]
PROG: error: no such option: -2

>>> # negative number options present, so both -1s are options
>>> parser.parse_args(['-1', '-1'])
usage: PROG [-h] [-1 ONE] [foo]
PROG: error: argument -1: expected one argument
@end example

If you have positional arguments that must begin with @code{-} and don’t look
like negative numbers, you can insert the pseudo-argument @code{'--'} which tells
@ref{24cd,,parse_args()} that everything after that is a positional
argument:

@example
>>> parser.parse_args(['--', '-f'])
Namespace(foo='-f', one=None)
@end example

@node Argument abbreviations prefix matching,Beyond sys argv,Arguments containing -,The parse_args method
@anchor{library/argparse argument-abbreviations-prefix-matching}@anchor{24f7}@anchor{library/argparse prefix-matching}@anchor{91d}
@subsubsection Argument abbreviations (prefix matching)


The @ref{24cd,,parse_args()} method @ref{91e,,by default}
allows long options to be abbreviated to a prefix, if the abbreviation is
unambiguous (the prefix matches a unique option):

@example
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('-bacon')
>>> parser.add_argument('-badger')
>>> parser.parse_args('-bac MMM'.split())
Namespace(bacon='MMM', badger=None)
>>> parser.parse_args('-bad WOOD'.split())
Namespace(bacon=None, badger='WOOD')
>>> parser.parse_args('-ba BA'.split())
usage: PROG [-h] [-bacon BACON] [-badger BADGER]
PROG: error: ambiguous option: -ba could match -badger, -bacon
@end example

An error is produced for arguments that could produce more than one options.
This feature can be disabled by setting @ref{91e,,allow_abbrev} to @code{False}.

@node Beyond sys argv,The Namespace object,Argument abbreviations prefix matching,The parse_args method
@anchor{library/argparse args}@anchor{24f2}@anchor{library/argparse beyond-sys-argv}@anchor{24f8}
@subsubsection Beyond @code{sys.argv}


Sometimes it may be useful to have an ArgumentParser parse arguments other than those
of @ref{e15,,sys.argv}.  This can be accomplished by passing a list of strings to
@ref{24cd,,parse_args()}.  This is useful for testing at the
interactive prompt:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument(
...     'integers', metavar='int', type=int, choices=range(10),
...     nargs='+', help='an integer in the range 0..9')
>>> parser.add_argument(
...     '--sum', dest='accumulate', action='store_const', const=sum,
...     default=max, help='sum the integers (default: find the max)')
>>> parser.parse_args(['1', '2', '3', '4'])
Namespace(accumulate=<built-in function max>, integers=[1, 2, 3, 4])
>>> parser.parse_args(['1', '2', '3', '4', '--sum'])
Namespace(accumulate=<built-in function sum>, integers=[1, 2, 3, 4])
@end example

@node The Namespace object,,Beyond sys argv,The parse_args method
@anchor{library/argparse namespace}@anchor{24f3}@anchor{library/argparse the-namespace-object}@anchor{24f9}
@subsubsection The Namespace object


@geindex Namespace (class in argparse)
@anchor{library/argparse argparse Namespace}@anchor{24cf}
@deffn {Class} argparse.Namespace

Simple class used by default by @ref{24cd,,parse_args()} to create
an object holding attributes and return it.
@end deffn

This class is deliberately simple, just an @ref{5a5,,object} subclass with a
readable string representation. If you prefer to have dict-like view of the
attributes, you can use the standard Python idiom, @ref{12fc,,vars()}:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo')
>>> args = parser.parse_args(['--foo', 'BAR'])
>>> vars(args)
@{'foo': 'BAR'@}
@end example

It may also be useful to have an @ref{91c,,ArgumentParser} assign attributes to an
already existing object, rather than a new @ref{24cf,,Namespace} object.  This can
be achieved by specifying the @code{namespace=} keyword argument:

@example
>>> class C:
...     pass
...
>>> c = C()
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo')
>>> parser.parse_args(args=['--foo', 'BAR'], namespace=c)
>>> c.foo
'BAR'
@end example

@node Other utilities,Upgrading optparse code,The parse_args method,argparse — Parser for command-line options arguments and sub-commands
@anchor{library/argparse other-utilities}@anchor{24fa}
@subsubsection Other utilities


@menu
* Sub-commands:: 
* FileType objects:: 
* Argument groups:: 
* Mutual exclusion:: 
* Parser defaults:: 
* Printing help:: 
* Partial parsing:: 
* Customizing file parsing:: 
* Exiting methods:: 
* Intermixed parsing:: 

@end menu

@node Sub-commands,FileType objects,,Other utilities
@anchor{library/argparse sub-commands}@anchor{24fb}
@subsubsection Sub-commands


@geindex add_subparsers() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser add_subparsers}@anchor{753}
@deffn {Method} ArgumentParser.add_subparsers ([title][, description][, prog][, parser_class][, action][, option_string][, dest][, required][, help][, metavar])

Many programs split up their functionality into a number of sub-commands,
for example, the @code{svn} program can invoke sub-commands like @code{svn
checkout}, @code{svn update}, and @code{svn commit}.  Splitting up functionality
this way can be a particularly good idea when a program performs several
different functions which require different kinds of command-line arguments.
@ref{91c,,ArgumentParser} supports the creation of such sub-commands with the
@ref{753,,add_subparsers()} method.  The @ref{753,,add_subparsers()} method is normally
called with no arguments and returns a special action object.  This object
has a single method, @code{add_parser()}, which takes a
command name and any @ref{91c,,ArgumentParser} constructor arguments, and
returns an @ref{91c,,ArgumentParser} object that can be modified as usual.

Description of parameters:


@itemize *

@item 
title - title for the sub-parser group in help output; by default
“subcommands” if description is provided, otherwise uses title for
positional arguments

@item 
description - description for the sub-parser group in help output, by
default @code{None}

@item 
prog - usage information that will be displayed with sub-command help,
by default the name of the program and any positional arguments before the
subparser argument

@item 
parser_class - class which will be used to create sub-parser instances, by
default the class of the current parser (e.g. ArgumentParser)

@item 
@ref{24e7,,action} - the basic type of action to be taken when this argument is
encountered at the command line

@item 
@ref{24e1,,dest} - name of the attribute under which sub-command name will be
stored; by default @code{None} and no value is stored

@item 
@ref{24ec,,required} - Whether or not a subcommand must be provided, by default
@code{False} (added in 3.7)

@item 
@ref{24ed,,help} - help for sub-parser group in help output, by default @code{None}

@item 
@ref{24ee,,metavar} - string presenting available sub-commands in help; by default it
is @code{None} and presents sub-commands in form @{cmd1, cmd2, ..@}
@end itemize

Some example usage:

@example
>>> # create the top-level parser
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--foo', action='store_true', help='foo help')
>>> subparsers = parser.add_subparsers(help='sub-command help')
>>>
>>> # create the parser for the "a" command
>>> parser_a = subparsers.add_parser('a', help='a help')
>>> parser_a.add_argument('bar', type=int, help='bar help')
>>>
>>> # create the parser for the "b" command
>>> parser_b = subparsers.add_parser('b', help='b help')
>>> parser_b.add_argument('--baz', choices='XYZ', help='baz help')
>>>
>>> # parse some argument lists
>>> parser.parse_args(['a', '12'])
Namespace(bar=12, foo=False)
>>> parser.parse_args(['--foo', 'b', '--baz', 'Z'])
Namespace(baz='Z', foo=True)
@end example

Note that the object returned by @ref{24cd,,parse_args()} will only contain
attributes for the main parser and the subparser that was selected by the
command line (and not any other subparsers).  So in the example above, when
the @code{a} command is specified, only the @code{foo} and @code{bar} attributes are
present, and when the @code{b} command is specified, only the @code{foo} and
@code{baz} attributes are present.

Similarly, when a help message is requested from a subparser, only the help
for that particular parser will be printed.  The help message will not
include parent parser or sibling parser messages.  (A help message for each
subparser command, however, can be given by supplying the @code{help=} argument
to @code{add_parser()} as above.)

@example
>>> parser.parse_args(['--help'])
usage: PROG [-h] [--foo] @{a,b@} ...

positional arguments:
  @{a,b@}   sub-command help
    a     a help
    b     b help

options:
  -h, --help  show this help message and exit
  --foo   foo help

>>> parser.parse_args(['a', '--help'])
usage: PROG a [-h] bar

positional arguments:
  bar     bar help

options:
  -h, --help  show this help message and exit

>>> parser.parse_args(['b', '--help'])
usage: PROG b [-h] [--baz @{X,Y,Z@}]

options:
  -h, --help     show this help message and exit
  --baz @{X,Y,Z@}  baz help
@end example

The @ref{753,,add_subparsers()} method also supports @code{title} and @code{description}
keyword arguments.  When either is present, the subparser’s commands will
appear in their own group in the help output.  For example:

@example
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers(title='subcommands',
...                                    description='valid subcommands',
...                                    help='additional help')
>>> subparsers.add_parser('foo')
>>> subparsers.add_parser('bar')
>>> parser.parse_args(['-h'])
usage:  [-h] @{foo,bar@} ...

options:
  -h, --help  show this help message and exit

subcommands:
  valid subcommands

  @{foo,bar@}   additional help
@end example

Furthermore, @code{add_parser} supports an additional @code{aliases} argument,
which allows multiple strings to refer to the same subparser. This example,
like @code{svn}, aliases @code{co} as a shorthand for @code{checkout}:

@example
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers()
>>> checkout = subparsers.add_parser('checkout', aliases=['co'])
>>> checkout.add_argument('foo')
>>> parser.parse_args(['co', 'bar'])
Namespace(foo='bar')
@end example

One particularly effective way of handling sub-commands is to combine the use
of the @ref{753,,add_subparsers()} method with calls to @ref{24e3,,set_defaults()} so
that each subparser knows which Python function it should execute.  For
example:

@example
>>> # sub-command functions
>>> def foo(args):
...     print(args.x * args.y)
...
>>> def bar(args):
...     print('((%s))' % args.z)
...
>>> # create the top-level parser
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers()
>>>
>>> # create the parser for the "foo" command
>>> parser_foo = subparsers.add_parser('foo')
>>> parser_foo.add_argument('-x', type=int, default=1)
>>> parser_foo.add_argument('y', type=float)
>>> parser_foo.set_defaults(func=foo)
>>>
>>> # create the parser for the "bar" command
>>> parser_bar = subparsers.add_parser('bar')
>>> parser_bar.add_argument('z')
>>> parser_bar.set_defaults(func=bar)
>>>
>>> # parse the args and call whatever function was selected
>>> args = parser.parse_args('foo 1 -x 2'.split())
>>> args.func(args)
2.0
>>>
>>> # parse the args and call whatever function was selected
>>> args = parser.parse_args('bar XYZYX'.split())
>>> args.func(args)
((XYZYX))
@end example

This way, you can let @ref{24cd,,parse_args()} do the job of calling the
appropriate function after argument parsing is complete.  Associating
functions with actions like this is typically the easiest way to handle the
different actions for each of your subparsers.  However, if it is necessary
to check the name of the subparser that was invoked, the @code{dest} keyword
argument to the @ref{753,,add_subparsers()} call will work:

@example
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers(dest='subparser_name')
>>> subparser1 = subparsers.add_parser('1')
>>> subparser1.add_argument('-x')
>>> subparser2 = subparsers.add_parser('2')
>>> subparser2.add_argument('y')
>>> parser.parse_args(['2', 'frobble'])
Namespace(subparser_name='2', y='frobble')
@end example


Changed in version 3.7: New @emph{required} keyword argument.

@end deffn

@node FileType objects,Argument groups,Sub-commands,Other utilities
@anchor{library/argparse filetype-objects}@anchor{24fc}
@subsubsection FileType objects


@geindex FileType (class in argparse)
@anchor{library/argparse argparse FileType}@anchor{a94}
@deffn {Class} argparse.FileType (mode='r', bufsize=- 1, encoding=None, errors=None)

The @ref{a94,,FileType} factory creates objects that can be passed to the type
argument of @ref{24cc,,ArgumentParser.add_argument()}.  Arguments that have
@ref{a94,,FileType} objects as their type will open command-line arguments as
files with the requested modes, buffer sizes, encodings and error handling
(see the @ref{1c8,,open()} function for more details):

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--raw', type=argparse.FileType('wb', 0))
>>> parser.add_argument('out', type=argparse.FileType('w', encoding='UTF-8'))
>>> parser.parse_args(['--raw', 'raw.dat', 'file.txt'])
Namespace(out=<_io.TextIOWrapper name='file.txt' mode='w' encoding='UTF-8'>, raw=<_io.FileIO name='raw.dat' mode='wb'>)
@end example

FileType objects understand the pseudo-argument @code{'-'} and automatically
convert this into @code{sys.stdin} for readable @ref{a94,,FileType} objects and
@code{sys.stdout} for writable @ref{a94,,FileType} objects:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('infile', type=argparse.FileType('r'))
>>> parser.parse_args(['-'])
Namespace(infile=<_io.TextIOWrapper name='<stdin>' encoding='UTF-8'>)
@end example


New in version 3.4: The @emph{encodings} and @emph{errors} keyword arguments.

@end deffn

@node Argument groups,Mutual exclusion,FileType objects,Other utilities
@anchor{library/argparse argument-groups}@anchor{24fd}
@subsubsection Argument groups


@geindex add_argument_group() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser add_argument_group}@anchor{24fe}
@deffn {Method} ArgumentParser.add_argument_group (title=None, description=None)

By default, @ref{91c,,ArgumentParser} groups command-line arguments into
“positional arguments” and “optional arguments” when displaying help
messages. When there is a better conceptual grouping of arguments than this
default one, appropriate groups can be created using the
@ref{24fe,,add_argument_group()} method:

@example
>>> parser = argparse.ArgumentParser(prog='PROG', add_help=False)
>>> group = parser.add_argument_group('group')
>>> group.add_argument('--foo', help='foo help')
>>> group.add_argument('bar', help='bar help')
>>> parser.print_help()
usage: PROG [--foo FOO] bar

group:
  bar    bar help
  --foo FOO  foo help
@end example

The @ref{24fe,,add_argument_group()} method returns an argument group object which
has an @ref{24cc,,add_argument()} method just like a regular
@ref{91c,,ArgumentParser}.  When an argument is added to the group, the parser
treats it just like a normal argument, but displays the argument in a
separate group for help messages.  The @ref{24fe,,add_argument_group()} method
accepts @emph{title} and @emph{description} arguments which can be used to
customize this display:

@example
>>> parser = argparse.ArgumentParser(prog='PROG', add_help=False)
>>> group1 = parser.add_argument_group('group1', 'group1 description')
>>> group1.add_argument('foo', help='foo help')
>>> group2 = parser.add_argument_group('group2', 'group2 description')
>>> group2.add_argument('--bar', help='bar help')
>>> parser.print_help()
usage: PROG [--bar BAR] foo

group1:
  group1 description

  foo    foo help

group2:
  group2 description

  --bar BAR  bar help
@end example

Note that any arguments not in your user-defined groups will end up back
in the usual “positional arguments” and “optional arguments” sections.


Changed in version 3.11: Calling @ref{24fe,,add_argument_group()} on an argument group is deprecated.
This feature was never supported and does not always work correctly.
The function exists on the API by accident through inheritance and
will be removed in the future.

@end deffn

@node Mutual exclusion,Parser defaults,Argument groups,Other utilities
@anchor{library/argparse mutual-exclusion}@anchor{24ff}
@subsubsection Mutual exclusion


@geindex add_mutually_exclusive_group() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser add_mutually_exclusive_group}@anchor{2500}
@deffn {Method} ArgumentParser.add_mutually_exclusive_group (required=False)

Create a mutually exclusive group. @ref{5,,argparse} will make sure that only
one of the arguments in the mutually exclusive group was present on the
command line:

@example
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> group = parser.add_mutually_exclusive_group()
>>> group.add_argument('--foo', action='store_true')
>>> group.add_argument('--bar', action='store_false')
>>> parser.parse_args(['--foo'])
Namespace(bar=True, foo=True)
>>> parser.parse_args(['--bar'])
Namespace(bar=False, foo=False)
>>> parser.parse_args(['--foo', '--bar'])
usage: PROG [-h] [--foo | --bar]
PROG: error: argument --bar: not allowed with argument --foo
@end example

The @ref{2500,,add_mutually_exclusive_group()} method also accepts a @emph{required}
argument, to indicate that at least one of the mutually exclusive arguments
is required:

@example
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> group = parser.add_mutually_exclusive_group(required=True)
>>> group.add_argument('--foo', action='store_true')
>>> group.add_argument('--bar', action='store_false')
>>> parser.parse_args([])
usage: PROG [-h] (--foo | --bar)
PROG: error: one of the arguments --foo --bar is required
@end example

Note that currently mutually exclusive argument groups do not support the
@emph{title} and @emph{description} arguments of
@ref{24fe,,add_argument_group()}.


Changed in version 3.11: Calling @ref{24fe,,add_argument_group()} or @ref{2500,,add_mutually_exclusive_group()}
on a mutually exclusive group is deprecated. These features were never
supported and do not always work correctly. The functions exist on the
API by accident through inheritance and will be removed in the future.

@end deffn

@node Parser defaults,Printing help,Mutual exclusion,Other utilities
@anchor{library/argparse parser-defaults}@anchor{2501}
@subsubsection Parser defaults


@geindex set_defaults() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser set_defaults}@anchor{24e3}
@deffn {Method} ArgumentParser.set_defaults (**kwargs)

Most of the time, the attributes of the object returned by @ref{24cd,,parse_args()}
will be fully determined by inspecting the command-line arguments and the argument
actions.  @ref{24e3,,set_defaults()} allows some additional
attributes that are determined without any inspection of the command line to
be added:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('foo', type=int)
>>> parser.set_defaults(bar=42, baz='badger')
>>> parser.parse_args(['736'])
Namespace(bar=42, baz='badger', foo=736)
@end example

Note that parser-level defaults always override argument-level defaults:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', default='bar')
>>> parser.set_defaults(foo='spam')
>>> parser.parse_args([])
Namespace(foo='spam')
@end example

Parser-level defaults can be particularly useful when working with multiple
parsers.  See the @ref{753,,add_subparsers()} method for an
example of this type.
@end deffn

@geindex get_default() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser get_default}@anchor{2502}
@deffn {Method} ArgumentParser.get_default (dest)

Get the default value for a namespace attribute, as set by either
@ref{24cc,,add_argument()} or by
@ref{24e3,,set_defaults()}:

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', default='badger')
>>> parser.get_default('foo')
'badger'
@end example
@end deffn

@node Printing help,Partial parsing,Parser defaults,Other utilities
@anchor{library/argparse printing-help}@anchor{2503}
@subsubsection Printing help


In most typical applications, @ref{24cd,,parse_args()} will take
care of formatting and printing any usage or error messages.  However, several
formatting methods are available:

@geindex print_usage() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser print_usage}@anchor{2504}
@deffn {Method} ArgumentParser.print_usage (file=None)

Print a brief description of how the @ref{91c,,ArgumentParser} should be
invoked on the command line.  If @emph{file} is @code{None}, @ref{5ef,,sys.stdout} is
assumed.
@end deffn

@geindex print_help() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser print_help}@anchor{2505}
@deffn {Method} ArgumentParser.print_help (file=None)

Print a help message, including the program usage and information about the
arguments registered with the @ref{91c,,ArgumentParser}.  If @emph{file} is
@code{None}, @ref{5ef,,sys.stdout} is assumed.
@end deffn

There are also variants of these methods that simply return a string instead of
printing it:

@geindex format_usage() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser format_usage}@anchor{1269}
@deffn {Method} ArgumentParser.format_usage ()

Return a string containing a brief description of how the
@ref{91c,,ArgumentParser} should be invoked on the command line.
@end deffn

@geindex format_help() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser format_help}@anchor{2506}
@deffn {Method} ArgumentParser.format_help ()

Return a string containing a help message, including the program usage and
information about the arguments registered with the @ref{91c,,ArgumentParser}.
@end deffn

@node Partial parsing,Customizing file parsing,Printing help,Other utilities
@anchor{library/argparse partial-parsing}@anchor{2507}
@subsubsection Partial parsing


@geindex parse_known_args() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser parse_known_args}@anchor{2508}
@deffn {Method} ArgumentParser.parse_known_args (args=None, namespace=None)
@end deffn

Sometimes a script may only parse a few of the command-line arguments, passing
the remaining arguments on to another script or program. In these cases, the
@ref{2508,,parse_known_args()} method can be useful.  It works much like
@ref{24cd,,parse_args()} except that it does not produce an error when
extra arguments are present.  Instead, it returns a two item tuple containing
the populated namespace and the list of remaining argument strings.

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', action='store_true')
>>> parser.add_argument('bar')
>>> parser.parse_known_args(['--foo', '--badger', 'BAR', 'spam'])
(Namespace(bar='BAR', foo=True), ['--badger', 'spam'])
@end example

@cartouche
@quotation Warning 
@ref{91d,,Prefix matching} rules apply to
@code{parse_known_args()}. The parser may consume an option even if it’s just
a prefix of one of its known options, instead of leaving it in the remaining
arguments list.
@end quotation
@end cartouche

@node Customizing file parsing,Exiting methods,Partial parsing,Other utilities
@anchor{library/argparse customizing-file-parsing}@anchor{2509}
@subsubsection Customizing file parsing


@geindex convert_arg_line_to_args() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser convert_arg_line_to_args}@anchor{24e2}
@deffn {Method} ArgumentParser.convert_arg_line_to_args (arg_line)

Arguments that are read from a file (see the @emph{fromfile_prefix_chars}
keyword argument to the @ref{91c,,ArgumentParser} constructor) are read one
argument per line. @ref{24e2,,convert_arg_line_to_args()} can be overridden for
fancier reading.

This method takes a single argument @emph{arg_line} which is a string read from
the argument file.  It returns a list of arguments parsed from this string.
The method is called once per line read from the argument file, in order.

A useful override of this method is one that treats each space-separated word
as an argument.  The following example demonstrates how to do this:

@example
class MyArgumentParser(argparse.ArgumentParser):
    def convert_arg_line_to_args(self, arg_line):
        return arg_line.split()
@end example
@end deffn

@node Exiting methods,Intermixed parsing,Customizing file parsing,Other utilities
@anchor{library/argparse exiting-methods}@anchor{250a}
@subsubsection Exiting methods


@geindex exit() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser exit}@anchor{250b}
@deffn {Method} ArgumentParser.exit (status=0, message=None)

This method terminates the program, exiting with the specified @emph{status}
and, if given, it prints a @emph{message} before that. The user can override
this method to handle these steps differently:

@example
class ErrorCatchingArgumentParser(argparse.ArgumentParser):
    def exit(self, status=0, message=None):
        if status:
            raise Exception(f'Exiting because of an error: @{message@}')
        exit(status)
@end example
@end deffn

@geindex error() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser error}@anchor{250c}
@deffn {Method} ArgumentParser.error (message)

This method prints a usage message including the @emph{message} to the
standard error and terminates the program with a status code of 2.
@end deffn

@node Intermixed parsing,,Exiting methods,Other utilities
@anchor{library/argparse intermixed-parsing}@anchor{250d}
@subsubsection Intermixed parsing


@geindex parse_intermixed_args() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser parse_intermixed_args}@anchor{61e}
@deffn {Method} ArgumentParser.parse_intermixed_args (args=None, namespace=None)
@end deffn

@geindex parse_known_intermixed_args() (argparse.ArgumentParser method)
@anchor{library/argparse argparse ArgumentParser parse_known_intermixed_args}@anchor{250e}
@deffn {Method} ArgumentParser.parse_known_intermixed_args (args=None, namespace=None)
@end deffn

A number of Unix commands allow the user to intermix optional arguments with
positional arguments.  The @ref{61e,,parse_intermixed_args()}
and @ref{250e,,parse_known_intermixed_args()} methods
support this parsing style.

These parsers do not support all the argparse features, and will raise
exceptions if unsupported features are used.  In particular, subparsers,
@code{argparse.REMAINDER}, and mutually exclusive groups that include both
optionals and positionals are not supported.

The following example shows the difference between
@ref{2508,,parse_known_args()} and
@ref{61e,,parse_intermixed_args()}: the former returns @code{['2',
'3']} as unparsed arguments, while the latter collects all the positionals
into @code{rest}.

@example
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo')
>>> parser.add_argument('cmd')
>>> parser.add_argument('rest', nargs='*', type=int)
>>> parser.parse_known_args('doit 1 --foo bar 2 3'.split())
(Namespace(cmd='doit', foo='bar', rest=[1]), ['2', '3'])
>>> parser.parse_intermixed_args('doit 1 --foo bar 2 3'.split())
Namespace(cmd='doit', foo='bar', rest=[1, 2, 3])
@end example

@ref{250e,,parse_known_intermixed_args()} returns a two item tuple
containing the populated namespace and the list of remaining argument strings.
@ref{61e,,parse_intermixed_args()} raises an error if there are any
remaining unparsed argument strings.


New in version 3.7.


@node Upgrading optparse code,,Other utilities,argparse — Parser for command-line options arguments and sub-commands
@anchor{library/argparse id2}@anchor{250f}@anchor{library/argparse upgrading-optparse-code}@anchor{d53}
@subsubsection Upgrading optparse code


Originally, the @ref{5,,argparse} module had attempted to maintain compatibility
with @ref{bf,,optparse}.  However, @ref{bf,,optparse} was difficult to extend
transparently, particularly with the changes required to support the new
@code{nargs=} specifiers and better usage messages.  When most everything in
@ref{bf,,optparse} had either been copy-pasted over or monkey-patched, it no
longer seemed practical to try to maintain the backwards compatibility.

The @ref{5,,argparse} module improves on the standard library @ref{bf,,optparse}
module in a number of ways including:


@itemize *

@item 
Handling positional arguments.

@item 
Supporting sub-commands.

@item 
Allowing alternative option prefixes like @code{+} and @code{/}.

@item 
Handling zero-or-more and one-or-more style arguments.

@item 
Producing more informative usage messages.

@item 
Providing a much simpler interface for custom @code{type} and @code{action}.
@end itemize

A partial upgrade path from @ref{bf,,optparse} to @ref{5,,argparse}:


@itemize *

@item 
Replace all @ref{2510,,optparse.OptionParser.add_option()} calls with
@ref{24cc,,ArgumentParser.add_argument()} calls.

@item 
Replace @code{(options, args) = parser.parse_args()} with @code{args =
parser.parse_args()} and add additional @ref{24cc,,ArgumentParser.add_argument()}
calls for the positional arguments. Keep in mind that what was previously
called @code{options}, now in the @ref{5,,argparse} context is called @code{args}.

@item 
Replace @ref{2511,,optparse.OptionParser.disable_interspersed_args()}
by using @ref{61e,,parse_intermixed_args()} instead of
@ref{24cd,,parse_args()}.

@item 
Replace callback actions and the @code{callback_*} keyword arguments with
@code{type} or @code{action} arguments.

@item 
Replace string names for @code{type} keyword arguments with the corresponding
type objects (e.g. int, float, complex, etc).

@item 
Replace @code{optparse.Values} with @ref{24cf,,Namespace} and
@code{optparse.OptionError} and @code{optparse.OptionValueError} with
@code{ArgumentError}.

@item 
Replace strings with implicit arguments such as @code{%default} or @code{%prog} with
the standard Python syntax to use dictionaries to format strings, that is,
@code{%(default)s} and @code{%(prog)s}.

@item 
Replace the OptionParser constructor @code{version} argument with a call to
@code{parser.add_argument('--version', action='version', version='<the version>')}.
@end itemize

@node getopt — C-style parser for command line options,logging — Logging facility for Python,argparse — Parser for command-line options arguments and sub-commands,Generic Operating System Services
@anchor{library/getopt doc}@anchor{2512}@anchor{library/getopt getopt-c-style-parser-for-command-line-options}@anchor{2513}@anchor{library/getopt module-getopt}@anchor{81}
@subsection @code{getopt} — C-style parser for command line options


@geindex module; getopt

@strong{Source code:} Lib/getopt.py@footnote{https://github.com/python/cpython/tree/main/Lib/getopt.py}

@cartouche
@quotation Note 
The @ref{81,,getopt} module is a parser for command line options whose API is
designed to be familiar to users of the C @code{getopt()} function. Users who
are unfamiliar with the C @code{getopt()} function or who would like to write
less code and get better help and error messages should consider using the
@ref{5,,argparse} module instead.
@end quotation
@end cartouche


__________________________________________________________________


This module helps scripts to parse the command line arguments in @code{sys.argv}.
It supports the same conventions as the Unix @code{getopt()} function (including
the special meanings of arguments of the form ‘@code{-}’ and ‘@code{--}‘).  Long
options similar to those supported by GNU software may be used as well via an
optional third argument.

This module provides two functions and an
exception:

@geindex getopt() (in module getopt)
@anchor{library/getopt getopt getopt}@anchor{2514}
@deffn {Function} getopt.getopt (args, shortopts, longopts=[])

Parses command line options and parameter list.  @emph{args} is the argument list to
be parsed, without the leading reference to the running program. Typically, this
means @code{sys.argv[1:]}. @emph{shortopts} is the string of option letters that the
script wants to recognize, with options that require an argument followed by a
colon (@code{':'}; i.e., the same format that Unix @code{getopt()} uses).

@cartouche
@quotation Note 
Unlike GNU @code{getopt()}, after a non-option argument, all further
arguments are considered also non-options. This is similar to the way
non-GNU Unix systems work.
@end quotation
@end cartouche

@emph{longopts}, if specified, must be a list of strings with the names of the
long options which should be supported.  The leading @code{'--'} characters
should not be included in the option name.  Long options which require an
argument should be followed by an equal sign (@code{'='}).  Optional arguments
are not supported.  To accept only long options, @emph{shortopts} should be an
empty string.  Long options on the command line can be recognized so long as
they provide a prefix of the option name that matches exactly one of the
accepted options.  For example, if @emph{longopts} is @code{['foo', 'frob']}, the
option @code{--fo} will match as @code{--foo}, but @code{--f} will
not match uniquely, so @ref{2515,,GetoptError} will be raised.

The return value consists of two elements: the first is a list of @code{(option,
value)} pairs; the second is the list of program arguments left after the
option list was stripped (this is a trailing slice of @emph{args}).  Each
option-and-value pair returned has the option as its first element, prefixed
with a hyphen for short options (e.g., @code{'-x'}) or two hyphens for long
options (e.g., @code{'--long-option'}), and the option argument as its
second element, or an empty string if the option has no argument.  The
options occur in the list in the same order in which they were found, thus
allowing multiple occurrences.  Long and short options may be mixed.
@end deffn

@geindex gnu_getopt() (in module getopt)
@anchor{library/getopt getopt gnu_getopt}@anchor{2516}
@deffn {Function} getopt.gnu_getopt (args, shortopts, longopts=[])

This function works like @ref{81,,getopt()}, except that GNU style scanning mode is
used by default. This means that option and non-option arguments may be
intermixed. The @ref{81,,getopt()} function stops processing options as soon as a
non-option argument is encountered.

If the first character of the option string is @code{'+'}, or if the environment
variable 
@geindex POSIXLY_CORRECT
@geindex environment variable; POSIXLY_CORRECT
@code{POSIXLY_CORRECT} is set, then option processing stops as
soon as a non-option argument is encountered.
@end deffn

@geindex GetoptError
@anchor{library/getopt getopt GetoptError}@anchor{2515}
@deffn {Exception} getopt.GetoptError

This is raised when an unrecognized option is found in the argument list or when
an option requiring an argument is given none. The argument to the exception is
a string indicating the cause of the error.  For long options, an argument given
to an option which does not require one will also cause this exception to be
raised.  The attributes @code{msg} and @code{opt} give the error message and
related option; if there is no specific option to which the exception relates,
@code{opt} is an empty string.
@end deffn

@c XXX deprecated?

@geindex error
@anchor{library/getopt getopt error}@anchor{2517}
@deffn {Exception} getopt.error

Alias for @ref{2515,,GetoptError}; for backward compatibility.
@end deffn

An example using only Unix style options:

@example
>>> import getopt
>>> args = '-a -b -cfoo -d bar a1 a2'.split()
>>> args
['-a', '-b', '-cfoo', '-d', 'bar', 'a1', 'a2']
>>> optlist, args = getopt.getopt(args, 'abc:d:')
>>> optlist
[('-a', '@w{'}), ('-b', '@w{'}), ('-c', 'foo'), ('-d', 'bar')]
>>> args
['a1', 'a2']
@end example

Using long option names is equally easy:

@example
>>> s = '--condition=foo --testing --output-file abc.def -x a1 a2'
>>> args = s.split()
>>> args
['--condition=foo', '--testing', '--output-file', 'abc.def', '-x', 'a1', 'a2']
>>> optlist, args = getopt.getopt(args, 'x', [
...     'condition=', 'output-file=', 'testing'])
>>> optlist
[('--condition', 'foo'), ('--testing', '@w{'}), ('--output-file', 'abc.def'), ('-x', '@w{'})]
>>> args
['a1', 'a2']
@end example

In a script, typical usage is something like this:

@example
import getopt, sys

def main():
    try:
        opts, args = getopt.getopt(sys.argv[1:], "ho:v", ["help", "output="])
    except getopt.GetoptError as err:
        # print help information and exit:
        print(err)  # will print something like "option -a not recognized"
        usage()
        sys.exit(2)
    output = None
    verbose = False
    for o, a in opts:
        if o == "-v":
            verbose = True
        elif o in ("-h", "--help"):
            usage()
            sys.exit()
        elif o in ("-o", "--output"):
            output = a
        else:
            assert False, "unhandled option"
    # ...

if __name__ == "__main__":
    main()
@end example

Note that an equivalent command line interface could be produced with less code
and more informative help and error messages by using the @ref{5,,argparse} module:

@example
import argparse

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-o', '--output')
    parser.add_argument('-v', dest='verbose', action='store_true')
    args = parser.parse_args()
    # ... do something with args.output ...
    # ... do something with args.verbose ..
@end example


@subsubheading See also



@table @asis

@item Module @ref{5,,argparse}

Alternative command line option and argument parsing library.
@end table


@node logging — Logging facility for Python,logging config — Logging configuration,getopt — C-style parser for command line options,Generic Operating System Services
@anchor{library/logging doc}@anchor{2518}@anchor{library/logging logging-logging-facility-for-python}@anchor{2519}@anchor{library/logging module-logging}@anchor{a6}
@subsection @code{logging} — Logging facility for Python


@geindex module; logging

@strong{Source code:} Lib/logging/__init__.py@footnote{https://github.com/python/cpython/tree/main/Lib/logging/__init__.py}

@geindex Errors; logging

@subsubheading Important



This page contains the API reference information. For tutorial
information and discussion of more advanced topics, see


@itemize *

@item 
@ref{d97,,Basic Tutorial}

@item 
@ref{d98,,Advanced Tutorial}

@item 
@ref{d99,,Logging Cookbook}
@end itemize


__________________________________________________________________


This module defines functions and classes which implement a flexible event
logging system for applications and libraries.

The key benefit of having the logging API provided by a standard library module
is that all Python modules can participate in logging, so your application log
can include your own messages integrated with messages from third-party
modules.

The module provides a lot of functionality and flexibility.  If you are
unfamiliar with logging, the best way to get to grips with it is to see the
tutorials (see the links on the right).

The basic classes defined by the module, together with their functions, are
listed below.


@itemize *

@item 
Loggers expose the interface that application code directly uses.

@item 
Handlers send the log records (created by loggers) to the appropriate
destination.

@item 
Filters provide a finer grained facility for determining which log records
to output.

@item 
Formatters specify the layout of log records in the final output.
@end itemize

@menu
* Logger Objects:: 
* Logging Levels:: 
* Handler Objects:: 
* Formatter Objects:: 
* Filter Objects:: 
* LogRecord Objects:: 
* LogRecord attributes:: 
* LoggerAdapter Objects:: 
* Thread Safety:: 
* Module-Level Functions:: 
* Module-Level Attributes:: 
* Integration with the warnings module:: 

@end menu

@node Logger Objects,Logging Levels,,logging — Logging facility for Python
@anchor{library/logging logger}@anchor{251a}@anchor{library/logging logger-objects}@anchor{251b}
@subsubsection Logger Objects


Loggers have the following attributes and methods.  Note that Loggers should
@emph{NEVER} be instantiated directly, but always through the module-level function
@code{logging.getLogger(name)}.  Multiple calls to @ref{463,,getLogger()} with the same
name will always return a reference to the same Logger object.

The @code{name} is potentially a period-separated hierarchical value, like
@code{foo.bar.baz} (though it could also be just plain @code{foo}, for example).
Loggers that are further down in the hierarchical list are children of loggers
higher up in the list.  For example, given a logger with a name of @code{foo},
loggers with names of @code{foo.bar}, @code{foo.bar.baz}, and @code{foo.bam} are all
descendants of @code{foo}.  The logger name hierarchy is analogous to the Python
package hierarchy, and identical to it if you organise your loggers on a
per-module basis using the recommended construction
@code{logging.getLogger(__name__)}.  That’s because in a module, @code{__name__}
is the module’s name in the Python package namespace.

@geindex Logger (class in logging)
@anchor{library/logging logging Logger}@anchor{676}
@deffn {Class} logging.Logger

@geindex propagate (logging.Logger attribute)
@anchor{library/logging logging Logger propagate}@anchor{251c}
@deffn {Attribute} propagate

If this attribute evaluates to true, events logged to this logger will be
passed to the handlers of higher level (ancestor) loggers, in addition to
any handlers attached to this logger. Messages are passed directly to the
ancestor loggers’ handlers - neither the level nor filters of the ancestor
loggers in question are considered.

If this evaluates to false, logging messages are not passed to the handlers
of ancestor loggers.

Spelling it out with an example: If the propagate attribute of the logger named
@code{A.B.C} evaluates to true, any event logged to @code{A.B.C} via a method call such as
@code{logging.getLogger('A.B.C').error(...)} will [subject to passing that logger’s
level and filter settings] be passed in turn to any handlers attached to loggers
named @code{A.B}, @code{A} and the root logger, after first being passed to any handlers
attached to @code{A.B.C}. If any logger in the chain @code{A.B.C}, @code{A.B}, @code{A} has its
@code{propagate} attribute set to false, then that is the last logger whose handlers
are offered the event to handle, and propagation stops at that point.

The constructor sets this attribute to @code{True}.

@cartouche
@quotation Note 
If you attach a handler to a logger @emph{and} one or more of its
ancestors, it may emit the same record multiple times. In general, you
should not need to attach a handler to more than one logger - if you just
attach it to the appropriate logger which is highest in the logger
hierarchy, then it will see all events logged by all descendant loggers,
provided that their propagate setting is left set to @code{True}. A common
scenario is to attach handlers only to the root logger, and to let
propagation take care of the rest.
@end quotation
@end cartouche
@end deffn

@geindex setLevel() (logging.Logger method)
@anchor{library/logging logging Logger setLevel}@anchor{251d}
@deffn {Method} setLevel (level)

Sets the threshold for this logger to @emph{level}. Logging messages which are less
severe than @emph{level} will be ignored; logging messages which have severity @emph{level}
or higher will be emitted by whichever handler or handlers service this logger,
unless a handler’s level has been set to a higher severity level than @emph{level}.

When a logger is created, the level is set to @code{NOTSET} (which causes
all messages to be processed when the logger is the root logger, or delegation
to the parent when the logger is a non-root logger). Note that the root logger
is created with level @code{WARNING}.

The term ‘delegation to the parent’ means that if a logger has a level of
NOTSET, its chain of ancestor loggers is traversed until either an ancestor with
a level other than NOTSET is found, or the root is reached.

If an ancestor is found with a level other than NOTSET, then that ancestor’s
level is treated as the effective level of the logger where the ancestor search
began, and is used to determine how a logging event is handled.

If the root is reached, and it has a level of NOTSET, then all messages will be
processed. Otherwise, the root’s level will be used as the effective level.

See @ref{251e,,Logging Levels} for a list of levels.


Changed in version 3.2: The @emph{level} parameter now accepts a string representation of the
level such as ‘INFO’ as an alternative to the integer constants
such as @code{INFO}. Note, however, that levels are internally stored
as integers, and methods such as e.g. @ref{251f,,getEffectiveLevel()} and
@ref{2520,,isEnabledFor()} will return/expect to be passed integers.

@end deffn

@geindex isEnabledFor() (logging.Logger method)
@anchor{library/logging logging Logger isEnabledFor}@anchor{2520}
@deffn {Method} isEnabledFor (level)

Indicates if a message of severity @emph{level} would be processed by this logger.
This method checks first the module-level level set by
@code{logging.disable(level)} and then the logger’s effective level as determined
by @ref{251f,,getEffectiveLevel()}.
@end deffn

@geindex getEffectiveLevel() (logging.Logger method)
@anchor{library/logging logging Logger getEffectiveLevel}@anchor{251f}
@deffn {Method} getEffectiveLevel ()

Indicates the effective level for this logger. If a value other than
@code{NOTSET} has been set using @ref{251d,,setLevel()}, it is returned. Otherwise,
the hierarchy is traversed towards the root until a value other than
@code{NOTSET} is found, and that value is returned. The value returned is
an integer, typically one of @code{logging.DEBUG}, @code{logging.INFO}
etc.
@end deffn

@geindex getChild() (logging.Logger method)
@anchor{library/logging logging Logger getChild}@anchor{e92}
@deffn {Method} getChild (suffix)

Returns a logger which is a descendant to this logger, as determined by the suffix.
Thus, @code{logging.getLogger('abc').getChild('def.ghi')} would return the same
logger as would be returned by @code{logging.getLogger('abc.def.ghi')}. This is a
convenience method, useful when the parent logger is named using e.g. @code{__name__}
rather than a literal string.


New in version 3.2.

@end deffn

@geindex debug() (logging.Logger method)
@anchor{library/logging logging Logger debug}@anchor{991}
@deffn {Method} debug (msg, *args, **kwargs)

Logs a message with level @code{DEBUG} on this logger. The @emph{msg} is the
message format string, and the @emph{args} are the arguments which are merged into
@emph{msg} using the string formatting operator. (Note that this means that you can
use keywords in the format string, together with a single dictionary argument.)
No % formatting operation is performed on @emph{msg} when no @emph{args} are supplied.

There are four keyword arguments in @emph{kwargs} which are inspected:
@emph{exc_info}, @emph{stack_info}, @emph{stacklevel} and @emph{extra}.

If @emph{exc_info} does not evaluate as false, it causes exception information to be
added to the logging message. If an exception tuple (in the format returned by
@ref{18f,,sys.exc_info()}) or an exception instance is provided, it is used;
otherwise, @ref{18f,,sys.exc_info()} is called to get the exception information.

The second optional keyword argument is @emph{stack_info}, which defaults to
@code{False}. If true, stack information is added to the logging
message, including the actual logging call. Note that this is not the same
stack information as that displayed through specifying @emph{exc_info}: The
former is stack frames from the bottom of the stack up to the logging call
in the current thread, whereas the latter is information about stack frames
which have been unwound, following an exception, while searching for
exception handlers.

You can specify @emph{stack_info} independently of @emph{exc_info}, e.g. to just show
how you got to a certain point in your code, even when no exceptions were
raised. The stack frames are printed following a header line which says:

@example
Stack (most recent call last):
@end example

This mimics the @code{Traceback (most recent call last):} which is used when
displaying exception frames.

The third optional keyword argument is @emph{stacklevel}, which defaults to @code{1}.
If greater than 1, the corresponding number of stack frames are skipped
when computing the line number and function name set in the @ref{b6c,,LogRecord}
created for the logging event. This can be used in logging helpers so that
the function name, filename and line number recorded are not the information
for the helper function/method, but rather its caller. The name of this
parameter mirrors the equivalent one in the @ref{12d,,warnings} module.

The fourth keyword argument is @emph{extra} which can be used to pass a
dictionary which is used to populate the __dict__ of the @ref{b6c,,LogRecord}
created for the logging event with user-defined attributes. These custom
attributes can then be used as you like. For example, they could be
incorporated into logged messages. For example:

@example
FORMAT = '%(asctime)s %(clientip)-15s %(user)-8s %(message)s'
logging.basicConfig(format=FORMAT)
d = @{'clientip': '192.168.0.1', 'user': 'fbloggs'@}
logger = logging.getLogger('tcpserver')
logger.warning('Protocol problem: %s', 'connection reset', extra=d)
@end example

would print something like

@example
2006-02-08 22:20:02,165 192.168.0.1 fbloggs  Protocol problem: connection reset
@end example

The keys in the dictionary passed in @emph{extra} should not clash with the keys used
by the logging system. (See the @ref{11c6,,Formatter} documentation for more
information on which keys are used by the logging system.)

If you choose to use these attributes in logged messages, you need to exercise
some care. In the above example, for instance, the @ref{11c6,,Formatter} has been
set up with a format string which expects ‘clientip’ and ‘user’ in the attribute
dictionary of the @ref{b6c,,LogRecord}. If these are missing, the message will
not be logged because a string formatting exception will occur. So in this case,
you always need to pass the @emph{extra} dictionary with these keys.

While this might be annoying, this feature is intended for use in specialized
circumstances, such as multi-threaded servers where the same code executes in
many contexts, and interesting conditions which arise are dependent on this
context (such as remote client IP address and authenticated user name, in the
above example). In such circumstances, it is likely that specialized
@ref{11c6,,Formatter}s would be used with particular @ref{2521,,Handler}s.


Changed in version 3.2: The @emph{stack_info} parameter was added.



Changed in version 3.5: The @emph{exc_info} parameter can now accept exception instances.



Changed in version 3.8: The @emph{stacklevel} parameter was added.

@end deffn

@geindex info() (logging.Logger method)
@anchor{library/logging logging Logger info}@anchor{2522}
@deffn {Method} info (msg, *args, **kwargs)

Logs a message with level @code{INFO} on this logger. The arguments are
interpreted as for @ref{2523,,debug()}.
@end deffn

@geindex warning() (logging.Logger method)
@anchor{library/logging logging Logger warning}@anchor{2524}
@deffn {Method} warning (msg, *args, **kwargs)

Logs a message with level @code{WARNING} on this logger. The arguments are
interpreted as for @ref{2523,,debug()}.

@cartouche
@quotation Note 
There is an obsolete method @code{warn} which is functionally
identical to @code{warning}. As @code{warn} is deprecated, please do not use
it - use @code{warning} instead.
@end quotation
@end cartouche
@end deffn

@geindex error() (logging.Logger method)
@anchor{library/logging logging Logger error}@anchor{2525}
@deffn {Method} error (msg, *args, **kwargs)

Logs a message with level @code{ERROR} on this logger. The arguments are
interpreted as for @ref{2523,,debug()}.
@end deffn

@geindex critical() (logging.Logger method)
@anchor{library/logging logging Logger critical}@anchor{990}
@deffn {Method} critical (msg, *args, **kwargs)

Logs a message with level @code{CRITICAL} on this logger. The arguments are
interpreted as for @ref{2523,,debug()}.
@end deffn

@geindex log() (logging.Logger method)
@anchor{library/logging logging Logger log}@anchor{98e}
@deffn {Method} log (level, msg, *args, **kwargs)

Logs a message with integer level @emph{level} on this logger. The other arguments are
interpreted as for @ref{2523,,debug()}.
@end deffn

@geindex exception() (logging.Logger method)
@anchor{library/logging logging Logger exception}@anchor{98f}
@deffn {Method} exception (msg, *args, **kwargs)

Logs a message with level @code{ERROR} on this logger. The arguments are
interpreted as for @ref{2523,,debug()}. Exception info is added to the logging
message. This method should only be called from an exception handler.
@end deffn

@geindex addFilter() (logging.Logger method)
@anchor{library/logging logging Logger addFilter}@anchor{2526}
@deffn {Method} addFilter (filter)

Adds the specified filter @emph{filter} to this logger.
@end deffn

@geindex removeFilter() (logging.Logger method)
@anchor{library/logging logging Logger removeFilter}@anchor{2527}
@deffn {Method} removeFilter (filter)

Removes the specified filter @emph{filter} from this logger.
@end deffn

@geindex filter() (logging.Logger method)
@anchor{library/logging logging Logger filter}@anchor{2528}
@deffn {Method} filter (record)

Apply this logger’s filters to the record and return @code{True} if the
record is to be processed. The filters are consulted in turn, until one of
them returns a false value. If none of them return a false value, the record
will be processed (passed to handlers). If one returns a false value, no
further processing of the record occurs.
@end deffn

@geindex addHandler() (logging.Logger method)
@anchor{library/logging logging Logger addHandler}@anchor{2529}
@deffn {Method} addHandler (hdlr)

Adds the specified handler @emph{hdlr} to this logger.
@end deffn

@geindex removeHandler() (logging.Logger method)
@anchor{library/logging logging Logger removeHandler}@anchor{252a}
@deffn {Method} removeHandler (hdlr)

Removes the specified handler @emph{hdlr} from this logger.
@end deffn

@geindex findCaller() (logging.Logger method)
@anchor{library/logging logging Logger findCaller}@anchor{252b}
@deffn {Method} findCaller (stack_info=False, stacklevel=1)

Finds the caller’s source filename and line number. Returns the filename, line
number, function name and stack information as a 4-element tuple. The stack
information is returned as @code{None} unless @emph{stack_info} is @code{True}.

The @emph{stacklevel} parameter is passed from code calling the @ref{2523,,debug()}
and other APIs. If greater than 1, the excess is used to skip stack frames
before determining the values to be returned. This will generally be useful
when calling logging APIs from helper/wrapper code, so that the information
in the event log refers not to the helper/wrapper code, but to the code that
calls it.
@end deffn

@geindex handle() (logging.Logger method)
@anchor{library/logging logging Logger handle}@anchor{252c}
@deffn {Method} handle (record)

Handles a record by passing it to all handlers associated with this logger and
its ancestors (until a false value of @emph{propagate} is found). This method is used
for unpickled records received from a socket, as well as those created locally.
Logger-level filtering is applied using @ref{2528,,filter()}.
@end deffn

@geindex makeRecord() (logging.Logger method)
@anchor{library/logging logging Logger makeRecord}@anchor{252d}
@deffn {Method} makeRecord (name, level, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None)

This is a factory method which can be overridden in subclasses to create
specialized @ref{b6c,,LogRecord} instances.
@end deffn

@geindex hasHandlers() (logging.Logger method)
@anchor{library/logging logging Logger hasHandlers}@anchor{252e}
@deffn {Method} hasHandlers ()

Checks to see if this logger has any handlers configured. This is done by
looking for handlers in this logger and its parents in the logger hierarchy.
Returns @code{True} if a handler was found, else @code{False}. The method stops searching
up the hierarchy whenever a logger with the ‘propagate’ attribute set to
false is found - that will be the last logger which is checked for the
existence of handlers.


New in version 3.2.

@end deffn


Changed in version 3.7: Loggers can now be pickled and unpickled.

@end deffn

@node Logging Levels,Handler Objects,Logger Objects,logging — Logging facility for Python
@anchor{library/logging levels}@anchor{251e}@anchor{library/logging logging-levels}@anchor{252f}
@subsubsection Logging Levels


The numeric values of logging levels are given in the following table. These are
primarily of interest if you want to define your own levels, and need them to
have specific values relative to the predefined levels. If you define a level
with the same numeric value, it overwrites the predefined value; the predefined
name is lost.


@multitable {xxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxx} 
@headitem

Level

@tab

Numeric value

@item

@code{CRITICAL}

@tab

50

@item

@code{ERROR}

@tab

40

@item

@code{WARNING}

@tab

30

@item

@code{INFO}

@tab

20

@item

@code{DEBUG}

@tab

10

@item

@code{NOTSET}

@tab

0

@end multitable


@node Handler Objects,Formatter Objects,Logging Levels,logging — Logging facility for Python
@anchor{library/logging handler}@anchor{2530}@anchor{library/logging handler-objects}@anchor{2531}
@subsubsection Handler Objects


Handlers have the following attributes and methods. Note that @ref{2521,,Handler}
is never instantiated directly; this class acts as a base for more useful
subclasses. However, the @code{__init__()} method in subclasses needs to call
@ref{2532,,Handler.__init__()}.

@geindex Handler (class in logging)
@anchor{library/logging logging Handler}@anchor{2521}
@deffn {Class} logging.Handler

@geindex __init__() (logging.Handler method)
@anchor{library/logging logging Handler __init__}@anchor{2532}
@deffn {Method} __init__ (level=NOTSET)

Initializes the @ref{2521,,Handler} instance by setting its level, setting the list
of filters to the empty list and creating a lock (using @ref{2533,,createLock()}) for
serializing access to an I/O mechanism.
@end deffn

@geindex createLock() (logging.Handler method)
@anchor{library/logging logging Handler createLock}@anchor{2533}
@deffn {Method} createLock ()

Initializes a thread lock which can be used to serialize access to underlying
I/O functionality which may not be threadsafe.
@end deffn

@geindex acquire() (logging.Handler method)
@anchor{library/logging logging Handler acquire}@anchor{2534}
@deffn {Method} acquire ()

Acquires the thread lock created with @ref{2533,,createLock()}.
@end deffn

@geindex release() (logging.Handler method)
@anchor{library/logging logging Handler release}@anchor{2535}
@deffn {Method} release ()

Releases the thread lock acquired with @ref{2534,,acquire()}.
@end deffn

@geindex setLevel() (logging.Handler method)
@anchor{library/logging logging Handler setLevel}@anchor{2536}
@deffn {Method} setLevel (level)

Sets the threshold for this handler to @emph{level}. Logging messages which are
less severe than @emph{level} will be ignored. When a handler is created, the
level is set to @code{NOTSET} (which causes all messages to be
processed).

See @ref{251e,,Logging Levels} for a list of levels.


Changed in version 3.2: The @emph{level} parameter now accepts a string representation of the
level such as ‘INFO’ as an alternative to the integer constants
such as @code{INFO}.

@end deffn

@geindex setFormatter() (logging.Handler method)
@anchor{library/logging logging Handler setFormatter}@anchor{2537}
@deffn {Method} setFormatter (fmt)

Sets the @ref{11c6,,Formatter} for this handler to @emph{fmt}.
@end deffn

@geindex addFilter() (logging.Handler method)
@anchor{library/logging logging Handler addFilter}@anchor{2538}
@deffn {Method} addFilter (filter)

Adds the specified filter @emph{filter} to this handler.
@end deffn

@geindex removeFilter() (logging.Handler method)
@anchor{library/logging logging Handler removeFilter}@anchor{2539}
@deffn {Method} removeFilter (filter)

Removes the specified filter @emph{filter} from this handler.
@end deffn

@geindex filter() (logging.Handler method)
@anchor{library/logging logging Handler filter}@anchor{253a}
@deffn {Method} filter (record)

Apply this handler’s filters to the record and return @code{True} if the
record is to be processed. The filters are consulted in turn, until one of
them returns a false value. If none of them return a false value, the record
will be emitted. If one returns a false value, the handler will not emit the
record.
@end deffn

@geindex flush() (logging.Handler method)
@anchor{library/logging logging Handler flush}@anchor{253b}
@deffn {Method} flush ()

Ensure all logging output has been flushed. This version does nothing and is
intended to be implemented by subclasses.
@end deffn

@geindex close() (logging.Handler method)
@anchor{library/logging logging Handler close}@anchor{253c}
@deffn {Method} close ()

Tidy up any resources used by the handler. This version does no output but
removes the handler from an internal list of handlers which is closed when
@ref{253d,,shutdown()} is called. Subclasses should ensure that this gets called
from overridden @ref{253c,,close()} methods.
@end deffn

@geindex handle() (logging.Handler method)
@anchor{library/logging logging Handler handle}@anchor{253e}
@deffn {Method} handle (record)

Conditionally emits the specified logging record, depending on filters which may
have been added to the handler. Wraps the actual emission of the record with
acquisition/release of the I/O thread lock.
@end deffn

@geindex handleError() (logging.Handler method)
@anchor{library/logging logging Handler handleError}@anchor{253f}
@deffn {Method} handleError (record)

This method should be called from handlers when an exception is encountered
during an @ref{2540,,emit()} call. If the module-level attribute
@code{raiseExceptions} is @code{False}, exceptions get silently ignored. This is
what is mostly wanted for a logging system - most users will not care about
errors in the logging system, they are more interested in application
errors. You could, however, replace this with a custom handler if you wish.
The specified record is the one which was being processed when the exception
occurred. (The default value of @code{raiseExceptions} is @code{True}, as that is
more useful during development).
@end deffn

@geindex format() (logging.Handler method)
@anchor{library/logging logging Handler format}@anchor{2541}
@deffn {Method} format (record)

Do formatting for a record - if a formatter is set, use it. Otherwise, use the
default formatter for the module.
@end deffn

@geindex emit() (logging.Handler method)
@anchor{library/logging logging Handler emit}@anchor{2540}
@deffn {Method} emit (record)

Do whatever it takes to actually log the specified logging record. This version
is intended to be implemented by subclasses and so raises a
@ref{8a3,,NotImplementedError}.
@end deffn
@end deffn

For a list of handlers included as standard, see @ref{a8,,logging.handlers}.

@node Formatter Objects,Filter Objects,Handler Objects,logging — Logging facility for Python
@anchor{library/logging formatter-objects}@anchor{2542}@anchor{library/logging id1}@anchor{2543}
@subsubsection Formatter Objects


@ref{11c6,,Formatter} objects have the following attributes and methods. They are
responsible for converting a @ref{b6c,,LogRecord} to (usually) a string which can
be interpreted by either a human or an external system. The base
@ref{11c6,,Formatter} allows a formatting string to be specified. If none is
supplied, the default value of @code{'%(message)s'} is used, which just includes
the message in the logging call. To have additional items of information in the
formatted output (such as a timestamp), keep reading.

A Formatter can be initialized with a format string which makes use of knowledge
of the @ref{b6c,,LogRecord} attributes - such as the default value mentioned above
making use of the fact that the user’s message and arguments are pre-formatted
into a @ref{b6c,,LogRecord}’s @emph{message} attribute.  This format string contains
standard Python %-style mapping keys. See section @ref{14e4,,printf-style String Formatting}
for more information on string formatting.

The useful mapping keys in a @ref{b6c,,LogRecord} are given in the section on
@ref{2544,,LogRecord attributes}.

@geindex Formatter (class in logging)
@anchor{library/logging logging Formatter}@anchor{11c6}
@deffn {Class} logging.Formatter (fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)

Returns a new instance of the @ref{11c6,,Formatter} class.  The instance is
initialized with a format string for the message as a whole, as well as a
format string for the date/time portion of a message.  If no @emph{fmt} is
specified, @code{'%(message)s'} is used.  If no @emph{datefmt} is specified, a format
is used which is described in the @ref{2545,,formatTime()} documentation.

The @emph{style} parameter can be one of ‘%’, ‘@{’ or ‘$’ and determines how
the format string will be merged with its data: using one of %-formatting,
@ref{784,,str.format()} or @ref{6b5,,string.Template}. This only applies to the
format string @emph{fmt} (e.g. @code{'%(message)s'} or @code{@{message@}}), not to the
actual log messages passed to @code{Logger.debug} etc; see
@ref{2546,,Using particular formatting styles throughout your application} for more information on using @{- and $-formatting
for log messages.

The @emph{defaults} parameter can be a dictionary with default values to use in
custom fields. For example:
@code{logging.Formatter('%(ip)s %(message)s', defaults=@{"ip": None@})}


Changed in version 3.2: The @emph{style} parameter was added.



Changed in version 3.8: The @emph{validate} parameter was added. Incorrect or mismatched style and fmt
will raise a @code{ValueError}.
For example: @code{logging.Formatter('%(asctime)s - %(message)s', style='@{')}.



Changed in version 3.10: The @emph{defaults} parameter was added.


@geindex format() (logging.Formatter method)
@anchor{library/logging logging Formatter format}@anchor{2547}
@deffn {Method} format (record)

The record’s attribute dictionary is used as the operand to a string
formatting operation. Returns the resulting string. Before formatting the
dictionary, a couple of preparatory steps are carried out. The @emph{message}
attribute of the record is computed using @emph{msg} % @emph{args}. If the
formatting string contains @code{'(asctime)'}, @ref{2545,,formatTime()} is called
to format the event time. If there is exception information, it is
formatted using @ref{2548,,formatException()} and appended to the message. Note
that the formatted exception information is cached in attribute
@emph{exc_text}. This is useful because the exception information can be
pickled and sent across the wire, but you should be careful if you have
more than one @ref{11c6,,Formatter} subclass which customizes the formatting
of exception information. In this case, you will have to clear the cached
value (by setting the @emph{exc_text} attribute to @code{None}) after a formatter
has done its formatting, so that the next formatter to handle the event
doesn’t use the cached value, but recalculates it afresh.

If stack information is available, it’s appended after the exception
information, using @ref{2549,,formatStack()} to transform it if necessary.
@end deffn

@geindex formatTime() (logging.Formatter method)
@anchor{library/logging logging Formatter formatTime}@anchor{2545}
@deffn {Method} formatTime (record, datefmt=None)

This method should be called from @ref{785,,format()} by a formatter which
wants to make use of a formatted time. This method can be overridden in
formatters to provide for any specific requirement, but the basic behavior
is as follows: if @emph{datefmt} (a string) is specified, it is used with
@ref{d8b,,time.strftime()} to format the creation time of the
record. Otherwise, the format ‘%Y-%m-%d %H:%M:%S,uuu’ is used, where the
uuu part is a millisecond value and the other letters are as per the
@ref{d8b,,time.strftime()} documentation.  An example time in this format is
@code{2003-01-23 00:29:50,411}.  The resulting string is returned.

This function uses a user-configurable function to convert the creation
time to a tuple. By default, @ref{1224,,time.localtime()} is used; to change
this for a particular formatter instance, set the @code{converter} attribute
to a function with the same signature as @ref{1224,,time.localtime()} or
@ref{d65,,time.gmtime()}. To change it for all formatters, for example if you
want all logging times to be shown in GMT, set the @code{converter}
attribute in the @code{Formatter} class.


Changed in version 3.3: Previously, the default format was hard-coded as in this example:
@code{2010-09-06 22:38:15,292} where the part before the comma is
handled by a strptime format string (@code{'%Y-%m-%d %H:%M:%S'}), and the
part after the comma is a millisecond value. Because strptime does not
have a format placeholder for milliseconds, the millisecond value is
appended using another format string, @code{'%s,%03d'} — and both of these
format strings have been hardcoded into this method. With the change,
these strings are defined as class-level attributes which can be
overridden at the instance level when desired. The names of the
attributes are @code{default_time_format} (for the strptime format string)
and @code{default_msec_format} (for appending the millisecond value).



Changed in version 3.9: The @code{default_msec_format} can be @code{None}.

@end deffn

@geindex formatException() (logging.Formatter method)
@anchor{library/logging logging Formatter formatException}@anchor{2548}
@deffn {Method} formatException (exc_info)

Formats the specified exception information (a standard exception tuple as
returned by @ref{18f,,sys.exc_info()}) as a string. This default implementation
just uses @ref{2d5,,traceback.print_exception()}. The resulting string is
returned.
@end deffn

@geindex formatStack() (logging.Formatter method)
@anchor{library/logging logging Formatter formatStack}@anchor{2549}
@deffn {Method} formatStack (stack_info)

Formats the specified stack information (a string as returned by
@ref{9f8,,traceback.print_stack()}, but with the last newline removed) as a
string. This default implementation just returns the input value.
@end deffn
@end deffn

@node Filter Objects,LogRecord Objects,Formatter Objects,logging — Logging facility for Python
@anchor{library/logging filter}@anchor{254a}@anchor{library/logging filter-objects}@anchor{254b}
@subsubsection Filter Objects


@code{Filters} can be used by @code{Handlers} and @code{Loggers} for more sophisticated
filtering than is provided by levels. The base filter class only allows events
which are below a certain point in the logger hierarchy. For example, a filter
initialized with ‘A.B’ will allow events logged by loggers ‘A.B’, ‘A.B.C’,
‘A.B.C.D’, ‘A.B.D’ etc. but not ‘A.BB’, ‘B.A.B’ etc. If initialized with the
empty string, all events are passed.

@geindex Filter (class in logging)
@anchor{library/logging logging Filter}@anchor{d9c}
@deffn {Class} logging.Filter (name='@w{'})

Returns an instance of the @ref{d9c,,Filter} class. If @emph{name} is specified, it
names a logger which, together with its children, will have its events allowed
through the filter. If @emph{name} is the empty string, allows every event.

@geindex filter() (logging.Filter method)
@anchor{library/logging logging Filter filter}@anchor{254c}
@deffn {Method} filter (record)

Is the specified record to be logged? Returns zero for no, nonzero for
yes. If deemed appropriate, the record may be modified in-place by this
method.
@end deffn
@end deffn

Note that filters attached to handlers are consulted before an event is
emitted by the handler, whereas filters attached to loggers are consulted
whenever an event is logged (using @ref{2523,,debug()}, @ref{254d,,info()},
etc.), before sending an event to handlers. This means that events which have
been generated by descendant loggers will not be filtered by a logger’s filter
setting, unless the filter has also been applied to those descendant loggers.

You don’t actually need to subclass @code{Filter}: you can pass any instance
which has a @code{filter} method with the same semantics.


Changed in version 3.2: You don’t need to create specialized @code{Filter} classes, or use other
classes with a @code{filter} method: you can use a function (or other
callable) as a filter. The filtering logic will check to see if the filter
object has a @code{filter} attribute: if it does, it’s assumed to be a
@code{Filter} and its @ref{254c,,filter()} method is called. Otherwise, it’s
assumed to be a callable and called with the record as the single
parameter. The returned value should conform to that returned by
@ref{254c,,filter()}.


Although filters are used primarily to filter records based on more
sophisticated criteria than levels, they get to see every record which is
processed by the handler or logger they’re attached to: this can be useful if
you want to do things like counting how many records were processed by a
particular logger or handler, or adding, changing or removing attributes in
the @ref{b6c,,LogRecord} being processed. Obviously changing the LogRecord needs
to be done with some care, but it does allow the injection of contextual
information into logs (see @ref{254e,,Using Filters to impart contextual information}).

@node LogRecord Objects,LogRecord attributes,Filter Objects,logging — Logging facility for Python
@anchor{library/logging log-record}@anchor{254f}@anchor{library/logging logrecord-objects}@anchor{2550}
@subsubsection LogRecord Objects


@ref{b6c,,LogRecord} instances are created automatically by the @ref{676,,Logger}
every time something is logged, and can be created manually via
@ref{2551,,makeLogRecord()} (for example, from a pickled event received over the
wire).

@geindex LogRecord (class in logging)
@anchor{library/logging logging LogRecord}@anchor{b6c}
@deffn {Class} logging.LogRecord (name, level, pathname, lineno, msg, args, exc_info, func=None, sinfo=None)

Contains all the information pertinent to the event being logged.

The primary information is passed in @code{msg} and @code{args}, which
are combined using @code{msg % args} to create the @code{message} field of the
record.

@*Parameters: 

@itemize *

@item 
@code{name} – The name of the logger used to log the event represented by
this LogRecord. Note that this name will always have this
value, even though it may be emitted by a handler attached to
a different (ancestor) logger.

@item 
@code{level} – The numeric level of the logging event (one of DEBUG, INFO etc.)
Note that this is converted to @emph{two} attributes of the LogRecord:
@code{levelno} for the numeric value and @code{levelname} for the
corresponding level name.

@item 
@code{pathname} – The full pathname of the source file where the logging call
was made.

@item 
@code{lineno} – The line number in the source file where the logging call was
made.

@item 
@code{msg} – The event description message, possibly a format string with
placeholders for variable data.

@item 
@code{args} – Variable data to merge into the @emph{msg} argument to obtain the
event description.

@item 
@code{exc_info} – An exception tuple with the current exception information,
or @code{None} if no exception information is available.

@item 
@code{func} – The name of the function or method from which the logging call
was invoked.

@item 
@code{sinfo} – A text string representing stack information from the base of
the stack in the current thread, up to the logging call.
@end itemize


@geindex getMessage() (logging.LogRecord method)
@anchor{library/logging logging LogRecord getMessage}@anchor{2552}
@deffn {Method} getMessage ()

Returns the message for this @ref{b6c,,LogRecord} instance after merging any
user-supplied arguments with the message. If the user-supplied message
argument to the logging call is not a string, @ref{2ef,,str()} is called on it to
convert it to a string. This allows use of user-defined classes as
messages, whose @code{__str__} method can return the actual format string to
be used.
@end deffn


Changed in version 3.2: The creation of a @ref{b6c,,LogRecord} has been made more configurable by
providing a factory which is used to create the record. The factory can be
set using @ref{2553,,getLogRecordFactory()} and @ref{2554,,setLogRecordFactory()}
(see this for the factory’s signature).


This functionality can be used to inject your own values into a
@ref{b6c,,LogRecord} at creation time. You can use the following pattern:

@example
old_factory = logging.getLogRecordFactory()

def record_factory(*args, **kwargs):
    record = old_factory(*args, **kwargs)
    record.custom_attribute = 0xdecafbad
    return record

logging.setLogRecordFactory(record_factory)
@end example

With this pattern, multiple factories could be chained, and as long
as they don’t overwrite each other’s attributes or unintentionally
overwrite the standard attributes listed above, there should be no
surprises.
@end deffn

@node LogRecord attributes,LoggerAdapter Objects,LogRecord Objects,logging — Logging facility for Python
@anchor{library/logging id2}@anchor{2555}@anchor{library/logging logrecord-attributes}@anchor{2544}
@subsubsection LogRecord attributes


The LogRecord has a number of attributes, most of which are derived from the
parameters to the constructor. (Note that the names do not always correspond
exactly between the LogRecord constructor parameters and the LogRecord
attributes.) These attributes can be used to merge data from the record into
the format string. The following table lists (in alphabetical order) the
attribute names, their meanings and the corresponding placeholder in a %-style
format string.

If you are using @{@}-formatting (@ref{784,,str.format()}), you can use
@code{@{attrname@}} as the placeholder in the format string. If you are using
$-formatting (@ref{6b5,,string.Template}), use the form @code{$@{attrname@}}. In
both cases, of course, replace @code{attrname} with the actual attribute name
you want to use.

In the case of @{@}-formatting, you can specify formatting flags by placing them
after the attribute name, separated from it with a colon. For example: a
placeholder of @code{@{msecs:03d@}} would format a millisecond value of @code{4} as
@code{004}. Refer to the @ref{784,,str.format()} documentation for full details on
the options available to you.


@multitable {xxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Attribute name

@tab

Format

@tab

Description

@item

args

@tab

You shouldn’t need to
format this yourself.

@tab

The tuple of arguments merged into @code{msg} to
produce @code{message}, or a dict whose values
are used for the merge (when there is only one
argument, and it is a dictionary).

@item

asctime

@tab

@code{%(asctime)s}

@tab

Human-readable time when the
@ref{b6c,,LogRecord} was created.  By default
this is of the form ‘2003-07-08 16:49:45,896’
(the numbers after the comma are millisecond
portion of the time).

@item

created

@tab

@code{%(created)f}

@tab

Time when the @ref{b6c,,LogRecord} was created
(as returned by @ref{5fc,,time.time()}).

@item

exc_info

@tab

You shouldn’t need to
format this yourself.

@tab

Exception tuple (à la @code{sys.exc_info}) or,
if no exception has occurred, @code{None}.

@item

filename

@tab

@code{%(filename)s}

@tab

Filename portion of @code{pathname}.

@item

funcName

@tab

@code{%(funcName)s}

@tab

Name of function containing the logging call.

@item

levelname

@tab

@code{%(levelname)s}

@tab

Text logging level for the message
(@code{'DEBUG'}, @code{'INFO'}, @code{'WARNING'},
@code{'ERROR'}, @code{'CRITICAL'}).

@item

levelno

@tab

@code{%(levelno)s}

@tab

Numeric logging level for the message
(@code{DEBUG}, @code{INFO},
@code{WARNING}, @code{ERROR},
@code{CRITICAL}).

@item

lineno

@tab

@code{%(lineno)d}

@tab

Source line number where the logging call was
issued (if available).

@item

message

@tab

@code{%(message)s}

@tab

The logged message, computed as @code{msg %
args}. This is set when
@ref{2547,,Formatter.format()} is invoked.

@item

module

@tab

@code{%(module)s}

@tab

Module (name portion of @code{filename}).

@item

msecs

@tab

@code{%(msecs)d}

@tab

Millisecond portion of the time when the
@ref{b6c,,LogRecord} was created.

@item

msg

@tab

You shouldn’t need to
format this yourself.

@tab

The format string passed in the original
logging call. Merged with @code{args} to
produce @code{message}, or an arbitrary object
(see @ref{2556,,Using arbitrary objects as messages}).

@item

name

@tab

@code{%(name)s}

@tab

Name of the logger used to log the call.

@item

pathname

@tab

@code{%(pathname)s}

@tab

Full pathname of the source file where the
logging call was issued (if available).

@item

process

@tab

@code{%(process)d}

@tab

Process ID (if available).

@item

processName

@tab

@code{%(processName)s}

@tab

Process name (if available).

@item

relativeCreated

@tab

@code{%(relativeCreated)d}

@tab

Time in milliseconds when the LogRecord was
created, relative to the time the logging
module was loaded.

@item

stack_info

@tab

You shouldn’t need to
format this yourself.

@tab

Stack frame information (where available)
from the bottom of the stack in the current
thread, up to and including the stack frame
of the logging call which resulted in the
creation of this record.

@item

thread

@tab

@code{%(thread)d}

@tab

Thread ID (if available).

@item

threadName

@tab

@code{%(threadName)s}

@tab

Thread name (if available).

@end multitable



Changed in version 3.1: @emph{processName} was added.


@node LoggerAdapter Objects,Thread Safety,LogRecord attributes,logging — Logging facility for Python
@anchor{library/logging logger-adapter}@anchor{2557}@anchor{library/logging loggeradapter-objects}@anchor{2558}
@subsubsection LoggerAdapter Objects


@ref{e93,,LoggerAdapter} instances are used to conveniently pass contextual
information into logging calls. For a usage example, see the section on
@ref{2559,,adding contextual information to your logging output}.

@geindex LoggerAdapter (class in logging)
@anchor{library/logging logging LoggerAdapter}@anchor{e93}
@deffn {Class} logging.LoggerAdapter (logger, extra)

Returns an instance of @ref{e93,,LoggerAdapter} initialized with an
underlying @ref{676,,Logger} instance and a dict-like object.

@geindex process() (logging.LoggerAdapter method)
@anchor{library/logging logging LoggerAdapter process}@anchor{255a}
@deffn {Method} process (msg, kwargs)

Modifies the message and/or keyword arguments passed to a logging call in
order to insert contextual information. This implementation takes the object
passed as @emph{extra} to the constructor and adds it to @emph{kwargs} using key
‘extra’. The return value is a (@emph{msg}, @emph{kwargs}) tuple which has the
(possibly modified) versions of the arguments passed in.
@end deffn
@end deffn

In addition to the above, @ref{e93,,LoggerAdapter} supports the following
methods of @ref{676,,Logger}: @ref{991,,debug()}, @ref{2522,,info()},
@ref{2524,,warning()}, @ref{2525,,error()}, @ref{98f,,exception()},
@ref{990,,critical()}, @ref{98e,,log()}, @ref{2520,,isEnabledFor()},
@ref{251f,,getEffectiveLevel()}, @ref{251d,,setLevel()} and
@ref{252e,,hasHandlers()}. These methods have the same signatures as their
counterparts in @ref{676,,Logger}, so you can use the two types of instances
interchangeably.


Changed in version 3.2: The @ref{2520,,isEnabledFor()}, @ref{251f,,getEffectiveLevel()},
@ref{251d,,setLevel()} and @ref{252e,,hasHandlers()} methods were added
to @ref{e93,,LoggerAdapter}.  These methods delegate to the underlying logger.



Changed in version 3.6: Attribute @code{manager} and method @code{_log()} were added, which
delegate to the underlying logger and allow adapters to be nested.


@node Thread Safety,Module-Level Functions,LoggerAdapter Objects,logging — Logging facility for Python
@anchor{library/logging thread-safety}@anchor{255b}
@subsubsection Thread Safety


The logging module is intended to be thread-safe without any special work
needing to be done by its clients. It achieves this though using threading
locks; there is one lock to serialize access to the module’s shared data, and
each handler also creates a lock to serialize access to its underlying I/O.

If you are implementing asynchronous signal handlers using the @ref{e5,,signal}
module, you may not be able to use logging from within such handlers. This is
because lock implementations in the @ref{109,,threading} module are not always
re-entrant, and so cannot be invoked from such signal handlers.

@node Module-Level Functions,Module-Level Attributes,Thread Safety,logging — Logging facility for Python
@anchor{library/logging module-level-functions}@anchor{255c}
@subsubsection Module-Level Functions


In addition to the classes described above, there are a number of module-level
functions.

@geindex getLogger() (in module logging)
@anchor{library/logging logging getLogger}@anchor{463}
@deffn {Function} logging.getLogger (name=None)

Return a logger with the specified name or, if name is @code{None}, return a
logger which is the root logger of the hierarchy. If specified, the name is
typically a dot-separated hierarchical name like @emph{‘a’}, @emph{‘a.b’} or @emph{‘a.b.c.d’}.
Choice of these names is entirely up to the developer who is using logging.

All calls to this function with a given name return the same logger instance.
This means that logger instances never need to be passed between different parts
of an application.
@end deffn

@geindex getLoggerClass() (in module logging)
@anchor{library/logging logging getLoggerClass}@anchor{255d}
@deffn {Function} logging.getLoggerClass ()

Return either the standard @ref{676,,Logger} class, or the last class passed to
@ref{255e,,setLoggerClass()}. This function may be called from within a new class
definition, to ensure that installing a customized @ref{676,,Logger} class will
not undo customizations already applied by other code. For example:

@example
class MyLogger(logging.getLoggerClass()):
    # ... override behaviour here
@end example
@end deffn

@geindex getLogRecordFactory() (in module logging)
@anchor{library/logging logging getLogRecordFactory}@anchor{2553}
@deffn {Function} logging.getLogRecordFactory ()

Return a callable which is used to create a @ref{b6c,,LogRecord}.


New in version 3.2: This function has been provided, along with @ref{2554,,setLogRecordFactory()},
to allow developers more control over how the @ref{b6c,,LogRecord}
representing a logging event is constructed.


See @ref{2554,,setLogRecordFactory()} for more information about the how the
factory is called.
@end deffn

@geindex debug() (in module logging)
@anchor{library/logging logging debug}@anchor{2523}
@deffn {Function} logging.debug (msg, *args, **kwargs)

Logs a message with level @code{DEBUG} on the root logger. The @emph{msg} is the
message format string, and the @emph{args} are the arguments which are merged into
@emph{msg} using the string formatting operator. (Note that this means that you can
use keywords in the format string, together with a single dictionary argument.)

There are three keyword arguments in @emph{kwargs} which are inspected: @emph{exc_info}
which, if it does not evaluate as false, causes exception information to be
added to the logging message. If an exception tuple (in the format returned by
@ref{18f,,sys.exc_info()}) or an exception instance is provided, it is used;
otherwise, @ref{18f,,sys.exc_info()} is called to get the exception information.

The second optional keyword argument is @emph{stack_info}, which defaults to
@code{False}. If true, stack information is added to the logging
message, including the actual logging call. Note that this is not the same
stack information as that displayed through specifying @emph{exc_info}: The
former is stack frames from the bottom of the stack up to the logging call
in the current thread, whereas the latter is information about stack frames
which have been unwound, following an exception, while searching for
exception handlers.

You can specify @emph{stack_info} independently of @emph{exc_info}, e.g. to just show
how you got to a certain point in your code, even when no exceptions were
raised. The stack frames are printed following a header line which says:

@example
Stack (most recent call last):
@end example

This mimics the @code{Traceback (most recent call last):} which is used when
displaying exception frames.

The third optional keyword argument is @emph{extra} which can be used to pass a
dictionary which is used to populate the __dict__ of the LogRecord created for
the logging event with user-defined attributes. These custom attributes can then
be used as you like. For example, they could be incorporated into logged
messages. For example:

@example
FORMAT = '%(asctime)s %(clientip)-15s %(user)-8s %(message)s'
logging.basicConfig(format=FORMAT)
d = @{'clientip': '192.168.0.1', 'user': 'fbloggs'@}
logging.warning('Protocol problem: %s', 'connection reset', extra=d)
@end example

would print something like:

@example
2006-02-08 22:20:02,165 192.168.0.1 fbloggs  Protocol problem: connection reset
@end example

The keys in the dictionary passed in @emph{extra} should not clash with the keys used
by the logging system. (See the @ref{11c6,,Formatter} documentation for more
information on which keys are used by the logging system.)

If you choose to use these attributes in logged messages, you need to exercise
some care. In the above example, for instance, the @ref{11c6,,Formatter} has been
set up with a format string which expects ‘clientip’ and ‘user’ in the attribute
dictionary of the LogRecord. If these are missing, the message will not be
logged because a string formatting exception will occur. So in this case, you
always need to pass the @emph{extra} dictionary with these keys.

While this might be annoying, this feature is intended for use in specialized
circumstances, such as multi-threaded servers where the same code executes in
many contexts, and interesting conditions which arise are dependent on this
context (such as remote client IP address and authenticated user name, in the
above example). In such circumstances, it is likely that specialized
@ref{11c6,,Formatter}s would be used with particular @ref{2521,,Handler}s.


Changed in version 3.2: The @emph{stack_info} parameter was added.

@end deffn

@geindex info() (in module logging)
@anchor{library/logging logging info}@anchor{254d}
@deffn {Function} logging.info (msg, *args, **kwargs)

Logs a message with level @code{INFO} on the root logger. The arguments are
interpreted as for @ref{2523,,debug()}.
@end deffn

@geindex warning() (in module logging)
@anchor{library/logging logging warning}@anchor{255f}
@deffn {Function} logging.warning (msg, *args, **kwargs)

Logs a message with level @code{WARNING} on the root logger. The arguments
are interpreted as for @ref{2523,,debug()}.

@cartouche
@quotation Note 
There is an obsolete function @code{warn} which is functionally
identical to @code{warning}. As @code{warn} is deprecated, please do not use
it - use @code{warning} instead.
@end quotation
@end cartouche
@end deffn

@geindex error() (in module logging)
@anchor{library/logging logging error}@anchor{2560}
@deffn {Function} logging.error (msg, *args, **kwargs)

Logs a message with level @code{ERROR} on the root logger. The arguments are
interpreted as for @ref{2523,,debug()}.
@end deffn

@geindex critical() (in module logging)
@anchor{library/logging logging critical}@anchor{2561}
@deffn {Function} logging.critical (msg, *args, **kwargs)

Logs a message with level @code{CRITICAL} on the root logger. The arguments
are interpreted as for @ref{2523,,debug()}.
@end deffn

@geindex exception() (in module logging)
@anchor{library/logging logging exception}@anchor{2562}
@deffn {Function} logging.exception (msg, *args, **kwargs)

Logs a message with level @code{ERROR} on the root logger. The arguments are
interpreted as for @ref{2523,,debug()}. Exception info is added to the logging
message. This function should only be called from an exception handler.
@end deffn

@geindex log() (in module logging)
@anchor{library/logging logging log}@anchor{2563}
@deffn {Function} logging.log (level, msg, *args, **kwargs)

Logs a message with level @emph{level} on the root logger. The other arguments are
interpreted as for @ref{2523,,debug()}.

@cartouche
@quotation Note 
The above module-level convenience functions, which delegate to the
root logger, call @ref{508,,basicConfig()} to ensure that at least one handler
is available. Because of this, they should @emph{not} be used in threads,
in versions of Python earlier than 2.7.1 and 3.2, unless at least one
handler has been added to the root logger @emph{before} the threads are
started. In earlier versions of Python, due to a thread safety shortcoming
in @ref{508,,basicConfig()}, this can (under rare circumstances) lead to
handlers being added multiple times to the root logger, which can in turn
lead to multiple messages for the same event.
@end quotation
@end cartouche
@end deffn

@geindex disable() (in module logging)
@anchor{library/logging logging disable}@anchor{2564}
@deffn {Function} logging.disable (level=CRITICAL)

Provides an overriding level @emph{level} for all loggers which takes precedence over
the logger’s own level. When the need arises to temporarily throttle logging
output down across the whole application, this function can be useful. Its
effect is to disable all logging calls of severity @emph{level} and below, so that
if you call it with a value of INFO, then all INFO and DEBUG events would be
discarded, whereas those of severity WARNING and above would be processed
according to the logger’s effective level. If
@code{logging.disable(logging.NOTSET)} is called, it effectively removes this
overriding level, so that logging output again depends on the effective
levels of individual loggers.

Note that if you have defined any custom logging level higher than
@code{CRITICAL} (this is not recommended), you won’t be able to rely on the
default value for the @emph{level} parameter, but will have to explicitly supply a
suitable value.


Changed in version 3.7: The @emph{level} parameter was defaulted to level @code{CRITICAL}. See
bpo-28524@footnote{https://bugs.python.org/issue28524} for more information about this change.

@end deffn

@geindex addLevelName() (in module logging)
@anchor{library/logging logging addLevelName}@anchor{2565}
@deffn {Function} logging.addLevelName (level, levelName)

Associates level @emph{level} with text @emph{levelName} in an internal dictionary, which is
used to map numeric levels to a textual representation, for example when a
@ref{11c6,,Formatter} formats a message. This function can also be used to define
your own levels. The only constraints are that all levels used must be
registered using this function, levels should be positive integers and they
should increase in increasing order of severity.

@cartouche
@quotation Note 
If you are thinking of defining your own levels, please see the
section on @ref{2566,,Custom Levels}.
@end quotation
@end cartouche
@end deffn

@geindex getLevelNamesMapping() (in module logging)
@anchor{library/logging logging getLevelNamesMapping}@anchor{110e}
@deffn {Function} logging.getLevelNamesMapping ()

Returns a mapping from level names to their corresponding logging levels. For example, the
string “CRITICAL” maps to @code{CRITICAL}. The returned mapping is copied from an internal
mapping on each call to this function.


New in version 3.11.

@end deffn

@geindex getLevelName() (in module logging)
@anchor{library/logging logging getLevelName}@anchor{2567}
@deffn {Function} logging.getLevelName (level)

Returns the textual or numeric representation of logging level @emph{level}.

If @emph{level} is one of the predefined levels @code{CRITICAL}, @code{ERROR},
@code{WARNING}, @code{INFO} or @code{DEBUG} then you get the
corresponding string. If you have associated levels with names using
@ref{2565,,addLevelName()} then the name you have associated with @emph{level} is
returned. If a numeric value corresponding to one of the defined levels is
passed in, the corresponding string representation is returned.

The @emph{level} parameter also accepts a string representation of the level such
as ‘INFO’. In such cases, this functions returns the corresponding numeric
value of the level.

If no matching numeric or string value is passed in, the string
‘Level %s’ % level is returned.

@cartouche
@quotation Note 
Levels are internally integers (as they need to be compared in the
logging logic). This function is used to convert between an integer level
and the level name displayed in the formatted log output by means of the
@code{%(levelname)s} format specifier (see @ref{2544,,LogRecord attributes}), and
vice versa.
@end quotation
@end cartouche


Changed in version 3.4: In Python versions earlier than 3.4, this function could also be passed a
text level, and would return the corresponding numeric value of the level.
This undocumented behaviour was considered a mistake, and was removed in
Python 3.4, but reinstated in 3.4.2 due to retain backward compatibility.

@end deffn

@geindex makeLogRecord() (in module logging)
@anchor{library/logging logging makeLogRecord}@anchor{2551}
@deffn {Function} logging.makeLogRecord (attrdict)

Creates and returns a new @ref{b6c,,LogRecord} instance whose attributes are
defined by @emph{attrdict}. This function is useful for taking a pickled
@ref{b6c,,LogRecord} attribute dictionary, sent over a socket, and reconstituting
it as a @ref{b6c,,LogRecord} instance at the receiving end.
@end deffn

@geindex basicConfig() (in module logging)
@anchor{library/logging logging basicConfig}@anchor{508}
@deffn {Function} logging.basicConfig (**kwargs)

Does basic configuration for the logging system by creating a
@ref{d9a,,StreamHandler} with a default @ref{11c6,,Formatter} and adding it to the
root logger. The functions @ref{2523,,debug()}, @ref{254d,,info()}, @ref{255f,,warning()},
@ref{2560,,error()} and @ref{2561,,critical()} will call @ref{508,,basicConfig()} automatically
if no handlers are defined for the root logger.

This function does nothing if the root logger already has handlers
configured, unless the keyword argument @emph{force} is set to @code{True}.

@cartouche
@quotation Note 
This function should be called from the main thread
before other threads are started. In versions of Python prior to
2.7.1 and 3.2, if this function is called from multiple threads,
it is possible (in rare circumstances) that a handler will be added
to the root logger more than once, leading to unexpected results
such as messages being duplicated in the log.
@end quotation
@end cartouche

The following keyword arguments are supported.


@multitable {xxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Format

@tab

Description

@item

@emph{filename}

@tab

Specifies that a @ref{1104,,FileHandler} be
created, using the specified filename,
rather than a @ref{d9a,,StreamHandler}.

@item

@emph{filemode}

@tab

If @emph{filename} is specified, open the file
in this @ref{1a54,,mode}. Defaults
to @code{'a'}.

@item

@emph{format}

@tab

Use the specified format string for the
handler. Defaults to attributes
@code{levelname}, @code{name} and @code{message}
separated by colons.

@item

@emph{datefmt}

@tab

Use the specified date/time format, as
accepted by @ref{d8b,,time.strftime()}.

@item

@emph{style}

@tab

If @emph{format} is specified, use this style
for the format string. One of @code{'%'},
@code{'@{'} or @code{'$'} for @ref{14e4,,printf-style},
@ref{784,,str.format()} or
@ref{6b5,,string.Template} respectively.
Defaults to @code{'%'}.

@item

@emph{level}

@tab

Set the root logger level to the specified
@ref{251e,,level}.

@item

@emph{stream}

@tab

Use the specified stream to initialize the
@ref{d9a,,StreamHandler}. Note that this
argument is incompatible with @emph{filename} -
if both are present, a @code{ValueError} is
raised.

@item

@emph{handlers}

@tab

If specified, this should be an iterable of
already created handlers to add to the root
logger. Any handlers which don’t already
have a formatter set will be assigned the
default formatter created in this function.
Note that this argument is incompatible
with @emph{filename} or @emph{stream} - if both
are present, a @code{ValueError} is raised.

@item

@emph{force}

@tab

If this keyword argument is specified as
true, any existing handlers attached to the
root logger are removed and closed, before
carrying out the configuration as specified
by the other arguments.

@item

@emph{encoding}

@tab

If this keyword argument is specified along
with @emph{filename}, its value is used when the
@ref{1104,,FileHandler} is created, and thus
used when opening the output file.

@item

@emph{errors}

@tab

If this keyword argument is specified along
with @emph{filename}, its value is used when the
@ref{1104,,FileHandler} is created, and thus
used when opening the output file. If not
specified, the value ‘backslashreplace’ is
used. Note that if @code{None} is specified,
it will be passed as such to @ref{1c8,,open()},
which means that it will be treated the
same as passing ‘errors’.

@end multitable



Changed in version 3.2: The @emph{style} argument was added.



Changed in version 3.3: The @emph{handlers} argument was added. Additional checks were added to
catch situations where incompatible arguments are specified (e.g.
@emph{handlers} together with @emph{stream} or @emph{filename}, or @emph{stream}
together with @emph{filename}).



Changed in version 3.8: The @emph{force} argument was added.



Changed in version 3.9: The @emph{encoding} and @emph{errors} arguments were added.

@end deffn

@geindex shutdown() (in module logging)
@anchor{library/logging logging shutdown}@anchor{253d}
@deffn {Function} logging.shutdown ()

Informs the logging system to perform an orderly shutdown by flushing and
closing all handlers. This should be called at application exit and no
further use of the logging system should be made after this call.

When the logging module is imported, it registers this function as an exit
handler (see @ref{b,,atexit}), so normally there’s no need to do that
manually.
@end deffn

@geindex setLoggerClass() (in module logging)
@anchor{library/logging logging setLoggerClass}@anchor{255e}
@deffn {Function} logging.setLoggerClass (klass)

Tells the logging system to use the class @emph{klass} when instantiating a logger.
The class should define @code{__init__()} such that only a name argument is
required, and the @code{__init__()} should call @code{Logger.__init__()}. This
function is typically called before any loggers are instantiated by applications
which need to use custom logger behavior. After this call, as at any other
time, do not instantiate loggers directly using the subclass: continue to use
the @ref{463,,logging.getLogger()} API to get your loggers.
@end deffn

@geindex setLogRecordFactory() (in module logging)
@anchor{library/logging logging setLogRecordFactory}@anchor{2554}
@deffn {Function} logging.setLogRecordFactory (factory)

Set a callable which is used to create a @ref{b6c,,LogRecord}.

@*Parameters: 
@code{factory} – The factory callable to be used to instantiate a log record.



New in version 3.2: This function has been provided, along with @ref{2553,,getLogRecordFactory()}, to
allow developers more control over how the @ref{b6c,,LogRecord} representing
a logging event is constructed.


The factory has the following signature:

@code{factory(name, level, fn, lno, msg, args, exc_info, func=None, sinfo=None, **kwargs)}

@quotation

@*name: 
The logger name.


@*level: 
The logging level (numeric).


@*fn: 
The full pathname of the file where the logging call was made.


@*lno: 
The line number in the file where the logging call was made.


@*msg: 
The logging message.


@*args: 
The arguments for the logging message.


@*exc_info: 
An exception tuple, or @code{None}.


@*func: 
The name of the function or method which invoked the logging
call.


@*sinfo: 
A stack traceback such as is provided by
@ref{9f8,,traceback.print_stack()}, showing the call hierarchy.


@*kwargs: 
Additional keyword arguments.

@end quotation
@end deffn

@node Module-Level Attributes,Integration with the warnings module,Module-Level Functions,logging — Logging facility for Python
@anchor{library/logging module-level-attributes}@anchor{2568}
@subsubsection Module-Level Attributes


@geindex lastResort (in module logging)
@anchor{library/logging logging lastResort}@anchor{d9b}
@deffn {Attribute} logging.lastResort

A “handler of last resort” is available through this attribute. This
is a @ref{d9a,,StreamHandler} writing to @code{sys.stderr} with a level of
@code{WARNING}, and is used to handle logging events in the absence of any
logging configuration. The end result is to just print the message to
@code{sys.stderr}. This replaces the earlier error message saying that
“no handlers could be found for logger XYZ”. If you need the earlier
behaviour for some reason, @code{lastResort} can be set to @code{None}.


New in version 3.2.

@end deffn

@node Integration with the warnings module,,Module-Level Attributes,logging — Logging facility for Python
@anchor{library/logging integration-with-the-warnings-module}@anchor{2569}
@subsubsection Integration with the warnings module


The @ref{256a,,captureWarnings()} function can be used to integrate @ref{a6,,logging}
with the @ref{12d,,warnings} module.

@geindex captureWarnings() (in module logging)
@anchor{library/logging logging captureWarnings}@anchor{256a}
@deffn {Function} logging.captureWarnings (capture)

This function is used to turn the capture of warnings by logging on and
off.

If @emph{capture} is @code{True}, warnings issued by the @ref{12d,,warnings} module will
be redirected to the logging system. Specifically, a warning will be
formatted using @ref{12c0,,warnings.formatwarning()} and the resulting string
logged to a logger named @code{'py.warnings'} with a severity of @code{WARNING}.

If @emph{capture} is @code{False}, the redirection of warnings to the logging system
will stop, and warnings will be redirected to their original destinations
(i.e. those in effect before @code{captureWarnings(True)} was called).
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{a7,,logging.config}

Configuration API for the logging module.

@item Module @ref{a8,,logging.handlers}

Useful handlers included with the logging module.

@geindex Python Enhancement Proposals; PEP 282

@item PEP 282@footnote{https://www.python.org/dev/peps/pep-0282} - A Logging System

The proposal which described this feature for inclusion in the Python standard
library.

@item Original Python logging package@footnote{https://old.red-dove.com/python_logging.html}

This is the original source for the @ref{a6,,logging} package.  The version of the
package available from this site is suitable for use with Python 1.5.2, 2.1.x
and 2.2.x, which do not include the @ref{a6,,logging} package in the standard
library.
@end table


@node logging config — Logging configuration,logging handlers — Logging handlers,logging — Logging facility for Python,Generic Operating System Services
@anchor{library/logging config doc}@anchor{256b}@anchor{library/logging config logging-config-logging-configuration}@anchor{256c}@anchor{library/logging config module-logging config}@anchor{a7}
@subsection @code{logging.config} — Logging configuration


@geindex module; logging.config

@strong{Source code:} Lib/logging/config.py@footnote{https://github.com/python/cpython/tree/main/Lib/logging/config.py}

@subsubheading Important



This page contains only reference information. For tutorials,
please see


@itemize *

@item 
@ref{d97,,Basic Tutorial}

@item 
@ref{d98,,Advanced Tutorial}

@item 
@ref{d99,,Logging Cookbook}
@end itemize


__________________________________________________________________


This section describes the API for configuring the logging module.

@menu
* Configuration functions:: 
* Security considerations:: 
* Configuration dictionary schema:: 
* Configuration file format:: 

@end menu

@node Configuration functions,Security considerations,,logging config — Logging configuration
@anchor{library/logging config configuration-functions}@anchor{256d}@anchor{library/logging config logging-config-api}@anchor{e8f}
@subsubsection Configuration functions


The following functions configure the logging module. They are located in the
@ref{a7,,logging.config} module.  Their use is optional — you can configure the
logging module using these functions or by making calls to the main API (defined
in @ref{a6,,logging} itself) and defining handlers which are declared either in
@ref{a6,,logging} or @ref{a8,,logging.handlers}.

@geindex dictConfig() (in module logging.config)
@anchor{library/logging config logging config dictConfig}@anchor{d55}
@deffn {Function} logging.config.dictConfig (config)

Takes the logging configuration from a dictionary.  The contents of
this dictionary are described in @ref{256e,,Configuration dictionary schema}
below.

If an error is encountered during configuration, this function will
raise a @ref{17c,,ValueError}, @ref{162,,TypeError}, @ref{163,,AttributeError}
or @ref{3b8,,ImportError} with a suitably descriptive message.  The
following is a (possibly incomplete) list of conditions which will
raise an error:


@itemize *

@item 
A @code{level} which is not a string or which is a string not
corresponding to an actual logging level.

@item 
A @code{propagate} value which is not a boolean.

@item 
An id which does not have a corresponding destination.

@item 
A non-existent handler id found during an incremental call.

@item 
An invalid logger name.

@item 
Inability to resolve to an internal or external object.
@end itemize

Parsing is performed by the @code{DictConfigurator} class, whose
constructor is passed the dictionary used for configuration, and
has a @code{configure()} method.  The @ref{a7,,logging.config} module
has a callable attribute @code{dictConfigClass}
which is initially set to @code{DictConfigurator}.
You can replace the value of @code{dictConfigClass} with a
suitable implementation of your own.

@ref{d55,,dictConfig()} calls @code{dictConfigClass} passing
the specified dictionary, and then calls the @code{configure()} method on
the returned object to put the configuration into effect:

@example
def dictConfig(config):
    dictConfigClass(config).configure()
@end example

For example, a subclass of @code{DictConfigurator} could call
@code{DictConfigurator.__init__()} in its own @code{__init__()}, then
set up custom prefixes which would be usable in the subsequent
@code{configure()} call. @code{dictConfigClass} would be bound to
this new subclass, and then @ref{d55,,dictConfig()} could be called exactly as
in the default, uncustomized state.


New in version 3.2.

@end deffn

@geindex fileConfig() (in module logging.config)
@anchor{library/logging config logging config fileConfig}@anchor{678}
@deffn {Function} logging.config.fileConfig (fname, defaults=None, disable_existing_loggers=True, encoding=None)

Reads the logging configuration from a @ref{21,,configparser}-format file. The
format of the file should be as described in
@ref{256f,,Configuration file format}.
This function can be called several times from an application, allowing an
end user to select from various pre-canned configurations (if the developer
provides a mechanism to present the choices and load the chosen
configuration).

@*Parameters: 

@itemize *

@item 
@code{fname} – A filename, or a file-like object, or an instance derived
from @ref{ae2,,RawConfigParser}. If a
@code{RawConfigParser}-derived instance is passed, it is used as
is. Otherwise, a @code{Configparser} is
instantiated, and the configuration read by it from the
object passed in @code{fname}. If that has a @ref{d9,,readline()}
method, it is assumed to be a file-like object and read using
@ref{22f1,,read_file()}; otherwise,
it is assumed to be a filename and passed to
@ref{1366,,read()}.

@item 
@code{defaults} – Defaults to be passed to the ConfigParser can be specified
in this argument.

@item 
@code{disable_existing_loggers} – 

@table @asis

@item If specified as @code{False}, loggers which

exist when this call is made are left
enabled. The default is @code{True} because this
enables old behaviour in a
backward-compatible way. This behaviour is to
disable any existing non-root loggers unless
they or their ancestors are explicitly named
in the logging configuration.
@end table

@*param encoding: 
The encoding used to open file when @emph{fname} is filename.


@end itemize



Changed in version 3.4: An instance of a subclass of @ref{ae2,,RawConfigParser} is
  now accepted as a value for @code{fname}. This facilitates:

@quotation


@itemize *

@item 
Use of a configuration file where logging configuration is just part
of the overall application configuration.

@item 
Use of a configuration read from a file, and then modified by the using
application (e.g. based on command-line parameters or other aspects
of the runtime environment) before being passed to @code{fileConfig}.
@end itemize
@end quotation


New in version 3.10: The @emph{encoding} parameter is added.


@end deffn

@geindex listen() (in module logging.config)
@anchor{library/logging config logging config listen}@anchor{ae3}
@deffn {Function} logging.config.listen (port=DEFAULT_LOGGING_CONFIG_PORT, verify=None)

Starts up a socket server on the specified port, and listens for new
configurations. If no port is specified, the module’s default
@code{DEFAULT_LOGGING_CONFIG_PORT} is used. Logging configurations will be
sent as a file suitable for processing by @ref{d55,,dictConfig()} or
@ref{678,,fileConfig()}. Returns a @ref{44b,,Thread} instance on which
you can call @ref{2570,,start()} to start the server, and which
you can @ref{d86,,join()} when appropriate. To stop the server,
call @ref{2571,,stopListening()}.

The @code{verify} argument, if specified, should be a callable which should
verify whether bytes received across the socket are valid and should be
processed. This could be done by encrypting and/or signing what is sent
across the socket, such that the @code{verify} callable can perform
signature verification and/or decryption. The @code{verify} callable is called
with a single argument - the bytes received across the socket - and should
return the bytes to be processed, or @code{None} to indicate that the bytes should
be discarded. The returned bytes could be the same as the passed in bytes
(e.g. when only verification is done), or they could be completely different
(perhaps if decryption were performed).

To send a configuration to the socket, read in the configuration file and
send it to the socket as a sequence of bytes preceded by a four-byte length
string packed in binary using @code{struct.pack('>L', n)}.
@anchor{library/logging config logging-eval-security}@anchor{2572}
@cartouche
@quotation Note 
Because portions of the configuration are passed through
@ref{34a,,eval()}, use of this function may open its users to a security risk.
While the function only binds to a socket on @code{localhost}, and so does
not accept connections from remote machines, there are scenarios where
untrusted code could be run under the account of the process which calls
@ref{ae3,,listen()}. Specifically, if the process calling @ref{ae3,,listen()} runs
on a multi-user machine where users cannot trust each other, then a
malicious user could arrange to run essentially arbitrary code in a
victim user’s process, simply by connecting to the victim’s
@ref{ae3,,listen()} socket and sending a configuration which runs whatever
code the attacker wants to have executed in the victim’s process. This is
especially easy to do if the default port is used, but not hard even if a
different port is used. To avoid the risk of this happening, use the
@code{verify} argument to @ref{ae3,,listen()} to prevent unrecognised
configurations from being applied.
@end quotation
@end cartouche


Changed in version 3.4: The @code{verify} argument was added.


@cartouche
@quotation Note 
If you want to send configurations to the listener which don’t
disable existing loggers, you will need to use a JSON format for
the configuration, which will use @ref{d55,,dictConfig()} for configuration.
This method allows you to specify @code{disable_existing_loggers} as
@code{False} in the configuration you send.
@end quotation
@end cartouche
@end deffn

@geindex stopListening() (in module logging.config)
@anchor{library/logging config logging config stopListening}@anchor{2571}
@deffn {Function} logging.config.stopListening ()

Stops the listening server which was created with a call to @ref{ae3,,listen()}.
This is typically called before calling @code{join()} on the return value from
@ref{ae3,,listen()}.
@end deffn

@node Security considerations,Configuration dictionary schema,Configuration functions,logging config — Logging configuration
@anchor{library/logging config security-considerations}@anchor{2573}
@subsubsection Security considerations


The logging configuration functionality tries to offer convenience, and in part this
is done by offering the ability to convert text in configuration files into Python
objects used in logging configuration - for example, as described in
@ref{2574,,User-defined objects}. However, these same mechanisms (importing
callables from user-defined modules and calling them with parameters from the
configuration) could be used to invoke any code you like, and for this reason you
should treat configuration files from untrusted sources with @emph{extreme caution} and
satisfy yourself that nothing bad can happen if you load them, before actually loading
them.

@node Configuration dictionary schema,Configuration file format,Security considerations,logging config — Logging configuration
@anchor{library/logging config configuration-dictionary-schema}@anchor{2575}@anchor{library/logging config logging-config-dictschema}@anchor{256e}
@subsubsection Configuration dictionary schema


Describing a logging configuration requires listing the various
objects to create and the connections between them; for example, you
may create a handler named ‘console’ and then say that the logger
named ‘startup’ will send its messages to the ‘console’ handler.
These objects aren’t limited to those provided by the @ref{a6,,logging}
module because you might write your own formatter or handler class.
The parameters to these classes may also need to include external
objects such as @code{sys.stderr}.  The syntax for describing these
objects and connections is defined in @ref{2576,,Object connections}
below.

@menu
* Dictionary Schema Details:: 
* Incremental Configuration:: 
* Object connections:: 
* User-defined objects:: 
* Access to external objects:: 
* Access to internal objects:: 
* Import resolution and custom importers:: 

@end menu

@node Dictionary Schema Details,Incremental Configuration,,Configuration dictionary schema
@anchor{library/logging config dictionary-schema-details}@anchor{2577}
@subsubsection Dictionary Schema Details


The dictionary passed to @ref{d55,,dictConfig()} must contain the following
keys:


@itemize *

@item 
@emph{version} - to be set to an integer value representing the schema
version.  The only valid value at present is 1, but having this key
allows the schema to evolve while still preserving backwards
compatibility.
@end itemize

All other keys are optional, but if present they will be interpreted
as described below.  In all cases below where a ‘configuring dict’ is
mentioned, it will be checked for the special @code{'()'} key to see if a
custom instantiation is required.  If so, the mechanism described in
@ref{2574,,User-defined objects} below is used to create an instance;
otherwise, the context is used to determine what to instantiate.
@anchor{library/logging config logging-config-dictschema-formatters}@anchor{2578}

@itemize *

@item 
@emph{formatters} - the corresponding value will be a dict in which each
key is a formatter id and each value is a dict describing how to
configure the corresponding @ref{11c6,,Formatter} instance.

The configuring dict is searched for the following optional keys
which correspond to the arguments passed to create a
@ref{11c6,,Formatter} object:

@quotation


@itemize *

@item 
@code{format}

@item 
@code{datefmt}

@item 
@code{style}

@item 
@code{validate} (since version >=3.8)
@end itemize
@end quotation

An optional @code{class} key indicates the name of the formatter’s
class (as a dotted module and class name).  The instantiation
arguments are as for @ref{11c6,,Formatter}, thus this key is
most useful for instantiating a customised subclass of
@ref{11c6,,Formatter}.  For example, the alternative class
might present exception tracebacks in an expanded or condensed
format.  If your formatter requires different or extra configuration
keys, you should use @ref{2574,,User-defined objects}.

@item 
@emph{filters} - the corresponding value will be a dict in which each key
is a filter id and each value is a dict describing how to configure
the corresponding Filter instance.

The configuring dict is searched for the key @code{name} (defaulting to the
empty string) and this is used to construct a @ref{d9c,,logging.Filter}
instance.

@item 
@emph{handlers} - the corresponding value will be a dict in which each
key is a handler id and each value is a dict describing how to
configure the corresponding Handler instance.

The configuring dict is searched for the following keys:


@itemize *

@item 
@code{class} (mandatory).  This is the fully qualified name of the
handler class.

@item 
@code{level} (optional).  The level of the handler.

@item 
@code{formatter} (optional).  The id of the formatter for this
handler.

@item 
@code{filters} (optional).  A list of ids of the filters for this
handler.


Changed in version 3.11: @code{filters} can take filter instances in addition to ids.

@end itemize

All @emph{other} keys are passed through as keyword arguments to the
handler’s constructor.  For example, given the snippet:

@example
handlers:
  console:
    class : logging.StreamHandler
    formatter: brief
    level   : INFO
    filters: [allow_foo]
    stream  : ext://sys.stdout
  file:
    class : logging.handlers.RotatingFileHandler
    formatter: precise
    filename: logconfig.log
    maxBytes: 1024
    backupCount: 3
@end example

the handler with id @code{console} is instantiated as a
@ref{d9a,,logging.StreamHandler}, using @code{sys.stdout} as the underlying
stream.  The handler with id @code{file} is instantiated as a
@ref{2579,,logging.handlers.RotatingFileHandler} with the keyword arguments
@code{filename='logconfig.log', maxBytes=1024, backupCount=3}.

@item 
@emph{loggers} - the corresponding value will be a dict in which each key
is a logger name and each value is a dict describing how to
configure the corresponding Logger instance.

The configuring dict is searched for the following keys:


@itemize *

@item 
@code{level} (optional).  The level of the logger.

@item 
@code{propagate} (optional).  The propagation setting of the logger.

@item 
@code{filters} (optional).  A list of ids of the filters for this
logger.


Changed in version 3.11: @code{filters} can take filter instances in addition to ids.


@item 
@code{handlers} (optional).  A list of ids of the handlers for this
logger.
@end itemize

The specified loggers will be configured according to the level,
propagation, filters and handlers specified.

@item 
@emph{root} - this will be the configuration for the root logger.
Processing of the configuration will be as for any logger, except
that the @code{propagate} setting will not be applicable.

@item 
@emph{incremental} - whether the configuration is to be interpreted as
incremental to the existing configuration.  This value defaults to
@code{False}, which means that the specified configuration replaces the
existing configuration with the same semantics as used by the
existing @ref{678,,fileConfig()} API.

If the specified value is @code{True}, the configuration is processed
as described in the section on @ref{257a,,Incremental Configuration}.

@item 
@emph{disable_existing_loggers} - whether any existing non-root loggers are
to be disabled. This setting mirrors the parameter of the same name in
@ref{678,,fileConfig()}. If absent, this parameter defaults to @code{True}.
This value is ignored if @emph{incremental} is @code{True}.
@end itemize

@node Incremental Configuration,Object connections,Dictionary Schema Details,Configuration dictionary schema
@anchor{library/logging config incremental-configuration}@anchor{257b}@anchor{library/logging config logging-config-dict-incremental}@anchor{257a}
@subsubsection Incremental Configuration


It is difficult to provide complete flexibility for incremental
configuration.  For example, because objects such as filters
and formatters are anonymous, once a configuration is set up, it is
not possible to refer to such anonymous objects when augmenting a
configuration.

Furthermore, there is not a compelling case for arbitrarily altering
the object graph of loggers, handlers, filters, formatters at
run-time, once a configuration is set up; the verbosity of loggers and
handlers can be controlled just by setting levels (and, in the case of
loggers, propagation flags).  Changing the object graph arbitrarily in
a safe way is problematic in a multi-threaded environment; while not
impossible, the benefits are not worth the complexity it adds to the
implementation.

Thus, when the @code{incremental} key of a configuration dict is present
and is @code{True}, the system will completely ignore any @code{formatters} and
@code{filters} entries, and process only the @code{level}
settings in the @code{handlers} entries, and the @code{level} and
@code{propagate} settings in the @code{loggers} and @code{root} entries.

Using a value in the configuration dict lets configurations to be sent
over the wire as pickled dicts to a socket listener. Thus, the logging
verbosity of a long-running application can be altered over time with
no need to stop and restart the application.

@node Object connections,User-defined objects,Incremental Configuration,Configuration dictionary schema
@anchor{library/logging config logging-config-dict-connections}@anchor{2576}@anchor{library/logging config object-connections}@anchor{257c}
@subsubsection Object connections


The schema describes a set of logging objects - loggers,
handlers, formatters, filters - which are connected to each other in
an object graph.  Thus, the schema needs to represent connections
between the objects.  For example, say that, once configured, a
particular logger has attached to it a particular handler.  For the
purposes of this discussion, we can say that the logger represents the
source, and the handler the destination, of a connection between the
two.  Of course in the configured objects this is represented by the
logger holding a reference to the handler.  In the configuration dict,
this is done by giving each destination object an id which identifies
it unambiguously, and then using the id in the source object’s
configuration to indicate that a connection exists between the source
and the destination object with that id.

So, for example, consider the following YAML snippet:

@example
formatters:
  brief:
    # configuration for formatter with id 'brief' goes here
  precise:
    # configuration for formatter with id 'precise' goes here
handlers:
  h1: #This is an id
   # configuration of handler with id 'h1' goes here
   formatter: brief
  h2: #This is another id
   # configuration of handler with id 'h2' goes here
   formatter: precise
loggers:
  foo.bar.baz:
    # other configuration for logger 'foo.bar.baz'
    handlers: [h1, h2]
@end example

(Note: YAML used here because it’s a little more readable than the
equivalent Python source form for the dictionary.)

The ids for loggers are the logger names which would be used
programmatically to obtain a reference to those loggers, e.g.
@code{foo.bar.baz}.  The ids for Formatters and Filters can be any string
value (such as @code{brief}, @code{precise} above) and they are transient,
in that they are only meaningful for processing the configuration
dictionary and used to determine connections between objects, and are
not persisted anywhere when the configuration call is complete.

The above snippet indicates that logger named @code{foo.bar.baz} should
have two handlers attached to it, which are described by the handler
ids @code{h1} and @code{h2}. The formatter for @code{h1} is that described by id
@code{brief}, and the formatter for @code{h2} is that described by id
@code{precise}.

@node User-defined objects,Access to external objects,Object connections,Configuration dictionary schema
@anchor{library/logging config logging-config-dict-userdef}@anchor{2574}@anchor{library/logging config user-defined-objects}@anchor{257d}
@subsubsection User-defined objects


The schema supports user-defined objects for handlers, filters and
formatters.  (Loggers do not need to have different types for
different instances, so there is no support in this configuration
schema for user-defined logger classes.)

Objects to be configured are described by dictionaries
which detail their configuration.  In some places, the logging system
will be able to infer from the context how an object is to be
instantiated, but when a user-defined object is to be instantiated,
the system will not know how to do this.  In order to provide complete
flexibility for user-defined object instantiation, the user needs
to provide a ‘factory’ - a callable which is called with a
configuration dictionary and which returns the instantiated object.
This is signalled by an absolute import path to the factory being
made available under the special key @code{'()'}.  Here’s a concrete
example:

@example
formatters:
  brief:
    format: '%(message)s'
  default:
    format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'
    datefmt: '%Y-%m-%d %H:%M:%S'
  custom:
      (): my.package.customFormatterFactory
      bar: baz
      spam: 99.9
      answer: 42
@end example

The above YAML snippet defines three formatters.  The first, with id
@code{brief}, is a standard @ref{11c6,,logging.Formatter} instance with the
specified format string.  The second, with id @code{default}, has a
longer format and also defines the time format explicitly, and will
result in a @ref{11c6,,logging.Formatter} initialized with those two format
strings.  Shown in Python source form, the @code{brief} and @code{default}
formatters have configuration sub-dictionaries:

@example
@{
  'format' : '%(message)s'
@}
@end example

and:

@example
@{
  'format' : '%(asctime)s %(levelname)-8s %(name)-15s %(message)s',
  'datefmt' : '%Y-%m-%d %H:%M:%S'
@}
@end example

respectively, and as these dictionaries do not contain the special key
@code{'()'}, the instantiation is inferred from the context: as a result,
standard @ref{11c6,,logging.Formatter} instances are created.  The
configuration sub-dictionary for the third formatter, with id
@code{custom}, is:

@example
@{
  '()' : 'my.package.customFormatterFactory',
  'bar' : 'baz',
  'spam' : 99.9,
  'answer' : 42
@}
@end example

and this contains the special key @code{'()'}, which means that
user-defined instantiation is wanted.  In this case, the specified
factory callable will be used. If it is an actual callable it will be
used directly - otherwise, if you specify a string (as in the example)
the actual callable will be located using normal import mechanisms.
The callable will be called with the @strong{remaining} items in the
configuration sub-dictionary as keyword arguments.  In the above
example, the formatter with id @code{custom} will be assumed to be
returned by the call:

@example
my.package.customFormatterFactory(bar='baz', spam=99.9, answer=42)
@end example

The key @code{'()'} has been used as the special key because it is not a
valid keyword parameter name, and so will not clash with the names of
the keyword arguments used in the call.  The @code{'()'} also serves as a
mnemonic that the corresponding value is a callable.

@quotation


Changed in version 3.11: The @code{filters} member of @code{handlers} and @code{loggers} can take
filter instances in addition to ids.

@end quotation

@node Access to external objects,Access to internal objects,User-defined objects,Configuration dictionary schema
@anchor{library/logging config access-to-external-objects}@anchor{257e}@anchor{library/logging config logging-config-dict-externalobj}@anchor{257f}
@subsubsection Access to external objects


There are times where a configuration needs to refer to objects
external to the configuration, for example @code{sys.stderr}.  If the
configuration dict is constructed using Python code, this is
straightforward, but a problem arises when the configuration is
provided via a text file (e.g. JSON, YAML).  In a text file, there is
no standard way to distinguish @code{sys.stderr} from the literal string
@code{'sys.stderr'}.  To facilitate this distinction, the configuration
system looks for certain special prefixes in string values and
treat them specially.  For example, if the literal string
@code{'ext://sys.stderr'} is provided as a value in the configuration,
then the @code{ext://} will be stripped off and the remainder of the
value processed using normal import mechanisms.

The handling of such prefixes is done in a way analogous to protocol
handling: there is a generic mechanism to look for prefixes which
match the regular expression @code{^(?P<prefix>[a-z]+)://(?P<suffix>.*)$}
whereby, if the @code{prefix} is recognised, the @code{suffix} is processed
in a prefix-dependent manner and the result of the processing replaces
the string value.  If the prefix is not recognised, then the string
value will be left as-is.

@node Access to internal objects,Import resolution and custom importers,Access to external objects,Configuration dictionary schema
@anchor{library/logging config access-to-internal-objects}@anchor{2580}@anchor{library/logging config logging-config-dict-internalobj}@anchor{2581}
@subsubsection Access to internal objects


As well as external objects, there is sometimes also a need to refer
to objects in the configuration.  This will be done implicitly by the
configuration system for things that it knows about.  For example, the
string value @code{'DEBUG'} for a @code{level} in a logger or handler will
automatically be converted to the value @code{logging.DEBUG}, and the
@code{handlers}, @code{filters} and @code{formatter} entries will take an
object id and resolve to the appropriate destination object.

However, a more generic mechanism is needed for user-defined
objects which are not known to the @ref{a6,,logging} module.  For
example, consider @ref{2582,,logging.handlers.MemoryHandler}, which takes
a @code{target} argument which is another handler to delegate to. Since
the system already knows about this class, then in the configuration,
the given @code{target} just needs to be the object id of the relevant
target handler, and the system will resolve to the handler from the
id.  If, however, a user defines a @code{my.package.MyHandler} which has
an @code{alternate} handler, the configuration system would not know that
the @code{alternate} referred to a handler.  To cater for this, a generic
resolution system allows the user to specify:

@example
handlers:
  file:
    # configuration of file handler goes here

  custom:
    (): my.package.MyHandler
    alternate: cfg://handlers.file
@end example

The literal string @code{'cfg://handlers.file'} will be resolved in an
analogous way to strings with the @code{ext://} prefix, but looking
in the configuration itself rather than the import namespace.  The
mechanism allows access by dot or by index, in a similar way to
that provided by @code{str.format}.  Thus, given the following snippet:

@example
handlers:
  email:
    class: logging.handlers.SMTPHandler
    mailhost: localhost
    fromaddr: my_app@@domain.tld
    toaddrs:
      - support_team@@domain.tld
      - dev_team@@domain.tld
    subject: Houston, we have a problem.
@end example

in the configuration, the string @code{'cfg://handlers'} would resolve to
the dict with key @code{handlers}, the string @code{'cfg://handlers.email}
would resolve to the dict with key @code{email} in the @code{handlers} dict,
and so on.  The string @code{'cfg://handlers.email.toaddrs[1]} would
resolve to @code{'dev_team@@domain.tld'} and the string
@code{'cfg://handlers.email.toaddrs[0]'} would resolve to the value
@code{'support_team@@domain.tld'}. The @code{subject} value could be accessed
using either @code{'cfg://handlers.email.subject'} or, equivalently,
@code{'cfg://handlers.email[subject]'}.  The latter form only needs to be
used if the key contains spaces or non-alphanumeric characters.  If an
index value consists only of decimal digits, access will be attempted
using the corresponding integer value, falling back to the string
value if needed.

Given a string @code{cfg://handlers.myhandler.mykey.123}, this will
resolve to @code{config_dict['handlers']['myhandler']['mykey']['123']}.
If the string is specified as @code{cfg://handlers.myhandler.mykey[123]},
the system will attempt to retrieve the value from
@code{config_dict['handlers']['myhandler']['mykey'][123]}, and fall back
to @code{config_dict['handlers']['myhandler']['mykey']['123']} if that
fails.

@node Import resolution and custom importers,,Access to internal objects,Configuration dictionary schema
@anchor{library/logging config import-resolution-and-custom-importers}@anchor{2583}@anchor{library/logging config logging-import-resolution}@anchor{2584}
@subsubsection Import resolution and custom importers


Import resolution, by default, uses the builtin @ref{3b7,,__import__()} function
to do its importing. You may want to replace this with your own importing
mechanism: if so, you can replace the @code{importer} attribute of the
@code{DictConfigurator} or its superclass, the
@code{BaseConfigurator} class. However, you need to be
careful because of the way functions are accessed from classes via
descriptors. If you are using a Python callable to do your imports, and you
want to define it at class level rather than instance level, you need to wrap
it with @ref{253,,staticmethod()}. For example:

@example
from importlib import import_module
from logging.config import BaseConfigurator

BaseConfigurator.importer = staticmethod(import_module)
@end example

You don’t need to wrap with @ref{253,,staticmethod()} if you’re setting the import
callable on a configurator @emph{instance}.

@node Configuration file format,,Configuration dictionary schema,logging config — Logging configuration
@anchor{library/logging config configuration-file-format}@anchor{2585}@anchor{library/logging config logging-config-fileformat}@anchor{256f}
@subsubsection Configuration file format


The configuration file format understood by @ref{678,,fileConfig()} is based on
@ref{21,,configparser} functionality. The file must contain sections called
@code{[loggers]}, @code{[handlers]} and @code{[formatters]} which identify by name the
entities of each type which are defined in the file. For each such entity, there
is a separate section which identifies how that entity is configured.  Thus, for
a logger named @code{log01} in the @code{[loggers]} section, the relevant
configuration details are held in a section @code{[logger_log01]}. Similarly, a
handler called @code{hand01} in the @code{[handlers]} section will have its
configuration held in a section called @code{[handler_hand01]}, while a formatter
called @code{form01} in the @code{[formatters]} section will have its configuration
specified in a section called @code{[formatter_form01]}. The root logger
configuration must be specified in a section called @code{[logger_root]}.

@cartouche
@quotation Note 
The @ref{678,,fileConfig()} API is older than the @ref{d55,,dictConfig()} API and does
not provide functionality to cover certain aspects of logging. For example,
you cannot configure @ref{d9c,,Filter} objects, which provide for
filtering of messages beyond simple integer levels, using @ref{678,,fileConfig()}.
If you need to have instances of @ref{d9c,,Filter} in your logging
configuration, you will need to use @ref{d55,,dictConfig()}. Note that future
enhancements to configuration functionality will be added to
@ref{d55,,dictConfig()}, so it’s worth considering transitioning to this newer
API when it’s convenient to do so.
@end quotation
@end cartouche

Examples of these sections in the file are given below.

@example
[loggers]
keys=root,log02,log03,log04,log05,log06,log07

[handlers]
keys=hand01,hand02,hand03,hand04,hand05,hand06,hand07,hand08,hand09

[formatters]
keys=form01,form02,form03,form04,form05,form06,form07,form08,form09
@end example

The root logger must specify a level and a list of handlers. An example of a
root logger section is given below.

@example
[logger_root]
level=NOTSET
handlers=hand01
@end example

The @code{level} entry can be one of @code{DEBUG, INFO, WARNING, ERROR, CRITICAL} or
@code{NOTSET}. For the root logger only, @code{NOTSET} means that all messages will be
logged. Level values are @ref{34a,,eval()}uated in the context of the @code{logging}
package’s namespace.

The @code{handlers} entry is a comma-separated list of handler names, which must
appear in the @code{[handlers]} section. These names must appear in the
@code{[handlers]} section and have corresponding sections in the configuration
file.

For loggers other than the root logger, some additional information is required.
This is illustrated by the following example.

@example
[logger_parser]
level=DEBUG
handlers=hand01
propagate=1
qualname=compiler.parser
@end example

The @code{level} and @code{handlers} entries are interpreted as for the root logger,
except that if a non-root logger’s level is specified as @code{NOTSET}, the system
consults loggers higher up the hierarchy to determine the effective level of the
logger. The @code{propagate} entry is set to 1 to indicate that messages must
propagate to handlers higher up the logger hierarchy from this logger, or 0 to
indicate that messages are @strong{not} propagated to handlers up the hierarchy. The
@code{qualname} entry is the hierarchical channel name of the logger, that is to
say the name used by the application to get the logger.

Sections which specify handler configuration are exemplified by the following.

@example
[handler_hand01]
class=StreamHandler
level=NOTSET
formatter=form01
args=(sys.stdout,)
@end example

The @code{class} entry indicates the handler’s class (as determined by @ref{34a,,eval()}
in the @code{logging} package’s namespace). The @code{level} is interpreted as for
loggers, and @code{NOTSET} is taken to mean ‘log everything’.

The @code{formatter} entry indicates the key name of the formatter for this
handler. If blank, a default formatter (@code{logging._defaultFormatter}) is used.
If a name is specified, it must appear in the @code{[formatters]} section and have
a corresponding section in the configuration file.

The @code{args} entry, when @ref{34a,,eval()}uated in the context of the @code{logging}
package’s namespace, is the list of arguments to the constructor for the handler
class. Refer to the constructors for the relevant handlers, or to the examples
below, to see how typical entries are constructed. If not provided, it defaults
to @code{()}.

The optional @code{kwargs} entry, when @ref{34a,,eval()}uated in the context of the
@code{logging} package’s namespace, is the keyword argument dict to the constructor
for the handler class. If not provided, it defaults to @code{@{@}}.

@example
[handler_hand02]
class=FileHandler
level=DEBUG
formatter=form02
args=('python.log', 'w')

[handler_hand03]
class=handlers.SocketHandler
level=INFO
formatter=form03
args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT)

[handler_hand04]
class=handlers.DatagramHandler
level=WARN
formatter=form04
args=('localhost', handlers.DEFAULT_UDP_LOGGING_PORT)

[handler_hand05]
class=handlers.SysLogHandler
level=ERROR
formatter=form05
args=(('localhost', handlers.SYSLOG_UDP_PORT), handlers.SysLogHandler.LOG_USER)

[handler_hand06]
class=handlers.NTEventLogHandler
level=CRITICAL
formatter=form06
args=('Python Application', '@w{'}, 'Application')

[handler_hand07]
class=handlers.SMTPHandler
level=WARN
formatter=form07
args=('localhost', 'from@@abc', ['user1@@abc', 'user2@@xyz'], 'Logger Subject')
kwargs=@{'timeout': 10.0@}

[handler_hand08]
class=handlers.MemoryHandler
level=NOTSET
formatter=form08
target=
args=(10, ERROR)

[handler_hand09]
class=handlers.HTTPHandler
level=NOTSET
formatter=form09
args=('localhost:9022', '/log', 'GET')
kwargs=@{'secure': True@}
@end example

Sections which specify formatter configuration are typified by the following.

@example
[formatter_form01]
format=F1 %(asctime)s %(levelname)s %(message)s
datefmt=
style='%'
validate=True
class=logging.Formatter
@end example

The arguments for the formatter configuration are the same as the keys
in the dictionary schema @ref{2578,,formatters section}.

@cartouche
@quotation Note 
Due to the use of @ref{34a,,eval()} as described above, there are
potential security risks which result from using the @ref{ae3,,listen()} to send
and receive configurations via sockets. The risks are limited to where
multiple users with no mutual trust run code on the same machine; see the
@ref{ae3,,listen()} documentation for more information.
@end quotation
@end cartouche


@subsubheading See also



@table @asis

@item Module @ref{a6,,logging}

API reference for the logging module.

@item Module @ref{a8,,logging.handlers}

Useful handlers included with the logging module.
@end table


@node logging handlers — Logging handlers,getpass — Portable password input,logging config — Logging configuration,Generic Operating System Services
@anchor{library/logging handlers doc}@anchor{2586}@anchor{library/logging handlers logging-handlers-logging-handlers}@anchor{2587}@anchor{library/logging handlers module-logging handlers}@anchor{a8}
@subsection @code{logging.handlers} — Logging handlers


@geindex module; logging.handlers

@strong{Source code:} Lib/logging/handlers.py@footnote{https://github.com/python/cpython/tree/main/Lib/logging/handlers.py}

@subsubheading Important



This page contains only reference information. For tutorials,
please see


@itemize *

@item 
@ref{d97,,Basic Tutorial}

@item 
@ref{d98,,Advanced Tutorial}

@item 
@ref{d99,,Logging Cookbook}
@end itemize


__________________________________________________________________


The following useful handlers are provided in the package. Note that three of
the handlers (@ref{d9a,,StreamHandler}, @ref{1104,,FileHandler} and
@ref{e34,,NullHandler}) are actually defined in the @ref{a6,,logging} module itself,
but have been documented here along with the other handlers.

@menu
* StreamHandler:: 
* FileHandler:: 
* NullHandler:: 
* WatchedFileHandler:: 
* BaseRotatingHandler:: 
* RotatingFileHandler:: 
* TimedRotatingFileHandler:: 
* SocketHandler:: 
* DatagramHandler:: 
* SysLogHandler:: 
* NTEventLogHandler:: 
* SMTPHandler:: 
* MemoryHandler:: 
* HTTPHandler:: 
* QueueHandler:: 
* QueueListener:: 

@end menu

@node StreamHandler,FileHandler,,logging handlers — Logging handlers
@anchor{library/logging handlers stream-handler}@anchor{2588}@anchor{library/logging handlers streamhandler}@anchor{2589}
@subsubsection StreamHandler


The @ref{d9a,,StreamHandler} class, located in the core @ref{a6,,logging} package,
sends logging output to streams such as @emph{sys.stdout}, @emph{sys.stderr} or any
file-like object (or, more precisely, any object which supports @code{write()}
and @code{flush()} methods).

@geindex StreamHandler (class in logging)
@anchor{library/logging handlers logging StreamHandler}@anchor{d9a}
@deffn {Class} logging.StreamHandler (stream=None)

Returns a new instance of the @ref{d9a,,StreamHandler} class. If @emph{stream} is
specified, the instance will use it for logging output; otherwise, @emph{sys.stderr}
will be used.

@geindex emit() (logging.StreamHandler method)
@anchor{library/logging handlers logging StreamHandler emit}@anchor{258a}
@deffn {Method} emit (record)

If a formatter is specified, it is used to format the record. The record
is then written to the stream followed by @ref{258b,,terminator}. If exception information
is present, it is formatted using @ref{2d5,,traceback.print_exception()} and
appended to the stream.
@end deffn

@geindex flush() (logging.StreamHandler method)
@anchor{library/logging handlers logging StreamHandler flush}@anchor{258c}
@deffn {Method} flush ()

Flushes the stream by calling its @ref{258c,,flush()} method. Note that the
@code{close()} method is inherited from @ref{2521,,Handler} and so
does no output, so an explicit @ref{258c,,flush()} call may be needed at times.
@end deffn

@geindex setStream() (logging.StreamHandler method)
@anchor{library/logging handlers logging StreamHandler setStream}@anchor{677}
@deffn {Method} setStream (stream)

Sets the instance’s stream to the specified value, if it is different.
The old stream is flushed before the new stream is set.

@*Parameters: 
@code{stream} – The stream that the handler should use.


@*Returns: 
the old stream, if the stream was changed, or @emph{None} if it wasn’t.



New in version 3.7.

@end deffn

@geindex terminator (logging.StreamHandler attribute)
@anchor{library/logging handlers logging StreamHandler terminator}@anchor{258b}
@deffn {Attribute} terminator

String used as the terminator when writing a formatted record to a stream.
Default value is @code{'\n'}.

If you don’t want a newline termination, you can set the handler instance’s
@code{terminator} attribute to the empty string.

In earlier versions, the terminator was hardcoded as @code{'\n'}.


New in version 3.2.

@end deffn
@end deffn

@node FileHandler,NullHandler,StreamHandler,logging handlers — Logging handlers
@anchor{library/logging handlers file-handler}@anchor{258d}@anchor{library/logging handlers filehandler}@anchor{258e}
@subsubsection FileHandler


The @ref{1104,,FileHandler} class, located in the core @ref{a6,,logging} package,
sends logging output to a disk file.  It inherits the output functionality from
@ref{d9a,,StreamHandler}.

@geindex FileHandler (class in logging)
@anchor{library/logging handlers logging FileHandler}@anchor{1104}
@deffn {Class} logging.FileHandler (filename, mode='a', encoding=None, delay=False, errors=None)

Returns a new instance of the @ref{1104,,FileHandler} class. The specified file is
opened and used as the stream for logging. If @emph{mode} is not specified,
@code{'a'} is used.  If @emph{encoding} is not @code{None}, it is used to open the file
with that encoding.  If @emph{delay} is true, then file opening is deferred until the
first call to @ref{258f,,emit()}. By default, the file grows indefinitely. If
@emph{errors} is specified, it’s used to determine how encoding errors are handled.


Changed in version 3.6: As well as string values, @ref{522,,Path} objects are also accepted
for the @emph{filename} argument.



Changed in version 3.9: The @emph{errors} parameter was added.


@geindex close() (logging.FileHandler method)
@anchor{library/logging handlers logging FileHandler close}@anchor{2590}
@deffn {Method} close ()

Closes the file.
@end deffn

@geindex emit() (logging.FileHandler method)
@anchor{library/logging handlers logging FileHandler emit}@anchor{258f}
@deffn {Method} emit (record)

Outputs the record to the file.

Note that if the file was closed due to logging shutdown at exit and the file
mode is ‘w’, the record will not be emitted (see bpo-42378@footnote{https://bugs.python.org/issue42378}).
@end deffn
@end deffn

@node NullHandler,WatchedFileHandler,FileHandler,logging handlers — Logging handlers
@anchor{library/logging handlers null-handler}@anchor{2591}@anchor{library/logging handlers nullhandler}@anchor{2592}
@subsubsection NullHandler



New in version 3.1.


The @ref{e34,,NullHandler} class, located in the core @ref{a6,,logging} package,
does not do any formatting or output. It is essentially a ‘no-op’ handler
for use by library developers.

@geindex NullHandler (class in logging)
@anchor{library/logging handlers logging NullHandler}@anchor{e34}
@deffn {Class} logging.NullHandler

Returns a new instance of the @ref{e34,,NullHandler} class.

@geindex emit() (logging.NullHandler method)
@anchor{library/logging handlers logging NullHandler emit}@anchor{2593}
@deffn {Method} emit (record)

This method does nothing.
@end deffn

@geindex handle() (logging.NullHandler method)
@anchor{library/logging handlers logging NullHandler handle}@anchor{2594}
@deffn {Method} handle (record)

This method does nothing.
@end deffn

@geindex createLock() (logging.NullHandler method)
@anchor{library/logging handlers logging NullHandler createLock}@anchor{2595}
@deffn {Method} createLock ()

This method returns @code{None} for the lock, since there is no
underlying I/O to which access needs to be serialized.
@end deffn
@end deffn

See @ref{2596,,Configuring Logging for a Library} for more information on how to use
@ref{e34,,NullHandler}.

@node WatchedFileHandler,BaseRotatingHandler,NullHandler,logging handlers — Logging handlers
@anchor{library/logging handlers watched-file-handler}@anchor{2597}@anchor{library/logging handlers watchedfilehandler}@anchor{2598}
@subsubsection WatchedFileHandler


The @ref{2599,,WatchedFileHandler} class, located in the @ref{a8,,logging.handlers}
module, is a @code{FileHandler} which watches the file it is logging to. If
the file changes, it is closed and reopened using the file name.

A file change can happen because of usage of programs such as @emph{newsyslog} and
@emph{logrotate} which perform log file rotation. This handler, intended for use
under Unix/Linux, watches the file to see if it has changed since the last emit.
(A file is deemed to have changed if its device or inode have changed.) If the
file has changed, the old file stream is closed, and the file opened to get a
new stream.

This handler is not appropriate for use under Windows, because under Windows
open log files cannot be moved or renamed - logging opens the files with
exclusive locks - and so there is no need for such a handler. Furthermore,
@emph{ST_INO} is not supported under Windows; @ref{516,,stat()} always returns zero
for this value.

@geindex WatchedFileHandler (class in logging.handlers)
@anchor{library/logging handlers logging handlers WatchedFileHandler}@anchor{2599}
@deffn {Class} logging.handlers.WatchedFileHandler (filename, mode='a', encoding=None, delay=False, errors=None)

Returns a new instance of the @ref{2599,,WatchedFileHandler} class. The specified
file is opened and used as the stream for logging. If @emph{mode} is not specified,
@code{'a'} is used.  If @emph{encoding} is not @code{None}, it is used to open the file
with that encoding.  If @emph{delay} is true, then file opening is deferred until the
first call to @ref{259a,,emit()}.  By default, the file grows indefinitely. If
@emph{errors} is provided, it determines how encoding errors are handled.


Changed in version 3.6: As well as string values, @ref{522,,Path} objects are also accepted
for the @emph{filename} argument.



Changed in version 3.9: The @emph{errors} parameter was added.


@geindex reopenIfNeeded() (logging.handlers.WatchedFileHandler method)
@anchor{library/logging handlers logging handlers WatchedFileHandler reopenIfNeeded}@anchor{800}
@deffn {Method} reopenIfNeeded ()

Checks to see if the file has changed.  If it has, the existing stream is
flushed and closed and the file opened again, typically as a precursor to
outputting the record to the file.


New in version 3.6.

@end deffn

@geindex emit() (logging.handlers.WatchedFileHandler method)
@anchor{library/logging handlers logging handlers WatchedFileHandler emit}@anchor{259a}
@deffn {Method} emit (record)

Outputs the record to the file, but first calls @ref{800,,reopenIfNeeded()} to
reopen the file if it has changed.
@end deffn
@end deffn

@node BaseRotatingHandler,RotatingFileHandler,WatchedFileHandler,logging handlers — Logging handlers
@anchor{library/logging handlers base-rotating-handler}@anchor{259b}@anchor{library/logging handlers baserotatinghandler}@anchor{259c}
@subsubsection BaseRotatingHandler


The @ref{259d,,BaseRotatingHandler} class, located in the @ref{a8,,logging.handlers}
module, is the base class for the rotating file handlers,
@ref{2579,,RotatingFileHandler} and @ref{adf,,TimedRotatingFileHandler}. You should
not need to instantiate this class, but it has attributes and methods you may
need to override.

@geindex BaseRotatingHandler (class in logging.handlers)
@anchor{library/logging handlers logging handlers BaseRotatingHandler}@anchor{259d}
@deffn {Class} logging.handlers.BaseRotatingHandler (filename, mode, encoding=None, delay=False, errors=None)

The parameters are as for @code{FileHandler}. The attributes are:

@geindex namer (logging.handlers.BaseRotatingHandler attribute)
@anchor{library/logging handlers logging handlers BaseRotatingHandler namer}@anchor{259e}
@deffn {Attribute} namer

If this attribute is set to a callable, the @ref{259f,,rotation_filename()}
method delegates to this callable. The parameters passed to the callable
are those passed to @ref{259f,,rotation_filename()}.

@cartouche
@quotation Note 
The namer function is called quite a few times during rollover,
so it should be as simple and as fast as possible. It should also
return the same output every time for a given input, otherwise the
rollover behaviour may not work as expected.

It’s also worth noting that care should be taken when using a namer to
preserve certain attributes in the filename which are used during rotation.
For example, @ref{2579,,RotatingFileHandler} expects to have a set of log files
whose names contain successive integers, so that rotation works as expected,
and @ref{adf,,TimedRotatingFileHandler} deletes old log files (based on the
@code{backupCount} parameter passed to the handler’s initializer) by determining
the oldest files to delete. For this to happen, the filenames should be
sortable using the date/time portion of the filename, and a namer needs to
respect this. (If a namer is wanted that doesn’t respect this scheme, it will
need to be used in a subclass of @ref{adf,,TimedRotatingFileHandler} which
overrides the @ref{25a0,,getFilesToDelete()} method to
fit in with the custom naming scheme.)
@end quotation
@end cartouche


New in version 3.3.

@end deffn

@geindex rotator (logging.handlers.BaseRotatingHandler attribute)
@anchor{library/logging handlers logging handlers BaseRotatingHandler rotator}@anchor{25a1}
@deffn {Attribute} rotator

If this attribute is set to a callable, the @ref{25a2,,rotate()} method
delegates to this callable.  The parameters passed to the callable are
those passed to @ref{25a2,,rotate()}.


New in version 3.3.

@end deffn

@geindex rotation_filename() (logging.handlers.BaseRotatingHandler method)
@anchor{library/logging handlers logging handlers BaseRotatingHandler rotation_filename}@anchor{259f}
@deffn {Method} rotation_filename (default_name)

Modify the filename of a log file when rotating.

This is provided so that a custom filename can be provided.

The default implementation calls the ‘namer’ attribute of the handler,
if it’s callable, passing the default name to it. If the attribute isn’t
callable (the default is @code{None}), the name is returned unchanged.

@*Parameters: 
@code{default_name} – The default name for the log file.



New in version 3.3.

@end deffn

@geindex rotate() (logging.handlers.BaseRotatingHandler method)
@anchor{library/logging handlers logging handlers BaseRotatingHandler rotate}@anchor{25a2}
@deffn {Method} rotate (source, dest)

When rotating, rotate the current log.

The default implementation calls the ‘rotator’ attribute of the handler,
if it’s callable, passing the source and dest arguments to it. If the
attribute isn’t callable (the default is @code{None}), the source is simply
renamed to the destination.

@*Parameters: 

@itemize *

@item 
@code{source} – The source filename. This is normally the base
filename, e.g. ‘test.log’.

@item 
@code{dest} – The destination filename. This is normally
what the source is rotated to, e.g. ‘test.log.1’.
@end itemize



New in version 3.3.

@end deffn
@end deffn

The reason the attributes exist is to save you having to subclass - you can use
the same callables for instances of @ref{2579,,RotatingFileHandler} and
@ref{adf,,TimedRotatingFileHandler}. If either the namer or rotator callable
raises an exception, this will be handled in the same way as any other
exception during an @code{emit()} call, i.e. via the @code{handleError()} method
of the handler.

If you need to make more significant changes to rotation processing, you can
override the methods.

For an example, see @ref{25a3,,Using a rotator and namer to customize log rotation processing}.

@node RotatingFileHandler,TimedRotatingFileHandler,BaseRotatingHandler,logging handlers — Logging handlers
@anchor{library/logging handlers rotating-file-handler}@anchor{25a4}@anchor{library/logging handlers rotatingfilehandler}@anchor{25a5}
@subsubsection RotatingFileHandler


The @ref{2579,,RotatingFileHandler} class, located in the @ref{a8,,logging.handlers}
module, supports rotation of disk log files.

@geindex RotatingFileHandler (class in logging.handlers)
@anchor{library/logging handlers logging handlers RotatingFileHandler}@anchor{2579}
@deffn {Class} logging.handlers.RotatingFileHandler (filename, mode='a', maxBytes=0, backupCount=0, encoding=None, delay=False, errors=None)

Returns a new instance of the @ref{2579,,RotatingFileHandler} class. The specified
file is opened and used as the stream for logging. If @emph{mode} is not specified,
@code{'a'} is used.  If @emph{encoding} is not @code{None}, it is used to open the file
with that encoding.  If @emph{delay} is true, then file opening is deferred until the
first call to @ref{25a6,,emit()}.  By default, the file grows indefinitely. If
@emph{errors} is provided, it determines how encoding errors are handled.

You can use the @emph{maxBytes} and @emph{backupCount} values to allow the file to
@emph{rollover} at a predetermined size. When the size is about to be exceeded,
the file is closed and a new file is silently opened for output. Rollover occurs
whenever the current log file is nearly @emph{maxBytes} in length; but if either of
@emph{maxBytes} or @emph{backupCount} is zero, rollover never occurs, so you generally want
to set @emph{backupCount} to at least 1, and have a non-zero @emph{maxBytes}.
When @emph{backupCount} is non-zero, the system will save old log files by appending
the extensions ‘.1’, ‘.2’ etc., to the filename. For example, with a @emph{backupCount}
of 5 and a base file name of @code{app.log}, you would get @code{app.log},
@code{app.log.1}, @code{app.log.2}, up to @code{app.log.5}. The file being
written to is always @code{app.log}.  When this file is filled, it is closed
and renamed to @code{app.log.1}, and if files @code{app.log.1},
@code{app.log.2}, etc. exist, then they are renamed to @code{app.log.2},
@code{app.log.3} etc. respectively.


Changed in version 3.6: As well as string values, @ref{522,,Path} objects are also accepted
for the @emph{filename} argument.



Changed in version 3.9: The @emph{errors} parameter was added.


@geindex doRollover() (logging.handlers.RotatingFileHandler method)
@anchor{library/logging handlers logging handlers RotatingFileHandler doRollover}@anchor{25a7}
@deffn {Method} doRollover ()

Does a rollover, as described above.
@end deffn

@geindex emit() (logging.handlers.RotatingFileHandler method)
@anchor{library/logging handlers logging handlers RotatingFileHandler emit}@anchor{25a6}
@deffn {Method} emit (record)

Outputs the record to the file, catering for rollover as described
previously.
@end deffn
@end deffn

@node TimedRotatingFileHandler,SocketHandler,RotatingFileHandler,logging handlers — Logging handlers
@anchor{library/logging handlers timed-rotating-file-handler}@anchor{25a8}@anchor{library/logging handlers timedrotatingfilehandler}@anchor{25a9}
@subsubsection TimedRotatingFileHandler


The @ref{adf,,TimedRotatingFileHandler} class, located in the
@ref{a8,,logging.handlers} module, supports rotation of disk log files at certain
timed intervals.

@geindex TimedRotatingFileHandler (class in logging.handlers)
@anchor{library/logging handlers logging handlers TimedRotatingFileHandler}@anchor{adf}
@deffn {Class} logging.handlers.TimedRotatingFileHandler (filename, when='h', interval=1, backupCount=0, encoding=None, delay=False, utc=False, atTime=None, errors=None)

Returns a new instance of the @ref{adf,,TimedRotatingFileHandler} class. The
specified file is opened and used as the stream for logging. On rotating it also
sets the filename suffix. Rotating happens based on the product of @emph{when} and
@emph{interval}.

You can use the @emph{when} to specify the type of @emph{interval}. The list of possible
values is below.  Note that they are not case sensitive.


@multitable {xxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Value

@tab

Type of interval

@tab

If/how @emph{atTime} is used

@item

@code{'S'}

@tab

Seconds

@tab

Ignored

@item

@code{'M'}

@tab

Minutes

@tab

Ignored

@item

@code{'H'}

@tab

Hours

@tab

Ignored

@item

@code{'D'}

@tab

Days

@tab

Ignored

@item

@code{'W0'-'W6'}

@tab

Weekday (0=Monday)

@tab

Used to compute initial
rollover time

@item

@code{'midnight'}

@tab

Roll over at midnight, if
@emph{atTime} not specified,
else at time @emph{atTime}

@tab

Used to compute initial
rollover time

@end multitable


When using weekday-based rotation, specify ‘W0’ for Monday, ‘W1’ for
Tuesday, and so on up to ‘W6’ for Sunday. In this case, the value passed for
@emph{interval} isn’t used.

The system will save old log files by appending extensions to the filename.
The extensions are date-and-time based, using the strftime format
@code{%Y-%m-%d_%H-%M-%S} or a leading portion thereof, depending on the
rollover interval.

When computing the next rollover time for the first time (when the handler
is created), the last modification time of an existing log file, or else
the current time, is used to compute when the next rotation will occur.

If the @emph{utc} argument is true, times in UTC will be used; otherwise
local time is used.

If @emph{backupCount} is nonzero, at most @emph{backupCount} files
will be kept, and if more would be created when rollover occurs, the oldest
one is deleted. The deletion logic uses the interval to determine which
files to delete, so changing the interval may leave old files lying around.

If @emph{delay} is true, then file opening is deferred until the first call to
@ref{25aa,,emit()}.

If @emph{atTime} is not @code{None}, it must be a @code{datetime.time} instance which
specifies the time of day when rollover occurs, for the cases where rollover
is set to happen “at midnight” or “on a particular weekday”. Note that in
these cases, the @emph{atTime} value is effectively used to compute the @emph{initial}
rollover, and subsequent rollovers would be calculated via the normal
interval calculation.

If @emph{errors} is specified, it’s used to determine how encoding errors are
handled.

@cartouche
@quotation Note 
Calculation of the initial rollover time is done when the handler
is initialised. Calculation of subsequent rollover times is done only
when rollover occurs, and rollover occurs only when emitting output. If
this is not kept in mind, it might lead to some confusion. For example,
if an interval of “every minute” is set, that does not mean you will
always see log files with times (in the filename) separated by a minute;
if, during application execution, logging output is generated more
frequently than once a minute, @emph{then} you can expect to see log files
with times separated by a minute. If, on the other hand, logging messages
are only output once every five minutes (say), then there will be gaps in
the file times corresponding to the minutes where no output (and hence no
rollover) occurred.
@end quotation
@end cartouche


Changed in version 3.4: @emph{atTime} parameter was added.



Changed in version 3.6: As well as string values, @ref{522,,Path} objects are also accepted
for the @emph{filename} argument.



Changed in version 3.9: The @emph{errors} parameter was added.


@geindex doRollover() (logging.handlers.TimedRotatingFileHandler method)
@anchor{library/logging handlers logging handlers TimedRotatingFileHandler doRollover}@anchor{25ab}
@deffn {Method} doRollover ()

Does a rollover, as described above.
@end deffn

@geindex emit() (logging.handlers.TimedRotatingFileHandler method)
@anchor{library/logging handlers logging handlers TimedRotatingFileHandler emit}@anchor{25aa}
@deffn {Method} emit (record)

Outputs the record to the file, catering for rollover as described above.
@end deffn

@geindex getFilesToDelete() (logging.handlers.TimedRotatingFileHandler method)
@anchor{library/logging handlers logging handlers TimedRotatingFileHandler getFilesToDelete}@anchor{25a0}
@deffn {Method} getFilesToDelete ()

Returns a list of filenames which should be deleted as part of rollover. These
are the absolute paths of the oldest backup log files written by the handler.
@end deffn
@end deffn

@node SocketHandler,DatagramHandler,TimedRotatingFileHandler,logging handlers — Logging handlers
@anchor{library/logging handlers socket-handler}@anchor{25ac}@anchor{library/logging handlers sockethandler}@anchor{25ad}
@subsubsection SocketHandler


The @ref{ae0,,SocketHandler} class, located in the @ref{a8,,logging.handlers} module,
sends logging output to a network socket. The base class uses a TCP socket.

@geindex SocketHandler (class in logging.handlers)
@anchor{library/logging handlers logging handlers SocketHandler}@anchor{ae0}
@deffn {Class} logging.handlers.SocketHandler (host, port)

Returns a new instance of the @ref{ae0,,SocketHandler} class intended to
communicate with a remote machine whose address is given by @emph{host} and @emph{port}.


Changed in version 3.4: If @code{port} is specified as @code{None}, a Unix domain socket is created
using the value in @code{host} - otherwise, a TCP socket is created.


@geindex close() (logging.handlers.SocketHandler method)
@anchor{library/logging handlers logging handlers SocketHandler close}@anchor{25ae}
@deffn {Method} close ()

Closes the socket.
@end deffn

@geindex emit() (logging.handlers.SocketHandler method)
@anchor{library/logging handlers logging handlers SocketHandler emit}@anchor{25af}
@deffn {Method} emit ()

Pickles the record’s attribute dictionary and writes it to the socket in
binary format. If there is an error with the socket, silently drops the
packet. If the connection was previously lost, re-establishes the
connection. To unpickle the record at the receiving end into a
@ref{b6c,,LogRecord}, use the @ref{2551,,makeLogRecord()}
function.
@end deffn

@geindex handleError() (logging.handlers.SocketHandler method)
@anchor{library/logging handlers logging handlers SocketHandler handleError}@anchor{25b0}
@deffn {Method} handleError ()

Handles an error which has occurred during @ref{25af,,emit()}. The most likely
cause is a lost connection. Closes the socket so that we can retry on the
next event.
@end deffn

@geindex makeSocket() (logging.handlers.SocketHandler method)
@anchor{library/logging handlers logging handlers SocketHandler makeSocket}@anchor{25b1}
@deffn {Method} makeSocket ()

This is a factory method which allows subclasses to define the precise
type of socket they want. The default implementation creates a TCP socket
(@ref{e91,,socket.SOCK_STREAM}).
@end deffn

@geindex makePickle() (logging.handlers.SocketHandler method)
@anchor{library/logging handlers logging handlers SocketHandler makePickle}@anchor{25b2}
@deffn {Method} makePickle (record)

Pickles the record’s attribute dictionary in binary format with a length
prefix, and returns it ready for transmission across the socket. The
details of this operation are equivalent to:

@example
data = pickle.dumps(record_attr_dict, 1)
datalen = struct.pack('>L', len(data))
return datalen + data
@end example

Note that pickles aren’t completely secure. If you are concerned about
security, you may want to override this method to implement a more secure
mechanism. For example, you can sign pickles using HMAC and then verify
them on the receiving end, or alternatively you can disable unpickling of
global objects on the receiving end.
@end deffn

@geindex send() (logging.handlers.SocketHandler method)
@anchor{library/logging handlers logging handlers SocketHandler send}@anchor{25b3}
@deffn {Method} send (packet)

Send a pickled byte-string @emph{packet} to the socket. The format of the sent
byte-string is as described in the documentation for
@ref{25b2,,makePickle()}.

This function allows for partial sends, which can happen when the network
is busy.
@end deffn

@geindex createSocket() (logging.handlers.SocketHandler method)
@anchor{library/logging handlers logging handlers SocketHandler createSocket}@anchor{25b4}
@deffn {Method} createSocket ()

Tries to create a socket; on failure, uses an exponential back-off
algorithm.  On initial failure, the handler will drop the message it was
trying to send.  When subsequent messages are handled by the same
instance, it will not try connecting until some time has passed.  The
default parameters are such that the initial delay is one second, and if
after that delay the connection still can’t be made, the handler will
double the delay each time up to a maximum of 30 seconds.

This behaviour is controlled by the following handler attributes:


@itemize *

@item 
@code{retryStart} (initial delay, defaulting to 1.0 seconds).

@item 
@code{retryFactor} (multiplier, defaulting to 2.0).

@item 
@code{retryMax} (maximum delay, defaulting to 30.0 seconds).
@end itemize

This means that if the remote listener starts up @emph{after} the handler has
been used, you could lose messages (since the handler won’t even attempt
a connection until the delay has elapsed, but just silently drop messages
during the delay period).
@end deffn
@end deffn

@node DatagramHandler,SysLogHandler,SocketHandler,logging handlers — Logging handlers
@anchor{library/logging handlers datagram-handler}@anchor{25b5}@anchor{library/logging handlers datagramhandler}@anchor{25b6}
@subsubsection DatagramHandler


The @ref{ae1,,DatagramHandler} class, located in the @ref{a8,,logging.handlers}
module, inherits from @ref{ae0,,SocketHandler} to support sending logging messages
over UDP sockets.

@geindex DatagramHandler (class in logging.handlers)
@anchor{library/logging handlers logging handlers DatagramHandler}@anchor{ae1}
@deffn {Class} logging.handlers.DatagramHandler (host, port)

Returns a new instance of the @ref{ae1,,DatagramHandler} class intended to
communicate with a remote machine whose address is given by @emph{host} and @emph{port}.


Changed in version 3.4: If @code{port} is specified as @code{None}, a Unix domain socket is created
using the value in @code{host} - otherwise, a UDP socket is created.


@geindex emit() (logging.handlers.DatagramHandler method)
@anchor{library/logging handlers logging handlers DatagramHandler emit}@anchor{25b7}
@deffn {Method} emit ()

Pickles the record’s attribute dictionary and writes it to the socket in
binary format. If there is an error with the socket, silently drops the
packet. To unpickle the record at the receiving end into a
@ref{b6c,,LogRecord}, use the @ref{2551,,makeLogRecord()}
function.
@end deffn

@geindex makeSocket() (logging.handlers.DatagramHandler method)
@anchor{library/logging handlers logging handlers DatagramHandler makeSocket}@anchor{25b8}
@deffn {Method} makeSocket ()

The factory method of @ref{ae0,,SocketHandler} is here overridden to create
a UDP socket (@ref{e90,,socket.SOCK_DGRAM}).
@end deffn

@geindex send() (logging.handlers.DatagramHandler method)
@anchor{library/logging handlers logging handlers DatagramHandler send}@anchor{25b9}
@deffn {Method} send (s)

Send a pickled byte-string to a socket. The format of the sent byte-string
is as described in the documentation for @ref{25b2,,SocketHandler.makePickle()}.
@end deffn
@end deffn

@node SysLogHandler,NTEventLogHandler,DatagramHandler,logging handlers — Logging handlers
@anchor{library/logging handlers syslog-handler}@anchor{25ba}@anchor{library/logging handlers sysloghandler}@anchor{25bb}
@subsubsection SysLogHandler


The @ref{c6c,,SysLogHandler} class, located in the @ref{a8,,logging.handlers} module,
supports sending logging messages to a remote or local Unix syslog.

@geindex SysLogHandler (class in logging.handlers)
@anchor{library/logging handlers logging handlers SysLogHandler}@anchor{c6c}
@deffn {Class} logging.handlers.SysLogHandler (address=('localhost', SYSLOG_UDP_PORT), facility=LOG_USER, socktype=socket.SOCK_DGRAM)

Returns a new instance of the @ref{c6c,,SysLogHandler} class intended to
communicate with a remote Unix machine whose address is given by @emph{address} in
the form of a @code{(host, port)} tuple.  If @emph{address} is not specified,
@code{('localhost', 514)} is used.  The address is used to open a socket.  An
alternative to providing a @code{(host, port)} tuple is providing an address as a
string, for example ‘/dev/log’. In this case, a Unix domain socket is used to
send the message to the syslog. If @emph{facility} is not specified,
@code{LOG_USER} is used. The type of socket opened depends on the
@emph{socktype} argument, which defaults to @ref{e90,,socket.SOCK_DGRAM} and thus
opens a UDP socket. To open a TCP socket (for use with the newer syslog
daemons such as rsyslog), specify a value of @ref{e91,,socket.SOCK_STREAM}.

Note that if your server is not listening on UDP port 514,
@ref{c6c,,SysLogHandler} may appear not to work. In that case, check what
address you should be using for a domain socket - it’s system dependent.
For example, on Linux it’s usually ‘/dev/log’ but on OS/X it’s
‘/var/run/syslog’. You’ll need to check your platform and use the
appropriate address (you may need to do this check at runtime if your
application needs to run on several platforms). On Windows, you pretty
much have to use the UDP option.


Changed in version 3.2: @emph{socktype} was added.


@geindex close() (logging.handlers.SysLogHandler method)
@anchor{library/logging handlers logging handlers SysLogHandler close}@anchor{25bc}
@deffn {Method} close ()

Closes the socket to the remote host.
@end deffn

@geindex emit() (logging.handlers.SysLogHandler method)
@anchor{library/logging handlers logging handlers SysLogHandler emit}@anchor{25bd}
@deffn {Method} emit (record)

The record is formatted, and then sent to the syslog server. If exception
information is present, it is @emph{not} sent to the server.


Changed in version 3.2.1: (See: bpo-12168@footnote{https://bugs.python.org/issue12168}.) In earlier versions, the message sent to the
syslog daemons was always terminated with a NUL byte, because early
versions of these daemons expected a NUL terminated message - even
though it’s not in the relevant specification (
@geindex RFC; RFC 5424
RFC 5424@footnote{https://tools.ietf.org/html/rfc5424.html}). More recent
versions of these daemons don’t expect the NUL byte but strip it off
if it’s there, and even more recent daemons (which adhere more closely
to RFC 5424) pass the NUL byte on as part of the message.

To enable easier handling of syslog messages in the face of all these
differing daemon behaviours, the appending of the NUL byte has been
made configurable, through the use of a class-level attribute,
@code{append_nul}. This defaults to @code{True} (preserving the existing
behaviour) but can be set to @code{False} on a @code{SysLogHandler} instance
in order for that instance to @emph{not} append the NUL terminator.



Changed in version 3.3: (See: bpo-12419@footnote{https://bugs.python.org/issue12419}.) In earlier versions, there was no facility for
an “ident” or “tag” prefix to identify the source of the message. This
can now be specified using a class-level attribute, defaulting to
@code{""} to preserve existing behaviour, but which can be overridden on
a @code{SysLogHandler} instance in order for that instance to prepend
the ident to every message handled. Note that the provided ident must
be text, not bytes, and is prepended to the message exactly as is.

@end deffn

@geindex encodePriority() (logging.handlers.SysLogHandler method)
@anchor{library/logging handlers logging handlers SysLogHandler encodePriority}@anchor{25be}
@deffn {Method} encodePriority (facility, priority)

Encodes the facility and priority into an integer. You can pass in strings
or integers - if strings are passed, internal mapping dictionaries are
used to convert them to integers.

The symbolic @code{LOG_} values are defined in @ref{c6c,,SysLogHandler} and
mirror the values defined in the @code{sys/syslog.h} header file.

@strong{Priorities}


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxx} 
@headitem

Name (string)

@tab

Symbolic value

@item

@code{alert}

@tab

LOG_ALERT

@item

@code{crit} or @code{critical}

@tab

LOG_CRIT

@item

@code{debug}

@tab

LOG_DEBUG

@item

@code{emerg} or @code{panic}

@tab

LOG_EMERG

@item

@code{err} or @code{error}

@tab

LOG_ERR

@item

@code{info}

@tab

LOG_INFO

@item

@code{notice}

@tab

LOG_NOTICE

@item

@code{warn} or @code{warning}

@tab

LOG_WARNING

@end multitable


@strong{Facilities}


@multitable {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxx} 
@headitem

Name (string)

@tab

Symbolic value

@item

@code{auth}

@tab

LOG_AUTH

@item

@code{authpriv}

@tab

LOG_AUTHPRIV

@item

@code{cron}

@tab

LOG_CRON

@item

@code{daemon}

@tab

LOG_DAEMON

@item

@code{ftp}

@tab

LOG_FTP

@item

@code{kern}

@tab

LOG_KERN

@item

@code{lpr}

@tab

LOG_LPR

@item

@code{mail}

@tab

LOG_MAIL

@item

@code{news}

@tab

LOG_NEWS

@item

@code{syslog}

@tab

LOG_SYSLOG

@item

@code{user}

@tab

LOG_USER

@item

@code{uucp}

@tab

LOG_UUCP

@item

@code{local0}

@tab

LOG_LOCAL0

@item

@code{local1}

@tab

LOG_LOCAL1

@item

@code{local2}

@tab

LOG_LOCAL2

@item

@code{local3}

@tab

LOG_LOCAL3

@item

@code{local4}

@tab

LOG_LOCAL4

@item

@code{local5}

@tab

LOG_LOCAL5

@item

@code{local6}

@tab

LOG_LOCAL6

@item

@code{local7}

@tab

LOG_LOCAL7

@end multitable

@end deffn

@geindex mapPriority() (logging.handlers.SysLogHandler method)
@anchor{library/logging handlers logging handlers SysLogHandler mapPriority}@anchor{25bf}
@deffn {Method} mapPriority (levelname)

Maps a logging level name to a syslog priority name.
You may need to override this if you are using custom levels, or
if the default algorithm is not suitable for your needs. The
default algorithm maps @code{DEBUG}, @code{INFO}, @code{WARNING}, @code{ERROR} and
@code{CRITICAL} to the equivalent syslog names, and all other level
names to ‘warning’.
@end deffn
@end deffn

@node NTEventLogHandler,SMTPHandler,SysLogHandler,logging handlers — Logging handlers
@anchor{library/logging handlers nt-eventlog-handler}@anchor{25c0}@anchor{library/logging handlers nteventloghandler}@anchor{25c1}
@subsubsection NTEventLogHandler


The @ref{25c2,,NTEventLogHandler} class, located in the @ref{a8,,logging.handlers}
module, supports sending logging messages to a local Windows NT, Windows 2000 or
Windows XP event log. Before you can use it, you need Mark Hammond’s Win32
extensions for Python installed.

@geindex NTEventLogHandler (class in logging.handlers)
@anchor{library/logging handlers logging handlers NTEventLogHandler}@anchor{25c2}
@deffn {Class} logging.handlers.NTEventLogHandler (appname, dllname=None, logtype='Application')

Returns a new instance of the @ref{25c2,,NTEventLogHandler} class. The @emph{appname} is
used to define the application name as it appears in the event log. An
appropriate registry entry is created using this name. The @emph{dllname} should give
the fully qualified pathname of a .dll or .exe which contains message
definitions to hold in the log (if not specified, @code{'win32service.pyd'} is used
- this is installed with the Win32 extensions and contains some basic
placeholder message definitions. Note that use of these placeholders will make
your event logs big, as the entire message source is held in the log. If you
want slimmer logs, you have to pass in the name of your own .dll or .exe which
contains the message definitions you want to use in the event log). The
@emph{logtype} is one of @code{'Application'}, @code{'System'} or @code{'Security'}, and
defaults to @code{'Application'}.

@geindex close() (logging.handlers.NTEventLogHandler method)
@anchor{library/logging handlers logging handlers NTEventLogHandler close}@anchor{25c3}
@deffn {Method} close ()

At this point, you can remove the application name from the registry as a
source of event log entries. However, if you do this, you will not be able
to see the events as you intended in the Event Log Viewer - it needs to be
able to access the registry to get the .dll name. The current version does
not do this.
@end deffn

@geindex emit() (logging.handlers.NTEventLogHandler method)
@anchor{library/logging handlers logging handlers NTEventLogHandler emit}@anchor{25c4}
@deffn {Method} emit (record)

Determines the message ID, event category and event type, and then logs
the message in the NT event log.
@end deffn

@geindex getEventCategory() (logging.handlers.NTEventLogHandler method)
@anchor{library/logging handlers logging handlers NTEventLogHandler getEventCategory}@anchor{25c5}
@deffn {Method} getEventCategory (record)

Returns the event category for the record. Override this if you want to
specify your own categories. This version returns 0.
@end deffn

@geindex getEventType() (logging.handlers.NTEventLogHandler method)
@anchor{library/logging handlers logging handlers NTEventLogHandler getEventType}@anchor{25c6}
@deffn {Method} getEventType (record)

Returns the event type for the record. Override this if you want to
specify your own types. This version does a mapping using the handler’s
typemap attribute, which is set up in @code{__init__()} to a dictionary
which contains mappings for @code{DEBUG}, @code{INFO},
@code{WARNING}, @code{ERROR} and @code{CRITICAL}. If you are using
your own levels, you will either need to override this method or place a
suitable dictionary in the handler’s @emph{typemap} attribute.
@end deffn

@geindex getMessageID() (logging.handlers.NTEventLogHandler method)
@anchor{library/logging handlers logging handlers NTEventLogHandler getMessageID}@anchor{25c7}
@deffn {Method} getMessageID (record)

Returns the message ID for the record. If you are using your own messages,
you could do this by having the @emph{msg} passed to the logger being an ID
rather than a format string. Then, in here, you could use a dictionary
lookup to get the message ID. This version returns 1, which is the base
message ID in @code{win32service.pyd}.
@end deffn
@end deffn

@node SMTPHandler,MemoryHandler,NTEventLogHandler,logging handlers — Logging handlers
@anchor{library/logging handlers smtp-handler}@anchor{25c8}@anchor{library/logging handlers smtphandler}@anchor{25c9}
@subsubsection SMTPHandler


The @ref{25ca,,SMTPHandler} class, located in the @ref{a8,,logging.handlers} module,
supports sending logging messages to an email address via SMTP.

@geindex SMTPHandler (class in logging.handlers)
@anchor{library/logging handlers logging handlers SMTPHandler}@anchor{25ca}
@deffn {Class} logging.handlers.SMTPHandler (mailhost, fromaddr, toaddrs, subject, credentials=None, secure=None, timeout=1.0)

Returns a new instance of the @ref{25ca,,SMTPHandler} class. The instance is
initialized with the from and to addresses and subject line of the email. The
@emph{toaddrs} should be a list of strings. To specify a non-standard SMTP port, use
the (host, port) tuple format for the @emph{mailhost} argument. If you use a string,
the standard SMTP port is used. If your SMTP server requires authentication, you
can specify a (username, password) tuple for the @emph{credentials} argument.

To specify the use of a secure protocol (TLS), pass in a tuple to the
@emph{secure} argument. This will only be used when authentication credentials are
supplied. The tuple should be either an empty tuple, or a single-value tuple
with the name of a keyfile, or a 2-value tuple with the names of the keyfile
and certificate file. (This tuple is passed to the
@ref{ccd,,smtplib.SMTP.starttls()} method.)

A timeout can be specified for communication with the SMTP server using the
@emph{timeout} argument.


New in version 3.3: The @emph{timeout} argument was added.


@geindex emit() (logging.handlers.SMTPHandler method)
@anchor{library/logging handlers logging handlers SMTPHandler emit}@anchor{25cb}
@deffn {Method} emit (record)

Formats the record and sends it to the specified addressees.
@end deffn

@geindex getSubject() (logging.handlers.SMTPHandler method)
@anchor{library/logging handlers logging handlers SMTPHandler getSubject}@anchor{25cc}
@deffn {Method} getSubject (record)

If you want to specify a subject line which is record-dependent, override
this method.
@end deffn
@end deffn

@node MemoryHandler,HTTPHandler,SMTPHandler,logging handlers — Logging handlers
@anchor{library/logging handlers memory-handler}@anchor{25cd}@anchor{library/logging handlers memoryhandler}@anchor{25ce}
@subsubsection MemoryHandler


The @ref{2582,,MemoryHandler} class, located in the @ref{a8,,logging.handlers} module,
supports buffering of logging records in memory, periodically flushing them to a
@emph{target} handler. Flushing occurs whenever the buffer is full, or when an
event of a certain severity or greater is seen.

@ref{2582,,MemoryHandler} is a subclass of the more general
@ref{25cf,,BufferingHandler}, which is an abstract class. This buffers logging
records in memory. Whenever each record is added to the buffer, a check is made
by calling @code{shouldFlush()} to see if the buffer should be flushed.  If it
should, then @code{flush()} is expected to do the flushing.

@geindex BufferingHandler (class in logging.handlers)
@anchor{library/logging handlers logging handlers BufferingHandler}@anchor{25cf}
@deffn {Class} logging.handlers.BufferingHandler (capacity)

Initializes the handler with a buffer of the specified capacity. Here,
@emph{capacity} means the number of logging records buffered.

@geindex emit() (logging.handlers.BufferingHandler method)
@anchor{library/logging handlers logging handlers BufferingHandler emit}@anchor{25d0}
@deffn {Method} emit (record)

Append the record to the buffer. If @ref{25d1,,shouldFlush()} returns true,
call @ref{25d2,,flush()} to process the buffer.
@end deffn

@geindex flush() (logging.handlers.BufferingHandler method)
@anchor{library/logging handlers logging handlers BufferingHandler flush}@anchor{25d2}
@deffn {Method} flush ()

You can override this to implement custom flushing behavior. This version
just zaps the buffer to empty.
@end deffn

@geindex shouldFlush() (logging.handlers.BufferingHandler method)
@anchor{library/logging handlers logging handlers BufferingHandler shouldFlush}@anchor{25d1}
@deffn {Method} shouldFlush (record)

Return @code{True} if the buffer is up to capacity. This method can be
overridden to implement custom flushing strategies.
@end deffn
@end deffn

@geindex MemoryHandler (class in logging.handlers)
@anchor{library/logging handlers logging handlers MemoryHandler}@anchor{2582}
@deffn {Class} logging.handlers.MemoryHandler (capacity, flushLevel=ERROR, target=None, flushOnClose=True)

Returns a new instance of the @ref{2582,,MemoryHandler} class. The instance is
initialized with a buffer size of @emph{capacity} (number of records buffered).
If @emph{flushLevel} is not specified, @code{ERROR} is used. If no @emph{target} is
specified, the target will need to be set using @ref{25d3,,setTarget()} before this
handler does anything useful. If @emph{flushOnClose} is specified as @code{False},
then the buffer is @emph{not} flushed when the handler is closed. If not specified
or specified as @code{True}, the previous behaviour of flushing the buffer will
occur when the handler is closed.


Changed in version 3.6: The @emph{flushOnClose} parameter was added.


@geindex close() (logging.handlers.MemoryHandler method)
@anchor{library/logging handlers logging handlers MemoryHandler close}@anchor{25d4}
@deffn {Method} close ()

Calls @ref{25d5,,flush()}, sets the target to @code{None} and clears the
buffer.
@end deffn

@geindex flush() (logging.handlers.MemoryHandler method)
@anchor{library/logging handlers logging handlers MemoryHandler flush}@anchor{25d5}
@deffn {Method} flush ()

For a @ref{2582,,MemoryHandler}, flushing means just sending the buffered
records to the target, if there is one. The buffer is also cleared when
this happens. Override if you want different behavior.
@end deffn

@geindex setTarget() (logging.handlers.MemoryHandler method)
@anchor{library/logging handlers logging handlers MemoryHandler setTarget}@anchor{25d3}
@deffn {Method} setTarget (target)

Sets the target handler for this handler.
@end deffn

@geindex shouldFlush() (logging.handlers.MemoryHandler method)
@anchor{library/logging handlers logging handlers MemoryHandler shouldFlush}@anchor{25d6}
@deffn {Method} shouldFlush (record)

Checks for buffer full or a record at the @emph{flushLevel} or higher.
@end deffn
@end deffn

@node HTTPHandler,QueueHandler,MemoryHandler,logging handlers — Logging handlers
@anchor{library/logging handlers http-handler}@anchor{25d7}@anchor{library/logging handlers httphandler}@anchor{25d8}
@subsubsection HTTPHandler


The @ref{992,,HTTPHandler} class, located in the @ref{a8,,logging.handlers} module,
supports sending logging messages to a web server, using either @code{GET} or
@code{POST} semantics.

@geindex HTTPHandler (class in logging.handlers)
@anchor{library/logging handlers logging handlers HTTPHandler}@anchor{992}
@deffn {Class} logging.handlers.HTTPHandler (host, url, method='GET', secure=False, credentials=None, context=None)

Returns a new instance of the @ref{992,,HTTPHandler} class. The @emph{host} can be
of the form @code{host:port}, should you need to use a specific port number.  If
no @emph{method} is specified, @code{GET} is used. If @emph{secure} is true, a HTTPS
connection will be used. The @emph{context} parameter may be set to a
@ref{2bf,,ssl.SSLContext} instance to configure the SSL settings used for the
HTTPS connection. If @emph{credentials} is specified, it should be a 2-tuple
consisting of userid and password, which will be placed in a HTTP
‘Authorization’ header using Basic authentication. If you specify
credentials, you should also specify secure=True so that your userid and
password are not passed in cleartext across the wire.


Changed in version 3.5: The @emph{context} parameter was added.


@geindex mapLogRecord() (logging.handlers.HTTPHandler method)
@anchor{library/logging handlers logging handlers HTTPHandler mapLogRecord}@anchor{25d9}
@deffn {Method} mapLogRecord (record)

Provides a dictionary, based on @code{record}, which is to be URL-encoded
and sent to the web server. The default implementation just returns
@code{record.__dict__}. This method can be overridden if e.g. only a
subset of @ref{b6c,,LogRecord} is to be sent to the web server, or
if more specific customization of what’s sent to the server is required.
@end deffn

@geindex emit() (logging.handlers.HTTPHandler method)
@anchor{library/logging handlers logging handlers HTTPHandler emit}@anchor{25da}
@deffn {Method} emit (record)

Sends the record to the web server as a URL-encoded dictionary. The
@ref{25d9,,mapLogRecord()} method is used to convert the record to the
dictionary to be sent.
@end deffn

@cartouche
@quotation Note 
Since preparing a record for sending it to a web server is not
the same as a generic formatting operation, using
@ref{2537,,setFormatter()} to specify a
@ref{11c6,,Formatter} for a @ref{992,,HTTPHandler} has no effect.
Instead of calling @ref{2541,,format()}, this handler calls
@ref{25d9,,mapLogRecord()} and then @ref{a05,,urllib.parse.urlencode()} to encode the
dictionary in a form suitable for sending to a web server.
@end quotation
@end cartouche
@end deffn

@node QueueHandler,QueueListener,HTTPHandler,logging handlers — Logging handlers
@anchor{library/logging handlers queue-handler}@anchor{25db}@anchor{library/logging handlers queuehandler}@anchor{25dc}
@subsubsection QueueHandler



New in version 3.2.


The @ref{25dd,,QueueHandler} class, located in the @ref{a8,,logging.handlers} module,
supports sending logging messages to a queue, such as those implemented in the
@ref{d5,,queue} or @ref{b3,,multiprocessing} modules.

Along with the @ref{993,,QueueListener} class, @ref{25dd,,QueueHandler} can be used
to let handlers do their work on a separate thread from the one which does the
logging. This is important in web applications and also other service
applications where threads servicing clients need to respond as quickly as
possible, while any potentially slow operations (such as sending an email via
@ref{25ca,,SMTPHandler}) are done on a separate thread.

@geindex QueueHandler (class in logging.handlers)
@anchor{library/logging handlers logging handlers QueueHandler}@anchor{25dd}
@deffn {Class} logging.handlers.QueueHandler (queue)

Returns a new instance of the @ref{25dd,,QueueHandler} class. The instance is
initialized with the queue to send messages to. The @emph{queue} can be any
queue-like object; it’s used as-is by the @ref{25de,,enqueue()} method, which
needs to know how to send messages to it. The queue is not @emph{required} to
have the task tracking API, which means that you can use
@ref{695,,SimpleQueue} instances for @emph{queue}.

@geindex emit() (logging.handlers.QueueHandler method)
@anchor{library/logging handlers logging handlers QueueHandler emit}@anchor{25df}
@deffn {Method} emit (record)

Enqueues the result of preparing the LogRecord. Should an exception
occur (e.g. because a bounded queue has filled up), the
@ref{253f,,handleError()} method is called to handle the
error. This can result in the record silently being dropped (if
@code{logging.raiseExceptions} is @code{False}) or a message printed to
@code{sys.stderr} (if @code{logging.raiseExceptions} is @code{True}).
@end deffn

@geindex prepare() (logging.handlers.QueueHandler method)
@anchor{library/logging handlers logging handlers QueueHandler prepare}@anchor{25e0}
@deffn {Method} prepare (record)

Prepares a record for queuing. The object returned by this
method is enqueued.

The base implementation formats the record to merge the message,
arguments, and exception information, if present.  It also removes
unpickleable items from the record in-place. Specifically, it overwrites
the record’s @code{msg} and @code{message} attributes with the merged
message (obtained by calling the handler’s @ref{785,,format()} method), and
sets the @code{args}, @code{exc_info} and @code{exc_text} attributes
to @code{None}.

You might want to override this method if you want to convert
the record to a dict or JSON string, or send a modified copy
of the record while leaving the original intact.
@end deffn

@geindex enqueue() (logging.handlers.QueueHandler method)
@anchor{library/logging handlers logging handlers QueueHandler enqueue}@anchor{25de}
@deffn {Method} enqueue (record)

Enqueues the record on the queue using @code{put_nowait()}; you may
want to override this if you want to use blocking behaviour, or a
timeout, or a customized queue implementation.
@end deffn
@end deffn

@node QueueListener,,QueueHandler,logging handlers — Logging handlers
@anchor{library/logging handlers queue-listener}@anchor{25e1}@anchor{library/logging handlers queuelistener}@anchor{25e2}
@subsubsection QueueListener



New in version 3.2.


The @ref{993,,QueueListener} class, located in the @ref{a8,,logging.handlers}
module, supports receiving logging messages from a queue, such as those
implemented in the @ref{d5,,queue} or @ref{b3,,multiprocessing} modules. The
messages are received from a queue in an internal thread and passed, on
the same thread, to one or more handlers for processing. While
@ref{993,,QueueListener} is not itself a handler, it is documented here
because it works hand-in-hand with @ref{25dd,,QueueHandler}.

Along with the @ref{25dd,,QueueHandler} class, @ref{993,,QueueListener} can be used
to let handlers do their work on a separate thread from the one which does the
logging. This is important in web applications and also other service
applications where threads servicing clients need to respond as quickly as
possible, while any potentially slow operations (such as sending an email via
@ref{25ca,,SMTPHandler}) are done on a separate thread.

@geindex QueueListener (class in logging.handlers)
@anchor{library/logging handlers logging handlers QueueListener}@anchor{993}
@deffn {Class} logging.handlers.QueueListener (queue, *handlers, respect_handler_level=False)

Returns a new instance of the @ref{993,,QueueListener} class. The instance is
initialized with the queue to send messages to and a list of handlers which
will handle entries placed on the queue. The queue can be any queue-like
object; it’s passed as-is to the @ref{25e3,,dequeue()} method, which needs
to know how to get messages from it. The queue is not @emph{required} to have the
task tracking API (though it’s used if available), which means that you can
use @ref{695,,SimpleQueue} instances for @emph{queue}.

If @code{respect_handler_level} is @code{True}, a handler’s level is respected
(compared with the level for the message) when deciding whether to pass
messages to that handler; otherwise, the behaviour is as in previous Python
versions - to always pass each message to each handler.


Changed in version 3.5: The @code{respect_handler_level} argument was added.


@geindex dequeue() (logging.handlers.QueueListener method)
@anchor{library/logging handlers logging handlers QueueListener dequeue}@anchor{25e3}
@deffn {Method} dequeue (block)

Dequeues a record and return it, optionally blocking.

The base implementation uses @code{get()}. You may want to override this
method if you want to use timeouts or work with custom queue
implementations.
@end deffn

@geindex prepare() (logging.handlers.QueueListener method)
@anchor{library/logging handlers logging handlers QueueListener prepare}@anchor{25e4}
@deffn {Method} prepare (record)

Prepare a record for handling.

This implementation just returns the passed-in record. You may want to
override this method if you need to do any custom marshalling or
manipulation of the record before passing it to the handlers.
@end deffn

@geindex handle() (logging.handlers.QueueListener method)
@anchor{library/logging handlers logging handlers QueueListener handle}@anchor{25e5}
@deffn {Method} handle (record)

Handle a record.

This just loops through the handlers offering them the record
to handle. The actual object passed to the handlers is that which
is returned from @ref{25e4,,prepare()}.
@end deffn

@geindex start() (logging.handlers.QueueListener method)
@anchor{library/logging handlers logging handlers QueueListener start}@anchor{25e6}
@deffn {Method} start ()

Starts the listener.

This starts up a background thread to monitor the queue for
LogRecords to process.
@end deffn

@geindex stop() (logging.handlers.QueueListener method)
@anchor{library/logging handlers logging handlers QueueListener stop}@anchor{25e7}
@deffn {Method} stop ()

Stops the listener.

This asks the thread to terminate, and then waits for it to do so.
Note that if you don’t call this before your application exits, there
may be some records still left on the queue, which won’t be processed.
@end deffn

@geindex enqueue_sentinel() (logging.handlers.QueueListener method)
@anchor{library/logging handlers logging handlers QueueListener enqueue_sentinel}@anchor{25e8}
@deffn {Method} enqueue_sentinel ()

Writes a sentinel to the queue to tell the listener to quit. This
implementation uses @code{put_nowait()}.  You may want to override this
method if you want to use timeouts or work with custom queue
implementations.


New in version 3.3.

@end deffn
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{a6,,logging}

API reference for the logging module.

@item Module @ref{a7,,logging.config}

Configuration API for the logging module.
@end table


@node getpass — Portable password input,curses — Terminal handling for character-cell displays,logging handlers — Logging handlers,Generic Operating System Services
@anchor{library/getpass doc}@anchor{25e9}@anchor{library/getpass getpass-portable-password-input}@anchor{25ea}@anchor{library/getpass module-getpass}@anchor{82}
@subsection @code{getpass} — Portable password input


@geindex module; getpass

@c Windows (& Mac?) support by Guido van Rossum.

@strong{Source code:} Lib/getpass.py@footnote{https://github.com/python/cpython/tree/main/Lib/getpass.py}


__________________________________________________________________


The @ref{82,,getpass} module provides two functions:

@geindex getpass() (in module getpass)
@anchor{library/getpass getpass getpass}@anchor{25eb}
@deffn {Function} getpass.getpass (prompt='Password: ', stream=None)

Prompt the user for a password without echoing.  The user is prompted using
the string @emph{prompt}, which defaults to @code{'Password: '}.  On Unix, the
prompt is written to the file-like object @emph{stream} using the replace error
handler if needed.  @emph{stream} defaults to the controlling terminal
(@code{/dev/tty}) or if that is unavailable to @code{sys.stderr} (this
argument is ignored on Windows).

If echo free input is unavailable getpass() falls back to printing
a warning message to @emph{stream} and reading from @code{sys.stdin} and
issuing a @ref{25ec,,GetPassWarning}.

@cartouche
@quotation Note 
If you call getpass from within IDLE, the input may be done in the
terminal you launched IDLE from rather than the idle window itself.
@end quotation
@end cartouche
@end deffn

@geindex GetPassWarning
@anchor{library/getpass getpass GetPassWarning}@anchor{25ec}
@deffn {Exception} getpass.GetPassWarning

A @ref{1b8e,,UserWarning} subclass issued when password input may be echoed.
@end deffn

@geindex getuser() (in module getpass)
@anchor{library/getpass getpass getuser}@anchor{2388}
@deffn {Function} getpass.getuser ()

Return the “login name” of the user.

This function checks the environment variables 
@geindex LOGNAME
@geindex environment variable; LOGNAME
@code{LOGNAME},
@geindex USER
@geindex environment variable; USER
@code{USER}, 
@geindex LNAME
@geindex environment variable; LNAME
@code{LNAME} and 
@geindex USERNAME
@geindex environment variable; USERNAME
@code{USERNAME}, in order, and
returns the value of the first one which is set to a non-empty string.  If
none are set, the login name from the password database is returned on
systems which support the @ref{d1,,pwd} module, otherwise, an exception is
raised.

In general, this function should be preferred over @ref{2387,,os.getlogin()}.
@end deffn

@node curses — Terminal handling for character-cell displays,curses textpad — Text input widget for curses programs,getpass — Portable password input,Generic Operating System Services
@anchor{library/curses doc}@anchor{25ed}@anchor{library/curses curses-terminal-handling-for-character-cell-displays}@anchor{25ee}@anchor{library/curses module-curses}@anchor{2a}
@subsection @code{curses} — Terminal handling for character-cell displays


@geindex module; curses


__________________________________________________________________


The @ref{2a,,curses} module provides an interface to the curses library, the
de-facto standard for portable advanced terminal handling.

While curses is most widely used in the Unix environment, versions are available
for Windows, DOS, and possibly other systems as well.  This extension module is
designed to match the API of ncurses, an open-source curses library hosted on
Linux and the BSD variants of Unix.

@cartouche
@quotation Note 
Whenever the documentation mentions a @emph{character} it can be specified
as an integer, a one-character Unicode string or a one-byte byte string.

Whenever the documentation mentions a @emph{character string} it can be specified
as a Unicode string or a byte string.
@end quotation
@end cartouche

@cartouche
@quotation Note 
Since version 5.4, the ncurses library decides how to interpret non-ASCII data
using the @code{nl_langinfo} function.  That means that you have to call
@ref{1b4,,locale.setlocale()} in the application and encode Unicode strings
using one of the system’s available encodings.  This example uses the
system’s default encoding:

@example
import locale
locale.setlocale(locale.LC_ALL, '@w{'})
code = locale.getpreferredencoding()
@end example

Then use @emph{code} as the encoding for @ref{3ba,,str.encode()} calls.
@end quotation
@end cartouche


@subsubheading See also



@table @asis

@item Module @ref{2b,,curses.ascii}

Utilities for working with ASCII characters, regardless of your locale settings.

@item Module @ref{2c,,curses.panel}

A panel stack extension that adds depth to  curses windows.

@item Module @ref{2d,,curses.textpad}

Editable text widget for curses supporting  @code{Emacs}-like bindings.

@item @ref{25ef,,Curses Programming with Python}

Tutorial material on using curses with Python, by Andrew Kuchling and Eric
Raymond.
@end table

The Tools/demo/@footnote{https://github.com/python/cpython/tree/main/Tools/demo/} directory in the Python source distribution contains
some example programs using the curses bindings provided by this module.


@menu
* Functions: Functions<4>. 
* Window Objects:: 
* Constants: Constants<6>. 

@end menu

@node Functions<4>,Window Objects,,curses — Terminal handling for character-cell displays
@anchor{library/curses curses-functions}@anchor{25f0}@anchor{library/curses functions}@anchor{25f1}
@subsubsection Functions


The module @ref{2a,,curses} defines the following exception:

@geindex error
@anchor{library/curses curses error}@anchor{25f2}
@deffn {Exception} curses.error

Exception raised when a curses library function returns an error.
@end deffn

@cartouche
@quotation Note 
Whenever @emph{x} or @emph{y} arguments to a function or a method are optional, they
default to the current cursor location. Whenever @emph{attr} is optional, it defaults
to @code{A_NORMAL}.
@end quotation
@end cartouche

The module @ref{2a,,curses} defines the following functions:

@geindex baudrate() (in module curses)
@anchor{library/curses curses baudrate}@anchor{25f3}
@deffn {Function} curses.baudrate ()

Return the output speed of the terminal in bits per second.  On software
terminal emulators it will have a fixed high value. Included for historical
reasons; in former times, it was used to  write output loops for time delays and
occasionally to change interfaces depending on the line speed.
@end deffn

@geindex beep() (in module curses)
@anchor{library/curses curses beep}@anchor{25f4}
@deffn {Function} curses.beep ()

Emit a short attention sound.
@end deffn

@geindex can_change_color() (in module curses)
@anchor{library/curses curses can_change_color}@anchor{25f5}
@deffn {Function} curses.can_change_color ()

Return @code{True} or @code{False}, depending on whether the programmer can change the colors
displayed by the terminal.
@end deffn

@geindex cbreak() (in module curses)
@anchor{library/curses curses cbreak}@anchor{25f6}
@deffn {Function} curses.cbreak ()

Enter cbreak mode.  In cbreak mode (sometimes called “rare” mode) normal tty
line buffering is turned off and characters are available to be read one by one.
However, unlike raw mode, special characters (interrupt, quit, suspend, and flow
control) retain their effects on the tty driver and calling program.  Calling
first @ref{25f7,,raw()} then @ref{25f6,,cbreak()} leaves the terminal in cbreak mode.
@end deffn

@geindex color_content() (in module curses)
@anchor{library/curses curses color_content}@anchor{26f}
@deffn {Function} curses.color_content (color_number)

Return the intensity of the red, green, and blue (RGB) components in the color
@emph{color_number}, which must be between @code{0} and @code{COLORS - 1}.  Return a 3-tuple,
containing the R,G,B values for the given color, which will be between
@code{0} (no component) and @code{1000} (maximum amount of component).
@end deffn

@geindex color_pair() (in module curses)
@anchor{library/curses curses color_pair}@anchor{25f8}
@deffn {Function} curses.color_pair (pair_number)

Return the attribute value for displaying text in the specified color pair.
Only the first 256 color pairs are supported. This
attribute value can be combined with @code{A_STANDOUT}, @code{A_REVERSE},
and the other @code{A_*} attributes.  @ref{25f9,,pair_number()} is the counterpart
to this function.
@end deffn

@geindex curs_set() (in module curses)
@anchor{library/curses curses curs_set}@anchor{25fa}
@deffn {Function} curses.curs_set (visibility)

Set the cursor state.  @emph{visibility} can be set to @code{0}, @code{1}, or @code{2}, for invisible,
normal, or very visible.  If the terminal supports the visibility requested, return the
previous cursor state; otherwise raise an exception.  On many
terminals, the “visible” mode is an underline cursor and the “very visible” mode
is a block cursor.
@end deffn

@geindex def_prog_mode() (in module curses)
@anchor{library/curses curses def_prog_mode}@anchor{25fb}
@deffn {Function} curses.def_prog_mode ()

Save the current terminal mode as the “program” mode, the mode when the running
program is using curses.  (Its counterpart is the “shell” mode, for when the
program is not in curses.)  Subsequent calls to @ref{25fc,,reset_prog_mode()} will
restore this mode.
@end deffn

@geindex def_shell_mode() (in module curses)
@anchor{library/curses curses def_shell_mode}@anchor{25fd}
@deffn {Function} curses.def_shell_mode ()

Save the current terminal mode as the “shell” mode, the mode when the running
program is not using curses.  (Its counterpart is the “program” mode, when the
program is using curses capabilities.) Subsequent calls to
@ref{25fe,,reset_shell_mode()} will restore this mode.
@end deffn

@geindex delay_output() (in module curses)
@anchor{library/curses curses delay_output}@anchor{25ff}
@deffn {Function} curses.delay_output (ms)

Insert an @emph{ms} millisecond pause in output.
@end deffn

@geindex doupdate() (in module curses)
@anchor{library/curses curses doupdate}@anchor{2600}
@deffn {Function} curses.doupdate ()

Update the physical screen.  The curses library keeps two data structures, one
representing the current physical screen contents and a virtual screen
representing the desired next state.  The @ref{2600,,doupdate()} ground updates the
physical screen to match the virtual screen.

The virtual screen may be updated by a @ref{2601,,noutrefresh()} call after write
operations such as @ref{2602,,addstr()} have been performed on a window.  The normal
@ref{2603,,refresh()} call is simply @code{noutrefresh()} followed by @code{doupdate()};
if you have to update multiple windows, you can speed performance and perhaps
reduce screen flicker by issuing @code{noutrefresh()} calls on all windows,
followed by a single @code{doupdate()}.
@end deffn

@geindex echo() (in module curses)
@anchor{library/curses curses echo}@anchor{2604}
@deffn {Function} curses.echo ()

Enter echo mode.  In echo mode, each character input is echoed to the screen as
it is entered.
@end deffn

@geindex endwin() (in module curses)
@anchor{library/curses curses endwin}@anchor{2605}
@deffn {Function} curses.endwin ()

De-initialize the library, and return terminal to normal status.
@end deffn

@geindex erasechar() (in module curses)
@anchor{library/curses curses erasechar}@anchor{2606}
@deffn {Function} curses.erasechar ()

Return the user’s current erase character as a one-byte bytes object.  Under Unix operating systems this
is a property of the controlling tty of the curses program, and is not set by
the curses library itself.
@end deffn

@geindex filter() (in module curses)
@anchor{library/curses curses filter}@anchor{2607}
@deffn {Function} curses.filter ()

The @ref{2607,,filter()} routine, if used, must be called before @ref{2608,,initscr()} is
called.  The effect is that, during those calls, 
@geindex LINES
@geindex environment variable; LINES
@code{LINES} is set to @code{1}; the
capabilities @code{clear}, @code{cup}, @code{cud}, @code{cud1}, @code{cuu1}, @code{cuu}, @code{vpa} are disabled; and the @code{home}
string is set to the value of @code{cr}. The effect is that the cursor is confined to
the current line, and so are screen updates.  This may be used for enabling
character-at-a-time  line editing without touching the rest of the screen.
@end deffn

@geindex flash() (in module curses)
@anchor{library/curses curses flash}@anchor{2609}
@deffn {Function} curses.flash ()

Flash the screen.  That is, change it to reverse-video and then change it back
in a short interval.  Some people prefer such as ‘visible bell’ to the audible
attention signal produced by @ref{25f4,,beep()}.
@end deffn

@geindex flushinp() (in module curses)
@anchor{library/curses curses flushinp}@anchor{260a}
@deffn {Function} curses.flushinp ()

Flush all input buffers.  This throws away any  typeahead  that  has been typed
by the user and has not yet been processed by the program.
@end deffn

@geindex getmouse() (in module curses)
@anchor{library/curses curses getmouse}@anchor{260b}
@deffn {Function} curses.getmouse ()

After @ref{260c,,getch()} returns @code{KEY_MOUSE} to signal a mouse event, this
method should be called to retrieve the queued mouse event, represented as a
5-tuple @code{(id, x, y, z, bstate)}. @emph{id} is an ID value used to distinguish
multiple devices, and @emph{x}, @emph{y}, @emph{z} are the event’s coordinates.  (@emph{z} is
currently unused.)  @emph{bstate} is an integer value whose bits will be set to
indicate the type of event, and will be the bitwise OR of one or more of the
following constants, where @emph{n} is the button number from 1 to 5:
@code{BUTTONn_PRESSED}, @code{BUTTONn_RELEASED}, @code{BUTTONn_CLICKED},
@code{BUTTONn_DOUBLE_CLICKED}, @code{BUTTONn_TRIPLE_CLICKED},
@code{BUTTON_SHIFT}, @code{BUTTON_CTRL}, @code{BUTTON_ALT}.


Changed in version 3.10: The @code{BUTTON5_*} constants are now exposed if they are provided by the
underlying curses library.

@end deffn

@geindex getsyx() (in module curses)
@anchor{library/curses curses getsyx}@anchor{260d}
@deffn {Function} curses.getsyx ()

Return the current coordinates of the virtual screen cursor as a tuple
@code{(y, x)}.  If @ref{260e,,leaveok} is currently @code{True}, then return @code{(-1, -1)}.
@end deffn

@geindex getwin() (in module curses)
@anchor{library/curses curses getwin}@anchor{260f}
@deffn {Function} curses.getwin (file)

Read window related data stored in the file by an earlier @code{putwin()} call.
The routine then creates and initializes a new window using that data, returning
the new window object.
@end deffn

@geindex has_colors() (in module curses)
@anchor{library/curses curses has_colors}@anchor{2610}
@deffn {Function} curses.has_colors ()

Return @code{True} if the terminal can display colors; otherwise, return @code{False}.
@end deffn

@geindex has_extended_color_support() (in module curses)
@anchor{library/curses curses has_extended_color_support}@anchor{273}
@deffn {Function} curses.has_extended_color_support ()

Return @code{True} if the module supports extended colors; otherwise, return
@code{False}. Extended color support allows more than 256 color pairs for
terminals that support more than 16 colors (e.g. xterm-256color).

Extended color support requires ncurses version 6.1 or later.


New in version 3.10.

@end deffn

@geindex has_ic() (in module curses)
@anchor{library/curses curses has_ic}@anchor{2611}
@deffn {Function} curses.has_ic ()

Return @code{True} if the terminal has insert- and delete-character capabilities.
This function is included for historical reasons only, as all modern software
terminal emulators have such capabilities.
@end deffn

@geindex has_il() (in module curses)
@anchor{library/curses curses has_il}@anchor{2612}
@deffn {Function} curses.has_il ()

Return @code{True} if the terminal has insert- and delete-line capabilities, or can
simulate  them  using scrolling regions. This function is included for
historical reasons only, as all modern software terminal emulators have such
capabilities.
@end deffn

@geindex has_key() (in module curses)
@anchor{library/curses curses has_key}@anchor{2613}
@deffn {Function} curses.has_key (ch)

Take a key value @emph{ch}, and return @code{True} if the current terminal type recognizes
a key with that value.
@end deffn

@geindex halfdelay() (in module curses)
@anchor{library/curses curses halfdelay}@anchor{2614}
@deffn {Function} curses.halfdelay (tenths)

Used for half-delay mode, which is similar to cbreak mode in that characters
typed by the user are immediately available to the program. However, after
blocking for @emph{tenths} tenths of seconds, raise an exception if nothing has
been typed.  The value of @emph{tenths} must be a number between @code{1} and @code{255}.  Use
@ref{2615,,nocbreak()} to leave half-delay mode.
@end deffn

@geindex init_color() (in module curses)
@anchor{library/curses curses init_color}@anchor{270}
@deffn {Function} curses.init_color (color_number, r, g, b)

Change the definition of a color, taking the number of the color to be changed
followed by three RGB values (for the amounts of red, green, and blue
components).  The value of @emph{color_number} must be between @code{0} and
@cite{COLORS - 1}.  Each of @emph{r}, @emph{g}, @emph{b}, must be a value between @code{0} and
@code{1000}.  When @ref{270,,init_color()} is used, all occurrences of that color on the
screen immediately change to the new definition.  This function is a no-op on
most terminals; it is active only if @ref{25f5,,can_change_color()} returns @code{True}.
@end deffn

@geindex init_pair() (in module curses)
@anchor{library/curses curses init_pair}@anchor{271}
@deffn {Function} curses.init_pair (pair_number, fg, bg)

Change the definition of a color-pair.  It takes three arguments: the number of
the color-pair to be changed, the foreground color number, and the background
color number.  The value of @emph{pair_number} must be between @code{1} and
@code{COLOR_PAIRS - 1} (the @code{0} color pair is wired to white on black and cannot
be changed).  The value of @emph{fg} and @emph{bg} arguments must be between @code{0} and
@code{COLORS - 1}, or, after calling @ref{2616,,use_default_colors()}, @code{-1}.
If the color-pair was previously initialized, the screen is
refreshed and all occurrences of that color-pair are changed to the new
definition.
@end deffn

@geindex initscr() (in module curses)
@anchor{library/curses curses initscr}@anchor{2608}
@deffn {Function} curses.initscr ()

Initialize the library. Return a @ref{2617,,window} object
which represents the whole screen.

@cartouche
@quotation Note 
If there is an error opening the terminal, the underlying curses library may
cause the interpreter to exit.
@end quotation
@end cartouche
@end deffn

@geindex is_term_resized() (in module curses)
@anchor{library/curses curses is_term_resized}@anchor{2618}
@deffn {Function} curses.is_term_resized (nlines, ncols)

Return @code{True} if @ref{2619,,resize_term()} would modify the window structure,
@code{False} otherwise.
@end deffn

@geindex isendwin() (in module curses)
@anchor{library/curses curses isendwin}@anchor{261a}
@deffn {Function} curses.isendwin ()

Return @code{True} if @ref{2605,,endwin()} has been called (that is, the  curses library has
been deinitialized).
@end deffn

@geindex keyname() (in module curses)
@anchor{library/curses curses keyname}@anchor{261b}
@deffn {Function} curses.keyname (k)

Return the name of the key numbered @emph{k} as a bytes object.  The name of a key generating printable
ASCII character is the key’s character.  The name of a control-key combination
is a two-byte bytes object consisting of a caret (@code{b'^'}) followed by the corresponding
printable ASCII character.  The name of an alt-key combination (128–255) is a
bytes object consisting of the prefix @code{b'M-'} followed by the name of the corresponding
ASCII character.
@end deffn

@geindex killchar() (in module curses)
@anchor{library/curses curses killchar}@anchor{261c}
@deffn {Function} curses.killchar ()

Return the user’s current line kill character as a one-byte bytes object. Under Unix operating systems
this is a property of the controlling tty of the curses program, and is not set
by the curses library itself.
@end deffn

@geindex longname() (in module curses)
@anchor{library/curses curses longname}@anchor{261d}
@deffn {Function} curses.longname ()

Return a bytes object containing the terminfo long name field describing the current
terminal.  The maximum length of a verbose description is 128 characters.  It is
defined only after the call to @ref{2608,,initscr()}.
@end deffn

@geindex meta() (in module curses)
@anchor{library/curses curses meta}@anchor{261e}
@deffn {Function} curses.meta (flag)

If @emph{flag} is @code{True}, allow 8-bit characters to be input.  If
@emph{flag} is @code{False},  allow only 7-bit chars.
@end deffn

@geindex mouseinterval() (in module curses)
@anchor{library/curses curses mouseinterval}@anchor{261f}
@deffn {Function} curses.mouseinterval (interval)

Set the maximum time in milliseconds that can elapse between press and release
events in order for them to be recognized as a click, and return the previous
interval value.  The default value is 200 milliseconds, or one fifth of a second.
@end deffn

@geindex mousemask() (in module curses)
@anchor{library/curses curses mousemask}@anchor{2620}
@deffn {Function} curses.mousemask (mousemask)

Set the mouse events to be reported, and return a tuple @code{(availmask,
oldmask)}.   @emph{availmask} indicates which of the specified mouse events can be
reported; on complete failure it returns @code{0}.  @emph{oldmask} is the previous value of
the given window’s mouse event mask.  If this function is never called, no mouse
events are ever reported.
@end deffn

@geindex napms() (in module curses)
@anchor{library/curses curses napms}@anchor{2621}
@deffn {Function} curses.napms (ms)

Sleep for @emph{ms} milliseconds.
@end deffn

@geindex newpad() (in module curses)
@anchor{library/curses curses newpad}@anchor{2622}
@deffn {Function} curses.newpad (nlines, ncols)

Create and return a pointer to a new pad data structure with the given number
of lines and columns.  Return a pad as a window object.

A pad is like a window, except that it is not restricted by the screen size, and
is not necessarily associated with a particular part of the screen.  Pads can be
used when a large window is needed, and only a part of the window will be on the
screen at one time.  Automatic refreshes of pads (such as from scrolling or
echoing of input) do not occur.  The @ref{2603,,refresh()} and @ref{2601,,noutrefresh()}
methods of a pad require 6 arguments to specify the part of the pad to be
displayed and the location on the screen to be used for the display. The
arguments are @emph{pminrow}, @emph{pmincol}, @emph{sminrow}, @emph{smincol}, @emph{smaxrow}, @emph{smaxcol}; the @emph{p}
arguments refer to the upper left corner of the pad region to be displayed and
the @emph{s} arguments define a clipping box on the screen within which the pad region
is to be displayed.
@end deffn

@geindex newwin() (in module curses)
@anchor{library/curses curses newwin}@anchor{2623}
@deffn {Function} curses.newwin (nlines, ncols)

@deffnx {Function} curses.newwin (nlines, ncols, begin_y, begin_x)

Return a new @ref{2617,,window}, whose left-upper corner
is at  @code{(begin_y, begin_x)}, and whose height/width is  @emph{nlines}/@emph{ncols}.

By default, the window will extend from the  specified position to the lower
right corner of the screen.
@end deffn

@geindex nl() (in module curses)
@anchor{library/curses curses nl}@anchor{2624}
@deffn {Function} curses.nl ()

Enter newline mode.  This mode translates the return key into newline on input,
and translates newline into return and line-feed on output. Newline mode is
initially on.
@end deffn

@geindex nocbreak() (in module curses)
@anchor{library/curses curses nocbreak}@anchor{2615}
@deffn {Function} curses.nocbreak ()

Leave cbreak mode.  Return to normal “cooked” mode with line buffering.
@end deffn

@geindex noecho() (in module curses)
@anchor{library/curses curses noecho}@anchor{2625}
@deffn {Function} curses.noecho ()

Leave echo mode.  Echoing of input characters is turned off.
@end deffn

@geindex nonl() (in module curses)
@anchor{library/curses curses nonl}@anchor{2626}
@deffn {Function} curses.nonl ()

Leave newline mode.  Disable translation of return into newline on input, and
disable low-level translation of newline into newline/return on output (but this
does not change the behavior of @code{addch('\n')}, which always does the
equivalent of return and line feed on the virtual screen).  With translation
off, curses can sometimes speed up vertical motion a little; also, it will be
able to detect the return key on input.
@end deffn

@geindex noqiflush() (in module curses)
@anchor{library/curses curses noqiflush}@anchor{2627}
@deffn {Function} curses.noqiflush ()

When the @code{noqiflush()} routine is used, normal flush of input and output queues
associated with the @code{INTR}, @code{QUIT} and @code{SUSP} characters will not be done.  You may
want to call @code{noqiflush()} in a signal handler if you want output to
continue as though the interrupt had not occurred, after the handler exits.
@end deffn

@geindex noraw() (in module curses)
@anchor{library/curses curses noraw}@anchor{2628}
@deffn {Function} curses.noraw ()

Leave raw mode. Return to normal “cooked” mode with line buffering.
@end deffn

@geindex pair_content() (in module curses)
@anchor{library/curses curses pair_content}@anchor{272}
@deffn {Function} curses.pair_content (pair_number)

Return a tuple @code{(fg, bg)} containing the colors for the requested color pair.
The value of @emph{pair_number} must be between @code{0} and @code{COLOR_PAIRS - 1}.
@end deffn

@geindex pair_number() (in module curses)
@anchor{library/curses curses pair_number}@anchor{25f9}
@deffn {Function} curses.pair_number (attr)

Return the number of the color-pair set by the attribute value @emph{attr}.
@ref{25f8,,color_pair()} is the counterpart to this function.
@end deffn

@geindex putp() (in module curses)
@anchor{library/curses curses putp}@anchor{2629}
@deffn {Function} curses.putp (str)

Equivalent to @code{tputs(str, 1, putchar)}; emit the value of a specified
terminfo capability for the current terminal.  Note that the output of @ref{2629,,putp()}
always goes to standard output.
@end deffn

@geindex qiflush() (in module curses)
@anchor{library/curses curses qiflush}@anchor{262a}
@deffn {Function} curses.qiflush ([flag])

If @emph{flag} is @code{False}, the effect is the same as calling @ref{2627,,noqiflush()}. If
@emph{flag} is @code{True}, or no argument is provided, the queues will be flushed when
these control characters are read.
@end deffn

@geindex raw() (in module curses)
@anchor{library/curses curses raw}@anchor{25f7}
@deffn {Function} curses.raw ()

Enter raw mode.  In raw mode, normal line buffering and  processing of
interrupt, quit, suspend, and flow control keys are turned off; characters are
presented to curses input functions one by one.
@end deffn

@geindex reset_prog_mode() (in module curses)
@anchor{library/curses curses reset_prog_mode}@anchor{25fc}
@deffn {Function} curses.reset_prog_mode ()

Restore the  terminal  to “program” mode, as previously saved  by
@ref{25fb,,def_prog_mode()}.
@end deffn

@geindex reset_shell_mode() (in module curses)
@anchor{library/curses curses reset_shell_mode}@anchor{25fe}
@deffn {Function} curses.reset_shell_mode ()

Restore the  terminal  to “shell” mode, as previously saved  by
@ref{25fd,,def_shell_mode()}.
@end deffn

@geindex resetty() (in module curses)
@anchor{library/curses curses resetty}@anchor{262b}
@deffn {Function} curses.resetty ()

Restore the state of the terminal modes to what it was at the last call to
@ref{262c,,savetty()}.
@end deffn

@geindex resize_term() (in module curses)
@anchor{library/curses curses resize_term}@anchor{2619}
@deffn {Function} curses.resize_term (nlines, ncols)

Backend function used by @ref{262d,,resizeterm()}, performing most of the work;
when resizing the windows, @ref{2619,,resize_term()} blank-fills the areas that are
extended.  The calling application should fill in these areas with
appropriate data.  The @code{resize_term()} function attempts to resize all
windows.  However, due to the calling convention of pads, it is not possible
to resize these without additional interaction with the application.
@end deffn

@geindex resizeterm() (in module curses)
@anchor{library/curses curses resizeterm}@anchor{262d}
@deffn {Function} curses.resizeterm (nlines, ncols)

Resize the standard and current windows to the specified dimensions, and
adjusts other bookkeeping data used by the curses library that record the
window dimensions (in particular the SIGWINCH handler).
@end deffn

@geindex savetty() (in module curses)
@anchor{library/curses curses savetty}@anchor{262c}
@deffn {Function} curses.savetty ()

Save the current state of the terminal modes in a buffer, usable by
@ref{262b,,resetty()}.
@end deffn

@geindex get_escdelay() (in module curses)
@anchor{library/curses curses get_escdelay}@anchor{3dc}
@deffn {Function} curses.get_escdelay ()

Retrieves the value set by @ref{3dd,,set_escdelay()}.


New in version 3.9.

@end deffn

@geindex set_escdelay() (in module curses)
@anchor{library/curses curses set_escdelay}@anchor{3dd}
@deffn {Function} curses.set_escdelay (ms)

Sets the number of milliseconds to wait after reading an escape character,
to distinguish between an individual escape character entered on the
keyboard from escape sequences sent by cursor and function keys.


New in version 3.9.

@end deffn

@geindex get_tabsize() (in module curses)
@anchor{library/curses curses get_tabsize}@anchor{3de}
@deffn {Function} curses.get_tabsize ()

Retrieves the value set by @ref{3df,,set_tabsize()}.


New in version 3.9.

@end deffn

@geindex set_tabsize() (in module curses)
@anchor{library/curses curses set_tabsize}@anchor{3df}
@deffn {Function} curses.set_tabsize (size)

Sets the number of columns used by the curses library when converting a tab
character to spaces as it adds the tab to a window.


New in version 3.9.

@end deffn

@geindex setsyx() (in module curses)
@anchor{library/curses curses setsyx}@anchor{262e}
@deffn {Function} curses.setsyx (y, x)

Set the virtual screen cursor to @emph{y}, @emph{x}. If @emph{y} and @emph{x} are both @code{-1}, then
@ref{260e,,leaveok} is set @code{True}.
@end deffn

@geindex setupterm() (in module curses)
@anchor{library/curses curses setupterm}@anchor{262f}
@deffn {Function} curses.setupterm (term=None, fd=- 1)

Initialize the terminal.  @emph{term} is a string giving
the terminal name, or @code{None}; if omitted or @code{None}, the value of the
@geindex TERM
@geindex environment variable; TERM
@code{TERM} environment variable will be used.  @emph{fd} is the
file descriptor to which any initialization sequences will be sent; if not
supplied or @code{-1}, the file descriptor for @code{sys.stdout} will be used.
@end deffn

@geindex start_color() (in module curses)
@anchor{library/curses curses start_color}@anchor{2630}
@deffn {Function} curses.start_color ()

Must be called if the programmer wants to use colors, and before any other color
manipulation routine is called.  It is good practice to call this routine right
after @ref{2608,,initscr()}.

@ref{2630,,start_color()} initializes eight basic colors (black, red,  green, yellow,
blue, magenta, cyan, and white), and two global variables in the @ref{2a,,curses}
module, @code{COLORS} and @code{COLOR_PAIRS}, containing the maximum number
of colors and color-pairs the terminal can support.  It also restores the colors
on the terminal to the values they had when the terminal was just turned on.
@end deffn

@geindex termattrs() (in module curses)
@anchor{library/curses curses termattrs}@anchor{2631}
@deffn {Function} curses.termattrs ()

Return a logical OR of all video attributes supported by the terminal.  This
information is useful when a curses program needs complete control over the
appearance of the screen.
@end deffn

@geindex termname() (in module curses)
@anchor{library/curses curses termname}@anchor{2632}
@deffn {Function} curses.termname ()

Return the value of the environment variable 
@geindex TERM
@geindex environment variable; TERM
@code{TERM}, as a bytes object,
truncated to 14 characters.
@end deffn

@geindex tigetflag() (in module curses)
@anchor{library/curses curses tigetflag}@anchor{2633}
@deffn {Function} curses.tigetflag (capname)

Return the value of the Boolean capability corresponding to the terminfo
capability name @emph{capname} as an integer.  Return the value @code{-1} if @emph{capname} is not a
Boolean capability, or @code{0} if it is canceled or absent from the terminal
description.
@end deffn

@geindex tigetnum() (in module curses)
@anchor{library/curses curses tigetnum}@anchor{2634}
@deffn {Function} curses.tigetnum (capname)

Return the value of the numeric capability corresponding to the terminfo
capability name @emph{capname} as an integer.  Return the value @code{-2} if @emph{capname} is not a
numeric capability, or @code{-1} if it is canceled or absent from the terminal
description.
@end deffn

@geindex tigetstr() (in module curses)
@anchor{library/curses curses tigetstr}@anchor{2635}
@deffn {Function} curses.tigetstr (capname)

Return the value of the string capability corresponding to the terminfo
capability name @emph{capname} as a bytes object.  Return @code{None} if @emph{capname}
is not a terminfo “string capability”, or is canceled or absent from the
terminal description.
@end deffn

@geindex tparm() (in module curses)
@anchor{library/curses curses tparm}@anchor{2636}
@deffn {Function} curses.tparm (str[, ...])

Instantiate the bytes object @emph{str} with the supplied parameters, where @emph{str} should
be a parameterized string obtained from the terminfo database.  E.g.
@code{tparm(tigetstr("cup"), 5, 3)} could result in @code{b'\033[6;4H'}, the exact
result depending on terminal type.
@end deffn

@geindex typeahead() (in module curses)
@anchor{library/curses curses typeahead}@anchor{2637}
@deffn {Function} curses.typeahead (fd)

Specify that the file descriptor @emph{fd} be used for typeahead checking.  If @emph{fd}
is @code{-1}, then no typeahead checking is done.

The curses library does “line-breakout optimization” by looking for typeahead
periodically while updating the screen.  If input is found, and it is coming
from a tty, the current update is postponed until refresh or doupdate is called
again, allowing faster response to commands typed in advance. This function
allows specifying a different file descriptor for typeahead checking.
@end deffn

@geindex unctrl() (in module curses)
@anchor{library/curses curses unctrl}@anchor{2638}
@deffn {Function} curses.unctrl (ch)

Return a bytes object which is a printable representation of the character @emph{ch}.
Control characters are represented as a caret followed by the character, for
example as @code{b'^C'}. Printing characters are left as they are.
@end deffn

@geindex ungetch() (in module curses)
@anchor{library/curses curses ungetch}@anchor{2639}
@deffn {Function} curses.ungetch (ch)

Push @emph{ch} so the next @ref{260c,,getch()} will return it.

@cartouche
@quotation Note 
Only one @emph{ch} can be pushed before @code{getch()} is called.
@end quotation
@end cartouche
@end deffn

@geindex update_lines_cols() (in module curses)
@anchor{library/curses curses update_lines_cols}@anchor{94b}
@deffn {Function} curses.update_lines_cols ()

Update 
@geindex LINES
@geindex environment variable; LINES
@code{LINES} and 
@geindex COLS
@geindex environment variable; COLS
@code{COLS}. Useful for detecting manual screen resize.


New in version 3.5.

@end deffn

@geindex unget_wch() (in module curses)
@anchor{library/curses curses unget_wch}@anchor{c30}
@deffn {Function} curses.unget_wch (ch)

Push @emph{ch} so the next @ref{c2f,,get_wch()} will return it.

@cartouche
@quotation Note 
Only one @emph{ch} can be pushed before @code{get_wch()} is called.
@end quotation
@end cartouche


New in version 3.3.

@end deffn

@geindex ungetmouse() (in module curses)
@anchor{library/curses curses ungetmouse}@anchor{263a}
@deffn {Function} curses.ungetmouse (id, x, y, z, bstate)

Push a @code{KEY_MOUSE} event onto the input queue, associating the given
state data with it.
@end deffn

@geindex use_env() (in module curses)
@anchor{library/curses curses use_env}@anchor{263b}
@deffn {Function} curses.use_env (flag)

If used, this function should be called before @ref{2608,,initscr()} or newterm are
called.  When @emph{flag} is @code{False}, the values of lines and columns specified in the
terminfo database will be used, even if environment variables 
@geindex LINES
@geindex environment variable; LINES
@code{LINES}
and 
@geindex COLUMNS
@geindex environment variable; COLUMNS
@code{COLUMNS} (used by default) are set, or if curses is running in a
window (in which case default behavior would be to use the window size if
@geindex LINES
@geindex environment variable; LINES
@code{LINES} and 
@geindex COLUMNS
@geindex environment variable; COLUMNS
@code{COLUMNS} are not set).
@end deffn

@geindex use_default_colors() (in module curses)
@anchor{library/curses curses use_default_colors}@anchor{2616}
@deffn {Function} curses.use_default_colors ()

Allow use of default values for colors on terminals supporting this feature. Use
this to support transparency in your application.  The default color is assigned
to the color number @code{-1}. After calling this function,  @code{init_pair(x,
curses.COLOR_RED, -1)} initializes, for instance, color pair @emph{x} to a red
foreground color on the default background.
@end deffn

@geindex wrapper() (in module curses)
@anchor{library/curses curses wrapper}@anchor{598}
@deffn {Function} curses.wrapper (func, /, *args, **kwargs)

Initialize curses and call another callable object, @emph{func}, which should be the
rest of your curses-using application.  If the application raises an exception,
this function will restore the terminal to a sane state before re-raising the
exception and generating a traceback.  The callable object @emph{func} is then passed
the main window ‘stdscr’ as its first argument, followed by any other arguments
passed to @code{wrapper()}.  Before calling @emph{func}, @code{wrapper()} turns on
cbreak mode, turns off echo, enables the terminal keypad, and initializes colors
if the terminal has color support.  On exit (whether normally or by exception)
it restores cooked mode, turns on echo, and disables the terminal keypad.
@end deffn

@node Window Objects,Constants<6>,Functions<4>,curses — Terminal handling for character-cell displays
@anchor{library/curses curses-window-objects}@anchor{2617}@anchor{library/curses window-objects}@anchor{263c}
@subsubsection Window Objects


Window objects, as returned by @ref{2608,,initscr()} and @ref{2623,,newwin()} above, have
the following methods and attributes:

@geindex addch() (curses.window method)
@anchor{library/curses curses window addch}@anchor{263d}
@deffn {Method} window.addch (ch[, attr])

@deffnx {Method} window.addch (y, x, ch[, attr])

Paint character @emph{ch} at @code{(y, x)} with attributes @emph{attr}, overwriting any
character previously painted at that location.  By default, the character
position and attributes are the current settings for the window object.

@cartouche
@quotation Note 
Writing outside the window, subwindow, or pad raises a @ref{25f2,,curses.error}.
Attempting to write to the lower right corner of a window, subwindow,
or pad will cause an exception to be raised after the character is printed.
@end quotation
@end cartouche
@end deffn

@geindex addnstr() (curses.window method)
@anchor{library/curses curses window addnstr}@anchor{263e}
@deffn {Method} window.addnstr (str, n[, attr])

@deffnx {Method} window.addnstr (y, x, str, n[, attr])

Paint at most @emph{n} characters of the character string @emph{str} at
@code{(y, x)} with attributes
@emph{attr}, overwriting anything previously on the display.
@end deffn

@geindex addstr() (curses.window method)
@anchor{library/curses curses window addstr}@anchor{2602}
@deffn {Method} window.addstr (str[, attr])

@deffnx {Method} window.addstr (y, x, str[, attr])

Paint the character string @emph{str} at @code{(y, x)} with attributes
@emph{attr}, overwriting anything previously on the display.

@cartouche
@quotation Note 

@itemize *

@item 
Writing outside the window, subwindow, or pad raises @ref{25f2,,curses.error}.
Attempting to write to the lower right corner of a window, subwindow,
or pad will cause an exception to be raised after the string is printed.

@item 
A bug in ncurses@footnote{https://bugs.python.org/issue35924}, the backend
for this Python module, can cause SegFaults when resizing windows. This
is fixed in ncurses-6.1-20190511.  If you are stuck with an earlier
ncurses, you can avoid triggering this if you do not call @ref{2602,,addstr()}
with a @emph{str} that has embedded newlines.  Instead, call @ref{2602,,addstr()}
separately for each line.
@end itemize
@end quotation
@end cartouche
@end deffn

@geindex attroff() (curses.window method)
@anchor{library/curses curses window attroff}@anchor{263f}
@deffn {Method} window.attroff (attr)

Remove attribute @emph{attr} from the “background” set applied to all writes to the
current window.
@end deffn

@geindex attron() (curses.window method)
@anchor{library/curses curses window attron}@anchor{2640}
@deffn {Method} window.attron (attr)

Add attribute @emph{attr} from the “background” set applied to all writes to the
current window.
@end deffn

@geindex attrset() (curses.window method)
@anchor{library/curses curses window attrset}@anchor{2641}
@deffn {Method} window.attrset (attr)

Set the “background” set of attributes to @emph{attr}.  This set is initially
@code{0} (no attributes).
@end deffn

@geindex bkgd() (curses.window method)
@anchor{library/curses curses window bkgd}@anchor{2642}
@deffn {Method} window.bkgd (ch[, attr])

Set the background property of the window to the character @emph{ch}, with
attributes @emph{attr}.  The change is then applied to every character position in
that window:


@itemize *

@item 
The attribute of every character in the window  is changed to the new
background attribute.

@item 
Wherever  the  former background character appears, it is changed to the new
background character.
@end itemize
@end deffn

@geindex bkgdset() (curses.window method)
@anchor{library/curses curses window bkgdset}@anchor{2643}
@deffn {Method} window.bkgdset (ch[, attr])

Set the window’s background.  A window’s background consists of a character and
any combination of attributes.  The attribute part of the background is combined
(OR’ed) with all non-blank characters that are written into the window.  Both
the character and attribute parts of the background are combined with the blank
characters.  The background becomes a property of the character and moves with
the character through any scrolling and insert/delete line/character operations.
@end deffn

@geindex border() (curses.window method)
@anchor{library/curses curses window border}@anchor{2644}
@deffn {Method} window.border ([ls[, rs[, ts[, bs[, tl[, tr[, bl[, br]]]]]]]])

Draw a border around the edges of the window. Each parameter specifies  the
character to use for a specific part of the border; see the table below for more
details.

@cartouche
@quotation Note 
A @code{0} value for any parameter will cause the default character to be used for
that parameter.  Keyword parameters can @emph{not} be used.  The defaults are listed
in this table:
@end quotation
@end cartouche


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Parameter

@tab

Description

@tab

Default value

@item

@emph{ls}

@tab

Left side

@tab

@code{ACS_VLINE}

@item

@emph{rs}

@tab

Right side

@tab

@code{ACS_VLINE}

@item

@emph{ts}

@tab

Top

@tab

@code{ACS_HLINE}

@item

@emph{bs}

@tab

Bottom

@tab

@code{ACS_HLINE}

@item

@emph{tl}

@tab

Upper-left corner

@tab

@code{ACS_ULCORNER}

@item

@emph{tr}

@tab

Upper-right corner

@tab

@code{ACS_URCORNER}

@item

@emph{bl}

@tab

Bottom-left corner

@tab

@code{ACS_LLCORNER}

@item

@emph{br}

@tab

Bottom-right corner

@tab

@code{ACS_LRCORNER}

@end multitable

@end deffn

@geindex box() (curses.window method)
@anchor{library/curses curses window box}@anchor{2645}
@deffn {Method} window.box ([vertch, horch])

Similar to @ref{2644,,border()}, but both @emph{ls} and @emph{rs} are @emph{vertch} and both @emph{ts} and
@emph{bs} are @emph{horch}.  The default corner characters are always used by this function.
@end deffn

@geindex chgat() (curses.window method)
@anchor{library/curses curses window chgat}@anchor{2646}
@deffn {Method} window.chgat (attr)

@deffnx {Method} window.chgat (num, attr)

@deffnx {Method} window.chgat (y, x, attr)

@deffnx {Method} window.chgat (y, x, num, attr)

Set the attributes of @emph{num} characters at the current cursor position, or at
position @code{(y, x)} if supplied. If @emph{num} is not given or is @code{-1},
the attribute will be set on all the characters to the end of the line.  This
function moves cursor to position @code{(y, x)} if supplied. The changed line
will be touched using the @ref{2647,,touchline()} method so that the contents will
be redisplayed by the next window refresh.
@end deffn

@geindex clear() (curses.window method)
@anchor{library/curses curses window clear}@anchor{2648}
@deffn {Method} window.clear ()

Like @ref{2649,,erase()}, but also cause the whole window to be repainted upon next
call to @ref{2603,,refresh()}.
@end deffn

@geindex clearok() (curses.window method)
@anchor{library/curses curses window clearok}@anchor{264a}
@deffn {Method} window.clearok (flag)

If @emph{flag} is @code{True}, the next call to @ref{2603,,refresh()} will clear the window
completely.
@end deffn

@geindex clrtobot() (curses.window method)
@anchor{library/curses curses window clrtobot}@anchor{264b}
@deffn {Method} window.clrtobot ()

Erase from cursor to the end of the window: all lines below the cursor are
deleted, and then the equivalent of @ref{264c,,clrtoeol()} is performed.
@end deffn

@geindex clrtoeol() (curses.window method)
@anchor{library/curses curses window clrtoeol}@anchor{264c}
@deffn {Method} window.clrtoeol ()

Erase from cursor to the end of the line.
@end deffn

@geindex cursyncup() (curses.window method)
@anchor{library/curses curses window cursyncup}@anchor{264d}
@deffn {Method} window.cursyncup ()

Update the current cursor position of all the ancestors of the window to
reflect the current cursor position of the window.
@end deffn

@geindex delch() (curses.window method)
@anchor{library/curses curses window delch}@anchor{264e}
@deffn {Method} window.delch ([y, x])

Delete any character at @code{(y, x)}.
@end deffn

@geindex deleteln() (curses.window method)
@anchor{library/curses curses window deleteln}@anchor{264f}
@deffn {Method} window.deleteln ()

Delete the line under the cursor. All following lines are moved up by one line.
@end deffn

@geindex derwin() (curses.window method)
@anchor{library/curses curses window derwin}@anchor{2650}
@deffn {Method} window.derwin (begin_y, begin_x)

@deffnx {Method} window.derwin (nlines, ncols, begin_y, begin_x)

An abbreviation for “derive window”, @ref{2650,,derwin()} is the same as calling
@ref{2651,,subwin()}, except that @emph{begin_y} and @emph{begin_x} are relative to the origin
of the window, rather than relative to the entire screen.  Return a window
object for the derived window.
@end deffn

@geindex echochar() (curses.window method)
@anchor{library/curses curses window echochar}@anchor{2652}
@deffn {Method} window.echochar (ch[, attr])

Add character @emph{ch} with attribute @emph{attr}, and immediately  call @ref{2603,,refresh()}
on the window.
@end deffn

@geindex enclose() (curses.window method)
@anchor{library/curses curses window enclose}@anchor{1150}
@deffn {Method} window.enclose (y, x)

Test whether the given pair of screen-relative character-cell coordinates are
enclosed by the given window, returning @code{True} or @code{False}.  It is useful for
determining what subset of the screen windows enclose the location of a mouse
event.


Changed in version 3.10: Previously it returned @code{1} or @code{0} instead of @code{True} or @code{False}.

@end deffn

@geindex encoding (curses.window attribute)
@anchor{library/curses curses window encoding}@anchor{c2e}
@deffn {Attribute} window.encoding

Encoding used to encode method arguments (Unicode strings and characters).
The encoding attribute is inherited from the parent window when a subwindow
is created, for example with @ref{2651,,window.subwin()}. By default, the locale
encoding is used (see @ref{1b5,,locale.getpreferredencoding()}).


New in version 3.3.

@end deffn

@geindex erase() (curses.window method)
@anchor{library/curses curses window erase}@anchor{2649}
@deffn {Method} window.erase ()

Clear the window.
@end deffn

@geindex getbegyx() (curses.window method)
@anchor{library/curses curses window getbegyx}@anchor{2653}
@deffn {Method} window.getbegyx ()

Return a tuple @code{(y, x)} of co-ordinates of upper-left corner.
@end deffn

@geindex getbkgd() (curses.window method)
@anchor{library/curses curses window getbkgd}@anchor{2654}
@deffn {Method} window.getbkgd ()

Return the given window’s current background character/attribute pair.
@end deffn

@geindex getch() (curses.window method)
@anchor{library/curses curses window getch}@anchor{260c}
@deffn {Method} window.getch ([y, x])

Get a character. Note that the integer returned does @emph{not} have to be in ASCII
range: function keys, keypad keys and so on are represented by numbers higher
than 255.  In no-delay mode, return @code{-1} if there is no input, otherwise
wait until a key is pressed.
@end deffn

@geindex get_wch() (curses.window method)
@anchor{library/curses curses window get_wch}@anchor{c2f}
@deffn {Method} window.get_wch ([y, x])

Get a wide character. Return a character for most keys, or an integer for
function keys, keypad keys, and other special keys.
In no-delay mode, raise an exception if there is no input.


New in version 3.3.

@end deffn

@geindex getkey() (curses.window method)
@anchor{library/curses curses window getkey}@anchor{2655}
@deffn {Method} window.getkey ([y, x])

Get a character, returning a string instead of an integer, as @ref{260c,,getch()}
does. Function keys, keypad keys and other special keys return a multibyte
string containing the key name.  In no-delay mode, raise an exception if
there is no input.
@end deffn

@geindex getmaxyx() (curses.window method)
@anchor{library/curses curses window getmaxyx}@anchor{2656}
@deffn {Method} window.getmaxyx ()

Return a tuple @code{(y, x)} of the height and width of the window.
@end deffn

@geindex getparyx() (curses.window method)
@anchor{library/curses curses window getparyx}@anchor{2657}
@deffn {Method} window.getparyx ()

Return the beginning coordinates of this window relative to its parent window
as a tuple @code{(y, x)}.  Return @code{(-1, -1)} if this window has no
parent.
@end deffn

@geindex getstr() (curses.window method)
@anchor{library/curses curses window getstr}@anchor{2658}
@deffn {Method} window.getstr ()

@deffnx {Method} window.getstr (n)

@deffnx {Method} window.getstr (y, x)

@deffnx {Method} window.getstr (y, x, n)

Read a bytes object from the user, with primitive line editing capacity.
@end deffn

@geindex getyx() (curses.window method)
@anchor{library/curses curses window getyx}@anchor{2659}
@deffn {Method} window.getyx ()

Return a tuple @code{(y, x)} of current cursor position  relative to the window’s
upper-left corner.
@end deffn

@geindex hline() (curses.window method)
@anchor{library/curses curses window hline}@anchor{265a}
@deffn {Method} window.hline (ch, n)

@deffnx {Method} window.hline (y, x, ch, n)

Display a horizontal line starting at @code{(y, x)} with length @emph{n} consisting of
the character @emph{ch}.
@end deffn

@geindex idcok() (curses.window method)
@anchor{library/curses curses window idcok}@anchor{265b}
@deffn {Method} window.idcok (flag)

If @emph{flag} is @code{False}, curses no longer considers using the hardware insert/delete
character feature of the terminal; if @emph{flag} is @code{True}, use of character insertion
and deletion is enabled.  When curses is first initialized, use of character
insert/delete is enabled by default.
@end deffn

@geindex idlok() (curses.window method)
@anchor{library/curses curses window idlok}@anchor{265c}
@deffn {Method} window.idlok (flag)

If @emph{flag} is @code{True}, @ref{2a,,curses} will try and use hardware line
editing facilities. Otherwise, line insertion/deletion are disabled.
@end deffn

@geindex immedok() (curses.window method)
@anchor{library/curses curses window immedok}@anchor{265d}
@deffn {Method} window.immedok (flag)

If @emph{flag} is @code{True}, any change in the window image automatically causes the
window to be refreshed; you no longer have to call @ref{2603,,refresh()} yourself.
However, it may degrade performance considerably, due to repeated calls to
wrefresh.  This option is disabled by default.
@end deffn

@geindex inch() (curses.window method)
@anchor{library/curses curses window inch}@anchor{265e}
@deffn {Method} window.inch ([y, x])

Return the character at the given position in the window. The bottom 8 bits are
the character proper, and upper bits are the attributes.
@end deffn

@geindex insch() (curses.window method)
@anchor{library/curses curses window insch}@anchor{265f}
@deffn {Method} window.insch (ch[, attr])

@deffnx {Method} window.insch (y, x, ch[, attr])

Paint character @emph{ch} at @code{(y, x)} with attributes @emph{attr}, moving the line from
position @emph{x} right by one character.
@end deffn

@geindex insdelln() (curses.window method)
@anchor{library/curses curses window insdelln}@anchor{2660}
@deffn {Method} window.insdelln (nlines)

Insert @emph{nlines} lines into the specified window above the current line.  The
@emph{nlines} bottom lines are lost.  For negative @emph{nlines}, delete @emph{nlines} lines
starting with the one under the cursor, and move the remaining lines up.  The
bottom @emph{nlines} lines are cleared.  The current cursor position remains the
same.
@end deffn

@geindex insertln() (curses.window method)
@anchor{library/curses curses window insertln}@anchor{2661}
@deffn {Method} window.insertln ()

Insert a blank line under the cursor. All following lines are moved down by one
line.
@end deffn

@geindex insnstr() (curses.window method)
@anchor{library/curses curses window insnstr}@anchor{2662}
@deffn {Method} window.insnstr (str, n[, attr])

@deffnx {Method} window.insnstr (y, x, str, n[, attr])

Insert a character string (as many characters as will fit on the line) before
the character under the cursor, up to @emph{n} characters.   If @emph{n} is zero or
negative, the entire string is inserted. All characters to the right of the
cursor are shifted right, with the rightmost characters on the line being lost.
The cursor position does not change (after moving to @emph{y}, @emph{x}, if specified).
@end deffn

@geindex insstr() (curses.window method)
@anchor{library/curses curses window insstr}@anchor{2663}
@deffn {Method} window.insstr (str[, attr])

@deffnx {Method} window.insstr (y, x, str[, attr])

Insert a character string (as many characters as will fit on the line) before
the character under the cursor.  All characters to the right of the cursor are
shifted right, with the rightmost characters on the line being lost.  The cursor
position does not change (after moving to @emph{y}, @emph{x}, if specified).
@end deffn

@geindex instr() (curses.window method)
@anchor{library/curses curses window instr}@anchor{2664}
@deffn {Method} window.instr ([n])

@deffnx {Method} window.instr (y, x[, n])

Return a bytes object of characters, extracted from the window starting at the
current cursor position, or at @emph{y}, @emph{x} if specified. Attributes are stripped
from the characters.  If @emph{n} is specified, @ref{2664,,instr()} returns a string
at most @emph{n} characters long (exclusive of the trailing NUL).
@end deffn

@geindex is_linetouched() (curses.window method)
@anchor{library/curses curses window is_linetouched}@anchor{2665}
@deffn {Method} window.is_linetouched (line)

Return @code{True} if the specified line was modified since the last call to
@ref{2603,,refresh()}; otherwise return @code{False}.  Raise a @ref{25f2,,curses.error}
exception if @emph{line} is not valid for the given window.
@end deffn

@geindex is_wintouched() (curses.window method)
@anchor{library/curses curses window is_wintouched}@anchor{2666}
@deffn {Method} window.is_wintouched ()

Return @code{True} if the specified window was modified since the last call to
@ref{2603,,refresh()}; otherwise return @code{False}.
@end deffn

@geindex keypad() (curses.window method)
@anchor{library/curses curses window keypad}@anchor{2667}
@deffn {Method} window.keypad (flag)

If @emph{flag} is @code{True}, escape sequences generated by some keys (keypad,  function keys)
will be interpreted by @ref{2a,,curses}. If @emph{flag} is @code{False}, escape sequences will be
left as is in the input stream.
@end deffn

@geindex leaveok() (curses.window method)
@anchor{library/curses curses window leaveok}@anchor{260e}
@deffn {Method} window.leaveok (flag)

If @emph{flag} is @code{True}, cursor is left where it is on update, instead of being at “cursor
position.”  This reduces cursor movement where possible. If possible the cursor
will be made invisible.

If @emph{flag} is @code{False}, cursor will always be at “cursor position” after an update.
@end deffn

@geindex move() (curses.window method)
@anchor{library/curses curses window move}@anchor{2668}
@deffn {Method} window.move (new_y, new_x)

Move cursor to @code{(new_y, new_x)}.
@end deffn

@geindex mvderwin() (curses.window method)
@anchor{library/curses curses window mvderwin}@anchor{2669}
@deffn {Method} window.mvderwin (y, x)

Move the window inside its parent window.  The screen-relative parameters of
the window are not changed.  This routine is used to display different parts of
the parent window at the same physical position on the screen.
@end deffn

@geindex mvwin() (curses.window method)
@anchor{library/curses curses window mvwin}@anchor{266a}
@deffn {Method} window.mvwin (new_y, new_x)

Move the window so its upper-left corner is at @code{(new_y, new_x)}.
@end deffn

@geindex nodelay() (curses.window method)
@anchor{library/curses curses window nodelay}@anchor{266b}
@deffn {Method} window.nodelay (flag)

If @emph{flag} is @code{True}, @ref{260c,,getch()} will be non-blocking.
@end deffn

@geindex notimeout() (curses.window method)
@anchor{library/curses curses window notimeout}@anchor{266c}
@deffn {Method} window.notimeout (flag)

If @emph{flag} is @code{True}, escape sequences will not be timed out.

If @emph{flag} is @code{False}, after a few milliseconds, an escape sequence will not be
interpreted, and will be left in the input stream as is.
@end deffn

@geindex noutrefresh() (curses.window method)
@anchor{library/curses curses window noutrefresh}@anchor{2601}
@deffn {Method} window.noutrefresh ()

Mark for refresh but wait.  This function updates the data structure
representing the desired state of the window, but does not force an update of
the physical screen.  To accomplish that, call  @ref{2600,,doupdate()}.
@end deffn

@geindex overlay() (curses.window method)
@anchor{library/curses curses window overlay}@anchor{266d}
@deffn {Method} window.overlay (destwin[, sminrow, smincol, dminrow, dmincol, dmaxrow, dmaxcol])

Overlay the window on top of @emph{destwin}. The windows need not be the same size,
only the overlapping region is copied. This copy is non-destructive, which means
that the current background character does not overwrite the old contents of
@emph{destwin}.

To get fine-grained control over the copied region, the second form of
@ref{266d,,overlay()} can be used. @emph{sminrow} and @emph{smincol} are the upper-left
coordinates of the source window, and the other variables mark a rectangle in
the destination window.
@end deffn

@geindex overwrite() (curses.window method)
@anchor{library/curses curses window overwrite}@anchor{266e}
@deffn {Method} window.overwrite (destwin[, sminrow, smincol, dminrow, dmincol, dmaxrow, dmaxcol])

Overwrite the window on top of @emph{destwin}. The windows need not be the same size,
in which case only the overlapping region is copied. This copy is destructive,
which means that the current background character overwrites the old contents of
@emph{destwin}.

To get fine-grained control over the copied region, the second form of
@ref{266e,,overwrite()} can be used. @emph{sminrow} and @emph{smincol} are the upper-left
coordinates of the source window, the other variables mark a rectangle in the
destination window.
@end deffn

@geindex putwin() (curses.window method)
@anchor{library/curses curses window putwin}@anchor{266f}
@deffn {Method} window.putwin (file)

Write all data associated with the window into the provided file object.  This
information can be later retrieved using the @ref{260f,,getwin()} function.
@end deffn

@geindex redrawln() (curses.window method)
@anchor{library/curses curses window redrawln}@anchor{2670}
@deffn {Method} window.redrawln (beg, num)

Indicate that the @emph{num} screen lines, starting at line @emph{beg}, are corrupted and
should be completely redrawn on the next @ref{2603,,refresh()} call.
@end deffn

@geindex redrawwin() (curses.window method)
@anchor{library/curses curses window redrawwin}@anchor{2671}
@deffn {Method} window.redrawwin ()

Touch the entire window, causing it to be completely redrawn on the next
@ref{2603,,refresh()} call.
@end deffn

@geindex refresh() (curses.window method)
@anchor{library/curses curses window refresh}@anchor{2603}
@deffn {Method} window.refresh ([pminrow, pmincol, sminrow, smincol, smaxrow, smaxcol])

Update the display immediately (sync actual screen with previous
drawing/deleting methods).

The 6 optional arguments can only be specified when the window is a pad created
with @ref{2622,,newpad()}.  The additional parameters are needed to indicate what part
of the pad and screen are involved. @emph{pminrow} and @emph{pmincol} specify the upper
left-hand corner of the rectangle to be displayed in the pad.  @emph{sminrow},
@emph{smincol}, @emph{smaxrow}, and @emph{smaxcol} specify the edges of the rectangle to be
displayed on the screen.  The lower right-hand corner of the rectangle to be
displayed in the pad is calculated from the screen coordinates, since the
rectangles must be the same size.  Both rectangles must be entirely contained
within their respective structures.  Negative values of @emph{pminrow}, @emph{pmincol},
@emph{sminrow}, or @emph{smincol} are treated as if they were zero.
@end deffn

@geindex resize() (curses.window method)
@anchor{library/curses curses window resize}@anchor{2672}
@deffn {Method} window.resize (nlines, ncols)

Reallocate storage for a curses window to adjust its dimensions to the
specified values.  If either dimension is larger than the current values, the
window’s data is filled with blanks that have the current background
rendition (as set by @ref{2643,,bkgdset()}) merged into them.
@end deffn

@geindex scroll() (curses.window method)
@anchor{library/curses curses window scroll}@anchor{2673}
@deffn {Method} window.scroll ([lines=1])

Scroll the screen or scrolling region upward by @emph{lines} lines.
@end deffn

@geindex scrollok() (curses.window method)
@anchor{library/curses curses window scrollok}@anchor{2674}
@deffn {Method} window.scrollok (flag)

Control what happens when the cursor of a window is moved off the edge of the
window or scrolling region, either as a result of a newline action on the bottom
line, or typing the last character of the last line.  If @emph{flag} is @code{False}, the
cursor is left on the bottom line.  If @emph{flag} is @code{True}, the window is scrolled up
one line.  Note that in order to get the physical scrolling effect on the
terminal, it is also necessary to call @ref{265c,,idlok()}.
@end deffn

@geindex setscrreg() (curses.window method)
@anchor{library/curses curses window setscrreg}@anchor{2675}
@deffn {Method} window.setscrreg (top, bottom)

Set the scrolling region from line @emph{top} to line @emph{bottom}. All scrolling actions
will take place in this region.
@end deffn

@geindex standend() (curses.window method)
@anchor{library/curses curses window standend}@anchor{2676}
@deffn {Method} window.standend ()

Turn off the standout attribute.  On some terminals this has the side effect of
turning off all attributes.
@end deffn

@geindex standout() (curses.window method)
@anchor{library/curses curses window standout}@anchor{2677}
@deffn {Method} window.standout ()

Turn on attribute @emph{A_STANDOUT}.
@end deffn

@geindex subpad() (curses.window method)
@anchor{library/curses curses window subpad}@anchor{2678}
@deffn {Method} window.subpad (begin_y, begin_x)

@deffnx {Method} window.subpad (nlines, ncols, begin_y, begin_x)

Return a sub-window, whose upper-left corner is at @code{(begin_y, begin_x)}, and
whose width/height is @emph{ncols}/@emph{nlines}.
@end deffn

@geindex subwin() (curses.window method)
@anchor{library/curses curses window subwin}@anchor{2651}
@deffn {Method} window.subwin (begin_y, begin_x)

@deffnx {Method} window.subwin (nlines, ncols, begin_y, begin_x)

Return a sub-window, whose upper-left corner is at @code{(begin_y, begin_x)}, and
whose width/height is @emph{ncols}/@emph{nlines}.

By default, the sub-window will extend from the specified position to the lower
right corner of the window.
@end deffn

@geindex syncdown() (curses.window method)
@anchor{library/curses curses window syncdown}@anchor{2679}
@deffn {Method} window.syncdown ()

Touch each location in the window that has been touched in any of its ancestor
windows.  This routine is called by @ref{2603,,refresh()}, so it should almost never
be necessary to call it manually.
@end deffn

@geindex syncok() (curses.window method)
@anchor{library/curses curses window syncok}@anchor{267a}
@deffn {Method} window.syncok (flag)

If @emph{flag} is @code{True}, then @ref{267b,,syncup()} is called automatically
whenever there is a change in the window.
@end deffn

@geindex syncup() (curses.window method)
@anchor{library/curses curses window syncup}@anchor{267b}
@deffn {Method} window.syncup ()

Touch all locations in ancestors of the window that have been changed in  the
window.
@end deffn

@geindex timeout() (curses.window method)
@anchor{library/curses curses window timeout}@anchor{267c}
@deffn {Method} window.timeout (delay)

Set blocking or non-blocking read behavior for the window.  If @emph{delay} is
negative, blocking read is used (which will wait indefinitely for input).  If
@emph{delay} is zero, then non-blocking read is used, and @ref{260c,,getch()} will
return @code{-1} if no input is waiting.  If @emph{delay} is positive, then
@ref{260c,,getch()} will block for @emph{delay} milliseconds, and return @code{-1} if there is
still no input at the end of that time.
@end deffn

@geindex touchline() (curses.window method)
@anchor{library/curses curses window touchline}@anchor{2647}
@deffn {Method} window.touchline (start, count[, changed])

Pretend @emph{count} lines have been changed, starting with line @emph{start}.  If
@emph{changed} is supplied, it specifies whether the affected lines are marked as
having been changed (@emph{changed}@code{=True}) or unchanged (@emph{changed}@code{=False}).
@end deffn

@geindex touchwin() (curses.window method)
@anchor{library/curses curses window touchwin}@anchor{267d}
@deffn {Method} window.touchwin ()

Pretend the whole window has been changed, for purposes of drawing
optimizations.
@end deffn

@geindex untouchwin() (curses.window method)
@anchor{library/curses curses window untouchwin}@anchor{267e}
@deffn {Method} window.untouchwin ()

Mark all lines in  the  window  as unchanged since the last call to
@ref{2603,,refresh()}.
@end deffn

@geindex vline() (curses.window method)
@anchor{library/curses curses window vline}@anchor{267f}
@deffn {Method} window.vline (ch, n)

@deffnx {Method} window.vline (y, x, ch, n)

Display a vertical line starting at @code{(y, x)} with length @emph{n} consisting of the
character @emph{ch}.
@end deffn

@node Constants<6>,,Window Objects,curses — Terminal handling for character-cell displays
@anchor{library/curses constants}@anchor{2680}
@subsubsection Constants


The @ref{2a,,curses} module defines the following data members:

@geindex ERR (in module curses)
@anchor{library/curses curses ERR}@anchor{2681}
@deffn {Data} curses.ERR

Some curses routines  that  return  an integer, such as @ref{260c,,getch()}, return
@ref{2681,,ERR} upon failure.
@end deffn

@geindex OK (in module curses)
@anchor{library/curses curses OK}@anchor{2682}
@deffn {Data} curses.OK

Some curses routines  that  return  an integer, such as  @ref{2621,,napms()}, return
@ref{2682,,OK} upon success.
@end deffn

@geindex version (in module curses)
@anchor{library/curses curses version}@anchor{2683}
@deffn {Data} curses.version

A bytes object representing the current version of the module.  Also available as
@code{__version__}.
@end deffn

@geindex ncurses_version (in module curses)
@anchor{library/curses curses ncurses_version}@anchor{4eb}
@deffn {Data} curses.ncurses_version

A named tuple containing the three components of the ncurses library
version: @emph{major}, @emph{minor}, and @emph{patch}.  All values are integers.  The
components can also be accessed by name,  so @code{curses.ncurses_version[0]}
is equivalent to @code{curses.ncurses_version.major} and so on.

Availability: if the ncurses library is used.


New in version 3.8.

@end deffn

Some constants are available to specify character cell attributes.
The exact constants available are system dependent.


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Attribute

@tab

Meaning

@item

@code{A_ALTCHARSET}

@tab

Alternate character set mode

@item

@code{A_BLINK}

@tab

Blink mode

@item

@code{A_BOLD}

@tab

Bold mode

@item

@code{A_DIM}

@tab

Dim mode

@item

@code{A_INVIS}

@tab

Invisible or blank mode

@item

@code{A_ITALIC}

@tab

Italic mode

@item

@code{A_NORMAL}

@tab

Normal attribute

@item

@code{A_PROTECT}

@tab

Protected mode

@item

@code{A_REVERSE}

@tab

Reverse background and
foreground colors

@item

@code{A_STANDOUT}

@tab

Standout mode

@item

@code{A_UNDERLINE}

@tab

Underline mode

@item

@code{A_HORIZONTAL}

@tab

Horizontal highlight

@item

@code{A_LEFT}

@tab

Left highlight

@item

@code{A_LOW}

@tab

Low highlight

@item

@code{A_RIGHT}

@tab

Right highlight

@item

@code{A_TOP}

@tab

Top highlight

@item

@code{A_VERTICAL}

@tab

Vertical highlight

@item

@code{A_CHARTEXT}

@tab

Bit-mask to extract a
character

@end multitable



New in version 3.7: @code{A_ITALIC} was added.


Several constants are available to extract corresponding attributes returned
by some methods.


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Bit-mask

@tab

Meaning

@item

@code{A_ATTRIBUTES}

@tab

Bit-mask to extract
attributes

@item

@code{A_CHARTEXT}

@tab

Bit-mask to extract a
character

@item

@code{A_COLOR}

@tab

Bit-mask to extract
color-pair field information

@end multitable


Keys are referred to by integer constants with names starting with  @code{KEY_}.
The exact keycaps available are system dependent.

@c XXX this table is far too large! should it be alphabetized?


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Key constant

@tab

Key

@item

@code{KEY_MIN}

@tab

Minimum key value

@item

@code{KEY_BREAK}

@tab

Break key (unreliable)

@item

@code{KEY_DOWN}

@tab

Down-arrow

@item

@code{KEY_UP}

@tab

Up-arrow

@item

@code{KEY_LEFT}

@tab

Left-arrow

@item

@code{KEY_RIGHT}

@tab

Right-arrow

@item

@code{KEY_HOME}

@tab

Home key (upward+left arrow)

@item

@code{KEY_BACKSPACE}

@tab

Backspace (unreliable)

@item

@code{KEY_F0}

@tab

Function keys.  Up to 64 function keys are
supported.

@item

@code{KEY_Fn}

@tab

Value of function key @emph{n}

@item

@code{KEY_DL}

@tab

Delete line

@item

@code{KEY_IL}

@tab

Insert line

@item

@code{KEY_DC}

@tab

Delete character

@item

@code{KEY_IC}

@tab

Insert char or enter insert mode

@item

@code{KEY_EIC}

@tab

Exit insert char mode

@item

@code{KEY_CLEAR}

@tab

Clear screen

@item

@code{KEY_EOS}

@tab

Clear to end of screen

@item

@code{KEY_EOL}

@tab

Clear to end of line

@item

@code{KEY_SF}

@tab

Scroll 1 line forward

@item

@code{KEY_SR}

@tab

Scroll 1 line backward (reverse)

@item

@code{KEY_NPAGE}

@tab

Next page

@item

@code{KEY_PPAGE}

@tab

Previous page

@item

@code{KEY_STAB}

@tab

Set tab

@item

@code{KEY_CTAB}

@tab

Clear tab

@item

@code{KEY_CATAB}

@tab

Clear all tabs

@item

@code{KEY_ENTER}

@tab

Enter or send (unreliable)

@item

@code{KEY_SRESET}

@tab

Soft (partial) reset (unreliable)

@item

@code{KEY_RESET}

@tab

Reset or hard reset (unreliable)

@item

@code{KEY_PRINT}

@tab

Print

@item

@code{KEY_LL}

@tab

Home down or bottom (lower left)

@item

@code{KEY_A1}

@tab

Upper left of keypad

@item

@code{KEY_A3}

@tab

Upper right of keypad

@item

@code{KEY_B2}

@tab

Center of keypad

@item

@code{KEY_C1}

@tab

Lower left of keypad

@item

@code{KEY_C3}

@tab

Lower right of keypad

@item

@code{KEY_BTAB}

@tab

Back tab

@item

@code{KEY_BEG}

@tab

Beg (beginning)

@item

@code{KEY_CANCEL}

@tab

Cancel

@item

@code{KEY_CLOSE}

@tab

Close

@item

@code{KEY_COMMAND}

@tab

Cmd (command)

@item

@code{KEY_COPY}

@tab

Copy

@item

@code{KEY_CREATE}

@tab

Create

@item

@code{KEY_END}

@tab

End

@item

@code{KEY_EXIT}

@tab

Exit

@item

@code{KEY_FIND}

@tab

Find

@item

@code{KEY_HELP}

@tab

Help

@item

@code{KEY_MARK}

@tab

Mark

@item

@code{KEY_MESSAGE}

@tab

Message

@item

@code{KEY_MOVE}

@tab

Move

@item

@code{KEY_NEXT}

@tab

Next

@item

@code{KEY_OPEN}

@tab

Open

@item

@code{KEY_OPTIONS}

@tab

Options

@item

@code{KEY_PREVIOUS}

@tab

Prev (previous)

@item

@code{KEY_REDO}

@tab

Redo

@item

@code{KEY_REFERENCE}

@tab

Ref (reference)

@item

@code{KEY_REFRESH}

@tab

Refresh

@item

@code{KEY_REPLACE}

@tab

Replace

@item

@code{KEY_RESTART}

@tab

Restart

@item

@code{KEY_RESUME}

@tab

Resume

@item

@code{KEY_SAVE}

@tab

Save

@item

@code{KEY_SBEG}

@tab

Shifted Beg (beginning)

@item

@code{KEY_SCANCEL}

@tab

Shifted Cancel

@item

@code{KEY_SCOMMAND}

@tab

Shifted Command

@item

@code{KEY_SCOPY}

@tab

Shifted Copy

@item

@code{KEY_SCREATE}

@tab

Shifted Create

@item

@code{KEY_SDC}

@tab

Shifted Delete char

@item

@code{KEY_SDL}

@tab

Shifted Delete line

@item

@code{KEY_SELECT}

@tab

Select

@item

@code{KEY_SEND}

@tab

Shifted End

@item

@code{KEY_SEOL}

@tab

Shifted Clear line

@item

@code{KEY_SEXIT}

@tab

Shifted Exit

@item

@code{KEY_SFIND}

@tab

Shifted Find

@item

@code{KEY_SHELP}

@tab

Shifted Help

@item

@code{KEY_SHOME}

@tab

Shifted Home

@item

@code{KEY_SIC}

@tab

Shifted Input

@item

@code{KEY_SLEFT}

@tab

Shifted Left arrow

@item

@code{KEY_SMESSAGE}

@tab

Shifted Message

@item

@code{KEY_SMOVE}

@tab

Shifted Move

@item

@code{KEY_SNEXT}

@tab

Shifted Next

@item

@code{KEY_SOPTIONS}

@tab

Shifted Options

@item

@code{KEY_SPREVIOUS}

@tab

Shifted Prev

@item

@code{KEY_SPRINT}

@tab

Shifted Print

@item

@code{KEY_SREDO}

@tab

Shifted Redo

@item

@code{KEY_SREPLACE}

@tab

Shifted Replace

@item

@code{KEY_SRIGHT}

@tab

Shifted Right arrow

@item

@code{KEY_SRSUME}

@tab

Shifted Resume

@item

@code{KEY_SSAVE}

@tab

Shifted Save

@item

@code{KEY_SSUSPEND}

@tab

Shifted Suspend

@item

@code{KEY_SUNDO}

@tab

Shifted Undo

@item

@code{KEY_SUSPEND}

@tab

Suspend

@item

@code{KEY_UNDO}

@tab

Undo

@item

@code{KEY_MOUSE}

@tab

Mouse event has occurred

@item

@code{KEY_RESIZE}

@tab

Terminal resize event

@item

@code{KEY_MAX}

@tab

Maximum key value

@end multitable


On VT100s and their software emulations, such as X terminal emulators, there are
normally at least four function keys (@code{KEY_F1}, @code{KEY_F2},
@code{KEY_F3}, @code{KEY_F4}) available, and the arrow keys mapped to
@code{KEY_UP}, @code{KEY_DOWN}, @code{KEY_LEFT} and @code{KEY_RIGHT} in
the obvious way.  If your machine has a PC keyboard, it is safe to expect arrow
keys and twelve function keys (older PC keyboards may have only ten function
keys); also, the following keypad mappings are standard:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxx} 
@headitem

Keycap

@tab

Constant

@item

@code{Insert}

@tab

KEY_IC

@item

@code{Delete}

@tab

KEY_DC

@item

@code{Home}

@tab

KEY_HOME

@item

@code{End}

@tab

KEY_END

@item

@code{Page Up}

@tab

KEY_PPAGE

@item

@code{Page Down}

@tab

KEY_NPAGE

@end multitable


The following table lists characters from the alternate character set. These are
inherited from the VT100 terminal, and will generally be  available on software
emulations such as X terminals.  When there is no graphic available, curses
falls back on a crude printable ASCII approximation.

@cartouche
@quotation Note 
These are available only after @ref{2608,,initscr()} has  been called.
@end quotation
@end cartouche


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

ACS code

@tab

Meaning

@item

@code{ACS_BBSS}

@tab

alternate name for upper right corner

@item

@code{ACS_BLOCK}

@tab

solid square block

@item

@code{ACS_BOARD}

@tab

board of squares

@item

@code{ACS_BSBS}

@tab

alternate name for horizontal line

@item

@code{ACS_BSSB}

@tab

alternate name for upper left corner

@item

@code{ACS_BSSS}

@tab

alternate name for top tee

@item

@code{ACS_BTEE}

@tab

bottom tee

@item

@code{ACS_BULLET}

@tab

bullet

@item

@code{ACS_CKBOARD}

@tab

checker board (stipple)

@item

@code{ACS_DARROW}

@tab

arrow pointing down

@item

@code{ACS_DEGREE}

@tab

degree symbol

@item

@code{ACS_DIAMOND}

@tab

diamond

@item

@code{ACS_GEQUAL}

@tab

greater-than-or-equal-to

@item

@code{ACS_HLINE}

@tab

horizontal line

@item

@code{ACS_LANTERN}

@tab

lantern symbol

@item

@code{ACS_LARROW}

@tab

left arrow

@item

@code{ACS_LEQUAL}

@tab

less-than-or-equal-to

@item

@code{ACS_LLCORNER}

@tab

lower left-hand corner

@item

@code{ACS_LRCORNER}

@tab

lower right-hand corner

@item

@code{ACS_LTEE}

@tab

left tee

@item

@code{ACS_NEQUAL}

@tab

not-equal sign

@item

@code{ACS_PI}

@tab

letter pi

@item

@code{ACS_PLMINUS}

@tab

plus-or-minus sign

@item

@code{ACS_PLUS}

@tab

big plus sign

@item

@code{ACS_RARROW}

@tab

right arrow

@item

@code{ACS_RTEE}

@tab

right tee

@item

@code{ACS_S1}

@tab

scan line 1

@item

@code{ACS_S3}

@tab

scan line 3

@item

@code{ACS_S7}

@tab

scan line 7

@item

@code{ACS_S9}

@tab

scan line 9

@item

@code{ACS_SBBS}

@tab

alternate name for lower right corner

@item

@code{ACS_SBSB}

@tab

alternate name for vertical line

@item

@code{ACS_SBSS}

@tab

alternate name for right tee

@item

@code{ACS_SSBB}

@tab

alternate name for lower left corner

@item

@code{ACS_SSBS}

@tab

alternate name for bottom tee

@item

@code{ACS_SSSB}

@tab

alternate name for left tee

@item

@code{ACS_SSSS}

@tab

alternate name for crossover or big plus

@item

@code{ACS_STERLING}

@tab

pound sterling

@item

@code{ACS_TTEE}

@tab

top tee

@item

@code{ACS_UARROW}

@tab

up arrow

@item

@code{ACS_ULCORNER}

@tab

upper left corner

@item

@code{ACS_URCORNER}

@tab

upper right corner

@item

@code{ACS_VLINE}

@tab

vertical line

@end multitable


The following table lists the predefined colors:


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Color

@item

@code{COLOR_BLACK}

@tab

Black

@item

@code{COLOR_BLUE}

@tab

Blue

@item

@code{COLOR_CYAN}

@tab

Cyan (light greenish blue)

@item

@code{COLOR_GREEN}

@tab

Green

@item

@code{COLOR_MAGENTA}

@tab

Magenta (purplish red)

@item

@code{COLOR_RED}

@tab

Red

@item

@code{COLOR_WHITE}

@tab

White

@item

@code{COLOR_YELLOW}

@tab

Yellow

@end multitable


@node curses textpad — Text input widget for curses programs,curses ascii — Utilities for ASCII characters,curses — Terminal handling for character-cell displays,Generic Operating System Services
@anchor{library/curses curses-textpad-text-input-widget-for-curses-programs}@anchor{2684}@anchor{library/curses module-curses textpad}@anchor{2d}
@subsection @code{curses.textpad} — Text input widget for curses programs


@geindex module; curses.textpad

The @ref{2d,,curses.textpad} module provides a @ref{2685,,Textbox} class that handles
elementary text editing in a curses window, supporting a set of keybindings
resembling those of Emacs (thus, also of Netscape Navigator, BBedit 6.x,
FrameMaker, and many other programs).  The module also provides a
rectangle-drawing function useful for framing text boxes or for other purposes.

The module @ref{2d,,curses.textpad} defines the following function:

@geindex rectangle() (in module curses.textpad)
@anchor{library/curses curses textpad rectangle}@anchor{2686}
@deffn {Function} curses.textpad.rectangle (win, uly, ulx, lry, lrx)

Draw a rectangle.  The first argument must be a window object; the remaining
arguments are coordinates relative to that window.  The second and third
arguments are the y and x coordinates of the upper left hand corner of the
rectangle to be drawn; the fourth and fifth arguments are the y and x
coordinates of the lower right hand corner. The rectangle will be drawn using
VT100/IBM PC forms characters on terminals that make this possible (including
xterm and most other software terminal emulators).  Otherwise it will be drawn
with ASCII  dashes, vertical bars, and plus signs.
@end deffn

@menu
* Textbox objects:: 

@end menu

@node Textbox objects,,,curses textpad — Text input widget for curses programs
@anchor{library/curses curses-textpad-objects}@anchor{2687}@anchor{library/curses textbox-objects}@anchor{2688}
@subsubsection Textbox objects


You can instantiate a @ref{2685,,Textbox} object as follows:

@geindex Textbox (class in curses.textpad)
@anchor{library/curses curses textpad Textbox}@anchor{2685}
@deffn {Class} curses.textpad.Textbox (win)

Return a textbox widget object.  The @emph{win} argument should be a curses
@ref{2617,,window} object in which the textbox is to
be contained. The edit cursor of the textbox is initially located at the
upper left hand corner of the containing window, with coordinates @code{(0, 0)}.
The instance’s @ref{2689,,stripspaces} flag is initially on.

@ref{2685,,Textbox} objects have the following methods:

@geindex edit() (curses.textpad.Textbox method)
@anchor{library/curses curses textpad Textbox edit}@anchor{268a}
@deffn {Method} edit ([validator])

This is the entry point you will normally use.  It accepts editing
keystrokes until one of the termination keystrokes is entered.  If
@emph{validator} is supplied, it must be a function.  It will be called for
each keystroke entered with the keystroke as a parameter; command dispatch
is done on the result. This method returns the window contents as a
string; whether blanks in the window are included is affected by the
@ref{2689,,stripspaces} attribute.
@end deffn

@geindex do_command() (curses.textpad.Textbox method)
@anchor{library/curses curses textpad Textbox do_command}@anchor{268b}
@deffn {Method} do_command (ch)

Process a single command keystroke.  Here are the supported special
keystrokes:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Keystroke

@tab

Action

@item

@code{Control-A}

@tab

Go to left edge of window.

@item

@code{Control-B}

@tab

Cursor left, wrapping to previous line if
appropriate.

@item

@code{Control-D}

@tab

Delete character under cursor.

@item

@code{Control-E}

@tab

Go to right edge (stripspaces off) or end
of line (stripspaces on).

@item

@code{Control-F}

@tab

Cursor right, wrapping to next line when
appropriate.

@item

@code{Control-G}

@tab

Terminate, returning the window contents.

@item

@code{Control-H}

@tab

Delete character backward.

@item

@code{Control-J}

@tab

Terminate if the window is 1 line,
otherwise insert newline.

@item

@code{Control-K}

@tab

If line is blank, delete it, otherwise
clear to end of line.

@item

@code{Control-L}

@tab

Refresh screen.

@item

@code{Control-N}

@tab

Cursor down; move down one line.

@item

@code{Control-O}

@tab

Insert a blank line at cursor location.

@item

@code{Control-P}

@tab

Cursor up; move up one line.

@end multitable


Move operations do nothing if the cursor is at an edge where the movement
is not possible.  The following synonyms are supported where possible:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Keystroke

@item

@code{KEY_LEFT}

@tab

@code{Control-B}

@item

@code{KEY_RIGHT}

@tab

@code{Control-F}

@item

@code{KEY_UP}

@tab

@code{Control-P}

@item

@code{KEY_DOWN}

@tab

@code{Control-N}

@item

@code{KEY_BACKSPACE}

@tab

@code{Control-h}

@end multitable


All other keystrokes are treated as a command to insert the given
character and move right (with line wrapping).
@end deffn

@geindex gather() (curses.textpad.Textbox method)
@anchor{library/curses curses textpad Textbox gather}@anchor{268c}
@deffn {Method} gather ()

Return the window contents as a string; whether blanks in the
window are included is affected by the @ref{2689,,stripspaces} member.
@end deffn

@geindex stripspaces (curses.textpad.Textbox attribute)
@anchor{library/curses curses textpad Textbox stripspaces}@anchor{2689}
@deffn {Attribute} stripspaces

This attribute is a flag which controls the interpretation of blanks in
the window.  When it is on, trailing blanks on each line are ignored; any
cursor motion that would land the cursor on a trailing blank goes to the
end of that line instead, and trailing blanks are stripped when the window
contents are gathered.
@end deffn
@end deffn

@node curses ascii — Utilities for ASCII characters,curses panel — A panel stack extension for curses,curses textpad — Text input widget for curses programs,Generic Operating System Services
@anchor{library/curses ascii doc}@anchor{268d}@anchor{library/curses ascii curses-ascii-utilities-for-ascii-characters}@anchor{268e}@anchor{library/curses ascii module-curses ascii}@anchor{2b}
@subsection @code{curses.ascii} — Utilities for ASCII characters


@geindex module; curses.ascii


__________________________________________________________________


The @ref{2b,,curses.ascii} module supplies name constants for ASCII characters and
functions to test membership in various ASCII character classes.  The constants
supplied are names for control characters as follows:


@multitable {xxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Name

@tab

Meaning

@item

@code{NUL}

@tab

@item

@code{SOH}

@tab

Start of heading, console interrupt

@item

@code{STX}

@tab

Start of text

@item

@code{ETX}

@tab

End of text

@item

@code{EOT}

@tab

End of transmission

@item

@code{ENQ}

@tab

Enquiry, goes with @code{ACK} flow control

@item

@code{ACK}

@tab

Acknowledgement

@item

@code{BEL}

@tab

Bell

@item

@code{BS}

@tab

Backspace

@item

@code{TAB}

@tab

Tab

@item

@code{HT}

@tab

Alias for @code{TAB}: “Horizontal tab”

@item

@code{LF}

@tab

Line feed

@item

@code{NL}

@tab

Alias for @code{LF}: “New line”

@item

@code{VT}

@tab

Vertical tab

@item

@code{FF}

@tab

Form feed

@item

@code{CR}

@tab

Carriage return

@item

@code{SO}

@tab

Shift-out, begin alternate character set

@item

@code{SI}

@tab

Shift-in, resume default character set

@item

@code{DLE}

@tab

Data-link escape

@item

@code{DC1}

@tab

XON, for flow control

@item

@code{DC2}

@tab

Device control 2, block-mode flow control

@item

@code{DC3}

@tab

XOFF, for flow control

@item

@code{DC4}

@tab

Device control 4

@item

@code{NAK}

@tab

Negative acknowledgement

@item

@code{SYN}

@tab

Synchronous idle

@item

@code{ETB}

@tab

End transmission block

@item

@code{CAN}

@tab

Cancel

@item

@code{EM}

@tab

End of medium

@item

@code{SUB}

@tab

Substitute

@item

@code{ESC}

@tab

Escape

@item

@code{FS}

@tab

File separator

@item

@code{GS}

@tab

Group separator

@item

@code{RS}

@tab

Record separator, block-mode terminator

@item

@code{US}

@tab

Unit separator

@item

@code{SP}

@tab

Space

@item

@code{DEL}

@tab

Delete

@end multitable


Note that many of these have little practical significance in modern usage.  The
mnemonics derive from teleprinter conventions that predate digital computers.

The module supplies the following functions, patterned on those in the standard
C library:

@geindex isalnum() (in module curses.ascii)
@anchor{library/curses ascii curses ascii isalnum}@anchor{268f}
@deffn {Function} curses.ascii.isalnum (c)

Checks for an ASCII alphanumeric character; it is equivalent to @code{isalpha(c) or
isdigit(c)}.
@end deffn

@geindex isalpha() (in module curses.ascii)
@anchor{library/curses ascii curses ascii isalpha}@anchor{2690}
@deffn {Function} curses.ascii.isalpha (c)

Checks for an ASCII alphabetic character; it is equivalent to @code{isupper(c) or
islower(c)}.
@end deffn

@geindex isascii() (in module curses.ascii)
@anchor{library/curses ascii curses ascii isascii}@anchor{2691}
@deffn {Function} curses.ascii.isascii (c)

Checks for a character value that fits in the 7-bit ASCII set.
@end deffn

@geindex isblank() (in module curses.ascii)
@anchor{library/curses ascii curses ascii isblank}@anchor{2692}
@deffn {Function} curses.ascii.isblank (c)

Checks for an ASCII whitespace character; space or horizontal tab.
@end deffn

@geindex iscntrl() (in module curses.ascii)
@anchor{library/curses ascii curses ascii iscntrl}@anchor{2693}
@deffn {Function} curses.ascii.iscntrl (c)

Checks for an ASCII control character (in the range 0x00 to 0x1f or 0x7f).
@end deffn

@geindex isdigit() (in module curses.ascii)
@anchor{library/curses ascii curses ascii isdigit}@anchor{2694}
@deffn {Function} curses.ascii.isdigit (c)

Checks for an ASCII decimal digit, @code{'0'} through @code{'9'}.  This is equivalent
to @code{c in string.digits}.
@end deffn

@geindex isgraph() (in module curses.ascii)
@anchor{library/curses ascii curses ascii isgraph}@anchor{2695}
@deffn {Function} curses.ascii.isgraph (c)

Checks for ASCII any printable character except space.
@end deffn

@geindex islower() (in module curses.ascii)
@anchor{library/curses ascii curses ascii islower}@anchor{2696}
@deffn {Function} curses.ascii.islower (c)

Checks for an ASCII lower-case character.
@end deffn

@geindex isprint() (in module curses.ascii)
@anchor{library/curses ascii curses ascii isprint}@anchor{2697}
@deffn {Function} curses.ascii.isprint (c)

Checks for any ASCII printable character including space.
@end deffn

@geindex ispunct() (in module curses.ascii)
@anchor{library/curses ascii curses ascii ispunct}@anchor{2698}
@deffn {Function} curses.ascii.ispunct (c)

Checks for any printable ASCII character which is not a space or an alphanumeric
character.
@end deffn

@geindex isspace() (in module curses.ascii)
@anchor{library/curses ascii curses ascii isspace}@anchor{2699}
@deffn {Function} curses.ascii.isspace (c)

Checks for ASCII white-space characters; space, line feed, carriage return, form
feed, horizontal tab, vertical tab.
@end deffn

@geindex isupper() (in module curses.ascii)
@anchor{library/curses ascii curses ascii isupper}@anchor{269a}
@deffn {Function} curses.ascii.isupper (c)

Checks for an ASCII uppercase letter.
@end deffn

@geindex isxdigit() (in module curses.ascii)
@anchor{library/curses ascii curses ascii isxdigit}@anchor{269b}
@deffn {Function} curses.ascii.isxdigit (c)

Checks for an ASCII hexadecimal digit.  This is equivalent to @code{c in
string.hexdigits}.
@end deffn

@geindex isctrl() (in module curses.ascii)
@anchor{library/curses ascii curses ascii isctrl}@anchor{269c}
@deffn {Function} curses.ascii.isctrl (c)

Checks for an ASCII control character (ordinal values 0 to 31).
@end deffn

@geindex ismeta() (in module curses.ascii)
@anchor{library/curses ascii curses ascii ismeta}@anchor{269d}
@deffn {Function} curses.ascii.ismeta (c)

Checks for a non-ASCII character (ordinal values 0x80 and above).
@end deffn

These functions accept either integers or single-character strings; when the argument is a
string, it is first converted using the built-in function @ref{178e,,ord()}.

Note that all these functions check ordinal bit values derived from the
character of the string you pass in; they do not actually know anything about
the host machine’s character encoding.

The following two functions take either a single-character string or integer
byte value; they return a value of the same type.

@geindex ascii() (in module curses.ascii)
@anchor{library/curses ascii curses ascii ascii}@anchor{269e}
@deffn {Function} curses.ascii.ascii (c)

Return the ASCII value corresponding to the low 7 bits of @emph{c}.
@end deffn

@geindex ctrl() (in module curses.ascii)
@anchor{library/curses ascii curses ascii ctrl}@anchor{269f}
@deffn {Function} curses.ascii.ctrl (c)

Return the control character corresponding to the given character (the character
bit value is bitwise-anded with 0x1f).
@end deffn

@geindex alt() (in module curses.ascii)
@anchor{library/curses ascii curses ascii alt}@anchor{26a0}
@deffn {Function} curses.ascii.alt (c)

Return the 8-bit character corresponding to the given ASCII character (the
character bit value is bitwise-ored with 0x80).
@end deffn

The following function takes either a single-character string or integer value;
it returns a string.

@geindex ^ (caret); in curses module
@geindex ! (exclamation); in curses module

@geindex unctrl() (in module curses.ascii)
@anchor{library/curses ascii curses ascii unctrl}@anchor{26a1}
@deffn {Function} curses.ascii.unctrl (c)

Return a string representation of the ASCII character @emph{c}.  If @emph{c} is printable,
this string is the character itself.  If the character is a control character
(0x00–0x1f) the string consists of a caret (@code{'^'}) followed by the
corresponding uppercase letter. If the character is an ASCII delete (0x7f) the
string is @code{'^?'}.  If the character has its meta bit (0x80) set, the meta bit
is stripped, the preceding rules applied, and @code{'!'} prepended to the result.
@end deffn

@geindex controlnames (in module curses.ascii)
@anchor{library/curses ascii curses ascii controlnames}@anchor{26a2}
@deffn {Data} curses.ascii.controlnames

A 33-element string array that contains the ASCII mnemonics for the thirty-two
ASCII control characters from 0 (NUL) to 0x1f (US), in order, plus the mnemonic
@code{SP} for the space character.
@end deffn

@node curses panel — A panel stack extension for curses,platform — Access to underlying platform’s identifying data,curses ascii — Utilities for ASCII characters,Generic Operating System Services
@anchor{library/curses panel doc}@anchor{26a3}@anchor{library/curses panel curses-panel-a-panel-stack-extension-for-curses}@anchor{26a4}@anchor{library/curses panel module-curses panel}@anchor{2c}
@subsection @code{curses.panel} — A panel stack extension for curses


@geindex module; curses.panel


__________________________________________________________________


Panels are windows with the added feature of depth, so they can be stacked on
top of each other, and only the visible portions of each window will be
displayed.  Panels can be added, moved up or down in the stack, and removed.

@menu
* Functions: Functions<5>. 
* Panel Objects:: 

@end menu

@node Functions<5>,Panel Objects,,curses panel — A panel stack extension for curses
@anchor{library/curses panel cursespanel-functions}@anchor{26a5}@anchor{library/curses panel functions}@anchor{26a6}
@subsubsection Functions


The module @ref{2c,,curses.panel} defines the following functions:

@geindex bottom_panel() (in module curses.panel)
@anchor{library/curses panel curses panel bottom_panel}@anchor{26a7}
@deffn {Function} curses.panel.bottom_panel ()

Returns the bottom panel in the panel stack.
@end deffn

@geindex new_panel() (in module curses.panel)
@anchor{library/curses panel curses panel new_panel}@anchor{26a8}
@deffn {Function} curses.panel.new_panel (win)

Returns a panel object, associating it with the given window @emph{win}. Be aware
that you need to keep the returned panel object referenced explicitly.  If you
don’t, the panel object is garbage collected and removed from the panel stack.
@end deffn

@geindex top_panel() (in module curses.panel)
@anchor{library/curses panel curses panel top_panel}@anchor{26a9}
@deffn {Function} curses.panel.top_panel ()

Returns the top panel in the panel stack.
@end deffn

@geindex update_panels() (in module curses.panel)
@anchor{library/curses panel curses panel update_panels}@anchor{26aa}
@deffn {Function} curses.panel.update_panels ()

Updates the virtual screen after changes in the panel stack. This does not call
@ref{2600,,curses.doupdate()}, so you’ll have to do this yourself.
@end deffn

@node Panel Objects,,Functions<5>,curses panel — A panel stack extension for curses
@anchor{library/curses panel curses-panel-objects}@anchor{26ab}@anchor{library/curses panel panel-objects}@anchor{26ac}
@subsubsection Panel Objects


Panel objects, as returned by @ref{26a8,,new_panel()} above, are windows with a
stacking order. There’s always a window associated with a panel which determines
the content, while the panel methods are responsible for the window’s depth in
the panel stack.

Panel objects have the following methods:

@geindex above() (curses.panel.Panel method)
@anchor{library/curses panel curses panel Panel above}@anchor{26ad}
@deffn {Method} Panel.above ()

Returns the panel above the current panel.
@end deffn

@geindex below() (curses.panel.Panel method)
@anchor{library/curses panel curses panel Panel below}@anchor{26ae}
@deffn {Method} Panel.below ()

Returns the panel below the current panel.
@end deffn

@geindex bottom() (curses.panel.Panel method)
@anchor{library/curses panel curses panel Panel bottom}@anchor{26af}
@deffn {Method} Panel.bottom ()

Push the panel to the bottom of the stack.
@end deffn

@geindex hidden() (curses.panel.Panel method)
@anchor{library/curses panel curses panel Panel hidden}@anchor{26b0}
@deffn {Method} Panel.hidden ()

Returns @code{True} if the panel is hidden (not visible), @code{False} otherwise.
@end deffn

@geindex hide() (curses.panel.Panel method)
@anchor{library/curses panel curses panel Panel hide}@anchor{26b1}
@deffn {Method} Panel.hide ()

Hide the panel. This does not delete the object, it just makes the window on
screen invisible.
@end deffn

@geindex move() (curses.panel.Panel method)
@anchor{library/curses panel curses panel Panel move}@anchor{26b2}
@deffn {Method} Panel.move (y, x)

Move the panel to the screen coordinates @code{(y, x)}.
@end deffn

@geindex replace() (curses.panel.Panel method)
@anchor{library/curses panel curses panel Panel replace}@anchor{26b3}
@deffn {Method} Panel.replace (win)

Change the window associated with the panel to the window @emph{win}.
@end deffn

@geindex set_userptr() (curses.panel.Panel method)
@anchor{library/curses panel curses panel Panel set_userptr}@anchor{26b4}
@deffn {Method} Panel.set_userptr (obj)

Set the panel’s user pointer to @emph{obj}. This is used to associate an arbitrary
piece of data with the panel, and can be any Python object.
@end deffn

@geindex show() (curses.panel.Panel method)
@anchor{library/curses panel curses panel Panel show}@anchor{26b5}
@deffn {Method} Panel.show ()

Display the panel (which might have been hidden).
@end deffn

@geindex top() (curses.panel.Panel method)
@anchor{library/curses panel curses panel Panel top}@anchor{26b6}
@deffn {Method} Panel.top ()

Push panel to the top of the stack.
@end deffn

@geindex userptr() (curses.panel.Panel method)
@anchor{library/curses panel curses panel Panel userptr}@anchor{26b7}
@deffn {Method} Panel.userptr ()

Returns the user pointer for the panel.  This might be any Python object.
@end deffn

@geindex window() (curses.panel.Panel method)
@anchor{library/curses panel curses panel Panel window}@anchor{26b8}
@deffn {Method} Panel.window ()

Returns the window object associated with the panel.
@end deffn

@node platform — Access to underlying platform’s identifying data,errno — Standard errno system symbols,curses panel — A panel stack extension for curses,Generic Operating System Services
@anchor{library/platform doc}@anchor{26b9}@anchor{library/platform module-platform}@anchor{c9}@anchor{library/platform platform-access-to-underlying-platform-s-identifying-data}@anchor{26ba}
@subsection @code{platform} —  Access to underlying platform’s identifying data


@geindex module; platform

@strong{Source code:} Lib/platform.py@footnote{https://github.com/python/cpython/tree/main/Lib/platform.py}


__________________________________________________________________


@cartouche
@quotation Note 
Specific platforms listed alphabetically, with Linux included in the Unix
section.
@end quotation
@end cartouche

@menu
* Cross Platform:: 
* Java Platform:: 
* Windows Platform:: 
* macOS Platform:: 
* Unix Platforms:: 
* Linux Platforms:: 

@end menu

@node Cross Platform,Java Platform,,platform — Access to underlying platform’s identifying data
@anchor{library/platform cross-platform}@anchor{26bb}
@subsubsection Cross Platform


@geindex architecture() (in module platform)
@anchor{library/platform platform architecture}@anchor{12de}
@deffn {Function} platform.architecture (executable=sys.executable, bits='@w{'}, linkage='@w{'})

Queries the given executable (defaults to the Python interpreter binary) for
various architecture information.

Returns a tuple @code{(bits, linkage)} which contain information about the bit
architecture and the linkage format used for the executable. Both values are
returned as strings.

Values that cannot be determined are returned as given by the parameter presets.
If bits is given as @code{'@w{'}}, the @code{sizeof(pointer)} (or
@code{sizeof(long)} on Python version < 1.5.2) is used as indicator for the
supported pointer size.

The function relies on the system’s @code{file} command to do the actual work.
This is available on most if not all Unix  platforms and some non-Unix platforms
and then only if the executable points to the Python interpreter.  Reasonable
defaults are used when the above needs are not met.

@cartouche
@quotation Note 
On macOS (and perhaps other platforms), executable files may be
universal files containing multiple architectures.

To get at the “64-bitness” of the current interpreter, it is more
reliable to query the @ref{d69,,sys.maxsize} attribute:

@example
is_64bits = sys.maxsize > 2**32
@end example
@end quotation
@end cartouche
@end deffn

@geindex machine() (in module platform)
@anchor{library/platform platform machine}@anchor{26bc}
@deffn {Function} platform.machine ()

Returns the machine type, e.g. @code{'i386'}. An empty string is returned if the
value cannot be determined.
@end deffn

@geindex node() (in module platform)
@anchor{library/platform platform node}@anchor{26bd}
@deffn {Function} platform.node ()

Returns the computer’s network name (may not be fully qualified!). An empty
string is returned if the value cannot be determined.
@end deffn

@geindex platform() (in module platform)
@anchor{library/platform platform platform}@anchor{12e3}
@deffn {Function} platform.platform (aliased=0, terse=0)

Returns a single string identifying the underlying platform with as much useful
information as possible.

The output is intended to be @emph{human readable} rather than machine parseable. It
may look different on different platforms and this is intended.

If @emph{aliased} is true, the function will use aliases for various platforms that
report system names which differ from their common names, for example SunOS will
be reported as Solaris.  The @ref{26be,,system_alias()} function is used to implement
this.

Setting @emph{terse} to true causes the function to return only the absolute minimum
information needed to identify the platform.


Changed in version 3.8: On macOS, the function now uses @ref{12e4,,mac_ver()}, if it returns a
non-empty release string, to get the macOS version rather than the darwin
version.

@end deffn

@geindex processor() (in module platform)
@anchor{library/platform platform processor}@anchor{26bf}
@deffn {Function} platform.processor ()

Returns the (real) processor name, e.g. @code{'amdk6'}.

An empty string is returned if the value cannot be determined. Note that many
platforms do not provide this information or simply return the same value as for
@ref{26bc,,machine()}.  NetBSD does this.
@end deffn

@geindex python_build() (in module platform)
@anchor{library/platform platform python_build}@anchor{26c0}
@deffn {Function} platform.python_build ()

Returns a tuple @code{(buildno, builddate)} stating the Python build number and
date as strings.
@end deffn

@geindex python_compiler() (in module platform)
@anchor{library/platform platform python_compiler}@anchor{26c1}
@deffn {Function} platform.python_compiler ()

Returns a string identifying the compiler used for compiling Python.
@end deffn

@geindex python_branch() (in module platform)
@anchor{library/platform platform python_branch}@anchor{26c2}
@deffn {Function} platform.python_branch ()

Returns a string identifying the Python implementation SCM branch.
@end deffn

@geindex python_implementation() (in module platform)
@anchor{library/platform platform python_implementation}@anchor{26c3}
@deffn {Function} platform.python_implementation ()

Returns a string identifying the Python implementation. Possible return values
are: ‘CPython’, ‘IronPython’, ‘Jython’, ‘PyPy’.
@end deffn

@geindex python_revision() (in module platform)
@anchor{library/platform platform python_revision}@anchor{26c4}
@deffn {Function} platform.python_revision ()

Returns a string identifying the Python implementation SCM revision.
@end deffn

@geindex python_version() (in module platform)
@anchor{library/platform platform python_version}@anchor{26c5}
@deffn {Function} platform.python_version ()

Returns the Python version as string @code{'major.minor.patchlevel'}.

Note that unlike the Python @code{sys.version}, the returned value will always
include the patchlevel (it defaults to 0).
@end deffn

@geindex python_version_tuple() (in module platform)
@anchor{library/platform platform python_version_tuple}@anchor{26c6}
@deffn {Function} platform.python_version_tuple ()

Returns the Python version as tuple @code{(major, minor, patchlevel)} of strings.

Note that unlike the Python @code{sys.version}, the returned value will always
include the patchlevel (it defaults to @code{'0'}).
@end deffn

@geindex release() (in module platform)
@anchor{library/platform platform release}@anchor{26c7}
@deffn {Function} platform.release ()

Returns the system’s release, e.g. @code{'2.2.0'} or @code{'NT'} An empty string is
returned if the value cannot be determined.
@end deffn

@geindex system() (in module platform)
@anchor{library/platform platform system}@anchor{26c8}
@deffn {Function} platform.system ()

Returns the system/OS name, such as @code{'Linux'}, @code{'Darwin'}, @code{'Java'},
@code{'Windows'}. An empty string is returned if the value cannot be determined.
@end deffn

@geindex system_alias() (in module platform)
@anchor{library/platform platform system_alias}@anchor{26be}
@deffn {Function} platform.system_alias (system, release, version)

Returns @code{(system, release, version)} aliased to common marketing names used
for some systems.  It also does some reordering of the information in some cases
where it would otherwise cause confusion.
@end deffn

@geindex version() (in module platform)
@anchor{library/platform platform version}@anchor{26c9}
@deffn {Function} platform.version ()

Returns the system’s release version, e.g. @code{'#3 on degas'}. An empty string is
returned if the value cannot be determined.
@end deffn

@geindex uname() (in module platform)
@anchor{library/platform platform uname}@anchor{12e1}
@deffn {Function} platform.uname ()

Fairly portable uname interface. Returns a @ref{3e4,,namedtuple()}
containing six attributes: @ref{26c8,,system}, @ref{26bd,,node}, @ref{26c7,,release},
@ref{26c9,,version}, @ref{26bc,,machine}, and @ref{26bf,,processor}.

Note that this adds a sixth attribute (@ref{26bf,,processor}) not present
in the @ref{ca9,,os.uname()} result.  Also, the attribute names are different
for the first two attributes; @ref{ca9,,os.uname()} names them
@code{sysname} and @code{nodename}.

Entries which cannot be determined are set to @code{'@w{'}}.


Changed in version 3.3: Result changed from a tuple to a namedtuple.

@end deffn

@node Java Platform,Windows Platform,Cross Platform,platform — Access to underlying platform’s identifying data
@anchor{library/platform java-platform}@anchor{26ca}
@subsubsection Java Platform


@geindex java_ver() (in module platform)
@anchor{library/platform platform java_ver}@anchor{26cb}
@deffn {Function} platform.java_ver (release='@w{'}, vendor='@w{'}, vminfo=('@w{'}, '@w{'}, '@w{'}), osinfo=('@w{'}, '@w{'}, '@w{'}))

Version interface for Jython.

Returns a tuple @code{(release, vendor, vminfo, osinfo)} with @emph{vminfo} being a
tuple @code{(vm_name, vm_release, vm_vendor)} and @emph{osinfo} being a tuple
@code{(os_name, os_version, os_arch)}. Values which cannot be determined are set to
the defaults given as parameters (which all default to @code{'@w{'}}).
@end deffn

@node Windows Platform,macOS Platform,Java Platform,platform — Access to underlying platform’s identifying data
@anchor{library/platform windows-platform}@anchor{26cc}
@subsubsection Windows Platform


@geindex win32_ver() (in module platform)
@anchor{library/platform platform win32_ver}@anchor{11e7}
@deffn {Function} platform.win32_ver (release='@w{'}, version='@w{'}, csd='@w{'}, ptype='@w{'})

Get additional version information from the Windows Registry and return a tuple
@code{(release, version, csd, ptype)} referring to OS release, version number,
CSD level (service pack) and OS type (multi/single processor).

As a hint: @emph{ptype} is @code{'Uniprocessor Free'} on single processor NT machines
and @code{'Multiprocessor Free'} on multi processor machines. The @emph{‘Free’} refers
to the OS version being free of debugging code. It could also state @emph{‘Checked’}
which means the OS version uses debugging code, i.e. code that checks arguments,
ranges, etc.
@end deffn

@geindex win32_edition() (in module platform)
@anchor{library/platform platform win32_edition}@anchor{26cd}
@deffn {Function} platform.win32_edition ()

Returns a string representing the current Windows edition.  Possible
values include but are not limited to @code{'Enterprise'}, @code{'IoTUAP'},
@code{'ServerStandard'}, and @code{'nanoserver'}.


New in version 3.8.

@end deffn

@geindex win32_is_iot() (in module platform)
@anchor{library/platform platform win32_is_iot}@anchor{26ce}
@deffn {Function} platform.win32_is_iot ()

Return @code{True} if the Windows edition returned by @ref{26cd,,win32_edition()}
is recognized as an IoT edition.


New in version 3.8.

@end deffn

@node macOS Platform,Unix Platforms,Windows Platform,platform — Access to underlying platform’s identifying data
@anchor{library/platform macos-platform}@anchor{26cf}
@subsubsection macOS Platform


@geindex mac_ver() (in module platform)
@anchor{library/platform platform mac_ver}@anchor{12e4}
@deffn {Function} platform.mac_ver (release='@w{'}, versioninfo=('@w{'}, '@w{'}, '@w{'}), machine='@w{'})

Get macOS version information and return it as tuple @code{(release, versioninfo,
machine)} with @emph{versioninfo} being a tuple @code{(version, dev_stage,
non_release_version)}.

Entries which cannot be determined are set to @code{'@w{'}}.  All tuple entries are
strings.
@end deffn

@node Unix Platforms,Linux Platforms,macOS Platform,platform — Access to underlying platform’s identifying data
@anchor{library/platform unix-platforms}@anchor{26d0}
@subsubsection Unix Platforms


@geindex libc_ver() (in module platform)
@anchor{library/platform platform libc_ver}@anchor{12e0}
@deffn {Function} platform.libc_ver (executable=sys.executable, lib='@w{'}, version='@w{'}, chunksize=16384)

Tries to determine the libc version against which the file executable (defaults
to the Python interpreter) is linked.  Returns a tuple of strings @code{(lib,
version)} which default to the given parameters in case the lookup fails.

Note that this function has intimate knowledge of how different libc versions
add symbols to the executable is probably only usable for executables compiled
using @code{gcc}.

The file is read and scanned in chunks of @emph{chunksize} bytes.
@end deffn

@node Linux Platforms,,Unix Platforms,platform — Access to underlying platform’s identifying data
@anchor{library/platform linux-platforms}@anchor{26d1}
@subsubsection Linux Platforms


@geindex freedesktop_os_release() (in module platform)
@anchor{library/platform platform freedesktop_os_release}@anchor{2aa}
@deffn {Function} platform.freedesktop_os_release ()

Get operating system identification from @code{os-release} file and return
it as a dict. The @code{os-release} file is a freedesktop.org standard@footnote{https://www.freedesktop.org/software/systemd/man/os-release.html} and
is available in most Linux distributions. A noticeable exception is
Android and Android-based distributions.

Raises @ref{2a1,,OSError} or subclass when neither @code{/etc/os-release} nor
@code{/usr/lib/os-release} can be read.

On success, the function returns a dictionary where keys and values are
strings. Values have their special characters like @code{"} and @code{$}
unquoted. The fields @code{NAME}, @code{ID}, and @code{PRETTY_NAME} are always
defined according to the standard. All other fields are optional. Vendors
may include additional fields.

Note that fields like @code{NAME}, @code{VERSION}, and @code{VARIANT} are strings
suitable for presentation to users. Programs should use fields like
@code{ID}, @code{ID_LIKE}, @code{VERSION_ID}, or @code{VARIANT_ID} to identify
Linux distributions.

Example:

@example
def get_like_distro():
    info = platform.freedesktop_os_release()
    ids = [info["ID"]]
    if "ID_LIKE" in info:
        # ids are space separated and ordered by precedence
        ids.extend(info["ID_LIKE"].split())
    return ids
@end example


New in version 3.10.

@end deffn

@node errno — Standard errno system symbols,ctypes — A foreign function library for Python,platform — Access to underlying platform’s identifying data,Generic Operating System Services
@anchor{library/errno doc}@anchor{26d2}@anchor{library/errno errno-standard-errno-system-symbols}@anchor{26d3}@anchor{library/errno module-errno}@anchor{77}
@subsection @code{errno} — Standard errno system symbols


@geindex module; errno


__________________________________________________________________


This module makes available standard @code{errno} system symbols. The value of each
symbol is the corresponding integer value. The names and descriptions are
borrowed from @code{linux/include/errno.h}, which should be pretty
all-inclusive.

@geindex errorcode (in module errno)
@anchor{library/errno errno errorcode}@anchor{26d4}
@deffn {Data} errno.errorcode

Dictionary providing a mapping from the errno value to the string name in the
underlying system.  For instance, @code{errno.errorcode[errno.EPERM]} maps to
@code{'EPERM'}.
@end deffn

To translate a numeric error code to an error message, use @ref{239b,,os.strerror()}.

Of the following list, symbols that are not used on the current platform are not
defined by the module.  The specific list of defined symbols is available as
@code{errno.errorcode.keys()}.  Symbols available can include:

@geindex EPERM (in module errno)
@anchor{library/errno errno EPERM}@anchor{26d5}
@deffn {Data} errno.EPERM

Operation not permitted
@end deffn

@geindex ENOENT (in module errno)
@anchor{library/errno errno ENOENT}@anchor{26d6}
@deffn {Data} errno.ENOENT

No such file or directory
@end deffn

@geindex ESRCH (in module errno)
@anchor{library/errno errno ESRCH}@anchor{26d7}
@deffn {Data} errno.ESRCH

No such process
@end deffn

@geindex EINTR (in module errno)
@anchor{library/errno errno EINTR}@anchor{8e2}
@deffn {Data} errno.EINTR

Interrupted system call.


@subsubheading See also


This error is mapped to the exception @ref{8e3,,InterruptedError}.

@end deffn

@geindex EIO (in module errno)
@anchor{library/errno errno EIO}@anchor{26d8}
@deffn {Data} errno.EIO

I/O error
@end deffn

@geindex ENXIO (in module errno)
@anchor{library/errno errno ENXIO}@anchor{26d9}
@deffn {Data} errno.ENXIO

No such device or address
@end deffn

@geindex E2BIG (in module errno)
@anchor{library/errno errno E2BIG}@anchor{26da}
@deffn {Data} errno.E2BIG

Arg list too long
@end deffn

@geindex ENOEXEC (in module errno)
@anchor{library/errno errno ENOEXEC}@anchor{26db}
@deffn {Data} errno.ENOEXEC

Exec format error
@end deffn

@geindex EBADF (in module errno)
@anchor{library/errno errno EBADF}@anchor{45e}
@deffn {Data} errno.EBADF

Bad file number
@end deffn

@geindex ECHILD (in module errno)
@anchor{library/errno errno ECHILD}@anchor{26dc}
@deffn {Data} errno.ECHILD

No child processes
@end deffn

@geindex EAGAIN (in module errno)
@anchor{library/errno errno EAGAIN}@anchor{23ce}
@deffn {Data} errno.EAGAIN

Try again
@end deffn

@geindex ENOMEM (in module errno)
@anchor{library/errno errno ENOMEM}@anchor{26dd}
@deffn {Data} errno.ENOMEM

Out of memory
@end deffn

@geindex EACCES (in module errno)
@anchor{library/errno errno EACCES}@anchor{26de}
@deffn {Data} errno.EACCES

Permission denied
@end deffn

@geindex EFAULT (in module errno)
@anchor{library/errno errno EFAULT}@anchor{26df}
@deffn {Data} errno.EFAULT

Bad address
@end deffn

@geindex ENOTBLK (in module errno)
@anchor{library/errno errno ENOTBLK}@anchor{26e0}
@deffn {Data} errno.ENOTBLK

Block device required
@end deffn

@geindex EBUSY (in module errno)
@anchor{library/errno errno EBUSY}@anchor{26e1}
@deffn {Data} errno.EBUSY

Device or resource busy
@end deffn

@geindex EEXIST (in module errno)
@anchor{library/errno errno EEXIST}@anchor{26e2}
@deffn {Data} errno.EEXIST

File exists
@end deffn

@geindex EXDEV (in module errno)
@anchor{library/errno errno EXDEV}@anchor{23a6}
@deffn {Data} errno.EXDEV

Cross-device link
@end deffn

@geindex ENODEV (in module errno)
@anchor{library/errno errno ENODEV}@anchor{26e3}
@deffn {Data} errno.ENODEV

No such device
@end deffn

@geindex ENOTDIR (in module errno)
@anchor{library/errno errno ENOTDIR}@anchor{26e4}
@deffn {Data} errno.ENOTDIR

Not a directory
@end deffn

@geindex EISDIR (in module errno)
@anchor{library/errno errno EISDIR}@anchor{26e5}
@deffn {Data} errno.EISDIR

Is a directory
@end deffn

@geindex EINVAL (in module errno)
@anchor{library/errno errno EINVAL}@anchor{1298}
@deffn {Data} errno.EINVAL

Invalid argument
@end deffn

@geindex ENFILE (in module errno)
@anchor{library/errno errno ENFILE}@anchor{26e6}
@deffn {Data} errno.ENFILE

File table overflow
@end deffn

@geindex EMFILE (in module errno)
@anchor{library/errno errno EMFILE}@anchor{26e7}
@deffn {Data} errno.EMFILE

Too many open files
@end deffn

@geindex ENOTTY (in module errno)
@anchor{library/errno errno ENOTTY}@anchor{26e8}
@deffn {Data} errno.ENOTTY

Not a typewriter
@end deffn

@geindex ETXTBSY (in module errno)
@anchor{library/errno errno ETXTBSY}@anchor{26e9}
@deffn {Data} errno.ETXTBSY

Text file busy
@end deffn

@geindex EFBIG (in module errno)
@anchor{library/errno errno EFBIG}@anchor{26ea}
@deffn {Data} errno.EFBIG

File too large
@end deffn

@geindex ENOSPC (in module errno)
@anchor{library/errno errno ENOSPC}@anchor{26eb}
@deffn {Data} errno.ENOSPC

No space left on device
@end deffn

@geindex ESPIPE (in module errno)
@anchor{library/errno errno ESPIPE}@anchor{26ec}
@deffn {Data} errno.ESPIPE

Illegal seek
@end deffn

@geindex EROFS (in module errno)
@anchor{library/errno errno EROFS}@anchor{26ed}
@deffn {Data} errno.EROFS

Read-only file system
@end deffn

@geindex EMLINK (in module errno)
@anchor{library/errno errno EMLINK}@anchor{26ee}
@deffn {Data} errno.EMLINK

Too many links
@end deffn

@geindex EPIPE (in module errno)
@anchor{library/errno errno EPIPE}@anchor{26ef}
@deffn {Data} errno.EPIPE

Broken pipe
@end deffn

@geindex EDOM (in module errno)
@anchor{library/errno errno EDOM}@anchor{26f0}
@deffn {Data} errno.EDOM

Math argument out of domain of func
@end deffn

@geindex ERANGE (in module errno)
@anchor{library/errno errno ERANGE}@anchor{26f1}
@deffn {Data} errno.ERANGE

Math result not representable
@end deffn

@geindex EDEADLK (in module errno)
@anchor{library/errno errno EDEADLK}@anchor{26f2}
@deffn {Data} errno.EDEADLK

Resource deadlock would occur
@end deffn

@geindex ENAMETOOLONG (in module errno)
@anchor{library/errno errno ENAMETOOLONG}@anchor{26f3}
@deffn {Data} errno.ENAMETOOLONG

File name too long
@end deffn

@geindex ENOLCK (in module errno)
@anchor{library/errno errno ENOLCK}@anchor{26f4}
@deffn {Data} errno.ENOLCK

No record locks available
@end deffn

@geindex ENOSYS (in module errno)
@anchor{library/errno errno ENOSYS}@anchor{26f5}
@deffn {Data} errno.ENOSYS

Function not implemented
@end deffn

@geindex ENOTEMPTY (in module errno)
@anchor{library/errno errno ENOTEMPTY}@anchor{26f6}
@deffn {Data} errno.ENOTEMPTY

Directory not empty
@end deffn

@geindex ELOOP (in module errno)
@anchor{library/errno errno ELOOP}@anchor{26f7}
@deffn {Data} errno.ELOOP

Too many symbolic links encountered
@end deffn

@geindex EWOULDBLOCK (in module errno)
@anchor{library/errno errno EWOULDBLOCK}@anchor{26f8}
@deffn {Data} errno.EWOULDBLOCK

Operation would block
@end deffn

@geindex ENOMSG (in module errno)
@anchor{library/errno errno ENOMSG}@anchor{26f9}
@deffn {Data} errno.ENOMSG

No message of desired type
@end deffn

@geindex EIDRM (in module errno)
@anchor{library/errno errno EIDRM}@anchor{26fa}
@deffn {Data} errno.EIDRM

Identifier removed
@end deffn

@geindex ECHRNG (in module errno)
@anchor{library/errno errno ECHRNG}@anchor{26fb}
@deffn {Data} errno.ECHRNG

Channel number out of range
@end deffn

@geindex EL2NSYNC (in module errno)
@anchor{library/errno errno EL2NSYNC}@anchor{26fc}
@deffn {Data} errno.EL2NSYNC

Level 2 not synchronized
@end deffn

@geindex EL3HLT (in module errno)
@anchor{library/errno errno EL3HLT}@anchor{26fd}
@deffn {Data} errno.EL3HLT

Level 3 halted
@end deffn

@geindex EL3RST (in module errno)
@anchor{library/errno errno EL3RST}@anchor{26fe}
@deffn {Data} errno.EL3RST

Level 3 reset
@end deffn

@geindex ELNRNG (in module errno)
@anchor{library/errno errno ELNRNG}@anchor{26ff}
@deffn {Data} errno.ELNRNG

Link number out of range
@end deffn

@geindex EUNATCH (in module errno)
@anchor{library/errno errno EUNATCH}@anchor{2700}
@deffn {Data} errno.EUNATCH

Protocol driver not attached
@end deffn

@geindex ENOCSI (in module errno)
@anchor{library/errno errno ENOCSI}@anchor{2701}
@deffn {Data} errno.ENOCSI

No CSI structure available
@end deffn

@geindex EL2HLT (in module errno)
@anchor{library/errno errno EL2HLT}@anchor{2702}
@deffn {Data} errno.EL2HLT

Level 2 halted
@end deffn

@geindex EBADE (in module errno)
@anchor{library/errno errno EBADE}@anchor{2703}
@deffn {Data} errno.EBADE

Invalid exchange
@end deffn

@geindex EBADR (in module errno)
@anchor{library/errno errno EBADR}@anchor{2704}
@deffn {Data} errno.EBADR

Invalid request descriptor
@end deffn

@geindex EXFULL (in module errno)
@anchor{library/errno errno EXFULL}@anchor{2705}
@deffn {Data} errno.EXFULL

Exchange full
@end deffn

@geindex ENOANO (in module errno)
@anchor{library/errno errno ENOANO}@anchor{2706}
@deffn {Data} errno.ENOANO

No anode
@end deffn

@geindex EBADRQC (in module errno)
@anchor{library/errno errno EBADRQC}@anchor{2707}
@deffn {Data} errno.EBADRQC

Invalid request code
@end deffn

@geindex EBADSLT (in module errno)
@anchor{library/errno errno EBADSLT}@anchor{2708}
@deffn {Data} errno.EBADSLT

Invalid slot
@end deffn

@geindex EDEADLOCK (in module errno)
@anchor{library/errno errno EDEADLOCK}@anchor{2709}
@deffn {Data} errno.EDEADLOCK

File locking deadlock error
@end deffn

@geindex EBFONT (in module errno)
@anchor{library/errno errno EBFONT}@anchor{270a}
@deffn {Data} errno.EBFONT

Bad font file format
@end deffn

@geindex ENOSTR (in module errno)
@anchor{library/errno errno ENOSTR}@anchor{270b}
@deffn {Data} errno.ENOSTR

Device not a stream
@end deffn

@geindex ENODATA (in module errno)
@anchor{library/errno errno ENODATA}@anchor{270c}
@deffn {Data} errno.ENODATA

No data available
@end deffn

@geindex ETIME (in module errno)
@anchor{library/errno errno ETIME}@anchor{270d}
@deffn {Data} errno.ETIME

Timer expired
@end deffn

@geindex ENOSR (in module errno)
@anchor{library/errno errno ENOSR}@anchor{270e}
@deffn {Data} errno.ENOSR

Out of streams resources
@end deffn

@geindex ENONET (in module errno)
@anchor{library/errno errno ENONET}@anchor{270f}
@deffn {Data} errno.ENONET

Machine is not on the network
@end deffn

@geindex ENOPKG (in module errno)
@anchor{library/errno errno ENOPKG}@anchor{2710}
@deffn {Data} errno.ENOPKG

Package not installed
@end deffn

@geindex EREMOTE (in module errno)
@anchor{library/errno errno EREMOTE}@anchor{2711}
@deffn {Data} errno.EREMOTE

Object is remote
@end deffn

@geindex ENOLINK (in module errno)
@anchor{library/errno errno ENOLINK}@anchor{2712}
@deffn {Data} errno.ENOLINK

Link has been severed
@end deffn

@geindex EADV (in module errno)
@anchor{library/errno errno EADV}@anchor{2713}
@deffn {Data} errno.EADV

Advertise error
@end deffn

@geindex ESRMNT (in module errno)
@anchor{library/errno errno ESRMNT}@anchor{2714}
@deffn {Data} errno.ESRMNT

Srmount error
@end deffn

@geindex ECOMM (in module errno)
@anchor{library/errno errno ECOMM}@anchor{2715}
@deffn {Data} errno.ECOMM

Communication error on send
@end deffn

@geindex EPROTO (in module errno)
@anchor{library/errno errno EPROTO}@anchor{2716}
@deffn {Data} errno.EPROTO

Protocol error
@end deffn

@geindex EMULTIHOP (in module errno)
@anchor{library/errno errno EMULTIHOP}@anchor{2717}
@deffn {Data} errno.EMULTIHOP

Multihop attempted
@end deffn

@geindex EDOTDOT (in module errno)
@anchor{library/errno errno EDOTDOT}@anchor{2718}
@deffn {Data} errno.EDOTDOT

RFS specific error
@end deffn

@geindex EBADMSG (in module errno)
@anchor{library/errno errno EBADMSG}@anchor{2719}
@deffn {Data} errno.EBADMSG

Not a data message
@end deffn

@geindex EOVERFLOW (in module errno)
@anchor{library/errno errno EOVERFLOW}@anchor{271a}
@deffn {Data} errno.EOVERFLOW

Value too large for defined data type
@end deffn

@geindex ENOTUNIQ (in module errno)
@anchor{library/errno errno ENOTUNIQ}@anchor{271b}
@deffn {Data} errno.ENOTUNIQ

Name not unique on network
@end deffn

@geindex EBADFD (in module errno)
@anchor{library/errno errno EBADFD}@anchor{271c}
@deffn {Data} errno.EBADFD

File descriptor in bad state
@end deffn

@geindex EREMCHG (in module errno)
@anchor{library/errno errno EREMCHG}@anchor{271d}
@deffn {Data} errno.EREMCHG

Remote address changed
@end deffn

@geindex ELIBACC (in module errno)
@anchor{library/errno errno ELIBACC}@anchor{271e}
@deffn {Data} errno.ELIBACC

Can not access a needed shared library
@end deffn

@geindex ELIBBAD (in module errno)
@anchor{library/errno errno ELIBBAD}@anchor{271f}
@deffn {Data} errno.ELIBBAD

Accessing a corrupted shared library
@end deffn

@geindex ELIBSCN (in module errno)
@anchor{library/errno errno ELIBSCN}@anchor{2720}
@deffn {Data} errno.ELIBSCN

.lib section in a.out corrupted
@end deffn

@geindex ELIBMAX (in module errno)
@anchor{library/errno errno ELIBMAX}@anchor{2721}
@deffn {Data} errno.ELIBMAX

Attempting to link in too many shared libraries
@end deffn

@geindex ELIBEXEC (in module errno)
@anchor{library/errno errno ELIBEXEC}@anchor{2722}
@deffn {Data} errno.ELIBEXEC

Cannot exec a shared library directly
@end deffn

@geindex EILSEQ (in module errno)
@anchor{library/errno errno EILSEQ}@anchor{2723}
@deffn {Data} errno.EILSEQ

Illegal byte sequence
@end deffn

@geindex ERESTART (in module errno)
@anchor{library/errno errno ERESTART}@anchor{2724}
@deffn {Data} errno.ERESTART

Interrupted system call should be restarted
@end deffn

@geindex ESTRPIPE (in module errno)
@anchor{library/errno errno ESTRPIPE}@anchor{2725}
@deffn {Data} errno.ESTRPIPE

Streams pipe error
@end deffn

@geindex EUSERS (in module errno)
@anchor{library/errno errno EUSERS}@anchor{2726}
@deffn {Data} errno.EUSERS

Too many users
@end deffn

@geindex ENOTSOCK (in module errno)
@anchor{library/errno errno ENOTSOCK}@anchor{2727}
@deffn {Data} errno.ENOTSOCK

Socket operation on non-socket
@end deffn

@geindex EDESTADDRREQ (in module errno)
@anchor{library/errno errno EDESTADDRREQ}@anchor{2728}
@deffn {Data} errno.EDESTADDRREQ

Destination address required
@end deffn

@geindex EMSGSIZE (in module errno)
@anchor{library/errno errno EMSGSIZE}@anchor{2729}
@deffn {Data} errno.EMSGSIZE

Message too long
@end deffn

@geindex EPROTOTYPE (in module errno)
@anchor{library/errno errno EPROTOTYPE}@anchor{272a}
@deffn {Data} errno.EPROTOTYPE

Protocol wrong type for socket
@end deffn

@geindex ENOPROTOOPT (in module errno)
@anchor{library/errno errno ENOPROTOOPT}@anchor{272b}
@deffn {Data} errno.ENOPROTOOPT

Protocol not available
@end deffn

@geindex EPROTONOSUPPORT (in module errno)
@anchor{library/errno errno EPROTONOSUPPORT}@anchor{272c}
@deffn {Data} errno.EPROTONOSUPPORT

Protocol not supported
@end deffn

@geindex ESOCKTNOSUPPORT (in module errno)
@anchor{library/errno errno ESOCKTNOSUPPORT}@anchor{272d}
@deffn {Data} errno.ESOCKTNOSUPPORT

Socket type not supported
@end deffn

@geindex EOPNOTSUPP (in module errno)
@anchor{library/errno errno EOPNOTSUPP}@anchor{272e}
@deffn {Data} errno.EOPNOTSUPP

Operation not supported on transport endpoint
@end deffn

@geindex EPFNOSUPPORT (in module errno)
@anchor{library/errno errno EPFNOSUPPORT}@anchor{272f}
@deffn {Data} errno.EPFNOSUPPORT

Protocol family not supported
@end deffn

@geindex EAFNOSUPPORT (in module errno)
@anchor{library/errno errno EAFNOSUPPORT}@anchor{2730}
@deffn {Data} errno.EAFNOSUPPORT

Address family not supported by protocol
@end deffn

@geindex EADDRINUSE (in module errno)
@anchor{library/errno errno EADDRINUSE}@anchor{2731}
@deffn {Data} errno.EADDRINUSE

Address already in use
@end deffn

@geindex EADDRNOTAVAIL (in module errno)
@anchor{library/errno errno EADDRNOTAVAIL}@anchor{2732}
@deffn {Data} errno.EADDRNOTAVAIL

Cannot assign requested address
@end deffn

@geindex ENETDOWN (in module errno)
@anchor{library/errno errno ENETDOWN}@anchor{2733}
@deffn {Data} errno.ENETDOWN

Network is down
@end deffn

@geindex ENETUNREACH (in module errno)
@anchor{library/errno errno ENETUNREACH}@anchor{12b2}
@deffn {Data} errno.ENETUNREACH

Network is unreachable
@end deffn

@geindex ENETRESET (in module errno)
@anchor{library/errno errno ENETRESET}@anchor{2734}
@deffn {Data} errno.ENETRESET

Network dropped connection because of reset
@end deffn

@geindex ECONNABORTED (in module errno)
@anchor{library/errno errno ECONNABORTED}@anchor{2735}
@deffn {Data} errno.ECONNABORTED

Software caused connection abort
@end deffn

@geindex ECONNRESET (in module errno)
@anchor{library/errno errno ECONNRESET}@anchor{2736}
@deffn {Data} errno.ECONNRESET

Connection reset by peer
@end deffn

@geindex ENOBUFS (in module errno)
@anchor{library/errno errno ENOBUFS}@anchor{2737}
@deffn {Data} errno.ENOBUFS

No buffer space available
@end deffn

@geindex EISCONN (in module errno)
@anchor{library/errno errno EISCONN}@anchor{2738}
@deffn {Data} errno.EISCONN

Transport endpoint is already connected
@end deffn

@geindex ENOTCONN (in module errno)
@anchor{library/errno errno ENOTCONN}@anchor{2739}
@deffn {Data} errno.ENOTCONN

Transport endpoint is not connected
@end deffn

@geindex ESHUTDOWN (in module errno)
@anchor{library/errno errno ESHUTDOWN}@anchor{273a}
@deffn {Data} errno.ESHUTDOWN

Cannot send after transport endpoint shutdown
@end deffn

@geindex ETOOMANYREFS (in module errno)
@anchor{library/errno errno ETOOMANYREFS}@anchor{273b}
@deffn {Data} errno.ETOOMANYREFS

Too many references: cannot splice
@end deffn

@geindex ETIMEDOUT (in module errno)
@anchor{library/errno errno ETIMEDOUT}@anchor{273c}
@deffn {Data} errno.ETIMEDOUT

Connection timed out
@end deffn

@geindex ECONNREFUSED (in module errno)
@anchor{library/errno errno ECONNREFUSED}@anchor{273d}
@deffn {Data} errno.ECONNREFUSED

Connection refused
@end deffn

@geindex EHOSTDOWN (in module errno)
@anchor{library/errno errno EHOSTDOWN}@anchor{273e}
@deffn {Data} errno.EHOSTDOWN

Host is down
@end deffn

@geindex EHOSTUNREACH (in module errno)
@anchor{library/errno errno EHOSTUNREACH}@anchor{273f}
@deffn {Data} errno.EHOSTUNREACH

No route to host
@end deffn

@geindex EALREADY (in module errno)
@anchor{library/errno errno EALREADY}@anchor{2740}
@deffn {Data} errno.EALREADY

Operation already in progress
@end deffn

@geindex EINPROGRESS (in module errno)
@anchor{library/errno errno EINPROGRESS}@anchor{2741}
@deffn {Data} errno.EINPROGRESS

Operation now in progress
@end deffn

@geindex ESTALE (in module errno)
@anchor{library/errno errno ESTALE}@anchor{2742}
@deffn {Data} errno.ESTALE

Stale NFS file handle
@end deffn

@geindex EUCLEAN (in module errno)
@anchor{library/errno errno EUCLEAN}@anchor{2743}
@deffn {Data} errno.EUCLEAN

Structure needs cleaning
@end deffn

@geindex ENOTNAM (in module errno)
@anchor{library/errno errno ENOTNAM}@anchor{2744}
@deffn {Data} errno.ENOTNAM

Not a XENIX named type file
@end deffn

@geindex ENAVAIL (in module errno)
@anchor{library/errno errno ENAVAIL}@anchor{2745}
@deffn {Data} errno.ENAVAIL

No XENIX semaphores available
@end deffn

@geindex EISNAM (in module errno)
@anchor{library/errno errno EISNAM}@anchor{2746}
@deffn {Data} errno.EISNAM

Is a named type file
@end deffn

@geindex EREMOTEIO (in module errno)
@anchor{library/errno errno EREMOTEIO}@anchor{2747}
@deffn {Data} errno.EREMOTEIO

Remote I/O error
@end deffn

@geindex EDQUOT (in module errno)
@anchor{library/errno errno EDQUOT}@anchor{2748}
@deffn {Data} errno.EDQUOT

Quota exceeded
@end deffn

@geindex EQFULL (in module errno)
@anchor{library/errno errno EQFULL}@anchor{2749}
@deffn {Data} errno.EQFULL

Interface output queue is full


New in version 3.11.

@end deffn

@node ctypes — A foreign function library for Python,,errno — Standard errno system symbols,Generic Operating System Services
@anchor{library/ctypes doc}@anchor{274a}@anchor{library/ctypes ctypes-a-foreign-function-library-for-python}@anchor{274b}@anchor{library/ctypes module-ctypes}@anchor{29}
@subsection @code{ctypes} — A foreign function library for Python


@geindex module; ctypes


__________________________________________________________________


@ref{29,,ctypes} is a foreign function library for Python.  It provides C compatible
data types, and allows calling functions in DLLs or shared libraries.  It can be
used to wrap these libraries in pure Python.

@menu
* ctypes tutorial:: 
* ctypes reference:: 

@end menu

@node ctypes tutorial,ctypes reference,,ctypes — A foreign function library for Python
@anchor{library/ctypes ctypes-ctypes-tutorial}@anchor{274c}@anchor{library/ctypes ctypes-tutorial}@anchor{274d}
@subsubsection ctypes tutorial


Note: The code samples in this tutorial use @ref{63,,doctest} to make sure that
they actually work.  Since some code samples behave differently under Linux,
Windows, or macOS, they contain doctest directives in comments.

Note: Some code samples reference the ctypes @ref{274e,,c_int} type.  On platforms
where @code{sizeof(long) == sizeof(int)} it is an alias to @ref{274f,,c_long}.
So, you should not be confused if @ref{274f,,c_long} is printed if you would expect
@ref{274e,,c_int} — they are actually the same type.

@menu
* Loading dynamic link libraries:: 
* Accessing functions from loaded dlls:: 
* Calling functions:: 
* Fundamental data types:: 
* Calling functions@comma{} continued: Calling functions continued. 
* Calling functions with your own custom data types:: 
* Specifying the required argument types (function prototypes): Specifying the required argument types function prototypes. 
* Return types:: 
* Passing pointers (or; passing parameters by reference): Passing pointers or passing parameters by reference. 
* Structures and unions:: 
* Structure/union alignment and byte order:: 
* Bit fields in structures and unions:: 
* Arrays:: 
* Pointers:: 
* Type conversions:: 
* Incomplete Types:: 
* Callback functions:: 
* Accessing values exported from dlls:: 
* Surprises:: 
* Variable-sized data types:: 

@end menu

@node Loading dynamic link libraries,Accessing functions from loaded dlls,,ctypes tutorial
@anchor{library/ctypes ctypes-loading-dynamic-link-libraries}@anchor{2750}@anchor{library/ctypes loading-dynamic-link-libraries}@anchor{2751}
@subsubsection Loading dynamic link libraries


@ref{29,,ctypes} exports the @emph{cdll}, and on Windows @emph{windll} and @emph{oledll}
objects, for loading dynamic link libraries.

You load libraries by accessing them as attributes of these objects. @emph{cdll}
loads libraries which export functions using the standard @code{cdecl} calling
convention, while @emph{windll} libraries call functions using the @code{stdcall}
calling convention. @emph{oledll} also uses the @code{stdcall} calling convention, and
assumes the functions return a Windows @code{HRESULT} error code. The error
code is used to automatically raise an @ref{2a1,,OSError} exception when the
function call fails.


Changed in version 3.3: Windows errors used to raise @ref{bea,,WindowsError}, which is now an alias
of @ref{2a1,,OSError}.


Here are some examples for Windows. Note that @code{msvcrt} is the MS standard C
library containing most standard C functions, and uses the cdecl calling
convention:

@example
>>> from ctypes import *
>>> print(windll.kernel32)  
<WinDLL 'kernel32', handle ... at ...>
>>> print(cdll.msvcrt)      
<CDLL 'msvcrt', handle ... at ...>
>>> libc = cdll.msvcrt      
>>>
@end example

Windows appends the usual @code{.dll} file suffix automatically.

@cartouche
@quotation Note 
Accessing the standard C library through @code{cdll.msvcrt} will use an
outdated version of the library that may be incompatible with the one
being used by Python. Where possible, use native Python functionality,
or else import and use the @code{msvcrt} module.
@end quotation
@end cartouche

On Linux, it is required to specify the filename @emph{including} the extension to
load a library, so attribute access can not be used to load libraries. Either the
@code{LoadLibrary()} method of the dll loaders should be used, or you should load
the library by creating an instance of CDLL by calling the constructor:

@example
>>> cdll.LoadLibrary("libc.so.6")  
<CDLL 'libc.so.6', handle ... at ...>
>>> libc = CDLL("libc.so.6")       
>>> libc                           
<CDLL 'libc.so.6', handle ... at ...>
>>>
@end example

@c XXX Add section for macOS.

@node Accessing functions from loaded dlls,Calling functions,Loading dynamic link libraries,ctypes tutorial
@anchor{library/ctypes accessing-functions-from-loaded-dlls}@anchor{2752}@anchor{library/ctypes ctypes-accessing-functions-from-loaded-dlls}@anchor{2753}
@subsubsection Accessing functions from loaded dlls


Functions are accessed as attributes of dll objects:

@example
>>> from ctypes import *
>>> libc.printf
<_FuncPtr object at 0x...>
>>> print(windll.kernel32.GetModuleHandleA)  
<_FuncPtr object at 0x...>
>>> print(windll.kernel32.MyOwnFunction)     
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "ctypes.py", line 239, in __getattr__
    func = _StdcallFuncPtr(name, self)
AttributeError: function 'MyOwnFunction' not found
>>>
@end example

Note that win32 system dlls like @code{kernel32} and @code{user32} often export ANSI
as well as UNICODE versions of a function. The UNICODE version is exported with
an @code{W} appended to the name, while the ANSI version is exported with an @code{A}
appended to the name. The win32 @code{GetModuleHandle} function, which returns a
@emph{module handle} for a given module name, has the following C prototype, and a
macro is used to expose one of them as @code{GetModuleHandle} depending on whether
UNICODE is defined or not:

@example
/* ANSI version */
HMODULE GetModuleHandleA(LPCSTR lpModuleName);
/* UNICODE version */
HMODULE GetModuleHandleW(LPCWSTR lpModuleName);
@end example

@emph{windll} does not try to select one of them by magic, you must access the
version you need by specifying @code{GetModuleHandleA} or @code{GetModuleHandleW}
explicitly, and then call it with bytes or string objects respectively.

Sometimes, dlls export functions with names which aren’t valid Python
identifiers, like @code{"??2@@YAPAXI@@Z"}. In this case you have to use
@ref{708,,getattr()} to retrieve the function:

@example
>>> getattr(cdll.msvcrt, "??2@@YAPAXI@@Z")  
<_FuncPtr object at 0x...>
>>>
@end example

On Windows, some dlls export functions not by name but by ordinal. These
functions can be accessed by indexing the dll object with the ordinal number:

@example
>>> cdll.kernel32[1]  
<_FuncPtr object at 0x...>
>>> cdll.kernel32[0]  
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "ctypes.py", line 310, in __getitem__
    func = _StdcallFuncPtr(name, self)
AttributeError: function ordinal 0 not found
>>>
@end example

@node Calling functions,Fundamental data types,Accessing functions from loaded dlls,ctypes tutorial
@anchor{library/ctypes calling-functions}@anchor{2754}@anchor{library/ctypes ctypes-calling-functions}@anchor{2755}
@subsubsection Calling functions


You can call these functions like any other Python callable. This example uses
the @code{time()} function, which returns system time in seconds since the Unix
epoch, and the @code{GetModuleHandleA()} function, which returns a win32 module
handle.

This example calls both functions with a @code{NULL} pointer (@code{None} should be used
as the @code{NULL} pointer):

@example
>>> print(libc.time(None))  
1150640792
>>> print(hex(windll.kernel32.GetModuleHandleA(None)))  
0x1d000000
>>>
@end example

@ref{17c,,ValueError} is raised when you call an @code{stdcall} function with the
@code{cdecl} calling convention, or vice versa:

@example
>>> cdll.kernel32.GetModuleHandleA(None)  
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Procedure probably called with not enough arguments (4 bytes missing)
>>>

>>> windll.msvcrt.printf(b"spam")  
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Procedure probably called with too many arguments (4 bytes in excess)
>>>
@end example

To find out the correct calling convention you have to look into the C header
file or the documentation for the function you want to call.

On Windows, @ref{29,,ctypes} uses win32 structured exception handling to prevent
crashes from general protection faults when functions are called with invalid
argument values:

@example
>>> windll.kernel32.GetModuleHandleA(32)  
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: exception: access violation reading 0x00000020
>>>
@end example

There are, however, enough ways to crash Python with @ref{29,,ctypes}, so you
should be careful anyway.  The @ref{78,,faulthandler} module can be helpful in
debugging crashes (e.g. from segmentation faults produced by erroneous C library
calls).

@code{None}, integers, bytes objects and (unicode) strings are the only native
Python objects that can directly be used as parameters in these function calls.
@code{None} is passed as a C @code{NULL} pointer, bytes objects and strings are passed
as pointer to the memory block that contains their data (char* or
wchar_t*).  Python integers are passed as the platforms default C
int type, their value is masked to fit into the C type.

Before we move on calling functions with other parameter types, we have to learn
more about @ref{29,,ctypes} data types.

@node Fundamental data types,Calling functions continued,Calling functions,ctypes tutorial
@anchor{library/ctypes ctypes-fundamental-data-types}@anchor{2756}@anchor{library/ctypes fundamental-data-types}@anchor{2757}
@subsubsection Fundamental data types


@ref{29,,ctypes} defines a number of primitive C compatible data types:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

ctypes type

@tab

C type

@tab

Python type

@item

@ref{2758,,c_bool}

@tab

_Bool

@tab

bool (1)

@item

@ref{2759,,c_char}

@tab

char

@tab

1-character bytes object

@item

@ref{275a,,c_wchar}

@tab

@code{wchar_t}

@tab

1-character string

@item

@ref{275b,,c_byte}

@tab

char

@tab

int

@item

@ref{275c,,c_ubyte}

@tab

unsigned  char

@tab

int

@item

@ref{275d,,c_short}

@tab

short

@tab

int

@item

@ref{275e,,c_ushort}

@tab

unsigned  short

@tab

int

@item

@ref{274e,,c_int}

@tab

int

@tab

int

@item

@ref{275f,,c_uint}

@tab

unsigned  int

@tab

int

@item

@ref{274f,,c_long}

@tab

long

@tab

int

@item

@ref{2760,,c_ulong}

@tab

unsigned  long

@tab

int

@item

@ref{2761,,c_longlong}

@tab

@code{__int64} or long  long

@tab

int

@item

@ref{2762,,c_ulonglong}

@tab

unsigned  __int64 or
unsigned  long  long

@tab

int

@item

@ref{2763,,c_size_t}

@tab

@code{size_t}

@tab

int

@item

@ref{dee,,c_ssize_t}

@tab

@code{ssize_t} or
@code{Py_ssize_t}

@tab

int

@item

@ref{2764,,c_float}

@tab

float

@tab

float

@item

@ref{1172,,c_double}

@tab

double

@tab

float

@item

@ref{1173,,c_longdouble}

@tab

long  double

@tab

float

@item

@ref{2765,,c_char_p}

@tab

char* (NUL terminated)

@tab

bytes object or @code{None}

@item

@ref{113e,,c_wchar_p}

@tab

wchar_t* (NUL terminated)

@tab

string or @code{None}

@item

@ref{2766,,c_void_p}

@tab

void*

@tab

int or @code{None}

@end multitable



@enumerate 

@item 
The constructor accepts any object with a truth value.
@end enumerate

All these types can be created by calling them with an optional initializer of
the correct type and value:

@example
>>> c_int()
c_long(0)
>>> c_wchar_p("Hello, World")
c_wchar_p(140018365411392)
>>> c_ushort(-3)
c_ushort(65533)
>>>
@end example

Since these types are mutable, their value can also be changed afterwards:

@example
>>> i = c_int(42)
>>> print(i)
c_long(42)
>>> print(i.value)
42
>>> i.value = -99
>>> print(i.value)
-99
>>>
@end example

Assigning a new value to instances of the pointer types @ref{2765,,c_char_p},
@ref{113e,,c_wchar_p}, and @ref{2766,,c_void_p} changes the @emph{memory location} they
point to, @emph{not the contents} of the memory block (of course not, because Python
bytes objects are immutable):

@example
>>> s = "Hello, World"
>>> c_s = c_wchar_p(s)
>>> print(c_s)
c_wchar_p(139966785747344)
>>> print(c_s.value)
Hello World
>>> c_s.value = "Hi, there"
>>> print(c_s)              # the memory location has changed
c_wchar_p(139966783348904)
>>> print(c_s.value)
Hi, there
>>> print(s)                # first object is unchanged
Hello, World
>>>
@end example

You should be careful, however, not to pass them to functions expecting pointers
to mutable memory. If you need mutable memory blocks, ctypes has a
@ref{2767,,create_string_buffer()} function which creates these in various ways.  The
current memory block contents can be accessed (or changed) with the @code{raw}
property; if you want to access it as NUL terminated string, use the @code{value}
property:

@example
>>> from ctypes import *
>>> p = create_string_buffer(3)            # create a 3 byte buffer, initialized to NUL bytes
>>> print(sizeof(p), repr(p.raw))
3 b'\x00\x00\x00'
>>> p = create_string_buffer(b"Hello")     # create a buffer containing a NUL terminated string
>>> print(sizeof(p), repr(p.raw))
6 b'Hello\x00'
>>> print(repr(p.value))
b'Hello'
>>> p = create_string_buffer(b"Hello", 10) # create a 10 byte buffer
>>> print(sizeof(p), repr(p.raw))
10 b'Hello\x00\x00\x00\x00\x00'
>>> p.value = b"Hi"
>>> print(sizeof(p), repr(p.raw))
10 b'Hi\x00lo\x00\x00\x00\x00\x00'
>>>
@end example

The @ref{2767,,create_string_buffer()} function replaces the old @code{c_buffer()}
function (which is still available as an alias).  To create a mutable memory
block containing unicode characters of the C type @code{wchar_t}, use the
@ref{126f,,create_unicode_buffer()} function.

@node Calling functions continued,Calling functions with your own custom data types,Fundamental data types,ctypes tutorial
@anchor{library/ctypes calling-functions-continued}@anchor{2768}@anchor{library/ctypes ctypes-calling-functions-continued}@anchor{2769}
@subsubsection Calling functions, continued


Note that printf prints to the real standard output channel, @emph{not} to
@ref{5ef,,sys.stdout}, so these examples will only work at the console prompt, not
from within @emph{IDLE} or @emph{PythonWin}:

@example
>>> printf = libc.printf
>>> printf(b"Hello, %s\n", b"World!")
Hello, World!
14
>>> printf(b"Hello, %S\n", "World!")
Hello, World!
14
>>> printf(b"%d bottles of beer\n", 42)
42 bottles of beer
19
>>> printf(b"%f bottles of beer\n", 42.5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ArgumentError: argument 2: exceptions.TypeError: Don't know how to convert parameter 2
>>>
@end example

As has been mentioned before, all Python types except integers, strings, and
bytes objects have to be wrapped in their corresponding @ref{29,,ctypes} type, so
that they can be converted to the required C data type:

@example
>>> printf(b"An int %d, a double %f\n", 1234, c_double(3.14))
An int 1234, a double 3.140000
31
>>>
@end example

@node Calling functions with your own custom data types,Specifying the required argument types function prototypes,Calling functions continued,ctypes tutorial
@anchor{library/ctypes calling-functions-with-your-own-custom-data-types}@anchor{276a}@anchor{library/ctypes ctypes-calling-functions-with-own-custom-data-types}@anchor{276b}
@subsubsection Calling functions with your own custom data types


You can also customize @ref{29,,ctypes} argument conversion to allow instances of
your own classes be used as function arguments.  @ref{29,,ctypes} looks for an
@code{_as_parameter_} attribute and uses this as the function argument.  Of
course, it must be one of integer, string, or bytes:

@example
>>> class Bottles:
...     def __init__(self, number):
...         self._as_parameter_ = number
...
>>> bottles = Bottles(42)
>>> printf(b"%d bottles of beer\n", bottles)
42 bottles of beer
19
>>>
@end example

If you don’t want to store the instance’s data in the @code{_as_parameter_}
instance variable, you could define a @ref{501,,property} which makes the
attribute available on request.

@node Specifying the required argument types function prototypes,Return types,Calling functions with your own custom data types,ctypes tutorial
@anchor{library/ctypes ctypes-specifying-required-argument-types}@anchor{276c}@anchor{library/ctypes specifying-the-required-argument-types-function-prototypes}@anchor{276d}
@subsubsection Specifying the required argument types (function prototypes)


It is possible to specify the required argument types of functions exported from
DLLs by setting the @code{argtypes} attribute.

@code{argtypes} must be a sequence of C data types (the @code{printf} function is
probably not a good example here, because it takes a variable number and
different types of parameters depending on the format string, on the other hand
this is quite handy to experiment with this feature):

@example
>>> printf.argtypes = [c_char_p, c_char_p, c_int, c_double]
>>> printf(b"String '%s', Int %d, Double %f\n", b"Hi", 10, 2.2)
String 'Hi', Int 10, Double 2.200000
37
>>>
@end example

Specifying a format protects against incompatible argument types (just as a
prototype for a C function), and tries to convert the arguments to valid types:

@example
>>> printf(b"%d %d %d", 1, 2, 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ArgumentError: argument 2: exceptions.TypeError: wrong type
>>> printf(b"%s %d %f\n", b"X", 2, 3)
X 2 3.000000
13
>>>
@end example

If you have defined your own classes which you pass to function calls, you have
to implement a @code{from_param()} class method for them to be able to use them
in the @code{argtypes} sequence. The @code{from_param()} class method receives
the Python object passed to the function call, it should do a typecheck or
whatever is needed to make sure this object is acceptable, and then return the
object itself, its @code{_as_parameter_} attribute, or whatever you want to
pass as the C function argument in this case. Again, the result should be an
integer, string, bytes, a @ref{29,,ctypes} instance, or an object with an
@code{_as_parameter_} attribute.

@node Return types,Passing pointers or passing parameters by reference,Specifying the required argument types function prototypes,ctypes tutorial
@anchor{library/ctypes ctypes-return-types}@anchor{276e}@anchor{library/ctypes return-types}@anchor{276f}
@subsubsection Return types


By default functions are assumed to return the C int type.  Other
return types can be specified by setting the @code{restype} attribute of the
function object.

Here is a more advanced example, it uses the @code{strchr} function, which expects
a string pointer and a char, and returns a pointer to a string:

@example
>>> strchr = libc.strchr
>>> strchr(b"abcdef", ord("d"))  
8059983
>>> strchr.restype = c_char_p    # c_char_p is a pointer to a string
>>> strchr(b"abcdef", ord("d"))
b'def'
>>> print(strchr(b"abcdef", ord("x")))
None
>>>
@end example

If you want to avoid the @code{ord("x")} calls above, you can set the
@code{argtypes} attribute, and the second argument will be converted from a
single character Python bytes object into a C char:

@example
>>> strchr.restype = c_char_p
>>> strchr.argtypes = [c_char_p, c_char]
>>> strchr(b"abcdef", b"d")
'def'
>>> strchr(b"abcdef", b"def")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ArgumentError: argument 2: exceptions.TypeError: one character string expected
>>> print(strchr(b"abcdef", b"x"))
None
>>> strchr(b"abcdef", b"d")
'def'
>>>
@end example

You can also use a callable Python object (a function or a class for example) as
the @code{restype} attribute, if the foreign function returns an integer.  The
callable will be called with the @emph{integer} the C function returns, and the
result of this call will be used as the result of your function call. This is
useful to check for error return values and automatically raise an exception:

@example
>>> GetModuleHandle = windll.kernel32.GetModuleHandleA  
>>> def ValidHandle(value):
...     if value == 0:
...         raise WinError()
...     return value
...
>>>
>>> GetModuleHandle.restype = ValidHandle  
>>> GetModuleHandle(None)  
486539264
>>> GetModuleHandle("something silly")  
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in ValidHandle
OSError: [Errno 126] The specified module could not be found.
>>>
@end example

@code{WinError} is a function which will call Windows @code{FormatMessage()} api to
get the string representation of an error code, and @emph{returns} an exception.
@code{WinError} takes an optional error code parameter, if no one is used, it calls
@ref{2770,,GetLastError()} to retrieve it.

Please note that a much more powerful error checking mechanism is available
through the @code{errcheck} attribute; see the reference manual for details.

@node Passing pointers or passing parameters by reference,Structures and unions,Return types,ctypes tutorial
@anchor{library/ctypes ctypes-passing-pointers}@anchor{2771}@anchor{library/ctypes passing-pointers-or-passing-parameters-by-reference}@anchor{2772}
@subsubsection Passing pointers (or: passing parameters by reference)


Sometimes a C api function expects a @emph{pointer} to a data type as parameter,
probably to write into the corresponding location, or if the data is too large
to be passed by value. This is also known as @emph{passing parameters by reference}.

@ref{29,,ctypes} exports the @ref{2773,,byref()} function which is used to pass parameters
by reference.  The same effect can be achieved with the @ref{2774,,pointer()} function,
although @ref{2774,,pointer()} does a lot more work since it constructs a real pointer
object, so it is faster to use @ref{2773,,byref()} if you don’t need the pointer
object in Python itself:

@example
>>> i = c_int()
>>> f = c_float()
>>> s = create_string_buffer(b'\000' * 32)
>>> print(i.value, f.value, repr(s.value))
0 0.0 b'@w{'}
>>> libc.sscanf(b"1 3.14 Hello", b"%d %f %s",
...             byref(i), byref(f), s)
3
>>> print(i.value, f.value, repr(s.value))
1 3.1400001049 b'Hello'
>>>
@end example

@node Structures and unions,Structure/union alignment and byte order,Passing pointers or passing parameters by reference,ctypes tutorial
@anchor{library/ctypes ctypes-structures-unions}@anchor{2775}@anchor{library/ctypes structures-and-unions}@anchor{2776}
@subsubsection Structures and unions


Structures and unions must derive from the @ref{2777,,Structure} and @ref{2778,,Union}
base classes which are defined in the @ref{29,,ctypes} module. Each subclass must
define a @code{_fields_} attribute.  @code{_fields_} must be a list of
@emph{2-tuples}, containing a @emph{field name} and a @emph{field type}.

The field type must be a @ref{29,,ctypes} type like @ref{274e,,c_int}, or any other
derived @ref{29,,ctypes} type: structure, union, array, pointer.

Here is a simple example of a POINT structure, which contains two integers named
@emph{x} and @emph{y}, and also shows how to initialize a structure in the constructor:

@example
>>> from ctypes import *
>>> class POINT(Structure):
...     _fields_ = [("x", c_int),
...                 ("y", c_int)]
...
>>> point = POINT(10, 20)
>>> print(point.x, point.y)
10 20
>>> point = POINT(y=5)
>>> print(point.x, point.y)
0 5
>>> POINT(1, 2, 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: too many initializers
>>>
@end example

You can, however, build much more complicated structures.  A structure can
itself contain other structures by using a structure as a field type.

Here is a RECT structure which contains two POINTs named @emph{upperleft} and
@emph{lowerright}:

@example
>>> class RECT(Structure):
...     _fields_ = [("upperleft", POINT),
...                 ("lowerright", POINT)]
...
>>> rc = RECT(point)
>>> print(rc.upperleft.x, rc.upperleft.y)
0 5
>>> print(rc.lowerright.x, rc.lowerright.y)
0 0
>>>
@end example

Nested structures can also be initialized in the constructor in several ways:

@example
>>> r = RECT(POINT(1, 2), POINT(3, 4))
>>> r = RECT((1, 2), (3, 4))
@end example

Field @ref{17bc,,descriptor}s can be retrieved from the @emph{class}, they are useful
for debugging because they can provide useful information:

@example
>>> print(POINT.x)
<Field type=c_long, ofs=0, size=4>
>>> print(POINT.y)
<Field type=c_long, ofs=4, size=4>
>>>
@end example
@anchor{library/ctypes ctypes-structureunion-alignment-byte-order}@anchor{2779}
@cartouche
@quotation Warning 
@ref{29,,ctypes} does not support passing unions or structures with bit-fields
to functions by value.  While this may work on 32-bit x86, it’s not
guaranteed by the library to work in the general case.  Unions and
structures with bit-fields should always be passed to functions by pointer.
@end quotation
@end cartouche

@node Structure/union alignment and byte order,Bit fields in structures and unions,Structures and unions,ctypes tutorial
@anchor{library/ctypes structure-union-alignment-and-byte-order}@anchor{277a}
@subsubsection Structure/union alignment and byte order


By default, Structure and Union fields are aligned in the same way the C
compiler does it. It is possible to override this behavior by specifying a
@code{_pack_} class attribute in the subclass definition. This must be set to a
positive integer and specifies the maximum alignment for the fields. This is
what @code{#pragma pack(n)} also does in MSVC.

@ref{29,,ctypes} uses the native byte order for Structures and Unions.  To build
structures with non-native byte order, you can use one of the
@ref{277b,,BigEndianStructure}, @ref{277c,,LittleEndianStructure},
@code{BigEndianUnion}, and @code{LittleEndianUnion} base classes.  These
classes cannot contain pointer fields.

@node Bit fields in structures and unions,Arrays,Structure/union alignment and byte order,ctypes tutorial
@anchor{library/ctypes bit-fields-in-structures-and-unions}@anchor{277d}@anchor{library/ctypes ctypes-bit-fields-in-structures-unions}@anchor{277e}
@subsubsection Bit fields in structures and unions


It is possible to create structures and unions containing bit fields. Bit fields
are only possible for integer fields, the bit width is specified as the third
item in the @code{_fields_} tuples:

@example
>>> class Int(Structure):
...     _fields_ = [("first_16", c_int, 16),
...                 ("second_16", c_int, 16)]
...
>>> print(Int.first_16)
<Field type=c_long, ofs=0:0, bits=16>
>>> print(Int.second_16)
<Field type=c_long, ofs=0:16, bits=16>
>>>
@end example

@node Arrays,Pointers,Bit fields in structures and unions,ctypes tutorial
@anchor{library/ctypes arrays}@anchor{277f}@anchor{library/ctypes ctypes-arrays}@anchor{2780}
@subsubsection Arrays


Arrays are sequences, containing a fixed number of instances of the same type.

The recommended way to create array types is by multiplying a data type with a
positive integer:

@example
TenPointsArrayType = POINT * 10
@end example

Here is an example of a somewhat artificial data type, a structure containing 4
POINTs among other stuff:

@example
>>> from ctypes import *
>>> class POINT(Structure):
...     _fields_ = ("x", c_int), ("y", c_int)
...
>>> class MyStruct(Structure):
...     _fields_ = [("a", c_int),
...                 ("b", c_float),
...                 ("point_array", POINT * 4)]
>>>
>>> print(len(MyStruct().point_array))
4
>>>
@end example

Instances are created in the usual way, by calling the class:

@example
arr = TenPointsArrayType()
for pt in arr:
    print(pt.x, pt.y)
@end example

The above code print a series of @code{0 0} lines, because the array contents is
initialized to zeros.

Initializers of the correct type can also be specified:

@example
>>> from ctypes import *
>>> TenIntegers = c_int * 10
>>> ii = TenIntegers(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
>>> print(ii)
<c_long_Array_10 object at 0x...>
>>> for i in ii: print(i, end=" ")
...
1 2 3 4 5 6 7 8 9 10
>>>
@end example

@node Pointers,Type conversions,Arrays,ctypes tutorial
@anchor{library/ctypes ctypes-pointers}@anchor{2781}@anchor{library/ctypes pointers}@anchor{2782}
@subsubsection Pointers


Pointer instances are created by calling the @ref{2774,,pointer()} function on a
@ref{29,,ctypes} type:

@example
>>> from ctypes import *
>>> i = c_int(42)
>>> pi = pointer(i)
>>>
@end example

Pointer instances have a @ref{2783,,contents} attribute which
returns the object to which the pointer points, the @code{i} object above:

@example
>>> pi.contents
c_long(42)
>>>
@end example

Note that @ref{29,,ctypes} does not have OOR (original object return), it constructs a
new, equivalent object each time you retrieve an attribute:

@example
>>> pi.contents is i
False
>>> pi.contents is pi.contents
False
>>>
@end example

Assigning another @ref{274e,,c_int} instance to the pointer’s contents attribute
would cause the pointer to point to the memory location where this is stored:

@example
>>> i = c_int(99)
>>> pi.contents = i
>>> pi.contents
c_long(99)
>>>
@end example

@c XXX Document dereferencing pointers, and that it is preferred over the
@c .contents attribute.

Pointer instances can also be indexed with integers:

@example
>>> pi[0]
99
>>>
@end example

Assigning to an integer index changes the pointed to value:

@example
>>> print(i)
c_long(99)
>>> pi[0] = 22
>>> print(i)
c_long(22)
>>>
@end example

It is also possible to use indexes different from 0, but you must know what
you’re doing, just as in C: You can access or change arbitrary memory locations.
Generally you only use this feature if you receive a pointer from a C function,
and you @emph{know} that the pointer actually points to an array instead of a single
item.

Behind the scenes, the @ref{2774,,pointer()} function does more than simply create
pointer instances, it has to create pointer @emph{types} first. This is done with the
@ref{2784,,POINTER()} function, which accepts any @ref{29,,ctypes} type, and returns a
new type:

@example
>>> PI = POINTER(c_int)
>>> PI
<class 'ctypes.LP_c_long'>
>>> PI(42)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: expected c_long instead of int
>>> PI(c_int(42))
<ctypes.LP_c_long object at 0x...>
>>>
@end example

Calling the pointer type without an argument creates a @code{NULL} pointer.
@code{NULL} pointers have a @code{False} boolean value:

@example
>>> null_ptr = POINTER(c_int)()
>>> print(bool(null_ptr))
False
>>>
@end example

@ref{29,,ctypes} checks for @code{NULL} when dereferencing pointers (but dereferencing
invalid non-@code{NULL} pointers would crash Python):

@example
>>> null_ptr[0]
Traceback (most recent call last):
    ....
ValueError: NULL pointer access
>>>

>>> null_ptr[0] = 1234
Traceback (most recent call last):
    ....
ValueError: NULL pointer access
>>>
@end example

@node Type conversions,Incomplete Types,Pointers,ctypes tutorial
@anchor{library/ctypes ctypes-type-conversions}@anchor{2785}@anchor{library/ctypes type-conversions}@anchor{2786}
@subsubsection Type conversions


Usually, ctypes does strict type checking.  This means, if you have
@code{POINTER(c_int)} in the @code{argtypes} list of a function or as the type of
a member field in a structure definition, only instances of exactly the same
type are accepted.  There are some exceptions to this rule, where ctypes accepts
other objects.  For example, you can pass compatible array instances instead of
pointer types.  So, for @code{POINTER(c_int)}, ctypes accepts an array of c_int:

@example
>>> class Bar(Structure):
...     _fields_ = [("count", c_int), ("values", POINTER(c_int))]
...
>>> bar = Bar()
>>> bar.values = (c_int * 3)(1, 2, 3)
>>> bar.count = 3
>>> for i in range(bar.count):
...     print(bar.values[i])
...
1
2
3
>>>
@end example

In addition, if a function argument is explicitly declared to be a pointer type
(such as @code{POINTER(c_int)}) in @code{argtypes}, an object of the pointed
type (@code{c_int} in this case) can be passed to the function.  ctypes will apply
the required @ref{2773,,byref()} conversion in this case automatically.

To set a POINTER type field to @code{NULL}, you can assign @code{None}:

@example
>>> bar.values = None
>>>
@end example

@c XXX list other conversions...

Sometimes you have instances of incompatible types.  In C, you can cast one type
into another type.  @ref{29,,ctypes} provides a @ref{12db,,cast()} function which can be
used in the same way.  The @code{Bar} structure defined above accepts
@code{POINTER(c_int)} pointers or @ref{274e,,c_int} arrays for its @code{values} field,
but not instances of other types:

@example
>>> bar.values = (c_byte * 4)()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: incompatible types, c_byte_Array_4 instance instead of LP_c_long instance
>>>
@end example

For these cases, the @ref{12db,,cast()} function is handy.

The @ref{12db,,cast()} function can be used to cast a ctypes instance into a pointer
to a different ctypes data type.  @ref{12db,,cast()} takes two parameters, a ctypes
object that is or can be converted to a pointer of some kind, and a ctypes
pointer type.  It returns an instance of the second argument, which references
the same memory block as the first argument:

@example
>>> a = (c_byte * 4)()
>>> cast(a, POINTER(c_int))
<ctypes.LP_c_long object at ...>
>>>
@end example

So, @ref{12db,,cast()} can be used to assign to the @code{values} field of @code{Bar} the
structure:

@example
>>> bar = Bar()
>>> bar.values = cast((c_byte * 4)(), POINTER(c_int))
>>> print(bar.values[0])
0
>>>
@end example

@node Incomplete Types,Callback functions,Type conversions,ctypes tutorial
@anchor{library/ctypes ctypes-incomplete-types}@anchor{2787}@anchor{library/ctypes incomplete-types}@anchor{2788}
@subsubsection Incomplete Types


@emph{Incomplete Types} are structures, unions or arrays whose members are not yet
specified. In C, they are specified by forward declarations, which are defined
later:

@example
struct cell; /* forward declaration */

struct cell @{
    char *name;
    struct cell *next;
@};
@end example

The straightforward translation into ctypes code would be this, but it does not
work:

@example
>>> class cell(Structure):
...     _fields_ = [("name", c_char_p),
...                 ("next", POINTER(cell))]
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in cell
NameError: name 'cell' is not defined
>>>
@end example

because the new @code{class cell} is not available in the class statement itself.
In @ref{29,,ctypes}, we can define the @code{cell} class and set the @code{_fields_}
attribute later, after the class statement:

@example
>>> from ctypes import *
>>> class cell(Structure):
...     pass
...
>>> cell._fields_ = [("name", c_char_p),
...                  ("next", POINTER(cell))]
>>>
@end example

Let’s try it. We create two instances of @code{cell}, and let them point to each
other, and finally follow the pointer chain a few times:

@example
>>> c1 = cell()
>>> c1.name = b"foo"
>>> c2 = cell()
>>> c2.name = b"bar"
>>> c1.next = pointer(c2)
>>> c2.next = pointer(c1)
>>> p = c1
>>> for i in range(8):
...     print(p.name, end=" ")
...     p = p.next[0]
...
foo bar foo bar foo bar foo bar
>>>
@end example

@node Callback functions,Accessing values exported from dlls,Incomplete Types,ctypes tutorial
@anchor{library/ctypes callback-functions}@anchor{2789}@anchor{library/ctypes ctypes-callback-functions}@anchor{278a}
@subsubsection Callback functions


@ref{29,,ctypes} allows creating C callable function pointers from Python callables.
These are sometimes called @emph{callback functions}.

First, you must create a class for the callback function. The class knows the
calling convention, the return type, and the number and types of arguments this
function will receive.

The @ref{278b,,CFUNCTYPE()} factory function creates types for callback functions
using the @code{cdecl} calling convention. On Windows, the @ref{278c,,WINFUNCTYPE()}
factory function creates types for callback functions using the @code{stdcall}
calling convention.

Both of these factory functions are called with the result type as first
argument, and the callback functions expected argument types as the remaining
arguments.

I will present an example here which uses the standard C library’s
@code{qsort()} function, that is used to sort items with the help of a callback
function.  @code{qsort()} will be used to sort an array of integers:

@example
>>> IntArray5 = c_int * 5
>>> ia = IntArray5(5, 1, 7, 33, 99)
>>> qsort = libc.qsort
>>> qsort.restype = None
>>>
@end example

@code{qsort()} must be called with a pointer to the data to sort, the number of
items in the data array, the size of one item, and a pointer to the comparison
function, the callback. The callback will then be called with two pointers to
items, and it must return a negative integer if the first item is smaller than
the second, a zero if they are equal, and a positive integer otherwise.

So our callback function receives pointers to integers, and must return an
integer. First we create the @code{type} for the callback function:

@example
>>> CMPFUNC = CFUNCTYPE(c_int, POINTER(c_int), POINTER(c_int))
>>>
@end example

To get started, here is a simple callback that shows the values it gets
passed:

@example
>>> def py_cmp_func(a, b):
...     print("py_cmp_func", a[0], b[0])
...     return 0
...
>>> cmp_func = CMPFUNC(py_cmp_func)
>>>
@end example

The result:

@example
>>> qsort(ia, len(ia), sizeof(c_int), cmp_func)  
py_cmp_func 5 1
py_cmp_func 33 99
py_cmp_func 7 33
py_cmp_func 5 7
py_cmp_func 1 7
>>>
@end example

Now we can actually compare the two items and return a useful result:

@example
>>> def py_cmp_func(a, b):
...     print("py_cmp_func", a[0], b[0])
...     return a[0] - b[0]
...
>>>
>>> qsort(ia, len(ia), sizeof(c_int), CMPFUNC(py_cmp_func)) 
py_cmp_func 5 1
py_cmp_func 33 99
py_cmp_func 7 33
py_cmp_func 1 7
py_cmp_func 5 7
>>>
@end example

As we can easily check, our array is sorted now:

@example
>>> for i in ia: print(i, end=" ")
...
1 5 7 33 99
>>>
@end example

The function factories can be used as decorator factories, so we may as well
write:

@example
>>> @@CFUNCTYPE(c_int, POINTER(c_int), POINTER(c_int))
... def py_cmp_func(a, b):
...     print("py_cmp_func", a[0], b[0])
...     return a[0] - b[0]
...
>>> qsort(ia, len(ia), sizeof(c_int), py_cmp_func)
py_cmp_func 5 1
py_cmp_func 33 99
py_cmp_func 7 33
py_cmp_func 1 7
py_cmp_func 5 7
>>>
@end example

@cartouche
@quotation Note 
Make sure you keep references to @ref{278b,,CFUNCTYPE()} objects as long as they
are used from C code. @ref{29,,ctypes} doesn’t, and if you don’t, they may be
garbage collected, crashing your program when a callback is made.

Also, note that if the callback function is called in a thread created
outside of Python’s control (e.g. by the foreign code that calls the
callback), ctypes creates a new dummy Python thread on every invocation. This
behavior is correct for most purposes, but it means that values stored with
@ref{278d,,threading.local} will @emph{not} survive across different callbacks, even when
those calls are made from the same C thread.
@end quotation
@end cartouche

@node Accessing values exported from dlls,Surprises,Callback functions,ctypes tutorial
@anchor{library/ctypes accessing-values-exported-from-dlls}@anchor{278e}@anchor{library/ctypes ctypes-accessing-values-exported-from-dlls}@anchor{278f}
@subsubsection Accessing values exported from dlls


Some shared libraries not only export functions, they also export variables. An
example in the Python library itself is the @ref{2790,,Py_OptimizeFlag}, an integer
set to 0, 1, or 2, depending on the @ref{912,,-O} or @ref{913,,-OO} flag given on
startup.

@ref{29,,ctypes} can access values like this with the @code{in_dll()} class methods of
the type.  @emph{pythonapi} is a predefined symbol giving access to the Python C
api:

@example
>>> opt_flag = c_int.in_dll(pythonapi, "Py_OptimizeFlag")
>>> print(opt_flag)
c_long(0)
>>>
@end example

If the interpreter would have been started with @ref{912,,-O}, the sample would
have printed @code{c_long(1)}, or @code{c_long(2)} if @ref{913,,-OO} would have been
specified.

An extended example which also demonstrates the use of pointers accesses the
@ref{2791,,PyImport_FrozenModules} pointer exported by Python.

Quoting the docs for that value:

@quotation

This pointer is initialized to point to an array of struct  @ref{1d3,,_frozen}
records, terminated by one whose members are all @code{NULL} or zero.  When a frozen
module is imported, it is searched in this table.  Third-party code could play
tricks with this to provide a dynamically created collection of frozen modules.
@end quotation

So manipulating this pointer could even prove useful. To restrict the example
size, we show only how this table can be read with @ref{29,,ctypes}:

@example
>>> from ctypes import *
>>>
>>> class struct_frozen(Structure):
...     _fields_ = [("name", c_char_p),
...                 ("code", POINTER(c_ubyte)),
...                 ("size", c_int),
...                 ("get_code", POINTER(c_ubyte)),  # Function pointer
...                ]
...
>>>
@end example

We have defined the struct  @ref{1d3,,_frozen} data type, so we can get the pointer
to the table:

@example
>>> FrozenTable = POINTER(struct_frozen)
>>> table = FrozenTable.in_dll(pythonapi, "_PyImport_FrozenBootstrap")
>>>
@end example

Since @code{table} is a @code{pointer} to the array of @code{struct_frozen} records, we
can iterate over it, but we just have to make sure that our loop terminates,
because pointers have no size. Sooner or later it would probably crash with an
access violation or whatever, so it’s better to break out of the loop when we
hit the @code{NULL} entry:

@example
>>> for item in table:
...     if item.name is None:
...         break
...     print(item.name.decode("ascii"), item.size)
...
_frozen_importlib 31764
_frozen_importlib_external 41499
zipimport 12345
>>>
@end example

The fact that standard Python has a frozen module and a frozen package
(indicated by the negative @code{size} member) is not well known, it is only used
for testing. Try it out with @code{import __hello__} for example.

@node Surprises,Variable-sized data types,Accessing values exported from dlls,ctypes tutorial
@anchor{library/ctypes ctypes-surprises}@anchor{2792}@anchor{library/ctypes surprises}@anchor{2793}
@subsubsection Surprises


There are some edges in @ref{29,,ctypes} where you might expect something other
than what actually happens.

Consider the following example:

@example
>>> from ctypes import *
>>> class POINT(Structure):
...     _fields_ = ("x", c_int), ("y", c_int)
...
>>> class RECT(Structure):
...     _fields_ = ("a", POINT), ("b", POINT)
...
>>> p1 = POINT(1, 2)
>>> p2 = POINT(3, 4)
>>> rc = RECT(p1, p2)
>>> print(rc.a.x, rc.a.y, rc.b.x, rc.b.y)
1 2 3 4
>>> # now swap the two points
>>> rc.a, rc.b = rc.b, rc.a
>>> print(rc.a.x, rc.a.y, rc.b.x, rc.b.y)
3 4 3 4
>>>
@end example

Hm. We certainly expected the last statement to print @code{3 4 1 2}. What
happened? Here are the steps of the @code{rc.a, rc.b = rc.b, rc.a} line above:

@example
>>> temp0, temp1 = rc.b, rc.a
>>> rc.a = temp0
>>> rc.b = temp1
>>>
@end example

Note that @code{temp0} and @code{temp1} are objects still using the internal buffer of
the @code{rc} object above. So executing @code{rc.a = temp0} copies the buffer
contents of @code{temp0} into @code{rc} ‘s buffer.  This, in turn, changes the
contents of @code{temp1}. So, the last assignment @code{rc.b = temp1}, doesn’t have
the expected effect.

Keep in mind that retrieving sub-objects from Structure, Unions, and Arrays
doesn’t @emph{copy} the sub-object, instead it retrieves a wrapper object accessing
the root-object’s underlying buffer.

Another example that may behave differently from what one would expect is this:

@example
>>> s = c_char_p()
>>> s.value = b"abc def ghi"
>>> s.value
b'abc def ghi'
>>> s.value is s.value
False
>>>
@end example

@cartouche
@quotation Note 
Objects instantiated from @ref{2765,,c_char_p} can only have their value set to bytes
or integers.
@end quotation
@end cartouche

Why is it printing @code{False}?  ctypes instances are objects containing a memory
block plus some @ref{17bc,,descriptor}s accessing the contents of the memory.
Storing a Python object in the memory block does not store the object itself,
instead the @code{contents} of the object is stored.  Accessing the contents again
constructs a new Python object each time!

@node Variable-sized data types,,Surprises,ctypes tutorial
@anchor{library/ctypes ctypes-variable-sized-data-types}@anchor{2794}@anchor{library/ctypes variable-sized-data-types}@anchor{2795}
@subsubsection Variable-sized data types


@ref{29,,ctypes} provides some support for variable-sized arrays and structures.

The @ref{2796,,resize()} function can be used to resize the memory buffer of an
existing ctypes object.  The function takes the object as first argument, and
the requested size in bytes as the second argument.  The memory block cannot be
made smaller than the natural memory block specified by the objects type, a
@ref{17c,,ValueError} is raised if this is tried:

@example
>>> short_array = (c_short * 4)()
>>> print(sizeof(short_array))
8
>>> resize(short_array, 4)
Traceback (most recent call last):
    ...
ValueError: minimum size is 8
>>> resize(short_array, 32)
>>> sizeof(short_array)
32
>>> sizeof(type(short_array))
8
>>>
@end example

This is nice and fine, but how would one access the additional elements
contained in this array?  Since the type still only knows about 4 elements, we
get errors accessing other elements:

@example
>>> short_array[:]
[0, 0, 0, 0]
>>> short_array[7]
Traceback (most recent call last):
    ...
IndexError: invalid index
>>>
@end example

Another way to use variable-sized data types with @ref{29,,ctypes} is to use the
dynamic nature of Python, and (re-)define the data type after the required size
is already known, on a case by case basis.

@node ctypes reference,,ctypes tutorial,ctypes — A foreign function library for Python
@anchor{library/ctypes ctypes-ctypes-reference}@anchor{2797}@anchor{library/ctypes ctypes-reference}@anchor{2798}
@subsubsection ctypes reference


@menu
* Finding shared libraries:: 
* Loading shared libraries:: 
* Foreign functions:: 
* Function prototypes:: 
* Utility functions:: 
* Data types:: 
* Fundamental data types: Fundamental data types<2>. 
* Structured data types:: 
* Arrays and pointers:: 

@end menu

@node Finding shared libraries,Loading shared libraries,,ctypes reference
@anchor{library/ctypes ctypes-finding-shared-libraries}@anchor{2799}@anchor{library/ctypes finding-shared-libraries}@anchor{279a}
@subsubsection Finding shared libraries


When programming in a compiled language, shared libraries are accessed when
compiling/linking a program, and when the program is run.

The purpose of the @code{find_library()} function is to locate a library in a way
similar to what the compiler or runtime loader does (on platforms with several
versions of a shared library the most recent should be loaded), while the ctypes
library loaders act like when a program is run, and call the runtime loader
directly.

The @code{ctypes.util} module provides a function which can help to determine
the library to load.


@deffn {Data} ctypes.util.find_library (name)

Try to find a library and return a pathname.  @emph{name} is the library name without
any prefix like @emph{lib}, suffix like @code{.so}, @code{.dylib} or version number (this
is the form used for the posix linker option @code{-l}).  If no library can
be found, returns @code{None}.
@end deffn

The exact functionality is system dependent.

On Linux, @code{find_library()} tries to run external programs
(@code{/sbin/ldconfig}, @code{gcc}, @code{objdump} and @code{ld}) to find the library file.
It returns the filename of the library file.


Changed in version 3.6: On Linux, the value of the environment variable @code{LD_LIBRARY_PATH} is used
when searching for libraries, if a library cannot be found by any other means.


Here are some examples:

@example
>>> from ctypes.util import find_library
>>> find_library("m")
'libm.so.6'
>>> find_library("c")
'libc.so.6'
>>> find_library("bz2")
'libbz2.so.1.0'
>>>
@end example

On macOS, @code{find_library()} tries several predefined naming schemes and paths
to locate the library, and returns a full pathname if successful:

@example
>>> from ctypes.util import find_library
>>> find_library("c")
'/usr/lib/libc.dylib'
>>> find_library("m")
'/usr/lib/libm.dylib'
>>> find_library("bz2")
'/usr/lib/libbz2.dylib'
>>> find_library("AGL")
'/System/Library/Frameworks/AGL.framework/AGL'
>>>
@end example

On Windows, @code{find_library()} searches along the system search path, and
returns the full pathname, but since there is no predefined naming scheme a call
like @code{find_library("c")} will fail and return @code{None}.

If wrapping a shared library with @ref{29,,ctypes}, it @emph{may} be better to determine
the shared library name at development time, and hardcode that into the wrapper
module instead of using @code{find_library()} to locate the library at runtime.

@node Loading shared libraries,Foreign functions,Finding shared libraries,ctypes reference
@anchor{library/ctypes ctypes-loading-shared-libraries}@anchor{279b}@anchor{library/ctypes loading-shared-libraries}@anchor{279c}
@subsubsection Loading shared libraries


There are several ways to load shared libraries into the Python process.  One
way is to instantiate one of the following classes:

@geindex CDLL (class in ctypes)
@anchor{library/ctypes ctypes CDLL}@anchor{4ed}
@deffn {Class} ctypes.CDLL (name, mode=DEFAULT_MODE, handle=None, use_errno=False, use_last_error=False, winmode=None)

Instances of this class represent loaded shared libraries. Functions in these
libraries use the standard C calling convention, and are assumed to return
int.

On Windows creating a @ref{4ed,,CDLL} instance may fail even if the DLL name
exists. When a dependent DLL of the loaded DLL is not found, a
@ref{2a1,,OSError} error is raised with the message @emph{“[WinError 126] The
specified module could not be found”.} This error message does not contain
the name of the missing DLL because the Windows API does not return this
information making this error hard to diagnose. To resolve this error and
determine which DLL is not found, you need to find the list of dependent
DLLs and determine which one is not found using Windows debugging and
tracing tools.
@end deffn


@subsubheading See also


Microsoft DUMPBIN tool@footnote{https://docs.microsoft.com/cpp/build/reference/dependents}
– A tool to find DLL dependents.


@geindex OleDLL (class in ctypes)
@anchor{library/ctypes ctypes OleDLL}@anchor{279d}
@deffn {Class} ctypes.OleDLL (name, mode=DEFAULT_MODE, handle=None, use_errno=False, use_last_error=False, winmode=None)

Windows only: Instances of this class represent loaded shared libraries,
functions in these libraries use the @code{stdcall} calling convention, and are
assumed to return the windows specific @ref{279e,,HRESULT} code.  @ref{279e,,HRESULT}
values contain information specifying whether the function call failed or
succeeded, together with additional error code.  If the return value signals a
failure, an @ref{2a1,,OSError} is automatically raised.


Changed in version 3.3: @ref{bea,,WindowsError} used to be raised.

@end deffn

@geindex WinDLL (class in ctypes)
@anchor{library/ctypes ctypes WinDLL}@anchor{279f}
@deffn {Class} ctypes.WinDLL (name, mode=DEFAULT_MODE, handle=None, use_errno=False, use_last_error=False, winmode=None)

Windows only: Instances of this class represent loaded shared libraries,
functions in these libraries use the @code{stdcall} calling convention, and are
assumed to return int by default.

On Windows CE only the standard calling convention is used, for convenience the
@ref{279f,,WinDLL} and @ref{279d,,OleDLL} use the standard calling convention on this
platform.
@end deffn

The Python @ref{7ae,,global interpreter lock} is released before calling any
function exported by these libraries, and reacquired afterwards.

@geindex PyDLL (class in ctypes)
@anchor{library/ctypes ctypes PyDLL}@anchor{27a0}
@deffn {Class} ctypes.PyDLL (name, mode=DEFAULT_MODE, handle=None)

Instances of this class behave like @ref{4ed,,CDLL} instances, except that the
Python GIL is @emph{not} released during the function call, and after the function
execution the Python error flag is checked. If the error flag is set, a Python
exception is raised.

Thus, this is only useful to call Python C api functions directly.
@end deffn

All these classes can be instantiated by calling them with at least one
argument, the pathname of the shared library.  If you have an existing handle to
an already loaded shared library, it can be passed as the @code{handle} named
parameter, otherwise the underlying platforms @code{dlopen} or @code{LoadLibrary}
function is used to load the library into the process, and to get a handle to
it.

The @emph{mode} parameter can be used to specify how the library is loaded.  For
details, consult the @code{dlopen(3)} manpage.  On Windows, @emph{mode} is
ignored.  On posix systems, RTLD_NOW is always added, and is not
configurable.

The @emph{use_errno} parameter, when set to true, enables a ctypes mechanism that
allows accessing the system @ref{77,,errno} error number in a safe way.
@ref{29,,ctypes} maintains a thread-local copy of the systems @ref{77,,errno}
variable; if you call foreign functions created with @code{use_errno=True} then the
@ref{77,,errno} value before the function call is swapped with the ctypes private
copy, the same happens immediately after the function call.

The function @ref{27a1,,ctypes.get_errno()} returns the value of the ctypes private
copy, and the function @ref{27a2,,ctypes.set_errno()} changes the ctypes private copy
to a new value and returns the former value.

The @emph{use_last_error} parameter, when set to true, enables the same mechanism for
the Windows error code which is managed by the @ref{2770,,GetLastError()} and
@code{SetLastError()} Windows API functions; @ref{27a3,,ctypes.get_last_error()} and
@ref{27a4,,ctypes.set_last_error()} are used to request and change the ctypes private
copy of the windows error code.

The @emph{winmode} parameter is used on Windows to specify how the library is loaded
(since @emph{mode} is ignored). It takes any value that is valid for the Win32 API
@code{LoadLibraryEx} flags parameter. When omitted, the default is to use the flags
that result in the most secure DLL load to avoiding issues such as DLL
hijacking. Passing the full path to the DLL is the safest way to ensure the
correct library and dependencies are loaded.


Changed in version 3.8: Added @emph{winmode} parameter.



@deffn {Data} ctypes.RTLD_GLOBAL

Flag to use as @emph{mode} parameter.  On platforms where this flag is not available,
it is defined as the integer zero.
@end deffn


@deffn {Data} ctypes.RTLD_LOCAL

Flag to use as @emph{mode} parameter.  On platforms where this is not available, it
is the same as @emph{RTLD_GLOBAL}.
@end deffn


@deffn {Data} ctypes.DEFAULT_MODE

The default mode which is used to load shared libraries.  On OSX 10.3, this is
@emph{RTLD_GLOBAL}, otherwise it is the same as @emph{RTLD_LOCAL}.
@end deffn

Instances of these classes have no public methods.  Functions exported by the
shared library can be accessed as attributes or by index.  Please note that
accessing the function through an attribute caches the result and therefore
accessing it repeatedly returns the same object each time.  On the other hand,
accessing it through an index returns a new object each time:

@example
>>> from ctypes import CDLL
>>> libc = CDLL("libc.so.6")  # On Linux
>>> libc.time == libc.time
True
>>> libc['time'] == libc['time']
False
@end example

The following public attributes are available, their name starts with an
underscore to not clash with exported function names:

@geindex _handle (ctypes.PyDLL attribute)
@anchor{library/ctypes ctypes PyDLL _handle}@anchor{27a5}
@deffn {Attribute} PyDLL._handle

The system handle used to access the library.
@end deffn

@geindex _name (ctypes.PyDLL attribute)
@anchor{library/ctypes ctypes PyDLL _name}@anchor{27a6}
@deffn {Attribute} PyDLL._name

The name of the library passed in the constructor.
@end deffn

Shared libraries can also be loaded by using one of the prefabricated objects,
which are instances of the @ref{27a7,,LibraryLoader} class, either by calling the
@code{LoadLibrary()} method, or by retrieving the library as attribute of the
loader instance.

@geindex LibraryLoader (class in ctypes)
@anchor{library/ctypes ctypes LibraryLoader}@anchor{27a7}
@deffn {Class} ctypes.LibraryLoader (dlltype)

Class which loads shared libraries.  @emph{dlltype} should be one of the
@ref{4ed,,CDLL}, @ref{27a0,,PyDLL}, @ref{279f,,WinDLL}, or @ref{279d,,OleDLL} types.

@code{__getattr__()} has special behavior: It allows loading a shared library by
accessing it as attribute of a library loader instance.  The result is cached,
so repeated attribute accesses return the same library each time.

@geindex LoadLibrary() (ctypes.LibraryLoader method)
@anchor{library/ctypes ctypes LibraryLoader LoadLibrary}@anchor{27a8}
@deffn {Method} LoadLibrary (name)

Load a shared library into the process and return it.  This method always
returns a new instance of the library.
@end deffn
@end deffn

These prefabricated library loaders are available:


@deffn {Data} ctypes.cdll

Creates @ref{4ed,,CDLL} instances.
@end deffn


@deffn {Data} ctypes.windll

Windows only: Creates @ref{279f,,WinDLL} instances.
@end deffn


@deffn {Data} ctypes.oledll

Windows only: Creates @ref{279d,,OleDLL} instances.
@end deffn


@deffn {Data} ctypes.pydll

Creates @ref{27a0,,PyDLL} instances.
@end deffn

For accessing the C Python api directly, a ready-to-use Python shared library
object is available:


@deffn {Data} ctypes.pythonapi

An instance of @ref{27a0,,PyDLL} that exposes Python C API functions as
attributes.  Note that all these functions are assumed to return C
int, which is of course not always the truth, so you have to assign
the correct @code{restype} attribute to use these functions.
@end deffn


Loading a library through any of these objects raises an
@ref{1130,,auditing event} @code{ctypes.dlopen} with string argument
@code{name}, the name used to load the library.



Accessing a function on a loaded library raises an auditing event
@code{ctypes.dlsym} with arguments @code{library} (the library object) and @code{name}
(the symbol’s name as a string or integer).



In cases when only the library handle is available rather than the object,
accessing a function raises an auditing event @code{ctypes.dlsym/handle} with
arguments @code{handle} (the raw library handle) and @code{name}.


@node Foreign functions,Function prototypes,Loading shared libraries,ctypes reference
@anchor{library/ctypes ctypes-foreign-functions}@anchor{27a9}@anchor{library/ctypes foreign-functions}@anchor{27aa}
@subsubsection Foreign functions


As explained in the previous section, foreign functions can be accessed as
attributes of loaded shared libraries.  The function objects created in this way
by default accept any number of arguments, accept any ctypes data instances as
arguments, and return the default result type specified by the library loader.
They are instances of a private class:

@geindex _FuncPtr (class in ctypes)
@anchor{library/ctypes ctypes _FuncPtr}@anchor{27ab}
@deffn {Class} ctypes._FuncPtr

Base class for C callable foreign functions.

Instances of foreign functions are also C compatible data types; they
represent C function pointers.

This behavior can be customized by assigning to special attributes of the
foreign function object.

@geindex restype (ctypes._FuncPtr attribute)
@anchor{library/ctypes ctypes _FuncPtr restype}@anchor{27ac}
@deffn {Attribute} restype

Assign a ctypes type to specify the result type of the foreign function.
Use @code{None} for void, a function not returning anything.

It is possible to assign a callable Python object that is not a ctypes
type, in this case the function is assumed to return a C int, and
the callable will be called with this integer, allowing further
processing or error checking.  Using this is deprecated, for more flexible
post processing or error checking use a ctypes data type as
@ref{27ac,,restype} and assign a callable to the @ref{27ad,,errcheck} attribute.
@end deffn

@geindex argtypes (ctypes._FuncPtr attribute)
@anchor{library/ctypes ctypes _FuncPtr argtypes}@anchor{27ae}
@deffn {Attribute} argtypes

Assign a tuple of ctypes types to specify the argument types that the
function accepts.  Functions using the @code{stdcall} calling convention can
only be called with the same number of arguments as the length of this
tuple; functions using the C calling convention accept additional,
unspecified arguments as well.

When a foreign function is called, each actual argument is passed to the
@code{from_param()} class method of the items in the @ref{27ae,,argtypes}
tuple, this method allows adapting the actual argument to an object that
the foreign function accepts.  For example, a @ref{2765,,c_char_p} item in
the @ref{27ae,,argtypes} tuple will convert a string passed as argument into
a bytes object using ctypes conversion rules.

New: It is now possible to put items in argtypes which are not ctypes
types, but each item must have a @code{from_param()} method which returns a
value usable as argument (integer, string, ctypes instance).  This allows
defining adapters that can adapt custom objects as function parameters.
@end deffn

@geindex errcheck (ctypes._FuncPtr attribute)
@anchor{library/ctypes ctypes _FuncPtr errcheck}@anchor{27ad}
@deffn {Attribute} errcheck

Assign a Python function or another callable to this attribute. The
callable will be called with three or more arguments:


@deffn {Function} callable (result, func, arguments)

@emph{result} is what the foreign function returns, as specified by the
@code{restype} attribute.

@emph{func} is the foreign function object itself, this allows reusing the
same callable object to check or post process the results of several
functions.

@emph{arguments} is a tuple containing the parameters originally passed to
the function call, this allows specializing the behavior on the
arguments used.
@end deffn

The object that this function returns will be returned from the
foreign function call, but it can also check the result value
and raise an exception if the foreign function call failed.
@end deffn
@end deffn

@geindex ArgumentError
@anchor{library/ctypes ctypes ArgumentError}@anchor{27af}
@deffn {Exception} ctypes.ArgumentError

This exception is raised when a foreign function call cannot convert one of the
passed arguments.
@end deffn


On Windows, when a foreign function call raises a system exception (for
example, due to an access violation), it will be captured and replaced with
a suitable Python exception. Further, an auditing event
@code{ctypes.seh_exception} with argument @code{code} will be raised, allowing an
audit hook to replace the exception with its own.



Some ways to invoke foreign function calls may raise an auditing event
@code{ctypes.call_function} with arguments @code{function pointer} and @code{arguments}.


@node Function prototypes,Utility functions,Foreign functions,ctypes reference
@anchor{library/ctypes ctypes-function-prototypes}@anchor{27b0}@anchor{library/ctypes function-prototypes}@anchor{27b1}
@subsubsection Function prototypes


Foreign functions can also be created by instantiating function prototypes.
Function prototypes are similar to function prototypes in C; they describe a
function (return type, argument types, calling convention) without defining an
implementation.  The factory functions must be called with the desired result
type and the argument types of the function, and can be used as decorator
factories, and as such, be applied to functions through the @code{@@wrapper} syntax.
See @ref{278a,,Callback functions} for examples.

@geindex CFUNCTYPE() (in module ctypes)
@anchor{library/ctypes ctypes CFUNCTYPE}@anchor{278b}
@deffn {Function} ctypes.CFUNCTYPE (restype, *argtypes, use_errno=False, use_last_error=False)

The returned function prototype creates functions that use the standard C
calling convention.  The function will release the GIL during the call.  If
@emph{use_errno} is set to true, the ctypes private copy of the system
@ref{77,,errno} variable is exchanged with the real @ref{77,,errno} value before
and after the call; @emph{use_last_error} does the same for the Windows error
code.
@end deffn

@geindex WINFUNCTYPE() (in module ctypes)
@anchor{library/ctypes ctypes WINFUNCTYPE}@anchor{278c}
@deffn {Function} ctypes.WINFUNCTYPE (restype, *argtypes, use_errno=False, use_last_error=False)

Windows only: The returned function prototype creates functions that use the
@code{stdcall} calling convention, except on Windows CE where
@ref{278c,,WINFUNCTYPE()} is the same as @ref{278b,,CFUNCTYPE()}.  The function will
release the GIL during the call.  @emph{use_errno} and @emph{use_last_error} have the
same meaning as above.
@end deffn

@geindex PYFUNCTYPE() (in module ctypes)
@anchor{library/ctypes ctypes PYFUNCTYPE}@anchor{27b2}
@deffn {Function} ctypes.PYFUNCTYPE (restype, *argtypes)

The returned function prototype creates functions that use the Python calling
convention.  The function will @emph{not} release the GIL during the call.
@end deffn

Function prototypes created by these factory functions can be instantiated in
different ways, depending on the type and number of the parameters in the call:

@quotation


@deffn {Function} prototype (address)

Returns a foreign function at the specified address which must be an integer.
@end deffn


@deffn {Function} prototype (callable)

Create a C callable function (a callback function) from a Python @emph{callable}.
@end deffn


@deffn {Function} prototype (func_spec[, paramflags])

Returns a foreign function exported by a shared library. @emph{func_spec} must
be a 2-tuple @code{(name_or_ordinal, library)}. The first item is the name of
the exported function as string, or the ordinal of the exported function
as small integer.  The second item is the shared library instance.
@end deffn


@deffn {Function} prototype (vtbl_index, name[, paramflags[, iid]])

Returns a foreign function that will call a COM method. @emph{vtbl_index} is
the index into the virtual function table, a small non-negative
integer. @emph{name} is name of the COM method. @emph{iid} is an optional pointer to
the interface identifier which is used in extended error reporting.

COM methods use a special calling convention: They require a pointer to
the COM interface as first argument, in addition to those parameters that
are specified in the @code{argtypes} tuple.
@end deffn

The optional @emph{paramflags} parameter creates foreign function wrappers with much
more functionality than the features described above.

@emph{paramflags} must be a tuple of the same length as @code{argtypes}.

Each item in this tuple contains further information about a parameter, it must
be a tuple containing one, two, or three items.

The first item is an integer containing a combination of direction
flags for the parameter:

@quotation


@table @asis

@item 1

Specifies an input parameter to the function.

@item 2

Output parameter.  The foreign function fills in a value.

@item 4

Input parameter which defaults to the integer zero.
@end table
@end quotation

The optional second item is the parameter name as string.  If this is specified,
the foreign function can be called with named parameters.

The optional third item is the default value for this parameter.
@end quotation

This example demonstrates how to wrap the Windows @code{MessageBoxW} function so
that it supports default parameters and named arguments. The C declaration from
the windows header file is this:

@example
WINUSERAPI int WINAPI
MessageBoxW(
    HWND hWnd,
    LPCWSTR lpText,
    LPCWSTR lpCaption,
    UINT uType);
@end example

Here is the wrapping with @ref{29,,ctypes}:

@example
>>> from ctypes import c_int, WINFUNCTYPE, windll
>>> from ctypes.wintypes import HWND, LPCWSTR, UINT
>>> prototype = WINFUNCTYPE(c_int, HWND, LPCWSTR, LPCWSTR, UINT)
>>> paramflags = (1, "hwnd", 0), (1, "text", "Hi"), (1, "caption", "Hello from ctypes"), (1, "flags", 0)
>>> MessageBox = prototype(("MessageBoxW", windll.user32), paramflags)
@end example

The @code{MessageBox} foreign function can now be called in these ways:

@example
>>> MessageBox()
>>> MessageBox(text="Spam, spam, spam")
>>> MessageBox(flags=2, text="foo bar")
@end example

A second example demonstrates output parameters.  The win32 @code{GetWindowRect}
function retrieves the dimensions of a specified window by copying them into
@code{RECT} structure that the caller has to supply.  Here is the C declaration:

@example
WINUSERAPI BOOL WINAPI
GetWindowRect(
     HWND hWnd,
     LPRECT lpRect);
@end example

Here is the wrapping with @ref{29,,ctypes}:

@example
>>> from ctypes import POINTER, WINFUNCTYPE, windll, WinError
>>> from ctypes.wintypes import BOOL, HWND, RECT
>>> prototype = WINFUNCTYPE(BOOL, HWND, POINTER(RECT))
>>> paramflags = (1, "hwnd"), (2, "lprect")
>>> GetWindowRect = prototype(("GetWindowRect", windll.user32), paramflags)
>>>
@end example

Functions with output parameters will automatically return the output parameter
value if there is a single one, or a tuple containing the output parameter
values when there are more than one, so the GetWindowRect function now returns a
RECT instance, when called.

Output parameters can be combined with the @code{errcheck} protocol to do
further output processing and error checking.  The win32 @code{GetWindowRect} api
function returns a @code{BOOL} to signal success or failure, so this function could
do the error checking, and raises an exception when the api call failed:

@example
>>> def errcheck(result, func, args):
...     if not result:
...         raise WinError()
...     return args
...
>>> GetWindowRect.errcheck = errcheck
>>>
@end example

If the @code{errcheck} function returns the argument tuple it receives
unchanged, @ref{29,,ctypes} continues the normal processing it does on the output
parameters.  If you want to return a tuple of window coordinates instead of a
@code{RECT} instance, you can retrieve the fields in the function and return them
instead, the normal processing will no longer take place:

@example
>>> def errcheck(result, func, args):
...     if not result:
...         raise WinError()
...     rc = args[1]
...     return rc.left, rc.top, rc.bottom, rc.right
...
>>> GetWindowRect.errcheck = errcheck
>>>
@end example

@node Utility functions,Data types,Function prototypes,ctypes reference
@anchor{library/ctypes ctypes-utility-functions}@anchor{27b3}@anchor{library/ctypes utility-functions}@anchor{27b4}
@subsubsection Utility functions


@geindex addressof() (in module ctypes)
@anchor{library/ctypes ctypes addressof}@anchor{27b5}
@deffn {Function} ctypes.addressof (obj)

Returns the address of the memory buffer as integer.  @emph{obj} must be an
instance of a ctypes type.

Raises an @ref{1130,,auditing event} @code{ctypes.addressof} with argument @code{obj}.
@end deffn

@geindex alignment() (in module ctypes)
@anchor{library/ctypes ctypes alignment}@anchor{27b6}
@deffn {Function} ctypes.alignment (obj_or_type)

Returns the alignment requirements of a ctypes type. @emph{obj_or_type} must be a
ctypes type or instance.
@end deffn

@geindex byref() (in module ctypes)
@anchor{library/ctypes ctypes byref}@anchor{2773}
@deffn {Function} ctypes.byref (obj[, offset])

Returns a light-weight pointer to @emph{obj}, which must be an instance of a
ctypes type.  @emph{offset} defaults to zero, and must be an integer that will be
added to the internal pointer value.

@code{byref(obj, offset)} corresponds to this C code:

@example
(((char *)&obj) + offset)
@end example

The returned object can only be used as a foreign function call parameter.
It behaves similar to @code{pointer(obj)}, but the construction is a lot faster.
@end deffn

@geindex cast() (in module ctypes)
@anchor{library/ctypes ctypes cast}@anchor{12db}
@deffn {Function} ctypes.cast (obj, type)

This function is similar to the cast operator in C. It returns a new instance
of @emph{type} which points to the same memory block as @emph{obj}.  @emph{type} must be a
pointer type, and @emph{obj} must be an object that can be interpreted as a
pointer.
@end deffn

@geindex create_string_buffer() (in module ctypes)
@anchor{library/ctypes ctypes create_string_buffer}@anchor{2767}
@deffn {Function} ctypes.create_string_buffer (init_or_size, size=None)

This function creates a mutable character buffer. The returned object is a
ctypes array of @ref{2759,,c_char}.

@emph{init_or_size} must be an integer which specifies the size of the array, or a
bytes object which will be used to initialize the array items.

If a bytes object is specified as first argument, the buffer is made one item
larger than its length so that the last element in the array is a NUL
termination character. An integer can be passed as second argument which allows
specifying the size of the array if the length of the bytes should not be used.

Raises an @ref{1130,,auditing event} @code{ctypes.create_string_buffer} with arguments @code{init}, @code{size}.
@end deffn

@geindex create_unicode_buffer() (in module ctypes)
@anchor{library/ctypes ctypes create_unicode_buffer}@anchor{126f}
@deffn {Function} ctypes.create_unicode_buffer (init_or_size, size=None)

This function creates a mutable unicode character buffer. The returned object is
a ctypes array of @ref{275a,,c_wchar}.

@emph{init_or_size} must be an integer which specifies the size of the array, or a
string which will be used to initialize the array items.

If a string is specified as first argument, the buffer is made one item
larger than the length of the string so that the last element in the array is a
NUL termination character. An integer can be passed as second argument which
allows specifying the size of the array if the length of the string should not
be used.

Raises an @ref{1130,,auditing event} @code{ctypes.create_unicode_buffer} with arguments @code{init}, @code{size}.
@end deffn

@geindex DllCanUnloadNow() (in module ctypes)
@anchor{library/ctypes ctypes DllCanUnloadNow}@anchor{27b7}
@deffn {Function} ctypes.DllCanUnloadNow ()

Windows only: This function is a hook which allows implementing in-process
COM servers with ctypes.  It is called from the DllCanUnloadNow function that
the _ctypes extension dll exports.
@end deffn

@geindex DllGetClassObject() (in module ctypes)
@anchor{library/ctypes ctypes DllGetClassObject}@anchor{27b8}
@deffn {Function} ctypes.DllGetClassObject ()

Windows only: This function is a hook which allows implementing in-process
COM servers with ctypes.  It is called from the DllGetClassObject function
that the @code{_ctypes} extension dll exports.
@end deffn

@geindex find_library() (in module ctypes.util)
@anchor{library/ctypes ctypes util find_library}@anchor{8a0}
@deffn {Function} ctypes.util.find_library (name)

Try to find a library and return a pathname.  @emph{name} is the library name
without any prefix like @code{lib}, suffix like @code{.so}, @code{.dylib} or version
number (this is the form used for the posix linker option @code{-l}).  If
no library can be found, returns @code{None}.

The exact functionality is system dependent.
@end deffn

@geindex find_msvcrt() (in module ctypes.util)
@anchor{library/ctypes ctypes util find_msvcrt}@anchor{27b9}
@deffn {Function} ctypes.util.find_msvcrt ()

Windows only: return the filename of the VC runtime library used by Python,
and by the extension modules.  If the name of the library cannot be
determined, @code{None} is returned.

If you need to free memory, for example, allocated by an extension module
with a call to the @code{free(void *)}, it is important that you use the
function in the same library that allocated the memory.
@end deffn

@geindex FormatError() (in module ctypes)
@anchor{library/ctypes ctypes FormatError}@anchor{27ba}
@deffn {Function} ctypes.FormatError ([code])

Windows only: Returns a textual description of the error code @emph{code}.  If no
error code is specified, the last error code is used by calling the Windows
api function GetLastError.
@end deffn

@geindex GetLastError() (in module ctypes)
@anchor{library/ctypes ctypes GetLastError}@anchor{2770}
@deffn {Function} ctypes.GetLastError ()

Windows only: Returns the last error code set by Windows in the calling thread.
This function calls the Windows @cite{GetLastError()} function directly,
it does not return the ctypes-private copy of the error code.
@end deffn

@geindex get_errno() (in module ctypes)
@anchor{library/ctypes ctypes get_errno}@anchor{27a1}
@deffn {Function} ctypes.get_errno ()

Returns the current value of the ctypes-private copy of the system
@ref{77,,errno} variable in the calling thread.

Raises an @ref{1130,,auditing event} @code{ctypes.get_errno} with no arguments.
@end deffn

@geindex get_last_error() (in module ctypes)
@anchor{library/ctypes ctypes get_last_error}@anchor{27a3}
@deffn {Function} ctypes.get_last_error ()

Windows only: returns the current value of the ctypes-private copy of the system
@code{LastError} variable in the calling thread.

Raises an @ref{1130,,auditing event} @code{ctypes.get_last_error} with no arguments.
@end deffn

@geindex memmove() (in module ctypes)
@anchor{library/ctypes ctypes memmove}@anchor{27bb}
@deffn {Function} ctypes.memmove (dst, src, count)

Same as the standard C memmove library function: copies @emph{count} bytes from
@emph{src} to @emph{dst}. @emph{dst} and @emph{src} must be integers or ctypes instances that can
be converted to pointers.
@end deffn

@geindex memset() (in module ctypes)
@anchor{library/ctypes ctypes memset}@anchor{27bc}
@deffn {Function} ctypes.memset (dst, c, count)

Same as the standard C memset library function: fills the memory block at
address @emph{dst} with @emph{count} bytes of value @emph{c}. @emph{dst} must be an integer
specifying an address, or a ctypes instance.
@end deffn

@geindex POINTER() (in module ctypes)
@anchor{library/ctypes ctypes POINTER}@anchor{2784}
@deffn {Function} ctypes.POINTER (type)

This factory function creates and returns a new ctypes pointer type. Pointer
types are cached and reused internally, so calling this function repeatedly is
cheap. @emph{type} must be a ctypes type.
@end deffn

@geindex pointer() (in module ctypes)
@anchor{library/ctypes ctypes pointer}@anchor{2774}
@deffn {Function} ctypes.pointer (obj)

This function creates a new pointer instance, pointing to @emph{obj}. The returned
object is of the type @code{POINTER(type(obj))}.

Note: If you just want to pass a pointer to an object to a foreign function
call, you should use @code{byref(obj)} which is much faster.
@end deffn

@geindex resize() (in module ctypes)
@anchor{library/ctypes ctypes resize}@anchor{2796}
@deffn {Function} ctypes.resize (obj, size)

This function resizes the internal memory buffer of @emph{obj}, which must be an
instance of a ctypes type.  It is not possible to make the buffer smaller
than the native size of the objects type, as given by @code{sizeof(type(obj))},
but it is possible to enlarge the buffer.
@end deffn

@geindex set_errno() (in module ctypes)
@anchor{library/ctypes ctypes set_errno}@anchor{27a2}
@deffn {Function} ctypes.set_errno (value)

Set the current value of the ctypes-private copy of the system @ref{77,,errno}
variable in the calling thread to @emph{value} and return the previous value.

Raises an @ref{1130,,auditing event} @code{ctypes.set_errno} with argument @code{errno}.
@end deffn

@geindex set_last_error() (in module ctypes)
@anchor{library/ctypes ctypes set_last_error}@anchor{27a4}
@deffn {Function} ctypes.set_last_error (value)

Windows only: set the current value of the ctypes-private copy of the system
@code{LastError} variable in the calling thread to @emph{value} and return the
previous value.

Raises an @ref{1130,,auditing event} @code{ctypes.set_last_error} with argument @code{error}.
@end deffn

@geindex sizeof() (in module ctypes)
@anchor{library/ctypes ctypes sizeof}@anchor{27bd}
@deffn {Function} ctypes.sizeof (obj_or_type)

Returns the size in bytes of a ctypes type or instance memory buffer.
Does the same as the C @code{sizeof} operator.
@end deffn

@geindex string_at() (in module ctypes)
@anchor{library/ctypes ctypes string_at}@anchor{27be}
@deffn {Function} ctypes.string_at (address, size=- 1)

This function returns the C string starting at memory address @emph{address} as a bytes
object. If size is specified, it is used as size, otherwise the string is assumed
to be zero-terminated.

Raises an @ref{1130,,auditing event} @code{ctypes.string_at} with arguments @code{address}, @code{size}.
@end deffn

@geindex WinError() (in module ctypes)
@anchor{library/ctypes ctypes WinError}@anchor{27bf}
@deffn {Function} ctypes.WinError (code=None, descr=None)

Windows only: this function is probably the worst-named thing in ctypes. It
creates an instance of OSError.  If @emph{code} is not specified,
@code{GetLastError} is called to determine the error code. If @emph{descr} is not
specified, @ref{27ba,,FormatError()} is called to get a textual description of the
error.


Changed in version 3.3: An instance of @ref{bea,,WindowsError} used to be created.

@end deffn

@geindex wstring_at() (in module ctypes)
@anchor{library/ctypes ctypes wstring_at}@anchor{27c0}
@deffn {Function} ctypes.wstring_at (address, size=- 1)

This function returns the wide character string starting at memory address
@emph{address} as a string.  If @emph{size} is specified, it is used as the number of
characters of the string, otherwise the string is assumed to be
zero-terminated.

Raises an @ref{1130,,auditing event} @code{ctypes.wstring_at} with arguments @code{address}, @code{size}.
@end deffn

@node Data types,Fundamental data types<2>,Utility functions,ctypes reference
@anchor{library/ctypes ctypes-data-types}@anchor{27c1}@anchor{library/ctypes data-types}@anchor{27c2}
@subsubsection Data types


@geindex _CData (class in ctypes)
@anchor{library/ctypes ctypes _CData}@anchor{1371}
@deffn {Class} ctypes._CData

This non-public class is the common base class of all ctypes data types.
Among other things, all ctypes type instances contain a memory block that
hold C compatible data; the address of the memory block is returned by the
@ref{27b5,,addressof()} helper function. Another instance variable is exposed as
@ref{27c3,,_objects}; this contains other Python objects that need to be kept
alive in case the memory block contains pointers.

Common methods of ctypes data types, these are all class methods (to be
exact, they are methods of the @ref{17d3,,metaclass}):

@geindex from_buffer() (ctypes._CData method)
@anchor{library/ctypes ctypes _CData from_buffer}@anchor{27c4}
@deffn {Method} from_buffer (source[, offset])

This method returns a ctypes instance that shares the buffer of the
@emph{source} object.  The @emph{source} object must support the writeable buffer
interface.  The optional @emph{offset} parameter specifies an offset into the
source buffer in bytes; the default is zero.  If the source buffer is not
large enough a @ref{17c,,ValueError} is raised.

Raises an @ref{1130,,auditing event} @code{ctypes.cdata/buffer} with arguments @code{pointer}, @code{size}, @code{offset}.
@end deffn

@geindex from_buffer_copy() (ctypes._CData method)
@anchor{library/ctypes ctypes _CData from_buffer_copy}@anchor{27c5}
@deffn {Method} from_buffer_copy (source[, offset])

This method creates a ctypes instance, copying the buffer from the
@emph{source} object buffer which must be readable.  The optional @emph{offset}
parameter specifies an offset into the source buffer in bytes; the default
is zero.  If the source buffer is not large enough a @ref{17c,,ValueError} is
raised.

Raises an @ref{1130,,auditing event} @code{ctypes.cdata/buffer} with arguments @code{pointer}, @code{size}, @code{offset}.
@end deffn

@geindex from_address() (ctypes._CData method)
@anchor{library/ctypes ctypes _CData from_address}@anchor{27c6}
@deffn {Method} from_address (address)

This method returns a ctypes type instance using the memory specified by
@emph{address} which must be an integer.


This method, and others that indirectly call this method, raises an
@ref{1130,,auditing event} @code{ctypes.cdata} with argument
@code{address}.

@end deffn

@geindex from_param() (ctypes._CData method)
@anchor{library/ctypes ctypes _CData from_param}@anchor{27c7}
@deffn {Method} from_param (obj)

This method adapts @emph{obj} to a ctypes type.  It is called with the actual
object used in a foreign function call when the type is present in the
foreign function’s @code{argtypes} tuple; it must return an object that
can be used as a function call parameter.

All ctypes data types have a default implementation of this classmethod
that normally returns @emph{obj} if that is an instance of the type.  Some
types accept other objects as well.
@end deffn

@geindex in_dll() (ctypes._CData method)
@anchor{library/ctypes ctypes _CData in_dll}@anchor{27c8}
@deffn {Method} in_dll (library, name)

This method returns a ctypes type instance exported by a shared
library. @emph{name} is the name of the symbol that exports the data, @emph{library}
is the loaded shared library.
@end deffn

Common instance variables of ctypes data types:

@geindex _b_base_ (ctypes._CData attribute)
@anchor{library/ctypes ctypes _CData _b_base_}@anchor{27c9}
@deffn {Attribute} _b_base_

Sometimes ctypes data instances do not own the memory block they contain,
instead they share part of the memory block of a base object.  The
@ref{27c9,,_b_base_} read-only member is the root ctypes object that owns the
memory block.
@end deffn

@geindex _b_needsfree_ (ctypes._CData attribute)
@anchor{library/ctypes ctypes _CData _b_needsfree_}@anchor{27ca}
@deffn {Attribute} _b_needsfree_

This read-only variable is true when the ctypes data instance has
allocated the memory block itself, false otherwise.
@end deffn

@geindex _objects (ctypes._CData attribute)
@anchor{library/ctypes ctypes _CData _objects}@anchor{27c3}
@deffn {Attribute} _objects

This member is either @code{None} or a dictionary containing Python objects
that need to be kept alive so that the memory block contents is kept
valid.  This object is only exposed for debugging; never modify the
contents of this dictionary.
@end deffn
@end deffn

@node Fundamental data types<2>,Structured data types,Data types,ctypes reference
@anchor{library/ctypes ctypes-fundamental-data-types-2}@anchor{27cb}@anchor{library/ctypes id1}@anchor{27cc}
@subsubsection Fundamental data types


@geindex _SimpleCData (class in ctypes)
@anchor{library/ctypes ctypes _SimpleCData}@anchor{27cd}
@deffn {Class} ctypes._SimpleCData

This non-public class is the base class of all fundamental ctypes data
types. It is mentioned here because it contains the common attributes of the
fundamental ctypes data types.  @ref{27cd,,_SimpleCData} is a subclass of
@ref{1371,,_CData}, so it inherits their methods and attributes. ctypes data
types that are not and do not contain pointers can now be pickled.

Instances have a single attribute:

@geindex value (ctypes._SimpleCData attribute)
@anchor{library/ctypes ctypes _SimpleCData value}@anchor{27ce}
@deffn {Attribute} value

This attribute contains the actual value of the instance. For integer and
pointer types, it is an integer, for character types, it is a single
character bytes object or string, for character pointer types it is a
Python bytes object or string.

When the @code{value} attribute is retrieved from a ctypes instance, usually
a new object is returned each time.  @ref{29,,ctypes} does @emph{not} implement
original object return, always a new object is constructed.  The same is
true for all other ctypes object instances.
@end deffn
@end deffn

Fundamental data types, when returned as foreign function call results, or, for
example, by retrieving structure field members or array items, are transparently
converted to native Python types.  In other words, if a foreign function has a
@code{restype} of @ref{2765,,c_char_p}, you will always receive a Python bytes
object, @emph{not} a @ref{2765,,c_char_p} instance.

@c XXX above is false, it actually returns a Unicode string

Subclasses of fundamental data types do @emph{not} inherit this behavior. So, if a
foreign functions @code{restype} is a subclass of @ref{2766,,c_void_p}, you will
receive an instance of this subclass from the function call. Of course, you can
get the value of the pointer by accessing the @code{value} attribute.

These are the fundamental ctypes data types:

@geindex c_byte (class in ctypes)
@anchor{library/ctypes ctypes c_byte}@anchor{275b}
@deffn {Class} ctypes.c_byte

Represents the C signed  char datatype, and interprets the value as
small integer.  The constructor accepts an optional integer initializer; no
overflow checking is done.
@end deffn

@geindex c_char (class in ctypes)
@anchor{library/ctypes ctypes c_char}@anchor{2759}
@deffn {Class} ctypes.c_char

Represents the C char datatype, and interprets the value as a single
character.  The constructor accepts an optional string initializer, the
length of the string must be exactly one character.
@end deffn

@geindex c_char_p (class in ctypes)
@anchor{library/ctypes ctypes c_char_p}@anchor{2765}
@deffn {Class} ctypes.c_char_p

Represents the C char* datatype when it points to a zero-terminated
string.  For a general character pointer that may also point to binary data,
@code{POINTER(c_char)} must be used.  The constructor accepts an integer
address, or a bytes object.
@end deffn

@geindex c_double (class in ctypes)
@anchor{library/ctypes ctypes c_double}@anchor{1172}
@deffn {Class} ctypes.c_double

Represents the C double datatype.  The constructor accepts an
optional float initializer.
@end deffn

@geindex c_longdouble (class in ctypes)
@anchor{library/ctypes ctypes c_longdouble}@anchor{1173}
@deffn {Class} ctypes.c_longdouble

Represents the C long  double datatype.  The constructor accepts an
optional float initializer.  On platforms where @code{sizeof(long double) ==
sizeof(double)} it is an alias to @ref{1172,,c_double}.
@end deffn

@geindex c_float (class in ctypes)
@anchor{library/ctypes ctypes c_float}@anchor{2764}
@deffn {Class} ctypes.c_float

Represents the C float datatype.  The constructor accepts an
optional float initializer.
@end deffn

@geindex c_int (class in ctypes)
@anchor{library/ctypes ctypes c_int}@anchor{274e}
@deffn {Class} ctypes.c_int

Represents the C signed  int datatype.  The constructor accepts an
optional integer initializer; no overflow checking is done.  On platforms
where @code{sizeof(int) == sizeof(long)} it is an alias to @ref{274f,,c_long}.
@end deffn

@geindex c_int8 (class in ctypes)
@anchor{library/ctypes ctypes c_int8}@anchor{27cf}
@deffn {Class} ctypes.c_int8

Represents the C 8-bit signed  int datatype.  Usually an alias for
@ref{275b,,c_byte}.
@end deffn

@geindex c_int16 (class in ctypes)
@anchor{library/ctypes ctypes c_int16}@anchor{27d0}
@deffn {Class} ctypes.c_int16

Represents the C 16-bit signed  int datatype.  Usually an alias for
@ref{275d,,c_short}.
@end deffn

@geindex c_int32 (class in ctypes)
@anchor{library/ctypes ctypes c_int32}@anchor{27d1}
@deffn {Class} ctypes.c_int32

Represents the C 32-bit signed  int datatype.  Usually an alias for
@ref{274e,,c_int}.
@end deffn

@geindex c_int64 (class in ctypes)
@anchor{library/ctypes ctypes c_int64}@anchor{27d2}
@deffn {Class} ctypes.c_int64

Represents the C 64-bit signed  int datatype.  Usually an alias for
@ref{2761,,c_longlong}.
@end deffn

@geindex c_long (class in ctypes)
@anchor{library/ctypes ctypes c_long}@anchor{274f}
@deffn {Class} ctypes.c_long

Represents the C signed  long datatype.  The constructor accepts an
optional integer initializer; no overflow checking is done.
@end deffn

@geindex c_longlong (class in ctypes)
@anchor{library/ctypes ctypes c_longlong}@anchor{2761}
@deffn {Class} ctypes.c_longlong

Represents the C signed  long  long datatype.  The constructor accepts
an optional integer initializer; no overflow checking is done.
@end deffn

@geindex c_short (class in ctypes)
@anchor{library/ctypes ctypes c_short}@anchor{275d}
@deffn {Class} ctypes.c_short

Represents the C signed  short datatype.  The constructor accepts an
optional integer initializer; no overflow checking is done.
@end deffn

@geindex c_size_t (class in ctypes)
@anchor{library/ctypes ctypes c_size_t}@anchor{2763}
@deffn {Class} ctypes.c_size_t

Represents the C @code{size_t} datatype.
@end deffn

@geindex c_ssize_t (class in ctypes)
@anchor{library/ctypes ctypes c_ssize_t}@anchor{dee}
@deffn {Class} ctypes.c_ssize_t

Represents the C @code{ssize_t} datatype.


New in version 3.2.

@end deffn

@geindex c_ubyte (class in ctypes)
@anchor{library/ctypes ctypes c_ubyte}@anchor{275c}
@deffn {Class} ctypes.c_ubyte

Represents the C unsigned  char datatype, it interprets the value as
small integer.  The constructor accepts an optional integer initializer; no
overflow checking is done.
@end deffn

@geindex c_uint (class in ctypes)
@anchor{library/ctypes ctypes c_uint}@anchor{275f}
@deffn {Class} ctypes.c_uint

Represents the C unsigned  int datatype.  The constructor accepts an
optional integer initializer; no overflow checking is done.  On platforms
where @code{sizeof(int) == sizeof(long)} it is an alias for @ref{2760,,c_ulong}.
@end deffn

@geindex c_uint8 (class in ctypes)
@anchor{library/ctypes ctypes c_uint8}@anchor{27d3}
@deffn {Class} ctypes.c_uint8

Represents the C 8-bit unsigned  int datatype.  Usually an alias for
@ref{275c,,c_ubyte}.
@end deffn

@geindex c_uint16 (class in ctypes)
@anchor{library/ctypes ctypes c_uint16}@anchor{27d4}
@deffn {Class} ctypes.c_uint16

Represents the C 16-bit unsigned  int datatype.  Usually an alias for
@ref{275e,,c_ushort}.
@end deffn

@geindex c_uint32 (class in ctypes)
@anchor{library/ctypes ctypes c_uint32}@anchor{27d5}
@deffn {Class} ctypes.c_uint32

Represents the C 32-bit unsigned  int datatype.  Usually an alias for
@ref{275f,,c_uint}.
@end deffn

@geindex c_uint64 (class in ctypes)
@anchor{library/ctypes ctypes c_uint64}@anchor{27d6}
@deffn {Class} ctypes.c_uint64

Represents the C 64-bit unsigned  int datatype.  Usually an alias for
@ref{2762,,c_ulonglong}.
@end deffn

@geindex c_ulong (class in ctypes)
@anchor{library/ctypes ctypes c_ulong}@anchor{2760}
@deffn {Class} ctypes.c_ulong

Represents the C unsigned  long datatype.  The constructor accepts an
optional integer initializer; no overflow checking is done.
@end deffn

@geindex c_ulonglong (class in ctypes)
@anchor{library/ctypes ctypes c_ulonglong}@anchor{2762}
@deffn {Class} ctypes.c_ulonglong

Represents the C unsigned  long  long datatype.  The constructor
accepts an optional integer initializer; no overflow checking is done.
@end deffn

@geindex c_ushort (class in ctypes)
@anchor{library/ctypes ctypes c_ushort}@anchor{275e}
@deffn {Class} ctypes.c_ushort

Represents the C unsigned  short datatype.  The constructor accepts
an optional integer initializer; no overflow checking is done.
@end deffn

@geindex c_void_p (class in ctypes)
@anchor{library/ctypes ctypes c_void_p}@anchor{2766}
@deffn {Class} ctypes.c_void_p

Represents the C void* type.  The value is represented as integer.
The constructor accepts an optional integer initializer.
@end deffn

@geindex c_wchar (class in ctypes)
@anchor{library/ctypes ctypes c_wchar}@anchor{275a}
@deffn {Class} ctypes.c_wchar

Represents the C @code{wchar_t} datatype, and interprets the value as a
single character unicode string.  The constructor accepts an optional string
initializer, the length of the string must be exactly one character.
@end deffn

@geindex c_wchar_p (class in ctypes)
@anchor{library/ctypes ctypes c_wchar_p}@anchor{113e}
@deffn {Class} ctypes.c_wchar_p

Represents the C wchar_t* datatype, which must be a pointer to a
zero-terminated wide character string.  The constructor accepts an integer
address, or a string.
@end deffn

@geindex c_bool (class in ctypes)
@anchor{library/ctypes ctypes c_bool}@anchor{2758}
@deffn {Class} ctypes.c_bool

Represent the C bool datatype (more accurately, _Bool from
C99).  Its value can be @code{True} or @code{False}, and the constructor accepts any object
that has a truth value.
@end deffn

@geindex HRESULT (class in ctypes)
@anchor{library/ctypes ctypes HRESULT}@anchor{279e}
@deffn {Class} ctypes.HRESULT

Windows only: Represents a @code{HRESULT} value, which contains success or
error information for a function or method call.
@end deffn

@geindex py_object (class in ctypes)
@anchor{library/ctypes ctypes py_object}@anchor{27d7}
@deffn {Class} ctypes.py_object

Represents the C @ref{375,,PyObject}* datatype.  Calling this without an
argument creates a @code{NULL} @ref{375,,PyObject}* pointer.
@end deffn

The @code{ctypes.wintypes} module provides quite some other Windows specific
data types, for example @code{HWND}, @code{WPARAM}, or @code{DWORD}.  Some
useful structures like @code{MSG} or @code{RECT} are also defined.

@node Structured data types,Arrays and pointers,Fundamental data types<2>,ctypes reference
@anchor{library/ctypes ctypes-structured-data-types}@anchor{27d8}@anchor{library/ctypes structured-data-types}@anchor{27d9}
@subsubsection Structured data types


@geindex Union (class in ctypes)
@anchor{library/ctypes ctypes Union}@anchor{2778}
@deffn {Class} ctypes.Union (*args, **kw)

Abstract base class for unions in native byte order.
@end deffn

@geindex BigEndianStructure (class in ctypes)
@anchor{library/ctypes ctypes BigEndianStructure}@anchor{277b}
@deffn {Class} ctypes.BigEndianStructure (*args, **kw)

Abstract base class for structures in @emph{big endian} byte order.
@end deffn

@geindex LittleEndianStructure (class in ctypes)
@anchor{library/ctypes ctypes LittleEndianStructure}@anchor{277c}
@deffn {Class} ctypes.LittleEndianStructure (*args, **kw)

Abstract base class for structures in @emph{little endian} byte order.
@end deffn

Structures with non-native byte order cannot contain pointer type fields, or any
other data types containing pointer type fields.

@geindex Structure (class in ctypes)
@anchor{library/ctypes ctypes Structure}@anchor{2777}
@deffn {Class} ctypes.Structure (*args, **kw)

Abstract base class for structures in @emph{native} byte order.

Concrete structure and union types must be created by subclassing one of these
types, and at least define a @ref{27da,,_fields_} class variable. @ref{29,,ctypes} will
create @ref{17bc,,descriptor}s which allow reading and writing the fields by direct
attribute accesses.  These are the

@geindex _fields_ (ctypes.Structure attribute)
@anchor{library/ctypes ctypes Structure _fields_}@anchor{27da}
@deffn {Attribute} _fields_

A sequence defining the structure fields.  The items must be 2-tuples or
3-tuples.  The first item is the name of the field, the second item
specifies the type of the field; it can be any ctypes data type.

For integer type fields like @ref{274e,,c_int}, a third optional item can be
given.  It must be a small positive integer defining the bit width of the
field.

Field names must be unique within one structure or union.  This is not
checked, only one field can be accessed when names are repeated.

It is possible to define the @ref{27da,,_fields_} class variable @emph{after} the
class statement that defines the Structure subclass, this allows creating
data types that directly or indirectly reference themselves:

@example
class List(Structure):
    pass
List._fields_ = [("pnext", POINTER(List)),
                 ...
                ]
@end example

The @ref{27da,,_fields_} class variable must, however, be defined before the
type is first used (an instance is created, @ref{27bd,,sizeof()} is called on it,
and so on).  Later assignments to the @ref{27da,,_fields_} class variable will
raise an AttributeError.

It is possible to define sub-subclasses of structure types, they inherit
the fields of the base class plus the @ref{27da,,_fields_} defined in the
sub-subclass, if any.
@end deffn

@geindex _pack_ (ctypes.Structure attribute)
@anchor{library/ctypes ctypes Structure _pack_}@anchor{27db}
@deffn {Attribute} _pack_

An optional small integer that allows overriding the alignment of
structure fields in the instance.  @ref{27db,,_pack_} must already be defined
when @ref{27da,,_fields_} is assigned, otherwise it will have no effect.
@end deffn

@geindex _anonymous_ (ctypes.Structure attribute)
@anchor{library/ctypes ctypes Structure _anonymous_}@anchor{27dc}
@deffn {Attribute} _anonymous_

An optional sequence that lists the names of unnamed (anonymous) fields.
@ref{27dc,,_anonymous_} must be already defined when @ref{27da,,_fields_} is
assigned, otherwise it will have no effect.

The fields listed in this variable must be structure or union type fields.
@ref{29,,ctypes} will create descriptors in the structure type that allows
accessing the nested fields directly, without the need to create the
structure or union field.

Here is an example type (Windows):

@example
class _U(Union):
    _fields_ = [("lptdesc", POINTER(TYPEDESC)),
                ("lpadesc", POINTER(ARRAYDESC)),
                ("hreftype", HREFTYPE)]

class TYPEDESC(Structure):
    _anonymous_ = ("u",)
    _fields_ = [("u", _U),
                ("vt", VARTYPE)]
@end example

The @code{TYPEDESC} structure describes a COM data type, the @code{vt} field
specifies which one of the union fields is valid.  Since the @code{u} field
is defined as anonymous field, it is now possible to access the members
directly off the TYPEDESC instance. @code{td.lptdesc} and @code{td.u.lptdesc}
are equivalent, but the former is faster since it does not need to create
a temporary union instance:

@example
td = TYPEDESC()
td.vt = VT_PTR
td.lptdesc = POINTER(some_type)
td.u.lptdesc = POINTER(some_type)
@end example
@end deffn

It is possible to define sub-subclasses of structures, they inherit the
fields of the base class.  If the subclass definition has a separate
@ref{27da,,_fields_} variable, the fields specified in this are appended to the
fields of the base class.

Structure and union constructors accept both positional and keyword
arguments.  Positional arguments are used to initialize member fields in the
same order as they are appear in @ref{27da,,_fields_}.  Keyword arguments in the
constructor are interpreted as attribute assignments, so they will initialize
@ref{27da,,_fields_} with the same name, or create new attributes for names not
present in @ref{27da,,_fields_}.
@end deffn

@node Arrays and pointers,,Structured data types,ctypes reference
@anchor{library/ctypes arrays-and-pointers}@anchor{27dd}@anchor{library/ctypes ctypes-arrays-pointers}@anchor{27de}
@subsubsection Arrays and pointers


@geindex Array (class in ctypes)
@anchor{library/ctypes ctypes Array}@anchor{12d6}
@deffn {Class} ctypes.Array (*args)

Abstract base class for arrays.

The recommended way to create concrete array types is by multiplying any
@ref{29,,ctypes} data type with a positive integer.  Alternatively, you can subclass
this type and define @ref{27df,,_length_} and @ref{27e0,,_type_} class variables.
Array elements can be read and written using standard
subscript and slice accesses; for slice reads, the resulting object is
@emph{not} itself an @ref{12d6,,Array}.

@geindex _length_ (ctypes.Array attribute)
@anchor{library/ctypes ctypes Array _length_}@anchor{27df}
@deffn {Attribute} _length_

A positive integer specifying the number of elements in the array.
Out-of-range subscripts result in an @ref{1066,,IndexError}. Will be
returned by @ref{4a6,,len()}.
@end deffn

@geindex _type_ (ctypes.Array attribute)
@anchor{library/ctypes ctypes Array _type_}@anchor{27e0}
@deffn {Attribute} _type_

Specifies the type of each element in the array.
@end deffn

Array subclass constructors accept positional arguments, used to
initialize the elements in order.
@end deffn

@geindex _Pointer (class in ctypes)
@anchor{library/ctypes ctypes _Pointer}@anchor{27e1}
@deffn {Class} ctypes._Pointer

Private, abstract base class for pointers.

Concrete pointer types are created by calling @ref{2784,,POINTER()} with the
type that will be pointed to; this is done automatically by
@ref{2774,,pointer()}.

If a pointer points to an array, its elements can be read and
written using standard subscript and slice accesses.  Pointer objects
have no size, so @ref{4a6,,len()} will raise @ref{162,,TypeError}.  Negative
subscripts will read from the memory @emph{before} the pointer (as in C), and
out-of-range subscripts will probably crash with an access violation (if
you’re lucky).

@geindex _type_ (ctypes._Pointer attribute)
@anchor{library/ctypes ctypes _Pointer _type_}@anchor{27e2}
@deffn {Attribute} _type_

Specifies the type pointed to.
@end deffn

@geindex contents (ctypes._Pointer attribute)
@anchor{library/ctypes ctypes _Pointer contents}@anchor{2783}
@deffn {Attribute} contents

Returns the object to which to pointer points.  Assigning to this
attribute changes the pointer to point to the assigned object.
@end deffn
@end deffn

@node Concurrent Execution,Networking and Interprocess Communication,Generic Operating System Services,The Python Standard Library
@anchor{library/concurrency doc}@anchor{27e3}@anchor{library/concurrency concurrency}@anchor{27e4}@anchor{library/concurrency concurrent-execution}@anchor{27e5}
@section Concurrent Execution


The modules described in this chapter provide support for concurrent
execution of code. The appropriate choice of tool will depend on the
task to be executed (CPU bound vs IO bound) and preferred style of
development (event driven cooperative multitasking vs preemptive
multitasking). Here’s an overview:

@menu
* threading — Thread-based parallelism:: 
* multiprocessing — Process-based parallelism:: 
* multiprocessing.shared_memory — Provides shared memory for direct access across processes: multiprocessing shared_memory — Provides shared memory for direct access across processes. 
* The concurrent package:: 
* concurrent.futures — Launching parallel tasks: concurrent futures — Launching parallel tasks. 
* subprocess — Subprocess management:: 
* sched — Event scheduler:: 
* queue — A synchronized queue class:: 
* contextvars — Context Variables:: 
* _thread — Low-level threading API:: 

@end menu

@node threading — Thread-based parallelism,multiprocessing — Process-based parallelism,,Concurrent Execution
@anchor{library/threading doc}@anchor{27e6}@anchor{library/threading module-threading}@anchor{109}@anchor{library/threading threading-thread-based-parallelism}@anchor{27e7}
@subsection @code{threading} — Thread-based parallelism


@geindex module; threading

@strong{Source code:} Lib/threading.py@footnote{https://github.com/python/cpython/tree/main/Lib/threading.py}


__________________________________________________________________


This module constructs higher-level threading interfaces on top of the lower
level @ref{2,,_thread} module.  See also the @ref{d5,,queue} module.


Changed in version 3.7: This module used to be optional, it is now always available.


@cartouche
@quotation Note 
In the Python 2.x series, this module contained @code{camelCase} names
for some methods and functions. These are deprecated as of Python 3.10,
but they are still supported for compatibility with Python 2.5 and lower.
@end quotation
@end cartouche

@strong{CPython implementation detail:} In CPython, due to the @ref{7ae,,Global Interpreter Lock}, only one thread
can execute Python code at once (even though certain performance-oriented
libraries might overcome this limitation).
If you want your application to make better use of the computational
resources of multi-core machines, you are advised to use
@ref{b3,,multiprocessing} or @ref{3da,,concurrent.futures.ProcessPoolExecutor}.
However, threading is still an appropriate model if you want to run
multiple I/O-bound tasks simultaneously.

This module defines the following functions:

@geindex active_count() (in module threading)
@anchor{library/threading threading active_count}@anchor{31f}
@deffn {Function} threading.active_count ()

Return the number of @ref{44b,,Thread} objects currently alive.  The returned
count is equal to the length of the list returned by @ref{110d,,enumerate()}.

The function @code{activeCount} is a deprecated alias for this function.
@end deffn

@geindex current_thread() (in module threading)
@anchor{library/threading threading current_thread}@anchor{31e}
@deffn {Function} threading.current_thread ()

Return the current @ref{44b,,Thread} object, corresponding to the caller’s thread
of control.  If the caller’s thread of control was not created through the
@ref{109,,threading} module, a dummy thread object with limited functionality is
returned.

The function @code{currentThread} is a deprecated alias for this function.
@end deffn

@geindex excepthook() (in module threading)
@anchor{library/threading threading excepthook}@anchor{2d1}
@deffn {Function} threading.excepthook (args, /)

Handle uncaught exception raised by @ref{54e,,Thread.run()}.

The @emph{args} argument has the following attributes:


@itemize *

@item 
@emph{exc_type}: Exception type.

@item 
@emph{exc_value}: Exception value, can be @code{None}.

@item 
@emph{exc_traceback}: Exception traceback, can be @code{None}.

@item 
@emph{thread}: Thread which raised the exception, can be @code{None}.
@end itemize

If @emph{exc_type} is @ref{892,,SystemExit}, the exception is silently ignored.
Otherwise, the exception is printed out on @ref{42d,,sys.stderr}.

If  this function raises an exception, @ref{28e,,sys.excepthook()} is called to
handle it.

@ref{2d1,,threading.excepthook()} can be overridden to control how uncaught
exceptions raised by @ref{54e,,Thread.run()} are handled.

Storing @emph{exc_value} using a custom hook can create a reference cycle. It
should be cleared explicitly to break the reference cycle when the
exception is no longer needed.

Storing @emph{thread} using a custom hook can resurrect it if it is set to an
object which is being finalized. Avoid storing @emph{thread} after the custom
hook completes to avoid resurrecting objects.


@subsubheading See also


@ref{28e,,sys.excepthook()} handles uncaught exceptions.



New in version 3.8.

@end deffn

@geindex __excepthook__ (in module threading)
@anchor{library/threading threading __excepthook__}@anchor{2d0}
@deffn {Data} threading.__excepthook__

Holds the original value of @ref{2d1,,threading.excepthook()}. It is saved so that the
original value can be restored in case they happen to get replaced with
broken or alternative objects.


New in version 3.10.

@end deffn

@geindex get_ident() (in module threading)
@anchor{library/threading threading get_ident}@anchor{cf5}
@deffn {Function} threading.get_ident ()

Return the ‘thread identifier’ of the current thread.  This is a nonzero
integer.  Its value has no direct meaning; it is intended as a magic cookie
to be used e.g. to index a dictionary of thread-specific data.  Thread
identifiers may be recycled when a thread exits and another thread is
created.


New in version 3.3.

@end deffn

@geindex get_native_id() (in module threading)
@anchor{library/threading threading get_native_id}@anchor{54f}
@deffn {Function} threading.get_native_id ()

Return the native integral Thread ID of the current thread assigned by the kernel.
This is a non-negative integer.
Its value may be used to uniquely identify this particular thread system-wide
(until the thread terminates, after which the value may be recycled by the OS).

@ref{1629,,Availability}: Windows, FreeBSD, Linux, macOS, OpenBSD, NetBSD, AIX.


New in version 3.8.

@end deffn

@geindex enumerate() (in module threading)
@anchor{library/threading threading enumerate}@anchor{110d}
@deffn {Function} threading.enumerate ()

Return a list of all @ref{44b,,Thread} objects currently active.  The list
includes daemonic threads and dummy thread objects created by
@ref{31e,,current_thread()}.  It excludes terminated threads and threads
that have not yet been started.  However, the main thread is always part
of the result, even when terminated.
@end deffn

@geindex main_thread() (in module threading)
@anchor{library/threading threading main_thread}@anchor{b57}
@deffn {Function} threading.main_thread ()

Return the main @ref{44b,,Thread} object.  In normal conditions, the
main thread is the thread from which the Python interpreter was
started.


New in version 3.4.

@end deffn

@geindex settrace() (in module threading)
@anchor{library/threading threading settrace}@anchor{2ce}
@deffn {Function} threading.settrace (func)

@geindex trace function

Set a trace function for all threads started from the @ref{109,,threading} module.
The @emph{func} will be passed to  @ref{1034,,sys.settrace()} for each thread, before its
@ref{54e,,run()} method is called.
@end deffn

@geindex gettrace() (in module threading)
@anchor{library/threading threading gettrace}@anchor{2cc}
@deffn {Function} threading.gettrace ()

@geindex trace function
@geindex debugger

Get the trace function as set by @ref{2ce,,settrace()}.


New in version 3.10.

@end deffn

@geindex setprofile() (in module threading)
@anchor{library/threading threading setprofile}@anchor{2cf}
@deffn {Function} threading.setprofile (func)

@geindex profile function

Set a profile function for all threads started from the @ref{109,,threading} module.
The @emph{func} will be passed to  @ref{1033,,sys.setprofile()} for each thread, before its
@ref{54e,,run()} method is called.
@end deffn

@geindex getprofile() (in module threading)
@anchor{library/threading threading getprofile}@anchor{2cd}
@deffn {Function} threading.getprofile ()

@geindex profile function

Get the profiler function as set by @ref{2cf,,setprofile()}.


New in version 3.10.

@end deffn

@geindex stack_size() (in module threading)
@anchor{library/threading threading stack_size}@anchor{27e8}
@deffn {Function} threading.stack_size ([size])

Return the thread stack size used when creating new threads.  The optional
@emph{size} argument specifies the stack size to be used for subsequently created
threads, and must be 0 (use platform or configured default) or a positive
integer value of at least 32,768 (32 KiB). If @emph{size} is not specified,
0 is used.  If changing the thread stack size is
unsupported, a @ref{5af,,RuntimeError} is raised.  If the specified stack size is
invalid, a @ref{17c,,ValueError} is raised and the stack size is unmodified.  32 KiB
is currently the minimum supported stack size value to guarantee sufficient
stack space for the interpreter itself.  Note that some platforms may have
particular restrictions on values for the stack size, such as requiring a
minimum stack size > 32 KiB or requiring allocation in multiples of the system
memory page size - platform documentation should be referred to for more
information (4 KiB pages are common; using multiples of 4096 for the stack size is
the suggested approach in the absence of more specific information).

@ref{1629,,Availability}: Windows, systems with POSIX threads.
@end deffn

This module also defines the following constant:

@geindex TIMEOUT_MAX (in module threading)
@anchor{library/threading threading TIMEOUT_MAX}@anchor{27e9}
@deffn {Data} threading.TIMEOUT_MAX

The maximum value allowed for the @emph{timeout} parameter of blocking functions
(@ref{192,,Lock.acquire()}, @ref{9eb,,RLock.acquire()}, @ref{1124,,Condition.wait()}, etc.).
Specifying a timeout greater than this value will raise an
@ref{347,,OverflowError}.


New in version 3.2.

@end deffn

This module defines a number of classes, which are detailed in the sections
below.

The design of this module is loosely based on Java’s threading model. However,
where Java makes locks and condition variables basic behavior of every object,
they are separate objects in Python.  Python’s @ref{44b,,Thread} class supports a
subset of the behavior of Java’s Thread class; currently, there are no
priorities, no thread groups, and threads cannot be destroyed, stopped,
suspended, resumed, or interrupted.  The static methods of Java’s Thread class,
when implemented, are mapped to module-level functions.

All of the methods described below are executed atomically.

@menu
* Thread-Local Data:: 
* Thread Objects:: 
* Lock Objects:: 
* RLock Objects:: 
* Condition Objects:: 
* Semaphore Objects:: 
* Event Objects:: 
* Timer Objects:: 
* Barrier Objects:: 
* Using locks@comma{} conditions@comma{} and semaphores in the with statement: Using locks conditions and semaphores in the with statement. 

@end menu

@node Thread-Local Data,Thread Objects,,threading — Thread-based parallelism
@anchor{library/threading thread-local-data}@anchor{27ea}
@subsubsection Thread-Local Data


Thread-local data is data whose values are thread specific.  To manage
thread-local data, just create an instance of @ref{278d,,local} (or a
subclass) and store attributes on it:

@example
mydata = threading.local()
mydata.x = 1
@end example

The instance’s values will be different for separate threads.

@geindex local (class in threading)
@anchor{library/threading threading local}@anchor{278d}
@deffn {Class} threading.local

A class that represents thread-local data.

For more details and extensive examples, see the documentation string of the
@code{_threading_local} module.
@end deffn

@node Thread Objects,Lock Objects,Thread-Local Data,threading — Thread-based parallelism
@anchor{library/threading id1}@anchor{27eb}@anchor{library/threading thread-objects}@anchor{27ec}
@subsubsection Thread Objects


The @ref{44b,,Thread} class represents an activity that is run in a separate
thread of control.  There are two ways to specify the activity: by passing a
callable object to the constructor, or by overriding the @ref{54e,,run()}
method in a subclass.  No other methods (except for the constructor) should be
overridden in a subclass.  In other words, @emph{only}  override the
@code{__init__()} and @ref{54e,,run()} methods of this class.

Once a thread object is created, its activity must be started by calling the
thread’s @ref{2570,,start()} method.  This invokes the @ref{54e,,run()}
method in a separate thread of control.

Once the thread’s activity is started, the thread is considered ‘alive’. It
stops being alive when its @ref{54e,,run()} method terminates – either
normally, or by raising an unhandled exception.  The @ref{44c,,is_alive()}
method tests whether the thread is alive.

Other threads can call a thread’s @ref{d86,,join()} method.  This blocks
the calling thread until the thread whose @ref{d86,,join()} method is
called is terminated.

A thread has a name.  The name can be passed to the constructor, and read or
changed through the @ref{322,,name} attribute.

If the @ref{54e,,run()} method raises an exception,
@ref{2d1,,threading.excepthook()} is called to handle it. By default,
@ref{2d1,,threading.excepthook()} ignores silently @ref{892,,SystemExit}.

A thread can be flagged as a “daemon thread”.  The significance of this flag is
that the entire Python program exits when only daemon threads are left.  The
initial value is inherited from the creating thread.  The flag can be set
through the @ref{323,,daemon} property or the @emph{daemon} constructor
argument.

@cartouche
@quotation Note 
Daemon threads are abruptly stopped at shutdown.  Their resources (such
as open files, database transactions, etc.) may not be released properly.
If you want your threads to stop gracefully, make them non-daemonic and
use a suitable signalling mechanism such as an @ref{cf3,,Event}.
@end quotation
@end cartouche

There is a “main thread” object; this corresponds to the initial thread of
control in the Python program.  It is not a daemon thread.

There is the possibility that “dummy thread objects” are created. These are
thread objects corresponding to “alien threads”, which are threads of control
started outside the threading module, such as directly from C code.  Dummy
thread objects have limited functionality; they are always considered alive and
daemonic, and cannot be @ref{d86,,join()}ed.  They are never deleted,
since it is impossible to detect the termination of alien threads.

@geindex Thread (class in threading)
@anchor{library/threading threading Thread}@anchor{44b}
@deffn {Class} threading.Thread (group=None, target=None, name=None, args=(), kwargs=@{@}, *, daemon=None)

This constructor should always be called with keyword arguments.  Arguments
are:

@emph{group} should be @code{None}; reserved for future extension when a
@code{ThreadGroup} class is implemented.

@emph{target} is the callable object to be invoked by the @ref{54e,,run()} method.
Defaults to @code{None}, meaning nothing is called.

@emph{name} is the thread name. By default, a unique name is constructed
of the form “Thread-@emph{N}” where @emph{N} is a small decimal number,
or “Thread-@emph{N} (target)” where “target” is @code{target.__name__} if the
@emph{target} argument is specified.

@emph{args} is a list or tuple of arguments for the target invocation.  Defaults to @code{()}.

@emph{kwargs} is a dictionary of keyword arguments for the target invocation.
Defaults to @code{@{@}}.

If not @code{None}, @emph{daemon} explicitly sets whether the thread is daemonic.
If @code{None} (the default), the daemonic property is inherited from the
current thread.

If the subclass overrides the constructor, it must make sure to invoke the
base class constructor (@code{Thread.__init__()}) before doing anything else to
the thread.


Changed in version 3.10: Use the @emph{target} name if @emph{name} argument is omitted.



Changed in version 3.3: Added the @emph{daemon} argument.


@geindex start() (threading.Thread method)
@anchor{library/threading threading Thread start}@anchor{2570}
@deffn {Method} start ()

Start the thread’s activity.

It must be called at most once per thread object.  It arranges for the
object’s @ref{54e,,run()} method to be invoked in a separate thread
of control.

This method will raise a @ref{5af,,RuntimeError} if called more than once
on the same thread object.
@end deffn

@geindex run() (threading.Thread method)
@anchor{library/threading threading Thread run}@anchor{54e}
@deffn {Method} run ()

Method representing the thread’s activity.

You may override this method in a subclass.  The standard @ref{54e,,run()}
method invokes the callable object passed to the object’s constructor as
the @emph{target} argument, if any, with positional and keyword arguments taken
from the @emph{args} and @emph{kwargs} arguments, respectively.

Using list or tuple as the @emph{args} argument which passed to the @ref{44b,,Thread}
could achieve the same effect.

Example:

@example
>>> from threading import Thread
>>> t = Thread(target=print, args=[1])
>>> t.run()
1
>>> t = Thread(target=print, args=(1,))
>>> t.run()
1
@end example
@end deffn

@geindex join() (threading.Thread method)
@anchor{library/threading threading Thread join}@anchor{d86}
@deffn {Method} join (timeout=None)

Wait until the thread terminates. This blocks the calling thread until
the thread whose @ref{d86,,join()} method is called terminates – either
normally or through an unhandled exception – or until the optional
timeout occurs.

When the @emph{timeout} argument is present and not @code{None}, it should be a
floating point number specifying a timeout for the operation in seconds
(or fractions thereof). As @ref{d86,,join()} always returns @code{None},
you must call @ref{44c,,is_alive()} after @ref{d86,,join()} to
decide whether a timeout happened – if the thread is still alive, the
@ref{d86,,join()} call timed out.

When the @emph{timeout} argument is not present or @code{None}, the operation will
block until the thread terminates.

A thread can be @ref{d86,,join()}ed many times.

@ref{d86,,join()} raises a @ref{5af,,RuntimeError} if an attempt is made
to join the current thread as that would cause a deadlock. It is also
an error to @ref{d86,,join()} a thread before it has been started
and attempts to do so raise the same exception.
@end deffn

@geindex name (threading.Thread attribute)
@anchor{library/threading threading Thread name}@anchor{322}
@deffn {Attribute} name

A string used for identification purposes only. It has no semantics.
Multiple threads may be given the same name.  The initial name is set by
the constructor.
@end deffn

@geindex getName() (threading.Thread method)
@geindex setName() (threading.Thread method)
@anchor{library/threading threading Thread getName}@anchor{27ed}
@deffn {Method} getName ()
@anchor{library/threading threading Thread setName}@anchor{27ee}
@deffnx {Method} setName ()

Deprecated getter/setter API for @ref{322,,name}; use it directly as a
property instead.


Deprecated since version 3.10.

@end deffn

@geindex ident (threading.Thread attribute)
@anchor{library/threading threading Thread ident}@anchor{24b8}
@deffn {Attribute} ident

The ‘thread identifier’ of this thread or @code{None} if the thread has not
been started.  This is a nonzero integer.  See the @ref{cf5,,get_ident()}
function.  Thread identifiers may be recycled when a thread exits and
another thread is created.  The identifier is available even after the
thread has exited.
@end deffn

@geindex native_id (threading.Thread attribute)
@anchor{library/threading threading Thread native_id}@anchor{550}
@deffn {Attribute} native_id

The Thread ID (@code{TID}) of this thread, as assigned by the OS (kernel).
This is a non-negative integer, or @code{None} if the thread has not
been started. See the @ref{54f,,get_native_id()} function.
This value may be used to uniquely identify this particular thread
system-wide (until the thread terminates, after which the value
may be recycled by the OS).

@cartouche
@quotation Note 
Similar to Process IDs, Thread IDs are only valid (guaranteed unique
system-wide) from the time the thread is created until the thread
has been terminated.
@end quotation
@end cartouche

@ref{1629,,Availability}: Requires @ref{54f,,get_native_id()} function.


New in version 3.8.

@end deffn

@geindex is_alive() (threading.Thread method)
@anchor{library/threading threading Thread is_alive}@anchor{44c}
@deffn {Method} is_alive ()

Return whether the thread is alive.

This method returns @code{True} just before the @ref{54e,,run()} method
starts until just after the @ref{54e,,run()} method terminates.  The
module function @ref{110d,,enumerate()} returns a list of all alive threads.
@end deffn

@geindex daemon (threading.Thread attribute)
@anchor{library/threading threading Thread daemon}@anchor{323}
@deffn {Attribute} daemon

A boolean value indicating whether this thread is a daemon thread (@code{True})
or not (@code{False}).  This must be set before @ref{2570,,start()} is called,
otherwise @ref{5af,,RuntimeError} is raised.  Its initial value is inherited
from the creating thread; the main thread is not a daemon thread and
therefore all threads created in the main thread default to
@ref{323,,daemon} = @code{False}.

The entire Python program exits when no alive non-daemon threads are left.
@end deffn

@geindex isDaemon() (threading.Thread method)
@geindex setDaemon() (threading.Thread method)
@anchor{library/threading threading Thread isDaemon}@anchor{27ef}
@deffn {Method} isDaemon ()
@anchor{library/threading threading Thread setDaemon}@anchor{27f0}
@deffnx {Method} setDaemon ()

Deprecated getter/setter API for @ref{323,,daemon}; use it directly as a
property instead.


Deprecated since version 3.10.

@end deffn
@end deffn

@node Lock Objects,RLock Objects,Thread Objects,threading — Thread-based parallelism
@anchor{library/threading id2}@anchor{27f1}@anchor{library/threading lock-objects}@anchor{27f2}
@subsubsection Lock Objects


A primitive lock is a synchronization primitive that is not owned by a
particular thread when locked.  In Python, it is currently the lowest level
synchronization primitive available, implemented directly by the @ref{2,,_thread}
extension module.

A primitive lock is in one of two states, “locked” or “unlocked”. It is created
in the unlocked state.  It has two basic methods, @ref{192,,acquire()} and
@ref{27f3,,release()}.  When the state is unlocked, @ref{192,,acquire()}
changes the state to locked and returns immediately.  When the state is locked,
@ref{192,,acquire()} blocks until a call to @ref{27f3,,release()} in another
thread changes it to unlocked, then the @ref{192,,acquire()} call resets it
to locked and returns.  The @ref{27f3,,release()} method should only be
called in the locked state; it changes the state to unlocked and returns
immediately. If an attempt is made to release an unlocked lock, a
@ref{5af,,RuntimeError} will be raised.

Locks also support the @ref{27f4,,context management protocol}.

When more than one thread is blocked in @ref{192,,acquire()} waiting for the
state to turn to unlocked, only one thread proceeds when a @ref{27f3,,release()}
call resets the state to unlocked; which one of the waiting threads proceeds
is not defined, and may vary across implementations.

All methods are executed atomically.

@geindex Lock (class in threading)
@anchor{library/threading threading Lock}@anchor{27f5}
@deffn {Class} threading.Lock

The class implementing primitive lock objects.  Once a thread has acquired a
lock, subsequent attempts to acquire it block, until it is released; any
thread may release it.

Note that @code{Lock} is actually a factory function which returns an instance
of the most efficient version of the concrete Lock class that is supported
by the platform.

@geindex acquire() (threading.Lock method)
@anchor{library/threading threading Lock acquire}@anchor{192}
@deffn {Method} acquire (blocking=True, timeout=- 1)

Acquire a lock, blocking or non-blocking.

When invoked with the @emph{blocking} argument set to @code{True} (the default),
block until the lock is unlocked, then set it to locked and return @code{True}.

When invoked with the @emph{blocking} argument set to @code{False}, do not block.
If a call with @emph{blocking} set to @code{True} would block, return @code{False}
immediately; otherwise, set the lock to locked and return @code{True}.

When invoked with the floating-point @emph{timeout} argument set to a positive
value, block for at most the number of seconds specified by @emph{timeout}
and as long as the lock cannot be acquired.  A @emph{timeout} argument of @code{-1}
specifies an unbounded wait.  It is forbidden to specify a @emph{timeout}
when @emph{blocking} is @code{False}.

The return value is @code{True} if the lock is acquired successfully,
@code{False} if not (for example if the @emph{timeout} expired).


Changed in version 3.2: The @emph{timeout} parameter is new.



Changed in version 3.2: Lock acquisition can now be interrupted by signals on POSIX if the
underlying threading implementation supports it.

@end deffn

@geindex release() (threading.Lock method)
@anchor{library/threading threading Lock release}@anchor{27f3}
@deffn {Method} release ()

Release a lock.  This can be called from any thread, not only the thread
which has acquired the lock.

When the lock is locked, reset it to unlocked, and return.  If any other threads
are blocked waiting for the lock to become unlocked, allow exactly one of them
to proceed.

When invoked on an unlocked lock, a @ref{5af,,RuntimeError} is raised.

There is no return value.
@end deffn

@geindex locked() (threading.Lock method)
@anchor{library/threading threading Lock locked}@anchor{27f6}
@deffn {Method} locked ()

Return @code{True} if the lock is acquired.
@end deffn
@end deffn

@node RLock Objects,Condition Objects,Lock Objects,threading — Thread-based parallelism
@anchor{library/threading id3}@anchor{27f7}@anchor{library/threading rlock-objects}@anchor{27f8}
@subsubsection RLock Objects


A reentrant lock is a synchronization primitive that may be acquired multiple
times by the same thread.  Internally, it uses the concepts of “owning thread”
and “recursion level” in addition to the locked/unlocked state used by primitive
locks.  In the locked state, some thread owns the lock; in the unlocked state,
no thread owns it.

To lock the lock, a thread calls its @ref{9eb,,acquire()} method; this
returns once the thread owns the lock.  To unlock the lock, a thread calls
its @ref{27f3,,release()} method. @ref{192,,acquire()}/@ref{27f3,,release()}
call pairs may be nested; only the final @ref{27f3,,release()} (the
@ref{27f3,,release()} of the outermost pair) resets the lock to unlocked and
allows another thread blocked in @ref{192,,acquire()} to proceed.

Reentrant locks also support the @ref{27f4,,context management protocol}.

@geindex RLock (class in threading)
@anchor{library/threading threading RLock}@anchor{e09}
@deffn {Class} threading.RLock

This class implements reentrant lock objects.  A reentrant lock must be
released by the thread that acquired it.  Once a thread has acquired a
reentrant lock, the same thread may acquire it again without blocking; the
thread must release it once for each time it has acquired it.

Note that @code{RLock} is actually a factory function which returns an instance
of the most efficient version of the concrete RLock class that is supported
by the platform.

@geindex acquire() (threading.RLock method)
@anchor{library/threading threading RLock acquire}@anchor{9eb}
@deffn {Method} acquire (blocking=True, timeout=- 1)

Acquire a lock, blocking or non-blocking.

When invoked without arguments: if this thread already owns the lock, increment
the recursion level by one, and return immediately.  Otherwise, if another
thread owns the lock, block until the lock is unlocked.  Once the lock is
unlocked (not owned by any thread), then grab ownership, set the recursion level
to one, and return.  If more than one thread is blocked waiting until the lock
is unlocked, only one at a time will be able to grab ownership of the lock.
There is no return value in this case.

When invoked with the @emph{blocking} argument set to @code{True}, do the same thing as when
called without arguments, and return @code{True}.

When invoked with the @emph{blocking} argument set to @code{False}, do not block.  If a call
without an argument would block, return @code{False} immediately; otherwise, do the
same thing as when called without arguments, and return @code{True}.

When invoked with the floating-point @emph{timeout} argument set to a positive
value, block for at most the number of seconds specified by @emph{timeout}
and as long as the lock cannot be acquired.  Return @code{True} if the lock has
been acquired, @code{False} if the timeout has elapsed.


Changed in version 3.2: The @emph{timeout} parameter is new.

@end deffn

@geindex release() (threading.RLock method)
@anchor{library/threading threading RLock release}@anchor{27f9}
@deffn {Method} release ()

Release a lock, decrementing the recursion level.  If after the decrement it is
zero, reset the lock to unlocked (not owned by any thread), and if any other
threads are blocked waiting for the lock to become unlocked, allow exactly one
of them to proceed.  If after the decrement the recursion level is still
nonzero, the lock remains locked and owned by the calling thread.

Only call this method when the calling thread owns the lock. A
@ref{5af,,RuntimeError} is raised if this method is called when the lock is
unlocked.

There is no return value.
@end deffn
@end deffn

@node Condition Objects,Semaphore Objects,RLock Objects,threading — Thread-based parallelism
@anchor{library/threading condition-objects}@anchor{27fa}@anchor{library/threading id4}@anchor{27fb}
@subsubsection Condition Objects


A condition variable is always associated with some kind of lock; this can be
passed in or one will be created by default.  Passing one in is useful when
several condition variables must share the same lock.  The lock is part of
the condition object: you don’t have to track it separately.

A condition variable obeys the @ref{27f4,,context management protocol}:
using the @code{with} statement acquires the associated lock for the duration of
the enclosed block.  The @ref{27fc,,acquire()} and
@ref{27fd,,release()} methods also call the corresponding methods of
the associated lock.

Other methods must be called with the associated lock held.  The
@ref{1124,,wait()} method releases the lock, and then blocks until
another thread awakens it by calling @ref{27fe,,notify()} or
@ref{320,,notify_all()}.  Once awakened, @ref{1124,,wait()}
re-acquires the lock and returns.  It is also possible to specify a timeout.

The @ref{27fe,,notify()} method wakes up one of the threads waiting for
the condition variable, if any are waiting.  The @ref{320,,notify_all()}
method wakes up all threads waiting for the condition variable.

Note: the @ref{27fe,,notify()} and @ref{320,,notify_all()} methods
don’t release the lock; this means that the thread or threads awakened will
not return from their @ref{1124,,wait()} call immediately, but only when
the thread that called @ref{27fe,,notify()} or @ref{320,,notify_all()}
finally relinquishes ownership of the lock.

The typical programming style using condition variables uses the lock to
synchronize access to some shared state; threads that are interested in a
particular change of state call @ref{1124,,wait()} repeatedly until they
see the desired state, while threads that modify the state call
@ref{27fe,,notify()} or @ref{320,,notify_all()} when they change
the state in such a way that it could possibly be a desired state for one
of the waiters.  For example, the following code is a generic
producer-consumer situation with unlimited buffer capacity:

@example
# Consume one item
with cv:
    while not an_item_is_available():
        cv.wait()
    get_an_available_item()

# Produce one item
with cv:
    make_an_item_available()
    cv.notify()
@end example

The @code{while} loop checking for the application’s condition is necessary
because @ref{1124,,wait()} can return after an arbitrary long time,
and the condition which prompted the @ref{27fe,,notify()} call may
no longer hold true.  This is inherent to multi-threaded programming.  The
@ref{27ff,,wait_for()} method can be used to automate the condition
checking, and eases the computation of timeouts:

@example
# Consume an item
with cv:
    cv.wait_for(an_item_is_available)
    get_an_available_item()
@end example

To choose between @ref{27fe,,notify()} and @ref{320,,notify_all()},
consider whether one state change can be interesting for only one or several
waiting threads.  E.g. in a typical producer-consumer situation, adding one
item to the buffer only needs to wake up one consumer thread.

@geindex Condition (class in threading)
@anchor{library/threading threading Condition}@anchor{cf0}
@deffn {Class} threading.Condition (lock=None)

This class implements condition variable objects.  A condition variable
allows one or more threads to wait until they are notified by another thread.

If the @emph{lock} argument is given and not @code{None}, it must be a @ref{27f5,,Lock}
or @ref{e09,,RLock} object, and it is used as the underlying lock.  Otherwise,
a new @ref{e09,,RLock} object is created and used as the underlying lock.


Changed in version 3.3: changed from a factory function to a class.


@geindex acquire() (threading.Condition method)
@anchor{library/threading threading Condition acquire}@anchor{27fc}
@deffn {Method} acquire (*args)

Acquire the underlying lock. This method calls the corresponding method on
the underlying lock; the return value is whatever that method returns.
@end deffn

@geindex release() (threading.Condition method)
@anchor{library/threading threading Condition release}@anchor{27fd}
@deffn {Method} release ()

Release the underlying lock. This method calls the corresponding method on
the underlying lock; there is no return value.
@end deffn

@geindex wait() (threading.Condition method)
@anchor{library/threading threading Condition wait}@anchor{1124}
@deffn {Method} wait (timeout=None)

Wait until notified or until a timeout occurs. If the calling thread has
not acquired the lock when this method is called, a @ref{5af,,RuntimeError} is
raised.

This method releases the underlying lock, and then blocks until it is
awakened by a @ref{27fe,,notify()} or @ref{320,,notify_all()} call for the same
condition variable in another thread, or until the optional timeout
occurs.  Once awakened or timed out, it re-acquires the lock and returns.

When the @emph{timeout} argument is present and not @code{None}, it should be a
floating point number specifying a timeout for the operation in seconds
(or fractions thereof).

When the underlying lock is an @ref{e09,,RLock}, it is not released using
its @ref{27fd,,release()} method, since this may not actually unlock the lock
when it was acquired multiple times recursively.  Instead, an internal
interface of the @ref{e09,,RLock} class is used, which really unlocks it
even when it has been recursively acquired several times. Another internal
interface is then used to restore the recursion level when the lock is
reacquired.

The return value is @code{True} unless a given @emph{timeout} expired, in which
case it is @code{False}.


Changed in version 3.2: Previously, the method always returned @code{None}.

@end deffn

@geindex wait_for() (threading.Condition method)
@anchor{library/threading threading Condition wait_for}@anchor{27ff}
@deffn {Method} wait_for (predicate, timeout=None)

Wait until a condition evaluates to true.  @emph{predicate} should be a
callable which result will be interpreted as a boolean value.
A @emph{timeout} may be provided giving the maximum time to wait.

This utility method may call @ref{1124,,wait()} repeatedly until the predicate
is satisfied, or until a timeout occurs. The return value is
the last return value of the predicate and will evaluate to
@code{False} if the method timed out.

Ignoring the timeout feature, calling this method is roughly equivalent to
writing:

@example
while not predicate():
    cv.wait()
@end example

Therefore, the same rules apply as with @ref{1124,,wait()}: The lock must be
held when called and is re-acquired on return.  The predicate is evaluated
with the lock held.


New in version 3.2.

@end deffn

@geindex notify() (threading.Condition method)
@anchor{library/threading threading Condition notify}@anchor{27fe}
@deffn {Method} notify (n=1)

By default, wake up one thread waiting on this condition, if any.  If the
calling thread has not acquired the lock when this method is called, a
@ref{5af,,RuntimeError} is raised.

This method wakes up at most @emph{n} of the threads waiting for the condition
variable; it is a no-op if no threads are waiting.

The current implementation wakes up exactly @emph{n} threads, if at least @emph{n}
threads are waiting.  However, it’s not safe to rely on this behavior.
A future, optimized implementation may occasionally wake up more than
@emph{n} threads.

Note: an awakened thread does not actually return from its @ref{1124,,wait()}
call until it can reacquire the lock.  Since @ref{27fe,,notify()} does not
release the lock, its caller should.
@end deffn

@geindex notify_all() (threading.Condition method)
@anchor{library/threading threading Condition notify_all}@anchor{320}
@deffn {Method} notify_all ()

Wake up all threads waiting on this condition.  This method acts like
@ref{27fe,,notify()}, but wakes up all waiting threads instead of one. If the
calling thread has not acquired the lock when this method is called, a
@ref{5af,,RuntimeError} is raised.

The method @code{notifyAll} is a deprecated alias for this method.
@end deffn
@end deffn

@node Semaphore Objects,Event Objects,Condition Objects,threading — Thread-based parallelism
@anchor{library/threading id5}@anchor{2800}@anchor{library/threading semaphore-objects}@anchor{2801}
@subsubsection Semaphore Objects


This is one of the oldest synchronization primitives in the history of computer
science, invented by the early Dutch computer scientist Edsger W. Dijkstra (he
used the names @code{P()} and @code{V()} instead of @ref{e07,,acquire()} and
@ref{2802,,release()}).

A semaphore manages an internal counter which is decremented by each
@ref{e07,,acquire()} call and incremented by each @ref{2802,,release()}
call.  The counter can never go below zero; when @ref{e07,,acquire()}
finds that it is zero, it blocks, waiting until some other thread calls
@ref{2802,,release()}.

Semaphores also support the @ref{27f4,,context management protocol}.

@geindex Semaphore (class in threading)
@anchor{library/threading threading Semaphore}@anchor{cf1}
@deffn {Class} threading.Semaphore (value=1)

This class implements semaphore objects.  A semaphore manages an atomic
counter representing the number of @ref{2802,,release()} calls minus the number of
@ref{e07,,acquire()} calls, plus an initial value.  The @ref{e07,,acquire()} method
blocks if necessary until it can return without making the counter negative.
If not given, @emph{value} defaults to 1.

The optional argument gives the initial @emph{value} for the internal counter; it
defaults to @code{1}. If the @emph{value} given is less than 0, @ref{17c,,ValueError} is
raised.


Changed in version 3.3: changed from a factory function to a class.


@geindex acquire() (threading.Semaphore method)
@anchor{library/threading threading Semaphore acquire}@anchor{e07}
@deffn {Method} acquire (blocking=True, timeout=None)

Acquire a semaphore.

When invoked without arguments:


@itemize *

@item 
If the internal counter is larger than zero on entry, decrement it by
one and return @code{True} immediately.

@item 
If the internal counter is zero on entry, block until awoken by a call to
@ref{2802,,release()}.  Once awoken (and the counter is greater
than 0), decrement the counter by 1 and return @code{True}.  Exactly one
thread will be awoken by each call to @ref{2802,,release()}.  The
order in which threads are awoken should not be relied on.
@end itemize

When invoked with @emph{blocking} set to @code{False}, do not block.  If a call
without an argument would block, return @code{False} immediately; otherwise, do
the same thing as when called without arguments, and return @code{True}.

When invoked with a @emph{timeout} other than @code{None}, it will block for at
most @emph{timeout} seconds.  If acquire does not complete successfully in
that interval, return @code{False}.  Return @code{True} otherwise.


Changed in version 3.2: The @emph{timeout} parameter is new.

@end deffn

@geindex release() (threading.Semaphore method)
@anchor{library/threading threading Semaphore release}@anchor{2802}
@deffn {Method} release (n=1)

Release a semaphore, incrementing the internal counter by @emph{n}.  When it
was zero on entry and other threads are waiting for it to become larger
than zero again, wake up @emph{n} of those threads.


Changed in version 3.9: Added the @emph{n} parameter to release multiple waiting threads at once.

@end deffn
@end deffn

@geindex BoundedSemaphore (class in threading)
@anchor{library/threading threading BoundedSemaphore}@anchor{cf2}
@deffn {Class} threading.BoundedSemaphore (value=1)

Class implementing bounded semaphore objects.  A bounded semaphore checks to
make sure its current value doesn’t exceed its initial value.  If it does,
@ref{17c,,ValueError} is raised. In most situations semaphores are used to guard
resources with limited capacity.  If the semaphore is released too many times
it’s a sign of a bug.  If not given, @emph{value} defaults to 1.


Changed in version 3.3: changed from a factory function to a class.

@end deffn

@menu
* Semaphore Example:: 

@end menu

@node Semaphore Example,,,Semaphore Objects
@anchor{library/threading semaphore-example}@anchor{2803}@anchor{library/threading semaphore-examples}@anchor{2804}
@subsubsection @code{Semaphore} Example


Semaphores are often used to guard resources with limited capacity, for example,
a database server.  In any situation where the size of the resource is fixed,
you should use a bounded semaphore.  Before spawning any worker threads, your
main thread would initialize the semaphore:

@example
maxconnections = 5
# ...
pool_sema = BoundedSemaphore(value=maxconnections)
@end example

Once spawned, worker threads call the semaphore’s acquire and release methods
when they need to connect to the server:

@example
with pool_sema:
    conn = connectdb()
    try:
        # ... use connection ...
    finally:
        conn.close()
@end example

The use of a bounded semaphore reduces the chance that a programming error which
causes the semaphore to be released more than it’s acquired will go undetected.

@node Event Objects,Timer Objects,Semaphore Objects,threading — Thread-based parallelism
@anchor{library/threading event-objects}@anchor{2805}@anchor{library/threading id6}@anchor{2806}
@subsubsection Event Objects


This is one of the simplest mechanisms for communication between threads: one
thread signals an event and other threads wait for it.

An event object manages an internal flag that can be set to true with the
@ref{2807,,set()} method and reset to false with the @ref{2808,,clear()}
method.  The @ref{eb9,,wait()} method blocks until the flag is true.

@geindex Event (class in threading)
@anchor{library/threading threading Event}@anchor{cf3}
@deffn {Class} threading.Event

Class implementing event objects.  An event manages a flag that can be set to
true with the @ref{2807,,set()} method and reset to false with the
@ref{2808,,clear()} method.  The @ref{eb9,,wait()} method blocks until the flag is true.
The flag is initially false.


Changed in version 3.3: changed from a factory function to a class.


@geindex is_set() (threading.Event method)
@anchor{library/threading threading Event is_set}@anchor{321}
@deffn {Method} is_set ()

Return @code{True} if and only if the internal flag is true.

The method @code{isSet} is a deprecated alias for this method.
@end deffn

@geindex set() (threading.Event method)
@anchor{library/threading threading Event set}@anchor{2807}
@deffn {Method} set ()

Set the internal flag to true. All threads waiting for it to become true
are awakened. Threads that call @ref{eb9,,wait()} once the flag is true will
not block at all.
@end deffn

@geindex clear() (threading.Event method)
@anchor{library/threading threading Event clear}@anchor{2808}
@deffn {Method} clear ()

Reset the internal flag to false. Subsequently, threads calling
@ref{eb9,,wait()} will block until @ref{2807,,set()} is called to set the internal
flag to true again.
@end deffn

@geindex wait() (threading.Event method)
@anchor{library/threading threading Event wait}@anchor{eb9}
@deffn {Method} wait (timeout=None)

Block until the internal flag is true.  If the internal flag is true on
entry, return immediately.  Otherwise, block until another thread calls
@ref{2807,,set()} to set the flag to true, or until the optional timeout occurs.

When the timeout argument is present and not @code{None}, it should be a
floating point number specifying a timeout for the operation in seconds
(or fractions thereof).

This method returns @code{True} if and only if the internal flag has been set to
true, either before the wait call or after the wait starts, so it will
always return @code{True} except if a timeout is given and the operation
times out.


Changed in version 3.1: Previously, the method always returned @code{None}.

@end deffn
@end deffn

@node Timer Objects,Barrier Objects,Event Objects,threading — Thread-based parallelism
@anchor{library/threading id7}@anchor{2809}@anchor{library/threading timer-objects}@anchor{280a}
@subsubsection Timer Objects


This class represents an action that should be run only after a certain amount
of time has passed — a timer.  @ref{cf4,,Timer} is a subclass of @ref{44b,,Thread}
and as such also functions as an example of creating custom threads.

Timers are started, as with threads, by calling their @code{start()}
method.  The timer can be stopped (before its action has begun) by calling the
@ref{280b,,cancel()} method.  The interval the timer will wait before
executing its action may not be exactly the same as the interval specified by
the user.

For example:

@example
def hello():
    print("hello, world")

t = Timer(30.0, hello)
t.start()  # after 30 seconds, "hello, world" will be printed
@end example

@geindex Timer (class in threading)
@anchor{library/threading threading Timer}@anchor{cf4}
@deffn {Class} threading.Timer (interval, function, args=None, kwargs=None)

Create a timer that will run @emph{function} with arguments @emph{args} and  keyword
arguments @emph{kwargs}, after @emph{interval} seconds have passed.
If @emph{args} is @code{None} (the default) then an empty list will be used.
If @emph{kwargs} is @code{None} (the default) then an empty dict will be used.


Changed in version 3.3: changed from a factory function to a class.


@geindex cancel() (threading.Timer method)
@anchor{library/threading threading Timer cancel}@anchor{280b}
@deffn {Method} cancel ()

Stop the timer, and cancel the execution of the timer’s action.  This will
only work if the timer is still in its waiting stage.
@end deffn
@end deffn

@node Barrier Objects,Using locks conditions and semaphores in the with statement,Timer Objects,threading — Thread-based parallelism
@anchor{library/threading barrier-objects}@anchor{280c}
@subsubsection Barrier Objects



New in version 3.2.


This class provides a simple synchronization primitive for use by a fixed number
of threads that need to wait for each other.  Each of the threads tries to pass
the barrier by calling the @ref{280d,,wait()} method and will block until
all of the threads have made their @ref{280d,,wait()} calls. At this point,
the threads are released simultaneously.

The barrier can be reused any number of times for the same number of threads.

As an example, here is a simple way to synchronize a client and server thread:

@example
b = Barrier(2, timeout=5)

def server():
    start_server()
    b.wait()
    while True:
        connection = accept_connection()
        process_server_connection(connection)

def client():
    b.wait()
    while True:
        connection = make_connection()
        process_client_connection(connection)
@end example

@geindex Barrier (class in threading)
@anchor{library/threading threading Barrier}@anchor{d85}
@deffn {Class} threading.Barrier (parties, action=None, timeout=None)

Create a barrier object for @emph{parties} number of threads.  An @emph{action}, when
provided, is a callable to be called by one of the threads when they are
released.  @emph{timeout} is the default timeout value if none is specified for
the @ref{280d,,wait()} method.

@geindex wait() (threading.Barrier method)
@anchor{library/threading threading Barrier wait}@anchor{280d}
@deffn {Method} wait (timeout=None)

Pass the barrier.  When all the threads party to the barrier have called
this function, they are all released simultaneously.  If a @emph{timeout} is
provided, it is used in preference to any that was supplied to the class
constructor.

The return value is an integer in the range 0 to @emph{parties} – 1, different
for each thread.  This can be used to select a thread to do some special
housekeeping, e.g.:

@example
i = barrier.wait()
if i == 0:
    # Only one thread needs to print this
    print("passed the barrier")
@end example

If an @emph{action} was provided to the constructor, one of the threads will
have called it prior to being released.  Should this call raise an error,
the barrier is put into the broken state.

If the call times out, the barrier is put into the broken state.

This method may raise a @ref{d87,,BrokenBarrierError} exception if the
barrier is broken or reset while a thread is waiting.
@end deffn

@geindex reset() (threading.Barrier method)
@anchor{library/threading threading Barrier reset}@anchor{280e}
@deffn {Method} reset ()

Return the barrier to the default, empty state.  Any threads waiting on it
will receive the @ref{d87,,BrokenBarrierError} exception.

Note that using this function may require some external
synchronization if there are other threads whose state is unknown.  If a
barrier is broken it may be better to just leave it and create a new one.
@end deffn

@geindex abort() (threading.Barrier method)
@anchor{library/threading threading Barrier abort}@anchor{280f}
@deffn {Method} abort ()

Put the barrier into a broken state.  This causes any active or future
calls to @ref{280d,,wait()} to fail with the @ref{d87,,BrokenBarrierError}.  Use
this for example if one of the threads needs to abort, to avoid deadlocking the
application.

It may be preferable to simply create the barrier with a sensible
@emph{timeout} value to automatically guard against one of the threads going
awry.
@end deffn

@geindex parties (threading.Barrier attribute)
@anchor{library/threading threading Barrier parties}@anchor{2810}
@deffn {Attribute} parties

The number of threads required to pass the barrier.
@end deffn

@geindex n_waiting (threading.Barrier attribute)
@anchor{library/threading threading Barrier n_waiting}@anchor{2811}
@deffn {Attribute} n_waiting

The number of threads currently waiting in the barrier.
@end deffn

@geindex broken (threading.Barrier attribute)
@anchor{library/threading threading Barrier broken}@anchor{2812}
@deffn {Attribute} broken

A boolean that is @code{True} if the barrier is in the broken state.
@end deffn
@end deffn

@geindex BrokenBarrierError
@anchor{library/threading threading BrokenBarrierError}@anchor{d87}
@deffn {Exception} threading.BrokenBarrierError

This exception, a subclass of @ref{5af,,RuntimeError}, is raised when the
@ref{d85,,Barrier} object is reset or broken.
@end deffn

@node Using locks conditions and semaphores in the with statement,,Barrier Objects,threading — Thread-based parallelism
@anchor{library/threading using-locks-conditions-and-semaphores-in-the-with-statement}@anchor{2813}@anchor{library/threading with-locks}@anchor{27f4}
@subsubsection Using locks, conditions, and semaphores in the @code{with} statement


All of the objects provided by this module that have @code{acquire()} and
@code{release()} methods can be used as context managers for a @ref{168,,with}
statement.  The @code{acquire()} method will be called when the block is
entered, and @code{release()} will be called when the block is exited.  Hence,
the following snippet:

@example
with some_lock:
    # do something...
@end example

is equivalent to:

@example
some_lock.acquire()
try:
    # do something...
finally:
    some_lock.release()
@end example

Currently, @ref{27f5,,Lock}, @ref{e09,,RLock}, @ref{cf0,,Condition},
@ref{cf1,,Semaphore}, and @ref{cf2,,BoundedSemaphore} objects may be used as
@ref{168,,with} statement context managers.

@node multiprocessing — Process-based parallelism,multiprocessing shared_memory — Provides shared memory for direct access across processes,threading — Thread-based parallelism,Concurrent Execution
@anchor{library/multiprocessing doc}@anchor{2814}@anchor{library/multiprocessing module-multiprocessing}@anchor{b3}@anchor{library/multiprocessing multiprocessing-process-based-parallelism}@anchor{2815}
@subsection @code{multiprocessing} — Process-based parallelism


@geindex module; multiprocessing

@strong{Source code:} Lib/multiprocessing/@footnote{https://github.com/python/cpython/tree/main/Lib/multiprocessing/}


__________________________________________________________________


@menu
* Introduction: Introduction<8>. 
* Reference:: 
* Programming guidelines:: 
* Examples: Examples<12>. 

@end menu

@node Introduction<8>,Reference,,multiprocessing — Process-based parallelism
@anchor{library/multiprocessing introduction}@anchor{2816}
@subsubsection Introduction


@ref{b3,,multiprocessing} is a package that supports spawning processes using an
API similar to the @ref{109,,threading} module.  The @ref{b3,,multiprocessing} package
offers both local and remote concurrency, effectively side-stepping the
@ref{7ae,,Global Interpreter Lock} by using
subprocesses instead of threads.  Due
to this, the @ref{b3,,multiprocessing} module allows the programmer to fully
leverage multiple processors on a given machine.  It runs on both Unix and
Windows.

The @ref{b3,,multiprocessing} module also introduces APIs which do not have
analogs in the @ref{109,,threading} module.  A prime example of this is the
@ref{aec,,Pool} object which offers a convenient means of
parallelizing the execution of a function across multiple input values,
distributing the input data across processes (data parallelism).  The following
example demonstrates the common practice of defining such functions in a module
so that child processes can successfully import that module.  This basic example
of data parallelism using @ref{aec,,Pool},

@example
from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))
@end example

will print to standard output

@example
[1, 4, 9]
@end example

@menu
* The Process class:: 
* Contexts and start methods:: 
* Exchanging objects between processes:: 
* Synchronization between processes:: 
* Sharing state between processes:: 
* Using a pool of workers:: 

@end menu

@node The Process class,Contexts and start methods,,Introduction<8>
@anchor{library/multiprocessing the-process-class}@anchor{2817}
@subsubsection The @code{Process} class


In @ref{b3,,multiprocessing}, processes are spawned by creating a @ref{681,,Process}
object and then calling its @ref{2818,,start()} method.  @ref{681,,Process}
follows the API of @ref{44b,,threading.Thread}.  A trivial example of a
multiprocess program is

@example
from multiprocessing import Process

def f(name):
    print('hello', name)

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()
@end example

To show the individual process IDs involved, here is an expanded example:

@example
from multiprocessing import Process
import os

def info(title):
    print(title)
    print('module name:', __name__)
    print('parent process:', os.getppid())
    print('process id:', os.getpid())

def f(name):
    info('function f')
    print('hello', name)

if __name__ == '__main__':
    info('main line')
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()
@end example

For an explanation of why the @code{if __name__ == '__main__'} part is
necessary, see @ref{2819,,Programming guidelines}.

@node Contexts and start methods,Exchanging objects between processes,The Process class,Introduction<8>
@anchor{library/multiprocessing contexts-and-start-methods}@anchor{281a}
@subsubsection Contexts and start methods

@anchor{library/multiprocessing multiprocessing-start-methods}@anchor{ae7}
Depending on the platform, @ref{b3,,multiprocessing} supports three ways
to start a process.  These @emph{start methods} are

@quotation


@table @asis

@item @emph{spawn}

The parent process starts a fresh python interpreter process.  The
child process will only inherit those resources necessary to run
the process object’s @ref{281b,,run()} method.  In particular,
unnecessary file descriptors and handles from the parent process
will not be inherited.  Starting a process using this method is
rather slow compared to using @emph{fork} or @emph{forkserver}.

Available on Unix and Windows.  The default on Windows and macOS.

@item @emph{fork}

The parent process uses @ref{bb9,,os.fork()} to fork the Python
interpreter.  The child process, when it begins, is effectively
identical to the parent process.  All resources of the parent are
inherited by the child process.  Note that safely forking a
multithreaded process is problematic.

Available on Unix only.  The default on Unix.

@item @emph{forkserver}

When the program starts and selects the @emph{forkserver} start method,
a server process is started.  From then on, whenever a new process
is needed, the parent process connects to the server and requests
that it fork a new process.  The fork server process is single
threaded so it is safe for it to use @ref{bb9,,os.fork()}.  No
unnecessary resources are inherited.

Available on Unix platforms which support passing file descriptors
over Unix pipes.
@end table
@end quotation


Changed in version 3.8: On macOS, the @emph{spawn} start method is now the default.  The @emph{fork} start
method should be considered unsafe as it can lead to crashes of the
subprocess. See bpo-33725@footnote{https://bugs.python.org/issue33725}.



Changed in version 3.4: @emph{spawn} added on all unix platforms, and @emph{forkserver} added for
some unix platforms.
Child processes no longer inherit all of the parents inheritable
handles on Windows.


On Unix using the @emph{spawn} or @emph{forkserver} start methods will also
start a @emph{resource tracker} process which tracks the unlinked named
system resources (such as named semaphores or
@ref{281c,,SharedMemory} objects) created
by processes of the program.  When all processes
have exited the resource tracker unlinks any remaining tracked object.
Usually there should be none, but if a process was killed by a signal
there may be some “leaked” resources.  (Neither leaked semaphores nor shared
memory segments will be automatically unlinked until the next reboot. This is
problematic for both objects because the system allows only a limited number of
named semaphores, and shared memory segments occupy some space in the main
memory.)

To select a start method you use the @ref{aea,,set_start_method()} in
the @code{if __name__ == '__main__'} clause of the main module.  For
example:

@example
import multiprocessing as mp

def foo(q):
    q.put('hello')

if __name__ == '__main__':
    mp.set_start_method('spawn')
    q = mp.Queue()
    p = mp.Process(target=foo, args=(q,))
    p.start()
    print(q.get())
    p.join()
@end example

@ref{aea,,set_start_method()} should not be used more than once in the
program.

Alternatively, you can use @ref{aeb,,get_context()} to obtain a context
object.  Context objects have the same API as the multiprocessing
module, and allow one to use multiple start methods in the same
program.

@example
import multiprocessing as mp

def foo(q):
    q.put('hello')

if __name__ == '__main__':
    ctx = mp.get_context('spawn')
    q = ctx.Queue()
    p = ctx.Process(target=foo, args=(q,))
    p.start()
    print(q.get())
    p.join()
@end example

Note that objects related to one context may not be compatible with
processes for a different context.  In particular, locks created using
the @emph{fork} context cannot be passed to processes started using the
@emph{spawn} or @emph{forkserver} start methods.

A library which wants to use a particular start method should probably
use @ref{aeb,,get_context()} to avoid interfering with the choice of the
library user.

@cartouche
@quotation Warning 
The @code{'spawn'} and @code{'forkserver'} start methods cannot currently
be used with “frozen” executables (i.e., binaries produced by
packages like @strong{PyInstaller} and @strong{cx_Freeze}) on Unix.
The @code{'fork'} start method does work.
@end quotation
@end cartouche

@node Exchanging objects between processes,Synchronization between processes,Contexts and start methods,Introduction<8>
@anchor{library/multiprocessing exchanging-objects-between-processes}@anchor{281d}
@subsubsection Exchanging objects between processes


@ref{b3,,multiprocessing} supports two types of communication channel between
processes:

@strong{Queues}

@quotation

The @ref{281e,,Queue} class is a near clone of @ref{f18,,queue.Queue}.  For
example:

@example
from multiprocessing import Process, Queue

def f(q):
    q.put([42, None, 'hello'])

if __name__ == '__main__':
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    print(q.get())    # prints "[42, None, 'hello']"
    p.join()
@end example

Queues are thread and process safe.
@end quotation

@strong{Pipes}

@quotation

The @ref{281f,,Pipe()} function returns a pair of connection objects connected by a
pipe which by default is duplex (two-way).  For example:

@example
from multiprocessing import Process, Pipe

def f(conn):
    conn.send([42, None, 'hello'])
    conn.close()

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()
    p = Process(target=f, args=(child_conn,))
    p.start()
    print(parent_conn.recv())   # prints "[42, None, 'hello']"
    p.join()
@end example

The two connection objects returned by @ref{281f,,Pipe()} represent the two ends of
the pipe.  Each connection object has @code{send()} and
@code{recv()} methods (among others).  Note that data in a pipe
may become corrupted if two processes (or threads) try to read from or write
to the @emph{same} end of the pipe at the same time.  Of course there is no risk
of corruption from processes using different ends of the pipe at the same
time.
@end quotation

@node Synchronization between processes,Sharing state between processes,Exchanging objects between processes,Introduction<8>
@anchor{library/multiprocessing synchronization-between-processes}@anchor{2820}
@subsubsection Synchronization between processes


@ref{b3,,multiprocessing} contains equivalents of all the synchronization
primitives from @ref{109,,threading}.  For instance one can use a lock to ensure
that only one process prints to standard output at a time:

@example
from multiprocessing import Process, Lock

def f(l, i):
    l.acquire()
    try:
        print('hello world', i)
    finally:
        l.release()

if __name__ == '__main__':
    lock = Lock()

    for num in range(10):
        Process(target=f, args=(lock, num)).start()
@end example

Without using the lock output from the different processes is liable to get all
mixed up.

@node Sharing state between processes,Using a pool of workers,Synchronization between processes,Introduction<8>
@anchor{library/multiprocessing sharing-state-between-processes}@anchor{2821}
@subsubsection Sharing state between processes


As mentioned above, when doing concurrent programming it is usually best to
avoid using shared state as far as possible.  This is particularly true when
using multiple processes.

However, if you really do need to use some shared data then
@ref{b3,,multiprocessing} provides a couple of ways of doing so.

@strong{Shared memory}

@quotation

Data can be stored in a shared memory map using @ref{2822,,Value} or
@ref{2823,,Array}.  For example, the following code

@example
from multiprocessing import Process, Value, Array

def f(n, a):
    n.value = 3.1415927
    for i in range(len(a)):
        a[i] = -a[i]

if __name__ == '__main__':
    num = Value('d', 0.0)
    arr = Array('i', range(10))

    p = Process(target=f, args=(num, arr))
    p.start()
    p.join()

    print(num.value)
    print(arr[:])
@end example

will print

@example
3.1415927
[0, -1, -2, -3, -4, -5, -6, -7, -8, -9]
@end example

The @code{'d'} and @code{'i'} arguments used when creating @code{num} and @code{arr} are
typecodes of the kind used by the @ref{6,,array} module: @code{'d'} indicates a
double precision float and @code{'i'} indicates a signed integer.  These shared
objects will be process and thread-safe.

For more flexibility in using shared memory one can use the
@ref{b9,,multiprocessing.sharedctypes} module which supports the creation of
arbitrary ctypes objects allocated from shared memory.
@end quotation

@strong{Server process}

@quotation

A manager object returned by @code{Manager()} controls a server process which
holds Python objects and allows other processes to manipulate them using
proxies.

A manager returned by @code{Manager()} will support types
@ref{43a,,list}, @ref{3b0,,dict}, @ref{2824,,Namespace}, @ref{2825,,Lock},
@ref{2826,,RLock}, @ref{2827,,Semaphore}, @ref{2828,,BoundedSemaphore},
@ref{2829,,Condition}, @ref{10be,,Event}, @ref{282a,,Barrier},
@ref{281e,,Queue}, @ref{2822,,Value} and @ref{2823,,Array}.  For example,

@example
from multiprocessing import Process, Manager

def f(d, l):
    d[1] = '1'
    d['2'] = 2
    d[0.25] = None
    l.reverse()

if __name__ == '__main__':
    with Manager() as manager:
        d = manager.dict()
        l = manager.list(range(10))

        p = Process(target=f, args=(d, l))
        p.start()
        p.join()

        print(d)
        print(l)
@end example

will print

@example
@{0.25: None, 1: '1', '2': 2@}
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
@end example

Server process managers are more flexible than using shared memory objects
because they can be made to support arbitrary object types.  Also, a single
manager can be shared by processes on different computers over a network.
They are, however, slower than using shared memory.
@end quotation

@node Using a pool of workers,,Sharing state between processes,Introduction<8>
@anchor{library/multiprocessing using-a-pool-of-workers}@anchor{282b}
@subsubsection Using a pool of workers


The @ref{aec,,Pool} class represents a pool of worker
processes.  It has methods which allows tasks to be offloaded to the worker
processes in a few different ways.

For example:

@example
from multiprocessing import Pool, TimeoutError
import time
import os

def f(x):
    return x*x

if __name__ == '__main__':
    # start 4 worker processes
    with Pool(processes=4) as pool:

        # print "[0, 1, 4,..., 81]"
        print(pool.map(f, range(10)))

        # print same numbers in arbitrary order
        for i in pool.imap_unordered(f, range(10)):
            print(i)

        # evaluate "f(20)" asynchronously
        res = pool.apply_async(f, (20,))      # runs in *only* one process
        print(res.get(timeout=1))             # prints "400"

        # evaluate "os.getpid()" asynchronously
        res = pool.apply_async(os.getpid, ()) # runs in *only* one process
        print(res.get(timeout=1))             # prints the PID of that process

        # launching multiple evaluations asynchronously *may* use more processes
        multiple_results = [pool.apply_async(os.getpid, ()) for i in range(4)]
        print([res.get(timeout=1) for res in multiple_results])

        # make a single worker sleep for 10 seconds
        res = pool.apply_async(time.sleep, (10,))
        try:
            print(res.get(timeout=1))
        except TimeoutError:
            print("We lacked patience and got a multiprocessing.TimeoutError")

        print("For the moment, the pool remains available for more work")

    # exiting the 'with'-block has stopped the pool
    print("Now the pool is closed and no longer available")
@end example

Note that the methods of a pool should only ever be used by the
process which created it.

@cartouche
@quotation Note 
Functionality within this package requires that the @code{__main__} module be
importable by the children. This is covered in @ref{2819,,Programming guidelines}
however it is worth pointing out here. This means that some examples, such
as the @ref{aec,,multiprocessing.pool.Pool} examples will not work in the
interactive interpreter. For example:

@example
>>> from multiprocessing import Pool
>>> p = Pool(5)
>>> def f(x):
...     return x*x
...
>>> with p:
...   p.map(f, [1,2,3])
Process PoolWorker-1:
Process PoolWorker-2:
Process PoolWorker-3:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
AttributeError: 'module' object has no attribute 'f'
AttributeError: 'module' object has no attribute 'f'
AttributeError: 'module' object has no attribute 'f'
@end example

(If you try this it will actually output three full tracebacks
interleaved in a semi-random fashion, and then you may have to
stop the parent process somehow.)
@end quotation
@end cartouche

@node Reference,Programming guidelines,Introduction<8>,multiprocessing — Process-based parallelism
@anchor{library/multiprocessing reference}@anchor{282c}
@subsubsection Reference


The @ref{b3,,multiprocessing} package mostly replicates the API of the
@ref{109,,threading} module.

@menu
* Process and exceptions:: 
* Pipes and Queues:: 
* Miscellaneous: Miscellaneous<3>. 
* Connection Objects: Connection Objects<2>. 
* Synchronization primitives:: 
* Shared ctypes Objects:: 
* Managers:: 
* Proxy Objects:: 
* Process Pools:: 
* Listeners and Clients:: 
* Authentication keys:: 
* Logging: Logging<2>. 
* The multiprocessing.dummy module: The multiprocessing dummy module. 

@end menu

@node Process and exceptions,Pipes and Queues,,Reference
@anchor{library/multiprocessing process-and-exceptions}@anchor{282d}
@subsubsection @code{Process} and exceptions


@geindex Process (class in multiprocessing)
@anchor{library/multiprocessing multiprocessing Process}@anchor{681}
@deffn {Class} multiprocessing.Process (group=None, target=None, name=None, args=(), kwargs=@{@}, *, daemon=None)

Process objects represent activity that is run in a separate process. The
@ref{681,,Process} class has equivalents of all the methods of
@ref{44b,,threading.Thread}.

The constructor should always be called with keyword arguments. @emph{group}
should always be @code{None}; it exists solely for compatibility with
@ref{44b,,threading.Thread}.  @emph{target} is the callable object to be invoked by
the @ref{281b,,run()} method.  It defaults to @code{None}, meaning nothing is
called. @emph{name} is the process name (see @ref{282e,,name} for more details).
@emph{args} is the argument tuple for the target invocation.  @emph{kwargs} is a
dictionary of keyword arguments for the target invocation.  If provided,
the keyword-only @emph{daemon} argument sets the process @ref{282f,,daemon} flag
to @code{True} or @code{False}.  If @code{None} (the default), this flag will be
inherited from the creating process.

By default, no arguments are passed to @emph{target}. The @emph{args} argument,
which defaults to @code{()}, can be used to specify a list or tuple of the arguments
to pass to @emph{target}.

If a subclass overrides the constructor, it must make sure it invokes the
base class constructor (@code{Process.__init__()}) before doing anything else
to the process.


Changed in version 3.3: Added the @emph{daemon} argument.


@geindex run() (multiprocessing.Process method)
@anchor{library/multiprocessing multiprocessing Process run}@anchor{281b}
@deffn {Method} run ()

Method representing the process’s activity.

You may override this method in a subclass.  The standard @ref{281b,,run()}
method invokes the callable object passed to the object’s constructor as
the target argument, if any, with sequential and keyword arguments taken
from the @emph{args} and @emph{kwargs} arguments, respectively.

Using a list or tuple as the @emph{args} argument passed to @ref{681,,Process}
achieves the same effect.

Example:

@example
>>> from multiprocessing import Process
>>> p = Process(target=print, args=[1])
>>> p.run()
1
>>> p = Process(target=print, args=(1,))
>>> p.run()
1
@end example
@end deffn

@geindex start() (multiprocessing.Process method)
@anchor{library/multiprocessing multiprocessing Process start}@anchor{2818}
@deffn {Method} start ()

Start the process’s activity.

This must be called at most once per process object.  It arranges for the
object’s @ref{281b,,run()} method to be invoked in a separate process.
@end deffn

@geindex join() (multiprocessing.Process method)
@anchor{library/multiprocessing multiprocessing Process join}@anchor{2830}
@deffn {Method} join ([timeout])

If the optional argument @emph{timeout} is @code{None} (the default), the method
blocks until the process whose @ref{2830,,join()} method is called terminates.
If @emph{timeout} is a positive number, it blocks at most @emph{timeout} seconds.
Note that the method returns @code{None} if its process terminates or if the
method times out.  Check the process’s @ref{2831,,exitcode} to determine if
it terminated.

A process can be joined many times.

A process cannot join itself because this would cause a deadlock.  It is
an error to attempt to join a process before it has been started.
@end deffn

@geindex name (multiprocessing.Process attribute)
@anchor{library/multiprocessing multiprocessing Process name}@anchor{282e}
@deffn {Attribute} name

The process’s name.  The name is a string used for identification purposes
only.  It has no semantics.  Multiple processes may be given the same
name.

The initial name is set by the constructor.  If no explicit name is
provided to the constructor, a name of the form
‘Process-N@w{[1]}:N@w{[2]}:…:N@w{[k]}’ is constructed, where
each N@w{[k]} is the N-th child of its parent.
@end deffn

@geindex is_alive() (multiprocessing.Process method)
@anchor{library/multiprocessing multiprocessing Process is_alive}@anchor{2832}
@deffn {Method} is_alive ()

Return whether the process is alive.

Roughly, a process object is alive from the moment the @ref{2818,,start()}
method returns until the child process terminates.
@end deffn

@geindex daemon (multiprocessing.Process attribute)
@anchor{library/multiprocessing multiprocessing Process daemon}@anchor{282f}
@deffn {Attribute} daemon

The process’s daemon flag, a Boolean value.  This must be set before
@ref{2818,,start()} is called.

The initial value is inherited from the creating process.

When a process exits, it attempts to terminate all of its daemonic child
processes.

Note that a daemonic process is not allowed to create child processes.
Otherwise a daemonic process would leave its children orphaned if it gets
terminated when its parent process exits. Additionally, these are @strong{not}
Unix daemons or services, they are normal processes that will be
terminated (and not joined) if non-daemonic processes have exited.
@end deffn

In addition to the  @ref{44b,,threading.Thread} API, @ref{681,,Process} objects
also support the following attributes and methods:

@geindex pid (multiprocessing.Process attribute)
@anchor{library/multiprocessing multiprocessing Process pid}@anchor{2833}
@deffn {Attribute} pid

Return the process ID.  Before the process is spawned, this will be
@code{None}.
@end deffn

@geindex exitcode (multiprocessing.Process attribute)
@anchor{library/multiprocessing multiprocessing Process exitcode}@anchor{2831}
@deffn {Attribute} exitcode

The child’s exit code.  This will be @code{None} if the process has not yet
terminated.

If the child’s @ref{281b,,run()} method returned normally, the exit code
will be 0.  If it terminated via @ref{ee4,,sys.exit()} with an integer
argument @emph{N}, the exit code will be @emph{N}.

If the child terminated due to an exception not caught within
@ref{281b,,run()}, the exit code will be 1.  If it was terminated by
signal @emph{N}, the exit code will be the negative value @emph{-N}.
@end deffn

@geindex authkey (multiprocessing.Process attribute)
@anchor{library/multiprocessing multiprocessing Process authkey}@anchor{2834}
@deffn {Attribute} authkey

The process’s authentication key (a byte string).

When @ref{b3,,multiprocessing} is initialized the main process is assigned a
random string using @ref{17f,,os.urandom()}.

When a @ref{681,,Process} object is created, it will inherit the
authentication key of its parent process, although this may be changed by
setting @ref{2834,,authkey} to another byte string.

See @ref{2835,,Authentication keys}.
@end deffn

@geindex sentinel (multiprocessing.Process attribute)
@anchor{library/multiprocessing multiprocessing Process sentinel}@anchor{c73}
@deffn {Attribute} sentinel

A numeric handle of a system object which will become “ready” when
the process ends.

You can use this value if you want to wait on several events at
once using @ref{c72,,multiprocessing.connection.wait()}.  Otherwise
calling @ref{2830,,join()} is simpler.

On Windows, this is an OS handle usable with the @code{WaitForSingleObject}
and @code{WaitForMultipleObjects} family of API calls.  On Unix, this is
a file descriptor usable with primitives from the @ref{e0,,select} module.


New in version 3.3.

@end deffn

@geindex terminate() (multiprocessing.Process method)
@anchor{library/multiprocessing multiprocessing Process terminate}@anchor{2836}
@deffn {Method} terminate ()

Terminate the process.  On Unix this is done using the @code{SIGTERM} signal;
on Windows @code{TerminateProcess()} is used.  Note that exit handlers and
finally clauses, etc., will not be executed.

Note that descendant processes of the process will @emph{not} be terminated –
they will simply become orphaned.

@cartouche
@quotation Warning 
If this method is used when the associated process is using a pipe or
queue then the pipe or queue is liable to become corrupted and may
become unusable by other process.  Similarly, if the process has
acquired a lock or semaphore etc. then terminating it is liable to
cause other processes to deadlock.
@end quotation
@end cartouche
@end deffn

@geindex kill() (multiprocessing.Process method)
@anchor{library/multiprocessing multiprocessing Process kill}@anchor{680}
@deffn {Method} kill ()

Same as @ref{2836,,terminate()} but using the @code{SIGKILL} signal on Unix.


New in version 3.7.

@end deffn

@geindex close() (multiprocessing.Process method)
@anchor{library/multiprocessing multiprocessing Process close}@anchor{67f}
@deffn {Method} close ()

Close the @ref{681,,Process} object, releasing all resources associated
with it.  @ref{17c,,ValueError} is raised if the underlying process
is still running.  Once @ref{67f,,close()} returns successfully, most
other methods and attributes of the @ref{681,,Process} object will
raise @ref{17c,,ValueError}.


New in version 3.7.

@end deffn

Note that the @ref{2818,,start()}, @ref{2830,,join()}, @ref{2832,,is_alive()},
@ref{2836,,terminate()} and @ref{2831,,exitcode} methods should only be called by
the process that created the process object.

Example usage of some of the methods of @ref{681,,Process}:

@example
 >>> import multiprocessing, time, signal
 >>> p = multiprocessing.Process(target=time.sleep, args=(1000,))
 >>> print(p, p.is_alive())
 <Process ... initial> False
 >>> p.start()
 >>> print(p, p.is_alive())
 <Process ... started> True
 >>> p.terminate()
 >>> time.sleep(0.1)
 >>> print(p, p.is_alive())
 <Process ... stopped exitcode=-SIGTERM> False
 >>> p.exitcode == -signal.SIGTERM
 True
@end example
@end deffn

@geindex ProcessError
@anchor{library/multiprocessing multiprocessing ProcessError}@anchor{2837}
@deffn {Exception} multiprocessing.ProcessError

The base class of all @ref{b3,,multiprocessing} exceptions.
@end deffn

@geindex BufferTooShort
@anchor{library/multiprocessing multiprocessing BufferTooShort}@anchor{2838}
@deffn {Exception} multiprocessing.BufferTooShort

Exception raised by @code{Connection.recv_bytes_into()} when the supplied
buffer object is too small for the message read.

If @code{e} is an instance of @ref{2838,,BufferTooShort} then @code{e.args[0]} will give
the message as a byte string.
@end deffn

@geindex AuthenticationError
@anchor{library/multiprocessing multiprocessing AuthenticationError}@anchor{2839}
@deffn {Exception} multiprocessing.AuthenticationError

Raised when there is an authentication error.
@end deffn

@geindex TimeoutError
@anchor{library/multiprocessing multiprocessing TimeoutError}@anchor{283a}
@deffn {Exception} multiprocessing.TimeoutError

Raised by methods with a timeout when the timeout expires.
@end deffn

@node Pipes and Queues,Miscellaneous<3>,Process and exceptions,Reference
@anchor{library/multiprocessing pipes-and-queues}@anchor{283b}
@subsubsection Pipes and Queues


When using multiple processes, one generally uses message passing for
communication between processes and avoids having to use any synchronization
primitives like locks.

For passing messages one can use @ref{281f,,Pipe()} (for a connection between two
processes) or a queue (which allows multiple producers and consumers).

The @ref{281e,,Queue}, @ref{406,,SimpleQueue} and @ref{283c,,JoinableQueue} types
are multi-producer, multi-consumer @abbr{FIFO}
queues modelled on the @ref{f18,,queue.Queue} class in the
standard library.  They differ in that @ref{281e,,Queue} lacks the
@ref{283d,,task_done()} and @ref{283e,,join()} methods introduced
into Python 2.5’s @ref{f18,,queue.Queue} class.

If you use @ref{283c,,JoinableQueue} then you @strong{must} call
@ref{283f,,JoinableQueue.task_done()} for each task removed from the queue or else the
semaphore used to count the number of unfinished tasks may eventually overflow,
raising an exception.

Note that one can also create a shared queue by using a manager object – see
@ref{2840,,Managers}.

@cartouche
@quotation Note 
@ref{b3,,multiprocessing} uses the usual @ref{2841,,queue.Empty} and
@ref{2842,,queue.Full} exceptions to signal a timeout.  They are not available in
the @ref{b3,,multiprocessing} namespace so you need to import them from
@ref{d5,,queue}.
@end quotation
@end cartouche

@cartouche
@quotation Note 
When an object is put on a queue, the object is pickled and a
background thread later flushes the pickled data to an underlying
pipe.  This has some consequences which are a little surprising,
but should not cause any practical difficulties – if they really
bother you then you can instead use a queue created with a
@ref{2840,,manager}.


@enumerate 

@item 
After putting an object on an empty queue there may be an
infinitesimal delay before the queue’s @ref{2843,,empty()}
method returns @ref{659,,False} and @ref{2844,,get_nowait()} can
return without raising @ref{2841,,queue.Empty}.

@item 
If multiple processes are enqueuing objects, it is possible for
the objects to be received at the other end out-of-order.
However, objects enqueued by the same process will always be in
the expected order with respect to each other.
@end enumerate
@end quotation
@end cartouche

@cartouche
@quotation Warning 
If a process is killed using @ref{2836,,Process.terminate()} or @ref{ef3,,os.kill()}
while it is trying to use a @ref{281e,,Queue}, then the data in the queue is
likely to become corrupted.  This may cause any other process to get an
exception when it tries to use the queue later on.
@end quotation
@end cartouche

@cartouche
@quotation Warning 
As mentioned above, if a child process has put items on a queue (and it has
not used @ref{2845,,JoinableQueue.cancel_join_thread}), then that process will
not terminate until all buffered items have been flushed to the pipe.

This means that if you try joining that process you may get a deadlock unless
you are sure that all items which have been put on the queue have been
consumed.  Similarly, if the child process is non-daemonic then the parent
process may hang on exit when it tries to join all its non-daemonic children.

Note that a queue created using a manager does not have this issue.  See
@ref{2819,,Programming guidelines}.
@end quotation
@end cartouche

For an example of the usage of queues for interprocess communication see
@ref{2846,,Examples}.

@geindex Pipe() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing Pipe}@anchor{281f}
@deffn {Function} multiprocessing.Pipe ([duplex])

Returns a pair @code{(conn1, conn2)} of
@ref{2847,,Connection} objects representing the
ends of a pipe.

If @emph{duplex} is @code{True} (the default) then the pipe is bidirectional.  If
@emph{duplex} is @code{False} then the pipe is unidirectional: @code{conn1} can only be
used for receiving messages and @code{conn2} can only be used for sending
messages.
@end deffn

@geindex Queue (class in multiprocessing)
@anchor{library/multiprocessing multiprocessing Queue}@anchor{281e}
@deffn {Class} multiprocessing.Queue ([maxsize])

Returns a process shared queue implemented using a pipe and a few
locks/semaphores.  When a process first puts an item on the queue a feeder
thread is started which transfers objects from a buffer into the pipe.

The usual @ref{2841,,queue.Empty} and @ref{2842,,queue.Full} exceptions from the
standard library’s @ref{d5,,queue} module are raised to signal timeouts.

@ref{281e,,Queue} implements all the methods of @ref{f18,,queue.Queue} except for
@ref{283d,,task_done()} and @ref{283e,,join()}.

@geindex qsize() (multiprocessing.Queue method)
@anchor{library/multiprocessing multiprocessing Queue qsize}@anchor{2848}
@deffn {Method} qsize ()

Return the approximate size of the queue.  Because of
multithreading/multiprocessing semantics, this number is not reliable.

Note that this may raise @ref{8a3,,NotImplementedError} on Unix platforms like
macOS where @code{sem_getvalue()} is not implemented.
@end deffn

@geindex empty() (multiprocessing.Queue method)
@anchor{library/multiprocessing multiprocessing Queue empty}@anchor{2843}
@deffn {Method} empty ()

Return @code{True} if the queue is empty, @code{False} otherwise.  Because of
multithreading/multiprocessing semantics, this is not reliable.
@end deffn

@geindex full() (multiprocessing.Queue method)
@anchor{library/multiprocessing multiprocessing Queue full}@anchor{2849}
@deffn {Method} full ()

Return @code{True} if the queue is full, @code{False} otherwise.  Because of
multithreading/multiprocessing semantics, this is not reliable.
@end deffn

@geindex put() (multiprocessing.Queue method)
@anchor{library/multiprocessing multiprocessing Queue put}@anchor{12f3}
@deffn {Method} put (obj[, block[, timeout]])

Put obj into the queue.  If the optional argument @emph{block} is @code{True}
(the default) and @emph{timeout} is @code{None} (the default), block if necessary until
a free slot is available.  If @emph{timeout} is a positive number, it blocks at
most @emph{timeout} seconds and raises the @ref{2842,,queue.Full} exception if no
free slot was available within that time.  Otherwise (@emph{block} is
@code{False}), put an item on the queue if a free slot is immediately
available, else raise the @ref{2842,,queue.Full} exception (@emph{timeout} is
ignored in that case).


Changed in version 3.8: If the queue is closed, @ref{17c,,ValueError} is raised instead of
@ref{46e,,AssertionError}.

@end deffn

@geindex put_nowait() (multiprocessing.Queue method)
@anchor{library/multiprocessing multiprocessing Queue put_nowait}@anchor{284a}
@deffn {Method} put_nowait (obj)

Equivalent to @code{put(obj, False)}.
@end deffn

@geindex get() (multiprocessing.Queue method)
@anchor{library/multiprocessing multiprocessing Queue get}@anchor{12f4}
@deffn {Method} get ([block[, timeout]])

Remove and return an item from the queue.  If optional args @emph{block} is
@code{True} (the default) and @emph{timeout} is @code{None} (the default), block if
necessary until an item is available.  If @emph{timeout} is a positive number,
it blocks at most @emph{timeout} seconds and raises the @ref{2841,,queue.Empty}
exception if no item was available within that time.  Otherwise (block is
@code{False}), return an item if one is immediately available, else raise the
@ref{2841,,queue.Empty} exception (@emph{timeout} is ignored in that case).


Changed in version 3.8: If the queue is closed, @ref{17c,,ValueError} is raised instead of
@ref{2a1,,OSError}.

@end deffn

@geindex get_nowait() (multiprocessing.Queue method)
@anchor{library/multiprocessing multiprocessing Queue get_nowait}@anchor{2844}
@deffn {Method} get_nowait ()

Equivalent to @code{get(False)}.
@end deffn

@ref{281e,,multiprocessing.Queue} has a few additional methods not found in
@ref{f18,,queue.Queue}.  These methods are usually unnecessary for most
code:

@geindex close() (multiprocessing.Queue method)
@anchor{library/multiprocessing multiprocessing Queue close}@anchor{284b}
@deffn {Method} close ()

Indicate that no more data will be put on this queue by the current
process.  The background thread will quit once it has flushed all buffered
data to the pipe.  This is called automatically when the queue is garbage
collected.
@end deffn

@geindex join_thread() (multiprocessing.Queue method)
@anchor{library/multiprocessing multiprocessing Queue join_thread}@anchor{284c}
@deffn {Method} join_thread ()

Join the background thread.  This can only be used after @ref{284b,,close()} has
been called.  It blocks until the background thread exits, ensuring that
all data in the buffer has been flushed to the pipe.

By default if a process is not the creator of the queue then on exit it
will attempt to join the queue’s background thread.  The process can call
@ref{2845,,cancel_join_thread()} to make @ref{284c,,join_thread()} do nothing.
@end deffn

@geindex cancel_join_thread() (multiprocessing.Queue method)
@anchor{library/multiprocessing multiprocessing Queue cancel_join_thread}@anchor{2845}
@deffn {Method} cancel_join_thread ()

Prevent @ref{284c,,join_thread()} from blocking.  In particular, this prevents
the background thread from being joined automatically when the process
exits – see @ref{284c,,join_thread()}.

A better name for this method might be
@code{allow_exit_without_flush()}.  It is likely to cause enqueued
data to be lost, and you almost certainly will not need to use it.
It is really only there if you need the current process to exit
immediately without waiting to flush enqueued data to the
underlying pipe, and you don’t care about lost data.
@end deffn

@cartouche
@quotation Note 
This class’s functionality requires a functioning shared semaphore
implementation on the host operating system. Without one, the
functionality in this class will be disabled, and attempts to
instantiate a @ref{281e,,Queue} will result in an @ref{3b8,,ImportError}. See
bpo-3770@footnote{https://bugs.python.org/issue3770} for additional information.  The same holds true for any
of the specialized queue types listed below.
@end quotation
@end cartouche
@end deffn

@geindex SimpleQueue (class in multiprocessing)
@anchor{library/multiprocessing multiprocessing SimpleQueue}@anchor{406}
@deffn {Class} multiprocessing.SimpleQueue

It is a simplified @ref{281e,,Queue} type, very close to a locked @ref{281f,,Pipe}.

@geindex close() (multiprocessing.SimpleQueue method)
@anchor{library/multiprocessing multiprocessing SimpleQueue close}@anchor{407}
@deffn {Method} close ()

Close the queue: release internal resources.

A queue must not be used anymore after it is closed. For example,
@ref{284d,,get()}, @ref{284e,,put()} and @ref{284f,,empty()} methods must no longer be
called.


New in version 3.9.

@end deffn

@geindex empty() (multiprocessing.SimpleQueue method)
@anchor{library/multiprocessing multiprocessing SimpleQueue empty}@anchor{284f}
@deffn {Method} empty ()

Return @code{True} if the queue is empty, @code{False} otherwise.
@end deffn

@geindex get() (multiprocessing.SimpleQueue method)
@anchor{library/multiprocessing multiprocessing SimpleQueue get}@anchor{284d}
@deffn {Method} get ()

Remove and return an item from the queue.
@end deffn

@geindex put() (multiprocessing.SimpleQueue method)
@anchor{library/multiprocessing multiprocessing SimpleQueue put}@anchor{284e}
@deffn {Method} put (item)

Put @emph{item} into the queue.
@end deffn
@end deffn

@geindex JoinableQueue (class in multiprocessing)
@anchor{library/multiprocessing multiprocessing JoinableQueue}@anchor{283c}
@deffn {Class} multiprocessing.JoinableQueue ([maxsize])

@ref{283c,,JoinableQueue}, a @ref{281e,,Queue} subclass, is a queue which
additionally has @ref{283f,,task_done()} and @ref{2850,,join()} methods.

@geindex task_done() (multiprocessing.JoinableQueue method)
@anchor{library/multiprocessing multiprocessing JoinableQueue task_done}@anchor{283f}
@deffn {Method} task_done ()

Indicate that a formerly enqueued task is complete. Used by queue
consumers.  For each @ref{12f4,,get()} used to fetch a task, a subsequent
call to @ref{283f,,task_done()} tells the queue that the processing on the task
is complete.

If a @ref{283e,,join()} is currently blocking, it will resume when all
items have been processed (meaning that a @ref{283f,,task_done()} call was
received for every item that had been @ref{12f3,,put()} into the queue).

Raises a @ref{17c,,ValueError} if called more times than there were items
placed in the queue.
@end deffn

@geindex join() (multiprocessing.JoinableQueue method)
@anchor{library/multiprocessing multiprocessing JoinableQueue join}@anchor{2850}
@deffn {Method} join ()

Block until all items in the queue have been gotten and processed.

The count of unfinished tasks goes up whenever an item is added to the
queue.  The count goes down whenever a consumer calls
@ref{283f,,task_done()} to indicate that the item was retrieved and all work on
it is complete.  When the count of unfinished tasks drops to zero,
@ref{283e,,join()} unblocks.
@end deffn
@end deffn

@node Miscellaneous<3>,Connection Objects<2>,Pipes and Queues,Reference
@anchor{library/multiprocessing miscellaneous}@anchor{2851}
@subsubsection Miscellaneous


@geindex active_children() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing active_children}@anchor{2852}
@deffn {Function} multiprocessing.active_children ()

Return list of all live children of the current process.

Calling this has the side effect of “joining” any processes which have
already finished.
@end deffn

@geindex cpu_count() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing cpu_count}@anchor{af0}
@deffn {Function} multiprocessing.cpu_count ()

Return the number of CPUs in the system.

This number is not equivalent to the number of CPUs the current process can
use.  The number of usable CPUs can be obtained with
@code{len(os.sched_getaffinity(0))}

When the number of CPUs cannot be determined a @ref{8a3,,NotImplementedError}
is raised.


@subsubheading See also


@ref{298,,os.cpu_count()}

@end deffn

@geindex current_process() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing current_process}@anchor{2853}
@deffn {Function} multiprocessing.current_process ()

Return the @ref{681,,Process} object corresponding to the current process.

An analogue of @ref{31e,,threading.current_thread()}.
@end deffn

@geindex parent_process() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing parent_process}@anchor{2854}
@deffn {Function} multiprocessing.parent_process ()

Return the @ref{681,,Process} object corresponding to the parent process of
the @ref{2853,,current_process()}. For the main process, @code{parent_process} will
be @code{None}.


New in version 3.8.

@end deffn

@geindex freeze_support() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing freeze_support}@anchor{2855}
@deffn {Function} multiprocessing.freeze_support ()

Add support for when a program which uses @ref{b3,,multiprocessing} has been
frozen to produce a Windows executable.  (Has been tested with @strong{py2exe},
@strong{PyInstaller} and @strong{cx_Freeze}.)

One needs to call this function straight after the @code{if __name__ ==
'__main__'} line of the main module.  For example:

@example
from multiprocessing import Process, freeze_support

def f():
    print('hello world!')

if __name__ == '__main__':
    freeze_support()
    Process(target=f).start()
@end example

If the @code{freeze_support()} line is omitted then trying to run the frozen
executable will raise @ref{5af,,RuntimeError}.

Calling @code{freeze_support()} has no effect when invoked on any operating
system other than Windows.  In addition, if the module is being run
normally by the Python interpreter on Windows (the program has not been
frozen), then @code{freeze_support()} has no effect.
@end deffn

@geindex get_all_start_methods() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing get_all_start_methods}@anchor{ae8}
@deffn {Function} multiprocessing.get_all_start_methods ()

Returns a list of the supported start methods, the first of which
is the default.  The possible start methods are @code{'fork'},
@code{'spawn'} and @code{'forkserver'}.  On Windows only @code{'spawn'} is
available.  On Unix @code{'fork'} and @code{'spawn'} are always
supported, with @code{'fork'} being the default.


New in version 3.4.

@end deffn

@geindex get_context() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing get_context}@anchor{aeb}
@deffn {Function} multiprocessing.get_context (method=None)

Return a context object which has the same attributes as the
@ref{b3,,multiprocessing} module.

If @emph{method} is @code{None} then the default context is returned.
Otherwise @emph{method} should be @code{'fork'}, @code{'spawn'},
@code{'forkserver'}.  @ref{17c,,ValueError} is raised if the specified
start method is not available.


New in version 3.4.

@end deffn

@geindex get_start_method() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing get_start_method}@anchor{ae9}
@deffn {Function} multiprocessing.get_start_method (allow_none=False)

Return the name of start method used for starting processes.

If the start method has not been fixed and @emph{allow_none} is false,
then the start method is fixed to the default and the name is
returned.  If the start method has not been fixed and @emph{allow_none}
is true then @code{None} is returned.

The return value can be @code{'fork'}, @code{'spawn'}, @code{'forkserver'}
or @code{None}.  @code{'fork'} is the default on Unix, while @code{'spawn'} is
the default on Windows and macOS.
@end deffn


Changed in version 3.8: On macOS, the @emph{spawn} start method is now the default.  The @emph{fork} start
method should be considered unsafe as it can lead to crashes of the
subprocess. See bpo-33725@footnote{https://bugs.python.org/issue33725}.


New in version 3.4.



@geindex set_executable() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing set_executable}@anchor{2856}
@deffn {Function} multiprocessing.set_executable (executable)

Set the path of the Python interpreter to use when starting a child process.
(By default @ref{585,,sys.executable} is used).  Embedders will probably need to
do some thing like

@example
set_executable(os.path.join(sys.exec_prefix, 'pythonw.exe'))
@end example

before they can create child processes.


Changed in version 3.4: Now supported on Unix when the @code{'spawn'} start method is used.

@end deffn

@geindex set_start_method() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing set_start_method}@anchor{aea}
@deffn {Function} multiprocessing.set_start_method (method)

Set the method which should be used to start child processes.
@emph{method} can be @code{'fork'}, @code{'spawn'} or @code{'forkserver'}.

Note that this should be called at most once, and it should be
protected inside the @code{if __name__ == '__main__'} clause of the
main module.


New in version 3.4.

@end deffn

@cartouche
@quotation Note 
@ref{b3,,multiprocessing} contains no analogues of
@ref{31f,,threading.active_count()}, @ref{110d,,threading.enumerate()},
@ref{2ce,,threading.settrace()}, @ref{2cf,,threading.setprofile()},
@ref{cf4,,threading.Timer}, or @ref{278d,,threading.local}.
@end quotation
@end cartouche

@node Connection Objects<2>,Synchronization primitives,Miscellaneous<3>,Reference
@anchor{library/multiprocessing connection-objects}@anchor{2857}
@subsubsection Connection Objects


Connection objects allow the sending and receiving of picklable objects or
strings.  They can be thought of as message oriented connected sockets.

Connection objects are usually created using
@ref{281f,,Pipe} – see also
@ref{2858,,Listeners and Clients}.

@geindex Connection (class in multiprocessing.connection)
@anchor{library/multiprocessing multiprocessing connection Connection}@anchor{2847}
@deffn {Class} multiprocessing.connection.Connection

@geindex send() (multiprocessing.connection.Connection method)
@anchor{library/multiprocessing multiprocessing connection Connection send}@anchor{2859}
@deffn {Method} send (obj)

Send an object to the other end of the connection which should be read
using @ref{285a,,recv()}.

The object must be picklable.  Very large pickles (approximately 32 MiB+,
though it depends on the OS) may raise a @ref{17c,,ValueError} exception.
@end deffn

@geindex recv() (multiprocessing.connection.Connection method)
@anchor{library/multiprocessing multiprocessing connection Connection recv}@anchor{285a}
@deffn {Method} recv ()

Return an object sent from the other end of the connection using
@ref{2859,,send()}.  Blocks until there is something to receive.  Raises
@ref{e74,,EOFError} if there is nothing left to receive
and the other end was closed.
@end deffn

@geindex fileno() (multiprocessing.connection.Connection method)
@anchor{library/multiprocessing multiprocessing connection Connection fileno}@anchor{285b}
@deffn {Method} fileno ()

Return the file descriptor or handle used by the connection.
@end deffn

@geindex close() (multiprocessing.connection.Connection method)
@anchor{library/multiprocessing multiprocessing connection Connection close}@anchor{285c}
@deffn {Method} close ()

Close the connection.

This is called automatically when the connection is garbage collected.
@end deffn

@geindex poll() (multiprocessing.connection.Connection method)
@anchor{library/multiprocessing multiprocessing connection Connection poll}@anchor{285d}
@deffn {Method} poll ([timeout])

Return whether there is any data available to be read.

If @emph{timeout} is not specified then it will return immediately.  If
@emph{timeout} is a number then this specifies the maximum time in seconds to
block.  If @emph{timeout} is @code{None} then an infinite timeout is used.

Note that multiple connection objects may be polled at once by
using @ref{c72,,multiprocessing.connection.wait()}.
@end deffn

@geindex send_bytes() (multiprocessing.connection.Connection method)
@anchor{library/multiprocessing multiprocessing connection Connection send_bytes}@anchor{285e}
@deffn {Method} send_bytes (buffer[, offset[, size]])

Send byte data from a @ref{87e,,bytes-like object} as a complete message.

If @emph{offset} is given then data is read from that position in @emph{buffer}.  If
@emph{size} is given then that many bytes will be read from buffer.  Very large
buffers (approximately 32 MiB+, though it depends on the OS) may raise a
@ref{17c,,ValueError} exception
@end deffn

@geindex recv_bytes() (multiprocessing.connection.Connection method)
@anchor{library/multiprocessing multiprocessing connection Connection recv_bytes}@anchor{285f}
@deffn {Method} recv_bytes ([maxlength])

Return a complete message of byte data sent from the other end of the
connection as a string.  Blocks until there is something to receive.
Raises @ref{e74,,EOFError} if there is nothing left
to receive and the other end has closed.

If @emph{maxlength} is specified and the message is longer than @emph{maxlength}
then @ref{2a1,,OSError} is raised and the connection will no longer be
readable.


Changed in version 3.3: This function used to raise @ref{be8,,IOError}, which is now an
alias of @ref{2a1,,OSError}.

@end deffn

@geindex recv_bytes_into() (multiprocessing.connection.Connection method)
@anchor{library/multiprocessing multiprocessing connection Connection recv_bytes_into}@anchor{2860}
@deffn {Method} recv_bytes_into (buffer[, offset])

Read into @emph{buffer} a complete message of byte data sent from the other end
of the connection and return the number of bytes in the message.  Blocks
until there is something to receive.  Raises
@ref{e74,,EOFError} if there is nothing left to receive and the other end was
closed.

@emph{buffer} must be a writable @ref{87e,,bytes-like object}.  If
@emph{offset} is given then the message will be written into the buffer from
that position.  Offset must be a non-negative integer less than the
length of @emph{buffer} (in bytes).

If the buffer is too short then a @code{BufferTooShort} exception is
raised and the complete message is available as @code{e.args[0]} where @code{e}
is the exception instance.
@end deffn


Changed in version 3.3: Connection objects themselves can now be transferred between processes
using @ref{2859,,Connection.send()} and @ref{285a,,Connection.recv()}.



New in version 3.3: Connection objects now support the context management protocol – see
@ref{1810,,Context Manager Types}.  @ref{1b35,,__enter__()} returns the
connection object, and @ref{1b37,,__exit__()} calls @ref{285c,,close()}.

@end deffn

For example:

@example
>>> from multiprocessing import Pipe
>>> a, b = Pipe()
>>> a.send([1, 'hello', None])
>>> b.recv()
[1, 'hello', None]
>>> b.send_bytes(b'thank you')
>>> a.recv_bytes()
b'thank you'
>>> import array
>>> arr1 = array.array('i', range(5))
>>> arr2 = array.array('i', [0] * 10)
>>> a.send_bytes(arr1)
>>> count = b.recv_bytes_into(arr2)
>>> assert count == len(arr1) * arr1.itemsize
>>> arr2
array('i', [0, 1, 2, 3, 4, 0, 0, 0, 0, 0])
@end example
@anchor{library/multiprocessing multiprocessing-recv-pickle-security}@anchor{2861}
@cartouche
@quotation Warning 
The @ref{285a,,Connection.recv()} method automatically unpickles the data it
receives, which can be a security risk unless you can trust the process
which sent the message.

Therefore, unless the connection object was produced using @code{Pipe()} you
should only use the @ref{285a,,recv()} and @ref{2859,,send()}
methods after performing some sort of authentication.  See
@ref{2835,,Authentication keys}.
@end quotation
@end cartouche

@cartouche
@quotation Warning 
If a process is killed while it is trying to read or write to a pipe then
the data in the pipe is likely to become corrupted, because it may become
impossible to be sure where the message boundaries lie.
@end quotation
@end cartouche

@node Synchronization primitives,Shared ctypes Objects,Connection Objects<2>,Reference
@anchor{library/multiprocessing synchronization-primitives}@anchor{2862}
@subsubsection Synchronization primitives


Generally synchronization primitives are not as necessary in a multiprocess
program as they are in a multithreaded program.  See the documentation for
@ref{109,,threading} module.

Note that one can also create synchronization primitives by using a manager
object – see @ref{2840,,Managers}.

@geindex Barrier (class in multiprocessing)
@anchor{library/multiprocessing multiprocessing Barrier}@anchor{282a}
@deffn {Class} multiprocessing.Barrier (parties[, action[, timeout]])

A barrier object: a clone of @ref{d85,,threading.Barrier}.


New in version 3.3.

@end deffn

@geindex BoundedSemaphore (class in multiprocessing)
@anchor{library/multiprocessing multiprocessing BoundedSemaphore}@anchor{2828}
@deffn {Class} multiprocessing.BoundedSemaphore ([value])

A bounded semaphore object: a close analog of
@ref{cf2,,threading.BoundedSemaphore}.

A solitary difference from its close analog exists: its @code{acquire} method’s
first argument is named @emph{block}, as is consistent with @ref{2863,,Lock.acquire()}.

@cartouche
@quotation Note 
On macOS, this is indistinguishable from @ref{2827,,Semaphore} because
@code{sem_getvalue()} is not implemented on that platform.
@end quotation
@end cartouche
@end deffn

@geindex Condition (class in multiprocessing)
@anchor{library/multiprocessing multiprocessing Condition}@anchor{2829}
@deffn {Class} multiprocessing.Condition ([lock])

A condition variable: an alias for @ref{cf0,,threading.Condition}.

If @emph{lock} is specified then it should be a @ref{2825,,Lock} or @ref{2826,,RLock}
object from @ref{b3,,multiprocessing}.


Changed in version 3.3: The @ref{27ff,,wait_for()} method was added.

@end deffn

@geindex Event (class in multiprocessing)
@anchor{library/multiprocessing multiprocessing Event}@anchor{10be}
@deffn {Class} multiprocessing.Event

A clone of @ref{cf3,,threading.Event}.
@end deffn

@geindex Lock (class in multiprocessing)
@anchor{library/multiprocessing multiprocessing Lock}@anchor{2825}
@deffn {Class} multiprocessing.Lock

A non-recursive lock object: a close analog of @ref{27f5,,threading.Lock}.
Once a process or thread has acquired a lock, subsequent attempts to
acquire it from any process or thread will block until it is released;
any process or thread may release it.  The concepts and behaviors of
@ref{27f5,,threading.Lock} as it applies to threads are replicated here in
@ref{2825,,multiprocessing.Lock} as it applies to either processes or threads,
except as noted.

Note that @ref{2825,,Lock} is actually a factory function which returns an
instance of @code{multiprocessing.synchronize.Lock} initialized with a
default context.

@ref{2825,,Lock} supports the @ref{166,,context manager} protocol and thus may be
used in @ref{168,,with} statements.

@geindex acquire() (multiprocessing.Lock method)
@anchor{library/multiprocessing multiprocessing Lock acquire}@anchor{2863}
@deffn {Method} acquire (block=True, timeout=None)

Acquire a lock, blocking or non-blocking.

With the @emph{block} argument set to @code{True} (the default), the method call
will block until the lock is in an unlocked state, then set it to locked
and return @code{True}.  Note that the name of this first argument differs
from that in @ref{192,,threading.Lock.acquire()}.

With the @emph{block} argument set to @code{False}, the method call does not
block.  If the lock is currently in a locked state, return @code{False};
otherwise set the lock to a locked state and return @code{True}.

When invoked with a positive, floating-point value for @emph{timeout}, block
for at most the number of seconds specified by @emph{timeout} as long as
the lock can not be acquired.  Invocations with a negative value for
@emph{timeout} are equivalent to a @emph{timeout} of zero.  Invocations with a
@emph{timeout} value of @code{None} (the default) set the timeout period to
infinite.  Note that the treatment of negative or @code{None} values for
@emph{timeout} differs from the implemented behavior in
@ref{192,,threading.Lock.acquire()}.  The @emph{timeout} argument has no practical
implications if the @emph{block} argument is set to @code{False} and is thus
ignored.  Returns @code{True} if the lock has been acquired or @code{False} if
the timeout period has elapsed.
@end deffn

@geindex release() (multiprocessing.Lock method)
@anchor{library/multiprocessing multiprocessing Lock release}@anchor{2864}
@deffn {Method} release ()

Release a lock.  This can be called from any process or thread, not only
the process or thread which originally acquired the lock.

Behavior is the same as in @ref{27f3,,threading.Lock.release()} except that
when invoked on an unlocked lock, a @ref{17c,,ValueError} is raised.
@end deffn
@end deffn

@geindex RLock (class in multiprocessing)
@anchor{library/multiprocessing multiprocessing RLock}@anchor{2826}
@deffn {Class} multiprocessing.RLock

A recursive lock object: a close analog of @ref{e09,,threading.RLock}.  A
recursive lock must be released by the process or thread that acquired it.
Once a process or thread has acquired a recursive lock, the same process
or thread may acquire it again without blocking; that process or thread
must release it once for each time it has been acquired.

Note that @ref{2826,,RLock} is actually a factory function which returns an
instance of @code{multiprocessing.synchronize.RLock} initialized with a
default context.

@ref{2826,,RLock} supports the @ref{166,,context manager} protocol and thus may be
used in @ref{168,,with} statements.

@geindex acquire() (multiprocessing.RLock method)
@anchor{library/multiprocessing multiprocessing RLock acquire}@anchor{2865}
@deffn {Method} acquire (block=True, timeout=None)

Acquire a lock, blocking or non-blocking.

When invoked with the @emph{block} argument set to @code{True}, block until the
lock is in an unlocked state (not owned by any process or thread) unless
the lock is already owned by the current process or thread.  The current
process or thread then takes ownership of the lock (if it does not
already have ownership) and the recursion level inside the lock increments
by one, resulting in a return value of @code{True}.  Note that there are
several differences in this first argument’s behavior compared to the
implementation of @ref{9eb,,threading.RLock.acquire()}, starting with the name
of the argument itself.

When invoked with the @emph{block} argument set to @code{False}, do not block.
If the lock has already been acquired (and thus is owned) by another
process or thread, the current process or thread does not take ownership
and the recursion level within the lock is not changed, resulting in
a return value of @code{False}.  If the lock is in an unlocked state, the
current process or thread takes ownership and the recursion level is
incremented, resulting in a return value of @code{True}.

Use and behaviors of the @emph{timeout} argument are the same as in
@ref{2863,,Lock.acquire()}.  Note that some of these behaviors of @emph{timeout}
differ from the implemented behaviors in @ref{9eb,,threading.RLock.acquire()}.
@end deffn

@geindex release() (multiprocessing.RLock method)
@anchor{library/multiprocessing multiprocessing RLock release}@anchor{2866}
@deffn {Method} release ()

Release a lock, decrementing the recursion level.  If after the
decrement the recursion level is zero, reset the lock to unlocked (not
owned by any process or thread) and if any other processes or threads
are blocked waiting for the lock to become unlocked, allow exactly one
of them to proceed.  If after the decrement the recursion level is still
nonzero, the lock remains locked and owned by the calling process or
thread.

Only call this method when the calling process or thread owns the lock.
An @ref{46e,,AssertionError} is raised if this method is called by a process
or thread other than the owner or if the lock is in an unlocked (unowned)
state.  Note that the type of exception raised in this situation
differs from the implemented behavior in @ref{27f9,,threading.RLock.release()}.
@end deffn
@end deffn

@geindex Semaphore (class in multiprocessing)
@anchor{library/multiprocessing multiprocessing Semaphore}@anchor{2827}
@deffn {Class} multiprocessing.Semaphore ([value])

A semaphore object: a close analog of @ref{cf1,,threading.Semaphore}.

A solitary difference from its close analog exists: its @code{acquire} method’s
first argument is named @emph{block}, as is consistent with @ref{2863,,Lock.acquire()}.
@end deffn

@cartouche
@quotation Note 
On macOS, @code{sem_timedwait} is unsupported, so calling @code{acquire()} with
a timeout will emulate that function’s behavior using a sleeping loop.
@end quotation
@end cartouche

@cartouche
@quotation Note 
If the SIGINT signal generated by @code{Ctrl-C} arrives while the main thread is
blocked by a call to @code{BoundedSemaphore.acquire()}, @ref{2863,,Lock.acquire()},
@ref{2865,,RLock.acquire()}, @code{Semaphore.acquire()}, @code{Condition.acquire()}
or @code{Condition.wait()} then the call will be immediately interrupted and
@ref{4ca,,KeyboardInterrupt} will be raised.

This differs from the behaviour of @ref{109,,threading} where SIGINT will be
ignored while the equivalent blocking calls are in progress.
@end quotation
@end cartouche

@cartouche
@quotation Note 
Some of this package’s functionality requires a functioning shared semaphore
implementation on the host operating system. Without one, the
@code{multiprocessing.synchronize} module will be disabled, and attempts to
import it will result in an @ref{3b8,,ImportError}. See
bpo-3770@footnote{https://bugs.python.org/issue3770} for additional information.
@end quotation
@end cartouche

@node Shared ctypes Objects,Managers,Synchronization primitives,Reference
@anchor{library/multiprocessing shared-ctypes-objects}@anchor{2867}
@subsubsection Shared @code{ctypes} Objects


It is possible to create shared objects using shared memory which can be
inherited by child processes.

@geindex Value() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing Value}@anchor{2822}
@deffn {Function} multiprocessing.Value (typecode_or_type, *args, lock=True)

Return a @ref{29,,ctypes} object allocated from shared memory.  By default the
return value is actually a synchronized wrapper for the object.  The object
itself can be accessed via the @emph{value} attribute of a @ref{2822,,Value}.

@emph{typecode_or_type} determines the type of the returned object: it is either a
ctypes type or a one character typecode of the kind used by the @ref{6,,array}
module.  @emph{*args} is passed on to the constructor for the type.

If @emph{lock} is @code{True} (the default) then a new recursive lock
object is created to synchronize access to the value.  If @emph{lock} is
a @ref{2825,,Lock} or @ref{2826,,RLock} object then that will be used to
synchronize access to the value.  If @emph{lock} is @code{False} then
access to the returned object will not be automatically protected
by a lock, so it will not necessarily be “process-safe”.

Operations like @code{+=} which involve a read and write are not
atomic.  So if, for instance, you want to atomically increment a
shared value it is insufficient to just do

@example
counter.value += 1
@end example

Assuming the associated lock is recursive (which it is by default)
you can instead do

@example
with counter.get_lock():
    counter.value += 1
@end example

Note that @emph{lock} is a keyword-only argument.
@end deffn

@geindex Array() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing Array}@anchor{2823}
@deffn {Function} multiprocessing.Array (typecode_or_type, size_or_initializer, *, lock=True)

Return a ctypes array allocated from shared memory.  By default the return
value is actually a synchronized wrapper for the array.

@emph{typecode_or_type} determines the type of the elements of the returned array:
it is either a ctypes type or a one character typecode of the kind used by
the @ref{6,,array} module.  If @emph{size_or_initializer} is an integer, then it
determines the length of the array, and the array will be initially zeroed.
Otherwise, @emph{size_or_initializer} is a sequence which is used to initialize
the array and whose length determines the length of the array.

If @emph{lock} is @code{True} (the default) then a new lock object is created to
synchronize access to the value.  If @emph{lock} is a @ref{2825,,Lock} or
@ref{2826,,RLock} object then that will be used to synchronize access to the
value.  If @emph{lock} is @code{False} then access to the returned object will not be
automatically protected by a lock, so it will not necessarily be
“process-safe”.

Note that @emph{lock} is a keyword only argument.

Note that an array of @ref{2759,,ctypes.c_char} has @emph{value} and @emph{raw}
attributes which allow one to use it to store and retrieve strings.
@end deffn

@menu
* The multiprocessing.sharedctypes module: The multiprocessing sharedctypes module. 

@end menu

@node The multiprocessing sharedctypes module,,,Shared ctypes Objects
@anchor{library/multiprocessing module-multiprocessing sharedctypes}@anchor{b9}@anchor{library/multiprocessing the-multiprocessing-sharedctypes-module}@anchor{2868}
@subsubsection The @code{multiprocessing.sharedctypes} module


@geindex module; multiprocessing.sharedctypes

The @ref{b9,,multiprocessing.sharedctypes} module provides functions for allocating
@ref{29,,ctypes} objects from shared memory which can be inherited by child
processes.

@cartouche
@quotation Note 
Although it is possible to store a pointer in shared memory remember that
this will refer to a location in the address space of a specific process.
However, the pointer is quite likely to be invalid in the context of a second
process and trying to dereference the pointer from the second process may
cause a crash.
@end quotation
@end cartouche

@geindex RawArray() (in module multiprocessing.sharedctypes)
@anchor{library/multiprocessing multiprocessing sharedctypes RawArray}@anchor{2869}
@deffn {Function} multiprocessing.sharedctypes.RawArray (typecode_or_type, size_or_initializer)

Return a ctypes array allocated from shared memory.

@emph{typecode_or_type} determines the type of the elements of the returned array:
it is either a ctypes type or a one character typecode of the kind used by
the @ref{6,,array} module.  If @emph{size_or_initializer} is an integer then it
determines the length of the array, and the array will be initially zeroed.
Otherwise @emph{size_or_initializer} is a sequence which is used to initialize the
array and whose length determines the length of the array.

Note that setting and getting an element is potentially non-atomic – use
@ref{286a,,Array()} instead to make sure that access is automatically synchronized
using a lock.
@end deffn

@geindex RawValue() (in module multiprocessing.sharedctypes)
@anchor{library/multiprocessing multiprocessing sharedctypes RawValue}@anchor{286b}
@deffn {Function} multiprocessing.sharedctypes.RawValue (typecode_or_type, *args)

Return a ctypes object allocated from shared memory.

@emph{typecode_or_type} determines the type of the returned object: it is either a
ctypes type or a one character typecode of the kind used by the @ref{6,,array}
module.  @emph{*args} is passed on to the constructor for the type.

Note that setting and getting the value is potentially non-atomic – use
@ref{286c,,Value()} instead to make sure that access is automatically synchronized
using a lock.

Note that an array of @ref{2759,,ctypes.c_char} has @code{value} and @code{raw}
attributes which allow one to use it to store and retrieve strings – see
documentation for @ref{29,,ctypes}.
@end deffn

@geindex Array() (in module multiprocessing.sharedctypes)
@anchor{library/multiprocessing multiprocessing sharedctypes Array}@anchor{286a}
@deffn {Function} multiprocessing.sharedctypes.Array (typecode_or_type, size_or_initializer, *, lock=True)

The same as @ref{2869,,RawArray()} except that depending on the value of @emph{lock} a
process-safe synchronization wrapper may be returned instead of a raw ctypes
array.

If @emph{lock} is @code{True} (the default) then a new lock object is created to
synchronize access to the value.  If @emph{lock} is a
@ref{2825,,Lock} or @ref{2826,,RLock} object
then that will be used to synchronize access to the
value.  If @emph{lock} is @code{False} then access to the returned object will not be
automatically protected by a lock, so it will not necessarily be
“process-safe”.

Note that @emph{lock} is a keyword-only argument.
@end deffn

@geindex Value() (in module multiprocessing.sharedctypes)
@anchor{library/multiprocessing multiprocessing sharedctypes Value}@anchor{286c}
@deffn {Function} multiprocessing.sharedctypes.Value (typecode_or_type, *args, lock=True)

The same as @ref{286b,,RawValue()} except that depending on the value of @emph{lock} a
process-safe synchronization wrapper may be returned instead of a raw ctypes
object.

If @emph{lock} is @code{True} (the default) then a new lock object is created to
synchronize access to the value.  If @emph{lock} is a @ref{2825,,Lock} or
@ref{2826,,RLock} object then that will be used to synchronize access to the
value.  If @emph{lock} is @code{False} then access to the returned object will not be
automatically protected by a lock, so it will not necessarily be
“process-safe”.

Note that @emph{lock} is a keyword-only argument.
@end deffn

@geindex copy() (in module multiprocessing.sharedctypes)
@anchor{library/multiprocessing multiprocessing sharedctypes copy}@anchor{286d}
@deffn {Function} multiprocessing.sharedctypes.copy (obj)

Return a ctypes object allocated from shared memory which is a copy of the
ctypes object @emph{obj}.
@end deffn

@geindex synchronized() (in module multiprocessing.sharedctypes)
@anchor{library/multiprocessing multiprocessing sharedctypes synchronized}@anchor{998}
@deffn {Function} multiprocessing.sharedctypes.synchronized (obj[, lock])

Return a process-safe wrapper object for a ctypes object which uses @emph{lock} to
synchronize access.  If @emph{lock} is @code{None} (the default) then a
@ref{2826,,multiprocessing.RLock} object is created automatically.

A synchronized wrapper will have two methods in addition to those of the
object it wraps: @code{get_obj()} returns the wrapped object and
@code{get_lock()} returns the lock object used for synchronization.

Note that accessing the ctypes object through the wrapper can be a lot slower
than accessing the raw ctypes object.


Changed in version 3.5: Synchronized objects support the @ref{166,,context manager} protocol.

@end deffn

The table below compares the syntax for creating shared ctypes objects from
shared memory with the normal ctypes syntax.  (In the table @code{MyStruct} is some
subclass of @ref{2777,,ctypes.Structure}.)


@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

ctypes

@tab

sharedctypes using type

@tab

sharedctypes using typecode

@item

c_double(2.4)

@tab

RawValue(c_double, 2.4)

@tab

RawValue(‘d’, 2.4)

@item

MyStruct(4, 6)

@tab

RawValue(MyStruct, 4, 6)

@tab

@item

(c_short * 7)()

@tab

RawArray(c_short, 7)

@tab

RawArray(‘h’, 7)

@item

(c_int * 3)(9, 2, 8)

@tab

RawArray(c_int, (9, 2, 8))

@tab

RawArray(‘i’, (9, 2, 8))

@end multitable


Below is an example where a number of ctypes objects are modified by a child
process:

@example
from multiprocessing import Process, Lock
from multiprocessing.sharedctypes import Value, Array
from ctypes import Structure, c_double

class Point(Structure):
    _fields_ = [('x', c_double), ('y', c_double)]

def modify(n, x, s, A):
    n.value **= 2
    x.value **= 2
    s.value = s.value.upper()
    for a in A:
        a.x **= 2
        a.y **= 2

if __name__ == '__main__':
    lock = Lock()

    n = Value('i', 7)
    x = Value(c_double, 1.0/3.0, lock=False)
    s = Array('c', b'hello world', lock=lock)
    A = Array(Point, [(1.875,-6.25), (-5.75,2.0), (2.375,9.5)], lock=lock)

    p = Process(target=modify, args=(n, x, s, A))
    p.start()
    p.join()

    print(n.value)
    print(x.value)
    print(s.value)
    print([(a.x, a.y) for a in A])
@end example

The results printed are

@example
49
0.1111111111111111
HELLO WORLD
[(3.515625, 39.0625), (33.0625, 4.0), (5.640625, 90.25)]
@end example

@node Managers,Proxy Objects,Shared ctypes Objects,Reference
@anchor{library/multiprocessing managers}@anchor{286e}@anchor{library/multiprocessing multiprocessing-managers}@anchor{2840}
@subsubsection Managers


Managers provide a way to create data which can be shared between different
processes, including sharing over a network between processes running on
different machines. A manager object controls a server process which manages
@emph{shared objects}.  Other processes can access the shared objects by using
proxies.

@geindex multiprocessing.Manager() (in module multiprocessing.sharedctypes)
@anchor{library/multiprocessing multiprocessing sharedctypes multiprocessing Manager}@anchor{286f}
@deffn {Function} multiprocessing.Manager ()

Returns a started @ref{2870,,SyncManager} object which
can be used for sharing objects between processes.  The returned manager
object corresponds to a spawned child process and has methods which will
create shared objects and return corresponding proxies.
@end deffn
@anchor{library/multiprocessing module-multiprocessing managers}@anchor{b6}
@geindex module; multiprocessing.managers

Manager processes will be shutdown as soon as they are garbage collected or
their parent process exits.  The manager classes are defined in the
@ref{b6,,multiprocessing.managers} module:

@geindex BaseManager (class in multiprocessing.managers)
@anchor{library/multiprocessing multiprocessing managers BaseManager}@anchor{2871}
@deffn {Class} multiprocessing.managers.BaseManager ([address[, authkey]])

Create a BaseManager object.

Once created one should call @ref{2872,,start()} or @code{get_server().serve_forever()} to ensure
that the manager object refers to a started manager process.

@emph{address} is the address on which the manager process listens for new
connections.  If @emph{address} is @code{None} then an arbitrary one is chosen.

@emph{authkey} is the authentication key which will be used to check the
validity of incoming connections to the server process.  If
@emph{authkey} is @code{None} then @code{current_process().authkey} is used.
Otherwise @emph{authkey} is used and it must be a byte string.

@geindex start() (multiprocessing.managers.BaseManager method)
@anchor{library/multiprocessing multiprocessing managers BaseManager start}@anchor{2872}
@deffn {Method} start ([initializer[, initargs]])

Start a subprocess to start the manager.  If @emph{initializer} is not @code{None}
then the subprocess will call @code{initializer(*initargs)} when it starts.
@end deffn

@geindex get_server() (multiprocessing.managers.BaseManager method)
@anchor{library/multiprocessing multiprocessing managers BaseManager get_server}@anchor{2873}
@deffn {Method} get_server ()

Returns a @code{Server} object which represents the actual server under
the control of the Manager. The @code{Server} object supports the
@code{serve_forever()} method:

@example
>>> from multiprocessing.managers import BaseManager
>>> manager = BaseManager(address=('@w{'}, 50000), authkey=b'abc')
>>> server = manager.get_server()
>>> server.serve_forever()
@end example

@code{Server} additionally has an @ref{2874,,address} attribute.
@end deffn

@geindex connect() (multiprocessing.managers.BaseManager method)
@anchor{library/multiprocessing multiprocessing managers BaseManager connect}@anchor{2875}
@deffn {Method} connect ()

Connect a local manager object to a remote manager process:

@example
>>> from multiprocessing.managers import BaseManager
>>> m = BaseManager(address=('127.0.0.1', 50000), authkey=b'abc')
>>> m.connect()
@end example
@end deffn

@geindex shutdown() (multiprocessing.managers.BaseManager method)
@anchor{library/multiprocessing multiprocessing managers BaseManager shutdown}@anchor{2876}
@deffn {Method} shutdown ()

Stop the process used by the manager.  This is only available if
@ref{2872,,start()} has been used to start the server process.

This can be called multiple times.
@end deffn

@geindex register() (multiprocessing.managers.BaseManager method)
@anchor{library/multiprocessing multiprocessing managers BaseManager register}@anchor{2877}
@deffn {Method} register (typeid[, callable[, proxytype[, exposed[, method_to_typeid[, create_method]]]]])

A classmethod which can be used for registering a type or callable with
the manager class.

@emph{typeid} is a “type identifier” which is used to identify a particular
type of shared object.  This must be a string.

@emph{callable} is a callable used for creating objects for this type
identifier.  If a manager instance will be connected to the
server using the @ref{2875,,connect()} method, or if the
@emph{create_method} argument is @code{False} then this can be left as
@code{None}.

@emph{proxytype} is a subclass of @ref{2878,,BaseProxy} which is used to create
proxies for shared objects with this @emph{typeid}.  If @code{None} then a proxy
class is created automatically.

@emph{exposed} is used to specify a sequence of method names which proxies for
this typeid should be allowed to access using
@ref{2879,,BaseProxy._callmethod()}.  (If @emph{exposed} is @code{None} then
@code{proxytype._exposed_} is used instead if it exists.)  In the case
where no exposed list is specified, all “public methods” of the shared
object will be accessible.  (Here a “public method” means any attribute
which has a @ref{1795,,__call__()} method and whose name does not begin
with @code{'_'}.)

@emph{method_to_typeid} is a mapping used to specify the return type of those
exposed methods which should return a proxy.  It maps method names to
typeid strings.  (If @emph{method_to_typeid} is @code{None} then
@code{proxytype._method_to_typeid_} is used instead if it exists.)  If a
method’s name is not a key of this mapping or if the mapping is @code{None}
then the object returned by the method will be copied by value.

@emph{create_method} determines whether a method should be created with name
@emph{typeid} which can be used to tell the server process to create a new
shared object and return a proxy for it.  By default it is @code{True}.
@end deffn

@ref{2871,,BaseManager} instances also have one read-only property:

@geindex address (multiprocessing.managers.BaseManager attribute)
@anchor{library/multiprocessing multiprocessing managers BaseManager address}@anchor{2874}
@deffn {Attribute} address

The address used by the manager.
@end deffn


Changed in version 3.3: Manager objects support the context management protocol – see
@ref{1810,,Context Manager Types}.  @ref{1b35,,__enter__()} starts the
server process (if it has not already started) and then returns the
manager object.  @ref{1b37,,__exit__()} calls @ref{2876,,shutdown()}.

In previous versions @ref{1b35,,__enter__()} did not start the
manager’s server process if it was not already started.

@end deffn

@geindex SyncManager (class in multiprocessing.managers)
@anchor{library/multiprocessing multiprocessing managers SyncManager}@anchor{2870}
@deffn {Class} multiprocessing.managers.SyncManager

A subclass of @ref{2871,,BaseManager} which can be used for the synchronization
of processes.  Objects of this type are returned by
@code{multiprocessing.Manager()}.

Its methods create and return @ref{803,,Proxy Objects} for a
number of commonly used data types to be synchronized across processes.
This notably includes shared lists and dictionaries.

@geindex Barrier() (multiprocessing.managers.SyncManager method)
@anchor{library/multiprocessing multiprocessing managers SyncManager Barrier}@anchor{287a}
@deffn {Method} Barrier (parties[, action[, timeout]])

Create a shared @ref{d85,,threading.Barrier} object and return a
proxy for it.


New in version 3.3.

@end deffn

@geindex BoundedSemaphore() (multiprocessing.managers.SyncManager method)
@anchor{library/multiprocessing multiprocessing managers SyncManager BoundedSemaphore}@anchor{287b}
@deffn {Method} BoundedSemaphore ([value])

Create a shared @ref{cf2,,threading.BoundedSemaphore} object and return a
proxy for it.
@end deffn

@geindex Condition() (multiprocessing.managers.SyncManager method)
@anchor{library/multiprocessing multiprocessing managers SyncManager Condition}@anchor{287c}
@deffn {Method} Condition ([lock])

Create a shared @ref{cf0,,threading.Condition} object and return a proxy for
it.

If @emph{lock} is supplied then it should be a proxy for a
@ref{27f5,,threading.Lock} or @ref{e09,,threading.RLock} object.


Changed in version 3.3: The @ref{27ff,,wait_for()} method was added.

@end deffn

@geindex Event() (multiprocessing.managers.SyncManager method)
@anchor{library/multiprocessing multiprocessing managers SyncManager Event}@anchor{287d}
@deffn {Method} Event ()

Create a shared @ref{cf3,,threading.Event} object and return a proxy for it.
@end deffn

@geindex Lock() (multiprocessing.managers.SyncManager method)
@anchor{library/multiprocessing multiprocessing managers SyncManager Lock}@anchor{287e}
@deffn {Method} Lock ()

Create a shared @ref{27f5,,threading.Lock} object and return a proxy for it.
@end deffn

@geindex Namespace() (multiprocessing.managers.SyncManager method)
@anchor{library/multiprocessing multiprocessing managers SyncManager Namespace}@anchor{287f}
@deffn {Method} Namespace ()

Create a shared @ref{2824,,Namespace} object and return a proxy for it.
@end deffn

@geindex Queue() (multiprocessing.managers.SyncManager method)
@anchor{library/multiprocessing multiprocessing managers SyncManager Queue}@anchor{2880}
@deffn {Method} Queue ([maxsize])

Create a shared @ref{f18,,queue.Queue} object and return a proxy for it.
@end deffn

@geindex RLock() (multiprocessing.managers.SyncManager method)
@anchor{library/multiprocessing multiprocessing managers SyncManager RLock}@anchor{2881}
@deffn {Method} RLock ()

Create a shared @ref{e09,,threading.RLock} object and return a proxy for it.
@end deffn

@geindex Semaphore() (multiprocessing.managers.SyncManager method)
@anchor{library/multiprocessing multiprocessing managers SyncManager Semaphore}@anchor{2882}
@deffn {Method} Semaphore ([value])

Create a shared @ref{cf1,,threading.Semaphore} object and return a proxy for
it.
@end deffn

@geindex Array() (multiprocessing.managers.SyncManager method)
@anchor{library/multiprocessing multiprocessing managers SyncManager Array}@anchor{2883}
@deffn {Method} Array (typecode, sequence)

Create an array and return a proxy for it.
@end deffn

@geindex Value() (multiprocessing.managers.SyncManager method)
@anchor{library/multiprocessing multiprocessing managers SyncManager Value}@anchor{2884}
@deffn {Method} Value (typecode, value)

Create an object with a writable @code{value} attribute and return a proxy
for it.
@end deffn

@geindex dict() (multiprocessing.managers.SyncManager method)
@anchor{library/multiprocessing multiprocessing managers SyncManager dict}@anchor{2885}
@deffn {Method} dict ()

@deffnx {Method} dict (mapping)

@deffnx {Method} dict (sequence)

Create a shared @ref{3b0,,dict} object and return a proxy for it.
@end deffn

@geindex list() (multiprocessing.managers.SyncManager method)
@anchor{library/multiprocessing multiprocessing managers SyncManager list}@anchor{2886}
@deffn {Method} list ()

@deffnx {Method} list (sequence)

Create a shared @ref{43a,,list} object and return a proxy for it.
@end deffn


Changed in version 3.6: Shared objects are capable of being nested.  For example, a shared
container object such as a shared list can contain other shared objects
which will all be managed and synchronized by the @ref{2870,,SyncManager}.

@end deffn

@geindex Namespace (class in multiprocessing.managers)
@anchor{library/multiprocessing multiprocessing managers Namespace}@anchor{2824}
@deffn {Class} multiprocessing.managers.Namespace

A type that can register with @ref{2870,,SyncManager}.

A namespace object has no public methods, but does have writable attributes.
Its representation shows the values of its attributes.

However, when using a proxy for a namespace object, an attribute beginning
with @code{'_'} will be an attribute of the proxy and not an attribute of the
referent:

@example
>>> manager = multiprocessing.Manager()
>>> Global = manager.Namespace()
>>> Global.x = 10
>>> Global.y = 'hello'
>>> Global._z = 12.3    # this is an attribute of the proxy
>>> print(Global)
Namespace(x=10, y='hello')
@end example
@end deffn

@menu
* Customized managers:: 
* Using a remote manager:: 

@end menu

@node Customized managers,Using a remote manager,,Managers
@anchor{library/multiprocessing customized-managers}@anchor{2887}
@subsubsection Customized managers


To create one’s own manager, one creates a subclass of @ref{2871,,BaseManager} and
uses the @ref{2877,,register()} classmethod to register new types or
callables with the manager class.  For example:

@example
from multiprocessing.managers import BaseManager

class MathsClass:
    def add(self, x, y):
        return x + y
    def mul(self, x, y):
        return x * y

class MyManager(BaseManager):
    pass

MyManager.register('Maths', MathsClass)

if __name__ == '__main__':
    with MyManager() as manager:
        maths = manager.Maths()
        print(maths.add(4, 3))         # prints 7
        print(maths.mul(7, 8))         # prints 56
@end example

@node Using a remote manager,,Customized managers,Managers
@anchor{library/multiprocessing using-a-remote-manager}@anchor{2888}
@subsubsection Using a remote manager


It is possible to run a manager server on one machine and have clients use it
from other machines (assuming that the firewalls involved allow it).

Running the following commands creates a server for a single shared queue which
remote clients can access:

@example
>>> from multiprocessing.managers import BaseManager
>>> from queue import Queue
>>> queue = Queue()
>>> class QueueManager(BaseManager): pass
>>> QueueManager.register('get_queue', callable=lambda:queue)
>>> m = QueueManager(address=('@w{'}, 50000), authkey=b'abracadabra')
>>> s = m.get_server()
>>> s.serve_forever()
@end example

One client can access the server as follows:

@example
>>> from multiprocessing.managers import BaseManager
>>> class QueueManager(BaseManager): pass
>>> QueueManager.register('get_queue')
>>> m = QueueManager(address=('foo.bar.org', 50000), authkey=b'abracadabra')
>>> m.connect()
>>> queue = m.get_queue()
>>> queue.put('hello')
@end example

Another client can also use it:

@example
>>> from multiprocessing.managers import BaseManager
>>> class QueueManager(BaseManager): pass
>>> QueueManager.register('get_queue')
>>> m = QueueManager(address=('foo.bar.org', 50000), authkey=b'abracadabra')
>>> m.connect()
>>> queue = m.get_queue()
>>> queue.get()
'hello'
@end example

Local processes can also access that queue, using the code from above on the
client to access it remotely:

@example
>>> from multiprocessing import Process, Queue
>>> from multiprocessing.managers import BaseManager
>>> class Worker(Process):
...     def __init__(self, q):
...         self.q = q
...         super().__init__()
...     def run(self):
...         self.q.put('local hello')
...
>>> queue = Queue()
>>> w = Worker(queue)
>>> w.start()
>>> class QueueManager(BaseManager): pass
...
>>> QueueManager.register('get_queue', callable=lambda: queue)
>>> m = QueueManager(address=('@w{'}, 50000), authkey=b'abracadabra')
>>> s = m.get_server()
>>> s.serve_forever()
@end example

@node Proxy Objects,Process Pools,Managers,Reference
@anchor{library/multiprocessing multiprocessing-proxy-objects}@anchor{803}@anchor{library/multiprocessing proxy-objects}@anchor{2889}
@subsubsection Proxy Objects


A proxy is an object which @emph{refers} to a shared object which lives (presumably)
in a different process.  The shared object is said to be the @emph{referent} of the
proxy.  Multiple proxy objects may have the same referent.

A proxy object has methods which invoke corresponding methods of its referent
(although not every method of the referent will necessarily be available through
the proxy).  In this way, a proxy can be used just like its referent can:

@example
>>> from multiprocessing import Manager
>>> manager = Manager()
>>> l = manager.list([i*i for i in range(10)])
>>> print(l)
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>> print(repr(l))
<ListProxy object, typeid 'list' at 0x...>
>>> l[4]
16
>>> l[2:5]
[4, 9, 16]
@end example

Notice that applying @ref{2ef,,str()} to a proxy will return the representation of
the referent, whereas applying @ref{27d,,repr()} will return the representation of
the proxy.

An important feature of proxy objects is that they are picklable so they can be
passed between processes.  As such, a referent can contain
@ref{803,,Proxy Objects}.  This permits nesting of these managed
lists, dicts, and other @ref{803,,Proxy Objects}:

@example
>>> a = manager.list()
>>> b = manager.list()
>>> a.append(b)         # referent of a now contains referent of b
>>> print(a, b)
[<ListProxy object, typeid 'list' at ...>] []
>>> b.append('hello')
>>> print(a[0], b)
['hello'] ['hello']
@end example

Similarly, dict and list proxies may be nested inside one another:

@example
>>> l_outer = manager.list([ manager.dict() for i in range(2) ])
>>> d_first_inner = l_outer[0]
>>> d_first_inner['a'] = 1
>>> d_first_inner['b'] = 2
>>> l_outer[1]['c'] = 3
>>> l_outer[1]['z'] = 26
>>> print(l_outer[0])
@{'a': 1, 'b': 2@}
>>> print(l_outer[1])
@{'c': 3, 'z': 26@}
@end example

If standard (non-proxy) @ref{43a,,list} or @ref{3b0,,dict} objects are contained
in a referent, modifications to those mutable values will not be propagated
through the manager because the proxy has no way of knowing when the values
contained within are modified.  However, storing a value in a container proxy
(which triggers a @code{__setitem__} on the proxy object) does propagate through
the manager and so to effectively modify such an item, one could re-assign the
modified value to the container proxy:

@example
# create a list proxy and append a mutable object (a dictionary)
lproxy = manager.list()
lproxy.append(@{@})
# now mutate the dictionary
d = lproxy[0]
d['a'] = 1
d['b'] = 2
# at this point, the changes to d are not yet synced, but by
# updating the dictionary, the proxy is notified of the change
lproxy[0] = d
@end example

This approach is perhaps less convenient than employing nested
@ref{803,,Proxy Objects} for most use cases but also
demonstrates a level of control over the synchronization.

@cartouche
@quotation Note 
The proxy types in @ref{b3,,multiprocessing} do nothing to support comparisons
by value.  So, for instance, we have:

@example
>>> manager.list([1,2,3]) == [1,2,3]
False
@end example

One should just use a copy of the referent instead when making comparisons.
@end quotation
@end cartouche

@geindex BaseProxy (class in multiprocessing.managers)
@anchor{library/multiprocessing multiprocessing managers BaseProxy}@anchor{2878}
@deffn {Class} multiprocessing.managers.BaseProxy

Proxy objects are instances of subclasses of @ref{2878,,BaseProxy}.

@geindex _callmethod() (multiprocessing.managers.BaseProxy method)
@anchor{library/multiprocessing multiprocessing managers BaseProxy _callmethod}@anchor{2879}
@deffn {Method} _callmethod (methodname[, args[, kwds]])

Call and return the result of a method of the proxy’s referent.

If @code{proxy} is a proxy whose referent is @code{obj} then the expression

@example
proxy._callmethod(methodname, args, kwds)
@end example

will evaluate the expression

@example
getattr(obj, methodname)(*args, **kwds)
@end example

in the manager’s process.

The returned value will be a copy of the result of the call or a proxy to
a new shared object – see documentation for the @emph{method_to_typeid}
argument of @ref{2877,,BaseManager.register()}.

If an exception is raised by the call, then is re-raised by
@ref{2879,,_callmethod()}.  If some other exception is raised in the manager’s
process then this is converted into a @code{RemoteError} exception and is
raised by @ref{2879,,_callmethod()}.

Note in particular that an exception will be raised if @emph{methodname} has
not been @emph{exposed}.

An example of the usage of @ref{2879,,_callmethod()}:

@example
>>> l = manager.list(range(10))
>>> l._callmethod('__len__')
10
>>> l._callmethod('__getitem__', (slice(2, 7),)) # equivalent to l[2:7]
[2, 3, 4, 5, 6]
>>> l._callmethod('__getitem__', (20,))          # equivalent to l[20]
Traceback (most recent call last):
...
IndexError: list index out of range
@end example
@end deffn

@geindex _getvalue() (multiprocessing.managers.BaseProxy method)
@anchor{library/multiprocessing multiprocessing managers BaseProxy _getvalue}@anchor{288a}
@deffn {Method} _getvalue ()

Return a copy of the referent.

If the referent is unpicklable then this will raise an exception.
@end deffn

@geindex __repr__() (multiprocessing.managers.BaseProxy method)
@anchor{library/multiprocessing multiprocessing managers BaseProxy __repr__}@anchor{288b}
@deffn {Method} __repr__ ()

Return a representation of the proxy object.
@end deffn

@geindex __str__() (multiprocessing.managers.BaseProxy method)
@anchor{library/multiprocessing multiprocessing managers BaseProxy __str__}@anchor{288c}
@deffn {Method} __str__ ()

Return the representation of the referent.
@end deffn
@end deffn

@menu
* Cleanup:: 

@end menu

@node Cleanup,,,Proxy Objects
@anchor{library/multiprocessing cleanup}@anchor{288d}
@subsubsection Cleanup


A proxy object uses a weakref callback so that when it gets garbage collected it
deregisters itself from the manager which owns its referent.

A shared object gets deleted from the manager process when there are no longer
any proxies referring to it.

@node Process Pools,Listeners and Clients,Proxy Objects,Reference
@anchor{library/multiprocessing module-multiprocessing pool}@anchor{b7}@anchor{library/multiprocessing process-pools}@anchor{288e}
@subsubsection Process Pools


@geindex module; multiprocessing.pool

One can create a pool of processes which will carry out tasks submitted to it
with the @ref{aec,,Pool} class.

@geindex Pool (class in multiprocessing.pool)
@anchor{library/multiprocessing multiprocessing pool Pool}@anchor{aec}
@deffn {Class} multiprocessing.pool.Pool ([processes[, initializer[, initargs[, maxtasksperchild[, context]]]]])

A process pool object which controls a pool of worker processes to which jobs
can be submitted.  It supports asynchronous results with timeouts and
callbacks and has a parallel map implementation.

@emph{processes} is the number of worker processes to use.  If @emph{processes} is
@code{None} then the number returned by @ref{298,,os.cpu_count()} is used.

If @emph{initializer} is not @code{None} then each worker process will call
@code{initializer(*initargs)} when it starts.

@emph{maxtasksperchild} is the number of tasks a worker process can complete
before it will exit and be replaced with a fresh worker process, to enable
unused resources to be freed. The default @emph{maxtasksperchild} is @code{None}, which
means worker processes will live as long as the pool.

@emph{context} can be used to specify the context used for starting
the worker processes.  Usually a pool is created using the
function @code{multiprocessing.Pool()} or the @ref{aec,,Pool()} method
of a context object.  In both cases @emph{context} is set
appropriately.

Note that the methods of the pool object should only be called by
the process which created the pool.

@cartouche
@quotation Warning 
@ref{b7,,multiprocessing.pool} objects have internal resources that need to be
properly managed (like any other resource) by using the pool as a context manager
or by calling @ref{288f,,close()} and @ref{2890,,terminate()} manually. Failure to do this
can lead to the process hanging on finalization.

Note that it is @strong{not correct} to rely on the garbage collector to destroy the pool
as CPython does not assure that the finalizer of the pool will be called
(see @ref{17a3,,object.__del__()} for more information).
@end quotation
@end cartouche


New in version 3.2: @emph{maxtasksperchild}



New in version 3.4: @emph{context}


@cartouche
@quotation Note 
Worker processes within a @ref{aec,,Pool} typically live for the complete
duration of the Pool’s work queue. A frequent pattern found in other
systems (such as Apache, mod_wsgi, etc) to free resources held by
workers is to allow a worker within a pool to complete only a set
amount of work before being exiting, being cleaned up and a new
process spawned to replace the old one. The @emph{maxtasksperchild}
argument to the @ref{aec,,Pool} exposes this ability to the end user.
@end quotation
@end cartouche

@geindex apply() (multiprocessing.pool.Pool method)
@anchor{library/multiprocessing multiprocessing pool Pool apply}@anchor{2891}
@deffn {Method} apply (func[, args[, kwds]])

Call @emph{func} with arguments @emph{args} and keyword arguments @emph{kwds}.  It blocks
until the result is ready. Given this blocks, @ref{2892,,apply_async()} is
better suited for performing work in parallel. Additionally, @emph{func}
is only executed in one of the workers of the pool.
@end deffn

@geindex apply_async() (multiprocessing.pool.Pool method)
@anchor{library/multiprocessing multiprocessing pool Pool apply_async}@anchor{2892}
@deffn {Method} apply_async (func[, args[, kwds[, callback[, error_callback]]]])

A variant of the @ref{2891,,apply()} method which returns a
@ref{2893,,AsyncResult} object.

If @emph{callback} is specified then it should be a callable which accepts a
single argument.  When the result becomes ready @emph{callback} is applied to
it, that is unless the call failed, in which case the @emph{error_callback}
is applied instead.

If @emph{error_callback} is specified then it should be a callable which
accepts a single argument.  If the target function fails, then
the @emph{error_callback} is called with the exception instance.

Callbacks should complete immediately since otherwise the thread which
handles the results will get blocked.
@end deffn

@geindex map() (multiprocessing.pool.Pool method)
@anchor{library/multiprocessing multiprocessing pool Pool map}@anchor{c77}
@deffn {Method} map (func, iterable[, chunksize])

A parallel equivalent of the @ref{2f4,,map()} built-in function (it supports only
one @emph{iterable} argument though, for multiple iterables see @ref{c74,,starmap()}).
It blocks until the result is ready.

This method chops the iterable into a number of chunks which it submits to
the process pool as separate tasks.  The (approximate) size of these
chunks can be specified by setting @emph{chunksize} to a positive integer.

Note that it may cause high memory usage for very long iterables. Consider
using @ref{2894,,imap()} or @ref{2895,,imap_unordered()} with explicit @emph{chunksize}
option for better efficiency.
@end deffn

@geindex map_async() (multiprocessing.pool.Pool method)
@anchor{library/multiprocessing multiprocessing pool Pool map_async}@anchor{c78}
@deffn {Method} map_async (func, iterable[, chunksize[, callback[, error_callback]]])

A variant of the @ref{c77,,map()} method which returns a
@ref{2893,,AsyncResult} object.

If @emph{callback} is specified then it should be a callable which accepts a
single argument.  When the result becomes ready @emph{callback} is applied to
it, that is unless the call failed, in which case the @emph{error_callback}
is applied instead.

If @emph{error_callback} is specified then it should be a callable which
accepts a single argument.  If the target function fails, then
the @emph{error_callback} is called with the exception instance.

Callbacks should complete immediately since otherwise the thread which
handles the results will get blocked.
@end deffn

@geindex imap() (multiprocessing.pool.Pool method)
@anchor{library/multiprocessing multiprocessing pool Pool imap}@anchor{2894}
@deffn {Method} imap (func, iterable[, chunksize])

A lazier version of @ref{c77,,map()}.

The @emph{chunksize} argument is the same as the one used by the @ref{c77,,map()}
method.  For very long iterables using a large value for @emph{chunksize} can
make the job complete @strong{much} faster than using the default value of
@code{1}.

Also if @emph{chunksize} is @code{1} then the @code{next()} method of the iterator
returned by the @ref{2894,,imap()} method has an optional @emph{timeout} parameter:
@code{next(timeout)} will raise @ref{283a,,multiprocessing.TimeoutError} if the
result cannot be returned within @emph{timeout} seconds.
@end deffn

@geindex imap_unordered() (multiprocessing.pool.Pool method)
@anchor{library/multiprocessing multiprocessing pool Pool imap_unordered}@anchor{2895}
@deffn {Method} imap_unordered (func, iterable[, chunksize])

The same as @ref{2894,,imap()} except that the ordering of the results from the
returned iterator should be considered arbitrary.  (Only when there is
only one worker process is the order guaranteed to be “correct”.)
@end deffn

@geindex starmap() (multiprocessing.pool.Pool method)
@anchor{library/multiprocessing multiprocessing pool Pool starmap}@anchor{c74}
@deffn {Method} starmap (func, iterable[, chunksize])

Like @ref{c77,,map()} except that the
elements of the @emph{iterable} are expected to be iterables that are
unpacked as arguments.

Hence an @emph{iterable} of @code{[(1,2), (3, 4)]} results in @code{[func(1,2),
func(3,4)]}.


New in version 3.3.

@end deffn

@geindex starmap_async() (multiprocessing.pool.Pool method)
@anchor{library/multiprocessing multiprocessing pool Pool starmap_async}@anchor{c75}
@deffn {Method} starmap_async (func, iterable[, chunksize[, callback[, error_callback]]])

A combination of @ref{c74,,starmap()} and @ref{c78,,map_async()} that iterates over
@emph{iterable} of iterables and calls @emph{func} with the iterables unpacked.
Returns a result object.


New in version 3.3.

@end deffn

@geindex close() (multiprocessing.pool.Pool method)
@anchor{library/multiprocessing multiprocessing pool Pool close}@anchor{288f}
@deffn {Method} close ()

Prevents any more tasks from being submitted to the pool.  Once all the
tasks have been completed the worker processes will exit.
@end deffn

@geindex terminate() (multiprocessing.pool.Pool method)
@anchor{library/multiprocessing multiprocessing pool Pool terminate}@anchor{2890}
@deffn {Method} terminate ()

Stops the worker processes immediately without completing outstanding
work.  When the pool object is garbage collected @ref{2890,,terminate()} will be
called immediately.
@end deffn

@geindex join() (multiprocessing.pool.Pool method)
@anchor{library/multiprocessing multiprocessing pool Pool join}@anchor{2896}
@deffn {Method} join ()

Wait for the worker processes to exit.  One must call @ref{288f,,close()} or
@ref{2890,,terminate()} before using @ref{2896,,join()}.
@end deffn


New in version 3.3: Pool objects now support the context management protocol – see
@ref{1810,,Context Manager Types}.  @ref{1b35,,__enter__()} returns the
pool object, and @ref{1b37,,__exit__()} calls @ref{2890,,terminate()}.

@end deffn

@geindex AsyncResult (class in multiprocessing.pool)
@anchor{library/multiprocessing multiprocessing pool AsyncResult}@anchor{2893}
@deffn {Class} multiprocessing.pool.AsyncResult

The class of the result returned by @ref{2892,,Pool.apply_async()} and
@ref{c78,,Pool.map_async()}.

@geindex get() (multiprocessing.pool.AsyncResult method)
@anchor{library/multiprocessing multiprocessing pool AsyncResult get}@anchor{2897}
@deffn {Method} get ([timeout])

Return the result when it arrives.  If @emph{timeout} is not @code{None} and the
result does not arrive within @emph{timeout} seconds then
@ref{283a,,multiprocessing.TimeoutError} is raised.  If the remote call raised
an exception then that exception will be reraised by @ref{2897,,get()}.
@end deffn

@geindex wait() (multiprocessing.pool.AsyncResult method)
@anchor{library/multiprocessing multiprocessing pool AsyncResult wait}@anchor{2898}
@deffn {Method} wait ([timeout])

Wait until the result is available or until @emph{timeout} seconds pass.
@end deffn

@geindex ready() (multiprocessing.pool.AsyncResult method)
@anchor{library/multiprocessing multiprocessing pool AsyncResult ready}@anchor{2899}
@deffn {Method} ready ()

Return whether the call has completed.
@end deffn

@geindex successful() (multiprocessing.pool.AsyncResult method)
@anchor{library/multiprocessing multiprocessing pool AsyncResult successful}@anchor{289a}
@deffn {Method} successful ()

Return whether the call completed without raising an exception.  Will
raise @ref{17c,,ValueError} if the result is not ready.


Changed in version 3.7: If the result is not ready, @ref{17c,,ValueError} is raised instead of
@ref{46e,,AssertionError}.

@end deffn
@end deffn

The following example demonstrates the use of a pool:

@example
from multiprocessing import Pool
import time

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(processes=4) as pool:         # start 4 worker processes
        result = pool.apply_async(f, (10,)) # evaluate "f(10)" asynchronously in a single process
        print(result.get(timeout=1))        # prints "100" unless your computer is *very* slow

        print(pool.map(f, range(10)))       # prints "[0, 1, 4,..., 81]"

        it = pool.imap(f, range(10))
        print(next(it))                     # prints "0"
        print(next(it))                     # prints "1"
        print(it.next(timeout=1))           # prints "4" unless your computer is *very* slow

        result = pool.apply_async(time.sleep, (10,))
        print(result.get(timeout=1))        # raises multiprocessing.TimeoutError
@end example

@node Listeners and Clients,Authentication keys,Process Pools,Reference
@anchor{library/multiprocessing listeners-and-clients}@anchor{289b}@anchor{library/multiprocessing module-multiprocessing connection}@anchor{b4}@anchor{library/multiprocessing multiprocessing-listeners-clients}@anchor{2858}
@subsubsection Listeners and Clients


@geindex module; multiprocessing.connection

Usually message passing between processes is done using queues or by using
@ref{2847,,Connection} objects returned by
@ref{281f,,Pipe()}.

However, the @ref{b4,,multiprocessing.connection} module allows some extra
flexibility.  It basically gives a high level message oriented API for dealing
with sockets or Windows named pipes.  It also has support for @emph{digest
authentication} using the @ref{8a,,hmac} module, and for polling
multiple connections at the same time.

@geindex deliver_challenge() (in module multiprocessing.connection)
@anchor{library/multiprocessing multiprocessing connection deliver_challenge}@anchor{289c}
@deffn {Function} multiprocessing.connection.deliver_challenge (connection, authkey)

Send a randomly generated message to the other end of the connection and wait
for a reply.

If the reply matches the digest of the message using @emph{authkey} as the key
then a welcome message is sent to the other end of the connection.  Otherwise
@ref{2839,,AuthenticationError} is raised.
@end deffn

@geindex answer_challenge() (in module multiprocessing.connection)
@anchor{library/multiprocessing multiprocessing connection answer_challenge}@anchor{289d}
@deffn {Function} multiprocessing.connection.answer_challenge (connection, authkey)

Receive a message, calculate the digest of the message using @emph{authkey} as the
key, and then send the digest back.

If a welcome message is not received, then
@ref{2839,,AuthenticationError} is raised.
@end deffn

@geindex Client() (in module multiprocessing.connection)
@anchor{library/multiprocessing multiprocessing connection Client}@anchor{289e}
@deffn {Function} multiprocessing.connection.Client (address[, family[, authkey]])

Attempt to set up a connection to the listener which is using address
@emph{address}, returning a @ref{2847,,Connection}.

The type of the connection is determined by @emph{family} argument, but this can
generally be omitted since it can usually be inferred from the format of
@emph{address}. (See @ref{289f,,Address Formats})

If @emph{authkey} is given and not None, it should be a byte string and will be
used as the secret key for an HMAC-based authentication challenge. No
authentication is done if @emph{authkey} is None.
@ref{2839,,AuthenticationError} is raised if authentication fails.
See @ref{2835,,Authentication keys}.
@end deffn

@geindex Listener (class in multiprocessing.connection)
@anchor{library/multiprocessing multiprocessing connection Listener}@anchor{1210}
@deffn {Class} multiprocessing.connection.Listener ([address[, family[, backlog[, authkey]]]])

A wrapper for a bound socket or Windows named pipe which is ‘listening’ for
connections.

@emph{address} is the address to be used by the bound socket or named pipe of the
listener object.

@cartouche
@quotation Note 
If an address of ‘0.0.0.0’ is used, the address will not be a connectable
end point on Windows. If you require a connectable end-point,
you should use ‘127.0.0.1’.
@end quotation
@end cartouche

@emph{family} is the type of socket (or named pipe) to use.  This can be one of
the strings @code{'AF_INET'} (for a TCP socket), @code{'AF_UNIX'} (for a Unix
domain socket) or @code{'AF_PIPE'} (for a Windows named pipe).  Of these only
the first is guaranteed to be available.  If @emph{family} is @code{None} then the
family is inferred from the format of @emph{address}.  If @emph{address} is also
@code{None} then a default is chosen.  This default is the family which is
assumed to be the fastest available.  See
@ref{289f,,Address Formats}.  Note that if @emph{family} is
@code{'AF_UNIX'} and address is @code{None} then the socket will be created in a
private temporary directory created using @ref{2102,,tempfile.mkstemp()}.

If the listener object uses a socket then @emph{backlog} (1 by default) is passed
to the @ref{9ca,,listen()} method of the socket once it has been
bound.

If @emph{authkey} is given and not None, it should be a byte string and will be
used as the secret key for an HMAC-based authentication challenge. No
authentication is done if @emph{authkey} is None.
@ref{2839,,AuthenticationError} is raised if authentication fails.
See @ref{2835,,Authentication keys}.

@geindex accept() (multiprocessing.connection.Listener method)
@anchor{library/multiprocessing multiprocessing connection Listener accept}@anchor{28a0}
@deffn {Method} accept ()

Accept a connection on the bound socket or named pipe of the listener
object and return a @ref{2847,,Connection} object.
If authentication is attempted and fails, then
@ref{2839,,AuthenticationError} is raised.
@end deffn

@geindex close() (multiprocessing.connection.Listener method)
@anchor{library/multiprocessing multiprocessing connection Listener close}@anchor{28a1}
@deffn {Method} close ()

Close the bound socket or named pipe of the listener object.  This is
called automatically when the listener is garbage collected.  However it
is advisable to call it explicitly.
@end deffn

Listener objects have the following read-only properties:

@geindex address (multiprocessing.connection.Listener attribute)
@anchor{library/multiprocessing multiprocessing connection Listener address}@anchor{28a2}
@deffn {Attribute} address

The address which is being used by the Listener object.
@end deffn

@geindex last_accepted (multiprocessing.connection.Listener attribute)
@anchor{library/multiprocessing multiprocessing connection Listener last_accepted}@anchor{28a3}
@deffn {Attribute} last_accepted

The address from which the last accepted connection came.  If this is
unavailable then it is @code{None}.
@end deffn


New in version 3.3: Listener objects now support the context management protocol – see
@ref{1810,,Context Manager Types}.  @ref{1b35,,__enter__()} returns the
listener object, and @ref{1b37,,__exit__()} calls @ref{28a1,,close()}.

@end deffn

@geindex wait() (in module multiprocessing.connection)
@anchor{library/multiprocessing multiprocessing connection wait}@anchor{c72}
@deffn {Function} multiprocessing.connection.wait (object_list, timeout=None)

Wait till an object in @emph{object_list} is ready.  Returns the list of
those objects in @emph{object_list} which are ready.  If @emph{timeout} is a
float then the call blocks for at most that many seconds.  If
@emph{timeout} is @code{None} then it will block for an unlimited period.
A negative timeout is equivalent to a zero timeout.

For both Unix and Windows, an object can appear in @emph{object_list} if
it is


@itemize *

@item 
a readable @ref{2847,,Connection} object;

@item 
a connected and readable @ref{8fd,,socket.socket} object; or

@item 
the @ref{c73,,sentinel} attribute of a
@ref{681,,Process} object.
@end itemize

A connection or socket object is ready when there is data available
to be read from it, or the other end has been closed.

@strong{Unix}: @code{wait(object_list, timeout)} almost equivalent
@code{select.select(object_list, [], [], timeout)}.  The difference is
that, if @ref{8fc,,select.select()} is interrupted by a signal, it can
raise @ref{2a1,,OSError} with an error number of @code{EINTR}, whereas
@ref{c72,,wait()} will not.

@strong{Windows}: An item in @emph{object_list} must either be an integer
handle which is waitable (according to the definition used by the
documentation of the Win32 function @code{WaitForMultipleObjects()})
or it can be an object with a @code{fileno()} method which returns a
socket handle or pipe handle.  (Note that pipe handles and socket
handles are @strong{not} waitable handles.)


New in version 3.3.

@end deffn

@strong{Examples}

The following server code creates a listener which uses @code{'secret password'} as
an authentication key.  It then waits for a connection and sends some data to
the client:

@example
from multiprocessing.connection import Listener
from array import array

address = ('localhost', 6000)     # family is deduced to be 'AF_INET'

with Listener(address, authkey=b'secret password') as listener:
    with listener.accept() as conn:
        print('connection accepted from', listener.last_accepted)

        conn.send([2.25, None, 'junk', float])

        conn.send_bytes(b'hello')

        conn.send_bytes(array('i', [42, 1729]))
@end example

The following code connects to the server and receives some data from the
server:

@example
from multiprocessing.connection import Client
from array import array

address = ('localhost', 6000)

with Client(address, authkey=b'secret password') as conn:
    print(conn.recv())                  # => [2.25, None, 'junk', float]

    print(conn.recv_bytes())            # => 'hello'

    arr = array('i', [0, 0, 0, 0, 0])
    print(conn.recv_bytes_into(arr))    # => 8
    print(arr)                          # => array('i', [42, 1729, 0, 0, 0])
@end example

The following code uses @ref{c72,,wait()} to
wait for messages from multiple processes at once:

@example
import time, random
from multiprocessing import Process, Pipe, current_process
from multiprocessing.connection import wait

def foo(w):
    for i in range(10):
        w.send((i, current_process().name))
    w.close()

if __name__ == '__main__':
    readers = []

    for i in range(4):
        r, w = Pipe(duplex=False)
        readers.append(r)
        p = Process(target=foo, args=(w,))
        p.start()
        # We close the writable end of the pipe now to be sure that
        # p is the only process which owns a handle for it.  This
        # ensures that when p closes its handle for the writable end,
        # wait() will promptly report the readable end as being ready.
        w.close()

    while readers:
        for r in wait(readers):
            try:
                msg = r.recv()
            except EOFError:
                readers.remove(r)
            else:
                print(msg)
@end example

@menu
* Address Formats:: 

@end menu

@node Address Formats,,,Listeners and Clients
@anchor{library/multiprocessing address-formats}@anchor{28a4}@anchor{library/multiprocessing multiprocessing-address-formats}@anchor{289f}
@subsubsection Address Formats



@itemize *

@item 
An @code{'AF_INET'} address is a tuple of the form @code{(hostname, port)} where
@emph{hostname} is a string and @emph{port} is an integer.

@item 
An @code{'AF_UNIX'} address is a string representing a filename on the
filesystem.

@item 
An @code{'AF_PIPE'} address is a string of the form
@code{r'\.\pipe@{PipeName@}'}.  To use @ref{289e,,Client()} to connect to a named
pipe on a remote computer called @emph{ServerName} one should use an address of the
form @code{r'\@emph{ServerName}\pipe@{PipeName@}'} instead.
@end itemize

Note that any string beginning with two backslashes is assumed by default to be
an @code{'AF_PIPE'} address rather than an @code{'AF_UNIX'} address.

@node Authentication keys,Logging<2>,Listeners and Clients,Reference
@anchor{library/multiprocessing authentication-keys}@anchor{28a5}@anchor{library/multiprocessing multiprocessing-auth-keys}@anchor{2835}
@subsubsection Authentication keys


When one uses @ref{285a,,Connection.recv}, the
data received is automatically
unpickled. Unfortunately unpickling data from an untrusted source is a security
risk. Therefore @ref{1210,,Listener} and @ref{289e,,Client()} use the @ref{8a,,hmac} module
to provide digest authentication.

An authentication key is a byte string which can be thought of as a
password: once a connection is established both ends will demand proof
that the other knows the authentication key.  (Demonstrating that both
ends are using the same key does @strong{not} involve sending the key over
the connection.)

If authentication is requested but no authentication key is specified then the
return value of @code{current_process().authkey} is used (see
@ref{681,,Process}).  This value will be automatically inherited by
any @ref{681,,Process} object that the current process creates.
This means that (by default) all processes of a multi-process program will share
a single authentication key which can be used when setting up connections
between themselves.

Suitable authentication keys can also be generated by using @ref{17f,,os.urandom()}.

@node Logging<2>,The multiprocessing dummy module,Authentication keys,Reference
@anchor{library/multiprocessing logging}@anchor{28a6}
@subsubsection Logging


Some support for logging is available.  Note, however, that the @ref{a6,,logging}
package does not use process shared locks so it is possible (depending on the
handler type) for messages from different processes to get mixed up.

@geindex get_logger() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing get_logger}@anchor{28a7}
@deffn {Function} multiprocessing.get_logger ()

Returns the logger used by @ref{b3,,multiprocessing}.  If necessary, a new one
will be created.

When first created the logger has level @code{logging.NOTSET} and no
default handler. Messages sent to this logger will not by default propagate
to the root logger.

Note that on Windows child processes will only inherit the level of the
parent process’s logger – any other customization of the logger will not be
inherited.
@end deffn

@geindex log_to_stderr() (in module multiprocessing)
@anchor{library/multiprocessing multiprocessing log_to_stderr}@anchor{28a8}
@deffn {Function} multiprocessing.log_to_stderr (level=None)

This function performs a call to @ref{28a7,,get_logger()} but in addition to
returning the logger created by get_logger, it adds a handler which sends
output to @ref{42d,,sys.stderr} using format
@code{'[%(levelname)s/%(processName)s] %(message)s'}.
You can modify @code{levelname} of the logger by passing a @code{level} argument.
@end deffn

Below is an example session with logging turned on:

@example
>>> import multiprocessing, logging
>>> logger = multiprocessing.log_to_stderr()
>>> logger.setLevel(logging.INFO)
>>> logger.warning('doomed')
[WARNING/MainProcess] doomed
>>> m = multiprocessing.Manager()
[INFO/SyncManager-...] child process calling self.run()
[INFO/SyncManager-...] created temp directory /.../pymp-...
[INFO/SyncManager-...] manager serving at '/.../listener-...'
>>> del m
[INFO/MainProcess] sending shutdown message to manager
[INFO/SyncManager-...] manager exiting with exitcode 0
@end example

For a full table of logging levels, see the @ref{a6,,logging} module.

@node The multiprocessing dummy module,,Logging<2>,Reference
@anchor{library/multiprocessing module-multiprocessing dummy}@anchor{b5}@anchor{library/multiprocessing the-multiprocessing-dummy-module}@anchor{28a9}
@subsubsection The @code{multiprocessing.dummy} module


@geindex module; multiprocessing.dummy

@ref{b5,,multiprocessing.dummy} replicates the API of @ref{b3,,multiprocessing} but is
no more than a wrapper around the @ref{109,,threading} module.

In particular, the @code{Pool} function provided by @ref{b5,,multiprocessing.dummy}
returns an instance of @ref{1189,,ThreadPool}, which is a subclass of
@ref{aec,,Pool} that supports all the same method calls but uses a pool of
worker threads rather than worker processes.

@geindex ThreadPool (class in multiprocessing.pool)
@anchor{library/multiprocessing multiprocessing pool ThreadPool}@anchor{1189}
@deffn {Class} multiprocessing.pool.ThreadPool ([processes[, initializer[, initargs]]])

A thread pool object which controls a pool of worker threads to which jobs
can be submitted.  @ref{1189,,ThreadPool} instances are fully interface
compatible with @ref{aec,,Pool} instances, and their resources must also be
properly managed, either by using the pool as a context manager or by
calling @ref{288f,,close()} and
@ref{2890,,terminate()} manually.

@emph{processes} is the number of worker threads to use.  If @emph{processes} is
@code{None} then the number returned by @ref{298,,os.cpu_count()} is used.

If @emph{initializer} is not @code{None} then each worker process will call
@code{initializer(*initargs)} when it starts.

Unlike @ref{aec,,Pool}, @emph{maxtasksperchild} and @emph{context} cannot be provided.

@quotation

@cartouche
@quotation Note 
A @ref{1189,,ThreadPool} shares the same interface as @ref{aec,,Pool}, which
is designed around a pool of processes and predates the introduction of
the @ref{20,,concurrent.futures} module.  As such, it inherits some
operations that don’t make sense for a pool backed by threads, and it
has its own type for representing the status of asynchronous jobs,
@ref{2893,,AsyncResult}, that is not understood by any other libraries.

Users should generally prefer to use
@ref{1c7,,concurrent.futures.ThreadPoolExecutor}, which has a simpler
interface that was designed around threads from the start, and which
returns @ref{d57,,concurrent.futures.Future} instances that are
compatible with many other libraries, including @ref{9,,asyncio}.
@end quotation
@end cartouche
@end quotation
@end deffn

@node Programming guidelines,Examples<12>,Reference,multiprocessing — Process-based parallelism
@anchor{library/multiprocessing multiprocessing-programming}@anchor{2819}@anchor{library/multiprocessing programming-guidelines}@anchor{28aa}
@subsubsection Programming guidelines


There are certain guidelines and idioms which should be adhered to when using
@ref{b3,,multiprocessing}.

@menu
* All start methods:: 
* The spawn and forkserver start methods:: 

@end menu

@node All start methods,The spawn and forkserver start methods,,Programming guidelines
@anchor{library/multiprocessing all-start-methods}@anchor{28ab}
@subsubsection All start methods


The following applies to all start methods.

Avoid shared state

@quotation

As far as possible one should try to avoid shifting large amounts of data
between processes.

It is probably best to stick to using queues or pipes for communication
between processes rather than using the lower level synchronization
primitives.
@end quotation

Picklability

@quotation

Ensure that the arguments to the methods of proxies are picklable.
@end quotation

Thread safety of proxies

@quotation

Do not use a proxy object from more than one thread unless you protect it
with a lock.

(There is never a problem with different processes using the @emph{same} proxy.)
@end quotation

Joining zombie processes

@quotation

On Unix when a process finishes but has not been joined it becomes a zombie.
There should never be very many because each time a new process starts (or
@ref{2852,,active_children()} is called) all completed processes
which have not yet been joined will be joined.  Also calling a finished
process’s @ref{2832,,Process.is_alive} will
join the process.  Even so it is probably good
practice to explicitly join all the processes that you start.
@end quotation

Better to inherit than pickle/unpickle

@quotation

When using the @emph{spawn} or @emph{forkserver} start methods many types
from @ref{b3,,multiprocessing} need to be picklable so that child
processes can use them.  However, one should generally avoid
sending shared objects to other processes using pipes or queues.
Instead you should arrange the program so that a process which
needs access to a shared resource created elsewhere can inherit it
from an ancestor process.
@end quotation

Avoid terminating processes

@quotation

Using the @ref{2836,,Process.terminate}
method to stop a process is liable to
cause any shared resources (such as locks, semaphores, pipes and queues)
currently being used by the process to become broken or unavailable to other
processes.

Therefore it is probably best to only consider using
@ref{2836,,Process.terminate} on processes
which never use any shared resources.
@end quotation

Joining processes that use queues

@quotation

Bear in mind that a process that has put items in a queue will wait before
terminating until all the buffered items are fed by the “feeder” thread to
the underlying pipe.  (The child process can call the
@ref{2845,,Queue.cancel_join_thread}
method of the queue to avoid this behaviour.)

This means that whenever you use a queue you need to make sure that all
items which have been put on the queue will eventually be removed before the
process is joined.  Otherwise you cannot be sure that processes which have
put items on the queue will terminate.  Remember also that non-daemonic
processes will be joined automatically.

An example which will deadlock is the following:

@example
from multiprocessing import Process, Queue

def f(q):
    q.put('X' * 1000000)

if __name__ == '__main__':
    queue = Queue()
    p = Process(target=f, args=(queue,))
    p.start()
    p.join()                    # this deadlocks
    obj = queue.get()
@end example

A fix here would be to swap the last two lines (or simply remove the
@code{p.join()} line).
@end quotation

Explicitly pass resources to child processes

@quotation

On Unix using the @emph{fork} start method, a child process can make
use of a shared resource created in a parent process using a
global resource.  However, it is better to pass the object as an
argument to the constructor for the child process.

Apart from making the code (potentially) compatible with Windows
and the other start methods this also ensures that as long as the
child process is still alive the object will not be garbage
collected in the parent process.  This might be important if some
resource is freed when the object is garbage collected in the
parent process.

So for instance

@example
from multiprocessing import Process, Lock

def f():
    ... do something using "lock" ...

if __name__ == '__main__':
    lock = Lock()
    for i in range(10):
        Process(target=f).start()
@end example

should be rewritten as

@example
from multiprocessing import Process, Lock

def f(l):
    ... do something using "l" ...

if __name__ == '__main__':
    lock = Lock()
    for i in range(10):
        Process(target=f, args=(lock,)).start()
@end example
@end quotation

Beware of replacing @ref{5ee,,sys.stdin} with a “file like object”

@quotation

@ref{b3,,multiprocessing} originally unconditionally called:

@example
os.close(sys.stdin.fileno())
@end example

in the @code{multiprocessing.Process._bootstrap()} method — this resulted
in issues with processes-in-processes. This has been changed to:

@example
sys.stdin.close()
sys.stdin = open(os.open(os.devnull, os.O_RDONLY), closefd=False)
@end example

Which solves the fundamental issue of processes colliding with each other
resulting in a bad file descriptor error, but introduces a potential danger
to applications which replace @ref{5ee,,sys.stdin()} with a “file-like object”
with output buffering.  This danger is that if multiple processes call
@ref{12d8,,close()} on this file-like object, it could result in the same
data being flushed to the object multiple times, resulting in corruption.

If you write a file-like object and implement your own caching, you can
make it fork-safe by storing the pid whenever you append to the cache,
and discarding the cache when the pid changes. For example:

@example
@@property
def cache(self):
    pid = os.getpid()
    if pid != self._pid:
        self._pid = pid
        self._cache = []
    return self._cache
@end example

For more information, see bpo-5155@footnote{https://bugs.python.org/issue5155}, bpo-5313@footnote{https://bugs.python.org/issue5313} and bpo-5331@footnote{https://bugs.python.org/issue5331}
@end quotation

@node The spawn and forkserver start methods,,All start methods,Programming guidelines
@anchor{library/multiprocessing the-spawn-and-forkserver-start-methods}@anchor{28ac}
@subsubsection The @emph{spawn} and @emph{forkserver} start methods


There are a few extra restriction which don’t apply to the @emph{fork}
start method.

More picklability

@quotation

Ensure that all arguments to @code{Process.__init__()} are picklable.
Also, if you subclass @ref{681,,Process} then make sure that
instances will be picklable when the @ref{2818,,Process.start} method is called.
@end quotation

Global variables

@quotation

Bear in mind that if code run in a child process tries to access a global
variable, then the value it sees (if any) may not be the same as the value
in the parent process at the time that @ref{2818,,Process.start} was called.

However, global variables which are just module level constants cause no
problems.
@end quotation

Safe importing of main module

@quotation

Make sure that the main module can be safely imported by a new Python
interpreter without causing unintended side effects (such a starting a new
process).

For example, using the @emph{spawn} or @emph{forkserver} start method
running the following module would fail with a
@ref{5af,,RuntimeError}:

@example
from multiprocessing import Process

def foo():
    print('hello')

p = Process(target=foo)
p.start()
@end example

Instead one should protect the “entry point” of the program by using @code{if
__name__ == '__main__':} as follows:

@example
from multiprocessing import Process, freeze_support, set_start_method

def foo():
    print('hello')

if __name__ == '__main__':
    freeze_support()
    set_start_method('spawn')
    p = Process(target=foo)
    p.start()
@end example

(The @code{freeze_support()} line can be omitted if the program will be run
normally instead of frozen.)

This allows the newly spawned Python interpreter to safely import the module
and then run the module’s @code{foo()} function.

Similar restrictions apply if a pool or manager is created in the main
module.
@end quotation

@node Examples<12>,,Programming guidelines,multiprocessing — Process-based parallelism
@anchor{library/multiprocessing examples}@anchor{28ad}@anchor{library/multiprocessing multiprocessing-examples}@anchor{2846}
@subsubsection Examples


Demonstration of how to create and use customized managers and proxies:

@example
from multiprocessing import freeze_support
from multiprocessing.managers import BaseManager, BaseProxy
import operator

##

class Foo:
    def f(self):
        print('you called Foo.f()')
    def g(self):
        print('you called Foo.g()')
    def _h(self):
        print('you called Foo._h()')

# A simple generator function
def baz():
    for i in range(10):
        yield i*i

# Proxy type for generator objects
class GeneratorProxy(BaseProxy):
    _exposed_ = ['__next__']
    def __iter__(self):
        return self
    def __next__(self):
        return self._callmethod('__next__')

# Function to return the operator module
def get_operator_module():
    return operator

##

class MyManager(BaseManager):
    pass

# register the Foo class; make `f()` and `g()` accessible via proxy
MyManager.register('Foo1', Foo)

# register the Foo class; make `g()` and `_h()` accessible via proxy
MyManager.register('Foo2', Foo, exposed=('g', '_h'))

# register the generator function baz; use `GeneratorProxy` to make proxies
MyManager.register('baz', baz, proxytype=GeneratorProxy)

# register get_operator_module(); make public functions accessible via proxy
MyManager.register('operator', get_operator_module)

##

def test():
    manager = MyManager()
    manager.start()

    print('-' * 20)

    f1 = manager.Foo1()
    f1.f()
    f1.g()
    assert not hasattr(f1, '_h')
    assert sorted(f1._exposed_) == sorted(['f', 'g'])

    print('-' * 20)

    f2 = manager.Foo2()
    f2.g()
    f2._h()
    assert not hasattr(f2, 'f')
    assert sorted(f2._exposed_) == sorted(['g', '_h'])

    print('-' * 20)

    it = manager.baz()
    for i in it:
        print('<%d>' % i, end=' ')
    print()

    print('-' * 20)

    op = manager.operator()
    print('op.add(23, 45) =', op.add(23, 45))
    print('op.pow(2, 94) =', op.pow(2, 94))
    print('op._exposed_ =', op._exposed_)

##

if __name__ == '__main__':
    freeze_support()
    test()
@end example

Using @ref{aec,,Pool}:

@example
import multiprocessing
import time
import random
import sys

#
# Functions used by test code
#

def calculate(func, args):
    result = func(*args)
    return '%s says that %s%s = %s' % (
        multiprocessing.current_process().name,
        func.__name__, args, result
        )

def calculatestar(args):
    return calculate(*args)

def mul(a, b):
    time.sleep(0.5 * random.random())
    return a * b

def plus(a, b):
    time.sleep(0.5 * random.random())
    return a + b

def f(x):
    return 1.0 / (x - 5.0)

def pow3(x):
    return x ** 3

def noop(x):
    pass

#
# Test code
#

def test():
    PROCESSES = 4
    print('Creating pool with %d processes\n' % PROCESSES)

    with multiprocessing.Pool(PROCESSES) as pool:
        #
        # Tests
        #

        TASKS = [(mul, (i, 7)) for i in range(10)] + \
                [(plus, (i, 8)) for i in range(10)]

        results = [pool.apply_async(calculate, t) for t in TASKS]
        imap_it = pool.imap(calculatestar, TASKS)
        imap_unordered_it = pool.imap_unordered(calculatestar, TASKS)

        print('Ordered results using pool.apply_async():')
        for r in results:
            print('\t', r.get())
        print()

        print('Ordered results using pool.imap():')
        for x in imap_it:
            print('\t', x)
        print()

        print('Unordered results using pool.imap_unordered():')
        for x in imap_unordered_it:
            print('\t', x)
        print()

        print('Ordered results using pool.map() --- will block till complete:')
        for x in pool.map(calculatestar, TASKS):
            print('\t', x)
        print()

        #
        # Test error handling
        #

        print('Testing error handling:')

        try:
            print(pool.apply(f, (5,)))
        except ZeroDivisionError:
            print('\tGot ZeroDivisionError as expected from pool.apply()')
        else:
            raise AssertionError('expected ZeroDivisionError')

        try:
            print(pool.map(f, list(range(10))))
        except ZeroDivisionError:
            print('\tGot ZeroDivisionError as expected from pool.map()')
        else:
            raise AssertionError('expected ZeroDivisionError')

        try:
            print(list(pool.imap(f, list(range(10)))))
        except ZeroDivisionError:
            print('\tGot ZeroDivisionError as expected from list(pool.imap())')
        else:
            raise AssertionError('expected ZeroDivisionError')

        it = pool.imap(f, list(range(10)))
        for i in range(10):
            try:
                x = next(it)
            except ZeroDivisionError:
                if i == 5:
                    pass
            except StopIteration:
                break
            else:
                if i == 5:
                    raise AssertionError('expected ZeroDivisionError')

        assert i == 9
        print('\tGot ZeroDivisionError as expected from IMapIterator.next()')
        print()

        #
        # Testing timeouts
        #

        print('Testing ApplyResult.get() with timeout:', end=' ')
        res = pool.apply_async(calculate, TASKS[0])
        while 1:
            sys.stdout.flush()
            try:
                sys.stdout.write('\n\t%s' % res.get(0.02))
                break
            except multiprocessing.TimeoutError:
                sys.stdout.write('.')
        print()
        print()

        print('Testing IMapIterator.next() with timeout:', end=' ')
        it = pool.imap(calculatestar, TASKS)
        while 1:
            sys.stdout.flush()
            try:
                sys.stdout.write('\n\t%s' % it.next(0.02))
            except StopIteration:
                break
            except multiprocessing.TimeoutError:
                sys.stdout.write('.')
        print()
        print()


if __name__ == '__main__':
    multiprocessing.freeze_support()
    test()
@end example

An example showing how to use queues to feed tasks to a collection of worker
processes and collect the results:

@example
import time
import random

from multiprocessing import Process, Queue, current_process, freeze_support

#
# Function run by worker processes
#

def worker(input, output):
    for func, args in iter(input.get, 'STOP'):
        result = calculate(func, args)
        output.put(result)

#
# Function used to calculate result
#

def calculate(func, args):
    result = func(*args)
    return '%s says that %s%s = %s' % \
        (current_process().name, func.__name__, args, result)

#
# Functions referenced by tasks
#

def mul(a, b):
    time.sleep(0.5*random.random())
    return a * b

def plus(a, b):
    time.sleep(0.5*random.random())
    return a + b

#
#
#

def test():
    NUMBER_OF_PROCESSES = 4
    TASKS1 = [(mul, (i, 7)) for i in range(20)]
    TASKS2 = [(plus, (i, 8)) for i in range(10)]

    # Create queues
    task_queue = Queue()
    done_queue = Queue()

    # Submit tasks
    for task in TASKS1:
        task_queue.put(task)

    # Start worker processes
    for i in range(NUMBER_OF_PROCESSES):
        Process(target=worker, args=(task_queue, done_queue)).start()

    # Get and print results
    print('Unordered results:')
    for i in range(len(TASKS1)):
        print('\t', done_queue.get())

    # Add more tasks using `put()`
    for task in TASKS2:
        task_queue.put(task)

    # Get and print some more results
    for i in range(len(TASKS2)):
        print('\t', done_queue.get())

    # Tell child processes to stop
    for i in range(NUMBER_OF_PROCESSES):
        task_queue.put('STOP')


if __name__ == '__main__':
    freeze_support()
    test()
@end example

@node multiprocessing shared_memory — Provides shared memory for direct access across processes,The concurrent package,multiprocessing — Process-based parallelism,Concurrent Execution
@anchor{library/multiprocessing shared_memory doc}@anchor{28ae}@anchor{library/multiprocessing shared_memory module-multiprocessing shared_memory}@anchor{b8}@anchor{library/multiprocessing shared_memory multiprocessing-shared-memory-provides-shared-memory-for-direct-access-across-processes}@anchor{28af}
@subsection @code{multiprocessing.shared_memory} —  Provides shared memory for direct access across processes


@geindex module; multiprocessing.shared_memory

@strong{Source code:} Lib/multiprocessing/shared_memory.py@footnote{https://github.com/python/cpython/tree/main/Lib/multiprocessing/shared_memory.py}


New in version 3.8.


@geindex Shared Memory
@geindex POSIX Shared Memory
@geindex Named Shared Memory


__________________________________________________________________


This module provides a class, @ref{281c,,SharedMemory}, for the allocation
and management of shared memory to be accessed by one or more processes
on a multicore or symmetric multiprocessor (SMP) machine.  To assist with
the life-cycle management of shared memory especially across distinct
processes, a @ref{2871,,BaseManager} subclass,
@code{SharedMemoryManager}, is also provided in the
@code{multiprocessing.managers} module.

In this module, shared memory refers to “System V style” shared memory blocks
(though is not necessarily implemented explicitly as such) and does not refer
to “distributed shared memory”.  This style of shared memory permits distinct
processes to potentially read and write to a common (or shared) region of
volatile memory.  Processes are conventionally limited to only have access to
their own process memory space but shared memory permits the sharing
of data between processes, avoiding the need to instead send messages between
processes containing that data.  Sharing data directly via memory can provide
significant performance benefits compared to sharing data via disk or socket
or other communications requiring the serialization/deserialization and
copying of data.

@geindex SharedMemory (class in multiprocessing.shared_memory)
@anchor{library/multiprocessing shared_memory multiprocessing shared_memory SharedMemory}@anchor{281c}
@deffn {Class} multiprocessing.shared_memory.SharedMemory (name=None, create=False, size=0)

Creates a new shared memory block or attaches to an existing shared
memory block.  Each shared memory block is assigned a unique name.
In this way, one process can create a shared memory block with a
particular name and a different process can attach to that same shared
memory block using that same name.

As a resource for sharing data across processes, shared memory blocks
may outlive the original process that created them.  When one process
no longer needs access to a shared memory block that might still be
needed by other processes, the @ref{28b0,,close()} method should be called.
When a shared memory block is no longer needed by any process, the
@ref{28b1,,unlink()} method should be called to ensure proper cleanup.

@emph{name} is the unique name for the requested shared memory, specified as
a string.  When creating a new shared memory block, if @code{None} (the
default) is supplied for the name, a novel name will be generated.

@emph{create} controls whether a new shared memory block is created (@code{True})
or an existing shared memory block is attached (@code{False}).

@emph{size} specifies the requested number of bytes when creating a new shared
memory block.  Because some platforms choose to allocate chunks of memory
based upon that platform’s memory page size, the exact size of the shared
memory block may be larger or equal to the size requested.  When attaching
to an existing shared memory block, the @code{size} parameter is ignored.

@geindex close() (multiprocessing.shared_memory.SharedMemory method)
@anchor{library/multiprocessing shared_memory multiprocessing shared_memory SharedMemory close}@anchor{28b0}
@deffn {Method} close ()

Closes access to the shared memory from this instance.  In order to
ensure proper cleanup of resources, all instances should call
@code{close()} once the instance is no longer needed.  Note that calling
@code{close()} does not cause the shared memory block itself to be
destroyed.
@end deffn

@geindex unlink() (multiprocessing.shared_memory.SharedMemory method)
@anchor{library/multiprocessing shared_memory multiprocessing shared_memory SharedMemory unlink}@anchor{28b1}
@deffn {Method} unlink ()

Requests that the underlying shared memory block be destroyed.  In
order to ensure proper cleanup of resources, @code{unlink()} should be
called once (and only once) across all processes which have need
for the shared memory block.  After requesting its destruction, a
shared memory block may or may not be immediately destroyed and
this behavior may differ across platforms.  Attempts to access data
inside the shared memory block after @code{unlink()} has been called may
result in memory access errors.  Note: the last process relinquishing
its hold on a shared memory block may call @code{unlink()} and
@ref{28b0,,close()} in either order.
@end deffn

@geindex buf (multiprocessing.shared_memory.SharedMemory attribute)
@anchor{library/multiprocessing shared_memory multiprocessing shared_memory SharedMemory buf}@anchor{28b2}
@deffn {Attribute} buf

A memoryview of contents of the shared memory block.
@end deffn

@geindex name (multiprocessing.shared_memory.SharedMemory attribute)
@anchor{library/multiprocessing shared_memory multiprocessing shared_memory SharedMemory name}@anchor{28b3}
@deffn {Attribute} name

Read-only access to the unique name of the shared memory block.
@end deffn

@geindex size (multiprocessing.shared_memory.SharedMemory attribute)
@anchor{library/multiprocessing shared_memory multiprocessing shared_memory SharedMemory size}@anchor{28b4}
@deffn {Attribute} size

Read-only access to size in bytes of the shared memory block.
@end deffn
@end deffn

The following example demonstrates low-level use of @ref{281c,,SharedMemory}
instances:

@example
>>> from multiprocessing import shared_memory
>>> shm_a = shared_memory.SharedMemory(create=True, size=10)
>>> type(shm_a.buf)
<class 'memoryview'>
>>> buffer = shm_a.buf
>>> len(buffer)
10
>>> buffer[:4] = bytearray([22, 33, 44, 55])  # Modify multiple at once
>>> buffer[4] = 100                           # Modify single byte at a time
>>> # Attach to an existing shared memory block
>>> shm_b = shared_memory.SharedMemory(shm_a.name)
>>> import array
>>> array.array('b', shm_b.buf[:5])  # Copy the data into a new array.array
array('b', [22, 33, 44, 55, 100])
>>> shm_b.buf[:5] = b'howdy'  # Modify via shm_b using bytes
>>> bytes(shm_a.buf[:5])      # Access via shm_a
b'howdy'
>>> shm_b.close()   # Close each SharedMemory instance
>>> shm_a.close()
>>> shm_a.unlink()  # Call unlink only once to release the shared memory
@end example

The following example demonstrates a practical use of the @ref{281c,,SharedMemory}
class with NumPy arrays@footnote{https://www.numpy.org/}, accessing the
same @code{numpy.ndarray} from two distinct Python shells:

@example
>>> # In the first Python interactive shell
>>> import numpy as np
>>> a = np.array([1, 1, 2, 3, 5, 8])  # Start with an existing NumPy array
>>> from multiprocessing import shared_memory
>>> shm = shared_memory.SharedMemory(create=True, size=a.nbytes)
>>> # Now create a NumPy array backed by shared memory
>>> b = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf)
>>> b[:] = a[:]  # Copy the original data into shared memory
>>> b
array([1, 1, 2, 3, 5, 8])
>>> type(b)
<class 'numpy.ndarray'>
>>> type(a)
<class 'numpy.ndarray'>
>>> shm.name  # We did not specify a name so one was chosen for us
'psm_21467_46075'

>>> # In either the same shell or a new Python shell on the same machine
>>> import numpy as np
>>> from multiprocessing import shared_memory
>>> # Attach to the existing shared memory block
>>> existing_shm = shared_memory.SharedMemory(name='psm_21467_46075')
>>> # Note that a.shape is (6,) and a.dtype is np.int64 in this example
>>> c = np.ndarray((6,), dtype=np.int64, buffer=existing_shm.buf)
>>> c
array([1, 1, 2, 3, 5, 8])
>>> c[-1] = 888
>>> c
array([  1,   1,   2,   3,   5, 888])

>>> # Back in the first Python interactive shell, b reflects this change
>>> b
array([  1,   1,   2,   3,   5, 888])

>>> # Clean up from within the second Python shell
>>> del c  # Unnecessary; merely emphasizing the array is no longer used
>>> existing_shm.close()

>>> # Clean up from within the first Python shell
>>> del b  # Unnecessary; merely emphasizing the array is no longer used
>>> shm.close()
>>> shm.unlink()  # Free and release the shared memory block at the very end
@end example

@geindex SharedMemoryManager (class in multiprocessing.managers)
@anchor{library/multiprocessing shared_memory multiprocessing managers SharedMemoryManager}@anchor{28b5}
@deffn {Class} multiprocessing.managers.SharedMemoryManager ([address[, authkey]])

A subclass of @ref{2871,,BaseManager} which can be
used for the management of shared memory blocks across processes.

A call to @ref{2872,,start()} on a
@ref{28b5,,SharedMemoryManager} instance causes a new process to be started.
This new process’s sole purpose is to manage the life cycle
of all shared memory blocks created through it.  To trigger the release
of all shared memory blocks managed by that process, call
@ref{2876,,shutdown()} on the instance.
This triggers a @code{SharedMemory.unlink()} call on all of the
@ref{28b6,,SharedMemory} objects managed by that process and then
stops the process itself.  By creating @code{SharedMemory} instances
through a @code{SharedMemoryManager}, we avoid the need to manually track
and trigger the freeing of shared memory resources.

This class provides methods for creating and returning @ref{28b6,,SharedMemory}
instances and for creating a list-like object (@ref{28b7,,ShareableList})
backed by shared memory.

Refer to @ref{2871,,multiprocessing.managers.BaseManager} for a description
of the inherited @emph{address} and @emph{authkey} optional input arguments and how
they may be used to connect to an existing @code{SharedMemoryManager} service
from other processes.

@geindex SharedMemory() (multiprocessing.managers.SharedMemoryManager method)
@anchor{library/multiprocessing shared_memory multiprocessing managers SharedMemoryManager SharedMemory}@anchor{28b6}
@deffn {Method} SharedMemory (size)

Create and return a new @ref{28b6,,SharedMemory} object with the
specified @code{size} in bytes.
@end deffn

@geindex ShareableList() (multiprocessing.managers.SharedMemoryManager method)
@anchor{library/multiprocessing shared_memory multiprocessing managers SharedMemoryManager ShareableList}@anchor{28b7}
@deffn {Method} ShareableList (sequence)

Create and return a new @ref{28b7,,ShareableList} object, initialized
by the values from the input @code{sequence}.
@end deffn
@end deffn

The following example demonstrates the basic mechanisms of a
@code{SharedMemoryManager}:

@example
>>> from multiprocessing.managers import SharedMemoryManager
>>> smm = SharedMemoryManager()
>>> smm.start()  # Start the process that manages the shared memory blocks
>>> sl = smm.ShareableList(range(4))
>>> sl
ShareableList([0, 1, 2, 3], name='psm_6572_7512')
>>> raw_shm = smm.SharedMemory(size=128)
>>> another_sl = smm.ShareableList('alpha')
>>> another_sl
ShareableList(['a', 'l', 'p', 'h', 'a'], name='psm_6572_12221')
>>> smm.shutdown()  # Calls unlink() on sl, raw_shm, and another_sl
@end example

The following example depicts a potentially more convenient pattern for using
@code{SharedMemoryManager} objects via the @ref{168,,with} statement to
ensure that all shared memory blocks are released after they are no longer
needed:

@example
>>> with SharedMemoryManager() as smm:
...     sl = smm.ShareableList(range(2000))
...     # Divide the work among two processes, storing partial results in sl
...     p1 = Process(target=do_work, args=(sl, 0, 1000))
...     p2 = Process(target=do_work, args=(sl, 1000, 2000))
...     p1.start()
...     p2.start()  # A multiprocessing.Pool might be more efficient
...     p1.join()
...     p2.join()   # Wait for all work to complete in both processes
...     total_result = sum(sl)  # Consolidate the partial results now in sl
@end example

When using a @code{SharedMemoryManager} in a @ref{168,,with} statement, the
shared memory blocks created using that manager are all released when the
@ref{168,,with} statement’s code block finishes execution.

@geindex ShareableList (class in multiprocessing.shared_memory)
@anchor{library/multiprocessing shared_memory multiprocessing shared_memory ShareableList}@anchor{11fb}
@deffn {Class} multiprocessing.shared_memory.ShareableList (sequence=None, *, name=None)

Provides a mutable list-like object where all values stored within are
stored in a shared memory block.  This constrains storable values to
only the @code{int}, @code{float}, @code{bool}, @code{str} (less than 10M bytes each),
@code{bytes} (less than 10M bytes each), and @code{None} built-in data types.
It also notably differs from the built-in @code{list} type in that these
lists can not change their overall length (i.e. no append, insert, etc.)
and do not support the dynamic creation of new @ref{11fb,,ShareableList}
instances via slicing.

@emph{sequence} is used in populating a new @code{ShareableList} full of values.
Set to @code{None} to instead attach to an already existing
@code{ShareableList} by its unique shared memory name.

@emph{name} is the unique name for the requested shared memory, as described
in the definition for @ref{281c,,SharedMemory}.  When attaching to an
existing @code{ShareableList}, specify its shared memory block’s unique
name while leaving @code{sequence} set to @code{None}.

@geindex count() (multiprocessing.shared_memory.ShareableList method)
@anchor{library/multiprocessing shared_memory multiprocessing shared_memory ShareableList count}@anchor{28b8}
@deffn {Method} count (value)

Returns the number of occurrences of @code{value}.
@end deffn

@geindex index() (multiprocessing.shared_memory.ShareableList method)
@anchor{library/multiprocessing shared_memory multiprocessing shared_memory ShareableList index}@anchor{28b9}
@deffn {Method} index (value)

Returns first index position of @code{value}.  Raises @ref{17c,,ValueError} if
@code{value} is not present.
@end deffn

@geindex format (multiprocessing.shared_memory.ShareableList attribute)
@anchor{library/multiprocessing shared_memory multiprocessing shared_memory ShareableList format}@anchor{28ba}
@deffn {Attribute} format

Read-only attribute containing the @ref{f3,,struct} packing format used by
all currently stored values.
@end deffn

@geindex shm (multiprocessing.shared_memory.ShareableList attribute)
@anchor{library/multiprocessing shared_memory multiprocessing shared_memory ShareableList shm}@anchor{28bb}
@deffn {Attribute} shm

The @ref{281c,,SharedMemory} instance where the values are stored.
@end deffn
@end deffn

The following example demonstrates basic use of a @ref{11fb,,ShareableList}
instance:

@example
>>> from multiprocessing import shared_memory
>>> a = shared_memory.ShareableList(['howdy', b'HoWdY', -273.154, 100, None, True, 42])
>>> [ type(entry) for entry in a ]
[<class 'str'>, <class 'bytes'>, <class 'float'>, <class 'int'>, <class 'NoneType'>, <class 'bool'>, <class 'int'>]
>>> a[2]
-273.154
>>> a[2] = -78.5
>>> a[2]
-78.5
>>> a[2] = 'dry ice'  # Changing data types is supported as well
>>> a[2]
'dry ice'
>>> a[2] = 'larger than previously allocated storage space'
Traceback (most recent call last):
  ...
ValueError: exceeds available storage for existing str
>>> a[2]
'dry ice'
>>> len(a)
7
>>> a.index(42)
6
>>> a.count(b'howdy')
0
>>> a.count(b'HoWdY')
1
>>> a.shm.close()
>>> a.shm.unlink()
>>> del a  # Use of a ShareableList after call to unlink() is unsupported
@end example

The following example depicts how one, two, or many processes may access the
same @ref{11fb,,ShareableList} by supplying the name of the shared memory block
behind it:

@example
>>> b = shared_memory.ShareableList(range(5))         # In a first process
>>> c = shared_memory.ShareableList(name=b.shm.name)  # In a second process
>>> c
ShareableList([0, 1, 2, 3, 4], name='...')
>>> c[-1] = -999
>>> b[-1]
-999
>>> b.shm.close()
>>> c.shm.close()
>>> c.shm.unlink()
@end example

The following examples demonstrates that @code{ShareableList}
(and underlying @code{SharedMemory}) objects
can be pickled and unpickled if needed.
Note, that it will still be the same shared object.
This happens, because the deserialized object has
the same unique name and is just attached to an existing
object with the same name (if the object is still alive):

@example
>>> import pickle
>>> from multiprocessing import shared_memory
>>> sl = shared_memory.ShareableList(range(10))
>>> list(sl)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
@end example

@example
>>> deserialized_sl = pickle.loads(pickle.dumps(sl))
>>> list(deserialized_sl)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
@end example

@example
>>> sl[0] = -1
>>> deserialized_sl[1] = -2
>>> list(sl)
[-1, -2, 2, 3, 4, 5, 6, 7, 8, 9]
>>> list(deserialized_sl)
[-1, -2, 2, 3, 4, 5, 6, 7, 8, 9]
@end example

@example
>>> sl.shm.close()
>>> sl.shm.unlink()
@end example

@node The concurrent package,concurrent futures — Launching parallel tasks,multiprocessing shared_memory — Provides shared memory for direct access across processes,Concurrent Execution
@anchor{library/concurrent doc}@anchor{28bc}@anchor{library/concurrent the-concurrent-package}@anchor{28bd}
@subsection The @code{concurrent} package


Currently, there is only one module in this package:


@itemize *

@item 
@ref{20,,concurrent.futures} – Launching parallel tasks
@end itemize

@node concurrent futures — Launching parallel tasks,subprocess — Subprocess management,The concurrent package,Concurrent Execution
@anchor{library/concurrent futures doc}@anchor{28be}@anchor{library/concurrent futures concurrent-futures-launching-parallel-tasks}@anchor{28bf}@anchor{library/concurrent futures module-concurrent futures}@anchor{20}
@subsection @code{concurrent.futures} — Launching parallel tasks


@geindex module; concurrent.futures


New in version 3.2.


@strong{Source code:} Lib/concurrent/futures/thread.py@footnote{https://github.com/python/cpython/tree/main/Lib/concurrent/futures/thread.py}
and Lib/concurrent/futures/process.py@footnote{https://github.com/python/cpython/tree/main/Lib/concurrent/futures/process.py}


__________________________________________________________________


The @ref{20,,concurrent.futures} module provides a high-level interface for
asynchronously executing callables.

The asynchronous execution can be performed with threads, using
@ref{1c7,,ThreadPoolExecutor}, or separate processes, using
@ref{3da,,ProcessPoolExecutor}.  Both implement the same interface, which is
defined by the abstract @ref{28c0,,Executor} class.

@menu
* Executor Objects:: 
* ThreadPoolExecutor:: 
* ProcessPoolExecutor:: 
* Future Objects:: 
* Module Functions:: 
* Exception classes:: 

@end menu

@node Executor Objects,ThreadPoolExecutor,,concurrent futures — Launching parallel tasks
@anchor{library/concurrent futures executor-objects}@anchor{28c1}
@subsubsection Executor Objects


@geindex Executor (class in concurrent.futures)
@anchor{library/concurrent futures concurrent futures Executor}@anchor{28c0}
@deffn {Class} concurrent.futures.Executor

An abstract class that provides methods to execute calls asynchronously.  It
should not be used directly, but through its concrete subclasses.

@quotation

@geindex submit() (concurrent.futures.Executor method)
@anchor{library/concurrent futures concurrent futures Executor submit}@anchor{59a}
@deffn {Method} submit (fn, /, *args, **kwargs)

Schedules the callable, @emph{fn}, to be executed as @code{fn(*args, **kwargs)}
and returns a @ref{d57,,Future} object representing the execution of the
callable.

@example
with ThreadPoolExecutor(max_workers=1) as executor:
    future = executor.submit(pow, 323, 1235)
    print(future.result())
@end example
@end deffn

@geindex map() (concurrent.futures.Executor method)
@anchor{library/concurrent futures concurrent futures Executor map}@anchor{943}
@deffn {Method} map (func, *iterables, timeout=None, chunksize=1)

Similar to @ref{2f4,,map(func@comma{} *iterables)} except:


@itemize *

@item 
the @emph{iterables} are collected immediately rather than lazily;

@item 
@emph{func} is executed asynchronously and several calls to
@emph{func} may be made concurrently.
@end itemize

The returned iterator raises a @ref{2b8,,TimeoutError}
if @ref{e6d,,__next__()} is called and the result isn’t available
after @emph{timeout} seconds from the original call to @ref{943,,Executor.map()}.
@emph{timeout} can be an int or a float.  If @emph{timeout} is not specified or
@code{None}, there is no limit to the wait time.

If a @emph{func} call raises an exception, then that exception will be
raised when its value is retrieved from the iterator.

When using @ref{3da,,ProcessPoolExecutor}, this method chops @emph{iterables}
into a number of chunks which it submits to the pool as separate
tasks.  The (approximate) size of these chunks can be specified by
setting @emph{chunksize} to a positive integer.  For very long iterables,
using a large value for @emph{chunksize} can significantly improve
performance compared to the default size of 1.  With
@ref{1c7,,ThreadPoolExecutor}, @emph{chunksize} has no effect.


Changed in version 3.5: Added the @emph{chunksize} argument.

@end deffn

@geindex shutdown() (concurrent.futures.Executor method)
@anchor{library/concurrent futures concurrent futures Executor shutdown}@anchor{3d9}
@deffn {Method} shutdown (wait=True, *, cancel_futures=False)

Signal the executor that it should free any resources that it is using
when the currently pending futures are done executing.  Calls to
@ref{59a,,Executor.submit()} and @ref{943,,Executor.map()} made after shutdown will
raise @ref{5af,,RuntimeError}.

If @emph{wait} is @code{True} then this method will not return until all the
pending futures are done executing and the resources associated with the
executor have been freed.  If @emph{wait} is @code{False} then this method will
return immediately and the resources associated with the executor will be
freed when all pending futures are done executing.  Regardless of the
value of @emph{wait}, the entire Python program will not exit until all
pending futures are done executing.

If @emph{cancel_futures} is @code{True}, this method will cancel all pending
futures that the executor has not started running. Any futures that
are completed or running won’t be cancelled, regardless of the value
of @emph{cancel_futures}.

If both @emph{cancel_futures} and @emph{wait} are @code{True}, all futures that the
executor has started running will be completed prior to this method
returning. The remaining futures are cancelled.

You can avoid having to call this method explicitly if you use the
@ref{168,,with} statement, which will shutdown the @ref{28c0,,Executor}
(waiting as if @ref{3d9,,Executor.shutdown()} were called with @emph{wait} set to
@code{True}):

@example
import shutil
with ThreadPoolExecutor(max_workers=4) as e:
    e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
    e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
    e.submit(shutil.copy, 'src4.txt', 'dest4.txt')
@end example


Changed in version 3.9: Added @emph{cancel_futures}.

@end deffn
@end quotation
@end deffn

@node ThreadPoolExecutor,ProcessPoolExecutor,Executor Objects,concurrent futures — Launching parallel tasks
@anchor{library/concurrent futures threadpoolexecutor}@anchor{28c2}
@subsubsection ThreadPoolExecutor


@ref{1c7,,ThreadPoolExecutor} is an @ref{28c0,,Executor} subclass that uses a pool of
threads to execute calls asynchronously.

Deadlocks can occur when the callable associated with a @ref{d57,,Future} waits on
the results of another @ref{d57,,Future}.  For example:

@example
import time
def wait_on_b():
    time.sleep(5)
    print(b.result())  # b will never complete because it is waiting on a.
    return 5

def wait_on_a():
    time.sleep(5)
    print(a.result())  # a will never complete because it is waiting on b.
    return 6


executor = ThreadPoolExecutor(max_workers=2)
a = executor.submit(wait_on_b)
b = executor.submit(wait_on_a)
@end example

And:

@example
def wait_on_future():
    f = executor.submit(pow, 5, 2)
    # This will never complete because there is only one worker thread and
    # it is executing this function.
    print(f.result())

executor = ThreadPoolExecutor(max_workers=1)
executor.submit(wait_on_future)
@end example

@geindex ThreadPoolExecutor (class in concurrent.futures)
@anchor{library/concurrent futures concurrent futures ThreadPoolExecutor}@anchor{1c7}
@deffn {Class} concurrent.futures.ThreadPoolExecutor (max_workers=None, thread_name_prefix='@w{'}, initializer=None, initargs=())

An @ref{28c0,,Executor} subclass that uses a pool of at most @emph{max_workers}
threads to execute calls asynchronously.

@emph{initializer} is an optional callable that is called at the start of
each worker thread; @emph{initargs} is a tuple of arguments passed to the
initializer.  Should @emph{initializer} raise an exception, all currently
pending jobs will raise a @ref{28c3,,BrokenThreadPool},
as well as any attempt to submit more jobs to the pool.


Changed in version 3.5: If @emph{max_workers} is @code{None} or
not given, it will default to the number of processors on the machine,
multiplied by @code{5}, assuming that @ref{1c7,,ThreadPoolExecutor} is often
used to overlap I/O instead of CPU work and the number of workers
should be higher than the number of workers
for @ref{3da,,ProcessPoolExecutor}.



New in version 3.6: The @emph{thread_name_prefix} argument was added to allow users to
control the @ref{44b,,threading.Thread} names for worker threads created by
the pool for easier debugging.



Changed in version 3.7: Added the @emph{initializer} and @emph{initargs} arguments.



Changed in version 3.8: Default value of @emph{max_workers} is changed to @code{min(32, os.cpu_count() + 4)}.
This default value preserves at least 5 workers for I/O bound tasks.
It utilizes at most 32 CPU cores for CPU bound tasks which release the GIL.
And it avoids using very large resources implicitly on many-core machines.

ThreadPoolExecutor now reuses idle worker threads before starting
@emph{max_workers} worker threads too.

@end deffn

@menu
* ThreadPoolExecutor Example:: 

@end menu

@node ThreadPoolExecutor Example,,,ThreadPoolExecutor
@anchor{library/concurrent futures id1}@anchor{28c4}@anchor{library/concurrent futures threadpoolexecutor-example}@anchor{d58}
@subsubsection ThreadPoolExecutor Example


@example
import concurrent.futures
import urllib.request

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

# Retrieve a single page and report the URL and contents
def load_url(url, timeout):
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()

# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    # Start the load operations and mark each future with its URL
    future_to_url = @{executor.submit(load_url, url, 60): url for url in URLS@}
    for future in concurrent.futures.as_completed(future_to_url):
        url = future_to_url[future]
        try:
            data = future.result()
        except Exception as exc:
            print('%r generated an exception: %s' % (url, exc))
        else:
            print('%r page is %d bytes' % (url, len(data)))
@end example

@node ProcessPoolExecutor,Future Objects,ThreadPoolExecutor,concurrent futures — Launching parallel tasks
@anchor{library/concurrent futures processpoolexecutor}@anchor{28c5}
@subsubsection ProcessPoolExecutor


The @ref{3da,,ProcessPoolExecutor} class is an @ref{28c0,,Executor} subclass that
uses a pool of processes to execute calls asynchronously.
@ref{3da,,ProcessPoolExecutor} uses the @ref{b3,,multiprocessing} module, which
allows it to side-step the @ref{7ae,,Global Interpreter Lock} but also means that
only picklable objects can be executed and returned.

The @code{__main__} module must be importable by worker subprocesses. This means
that @ref{3da,,ProcessPoolExecutor} will not work in the interactive interpreter.

Calling @ref{28c0,,Executor} or @ref{d57,,Future} methods from a callable submitted
to a @ref{3da,,ProcessPoolExecutor} will result in deadlock.

@geindex ProcessPoolExecutor (class in concurrent.futures)
@anchor{library/concurrent futures concurrent futures ProcessPoolExecutor}@anchor{3da}
@deffn {Class} concurrent.futures.ProcessPoolExecutor (max_workers=None, mp_context=None, initializer=None, initargs=(), max_tasks_per_child=None)

An @ref{28c0,,Executor} subclass that executes calls asynchronously using a pool
of at most @emph{max_workers} processes.  If @emph{max_workers} is @code{None} or not
given, it will default to the number of processors on the machine.
If @emph{max_workers} is less than or equal to @code{0}, then a @ref{17c,,ValueError}
will be raised.
On Windows, @emph{max_workers} must be less than or equal to @code{61}. If it is not
then @ref{17c,,ValueError} will be raised. If @emph{max_workers} is @code{None}, then
the default chosen will be at most @code{61}, even if more processors are
available.
@emph{mp_context} can be a multiprocessing context or None. It will be used to
launch the workers. If @emph{mp_context} is @code{None} or not given, the default
multiprocessing context is used.

@emph{initializer} is an optional callable that is called at the start of
each worker process; @emph{initargs} is a tuple of arguments passed to the
initializer.  Should @emph{initializer} raise an exception, all currently
pending jobs will raise a @ref{28c6,,BrokenProcessPool},
as well as any attempt to submit more jobs to the pool.

@emph{max_tasks_per_child} is an optional argument that specifies the maximum
number of tasks a single process can execute before it will exit and be
replaced with a fresh worker process. The default @emph{max_tasks_per_child} is
@code{None} which means worker processes will live as long as the pool.


Changed in version 3.3: When one of the worker processes terminates abruptly, a
@code{BrokenProcessPool} error is now raised.  Previously, behaviour
was undefined but operations on the executor or its futures would often
freeze or deadlock.



Changed in version 3.7: The @emph{mp_context} argument was added to allow users to control the
start_method for worker processes created by the pool.

Added the @emph{initializer} and @emph{initargs} arguments.



Changed in version 3.11: The @emph{max_tasks_per_child} argument was added to allow users to
control the lifetime of workers in the pool.

@end deffn

@menu
* ProcessPoolExecutor Example:: 

@end menu

@node ProcessPoolExecutor Example,,,ProcessPoolExecutor
@anchor{library/concurrent futures id2}@anchor{28c7}@anchor{library/concurrent futures processpoolexecutor-example}@anchor{d59}
@subsubsection ProcessPoolExecutor Example


@example
import concurrent.futures
import math

PRIMES = [
    112272535095293,
    112582705942171,
    112272535095293,
    115280095190773,
    115797848077099,
    1099726899285419]

def is_prime(n):
    if n < 2:
        return False
    if n == 2:
        return True
    if n % 2 == 0:
        return False

    sqrt_n = int(math.floor(math.sqrt(n)))
    for i in range(3, sqrt_n + 1, 2):
        if n % i == 0:
            return False
    return True

def main():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
            print('%d is prime: %s' % (number, prime))

if __name__ == '__main__':
    main()
@end example

@node Future Objects,Module Functions,ProcessPoolExecutor,concurrent futures — Launching parallel tasks
@anchor{library/concurrent futures future-objects}@anchor{28c8}
@subsubsection Future Objects


The @ref{d57,,Future} class encapsulates the asynchronous execution of a callable.
@ref{d57,,Future} instances are created by @ref{59a,,Executor.submit()}.

@geindex Future (class in concurrent.futures)
@anchor{library/concurrent futures concurrent futures Future}@anchor{d57}
@deffn {Class} concurrent.futures.Future

Encapsulates the asynchronous execution of a callable.  @ref{d57,,Future}
instances are created by @ref{59a,,Executor.submit()} and should not be created
directly except for testing.

@quotation

@geindex cancel() (concurrent.futures.Future method)
@anchor{library/concurrent futures concurrent futures Future cancel}@anchor{28c9}
@deffn {Method} cancel ()

Attempt to cancel the call.  If the call is currently being executed or
finished running and cannot be cancelled then the method will return
@code{False}, otherwise the call will be cancelled and the method will
return @code{True}.
@end deffn

@geindex cancelled() (concurrent.futures.Future method)
@anchor{library/concurrent futures concurrent futures Future cancelled}@anchor{28ca}
@deffn {Method} cancelled ()

Return @code{True} if the call was successfully cancelled.
@end deffn

@geindex running() (concurrent.futures.Future method)
@anchor{library/concurrent futures concurrent futures Future running}@anchor{28cb}
@deffn {Method} running ()

Return @code{True} if the call is currently being executed and cannot be
cancelled.
@end deffn

@geindex done() (concurrent.futures.Future method)
@anchor{library/concurrent futures concurrent futures Future done}@anchor{28cc}
@deffn {Method} done ()

Return @code{True} if the call was successfully cancelled or finished
running.
@end deffn

@geindex result() (concurrent.futures.Future method)
@anchor{library/concurrent futures concurrent futures Future result}@anchor{28cd}
@deffn {Method} result (timeout=None)

Return the value returned by the call. If the call hasn’t yet completed
then this method will wait up to @emph{timeout} seconds.  If the call hasn’t
completed in @emph{timeout} seconds, then a
@ref{2b8,,TimeoutError} will be raised. @emph{timeout} can be
an int or float.  If @emph{timeout} is not specified or @code{None}, there is no
limit to the wait time.

If the future is cancelled before completing then @ref{4d9,,CancelledError}
will be raised.

If the call raised an exception, this method will raise the same exception.
@end deffn

@geindex exception() (concurrent.futures.Future method)
@anchor{library/concurrent futures concurrent futures Future exception}@anchor{28ce}
@deffn {Method} exception (timeout=None)

Return the exception raised by the call.  If the call hasn’t yet
completed then this method will wait up to @emph{timeout} seconds.  If the
call hasn’t completed in @emph{timeout} seconds, then a
@ref{2b8,,TimeoutError} will be raised.  @emph{timeout} can be
an int or float.  If @emph{timeout} is not specified or @code{None}, there is no
limit to the wait time.

If the future is cancelled before completing then @ref{4d9,,CancelledError}
will be raised.

If the call completed without raising, @code{None} is returned.
@end deffn

@geindex add_done_callback() (concurrent.futures.Future method)
@anchor{library/concurrent futures concurrent futures Future add_done_callback}@anchor{28cf}
@deffn {Method} add_done_callback (fn)

Attaches the callable @emph{fn} to the future.  @emph{fn} will be called, with the
future as its only argument, when the future is cancelled or finishes
running.

Added callables are called in the order that they were added and are
always called in a thread belonging to the process that added them.  If
the callable raises an @ref{4d8,,Exception} subclass, it will be logged and
ignored.  If the callable raises a @ref{15f,,BaseException} subclass, the
behavior is undefined.

If the future has already completed or been cancelled, @emph{fn} will be
called immediately.
@end deffn
@end quotation

The following @ref{d57,,Future} methods are meant for use in unit tests and
@ref{28c0,,Executor} implementations.

@quotation

@geindex set_running_or_notify_cancel() (concurrent.futures.Future method)
@anchor{library/concurrent futures concurrent futures Future set_running_or_notify_cancel}@anchor{28d0}
@deffn {Method} set_running_or_notify_cancel ()

This method should only be called by @ref{28c0,,Executor} implementations
before executing the work associated with the @ref{d57,,Future} and by unit
tests.

If the method returns @code{False} then the @ref{d57,,Future} was cancelled,
i.e. @ref{28c9,,Future.cancel()} was called and returned @cite{True}.  Any threads
waiting on the @ref{d57,,Future} completing (i.e. through
@ref{13af,,as_completed()} or @ref{10bc,,wait()}) will be woken up.

If the method returns @code{True} then the @ref{d57,,Future} was not cancelled
and has been put in the running state, i.e. calls to
@ref{28cb,,Future.running()} will return @cite{True}.

This method can only be called once and cannot be called after
@ref{28d1,,Future.set_result()} or @ref{28d2,,Future.set_exception()} have been
called.
@end deffn

@geindex set_result() (concurrent.futures.Future method)
@anchor{library/concurrent futures concurrent futures Future set_result}@anchor{28d1}
@deffn {Method} set_result (result)

Sets the result of the work associated with the @ref{d57,,Future} to
@emph{result}.

This method should only be used by @ref{28c0,,Executor} implementations and
unit tests.


Changed in version 3.8: This method raises
@ref{28d3,,concurrent.futures.InvalidStateError} if the @ref{d57,,Future} is
already done.

@end deffn

@geindex set_exception() (concurrent.futures.Future method)
@anchor{library/concurrent futures concurrent futures Future set_exception}@anchor{28d2}
@deffn {Method} set_exception (exception)

Sets the result of the work associated with the @ref{d57,,Future} to the
@ref{4d8,,Exception} @emph{exception}.

This method should only be used by @ref{28c0,,Executor} implementations and
unit tests.


Changed in version 3.8: This method raises
@ref{28d3,,concurrent.futures.InvalidStateError} if the @ref{d57,,Future} is
already done.

@end deffn
@end quotation
@end deffn

@node Module Functions,Exception classes,Future Objects,concurrent futures — Launching parallel tasks
@anchor{library/concurrent futures module-functions}@anchor{28d4}
@subsubsection Module Functions


@geindex wait() (in module concurrent.futures)
@anchor{library/concurrent futures concurrent futures wait}@anchor{10bc}
@deffn {Function} concurrent.futures.wait (fs, timeout=None, return_when=ALL_COMPLETED)

Wait for the @ref{d57,,Future} instances (possibly created by different
@ref{28c0,,Executor} instances) given by @emph{fs} to complete. Duplicate futures
given to @emph{fs} are removed and will be returned only once. Returns a named
2-tuple of sets.  The first set, named @code{done}, contains the futures that
completed (finished or cancelled futures) before the wait completed.  The
second set, named @code{not_done}, contains the futures that did not complete
(pending or running futures).

@emph{timeout} can be used to control the maximum number of seconds to wait before
returning.  @emph{timeout} can be an int or float.  If @emph{timeout} is not specified
or @code{None}, there is no limit to the wait time.

@emph{return_when} indicates when this function should return.  It must be one of
the following constants:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Description

@item

@code{FIRST_COMPLETED}

@tab

The function will return when any
future finishes or is cancelled.

@item

@code{FIRST_EXCEPTION}

@tab

The function will return when any
future finishes by raising an
exception.  If no future raises an
exception then it is equivalent to
@code{ALL_COMPLETED}.

@item

@code{ALL_COMPLETED}

@tab

The function will return when all
futures finish or are cancelled.

@end multitable

@end deffn

@geindex as_completed() (in module concurrent.futures)
@anchor{library/concurrent futures concurrent futures as_completed}@anchor{13af}
@deffn {Function} concurrent.futures.as_completed (fs, timeout=None)

Returns an iterator over the @ref{d57,,Future} instances (possibly created by
different @ref{28c0,,Executor} instances) given by @emph{fs} that yields futures as
they complete (finished or cancelled futures). Any futures given by @emph{fs} that
are duplicated will be returned once. Any futures that completed before
@ref{13af,,as_completed()} is called will be yielded first.  The returned iterator
raises a @ref{2b8,,TimeoutError} if @ref{e6d,,__next__()}
is called and the result isn’t available after @emph{timeout} seconds from the
original call to @ref{13af,,as_completed()}.  @emph{timeout} can be an int or float. If
@emph{timeout} is not specified or @code{None}, there is no limit to the wait time.
@end deffn


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 3148

@item PEP 3148@footnote{https://www.python.org/dev/peps/pep-3148} – futures - execute computations asynchronously

The proposal which described this feature for inclusion in the Python
standard library.
@end table


@node Exception classes,,Module Functions,concurrent futures — Launching parallel tasks
@anchor{library/concurrent futures exception-classes}@anchor{28d5}
@subsubsection Exception classes


@geindex CancelledError
@anchor{library/concurrent futures concurrent futures CancelledError}@anchor{4d9}
@deffn {Exception} concurrent.futures.CancelledError

Raised when a future is cancelled.
@end deffn

@geindex TimeoutError
@anchor{library/concurrent futures concurrent futures TimeoutError}@anchor{28d6}
@deffn {Exception} concurrent.futures.TimeoutError

A deprecated alias of @ref{2b8,,TimeoutError},
raised when a future operation exceeds the given timeout.


Changed in version 3.11: This class was made an alias of @ref{2b8,,TimeoutError}.

@end deffn

@geindex BrokenExecutor
@anchor{library/concurrent futures concurrent futures BrokenExecutor}@anchor{28d7}
@deffn {Exception} concurrent.futures.BrokenExecutor

Derived from @ref{5af,,RuntimeError}, this exception class is raised
when an executor is broken for some reason, and cannot be used
to submit or execute new tasks.


New in version 3.7.

@end deffn

@geindex InvalidStateError
@anchor{library/concurrent futures concurrent futures InvalidStateError}@anchor{28d3}
@deffn {Exception} concurrent.futures.InvalidStateError

Raised when an operation is performed on a future that is not allowed
in the current state.


New in version 3.8.

@end deffn

@geindex BrokenThreadPool
@anchor{library/concurrent futures concurrent futures thread BrokenThreadPool}@anchor{28c3}
@deffn {Exception} concurrent.futures.thread.BrokenThreadPool

Derived from @ref{28d7,,BrokenExecutor}, this exception
class is raised when one of the workers of a @code{ThreadPoolExecutor}
has failed initializing.


New in version 3.7.

@end deffn

@geindex BrokenProcessPool
@anchor{library/concurrent futures concurrent futures process BrokenProcessPool}@anchor{28c6}
@deffn {Exception} concurrent.futures.process.BrokenProcessPool

Derived from @ref{28d7,,BrokenExecutor} (formerly
@ref{5af,,RuntimeError}), this exception class is raised when one of the
workers of a @code{ProcessPoolExecutor} has terminated in a non-clean
fashion (for example, if it was killed from the outside).


New in version 3.3.

@end deffn

@node subprocess — Subprocess management,sched — Event scheduler,concurrent futures — Launching parallel tasks,Concurrent Execution
@anchor{library/subprocess doc}@anchor{28d8}@anchor{library/subprocess module-subprocess}@anchor{f4}@anchor{library/subprocess subprocess-subprocess-management}@anchor{28d9}
@subsection @code{subprocess} — Subprocess management


@geindex module; subprocess

@strong{Source code:} Lib/subprocess.py@footnote{https://github.com/python/cpython/tree/main/Lib/subprocess.py}


__________________________________________________________________


The @ref{f4,,subprocess} module allows you to spawn new processes, connect to their
input/output/error pipes, and obtain their return codes.  This module intends to
replace several older modules and functions:

@example
os.system
os.spawn*
@end example

Information about how the @ref{f4,,subprocess} module can be used to replace these
modules and functions can be found in the following sections.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 324
PEP 324@footnote{https://www.python.org/dev/peps/pep-0324} – PEP proposing the subprocess module


@menu
* Using the subprocess Module:: 
* Security Considerations:: 
* Popen Objects:: 
* Windows Popen Helpers:: 
* Older high-level API:: 
* Replacing Older Functions with the subprocess Module:: 
* Legacy Shell Invocation Functions:: 
* Notes: Notes<2>. 

@end menu

@node Using the subprocess Module,Security Considerations,,subprocess — Subprocess management
@anchor{library/subprocess using-the-subprocess-module}@anchor{28da}
@subsubsection Using the @code{subprocess} Module


The recommended approach to invoking subprocesses is to use the @ref{6b7,,run()}
function for all use cases it can handle. For more advanced use cases, the
underlying @ref{5ae,,Popen} interface can be used directly.

The @ref{6b7,,run()} function was added in Python 3.5; if you need to retain
compatibility with older versions, see the @ref{28db,,Older high-level API} section.

@geindex run() (in module subprocess)
@anchor{library/subprocess subprocess run}@anchor{6b7}
@deffn {Function} subprocess.run (args, *, stdin=None, input=None, stdout=None, stderr=None, capture_output=False, shell=False, cwd=None, timeout=None, check=False, encoding=None, errors=None, text=None, env=None, universal_newlines=None, **other_popen_kwargs)

Run the command described by @emph{args}.  Wait for command to complete, then
return a @ref{9df,,CompletedProcess} instance.

The arguments shown above are merely the most common ones, described below
in @ref{28dc,,Frequently Used Arguments} (hence the use of keyword-only notation
in the abbreviated signature). The full function signature is largely the
same as that of the @ref{5ae,,Popen} constructor - most of the arguments to
this function are passed through to that interface. (@emph{timeout},  @emph{input},
@emph{check}, and @emph{capture_output} are not.)

If @emph{capture_output} is true, stdout and stderr will be captured.
When used, the internal @ref{5ae,,Popen} object is automatically created with
@code{stdout=PIPE} and @code{stderr=PIPE}. The @emph{stdout} and @emph{stderr} arguments may
not be supplied at the same time as @emph{capture_output}.  If you wish to capture
and combine both streams into one, use @code{stdout=PIPE} and @code{stderr=STDOUT}
instead of @emph{capture_output}.

The @emph{timeout} argument is passed to @ref{28dd,,Popen.communicate()}. If the timeout
expires, the child process will be killed and waited for.  The
@ref{28de,,TimeoutExpired} exception will be re-raised after the child process
has terminated.

The @emph{input} argument is passed to @ref{28dd,,Popen.communicate()} and thus to the
subprocess’s stdin.  If used it must be a byte sequence, or a string if
@emph{encoding} or @emph{errors} is specified or @emph{text} is true.  When
used, the internal @ref{5ae,,Popen} object is automatically created with
@code{stdin=PIPE}, and the @emph{stdin} argument may not be used as well.

If @emph{check} is true, and the process exits with a non-zero exit code, a
@ref{eb7,,CalledProcessError} exception will be raised. Attributes of that
exception hold the arguments, the exit code, and stdout and stderr if they
were captured.

If @emph{encoding} or @emph{errors} are specified, or @emph{text} is true,
file objects for stdin, stdout and stderr are opened in text mode using the
specified @emph{encoding} and @emph{errors} or the @ref{889,,io.TextIOWrapper} default.
The @emph{universal_newlines} argument is equivalent  to @emph{text} and is provided
for backwards compatibility. By default, file objects are opened in binary mode.

If @emph{env} is not @code{None}, it must be a mapping that defines the environment
variables for the new process; these are used instead of the default
behavior of inheriting the current process’ environment. It is passed directly
to @ref{5ae,,Popen}.

Examples:

@example
>>> subprocess.run(["ls", "-l"])  # doesn't capture output
CompletedProcess(args=['ls', '-l'], returncode=0)

>>> subprocess.run("exit 1", shell=True, check=True)
Traceback (most recent call last):
  ...
subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1

>>> subprocess.run(["ls", "-l", "/dev/null"], capture_output=True)
CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0,
stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n', stderr=b'@w{'})
@end example


New in version 3.5.



Changed in version 3.6: Added @emph{encoding} and @emph{errors} parameters



Changed in version 3.7: Added the @emph{text} parameter, as a more understandable alias of @emph{universal_newlines}.
Added the @emph{capture_output} parameter.

@end deffn

@geindex CompletedProcess (class in subprocess)
@anchor{library/subprocess subprocess CompletedProcess}@anchor{9df}
@deffn {Class} subprocess.CompletedProcess

The return value from @ref{6b7,,run()}, representing a process that has finished.

@geindex args (subprocess.CompletedProcess attribute)
@anchor{library/subprocess subprocess CompletedProcess args}@anchor{28df}
@deffn {Attribute} args

The arguments used to launch the process. This may be a list or a string.
@end deffn

@geindex returncode (subprocess.CompletedProcess attribute)
@anchor{library/subprocess subprocess CompletedProcess returncode}@anchor{28e0}
@deffn {Attribute} returncode

Exit status of the child process. Typically, an exit status of 0 indicates
that it ran successfully.

A negative value @code{-N} indicates that the child was terminated by signal
@code{N} (POSIX only).
@end deffn

@geindex stdout (subprocess.CompletedProcess attribute)
@anchor{library/subprocess subprocess CompletedProcess stdout}@anchor{28e1}
@deffn {Attribute} stdout

Captured stdout from the child process. A bytes sequence, or a string if
@ref{6b7,,run()} was called with an encoding, errors, or text=True.
@code{None} if stdout was not captured.

If you ran the process with @code{stderr=subprocess.STDOUT}, stdout and
stderr will be combined in this attribute, and @ref{28e2,,stderr} will be
@code{None}.
@end deffn

@geindex stderr (subprocess.CompletedProcess attribute)
@anchor{library/subprocess subprocess CompletedProcess stderr}@anchor{28e2}
@deffn {Attribute} stderr

Captured stderr from the child process. A bytes sequence, or a string if
@ref{6b7,,run()} was called with an encoding, errors, or text=True.
@code{None} if stderr was not captured.
@end deffn

@geindex check_returncode() (subprocess.CompletedProcess method)
@anchor{library/subprocess subprocess CompletedProcess check_returncode}@anchor{28e3}
@deffn {Method} check_returncode ()

If @ref{28e0,,returncode} is non-zero, raise a @ref{eb7,,CalledProcessError}.
@end deffn


New in version 3.5.

@end deffn

@geindex DEVNULL (in module subprocess)
@anchor{library/subprocess subprocess DEVNULL}@anchor{ce6}
@deffn {Data} subprocess.DEVNULL

Special value that can be used as the @emph{stdin}, @emph{stdout} or @emph{stderr} argument
to @ref{5ae,,Popen} and indicates that the special file @ref{12e2,,os.devnull}
will be used.


New in version 3.3.

@end deffn

@geindex PIPE (in module subprocess)
@anchor{library/subprocess subprocess PIPE}@anchor{6b8}
@deffn {Data} subprocess.PIPE

Special value that can be used as the @emph{stdin}, @emph{stdout} or @emph{stderr} argument
to @ref{5ae,,Popen} and indicates that a pipe to the standard stream should be
opened.  Most useful with @ref{28dd,,Popen.communicate()}.
@end deffn

@geindex STDOUT (in module subprocess)
@anchor{library/subprocess subprocess STDOUT}@anchor{28e4}
@deffn {Data} subprocess.STDOUT

Special value that can be used as the @emph{stderr} argument to @ref{5ae,,Popen} and
indicates that standard error should go into the same handle as standard
output.
@end deffn

@geindex SubprocessError
@anchor{library/subprocess subprocess SubprocessError}@anchor{28e5}
@deffn {Exception} subprocess.SubprocessError

Base class for all other exceptions from this module.


New in version 3.3.

@end deffn

@geindex TimeoutExpired
@anchor{library/subprocess subprocess TimeoutExpired}@anchor{28de}
@deffn {Exception} subprocess.TimeoutExpired

Subclass of @ref{28e5,,SubprocessError}, raised when a timeout expires
while waiting for a child process.

@geindex cmd (subprocess.TimeoutExpired attribute)
@anchor{library/subprocess subprocess TimeoutExpired cmd}@anchor{28e6}
@deffn {Attribute} cmd

Command that was used to spawn the child process.
@end deffn

@geindex timeout (subprocess.TimeoutExpired attribute)
@anchor{library/subprocess subprocess TimeoutExpired timeout}@anchor{28e7}
@deffn {Attribute} timeout

Timeout in seconds.
@end deffn

@geindex output (subprocess.TimeoutExpired attribute)
@anchor{library/subprocess subprocess TimeoutExpired output}@anchor{28e8}
@deffn {Attribute} output

Output of the child process if it was captured by @ref{6b7,,run()} or
@ref{b43,,check_output()}.  Otherwise, @code{None}.
@end deffn

@geindex stdout (subprocess.TimeoutExpired attribute)
@anchor{library/subprocess subprocess TimeoutExpired stdout}@anchor{28e9}
@deffn {Attribute} stdout

Alias for output, for symmetry with @ref{28ea,,stderr}.
@end deffn

@geindex stderr (subprocess.TimeoutExpired attribute)
@anchor{library/subprocess subprocess TimeoutExpired stderr}@anchor{28ea}
@deffn {Attribute} stderr

Stderr output of the child process if it was captured by @ref{6b7,,run()}.
Otherwise, @code{None}.
@end deffn


New in version 3.3.



Changed in version 3.5: @emph{stdout} and @emph{stderr} attributes added

@end deffn

@geindex CalledProcessError
@anchor{library/subprocess subprocess CalledProcessError}@anchor{eb7}
@deffn {Exception} subprocess.CalledProcessError

Subclass of @ref{28e5,,SubprocessError}, raised when a process run by
@ref{28eb,,check_call()} or @ref{b43,,check_output()} returns a non-zero exit status.

@geindex returncode (subprocess.CalledProcessError attribute)
@anchor{library/subprocess subprocess CalledProcessError returncode}@anchor{28ec}
@deffn {Attribute} returncode

Exit status of the child process.  If the process exited due to a
signal, this will be the negative signal number.
@end deffn

@geindex cmd (subprocess.CalledProcessError attribute)
@anchor{library/subprocess subprocess CalledProcessError cmd}@anchor{28ed}
@deffn {Attribute} cmd

Command that was used to spawn the child process.
@end deffn

@geindex output (subprocess.CalledProcessError attribute)
@anchor{library/subprocess subprocess CalledProcessError output}@anchor{28ee}
@deffn {Attribute} output

Output of the child process if it was captured by @ref{6b7,,run()} or
@ref{b43,,check_output()}.  Otherwise, @code{None}.
@end deffn

@geindex stdout (subprocess.CalledProcessError attribute)
@anchor{library/subprocess subprocess CalledProcessError stdout}@anchor{28ef}
@deffn {Attribute} stdout

Alias for output, for symmetry with @ref{28f0,,stderr}.
@end deffn

@geindex stderr (subprocess.CalledProcessError attribute)
@anchor{library/subprocess subprocess CalledProcessError stderr}@anchor{28f0}
@deffn {Attribute} stderr

Stderr output of the child process if it was captured by @ref{6b7,,run()}.
Otherwise, @code{None}.
@end deffn


Changed in version 3.5: @emph{stdout} and @emph{stderr} attributes added

@end deffn

@menu
* Frequently Used Arguments:: 
* Popen Constructor:: 
* Exceptions: Exceptions<8>. 

@end menu

@node Frequently Used Arguments,Popen Constructor,,Using the subprocess Module
@anchor{library/subprocess frequently-used-arguments}@anchor{28dc}@anchor{library/subprocess id1}@anchor{28f1}
@subsubsection Frequently Used Arguments


To support a wide variety of use cases, the @ref{5ae,,Popen} constructor (and
the convenience functions) accept a large number of optional arguments. For
most typical use cases, many of these arguments can be safely left at their
default values. The arguments that are most commonly needed are:

@quotation

@emph{args} is required for all calls and should be a string, or a sequence of
program arguments. Providing a sequence of arguments is generally
preferred, as it allows the module to take care of any required escaping
and quoting of arguments (e.g. to permit spaces in file names). If passing
a single string, either @emph{shell} must be @ref{748,,True} (see below) or else
the string must simply name the program to be executed without specifying
any arguments.

@emph{stdin}, @emph{stdout} and @emph{stderr} specify the executed program’s standard input,
standard output and standard error file handles, respectively.  Valid values
are @ref{6b8,,PIPE}, @ref{ce6,,DEVNULL}, an existing file descriptor (a positive
integer), an existing file object with a valid file descriptor, and @code{None}.
@ref{6b8,,PIPE} indicates that a new pipe to the child should be created.
@ref{ce6,,DEVNULL} indicates that the special file @ref{12e2,,os.devnull} will
be used.  With the default settings of @code{None}, no redirection will occur;
the child’s file handles will be inherited from the parent.
Additionally, @emph{stderr} can be @ref{28e4,,STDOUT}, which indicates that the
stderr data from the child process should be captured into the same file
handle as for @emph{stdout}.

@geindex universal newlines; subprocess module

If @emph{encoding} or @emph{errors} are specified, or @emph{text} (also known as
@emph{universal_newlines}) is true,
the file objects @emph{stdin}, @emph{stdout} and @emph{stderr} will be opened in text
mode using the @emph{encoding} and @emph{errors} specified in the call or the
defaults for @ref{889,,io.TextIOWrapper}.

For @emph{stdin}, line ending characters @code{'\n'} in the input will be converted
to the default line separator @ref{1a59,,os.linesep}. For @emph{stdout} and @emph{stderr},
all line endings in the output will be converted to @code{'\n'}.  For more
information see the documentation of the @ref{889,,io.TextIOWrapper} class
when the @emph{newline} argument to its constructor is @code{None}.

If text mode is not used, @emph{stdin}, @emph{stdout} and @emph{stderr} will be opened as
binary streams. No encoding or line ending conversion is performed.


New in version 3.6: Added @emph{encoding} and @emph{errors} parameters.



New in version 3.7: Added the @emph{text} parameter as an alias for @emph{universal_newlines}.


@cartouche
@quotation Note 
The newlines attribute of the file objects @ref{28f2,,Popen.stdin},
@ref{28f3,,Popen.stdout} and @ref{28f4,,Popen.stderr} are not updated by
the @ref{28dd,,Popen.communicate()} method.
@end quotation
@end cartouche

If @emph{shell} is @code{True}, the specified command will be executed through
the shell.  This can be useful if you are using Python primarily for the
enhanced control flow it offers over most system shells and still want
convenient access to other shell features such as shell pipes, filename
wildcards, environment variable expansion, and expansion of @code{~} to a
user’s home directory.  However, note that Python itself offers
implementations of many shell-like features (in particular, @ref{84,,glob},
@ref{7c,,fnmatch}, @ref{8de,,os.walk()}, @ref{f3f,,os.path.expandvars()},
@ref{520,,os.path.expanduser()}, and @ref{e4,,shutil}).


Changed in version 3.3: When @emph{universal_newlines} is @code{True}, the class uses the encoding
@ref{1b5,,locale.getpreferredencoding(False)}
instead of @code{locale.getpreferredencoding()}.  See the
@ref{889,,io.TextIOWrapper} class for more information on this change.


@cartouche
@quotation Note 
Read the @ref{28f5,,Security Considerations} section before using @code{shell=True}.
@end quotation
@end cartouche
@end quotation

These options, along with all of the other options, are described in more
detail in the @ref{5ae,,Popen} constructor documentation.

@node Popen Constructor,Exceptions<8>,Frequently Used Arguments,Using the subprocess Module
@anchor{library/subprocess popen-constructor}@anchor{28f6}
@subsubsection Popen Constructor


The underlying process creation and management in this module is handled by
the @ref{5ae,,Popen} class. It offers a lot of flexibility so that developers
are able to handle the less common cases not covered by the convenience
functions.

@geindex Popen (class in subprocess)
@anchor{library/subprocess subprocess Popen}@anchor{5ae}
@deffn {Class} subprocess.Popen (args, bufsize=- 1, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=True, shell=False, cwd=None, env=None, universal_newlines=None, startupinfo=None, creationflags=0, restore_signals=True, start_new_session=False, pass_fds=(), *, group=None, extra_groups=None, user=None, umask=- 1, encoding=None, errors=None, text=None, pipesize=- 1)

Execute a child program in a new process.  On POSIX, the class uses
@ref{242f,,os.execvpe()}-like behavior to execute the child program.  On Windows,
the class uses the Windows @code{CreateProcess()} function.  The arguments to
@ref{5ae,,Popen} are as follows.

@emph{args} should be a sequence of program arguments or else a single string
or @ref{63d,,path-like object}.
By default, the program to execute is the first item in @emph{args} if @emph{args} is
a sequence.  If @emph{args} is a string, the interpretation is
platform-dependent and described below.  See the @emph{shell} and @emph{executable}
arguments for additional differences from the default behavior.  Unless
otherwise stated, it is recommended to pass @emph{args} as a sequence.

@cartouche
@quotation Warning 
For maximum reliability, use a fully-qualified path for the executable.
To search for an unqualified name on 
@geindex PATH
@geindex environment variable; PATH
@code{PATH}, use
@ref{11ac,,shutil.which()}. On all platforms, passing @ref{585,,sys.executable}
is the recommended way to launch the current Python interpreter again,
and use the @code{-m} command-line format to launch an installed module.

Resolving the path of @emph{executable} (or the first item of @emph{args}) is
platform dependent. For POSIX, see @ref{242f,,os.execvpe()}, and note that
when resolving or searching for the executable path, @emph{cwd} overrides the
current working directory and @emph{env} can override the @code{PATH}
environment variable. For Windows, see the documentation of the
@code{lpApplicationName} and @code{lpCommandLine} parameters of WinAPI
@code{CreateProcess}, and note that when resolving or searching for the
executable path with @code{shell=False}, @emph{cwd} does not override the
current working directory and @emph{env} cannot override the @code{PATH}
environment variable. Using a full path avoids all of these variations.
@end quotation
@end cartouche

An example of passing some arguments to an external program
as a sequence is:

@example
Popen(["/usr/bin/git", "commit", "-m", "Fixes a bug."])
@end example

On POSIX, if @emph{args} is a string, the string is interpreted as the name or
path of the program to execute.  However, this can only be done if not
passing arguments to the program.

@cartouche
@quotation Note 
It may not be obvious how to break a shell command into a sequence of arguments,
especially in complex cases. @ref{443,,shlex.split()} can illustrate how to
determine the correct tokenization for @emph{args}:

@example
>>> import shlex, subprocess
>>> command_line = input()
/bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'"
>>> args = shlex.split(command_line)
>>> print(args)
['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"]
>>> p = subprocess.Popen(args) # Success!
@end example

Note in particular that options (such as @emph{-input}) and arguments (such
as @emph{eggs.txt}) that are separated by whitespace in the shell go in separate
list elements, while arguments that need quoting or backslash escaping when
used in the shell (such as filenames containing spaces or the @emph{echo} command
shown above) are single list elements.
@end quotation
@end cartouche

On Windows, if @emph{args} is a sequence, it will be converted to a string in a
manner described in @ref{28f7,,Converting an argument sequence to a string on Windows}.  This is because
the underlying @code{CreateProcess()} operates on strings.


Changed in version 3.6: @emph{args} parameter accepts a @ref{63d,,path-like object} if @emph{shell} is
@code{False} and a sequence containing path-like objects on POSIX.



Changed in version 3.8: @emph{args} parameter accepts a @ref{63d,,path-like object} if @emph{shell} is
@code{False} and a sequence containing bytes and path-like objects
on Windows.


The @emph{shell} argument (which defaults to @code{False}) specifies whether to use
the shell as the program to execute.  If @emph{shell} is @code{True}, it is
recommended to pass @emph{args} as a string rather than as a sequence.

On POSIX with @code{shell=True}, the shell defaults to @code{/bin/sh}.  If
@emph{args} is a string, the string specifies the command
to execute through the shell.  This means that the string must be
formatted exactly as it would be when typed at the shell prompt.  This
includes, for example, quoting or backslash escaping filenames with spaces in
them.  If @emph{args} is a sequence, the first item specifies the command string, and
any additional items will be treated as additional arguments to the shell
itself.  That is to say, @ref{5ae,,Popen} does the equivalent of:

@example
Popen(['/bin/sh', '-c', args[0], args[1], ...])
@end example

On Windows with @code{shell=True}, the 
@geindex COMSPEC
@geindex environment variable; COMSPEC
@code{COMSPEC} environment variable
specifies the default shell.  The only time you need to specify
@code{shell=True} on Windows is when the command you wish to execute is built
into the shell (e.g. @code{dir} or @code{copy}).  You do not need
@code{shell=True} to run a batch file or console-based executable.

@cartouche
@quotation Note 
Read the @ref{28f5,,Security Considerations} section before using @code{shell=True}.
@end quotation
@end cartouche

@emph{bufsize} will be supplied as the corresponding argument to the
@ref{1c8,,open()} function when creating the stdin/stdout/stderr pipe
file objects:


@itemize -

@item 
@code{0} means unbuffered (read and write are one
system call and can return short)

@item 
@code{1} means line buffered
(only usable if @code{universal_newlines=True} i.e., in a text mode)

@item 
any other positive value means use a buffer of approximately that
size

@item 
negative bufsize (the default) means the system default of
io.DEFAULT_BUFFER_SIZE will be used.
@end itemize


Changed in version 3.3.1: @emph{bufsize} now defaults to -1 to enable buffering by default to match the
behavior that most code expects.  In versions prior to Python 3.2.4 and
3.3.1 it incorrectly defaulted to @code{0} which was unbuffered
and allowed short reads.  This was unintentional and did not match the
behavior of Python 2 as most code expected.


The @emph{executable} argument specifies a replacement program to execute.   It
is very seldom needed.  When @code{shell=False}, @emph{executable} replaces the
program to execute specified by @emph{args}.  However, the original @emph{args} is
still passed to the program.  Most programs treat the program specified
by @emph{args} as the command name, which can then be different from the program
actually executed.  On POSIX, the @emph{args} name
becomes the display name for the executable in utilities such as
@code{ps}.  If @code{shell=True}, on POSIX the @emph{executable} argument
specifies a replacement shell for the default @code{/bin/sh}.


Changed in version 3.6: @emph{executable} parameter accepts a @ref{63d,,path-like object} on POSIX.



Changed in version 3.8: @emph{executable} parameter accepts a bytes and @ref{63d,,path-like object}
on Windows.


@emph{stdin}, @emph{stdout} and @emph{stderr} specify the executed program’s standard input,
standard output and standard error file handles, respectively.  Valid values
are @ref{6b8,,PIPE}, @ref{ce6,,DEVNULL}, an existing file descriptor (a positive
integer), an existing @ref{d68,,file object} with a valid file descriptor,
and @code{None}.  @ref{6b8,,PIPE} indicates that a new pipe to the child should
be created.  @ref{ce6,,DEVNULL} indicates that the special file
@ref{12e2,,os.devnull} will be used. With the default settings of @code{None},
no redirection will occur; the child’s file handles will be inherited from
the parent.  Additionally, @emph{stderr} can be @ref{28e4,,STDOUT}, which indicates
that the stderr data from the applications should be captured into the same
file handle as for stdout.

If @emph{preexec_fn} is set to a callable object, this object will be called in the
child process just before the child is executed.
(POSIX only)

@cartouche
@quotation Warning 
The @emph{preexec_fn} parameter is not safe to use in the presence of threads
in your application.  The child process could deadlock before exec is
called.
If you must use it, keep it trivial!  Minimize the number of libraries
you call into.
@end quotation
@end cartouche

@cartouche
@quotation Note 
If you need to modify the environment for the child use the @emph{env}
parameter rather than doing it in a @emph{preexec_fn}.
The @emph{start_new_session} parameter can take the place of a previously
common use of @emph{preexec_fn} to call os.setsid() in the child.
@end quotation
@end cartouche


Changed in version 3.8: The @emph{preexec_fn} parameter is no longer supported in subinterpreters.
The use of the parameter in a subinterpreter raises
@ref{5af,,RuntimeError}. The new restriction may affect applications that
are deployed in mod_wsgi, uWSGI, and other embedded environments.


If @emph{close_fds} is true, all file descriptors except @code{0}, @code{1} and
@code{2} will be closed before the child process is executed.  Otherwise
when @emph{close_fds} is false, file descriptors obey their inheritable flag
as described in @ref{a6e,,Inheritance of File Descriptors}.

On Windows, if @emph{close_fds} is true then no handles will be inherited by the
child process unless explicitly passed in the @code{handle_list} element of
@ref{749,,STARTUPINFO.lpAttributeList}, or by standard handle redirection.


Changed in version 3.2: The default for @emph{close_fds} was changed from @ref{659,,False} to
what is described above.



Changed in version 3.7: On Windows the default for @emph{close_fds} was changed from @ref{659,,False} to
@ref{748,,True} when redirecting the standard handles. It’s now possible to
set @emph{close_fds} to @ref{748,,True} when redirecting the standard handles.


@emph{pass_fds} is an optional sequence of file descriptors to keep open
between the parent and child.  Providing any @emph{pass_fds} forces
@emph{close_fds} to be @ref{748,,True}.  (POSIX only)


Changed in version 3.2: The @emph{pass_fds} parameter was added.


If @emph{cwd} is not @code{None}, the function changes the working directory to
@emph{cwd} before executing the child.  @emph{cwd} can be a string, bytes or
@ref{63d,,path-like} object.  On POSIX, the function
looks for @emph{executable} (or for the first item in @emph{args}) relative to @emph{cwd}
if the executable path is a relative path.


Changed in version 3.6: @emph{cwd} parameter accepts a @ref{63d,,path-like object} on POSIX.



Changed in version 3.7: @emph{cwd} parameter accepts a @ref{63d,,path-like object} on Windows.



Changed in version 3.8: @emph{cwd} parameter accepts a bytes object on Windows.


If @emph{restore_signals} is true (the default) all signals that Python has set to
SIG_IGN are restored to SIG_DFL in the child process before the exec.
Currently this includes the SIGPIPE, SIGXFZ and SIGXFSZ signals.
(POSIX only)


Changed in version 3.2: @emph{restore_signals} was added.


If @emph{start_new_session} is true the setsid() system call will be made in the
child process prior to the execution of the subprocess.  (POSIX only)


Changed in version 3.2: @emph{start_new_session} was added.


If @emph{group} is not @code{None}, the setregid() system call will be made in the
child process prior to the execution of the subprocess. If the provided
value is a string, it will be looked up via @ref{12f2,,grp.getgrnam()} and
the value in @code{gr_gid} will be used. If the value is an integer, it
will be passed verbatim. (POSIX only)

@ref{1629,,Availability}: POSIX


New in version 3.9.


If @emph{extra_groups} is not @code{None}, the setgroups() system call will be
made in the child process prior to the execution of the subprocess.
Strings provided in @emph{extra_groups} will be looked up via
@ref{12f2,,grp.getgrnam()} and the values in @code{gr_gid} will be used.
Integer values will be passed verbatim. (POSIX only)

@ref{1629,,Availability}: POSIX


New in version 3.9.


If @emph{user} is not @code{None}, the setreuid() system call will be made in the
child process prior to the execution of the subprocess. If the provided
value is a string, it will be looked up via @ref{12f1,,pwd.getpwnam()} and
the value in @code{pw_uid} will be used. If the value is an integer, it will
be passed verbatim. (POSIX only)

@ref{1629,,Availability}: POSIX


New in version 3.9.


If @emph{umask} is not negative, the umask() system call will be made in the
child process prior to the execution of the subprocess.

@ref{1629,,Availability}: POSIX


New in version 3.9.


If @emph{env} is not @code{None}, it must be a mapping that defines the environment
variables for the new process; these are used instead of the default
behavior of inheriting the current process’ environment.

@cartouche
@quotation Note 
If specified, @emph{env} must provide any variables required for the program to
execute.  On Windows, in order to run a side-by-side assembly@footnote{https://en.wikipedia.org/wiki/Side-by-Side_Assembly} the
specified @emph{env} @strong{must} include a valid 
@geindex SystemRoot
@geindex environment variable; SystemRoot
@code{SystemRoot}.
@end quotation
@end cartouche

If @emph{encoding} or @emph{errors} are specified, or @emph{text} is true, the file objects
@emph{stdin}, @emph{stdout} and @emph{stderr} are opened in text mode with the specified
encoding and @emph{errors}, as described above in @ref{28dc,,Frequently Used Arguments}.
The @emph{universal_newlines} argument is equivalent  to @emph{text} and is provided
for backwards compatibility. By default, file objects are opened in binary mode.


New in version 3.6: @emph{encoding} and @emph{errors} were added.



New in version 3.7: @emph{text} was added as a more readable alias for @emph{universal_newlines}.


If given, @emph{startupinfo} will be a @ref{28f8,,STARTUPINFO} object, which is
passed to the underlying @code{CreateProcess} function.
@emph{creationflags}, if given, can be one or more of the following flags:

@quotation


@itemize *

@item 
@ref{28f9,,CREATE_NEW_CONSOLE}

@item 
@ref{28fa,,CREATE_NEW_PROCESS_GROUP}

@item 
@ref{28fb,,ABOVE_NORMAL_PRIORITY_CLASS}

@item 
@ref{28fc,,BELOW_NORMAL_PRIORITY_CLASS}

@item 
@ref{28fd,,HIGH_PRIORITY_CLASS}

@item 
@ref{28fe,,IDLE_PRIORITY_CLASS}

@item 
@ref{28ff,,NORMAL_PRIORITY_CLASS}

@item 
@ref{2900,,REALTIME_PRIORITY_CLASS}

@item 
@ref{2901,,CREATE_NO_WINDOW}

@item 
@ref{2902,,DETACHED_PROCESS}

@item 
@ref{2903,,CREATE_DEFAULT_ERROR_MODE}

@item 
@ref{2904,,CREATE_BREAKAWAY_FROM_JOB}
@end itemize
@end quotation

@emph{pipesize} can be used to change the size of the pipe when
@ref{6b8,,PIPE} is used for @emph{stdin}, @emph{stdout} or @emph{stderr}. The size of the pipe
is only changed on platforms that support this (only Linux at this time of
writing). Other platforms will ignore this parameter.


New in version 3.10: The @code{pipesize} parameter was added.


Popen objects are supported as context managers via the @ref{168,,with} statement:
on exit, standard file descriptors are closed, and the process is waited for.

@example
with Popen(["ifconfig"], stdout=PIPE) as proc:
    log.write(proc.stdout.read())
@end example


Popen and the other functions in this module that use it raise an
@ref{1130,,auditing event} @code{subprocess.Popen} with arguments
@code{executable}, @code{args}, @code{cwd}, and @code{env}. The value for @code{args}
may be a single string or a list of strings, depending on platform.



Changed in version 3.2: Added context manager support.



Changed in version 3.6: Popen destructor now emits a @ref{77b,,ResourceWarning} warning if the child
process is still running.



Changed in version 3.8: Popen can use @ref{56e,,os.posix_spawn()} in some cases for better
performance. On Windows Subsystem for Linux and QEMU User Emulation,
Popen constructor using @ref{56e,,os.posix_spawn()} no longer raise an
exception on errors like missing program, but the child process fails
with a non-zero @ref{1241,,returncode}.

@end deffn

@node Exceptions<8>,,Popen Constructor,Using the subprocess Module
@anchor{library/subprocess exceptions}@anchor{2905}
@subsubsection Exceptions


Exceptions raised in the child process, before the new program has started to
execute, will be re-raised in the parent.

The most common exception raised is @ref{2a1,,OSError}.  This occurs, for example,
when trying to execute a non-existent file.  Applications should prepare for
@ref{2a1,,OSError} exceptions. Note that, when @code{shell=True}, @ref{2a1,,OSError}
will be raised by the child only if the selected shell itself was not found.
To determine if the shell failed to find the requested application, it is
necessary to check the return code or output from the subprocess.

A @ref{17c,,ValueError} will be raised if @ref{5ae,,Popen} is called with invalid
arguments.

@ref{28eb,,check_call()} and @ref{b43,,check_output()} will raise
@ref{eb7,,CalledProcessError} if the called process returns a non-zero return
code.

All of the functions and methods that accept a @emph{timeout} parameter, such as
@ref{6b9,,call()} and @ref{28dd,,Popen.communicate()} will raise @ref{28de,,TimeoutExpired} if
the timeout expires before the process exits.

Exceptions defined in this module all inherit from @ref{28e5,,SubprocessError}.

@quotation


New in version 3.3: The @ref{28e5,,SubprocessError} base class was added.

@end quotation

@node Security Considerations,Popen Objects,Using the subprocess Module,subprocess — Subprocess management
@anchor{library/subprocess security-considerations}@anchor{28f5}@anchor{library/subprocess subprocess-security}@anchor{2906}
@subsubsection Security Considerations


Unlike some other popen functions, this implementation will never
implicitly call a system shell.  This means that all characters,
including shell metacharacters, can safely be passed to child processes.
If the shell is invoked explicitly, via @code{shell=True}, it is the application’s
responsibility to ensure that all whitespace and metacharacters are
quoted appropriately to avoid
shell injection@footnote{https://en.wikipedia.org/wiki/Shell_injection#Shell_injection}
vulnerabilities. On @ref{2907,,some platforms}, it is possible
to use @ref{cbf,,shlex.quote()} for this escaping.

@node Popen Objects,Windows Popen Helpers,Security Considerations,subprocess — Subprocess management
@anchor{library/subprocess popen-objects}@anchor{2908}
@subsubsection Popen Objects


Instances of the @ref{5ae,,Popen} class have the following methods:

@geindex poll() (subprocess.Popen method)
@anchor{library/subprocess subprocess Popen poll}@anchor{2909}
@deffn {Method} Popen.poll ()

Check if child process has terminated.  Set and return
@ref{1241,,returncode} attribute. Otherwise, returns @code{None}.
@end deffn

@geindex wait() (subprocess.Popen method)
@anchor{library/subprocess subprocess Popen wait}@anchor{82d}
@deffn {Method} Popen.wait (timeout=None)

Wait for child process to terminate.  Set and return
@ref{1241,,returncode} attribute.

If the process does not terminate after @emph{timeout} seconds, raise a
@ref{28de,,TimeoutExpired} exception.  It is safe to catch this exception and
retry the wait.

@cartouche
@quotation Note 
This will deadlock when using @code{stdout=PIPE} or @code{stderr=PIPE}
and the child process generates enough output to a pipe such that
it blocks waiting for the OS pipe buffer to accept more data.
Use @ref{28dd,,Popen.communicate()} when using pipes to avoid that.
@end quotation
@end cartouche

@cartouche
@quotation Note 
The function is implemented using a busy loop (non-blocking call and
short sleeps). Use the @ref{9,,asyncio} module for an asynchronous wait:
see @ref{591,,asyncio.create_subprocess_exec}.
@end quotation
@end cartouche


Changed in version 3.3: @emph{timeout} was added.

@end deffn

@geindex communicate() (subprocess.Popen method)
@anchor{library/subprocess subprocess Popen communicate}@anchor{28dd}
@deffn {Method} Popen.communicate (input=None, timeout=None)

Interact with process: Send data to stdin.  Read data from stdout and stderr,
until end-of-file is reached.  Wait for process to terminate and set the
@ref{1241,,returncode} attribute.  The optional @emph{input} argument should be
data to be sent to the child process, or @code{None}, if no data should be sent
to the child.  If streams were opened in text mode, @emph{input} must be a string.
Otherwise, it must be bytes.

@ref{28dd,,communicate()} returns a tuple @code{(stdout_data, stderr_data)}.
The data will be strings if streams were opened in text mode; otherwise,
bytes.

Note that if you want to send data to the process’s stdin, you need to create
the Popen object with @code{stdin=PIPE}.  Similarly, to get anything other than
@code{None} in the result tuple, you need to give @code{stdout=PIPE} and/or
@code{stderr=PIPE} too.

If the process does not terminate after @emph{timeout} seconds, a
@ref{28de,,TimeoutExpired} exception will be raised.  Catching this exception and
retrying communication will not lose any output.

The child process is not killed if the timeout expires, so in order to
cleanup properly a well-behaved application should kill the child process and
finish communication:

@example
proc = subprocess.Popen(...)
try:
    outs, errs = proc.communicate(timeout=15)
except TimeoutExpired:
    proc.kill()
    outs, errs = proc.communicate()
@end example

@cartouche
@quotation Note 
The data read is buffered in memory, so do not use this method if the data
size is large or unlimited.
@end quotation
@end cartouche


Changed in version 3.3: @emph{timeout} was added.

@end deffn

@geindex send_signal() (subprocess.Popen method)
@anchor{library/subprocess subprocess Popen send_signal}@anchor{1240}
@deffn {Method} Popen.send_signal (signal)

Sends the signal @emph{signal} to the child.

Do nothing if the process completed.

@cartouche
@quotation Note 
On Windows, SIGTERM is an alias for @ref{290a,,terminate()}. CTRL_C_EVENT and
CTRL_BREAK_EVENT can be sent to processes started with a @emph{creationflags}
parameter which includes @cite{CREATE_NEW_PROCESS_GROUP}.
@end quotation
@end cartouche
@end deffn

@geindex terminate() (subprocess.Popen method)
@anchor{library/subprocess subprocess Popen terminate}@anchor{290a}
@deffn {Method} Popen.terminate ()

Stop the child. On POSIX OSs the method sends SIGTERM to the
child. On Windows the Win32 API function @code{TerminateProcess()} is called
to stop the child.
@end deffn

@geindex kill() (subprocess.Popen method)
@anchor{library/subprocess subprocess Popen kill}@anchor{290b}
@deffn {Method} Popen.kill ()

Kills the child. On POSIX OSs the function sends SIGKILL to the child.
On Windows @ref{290b,,kill()} is an alias for @ref{290a,,terminate()}.
@end deffn

The following attributes are also available:

@geindex args (subprocess.Popen attribute)
@anchor{library/subprocess subprocess Popen args}@anchor{290c}
@deffn {Attribute} Popen.args

The @emph{args} argument as it was passed to @ref{5ae,,Popen} – a
sequence of program arguments or else a single string.


New in version 3.3.

@end deffn

@geindex stdin (subprocess.Popen attribute)
@anchor{library/subprocess subprocess Popen stdin}@anchor{28f2}
@deffn {Attribute} Popen.stdin

If the @emph{stdin} argument was @ref{6b8,,PIPE}, this attribute is a writeable
stream object as returned by @ref{1c8,,open()}. If the @emph{encoding} or @emph{errors}
arguments were specified or the @emph{universal_newlines} argument was @code{True},
the stream is a text stream, otherwise it is a byte stream. If the @emph{stdin}
argument was not @ref{6b8,,PIPE}, this attribute is @code{None}.
@end deffn

@geindex stdout (subprocess.Popen attribute)
@anchor{library/subprocess subprocess Popen stdout}@anchor{28f3}
@deffn {Attribute} Popen.stdout

If the @emph{stdout} argument was @ref{6b8,,PIPE}, this attribute is a readable
stream object as returned by @ref{1c8,,open()}. Reading from the stream provides
output from the child process. If the @emph{encoding} or @emph{errors} arguments were
specified or the @emph{universal_newlines} argument was @code{True}, the stream is a
text stream, otherwise it is a byte stream. If the @emph{stdout} argument was not
@ref{6b8,,PIPE}, this attribute is @code{None}.
@end deffn

@geindex stderr (subprocess.Popen attribute)
@anchor{library/subprocess subprocess Popen stderr}@anchor{28f4}
@deffn {Attribute} Popen.stderr

If the @emph{stderr} argument was @ref{6b8,,PIPE}, this attribute is a readable
stream object as returned by @ref{1c8,,open()}. Reading from the stream provides
error output from the child process. If the @emph{encoding} or @emph{errors} arguments
were specified or the @emph{universal_newlines} argument was @code{True}, the stream
is a text stream, otherwise it is a byte stream. If the @emph{stderr} argument was
not @ref{6b8,,PIPE}, this attribute is @code{None}.
@end deffn

@cartouche
@quotation Warning 
Use @ref{28dd,,communicate()} rather than @ref{28f2,,.stdin.write},
@ref{28f3,,.stdout.read} or @ref{28f4,,.stderr.read} to avoid
deadlocks due to any of the other OS pipe buffers filling up and blocking the
child process.
@end quotation
@end cartouche

@geindex pid (subprocess.Popen attribute)
@anchor{library/subprocess subprocess Popen pid}@anchor{290d}
@deffn {Attribute} Popen.pid

The process ID of the child process.

Note that if you set the @emph{shell} argument to @code{True}, this is the process ID
of the spawned shell.
@end deffn

@geindex returncode (subprocess.Popen attribute)
@anchor{library/subprocess subprocess Popen returncode}@anchor{1241}
@deffn {Attribute} Popen.returncode

The child return code, set by @ref{2909,,poll()} and @ref{82d,,wait()} (and indirectly
by @ref{28dd,,communicate()}).  A @code{None} value indicates that the process
hasn’t terminated yet.

A negative value @code{-N} indicates that the child was terminated by signal
@code{N} (POSIX only).
@end deffn

@node Windows Popen Helpers,Older high-level API,Popen Objects,subprocess — Subprocess management
@anchor{library/subprocess windows-popen-helpers}@anchor{290e}
@subsubsection Windows Popen Helpers


The @ref{28f8,,STARTUPINFO} class and following constants are only available
on Windows.

@geindex STARTUPINFO (class in subprocess)
@anchor{library/subprocess subprocess STARTUPINFO}@anchor{28f8}
@deffn {Class} subprocess.STARTUPINFO (*, dwFlags=0, hStdInput=None, hStdOutput=None, hStdError=None, wShowWindow=0, lpAttributeList=None)

Partial support of the Windows
STARTUPINFO@footnote{https://msdn.microsoft.com/en-us/library/ms686331(v=vs.85).aspx}
structure is used for @ref{5ae,,Popen} creation.  The following attributes can
be set by passing them as keyword-only arguments.


Changed in version 3.7: Keyword-only argument support was added.


@geindex dwFlags (subprocess.STARTUPINFO attribute)
@anchor{library/subprocess subprocess STARTUPINFO dwFlags}@anchor{290f}
@deffn {Attribute} dwFlags

A bit field that determines whether certain @ref{28f8,,STARTUPINFO}
attributes are used when the process creates a window.

@example
si = subprocess.STARTUPINFO()
si.dwFlags = subprocess.STARTF_USESTDHANDLES | subprocess.STARTF_USESHOWWINDOW
@end example
@end deffn

@geindex hStdInput (subprocess.STARTUPINFO attribute)
@anchor{library/subprocess subprocess STARTUPINFO hStdInput}@anchor{2910}
@deffn {Attribute} hStdInput

If @ref{290f,,dwFlags} specifies @ref{2911,,STARTF_USESTDHANDLES}, this attribute
is the standard input handle for the process. If
@ref{2911,,STARTF_USESTDHANDLES} is not specified, the default for standard
input is the keyboard buffer.
@end deffn

@geindex hStdOutput (subprocess.STARTUPINFO attribute)
@anchor{library/subprocess subprocess STARTUPINFO hStdOutput}@anchor{2912}
@deffn {Attribute} hStdOutput

If @ref{290f,,dwFlags} specifies @ref{2911,,STARTF_USESTDHANDLES}, this attribute
is the standard output handle for the process. Otherwise, this attribute
is ignored and the default for standard output is the console window’s
buffer.
@end deffn

@geindex hStdError (subprocess.STARTUPINFO attribute)
@anchor{library/subprocess subprocess STARTUPINFO hStdError}@anchor{2913}
@deffn {Attribute} hStdError

If @ref{290f,,dwFlags} specifies @ref{2911,,STARTF_USESTDHANDLES}, this attribute
is the standard error handle for the process. Otherwise, this attribute is
ignored and the default for standard error is the console window’s buffer.
@end deffn

@geindex wShowWindow (subprocess.STARTUPINFO attribute)
@anchor{library/subprocess subprocess STARTUPINFO wShowWindow}@anchor{2914}
@deffn {Attribute} wShowWindow

If @ref{290f,,dwFlags} specifies @ref{2915,,STARTF_USESHOWWINDOW}, this attribute
can be any of the values that can be specified in the @code{nCmdShow}
parameter for the
ShowWindow@footnote{https://msdn.microsoft.com/en-us/library/ms633548(v=vs.85).aspx}
function, except for @code{SW_SHOWDEFAULT}. Otherwise, this attribute is
ignored.

@ref{2916,,SW_HIDE} is provided for this attribute. It is used when
@ref{5ae,,Popen} is called with @code{shell=True}.
@end deffn

@geindex lpAttributeList (subprocess.STARTUPINFO attribute)
@anchor{library/subprocess subprocess STARTUPINFO lpAttributeList}@anchor{749}
@deffn {Attribute} lpAttributeList

A dictionary of additional attributes for process creation as given in
@code{STARTUPINFOEX}, see
UpdateProcThreadAttribute@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/ms686880(v=vs.85).aspx}.

Supported attributes:


@table @asis

@item @strong{handle_list}

Sequence of handles that will be inherited. @emph{close_fds} must be true if
non-empty.

The handles must be temporarily made inheritable by
@ref{a72,,os.set_handle_inheritable()} when passed to the @ref{5ae,,Popen}
constructor, else @ref{2a1,,OSError} will be raised with Windows error
@code{ERROR_INVALID_PARAMETER} (87).

@cartouche
@quotation Warning 
In a multithreaded process, use caution to avoid leaking handles
that are marked inheritable when combining this feature with
concurrent calls to other process creation functions that inherit
all handles such as @ref{fbb,,os.system()}.  This also applies to
standard handle redirection, which temporarily creates inheritable
handles.
@end quotation
@end cartouche
@end table


New in version 3.7.

@end deffn
@end deffn

@menu
* Windows Constants:: 

@end menu

@node Windows Constants,,,Windows Popen Helpers
@anchor{library/subprocess windows-constants}@anchor{2917}
@subsubsection Windows Constants


The @ref{f4,,subprocess} module exposes the following constants.

@geindex STD_INPUT_HANDLE (in module subprocess)
@anchor{library/subprocess subprocess STD_INPUT_HANDLE}@anchor{2918}
@deffn {Data} subprocess.STD_INPUT_HANDLE

The standard input device. Initially, this is the console input buffer,
@code{CONIN$}.
@end deffn

@geindex STD_OUTPUT_HANDLE (in module subprocess)
@anchor{library/subprocess subprocess STD_OUTPUT_HANDLE}@anchor{2919}
@deffn {Data} subprocess.STD_OUTPUT_HANDLE

The standard output device. Initially, this is the active console screen
buffer, @code{CONOUT$}.
@end deffn

@geindex STD_ERROR_HANDLE (in module subprocess)
@anchor{library/subprocess subprocess STD_ERROR_HANDLE}@anchor{291a}
@deffn {Data} subprocess.STD_ERROR_HANDLE

The standard error device. Initially, this is the active console screen
buffer, @code{CONOUT$}.
@end deffn

@geindex SW_HIDE (in module subprocess)
@anchor{library/subprocess subprocess SW_HIDE}@anchor{2916}
@deffn {Data} subprocess.SW_HIDE

Hides the window. Another window will be activated.
@end deffn

@geindex STARTF_USESTDHANDLES (in module subprocess)
@anchor{library/subprocess subprocess STARTF_USESTDHANDLES}@anchor{2911}
@deffn {Data} subprocess.STARTF_USESTDHANDLES

Specifies that the @ref{2910,,STARTUPINFO.hStdInput},
@ref{2912,,STARTUPINFO.hStdOutput}, and @ref{2913,,STARTUPINFO.hStdError} attributes
contain additional information.
@end deffn

@geindex STARTF_USESHOWWINDOW (in module subprocess)
@anchor{library/subprocess subprocess STARTF_USESHOWWINDOW}@anchor{2915}
@deffn {Data} subprocess.STARTF_USESHOWWINDOW

Specifies that the @ref{2914,,STARTUPINFO.wShowWindow} attribute contains
additional information.
@end deffn

@geindex CREATE_NEW_CONSOLE (in module subprocess)
@anchor{library/subprocess subprocess CREATE_NEW_CONSOLE}@anchor{28f9}
@deffn {Data} subprocess.CREATE_NEW_CONSOLE

The new process has a new console, instead of inheriting its parent’s
console (the default).
@end deffn

@geindex CREATE_NEW_PROCESS_GROUP (in module subprocess)
@anchor{library/subprocess subprocess CREATE_NEW_PROCESS_GROUP}@anchor{28fa}
@deffn {Data} subprocess.CREATE_NEW_PROCESS_GROUP

A @ref{5ae,,Popen} @code{creationflags} parameter to specify that a new process
group will be created. This flag is necessary for using @ref{ef3,,os.kill()}
on the subprocess.

This flag is ignored if @ref{28f9,,CREATE_NEW_CONSOLE} is specified.
@end deffn

@geindex ABOVE_NORMAL_PRIORITY_CLASS (in module subprocess)
@anchor{library/subprocess subprocess ABOVE_NORMAL_PRIORITY_CLASS}@anchor{28fb}
@deffn {Data} subprocess.ABOVE_NORMAL_PRIORITY_CLASS

A @ref{5ae,,Popen} @code{creationflags} parameter to specify that a new process
will have an above average priority.


New in version 3.7.

@end deffn

@geindex BELOW_NORMAL_PRIORITY_CLASS (in module subprocess)
@anchor{library/subprocess subprocess BELOW_NORMAL_PRIORITY_CLASS}@anchor{28fc}
@deffn {Data} subprocess.BELOW_NORMAL_PRIORITY_CLASS

A @ref{5ae,,Popen} @code{creationflags} parameter to specify that a new process
will have a below average priority.


New in version 3.7.

@end deffn

@geindex HIGH_PRIORITY_CLASS (in module subprocess)
@anchor{library/subprocess subprocess HIGH_PRIORITY_CLASS}@anchor{28fd}
@deffn {Data} subprocess.HIGH_PRIORITY_CLASS

A @ref{5ae,,Popen} @code{creationflags} parameter to specify that a new process
will have a high priority.


New in version 3.7.

@end deffn

@geindex IDLE_PRIORITY_CLASS (in module subprocess)
@anchor{library/subprocess subprocess IDLE_PRIORITY_CLASS}@anchor{28fe}
@deffn {Data} subprocess.IDLE_PRIORITY_CLASS

A @ref{5ae,,Popen} @code{creationflags} parameter to specify that a new process
will have an idle (lowest) priority.


New in version 3.7.

@end deffn

@geindex NORMAL_PRIORITY_CLASS (in module subprocess)
@anchor{library/subprocess subprocess NORMAL_PRIORITY_CLASS}@anchor{28ff}
@deffn {Data} subprocess.NORMAL_PRIORITY_CLASS

A @ref{5ae,,Popen} @code{creationflags} parameter to specify that a new process
will have an normal priority. (default)


New in version 3.7.

@end deffn

@geindex REALTIME_PRIORITY_CLASS (in module subprocess)
@anchor{library/subprocess subprocess REALTIME_PRIORITY_CLASS}@anchor{2900}
@deffn {Data} subprocess.REALTIME_PRIORITY_CLASS

A @ref{5ae,,Popen} @code{creationflags} parameter to specify that a new process
will have realtime priority.
You should almost never use REALTIME_PRIORITY_CLASS, because this interrupts
system threads that manage mouse input, keyboard input, and background disk
flushing. This class can be appropriate for applications that “talk” directly
to hardware or that perform brief tasks that should have limited interruptions.


New in version 3.7.

@end deffn

@geindex CREATE_NO_WINDOW (in module subprocess)
@anchor{library/subprocess subprocess CREATE_NO_WINDOW}@anchor{2901}
@deffn {Data} subprocess.CREATE_NO_WINDOW

A @ref{5ae,,Popen} @code{creationflags} parameter to specify that a new process
will not create a window.


New in version 3.7.

@end deffn

@geindex DETACHED_PROCESS (in module subprocess)
@anchor{library/subprocess subprocess DETACHED_PROCESS}@anchor{2902}
@deffn {Data} subprocess.DETACHED_PROCESS

A @ref{5ae,,Popen} @code{creationflags} parameter to specify that a new process
will not inherit its parent’s console.
This value cannot be used with CREATE_NEW_CONSOLE.


New in version 3.7.

@end deffn

@geindex CREATE_DEFAULT_ERROR_MODE (in module subprocess)
@anchor{library/subprocess subprocess CREATE_DEFAULT_ERROR_MODE}@anchor{2903}
@deffn {Data} subprocess.CREATE_DEFAULT_ERROR_MODE

A @ref{5ae,,Popen} @code{creationflags} parameter to specify that a new process
does not inherit the error mode of the calling process. Instead, the new
process gets the default error mode.
This feature is particularly useful for multithreaded shell applications
that run with hard errors disabled.


New in version 3.7.

@end deffn

@geindex CREATE_BREAKAWAY_FROM_JOB (in module subprocess)
@anchor{library/subprocess subprocess CREATE_BREAKAWAY_FROM_JOB}@anchor{2904}
@deffn {Data} subprocess.CREATE_BREAKAWAY_FROM_JOB

A @ref{5ae,,Popen} @code{creationflags} parameter to specify that a new process
is not associated with the job.


New in version 3.7.

@end deffn

@node Older high-level API,Replacing Older Functions with the subprocess Module,Windows Popen Helpers,subprocess — Subprocess management
@anchor{library/subprocess call-function-trio}@anchor{28db}@anchor{library/subprocess older-high-level-api}@anchor{291b}
@subsubsection Older high-level API


Prior to Python 3.5, these three functions comprised the high level API to
subprocess. You can now use @ref{6b7,,run()} in many cases, but lots of existing code
calls these functions.

@geindex call() (in module subprocess)
@anchor{library/subprocess subprocess call}@anchor{6b9}
@deffn {Function} subprocess.call (args, *, stdin=None, stdout=None, stderr=None, shell=False, cwd=None, timeout=None, **other_popen_kwargs)

Run the command described by @emph{args}.  Wait for command to complete, then
return the @ref{1241,,returncode} attribute.

Code needing to capture stdout or stderr should use @ref{6b7,,run()} instead:

@example
run(...).returncode
@end example

To suppress stdout or stderr, supply a value of @ref{ce6,,DEVNULL}.

The arguments shown above are merely some common ones.
The full function signature is the
same as that of the @ref{5ae,,Popen} constructor - this function passes all
supplied arguments other than @emph{timeout} directly through to that interface.

@cartouche
@quotation Note 
Do not use @code{stdout=PIPE} or @code{stderr=PIPE} with this
function.  The child process will block if it generates enough
output to a pipe to fill up the OS pipe buffer as the pipes are
not being read from.
@end quotation
@end cartouche


Changed in version 3.3: @emph{timeout} was added.

@end deffn

@geindex check_call() (in module subprocess)
@anchor{library/subprocess subprocess check_call}@anchor{28eb}
@deffn {Function} subprocess.check_call (args, *, stdin=None, stdout=None, stderr=None, shell=False, cwd=None, timeout=None, **other_popen_kwargs)

Run command with arguments.  Wait for command to complete. If the return
code was zero then return, otherwise raise @ref{eb7,,CalledProcessError}. The
@ref{eb7,,CalledProcessError} object will have the return code in the
@ref{28ec,,returncode} attribute.
If @ref{28eb,,check_call()} was unable to start the process it will propagate the exception
that was raised.

Code needing to capture stdout or stderr should use @ref{6b7,,run()} instead:

@example
run(..., check=True)
@end example

To suppress stdout or stderr, supply a value of @ref{ce6,,DEVNULL}.

The arguments shown above are merely some common ones.
The full function signature is the
same as that of the @ref{5ae,,Popen} constructor - this function passes all
supplied arguments other than @emph{timeout} directly through to that interface.

@cartouche
@quotation Note 
Do not use @code{stdout=PIPE} or @code{stderr=PIPE} with this
function.  The child process will block if it generates enough
output to a pipe to fill up the OS pipe buffer as the pipes are
not being read from.
@end quotation
@end cartouche


Changed in version 3.3: @emph{timeout} was added.

@end deffn

@geindex check_output() (in module subprocess)
@anchor{library/subprocess subprocess check_output}@anchor{b43}
@deffn {Function} subprocess.check_output (args, *, stdin=None, stderr=None, shell=False, cwd=None, encoding=None, errors=None, universal_newlines=None, timeout=None, text=None, **other_popen_kwargs)

Run command with arguments and return its output.

If the return code was non-zero it raises a @ref{eb7,,CalledProcessError}. The
@ref{eb7,,CalledProcessError} object will have the return code in the
@ref{28ec,,returncode} attribute and any output in the
@ref{28ee,,output} attribute.

This is equivalent to:

@example
run(..., check=True, stdout=PIPE).stdout
@end example

The arguments shown above are merely some common ones.
The full function signature is largely the same as that of @ref{6b7,,run()} -
most arguments are passed directly through to that interface.
One API deviation from @ref{6b7,,run()} behavior exists: passing @code{input=None}
will behave the same as @code{input=b'@w{'}} (or @code{input='@w{'}}, depending on other
arguments) rather than using the parent’s standard input file handle.

By default, this function will return the data as encoded bytes. The actual
encoding of the output data may depend on the command being invoked, so the
decoding to text will often need to be handled at the application level.

This behaviour may be overridden by setting @emph{text}, @emph{encoding}, @emph{errors},
or @emph{universal_newlines} to @code{True} as described in
@ref{28dc,,Frequently Used Arguments} and @ref{6b7,,run()}.

To also capture standard error in the result, use
@code{stderr=subprocess.STDOUT}:

@example
>>> subprocess.check_output(
...     "ls non_existent_file; exit 0",
...     stderr=subprocess.STDOUT,
...     shell=True)
'ls: non_existent_file: No such file or directory\n'
@end example


New in version 3.1.



Changed in version 3.3: @emph{timeout} was added.



Changed in version 3.4: Support for the @emph{input} keyword argument was added.



Changed in version 3.6: @emph{encoding} and @emph{errors} were added.  See @ref{6b7,,run()} for details.



New in version 3.7: @emph{text} was added as a more readable alias for @emph{universal_newlines}.

@end deffn

@node Replacing Older Functions with the subprocess Module,Legacy Shell Invocation Functions,Older high-level API,subprocess — Subprocess management
@anchor{library/subprocess replacing-older-functions-with-the-subprocess-module}@anchor{291c}@anchor{library/subprocess subprocess-replacements}@anchor{d28}
@subsubsection Replacing Older Functions with the @code{subprocess} Module


In this section, “a becomes b” means that b can be used as a replacement for a.

@cartouche
@quotation Note 
All “a” functions in this section fail (more or less) silently if the
executed program cannot be found; the “b” replacements raise @ref{2a1,,OSError}
instead.

In addition, the replacements using @ref{b43,,check_output()} will fail with a
@ref{eb7,,CalledProcessError} if the requested operation produces a non-zero
return code. The output is still available as the
@ref{28ee,,output} attribute of the raised exception.
@end quotation
@end cartouche

In the following examples, we assume that the relevant functions have already
been imported from the @ref{f4,,subprocess} module.

@menu
* Replacing /bin/sh shell command substitution:: 
* Replacing shell pipeline:: 
* Replacing os.system(): Replacing os system. 
* Replacing the os.spawn family: Replacing the os spawn family. 
* Replacing os.popen()@comma{} os.popen2()@comma{} os.popen3(): Replacing os popen os popen2 os popen3. 
* Replacing functions from the popen2 module:: 

@end menu

@node Replacing /bin/sh shell command substitution,Replacing shell pipeline,,Replacing Older Functions with the subprocess Module
@anchor{library/subprocess replacing-bin-sh-shell-command-substitution}@anchor{291d}
@subsubsection Replacing @code{/bin/sh} shell command substitution


@example
output=$(mycmd myarg)
@end example

becomes:

@example
output = check_output(["mycmd", "myarg"])
@end example

@node Replacing shell pipeline,Replacing os system,Replacing /bin/sh shell command substitution,Replacing Older Functions with the subprocess Module
@anchor{library/subprocess replacing-shell-pipeline}@anchor{291e}
@subsubsection Replacing shell pipeline


@example
output=$(dmesg | grep hda)
@end example

becomes:

@example
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]
@end example

The @code{p1.stdout.close()} call after starting the p2 is important in order for
p1 to receive a SIGPIPE if p2 exits before p1.

Alternatively, for trusted input, the shell’s own pipeline support may still
be used directly:

@example
output=$(dmesg | grep hda)
@end example

becomes:

@example
output = check_output("dmesg | grep hda", shell=True)
@end example

@node Replacing os system,Replacing the os spawn family,Replacing shell pipeline,Replacing Older Functions with the subprocess Module
@anchor{library/subprocess replacing-os-system}@anchor{291f}
@subsubsection Replacing @code{os.system()}


@example
sts = os.system("mycmd" + " myarg")
# becomes
retcode = call("mycmd" + " myarg", shell=True)
@end example

Notes:


@itemize *

@item 
Calling the program through the shell is usually not required.

@item 
The @ref{6b9,,call()} return value is encoded differently to that of
@ref{fbb,,os.system()}.

@item 
The @ref{fbb,,os.system()} function ignores SIGINT and SIGQUIT signals while
the command is running, but the caller must do this separately when
using the @ref{f4,,subprocess} module.
@end itemize

A more realistic example would look like this:

@example
try:
    retcode = call("mycmd" + " myarg", shell=True)
    if retcode < 0:
        print("Child was terminated by signal", -retcode, file=sys.stderr)
    else:
        print("Child returned", retcode, file=sys.stderr)
except OSError as e:
    print("Execution failed:", e, file=sys.stderr)
@end example

@node Replacing the os spawn family,Replacing os popen os popen2 os popen3,Replacing os system,Replacing Older Functions with the subprocess Module
@anchor{library/subprocess replacing-the-os-spawn-family}@anchor{2920}
@subsubsection Replacing the @code{os.spawn} family


P_NOWAIT example:

@example
pid = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg")
==>
pid = Popen(["/bin/mycmd", "myarg"]).pid
@end example

P_WAIT example:

@example
retcode = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg")
==>
retcode = call(["/bin/mycmd", "myarg"])
@end example

Vector example:

@example
os.spawnvp(os.P_NOWAIT, path, args)
==>
Popen([path] + args[1:])
@end example

Environment example:

@example
os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)
==>
Popen(["/bin/mycmd", "myarg"], env=@{"PATH": "/usr/bin"@})
@end example

@node Replacing os popen os popen2 os popen3,Replacing functions from the popen2 module,Replacing the os spawn family,Replacing Older Functions with the subprocess Module
@anchor{library/subprocess replacing-os-popen-os-popen2-os-popen3}@anchor{2921}
@subsubsection Replacing @code{os.popen()}, @code{os.popen2()}, @code{os.popen3()}


@example
(child_stdin, child_stdout) = os.popen2(cmd, mode, bufsize)
==>
p = Popen(cmd, shell=True, bufsize=bufsize,
          stdin=PIPE, stdout=PIPE, close_fds=True)
(child_stdin, child_stdout) = (p.stdin, p.stdout)
@end example

@example
(child_stdin,
 child_stdout,
 child_stderr) = os.popen3(cmd, mode, bufsize)
==>
p = Popen(cmd, shell=True, bufsize=bufsize,
          stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True)
(child_stdin,
 child_stdout,
 child_stderr) = (p.stdin, p.stdout, p.stderr)
@end example

@example
(child_stdin, child_stdout_and_stderr) = os.popen4(cmd, mode, bufsize)
==>
p = Popen(cmd, shell=True, bufsize=bufsize,
          stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
(child_stdin, child_stdout_and_stderr) = (p.stdin, p.stdout)
@end example

Return code handling translates as follows:

@example
pipe = os.popen(cmd, 'w')
...
rc = pipe.close()
if rc is not None and rc >> 8:
    print("There were some errors")
==>
process = Popen(cmd, stdin=PIPE)
...
process.stdin.close()
if process.wait() != 0:
    print("There were some errors")
@end example

@node Replacing functions from the popen2 module,,Replacing os popen os popen2 os popen3,Replacing Older Functions with the subprocess Module
@anchor{library/subprocess replacing-functions-from-the-popen2-module}@anchor{2922}
@subsubsection Replacing functions from the @code{popen2} module


@cartouche
@quotation Note 
If the cmd argument to popen2 functions is a string, the command is executed
through /bin/sh.  If it is a list, the command is directly executed.
@end quotation
@end cartouche

@example
(child_stdout, child_stdin) = popen2.popen2("somestring", bufsize, mode)
==>
p = Popen("somestring", shell=True, bufsize=bufsize,
          stdin=PIPE, stdout=PIPE, close_fds=True)
(child_stdout, child_stdin) = (p.stdout, p.stdin)
@end example

@example
(child_stdout, child_stdin) = popen2.popen2(["mycmd", "myarg"], bufsize, mode)
==>
p = Popen(["mycmd", "myarg"], bufsize=bufsize,
          stdin=PIPE, stdout=PIPE, close_fds=True)
(child_stdout, child_stdin) = (p.stdout, p.stdin)
@end example

@code{popen2.Popen3} and @code{popen2.Popen4} basically work as
@ref{5ae,,subprocess.Popen}, except that:


@itemize *

@item 
@ref{5ae,,Popen} raises an exception if the execution fails.

@item 
The @emph{capturestderr} argument is replaced with the @emph{stderr} argument.

@item 
@code{stdin=PIPE} and @code{stdout=PIPE} must be specified.

@item 
popen2 closes all file descriptors by default, but you have to specify
@code{close_fds=True} with @ref{5ae,,Popen} to guarantee this behavior on
all platforms or past Python versions.
@end itemize

@node Legacy Shell Invocation Functions,Notes<2>,Replacing Older Functions with the subprocess Module,subprocess — Subprocess management
@anchor{library/subprocess legacy-shell-invocation-functions}@anchor{2923}
@subsubsection Legacy Shell Invocation Functions


This module also provides the following legacy functions from the 2.x
@code{commands} module. These operations implicitly invoke the system shell and
none of the guarantees described above regarding security and exception
handling consistency are valid for these functions.

@geindex getstatusoutput() (in module subprocess)
@anchor{library/subprocess subprocess getstatusoutput}@anchor{b44}
@deffn {Function} subprocess.getstatusoutput (cmd)

Return @code{(exitcode, output)} of executing @emph{cmd} in a shell.

Execute the string @emph{cmd} in a shell with @code{Popen.check_output()} and
return a 2-tuple @code{(exitcode, output)}. The locale encoding is used;
see the notes on @ref{28dc,,Frequently Used Arguments} for more details.

A trailing newline is stripped from the output.
The exit code for the command can be interpreted as the return code
of subprocess.  Example:

@example
>>> subprocess.getstatusoutput('ls /bin/ls')
(0, '/bin/ls')
>>> subprocess.getstatusoutput('cat /bin/junk')
(1, 'cat: /bin/junk: No such file or directory')
>>> subprocess.getstatusoutput('/bin/junk')
(127, 'sh: /bin/junk: not found')
>>> subprocess.getstatusoutput('/bin/kill $$')
(-15, '@w{'})
@end example

@ref{1629,,Availability}: POSIX & Windows.


Changed in version 3.3.4: Windows support was added.

The function now returns (exitcode, output) instead of (status, output)
as it did in Python 3.3.3 and earlier.  exitcode has the same value as
@ref{1241,,returncode}.

@end deffn

@geindex getoutput() (in module subprocess)
@anchor{library/subprocess subprocess getoutput}@anchor{2924}
@deffn {Function} subprocess.getoutput (cmd)

Return output (stdout and stderr) of executing @emph{cmd} in a shell.

Like @ref{b44,,getstatusoutput()}, except the exit code is ignored and the return
value is a string containing the command’s output.  Example:

@example
>>> subprocess.getoutput('ls /bin/ls')
'/bin/ls'
@end example

@ref{1629,,Availability}: POSIX & Windows.


Changed in version 3.3.4: Windows support added

@end deffn

@node Notes<2>,,Legacy Shell Invocation Functions,subprocess — Subprocess management
@anchor{library/subprocess notes}@anchor{2925}
@subsubsection Notes


@menu
* Converting an argument sequence to a string on Windows:: 

@end menu

@node Converting an argument sequence to a string on Windows,,,Notes<2>
@anchor{library/subprocess converting-an-argument-sequence-to-a-string-on-windows}@anchor{2926}@anchor{library/subprocess converting-argument-sequence}@anchor{28f7}
@subsubsection Converting an argument sequence to a string on Windows


On Windows, an @emph{args} sequence is converted to a string that can be parsed
using the following rules (which correspond to the rules used by the MS C
runtime):


@enumerate 

@item 
Arguments are delimited by white space, which is either a
space or a tab.

@item 
A string surrounded by double quotation marks is
interpreted as a single argument, regardless of white space
contained within.  A quoted string can be embedded in an
argument.

@item 
A double quotation mark preceded by a backslash is
interpreted as a literal double quotation mark.

@item 
Backslashes are interpreted literally, unless they
immediately precede a double quotation mark.

@item 
If backslashes immediately precede a double quotation mark,
every pair of backslashes is interpreted as a literal
backslash.  If the number of backslashes is odd, the last
backslash escapes the next double quotation mark as
described in rule 3.
@end enumerate


@subsubheading See also



@table @asis

@item @ref{e3,,shlex}

Module which provides function to parse and escape command lines.
@end table


@node sched — Event scheduler,queue — A synchronized queue class,subprocess — Subprocess management,Concurrent Execution
@anchor{library/sched doc}@anchor{2927}@anchor{library/sched module-sched}@anchor{de}@anchor{library/sched sched-event-scheduler}@anchor{2928}
@subsection @code{sched} — Event scheduler


@geindex module; sched

@strong{Source code:} Lib/sched.py@footnote{https://github.com/python/cpython/tree/main/Lib/sched.py}

@geindex event scheduling


__________________________________________________________________


The @ref{de,,sched} module defines a class which implements a general purpose event
scheduler:

@geindex scheduler (class in sched)
@anchor{library/sched sched scheduler}@anchor{cba}
@deffn {Class} sched.scheduler (timefunc=time.monotonic, delayfunc=time.sleep)

The @ref{cba,,scheduler} class defines a generic interface to scheduling events.
It needs two functions to actually deal with the “outside world” — @emph{timefunc}
should be callable without arguments, and return  a number (the “time”, in any
units whatsoever).  The @emph{delayfunc} function should be callable with one
argument, compatible with the output of @emph{timefunc}, and should delay that many
time units. @emph{delayfunc} will also be called with the argument @code{0} after each
event is run to allow other threads an opportunity to run in multi-threaded
applications.


Changed in version 3.3: @emph{timefunc} and @emph{delayfunc} parameters are optional.



Changed in version 3.3: @ref{cba,,scheduler} class can be safely used in multi-threaded
environments.

@end deffn

Example:

@example
>>> import sched, time
>>> s = sched.scheduler(time.time, time.sleep)
>>> def print_time(a='default'):
...     print("From print_time", time.time(), a)
...
>>> def print_some_times():
...     print(time.time())
...     s.enter(10, 1, print_time)
...     s.enter(5, 2, print_time, argument=('positional',))
...     s.enter(5, 1, print_time, kwargs=@{'a': 'keyword'@})
...     s.run()
...     print(time.time())
...
>>> print_some_times()
930343690.257
From print_time 930343695.274 positional
From print_time 930343695.275 keyword
From print_time 930343700.273 default
930343700.276
@end example

@menu
* Scheduler Objects:: 

@end menu

@node Scheduler Objects,,,sched — Event scheduler
@anchor{library/sched id1}@anchor{2929}@anchor{library/sched scheduler-objects}@anchor{292a}
@subsubsection Scheduler Objects


@ref{cba,,scheduler} instances have the following methods and attributes:

@geindex enterabs() (sched.scheduler method)
@anchor{library/sched sched scheduler enterabs}@anchor{cbc}
@deffn {Method} scheduler.enterabs (time, priority, action, argument=(), kwargs=@{@})

Schedule a new event. The @emph{time} argument should be a numeric type compatible
with the return value of the @emph{timefunc} function passed  to the constructor.
Events scheduled for the same @emph{time} will be executed in the order of their
@emph{priority}. A lower number represents a higher priority.

Executing the event means executing @code{action(*argument, **kwargs)}.
@emph{argument} is a sequence holding the positional arguments for @emph{action}.
@emph{kwargs} is a dictionary holding the keyword arguments for @emph{action}.

Return value is an event which may be used for later cancellation of the event
(see @ref{11aa,,cancel()}).


Changed in version 3.3: @emph{argument} parameter is optional.



Changed in version 3.3: @emph{kwargs} parameter was added.

@end deffn

@geindex enter() (sched.scheduler method)
@anchor{library/sched sched scheduler enter}@anchor{cbb}
@deffn {Method} scheduler.enter (delay, priority, action, argument=(), kwargs=@{@})

Schedule an event for @emph{delay} more time units. Other than the relative time, the
other arguments, the effect and the return value are the same as those for
@ref{cbc,,enterabs()}.


Changed in version 3.3: @emph{argument} parameter is optional.



Changed in version 3.3: @emph{kwargs} parameter was added.

@end deffn

@geindex cancel() (sched.scheduler method)
@anchor{library/sched sched scheduler cancel}@anchor{11aa}
@deffn {Method} scheduler.cancel (event)

Remove the event from the queue. If @emph{event} is not an event currently in the
queue, this method will raise a @ref{17c,,ValueError}.
@end deffn

@geindex empty() (sched.scheduler method)
@anchor{library/sched sched scheduler empty}@anchor{292b}
@deffn {Method} scheduler.empty ()

Return @code{True} if the event queue is empty.
@end deffn

@geindex run() (sched.scheduler method)
@anchor{library/sched sched scheduler run}@anchor{cb9}
@deffn {Method} scheduler.run (blocking=True)

Run all scheduled events. This method will wait  (using the @code{delayfunc()}
function passed to the constructor) for the next event, then execute it and so
on until there are no more scheduled events.

If @emph{blocking} is false executes the scheduled events due to expire soonest
(if any) and then return the deadline of the next scheduled call in the
scheduler (if any).

Either @emph{action} or @emph{delayfunc} can raise an exception.  In either case, the
scheduler will maintain a consistent state and propagate the exception.  If an
exception is raised by @emph{action}, the event will not be attempted in future calls
to @ref{cb9,,run()}.

If a sequence of events takes longer to run than the time available before the
next event, the scheduler will simply fall behind.  No events will be dropped;
the calling code is responsible for canceling  events which are no longer
pertinent.


Changed in version 3.3: @emph{blocking} parameter was added.

@end deffn

@geindex queue (sched.scheduler attribute)
@anchor{library/sched sched scheduler queue}@anchor{292c}
@deffn {Attribute} scheduler.queue

Read-only attribute returning a list of upcoming events in the order they
will be run.  Each event is shown as a @ref{ce9,,named tuple} with the
following fields:  time, priority, action, argument, kwargs.
@end deffn

@node queue — A synchronized queue class,contextvars — Context Variables,sched — Event scheduler,Concurrent Execution
@anchor{library/queue doc}@anchor{292d}@anchor{library/queue module-queue}@anchor{d5}@anchor{library/queue queue-a-synchronized-queue-class}@anchor{292e}
@subsection @code{queue} — A synchronized queue class


@geindex module; queue

@strong{Source code:} Lib/queue.py@footnote{https://github.com/python/cpython/tree/main/Lib/queue.py}


__________________________________________________________________


The @ref{d5,,queue} module implements multi-producer, multi-consumer queues.
It is especially useful in threaded programming when information must be
exchanged safely between multiple threads.  The @ref{f18,,Queue} class in this
module implements all the required locking semantics.

The module implements three types of queue, which differ only in the order in
which the entries are retrieved.  In a @abbr{FIFO}
queue, the first tasks added are the first retrieved. In a
@abbr{LIFO} queue, the most recently added entry is
the first retrieved (operating like a stack).  With a priority queue,
the entries are kept sorted (using the @ref{89,,heapq} module) and the
lowest valued entry is retrieved first.

Internally, those three types of queues use locks to temporarily block
competing threads; however, they are not designed to handle reentrancy
within a thread.

In addition, the module implements a “simple”
@abbr{FIFO} queue type, @ref{695,,SimpleQueue}, whose
specific implementation provides additional guarantees
in exchange for the smaller functionality.

The @ref{d5,,queue} module defines the following classes and exceptions:

@geindex Queue (class in queue)
@anchor{library/queue queue Queue}@anchor{f18}
@deffn {Class} queue.Queue (maxsize=0)

Constructor for a @abbr{FIFO} queue.  @emph{maxsize} is
an integer that sets the upperbound
limit on the number of items that can be placed in the queue.  Insertion will
block once this size has been reached, until queue items are consumed.  If
@emph{maxsize} is less than or equal to zero, the queue size is infinite.
@end deffn

@geindex LifoQueue (class in queue)
@anchor{library/queue queue LifoQueue}@anchor{1b46}
@deffn {Class} queue.LifoQueue (maxsize=0)

Constructor for a @abbr{LIFO} queue.  @emph{maxsize} is
an integer that sets the upperbound
limit on the number of items that can be placed in the queue.  Insertion will
block once this size has been reached, until queue items are consumed.  If
@emph{maxsize} is less than or equal to zero, the queue size is infinite.
@end deffn

@geindex PriorityQueue (class in queue)
@anchor{library/queue queue PriorityQueue}@anchor{1b47}
@deffn {Class} queue.PriorityQueue (maxsize=0)

Constructor for a priority queue.  @emph{maxsize} is an integer that sets the upperbound
limit on the number of items that can be placed in the queue.  Insertion will
block once this size has been reached, until queue items are consumed.  If
@emph{maxsize} is less than or equal to zero, the queue size is infinite.

The lowest valued entries are retrieved first (the lowest valued entry is the
one returned by @code{sorted(list(entries))[0]}).  A typical pattern for entries
is a tuple in the form: @code{(priority_number, data)}.

If the @emph{data} elements are not comparable, the data can be wrapped in a class
that ignores the data item and only compares the priority number:

@example
from dataclasses import dataclass, field
from typing import Any

@@dataclass(order=True)
class PrioritizedItem:
    priority: int
    item: Any=field(compare=False)
@end example
@end deffn

@geindex SimpleQueue (class in queue)
@anchor{library/queue queue SimpleQueue}@anchor{695}
@deffn {Class} queue.SimpleQueue

Constructor for an unbounded @abbr{FIFO} queue.
Simple queues lack advanced functionality such as task tracking.


New in version 3.7.

@end deffn

@geindex Empty
@anchor{library/queue queue Empty}@anchor{2841}
@deffn {Exception} queue.Empty

Exception raised when non-blocking @ref{292f,,get()} (or
@ref{2930,,get_nowait()}) is called
on a @ref{f18,,Queue} object which is empty.
@end deffn

@geindex Full
@anchor{library/queue queue Full}@anchor{2842}
@deffn {Exception} queue.Full

Exception raised when non-blocking @ref{2931,,put()} (or
@ref{2932,,put_nowait()}) is called
on a @ref{f18,,Queue} object which is full.
@end deffn

@menu
* Queue Objects:: 
* SimpleQueue Objects:: 

@end menu

@node Queue Objects,SimpleQueue Objects,,queue — A synchronized queue class
@anchor{library/queue queue-objects}@anchor{2933}@anchor{library/queue queueobjects}@anchor{2934}
@subsubsection Queue Objects


Queue objects (@ref{f18,,Queue}, @ref{1b46,,LifoQueue}, or @ref{1b47,,PriorityQueue})
provide the public methods described below.

@geindex qsize() (queue.Queue method)
@anchor{library/queue queue Queue qsize}@anchor{2935}
@deffn {Method} Queue.qsize ()

Return the approximate size of the queue.  Note, qsize() > 0 doesn’t
guarantee that a subsequent get() will not block, nor will qsize() < maxsize
guarantee that put() will not block.
@end deffn

@geindex empty() (queue.Queue method)
@anchor{library/queue queue Queue empty}@anchor{2936}
@deffn {Method} Queue.empty ()

Return @code{True} if the queue is empty, @code{False} otherwise.  If empty()
returns @code{True} it doesn’t guarantee that a subsequent call to put()
will not block.  Similarly, if empty() returns @code{False} it doesn’t
guarantee that a subsequent call to get() will not block.
@end deffn

@geindex full() (queue.Queue method)
@anchor{library/queue queue Queue full}@anchor{2937}
@deffn {Method} Queue.full ()

Return @code{True} if the queue is full, @code{False} otherwise.  If full()
returns @code{True} it doesn’t guarantee that a subsequent call to get()
will not block.  Similarly, if full() returns @code{False} it doesn’t
guarantee that a subsequent call to put() will not block.
@end deffn

@geindex put() (queue.Queue method)
@anchor{library/queue queue Queue put}@anchor{2931}
@deffn {Method} Queue.put (item, block=True, timeout=None)

Put @emph{item} into the queue. If optional args @emph{block} is true and @emph{timeout} is
@code{None} (the default), block if necessary until a free slot is available. If
@emph{timeout} is a positive number, it blocks at most @emph{timeout} seconds and raises
the @ref{2842,,Full} exception if no free slot was available within that time.
Otherwise (@emph{block} is false), put an item on the queue if a free slot is
immediately available, else raise the @ref{2842,,Full} exception (@emph{timeout} is
ignored in that case).
@end deffn

@geindex put_nowait() (queue.Queue method)
@anchor{library/queue queue Queue put_nowait}@anchor{2932}
@deffn {Method} Queue.put_nowait (item)

Equivalent to @code{put(item, False)}.
@end deffn

@geindex get() (queue.Queue method)
@anchor{library/queue queue Queue get}@anchor{292f}
@deffn {Method} Queue.get (block=True, timeout=None)

Remove and return an item from the queue. If optional args @emph{block} is true and
@emph{timeout} is @code{None} (the default), block if necessary until an item is available.
If @emph{timeout} is a positive number, it blocks at most @emph{timeout} seconds and
raises the @ref{2841,,Empty} exception if no item was available within that time.
Otherwise (@emph{block} is false), return an item if one is immediately available,
else raise the @ref{2841,,Empty} exception (@emph{timeout} is ignored in that case).

Prior to 3.0 on POSIX systems, and for all versions on Windows, if
@emph{block} is true and @emph{timeout} is @code{None}, this operation goes into
an uninterruptible wait on an underlying lock. This means that no exceptions
can occur, and in particular a SIGINT will not trigger a @ref{4ca,,KeyboardInterrupt}.
@end deffn

@geindex get_nowait() (queue.Queue method)
@anchor{library/queue queue Queue get_nowait}@anchor{2930}
@deffn {Method} Queue.get_nowait ()

Equivalent to @code{get(False)}.
@end deffn

Two methods are offered to support tracking whether enqueued tasks have been
fully processed by daemon consumer threads.

@geindex task_done() (queue.Queue method)
@anchor{library/queue queue Queue task_done}@anchor{283d}
@deffn {Method} Queue.task_done ()

Indicate that a formerly enqueued task is complete.  Used by queue consumer
threads.  For each @ref{292f,,get()} used to fetch a task, a subsequent call to
@ref{283d,,task_done()} tells the queue that the processing on the task is complete.

If a @ref{283e,,join()} is currently blocking, it will resume when all items have been
processed (meaning that a @ref{283d,,task_done()} call was received for every item
that had been @ref{2931,,put()} into the queue).

Raises a @ref{17c,,ValueError} if called more times than there were items placed in
the queue.
@end deffn

@geindex join() (queue.Queue method)
@anchor{library/queue queue Queue join}@anchor{283e}
@deffn {Method} Queue.join ()

Blocks until all items in the queue have been gotten and processed.

The count of unfinished tasks goes up whenever an item is added to the queue.
The count goes down whenever a consumer thread calls @ref{283d,,task_done()} to
indicate that the item was retrieved and all work on it is complete. When the
count of unfinished tasks drops to zero, @ref{283e,,join()} unblocks.
@end deffn

Example of how to wait for enqueued tasks to be completed:

@example
import threading, queue

q = queue.Queue()

def worker():
    while True:
        item = q.get()
        print(f'Working on @{item@}')
        print(f'Finished @{item@}')
        q.task_done()

# turn-on the worker thread
threading.Thread(target=worker, daemon=True).start()

# send thirty task requests to the worker
for item in range(30):
    q.put(item)
print('All task requests sent\n', end='@w{'})

# block until all tasks are done
q.join()
print('All work completed')
@end example

@node SimpleQueue Objects,,Queue Objects,queue — A synchronized queue class
@anchor{library/queue simplequeue-objects}@anchor{2938}
@subsubsection SimpleQueue Objects


@ref{695,,SimpleQueue} objects provide the public methods described below.

@geindex qsize() (queue.SimpleQueue method)
@anchor{library/queue queue SimpleQueue qsize}@anchor{2939}
@deffn {Method} SimpleQueue.qsize ()

Return the approximate size of the queue.  Note, qsize() > 0 doesn’t
guarantee that a subsequent get() will not block.
@end deffn

@geindex empty() (queue.SimpleQueue method)
@anchor{library/queue queue SimpleQueue empty}@anchor{293a}
@deffn {Method} SimpleQueue.empty ()

Return @code{True} if the queue is empty, @code{False} otherwise. If empty()
returns @code{False} it doesn’t guarantee that a subsequent call to get()
will not block.
@end deffn

@geindex put() (queue.SimpleQueue method)
@anchor{library/queue queue SimpleQueue put}@anchor{293b}
@deffn {Method} SimpleQueue.put (item, block=True, timeout=None)

Put @emph{item} into the queue.  The method never blocks and always succeeds
(except for potential low-level errors such as failure to allocate memory).
The optional args @emph{block} and @emph{timeout} are ignored and only provided
for compatibility with @ref{2931,,Queue.put()}.

@strong{CPython implementation detail:} This method has a C implementation which is reentrant.  That is, a
@code{put()} or @code{get()} call can be interrupted by another @code{put()}
call in the same thread without deadlocking or corrupting internal
state inside the queue.  This makes it appropriate for use in
destructors such as @code{__del__} methods or @ref{12f,,weakref} callbacks.
@end deffn

@geindex put_nowait() (queue.SimpleQueue method)
@anchor{library/queue queue SimpleQueue put_nowait}@anchor{293c}
@deffn {Method} SimpleQueue.put_nowait (item)

Equivalent to @code{put(item)}, provided for compatibility with
@ref{2932,,Queue.put_nowait()}.
@end deffn

@geindex get() (queue.SimpleQueue method)
@anchor{library/queue queue SimpleQueue get}@anchor{293d}
@deffn {Method} SimpleQueue.get (block=True, timeout=None)

Remove and return an item from the queue.  If optional args @emph{block} is true and
@emph{timeout} is @code{None} (the default), block if necessary until an item is available.
If @emph{timeout} is a positive number, it blocks at most @emph{timeout} seconds and
raises the @ref{2841,,Empty} exception if no item was available within that time.
Otherwise (@emph{block} is false), return an item if one is immediately available,
else raise the @ref{2841,,Empty} exception (@emph{timeout} is ignored in that case).
@end deffn

@geindex get_nowait() (queue.SimpleQueue method)
@anchor{library/queue queue SimpleQueue get_nowait}@anchor{293e}
@deffn {Method} SimpleQueue.get_nowait ()

Equivalent to @code{get(False)}.
@end deffn


@subsubheading See also



@table @asis

@item Class @ref{281e,,multiprocessing.Queue}

A queue class for use in a multi-processing (rather than multi-threading)
context.
@end table

@ref{7d3,,collections.deque} is an alternative implementation of unbounded
queues with fast atomic @ref{1dcc,,append()} and
@ref{1dd2,,popleft()} operations that do not require locking
and also support indexing.


@node contextvars — Context Variables,_thread — Low-level threading API,queue — A synchronized queue class,Concurrent Execution
@anchor{library/contextvars doc}@anchor{293f}@anchor{library/contextvars contextvars-context-variables}@anchor{2940}@anchor{library/contextvars module-contextvars}@anchor{23}
@subsection @code{contextvars} — Context Variables


@geindex module; contextvars


__________________________________________________________________


This module provides APIs to manage, store, and access context-local
state.  The @ref{2941,,ContextVar} class is used to declare
and work with @emph{Context Variables}.  The @ref{2942,,copy_context()}
function and the @ref{2943,,Context} class should be used to
manage the current context in asynchronous frameworks.

Context managers that have state should use Context Variables
instead of @ref{278d,,threading.local()} to prevent their state from
bleeding to other code unexpectedly, when used in concurrent code.

See also 
@geindex Python Enhancement Proposals; PEP 567
PEP 567@footnote{https://www.python.org/dev/peps/pep-0567} for additional details.


New in version 3.7.


@menu
* Context Variables:: 
* Manual Context Management:: 
* asyncio support:: 

@end menu

@node Context Variables,Manual Context Management,,contextvars — Context Variables
@anchor{library/contextvars context-variables}@anchor{2944}
@subsubsection Context Variables


@geindex ContextVar (class in contextvars)
@anchor{library/contextvars contextvars ContextVar}@anchor{2941}
@deffn {Class} contextvars.ContextVar (name[, *, default])

This class is used to declare a new Context Variable, e.g.:

@example
var: ContextVar[int] = ContextVar('var', default=42)
@end example

The required @emph{name} parameter is used for introspection and debug
purposes.

The optional keyword-only @emph{default} parameter is returned by
@ref{2945,,ContextVar.get()} when no value for the variable is found
in the current context.

@strong{Important:} Context Variables should be created at the top module
level and never in closures.  @ref{2943,,Context} objects hold strong
references to context variables which prevents context variables
from being properly garbage collected.

@geindex name (contextvars.ContextVar attribute)
@anchor{library/contextvars contextvars ContextVar name}@anchor{2946}
@deffn {Attribute} name

The name of the variable.  This is a read-only property.


New in version 3.7.1.

@end deffn

@geindex get() (contextvars.ContextVar method)
@anchor{library/contextvars contextvars ContextVar get}@anchor{2945}
@deffn {Method} get ([default])

Return a value for the context variable for the current context.

If there is no value for the variable in the current context,
the method will:


@itemize *

@item 
return the value of the @emph{default} argument of the method,
if provided; or

@item 
return the default value for the context variable,
if it was created with one; or

@item 
raise a @ref{1ab5,,LookupError}.
@end itemize
@end deffn

@geindex set() (contextvars.ContextVar method)
@anchor{library/contextvars contextvars ContextVar set}@anchor{2947}
@deffn {Method} set (value)

Call to set a new value for the context variable in the current
context.

The required @emph{value} argument is the new value for the context
variable.

Returns a @ref{2948,,Token} object that can be used
to restore the variable to its previous value via the
@ref{2949,,ContextVar.reset()} method.
@end deffn

@geindex reset() (contextvars.ContextVar method)
@anchor{library/contextvars contextvars ContextVar reset}@anchor{2949}
@deffn {Method} reset (token)

Reset the context variable to the value it had before the
@ref{2947,,ContextVar.set()} that created the @emph{token} was used.

For example:

@example
var = ContextVar('var')

token = var.set('new value')
# code that uses 'var'; var.get() returns 'new value'.
var.reset(token)

# After the reset call the var has no value again, so
# var.get() would raise a LookupError.
@end example
@end deffn
@end deffn

@geindex Token (class in contextvars)
@anchor{library/contextvars contextvars Token}@anchor{2948}
@deffn {Class} contextvars.Token

@emph{Token} objects are returned by the @ref{2947,,ContextVar.set()} method.
They can be passed to the @ref{2949,,ContextVar.reset()} method to revert
the value of the variable to what it was before the corresponding
@emph{set}.

@geindex var (contextvars.Token attribute)
@anchor{library/contextvars contextvars Token var}@anchor{294a}
@deffn {Attribute} var

A read-only property.  Points to the @ref{2941,,ContextVar} object
that created the token.
@end deffn

@geindex old_value (contextvars.Token attribute)
@anchor{library/contextvars contextvars Token old_value}@anchor{294b}
@deffn {Attribute} old_value

A read-only property.  Set to the value the variable had before
the @ref{2947,,ContextVar.set()} method call that created the token.
It points to @ref{294c,,Token.MISSING} is the variable was not set
before the call.
@end deffn

@geindex MISSING (contextvars.Token attribute)
@anchor{library/contextvars contextvars Token MISSING}@anchor{294c}
@deffn {Attribute} MISSING

A marker object used by @ref{294b,,Token.old_value}.
@end deffn
@end deffn

@node Manual Context Management,asyncio support,Context Variables,contextvars — Context Variables
@anchor{library/contextvars manual-context-management}@anchor{294d}
@subsubsection Manual Context Management


@geindex copy_context() (in module contextvars)
@anchor{library/contextvars contextvars copy_context}@anchor{2942}
@deffn {Function} contextvars.copy_context ()

Returns a copy of the current @ref{2943,,Context} object.

The following snippet gets a copy of the current context and prints
all variables and their values that are set in it:

@example
ctx: Context = copy_context()
print(list(ctx.items()))
@end example

The function has an O(1) complexity, i.e. works equally fast for
contexts with a few context variables and for contexts that have
a lot of them.
@end deffn

@geindex Context (class in contextvars)
@anchor{library/contextvars contextvars Context}@anchor{2943}
@deffn {Class} contextvars.Context

A mapping of @ref{2941,,ContextVars} to their values.

@code{Context()} creates an empty context with no values in it.
To get a copy of the current context use the
@ref{2942,,copy_context()} function.

Context implements the @ref{3ab,,collections.abc.Mapping} interface.

@geindex run() (contextvars.Context method)
@anchor{library/contextvars contextvars Context run}@anchor{294e}
@deffn {Method} run (callable, *args, **kwargs)

Execute @code{callable(*args, **kwargs)} code in the context object
the @emph{run} method is called on.  Return the result of the execution
or propagate an exception if one occurred.

Any changes to any context variables that @emph{callable} makes will
be contained in the context object:

@example
var = ContextVar('var')
var.set('spam')

def main():
    # 'var' was set to 'spam' before
    # calling 'copy_context()' and 'ctx.run(main)', so:
    # var.get() == ctx[var] == 'spam'

    var.set('ham')

    # Now, after setting 'var' to 'ham':
    # var.get() == ctx[var] == 'ham'

ctx = copy_context()

# Any changes that the 'main' function makes to 'var'
# will be contained in 'ctx'.
ctx.run(main)

# The 'main()' function was run in the 'ctx' context,
# so changes to 'var' are contained in it:
# ctx[var] == 'ham'

# However, outside of 'ctx', 'var' is still set to 'spam':
# var.get() == 'spam'
@end example

The method raises a @ref{5af,,RuntimeError} when called on the same
context object from more than one OS thread, or when called
recursively.
@end deffn

@geindex copy() (contextvars.Context method)
@anchor{library/contextvars contextvars Context copy}@anchor{294f}
@deffn {Method} copy ()

Return a shallow copy of the context object.
@end deffn


@deffn {Describe} var in context

Return @code{True} if the @emph{context} has a value for @emph{var} set;
return @code{False} otherwise.
@end deffn


@deffn {Describe} context[var]

Return the value of the @emph{var} @ref{2941,,ContextVar} variable.
If the variable is not set in the context object, a
@ref{5bb,,KeyError} is raised.
@end deffn

@geindex get() (contextvars.Context method)
@anchor{library/contextvars contextvars Context get}@anchor{2950}
@deffn {Method} get (var[, default])

Return the value for @emph{var} if @emph{var} has the value in the context
object.  Return @emph{default} otherwise.  If @emph{default} is not given,
return @code{None}.
@end deffn


@deffn {Describe} iter(context)

Return an iterator over the variables stored in the context
object.
@end deffn


@deffn {Describe} len(proxy)

Return the number of variables set in the context object.
@end deffn

@geindex keys() (contextvars.Context method)
@anchor{library/contextvars contextvars Context keys}@anchor{2951}
@deffn {Method} keys ()

Return a list of all variables in the context object.
@end deffn

@geindex values() (contextvars.Context method)
@anchor{library/contextvars contextvars Context values}@anchor{2952}
@deffn {Method} values ()

Return a list of all variables’ values in the context object.
@end deffn

@geindex items() (contextvars.Context method)
@anchor{library/contextvars contextvars Context items}@anchor{2953}
@deffn {Method} items ()

Return a list of 2-tuples containing all variables and their
values in the context object.
@end deffn
@end deffn

@node asyncio support,,Manual Context Management,contextvars — Context Variables
@anchor{library/contextvars asyncio-support}@anchor{2954}
@subsubsection asyncio support


Context variables are natively supported in @ref{9,,asyncio} and are
ready to be used without any extra configuration.  For example, here
is a simple echo server, that uses a context variable to make the
address of a remote client available in the Task that handles that
client:

@example
import asyncio
import contextvars

client_addr_var = contextvars.ContextVar('client_addr')

def render_goodbye():
    # The address of the currently handled client can be accessed
    # without passing it explicitly to this function.

    client_addr = client_addr_var.get()
    return f'Good bye, client @@ @{client_addr@}\n'.encode()

async def handle_request(reader, writer):
    addr = writer.transport.get_extra_info('socket').getpeername()
    client_addr_var.set(addr)

    # In any code that we call is now possible to get
    # client's address by calling 'client_addr_var.get()'.

    while True:
        line = await reader.readline()
        print(line)
        if not line.strip():
            break
        writer.write(line)

    writer.write(render_goodbye())
    writer.close()

async def main():
    srv = await asyncio.start_server(
        handle_request, '127.0.0.1', 8081)

    async with srv:
        await srv.serve_forever()

asyncio.run(main())

# To test it you can use telnet:
#     telnet 127.0.0.1 8081
@end example

The following are support modules for some of the above services:

@node _thread — Low-level threading API,,contextvars — Context Variables,Concurrent Execution
@anchor{library/_thread doc}@anchor{2955}@anchor{library/_thread module-_thread}@anchor{2}@anchor{library/_thread thread-low-level-threading-api}@anchor{2956}
@subsection @code{_thread} — Low-level threading API


@geindex module; _thread

@geindex light-weight processes
@geindex processes@comma{} light-weight
@geindex binary semaphores
@geindex semaphores@comma{} binary


__________________________________________________________________


This module provides low-level primitives for working with multiple threads
(also called @emph{light-weight processes} or @emph{tasks}) — multiple threads of
control sharing their global data space.  For synchronization, simple locks
(also called @emph{mutexes} or @emph{binary semaphores}) are provided.
The @ref{109,,threading} module provides an easier to use and higher-level
threading API built on top of this module.

@geindex pthreads
@geindex threads; POSIX


Changed in version 3.7: This module used to be optional, it is now always available.


This module defines the following constants and functions:

@geindex error
@anchor{library/_thread thread error}@anchor{2957}@anchor{library/_thread _thread error}@anchor{2958}
@deffn {Exception} _thread.error

Raised on thread-specific errors.


Changed in version 3.3: This is now a synonym of the built-in @ref{5af,,RuntimeError}.

@end deffn

@geindex LockType (in module _thread)
@anchor{library/_thread thread LockType}@anchor{2959}@anchor{library/_thread _thread LockType}@anchor{295a}
@deffn {Data} _thread.LockType

This is the type of lock objects.
@end deffn

@geindex start_new_thread() (in module _thread)
@anchor{library/_thread thread start_new_thread}@anchor{1294}@anchor{library/_thread _thread start_new_thread}@anchor{295b}
@deffn {Function} _thread.start_new_thread (function, args[, kwargs])

Start a new thread and return its identifier.  The thread executes the
function @emph{function} with the argument list @emph{args} (which must be a tuple).
The optional @emph{kwargs} argument specifies a dictionary of keyword arguments.

When the function returns, the thread silently exits.

When the function terminates with an unhandled exception,
@ref{18d,,sys.unraisablehook()} is called to handle the exception. The @emph{object}
attribute of the hook argument is @emph{function}. By default, a stack trace is
printed and then the thread exits (but other threads continue to run).

When the function raises a @ref{892,,SystemExit} exception, it is silently
ignored.


Changed in version 3.8: @ref{18d,,sys.unraisablehook()} is now used to handle unhandled exceptions.

@end deffn

@geindex interrupt_main() (in module _thread)
@anchor{library/_thread thread interrupt_main}@anchor{2c9}@anchor{library/_thread _thread interrupt_main}@anchor{295c}
@deffn {Function} _thread.interrupt_main (signum=signal.SIGINT, /)

Simulate the effect of a signal arriving in the main thread.
A thread can use this function to interrupt the main thread, though
there is no guarantee that the interruption will happen immediately.

If given, @emph{signum} is the number of the signal to simulate.
If @emph{signum} is not given, @ref{2ca,,signal.SIGINT} is simulated.

If the given signal isn’t handled by Python (it was set to
@ref{295d,,signal.SIG_DFL} or @ref{295e,,signal.SIG_IGN}), this function does
nothing.


Changed in version 3.10: The @emph{signum} argument is added to customize the signal number.


@cartouche
@quotation Note 
This does not emit the corresponding signal but schedules a call to
the associated handler (if it exists).
If you want to truly emit the signal, use @ref{295f,,signal.raise_signal()}.
@end quotation
@end cartouche
@end deffn

@geindex exit() (in module _thread)
@anchor{library/_thread thread exit}@anchor{2960}@anchor{library/_thread _thread exit}@anchor{2961}
@deffn {Function} _thread.exit ()

Raise the @ref{892,,SystemExit} exception.  When not caught, this will cause the
thread to exit silently.
@end deffn

@c function:: exit_prog(status)
@c 
@c    Exit all threads and report the value of the integer argument
@c    *status* as the exit status of the entire program.
@c    **Caveat:** code in pending :keyword:`finally` clauses, in this thread
@c    or in other threads, is not executed.

@geindex allocate_lock() (in module _thread)
@anchor{library/_thread thread allocate_lock}@anchor{2962}@anchor{library/_thread _thread allocate_lock}@anchor{2963}
@deffn {Function} _thread.allocate_lock ()

Return a new lock object.  Methods of locks are described below.  The lock is
initially unlocked.
@end deffn

@geindex get_ident() (in module _thread)
@anchor{library/_thread thread get_ident}@anchor{2964}@anchor{library/_thread _thread get_ident}@anchor{2965}
@deffn {Function} _thread.get_ident ()

Return the ‘thread identifier’ of the current thread.  This is a nonzero
integer.  Its value has no direct meaning; it is intended as a magic cookie to
be used e.g. to index a dictionary of thread-specific data.  Thread identifiers
may be recycled when a thread exits and another thread is created.
@end deffn

@geindex get_native_id() (in module _thread)
@anchor{library/_thread thread get_native_id}@anchor{2966}@anchor{library/_thread _thread get_native_id}@anchor{2967}
@deffn {Function} _thread.get_native_id ()

Return the native integral Thread ID of the current thread assigned by the kernel.
This is a non-negative integer.
Its value may be used to uniquely identify this particular thread system-wide
(until the thread terminates, after which the value may be recycled by the OS).

@ref{1629,,Availability}: Windows, FreeBSD, Linux, macOS, OpenBSD, NetBSD, AIX.


New in version 3.8.

@end deffn

@geindex stack_size() (in module _thread)
@anchor{library/_thread thread stack_size}@anchor{2968}@anchor{library/_thread _thread stack_size}@anchor{2969}
@deffn {Function} _thread.stack_size ([size])

Return the thread stack size used when creating new threads.  The optional
@emph{size} argument specifies the stack size to be used for subsequently created
threads, and must be 0 (use platform or configured default) or a positive
integer value of at least 32,768 (32 KiB). If @emph{size} is not specified,
0 is used.  If changing the thread stack size is
unsupported, a @ref{5af,,RuntimeError} is raised.  If the specified stack size is
invalid, a @ref{17c,,ValueError} is raised and the stack size is unmodified.  32 KiB
is currently the minimum supported stack size value to guarantee sufficient
stack space for the interpreter itself.  Note that some platforms may have
particular restrictions on values for the stack size, such as requiring a
minimum stack size > 32 KiB or requiring allocation in multiples of the system
memory page size - platform documentation should be referred to for more
information (4 KiB pages are common; using multiples of 4096 for the stack size is
the suggested approach in the absence of more specific information).

@ref{1629,,Availability}: Windows, systems with POSIX threads.
@end deffn

@geindex TIMEOUT_MAX (in module _thread)
@anchor{library/_thread thread TIMEOUT_MAX}@anchor{296a}@anchor{library/_thread _thread TIMEOUT_MAX}@anchor{296b}
@deffn {Data} _thread.TIMEOUT_MAX

The maximum value allowed for the @emph{timeout} parameter of
@code{Lock.acquire()}. Specifying a timeout greater than this value will
raise an @ref{347,,OverflowError}.


New in version 3.2.

@end deffn

Lock objects have the following methods:

@geindex acquire() (_thread.lock method)
@anchor{library/_thread thread lock acquire}@anchor{296c}@anchor{library/_thread _thread lock acquire}@anchor{296d}
@deffn {Method} lock.acquire (waitflag=1, timeout=- 1)

Without any optional argument, this method acquires the lock unconditionally, if
necessary waiting until it is released by another thread (only one thread at a
time can acquire a lock — that’s their reason for existence).

If the integer @emph{waitflag} argument is present, the action depends on its
value: if it is zero, the lock is only acquired if it can be acquired
immediately without waiting, while if it is nonzero, the lock is acquired
unconditionally as above.

If the floating-point @emph{timeout} argument is present and positive, it
specifies the maximum wait time in seconds before returning.  A negative
@emph{timeout} argument specifies an unbounded wait.  You cannot specify
a @emph{timeout} if @emph{waitflag} is zero.

The return value is @code{True} if the lock is acquired successfully,
@code{False} if not.


Changed in version 3.2: The @emph{timeout} parameter is new.



Changed in version 3.2: Lock acquires can now be interrupted by signals on POSIX.

@end deffn

@geindex release() (_thread.lock method)
@anchor{library/_thread thread lock release}@anchor{296e}@anchor{library/_thread _thread lock release}@anchor{296f}
@deffn {Method} lock.release ()

Releases the lock.  The lock must have been acquired earlier, but not
necessarily by the same thread.
@end deffn

@geindex locked() (_thread.lock method)
@anchor{library/_thread thread lock locked}@anchor{2970}@anchor{library/_thread _thread lock locked}@anchor{2971}
@deffn {Method} lock.locked ()

Return the status of the lock: @code{True} if it has been acquired by some thread,
@code{False} if not.
@end deffn

In addition to these methods, lock objects can also be used via the
@ref{168,,with} statement, e.g.:

@example
import _thread

a_lock = _thread.allocate_lock()

with a_lock:
    print("a_lock is locked while this executes")
@end example

@strong{Caveats:}

@quotation

@geindex module; signal
@end quotation


@itemize *

@item 
Threads interact strangely with interrupts: the @ref{4ca,,KeyboardInterrupt}
exception will be received by an arbitrary thread.  (When the @ref{e5,,signal}
module is available, interrupts always go to the main thread.)

@item 
Calling @ref{ee4,,sys.exit()} or raising the @ref{892,,SystemExit} exception is
equivalent to calling @ref{2960,,_thread.exit()}.

@item 
It is not possible to interrupt the @code{acquire()} method on a lock — the
@ref{4ca,,KeyboardInterrupt} exception will happen after the lock has been acquired.

@item 
When the main thread exits, it is system defined whether the other threads
survive.  On most systems, they are killed without executing
@ref{f6d,,try} … @ref{4c1,,finally} clauses or executing object
destructors.

@item 
When the main thread exits, it does not do any of its usual cleanup (except
that @ref{f6d,,try} … @ref{4c1,,finally} clauses are honored), and the
standard I/O files are not flushed.
@end itemize

@node Networking and Interprocess Communication,Internet Data Handling,Concurrent Execution,The Python Standard Library
@anchor{library/ipc doc}@anchor{2972}@anchor{library/ipc ipc}@anchor{2973}@anchor{library/ipc networking-and-interprocess-communication}@anchor{2974}
@section Networking and Interprocess Communication


The modules described in this chapter provide mechanisms for
networking and inter-processes communication.

Some modules only work for two processes that are on the same machine, e.g.
@ref{e5,,signal} and @ref{af,,mmap}.  Other modules support networking protocols
that two or more processes can use to communicate across machines.

The list of modules described in this chapter is:

@menu
* asyncio — Asynchronous I/O:: 
* socket — Low-level networking interface:: 
* ssl — TLS/SSL wrapper for socket objects:: 
* select — Waiting for I/O completion:: 
* selectors — High-level I/O multiplexing:: 
* asyncore — Asynchronous socket handler:: 
* asynchat — Asynchronous socket command/response handler:: 
* signal — Set handlers for asynchronous events:: 
* mmap — Memory-mapped file support:: 

@end menu

@node asyncio — Asynchronous I/O,socket — Low-level networking interface,,Networking and Interprocess Communication
@anchor{library/asyncio doc}@anchor{2975}@anchor{library/asyncio asyncio-asynchronous-i-o}@anchor{2976}@anchor{library/asyncio module-asyncio}@anchor{9}
@subsection @code{asyncio} — Asynchronous I/O


@geindex module; asyncio


__________________________________________________________________


@subsubheading Hello World!



@example
import asyncio

async def main():
    print('Hello ...')
    await asyncio.sleep(1)
    print('... World!')

# Python 3.7+
asyncio.run(main())
@end example

asyncio is a library to write @strong{concurrent} code using
the @strong{async/await} syntax.

asyncio is used as a foundation for multiple Python asynchronous
frameworks that provide high-performance network and web-servers,
database connection libraries, distributed task queues, etc.

asyncio is often a perfect fit for IO-bound and high-level
@strong{structured} network code.

asyncio provides a set of @strong{high-level} APIs to:


@itemize *

@item 
@ref{2977,,run Python coroutines} concurrently and
have full control over their execution;

@item 
perform @ref{2978,,network IO and IPC};

@item 
control @ref{2979,,subprocesses};

@item 
distribute tasks via @ref{297a,,queues};

@item 
@ref{297b,,synchronize} concurrent code;
@end itemize

Additionally, there are @strong{low-level} APIs for
@emph{library and framework developers} to:


@itemize *

@item 
create and manage @ref{8d1,,event loops}, which
provide asynchronous APIs for @ref{630,,networking},
running @ref{297c,,subprocesses},
handling @ref{297d,,OS signals}, etc;

@item 
implement efficient protocols using
@ref{297e,,transports};

@item 
@ref{297f,,bridge} callback-based libraries and code
with async/await syntax.
@end itemize

@c We use the "rubric" directive here to avoid creating
@c the "Reference" subsection in the TOC.

@subsubheading Reference


@menu
* Coroutines and Tasks:: 
* Streams:: 
* Synchronization Primitives:: 
* Subprocesses:: 
* Queues:: 
* Exceptions: Exceptions<10>. 
* Event Loop:: 
* Futures:: 
* Transports and Protocols:: 
* Policies:: 
* Platform Support:: 
* High-level API Index:: 
* Low-level API Index:: 
* Developing with asyncio:: 

@end menu

@node Coroutines and Tasks,Streams,,asyncio — Asynchronous I/O
@anchor{library/asyncio-task doc}@anchor{2980}@anchor{library/asyncio-task coroutines-and-tasks}@anchor{2981}
@subsubsection Coroutines and Tasks


This section outlines high-level asyncio APIs to work with coroutines
and Tasks.

@menu
* Coroutines: Coroutines<3>. 
* Awaitables:: 
* Running an asyncio Program:: 
* Creating Tasks:: 
* Sleeping:: 
* Running Tasks Concurrently:: 
* Shielding From Cancellation:: 
* Timeouts:: 
* Waiting Primitives:: 
* Running in Threads:: 
* Scheduling From Other Threads:: 
* Introspection:: 
* Task Object:: 

@end menu

@node Coroutines<3>,Awaitables,,Coroutines and Tasks
@anchor{library/asyncio-task coroutine}@anchor{2977}@anchor{library/asyncio-task coroutines}@anchor{2982}
@subsubsection Coroutines


@ref{3cf,,Coroutines} declared with the async/await syntax is the
preferred way of writing asyncio applications.  For example, the following
snippet of code (requires Python 3.7+) prints “hello”, waits 1 second,
and then prints “world”:

@example
>>> import asyncio

>>> async def main():
...     print('hello')
...     await asyncio.sleep(1)
...     print('world')

>>> asyncio.run(main())
hello
world
@end example

Note that simply calling a coroutine will not schedule it to
be executed:

@example
>>> main()
<coroutine object main at 0x1053bb7c8>
@end example

To actually run a coroutine, asyncio provides three main mechanisms:


@itemize *

@item 
The @ref{3d1,,asyncio.run()} function to run the top-level
entry point “main()” function (see the above example.)

@item 
Awaiting on a coroutine.  The following snippet of code will
print “hello” after waiting for 1 second, and then print “world”
after waiting for @emph{another} 2 seconds:

@example
import asyncio
import time

async def say_after(delay, what):
    await asyncio.sleep(delay)
    print(what)

async def main():
    print(f"started at @{time.strftime('%X')@}")

    await say_after(1, 'hello')
    await say_after(2, 'world')

    print(f"finished at @{time.strftime('%X')@}")

asyncio.run(main())
@end example

Expected output:

@example
started at 17:13:52
hello
world
finished at 17:13:55
@end example

@item 
The @ref{4dc,,asyncio.create_task()} function to run coroutines
concurrently as asyncio @ref{317,,Tasks}.

Let’s modify the above example and run two @code{say_after} coroutines
@emph{concurrently}:

@example
async def main():
    task1 = asyncio.create_task(
        say_after(1, 'hello'))

    task2 = asyncio.create_task(
        say_after(2, 'world'))

    print(f"started at @{time.strftime('%X')@}")

    # Wait until both tasks are completed (should take
    # around 2 seconds.)
    await task1
    await task2

    print(f"finished at @{time.strftime('%X')@}")
@end example

Note that expected output now shows that the snippet runs
1 second faster than before:

@example
started at 17:14:32
hello
world
finished at 17:14:34
@end example
@end itemize

@node Awaitables,Running an asyncio Program,Coroutines<3>,Coroutines and Tasks
@anchor{library/asyncio-task asyncio-awaitables}@anchor{2983}@anchor{library/asyncio-task awaitables}@anchor{2984}
@subsubsection Awaitables


We say that an object is an @strong{awaitable} object if it can be used
in an @ref{4d5,,await} expression.  Many asyncio APIs are designed to
accept awaitables.

There are three main types of @emph{awaitable} objects:
@strong{coroutines}, @strong{Tasks}, and @strong{Futures}.

@subsubheading Coroutines


Python coroutines are @emph{awaitables} and therefore can be awaited from
other coroutines:

@example
import asyncio

async def nested():
    return 42

async def main():
    # Nothing happens if we just call "nested()".
    # A coroutine object is created but not awaited,
    # so it *won't run at all*.
    nested()

    # Let's do it differently now and await it:
    print(await nested())  # will print "42".

asyncio.run(main())
@end example

@cartouche
@quotation Important 
In this documentation the term “coroutine” can be used for
two closely related concepts:


@itemize *

@item 
a @emph{coroutine function}: an @ref{1c0,,async def} function;

@item 
a @emph{coroutine object}: an object returned by calling a
@emph{coroutine function}.
@end itemize
@end quotation
@end cartouche

@subsubheading Tasks


@emph{Tasks} are used to schedule coroutines @emph{concurrently}.

When a coroutine is wrapped into a @emph{Task} with functions like
@ref{4dc,,asyncio.create_task()} the coroutine is automatically
scheduled to run soon:

@example
import asyncio

async def nested():
    return 42

async def main():
    # Schedule nested() to run soon concurrently
    # with "main()".
    task = asyncio.create_task(nested())

    # "task" can now be used to cancel "nested()", or
    # can simply be awaited to wait until it is complete:
    await task

asyncio.run(main())
@end example

@subsubheading Futures


A @ref{316,,Future} is a special @strong{low-level} awaitable object that
represents an @strong{eventual result} of an asynchronous operation.

When a Future object is @emph{awaited} it means that the coroutine will
wait until the Future is resolved in some other place.

Future objects in asyncio are needed to allow callback-based code
to be used with async/await.

Normally @strong{there is no need} to create Future objects at the
application level code.

Future objects, sometimes exposed by libraries and some asyncio
APIs, can be awaited:

@example
async def main():
    await function_that_returns_a_future_object()

    # this is also valid:
    await asyncio.gather(
        function_that_returns_a_future_object(),
        some_python_coroutine()
    )
@end example

A good example of a low-level function that returns a Future object
is @ref{3d4,,loop.run_in_executor()}.

@node Running an asyncio Program,Creating Tasks,Awaitables,Coroutines and Tasks
@anchor{library/asyncio-task running-an-asyncio-program}@anchor{2985}
@subsubsection Running an asyncio Program


@geindex run() (in module asyncio)
@anchor{library/asyncio-task asyncio run}@anchor{3d1}
@deffn {Function} asyncio.run (coro, *, debug=False)

Execute the @ref{3cf,,coroutine} @emph{coro} and return the result.

This function runs the passed coroutine, taking care of
managing the asyncio event loop, @emph{finalizing asynchronous
generators}, and closing the threadpool.

This function cannot be called when another asyncio event loop is
running in the same thread.

If @emph{debug} is @code{True}, the event loop will be run in debug mode.

This function always creates a new event loop and closes it at
the end.  It should be used as a main entry point for asyncio
programs, and should ideally only be called once.

Example:

@example
async def main():
    await asyncio.sleep(1)
    print('hello')

asyncio.run(main())
@end example


New in version 3.7.



Changed in version 3.9: Updated to use @ref{3d0,,loop.shutdown_default_executor()}.


@cartouche
@quotation Note 
The source code for @code{asyncio.run()} can be found in
Lib/asyncio/runners.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/runners.py}.
@end quotation
@end cartouche
@end deffn

@node Creating Tasks,Sleeping,Running an asyncio Program,Coroutines and Tasks
@anchor{library/asyncio-task creating-tasks}@anchor{2986}
@subsubsection Creating Tasks


@geindex create_task() (in module asyncio)
@anchor{library/asyncio-task asyncio create_task}@anchor{4dc}
@deffn {Function} asyncio.create_task (coro, *, name=None)

Wrap the @emph{coro} @ref{2977,,coroutine} into a @ref{317,,Task}
and schedule its execution.  Return the Task object.

If @emph{name} is not @code{None}, it is set as the name of the task using
@ref{4de,,Task.set_name()}.

The task is executed in the loop returned by @ref{315,,get_running_loop()},
@ref{5af,,RuntimeError} is raised if there is no running loop in
current thread.

This function has been @strong{added in Python 3.7}.  Prior to
Python 3.7, the low-level @ref{318,,asyncio.ensure_future()} function
can be used instead:

@example
async def coro():
    ...

# In Python 3.7+
task = asyncio.create_task(coro())
...

# This works in all Python versions but is less readable
task = asyncio.ensure_future(coro())
...
@end example

@cartouche
@quotation Important 
Save a reference to the result of this function, to avoid
a task disappearing mid execution.
@end quotation
@end cartouche


New in version 3.7.



Changed in version 3.8: Added the @emph{name} parameter.

@end deffn

@node Sleeping,Running Tasks Concurrently,Creating Tasks,Coroutines and Tasks
@anchor{library/asyncio-task sleeping}@anchor{2987}
@subsubsection Sleeping


@geindex sleep() (in module asyncio)
@anchor{library/asyncio-task asyncio sleep}@anchor{58a}
@deffn {Function} coroutine asyncio.sleep (delay, result=None)

Block for @emph{delay} seconds.

If @emph{result} is provided, it is returned to the caller
when the coroutine completes.

@code{sleep()} always suspends the current task, allowing other tasks
to run.

Setting the delay to 0 provides an optimized path to allow other
tasks to run. This can be used by long-running functions to avoid
blocking the event loop for the full duration of the function call.
@anchor{library/asyncio-task asyncio-example-sleep}@anchor{2988}
Example of coroutine displaying the current date every second
for 5 seconds:

@example
import asyncio
import datetime

async def display_date():
    loop = asyncio.get_running_loop()
    end_time = loop.time() + 5.0
    while True:
        print(datetime.datetime.now())
        if (loop.time() + 1.0) >= end_time:
            break
        await asyncio.sleep(1)

asyncio.run(display_date())
@end example


Changed in version 3.10: Removed the @emph{loop} parameter.

@end deffn

@node Running Tasks Concurrently,Shielding From Cancellation,Sleeping,Coroutines and Tasks
@anchor{library/asyncio-task running-tasks-concurrently}@anchor{2989}
@subsubsection Running Tasks Concurrently


@geindex gather() (in module asyncio)
@anchor{library/asyncio-task asyncio gather}@anchor{31a}
@deffn {Function} awaitable asyncio.gather (*aws, return_exceptions=False)

Run @ref{2983,,awaitable objects} in the @emph{aws}
sequence @emph{concurrently}.

If any awaitable in @emph{aws} is a coroutine, it is automatically
scheduled as a Task.

If all awaitables are completed successfully, the result is an
aggregate list of returned values.  The order of result values
corresponds to the order of awaitables in @emph{aws}.

If @emph{return_exceptions} is @code{False} (default), the first
raised exception is immediately propagated to the task that
awaits on @code{gather()}.  Other awaitables in the @emph{aws} sequence
@strong{won’t be cancelled} and will continue to run.

If @emph{return_exceptions} is @code{True}, exceptions are treated the
same as successful results, and aggregated in the result list.

If @code{gather()} is @emph{cancelled}, all submitted awaitables
(that have not completed yet) are also @emph{cancelled}.

If any Task or Future from the @emph{aws} sequence is @emph{cancelled}, it is
treated as if it raised @ref{4d7,,CancelledError} – the @code{gather()}
call is @strong{not} cancelled in this case.  This is to prevent the
cancellation of one submitted Task/Future to cause other
Tasks/Futures to be cancelled.


Changed in version 3.10: Removed the @emph{loop} parameter.

@anchor{library/asyncio-task asyncio-example-gather}@anchor{298a}
Example:

@example
import asyncio

async def factorial(name, number):
    f = 1
    for i in range(2, number + 1):
        print(f"Task @{name@}: Compute factorial(@{number@}), currently i=@{i@}...")
        await asyncio.sleep(1)
        f *= i
    print(f"Task @{name@}: factorial(@{number@}) = @{f@}")
    return f

async def main():
    # Schedule three calls *concurrently*:
    L = await asyncio.gather(
        factorial("A", 2),
        factorial("B", 3),
        factorial("C", 4),
    )
    print(L)

asyncio.run(main())

# Expected output:
#
#     Task A: Compute factorial(2), currently i=2...
#     Task B: Compute factorial(3), currently i=2...
#     Task C: Compute factorial(4), currently i=2...
#     Task A: factorial(2) = 2
#     Task B: Compute factorial(3), currently i=3...
#     Task C: Compute factorial(4), currently i=3...
#     Task B: factorial(3) = 6
#     Task C: Compute factorial(4), currently i=4...
#     Task C: factorial(4) = 24
#     [2, 6, 24]
@end example

@cartouche
@quotation Note 
If @emph{return_exceptions} is False, cancelling gather() after it
has been marked done won’t cancel any submitted awaitables.
For instance, gather can be marked done after propagating an
exception to the caller, therefore, calling @code{gather.cancel()}
after catching an exception (raised by one of the awaitables) from
gather won’t cancel any other awaitables.
@end quotation
@end cartouche


Changed in version 3.7: If the @emph{gather} itself is cancelled, the cancellation is
propagated regardless of @emph{return_exceptions}.



Changed in version 3.10: Removed the @emph{loop} parameter.



Deprecated since version 3.10: Deprecation warning is emitted if no positional arguments are provided
or not all positional arguments are Future-like objects
and there is no running event loop.

@end deffn

@node Shielding From Cancellation,Timeouts,Running Tasks Concurrently,Coroutines and Tasks
@anchor{library/asyncio-task shielding-from-cancellation}@anchor{298b}
@subsubsection Shielding From Cancellation


@geindex shield() (in module asyncio)
@anchor{library/asyncio-task asyncio shield}@anchor{31b}
@deffn {Function} awaitable asyncio.shield (aw)

Protect an @ref{2983,,awaitable object}
from being @ref{298c,,cancelled}.

If @emph{aw} is a coroutine it is automatically scheduled as a Task.

The statement:

@example
res = await shield(something())
@end example

is equivalent to:

@example
res = await something()
@end example

@emph{except} that if the coroutine containing it is cancelled, the
Task running in @code{something()} is not cancelled.  From the point
of view of @code{something()}, the cancellation did not happen.
Although its caller is still cancelled, so the “await” expression
still raises a @ref{4d7,,CancelledError}.

If @code{something()} is cancelled by other means (i.e. from within
itself) that would also cancel @code{shield()}.

If it is desired to completely ignore cancellation (not recommended)
the @code{shield()} function should be combined with a try/except
clause, as follows:

@example
try:
    res = await shield(something())
except CancelledError:
    res = None
@end example


Changed in version 3.10: Removed the @emph{loop} parameter.



Deprecated since version 3.10: Deprecation warning is emitted if @emph{aw} is not Future-like object
and there is no running event loop.

@end deffn

@node Timeouts,Waiting Primitives,Shielding From Cancellation,Coroutines and Tasks
@anchor{library/asyncio-task timeouts}@anchor{298d}
@subsubsection Timeouts


@geindex wait_for() (in module asyncio)
@anchor{library/asyncio-task asyncio wait_for}@anchor{3d5}
@deffn {Function} coroutine asyncio.wait_for (aw, timeout)

Wait for the @emph{aw} @ref{2983,,awaitable}
to complete with a timeout.

If @emph{aw} is a coroutine it is automatically scheduled as a Task.

@emph{timeout} can either be @code{None} or a float or int number of seconds
to wait for.  If @emph{timeout} is @code{None}, block until the future
completes.

If a timeout occurs, it cancels the task and raises
@ref{2b8,,TimeoutError}.

To avoid the task @ref{298c,,cancellation},
wrap it in @ref{31b,,shield()}.

The function will wait until the future is actually cancelled,
so the total wait time may exceed the @emph{timeout}. If an exception
happens during cancellation, it is propagated.

If the wait is cancelled, the future @emph{aw} is also cancelled.


Changed in version 3.10: Removed the @emph{loop} parameter.

@anchor{library/asyncio-task asyncio-example-waitfor}@anchor{298e}
Example:

@example
async def eternity():
    # Sleep for one hour
    await asyncio.sleep(3600)
    print('yay!')

async def main():
    # Wait for at most 1 second
    try:
        await asyncio.wait_for(eternity(), timeout=1.0)
    except TimeoutError:
        print('timeout!')

asyncio.run(main())

# Expected output:
#
#     timeout!
@end example


Changed in version 3.7: When @emph{aw} is cancelled due to a timeout, @code{wait_for} waits
for @emph{aw} to be cancelled.  Previously, it raised
@ref{2b8,,TimeoutError} immediately.



Changed in version 3.10: Removed the @emph{loop} parameter.

@end deffn

@node Waiting Primitives,Running in Threads,Timeouts,Coroutines and Tasks
@anchor{library/asyncio-task waiting-primitives}@anchor{298f}
@subsubsection Waiting Primitives


@geindex wait() (in module asyncio)
@anchor{library/asyncio-task asyncio wait}@anchor{440}
@deffn {Function} coroutine asyncio.wait (aws, *, timeout=None, return_when=ALL_COMPLETED)

Run @ref{2983,,awaitable objects} in the @emph{aws}
iterable concurrently and block until the condition specified
by @emph{return_when}.

The @emph{aws} iterable must not be empty.

Returns two sets of Tasks/Futures: @code{(done, pending)}.

Usage:

@example
done, pending = await asyncio.wait(aws)
@end example

@emph{timeout} (a float or int), if specified, can be used to control
the maximum number of seconds to wait before returning.

Note that this function does not raise @ref{2b8,,TimeoutError}.
Futures or Tasks that aren’t done when the timeout occurs are simply
returned in the second set.

@emph{return_when} indicates when this function should return.  It must
be one of the following constants:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Description

@item

@code{FIRST_COMPLETED}

@tab

The function will return when any
future finishes or is cancelled.

@item

@code{FIRST_EXCEPTION}

@tab

The function will return when any
future finishes by raising an
exception.  If no future raises an
exception then it is equivalent to
@code{ALL_COMPLETED}.

@item

@code{ALL_COMPLETED}

@tab

The function will return when all
futures finish or are cancelled.

@end multitable


Unlike @ref{3d5,,wait_for()}, @code{wait()} does not cancel the
futures when a timeout occurs.


Deprecated since version 3.8: If any awaitable in @emph{aws} is a coroutine, it is automatically
scheduled as a Task.  Passing coroutines objects to
@code{wait()} directly is deprecated as it leads to
@ref{2990,,confusing behavior}.



Changed in version 3.10: Removed the @emph{loop} parameter.

@anchor{library/asyncio-task asyncio-example-wait-coroutine}@anchor{2990}
@cartouche
@quotation Note 
@code{wait()} schedules coroutines as Tasks automatically and later
returns those implicitly created Task objects in @code{(done, pending)}
sets.  Therefore the following code won’t work as expected:

@example
async def foo():
    return 42

coro = foo()
done, pending = await asyncio.wait(@{coro@})

if coro in done:
    # This branch will never be run!
@end example

Here is how the above snippet can be fixed:

@example
async def foo():
    return 42

task = asyncio.create_task(foo())
done, pending = await asyncio.wait(@{task@})

if task in done:
    # Everything will work as expected now.
@end example
@end quotation
@end cartouche


Deprecated since version 3.8, removed in version 3.11: Passing coroutine objects to @code{wait()} directly is
deprecated.



Changed in version 3.10: Removed the @emph{loop} parameter.

@end deffn

@geindex as_completed() (in module asyncio)
@anchor{library/asyncio-task asyncio as_completed}@anchor{31c}
@deffn {Function} asyncio.as_completed (aws, *, timeout=None)

Run @ref{2983,,awaitable objects} in the @emph{aws}
iterable concurrently.  Return an iterator of coroutines.
Each coroutine returned can be awaited to get the earliest next
result from the iterable of the remaining awaitables.

Raises @ref{2b8,,TimeoutError} if the timeout occurs before
all Futures are done.


Changed in version 3.10: Removed the @emph{loop} parameter.


Example:

@example
for coro in as_completed(aws):
    earliest_result = await coro
    # ...
@end example


Changed in version 3.10: Removed the @emph{loop} parameter.



Deprecated since version 3.10: Deprecation warning is emitted if not all awaitable objects in the @emph{aws}
iterable are Future-like objects and there is no running event loop.

@end deffn

@node Running in Threads,Scheduling From Other Threads,Waiting Primitives,Coroutines and Tasks
@anchor{library/asyncio-task running-in-threads}@anchor{2991}
@subsubsection Running in Threads


@geindex to_thread() (in module asyncio)
@anchor{library/asyncio-task asyncio to_thread}@anchor{3d3}
@deffn {Function} coroutine asyncio.to_thread (func, /, *args, **kwargs)

Asynchronously run function @emph{func} in a separate thread.

Any *args and **kwargs supplied for this function are directly passed
to @emph{func}. Also, the current @ref{2943,,contextvars.Context} is propagated,
allowing context variables from the event loop thread to be accessed in the
separate thread.

Return a coroutine that can be awaited to get the eventual result of @emph{func}.

This coroutine function is primarily intended to be used for executing
IO-bound functions/methods that would otherwise block the event loop if
they were run in the main thread. For example:

@example
def blocking_io():
    print(f"start blocking_io at @{time.strftime('%X')@}")
    # Note that time.sleep() can be replaced with any blocking
    # IO-bound operation, such as file operations.
    time.sleep(1)
    print(f"blocking_io complete at @{time.strftime('%X')@}")

async def main():
    print(f"started main at @{time.strftime('%X')@}")

    await asyncio.gather(
        asyncio.to_thread(blocking_io),
        asyncio.sleep(1))

    print(f"finished main at @{time.strftime('%X')@}")


asyncio.run(main())

# Expected output:
#
# started main at 19:50:53
# start blocking_io at 19:50:53
# blocking_io complete at 19:50:54
# finished main at 19:50:54
@end example

Directly calling @cite{blocking_io()} in any coroutine would block the event loop
for its duration, resulting in an additional 1 second of run time. Instead,
by using @cite{asyncio.to_thread()}, we can run it in a separate thread without
blocking the event loop.

@cartouche
@quotation Note 
Due to the @ref{385,,GIL}, @cite{asyncio.to_thread()} can typically only be used
to make IO-bound functions non-blocking. However, for extension modules
that release the GIL or alternative Python implementations that don’t
have one, @cite{asyncio.to_thread()} can also be used for CPU-bound functions.
@end quotation
@end cartouche


New in version 3.9.

@end deffn

@node Scheduling From Other Threads,Introspection,Running in Threads,Coroutines and Tasks
@anchor{library/asyncio-task scheduling-from-other-threads}@anchor{2992}
@subsubsection Scheduling From Other Threads


@geindex run_coroutine_threadsafe() (in module asyncio)
@anchor{library/asyncio-task asyncio run_coroutine_threadsafe}@anchor{348}
@deffn {Function} asyncio.run_coroutine_threadsafe (coro, loop)

Submit a coroutine to the given event loop.  Thread-safe.

Return a @ref{d57,,concurrent.futures.Future} to wait for the result
from another OS thread.

This function is meant to be called from a different OS thread
than the one where the event loop is running.  Example:

@example
# Create a coroutine
coro = asyncio.sleep(1, result=3)

# Submit the coroutine to a given loop
future = asyncio.run_coroutine_threadsafe(coro, loop)

# Wait for the result with an optional timeout argument
assert future.result(timeout) == 3
@end example

If an exception is raised in the coroutine, the returned Future
will be notified.  It can also be used to cancel the task in
the event loop:

@example
try:
    result = future.result(timeout)
except TimeoutError:
    print('The coroutine took too long, cancelling the task...')
    future.cancel()
except Exception as exc:
    print(f'The coroutine raised an exception: @{exc!r@}')
else:
    print(f'The coroutine returned: @{result!r@}')
@end example

See the @ref{2993,,concurrency and multithreading}
section of the documentation.

Unlike other asyncio functions this function requires the @emph{loop}
argument to be passed explicitly.


New in version 3.5.1.

@end deffn

@node Introspection,Task Object,Scheduling From Other Threads,Coroutines and Tasks
@anchor{library/asyncio-task introspection}@anchor{2994}
@subsubsection Introspection


@geindex current_task() (in module asyncio)
@anchor{library/asyncio-task asyncio current_task}@anchor{458}
@deffn {Function} asyncio.current_task (loop=None)

Return the currently running @ref{317,,Task} instance, or @code{None} if
no task is running.

If @emph{loop} is @code{None} @ref{315,,get_running_loop()} is used to get
the current loop.


New in version 3.7.

@end deffn

@geindex all_tasks() (in module asyncio)
@anchor{library/asyncio-task asyncio all_tasks}@anchor{459}
@deffn {Function} asyncio.all_tasks (loop=None)

Return a set of not yet finished @ref{317,,Task} objects run by
the loop.

If @emph{loop} is @code{None}, @ref{315,,get_running_loop()} is used for getting
current loop.


New in version 3.7.

@end deffn

@node Task Object,,Introspection,Coroutines and Tasks
@anchor{library/asyncio-task task-object}@anchor{2995}
@subsubsection Task Object


@geindex Task (class in asyncio)
@anchor{library/asyncio-task asyncio Task}@anchor{317}
@deffn {Class} asyncio.Task (coro, *, loop=None, name=None)

A @ref{316,,Future-like} object that runs a Python
@ref{2977,,coroutine}.  Not thread-safe.

Tasks are used to run coroutines in event loops.
If a coroutine awaits on a Future, the Task suspends
the execution of the coroutine and waits for the completion
of the Future.  When the Future is @emph{done}, the execution of
the wrapped coroutine resumes.

Event loops use cooperative scheduling: an event loop runs
one Task at a time.  While a Task awaits for the completion of a
Future, the event loop runs other Tasks, callbacks, or performs
IO operations.

Use the high-level @ref{4dc,,asyncio.create_task()} function to create
Tasks, or the low-level @ref{4dd,,loop.create_task()} or
@ref{318,,ensure_future()} functions.  Manual instantiation of Tasks
is discouraged.

To cancel a running Task use the @ref{298c,,cancel()} method.  Calling it
will cause the Task to throw a @ref{4d7,,CancelledError} exception into
the wrapped coroutine.  If a coroutine is awaiting on a Future
object during cancellation, the Future object will be cancelled.

@ref{2996,,cancelled()} can be used to check if the Task was cancelled.
The method returns @code{True} if the wrapped coroutine did not
suppress the @ref{4d7,,CancelledError} exception and was actually
cancelled.

@ref{317,,asyncio.Task} inherits from @ref{316,,Future} all of its
APIs except @ref{2997,,Future.set_result()} and
@ref{2998,,Future.set_exception()}.

Tasks support the @ref{23,,contextvars} module.  When a Task
is created it copies the current context and later runs its
coroutine in the copied context.


Changed in version 3.7: Added support for the @ref{23,,contextvars} module.



Changed in version 3.8: Added the @emph{name} parameter.



Deprecated since version 3.10: Deprecation warning is emitted if @emph{loop} is not specified
and there is no running event loop.


@geindex cancel() (asyncio.Task method)
@anchor{library/asyncio-task asyncio Task cancel}@anchor{298c}
@deffn {Method} cancel (msg=None)

Request the Task to be cancelled.

This arranges for a @ref{4d7,,CancelledError} exception to be thrown
into the wrapped coroutine on the next cycle of the event loop.

The coroutine then has a chance to clean up or even deny the
request by suppressing the exception with a @ref{f6d,,try} …
… @code{except CancelledError} … @ref{4c1,,finally} block.
Therefore, unlike @ref{2999,,Future.cancel()}, @ref{298c,,Task.cancel()} does
not guarantee that the Task will be cancelled, although
suppressing cancellation completely is not common and is actively
discouraged.


Changed in version 3.9: Added the @emph{msg} parameter.



Changed in version 3.11: The @code{msg} parameter is propagated from cancelled task to its awaiter.

@anchor{library/asyncio-task asyncio-example-task-cancel}@anchor{299a}
The following example illustrates how coroutines can intercept
the cancellation request:

@example
async def cancel_me():
    print('cancel_me(): before sleep')

    try:
        # Wait for 1 hour
        await asyncio.sleep(3600)
    except asyncio.CancelledError:
        print('cancel_me(): cancel sleep')
        raise
    finally:
        print('cancel_me(): after sleep')

async def main():
    # Create a "cancel_me" Task
    task = asyncio.create_task(cancel_me())

    # Wait for 1 second
    await asyncio.sleep(1)

    task.cancel()
    try:
        await task
    except asyncio.CancelledError:
        print("main(): cancel_me is cancelled now")

asyncio.run(main())

# Expected output:
#
#     cancel_me(): before sleep
#     cancel_me(): cancel sleep
#     cancel_me(): after sleep
#     main(): cancel_me is cancelled now
@end example
@end deffn

@geindex cancelled() (asyncio.Task method)
@anchor{library/asyncio-task asyncio Task cancelled}@anchor{2996}
@deffn {Method} cancelled ()

Return @code{True} if the Task is @emph{cancelled}.

The Task is @emph{cancelled} when the cancellation was requested with
@ref{298c,,cancel()} and the wrapped coroutine propagated the
@ref{4d7,,CancelledError} exception thrown into it.
@end deffn

@geindex done() (asyncio.Task method)
@anchor{library/asyncio-task asyncio Task done}@anchor{299b}
@deffn {Method} done ()

Return @code{True} if the Task is @emph{done}.

A Task is @emph{done} when the wrapped coroutine either returned
a value, raised an exception, or the Task was cancelled.
@end deffn

@geindex result() (asyncio.Task method)
@anchor{library/asyncio-task asyncio Task result}@anchor{299c}
@deffn {Method} result ()

Return the result of the Task.

If the Task is @emph{done}, the result of the wrapped coroutine
is returned (or if the coroutine raised an exception, that
exception is re-raised.)

If the Task has been @emph{cancelled}, this method raises
a @ref{4d7,,CancelledError} exception.

If the Task’s result isn’t yet available, this method raises
a @ref{299d,,InvalidStateError} exception.
@end deffn

@geindex exception() (asyncio.Task method)
@anchor{library/asyncio-task asyncio Task exception}@anchor{299e}
@deffn {Method} exception ()

Return the exception of the Task.

If the wrapped coroutine raised an exception that exception
is returned.  If the wrapped coroutine returned normally
this method returns @code{None}.

If the Task has been @emph{cancelled}, this method raises a
@ref{4d7,,CancelledError} exception.

If the Task isn’t @emph{done} yet, this method raises an
@ref{299d,,InvalidStateError} exception.
@end deffn

@geindex add_done_callback() (asyncio.Task method)
@anchor{library/asyncio-task asyncio Task add_done_callback}@anchor{299f}
@deffn {Method} add_done_callback (callback, *, context=None)

Add a callback to be run when the Task is @emph{done}.

This method should only be used in low-level callback-based code.

See the documentation of @ref{626,,Future.add_done_callback()}
for more details.
@end deffn

@geindex remove_done_callback() (asyncio.Task method)
@anchor{library/asyncio-task asyncio Task remove_done_callback}@anchor{29a0}
@deffn {Method} remove_done_callback (callback)

Remove @emph{callback} from the callbacks list.

This method should only be used in low-level callback-based code.

See the documentation of @ref{29a1,,Future.remove_done_callback()}
for more details.
@end deffn

@geindex get_stack() (asyncio.Task method)
@anchor{library/asyncio-task asyncio Task get_stack}@anchor{29a2}
@deffn {Method} get_stack (*, limit=None)

Return the list of stack frames for this Task.

If the wrapped coroutine is not done, this returns the stack
where it is suspended.  If the coroutine has completed
successfully or was cancelled, this returns an empty list.
If the coroutine was terminated by an exception, this returns
the list of traceback frames.

The frames are always ordered from oldest to newest.

Only one stack frame is returned for a suspended coroutine.

The optional @emph{limit} argument sets the maximum number of frames
to return; by default all available frames are returned.
The ordering of the returned list differs depending on whether
a stack or a traceback is returned: the newest frames of a
stack are returned, but the oldest frames of a traceback are
returned.  (This matches the behavior of the traceback module.)
@end deffn

@geindex print_stack() (asyncio.Task method)
@anchor{library/asyncio-task asyncio Task print_stack}@anchor{29a3}
@deffn {Method} print_stack (*, limit=None, file=None)

Print the stack or traceback for this Task.

This produces output similar to that of the traceback module
for the frames retrieved by @ref{29a2,,get_stack()}.

The @emph{limit} argument is passed to @ref{29a2,,get_stack()} directly.

The @emph{file} argument is an I/O stream to which the output
is written; by default output is written to @ref{42d,,sys.stderr}.
@end deffn

@geindex get_coro() (asyncio.Task method)
@anchor{library/asyncio-task asyncio Task get_coro}@anchor{4db}
@deffn {Method} get_coro ()

Return the coroutine object wrapped by the @ref{317,,Task}.


New in version 3.8.

@end deffn

@geindex get_name() (asyncio.Task method)
@anchor{library/asyncio-task asyncio Task get_name}@anchor{4df}
@deffn {Method} get_name ()

Return the name of the Task.

If no name has been explicitly assigned to the Task, the default
asyncio Task implementation generates a default name during
instantiation.


New in version 3.8.

@end deffn

@geindex set_name() (asyncio.Task method)
@anchor{library/asyncio-task asyncio Task set_name}@anchor{4de}
@deffn {Method} set_name (value)

Set the name of the Task.

The @emph{value} argument can be any object, which is then
converted to a string.

In the default Task implementation, the name will be visible
in the @ref{27d,,repr()} output of a task object.


New in version 3.8.

@end deffn
@end deffn

@node Streams,Synchronization Primitives,Coroutines and Tasks,asyncio — Asynchronous I/O
@anchor{library/asyncio-stream doc}@anchor{29a4}@anchor{library/asyncio-stream asyncio-streams}@anchor{2978}@anchor{library/asyncio-stream streams}@anchor{29a5}
@subsubsection Streams


@strong{Source code:} Lib/asyncio/streams.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/streams.py}


__________________________________________________________________


Streams are high-level async/await-ready primitives to work with
network connections.  Streams allow sending and receiving data without
using callbacks or low-level protocols and transports.
@anchor{library/asyncio-stream asyncio-example-stream}@anchor{29a6}
Here is an example of a TCP echo client written using asyncio
streams:

@example
import asyncio

async def tcp_echo_client(message):
    reader, writer = await asyncio.open_connection(
        '127.0.0.1', 8888)

    print(f'Send: @{message!r@}')
    writer.write(message.encode())
    await writer.drain()

    data = await reader.read(100)
    print(f'Received: @{data.decode()!r@}')

    print('Close the connection')
    writer.close()
    await writer.wait_closed()

asyncio.run(tcp_echo_client('Hello World!'))
@end example

See also the @ref{29a7,,Examples} section below.

@subsubheading Stream Functions


The following top-level asyncio functions can be used to create
and work with streams:

@geindex open_connection() (in module asyncio)
@anchor{library/asyncio-stream asyncio open_connection}@anchor{636}
@deffn {Function} coroutine asyncio.open_connection (host=None, port=None, *, limit=None, ssl=None, family=0, proto=0, flags=0, sock=None, local_addr=None, server_hostname=None, ssl_handshake_timeout=None)

Establish a network connection and return a pair of
@code{(reader, writer)} objects.

The returned @emph{reader} and @emph{writer} objects are instances of
@ref{31d,,StreamReader} and @ref{1299,,StreamWriter} classes.

@emph{limit} determines the buffer size limit used by the
returned @ref{31d,,StreamReader} instance.  By default the @emph{limit}
is set to 64 KiB.

The rest of the arguments are passed directly to
@ref{4e0,,loop.create_connection()}.


Changed in version 3.7: Added the @emph{ssl_handshake_timeout} parameter.



Changed in version 3.10: Removed the @emph{loop} parameter.

@end deffn

@geindex start_server() (in module asyncio)
@anchor{library/asyncio-stream asyncio start_server}@anchor{637}
@deffn {Function} coroutine asyncio.start_server (client_connected_cb, host=None, port=None, *, limit=None, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None, reuse_port=None, ssl_handshake_timeout=None, start_serving=True)

Start a socket server.

The @emph{client_connected_cb} callback is called whenever a new client
connection is established.  It receives a @code{(reader, writer)} pair
as two arguments, instances of the @ref{31d,,StreamReader} and
@ref{1299,,StreamWriter} classes.

@emph{client_connected_cb} can be a plain callable or a
@ref{2977,,coroutine function}; if it is a coroutine function,
it will be automatically scheduled as a @ref{317,,Task}.

@emph{limit} determines the buffer size limit used by the
returned @ref{31d,,StreamReader} instance.  By default the @emph{limit}
is set to 64 KiB.

The rest of the arguments are passed directly to
@ref{630,,loop.create_server()}.


Changed in version 3.7: Added the @emph{ssl_handshake_timeout} and @emph{start_serving} parameters.



Changed in version 3.10: Removed the @emph{loop} parameter.

@end deffn

@subsubheading Unix Sockets


@geindex open_unix_connection() (in module asyncio)
@anchor{library/asyncio-stream asyncio open_unix_connection}@anchor{29a8}
@deffn {Function} coroutine asyncio.open_unix_connection (path=None, *, limit=None, ssl=None, sock=None, server_hostname=None, ssl_handshake_timeout=None)

Establish a Unix socket connection and return a pair of
@code{(reader, writer)}.

Similar to @ref{636,,open_connection()} but operates on Unix sockets.

See also the documentation of @ref{29a9,,loop.create_unix_connection()}.

@ref{1629,,Availability}: Unix.


Changed in version 3.7: Added the @emph{ssl_handshake_timeout} parameter.
The @emph{path} parameter can now be a @ref{63d,,path-like object}



Changed in version 3.10: Removed the @emph{loop} parameter.

@end deffn

@geindex start_unix_server() (in module asyncio)
@anchor{library/asyncio-stream asyncio start_unix_server}@anchor{29aa}
@deffn {Function} coroutine asyncio.start_unix_server (client_connected_cb, path=None, *, limit=None, sock=None, backlog=100, ssl=None, ssl_handshake_timeout=None, start_serving=True)

Start a Unix socket server.

Similar to @ref{637,,start_server()} but works with Unix sockets.

See also the documentation of @ref{631,,loop.create_unix_server()}.

@ref{1629,,Availability}: Unix.


Changed in version 3.7: Added the @emph{ssl_handshake_timeout} and @emph{start_serving} parameters.
The @emph{path} parameter can now be a @ref{63d,,path-like object}.



Changed in version 3.10: Removed the @emph{loop} parameter.

@end deffn

@menu
* StreamReader:: 
* StreamWriter:: 
* Examples: Examples<13>. 

@end menu

@node StreamReader,StreamWriter,,Streams
@anchor{library/asyncio-stream streamreader}@anchor{29ab}
@subsubsection StreamReader


@geindex StreamReader (class in asyncio)
@anchor{library/asyncio-stream asyncio StreamReader}@anchor{31d}
@deffn {Class} asyncio.StreamReader

Represents a reader object that provides APIs to read data
from the IO stream.

It is not recommended to instantiate @emph{StreamReader} objects
directly; use @ref{636,,open_connection()} and @ref{637,,start_server()}
instead.

@geindex read() (asyncio.StreamReader method)
@anchor{library/asyncio-stream asyncio StreamReader read}@anchor{29ac}
@deffn {Method} coroutine read (n=- 1)

Read up to @emph{n} bytes.  If @emph{n} is not provided, or set to @code{-1},
read until EOF and return all read bytes.

If EOF was received and the internal buffer is empty,
return an empty @code{bytes} object.
@end deffn

@geindex readline() (asyncio.StreamReader method)
@anchor{library/asyncio-stream asyncio StreamReader readline}@anchor{29ad}
@deffn {Method} coroutine readline ()

Read one line, where “line” is a sequence of bytes
ending with @code{\n}.

If EOF is received and @code{\n} was not found, the method
returns partially read data.

If EOF is received and the internal buffer is empty,
return an empty @code{bytes} object.
@end deffn

@geindex readexactly() (asyncio.StreamReader method)
@anchor{library/asyncio-stream asyncio StreamReader readexactly}@anchor{7c2}
@deffn {Method} coroutine readexactly (n)

Read exactly @emph{n} bytes.

Raise an @ref{1198,,IncompleteReadError} if EOF is reached before @emph{n}
can be read.  Use the @ref{29ae,,IncompleteReadError.partial}
attribute to get the partially read data.
@end deffn

@geindex readuntil() (asyncio.StreamReader method)
@anchor{library/asyncio-stream asyncio StreamReader readuntil}@anchor{7c1}
@deffn {Method} coroutine readuntil (separator=b'\n')

Read data from the stream until @emph{separator} is found.

On success, the data and separator will be removed from the
internal buffer (consumed). Returned data will include the
separator at the end.

If the amount of data read exceeds the configured stream limit, a
@ref{29af,,LimitOverrunError} exception is raised, and the data
is left in the internal buffer and can be read again.

If EOF is reached before the complete separator is found,
an @ref{1198,,IncompleteReadError} exception is raised, and the internal
buffer is reset.  The @ref{29ae,,IncompleteReadError.partial} attribute
may contain a portion of the separator.


New in version 3.5.2.

@end deffn

@geindex at_eof() (asyncio.StreamReader method)
@anchor{library/asyncio-stream asyncio StreamReader at_eof}@anchor{29b0}
@deffn {Method} at_eof ()

Return @code{True} if the buffer is empty and @code{feed_eof()}
was called.
@end deffn
@end deffn

@node StreamWriter,Examples<13>,StreamReader,Streams
@anchor{library/asyncio-stream streamwriter}@anchor{29b1}
@subsubsection StreamWriter


@geindex StreamWriter (class in asyncio)
@anchor{library/asyncio-stream asyncio StreamWriter}@anchor{1299}
@deffn {Class} asyncio.StreamWriter

Represents a writer object that provides APIs to write data
to the IO stream.

It is not recommended to instantiate @emph{StreamWriter} objects
directly; use @ref{636,,open_connection()} and @ref{637,,start_server()}
instead.

@geindex write() (asyncio.StreamWriter method)
@anchor{library/asyncio-stream asyncio StreamWriter write}@anchor{29b2}
@deffn {Method} write (data)

The method attempts to write the @emph{data} to the underlying socket immediately.
If that fails, the data is queued in an internal write buffer until it can be
sent.

The method should be used along with the @code{drain()} method:

@example
stream.write(data)
await stream.drain()
@end example
@end deffn

@geindex writelines() (asyncio.StreamWriter method)
@anchor{library/asyncio-stream asyncio StreamWriter writelines}@anchor{29b3}
@deffn {Method} writelines (data)

The method writes a list (or any iterable) of bytes to the underlying socket
immediately.
If that fails, the data is queued in an internal write buffer until it can be
sent.

The method should be used along with the @code{drain()} method:

@example
stream.writelines(lines)
await stream.drain()
@end example
@end deffn

@geindex close() (asyncio.StreamWriter method)
@anchor{library/asyncio-stream asyncio StreamWriter close}@anchor{29b4}
@deffn {Method} close ()

The method closes the stream and the underlying socket.

The method should be used along with the @code{wait_closed()} method:

@example
stream.close()
await stream.wait_closed()
@end example
@end deffn

@geindex can_write_eof() (asyncio.StreamWriter method)
@anchor{library/asyncio-stream asyncio StreamWriter can_write_eof}@anchor{29b5}
@deffn {Method} can_write_eof ()

Return @code{True} if the underlying transport supports
the @ref{29b6,,write_eof()} method, @code{False} otherwise.
@end deffn

@geindex write_eof() (asyncio.StreamWriter method)
@anchor{library/asyncio-stream asyncio StreamWriter write_eof}@anchor{29b6}
@deffn {Method} write_eof ()

Close the write end of the stream after the buffered write
data is flushed.
@end deffn

@geindex transport (asyncio.StreamWriter attribute)
@anchor{library/asyncio-stream asyncio StreamWriter transport}@anchor{29b7}
@deffn {Attribute} transport

Return the underlying asyncio transport.
@end deffn

@geindex get_extra_info() (asyncio.StreamWriter method)
@anchor{library/asyncio-stream asyncio StreamWriter get_extra_info}@anchor{29b8}
@deffn {Method} get_extra_info (name, default=None)

Access optional transport information; see
@ref{5bc,,BaseTransport.get_extra_info()} for details.
@end deffn

@geindex drain() (asyncio.StreamWriter method)
@anchor{library/asyncio-stream asyncio StreamWriter drain}@anchor{29b9}
@deffn {Method} coroutine drain ()

Wait until it is appropriate to resume writing to the stream.
Example:

@example
writer.write(data)
await writer.drain()
@end example

This is a flow control method that interacts with the underlying
IO write buffer.  When the size of the buffer reaches
the high watermark, @emph{drain()} blocks until the size of the
buffer is drained down to the low watermark and writing can
be resumed.  When there is nothing to wait for, the @ref{29b9,,drain()}
returns immediately.
@end deffn

@geindex is_closing() (asyncio.StreamWriter method)
@anchor{library/asyncio-stream asyncio StreamWriter is_closing}@anchor{62a}
@deffn {Method} is_closing ()

Return @code{True} if the stream is closed or in the process of
being closed.


New in version 3.7.

@end deffn

@geindex wait_closed() (asyncio.StreamWriter method)
@anchor{library/asyncio-stream asyncio StreamWriter wait_closed}@anchor{629}
@deffn {Method} coroutine wait_closed ()

Wait until the stream is closed.

Should be called after @ref{29b4,,close()} to wait until the underlying
connection is closed.


New in version 3.7.

@end deffn
@end deffn

@node Examples<13>,,StreamWriter,Streams
@anchor{library/asyncio-stream examples}@anchor{29a7}
@subsubsection Examples


@menu
* TCP echo client using streams:: 
* TCP echo server using streams:: 
* Get HTTP headers:: 
* Register an open socket to wait for data using streams:: 

@end menu

@node TCP echo client using streams,TCP echo server using streams,,Examples<13>
@anchor{library/asyncio-stream asyncio-tcp-echo-client-streams}@anchor{29ba}@anchor{library/asyncio-stream tcp-echo-client-using-streams}@anchor{29bb}
@subsubsection TCP echo client using streams


TCP echo client using the @ref{636,,asyncio.open_connection()} function:

@example
import asyncio

async def tcp_echo_client(message):
    reader, writer = await asyncio.open_connection(
        '127.0.0.1', 8888)

    print(f'Send: @{message!r@}')
    writer.write(message.encode())
    await writer.drain()

    data = await reader.read(100)
    print(f'Received: @{data.decode()!r@}')

    print('Close the connection')
    writer.close()

asyncio.run(tcp_echo_client('Hello World!'))
@end example


@subsubheading See also


The @ref{29bc,,TCP echo client protocol}
example uses the low-level @ref{4e0,,loop.create_connection()} method.


@node TCP echo server using streams,Get HTTP headers,TCP echo client using streams,Examples<13>
@anchor{library/asyncio-stream asyncio-tcp-echo-server-streams}@anchor{29bd}@anchor{library/asyncio-stream tcp-echo-server-using-streams}@anchor{29be}
@subsubsection TCP echo server using streams


TCP echo server using the @ref{637,,asyncio.start_server()} function:

@example
import asyncio

async def handle_echo(reader, writer):
    data = await reader.read(100)
    message = data.decode()
    addr = writer.get_extra_info('peername')

    print(f"Received @{message!r@} from @{addr!r@}")

    print(f"Send: @{message!r@}")
    writer.write(data)
    await writer.drain()

    print("Close the connection")
    writer.close()

async def main():
    server = await asyncio.start_server(
        handle_echo, '127.0.0.1', 8888)

    addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets)
    print(f'Serving on @{addrs@}')

    async with server:
        await server.serve_forever()

asyncio.run(main())
@end example


@subsubheading See also


The @ref{29bf,,TCP echo server protocol}
example uses the @ref{630,,loop.create_server()} method.


@node Get HTTP headers,Register an open socket to wait for data using streams,TCP echo server using streams,Examples<13>
@anchor{library/asyncio-stream get-http-headers}@anchor{29c0}
@subsubsection Get HTTP headers


Simple example querying HTTP headers of the URL passed on the command line:

@example
import asyncio
import urllib.parse
import sys

async def print_http_headers(url):
    url = urllib.parse.urlsplit(url)
    if url.scheme == 'https':
        reader, writer = await asyncio.open_connection(
            url.hostname, 443, ssl=True)
    else:
        reader, writer = await asyncio.open_connection(
            url.hostname, 80)

    query = (
        f"HEAD @{url.path or '/'@} HTTP/1.0\r\n"
        f"Host: @{url.hostname@}\r\n"
        f"\r\n"
    )

    writer.write(query.encode('latin-1'))
    while True:
        line = await reader.readline()
        if not line:
            break

        line = line.decode('latin1').rstrip()
        if line:
            print(f'HTTP header> @{line@}')

    # Ignore the body, close the socket
    writer.close()

url = sys.argv[1]
asyncio.run(print_http_headers(url))
@end example

Usage:

@example
python example.py http://example.com/path/page.html
@end example

or with HTTPS:

@example
python example.py https://example.com/path/page.html
@end example

@node Register an open socket to wait for data using streams,,Get HTTP headers,Examples<13>
@anchor{library/asyncio-stream asyncio-example-create-connection-streams}@anchor{29c1}@anchor{library/asyncio-stream register-an-open-socket-to-wait-for-data-using-streams}@anchor{29c2}
@subsubsection Register an open socket to wait for data using streams


Coroutine waiting until a socket receives data using the
@ref{636,,open_connection()} function:

@example
import asyncio
import socket

async def wait_for_data():
    # Get a reference to the current event loop because
    # we want to access low-level APIs.
    loop = asyncio.get_running_loop()

    # Create a pair of connected sockets.
    rsock, wsock = socket.socketpair()

    # Register the open socket to wait for data.
    reader, writer = await asyncio.open_connection(sock=rsock)

    # Simulate the reception of data from the network
    loop.call_soon(wsock.send, 'abc'.encode())

    # Wait for data
    data = await reader.read(100)

    # Got data, we are done: close the socket
    print("Received:", data.decode())
    writer.close()

    # Close the second socket
    wsock.close()

asyncio.run(wait_for_data())
@end example


@subsubheading See also


The @ref{29c3,,register an open socket to wait for data using a protocol} example uses a low-level protocol and
the @ref{4e0,,loop.create_connection()} method.

The @ref{29c4,,watch a file descriptor for read events} example uses the low-level
@ref{29c5,,loop.add_reader()} method to watch a file descriptor.


@node Synchronization Primitives,Subprocesses,Streams,asyncio — Asynchronous I/O
@anchor{library/asyncio-sync doc}@anchor{29c6}@anchor{library/asyncio-sync asyncio-sync}@anchor{297b}@anchor{library/asyncio-sync synchronization-primitives}@anchor{29c7}
@subsubsection Synchronization Primitives


@strong{Source code:} Lib/asyncio/locks.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/locks.py}


__________________________________________________________________


asyncio synchronization primitives are designed to be similar to
those of the @ref{109,,threading} module with two important caveats:


@itemize *

@item 
asyncio primitives are not thread-safe, therefore they should not
be used for OS thread synchronization (use @ref{109,,threading} for
that);

@item 
methods of these synchronization primitives do not accept the @emph{timeout}
argument; use the @ref{3d5,,asyncio.wait_for()} function to perform
operations with timeouts.
@end itemize

asyncio has the following basic synchronization primitives:


@itemize *

@item 
@ref{58b,,Lock}

@item 
@ref{58c,,Event}

@item 
@ref{58d,,Condition}

@item 
@ref{58e,,Semaphore}

@item 
@ref{58f,,BoundedSemaphore}
@end itemize


__________________________________________________________________


@menu
* Lock:: 
* Event:: 
* Condition:: 
* Semaphore:: 
* BoundedSemaphore:: 

@end menu

@node Lock,Event,,Synchronization Primitives
@anchor{library/asyncio-sync lock}@anchor{29c8}
@subsubsection Lock


@geindex Lock (class in asyncio)
@anchor{library/asyncio-sync asyncio Lock}@anchor{58b}
@deffn {Class} asyncio.Lock

Implements a mutex lock for asyncio tasks.  Not thread-safe.

An asyncio lock can be used to guarantee exclusive access to a
shared resource.

The preferred way to use a Lock is an @ref{169,,async with}
statement:

@example
lock = asyncio.Lock()

# ... later
async with lock:
    # access shared state
@end example

which is equivalent to:

@example
lock = asyncio.Lock()

# ... later
await lock.acquire()
try:
    # access shared state
finally:
    lock.release()
@end example


Changed in version 3.10: Removed the @emph{loop} parameter.


@geindex acquire() (asyncio.Lock method)
@anchor{library/asyncio-sync asyncio Lock acquire}@anchor{29c9}
@deffn {Method} coroutine acquire ()

Acquire the lock.

This method waits until the lock is @emph{unlocked}, sets it to
@emph{locked} and returns @code{True}.

When more than one coroutine is blocked in @ref{29c9,,acquire()}
waiting for the lock to be unlocked, only one coroutine
eventually proceeds.

Acquiring a lock is @emph{fair}: the coroutine that proceeds will be
the first coroutine that started waiting on the lock.
@end deffn

@geindex release() (asyncio.Lock method)
@anchor{library/asyncio-sync asyncio Lock release}@anchor{29ca}
@deffn {Method} release ()

Release the lock.

When the lock is @emph{locked}, reset it to @emph{unlocked} and return.

If the lock is @emph{unlocked}, a @ref{5af,,RuntimeError} is raised.
@end deffn

@geindex locked() (asyncio.Lock method)
@anchor{library/asyncio-sync asyncio Lock locked}@anchor{29cb}
@deffn {Method} locked ()

Return @code{True} if the lock is @emph{locked}.
@end deffn
@end deffn

@node Event,Condition,Lock,Synchronization Primitives
@anchor{library/asyncio-sync event}@anchor{29cc}
@subsubsection Event


@geindex Event (class in asyncio)
@anchor{library/asyncio-sync asyncio Event}@anchor{58c}
@deffn {Class} asyncio.Event

An event object.  Not thread-safe.

An asyncio event can be used to notify multiple asyncio tasks
that some event has happened.

An Event object manages an internal flag that can be set to @emph{true}
with the @ref{29cd,,set()} method and reset to @emph{false} with the
@ref{29ce,,clear()} method.  The @ref{29cf,,wait()} method blocks until the
flag is set to @emph{true}.  The flag is set to @emph{false} initially.


Changed in version 3.10: Removed the @emph{loop} parameter.

@anchor{library/asyncio-sync asyncio-example-sync-event}@anchor{29d0}
Example:

@example
async def waiter(event):
    print('waiting for it ...')
    await event.wait()
    print('... got it!')

async def main():
    # Create an Event object.
    event = asyncio.Event()

    # Spawn a Task to wait until 'event' is set.
    waiter_task = asyncio.create_task(waiter(event))

    # Sleep for 1 second and set the event.
    await asyncio.sleep(1)
    event.set()

    # Wait until the waiter task is finished.
    await waiter_task

asyncio.run(main())
@end example

@geindex wait() (asyncio.Event method)
@anchor{library/asyncio-sync asyncio Event wait}@anchor{29cf}
@deffn {Method} coroutine wait ()

Wait until the event is set.

If the event is set, return @code{True} immediately.
Otherwise block until another task calls @ref{29cd,,set()}.
@end deffn

@geindex set() (asyncio.Event method)
@anchor{library/asyncio-sync asyncio Event set}@anchor{29cd}
@deffn {Method} set ()

Set the event.

All tasks waiting for event to be set will be immediately
awakened.
@end deffn

@geindex clear() (asyncio.Event method)
@anchor{library/asyncio-sync asyncio Event clear}@anchor{29ce}
@deffn {Method} clear ()

Clear (unset) the event.

Tasks awaiting on @ref{29cf,,wait()} will now block until the
@ref{29cd,,set()} method is called again.
@end deffn

@geindex is_set() (asyncio.Event method)
@anchor{library/asyncio-sync asyncio Event is_set}@anchor{29d1}
@deffn {Method} is_set ()

Return @code{True} if the event is set.
@end deffn
@end deffn

@node Condition,Semaphore,Event,Synchronization Primitives
@anchor{library/asyncio-sync condition}@anchor{29d2}
@subsubsection Condition


@geindex Condition (class in asyncio)
@anchor{library/asyncio-sync asyncio Condition}@anchor{58d}
@deffn {Class} asyncio.Condition (lock=None)

A Condition object.  Not thread-safe.

An asyncio condition primitive can be used by a task to wait for
some event to happen and then get exclusive access to a shared
resource.

In essence, a Condition object combines the functionality
of an @ref{58c,,Event} and a @ref{58b,,Lock}.  It is possible to have
multiple Condition objects share one Lock, which allows coordinating
exclusive access to a shared resource between different tasks
interested in particular states of that shared resource.

The optional @emph{lock} argument must be a @ref{58b,,Lock} object or
@code{None}.  In the latter case a new Lock object is created
automatically.


Changed in version 3.10: Removed the @emph{loop} parameter.


The preferred way to use a Condition is an @ref{169,,async with}
statement:

@example
cond = asyncio.Condition()

# ... later
async with cond:
    await cond.wait()
@end example

which is equivalent to:

@example
cond = asyncio.Condition()

# ... later
await cond.acquire()
try:
    await cond.wait()
finally:
    cond.release()
@end example

@geindex acquire() (asyncio.Condition method)
@anchor{library/asyncio-sync asyncio Condition acquire}@anchor{29d3}
@deffn {Method} coroutine acquire ()

Acquire the underlying lock.

This method waits until the underlying lock is @emph{unlocked},
sets it to @emph{locked} and returns @code{True}.
@end deffn

@geindex notify() (asyncio.Condition method)
@anchor{library/asyncio-sync asyncio Condition notify}@anchor{29d4}
@deffn {Method} notify (n=1)

Wake up at most @emph{n} tasks (1 by default) waiting on this
condition.  The method is no-op if no tasks are waiting.

The lock must be acquired before this method is called and
released shortly after.  If called with an @emph{unlocked} lock
a @ref{5af,,RuntimeError} error is raised.
@end deffn

@geindex locked() (asyncio.Condition method)
@anchor{library/asyncio-sync asyncio Condition locked}@anchor{29d5}
@deffn {Method} locked ()

Return @code{True} if the underlying lock is acquired.
@end deffn

@geindex notify_all() (asyncio.Condition method)
@anchor{library/asyncio-sync asyncio Condition notify_all}@anchor{29d6}
@deffn {Method} notify_all ()

Wake up all tasks waiting on this condition.

This method acts like @ref{29d4,,notify()}, but wakes up all waiting
tasks.

The lock must be acquired before this method is called and
released shortly after.  If called with an @emph{unlocked} lock
a @ref{5af,,RuntimeError} error is raised.
@end deffn

@geindex release() (asyncio.Condition method)
@anchor{library/asyncio-sync asyncio Condition release}@anchor{29d7}
@deffn {Method} release ()

Release the underlying lock.

When invoked on an unlocked lock, a @ref{5af,,RuntimeError} is
raised.
@end deffn

@geindex wait() (asyncio.Condition method)
@anchor{library/asyncio-sync asyncio Condition wait}@anchor{29d8}
@deffn {Method} coroutine wait ()

Wait until notified.

If the calling task has not acquired the lock when this method is
called, a @ref{5af,,RuntimeError} is raised.

This method releases the underlying lock, and then blocks until
it is awakened by a @ref{29d4,,notify()} or @ref{29d6,,notify_all()} call.
Once awakened, the Condition re-acquires its lock and this method
returns @code{True}.
@end deffn

@geindex wait_for() (asyncio.Condition method)
@anchor{library/asyncio-sync asyncio Condition wait_for}@anchor{29d9}
@deffn {Method} coroutine wait_for (predicate)

Wait until a predicate becomes @emph{true}.

The predicate must be a callable which result will be
interpreted as a boolean value.  The final value is the
return value.
@end deffn
@end deffn

@node Semaphore,BoundedSemaphore,Condition,Synchronization Primitives
@anchor{library/asyncio-sync semaphore}@anchor{29da}
@subsubsection Semaphore


@geindex Semaphore (class in asyncio)
@anchor{library/asyncio-sync asyncio Semaphore}@anchor{58e}
@deffn {Class} asyncio.Semaphore (value=1)

A Semaphore object.  Not thread-safe.

A semaphore manages an internal counter which is decremented by each
@ref{29db,,acquire()} call and incremented by each @ref{29dc,,release()} call.
The counter can never go below zero; when @ref{29db,,acquire()} finds
that it is zero, it blocks, waiting until some task calls
@ref{29dc,,release()}.

The optional @emph{value} argument gives the initial value for the
internal counter (@code{1} by default). If the given value is
less than @code{0} a @ref{17c,,ValueError} is raised.


Changed in version 3.10: Removed the @emph{loop} parameter.


The preferred way to use a Semaphore is an @ref{169,,async with}
statement:

@example
sem = asyncio.Semaphore(10)

# ... later
async with sem:
    # work with shared resource
@end example

which is equivalent to:

@example
sem = asyncio.Semaphore(10)

# ... later
await sem.acquire()
try:
    # work with shared resource
finally:
    sem.release()
@end example

@geindex acquire() (asyncio.Semaphore method)
@anchor{library/asyncio-sync asyncio Semaphore acquire}@anchor{29db}
@deffn {Method} coroutine acquire ()

Acquire a semaphore.

If the internal counter is greater than zero, decrement
it by one and return @code{True} immediately.  If it is zero, wait
until a @ref{29dc,,release()} is called and return @code{True}.
@end deffn

@geindex locked() (asyncio.Semaphore method)
@anchor{library/asyncio-sync asyncio Semaphore locked}@anchor{29dd}
@deffn {Method} locked ()

Returns @code{True} if semaphore can not be acquired immediately.
@end deffn

@geindex release() (asyncio.Semaphore method)
@anchor{library/asyncio-sync asyncio Semaphore release}@anchor{29dc}
@deffn {Method} release ()

Release a semaphore, incrementing the internal counter by one.
Can wake up a task waiting to acquire the semaphore.

Unlike @ref{58f,,BoundedSemaphore}, @ref{58e,,Semaphore} allows
making more @code{release()} calls than @code{acquire()} calls.
@end deffn
@end deffn

@node BoundedSemaphore,,Semaphore,Synchronization Primitives
@anchor{library/asyncio-sync boundedsemaphore}@anchor{29de}
@subsubsection BoundedSemaphore


@geindex BoundedSemaphore (class in asyncio)
@anchor{library/asyncio-sync asyncio BoundedSemaphore}@anchor{58f}
@deffn {Class} asyncio.BoundedSemaphore (value=1)

A bounded semaphore object.  Not thread-safe.

Bounded Semaphore is a version of @ref{58e,,Semaphore} that raises
a @ref{17c,,ValueError} in @ref{29dc,,release()} if it
increases the internal counter above the initial @emph{value}.


Changed in version 3.10: Removed the @emph{loop} parameter.

@end deffn


__________________________________________________________________



Changed in version 3.9: Acquiring a lock using @code{await lock} or @code{yield from lock} and/or
@ref{168,,with} statement (@code{with await lock}, @code{with (yield from
lock)}) was removed.  Use @code{async with lock} instead.


@node Subprocesses,Queues,Synchronization Primitives,asyncio — Asynchronous I/O
@anchor{library/asyncio-subprocess doc}@anchor{29df}@anchor{library/asyncio-subprocess asyncio-subprocess}@anchor{2979}@anchor{library/asyncio-subprocess subprocesses}@anchor{29e0}
@subsubsection Subprocesses


@strong{Source code:} Lib/asyncio/subprocess.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/subprocess.py},
Lib/asyncio/base_subprocess.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/base_subprocess.py}


__________________________________________________________________


This section describes high-level async/await asyncio APIs to
create and manage subprocesses.
@anchor{library/asyncio-subprocess asyncio-example-subprocess-shell}@anchor{29e1}
Here’s an example of how asyncio can run a shell command and
obtain its result:

@example
import asyncio

async def run(cmd):
    proc = await asyncio.create_subprocess_shell(
        cmd,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE)

    stdout, stderr = await proc.communicate()

    print(f'[@{cmd!r@} exited with @{proc.returncode@}]')
    if stdout:
        print(f'[stdout]\n@{stdout.decode()@}')
    if stderr:
        print(f'[stderr]\n@{stderr.decode()@}')

asyncio.run(run('ls /zzz'))
@end example

will print:

@example
['ls /zzz' exited with 1]
[stderr]
ls: /zzz: No such file or directory
@end example

Because all asyncio subprocess functions are asynchronous and asyncio
provides many tools to work with such functions, it is easy to execute
and monitor multiple subprocesses in parallel.  It is indeed trivial
to modify the above example to run several commands simultaneously:

@example
async def main():
    await asyncio.gather(
        run('ls /zzz'),
        run('sleep 1; echo "hello"'))

asyncio.run(main())
@end example

See also the @ref{29e2,,Examples} subsection.

@menu
* Creating Subprocesses:: 
* Constants: Constants<7>. 
* Interacting with Subprocesses:: 

@end menu

@node Creating Subprocesses,Constants<7>,,Subprocesses
@anchor{library/asyncio-subprocess creating-subprocesses}@anchor{29e3}
@subsubsection Creating Subprocesses


@geindex create_subprocess_exec() (in module asyncio)
@anchor{library/asyncio-subprocess asyncio create_subprocess_exec}@anchor{591}
@deffn {Function} coroutine asyncio.create_subprocess_exec (program, *args, stdin=None, stdout=None, stderr=None, limit=None, **kwds)

Create a subprocess.

The @emph{limit} argument sets the buffer limit for @ref{31d,,StreamReader}
wrappers for @code{Process.stdout} and @code{Process.stderr}
(if @ref{6b8,,subprocess.PIPE} is passed to @emph{stdout} and @emph{stderr} arguments).

Return a @ref{29e4,,Process} instance.

See the documentation of @ref{297c,,loop.subprocess_exec()} for other
parameters.


Changed in version 3.10: Removed the @emph{loop} parameter.

@end deffn

@geindex create_subprocess_shell() (in module asyncio)
@anchor{library/asyncio-subprocess asyncio create_subprocess_shell}@anchor{592}
@deffn {Function} coroutine asyncio.create_subprocess_shell (cmd, stdin=None, stdout=None, stderr=None, limit=None, **kwds)

Run the @emph{cmd} shell command.

The @emph{limit} argument sets the buffer limit for @ref{31d,,StreamReader}
wrappers for @code{Process.stdout} and @code{Process.stderr}
(if @ref{6b8,,subprocess.PIPE} is passed to @emph{stdout} and @emph{stderr} arguments).

Return a @ref{29e4,,Process} instance.

See the documentation of @ref{29e5,,loop.subprocess_shell()} for other
parameters.

@cartouche
@quotation Important 
It is the application’s responsibility to ensure that all whitespace and
special characters are quoted appropriately to avoid shell injection@footnote{https://en.wikipedia.org/wiki/Shell_injection#Shell_injection}
vulnerabilities. The @ref{cbf,,shlex.quote()} function can be used to properly
escape whitespace and special shell characters in strings that are going
to be used to construct shell commands.
@end quotation
@end cartouche


Changed in version 3.10: Removed the @emph{loop} parameter.

@end deffn

@cartouche
@quotation Note 
Subprocesses are available for Windows if a @ref{4da,,ProactorEventLoop} is
used. See @ref{29e6,,Subprocess Support on Windows}
for details.
@end quotation
@end cartouche


@subsubheading See also


asyncio also has the following @emph{low-level} APIs to work with subprocesses:
@ref{297c,,loop.subprocess_exec()}, @ref{29e5,,loop.subprocess_shell()},
@ref{29e7,,loop.connect_read_pipe()}, @ref{29e8,,loop.connect_write_pipe()},
as well as the @ref{29e9,,Subprocess Transports}
and @ref{29ea,,Subprocess Protocols}.


@node Constants<7>,Interacting with Subprocesses,Creating Subprocesses,Subprocesses
@anchor{library/asyncio-subprocess constants}@anchor{29eb}
@subsubsection Constants


@geindex asyncio.subprocess.PIPE (in module asyncio)
@anchor{library/asyncio-subprocess asyncio asyncio subprocess PIPE}@anchor{29ec}
@deffn {Data} asyncio.subprocess.PIPE

Can be passed to the @emph{stdin}, @emph{stdout} or @emph{stderr} parameters.

If @emph{PIPE} is passed to @emph{stdin} argument, the
@ref{29ed,,Process.stdin} attribute
will point to a @ref{1299,,StreamWriter} instance.

If @emph{PIPE} is passed to @emph{stdout} or @emph{stderr} arguments, the
@ref{29ee,,Process.stdout} and
@ref{29ef,,Process.stderr}
attributes will point to @ref{31d,,StreamReader} instances.
@end deffn

@geindex asyncio.subprocess.STDOUT (in module asyncio)
@anchor{library/asyncio-subprocess asyncio asyncio subprocess STDOUT}@anchor{29f0}
@deffn {Data} asyncio.subprocess.STDOUT

Special value that can be used as the @emph{stderr} argument and indicates
that standard error should be redirected into standard output.
@end deffn

@geindex asyncio.subprocess.DEVNULL (in module asyncio)
@anchor{library/asyncio-subprocess asyncio asyncio subprocess DEVNULL}@anchor{29f1}
@deffn {Data} asyncio.subprocess.DEVNULL

Special value that can be used as the @emph{stdin}, @emph{stdout} or @emph{stderr} argument
to process creation functions.  It indicates that the special file
@ref{12e2,,os.devnull} will be used for the corresponding subprocess stream.
@end deffn

@node Interacting with Subprocesses,,Constants<7>,Subprocesses
@anchor{library/asyncio-subprocess interacting-with-subprocesses}@anchor{29f2}
@subsubsection Interacting with Subprocesses


Both @ref{591,,create_subprocess_exec()} and @ref{592,,create_subprocess_shell()}
functions return instances of the @emph{Process} class.  @emph{Process} is a high-level
wrapper that allows communicating with subprocesses and watching for
their completion.

@geindex asyncio.subprocess.Process (class in asyncio)
@anchor{library/asyncio-subprocess asyncio asyncio subprocess Process}@anchor{29e4}
@deffn {Class} asyncio.subprocess.Process

An object that wraps OS processes created by the
@ref{591,,create_subprocess_exec()} and @ref{592,,create_subprocess_shell()}
functions.

This class is designed to have a similar API to the
@ref{5ae,,subprocess.Popen} class, but there are some
notable differences:


@itemize *

@item 
unlike Popen, Process instances do not have an equivalent to
the @ref{2909,,poll()} method;

@item 
the @ref{29f3,,communicate()} and
@ref{29f4,,wait()} methods don’t have a
@emph{timeout} parameter: use the @ref{3d5,,wait_for()} function;

@item 
the @ref{29f4,,Process.wait()} method
is asynchronous, whereas @ref{82d,,subprocess.Popen.wait()} method
is implemented as a blocking busy loop;

@item 
the @emph{universal_newlines} parameter is not supported.
@end itemize

This class is @ref{2993,,not thread safe}.

See also the @ref{29f5,,Subprocess and Threads}
section.

@geindex wait() (asyncio.asyncio.subprocess.Process method)
@anchor{library/asyncio-subprocess asyncio asyncio subprocess Process wait}@anchor{29f4}
@deffn {Method} coroutine wait ()

Wait for the child process to terminate.

Set and return the @ref{29f6,,returncode} attribute.

@cartouche
@quotation Note 
This method can deadlock when using @code{stdout=PIPE} or
@code{stderr=PIPE} and the child process generates so much output
that it blocks waiting for the OS pipe buffer to accept
more data. Use the @ref{29f3,,communicate()} method when using pipes
to avoid this condition.
@end quotation
@end cartouche
@end deffn

@geindex communicate() (asyncio.asyncio.subprocess.Process method)
@anchor{library/asyncio-subprocess asyncio asyncio subprocess Process communicate}@anchor{29f3}
@deffn {Method} coroutine communicate (input=None)

Interact with process:


@enumerate 

@item 
send data to @emph{stdin} (if @emph{input} is not @code{None});

@item 
read data from @emph{stdout} and @emph{stderr}, until EOF is reached;

@item 
wait for process to terminate.
@end enumerate

The optional @emph{input} argument is the data (@ref{2f0,,bytes} object)
that will be sent to the child process.

Return a tuple @code{(stdout_data, stderr_data)}.

If either @ref{bf2,,BrokenPipeError} or @ref{bf5,,ConnectionResetError}
exception is raised when writing @emph{input} into @emph{stdin}, the
exception is ignored.  This condition occurs when the process
exits before all data are written into @emph{stdin}.

If it is desired to send data to the process’ @emph{stdin},
the process needs to be created with @code{stdin=PIPE}.  Similarly,
to get anything other than @code{None} in the result tuple, the
process has to be created with @code{stdout=PIPE} and/or
@code{stderr=PIPE} arguments.

Note, that the data read is buffered in memory, so do not use
this method if the data size is large or unlimited.
@end deffn

@geindex send_signal() (asyncio.asyncio.subprocess.Process method)
@anchor{library/asyncio-subprocess asyncio asyncio subprocess Process send_signal}@anchor{29f7}
@deffn {Method} send_signal (signal)

Sends the signal @emph{signal} to the child process.

@cartouche
@quotation Note 
On Windows, @code{SIGTERM} is an alias for @ref{29f8,,terminate()}.
@code{CTRL_C_EVENT} and @code{CTRL_BREAK_EVENT} can be sent to processes
started with a @emph{creationflags} parameter which includes
@code{CREATE_NEW_PROCESS_GROUP}.
@end quotation
@end cartouche
@end deffn

@geindex terminate() (asyncio.asyncio.subprocess.Process method)
@anchor{library/asyncio-subprocess asyncio asyncio subprocess Process terminate}@anchor{29f8}
@deffn {Method} terminate ()

Stop the child process.

On POSIX systems this method sends @ref{29f9,,signal.SIGTERM} to the
child process.

On Windows the Win32 API function @code{TerminateProcess()} is
called to stop the child process.
@end deffn

@geindex kill() (asyncio.asyncio.subprocess.Process method)
@anchor{library/asyncio-subprocess asyncio asyncio subprocess Process kill}@anchor{29fa}
@deffn {Method} kill ()

Kill the child process.

On POSIX systems this method sends @code{SIGKILL} to the child
process.

On Windows this method is an alias for @ref{29f8,,terminate()}.
@end deffn

@geindex stdin (asyncio.asyncio.subprocess.Process attribute)
@anchor{library/asyncio-subprocess asyncio asyncio subprocess Process stdin}@anchor{29ed}
@deffn {Attribute} stdin

Standard input stream (@ref{1299,,StreamWriter}) or @code{None}
if the process was created with @code{stdin=None}.
@end deffn

@geindex stdout (asyncio.asyncio.subprocess.Process attribute)
@anchor{library/asyncio-subprocess asyncio asyncio subprocess Process stdout}@anchor{29ee}
@deffn {Attribute} stdout

Standard output stream (@ref{31d,,StreamReader}) or @code{None}
if the process was created with @code{stdout=None}.
@end deffn

@geindex stderr (asyncio.asyncio.subprocess.Process attribute)
@anchor{library/asyncio-subprocess asyncio asyncio subprocess Process stderr}@anchor{29ef}
@deffn {Attribute} stderr

Standard error stream (@ref{31d,,StreamReader}) or @code{None}
if the process was created with @code{stderr=None}.
@end deffn

@cartouche
@quotation Warning 
Use the @ref{29f3,,communicate()} method rather than
@ref{29ed,,process.stdin.write()},
@ref{29ee,,await process.stdout.read()} or
@ref{29ef,,await process.stderr.read}.
This avoids deadlocks due to streams pausing reading or writing
and blocking the child process.
@end quotation
@end cartouche

@geindex pid (asyncio.asyncio.subprocess.Process attribute)
@anchor{library/asyncio-subprocess asyncio asyncio subprocess Process pid}@anchor{29fb}
@deffn {Attribute} pid

Process identification number (PID).

Note that for processes created by the @ref{592,,create_subprocess_shell()}
function, this attribute is the PID of the spawned shell.
@end deffn

@geindex returncode (asyncio.asyncio.subprocess.Process attribute)
@anchor{library/asyncio-subprocess asyncio asyncio subprocess Process returncode}@anchor{29f6}
@deffn {Attribute} returncode

Return code of the process when it exits.

A @code{None} value indicates that the process has not terminated yet.

A negative value @code{-N} indicates that the child was terminated
by signal @code{N} (POSIX only).
@end deffn
@end deffn

@menu
* Subprocess and Threads:: 
* Examples: Examples<14>. 

@end menu

@node Subprocess and Threads,Examples<14>,,Interacting with Subprocesses
@anchor{library/asyncio-subprocess asyncio-subprocess-threads}@anchor{29f5}@anchor{library/asyncio-subprocess subprocess-and-threads}@anchor{29fc}
@subsubsection Subprocess and Threads


Standard asyncio event loop supports running subprocesses from different threads by
default.

On Windows subprocesses are provided by @ref{4da,,ProactorEventLoop} only (default),
@ref{29fd,,SelectorEventLoop} has no subprocess support.

On UNIX @emph{child watchers} are used for subprocess finish waiting, see
@ref{29fe,,Process Watchers} for more info.


Changed in version 3.8: UNIX switched to use @ref{29ff,,ThreadedChildWatcher} for spawning subprocesses from
different threads without any limitation.

Spawning a subprocess with @emph{inactive} current child watcher raises
@ref{5af,,RuntimeError}.


Note that alternative event loop implementations might have own limitations;
please refer to their documentation.


@subsubheading See also


The @ref{2993,,Concurrency and multithreading in asyncio} section.


@node Examples<14>,,Subprocess and Threads,Interacting with Subprocesses
@anchor{library/asyncio-subprocess examples}@anchor{29e2}
@subsubsection Examples


An example using the @ref{29e4,,Process} class to
control a subprocess and the @ref{31d,,StreamReader} class to read from
its standard output.
@anchor{library/asyncio-subprocess asyncio-example-create-subprocess-exec}@anchor{2a00}
The subprocess is created by the @ref{591,,create_subprocess_exec()}
function:

@example
import asyncio
import sys

async def get_date():
    code = 'import datetime; print(datetime.datetime.now())'

    # Create the subprocess; redirect the standard output
    # into a pipe.
    proc = await asyncio.create_subprocess_exec(
        sys.executable, '-c', code,
        stdout=asyncio.subprocess.PIPE)

    # Read one line of output.
    data = await proc.stdout.readline()
    line = data.decode('ascii').rstrip()

    # Wait for the subprocess exit.
    await proc.wait()
    return line

date = asyncio.run(get_date())
print(f"Current date: @{date@}")
@end example

See also the @ref{2a01,,same example}
written using low-level APIs.

@node Queues,Exceptions<10>,Subprocesses,asyncio — Asynchronous I/O
@anchor{library/asyncio-queue doc}@anchor{2a02}@anchor{library/asyncio-queue asyncio-queues}@anchor{297a}@anchor{library/asyncio-queue queues}@anchor{2a03}
@subsubsection Queues


@strong{Source code:} Lib/asyncio/queues.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/queues.py}


__________________________________________________________________


asyncio queues are designed to be similar to classes of the
@ref{d5,,queue} module.  Although asyncio queues are not thread-safe,
they are designed to be used specifically in async/await code.

Note that methods of asyncio queues don’t have a @emph{timeout} parameter;
use @ref{3d5,,asyncio.wait_for()} function to do queue operations with a
timeout.

See also the @ref{2a04,,Examples} section below.

@menu
* Queue:: 
* Priority Queue:: 
* LIFO Queue:: 
* Exceptions: Exceptions<9>. 
* Examples: Examples<15>. 

@end menu

@node Queue,Priority Queue,,Queues
@anchor{library/asyncio-queue queue}@anchor{2a05}
@subsubsection Queue


@geindex Queue (class in asyncio)
@anchor{library/asyncio-queue asyncio Queue}@anchor{590}
@deffn {Class} asyncio.Queue (maxsize=0)

A first in, first out (FIFO) queue.

If @emph{maxsize} is less than or equal to zero, the queue size is
infinite.  If it is an integer greater than @code{0}, then
@code{await put()} blocks when the queue reaches @emph{maxsize}
until an item is removed by @ref{2a06,,get()}.

Unlike the standard library threading @ref{d5,,queue}, the size of
the queue is always known and can be returned by calling the
@ref{2a07,,qsize()} method.


Changed in version 3.10: Removed the @emph{loop} parameter.


This class is @ref{2993,,not thread safe}.

@geindex maxsize (asyncio.Queue attribute)
@anchor{library/asyncio-queue asyncio Queue maxsize}@anchor{2a08}
@deffn {Attribute} maxsize

Number of items allowed in the queue.
@end deffn

@geindex empty() (asyncio.Queue method)
@anchor{library/asyncio-queue asyncio Queue empty}@anchor{2a09}
@deffn {Method} empty ()

Return @code{True} if the queue is empty, @code{False} otherwise.
@end deffn

@geindex full() (asyncio.Queue method)
@anchor{library/asyncio-queue asyncio Queue full}@anchor{2a0a}
@deffn {Method} full ()

Return @code{True} if there are @ref{2a08,,maxsize} items in the queue.

If the queue was initialized with @code{maxsize=0} (the default),
then @ref{2a0a,,full()} never returns @code{True}.
@end deffn

@geindex get() (asyncio.Queue method)
@anchor{library/asyncio-queue asyncio Queue get}@anchor{2a06}
@deffn {Method} coroutine get ()

Remove and return an item from the queue. If queue is empty,
wait until an item is available.
@end deffn

@geindex get_nowait() (asyncio.Queue method)
@anchor{library/asyncio-queue asyncio Queue get_nowait}@anchor{2a0b}
@deffn {Method} get_nowait ()

Return an item if one is immediately available, else raise
@ref{2a0c,,QueueEmpty}.
@end deffn

@geindex join() (asyncio.Queue method)
@anchor{library/asyncio-queue asyncio Queue join}@anchor{926}
@deffn {Method} coroutine join ()

Block until all items in the queue have been received and processed.

The count of unfinished tasks goes up whenever an item is added
to the queue. The count goes down whenever a consumer coroutine calls
@ref{927,,task_done()} to indicate that the item was retrieved and all
work on it is complete.  When the count of unfinished tasks drops
to zero, @ref{926,,join()} unblocks.
@end deffn

@geindex put() (asyncio.Queue method)
@anchor{library/asyncio-queue asyncio Queue put}@anchor{2a0d}
@deffn {Method} coroutine put (item)

Put an item into the queue. If the queue is full, wait until a
free slot is available before adding the item.
@end deffn

@geindex put_nowait() (asyncio.Queue method)
@anchor{library/asyncio-queue asyncio Queue put_nowait}@anchor{2a0e}
@deffn {Method} put_nowait (item)

Put an item into the queue without blocking.

If no free slot is immediately available, raise @ref{2a0f,,QueueFull}.
@end deffn

@geindex qsize() (asyncio.Queue method)
@anchor{library/asyncio-queue asyncio Queue qsize}@anchor{2a07}
@deffn {Method} qsize ()

Return the number of items in the queue.
@end deffn

@geindex task_done() (asyncio.Queue method)
@anchor{library/asyncio-queue asyncio Queue task_done}@anchor{927}
@deffn {Method} task_done ()

Indicate that a formerly enqueued task is complete.

Used by queue consumers. For each @ref{2a06,,get()} used to
fetch a task, a subsequent call to @ref{927,,task_done()} tells the
queue that the processing on the task is complete.

If a @ref{926,,join()} is currently blocking, it will resume when all
items have been processed (meaning that a @ref{927,,task_done()}
call was received for every item that had been @ref{2a0d,,put()}
into the queue).

Raises @ref{17c,,ValueError} if called more times than there were
items placed in the queue.
@end deffn
@end deffn

@node Priority Queue,LIFO Queue,Queue,Queues
@anchor{library/asyncio-queue priority-queue}@anchor{2a10}
@subsubsection Priority Queue


@geindex PriorityQueue (class in asyncio)
@anchor{library/asyncio-queue asyncio PriorityQueue}@anchor{2a11}
@deffn {Class} asyncio.PriorityQueue

A variant of @ref{590,,Queue}; retrieves entries in priority order
(lowest first).

Entries are typically tuples of the form
@code{(priority_number, data)}.
@end deffn

@node LIFO Queue,Exceptions<9>,Priority Queue,Queues
@anchor{library/asyncio-queue lifo-queue}@anchor{2a12}
@subsubsection LIFO Queue


@geindex LifoQueue (class in asyncio)
@anchor{library/asyncio-queue asyncio LifoQueue}@anchor{2a13}
@deffn {Class} asyncio.LifoQueue

A variant of @ref{590,,Queue} that retrieves most recently added
entries first (last in, first out).
@end deffn

@node Exceptions<9>,Examples<15>,LIFO Queue,Queues
@anchor{library/asyncio-queue exceptions}@anchor{2a14}
@subsubsection Exceptions


@geindex QueueEmpty
@anchor{library/asyncio-queue asyncio QueueEmpty}@anchor{2a0c}
@deffn {Exception} asyncio.QueueEmpty

This exception is raised when the @ref{2a0b,,get_nowait()} method
is called on an empty queue.
@end deffn

@geindex QueueFull
@anchor{library/asyncio-queue asyncio QueueFull}@anchor{2a0f}
@deffn {Exception} asyncio.QueueFull

Exception raised when the @ref{2a0e,,put_nowait()} method is called
on a queue that has reached its @emph{maxsize}.
@end deffn

@node Examples<15>,,Exceptions<9>,Queues
@anchor{library/asyncio-queue examples}@anchor{2a04}
@subsubsection Examples

@anchor{library/asyncio-queue asyncio-example-queue-dist}@anchor{2a15}
Queues can be used to distribute workload between several
concurrent tasks:

@example
import asyncio
import random
import time


async def worker(name, queue):
    while True:
        # Get a "work item" out of the queue.
        sleep_for = await queue.get()

        # Sleep for the "sleep_for" seconds.
        await asyncio.sleep(sleep_for)

        # Notify the queue that the "work item" has been processed.
        queue.task_done()

        print(f'@{name@} has slept for @{sleep_for:.2f@} seconds')


async def main():
    # Create a queue that we will use to store our "workload".
    queue = asyncio.Queue()

    # Generate random timings and put them into the queue.
    total_sleep_time = 0
    for _ in range(20):
        sleep_for = random.uniform(0.05, 1.0)
        total_sleep_time += sleep_for
        queue.put_nowait(sleep_for)

    # Create three worker tasks to process the queue concurrently.
    tasks = []
    for i in range(3):
        task = asyncio.create_task(worker(f'worker-@{i@}', queue))
        tasks.append(task)

    # Wait until the queue is fully processed.
    started_at = time.monotonic()
    await queue.join()
    total_slept_for = time.monotonic() - started_at

    # Cancel our worker tasks.
    for task in tasks:
        task.cancel()
    # Wait until all worker tasks are cancelled.
    await asyncio.gather(*tasks, return_exceptions=True)

    print('====')
    print(f'3 workers slept in parallel for @{total_slept_for:.2f@} seconds')
    print(f'total expected sleep time: @{total_sleep_time:.2f@} seconds')


asyncio.run(main())
@end example

@node Exceptions<10>,Event Loop,Queues,asyncio — Asynchronous I/O
@anchor{library/asyncio-exceptions doc}@anchor{2a16}@anchor{library/asyncio-exceptions asyncio-exceptions}@anchor{2a17}@anchor{library/asyncio-exceptions exceptions}@anchor{2a18}
@subsubsection Exceptions


@strong{Source code:} Lib/asyncio/exceptions.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/exceptions.py}


__________________________________________________________________


@geindex TimeoutError
@anchor{library/asyncio-exceptions asyncio TimeoutError}@anchor{2a19}
@deffn {Exception} asyncio.TimeoutError

A deprecated alias of @ref{2b8,,TimeoutError},
raised when the operation has exceeded the given deadline.


Changed in version 3.11: This class was made an alias of @ref{2b8,,TimeoutError}.

@end deffn

@geindex CancelledError
@anchor{library/asyncio-exceptions asyncio CancelledError}@anchor{4d7}
@deffn {Exception} asyncio.CancelledError

The operation has been cancelled.

This exception can be caught to perform custom operations
when asyncio Tasks are cancelled.  In almost all situations the
exception must be re-raised.


Changed in version 3.8: @ref{4d7,,CancelledError} is now a subclass of @ref{15f,,BaseException}.

@end deffn

@geindex InvalidStateError
@anchor{library/asyncio-exceptions asyncio InvalidStateError}@anchor{299d}
@deffn {Exception} asyncio.InvalidStateError

Invalid internal state of @ref{317,,Task} or @ref{316,,Future}.

Can be raised in situations like setting a result value for a
@emph{Future} object that already has a result value set.
@end deffn

@geindex SendfileNotAvailableError
@anchor{library/asyncio-exceptions asyncio SendfileNotAvailableError}@anchor{2a1a}
@deffn {Exception} asyncio.SendfileNotAvailableError

The “sendfile” syscall is not available for the given
socket or file type.

A subclass of @ref{5af,,RuntimeError}.
@end deffn

@geindex IncompleteReadError
@anchor{library/asyncio-exceptions asyncio IncompleteReadError}@anchor{1198}
@deffn {Exception} asyncio.IncompleteReadError

The requested read operation did not complete fully.

Raised by the @ref{2978,,asyncio stream APIs}.

This exception is a subclass of @ref{e74,,EOFError}.

@geindex expected (asyncio.IncompleteReadError attribute)
@anchor{library/asyncio-exceptions asyncio IncompleteReadError expected}@anchor{2a1b}
@deffn {Attribute} expected

The total number (@ref{1b1,,int}) of expected bytes.
@end deffn

@geindex partial (asyncio.IncompleteReadError attribute)
@anchor{library/asyncio-exceptions asyncio IncompleteReadError partial}@anchor{29ae}
@deffn {Attribute} partial

A string of @ref{2f0,,bytes} read before the end of stream was reached.
@end deffn
@end deffn

@geindex LimitOverrunError
@anchor{library/asyncio-exceptions asyncio LimitOverrunError}@anchor{29af}
@deffn {Exception} asyncio.LimitOverrunError

Reached the buffer size limit while looking for a separator.

Raised by the @ref{2978,,asyncio stream APIs}.

@geindex consumed (asyncio.LimitOverrunError attribute)
@anchor{library/asyncio-exceptions asyncio LimitOverrunError consumed}@anchor{2a1c}
@deffn {Attribute} consumed

The total number of to be consumed bytes.
@end deffn
@end deffn

@node Event Loop,Futures,Exceptions<10>,asyncio — Asynchronous I/O
@anchor{library/asyncio-eventloop doc}@anchor{2a1d}@anchor{library/asyncio-eventloop event-loop}@anchor{2a1e}
@subsubsection Event Loop


@strong{Source code:} Lib/asyncio/events.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/events.py},
Lib/asyncio/base_events.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/base_events.py}


__________________________________________________________________


@subsubheading Preface


The event loop is the core of every asyncio application.
Event loops run asynchronous tasks and callbacks, perform network
IO operations, and run subprocesses.

Application developers should typically use the high-level asyncio functions,
such as @ref{3d1,,asyncio.run()}, and should rarely need to reference the loop
object or call its methods.  This section is intended mostly for authors
of lower-level code, libraries, and frameworks, who need finer control over
the event loop behavior.

@subsubheading Obtaining the Event Loop


The following low-level functions can be used to get, set, or create
an event loop:

@geindex get_running_loop() (in module asyncio)
@anchor{library/asyncio-eventloop asyncio get_running_loop}@anchor{315}
@deffn {Function} asyncio.get_running_loop ()

Return the running event loop in the current OS thread.

If there is no running event loop a @ref{5af,,RuntimeError} is raised.
This function can only be called from a coroutine or a callback.


New in version 3.7.

@end deffn

@geindex get_event_loop() (in module asyncio)
@anchor{library/asyncio-eventloop asyncio get_event_loop}@anchor{314}
@deffn {Function} asyncio.get_event_loop ()

Get the current event loop.

If there is no current event loop set in the current OS thread,
the OS thread is main, and @ref{2a1f,,set_event_loop()} has not yet
been called, asyncio will create a new event loop and set it as the
current one.

Because this function has rather complex behavior (especially
when custom event loop policies are in use), using the
@ref{315,,get_running_loop()} function is preferred to @ref{314,,get_event_loop()}
in coroutines and callbacks.

Consider also using the @ref{3d1,,asyncio.run()} function instead of using
lower level functions to manually create and close an event loop.


Deprecated since version 3.10: Deprecation warning is emitted if there is no running event loop.
In future Python releases, this function will be an alias of
@ref{315,,get_running_loop()}.

@end deffn

@geindex set_event_loop() (in module asyncio)
@anchor{library/asyncio-eventloop asyncio set_event_loop}@anchor{2a1f}
@deffn {Function} asyncio.set_event_loop (loop)

Set @emph{loop} as a current event loop for the current OS thread.
@end deffn

@geindex new_event_loop() (in module asyncio)
@anchor{library/asyncio-eventloop asyncio new_event_loop}@anchor{2a20}
@deffn {Function} asyncio.new_event_loop ()

Create and return a new event loop object.
@end deffn

Note that the behaviour of @ref{314,,get_event_loop()}, @ref{2a1f,,set_event_loop()},
and @ref{2a20,,new_event_loop()} functions can be altered by
@ref{2a21,,setting a custom event loop policy}.

@subsubheading Contents


This documentation page contains the following sections:


@itemize *

@item 
The @ref{2a22,,Event Loop Methods} section is the reference documentation of
the event loop APIs;

@item 
The @ref{2a23,,Callback Handles} section documents the @ref{1364,,Handle} and
@ref{2a24,,TimerHandle} instances which are returned from scheduling
methods such as @ref{622,,loop.call_soon()} and @ref{624,,loop.call_later()};

@item 
The @ref{2a25,,Server Objects} section documents types returned from
event loop methods like @ref{630,,loop.create_server()};

@item 
The @ref{2a26,,Event Loop Implementations} section documents the
@ref{29fd,,SelectorEventLoop} and @ref{4da,,ProactorEventLoop} classes;

@item 
The @ref{2a27,,Examples} section showcases how to work with some event
loop APIs.
@end itemize

@menu
* Event Loop Methods:: 
* Callback Handles:: 
* Server Objects:: 
* Event Loop Implementations:: 
* Examples: Examples<16>. 

@end menu

@node Event Loop Methods,Callback Handles,,Event Loop
@anchor{library/asyncio-eventloop asyncio-event-loop}@anchor{8d1}@anchor{library/asyncio-eventloop event-loop-methods}@anchor{2a22}
@subsubsection Event Loop Methods


Event loops have @strong{low-level} APIs for the following:

@menu
* Running and stopping the loop:: 
* Scheduling callbacks:: 
* Scheduling delayed callbacks:: 
* Creating Futures and Tasks:: 
* Opening network connections:: 
* Creating network servers:: 
* Transferring files:: 
* TLS Upgrade:: 
* Watching file descriptors:: 
* Working with socket objects directly:: 
* DNS:: 
* Working with pipes:: 
* Unix signals:: 
* Executing code in thread or process pools:: 
* Error Handling API:: 
* Enabling debug mode:: 
* Running Subprocesses:: 

@end menu

@node Running and stopping the loop,Scheduling callbacks,,Event Loop Methods
@anchor{library/asyncio-eventloop running-and-stopping-the-loop}@anchor{2a28}
@subsubsection Running and stopping the loop


@geindex run_until_complete() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop run_until_complete}@anchor{7bc}
@deffn {Method} loop.run_until_complete (future)

Run until the @emph{future} (an instance of @ref{316,,Future}) has
completed.

If the argument is a @ref{2977,,coroutine object} it
is implicitly scheduled to run as a @ref{317,,asyncio.Task}.

Return the Future’s result or raise its exception.
@end deffn

@geindex run_forever() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop run_forever}@anchor{2a29}
@deffn {Method} loop.run_forever ()

Run the event loop until @ref{7c3,,stop()} is called.

If @ref{7c3,,stop()} is called before @ref{2a29,,run_forever()} is called,
the loop will poll the I/O selector once with a timeout of zero,
run all callbacks scheduled in response to I/O events (and
those that were already scheduled), and then exit.

If @ref{7c3,,stop()} is called while @ref{2a29,,run_forever()} is running,
the loop will run the current batch of callbacks and then exit.
Note that new callbacks scheduled by callbacks will not run in this
case; instead, they will run the next time @ref{2a29,,run_forever()} or
@ref{7bc,,run_until_complete()} is called.
@end deffn

@geindex stop() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop stop}@anchor{7c3}
@deffn {Method} loop.stop ()

Stop the event loop.
@end deffn

@geindex is_running() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop is_running}@anchor{2a2a}
@deffn {Method} loop.is_running ()

Return @code{True} if the event loop is currently running.
@end deffn

@geindex is_closed() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop is_closed}@anchor{922}
@deffn {Method} loop.is_closed ()

Return @code{True} if the event loop was closed.
@end deffn

@geindex close() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop close}@anchor{2a2b}
@deffn {Method} loop.close ()

Close the event loop.

The loop must not be running when this function is called.
Any pending callbacks will be discarded.

This method clears all queues and shuts down the executor, but does
not wait for the executor to finish.

This method is idempotent and irreversible.  No other methods
should be called after the event loop is closed.
@end deffn

@geindex shutdown_asyncgens() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop shutdown_asyncgens}@anchor{7c4}
@deffn {Method} coroutine loop.shutdown_asyncgens ()

Schedule all currently open @ref{18ac,,asynchronous generator} objects to
close with an @ref{3bd,,aclose()} call.  After calling this method,
the event loop will issue a warning if a new asynchronous generator
is iterated. This should be used to reliably finalize all scheduled
asynchronous generators.

Note that there is no need to call this function when
@ref{3d1,,asyncio.run()} is used.

Example:

@example
try:
    loop.run_forever()
finally:
    loop.run_until_complete(loop.shutdown_asyncgens())
    loop.close()
@end example


New in version 3.6.

@end deffn

@geindex shutdown_default_executor() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop shutdown_default_executor}@anchor{3d0}
@deffn {Method} coroutine loop.shutdown_default_executor ()

Schedule the closure of the default executor and wait for it to join all of
the threads in the @code{ThreadPoolExecutor}. After calling this method, a
@ref{5af,,RuntimeError} will be raised if @ref{3d4,,loop.run_in_executor()} is called
while using the default executor.

Note that there is no need to call this function when
@ref{3d1,,asyncio.run()} is used.


New in version 3.9.

@end deffn

@node Scheduling callbacks,Scheduling delayed callbacks,Running and stopping the loop,Event Loop Methods
@anchor{library/asyncio-eventloop scheduling-callbacks}@anchor{2a2c}
@subsubsection Scheduling callbacks


@geindex call_soon() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop call_soon}@anchor{622}
@deffn {Method} loop.call_soon (callback, *args, context=None)

Schedule the @emph{callback} @ref{2a2d,,callback} to be called with
@emph{args} arguments at the next iteration of the event loop.

Callbacks are called in the order in which they are registered.
Each callback will be called exactly once.

An optional keyword-only @emph{context} argument allows specifying a
custom @ref{2943,,contextvars.Context} for the @emph{callback} to run in.
The current context is used when no @emph{context} is provided.

An instance of @ref{1364,,asyncio.Handle} is returned, which can be
used later to cancel the callback.

This method is not thread-safe.
@end deffn

@geindex call_soon_threadsafe() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop call_soon_threadsafe}@anchor{623}
@deffn {Method} loop.call_soon_threadsafe (callback, *args, context=None)

A thread-safe variant of @ref{622,,call_soon()}.  Must be used to
schedule callbacks @emph{from another thread}.

Raises @ref{5af,,RuntimeError} if called on a loop that’s been closed.
This can happen on a secondary thread when the main application is
shutting down.

See the @ref{2993,,concurrency and multithreading}
section of the documentation.
@end deffn


Changed in version 3.7: The @emph{context} keyword-only parameter was added. See 
@geindex Python Enhancement Proposals; PEP 567
PEP 567@footnote{https://www.python.org/dev/peps/pep-0567}
for more details.

@anchor{library/asyncio-eventloop asyncio-pass-keywords}@anchor{2a2e}
@cartouche
@quotation Note 
Most @ref{9,,asyncio} scheduling functions don’t allow passing
keyword arguments.  To do that, use @ref{a3f,,functools.partial()}:

@example
# will schedule "print("Hello", flush=True)"
loop.call_soon(
    functools.partial(print, "Hello", flush=True))
@end example

Using partial objects is usually more convenient than using lambdas,
as asyncio can render partial objects better in debug and error
messages.
@end quotation
@end cartouche

@node Scheduling delayed callbacks,Creating Futures and Tasks,Scheduling callbacks,Event Loop Methods
@anchor{library/asyncio-eventloop asyncio-delayed-calls}@anchor{2a2f}@anchor{library/asyncio-eventloop scheduling-delayed-callbacks}@anchor{2a30}
@subsubsection Scheduling delayed callbacks


Event loop provides mechanisms to schedule callback functions
to be called at some point in the future.  Event loop uses monotonic
clocks to track time.

@geindex call_later() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop call_later}@anchor{624}
@deffn {Method} loop.call_later (delay, callback, *args, context=None)

Schedule @emph{callback} to be called after the given @emph{delay}
number of seconds (can be either an int or a float).

An instance of @ref{2a24,,asyncio.TimerHandle} is returned which can
be used to cancel the callback.

@emph{callback} will be called exactly once.  If two callbacks are
scheduled for exactly the same time, the order in which they
are called is undefined.

The optional positional @emph{args} will be passed to the callback when
it is called. If you want the callback to be called with keyword
arguments use @ref{a3f,,functools.partial()}.

An optional keyword-only @emph{context} argument allows specifying a
custom @ref{2943,,contextvars.Context} for the @emph{callback} to run in.
The current context is used when no @emph{context} is provided.


Changed in version 3.7: The @emph{context} keyword-only parameter was added. See 
@geindex Python Enhancement Proposals; PEP 567
PEP 567@footnote{https://www.python.org/dev/peps/pep-0567}
for more details.



Changed in version 3.8: In Python 3.7 and earlier with the default event loop implementation,
the @emph{delay} could not exceed one day.
This has been fixed in Python 3.8.

@end deffn

@geindex call_at() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop call_at}@anchor{625}
@deffn {Method} loop.call_at (when, callback, *args, context=None)

Schedule @emph{callback} to be called at the given absolute timestamp
@emph{when} (an int or a float), using the same time reference as
@ref{2a31,,loop.time()}.

This method’s behavior is the same as @ref{624,,call_later()}.

An instance of @ref{2a24,,asyncio.TimerHandle} is returned which can
be used to cancel the callback.


Changed in version 3.7: The @emph{context} keyword-only parameter was added. See 
@geindex Python Enhancement Proposals; PEP 567
PEP 567@footnote{https://www.python.org/dev/peps/pep-0567}
for more details.



Changed in version 3.8: In Python 3.7 and earlier with the default event loop implementation,
the difference between @emph{when} and the current time could not exceed
one day.  This has been fixed in Python 3.8.

@end deffn

@geindex time() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop time}@anchor{2a31}
@deffn {Method} loop.time ()

Return the current time, as a @ref{256,,float} value, according to
the event loop’s internal monotonic clock.
@end deffn

@cartouche
@quotation Note 

Changed in version 3.8: In Python 3.7 and earlier timeouts (relative @emph{delay} or absolute @emph{when})
should not exceed one day.  This has been fixed in Python 3.8.

@end quotation
@end cartouche


@subsubheading See also


The @ref{58a,,asyncio.sleep()} function.


@node Creating Futures and Tasks,Opening network connections,Scheduling delayed callbacks,Event Loop Methods
@anchor{library/asyncio-eventloop creating-futures-and-tasks}@anchor{2a32}
@subsubsection Creating Futures and Tasks


@geindex create_future() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop create_future}@anchor{7bf}
@deffn {Method} loop.create_future ()

Create an @ref{316,,asyncio.Future} object attached to the event loop.

This is the preferred way to create Futures in asyncio. This lets
third-party event loops provide alternative implementations of
the Future object (with better performance or instrumentation).


New in version 3.5.2.

@end deffn

@geindex create_task() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop create_task}@anchor{4dd}
@deffn {Method} loop.create_task (coro, *, name=None)

Schedule the execution of a @ref{2977,,Coroutines}.
Return a @ref{317,,Task} object.

Third-party event loops can use their own subclass of @ref{317,,Task}
for interoperability. In this case, the result type is a subclass
of @ref{317,,Task}.

If the @emph{name} argument is provided and not @code{None}, it is set as
the name of the task using @ref{4de,,Task.set_name()}.


Changed in version 3.8: Added the @emph{name} parameter.

@end deffn

@geindex set_task_factory() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop set_task_factory}@anchor{924}
@deffn {Method} loop.set_task_factory (factory)

Set a task factory that will be used by
@ref{4dd,,loop.create_task()}.

If @emph{factory} is @code{None} the default task factory will be set.
Otherwise, @emph{factory} must be a @emph{callable} with the signature matching
@code{(loop, coro)}, where @emph{loop} is a reference to the active
event loop, and @emph{coro} is a coroutine object.  The callable
must return a @ref{316,,asyncio.Future}-compatible object.
@end deffn

@geindex get_task_factory() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop get_task_factory}@anchor{925}
@deffn {Method} loop.get_task_factory ()

Return a task factory or @code{None} if the default one is in use.
@end deffn

@node Opening network connections,Creating network servers,Creating Futures and Tasks,Event Loop Methods
@anchor{library/asyncio-eventloop opening-network-connections}@anchor{2a33}
@subsubsection Opening network connections


@geindex create_connection() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop create_connection}@anchor{4e0}
@deffn {Method} coroutine loop.create_connection (protocol_factory, host=None, port=None, *, ssl=None, family=0, proto=0, flags=0, sock=None, local_addr=None, server_hostname=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None, happy_eyeballs_delay=None, interleave=None)

Open a streaming transport connection to a given
address specified by @emph{host} and @emph{port}.

The socket family can be either @ref{2a34,,AF_INET} or
@ref{2a35,,AF_INET6} depending on @emph{host} (or the @emph{family}
argument, if provided).

The socket type will be @ref{e91,,SOCK_STREAM}.

@emph{protocol_factory} must be a callable returning an
@ref{2a36,,asyncio protocol} implementation.

This method will try to establish the connection in the background.
When successful, it returns a @code{(transport, protocol)} pair.

The chronological synopsis of the underlying operation is as follows:


@enumerate 

@item 
The connection is established and a @ref{2a37,,transport}
is created for it.

@item 
@emph{protocol_factory} is called without arguments and is expected to
return a @ref{2a36,,protocol} instance.

@item 
The protocol instance is coupled with the transport by calling its
@ref{2a38,,connection_made()} method.

@item 
A @code{(transport, protocol)} tuple is returned on success.
@end enumerate

The created transport is an implementation-dependent bidirectional
stream.

Other arguments:


@itemize *

@item 
@emph{ssl}: if given and not false, a SSL/TLS transport is created
(by default a plain TCP transport is created).  If @emph{ssl} is
a @ref{2bf,,ssl.SSLContext} object, this context is used to create
the transport; if @emph{ssl} is @ref{748,,True}, a default context returned
from @ref{b29,,ssl.create_default_context()} is used.


@subsubheading See also


@ref{2a39,,SSL/TLS security considerations}


@item 
@emph{server_hostname} sets or overrides the hostname that the target
server’s certificate will be matched against.  Should only be passed
if @emph{ssl} is not @code{None}.  By default the value of the @emph{host} argument
is used.  If @emph{host} is empty, there is no default and you must pass a
value for @emph{server_hostname}.  If @emph{server_hostname} is an empty
string, hostname matching is disabled (which is a serious security
risk, allowing for potential man-in-the-middle attacks).

@item 
@emph{family}, @emph{proto}, @emph{flags} are the optional address family, protocol
and flags to be passed through to getaddrinfo() for @emph{host} resolution.
If given, these should all be integers from the corresponding
@ref{ea,,socket} module constants.

@item 
@emph{happy_eyeballs_delay}, if given, enables Happy Eyeballs for this
connection. It should
be a floating-point number representing the amount of time in seconds
to wait for a connection attempt to complete, before starting the next
attempt in parallel. This is the “Connection Attempt Delay” as defined
in 
@geindex RFC; RFC 8305
RFC 8305@footnote{https://tools.ietf.org/html/rfc8305.html}. A sensible default value recommended by the RFC is @code{0.25}
(250 milliseconds).

@item 
@emph{interleave} controls address reordering when a host name resolves to
multiple IP addresses.
If @code{0} or unspecified, no reordering is done, and addresses are
tried in the order returned by @ref{750,,getaddrinfo()}. If a positive integer
is specified, the addresses are interleaved by address family, and the
given integer is interpreted as “First Address Family Count” as defined
in 
@geindex RFC; RFC 8305
RFC 8305@footnote{https://tools.ietf.org/html/rfc8305.html}. The default is @code{0} if @emph{happy_eyeballs_delay} is not
specified, and @code{1} if it is.

@item 
@emph{sock}, if given, should be an existing, already connected
@ref{8fd,,socket.socket} object to be used by the transport.
If @emph{sock} is given, none of @emph{host}, @emph{port}, @emph{family}, @emph{proto}, @emph{flags},
@emph{happy_eyeballs_delay}, @emph{interleave}
and @emph{local_addr} should be specified.

@item 
@emph{local_addr}, if given, is a @code{(local_host, local_port)} tuple used
to bind the socket locally.  The @emph{local_host} and @emph{local_port}
are looked up using @code{getaddrinfo()}, similarly to @emph{host} and @emph{port}.

@item 
@emph{ssl_handshake_timeout} is (for a TLS connection) the time in seconds
to wait for the TLS handshake to complete before aborting the connection.
@code{60.0} seconds if @code{None} (default).

@item 
@emph{ssl_shutdown_timeout} is the time in seconds to wait for the SSL shutdown
to complete before aborting the connection. @code{30.0} seconds if @code{None}
(default).
@end itemize


Changed in version 3.5: Added support for SSL/TLS in @ref{4da,,ProactorEventLoop}.



Changed in version 3.6: The socket option @code{TCP_NODELAY} is set by default
for all TCP connections.



Changed in version 3.7: Added the @emph{ssl_handshake_timeout} parameter.



Changed in version 3.8: Added the @emph{happy_eyeballs_delay} and @emph{interleave} parameters.

Happy Eyeballs Algorithm: Success with Dual-Stack Hosts.
When a server’s IPv4 path and protocol are working, but the server’s
IPv6 path and protocol are not working, a dual-stack client
application experiences significant connection delay compared to an
IPv4-only client.  This is undesirable because it causes the dual-
stack client to have a worse user experience.  This document
specifies requirements for algorithms that reduce this user-visible
delay and provides an algorithm.

For more information: @indicateurl{https://tools.ietf.org/html/rfc6555}



Changed in version 3.11: Added the @emph{ssl_shutdown_timeout} parameter.



@subsubheading See also


The @ref{636,,open_connection()} function is a high-level alternative
API.  It returns a pair of (@ref{31d,,StreamReader}, @ref{1299,,StreamWriter})
that can be used directly in async/await code.

@end deffn

@geindex create_datagram_endpoint() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop create_datagram_endpoint}@anchor{1b9}
@deffn {Method} coroutine loop.create_datagram_endpoint (protocol_factory, local_addr=None, remote_addr=None, *, family=0, proto=0, flags=0, reuse_port=None, allow_broadcast=None, sock=None)

Create a datagram connection.

The socket family can be either @ref{2a34,,AF_INET},
@ref{2a35,,AF_INET6}, or @ref{2a3a,,AF_UNIX},
depending on @emph{host} (or the @emph{family} argument, if provided).

The socket type will be @ref{e90,,SOCK_DGRAM}.

@emph{protocol_factory} must be a callable returning a
@ref{2a36,,protocol} implementation.

A tuple of @code{(transport, protocol)} is returned on success.

Other arguments:


@itemize *

@item 
@emph{local_addr}, if given, is a @code{(local_host, local_port)} tuple used
to bind the socket locally.  The @emph{local_host} and @emph{local_port}
are looked up using @ref{750,,getaddrinfo()}.

@item 
@emph{remote_addr}, if given, is a @code{(remote_host, remote_port)} tuple used
to connect the socket to a remote address.  The @emph{remote_host} and
@emph{remote_port} are looked up using @ref{750,,getaddrinfo()}.

@item 
@emph{family}, @emph{proto}, @emph{flags} are the optional address family, protocol
and flags to be passed through to @ref{750,,getaddrinfo()} for @emph{host}
resolution. If given, these should all be integers from the
corresponding @ref{ea,,socket} module constants.

@item 
@emph{reuse_port} tells the kernel to allow this endpoint to be bound to the
same port as other existing endpoints are bound to, so long as they all
set this flag when being created. This option is not supported on Windows
and some Unixes. If the @code{SO_REUSEPORT} constant is not
defined then this capability is unsupported.

@item 
@emph{allow_broadcast} tells the kernel to allow this endpoint to send
messages to the broadcast address.

@item 
@emph{sock} can optionally be specified in order to use a preexisting,
already connected, @ref{8fd,,socket.socket} object to be used by the
transport. If specified, @emph{local_addr} and @emph{remote_addr} should be omitted
(must be @ref{182,,None}).
@end itemize

See @ref{2a3b,,UDP echo client protocol} and
@ref{2a3c,,UDP echo server protocol} examples.


Changed in version 3.4.4: The @emph{family}, @emph{proto}, @emph{flags}, @emph{reuse_address}, @emph{reuse_port},
@emph{allow_broadcast}, and @emph{sock} parameters were added.



Changed in version 3.8.1: The @emph{reuse_address} parameter is no longer supported, as using
@code{SO_REUSEADDR} poses a significant security concern for
UDP. Explicitly passing @code{reuse_address=True} will raise an exception.

When multiple processes with differing UIDs assign sockets to an
identical UDP socket address with @code{SO_REUSEADDR}, incoming packets can
become randomly distributed among the sockets.

For supported platforms, @emph{reuse_port} can be used as a replacement for
similar functionality. With @emph{reuse_port},
@code{SO_REUSEPORT} is used instead, which specifically
prevents processes with differing UIDs from assigning sockets to the same
socket address.



Changed in version 3.8: Added support for Windows.



Changed in version 3.11: The @emph{reuse_address} parameter, disabled since Python 3.9.0, 3.8.1,
3.7.6 and 3.6.10, has been entirely removed.

@end deffn

@geindex create_unix_connection() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop create_unix_connection}@anchor{29a9}
@deffn {Method} coroutine loop.create_unix_connection (protocol_factory, path=None, *, ssl=None, sock=None, server_hostname=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None)

Create a Unix connection.

The socket family will be @ref{2a3a,,AF_UNIX}; socket
type will be @ref{e91,,SOCK_STREAM}.

A tuple of @code{(transport, protocol)} is returned on success.

@emph{path} is the name of a Unix domain socket and is required,
unless a @emph{sock} parameter is specified.  Abstract Unix sockets,
@ref{2ef,,str}, @ref{2f0,,bytes}, and @ref{522,,Path} paths are
supported.

See the documentation of the @ref{4e0,,loop.create_connection()} method
for information about arguments to this method.

@ref{1629,,Availability}: Unix.


Changed in version 3.7: Added the @emph{ssl_handshake_timeout} parameter.
The @emph{path} parameter can now be a @ref{63d,,path-like object}.



Changed in version 3.11: Added the @emph{ssl_shutdown_timeout} parameter.

@end deffn

@node Creating network servers,Transferring files,Opening network connections,Event Loop Methods
@anchor{library/asyncio-eventloop creating-network-servers}@anchor{2a3d}
@subsubsection Creating network servers


@geindex create_server() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop create_server}@anchor{630}
@deffn {Method} coroutine loop.create_server (protocol_factory, host=None, port=None, *, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None, reuse_port=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None, start_serving=True)

Create a TCP server (socket type @ref{e91,,SOCK_STREAM}) listening
on @emph{port} of the @emph{host} address.

Returns a @ref{62f,,Server} object.

Arguments:


@itemize *

@item 
@emph{protocol_factory} must be a callable returning a
@ref{2a36,,protocol} implementation.

@item 
The @emph{host} parameter can be set to several types which determine where
the server would be listening:


@itemize -

@item 
If @emph{host} is a string, the TCP server is bound to a single network
interface specified by @emph{host}.

@item 
If @emph{host} is a sequence of strings, the TCP server is bound to all
network interfaces specified by the sequence.

@item 
If @emph{host} is an empty string or @code{None}, all interfaces are
assumed and a list of multiple sockets will be returned (most likely
one for IPv4 and another one for IPv6).
@end itemize

@item 
The @emph{port} parameter can be set to specify which port the server should
listen on. If @code{0} or @code{None} (the default), a random unused port will
be selected (note that if @emph{host} resolves to multiple network interfaces,
a different random port will be selected for each interface).

@item 
@emph{family} can be set to either @ref{2a34,,socket.AF_INET} or
@ref{2a35,,AF_INET6} to force the socket to use IPv4 or IPv6.
If not set, the @emph{family} will be determined from host name
(defaults to @code{AF_UNSPEC}).

@item 
@emph{flags} is a bitmask for @ref{750,,getaddrinfo()}.

@item 
@emph{sock} can optionally be specified in order to use a preexisting
socket object. If specified, @emph{host} and @emph{port} must not be specified.

@item 
@emph{backlog} is the maximum number of queued connections passed to
@ref{9ca,,listen()} (defaults to 100).

@item 
@emph{ssl} can be set to an @ref{2bf,,SSLContext} instance to enable
TLS over the accepted connections.

@item 
@emph{reuse_address} tells the kernel to reuse a local socket in
@code{TIME_WAIT} state, without waiting for its natural timeout to
expire. If not specified will automatically be set to @code{True} on
Unix.

@item 
@emph{reuse_port} tells the kernel to allow this endpoint to be bound to the
same port as other existing endpoints are bound to, so long as they all
set this flag when being created. This option is not supported on
Windows.

@item 
@emph{ssl_handshake_timeout} is (for a TLS server) the time in seconds to wait
for the TLS handshake to complete before aborting the connection.
@code{60.0} seconds if @code{None} (default).

@item 
@emph{ssl_shutdown_timeout} is the time in seconds to wait for the SSL shutdown
to complete before aborting the connection. @code{30.0} seconds if @code{None}
(default).

@item 
@emph{start_serving} set to @code{True} (the default) causes the created server
to start accepting connections immediately.  When set to @code{False},
the user should await on @ref{632,,Server.start_serving()} or
@ref{633,,Server.serve_forever()} to make the server to start accepting
connections.
@end itemize


Changed in version 3.5: Added support for SSL/TLS in @ref{4da,,ProactorEventLoop}.



Changed in version 3.5.1: The @emph{host} parameter can be a sequence of strings.



Changed in version 3.6: Added @emph{ssl_handshake_timeout} and @emph{start_serving} parameters.
The socket option @code{TCP_NODELAY} is set by default
for all TCP connections.



Changed in version 3.11: Added the @emph{ssl_shutdown_timeout} parameter.



@subsubheading See also


The @ref{637,,start_server()} function is a higher-level alternative API
that returns a pair of @ref{31d,,StreamReader} and @ref{1299,,StreamWriter}
that can be used in an async/await code.

@end deffn

@geindex create_unix_server() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop create_unix_server}@anchor{631}
@deffn {Method} coroutine loop.create_unix_server (protocol_factory, path=None, *, sock=None, backlog=100, ssl=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None, start_serving=True)

Similar to @ref{630,,loop.create_server()} but works with the
@ref{2a3a,,AF_UNIX} socket family.

@emph{path} is the name of a Unix domain socket, and is required,
unless a @emph{sock} argument is provided.  Abstract Unix sockets,
@ref{2ef,,str}, @ref{2f0,,bytes}, and @ref{522,,Path} paths
are supported.

See the documentation of the @ref{630,,loop.create_server()} method
for information about arguments to this method.

@ref{1629,,Availability}: Unix.


Changed in version 3.7: Added the @emph{ssl_handshake_timeout} and @emph{start_serving} parameters.
The @emph{path} parameter can now be a @ref{522,,Path} object.



Changed in version 3.11: Added the @emph{ssl_shutdown_timeout} parameter.

@end deffn

@geindex connect_accepted_socket() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop connect_accepted_socket}@anchor{638}
@deffn {Method} coroutine loop.connect_accepted_socket (protocol_factory, sock, *, ssl=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None)

Wrap an already accepted connection into a transport/protocol pair.

This method can be used by servers that accept connections outside
of asyncio but that use asyncio to handle them.

Parameters:


@itemize *

@item 
@emph{protocol_factory} must be a callable returning a
@ref{2a36,,protocol} implementation.

@item 
@emph{sock} is a preexisting socket object returned from
@ref{8fe,,socket.accept}.

@item 
@emph{ssl} can be set to an @ref{2bf,,SSLContext} to enable SSL over
the accepted connections.

@item 
@emph{ssl_handshake_timeout} is (for an SSL connection) the time in seconds to
wait for the SSL handshake to complete before aborting the connection.
@code{60.0} seconds if @code{None} (default).

@item 
@emph{ssl_shutdown_timeout} is the time in seconds to wait for the SSL shutdown
to complete before aborting the connection. @code{30.0} seconds if @code{None}
(default).
@end itemize

Returns a @code{(transport, protocol)} pair.


New in version 3.5.3.



Changed in version 3.7: Added the @emph{ssl_handshake_timeout} parameter.



Changed in version 3.11: Added the @emph{ssl_shutdown_timeout} parameter.

@end deffn

@node Transferring files,TLS Upgrade,Creating network servers,Event Loop Methods
@anchor{library/asyncio-eventloop transferring-files}@anchor{2a3e}
@subsubsection Transferring files


@geindex sendfile() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop sendfile}@anchor{2a3f}
@deffn {Method} coroutine loop.sendfile (transport, file, offset=0, count=None, *, fallback=True)

Send a @emph{file} over a @emph{transport}.  Return the total number of bytes
sent.

The method uses high-performance @ref{62c,,os.sendfile()} if available.

@emph{file} must be a regular file object opened in binary mode.

@emph{offset} tells from where to start reading the file. If specified,
@emph{count} is the total number of bytes to transmit as opposed to
sending the file until EOF is reached. File position is always updated,
even when this method raises an error, and
@ref{2232,,file.tell()} can be used to obtain the actual
number of bytes sent.

@emph{fallback} set to @code{True} makes asyncio to manually read and send
the file when the platform does not support the sendfile system call
(e.g. Windows or SSL socket on Unix).

Raise @ref{2a1a,,SendfileNotAvailableError} if the system does not support
the @emph{sendfile} syscall and @emph{fallback} is @code{False}.


New in version 3.7.

@end deffn

@node TLS Upgrade,Watching file descriptors,Transferring files,Event Loop Methods
@anchor{library/asyncio-eventloop tls-upgrade}@anchor{2a40}
@subsubsection TLS Upgrade


@geindex start_tls() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop start_tls}@anchor{627}
@deffn {Method} coroutine loop.start_tls (transport, protocol, sslcontext, *, server_side=False, server_hostname=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None)

Upgrade an existing transport-based connection to TLS.

Return a new transport instance, that the @emph{protocol} must start using
immediately after the @emph{await}.  The @emph{transport} instance passed to
the @emph{start_tls} method should never be used again.

Parameters:


@itemize *

@item 
@emph{transport} and @emph{protocol} instances that methods like
@ref{630,,create_server()} and
@ref{4e0,,create_connection()} return.

@item 
@emph{sslcontext}: a configured instance of @ref{2bf,,SSLContext}.

@item 
@emph{server_side} pass @code{True} when a server-side connection is being
upgraded (like the one created by @ref{630,,create_server()}).

@item 
@emph{server_hostname}: sets or overrides the host name that the target
server’s certificate will be matched against.

@item 
@emph{ssl_handshake_timeout} is (for a TLS connection) the time in seconds to
wait for the TLS handshake to complete before aborting the connection.
@code{60.0} seconds if @code{None} (default).

@item 
@emph{ssl_shutdown_timeout} is the time in seconds to wait for the SSL shutdown
to complete before aborting the connection. @code{30.0} seconds if @code{None}
(default).
@end itemize


New in version 3.7.



Changed in version 3.11: Added the @emph{ssl_shutdown_timeout} parameter.

@end deffn

@node Watching file descriptors,Working with socket objects directly,TLS Upgrade,Event Loop Methods
@anchor{library/asyncio-eventloop watching-file-descriptors}@anchor{2a41}
@subsubsection Watching file descriptors


@geindex add_reader() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop add_reader}@anchor{29c5}
@deffn {Method} loop.add_reader (fd, callback, *args)

Start monitoring the @emph{fd} file descriptor for read availability and
invoke @emph{callback} with the specified arguments once @emph{fd} is available for
reading.
@end deffn

@geindex remove_reader() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop remove_reader}@anchor{2a42}
@deffn {Method} loop.remove_reader (fd)

Stop monitoring the @emph{fd} file descriptor for read availability.
@end deffn

@geindex add_writer() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop add_writer}@anchor{2a43}
@deffn {Method} loop.add_writer (fd, callback, *args)

Start monitoring the @emph{fd} file descriptor for write availability and
invoke @emph{callback} with the specified arguments once @emph{fd} is available for
writing.

Use @ref{a3f,,functools.partial()} @ref{2a2e,,to pass keyword arguments} to @emph{callback}.
@end deffn

@geindex remove_writer() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop remove_writer}@anchor{2a44}
@deffn {Method} loop.remove_writer (fd)

Stop monitoring the @emph{fd} file descriptor for write availability.
@end deffn

See also @ref{2a45,,Platform Support} section
for some limitations of these methods.

@node Working with socket objects directly,DNS,Watching file descriptors,Event Loop Methods
@anchor{library/asyncio-eventloop working-with-socket-objects-directly}@anchor{2a46}
@subsubsection Working with socket objects directly


In general, protocol implementations that use transport-based APIs
such as @ref{4e0,,loop.create_connection()} and @ref{630,,loop.create_server()}
are faster than implementations that work with sockets directly.
However, there are some use cases when performance is not critical, and
working with @ref{8fd,,socket} objects directly is more
convenient.

@geindex sock_recv() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop sock_recv}@anchor{74d}
@deffn {Method} coroutine loop.sock_recv (sock, nbytes)

Receive up to @emph{nbytes} from @emph{sock}.  Asynchronous version of
@ref{900,,socket.recv()}.

Return the received data as a bytes object.

@emph{sock} must be a non-blocking socket.


Changed in version 3.7: Even though this method was always documented as a coroutine
method, releases before Python 3.7 returned a @ref{316,,Future}.
Since Python 3.7 this is an @code{async def} method.

@end deffn

@geindex sock_recv_into() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop sock_recv_into}@anchor{628}
@deffn {Method} coroutine loop.sock_recv_into (sock, buf)

Receive data from @emph{sock} into the @emph{buf} buffer.  Modeled after the blocking
@ref{eb4,,socket.recv_into()} method.

Return the number of bytes written to the buffer.

@emph{sock} must be a non-blocking socket.


New in version 3.7.

@end deffn

@geindex sock_sendall() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop sock_sendall}@anchor{74e}
@deffn {Method} coroutine loop.sock_sendall (sock, data)

Send @emph{data} to the @emph{sock} socket. Asynchronous version of
@ref{904,,socket.sendall()}.

This method continues to send to the socket until either all data
in @emph{data} has been sent or an error occurs.  @code{None} is returned
on success.  On error, an exception is raised. Additionally, there is no way
to determine how much data, if any, was successfully processed by the
receiving end of the connection.

@emph{sock} must be a non-blocking socket.


Changed in version 3.7: Even though the method was always documented as a coroutine
method, before Python 3.7 it returned a @ref{316,,Future}.
Since Python 3.7, this is an @code{async def} method.

@end deffn

@geindex sock_connect() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop sock_connect}@anchor{928}
@deffn {Method} coroutine loop.sock_connect (sock, address)

Connect @emph{sock} to a remote socket at @emph{address}.

Asynchronous version of @ref{8ff,,socket.connect()}.

@emph{sock} must be a non-blocking socket.


Changed in version 3.5.2: @code{address} no longer needs to be resolved.  @code{sock_connect}
will try to check if the @emph{address} is already resolved by calling
@ref{465,,socket.inet_pton()}.  If not,
@ref{750,,loop.getaddrinfo()} will be used to resolve the
@emph{address}.



@subsubheading See also


@ref{4e0,,loop.create_connection()}
and  @ref{636,,asyncio.open_connection()}.

@end deffn

@geindex sock_accept() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop sock_accept}@anchor{74f}
@deffn {Method} coroutine loop.sock_accept (sock)

Accept a connection.  Modeled after the blocking
@ref{8fe,,socket.accept()} method.

The socket must be bound to an address and listening
for connections. The return value is a pair @code{(conn, address)} where @emph{conn}
is a @emph{new} socket object usable to send and receive data on the connection,
and @emph{address} is the address bound to the socket on the other end of the
connection.

@emph{sock} must be a non-blocking socket.


Changed in version 3.7: Even though the method was always documented as a coroutine
method, before Python 3.7 it returned a @ref{316,,Future}.
Since Python 3.7, this is an @code{async def} method.



@subsubheading See also


@ref{630,,loop.create_server()} and @ref{637,,start_server()}.

@end deffn

@geindex sock_sendfile() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop sock_sendfile}@anchor{62b}
@deffn {Method} coroutine loop.sock_sendfile (sock, file, offset=0, count=None, *, fallback=True)

Send a file using high-performance @ref{62c,,os.sendfile} if possible.
Return the total number of bytes sent.

Asynchronous version of @ref{9c9,,socket.sendfile()}.

@emph{sock} must be a non-blocking @ref{e91,,socket.SOCK_STREAM}
@ref{8fd,,socket}.

@emph{file} must be a regular file object open in binary mode.

@emph{offset} tells from where to start reading the file. If specified,
@emph{count} is the total number of bytes to transmit as opposed to
sending the file until EOF is reached. File position is always updated,
even when this method raises an error, and
@ref{2232,,file.tell()} can be used to obtain the actual
number of bytes sent.

@emph{fallback}, when set to @code{True}, makes asyncio manually read and send
the file when the platform does not support the sendfile syscall
(e.g. Windows or SSL socket on Unix).

Raise @ref{2a1a,,SendfileNotAvailableError} if the system does not support
@emph{sendfile} syscall and @emph{fallback} is @code{False}.

@emph{sock} must be a non-blocking socket.


New in version 3.7.

@end deffn

@node DNS,Working with pipes,Working with socket objects directly,Event Loop Methods
@anchor{library/asyncio-eventloop dns}@anchor{2a47}
@subsubsection DNS


@geindex getaddrinfo() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop getaddrinfo}@anchor{750}
@deffn {Method} coroutine loop.getaddrinfo (host, port, *, family=0, type=0, proto=0, flags=0)

Asynchronous version of @ref{11cc,,socket.getaddrinfo()}.
@end deffn

@geindex getnameinfo() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop getnameinfo}@anchor{751}
@deffn {Method} coroutine loop.getnameinfo (sockaddr, flags=0)

Asynchronous version of @ref{2a48,,socket.getnameinfo()}.
@end deffn


Changed in version 3.7: Both @emph{getaddrinfo} and @emph{getnameinfo} methods were always documented
to return a coroutine, but prior to Python 3.7 they were, in fact,
returning @ref{316,,asyncio.Future} objects.  Starting with Python 3.7
both methods are coroutines.


@node Working with pipes,Unix signals,DNS,Event Loop Methods
@anchor{library/asyncio-eventloop working-with-pipes}@anchor{2a49}
@subsubsection Working with pipes


@geindex connect_read_pipe() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop connect_read_pipe}@anchor{29e7}
@deffn {Method} coroutine loop.connect_read_pipe (protocol_factory, pipe)

Register the read end of @emph{pipe} in the event loop.

@emph{protocol_factory} must be a callable returning an
@ref{2a36,,asyncio protocol} implementation.

@emph{pipe} is a @ref{d68,,file-like object}.

Return pair @code{(transport, protocol)}, where @emph{transport} supports
the @ref{2a4a,,ReadTransport} interface and @emph{protocol} is an object
instantiated by the @emph{protocol_factory}.

With @ref{29fd,,SelectorEventLoop} event loop, the @emph{pipe} is set to
non-blocking mode.
@end deffn

@geindex connect_write_pipe() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop connect_write_pipe}@anchor{29e8}
@deffn {Method} coroutine loop.connect_write_pipe (protocol_factory, pipe)

Register the write end of @emph{pipe} in the event loop.

@emph{protocol_factory} must be a callable returning an
@ref{2a36,,asyncio protocol} implementation.

@emph{pipe} is @ref{d68,,file-like object}.

Return pair @code{(transport, protocol)}, where @emph{transport} supports
@ref{2a4b,,WriteTransport} interface and @emph{protocol} is an object
instantiated by the @emph{protocol_factory}.

With @ref{29fd,,SelectorEventLoop} event loop, the @emph{pipe} is set to
non-blocking mode.
@end deffn

@cartouche
@quotation Note 
@ref{29fd,,SelectorEventLoop} does not support the above methods on
Windows.  Use @ref{4da,,ProactorEventLoop} instead for Windows.
@end quotation
@end cartouche


@subsubheading See also


The @ref{297c,,loop.subprocess_exec()} and
@ref{29e5,,loop.subprocess_shell()} methods.


@node Unix signals,Executing code in thread or process pools,Working with pipes,Event Loop Methods
@anchor{library/asyncio-eventloop unix-signals}@anchor{2a4c}
@subsubsection Unix signals


@geindex add_signal_handler() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop add_signal_handler}@anchor{297d}
@deffn {Method} loop.add_signal_handler (signum, callback, *args)

Set @emph{callback} as the handler for the @emph{signum} signal.

The callback will be invoked by @emph{loop}, along with other queued callbacks
and runnable coroutines of that event loop. Unlike signal handlers
registered using @ref{cc9,,signal.signal()}, a callback registered with this
function is allowed to interact with the event loop.

Raise @ref{17c,,ValueError} if the signal number is invalid or uncatchable.
Raise @ref{5af,,RuntimeError} if there is a problem setting up the handler.

Use @ref{a3f,,functools.partial()} @ref{2a2e,,to pass keyword arguments} to @emph{callback}.

Like @ref{cc9,,signal.signal()}, this function must be invoked in the main
thread.
@end deffn

@geindex remove_signal_handler() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop remove_signal_handler}@anchor{2a4d}
@deffn {Method} loop.remove_signal_handler (sig)

Remove the handler for the @emph{sig} signal.

Return @code{True} if the signal handler was removed, or @code{False} if
no handler was set for the given signal.

@ref{1629,,Availability}: Unix.
@end deffn


@subsubheading See also


The @ref{e5,,signal} module.


@node Executing code in thread or process pools,Error Handling API,Unix signals,Event Loop Methods
@anchor{library/asyncio-eventloop executing-code-in-thread-or-process-pools}@anchor{2a4e}
@subsubsection Executing code in thread or process pools


@geindex run_in_executor() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop run_in_executor}@anchor{3d4}
@deffn {Method} awaitable loop.run_in_executor (executor, func, *args)

Arrange for @emph{func} to be called in the specified executor.

The @emph{executor} argument should be an @ref{28c0,,concurrent.futures.Executor}
instance. The default executor is used if @emph{executor} is @code{None}.

Example:

@example
import asyncio
import concurrent.futures

def blocking_io():
    # File operations (such as logging) can block the
    # event loop: run them in a thread pool.
    with open('/dev/urandom', 'rb') as f:
        return f.read(100)

def cpu_bound():
    # CPU-bound operations will block the event loop:
    # in general it is preferable to run them in a
    # process pool.
    return sum(i * i for i in range(10 ** 7))

async def main():
    loop = asyncio.get_running_loop()

    ## Options:

    # 1. Run in the default loop's executor:
    result = await loop.run_in_executor(
        None, blocking_io)
    print('default thread pool', result)

    # 2. Run in a custom thread pool:
    with concurrent.futures.ThreadPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, blocking_io)
        print('custom thread pool', result)

    # 3. Run in a custom process pool:
    with concurrent.futures.ProcessPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, cpu_bound)
        print('custom process pool', result)

asyncio.run(main())
@end example

This method returns a @ref{316,,asyncio.Future} object.

Use @ref{a3f,,functools.partial()} @ref{2a2e,,to pass keyword arguments} to @emph{func}.


Changed in version 3.5.3: @ref{3d4,,loop.run_in_executor()} no longer configures the
@code{max_workers} of the thread pool executor it creates, instead
leaving it up to the thread pool executor
(@ref{1c7,,ThreadPoolExecutor}) to set the
default.

@end deffn

@geindex set_default_executor() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop set_default_executor}@anchor{587}
@deffn {Method} loop.set_default_executor (executor)

Set @emph{executor} as the default executor used by @ref{3d4,,run_in_executor()}.
@emph{executor} must be an instance of
@ref{1c7,,ThreadPoolExecutor}.


Changed in version 3.11: @emph{executor} must be an instance of
@ref{1c7,,ThreadPoolExecutor}.

@end deffn

@node Error Handling API,Enabling debug mode,Executing code in thread or process pools,Event Loop Methods
@anchor{library/asyncio-eventloop error-handling-api}@anchor{2a4f}
@subsubsection Error Handling API


Allows customizing how exceptions are handled in the event loop.

@geindex set_exception_handler() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop set_exception_handler}@anchor{2a50}
@deffn {Method} loop.set_exception_handler (handler)

Set @emph{handler} as the new event loop exception handler.

If @emph{handler} is @code{None}, the default exception handler will
be set.  Otherwise, @emph{handler} must be a callable with the signature
matching @code{(loop, context)}, where @code{loop}
is a reference to the active event loop, and @code{context}
is a @code{dict} object containing the details of the exception
(see @ref{2a51,,call_exception_handler()} documentation for details
about context).
@end deffn

@geindex get_exception_handler() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop get_exception_handler}@anchor{7c0}
@deffn {Method} loop.get_exception_handler ()

Return the current exception handler, or @code{None} if no custom
exception handler was set.


New in version 3.5.2.

@end deffn

@geindex default_exception_handler() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop default_exception_handler}@anchor{2a52}
@deffn {Method} loop.default_exception_handler (context)

Default exception handler.

This is called when an exception occurs and no exception
handler is set. This can be called by a custom exception
handler that wants to defer to the default handler behavior.

@emph{context} parameter has the same meaning as in
@ref{2a51,,call_exception_handler()}.
@end deffn

@geindex call_exception_handler() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop call_exception_handler}@anchor{2a51}
@deffn {Method} loop.call_exception_handler (context)

Call the current event loop exception handler.

@emph{context} is a @code{dict} object containing the following keys
(new keys may be introduced in future Python versions):


@itemize *

@item 
‘message’: Error message;

@item 
‘exception’ (optional): Exception object;

@item 
‘future’ (optional): @ref{316,,asyncio.Future} instance;

@item 
‘task’ (optional): @ref{317,,asyncio.Task} instance;

@item 
‘handle’ (optional): @ref{1364,,asyncio.Handle} instance;

@item 
‘protocol’ (optional): @ref{2a36,,Protocol} instance;

@item 
‘transport’ (optional): @ref{2a37,,Transport} instance;

@item 
‘socket’ (optional): @ref{8fd,,socket.socket} instance;

@item 

@table @asis

@item ‘asyncgen’ (optional): Asynchronous generator that caused

the exception.
@end table
@end itemize

@cartouche
@quotation Note 
This method should not be overloaded in subclassed
event loops.  For custom exception handling, use
the @ref{2a50,,set_exception_handler()} method.
@end quotation
@end cartouche
@end deffn

@node Enabling debug mode,Running Subprocesses,Error Handling API,Event Loop Methods
@anchor{library/asyncio-eventloop enabling-debug-mode}@anchor{2a53}
@subsubsection Enabling debug mode


@geindex get_debug() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop get_debug}@anchor{921}
@deffn {Method} loop.get_debug ()

Get the debug mode (@ref{2f7,,bool}) of the event loop.

The default value is @code{True} if the environment variable
@geindex PYTHONASYNCIODEBUG
@geindex environment variable; PYTHONASYNCIODEBUG
@ref{1624,,PYTHONASYNCIODEBUG} is set to a non-empty string, @code{False}
otherwise.
@end deffn

@geindex set_debug() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop set_debug}@anchor{920}
@deffn {Method} loop.set_debug (enabled: bool)

Set the debug mode of the event loop.


Changed in version 3.7: The new @ref{3ac,,Python Development Mode} can now also be used
to enable the debug mode.

@end deffn


@subsubheading See also


The @ref{1625,,debug mode of asyncio}.


@node Running Subprocesses,,Enabling debug mode,Event Loop Methods
@anchor{library/asyncio-eventloop running-subprocesses}@anchor{2a54}
@subsubsection Running Subprocesses


Methods described in this subsections are low-level.  In regular
async/await code consider using the high-level
@ref{592,,asyncio.create_subprocess_shell()} and
@ref{591,,asyncio.create_subprocess_exec()} convenience functions instead.

@cartouche
@quotation Note 
On Windows, the default event loop @ref{4da,,ProactorEventLoop} supports
subprocesses, whereas @ref{29fd,,SelectorEventLoop} does not. See
@ref{29e6,,Subprocess Support on Windows} for
details.
@end quotation
@end cartouche

@geindex subprocess_exec() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop subprocess_exec}@anchor{297c}
@deffn {Method} coroutine loop.subprocess_exec (protocol_factory, *args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs)

Create a subprocess from one or more string arguments specified by
@emph{args}.

@emph{args} must be a list of strings represented by:


@itemize *

@item 
@ref{2ef,,str};

@item 
or @ref{2f0,,bytes}, encoded to the
@ref{2377,,filesystem encoding}.
@end itemize

The first string specifies the program executable,
and the remaining strings specify the arguments.  Together, string
arguments form the @code{argv} of the program.

This is similar to the standard library @ref{5ae,,subprocess.Popen}
class called with @code{shell=False} and the list of strings passed as
the first argument; however, where @ref{5ae,,Popen} takes
a single argument which is list of strings, @emph{subprocess_exec}
takes multiple string arguments.

The @emph{protocol_factory} must be a callable returning a subclass of the
@ref{2a55,,asyncio.SubprocessProtocol} class.

Other parameters:


@itemize *

@item 
@emph{stdin} can be any of these:


@itemize *

@item 
a file-like object representing a pipe to be connected to the
subprocess’s standard input stream using
@ref{29e8,,connect_write_pipe()}

@item 
the @ref{6b8,,subprocess.PIPE} constant (default) which will create a new
pipe and connect it,

@item 
the value @code{None} which will make the subprocess inherit the file
descriptor from this process

@item 
the @ref{ce6,,subprocess.DEVNULL} constant which indicates that the
special @ref{12e2,,os.devnull} file will be used
@end itemize

@item 
@emph{stdout} can be any of these:


@itemize *

@item 
a file-like object representing a pipe to be connected to the
subprocess’s standard output stream using
@ref{29e8,,connect_write_pipe()}

@item 
the @ref{6b8,,subprocess.PIPE} constant (default) which will create a new
pipe and connect it,

@item 
the value @code{None} which will make the subprocess inherit the file
descriptor from this process

@item 
the @ref{ce6,,subprocess.DEVNULL} constant which indicates that the
special @ref{12e2,,os.devnull} file will be used
@end itemize

@item 
@emph{stderr} can be any of these:


@itemize *

@item 
a file-like object representing a pipe to be connected to the
subprocess’s standard error stream using
@ref{29e8,,connect_write_pipe()}

@item 
the @ref{6b8,,subprocess.PIPE} constant (default) which will create a new
pipe and connect it,

@item 
the value @code{None} which will make the subprocess inherit the file
descriptor from this process

@item 
the @ref{ce6,,subprocess.DEVNULL} constant which indicates that the
special @ref{12e2,,os.devnull} file will be used

@item 
the @ref{28e4,,subprocess.STDOUT} constant which will connect the standard
error stream to the process’ standard output stream
@end itemize

@item 
All other keyword arguments are passed to @ref{5ae,,subprocess.Popen}
without interpretation, except for @emph{bufsize}, @emph{universal_newlines},
@emph{shell}, @emph{text}, @emph{encoding} and @emph{errors}, which should not be specified
at all.

The @code{asyncio} subprocess API does not support decoding the streams
as text. @ref{3bb,,bytes.decode()} can be used to convert the bytes returned
from the stream to text.
@end itemize

See the constructor of the @ref{5ae,,subprocess.Popen} class
for documentation on other arguments.

Returns a pair of @code{(transport, protocol)}, where @emph{transport}
conforms to the @ref{2a56,,asyncio.SubprocessTransport} base class and
@emph{protocol} is an object instantiated by the @emph{protocol_factory}.
@end deffn

@geindex subprocess_shell() (asyncio.loop method)
@anchor{library/asyncio-eventloop asyncio loop subprocess_shell}@anchor{29e5}
@deffn {Method} coroutine loop.subprocess_shell (protocol_factory, cmd, *, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs)

Create a subprocess from @emph{cmd}, which can be a @ref{2ef,,str} or a
@ref{2f0,,bytes} string encoded to the
@ref{2377,,filesystem encoding},
using the platform’s “shell” syntax.

This is similar to the standard library @ref{5ae,,subprocess.Popen}
class called with @code{shell=True}.

The @emph{protocol_factory} must be a callable returning a subclass of the
@ref{2a55,,SubprocessProtocol} class.

See @ref{297c,,subprocess_exec()} for more details about
the remaining arguments.

Returns a pair of @code{(transport, protocol)}, where @emph{transport}
conforms to the @ref{2a56,,SubprocessTransport} base class and
@emph{protocol} is an object instantiated by the @emph{protocol_factory}.
@end deffn

@cartouche
@quotation Note 
It is the application’s responsibility to ensure that all whitespace
and special characters are quoted appropriately to avoid shell injection@footnote{https://en.wikipedia.org/wiki/Shell_injection#Shell_injection}
vulnerabilities. The @ref{cbf,,shlex.quote()} function can be used to
properly escape whitespace and special characters in strings that
are going to be used to construct shell commands.
@end quotation
@end cartouche

@node Callback Handles,Server Objects,Event Loop Methods,Event Loop
@anchor{library/asyncio-eventloop callback-handles}@anchor{2a23}
@subsubsection Callback Handles


@geindex Handle (class in asyncio)
@anchor{library/asyncio-eventloop asyncio Handle}@anchor{1364}
@deffn {Class} asyncio.Handle

A callback wrapper object returned by @ref{622,,loop.call_soon()},
@ref{623,,loop.call_soon_threadsafe()}.

@geindex cancel() (asyncio.Handle method)
@anchor{library/asyncio-eventloop asyncio Handle cancel}@anchor{2a57}
@deffn {Method} cancel ()

Cancel the callback.  If the callback has already been canceled
or executed, this method has no effect.
@end deffn

@geindex cancelled() (asyncio.Handle method)
@anchor{library/asyncio-eventloop asyncio Handle cancelled}@anchor{639}
@deffn {Method} cancelled ()

Return @code{True} if the callback was cancelled.


New in version 3.7.

@end deffn
@end deffn

@geindex TimerHandle (class in asyncio)
@anchor{library/asyncio-eventloop asyncio TimerHandle}@anchor{2a24}
@deffn {Class} asyncio.TimerHandle

A callback wrapper object returned by @ref{624,,loop.call_later()},
and @ref{625,,loop.call_at()}.

This class is a subclass of @ref{1364,,Handle}.

@geindex when() (asyncio.TimerHandle method)
@anchor{library/asyncio-eventloop asyncio TimerHandle when}@anchor{635}
@deffn {Method} when ()

Return a scheduled callback time as @ref{256,,float} seconds.

The time is an absolute timestamp, using the same time
reference as @ref{2a31,,loop.time()}.


New in version 3.7.

@end deffn
@end deffn

@node Server Objects,Event Loop Implementations,Callback Handles,Event Loop
@anchor{library/asyncio-eventloop server-objects}@anchor{2a25}
@subsubsection Server Objects


Server objects are created by @ref{630,,loop.create_server()},
@ref{631,,loop.create_unix_server()}, @ref{637,,start_server()},
and @ref{29aa,,start_unix_server()} functions.

Do not instantiate the class directly.

@geindex Server (class in asyncio)
@anchor{library/asyncio-eventloop asyncio Server}@anchor{62f}
@deffn {Class} asyncio.Server

@emph{Server} objects are asynchronous context managers.  When used in an
@code{async with} statement, it’s guaranteed that the Server object is
closed and not accepting new connections when the @code{async with}
statement is completed:

@example
srv = await loop.create_server(...)

async with srv:
    # some code

# At this point, srv is closed and no longer accepts new connections.
@end example


Changed in version 3.7: Server object is an asynchronous context manager since Python 3.7.


@geindex close() (asyncio.Server method)
@anchor{library/asyncio-eventloop asyncio Server close}@anchor{2a58}
@deffn {Method} close ()

Stop serving: close listening sockets and set the @ref{752,,sockets}
attribute to @code{None}.

The sockets that represent existing incoming client connections
are left open.

The server is closed asynchronously, use the @ref{2a59,,wait_closed()}
coroutine to wait until the server is closed.
@end deffn

@geindex get_loop() (asyncio.Server method)
@anchor{library/asyncio-eventloop asyncio Server get_loop}@anchor{62e}
@deffn {Method} get_loop ()

Return the event loop associated with the server object.


New in version 3.7.

@end deffn

@geindex start_serving() (asyncio.Server method)
@anchor{library/asyncio-eventloop asyncio Server start_serving}@anchor{632}
@deffn {Method} coroutine start_serving ()

Start accepting connections.

This method is idempotent, so it can be called when
the server is already serving.

The @emph{start_serving} keyword-only parameter to
@ref{630,,loop.create_server()} and
@ref{637,,asyncio.start_server()} allows creating a Server object
that is not accepting connections initially.  In this case
@code{Server.start_serving()}, or @ref{633,,Server.serve_forever()} can be used
to make the Server start accepting connections.


New in version 3.7.

@end deffn

@geindex serve_forever() (asyncio.Server method)
@anchor{library/asyncio-eventloop asyncio Server serve_forever}@anchor{633}
@deffn {Method} coroutine serve_forever ()

Start accepting connections until the coroutine is cancelled.
Cancellation of @code{serve_forever} task causes the server
to be closed.

This method can be called if the server is already accepting
connections.  Only one @code{serve_forever} task can exist per
one @emph{Server} object.

Example:

@example
async def client_connected(reader, writer):
    # Communicate with the client with
    # reader/writer streams.  For example:
    await reader.readline()

async def main(host, port):
    srv = await asyncio.start_server(
        client_connected, host, port)
    await srv.serve_forever()

asyncio.run(main('127.0.0.1', 0))
@end example


New in version 3.7.

@end deffn

@geindex is_serving() (asyncio.Server method)
@anchor{library/asyncio-eventloop asyncio Server is_serving}@anchor{634}
@deffn {Method} is_serving ()

Return @code{True} if the server is accepting new connections.


New in version 3.7.

@end deffn

@geindex wait_closed() (asyncio.Server method)
@anchor{library/asyncio-eventloop asyncio Server wait_closed}@anchor{2a59}
@deffn {Method} coroutine wait_closed ()

Wait until the @ref{2a58,,close()} method completes.
@end deffn

@geindex sockets (asyncio.Server attribute)
@anchor{library/asyncio-eventloop asyncio Server sockets}@anchor{752}
@deffn {Attribute} sockets

List of @ref{8fd,,socket.socket} objects the server is listening on.


Changed in version 3.7: Prior to Python 3.7 @code{Server.sockets} used to return an
internal list of server sockets directly.  In 3.7 a copy
of that list is returned.

@end deffn
@end deffn

@node Event Loop Implementations,Examples<16>,Server Objects,Event Loop
@anchor{library/asyncio-eventloop asyncio-event-loops}@anchor{2a5a}@anchor{library/asyncio-eventloop event-loop-implementations}@anchor{2a26}
@subsubsection Event Loop Implementations


asyncio ships with two different event loop implementations:
@ref{29fd,,SelectorEventLoop} and @ref{4da,,ProactorEventLoop}.

By default asyncio is configured to use @ref{29fd,,SelectorEventLoop}
on Unix and @ref{4da,,ProactorEventLoop} on Windows.

@geindex SelectorEventLoop (class in asyncio)
@anchor{library/asyncio-eventloop asyncio SelectorEventLoop}@anchor{29fd}
@deffn {Class} asyncio.SelectorEventLoop

An event loop based on the @ref{e1,,selectors} module.

Uses the most efficient @emph{selector} available for the given
platform.  It is also possible to manually configure the
exact selector implementation to be used:

@example
import asyncio
import selectors

selector = selectors.SelectSelector()
loop = asyncio.SelectorEventLoop(selector)
asyncio.set_event_loop(loop)
@end example

@ref{1629,,Availability}: Unix, Windows.
@end deffn

@geindex ProactorEventLoop (class in asyncio)
@anchor{library/asyncio-eventloop asyncio ProactorEventLoop}@anchor{4da}
@deffn {Class} asyncio.ProactorEventLoop

An event loop for Windows that uses “I/O Completion Ports” (IOCP).

@ref{1629,,Availability}: Windows.


@subsubheading See also


MSDN documentation on I/O Completion Ports@footnote{https://docs.microsoft.com/en-ca/windows/desktop/FileIO/i-o-completion-ports}.

@end deffn

@geindex AbstractEventLoop (class in asyncio)
@anchor{library/asyncio-eventloop asyncio AbstractEventLoop}@anchor{462}
@deffn {Class} asyncio.AbstractEventLoop

Abstract base class for asyncio-compliant event loops.

The @ref{8d1,,Event Loop Methods} section lists all
methods that an alternative implementation of @code{AbstractEventLoop}
should have defined.
@end deffn

@node Examples<16>,,Event Loop Implementations,Event Loop
@anchor{library/asyncio-eventloop examples}@anchor{2a27}
@subsubsection Examples


Note that all examples in this section @strong{purposefully} show how
to use the low-level event loop APIs, such as @ref{2a29,,loop.run_forever()}
and @ref{622,,loop.call_soon()}.  Modern asyncio applications rarely
need to be written this way; consider using the high-level functions
like @ref{3d1,,asyncio.run()}.

@menu
* Hello World with call_soon(): Hello World with call_soon. 
* Display the current date with call_later(): Display the current date with call_later. 
* Watch a file descriptor for read events:: 
* Set signal handlers for SIGINT and SIGTERM:: 

@end menu

@node Hello World with call_soon,Display the current date with call_later,,Examples<16>
@anchor{library/asyncio-eventloop asyncio-example-lowlevel-helloworld}@anchor{2a5b}@anchor{library/asyncio-eventloop hello-world-with-call-soon}@anchor{2a5c}
@subsubsection Hello World with call_soon()


An example using the @ref{622,,loop.call_soon()} method to schedule a
callback. The callback displays @code{"Hello World"} and then stops the
event loop:

@example
import asyncio

def hello_world(loop):
    """A callback to print 'Hello World' and stop the event loop"""
    print('Hello World')
    loop.stop()

loop = asyncio.get_event_loop()

# Schedule a call to hello_world()
loop.call_soon(hello_world, loop)

# Blocking call interrupted by loop.stop()
try:
    loop.run_forever()
finally:
    loop.close()
@end example


@subsubheading See also


A similar @ref{2977,,Hello World}
example created with a coroutine and the @ref{3d1,,run()} function.


@node Display the current date with call_later,Watch a file descriptor for read events,Hello World with call_soon,Examples<16>
@anchor{library/asyncio-eventloop asyncio-example-call-later}@anchor{2a5d}@anchor{library/asyncio-eventloop display-the-current-date-with-call-later}@anchor{2a5e}
@subsubsection Display the current date with call_later()


An example of a callback displaying the current date every second. The
callback uses the @ref{624,,loop.call_later()} method to reschedule itself
after 5 seconds, and then stops the event loop:

@example
import asyncio
import datetime

def display_date(end_time, loop):
    print(datetime.datetime.now())
    if (loop.time() + 1.0) < end_time:
        loop.call_later(1, display_date, end_time, loop)
    else:
        loop.stop()

loop = asyncio.get_event_loop()

# Schedule the first call to display_date()
end_time = loop.time() + 5.0
loop.call_soon(display_date, end_time, loop)

# Blocking call interrupted by loop.stop()
try:
    loop.run_forever()
finally:
    loop.close()
@end example


@subsubheading See also


A similar @ref{2988,,current date} example
created with a coroutine and the @ref{3d1,,run()} function.


@node Watch a file descriptor for read events,Set signal handlers for SIGINT and SIGTERM,Display the current date with call_later,Examples<16>
@anchor{library/asyncio-eventloop asyncio-example-watch-fd}@anchor{29c4}@anchor{library/asyncio-eventloop watch-a-file-descriptor-for-read-events}@anchor{2a5f}
@subsubsection Watch a file descriptor for read events


Wait until a file descriptor received some data using the
@ref{29c5,,loop.add_reader()} method and then close the event loop:

@example
import asyncio
from socket import socketpair

# Create a pair of connected file descriptors
rsock, wsock = socketpair()

loop = asyncio.get_event_loop()

def reader():
    data = rsock.recv(100)
    print("Received:", data.decode())

    # We are done: unregister the file descriptor
    loop.remove_reader(rsock)

    # Stop the event loop
    loop.stop()

# Register the file descriptor for read event
loop.add_reader(rsock, reader)

# Simulate the reception of data from the network
loop.call_soon(wsock.send, 'abc'.encode())

try:
    # Run the event loop
    loop.run_forever()
finally:
    # We are done. Close sockets and the event loop.
    rsock.close()
    wsock.close()
    loop.close()
@end example


@subsubheading See also



@itemize *

@item 
A similar @ref{29c3,,example}
using transports, protocols, and the
@ref{4e0,,loop.create_connection()} method.

@item 
Another similar @ref{29c1,,example}
using the high-level @ref{636,,asyncio.open_connection()} function
and streams.
@end itemize


@node Set signal handlers for SIGINT and SIGTERM,,Watch a file descriptor for read events,Examples<16>
@anchor{library/asyncio-eventloop asyncio-example-unix-signals}@anchor{2a60}@anchor{library/asyncio-eventloop set-signal-handlers-for-sigint-and-sigterm}@anchor{2a61}
@subsubsection Set signal handlers for SIGINT and SIGTERM


(This @code{signals} example only works on Unix.)

Register handlers for signals @code{SIGINT} and @code{SIGTERM}
using the @ref{297d,,loop.add_signal_handler()} method:

@example
import asyncio
import functools
import os
import signal

def ask_exit(signame, loop):
    print("got signal %s: exit" % signame)
    loop.stop()

async def main():
    loop = asyncio.get_running_loop()

    for signame in @{'SIGINT', 'SIGTERM'@}:
        loop.add_signal_handler(
            getattr(signal, signame),
            functools.partial(ask_exit, signame, loop))

    await asyncio.sleep(3600)

print("Event loop running for 1 hour, press Ctrl+C to interrupt.")
print(f"pid @{os.getpid()@}: send SIGINT or SIGTERM to exit.")

asyncio.run(main())
@end example

@node Futures,Transports and Protocols,Event Loop,asyncio — Asynchronous I/O
@anchor{library/asyncio-future doc}@anchor{2a62}@anchor{library/asyncio-future asyncio-futures}@anchor{297f}@anchor{library/asyncio-future futures}@anchor{2a63}
@subsubsection Futures


@strong{Source code:} Lib/asyncio/futures.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/futures.py},
Lib/asyncio/base_futures.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/base_futures.py}


__________________________________________________________________


@emph{Future} objects are used to bridge @strong{low-level callback-based code}
with high-level async/await code.

@menu
* Future Functions:: 
* Future Object:: 

@end menu

@node Future Functions,Future Object,,Futures
@anchor{library/asyncio-future future-functions}@anchor{2a64}
@subsubsection Future Functions


@geindex isfuture() (in module asyncio)
@anchor{library/asyncio-future asyncio isfuture}@anchor{2a65}
@deffn {Function} asyncio.isfuture (obj)

Return @code{True} if @emph{obj} is either of:


@itemize *

@item 
an instance of @ref{316,,asyncio.Future},

@item 
an instance of @ref{317,,asyncio.Task},

@item 
a Future-like object with a @code{_asyncio_future_blocking}
attribute.
@end itemize


New in version 3.5.

@end deffn

@geindex ensure_future() (in module asyncio)
@anchor{library/asyncio-future asyncio ensure_future}@anchor{318}
@deffn {Function} asyncio.ensure_future (obj, *, loop=None)

Return:


@itemize *

@item 
@emph{obj} argument as is, if @emph{obj} is a @ref{316,,Future},
a @ref{317,,Task}, or a Future-like object (@ref{2a65,,isfuture()}
is used for the test.)

@item 
a @ref{317,,Task} object wrapping @emph{obj}, if @emph{obj} is a
coroutine (@code{iscoroutine()} is used for the test);
in this case the coroutine will be scheduled by
@code{ensure_future()}.

@item 
a @ref{317,,Task} object that would await on @emph{obj}, if @emph{obj} is an
awaitable (@ref{97a,,inspect.isawaitable()} is used for the test.)
@end itemize

If @emph{obj} is neither of the above a @ref{162,,TypeError} is raised.

@cartouche
@quotation Important 
See also the @ref{4dc,,create_task()} function which is the
preferred way for creating new Tasks.

Save a reference to the result of this function, to avoid
a task disappearing mid execution.
@end quotation
@end cartouche


Changed in version 3.5.1: The function accepts any @ref{7bd,,awaitable} object.



Deprecated since version 3.10: Deprecation warning is emitted if @emph{obj} is not a Future-like object
and @emph{loop} is not specified and there is no running event loop.

@end deffn

@geindex wrap_future() (in module asyncio)
@anchor{library/asyncio-future asyncio wrap_future}@anchor{319}
@deffn {Function} asyncio.wrap_future (future, *, loop=None)

Wrap a @ref{d57,,concurrent.futures.Future} object in a
@ref{316,,asyncio.Future} object.


Deprecated since version 3.10: Deprecation warning is emitted if @emph{future} is not a Future-like object
and @emph{loop} is not specified and there is no running event loop.

@end deffn

@node Future Object,,Future Functions,Futures
@anchor{library/asyncio-future future-object}@anchor{2a66}
@subsubsection Future Object


@geindex Future (class in asyncio)
@anchor{library/asyncio-future asyncio Future}@anchor{316}
@deffn {Class} asyncio.Future (*, loop=None)

A Future represents an eventual result of an asynchronous
operation.  Not thread-safe.

Future is an @ref{7bd,,awaitable} object.  Coroutines can await on
Future objects until they either have a result or an exception
set, or until they are cancelled.

Typically Futures are used to enable low-level
callback-based code (e.g. in protocols implemented using asyncio
@ref{297e,,transports})
to interoperate with high-level async/await code.

The rule of thumb is to never expose Future objects in user-facing
APIs, and the recommended way to create a Future object is to call
@ref{7bf,,loop.create_future()}.  This way alternative event loop
implementations can inject their own optimized implementations
of a Future object.


Changed in version 3.7: Added support for the @ref{23,,contextvars} module.



Deprecated since version 3.10: Deprecation warning is emitted if @emph{loop} is not specified
and there is no running event loop.


@geindex result() (asyncio.Future method)
@anchor{library/asyncio-future asyncio Future result}@anchor{2a67}
@deffn {Method} result ()

Return the result of the Future.

If the Future is @emph{done} and has a result set by the
@ref{2997,,set_result()} method, the result value is returned.

If the Future is @emph{done} and has an exception set by the
@ref{2998,,set_exception()} method, this method raises the exception.

If the Future has been @emph{cancelled}, this method raises
a @ref{4d7,,CancelledError} exception.

If the Future’s result isn’t yet available, this method raises
a @ref{299d,,InvalidStateError} exception.
@end deffn

@geindex set_result() (asyncio.Future method)
@anchor{library/asyncio-future asyncio Future set_result}@anchor{2997}
@deffn {Method} set_result (result)

Mark the Future as @emph{done} and set its result.

Raises a @ref{299d,,InvalidStateError} error if the Future is
already @emph{done}.
@end deffn

@geindex set_exception() (asyncio.Future method)
@anchor{library/asyncio-future asyncio Future set_exception}@anchor{2998}
@deffn {Method} set_exception (exception)

Mark the Future as @emph{done} and set an exception.

Raises a @ref{299d,,InvalidStateError} error if the Future is
already @emph{done}.
@end deffn

@geindex done() (asyncio.Future method)
@anchor{library/asyncio-future asyncio Future done}@anchor{2a68}
@deffn {Method} done ()

Return @code{True} if the Future is @emph{done}.

A Future is @emph{done} if it was @emph{cancelled} or if it has a result
or an exception set with @ref{2997,,set_result()} or
@ref{2998,,set_exception()} calls.
@end deffn

@geindex cancelled() (asyncio.Future method)
@anchor{library/asyncio-future asyncio Future cancelled}@anchor{2a69}
@deffn {Method} cancelled ()

Return @code{True} if the Future was @emph{cancelled}.

The method is usually used to check if a Future is not
@emph{cancelled} before setting a result or an exception for it:

@example
if not fut.cancelled():
    fut.set_result(42)
@end example
@end deffn

@geindex add_done_callback() (asyncio.Future method)
@anchor{library/asyncio-future asyncio Future add_done_callback}@anchor{626}
@deffn {Method} add_done_callback (callback, *, context=None)

Add a callback to be run when the Future is @emph{done}.

The @emph{callback} is called with the Future object as its only
argument.

If the Future is already @emph{done} when this method is called,
the callback is scheduled with @ref{622,,loop.call_soon()}.

An optional keyword-only @emph{context} argument allows specifying a
custom @ref{2943,,contextvars.Context} for the @emph{callback} to run in.
The current context is used when no @emph{context} is provided.

@ref{a3f,,functools.partial()} can be used to pass parameters
to the callback, e.g.:

@example
# Call 'print("Future:", fut)' when "fut" is done.
fut.add_done_callback(
    functools.partial(print, "Future:"))
@end example


Changed in version 3.7: The @emph{context} keyword-only parameter was added.
See 
@geindex Python Enhancement Proposals; PEP 567
PEP 567@footnote{https://www.python.org/dev/peps/pep-0567} for more details.

@end deffn

@geindex remove_done_callback() (asyncio.Future method)
@anchor{library/asyncio-future asyncio Future remove_done_callback}@anchor{29a1}
@deffn {Method} remove_done_callback (callback)

Remove @emph{callback} from the callbacks list.

Returns the number of callbacks removed, which is typically 1,
unless a callback was added more than once.
@end deffn

@geindex cancel() (asyncio.Future method)
@anchor{library/asyncio-future asyncio Future cancel}@anchor{2999}
@deffn {Method} cancel (msg=None)

Cancel the Future and schedule callbacks.

If the Future is already @emph{done} or @emph{cancelled}, return @code{False}.
Otherwise, change the Future’s state to @emph{cancelled},
schedule the callbacks, and return @code{True}.


Changed in version 3.9: Added the @emph{msg} parameter.

@end deffn

@geindex exception() (asyncio.Future method)
@anchor{library/asyncio-future asyncio Future exception}@anchor{2a6a}
@deffn {Method} exception ()

Return the exception that was set on this Future.

The exception (or @code{None} if no exception was set) is
returned only if the Future is @emph{done}.

If the Future has been @emph{cancelled}, this method raises a
@ref{4d7,,CancelledError} exception.

If the Future isn’t @emph{done} yet, this method raises an
@ref{299d,,InvalidStateError} exception.
@end deffn

@geindex get_loop() (asyncio.Future method)
@anchor{library/asyncio-future asyncio Future get_loop}@anchor{62d}
@deffn {Method} get_loop ()

Return the event loop the Future object is bound to.


New in version 3.7.

@end deffn
@end deffn
@anchor{library/asyncio-future asyncio-example-future}@anchor{2a6b}
This example creates a Future object, creates and schedules an
asynchronous Task to set result for the Future, and waits until
the Future has a result:

@example
async def set_after(fut, delay, value):
    # Sleep for *delay* seconds.
    await asyncio.sleep(delay)

    # Set *value* as a result of *fut* Future.
    fut.set_result(value)

async def main():
    # Get the current event loop.
    loop = asyncio.get_running_loop()

    # Create a new Future object.
    fut = loop.create_future()

    # Run "set_after()" coroutine in a parallel Task.
    # We are using the low-level "loop.create_task()" API here because
    # we already have a reference to the event loop at hand.
    # Otherwise we could have just used "asyncio.create_task()".
    loop.create_task(
        set_after(fut, 1, '... world'))

    print('hello ...')

    # Wait until *fut* has a result (1 second) and print it.
    print(await fut)

asyncio.run(main())
@end example

@cartouche
@quotation Important 
The Future object was designed to mimic
@ref{d57,,concurrent.futures.Future}.  Key differences include:


@itemize -

@item 
unlike asyncio Futures, @ref{d57,,concurrent.futures.Future}
instances cannot be awaited.

@item 
@ref{2a67,,asyncio.Future.result()} and @ref{2a6a,,asyncio.Future.exception()}
do not accept the @emph{timeout} argument.

@item 
@ref{2a67,,asyncio.Future.result()} and @ref{2a6a,,asyncio.Future.exception()}
raise an @ref{299d,,InvalidStateError} exception when the Future is not
@emph{done}.

@item 
Callbacks registered with @ref{626,,asyncio.Future.add_done_callback()}
are not called immediately.  They are scheduled with
@ref{622,,loop.call_soon()} instead.

@item 
asyncio Future is not compatible with the
@ref{10bc,,concurrent.futures.wait()} and
@ref{13af,,concurrent.futures.as_completed()} functions.

@item 
@ref{2999,,asyncio.Future.cancel()} accepts an optional @code{msg} argument,
but @code{concurrent.futures.cancel()} does not.
@end itemize
@end quotation
@end cartouche

@node Transports and Protocols,Policies,Futures,asyncio — Asynchronous I/O
@anchor{library/asyncio-protocol doc}@anchor{2a6c}@anchor{library/asyncio-protocol asyncio-transports-protocols}@anchor{297e}@anchor{library/asyncio-protocol transports-and-protocols}@anchor{2a6d}
@subsubsection Transports and Protocols


@subsubheading Preface


Transports and Protocols are used by the @strong{low-level} event loop
APIs such as @ref{4e0,,loop.create_connection()}.  They use
callback-based programming style and enable high-performance
implementations of network or IPC protocols (e.g. HTTP).

Essentially, transports and protocols should only be used in
libraries and frameworks and never in high-level asyncio
applications.

This documentation page covers both @ref{2a6e,,Transports} and @ref{2a6f,,Protocols}.

@subsubheading Introduction


At the highest level, the transport is concerned with @emph{how} bytes
are transmitted, while the protocol determines @emph{which} bytes to
transmit (and to some extent when).

A different way of saying the same thing: a transport is an
abstraction for a socket (or similar I/O endpoint) while a protocol
is an abstraction for an application, from the transport’s point
of view.

Yet another view is the transport and protocol interfaces
together define an abstract interface for using network I/O and
interprocess I/O.

There is always a 1:1 relationship between transport and protocol
objects: the protocol calls transport methods to send data,
while the transport calls protocol methods to pass it data that
has been received.

Most of connection oriented event loop methods
(such as @ref{4e0,,loop.create_connection()}) usually accept a
@emph{protocol_factory} argument used to create a @emph{Protocol} object
for an accepted connection, represented by a @emph{Transport} object.
Such methods usually return a tuple of @code{(transport, protocol)}.

@subsubheading Contents


This documentation page contains the following sections:


@itemize *

@item 
The @ref{2a6e,,Transports} section documents asyncio @ref{2a70,,BaseTransport},
@ref{2a4a,,ReadTransport}, @ref{2a4b,,WriteTransport}, @ref{2a71,,Transport},
@ref{108b,,DatagramTransport}, and @ref{2a56,,SubprocessTransport}
classes.

@item 
The @ref{2a6f,,Protocols} section documents asyncio @ref{2a72,,BaseProtocol},
@ref{2a73,,Protocol}, @ref{5bd,,BufferedProtocol},
@ref{2a74,,DatagramProtocol}, and @ref{2a55,,SubprocessProtocol} classes.

@item 
The @ref{2a75,,Examples} section showcases how to work with transports,
protocols, and low-level event loop APIs.
@end itemize

@menu
* Transports:: 
* Protocols:: 
* Examples: Examples<17>. 

@end menu

@node Transports,Protocols,,Transports and Protocols
@anchor{library/asyncio-protocol asyncio-transport}@anchor{2a37}@anchor{library/asyncio-protocol transports}@anchor{2a6e}
@subsubsection Transports


@strong{Source code:} Lib/asyncio/transports.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/transports.py}


__________________________________________________________________


Transports are classes provided by @ref{9,,asyncio} in order to abstract
various kinds of communication channels.

Transport objects are always instantiated by an
@ref{8d1,,asyncio event loop}.

asyncio implements transports for TCP, UDP, SSL, and subprocess pipes.
The methods available on a transport depend on the transport’s kind.

The transport classes are @ref{2993,,not thread safe}.

@menu
* Transports Hierarchy:: 
* Base Transport:: 
* Read-only Transports:: 
* Write-only Transports:: 
* Datagram Transports:: 
* Subprocess Transports:: 

@end menu

@node Transports Hierarchy,Base Transport,,Transports
@anchor{library/asyncio-protocol transports-hierarchy}@anchor{2a76}
@subsubsection Transports Hierarchy


@geindex BaseTransport (class in asyncio)
@anchor{library/asyncio-protocol asyncio BaseTransport}@anchor{2a70}
@deffn {Class} asyncio.BaseTransport

Base class for all transports.  Contains methods that all
asyncio transports share.
@end deffn

@geindex WriteTransport (class in asyncio)
@anchor{library/asyncio-protocol asyncio WriteTransport}@anchor{2a4b}
@deffn {Class} asyncio.WriteTransport (BaseTransport)

A base transport for write-only connections.

Instances of the @emph{WriteTransport} class are returned from
the @ref{29e8,,loop.connect_write_pipe()} event loop method and
are also used by subprocess-related methods like
@ref{297c,,loop.subprocess_exec()}.
@end deffn

@geindex ReadTransport (class in asyncio)
@anchor{library/asyncio-protocol asyncio ReadTransport}@anchor{2a4a}
@deffn {Class} asyncio.ReadTransport (BaseTransport)

A base transport for read-only connections.

Instances of the @emph{ReadTransport} class are returned from
the @ref{29e7,,loop.connect_read_pipe()} event loop method and
are also used by subprocess-related methods like
@ref{297c,,loop.subprocess_exec()}.
@end deffn

@geindex Transport (class in asyncio)
@anchor{library/asyncio-protocol asyncio Transport}@anchor{2a71}
@deffn {Class} asyncio.Transport (WriteTransport, ReadTransport)

Interface representing a bidirectional transport, such as a
TCP connection.

The user does not instantiate a transport directly; they call a
utility function, passing it a protocol factory and other
information necessary to create the transport and protocol.

Instances of the @emph{Transport} class are returned from or used by
event loop methods like @ref{4e0,,loop.create_connection()},
@ref{29a9,,loop.create_unix_connection()},
@ref{630,,loop.create_server()}, @ref{2a3f,,loop.sendfile()}, etc.
@end deffn

@geindex DatagramTransport (class in asyncio)
@anchor{library/asyncio-protocol asyncio DatagramTransport}@anchor{108b}
@deffn {Class} asyncio.DatagramTransport (BaseTransport)

A transport for datagram (UDP) connections.

Instances of the @emph{DatagramTransport} class are returned from
the @ref{1b9,,loop.create_datagram_endpoint()} event loop method.
@end deffn

@geindex SubprocessTransport (class in asyncio)
@anchor{library/asyncio-protocol asyncio SubprocessTransport}@anchor{2a56}
@deffn {Class} asyncio.SubprocessTransport (BaseTransport)

An abstraction to represent a connection between a parent and its
child OS process.

Instances of the @emph{SubprocessTransport} class are returned from
event loop methods @ref{29e5,,loop.subprocess_shell()} and
@ref{297c,,loop.subprocess_exec()}.
@end deffn

@node Base Transport,Read-only Transports,Transports Hierarchy,Transports
@anchor{library/asyncio-protocol base-transport}@anchor{2a77}
@subsubsection Base Transport


@geindex close() (asyncio.BaseTransport method)
@anchor{library/asyncio-protocol asyncio BaseTransport close}@anchor{2a78}
@deffn {Method} BaseTransport.close ()

Close the transport.

If the transport has a buffer for outgoing
data, buffered data will be flushed asynchronously.  No more data
will be received.  After all buffered data is flushed, the
protocol’s @ref{2a79,,protocol.connection_lost()} method will be called with
@ref{182,,None} as its argument.
@end deffn

@geindex is_closing() (asyncio.BaseTransport method)
@anchor{library/asyncio-protocol asyncio BaseTransport is_closing}@anchor{7be}
@deffn {Method} BaseTransport.is_closing ()

Return @code{True} if the transport is closing or is closed.
@end deffn

@geindex get_extra_info() (asyncio.BaseTransport method)
@anchor{library/asyncio-protocol asyncio BaseTransport get_extra_info}@anchor{5bc}
@deffn {Method} BaseTransport.get_extra_info (name, default=None)

Return information about the transport or underlying resources
it uses.

@emph{name} is a string representing the piece of transport-specific
information to get.

@emph{default} is the value to return if the information is not
available, or if the transport does not support querying it
with the given third-party event loop implementation or on the
current platform.

For example, the following code attempts to get the underlying
socket object of the transport:

@example
sock = transport.get_extra_info('socket')
if sock is not None:
    print(sock.getsockopt(...))
@end example

Categories of information that can be queried on some transports:


@itemize *

@item 
socket:


@itemize -

@item 
@code{'peername'}: the remote address to which the socket is
connected, result of @ref{2a7a,,socket.socket.getpeername()}
(@code{None} on error)

@item 
@code{'socket'}: @ref{8fd,,socket.socket} instance

@item 
@code{'sockname'}: the socket’s own address,
result of @ref{2a7b,,socket.socket.getsockname()}
@end itemize

@item 
SSL socket:


@itemize -

@item 
@code{'compression'}: the compression algorithm being used as a
string, or @code{None} if the connection isn’t compressed; result
of @ref{cdd,,ssl.SSLSocket.compression()}

@item 
@code{'cipher'}: a three-value tuple containing the name of the
cipher being used, the version of the SSL protocol that defines
its use, and the number of secret bits being used; result of
@ref{2a7c,,ssl.SSLSocket.cipher()}

@item 
@code{'peercert'}: peer certificate; result of
@ref{b39,,ssl.SSLSocket.getpeercert()}

@item 
@code{'sslcontext'}: @ref{2bf,,ssl.SSLContext} instance

@item 
@code{'ssl_object'}: @ref{6af,,ssl.SSLObject} or
@ref{3d6,,ssl.SSLSocket} instance
@end itemize

@item 
pipe:


@itemize -

@item 
@code{'pipe'}: pipe object
@end itemize

@item 
subprocess:


@itemize -

@item 
@code{'subprocess'}: @ref{5ae,,subprocess.Popen} instance
@end itemize
@end itemize
@end deffn

@geindex set_protocol() (asyncio.BaseTransport method)
@anchor{library/asyncio-protocol asyncio BaseTransport set_protocol}@anchor{2a7d}
@deffn {Method} BaseTransport.set_protocol (protocol)

Set a new protocol.

Switching protocol should only be done when both
protocols are documented to support the switch.
@end deffn

@geindex get_protocol() (asyncio.BaseTransport method)
@anchor{library/asyncio-protocol asyncio BaseTransport get_protocol}@anchor{2a7e}
@deffn {Method} BaseTransport.get_protocol ()

Return the current protocol.
@end deffn

@node Read-only Transports,Write-only Transports,Base Transport,Transports
@anchor{library/asyncio-protocol read-only-transports}@anchor{2a7f}
@subsubsection Read-only Transports


@geindex is_reading() (asyncio.ReadTransport method)
@anchor{library/asyncio-protocol asyncio ReadTransport is_reading}@anchor{63a}
@deffn {Method} ReadTransport.is_reading ()

Return @code{True} if the transport is receiving new data.


New in version 3.7.

@end deffn

@geindex pause_reading() (asyncio.ReadTransport method)
@anchor{library/asyncio-protocol asyncio ReadTransport pause_reading}@anchor{63c}
@deffn {Method} ReadTransport.pause_reading ()

Pause the receiving end of the transport.  No data will be passed to
the protocol’s @ref{2a80,,protocol.data_received()}
method until @ref{63b,,resume_reading()} is called.


Changed in version 3.7: The method is idempotent, i.e. it can be called when the
transport is already paused or closed.

@end deffn

@geindex resume_reading() (asyncio.ReadTransport method)
@anchor{library/asyncio-protocol asyncio ReadTransport resume_reading}@anchor{63b}
@deffn {Method} ReadTransport.resume_reading ()

Resume the receiving end.  The protocol’s
@ref{2a80,,protocol.data_received()} method
will be called once again if some data is available for reading.


Changed in version 3.7: The method is idempotent, i.e. it can be called when the
transport is already reading.

@end deffn

@node Write-only Transports,Datagram Transports,Read-only Transports,Transports
@anchor{library/asyncio-protocol write-only-transports}@anchor{2a81}
@subsubsection Write-only Transports


@geindex abort() (asyncio.WriteTransport method)
@anchor{library/asyncio-protocol asyncio WriteTransport abort}@anchor{2a82}
@deffn {Method} WriteTransport.abort ()

Close the transport immediately, without waiting for pending operations
to complete.  Buffered data will be lost.  No more data will be received.
The protocol’s @ref{2a79,,protocol.connection_lost()} method will eventually be
called with @ref{182,,None} as its argument.
@end deffn

@geindex can_write_eof() (asyncio.WriteTransport method)
@anchor{library/asyncio-protocol asyncio WriteTransport can_write_eof}@anchor{2a83}
@deffn {Method} WriteTransport.can_write_eof ()

Return @ref{748,,True} if the transport supports
@ref{2a84,,write_eof()}, @ref{659,,False} if not.
@end deffn

@geindex get_write_buffer_size() (asyncio.WriteTransport method)
@anchor{library/asyncio-protocol asyncio WriteTransport get_write_buffer_size}@anchor{2a85}
@deffn {Method} WriteTransport.get_write_buffer_size ()

Return the current size of the output buffer used by the transport.
@end deffn

@geindex get_write_buffer_limits() (asyncio.WriteTransport method)
@anchor{library/asyncio-protocol asyncio WriteTransport get_write_buffer_limits}@anchor{923}
@deffn {Method} WriteTransport.get_write_buffer_limits ()

Get the @emph{high} and @emph{low} watermarks for write flow control. Return a
tuple @code{(low, high)} where @emph{low} and @emph{high} are positive number of
bytes.

Use @ref{2a86,,set_write_buffer_limits()} to set the limits.


New in version 3.4.2.

@end deffn

@geindex set_write_buffer_limits() (asyncio.WriteTransport method)
@anchor{library/asyncio-protocol asyncio WriteTransport set_write_buffer_limits}@anchor{2a86}
@deffn {Method} WriteTransport.set_write_buffer_limits (high=None, low=None)

Set the @emph{high} and @emph{low} watermarks for write flow control.

These two values (measured in number of
bytes) control when the protocol’s
@ref{2a87,,protocol.pause_writing()}
and @ref{2a88,,protocol.resume_writing()}
methods are called. If specified, the low watermark must be less
than or equal to the high watermark.  Neither @emph{high} nor @emph{low}
can be negative.

@ref{2a87,,pause_writing()} is called when the buffer size
becomes greater than or equal to the @emph{high} value. If writing has
been paused, @ref{2a88,,resume_writing()} is called when
the buffer size becomes less than or equal to the @emph{low} value.

The defaults are implementation-specific.  If only the
high watermark is given, the low watermark defaults to an
implementation-specific value less than or equal to the
high watermark.  Setting @emph{high} to zero forces @emph{low} to zero as
well, and causes @ref{2a87,,pause_writing()} to be called
whenever the buffer becomes non-empty.  Setting @emph{low} to zero causes
@ref{2a88,,resume_writing()} to be called only once the
buffer is empty. Use of zero for either limit is generally
sub-optimal as it reduces opportunities for doing I/O and
computation concurrently.

Use @ref{923,,get_write_buffer_limits()}
to get the limits.
@end deffn

@geindex write() (asyncio.WriteTransport method)
@anchor{library/asyncio-protocol asyncio WriteTransport write}@anchor{2a89}
@deffn {Method} WriteTransport.write (data)

Write some @emph{data} bytes to the transport.

This method does not block; it buffers the data and arranges for it
to be sent out asynchronously.
@end deffn

@geindex writelines() (asyncio.WriteTransport method)
@anchor{library/asyncio-protocol asyncio WriteTransport writelines}@anchor{2a8a}
@deffn {Method} WriteTransport.writelines (list_of_data)

Write a list (or any iterable) of data bytes to the transport.
This is functionally equivalent to calling @ref{2a89,,write()} on each
element yielded by the iterable, but may be implemented more
efficiently.
@end deffn

@geindex write_eof() (asyncio.WriteTransport method)
@anchor{library/asyncio-protocol asyncio WriteTransport write_eof}@anchor{2a84}
@deffn {Method} WriteTransport.write_eof ()

Close the write end of the transport after flushing all buffered data.
Data may still be received.

This method can raise @ref{8a3,,NotImplementedError} if the transport
(e.g. SSL) doesn’t support half-closed connections.
@end deffn

@node Datagram Transports,Subprocess Transports,Write-only Transports,Transports
@anchor{library/asyncio-protocol datagram-transports}@anchor{2a8b}
@subsubsection Datagram Transports


@geindex sendto() (asyncio.DatagramTransport method)
@anchor{library/asyncio-protocol asyncio DatagramTransport sendto}@anchor{2a8c}
@deffn {Method} DatagramTransport.sendto (data, addr=None)

Send the @emph{data} bytes to the remote peer given by @emph{addr} (a
transport-dependent target address).  If @emph{addr} is @ref{182,,None},
the data is sent to the target address given on transport
creation.

This method does not block; it buffers the data and arranges
for it to be sent out asynchronously.
@end deffn

@geindex abort() (asyncio.DatagramTransport method)
@anchor{library/asyncio-protocol asyncio DatagramTransport abort}@anchor{2a8d}
@deffn {Method} DatagramTransport.abort ()

Close the transport immediately, without waiting for pending
operations to complete.  Buffered data will be lost.
No more data will be received.  The protocol’s
@ref{2a79,,protocol.connection_lost()}
method will eventually be called with @ref{182,,None} as its argument.
@end deffn

@node Subprocess Transports,,Datagram Transports,Transports
@anchor{library/asyncio-protocol asyncio-subprocess-transports}@anchor{29e9}@anchor{library/asyncio-protocol subprocess-transports}@anchor{2a8e}
@subsubsection Subprocess Transports


@geindex get_pid() (asyncio.SubprocessTransport method)
@anchor{library/asyncio-protocol asyncio SubprocessTransport get_pid}@anchor{2a8f}
@deffn {Method} SubprocessTransport.get_pid ()

Return the subprocess process id as an integer.
@end deffn

@geindex get_pipe_transport() (asyncio.SubprocessTransport method)
@anchor{library/asyncio-protocol asyncio SubprocessTransport get_pipe_transport}@anchor{2a90}
@deffn {Method} SubprocessTransport.get_pipe_transport (fd)

Return the transport for the communication pipe corresponding to the
integer file descriptor @emph{fd}:


@itemize *

@item 
@code{0}: readable streaming transport of the standard input (@emph{stdin}),
or @ref{182,,None} if the subprocess was not created with @code{stdin=PIPE}

@item 
@code{1}: writable streaming transport of the standard output (@emph{stdout}),
or @ref{182,,None} if the subprocess was not created with @code{stdout=PIPE}

@item 
@code{2}: writable streaming transport of the standard error (@emph{stderr}),
or @ref{182,,None} if the subprocess was not created with @code{stderr=PIPE}

@item 
other @emph{fd}: @ref{182,,None}
@end itemize
@end deffn

@geindex get_returncode() (asyncio.SubprocessTransport method)
@anchor{library/asyncio-protocol asyncio SubprocessTransport get_returncode}@anchor{2a91}
@deffn {Method} SubprocessTransport.get_returncode ()

Return the subprocess return code as an integer or @ref{182,,None}
if it hasn’t returned, which is similar to the
@ref{1241,,subprocess.Popen.returncode} attribute.
@end deffn

@geindex kill() (asyncio.SubprocessTransport method)
@anchor{library/asyncio-protocol asyncio SubprocessTransport kill}@anchor{2a92}
@deffn {Method} SubprocessTransport.kill ()

Kill the subprocess.

On POSIX systems, the function sends SIGKILL to the subprocess.
On Windows, this method is an alias for @ref{2a93,,terminate()}.

See also @ref{290b,,subprocess.Popen.kill()}.
@end deffn

@geindex send_signal() (asyncio.SubprocessTransport method)
@anchor{library/asyncio-protocol asyncio SubprocessTransport send_signal}@anchor{2a94}
@deffn {Method} SubprocessTransport.send_signal (signal)

Send the @emph{signal} number to the subprocess, as in
@ref{1240,,subprocess.Popen.send_signal()}.
@end deffn

@geindex terminate() (asyncio.SubprocessTransport method)
@anchor{library/asyncio-protocol asyncio SubprocessTransport terminate}@anchor{2a93}
@deffn {Method} SubprocessTransport.terminate ()

Stop the subprocess.

On POSIX systems, this method sends SIGTERM to the subprocess.
On Windows, the Windows API function TerminateProcess() is called to
stop the subprocess.

See also @ref{290a,,subprocess.Popen.terminate()}.
@end deffn

@geindex close() (asyncio.SubprocessTransport method)
@anchor{library/asyncio-protocol asyncio SubprocessTransport close}@anchor{2a95}
@deffn {Method} SubprocessTransport.close ()

Kill the subprocess by calling the @ref{2a92,,kill()} method.

If the subprocess hasn’t returned yet, and close transports of
@emph{stdin}, @emph{stdout}, and @emph{stderr} pipes.
@end deffn

@node Protocols,Examples<17>,Transports,Transports and Protocols
@anchor{library/asyncio-protocol asyncio-protocol}@anchor{2a36}@anchor{library/asyncio-protocol protocols}@anchor{2a6f}
@subsubsection Protocols


@strong{Source code:} Lib/asyncio/protocols.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/protocols.py}


__________________________________________________________________


asyncio provides a set of abstract base classes that should be used
to implement network protocols.  Those classes are meant to be used
together with @ref{2a37,,transports}.

Subclasses of abstract base protocol classes may implement some or
all methods.  All these methods are callbacks: they are called by
transports on certain events, for example when some data is received.
A base protocol method should be called by the corresponding transport.

@menu
* Base Protocols:: 
* Base Protocol:: 
* Streaming Protocols:: 
* Buffered Streaming Protocols:: 
* Datagram Protocols:: 
* Subprocess Protocols:: 

@end menu

@node Base Protocols,Base Protocol,,Protocols
@anchor{library/asyncio-protocol base-protocols}@anchor{2a96}
@subsubsection Base Protocols


@geindex BaseProtocol (class in asyncio)
@anchor{library/asyncio-protocol asyncio BaseProtocol}@anchor{2a72}
@deffn {Class} asyncio.BaseProtocol

Base protocol with methods that all protocols share.
@end deffn

@geindex Protocol (class in asyncio)
@anchor{library/asyncio-protocol asyncio Protocol}@anchor{2a73}
@deffn {Class} asyncio.Protocol (BaseProtocol)

The base class for implementing streaming protocols
(TCP, Unix sockets, etc).
@end deffn

@geindex BufferedProtocol (class in asyncio)
@anchor{library/asyncio-protocol asyncio BufferedProtocol}@anchor{5bd}
@deffn {Class} asyncio.BufferedProtocol (BaseProtocol)

A base class for implementing streaming protocols with manual
control of the receive buffer.
@end deffn

@geindex DatagramProtocol (class in asyncio)
@anchor{library/asyncio-protocol asyncio DatagramProtocol}@anchor{2a74}
@deffn {Class} asyncio.DatagramProtocol (BaseProtocol)

The base class for implementing datagram (UDP) protocols.
@end deffn

@geindex SubprocessProtocol (class in asyncio)
@anchor{library/asyncio-protocol asyncio SubprocessProtocol}@anchor{2a55}
@deffn {Class} asyncio.SubprocessProtocol (BaseProtocol)

The base class for implementing protocols communicating with child
processes (unidirectional pipes).
@end deffn

@node Base Protocol,Streaming Protocols,Base Protocols,Protocols
@anchor{library/asyncio-protocol base-protocol}@anchor{2a97}
@subsubsection Base Protocol


All asyncio protocols can implement Base Protocol callbacks.

@subsubheading Connection Callbacks


Connection callbacks are called on all protocols, exactly once per
a successful connection.  All other protocol callbacks can only be
called between those two methods.

@geindex connection_made() (asyncio.BaseProtocol method)
@anchor{library/asyncio-protocol asyncio BaseProtocol connection_made}@anchor{2a38}
@deffn {Method} BaseProtocol.connection_made (transport)

Called when a connection is made.

The @emph{transport} argument is the transport representing the
connection.  The protocol is responsible for storing the reference
to its transport.
@end deffn

@geindex connection_lost() (asyncio.BaseProtocol method)
@anchor{library/asyncio-protocol asyncio BaseProtocol connection_lost}@anchor{2a79}
@deffn {Method} BaseProtocol.connection_lost (exc)

Called when the connection is lost or closed.

The argument is either an exception object or @ref{182,,None}.
The latter means a regular EOF is received, or the connection was
aborted or closed by this side of the connection.
@end deffn

@subsubheading Flow Control Callbacks


Flow control callbacks can be called by transports to pause or
resume writing performed by the protocol.

See the documentation of the @ref{2a86,,set_write_buffer_limits()}
method for more details.

@geindex pause_writing() (asyncio.BaseProtocol method)
@anchor{library/asyncio-protocol asyncio BaseProtocol pause_writing}@anchor{2a87}
@deffn {Method} BaseProtocol.pause_writing ()

Called when the transport’s buffer goes over the high watermark.
@end deffn

@geindex resume_writing() (asyncio.BaseProtocol method)
@anchor{library/asyncio-protocol asyncio BaseProtocol resume_writing}@anchor{2a88}
@deffn {Method} BaseProtocol.resume_writing ()

Called when the transport’s buffer drains below the low watermark.
@end deffn

If the buffer size equals the high watermark,
@ref{2a87,,pause_writing()} is not called: the buffer size must
go strictly over.

Conversely, @ref{2a88,,resume_writing()} is called when the
buffer size is equal or lower than the low watermark.  These end
conditions are important to ensure that things go as expected when
either mark is zero.

@node Streaming Protocols,Buffered Streaming Protocols,Base Protocol,Protocols
@anchor{library/asyncio-protocol streaming-protocols}@anchor{2a98}
@subsubsection Streaming Protocols


Event methods, such as @ref{630,,loop.create_server()},
@ref{631,,loop.create_unix_server()}, @ref{4e0,,loop.create_connection()},
@ref{29a9,,loop.create_unix_connection()}, @ref{638,,loop.connect_accepted_socket()},
@ref{29e7,,loop.connect_read_pipe()}, and @ref{29e8,,loop.connect_write_pipe()}
accept factories that return streaming protocols.

@geindex data_received() (asyncio.Protocol method)
@anchor{library/asyncio-protocol asyncio Protocol data_received}@anchor{2a80}
@deffn {Method} Protocol.data_received (data)

Called when some data is received.  @emph{data} is a non-empty bytes
object containing the incoming data.

Whether the data is buffered, chunked or reassembled depends on
the transport.  In general, you shouldn’t rely on specific semantics
and instead make your parsing generic and flexible. However,
data is always received in the correct order.

The method can be called an arbitrary number of times while
a connection is open.

However, @ref{2a99,,protocol.eof_received()}
is called at most once.  Once @cite{eof_received()} is called,
@code{data_received()} is not called anymore.
@end deffn

@geindex eof_received() (asyncio.Protocol method)
@anchor{library/asyncio-protocol asyncio Protocol eof_received}@anchor{2a99}
@deffn {Method} Protocol.eof_received ()

Called when the other end signals it won’t send any more data
(for example by calling @ref{2a84,,transport.write_eof()}, if the other end also uses
asyncio).

This method may return a false value (including @code{None}), in which case
the transport will close itself.  Conversely, if this method returns a
true value, the protocol used determines whether to close the transport.
Since the default implementation returns @code{None}, it implicitly closes the
connection.

Some transports, including SSL, don’t support half-closed connections,
in which case returning true from this method will result in the connection
being closed.
@end deffn

State machine:

@example
start -> connection_made
    [-> data_received]*
    [-> eof_received]?
-> connection_lost -> end
@end example

@node Buffered Streaming Protocols,Datagram Protocols,Streaming Protocols,Protocols
@anchor{library/asyncio-protocol buffered-streaming-protocols}@anchor{2a9a}
@subsubsection Buffered Streaming Protocols



New in version 3.7.


Buffered Protocols can be used with any event loop method
that supports @ref{2a98,,Streaming Protocols}.

@code{BufferedProtocol} implementations allow explicit manual allocation
and control of the receive buffer.  Event loops can then use the buffer
provided by the protocol to avoid unnecessary data copies.  This
can result in noticeable performance improvement for protocols that
receive big amounts of data.  Sophisticated protocol implementations
can significantly reduce the number of buffer allocations.

The following callbacks are called on @ref{5bd,,BufferedProtocol}
instances:

@geindex get_buffer() (asyncio.BufferedProtocol method)
@anchor{library/asyncio-protocol asyncio BufferedProtocol get_buffer}@anchor{2a9b}
@deffn {Method} BufferedProtocol.get_buffer (sizehint)

Called to allocate a new receive buffer.

@emph{sizehint} is the recommended minimum size for the returned
buffer.  It is acceptable to return smaller or larger buffers
than what @emph{sizehint} suggests.  When set to -1, the buffer size
can be arbitrary. It is an error to return a buffer with a zero size.

@code{get_buffer()} must return an object implementing the
@ref{1a3e,,buffer protocol}.
@end deffn

@geindex buffer_updated() (asyncio.BufferedProtocol method)
@anchor{library/asyncio-protocol asyncio BufferedProtocol buffer_updated}@anchor{2a9c}
@deffn {Method} BufferedProtocol.buffer_updated (nbytes)

Called when the buffer was updated with the received data.

@emph{nbytes} is the total number of bytes that were written to the buffer.
@end deffn

@geindex eof_received() (asyncio.BufferedProtocol method)
@anchor{library/asyncio-protocol asyncio BufferedProtocol eof_received}@anchor{2a9d}
@deffn {Method} BufferedProtocol.eof_received ()

See the documentation of the @ref{2a99,,protocol.eof_received()} method.
@end deffn

@ref{2a9b,,get_buffer()} can be called an arbitrary number
of times during a connection.  However, @ref{2a99,,protocol.eof_received()} is called at most once
and, if called, @ref{2a9b,,get_buffer()} and
@ref{2a9c,,buffer_updated()} won’t be called after it.

State machine:

@example
start -> connection_made
    [-> get_buffer
        [-> buffer_updated]?
    ]*
    [-> eof_received]?
-> connection_lost -> end
@end example

@node Datagram Protocols,Subprocess Protocols,Buffered Streaming Protocols,Protocols
@anchor{library/asyncio-protocol datagram-protocols}@anchor{2a9e}
@subsubsection Datagram Protocols


Datagram Protocol instances should be constructed by protocol
factories passed to the @ref{1b9,,loop.create_datagram_endpoint()} method.

@geindex datagram_received() (asyncio.DatagramProtocol method)
@anchor{library/asyncio-protocol asyncio DatagramProtocol datagram_received}@anchor{2a9f}
@deffn {Method} DatagramProtocol.datagram_received (data, addr)

Called when a datagram is received.  @emph{data} is a bytes object containing
the incoming data.  @emph{addr} is the address of the peer sending the data;
the exact format depends on the transport.
@end deffn

@geindex error_received() (asyncio.DatagramProtocol method)
@anchor{library/asyncio-protocol asyncio DatagramProtocol error_received}@anchor{2aa0}
@deffn {Method} DatagramProtocol.error_received (exc)

Called when a previous send or receive operation raises an
@ref{2a1,,OSError}.  @emph{exc} is the @ref{2a1,,OSError} instance.

This method is called in rare conditions, when the transport (e.g. UDP)
detects that a datagram could not be delivered to its recipient.
In many conditions though, undeliverable datagrams will be silently
dropped.
@end deffn

@cartouche
@quotation Note 
On BSD systems (macOS, FreeBSD, etc.) flow control is not supported
for datagram protocols, because there is no reliable way to detect send
failures caused by writing too many packets.

The socket always appears ‘ready’ and excess packets are dropped. An
@ref{2a1,,OSError} with @code{errno} set to @ref{2737,,errno.ENOBUFS} may
or may not be raised; if it is raised, it will be reported to
@ref{2aa0,,DatagramProtocol.error_received()} but otherwise ignored.
@end quotation
@end cartouche

@node Subprocess Protocols,,Datagram Protocols,Protocols
@anchor{library/asyncio-protocol asyncio-subprocess-protocols}@anchor{29ea}@anchor{library/asyncio-protocol subprocess-protocols}@anchor{2aa1}
@subsubsection Subprocess Protocols


Subprocess Protocol instances should be constructed by protocol
factories passed to the @ref{297c,,loop.subprocess_exec()} and
@ref{29e5,,loop.subprocess_shell()} methods.

@geindex pipe_data_received() (asyncio.SubprocessProtocol method)
@anchor{library/asyncio-protocol asyncio SubprocessProtocol pipe_data_received}@anchor{2aa2}
@deffn {Method} SubprocessProtocol.pipe_data_received (fd, data)

Called when the child process writes data into its stdout or stderr
pipe.

@emph{fd} is the integer file descriptor of the pipe.

@emph{data} is a non-empty bytes object containing the received data.
@end deffn

@geindex pipe_connection_lost() (asyncio.SubprocessProtocol method)
@anchor{library/asyncio-protocol asyncio SubprocessProtocol pipe_connection_lost}@anchor{2aa3}
@deffn {Method} SubprocessProtocol.pipe_connection_lost (fd, exc)

Called when one of the pipes communicating with the child process
is closed.

@emph{fd} is the integer file descriptor that was closed.
@end deffn

@geindex process_exited() (asyncio.SubprocessProtocol method)
@anchor{library/asyncio-protocol asyncio SubprocessProtocol process_exited}@anchor{2aa4}
@deffn {Method} SubprocessProtocol.process_exited ()

Called when the child process has exited.
@end deffn

@node Examples<17>,,Protocols,Transports and Protocols
@anchor{library/asyncio-protocol examples}@anchor{2a75}
@subsubsection Examples


@menu
* TCP Echo Server:: 
* TCP Echo Client:: 
* UDP Echo Server:: 
* UDP Echo Client:: 
* Connecting Existing Sockets:: 
* loop.subprocess_exec() and SubprocessProtocol: loop subprocess_exec and SubprocessProtocol. 

@end menu

@node TCP Echo Server,TCP Echo Client,,Examples<17>
@anchor{library/asyncio-protocol asyncio-example-tcp-echo-server-protocol}@anchor{29bf}@anchor{library/asyncio-protocol tcp-echo-server}@anchor{2aa5}
@subsubsection TCP Echo Server


Create a TCP echo server using the @ref{630,,loop.create_server()} method, send back
received data, and close the connection:

@example
import asyncio


class EchoServerProtocol(asyncio.Protocol):
    def connection_made(self, transport):
        peername = transport.get_extra_info('peername')
        print('Connection from @{@}'.format(peername))
        self.transport = transport

    def data_received(self, data):
        message = data.decode()
        print('Data received: @{!r@}'.format(message))

        print('Send: @{!r@}'.format(message))
        self.transport.write(data)

        print('Close the client socket')
        self.transport.close()


async def main():
    # Get a reference to the event loop as we plan to use
    # low-level APIs.
    loop = asyncio.get_running_loop()

    server = await loop.create_server(
        lambda: EchoServerProtocol(),
        '127.0.0.1', 8888)

    async with server:
        await server.serve_forever()


asyncio.run(main())
@end example


@subsubheading See also


The @ref{29bd,,TCP echo server using streams}
example uses the high-level @ref{637,,asyncio.start_server()} function.


@node TCP Echo Client,UDP Echo Server,TCP Echo Server,Examples<17>
@anchor{library/asyncio-protocol asyncio-example-tcp-echo-client-protocol}@anchor{29bc}@anchor{library/asyncio-protocol tcp-echo-client}@anchor{2aa6}
@subsubsection TCP Echo Client


A TCP echo client using the @ref{4e0,,loop.create_connection()} method, sends
data, and waits until the connection is closed:

@example
import asyncio


class EchoClientProtocol(asyncio.Protocol):
    def __init__(self, message, on_con_lost):
        self.message = message
        self.on_con_lost = on_con_lost

    def connection_made(self, transport):
        transport.write(self.message.encode())
        print('Data sent: @{!r@}'.format(self.message))

    def data_received(self, data):
        print('Data received: @{!r@}'.format(data.decode()))

    def connection_lost(self, exc):
        print('The server closed the connection')
        self.on_con_lost.set_result(True)


async def main():
    # Get a reference to the event loop as we plan to use
    # low-level APIs.
    loop = asyncio.get_running_loop()

    on_con_lost = loop.create_future()
    message = 'Hello World!'

    transport, protocol = await loop.create_connection(
        lambda: EchoClientProtocol(message, on_con_lost),
        '127.0.0.1', 8888)

    # Wait until the protocol signals that the connection
    # is lost and close the transport.
    try:
        await on_con_lost
    finally:
        transport.close()


asyncio.run(main())
@end example


@subsubheading See also


The @ref{29ba,,TCP echo client using streams}
example uses the high-level @ref{636,,asyncio.open_connection()} function.


@node UDP Echo Server,UDP Echo Client,TCP Echo Client,Examples<17>
@anchor{library/asyncio-protocol asyncio-udp-echo-server-protocol}@anchor{2a3c}@anchor{library/asyncio-protocol udp-echo-server}@anchor{2aa7}
@subsubsection UDP Echo Server


A UDP echo server, using the @ref{1b9,,loop.create_datagram_endpoint()}
method, sends back received data:

@example
import asyncio


class EchoServerProtocol:
    def connection_made(self, transport):
        self.transport = transport

    def datagram_received(self, data, addr):
        message = data.decode()
        print('Received %r from %s' % (message, addr))
        print('Send %r to %s' % (message, addr))
        self.transport.sendto(data, addr)


async def main():
    print("Starting UDP server")

    # Get a reference to the event loop as we plan to use
    # low-level APIs.
    loop = asyncio.get_running_loop()

    # One protocol instance will be created to serve all
    # client requests.
    transport, protocol = await loop.create_datagram_endpoint(
        lambda: EchoServerProtocol(),
        local_addr=('127.0.0.1', 9999))

    try:
        await asyncio.sleep(3600)  # Serve for 1 hour.
    finally:
        transport.close()


asyncio.run(main())
@end example

@node UDP Echo Client,Connecting Existing Sockets,UDP Echo Server,Examples<17>
@anchor{library/asyncio-protocol asyncio-udp-echo-client-protocol}@anchor{2a3b}@anchor{library/asyncio-protocol udp-echo-client}@anchor{2aa8}
@subsubsection UDP Echo Client


A UDP echo client, using the @ref{1b9,,loop.create_datagram_endpoint()}
method, sends data and closes the transport when it receives the answer:

@example
import asyncio


class EchoClientProtocol:
    def __init__(self, message, on_con_lost):
        self.message = message
        self.on_con_lost = on_con_lost
        self.transport = None

    def connection_made(self, transport):
        self.transport = transport
        print('Send:', self.message)
        self.transport.sendto(self.message.encode())

    def datagram_received(self, data, addr):
        print("Received:", data.decode())

        print("Close the socket")
        self.transport.close()

    def error_received(self, exc):
        print('Error received:', exc)

    def connection_lost(self, exc):
        print("Connection closed")
        self.on_con_lost.set_result(True)


async def main():
    # Get a reference to the event loop as we plan to use
    # low-level APIs.
    loop = asyncio.get_running_loop()

    on_con_lost = loop.create_future()
    message = "Hello World!"

    transport, protocol = await loop.create_datagram_endpoint(
        lambda: EchoClientProtocol(message, on_con_lost),
        remote_addr=('127.0.0.1', 9999))

    try:
        await on_con_lost
    finally:
        transport.close()


asyncio.run(main())
@end example

@node Connecting Existing Sockets,loop subprocess_exec and SubprocessProtocol,UDP Echo Client,Examples<17>
@anchor{library/asyncio-protocol asyncio-example-create-connection}@anchor{29c3}@anchor{library/asyncio-protocol connecting-existing-sockets}@anchor{2aa9}
@subsubsection Connecting Existing Sockets


Wait until a socket receives data using the
@ref{4e0,,loop.create_connection()} method with a protocol:

@example
import asyncio
import socket


class MyProtocol(asyncio.Protocol):

    def __init__(self, on_con_lost):
        self.transport = None
        self.on_con_lost = on_con_lost

    def connection_made(self, transport):
        self.transport = transport

    def data_received(self, data):
        print("Received:", data.decode())

        # We are done: close the transport;
        # connection_lost() will be called automatically.
        self.transport.close()

    def connection_lost(self, exc):
        # The socket has been closed
        self.on_con_lost.set_result(True)


async def main():
    # Get a reference to the event loop as we plan to use
    # low-level APIs.
    loop = asyncio.get_running_loop()
    on_con_lost = loop.create_future()

    # Create a pair of connected sockets
    rsock, wsock = socket.socketpair()

    # Register the socket to wait for data.
    transport, protocol = await loop.create_connection(
        lambda: MyProtocol(on_con_lost), sock=rsock)

    # Simulate the reception of data from the network.
    loop.call_soon(wsock.send, 'abc'.encode())

    try:
        await protocol.on_con_lost
    finally:
        transport.close()
        wsock.close()

asyncio.run(main())
@end example


@subsubheading See also


The @ref{29c4,,watch a file descriptor for read events} example uses the low-level
@ref{29c5,,loop.add_reader()} method to register an FD.

The @ref{29c1,,register an open socket to wait for data using streams} example uses high-level streams
created by the @ref{636,,open_connection()} function in a coroutine.


@node loop subprocess_exec and SubprocessProtocol,,Connecting Existing Sockets,Examples<17>
@anchor{library/asyncio-protocol asyncio-example-subprocess-proto}@anchor{2a01}@anchor{library/asyncio-protocol loop-subprocess-exec-and-subprocessprotocol}@anchor{2aaa}
@subsubsection loop.subprocess_exec() and SubprocessProtocol


An example of a subprocess protocol used to get the output of a
subprocess and to wait for the subprocess exit.

The subprocess is created by the @ref{297c,,loop.subprocess_exec()} method:

@example
import asyncio
import sys

class DateProtocol(asyncio.SubprocessProtocol):
    def __init__(self, exit_future):
        self.exit_future = exit_future
        self.output = bytearray()

    def pipe_data_received(self, fd, data):
        self.output.extend(data)

    def process_exited(self):
        self.exit_future.set_result(True)

async def get_date():
    # Get a reference to the event loop as we plan to use
    # low-level APIs.
    loop = asyncio.get_running_loop()

    code = 'import datetime; print(datetime.datetime.now())'
    exit_future = asyncio.Future(loop=loop)

    # Create the subprocess controlled by DateProtocol;
    # redirect the standard output into a pipe.
    transport, protocol = await loop.subprocess_exec(
        lambda: DateProtocol(exit_future),
        sys.executable, '-c', code,
        stdin=None, stderr=None)

    # Wait for the subprocess exit using the process_exited()
    # method of the protocol.
    await exit_future

    # Close the stdout pipe.
    transport.close()

    # Read the output which was collected by the
    # pipe_data_received() method of the protocol.
    data = bytes(protocol.output)
    return data.decode('ascii').rstrip()

date = asyncio.run(get_date())
print(f"Current date: @{date@}")
@end example

See also the @ref{2a00,,same example}
written using high-level APIs.

@node Policies,Platform Support,Transports and Protocols,asyncio — Asynchronous I/O
@anchor{library/asyncio-policy doc}@anchor{2aab}@anchor{library/asyncio-policy asyncio-policies}@anchor{2a21}@anchor{library/asyncio-policy policies}@anchor{2aac}
@subsubsection Policies


An event loop policy is a global per-process object that controls
the management of the event loop. Each event loop has a default
policy, which can be changed and customized using the policy API.

A policy defines the notion of @emph{context} and manages a
separate event loop per context. The default policy
defines @emph{context} to be the current thread.

By using a custom event loop policy, the behavior of
@ref{314,,get_event_loop()}, @ref{2a1f,,set_event_loop()}, and
@ref{2a20,,new_event_loop()} functions can be customized.

Policy objects should implement the APIs defined
in the @ref{2aad,,AbstractEventLoopPolicy} abstract base class.

@menu
* Getting and Setting the Policy:: 
* Policy Objects:: 
* Process Watchers:: 
* Custom Policies:: 

@end menu

@node Getting and Setting the Policy,Policy Objects,,Policies
@anchor{library/asyncio-policy getting-and-setting-the-policy}@anchor{2aae}
@subsubsection Getting and Setting the Policy


The following functions can be used to get and set the policy
for the current process:

@geindex get_event_loop_policy() (in module asyncio)
@anchor{library/asyncio-policy asyncio get_event_loop_policy}@anchor{2aaf}
@deffn {Function} asyncio.get_event_loop_policy ()

Return the current process-wide policy.
@end deffn

@geindex set_event_loop_policy() (in module asyncio)
@anchor{library/asyncio-policy asyncio set_event_loop_policy}@anchor{2ab0}
@deffn {Function} asyncio.set_event_loop_policy (policy)

Set the current process-wide policy to @emph{policy}.

If @emph{policy} is set to @code{None}, the default policy is restored.
@end deffn

@node Policy Objects,Process Watchers,Getting and Setting the Policy,Policies
@anchor{library/asyncio-policy policy-objects}@anchor{2ab1}
@subsubsection Policy Objects


The abstract event loop policy base class is defined as follows:

@geindex AbstractEventLoopPolicy (class in asyncio)
@anchor{library/asyncio-policy asyncio AbstractEventLoopPolicy}@anchor{2aad}
@deffn {Class} asyncio.AbstractEventLoopPolicy

An abstract base class for asyncio policies.

@geindex get_event_loop() (asyncio.AbstractEventLoopPolicy method)
@anchor{library/asyncio-policy asyncio AbstractEventLoopPolicy get_event_loop}@anchor{2ab2}
@deffn {Method} get_event_loop ()

Get the event loop for the current context.

Return an event loop object implementing the
@ref{462,,AbstractEventLoop} interface.

This method should never return @code{None}.


Changed in version 3.6.

@end deffn

@geindex set_event_loop() (asyncio.AbstractEventLoopPolicy method)
@anchor{library/asyncio-policy asyncio AbstractEventLoopPolicy set_event_loop}@anchor{2ab3}
@deffn {Method} set_event_loop (loop)

Set the event loop for the current context to @emph{loop}.
@end deffn

@geindex new_event_loop() (asyncio.AbstractEventLoopPolicy method)
@anchor{library/asyncio-policy asyncio AbstractEventLoopPolicy new_event_loop}@anchor{2ab4}
@deffn {Method} new_event_loop ()

Create and return a new event loop object.

This method should never return @code{None}.
@end deffn

@geindex get_child_watcher() (asyncio.AbstractEventLoopPolicy method)
@anchor{library/asyncio-policy asyncio AbstractEventLoopPolicy get_child_watcher}@anchor{2ab5}
@deffn {Method} get_child_watcher ()

Get a child process watcher object.

Return a watcher object implementing the
@ref{2ab6,,AbstractChildWatcher} interface.

This function is Unix specific.
@end deffn

@geindex set_child_watcher() (asyncio.AbstractEventLoopPolicy method)
@anchor{library/asyncio-policy asyncio AbstractEventLoopPolicy set_child_watcher}@anchor{2ab7}
@deffn {Method} set_child_watcher (watcher)

Set the current child process watcher to @emph{watcher}.

This function is Unix specific.
@end deffn
@end deffn

asyncio ships with the following built-in policies:

@geindex DefaultEventLoopPolicy (class in asyncio)
@anchor{library/asyncio-policy asyncio DefaultEventLoopPolicy}@anchor{2ab8}
@deffn {Class} asyncio.DefaultEventLoopPolicy

The default asyncio policy.  Uses @ref{29fd,,SelectorEventLoop}
on Unix and @ref{4da,,ProactorEventLoop} on Windows.

There is no need to install the default policy manually. asyncio
is configured to use the default policy automatically.


Changed in version 3.8: On Windows, @ref{4da,,ProactorEventLoop} is now used by default.

@end deffn

@geindex WindowsSelectorEventLoopPolicy (class in asyncio)
@anchor{library/asyncio-policy asyncio WindowsSelectorEventLoopPolicy}@anchor{2ab9}
@deffn {Class} asyncio.WindowsSelectorEventLoopPolicy

An alternative event loop policy that uses the
@ref{29fd,,SelectorEventLoop} event loop implementation.

@ref{1629,,Availability}: Windows.
@end deffn

@geindex WindowsProactorEventLoopPolicy (class in asyncio)
@anchor{library/asyncio-policy asyncio WindowsProactorEventLoopPolicy}@anchor{2aba}
@deffn {Class} asyncio.WindowsProactorEventLoopPolicy

An alternative event loop policy that uses the
@ref{4da,,ProactorEventLoop} event loop implementation.

@ref{1629,,Availability}: Windows.
@end deffn

@node Process Watchers,Custom Policies,Policy Objects,Policies
@anchor{library/asyncio-policy asyncio-watchers}@anchor{29fe}@anchor{library/asyncio-policy process-watchers}@anchor{2abb}
@subsubsection Process Watchers


A process watcher allows customization of how an event loop monitors
child processes on Unix. Specifically, the event loop needs to know
when a child process has exited.

In asyncio, child processes are created with
@ref{591,,create_subprocess_exec()} and @ref{297c,,loop.subprocess_exec()}
functions.

asyncio defines the @ref{2ab6,,AbstractChildWatcher} abstract base class, which child
watchers should implement, and has four different implementations:
@ref{29ff,,ThreadedChildWatcher} (configured to be used by default),
@ref{2abc,,MultiLoopChildWatcher}, @ref{2abd,,SafeChildWatcher}, and
@ref{2abe,,FastChildWatcher}.

See also the @ref{29f5,,Subprocess and Threads}
section.

The following two functions can be used to customize the child process watcher
implementation used by the asyncio event loop:

@geindex get_child_watcher() (in module asyncio)
@anchor{library/asyncio-policy asyncio get_child_watcher}@anchor{2abf}
@deffn {Function} asyncio.get_child_watcher ()

Return the current child watcher for the current policy.
@end deffn

@geindex set_child_watcher() (in module asyncio)
@anchor{library/asyncio-policy asyncio set_child_watcher}@anchor{2ac0}
@deffn {Function} asyncio.set_child_watcher (watcher)

Set the current child watcher to @emph{watcher} for the current
policy.  @emph{watcher} must implement methods defined in the
@ref{2ab6,,AbstractChildWatcher} base class.
@end deffn

@cartouche
@quotation Note 
Third-party event loops implementations might not support
custom child watchers.  For such event loops, using
@ref{2ac0,,set_child_watcher()} might be prohibited or have no effect.
@end quotation
@end cartouche

@geindex AbstractChildWatcher (class in asyncio)
@anchor{library/asyncio-policy asyncio AbstractChildWatcher}@anchor{2ab6}
@deffn {Class} asyncio.AbstractChildWatcher

@geindex add_child_handler() (asyncio.AbstractChildWatcher method)
@anchor{library/asyncio-policy asyncio AbstractChildWatcher add_child_handler}@anchor{2ac1}
@deffn {Method} add_child_handler (pid, callback, *args)

Register a new child handler.

Arrange for @code{callback(pid, returncode, *args)} to be called
when a process with PID equal to @emph{pid} terminates.  Specifying
another callback for the same process replaces the previous
handler.

The @emph{callback} callable must be thread-safe.
@end deffn

@geindex remove_child_handler() (asyncio.AbstractChildWatcher method)
@anchor{library/asyncio-policy asyncio AbstractChildWatcher remove_child_handler}@anchor{2ac2}
@deffn {Method} remove_child_handler (pid)

Removes the handler for process with PID equal to @emph{pid}.

The function returns @code{True} if the handler was successfully
removed, @code{False} if there was nothing to remove.
@end deffn

@geindex attach_loop() (asyncio.AbstractChildWatcher method)
@anchor{library/asyncio-policy asyncio AbstractChildWatcher attach_loop}@anchor{2ac3}
@deffn {Method} attach_loop (loop)

Attach the watcher to an event loop.

If the watcher was previously attached to an event loop, then
it is first detached before attaching to the new loop.

Note: loop may be @code{None}.
@end deffn

@geindex is_active() (asyncio.AbstractChildWatcher method)
@anchor{library/asyncio-policy asyncio AbstractChildWatcher is_active}@anchor{2ac4}
@deffn {Method} is_active ()

Return @code{True} if the watcher is ready to use.

Spawning a subprocess with @emph{inactive} current child watcher raises
@ref{5af,,RuntimeError}.


New in version 3.8.

@end deffn

@geindex close() (asyncio.AbstractChildWatcher method)
@anchor{library/asyncio-policy asyncio AbstractChildWatcher close}@anchor{2ac5}
@deffn {Method} close ()

Close the watcher.

This method has to be called to ensure that underlying
resources are cleaned-up.
@end deffn
@end deffn

@geindex ThreadedChildWatcher (class in asyncio)
@anchor{library/asyncio-policy asyncio ThreadedChildWatcher}@anchor{29ff}
@deffn {Class} asyncio.ThreadedChildWatcher

This implementation starts a new waiting thread for every subprocess spawn.

It works reliably even when the asyncio event loop is run in a non-main OS thread.

There is no noticeable overhead when handling a big number of children (@emph{O(1)} each
time a child terminates), but starting a thread per process requires extra memory.

This watcher is used by default.


New in version 3.8.

@end deffn

@geindex MultiLoopChildWatcher (class in asyncio)
@anchor{library/asyncio-policy asyncio MultiLoopChildWatcher}@anchor{2abc}
@deffn {Class} asyncio.MultiLoopChildWatcher

This implementation registers a @code{SIGCHLD} signal handler on
instantiation. That can break third-party code that installs a custom handler for
@code{SIGCHLD} signal.

The watcher avoids disrupting other code spawning processes
by polling every process explicitly on a @code{SIGCHLD} signal.

There is no limitation for running subprocesses from different threads once the
watcher is installed.

The solution is safe but it has a significant overhead when
handling a big number of processes (@emph{O(n)} each time a
@code{SIGCHLD} is received).


New in version 3.8.

@end deffn

@geindex SafeChildWatcher (class in asyncio)
@anchor{library/asyncio-policy asyncio SafeChildWatcher}@anchor{2abd}
@deffn {Class} asyncio.SafeChildWatcher

This implementation uses active event loop from the main thread to handle
@code{SIGCHLD} signal. If the main thread has no running event loop another
thread cannot spawn a subprocess (@ref{5af,,RuntimeError} is raised).

The watcher avoids disrupting other code spawning processes
by polling every process explicitly on a @code{SIGCHLD} signal.

This solution is as safe as @ref{2abc,,MultiLoopChildWatcher} and has the same @emph{O(N)}
complexity but requires a running event loop in the main thread to work.
@end deffn

@geindex FastChildWatcher (class in asyncio)
@anchor{library/asyncio-policy asyncio FastChildWatcher}@anchor{2abe}
@deffn {Class} asyncio.FastChildWatcher

This implementation reaps every terminated processes by calling
@code{os.waitpid(-1)} directly, possibly breaking other code spawning
processes and waiting for their termination.

There is no noticeable overhead when handling a big number of
children (@emph{O(1)} each time a child terminates).

This solution requires a running event loop in the main thread to work, as
@ref{2abd,,SafeChildWatcher}.
@end deffn

@geindex PidfdChildWatcher (class in asyncio)
@anchor{library/asyncio-policy asyncio PidfdChildWatcher}@anchor{3d2}
@deffn {Class} asyncio.PidfdChildWatcher

This implementation polls process file descriptors (pidfds) to await child
process termination. In some respects, @ref{3d2,,PidfdChildWatcher} is a
“Goldilocks” child watcher implementation. It doesn’t require signals or
threads, doesn’t interfere with any processes launched outside the event
loop, and scales linearly with the number of subprocesses launched by the
event loop. The main disadvantage is that pidfds are specific to Linux, and
only work on recent (5.3+) kernels.


New in version 3.9.

@end deffn

@node Custom Policies,,Process Watchers,Policies
@anchor{library/asyncio-policy custom-policies}@anchor{2ac6}
@subsubsection Custom Policies


To implement a new event loop policy, it is recommended to subclass
@ref{2ab8,,DefaultEventLoopPolicy} and override the methods for which
custom behavior is wanted, e.g.:

@example
class MyEventLoopPolicy(asyncio.DefaultEventLoopPolicy):

    def get_event_loop(self):
        """Get the event loop.

        This may be None or an instance of EventLoop.
        """
        loop = super().get_event_loop()
        # Do something with loop ...
        return loop

asyncio.set_event_loop_policy(MyEventLoopPolicy())
@end example

@node Platform Support,High-level API Index,Policies,asyncio — Asynchronous I/O
@anchor{library/asyncio-platforms doc}@anchor{2ac7}@anchor{library/asyncio-platforms asyncio-platform-support}@anchor{2a45}@anchor{library/asyncio-platforms platform-support}@anchor{2ac8}
@subsubsection Platform Support


The @ref{9,,asyncio} module is designed to be portable,
but some platforms have subtle differences and limitations
due to the platforms’ underlying architecture and capabilities.

@menu
* All Platforms:: 
* Windows: Windows<55>. 
* macOS: macOS<30>. 

@end menu

@node All Platforms,Windows<55>,,Platform Support
@anchor{library/asyncio-platforms all-platforms}@anchor{2ac9}
@subsubsection All Platforms



@itemize *

@item 
@ref{29c5,,loop.add_reader()} and @ref{2a43,,loop.add_writer()}
cannot be used to monitor file I/O.
@end itemize

@node Windows<55>,macOS<30>,All Platforms,Platform Support
@anchor{library/asyncio-platforms windows}@anchor{2aca}
@subsubsection Windows


@strong{Source code:} Lib/asyncio/proactor_events.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/proactor_events.py},
Lib/asyncio/windows_events.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/windows_events.py},
Lib/asyncio/windows_utils.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/windows_utils.py}


__________________________________________________________________



Changed in version 3.8: On Windows, @ref{4da,,ProactorEventLoop} is now the default event loop.


All event loops on Windows do not support the following methods:


@itemize *

@item 
@ref{29a9,,loop.create_unix_connection()} and
@ref{631,,loop.create_unix_server()} are not supported.
The @ref{2a3a,,socket.AF_UNIX} socket family is specific to Unix.

@item 
@ref{297d,,loop.add_signal_handler()} and
@ref{2a4d,,loop.remove_signal_handler()} are not supported.
@end itemize

@ref{29fd,,SelectorEventLoop} has the following limitations:


@itemize *

@item 
@ref{2acb,,SelectSelector} is used to wait on socket events:
it supports sockets and is limited to 512 sockets.

@item 
@ref{29c5,,loop.add_reader()} and @ref{2a43,,loop.add_writer()} only accept
socket handles (e.g. pipe file descriptors are not supported).

@item 
Pipes are not supported, so the @ref{29e7,,loop.connect_read_pipe()}
and @ref{29e8,,loop.connect_write_pipe()} methods are not implemented.

@item 
@ref{2979,,Subprocesses} are not supported, i.e.
@ref{297c,,loop.subprocess_exec()} and @ref{29e5,,loop.subprocess_shell()}
methods are not implemented.
@end itemize

@ref{4da,,ProactorEventLoop} has the following limitations:


@itemize *

@item 
The @ref{29c5,,loop.add_reader()} and @ref{2a43,,loop.add_writer()}
methods are not supported.
@end itemize

The resolution of the monotonic clock on Windows is usually around 15.6
milliseconds.  The best resolution is 0.5 milliseconds. The resolution depends on the
hardware (availability of HPET@footnote{https://en.wikipedia.org/wiki/High_Precision_Event_Timer}) and on the
Windows configuration.

@menu
* Subprocess Support on Windows:: 

@end menu

@node Subprocess Support on Windows,,,Windows<55>
@anchor{library/asyncio-platforms asyncio-windows-subprocess}@anchor{29e6}@anchor{library/asyncio-platforms subprocess-support-on-windows}@anchor{2acc}
@subsubsection Subprocess Support on Windows


On Windows, the default event loop @ref{4da,,ProactorEventLoop} supports
subprocesses, whereas @ref{29fd,,SelectorEventLoop} does not.

The @ref{2ab7,,policy.set_child_watcher()} function is also
not supported, as @ref{4da,,ProactorEventLoop} has a different mechanism
to watch child processes.

@node macOS<30>,,Windows<55>,Platform Support
@anchor{library/asyncio-platforms macos}@anchor{2acd}
@subsubsection macOS


Modern macOS versions are fully supported.

@subsubheading macOS <= 10.8


On macOS 10.6, 10.7 and 10.8, the default event loop
uses @ref{2ace,,selectors.KqueueSelector}, which does not support
character devices on these versions.  The @ref{29fd,,SelectorEventLoop}
can be manually configured to use @ref{2acb,,SelectSelector}
or @ref{70e,,PollSelector} to support character devices on
these older versions of macOS.  Example:

@example
import asyncio
import selectors

selector = selectors.SelectSelector()
loop = asyncio.SelectorEventLoop(selector)
asyncio.set_event_loop(loop)
@end example

@node High-level API Index,Low-level API Index,Platform Support,asyncio — Asynchronous I/O
@anchor{library/asyncio-api-index doc}@anchor{33e}@anchor{library/asyncio-api-index high-level-api-index}@anchor{2acf}
@subsubsection High-level API Index


This page lists all high-level async/await enabled asyncio APIs.

@menu
* Tasks:: 
* Queues: Queues<2>. 
* Subprocesses: Subprocesses<2>. 
* Streams: Streams<2>. 
* Synchronization:: 
* Exceptions: Exceptions<11>. 

@end menu

@node Tasks,Queues<2>,,High-level API Index
@anchor{library/asyncio-api-index tasks}@anchor{2ad0}
@subsubsection Tasks


Utilities to run asyncio programs, create Tasks, and
await on multiple things with timeouts.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{3d1,,run()}

@tab

Create event loop, run a coroutine, close the loop.

@item

@ref{4dc,,create_task()}

@tab

Start an asyncio Task.

@item

@code{await} @ref{58a,,sleep()}

@tab

Sleep for a number of seconds.

@item

@code{await} @ref{31a,,gather()}

@tab

Schedule and wait for things concurrently.

@item

@code{await} @ref{3d5,,wait_for()}

@tab

Run with a timeout.

@item

@code{await} @ref{31b,,shield()}

@tab

Shield from cancellation.

@item

@code{await} @ref{440,,wait()}

@tab

Monitor for completion.

@item

@ref{458,,current_task()}

@tab

Return the current Task.

@item

@ref{459,,all_tasks()}

@tab

Return all tasks for an event loop.

@item

@ref{317,,Task}

@tab

Task object.

@item

@ref{3d3,,to_thread()}

@tab

Asynchronously run a function in a separate OS thread.

@item

@ref{348,,run_coroutine_threadsafe()}

@tab

Schedule a coroutine from another OS thread.

@item

@code{for in} @ref{31c,,as_completed()}

@tab

Monitor for completion with a @code{for} loop.

@end multitable


@subsubheading Examples



@itemize *

@item 
@ref{298a,,Using asyncio.gather() to run things in parallel}.

@item 
@ref{298e,,Using asyncio.wait_for() to enforce a timeout}.

@item 
@ref{299a,,Cancellation}.

@item 
@ref{2988,,Using asyncio.sleep()}.

@item 
See also the main @ref{2977,,Tasks documentation page}.
@end itemize

@node Queues<2>,Subprocesses<2>,Tasks,High-level API Index
@anchor{library/asyncio-api-index queues}@anchor{2ad1}
@subsubsection Queues


Queues should be used to distribute work amongst multiple asyncio Tasks,
implement connection pools, and pub/sub patterns.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{590,,Queue}

@tab

A FIFO queue.

@item

@ref{2a11,,PriorityQueue}

@tab

A priority queue.

@item

@ref{2a13,,LifoQueue}

@tab

A LIFO queue.

@end multitable


@subsubheading Examples



@itemize *

@item 
@ref{2a15,,Using asyncio.Queue to distribute workload between several Tasks}.

@item 
See also the @ref{297a,,Queues documentation page}.
@end itemize

@node Subprocesses<2>,Streams<2>,Queues<2>,High-level API Index
@anchor{library/asyncio-api-index subprocesses}@anchor{2ad2}
@subsubsection Subprocesses


Utilities to spawn subprocesses and run shell commands.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{await} @ref{591,,create_subprocess_exec()}

@tab

Create a subprocess.

@item

@code{await} @ref{592,,create_subprocess_shell()}

@tab

Run a shell command.

@end multitable


@subsubheading Examples



@itemize *

@item 
@ref{29e1,,Executing a shell command}.

@item 
See also the @ref{2979,,subprocess APIs}
documentation.
@end itemize

@node Streams<2>,Synchronization,Subprocesses<2>,High-level API Index
@anchor{library/asyncio-api-index streams}@anchor{2ad3}
@subsubsection Streams


High-level APIs to work with network IO.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{await} @ref{636,,open_connection()}

@tab

Establish a TCP connection.

@item

@code{await} @ref{29a8,,open_unix_connection()}

@tab

Establish a Unix socket connection.

@item

@code{await} @ref{637,,start_server()}

@tab

Start a TCP server.

@item

@code{await} @ref{29aa,,start_unix_server()}

@tab

Start a Unix socket server.

@item

@ref{31d,,StreamReader}

@tab

High-level async/await object to receive network data.

@item

@ref{1299,,StreamWriter}

@tab

High-level async/await object to send network data.

@end multitable


@subsubheading Examples



@itemize *

@item 
@ref{29a6,,Example TCP client}.

@item 
See also the @ref{2978,,streams APIs}
documentation.
@end itemize

@node Synchronization,Exceptions<11>,Streams<2>,High-level API Index
@anchor{library/asyncio-api-index synchronization}@anchor{2ad4}
@subsubsection Synchronization


Threading-like synchronization primitives that can be used in Tasks.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{58b,,Lock}

@tab

A mutex lock.

@item

@ref{58c,,Event}

@tab

An event object.

@item

@ref{58d,,Condition}

@tab

A condition object.

@item

@ref{58e,,Semaphore}

@tab

A semaphore.

@item

@ref{58f,,BoundedSemaphore}

@tab

A bounded semaphore.

@end multitable


@subsubheading Examples



@itemize *

@item 
@ref{29d0,,Using asyncio.Event}.

@item 
See also the documentation of asyncio
@ref{297b,,synchronization primitives}.
@end itemize

@node Exceptions<11>,,Synchronization,High-level API Index
@anchor{library/asyncio-api-index exceptions}@anchor{2ad5}
@subsubsection Exceptions



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{4d7,,asyncio.CancelledError}

@tab

Raised when a Task is cancelled. See also @ref{298c,,Task.cancel()}.

@end multitable


@subsubheading Examples



@itemize *

@item 
@ref{299a,,Handling CancelledError to run code on cancellation request}.

@item 
See also the full list of
@ref{2a17,,asyncio-specific exceptions}.
@end itemize

@node Low-level API Index,Developing with asyncio,High-level API Index,asyncio — Asynchronous I/O
@anchor{library/asyncio-llapi-index doc}@anchor{2ad6}@anchor{library/asyncio-llapi-index low-level-api-index}@anchor{2ad7}
@subsubsection Low-level API Index


This page lists all low-level asyncio APIs.

@menu
* Obtaining the Event Loop:: 
* Event Loop Methods: Event Loop Methods<2>. 
* Transports: Transports<2>. 
* Protocols: Protocols<2>. 
* Event Loop Policies:: 

@end menu

@node Obtaining the Event Loop,Event Loop Methods<2>,,Low-level API Index
@anchor{library/asyncio-llapi-index obtaining-the-event-loop}@anchor{2ad8}
@subsubsection Obtaining the Event Loop



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{315,,asyncio.get_running_loop()}

@tab

The @strong{preferred} function to get the running event loop.

@item

@ref{314,,asyncio.get_event_loop()}

@tab

Get an event loop instance (current or via the policy).

@item

@ref{2a1f,,asyncio.set_event_loop()}

@tab

Set the event loop as current via the current policy.

@item

@ref{2a20,,asyncio.new_event_loop()}

@tab

Create a new event loop.

@end multitable


@subsubheading Examples



@itemize *

@item 
@ref{2a6b,,Using asyncio.get_running_loop()}.
@end itemize

@node Event Loop Methods<2>,Transports<2>,Obtaining the Event Loop,Low-level API Index
@anchor{library/asyncio-llapi-index event-loop-methods}@anchor{2ad9}
@subsubsection Event Loop Methods


See also the main documentation section about the
@ref{8d1,,event loop methods}.

@subsubheading Lifecycle



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{7bc,,loop.run_until_complete()}

@tab

Run a Future/Task/awaitable until complete.

@item

@ref{2a29,,loop.run_forever()}

@tab

Run the event loop forever.

@item

@ref{7c3,,loop.stop()}

@tab

Stop the event loop.

@item

@ref{2a2b,,loop.close()}

@tab

Close the event loop.

@item

@ref{2a2a,,loop.is_running()}

@tab

Return @code{True} if the event loop is running.

@item

@ref{922,,loop.is_closed()}

@tab

Return @code{True} if the event loop is closed.

@item

@code{await} @ref{7c4,,loop.shutdown_asyncgens()}

@tab

Close asynchronous generators.

@end multitable


@subsubheading Debugging



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{920,,loop.set_debug()}

@tab

Enable or disable the debug mode.

@item

@ref{921,,loop.get_debug()}

@tab

Get the current debug mode.

@end multitable


@subsubheading Scheduling Callbacks



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{622,,loop.call_soon()}

@tab

Invoke a callback soon.

@item

@ref{623,,loop.call_soon_threadsafe()}

@tab

A thread-safe variant of @ref{622,,loop.call_soon()}.

@item

@ref{624,,loop.call_later()}

@tab

Invoke a callback @emph{after} the given time.

@item

@ref{625,,loop.call_at()}

@tab

Invoke a callback @emph{at} the given time.

@end multitable


@subsubheading Thread/Process Pool



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{await} @ref{3d4,,loop.run_in_executor()}

@tab

Run a CPU-bound or other blocking function in
a @ref{20,,concurrent.futures} executor.

@item

@ref{587,,loop.set_default_executor()}

@tab

Set the default executor for @ref{3d4,,loop.run_in_executor()}.

@end multitable


@subsubheading Tasks and Futures



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{7bf,,loop.create_future()}

@tab

Create a @ref{316,,Future} object.

@item

@ref{4dd,,loop.create_task()}

@tab

Schedule coroutine as a @ref{317,,Task}.

@item

@ref{924,,loop.set_task_factory()}

@tab

Set a factory used by @ref{4dd,,loop.create_task()} to
create @ref{317,,Tasks}.

@item

@ref{925,,loop.get_task_factory()}

@tab

Get the factory @ref{4dd,,loop.create_task()} uses
to create @ref{317,,Tasks}.

@end multitable


@subsubheading DNS



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{await} @ref{750,,loop.getaddrinfo()}

@tab

Asynchronous version of @ref{11cc,,socket.getaddrinfo()}.

@item

@code{await} @ref{751,,loop.getnameinfo()}

@tab

Asynchronous version of @ref{2a48,,socket.getnameinfo()}.

@end multitable


@subsubheading Networking and IPC



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{await} @ref{4e0,,loop.create_connection()}

@tab

Open a TCP connection.

@item

@code{await} @ref{630,,loop.create_server()}

@tab

Create a TCP server.

@item

@code{await} @ref{29a9,,loop.create_unix_connection()}

@tab

Open a Unix socket connection.

@item

@code{await} @ref{631,,loop.create_unix_server()}

@tab

Create a Unix socket server.

@item

@code{await} @ref{638,,loop.connect_accepted_socket()}

@tab

Wrap a @ref{8fd,,socket} into a @code{(transport, protocol)}
pair.

@item

@code{await} @ref{1b9,,loop.create_datagram_endpoint()}

@tab

Open a datagram (UDP) connection.

@item

@code{await} @ref{2a3f,,loop.sendfile()}

@tab

Send a file over a transport.

@item

@code{await} @ref{627,,loop.start_tls()}

@tab

Upgrade an existing connection to TLS.

@item

@code{await} @ref{29e7,,loop.connect_read_pipe()}

@tab

Wrap a read end of a pipe into a @code{(transport, protocol)} pair.

@item

@code{await} @ref{29e8,,loop.connect_write_pipe()}

@tab

Wrap a write end of a pipe into a @code{(transport, protocol)} pair.

@end multitable


@subsubheading Sockets



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{await} @ref{74d,,loop.sock_recv()}

@tab

Receive data from the @ref{8fd,,socket}.

@item

@code{await} @ref{628,,loop.sock_recv_into()}

@tab

Receive data from the @ref{8fd,,socket} into a buffer.

@item

@code{await} @ref{74e,,loop.sock_sendall()}

@tab

Send data to the @ref{8fd,,socket}.

@item

@code{await} @ref{928,,loop.sock_connect()}

@tab

Connect the @ref{8fd,,socket}.

@item

@code{await} @ref{74f,,loop.sock_accept()}

@tab

Accept a @ref{8fd,,socket} connection.

@item

@code{await} @ref{62b,,loop.sock_sendfile()}

@tab

Send a file over the @ref{8fd,,socket}.

@item

@ref{29c5,,loop.add_reader()}

@tab

Start watching a file descriptor for read availability.

@item

@ref{2a42,,loop.remove_reader()}

@tab

Stop watching a file descriptor for read availability.

@item

@ref{2a43,,loop.add_writer()}

@tab

Start watching a file descriptor for write availability.

@item

@ref{2a44,,loop.remove_writer()}

@tab

Stop watching a file descriptor for write availability.

@end multitable


@subsubheading Unix Signals



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{297d,,loop.add_signal_handler()}

@tab

Add a handler for a @ref{e5,,signal}.

@item

@ref{2a4d,,loop.remove_signal_handler()}

@tab

Remove a handler for a @ref{e5,,signal}.

@end multitable


@subsubheading Subprocesses



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{297c,,loop.subprocess_exec()}

@tab

Spawn a subprocess.

@item

@ref{29e5,,loop.subprocess_shell()}

@tab

Spawn a subprocess from a shell command.

@end multitable


@subsubheading Error Handling



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{2a51,,loop.call_exception_handler()}

@tab

Call the exception handler.

@item

@ref{2a50,,loop.set_exception_handler()}

@tab

Set a new exception handler.

@item

@ref{7c0,,loop.get_exception_handler()}

@tab

Get the current exception handler.

@item

@ref{2a52,,loop.default_exception_handler()}

@tab

The default exception handler implementation.

@end multitable


@subsubheading Examples



@itemize *

@item 
@ref{2a5b,,Using asyncio.get_event_loop() and loop.run_forever()}.

@item 
@ref{2a5d,,Using loop.call_later()}.

@item 
Using @code{loop.create_connection()} to implement
@ref{29bc,,an echo-client}.

@item 
Using @code{loop.create_connection()} to
@ref{29c3,,connect a socket}.

@item 
@ref{29c4,,Using add_reader() to watch an FD for read events}.

@item 
@ref{2a60,,Using loop.add_signal_handler()}.

@item 
@ref{2a01,,Using loop.subprocess_exec()}.
@end itemize

@node Transports<2>,Protocols<2>,Event Loop Methods<2>,Low-level API Index
@anchor{library/asyncio-llapi-index transports}@anchor{2ada}
@subsubsection Transports


All transports implement the following methods:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{2a78,,transport.close()}

@tab

Close the transport.

@item

@ref{7be,,transport.is_closing()}

@tab

Return @code{True} if the transport is closing or is closed.

@item

@ref{5bc,,transport.get_extra_info()}

@tab

Request for information about the transport.

@item

@ref{2a7d,,transport.set_protocol()}

@tab

Set a new protocol.

@item

@ref{2a7e,,transport.get_protocol()}

@tab

Return the current protocol.

@end multitable


Transports that can receive data (TCP and Unix connections,
pipes, etc).  Returned from methods like
@ref{4e0,,loop.create_connection()}, @ref{29a9,,loop.create_unix_connection()},
@ref{29e7,,loop.connect_read_pipe()}, etc:

@subsubheading Read Transports



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{63a,,transport.is_reading()}

@tab

Return @code{True} if the transport is receiving.

@item

@ref{63c,,transport.pause_reading()}

@tab

Pause receiving.

@item

@ref{63b,,transport.resume_reading()}

@tab

Resume receiving.

@end multitable


Transports that can Send data (TCP and Unix connections,
pipes, etc).  Returned from methods like
@ref{4e0,,loop.create_connection()}, @ref{29a9,,loop.create_unix_connection()},
@ref{29e8,,loop.connect_write_pipe()}, etc:

@subsubheading Write Transports



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{2a89,,transport.write()}

@tab

Write data to the transport.

@item

@ref{2a8a,,transport.writelines()}

@tab

Write buffers to the transport.

@item

@ref{2a83,,transport.can_write_eof()}

@tab

Return @ref{748,,True} if the transport supports sending EOF.

@item

@ref{2a84,,transport.write_eof()}

@tab

Close and send EOF after flushing buffered data.

@item

@ref{2a82,,transport.abort()}

@tab

Close the transport immediately.

@item

@ref{2a85,,transport.get_write_buffer_size()}

@tab

Return high and low water marks for write flow control.

@item

@ref{2a86,,transport.set_write_buffer_limits()}

@tab

Set new high and low water marks for write flow control.

@end multitable


Transports returned by @ref{1b9,,loop.create_datagram_endpoint()}:

@subsubheading Datagram Transports



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{2a8c,,transport.sendto()}

@tab

Send data to the remote peer.

@item

@ref{2a8d,,transport.abort()}

@tab

Close the transport immediately.

@end multitable


Low-level transport abstraction over subprocesses.
Returned by @ref{297c,,loop.subprocess_exec()} and
@ref{29e5,,loop.subprocess_shell()}:

@subsubheading Subprocess Transports



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{2a8f,,transport.get_pid()}

@tab

Return the subprocess process id.

@item

@ref{2a90,,transport.get_pipe_transport()}

@tab

Return the transport for the requested communication pipe
(@emph{stdin}, @emph{stdout}, or @emph{stderr}).

@item

@ref{2a91,,transport.get_returncode()}

@tab

Return the subprocess return code.

@item

@ref{2a92,,transport.kill()}

@tab

Kill the subprocess.

@item

@ref{2a94,,transport.send_signal()}

@tab

Send a signal to the subprocess.

@item

@ref{2a93,,transport.terminate()}

@tab

Stop the subprocess.

@item

@ref{2a95,,transport.close()}

@tab

Kill the subprocess and close all pipes.

@end multitable


@node Protocols<2>,Event Loop Policies,Transports<2>,Low-level API Index
@anchor{library/asyncio-llapi-index protocols}@anchor{2adb}
@subsubsection Protocols


Protocol classes can implement the following @strong{callback methods}:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{callback} @ref{2a38,,connection_made()}

@tab

Called when a connection is made.

@item

@code{callback} @ref{2a79,,connection_lost()}

@tab

Called when the connection is lost or closed.

@item

@code{callback} @ref{2a87,,pause_writing()}

@tab

Called when the transport’s buffer goes over the high water mark.

@item

@code{callback} @ref{2a88,,resume_writing()}

@tab

Called when the transport’s buffer drains below the low water mark.

@end multitable


@subsubheading Streaming Protocols (TCP, Unix Sockets, Pipes)



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{callback} @ref{2a80,,data_received()}

@tab

Called when some data is received.

@item

@code{callback} @ref{2a99,,eof_received()}

@tab

Called when an EOF is received.

@end multitable


@subsubheading Buffered Streaming Protocols



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{callback} @ref{2a9b,,get_buffer()}

@tab

Called to allocate a new receive buffer.

@item

@code{callback} @ref{2a9c,,buffer_updated()}

@tab

Called when the buffer was updated with the received data.

@item

@code{callback} @ref{2a9d,,eof_received()}

@tab

Called when an EOF is received.

@end multitable


@subsubheading Datagram Protocols



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{callback} @ref{2a9f,,datagram_received()}

@tab

Called when a datagram is received.

@item

@code{callback} @ref{2aa0,,error_received()}

@tab

Called when a previous send or receive operation raises an
@ref{2a1,,OSError}.

@end multitable


@subsubheading Subprocess Protocols



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{callback} @ref{2aa2,,pipe_data_received()}

@tab

Called when the child process writes data into its
@emph{stdout} or @emph{stderr} pipe.

@item

@code{callback} @ref{2aa3,,pipe_connection_lost()}

@tab

Called when one of the pipes communicating with
the child process is closed.

@item

@code{callback} @ref{2aa4,,process_exited()}

@tab

Called when the child process has exited.

@end multitable


@node Event Loop Policies,,Protocols<2>,Low-level API Index
@anchor{library/asyncio-llapi-index event-loop-policies}@anchor{2adc}
@subsubsection Event Loop Policies


Policies is a low-level mechanism to alter the behavior of
functions like @ref{314,,asyncio.get_event_loop()}.  See also
the main @ref{2a21,,policies section} for more
details.

@subsubheading Accessing Policies



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@ref{2aaf,,asyncio.get_event_loop_policy()}

@tab

Return the current process-wide policy.

@item

@ref{2ab0,,asyncio.set_event_loop_policy()}

@tab

Set a new process-wide policy.

@item

@ref{2aad,,AbstractEventLoopPolicy}

@tab

Base class for policy objects.

@end multitable


@node Developing with asyncio,,Low-level API Index,asyncio — Asynchronous I/O
@anchor{library/asyncio-dev doc}@anchor{2add}@anchor{library/asyncio-dev asyncio-dev}@anchor{2ade}@anchor{library/asyncio-dev developing-with-asyncio}@anchor{2adf}
@subsubsection Developing with asyncio


Asynchronous programming is different from classic “sequential”
programming.

This page lists common mistakes and traps and explains how
to avoid them.

@menu
* Debug Mode:: 
* Concurrency and Multithreading:: 
* Running Blocking Code:: 
* Logging: Logging<3>. 
* Detect never-awaited coroutines:: 
* Detect never-retrieved exceptions:: 

@end menu

@node Debug Mode,Concurrency and Multithreading,,Developing with asyncio
@anchor{library/asyncio-dev asyncio-debug-mode}@anchor{1625}@anchor{library/asyncio-dev debug-mode}@anchor{2ae0}
@subsubsection Debug Mode


By default asyncio runs in production mode.  In order to ease
the development asyncio has a @emph{debug mode}.

There are several ways to enable asyncio debug mode:


@itemize *

@item 
Setting the 
@geindex PYTHONASYNCIODEBUG
@geindex environment variable; PYTHONASYNCIODEBUG
@ref{1624,,PYTHONASYNCIODEBUG} environment variable to @code{1}.

@item 
Using the @ref{3ac,,Python Development Mode}.

@item 
Passing @code{debug=True} to @ref{3d1,,asyncio.run()}.

@item 
Calling @ref{920,,loop.set_debug()}.
@end itemize

In addition to enabling the debug mode, consider also:


@itemize *

@item 
setting the log level of the @ref{2ae1,,asyncio logger} to
@code{logging.DEBUG}, for example the following snippet of code
can be run at startup of the application:

@example
logging.basicConfig(level=logging.DEBUG)
@end example

@item 
configuring the @ref{12d,,warnings} module to display
@ref{77b,,ResourceWarning} warnings.  One way of doing that is by
using the @ref{3a8,,-W} @code{default} command line option.
@end itemize

When the debug mode is enabled:


@itemize *

@item 
asyncio checks for @ref{2ae2,,coroutines that were not awaited} and logs them; this mitigates
the “forgotten await” pitfall.

@item 
Many non-threadsafe asyncio APIs (such as @ref{622,,loop.call_soon()} and
@ref{625,,loop.call_at()} methods) raise an exception if they are called
from a wrong thread.

@item 
The execution time of the I/O selector is logged if it takes too long to
perform an I/O operation.

@item 
Callbacks taking longer than 100 milliseconds are logged.  The
@code{loop.slow_callback_duration} attribute can be used to set the
minimum execution duration in seconds that is considered “slow”.
@end itemize

@node Concurrency and Multithreading,Running Blocking Code,Debug Mode,Developing with asyncio
@anchor{library/asyncio-dev asyncio-multithreading}@anchor{2993}@anchor{library/asyncio-dev concurrency-and-multithreading}@anchor{2ae3}
@subsubsection Concurrency and Multithreading


An event loop runs in a thread (typically the main thread) and executes
all callbacks and Tasks in its thread.  While a Task is running in the
event loop, no other Tasks can run in the same thread.  When a Task
executes an @code{await} expression, the running Task gets suspended, and
the event loop executes the next Task.

To schedule a @ref{2a2d,,callback} from another OS thread, the
@ref{623,,loop.call_soon_threadsafe()} method should be used. Example:

@example
loop.call_soon_threadsafe(callback, *args)
@end example

Almost all asyncio objects are not thread safe, which is typically
not a problem unless there is code that works with them from outside
of a Task or a callback.  If there’s a need for such code to call a
low-level asyncio API, the @ref{623,,loop.call_soon_threadsafe()} method
should be used, e.g.:

@example
loop.call_soon_threadsafe(fut.cancel)
@end example

To schedule a coroutine object from a different OS thread, the
@ref{348,,run_coroutine_threadsafe()} function should be used. It returns a
@ref{d57,,concurrent.futures.Future} to access the result:

@example
async def coro_func():
     return await asyncio.sleep(1, 42)

# Later in another OS thread:

future = asyncio.run_coroutine_threadsafe(coro_func(), loop)
# Wait for the result:
result = future.result()
@end example

To handle signals and to execute subprocesses, the event loop must be
run in the main thread.

The @ref{3d4,,loop.run_in_executor()} method can be used with a
@ref{1c7,,concurrent.futures.ThreadPoolExecutor} to execute
blocking code in a different OS thread without blocking the OS thread
that the event loop runs in.

There is currently no way to schedule coroutines or callbacks directly
from a different process (such as one started with
@ref{b3,,multiprocessing}). The @ref{8d1,,Event Loop Methods}
section lists APIs that can read from pipes and watch file descriptors
without blocking the event loop. In addition, asyncio’s
@ref{2979,,Subprocess} APIs provide a way to start a
process and communicate with it from the event loop. Lastly, the
aforementioned @ref{3d4,,loop.run_in_executor()} method can also be used
with a @ref{3da,,concurrent.futures.ProcessPoolExecutor} to execute
code in a different process.

@node Running Blocking Code,Logging<3>,Concurrency and Multithreading,Developing with asyncio
@anchor{library/asyncio-dev asyncio-handle-blocking}@anchor{2ae4}@anchor{library/asyncio-dev running-blocking-code}@anchor{2ae5}
@subsubsection Running Blocking Code


Blocking (CPU-bound) code should not be called directly.  For example,
if a function performs a CPU-intensive calculation for 1 second,
all concurrent asyncio Tasks and IO operations would be delayed
by 1 second.

An executor can be used to run a task in a different thread or even in
a different process to avoid blocking the OS thread with the
event loop.  See the @ref{3d4,,loop.run_in_executor()} method for more
details.

@node Logging<3>,Detect never-awaited coroutines,Running Blocking Code,Developing with asyncio
@anchor{library/asyncio-dev asyncio-logger}@anchor{2ae1}@anchor{library/asyncio-dev logging}@anchor{2ae6}
@subsubsection Logging


asyncio uses the @ref{a6,,logging} module and all logging is performed
via the @code{"asyncio"} logger.

The default log level is @code{logging.INFO}, which can be easily
adjusted:

@example
logging.getLogger("asyncio").setLevel(logging.WARNING)
@end example

@node Detect never-awaited coroutines,Detect never-retrieved exceptions,Logging<3>,Developing with asyncio
@anchor{library/asyncio-dev asyncio-coroutine-not-scheduled}@anchor{2ae2}@anchor{library/asyncio-dev detect-never-awaited-coroutines}@anchor{2ae7}
@subsubsection Detect never-awaited coroutines


When a coroutine function is called, but not awaited
(e.g. @code{coro()} instead of @code{await coro()})
or the coroutine is not scheduled with @ref{4dc,,asyncio.create_task()}, asyncio
will emit a @ref{5b5,,RuntimeWarning}:

@example
import asyncio

async def test():
    print("never scheduled")

async def main():
    test()

asyncio.run(main())
@end example

Output:

@example
test.py:7: RuntimeWarning: coroutine 'test' was never awaited
  test()
@end example

Output in debug mode:

@example
test.py:7: RuntimeWarning: coroutine 'test' was never awaited
Coroutine created at (most recent call last)
  File "../t.py", line 9, in <module>
    asyncio.run(main(), debug=True)

  < .. >

  File "../t.py", line 7, in main
    test()
  test()
@end example

The usual fix is to either await the coroutine or call the
@ref{4dc,,asyncio.create_task()} function:

@example
async def main():
    await test()
@end example

@node Detect never-retrieved exceptions,,Detect never-awaited coroutines,Developing with asyncio
@anchor{library/asyncio-dev detect-never-retrieved-exceptions}@anchor{2ae8}
@subsubsection Detect never-retrieved exceptions


If a @ref{2998,,Future.set_exception()} is called but the Future object is
never awaited on, the exception would never be propagated to the
user code.  In this case, asyncio would emit a log message when the
Future object is garbage collected.

Example of an unhandled exception:

@example
import asyncio

async def bug():
    raise Exception("not consumed")

async def main():
    asyncio.create_task(bug())

asyncio.run(main())
@end example

Output:

@example
Task exception was never retrieved
future: <Task finished coro=<bug() done, defined at test.py:3>
  exception=Exception('not consumed')>

Traceback (most recent call last):
  File "test.py", line 4, in bug
    raise Exception("not consumed")
Exception: not consumed
@end example

@ref{1625,,Enable the debug mode} to get the
traceback where the task was created:

@example
asyncio.run(main(), debug=True)
@end example

Output in debug mode:

@example
Task exception was never retrieved
future: <Task finished coro=<bug() done, defined at test.py:3>
    exception=Exception('not consumed') created at asyncio/tasks.py:321>

source_traceback: Object created at (most recent call last):
  File "../t.py", line 9, in <module>
    asyncio.run(main(), debug=True)

< .. >

Traceback (most recent call last):
  File "../t.py", line 4, in bug
    raise Exception("not consumed")
Exception: not consumed
@end example

@cartouche
@quotation Note 
The source code for asyncio can be found in Lib/asyncio/@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/}.
@end quotation
@end cartouche

@node socket — Low-level networking interface,ssl — TLS/SSL wrapper for socket objects,asyncio — Asynchronous I/O,Networking and Interprocess Communication
@anchor{library/socket doc}@anchor{2ae9}@anchor{library/socket module-socket}@anchor{ea}@anchor{library/socket socket-low-level-networking-interface}@anchor{2aea}
@subsection @code{socket} — Low-level networking interface


@geindex module; socket

@strong{Source code:} Lib/socket.py@footnote{https://github.com/python/cpython/tree/main/Lib/socket.py}


__________________________________________________________________


This module provides access to the BSD @emph{socket} interface. It is available on
all modern Unix systems, Windows, MacOS, and probably additional platforms.

@cartouche
@quotation Note 
Some behavior may be platform dependent, since calls are made to the operating
system socket APIs.
@end quotation
@end cartouche

@geindex object; socket

The Python interface is a straightforward transliteration of the Unix system
call and library interface for sockets to Python’s object-oriented style: the
@code{socket()} function returns a @emph{socket object} whose methods implement
the various socket system calls.  Parameter types are somewhat higher-level than
in the C interface: as with @code{read()} and @code{write()} operations on Python
files, buffer allocation on receive operations is automatic, and buffer length
is implicit on send operations.


@subsubheading See also



@table @asis

@item Module @ref{eb,,socketserver}

Classes that simplify writing network servers.

@item Module @ref{ee,,ssl}

A TLS/SSL wrapper for socket objects.
@end table


@menu
* Socket families:: 
* Module contents:: 
* Socket Objects:: 
* Notes on socket timeouts:: 
* Example: Example<8>. 

@end menu

@node Socket families,Module contents,,socket — Low-level networking interface
@anchor{library/socket socket-families}@anchor{2aeb}
@subsubsection Socket families


Depending on the system and the build options, various socket families
are supported by this module.

The address format required by a particular socket object is automatically
selected based on the address family specified when the socket object was
created.  Socket addresses are represented as follows:


@itemize -

@item 
The address of an @ref{2a3a,,AF_UNIX} socket bound to a file system node
is represented as a string, using the file system encoding and the
@code{'surrogateescape'} error handler (see 
@geindex Python Enhancement Proposals; PEP 383
PEP 383@footnote{https://www.python.org/dev/peps/pep-0383}).  An address in
Linux’s abstract namespace is returned as a @ref{87e,,bytes-like object} with
an initial null byte; note that sockets in this namespace can
communicate with normal file system sockets, so programs intended to
run on Linux may need to deal with both types of address.  A string or
bytes-like object can be used for either type of address when
passing it as an argument.


Changed in version 3.3: Previously, @ref{2a3a,,AF_UNIX} socket paths were assumed to use UTF-8
encoding.



Changed in version 3.5: Writable @ref{87e,,bytes-like object} is now accepted.

@end itemize
@anchor{library/socket host-port}@anchor{2aec}

@itemize -

@item 
A pair @code{(host, port)} is used for the @ref{2a34,,AF_INET} address family,
where @emph{host} is a string representing either a hostname in internet domain
notation like @code{'daring.cwi.nl'} or an IPv4 address like @code{'100.50.200.5'},
and @emph{port} is an integer.


@itemize -

@item 
For IPv4 addresses, two special forms are accepted instead of a host
address: @code{'@w{'}} represents @code{INADDR_ANY}, which is used to bind to all
interfaces, and the string @code{'<broadcast>'} represents
@code{INADDR_BROADCAST}.  This behavior is not compatible with IPv6,
therefore, you may want to avoid these if you intend to support IPv6 with your
Python programs.
@end itemize

@item 
For @ref{2a35,,AF_INET6} address family, a four-tuple @code{(host, port, flowinfo,
scope_id)} is used, where @emph{flowinfo} and @emph{scope_id} represent the @code{sin6_flowinfo}
and @code{sin6_scope_id} members in @code{struct sockaddr_in6} in C.  For
@ref{ea,,socket} module methods, @emph{flowinfo} and @emph{scope_id} can be omitted just for
backward compatibility.  Note, however, omission of @emph{scope_id} can cause problems
in manipulating scoped IPv6 addresses.


Changed in version 3.7: For multicast addresses (with @emph{scope_id} meaningful) @emph{address} may not contain
@code{%scope_id} (or @code{zone id}) part. This information is superfluous and may
be safely omitted (recommended).


@item 
@code{AF_NETLINK} sockets are represented as pairs @code{(pid, groups)}.

@item 
Linux-only support for TIPC is available using the @code{AF_TIPC}
address family.  TIPC is an open, non-IP based networked protocol designed
for use in clustered computer environments.  Addresses are represented by a
tuple, and the fields depend on the address type. The general tuple form is
@code{(addr_type, v1, v2, v3 [, scope])}, where:


@itemize -

@item 
@emph{addr_type} is one of @code{TIPC_ADDR_NAMESEQ}, @code{TIPC_ADDR_NAME},
or @code{TIPC_ADDR_ID}.

@item 
@emph{scope} is one of @code{TIPC_ZONE_SCOPE}, @code{TIPC_CLUSTER_SCOPE}, and
@code{TIPC_NODE_SCOPE}.

@item 
If @emph{addr_type} is @code{TIPC_ADDR_NAME}, then @emph{v1} is the server type, @emph{v2} is
the port identifier, and @emph{v3} should be 0.

If @emph{addr_type} is @code{TIPC_ADDR_NAMESEQ}, then @emph{v1} is the server type, @emph{v2}
is the lower port number, and @emph{v3} is the upper port number.

If @emph{addr_type} is @code{TIPC_ADDR_ID}, then @emph{v1} is the node, @emph{v2} is the
reference, and @emph{v3} should be set to 0.
@end itemize

@item 
A tuple @code{(interface, )} is used for the @ref{2aed,,AF_CAN} address family,
where @emph{interface} is a string representing a network interface name like
@code{'can0'}. The network interface name @code{'@w{'}} can be used to receive packets
from all network interfaces of this family.


@itemize -

@item 
@ref{2aee,,CAN_ISOTP} protocol require a tuple @code{(interface, rx_addr, tx_addr)}
where both additional parameters are unsigned long integer that represent a
CAN identifier (standard or extended).

@item 
@ref{426,,CAN_J1939} protocol require a tuple @code{(interface, name, pgn, addr)}
where additional parameters are 64-bit unsigned integer representing the
ECU name, a 32-bit unsigned integer representing the Parameter Group Number
(PGN), and an 8-bit integer representing the address.
@end itemize

@item 
A string or a tuple @code{(id, unit)} is used for the @code{SYSPROTO_CONTROL}
protocol of the @code{PF_SYSTEM} family. The string is the name of a
kernel control using a dynamically-assigned ID. The tuple can be used if ID
and unit number of the kernel control are known or if a registered ID is
used.


New in version 3.3.


@item 
@code{AF_BLUETOOTH} supports the following protocols and address
formats:


@itemize -

@item 
@code{BTPROTO_L2CAP} accepts @code{(bdaddr, psm)} where @code{bdaddr} is
the Bluetooth address as a string and @code{psm} is an integer.

@item 
@code{BTPROTO_RFCOMM} accepts @code{(bdaddr, channel)} where @code{bdaddr}
is the Bluetooth address as a string and @code{channel} is an integer.

@item 
@code{BTPROTO_HCI} accepts @code{(device_id,)} where @code{device_id} is
either an integer or a string with the Bluetooth address of the
interface. (This depends on your OS; NetBSD and DragonFlyBSD expect
a Bluetooth address while everything else expects an integer.)


Changed in version 3.2: NetBSD and DragonFlyBSD support added.


@item 
@code{BTPROTO_SCO} accepts @code{bdaddr} where @code{bdaddr} is a
@ref{2f0,,bytes} object containing the Bluetooth address in a
string format. (ex. @code{b'12:23:34:45:56:67'}) This protocol is not
supported under FreeBSD.
@end itemize

@item 
@ref{81f,,AF_ALG} is a Linux-only socket based interface to Kernel
cryptography. An algorithm socket is configured with a tuple of two to four
elements @code{(type, name [, feat [, mask]])}, where:


@itemize -

@item 
@emph{type} is the algorithm type as string, e.g. @code{aead}, @code{hash},
@code{skcipher} or @code{rng}.

@item 
@emph{name} is the algorithm name and operation mode as string, e.g.
@code{sha256}, @code{hmac(sha256)}, @code{cbc(aes)} or @code{drbg_nopr_ctr_aes256}.

@item 
@emph{feat} and @emph{mask} are unsigned 32bit integers.
@end itemize

@ref{1629,,Availability}: Linux 2.6.38, some algorithm types require more recent Kernels.


New in version 3.6.


@item 
@ref{6a2,,AF_VSOCK} allows communication between virtual machines and
their hosts. The sockets are represented as a @code{(CID, port)} tuple
where the context ID or CID and port are integers.

@ref{1629,,Availability}: Linux >= 4.8 QEMU >= 2.8 ESX >= 4.0 ESX Workstation >= 6.5.


New in version 3.7.


@item 
@ref{2aef,,AF_PACKET} is a low-level interface directly to network devices.
The packets are represented by the tuple
@code{(ifname, proto[, pkttype[, hatype[, addr]]])} where:


@itemize -

@item 
@emph{ifname} - String specifying the device name.

@item 
@emph{proto} - An in network-byte-order integer specifying the Ethernet
protocol number.

@item 
@emph{pkttype} - Optional integer specifying the packet type:


@itemize -

@item 
@code{PACKET_HOST} (the default) - Packet addressed to the local host.

@item 
@code{PACKET_BROADCAST} - Physical-layer broadcast packet.

@item 
@code{PACKET_MULTICAST} - Packet sent to a physical-layer multicast address.

@item 
@code{PACKET_OTHERHOST} - Packet to some other host that has been caught by
a device driver in promiscuous mode.

@item 
@code{PACKET_OUTGOING} - Packet originating from the local host that is
looped back to a packet socket.
@end itemize

@item 
@emph{hatype} - Optional integer specifying the ARP hardware address type.

@item 
@emph{addr} - Optional bytes-like object specifying the hardware physical
address, whose interpretation depends on the device.
@end itemize

@quotation

@ref{1629,,Availability}: Linux >= 2.2.
@end quotation

@item 
@ref{2af0,,AF_QIPCRTR} is a Linux-only socket based interface for communicating
with services running on co-processors in Qualcomm platforms. The address
family is represented as a @code{(node, port)} tuple where the @emph{node} and @emph{port}
are non-negative integers.

@quotation

@ref{1629,,Availability}: Linux >= 4.7.
@end quotation


New in version 3.8.


@item 
@code{IPPROTO_UDPLITE} is a variant of UDP which allows you to specify
what portion of a packet is covered with the checksum. It adds two socket
options that you can change.
@code{self.setsockopt(IPPROTO_UDPLITE, UDPLITE_SEND_CSCOV, length)} will
change what portion of outgoing packets are covered by the checksum and
@code{self.setsockopt(IPPROTO_UDPLITE, UDPLITE_RECV_CSCOV, length)} will
filter out packets which cover too little of their data. In both cases
@code{length} should be in @code{range(8, 2**16, 8)}.

Such a socket should be constructed with
@code{socket(AF_INET, SOCK_DGRAM, IPPROTO_UDPLITE)} for IPv4 or
@code{socket(AF_INET6, SOCK_DGRAM, IPPROTO_UDPLITE)} for IPv6.

@ref{1629,,Availability}: Linux >= 2.6.20, FreeBSD >= 10.1-RELEASE


New in version 3.9.

@end itemize

If you use a hostname in the @emph{host} portion of IPv4/v6 socket address, the
program may show a nondeterministic behavior, as Python uses the first address
returned from the DNS resolution.  The socket address will be resolved
differently into an actual IPv4/v6 address, depending on the results from DNS
resolution and/or the host configuration.  For deterministic behavior use a
numeric address in @emph{host} portion.

All errors raise exceptions.  The normal exceptions for invalid argument types
and out-of-memory conditions can be raised; starting from Python 3.3, errors
related to socket or address semantics raise @ref{2a1,,OSError} or one of its
subclasses (they used to raise @ref{beb,,socket.error}).

Non-blocking mode is supported through @ref{23a8,,setblocking()}.  A
generalization of this based on timeouts is supported through
@ref{2af1,,settimeout()}.

@node Module contents,Socket Objects,Socket families,socket — Low-level networking interface
@anchor{library/socket module-contents}@anchor{2af2}
@subsubsection Module contents


The module @ref{ea,,socket} exports the following elements.

@menu
* Exceptions: Exceptions<12>. 
* Constants: Constants<8>. 
* Functions: Functions<6>. 

@end menu

@node Exceptions<12>,Constants<8>,,Module contents
@anchor{library/socket exceptions}@anchor{2af3}
@subsubsection Exceptions


@geindex error
@anchor{library/socket socket error}@anchor{beb}
@deffn {Exception} socket.error

A deprecated alias of @ref{2a1,,OSError}.


Changed in version 3.3: Following 
@geindex Python Enhancement Proposals; PEP 3151
PEP 3151@footnote{https://www.python.org/dev/peps/pep-3151}, this class was made an alias of @ref{2a1,,OSError}.

@end deffn

@geindex herror
@anchor{library/socket socket herror}@anchor{2af4}
@deffn {Exception} socket.herror

A subclass of @ref{2a1,,OSError}, this exception is raised for
address-related errors, i.e. for functions that use @emph{h_errno} in the POSIX
C API, including @ref{1145,,gethostbyname_ex()} and @ref{1144,,gethostbyaddr()}.
The accompanying value is a pair @code{(h_errno, string)} representing an
error returned by a library call.  @emph{h_errno} is a numeric value, while
@emph{string} represents the description of @emph{h_errno}, as returned by the
@code{hstrerror()} C function.


Changed in version 3.3: This class was made a subclass of @ref{2a1,,OSError}.

@end deffn

@geindex gaierror
@anchor{library/socket socket gaierror}@anchor{2af5}
@deffn {Exception} socket.gaierror

A subclass of @ref{2a1,,OSError}, this exception is raised for
address-related errors by @ref{11cc,,getaddrinfo()} and @ref{2a48,,getnameinfo()}.
The accompanying value is a pair @code{(error, string)} representing an error
returned by a library call.  @emph{string} represents the description of
@emph{error}, as returned by the @code{gai_strerror()} C function.  The
numeric @emph{error} value will match one of the @code{EAI_*} constants
defined in this module.


Changed in version 3.3: This class was made a subclass of @ref{2a1,,OSError}.

@end deffn

@geindex timeout
@anchor{library/socket socket timeout}@anchor{2b7}
@deffn {Exception} socket.timeout

A deprecated alias of @ref{2b8,,TimeoutError}.

A subclass of @ref{2a1,,OSError}, this exception is raised when a timeout
occurs on a socket which has had timeouts enabled via a prior call to
@ref{2af1,,settimeout()} (or implicitly through
@ref{2af6,,setdefaulttimeout()}).  The accompanying value is a string
whose value is currently always “timed out”.


Changed in version 3.3: This class was made a subclass of @ref{2a1,,OSError}.



Changed in version 3.10: This class was made an alias of @ref{2b8,,TimeoutError}.

@end deffn

@node Constants<8>,Functions<6>,Exceptions<12>,Module contents
@anchor{library/socket constants}@anchor{2af7}
@subsubsection Constants


@quotation

The AF_* and SOCK_* constants are now @code{AddressFamily} and
@code{SocketKind} @ref{828,,IntEnum} collections.


New in version 3.4.

@end quotation

@geindex AF_UNIX (in module socket)
@geindex AF_INET (in module socket)
@geindex AF_INET6 (in module socket)
@anchor{library/socket socket AF_UNIX}@anchor{2a3a}
@deffn {Data} socket.AF_UNIX
@anchor{library/socket socket AF_INET}@anchor{2a34}
@deffnx {Data} socket.AF_INET
@anchor{library/socket socket AF_INET6}@anchor{2a35}
@deffnx {Data} socket.AF_INET6

These constants represent the address (and protocol) families, used for the
first argument to @code{socket()}.  If the @ref{2a3a,,AF_UNIX} constant is not
defined then this protocol is unsupported.  More constants may be available
depending on the system.
@end deffn

@geindex SOCK_STREAM (in module socket)
@geindex SOCK_DGRAM (in module socket)
@geindex SOCK_RAW (in module socket)
@geindex SOCK_RDM (in module socket)
@geindex SOCK_SEQPACKET (in module socket)
@anchor{library/socket socket SOCK_STREAM}@anchor{e91}
@deffn {Data} socket.SOCK_STREAM
@anchor{library/socket socket SOCK_DGRAM}@anchor{e90}
@deffnx {Data} socket.SOCK_DGRAM
@anchor{library/socket socket SOCK_RAW}@anchor{2af8}
@deffnx {Data} socket.SOCK_RAW
@anchor{library/socket socket SOCK_RDM}@anchor{2af9}
@deffnx {Data} socket.SOCK_RDM
@anchor{library/socket socket SOCK_SEQPACKET}@anchor{2afa}
@deffnx {Data} socket.SOCK_SEQPACKET

These constants represent the socket types, used for the second argument to
@code{socket()}.  More constants may be available depending on the system.
(Only @ref{e91,,SOCK_STREAM} and @ref{e90,,SOCK_DGRAM} appear to be generally
useful.)
@end deffn

@geindex SOCK_CLOEXEC (in module socket)
@geindex SOCK_NONBLOCK (in module socket)
@anchor{library/socket socket SOCK_CLOEXEC}@anchor{746}
@deffn {Data} socket.SOCK_CLOEXEC
@anchor{library/socket socket SOCK_NONBLOCK}@anchor{745}
@deffnx {Data} socket.SOCK_NONBLOCK

These two constants, if defined, can be combined with the socket types and
allow you to set some flags atomically (thus avoiding possible race
conditions and the need for separate calls).


@subsubheading See also


Secure File Descriptor Handling@footnote{http://udrepper.livejournal.com/20407.html}
for a more thorough explanation.


@ref{1629,,Availability}: Linux >= 2.6.27.


New in version 3.2.

@end deffn

@geindex SOMAXCONN (in module socket)

@deffn {Data} SO_*
@anchor{library/socket socket SOMAXCONN}@anchor{9cb}
@deffnx {Data} socket.SOMAXCONN

@deffnx {Data} MSG_*

@deffnx {Data} SOL_*

@deffnx {Data} SCM_*

@deffnx {Data} IPPROTO_*

@deffnx {Data} IPPORT_*

@deffnx {Data} INADDR_*

@deffnx {Data} IP_*

@deffnx {Data} IPV6_*

@deffnx {Data} EAI_*

@deffnx {Data} AI_*

@deffnx {Data} NI_*

@deffnx {Data} TCP_*

Many constants of these forms, documented in the Unix documentation on sockets
and/or the IP protocol, are also defined in the socket module. They are
generally used in arguments to the @code{setsockopt()} and @code{getsockopt()}
methods of socket objects.  In most cases, only those symbols that are defined
in the Unix header files are defined; for a few symbols, default values are
provided.


Changed in version 3.6: @code{SO_DOMAIN}, @code{SO_PROTOCOL}, @code{SO_PEERSEC}, @code{SO_PASSSEC},
@code{TCP_USER_TIMEOUT}, @code{TCP_CONGESTION} were added.



Changed in version 3.6.5: On Windows, @code{TCP_FASTOPEN}, @code{TCP_KEEPCNT} appear if run-time Windows
supports.



Changed in version 3.7: @code{TCP_NOTSENT_LOWAT} was added.

On Windows, @code{TCP_KEEPIDLE}, @code{TCP_KEEPINTVL} appear if run-time Windows
supports.



Changed in version 3.10: @code{IP_RECVTOS} was added.
 Added @code{TCP_KEEPALIVE}. On MacOS this constant can be used in the same
 way that @code{TCP_KEEPIDLE} is used on Linux.

@end deffn

@geindex AF_CAN (in module socket)
@geindex PF_CAN (in module socket)
@anchor{library/socket socket AF_CAN}@anchor{2aed}
@deffn {Data} socket.AF_CAN
@anchor{library/socket socket PF_CAN}@anchor{2afb}
@deffnx {Data} socket.PF_CAN

@deffnx {Data} SOL_CAN_*

@deffnx {Data} CAN_*

Many constants of these forms, documented in the Linux documentation, are
also defined in the socket module.

@ref{1629,,Availability}: Linux >= 2.6.25, NetBSD >= 8.


New in version 3.3.



Changed in version 3.11: NetBSD support was added.

@end deffn

@geindex CAN_BCM (in module socket)
@anchor{library/socket socket CAN_BCM}@anchor{b24}
@deffn {Data} socket.CAN_BCM

@deffnx {Data} CAN_BCM_*

CAN_BCM, in the CAN protocol family, is the broadcast manager (BCM) protocol.
Broadcast manager constants, documented in the Linux documentation, are also
defined in the socket module.

@ref{1629,,Availability}: Linux >= 2.6.25.

@cartouche
@quotation Note 
The @code{CAN_BCM_CAN_FD_FRAME} flag is only available on Linux >= 4.8.
@end quotation
@end cartouche


New in version 3.4.

@end deffn

@geindex CAN_RAW_FD_FRAMES (in module socket)
@anchor{library/socket socket CAN_RAW_FD_FRAMES}@anchor{a3c}
@deffn {Data} socket.CAN_RAW_FD_FRAMES

Enables CAN FD support in a CAN_RAW socket. This is disabled by default.
This allows your application to send both CAN and CAN FD frames; however,
you must accept both CAN and CAN FD frames when reading from the socket.

This constant is documented in the Linux documentation.

@ref{1629,,Availability}: Linux >= 3.6.


New in version 3.5.

@end deffn

@geindex CAN_RAW_JOIN_FILTERS (in module socket)
@anchor{library/socket socket CAN_RAW_JOIN_FILTERS}@anchor{425}
@deffn {Data} socket.CAN_RAW_JOIN_FILTERS

Joins the applied CAN filters such that only CAN frames that match all
given CAN filters are passed to user space.

This constant is documented in the Linux documentation.

@ref{1629,,Availability}: Linux >= 4.1.


New in version 3.9.

@end deffn

@geindex CAN_ISOTP (in module socket)
@anchor{library/socket socket CAN_ISOTP}@anchor{2aee}
@deffn {Data} socket.CAN_ISOTP

CAN_ISOTP, in the CAN protocol family, is the ISO-TP (ISO 15765-2) protocol.
ISO-TP constants, documented in the Linux documentation.

@ref{1629,,Availability}: Linux >= 2.6.25.


New in version 3.7.

@end deffn

@geindex CAN_J1939 (in module socket)
@anchor{library/socket socket CAN_J1939}@anchor{426}
@deffn {Data} socket.CAN_J1939

CAN_J1939, in the CAN protocol family, is the SAE J1939 protocol.
J1939 constants, documented in the Linux documentation.

@ref{1629,,Availability}: Linux >= 5.4.


New in version 3.9.

@end deffn

@geindex AF_PACKET (in module socket)
@geindex PF_PACKET (in module socket)
@anchor{library/socket socket AF_PACKET}@anchor{2aef}
@deffn {Data} socket.AF_PACKET
@anchor{library/socket socket PF_PACKET}@anchor{2afc}
@deffnx {Data} socket.PF_PACKET

@deffnx {Data} PACKET_*

Many constants of these forms, documented in the Linux documentation, are
also defined in the socket module.

@ref{1629,,Availability}: Linux >= 2.2.
@end deffn

@geindex AF_RDS (in module socket)
@geindex PF_RDS (in module socket)
@geindex SOL_RDS (in module socket)
@anchor{library/socket socket AF_RDS}@anchor{2afd}
@deffn {Data} socket.AF_RDS
@anchor{library/socket socket PF_RDS}@anchor{2afe}
@deffnx {Data} socket.PF_RDS
@anchor{library/socket socket SOL_RDS}@anchor{2aff}
@deffnx {Data} socket.SOL_RDS

@deffnx {Data} RDS_*

Many constants of these forms, documented in the Linux documentation, are
also defined in the socket module.

@ref{1629,,Availability}: Linux >= 2.6.30.


New in version 3.3.

@end deffn

@geindex SIO_RCVALL (in module socket)
@geindex SIO_KEEPALIVE_VALS (in module socket)
@geindex SIO_LOOPBACK_FAST_PATH (in module socket)
@anchor{library/socket socket SIO_RCVALL}@anchor{2b00}
@deffn {Data} socket.SIO_RCVALL
@anchor{library/socket socket SIO_KEEPALIVE_VALS}@anchor{2b01}
@deffnx {Data} socket.SIO_KEEPALIVE_VALS
@anchor{library/socket socket SIO_LOOPBACK_FAST_PATH}@anchor{81c}
@deffnx {Data} socket.SIO_LOOPBACK_FAST_PATH

@deffnx {Data} RCVALL_*

Constants for Windows’ WSAIoctl(). The constants are used as arguments to the
@ref{81b,,ioctl()} method of socket objects.


Changed in version 3.6: @code{SIO_LOOPBACK_FAST_PATH} was added.

@end deffn


@deffn {Data} TIPC_*

TIPC related constants, matching the ones exported by the C socket API. See
the TIPC documentation for more information.
@end deffn

@geindex AF_ALG (in module socket)
@geindex SOL_ALG (in module socket)
@anchor{library/socket socket AF_ALG}@anchor{81f}
@deffn {Data} socket.AF_ALG
@anchor{library/socket socket SOL_ALG}@anchor{2b02}
@deffnx {Data} socket.SOL_ALG

@deffnx {Data} ALG_*

Constants for Linux Kernel cryptography.

@ref{1629,,Availability}: Linux >= 2.6.38.


New in version 3.6.

@end deffn

@geindex AF_VSOCK (in module socket)
@geindex IOCTL_VM_SOCKETS_GET_LOCAL_CID (in module socket)
@anchor{library/socket socket AF_VSOCK}@anchor{6a2}
@deffn {Data} socket.AF_VSOCK
@anchor{library/socket socket IOCTL_VM_SOCKETS_GET_LOCAL_CID}@anchor{2b03}
@deffnx {Data} socket.IOCTL_VM_SOCKETS_GET_LOCAL_CID

@deffnx {Data} VMADDR*

@deffnx {Data} SO_VM*

Constants for Linux host/guest communication.

@ref{1629,,Availability}: Linux >= 4.8.


New in version 3.7.

@end deffn

@geindex AF_LINK (in module socket)
@anchor{library/socket socket AF_LINK}@anchor{b25}
@deffn {Data} socket.AF_LINK

@ref{1629,,Availability}: BSD, macOS.


New in version 3.4.

@end deffn

@geindex has_ipv6 (in module socket)
@anchor{library/socket socket has_ipv6}@anchor{2b04}
@deffn {Data} socket.has_ipv6

This constant contains a boolean value which indicates if IPv6 is supported on
this platform.
@end deffn

@geindex BDADDR_ANY (in module socket)
@geindex BDADDR_LOCAL (in module socket)
@anchor{library/socket socket BDADDR_ANY}@anchor{2b05}
@deffn {Data} socket.BDADDR_ANY
@anchor{library/socket socket BDADDR_LOCAL}@anchor{2b06}
@deffnx {Data} socket.BDADDR_LOCAL

These are string constants containing Bluetooth addresses with special
meanings. For example, @ref{2b05,,BDADDR_ANY} can be used to indicate
any address when specifying the binding socket with
@code{BTPROTO_RFCOMM}.
@end deffn

@geindex HCI_FILTER (in module socket)
@geindex HCI_TIME_STAMP (in module socket)
@geindex HCI_DATA_DIR (in module socket)
@anchor{library/socket socket HCI_FILTER}@anchor{2b07}
@deffn {Data} socket.HCI_FILTER
@anchor{library/socket socket HCI_TIME_STAMP}@anchor{2b08}
@deffnx {Data} socket.HCI_TIME_STAMP
@anchor{library/socket socket HCI_DATA_DIR}@anchor{2b09}
@deffnx {Data} socket.HCI_DATA_DIR

For use with @code{BTPROTO_HCI}. @ref{2b07,,HCI_FILTER} is not
available for NetBSD or DragonFlyBSD. @ref{2b08,,HCI_TIME_STAMP} and
@ref{2b09,,HCI_DATA_DIR} are not available for FreeBSD, NetBSD, or
DragonFlyBSD.
@end deffn

@geindex AF_QIPCRTR (in module socket)
@anchor{library/socket socket AF_QIPCRTR}@anchor{2af0}
@deffn {Data} socket.AF_QIPCRTR

Constant for Qualcomm’s IPC router protocol, used to communicate with
service providing remote processors.

@ref{1629,,Availability}: Linux >= 4.7.
@end deffn

@node Functions<6>,,Constants<8>,Module contents
@anchor{library/socket functions}@anchor{2b0a}
@subsubsection Functions


@menu
* Creating sockets:: 
* Other functions: Other functions<2>. 

@end menu

@node Creating sockets,Other functions<2>,,Functions<6>
@anchor{library/socket creating-sockets}@anchor{2b0b}
@subsubsection Creating sockets


The following functions all create @ref{2b0c,,socket objects}.

@geindex socket (class in socket)
@anchor{library/socket socket socket}@anchor{8fd}
@deffn {Class} socket.socket (family=AF_INET, type=SOCK_STREAM, proto=0, fileno=None)

Create a new socket using the given address family, socket type and protocol
number.  The address family should be @ref{2a34,,AF_INET} (the default),
@ref{2a35,,AF_INET6}, @ref{2a3a,,AF_UNIX}, @ref{2aed,,AF_CAN}, @ref{2aef,,AF_PACKET},
or @ref{2afd,,AF_RDS}. The socket type should be @ref{e91,,SOCK_STREAM} (the
default), @ref{e90,,SOCK_DGRAM}, @ref{2af8,,SOCK_RAW} or perhaps one of the other
@code{SOCK_} constants. The protocol number is usually zero and may be omitted
or in the case where the address family is @ref{2aed,,AF_CAN} the protocol
should be one of @code{CAN_RAW}, @ref{b24,,CAN_BCM}, @ref{2aee,,CAN_ISOTP} or
@ref{426,,CAN_J1939}.

If @emph{fileno} is specified, the values for @emph{family}, @emph{type}, and @emph{proto} are
auto-detected from the specified file descriptor.  Auto-detection can be
overruled by calling the function with explicit @emph{family}, @emph{type}, or @emph{proto}
arguments.  This only affects how Python represents e.g. the return value
of @ref{2a7a,,socket.getpeername()} but not the actual OS resource.  Unlike
@ref{2b0d,,socket.fromfd()}, @emph{fileno} will return the same socket and not a
duplicate. This may help close a detached socket using
@ref{6a0,,socket.close()}.

The newly created socket is @ref{a6e,,non-inheritable}.

Raises an @ref{1130,,auditing event} @code{socket.__new__} with arguments @code{self}, @code{family}, @code{type}, @code{protocol}.


Changed in version 3.3: The AF_CAN family was added.
The AF_RDS family was added.



Changed in version 3.4: The CAN_BCM protocol was added.



Changed in version 3.4: The returned socket is now non-inheritable.



Changed in version 3.7: The CAN_ISOTP protocol was added.



Changed in version 3.7: When @ref{745,,SOCK_NONBLOCK} or @ref{746,,SOCK_CLOEXEC}
bit flags are applied to @emph{type} they are cleared, and
@ref{747,,socket.type} will not reflect them.  They are still passed
to the underlying system @cite{socket()} call.  Therefore,

@example
sock = socket.socket(
    socket.AF_INET,
    socket.SOCK_STREAM | socket.SOCK_NONBLOCK)
@end example

will still create a non-blocking socket on OSes that support
@code{SOCK_NONBLOCK}, but @code{sock.type} will be set to
@code{socket.SOCK_STREAM}.



Changed in version 3.9: The CAN_J1939 protocol was added.



Changed in version 3.10: The IPPROTO_MPTCP protocol was added.

@end deffn

@geindex socketpair() (in module socket)
@anchor{library/socket socket socketpair}@anchor{731}
@deffn {Function} socket.socketpair ([family[, type[, proto]]])

Build a pair of connected socket objects using the given address family, socket
type, and protocol number.  Address family, socket type, and protocol number are
as for the @code{socket()} function above. The default family is @ref{2a3a,,AF_UNIX}
if defined on the platform; otherwise, the default is @ref{2a34,,AF_INET}.

The newly created sockets are @ref{a6e,,non-inheritable}.


Changed in version 3.2: The returned socket objects now support the whole socket API, rather
than a subset.



Changed in version 3.4: The returned sockets are now non-inheritable.



Changed in version 3.5: Windows support added.

@end deffn

@geindex create_connection() (in module socket)
@anchor{library/socket socket create_connection}@anchor{dc0}
@deffn {Function} socket.create_connection (address[, timeout[, source_address]])

Connect to a TCP service listening on the internet @emph{address} (a 2-tuple
@code{(host, port)}), and return the socket object.  This is a higher-level
function than @ref{8ff,,socket.connect()}: if @emph{host} is a non-numeric hostname,
it will try to resolve it for both @ref{2a34,,AF_INET} and @ref{2a35,,AF_INET6},
and then try to connect to all possible addresses in turn until a
connection succeeds.  This makes it easy to write clients that are
compatible to both IPv4 and IPv6.

Passing the optional @emph{timeout} parameter will set the timeout on the
socket instance before attempting to connect.  If no @emph{timeout} is
supplied, the global default timeout setting returned by
@ref{2b0e,,getdefaulttimeout()} is used.

If supplied, @emph{source_address} must be a 2-tuple @code{(host, port)} for the
socket to bind to as its source address before connecting.  If host or port
are ‘’ or 0 respectively the OS default behavior will be used.


Changed in version 3.2: @emph{source_address} was added.

@end deffn

@geindex create_server() (in module socket)
@anchor{library/socket socket create_server}@anchor{53c}
@deffn {Function} socket.create_server (address, *, family=AF_INET, backlog=None, reuse_port=False, dualstack_ipv6=False)

Convenience function which creates a TCP socket bound to @emph{address} (a 2-tuple
@code{(host, port)}) and return the socket object.

@emph{family} should be either @ref{2a34,,AF_INET} or @ref{2a35,,AF_INET6}.
@emph{backlog} is the queue size passed to @ref{9ca,,socket.listen()}; when @code{0}
a default reasonable value is chosen.
@emph{reuse_port} dictates whether to set the @code{SO_REUSEPORT} socket option.

If @emph{dualstack_ipv6} is true and the platform supports it the socket will
be able to accept both IPv4 and IPv6 connections, else it will raise
@ref{17c,,ValueError}. Most POSIX platforms and Windows are supposed to support
this functionality.
When this functionality is enabled the address returned by
@ref{2a7a,,socket.getpeername()} when an IPv4 connection occurs will be an IPv6
address represented as an IPv4-mapped IPv6 address.
If @emph{dualstack_ipv6} is false it will explicitly disable this functionality
on platforms that enable it by default (e.g. Linux).
This parameter can be used in conjunction with @ref{53d,,has_dualstack_ipv6()}:

@example
import socket

addr = ("", 8080)  # all interfaces, port 8080
if socket.has_dualstack_ipv6():
    s = socket.create_server(addr, family=socket.AF_INET6, dualstack_ipv6=True)
else:
    s = socket.create_server(addr)
@end example

@cartouche
@quotation Note 
On POSIX platforms the @code{SO_REUSEADDR} socket option is set in order to
immediately reuse previous sockets which were bound on the same @emph{address}
and remained in TIME_WAIT state.
@end quotation
@end cartouche


New in version 3.8.

@end deffn

@geindex has_dualstack_ipv6() (in module socket)
@anchor{library/socket socket has_dualstack_ipv6}@anchor{53d}
@deffn {Function} socket.has_dualstack_ipv6 ()

Return @code{True} if the platform supports creating a TCP socket which can
handle both IPv4 and IPv6 connections.


New in version 3.8.

@end deffn

@geindex fromfd() (in module socket)
@anchor{library/socket socket fromfd}@anchor{2b0d}
@deffn {Function} socket.fromfd (fd, family, type, proto=0)

Duplicate the file descriptor @emph{fd} (an integer as returned by a file object’s
@code{fileno()} method) and build a socket object from the result.  Address
family, socket type and protocol number are as for the @code{socket()} function
above. The file descriptor should refer to a socket, but this is not checked —
subsequent operations on the object may fail if the file descriptor is invalid.
This function is rarely needed, but can be used to get or set socket options on
a socket passed to a program as standard input or output (such as a server
started by the Unix inet daemon).  The socket is assumed to be in blocking mode.

The newly created socket is @ref{a6e,,non-inheritable}.


Changed in version 3.4: The returned socket is now non-inheritable.

@end deffn

@geindex fromshare() (in module socket)
@anchor{library/socket socket fromshare}@anchor{742}
@deffn {Function} socket.fromshare (data)

Instantiate a socket from data obtained from the @ref{743,,socket.share()}
method.  The socket is assumed to be in blocking mode.

@ref{1629,,Availability}: Windows.


New in version 3.3.

@end deffn

@geindex SocketType (in module socket)
@anchor{library/socket socket SocketType}@anchor{2b0f}
@deffn {Data} socket.SocketType

This is a Python type object that represents the socket object type. It is the
same as @code{type(socket(...))}.
@end deffn

@node Other functions<2>,,Creating sockets,Functions<6>
@anchor{library/socket other-functions}@anchor{2b10}
@subsubsection Other functions


The @ref{ea,,socket} module also offers various network-related services:

@geindex close() (in module socket)
@anchor{library/socket socket close}@anchor{6a0}
@deffn {Function} socket.close (fd)

Close a socket file descriptor. This is like @ref{6a1,,os.close()}, but for
sockets. On some platforms (most noticeable Windows) @ref{6a1,,os.close()}
does not work for socket file descriptors.


New in version 3.7.

@end deffn

@geindex getaddrinfo() (in module socket)
@anchor{library/socket socket getaddrinfo}@anchor{11cc}
@deffn {Function} socket.getaddrinfo (host, port, family=0, type=0, proto=0, flags=0)

Translate the @emph{host}/@emph{port} argument into a sequence of 5-tuples that contain
all the necessary arguments for creating a socket connected to that service.
@emph{host} is a domain name, a string representation of an IPv4/v6 address
or @code{None}. @emph{port} is a string service name such as @code{'http'}, a numeric
port number or @code{None}.  By passing @code{None} as the value of @emph{host}
and @emph{port}, you can pass @code{NULL} to the underlying C API.

The @emph{family}, @emph{type} and @emph{proto} arguments can be optionally specified
in order to narrow the list of addresses returned.  Passing zero as a
value for each of these arguments selects the full range of results.
The @emph{flags} argument can be one or several of the @code{AI_*} constants,
and will influence how results are computed and returned.
For example, @code{AI_NUMERICHOST} will disable domain name resolution
and will raise an error if @emph{host} is a domain name.

The function returns a list of 5-tuples with the following structure:

@code{(family, type, proto, canonname, sockaddr)}

In these tuples, @emph{family}, @emph{type}, @emph{proto} are all integers and are
meant to be passed to the @code{socket()} function.  @emph{canonname} will be
a string representing the canonical name of the @emph{host} if
@code{AI_CANONNAME} is part of the @emph{flags} argument; else @emph{canonname}
will be empty.  @emph{sockaddr} is a tuple describing a socket address, whose
format depends on the returned @emph{family} (a @code{(address, port)} 2-tuple for
@ref{2a34,,AF_INET}, a @code{(address, port, flowinfo, scope_id)} 4-tuple for
@ref{2a35,,AF_INET6}), and is meant to be passed to the @ref{8ff,,socket.connect()}
method.

Raises an @ref{1130,,auditing event} @code{socket.getaddrinfo} with arguments @code{host}, @code{port}, @code{family}, @code{type}, @code{protocol}.

The following example fetches address information for a hypothetical TCP
connection to @code{example.org} on port 80 (results may differ on your
system if IPv6 isn’t enabled):

@example
>>> socket.getaddrinfo("example.org", 80, proto=socket.IPPROTO_TCP)
[(socket.AF_INET6, socket.SOCK_STREAM,
 6, '@w{'}, ('2606:2800:220:1:248:1893:25c8:1946', 80, 0, 0)),
 (socket.AF_INET, socket.SOCK_STREAM,
 6, '@w{'}, ('93.184.216.34', 80))]
@end example


Changed in version 3.2: parameters can now be passed using keyword arguments.



Changed in version 3.7: for IPv6 multicast addresses, string representing an address will not
contain @code{%scope_id} part.

@end deffn

@geindex getfqdn() (in module socket)
@anchor{library/socket socket getfqdn}@anchor{2b11}
@deffn {Function} socket.getfqdn ([name])

Return a fully qualified domain name for @emph{name}. If @emph{name} is omitted or empty,
it is interpreted as the local host.  To find the fully qualified name, the
hostname returned by @ref{1144,,gethostbyaddr()} is checked, followed by aliases for the
host, if available.  The first name which includes a period is selected.  In
case no fully qualified domain name is available and @emph{name} was provided,
it is returned unchanged.  If @emph{name} was empty or equal to @code{'0.0.0.0'},
the hostname from @ref{239d,,gethostname()} is returned.
@end deffn

@geindex gethostbyname() (in module socket)
@anchor{library/socket socket gethostbyname}@anchor{2b12}
@deffn {Function} socket.gethostbyname (hostname)

Translate a host name to IPv4 address format.  The IPv4 address is returned as a
string, such as  @code{'100.50.200.5'}.  If the host name is an IPv4 address itself
it is returned unchanged.  See @ref{1145,,gethostbyname_ex()} for a more complete
interface. @ref{2b12,,gethostbyname()} does not support IPv6 name resolution, and
@ref{11cc,,getaddrinfo()} should be used instead for IPv4/v6 dual stack support.

Raises an @ref{1130,,auditing event} @code{socket.gethostbyname} with argument @code{hostname}.
@end deffn

@geindex gethostbyname_ex() (in module socket)
@anchor{library/socket socket gethostbyname_ex}@anchor{1145}
@deffn {Function} socket.gethostbyname_ex (hostname)

Translate a host name to IPv4 address format, extended interface. Return a
triple @code{(hostname, aliaslist, ipaddrlist)} where @emph{hostname} is the host’s
primary host name, @emph{aliaslist} is a (possibly
empty) list of alternative host names for the same address, and @emph{ipaddrlist} is
a list of IPv4 addresses for the same interface on the same host (often but not
always a single address). @ref{1145,,gethostbyname_ex()} does not support IPv6 name
resolution, and @ref{11cc,,getaddrinfo()} should be used instead for IPv4/v6 dual
stack support.

Raises an @ref{1130,,auditing event} @code{socket.gethostbyname} with argument @code{hostname}.
@end deffn

@geindex gethostname() (in module socket)
@anchor{library/socket socket gethostname}@anchor{239d}
@deffn {Function} socket.gethostname ()

Return a string containing the hostname of the machine where  the Python
interpreter is currently executing.

Raises an @ref{1130,,auditing event} @code{socket.gethostname} with no arguments.

Note: @ref{239d,,gethostname()} doesn’t always return the fully qualified domain
name; use @ref{2b11,,getfqdn()} for that.
@end deffn

@geindex gethostbyaddr() (in module socket)
@anchor{library/socket socket gethostbyaddr}@anchor{1144}
@deffn {Function} socket.gethostbyaddr (ip_address)

Return a triple @code{(hostname, aliaslist, ipaddrlist)} where @emph{hostname} is the
primary host name responding to the given @emph{ip_address}, @emph{aliaslist} is a
(possibly empty) list of alternative host names for the same address, and
@emph{ipaddrlist} is a list of IPv4/v6 addresses for the same interface on the same
host (most likely containing only a single address). To find the fully qualified
domain name, use the function @ref{2b11,,getfqdn()}. @ref{1144,,gethostbyaddr()} supports
both IPv4 and IPv6.

Raises an @ref{1130,,auditing event} @code{socket.gethostbyaddr} with argument @code{ip_address}.
@end deffn

@geindex getnameinfo() (in module socket)
@anchor{library/socket socket getnameinfo}@anchor{2a48}
@deffn {Function} socket.getnameinfo (sockaddr, flags)

Translate a socket address @emph{sockaddr} into a 2-tuple @code{(host, port)}. Depending
on the settings of @emph{flags}, the result can contain a fully-qualified domain name
or numeric address representation in @emph{host}.  Similarly, @emph{port} can contain a
string port name or a numeric port number.

For IPv6 addresses, @code{%scope_id} is appended to the host part if @emph{sockaddr}
contains meaningful @emph{scope_id}. Usually this happens for multicast addresses.

For more information about @emph{flags} you can consult @code{getnameinfo(3)}.

Raises an @ref{1130,,auditing event} @code{socket.getnameinfo} with argument @code{sockaddr}.
@end deffn

@geindex getprotobyname() (in module socket)
@anchor{library/socket socket getprotobyname}@anchor{2b13}
@deffn {Function} socket.getprotobyname (protocolname)

Translate an internet protocol name (for example, @code{'icmp'}) to a constant
suitable for passing as the (optional) third argument to the @code{socket()}
function.  This is usually only needed for sockets opened in “raw” mode
(@ref{2af8,,SOCK_RAW}); for the normal socket modes, the correct protocol is chosen
automatically if the protocol is omitted or zero.
@end deffn

@geindex getservbyname() (in module socket)
@anchor{library/socket socket getservbyname}@anchor{2b14}
@deffn {Function} socket.getservbyname (servicename[, protocolname])

Translate an internet service name and protocol name to a port number for that
service.  The optional protocol name, if given, should be @code{'tcp'} or
@code{'udp'}, otherwise any protocol will match.

Raises an @ref{1130,,auditing event} @code{socket.getservbyname} with arguments @code{servicename}, @code{protocolname}.
@end deffn

@geindex getservbyport() (in module socket)
@anchor{library/socket socket getservbyport}@anchor{2b15}
@deffn {Function} socket.getservbyport (port[, protocolname])

Translate an internet port number and protocol name to a service name for that
service.  The optional protocol name, if given, should be @code{'tcp'} or
@code{'udp'}, otherwise any protocol will match.

Raises an @ref{1130,,auditing event} @code{socket.getservbyport} with arguments @code{port}, @code{protocolname}.
@end deffn

@geindex ntohl() (in module socket)
@anchor{library/socket socket ntohl}@anchor{2b16}
@deffn {Function} socket.ntohl (x)

Convert 32-bit positive integers from network to host byte order.  On machines
where the host byte order is the same as network byte order, this is a no-op;
otherwise, it performs a 4-byte swap operation.
@end deffn

@geindex ntohs() (in module socket)
@anchor{library/socket socket ntohs}@anchor{346}
@deffn {Function} socket.ntohs (x)

Convert 16-bit positive integers from network to host byte order.  On machines
where the host byte order is the same as network byte order, this is a no-op;
otherwise, it performs a 2-byte swap operation.


Changed in version 3.10: Raises @ref{347,,OverflowError} if @emph{x} does not fit in a 16-bit unsigned
integer.

@end deffn

@geindex htonl() (in module socket)
@anchor{library/socket socket htonl}@anchor{2b17}
@deffn {Function} socket.htonl (x)

Convert 32-bit positive integers from host to network byte order.  On machines
where the host byte order is the same as network byte order, this is a no-op;
otherwise, it performs a 4-byte swap operation.
@end deffn

@geindex htons() (in module socket)
@anchor{library/socket socket htons}@anchor{345}
@deffn {Function} socket.htons (x)

Convert 16-bit positive integers from host to network byte order.  On machines
where the host byte order is the same as network byte order, this is a no-op;
otherwise, it performs a 2-byte swap operation.


Changed in version 3.10: Raises @ref{347,,OverflowError} if @emph{x} does not fit in a 16-bit unsigned
integer.

@end deffn

@geindex inet_aton() (in module socket)
@anchor{library/socket socket inet_aton}@anchor{464}
@deffn {Function} socket.inet_aton (ip_string)

Convert an IPv4 address from dotted-quad string format (for example,
‘123.45.67.89’) to 32-bit packed binary format, as a bytes object four characters in
length.  This is useful when conversing with a program that uses the standard C
library and needs objects of type struct  in_addr, which is the C type
for the 32-bit packed binary this function returns.

@ref{464,,inet_aton()} also accepts strings with less than three dots; see the
Unix manual page @code{inet(3)} for details.

If the IPv4 address string passed to this function is invalid,
@ref{2a1,,OSError} will be raised. Note that exactly what is valid depends on
the underlying C implementation of @code{inet_aton()}.

@ref{464,,inet_aton()} does not support IPv6, and @ref{465,,inet_pton()} should be used
instead for IPv4/v6 dual stack support.
@end deffn

@geindex inet_ntoa() (in module socket)
@anchor{library/socket socket inet_ntoa}@anchor{2b18}
@deffn {Function} socket.inet_ntoa (packed_ip)

Convert a 32-bit packed IPv4 address (a @ref{87e,,bytes-like object} four
bytes in length) to its standard dotted-quad string representation (for example,
‘123.45.67.89’).  This is useful when conversing with a program that uses the
standard C library and needs objects of type struct  in_addr, which
is the C type for the 32-bit packed binary data this function takes as an
argument.

If the byte sequence passed to this function is not exactly 4 bytes in
length, @ref{2a1,,OSError} will be raised. @ref{2b18,,inet_ntoa()} does not
support IPv6, and @ref{b26,,inet_ntop()} should be used instead for IPv4/v6 dual
stack support.


Changed in version 3.5: Writable @ref{87e,,bytes-like object} is now accepted.

@end deffn

@geindex inet_pton() (in module socket)
@anchor{library/socket socket inet_pton}@anchor{465}
@deffn {Function} socket.inet_pton (address_family, ip_string)

Convert an IP address from its family-specific string format to a packed,
binary format. @ref{465,,inet_pton()} is useful when a library or network protocol
calls for an object of type struct  in_addr (similar to
@ref{464,,inet_aton()}) or struct  in6_addr.

Supported values for @emph{address_family} are currently @ref{2a34,,AF_INET} and
@ref{2a35,,AF_INET6}. If the IP address string @emph{ip_string} is invalid,
@ref{2a1,,OSError} will be raised. Note that exactly what is valid depends on
both the value of @emph{address_family} and the underlying implementation of
@code{inet_pton()}.

@ref{1629,,Availability}: Unix (maybe not all platforms), Windows.


Changed in version 3.4: Windows support added

@end deffn

@geindex inet_ntop() (in module socket)
@anchor{library/socket socket inet_ntop}@anchor{b26}
@deffn {Function} socket.inet_ntop (address_family, packed_ip)

Convert a packed IP address (a @ref{87e,,bytes-like object} of some number of
bytes) to its standard, family-specific string representation (for
example, @code{'7.10.0.5'} or @code{'5aef:2b::8'}).
@ref{b26,,inet_ntop()} is useful when a library or network protocol returns an
object of type struct  in_addr (similar to @ref{2b18,,inet_ntoa()}) or
struct  in6_addr.

Supported values for @emph{address_family} are currently @ref{2a34,,AF_INET} and
@ref{2a35,,AF_INET6}. If the bytes object @emph{packed_ip} is not the correct
length for the specified address family, @ref{17c,,ValueError} will be raised.
@ref{2a1,,OSError} is raised for errors from the call to @ref{b26,,inet_ntop()}.

@ref{1629,,Availability}: Unix (maybe not all platforms), Windows.


Changed in version 3.4: Windows support added



Changed in version 3.5: Writable @ref{87e,,bytes-like object} is now accepted.

@end deffn

@c XXX: Are sendmsg(), recvmsg() and CMSG_*() available on any
@c non-Unix platforms?  The old (obsolete?) 4.2BSD form of the
@c interface, in which struct msghdr has no msg_control or
@c msg_controllen members, is not currently supported.

@geindex CMSG_LEN() (in module socket)
@anchor{library/socket socket CMSG_LEN}@anchor{12e9}
@deffn {Function} socket.CMSG_LEN (length)

Return the total length, without trailing padding, of an ancillary
data item with associated data of the given @emph{length}.  This value
can often be used as the buffer size for @ref{902,,recvmsg()} to
receive a single item of ancillary data, but 
@geindex RFC; RFC 3542
RFC 3542@footnote{https://tools.ietf.org/html/rfc3542.html} requires
portable applications to use @ref{12e8,,CMSG_SPACE()} and thus include
space for padding, even when the item will be the last in the
buffer.  Raises @ref{347,,OverflowError} if @emph{length} is outside the
permissible range of values.

@ref{1629,,Availability}: most Unix platforms, possibly others.


New in version 3.3.

@end deffn

@geindex CMSG_SPACE() (in module socket)
@anchor{library/socket socket CMSG_SPACE}@anchor{12e8}
@deffn {Function} socket.CMSG_SPACE (length)

Return the buffer size needed for @ref{902,,recvmsg()} to
receive an ancillary data item with associated data of the given
@emph{length}, along with any trailing padding.  The buffer space needed
to receive multiple items is the sum of the @ref{12e8,,CMSG_SPACE()}
values for their associated data lengths.  Raises
@ref{347,,OverflowError} if @emph{length} is outside the permissible range
of values.

Note that some systems might support ancillary data without
providing this function.  Also note that setting the buffer size
using the results of this function may not precisely limit the
amount of ancillary data that can be received, since additional
data may be able to fit into the padding area.

@ref{1629,,Availability}: most Unix platforms, possibly others.


New in version 3.3.

@end deffn

@geindex getdefaulttimeout() (in module socket)
@anchor{library/socket socket getdefaulttimeout}@anchor{2b0e}
@deffn {Function} socket.getdefaulttimeout ()

Return the default timeout in seconds (float) for new socket objects. A value
of @code{None} indicates that new socket objects have no timeout. When the socket
module is first imported, the default is @code{None}.
@end deffn

@geindex setdefaulttimeout() (in module socket)
@anchor{library/socket socket setdefaulttimeout}@anchor{2af6}
@deffn {Function} socket.setdefaulttimeout (timeout)

Set the default timeout in seconds (float) for new socket objects.  When
the socket module is first imported, the default is @code{None}.  See
@ref{2af1,,settimeout()} for possible values and their respective
meanings.
@end deffn

@geindex sethostname() (in module socket)
@anchor{library/socket socket sethostname}@anchor{cd0}
@deffn {Function} socket.sethostname (name)

Set the machine’s hostname to @emph{name}.  This will raise an
@ref{2a1,,OSError} if you don’t have enough rights.

Raises an @ref{1130,,auditing event} @code{socket.sethostname} with argument @code{name}.

@ref{1629,,Availability}: Unix.


New in version 3.3.

@end deffn

@geindex if_nameindex() (in module socket)
@anchor{library/socket socket if_nameindex}@anchor{53e}
@deffn {Function} socket.if_nameindex ()

Return a list of network interface information
(index int, name string) tuples.
@ref{2a1,,OSError} if the system call fails.

@ref{1629,,Availability}: Unix, Windows.


New in version 3.3.



Changed in version 3.8: Windows support was added.


@cartouche
@quotation Note 
On Windows network interfaces have different names in different contexts
(all names are examples):


@itemize *

@item 
UUID: @code{@{FB605B73-AAC2-49A6-9A2F-25416AEA0573@}}

@item 
name: @code{ethernet_32770}

@item 
friendly name: @code{vEthernet (nat)}

@item 
description: @code{Hyper-V Virtual Ethernet Adapter}
@end itemize

This function returns names of the second form from the list, @code{ethernet_32770}
in this example case.
@end quotation
@end cartouche
@end deffn

@geindex if_nametoindex() (in module socket)
@anchor{library/socket socket if_nametoindex}@anchor{53f}
@deffn {Function} socket.if_nametoindex (if_name)

Return a network interface index number corresponding to an
interface name.
@ref{2a1,,OSError} if no interface with the given name exists.

@ref{1629,,Availability}: Unix, Windows.


New in version 3.3.



Changed in version 3.8: Windows support was added.



@subsubheading See also


“Interface name” is a name as documented in @ref{53e,,if_nameindex()}.

@end deffn

@geindex if_indextoname() (in module socket)
@anchor{library/socket socket if_indextoname}@anchor{540}
@deffn {Function} socket.if_indextoname (if_index)

Return a network interface name corresponding to an
interface index number.
@ref{2a1,,OSError} if no interface with the given index exists.

@ref{1629,,Availability}: Unix, Windows.


New in version 3.3.



Changed in version 3.8: Windows support was added.



@subsubheading See also


“Interface name” is a name as documented in @ref{53e,,if_nameindex()}.

@end deffn

@geindex send_fds() (in module socket)
@anchor{library/socket socket send_fds}@anchor{427}
@deffn {Function} socket.send_fds (sock, buffers, fds[, flags[, address]])

Send the list of file descriptors @emph{fds} over an @ref{2a3a,,AF_UNIX} socket @emph{sock}.
The @emph{fds} parameter is a sequence of file descriptors.
Consult @code{sendmsg()} for the documentation of these parameters.

@ref{1629,,Availability}: Unix supporting @ref{905,,sendmsg()} and @code{SCM_RIGHTS} mechanism.


New in version 3.9.

@end deffn

@geindex recv_fds() (in module socket)
@anchor{library/socket socket recv_fds}@anchor{428}
@deffn {Function} socket.recv_fds (sock, bufsize, maxfds[, flags])

Receive up to @emph{maxfds} file descriptors from an @ref{2a3a,,AF_UNIX} socket @emph{sock}.
Return @code{(msg, list(fds), flags, addr)}.
Consult @code{recvmsg()} for the documentation of these parameters.

@ref{1629,,Availability}: Unix supporting @ref{902,,recvmsg()} and @code{SCM_RIGHTS} mechanism.


New in version 3.9.


@cartouche
@quotation Note 
Any truncated integers at the end of the list of file descriptors.
@end quotation
@end cartouche
@end deffn

@node Socket Objects,Notes on socket timeouts,Module contents,socket — Low-level networking interface
@anchor{library/socket id1}@anchor{2b19}@anchor{library/socket socket-objects}@anchor{2b0c}
@subsubsection Socket Objects


Socket objects have the following methods.  Except for
@ref{d44,,makefile()}, these correspond to Unix system calls applicable
to sockets.


Changed in version 3.2: Support for the @ref{166,,context manager} protocol was added.  Exiting the
context manager is equivalent to calling @ref{6a0,,close()}.


@geindex accept() (socket.socket method)
@anchor{library/socket socket socket accept}@anchor{8fe}
@deffn {Method} socket.accept ()

Accept a connection. The socket must be bound to an address and listening for
connections. The return value is a pair @code{(conn, address)} where @emph{conn} is a
@emph{new} socket object usable to send and receive data on the connection, and
@emph{address} is the address bound to the socket on the other end of the connection.

The newly created socket is @ref{a6e,,non-inheritable}.


Changed in version 3.4: The socket is now non-inheritable.



Changed in version 3.5: If the system call is interrupted and the signal handler does not raise
an exception, the method now retries the system call instead of raising
an @ref{8e3,,InterruptedError} exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).

@end deffn

@geindex bind() (socket.socket method)
@anchor{library/socket socket socket bind}@anchor{2b1a}
@deffn {Method} socket.bind (address)

Bind the socket to @emph{address}.  The socket must not already be bound. (The format
of @emph{address} depends on the address family — see above.)

Raises an @ref{1130,,auditing event} @code{socket.bind} with arguments @code{self}, @code{address}.
@end deffn

@geindex close() (socket.socket method)
@anchor{library/socket socket socket close}@anchor{896}
@deffn {Method} socket.close ()

Mark the socket closed.  The underlying system resource (e.g. a file
descriptor) is also closed when all file objects from @ref{d44,,makefile()}
are closed.  Once that happens, all future operations on the socket
object will fail. The remote end will receive no more data (after
queued data is flushed).

Sockets are automatically closed when they are garbage-collected, but
it is recommended to @ref{6a0,,close()} them explicitly, or to use a
@ref{168,,with} statement around them.


Changed in version 3.6: @ref{2a1,,OSError} is now raised if an error occurs when the underlying
@code{close()} call is made.


@cartouche
@quotation Note 
@ref{6a0,,close()} releases the resource associated with a connection but
does not necessarily close the connection immediately.  If you want
to close the connection in a timely fashion, call @ref{2b1b,,shutdown()}
before @ref{6a0,,close()}.
@end quotation
@end cartouche
@end deffn

@geindex connect() (socket.socket method)
@anchor{library/socket socket socket connect}@anchor{8ff}
@deffn {Method} socket.connect (address)

Connect to a remote socket at @emph{address}. (The format of @emph{address} depends on the
address family — see above.)

If the connection is interrupted by a signal, the method waits until the
connection completes, or raise a @ref{2b8,,TimeoutError} on timeout, if the
signal handler doesn’t raise an exception and the socket is blocking or has
a timeout. For non-blocking sockets, the method raises an
@ref{8e3,,InterruptedError} exception if the connection is interrupted by a
signal (or the exception raised by the signal handler).

Raises an @ref{1130,,auditing event} @code{socket.connect} with arguments @code{self}, @code{address}.


Changed in version 3.5: The method now waits until the connection completes instead of raising an
@ref{8e3,,InterruptedError} exception if the connection is interrupted by a
signal, the signal handler doesn’t raise an exception and the socket is
blocking or has a timeout (see the 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).

@end deffn

@geindex connect_ex() (socket.socket method)
@anchor{library/socket socket socket connect_ex}@anchor{2b1c}
@deffn {Method} socket.connect_ex (address)

Like @code{connect(address)}, but return an error indicator instead of raising an
exception for errors returned by the C-level @code{connect()} call (other
problems, such as “host not found,” can still raise exceptions).  The error
indicator is @code{0} if the operation succeeded, otherwise the value of the
@code{errno} variable.  This is useful to support, for example, asynchronous
connects.

Raises an @ref{1130,,auditing event} @code{socket.connect} with arguments @code{self}, @code{address}.
@end deffn

@geindex detach() (socket.socket method)
@anchor{library/socket socket socket detach}@anchor{dbf}
@deffn {Method} socket.detach ()

Put the socket object into closed state without actually closing the
underlying file descriptor.  The file descriptor is returned, and can
be reused for other purposes.


New in version 3.2.

@end deffn

@geindex dup() (socket.socket method)
@anchor{library/socket socket socket dup}@anchor{2b1d}
@deffn {Method} socket.dup ()

Duplicate the socket.

The newly created socket is @ref{a6e,,non-inheritable}.


Changed in version 3.4: The socket is now non-inheritable.

@end deffn

@geindex fileno() (socket.socket method)
@anchor{library/socket socket socket fileno}@anchor{2b1e}
@deffn {Method} socket.fileno ()

Return the socket’s file descriptor (a small integer), or -1 on failure. This
is useful with @ref{8fc,,select.select()}.

Under Windows the small integer returned by this method cannot be used where a
file descriptor can be used (such as @ref{179a,,os.fdopen()}).  Unix does not have
this limitation.
@end deffn

@geindex get_inheritable() (socket.socket method)
@anchor{library/socket socket socket get_inheritable}@anchor{a73}
@deffn {Method} socket.get_inheritable ()

Get the @ref{a6e,,inheritable flag} of the socket’s file
descriptor or socket’s handle: @code{True} if the socket can be inherited in
child processes, @code{False} if it cannot.


New in version 3.4.

@end deffn

@geindex getpeername() (socket.socket method)
@anchor{library/socket socket socket getpeername}@anchor{2a7a}
@deffn {Method} socket.getpeername ()

Return the remote address to which the socket is connected.  This is useful to
find out the port number of a remote IPv4/v6 socket, for instance. (The format
of the address returned depends on the address family — see above.)  On some
systems this function is not supported.
@end deffn

@geindex getsockname() (socket.socket method)
@anchor{library/socket socket socket getsockname}@anchor{2a7b}
@deffn {Method} socket.getsockname ()

Return the socket’s own address.  This is useful to find out the port number of
an IPv4/v6 socket, for instance. (The format of the address returned depends on
the address family — see above.)
@end deffn

@geindex getsockopt() (socket.socket method)
@anchor{library/socket socket socket getsockopt}@anchor{81d}
@deffn {Method} socket.getsockopt (level, optname[, buflen])

Return the value of the given socket option (see the Unix man page
@code{getsockopt(2)}).  The needed symbolic constants (@code{SO_*} etc.)
are defined in this module.  If @emph{buflen} is absent, an integer option is assumed
and its integer value is returned by the function.  If @emph{buflen} is present, it
specifies the maximum length of the buffer used to receive the option in, and
this buffer is returned as a bytes object.  It is up to the caller to decode the
contents of the buffer (see the optional built-in module @ref{f3,,struct} for a way
to decode C structures encoded as byte strings).
@end deffn

@geindex getblocking() (socket.socket method)
@anchor{library/socket socket socket getblocking}@anchor{69f}
@deffn {Method} socket.getblocking ()

Return @code{True} if socket is in blocking mode, @code{False} if in
non-blocking.

This is equivalent to checking @code{socket.gettimeout() == 0}.


New in version 3.7.

@end deffn

@geindex gettimeout() (socket.socket method)
@anchor{library/socket socket socket gettimeout}@anchor{2b1f}
@deffn {Method} socket.gettimeout ()

Return the timeout in seconds (float) associated with socket operations,
or @code{None} if no timeout is set.  This reflects the last call to
@ref{23a8,,setblocking()} or @ref{2af1,,settimeout()}.
@end deffn

@geindex ioctl() (socket.socket method)
@anchor{library/socket socket socket ioctl}@anchor{81b}
@deffn {Method} socket.ioctl (control, option)

@*Platform: 
Windows


The @ref{81b,,ioctl()} method is a limited interface to the WSAIoctl system
interface.  Please refer to the Win32 documentation@footnote{https://msdn.microsoft.com/en-us/library/ms741621%28VS.85%29.aspx} for more
information.

On other platforms, the generic @ref{2b20,,fcntl.fcntl()} and @ref{fd7,,fcntl.ioctl()}
functions may be used; they accept a socket object as their first argument.

Currently only the following control codes are supported:
@code{SIO_RCVALL}, @code{SIO_KEEPALIVE_VALS}, and @code{SIO_LOOPBACK_FAST_PATH}.


Changed in version 3.6: @code{SIO_LOOPBACK_FAST_PATH} was added.

@end deffn

@geindex listen() (socket.socket method)
@anchor{library/socket socket socket listen}@anchor{9ca}
@deffn {Method} socket.listen ([backlog])

Enable a server to accept connections.  If @emph{backlog} is specified, it must
be at least 0 (if it is lower, it is set to 0); it specifies the number of
unaccepted connections that the system will allow before refusing new
connections. If not specified, a default reasonable value is chosen.


Changed in version 3.5: The @emph{backlog} parameter is now optional.

@end deffn

@geindex makefile() (socket.socket method)
@anchor{library/socket socket socket makefile}@anchor{d44}
@deffn {Method} socket.makefile (mode='r', buffering=None, *, encoding=None, errors=None, newline=None)

@geindex I/O control; buffering

Return a @ref{d68,,file object} associated with the socket.  The exact returned
type depends on the arguments given to @ref{d44,,makefile()}.  These arguments are
interpreted the same way as by the built-in @ref{1c8,,open()} function, except
the only supported @emph{mode} values are @code{'r'} (default), @code{'w'} and @code{'b'}.

The socket must be in blocking mode; it can have a timeout, but the file
object’s internal buffer may end up in an inconsistent state if a timeout
occurs.

Closing the file object returned by @ref{d44,,makefile()} won’t close the
original socket unless all other file objects have been closed and
@ref{6a0,,socket.close()} has been called on the socket object.

@cartouche
@quotation Note 
On Windows, the file-like object created by @ref{d44,,makefile()} cannot be
used where a file object with a file descriptor is expected, such as the
stream arguments of @ref{5ae,,subprocess.Popen()}.
@end quotation
@end cartouche
@end deffn

@geindex recv() (socket.socket method)
@anchor{library/socket socket socket recv}@anchor{900}
@deffn {Method} socket.recv (bufsize[, flags])

Receive data from the socket.  The return value is a bytes object representing the
data received.  The maximum amount of data to be received at once is specified
by @emph{bufsize}.  See the Unix manual page @code{recv(2)} for the meaning of
the optional argument @emph{flags}; it defaults to zero.

@cartouche
@quotation Note 
For best match with hardware and network realities, the value of  @emph{bufsize}
should be a relatively small power of 2, for example, 4096.
@end quotation
@end cartouche


Changed in version 3.5: If the system call is interrupted and the signal handler does not raise
an exception, the method now retries the system call instead of raising
an @ref{8e3,,InterruptedError} exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).

@end deffn

@geindex recvfrom() (socket.socket method)
@anchor{library/socket socket socket recvfrom}@anchor{901}
@deffn {Method} socket.recvfrom (bufsize[, flags])

Receive data from the socket.  The return value is a pair @code{(bytes, address)}
where @emph{bytes} is a bytes object representing the data received and @emph{address} is the
address of the socket sending the data.  See the Unix manual page
@code{recv(2)} for the meaning of the optional argument @emph{flags}; it defaults
to zero. (The format of @emph{address} depends on the address family — see above.)


Changed in version 3.5: If the system call is interrupted and the signal handler does not raise
an exception, the method now retries the system call instead of raising
an @ref{8e3,,InterruptedError} exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).



Changed in version 3.7: For multicast IPv6 address, first item of @emph{address} does not contain
@code{%scope_id} part anymore. In order to get full IPv6 address use
@ref{2a48,,getnameinfo()}.

@end deffn

@geindex recvmsg() (socket.socket method)
@anchor{library/socket socket socket recvmsg}@anchor{902}
@deffn {Method} socket.recvmsg (bufsize[, ancbufsize[, flags]])

Receive normal data (up to @emph{bufsize} bytes) and ancillary data from
the socket.  The @emph{ancbufsize} argument sets the size in bytes of
the internal buffer used to receive the ancillary data; it defaults
to 0, meaning that no ancillary data will be received.  Appropriate
buffer sizes for ancillary data can be calculated using
@ref{12e8,,CMSG_SPACE()} or @ref{12e9,,CMSG_LEN()}, and items which do not fit
into the buffer might be truncated or discarded.  The @emph{flags}
argument defaults to 0 and has the same meaning as for
@ref{900,,recv()}.

The return value is a 4-tuple: @code{(data, ancdata, msg_flags,
address)}.  The @emph{data} item is a @ref{2f0,,bytes} object holding the
non-ancillary data received.  The @emph{ancdata} item is a list of zero
or more tuples @code{(cmsg_level, cmsg_type, cmsg_data)} representing
the ancillary data (control messages) received: @emph{cmsg_level} and
@emph{cmsg_type} are integers specifying the protocol level and
protocol-specific type respectively, and @emph{cmsg_data} is a
@ref{2f0,,bytes} object holding the associated data.  The @emph{msg_flags}
item is the bitwise OR of various flags indicating conditions on
the received message; see your system documentation for details.
If the receiving socket is unconnected, @emph{address} is the address of
the sending socket, if available; otherwise, its value is
unspecified.

On some systems, @ref{905,,sendmsg()} and @ref{902,,recvmsg()} can be used to
pass file descriptors between processes over an @ref{2a3a,,AF_UNIX}
socket.  When this facility is used (it is often restricted to
@ref{e91,,SOCK_STREAM} sockets), @ref{902,,recvmsg()} will return, in its
ancillary data, items of the form @code{(socket.SOL_SOCKET,
socket.SCM_RIGHTS, fds)}, where @emph{fds} is a @ref{2f0,,bytes} object
representing the new file descriptors as a binary array of the
native C int type.  If @ref{902,,recvmsg()} raises an
exception after the system call returns, it will first attempt to
close any file descriptors received via this mechanism.

Some systems do not indicate the truncated length of ancillary data
items which have been only partially received.  If an item appears
to extend beyond the end of the buffer, @ref{902,,recvmsg()} will issue
a @ref{5b5,,RuntimeWarning}, and will return the part of it which is
inside the buffer provided it has not been truncated before the
start of its associated data.

On systems which support the @code{SCM_RIGHTS} mechanism, the
following function will receive up to @emph{maxfds} file descriptors,
returning the message data and a list containing the descriptors
(while ignoring unexpected conditions such as unrelated control
messages being received).  See also @ref{905,,sendmsg()}.

@example
import socket, array

def recv_fds(sock, msglen, maxfds):
    fds = array.array("i")   # Array of ints
    msg, ancdata, flags, addr = sock.recvmsg(msglen, socket.CMSG_LEN(maxfds * fds.itemsize))
    for cmsg_level, cmsg_type, cmsg_data in ancdata:
        if cmsg_level == socket.SOL_SOCKET and cmsg_type == socket.SCM_RIGHTS:
            # Append data, ignoring any truncated integers at the end.
            fds.frombytes(cmsg_data[:len(cmsg_data) - (len(cmsg_data) % fds.itemsize)])
    return msg, list(fds)
@end example

@ref{1629,,Availability}: most Unix platforms, possibly others.


New in version 3.3.



Changed in version 3.5: If the system call is interrupted and the signal handler does not raise
an exception, the method now retries the system call instead of raising
an @ref{8e3,,InterruptedError} exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).

@end deffn

@geindex recvmsg_into() (socket.socket method)
@anchor{library/socket socket socket recvmsg_into}@anchor{ccf}
@deffn {Method} socket.recvmsg_into (buffers[, ancbufsize[, flags]])

Receive normal data and ancillary data from the socket, behaving as
@ref{902,,recvmsg()} would, but scatter the non-ancillary data into a
series of buffers instead of returning a new bytes object.  The
@emph{buffers} argument must be an iterable of objects that export
writable buffers (e.g. @ref{2f1,,bytearray} objects); these will be
filled with successive chunks of the non-ancillary data until it
has all been written or there are no more buffers.  The operating
system may set a limit (@ref{23cd,,sysconf()} value @code{SC_IOV_MAX})
on the number of buffers that can be used.  The @emph{ancbufsize} and
@emph{flags} arguments have the same meaning as for @ref{902,,recvmsg()}.

The return value is a 4-tuple: @code{(nbytes, ancdata, msg_flags,
address)}, where @emph{nbytes} is the total number of bytes of
non-ancillary data written into the buffers, and @emph{ancdata},
@emph{msg_flags} and @emph{address} are the same as for @ref{902,,recvmsg()}.

Example:

@example
>>> import socket
>>> s1, s2 = socket.socketpair()
>>> b1 = bytearray(b'----')
>>> b2 = bytearray(b'0123456789')
>>> b3 = bytearray(b'--------------')
>>> s1.send(b'Mary had a little lamb')
22
>>> s2.recvmsg_into([b1, memoryview(b2)[2:9], b3])
(22, [], 0, None)
>>> [b1, b2, b3]
[bytearray(b'Mary'), bytearray(b'01 had a 9'), bytearray(b'little lamb---')]
@end example

@ref{1629,,Availability}: most Unix platforms, possibly others.


New in version 3.3.

@end deffn

@geindex recvfrom_into() (socket.socket method)
@anchor{library/socket socket socket recvfrom_into}@anchor{eb5}
@deffn {Method} socket.recvfrom_into (buffer[, nbytes[, flags]])

Receive data from the socket, writing it into @emph{buffer} instead of creating a
new bytestring.  The return value is a pair @code{(nbytes, address)} where @emph{nbytes} is
the number of bytes received and @emph{address} is the address of the socket sending
the data.  See the Unix manual page @code{recv(2)} for the meaning of the
optional argument @emph{flags}; it defaults to zero.  (The format of @emph{address}
depends on the address family — see above.)
@end deffn

@geindex recv_into() (socket.socket method)
@anchor{library/socket socket socket recv_into}@anchor{eb4}
@deffn {Method} socket.recv_into (buffer[, nbytes[, flags]])

Receive up to @emph{nbytes} bytes from the socket, storing the data into a buffer
rather than creating a new bytestring.  If @emph{nbytes} is not specified (or 0),
receive up to the size available in the given buffer.  Returns the number of
bytes received.  See the Unix manual page @code{recv(2)} for the meaning
of the optional argument @emph{flags}; it defaults to zero.
@end deffn

@geindex send() (socket.socket method)
@anchor{library/socket socket socket send}@anchor{903}
@deffn {Method} socket.send (bytes[, flags])

Send data to the socket.  The socket must be connected to a remote socket.  The
optional @emph{flags} argument has the same meaning as for @ref{900,,recv()} above.
Returns the number of bytes sent. Applications are responsible for checking that
all data has been sent; if only some of the data was transmitted, the
application needs to attempt delivery of the remaining data. For further
information on this topic, consult the @ref{2b21,,Socket Programming HOWTO}.


Changed in version 3.5: If the system call is interrupted and the signal handler does not raise
an exception, the method now retries the system call instead of raising
an @ref{8e3,,InterruptedError} exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).

@end deffn

@geindex sendall() (socket.socket method)
@anchor{library/socket socket socket sendall}@anchor{904}
@deffn {Method} socket.sendall (bytes[, flags])

Send data to the socket.  The socket must be connected to a remote socket.  The
optional @emph{flags} argument has the same meaning as for @ref{900,,recv()} above.
Unlike @ref{903,,send()}, this method continues to send data from @emph{bytes} until
either all data has been sent or an error occurs.  @code{None} is returned on
success.  On error, an exception is raised, and there is no way to determine how
much data, if any, was successfully sent.


Changed in version 3.5: The socket timeout is no more reset each time data is sent successfully.
The socket timeout is now the maximum total duration to send all data.



Changed in version 3.5: If the system call is interrupted and the signal handler does not raise
an exception, the method now retries the system call instead of raising
an @ref{8e3,,InterruptedError} exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).

@end deffn

@geindex sendto() (socket.socket method)
@anchor{library/socket socket socket sendto}@anchor{906}
@deffn {Method} socket.sendto (bytes, address)

@deffnx {Method} socket.sendto (bytes, flags, address)

Send data to the socket.  The socket should not be connected to a remote socket,
since the destination socket is specified by @emph{address}.  The optional @emph{flags}
argument has the same meaning as for @ref{900,,recv()} above.  Return the number of
bytes sent. (The format of @emph{address} depends on the address family — see
above.)

Raises an @ref{1130,,auditing event} @code{socket.sendto} with arguments @code{self}, @code{address}.


Changed in version 3.5: If the system call is interrupted and the signal handler does not raise
an exception, the method now retries the system call instead of raising
an @ref{8e3,,InterruptedError} exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).

@end deffn

@geindex sendmsg() (socket.socket method)
@anchor{library/socket socket socket sendmsg}@anchor{905}
@deffn {Method} socket.sendmsg (buffers[, ancdata[, flags[, address]]])

Send normal and ancillary data to the socket, gathering the
non-ancillary data from a series of buffers and concatenating it
into a single message.  The @emph{buffers} argument specifies the
non-ancillary data as an iterable of
@ref{87e,,bytes-like objects}
(e.g. @ref{2f0,,bytes} objects); the operating system may set a limit
(@ref{23cd,,sysconf()} value @code{SC_IOV_MAX}) on the number of buffers
that can be used.  The @emph{ancdata} argument specifies the ancillary
data (control messages) as an iterable of zero or more tuples
@code{(cmsg_level, cmsg_type, cmsg_data)}, where @emph{cmsg_level} and
@emph{cmsg_type} are integers specifying the protocol level and
protocol-specific type respectively, and @emph{cmsg_data} is a
bytes-like object holding the associated data.  Note that
some systems (in particular, systems without @ref{12e8,,CMSG_SPACE()})
might support sending only one control message per call.  The
@emph{flags} argument defaults to 0 and has the same meaning as for
@ref{903,,send()}.  If @emph{address} is supplied and not @code{None}, it sets a
destination address for the message.  The return value is the
number of bytes of non-ancillary data sent.

The following function sends the list of file descriptors @emph{fds}
over an @ref{2a3a,,AF_UNIX} socket, on systems which support the
@code{SCM_RIGHTS} mechanism.  See also @ref{902,,recvmsg()}.

@example
import socket, array

def send_fds(sock, msg, fds):
    return sock.sendmsg([msg], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, array.array("i", fds))])
@end example

@ref{1629,,Availability}: most Unix platforms, possibly others.

Raises an @ref{1130,,auditing event} @code{socket.sendmsg} with arguments @code{self}, @code{address}.


New in version 3.3.



Changed in version 3.5: If the system call is interrupted and the signal handler does not raise
an exception, the method now retries the system call instead of raising
an @ref{8e3,,InterruptedError} exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).

@end deffn

@geindex sendmsg_afalg() (socket.socket method)
@anchor{library/socket socket socket sendmsg_afalg}@anchor{820}
@deffn {Method} socket.sendmsg_afalg ([msg], *, op[, iv[, assoclen[, flags]]])

Specialized version of @ref{905,,sendmsg()} for @ref{81f,,AF_ALG} socket.
Set mode, IV, AEAD associated data length and flags for @ref{81f,,AF_ALG} socket.

@ref{1629,,Availability}: Linux >= 2.6.38.


New in version 3.6.

@end deffn

@geindex sendfile() (socket.socket method)
@anchor{library/socket socket socket sendfile}@anchor{9c9}
@deffn {Method} socket.sendfile (file, offset=0, count=None)

Send a file until EOF is reached by using high-performance
@ref{62c,,os.sendfile} and return the total number of bytes which were sent.
@emph{file} must be a regular file object opened in binary mode. If
@ref{62c,,os.sendfile} is not available (e.g. Windows) or @emph{file} is not a
regular file @ref{903,,send()} will be used instead. @emph{offset} tells from where to
start reading the file. If specified, @emph{count} is the total number of bytes
to transmit as opposed to sending the file until EOF is reached. File
position is updated on return or also in case of error in which case
@ref{2232,,file.tell()} can be used to figure out the number of
bytes which were sent. The socket must be of @ref{e91,,SOCK_STREAM} type.
Non-blocking sockets are not supported.


New in version 3.5.

@end deffn

@geindex set_inheritable() (socket.socket method)
@anchor{library/socket socket socket set_inheritable}@anchor{a74}
@deffn {Method} socket.set_inheritable (inheritable)

Set the @ref{a6e,,inheritable flag} of the socket’s file
descriptor or socket’s handle.


New in version 3.4.

@end deffn

@geindex setblocking() (socket.socket method)
@anchor{library/socket socket socket setblocking}@anchor{23a8}
@deffn {Method} socket.setblocking (flag)

Set blocking or non-blocking mode of the socket: if @emph{flag} is false, the
socket is set to non-blocking, else to blocking mode.

This method is a shorthand for certain @ref{2af1,,settimeout()} calls:


@itemize *

@item 
@code{sock.setblocking(True)} is equivalent to @code{sock.settimeout(None)}

@item 
@code{sock.setblocking(False)} is equivalent to @code{sock.settimeout(0.0)}
@end itemize


Changed in version 3.7: The method no longer applies @ref{745,,SOCK_NONBLOCK} flag on
@ref{747,,socket.type}.

@end deffn

@geindex settimeout() (socket.socket method)
@anchor{library/socket socket socket settimeout}@anchor{2af1}
@deffn {Method} socket.settimeout (value)

Set a timeout on blocking socket operations.  The @emph{value} argument can be a
nonnegative floating point number expressing seconds, or @code{None}.
If a non-zero value is given, subsequent socket operations will raise a
@ref{2b7,,timeout} exception if the timeout period @emph{value} has elapsed before
the operation has completed.  If zero is given, the socket is put in
non-blocking mode. If @code{None} is given, the socket is put in blocking mode.

For further information, please consult the @ref{2b22,,notes on socket timeouts}.


Changed in version 3.7: The method no longer toggles @ref{745,,SOCK_NONBLOCK} flag on
@ref{747,,socket.type}.

@end deffn

@geindex setsockopt() (socket.socket method)
@anchor{library/socket socket socket setsockopt}@anchor{81e}
@deffn {Method} socket.setsockopt (level, optname, value: int)
@end deffn


@deffn {Method} socket.setsockopt (level, optname, value: buffer)
@end deffn


@deffn {Method} socket.setsockopt (level, optname, None, optlen: int)

@geindex module; struct

Set the value of the given socket option (see the Unix manual page
@code{setsockopt(2)}).  The needed symbolic constants are defined in the
@ref{ea,,socket} module (@code{SO_*} etc.).  The value can be an integer,
@code{None} or a @ref{87e,,bytes-like object} representing a buffer. In the later
case it is up to the caller to ensure that the bytestring contains the
proper bits (see the optional built-in module @ref{f3,,struct} for a way to
encode C structures as bytestrings). When @emph{value} is set to @code{None},
@emph{optlen} argument is required. It’s equivalent to call @code{setsockopt()} C
function with @code{optval=NULL} and @code{optlen=optlen}.


Changed in version 3.5: Writable @ref{87e,,bytes-like object} is now accepted.



Changed in version 3.6: setsockopt(level, optname, None, optlen: int) form added.

@end deffn

@geindex shutdown() (socket.socket method)
@anchor{library/socket socket socket shutdown}@anchor{2b1b}
@deffn {Method} socket.shutdown (how)

Shut down one or both halves of the connection.  If @emph{how} is @code{SHUT_RD},
further receives are disallowed.  If @emph{how} is @code{SHUT_WR}, further sends
are disallowed.  If @emph{how} is @code{SHUT_RDWR}, further sends and receives are
disallowed.
@end deffn

@geindex share() (socket.socket method)
@anchor{library/socket socket socket share}@anchor{743}
@deffn {Method} socket.share (process_id)

Duplicate a socket and prepare it for sharing with a target process.  The
target process must be provided with @emph{process_id}.  The resulting bytes object
can then be passed to the target process using some form of interprocess
communication and the socket can be recreated there using @ref{742,,fromshare()}.
Once this method has been called, it is safe to close the socket since
the operating system has already duplicated it for the target process.

@ref{1629,,Availability}: Windows.


New in version 3.3.

@end deffn

Note that there are no methods @code{read()} or @code{write()}; use
@ref{900,,recv()} and @ref{903,,send()} without @emph{flags} argument instead.

Socket objects also have these (read-only) attributes that correspond to the
values given to the @ref{8fd,,socket} constructor.

@geindex family (socket.socket attribute)
@anchor{library/socket socket socket family}@anchor{2b23}
@deffn {Attribute} socket.family

The socket family.
@end deffn

@geindex type (socket.socket attribute)
@anchor{library/socket socket socket type}@anchor{747}
@deffn {Attribute} socket.type

The socket type.
@end deffn

@geindex proto (socket.socket attribute)
@anchor{library/socket socket socket proto}@anchor{2b24}
@deffn {Attribute} socket.proto

The socket protocol.
@end deffn

@node Notes on socket timeouts,Example<8>,Socket Objects,socket — Low-level networking interface
@anchor{library/socket notes-on-socket-timeouts}@anchor{2b25}@anchor{library/socket socket-timeouts}@anchor{2b22}
@subsubsection Notes on socket timeouts


A socket object can be in one of three modes: blocking, non-blocking, or
timeout.  Sockets are by default always created in blocking mode, but this
can be changed by calling @ref{2af6,,setdefaulttimeout()}.


@itemize *

@item 
In @emph{blocking mode}, operations block until complete or the system returns
an error (such as connection timed out).

@item 
In @emph{non-blocking mode}, operations fail (with an error that is unfortunately
system-dependent) if they cannot be completed immediately: functions from the
@ref{e0,,select} can be used to know when and whether a socket is available for
reading or writing.

@item 
In @emph{timeout mode}, operations fail if they cannot be completed within the
timeout specified for the socket (they raise a @ref{2b7,,timeout} exception)
or if the system returns an error.
@end itemize

@cartouche
@quotation Note 
At the operating system level, sockets in @emph{timeout mode} are internally set
in non-blocking mode.  Also, the blocking and timeout modes are shared between
file descriptors and socket objects that refer to the same network endpoint.
This implementation detail can have visible consequences if e.g. you decide
to use the @ref{2b1e,,fileno()} of a socket.
@end quotation
@end cartouche

@menu
* Timeouts and the connect method:: 
* Timeouts and the accept method:: 

@end menu

@node Timeouts and the connect method,Timeouts and the accept method,,Notes on socket timeouts
@anchor{library/socket timeouts-and-the-connect-method}@anchor{2b26}
@subsubsection Timeouts and the @code{connect} method


The @ref{8ff,,connect()} operation is also subject to the timeout
setting, and in general it is recommended to call @ref{2af1,,settimeout()}
before calling @ref{8ff,,connect()} or pass a timeout parameter to
@ref{dc0,,create_connection()}.  However, the system network stack may also
return a connection timeout error of its own regardless of any Python socket
timeout setting.

@node Timeouts and the accept method,,Timeouts and the connect method,Notes on socket timeouts
@anchor{library/socket timeouts-and-the-accept-method}@anchor{2b27}
@subsubsection Timeouts and the @code{accept} method


If @ref{2b0e,,getdefaulttimeout()} is not @ref{182,,None}, sockets returned by
the @ref{8fe,,accept()} method inherit that timeout.  Otherwise, the
behaviour depends on settings of the listening socket:


@itemize *

@item 
if the listening socket is in @emph{blocking mode} or in @emph{timeout mode},
the socket returned by @ref{8fe,,accept()} is in @emph{blocking mode};

@item 
if the listening socket is in @emph{non-blocking mode}, whether the socket
returned by @ref{8fe,,accept()} is in blocking or non-blocking mode
is operating system-dependent.  If you want to ensure cross-platform
behaviour, it is recommended you manually override this setting.
@end itemize

@node Example<8>,,Notes on socket timeouts,socket — Low-level networking interface
@anchor{library/socket example}@anchor{2b28}@anchor{library/socket socket-example}@anchor{2b29}
@subsubsection Example


Here are four minimal example programs using the TCP/IP protocol: a server that
echoes all data that it receives back (servicing only one client), and a client
using it.  Note that a server must perform the sequence @code{socket()},
@ref{2b1a,,bind()}, @ref{9ca,,listen()}, @ref{8fe,,accept()} (possibly
repeating the @ref{8fe,,accept()} to service more than one client), while a
client only needs the sequence @code{socket()}, @ref{8ff,,connect()}.  Also
note that the server does not @ref{904,,sendall()}/@ref{900,,recv()} on
the socket it is listening on but on the new socket returned by
@ref{8fe,,accept()}.

The first two examples support IPv4 only.

@example
# Echo server program
import socket

HOST = '@w{'}                 # Symbolic name meaning all available interfaces
PORT = 50007              # Arbitrary non-privileged port
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.bind((HOST, PORT))
    s.listen(1)
    conn, addr = s.accept()
    with conn:
        print('Connected by', addr)
        while True:
            data = conn.recv(1024)
            if not data: break
            conn.sendall(data)
@end example

@example
# Echo client program
import socket

HOST = 'daring.cwi.nl'    # The remote host
PORT = 50007              # The same port as used by the server
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.connect((HOST, PORT))
    s.sendall(b'Hello, world')
    data = s.recv(1024)
print('Received', repr(data))
@end example

The next two examples are identical to the above two, but support both IPv4 and
IPv6. The server side will listen to the first address family available (it
should listen to both instead). On most of IPv6-ready systems, IPv6 will take
precedence and the server may not accept IPv4 traffic. The client side will try
to connect to the all addresses returned as a result of the name resolution, and
sends traffic to the first one connected successfully.

@example
# Echo server program
import socket
import sys

HOST = None               # Symbolic name meaning all available interfaces
PORT = 50007              # Arbitrary non-privileged port
s = None
for res in socket.getaddrinfo(HOST, PORT, socket.AF_UNSPEC,
                              socket.SOCK_STREAM, 0, socket.AI_PASSIVE):
    af, socktype, proto, canonname, sa = res
    try:
        s = socket.socket(af, socktype, proto)
    except OSError as msg:
        s = None
        continue
    try:
        s.bind(sa)
        s.listen(1)
    except OSError as msg:
        s.close()
        s = None
        continue
    break
if s is None:
    print('could not open socket')
    sys.exit(1)
conn, addr = s.accept()
with conn:
    print('Connected by', addr)
    while True:
        data = conn.recv(1024)
        if not data: break
        conn.send(data)
@end example

@example
# Echo client program
import socket
import sys

HOST = 'daring.cwi.nl'    # The remote host
PORT = 50007              # The same port as used by the server
s = None
for res in socket.getaddrinfo(HOST, PORT, socket.AF_UNSPEC, socket.SOCK_STREAM):
    af, socktype, proto, canonname, sa = res
    try:
        s = socket.socket(af, socktype, proto)
    except OSError as msg:
        s = None
        continue
    try:
        s.connect(sa)
    except OSError as msg:
        s.close()
        s = None
        continue
    break
if s is None:
    print('could not open socket')
    sys.exit(1)
with s:
    s.sendall(b'Hello, world')
    data = s.recv(1024)
print('Received', repr(data))
@end example

The next example shows how to write a very simple network sniffer with raw
sockets on Windows. The example requires administrator privileges to modify
the interface:

@example
import socket

# the public network interface
HOST = socket.gethostbyname(socket.gethostname())

# create a raw socket and bind it to the public interface
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_IP)
s.bind((HOST, 0))

# Include IP headers
s.setsockopt(socket.IPPROTO_IP, socket.IP_HDRINCL, 1)

# receive all packages
s.ioctl(socket.SIO_RCVALL, socket.RCVALL_ON)

# receive a package
print(s.recvfrom(65565))

# disabled promiscuous mode
s.ioctl(socket.SIO_RCVALL, socket.RCVALL_OFF)
@end example

The next example shows how to use the socket interface to communicate to a CAN
network using the raw socket protocol. To use CAN with the broadcast
manager protocol instead, open a socket with:

@example
socket.socket(socket.AF_CAN, socket.SOCK_DGRAM, socket.CAN_BCM)
@end example

After binding (@code{CAN_RAW}) or connecting (@ref{b24,,CAN_BCM}) the socket, you
can use the @ref{903,,socket.send()}, and the @ref{900,,socket.recv()} operations (and
their counterparts) on the socket object as usual.

This last example might require special privileges:

@example
import socket
import struct


# CAN frame packing/unpacking (see 'struct can_frame' in <linux/can.h>)

can_frame_fmt = "=IB3x8s"
can_frame_size = struct.calcsize(can_frame_fmt)

def build_can_frame(can_id, data):
    can_dlc = len(data)
    data = data.ljust(8, b'\x00')
    return struct.pack(can_frame_fmt, can_id, can_dlc, data)

def dissect_can_frame(frame):
    can_id, can_dlc, data = struct.unpack(can_frame_fmt, frame)
    return (can_id, can_dlc, data[:can_dlc])


# create a raw socket and bind it to the 'vcan0' interface
s = socket.socket(socket.AF_CAN, socket.SOCK_RAW, socket.CAN_RAW)
s.bind(('vcan0',))

while True:
    cf, addr = s.recvfrom(can_frame_size)

    print('Received: can_id=%x, can_dlc=%x, data=%s' % dissect_can_frame(cf))

    try:
        s.send(cf)
    except OSError:
        print('Error sending CAN frame')

    try:
        s.send(build_can_frame(0x01, b'\x01\x02\x03'))
    except OSError:
        print('Error sending CAN frame')
@end example

Running an example several times with too small delay between executions, could
lead to this error:

@example
OSError: [Errno 98] Address already in use
@end example

This is because the previous execution has left the socket in a @code{TIME_WAIT}
state, and can’t be immediately reused.

There is a @ref{ea,,socket} flag to set, in order to prevent this,
@code{socket.SO_REUSEADDR}:

@example
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
@end example

the @code{SO_REUSEADDR} flag tells the kernel to reuse a local socket in
@code{TIME_WAIT} state, without waiting for its natural timeout to expire.


@subsubheading See also


For an introduction to socket programming (in C), see the following papers:


@itemize -

@item 
@emph{An Introductory 4.3BSD Interprocess Communication Tutorial}, by Stuart Sechrest

@item 
@emph{An Advanced 4.3BSD Interprocess Communication Tutorial}, by Samuel J.  Leffler et
al,
@end itemize

both in the UNIX Programmer’s Manual, Supplementary Documents 1 (sections
PS1:7 and PS1:8).  The platform-specific reference material for the various
socket-related system calls are also a valuable source of information on the
details of socket semantics.  For Unix, refer to the manual pages; for Windows,
see the WinSock (or Winsock 2) specification.  For IPv6-ready APIs, readers may
want to refer to 
@geindex RFC; RFC 3493
RFC 3493@footnote{https://tools.ietf.org/html/rfc3493.html} titled Basic Socket Interface Extensions for IPv6.


@node ssl — TLS/SSL wrapper for socket objects,select — Waiting for I/O completion,socket — Low-level networking interface,Networking and Interprocess Communication
@anchor{library/ssl doc}@anchor{2b2a}@anchor{library/ssl module-ssl}@anchor{ee}@anchor{library/ssl ssl-tls-ssl-wrapper-for-socket-objects}@anchor{2b2b}
@subsection @code{ssl} — TLS/SSL wrapper for socket objects


@geindex module; ssl

@strong{Source code:} Lib/ssl.py@footnote{https://github.com/python/cpython/tree/main/Lib/ssl.py}

@geindex OpenSSL; (use in module ssl)

@geindex TLS
@geindex SSL
@geindex Transport Layer Security
@geindex Secure Sockets Layer


__________________________________________________________________


This module provides access to Transport Layer Security (often known as “Secure
Sockets Layer”) encryption and peer authentication facilities for network
sockets, both client-side and server-side.  This module uses the OpenSSL
library. It is available on all modern Unix systems, Windows, macOS, and
probably additional platforms, as long as OpenSSL is installed on that platform.

@cartouche
@quotation Note 
Some behavior may be platform dependent, since calls are made to the
operating system socket APIs.  The installed version of OpenSSL may also
cause variations in behavior. For example, TLSv1.3 with OpenSSL version
1.1.1.
@end quotation
@end cartouche

@cartouche
@quotation Warning 
Don’t use this module without reading the @ref{2a39,,Security considerations}.  Doing so
may lead to a false sense of security, as the default settings of the
ssl module are not necessarily appropriate for your application.
@end quotation
@end cartouche

This section documents the objects and functions in the @code{ssl} module; for more
general information about TLS, SSL, and certificates, the reader is referred to
the documents in the “See Also” section at the bottom.

This module provides a class, @ref{3d6,,ssl.SSLSocket}, which is derived from the
@ref{8fd,,socket.socket} type, and provides a socket-like wrapper that also
encrypts and decrypts the data going over the socket with SSL.  It supports
additional methods such as @code{getpeercert()}, which retrieves the
certificate of the other side of the connection, and @code{cipher()}, which
retrieves the cipher being used for the secure connection.

For more sophisticated applications, the @ref{2bf,,ssl.SSLContext} class
helps manage settings and certificates, which can then be inherited
by SSL sockets created through the @ref{332,,SSLContext.wrap_socket()} method.


Changed in version 3.5.3: Updated to support linking with OpenSSL 1.1.0



Changed in version 3.6: OpenSSL 0.9.8, 1.0.0 and 1.0.1 are deprecated and no longer supported.
In the future the ssl module will require at least OpenSSL 1.0.2 or
1.1.0.



Changed in version 3.10: 
@geindex Python Enhancement Proposals; PEP 644
PEP 644@footnote{https://www.python.org/dev/peps/pep-0644} has been implemented. The ssl module requires OpenSSL 1.1.1
or newer.

Use of deprecated constants and functions result in deprecation warnings.


@menu
* Functions@comma{} Constants@comma{} and Exceptions: Functions Constants and Exceptions. 
* SSL Sockets:: 
* SSL Contexts:: 
* Certificates:: 
* Examples: Examples<18>. 
* Notes on non-blocking sockets:: 
* Memory BIO Support: Memory BIO Support<2>. 
* SSL session:: 
* Security considerations: Security considerations<2>. 
* TLS 1.3: TLS 1 3. 

@end menu

@node Functions Constants and Exceptions,SSL Sockets,,ssl — TLS/SSL wrapper for socket objects
@anchor{library/ssl functions-constants-and-exceptions}@anchor{2b2c}
@subsubsection Functions, Constants, and Exceptions


@menu
* Socket creation:: 
* Context creation:: 
* Exceptions: Exceptions<13>. 
* Random generation:: 
* Certificate handling:: 
* Constants: Constants<9>. 

@end menu

@node Socket creation,Context creation,,Functions Constants and Exceptions
@anchor{library/ssl socket-creation}@anchor{2b2d}
@subsubsection Socket creation


Since Python 3.2 and 2.7.9, it is recommended to use the
@ref{332,,SSLContext.wrap_socket()} of an @ref{2bf,,SSLContext} instance to wrap
sockets as @ref{3d6,,SSLSocket} objects. The helper functions
@ref{b29,,create_default_context()} returns a new context with secure default
settings. The old @ref{331,,wrap_socket()} function is deprecated since it is
both inefficient and has no support for server name indication (SNI) and
hostname matching.

Client socket example with default context and IPv4/IPv6 dual stack:

@example
import socket
import ssl

hostname = 'www.python.org'
context = ssl.create_default_context()

with socket.create_connection((hostname, 443)) as sock:
    with context.wrap_socket(sock, server_hostname=hostname) as ssock:
        print(ssock.version())
@end example

Client socket example with custom context and IPv4:

@example
hostname = 'www.python.org'
# PROTOCOL_TLS_CLIENT requires valid cert chain and hostname
context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
context.load_verify_locations('path/to/cabundle.pem')

with socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) as sock:
    with context.wrap_socket(sock, server_hostname=hostname) as ssock:
        print(ssock.version())
@end example

Server socket example listening on localhost IPv4:

@example
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.load_cert_chain('/path/to/certchain.pem', '/path/to/private.key')

with socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) as sock:
    sock.bind(('127.0.0.1', 8443))
    sock.listen(5)
    with context.wrap_socket(sock, server_side=True) as ssock:
        conn, addr = ssock.accept()
        ...
@end example

@node Context creation,Exceptions<13>,Socket creation,Functions Constants and Exceptions
@anchor{library/ssl context-creation}@anchor{2b2e}
@subsubsection Context creation


A convenience function helps create @ref{2bf,,SSLContext} objects for common
purposes.

@geindex create_default_context() (in module ssl)
@anchor{library/ssl ssl create_default_context}@anchor{b29}
@deffn {Function} ssl.create_default_context (purpose=Purpose.SERVER_AUTH, cafile=None, capath=None, cadata=None)

Return a new @ref{2bf,,SSLContext} object with default settings for
the given @emph{purpose}.  The settings are chosen by the @ref{ee,,ssl} module,
and usually represent a higher security level than when calling the
@ref{2bf,,SSLContext} constructor directly.

@emph{cafile}, @emph{capath}, @emph{cadata} represent optional CA certificates to
trust for certificate verification, as in
@ref{a5f,,SSLContext.load_verify_locations()}.  If all three are
@ref{182,,None}, this function can choose to trust the system’s default
CA certificates instead.

The settings are: @ref{32f,,PROTOCOL_TLS_CLIENT} or
@ref{330,,PROTOCOL_TLS_SERVER}, @ref{2bc,,OP_NO_SSLv2}, and @ref{2bd,,OP_NO_SSLv3}
with high encryption cipher suites without RC4 and
without unauthenticated cipher suites. Passing @ref{b34,,SERVER_AUTH}
as @emph{purpose} sets @ref{2b2f,,verify_mode} to @ref{2b30,,CERT_REQUIRED}
and either loads CA certificates (when at least one of @emph{cafile}, @emph{capath} or
@emph{cadata} is given) or uses @ref{b33,,SSLContext.load_default_certs()} to load
default CA certificates.

When @ref{2b31,,keylog_filename} is supported and the environment
variable 
@geindex SSLKEYLOGFILE
@geindex environment variable; SSLKEYLOGFILE
@code{SSLKEYLOGFILE} is set, @ref{b29,,create_default_context()}
enables key logging.

@cartouche
@quotation Note 
The protocol, options, cipher and other settings may change to more
restrictive values anytime without prior deprecation.  The values
represent a fair balance between compatibility and security.

If your application needs specific settings, you should create a
@ref{2bf,,SSLContext} and apply the settings yourself.
@end quotation
@end cartouche

@cartouche
@quotation Note 
If you find that when certain older clients or servers attempt to connect
with a @ref{2bf,,SSLContext} created by this function that they get an error
stating “Protocol or cipher suite mismatch”, it may be that they only
support SSL3.0 which this function excludes using the
@ref{2bd,,OP_NO_SSLv3}. SSL3.0 is widely considered to be completely broken@footnote{https://en.wikipedia.org/wiki/POODLE}. If you still wish to continue to
use this function but still allow SSL 3.0 connections you can re-enable
them using:

@example
ctx = ssl.create_default_context(Purpose.CLIENT_AUTH)
ctx.options &= ~ssl.OP_NO_SSLv3
@end example
@end quotation
@end cartouche


New in version 3.4.



Changed in version 3.4.4: RC4 was dropped from the default cipher string.



Changed in version 3.6: ChaCha20/Poly1305 was added to the default cipher string.

3DES was dropped from the default cipher string.



Changed in version 3.8: Support for key logging to 
@geindex SSLKEYLOGFILE
@geindex environment variable; SSLKEYLOGFILE
@code{SSLKEYLOGFILE} was added.



Changed in version 3.10: The context now uses @ref{32f,,PROTOCOL_TLS_CLIENT} or
@ref{330,,PROTOCOL_TLS_SERVER} protocol instead of generic
@ref{32e,,PROTOCOL_TLS}.

@end deffn

@node Exceptions<13>,Random generation,Context creation,Functions Constants and Exceptions
@anchor{library/ssl exceptions}@anchor{2b32}
@subsubsection Exceptions


@geindex SSLError
@anchor{library/ssl ssl SSLError}@anchor{e28}
@deffn {Exception} ssl.SSLError

Raised to signal an error from the underlying SSL implementation
(currently provided by the OpenSSL library).  This signifies some
problem in the higher-level encryption and authentication layer that’s
superimposed on the underlying network connection.  This error
is a subtype of @ref{2a1,,OSError}.  The error code and message of
@ref{e28,,SSLError} instances are provided by the OpenSSL library.


Changed in version 3.3: @ref{e28,,SSLError} used to be a subtype of @ref{beb,,socket.error}.


@geindex library (ssl.SSLError attribute)
@anchor{library/ssl ssl SSLError library}@anchor{cdf}
@deffn {Attribute} library

A string mnemonic designating the OpenSSL submodule in which the error
occurred, such as @code{SSL}, @code{PEM} or @code{X509}.  The range of possible
values depends on the OpenSSL version.


New in version 3.3.

@end deffn

@geindex reason (ssl.SSLError attribute)
@anchor{library/ssl ssl SSLError reason}@anchor{ce0}
@deffn {Attribute} reason

A string mnemonic designating the reason this error occurred, for
example @code{CERTIFICATE_VERIFY_FAILED}.  The range of possible
values depends on the OpenSSL version.


New in version 3.3.

@end deffn
@end deffn

@geindex SSLZeroReturnError
@anchor{library/ssl ssl SSLZeroReturnError}@anchor{2b33}
@deffn {Exception} ssl.SSLZeroReturnError

A subclass of @ref{e28,,SSLError} raised when trying to read or write and
the SSL connection has been closed cleanly.  Note that this doesn’t
mean that the underlying transport (read TCP) has been closed.


New in version 3.3.

@end deffn

@geindex SSLWantReadError
@anchor{library/ssl ssl SSLWantReadError}@anchor{9d5}
@deffn {Exception} ssl.SSLWantReadError

A subclass of @ref{e28,,SSLError} raised by a @ref{2b34,,non-blocking SSL socket} when trying to read or write data, but more data needs
to be received on the underlying TCP transport before the request can be
fulfilled.


New in version 3.3.

@end deffn

@geindex SSLWantWriteError
@anchor{library/ssl ssl SSLWantWriteError}@anchor{9d6}
@deffn {Exception} ssl.SSLWantWriteError

A subclass of @ref{e28,,SSLError} raised by a @ref{2b34,,non-blocking SSL socket} when trying to read or write data, but more data needs
to be sent on the underlying TCP transport before the request can be
fulfilled.


New in version 3.3.

@end deffn

@geindex SSLSyscallError
@anchor{library/ssl ssl SSLSyscallError}@anchor{2b35}
@deffn {Exception} ssl.SSLSyscallError

A subclass of @ref{e28,,SSLError} raised when a system error was encountered
while trying to fulfill an operation on a SSL socket.  Unfortunately,
there is no easy way to inspect the original errno number.


New in version 3.3.

@end deffn

@geindex SSLEOFError
@anchor{library/ssl ssl SSLEOFError}@anchor{1121}
@deffn {Exception} ssl.SSLEOFError

A subclass of @ref{e28,,SSLError} raised when the SSL connection has been
terminated abruptly.  Generally, you shouldn’t try to reuse the underlying
transport when this error is encountered.


New in version 3.3.

@end deffn

@geindex SSLCertVerificationError
@anchor{library/ssl ssl SSLCertVerificationError}@anchor{6aa}
@deffn {Exception} ssl.SSLCertVerificationError

A subclass of @ref{e28,,SSLError} raised when certificate validation has
failed.


New in version 3.7.


@geindex verify_code (ssl.SSLCertVerificationError attribute)
@anchor{library/ssl ssl SSLCertVerificationError verify_code}@anchor{2b36}
@deffn {Attribute} verify_code

A numeric error number that denotes the verification error.
@end deffn

@geindex verify_message (ssl.SSLCertVerificationError attribute)
@anchor{library/ssl ssl SSLCertVerificationError verify_message}@anchor{2b37}
@deffn {Attribute} verify_message

A human readable string of the verification error.
@end deffn
@end deffn

@geindex CertificateError
@anchor{library/ssl ssl CertificateError}@anchor{2b38}
@deffn {Exception} ssl.CertificateError

An alias for @ref{6aa,,SSLCertVerificationError}.


Changed in version 3.7: The exception is now an alias for @ref{6aa,,SSLCertVerificationError}.

@end deffn

@node Random generation,Certificate handling,Exceptions<13>,Functions Constants and Exceptions
@anchor{library/ssl random-generation}@anchor{2b39}
@subsubsection Random generation


@geindex RAND_bytes() (in module ssl)
@anchor{library/ssl ssl RAND_bytes}@anchor{cd8}
@deffn {Function} ssl.RAND_bytes (num)

Return @emph{num} cryptographically strong pseudo-random bytes. Raises an
@ref{e28,,SSLError} if the PRNG has not been seeded with enough data or if the
operation is not supported by the current RAND method. @ref{113d,,RAND_status()}
can be used to check the status of the PRNG and @ref{2b3a,,RAND_add()} can be used
to seed the PRNG.

For almost all applications @ref{17f,,os.urandom()} is preferable.

Read the Wikipedia article, Cryptographically secure pseudorandom number generator (CSPRNG)@footnote{https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator},
to get the requirements of a cryptographically strong generator.


New in version 3.3.

@end deffn

@geindex RAND_pseudo_bytes() (in module ssl)
@anchor{library/ssl ssl RAND_pseudo_bytes}@anchor{334}
@deffn {Function} ssl.RAND_pseudo_bytes (num)

Return (bytes, is_cryptographic): bytes are @emph{num} pseudo-random bytes,
is_cryptographic is @code{True} if the bytes generated are cryptographically
strong. Raises an @ref{e28,,SSLError} if the operation is not supported by the
current RAND method.

Generated pseudo-random byte sequences will be unique if they are of
sufficient length, but are not necessarily unpredictable. They can be used
for non-cryptographic purposes and for certain purposes in cryptographic
protocols, but usually not for key generation etc.

For almost all applications @ref{17f,,os.urandom()} is preferable.


New in version 3.3.



Deprecated since version 3.6: OpenSSL has deprecated @ref{334,,ssl.RAND_pseudo_bytes()}, use
@ref{cd8,,ssl.RAND_bytes()} instead.

@end deffn

@geindex RAND_status() (in module ssl)
@anchor{library/ssl ssl RAND_status}@anchor{113d}
@deffn {Function} ssl.RAND_status ()

Return @code{True} if the SSL pseudo-random number generator has been seeded
with ‘enough’ randomness, and @code{False} otherwise.  You can use
@code{ssl.RAND_egd()} and @ref{2b3a,,ssl.RAND_add()} to increase the randomness of
the pseudo-random number generator.
@end deffn

@geindex RAND_add() (in module ssl)
@anchor{library/ssl ssl RAND_add}@anchor{2b3a}
@deffn {Function} ssl.RAND_add (bytes, entropy)

Mix the given @emph{bytes} into the SSL pseudo-random number generator.  The
parameter @emph{entropy} (a float) is a lower bound on the entropy contained in
string (so you can always use @code{0.0}).  See 
@geindex RFC; RFC 1750
RFC 1750@footnote{https://tools.ietf.org/html/rfc1750.html} for more
information on sources of entropy.


Changed in version 3.5: Writable @ref{87e,,bytes-like object} is now accepted.

@end deffn

@node Certificate handling,Constants<9>,Random generation,Functions Constants and Exceptions
@anchor{library/ssl certificate-handling}@anchor{2b3b}
@subsubsection Certificate handling


@c import ssl

@geindex match_hostname() (in module ssl)
@anchor{library/ssl ssl match_hostname}@anchor{333}
@deffn {Function} ssl.match_hostname (cert, hostname)

Verify that @emph{cert} (in decoded format as returned by
@ref{b39,,SSLSocket.getpeercert()}) matches the given @emph{hostname}.  The rules
applied are those for checking the identity of HTTPS servers as outlined
in 
@geindex RFC; RFC 2818
RFC 2818@footnote{https://tools.ietf.org/html/rfc2818.html}, 
@geindex RFC; RFC 5280
RFC 5280@footnote{https://tools.ietf.org/html/rfc5280.html} and 
@geindex RFC; RFC 6125
RFC 6125@footnote{https://tools.ietf.org/html/rfc6125.html}.  In addition to HTTPS, this
function should be suitable for checking the identity of servers in
various SSL-based protocols such as FTPS, IMAPS, POPS and others.

@ref{2b38,,CertificateError} is raised on failure. On success, the function
returns nothing:

@example
>>> cert = @{'subject': ((('commonName', 'example.com'),),)@}
>>> ssl.match_hostname(cert, "example.com")
>>> ssl.match_hostname(cert, "example.org")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/py3k/Lib/ssl.py", line 130, in match_hostname
ssl.CertificateError: hostname 'example.org' doesn't match 'example.com'
@end example


New in version 3.2.



Changed in version 3.3.3: The function now follows 
@geindex RFC; RFC 6125
RFC 6125@footnote{https://tools.ietf.org/html/rfc6125.html}, section 6.4.3 and does neither
match multiple wildcards (e.g. @code{*.*.com} or @code{*a*.example.org}) nor
a wildcard inside an internationalized domain names (IDN) fragment.
IDN A-labels such as @code{www*.xn--pthon-kva.org} are still supported,
but @code{x*.python.org} no longer matches @code{xn--tda.python.org}.



Changed in version 3.5: Matching of IP addresses, when present in the subjectAltName field
of the certificate, is now supported.



Changed in version 3.7: The function is no longer used to TLS connections. Hostname matching
is now performed by OpenSSL.

Allow wildcard when it is the leftmost and the only character
in that segment. Partial wildcards like @code{www*.example.com} are no
longer supported.



Deprecated since version 3.7.

@end deffn

@geindex cert_time_to_seconds() (in module ssl)
@anchor{library/ssl ssl cert_time_to_seconds}@anchor{9d7}
@deffn {Function} ssl.cert_time_to_seconds (cert_time)

Return the time in seconds since the Epoch, given the @code{cert_time}
string representing the “notBefore” or “notAfter” date from a
certificate in @code{"%b %d %H:%M:%S %Y %Z"} strptime format (C
locale).

Here’s an example:

@example
>>> import ssl
>>> timestamp = ssl.cert_time_to_seconds("Jan  5 09:34:43 2018 GMT")
>>> timestamp  
1515144883
>>> from datetime import datetime
>>> print(datetime.utcfromtimestamp(timestamp))  
2018-01-05 09:34:43
@end example

“notBefore” or “notAfter” dates must use GMT (
@geindex RFC; RFC 5280
RFC 5280@footnote{https://tools.ietf.org/html/rfc5280.html}).


Changed in version 3.5: Interpret the input time as a time in UTC as specified by ‘GMT’
timezone in the input string. Local timezone was used
previously. Return an integer (no fractions of a second in the
input format)

@end deffn

@geindex get_server_certificate() (in module ssl)
@anchor{library/ssl ssl get_server_certificate}@anchor{2c0}
@deffn {Function} ssl.get_server_certificate (addr, ssl_version=PROTOCOL_TLS_CLIENT, ca_certs=None[, timeout])

Given the address @code{addr} of an SSL-protected server, as a (@emph{hostname},
@emph{port-number}) pair, fetches the server’s certificate, and returns it as a
PEM-encoded string.  If @code{ssl_version} is specified, uses that version of
the SSL protocol to attempt to connect to the server.  If @code{ca_certs} is
specified, it should be a file containing a list of root certificates, the
same format as used for the same parameter in
@ref{332,,SSLContext.wrap_socket()}.  The call will attempt to validate the
server certificate against that set of root certificates, and will fail
if the validation attempt fails.  A timeout can be specified with the
@code{timeout} parameter.


Changed in version 3.3: This function is now IPv6-compatible.



Changed in version 3.5: The default @emph{ssl_version} is changed from @ref{329,,PROTOCOL_SSLv3} to
@ref{32e,,PROTOCOL_TLS} for maximum compatibility with modern servers.



Changed in version 3.10: The @emph{timeout} parameter was added.

@end deffn

@geindex DER_cert_to_PEM_cert() (in module ssl)
@anchor{library/ssl ssl DER_cert_to_PEM_cert}@anchor{2b3c}
@deffn {Function} ssl.DER_cert_to_PEM_cert (DER_cert_bytes)

Given a certificate as a DER-encoded blob of bytes, returns a PEM-encoded
string version of the same certificate.
@end deffn

@geindex PEM_cert_to_DER_cert() (in module ssl)
@anchor{library/ssl ssl PEM_cert_to_DER_cert}@anchor{2b3d}
@deffn {Function} ssl.PEM_cert_to_DER_cert (PEM_cert_string)

Given a certificate as an ASCII PEM string, returns a DER-encoded sequence of
bytes for that same certificate.
@end deffn

@geindex get_default_verify_paths() (in module ssl)
@anchor{library/ssl ssl get_default_verify_paths}@anchor{b2a}
@deffn {Function} ssl.get_default_verify_paths ()

Returns a named tuple with paths to OpenSSL’s default cafile and capath.
The paths are the same as used by
@ref{b2b,,SSLContext.set_default_verify_paths()}. The return value is a
@ref{ce9,,named tuple} @code{DefaultVerifyPaths}:


@itemize *

@item 
@code{cafile} - resolved path to cafile or @code{None} if the file doesn’t exist,

@item 
@code{capath} - resolved path to capath or @code{None} if the directory doesn’t exist,

@item 
@code{openssl_cafile_env} - OpenSSL’s environment key that points to a cafile,

@item 
@code{openssl_cafile} - hard coded path to a cafile,

@item 
@code{openssl_capath_env} - OpenSSL’s environment key that points to a capath,

@item 
@code{openssl_capath} - hard coded path to a capath directory
@end itemize

@ref{1629,,Availability}: LibreSSL ignores the environment vars
@code{openssl_cafile_env} and @code{openssl_capath_env}.


New in version 3.4.

@end deffn

@geindex enum_certificates() (in module ssl)
@anchor{library/ssl ssl enum_certificates}@anchor{b36}
@deffn {Function} ssl.enum_certificates (store_name)

Retrieve certificates from Windows’ system cert store. @emph{store_name} may be
one of @code{CA}, @code{ROOT} or @code{MY}. Windows may provide additional cert
stores, too.

The function returns a list of (cert_bytes, encoding_type, trust) tuples.
The encoding_type specifies the encoding of cert_bytes. It is either
@code{x509_asn} for X.509 ASN.1 data or @code{pkcs_7_asn} for
PKCS#7 ASN.1 data. Trust specifies the purpose of the certificate as a set
of OIDS or exactly @code{True} if the certificate is trustworthy for all
purposes.

Example:

@example
>>> ssl.enum_certificates("CA")
[(b'data...', 'x509_asn', @{'1.3.6.1.5.5.7.3.1', '1.3.6.1.5.5.7.3.2'@}),
 (b'data...', 'x509_asn', True)]
@end example

@ref{1629,,Availability}: Windows.


New in version 3.4.

@end deffn

@geindex enum_crls() (in module ssl)
@anchor{library/ssl ssl enum_crls}@anchor{b37}
@deffn {Function} ssl.enum_crls (store_name)

Retrieve CRLs from Windows’ system cert store. @emph{store_name} may be
one of @code{CA}, @code{ROOT} or @code{MY}. Windows may provide additional cert
stores, too.

The function returns a list of (cert_bytes, encoding_type, trust) tuples.
The encoding_type specifies the encoding of cert_bytes. It is either
@code{x509_asn} for X.509 ASN.1 data or @code{pkcs_7_asn} for
PKCS#7 ASN.1 data.

@ref{1629,,Availability}: Windows.


New in version 3.4.

@end deffn

@geindex wrap_socket() (in module ssl)
@anchor{library/ssl ssl wrap_socket}@anchor{331}
@deffn {Function} ssl.wrap_socket (sock, keyfile=None, certfile=None, server_side=False, cert_reqs=CERT_NONE, ssl_version=PROTOCOL_TLS, ca_certs=None, do_handshake_on_connect=True, suppress_ragged_eofs=True, ciphers=None)

Takes an instance @code{sock} of @ref{8fd,,socket.socket}, and returns an instance
of @ref{3d6,,ssl.SSLSocket}, a subtype of @ref{8fd,,socket.socket}, which wraps
the underlying socket in an SSL context.  @code{sock} must be a
@ref{e91,,SOCK_STREAM} socket; other socket types are unsupported.

Internally, function creates a @ref{2bf,,SSLContext} with protocol
@emph{ssl_version} and @ref{2bb,,SSLContext.options} set to @emph{cert_reqs}. If
parameters @emph{keyfile}, @emph{certfile}, @emph{ca_certs} or @emph{ciphers} are set, then
the values are passed to @ref{cd9,,SSLContext.load_cert_chain()},
@ref{a5f,,SSLContext.load_verify_locations()}, and
@ref{2b3e,,SSLContext.set_ciphers()}.

The arguments @emph{server_side}, @emph{do_handshake_on_connect}, and
@emph{suppress_ragged_eofs} have the same meaning as
@ref{332,,SSLContext.wrap_socket()}.


Deprecated since version 3.7: Since Python 3.2 and 2.7.9, it is recommended to use the
@ref{332,,SSLContext.wrap_socket()} instead of @ref{331,,wrap_socket()}. The
top-level function is limited and creates an insecure client socket
without server name indication or hostname matching.

@end deffn

@node Constants<9>,,Certificate handling,Functions Constants and Exceptions
@anchor{library/ssl constants}@anchor{2b3f}
@subsubsection Constants


@quotation

All constants are now @ref{828,,enum.IntEnum} or @ref{11c2,,enum.IntFlag} collections.


New in version 3.6.

@end quotation

@geindex CERT_NONE (in module ssl)
@anchor{library/ssl ssl CERT_NONE}@anchor{2b40}
@deffn {Data} ssl.CERT_NONE

Possible value for @ref{2b2f,,SSLContext.verify_mode}, or the @code{cert_reqs}
parameter to @ref{331,,wrap_socket()}.  Except for @ref{32f,,PROTOCOL_TLS_CLIENT},
it is the default mode.  With client-side sockets, just about any
cert is accepted.  Validation errors, such as untrusted or expired cert,
are ignored and do not abort the TLS/SSL handshake.

In server mode, no certificate is requested from the client, so the client
does not send any for client cert authentication.

See the discussion of @ref{2a39,,Security considerations} below.
@end deffn

@geindex CERT_OPTIONAL (in module ssl)
@anchor{library/ssl ssl CERT_OPTIONAL}@anchor{2b41}
@deffn {Data} ssl.CERT_OPTIONAL

Possible value for @ref{2b2f,,SSLContext.verify_mode}, or the @code{cert_reqs}
parameter to @ref{331,,wrap_socket()}.  In client mode, @ref{2b41,,CERT_OPTIONAL}
has the same meaning as @ref{2b30,,CERT_REQUIRED}. It is recommended to
use @ref{2b30,,CERT_REQUIRED} for client-side sockets instead.

In server mode, a client certificate request is sent to the client.  The
client may either ignore the request or send a certificate in order
perform TLS client cert authentication.  If the client chooses to send
a certificate, it is verified.  Any verification error immediately aborts
the TLS handshake.

Use of this setting requires a valid set of CA certificates to
be passed, either to @ref{a5f,,SSLContext.load_verify_locations()} or as a
value of the @code{ca_certs} parameter to @ref{331,,wrap_socket()}.
@end deffn

@geindex CERT_REQUIRED (in module ssl)
@anchor{library/ssl ssl CERT_REQUIRED}@anchor{2b30}
@deffn {Data} ssl.CERT_REQUIRED

Possible value for @ref{2b2f,,SSLContext.verify_mode}, or the @code{cert_reqs}
parameter to @ref{331,,wrap_socket()}.  In this mode, certificates are
required from the other side of the socket connection; an @ref{e28,,SSLError}
will be raised if no certificate is provided, or if its validation fails.
This mode is @strong{not} sufficient to verify a certificate in client mode as
it does not match hostnames.  @ref{2b42,,check_hostname} must be
enabled as well to verify the authenticity of a cert.
@ref{32f,,PROTOCOL_TLS_CLIENT} uses @ref{2b30,,CERT_REQUIRED} and
enables @ref{2b42,,check_hostname} by default.

With server socket, this mode provides mandatory TLS client cert
authentication.  A client certificate request is sent to the client and
the client must provide a valid and trusted certificate.

Use of this setting requires a valid set of CA certificates to
be passed, either to @ref{a5f,,SSLContext.load_verify_locations()} or as a
value of the @code{ca_certs} parameter to @ref{331,,wrap_socket()}.
@end deffn

@geindex VerifyMode (class in ssl)
@anchor{library/ssl ssl VerifyMode}@anchor{2b43}
@deffn {Class} ssl.VerifyMode

@ref{828,,enum.IntEnum} collection of CERT_* constants.


New in version 3.6.

@end deffn

@geindex VERIFY_DEFAULT (in module ssl)
@anchor{library/ssl ssl VERIFY_DEFAULT}@anchor{b2f}
@deffn {Data} ssl.VERIFY_DEFAULT

Possible value for @ref{b2e,,SSLContext.verify_flags}. In this mode, certificate
revocation lists (CRLs) are not checked. By default OpenSSL does neither
require nor verify CRLs.


New in version 3.4.

@end deffn

@geindex VERIFY_CRL_CHECK_LEAF (in module ssl)
@anchor{library/ssl ssl VERIFY_CRL_CHECK_LEAF}@anchor{b30}
@deffn {Data} ssl.VERIFY_CRL_CHECK_LEAF

Possible value for @ref{b2e,,SSLContext.verify_flags}. In this mode, only the
peer cert is checked but none of the intermediate CA certificates. The mode
requires a valid CRL that is signed by the peer cert’s issuer (its direct
ancestor CA). If no proper CRL has been loaded with
@ref{a5f,,SSLContext.load_verify_locations}, validation will fail.


New in version 3.4.

@end deffn

@geindex VERIFY_CRL_CHECK_CHAIN (in module ssl)
@anchor{library/ssl ssl VERIFY_CRL_CHECK_CHAIN}@anchor{b31}
@deffn {Data} ssl.VERIFY_CRL_CHECK_CHAIN

Possible value for @ref{b2e,,SSLContext.verify_flags}. In this mode, CRLs of
all certificates in the peer cert chain are checked.


New in version 3.4.

@end deffn

@geindex VERIFY_X509_STRICT (in module ssl)
@anchor{library/ssl ssl VERIFY_X509_STRICT}@anchor{b32}
@deffn {Data} ssl.VERIFY_X509_STRICT

Possible value for @ref{b2e,,SSLContext.verify_flags} to disable workarounds
for broken X.509 certificates.


New in version 3.4.

@end deffn

@geindex VERIFY_ALLOW_PROXY_CERTS (in module ssl)
@anchor{library/ssl ssl VERIFY_ALLOW_PROXY_CERTS}@anchor{1155}
@deffn {Data} ssl.VERIFY_ALLOW_PROXY_CERTS

Possible value for @ref{b2e,,SSLContext.verify_flags} to enables proxy
certificate verification.


New in version 3.10.

@end deffn

@geindex VERIFY_X509_TRUSTED_FIRST (in module ssl)
@anchor{library/ssl ssl VERIFY_X509_TRUSTED_FIRST}@anchor{2b44}
@deffn {Data} ssl.VERIFY_X509_TRUSTED_FIRST

Possible value for @ref{b2e,,SSLContext.verify_flags}. It instructs OpenSSL to
prefer trusted certificates when building the trust chain to validate a
certificate. This flag is enabled by default.


New in version 3.4.4.

@end deffn

@geindex VERIFY_X509_PARTIAL_CHAIN (in module ssl)
@anchor{library/ssl ssl VERIFY_X509_PARTIAL_CHAIN}@anchor{2c1}
@deffn {Data} ssl.VERIFY_X509_PARTIAL_CHAIN

Possible value for @ref{b2e,,SSLContext.verify_flags}. It instructs OpenSSL to
accept intermediate CAs in the trust store to be treated as trust-anchors,
in the same way as the self-signed root CA certificates. This makes it
possible to trust certificates issued by an intermediate CA without having
to trust its ancestor root CA.


New in version 3.10.

@end deffn

@geindex VerifyFlags (class in ssl)
@anchor{library/ssl ssl VerifyFlags}@anchor{2b45}
@deffn {Class} ssl.VerifyFlags

@ref{11c2,,enum.IntFlag} collection of VERIFY_* constants.


New in version 3.6.

@end deffn

@geindex PROTOCOL_TLS (in module ssl)
@anchor{library/ssl ssl PROTOCOL_TLS}@anchor{32e}
@deffn {Data} ssl.PROTOCOL_TLS

Selects the highest protocol version that both the client and server support.
Despite the name, this option can select both “SSL” and “TLS” protocols.


New in version 3.6.



Deprecated since version 3.10: TLS clients and servers require different default settings for secure
communication. The generic TLS protocol constant is deprecated in
favor of @ref{32f,,PROTOCOL_TLS_CLIENT} and @ref{330,,PROTOCOL_TLS_SERVER}.

@end deffn

@geindex PROTOCOL_TLS_CLIENT (in module ssl)
@anchor{library/ssl ssl PROTOCOL_TLS_CLIENT}@anchor{32f}
@deffn {Data} ssl.PROTOCOL_TLS_CLIENT

Auto-negotiate the highest protocol version that both the client and
server support, and configure the context client-side connections. The
protocol enables @ref{2b30,,CERT_REQUIRED} and
@ref{2b42,,check_hostname} by default.


New in version 3.6.

@end deffn

@geindex PROTOCOL_TLS_SERVER (in module ssl)
@anchor{library/ssl ssl PROTOCOL_TLS_SERVER}@anchor{330}
@deffn {Data} ssl.PROTOCOL_TLS_SERVER

Auto-negotiate the highest protocol version that both the client and
server support, and configure the context server-side connections.


New in version 3.6.

@end deffn

@geindex PROTOCOL_SSLv23 (in module ssl)
@anchor{library/ssl ssl PROTOCOL_SSLv23}@anchor{32a}
@deffn {Data} ssl.PROTOCOL_SSLv23

Alias for @ref{32e,,PROTOCOL_TLS}.


Deprecated since version 3.6: Use @ref{32e,,PROTOCOL_TLS} instead.

@end deffn

@geindex PROTOCOL_SSLv2 (in module ssl)
@anchor{library/ssl ssl PROTOCOL_SSLv2}@anchor{328}
@deffn {Data} ssl.PROTOCOL_SSLv2

Selects SSL version 2 as the channel encryption protocol.

This protocol is not available if OpenSSL is compiled with the
@code{OPENSSL_NO_SSL2} flag.

@cartouche
@quotation Warning 
SSL version 2 is insecure.  Its use is highly discouraged.
@end quotation
@end cartouche


Deprecated since version 3.6: OpenSSL has removed support for SSLv2.

@end deffn

@geindex PROTOCOL_SSLv3 (in module ssl)
@anchor{library/ssl ssl PROTOCOL_SSLv3}@anchor{329}
@deffn {Data} ssl.PROTOCOL_SSLv3

Selects SSL version 3 as the channel encryption protocol.

This protocol is not be available if OpenSSL is compiled with the
@code{OPENSSL_NO_SSLv3} flag.

@cartouche
@quotation Warning 
SSL version 3 is insecure.  Its use is highly discouraged.
@end quotation
@end cartouche


Deprecated since version 3.6: OpenSSL has deprecated all version specific protocols. Use the default
protocol @ref{330,,PROTOCOL_TLS_SERVER} or @ref{32f,,PROTOCOL_TLS_CLIENT}
with @ref{6b1,,SSLContext.minimum_version} and
@ref{6b2,,SSLContext.maximum_version} instead.

@end deffn

@geindex PROTOCOL_TLSv1 (in module ssl)
@anchor{library/ssl ssl PROTOCOL_TLSv1}@anchor{32b}
@deffn {Data} ssl.PROTOCOL_TLSv1

Selects TLS version 1.0 as the channel encryption protocol.


Deprecated since version 3.6: OpenSSL has deprecated all version specific protocols.

@end deffn

@geindex PROTOCOL_TLSv1_1 (in module ssl)
@anchor{library/ssl ssl PROTOCOL_TLSv1_1}@anchor{32c}
@deffn {Data} ssl.PROTOCOL_TLSv1_1

Selects TLS version 1.1 as the channel encryption protocol.
Available only with openssl version 1.0.1+.


New in version 3.4.



Deprecated since version 3.6: OpenSSL has deprecated all version specific protocols.

@end deffn

@geindex PROTOCOL_TLSv1_2 (in module ssl)
@anchor{library/ssl ssl PROTOCOL_TLSv1_2}@anchor{32d}
@deffn {Data} ssl.PROTOCOL_TLSv1_2

Selects TLS version 1.2 as the channel encryption protocol.
Available only with openssl version 1.0.1+.


New in version 3.4.



Deprecated since version 3.6: OpenSSL has deprecated all version specific protocols.

@end deffn

@geindex OP_ALL (in module ssl)
@anchor{library/ssl ssl OP_ALL}@anchor{2b46}
@deffn {Data} ssl.OP_ALL

Enables workarounds for various bugs present in other SSL implementations.
This option is set by default.  It does not necessarily set the same
flags as OpenSSL’s @code{SSL_OP_ALL} constant.


New in version 3.2.

@end deffn

@geindex OP_NO_SSLv2 (in module ssl)
@anchor{library/ssl ssl OP_NO_SSLv2}@anchor{2bc}
@deffn {Data} ssl.OP_NO_SSLv2

Prevents an SSLv2 connection.  This option is only applicable in
conjunction with @ref{32e,,PROTOCOL_TLS}.  It prevents the peers from
choosing SSLv2 as the protocol version.


New in version 3.2.



Deprecated since version 3.6: SSLv2 is deprecated

@end deffn

@geindex OP_NO_SSLv3 (in module ssl)
@anchor{library/ssl ssl OP_NO_SSLv3}@anchor{2bd}
@deffn {Data} ssl.OP_NO_SSLv3

Prevents an SSLv3 connection.  This option is only applicable in
conjunction with @ref{32e,,PROTOCOL_TLS}.  It prevents the peers from
choosing SSLv3 as the protocol version.


New in version 3.2.



Deprecated since version 3.6: SSLv3 is deprecated

@end deffn

@geindex OP_NO_TLSv1 (in module ssl)
@anchor{library/ssl ssl OP_NO_TLSv1}@anchor{324}
@deffn {Data} ssl.OP_NO_TLSv1

Prevents a TLSv1 connection.  This option is only applicable in
conjunction with @ref{32e,,PROTOCOL_TLS}.  It prevents the peers from
choosing TLSv1 as the protocol version.


New in version 3.2.



Deprecated since version 3.7: The option is deprecated since OpenSSL 1.1.0, use the new
@ref{6b1,,SSLContext.minimum_version} and
@ref{6b2,,SSLContext.maximum_version} instead.

@end deffn

@geindex OP_NO_TLSv1_1 (in module ssl)
@anchor{library/ssl ssl OP_NO_TLSv1_1}@anchor{325}
@deffn {Data} ssl.OP_NO_TLSv1_1

Prevents a TLSv1.1 connection. This option is only applicable in conjunction
with @ref{32e,,PROTOCOL_TLS}. It prevents the peers from choosing TLSv1.1 as
the protocol version. Available only with openssl version 1.0.1+.


New in version 3.4.



Deprecated since version 3.7: The option is deprecated since OpenSSL 1.1.0.

@end deffn

@geindex OP_NO_TLSv1_2 (in module ssl)
@anchor{library/ssl ssl OP_NO_TLSv1_2}@anchor{326}
@deffn {Data} ssl.OP_NO_TLSv1_2

Prevents a TLSv1.2 connection. This option is only applicable in conjunction
with @ref{32e,,PROTOCOL_TLS}. It prevents the peers from choosing TLSv1.2 as
the protocol version. Available only with openssl version 1.0.1+.


New in version 3.4.



Deprecated since version 3.7: The option is deprecated since OpenSSL 1.1.0.

@end deffn

@geindex OP_NO_TLSv1_3 (in module ssl)
@anchor{library/ssl ssl OP_NO_TLSv1_3}@anchor{327}
@deffn {Data} ssl.OP_NO_TLSv1_3

Prevents a TLSv1.3 connection. This option is only applicable in conjunction
with @ref{32e,,PROTOCOL_TLS}. It prevents the peers from choosing TLSv1.3 as
the protocol version. TLS 1.3 is available with OpenSSL 1.1.1 or later.
When Python has been compiled against an older version of OpenSSL, the
flag defaults to @emph{0}.


New in version 3.7.



Deprecated since version 3.7: The option is deprecated since OpenSSL 1.1.0. It was added to 2.7.15,
3.6.3 and 3.7.0 for backwards compatibility with OpenSSL 1.0.2.

@end deffn

@geindex OP_NO_RENEGOTIATION (in module ssl)
@anchor{library/ssl ssl OP_NO_RENEGOTIATION}@anchor{2b47}
@deffn {Data} ssl.OP_NO_RENEGOTIATION

Disable all renegotiation in TLSv1.2 and earlier. Do not send
HelloRequest messages, and ignore renegotiation requests via ClientHello.

This option is only available with OpenSSL 1.1.0h and later.


New in version 3.7.

@end deffn

@geindex OP_CIPHER_SERVER_PREFERENCE (in module ssl)
@anchor{library/ssl ssl OP_CIPHER_SERVER_PREFERENCE}@anchor{ce1}
@deffn {Data} ssl.OP_CIPHER_SERVER_PREFERENCE

Use the server’s cipher ordering preference, rather than the client’s.
This option has no effect on client sockets and SSLv2 server sockets.


New in version 3.3.

@end deffn

@geindex OP_SINGLE_DH_USE (in module ssl)
@anchor{library/ssl ssl OP_SINGLE_DH_USE}@anchor{2b48}
@deffn {Data} ssl.OP_SINGLE_DH_USE

Prevents re-use of the same DH key for distinct SSL sessions.  This
improves forward secrecy but requires more computational resources.
This option only applies to server sockets.


New in version 3.3.

@end deffn

@geindex OP_SINGLE_ECDH_USE (in module ssl)
@anchor{library/ssl ssl OP_SINGLE_ECDH_USE}@anchor{2b49}
@deffn {Data} ssl.OP_SINGLE_ECDH_USE

Prevents re-use of the same ECDH key for distinct SSL sessions.  This
improves forward secrecy but requires more computational resources.
This option only applies to server sockets.


New in version 3.3.

@end deffn

@geindex OP_ENABLE_MIDDLEBOX_COMPAT (in module ssl)
@anchor{library/ssl ssl OP_ENABLE_MIDDLEBOX_COMPAT}@anchor{2b4a}
@deffn {Data} ssl.OP_ENABLE_MIDDLEBOX_COMPAT

Send dummy Change Cipher Spec (CCS) messages in TLS 1.3 handshake to make
a TLS 1.3 connection look more like a TLS 1.2 connection.

This option is only available with OpenSSL 1.1.1 and later.


New in version 3.8.

@end deffn

@geindex OP_NO_COMPRESSION (in module ssl)
@anchor{library/ssl ssl OP_NO_COMPRESSION}@anchor{cde}
@deffn {Data} ssl.OP_NO_COMPRESSION

Disable compression on the SSL channel.  This is useful if the application
protocol supports its own compression scheme.


New in version 3.3.

@end deffn

@geindex Options (class in ssl)
@anchor{library/ssl ssl Options}@anchor{2b4b}
@deffn {Class} ssl.Options

@ref{11c2,,enum.IntFlag} collection of OP_* constants.
@end deffn

@geindex OP_NO_TICKET (in module ssl)
@anchor{library/ssl ssl OP_NO_TICKET}@anchor{2b4c}
@deffn {Data} ssl.OP_NO_TICKET

Prevent client side from requesting a session ticket.


New in version 3.6.

@end deffn

@geindex OP_IGNORE_UNEXPECTED_EOF (in module ssl)
@anchor{library/ssl ssl OP_IGNORE_UNEXPECTED_EOF}@anchor{2ba}
@deffn {Data} ssl.OP_IGNORE_UNEXPECTED_EOF

Ignore unexpected shutdown of TLS connections.

This option is only available with OpenSSL 3.0.0 and later.


New in version 3.10.

@end deffn

@geindex HAS_ALPN (in module ssl)
@anchor{library/ssl ssl HAS_ALPN}@anchor{9d2}
@deffn {Data} ssl.HAS_ALPN

Whether the OpenSSL library has built-in support for the @emph{Application-Layer
Protocol Negotiation} TLS extension as described in 
@geindex RFC; RFC 7301
RFC 7301@footnote{https://tools.ietf.org/html/rfc7301.html}.


New in version 3.5.

@end deffn

@geindex HAS_NEVER_CHECK_COMMON_NAME (in module ssl)
@anchor{library/ssl ssl HAS_NEVER_CHECK_COMMON_NAME}@anchor{2b4d}
@deffn {Data} ssl.HAS_NEVER_CHECK_COMMON_NAME

Whether the OpenSSL library has built-in support not checking subject
common name and @ref{6ab,,SSLContext.hostname_checks_common_name} is
writeable.


New in version 3.7.

@end deffn

@geindex HAS_ECDH (in module ssl)
@anchor{library/ssl ssl HAS_ECDH}@anchor{2b4e}
@deffn {Data} ssl.HAS_ECDH

Whether the OpenSSL library has built-in support for the Elliptic Curve-based
Diffie-Hellman key exchange.  This should be true unless the feature was
explicitly disabled by the distributor.


New in version 3.3.

@end deffn

@geindex HAS_SNI (in module ssl)
@anchor{library/ssl ssl HAS_SNI}@anchor{2b4f}
@deffn {Data} ssl.HAS_SNI

Whether the OpenSSL library has built-in support for the @emph{Server Name
Indication} extension (as defined in 
@geindex RFC; RFC 6066
RFC 6066@footnote{https://tools.ietf.org/html/rfc6066.html}).


New in version 3.2.

@end deffn

@geindex HAS_NPN (in module ssl)
@anchor{library/ssl ssl HAS_NPN}@anchor{2b50}
@deffn {Data} ssl.HAS_NPN

Whether the OpenSSL library has built-in support for the @emph{Next Protocol
Negotiation} as described in the Application Layer Protocol Negotiation@footnote{https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation}.
When true, you can use the @ref{336,,SSLContext.set_npn_protocols()} method to advertise
which protocols you want to support.


New in version 3.3.

@end deffn

@geindex HAS_SSLv2 (in module ssl)
@anchor{library/ssl ssl HAS_SSLv2}@anchor{2b51}
@deffn {Data} ssl.HAS_SSLv2

Whether the OpenSSL library has built-in support for the SSL 2.0 protocol.


New in version 3.7.

@end deffn

@geindex HAS_SSLv3 (in module ssl)
@anchor{library/ssl ssl HAS_SSLv3}@anchor{2b52}
@deffn {Data} ssl.HAS_SSLv3

Whether the OpenSSL library has built-in support for the SSL 3.0 protocol.


New in version 3.7.

@end deffn

@geindex HAS_TLSv1 (in module ssl)
@anchor{library/ssl ssl HAS_TLSv1}@anchor{2b53}
@deffn {Data} ssl.HAS_TLSv1

Whether the OpenSSL library has built-in support for the TLS 1.0 protocol.


New in version 3.7.

@end deffn

@geindex HAS_TLSv1_1 (in module ssl)
@anchor{library/ssl ssl HAS_TLSv1_1}@anchor{6b3}
@deffn {Data} ssl.HAS_TLSv1_1

Whether the OpenSSL library has built-in support for the TLS 1.1 protocol.


New in version 3.7.

@end deffn

@geindex HAS_TLSv1_2 (in module ssl)
@anchor{library/ssl ssl HAS_TLSv1_2}@anchor{2b54}
@deffn {Data} ssl.HAS_TLSv1_2

Whether the OpenSSL library has built-in support for the TLS 1.2 protocol.


New in version 3.7.

@end deffn

@geindex HAS_TLSv1_3 (in module ssl)
@anchor{library/ssl ssl HAS_TLSv1_3}@anchor{2b55}
@deffn {Data} ssl.HAS_TLSv1_3

Whether the OpenSSL library has built-in support for the TLS 1.3 protocol.


New in version 3.7.

@end deffn

@geindex CHANNEL_BINDING_TYPES (in module ssl)
@anchor{library/ssl ssl CHANNEL_BINDING_TYPES}@anchor{2b56}
@deffn {Data} ssl.CHANNEL_BINDING_TYPES

List of supported TLS channel binding types.  Strings in this list
can be used as arguments to @ref{cdc,,SSLSocket.get_channel_binding()}.


New in version 3.3.

@end deffn

@geindex OPENSSL_VERSION (in module ssl)
@anchor{library/ssl ssl OPENSSL_VERSION}@anchor{dc2}
@deffn {Data} ssl.OPENSSL_VERSION

The version string of the OpenSSL library loaded by the interpreter:

@example
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.2k  26 Jan 2017'
@end example


New in version 3.2.

@end deffn

@geindex OPENSSL_VERSION_INFO (in module ssl)
@anchor{library/ssl ssl OPENSSL_VERSION_INFO}@anchor{dc3}
@deffn {Data} ssl.OPENSSL_VERSION_INFO

A tuple of five integers representing version information about the
OpenSSL library:

@example
>>> ssl.OPENSSL_VERSION_INFO
(1, 0, 2, 11, 15)
@end example


New in version 3.2.

@end deffn

@geindex OPENSSL_VERSION_NUMBER (in module ssl)
@anchor{library/ssl ssl OPENSSL_VERSION_NUMBER}@anchor{dc4}
@deffn {Data} ssl.OPENSSL_VERSION_NUMBER

The raw version number of the OpenSSL library, as a single integer:

@example
>>> ssl.OPENSSL_VERSION_NUMBER
268443839
>>> hex(ssl.OPENSSL_VERSION_NUMBER)
'0x100020bf'
@end example


New in version 3.2.

@end deffn

@geindex ALERT_DESCRIPTION_HANDSHAKE_FAILURE (in module ssl)
@geindex ALERT_DESCRIPTION_INTERNAL_ERROR (in module ssl)
@anchor{library/ssl ssl ALERT_DESCRIPTION_HANDSHAKE_FAILURE}@anchor{2b57}
@deffn {Data} ssl.ALERT_DESCRIPTION_HANDSHAKE_FAILURE
@anchor{library/ssl ssl ALERT_DESCRIPTION_INTERNAL_ERROR}@anchor{2b58}
@deffnx {Data} ssl.ALERT_DESCRIPTION_INTERNAL_ERROR

@deffnx {Data} ALERT_DESCRIPTION_*

Alert Descriptions from 
@geindex RFC; RFC 5246
RFC 5246@footnote{https://tools.ietf.org/html/rfc5246.html} and others. The IANA TLS Alert Registry@footnote{https://www.iana.org/assignments/tls-parameters/tls-parameters.xml#tls-parameters-6}
contains this list and references to the RFCs where their meaning is defined.

Used as the return value of the callback function in
@ref{b38,,SSLContext.set_servername_callback()}.


New in version 3.4.

@end deffn

@geindex AlertDescription (class in ssl)
@anchor{library/ssl ssl AlertDescription}@anchor{2b59}
@deffn {Class} ssl.AlertDescription

@ref{828,,enum.IntEnum} collection of ALERT_DESCRIPTION_* constants.


New in version 3.6.

@end deffn

@geindex Purpose.SERVER_AUTH (in module ssl)
@anchor{library/ssl ssl Purpose SERVER_AUTH}@anchor{b34}
@deffn {Data} Purpose.SERVER_AUTH

Option for @ref{b29,,create_default_context()} and
@ref{b33,,SSLContext.load_default_certs()}.  This value indicates that the
context may be used to authenticate web servers (therefore, it will
be used to create client-side sockets).


New in version 3.4.

@end deffn

@geindex Purpose.CLIENT_AUTH (in module ssl)
@anchor{library/ssl ssl Purpose CLIENT_AUTH}@anchor{b35}
@deffn {Data} Purpose.CLIENT_AUTH

Option for @ref{b29,,create_default_context()} and
@ref{b33,,SSLContext.load_default_certs()}.  This value indicates that the
context may be used to authenticate web clients (therefore, it will
be used to create server-side sockets).


New in version 3.4.

@end deffn

@geindex SSLErrorNumber (class in ssl)
@anchor{library/ssl ssl SSLErrorNumber}@anchor{2b5a}
@deffn {Class} ssl.SSLErrorNumber

@ref{828,,enum.IntEnum} collection of SSL_ERROR_* constants.


New in version 3.6.

@end deffn

@geindex TLSVersion (class in ssl)
@anchor{library/ssl ssl TLSVersion}@anchor{2b5b}
@deffn {Class} ssl.TLSVersion

@ref{828,,enum.IntEnum} collection of SSL and TLS versions for
@ref{6b2,,SSLContext.maximum_version} and @ref{6b1,,SSLContext.minimum_version}.


New in version 3.7.

@end deffn

@geindex MINIMUM_SUPPORTED (ssl.TLSVersion attribute)
@anchor{library/ssl ssl TLSVersion MINIMUM_SUPPORTED}@anchor{2b5c}
@deffn {Attribute} TLSVersion.MINIMUM_SUPPORTED
@end deffn

@geindex MAXIMUM_SUPPORTED (ssl.TLSVersion attribute)
@anchor{library/ssl ssl TLSVersion MAXIMUM_SUPPORTED}@anchor{2b5d}
@deffn {Attribute} TLSVersion.MAXIMUM_SUPPORTED

The minimum or maximum supported SSL or TLS version. These are magic
constants. Their values don’t reflect the lowest and highest available
TLS/SSL versions.
@end deffn

@geindex SSLv3 (ssl.TLSVersion attribute)
@anchor{library/ssl ssl TLSVersion SSLv3}@anchor{2b5e}
@deffn {Attribute} TLSVersion.SSLv3
@end deffn

@geindex TLSv1 (ssl.TLSVersion attribute)
@anchor{library/ssl ssl TLSVersion TLSv1}@anchor{2b5f}
@deffn {Attribute} TLSVersion.TLSv1
@end deffn

@geindex TLSv1_1 (ssl.TLSVersion attribute)
@anchor{library/ssl ssl TLSVersion TLSv1_1}@anchor{2b60}
@deffn {Attribute} TLSVersion.TLSv1_1
@end deffn

@geindex TLSv1_2 (ssl.TLSVersion attribute)
@anchor{library/ssl ssl TLSVersion TLSv1_2}@anchor{2b61}
@deffn {Attribute} TLSVersion.TLSv1_2
@end deffn

@geindex TLSv1_3 (ssl.TLSVersion attribute)
@anchor{library/ssl ssl TLSVersion TLSv1_3}@anchor{2b62}
@deffn {Attribute} TLSVersion.TLSv1_3

SSL 3.0 to TLS 1.3.


Deprecated since version 3.10: All @ref{2b5b,,TLSVersion} members except @ref{2b61,,TLSVersion.TLSv1_2} and
@ref{2b62,,TLSVersion.TLSv1_3} are deprecated.

@end deffn

@node SSL Sockets,SSL Contexts,Functions Constants and Exceptions,ssl — TLS/SSL wrapper for socket objects
@anchor{library/ssl ssl-sockets}@anchor{2b63}
@subsubsection SSL Sockets


@geindex SSLSocket (class in ssl)
@anchor{library/ssl ssl SSLSocket}@anchor{3d6}
@deffn {Class} ssl.SSLSocket (socket.socket)

SSL sockets provide the following methods of @ref{2b0c,,Socket Objects}:


@itemize -

@item 
@ref{8fe,,accept()}

@item 
@ref{2b1a,,bind()}

@item 
@ref{896,,close()}

@item 
@ref{8ff,,connect()}

@item 
@ref{dbf,,detach()}

@item 
@ref{2b1e,,fileno()}

@item 
@ref{2a7a,,getpeername()}, @ref{2a7b,,getsockname()}

@item 
@ref{81d,,getsockopt()}, @ref{81e,,setsockopt()}

@item 
@ref{2b1f,,gettimeout()}, @ref{2af1,,settimeout()},
@ref{23a8,,setblocking()}

@item 
@ref{9ca,,listen()}

@item 
@ref{d44,,makefile()}

@item 
@ref{900,,recv()}, @ref{eb4,,recv_into()}
(but passing a non-zero @code{flags} argument is not allowed)

@item 
@ref{903,,send()}, @ref{904,,sendall()} (with
the same limitation)

@item 
@ref{9c9,,sendfile()} (but @ref{62c,,os.sendfile} will be used
for plain-text sockets only, else @ref{903,,send()} will be used)

@item 
@ref{2b1b,,shutdown()}
@end itemize

However, since the SSL (and TLS) protocol has its own framing atop
of TCP, the SSL sockets abstraction can, in certain respects, diverge from
the specification of normal, OS-level sockets.  See especially the
@ref{2b34,,notes on non-blocking sockets}.

Instances of @ref{3d6,,SSLSocket} must be created using the
@ref{332,,SSLContext.wrap_socket()} method.


Changed in version 3.5: The @code{sendfile()} method was added.



Changed in version 3.5: The @code{shutdown()} does not reset the socket timeout each time bytes
are received or sent. The socket timeout is now to maximum total duration
of the shutdown.



Deprecated since version 3.6: It is deprecated to create a @ref{3d6,,SSLSocket} instance directly, use
@ref{332,,SSLContext.wrap_socket()} to wrap a socket.



Changed in version 3.7: @ref{3d6,,SSLSocket} instances must to created with
@ref{332,,wrap_socket()}. In earlier versions, it was possible
to create instances directly. This was never documented or officially
supported.



Changed in version 3.10: Python now uses @code{SSL_read_ex} and @code{SSL_write_ex} internally. The
functions support reading and writing of data larger than 2 GB. Writing
zero-length data no longer fails with a protocol violation error.

@end deffn

SSL sockets also have the following additional methods and attributes:

@geindex read() (ssl.SSLSocket method)
@anchor{library/ssl ssl SSLSocket read}@anchor{9da}
@deffn {Method} SSLSocket.read (len=1024, buffer=None)

Read up to @emph{len} bytes of data from the SSL socket and return the result as
a @code{bytes} instance. If @emph{buffer} is specified, then read into the buffer
instead, and return the number of bytes read.

Raise @ref{9d5,,SSLWantReadError} or @ref{9d6,,SSLWantWriteError} if the socket is
@ref{2b34,,non-blocking} and the read would block.

As at any time a re-negotiation is possible, a call to @ref{9da,,read()} can also
cause write operations.


Changed in version 3.5: The socket timeout is no more reset each time bytes are received or sent.
The socket timeout is now to maximum total duration to read up to @emph{len}
bytes.



Deprecated since version 3.6: Use @code{recv()} instead of @ref{9da,,read()}.

@end deffn

@geindex write() (ssl.SSLSocket method)
@anchor{library/ssl ssl SSLSocket write}@anchor{9db}
@deffn {Method} SSLSocket.write (buf)

Write @emph{buf} to the SSL socket and return the number of bytes written. The
@emph{buf} argument must be an object supporting the buffer interface.

Raise @ref{9d5,,SSLWantReadError} or @ref{9d6,,SSLWantWriteError} if the socket is
@ref{2b34,,non-blocking} and the write would block.

As at any time a re-negotiation is possible, a call to @ref{9db,,write()} can
also cause read operations.


Changed in version 3.5: The socket timeout is no more reset each time bytes are received or sent.
The socket timeout is now to maximum total duration to write @emph{buf}.



Deprecated since version 3.6: Use @code{send()} instead of @ref{9db,,write()}.

@end deffn

@cartouche
@quotation Note 
The @ref{9da,,read()} and @ref{9db,,write()} methods are the
low-level methods that read and write unencrypted, application-level data
and decrypt/encrypt it to encrypted, wire-level data. These methods
require an active SSL connection, i.e. the handshake was completed and
@ref{2b64,,SSLSocket.unwrap()} was not called.

Normally you should use the socket API methods like
@ref{900,,recv()} and @ref{903,,send()} instead of these
methods.
@end quotation
@end cartouche

@geindex do_handshake() (ssl.SSLSocket method)
@anchor{library/ssl ssl SSLSocket do_handshake}@anchor{9d9}
@deffn {Method} SSLSocket.do_handshake ()

Perform the SSL setup handshake.


Changed in version 3.4: The handshake method also performs @ref{333,,match_hostname()} when the
@ref{2b42,,check_hostname} attribute of the socket’s
@ref{2b65,,context} is true.



Changed in version 3.5: The socket timeout is no more reset each time bytes are received or sent.
The socket timeout is now to maximum total duration of the handshake.



Changed in version 3.7: Hostname or IP address is matched by OpenSSL during handshake. The
function @ref{333,,match_hostname()} is no longer used. In case OpenSSL
refuses a hostname or IP address, the handshake is aborted early and
a TLS alert message is send to the peer.

@end deffn

@geindex getpeercert() (ssl.SSLSocket method)
@anchor{library/ssl ssl SSLSocket getpeercert}@anchor{b39}
@deffn {Method} SSLSocket.getpeercert (binary_form=False)

If there is no certificate for the peer on the other end of the connection,
return @code{None}.  If the SSL handshake hasn’t been done yet, raise
@ref{17c,,ValueError}.

If the @code{binary_form} parameter is @ref{659,,False}, and a certificate was
received from the peer, this method returns a @ref{3b0,,dict} instance.  If the
certificate was not validated, the dict is empty.  If the certificate was
validated, it returns a dict with several keys, amongst them @code{subject}
(the principal for which the certificate was issued) and @code{issuer}
(the principal issuing the certificate).  If a certificate contains an
instance of the @emph{Subject Alternative Name} extension (see 
@geindex RFC; RFC 3280
RFC 3280@footnote{https://tools.ietf.org/html/rfc3280.html}),
there will also be a @code{subjectAltName} key in the dictionary.

The @code{subject} and @code{issuer} fields are tuples containing the sequence
of relative distinguished names (RDNs) given in the certificate’s data
structure for the respective fields, and each RDN is a sequence of
name-value pairs.  Here is a real-world example:

@example
@{'issuer': ((('countryName', 'IL'),),
            (('organizationName', 'StartCom Ltd.'),),
            (('organizationalUnitName',
              'Secure Digital Certificate Signing'),),
            (('commonName',
              'StartCom Class 2 Primary Intermediate Server CA'),)),
 'notAfter': 'Nov 22 08:15:19 2013 GMT',
 'notBefore': 'Nov 21 03:09:52 2011 GMT',
 'serialNumber': '95F0',
 'subject': ((('description', '571208-SLe257oHY9fVQ07Z'),),
             (('countryName', 'US'),),
             (('stateOrProvinceName', 'California'),),
             (('localityName', 'San Francisco'),),
             (('organizationName', 'Electronic Frontier Foundation, Inc.'),),
             (('commonName', '*.eff.org'),),
             (('emailAddress', 'hostmaster@@eff.org'),)),
 'subjectAltName': (('DNS', '*.eff.org'), ('DNS', 'eff.org')),
 'version': 3@}
@end example

@cartouche
@quotation Note 
To validate a certificate for a particular service, you can use the
@ref{333,,match_hostname()} function.
@end quotation
@end cartouche

If the @code{binary_form} parameter is @ref{748,,True}, and a certificate was
provided, this method returns the DER-encoded form of the entire certificate
as a sequence of bytes, or @ref{182,,None} if the peer did not provide a
certificate.  Whether the peer provides a certificate depends on the SSL
socket’s role:


@itemize *

@item 
for a client SSL socket, the server will always provide a certificate,
regardless of whether validation was required;

@item 
for a server SSL socket, the client will only provide a certificate
when requested by the server; therefore @ref{b39,,getpeercert()} will return
@ref{182,,None} if you used @ref{2b40,,CERT_NONE} (rather than
@ref{2b41,,CERT_OPTIONAL} or @ref{2b30,,CERT_REQUIRED}).
@end itemize


Changed in version 3.2: The returned dictionary includes additional items such as @code{issuer}
and @code{notBefore}.



Changed in version 3.4: @ref{17c,,ValueError} is raised when the handshake isn’t done.
The returned dictionary includes additional X509v3 extension items
  such as @code{crlDistributionPoints}, @code{caIssuers} and @code{OCSP} URIs.



Changed in version 3.9: IPv6 address strings no longer have a trailing new line.

@end deffn

@geindex cipher() (ssl.SSLSocket method)
@anchor{library/ssl ssl SSLSocket cipher}@anchor{2a7c}
@deffn {Method} SSLSocket.cipher ()

Returns a three-value tuple containing the name of the cipher being used, the
version of the SSL protocol that defines its use, and the number of secret
bits being used.  If no connection has been established, returns @code{None}.
@end deffn

@geindex shared_ciphers() (ssl.SSLSocket method)
@anchor{library/ssl ssl SSLSocket shared_ciphers}@anchor{9d8}
@deffn {Method} SSLSocket.shared_ciphers ()

Return the list of ciphers shared by the client during the handshake.  Each
entry of the returned list is a three-value tuple containing the name of the
cipher, the version of the SSL protocol that defines its use, and the number
of secret bits the cipher uses.  @ref{9d8,,shared_ciphers()} returns
@code{None} if no connection has been established or the socket is a client
socket.


New in version 3.5.

@end deffn

@geindex compression() (ssl.SSLSocket method)
@anchor{library/ssl ssl SSLSocket compression}@anchor{cdd}
@deffn {Method} SSLSocket.compression ()

Return the compression algorithm being used as a string, or @code{None}
if the connection isn’t compressed.

If the higher-level protocol supports its own compression mechanism,
you can use @ref{cde,,OP_NO_COMPRESSION} to disable SSL-level compression.


New in version 3.3.

@end deffn

@geindex get_channel_binding() (ssl.SSLSocket method)
@anchor{library/ssl ssl SSLSocket get_channel_binding}@anchor{cdc}
@deffn {Method} SSLSocket.get_channel_binding (cb_type='tls-unique')

Get channel binding data for current connection, as a bytes object.  Returns
@code{None} if not connected or the handshake has not been completed.

The @emph{cb_type} parameter allow selection of the desired channel binding
type. Valid channel binding types are listed in the
@ref{2b56,,CHANNEL_BINDING_TYPES} list.  Currently only the ‘tls-unique’ channel
binding, defined by 
@geindex RFC; RFC 5929
RFC 5929@footnote{https://tools.ietf.org/html/rfc5929.html}, is supported.  @ref{17c,,ValueError} will be
raised if an unsupported channel binding type is requested.


New in version 3.3.

@end deffn

@geindex selected_alpn_protocol() (ssl.SSLSocket method)
@anchor{library/ssl ssl SSLSocket selected_alpn_protocol}@anchor{9d1}
@deffn {Method} SSLSocket.selected_alpn_protocol ()

Return the protocol that was selected during the TLS handshake.  If
@ref{9d0,,SSLContext.set_alpn_protocols()} was not called, if the other party does
not support ALPN, if this socket does not support any of the client’s
proposed protocols, or if the handshake has not happened yet, @code{None} is
returned.


New in version 3.5.

@end deffn

@geindex selected_npn_protocol() (ssl.SSLSocket method)
@anchor{library/ssl ssl SSLSocket selected_npn_protocol}@anchor{335}
@deffn {Method} SSLSocket.selected_npn_protocol ()

Return the higher-level protocol that was selected during the TLS/SSL
handshake. If @ref{336,,SSLContext.set_npn_protocols()} was not called, or
if the other party does not support NPN, or if the handshake has not yet
happened, this will return @code{None}.


New in version 3.3.



Deprecated since version 3.10: NPN has been superseded by ALPN

@end deffn

@geindex unwrap() (ssl.SSLSocket method)
@anchor{library/ssl ssl SSLSocket unwrap}@anchor{2b64}
@deffn {Method} SSLSocket.unwrap ()

Performs the SSL shutdown handshake, which removes the TLS layer from the
underlying socket, and returns the underlying socket object.  This can be
used to go from encrypted operation over a connection to unencrypted.  The
returned socket should always be used for further communication with the
other side of the connection, rather than the original socket.
@end deffn

@geindex verify_client_post_handshake() (ssl.SSLSocket method)
@anchor{library/ssl ssl SSLSocket verify_client_post_handshake}@anchor{543}
@deffn {Method} SSLSocket.verify_client_post_handshake ()

Requests post-handshake authentication (PHA) from a TLS 1.3 client. PHA
can only be initiated for a TLS 1.3 connection from a server-side socket,
after the initial TLS handshake and with PHA enabled on both sides, see
@ref{542,,SSLContext.post_handshake_auth}.

The method does not perform a cert exchange immediately. The server-side
sends a CertificateRequest during the next write event and expects the
client to respond with a certificate on the next read event.

If any precondition isn’t met (e.g. not TLS 1.3, PHA not enabled), an
@ref{e28,,SSLError} is raised.

@cartouche
@quotation Note 
Only available with OpenSSL 1.1.1 and TLS 1.3 enabled. Without TLS 1.3
support, the method raises @ref{8a3,,NotImplementedError}.
@end quotation
@end cartouche


New in version 3.8.

@end deffn

@geindex version() (ssl.SSLSocket method)
@anchor{library/ssl ssl SSLSocket version}@anchor{9d4}
@deffn {Method} SSLSocket.version ()

Return the actual SSL protocol version negotiated by the connection
as a string, or @code{None} is no secure connection is established.
As of this writing, possible return values include @code{"SSLv2"},
@code{"SSLv3"}, @code{"TLSv1"}, @code{"TLSv1.1"} and @code{"TLSv1.2"}.
Recent OpenSSL versions may define more return values.


New in version 3.5.

@end deffn

@geindex pending() (ssl.SSLSocket method)
@anchor{library/ssl ssl SSLSocket pending}@anchor{2b66}
@deffn {Method} SSLSocket.pending ()

Returns the number of already decrypted bytes available for read, pending on
the connection.
@end deffn

@geindex context (ssl.SSLSocket attribute)
@anchor{library/ssl ssl SSLSocket context}@anchor{2b65}
@deffn {Attribute} SSLSocket.context

The @ref{2bf,,SSLContext} object this SSL socket is tied to.  If the SSL
socket was created using the deprecated @ref{331,,wrap_socket()} function
(rather than @ref{332,,SSLContext.wrap_socket()}), this is a custom context
object created for this SSL socket.


New in version 3.2.

@end deffn

@geindex server_side (ssl.SSLSocket attribute)
@anchor{library/ssl ssl SSLSocket server_side}@anchor{2b67}
@deffn {Attribute} SSLSocket.server_side

A boolean which is @code{True} for server-side sockets and @code{False} for
client-side sockets.


New in version 3.2.

@end deffn

@geindex server_hostname (ssl.SSLSocket attribute)
@anchor{library/ssl ssl SSLSocket server_hostname}@anchor{6ad}
@deffn {Attribute} SSLSocket.server_hostname

Hostname of the server: @ref{2ef,,str} type, or @code{None} for server-side
socket or if the hostname was not specified in the constructor.


New in version 3.2.



Changed in version 3.7: The attribute is now always ASCII text. When @code{server_hostname} is
an internationalized domain name (IDN), this attribute now stores the
A-label form (@code{"xn--pythn-mua.org"}), rather than the U-label form
(@code{"pythön.org"}).

@end deffn

@geindex session (ssl.SSLSocket attribute)
@anchor{library/ssl ssl SSLSocket session}@anchor{2b68}
@deffn {Attribute} SSLSocket.session

The @ref{826,,SSLSession} for this SSL connection. The session is available
for client and server side sockets after the TLS handshake has been
performed. For client sockets the session can be set before
@ref{9d9,,do_handshake()} has been called to reuse a session.


New in version 3.6.

@end deffn

@geindex session_reused (ssl.SSLSocket attribute)
@anchor{library/ssl ssl SSLSocket session_reused}@anchor{2b69}
@deffn {Attribute} SSLSocket.session_reused


New in version 3.6.

@end deffn

@node SSL Contexts,Certificates,SSL Sockets,ssl — TLS/SSL wrapper for socket objects
@anchor{library/ssl ssl-contexts}@anchor{2b6a}
@subsubsection SSL Contexts



New in version 3.2.


An SSL context holds various data longer-lived than single SSL connections,
such as SSL configuration options, certificate(s) and private key(s).
It also manages a cache of SSL sessions for server-side sockets, in order
to speed up repeated connections from the same clients.

@geindex SSLContext (class in ssl)
@anchor{library/ssl ssl SSLContext}@anchor{2bf}
@deffn {Class} ssl.SSLContext (protocol=None)

Create a new SSL context.  You may pass @emph{protocol} which must be one
of the @code{PROTOCOL_*} constants defined in this module.  The parameter
specifies which version of the SSL protocol to use.  Typically, the
server chooses a particular protocol version, and the client must adapt
to the server’s choice.  Most of the versions are not interoperable
with the other versions.  If not specified, the default is
@ref{32e,,PROTOCOL_TLS}; it provides the most compatibility with other
versions.

Here’s a table showing which versions in a client (down the side) can connect
to which versions in a server (along the top):

@quotation


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxx} {xxxxxxxxxxxxxx} {xxxxxxxxxxxxxxx} {xxxxxxxxxxx} {xxxxxxxxxxxxx} {xxxxxxxxxxxxx} 
@item

@emph{client} / @strong{server}

@tab

@strong{SSLv2}

@tab

@strong{SSLv3}

@tab

@strong{TLS} @footnote{@w{(3)} 
TLS 1.3 protocol will be available with @ref{32e,,PROTOCOL_TLS} in
OpenSSL >= 1.1.1. There is no dedicated PROTOCOL constant for just
TLS 1.3.
}

@tab

@strong{TLSv1}

@tab

@strong{TLSv1.1}

@tab

@strong{TLSv1.2}

@item

@emph{SSLv2}

@tab

yes

@tab

no

@tab

no @footnote{@w{(1)} 
@ref{2bf,,SSLContext} disables SSLv2 with @ref{2bc,,OP_NO_SSLv2} by default.
}

@tab

no

@tab

no

@tab

no

@item

@emph{SSLv3}

@tab

no

@tab

yes

@tab

no @footnote{@w{(2)} 
@ref{2bf,,SSLContext} disables SSLv3 with @ref{2bd,,OP_NO_SSLv3} by default.
}

@tab

no

@tab

no

@tab

no

@item

@emph{TLS} (@emph{SSLv23}) @footnote{@w{(3)} 
TLS 1.3 protocol will be available with @ref{32e,,PROTOCOL_TLS} in
OpenSSL >= 1.1.1. There is no dedicated PROTOCOL constant for just
TLS 1.3.
}

@tab

no @footnote{@w{(1)} 
@ref{2bf,,SSLContext} disables SSLv2 with @ref{2bc,,OP_NO_SSLv2} by default.
}

@tab

no @footnote{@w{(2)} 
@ref{2bf,,SSLContext} disables SSLv3 with @ref{2bd,,OP_NO_SSLv3} by default.
}

@tab

yes

@tab

yes

@tab

yes

@tab

yes

@item

@emph{TLSv1}

@tab

no

@tab

no

@tab

yes

@tab

yes

@tab

no

@tab

no

@item

@emph{TLSv1.1}

@tab

no

@tab

no

@tab

yes

@tab

no

@tab

yes

@tab

no

@item

@emph{TLSv1.2}

@tab

no

@tab

no

@tab

yes

@tab

no

@tab

no

@tab

yes

@end multitable

@end quotation


@subsubheading See also


@ref{b29,,create_default_context()} lets the @ref{ee,,ssl} module choose
security settings for a given purpose.



Changed in version 3.6: The context is created with secure default values. The options
@ref{cde,,OP_NO_COMPRESSION}, @ref{ce1,,OP_CIPHER_SERVER_PREFERENCE},
@ref{2b48,,OP_SINGLE_DH_USE}, @ref{2b49,,OP_SINGLE_ECDH_USE},
@ref{2bc,,OP_NO_SSLv2} (except for @ref{328,,PROTOCOL_SSLv2}),
and @ref{2bd,,OP_NO_SSLv3} (except for @ref{329,,PROTOCOL_SSLv3}) are
set by default. The initial cipher suite list contains only @code{HIGH}
ciphers, no @code{NULL} ciphers and no @code{MD5} ciphers (except for
@ref{328,,PROTOCOL_SSLv2}).



Deprecated since version 3.10: @ref{2bf,,SSLContext} without protocol argument is deprecated. The
context class will either require @ref{32f,,PROTOCOL_TLS_CLIENT} or
@ref{330,,PROTOCOL_TLS_SERVER} protocol in the future.



Changed in version 3.10: The default cipher suites now include only secure AES and ChaCha20
ciphers with forward secrecy and security level 2. RSA and DH keys with
less than 2048 bits and ECC keys with less than 224 bits are prohibited.
@ref{32e,,PROTOCOL_TLS}, @ref{32f,,PROTOCOL_TLS_CLIENT}, and
@ref{330,,PROTOCOL_TLS_SERVER} use TLS 1.2 as minimum TLS version.

@end deffn

@ref{2bf,,SSLContext} objects have the following methods and attributes:

@geindex cert_store_stats() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext cert_store_stats}@anchor{b2c}
@deffn {Method} SSLContext.cert_store_stats ()

Get statistics about quantities of loaded X.509 certificates, count of
X.509 certificates flagged as CA certificates and certificate revocation
lists as dictionary.

Example for a context with one CA cert and one other cert:

@example
>>> context.cert_store_stats()
@{'crl': 0, 'x509_ca': 1, 'x509': 2@}
@end example


New in version 3.4.

@end deffn

@geindex load_cert_chain() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext load_cert_chain}@anchor{cd9}
@deffn {Method} SSLContext.load_cert_chain (certfile, keyfile=None, password=None)

Load a private key and the corresponding certificate.  The @emph{certfile}
string must be the path to a single file in PEM format containing the
certificate as well as any number of CA certificates needed to establish
the certificate’s authenticity.  The @emph{keyfile} string, if present, must
point to a file containing the private key in.  Otherwise the private
key will be taken from @emph{certfile} as well.  See the discussion of
@ref{2b6b,,Certificates} for more information on how the certificate
is stored in the @emph{certfile}.

The @emph{password} argument may be a function to call to get the password for
decrypting the private key.  It will only be called if the private key is
encrypted and a password is necessary.  It will be called with no arguments,
and it should return a string, bytes, or bytearray.  If the return value is
a string it will be encoded as UTF-8 before using it to decrypt the key.
Alternatively a string, bytes, or bytearray value may be supplied directly
as the @emph{password} argument.  It will be ignored if the private key is not
encrypted and no password is needed.

If the @emph{password} argument is not specified and a password is required,
OpenSSL’s built-in password prompting mechanism will be used to
interactively prompt the user for a password.

An @ref{e28,,SSLError} is raised if the private key doesn’t
match with the certificate.


Changed in version 3.3: New optional argument @emph{password}.

@end deffn

@geindex load_default_certs() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext load_default_certs}@anchor{b33}
@deffn {Method} SSLContext.load_default_certs (purpose=Purpose.SERVER_AUTH)

Load a set of default “certification authority” (CA) certificates from
default locations. On Windows it loads CA certs from the @code{CA} and
@code{ROOT} system stores. On all systems it calls
@ref{b2b,,SSLContext.set_default_verify_paths()}. In the future the method may
load CA certificates from other locations, too.

The @emph{purpose} flag specifies what kind of CA certificates are loaded. The
default settings @ref{b34,,Purpose.SERVER_AUTH} loads certificates, that are
flagged and trusted for TLS web server authentication (client side
sockets). @ref{b35,,Purpose.CLIENT_AUTH} loads CA certificates for client
certificate verification on the server side.


New in version 3.4.

@end deffn

@geindex load_verify_locations() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext load_verify_locations}@anchor{a5f}
@deffn {Method} SSLContext.load_verify_locations (cafile=None, capath=None, cadata=None)

Load a set of “certification authority” (CA) certificates used to validate
other peers’ certificates when @ref{2b2f,,verify_mode} is other than
@ref{2b40,,CERT_NONE}.  At least one of @emph{cafile} or @emph{capath} must be specified.

This method can also load certification revocation lists (CRLs) in PEM or
DER format. In order to make use of CRLs, @ref{b2e,,SSLContext.verify_flags}
must be configured properly.

The @emph{cafile} string, if present, is the path to a file of concatenated
CA certificates in PEM format. See the discussion of
@ref{2b6b,,Certificates} for more information about how to arrange the
certificates in this file.

The @emph{capath} string, if present, is
the path to a directory containing several CA certificates in PEM format,
following an OpenSSL specific layout@footnote{https://www.openssl.org/docs/manmaster/man3/SSL_CTX_load_verify_locations.html}.

The @emph{cadata} object, if present, is either an ASCII string of one or more
PEM-encoded certificates or a @ref{87e,,bytes-like object} of DER-encoded
certificates. Like with @emph{capath} extra lines around PEM-encoded
certificates are ignored but at least one certificate must be present.


Changed in version 3.4: New optional argument @emph{cadata}

@end deffn

@geindex get_ca_certs() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext get_ca_certs}@anchor{b2d}
@deffn {Method} SSLContext.get_ca_certs (binary_form=False)

Get a list of loaded “certification authority” (CA) certificates. If the
@code{binary_form} parameter is @ref{659,,False} each list
entry is a dict like the output of @ref{b39,,SSLSocket.getpeercert()}. Otherwise
the method returns a list of DER-encoded certificates. The returned list
does not contain certificates from @emph{capath} unless a certificate was
requested and loaded by a SSL connection.

@cartouche
@quotation Note 
Certificates in a capath directory aren’t loaded unless they have
been used at least once.
@end quotation
@end cartouche


New in version 3.4.

@end deffn

@geindex get_ciphers() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext get_ciphers}@anchor{827}
@deffn {Method} SSLContext.get_ciphers ()

Get a list of enabled ciphers. The list is in order of cipher priority.
See @ref{2b3e,,SSLContext.set_ciphers()}.

Example:

@example
>>> ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
>>> ctx.set_ciphers('ECDHE+AESGCM:!ECDSA')
>>> ctx.get_ciphers()
[@{'aead': True,
  'alg_bits': 256,
  'auth': 'auth-rsa',
  'description': 'ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  '
                 'Enc=AESGCM(256) Mac=AEAD',
  'digest': None,
  'id': 50380848,
  'kea': 'kx-ecdhe',
  'name': 'ECDHE-RSA-AES256-GCM-SHA384',
  'protocol': 'TLSv1.2',
  'strength_bits': 256,
  'symmetric': 'aes-256-gcm'@},
 @{'aead': True,
  'alg_bits': 128,
  'auth': 'auth-rsa',
  'description': 'ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  '
                 'Enc=AESGCM(128) Mac=AEAD',
  'digest': None,
  'id': 50380847,
  'kea': 'kx-ecdhe',
  'name': 'ECDHE-RSA-AES128-GCM-SHA256',
  'protocol': 'TLSv1.2',
  'strength_bits': 128,
  'symmetric': 'aes-128-gcm'@}]
@end example


New in version 3.6.

@end deffn

@geindex set_default_verify_paths() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext set_default_verify_paths}@anchor{b2b}
@deffn {Method} SSLContext.set_default_verify_paths ()

Load a set of default “certification authority” (CA) certificates from
a filesystem path defined when building the OpenSSL library.  Unfortunately,
there’s no easy way to know whether this method succeeds: no error is
returned if no certificates are to be found.  When the OpenSSL library is
provided as part of the operating system, though, it is likely to be
configured properly.
@end deffn

@geindex set_ciphers() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext set_ciphers}@anchor{2b3e}
@deffn {Method} SSLContext.set_ciphers (ciphers)

Set the available ciphers for sockets created with this context.
It should be a string in the OpenSSL cipher list format@footnote{https://www.openssl.org/docs/manmaster/man1/ciphers.html}.
If no cipher can be selected (because compile-time options or other
configuration forbids use of all the specified ciphers), an
@ref{e28,,SSLError} will be raised.

@cartouche
@quotation Note 
when connected, the @ref{2a7c,,SSLSocket.cipher()} method of SSL sockets will
give the currently selected cipher.

TLS 1.3 cipher suites cannot be disabled with
@ref{2b3e,,set_ciphers()}.
@end quotation
@end cartouche
@end deffn

@geindex set_alpn_protocols() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext set_alpn_protocols}@anchor{9d0}
@deffn {Method} SSLContext.set_alpn_protocols (protocols)

Specify which protocols the socket should advertise during the SSL/TLS
handshake. It should be a list of ASCII strings, like @code{['http/1.1',
'spdy/2']}, ordered by preference. The selection of a protocol will happen
during the handshake, and will play out according to 
@geindex RFC; RFC 7301
RFC 7301@footnote{https://tools.ietf.org/html/rfc7301.html}. After a
successful handshake, the @ref{9d1,,SSLSocket.selected_alpn_protocol()} method will
return the agreed-upon protocol.

This method will raise @ref{8a3,,NotImplementedError} if @ref{9d2,,HAS_ALPN} is
@code{False}.


New in version 3.5.

@end deffn

@geindex set_npn_protocols() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext set_npn_protocols}@anchor{336}
@deffn {Method} SSLContext.set_npn_protocols (protocols)

Specify which protocols the socket should advertise during the SSL/TLS
handshake. It should be a list of strings, like @code{['http/1.1', 'spdy/2']},
ordered by preference. The selection of a protocol will happen during the
handshake, and will play out according to the Application Layer Protocol Negotiation@footnote{https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation}. After a
successful handshake, the @ref{335,,SSLSocket.selected_npn_protocol()} method will
return the agreed-upon protocol.

This method will raise @ref{8a3,,NotImplementedError} if @ref{2b50,,HAS_NPN} is
@code{False}.


New in version 3.3.



Deprecated since version 3.10: NPN has been superseded by ALPN

@end deffn

@geindex sni_callback (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext sni_callback}@anchor{114d}
@deffn {Attribute} SSLContext.sni_callback

Register a callback function that will be called after the TLS Client Hello
handshake message has been received by the SSL/TLS server when the TLS client
specifies a server name indication. The server name indication mechanism
is specified in 
@geindex RFC; RFC 6066
RFC 6066@footnote{https://tools.ietf.org/html/rfc6066.html} section 3 - Server Name Indication.

Only one callback can be set per @code{SSLContext}.  If @emph{sni_callback}
is set to @code{None} then the callback is disabled. Calling this function a
subsequent time will disable the previously registered callback.

The callback function will be called with three
arguments; the first being the @ref{3d6,,ssl.SSLSocket}, the second is a string
that represents the server name that the client is intending to communicate
(or @ref{182,,None} if the TLS Client Hello does not contain a server name)
and the third argument is the original @ref{2bf,,SSLContext}. The server name
argument is text. For internationalized domain name, the server
name is an IDN A-label (@code{"xn--pythn-mua.org"}).

A typical use of this callback is to change the @ref{3d6,,ssl.SSLSocket}’s
@ref{2b65,,SSLSocket.context} attribute to a new object of type
@ref{2bf,,SSLContext} representing a certificate chain that matches the server
name.

Due to the early negotiation phase of the TLS connection, only limited
methods and attributes are usable like
@ref{9d1,,SSLSocket.selected_alpn_protocol()} and @ref{2b65,,SSLSocket.context}.
The @ref{b39,,SSLSocket.getpeercert()},
@ref{2a7c,,SSLSocket.cipher()} and @ref{cdd,,SSLSocket.compression()} methods require that
the TLS connection has progressed beyond the TLS Client Hello and therefore
will not return meaningful values nor can they be called safely.

The @emph{sni_callback} function must return @code{None} to allow the
TLS negotiation to continue.  If a TLS failure is required, a constant
@ref{2b58,,ALERT_DESCRIPTION_*} can be
returned.  Other return values will result in a TLS fatal error with
@ref{2b58,,ALERT_DESCRIPTION_INTERNAL_ERROR}.

If an exception is raised from the @emph{sni_callback} function the TLS
connection will terminate with a fatal TLS alert message
@ref{2b57,,ALERT_DESCRIPTION_HANDSHAKE_FAILURE}.

This method will raise @ref{8a3,,NotImplementedError} if the OpenSSL library
had OPENSSL_NO_TLSEXT defined when it was built.


New in version 3.7.

@end deffn

@geindex set_servername_callback (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext set_servername_callback}@anchor{b38}
@deffn {Attribute} SSLContext.set_servername_callback (server_name_callback)

This is a legacy API retained for backwards compatibility. When possible,
you should use @ref{114d,,sni_callback} instead. The given @emph{server_name_callback}
is similar to @emph{sni_callback}, except that when the server hostname is an
IDN-encoded internationalized domain name, the @emph{server_name_callback}
receives a decoded U-label (@code{"pythön.org"}).

If there is an decoding error on the server name, the TLS connection will
terminate with an @ref{2b58,,ALERT_DESCRIPTION_INTERNAL_ERROR} fatal TLS
alert message to the client.


New in version 3.4.

@end deffn

@geindex load_dh_params() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext load_dh_params}@anchor{cda}
@deffn {Method} SSLContext.load_dh_params (dhfile)

Load the key generation parameters for Diffie-Hellman (DH) key exchange.
Using DH key exchange improves forward secrecy at the expense of
computational resources (both on the server and on the client).
The @emph{dhfile} parameter should be the path to a file containing DH
parameters in PEM format.

This setting doesn’t apply to client sockets.  You can also use the
@ref{2b48,,OP_SINGLE_DH_USE} option to further improve security.


New in version 3.3.

@end deffn

@geindex set_ecdh_curve() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext set_ecdh_curve}@anchor{cdb}
@deffn {Method} SSLContext.set_ecdh_curve (curve_name)

Set the curve name for Elliptic Curve-based Diffie-Hellman (ECDH) key
exchange.  ECDH is significantly faster than regular DH while arguably
as secure.  The @emph{curve_name} parameter should be a string describing
a well-known elliptic curve, for example @code{prime256v1} for a widely
supported curve.

This setting doesn’t apply to client sockets.  You can also use the
@ref{2b49,,OP_SINGLE_ECDH_USE} option to further improve security.

This method is not available if @ref{2b4e,,HAS_ECDH} is @code{False}.


New in version 3.3.



@subsubheading See also



@table @asis

@item SSL/TLS & Perfect Forward Secrecy@footnote{https://vincent.bernat.im/en/blog/2011-ssl-perfect-forward-secrecy}

Vincent Bernat.
@end table

@end deffn

@geindex wrap_socket() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext wrap_socket}@anchor{332}
@deffn {Method} SSLContext.wrap_socket (sock, server_side=False, do_handshake_on_connect=True, suppress_ragged_eofs=True, server_hostname=None, session=None)

Wrap an existing Python socket @emph{sock} and return an instance of
@ref{2b6c,,SSLContext.sslsocket_class} (default @ref{3d6,,SSLSocket}). The
returned SSL socket is tied to the context, its settings and certificates.
@emph{sock} must be a @ref{e91,,SOCK_STREAM} socket; other
socket types are unsupported.

The parameter @code{server_side} is a boolean which identifies whether
server-side or client-side behavior is desired from this socket.

For client-side sockets, the context construction is lazy; if the
underlying socket isn’t connected yet, the context construction will be
performed after @code{connect()} is called on the socket.  For
server-side sockets, if the socket has no remote peer, it is assumed
to be a listening socket, and the server-side SSL wrapping is
automatically performed on client connections accepted via the
@code{accept()} method. The method may raise @ref{e28,,SSLError}.

On client connections, the optional parameter @emph{server_hostname} specifies
the hostname of the service which we are connecting to.  This allows a
single server to host multiple SSL-based services with distinct certificates,
quite similarly to HTTP virtual hosts. Specifying @emph{server_hostname} will
raise a @ref{17c,,ValueError} if @emph{server_side} is true.

The parameter @code{do_handshake_on_connect} specifies whether to do the SSL
handshake automatically after doing a @code{socket.connect()}, or whether the
application program will call it explicitly, by invoking the
@ref{9d9,,SSLSocket.do_handshake()} method.  Calling
@ref{9d9,,SSLSocket.do_handshake()} explicitly gives the program control over the
blocking behavior of the socket I/O involved in the handshake.

The parameter @code{suppress_ragged_eofs} specifies how the
@code{SSLSocket.recv()} method should signal unexpected EOF from the other end
of the connection.  If specified as @ref{748,,True} (the default), it returns a
normal EOF (an empty bytes object) in response to unexpected EOF errors
raised from the underlying socket; if @ref{659,,False}, it will raise the
exceptions back to the caller.

@emph{session}, see @ref{2b68,,session}.


Changed in version 3.5: Always allow a server_hostname to be passed, even if OpenSSL does not
have SNI.



Changed in version 3.6: @emph{session} argument was added.


Changed in version 3.7: The method returns an instance of @ref{2b6c,,SSLContext.sslsocket_class}
instead of hard-coded @ref{3d6,,SSLSocket}.


@end deffn

@geindex sslsocket_class (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext sslsocket_class}@anchor{2b6c}
@deffn {Attribute} SSLContext.sslsocket_class

The return type of @ref{332,,SSLContext.wrap_socket()}, defaults to
@ref{3d6,,SSLSocket}. The attribute can be overridden on instance of class
in order to return a custom subclass of @ref{3d6,,SSLSocket}.


New in version 3.7.

@end deffn

@geindex wrap_bio() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext wrap_bio}@anchor{6b0}
@deffn {Method} SSLContext.wrap_bio (incoming, outgoing, server_side=False, server_hostname=None, session=None)

Wrap the BIO objects @emph{incoming} and @emph{outgoing} and return an instance of
@ref{2b6d,,SSLContext.sslobject_class} (default @ref{6af,,SSLObject}). The SSL
routines will read input data from the incoming BIO and write data to the
outgoing BIO.

The @emph{server_side}, @emph{server_hostname} and @emph{session} parameters have the
same meaning as in @ref{332,,SSLContext.wrap_socket()}.


Changed in version 3.6: @emph{session} argument was added.



Changed in version 3.7: The method returns an instance of @ref{2b6d,,SSLContext.sslobject_class}
instead of hard-coded @ref{6af,,SSLObject}.

@end deffn

@geindex sslobject_class (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext sslobject_class}@anchor{2b6d}
@deffn {Attribute} SSLContext.sslobject_class

The return type of @ref{6b0,,SSLContext.wrap_bio()}, defaults to
@ref{6af,,SSLObject}. The attribute can be overridden on instance of class
in order to return a custom subclass of @ref{6af,,SSLObject}.


New in version 3.7.

@end deffn

@geindex session_stats() (ssl.SSLContext method)
@anchor{library/ssl ssl SSLContext session_stats}@anchor{2b6e}
@deffn {Method} SSLContext.session_stats ()

Get statistics about the SSL sessions created or managed by this context.
A dictionary is returned which maps the names of each piece of information@footnote{https://www.openssl.org/docs/man1.1.1/ssl/SSL_CTX_sess_number.html} to their
numeric values.  For example, here is the total number of hits and misses
in the session cache since the context was created:

@example
>>> stats = context.session_stats()
>>> stats['hits'], stats['misses']
(0, 0)
@end example
@end deffn

@geindex check_hostname (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext check_hostname}@anchor{2b42}
@deffn {Attribute} SSLContext.check_hostname

Whether to match the peer cert’s hostname in
@ref{9d9,,SSLSocket.do_handshake()}. The context’s
@ref{2b2f,,verify_mode} must be set to @ref{2b41,,CERT_OPTIONAL} or
@ref{2b30,,CERT_REQUIRED}, and you must pass @emph{server_hostname} to
@ref{332,,wrap_socket()} in order to match the hostname.  Enabling
hostname checking automatically sets @ref{2b2f,,verify_mode} from
@ref{2b40,,CERT_NONE} to @ref{2b30,,CERT_REQUIRED}.  It cannot be set back to
@ref{2b40,,CERT_NONE} as long as hostname checking is enabled. The
@ref{32f,,PROTOCOL_TLS_CLIENT} protocol enables hostname checking by default.
With other protocols, hostname checking must be enabled explicitly.

Example:

@example
import socket, ssl

context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
context.verify_mode = ssl.CERT_REQUIRED
context.check_hostname = True
context.load_default_certs()

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ssl_sock = context.wrap_socket(s, server_hostname='www.verisign.com')
ssl_sock.connect(('www.verisign.com', 443))
@end example


New in version 3.4.



Changed in version 3.7: @ref{2b2f,,verify_mode} is now automatically changed
to @ref{2b30,,CERT_REQUIRED}  when hostname checking is enabled and
@ref{2b2f,,verify_mode} is @ref{2b40,,CERT_NONE}. Previously
the same operation would have failed with a @ref{17c,,ValueError}.

@end deffn

@geindex keylog_filename (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext keylog_filename}@anchor{2b31}
@deffn {Attribute} SSLContext.keylog_filename

Write TLS keys to a keylog file, whenever key material is generated or
received. The keylog file is designed for debugging purposes only. The
file format is specified by NSS and used by many traffic analyzers such
as Wireshark. The log file is opened in append-only mode. Writes are
synchronized between threads, but not between processes.


New in version 3.8.

@end deffn

@geindex maximum_version (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext maximum_version}@anchor{6b2}
@deffn {Attribute} SSLContext.maximum_version

A @ref{2b5b,,TLSVersion} enum member representing the highest supported
TLS version. The value defaults to @ref{2b5d,,TLSVersion.MAXIMUM_SUPPORTED}.
The attribute is read-only for protocols other than @ref{32e,,PROTOCOL_TLS},
@ref{32f,,PROTOCOL_TLS_CLIENT}, and @ref{330,,PROTOCOL_TLS_SERVER}.

The attributes @ref{6b2,,maximum_version},
@ref{6b1,,minimum_version} and
@ref{2bb,,SSLContext.options} all affect the supported SSL
and TLS versions of the context. The implementation does not prevent
invalid combination. For example a context with
@ref{326,,OP_NO_TLSv1_2} in @ref{2bb,,options} and
@ref{6b2,,maximum_version} set to @ref{2b61,,TLSVersion.TLSv1_2}
will not be able to establish a TLS 1.2 connection.


New in version 3.7.

@end deffn

@geindex minimum_version (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext minimum_version}@anchor{6b1}
@deffn {Attribute} SSLContext.minimum_version

Like @ref{6b2,,SSLContext.maximum_version} except it is the lowest
supported version or @ref{2b5c,,TLSVersion.MINIMUM_SUPPORTED}.


New in version 3.7.

@end deffn

@geindex num_tickets (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext num_tickets}@anchor{2b6f}
@deffn {Attribute} SSLContext.num_tickets

Control the number of TLS 1.3 session tickets of a
@ref{330,,PROTOCOL_TLS_SERVER} context. The setting has no impact on TLS
1.0 to 1.2 connections.


New in version 3.8.

@end deffn

@geindex options (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext options}@anchor{2bb}
@deffn {Attribute} SSLContext.options

An integer representing the set of SSL options enabled on this context.
The default value is @ref{2b46,,OP_ALL}, but you can specify other options
such as @ref{2bc,,OP_NO_SSLv2} by ORing them together.


Changed in version 3.6: @ref{2bb,,SSLContext.options} returns @ref{2b4b,,Options} flags:

@example
>>> ssl.create_default_context().options  
<Options.OP_ALL|OP_NO_SSLv3|OP_NO_SSLv2|OP_NO_COMPRESSION: 2197947391>
@end example



Deprecated since version 3.7: All @code{OP_NO_SSL*} and @code{OP_NO_TLS*} options have been deprecated since
Python 3.7. Use @ref{6b1,,SSLContext.minimum_version} and
@ref{6b2,,SSLContext.maximum_version} instead.

@end deffn

@geindex post_handshake_auth (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext post_handshake_auth}@anchor{542}
@deffn {Attribute} SSLContext.post_handshake_auth

Enable TLS 1.3 post-handshake client authentication. Post-handshake auth
is disabled by default and a server can only request a TLS client
certificate during the initial handshake. When enabled, a server may
request a TLS client certificate at any time after the handshake.

When enabled on client-side sockets, the client signals the server that
it supports post-handshake authentication.

When enabled on server-side sockets, @ref{2b2f,,SSLContext.verify_mode} must
be set to @ref{2b41,,CERT_OPTIONAL} or @ref{2b30,,CERT_REQUIRED}, too. The
actual client cert exchange is delayed until
@ref{543,,SSLSocket.verify_client_post_handshake()} is called and some I/O is
performed.


New in version 3.8.

@end deffn

@geindex protocol (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext protocol}@anchor{2b70}
@deffn {Attribute} SSLContext.protocol

The protocol version chosen when constructing the context.  This attribute
is read-only.
@end deffn

@geindex hostname_checks_common_name (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext hostname_checks_common_name}@anchor{6ab}
@deffn {Attribute} SSLContext.hostname_checks_common_name

Whether @ref{2b42,,check_hostname} falls back to verify the cert’s
subject common name in the absence of a subject alternative name
extension (default: true).


New in version 3.7.



Changed in version 3.10: The flag had no effect with OpenSSL before version 1.1.1k. Python 3.8.9,
3.9.3, and 3.10 include workarounds for previous versions.

@end deffn

@geindex security_level (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext security_level}@anchor{2b71}
@deffn {Attribute} SSLContext.security_level

An integer representing the security level@footnote{https://www.openssl.org/docs/manmaster/man3/SSL_CTX_get_security_level.html}
for the context. This attribute is read-only.


New in version 3.10.

@end deffn

@geindex verify_flags (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext verify_flags}@anchor{b2e}
@deffn {Attribute} SSLContext.verify_flags

The flags for certificate verification operations. You can set flags like
@ref{b30,,VERIFY_CRL_CHECK_LEAF} by ORing them together. By default OpenSSL
does neither require nor verify certificate revocation lists (CRLs).


New in version 3.4.



Changed in version 3.6: @ref{b2e,,SSLContext.verify_flags} returns @ref{2b45,,VerifyFlags} flags:

@example
>>> ssl.create_default_context().verify_flags  
<VerifyFlags.VERIFY_X509_TRUSTED_FIRST: 32768>
@end example

@end deffn

@geindex verify_mode (ssl.SSLContext attribute)
@anchor{library/ssl ssl SSLContext verify_mode}@anchor{2b2f}
@deffn {Attribute} SSLContext.verify_mode

Whether to try to verify other peers’ certificates and how to behave
if verification fails.  This attribute must be one of
@ref{2b40,,CERT_NONE}, @ref{2b41,,CERT_OPTIONAL} or @ref{2b30,,CERT_REQUIRED}.


Changed in version 3.6: @ref{2b2f,,SSLContext.verify_mode} returns @ref{2b43,,VerifyMode} enum:

@example
>>> ssl.create_default_context().verify_mode  
<VerifyMode.CERT_REQUIRED: 2>
@end example

@end deffn

@geindex certificates

@geindex X509 certificate

@node Certificates,Examples<18>,SSL Contexts,ssl — TLS/SSL wrapper for socket objects
@anchor{library/ssl certificates}@anchor{2b72}@anchor{library/ssl ssl-certificates}@anchor{2b6b}
@subsubsection Certificates


Certificates in general are part of a public-key / private-key system.  In this
system, each @emph{principal}, (which may be a machine, or a person, or an
organization) is assigned a unique two-part encryption key.  One part of the key
is public, and is called the @emph{public key}; the other part is kept secret, and is
called the @emph{private key}.  The two parts are related, in that if you encrypt a
message with one of the parts, you can decrypt it with the other part, and
@strong{only} with the other part.

A certificate contains information about two principals.  It contains the name
of a @emph{subject}, and the subject’s public key.  It also contains a statement by a
second principal, the @emph{issuer}, that the subject is who they claim to be, and
that this is indeed the subject’s public key.  The issuer’s statement is signed
with the issuer’s private key, which only the issuer knows.  However, anyone can
verify the issuer’s statement by finding the issuer’s public key, decrypting the
statement with it, and comparing it to the other information in the certificate.
The certificate also contains information about the time period over which it is
valid.  This is expressed as two fields, called “notBefore” and “notAfter”.

In the Python use of certificates, a client or server can use a certificate to
prove who they are.  The other side of a network connection can also be required
to produce a certificate, and that certificate can be validated to the
satisfaction of the client or server that requires such validation.  The
connection attempt can be set to raise an exception if the validation fails.
Validation is done automatically, by the underlying OpenSSL framework; the
application need not concern itself with its mechanics.  But the application
does usually need to provide sets of certificates to allow this process to take
place.

Python uses files to contain certificates.  They should be formatted as “PEM”
(see 
@geindex RFC; RFC 1422
RFC 1422@footnote{https://tools.ietf.org/html/rfc1422.html}), which is a base-64 encoded form wrapped with a header line
and a footer line:

@example
-----BEGIN CERTIFICATE-----
... (certificate in base64 PEM encoding) ...
-----END CERTIFICATE-----
@end example

@menu
* Certificate chains:: 
* CA certificates:: 
* Combined key and certificate:: 
* Self-signed certificates:: 

@end menu

@node Certificate chains,CA certificates,,Certificates
@anchor{library/ssl certificate-chains}@anchor{2b73}
@subsubsection Certificate chains


The Python files which contain certificates can contain a sequence of
certificates, sometimes called a @emph{certificate chain}.  This chain should start
with the specific certificate for the principal who “is” the client or server,
and then the certificate for the issuer of that certificate, and then the
certificate for the issuer of @emph{that} certificate, and so on up the chain till
you get to a certificate which is @emph{self-signed}, that is, a certificate which
has the same subject and issuer, sometimes called a @emph{root certificate}.  The
certificates should just be concatenated together in the certificate file.  For
example, suppose we had a three certificate chain, from our server certificate
to the certificate of the certification authority that signed our server
certificate, to the root certificate of the agency which issued the
certification authority’s certificate:

@example
-----BEGIN CERTIFICATE-----
... (certificate for your server)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the certificate for the CA)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the root certificate for the CA's issuer)...
-----END CERTIFICATE-----
@end example

@node CA certificates,Combined key and certificate,Certificate chains,Certificates
@anchor{library/ssl ca-certificates}@anchor{2b74}
@subsubsection CA certificates


If you are going to require validation of the other side of the connection’s
certificate, you need to provide a “CA certs” file, filled with the certificate
chains for each issuer you are willing to trust.  Again, this file just contains
these chains concatenated together.  For validation, Python will use the first
chain it finds in the file which matches.  The platform’s certificates file can
be used by calling @ref{b33,,SSLContext.load_default_certs()}, this is done
automatically with @ref{b29,,create_default_context()}.

@node Combined key and certificate,Self-signed certificates,CA certificates,Certificates
@anchor{library/ssl combined-key-and-certificate}@anchor{2b75}
@subsubsection Combined key and certificate


Often the private key is stored in the same file as the certificate; in this
case, only the @code{certfile} parameter to @ref{cd9,,SSLContext.load_cert_chain()}
and @ref{331,,wrap_socket()} needs to be passed.  If the private key is stored
with the certificate, it should come before the first certificate in
the certificate chain:

@example
-----BEGIN RSA PRIVATE KEY-----
... (private key in base64 encoding) ...
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
... (certificate in base64 PEM encoding) ...
-----END CERTIFICATE-----
@end example

@node Self-signed certificates,,Combined key and certificate,Certificates
@anchor{library/ssl self-signed-certificates}@anchor{2b76}
@subsubsection Self-signed certificates


If you are going to create a server that provides SSL-encrypted connection
services, you will need to acquire a certificate for that service.  There are
many ways of acquiring appropriate certificates, such as buying one from a
certification authority.  Another common practice is to generate a self-signed
certificate.  The simplest way to do this is with the OpenSSL package, using
something like the following:

@example
% openssl req -new -x509 -days 365 -nodes -out cert.pem -keyout cert.pem
Generating a 1024 bit RSA private key
.......++++++
.............................++++++
writing new private key to 'cert.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:MyState
Locality Name (eg, city) []:Some City
Organization Name (eg, company) [Internet Widgits Pty Ltd]:My Organization, Inc.
Organizational Unit Name (eg, section) []:My Group
Common Name (eg, YOUR name) []:myserver.mygroup.myorganization.com
Email Address []:ops@@myserver.mygroup.myorganization.com
%
@end example

The disadvantage of a self-signed certificate is that it is its own root
certificate, and no one else will have it in their cache of known (and trusted)
root certificates.

@node Examples<18>,Notes on non-blocking sockets,Certificates,ssl — TLS/SSL wrapper for socket objects
@anchor{library/ssl examples}@anchor{2b77}
@subsubsection Examples


@menu
* Testing for SSL support:: 
* Client-side operation:: 
* Server-side operation:: 

@end menu

@node Testing for SSL support,Client-side operation,,Examples<18>
@anchor{library/ssl testing-for-ssl-support}@anchor{2b78}
@subsubsection Testing for SSL support


To test for the presence of SSL support in a Python installation, user code
should use the following idiom:

@example
try:
    import ssl
except ImportError:
    pass
else:
    ...  # do something that requires SSL support
@end example

@node Client-side operation,Server-side operation,Testing for SSL support,Examples<18>
@anchor{library/ssl client-side-operation}@anchor{2b79}
@subsubsection Client-side operation


This example creates a SSL context with the recommended security settings
for client sockets, including automatic certificate verification:

@example
>>> context = ssl.create_default_context()
@end example

If you prefer to tune security settings yourself, you might create
a context from scratch (but beware that you might not get the settings
right):

@example
>>> context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
>>> context.load_verify_locations("/etc/ssl/certs/ca-bundle.crt")
@end example

(this snippet assumes your operating system places a bundle of all CA
certificates in @code{/etc/ssl/certs/ca-bundle.crt}; if not, you’ll get an
error and have to adjust the location)

The @ref{32f,,PROTOCOL_TLS_CLIENT} protocol configures the context for cert
validation and hostname verification. @ref{2b2f,,verify_mode} is
set to @ref{2b30,,CERT_REQUIRED} and @ref{2b42,,check_hostname} is set
to @code{True}. All other protocols create SSL contexts with insecure defaults.

When you use the context to connect to a server, @ref{2b30,,CERT_REQUIRED}
and @ref{2b42,,check_hostname} validate the server certificate: it
ensures that the server certificate was signed with one of the CA
certificates, checks the signature for correctness, and verifies other
properties like validity and identity of the hostname:

@example
>>> conn = context.wrap_socket(socket.socket(socket.AF_INET),
...                            server_hostname="www.python.org")
>>> conn.connect(("www.python.org", 443))
@end example

You may then fetch the certificate:

@example
>>> cert = conn.getpeercert()
@end example

Visual inspection shows that the certificate does identify the desired service
(that is, the HTTPS host @code{www.python.org}):

@example
>>> pprint.pprint(cert)
@{'OCSP': ('http://ocsp.digicert.com',),
 'caIssuers': ('http://cacerts.digicert.com/DigiCertSHA2ExtendedValidationServerCA.crt',),
 'crlDistributionPoints': ('http://crl3.digicert.com/sha2-ev-server-g1.crl',
                           'http://crl4.digicert.com/sha2-ev-server-g1.crl'),
 'issuer': ((('countryName', 'US'),),
            (('organizationName', 'DigiCert Inc'),),
            (('organizationalUnitName', 'www.digicert.com'),),
            (('commonName', 'DigiCert SHA2 Extended Validation Server CA'),)),
 'notAfter': 'Sep  9 12:00:00 2016 GMT',
 'notBefore': 'Sep  5 00:00:00 2014 GMT',
 'serialNumber': '01BB6F00122B177F36CAB49CEA8B6B26',
 'subject': ((('businessCategory', 'Private Organization'),),
             (('1.3.6.1.4.1.311.60.2.1.3', 'US'),),
             (('1.3.6.1.4.1.311.60.2.1.2', 'Delaware'),),
             (('serialNumber', '3359300'),),
             (('streetAddress', '16 Allen Rd'),),
             (('postalCode', '03894-4801'),),
             (('countryName', 'US'),),
             (('stateOrProvinceName', 'NH'),),
             (('localityName', 'Wolfeboro'),),
             (('organizationName', 'Python Software Foundation'),),
             (('commonName', 'www.python.org'),)),
 'subjectAltName': (('DNS', 'www.python.org'),
                    ('DNS', 'python.org'),
                    ('DNS', 'pypi.org'),
                    ('DNS', 'docs.python.org'),
                    ('DNS', 'testpypi.org'),
                    ('DNS', 'bugs.python.org'),
                    ('DNS', 'wiki.python.org'),
                    ('DNS', 'hg.python.org'),
                    ('DNS', 'mail.python.org'),
                    ('DNS', 'packaging.python.org'),
                    ('DNS', 'pythonhosted.org'),
                    ('DNS', 'www.pythonhosted.org'),
                    ('DNS', 'test.pythonhosted.org'),
                    ('DNS', 'us.pycon.org'),
                    ('DNS', 'id.python.org')),
 'version': 3@}
@end example

Now the SSL channel is established and the certificate verified, you can
proceed to talk with the server:

@example
>>> conn.sendall(b"HEAD / HTTP/1.0\r\nHost: linuxfr.org\r\n\r\n")
>>> pprint.pprint(conn.recv(1024).split(b"\r\n"))
[b'HTTP/1.1 200 OK',
 b'Date: Sat, 18 Oct 2014 18:27:20 GMT',
 b'Server: nginx',
 b'Content-Type: text/html; charset=utf-8',
 b'X-Frame-Options: SAMEORIGIN',
 b'Content-Length: 45679',
 b'Accept-Ranges: bytes',
 b'Via: 1.1 varnish',
 b'Age: 2188',
 b'X-Served-By: cache-lcy1134-LCY',
 b'X-Cache: HIT',
 b'X-Cache-Hits: 11',
 b'Vary: Cookie',
 b'Strict-Transport-Security: max-age=63072000; includeSubDomains',
 b'Connection: close',
 b'@w{'},
 b'@w{'}]
@end example

See the discussion of @ref{2a39,,Security considerations} below.

@node Server-side operation,,Client-side operation,Examples<18>
@anchor{library/ssl server-side-operation}@anchor{2b7a}
@subsubsection Server-side operation


For server operation, typically you’ll need to have a server certificate, and
private key, each in a file.  You’ll first create a context holding the key
and the certificate, so that clients can check your authenticity.  Then
you’ll open a socket, bind it to a port, call @code{listen()} on it, and start
waiting for clients to connect:

@example
import socket, ssl

context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context.load_cert_chain(certfile="mycertfile", keyfile="mykeyfile")

bindsocket = socket.socket()
bindsocket.bind(('myaddr.mydomain.com', 10023))
bindsocket.listen(5)
@end example

When a client connects, you’ll call @code{accept()} on the socket to get the
new socket from the other end, and use the context’s @ref{332,,SSLContext.wrap_socket()}
method to create a server-side SSL socket for the connection:

@example
while True:
    newsocket, fromaddr = bindsocket.accept()
    connstream = context.wrap_socket(newsocket, server_side=True)
    try:
        deal_with_client(connstream)
    finally:
        connstream.shutdown(socket.SHUT_RDWR)
        connstream.close()
@end example

Then you’ll read data from the @code{connstream} and do something with it till you
are finished with the client (or the client is finished with you):

@example
def deal_with_client(connstream):
    data = connstream.recv(1024)
    # empty data means the client is finished with us
    while data:
        if not do_something(connstream, data):
            # we'll assume do_something returns False
            # when we're finished with client
            break
        data = connstream.recv(1024)
    # finished with client
@end example

And go back to listening for new client connections (of course, a real server
would probably handle each client connection in a separate thread, or put
the sockets in @ref{2b34,,non-blocking mode} and use an event loop).

@node Notes on non-blocking sockets,Memory BIO Support<2>,Examples<18>,ssl — TLS/SSL wrapper for socket objects
@anchor{library/ssl notes-on-non-blocking-sockets}@anchor{2b7b}@anchor{library/ssl ssl-nonblocking}@anchor{2b34}
@subsubsection Notes on non-blocking sockets


SSL sockets behave slightly different than regular sockets in
non-blocking mode. When working with non-blocking sockets, there are
thus several things you need to be aware of:


@itemize -

@item 
Most @ref{3d6,,SSLSocket} methods will raise either
@ref{9d6,,SSLWantWriteError} or @ref{9d5,,SSLWantReadError} instead of
@ref{bed,,BlockingIOError} if an I/O operation would
block. @ref{9d5,,SSLWantReadError} will be raised if a read operation on
the underlying socket is necessary, and @ref{9d6,,SSLWantWriteError} for
a write operation on the underlying socket. Note that attempts to
@emph{write} to an SSL socket may require @emph{reading} from the underlying
socket first, and attempts to @emph{read} from the SSL socket may require
a prior @emph{write} to the underlying socket.


Changed in version 3.5: In earlier Python versions, the @code{SSLSocket.send()} method
returned zero instead of raising @ref{9d6,,SSLWantWriteError} or
@ref{9d5,,SSLWantReadError}.


@item 
Calling @ref{8fc,,select()} tells you that the OS-level socket can be
read from (or written to), but it does not imply that there is sufficient
data at the upper SSL layer.  For example, only part of an SSL frame might
have arrived.  Therefore, you must be ready to handle @code{SSLSocket.recv()}
and @code{SSLSocket.send()} failures, and retry after another call to
@ref{8fc,,select()}.

@item 
Conversely, since the SSL layer has its own framing, a SSL socket may
still have data available for reading without @ref{8fc,,select()}
being aware of it.  Therefore, you should first call
@code{SSLSocket.recv()} to drain any potentially available data, and then
only block on a @ref{8fc,,select()} call if still necessary.

(of course, similar provisions apply when using other primitives such as
@ref{241d,,poll()}, or those in the @ref{e1,,selectors} module)

@item 
The SSL handshake itself will be non-blocking: the
@ref{9d9,,SSLSocket.do_handshake()} method has to be retried until it returns
successfully.  Here is a synopsis using @ref{8fc,,select()} to wait for
the socket’s readiness:

@example
while True:
    try:
        sock.do_handshake()
        break
    except ssl.SSLWantReadError:
        select.select([sock], [], [])
    except ssl.SSLWantWriteError:
        select.select([], [sock], [])
@end example
@end itemize


@subsubheading See also


The @ref{9,,asyncio} module supports @ref{2b34,,non-blocking SSL sockets} and provides a
higher level API. It polls for events using the @ref{e1,,selectors} module and
handles @ref{9d6,,SSLWantWriteError}, @ref{9d5,,SSLWantReadError} and
@ref{bed,,BlockingIOError} exceptions. It runs the SSL handshake asynchronously
as well.


@node Memory BIO Support<2>,SSL session,Notes on non-blocking sockets,ssl — TLS/SSL wrapper for socket objects
@anchor{library/ssl memory-bio-support}@anchor{2b7c}
@subsubsection Memory BIO Support



New in version 3.5.


Ever since the SSL module was introduced in Python 2.6, the @ref{3d6,,SSLSocket}
class has provided two related but distinct areas of functionality:


@itemize -

@item 
SSL protocol handling

@item 
Network IO
@end itemize

The network IO API is identical to that provided by @ref{8fd,,socket.socket},
from which @ref{3d6,,SSLSocket} also inherits. This allows an SSL socket to be
used as a drop-in replacement for a regular socket, making it very easy to add
SSL support to an existing application.

Combining SSL protocol handling and network IO usually works well, but there
are some cases where it doesn’t. An example is async IO frameworks that want to
use a different IO multiplexing model than the “select/poll on a file
descriptor” (readiness based) model that is assumed by @ref{8fd,,socket.socket}
and by the internal OpenSSL socket IO routines. This is mostly relevant for
platforms like Windows where this model is not efficient. For this purpose, a
reduced scope variant of @ref{3d6,,SSLSocket} called @ref{6af,,SSLObject} is
provided.

@geindex SSLObject (class in ssl)
@anchor{library/ssl ssl SSLObject}@anchor{6af}
@deffn {Class} ssl.SSLObject

A reduced-scope variant of @ref{3d6,,SSLSocket} representing an SSL protocol
instance that does not contain any network IO methods. This class is
typically used by framework authors that want to implement asynchronous IO
for SSL through memory buffers.

This class implements an interface on top of a low-level SSL object as
implemented by OpenSSL. This object captures the state of an SSL connection
but does not provide any network IO itself. IO needs to be performed through
separate “BIO” objects which are OpenSSL’s IO abstraction layer.

This class has no public constructor.  An @ref{6af,,SSLObject} instance
must be created using the @ref{6b0,,wrap_bio()} method. This
method will create the @ref{6af,,SSLObject} instance and bind it to a
pair of BIOs. The @emph{incoming} BIO is used to pass data from Python to the
SSL protocol instance, while the @emph{outgoing} BIO is used to pass data the
other way around.

The following methods are available:


@itemize -

@item 
@ref{2b65,,context}

@item 
@ref{2b67,,server_side}

@item 
@ref{6ad,,server_hostname}

@item 
@ref{2b68,,session}

@item 
@ref{2b69,,session_reused}

@item 
@ref{9da,,read()}

@item 
@ref{9db,,write()}

@item 
@ref{b39,,getpeercert()}

@item 
@ref{9d1,,selected_alpn_protocol()}

@item 
@ref{335,,selected_npn_protocol()}

@item 
@ref{2a7c,,cipher()}

@item 
@ref{9d8,,shared_ciphers()}

@item 
@ref{cdd,,compression()}

@item 
@ref{2b66,,pending()}

@item 
@ref{9d9,,do_handshake()}

@item 
@ref{543,,verify_client_post_handshake()}

@item 
@ref{2b64,,unwrap()}

@item 
@ref{cdc,,get_channel_binding()}

@item 
@ref{9d4,,version()}
@end itemize

When compared to @ref{3d6,,SSLSocket}, this object lacks the following
features:


@itemize -

@item 
Any form of network IO; @code{recv()} and @code{send()} read and write only to
the underlying @ref{9ce,,MemoryBIO} buffers.

@item 
There is no @emph{do_handshake_on_connect} machinery. You must always manually
call @ref{9d9,,do_handshake()} to start the handshake.

@item 
There is no handling of @emph{suppress_ragged_eofs}. All end-of-file conditions
that are in violation of the protocol are reported via the
@ref{1121,,SSLEOFError} exception.

@item 
The method @ref{2b64,,unwrap()} call does not return anything,
unlike for an SSL socket where it returns the underlying socket.

@item 
The @emph{server_name_callback} callback passed to
@ref{b38,,SSLContext.set_servername_callback()} will get an @ref{6af,,SSLObject}
instance instead of a @ref{3d6,,SSLSocket} instance as its first parameter.
@end itemize

Some notes related to the use of @ref{6af,,SSLObject}:


@itemize -

@item 
All IO on an @ref{6af,,SSLObject} is @ref{2b34,,non-blocking}.
This means that for example @ref{9da,,read()} will raise an
@ref{9d5,,SSLWantReadError} if it needs more data than the incoming BIO has
available.

@item 
There is no module-level @code{wrap_bio()} call like there is for
@ref{332,,wrap_socket()}. An @ref{6af,,SSLObject} is always created
via an @ref{2bf,,SSLContext}.
@end itemize


Changed in version 3.7: @ref{6af,,SSLObject} instances must to created with
@ref{6b0,,wrap_bio()}. In earlier versions, it was possible to
create instances directly. This was never documented or officially
supported.

@end deffn

An SSLObject communicates with the outside world using memory buffers. The
class @ref{9ce,,MemoryBIO} provides a memory buffer that can be used for this
purpose.  It wraps an OpenSSL memory BIO (Basic IO) object:

@geindex MemoryBIO (class in ssl)
@anchor{library/ssl ssl MemoryBIO}@anchor{9ce}
@deffn {Class} ssl.MemoryBIO

A memory buffer that can be used to pass data between Python and an SSL
protocol instance.

@geindex pending (ssl.MemoryBIO attribute)
@anchor{library/ssl ssl MemoryBIO pending}@anchor{2b7d}
@deffn {Attribute} pending

Return the number of bytes currently in the memory buffer.
@end deffn

@geindex eof (ssl.MemoryBIO attribute)
@anchor{library/ssl ssl MemoryBIO eof}@anchor{2b7e}
@deffn {Attribute} eof

A boolean indicating whether the memory BIO is current at the end-of-file
position.
@end deffn

@geindex read() (ssl.MemoryBIO method)
@anchor{library/ssl ssl MemoryBIO read}@anchor{2b7f}
@deffn {Method} read (n=- 1)

Read up to @emph{n} bytes from the memory buffer. If @emph{n} is not specified or
negative, all bytes are returned.
@end deffn

@geindex write() (ssl.MemoryBIO method)
@anchor{library/ssl ssl MemoryBIO write}@anchor{2b80}
@deffn {Method} write (buf)

Write the bytes from @emph{buf} to the memory BIO. The @emph{buf} argument must be an
object supporting the buffer protocol.

The return value is the number of bytes written, which is always equal to
the length of @emph{buf}.
@end deffn

@geindex write_eof() (ssl.MemoryBIO method)
@anchor{library/ssl ssl MemoryBIO write_eof}@anchor{2b81}
@deffn {Method} write_eof ()

Write an EOF marker to the memory BIO. After this method has been called, it
is illegal to call @ref{2b80,,write()}. The attribute @ref{2b7e,,eof} will
become true after all data currently in the buffer has been read.
@end deffn
@end deffn

@node SSL session,Security considerations<2>,Memory BIO Support<2>,ssl — TLS/SSL wrapper for socket objects
@anchor{library/ssl ssl-session}@anchor{2b82}
@subsubsection SSL session



New in version 3.6.


@geindex SSLSession (class in ssl)
@anchor{library/ssl ssl SSLSession}@anchor{826}
@deffn {Class} ssl.SSLSession

Session object used by @ref{2b68,,session}.

@geindex id (ssl.SSLSession attribute)
@anchor{library/ssl ssl SSLSession id}@anchor{2b83}
@deffn {Attribute} id
@end deffn

@geindex time (ssl.SSLSession attribute)
@anchor{library/ssl ssl SSLSession time}@anchor{2b84}
@deffn {Attribute} time
@end deffn

@geindex timeout (ssl.SSLSession attribute)
@anchor{library/ssl ssl SSLSession timeout}@anchor{2b85}
@deffn {Attribute} timeout
@end deffn

@geindex ticket_lifetime_hint (ssl.SSLSession attribute)
@anchor{library/ssl ssl SSLSession ticket_lifetime_hint}@anchor{2b86}
@deffn {Attribute} ticket_lifetime_hint
@end deffn

@geindex has_ticket (ssl.SSLSession attribute)
@anchor{library/ssl ssl SSLSession has_ticket}@anchor{2b87}
@deffn {Attribute} has_ticket
@end deffn
@end deffn

@node Security considerations<2>,TLS 1 3,SSL session,ssl — TLS/SSL wrapper for socket objects
@anchor{library/ssl security-considerations}@anchor{2b88}@anchor{library/ssl ssl-security}@anchor{2a39}
@subsubsection Security considerations


@menu
* Best defaults:: 
* Manual settings:: 
* Multi-processing:: 

@end menu

@node Best defaults,Manual settings,,Security considerations<2>
@anchor{library/ssl best-defaults}@anchor{2b89}
@subsubsection Best defaults


For @strong{client use}, if you don’t have any special requirements for your
security policy, it is highly recommended that you use the
@ref{b29,,create_default_context()} function to create your SSL context.
It will load the system’s trusted CA certificates, enable certificate
validation and hostname checking, and try to choose reasonably secure
protocol and cipher settings.

For example, here is how you would use the @ref{421,,smtplib.SMTP} class to
create a trusted, secure connection to a SMTP server:

@example
>>> import ssl, smtplib
>>> smtp = smtplib.SMTP("mail.python.org", port=587)
>>> context = ssl.create_default_context()
>>> smtp.starttls(context=context)
(220, b'2.0.0 Ready to start TLS')
@end example

If a client certificate is needed for the connection, it can be added with
@ref{cd9,,SSLContext.load_cert_chain()}.

By contrast, if you create the SSL context by calling the @ref{2bf,,SSLContext}
constructor yourself, it will not have certificate validation nor hostname
checking enabled by default.  If you do so, please read the paragraphs below
to achieve a good security level.

@node Manual settings,Multi-processing,Best defaults,Security considerations<2>
@anchor{library/ssl manual-settings}@anchor{2b8a}
@subsubsection Manual settings


@menu
* Verifying certificates:: 
* Protocol versions:: 
* Cipher selection:: 

@end menu

@node Verifying certificates,Protocol versions,,Manual settings
@anchor{library/ssl verifying-certificates}@anchor{2b8b}
@subsubsection Verifying certificates


When calling the @ref{2bf,,SSLContext} constructor directly,
@ref{2b40,,CERT_NONE} is the default.  Since it does not authenticate the other
peer, it can be insecure, especially in client mode where most of time you
would like to ensure the authenticity of the server you’re talking to.
Therefore, when in client mode, it is highly recommended to use
@ref{2b30,,CERT_REQUIRED}.  However, it is in itself not sufficient; you also
have to check that the server certificate, which can be obtained by calling
@ref{b39,,SSLSocket.getpeercert()}, matches the desired service.  For many
protocols and applications, the service can be identified by the hostname;
in this case, the @ref{333,,match_hostname()} function can be used.  This common
check is automatically performed when @ref{2b42,,SSLContext.check_hostname} is
enabled.


Changed in version 3.7: Hostname matchings is now performed by OpenSSL. Python no longer uses
@ref{333,,match_hostname()}.


In server mode, if you want to authenticate your clients using the SSL layer
(rather than using a higher-level authentication mechanism), you’ll also have
to specify @ref{2b30,,CERT_REQUIRED} and similarly check the client certificate.

@node Protocol versions,Cipher selection,Verifying certificates,Manual settings
@anchor{library/ssl protocol-versions}@anchor{2b8c}
@subsubsection Protocol versions


SSL versions 2 and 3 are considered insecure and are therefore dangerous to
use.  If you want maximum compatibility between clients and servers, it is
recommended to use @ref{32f,,PROTOCOL_TLS_CLIENT} or
@ref{330,,PROTOCOL_TLS_SERVER} as the protocol version. SSLv2 and SSLv3 are
disabled by default.

@example
>>> client_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
>>> client_context.minimum_version = ssl.TLSVersion.TLSv1_3
>>> client_context.maximum_version = ssl.TLSVersion.TLSv1_3
@end example

The SSL context created above will only allow TLSv1.2 and later (if
supported by your system) connections to a server. @ref{32f,,PROTOCOL_TLS_CLIENT}
implies certificate validation and hostname checks by default. You have to
load certificates into the context.

@node Cipher selection,,Protocol versions,Manual settings
@anchor{library/ssl cipher-selection}@anchor{2b8d}
@subsubsection Cipher selection


If you have advanced security requirements, fine-tuning of the ciphers
enabled when negotiating a SSL session is possible through the
@ref{2b3e,,SSLContext.set_ciphers()} method.  Starting from Python 3.2.3, the
ssl module disables certain weak ciphers by default, but you may want
to further restrict the cipher choice. Be sure to read OpenSSL’s documentation
about the cipher list format@footnote{https://www.openssl.org/docs/manmaster/man1/ciphers.html#CIPHER-LIST-FORMAT}.
If you want to check which ciphers are enabled by a given cipher list, use
@ref{827,,SSLContext.get_ciphers()} or the @code{openssl ciphers} command on your
system.

@node Multi-processing,,Manual settings,Security considerations<2>
@anchor{library/ssl multi-processing}@anchor{2b8e}
@subsubsection Multi-processing


If using this module as part of a multi-processed application (using,
for example the @ref{b3,,multiprocessing} or @ref{20,,concurrent.futures} modules),
be aware that OpenSSL’s internal random number generator does not properly
handle forked processes.  Applications must change the PRNG state of the
parent process if they use any SSL feature with @ref{bb9,,os.fork()}.  Any
successful call of @ref{2b3a,,RAND_add()}, @ref{cd8,,RAND_bytes()} or
@ref{334,,RAND_pseudo_bytes()} is sufficient.

@node TLS 1 3,,Security considerations<2>,ssl — TLS/SSL wrapper for socket objects
@anchor{library/ssl ssl-tlsv1-3}@anchor{6ae}@anchor{library/ssl tls-1-3}@anchor{2b8f}
@subsubsection TLS 1.3



New in version 3.7.


The TLS 1.3 protocol behaves slightly differently than previous version
of TLS/SSL. Some new TLS 1.3 features are not yet available.


@itemize -

@item 
TLS 1.3 uses a disjunct set of cipher suites. All AES-GCM and
ChaCha20 cipher suites are enabled by default.  The method
@ref{2b3e,,SSLContext.set_ciphers()} cannot enable or disable any TLS 1.3
ciphers yet, but @ref{827,,SSLContext.get_ciphers()} returns them.

@item 
Session tickets are no longer sent as part of the initial handshake and
are handled differently.  @ref{2b68,,SSLSocket.session} and @ref{826,,SSLSession}
are not compatible with TLS 1.3.

@item 
Client-side certificates are also no longer verified during the initial
handshake.  A server can request a certificate at any time.  Clients
process certificate requests while they send or receive application data
from the server.

@item 
TLS 1.3 features like early data, deferred TLS client cert request,
signature algorithm configuration, and rekeying are not supported yet.
@end itemize


@subsubheading See also



@table @asis

@item Class @ref{8fd,,socket.socket}

Documentation of underlying @ref{ea,,socket} class

@item SSL/TLS Strong Encryption: An Introduction@footnote{https://httpd.apache.org/docs/trunk/en/ssl/ssl_intro.html}

Intro from the Apache HTTP Server documentation

@geindex RFC; RFC 1422

@item RFC 1422: Privacy Enhancement for Internet Electronic Mail: Part II: Certificate-Based Key Management@footnote{https://tools.ietf.org/html/rfc1422.html}

Steve Kent

@geindex RFC; RFC 4086

@item RFC 4086: Randomness Requirements for Security@footnote{https://tools.ietf.org/html/rfc4086.html}

Donald E., Jeffrey I. Schiller

@geindex RFC; RFC 5280

@item RFC 5280: Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile@footnote{https://tools.ietf.org/html/rfc5280.html}

D. Cooper

@geindex RFC; RFC 5246

@item RFC 5246: The Transport Layer Security (TLS) Protocol Version 1.2@footnote{https://tools.ietf.org/html/rfc5246.html}

T. Dierks et. al.

@geindex RFC; RFC 6066

@item RFC 6066: Transport Layer Security (TLS) Extensions@footnote{https://tools.ietf.org/html/rfc6066.html}

D. Eastlake

@item IANA TLS: Transport Layer Security (TLS) Parameters@footnote{https://www.iana.org/assignments/tls-parameters/tls-parameters.xml}

IANA

@geindex RFC; RFC 7525

@item RFC 7525: Recommendations for Secure Use of Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS)@footnote{https://tools.ietf.org/html/rfc7525.html}

IETF

@item Mozilla's Server Side TLS recommendations@footnote{https://wiki.mozilla.org/Security/Server_Side_TLS}

Mozilla
@end table


@node select — Waiting for I/O completion,selectors — High-level I/O multiplexing,ssl — TLS/SSL wrapper for socket objects,Networking and Interprocess Communication
@anchor{library/select doc}@anchor{2b90}@anchor{library/select module-select}@anchor{e0}@anchor{library/select select-waiting-for-i-o-completion}@anchor{2b91}
@subsection @code{select} — Waiting for I/O completion


@geindex module; select


__________________________________________________________________


This module provides access to the @code{select()} and @code{poll()} functions
available in most operating systems, @code{devpoll()} available on
Solaris and derivatives, @code{epoll()} available on Linux 2.5+ and
@code{kqueue()} available on most BSD.
Note that on Windows, it only works for sockets; on other operating systems,
it also works for other file types (in particular, on Unix, it works on pipes).
It cannot be used on regular files to determine whether a file has grown since
it was last read.

@cartouche
@quotation Note 
The @ref{e1,,selectors} module allows high-level and efficient I/O
multiplexing, built upon the @ref{e0,,select} module primitives. Users are
encouraged to use the @ref{e1,,selectors} module instead, unless they want
precise control over the OS-level primitives used.
@end quotation
@end cartouche

The module defines the following:

@geindex error
@anchor{library/select select error}@anchor{bec}
@deffn {Exception} select.error

A deprecated alias of @ref{2a1,,OSError}.


Changed in version 3.3: Following 
@geindex Python Enhancement Proposals; PEP 3151
PEP 3151@footnote{https://www.python.org/dev/peps/pep-3151}, this class was made an alias of @ref{2a1,,OSError}.

@end deffn

@geindex devpoll() (in module select)
@anchor{library/select select devpoll}@anchor{b16}
@deffn {Function} select.devpoll ()

(Only supported on Solaris and derivatives.)  Returns a @code{/dev/poll}
polling object; see section @ref{2b92,,/dev/poll Polling Objects} below for the
methods supported by devpoll objects.

@code{devpoll()} objects are linked to the number of file
descriptors allowed at the time of instantiation. If your program
reduces this value, @code{devpoll()} will fail. If your program
increases this value, @code{devpoll()} may return an
incomplete list of active file descriptors.

The new file descriptor is @ref{a6e,,non-inheritable}.


New in version 3.3.



Changed in version 3.4: The new file descriptor is now non-inheritable.

@end deffn

@geindex epoll() (in module select)
@anchor{library/select select epoll}@anchor{b14}
@deffn {Function} select.epoll (sizehint=- 1, flags=0)

(Only supported on Linux 2.5.44 and newer.) Return an edge polling object,
which can be used as Edge or Level Triggered interface for I/O
events.

@emph{sizehint} informs epoll about the expected number of events to be
registered.  It must be positive, or @cite{-1} to use the default. It is only
used on older systems where @code{epoll_create1()} is not available;
otherwise it has no effect (though its value is still checked).

@emph{flags} is deprecated and completely ignored.  However, when supplied, its
value must be @code{0} or @code{select.EPOLL_CLOEXEC}, otherwise @code{OSError} is
raised.

See the @ref{2b93,,Edge and Level Trigger Polling (epoll) Objects} section below for the methods supported by
epolling objects.

@code{epoll} objects support the context management protocol: when used in a
@ref{168,,with} statement, the new file descriptor is automatically closed
at the end of the block.

The new file descriptor is @ref{a6e,,non-inheritable}.


Changed in version 3.3: Added the @emph{flags} parameter.



Changed in version 3.4: Support for the @ref{168,,with} statement was added.
The new file descriptor is now non-inheritable.



Deprecated since version 3.4: The @emph{flags} parameter.  @code{select.EPOLL_CLOEXEC} is used by default now.
Use @ref{a70,,os.set_inheritable()} to make the file descriptor inheritable.

@end deffn

@geindex poll() (in module select)
@anchor{library/select select poll}@anchor{241d}
@deffn {Function} select.poll ()

(Not supported by all operating systems.)  Returns a polling object, which
supports registering and unregistering file descriptors, and then polling them
for I/O events; see section @ref{2b94,,Polling Objects} below for the methods supported
by polling objects.
@end deffn

@geindex kqueue() (in module select)
@anchor{library/select select kqueue}@anchor{2b95}
@deffn {Function} select.kqueue ()

(Only supported on BSD.)  Returns a kernel queue object; see section
@ref{2b96,,Kqueue Objects} below for the methods supported by kqueue objects.

The new file descriptor is @ref{a6e,,non-inheritable}.


Changed in version 3.4: The new file descriptor is now non-inheritable.

@end deffn

@geindex kevent() (in module select)
@anchor{library/select select kevent}@anchor{2b97}
@deffn {Function} select.kevent (ident, filter=KQ_FILTER_READ, flags=KQ_EV_ADD, fflags=0, data=0, udata=0)

(Only supported on BSD.)  Returns a kernel event object; see section
@ref{2b98,,Kevent Objects} below for the methods supported by kevent objects.
@end deffn

@geindex select() (in module select)
@anchor{library/select select select}@anchor{8fc}
@deffn {Function} select.select (rlist, wlist, xlist[, timeout])

This is a straightforward interface to the Unix @code{select()} system call.
The first three arguments are iterables of ‘waitable objects’: either
integers representing file descriptors or objects with a parameterless method
named @ref{23a2,,fileno()} returning such an integer:


@itemize *

@item 
@emph{rlist}: wait until ready for reading

@item 
@emph{wlist}: wait until ready for writing

@item 
@emph{xlist}: wait for an “exceptional condition” (see the manual page for what
your system considers such a condition)
@end itemize

Empty iterables are allowed, but acceptance of three empty iterables is
platform-dependent. (It is known to work on Unix but not on Windows.)  The
optional @emph{timeout} argument specifies a time-out as a floating point number
in seconds.  When the @emph{timeout} argument is omitted the function blocks until
at least one file descriptor is ready.  A time-out value of zero specifies a
poll and never blocks.

The return value is a triple of lists of objects that are ready: subsets of the
first three arguments.  When the time-out is reached without a file descriptor
becoming ready, three empty lists are returned.

@geindex socket() (in module socket)
@geindex popen() (in module os)

Among the acceptable object types in the iterables are Python @ref{d68,,file objects} (e.g. @code{sys.stdin}, or objects returned by
@ref{1c8,,open()} or @ref{59f,,os.popen()}), socket objects returned by
@ref{8fd,,socket.socket()}.  You may also define a @emph{wrapper} class yourself,
as long as it has an appropriate @ref{23a2,,fileno()} method (that
really returns a file descriptor, not just a random integer).

@cartouche
@quotation Note 
@geindex WinSock

File objects on Windows are not acceptable, but sockets are.  On Windows,
the underlying @code{select()} function is provided by the WinSock
library, and does not handle file descriptors that don’t originate from
WinSock.
@end quotation
@end cartouche


Changed in version 3.5: The function is now retried with a recomputed timeout when interrupted by
a signal, except if the signal handler raises an exception (see
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale), instead of raising
@ref{8e3,,InterruptedError}.

@end deffn

@geindex PIPE_BUF (in module select)
@anchor{library/select select PIPE_BUF}@anchor{dab}
@deffn {Attribute} select.PIPE_BUF

The minimum number of bytes which can be written without blocking to a pipe
when the pipe has been reported as ready for writing by @ref{8fc,,select()},
@ref{241d,,poll()} or another interface in this module.  This doesn’t apply
to other kind of file-like objects such as sockets.

This value is guaranteed by POSIX to be at least 512.

@ref{1629,,Availability}: Unix


New in version 3.2.

@end deffn

@menu
* /dev/poll Polling Objects:: 
* Edge and Level Trigger Polling (epoll) Objects: Edge and Level Trigger Polling epoll Objects. 
* Polling Objects:: 
* Kqueue Objects:: 
* Kevent Objects:: 

@end menu

@node /dev/poll Polling Objects,Edge and Level Trigger Polling epoll Objects,,select — Waiting for I/O completion
@anchor{library/select dev-poll-polling-objects}@anchor{2b99}@anchor{library/select devpoll-objects}@anchor{2b92}
@subsubsection @code{/dev/poll} Polling Objects


Solaris and derivatives have @code{/dev/poll}. While @code{select()} is
O(highest file descriptor) and @code{poll()} is O(number of file
descriptors), @code{/dev/poll} is O(active file descriptors).

@code{/dev/poll} behaviour is very close to the standard @code{poll()}
object.

@geindex close() (select.devpoll method)
@anchor{library/select select devpoll close}@anchor{b18}
@deffn {Method} devpoll.close ()

Close the file descriptor of the polling object.


New in version 3.4.

@end deffn

@geindex closed (select.devpoll attribute)
@anchor{library/select select devpoll closed}@anchor{b19}
@deffn {Attribute} devpoll.closed

@code{True} if the polling object is closed.


New in version 3.4.

@end deffn

@geindex fileno() (select.devpoll method)
@anchor{library/select select devpoll fileno}@anchor{b17}
@deffn {Method} devpoll.fileno ()

Return the file descriptor number of the polling object.


New in version 3.4.

@end deffn

@geindex register() (select.devpoll method)
@anchor{library/select select devpoll register}@anchor{2b9a}
@deffn {Method} devpoll.register (fd[, eventmask])

Register a file descriptor with the polling object.  Future calls to the
@ref{241d,,poll()} method will then check whether the file descriptor has any
pending I/O events.  @emph{fd} can be either an integer, or an object with a
@ref{23a2,,fileno()} method that returns an integer.  File objects
implement @code{fileno()}, so they can also be used as the argument.

@emph{eventmask} is an optional bitmask describing the type of events you want to
check for. The constants are the same that with @code{poll()}
object. The default value is a combination of the constants @code{POLLIN},
@code{POLLPRI}, and @code{POLLOUT}.

@cartouche
@quotation Warning 
Registering a file descriptor that’s already registered is not an
error, but the result is undefined. The appropriate action is to
unregister or modify it first. This is an important difference
compared with @code{poll()}.
@end quotation
@end cartouche
@end deffn

@geindex modify() (select.devpoll method)
@anchor{library/select select devpoll modify}@anchor{2b9b}
@deffn {Method} devpoll.modify (fd[, eventmask])

This method does an @ref{2b9c,,unregister()} followed by a
@ref{2b9a,,register()}. It is (a bit) more efficient that doing the same
explicitly.
@end deffn

@geindex unregister() (select.devpoll method)
@anchor{library/select select devpoll unregister}@anchor{2b9c}
@deffn {Method} devpoll.unregister (fd)

Remove a file descriptor being tracked by a polling object.  Just like the
@ref{2b9a,,register()} method, @emph{fd} can be an integer or an object with a
@ref{23a2,,fileno()} method that returns an integer.

Attempting to remove a file descriptor that was never registered is
safely ignored.
@end deffn

@geindex poll() (select.devpoll method)
@anchor{library/select select devpoll poll}@anchor{8f8}
@deffn {Method} devpoll.poll ([timeout])

Polls the set of registered file descriptors, and returns a possibly-empty list
containing @code{(fd, event)} 2-tuples for the descriptors that have events or
errors to report. @emph{fd} is the file descriptor, and @emph{event} is a bitmask with
bits set for the reported events for that descriptor — @code{POLLIN} for
waiting input, @code{POLLOUT} to indicate that the descriptor can be written
to, and so forth. An empty list indicates that the call timed out and no file
descriptors had any events to report. If @emph{timeout} is given, it specifies the
length of time in milliseconds which the system will wait for events before
returning. If @emph{timeout} is omitted, -1, or @ref{182,,None}, the call will
block until there is an event for this poll object.


Changed in version 3.5: The function is now retried with a recomputed timeout when interrupted by
a signal, except if the signal handler raises an exception (see
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale), instead of raising
@ref{8e3,,InterruptedError}.

@end deffn

@node Edge and Level Trigger Polling epoll Objects,Polling Objects,/dev/poll Polling Objects,select — Waiting for I/O completion
@anchor{library/select edge-and-level-trigger-polling-epoll-objects}@anchor{2b9d}@anchor{library/select epoll-objects}@anchor{2b93}
@subsubsection Edge and Level Trigger Polling (epoll) Objects


@quotation

@indicateurl{https://linux.die.net/man/4/epoll}

@emph{eventmask}


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Meaning

@item

@code{EPOLLIN}

@tab

Available for read

@item

@code{EPOLLOUT}

@tab

Available for write

@item

@code{EPOLLPRI}

@tab

Urgent data for read

@item

@code{EPOLLERR}

@tab

Error condition happened on the assoc. fd

@item

@code{EPOLLHUP}

@tab

Hang up happened on the assoc. fd

@item

@code{EPOLLET}

@tab

Set Edge Trigger behavior, the default is
Level Trigger behavior

@item

@code{EPOLLONESHOT}

@tab

Set one-shot behavior. After one event is
pulled out, the fd is internally disabled

@item

@code{EPOLLEXCLUSIVE}

@tab

Wake only one epoll object when the
associated fd has an event. The default (if
this flag is not set) is to wake all epoll
objects polling on a fd.

@item

@code{EPOLLRDHUP}

@tab

Stream socket peer closed connection or shut
down writing half of connection.

@item

@code{EPOLLRDNORM}

@tab

Equivalent to @code{EPOLLIN}

@item

@code{EPOLLRDBAND}

@tab

Priority data band can be read.

@item

@code{EPOLLWRNORM}

@tab

Equivalent to @code{EPOLLOUT}

@item

@code{EPOLLWRBAND}

@tab

Priority data may be written.

@item

@code{EPOLLMSG}

@tab

Ignored.

@end multitable



New in version 3.6: @code{EPOLLEXCLUSIVE} was added.  It’s only supported by Linux Kernel 4.5
or later.

@end quotation

@geindex close() (select.epoll method)
@anchor{library/select select epoll close}@anchor{b15}
@deffn {Method} epoll.close ()

Close the control file descriptor of the epoll object.
@end deffn

@geindex closed (select.epoll attribute)
@anchor{library/select select epoll closed}@anchor{2b9e}
@deffn {Attribute} epoll.closed

@code{True} if the epoll object is closed.
@end deffn

@geindex fileno() (select.epoll method)
@anchor{library/select select epoll fileno}@anchor{2b9f}
@deffn {Method} epoll.fileno ()

Return the file descriptor number of the control fd.
@end deffn

@geindex fromfd() (select.epoll method)
@anchor{library/select select epoll fromfd}@anchor{2ba0}
@deffn {Method} epoll.fromfd (fd)

Create an epoll object from a given file descriptor.
@end deffn

@geindex register() (select.epoll method)
@anchor{library/select select epoll register}@anchor{2ba1}
@deffn {Method} epoll.register (fd[, eventmask])

Register a fd descriptor with the epoll object.
@end deffn

@geindex modify() (select.epoll method)
@anchor{library/select select epoll modify}@anchor{2ba2}
@deffn {Method} epoll.modify (fd, eventmask)

Modify a registered file descriptor.
@end deffn

@geindex unregister() (select.epoll method)
@anchor{library/select select epoll unregister}@anchor{45d}
@deffn {Method} epoll.unregister (fd)

Remove a registered file descriptor from the epoll object.


Changed in version 3.9: The method no longer ignores the @ref{45e,,EBADF} error.

@end deffn

@geindex poll() (select.epoll method)
@anchor{library/select select epoll poll}@anchor{8f9}
@deffn {Method} epoll.poll (timeout=None, maxevents=- 1)

Wait for events. timeout in seconds (float)


Changed in version 3.5: The function is now retried with a recomputed timeout when interrupted by
a signal, except if the signal handler raises an exception (see
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale), instead of raising
@ref{8e3,,InterruptedError}.

@end deffn

@node Polling Objects,Kqueue Objects,Edge and Level Trigger Polling epoll Objects,select — Waiting for I/O completion
@anchor{library/select poll-objects}@anchor{2b94}@anchor{library/select polling-objects}@anchor{2ba3}
@subsubsection Polling Objects


The @code{poll()} system call, supported on most Unix systems, provides better
scalability for network servers that service many, many clients at the same
time. @code{poll()} scales better because the system call only requires listing
the file descriptors of interest, while @code{select()} builds a bitmap, turns
on bits for the fds of interest, and then afterward the whole bitmap has to be
linearly scanned again. @code{select()} is O(highest file descriptor), while
@code{poll()} is O(number of file descriptors).

@geindex register() (select.poll method)
@anchor{library/select select poll register}@anchor{2ba4}
@deffn {Method} poll.register (fd[, eventmask])

Register a file descriptor with the polling object.  Future calls to the
@ref{241d,,poll()} method will then check whether the file descriptor has any
pending I/O events.  @emph{fd} can be either an integer, or an object with a
@ref{23a2,,fileno()} method that returns an integer.  File objects
implement @code{fileno()}, so they can also be used as the argument.

@emph{eventmask} is an optional bitmask describing the type of events you want to
check for, and can be a combination of the constants @code{POLLIN},
@code{POLLPRI}, and @code{POLLOUT}, described in the table below.  If not
specified, the default value used will check for all 3 types of events.


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Meaning

@item

@code{POLLIN}

@tab

There is data to read

@item

@code{POLLPRI}

@tab

There is urgent data to read

@item

@code{POLLOUT}

@tab

Ready for output: writing will not block

@item

@code{POLLERR}

@tab

Error condition of some sort

@item

@code{POLLHUP}

@tab

Hung up

@item

@code{POLLRDHUP}

@tab

Stream socket peer closed connection, or
shut down writing half of connection

@item

@code{POLLNVAL}

@tab

Invalid request: descriptor not open

@end multitable


Registering a file descriptor that’s already registered is not an error, and has
the same effect as registering the descriptor exactly once.
@end deffn

@geindex modify() (select.poll method)
@anchor{library/select select poll modify}@anchor{2ba5}
@deffn {Method} poll.modify (fd, eventmask)

Modifies an already registered fd. This has the same effect as
@code{register(fd, eventmask)}.  Attempting to modify a file descriptor
that was never registered causes an @ref{2a1,,OSError} exception with errno
@code{ENOENT} to be raised.
@end deffn

@geindex unregister() (select.poll method)
@anchor{library/select select poll unregister}@anchor{2ba6}
@deffn {Method} poll.unregister (fd)

Remove a file descriptor being tracked by a polling object.  Just like the
@ref{2ba4,,register()} method, @emph{fd} can be an integer or an object with a
@ref{23a2,,fileno()} method that returns an integer.

Attempting to remove a file descriptor that was never registered causes a
@ref{5bb,,KeyError} exception to be raised.
@end deffn

@geindex poll() (select.poll method)
@anchor{library/select select poll poll}@anchor{8fb}
@deffn {Method} poll.poll ([timeout])

Polls the set of registered file descriptors, and returns a possibly-empty list
containing @code{(fd, event)} 2-tuples for the descriptors that have events or
errors to report. @emph{fd} is the file descriptor, and @emph{event} is a bitmask with
bits set for the reported events for that descriptor — @code{POLLIN} for
waiting input, @code{POLLOUT} to indicate that the descriptor can be written
to, and so forth. An empty list indicates that the call timed out and no file
descriptors had any events to report. If @emph{timeout} is given, it specifies the
length of time in milliseconds which the system will wait for events before
returning. If @emph{timeout} is omitted, negative, or @ref{182,,None}, the call will
block until there is an event for this poll object.


Changed in version 3.5: The function is now retried with a recomputed timeout when interrupted by
a signal, except if the signal handler raises an exception (see
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale), instead of raising
@ref{8e3,,InterruptedError}.

@end deffn

@node Kqueue Objects,Kevent Objects,Polling Objects,select — Waiting for I/O completion
@anchor{library/select id1}@anchor{2ba7}@anchor{library/select kqueue-objects}@anchor{2b96}
@subsubsection Kqueue Objects


@geindex close() (select.kqueue method)
@anchor{library/select select kqueue close}@anchor{2ba8}
@deffn {Method} kqueue.close ()

Close the control file descriptor of the kqueue object.
@end deffn

@geindex closed (select.kqueue attribute)
@anchor{library/select select kqueue closed}@anchor{2ba9}
@deffn {Attribute} kqueue.closed

@code{True} if the kqueue object is closed.
@end deffn

@geindex fileno() (select.kqueue method)
@anchor{library/select select kqueue fileno}@anchor{2baa}
@deffn {Method} kqueue.fileno ()

Return the file descriptor number of the control fd.
@end deffn

@geindex fromfd() (select.kqueue method)
@anchor{library/select select kqueue fromfd}@anchor{2bab}
@deffn {Method} kqueue.fromfd (fd)

Create a kqueue object from a given file descriptor.
@end deffn

@geindex control() (select.kqueue method)
@anchor{library/select select kqueue control}@anchor{8fa}
@deffn {Method} kqueue.control (changelist, max_events[, timeout]) -> eventlist

Low level interface to kevent


@itemize -

@item 
changelist must be an iterable of kevent objects or @code{None}

@item 
max_events must be 0 or a positive integer

@item 
timeout in seconds (floats possible); the default is @code{None},
to wait forever
@end itemize


Changed in version 3.5: The function is now retried with a recomputed timeout when interrupted by
a signal, except if the signal handler raises an exception (see
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale), instead of raising
@ref{8e3,,InterruptedError}.

@end deffn

@node Kevent Objects,,Kqueue Objects,select — Waiting for I/O completion
@anchor{library/select id2}@anchor{2bac}@anchor{library/select kevent-objects}@anchor{2b98}
@subsubsection Kevent Objects


@indicateurl{https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2}

@geindex ident (select.kevent attribute)
@anchor{library/select select kevent ident}@anchor{2bad}
@deffn {Attribute} kevent.ident

Value used to identify the event. The interpretation depends on the filter
but it’s usually the file descriptor. In the constructor ident can either
be an int or an object with a @ref{23a2,,fileno()} method. kevent
stores the integer internally.
@end deffn

@geindex filter (select.kevent attribute)
@anchor{library/select select kevent filter}@anchor{2bae}
@deffn {Attribute} kevent.filter

Name of the kernel filter.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Meaning

@item

@code{KQ_FILTER_READ}

@tab

Takes a descriptor and returns whenever
there is data available to read

@item

@code{KQ_FILTER_WRITE}

@tab

Takes a descriptor and returns whenever
there is data available to write

@item

@code{KQ_FILTER_AIO}

@tab

AIO requests

@item

@code{KQ_FILTER_VNODE}

@tab

Returns when one or more of the requested
events watched in @emph{fflag} occurs

@item

@code{KQ_FILTER_PROC}

@tab

Watch for events on a process id

@item

@code{KQ_FILTER_NETDEV}

@tab

Watch for events on a network device
[not available on macOS]

@item

@code{KQ_FILTER_SIGNAL}

@tab

Returns whenever the watched signal is
delivered to the process

@item

@code{KQ_FILTER_TIMER}

@tab

Establishes an arbitrary timer

@end multitable

@end deffn

@geindex flags (select.kevent attribute)
@anchor{library/select select kevent flags}@anchor{2baf}
@deffn {Attribute} kevent.flags

Filter action.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Meaning

@item

@code{KQ_EV_ADD}

@tab

Adds or modifies an event

@item

@code{KQ_EV_DELETE}

@tab

Removes an event from the queue

@item

@code{KQ_EV_ENABLE}

@tab

Permitscontrol() to returns the event

@item

@code{KQ_EV_DISABLE}

@tab

Disablesevent

@item

@code{KQ_EV_ONESHOT}

@tab

Removes event after first occurrence

@item

@code{KQ_EV_CLEAR}

@tab

Reset the state after an event is retrieved

@item

@code{KQ_EV_SYSFLAGS}

@tab

internal event

@item

@code{KQ_EV_FLAG1}

@tab

internal event

@item

@code{KQ_EV_EOF}

@tab

Filter specific EOF condition

@item

@code{KQ_EV_ERROR}

@tab

See return values

@end multitable

@end deffn

@geindex fflags (select.kevent attribute)
@anchor{library/select select kevent fflags}@anchor{2bb0}
@deffn {Attribute} kevent.fflags

Filter specific flags.

@code{KQ_FILTER_READ} and  @code{KQ_FILTER_WRITE} filter flags:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Meaning

@item

@code{KQ_NOTE_LOWAT}

@tab

low water mark of a socket buffer

@end multitable


@code{KQ_FILTER_VNODE} filter flags:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Meaning

@item

@code{KQ_NOTE_DELETE}

@tab

@emph{unlink()} was called

@item

@code{KQ_NOTE_WRITE}

@tab

a write occurred

@item

@code{KQ_NOTE_EXTEND}

@tab

the file was extended

@item

@code{KQ_NOTE_ATTRIB}

@tab

an attribute was changed

@item

@code{KQ_NOTE_LINK}

@tab

the link count has changed

@item

@code{KQ_NOTE_RENAME}

@tab

the file was renamed

@item

@code{KQ_NOTE_REVOKE}

@tab

access to the file was revoked

@end multitable


@code{KQ_FILTER_PROC} filter flags:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Meaning

@item

@code{KQ_NOTE_EXIT}

@tab

the process has exited

@item

@code{KQ_NOTE_FORK}

@tab

the process has called @emph{fork()}

@item

@code{KQ_NOTE_EXEC}

@tab

the process has executed a new process

@item

@code{KQ_NOTE_PCTRLMASK}

@tab

internal filter flag

@item

@code{KQ_NOTE_PDATAMASK}

@tab

internal filter flag

@item

@code{KQ_NOTE_TRACK}

@tab

follow a process across @emph{fork()}

@item

@code{KQ_NOTE_CHILD}

@tab

returned on the child process for
@emph{NOTE_TRACK}

@item

@code{KQ_NOTE_TRACKERR}

@tab

unable to attach to a child

@end multitable


@code{KQ_FILTER_NETDEV} filter flags (not available on macOS):


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Meaning

@item

@code{KQ_NOTE_LINKUP}

@tab

link is up

@item

@code{KQ_NOTE_LINKDOWN}

@tab

link is down

@item

@code{KQ_NOTE_LINKINV}

@tab

link state is invalid

@end multitable

@end deffn

@geindex data (select.kevent attribute)
@anchor{library/select select kevent data}@anchor{2bb1}
@deffn {Attribute} kevent.data

Filter specific data.
@end deffn

@geindex udata (select.kevent attribute)
@anchor{library/select select kevent udata}@anchor{2bb2}
@deffn {Attribute} kevent.udata

User defined value.
@end deffn

@node selectors — High-level I/O multiplexing,asyncore — Asynchronous socket handler,select — Waiting for I/O completion,Networking and Interprocess Communication
@anchor{library/selectors doc}@anchor{2bb3}@anchor{library/selectors module-selectors}@anchor{e1}@anchor{library/selectors selectors-high-level-i-o-multiplexing}@anchor{2bb4}
@subsection @code{selectors} — High-level I/O multiplexing


@geindex module; selectors


New in version 3.4.


@strong{Source code:} Lib/selectors.py@footnote{https://github.com/python/cpython/tree/main/Lib/selectors.py}


__________________________________________________________________


@menu
* Introduction: Introduction<9>. 
* Classes: Classes<3>. 
* Examples: Examples<19>. 

@end menu

@node Introduction<9>,Classes<3>,,selectors — High-level I/O multiplexing
@anchor{library/selectors introduction}@anchor{2bb5}
@subsubsection Introduction


This module allows high-level and efficient I/O multiplexing, built upon the
@ref{e0,,select} module primitives. Users are encouraged to use this module
instead, unless they want precise control over the OS-level primitives used.

It defines a @ref{2bb6,,BaseSelector} abstract base class, along with several
concrete implementations (@ref{2ace,,KqueueSelector}, @ref{70d,,EpollSelector}…),
that can be used to wait for I/O readiness notification on multiple file
objects. In the following, “file object” refers to any object with a
@code{fileno()} method, or a raw file descriptor. See @ref{d68,,file object}.

@ref{2bb7,,DefaultSelector} is an alias to the most efficient implementation
available on the current platform: this should be the default choice for most
users.

@cartouche
@quotation Note 
The type of file objects supported depends on the platform: on Windows,
sockets are supported, but not pipes, whereas on Unix, both are supported
(some other types may be supported as well, such as fifos or special file
devices).
@end quotation
@end cartouche


@subsubheading See also



@table @asis

@item @ref{e0,,select}

Low-level I/O multiplexing module.
@end table


@node Classes<3>,Examples<19>,Introduction<9>,selectors — High-level I/O multiplexing
@anchor{library/selectors classes}@anchor{2bb8}
@subsubsection Classes


Classes hierarchy:

@example
BaseSelector
+-- SelectSelector
+-- PollSelector
+-- EpollSelector
+-- DevpollSelector
+-- KqueueSelector
@end example

In the following, @emph{events} is a bitwise mask indicating which I/O events should
be waited for on a given file object. It can be a combination of the modules
constants below:

@quotation


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Meaning

@item

@code{EVENT_READ}

@tab

Available for read

@item

@code{EVENT_WRITE}

@tab

Available for write

@end multitable

@end quotation

@geindex SelectorKey (class in selectors)
@anchor{library/selectors selectors SelectorKey}@anchor{2bb9}
@deffn {Class} selectors.SelectorKey

A @ref{2bb9,,SelectorKey} is a @ref{3e4,,namedtuple} used to
associate a file object to its underlying file descriptor, selected event
mask and attached data. It is returned by several @ref{2bb6,,BaseSelector}
methods.

@geindex fileobj (selectors.SelectorKey attribute)
@anchor{library/selectors selectors SelectorKey fileobj}@anchor{2bba}
@deffn {Attribute} fileobj

File object registered.
@end deffn

@geindex fd (selectors.SelectorKey attribute)
@anchor{library/selectors selectors SelectorKey fd}@anchor{2bbb}
@deffn {Attribute} fd

Underlying file descriptor.
@end deffn

@geindex events (selectors.SelectorKey attribute)
@anchor{library/selectors selectors SelectorKey events}@anchor{2bbc}
@deffn {Attribute} events

Events that must be waited for on this file object.
@end deffn

@geindex data (selectors.SelectorKey attribute)
@anchor{library/selectors selectors SelectorKey data}@anchor{2bbd}
@deffn {Attribute} data

Optional opaque data associated to this file object: for example, this
could be used to store a per-client session ID.
@end deffn
@end deffn

@geindex BaseSelector (class in selectors)
@anchor{library/selectors selectors BaseSelector}@anchor{2bb6}
@deffn {Class} selectors.BaseSelector

A @ref{2bb6,,BaseSelector} is used to wait for I/O event readiness on multiple
file objects. It supports file stream registration, unregistration, and a
method to wait for I/O events on those streams, with an optional timeout.
It’s an abstract base class, so cannot be instantiated. Use
@ref{2bb7,,DefaultSelector} instead, or one of @ref{2acb,,SelectSelector},
@ref{2ace,,KqueueSelector} etc. if you want to specifically use an
implementation, and your platform supports it.
@ref{2bb6,,BaseSelector} and its concrete implementations support the
@ref{166,,context manager} protocol.

@geindex register() (selectors.BaseSelector method)
@anchor{library/selectors selectors BaseSelector register}@anchor{2bbe}
@deffn {Method} abstractmethod register (fileobj, events, data=None)

Register a file object for selection, monitoring it for I/O events.

@emph{fileobj} is the file object to monitor.  It may either be an integer
file descriptor or an object with a @code{fileno()} method.
@emph{events} is a bitwise mask of events to monitor.
@emph{data} is an opaque object.

This returns a new @ref{2bb9,,SelectorKey} instance, or raises a
@ref{17c,,ValueError} in case of invalid event mask or file descriptor, or
@ref{5bb,,KeyError} if the file object is already registered.
@end deffn

@geindex unregister() (selectors.BaseSelector method)
@anchor{library/selectors selectors BaseSelector unregister}@anchor{2bbf}
@deffn {Method} abstractmethod unregister (fileobj)

Unregister a file object from selection, removing it from monitoring. A
file object shall be unregistered prior to being closed.

@emph{fileobj} must be a file object previously registered.

This returns the associated @ref{2bb9,,SelectorKey} instance, or raises a
@ref{5bb,,KeyError} if @emph{fileobj} is not registered.  It will raise
@ref{17c,,ValueError} if @emph{fileobj} is invalid (e.g. it has no @code{fileno()}
method or its @code{fileno()} method has an invalid return value).
@end deffn

@geindex modify() (selectors.BaseSelector method)
@anchor{library/selectors selectors BaseSelector modify}@anchor{70c}
@deffn {Method} modify (fileobj, events, data=None)

Change a registered file object’s monitored events or attached data.

This is equivalent to @code{BaseSelector.unregister(fileobj)()} followed
by @code{BaseSelector.register(fileobj, events, data)()}, except that it
can be implemented more efficiently.

This returns a new @ref{2bb9,,SelectorKey} instance, or raises a
@ref{17c,,ValueError} in case of invalid event mask or file descriptor, or
@ref{5bb,,KeyError} if the file object is not registered.
@end deffn

@geindex select() (selectors.BaseSelector method)
@anchor{library/selectors selectors BaseSelector select}@anchor{2bc0}
@deffn {Method} abstractmethod select (timeout=None)

Wait until some registered file objects become ready, or the timeout
expires.

If @code{timeout > 0}, this specifies the maximum wait time, in seconds.
If @code{timeout <= 0}, the call won’t block, and will report the currently
ready file objects.
If @emph{timeout} is @code{None}, the call will block until a monitored file object
becomes ready.

This returns a list of @code{(key, events)} tuples, one for each ready file
object.

@emph{key} is the @ref{2bb9,,SelectorKey} instance corresponding to a ready file
object.
@emph{events} is a bitmask of events ready on this file object.

@cartouche
@quotation Note 
This method can return before any file object becomes ready or the
timeout has elapsed if the current process receives a signal: in this
case, an empty list will be returned.
@end quotation
@end cartouche


Changed in version 3.5: The selector is now retried with a recomputed timeout when interrupted
by a signal if the signal handler did not raise an exception (see
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale), instead of returning an empty list
of events before the timeout.

@end deffn

@geindex close() (selectors.BaseSelector method)
@anchor{library/selectors selectors BaseSelector close}@anchor{2bc1}
@deffn {Method} close ()

Close the selector.

This must be called to make sure that any underlying resource is freed.
The selector shall not be used once it has been closed.
@end deffn

@geindex get_key() (selectors.BaseSelector method)
@anchor{library/selectors selectors BaseSelector get_key}@anchor{2bc2}
@deffn {Method} get_key (fileobj)

Return the key associated with a registered file object.

This returns the @ref{2bb9,,SelectorKey} instance associated to this file
object, or raises @ref{5bb,,KeyError} if the file object is not registered.
@end deffn

@geindex get_map() (selectors.BaseSelector method)
@anchor{library/selectors selectors BaseSelector get_map}@anchor{2bc3}
@deffn {Method} abstractmethod get_map ()

Return a mapping of file objects to selector keys.

This returns a @ref{3ab,,Mapping} instance mapping
registered file objects to their associated @ref{2bb9,,SelectorKey}
instance.
@end deffn
@end deffn

@geindex DefaultSelector (class in selectors)
@anchor{library/selectors selectors DefaultSelector}@anchor{2bb7}
@deffn {Class} selectors.DefaultSelector

The default selector class, using the most efficient implementation
available on the current platform. This should be the default choice for
most users.
@end deffn

@geindex SelectSelector (class in selectors)
@anchor{library/selectors selectors SelectSelector}@anchor{2acb}
@deffn {Class} selectors.SelectSelector

@ref{8fc,,select.select()}-based selector.
@end deffn

@geindex PollSelector (class in selectors)
@anchor{library/selectors selectors PollSelector}@anchor{70e}
@deffn {Class} selectors.PollSelector

@ref{241d,,select.poll()}-based selector.
@end deffn

@geindex EpollSelector (class in selectors)
@anchor{library/selectors selectors EpollSelector}@anchor{70d}
@deffn {Class} selectors.EpollSelector

@ref{b14,,select.epoll()}-based selector.

@geindex fileno() (selectors.EpollSelector method)
@anchor{library/selectors selectors EpollSelector fileno}@anchor{2bc4}
@deffn {Method} fileno ()

This returns the file descriptor used by the underlying
@ref{b14,,select.epoll()} object.
@end deffn
@end deffn

@geindex DevpollSelector (class in selectors)
@anchor{library/selectors selectors DevpollSelector}@anchor{70f}
@deffn {Class} selectors.DevpollSelector

@ref{b16,,select.devpoll()}-based selector.

@geindex fileno() (selectors.DevpollSelector method)
@anchor{library/selectors selectors DevpollSelector fileno}@anchor{2bc5}
@deffn {Method} fileno ()

This returns the file descriptor used by the underlying
@ref{b16,,select.devpoll()} object.
@end deffn


New in version 3.5.

@end deffn

@geindex KqueueSelector (class in selectors)
@anchor{library/selectors selectors KqueueSelector}@anchor{2ace}
@deffn {Class} selectors.KqueueSelector

@ref{2b95,,select.kqueue()}-based selector.

@geindex fileno() (selectors.KqueueSelector method)
@anchor{library/selectors selectors KqueueSelector fileno}@anchor{2bc6}
@deffn {Method} fileno ()

This returns the file descriptor used by the underlying
@ref{2b95,,select.kqueue()} object.
@end deffn
@end deffn

@node Examples<19>,,Classes<3>,selectors — High-level I/O multiplexing
@anchor{library/selectors examples}@anchor{2bc7}
@subsubsection Examples


Here is a simple echo server implementation:

@example
import selectors
import socket

sel = selectors.DefaultSelector()

def accept(sock, mask):
    conn, addr = sock.accept()  # Should be ready
    print('accepted', conn, 'from', addr)
    conn.setblocking(False)
    sel.register(conn, selectors.EVENT_READ, read)

def read(conn, mask):
    data = conn.recv(1000)  # Should be ready
    if data:
        print('echoing', repr(data), 'to', conn)
        conn.send(data)  # Hope it won't block
    else:
        print('closing', conn)
        sel.unregister(conn)
        conn.close()

sock = socket.socket()
sock.bind(('localhost', 1234))
sock.listen(100)
sock.setblocking(False)
sel.register(sock, selectors.EVENT_READ, accept)

while True:
    events = sel.select()
    for key, mask in events:
        callback = key.data
        callback(key.fileobj, mask)
@end example

@node asyncore — Asynchronous socket handler,asynchat — Asynchronous socket command/response handler,selectors — High-level I/O multiplexing,Networking and Interprocess Communication
@anchor{library/asyncore doc}@anchor{2bc8}@anchor{library/asyncore asyncore-asynchronous-socket-handler}@anchor{2bc9}@anchor{library/asyncore module-asyncore}@anchor{a}
@subsection @code{asyncore} — Asynchronous socket handler


@geindex module; asyncore

@c heavily adapted from original documentation by Sam Rushing

@strong{Source code:} Lib/asyncore.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncore.py}


Deprecated since version 3.6: Please use @ref{9,,asyncio} instead.



__________________________________________________________________


@cartouche
@quotation Note 
This module exists for backwards compatibility only.  For new code we
recommend using @ref{9,,asyncio}.
@end quotation
@end cartouche

This module provides the basic infrastructure for writing asynchronous  socket
service clients and servers.

There are only two ways to have a program on a single processor do  “more than
one thing at a time.” Multi-threaded programming is the  simplest and most
popular way to do it, but there is another very different technique, that lets
you have nearly all the advantages of  multi-threading, without actually using
multiple threads.  It’s really  only practical if your program is largely I/O
bound.  If your program is processor bound, then pre-emptive scheduled threads
are probably what you really need.  Network servers are rarely processor
bound, however.

If your operating system supports the @code{select()} system call in its I/O
library (and nearly all do), then you can use it to juggle multiple
communication channels at once; doing other work while your I/O is taking
place in the “background.”  Although this strategy can seem strange and
complex, especially at first, it is in many ways easier to understand and
control than multi-threaded programming.  The @ref{a,,asyncore} module solves
many of the difficult problems for you, making the task of building
sophisticated high-performance network servers and clients a snap.  For
“conversational” applications and protocols the companion @ref{8,,asynchat}
module is invaluable.

The basic idea behind both modules is to create one or more network
@emph{channels}, instances of class @ref{de0,,asyncore.dispatcher} and
@ref{b20,,asynchat.async_chat}.  Creating the channels adds them to a global
map, used by the @ref{2bca,,loop()} function if you do not provide it with your own
@emph{map}.

Once the initial channel(s) is(are) created, calling the @ref{2bca,,loop()} function
activates channel service, which continues until the last channel (including
any that have been added to the map during asynchronous service) is closed.

@geindex loop() (in module asyncore)
@anchor{library/asyncore asyncore loop}@anchor{2bca}
@deffn {Function} asyncore.loop ([timeout[, use_poll[, map[, count]]]])

Enter a polling loop that terminates after count passes or all open
channels have been closed.  All arguments are optional.  The @emph{count}
parameter defaults to @code{None}, resulting in the loop terminating only when all
channels have been closed.  The @emph{timeout} argument sets the timeout
parameter for the appropriate @ref{8fc,,select()} or @ref{241d,,poll()}
call, measured in seconds; the default is 30 seconds.  The @emph{use_poll}
parameter, if true, indicates that @ref{241d,,poll()} should be used in
preference to @ref{8fc,,select()} (the default is @code{False}).

The @emph{map} parameter is a dictionary whose items are the channels to watch.
As channels are closed they are deleted from their map.  If @emph{map} is
omitted, a global map is used. Channels (instances of
@ref{de0,,asyncore.dispatcher}, @ref{b20,,asynchat.async_chat} and subclasses
thereof) can freely be mixed in the map.
@end deffn

@geindex dispatcher (class in asyncore)
@anchor{library/asyncore asyncore dispatcher}@anchor{de0}
@deffn {Class} asyncore.dispatcher

The @ref{de0,,dispatcher} class is a thin wrapper around a low-level socket
object. To make it more useful, it has a few methods for event-handling
which are called from the asynchronous loop.   Otherwise, it can be treated
as a normal non-blocking socket object.

The firing of low-level events at certain times or in certain connection
states tells the asynchronous loop that certain higher-level events have
taken place.  For example, if we have asked for a socket to connect to
another host, we know that the connection has been made when the socket
becomes writable for the first time (at this point you know that you may
write to it with the expectation of success).  The implied higher-level
events are:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Event

@tab

Description

@item

@code{handle_connect()}

@tab

Implied by the first read or write
event

@item

@code{handle_close()}

@tab

Implied by a read event with no data
available

@item

@code{handle_accepted()}

@tab

Implied by a read event on a listening
socket

@end multitable


During asynchronous processing, each mapped channel’s @ref{2bcb,,readable()} and
@ref{2bcc,,writable()} methods are used to determine whether the channel’s socket
should be added to the list of channels @code{select()}ed or
@code{poll()}ed for read and write events.

Thus, the set of channel events is larger than the basic socket events.  The
full set of methods that can be overridden in your subclass follows:

@geindex handle_read() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher handle_read}@anchor{2bcd}
@deffn {Method} handle_read ()

Called when the asynchronous loop detects that a @code{read()} call on the
channel’s socket will succeed.
@end deffn

@geindex handle_write() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher handle_write}@anchor{2bce}
@deffn {Method} handle_write ()

Called when the asynchronous loop detects that a writable socket can be
written.  Often this method will implement the necessary buffering for
performance.  For example:

@example
def handle_write(self):
    sent = self.send(self.buffer)
    self.buffer = self.buffer[sent:]
@end example
@end deffn

@geindex handle_expt() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher handle_expt}@anchor{2bcf}
@deffn {Method} handle_expt ()

Called when there is out of band (OOB) data for a socket connection.  This
will almost never happen, as OOB is tenuously supported and rarely used.
@end deffn

@geindex handle_connect() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher handle_connect}@anchor{2bd0}
@deffn {Method} handle_connect ()

Called when the active opener’s socket actually makes a connection.  Might
send a “welcome” banner, or initiate a protocol negotiation with the
remote endpoint, for example.
@end deffn

@geindex handle_close() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher handle_close}@anchor{2bd1}
@deffn {Method} handle_close ()

Called when the socket is closed.
@end deffn

@geindex handle_error() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher handle_error}@anchor{2bd2}
@deffn {Method} handle_error ()

Called when an exception is raised and not otherwise handled.  The default
version prints a condensed traceback.
@end deffn

@geindex handle_accept() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher handle_accept}@anchor{de2}
@deffn {Method} handle_accept ()

Called on listening channels (passive openers) when a connection can be
established with a new remote endpoint that has issued a @ref{2bd3,,connect()}
call for the local endpoint. Deprecated in version 3.2; use
@ref{de1,,handle_accepted()} instead.


Deprecated since version 3.2.

@end deffn

@geindex handle_accepted() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher handle_accepted}@anchor{de1}
@deffn {Method} handle_accepted (sock, addr)

Called on listening channels (passive openers) when a connection has been
established with a new remote endpoint that has issued a @ref{2bd3,,connect()}
call for the local endpoint.  @emph{sock} is a @emph{new} socket object usable to
send and receive data on the connection, and @emph{addr} is the address
bound to the socket on the other end of the connection.


New in version 3.2.

@end deffn

@geindex readable() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher readable}@anchor{2bcb}
@deffn {Method} readable ()

Called each time around the asynchronous loop to determine whether a
channel’s socket should be added to the list on which read events can
occur.  The default method simply returns @code{True}, indicating that by
default, all channels will be interested in read events.
@end deffn

@geindex writable() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher writable}@anchor{2bcc}
@deffn {Method} writable ()

Called each time around the asynchronous loop to determine whether a
channel’s socket should be added to the list on which write events can
occur.  The default method simply returns @code{True}, indicating that by
default, all channels will be interested in write events.
@end deffn

In addition, each channel delegates or extends many of the socket methods.
Most of these are nearly identical to their socket partners.

@geindex create_socket() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher create_socket}@anchor{2bd4}
@deffn {Method} create_socket (family=socket.AF_INET, type=socket.SOCK_STREAM)

This is identical to the creation of a normal socket, and will use the
same options for creation.  Refer to the @ref{ea,,socket} documentation for
information on creating sockets.


Changed in version 3.3: @emph{family} and @emph{type} arguments can be omitted.

@end deffn

@geindex connect() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher connect}@anchor{2bd3}
@deffn {Method} connect (address)

As with the normal socket object, @emph{address} is a tuple with the first
element the host to connect to, and the second the port number.
@end deffn

@geindex send() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher send}@anchor{2bd5}
@deffn {Method} send (data)

Send @emph{data} to the remote end-point of the socket.
@end deffn

@geindex recv() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher recv}@anchor{2bd6}
@deffn {Method} recv (buffer_size)

Read at most @emph{buffer_size} bytes from the socket’s remote end-point.  An
empty bytes object implies that the channel has been closed from the
other end.

Note that @ref{2bd6,,recv()} may raise @ref{bed,,BlockingIOError} , even though
@ref{8fc,,select.select()} or @ref{241d,,select.poll()} has reported the socket
ready for reading.
@end deffn

@geindex listen() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher listen}@anchor{2bd7}
@deffn {Method} listen (backlog)

Listen for connections made to the socket.  The @emph{backlog} argument
specifies the maximum number of queued connections and should be at least
1; the maximum value is system-dependent (usually 5).
@end deffn

@geindex bind() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher bind}@anchor{2bd8}
@deffn {Method} bind (address)

Bind the socket to @emph{address}.  The socket must not already be bound.  (The
format of @emph{address} depends on the address family — refer to the
@ref{ea,,socket} documentation for more information.)  To mark
the socket as re-usable (setting the @code{SO_REUSEADDR} option), call
the @ref{de0,,dispatcher} object’s @code{set_reuse_addr()} method.
@end deffn

@geindex accept() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher accept}@anchor{de3}
@deffn {Method} accept ()

Accept a connection.  The socket must be bound to an address and listening
for connections.  The return value can be either @code{None} or a pair
@code{(conn, address)} where @emph{conn} is a @emph{new} socket object usable to send
and receive data on the connection, and @emph{address} is the address bound to
the socket on the other end of the connection.
When @code{None} is returned it means the connection didn’t take place, in
which case the server should just ignore this event and keep listening
for further incoming connections.
@end deffn

@geindex close() (asyncore.dispatcher method)
@anchor{library/asyncore asyncore dispatcher close}@anchor{2bd9}
@deffn {Method} close ()

Close the socket.  All future operations on the socket object will fail.
The remote end-point will receive no more data (after queued data is
flushed).  Sockets are automatically closed when they are
garbage-collected.
@end deffn
@end deffn

@geindex dispatcher_with_send (class in asyncore)
@anchor{library/asyncore asyncore dispatcher_with_send}@anchor{2bda}
@deffn {Class} asyncore.dispatcher_with_send

A @ref{de0,,dispatcher} subclass which adds simple buffered output capability,
useful for simple clients. For more sophisticated usage use
@ref{b20,,asynchat.async_chat}.
@end deffn

@geindex file_dispatcher (class in asyncore)
@anchor{library/asyncore asyncore file_dispatcher}@anchor{2bdb}
@deffn {Class} asyncore.file_dispatcher

A file_dispatcher takes a file descriptor or @ref{d68,,file object} along
with an optional map argument and wraps it for use with the @code{poll()}
or @code{loop()} functions.  If provided a file object or anything with a
@code{fileno()} method, that method will be called and passed to the
@ref{2bdc,,file_wrapper} constructor.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex file_wrapper (class in asyncore)
@anchor{library/asyncore asyncore file_wrapper}@anchor{2bdc}
@deffn {Class} asyncore.file_wrapper

A file_wrapper takes an integer file descriptor and calls @ref{127f,,os.dup()} to
duplicate the handle so that the original handle may be closed independently
of the file_wrapper.  This class implements sufficient methods to emulate a
socket for use by the @ref{2bdb,,file_dispatcher} class.

@ref{1629,,Availability}: Unix.
@end deffn

@menu
* asyncore Example basic HTTP client:: 
* asyncore Example basic echo server:: 

@end menu

@node asyncore Example basic HTTP client,asyncore Example basic echo server,,asyncore — Asynchronous socket handler
@anchor{library/asyncore asyncore-example-1}@anchor{2bdd}@anchor{library/asyncore asyncore-example-basic-http-client}@anchor{2bde}
@subsubsection asyncore Example basic HTTP client


Here is a very basic HTTP client that uses the @ref{de0,,dispatcher} class to
implement its socket handling:

@example
import asyncore

class HTTPClient(asyncore.dispatcher):

    def __init__(self, host, path):
        asyncore.dispatcher.__init__(self)
        self.create_socket()
        self.connect( (host, 80) )
        self.buffer = bytes('GET %s HTTP/1.0\r\nHost: %s\r\n\r\n' %
                            (path, host), 'ascii')

    def handle_connect(self):
        pass

    def handle_close(self):
        self.close()

    def handle_read(self):
        print(self.recv(8192))

    def writable(self):
        return (len(self.buffer) > 0)

    def handle_write(self):
        sent = self.send(self.buffer)
        self.buffer = self.buffer[sent:]


client = HTTPClient('www.python.org', '/')
asyncore.loop()
@end example

@node asyncore Example basic echo server,,asyncore Example basic HTTP client,asyncore — Asynchronous socket handler
@anchor{library/asyncore asyncore-example-2}@anchor{2bdf}@anchor{library/asyncore asyncore-example-basic-echo-server}@anchor{2be0}
@subsubsection asyncore Example basic echo server


Here is a basic echo server that uses the @ref{de0,,dispatcher} class to accept
connections and dispatches the incoming connections to a handler:

@example
import asyncore

class EchoHandler(asyncore.dispatcher_with_send):

    def handle_read(self):
        data = self.recv(8192)
        if data:
            self.send(data)

class EchoServer(asyncore.dispatcher):

    def __init__(self, host, port):
        asyncore.dispatcher.__init__(self)
        self.create_socket()
        self.set_reuse_addr()
        self.bind((host, port))
        self.listen(5)

    def handle_accepted(self, sock, addr):
        print('Incoming connection from %s' % repr(addr))
        handler = EchoHandler(sock)

server = EchoServer('localhost', 8080)
asyncore.loop()
@end example

@node asynchat — Asynchronous socket command/response handler,signal — Set handlers for asynchronous events,asyncore — Asynchronous socket handler,Networking and Interprocess Communication
@anchor{library/asynchat doc}@anchor{2be1}@anchor{library/asynchat asynchat-asynchronous-socket-command-response-handler}@anchor{2be2}@anchor{library/asynchat module-asynchat}@anchor{8}
@subsection @code{asynchat} — Asynchronous socket command/response handler


@geindex module; asynchat

@strong{Source code:} Lib/asynchat.py@footnote{https://github.com/python/cpython/tree/main/Lib/asynchat.py}


Deprecated since version 3.6: Please use @ref{9,,asyncio} instead.



__________________________________________________________________


@cartouche
@quotation Note 
This module exists for backwards compatibility only.  For new code we
recommend using @ref{9,,asyncio}.
@end quotation
@end cartouche

This module builds on the @ref{a,,asyncore} infrastructure, simplifying
asynchronous clients and servers and making it easier to handle protocols
whose elements are terminated by arbitrary strings, or are of variable length.
@ref{8,,asynchat} defines the abstract class @ref{b20,,async_chat} that you
subclass, providing implementations of the @code{collect_incoming_data()} and
@code{found_terminator()} methods. It uses the same asynchronous loop as
@ref{a,,asyncore}, and the two types of channel, @ref{de0,,asyncore.dispatcher}
and @ref{b20,,asynchat.async_chat}, can freely be mixed in the channel map.
Typically an @ref{de0,,asyncore.dispatcher} server channel generates new
@ref{b20,,asynchat.async_chat} channel objects as it receives incoming
connection requests.

@geindex async_chat (class in asynchat)
@anchor{library/asynchat asynchat async_chat}@anchor{b20}
@deffn {Class} asynchat.async_chat

This class is an abstract subclass of @ref{de0,,asyncore.dispatcher}. To make
practical use of the code you must subclass @ref{b20,,async_chat}, providing
meaningful @ref{2be3,,collect_incoming_data()} and @ref{2be4,,found_terminator()}
methods.
The @ref{de0,,asyncore.dispatcher} methods can be used, although not all make
sense in a message/response context.

Like @ref{de0,,asyncore.dispatcher}, @ref{b20,,async_chat} defines a set of
events that are generated by an analysis of socket conditions after a
@code{select()} call. Once the polling loop has been started the
@ref{b20,,async_chat} object’s methods are called by the event-processing
framework with no action on the part of the programmer.

Two class attributes can be modified, to improve performance, or possibly
even to conserve memory.

@geindex async_chat.ac_in_buffer_size (in module asynchat)
@anchor{library/asynchat asynchat async_chat ac_in_buffer_size}@anchor{2be5}
@deffn {Data} ac_in_buffer_size

The asynchronous input buffer size (default @code{4096}).
@end deffn

@geindex async_chat.ac_out_buffer_size (in module asynchat)
@anchor{library/asynchat asynchat async_chat ac_out_buffer_size}@anchor{2be6}
@deffn {Data} ac_out_buffer_size

The asynchronous output buffer size (default @code{4096}).
@end deffn

Unlike @ref{de0,,asyncore.dispatcher}, @ref{b20,,async_chat} allows you to
define a @abbr{FIFO} queue of @emph{producers}. A producer need
have only one method, @code{more()}, which should return data to be
transmitted on the channel.
The producer indicates exhaustion (@emph{i.e.} that it contains no more data) by
having its @code{more()} method return the empty bytes object. At this point
the @ref{b20,,async_chat} object removes the producer from the queue and starts
using the next producer, if any. When the producer queue is empty the
@code{handle_write()} method does nothing. You use the channel object’s
@ref{2be7,,set_terminator()} method to describe how to recognize the end of, or
an important breakpoint in, an incoming transmission from the remote
endpoint.

To build a functioning @ref{b20,,async_chat} subclass your  input methods
@ref{2be3,,collect_incoming_data()} and @ref{2be4,,found_terminator()} must handle the
data that the channel receives asynchronously. The methods are described
below.
@end deffn

@geindex close_when_done() (asynchat.async_chat method)
@anchor{library/asynchat asynchat async_chat close_when_done}@anchor{2be8}
@deffn {Method} async_chat.close_when_done ()

Pushes a @code{None} on to the producer queue. When this producer is popped off
the queue it causes the channel to be closed.
@end deffn

@geindex collect_incoming_data() (asynchat.async_chat method)
@anchor{library/asynchat asynchat async_chat collect_incoming_data}@anchor{2be3}
@deffn {Method} async_chat.collect_incoming_data (data)

Called with @emph{data} holding an arbitrary amount of received data.  The
default method, which must be overridden, raises a
@ref{8a3,,NotImplementedError} exception.
@end deffn

@geindex discard_buffers() (asynchat.async_chat method)
@anchor{library/asynchat asynchat async_chat discard_buffers}@anchor{2be9}
@deffn {Method} async_chat.discard_buffers ()

In emergencies this method will discard any data held in the input and/or
output buffers and the producer queue.
@end deffn

@geindex found_terminator() (asynchat.async_chat method)
@anchor{library/asynchat asynchat async_chat found_terminator}@anchor{2be4}
@deffn {Method} async_chat.found_terminator ()

Called when the incoming data stream  matches the termination condition set
by @ref{2be7,,set_terminator()}. The default method, which must be overridden,
raises a @ref{8a3,,NotImplementedError} exception. The buffered input data
should be available via an instance attribute.
@end deffn

@geindex get_terminator() (asynchat.async_chat method)
@anchor{library/asynchat asynchat async_chat get_terminator}@anchor{2bea}
@deffn {Method} async_chat.get_terminator ()

Returns the current terminator for the channel.
@end deffn

@geindex push() (asynchat.async_chat method)
@anchor{library/asynchat asynchat async_chat push}@anchor{2beb}
@deffn {Method} async_chat.push (data)

Pushes data on to the channel’s queue to ensure its transmission.
This is all you need to do to have the channel write the data out to the
network, although it is possible to use your own producers in more complex
schemes to implement encryption and chunking, for example.
@end deffn

@geindex push_with_producer() (asynchat.async_chat method)
@anchor{library/asynchat asynchat async_chat push_with_producer}@anchor{2bec}
@deffn {Method} async_chat.push_with_producer (producer)

Takes a producer object and adds it to the producer queue associated with
the channel.  When all currently-pushed producers have been exhausted the
channel will consume this producer’s data by calling its @code{more()}
method and send the data to the remote endpoint.
@end deffn

@geindex set_terminator() (asynchat.async_chat method)
@anchor{library/asynchat asynchat async_chat set_terminator}@anchor{2be7}
@deffn {Method} async_chat.set_terminator (term)

Sets the terminating condition to be recognized on the channel.  @code{term}
may be any of three types of value, corresponding to three different ways
to handle incoming protocol data.


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

term

@tab

Description

@item

@emph{string}

@tab

Will call @ref{2be4,,found_terminator()} when the
string is found in the input stream

@item

@emph{integer}

@tab

Will call @ref{2be4,,found_terminator()} when the
indicated number of characters have been
received

@item

@code{None}

@tab

The channel continues to collect data
forever

@end multitable


Note that any data following the terminator will be available for reading
by the channel after @ref{2be4,,found_terminator()} is called.
@end deffn

@menu
* asynchat Example:: 

@end menu

@node asynchat Example,,,asynchat — Asynchronous socket command/response handler
@anchor{library/asynchat asynchat-example}@anchor{2bed}@anchor{library/asynchat id1}@anchor{2bee}
@subsubsection asynchat Example


The following partial example shows how HTTP requests can be read with
@ref{b20,,async_chat}.  A web server might create an
@code{http_request_handler} object for each incoming client connection.
Notice that initially the channel terminator is set to match the blank line at
the end of the HTTP headers, and a flag indicates that the headers are being
read.

Once the headers have been read, if the request is of type POST (indicating
that further data are present in the input stream) then the
@code{Content-Length:} header is used to set a numeric terminator to read the
right amount of data from the channel.

The @code{handle_request()} method is called once all relevant input has been
marshalled, after setting the channel terminator to @code{None} to ensure that
any extraneous data sent by the web client are ignored.

@example
import asynchat

class http_request_handler(asynchat.async_chat):

    def __init__(self, sock, addr, sessions, log):
        asynchat.async_chat.__init__(self, sock=sock)
        self.addr = addr
        self.sessions = sessions
        self.ibuffer = []
        self.obuffer = b""
        self.set_terminator(b"\r\n\r\n")
        self.reading_headers = True
        self.handling = False
        self.cgi_data = None
        self.log = log

    def collect_incoming_data(self, data):
        """Buffer the data"""
        self.ibuffer.append(data)

    def found_terminator(self):
        if self.reading_headers:
            self.reading_headers = False
            self.parse_headers(b"".join(self.ibuffer))
            self.ibuffer = []
            if self.op.upper() == b"POST":
                clen = self.headers.getheader("content-length")
                self.set_terminator(int(clen))
            else:
                self.handling = True
                self.set_terminator(None)
                self.handle_request()
        elif not self.handling:
            self.set_terminator(None)  # browsers sometimes over-send
            self.cgi_data = parse(self.headers, b"".join(self.ibuffer))
            self.handling = True
            self.ibuffer = []
            self.handle_request()
@end example

@node signal — Set handlers for asynchronous events,mmap — Memory-mapped file support,asynchat — Asynchronous socket command/response handler,Networking and Interprocess Communication
@anchor{library/signal doc}@anchor{2bef}@anchor{library/signal module-signal}@anchor{e5}@anchor{library/signal signal-set-handlers-for-asynchronous-events}@anchor{2bf0}
@subsection @code{signal} — Set handlers for asynchronous events


@geindex module; signal


__________________________________________________________________


This module provides mechanisms to use signal handlers in Python.

@menu
* General rules:: 
* Module contents: Module contents<2>. 
* Examples: Examples<20>. 
* Note on SIGPIPE:: 

@end menu

@node General rules,Module contents<2>,,signal — Set handlers for asynchronous events
@anchor{library/signal general-rules}@anchor{2bf1}
@subsubsection General rules


The @ref{cc9,,signal.signal()} function allows defining custom handlers to be
executed when a signal is received.  A small number of default handlers are
installed: @ref{2bf2,,SIGPIPE} is ignored (so write errors on pipes and sockets
can be reported as ordinary Python exceptions) and @ref{2ca,,SIGINT} is
translated into a @ref{4ca,,KeyboardInterrupt} exception if the parent process
has not changed it.

A handler for a particular signal, once set, remains installed until it is
explicitly reset (Python emulates the BSD style interface regardless of the
underlying implementation), with the exception of the handler for
@ref{2bf3,,SIGCHLD}, which follows the underlying implementation.

@menu
* Execution of Python signal handlers:: 
* Signals and threads:: 

@end menu

@node Execution of Python signal handlers,Signals and threads,,General rules
@anchor{library/signal execution-of-python-signal-handlers}@anchor{2bf4}
@subsubsection Execution of Python signal handlers


A Python signal handler does not get executed inside the low-level (C) signal
handler.  Instead, the low-level signal handler sets a flag which tells the
@ref{2bf5,,virtual machine} to execute the corresponding Python signal handler
at a later point(for example at the next @ref{8a8,,bytecode} instruction).
This has consequences:


@itemize *

@item 
It makes little sense to catch synchronous errors like @ref{2bf6,,SIGFPE} or
@ref{2bf7,,SIGSEGV} that are caused by an invalid operation in C code.  Python
will return from the signal handler to the C code, which is likely to raise
the same signal again, causing Python to apparently hang.  From Python 3.3
onwards, you can use the @ref{78,,faulthandler} module to report on synchronous
errors.

@item 
A long-running calculation implemented purely in C (such as regular
expression matching on a large body of text) may run uninterrupted for an
arbitrary amount of time, regardless of any signals received.  The Python
signal handlers will be called when the calculation finishes.
@end itemize

@node Signals and threads,,Execution of Python signal handlers,General rules
@anchor{library/signal id1}@anchor{2bf8}@anchor{library/signal signals-and-threads}@anchor{2bf9}
@subsubsection Signals and threads


Python signal handlers are always executed in the main Python thread of the main interpreter,
even if the signal was received in another thread.  This means that signals
can’t be used as a means of inter-thread communication.  You can use
the synchronization primitives from the @ref{109,,threading} module instead.

Besides, only the main thread of the main interpreter is allowed to set a new signal handler.

@node Module contents<2>,Examples<20>,General rules,signal — Set handlers for asynchronous events
@anchor{library/signal module-contents}@anchor{2bfa}
@subsubsection Module contents



Changed in version 3.5: signal (SIG*), handler (@ref{295d,,SIG_DFL}, @ref{295e,,SIG_IGN}) and sigmask
(@ref{2bfb,,SIG_BLOCK}, @ref{2bfc,,SIG_UNBLOCK}, @ref{2bfd,,SIG_SETMASK})
related constants listed below were turned into
@ref{828,,enums} (@ref{2bfe,,Signals}, @ref{2bff,,Handlers} and @ref{2c00,,Sigmasks} respectively).
@ref{2c01,,getsignal()}, @ref{cc5,,pthread_sigmask()}, @ref{cc7,,sigpending()} and
@ref{cc8,,sigwait()} functions return human-readable
@ref{828,,enums} as @ref{2bfe,,Signals} objects.


The signal module defines three enums:

@geindex Signals (class in signal)
@anchor{library/signal signal Signals}@anchor{2bfe}
@deffn {Class} signal.Signals

@ref{828,,enum.IntEnum} collection of SIG* constants and the CTRL_* constants.


New in version 3.5.

@end deffn

@geindex Handlers (class in signal)
@anchor{library/signal signal Handlers}@anchor{2bff}
@deffn {Class} signal.Handlers

@ref{828,,enum.IntEnum} collection the constants @ref{295d,,SIG_DFL} and @ref{295e,,SIG_IGN}.


New in version 3.5.

@end deffn

@geindex Sigmasks (class in signal)
@anchor{library/signal signal Sigmasks}@anchor{2c00}
@deffn {Class} signal.Sigmasks

@ref{828,,enum.IntEnum} collection the constants @ref{2bfb,,SIG_BLOCK}, @ref{2bfc,,SIG_UNBLOCK} and @ref{2bfd,,SIG_SETMASK}.

Availability: Unix. See the man page @code{sigprocmask(3)} and
@code{pthread_sigmask(3)} for further information.


New in version 3.5.

@end deffn

The variables defined in the @ref{e5,,signal} module are:

@geindex SIG_DFL (in module signal)
@anchor{library/signal signal SIG_DFL}@anchor{295d}
@deffn {Data} signal.SIG_DFL

This is one of two standard signal handling options; it will simply perform
the default function for the signal.  For example, on most systems the
default action for @code{SIGQUIT} is to dump core and exit, while the
default action for @ref{2bf3,,SIGCHLD} is to simply ignore it.
@end deffn

@geindex SIG_IGN (in module signal)
@anchor{library/signal signal SIG_IGN}@anchor{295e}
@deffn {Data} signal.SIG_IGN

This is another standard signal handler, which will simply ignore the given
signal.
@end deffn

@geindex SIGABRT (in module signal)
@anchor{library/signal signal SIGABRT}@anchor{2c02}
@deffn {Data} signal.SIGABRT

Abort signal from @code{abort(3)}.
@end deffn

@geindex SIGALRM (in module signal)
@anchor{library/signal signal SIGALRM}@anchor{2c03}
@deffn {Data} signal.SIGALRM

Timer signal from @code{alarm(2)}.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex SIGBREAK (in module signal)
@anchor{library/signal signal SIGBREAK}@anchor{2c04}
@deffn {Data} signal.SIGBREAK

Interrupt from keyboard (CTRL + BREAK).

@ref{1629,,Availability}: Windows.
@end deffn

@geindex SIGBUS (in module signal)
@anchor{library/signal signal SIGBUS}@anchor{2c05}
@deffn {Data} signal.SIGBUS

Bus error (bad memory access).

@ref{1629,,Availability}: Unix.
@end deffn

@geindex SIGCHLD (in module signal)
@anchor{library/signal signal SIGCHLD}@anchor{2bf3}
@deffn {Data} signal.SIGCHLD

Child process stopped or terminated.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex SIGCLD (in module signal)
@anchor{library/signal signal SIGCLD}@anchor{2c06}
@deffn {Data} signal.SIGCLD

Alias to @ref{2bf3,,SIGCHLD}.
@end deffn

@geindex SIGCONT (in module signal)
@anchor{library/signal signal SIGCONT}@anchor{246a}
@deffn {Data} signal.SIGCONT

Continue the process if it is currently stopped

@ref{1629,,Availability}: Unix.
@end deffn

@geindex SIGFPE (in module signal)
@anchor{library/signal signal SIGFPE}@anchor{2bf6}
@deffn {Data} signal.SIGFPE

Floating-point exception. For example, division by zero.


@subsubheading See also


@ref{43c,,ZeroDivisionError} is raised when the second argument of a division
or modulo operation is zero.

@end deffn

@geindex SIGHUP (in module signal)
@anchor{library/signal signal SIGHUP}@anchor{2c07}
@deffn {Data} signal.SIGHUP

Hangup detected on controlling terminal or death of controlling process.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex SIGILL (in module signal)
@anchor{library/signal signal SIGILL}@anchor{2c08}
@deffn {Data} signal.SIGILL

Illegal instruction.
@end deffn

@geindex SIGINT (in module signal)
@anchor{library/signal signal SIGINT}@anchor{2ca}
@deffn {Data} signal.SIGINT

Interrupt from keyboard (CTRL + C).

Default action is to raise @ref{4ca,,KeyboardInterrupt}.
@end deffn

@geindex SIGKILL (in module signal)
@anchor{library/signal signal SIGKILL}@anchor{2c09}
@deffn {Data} signal.SIGKILL

Kill signal.

It cannot be caught, blocked, or ignored.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex SIGPIPE (in module signal)
@anchor{library/signal signal SIGPIPE}@anchor{2bf2}
@deffn {Data} signal.SIGPIPE

Broken pipe: write to pipe with no readers.

Default action is to ignore the signal.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex SIGSEGV (in module signal)
@anchor{library/signal signal SIGSEGV}@anchor{2bf7}
@deffn {Data} signal.SIGSEGV

Segmentation fault: invalid memory reference.
@end deffn

@geindex SIGSTKFLT (in module signal)
@anchor{library/signal signal SIGSTKFLT}@anchor{10bf}
@deffn {Data} signal.SIGSTKFLT

@quotation

Stack fault on coprocessor. The Linux kernel does not raise this signal: it
can only be raised in user space.
@end quotation

@ref{1629,,Availability}: Linux, on architectures where the signal is available. See
the man page @code{signal(7)} for further information.


New in version 3.11.

@end deffn

@geindex SIGTERM (in module signal)
@anchor{library/signal signal SIGTERM}@anchor{29f9}
@deffn {Data} signal.SIGTERM

Termination signal.
@end deffn

@geindex SIGUSR1 (in module signal)
@anchor{library/signal signal SIGUSR1}@anchor{2c0a}
@deffn {Data} signal.SIGUSR1

User-defined signal 1.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex SIGUSR2 (in module signal)
@anchor{library/signal signal SIGUSR2}@anchor{2c0b}
@deffn {Data} signal.SIGUSR2

User-defined signal 2.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex SIGWINCH (in module signal)
@anchor{library/signal signal SIGWINCH}@anchor{2c0c}
@deffn {Data} signal.SIGWINCH

Window resize signal.

@ref{1629,,Availability}: Unix.
@end deffn


@deffn {Data} SIG*

All the signal numbers are defined symbolically.  For example, the hangup signal
is defined as @ref{2c07,,signal.SIGHUP}; the variable names are identical to the
names used in C programs, as found in @code{<signal.h>}.  The Unix man page for
‘@code{signal()}’ lists the existing signals (on some systems this is
@code{signal(2)}, on others the list is in @code{signal(7)}). Note that
not all systems define the same set of signal names; only those names defined by
the system are defined by this module.
@end deffn

@geindex CTRL_C_EVENT (in module signal)
@anchor{library/signal signal CTRL_C_EVENT}@anchor{2442}
@deffn {Data} signal.CTRL_C_EVENT

The signal corresponding to the @code{Ctrl+C} keystroke event. This signal can
only be used with @ref{ef3,,os.kill()}.

@ref{1629,,Availability}: Windows.


New in version 3.2.

@end deffn

@geindex CTRL_BREAK_EVENT (in module signal)
@anchor{library/signal signal CTRL_BREAK_EVENT}@anchor{2443}
@deffn {Data} signal.CTRL_BREAK_EVENT

The signal corresponding to the @code{Ctrl+Break} keystroke event. This signal can
only be used with @ref{ef3,,os.kill()}.

@ref{1629,,Availability}: Windows.


New in version 3.2.

@end deffn

@geindex NSIG (in module signal)
@anchor{library/signal signal NSIG}@anchor{2c0d}
@deffn {Data} signal.NSIG

One more than the number of the highest signal number.
@end deffn

@geindex ITIMER_REAL (in module signal)
@anchor{library/signal signal ITIMER_REAL}@anchor{2c0e}
@deffn {Data} signal.ITIMER_REAL

Decrements interval timer in real time, and delivers @ref{2c03,,SIGALRM} upon
expiration.
@end deffn

@geindex ITIMER_VIRTUAL (in module signal)
@anchor{library/signal signal ITIMER_VIRTUAL}@anchor{2c0f}
@deffn {Data} signal.ITIMER_VIRTUAL

Decrements interval timer only when the process is executing, and delivers
SIGVTALRM upon expiration.
@end deffn

@geindex ITIMER_PROF (in module signal)
@anchor{library/signal signal ITIMER_PROF}@anchor{2c10}
@deffn {Data} signal.ITIMER_PROF

Decrements interval timer both when the process executes and when the
system is executing on behalf of the process. Coupled with ITIMER_VIRTUAL,
this timer is usually used to profile the time spent by the application
in user and kernel space. SIGPROF is delivered upon expiration.
@end deffn

@geindex SIG_BLOCK (in module signal)
@anchor{library/signal signal SIG_BLOCK}@anchor{2bfb}
@deffn {Data} signal.SIG_BLOCK

A possible value for the @emph{how} parameter to @ref{cc5,,pthread_sigmask()}
indicating that signals are to be blocked.


New in version 3.3.

@end deffn

@geindex SIG_UNBLOCK (in module signal)
@anchor{library/signal signal SIG_UNBLOCK}@anchor{2bfc}
@deffn {Data} signal.SIG_UNBLOCK

A possible value for the @emph{how} parameter to @ref{cc5,,pthread_sigmask()}
indicating that signals are to be unblocked.


New in version 3.3.

@end deffn

@geindex SIG_SETMASK (in module signal)
@anchor{library/signal signal SIG_SETMASK}@anchor{2bfd}
@deffn {Data} signal.SIG_SETMASK

A possible value for the @emph{how} parameter to @ref{cc5,,pthread_sigmask()}
indicating that the signal mask is to be replaced.


New in version 3.3.

@end deffn

The @ref{e5,,signal} module defines one exception:

@geindex ItimerError
@anchor{library/signal signal ItimerError}@anchor{2c11}
@deffn {Exception} signal.ItimerError

Raised to signal an error from the underlying @ref{2c12,,setitimer()} or
@ref{2c13,,getitimer()} implementation. Expect this error if an invalid
interval timer or a negative time is passed to @ref{2c12,,setitimer()}.
This error is a subtype of @ref{2a1,,OSError}.


New in version 3.3: This error used to be a subtype of @ref{be8,,IOError}, which is now an
alias of @ref{2a1,,OSError}.

@end deffn

The @ref{e5,,signal} module defines the following functions:

@geindex alarm() (in module signal)
@anchor{library/signal signal alarm}@anchor{2c14}
@deffn {Function} signal.alarm (time)

If @emph{time} is non-zero, this function requests that a @ref{2c03,,SIGALRM} signal be
sent to the process in @emph{time} seconds. Any previously scheduled alarm is
canceled (only one alarm can be scheduled at any time).  The returned value is
then the number of seconds before any previously set alarm was to have been
delivered. If @emph{time} is zero, no alarm is scheduled, and any scheduled alarm is
canceled.  If the return value is zero, no alarm is currently scheduled.

@ref{1629,,Availability}: Unix.  See the man page @code{alarm(2)} for further
information.
@end deffn

@geindex getsignal() (in module signal)
@anchor{library/signal signal getsignal}@anchor{2c01}
@deffn {Function} signal.getsignal (signalnum)

Return the current signal handler for the signal @emph{signalnum}. The returned value
may be a callable Python object, or one of the special values
@ref{295e,,signal.SIG_IGN}, @ref{295d,,signal.SIG_DFL} or @ref{182,,None}.  Here,
@ref{295e,,signal.SIG_IGN} means that the signal was previously ignored,
@ref{295d,,signal.SIG_DFL} means that the default way of handling the signal was
previously in use, and @code{None} means that the previous signal handler was not
installed from Python.
@end deffn

@geindex strsignal() (in module signal)
@anchor{library/signal signal strsignal}@anchor{12f5}
@deffn {Function} signal.strsignal (signalnum)

Return the system description of the signal @emph{signalnum}, such as
“Interrupt”, “Segmentation fault”, etc. Returns @ref{182,,None} if the signal
is not recognized.


New in version 3.8.

@end deffn

@geindex valid_signals() (in module signal)
@anchor{library/signal signal valid_signals}@anchor{2c15}
@deffn {Function} signal.valid_signals ()

Return the set of valid signal numbers on this platform.  This can be
less than @code{range(1, NSIG)} if some signals are reserved by the system
for internal use.


New in version 3.8.

@end deffn

@geindex pause() (in module signal)
@anchor{library/signal signal pause}@anchor{2c16}
@deffn {Function} signal.pause ()

Cause the process to sleep until a signal is received; the appropriate handler
will then be called.  Returns nothing.

@ref{1629,,Availability}: Unix.  See the man page @code{signal(2)} for further
information.

See also @ref{cc8,,sigwait()}, @ref{908,,sigwaitinfo()}, @ref{907,,sigtimedwait()} and
@ref{cc7,,sigpending()}.
@end deffn

@geindex raise_signal() (in module signal)
@anchor{library/signal signal raise_signal}@anchor{295f}
@deffn {Function} signal.raise_signal (signum)

Sends a signal to the calling process. Returns nothing.


New in version 3.8.

@end deffn

@geindex pidfd_send_signal() (in module signal)
@anchor{library/signal signal pidfd_send_signal}@anchor{41f}
@deffn {Function} signal.pidfd_send_signal (pidfd, sig, siginfo=None, flags=0)

Send signal @emph{sig} to the process referred to by file descriptor @emph{pidfd}.
Python does not currently support the @emph{siginfo} parameter; it must be
@code{None}.  The @emph{flags} argument is provided for future extensions; no flag
values are currently defined.

See the @code{pidfd_send_signal(2)} man page for more information.

@ref{1629,,Availability}: Linux 5.1+


New in version 3.9.

@end deffn

@geindex pthread_kill() (in module signal)
@anchor{library/signal signal pthread_kill}@anchor{cc6}
@deffn {Function} signal.pthread_kill (thread_id, signalnum)

Send the signal @emph{signalnum} to the thread @emph{thread_id}, another thread in the
same process as the caller.  The target thread can be executing any code
(Python or not).  However, if the target thread is executing the Python
interpreter, the Python signal handlers will be @ref{2bf9,,executed by the main thread of the main interpreter}.  Therefore, the only point of sending a
signal to a particular Python thread would be to force a running system call
to fail with @ref{8e3,,InterruptedError}.

Use @ref{cf5,,threading.get_ident()} or the @ref{24b8,,ident}
attribute of @ref{44b,,threading.Thread} objects to get a suitable value
for @emph{thread_id}.

If @emph{signalnum} is 0, then no signal is sent, but error checking is still
performed; this can be used to check if the target thread is still running.

Raises an @ref{1130,,auditing event} @code{signal.pthread_kill} with arguments @code{thread_id}, @code{signalnum}.

@ref{1629,,Availability}: Unix.  See the man page @code{pthread_kill(3)} for further
information.

See also @ref{ef3,,os.kill()}.


New in version 3.3.

@end deffn

@geindex pthread_sigmask() (in module signal)
@anchor{library/signal signal pthread_sigmask}@anchor{cc5}
@deffn {Function} signal.pthread_sigmask (how, mask)

Fetch and/or change the signal mask of the calling thread.  The signal mask
is the set of signals whose delivery is currently blocked for the caller.
Return the old signal mask as a set of signals.

The behavior of the call is dependent on the value of @emph{how}, as follows.


@itemize *

@item 
@ref{2bfb,,SIG_BLOCK}: The set of blocked signals is the union of the current
set and the @emph{mask} argument.

@item 
@ref{2bfc,,SIG_UNBLOCK}: The signals in @emph{mask} are removed from the current
set of blocked signals.  It is permissible to attempt to unblock a
signal which is not blocked.

@item 
@ref{2bfd,,SIG_SETMASK}: The set of blocked signals is set to the @emph{mask}
argument.
@end itemize

@emph{mask} is a set of signal numbers (e.g. @{@ref{2ca,,signal.SIGINT},
@ref{29f9,,signal.SIGTERM}@}). Use @ref{2c15,,valid_signals()} for a full
mask including all signals.

For example, @code{signal.pthread_sigmask(signal.SIG_BLOCK, [])} reads the
signal mask of the calling thread.

@ref{2c09,,SIGKILL} and @code{SIGSTOP} cannot be blocked.

@ref{1629,,Availability}: Unix.  See the man page @code{sigprocmask(2)} and
@code{pthread_sigmask(3)} for further information.

See also @ref{2c16,,pause()}, @ref{cc7,,sigpending()} and @ref{cc8,,sigwait()}.


New in version 3.3.

@end deffn

@geindex setitimer() (in module signal)
@anchor{library/signal signal setitimer}@anchor{2c12}
@deffn {Function} signal.setitimer (which, seconds, interval=0.0)

Sets given interval timer (one of @ref{2c0e,,signal.ITIMER_REAL},
@ref{2c0f,,signal.ITIMER_VIRTUAL} or @ref{2c10,,signal.ITIMER_PROF}) specified
by @emph{which} to fire after @emph{seconds} (float is accepted, different from
@ref{2c14,,alarm()}) and after that every @emph{interval} seconds (if @emph{interval}
is non-zero). The interval timer specified by @emph{which} can be cleared by
setting @emph{seconds} to zero.

When an interval timer fires, a signal is sent to the process.
The signal sent is dependent on the timer being used;
@ref{2c0e,,signal.ITIMER_REAL} will deliver @ref{2c03,,SIGALRM},
@ref{2c0f,,signal.ITIMER_VIRTUAL} sends @code{SIGVTALRM},
and @ref{2c10,,signal.ITIMER_PROF} will deliver @code{SIGPROF}.

The old values are returned as a tuple: (delay, interval).

Attempting to pass an invalid interval timer will cause an
@ref{2c11,,ItimerError}.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex getitimer() (in module signal)
@anchor{library/signal signal getitimer}@anchor{2c13}
@deffn {Function} signal.getitimer (which)

Returns current value of a given interval timer specified by @emph{which}.

@ref{1629,,Availability}: Unix.
@end deffn

@geindex set_wakeup_fd() (in module signal)
@anchor{library/signal signal set_wakeup_fd}@anchor{69d}
@deffn {Function} signal.set_wakeup_fd (fd, *, warn_on_full_buffer=True)

Set the wakeup file descriptor to @emph{fd}.  When a signal is received, the
signal number is written as a single byte into the fd.  This can be used by
a library to wakeup a poll or select call, allowing the signal to be fully
processed.

The old wakeup fd is returned (or -1 if file descriptor wakeup was not
enabled).  If @emph{fd} is -1, file descriptor wakeup is disabled.
If not -1, @emph{fd} must be non-blocking.  It is up to the library to remove
any bytes from @emph{fd} before calling poll or select again.

When threads are enabled, this function can only be called
from @ref{2bf9,,the main thread of the main interpreter};
attempting to call it from other threads will cause a @ref{17c,,ValueError}
exception to be raised.

There are two common ways to use this function. In both approaches,
you use the fd to wake up when a signal arrives, but then they
differ in how they determine @emph{which} signal or signals have
arrived.

In the first approach, we read the data out of the fd’s buffer, and
the byte values give you the signal numbers. This is simple, but in
rare cases it can run into a problem: generally the fd will have a
limited amount of buffer space, and if too many signals arrive too
quickly, then the buffer may become full, and some signals may be
lost. If you use this approach, then you should set
@code{warn_on_full_buffer=True}, which will at least cause a warning
to be printed to stderr when signals are lost.

In the second approach, we use the wakeup fd @emph{only} for wakeups,
and ignore the actual byte values. In this case, all we care about
is whether the fd’s buffer is empty or non-empty; a full buffer
doesn’t indicate a problem at all. If you use this approach, then
you should set @code{warn_on_full_buffer=False}, so that your users
are not confused by spurious warning messages.


Changed in version 3.5: On Windows, the function now also supports socket handles.



Changed in version 3.7: Added @code{warn_on_full_buffer} parameter.

@end deffn

@geindex siginterrupt() (in module signal)
@anchor{library/signal signal siginterrupt}@anchor{cca}
@deffn {Function} signal.siginterrupt (signalnum, flag)

Change system call restart behaviour: if @emph{flag} is @ref{659,,False}, system
calls will be restarted when interrupted by signal @emph{signalnum}, otherwise
system calls will be interrupted.  Returns nothing.

@ref{1629,,Availability}: Unix.  See the man page @code{siginterrupt(3)}
for further information.

Note that installing a signal handler with @ref{e5,,signal()} will reset the
restart behaviour to interruptible by implicitly calling
@code{siginterrupt()} with a true @emph{flag} value for the given signal.
@end deffn

@geindex signal() (in module signal)
@anchor{library/signal signal signal}@anchor{cc9}
@deffn {Function} signal.signal (signalnum, handler)

Set the handler for signal @emph{signalnum} to the function @emph{handler}.  @emph{handler} can
be a callable Python object taking two arguments (see below), or one of the
special values @ref{295e,,signal.SIG_IGN} or @ref{295d,,signal.SIG_DFL}.  The previous
signal handler will be returned (see the description of @ref{2c01,,getsignal()}
above).  (See the Unix man page @code{signal(2)} for further information.)

When threads are enabled, this function can only be called
from @ref{2bf9,,the main thread of the main interpreter};
attempting to call it from other threads will cause a @ref{17c,,ValueError}
exception to be raised.

The @emph{handler} is called with two arguments: the signal number and the current
stack frame (@code{None} or a frame object; for a description of frame objects,
see the @ref{179b,,description in the type hierarchy} or see the
attribute descriptions in the @ref{9c,,inspect} module).

On Windows, @ref{e5,,signal()} can only be called with @ref{2c02,,SIGABRT},
@ref{2bf6,,SIGFPE}, @ref{2c08,,SIGILL}, @ref{2ca,,SIGINT}, @ref{2bf7,,SIGSEGV},
@ref{29f9,,SIGTERM}, or @ref{2c04,,SIGBREAK}.
A @ref{17c,,ValueError} will be raised in any other case.
Note that not all systems define the same set of signal names; an
@ref{163,,AttributeError} will be raised if a signal name is not defined as
@code{SIG*} module level constant.
@end deffn

@geindex sigpending() (in module signal)
@anchor{library/signal signal sigpending}@anchor{cc7}
@deffn {Function} signal.sigpending ()

Examine the set of signals that are pending for delivery to the calling
thread (i.e., the signals which have been raised while blocked).  Return the
set of the pending signals.

@ref{1629,,Availability}: Unix.  See the man page @code{sigpending(2)} for further
information.

See also @ref{2c16,,pause()}, @ref{cc5,,pthread_sigmask()} and @ref{cc8,,sigwait()}.


New in version 3.3.

@end deffn

@geindex sigwait() (in module signal)
@anchor{library/signal signal sigwait}@anchor{cc8}
@deffn {Function} signal.sigwait (sigset)

Suspend execution of the calling thread until the delivery of one of the
signals specified in the signal set @emph{sigset}.  The function accepts the signal
(removes it from the pending list of signals), and returns the signal number.

@ref{1629,,Availability}: Unix.  See the man page @code{sigwait(3)} for further
information.

See also @ref{2c16,,pause()}, @ref{cc5,,pthread_sigmask()}, @ref{cc7,,sigpending()},
@ref{908,,sigwaitinfo()} and @ref{907,,sigtimedwait()}.


New in version 3.3.

@end deffn

@geindex sigwaitinfo() (in module signal)
@anchor{library/signal signal sigwaitinfo}@anchor{908}
@deffn {Function} signal.sigwaitinfo (sigset)

Suspend execution of the calling thread until the delivery of one of the
signals specified in the signal set @emph{sigset}.  The function accepts the
signal and removes it from the pending list of signals. If one of the
signals in @emph{sigset} is already pending for the calling thread, the function
will return immediately with information about that signal. The signal
handler is not called for the delivered signal. The function raises an
@ref{8e3,,InterruptedError} if it is interrupted by a signal that is not in
@emph{sigset}.

The return value is an object representing the data contained in the
@code{siginfo_t} structure, namely: @code{si_signo}, @code{si_code},
@code{si_errno}, @code{si_pid}, @code{si_uid}, @code{si_status},
@code{si_band}.

@ref{1629,,Availability}: Unix.  See the man page @code{sigwaitinfo(2)} for further
information.

See also @ref{2c16,,pause()}, @ref{cc8,,sigwait()} and @ref{907,,sigtimedwait()}.


New in version 3.3.



Changed in version 3.5: The function is now retried if interrupted by a signal not in @emph{sigset}
and the signal handler does not raise an exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for
the rationale).

@end deffn

@geindex sigtimedwait() (in module signal)
@anchor{library/signal signal sigtimedwait}@anchor{907}
@deffn {Function} signal.sigtimedwait (sigset, timeout)

Like @ref{908,,sigwaitinfo()}, but takes an additional @emph{timeout} argument
specifying a timeout. If @emph{timeout} is specified as @code{0}, a poll is
performed. Returns @ref{182,,None} if a timeout occurs.

@ref{1629,,Availability}: Unix.  See the man page @code{sigtimedwait(2)} for further
information.

See also @ref{2c16,,pause()}, @ref{cc8,,sigwait()} and @ref{908,,sigwaitinfo()}.


New in version 3.3.



Changed in version 3.5: The function is now retried with the recomputed @emph{timeout} if interrupted
by a signal not in @emph{sigset} and the signal handler does not raise an
exception (see 
@geindex Python Enhancement Proposals; PEP 475
PEP 475@footnote{https://www.python.org/dev/peps/pep-0475} for the rationale).

@end deffn

@node Examples<20>,Note on SIGPIPE,Module contents<2>,signal — Set handlers for asynchronous events
@anchor{library/signal examples}@anchor{2c17}@anchor{library/signal signal-example}@anchor{2c18}
@subsubsection Examples


Here is a minimal example program. It uses the @ref{2c14,,alarm()} function to limit
the time spent waiting to open a file; this is useful if the file is for a
serial device that may not be turned on, which would normally cause the
@ref{8ee,,os.open()} to hang indefinitely.  The solution is to set a 5-second alarm
before opening the file; if the operation takes too long, the alarm signal will
be sent, and the handler raises an exception.

@example
import signal, os

def handler(signum, frame):
    signame = signal.Signals(signum).name
    print(f'Signal handler called with signal @{signame@} (@{signum@})')
    raise OSError("Couldn't open device!")

# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)

# This open() may hang indefinitely
fd = os.open('/dev/ttyS0', os.O_RDWR)

signal.alarm(0)          # Disable the alarm
@end example

@node Note on SIGPIPE,,Examples<20>,signal — Set handlers for asynchronous events
@anchor{library/signal note-on-sigpipe}@anchor{2c19}
@subsubsection Note on SIGPIPE


Piping output of your program to tools like @code{head(1)} will
cause a @ref{2bf2,,SIGPIPE} signal to be sent to your process when the receiver
of its standard output closes early.  This results in an exception
like @code{BrokenPipeError: [Errno 32] Broken pipe}.  To handle this
case, wrap your entry point to catch this exception as follows:

@example
import os
import sys

def main():
    try:
        # simulate large output (your code replaces this loop)
        for x in range(10000):
            print("y")
        # flush output here to force SIGPIPE to be triggered
        # while inside this try block.
        sys.stdout.flush()
    except BrokenPipeError:
        # Python flushes standard streams on exit; redirect remaining output
        # to devnull to avoid another BrokenPipeError at shutdown
        devnull = os.open(os.devnull, os.O_WRONLY)
        os.dup2(devnull, sys.stdout.fileno())
        sys.exit(1)  # Python exits with error code 1 on EPIPE

if __name__ == '__main__':
    main()
@end example

Do not set @ref{2bf2,,SIGPIPE}’s disposition to @ref{295d,,SIG_DFL}
in order to avoid @ref{bf2,,BrokenPipeError}.  Doing that would cause
your program to exit unexpectedly also whenever any socket connection
is interrupted while your program is still writing to it.

@node mmap — Memory-mapped file support,,signal — Set handlers for asynchronous events,Networking and Interprocess Communication
@anchor{library/mmap doc}@anchor{2c1a}@anchor{library/mmap mmap-memory-mapped-file-support}@anchor{2c1b}@anchor{library/mmap module-mmap}@anchor{af}
@subsection @code{mmap} — Memory-mapped file support


@geindex module; mmap


__________________________________________________________________


Memory-mapped file objects behave like both @ref{2f1,,bytearray} and like
@ref{d68,,file objects}.  You can use mmap objects in most places
where @ref{2f1,,bytearray} are expected; for example, you can use the @ref{d8,,re}
module to search through a memory-mapped file.  You can also change a single
byte by doing @code{obj[index] = 97}, or change a subsequence by assigning to a
slice: @code{obj[i1:i2] = b'...'}.  You can also read and write data starting at
the current file position, and @code{seek()} through the file to different positions.

A memory-mapped file is created by the @ref{511,,mmap} constructor, which is
different on Unix and on Windows.  In either case you must provide a file
descriptor for a file opened for update. If you wish to map an existing Python
file object, use its @code{fileno()} method to obtain the correct value for the
@emph{fileno} parameter.  Otherwise, you can open the file using the
@ref{8ee,,os.open()} function, which returns a file descriptor directly (the file
still needs to be closed when done).

@cartouche
@quotation Note 
If you want to create a memory-mapping for a writable, buffered file, you
should @ref{248d,,flush()} the file first.  This is necessary to ensure
that local modifications to the buffers are actually available to the
mapping.
@end quotation
@end cartouche

For both the Unix and Windows versions of the constructor, @emph{access} may be
specified as an optional keyword parameter. @emph{access} accepts one of four
values: @code{ACCESS_READ}, @code{ACCESS_WRITE}, or @code{ACCESS_COPY} to
specify read-only, write-through or copy-on-write memory respectively, or
@code{ACCESS_DEFAULT} to defer to @emph{prot}.  @emph{access} can be used on both Unix
and Windows.  If @emph{access} is not specified, Windows mmap returns a
write-through mapping.  The initial memory values for all three access types
are taken from the specified file.  Assignment to an @code{ACCESS_READ}
memory map raises a @ref{162,,TypeError} exception.  Assignment to an
@code{ACCESS_WRITE} memory map affects both memory and the underlying file.
Assignment to an @code{ACCESS_COPY} memory map affects memory but does not
update the underlying file.


Changed in version 3.7: Added @code{ACCESS_DEFAULT} constant.


To map anonymous memory, -1 should be passed as the fileno along with the length.

@geindex mmap (class in mmap)
@anchor{library/mmap mmap mmap}@anchor{511}
@deffn {Class} mmap.mmap (fileno, length, tagname=None, access=ACCESS_DEFAULT[, offset])

@strong{(Windows version)} Maps @emph{length} bytes from the file specified by the
file handle @emph{fileno}, and creates a mmap object.  If @emph{length} is larger
than the current size of the file, the file is extended to contain @emph{length}
bytes.  If @emph{length} is @code{0}, the maximum length of the map is the current
size of the file, except that if the file is empty Windows raises an
exception (you cannot create an empty mapping on Windows).

@emph{tagname}, if specified and not @code{None}, is a string giving a tag name for
the mapping.  Windows allows you to have many different mappings against
the same file.  If you specify the name of an existing tag, that tag is
opened, otherwise a new tag of this name is created.  If this parameter is
omitted or @code{None}, the mapping is created without a name.  Avoiding the
use of the tag parameter will assist in keeping your code portable between
Unix and Windows.

@emph{offset} may be specified as a non-negative integer offset. mmap references
will be relative to the offset from the beginning of the file. @emph{offset}
defaults to 0.  @emph{offset} must be a multiple of the @code{ALLOCATIONGRANULARITY}.

Raises an @ref{1130,,auditing event} @code{mmap.__new__} with arguments @code{fileno}, @code{length}, @code{access}, @code{offset}.
@end deffn


@deffn {Class} mmap.mmap (fileno, length, flags=MAP_SHARED, prot=PROT_WRITE|PROT_READ, access=ACCESS_DEFAULT[, offset])

@strong{(Unix version)} Maps @emph{length} bytes from the file specified by the file
descriptor @emph{fileno}, and returns a mmap object.  If @emph{length} is @code{0}, the
maximum length of the map will be the current size of the file when
@ref{511,,mmap} is called.

@emph{flags} specifies the nature of the mapping. @ref{2c1c,,MAP_PRIVATE} creates a
private copy-on-write mapping, so changes to the contents of the mmap
object will be private to this process, and @ref{2c1d,,MAP_SHARED} creates a
mapping that’s shared with all other processes mapping the same areas of
the file.  The default value is @ref{2c1d,,MAP_SHARED}. Some systems have
additional possible flags with the full list specified in
@ref{2c1e,,MAP_* constants}.

@emph{prot}, if specified, gives the desired memory protection; the two most
useful values are @code{PROT_READ} and @code{PROT_WRITE}, to specify
that the pages may be read or written.  @emph{prot} defaults to
@code{PROT_READ | PROT_WRITE}.

@emph{access} may be specified in lieu of @emph{flags} and @emph{prot} as an optional
keyword parameter.  It is an error to specify both @emph{flags}, @emph{prot} and
@emph{access}.  See the description of @emph{access} above for information on how to
use this parameter.

@emph{offset} may be specified as a non-negative integer offset. mmap references
will be relative to the offset from the beginning of the file. @emph{offset}
defaults to 0. @emph{offset} must be a multiple of @code{ALLOCATIONGRANULARITY}
which is equal to @code{PAGESIZE} on Unix systems.

To ensure validity of the created memory mapping the file specified
by the descriptor @emph{fileno} is internally automatically synchronized
with physical backing store on macOS and OpenVMS.

This example shows a simple way of using @ref{511,,mmap}:

@example
import mmap

# write a simple example file
with open("hello.txt", "wb") as f:
    f.write(b"Hello Python!\n")

with open("hello.txt", "r+b") as f:
    # memory-map the file, size 0 means whole file
    mm = mmap.mmap(f.fileno(), 0)
    # read content via standard file methods
    print(mm.readline())  # prints b"Hello Python!\n"
    # read content via slice notation
    print(mm[:5])  # prints b"Hello"
    # update content using slice notation;
    # note that new content must have same size
    mm[6:] = b" world!\n"
    # ... and read again using standard file methods
    mm.seek(0)
    print(mm.readline())  # prints b"Hello  world!\n"
    # close the map
    mm.close()
@end example

@ref{511,,mmap} can also be used as a context manager in a @ref{168,,with}
statement:

@example
import mmap

with mmap.mmap(-1, 13) as mm:
    mm.write(b"Hello world!")
@end example


New in version 3.2: Context manager support.


The next example demonstrates how to create an anonymous map and exchange
data between the parent and child processes:

@example
import mmap
import os

mm = mmap.mmap(-1, 13)
mm.write(b"Hello world!")

pid = os.fork()

if pid == 0:  # In a child process
    mm.seek(0)
    print(mm.readline())

    mm.close()
@end example

Raises an @ref{1130,,auditing event} @code{mmap.__new__} with arguments @code{fileno}, @code{length}, @code{access}, @code{offset}.

Memory-mapped file objects support the following methods:

@geindex close() (mmap.mmap method)
@anchor{library/mmap mmap mmap close}@anchor{2c1f}
@deffn {Method} close ()

Closes the mmap. Subsequent calls to other methods of the object will
result in a ValueError exception being raised. This will not close
the open file.
@end deffn

@geindex closed (mmap.mmap attribute)
@anchor{library/mmap mmap mmap closed}@anchor{2c20}
@deffn {Attribute} closed

@code{True} if the file is closed.


New in version 3.2.

@end deffn

@geindex find() (mmap.mmap method)
@anchor{library/mmap mmap mmap find}@anchor{2c21}
@deffn {Method} find (sub[, start[, end]])

Returns the lowest index in the object where the subsequence @emph{sub} is
found, such that @emph{sub} is contained in the range [@emph{start}, @emph{end}].
Optional arguments @emph{start} and @emph{end} are interpreted as in slice notation.
Returns @code{-1} on failure.


Changed in version 3.5: Writable @ref{87e,,bytes-like object} is now accepted.

@end deffn

@geindex flush() (mmap.mmap method)
@anchor{library/mmap mmap mmap flush}@anchor{5b7}
@deffn {Method} flush ([offset[, size]])

Flushes changes made to the in-memory copy of a file back to disk. Without
use of this call there is no guarantee that changes are written back before
the object is destroyed.  If @emph{offset} and @emph{size} are specified, only
changes to the given range of bytes will be flushed to disk; otherwise, the
whole extent of the mapping is flushed.  @emph{offset} must be a multiple of the
@code{PAGESIZE} or @code{ALLOCATIONGRANULARITY}.

@code{None} is returned to indicate success.  An exception is raised when the
call failed.


Changed in version 3.8: Previously, a nonzero value was returned on success; zero was returned
on error under Windows.  A zero value was returned on success; an
exception was raised on error under Unix.

@end deffn

@geindex madvise() (mmap.mmap method)
@anchor{library/mmap mmap mmap madvise}@anchor{512}
@deffn {Method} madvise (option[, start[, length]])

Send advice @emph{option} to the kernel about the memory region beginning at
@emph{start} and extending @emph{length} bytes.  @emph{option} must be one of the
@ref{2c22,,MADV_* constants} available on the system.  If
@emph{start} and @emph{length} are omitted, the entire mapping is spanned.  On
some systems (including Linux), @emph{start} must be a multiple of the
@code{PAGESIZE}.

Availability: Systems with the @code{madvise()} system call.


New in version 3.8.

@end deffn

@geindex move() (mmap.mmap method)
@anchor{library/mmap mmap mmap move}@anchor{2c23}
@deffn {Method} move (dest, src, count)

Copy the @emph{count} bytes starting at offset @emph{src} to the destination index
@emph{dest}.  If the mmap was created with @code{ACCESS_READ}, then calls to
move will raise a @ref{162,,TypeError} exception.
@end deffn

@geindex read() (mmap.mmap method)
@anchor{library/mmap mmap mmap read}@anchor{c70}
@deffn {Method} read ([n])

Return a @ref{2f0,,bytes} containing up to @emph{n} bytes starting from the
current file position. If the argument is omitted, @code{None} or negative,
return all bytes from the current file position to the end of the
mapping. The file position is updated to point after the bytes that were
returned.


Changed in version 3.3: Argument can be omitted or @code{None}.

@end deffn

@geindex read_byte() (mmap.mmap method)
@anchor{library/mmap mmap mmap read_byte}@anchor{2c24}
@deffn {Method} read_byte ()

Returns a byte at the current file position as an integer, and advances
the file position by 1.
@end deffn

@geindex readline() (mmap.mmap method)
@anchor{library/mmap mmap mmap readline}@anchor{2c25}
@deffn {Method} readline ()

Returns a single line, starting at the current file position and up to the
next newline. The file position is updated to point after the bytes that were
returned.
@end deffn

@geindex resize() (mmap.mmap method)
@anchor{library/mmap mmap mmap resize}@anchor{2c26}
@deffn {Method} resize (newsize)

Resizes the map and the underlying file, if any. If the mmap was created
with @code{ACCESS_READ} or @code{ACCESS_COPY}, resizing the map will
raise a @ref{162,,TypeError} exception.

@strong{On Windows}: Resizing the map will raise an @ref{2a1,,OSError} if there are other
maps against the same named file. Resizing an anonymous map (ie against the
pagefile) will silently create a new map with the original data copied over
up to the length of the new size.


Changed in version 3.11: Correctly fails if attempting to resize when another map is held
Allows resize against an anonymous map on Windows

@end deffn

@geindex rfind() (mmap.mmap method)
@anchor{library/mmap mmap mmap rfind}@anchor{2c27}
@deffn {Method} rfind (sub[, start[, end]])

Returns the highest index in the object where the subsequence @emph{sub} is
found, such that @emph{sub} is contained in the range [@emph{start}, @emph{end}].
Optional arguments @emph{start} and @emph{end} are interpreted as in slice notation.
Returns @code{-1} on failure.


Changed in version 3.5: Writable @ref{87e,,bytes-like object} is now accepted.

@end deffn

@geindex seek() (mmap.mmap method)
@anchor{library/mmap mmap mmap seek}@anchor{2c28}
@deffn {Method} seek (pos[, whence])

Set the file’s current position.  @emph{whence} argument is optional and
defaults to @code{os.SEEK_SET} or @code{0} (absolute file positioning); other
values are @code{os.SEEK_CUR} or @code{1} (seek relative to the current
position) and @code{os.SEEK_END} or @code{2} (seek relative to the file’s end).
@end deffn

@geindex size() (mmap.mmap method)
@anchor{library/mmap mmap mmap size}@anchor{2c29}
@deffn {Method} size ()

Return the length of the file, which can be larger than the size of the
memory-mapped area.
@end deffn

@geindex tell() (mmap.mmap method)
@anchor{library/mmap mmap mmap tell}@anchor{2c2a}
@deffn {Method} tell ()

Returns the current position of the file pointer.
@end deffn

@geindex write() (mmap.mmap method)
@anchor{library/mmap mmap mmap write}@anchor{2c2b}
@deffn {Method} write (bytes)

Write the bytes in @emph{bytes} into memory at the current position of the
file pointer and return the number of bytes written (never less than
@code{len(bytes)}, since if the write fails, a @ref{17c,,ValueError} will be
raised).  The file position is updated to point after the bytes that
were written.  If the mmap was created with @code{ACCESS_READ}, then
writing to it will raise a @ref{162,,TypeError} exception.


Changed in version 3.5: Writable @ref{87e,,bytes-like object} is now accepted.



Changed in version 3.6: The number of bytes written is now returned.

@end deffn

@geindex write_byte() (mmap.mmap method)
@anchor{library/mmap mmap mmap write_byte}@anchor{2c2c}
@deffn {Method} write_byte (byte)

Write the integer @emph{byte} into memory at the current
position of the file pointer; the file position is advanced by @code{1}. If
the mmap was created with @code{ACCESS_READ}, then writing to it will
raise a @ref{162,,TypeError} exception.
@end deffn
@end deffn

@menu
* MADV_* Constants:: 
* MAP_* Constants:: 

@end menu

@node MADV_* Constants,MAP_* Constants,,mmap — Memory-mapped file support
@anchor{library/mmap madv-constants}@anchor{2c2d}@anchor{library/mmap madvise-constants}@anchor{2c22}
@subsubsection MADV_* Constants


@geindex MADV_NORMAL (in module mmap)
@geindex MADV_RANDOM (in module mmap)
@geindex MADV_SEQUENTIAL (in module mmap)
@geindex MADV_WILLNEED (in module mmap)
@geindex MADV_DONTNEED (in module mmap)
@geindex MADV_REMOVE (in module mmap)
@geindex MADV_DONTFORK (in module mmap)
@geindex MADV_DOFORK (in module mmap)
@geindex MADV_HWPOISON (in module mmap)
@geindex MADV_MERGEABLE (in module mmap)
@geindex MADV_UNMERGEABLE (in module mmap)
@geindex MADV_SOFT_OFFLINE (in module mmap)
@geindex MADV_HUGEPAGE (in module mmap)
@geindex MADV_NOHUGEPAGE (in module mmap)
@geindex MADV_DONTDUMP (in module mmap)
@geindex MADV_DODUMP (in module mmap)
@geindex MADV_FREE (in module mmap)
@geindex MADV_NOSYNC (in module mmap)
@geindex MADV_AUTOSYNC (in module mmap)
@geindex MADV_NOCORE (in module mmap)
@geindex MADV_CORE (in module mmap)
@geindex MADV_PROTECT (in module mmap)
@geindex MADV_FREE_REUSABLE (in module mmap)
@geindex MADV_FREE_REUSE (in module mmap)
@anchor{library/mmap mmap MADV_NORMAL}@anchor{2c2e}
@deffn {Data} mmap.MADV_NORMAL
@anchor{library/mmap mmap MADV_RANDOM}@anchor{2c2f}
@deffnx {Data} mmap.MADV_RANDOM
@anchor{library/mmap mmap MADV_SEQUENTIAL}@anchor{2c30}
@deffnx {Data} mmap.MADV_SEQUENTIAL
@anchor{library/mmap mmap MADV_WILLNEED}@anchor{2c31}
@deffnx {Data} mmap.MADV_WILLNEED
@anchor{library/mmap mmap MADV_DONTNEED}@anchor{2c32}
@deffnx {Data} mmap.MADV_DONTNEED
@anchor{library/mmap mmap MADV_REMOVE}@anchor{2c33}
@deffnx {Data} mmap.MADV_REMOVE
@anchor{library/mmap mmap MADV_DONTFORK}@anchor{2c34}
@deffnx {Data} mmap.MADV_DONTFORK
@anchor{library/mmap mmap MADV_DOFORK}@anchor{2c35}
@deffnx {Data} mmap.MADV_DOFORK
@anchor{library/mmap mmap MADV_HWPOISON}@anchor{2c36}
@deffnx {Data} mmap.MADV_HWPOISON
@anchor{library/mmap mmap MADV_MERGEABLE}@anchor{2c37}
@deffnx {Data} mmap.MADV_MERGEABLE
@anchor{library/mmap mmap MADV_UNMERGEABLE}@anchor{2c38}
@deffnx {Data} mmap.MADV_UNMERGEABLE
@anchor{library/mmap mmap MADV_SOFT_OFFLINE}@anchor{2c39}
@deffnx {Data} mmap.MADV_SOFT_OFFLINE
@anchor{library/mmap mmap MADV_HUGEPAGE}@anchor{2c3a}
@deffnx {Data} mmap.MADV_HUGEPAGE
@anchor{library/mmap mmap MADV_NOHUGEPAGE}@anchor{2c3b}
@deffnx {Data} mmap.MADV_NOHUGEPAGE
@anchor{library/mmap mmap MADV_DONTDUMP}@anchor{2c3c}
@deffnx {Data} mmap.MADV_DONTDUMP
@anchor{library/mmap mmap MADV_DODUMP}@anchor{2c3d}
@deffnx {Data} mmap.MADV_DODUMP
@anchor{library/mmap mmap MADV_FREE}@anchor{2c3e}
@deffnx {Data} mmap.MADV_FREE
@anchor{library/mmap mmap MADV_NOSYNC}@anchor{2c3f}
@deffnx {Data} mmap.MADV_NOSYNC
@anchor{library/mmap mmap MADV_AUTOSYNC}@anchor{2c40}
@deffnx {Data} mmap.MADV_AUTOSYNC
@anchor{library/mmap mmap MADV_NOCORE}@anchor{2c41}
@deffnx {Data} mmap.MADV_NOCORE
@anchor{library/mmap mmap MADV_CORE}@anchor{2c42}
@deffnx {Data} mmap.MADV_CORE
@anchor{library/mmap mmap MADV_PROTECT}@anchor{2c43}
@deffnx {Data} mmap.MADV_PROTECT
@anchor{library/mmap mmap MADV_FREE_REUSABLE}@anchor{2c44}
@deffnx {Data} mmap.MADV_FREE_REUSABLE
@anchor{library/mmap mmap MADV_FREE_REUSE}@anchor{2c45}
@deffnx {Data} mmap.MADV_FREE_REUSE

These options can be passed to @ref{512,,mmap.madvise()}.  Not every option will
be present on every system.

Availability: Systems with the madvise() system call.


New in version 3.8.

@end deffn

@node MAP_* Constants,,MADV_* Constants,mmap — Memory-mapped file support
@anchor{library/mmap id1}@anchor{2c46}@anchor{library/mmap map-constants}@anchor{2c1e}
@subsubsection MAP_* Constants


@geindex MAP_SHARED (in module mmap)
@geindex MAP_PRIVATE (in module mmap)
@geindex MAP_DENYWRITE (in module mmap)
@geindex MAP_EXECUTABLE (in module mmap)
@geindex MAP_ANON (in module mmap)
@geindex MAP_ANONYMOUS (in module mmap)
@geindex MAP_POPULATE (in module mmap)
@geindex MAP_STACK (in module mmap)
@anchor{library/mmap mmap MAP_SHARED}@anchor{2c1d}
@deffn {Data} mmap.MAP_SHARED
@anchor{library/mmap mmap MAP_PRIVATE}@anchor{2c1c}
@deffnx {Data} mmap.MAP_PRIVATE
@anchor{library/mmap mmap MAP_DENYWRITE}@anchor{2c47}
@deffnx {Data} mmap.MAP_DENYWRITE
@anchor{library/mmap mmap MAP_EXECUTABLE}@anchor{2c48}
@deffnx {Data} mmap.MAP_EXECUTABLE
@anchor{library/mmap mmap MAP_ANON}@anchor{2c49}
@deffnx {Data} mmap.MAP_ANON
@anchor{library/mmap mmap MAP_ANONYMOUS}@anchor{2c4a}
@deffnx {Data} mmap.MAP_ANONYMOUS
@anchor{library/mmap mmap MAP_POPULATE}@anchor{11cd}
@deffnx {Data} mmap.MAP_POPULATE
@anchor{library/mmap mmap MAP_STACK}@anchor{2c4b}
@deffnx {Data} mmap.MAP_STACK

These are the various flags that can be passed to @ref{511,,mmap.mmap()}. Note that some options might not be present on some systems.


Changed in version 3.10: Added MAP_POPULATE constant.



New in version 3.11: Added MAP_STACK constant.

@end deffn

@node Internet Data Handling,Structured Markup Processing Tools,Networking and Interprocess Communication,The Python Standard Library
@anchor{library/netdata doc}@anchor{2c4c}@anchor{library/netdata internet-data-handling}@anchor{2c4d}@anchor{library/netdata netdata}@anchor{2c4e}
@section Internet Data Handling


This chapter describes modules which support handling data formats commonly used
on the internet.

@menu
* email — An email and MIME handling package:: 
* json — JSON encoder and decoder:: 
* mailcap — Mailcap file handling:: 
* mailbox — Manipulate mailboxes in various formats:: 
* mimetypes — Map filenames to MIME types:: 
* base64 — Base16@comma{} Base32@comma{} Base64@comma{} Base85 Data Encodings: base64 — Base16 Base32 Base64 Base85 Data Encodings. 
* binascii — Convert between binary and ASCII:: 
* quopri — Encode and decode MIME quoted-printable data:: 
* uu — Encode and decode uuencode files:: 

@end menu

@node email — An email and MIME handling package,json — JSON encoder and decoder,,Internet Data Handling
@anchor{library/email doc}@anchor{2c4f}@anchor{library/email email-an-email-and-mime-handling-package}@anchor{2c50}@anchor{library/email module-email}@anchor{64}
@subsection @code{email} — An email and MIME handling package


@geindex module; email

@strong{Source code:} Lib/email/__init__.py@footnote{https://github.com/python/cpython/tree/main/Lib/email/__init__.py}


__________________________________________________________________


The @ref{64,,email} package is a library for managing email messages.  It is
specifically @emph{not} designed to do any sending of email messages to SMTP
(
@geindex RFC; RFC 2821
RFC 2821@footnote{https://tools.ietf.org/html/rfc2821.html}), NNTP, or other servers; those are functions of modules such as
@ref{e8,,smtplib} and @ref{bc,,nntplib}.  The @ref{64,,email} package attempts to be as
RFC-compliant as possible, supporting 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html} and 
@geindex RFC; RFC 6532
RFC 6532@footnote{https://tools.ietf.org/html/rfc6532.html}, as well as
such MIME-related RFCs as 
@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html}, 
@geindex RFC; RFC 2046
RFC 2046@footnote{https://tools.ietf.org/html/rfc2046.html}, 
@geindex RFC; RFC 2047
RFC 2047@footnote{https://tools.ietf.org/html/rfc2047.html}, 
@geindex RFC; RFC 2183
RFC 2183@footnote{https://tools.ietf.org/html/rfc2183.html},
and 
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html}.

The overall structure of the email package can be divided into three major
components, plus a fourth component that controls the behavior of the other
components.

The central component of the package is an “object model” that represents email
messages.  An application interacts with the package primarily through the
object model interface defined in the @ref{6d,,message} sub-module.  The
application can use this API to ask questions about an existing email, to
construct a new email, or to add or remove email subcomponents that themselves
use the same object model interface.  That is, following the nature of email
messages and their MIME subcomponents, the email object model is a tree
structure of objects that all provide the @ref{7e4,,EmailMessage}
API.

The other two major components of the package are the @ref{6f,,parser} and
the @ref{69,,generator}.  The parser takes the serialized version of an
email message (a stream of bytes) and converts it into a tree of
@ref{7e4,,EmailMessage} objects.  The generator takes an
@ref{7e4,,EmailMessage} and turns it back into a serialized byte
stream.  (The parser and generator also handle streams of text characters, but
this usage is discouraged as it is too easy to end up with messages that are
not valid in one way or another.)

The control component is the @ref{70,,policy} module.  Every
@ref{7e4,,EmailMessage}, every @ref{69,,generator}, and every
@ref{6f,,parser} has an associated @ref{70,,policy} object that
controls its behavior.  Usually an application only needs to specify the policy
when an @ref{7e4,,EmailMessage} is created, either by directly
instantiating an @ref{7e4,,EmailMessage}  to create a new email,
or by parsing an input stream using a @ref{6f,,parser}.  But the policy can
be changed when the message is serialized using a @ref{69,,generator}.
This allows, for example, a generic email message to be parsed from disk, but
to serialize it using standard SMTP settings when sending it to an email
server.

The email package does its best to hide the details of the various governing
RFCs from the application.  Conceptually the application should be able to
treat the email message as a structured tree of unicode text and binary
attachments, without having to worry about how these are represented when
serialized.  In practice, however, it is often necessary to be aware of at
least some of the rules governing MIME messages and their structure,
specifically the names and nature of the MIME “content types” and how they
identify multipart documents.  For the most part this knowledge should only be
required for more complex applications, and even then it should only be the
high level structure in question, and not the details of how those structures
are represented.  Since MIME content types are used widely in modern internet
software (not just email), this will be a familiar concept to many programmers.

The following sections describe the functionality of the @ref{64,,email} package.
We start with the @ref{6d,,message} object model, which is the primary
interface an application will use, and follow that with the
@ref{6f,,parser} and @ref{69,,generator} components.  Then we cover the
@ref{70,,policy} controls, which completes the treatment of the main
components of the library.

The next three sections cover the exceptions the package may raise and the
defects (non-compliance with the RFCs) that the @ref{6f,,parser} may
detect.  Then we cover the @ref{6b,,headerregistry} and the
@ref{66,,contentmanager} sub-components, which provide tools for doing more
detailed manipulation of headers and payloads, respectively.  Both of these
components contain features relevant to consuming and producing non-trivial
messages, but also document their extensibility APIs, which will be of interest
to advanced applications.

Following those is a set of examples of using the fundamental parts of the APIs
covered in the preceding sections.

The foregoing represent the modern (unicode friendly) API of the email package.
The remaining sections, starting with the @ref{7e3,,Message}
class, cover the legacy @ref{7e2,,compat32} API that deals much more
directly with the details of how email messages are represented.  The
@ref{7e2,,compat32} API does @emph{not} hide the details of the RFCs from
the application, but for applications that need to operate at that level, they
can be useful tools.  This documentation is also relevant for applications that
are still using the @ref{7e2,,compat32} API for backward
compatibility reasons.


Changed in version 3.6: Docs reorganized and rewritten to promote the new
@ref{7e4,,EmailMessage}/@ref{c48,,EmailPolicy}
API.


Contents of the @ref{64,,email} package documentation:

@menu
* email.message; Representing an email message: email message Representing an email message. 
* email.parser; Parsing email messages: email parser Parsing email messages. 
* email.generator; Generating MIME documents: email generator Generating MIME documents. 
* email.policy; Policy Objects: email policy Policy Objects. 
* email.errors; Exception and Defect classes: email errors Exception and Defect classes. 
* email.headerregistry; Custom Header Objects: email headerregistry Custom Header Objects. 
* email.contentmanager; Managing MIME Content: email contentmanager Managing MIME Content. 
* email; Examples: email Examples. 
* email.message.Message; Representing an email message using the compat32 API: email message Message Representing an email message using the compat32 API. 
* email.mime; Creating email and MIME objects from scratch: email mime Creating email and MIME objects from scratch. 
* email.header; Internationalized headers: email header Internationalized headers. 
* email.charset; Representing character sets: email charset Representing character sets. 
* email.encoders; Encoders: email encoders Encoders. 
* email.utils; Miscellaneous utilities: email utils Miscellaneous utilities. 
* email.iterators; Iterators: email iterators Iterators. 

@end menu

@node email message Representing an email message,email parser Parsing email messages,,email — An email and MIME handling package
@anchor{library/email message doc}@anchor{2c51}@anchor{library/email message email-message-representing-an-email-message}@anchor{2c52}@anchor{library/email message module-email message}@anchor{6d}
@subsubsection @code{email.message}: Representing an email message


@geindex module; email.message

@strong{Source code:} Lib/email/message.py@footnote{https://github.com/python/cpython/tree/main/Lib/email/message.py}


__________________________________________________________________



New in version 3.6: @footnote{@w{(1)} 
Originally added in 3.4 as a @ref{bd6,,provisional module}.  Docs for legacy message class moved to
@ref{2c53,,email.message.Message; Representing an email message using the compat32 API}.
}


The central class in the @ref{64,,email} package is the @ref{7e4,,EmailMessage}
class, imported from the @ref{6d,,email.message} module.  It is the base class for
the @ref{64,,email} object model.  @ref{7e4,,EmailMessage} provides the core
functionality for setting and querying header fields, for accessing message
bodies, and for creating or modifying structured messages.

An email message consists of @emph{headers} and a @emph{payload} (which is also referred
to as the @emph{content}).  Headers are 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html} or 
@geindex RFC; RFC 6532
RFC 6532@footnote{https://tools.ietf.org/html/rfc6532.html} style field names
and values, where the field name and value are separated by a colon.  The colon
is not part of either the field name or the field value.  The payload may be a
simple text message, or a binary object, or a structured sequence of
sub-messages each with their own set of headers and their own payload.  The
latter type of payload is indicated by the message having a MIME type such as
@code{multipart/*} or @code{message/rfc822}.

The conceptual model provided by an @ref{7e4,,EmailMessage} object is that of an
ordered dictionary of headers coupled with a @emph{payload} that represents the
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html} body of the message, which might be a list of sub-@code{EmailMessage}
objects.  In addition to the normal dictionary methods for accessing the header
names and values, there are methods for accessing specialized information from
the headers (for example the MIME content type), for operating on the payload,
for generating a serialized version of the message, and for recursively walking
over the object tree.

The @ref{7e4,,EmailMessage} dictionary-like interface is indexed by the header
names, which must be ASCII values.  The values of the dictionary are strings
with some extra methods.  Headers are stored and returned in case-preserving
form, but field names are matched case-insensitively.  Unlike a real dict,
there is an ordering to the keys, and there can be duplicate keys.  Additional
methods are provided for working with headers that have duplicate keys.

The @emph{payload} is either a string or bytes object, in the case of simple message
objects, or a list of @ref{7e4,,EmailMessage} objects, for MIME container
documents such as @code{multipart/*} and @code{message/rfc822}
message objects.

@geindex EmailMessage (class in email.message)
@anchor{library/email message email message EmailMessage}@anchor{7e4}
@deffn {Class} email.message.EmailMessage (policy=default)

If @emph{policy} is specified use the rules it specifies to update and serialize
the representation of the message.  If @emph{policy} is not set, use the
@ref{2c54,,default} policy, which follows the rules of the email
RFCs except for line endings (instead of the RFC mandated @code{\r\n}, it uses
the Python standard @code{\n} line endings).  For more information see the
@ref{70,,policy} documentation.

@geindex as_string() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage as_string}@anchor{2c55}
@deffn {Method} as_string (unixfrom=False, maxheaderlen=None, policy=None)

Return the entire message flattened as a string.  When optional
@emph{unixfrom} is true, the envelope header is included in the returned
string.  @emph{unixfrom} defaults to @code{False}.  For backward compatibility
with the base @ref{7e3,,Message} class @emph{maxheaderlen} is
accepted, but defaults to @code{None}, which means that by default the line
length is controlled by the
@code{max_line_length} of the policy.  The
@emph{policy} argument may be used to override the default policy obtained
from the message instance.  This can be used to control some of the
formatting produced by the method, since the specified @emph{policy} will be
passed to the @ref{d72,,Generator}.

Flattening the message may trigger changes to the @ref{7e4,,EmailMessage}
if defaults need to be filled in to complete the transformation to a
string (for example, MIME boundaries may be generated or modified).

Note that this method is provided as a convenience and may not be the
most useful way to serialize messages in your application, especially if
you are dealing with multiple messages.  See
@ref{d72,,email.generator.Generator} for a more flexible API for
serializing messages.  Note also that this method is restricted to
producing messages serialized as “7 bit clean” when
@ref{957,,utf8} is @code{False}, which is the default.


Changed in version 3.6: the default behavior when @emph{maxheaderlen}
is not specified was changed from defaulting to 0 to defaulting
to the value of @emph{max_line_length} from the policy.

@end deffn

@geindex __str__() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage __str__}@anchor{2c56}
@deffn {Method} __str__ ()

Equivalent to @code{as_string(policy=self.policy.clone(utf8=True))}.  Allows
@code{str(msg)} to produce a string containing the serialized message in a
readable format.


Changed in version 3.4: the method was changed to use @code{utf8=True},
thus producing an 
@geindex RFC; RFC 6531
RFC 6531@footnote{https://tools.ietf.org/html/rfc6531.html}-like message representation, instead of
being a direct alias for @ref{2c55,,as_string()}.

@end deffn

@geindex as_bytes() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage as_bytes}@anchor{2c57}
@deffn {Method} as_bytes (unixfrom=False, policy=None)

Return the entire message flattened as a bytes object.  When optional
@emph{unixfrom} is true, the envelope header is included in the returned
string.  @emph{unixfrom} defaults to @code{False}.  The @emph{policy} argument may be
used to override the default policy obtained from the message instance.
This can be used to control some of the formatting produced by the
method, since the specified @emph{policy} will be passed to the
@ref{d73,,BytesGenerator}.

Flattening the message may trigger changes to the @ref{7e4,,EmailMessage}
if defaults need to be filled in to complete the transformation to a
string (for example, MIME boundaries may be generated or modified).

Note that this method is provided as a convenience and may not be the
most useful way to serialize messages in your application, especially if
you are dealing with multiple messages.  See
@ref{d73,,email.generator.BytesGenerator} for a more flexible API for
serializing messages.
@end deffn

@geindex __bytes__() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage __bytes__}@anchor{2c58}
@deffn {Method} __bytes__ ()

Equivalent to @ref{2c57,,as_bytes()}.  Allows @code{bytes(msg)} to produce a
bytes object containing the serialized message.
@end deffn

@geindex is_multipart() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage is_multipart}@anchor{2c59}
@deffn {Method} is_multipart ()

Return @code{True} if the message’s payload is a list of
sub-@ref{7e4,,EmailMessage} objects, otherwise return @code{False}.  When
@ref{2c59,,is_multipart()} returns @code{False}, the payload should be a string
object (which might be a CTE encoded binary payload).  Note that
@ref{2c59,,is_multipart()} returning @code{True} does not necessarily mean that
“msg.get_content_maintype() == ‘multipart’” will return the @code{True}.
For example, @code{is_multipart} will return @code{True} when the
@ref{7e4,,EmailMessage} is of type @code{message/rfc822}.
@end deffn

@geindex set_unixfrom() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage set_unixfrom}@anchor{2c5a}
@deffn {Method} set_unixfrom (unixfrom)

Set the message’s envelope header to @emph{unixfrom}, which should be a
string.  (See @ref{2c5b,,mboxMessage} for a brief description of
this header.)
@end deffn

@geindex get_unixfrom() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage get_unixfrom}@anchor{2c5c}
@deffn {Method} get_unixfrom ()

Return the message’s envelope header.  Defaults to @code{None} if the
envelope header was never set.
@end deffn

The following methods implement the mapping-like interface for accessing the
message’s headers.  Note that there are some semantic differences
between these methods and a normal mapping (i.e. dictionary) interface.  For
example, in a dictionary there are no duplicate keys, but here there may be
duplicate message headers.  Also, in dictionaries there is no guaranteed
order to the keys returned by @ref{2c5d,,keys()}, but in an @ref{7e4,,EmailMessage}
object, headers are always returned in the order they appeared in the
original message, or in which they were added to the message later.  Any
header deleted and then re-added is always appended to the end of the
header list.

These semantic differences are intentional and are biased toward
convenience in the most common use cases.

Note that in all cases, any envelope header present in the message is not
included in the mapping interface.

@geindex __len__() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage __len__}@anchor{2c5e}
@deffn {Method} __len__ ()

Return the total number of headers, including duplicates.
@end deffn

@geindex __contains__() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage __contains__}@anchor{2c5f}
@deffn {Method} __contains__ (name)

Return @code{True} if the message object has a field named @emph{name}. Matching is
done without regard to case and @emph{name} does not include the trailing
colon.  Used for the @code{in} operator.  For example:

@example
if 'message-id' in myMessage:
   print('Message-ID:', myMessage['message-id'])
@end example
@end deffn

@geindex __getitem__() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage __getitem__}@anchor{2c60}
@deffn {Method} __getitem__ (name)

Return the value of the named header field.  @emph{name} does not include the
colon field separator.  If the header is missing, @code{None} is returned; a
@ref{5bb,,KeyError} is never raised.

Note that if the named field appears more than once in the message’s
headers, exactly which of those field values will be returned is
undefined.  Use the @ref{2c61,,get_all()} method to get the values of all the
extant headers named @emph{name}.

Using the standard (non-@code{compat32}) policies, the returned value is an
instance of a subclass of @ref{2c62,,email.headerregistry.BaseHeader}.
@end deffn

@geindex __setitem__() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage __setitem__}@anchor{2c63}
@deffn {Method} __setitem__ (name, val)

Add a header to the message with field name @emph{name} and value @emph{val}.  The
field is appended to the end of the message’s existing headers.

Note that this does @emph{not} overwrite or delete any existing header with the same
name.  If you want to ensure that the new header is the only one present in the
message with field name @emph{name}, delete the field first, e.g.:

@example
del msg['subject']
msg['subject'] = 'Python roolz!'
@end example

If the @code{policy} defines certain headers to be unique (as the standard
policies do), this method may raise a @ref{17c,,ValueError} when an attempt
is made to assign a value to such a header when one already exists.  This
behavior is intentional for consistency’s sake, but do not depend on it
as we may choose to make such assignments do an automatic deletion of the
existing header in the future.
@end deffn

@geindex __delitem__() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage __delitem__}@anchor{2c64}
@deffn {Method} __delitem__ (name)

Delete all occurrences of the field with name @emph{name} from the message’s
headers.  No exception is raised if the named field isn’t present in the
headers.
@end deffn

@geindex keys() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage keys}@anchor{2c5d}
@deffn {Method} keys ()

Return a list of all the message’s header field names.
@end deffn

@geindex values() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage values}@anchor{2c65}
@deffn {Method} values ()

Return a list of all the message’s field values.
@end deffn

@geindex items() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage items}@anchor{2c66}
@deffn {Method} items ()

Return a list of 2-tuples containing all the message’s field headers and
values.
@end deffn

@geindex get() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage get}@anchor{2c67}
@deffn {Method} get (name, failobj=None)

Return the value of the named header field.  This is identical to
@ref{2c60,,__getitem__()} except that optional @emph{failobj} is returned if the
named header is missing (@emph{failobj} defaults to @code{None}).
@end deffn

Here are some additional useful header related methods:

@geindex get_all() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage get_all}@anchor{2c61}
@deffn {Method} get_all (name, failobj=None)

Return a list of all the values for the field named @emph{name}. If there are
no such named headers in the message, @emph{failobj} is returned (defaults to
@code{None}).
@end deffn

@geindex add_header() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage add_header}@anchor{2c68}
@deffn {Method} add_header (_name, _value, **_params)

Extended header setting.  This method is similar to @ref{2c63,,__setitem__()}
except that additional header parameters can be provided as keyword
arguments.  @emph{_name} is the header field to add and @emph{_value} is the
@emph{primary} value for the header.

For each item in the keyword argument dictionary @emph{_params}, the key is
taken as the parameter name, with underscores converted to dashes (since
dashes are illegal in Python identifiers).  Normally, the parameter will
be added as @code{key="value"} unless the value is @code{None}, in which case
only the key will be added.

If the value contains non-ASCII characters, the charset and language may
be explicitly controlled by specifying the value as a three tuple in the
format @code{(CHARSET, LANGUAGE, VALUE)}, where @code{CHARSET} is a string
naming the charset to be used to encode the value, @code{LANGUAGE} can
usually be set to @code{None} or the empty string (see 
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html} for other
possibilities), and @code{VALUE} is the string value containing non-ASCII
code points.  If a three tuple is not passed and the value contains
non-ASCII characters, it is automatically encoded in 
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html} format
using a @code{CHARSET} of @code{utf-8} and a @code{LANGUAGE} of @code{None}.

Here is an example:

@example
msg.add_header('Content-Disposition', 'attachment', filename='bud.gif')
@end example

This will add a header that looks like

@example
Content-Disposition: attachment; filename="bud.gif"
@end example

An example of the extended interface with non-ASCII characters:

@example
msg.add_header('Content-Disposition', 'attachment',
               filename=('iso-8859-1', '@w{'}, 'Fußballer.ppt'))
@end example
@end deffn

@geindex replace_header() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage replace_header}@anchor{2c69}
@deffn {Method} replace_header (_name, _value)

Replace a header.  Replace the first header found in the message that
matches @emph{_name}, retaining header order and field name case of the
original header.  If no matching header is found, raise a
@ref{5bb,,KeyError}.
@end deffn

@geindex get_content_type() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage get_content_type}@anchor{2c6a}
@deffn {Method} get_content_type ()

Return the message’s content type, coerced to lower case of the form
@code{maintype/subtype}.  If there is no @code{Content-Type}
header in the message return the value returned by
@ref{2c6b,,get_default_type()}.  If the @code{Content-Type} header is
invalid, return @code{text/plain}.

(According to 
@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html}, messages always have a default type,
@ref{2c6a,,get_content_type()} will always return a value.  
@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html} defines
a message’s default type to be @code{text/plain} unless it appears
inside a @code{multipart/digest} container, in which case it would
be @code{message/rfc822}.  If the @code{Content-Type} header
has an invalid type specification, 
@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html} mandates that the default
type be @code{text/plain}.)
@end deffn

@geindex get_content_maintype() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage get_content_maintype}@anchor{2c6c}
@deffn {Method} get_content_maintype ()

Return the message’s main content type.  This is the @code{maintype}
part of the string returned by @ref{2c6a,,get_content_type()}.
@end deffn

@geindex get_content_subtype() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage get_content_subtype}@anchor{2c6d}
@deffn {Method} get_content_subtype ()

Return the message’s sub-content type.  This is the @code{subtype}
part of the string returned by @ref{2c6a,,get_content_type()}.
@end deffn

@geindex get_default_type() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage get_default_type}@anchor{2c6b}
@deffn {Method} get_default_type ()

Return the default content type.  Most messages have a default content
type of @code{text/plain}, except for messages that are subparts of
@code{multipart/digest} containers.  Such subparts have a default
content type of @code{message/rfc822}.
@end deffn

@geindex set_default_type() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage set_default_type}@anchor{2c6e}
@deffn {Method} set_default_type (ctype)

Set the default content type.  @emph{ctype} should either be
@code{text/plain} or @code{message/rfc822}, although this is
not enforced.  The default content type is not stored in the
@code{Content-Type} header, so it only affects the return value of
the @code{get_content_type} methods when no @code{Content-Type}
header is present in the message.
@end deffn

@geindex set_param() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage set_param}@anchor{2c6f}
@deffn {Method} set_param (param, value, header='Content-Type', requote=True, charset=None, language='@w{'}, replace=False)

Set a parameter in the @code{Content-Type} header.  If the
parameter already exists in the header, replace its value with @emph{value}.
When @emph{header} is @code{Content-Type} (the default) and the header does not
yet exist in the message, add it, set its value to
@code{text/plain}, and append the new parameter value.  Optional
@emph{header} specifies an alternative header to @code{Content-Type}.

If the value contains non-ASCII characters, the charset and language may
be explicitly specified using the optional @emph{charset} and @emph{language}
parameters.  Optional @emph{language} specifies the 
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html} language,
defaulting to the empty string.  Both @emph{charset} and @emph{language} should be
strings.  The default is to use the @code{utf8} @emph{charset} and @code{None} for
the @emph{language}.

If @emph{replace} is @code{False} (the default) the header is moved to the
end of the list of headers.  If @emph{replace} is @code{True}, the header
will be updated in place.

Use of the @emph{requote} parameter with @ref{7e4,,EmailMessage} objects is
deprecated.

Note that existing parameter values of headers may be accessed through
the @code{params} attribute of the
header value (for example, @code{msg['Content-Type'].params['charset']}).


Changed in version 3.4: @code{replace} keyword was added.

@end deffn

@geindex del_param() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage del_param}@anchor{2c70}
@deffn {Method} del_param (param, header='content-type', requote=True)

Remove the given parameter completely from the @code{Content-Type}
header.  The header will be re-written in place without the parameter or
its value.  Optional @emph{header} specifies an alternative to
@code{Content-Type}.

Use of the @emph{requote} parameter with @ref{7e4,,EmailMessage} objects is
deprecated.
@end deffn

@geindex get_filename() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage get_filename}@anchor{2c71}
@deffn {Method} get_filename (failobj=None)

Return the value of the @code{filename} parameter of the
@code{Content-Disposition} header of the message.  If the header
does not have a @code{filename} parameter, this method falls back to looking
for the @code{name} parameter on the @code{Content-Type} header.  If
neither is found, or the header is missing, then @emph{failobj} is returned.
The returned string will always be unquoted as per
@ref{2c72,,email.utils.unquote()}.
@end deffn

@geindex get_boundary() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage get_boundary}@anchor{2c73}
@deffn {Method} get_boundary (failobj=None)

Return the value of the @code{boundary} parameter of the
@code{Content-Type} header of the message, or @emph{failobj} if either
the header is missing, or has no @code{boundary} parameter.  The returned
string will always be unquoted as per @ref{2c72,,email.utils.unquote()}.
@end deffn

@geindex set_boundary() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage set_boundary}@anchor{2c74}
@deffn {Method} set_boundary (boundary)

Set the @code{boundary} parameter of the @code{Content-Type} header to
@emph{boundary}.  @ref{2c74,,set_boundary()} will always quote @emph{boundary} if
necessary.  A @ref{2c75,,HeaderParseError} is raised if the
message object has no @code{Content-Type} header.

Note that using this method is subtly different from deleting the old
@code{Content-Type} header and adding a new one with the new
boundary via @ref{2c68,,add_header()}, because @ref{2c74,,set_boundary()} preserves
the order of the @code{Content-Type} header in the list of
headers.
@end deffn

@geindex get_content_charset() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage get_content_charset}@anchor{2c76}
@deffn {Method} get_content_charset (failobj=None)

Return the @code{charset} parameter of the @code{Content-Type} header,
coerced to lower case.  If there is no @code{Content-Type} header, or if
that header has no @code{charset} parameter, @emph{failobj} is returned.
@end deffn

@geindex get_charsets() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage get_charsets}@anchor{2c77}
@deffn {Method} get_charsets (failobj=None)

Return a list containing the character set names in the message.  If the
message is a @code{multipart}, then the list will contain one element
for each subpart in the payload, otherwise, it will be a list of length 1.

Each item in the list will be a string which is the value of the
@code{charset} parameter in the @code{Content-Type} header for the
represented subpart.  If the subpart has no @code{Content-Type}
header, no @code{charset} parameter, or is not of the @code{text} main
MIME type, then that item in the returned list will be @emph{failobj}.
@end deffn

@geindex is_attachment() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage is_attachment}@anchor{2c78}
@deffn {Method} is_attachment ()

Return @code{True} if there is a @code{Content-Disposition} header
and its (case insensitive) value is @code{attachment}, @code{False} otherwise.


Changed in version 3.4.2: is_attachment is now a method instead of a property, for consistency
with @ref{2c79,,is_multipart()}.

@end deffn

@geindex get_content_disposition() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage get_content_disposition}@anchor{2c7a}
@deffn {Method} get_content_disposition ()

Return the lowercased value (without parameters) of the message’s
@code{Content-Disposition} header if it has one, or @code{None}.  The
possible values for this method are @emph{inline}, @emph{attachment} or @code{None}
if the message follows 
@geindex RFC; RFC 2183
RFC 2183@footnote{https://tools.ietf.org/html/rfc2183.html}.


New in version 3.5.

@end deffn

The following methods relate to interrogating and manipulating the content
(payload) of the message.

@geindex walk() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage walk}@anchor{2c7b}
@deffn {Method} walk ()

The @ref{2c7b,,walk()} method is an all-purpose generator which can be used to
iterate over all the parts and subparts of a message object tree, in
depth-first traversal order.  You will typically use @ref{2c7b,,walk()} as the
iterator in a @code{for} loop; each iteration returns the next subpart.

Here’s an example that prints the MIME type of every part of a multipart
message structure:

@c from email import message_from_binary_file
@c with open('../Lib/test/test_email/data/msg_16.txt', 'rb') as f:
@c     msg = message_from_binary_file(f)

@example
>>> for part in msg.walk():
...     print(part.get_content_type())
multipart/report
text/plain
message/delivery-status
text/plain
text/plain
message/rfc822
text/plain
@end example

@code{walk} iterates over the subparts of any part where
@ref{2c59,,is_multipart()} returns @code{True}, even though
@code{msg.get_content_maintype() == 'multipart'} may return @code{False}.  We
can see this in our example by making use of the @code{_structure} debug
helper function:

@example
>>> from email.iterators import _structure
>>> for part in msg.walk():
...     print(part.get_content_maintype() == 'multipart',
...           part.is_multipart())
True True
False False
False True
False False
False False
False True
False False
>>> _structure(msg)
multipart/report
    text/plain
    message/delivery-status
        text/plain
        text/plain
    message/rfc822
        text/plain
@end example

Here the @code{message} parts are not @code{multiparts}, but they do contain
subparts. @code{is_multipart()} returns @code{True} and @code{walk} descends
into the subparts.
@end deffn

@geindex get_body() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage get_body}@anchor{2c7c}
@deffn {Method} get_body (preferencelist=('related', 'html', 'plain'))

Return the MIME part that is the best candidate to be the “body” of the
message.

@emph{preferencelist} must be a sequence of strings from the set @code{related},
@code{html}, and @code{plain}, and indicates the order of preference for the
content type of the part returned.

Start looking for candidate matches with the object on which the
@code{get_body} method is called.

If @code{related} is not included in @emph{preferencelist}, consider the root
part (or subpart of the root part) of any related encountered as a
candidate if the (sub-)part matches a preference.

When encountering a @code{multipart/related}, check the @code{start} parameter
and if a part with a matching @code{Content-ID} is found, consider
only it when looking for candidate matches.  Otherwise consider only the
first (default root) part of the @code{multipart/related}.

If a part has a @code{Content-Disposition} header, only consider
the part a candidate match if the value of the header is @code{inline}.

If none of the candidates matches any of the preferences in
@emph{preferencelist}, return @code{None}.

Notes: (1) For most applications the only @emph{preferencelist} combinations
that really make sense are @code{('plain',)}, @code{('html', 'plain')}, and the
default @code{('related', 'html', 'plain')}.  (2) Because matching starts
with the object on which @code{get_body} is called, calling @code{get_body} on
a @code{multipart/related} will return the object itself unless
@emph{preferencelist} has a non-default value. (3) Messages (or message parts)
that do not specify a @code{Content-Type} or whose
@code{Content-Type} header is invalid will be treated as if they
are of type @code{text/plain}, which may occasionally cause @code{get_body} to
return unexpected results.
@end deffn

@geindex iter_attachments() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage iter_attachments}@anchor{2c7d}
@deffn {Method} iter_attachments ()

Return an iterator over all of the immediate sub-parts of the message
that are not candidate “body” parts.  That is, skip the first occurrence
of each of @code{text/plain}, @code{text/html}, @code{multipart/related}, or
@code{multipart/alternative} (unless they are explicitly marked as
attachments via @code{Content-Disposition: attachment}), and
return all remaining parts.  When applied directly to a
@code{multipart/related}, return an iterator over the all the related parts
except the root part (ie: the part pointed to by the @code{start} parameter,
or the first part if there is no @code{start} parameter or the @code{start}
parameter doesn’t match the @code{Content-ID} of any of the
parts).  When applied directly to a @code{multipart/alternative} or a
non-@code{multipart}, return an empty iterator.
@end deffn

@geindex iter_parts() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage iter_parts}@anchor{2c7e}
@deffn {Method} iter_parts ()

Return an iterator over all of the immediate sub-parts of the message,
which will be empty for a non-@code{multipart}.  (See also
@ref{2c7b,,walk()}.)
@end deffn

@geindex get_content() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage get_content}@anchor{2c7f}
@deffn {Method} get_content (*args, content_manager=None, **kw)

Call the @ref{2c80,,get_content()} method
of the @emph{content_manager}, passing self as the message object, and passing
along any other arguments or keywords as additional arguments.  If
@emph{content_manager} is not specified, use the @code{content_manager} specified
by the current @ref{70,,policy}.
@end deffn

@geindex set_content() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage set_content}@anchor{1116}
@deffn {Method} set_content (*args, content_manager=None, **kw)

Call the @ref{2c81,,set_content()} method
of the @emph{content_manager}, passing self as the message object, and passing
along any other arguments or keywords as additional arguments.  If
@emph{content_manager} is not specified, use the @code{content_manager} specified
by the current @ref{70,,policy}.
@end deffn

@geindex make_related() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage make_related}@anchor{2c82}
@deffn {Method} make_related (boundary=None)

Convert a non-@code{multipart} message into a @code{multipart/related} message,
moving any existing @code{Content-} headers and payload into a
(new) first part of the @code{multipart}.  If @emph{boundary} is specified, use
it as the boundary string in the multipart, otherwise leave the boundary
to be automatically created when it is needed (for example, when the
message is serialized).
@end deffn

@geindex make_alternative() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage make_alternative}@anchor{2c83}
@deffn {Method} make_alternative (boundary=None)

Convert a non-@code{multipart} or a @code{multipart/related} into a
@code{multipart/alternative}, moving any existing @code{Content-}
headers and payload into a (new) first part of the @code{multipart}.  If
@emph{boundary} is specified, use it as the boundary string in the multipart,
otherwise leave the boundary to be automatically created when it is
needed (for example, when the message is serialized).
@end deffn

@geindex make_mixed() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage make_mixed}@anchor{2c84}
@deffn {Method} make_mixed (boundary=None)

Convert a non-@code{multipart}, a @code{multipart/related}, or a
@code{multipart-alternative} into a @code{multipart/mixed}, moving any existing
@code{Content-} headers and payload into a (new) first part of the
@code{multipart}.  If @emph{boundary} is specified, use it as the boundary string
in the multipart, otherwise leave the boundary to be automatically
created when it is needed (for example, when the message is serialized).
@end deffn

@geindex add_related() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage add_related}@anchor{2c85}
@deffn {Method} add_related (*args, content_manager=None, **kw)

If the message is a @code{multipart/related}, create a new message
object, pass all of the arguments to its @ref{1116,,set_content()} method,
and @ref{2c86,,attach()} it to the @code{multipart}.  If
the message is a non-@code{multipart}, call @ref{2c82,,make_related()} and then
proceed as above.  If the message is any other type of @code{multipart},
raise a @ref{162,,TypeError}. If @emph{content_manager} is not specified, use
the @code{content_manager} specified by the current @ref{70,,policy}.
If the added part has no @code{Content-Disposition} header,
add one with the value @code{inline}.
@end deffn

@geindex add_alternative() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage add_alternative}@anchor{2c87}
@deffn {Method} add_alternative (*args, content_manager=None, **kw)

If the message is a @code{multipart/alternative}, create a new message
object, pass all of the arguments to its @ref{1116,,set_content()} method, and
@ref{2c86,,attach()} it to the @code{multipart}.  If the
message is a non-@code{multipart} or @code{multipart/related}, call
@ref{2c83,,make_alternative()} and then proceed as above.  If the message is
any other type of @code{multipart}, raise a @ref{162,,TypeError}. If
@emph{content_manager} is not specified, use the @code{content_manager} specified
by the current @ref{70,,policy}.
@end deffn

@geindex add_attachment() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage add_attachment}@anchor{2c88}
@deffn {Method} add_attachment (*args, content_manager=None, **kw)

If the message is a @code{multipart/mixed}, create a new message object,
pass all of the arguments to its @ref{1116,,set_content()} method, and
@ref{2c86,,attach()} it to the @code{multipart}.  If the
message is a non-@code{multipart}, @code{multipart/related}, or
@code{multipart/alternative}, call @ref{2c84,,make_mixed()} and then proceed as
above. If @emph{content_manager} is not specified, use the @code{content_manager}
specified by the current @ref{70,,policy}.  If the added part
has no @code{Content-Disposition} header, add one with the value
@code{attachment}.  This method can be used both for explicit attachments
(@code{Content-Disposition: attachment}) and @code{inline} attachments
(@code{Content-Disposition: inline}), by passing appropriate
options to the @code{content_manager}.
@end deffn

@geindex clear() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage clear}@anchor{2c89}
@deffn {Method} clear ()

Remove the payload and all of the headers.
@end deffn

@geindex clear_content() (email.message.EmailMessage method)
@anchor{library/email message email message EmailMessage clear_content}@anchor{2c8a}
@deffn {Method} clear_content ()

Remove the payload and all of the @code{Content-} headers, leaving
all other headers intact and in their original order.
@end deffn

@ref{7e4,,EmailMessage} objects have the following instance attributes:

@geindex preamble (email.message.EmailMessage attribute)
@anchor{library/email message email message EmailMessage preamble}@anchor{2c8b}
@deffn {Attribute} preamble

The format of a MIME document allows for some text between the blank line
following the headers, and the first multipart boundary string. Normally,
this text is never visible in a MIME-aware mail reader because it falls
outside the standard MIME armor.  However, when viewing the raw text of
the message, or when viewing the message in a non-MIME aware reader, this
text can become visible.

The @emph{preamble} attribute contains this leading extra-armor text for MIME
documents.  When the @ref{d43,,Parser} discovers some text
after the headers but before the first boundary string, it assigns this
text to the message’s @emph{preamble} attribute.  When the
@ref{d72,,Generator} is writing out the plain text
representation of a MIME message, and it finds the
message has a @emph{preamble} attribute, it will write this text in the area
between the headers and the first boundary.  See @ref{6f,,email.parser} and
@ref{69,,email.generator} for details.

Note that if the message object has no preamble, the @emph{preamble} attribute
will be @code{None}.
@end deffn

@geindex epilogue (email.message.EmailMessage attribute)
@anchor{library/email message email message EmailMessage epilogue}@anchor{2c8c}
@deffn {Attribute} epilogue

The @emph{epilogue} attribute acts the same way as the @emph{preamble} attribute,
except that it contains text that appears between the last boundary and
the end of the message.  As with the @ref{2c8b,,preamble},
if there is no epilog text this attribute will be @code{None}.
@end deffn

@geindex defects (email.message.EmailMessage attribute)
@anchor{library/email message email message EmailMessage defects}@anchor{2c8d}
@deffn {Attribute} defects

The @emph{defects} attribute contains a list of all the problems found when
parsing this message.  See @ref{68,,email.errors} for a detailed description
of the possible parsing defects.
@end deffn
@end deffn

@geindex MIMEPart (class in email.message)
@anchor{library/email message email message MIMEPart}@anchor{ab7}
@deffn {Class} email.message.MIMEPart (policy=default)

This class represents a subpart of a MIME message.  It is identical to
@ref{7e4,,EmailMessage}, except that no @code{MIME-Version} headers are
added when @ref{1116,,set_content()} is called, since sub-parts do
not need their own @code{MIME-Version} headers.
@end deffn

@node email parser Parsing email messages,email generator Generating MIME documents,email message Representing an email message,email — An email and MIME handling package
@anchor{library/email parser doc}@anchor{2c8e}@anchor{library/email parser email-parser-parsing-email-messages}@anchor{2c8f}@anchor{library/email parser module-email parser}@anchor{6f}
@subsubsection @code{email.parser}: Parsing email messages


@geindex module; email.parser

@strong{Source code:} Lib/email/parser.py@footnote{https://github.com/python/cpython/tree/main/Lib/email/parser.py}


__________________________________________________________________


Message object structures can be created in one of two ways: they can be
created from whole cloth by creating an @ref{7e4,,EmailMessage}
object, adding headers using the dictionary interface, and adding payload(s)
using @ref{1116,,set_content()} and related methods, or
they can be created by parsing a serialized representation of the email
message.

The @ref{64,,email} package provides a standard parser that understands most email
document structures, including MIME documents.  You can pass the parser a
bytes, string or file object, and the parser will return to you the root
@ref{7e4,,EmailMessage} instance of the object structure.  For
simple, non-MIME messages the payload of this root object will likely be a
string containing the text of the message.  For MIME messages, the root object
will return @code{True} from its @ref{2c59,,is_multipart()}
method, and the subparts can be accessed via the payload manipulation methods,
such as @ref{2c7c,,get_body()},
@ref{2c7e,,iter_parts()}, and
@ref{2c7b,,walk()}.

There are actually two parser interfaces available for use, the @ref{d43,,Parser}
API and the incremental @ref{2c90,,FeedParser} API.  The @ref{d43,,Parser} API is
most useful if you have the entire text of the message in memory, or if the
entire message lives in a file on the file system.  @ref{2c90,,FeedParser} is more
appropriate when you are reading the message from a stream which might block
waiting for more input (such as reading an email message from a socket).  The
@ref{2c90,,FeedParser} can consume and parse the message incrementally, and only
returns the root object when you close the parser.

Note that the parser can be extended in limited ways, and of course you can
implement your own parser completely from scratch.  All of the logic that
connects the @ref{64,,email} package’s bundled parser and the
@ref{7e4,,EmailMessage} class is embodied in the @code{policy}
class, so a custom parser can create message object trees any way it finds
necessary by implementing custom versions of the appropriate @code{policy}
methods.

@menu
* FeedParser API:: 
* Parser API:: 
* Additional notes:: 

@end menu

@node FeedParser API,Parser API,,email parser Parsing email messages
@anchor{library/email parser feedparser-api}@anchor{2c91}
@subsubsection FeedParser API


The @ref{d6f,,BytesFeedParser}, imported from the @code{email.feedparser} module,
provides an API that is conducive to incremental parsing of email messages,
such as would be necessary when reading the text of an email message from a
source that can block (such as a socket).  The @ref{d6f,,BytesFeedParser} can of
course be used to parse an email message fully contained in a @ref{87e,,bytes-like object}, string, or file, but the @ref{d70,,BytesParser} API may be more
convenient for such use cases.  The semantics and results of the two parser
APIs are identical.

The @ref{d6f,,BytesFeedParser}’s API is simple; you create an instance, feed it a
bunch of bytes until there’s no more to feed it, then close the parser to
retrieve the root message object.  The @ref{d6f,,BytesFeedParser} is extremely
accurate when parsing standards-compliant messages, and it does a very good job
of parsing non-compliant messages, providing information about how a message
was deemed broken.  It will populate a message object’s
@ref{2c8d,,defects} attribute with a list of any
problems it found in a message.  See the @ref{68,,email.errors} module for the
list of defects that it can find.

Here is the API for the @ref{d6f,,BytesFeedParser}:

@geindex BytesFeedParser (class in email.parser)
@anchor{library/email parser email parser BytesFeedParser}@anchor{d6f}
@deffn {Class} email.parser.BytesFeedParser (_factory=None, *, policy=policy.compat32)

Create a @ref{d6f,,BytesFeedParser} instance.  Optional @emph{_factory} is a
no-argument callable; if not specified use the
@ref{7e1,,message_factory} from the @emph{policy}.  Call
@emph{_factory} whenever a new message object is needed.

If @emph{policy} is specified use the rules it specifies to update the
representation of the message.  If @emph{policy} is not set, use the
@ref{c45,,compat32} policy, which maintains backward
compatibility with the Python 3.2 version of the email package and provides
@ref{7e3,,Message} as the default factory.  All other policies
provide @ref{7e4,,EmailMessage} as the default @emph{_factory}. For
more information on what else @emph{policy} controls, see the
@ref{70,,policy} documentation.

Note: @strong{The policy keyword should always be specified}; The default will
change to @ref{2c54,,email.policy.default} in a future version of Python.


New in version 3.2.



Changed in version 3.3: Added the @emph{policy} keyword.



Changed in version 3.6: @emph{_factory} defaults to the policy @code{message_factory}.


@geindex feed() (email.parser.BytesFeedParser method)
@anchor{library/email parser email parser BytesFeedParser feed}@anchor{2c92}
@deffn {Method} feed (data)

Feed the parser some more data.  @emph{data} should be a @ref{87e,,bytes-like object} containing one or more lines.  The lines can be partial and the
parser will stitch such partial lines together properly.  The lines can
have any of the three common line endings: carriage return, newline, or
carriage return and newline (they can even be mixed).
@end deffn

@geindex close() (email.parser.BytesFeedParser method)
@anchor{library/email parser email parser BytesFeedParser close}@anchor{2c93}
@deffn {Method} close ()

Complete the parsing of all previously fed data and return the root
message object.  It is undefined what happens if @ref{2c92,,feed()} is called
after this method has been called.
@end deffn
@end deffn

@geindex FeedParser (class in email.parser)
@anchor{library/email parser email parser FeedParser}@anchor{2c90}
@deffn {Class} email.parser.FeedParser (_factory=None, *, policy=policy.compat32)

Works like @ref{d6f,,BytesFeedParser} except that the input to the
@ref{2c92,,feed()} method must be a string.  This is of limited
utility, since the only way for such a message to be valid is for it to
contain only ASCII text or, if @code{utf8} is
@code{True}, no binary attachments.


Changed in version 3.3: Added the @emph{policy} keyword.

@end deffn

@node Parser API,Additional notes,FeedParser API,email parser Parsing email messages
@anchor{library/email parser parser-api}@anchor{2c94}
@subsubsection Parser API


The @ref{d70,,BytesParser} class, imported from the @ref{6f,,email.parser} module,
provides an API that can be used to parse a message when the complete contents
of the message are available in a @ref{87e,,bytes-like object} or file.  The
@ref{6f,,email.parser} module also provides @ref{d43,,Parser} for parsing strings,
and header-only parsers, @ref{c4c,,BytesHeaderParser} and
@ref{c4d,,HeaderParser}, which can be used if you’re only interested in the
headers of the message.  @ref{c4c,,BytesHeaderParser} and @ref{c4d,,HeaderParser}
can be much faster in these situations, since they do not attempt to parse the
message body, instead setting the payload to the raw body.

@geindex BytesParser (class in email.parser)
@anchor{library/email parser email parser BytesParser}@anchor{d70}
@deffn {Class} email.parser.BytesParser (_class=None, *, policy=policy.compat32)

Create a @ref{d70,,BytesParser} instance.  The @emph{_class} and @emph{policy}
arguments have the same meaning and semantics as the @emph{_factory}
and @emph{policy} arguments of @ref{d6f,,BytesFeedParser}.

Note: @strong{The policy keyword should always be specified}; The default will
change to @ref{2c54,,email.policy.default} in a future version of Python.


Changed in version 3.3: Removed the @emph{strict} argument that was deprecated in 2.4.  Added the
@emph{policy} keyword.



Changed in version 3.6: @emph{_class} defaults to the policy @code{message_factory}.


@geindex parse() (email.parser.BytesParser method)
@anchor{library/email parser email parser BytesParser parse}@anchor{2c95}
@deffn {Method} parse (fp, headersonly=False)

Read all the data from the binary file-like object @emph{fp}, parse the
resulting bytes, and return the message object.  @emph{fp} must support
both the @ref{1b74,,readline()} and the @code{read()}
methods.

The bytes contained in @emph{fp} must be formatted as a block of 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html}
(or, if @code{utf8} is @code{True}, 
@geindex RFC; RFC 6532
RFC 6532@footnote{https://tools.ietf.org/html/rfc6532.html})
style headers and header continuation lines, optionally preceded by an
envelope header.  The header block is terminated either by the end of the
data or by a blank line.  Following the header block is the body of the
message (which may contain MIME-encoded subparts, including subparts
with a @code{Content-Transfer-Encoding} of @code{8bit}).

Optional @emph{headersonly} is a flag specifying whether to stop parsing after
reading the headers or not.  The default is @code{False}, meaning it parses
the entire contents of the file.
@end deffn

@geindex parsebytes() (email.parser.BytesParser method)
@anchor{library/email parser email parser BytesParser parsebytes}@anchor{2c96}
@deffn {Method} parsebytes (bytes, headersonly=False)

Similar to the @ref{2c95,,parse()} method, except it takes a @ref{87e,,bytes-like object} instead of a file-like object.  Calling this method on a
@ref{87e,,bytes-like object} is equivalent to wrapping @emph{bytes} in a
@ref{a15,,BytesIO} instance first and calling @ref{2c95,,parse()}.

Optional @emph{headersonly} is as with the @ref{2c95,,parse()} method.
@end deffn


New in version 3.2.

@end deffn

@geindex BytesHeaderParser (class in email.parser)
@anchor{library/email parser email parser BytesHeaderParser}@anchor{c4c}
@deffn {Class} email.parser.BytesHeaderParser (_class=None, *, policy=policy.compat32)

Exactly like @ref{d70,,BytesParser}, except that @emph{headersonly}
defaults to @code{True}.


New in version 3.3.

@end deffn

@geindex Parser (class in email.parser)
@anchor{library/email parser email parser Parser}@anchor{d43}
@deffn {Class} email.parser.Parser (_class=None, *, policy=policy.compat32)

This class is parallel to @ref{d70,,BytesParser}, but handles string input.


Changed in version 3.3: Removed the @emph{strict} argument.  Added the @emph{policy} keyword.



Changed in version 3.6: @emph{_class} defaults to the policy @code{message_factory}.


@geindex parse() (email.parser.Parser method)
@anchor{library/email parser email parser Parser parse}@anchor{2c97}
@deffn {Method} parse (fp, headersonly=False)

Read all the data from the text-mode file-like object @emph{fp}, parse the
resulting text, and return the root message object.  @emph{fp} must support
both the @ref{209b,,readline()} and the
@ref{24a6,,read()} methods on file-like objects.

Other than the text mode requirement, this method operates like
@ref{2c95,,BytesParser.parse()}.
@end deffn

@geindex parsestr() (email.parser.Parser method)
@anchor{library/email parser email parser Parser parsestr}@anchor{2c98}
@deffn {Method} parsestr (text, headersonly=False)

Similar to the @ref{2c97,,parse()} method, except it takes a string object
instead of a file-like object.  Calling this method on a string is
equivalent to wrapping @emph{text} in a @ref{aa1,,StringIO} instance first
and calling @ref{2c97,,parse()}.

Optional @emph{headersonly} is as with the @ref{2c97,,parse()} method.
@end deffn
@end deffn

@geindex HeaderParser (class in email.parser)
@anchor{library/email parser email parser HeaderParser}@anchor{c4d}
@deffn {Class} email.parser.HeaderParser (_class=None, *, policy=policy.compat32)

Exactly like @ref{d43,,Parser}, except that @emph{headersonly}
defaults to @code{True}.
@end deffn

Since creating a message object structure from a string or a file object is such
a common task, four functions are provided as a convenience.  They are available
in the top-level @ref{64,,email} package namespace.

@geindex message_from_bytes() (in module email)
@anchor{library/email parser email message_from_bytes}@anchor{d6d}
@deffn {Function} email.message_from_bytes (s, _class=None, *, policy=policy.compat32)

Return a message object structure from a @ref{87e,,bytes-like object}.  This is
equivalent to @code{BytesParser().parsebytes(s)}.  Optional @emph{_class} and
@emph{policy} are interpreted as with the @ref{d70,,BytesParser} class
constructor.


New in version 3.2.



Changed in version 3.3: Removed the @emph{strict} argument.  Added the @emph{policy} keyword.

@end deffn

@geindex message_from_binary_file() (in module email)
@anchor{library/email parser email message_from_binary_file}@anchor{d6e}
@deffn {Function} email.message_from_binary_file (fp, _class=None, *, policy=policy.compat32)

Return a message object structure tree from an open binary @ref{d68,,file object}.  This is equivalent to @code{BytesParser().parse(fp)}.  @emph{_class} and
@emph{policy} are interpreted as with the @ref{d70,,BytesParser} class
constructor.


New in version 3.2.



Changed in version 3.3: Removed the @emph{strict} argument.  Added the @emph{policy} keyword.

@end deffn

@geindex message_from_string() (in module email)
@anchor{library/email parser email message_from_string}@anchor{2c99}
@deffn {Function} email.message_from_string (s, _class=None, *, policy=policy.compat32)

Return a message object structure from a string.  This is equivalent to
@code{Parser().parsestr(s)}.  @emph{_class} and @emph{policy} are interpreted as
with the @ref{d43,,Parser} class constructor.


Changed in version 3.3: Removed the @emph{strict} argument.  Added the @emph{policy} keyword.

@end deffn

@geindex message_from_file() (in module email)
@anchor{library/email parser email message_from_file}@anchor{2c9a}
@deffn {Function} email.message_from_file (fp, _class=None, *, policy=policy.compat32)

Return a message object structure tree from an open @ref{d68,,file object}.
This is equivalent to @code{Parser().parse(fp)}.  @emph{_class} and @emph{policy} are
interpreted as with the @ref{d43,,Parser} class constructor.


Changed in version 3.3: Removed the @emph{strict} argument.  Added the @emph{policy} keyword.



Changed in version 3.6: @emph{_class} defaults to the policy @code{message_factory}.

@end deffn

Here’s an example of how you might use @ref{d6d,,message_from_bytes()} at an
interactive Python prompt:

@example
>>> import email
>>> msg = email.message_from_bytes(myBytes)  
@end example

@node Additional notes,,Parser API,email parser Parsing email messages
@anchor{library/email parser additional-notes}@anchor{2c9b}
@subsubsection Additional notes


Here are some notes on the parsing semantics:


@itemize *

@item 
Most non-@code{multipart} type messages are parsed as a single message
object with a string payload.  These objects will return @code{False} for
@ref{2c59,,is_multipart()}, and
@ref{2c7e,,iter_parts()} will yield an empty list.

@item 
All @code{multipart} type messages will be parsed as a container message
object with a list of sub-message objects for their payload.  The outer
container message will return @code{True} for
@ref{2c59,,is_multipart()}, and
@ref{2c7e,,iter_parts()} will yield a list of subparts.

@item 
Most messages with a content type of @code{message/*} (such as
@code{message/delivery-status} and @code{message/rfc822}) will also
be parsed as container object containing a list payload of length 1.  Their
@ref{2c59,,is_multipart()} method will return @code{True}.
The single element yielded by @ref{2c7e,,iter_parts()}
will be a sub-message object.

@item 
Some non-standards-compliant messages may not be internally consistent about
their @code{multipart}-edness.  Such messages may have a
@code{Content-Type} header of type @code{multipart}, but their
@ref{2c59,,is_multipart()} method may return @code{False}.
If such messages were parsed with the @ref{2c90,,FeedParser},
they will have an instance of the
@code{MultipartInvariantViolationDefect} class in their
@emph{defects} attribute list.  See @ref{68,,email.errors} for details.
@end itemize

@node email generator Generating MIME documents,email policy Policy Objects,email parser Parsing email messages,email — An email and MIME handling package
@anchor{library/email generator doc}@anchor{2c9c}@anchor{library/email generator email-generator-generating-mime-documents}@anchor{2c9d}@anchor{library/email generator module-email generator}@anchor{69}
@subsubsection @code{email.generator}: Generating MIME documents


@geindex module; email.generator

@strong{Source code:} Lib/email/generator.py@footnote{https://github.com/python/cpython/tree/main/Lib/email/generator.py}


__________________________________________________________________


One of the most common tasks is to generate the flat (serialized) version of
the email message represented by a message object structure.  You will need to
do this if you want to send your message via @ref{9c3,,smtplib.SMTP.sendmail()} or
the @ref{bc,,nntplib} module, or print the message on the console.  Taking a
message object structure and producing a serialized representation is the job
of the generator classes.

As with the @ref{6f,,email.parser} module, you aren’t limited to the functionality
of the bundled generator; you could write one from scratch yourself.  However
the bundled generator knows how to generate most email in a standards-compliant
way, should handle MIME and non-MIME email messages just fine, and is designed
so that the bytes-oriented parsing and generation operations are inverses,
assuming the same non-transforming @ref{70,,policy} is used for both.  That
is, parsing the serialized byte stream via the
@ref{d70,,BytesParser} class and then regenerating the serialized
byte stream using @ref{d73,,BytesGenerator} should produce output identical to
the input @footnote{@w{(1)} 
This statement assumes that you use the appropriate setting for
@code{unixfrom}, and that there are no @code{policy} settings calling for
automatic adjustments (for example,
@code{refold_source} must be @code{none}, which is
@emph{not} the default).  It is also not 100% true, since if the message
does not conform to the RFC standards occasionally information about the
exact original text is lost during parsing error recovery.  It is a goal
to fix these latter edge cases when possible.
}.  (On the other hand, using the generator on an
@ref{7e4,,EmailMessage} constructed by program may result in
changes to the @ref{7e4,,EmailMessage} object as defaults are
filled in.)

The @ref{d72,,Generator} class can be used to flatten a message into a text (as
opposed to binary) serialized representation, but since Unicode cannot
represent binary data directly, the message is of necessity transformed into
something that contains only ASCII characters, using the standard email RFC
Content Transfer Encoding techniques for encoding email messages for transport
over channels that are not “8 bit clean”.

To accommodate reproducible processing of SMIME-signed messages
@ref{d72,,Generator} disables header folding for message parts of type
@code{multipart/signed} and all subparts.

@geindex BytesGenerator (class in email.generator)
@anchor{library/email generator email generator BytesGenerator}@anchor{d73}
@deffn {Class} email.generator.BytesGenerator (outfp, mangle_from_=None, maxheaderlen=None, *, policy=None)

Return a @ref{d73,,BytesGenerator} object that will write any message provided
to the @ref{2c9e,,flatten()} method, or any surrogateescape encoded text provided
to the @ref{2c9f,,write()} method, to the @ref{1e54,,file-like object} @emph{outfp}.
@emph{outfp} must support a @code{write} method that accepts binary data.

If optional @emph{mangle_from_} is @code{True}, put a @code{>} character in front of
any line in the body that starts with the exact string @code{"From "}, that is
@code{From} followed by a space at the beginning of a line.  @emph{mangle_from_}
defaults to the value of the @ref{955,,mangle_from_}
setting of the @emph{policy} (which is @code{True} for the
@ref{7e2,,compat32} policy and @code{False} for all others).
@emph{mangle_from_} is intended for use when messages are stored in unix mbox
format (see @ref{aa,,mailbox} and WHY THE CONTENT-LENGTH FORMAT IS BAD@footnote{https://www.jwz.org/doc/content-length.html}).

If @emph{maxheaderlen} is not @code{None}, refold any header lines that are longer
than @emph{maxheaderlen}, or if @code{0}, do not rewrap any headers.  If
@emph{manheaderlen} is @code{None} (the default), wrap headers and other message
lines according to the @emph{policy} settings.

If @emph{policy} is specified, use that policy to control message generation.  If
@emph{policy} is @code{None} (the default), use the policy associated with the
@ref{7e3,,Message} or @ref{7e4,,EmailMessage}
object passed to @code{flatten} to control the message generation.  See
@ref{70,,email.policy} for details on what @emph{policy} controls.


New in version 3.2.



Changed in version 3.3: Added the @emph{policy} keyword.



Changed in version 3.6: The default behavior of the @emph{mangle_from_}
and @emph{maxheaderlen} parameters is to follow the policy.


@geindex flatten() (email.generator.BytesGenerator method)
@anchor{library/email generator email generator BytesGenerator flatten}@anchor{2c9e}
@deffn {Method} flatten (msg, unixfrom=False, linesep=None)

Print the textual representation of the message object structure rooted
at @emph{msg} to the output file specified when the @ref{d73,,BytesGenerator}
instance was created.

If the @ref{70,,policy} option @ref{2ca0,,cte_type}
is @code{8bit} (the default), copy any headers in the original parsed
message that have not been modified to the output with any bytes with the
high bit set reproduced as in the original, and preserve the non-ASCII
@code{Content-Transfer-Encoding} of any body parts that have them.
If @code{cte_type} is @code{7bit}, convert the bytes with the high bit set as
needed using an ASCII-compatible @code{Content-Transfer-Encoding}.
That is, transform parts with non-ASCII
@code{Content-Transfer-Encoding}
(@code{Content-Transfer-Encoding: 8bit}) to an ASCII compatible
@code{Content-Transfer-Encoding}, and encode RFC-invalid non-ASCII
bytes in headers using the MIME @code{unknown-8bit} character set, thus
rendering them RFC-compliant.

@c XXX: There should be an option that just does the RFC
@c compliance transformation on headers but leaves CTE 8bit parts alone.

If @emph{unixfrom} is @code{True}, print the envelope header delimiter used by
the Unix mailbox format (see @ref{aa,,mailbox}) before the first of the
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html} headers of the root message object.  If the root object has
no envelope header, craft a standard one.  The default is @code{False}.
Note that for subparts, no envelope header is ever printed.

If @emph{linesep} is not @code{None}, use it as the separator character between
all the lines of the flattened message.  If @emph{linesep} is @code{None} (the
default), use the value specified in the @emph{policy}.

@c XXX: flatten should take a *policy* keyword.
@end deffn

@geindex clone() (email.generator.BytesGenerator method)
@anchor{library/email generator email generator BytesGenerator clone}@anchor{2ca1}
@deffn {Method} clone (fp)

Return an independent clone of this @ref{d73,,BytesGenerator} instance with
the exact same option settings, and @emph{fp} as the new @emph{outfp}.
@end deffn

@geindex write() (email.generator.BytesGenerator method)
@anchor{library/email generator email generator BytesGenerator write}@anchor{2c9f}
@deffn {Method} write (s)

Encode @emph{s} using the @code{ASCII} codec and the @code{surrogateescape} error
handler, and pass it to the @emph{write} method of the @emph{outfp} passed to the
@ref{d73,,BytesGenerator}’s constructor.
@end deffn
@end deffn

As a convenience, @ref{7e4,,EmailMessage} provides the methods
@ref{2c57,,as_bytes()} and @code{bytes(aMessage)} (a.k.a.
@ref{2c58,,__bytes__()}), which simplify the generation of
a serialized binary representation of a message object.  For more detail, see
@ref{6d,,email.message}.

Because strings cannot represent binary data, the @ref{d72,,Generator} class must
convert any binary data in any message it flattens to an ASCII compatible
format, by converting them to an ASCII compatible
@code{Content-Transfer_Encoding}.  Using the terminology of the email
RFCs, you can think of this as @ref{d72,,Generator} serializing to an I/O stream
that is not “8 bit clean”.  In other words, most applications will want
to be using @ref{d73,,BytesGenerator}, and not @ref{d72,,Generator}.

@geindex Generator (class in email.generator)
@anchor{library/email generator email generator Generator}@anchor{d72}
@deffn {Class} email.generator.Generator (outfp, mangle_from_=None, maxheaderlen=None, *, policy=None)

Return a @ref{d72,,Generator} object that will write any message provided
to the @ref{2ca2,,flatten()} method, or any text provided to the @ref{2ca3,,write()}
method, to the @ref{1e54,,file-like object} @emph{outfp}.  @emph{outfp} must support a
@code{write} method that accepts string data.

If optional @emph{mangle_from_} is @code{True}, put a @code{>} character in front of
any line in the body that starts with the exact string @code{"From "}, that is
@code{From} followed by a space at the beginning of a line.  @emph{mangle_from_}
defaults to the value of the @ref{955,,mangle_from_}
setting of the @emph{policy} (which is @code{True} for the
@ref{7e2,,compat32} policy and @code{False} for all others).
@emph{mangle_from_} is intended for use when messages are stored in unix mbox
format (see @ref{aa,,mailbox} and WHY THE CONTENT-LENGTH FORMAT IS BAD@footnote{https://www.jwz.org/doc/content-length.html}).

If @emph{maxheaderlen} is not @code{None}, refold any header lines that are longer
than @emph{maxheaderlen}, or if @code{0}, do not rewrap any headers.  If
@emph{manheaderlen} is @code{None} (the default), wrap headers and other message
lines according to the @emph{policy} settings.

If @emph{policy} is specified, use that policy to control message generation.  If
@emph{policy} is @code{None} (the default), use the policy associated with the
@ref{7e3,,Message} or @ref{7e4,,EmailMessage}
object passed to @code{flatten} to control the message generation.  See
@ref{70,,email.policy} for details on what @emph{policy} controls.


Changed in version 3.3: Added the @emph{policy} keyword.



Changed in version 3.6: The default behavior of the @emph{mangle_from_}
and @emph{maxheaderlen} parameters is to follow the policy.


@geindex flatten() (email.generator.Generator method)
@anchor{library/email generator email generator Generator flatten}@anchor{2ca2}
@deffn {Method} flatten (msg, unixfrom=False, linesep=None)

Print the textual representation of the message object structure rooted
at @emph{msg} to the output file specified when the @ref{d72,,Generator}
instance was created.

If the @ref{70,,policy} option @ref{2ca0,,cte_type}
is @code{8bit}, generate the message as if the option were set to @code{7bit}.
(This is required because strings cannot represent non-ASCII bytes.)
Convert any bytes with the high bit set as needed using an
ASCII-compatible @code{Content-Transfer-Encoding}.  That is,
transform parts with non-ASCII @code{Content-Transfer-Encoding}
(@code{Content-Transfer-Encoding: 8bit}) to an ASCII compatible
@code{Content-Transfer-Encoding}, and encode RFC-invalid non-ASCII
bytes in headers using the MIME @code{unknown-8bit} character set, thus
rendering them RFC-compliant.

If @emph{unixfrom} is @code{True}, print the envelope header delimiter used by
the Unix mailbox format (see @ref{aa,,mailbox}) before the first of the
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html} headers of the root message object.  If the root object has
no envelope header, craft a standard one.  The default is @code{False}.
Note that for subparts, no envelope header is ever printed.

If @emph{linesep} is not @code{None}, use it as the separator character between
all the lines of the flattened message.  If @emph{linesep} is @code{None} (the
default), use the value specified in the @emph{policy}.

@c XXX: flatten should take a *policy* keyword.


Changed in version 3.2: Added support for re-encoding @code{8bit} message bodies, and the
@emph{linesep} argument.

@end deffn

@geindex clone() (email.generator.Generator method)
@anchor{library/email generator email generator Generator clone}@anchor{2ca4}
@deffn {Method} clone (fp)

Return an independent clone of this @ref{d72,,Generator} instance with the
exact same options, and @emph{fp} as the new @emph{outfp}.
@end deffn

@geindex write() (email.generator.Generator method)
@anchor{library/email generator email generator Generator write}@anchor{2ca3}
@deffn {Method} write (s)

Write @emph{s} to the @emph{write} method of the @emph{outfp} passed to the
@ref{d72,,Generator}’s constructor.  This provides just enough file-like
API for @ref{d72,,Generator} instances to be used in the @ref{af6,,print()}
function.
@end deffn
@end deffn

As a convenience, @ref{7e4,,EmailMessage} provides the methods
@ref{2c55,,as_string()} and @code{str(aMessage)} (a.k.a.
@ref{2c56,,__str__()}), which simplify the generation of
a formatted string representation of a message object.  For more detail, see
@ref{6d,,email.message}.

The @ref{69,,email.generator} module also provides a derived class,
@ref{7e0,,DecodedGenerator}, which is like the @ref{d72,,Generator} base class,
except that non-@code{text} parts are not serialized, but are instead
represented in the output stream by a string derived from a template filled
in with information about the part.

@geindex DecodedGenerator (class in email.generator)
@anchor{library/email generator email generator DecodedGenerator}@anchor{7e0}
@deffn {Class} email.generator.DecodedGenerator (outfp, mangle_from_=None, maxheaderlen=None, fmt=None, *, policy=None)

Act like @ref{d72,,Generator}, except that for any subpart of the message
passed to @ref{2ca2,,Generator.flatten()}, if the subpart is of main type
@code{text}, print the decoded payload of the subpart, and if the main
type is not @code{text}, instead of printing it fill in the string
@emph{fmt} using information from the part and print the resulting
filled-in string.

To fill in @emph{fmt}, execute @code{fmt % part_info}, where @code{part_info}
is a dictionary composed of the following keys and values:


@itemize *

@item 
@code{type} – Full MIME type of the non-@code{text} part

@item 
@code{maintype} – Main MIME type of the non-@code{text} part

@item 
@code{subtype} – Sub-MIME type of the non-@code{text} part

@item 
@code{filename} – Filename of the non-@code{text} part

@item 
@code{description} – Description associated with the non-@code{text} part

@item 
@code{encoding} – Content transfer encoding of the non-@code{text} part
@end itemize

If @emph{fmt} is @code{None}, use the following default @emph{fmt}:

@quotation

“[Non-text (%(type)s) part of message omitted, filename %(filename)s]”
@end quotation

Optional @emph{_mangle_from_} and @emph{maxheaderlen} are as with the
@ref{d72,,Generator} base class.
@end deffn

@node email policy Policy Objects,email errors Exception and Defect classes,email generator Generating MIME documents,email — An email and MIME handling package
@anchor{library/email policy doc}@anchor{2ca5}@anchor{library/email policy email-policy-policy-objects}@anchor{2ca6}@anchor{library/email policy module-email policy}@anchor{70}
@subsubsection @code{email.policy}: Policy Objects


@geindex module; email.policy


New in version 3.3.


@strong{Source code:} Lib/email/policy.py@footnote{https://github.com/python/cpython/tree/main/Lib/email/policy.py}


__________________________________________________________________


The @ref{64,,email} package’s prime focus is the handling of email messages as
described by the various email and MIME RFCs.  However, the general format of
email messages (a block of header fields each consisting of a name followed by
a colon followed by a value, the whole block followed by a blank line and an
arbitrary ‘body’), is a format that has found utility outside of the realm of
email.  Some of these uses conform fairly closely to the main email RFCs, some
do not.  Even when working with email, there are times when it is desirable to
break strict compliance with the RFCs, such as generating emails that
interoperate with email servers that do not themselves follow the standards, or
that implement extensions you want to use in ways that violate the
standards.

Policy objects give the email package the flexibility to handle all these
disparate use cases.

A @ref{c44,,Policy} object encapsulates a set of attributes and methods that
control the behavior of various components of the email package during use.
@ref{c44,,Policy} instances can be passed to various classes and methods in the
email package to alter the default behavior.  The settable values and their
defaults are described below.

There is a default policy used by all classes in the email package.  For all of
the @ref{6f,,parser} classes and the related convenience functions, and for
the @ref{7e3,,Message} class, this is the @ref{c45,,Compat32}
policy, via its corresponding pre-defined instance @ref{7e2,,compat32}.  This
policy provides for complete backward compatibility (in some cases, including
bug compatibility) with the pre-Python3.3 version of the email package.

This default value for the @emph{policy} keyword to
@ref{7e4,,EmailMessage} is the @ref{c48,,EmailPolicy} policy, via
its pre-defined instance @ref{2c54,,default}.

When a @ref{7e3,,Message} or @ref{7e4,,EmailMessage}
object is created, it acquires a policy.  If the message is created by a
@ref{6f,,parser}, a policy passed to the parser will be the policy used by
the message it creates.  If the message is created by the program, then the
policy can be specified when it is created.  When a message is passed to a
@ref{69,,generator}, the generator uses the policy from the message by
default, but you can also pass a specific policy to the generator that will
override the one stored on the message object.

The default value for the @emph{policy} keyword for the @ref{6f,,email.parser} classes
and the parser convenience functions @strong{will be changing} in a future version of
Python.  Therefore you should @strong{always specify explicitly which policy you want
to use} when calling any of the classes and functions described in the
@ref{6f,,parser} module.

The first part of this documentation covers the features of @ref{c44,,Policy}, an
@ref{d5c,,abstract base class} that defines the features that are common to all
policy objects, including @ref{7e2,,compat32}.  This includes certain hook
methods that are called internally by the email package, which a custom policy
could override to obtain different behavior.  The second part describes the
concrete classes @ref{c48,,EmailPolicy} and @ref{c45,,Compat32}, which implement
the hooks that provide the standard behavior and the backward compatible
behavior and features, respectively.

@ref{c44,,Policy} instances are immutable, but they can be cloned, accepting the
same keyword arguments as the class constructor and returning a new
@ref{c44,,Policy} instance that is a copy of the original but with the specified
attributes values changed.

As an example, the following code could be used to read an email message from a
file on disk and pass it to the system @code{sendmail} program on a Unix system:

@c from unittest import mock
@c mocker = mock.patch('subprocess.Popen')
@c m = mocker.start()
@c proc = mock.MagicMock()
@c m.return_value = proc
@c proc.stdin.close.return_value = None
@c mymsg = open('mymsg.txt', 'w')
@c mymsg.write('To: abc@xyz.com\n\n')
@c mymsg.flush()

@example
>>> from email import message_from_binary_file
>>> from email.generator import BytesGenerator
>>> from email import policy
>>> from subprocess import Popen, PIPE
>>> with open('mymsg.txt', 'rb') as f:
...     msg = message_from_binary_file(f, policy=policy.default)
>>> p = Popen(['sendmail', msg['To'].addresses[0]], stdin=PIPE)
>>> g = BytesGenerator(p.stdin, policy=msg.policy.clone(linesep='\r\n'))
>>> g.flatten(msg)
>>> p.stdin.close()
>>> rc = p.wait()
@end example

@c mymsg.close()
@c mocker.stop()
@c import os
@c os.remove('mymsg.txt')

Here we are telling @ref{d73,,BytesGenerator} to use the RFC
correct line separator characters when creating the binary string to feed into
@code{sendmail's} @code{stdin}, where the default policy would use @code{\n} line
separators.

Some email package methods accept a @emph{policy} keyword argument, allowing the
policy to be overridden for that method.  For example, the following code uses
the @ref{ab4,,as_bytes()} method of the @emph{msg} object from
the previous example and writes the message to a file using the native line
separators for the platform on which it is running:

@example
>>> import os
>>> with open('converted.txt', 'wb') as f:
...     f.write(msg.as_bytes(policy=msg.policy.clone(linesep=os.linesep)))
17
@end example

Policy objects can also be combined using the addition operator, producing a
policy object whose settings are a combination of the non-default values of the
summed objects:

@example
>>> compat_SMTP = policy.compat32.clone(linesep='\r\n')
>>> compat_strict = policy.compat32.clone(raise_on_defect=True)
>>> compat_strict_SMTP = compat_SMTP + compat_strict
@end example

This operation is not commutative; that is, the order in which the objects are
added matters.  To illustrate:

@example
>>> policy100 = policy.compat32.clone(max_line_length=100)
>>> policy80 = policy.compat32.clone(max_line_length=80)
>>> apolicy = policy100 + policy80
>>> apolicy.max_line_length
80
>>> apolicy = policy80 + policy100
>>> apolicy.max_line_length
100
@end example

@geindex Policy (class in email.policy)
@anchor{library/email policy email policy Policy}@anchor{c44}
@deffn {Class} email.policy.Policy (**kw)

This is the @ref{d5c,,abstract base class} for all policy classes.  It provides
default implementations for a couple of trivial methods, as well as the
implementation of the immutability property, the @ref{c46,,clone()} method, and
the constructor semantics.

The constructor of a policy class can be passed various keyword arguments.
The arguments that may be specified are any non-method properties on this
class, plus any additional non-method properties on the concrete class.  A
value specified in the constructor will override the default value for the
corresponding attribute.

This class defines the following properties, and thus values for the
following may be passed in the constructor of any policy class:

@geindex max_line_length (email.policy.Policy attribute)
@anchor{library/email policy email policy Policy max_line_length}@anchor{2ca7}
@deffn {Attribute} max_line_length

The maximum length of any line in the serialized output, not counting the
end of line character(s).  Default is 78, per 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html}.  A value of
@code{0} or @ref{182,,None} indicates that no line wrapping should be
done at all.
@end deffn

@geindex linesep (email.policy.Policy attribute)
@anchor{library/email policy email policy Policy linesep}@anchor{2ca8}
@deffn {Attribute} linesep

The string to be used to terminate lines in serialized output.  The
default is @code{\n} because that’s the internal end-of-line discipline used
by Python, though @code{\r\n} is required by the RFCs.
@end deffn

@geindex cte_type (email.policy.Policy attribute)
@anchor{library/email policy email policy Policy cte_type}@anchor{2ca0}
@deffn {Attribute} cte_type

Controls the type of Content Transfer Encodings that may be or are
required to be used.  The possible values are:


@multitable {xxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{7bit}

@tab

all data must be “7 bit clean” (ASCII-only).  This means that
where necessary data will be encoded using either
quoted-printable or base64 encoding.

@item

@code{8bit}

@tab

data is not constrained to be 7 bit clean.  Data in headers is
still required to be ASCII-only and so will be encoded (see
@ref{2ca9,,fold_binary()} and @ref{957,,utf8} below for
exceptions), but body parts may use the @code{8bit} CTE.

@end multitable


A @code{cte_type} value of @code{8bit} only works with @code{BytesGenerator}, not
@code{Generator}, because strings cannot contain binary data.  If a
@code{Generator} is operating under a policy that specifies
@code{cte_type=8bit}, it will act as if @code{cte_type} is @code{7bit}.
@end deffn

@geindex raise_on_defect (email.policy.Policy attribute)
@anchor{library/email policy email policy Policy raise_on_defect}@anchor{2caa}
@deffn {Attribute} raise_on_defect

If @ref{748,,True}, any defects encountered will be raised as errors.  If
@ref{659,,False} (the default), defects will be passed to the
@ref{2cab,,register_defect()} method.
@end deffn

@geindex mangle_from_ (email.policy.Policy attribute)
@anchor{library/email policy email policy Policy mangle_from_}@anchor{955}
@deffn {Attribute} mangle_from_

If @ref{748,,True}, lines starting with @emph{“From “} in the body are
escaped by putting a @code{>} in front of them. This parameter is used when
the message is being serialized by a generator.
Default: @ref{659,,False}.


New in version 3.5: The @emph{mangle_from_} parameter.

@end deffn

@geindex message_factory (email.policy.Policy attribute)
@anchor{library/email policy email policy Policy message_factory}@anchor{7e1}
@deffn {Attribute} message_factory

A factory function for constructing a new empty message object.  Used
by the parser when building messages.  Defaults to @code{None}, in
which case @ref{7e3,,Message} is used.


New in version 3.6.

@end deffn

The following @ref{c44,,Policy} method is intended to be called by code using
the email library to create policy instances with custom settings:

@geindex clone() (email.policy.Policy method)
@anchor{library/email policy email policy Policy clone}@anchor{c46}
@deffn {Method} clone (**kw)

Return a new @ref{c44,,Policy} instance whose attributes have the same
values as the current instance, except where those attributes are
given new values by the keyword arguments.
@end deffn

The remaining @ref{c44,,Policy} methods are called by the email package code,
and are not intended to be called by an application using the email package.
A custom policy must implement all of these methods.

@geindex handle_defect() (email.policy.Policy method)
@anchor{library/email policy email policy Policy handle_defect}@anchor{2cac}
@deffn {Method} handle_defect (obj, defect)

Handle a @emph{defect} found on @emph{obj}.  When the email package calls this
method, @emph{defect} will always be a subclass of
@code{Defect}.

The default implementation checks the @ref{2caa,,raise_on_defect} flag.  If
it is @code{True}, @emph{defect} is raised as an exception.  If it is @code{False}
(the default), @emph{obj} and @emph{defect} are passed to @ref{2cab,,register_defect()}.
@end deffn

@geindex register_defect() (email.policy.Policy method)
@anchor{library/email policy email policy Policy register_defect}@anchor{2cab}
@deffn {Method} register_defect (obj, defect)

Register a @emph{defect} on @emph{obj}.  In the email package, @emph{defect} will always
be a subclass of @code{Defect}.

The default implementation calls the @code{append} method of the @code{defects}
attribute of @emph{obj}.  When the email package calls @ref{2cac,,handle_defect},
@emph{obj} will normally have a @code{defects} attribute that has an @code{append}
method.  Custom object types used with the email package (for example,
custom @code{Message} objects) should also provide such an attribute,
otherwise defects in parsed messages will raise unexpected errors.
@end deffn

@geindex header_max_count() (email.policy.Policy method)
@anchor{library/email policy email policy Policy header_max_count}@anchor{2cad}
@deffn {Method} header_max_count (name)

Return the maximum allowed number of headers named @emph{name}.

Called when a header is added to an @ref{7e4,,EmailMessage}
or @ref{7e3,,Message} object.  If the returned value is not
@code{0} or @code{None}, and there are already a number of headers with the
name @emph{name} greater than or equal to the value returned, a
@ref{17c,,ValueError} is raised.

Because the default behavior of @code{Message.__setitem__} is to append the
value to the list of headers, it is easy to create duplicate headers
without realizing it.  This method allows certain headers to be limited
in the number of instances of that header that may be added to a
@code{Message} programmatically.  (The limit is not observed by the parser,
which will faithfully produce as many headers as exist in the message
being parsed.)

The default implementation returns @code{None} for all header names.
@end deffn

@geindex header_source_parse() (email.policy.Policy method)
@anchor{library/email policy email policy Policy header_source_parse}@anchor{2cae}
@deffn {Method} header_source_parse (sourcelines)

The email package calls this method with a list of strings, each string
ending with the line separation characters found in the source being
parsed.  The first line includes the field header name and separator.
All whitespace in the source is preserved.  The method should return the
@code{(name, value)} tuple that is to be stored in the @code{Message} to
represent the parsed header.

If an implementation wishes to retain compatibility with the existing
email package policies, @emph{name} should be the case preserved name (all
characters up to the ‘@code{:}’ separator), while @emph{value} should be the
unfolded value (all line separator characters removed, but whitespace
kept intact), stripped of leading whitespace.

@emph{sourcelines} may contain surrogateescaped binary data.

There is no default implementation
@end deffn

@geindex header_store_parse() (email.policy.Policy method)
@anchor{library/email policy email policy Policy header_store_parse}@anchor{2caf}
@deffn {Method} header_store_parse (name, value)

The email package calls this method with the name and value provided by
the application program when the application program is modifying a
@code{Message} programmatically (as opposed to a @code{Message} created by a
parser).  The method should return the @code{(name, value)} tuple that is to
be stored in the @code{Message} to represent the header.

If an implementation wishes to retain compatibility with the existing
email package policies, the @emph{name} and @emph{value} should be strings or
string subclasses that do not change the content of the passed in
arguments.

There is no default implementation
@end deffn

@geindex header_fetch_parse() (email.policy.Policy method)
@anchor{library/email policy email policy Policy header_fetch_parse}@anchor{2cb0}
@deffn {Method} header_fetch_parse (name, value)

The email package calls this method with the @emph{name} and @emph{value} currently
stored in the @code{Message} when that header is requested by the
application program, and whatever the method returns is what is passed
back to the application as the value of the header being retrieved.
Note that there may be more than one header with the same name stored in
the @code{Message}; the method is passed the specific name and value of the
header destined to be returned to the application.

@emph{value} may contain surrogateescaped binary data.  There should be no
surrogateescaped binary data in the value returned by the method.

There is no default implementation
@end deffn

@geindex fold() (email.policy.Policy method)
@anchor{library/email policy email policy Policy fold}@anchor{2cb1}
@deffn {Method} fold (name, value)

The email package calls this method with the @emph{name} and @emph{value} currently
stored in the @code{Message} for a given header.  The method should return a
string that represents that header “folded” correctly (according to the
policy settings) by composing the @emph{name} with the @emph{value} and inserting
@ref{2ca8,,linesep} characters at the appropriate places.  See 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html}
for a discussion of the rules for folding email headers.

@emph{value} may contain surrogateescaped binary data.  There should be no
surrogateescaped binary data in the string returned by the method.
@end deffn

@geindex fold_binary() (email.policy.Policy method)
@anchor{library/email policy email policy Policy fold_binary}@anchor{2ca9}
@deffn {Method} fold_binary (name, value)

The same as @ref{2cb1,,fold()}, except that the returned value should be a
bytes object rather than a string.

@emph{value} may contain surrogateescaped binary data.  These could be
converted back into binary data in the returned bytes object.
@end deffn
@end deffn

@geindex EmailPolicy (class in email.policy)
@anchor{library/email policy email policy EmailPolicy}@anchor{c48}
@deffn {Class} email.policy.EmailPolicy (**kw)

This concrete @ref{c44,,Policy} provides behavior that is intended to be fully
compliant with the current email RFCs.  These include (but are not limited
to) 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html}, 
@geindex RFC; RFC 2047
RFC 2047@footnote{https://tools.ietf.org/html/rfc2047.html}, and the current MIME RFCs.

This policy adds new header parsing and folding algorithms.  Instead of
simple strings, headers are @code{str} subclasses with attributes that depend
on the type of the field.  The parsing and folding algorithm fully implement
@geindex RFC; RFC 2047
RFC 2047@footnote{https://tools.ietf.org/html/rfc2047.html} and 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html}.

The default value for the @ref{7e1,,message_factory}
attribute is @ref{7e4,,EmailMessage}.

In addition to the settable attributes listed above that apply to all
policies, this policy adds the following additional attributes:


New in version 3.6: @footnote{@w{(1)} 
Originally added in 3.3 as a @ref{bd6,,provisional feature}.
}


@geindex utf8 (email.policy.EmailPolicy attribute)
@anchor{library/email policy email policy EmailPolicy utf8}@anchor{957}
@deffn {Attribute} utf8

If @code{False}, follow 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html}, supporting non-ASCII characters in
headers by encoding them as “encoded words”.  If @code{True}, follow
@geindex RFC; RFC 6532
RFC 6532@footnote{https://tools.ietf.org/html/rfc6532.html} and use @code{utf-8} encoding for headers.  Messages
formatted in this way may be passed to SMTP servers that support
the @code{SMTPUTF8} extension (
@geindex RFC; RFC 6531
RFC 6531@footnote{https://tools.ietf.org/html/rfc6531.html}).
@end deffn

@geindex refold_source (email.policy.EmailPolicy attribute)
@anchor{library/email policy email policy EmailPolicy refold_source}@anchor{2cb2}
@deffn {Attribute} refold_source

If the value for a header in the @code{Message} object originated from a
@ref{6f,,parser} (as opposed to being set by a program), this
attribute indicates whether or not a generator should refold that value
when transforming the message back into serialized form.  The possible
values are:


@multitable {xxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{none}

@tab

all source values use original folding

@item

@code{long}

@tab

source values that have any line that is longer than
@code{max_line_length} will be refolded

@item

@code{all}

@tab

all values are refolded.

@end multitable


The default is @code{long}.
@end deffn

@geindex header_factory (email.policy.EmailPolicy attribute)
@anchor{library/email policy email policy EmailPolicy header_factory}@anchor{2cb3}
@deffn {Attribute} header_factory

A callable that takes two arguments, @code{name} and @code{value}, where
@code{name} is a header field name and @code{value} is an unfolded header field
value, and returns a string subclass that represents that header.  A
default @code{header_factory} (see @ref{6b,,headerregistry}) is provided
that supports custom parsing for the various address and date 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html}
header field types, and the major MIME header field stypes.  Support for
additional custom parsing will be added in the future.
@end deffn

@geindex content_manager (email.policy.EmailPolicy attribute)
@anchor{library/email policy email policy EmailPolicy content_manager}@anchor{ab8}
@deffn {Attribute} content_manager

An object with at least two methods: get_content and set_content.  When
the @ref{2c7f,,get_content()} or
@ref{1116,,set_content()} method of an
@ref{7e4,,EmailMessage} object is called, it calls the
corresponding method of this object, passing it the message object as its
first argument, and any arguments or keywords that were passed to it as
additional arguments.  By default @code{content_manager} is set to
@ref{2cb4,,raw_data_manager}.


New in version 3.4.

@end deffn

The class provides the following concrete implementations of the abstract
methods of @ref{c44,,Policy}:

@geindex header_max_count() (email.policy.EmailPolicy method)
@anchor{library/email policy email policy EmailPolicy header_max_count}@anchor{2cb5}
@deffn {Method} header_max_count (name)

Returns the value of the
@ref{2cb6,,max_count} attribute of the
specialized class used to represent the header with the given name.
@end deffn

@geindex header_source_parse() (email.policy.EmailPolicy method)
@anchor{library/email policy email policy EmailPolicy header_source_parse}@anchor{2cb7}
@deffn {Method} header_source_parse (sourcelines)

The name is parsed as everything up to the ‘@code{:}’ and returned
unmodified.  The value is determined by stripping leading whitespace off
the remainder of the first line, joining all subsequent lines together,
and stripping any trailing carriage return or linefeed characters.
@end deffn

@geindex header_store_parse() (email.policy.EmailPolicy method)
@anchor{library/email policy email policy EmailPolicy header_store_parse}@anchor{2cb8}
@deffn {Method} header_store_parse (name, value)

The name is returned unchanged.  If the input value has a @code{name}
attribute and it matches @emph{name} ignoring case, the value is returned
unchanged.  Otherwise the @emph{name} and @emph{value} are passed to
@code{header_factory}, and the resulting header object is returned as
the value.  In this case a @code{ValueError} is raised if the input value
contains CR or LF characters.
@end deffn

@geindex header_fetch_parse() (email.policy.EmailPolicy method)
@anchor{library/email policy email policy EmailPolicy header_fetch_parse}@anchor{2cb9}
@deffn {Method} header_fetch_parse (name, value)

If the value has a @code{name} attribute, it is returned to unmodified.
Otherwise the @emph{name}, and the @emph{value} with any CR or LF characters
removed, are passed to the @code{header_factory}, and the resulting
header object is returned.  Any surrogateescaped bytes get turned into
the unicode unknown-character glyph.
@end deffn

@geindex fold() (email.policy.EmailPolicy method)
@anchor{library/email policy email policy EmailPolicy fold}@anchor{2cba}
@deffn {Method} fold (name, value)

Header folding is controlled by the @ref{2cb2,,refold_source} policy setting.
A value is considered to be a ‘source value’ if and only if it does not
have a @code{name} attribute (having a @code{name} attribute means it is a
header object of some sort).  If a source value needs to be refolded
according to the policy, it is converted into a header object by
passing the @emph{name} and the @emph{value} with any CR and LF characters removed
to the @code{header_factory}.  Folding of a header object is done by
calling its @code{fold} method with the current policy.

Source values are split into lines using @ref{1ab1,,splitlines()}.  If
the value is not to be refolded, the lines are rejoined using the
@code{linesep} from the policy and returned.  The exception is lines
containing non-ascii binary data.  In that case the value is refolded
regardless of the @code{refold_source} setting, which causes the binary data
to be CTE encoded using the @code{unknown-8bit} charset.
@end deffn

@geindex fold_binary() (email.policy.EmailPolicy method)
@anchor{library/email policy email policy EmailPolicy fold_binary}@anchor{2cbb}
@deffn {Method} fold_binary (name, value)

The same as @ref{2cba,,fold()} if @ref{2ca0,,cte_type} is @code{7bit}, except
that the returned value is bytes.

If @ref{2ca0,,cte_type} is @code{8bit}, non-ASCII binary data is
converted back
into bytes.  Headers with binary data are not refolded, regardless of the
@code{refold_header} setting, since there is no way to know whether the
binary data consists of single byte characters or multibyte characters.
@end deffn
@end deffn

The following instances of @ref{c48,,EmailPolicy} provide defaults suitable for
specific application domains.  Note that in the future the behavior of these
instances (in particular the @code{HTTP} instance) may be adjusted to conform even
more closely to the RFCs relevant to their domains.

@geindex default (in module email.policy)
@anchor{library/email policy email policy default}@anchor{2c54}
@deffn {Data} email.policy.default

An instance of @code{EmailPolicy} with all defaults unchanged.  This policy
uses the standard Python @code{\n} line endings rather than the RFC-correct
@code{\r\n}.
@end deffn

@geindex SMTP (in module email.policy)
@anchor{library/email policy email policy SMTP}@anchor{2cbc}
@deffn {Data} email.policy.SMTP

Suitable for serializing messages in conformance with the email RFCs.
Like @code{default}, but with @code{linesep} set to @code{\r\n}, which is RFC
compliant.
@end deffn

@geindex SMTPUTF8 (in module email.policy)
@anchor{library/email policy email policy SMTPUTF8}@anchor{2cbd}
@deffn {Data} email.policy.SMTPUTF8

The same as @code{SMTP} except that @ref{957,,utf8} is @code{True}.
Useful for serializing messages to a message store without using encoded
words in the headers.  Should only be used for SMTP transmission if the
sender or recipient addresses have non-ASCII characters (the
@ref{9c4,,smtplib.SMTP.send_message()} method handles this automatically).
@end deffn

@geindex HTTP (in module email.policy)
@anchor{library/email policy email policy HTTP}@anchor{2cbe}
@deffn {Data} email.policy.HTTP

Suitable for serializing headers with for use in HTTP traffic.  Like
@code{SMTP} except that @code{max_line_length} is set to @code{None} (unlimited).
@end deffn

@geindex strict (in module email.policy)
@anchor{library/email policy email policy strict}@anchor{2cbf}
@deffn {Data} email.policy.strict

Convenience instance.  The same as @code{default} except that
@code{raise_on_defect} is set to @code{True}.  This allows any policy to be made
strict by writing:

@example
somepolicy + policy.strict
@end example
@end deffn

With all of these @ref{c48,,EmailPolicies}, the effective API of
the email package is changed from the Python 3.2 API in the following ways:

@quotation


@itemize *

@item 
Setting a header on a @ref{7e3,,Message} results in that
header being parsed and a header object created.

@item 
Fetching a header value from a @ref{7e3,,Message} results
in that header being parsed and a header object created and
returned.

@item 
Any header object, or any header that is refolded due to the
policy settings, is folded using an algorithm that fully implements the
RFC folding algorithms, including knowing where encoded words are required
and allowed.
@end itemize
@end quotation

From the application view, this means that any header obtained through the
@ref{7e4,,EmailMessage} is a header object with extra
attributes, whose string value is the fully decoded unicode value of the
header.  Likewise, a header may be assigned a new value, or a new header
created, using a unicode string, and the policy will take care of converting
the unicode string into the correct RFC encoded form.

The header objects and their attributes are described in
@ref{6b,,headerregistry}.

@geindex Compat32 (class in email.policy)
@anchor{library/email policy email policy Compat32}@anchor{c45}
@deffn {Class} email.policy.Compat32 (**kw)

This concrete @ref{c44,,Policy} is the backward compatibility policy.  It
replicates the behavior of the email package in Python 3.2.  The
@ref{70,,policy} module also defines an instance of this class,
@ref{7e2,,compat32}, that is used as the default policy.  Thus the default
behavior of the email package is to maintain compatibility with Python 3.2.

The following attributes have values that are different from the
@ref{c44,,Policy} default:

@geindex mangle_from_ (email.policy.Compat32 attribute)
@anchor{library/email policy email policy Compat32 mangle_from_}@anchor{2cc0}
@deffn {Attribute} mangle_from_

The default is @code{True}.
@end deffn

The class provides the following concrete implementations of the
abstract methods of @ref{c44,,Policy}:

@geindex header_source_parse() (email.policy.Compat32 method)
@anchor{library/email policy email policy Compat32 header_source_parse}@anchor{2cc1}
@deffn {Method} header_source_parse (sourcelines)

The name is parsed as everything up to the ‘@code{:}’ and returned
unmodified.  The value is determined by stripping leading whitespace off
the remainder of the first line, joining all subsequent lines together,
and stripping any trailing carriage return or linefeed characters.
@end deffn

@geindex header_store_parse() (email.policy.Compat32 method)
@anchor{library/email policy email policy Compat32 header_store_parse}@anchor{2cc2}
@deffn {Method} header_store_parse (name, value)

The name and value are returned unmodified.
@end deffn

@geindex header_fetch_parse() (email.policy.Compat32 method)
@anchor{library/email policy email policy Compat32 header_fetch_parse}@anchor{2cc3}
@deffn {Method} header_fetch_parse (name, value)

If the value contains binary data, it is converted into a
@ref{1118,,Header} object using the @code{unknown-8bit} charset.
Otherwise it is returned unmodified.
@end deffn

@geindex fold() (email.policy.Compat32 method)
@anchor{library/email policy email policy Compat32 fold}@anchor{2cc4}
@deffn {Method} fold (name, value)

Headers are folded using the @ref{1118,,Header} folding
algorithm, which preserves existing line breaks in the value, and wraps
each resulting line to the @code{max_line_length}.  Non-ASCII binary data are
CTE encoded using the @code{unknown-8bit} charset.
@end deffn

@geindex fold_binary() (email.policy.Compat32 method)
@anchor{library/email policy email policy Compat32 fold_binary}@anchor{2cc5}
@deffn {Method} fold_binary (name, value)

Headers are folded using the @ref{1118,,Header} folding
algorithm, which preserves existing line breaks in the value, and wraps
each resulting line to the @code{max_line_length}.  If @code{cte_type} is
@code{7bit}, non-ascii binary data is CTE encoded using the @code{unknown-8bit}
charset.  Otherwise the original source header is used, with its existing
line breaks and any (RFC invalid) binary data it may contain.
@end deffn
@end deffn

@geindex compat32 (in module email.policy)
@anchor{library/email policy email policy compat32}@anchor{7e2}
@deffn {Data} email.policy.compat32

An instance of @ref{c45,,Compat32}, providing  backward compatibility with the
behavior of the email package in Python 3.2.
@end deffn

@node email errors Exception and Defect classes,email headerregistry Custom Header Objects,email policy Policy Objects,email — An email and MIME handling package
@anchor{library/email errors doc}@anchor{2cc6}@anchor{library/email errors email-errors-exception-and-defect-classes}@anchor{2cc7}@anchor{library/email errors module-email errors}@anchor{68}
@subsubsection @code{email.errors}: Exception and Defect classes


@geindex module; email.errors

@strong{Source code:} Lib/email/errors.py@footnote{https://github.com/python/cpython/tree/main/Lib/email/errors.py}


__________________________________________________________________


The following exception classes are defined in the @ref{68,,email.errors} module:

@geindex MessageError
@anchor{library/email errors email errors MessageError}@anchor{2cc8}
@deffn {Exception} email.errors.MessageError

This is the base class for all exceptions that the @ref{64,,email} package can
raise.  It is derived from the standard @ref{4d8,,Exception} class and defines no
additional methods.
@end deffn

@geindex MessageParseError
@anchor{library/email errors email errors MessageParseError}@anchor{2cc9}
@deffn {Exception} email.errors.MessageParseError

This is the base class for exceptions raised by the
@ref{d43,,Parser} class.  It is derived from
@ref{2cc8,,MessageError}.  This class is also used internally by the parser used
by @ref{6b,,headerregistry}.
@end deffn

@geindex HeaderParseError
@anchor{library/email errors email errors HeaderParseError}@anchor{2c75}
@deffn {Exception} email.errors.HeaderParseError

Raised under some error conditions when parsing the 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html} headers of a
message, this class is derived from @ref{2cc9,,MessageParseError}.  The
@ref{2c74,,set_boundary()} method will raise this
error if the content type is unknown when the method is called.
@ref{1118,,Header} may raise this error for certain base64
decoding errors, and when an attempt is made to create a header that appears
to contain an embedded header (that is, there is what is supposed to be a
continuation line that has no leading whitespace and looks like a header).
@end deffn

@geindex BoundaryError
@anchor{library/email errors email errors BoundaryError}@anchor{2cca}
@deffn {Exception} email.errors.BoundaryError

Deprecated and no longer used.
@end deffn

@geindex MultipartConversionError
@anchor{library/email errors email errors MultipartConversionError}@anchor{2ccb}
@deffn {Exception} email.errors.MultipartConversionError

Raised when a payload is added to a @ref{7e3,,Message} object
using @code{add_payload()}, but the payload is already a scalar and the
message’s @code{Content-Type} main type is not either
@code{multipart} or missing.  @ref{2ccb,,MultipartConversionError} multiply
inherits from @ref{2cc8,,MessageError} and the built-in @ref{162,,TypeError}.

Since @code{Message.add_payload()} is deprecated, this exception is rarely
raised in practice.  However the exception may also be raised if the
@ref{2c86,,attach()}
method is called on an instance of a class derived from
@ref{2ccc,,MIMENonMultipart} (e.g.
@ref{2ccd,,MIMEImage}).
@end deffn

Here is the list of the defects that the @ref{2c90,,FeedParser}
can find while parsing messages.  Note that the defects are added to the message
where the problem was found, so for example, if a message nested inside a
@code{multipart/alternative} had a malformed header, that nested message
object would have a defect, but the containing messages would not.

All defect classes are subclassed from @code{email.errors.MessageDefect}.


@itemize *

@item 
@code{NoBoundaryInMultipartDefect} – A message claimed to be a multipart,
but had no @code{boundary} parameter.

@item 
@code{StartBoundaryNotFoundDefect} – The start boundary claimed in the
@code{Content-Type} header was never found.

@item 
@code{CloseBoundaryNotFoundDefect} – A start boundary was found, but
no corresponding close boundary was ever found.


New in version 3.3.


@item 
@code{FirstHeaderLineIsContinuationDefect} – The message had a continuation
line as its first header line.

@item 
@code{MisplacedEnvelopeHeaderDefect} - A “Unix From” header was found in the
middle of a header block.

@item 
@code{MissingHeaderBodySeparatorDefect} - A line was found while parsing
headers that had no leading white space but contained no ‘:’.  Parsing
continues assuming that the line represents the first line of the body.


New in version 3.3.


@item 
@code{MalformedHeaderDefect} – A header was found that was missing a colon,
or was otherwise malformed.


Deprecated since version 3.3: This defect has not been used for several Python versions.


@item 
@code{MultipartInvariantViolationDefect} – A message claimed to be a
@code{multipart}, but no subparts were found.  Note that when a message
has this defect, its @ref{2c79,,is_multipart()} method may
return @code{False} even though its content type claims to be @code{multipart}.

@item 
@code{InvalidBase64PaddingDefect} – When decoding a block of base64
encoded bytes, the padding was not correct.  Enough padding is added to
perform the decode, but the resulting decoded bytes may be invalid.

@item 
@code{InvalidBase64CharactersDefect} – When decoding a block of base64
encoded bytes, characters outside the base64 alphabet were encountered.
The characters are ignored, but the resulting decoded bytes may be invalid.

@item 
@code{InvalidBase64LengthDefect} – When decoding a block of base64 encoded
bytes, the number of non-padding base64 characters was invalid (1 more than
a multiple of 4).  The encoded block was kept as-is.

@item 
@code{InvalidDateDefect} – When decoding an invalid or unparsable date field.
The original value is kept as-is.
@end itemize

@node email headerregistry Custom Header Objects,email contentmanager Managing MIME Content,email errors Exception and Defect classes,email — An email and MIME handling package
@anchor{library/email headerregistry doc}@anchor{2cce}@anchor{library/email headerregistry email-headerregistry-custom-header-objects}@anchor{2ccf}@anchor{library/email headerregistry module-email headerregistry}@anchor{6b}
@subsubsection @code{email.headerregistry}: Custom Header Objects


@geindex module; email.headerregistry

@strong{Source code:} Lib/email/headerregistry.py@footnote{https://github.com/python/cpython/tree/main/Lib/email/headerregistry.py}


__________________________________________________________________



New in version 3.6: @footnote{@w{(1)} 
Originally added in 3.3 as a @ref{bd6,,provisional module}
}


Headers are represented by customized subclasses of @ref{2ef,,str}.  The
particular class used to represent a given header is determined by the
@ref{2cb3,,header_factory} of the @ref{70,,policy} in
effect when the headers are created.  This section documents the particular
@code{header_factory} implemented by the email package for handling 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html}
compliant email messages, which not only provides customized header objects for
various header types, but also provides an extension mechanism for applications
to add their own custom header types.

When using any of the policy objects derived from
@ref{c48,,EmailPolicy}, all headers are produced by
@ref{2cd0,,HeaderRegistry} and have @ref{2c62,,BaseHeader} as their last base
class.  Each header class has an additional base class that is determined by
the type of the header.  For example, many headers have the class
@ref{2cd1,,UnstructuredHeader} as their other base class.  The specialized second
class for a header is determined by the name of the header, using a lookup
table stored in the @ref{2cd0,,HeaderRegistry}.  All of this is managed
transparently for the typical application program, but interfaces are provided
for modifying the default behavior for use by more complex applications.

The sections below first document the header base classes and their attributes,
followed by the API for modifying the behavior of @ref{2cd0,,HeaderRegistry}, and
finally the support classes used to represent the data parsed from structured
headers.

@geindex BaseHeader (class in email.headerregistry)
@anchor{library/email headerregistry email headerregistry BaseHeader}@anchor{2c62}
@deffn {Class} email.headerregistry.BaseHeader (name, value)

@emph{name} and @emph{value} are passed to @code{BaseHeader} from the
@ref{2cb3,,header_factory} call.  The string value of
any header object is the @emph{value} fully decoded to unicode.

This base class defines the following read-only properties:

@geindex name (email.headerregistry.BaseHeader attribute)
@anchor{library/email headerregistry email headerregistry BaseHeader name}@anchor{2cd2}
@deffn {Attribute} name

The name of the header (the portion of the field before the ‘:’).  This
is exactly the value passed in the
@ref{2cb3,,header_factory} call for @emph{name}; that
is, case is preserved.
@end deffn

@geindex defects (email.headerregistry.BaseHeader attribute)
@anchor{library/email headerregistry email headerregistry BaseHeader defects}@anchor{2cd3}
@deffn {Attribute} defects

A tuple of @code{HeaderDefect} instances reporting any
RFC compliance problems found during parsing.  The email package tries to
be complete about detecting compliance issues.  See the @ref{68,,errors}
module for a discussion of the types of defects that may be reported.
@end deffn

@geindex max_count (email.headerregistry.BaseHeader attribute)
@anchor{library/email headerregistry email headerregistry BaseHeader max_count}@anchor{2cb6}
@deffn {Attribute} max_count

The maximum number of headers of this type that can have the same
@code{name}.  A value of @code{None} means unlimited.  The @code{BaseHeader} value
for this attribute is @code{None}; it is expected that specialized header
classes will override this value as needed.
@end deffn

@code{BaseHeader} also provides the following method, which is called by the
email library code and should not in general be called by application
programs:

@geindex fold() (email.headerregistry.BaseHeader method)
@anchor{library/email headerregistry email headerregistry BaseHeader fold}@anchor{2cd4}
@deffn {Method} fold (*, policy)

Return a string containing @ref{2ca8,,linesep}
characters as required to correctly fold the header according to
@emph{policy}.  A @ref{2ca0,,cte_type} of @code{8bit} will be
treated as if it were @code{7bit}, since headers may not contain arbitrary
binary data.  If @ref{957,,utf8} is @code{False},
non-ASCII data will be 
@geindex RFC; RFC 2047
RFC 2047@footnote{https://tools.ietf.org/html/rfc2047.html} encoded.
@end deffn

@code{BaseHeader} by itself cannot be used to create a header object.  It
defines a protocol that each specialized header cooperates with in order to
produce the header object.  Specifically, @code{BaseHeader} requires that
the specialized class provide a @ref{254,,classmethod()} named @code{parse}.  This
method is called as follows:

@example
parse(string, kwds)
@end example

@code{kwds} is a dictionary containing one pre-initialized key, @code{defects}.
@code{defects} is an empty list.  The parse method should append any detected
defects to this list.  On return, the @code{kwds} dictionary @emph{must} contain
values for at least the keys @code{decoded} and @code{defects}.  @code{decoded}
should be the string value for the header (that is, the header value fully
decoded to unicode).  The parse method should assume that @emph{string} may
contain content-transfer-encoded parts, but should correctly handle all valid
unicode characters as well so that it can parse un-encoded header values.

@code{BaseHeader}’s @code{__new__} then creates the header instance, and calls its
@code{init} method.  The specialized class only needs to provide an @code{init}
method if it wishes to set additional attributes beyond those provided by
@code{BaseHeader} itself.  Such an @code{init} method should look like this:

@example
def init(self, /, *args, **kw):
    self._myattr = kw.pop('myattr')
    super().init(*args, **kw)
@end example

That is, anything extra that the specialized class puts in to the @code{kwds}
dictionary should be removed and handled, and the remaining contents of
@code{kw} (and @code{args}) passed to the @code{BaseHeader} @code{init} method.
@end deffn

@geindex UnstructuredHeader (class in email.headerregistry)
@anchor{library/email headerregistry email headerregistry UnstructuredHeader}@anchor{2cd1}
@deffn {Class} email.headerregistry.UnstructuredHeader

An “unstructured” header is the default type of header in 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html}.
Any header that does not have a specified syntax is treated as
unstructured.  The classic example of an unstructured header is the
@code{Subject} header.

In 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html}, an unstructured header is a run of arbitrary text in the
ASCII character set.  
@geindex RFC; RFC 2047
RFC 2047@footnote{https://tools.ietf.org/html/rfc2047.html}, however, has an 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html} compatible
mechanism for encoding non-ASCII text as ASCII characters within a header
value.  When a @emph{value} containing encoded words is passed to the
constructor, the @code{UnstructuredHeader} parser converts such encoded words
into unicode, following the 
@geindex RFC; RFC 2047
RFC 2047@footnote{https://tools.ietf.org/html/rfc2047.html} rules for unstructured text.  The
parser uses heuristics to attempt to decode certain non-compliant encoded
words.  Defects are registered in such cases, as well as defects for issues
such as invalid characters within the encoded words or the non-encoded text.

This header type provides no additional attributes.
@end deffn

@geindex DateHeader (class in email.headerregistry)
@anchor{library/email headerregistry email headerregistry DateHeader}@anchor{2cd5}
@deffn {Class} email.headerregistry.DateHeader

@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html} specifies a very specific format for dates within email headers.
The @code{DateHeader} parser recognizes that date format, as well as
recognizing a number of variant forms that are sometimes found “in the
wild”.

This header type provides the following additional attributes:

@geindex datetime (email.headerregistry.DateHeader attribute)
@anchor{library/email headerregistry email headerregistry DateHeader datetime}@anchor{2cd6}
@deffn {Attribute} datetime

If the header value can be recognized as a valid date of one form or
another, this attribute will contain a @ref{363,,datetime}
instance representing that date.  If the timezone of the input date is
specified as @code{-0000} (indicating it is in UTC but contains no
information about the source timezone), then @ref{2cd6,,datetime} will be a
naive @ref{363,,datetime}.  If a specific timezone offset is
found (including @cite{+0000}), then @ref{2cd6,,datetime} will contain an aware
@code{datetime} that uses @ref{c51,,datetime.timezone} to record the timezone
offset.
@end deffn

The @code{decoded} value of the header is determined by formatting the
@code{datetime} according to the 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html} rules; that is, it is set to:

@example
email.utils.format_datetime(self.datetime)
@end example

When creating a @code{DateHeader}, @emph{value} may be
@ref{363,,datetime} instance.  This means, for example, that
the following code is valid and does what one would expect:

@example
msg['Date'] = datetime(2011, 7, 15, 21)
@end example

Because this is a naive @code{datetime} it will be interpreted as a UTC
timestamp, and the resulting value will have a timezone of @code{-0000}.  Much
more useful is to use the @ref{c50,,localtime()} function from the
@ref{71,,utils} module:

@example
msg['Date'] = utils.localtime()
@end example

This example sets the date header to the current time and date using
the current timezone offset.
@end deffn

@geindex AddressHeader (class in email.headerregistry)
@anchor{library/email headerregistry email headerregistry AddressHeader}@anchor{2cd7}
@deffn {Class} email.headerregistry.AddressHeader

Address headers are one of the most complex structured header types.
The @code{AddressHeader} class provides a generic interface to any address
header.

This header type provides the following additional attributes:

@geindex groups (email.headerregistry.AddressHeader attribute)
@anchor{library/email headerregistry email headerregistry AddressHeader groups}@anchor{2cd8}
@deffn {Attribute} groups

A tuple of @ref{2cd9,,Group} objects encoding the
addresses and groups found in the header value.  Addresses that are
not part of a group are represented in this list as single-address
@code{Groups} whose @ref{2cda,,display_name} is @code{None}.
@end deffn

@geindex addresses (email.headerregistry.AddressHeader attribute)
@anchor{library/email headerregistry email headerregistry AddressHeader addresses}@anchor{2cdb}
@deffn {Attribute} addresses

A tuple of @ref{2cdc,,Address} objects encoding all
of the individual addresses from the header value.  If the header value
contains any groups, the individual addresses from the group are included
in the list at the point where the group occurs in the value (that is,
the list of addresses is “flattened” into a one dimensional list).
@end deffn

The @code{decoded} value of the header will have all encoded words decoded to
unicode.  @ref{72,,idna} encoded domain names are also decoded to
unicode.  The @code{decoded} value is set by @ref{1a8b,,join}ing the
@ref{2ef,,str} value of the elements of the @code{groups} attribute with @code{',
'}.

A list of @ref{2cdc,,Address} and @ref{2cd9,,Group} objects in any combination
may be used to set the value of an address header.  @code{Group} objects whose
@code{display_name} is @code{None} will be interpreted as single addresses, which
allows an address list to be copied with groups intact by using the list
obtained from the @code{groups} attribute of the source header.
@end deffn

@geindex SingleAddressHeader (class in email.headerregistry)
@anchor{library/email headerregistry email headerregistry SingleAddressHeader}@anchor{2cdd}
@deffn {Class} email.headerregistry.SingleAddressHeader

A subclass of @ref{2cd7,,AddressHeader} that adds one
additional attribute:

@geindex address (email.headerregistry.SingleAddressHeader attribute)
@anchor{library/email headerregistry email headerregistry SingleAddressHeader address}@anchor{2cde}
@deffn {Attribute} address

The single address encoded by the header value.  If the header value
actually contains more than one address (which would be a violation of
the RFC under the default @ref{70,,policy}), accessing this attribute
will result in a @ref{17c,,ValueError}.
@end deffn
@end deffn

Many of the above classes also have a @code{Unique} variant (for example,
@code{UniqueUnstructuredHeader}).  The only difference is that in the @code{Unique}
variant, @ref{2cb6,,max_count} is set to 1.

@geindex MIMEVersionHeader (class in email.headerregistry)
@anchor{library/email headerregistry email headerregistry MIMEVersionHeader}@anchor{2cdf}
@deffn {Class} email.headerregistry.MIMEVersionHeader

There is really only one valid value for the @code{MIME-Version}
header, and that is @code{1.0}.  For future proofing, this header class
supports other valid version numbers.  If a version number has a valid value
per 
@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html}, then the header object will have non-@code{None} values for
the following attributes:

@geindex version (email.headerregistry.MIMEVersionHeader attribute)
@anchor{library/email headerregistry email headerregistry MIMEVersionHeader version}@anchor{2ce0}
@deffn {Attribute} version

The version number as a string, with any whitespace and/or comments
removed.
@end deffn

@geindex major (email.headerregistry.MIMEVersionHeader attribute)
@anchor{library/email headerregistry email headerregistry MIMEVersionHeader major}@anchor{2ce1}
@deffn {Attribute} major

The major version number as an integer
@end deffn

@geindex minor (email.headerregistry.MIMEVersionHeader attribute)
@anchor{library/email headerregistry email headerregistry MIMEVersionHeader minor}@anchor{2ce2}
@deffn {Attribute} minor

The minor version number as an integer
@end deffn
@end deffn

@geindex ParameterizedMIMEHeader (class in email.headerregistry)
@anchor{library/email headerregistry email headerregistry ParameterizedMIMEHeader}@anchor{2ce3}
@deffn {Class} email.headerregistry.ParameterizedMIMEHeader

MIME headers all start with the prefix ‘Content-’.  Each specific header has
a certain value, described under the class for that header.  Some can
also take a list of supplemental parameters, which have a common format.
This class serves as a base for all the MIME headers that take parameters.

@geindex params (email.headerregistry.ParameterizedMIMEHeader attribute)
@anchor{library/email headerregistry email headerregistry ParameterizedMIMEHeader params}@anchor{2ce4}
@deffn {Attribute} params

A dictionary mapping parameter names to parameter values.
@end deffn
@end deffn

@geindex ContentTypeHeader (class in email.headerregistry)
@anchor{library/email headerregistry email headerregistry ContentTypeHeader}@anchor{2ce5}
@deffn {Class} email.headerregistry.ContentTypeHeader

A @ref{2ce3,,ParameterizedMIMEHeader} class that handles the
@code{Content-Type} header.

@geindex content_type (email.headerregistry.ContentTypeHeader attribute)
@anchor{library/email headerregistry email headerregistry ContentTypeHeader content_type}@anchor{2ce6}
@deffn {Attribute} content_type

The content type string, in the form @code{maintype/subtype}.
@end deffn

@geindex maintype (email.headerregistry.ContentTypeHeader attribute)
@anchor{library/email headerregistry email headerregistry ContentTypeHeader maintype}@anchor{2ce7}
@deffn {Attribute} maintype
@end deffn

@geindex subtype (email.headerregistry.ContentTypeHeader attribute)
@anchor{library/email headerregistry email headerregistry ContentTypeHeader subtype}@anchor{2ce8}
@deffn {Attribute} subtype
@end deffn
@end deffn

@geindex ContentDispositionHeader (class in email.headerregistry)
@anchor{library/email headerregistry email headerregistry ContentDispositionHeader}@anchor{2ce9}
@deffn {Class} email.headerregistry.ContentDispositionHeader

A @ref{2ce3,,ParameterizedMIMEHeader} class that handles the
@code{Content-Disposition} header.

@geindex content_disposition (email.headerregistry.ContentDispositionHeader attribute)
@anchor{library/email headerregistry email headerregistry ContentDispositionHeader content_disposition}@anchor{2cea}
@deffn {Attribute} content_disposition

@code{inline} and @code{attachment} are the only valid values in common use.
@end deffn
@end deffn

@geindex ContentTransferEncoding (class in email.headerregistry)
@anchor{library/email headerregistry email headerregistry ContentTransferEncoding}@anchor{2ceb}
@deffn {Class} email.headerregistry.ContentTransferEncoding

Handles the @code{Content-Transfer-Encoding} header.

@geindex cte (email.headerregistry.ContentTransferEncoding attribute)
@anchor{library/email headerregistry email headerregistry ContentTransferEncoding cte}@anchor{2cec}
@deffn {Attribute} cte

Valid values are @code{7bit}, @code{8bit}, @code{base64}, and
@code{quoted-printable}.  See 
@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html} for more information.
@end deffn
@end deffn

@geindex HeaderRegistry (class in email.headerregistry)
@anchor{library/email headerregistry email headerregistry HeaderRegistry}@anchor{2cd0}
@deffn {Class} email.headerregistry.HeaderRegistry (base_class=BaseHeader, default_class=UnstructuredHeader, use_default_map=True)

This is the factory used by @ref{c48,,EmailPolicy} by default.
@code{HeaderRegistry} builds the class used to create a header instance
dynamically, using @emph{base_class} and a specialized class retrieved from a
registry that it holds.  When a given header name does not appear in the
registry, the class specified by @emph{default_class} is used as the specialized
class.  When @emph{use_default_map} is @code{True} (the default), the standard
mapping of header names to classes is copied in to the registry during
initialization.  @emph{base_class} is always the last class in the generated
class’s @code{__bases__} list.

The default mappings are:

@quotation

@*subject: 
UniqueUnstructuredHeader


@*date: 
UniqueDateHeader


@*resent-date: 
DateHeader


@*orig-date: 
UniqueDateHeader


@*sender: 
UniqueSingleAddressHeader


@*resent-sender: 
SingleAddressHeader


@*to: 
UniqueAddressHeader


@*resent-to: 
AddressHeader


@*cc: 
UniqueAddressHeader


@*resent-cc: 
AddressHeader


@*bcc: 
UniqueAddressHeader


@*resent-bcc: 
AddressHeader


@*from: 
UniqueAddressHeader


@*resent-from: 
AddressHeader


@*reply-to: 
UniqueAddressHeader


@*mime-version: 
MIMEVersionHeader


@*content-type: 
ContentTypeHeader


@*content-disposition: 
ContentDispositionHeader


@*content-transfer-encoding: 
ContentTransferEncodingHeader


@*message-id: 
MessageIDHeader

@end quotation

@code{HeaderRegistry} has the following methods:

@geindex map_to_type() (email.headerregistry.HeaderRegistry method)
@anchor{library/email headerregistry email headerregistry HeaderRegistry map_to_type}@anchor{2ced}
@deffn {Method} map_to_type (self, name, cls)

@emph{name} is the name of the header to be mapped.  It will be converted to
lower case in the registry.  @emph{cls} is the specialized class to be used,
along with @emph{base_class}, to create the class used to instantiate headers
that match @emph{name}.
@end deffn

@geindex __getitem__() (email.headerregistry.HeaderRegistry method)
@anchor{library/email headerregistry email headerregistry HeaderRegistry __getitem__}@anchor{2cee}
@deffn {Method} __getitem__ (name)

Construct and return a class to handle creating a @emph{name} header.
@end deffn

@geindex __call__() (email.headerregistry.HeaderRegistry method)
@anchor{library/email headerregistry email headerregistry HeaderRegistry __call__}@anchor{2cef}
@deffn {Method} __call__ (name, value)

Retrieves the specialized header associated with @emph{name} from the
registry (using @emph{default_class} if @emph{name} does not appear in the
registry) and composes it with @emph{base_class} to produce a class,
calls the constructed class’s constructor, passing it the same
argument list, and finally returns the class instance created thereby.
@end deffn
@end deffn

The following classes are the classes used to represent data parsed from
structured headers and can, in general, be used by an application program to
construct structured values to assign to specific headers.

@geindex Address (class in email.headerregistry)
@anchor{library/email headerregistry email headerregistry Address}@anchor{2cdc}
@deffn {Class} email.headerregistry.Address (display_name='@w{'}, username='@w{'}, domain='@w{'}, addr_spec=None)

The class used to represent an email address.  The general form of an
address is:

@example
[display_name] <username@@domain>
@end example

or:

@example
username@@domain
@end example

where each part must conform to specific syntax rules spelled out in
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html}.

As a convenience @emph{addr_spec} can be specified instead of @emph{username} and
@emph{domain}, in which case @emph{username} and @emph{domain} will be parsed from the
@emph{addr_spec}.  An @emph{addr_spec} must be a properly RFC quoted string; if it is
not @code{Address} will raise an error.  Unicode characters are allowed and
will be property encoded when serialized.  However, per the RFCs, unicode is
@emph{not} allowed in the username portion of the address.

@geindex display_name (email.headerregistry.Address attribute)
@anchor{library/email headerregistry email headerregistry Address display_name}@anchor{2cf0}
@deffn {Attribute} display_name

The display name portion of the address, if any, with all quoting
removed.  If the address does not have a display name, this attribute
will be an empty string.
@end deffn

@geindex username (email.headerregistry.Address attribute)
@anchor{library/email headerregistry email headerregistry Address username}@anchor{2cf1}
@deffn {Attribute} username

The @code{username} portion of the address, with all quoting removed.
@end deffn

@geindex domain (email.headerregistry.Address attribute)
@anchor{library/email headerregistry email headerregistry Address domain}@anchor{2cf2}
@deffn {Attribute} domain

The @code{domain} portion of the address.
@end deffn

@geindex addr_spec (email.headerregistry.Address attribute)
@anchor{library/email headerregistry email headerregistry Address addr_spec}@anchor{2cf3}
@deffn {Attribute} addr_spec

The @code{username@@domain} portion of the address, correctly quoted
for use as a bare address (the second form shown above).  This
attribute is not mutable.
@end deffn

@geindex __str__() (email.headerregistry.Address method)
@anchor{library/email headerregistry email headerregistry Address __str__}@anchor{2cf4}
@deffn {Method} __str__ ()

The @code{str} value of the object is the address quoted according to
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html} rules, but with no Content Transfer Encoding of any non-ASCII
characters.
@end deffn

To support SMTP (
@geindex RFC; RFC 5321
RFC 5321@footnote{https://tools.ietf.org/html/rfc5321.html}), @code{Address} handles one special case: if
@code{username} and @code{domain} are both the empty string (or @code{None}), then
the string value of the @code{Address} is @code{<>}.
@end deffn

@geindex Group (class in email.headerregistry)
@anchor{library/email headerregistry email headerregistry Group}@anchor{2cd9}
@deffn {Class} email.headerregistry.Group (display_name=None, addresses=None)

The class used to represent an address group.  The general form of an
address group is:

@example
display_name: [address-list];
@end example

As a convenience for processing lists of addresses that consist of a mixture
of groups and single addresses, a @code{Group} may also be used to represent
single addresses that are not part of a group by setting @emph{display_name} to
@code{None} and providing a list of the single address as @emph{addresses}.

@geindex display_name (email.headerregistry.Group attribute)
@anchor{library/email headerregistry email headerregistry Group display_name}@anchor{2cda}
@deffn {Attribute} display_name

The @code{display_name} of the group.  If it is @code{None} and there is
exactly one @code{Address} in @code{addresses}, then the @code{Group} represents a
single address that is not in a group.
@end deffn

@geindex addresses (email.headerregistry.Group attribute)
@anchor{library/email headerregistry email headerregistry Group addresses}@anchor{2cf5}
@deffn {Attribute} addresses

A possibly empty tuple of @ref{2cdc,,Address} objects representing the
addresses in the group.
@end deffn

@geindex __str__() (email.headerregistry.Group method)
@anchor{library/email headerregistry email headerregistry Group __str__}@anchor{2cf6}
@deffn {Method} __str__ ()

The @code{str} value of a @code{Group} is formatted according to 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html},
but with no Content Transfer Encoding of any non-ASCII characters.  If
@code{display_name} is none and there is a single @code{Address} in the
@code{addresses} list, the @code{str} value will be the same as the @code{str} of
that single @code{Address}.
@end deffn
@end deffn

@node email contentmanager Managing MIME Content,email Examples,email headerregistry Custom Header Objects,email — An email and MIME handling package
@anchor{library/email contentmanager doc}@anchor{2cf7}@anchor{library/email contentmanager email-contentmanager-managing-mime-content}@anchor{2cf8}@anchor{library/email contentmanager module-email contentmanager}@anchor{66}
@subsubsection @code{email.contentmanager}: Managing MIME Content


@geindex module; email.contentmanager

@strong{Source code:} Lib/email/contentmanager.py@footnote{https://github.com/python/cpython/tree/main/Lib/email/contentmanager.py}


__________________________________________________________________



New in version 3.6: @footnote{@w{(1)} 
Originally added in 3.4 as a @ref{bd6,,provisional module}
}


@geindex ContentManager (class in email.contentmanager)
@anchor{library/email contentmanager email contentmanager ContentManager}@anchor{2cf9}
@deffn {Class} email.contentmanager.ContentManager

Base class for content managers.  Provides the standard registry mechanisms
to register converters between MIME content and other representations, as
well as the @code{get_content} and @code{set_content} dispatch methods.

@geindex get_content() (email.contentmanager.ContentManager method)
@anchor{library/email contentmanager email contentmanager ContentManager get_content}@anchor{2c80}
@deffn {Method} get_content (msg, *args, **kw)

Look up a handler function based on the @code{mimetype} of @emph{msg} (see next
paragraph), call it, passing through all arguments, and return the result
of the call.  The expectation is that the handler will extract the
payload from @emph{msg} and return an object that encodes information about
the extracted data.

To find the handler, look for the following keys in the registry,
stopping with the first one found:

@quotation


@itemize *

@item 
the string representing the full MIME type (@code{maintype/subtype})

@item 
the string representing the @code{maintype}

@item 
the empty string
@end itemize
@end quotation

If none of these keys produce a handler, raise a @ref{5bb,,KeyError} for the
full MIME type.
@end deffn

@geindex set_content() (email.contentmanager.ContentManager method)
@anchor{library/email contentmanager email contentmanager ContentManager set_content}@anchor{2c81}
@deffn {Method} set_content (msg, obj, *args, **kw)

If the @code{maintype} is @code{multipart}, raise a @ref{162,,TypeError}; otherwise
look up a handler function based on the type of @emph{obj} (see next
paragraph), call @ref{2c8a,,clear_content()} on the
@emph{msg}, and call the handler function, passing through all arguments.  The
expectation is that the handler will transform and store @emph{obj} into
@emph{msg}, possibly making other changes to @emph{msg} as well, such as adding
various MIME headers to encode information needed to interpret the stored
data.

To find the handler, obtain the type of @emph{obj} (@code{typ = type(obj)}), and
look for the following keys in the registry, stopping with the first one
found:

@quotation


@itemize *

@item 
the type itself (@code{typ})

@item 
the type’s fully qualified name (@code{typ.__module__ + '.' +
typ.__qualname__}).

@item 
the type’s qualname (@code{typ.__qualname__})

@item 
the type’s name (@code{typ.__name__}).
@end itemize
@end quotation

If none of the above match, repeat all of the checks above for each of
the types in the @ref{2cfa,,MRO} (@code{typ.__mro__}).  Finally, if no other key
yields a handler, check for a handler for the key @code{None}.  If there is
no handler for @code{None}, raise a @ref{5bb,,KeyError} for the fully
qualified name of the type.

Also add a @code{MIME-Version} header if one is not present (see
also @ref{ab7,,MIMEPart}).
@end deffn

@geindex add_get_handler() (email.contentmanager.ContentManager method)
@anchor{library/email contentmanager email contentmanager ContentManager add_get_handler}@anchor{2cfb}
@deffn {Method} add_get_handler (key, handler)

Record the function @emph{handler} as the handler for @emph{key}.  For the possible
values of @emph{key}, see @ref{2cfc,,get_content()}.
@end deffn

@geindex add_set_handler() (email.contentmanager.ContentManager method)
@anchor{library/email contentmanager email contentmanager ContentManager add_set_handler}@anchor{2cfd}
@deffn {Method} add_set_handler (typekey, handler)

Record @emph{handler} as the function to call when an object of a type
matching @emph{typekey} is passed to @ref{2cfe,,set_content()}.  For the possible
values of @emph{typekey}, see @ref{2cfe,,set_content()}.
@end deffn
@end deffn

@menu
* Content Manager Instances:: 

@end menu

@node Content Manager Instances,,,email contentmanager Managing MIME Content
@anchor{library/email contentmanager content-manager-instances}@anchor{2cff}
@subsubsection Content Manager Instances


Currently the email package provides only one concrete content manager,
@ref{2cb4,,raw_data_manager}, although more may be added in the future.
@ref{2cb4,,raw_data_manager} is the
@ref{ab8,,content_manager} provided by
@ref{c48,,EmailPolicy} and its derivatives.

@geindex raw_data_manager (in module email.contentmanager)
@anchor{library/email contentmanager email contentmanager raw_data_manager}@anchor{2cb4}
@deffn {Data} email.contentmanager.raw_data_manager

This content manager provides only a minimum interface beyond that provided
by @ref{7e3,,Message} itself:  it deals only with text, raw
byte strings, and @ref{7e3,,Message} objects.  Nevertheless, it
provides significant advantages compared to the base API: @code{get_content} on
a text part will return a unicode string without the application needing to
manually decode it, @code{set_content} provides a rich set of options for
controlling the headers added to a part and controlling the content transfer
encoding, and it enables the use of the various @code{add_} methods, thereby
simplifying the creation of multipart messages.

@geindex get_content() (in module email.contentmanager)
@anchor{library/email contentmanager email contentmanager get_content}@anchor{2cfc}
@deffn {Method} email.contentmanager.get_content (msg, errors='replace')

Return the payload of the part as either a string (for @code{text} parts), an
@ref{7e4,,EmailMessage} object (for @code{message/rfc822}
parts), or a @code{bytes} object (for all other non-multipart types).  Raise
a @ref{5bb,,KeyError} if called on a @code{multipart}.  If the part is a
@code{text} part and @emph{errors} is specified, use it as the error handler when
decoding the payload to unicode.  The default error handler is
@code{replace}.
@end deffn

@geindex set_content() (in module email.contentmanager)
@anchor{library/email contentmanager email contentmanager set_content}@anchor{2cfe}
@deffn {Method} email.contentmanager.set_content (msg, <'str'>, subtype="plain", charset='utf-8', cte=None, disposition=None, filename=None, cid=None, params=None, headers=None)

@deffnx {Method} email.contentmanager.set_content (msg, <'bytes'>, maintype, subtype, cte="base64", disposition=None, filename=None, cid=None, params=None, headers=None)

@deffnx {Method} email.contentmanager.set_content (msg, <'EmailMessage'>, cte=None, disposition=None, filename=None, cid=None, params=None, headers=None)

Add headers and payload to @emph{msg}:

Add a @code{Content-Type} header with a @code{maintype/subtype}
value.

@quotation


@itemize *

@item 
For @code{str}, set the MIME @code{maintype} to @code{text}, and set the
subtype to @emph{subtype} if it is specified, or @code{plain} if it is not.

@item 
For @code{bytes}, use the specified @emph{maintype} and @emph{subtype}, or
raise a @ref{162,,TypeError} if they are not specified.

@item 
For @ref{7e4,,EmailMessage} objects, set the maintype
to @code{message}, and set the subtype to @emph{subtype} if it is
specified or @code{rfc822} if it is not.  If @emph{subtype} is
@code{partial}, raise an error (@code{bytes} objects must be used to
construct @code{message/partial} parts).
@end itemize
@end quotation

If @emph{charset} is provided (which is valid only for @code{str}), encode the
string to bytes using the specified character set.  The default is
@code{utf-8}.  If the specified @emph{charset} is a known alias for a standard
MIME charset name, use the standard charset instead.

If @emph{cte} is set, encode the payload using the specified content transfer
encoding, and set the @code{Content-Transfer-Encoding} header to
that value.  Possible values for @emph{cte} are @code{quoted-printable},
@code{base64}, @code{7bit}, @code{8bit}, and @code{binary}.  If the input cannot be
encoded in the specified encoding (for example, specifying a @emph{cte} of
@code{7bit} for an input that contains non-ASCII values), raise a
@ref{17c,,ValueError}.

@quotation


@itemize *

@item 
For @code{str} objects, if @emph{cte} is not set use heuristics to
determine the most compact encoding.

@item 
For @ref{7e4,,EmailMessage}, per 
@geindex RFC; RFC 2046
RFC 2046@footnote{https://tools.ietf.org/html/rfc2046.html}, raise
an error if a @emph{cte} of @code{quoted-printable} or @code{base64} is
requested for @emph{subtype} @code{rfc822}, and for any @emph{cte} other than
@code{7bit} for @emph{subtype} @code{external-body}.  For
@code{message/rfc822}, use @code{8bit} if @emph{cte} is not specified.  For
all other values of @emph{subtype}, use @code{7bit}.
@end itemize
@end quotation

@cartouche
@quotation Note 
A @emph{cte} of @code{binary} does not actually work correctly yet.
The @code{EmailMessage} object as modified by @code{set_content} is
correct, but @ref{d73,,BytesGenerator} does not
serialize it correctly.
@end quotation
@end cartouche

If @emph{disposition} is set, use it as the value of the
@code{Content-Disposition} header.  If not specified, and
@emph{filename} is specified, add the header with the value @code{attachment}.
If @emph{disposition} is not specified and @emph{filename} is also not specified,
do not add the header.  The only valid values for @emph{disposition} are
@code{attachment} and @code{inline}.

If @emph{filename} is specified, use it as the value of the @code{filename}
parameter of the @code{Content-Disposition} header.

If @emph{cid} is specified, add a @code{Content-ID} header with
@emph{cid} as its value.

If @emph{params} is specified, iterate its @code{items} method and use the
resulting @code{(key, value)} pairs to set additional parameters on the
@code{Content-Type} header.

If @emph{headers} is specified and is a list of strings of the form
@code{headername: headervalue} or a list of @code{header} objects
(distinguished from strings by having a @code{name} attribute), add the
headers to @emph{msg}.
@end deffn
@end deffn

@node email Examples,email message Message Representing an email message using the compat32 API,email contentmanager Managing MIME Content,email — An email and MIME handling package
@anchor{library/email examples doc}@anchor{2d00}@anchor{library/email examples email-examples}@anchor{ab9}@anchor{library/email examples id1}@anchor{2d01}
@subsubsection @code{email}: Examples


Here are a few examples of how to use the @ref{64,,email} package to read, write,
and send simple email messages, as well as more complex MIME messages.

First, let’s see how to create and send a simple text message (both the
text content and the addresses may contain unicode characters):

@example
# Import smtplib for the actual sending function
import smtplib

# Import the email modules we'll need
from email.message import EmailMessage

# Open the plain text file whose name is in textfile for reading.
with open(textfile) as fp:
    # Create a text/plain message
    msg = EmailMessage()
    msg.set_content(fp.read())

# me == the sender's email address
# you == the recipient's email address
msg['Subject'] = f'The contents of @{textfile@}'
msg['From'] = me
msg['To'] = you

# Send the message via our own SMTP server.
s = smtplib.SMTP('localhost')
s.send_message(msg)
s.quit()
@end example

Parsing 
@geindex RFC; RFC 822
RFC 822@footnote{https://tools.ietf.org/html/rfc822.html} headers can easily be done by the using the classes
from the @ref{6f,,parser} module:

@example
# Import the email modules we'll need
from email.parser import BytesParser, Parser
from email.policy import default

# If the e-mail headers are in a file, uncomment these two lines:
# with open(messagefile, 'rb') as fp:
#     headers = BytesParser(policy=default).parse(fp)

#  Or for parsing headers in a string (this is an uncommon operation), use:
headers = Parser(policy=default).parsestr(
        'From: Foo Bar <user@@example.com>\n'
        'To: <someone_else@@example.com>\n'
        'Subject: Test message\n'
        '\n'
        'Body would go here\n')

#  Now the header items can be accessed as a dictionary:
print('To: @{@}'.format(headers['to']))
print('From: @{@}'.format(headers['from']))
print('Subject: @{@}'.format(headers['subject']))

# You can also access the parts of the addresses:
print('Recipient username: @{@}'.format(headers['to'].addresses[0].username))
print('Sender name: @{@}'.format(headers['from'].addresses[0].display_name))
@end example

Here’s an example of how to send a MIME message containing a bunch of family
pictures that may be residing in a directory:

@example
# Import smtplib for the actual sending function
import smtplib

# And imghdr to find the types of our images
import imghdr

# Here are the email package modules we'll need
from email.message import EmailMessage

# Create the container email message.
msg = EmailMessage()
msg['Subject'] = 'Our family reunion'
# me == the sender's email address
# family = the list of all recipients' email addresses
msg['From'] = me
msg['To'] = ', '.join(family)
msg.preamble = 'You will not see this in a MIME-aware mail reader.\n'

# Open the files in binary mode.  Use imghdr to figure out the
# MIME subtype for each specific image.
for file in pngfiles:
    with open(file, 'rb') as fp:
        img_data = fp.read()
    msg.add_attachment(img_data, maintype='image',
                                 subtype=imghdr.what(None, img_data))

# Send the email via our own SMTP server.
with smtplib.SMTP('localhost') as s:
    s.send_message(msg)
@end example

Here’s an example of how to send the entire contents of a directory as an email
message: @footnote{@w{(1)} 
Thanks to Matthew Dixon Cowles for the original inspiration and examples.
}

@example
#!/usr/bin/env python3

"""Send the contents of a directory as a MIME message."""

import os
import smtplib
# For guessing MIME type based on file name extension
import mimetypes

from argparse import ArgumentParser

from email.message import EmailMessage
from email.policy import SMTP


def main():
    parser = ArgumentParser(description="""\
Send the contents of a directory as a MIME message.
Unless the -o option is given, the email is sent by forwarding to your local
SMTP server, which then does the normal delivery process.  Your local machine
must be running an SMTP server.
""")
    parser.add_argument('-d', '--directory',
                        help="""Mail the contents of the specified directory,
                        otherwise use the current directory.  Only the regular
                        files in the directory are sent, and we don't recurse to
                        subdirectories.""")
    parser.add_argument('-o', '--output',
                        metavar='FILE',
                        help="""Print the composed message to FILE instead of
                        sending the message to the SMTP server.""")
    parser.add_argument('-s', '--sender', required=True,
                        help='The value of the From: header (required)')
    parser.add_argument('-r', '--recipient', required=True,
                        action='append', metavar='RECIPIENT',
                        default=[], dest='recipients',
                        help='A To: header value (at least one required)')
    args = parser.parse_args()
    directory = args.directory
    if not directory:
        directory = '.'
    # Create the message
    msg = EmailMessage()
    msg['Subject'] = f'Contents of directory @{os.path.abspath(directory)@}'
    msg['To'] = ', '.join(args.recipients)
    msg['From'] = args.sender
    msg.preamble = 'You will not see this in a MIME-aware mail reader.\n'

    for filename in os.listdir(directory):
        path = os.path.join(directory, filename)
        if not os.path.isfile(path):
            continue
        # Guess the content type based on the file's extension.  Encoding
        # will be ignored, although we should check for simple things like
        # gzip'd or compressed files.
        ctype, encoding = mimetypes.guess_type(path)
        if ctype is None or encoding is not None:
            # No guess could be made, or the file is encoded (compressed), so
            # use a generic bag-of-bits type.
            ctype = 'application/octet-stream'
        maintype, subtype = ctype.split('/', 1)
        with open(path, 'rb') as fp:
            msg.add_attachment(fp.read(),
                               maintype=maintype,
                               subtype=subtype,
                               filename=filename)
    # Now send or store the message
    if args.output:
        with open(args.output, 'wb') as fp:
            fp.write(msg.as_bytes(policy=SMTP))
    else:
        with smtplib.SMTP('localhost') as s:
            s.send_message(msg)


if __name__ == '__main__':
    main()
@end example

Here’s an example of how to unpack a MIME message like the one
above, into a directory of files:

@example
#!/usr/bin/env python3

"""Unpack a MIME message into a directory of files."""

import os
import email
import mimetypes

from email.policy import default

from argparse import ArgumentParser


def main():
    parser = ArgumentParser(description="""\
Unpack a MIME message into a directory of files.
""")
    parser.add_argument('-d', '--directory', required=True,
                        help="""Unpack the MIME message into the named
                        directory, which will be created if it doesn't already
                        exist.""")
    parser.add_argument('msgfile')
    args = parser.parse_args()

    with open(args.msgfile, 'rb') as fp:
        msg = email.message_from_binary_file(fp, policy=default)

    try:
        os.mkdir(args.directory)
    except FileExistsError:
        pass

    counter = 1
    for part in msg.walk():
        # multipart/* are just containers
        if part.get_content_maintype() == 'multipart':
            continue
        # Applications should really sanitize the given filename so that an
        # email message can't be used to overwrite important files
        filename = part.get_filename()
        if not filename:
            ext = mimetypes.guess_extension(part.get_content_type())
            if not ext:
                # Use a generic bag-of-bits extension
                ext = '.bin'
            filename = f'part-@{counter:03d@}@{ext@}'
        counter += 1
        with open(os.path.join(args.directory, filename), 'wb') as fp:
            fp.write(part.get_payload(decode=True))


if __name__ == '__main__':
    main()
@end example

Here’s an example of how to create an HTML message with an alternative plain
text version.  To make things a bit more interesting, we include a related
image in the html part, and we save a copy of what we are going to send to
disk, as well as sending it.

@example
#!/usr/bin/env python3

import smtplib

from email.message import EmailMessage
from email.headerregistry import Address
from email.utils import make_msgid

# Create the base text message.
msg = EmailMessage()
msg['Subject'] = "Ayons asperges pour le déjeuner"
msg['From'] = Address("Pepé Le Pew", "pepe", "example.com")
msg['To'] = (Address("Penelope Pussycat", "penelope", "example.com"),
             Address("Fabrette Pussycat", "fabrette", "example.com"))
msg.set_content("""\
Salut!

Cela ressemble à un excellent recipie[1] déjeuner.

[1] http://www.yummly.com/recipe/Roasted-Asparagus-Epicurious-203718

--Pepé
""")

# Add the html version.  This converts the message into a multipart/alternative
# container, with the original text message as the first part and the new html
# message as the second part.
asparagus_cid = make_msgid()
msg.add_alternative("""\
<html>
  <head></head>
  <body>
    <p>Salut!</p>
    <p>Cela ressemble à un excellent
        <a href="http://www.yummly.com/recipe/Roasted-Asparagus-Epicurious-203718">
            recipie
        </a> déjeuner.
    </p>
    <img src="cid:@{asparagus_cid@}" />
  </body>
</html>
""".format(asparagus_cid=asparagus_cid[1:-1]), subtype='html')
# note that we needed to peel the <> off the msgid for use in the html.

# Now add the related image to the html part.
with open("roasted-asparagus.jpg", 'rb') as img:
    msg.get_payload()[1].add_related(img.read(), 'image', 'jpeg',
                                     cid=asparagus_cid)

# Make a local copy of what we are going to send.
with open('outgoing.msg', 'wb') as f:
    f.write(bytes(msg))

# Send the message via local SMTP server.
with smtplib.SMTP('localhost') as s:
    s.send_message(msg)
@end example

If we were sent the message from the last example, here is one way we could
process it:

@example
import os
import sys
import tempfile
import mimetypes
import webbrowser

# Import the email modules we'll need
from email import policy
from email.parser import BytesParser

# An imaginary module that would make this work and be safe.
from imaginary import magic_html_parser

# In a real program you'd get the filename from the arguments.
with open('outgoing.msg', 'rb') as fp:
    msg = BytesParser(policy=policy.default).parse(fp)

# Now the header items can be accessed as a dictionary, and any non-ASCII will
# be converted to unicode:
print('To:', msg['to'])
print('From:', msg['from'])
print('Subject:', msg['subject'])

# If we want to print a preview of the message content, we can extract whatever
# the least formatted payload is and print the first three lines.  Of course,
# if the message has no plain text part printing the first three lines of html
# is probably useless, but this is just a conceptual example.
simplest = msg.get_body(preferencelist=('plain', 'html'))
print()
print('@w{'}.join(simplest.get_content().splitlines(keepends=True)[:3]))

ans = input("View full message?")
if ans.lower()[0] == 'n':
    sys.exit()

# We can extract the richest alternative in order to display it:
richest = msg.get_body()
partfiles = @{@}
if richest['content-type'].maintype == 'text':
    if richest['content-type'].subtype == 'plain':
        for line in richest.get_content().splitlines():
            print(line)
        sys.exit()
    elif richest['content-type'].subtype == 'html':
        body = richest
    else:
        print("Don't know how to display @{@}".format(richest.get_content_type()))
        sys.exit()
elif richest['content-type'].content_type == 'multipart/related':
    body = richest.get_body(preferencelist=('html'))
    for part in richest.iter_attachments():
        fn = part.get_filename()
        if fn:
            extension = os.path.splitext(part.get_filename())[1]
        else:
            extension = mimetypes.guess_extension(part.get_content_type())
        with tempfile.NamedTemporaryFile(suffix=extension, delete=False) as f:
            f.write(part.get_content())
            # again strip the <> to go from email form of cid to html form.
            partfiles[part['content-id'][1:-1]] = f.name
else:
    print("Don't know how to display @{@}".format(richest.get_content_type()))
    sys.exit()
with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
    # The magic_html_parser has to rewrite the href="cid:...." attributes to
    # point to the filenames in partfiles.  It also has to do a safety-sanitize
    # of the html.  It could be written using html.parser.
    f.write(magic_html_parser(body.get_content(), partfiles))
webbrowser.open(f.name)
os.remove(f.name)
for fn in partfiles.values():
    os.remove(fn)

# Of course, there are lots of email messages that could break this simple
# minded program, but it will handle the most common ones.
@end example

Up to the prompt, the output from the above is:

@example
To: Penelope Pussycat <penelope@@example.com>, Fabrette Pussycat <fabrette@@example.com>
From: Pepé Le Pew <pepe@@example.com>
Subject: Ayons asperges pour le déjeuner

Salut!

Cela ressemble à un excellent recipie[1] déjeuner.
@end example

Legacy API:

@node email message Message Representing an email message using the compat32 API,email mime Creating email and MIME objects from scratch,email Examples,email — An email and MIME handling package
@anchor{library/email compat32-message doc}@anchor{2d02}@anchor{library/email compat32-message compat32-message}@anchor{2c53}@anchor{library/email compat32-message email-message-message-representing-an-email-message-using-the-compat32-api}@anchor{2d03}
@subsubsection @code{email.message.Message}: Representing an email message using the @code{compat32} API


The @ref{7e3,,Message} class is very similar to the
@ref{7e4,,EmailMessage} class, without the methods added by that
class, and with the default behavior of certain other methods being slightly
different.  We also document here some methods that, while supported by the
@ref{7e4,,EmailMessage} class, are not recommended unless you are
dealing with legacy code.

The philosophy and structure of the two classes is otherwise the same.

This document describes the behavior under the default (for @ref{7e3,,Message})
policy @ref{c45,,Compat32}.  If you are going to use another policy,
you should be using the @ref{7e4,,EmailMessage} class instead.

An email message consists of @emph{headers} and a @emph{payload}.  Headers must be
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html} style names and values, where the field name and value are
separated by a colon.  The colon is not part of either the field name or the
field value.  The payload may be a simple text message, or a binary object, or
a structured sequence of sub-messages each with their own set of headers and
their own payload.  The latter type of payload is indicated by the message
having a MIME type such as @code{multipart/*} or
@code{message/rfc822}.

The conceptual model provided by a @ref{7e3,,Message} object is that of an
ordered dictionary of headers with additional methods for accessing both
specialized information from the headers, for accessing the payload, for
generating a serialized version of the message, and for recursively walking
over the object tree.  Note that duplicate headers are supported but special
methods must be used to access them.

The @ref{7e3,,Message} pseudo-dictionary is indexed by the header names, which
must be ASCII values.  The values of the dictionary are strings that are
supposed to contain only ASCII characters; there is some special handling for
non-ASCII input, but it doesn’t always produce the correct results.  Headers
are stored and returned in case-preserving form, but field names are matched
case-insensitively.  There may also be a single envelope header, also known as
the @emph{Unix-From} header or the @code{From_} header.  The @emph{payload} is either a
string or bytes, in the case of simple message objects, or a list of
@ref{7e3,,Message} objects, for MIME container documents (e.g.
@code{multipart/*} and @code{message/rfc822}).

Here are the methods of the @ref{7e3,,Message} class:

@geindex Message (class in email.message)
@anchor{library/email compat32-message email message Message}@anchor{7e3}
@deffn {Class} email.message.Message (policy=compat32)

If @emph{policy} is specified (it must be an instance of a @ref{70,,policy}
class) use the rules it specifies to update and serialize the representation
of the message.  If @emph{policy} is not set, use the @ref{c45,,compat32} policy, which maintains backward compatibility with
the Python 3.2 version of the email package.  For more information see the
@ref{70,,policy} documentation.


Changed in version 3.3: The @emph{policy} keyword argument was added.


@geindex as_string() (email.message.Message method)
@anchor{library/email compat32-message email message Message as_string}@anchor{ab3}
@deffn {Method} as_string (unixfrom=False, maxheaderlen=0, policy=None)

Return the entire message flattened as a string.  When optional @emph{unixfrom}
is true, the envelope header is included in the returned string.
@emph{unixfrom} defaults to @code{False}.  For backward compatibility reasons,
@emph{maxheaderlen} defaults to @code{0}, so if you want a different value you
must override it explicitly (the value specified for @emph{max_line_length} in
the policy will be ignored by this method).  The @emph{policy} argument may be
used to override the default policy obtained from the message instance.
This can be used to control some of the formatting produced by the
method, since the specified @emph{policy} will be passed to the @code{Generator}.

Flattening the message may trigger changes to the @ref{7e3,,Message} if
defaults need to be filled in to complete the transformation to a string
(for example, MIME boundaries may be generated or modified).

Note that this method is provided as a convenience and may not always
format the message the way you want.  For example, by default it does
not do the mangling of lines that begin with @code{From} that is
required by the unix mbox format.  For more flexibility, instantiate a
@ref{d72,,Generator} instance and use its
@ref{2ca2,,flatten()} method directly.  For example:

@example
from io import StringIO
from email.generator import Generator
fp = StringIO()
g = Generator(fp, mangle_from_=True, maxheaderlen=60)
g.flatten(msg)
text = fp.getvalue()
@end example

If the message object contains binary data that is not encoded according
to RFC standards, the non-compliant data will be replaced by unicode
“unknown character” code points.  (See also @ref{ab4,,as_bytes()} and
@ref{d73,,BytesGenerator}.)


Changed in version 3.4: the @emph{policy} keyword argument was added.

@end deffn

@geindex __str__() (email.message.Message method)
@anchor{library/email compat32-message email message Message __str__}@anchor{2d04}
@deffn {Method} __str__ ()

Equivalent to @ref{ab3,,as_string()}.  Allows @code{str(msg)} to produce a
string containing the formatted message.
@end deffn

@geindex as_bytes() (email.message.Message method)
@anchor{library/email compat32-message email message Message as_bytes}@anchor{ab4}
@deffn {Method} as_bytes (unixfrom=False, policy=None)

Return the entire message flattened as a bytes object.  When optional
@emph{unixfrom} is true, the envelope header is included in the returned
string.  @emph{unixfrom} defaults to @code{False}.  The @emph{policy} argument may be
used to override the default policy obtained from the message instance.
This can be used to control some of the formatting produced by the
method, since the specified @emph{policy} will be passed to the
@code{BytesGenerator}.

Flattening the message may trigger changes to the @ref{7e3,,Message} if
defaults need to be filled in to complete the transformation to a string
(for example, MIME boundaries may be generated or modified).

Note that this method is provided as a convenience and may not always
format the message the way you want.  For example, by default it does
not do the mangling of lines that begin with @code{From} that is
required by the unix mbox format.  For more flexibility, instantiate a
@ref{d73,,BytesGenerator} instance and use its
@ref{2c9e,,flatten()} method directly.
For example:

@example
from io import BytesIO
from email.generator import BytesGenerator
fp = BytesIO()
g = BytesGenerator(fp, mangle_from_=True, maxheaderlen=60)
g.flatten(msg)
text = fp.getvalue()
@end example


New in version 3.4.

@end deffn

@geindex __bytes__() (email.message.Message method)
@anchor{library/email compat32-message email message Message __bytes__}@anchor{ab5}
@deffn {Method} __bytes__ ()

Equivalent to @ref{ab4,,as_bytes()}.  Allows @code{bytes(msg)} to produce a
bytes object containing the formatted message.


New in version 3.4.

@end deffn

@geindex is_multipart() (email.message.Message method)
@anchor{library/email compat32-message email message Message is_multipart}@anchor{2c79}
@deffn {Method} is_multipart ()

Return @code{True} if the message’s payload is a list of
sub-@ref{7e3,,Message} objects, otherwise return @code{False}.  When
@ref{2c79,,is_multipart()} returns @code{False}, the payload should be a string
object (which might be a CTE encoded binary payload).  (Note that
@ref{2c79,,is_multipart()} returning @code{True} does not necessarily mean that
“msg.get_content_maintype() == ‘multipart’” will return the @code{True}.
For example, @code{is_multipart} will return @code{True} when the
@ref{7e3,,Message} is of type @code{message/rfc822}.)
@end deffn

@geindex set_unixfrom() (email.message.Message method)
@anchor{library/email compat32-message email message Message set_unixfrom}@anchor{2d05}
@deffn {Method} set_unixfrom (unixfrom)

Set the message’s envelope header to @emph{unixfrom}, which should be a string.
@end deffn

@geindex get_unixfrom() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_unixfrom}@anchor{2d06}
@deffn {Method} get_unixfrom ()

Return the message’s envelope header.  Defaults to @code{None} if the
envelope header was never set.
@end deffn

@geindex attach() (email.message.Message method)
@anchor{library/email compat32-message email message Message attach}@anchor{2c86}
@deffn {Method} attach (payload)

Add the given @emph{payload} to the current payload, which must be @code{None} or
a list of @ref{7e3,,Message} objects before the call. After the call, the
payload will always be a list of @ref{7e3,,Message} objects.  If you want to
set the payload to a scalar object (e.g. a string), use
@ref{2d07,,set_payload()} instead.

This is a legacy method.  On the
@code{EmailMessage} class its functionality is
replaced by @ref{1116,,set_content()} and the
related @code{make} and @code{add} methods.
@end deffn

@geindex get_payload() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_payload}@anchor{d71}
@deffn {Method} get_payload (i=None, decode=False)

Return the current payload, which will be a list of
@ref{7e3,,Message} objects when @ref{2c79,,is_multipart()} is @code{True}, or a
string when @ref{2c79,,is_multipart()} is @code{False}.  If the payload is a list
and you mutate the list object, you modify the message’s payload in place.

With optional argument @emph{i}, @ref{d71,,get_payload()} will return the @emph{i}-th
element of the payload, counting from zero, if @ref{2c79,,is_multipart()} is
@code{True}.  An @ref{1066,,IndexError} will be raised if @emph{i} is less than 0 or
greater than or equal to the number of items in the payload.  If the
payload is a string (i.e.  @ref{2c79,,is_multipart()} is @code{False}) and @emph{i} is
given, a @ref{162,,TypeError} is raised.

Optional @emph{decode} is a flag indicating whether the payload should be
decoded or not, according to the @code{Content-Transfer-Encoding}
header. When @code{True} and the message is not a multipart, the payload will
be decoded if this header’s value is @code{quoted-printable} or @code{base64}.
If some other encoding is used, or @code{Content-Transfer-Encoding}
header is missing, the payload is
returned as-is (undecoded).  In all cases the returned value is binary
data.  If the message is a multipart and the @emph{decode} flag is @code{True},
then @code{None} is returned.  If the payload is base64 and it was not
perfectly formed (missing padding, characters outside the base64
alphabet), then an appropriate defect will be added to the message’s
defect property (@code{InvalidBase64PaddingDefect} or
@code{InvalidBase64CharactersDefect}, respectively).

When @emph{decode} is @code{False} (the default) the body is returned as a string
without decoding the @code{Content-Transfer-Encoding}.  However,
for a @code{Content-Transfer-Encoding} of 8bit, an attempt is made
to decode the original bytes using the @code{charset} specified by the
@code{Content-Type} header, using the @code{replace} error handler.
If no @code{charset} is specified, or if the @code{charset} given is not
recognized by the email package, the body is decoded using the default
ASCII charset.

This is a legacy method.  On the
@code{EmailMessage} class its functionality is
replaced by @ref{2c7f,,get_content()} and
@ref{2c7e,,iter_parts()}.
@end deffn

@geindex set_payload() (email.message.Message method)
@anchor{library/email compat32-message email message Message set_payload}@anchor{2d07}
@deffn {Method} set_payload (payload, charset=None)

Set the entire message object’s payload to @emph{payload}.  It is the client’s
responsibility to ensure the payload invariants.  Optional @emph{charset} sets
the message’s default character set; see @ref{2d08,,set_charset()} for details.

This is a legacy method.  On the
@code{EmailMessage} class its functionality is
replaced by @ref{1116,,set_content()}.
@end deffn

@geindex set_charset() (email.message.Message method)
@anchor{library/email compat32-message email message Message set_charset}@anchor{2d08}
@deffn {Method} set_charset (charset)

Set the character set of the payload to @emph{charset}, which can either be a
@ref{959,,Charset} instance (see @ref{65,,email.charset}), a
string naming a character set, or @code{None}.  If it is a string, it will
be converted to a @ref{959,,Charset} instance.  If @emph{charset}
is @code{None}, the @code{charset} parameter will be removed from the
@code{Content-Type} header (the message will not be otherwise
modified).  Anything else will generate a @ref{162,,TypeError}.

If there is no existing @code{MIME-Version} header one will be
added.  If there is no existing @code{Content-Type} header, one
will be added with a value of @code{text/plain}.  Whether the
@code{Content-Type} header already exists or not, its @code{charset}
parameter will be set to @emph{charset.output_charset}.   If
@emph{charset.input_charset} and @emph{charset.output_charset} differ, the payload
will be re-encoded to the @emph{output_charset}.  If there is no existing
@code{Content-Transfer-Encoding} header, then the payload will be
transfer-encoded, if needed, using the specified
@ref{959,,Charset}, and a header with the appropriate value
will be added.  If a @code{Content-Transfer-Encoding} header
already exists, the payload is assumed to already be correctly encoded
using that @code{Content-Transfer-Encoding} and is not modified.

This is a legacy method.  On the
@code{EmailMessage} class its functionality is
replaced by the @emph{charset} parameter of the
@code{email.emailmessage.EmailMessage.set_content()} method.
@end deffn

@geindex get_charset() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_charset}@anchor{2d09}
@deffn {Method} get_charset ()

Return the @ref{959,,Charset} instance associated with the
message’s payload.

This is a legacy method.  On the
@code{EmailMessage} class it always returns
@code{None}.
@end deffn

The following methods implement a mapping-like interface for accessing the
message’s 
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html} headers.  Note that there are some semantic differences
between these methods and a normal mapping (i.e. dictionary) interface.  For
example, in a dictionary there are no duplicate keys, but here there may be
duplicate message headers.  Also, in dictionaries there is no guaranteed
order to the keys returned by @ref{2d0a,,keys()}, but in a @ref{7e3,,Message} object,
headers are always returned in the order they appeared in the original
message, or were added to the message later.  Any header deleted and then
re-added are always appended to the end of the header list.

These semantic differences are intentional and are biased toward maximal
convenience.

Note that in all cases, any envelope header present in the message is not
included in the mapping interface.

In a model generated from bytes, any header values that (in contravention of
the RFCs) contain non-ASCII bytes will, when retrieved through this
interface, be represented as @ref{1118,,Header} objects with
a charset of @cite{unknown-8bit}.

@geindex __len__() (email.message.Message method)
@anchor{library/email compat32-message email message Message __len__}@anchor{2d0b}
@deffn {Method} __len__ ()

Return the total number of headers, including duplicates.
@end deffn

@geindex __contains__() (email.message.Message method)
@anchor{library/email compat32-message email message Message __contains__}@anchor{2d0c}
@deffn {Method} __contains__ (name)

Return @code{True} if the message object has a field named @emph{name}. Matching is
done case-insensitively and @emph{name} should not include the trailing colon.
Used for the @code{in} operator, e.g.:

@example
if 'message-id' in myMessage:
   print('Message-ID:', myMessage['message-id'])
@end example
@end deffn

@geindex __getitem__() (email.message.Message method)
@anchor{library/email compat32-message email message Message __getitem__}@anchor{2d0d}
@deffn {Method} __getitem__ (name)

Return the value of the named header field.  @emph{name} should not include the
colon field separator.  If the header is missing, @code{None} is returned; a
@ref{5bb,,KeyError} is never raised.

Note that if the named field appears more than once in the message’s
headers, exactly which of those field values will be returned is
undefined.  Use the @ref{2d0e,,get_all()} method to get the values of all the
extant named headers.
@end deffn

@geindex __setitem__() (email.message.Message method)
@anchor{library/email compat32-message email message Message __setitem__}@anchor{2d0f}
@deffn {Method} __setitem__ (name, val)

Add a header to the message with field name @emph{name} and value @emph{val}.  The
field is appended to the end of the message’s existing fields.

Note that this does @emph{not} overwrite or delete any existing header with the same
name.  If you want to ensure that the new header is the only one present in the
message with field name @emph{name}, delete the field first, e.g.:

@example
del msg['subject']
msg['subject'] = 'Python roolz!'
@end example
@end deffn

@geindex __delitem__() (email.message.Message method)
@anchor{library/email compat32-message email message Message __delitem__}@anchor{2d10}
@deffn {Method} __delitem__ (name)

Delete all occurrences of the field with name @emph{name} from the message’s
headers.  No exception is raised if the named field isn’t present in the
headers.
@end deffn

@geindex keys() (email.message.Message method)
@anchor{library/email compat32-message email message Message keys}@anchor{2d0a}
@deffn {Method} keys ()

Return a list of all the message’s header field names.
@end deffn

@geindex values() (email.message.Message method)
@anchor{library/email compat32-message email message Message values}@anchor{2d11}
@deffn {Method} values ()

Return a list of all the message’s field values.
@end deffn

@geindex items() (email.message.Message method)
@anchor{library/email compat32-message email message Message items}@anchor{2d12}
@deffn {Method} items ()

Return a list of 2-tuples containing all the message’s field headers and
values.
@end deffn

@geindex get() (email.message.Message method)
@anchor{library/email compat32-message email message Message get}@anchor{2d13}
@deffn {Method} get (name, failobj=None)

Return the value of the named header field.  This is identical to
@ref{2d0d,,__getitem__()} except that optional @emph{failobj} is returned if the
named header is missing (defaults to @code{None}).
@end deffn

Here are some additional useful methods:

@geindex get_all() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_all}@anchor{2d0e}
@deffn {Method} get_all (name, failobj=None)

Return a list of all the values for the field named @emph{name}. If there are
no such named headers in the message, @emph{failobj} is returned (defaults to
@code{None}).
@end deffn

@geindex add_header() (email.message.Message method)
@anchor{library/email compat32-message email message Message add_header}@anchor{2d14}
@deffn {Method} add_header (_name, _value, **_params)

Extended header setting.  This method is similar to @ref{2d0f,,__setitem__()}
except that additional header parameters can be provided as keyword
arguments.  @emph{_name} is the header field to add and @emph{_value} is the
@emph{primary} value for the header.

For each item in the keyword argument dictionary @emph{_params}, the key is
taken as the parameter name, with underscores converted to dashes (since
dashes are illegal in Python identifiers).  Normally, the parameter will
be added as @code{key="value"} unless the value is @code{None}, in which case
only the key will be added.  If the value contains non-ASCII characters,
it can be specified as a three tuple in the format
@code{(CHARSET, LANGUAGE, VALUE)}, where @code{CHARSET} is a string naming the
charset to be used to encode the value, @code{LANGUAGE} can usually be set
to @code{None} or the empty string (see 
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html} for other possibilities),
and @code{VALUE} is the string value containing non-ASCII code points.  If
a three tuple is not passed and the value contains non-ASCII characters,
it is automatically encoded in 
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html} format using a @code{CHARSET}
of @code{utf-8} and a @code{LANGUAGE} of @code{None}.

Here’s an example:

@example
msg.add_header('Content-Disposition', 'attachment', filename='bud.gif')
@end example

This will add a header that looks like

@example
Content-Disposition: attachment; filename="bud.gif"
@end example

An example with non-ASCII characters:

@example
msg.add_header('Content-Disposition', 'attachment',
               filename=('iso-8859-1', '@w{'}, 'Fußballer.ppt'))
@end example

Which produces

@example
Content-Disposition: attachment; filename*="iso-8859-1'@w{'}Fu%DFballer.ppt"
@end example
@end deffn

@geindex replace_header() (email.message.Message method)
@anchor{library/email compat32-message email message Message replace_header}@anchor{2d15}
@deffn {Method} replace_header (_name, _value)

Replace a header.  Replace the first header found in the message that
matches @emph{_name}, retaining header order and field name case.  If no
matching header was found, a @ref{5bb,,KeyError} is raised.
@end deffn

@geindex get_content_type() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_content_type}@anchor{2d16}
@deffn {Method} get_content_type ()

Return the message’s content type.  The returned string is coerced to
lower case of the form @code{maintype/subtype}.  If there was no
@code{Content-Type} header in the message the default type as given
by @ref{2d17,,get_default_type()} will be returned.  Since according to
@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html}, messages always have a default type, @ref{2d16,,get_content_type()}
will always return a value.

@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html} defines a message’s default type to be @code{text/plain}
unless it appears inside a @code{multipart/digest} container, in
which case it would be @code{message/rfc822}.  If the
@code{Content-Type} header has an invalid type specification,
@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html} mandates that the default type be @code{text/plain}.
@end deffn

@geindex get_content_maintype() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_content_maintype}@anchor{2d18}
@deffn {Method} get_content_maintype ()

Return the message’s main content type.  This is the @code{maintype}
part of the string returned by @ref{2d16,,get_content_type()}.
@end deffn

@geindex get_content_subtype() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_content_subtype}@anchor{2d19}
@deffn {Method} get_content_subtype ()

Return the message’s sub-content type.  This is the @code{subtype}
part of the string returned by @ref{2d16,,get_content_type()}.
@end deffn

@geindex get_default_type() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_default_type}@anchor{2d17}
@deffn {Method} get_default_type ()

Return the default content type.  Most messages have a default content
type of @code{text/plain}, except for messages that are subparts of
@code{multipart/digest} containers.  Such subparts have a default
content type of @code{message/rfc822}.
@end deffn

@geindex set_default_type() (email.message.Message method)
@anchor{library/email compat32-message email message Message set_default_type}@anchor{2d1a}
@deffn {Method} set_default_type (ctype)

Set the default content type.  @emph{ctype} should either be
@code{text/plain} or @code{message/rfc822}, although this is not
enforced.  The default content type is not stored in the
@code{Content-Type} header.
@end deffn

@geindex get_params() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_params}@anchor{2d1b}
@deffn {Method} get_params (failobj=None, header='content-type', unquote=True)

Return the message’s @code{Content-Type} parameters, as a list.
The elements of the returned list are 2-tuples of key/value pairs, as
split on the @code{'='} sign.  The left hand side of the @code{'='} is the key,
while the right hand side is the value.  If there is no @code{'='} sign in
the parameter the value is the empty string, otherwise the value is as
described in @ref{2d1c,,get_param()} and is unquoted if optional @emph{unquote} is
@code{True} (the default).

Optional @emph{failobj} is the object to return if there is no
@code{Content-Type} header.  Optional @emph{header} is the header to
search instead of @code{Content-Type}.

This is a legacy method.  On the
@code{EmailMessage} class its functionality is
replaced by the @emph{params} property of the individual header objects
returned by the header access methods.
@end deffn

@geindex get_param() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_param}@anchor{2d1c}
@deffn {Method} get_param (param, failobj=None, header='content-type', unquote=True)

Return the value of the @code{Content-Type} header’s parameter
@emph{param} as a string.  If the message has no @code{Content-Type}
header or if there is no such parameter, then @emph{failobj} is returned
(defaults to @code{None}).

Optional @emph{header} if given, specifies the message header to use instead of
@code{Content-Type}.

Parameter keys are always compared case insensitively.  The return value
can either be a string, or a 3-tuple if the parameter was 
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html}
encoded.  When it’s a 3-tuple, the elements of the value are of the form
@code{(CHARSET, LANGUAGE, VALUE)}.  Note that both @code{CHARSET} and
@code{LANGUAGE} can be @code{None}, in which case you should consider @code{VALUE}
to be encoded in the @code{us-ascii} charset.  You can usually ignore
@code{LANGUAGE}.

If your application doesn’t care whether the parameter was encoded as in
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html}, you can collapse the parameter value by calling
@ref{2d1d,,email.utils.collapse_rfc2231_value()}, passing in the return value
from @ref{2d1c,,get_param()}.  This will return a suitably decoded Unicode
string when the value is a tuple, or the original string unquoted if it
isn’t.  For example:

@example
rawparam = msg.get_param('foo')
param = email.utils.collapse_rfc2231_value(rawparam)
@end example

In any case, the parameter value (either the returned string, or the
@code{VALUE} item in the 3-tuple) is always unquoted, unless @emph{unquote} is set
to @code{False}.

This is a legacy method.  On the
@code{EmailMessage} class its functionality is
replaced by the @emph{params} property of the individual header objects
returned by the header access methods.
@end deffn

@geindex set_param() (email.message.Message method)
@anchor{library/email compat32-message email message Message set_param}@anchor{ab6}
@deffn {Method} set_param (param, value, header='Content-Type', requote=True, charset=None, language='@w{'}, replace=False)

Set a parameter in the @code{Content-Type} header.  If the
parameter already exists in the header, its value will be replaced with
@emph{value}.  If the @code{Content-Type} header as not yet been defined
for this message, it will be set to @code{text/plain} and the new
parameter value will be appended as per 
@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html}.

Optional @emph{header} specifies an alternative header to
@code{Content-Type}, and all parameters will be quoted as necessary
unless optional @emph{requote} is @code{False} (the default is @code{True}).

If optional @emph{charset} is specified, the parameter will be encoded
according to 
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html}. Optional @emph{language} specifies the RFC 2231
language, defaulting to the empty string.  Both @emph{charset} and @emph{language}
should be strings.

If @emph{replace} is @code{False} (the default) the header is moved to the
end of the list of headers.  If @emph{replace} is @code{True}, the header
will be updated in place.


Changed in version 3.4: @code{replace} keyword was added.

@end deffn

@geindex del_param() (email.message.Message method)
@anchor{library/email compat32-message email message Message del_param}@anchor{2d1e}
@deffn {Method} del_param (param, header='content-type', requote=True)

Remove the given parameter completely from the @code{Content-Type}
header.  The header will be re-written in place without the parameter or
its value.  All values will be quoted as necessary unless @emph{requote} is
@code{False} (the default is @code{True}).  Optional @emph{header} specifies an
alternative to @code{Content-Type}.
@end deffn

@geindex set_type() (email.message.Message method)
@anchor{library/email compat32-message email message Message set_type}@anchor{2d1f}
@deffn {Method} set_type (type, header='Content-Type', requote=True)

Set the main type and subtype for the @code{Content-Type}
header. @emph{type} must be a string in the form @code{maintype/subtype},
otherwise a @ref{17c,,ValueError} is raised.

This method replaces the @code{Content-Type} header, keeping all
the parameters in place.  If @emph{requote} is @code{False}, this leaves the
existing header’s quoting as is, otherwise the parameters will be quoted
(the default).

An alternative header can be specified in the @emph{header} argument. When the
@code{Content-Type} header is set a @code{MIME-Version}
header is also added.

This is a legacy method.  On the
@code{EmailMessage} class its functionality is
replaced by the @code{make_} and @code{add_} methods.
@end deffn

@geindex get_filename() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_filename}@anchor{2d20}
@deffn {Method} get_filename (failobj=None)

Return the value of the @code{filename} parameter of the
@code{Content-Disposition} header of the message.  If the header
does not have a @code{filename} parameter, this method falls back to looking
for the @code{name} parameter on the @code{Content-Type} header.  If
neither is found, or the header is missing, then @emph{failobj} is returned.
The returned string will always be unquoted as per
@ref{2c72,,email.utils.unquote()}.
@end deffn

@geindex get_boundary() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_boundary}@anchor{2d21}
@deffn {Method} get_boundary (failobj=None)

Return the value of the @code{boundary} parameter of the
@code{Content-Type} header of the message, or @emph{failobj} if either
the header is missing, or has no @code{boundary} parameter.  The returned
string will always be unquoted as per @ref{2c72,,email.utils.unquote()}.
@end deffn

@geindex set_boundary() (email.message.Message method)
@anchor{library/email compat32-message email message Message set_boundary}@anchor{2d22}
@deffn {Method} set_boundary (boundary)

Set the @code{boundary} parameter of the @code{Content-Type} header to
@emph{boundary}.  @ref{2d22,,set_boundary()} will always quote @emph{boundary} if
necessary.  A @ref{2c75,,HeaderParseError} is raised if the
message object has no @code{Content-Type} header.

Note that using this method is subtly different than deleting the old
@code{Content-Type} header and adding a new one with the new
boundary via @ref{2d14,,add_header()}, because @ref{2d22,,set_boundary()} preserves
the order of the @code{Content-Type} header in the list of
headers. However, it does @emph{not} preserve any continuation lines which may
have been present in the original @code{Content-Type} header.
@end deffn

@geindex get_content_charset() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_content_charset}@anchor{2d23}
@deffn {Method} get_content_charset (failobj=None)

Return the @code{charset} parameter of the @code{Content-Type} header,
coerced to lower case.  If there is no @code{Content-Type} header, or if
that header has no @code{charset} parameter, @emph{failobj} is returned.

Note that this method differs from @ref{2d09,,get_charset()} which returns the
@ref{959,,Charset} instance for the default encoding of the message body.
@end deffn

@geindex get_charsets() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_charsets}@anchor{2d24}
@deffn {Method} get_charsets (failobj=None)

Return a list containing the character set names in the message.  If the
message is a @code{multipart}, then the list will contain one element
for each subpart in the payload, otherwise, it will be a list of length 1.

Each item in the list will be a string which is the value of the
@code{charset} parameter in the @code{Content-Type} header for the
represented subpart.  However, if the subpart has no
@code{Content-Type} header, no @code{charset} parameter, or is not of
the @code{text} main MIME type, then that item in the returned list
will be @emph{failobj}.
@end deffn

@geindex get_content_disposition() (email.message.Message method)
@anchor{library/email compat32-message email message Message get_content_disposition}@anchor{956}
@deffn {Method} get_content_disposition ()

Return the lowercased value (without parameters) of the message’s
@code{Content-Disposition} header if it has one, or @code{None}.  The
possible values for this method are @emph{inline}, @emph{attachment} or @code{None}
if the message follows 
@geindex RFC; RFC 2183
RFC 2183@footnote{https://tools.ietf.org/html/rfc2183.html}.


New in version 3.5.

@end deffn

@geindex walk() (email.message.Message method)
@anchor{library/email compat32-message email message Message walk}@anchor{2d25}
@deffn {Method} walk ()

The @ref{2d25,,walk()} method is an all-purpose generator which can be used to
iterate over all the parts and subparts of a message object tree, in
depth-first traversal order.  You will typically use @ref{2d25,,walk()} as the
iterator in a @code{for} loop; each iteration returns the next subpart.

Here’s an example that prints the MIME type of every part of a multipart
message structure:

@c import email
@c from email import message_from_binary_file
@c from os.path import join, dirname
@c lib_dir = dirname(dirname(email.__file__))
@c file_path = join(lib_dir, 'test/test_email/data/msg_16.txt')
@c with open(file_path, 'rb') as f:
@c     msg = message_from_binary_file(f)
@c from email.iterators import _structure

@example
>>> for part in msg.walk():
...     print(part.get_content_type())
multipart/report
text/plain
message/delivery-status
text/plain
text/plain
message/rfc822
text/plain
@end example

@code{walk} iterates over the subparts of any part where
@ref{2c79,,is_multipart()} returns @code{True}, even though
@code{msg.get_content_maintype() == 'multipart'} may return @code{False}.  We
can see this in our example by making use of the @code{_structure} debug
helper function:

@example
>>> for part in msg.walk():
...     print(part.get_content_maintype() == 'multipart',
...           part.is_multipart())
True True
False False
False True
False False
False False
False True
False False
>>> _structure(msg)
multipart/report
    text/plain
    message/delivery-status
        text/plain
        text/plain
    message/rfc822
        text/plain
@end example

Here the @code{message} parts are not @code{multiparts}, but they do contain
subparts. @code{is_multipart()} returns @code{True} and @code{walk} descends
into the subparts.
@end deffn

@ref{7e3,,Message} objects can also optionally contain two instance attributes,
which can be used when generating the plain text of a MIME message.

@geindex preamble (email.message.Message attribute)
@anchor{library/email compat32-message email message Message preamble}@anchor{2d26}
@deffn {Attribute} preamble

The format of a MIME document allows for some text between the blank line
following the headers, and the first multipart boundary string. Normally,
this text is never visible in a MIME-aware mail reader because it falls
outside the standard MIME armor.  However, when viewing the raw text of
the message, or when viewing the message in a non-MIME aware reader, this
text can become visible.

The @emph{preamble} attribute contains this leading extra-armor text for MIME
documents.  When the @ref{d43,,Parser} discovers some text
after the headers but before the first boundary string, it assigns this
text to the message’s @emph{preamble} attribute.  When the
@ref{d72,,Generator} is writing out the plain text
representation of a MIME message, and it finds the
message has a @emph{preamble} attribute, it will write this text in the area
between the headers and the first boundary.  See @ref{6f,,email.parser} and
@ref{69,,email.generator} for details.

Note that if the message object has no preamble, the @emph{preamble} attribute
will be @code{None}.
@end deffn

@geindex epilogue (email.message.Message attribute)
@anchor{library/email compat32-message email message Message epilogue}@anchor{2d27}
@deffn {Attribute} epilogue

The @emph{epilogue} attribute acts the same way as the @emph{preamble} attribute,
except that it contains text that appears between the last boundary and
the end of the message.

You do not need to set the epilogue to the empty string in order for the
@ref{d72,,Generator} to print a newline at the end of the
file.
@end deffn

@geindex defects (email.message.Message attribute)
@anchor{library/email compat32-message email message Message defects}@anchor{2d28}
@deffn {Attribute} defects

The @emph{defects} attribute contains a list of all the problems found when
parsing this message.  See @ref{68,,email.errors} for a detailed description
of the possible parsing defects.
@end deffn
@end deffn

@node email mime Creating email and MIME objects from scratch,email header Internationalized headers,email message Message Representing an email message using the compat32 API,email — An email and MIME handling package
@anchor{library/email mime doc}@anchor{2d29}@anchor{library/email mime email-mime-creating-email-and-mime-objects-from-scratch}@anchor{2d2a}@anchor{library/email mime module-email mime}@anchor{6e}
@subsubsection @code{email.mime}: Creating email and MIME objects from scratch


@geindex module; email.mime

@strong{Source code:} Lib/email/mime/@footnote{https://github.com/python/cpython/tree/main/Lib/email/mime/}


__________________________________________________________________


This module is part of the legacy (@code{Compat32}) email API.  Its functionality
is partially replaced by the @ref{66,,contentmanager} in the new API, but
in certain applications these classes may still be useful, even in non-legacy
code.

Ordinarily, you get a message object structure by passing a file or some text to
a parser, which parses the text and returns the root message object.  However
you can also build a complete message structure from scratch, or even individual
@ref{7e3,,Message} objects by hand.  In fact, you can also take an
existing structure and add new @ref{7e3,,Message} objects, move them
around, etc.  This makes a very convenient interface for slicing-and-dicing MIME
messages.

You can create a new object structure by creating @ref{7e3,,Message}
instances, adding attachments and all the appropriate headers manually.  For MIME
messages though, the @ref{64,,email} package provides some convenient subclasses to
make things easier.

Here are the classes:

@geindex MIMEBase (class in email.mime.base)
@anchor{library/email mime email mime base MIMEBase}@anchor{2d2b}
@deffn {Class} email.mime.base.MIMEBase (_maintype, _subtype, *, policy=compat32, **_params)

Module: @code{email.mime.base}

This is the base class for all the MIME-specific subclasses of
@ref{7e3,,Message}.  Ordinarily you won’t create instances
specifically of @ref{2d2b,,MIMEBase}, although you could.  @ref{2d2b,,MIMEBase}
is provided primarily as a convenient base class for more specific
MIME-aware subclasses.

@emph{_maintype} is the @code{Content-Type} major type (e.g. @code{text}
or @code{image}), and @emph{_subtype} is the @code{Content-Type} minor
type  (e.g. @code{plain} or @code{gif}).  @emph{_params} is a parameter
key/value dictionary and is passed directly to @ref{2d14,,Message.add_header}.

If @emph{policy} is specified, (defaults to the
@ref{c45,,compat32} policy) it will be passed to
@ref{7e3,,Message}.

The @ref{2d2b,,MIMEBase} class always adds a @code{Content-Type} header
(based on @emph{_maintype}, @emph{_subtype}, and @emph{_params}), and a
@code{MIME-Version} header (always set to @code{1.0}).


Changed in version 3.6: Added @emph{policy} keyword-only parameter.

@end deffn

@geindex MIMENonMultipart (class in email.mime.nonmultipart)
@anchor{library/email mime email mime nonmultipart MIMENonMultipart}@anchor{2ccc}
@deffn {Class} email.mime.nonmultipart.MIMENonMultipart

Module: @code{email.mime.nonmultipart}

A subclass of @ref{2d2b,,MIMEBase}, this is an intermediate base
class for MIME messages that are not @code{multipart}.  The primary
purpose of this class is to prevent the use of the
@ref{2c86,,attach()} method, which only makes sense for
@code{multipart} messages.  If @ref{2c86,,attach()}
is called, a @ref{2ccb,,MultipartConversionError} exception is raised.
@end deffn

@geindex MIMEMultipart (class in email.mime.multipart)
@anchor{library/email mime email mime multipart MIMEMultipart}@anchor{2d2c}
@deffn {Class} email.mime.multipart.MIMEMultipart (_subtype='mixed', boundary=None, _subparts=None, *, policy=compat32, **_params)

Module: @code{email.mime.multipart}

A subclass of @ref{2d2b,,MIMEBase}, this is an intermediate base
class for MIME messages that are @code{multipart}.  Optional @emph{_subtype}
defaults to @code{mixed}, but can be used to specify the subtype of the
message.  A @code{Content-Type} header of @code{multipart/_subtype}
will be added to the message object.  A @code{MIME-Version} header will
also be added.

Optional @emph{boundary} is the multipart boundary string.  When @code{None} (the
default), the boundary is calculated when needed (for example, when the
message is serialized).

@emph{_subparts} is a sequence of initial subparts for the payload.  It must be
possible to convert this sequence to a list.  You can always attach new subparts
to the message by using the @ref{2c86,,Message.attach} method.

Optional @emph{policy} argument defaults to @ref{c45,,compat32}.

Additional parameters for the @code{Content-Type} header are taken from
the keyword arguments, or passed into the @emph{_params} argument, which is a keyword
dictionary.


Changed in version 3.6: Added @emph{policy} keyword-only parameter.

@end deffn

@geindex MIMEApplication (class in email.mime.application)
@anchor{library/email mime email mime application MIMEApplication}@anchor{2d2d}
@deffn {Class} email.mime.application.MIMEApplication (_data, _subtype='octet-stream', _encoder=email.encoders.encode_base64, *, policy=compat32, **_params)

Module: @code{email.mime.application}

A subclass of @ref{2ccc,,MIMENonMultipart}, the
@ref{2d2d,,MIMEApplication} class is used to represent MIME message objects of
major type @code{application}.  @emph{_data} is a string containing the raw
byte data.  Optional @emph{_subtype} specifies the MIME subtype and defaults to
@code{octet-stream}.

Optional @emph{_encoder} is a callable (i.e. function) which will perform the actual
encoding of the data for transport.  This callable takes one argument, which is
the @ref{2d2d,,MIMEApplication} instance. It should use
@ref{d71,,get_payload()} and
@ref{2d07,,set_payload()} to change the payload to encoded
form.  It should also add
any @code{Content-Transfer-Encoding} or other headers to the message
object as necessary.  The default encoding is base64.  See the
@ref{67,,email.encoders} module for a list of the built-in encoders.

Optional @emph{policy} argument defaults to @ref{c45,,compat32}.

@emph{_params} are passed straight through to the base class constructor.


Changed in version 3.6: Added @emph{policy} keyword-only parameter.

@end deffn

@geindex MIMEAudio (class in email.mime.audio)
@anchor{library/email mime email mime audio MIMEAudio}@anchor{2d2e}
@deffn {Class} email.mime.audio.MIMEAudio (_audiodata, _subtype=None, _encoder=email.encoders.encode_base64, *, policy=compat32, **_params)

Module: @code{email.mime.audio}

A subclass of @ref{2ccc,,MIMENonMultipart}, the
@ref{2d2e,,MIMEAudio} class is used to create MIME message objects of major type
@code{audio}. @emph{_audiodata} is a string containing the raw audio data.  If
this data can be decoded by the standard Python module @ref{e9,,sndhdr}, then the
subtype will be automatically included in the @code{Content-Type} header.
Otherwise you can explicitly specify the audio subtype via the @emph{_subtype}
argument.  If the minor type could not be guessed and @emph{_subtype} was not given,
then @ref{162,,TypeError} is raised.

Optional @emph{_encoder} is a callable (i.e. function) which will perform the actual
encoding of the audio data for transport.  This callable takes one argument,
which is the @ref{2d2e,,MIMEAudio} instance. It should use
@ref{d71,,get_payload()} and
@ref{2d07,,set_payload()} to change the payload to encoded
form.  It should also add
any @code{Content-Transfer-Encoding} or other headers to the message
object as necessary.  The default encoding is base64.  See the
@ref{67,,email.encoders} module for a list of the built-in encoders.

Optional @emph{policy} argument defaults to @ref{c45,,compat32}.

@emph{_params} are passed straight through to the base class constructor.


Changed in version 3.6: Added @emph{policy} keyword-only parameter.

@end deffn

@geindex MIMEImage (class in email.mime.image)
@anchor{library/email mime email mime image MIMEImage}@anchor{2ccd}
@deffn {Class} email.mime.image.MIMEImage (_imagedata, _subtype=None, _encoder=email.encoders.encode_base64, *, policy=compat32, **_params)

Module: @code{email.mime.image}

A subclass of @ref{2ccc,,MIMENonMultipart}, the
@ref{2ccd,,MIMEImage} class is used to create MIME message objects of major type
@code{image}. @emph{_imagedata} is a string containing the raw image data.  If
this data can be decoded by the standard Python module @ref{94,,imghdr}, then the
subtype will be automatically included in the @code{Content-Type} header.
Otherwise you can explicitly specify the image subtype via the @emph{_subtype}
argument.  If the minor type could not be guessed and @emph{_subtype} was not given,
then @ref{162,,TypeError} is raised.

Optional @emph{_encoder} is a callable (i.e. function) which will perform the actual
encoding of the image data for transport.  This callable takes one argument,
which is the @ref{2ccd,,MIMEImage} instance. It should use
@ref{d71,,get_payload()} and
@ref{2d07,,set_payload()} to change the payload to encoded
form.  It should also add
any @code{Content-Transfer-Encoding} or other headers to the message
object as necessary.  The default encoding is base64.  See the
@ref{67,,email.encoders} module for a list of the built-in encoders.

Optional @emph{policy} argument defaults to @ref{c45,,compat32}.

@emph{_params} are passed straight through to the @ref{2d2b,,MIMEBase}
constructor.


Changed in version 3.6: Added @emph{policy} keyword-only parameter.

@end deffn

@geindex MIMEMessage (class in email.mime.message)
@anchor{library/email mime email mime message MIMEMessage}@anchor{2d2f}
@deffn {Class} email.mime.message.MIMEMessage (_msg, _subtype='rfc822', *, policy=compat32)

Module: @code{email.mime.message}

A subclass of @ref{2ccc,,MIMENonMultipart}, the
@ref{2d2f,,MIMEMessage} class is used to create MIME objects of main type
@code{message}. @emph{_msg} is used as the payload, and must be an instance
of class @ref{7e3,,Message} (or a subclass thereof), otherwise
a @ref{162,,TypeError} is raised.

Optional @emph{_subtype} sets the subtype of the message; it defaults to
@code{rfc822}.

Optional @emph{policy} argument defaults to @ref{c45,,compat32}.


Changed in version 3.6: Added @emph{policy} keyword-only parameter.

@end deffn

@geindex MIMEText (class in email.mime.text)
@anchor{library/email mime email mime text MIMEText}@anchor{958}
@deffn {Class} email.mime.text.MIMEText (_text, _subtype='plain', _charset=None, *, policy=compat32)

Module: @code{email.mime.text}

A subclass of @ref{2ccc,,MIMENonMultipart}, the
@ref{958,,MIMEText} class is used to create MIME objects of major type
@code{text}. @emph{_text} is the string for the payload.  @emph{_subtype} is the
minor type and defaults to @code{plain}.  @emph{_charset} is the character
set of the text and is passed as an argument to the
@ref{2ccc,,MIMENonMultipart} constructor; it defaults
to @code{us-ascii} if the string contains only @code{ascii} code points, and
@code{utf-8} otherwise.  The @emph{_charset} parameter accepts either a string or a
@ref{959,,Charset} instance.

Unless the @emph{_charset} argument is explicitly set to @code{None}, the
MIMEText object created will have both a @code{Content-Type} header
with a @code{charset} parameter, and a @code{Content-Transfer-Encoding}
header.  This means that a subsequent @code{set_payload} call will not result
in an encoded payload, even if a charset is passed in the @code{set_payload}
command.  You can “reset” this behavior by deleting the
@code{Content-Transfer-Encoding} header, after which a @code{set_payload} call
will automatically encode the new payload (and add a new
@code{Content-Transfer-Encoding} header).

Optional @emph{policy} argument defaults to @ref{c45,,compat32}.


Changed in version 3.5: @emph{_charset} also accepts @ref{959,,Charset} instances.



Changed in version 3.6: Added @emph{policy} keyword-only parameter.

@end deffn

@node email header Internationalized headers,email charset Representing character sets,email mime Creating email and MIME objects from scratch,email — An email and MIME handling package
@anchor{library/email header doc}@anchor{2d30}@anchor{library/email header email-header-internationalized-headers}@anchor{2d31}@anchor{library/email header module-email header}@anchor{6a}
@subsubsection @code{email.header}: Internationalized headers


@geindex module; email.header

@strong{Source code:} Lib/email/header.py@footnote{https://github.com/python/cpython/tree/main/Lib/email/header.py}


__________________________________________________________________


This module is part of the legacy (@code{Compat32}) email API.  In the current API
encoding and decoding of headers is handled transparently by the
dictionary-like API of the @ref{7e4,,EmailMessage} class.  In
addition to uses in legacy code, this module can be useful in applications that
need to completely control the character sets used when encoding headers.

The remaining text in this section is the original documentation of the module.

@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html} is the base standard that describes the format of email messages.
It derives from the older 
@geindex RFC; RFC 822
RFC 822@footnote{https://tools.ietf.org/html/rfc822.html} standard which came into widespread use at
a time when most email was composed of ASCII characters only.  
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html} is a
specification written assuming email contains only 7-bit ASCII characters.

Of course, as email has been deployed worldwide, it has become
internationalized, such that language specific character sets can now be used in
email messages.  The base standard still requires email messages to be
transferred using only 7-bit ASCII characters, so a slew of RFCs have been
written describing how to encode email containing non-ASCII characters into
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html}-compliant format. These RFCs include 
@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html}, 
@geindex RFC; RFC 2046
RFC 2046@footnote{https://tools.ietf.org/html/rfc2046.html},
@geindex RFC; RFC 2047
RFC 2047@footnote{https://tools.ietf.org/html/rfc2047.html}, and 
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html}. The @ref{64,,email} package supports these standards
in its @ref{6a,,email.header} and @ref{65,,email.charset} modules.

If you want to include non-ASCII characters in your email headers, say in the
@code{Subject} or @code{To} fields, you should use the
@ref{1118,,Header} class and assign the field in the @ref{7e3,,Message}
object to an instance of @ref{1118,,Header} instead of using a string for the header
value.  Import the @ref{1118,,Header} class from the @ref{6a,,email.header} module.
For example:

@example
>>> from email.message import Message
>>> from email.header import Header
>>> msg = Message()
>>> h = Header('p\xf6stal', 'iso-8859-1')
>>> msg['Subject'] = h
>>> msg.as_string()
'Subject: =?iso-8859-1?q?p=F6stal?=\n\n'
@end example

Notice here how we wanted the @code{Subject} field to contain a non-ASCII
character?  We did this by creating a @ref{1118,,Header} instance and passing in
the character set that the byte string was encoded in.  When the subsequent
@ref{7e3,,Message} instance was flattened, the @code{Subject}
field was properly 
@geindex RFC; RFC 2047
RFC 2047@footnote{https://tools.ietf.org/html/rfc2047.html} encoded.  MIME-aware mail readers would show this
header using the embedded ISO-8859-1 character.

Here is the @ref{1118,,Header} class description:

@geindex Header (class in email.header)
@anchor{library/email header email header Header}@anchor{1118}
@deffn {Class} email.header.Header (s=None, charset=None, maxlinelen=None, header_name=None, continuation_ws=' ', errors='strict')

Create a MIME-compliant header that can contain strings in different character
sets.

Optional @emph{s} is the initial header value.  If @code{None} (the default), the
initial header value is not set.  You can later append to the header with
@ref{2d32,,append()} method calls.  @emph{s} may be an instance of @ref{2f0,,bytes} or
@ref{2ef,,str}, but see the @ref{2d32,,append()} documentation for semantics.

Optional @emph{charset} serves two purposes: it has the same meaning as the @emph{charset}
argument to the @ref{2d32,,append()} method.  It also sets the default character set
for all subsequent @ref{2d32,,append()} calls that omit the @emph{charset} argument.  If
@emph{charset} is not provided in the constructor (the default), the @code{us-ascii}
character set is used both as @emph{s}’s initial charset and as the default for
subsequent @ref{2d32,,append()} calls.

The maximum line length can be specified explicitly via @emph{maxlinelen}.  For
splitting the first line to a shorter value (to account for the field header
which isn’t included in @emph{s}, e.g. @code{Subject}) pass in the name of the
field in @emph{header_name}.  The default @emph{maxlinelen} is 76, and the default value
for @emph{header_name} is @code{None}, meaning it is not taken into account for the
first line of a long, split header.

Optional @emph{continuation_ws} must be 
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html}-compliant folding
whitespace, and is usually either a space or a hard tab character.  This
character will be prepended to continuation lines.  @emph{continuation_ws}
defaults to a single space character.

Optional @emph{errors} is passed straight through to the @ref{2d32,,append()} method.

@geindex append() (email.header.Header method)
@anchor{library/email header email header Header append}@anchor{2d32}
@deffn {Method} append (s, charset=None, errors='strict')

Append the string @emph{s} to the MIME header.

Optional @emph{charset}, if given, should be a @ref{959,,Charset}
instance (see @ref{65,,email.charset}) or the name of a character set, which
will be converted to a @ref{959,,Charset} instance.  A value
of @code{None} (the default) means that the @emph{charset} given in the constructor
is used.

@emph{s} may be an instance of @ref{2f0,,bytes} or @ref{2ef,,str}.  If it is an
instance of @ref{2f0,,bytes}, then @emph{charset} is the encoding of that byte
string, and a @ref{e49,,UnicodeError} will be raised if the string cannot be
decoded with that character set.

If @emph{s} is an instance of @ref{2ef,,str}, then @emph{charset} is a hint specifying
the character set of the characters in the string.

In either case, when producing an 
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html}-compliant header using
@geindex RFC; RFC 2047
RFC 2047@footnote{https://tools.ietf.org/html/rfc2047.html} rules, the string will be encoded using the output codec of
the charset.  If the string cannot be encoded using the output codec, a
UnicodeError will be raised.

Optional @emph{errors} is passed as the errors argument to the decode call
if @emph{s} is a byte string.
@end deffn

@geindex encode() (email.header.Header method)
@anchor{library/email header email header Header encode}@anchor{2d33}
@deffn {Method} encode (splitchars=';, \t', maxlinelen=None, linesep='\n')

Encode a message header into an RFC-compliant format, possibly wrapping
long lines and encapsulating non-ASCII parts in base64 or quoted-printable
encodings.

Optional @emph{splitchars} is a string containing characters which should be
given extra weight by the splitting algorithm during normal header
wrapping.  This is in very rough support of 
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html}'s ‘higher level
syntactic breaks’:  split points preceded by a splitchar are preferred
during line splitting, with the characters preferred in the order in
which they appear in the string.  Space and tab may be included in the
string to indicate whether preference should be given to one over the
other as a split point when other split chars do not appear in the line
being split.  Splitchars does not affect 
@geindex RFC; RFC 2047
RFC 2047@footnote{https://tools.ietf.org/html/rfc2047.html} encoded lines.

@emph{maxlinelen}, if given, overrides the instance’s value for the maximum
line length.

@emph{linesep} specifies the characters used to separate the lines of the
folded header.  It defaults to the most useful value for Python
application code (@code{\n}), but @code{\r\n} can be specified in order
to produce headers with RFC-compliant line separators.


Changed in version 3.2: Added the @emph{linesep} argument.

@end deffn

The @ref{1118,,Header} class also provides a number of methods to support
standard operators and built-in functions.

@geindex __str__() (email.header.Header method)
@anchor{library/email header email header Header __str__}@anchor{2d34}
@deffn {Method} __str__ ()

Returns an approximation of the @ref{1118,,Header} as a string, using an
unlimited line length.  All pieces are converted to unicode using the
specified encoding and joined together appropriately.  Any pieces with a
charset of @code{'unknown-8bit'} are decoded as ASCII using the @code{'replace'}
error handler.


Changed in version 3.2: Added handling for the @code{'unknown-8bit'} charset.

@end deffn

@geindex __eq__() (email.header.Header method)
@anchor{library/email header email header Header __eq__}@anchor{2d35}
@deffn {Method} __eq__ (other)

This method allows you to compare two @ref{1118,,Header} instances for
equality.
@end deffn

@geindex __ne__() (email.header.Header method)
@anchor{library/email header email header Header __ne__}@anchor{2d36}
@deffn {Method} __ne__ (other)

This method allows you to compare two @ref{1118,,Header} instances for
inequality.
@end deffn
@end deffn

The @ref{6a,,email.header} module also provides the following convenient functions.

@geindex decode_header() (in module email.header)
@anchor{library/email header email header decode_header}@anchor{c49}
@deffn {Function} email.header.decode_header (header)

Decode a message header value without converting the character set. The header
value is in @emph{header}.

This function returns a list of @code{(decoded_string, charset)} pairs containing
each of the decoded parts of the header.  @emph{charset} is @code{None} for non-encoded
parts of the header, otherwise a lower case string containing the name of the
character set specified in the encoded string.

Here’s an example:

@example
>>> from email.header import decode_header
>>> decode_header('=?iso-8859-1?q?p=F6stal?=')
[(b'p\xf6stal', 'iso-8859-1')]
@end example
@end deffn

@geindex make_header() (in module email.header)
@anchor{library/email header email header make_header}@anchor{c4a}
@deffn {Function} email.header.make_header (decoded_seq, maxlinelen=None, header_name=None, continuation_ws=' ')

Create a @ref{1118,,Header} instance from a sequence of pairs as returned by
@ref{c49,,decode_header()}.

@ref{c49,,decode_header()} takes a header value string and returns a sequence of
pairs of the format @code{(decoded_string, charset)} where @emph{charset} is the name of
the character set.

This function takes one of those sequence of pairs and returns a
@ref{1118,,Header} instance.  Optional @emph{maxlinelen}, @emph{header_name}, and
@emph{continuation_ws} are as in the @ref{1118,,Header} constructor.
@end deffn

@node email charset Representing character sets,email encoders Encoders,email header Internationalized headers,email — An email and MIME handling package
@anchor{library/email charset doc}@anchor{2d37}@anchor{library/email charset email-charset-representing-character-sets}@anchor{2d38}@anchor{library/email charset module-email charset}@anchor{65}
@subsubsection @code{email.charset}: Representing character sets


@geindex module; email.charset

@strong{Source code:} Lib/email/charset.py@footnote{https://github.com/python/cpython/tree/main/Lib/email/charset.py}


__________________________________________________________________


This module is part of the legacy (@code{Compat32}) email API.  In the new
API only the aliases table is used.

The remaining text in this section is the original documentation of the module.

This module provides a class @ref{959,,Charset} for representing character sets
and character set conversions in email messages, as well as a character set
registry and several convenience methods for manipulating this registry.
Instances of @ref{959,,Charset} are used in several other modules within the
@ref{64,,email} package.

Import this class from the @ref{65,,email.charset} module.

@geindex Charset (class in email.charset)
@anchor{library/email charset email charset Charset}@anchor{959}
@deffn {Class} email.charset.Charset (input_charset=DEFAULT_CHARSET)

Map character sets to their email properties.

This class provides information about the requirements imposed on email for a
specific character set.  It also provides convenience routines for converting
between character sets, given the availability of the applicable codecs.  Given
a character set, it will do its best to provide information on how to use that
character set in an email message in an RFC-compliant way.

Certain character sets must be encoded with quoted-printable or base64 when used
in email headers or bodies.  Certain character sets must be converted outright,
and are not allowed in email.

Optional @emph{input_charset} is as described below; it is always coerced to lower
case.  After being alias normalized it is also used as a lookup into the
registry of character sets to find out the header encoding, body encoding, and
output conversion codec to be used for the character set.  For example, if
@emph{input_charset} is @code{iso-8859-1}, then headers and bodies will be encoded using
quoted-printable and no output conversion codec is necessary.  If
@emph{input_charset} is @code{euc-jp}, then headers will be encoded with base64, bodies
will not be encoded, but output text will be converted from the @code{euc-jp}
character set to the @code{iso-2022-jp} character set.

@ref{959,,Charset} instances have the following data attributes:

@geindex input_charset (email.charset.Charset attribute)
@anchor{library/email charset email charset Charset input_charset}@anchor{2d39}
@deffn {Attribute} input_charset

The initial character set specified.  Common aliases are converted to
their @emph{official} email names (e.g. @code{latin_1} is converted to
@code{iso-8859-1}).  Defaults to 7-bit @code{us-ascii}.
@end deffn

@geindex header_encoding (email.charset.Charset attribute)
@anchor{library/email charset email charset Charset header_encoding}@anchor{2d3a}
@deffn {Attribute} header_encoding

If the character set must be encoded before it can be used in an email
header, this attribute will be set to @code{Charset.QP} (for
quoted-printable), @code{Charset.BASE64} (for base64 encoding), or
@code{Charset.SHORTEST} for the shortest of QP or BASE64 encoding. Otherwise,
it will be @code{None}.
@end deffn

@geindex body_encoding (email.charset.Charset attribute)
@anchor{library/email charset email charset Charset body_encoding}@anchor{2d3b}
@deffn {Attribute} body_encoding

Same as @emph{header_encoding}, but describes the encoding for the mail
message’s body, which indeed may be different than the header encoding.
@code{Charset.SHORTEST} is not allowed for @emph{body_encoding}.
@end deffn

@geindex output_charset (email.charset.Charset attribute)
@anchor{library/email charset email charset Charset output_charset}@anchor{2d3c}
@deffn {Attribute} output_charset

Some character sets must be converted before they can be used in email
headers or bodies.  If the @emph{input_charset} is one of them, this attribute
will contain the name of the character set output will be converted to.
Otherwise, it will be @code{None}.
@end deffn

@geindex input_codec (email.charset.Charset attribute)
@anchor{library/email charset email charset Charset input_codec}@anchor{2d3d}
@deffn {Attribute} input_codec

The name of the Python codec used to convert the @emph{input_charset} to
Unicode.  If no conversion codec is necessary, this attribute will be
@code{None}.
@end deffn

@geindex output_codec (email.charset.Charset attribute)
@anchor{library/email charset email charset Charset output_codec}@anchor{2d3e}
@deffn {Attribute} output_codec

The name of the Python codec used to convert Unicode to the
@emph{output_charset}.  If no conversion codec is necessary, this attribute
will have the same value as the @emph{input_codec}.
@end deffn

@ref{959,,Charset} instances also have the following methods:

@geindex get_body_encoding() (email.charset.Charset method)
@anchor{library/email charset email charset Charset get_body_encoding}@anchor{2d3f}
@deffn {Method} get_body_encoding ()

Return the content transfer encoding used for body encoding.

This is either the string @code{quoted-printable} or @code{base64} depending on
the encoding used, or it is a function, in which case you should call the
function with a single argument, the Message object being encoded.  The
function should then set the @code{Content-Transfer-Encoding}
header itself to whatever is appropriate.

Returns the string @code{quoted-printable} if @emph{body_encoding} is @code{QP},
returns the string @code{base64} if @emph{body_encoding} is @code{BASE64}, and
returns the string @code{7bit} otherwise.
@end deffn

@geindex get_output_charset() (email.charset.Charset method)
@anchor{library/email charset email charset Charset get_output_charset}@anchor{2d40}
@deffn {Method} get_output_charset ()

Return the output character set.

This is the @emph{output_charset} attribute if that is not @code{None}, otherwise
it is @emph{input_charset}.
@end deffn

@geindex header_encode() (email.charset.Charset method)
@anchor{library/email charset email charset Charset header_encode}@anchor{2d41}
@deffn {Method} header_encode (string)

Header-encode the string @emph{string}.

The type of encoding (base64 or quoted-printable) will be based on the
@emph{header_encoding} attribute.
@end deffn

@geindex header_encode_lines() (email.charset.Charset method)
@anchor{library/email charset email charset Charset header_encode_lines}@anchor{2d42}
@deffn {Method} header_encode_lines (string, maxlengths)

Header-encode a @emph{string} by converting it first to bytes.

This is similar to @ref{2d41,,header_encode()} except that the string is fit
into maximum line lengths as given by the argument @emph{maxlengths}, which
must be an iterator: each element returned from this iterator will provide
the next maximum line length.
@end deffn

@geindex body_encode() (email.charset.Charset method)
@anchor{library/email charset email charset Charset body_encode}@anchor{2d43}
@deffn {Method} body_encode (string)

Body-encode the string @emph{string}.

The type of encoding (base64 or quoted-printable) will be based on the
@emph{body_encoding} attribute.
@end deffn

The @ref{959,,Charset} class also provides a number of methods to support
standard operations and built-in functions.

@geindex __str__() (email.charset.Charset method)
@anchor{library/email charset email charset Charset __str__}@anchor{2d44}
@deffn {Method} __str__ ()

Returns @emph{input_charset} as a string coerced to lower
case. @code{__repr__()} is an alias for @ref{2d44,,__str__()}.
@end deffn

@geindex __eq__() (email.charset.Charset method)
@anchor{library/email charset email charset Charset __eq__}@anchor{2d45}
@deffn {Method} __eq__ (other)

This method allows you to compare two @ref{959,,Charset} instances for
equality.
@end deffn

@geindex __ne__() (email.charset.Charset method)
@anchor{library/email charset email charset Charset __ne__}@anchor{2d46}
@deffn {Method} __ne__ (other)

This method allows you to compare two @ref{959,,Charset} instances for
inequality.
@end deffn
@end deffn

The @ref{65,,email.charset} module also provides the following functions for adding
new entries to the global character set, alias, and codec registries:

@geindex add_charset() (in module email.charset)
@anchor{library/email charset email charset add_charset}@anchor{2d47}
@deffn {Function} email.charset.add_charset (charset, header_enc=None, body_enc=None, output_charset=None)

Add character properties to the global registry.

@emph{charset} is the input character set, and must be the canonical name of a
character set.

Optional @emph{header_enc} and @emph{body_enc} is either @code{Charset.QP} for
quoted-printable, @code{Charset.BASE64} for base64 encoding,
@code{Charset.SHORTEST} for the shortest of quoted-printable or base64 encoding,
or @code{None} for no encoding.  @code{SHORTEST} is only valid for
@emph{header_enc}. The default is @code{None} for no encoding.

Optional @emph{output_charset} is the character set that the output should be in.
Conversions will proceed from input charset, to Unicode, to the output charset
when the method @code{Charset.convert()} is called.  The default is to output in
the same character set as the input.

Both @emph{input_charset} and @emph{output_charset} must have Unicode codec entries in the
module’s character set-to-codec mapping; use @ref{2d48,,add_codec()} to add codecs the
module does not know about.  See the @ref{1a,,codecs} module’s documentation for
more information.

The global character set registry is kept in the module global dictionary
@code{CHARSETS}.
@end deffn

@geindex add_alias() (in module email.charset)
@anchor{library/email charset email charset add_alias}@anchor{2d49}
@deffn {Function} email.charset.add_alias (alias, canonical)

Add a character set alias.  @emph{alias} is the alias name, e.g. @code{latin-1}.
@emph{canonical} is the character set’s canonical name, e.g. @code{iso-8859-1}.

The global charset alias registry is kept in the module global dictionary
@code{ALIASES}.
@end deffn

@geindex add_codec() (in module email.charset)
@anchor{library/email charset email charset add_codec}@anchor{2d48}
@deffn {Function} email.charset.add_codec (charset, codecname)

Add a codec that map characters in the given character set to and from Unicode.

@emph{charset} is the canonical name of a character set. @emph{codecname} is the name of a
Python codec, as appropriate for the second argument to the @ref{2ef,,str}’s
@ref{3ba,,encode()} method.
@end deffn

@node email encoders Encoders,email utils Miscellaneous utilities,email charset Representing character sets,email — An email and MIME handling package
@anchor{library/email encoders doc}@anchor{2d4a}@anchor{library/email encoders email-encoders-encoders}@anchor{2d4b}@anchor{library/email encoders module-email encoders}@anchor{67}
@subsubsection @code{email.encoders}: Encoders


@geindex module; email.encoders

@strong{Source code:} Lib/email/encoders.py@footnote{https://github.com/python/cpython/tree/main/Lib/email/encoders.py}


__________________________________________________________________


This module is part of the legacy (@code{Compat32}) email API.  In the
new API the functionality is provided by the @emph{cte} parameter of
the @ref{1116,,set_content()} method.

This module is deprecated in Python 3.  The functions provided here
should not be called explicitly since the @ref{958,,MIMEText}
class sets the content type and CTE header using the @emph{_subtype} and @emph{_charset}
values passed during the instantiation of that class.

The remaining text in this section is the original documentation of the module.

When creating @ref{7e3,,Message} objects from scratch, you often
need to encode the payloads for transport through compliant mail servers. This
is especially true for @code{image/*} and @code{text/*} type messages
containing binary data.

The @ref{64,,email} package provides some convenient encoders in its
@code{encoders} module.  These encoders are actually used by the
@ref{2d2e,,MIMEAudio} and @ref{2ccd,,MIMEImage}
class constructors to provide default encodings.  All encoder functions take
exactly one argument, the message object to encode.  They usually extract the
payload, encode it, and reset the payload to this newly encoded value.  They
should also set the @code{Content-Transfer-Encoding} header as appropriate.

Note that these functions are not meaningful for a multipart message.  They
must be applied to individual subparts instead, and will raise a
@ref{162,,TypeError} if passed a message whose type is multipart.

Here are the encoding functions provided:

@geindex encode_quopri() (in module email.encoders)
@anchor{library/email encoders email encoders encode_quopri}@anchor{2d4c}
@deffn {Function} email.encoders.encode_quopri (msg)

Encodes the payload into quoted-printable form and sets the
@code{Content-Transfer-Encoding} header to @code{quoted-printable} @footnote{@w{(1)} 
Note that encoding with @ref{2d4c,,encode_quopri()} also encodes all tabs and space
characters in the data.
}.
This is a good encoding to use when most of your payload is normal printable
data, but contains a few unprintable characters.
@end deffn

@geindex encode_base64() (in module email.encoders)
@anchor{library/email encoders email encoders encode_base64}@anchor{2d4d}
@deffn {Function} email.encoders.encode_base64 (msg)

Encodes the payload into base64 form and sets the
@code{Content-Transfer-Encoding} header to @code{base64}.  This is a good
encoding to use when most of your payload is unprintable data since it is a more
compact form than quoted-printable.  The drawback of base64 encoding is that it
renders the text non-human readable.
@end deffn

@geindex encode_7or8bit() (in module email.encoders)
@anchor{library/email encoders email encoders encode_7or8bit}@anchor{2d4e}
@deffn {Function} email.encoders.encode_7or8bit (msg)

This doesn’t actually modify the message’s payload, but it does set the
@code{Content-Transfer-Encoding} header to either @code{7bit} or @code{8bit} as
appropriate, based on the payload data.
@end deffn

@geindex encode_noop() (in module email.encoders)
@anchor{library/email encoders email encoders encode_noop}@anchor{2d4f}
@deffn {Function} email.encoders.encode_noop (msg)

This does nothing; it doesn’t even set the
@code{Content-Transfer-Encoding} header.
@end deffn

@node email utils Miscellaneous utilities,email iterators Iterators,email encoders Encoders,email — An email and MIME handling package
@anchor{library/email utils doc}@anchor{2d50}@anchor{library/email utils email-utils-miscellaneous-utilities}@anchor{2d51}@anchor{library/email utils module-email utils}@anchor{71}
@subsubsection @code{email.utils}: Miscellaneous utilities


@geindex module; email.utils

@strong{Source code:} Lib/email/utils.py@footnote{https://github.com/python/cpython/tree/main/Lib/email/utils.py}


__________________________________________________________________


There are a couple of useful utilities provided in the @ref{71,,email.utils}
module:

@geindex localtime() (in module email.utils)
@anchor{library/email utils email utils localtime}@anchor{c50}
@deffn {Function} email.utils.localtime (dt=None)

Return local time as an aware datetime object.  If called without
arguments, return current time.  Otherwise @emph{dt} argument should be a
@ref{363,,datetime} instance, and it is converted to the local time
zone according to the system time zone database.  If @emph{dt} is naive (that
is, @code{dt.tzinfo} is @code{None}), it is assumed to be in local time.  In this
case, a positive or zero value for @emph{isdst} causes @code{localtime} to presume
initially that summer time (for example, Daylight Saving Time) is or is not
(respectively) in effect for the specified time.  A negative value for
@emph{isdst} causes the @code{localtime} to attempt to divine whether summer time
is in effect for the specified time.


New in version 3.3.

@end deffn

@geindex make_msgid() (in module email.utils)
@anchor{library/email utils email utils make_msgid}@anchor{2d52}
@deffn {Function} email.utils.make_msgid (idstring=None, domain=None)

Returns a string suitable for an 
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html}-compliant
@code{Message-ID} header.  Optional @emph{idstring} if given, is a string
used to strengthen the uniqueness of the message id.  Optional @emph{domain} if
given provides the portion of the msgid after the ‘@@’.  The default is the
local hostname.  It is not normally necessary to override this default, but
may be useful certain cases, such as a constructing distributed system that
uses a consistent domain name across multiple hosts.


Changed in version 3.2: Added the @emph{domain} keyword.

@end deffn

The remaining functions are part of the legacy (@code{Compat32}) email API.  There
is no need to directly use these with the new API, since the parsing and
formatting they provide is done automatically by the header parsing machinery
of the new API.

@geindex quote() (in module email.utils)
@anchor{library/email utils email utils quote}@anchor{2d53}
@deffn {Function} email.utils.quote (str)

Return a new string with backslashes in @emph{str} replaced by two backslashes, and
double quotes replaced by backslash-double quote.
@end deffn

@geindex unquote() (in module email.utils)
@anchor{library/email utils email utils unquote}@anchor{2c72}
@deffn {Function} email.utils.unquote (str)

Return a new string which is an @emph{unquoted} version of @emph{str}. If @emph{str} ends and
begins with double quotes, they are stripped off.  Likewise if @emph{str} ends and
begins with angle brackets, they are stripped off.
@end deffn

@geindex parseaddr() (in module email.utils)
@anchor{library/email utils email utils parseaddr}@anchor{2d54}
@deffn {Function} email.utils.parseaddr (address)

Parse address – which should be the value of some address-containing field such
as @code{To} or @code{Cc} – into its constituent @emph{realname} and
@emph{email address} parts.  Returns a tuple of that information, unless the parse
fails, in which case a 2-tuple of @code{('@w{'}, '@w{'})} is returned.
@end deffn

@geindex formataddr() (in module email.utils)
@anchor{library/email utils email utils formataddr}@anchor{d40}
@deffn {Function} email.utils.formataddr (pair, charset='utf-8')

The inverse of @ref{2d54,,parseaddr()}, this takes a 2-tuple of the form @code{(realname,
email_address)} and returns the string value suitable for a @code{To} or
@code{Cc} header.  If the first element of @emph{pair} is false, then the
second element is returned unmodified.

Optional @emph{charset} is the character set that will be used in the 
@geindex RFC; RFC 2047
RFC 2047@footnote{https://tools.ietf.org/html/rfc2047.html}
encoding of the @code{realname} if the @code{realname} contains non-ASCII
characters.  Can be an instance of @ref{2ef,,str} or a
@ref{959,,Charset}.  Defaults to @code{utf-8}.


Changed in version 3.3: Added the @emph{charset} option.

@end deffn

@geindex getaddresses() (in module email.utils)
@anchor{library/email utils email utils getaddresses}@anchor{1117}
@deffn {Function} email.utils.getaddresses (fieldvalues)

This method returns a list of 2-tuples of the form returned by @code{parseaddr()}.
@emph{fieldvalues} is a sequence of header field values as might be returned by
@ref{2d0e,,Message.get_all}.  Here’s a simple
example that gets all the recipients of a message:

@example
from email.utils import getaddresses

tos = msg.get_all('to', [])
ccs = msg.get_all('cc', [])
resent_tos = msg.get_all('resent-to', [])
resent_ccs = msg.get_all('resent-cc', [])
all_recipients = getaddresses(tos + ccs + resent_tos + resent_ccs)
@end example
@end deffn

@geindex parsedate() (in module email.utils)
@anchor{library/email utils email utils parsedate}@anchor{2d55}
@deffn {Function} email.utils.parsedate (date)

Attempts to parse a date according to the rules in 
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html}. however, some
mailers don’t follow that format as specified, so @ref{2d55,,parsedate()} tries to
guess correctly in such cases.  @emph{date} is a string containing an 
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html}
date, such as  @code{"Mon, 20 Nov 1995 19:12:08 -0500"}.  If it succeeds in parsing
the date, @ref{2d55,,parsedate()} returns a 9-tuple that can be passed directly to
@ref{d8a,,time.mktime()}; otherwise @code{None} will be returned.  Note that indexes 6,
7, and 8 of the result tuple are not usable.
@end deffn

@geindex parsedate_tz() (in module email.utils)
@anchor{library/email utils email utils parsedate_tz}@anchor{10e0}
@deffn {Function} email.utils.parsedate_tz (date)

Performs the same function as @ref{2d55,,parsedate()}, but returns either @code{None} or
a 10-tuple; the first 9 elements make up a tuple that can be passed directly to
@ref{d8a,,time.mktime()}, and the tenth is the offset of the date’s timezone from UTC
(which is the official term for Greenwich Mean Time) @footnote{@w{(1)} 
Note that the sign of the timezone offset is the opposite of the sign of the
@code{time.timezone} variable for the same timezone; the latter variable follows
the POSIX standard while this module follows 
@geindex RFC; RFC 2822
@uref{https://tools.ietf.org/html/rfc2822.html,RFC 2822}.
}.  If the input string
has no timezone, the last element of the tuple returned is @code{0}, which represents
UTC. Note that indexes 6, 7, and 8 of the result tuple are not usable.
@end deffn

@geindex parsedate_to_datetime() (in module email.utils)
@anchor{library/email utils email utils parsedate_to_datetime}@anchor{c4f}
@deffn {Function} email.utils.parsedate_to_datetime (date)

The inverse of @ref{c4e,,format_datetime()}.  Performs the same function as
@ref{2d55,,parsedate()}, but on success returns a @ref{363,,datetime};
otherwise @code{ValueError} is raised if @emph{date} contains an invalid value such
as an hour greater than 23 or a timezone offset not between -24 and 24 hours.
If the input date has a timezone of @code{-0000}, the @code{datetime} will be a naive
@code{datetime}, and if the date is conforming to the RFCs it will represent a
time in UTC but with no indication of the actual source timezone of the
message the date comes from.  If the input date has any other valid timezone
offset, the @code{datetime} will be an aware @code{datetime} with the
corresponding a @ref{c51,,timezone} @ref{3c6,,tzinfo}.


New in version 3.3.

@end deffn

@geindex mktime_tz() (in module email.utils)
@anchor{library/email utils email utils mktime_tz}@anchor{2d56}
@deffn {Function} email.utils.mktime_tz (tuple)

Turn a 10-tuple as returned by @ref{10e0,,parsedate_tz()} into a UTC
timestamp (seconds since the Epoch).  If the timezone item in the
tuple is @code{None}, assume local time.
@end deffn

@geindex formatdate() (in module email.utils)
@anchor{library/email utils email utils formatdate}@anchor{2d57}
@deffn {Function} email.utils.formatdate (timeval=None, localtime=False, usegmt=False)

Returns a date string as per 
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html}, e.g.:

@example
Fri, 09 Nov 2001 01:08:47 -0000
@end example

Optional @emph{timeval} if given is a floating point time value as accepted by
@ref{d65,,time.gmtime()} and @ref{1224,,time.localtime()}, otherwise the current time is
used.

Optional @emph{localtime} is a flag that when @code{True}, interprets @emph{timeval}, and
returns a date relative to the local timezone instead of UTC, properly taking
daylight savings time into account. The default is @code{False} meaning UTC is
used.

Optional @emph{usegmt} is a flag that when @code{True}, outputs a  date string with the
timezone as an ascii string @code{GMT}, rather than a numeric @code{-0000}. This is
needed for some protocols (such as HTTP). This only applies when @emph{localtime} is
@code{False}.  The default is @code{False}.
@end deffn

@geindex format_datetime() (in module email.utils)
@anchor{library/email utils email utils format_datetime}@anchor{c4e}
@deffn {Function} email.utils.format_datetime (dt, usegmt=False)

Like @code{formatdate}, but the input is a @ref{2f,,datetime} instance.  If it is
a naive datetime, it is assumed to be “UTC with no information about the
source timezone”, and the conventional @code{-0000} is used for the timezone.
If it is an aware @code{datetime}, then the numeric timezone offset is used.
If it is an aware timezone with offset zero, then @emph{usegmt} may be set to
@code{True}, in which case the string @code{GMT} is used instead of the numeric
timezone offset.  This provides a way to generate standards conformant HTTP
date headers.


New in version 3.3.

@end deffn

@geindex decode_rfc2231() (in module email.utils)
@anchor{library/email utils email utils decode_rfc2231}@anchor{2d58}
@deffn {Function} email.utils.decode_rfc2231 (s)

Decode the string @emph{s} according to 
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html}.
@end deffn

@geindex encode_rfc2231() (in module email.utils)
@anchor{library/email utils email utils encode_rfc2231}@anchor{2d59}
@deffn {Function} email.utils.encode_rfc2231 (s, charset=None, language=None)

Encode the string @emph{s} according to 
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html}.  Optional @emph{charset} and
@emph{language}, if given is the character set name and language name to use.  If
neither is given, @emph{s} is returned as-is.  If @emph{charset} is given but @emph{language}
is not, the string is encoded using the empty string for @emph{language}.
@end deffn

@geindex collapse_rfc2231_value() (in module email.utils)
@anchor{library/email utils email utils collapse_rfc2231_value}@anchor{2d1d}
@deffn {Function} email.utils.collapse_rfc2231_value (value, errors='replace', fallback_charset='us-ascii')

When a header parameter is encoded in 
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html} format,
@ref{2d1c,,Message.get_param} may return a
3-tuple containing the character set,
language, and value.  @ref{2d1d,,collapse_rfc2231_value()} turns this into a unicode
string.  Optional @emph{errors} is passed to the @emph{errors} argument of @ref{2ef,,str}’s
@ref{3ba,,encode()} method; it defaults to @code{'replace'}.  Optional
@emph{fallback_charset} specifies the character set to use if the one in the
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html} header is not known by Python; it defaults to @code{'us-ascii'}.

For convenience, if the @emph{value} passed to @ref{2d1d,,collapse_rfc2231_value()} is not
a tuple, it should be a string and it is returned unquoted.
@end deffn

@geindex decode_params() (in module email.utils)
@anchor{library/email utils email utils decode_params}@anchor{2d5a}
@deffn {Function} email.utils.decode_params (params)

Decode parameters list according to 
@geindex RFC; RFC 2231
RFC 2231@footnote{https://tools.ietf.org/html/rfc2231.html}.  @emph{params} is a sequence of
2-tuples containing elements of the form @code{(content-type, string-value)}.
@end deffn

@node email iterators Iterators,,email utils Miscellaneous utilities,email — An email and MIME handling package
@anchor{library/email iterators doc}@anchor{2d5b}@anchor{library/email iterators email-iterators-iterators}@anchor{2d5c}@anchor{library/email iterators module-email iterators}@anchor{6c}
@subsubsection @code{email.iterators}: Iterators


@geindex module; email.iterators

@strong{Source code:} Lib/email/iterators.py@footnote{https://github.com/python/cpython/tree/main/Lib/email/iterators.py}


__________________________________________________________________


Iterating over a message object tree is fairly easy with the
@ref{2d25,,Message.walk} method.  The
@ref{6c,,email.iterators} module provides some useful higher level iterations over
message object trees.

@geindex body_line_iterator() (in module email.iterators)
@anchor{library/email iterators email iterators body_line_iterator}@anchor{2d5d}
@deffn {Function} email.iterators.body_line_iterator (msg, decode=False)

This iterates over all the payloads in all the subparts of @emph{msg}, returning the
string payloads line-by-line.  It skips over all the subpart headers, and it
skips over any subpart with a payload that isn’t a Python string.  This is
somewhat equivalent to reading the flat text representation of the message from
a file using @ref{209b,,readline()}, skipping over all the
intervening headers.

Optional @emph{decode} is passed through to @ref{d71,,Message.get_payload}.
@end deffn

@geindex typed_subpart_iterator() (in module email.iterators)
@anchor{library/email iterators email iterators typed_subpart_iterator}@anchor{2d5e}
@deffn {Function} email.iterators.typed_subpart_iterator (msg, maintype='text', subtype=None)

This iterates over all the subparts of @emph{msg}, returning only those subparts that
match the MIME type specified by @emph{maintype} and @emph{subtype}.

Note that @emph{subtype} is optional; if omitted, then subpart MIME type matching is
done only with the main type.  @emph{maintype} is optional too; it defaults to
@code{text}.

Thus, by default @ref{2d5e,,typed_subpart_iterator()} returns each subpart that has a
MIME type of @code{text/*}.
@end deffn

The following function has been added as a useful debugging tool.  It should
@emph{not} be considered part of the supported public interface for the package.

@geindex _structure() (in module email.iterators)
@anchor{library/email iterators email iterators _structure}@anchor{2d5f}
@deffn {Function} email.iterators._structure (msg, fp=None, level=0, include_default=False)

Prints an indented representation of the content types of the message object
structure.  For example:

@c import email
@c from email.iterators import _structure
@c somefile = open('../Lib/test/test_email/data/msg_02.txt')

@example
>>> msg = email.message_from_file(somefile)
>>> _structure(msg)
multipart/mixed
    text/plain
    text/plain
    multipart/digest
        message/rfc822
            text/plain
        message/rfc822
            text/plain
        message/rfc822
            text/plain
        message/rfc822
            text/plain
        message/rfc822
            text/plain
    text/plain
@end example

@c somefile.close()

Optional @emph{fp} is a file-like object to print the output to.  It must be
suitable for Python’s @ref{af6,,print()} function.  @emph{level} is used internally.
@emph{include_default}, if true, prints the default type as well.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{e8,,smtplib}

SMTP (Simple Mail Transport Protocol) client

@item Module @ref{cb,,poplib}

POP (Post Office Protocol) client

@item Module @ref{93,,imaplib}

IMAP (Internet Message Access Protocol) client

@item Module @ref{bc,,nntplib}

NNTP (Net News Transport Protocol) client

@item Module @ref{aa,,mailbox}

Tools for creating, reading, and managing collections of messages on disk
using a variety standard formats.

@item Module @ref{e7,,smtpd}

SMTP server framework (primarily useful for testing)
@end table


@node json — JSON encoder and decoder,mailcap — Mailcap file handling,email — An email and MIME handling package,Internet Data Handling
@anchor{library/json doc}@anchor{2d60}@anchor{library/json json-json-encoder-and-decoder}@anchor{2d61}@anchor{library/json module-json}@anchor{a0}
@subsection @code{json} — JSON encoder and decoder


@geindex module; json

@strong{Source code:} Lib/json/__init__.py@footnote{https://github.com/python/cpython/tree/main/Lib/json/__init__.py}


__________________________________________________________________


JSON (JavaScript Object Notation)@footnote{https://json.org}, specified by
@geindex RFC; RFC 7159
RFC 7159@footnote{https://tools.ietf.org/html/rfc7159.html} (which obsoletes 
@geindex RFC; RFC 4627
RFC 4627@footnote{https://tools.ietf.org/html/rfc4627.html}) and by
ECMA-404@footnote{https://www.ecma-international.org/publications-and-standards/standards/ecma-404/},
is a lightweight data interchange format inspired by
JavaScript@footnote{https://en.wikipedia.org/wiki/JavaScript} object literal syntax
(although it is not a strict subset of JavaScript @footnote{@w{(1)} 
As noted in @uref{https://www.rfc-editor.org/errata_search.php?rfc=7159,the errata for RFC 7159},
JSON permits literal U+2028 (LINE SEPARATOR) and
U+2029 (PARAGRAPH SEPARATOR) characters in strings, whereas JavaScript
(as of ECMAScript Edition 5.1) does not.
} ).

@ref{a0,,json} exposes an API familiar to users of the standard library
@ref{ac,,marshal} and @ref{c5,,pickle} modules.

Encoding basic Python object hierarchies:

@example
>>> import json
>>> json.dumps(['foo', @{'bar': ('baz', None, 1.0, 2)@}])
'["foo", @{"bar": ["baz", null, 1.0, 2]@}]'
>>> print(json.dumps("\"foo\bar"))
"\"foo\bar"
>>> print(json.dumps('\u1234'))
"\u1234"
>>> print(json.dumps('\\'))
"\\"
>>> print(json.dumps(@{"c": 0, "b": 0, "a": 0@}, sort_keys=True))
@{"a": 0, "b": 0, "c": 0@}
>>> from io import StringIO
>>> io = StringIO()
>>> json.dump(['streaming API'], io)
>>> io.getvalue()
'["streaming API"]'
@end example

Compact encoding:

@example
>>> import json
>>> json.dumps([1, 2, 3, @{'4': 5, '6': 7@}], separators=(',', ':'))
'[1,2,3,@{"4":5,"6":7@}]'
@end example

Pretty printing:

@example
>>> import json
>>> print(json.dumps(@{'4': 5, '6': 7@}, sort_keys=True, indent=4))
@{
    "4": 5,
    "6": 7
@}
@end example

Decoding JSON:

@example
>>> import json
>>> json.loads('["foo", @{"bar":["baz", null, 1.0, 2]@}]')
['foo', @{'bar': ['baz', None, 1.0, 2]@}]
>>> json.loads('"\\"foo\\bar"')
'"foo\x08ar'
>>> from io import StringIO
>>> io = StringIO('["streaming API"]')
>>> json.load(io)
['streaming API']
@end example

Specializing JSON object decoding:

@example
>>> import json
>>> def as_complex(dct):
...     if '__complex__' in dct:
...         return complex(dct['real'], dct['imag'])
...     return dct
...
>>> json.loads('@{"__complex__": true, "real": 1, "imag": 2@}',
...     object_hook=as_complex)
(1+2j)
>>> import decimal
>>> json.loads('1.1', parse_float=decimal.Decimal)
Decimal('1.1')
@end example

Extending @ref{89c,,JSONEncoder}:

@example
>>> import json
>>> class ComplexEncoder(json.JSONEncoder):
...     def default(self, obj):
...         if isinstance(obj, complex):
...             return [obj.real, obj.imag]
...         # Let the base class default method raise the TypeError
...         return json.JSONEncoder.default(self, obj)
...
>>> json.dumps(2 + 1j, cls=ComplexEncoder)
'[2.0, 1.0]'
>>> ComplexEncoder().encode(2 + 1j)
'[2.0, 1.0]'
>>> list(ComplexEncoder().iterencode(2 + 1j))
['[2.0', ', 1.0', ']']
@end example

Using @ref{a1,,json.tool} from the shell to validate and pretty-print:

@example
$ echo '@{"json":"obj"@}' | python -m json.tool
@{
    "json": "obj"
@}
$ echo '@{1.2:3.4@}' | python -m json.tool
Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
@end example

See @ref{2d62,,Command Line Interface} for detailed documentation.

@cartouche
@quotation Note 
JSON is a subset of YAML@footnote{http://yaml.org/} 1.2.  The JSON produced by
this module’s default settings (in particular, the default @emph{separators}
value) is also a subset of YAML 1.0 and 1.1.  This module can thus also be
used as a YAML serializer.
@end quotation
@end cartouche

@cartouche
@quotation Note 
This module’s encoders and decoders preserve input and output order by
default.  Order is only lost if the underlying containers are unordered.

Prior to Python 3.7, @ref{3b0,,dict} was not guaranteed to be ordered, so
inputs and outputs were typically scrambled unless
@ref{4e5,,collections.OrderedDict} was specifically requested.  Starting
with Python 3.7, the regular @ref{3b0,,dict} became order preserving, so
it is no longer necessary to specify @ref{4e5,,collections.OrderedDict} for
JSON generation and parsing.
@end quotation
@end cartouche

@menu
* Basic Usage:: 
* Encoders and Decoders:: 
* Exceptions: Exceptions<14>. 
* Standard Compliance and Interoperability:: 
* Command Line Interface: Command Line Interface<2>. 

@end menu

@node Basic Usage,Encoders and Decoders,,json — JSON encoder and decoder
@anchor{library/json basic-usage}@anchor{2d63}
@subsubsection Basic Usage


@geindex dump() (in module json)
@anchor{library/json json dump}@anchor{89a}
@deffn {Function} json.dump (obj, fp, *, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, cls=None, indent=None, separators=None, default=None, sort_keys=False, **kw)

Serialize @emph{obj} as a JSON formatted stream to @emph{fp} (a @code{.write()}-supporting
@ref{1e54,,file-like object}) using this @ref{2d64,,conversion table}.

If @emph{skipkeys} is true (default: @code{False}), then dict keys that are not
of a basic type (@ref{2ef,,str}, @ref{1b1,,int}, @ref{256,,float}, @ref{2f7,,bool},
@code{None}) will be skipped instead of raising a @ref{162,,TypeError}.

The @ref{a0,,json} module always produces @ref{2ef,,str} objects, not
@ref{2f0,,bytes} objects. Therefore, @code{fp.write()} must support @ref{2ef,,str}
input.

If @emph{ensure_ascii} is true (the default), the output is guaranteed to
have all incoming non-ASCII characters escaped.  If @emph{ensure_ascii} is
false, these characters will be output as-is.

If @emph{check_circular} is false (default: @code{True}), then the circular
reference check for container types will be skipped and a circular reference
will result in a @ref{8c4,,RecursionError} (or worse).

If @emph{allow_nan} is false (default: @code{True}), then it will be a
@ref{17c,,ValueError} to serialize out of range @ref{256,,float} values (@code{nan},
@code{inf}, @code{-inf}) in strict compliance of the JSON specification.
If @emph{allow_nan} is true, their JavaScript equivalents (@code{NaN},
@code{Infinity}, @code{-Infinity}) will be used.

If @emph{indent} is a non-negative integer or string, then JSON array elements and
object members will be pretty-printed with that indent level.  An indent level
of 0, negative, or @code{""} will only insert newlines.  @code{None} (the default)
selects the most compact representation. Using a positive integer indent
indents that many spaces per level.  If @emph{indent} is a string (such as @code{"\t"}),
that string is used to indent each level.


Changed in version 3.2: Allow strings for @emph{indent} in addition to integers.


If specified, @emph{separators} should be an @code{(item_separator, key_separator)}
tuple.  The default is @code{(', ', ': ')} if @emph{indent} is @code{None} and
@code{(',', ': ')} otherwise.  To get the most compact JSON representation,
you should specify @code{(',', ':')} to eliminate whitespace.


Changed in version 3.4: Use @code{(',', ': ')} as default if @emph{indent} is not @code{None}.


If specified, @emph{default} should be a function that gets called for objects that
can’t otherwise be serialized.  It should return a JSON encodable version of
the object or raise a @ref{162,,TypeError}.  If not specified, @ref{162,,TypeError}
is raised.

If @emph{sort_keys} is true (default: @code{False}), then the output of
dictionaries will be sorted by key.

To use a custom @ref{89c,,JSONEncoder} subclass (e.g. one that overrides the
@code{default()} method to serialize additional types), specify it with the
@emph{cls} kwarg; otherwise @ref{89c,,JSONEncoder} is used.


Changed in version 3.6: All optional parameters are now @ref{5a2,,keyword-only}.


@cartouche
@quotation Note 
Unlike @ref{c5,,pickle} and @ref{ac,,marshal}, JSON is not a framed protocol,
so trying to serialize multiple objects with repeated calls to
@ref{89a,,dump()} using the same @emph{fp} will result in an invalid JSON file.
@end quotation
@end cartouche
@end deffn

@geindex dumps() (in module json)
@anchor{library/json json dumps}@anchor{89b}
@deffn {Function} json.dumps (obj, *, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, cls=None, indent=None, separators=None, default=None, sort_keys=False, **kw)

Serialize @emph{obj} to a JSON formatted @ref{2ef,,str} using this @ref{2d64,,conversion table}.  The arguments have the same meaning as in
@ref{89a,,dump()}.

@cartouche
@quotation Note 
Keys in key/value pairs of JSON are always of the type @ref{2ef,,str}. When
a dictionary is converted into JSON, all the keys of the dictionary are
coerced to strings. As a result of this, if a dictionary is converted
into JSON and then back into a dictionary, the dictionary may not equal
the original one. That is, @code{loads(dumps(x)) != x} if x has non-string
keys.
@end quotation
@end cartouche
@end deffn

@geindex load() (in module json)
@anchor{library/json json load}@anchor{7fe}
@deffn {Function} json.load (fp, *, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw)

Deserialize @emph{fp} (a @code{.read()}-supporting @ref{1566,,text file} or
@ref{2138,,binary file} containing a JSON document) to a Python object using
this @ref{2d65,,conversion table}.

@emph{object_hook} is an optional function that will be called with the result of
any object literal decoded (a @ref{3b0,,dict}).  The return value of
@emph{object_hook} will be used instead of the @ref{3b0,,dict}.  This feature can be used
to implement custom decoders (e.g. JSON-RPC@footnote{http://www.jsonrpc.org}
class hinting).

@emph{object_pairs_hook} is an optional function that will be called with the
result of any object literal decoded with an ordered list of pairs.  The
return value of @emph{object_pairs_hook} will be used instead of the
@ref{3b0,,dict}.  This feature can be used to implement custom decoders.
If @emph{object_hook} is also defined, the @emph{object_pairs_hook} takes priority.


Changed in version 3.1: Added support for @emph{object_pairs_hook}.


@emph{parse_float}, if specified, will be called with the string of every JSON
float to be decoded.  By default, this is equivalent to @code{float(num_str)}.
This can be used to use another datatype or parser for JSON floats
(e.g. @ref{248,,decimal.Decimal}).

@emph{parse_int}, if specified, will be called with the string of every JSON int
to be decoded.  By default, this is equivalent to @code{int(num_str)}.  This can
be used to use another datatype or parser for JSON integers
(e.g. @ref{256,,float}).

@emph{parse_constant}, if specified, will be called with one of the following
strings: @code{'-Infinity'}, @code{'Infinity'}, @code{'NaN'}.
This can be used to raise an exception if invalid JSON numbers
are encountered.


Changed in version 3.1: @emph{parse_constant} doesn’t get called on ‘null’, ‘true’, ‘false’ anymore.


To use a custom @ref{89d,,JSONDecoder} subclass, specify it with the @code{cls}
kwarg; otherwise @ref{89d,,JSONDecoder} is used.  Additional keyword arguments
will be passed to the constructor of the class.

If the data being deserialized is not a valid JSON document, a
@ref{987,,JSONDecodeError} will be raised.


Changed in version 3.6: All optional parameters are now @ref{5a2,,keyword-only}.



Changed in version 3.6: @emph{fp} can now be a @ref{2138,,binary file}. The input encoding should be
UTF-8, UTF-16 or UTF-32.

@end deffn

@geindex loads() (in module json)
@anchor{library/json json loads}@anchor{456}
@deffn {Function} json.loads (s, *, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw)

Deserialize @emph{s} (a @ref{2ef,,str}, @ref{2f0,,bytes} or @ref{2f1,,bytearray}
instance containing a JSON document) to a Python object using this
@ref{2d65,,conversion table}.

The other arguments have the same meaning as in @ref{7fe,,load()}.

If the data being deserialized is not a valid JSON document, a
@ref{987,,JSONDecodeError} will be raised.


Changed in version 3.6: @emph{s} can now be of type @ref{2f0,,bytes} or @ref{2f1,,bytearray}. The
input encoding should be UTF-8, UTF-16 or UTF-32.



Changed in version 3.9: The keyword argument @emph{encoding} has been removed.

@end deffn

@node Encoders and Decoders,Exceptions<14>,Basic Usage,json — JSON encoder and decoder
@anchor{library/json encoders-and-decoders}@anchor{2d66}
@subsubsection Encoders and Decoders


@geindex JSONDecoder (class in json)
@anchor{library/json json JSONDecoder}@anchor{89d}
@deffn {Class} json.JSONDecoder (*, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, strict=True, object_pairs_hook=None)

Simple JSON decoder.

Performs the following translations in decoding by default:
@anchor{library/json json-to-py-table}@anchor{2d65}

@multitable {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxx} 
@headitem

JSON

@tab

Python

@item

object

@tab

dict

@item

array

@tab

list

@item

string

@tab

str

@item

number (int)

@tab

int

@item

number (real)

@tab

float

@item

true

@tab

True

@item

false

@tab

False

@item

null

@tab

None

@end multitable


It also understands @code{NaN}, @code{Infinity}, and @code{-Infinity} as their
corresponding @code{float} values, which is outside the JSON spec.

@emph{object_hook}, if specified, will be called with the result of every JSON
object decoded and its return value will be used in place of the given
@ref{3b0,,dict}.  This can be used to provide custom deserializations (e.g. to
support JSON-RPC@footnote{http://www.jsonrpc.org} class hinting).

@emph{object_pairs_hook}, if specified will be called with the result of every
JSON object decoded with an ordered list of pairs.  The return value of
@emph{object_pairs_hook} will be used instead of the @ref{3b0,,dict}.  This
feature can be used to implement custom decoders.  If @emph{object_hook} is also
defined, the @emph{object_pairs_hook} takes priority.


Changed in version 3.1: Added support for @emph{object_pairs_hook}.


@emph{parse_float}, if specified, will be called with the string of every JSON
float to be decoded.  By default, this is equivalent to @code{float(num_str)}.
This can be used to use another datatype or parser for JSON floats
(e.g. @ref{248,,decimal.Decimal}).

@emph{parse_int}, if specified, will be called with the string of every JSON int
to be decoded.  By default, this is equivalent to @code{int(num_str)}.  This can
be used to use another datatype or parser for JSON integers
(e.g. @ref{256,,float}).

@emph{parse_constant}, if specified, will be called with one of the following
strings: @code{'-Infinity'}, @code{'Infinity'}, @code{'NaN'}.
This can be used to raise an exception if invalid JSON numbers
are encountered.

If @emph{strict} is false (@code{True} is the default), then control characters
will be allowed inside strings.  Control characters in this context are
those with character codes in the 0–31 range, including @code{'\t'} (tab),
@code{'\n'}, @code{'\r'} and @code{'\0'}.

If the data being deserialized is not a valid JSON document, a
@ref{987,,JSONDecodeError} will be raised.


Changed in version 3.6: All parameters are now @ref{5a2,,keyword-only}.


@geindex decode() (json.JSONDecoder method)
@anchor{library/json json JSONDecoder decode}@anchor{2d67}
@deffn {Method} decode (s)

Return the Python representation of @emph{s} (a @ref{2ef,,str} instance
containing a JSON document).

@ref{987,,JSONDecodeError} will be raised if the given JSON document is not
valid.
@end deffn

@geindex raw_decode() (json.JSONDecoder method)
@anchor{library/json json JSONDecoder raw_decode}@anchor{2d68}
@deffn {Method} raw_decode (s)

Decode a JSON document from @emph{s} (a @ref{2ef,,str} beginning with a
JSON document) and return a 2-tuple of the Python representation
and the index in @emph{s} where the document ended.

This can be used to decode a JSON document from a string that may have
extraneous data at the end.
@end deffn
@end deffn

@geindex JSONEncoder (class in json)
@anchor{library/json json JSONEncoder}@anchor{89c}
@deffn {Class} json.JSONEncoder (*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Extensible JSON encoder for Python data structures.

Supports the following objects and types by default:
@anchor{library/json py-to-json-table}@anchor{2d64}

@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxx} 
@headitem

Python

@tab

JSON

@item

dict

@tab

object

@item

list, tuple

@tab

array

@item

str

@tab

string

@item

int, float, int- & float-derived Enums

@tab

number

@item

True

@tab

true

@item

False

@tab

false

@item

None

@tab

null

@end multitable



Changed in version 3.4: Added support for int- and float-derived Enum classes.


To extend this to recognize other objects, subclass and implement a
@ref{2d69,,default()} method with another method that returns a serializable object
for @code{o} if possible, otherwise it should call the superclass implementation
(to raise @ref{162,,TypeError}).

If @emph{skipkeys} is false (the default), a @ref{162,,TypeError} will be raised when
trying to encode keys that are not @ref{2ef,,str}, @ref{1b1,,int}, @ref{256,,float}
or @code{None}.  If @emph{skipkeys} is true, such items are simply skipped.

If @emph{ensure_ascii} is true (the default), the output is guaranteed to
have all incoming non-ASCII characters escaped.  If @emph{ensure_ascii} is
false, these characters will be output as-is.

If @emph{check_circular} is true (the default), then lists, dicts, and custom
encoded objects will be checked for circular references during encoding to
prevent an infinite recursion (which would cause a @ref{8c4,,RecursionError}).
Otherwise, no such check takes place.

If @emph{allow_nan} is true (the default), then @code{NaN}, @code{Infinity}, and
@code{-Infinity} will be encoded as such.  This behavior is not JSON
specification compliant, but is consistent with most JavaScript based
encoders and decoders.  Otherwise, it will be a @ref{17c,,ValueError} to encode
such floats.

If @emph{sort_keys} is true (default: @code{False}), then the output of dictionaries
will be sorted by key; this is useful for regression tests to ensure that
JSON serializations can be compared on a day-to-day basis.

If @emph{indent} is a non-negative integer or string, then JSON array elements and
object members will be pretty-printed with that indent level.  An indent level
of 0, negative, or @code{""} will only insert newlines.  @code{None} (the default)
selects the most compact representation. Using a positive integer indent
indents that many spaces per level.  If @emph{indent} is a string (such as @code{"\t"}),
that string is used to indent each level.


Changed in version 3.2: Allow strings for @emph{indent} in addition to integers.


If specified, @emph{separators} should be an @code{(item_separator, key_separator)}
tuple.  The default is @code{(', ', ': ')} if @emph{indent} is @code{None} and
@code{(',', ': ')} otherwise.  To get the most compact JSON representation,
you should specify @code{(',', ':')} to eliminate whitespace.


Changed in version 3.4: Use @code{(',', ': ')} as default if @emph{indent} is not @code{None}.


If specified, @emph{default} should be a function that gets called for objects that
can’t otherwise be serialized.  It should return a JSON encodable version of
the object or raise a @ref{162,,TypeError}.  If not specified, @ref{162,,TypeError}
is raised.


Changed in version 3.6: All parameters are now @ref{5a2,,keyword-only}.


@geindex default() (json.JSONEncoder method)
@anchor{library/json json JSONEncoder default}@anchor{2d69}
@deffn {Method} default (o)

Implement this method in a subclass such that it returns a serializable
object for @emph{o}, or calls the base implementation (to raise a
@ref{162,,TypeError}).

For example, to support arbitrary iterators, you could implement
@ref{2d69,,default()} like this:

@example
def default(self, o):
   try:
       iterable = iter(o)
   except TypeError:
       pass
   else:
       return list(iterable)
   # Let the base class default method raise the TypeError
   return json.JSONEncoder.default(self, o)
@end example
@end deffn

@geindex encode() (json.JSONEncoder method)
@anchor{library/json json JSONEncoder encode}@anchor{2d6a}
@deffn {Method} encode (o)

Return a JSON string representation of a Python data structure, @emph{o}.  For
example:

@example
>>> json.JSONEncoder().encode(@{"foo": ["bar", "baz"]@})
'@{"foo": ["bar", "baz"]@}'
@end example
@end deffn

@geindex iterencode() (json.JSONEncoder method)
@anchor{library/json json JSONEncoder iterencode}@anchor{2d6b}
@deffn {Method} iterencode (o)

Encode the given object, @emph{o}, and yield each string representation as
available.  For example:

@example
for chunk in json.JSONEncoder().iterencode(bigobject):
    mysocket.write(chunk)
@end example
@end deffn
@end deffn

@node Exceptions<14>,Standard Compliance and Interoperability,Encoders and Decoders,json — JSON encoder and decoder
@anchor{library/json exceptions}@anchor{2d6c}
@subsubsection Exceptions


@geindex JSONDecodeError
@anchor{library/json json JSONDecodeError}@anchor{987}
@deffn {Exception} json.JSONDecodeError (msg, doc, pos)

Subclass of @ref{17c,,ValueError} with the following additional attributes:

@geindex msg (json.JSONDecodeError attribute)
@anchor{library/json json JSONDecodeError msg}@anchor{2d6d}
@deffn {Attribute} msg

The unformatted error message.
@end deffn

@geindex doc (json.JSONDecodeError attribute)
@anchor{library/json json JSONDecodeError doc}@anchor{2d6e}
@deffn {Attribute} doc

The JSON document being parsed.
@end deffn

@geindex pos (json.JSONDecodeError attribute)
@anchor{library/json json JSONDecodeError pos}@anchor{2d6f}
@deffn {Attribute} pos

The start index of @emph{doc} where parsing failed.
@end deffn

@geindex lineno (json.JSONDecodeError attribute)
@anchor{library/json json JSONDecodeError lineno}@anchor{2d70}
@deffn {Attribute} lineno

The line corresponding to @emph{pos}.
@end deffn

@geindex colno (json.JSONDecodeError attribute)
@anchor{library/json json JSONDecodeError colno}@anchor{2d71}
@deffn {Attribute} colno

The column corresponding to @emph{pos}.
@end deffn


New in version 3.5.

@end deffn

@node Standard Compliance and Interoperability,Command Line Interface<2>,Exceptions<14>,json — JSON encoder and decoder
@anchor{library/json standard-compliance-and-interoperability}@anchor{2d72}
@subsubsection Standard Compliance and Interoperability


The JSON format is specified by 
@geindex RFC; RFC 7159
RFC 7159@footnote{https://tools.ietf.org/html/rfc7159.html} and by
ECMA-404@footnote{https://www.ecma-international.org/publications-and-standards/standards/ecma-404/}.
This section details this module’s level of compliance with the RFC.
For simplicity, @ref{89c,,JSONEncoder} and @ref{89d,,JSONDecoder} subclasses, and
parameters other than those explicitly mentioned, are not considered.

This module does not comply with the RFC in a strict fashion, implementing some
extensions that are valid JavaScript but not valid JSON.  In particular:


@itemize -

@item 
Infinite and NaN number values are accepted and output;

@item 
Repeated names within an object are accepted, and only the value of the last
name-value pair is used.
@end itemize

Since the RFC permits RFC-compliant parsers to accept input texts that are not
RFC-compliant, this module’s deserializer is technically RFC-compliant under
default settings.

@menu
* Character Encodings:: 
* Infinite and NaN Number Values:: 
* Repeated Names Within an Object:: 
* Top-level Non-Object@comma{} Non-Array Values: Top-level Non-Object Non-Array Values. 
* Implementation Limitations:: 

@end menu

@node Character Encodings,Infinite and NaN Number Values,,Standard Compliance and Interoperability
@anchor{library/json character-encodings}@anchor{2d73}
@subsubsection Character Encodings


The RFC requires that JSON be represented using either UTF-8, UTF-16, or
UTF-32, with UTF-8 being the recommended default for maximum interoperability.

As permitted, though not required, by the RFC, this module’s serializer sets
@emph{ensure_ascii=True} by default, thus escaping the output so that the resulting
strings only contain ASCII characters.

Other than the @emph{ensure_ascii} parameter, this module is defined strictly in
terms of conversion between Python objects and
@ref{2ef,,Unicode strings}, and thus does not otherwise directly address
the issue of character encodings.

The RFC prohibits adding a byte order mark (BOM) to the start of a JSON text,
and this module’s serializer does not add a BOM to its output.
The RFC permits, but does not require, JSON deserializers to ignore an initial
BOM in their input.  This module’s deserializer raises a @ref{17c,,ValueError}
when an initial BOM is present.

The RFC does not explicitly forbid JSON strings which contain byte sequences
that don’t correspond to valid Unicode characters (e.g. unpaired UTF-16
surrogates), but it does note that they may cause interoperability problems.
By default, this module accepts and outputs (when present in the original
@ref{2ef,,str}) code points for such sequences.

@node Infinite and NaN Number Values,Repeated Names Within an Object,Character Encodings,Standard Compliance and Interoperability
@anchor{library/json infinite-and-nan-number-values}@anchor{2d74}
@subsubsection Infinite and NaN Number Values


The RFC does not permit the representation of infinite or NaN number values.
Despite that, by default, this module accepts and outputs @code{Infinity},
@code{-Infinity}, and @code{NaN} as if they were valid JSON number literal values:

@example
>>> # Neither of these calls raises an exception, but the results are not valid JSON
>>> json.dumps(float('-inf'))
'-Infinity'
>>> json.dumps(float('nan'))
'NaN'
>>> # Same when deserializing
>>> json.loads('-Infinity')
-inf
>>> json.loads('NaN')
nan
@end example

In the serializer, the @emph{allow_nan} parameter can be used to alter this
behavior.  In the deserializer, the @emph{parse_constant} parameter can be used to
alter this behavior.

@node Repeated Names Within an Object,Top-level Non-Object Non-Array Values,Infinite and NaN Number Values,Standard Compliance and Interoperability
@anchor{library/json repeated-names-within-an-object}@anchor{2d75}
@subsubsection Repeated Names Within an Object


The RFC specifies that the names within a JSON object should be unique, but
does not mandate how repeated names in JSON objects should be handled.  By
default, this module does not raise an exception; instead, it ignores all but
the last name-value pair for a given name:

@example
>>> weird_json = '@{"x": 1, "x": 2, "x": 3@}'
>>> json.loads(weird_json)
@{'x': 3@}
@end example

The @emph{object_pairs_hook} parameter can be used to alter this behavior.

@node Top-level Non-Object Non-Array Values,Implementation Limitations,Repeated Names Within an Object,Standard Compliance and Interoperability
@anchor{library/json top-level-non-object-non-array-values}@anchor{2d76}
@subsubsection Top-level Non-Object, Non-Array Values


The old version of JSON specified by the obsolete 
@geindex RFC; RFC 4627
RFC 4627@footnote{https://tools.ietf.org/html/rfc4627.html} required that
the top-level value of a JSON text must be either a JSON object or array
(Python @ref{3b0,,dict} or @ref{43a,,list}), and could not be a JSON null,
boolean, number, or string value.  
@geindex RFC; RFC 7159
RFC 7159@footnote{https://tools.ietf.org/html/rfc7159.html} removed that restriction, and
this module does not and has never implemented that restriction in either its
serializer or its deserializer.

Regardless, for maximum interoperability, you may wish to voluntarily adhere
to the restriction yourself.

@node Implementation Limitations,,Top-level Non-Object Non-Array Values,Standard Compliance and Interoperability
@anchor{library/json implementation-limitations}@anchor{2d77}
@subsubsection Implementation Limitations


Some JSON deserializer implementations may set limits on:


@itemize *

@item 
the size of accepted JSON texts

@item 
the maximum level of nesting of JSON objects and arrays

@item 
the range and precision of JSON numbers

@item 
the content and maximum length of JSON strings
@end itemize

This module does not impose any such limits beyond those of the relevant
Python datatypes themselves or the Python interpreter itself.

When serializing to JSON, beware any such limitations in applications that may
consume your JSON.  In particular, it is common for JSON numbers to be
deserialized into IEEE 754 double precision numbers and thus subject to that
representation’s range and precision limitations.  This is especially relevant
when serializing Python @ref{1b1,,int} values of extremely large magnitude, or
when serializing instances of “exotic” numerical types such as
@ref{248,,decimal.Decimal}.

@node Command Line Interface<2>,,Standard Compliance and Interoperability,json — JSON encoder and decoder
@anchor{library/json command-line-interface}@anchor{2d78}@anchor{library/json json-commandline}@anchor{2d62}@anchor{library/json module-json tool}@anchor{a1}
@subsubsection Command Line Interface


@geindex module; json.tool

@strong{Source code:} Lib/json/tool.py@footnote{https://github.com/python/cpython/tree/main/Lib/json/tool.py}


__________________________________________________________________


The @ref{a1,,json.tool} module provides a simple command line interface to validate
and pretty-print JSON objects.

If the optional @code{infile} and @code{outfile} arguments are not
specified, @ref{5ee,,sys.stdin} and @ref{5ef,,sys.stdout} will be used respectively:

@example
$ echo '@{"json": "obj"@}' | python -m json.tool
@{
    "json": "obj"
@}
$ echo '@{1.2:3.4@}' | python -m json.tool
Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
@end example


Changed in version 3.5: The output is now in the same order as the input. Use the
@ref{2d79,,--sort-keys} option to sort the output of dictionaries
alphabetically by key.


@menu
* Command line options: Command line options<2>. 

@end menu

@node Command line options<2>,,,Command Line Interface<2>
@anchor{library/json command-line-options}@anchor{2d7a}
@subsubsection Command line options


@geindex json.tool command line option; infile
@anchor{library/json cmdoption-json tool-arg-infile}@anchor{2d7b}@anchor{library/json cmdoption-json-tool-arg-infile}@anchor{2d7c}
@deffn {Program Option} infile

The JSON file to be validated or pretty-printed:

@example
$ python -m json.tool mp_films.json
[
    @{
        "title": "And Now for Something Completely Different",
        "year": 1971
    @},
    @{
        "title": "Monty Python and the Holy Grail",
        "year": 1975
    @}
]
@end example

If @emph{infile} is not specified, read from @ref{5ee,,sys.stdin}.
@end deffn

@geindex json.tool command line option; outfile
@anchor{library/json cmdoption-json tool-arg-outfile}@anchor{2d7d}@anchor{library/json cmdoption-json-tool-arg-outfile}@anchor{2d7e}
@deffn {Program Option} outfile

Write the output of the @emph{infile} to the given @emph{outfile}. Otherwise, write it
to @ref{5ef,,sys.stdout}.
@end deffn

@geindex json.tool command line option; --sort-keys
@anchor{library/json cmdoption-json tool-sort-keys}@anchor{2d79}@anchor{library/json cmdoption-json-tool-sort-keys}@anchor{2d7f}
@deffn {Program Option} @w{-}@w{-}sort@w{-}keys

Sort the output of dictionaries alphabetically by key.


New in version 3.5.

@end deffn

@geindex json.tool command line option; --no-ensure-ascii
@anchor{library/json cmdoption-json tool-no-ensure-ascii}@anchor{2d80}@anchor{library/json cmdoption-json-tool-no-ensure-ascii}@anchor{2d81}
@deffn {Program Option} @w{-}@w{-}no@w{-}ensure@w{-}ascii

Disable escaping of non-ascii characters, see @ref{89b,,json.dumps()} for more information.


New in version 3.9.

@end deffn

@geindex json.tool command line option; --json-lines
@anchor{library/json cmdoption-json tool-json-lines}@anchor{2d82}@anchor{library/json cmdoption-json-tool-json-lines}@anchor{2d83}
@deffn {Program Option} @w{-}@w{-}json@w{-}lines

Parse every input line as separate JSON object.


New in version 3.8.

@end deffn

@geindex json.tool command line option; --indent
@geindex json.tool command line option; --tab
@geindex json.tool command line option; --no-indent
@geindex json.tool command line option; --compact
@anchor{library/json cmdoption-json tool-indent}@anchor{2d84}@anchor{library/json cmdoption-json-tool-indent}@anchor{2d85}@anchor{library/json cmdoption-json tool-tab}@anchor{2d86}@anchor{library/json cmdoption-json-tool-tab}@anchor{2d87}@anchor{library/json cmdoption-json tool-no-indent}@anchor{2d88}@anchor{library/json cmdoption-json-tool-no-indent}@anchor{2d89}@anchor{library/json cmdoption-json tool-compact}@anchor{2d8a}@anchor{library/json cmdoption-json-tool-compact}@anchor{2d8b}
@deffn {Program Option} @w{-}@w{-}indent, @w{-}@w{-}tab, @w{-}@w{-}no@w{-}indent, @w{-}@w{-}compact

Mutually exclusive options for whitespace control.


New in version 3.9.

@end deffn

@geindex json.tool command line option; -h
@geindex json.tool command line option; --help
@anchor{library/json cmdoption-json tool-h}@anchor{2d8c}@anchor{library/json cmdoption-json-tool-h}@anchor{2d8d}@anchor{library/json cmdoption-json tool-help}@anchor{2d8e}@anchor{library/json cmdoption-json-tool-help}@anchor{2d8f}
@deffn {Program Option} @w{-}h, @w{-}@w{-}help

Show the help message.
@end deffn

@node mailcap — Mailcap file handling,mailbox — Manipulate mailboxes in various formats,json — JSON encoder and decoder,Internet Data Handling
@anchor{library/mailcap doc}@anchor{2d90}@anchor{library/mailcap mailcap-mailcap-file-handling}@anchor{2d91}@anchor{library/mailcap module-mailcap}@anchor{ab}
@subsection @code{mailcap} — Mailcap file handling


@geindex module; mailcap

@strong{Source code:} Lib/mailcap.py@footnote{https://github.com/python/cpython/tree/main/Lib/mailcap.py}


__________________________________________________________________


Mailcap files are used to configure how MIME-aware applications such as mail
readers and web browsers react to files with different MIME types. (The name
“mailcap” is derived from the phrase “mail capability”.)  For example, a mailcap
file might contain a line like @code{video/mpeg; xmpeg %s}.  Then, if the user
encounters an email message or web document with the MIME type
@code{video/mpeg}, @code{%s} will be replaced by a filename (usually one
belonging to a temporary file) and the @code{xmpeg} program can be
automatically started to view the file.

The mailcap format is documented in 
@geindex RFC; RFC 1524
RFC 1524@footnote{https://tools.ietf.org/html/rfc1524.html}, “A User Agent Configuration
Mechanism For Multimedia Mail Format Information”, but is not an internet
standard.  However, mailcap files are supported on most Unix systems.

@geindex findmatch() (in module mailcap)
@anchor{library/mailcap mailcap findmatch}@anchor{2d92}
@deffn {Function} mailcap.findmatch (caps, MIMEtype, key='view', filename='/dev/null', plist=[])

Return a 2-tuple; the first element is a string containing the command line to
be executed (which can be passed to @ref{fbb,,os.system()}), and the second element
is the mailcap entry for a given MIME type.  If no matching MIME type can be
found, @code{(None, None)} is returned.

@emph{key} is the name of the field desired, which represents the type of activity to
be performed; the default value is ‘view’, since in the  most common case you
simply want to view the body of the MIME-typed data.  Other possible values
might be ‘compose’ and ‘edit’, if you wanted to create a new body of the given
MIME type or alter the existing body data.  See 
@geindex RFC; RFC 1524
RFC 1524@footnote{https://tools.ietf.org/html/rfc1524.html} for a complete list
of these fields.

@emph{filename} is the filename to be substituted for @code{%s} in the command line; the
default value is @code{'/dev/null'} which is almost certainly not what you want, so
usually you’ll override it by specifying a filename.

@emph{plist} can be a list containing named parameters; the default value is simply
an empty list.  Each entry in the list must be a string containing the parameter
name, an equals sign (@code{'='}), and the parameter’s value.  Mailcap entries can
contain  named parameters like @code{%@{foo@}}, which will be replaced by the value
of the parameter named ‘foo’.  For example, if the command line @code{showpartial
%@{id@} %@{number@} %@{total@}} was in a mailcap file, and @emph{plist} was set to
@code{['id=1', 'number=2', 'total=3']}, the resulting command line would be
@code{'showpartial 1 2 3'}.

In a mailcap file, the “test” field can optionally be specified to test some
external condition (such as the machine architecture, or the window system in
use) to determine whether or not the mailcap line applies.  @ref{2d92,,findmatch()}
will automatically check such conditions and skip the entry if the check fails.
@end deffn

@geindex getcaps() (in module mailcap)
@anchor{library/mailcap mailcap getcaps}@anchor{2d93}
@deffn {Function} mailcap.getcaps ()

Returns a dictionary mapping MIME types to a list of mailcap file entries. This
dictionary must be passed to the @ref{2d92,,findmatch()} function.  An entry is stored
as a list of dictionaries, but it shouldn’t be necessary to know the details of
this representation.

The information is derived from all of the mailcap files found on the system.
Settings in the user’s mailcap file @code{$HOME/.mailcap} will override
settings in the system mailcap files @code{/etc/mailcap},
@code{/usr/etc/mailcap}, and @code{/usr/local/etc/mailcap}.
@end deffn

An example usage:

@example
>>> import mailcap
>>> d = mailcap.getcaps()
>>> mailcap.findmatch(d, 'video/mpeg', filename='tmp1223')
('xmpeg tmp1223', @{'view': 'xmpeg %s'@})
@end example

@node mailbox — Manipulate mailboxes in various formats,mimetypes — Map filenames to MIME types,mailcap — Mailcap file handling,Internet Data Handling
@anchor{library/mailbox doc}@anchor{2d94}@anchor{library/mailbox mailbox-manipulate-mailboxes-in-various-formats}@anchor{2d95}@anchor{library/mailbox module-mailbox}@anchor{aa}
@subsection @code{mailbox} — Manipulate mailboxes in various formats


@geindex module; mailbox

@strong{Source code:} Lib/mailbox.py@footnote{https://github.com/python/cpython/tree/main/Lib/mailbox.py}


__________________________________________________________________


This module defines two classes, @ref{e00,,Mailbox} and @ref{e04,,Message}, for
accessing and manipulating on-disk mailboxes and the messages they contain.
@ref{e00,,Mailbox} offers a dictionary-like mapping from keys to messages.
@ref{e04,,Message} extends the @ref{6d,,email.message} module’s
@ref{7e3,,Message} class with format-specific state and behavior.
Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF.


@subsubheading See also



@table @asis

@item Module @ref{64,,email}

Represent and manipulate messages.
@end table


@menu
* Mailbox objects:: 
* Message objects:: 
* Exceptions: Exceptions<15>. 
* Examples: Examples<21>. 

@end menu

@node Mailbox objects,Message objects,,mailbox — Manipulate mailboxes in various formats
@anchor{library/mailbox id1}@anchor{2d96}@anchor{library/mailbox mailbox-objects}@anchor{2d97}
@subsubsection @code{Mailbox} objects


@geindex Mailbox (class in mailbox)
@anchor{library/mailbox mailbox Mailbox}@anchor{e00}
@deffn {Class} mailbox.Mailbox

A mailbox, which may be inspected and modified.

The @ref{e00,,Mailbox} class defines an interface and is not intended to be
instantiated.  Instead, format-specific subclasses should inherit from
@ref{e00,,Mailbox} and your code should instantiate a particular subclass.

The @ref{e00,,Mailbox} interface is dictionary-like, with small keys
corresponding to messages. Keys are issued by the @ref{e00,,Mailbox} instance
with which they will be used and are only meaningful to that @ref{e00,,Mailbox}
instance. A key continues to identify a message even if the corresponding
message is modified, such as by replacing it with another message.

Messages may be added to a @ref{e00,,Mailbox} instance using the set-like
method @ref{dff,,add()} and removed using a @code{del} statement or the set-like
methods @ref{2d98,,remove()} and @ref{2d99,,discard()}.

@ref{e00,,Mailbox} interface semantics differ from dictionary semantics in some
noteworthy ways. Each time a message is requested, a new representation
(typically a @ref{e04,,Message} instance) is generated based upon the current
state of the mailbox. Similarly, when a message is added to a
@ref{e00,,Mailbox} instance, the provided message representation’s contents are
copied. In neither case is a reference to the message representation kept by
the @ref{e00,,Mailbox} instance.

The default @ref{e00,,Mailbox} iterator iterates over message representations,
not keys as the default dictionary iterator does. Moreover, modification of a
mailbox during iteration is safe and well-defined. Messages added to the
mailbox after an iterator is created will not be seen by the
iterator. Messages removed from the mailbox before the iterator yields them
will be silently skipped, though using a key from an iterator may result in a
@ref{5bb,,KeyError} exception if the corresponding message is subsequently
removed.

@cartouche
@quotation Warning 
Be very cautious when modifying mailboxes that might be simultaneously
changed by some other process.  The safest mailbox format to use for such
tasks is Maildir; try to avoid using single-file formats such as mbox for
concurrent writing.  If you’re modifying a mailbox, you @emph{must} lock it by
calling the @ref{2d9a,,lock()} and @ref{2d9b,,unlock()} methods @emph{before} reading any
messages in the file or making any changes by adding or deleting a
message.  Failing to lock the mailbox runs the risk of losing messages or
corrupting the entire mailbox.
@end quotation
@end cartouche

@ref{e00,,Mailbox} instances have the following methods:

@geindex add() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox add}@anchor{dff}
@deffn {Method} add (message)

Add @emph{message} to the mailbox and return the key that has been assigned to
it.

Parameter @emph{message} may be a @ref{e04,,Message} instance, an
@ref{7e3,,email.message.Message} instance, a string, a byte string, or a
file-like object (which should be open in binary mode). If @emph{message} is
an instance of the
appropriate format-specific @ref{e04,,Message} subclass (e.g., if it’s an
@ref{2c5b,,mboxMessage} instance and this is an @ref{2d9c,,mbox} instance), its
format-specific information is used. Otherwise, reasonable defaults for
format-specific information are used.


Changed in version 3.2: Support for binary input was added.

@end deffn

@geindex remove() (mailbox.Mailbox method)
@geindex __delitem__() (mailbox.Mailbox method)
@geindex discard() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox remove}@anchor{2d98}
@deffn {Method} remove (key)
@anchor{library/mailbox mailbox Mailbox __delitem__}@anchor{2d9d}
@deffnx {Method} __delitem__ (key)
@anchor{library/mailbox mailbox Mailbox discard}@anchor{2d99}
@deffnx {Method} discard (key)

Delete the message corresponding to @emph{key} from the mailbox.

If no such message exists, a @ref{5bb,,KeyError} exception is raised if the
method was called as @ref{2d98,,remove()} or @ref{2d9d,,__delitem__()} but no
exception is raised if the method was called as @ref{2d99,,discard()}. The
behavior of @ref{2d99,,discard()} may be preferred if the underlying mailbox
format supports concurrent modification by other processes.
@end deffn

@geindex __setitem__() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox __setitem__}@anchor{2d9e}
@deffn {Method} __setitem__ (key, message)

Replace the message corresponding to @emph{key} with @emph{message}. Raise a
@ref{5bb,,KeyError} exception if no message already corresponds to @emph{key}.

As with @ref{dff,,add()}, parameter @emph{message} may be a @ref{e04,,Message}
instance, an @ref{7e3,,email.message.Message} instance, a string, a byte
string, or a file-like object (which should be open in binary mode). If
@emph{message} is an
instance of the appropriate format-specific @ref{e04,,Message} subclass
(e.g., if it’s an @ref{2c5b,,mboxMessage} instance and this is an
@ref{2d9c,,mbox} instance), its format-specific information is
used. Otherwise, the format-specific information of the message that
currently corresponds to @emph{key} is left unchanged.
@end deffn

@geindex iterkeys() (mailbox.Mailbox method)
@geindex keys() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox iterkeys}@anchor{2d9f}
@deffn {Method} iterkeys ()
@anchor{library/mailbox mailbox Mailbox keys}@anchor{2da0}
@deffnx {Method} keys ()

Return an iterator over all keys if called as @ref{2d9f,,iterkeys()} or return a
list of keys if called as @ref{2da0,,keys()}.
@end deffn

@geindex itervalues() (mailbox.Mailbox method)
@geindex __iter__() (mailbox.Mailbox method)
@geindex values() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox itervalues}@anchor{2da1}
@deffn {Method} itervalues ()
@anchor{library/mailbox mailbox Mailbox __iter__}@anchor{2da2}
@deffnx {Method} __iter__ ()
@anchor{library/mailbox mailbox Mailbox values}@anchor{2da3}
@deffnx {Method} values ()

Return an iterator over representations of all messages if called as
@ref{2da1,,itervalues()} or @ref{2da2,,__iter__()} or return a list of such
representations if called as @ref{2da3,,values()}. The messages are represented
as instances of the appropriate format-specific @ref{e04,,Message} subclass
unless a custom message factory was specified when the @ref{e00,,Mailbox}
instance was initialized.

@cartouche
@quotation Note 
The behavior of @ref{2da2,,__iter__()} is unlike that of dictionaries, which
iterate over keys.
@end quotation
@end cartouche
@end deffn

@geindex iteritems() (mailbox.Mailbox method)
@geindex items() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox iteritems}@anchor{2da4}
@deffn {Method} iteritems ()
@anchor{library/mailbox mailbox Mailbox items}@anchor{2da5}
@deffnx {Method} items ()

Return an iterator over (@emph{key}, @emph{message}) pairs, where @emph{key} is a key and
@emph{message} is a message representation, if called as @ref{2da4,,iteritems()} or
return a list of such pairs if called as @ref{2da5,,items()}. The messages are
represented as instances of the appropriate format-specific
@ref{e04,,Message} subclass unless a custom message factory was specified
when the @ref{e00,,Mailbox} instance was initialized.
@end deffn

@geindex get() (mailbox.Mailbox method)
@geindex __getitem__() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox get}@anchor{2da6}
@deffn {Method} get (key, default=None)
@anchor{library/mailbox mailbox Mailbox __getitem__}@anchor{2da7}
@deffnx {Method} __getitem__ (key)

Return a representation of the message corresponding to @emph{key}. If no such
message exists, @emph{default} is returned if the method was called as
@ref{2da6,,get()} and a @ref{5bb,,KeyError} exception is raised if the method was
called as @ref{2da7,,__getitem__()}. The message is represented as an instance
of the appropriate format-specific @ref{e04,,Message} subclass unless a
custom message factory was specified when the @ref{e00,,Mailbox} instance
was initialized.
@end deffn

@geindex get_message() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox get_message}@anchor{2da8}
@deffn {Method} get_message (key)

Return a representation of the message corresponding to @emph{key} as an
instance of the appropriate format-specific @ref{e04,,Message} subclass, or
raise a @ref{5bb,,KeyError} exception if no such message exists.
@end deffn

@geindex get_bytes() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox get_bytes}@anchor{e02}
@deffn {Method} get_bytes (key)

Return a byte representation of the message corresponding to @emph{key}, or
raise a @ref{5bb,,KeyError} exception if no such message exists.


New in version 3.2.

@end deffn

@geindex get_string() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox get_string}@anchor{e03}
@deffn {Method} get_string (key)

Return a string representation of the message corresponding to @emph{key}, or
raise a @ref{5bb,,KeyError} exception if no such message exists.  The
message is processed through @ref{7e3,,email.message.Message} to
convert it to a 7bit clean representation.
@end deffn

@geindex get_file() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox get_file}@anchor{e01}
@deffn {Method} get_file (key)

Return a file-like representation of the message corresponding to @emph{key},
or raise a @ref{5bb,,KeyError} exception if no such message exists.  The
file-like object behaves as if open in binary mode.  This file should be
closed once it is no longer needed.


Changed in version 3.2: The file object really is a binary file; previously it was incorrectly
returned in text mode.  Also, the file-like object now supports the
context management protocol: you can use a @ref{168,,with} statement to
automatically close it.


@cartouche
@quotation Note 
Unlike other representations of messages, file-like representations are
not necessarily independent of the @ref{e00,,Mailbox} instance that
created them or of the underlying mailbox.  More specific documentation
is provided by each subclass.
@end quotation
@end cartouche
@end deffn

@geindex __contains__() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox __contains__}@anchor{2da9}
@deffn {Method} __contains__ (key)

Return @code{True} if @emph{key} corresponds to a message, @code{False} otherwise.
@end deffn

@geindex __len__() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox __len__}@anchor{2daa}
@deffn {Method} __len__ ()

Return a count of messages in the mailbox.
@end deffn

@geindex clear() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox clear}@anchor{2dab}
@deffn {Method} clear ()

Delete all messages from the mailbox.
@end deffn

@geindex pop() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox pop}@anchor{2dac}
@deffn {Method} pop (key, default=None)

Return a representation of the message corresponding to @emph{key} and delete
the message. If no such message exists, return @emph{default}. The message is
represented as an instance of the appropriate format-specific
@ref{e04,,Message} subclass unless a custom message factory was specified
when the @ref{e00,,Mailbox} instance was initialized.
@end deffn

@geindex popitem() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox popitem}@anchor{2dad}
@deffn {Method} popitem ()

Return an arbitrary (@emph{key}, @emph{message}) pair, where @emph{key} is a key and
@emph{message} is a message representation, and delete the corresponding
message. If the mailbox is empty, raise a @ref{5bb,,KeyError} exception. The
message is represented as an instance of the appropriate format-specific
@ref{e04,,Message} subclass unless a custom message factory was specified
when the @ref{e00,,Mailbox} instance was initialized.
@end deffn

@geindex update() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox update}@anchor{2dae}
@deffn {Method} update (arg)

Parameter @emph{arg} should be a @emph{key}-to-@emph{message} mapping or an iterable of
(@emph{key}, @emph{message}) pairs. Updates the mailbox so that, for each given
@emph{key} and @emph{message}, the message corresponding to @emph{key} is set to
@emph{message} as if by using @ref{2d9e,,__setitem__()}. As with @ref{2d9e,,__setitem__()},
each @emph{key} must already correspond to a message in the mailbox or else a
@ref{5bb,,KeyError} exception will be raised, so in general it is incorrect
for @emph{arg} to be a @ref{e00,,Mailbox} instance.

@cartouche
@quotation Note 
Unlike with dictionaries, keyword arguments are not supported.
@end quotation
@end cartouche
@end deffn

@geindex flush() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox flush}@anchor{2daf}
@deffn {Method} flush ()

Write any pending changes to the filesystem. For some @ref{e00,,Mailbox}
subclasses, changes are always written immediately and @ref{2daf,,flush()} does
nothing, but you should still make a habit of calling this method.
@end deffn

@geindex lock() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox lock}@anchor{2d9a}
@deffn {Method} lock ()

Acquire an exclusive advisory lock on the mailbox so that other processes
know not to modify it. An @ref{2db0,,ExternalClashError} is raised if the lock
is not available. The particular locking mechanisms used depend upon the
mailbox format.  You should @emph{always} lock the mailbox before making any
modifications to its contents.
@end deffn

@geindex unlock() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox unlock}@anchor{2d9b}
@deffn {Method} unlock ()

Release the lock on the mailbox, if any.
@end deffn

@geindex close() (mailbox.Mailbox method)
@anchor{library/mailbox mailbox Mailbox close}@anchor{2db1}
@deffn {Method} close ()

Flush the mailbox, unlock it if necessary, and close any open files. For
some @ref{e00,,Mailbox} subclasses, this method does nothing.
@end deffn
@end deffn

@menu
* Maildir:: 
* mbox:: 
* MH:: 
* Babyl:: 
* MMDF:: 

@end menu

@node Maildir,mbox,,Mailbox objects
@anchor{library/mailbox mailbox-maildir}@anchor{2db2}@anchor{library/mailbox maildir}@anchor{2db3}
@subsubsection @code{Maildir}


@geindex Maildir (class in mailbox)
@anchor{library/mailbox mailbox Maildir}@anchor{eaa}
@deffn {Class} mailbox.Maildir (dirname, factory=None, create=True)

A subclass of @ref{e00,,Mailbox} for mailboxes in Maildir format. Parameter
@emph{factory} is a callable object that accepts a file-like message representation
(which behaves as if opened in binary mode) and returns a custom representation.
If @emph{factory} is @code{None}, @ref{2db4,,MaildirMessage} is used as the default message
representation. If @emph{create} is @code{True}, the mailbox is created if it does not
exist.

If @emph{create} is @code{True} and the @emph{dirname} path exists, it will be treated as
an existing maildir without attempting to verify its directory layout.

It is for historical reasons that @emph{dirname} is named as such rather than @emph{path}.

Maildir is a directory-based mailbox format invented for the qmail mail
transfer agent and now widely supported by other programs. Messages in a
Maildir mailbox are stored in separate files within a common directory
structure. This design allows Maildir mailboxes to be accessed and modified
by multiple unrelated programs without data corruption, so file locking is
unnecessary.

Maildir mailboxes contain three subdirectories, namely: @code{tmp},
@code{new}, and @code{cur}. Messages are created momentarily in the
@code{tmp} subdirectory and then moved to the @code{new} subdirectory to
finalize delivery. A mail user agent may subsequently move the message to the
@code{cur} subdirectory and store information about the state of the message
in a special “info” section appended to its file name.

Folders of the style introduced by the Courier mail transfer agent are also
supported. Any subdirectory of the main mailbox is considered a folder if
@code{'.'} is the first character in its name. Folder names are represented by
@ref{eaa,,Maildir} without the leading @code{'.'}. Each folder is itself a Maildir
mailbox but should not contain other folders. Instead, a logical nesting is
indicated using @code{'.'} to delimit levels, e.g., “Archived.2005.07”.

@cartouche
@quotation Note 
The Maildir specification requires the use of a colon (@code{':'}) in certain
message file names. However, some operating systems do not permit this
character in file names, If you wish to use a Maildir-like format on such
an operating system, you should specify another character to use
instead. The exclamation point (@code{'!'}) is a popular choice. For
example:

@example
import mailbox
mailbox.Maildir.colon = '!'
@end example

The @code{colon} attribute may also be set on a per-instance basis.
@end quotation
@end cartouche

@ref{eaa,,Maildir} instances have all of the methods of @ref{e00,,Mailbox} in
addition to the following:

@geindex list_folders() (mailbox.Maildir method)
@anchor{library/mailbox mailbox Maildir list_folders}@anchor{2db5}
@deffn {Method} list_folders ()

Return a list of the names of all folders.
@end deffn

@geindex get_folder() (mailbox.Maildir method)
@anchor{library/mailbox mailbox Maildir get_folder}@anchor{2db6}
@deffn {Method} get_folder (folder)

Return a @ref{eaa,,Maildir} instance representing the folder whose name is
@emph{folder}. A @ref{2db7,,NoSuchMailboxError} exception is raised if the folder
does not exist.
@end deffn

@geindex add_folder() (mailbox.Maildir method)
@anchor{library/mailbox mailbox Maildir add_folder}@anchor{2db8}
@deffn {Method} add_folder (folder)

Create a folder whose name is @emph{folder} and return a @ref{eaa,,Maildir}
instance representing it.
@end deffn

@geindex remove_folder() (mailbox.Maildir method)
@anchor{library/mailbox mailbox Maildir remove_folder}@anchor{2db9}
@deffn {Method} remove_folder (folder)

Delete the folder whose name is @emph{folder}. If the folder contains any
messages, a @ref{2dba,,NotEmptyError} exception will be raised and the folder
will not be deleted.
@end deffn

@geindex clean() (mailbox.Maildir method)
@anchor{library/mailbox mailbox Maildir clean}@anchor{2dbb}
@deffn {Method} clean ()

Delete temporary files from the mailbox that have not been accessed in the
last 36 hours. The Maildir specification says that mail-reading programs
should do this occasionally.
@end deffn

Some @ref{e00,,Mailbox} methods implemented by @ref{eaa,,Maildir} deserve special
remarks:

@geindex add() (mailbox.Maildir method)
@geindex __setitem__() (mailbox.Maildir method)
@geindex update() (mailbox.Maildir method)
@anchor{library/mailbox mailbox Maildir add}@anchor{2dbc}
@deffn {Method} add (message)
@anchor{library/mailbox mailbox Maildir __setitem__}@anchor{2dbd}
@deffnx {Method} __setitem__ (key, message)
@anchor{library/mailbox mailbox Maildir update}@anchor{2dbe}
@deffnx {Method} update (arg)

@cartouche
@quotation Warning 
These methods generate unique file names based upon the current process
ID. When using multiple threads, undetected name clashes may occur and
cause corruption of the mailbox unless threads are coordinated to avoid
using these methods to manipulate the same mailbox simultaneously.
@end quotation
@end cartouche
@end deffn

@geindex flush() (mailbox.Maildir method)
@anchor{library/mailbox mailbox Maildir flush}@anchor{2dbf}
@deffn {Method} flush ()

All changes to Maildir mailboxes are immediately applied, so this method
does nothing.
@end deffn

@geindex lock() (mailbox.Maildir method)
@geindex unlock() (mailbox.Maildir method)
@anchor{library/mailbox mailbox Maildir lock}@anchor{2dc0}
@deffn {Method} lock ()
@anchor{library/mailbox mailbox Maildir unlock}@anchor{2dc1}
@deffnx {Method} unlock ()

Maildir mailboxes do not support (or require) locking, so these methods do
nothing.
@end deffn

@geindex close() (mailbox.Maildir method)
@anchor{library/mailbox mailbox Maildir close}@anchor{2dc2}
@deffn {Method} close ()

@ref{eaa,,Maildir} instances do not keep any open files and the underlying
mailboxes do not support locking, so this method does nothing.
@end deffn

@geindex get_file() (mailbox.Maildir method)
@anchor{library/mailbox mailbox Maildir get_file}@anchor{2dc3}
@deffn {Method} get_file (key)

Depending upon the host platform, it may not be possible to modify or
remove the underlying message while the returned file remains open.
@end deffn
@end deffn


@subsubheading See also



@table @asis

@item maildir man page from Courier@footnote{http://www.courier-mta.org/maildir.html}

A specification of the format. Describes a common extension for
supporting folders.

@item Using maildir format@footnote{https://cr.yp.to/proto/maildir.html}

Notes on Maildir by its inventor. Includes an updated name-creation scheme and
details on “info” semantics.
@end table


@node mbox,MH,Maildir,Mailbox objects
@anchor{library/mailbox mailbox-mbox}@anchor{2dc4}@anchor{library/mailbox mbox}@anchor{2dc5}
@subsubsection @code{mbox}


@geindex mbox (class in mailbox)
@anchor{library/mailbox mailbox mbox}@anchor{2d9c}
@deffn {Class} mailbox.mbox (path, factory=None, create=True)

A subclass of @ref{e00,,Mailbox} for mailboxes in mbox format. Parameter @emph{factory}
is a callable object that accepts a file-like message representation (which
behaves as if opened in binary mode) and returns a custom representation. If
@emph{factory} is @code{None}, @ref{2c5b,,mboxMessage} is used as the default message
representation. If @emph{create} is @code{True}, the mailbox is created if it does not
exist.

The mbox format is the classic format for storing mail on Unix systems. All
messages in an mbox mailbox are stored in a single file with the beginning of
each message indicated by a line whose first five characters are “From “.

Several variations of the mbox format exist to address perceived shortcomings in
the original. In the interest of compatibility, @ref{2d9c,,mbox} implements the
original format, which is sometimes referred to as @emph{mboxo}. This means that
the @code{Content-Length} header, if present, is ignored and that any
occurrences of “From ” at the beginning of a line in a message body are
transformed to “>From ” when storing the message, although occurrences of “>From
” are not transformed to “From ” when reading the message.

Some @ref{e00,,Mailbox} methods implemented by @ref{2d9c,,mbox} deserve special
remarks:

@geindex get_file() (mailbox.mbox method)
@anchor{library/mailbox mailbox mbox get_file}@anchor{2dc6}
@deffn {Method} get_file (key)

Using the file after calling @code{flush()} or @code{close()} on the
@ref{2d9c,,mbox} instance may yield unpredictable results or raise an
exception.
@end deffn

@geindex lock() (mailbox.mbox method)
@geindex unlock() (mailbox.mbox method)
@anchor{library/mailbox mailbox mbox lock}@anchor{2dc7}
@deffn {Method} lock ()
@anchor{library/mailbox mailbox mbox unlock}@anchor{2dc8}
@deffnx {Method} unlock ()

Three locking mechanisms are used—dot locking and, if available, the
@code{flock()} and @code{lockf()} system calls.
@end deffn
@end deffn


@subsubheading See also



@table @asis

@item mbox man page from tin@footnote{http://www.tin.org/bin/man.cgi?section=5&topic=mbox}

A specification of the format, with details on locking.

@item Configuring Netscape Mail on Unix: Why The Content-Length Format is Bad@footnote{https://www.jwz.org/doc/content-length.html}

An argument for using the original mbox format rather than a variation.

@item "mbox" is a family of several mutually incompatible mailbox formats@footnote{https://www.loc.gov/preservation/digital/formats/fdd/fdd000383.shtml}

A history of mbox variations.
@end table


@node MH,Babyl,mbox,Mailbox objects
@anchor{library/mailbox mailbox-mh}@anchor{2dc9}@anchor{library/mailbox mh}@anchor{2dca}
@subsubsection @code{MH}


@geindex MH (class in mailbox)
@anchor{library/mailbox mailbox MH}@anchor{2dcb}
@deffn {Class} mailbox.MH (path, factory=None, create=True)

A subclass of @ref{e00,,Mailbox} for mailboxes in MH format. Parameter @emph{factory}
is a callable object that accepts a file-like message representation (which
behaves as if opened in binary mode) and returns a custom representation. If
@emph{factory} is @code{None}, @ref{2dcc,,MHMessage} is used as the default message
representation. If @emph{create} is @code{True}, the mailbox is created if it does not
exist.

MH is a directory-based mailbox format invented for the MH Message Handling
System, a mail user agent. Each message in an MH mailbox resides in its own
file. An MH mailbox may contain other MH mailboxes (called @emph{folders}) in
addition to messages. Folders may be nested indefinitely. MH mailboxes also
support @emph{sequences}, which are named lists used to logically group
messages without moving them to sub-folders. Sequences are defined in a file
called @code{.mh_sequences} in each folder.

The @ref{2dcb,,MH} class manipulates MH mailboxes, but it does not attempt to
emulate all of @code{mh}’s behaviors. In particular, it does not modify
and is not affected by the @code{context} or @code{.mh_profile} files that
are used by @code{mh} to store its state and configuration.

@ref{2dcb,,MH} instances have all of the methods of @ref{e00,,Mailbox} in addition
to the following:

@geindex list_folders() (mailbox.MH method)
@anchor{library/mailbox mailbox MH list_folders}@anchor{2dcd}
@deffn {Method} list_folders ()

Return a list of the names of all folders.
@end deffn

@geindex get_folder() (mailbox.MH method)
@anchor{library/mailbox mailbox MH get_folder}@anchor{2dce}
@deffn {Method} get_folder (folder)

Return an @ref{2dcb,,MH} instance representing the folder whose name is
@emph{folder}. A @ref{2db7,,NoSuchMailboxError} exception is raised if the folder
does not exist.
@end deffn

@geindex add_folder() (mailbox.MH method)
@anchor{library/mailbox mailbox MH add_folder}@anchor{2dcf}
@deffn {Method} add_folder (folder)

Create a folder whose name is @emph{folder} and return an @ref{2dcb,,MH} instance
representing it.
@end deffn

@geindex remove_folder() (mailbox.MH method)
@anchor{library/mailbox mailbox MH remove_folder}@anchor{2dd0}
@deffn {Method} remove_folder (folder)

Delete the folder whose name is @emph{folder}. If the folder contains any
messages, a @ref{2dba,,NotEmptyError} exception will be raised and the folder
will not be deleted.
@end deffn

@geindex get_sequences() (mailbox.MH method)
@anchor{library/mailbox mailbox MH get_sequences}@anchor{2dd1}
@deffn {Method} get_sequences ()

Return a dictionary of sequence names mapped to key lists. If there are no
sequences, the empty dictionary is returned.
@end deffn

@geindex set_sequences() (mailbox.MH method)
@anchor{library/mailbox mailbox MH set_sequences}@anchor{2dd2}
@deffn {Method} set_sequences (sequences)

Re-define the sequences that exist in the mailbox based upon @emph{sequences},
a dictionary of names mapped to key lists, like returned by
@ref{2dd1,,get_sequences()}.
@end deffn

@geindex pack() (mailbox.MH method)
@anchor{library/mailbox mailbox MH pack}@anchor{2dd3}
@deffn {Method} pack ()

Rename messages in the mailbox as necessary to eliminate gaps in
numbering.  Entries in the sequences list are updated correspondingly.

@cartouche
@quotation Note 
Already-issued keys are invalidated by this operation and should not be
subsequently used.
@end quotation
@end cartouche
@end deffn

Some @ref{e00,,Mailbox} methods implemented by @ref{2dcb,,MH} deserve special
remarks:

@geindex remove() (mailbox.MH method)
@geindex __delitem__() (mailbox.MH method)
@geindex discard() (mailbox.MH method)
@anchor{library/mailbox mailbox MH remove}@anchor{2dd4}
@deffn {Method} remove (key)
@anchor{library/mailbox mailbox MH __delitem__}@anchor{2dd5}
@deffnx {Method} __delitem__ (key)
@anchor{library/mailbox mailbox MH discard}@anchor{2dd6}
@deffnx {Method} discard (key)

These methods immediately delete the message. The MH convention of marking
a message for deletion by prepending a comma to its name is not used.
@end deffn

@geindex lock() (mailbox.MH method)
@geindex unlock() (mailbox.MH method)
@anchor{library/mailbox mailbox MH lock}@anchor{2dd7}
@deffn {Method} lock ()
@anchor{library/mailbox mailbox MH unlock}@anchor{2dd8}
@deffnx {Method} unlock ()

Three locking mechanisms are used—dot locking and, if available, the
@code{flock()} and @code{lockf()} system calls. For MH mailboxes, locking
the mailbox means locking the @code{.mh_sequences} file and, only for the
duration of any operations that affect them, locking individual message
files.
@end deffn

@geindex get_file() (mailbox.MH method)
@anchor{library/mailbox mailbox MH get_file}@anchor{2dd9}
@deffn {Method} get_file (key)

Depending upon the host platform, it may not be possible to remove the
underlying message while the returned file remains open.
@end deffn

@geindex flush() (mailbox.MH method)
@anchor{library/mailbox mailbox MH flush}@anchor{2dda}
@deffn {Method} flush ()

All changes to MH mailboxes are immediately applied, so this method does
nothing.
@end deffn

@geindex close() (mailbox.MH method)
@anchor{library/mailbox mailbox MH close}@anchor{2ddb}
@deffn {Method} close ()

@ref{2dcb,,MH} instances do not keep any open files, so this method is
equivalent to @ref{2dd8,,unlock()}.
@end deffn
@end deffn


@subsubheading See also



@table @asis

@item nmh - Message Handling System@footnote{http://www.nongnu.org/nmh/}

Home page of @code{nmh}, an updated version of the original @code{mh}.

@item MH & nmh: Email for Users & Programmers@footnote{https://rand-mh.sourceforge.io/book/}

A GPL-licensed book on @code{mh} and @code{nmh}, with some information
on the mailbox format.
@end table


@node Babyl,MMDF,MH,Mailbox objects
@anchor{library/mailbox babyl}@anchor{2ddc}@anchor{library/mailbox mailbox-babyl}@anchor{2ddd}
@subsubsection @code{Babyl}


@geindex Babyl (class in mailbox)
@anchor{library/mailbox mailbox Babyl}@anchor{2dde}
@deffn {Class} mailbox.Babyl (path, factory=None, create=True)

A subclass of @ref{e00,,Mailbox} for mailboxes in Babyl format. Parameter
@emph{factory} is a callable object that accepts a file-like message representation
(which behaves as if opened in binary mode) and returns a custom representation.
If @emph{factory} is @code{None}, @ref{2ddf,,BabylMessage} is used as the default message
representation. If @emph{create} is @code{True}, the mailbox is created if it does not
exist.

Babyl is a single-file mailbox format used by the Rmail mail user agent
included with Emacs. The beginning of a message is indicated by a line
containing the two characters Control-Underscore (@code{'\037'}) and Control-L
(@code{'\014'}). The end of a message is indicated by the start of the next
message or, in the case of the last message, a line containing a
Control-Underscore (@code{'\037'}) character.

Messages in a Babyl mailbox have two sets of headers, original headers and
so-called visible headers. Visible headers are typically a subset of the
original headers that have been reformatted or abridged to be more
attractive. Each message in a Babyl mailbox also has an accompanying list of
@emph{labels}, or short strings that record extra information about the
message, and a list of all user-defined labels found in the mailbox is kept
in the Babyl options section.

@ref{2dde,,Babyl} instances have all of the methods of @ref{e00,,Mailbox} in
addition to the following:

@geindex get_labels() (mailbox.Babyl method)
@anchor{library/mailbox mailbox Babyl get_labels}@anchor{2de0}
@deffn {Method} get_labels ()

Return a list of the names of all user-defined labels used in the mailbox.

@cartouche
@quotation Note 
The actual messages are inspected to determine which labels exist in
the mailbox rather than consulting the list of labels in the Babyl
options section, but the Babyl section is updated whenever the mailbox
is modified.
@end quotation
@end cartouche
@end deffn

Some @ref{e00,,Mailbox} methods implemented by @ref{2dde,,Babyl} deserve special
remarks:

@geindex get_file() (mailbox.Babyl method)
@anchor{library/mailbox mailbox Babyl get_file}@anchor{2de1}
@deffn {Method} get_file (key)

In Babyl mailboxes, the headers of a message are not stored contiguously
with the body of the message. To generate a file-like representation, the
headers and body are copied together into an @ref{a15,,io.BytesIO} instance,
which has an API identical to that of a
file. As a result, the file-like object is truly independent of the
underlying mailbox but does not save memory compared to a string
representation.
@end deffn

@geindex lock() (mailbox.Babyl method)
@geindex unlock() (mailbox.Babyl method)
@anchor{library/mailbox mailbox Babyl lock}@anchor{2de2}
@deffn {Method} lock ()
@anchor{library/mailbox mailbox Babyl unlock}@anchor{2de3}
@deffnx {Method} unlock ()

Three locking mechanisms are used—dot locking and, if available, the
@code{flock()} and @code{lockf()} system calls.
@end deffn
@end deffn


@subsubheading See also



@table @asis

@item Format of Version 5 Babyl Files@footnote{https://quimby.gnus.org/notes/BABYL}

A specification of the Babyl format.

@item Reading Mail with Rmail@footnote{https://www.gnu.org/software/emacs/manual/html_node/emacs/Rmail.html}

The Rmail manual, with some information on Babyl semantics.
@end table


@node MMDF,,Babyl,Mailbox objects
@anchor{library/mailbox mailbox-mmdf}@anchor{2de4}@anchor{library/mailbox mmdf}@anchor{2de5}
@subsubsection @code{MMDF}


@geindex MMDF (class in mailbox)
@anchor{library/mailbox mailbox MMDF}@anchor{2de6}
@deffn {Class} mailbox.MMDF (path, factory=None, create=True)

A subclass of @ref{e00,,Mailbox} for mailboxes in MMDF format. Parameter @emph{factory}
is a callable object that accepts a file-like message representation (which
behaves as if opened in binary mode) and returns a custom representation. If
@emph{factory} is @code{None}, @ref{2de7,,MMDFMessage} is used as the default message
representation. If @emph{create} is @code{True}, the mailbox is created if it does not
exist.

MMDF is a single-file mailbox format invented for the Multichannel Memorandum
Distribution Facility, a mail transfer agent. Each message is in the same
form as an mbox message but is bracketed before and after by lines containing
four Control-A (@code{'\001'}) characters. As with the mbox format, the
beginning of each message is indicated by a line whose first five characters
are “From “, but additional occurrences of “From ” are not transformed to
“>From ” when storing messages because the extra message separator lines
prevent mistaking such occurrences for the starts of subsequent messages.

Some @ref{e00,,Mailbox} methods implemented by @ref{2de6,,MMDF} deserve special
remarks:

@geindex get_file() (mailbox.MMDF method)
@anchor{library/mailbox mailbox MMDF get_file}@anchor{2de8}
@deffn {Method} get_file (key)

Using the file after calling @code{flush()} or @code{close()} on the
@ref{2de6,,MMDF} instance may yield unpredictable results or raise an
exception.
@end deffn

@geindex lock() (mailbox.MMDF method)
@geindex unlock() (mailbox.MMDF method)
@anchor{library/mailbox mailbox MMDF lock}@anchor{2de9}
@deffn {Method} lock ()
@anchor{library/mailbox mailbox MMDF unlock}@anchor{2dea}
@deffnx {Method} unlock ()

Three locking mechanisms are used—dot locking and, if available, the
@code{flock()} and @code{lockf()} system calls.
@end deffn
@end deffn


@subsubheading See also



@table @asis

@item mmdf man page from tin@footnote{http://www.tin.org/bin/man.cgi?section=5&topic=mmdf}

A specification of MMDF format from the documentation of tin, a newsreader.

@item MMDF@footnote{https://en.wikipedia.org/wiki/MMDF}

A Wikipedia article describing the Multichannel Memorandum Distribution
Facility.
@end table


@node Message objects,Exceptions<15>,Mailbox objects,mailbox — Manipulate mailboxes in various formats
@anchor{library/mailbox mailbox-message-objects}@anchor{2deb}@anchor{library/mailbox message-objects}@anchor{2dec}
@subsubsection @code{Message} objects


@geindex Message (class in mailbox)
@anchor{library/mailbox mailbox Message}@anchor{e04}
@deffn {Class} mailbox.Message (message=None)

A subclass of the @ref{6d,,email.message} module’s
@ref{7e3,,Message}. Subclasses of @ref{e04,,mailbox.Message} add
mailbox-format-specific state and behavior.

If @emph{message} is omitted, the new instance is created in a default, empty state.
If @emph{message} is an @ref{7e3,,email.message.Message} instance, its contents are
copied; furthermore, any format-specific information is converted insofar as
possible if @emph{message} is a @ref{e04,,Message} instance. If @emph{message} is a string,
a byte string,
or a file, it should contain an 
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html}-compliant message, which is read
and parsed.  Files should be open in binary mode, but text mode files
are accepted for backward compatibility.

The format-specific state and behaviors offered by subclasses vary, but in
general it is only the properties that are not specific to a particular
mailbox that are supported (although presumably the properties are specific
to a particular mailbox format). For example, file offsets for single-file
mailbox formats and file names for directory-based mailbox formats are not
retained, because they are only applicable to the original mailbox. But state
such as whether a message has been read by the user or marked as important is
retained, because it applies to the message itself.

There is no requirement that @ref{e04,,Message} instances be used to represent
messages retrieved using @ref{e00,,Mailbox} instances. In some situations, the
time and memory required to generate @ref{e04,,Message} representations might
not be acceptable. For such situations, @ref{e00,,Mailbox} instances also
offer string and file-like representations, and a custom message factory may
be specified when a @ref{e00,,Mailbox} instance is initialized.
@end deffn

@menu
* MaildirMessage:: 
* mboxMessage:: 
* MHMessage:: 
* BabylMessage:: 
* MMDFMessage:: 

@end menu

@node MaildirMessage,mboxMessage,,Message objects
@anchor{library/mailbox mailbox-maildirmessage}@anchor{2ded}@anchor{library/mailbox maildirmessage}@anchor{2dee}
@subsubsection @code{MaildirMessage}


@geindex MaildirMessage (class in mailbox)
@anchor{library/mailbox mailbox MaildirMessage}@anchor{2db4}
@deffn {Class} mailbox.MaildirMessage (message=None)

A message with Maildir-specific behaviors. Parameter @emph{message} has the same
meaning as with the @ref{e04,,Message} constructor.

Typically, a mail user agent application moves all of the messages in the
@code{new} subdirectory to the @code{cur} subdirectory after the first time
the user opens and closes the mailbox, recording that the messages are old
whether or not they’ve actually been read. Each message in @code{cur} has an
“info” section added to its file name to store information about its state.
(Some mail readers may also add an “info” section to messages in
@code{new}.)  The “info” section may take one of two forms: it may contain
“2,” followed by a list of standardized flags (e.g., “2,FR”) or it may
contain “1,” followed by so-called experimental information. Standard flags
for Maildir messages are as follows:


@multitable {xxxxxxxx} {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Flag

@tab

Meaning

@tab

Explanation

@item

D

@tab

Draft

@tab

Under composition

@item

F

@tab

Flagged

@tab

Marked as important

@item

P

@tab

Passed

@tab

Forwarded, resent, or bounced

@item

R

@tab

Replied

@tab

Replied to

@item

S

@tab

Seen

@tab

Read

@item

T

@tab

Trashed

@tab

Marked for subsequent deletion

@end multitable


@ref{2db4,,MaildirMessage} instances offer the following methods:

@geindex get_subdir() (mailbox.MaildirMessage method)
@anchor{library/mailbox mailbox MaildirMessage get_subdir}@anchor{2def}
@deffn {Method} get_subdir ()

Return either “new” (if the message should be stored in the @code{new}
subdirectory) or “cur” (if the message should be stored in the @code{cur}
subdirectory).

@cartouche
@quotation Note 
A message is typically moved from @code{new} to @code{cur} after its
mailbox has been accessed, whether or not the message is has been
read. A message @code{msg} has been read if @code{"S" in msg.get_flags()} is
@code{True}.
@end quotation
@end cartouche
@end deffn

@geindex set_subdir() (mailbox.MaildirMessage method)
@anchor{library/mailbox mailbox MaildirMessage set_subdir}@anchor{2df0}
@deffn {Method} set_subdir (subdir)

Set the subdirectory the message should be stored in. Parameter @emph{subdir}
must be either “new” or “cur”.
@end deffn

@geindex get_flags() (mailbox.MaildirMessage method)
@anchor{library/mailbox mailbox MaildirMessage get_flags}@anchor{2df1}
@deffn {Method} get_flags ()

Return a string specifying the flags that are currently set. If the
message complies with the standard Maildir format, the result is the
concatenation in alphabetical order of zero or one occurrence of each of
@code{'D'}, @code{'F'}, @code{'P'}, @code{'R'}, @code{'S'}, and @code{'T'}. The empty string
is returned if no flags are set or if “info” contains experimental
semantics.
@end deffn

@geindex set_flags() (mailbox.MaildirMessage method)
@anchor{library/mailbox mailbox MaildirMessage set_flags}@anchor{2df2}
@deffn {Method} set_flags (flags)

Set the flags specified by @emph{flags} and unset all others.
@end deffn

@geindex add_flag() (mailbox.MaildirMessage method)
@anchor{library/mailbox mailbox MaildirMessage add_flag}@anchor{2df3}
@deffn {Method} add_flag (flag)

Set the flag(s) specified by @emph{flag} without changing other flags. To add
more than one flag at a time, @emph{flag} may be a string of more than one
character. The current “info” is overwritten whether or not it contains
experimental information rather than flags.
@end deffn

@geindex remove_flag() (mailbox.MaildirMessage method)
@anchor{library/mailbox mailbox MaildirMessage remove_flag}@anchor{2df4}
@deffn {Method} remove_flag (flag)

Unset the flag(s) specified by @emph{flag} without changing other flags. To
remove more than one flag at a time, @emph{flag} maybe a string of more than
one character.  If “info” contains experimental information rather than
flags, the current “info” is not modified.
@end deffn

@geindex get_date() (mailbox.MaildirMessage method)
@anchor{library/mailbox mailbox MaildirMessage get_date}@anchor{2df5}
@deffn {Method} get_date ()

Return the delivery date of the message as a floating-point number
representing seconds since the epoch.
@end deffn

@geindex set_date() (mailbox.MaildirMessage method)
@anchor{library/mailbox mailbox MaildirMessage set_date}@anchor{2df6}
@deffn {Method} set_date (date)

Set the delivery date of the message to @emph{date}, a floating-point number
representing seconds since the epoch.
@end deffn

@geindex get_info() (mailbox.MaildirMessage method)
@anchor{library/mailbox mailbox MaildirMessage get_info}@anchor{2df7}
@deffn {Method} get_info ()

Return a string containing the “info” for a message. This is useful for
accessing and modifying “info” that is experimental (i.e., not a list of
flags).
@end deffn

@geindex set_info() (mailbox.MaildirMessage method)
@anchor{library/mailbox mailbox MaildirMessage set_info}@anchor{2df8}
@deffn {Method} set_info (info)

Set “info” to @emph{info}, which should be a string.
@end deffn
@end deffn

When a @ref{2db4,,MaildirMessage} instance is created based upon an
@ref{2c5b,,mboxMessage} or @ref{2de7,,MMDFMessage} instance, the @code{Status}
and @code{X-Status} headers are omitted and the following conversions
take place:


@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2c5b,,mboxMessage} or @ref{2de7,,MMDFMessage}
state

@item

“cur” subdirectory

@tab

O flag

@item

F flag

@tab

F flag

@item

R flag

@tab

A flag

@item

S flag

@tab

R flag

@item

T flag

@tab

D flag

@end multitable


When a @ref{2db4,,MaildirMessage} instance is created based upon an
@ref{2dcc,,MHMessage} instance, the following conversions take place:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2dcc,,MHMessage} state

@item

“cur” subdirectory

@tab

“unseen” sequence

@item

“cur” subdirectory and S flag

@tab

no “unseen” sequence

@item

F flag

@tab

“flagged” sequence

@item

R flag

@tab

“replied” sequence

@end multitable


When a @ref{2db4,,MaildirMessage} instance is created based upon a
@ref{2ddf,,BabylMessage} instance, the following conversions take place:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2ddf,,BabylMessage} state

@item

“cur” subdirectory

@tab

“unseen” label

@item

“cur” subdirectory and S flag

@tab

no “unseen” label

@item

P flag

@tab

“forwarded” or “resent” label

@item

R flag

@tab

“answered” label

@item

T flag

@tab

“deleted” label

@end multitable


@node mboxMessage,MHMessage,MaildirMessage,Message objects
@anchor{library/mailbox mailbox-mboxmessage}@anchor{2df9}@anchor{library/mailbox mboxmessage}@anchor{2dfa}
@subsubsection @code{mboxMessage}


@geindex mboxMessage (class in mailbox)
@anchor{library/mailbox mailbox mboxMessage}@anchor{2c5b}
@deffn {Class} mailbox.mboxMessage (message=None)

A message with mbox-specific behaviors. Parameter @emph{message} has the same meaning
as with the @ref{e04,,Message} constructor.

Messages in an mbox mailbox are stored together in a single file. The
sender’s envelope address and the time of delivery are typically stored in a
line beginning with “From ” that is used to indicate the start of a message,
though there is considerable variation in the exact format of this data among
mbox implementations. Flags that indicate the state of the message, such as
whether it has been read or marked as important, are typically stored in
@code{Status} and @code{X-Status} headers.

Conventional flags for mbox messages are as follows:


@multitable {xxxxxxxx} {xxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Flag

@tab

Meaning

@tab

Explanation

@item

R

@tab

Read

@tab

Read

@item

O

@tab

Old

@tab

Previously detected by MUA

@item

D

@tab

Deleted

@tab

Marked for subsequent deletion

@item

F

@tab

Flagged

@tab

Marked as important

@item

A

@tab

Answered

@tab

Replied to

@end multitable


The “R” and “O” flags are stored in the @code{Status} header, and the
“D”, “F”, and “A” flags are stored in the @code{X-Status} header. The
flags and headers typically appear in the order mentioned.

@ref{2c5b,,mboxMessage} instances offer the following methods:

@geindex get_from() (mailbox.mboxMessage method)
@anchor{library/mailbox mailbox mboxMessage get_from}@anchor{2dfb}
@deffn {Method} get_from ()

Return a string representing the “From ” line that marks the start of the
message in an mbox mailbox. The leading “From ” and the trailing newline
are excluded.
@end deffn

@geindex set_from() (mailbox.mboxMessage method)
@anchor{library/mailbox mailbox mboxMessage set_from}@anchor{2dfc}
@deffn {Method} set_from (from_, time_=None)

Set the “From ” line to @emph{from_}, which should be specified without a
leading “From ” or trailing newline. For convenience, @emph{time_} may be
specified and will be formatted appropriately and appended to @emph{from_}. If
@emph{time_} is specified, it should be a @ref{834,,time.struct_time} instance, a
tuple suitable for passing to @ref{d8b,,time.strftime()}, or @code{True} (to use
@ref{d65,,time.gmtime()}).
@end deffn

@geindex get_flags() (mailbox.mboxMessage method)
@anchor{library/mailbox mailbox mboxMessage get_flags}@anchor{2dfd}
@deffn {Method} get_flags ()

Return a string specifying the flags that are currently set. If the
message complies with the conventional format, the result is the
concatenation in the following order of zero or one occurrence of each of
@code{'R'}, @code{'O'}, @code{'D'}, @code{'F'}, and @code{'A'}.
@end deffn

@geindex set_flags() (mailbox.mboxMessage method)
@anchor{library/mailbox mailbox mboxMessage set_flags}@anchor{2dfe}
@deffn {Method} set_flags (flags)

Set the flags specified by @emph{flags} and unset all others. Parameter @emph{flags}
should be the concatenation in any order of zero or more occurrences of
each of @code{'R'}, @code{'O'}, @code{'D'}, @code{'F'}, and @code{'A'}.
@end deffn

@geindex add_flag() (mailbox.mboxMessage method)
@anchor{library/mailbox mailbox mboxMessage add_flag}@anchor{2dff}
@deffn {Method} add_flag (flag)

Set the flag(s) specified by @emph{flag} without changing other flags. To add
more than one flag at a time, @emph{flag} may be a string of more than one
character.
@end deffn

@geindex remove_flag() (mailbox.mboxMessage method)
@anchor{library/mailbox mailbox mboxMessage remove_flag}@anchor{2e00}
@deffn {Method} remove_flag (flag)

Unset the flag(s) specified by @emph{flag} without changing other flags. To
remove more than one flag at a time, @emph{flag} maybe a string of more than
one character.
@end deffn
@end deffn

When an @ref{2c5b,,mboxMessage} instance is created based upon a
@ref{2db4,,MaildirMessage} instance, a “From ” line is generated based upon the
@ref{2db4,,MaildirMessage} instance’s delivery date, and the following conversions
take place:


@multitable {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2db4,,MaildirMessage} state

@item

R flag

@tab

S flag

@item

O flag

@tab

“cur” subdirectory

@item

D flag

@tab

T flag

@item

F flag

@tab

F flag

@item

A flag

@tab

R flag

@end multitable


When an @ref{2c5b,,mboxMessage} instance is created based upon an
@ref{2dcc,,MHMessage} instance, the following conversions take place:


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2dcc,,MHMessage} state

@item

R flag and O flag

@tab

no “unseen” sequence

@item

O flag

@tab

“unseen” sequence

@item

F flag

@tab

“flagged” sequence

@item

A flag

@tab

“replied” sequence

@end multitable


When an @ref{2c5b,,mboxMessage} instance is created based upon a
@ref{2ddf,,BabylMessage} instance, the following conversions take place:


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2ddf,,BabylMessage} state

@item

R flag and O flag

@tab

no “unseen” label

@item

O flag

@tab

“unseen” label

@item

D flag

@tab

“deleted” label

@item

A flag

@tab

“answered” label

@end multitable


When a @ref{e04,,Message} instance is created based upon an @ref{2de7,,MMDFMessage}
instance, the “From ” line is copied and all flags directly correspond:


@multitable {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2de7,,MMDFMessage} state

@item

R flag

@tab

R flag

@item

O flag

@tab

O flag

@item

D flag

@tab

D flag

@item

F flag

@tab

F flag

@item

A flag

@tab

A flag

@end multitable


@node MHMessage,BabylMessage,mboxMessage,Message objects
@anchor{library/mailbox mailbox-mhmessage}@anchor{2e01}@anchor{library/mailbox mhmessage}@anchor{2e02}
@subsubsection @code{MHMessage}


@geindex MHMessage (class in mailbox)
@anchor{library/mailbox mailbox MHMessage}@anchor{2dcc}
@deffn {Class} mailbox.MHMessage (message=None)

A message with MH-specific behaviors. Parameter @emph{message} has the same meaning
as with the @ref{e04,,Message} constructor.

MH messages do not support marks or flags in the traditional sense, but they
do support sequences, which are logical groupings of arbitrary messages. Some
mail reading programs (although not the standard @code{mh} and
@code{nmh}) use sequences in much the same way flags are used with other
formats, as follows:


@multitable {xxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Sequence

@tab

Explanation

@item

unseen

@tab

Not read, but previously detected by MUA

@item

replied

@tab

Replied to

@item

flagged

@tab

Marked as important

@end multitable


@ref{2dcc,,MHMessage} instances offer the following methods:

@geindex get_sequences() (mailbox.MHMessage method)
@anchor{library/mailbox mailbox MHMessage get_sequences}@anchor{2e03}
@deffn {Method} get_sequences ()

Return a list of the names of sequences that include this message.
@end deffn

@geindex set_sequences() (mailbox.MHMessage method)
@anchor{library/mailbox mailbox MHMessage set_sequences}@anchor{2e04}
@deffn {Method} set_sequences (sequences)

Set the list of sequences that include this message.
@end deffn

@geindex add_sequence() (mailbox.MHMessage method)
@anchor{library/mailbox mailbox MHMessage add_sequence}@anchor{2e05}
@deffn {Method} add_sequence (sequence)

Add @emph{sequence} to the list of sequences that include this message.
@end deffn

@geindex remove_sequence() (mailbox.MHMessage method)
@anchor{library/mailbox mailbox MHMessage remove_sequence}@anchor{2e06}
@deffn {Method} remove_sequence (sequence)

Remove @emph{sequence} from the list of sequences that include this message.
@end deffn
@end deffn

When an @ref{2dcc,,MHMessage} instance is created based upon a
@ref{2db4,,MaildirMessage} instance, the following conversions take place:


@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2db4,,MaildirMessage} state

@item

“unseen” sequence

@tab

no S flag

@item

“replied” sequence

@tab

R flag

@item

“flagged” sequence

@tab

F flag

@end multitable


When an @ref{2dcc,,MHMessage} instance is created based upon an
@ref{2c5b,,mboxMessage} or @ref{2de7,,MMDFMessage} instance, the @code{Status}
and @code{X-Status} headers are omitted and the following conversions
take place:


@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2c5b,,mboxMessage} or @ref{2de7,,MMDFMessage}
state

@item

“unseen” sequence

@tab

no R flag

@item

“replied” sequence

@tab

A flag

@item

“flagged” sequence

@tab

F flag

@end multitable


When an @ref{2dcc,,MHMessage} instance is created based upon a
@ref{2ddf,,BabylMessage} instance, the following conversions take place:


@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2ddf,,BabylMessage} state

@item

“unseen” sequence

@tab

“unseen” label

@item

“replied” sequence

@tab

“answered” label

@end multitable


@node BabylMessage,MMDFMessage,MHMessage,Message objects
@anchor{library/mailbox babylmessage}@anchor{2e07}@anchor{library/mailbox mailbox-babylmessage}@anchor{2e08}
@subsubsection @code{BabylMessage}


@geindex BabylMessage (class in mailbox)
@anchor{library/mailbox mailbox BabylMessage}@anchor{2ddf}
@deffn {Class} mailbox.BabylMessage (message=None)

A message with Babyl-specific behaviors. Parameter @emph{message} has the same
meaning as with the @ref{e04,,Message} constructor.

Certain message labels, called @emph{attributes}, are defined by convention
to have special meanings. The attributes are as follows:


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Label

@tab

Explanation

@item

unseen

@tab

Not read, but previously detected by MUA

@item

deleted

@tab

Marked for subsequent deletion

@item

filed

@tab

Copied to another file or mailbox

@item

answered

@tab

Replied to

@item

forwarded

@tab

Forwarded

@item

edited

@tab

Modified by the user

@item

resent

@tab

Resent

@end multitable


By default, Rmail displays only visible headers. The @ref{2ddf,,BabylMessage}
class, though, uses the original headers because they are more
complete. Visible headers may be accessed explicitly if desired.

@ref{2ddf,,BabylMessage} instances offer the following methods:

@geindex get_labels() (mailbox.BabylMessage method)
@anchor{library/mailbox mailbox BabylMessage get_labels}@anchor{2e09}
@deffn {Method} get_labels ()

Return a list of labels on the message.
@end deffn

@geindex set_labels() (mailbox.BabylMessage method)
@anchor{library/mailbox mailbox BabylMessage set_labels}@anchor{2e0a}
@deffn {Method} set_labels (labels)

Set the list of labels on the message to @emph{labels}.
@end deffn

@geindex add_label() (mailbox.BabylMessage method)
@anchor{library/mailbox mailbox BabylMessage add_label}@anchor{2e0b}
@deffn {Method} add_label (label)

Add @emph{label} to the list of labels on the message.
@end deffn

@geindex remove_label() (mailbox.BabylMessage method)
@anchor{library/mailbox mailbox BabylMessage remove_label}@anchor{2e0c}
@deffn {Method} remove_label (label)

Remove @emph{label} from the list of labels on the message.
@end deffn

@geindex get_visible() (mailbox.BabylMessage method)
@anchor{library/mailbox mailbox BabylMessage get_visible}@anchor{2e0d}
@deffn {Method} get_visible ()

Return an @ref{e04,,Message} instance whose headers are the message’s
visible headers and whose body is empty.
@end deffn

@geindex set_visible() (mailbox.BabylMessage method)
@anchor{library/mailbox mailbox BabylMessage set_visible}@anchor{2e0e}
@deffn {Method} set_visible (visible)

Set the message’s visible headers to be the same as the headers in
@emph{message}.  Parameter @emph{visible} should be a @ref{e04,,Message} instance, an
@ref{7e3,,email.message.Message} instance, a string, or a file-like object
(which should be open in text mode).
@end deffn

@geindex update_visible() (mailbox.BabylMessage method)
@anchor{library/mailbox mailbox BabylMessage update_visible}@anchor{2e0f}
@deffn {Method} update_visible ()

When a @ref{2ddf,,BabylMessage} instance’s original headers are modified, the
visible headers are not automatically modified to correspond. This method
updates the visible headers as follows: each visible header with a
corresponding original header is set to the value of the original header,
each visible header without a corresponding original header is removed,
and any of @code{Date}, @code{From}, @code{Reply-To},
@code{To}, @code{CC}, and @code{Subject} that are
present in the original headers but not the visible headers are added to
the visible headers.
@end deffn
@end deffn

When a @ref{2ddf,,BabylMessage} instance is created based upon a
@ref{2db4,,MaildirMessage} instance, the following conversions take place:


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2db4,,MaildirMessage} state

@item

“unseen” label

@tab

no S flag

@item

“deleted” label

@tab

T flag

@item

“answered” label

@tab

R flag

@item

“forwarded” label

@tab

P flag

@end multitable


When a @ref{2ddf,,BabylMessage} instance is created based upon an
@ref{2c5b,,mboxMessage} or @ref{2de7,,MMDFMessage} instance, the @code{Status}
and @code{X-Status} headers are omitted and the following conversions
take place:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2c5b,,mboxMessage} or @ref{2de7,,MMDFMessage}
state

@item

“unseen” label

@tab

no R flag

@item

“deleted” label

@tab

D flag

@item

“answered” label

@tab

A flag

@end multitable


When a @ref{2ddf,,BabylMessage} instance is created based upon an
@ref{2dcc,,MHMessage} instance, the following conversions take place:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2dcc,,MHMessage} state

@item

“unseen” label

@tab

“unseen” sequence

@item

“answered” label

@tab

“replied” sequence

@end multitable


@node MMDFMessage,,BabylMessage,Message objects
@anchor{library/mailbox mailbox-mmdfmessage}@anchor{2e10}@anchor{library/mailbox mmdfmessage}@anchor{2e11}
@subsubsection @code{MMDFMessage}


@geindex MMDFMessage (class in mailbox)
@anchor{library/mailbox mailbox MMDFMessage}@anchor{2de7}
@deffn {Class} mailbox.MMDFMessage (message=None)

A message with MMDF-specific behaviors. Parameter @emph{message} has the same meaning
as with the @ref{e04,,Message} constructor.

As with message in an mbox mailbox, MMDF messages are stored with the
sender’s address and the delivery date in an initial line beginning with
“From “.  Likewise, flags that indicate the state of the message are
typically stored in @code{Status} and @code{X-Status} headers.

Conventional flags for MMDF messages are identical to those of mbox message
and are as follows:


@multitable {xxxxxxxx} {xxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Flag

@tab

Meaning

@tab

Explanation

@item

R

@tab

Read

@tab

Read

@item

O

@tab

Old

@tab

Previously detected by MUA

@item

D

@tab

Deleted

@tab

Marked for subsequent deletion

@item

F

@tab

Flagged

@tab

Marked as important

@item

A

@tab

Answered

@tab

Replied to

@end multitable


The “R” and “O” flags are stored in the @code{Status} header, and the
“D”, “F”, and “A” flags are stored in the @code{X-Status} header. The
flags and headers typically appear in the order mentioned.

@ref{2de7,,MMDFMessage} instances offer the following methods, which are
identical to those offered by @ref{2c5b,,mboxMessage}:

@geindex get_from() (mailbox.MMDFMessage method)
@anchor{library/mailbox mailbox MMDFMessage get_from}@anchor{2e12}
@deffn {Method} get_from ()

Return a string representing the “From ” line that marks the start of the
message in an mbox mailbox. The leading “From ” and the trailing newline
are excluded.
@end deffn

@geindex set_from() (mailbox.MMDFMessage method)
@anchor{library/mailbox mailbox MMDFMessage set_from}@anchor{2e13}
@deffn {Method} set_from (from_, time_=None)

Set the “From ” line to @emph{from_}, which should be specified without a
leading “From ” or trailing newline. For convenience, @emph{time_} may be
specified and will be formatted appropriately and appended to @emph{from_}. If
@emph{time_} is specified, it should be a @ref{834,,time.struct_time} instance, a
tuple suitable for passing to @ref{d8b,,time.strftime()}, or @code{True} (to use
@ref{d65,,time.gmtime()}).
@end deffn

@geindex get_flags() (mailbox.MMDFMessage method)
@anchor{library/mailbox mailbox MMDFMessage get_flags}@anchor{2e14}
@deffn {Method} get_flags ()

Return a string specifying the flags that are currently set. If the
message complies with the conventional format, the result is the
concatenation in the following order of zero or one occurrence of each of
@code{'R'}, @code{'O'}, @code{'D'}, @code{'F'}, and @code{'A'}.
@end deffn

@geindex set_flags() (mailbox.MMDFMessage method)
@anchor{library/mailbox mailbox MMDFMessage set_flags}@anchor{2e15}
@deffn {Method} set_flags (flags)

Set the flags specified by @emph{flags} and unset all others. Parameter @emph{flags}
should be the concatenation in any order of zero or more occurrences of
each of @code{'R'}, @code{'O'}, @code{'D'}, @code{'F'}, and @code{'A'}.
@end deffn

@geindex add_flag() (mailbox.MMDFMessage method)
@anchor{library/mailbox mailbox MMDFMessage add_flag}@anchor{2e16}
@deffn {Method} add_flag (flag)

Set the flag(s) specified by @emph{flag} without changing other flags. To add
more than one flag at a time, @emph{flag} may be a string of more than one
character.
@end deffn

@geindex remove_flag() (mailbox.MMDFMessage method)
@anchor{library/mailbox mailbox MMDFMessage remove_flag}@anchor{2e17}
@deffn {Method} remove_flag (flag)

Unset the flag(s) specified by @emph{flag} without changing other flags. To
remove more than one flag at a time, @emph{flag} maybe a string of more than
one character.
@end deffn
@end deffn

When an @ref{2de7,,MMDFMessage} instance is created based upon a
@ref{2db4,,MaildirMessage} instance, a “From ” line is generated based upon the
@ref{2db4,,MaildirMessage} instance’s delivery date, and the following conversions
take place:


@multitable {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2db4,,MaildirMessage} state

@item

R flag

@tab

S flag

@item

O flag

@tab

“cur” subdirectory

@item

D flag

@tab

T flag

@item

F flag

@tab

F flag

@item

A flag

@tab

R flag

@end multitable


When an @ref{2de7,,MMDFMessage} instance is created based upon an
@ref{2dcc,,MHMessage} instance, the following conversions take place:


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2dcc,,MHMessage} state

@item

R flag and O flag

@tab

no “unseen” sequence

@item

O flag

@tab

“unseen” sequence

@item

F flag

@tab

“flagged” sequence

@item

A flag

@tab

“replied” sequence

@end multitable


When an @ref{2de7,,MMDFMessage} instance is created based upon a
@ref{2ddf,,BabylMessage} instance, the following conversions take place:


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2ddf,,BabylMessage} state

@item

R flag and O flag

@tab

no “unseen” label

@item

O flag

@tab

“unseen” label

@item

D flag

@tab

“deleted” label

@item

A flag

@tab

“answered” label

@end multitable


When an @ref{2de7,,MMDFMessage} instance is created based upon an
@ref{2c5b,,mboxMessage} instance, the “From ” line is copied and all flags directly
correspond:


@multitable {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Resulting state

@tab

@ref{2c5b,,mboxMessage} state

@item

R flag

@tab

R flag

@item

O flag

@tab

O flag

@item

D flag

@tab

D flag

@item

F flag

@tab

F flag

@item

A flag

@tab

A flag

@end multitable


@node Exceptions<15>,Examples<21>,Message objects,mailbox — Manipulate mailboxes in various formats
@anchor{library/mailbox exceptions}@anchor{2e18}
@subsubsection Exceptions


The following exception classes are defined in the @ref{aa,,mailbox} module:

@geindex Error
@anchor{library/mailbox mailbox Error}@anchor{2e19}
@deffn {Exception} mailbox.Error

The based class for all other module-specific exceptions.
@end deffn

@geindex NoSuchMailboxError
@anchor{library/mailbox mailbox NoSuchMailboxError}@anchor{2db7}
@deffn {Exception} mailbox.NoSuchMailboxError

Raised when a mailbox is expected but is not found, such as when instantiating a
@ref{e00,,Mailbox} subclass with a path that does not exist (and with the @emph{create}
parameter set to @code{False}), or when opening a folder that does not exist.
@end deffn

@geindex NotEmptyError
@anchor{library/mailbox mailbox NotEmptyError}@anchor{2dba}
@deffn {Exception} mailbox.NotEmptyError

Raised when a mailbox is not empty but is expected to be, such as when deleting
a folder that contains messages.
@end deffn

@geindex ExternalClashError
@anchor{library/mailbox mailbox ExternalClashError}@anchor{2db0}
@deffn {Exception} mailbox.ExternalClashError

Raised when some mailbox-related condition beyond the control of the program
causes it to be unable to proceed, such as when failing to acquire a lock that
another program already holds a lock, or when a uniquely-generated file name
already exists.
@end deffn

@geindex FormatError
@anchor{library/mailbox mailbox FormatError}@anchor{2e1a}
@deffn {Exception} mailbox.FormatError

Raised when the data in a file cannot be parsed, such as when an @ref{2dcb,,MH}
instance attempts to read a corrupted @code{.mh_sequences} file.
@end deffn

@node Examples<21>,,Exceptions<15>,mailbox — Manipulate mailboxes in various formats
@anchor{library/mailbox examples}@anchor{2e1b}@anchor{library/mailbox mailbox-examples}@anchor{2e1c}
@subsubsection Examples


A simple example of printing the subjects of all messages in a mailbox that seem
interesting:

@example
import mailbox
for message in mailbox.mbox('~/mbox'):
    subject = message['subject']       # Could possibly be None.
    if subject and 'python' in subject.lower():
        print(subject)
@end example

To copy all mail from a Babyl mailbox to an MH mailbox, converting all of the
format-specific information that can be converted:

@example
import mailbox
destination = mailbox.MH('~/Mail')
destination.lock()
for message in mailbox.Babyl('~/RMAIL'):
    destination.add(mailbox.MHMessage(message))
destination.flush()
destination.unlock()
@end example

This example sorts mail from several mailing lists into different mailboxes,
being careful to avoid mail corruption due to concurrent modification by other
programs, mail loss due to interruption of the program, or premature termination
due to malformed messages in the mailbox:

@example
import mailbox
import email.errors

list_names = ('python-list', 'python-dev', 'python-bugs')

boxes = @{name: mailbox.mbox('~/email/%s' % name) for name in list_names@}
inbox = mailbox.Maildir('~/Maildir', factory=None)

for key in inbox.iterkeys():
    try:
        message = inbox[key]
    except email.errors.MessageParseError:
        continue                # The message is malformed. Just leave it.

    for name in list_names:
        list_id = message['list-id']
        if list_id and name in list_id:
            # Get mailbox to use
            box = boxes[name]

            # Write copy to disk before removing original.
            # If there's a crash, you might duplicate a message, but
            # that's better than losing a message completely.
            box.lock()
            box.add(message)
            box.flush()
            box.unlock()

            # Remove original message
            inbox.lock()
            inbox.discard(key)
            inbox.flush()
            inbox.unlock()
            break               # Found destination, so stop looking.

for box in boxes.itervalues():
    box.close()
@end example

@node mimetypes — Map filenames to MIME types,base64 — Base16 Base32 Base64 Base85 Data Encodings,mailbox — Manipulate mailboxes in various formats,Internet Data Handling
@anchor{library/mimetypes doc}@anchor{2e1d}@anchor{library/mimetypes mimetypes-map-filenames-to-mime-types}@anchor{2e1e}@anchor{library/mimetypes module-mimetypes}@anchor{ae}
@subsection @code{mimetypes} — Map filenames to MIME types


@geindex module; mimetypes

@strong{Source code:} Lib/mimetypes.py@footnote{https://github.com/python/cpython/tree/main/Lib/mimetypes.py}

@geindex MIME; content type


__________________________________________________________________


The @ref{ae,,mimetypes} module converts between a filename or URL and the MIME type
associated with the filename extension.  Conversions are provided from filename
to MIME type and from MIME type to filename extension; encodings are not
supported for the latter conversion.

The module provides one class and a number of convenience functions. The
functions are the normal interface to this module, but some applications may be
interested in the class as well.

The functions described below provide the primary interface for this module.  If
the module has not been initialized, they will call @ref{123e,,init()} if they rely on
the information @ref{123e,,init()} sets up.

@geindex guess_type() (in module mimetypes)
@anchor{library/mimetypes mimetypes guess_type}@anchor{123d}
@deffn {Function} mimetypes.guess_type (url, strict=True)

@geindex MIME; headers

Guess the type of a file based on its filename, path or URL, given by @emph{url}.
URL can be a string or a @ref{63d,,path-like object}.

The return value is a tuple @code{(type, encoding)} where @emph{type} is @code{None} if the
type can’t be guessed (missing or unknown suffix) or a string of the form
@code{'type/subtype'}, usable for a MIME @code{content-type} header.

@emph{encoding} is @code{None} for no encoding or the name of the program used to encode
(e.g. @code{compress} or @code{gzip}). The encoding is suitable for use
as a @code{Content-Encoding} header, @strong{not} as a
@code{Content-Transfer-Encoding} header. The mappings are table driven.
Encoding suffixes are case sensitive; type suffixes are first tried case
sensitively, then case insensitively.

The optional @emph{strict} argument is a flag specifying whether the list of known MIME types
is limited to only the official types registered with IANA@footnote{https://www.iana.org/assignments/media-types/media-types.xhtml}.
When @emph{strict} is @code{True} (the default), only the IANA types are supported; when
@emph{strict} is @code{False}, some additional non-standard but commonly used MIME types
are also recognized.


Changed in version 3.8: Added support for url being a @ref{63d,,path-like object}.

@end deffn

@geindex guess_all_extensions() (in module mimetypes)
@anchor{library/mimetypes mimetypes guess_all_extensions}@anchor{10fa}
@deffn {Function} mimetypes.guess_all_extensions (type, strict=True)

Guess the extensions for a file based on its MIME type, given by @emph{type}. The
return value is a list of strings giving all possible filename extensions,
including the leading dot (@code{'.'}).  The extensions are not guaranteed to have
been associated with any particular data stream, but would be mapped to the MIME
type @emph{type} by @ref{123d,,guess_type()}.

The optional @emph{strict} argument has the same meaning as with the @ref{123d,,guess_type()} function.
@end deffn

@geindex guess_extension() (in module mimetypes)
@anchor{library/mimetypes mimetypes guess_extension}@anchor{2e1f}
@deffn {Function} mimetypes.guess_extension (type, strict=True)

Guess the extension for a file based on its MIME type, given by @emph{type}. The
return value is a string giving a filename extension, including the leading dot
(@code{'.'}).  The extension is not guaranteed to have been associated with any
particular data stream, but would be mapped to the MIME type @emph{type} by
@ref{123d,,guess_type()}.  If no extension can be guessed for @emph{type}, @code{None} is
returned.

The optional @emph{strict} argument has the same meaning as with the @ref{123d,,guess_type()} function.
@end deffn

Some additional functions and data items are available for controlling the
behavior of the module.

@geindex init() (in module mimetypes)
@anchor{library/mimetypes mimetypes init}@anchor{123e}
@deffn {Function} mimetypes.init (files=None)

Initialize the internal data structures.  If given, @emph{files} must be a sequence
of file names which should be used to augment the default type map.  If omitted,
the file names to use are taken from @ref{2e20,,knownfiles}; on Windows, the
current registry settings are loaded.  Each file named in @emph{files} or
@ref{2e20,,knownfiles} takes precedence over those named before it.  Calling
@ref{123e,,init()} repeatedly is allowed.

Specifying an empty list for @emph{files} will prevent the system defaults from
being applied: only the well-known values will be present from a built-in list.

If @emph{files} is @code{None} the internal data structure is completely rebuilt to its
initial default value. This is a stable operation and will produce the same results
when called multiple times.


Changed in version 3.2: Previously, Windows registry settings were ignored.

@end deffn

@geindex read_mime_types() (in module mimetypes)
@anchor{library/mimetypes mimetypes read_mime_types}@anchor{11c5}
@deffn {Function} mimetypes.read_mime_types (filename)

Load the type map given in the file @emph{filename}, if it exists.  The type map is
returned as a dictionary mapping filename extensions, including the leading dot
(@code{'.'}), to strings of the form @code{'type/subtype'}.  If the file @emph{filename}
does not exist or cannot be read, @code{None} is returned.
@end deffn

@geindex add_type() (in module mimetypes)
@anchor{library/mimetypes mimetypes add_type}@anchor{2e21}
@deffn {Function} mimetypes.add_type (type, ext, strict=True)

Add a mapping from the MIME type @emph{type} to the extension @emph{ext}. When the
extension is already known, the new type will replace the old one. When the type
is already known the extension will be added to the list of known extensions.

When @emph{strict} is @code{True} (the default), the mapping will be added to the
official MIME types, otherwise to the non-standard ones.
@end deffn

@geindex inited (in module mimetypes)
@anchor{library/mimetypes mimetypes inited}@anchor{2e22}
@deffn {Data} mimetypes.inited

Flag indicating whether or not the global data structures have been initialized.
This is set to @code{True} by @ref{123e,,init()}.
@end deffn

@geindex knownfiles (in module mimetypes)
@anchor{library/mimetypes mimetypes knownfiles}@anchor{2e20}
@deffn {Data} mimetypes.knownfiles

@geindex file; mime.types

List of type map file names commonly installed.  These files are typically named
@code{mime.types} and are installed in different locations by different
packages.
@end deffn

@geindex suffix_map (in module mimetypes)
@anchor{library/mimetypes mimetypes suffix_map}@anchor{2e23}
@deffn {Data} mimetypes.suffix_map

Dictionary mapping suffixes to suffixes.  This is used to allow recognition of
encoded files for which the encoding and the type are indicated by the same
extension.  For example, the @code{.tgz} extension is mapped to @code{.tar.gz}
to allow the encoding and type to be recognized separately.
@end deffn

@geindex encodings_map (in module mimetypes)
@anchor{library/mimetypes mimetypes encodings_map}@anchor{2e24}
@deffn {Data} mimetypes.encodings_map

Dictionary mapping filename extensions to encoding types.
@end deffn

@geindex types_map (in module mimetypes)
@anchor{library/mimetypes mimetypes types_map}@anchor{2e25}
@deffn {Data} mimetypes.types_map

Dictionary mapping filename extensions to MIME types.
@end deffn

@geindex common_types (in module mimetypes)
@anchor{library/mimetypes mimetypes common_types}@anchor{2e26}
@deffn {Data} mimetypes.common_types

Dictionary mapping filename extensions to non-standard, but commonly found MIME
types.
@end deffn

An example usage of the module:

@example
>>> import mimetypes
>>> mimetypes.init()
>>> mimetypes.knownfiles
['/etc/mime.types', '/etc/httpd/mime.types', ... ]
>>> mimetypes.suffix_map['.tgz']
'.tar.gz'
>>> mimetypes.encodings_map['.gz']
'gzip'
>>> mimetypes.types_map['.tgz']
'application/x-tar-gz'
@end example

@menu
* MimeTypes Objects:: 

@end menu

@node MimeTypes Objects,,,mimetypes — Map filenames to MIME types
@anchor{library/mimetypes id1}@anchor{2e27}@anchor{library/mimetypes mimetypes-objects}@anchor{2e28}
@subsubsection MimeTypes Objects


The @ref{2e29,,MimeTypes} class may be useful for applications which may want more
than one MIME-type database; it provides an interface similar to the one of the
@ref{ae,,mimetypes} module.

@geindex MimeTypes (class in mimetypes)
@anchor{library/mimetypes mimetypes MimeTypes}@anchor{2e29}
@deffn {Class} mimetypes.MimeTypes (filenames=(), strict=True)

This class represents a MIME-types database.  By default, it provides access to
the same database as the rest of this module. The initial database is a copy of
that provided by the module, and may be extended by loading additional
@code{mime.types}-style files into the database using the @ref{2e2a,,read()} or
@ref{2e2b,,readfp()} methods.  The mapping dictionaries may also be cleared before
loading additional data if the default data is not desired.

The optional @emph{filenames} parameter can be used to cause additional files to be
loaded “on top” of the default database.

@geindex suffix_map (mimetypes.MimeTypes attribute)
@anchor{library/mimetypes mimetypes MimeTypes suffix_map}@anchor{2e2c}
@deffn {Attribute} suffix_map

Dictionary mapping suffixes to suffixes.  This is used to allow recognition of
encoded files for which the encoding and the type are indicated by the same
extension.  For example, the @code{.tgz} extension is mapped to @code{.tar.gz}
to allow the encoding and type to be recognized separately.  This is initially a
copy of the global @ref{2e23,,suffix_map} defined in the module.
@end deffn

@geindex encodings_map (mimetypes.MimeTypes attribute)
@anchor{library/mimetypes mimetypes MimeTypes encodings_map}@anchor{2e2d}
@deffn {Attribute} encodings_map

Dictionary mapping filename extensions to encoding types.  This is initially a
copy of the global @ref{2e24,,encodings_map} defined in the module.
@end deffn

@geindex types_map (mimetypes.MimeTypes attribute)
@anchor{library/mimetypes mimetypes MimeTypes types_map}@anchor{2e2e}
@deffn {Attribute} types_map

Tuple containing two dictionaries, mapping filename extensions to MIME types:
the first dictionary is for the non-standards types and the second one is for
the standard types. They are initialized by @ref{2e26,,common_types} and
@ref{2e25,,types_map}.
@end deffn

@geindex types_map_inv (mimetypes.MimeTypes attribute)
@anchor{library/mimetypes mimetypes MimeTypes types_map_inv}@anchor{2e2f}
@deffn {Attribute} types_map_inv

Tuple containing two dictionaries, mapping MIME types to a list of filename
extensions: the first dictionary is for the non-standards types and the
second one is for the standard types. They are initialized by
@ref{2e26,,common_types} and @ref{2e25,,types_map}.
@end deffn

@geindex guess_extension() (mimetypes.MimeTypes method)
@anchor{library/mimetypes mimetypes MimeTypes guess_extension}@anchor{2e30}
@deffn {Method} guess_extension (type, strict=True)

Similar to the @ref{2e1f,,guess_extension()} function, using the tables stored as part
of the object.
@end deffn

@geindex guess_type() (mimetypes.MimeTypes method)
@anchor{library/mimetypes mimetypes MimeTypes guess_type}@anchor{12ec}
@deffn {Method} guess_type (url, strict=True)

Similar to the @ref{123d,,guess_type()} function, using the tables stored as part of
the object.
@end deffn

@geindex guess_all_extensions() (mimetypes.MimeTypes method)
@anchor{library/mimetypes mimetypes MimeTypes guess_all_extensions}@anchor{2e31}
@deffn {Method} guess_all_extensions (type, strict=True)

Similar to the @ref{10fa,,guess_all_extensions()} function, using the tables stored
as part of the object.
@end deffn

@geindex read() (mimetypes.MimeTypes method)
@anchor{library/mimetypes mimetypes MimeTypes read}@anchor{2e2a}
@deffn {Method} read (filename, strict=True)

Load MIME information from a file named @emph{filename}.  This uses @ref{2e2b,,readfp()} to
parse the file.

If @emph{strict} is @code{True}, information will be added to list of standard types,
else to the list of non-standard types.
@end deffn

@geindex readfp() (mimetypes.MimeTypes method)
@anchor{library/mimetypes mimetypes MimeTypes readfp}@anchor{2e2b}
@deffn {Method} readfp (fp, strict=True)

Load MIME type information from an open file @emph{fp}.  The file must have the format of
the standard @code{mime.types} files.

If @emph{strict} is @code{True}, information will be added to the list of standard
types, else to the list of non-standard types.
@end deffn

@geindex read_windows_registry() (mimetypes.MimeTypes method)
@anchor{library/mimetypes mimetypes MimeTypes read_windows_registry}@anchor{2e32}
@deffn {Method} read_windows_registry (strict=True)

Load MIME type information from the Windows registry.

@ref{1629,,Availability}: Windows.

If @emph{strict} is @code{True}, information will be added to the list of standard
types, else to the list of non-standard types.


New in version 3.2.

@end deffn
@end deffn

@node base64 — Base16 Base32 Base64 Base85 Data Encodings,binascii — Convert between binary and ASCII,mimetypes — Map filenames to MIME types,Internet Data Handling
@anchor{library/base64 doc}@anchor{2e33}@anchor{library/base64 base64-base16-base32-base64-base85-data-encodings}@anchor{2e34}@anchor{library/base64 module-base64}@anchor{d}
@subsection @code{base64} — Base16, Base32, Base64, Base85 Data Encodings


@geindex module; base64

@strong{Source code:} Lib/base64.py@footnote{https://github.com/python/cpython/tree/main/Lib/base64.py}

@geindex base64; encoding
@geindex MIME; base64 encoding


__________________________________________________________________


This module provides functions for encoding binary data to printable
ASCII characters and decoding such encodings back to binary data.
It provides encoding and decoding functions for the encodings specified in
@geindex RFC; RFC 4648
RFC 4648@footnote{https://tools.ietf.org/html/rfc4648.html}, which defines the Base16, Base32, and Base64 algorithms,
and for the de-facto standard Ascii85 and Base85 encodings.

The 
@geindex RFC; RFC 4648
RFC 4648@footnote{https://tools.ietf.org/html/rfc4648.html} encodings are suitable for encoding binary data so that it can be
safely sent by email, used as parts of URLs, or included as part of an HTTP
POST request.  The encoding algorithm is not the same as the
@code{uuencode} program.

There are two interfaces provided by this module.  The modern interface
supports encoding @ref{87e,,bytes-like objects} to ASCII
@ref{2f0,,bytes}, and decoding @ref{87e,,bytes-like objects} or
strings containing ASCII to @ref{2f0,,bytes}.  Both base-64 alphabets
defined in 
@geindex RFC; RFC 4648
RFC 4648@footnote{https://tools.ietf.org/html/rfc4648.html} (normal, and URL- and filesystem-safe) are supported.

The legacy interface does not support decoding from strings, but it does
provide functions for encoding and decoding to and from @ref{d68,,file objects}.  It only supports the Base64 standard alphabet, and it adds
newlines every 76 characters as per 
@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html}.  Note that if you are looking
for 
@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html} support you probably want to be looking at the @ref{64,,email}
package instead.


Changed in version 3.3: ASCII-only Unicode strings are now accepted by the decoding functions of
the modern interface.



Changed in version 3.4: Any @ref{87e,,bytes-like objects} are now accepted by all
encoding and decoding functions in this module.  Ascii85/Base85 support added.


The modern interface provides:

@geindex b64encode() (in module base64)
@anchor{library/base64 base64 b64encode}@anchor{2e35}
@deffn {Function} base64.b64encode (s, altchars=None)

Encode the @ref{87e,,bytes-like object} @emph{s} using Base64 and return the encoded
@ref{2f0,,bytes}.

Optional @emph{altchars} must be a @ref{87e,,bytes-like object} of at least
length 2 (additional characters are ignored) which specifies an alternative
alphabet for the @code{+} and @code{/} characters.  This allows an application to e.g.
generate URL or filesystem safe Base64 strings.  The default is @code{None}, for
which the standard Base64 alphabet is used.
@end deffn

@geindex b64decode() (in module base64)
@anchor{library/base64 base64 b64decode}@anchor{123f}
@deffn {Function} base64.b64decode (s, altchars=None, validate=False)

Decode the Base64 encoded @ref{87e,,bytes-like object} or ASCII string
@emph{s} and return the decoded @ref{2f0,,bytes}.

Optional @emph{altchars} must be a @ref{87e,,bytes-like object} or ASCII string of
at least length 2 (additional characters are ignored) which specifies the
alternative alphabet used instead of the @code{+} and @code{/} characters.

A @ref{bb7,,binascii.Error} exception is raised
if @emph{s} is incorrectly padded.

If @emph{validate} is @code{False} (the default), characters that are neither
in the normal base-64 alphabet nor the alternative alphabet are
discarded prior to the padding check.  If @emph{validate} is @code{True},
these non-alphabet characters in the input result in a
@ref{bb7,,binascii.Error}.

For more information about the strict base64 check, see @ref{1272,,binascii.a2b_base64()}
@end deffn

@geindex standard_b64encode() (in module base64)
@anchor{library/base64 base64 standard_b64encode}@anchor{2e36}
@deffn {Function} base64.standard_b64encode (s)

Encode @ref{87e,,bytes-like object} @emph{s} using the standard Base64 alphabet
and return the encoded @ref{2f0,,bytes}.
@end deffn

@geindex standard_b64decode() (in module base64)
@anchor{library/base64 base64 standard_b64decode}@anchor{2e37}
@deffn {Function} base64.standard_b64decode (s)

Decode @ref{87e,,bytes-like object} or ASCII string @emph{s} using the standard
Base64 alphabet and return the decoded @ref{2f0,,bytes}.
@end deffn

@geindex urlsafe_b64encode() (in module base64)
@anchor{library/base64 base64 urlsafe_b64encode}@anchor{2e38}
@deffn {Function} base64.urlsafe_b64encode (s)

Encode @ref{87e,,bytes-like object} @emph{s} using the
URL- and filesystem-safe alphabet, which
substitutes @code{-} instead of @code{+} and @code{_} instead of @code{/} in the
standard Base64 alphabet, and return the encoded @ref{2f0,,bytes}.  The result
can still contain @code{=}.
@end deffn

@geindex urlsafe_b64decode() (in module base64)
@anchor{library/base64 base64 urlsafe_b64decode}@anchor{2e39}
@deffn {Function} base64.urlsafe_b64decode (s)

Decode @ref{87e,,bytes-like object} or ASCII string @emph{s}
using the URL- and filesystem-safe
alphabet, which substitutes @code{-} instead of @code{+} and @code{_} instead of
@code{/} in the standard Base64 alphabet, and return the decoded
@ref{2f0,,bytes}.
@end deffn

@geindex b32encode() (in module base64)
@anchor{library/base64 base64 b32encode}@anchor{2e3a}
@deffn {Function} base64.b32encode (s)

Encode the @ref{87e,,bytes-like object} @emph{s} using Base32 and return the
encoded @ref{2f0,,bytes}.
@end deffn

@geindex b32decode() (in module base64)
@anchor{library/base64 base64 b32decode}@anchor{bb6}
@deffn {Function} base64.b32decode (s, casefold=False, map01=None)

Decode the Base32 encoded @ref{87e,,bytes-like object} or ASCII string @emph{s} and
return the decoded @ref{2f0,,bytes}.

Optional @emph{casefold} is a flag specifying
whether a lowercase alphabet is acceptable as input.  For security purposes,
the default is @code{False}.

@geindex RFC; RFC 4648
RFC 4648@footnote{https://tools.ietf.org/html/rfc4648.html} allows for optional mapping of the digit 0 (zero) to the letter O
(oh), and for optional mapping of the digit 1 (one) to either the letter I (eye)
or letter L (el).  The optional argument @emph{map01} when not @code{None}, specifies
which letter the digit 1 should be mapped to (when @emph{map01} is not @code{None}, the
digit 0 is always mapped to the letter O).  For security purposes the default is
@code{None}, so that 0 and 1 are not allowed in the input.

A @ref{bb7,,binascii.Error} is raised if @emph{s} is
incorrectly padded or if there are non-alphabet characters present in the
input.
@end deffn

@geindex b32hexencode() (in module base64)
@anchor{library/base64 base64 b32hexencode}@anchor{261}
@deffn {Function} base64.b32hexencode (s)

Similar to @ref{2e3a,,b32encode()} but uses the Extended Hex Alphabet, as defined in
@geindex RFC; RFC 4648
RFC 4648@footnote{https://tools.ietf.org/html/rfc4648.html}.


New in version 3.10.

@end deffn

@geindex b32hexdecode() (in module base64)
@anchor{library/base64 base64 b32hexdecode}@anchor{262}
@deffn {Function} base64.b32hexdecode (s, casefold=False)

Similar to @ref{bb6,,b32decode()} but uses the Extended Hex Alphabet, as defined in
@geindex RFC; RFC 4648
RFC 4648@footnote{https://tools.ietf.org/html/rfc4648.html}.

This version does not allow the digit 0 (zero) to the letter O (oh) and digit
1 (one) to either the letter I (eye) or letter L (el) mappings, all these
characters are included in the Extended Hex Alphabet and are not
interchangeable.


New in version 3.10.

@end deffn

@geindex b16encode() (in module base64)
@anchor{library/base64 base64 b16encode}@anchor{2e3b}
@deffn {Function} base64.b16encode (s)

Encode the @ref{87e,,bytes-like object} @emph{s} using Base16 and return the
encoded @ref{2f0,,bytes}.
@end deffn

@geindex b16decode() (in module base64)
@anchor{library/base64 base64 b16decode}@anchor{2e3c}
@deffn {Function} base64.b16decode (s, casefold=False)

Decode the Base16 encoded @ref{87e,,bytes-like object} or ASCII string @emph{s} and
return the decoded @ref{2f0,,bytes}.

Optional @emph{casefold} is a flag specifying whether a
lowercase alphabet is acceptable as input.  For security purposes, the default
is @code{False}.

A @ref{bb7,,binascii.Error} is raised if @emph{s} is
incorrectly padded or if there are non-alphabet characters present in the
input.
@end deffn

@geindex a85encode() (in module base64)
@anchor{library/base64 base64 a85encode}@anchor{a98}
@deffn {Function} base64.a85encode (b, *, foldspaces=False, wrapcol=0, pad=False, adobe=False)

Encode the @ref{87e,,bytes-like object} @emph{b} using Ascii85 and return the
encoded @ref{2f0,,bytes}.

@emph{foldspaces} is an optional flag that uses the special short sequence ‘y’
instead of 4 consecutive spaces (ASCII 0x20) as supported by ‘btoa’. This
feature is not supported by the “standard” Ascii85 encoding.

@emph{wrapcol} controls whether the output should have newline (@code{b'\n'})
characters added to it. If this is non-zero, each output line will be
at most this many characters long.

@emph{pad} controls whether the input is padded to a multiple of 4
before encoding. Note that the @code{btoa} implementation always pads.

@emph{adobe} controls whether the encoded byte sequence is framed with @code{<~}
and @code{~>}, which is used by the Adobe implementation.


New in version 3.4.

@end deffn

@geindex a85decode() (in module base64)
@anchor{library/base64 base64 a85decode}@anchor{a99}
@deffn {Function} base64.a85decode (b, *, foldspaces=False, adobe=False, ignorechars=b' \t\n\r\x0b')

Decode the Ascii85 encoded @ref{87e,,bytes-like object} or ASCII string @emph{b} and
return the decoded @ref{2f0,,bytes}.

@emph{foldspaces} is a flag that specifies whether the ‘y’ short sequence
should be accepted as shorthand for 4 consecutive spaces (ASCII 0x20).
This feature is not supported by the “standard” Ascii85 encoding.

@emph{adobe} controls whether the input sequence is in Adobe Ascii85 format
(i.e. is framed with <~ and ~>).

@emph{ignorechars} should be a @ref{87e,,bytes-like object} or ASCII string
containing characters to ignore
from the input. This should only contain whitespace characters, and by
default contains all whitespace characters in ASCII.


New in version 3.4.

@end deffn

@geindex b85encode() (in module base64)
@anchor{library/base64 base64 b85encode}@anchor{a9a}
@deffn {Function} base64.b85encode (b, pad=False)

Encode the @ref{87e,,bytes-like object} @emph{b} using base85 (as used in e.g.
git-style binary diffs) and return the encoded @ref{2f0,,bytes}.

If @emph{pad} is true, the input is padded with @code{b'\0'} so its length is a
multiple of 4 bytes before encoding.


New in version 3.4.

@end deffn

@geindex b85decode() (in module base64)
@anchor{library/base64 base64 b85decode}@anchor{a9b}
@deffn {Function} base64.b85decode (b)

Decode the base85-encoded @ref{87e,,bytes-like object} or ASCII string @emph{b} and
return the decoded @ref{2f0,,bytes}.  Padding is implicitly removed, if
necessary.


New in version 3.4.

@end deffn

The legacy interface:

@geindex decode() (in module base64)
@anchor{library/base64 base64 decode}@anchor{2e3d}
@deffn {Function} base64.decode (input, output)

Decode the contents of the binary @emph{input} file and write the resulting binary
data to the @emph{output} file. @emph{input} and @emph{output} must be @ref{d68,,file objects}. @emph{input} will be read until @code{input.readline()} returns an
empty bytes object.
@end deffn

@geindex decodebytes() (in module base64)
@anchor{library/base64 base64 decodebytes}@anchor{454}
@deffn {Function} base64.decodebytes (s)

Decode the @ref{87e,,bytes-like object} @emph{s}, which must contain one or more
lines of base64 encoded data, and return the decoded @ref{2f0,,bytes}.


New in version 3.1.

@end deffn

@geindex encode() (in module base64)
@anchor{library/base64 base64 encode}@anchor{2e3e}
@deffn {Function} base64.encode (input, output)

Encode the contents of the binary @emph{input} file and write the resulting base64
encoded data to the @emph{output} file. @emph{input} and @emph{output} must be @ref{d68,,file objects}. @emph{input} will be read until @code{input.read()} returns
an empty bytes object. @ref{2e3e,,encode()} inserts a newline character (@code{b'\n'})
after every 76 bytes of the output, as well as ensuring that the output
always ends with a newline, as per 
@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html} (MIME).
@end deffn

@geindex encodebytes() (in module base64)
@anchor{library/base64 base64 encodebytes}@anchor{453}
@deffn {Function} base64.encodebytes (s)

Encode the @ref{87e,,bytes-like object} @emph{s}, which can contain arbitrary binary
data, and return @ref{2f0,,bytes} containing the base64-encoded data, with newlines
(@code{b'\n'}) inserted after every 76 bytes of output, and ensuring that
there is a trailing newline, as per 
@geindex RFC; RFC 2045
RFC 2045@footnote{https://tools.ietf.org/html/rfc2045.html} (MIME).


New in version 3.1.

@end deffn

An example usage of the module:

@example
>>> import base64
>>> encoded = base64.b64encode(b'data to be encoded')
>>> encoded
b'ZGF0YSB0byBiZSBlbmNvZGVk'
>>> data = base64.b64decode(encoded)
>>> data
b'data to be encoded'
@end example

@menu
* Security Considerations: Security Considerations<2>. 

@end menu

@node Security Considerations<2>,,,base64 — Base16 Base32 Base64 Base85 Data Encodings
@anchor{library/base64 base64-security}@anchor{2e3f}@anchor{library/base64 security-considerations}@anchor{2e40}
@subsubsection Security Considerations


A new security considerations section was added to 
@geindex RFC; RFC 4648
RFC 4648@footnote{https://tools.ietf.org/html/rfc4648.html} (section 12); it’s
recommended to review the security section for any code deployed to production.


@subsubheading See also



@table @asis

@item Module @ref{f,,binascii}

Support module containing ASCII-to-binary and binary-to-ASCII conversions.

@geindex RFC; RFC 1521

@item RFC 1521@footnote{https://tools.ietf.org/html/rfc1521.html} - MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies

Section 5.2, “Base64 Content-Transfer-Encoding,” provides the definition of the
base64 encoding.
@end table


@node binascii — Convert between binary and ASCII,quopri — Encode and decode MIME quoted-printable data,base64 — Base16 Base32 Base64 Base85 Data Encodings,Internet Data Handling
@anchor{library/binascii doc}@anchor{2e41}@anchor{library/binascii binascii-convert-between-binary-and-ascii}@anchor{2e42}@anchor{library/binascii module-binascii}@anchor{f}
@subsection @code{binascii} — Convert between binary and ASCII


@geindex module; binascii

@geindex module; uu
@geindex module; base64


__________________________________________________________________


The @ref{f,,binascii} module contains a number of methods to convert between
binary and various ASCII-encoded binary representations. Normally, you will not
use these functions directly but use wrapper modules like @ref{12a,,uu} or
@ref{d,,base64} instead. The @ref{f,,binascii} module contains
low-level functions written in C for greater speed that are used by the
higher-level modules.

@cartouche
@quotation Note 
@code{a2b_*} functions accept Unicode strings containing only ASCII characters.
Other functions only accept @ref{87e,,bytes-like objects} (such as
@ref{2f0,,bytes}, @ref{2f1,,bytearray} and other objects that support the buffer
protocol).


Changed in version 3.3: ASCII-only unicode strings are now accepted by the @code{a2b_*} functions.

@end quotation
@end cartouche

The @ref{f,,binascii} module defines the following functions:

@geindex a2b_uu() (in module binascii)
@anchor{library/binascii binascii a2b_uu}@anchor{2e43}
@deffn {Function} binascii.a2b_uu (string)

Convert a single line of uuencoded data back to binary and return the binary
data. Lines normally contain 45 (binary) bytes, except for the last line. Line
data may be followed by whitespace.
@end deffn

@geindex b2a_uu() (in module binascii)
@anchor{library/binascii binascii b2a_uu}@anchor{640}
@deffn {Function} binascii.b2a_uu (data, *, backtick=False)

Convert binary data to a line of ASCII characters, the return value is the
converted line, including a newline char. The length of @emph{data} should be at most
45. If @emph{backtick} is true, zeros are represented by @code{'`'} instead of spaces.


Changed in version 3.7: Added the @emph{backtick} parameter.

@end deffn

@geindex a2b_base64() (in module binascii)
@anchor{library/binascii binascii a2b_base64}@anchor{1272}
@deffn {Function} binascii.a2b_base64 (string, strict_mode=False)

Convert a block of base64 data back to binary and return the binary data. More
than one line may be passed at a time.

If @emph{strict_mode} is true, only valid base64 data will be converted. Invalid base64
data will raise @ref{bb7,,binascii.Error}.


@table @asis

@item Valid base64:


@itemize *

@item 
Conforms to 
@geindex RFC; RFC 3548
RFC 3548@footnote{https://tools.ietf.org/html/rfc3548.html}.

@item 
Contains only characters from the base64 alphabet.

@item 
Contains no excess data after padding (including excess padding, newlines, etc.).

@item 
Does not start with a padding.
@end itemize
@end table


Changed in version 3.11: Added the @emph{strict_mode} parameter.

@end deffn

@geindex b2a_base64() (in module binascii)
@anchor{library/binascii binascii b2a_base64}@anchor{7c6}
@deffn {Function} binascii.b2a_base64 (data, *, newline=True)

Convert binary data to a line of ASCII characters in base64 coding. The return
value is the converted line, including a newline char if @emph{newline} is
true.  The output of this function conforms to 
@geindex RFC; RFC 3548
RFC 3548@footnote{https://tools.ietf.org/html/rfc3548.html}.


Changed in version 3.6: Added the @emph{newline} parameter.

@end deffn

@geindex a2b_qp() (in module binascii)
@anchor{library/binascii binascii a2b_qp}@anchor{2e44}
@deffn {Function} binascii.a2b_qp (data, header=False)

Convert a block of quoted-printable data back to binary and return the binary
data. More than one line may be passed at a time. If the optional argument
@emph{header} is present and true, underscores will be decoded as spaces.
@end deffn

@geindex b2a_qp() (in module binascii)
@anchor{library/binascii binascii b2a_qp}@anchor{2e45}
@deffn {Function} binascii.b2a_qp (data, quotetabs=False, istext=True, header=False)

Convert binary data to a line(s) of ASCII characters in quoted-printable
encoding.  The return value is the converted line(s). If the optional argument
@emph{quotetabs} is present and true, all tabs and spaces will be encoded.   If the
optional argument @emph{istext} is present and true, newlines are not encoded but
trailing whitespace will be encoded. If the optional argument @emph{header} is
present and true, spaces will be encoded as underscores per 
@geindex RFC; RFC 1522
RFC 1522@footnote{https://tools.ietf.org/html/rfc1522.html}. If the
optional argument @emph{header} is present and false, newline characters will be
encoded as well; otherwise linefeed conversion might corrupt the binary data
stream.
@end deffn

@geindex crc_hqx() (in module binascii)
@anchor{library/binascii binascii crc_hqx}@anchor{1b8}
@deffn {Function} binascii.crc_hqx (data, value)

Compute a 16-bit CRC value of @emph{data}, starting with @emph{value} as the
initial CRC, and return the result.  This uses the CRC-CCITT polynomial
@emph{x}@w{^16} + @emph{x}@w{^12} + @emph{x}@w{^5} + 1, often represented as
0x1021.  This CRC is used in the binhex4 format.
@end deffn

@geindex crc32() (in module binascii)
@anchor{library/binascii binascii crc32}@anchor{2e46}
@deffn {Function} binascii.crc32 (data[, value])

Compute CRC-32, the 32-bit checksum of @emph{data}, starting with an
initial CRC of @emph{value}.  The default initial CRC is zero.  The algorithm
is consistent with the ZIP file checksum.  Since the algorithm is designed for
use as a checksum algorithm, it is not suitable for use as a general hash
algorithm.  Use as follows:

@example
print(binascii.crc32(b"hello world"))
# Or, in two pieces:
crc = binascii.crc32(b"hello")
crc = binascii.crc32(b" world", crc)
print('crc32 = @{:#010x@}'.format(crc))
@end example


Changed in version 3.0: The result is always unsigned.
To generate the same numeric value across all Python versions and
platforms, use @code{crc32(data) & 0xffffffff}.

@end deffn

@geindex b2a_hex() (in module binascii)
@geindex hexlify() (in module binascii)
@anchor{library/binascii binascii b2a_hex}@anchor{1cf6}
@deffn {Function} binascii.b2a_hex (data[, sep[, bytes_per_sep=1]])
@anchor{library/binascii binascii hexlify}@anchor{128c}
@deffnx {Function} binascii.hexlify (data[, sep[, bytes_per_sep=1]])

Return the hexadecimal representation of the binary @emph{data}.  Every byte of
@emph{data} is converted into the corresponding 2-digit hex representation.  The
returned bytes object is therefore twice as long as the length of @emph{data}.

Similar functionality (but returning a text string) is also conveniently
accessible using the @ref{8c1,,bytes.hex()} method.

If @emph{sep} is specified, it must be a single character str or bytes object.
It will be inserted in the output after every @emph{bytes_per_sep} input bytes.
Separator placement is counted from the right end of the output by default,
if you wish to count from the left, supply a negative @emph{bytes_per_sep} value.

@example
>>> import binascii
>>> binascii.b2a_hex(b'\xb9\x01\xef')
b'b901ef'
>>> binascii.hexlify(b'\xb9\x01\xef', '-')
b'b9-01-ef'
>>> binascii.b2a_hex(b'\xb9\x01\xef', b'_', 2)
b'b9_01ef'
>>> binascii.b2a_hex(b'\xb9\x01\xef', b' ', -2)
b'b901 ef'
@end example


Changed in version 3.8: The @emph{sep} and @emph{bytes_per_sep} parameters were added.

@end deffn

@geindex a2b_hex() (in module binascii)
@geindex unhexlify() (in module binascii)
@anchor{library/binascii binascii a2b_hex}@anchor{1cf7}
@deffn {Function} binascii.a2b_hex (hexstr)
@anchor{library/binascii binascii unhexlify}@anchor{1302}
@deffnx {Function} binascii.unhexlify (hexstr)

Return the binary data represented by the hexadecimal string @emph{hexstr}.  This
function is the inverse of @ref{1cf6,,b2a_hex()}. @emph{hexstr} must contain an even number
of hexadecimal digits (which can be upper or lower case), otherwise an
@ref{bb7,,Error} exception is raised.

Similar functionality (accepting only text string arguments, but more
liberal towards whitespace) is also accessible using the
@ref{60c,,bytes.fromhex()} class method.
@end deffn

@geindex Error
@anchor{library/binascii binascii Error}@anchor{bb7}
@deffn {Exception} binascii.Error

Exception raised on errors. These are usually programming errors.
@end deffn

@geindex Incomplete
@anchor{library/binascii binascii Incomplete}@anchor{2e47}
@deffn {Exception} binascii.Incomplete

Exception raised on incomplete data. These are usually not programming errors,
but may be handled by reading a little more data and trying again.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{d,,base64}

Support for RFC compliant base64-style encoding in base 16, 32, 64,
and 85.

@item Module @ref{12a,,uu}

Support for UU encoding used on Unix.

@item Module @ref{d6,,quopri}

Support for quoted-printable encoding used in MIME email messages.
@end table


@node quopri — Encode and decode MIME quoted-printable data,uu — Encode and decode uuencode files,binascii — Convert between binary and ASCII,Internet Data Handling
@anchor{library/quopri doc}@anchor{2e48}@anchor{library/quopri module-quopri}@anchor{d6}@anchor{library/quopri quopri-encode-and-decode-mime-quoted-printable-data}@anchor{2e49}
@subsection @code{quopri} — Encode and decode MIME quoted-printable data


@geindex module; quopri

@strong{Source code:} Lib/quopri.py@footnote{https://github.com/python/cpython/tree/main/Lib/quopri.py}

@geindex quoted-printable; encoding
@geindex MIME; quoted-printable encoding


__________________________________________________________________


This module performs quoted-printable transport encoding and decoding, as
defined in 
@geindex RFC; RFC 1521
RFC 1521@footnote{https://tools.ietf.org/html/rfc1521.html}: “MIME (Multipurpose Internet Mail Extensions) Part One:
Mechanisms for Specifying and Describing the Format of Internet Message Bodies”.
The quoted-printable encoding is designed for data where there are relatively
few nonprintable characters; the base64 encoding scheme available via the
@ref{d,,base64} module is more compact if there are many such characters, as when
sending a graphics file.

@geindex decode() (in module quopri)
@anchor{library/quopri quopri decode}@anchor{1cf9}
@deffn {Function} quopri.decode (input, output, header=False)

Decode the contents of the @emph{input} file and write the resulting decoded binary
data to the @emph{output} file. @emph{input} and @emph{output} must be @ref{d68,,binary file objects}.  If the optional argument @emph{header} is present and true, underscore
will be decoded as space. This is used to decode “Q”-encoded headers as
described in 
@geindex RFC; RFC 1522
RFC 1522@footnote{https://tools.ietf.org/html/rfc1522.html}: “MIME (Multipurpose Internet Mail Extensions)
Part Two: Message Header Extensions for Non-ASCII Text”.
@end deffn

@geindex encode() (in module quopri)
@anchor{library/quopri quopri encode}@anchor{1cf8}
@deffn {Function} quopri.encode (input, output, quotetabs, header=False)

Encode the contents of the @emph{input} file and write the resulting quoted-printable
data to the @emph{output} file. @emph{input} and @emph{output} must be
@ref{d68,,binary file objects}. @emph{quotetabs}, a
non-optional flag which controls whether to encode embedded spaces
and tabs; when true it encodes such embedded whitespace, and when
false it leaves them unencoded.
Note that spaces and tabs appearing at the end of lines are always encoded,
as per 
@geindex RFC; RFC 1521
RFC 1521@footnote{https://tools.ietf.org/html/rfc1521.html}.  @emph{header} is a flag which controls if spaces are encoded
as underscores as per 
@geindex RFC; RFC 1522
RFC 1522@footnote{https://tools.ietf.org/html/rfc1522.html}.
@end deffn

@geindex decodestring() (in module quopri)
@anchor{library/quopri quopri decodestring}@anchor{2e4a}
@deffn {Function} quopri.decodestring (s, header=False)

Like @ref{1cf9,,decode()}, except that it accepts a source @ref{2f0,,bytes} and
returns the corresponding decoded @ref{2f0,,bytes}.
@end deffn

@geindex encodestring() (in module quopri)
@anchor{library/quopri quopri encodestring}@anchor{2e4b}
@deffn {Function} quopri.encodestring (s, quotetabs=False, header=False)

Like @ref{1cf8,,encode()}, except that it accepts a source @ref{2f0,,bytes} and
returns the corresponding encoded @ref{2f0,,bytes}. By default, it sends a
@code{False} value to @emph{quotetabs} parameter of the @ref{1cf8,,encode()} function.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{d,,base64}

Encode and decode MIME base64 data
@end table


@node uu — Encode and decode uuencode files,,quopri — Encode and decode MIME quoted-printable data,Internet Data Handling
@anchor{library/uu doc}@anchor{2e4c}@anchor{library/uu module-uu}@anchor{12a}@anchor{library/uu uu-encode-and-decode-uuencode-files}@anchor{2e4d}
@subsection @code{uu} — Encode and decode uuencode files


@geindex module; uu

@strong{Source code:} Lib/uu.py@footnote{https://github.com/python/cpython/tree/main/Lib/uu.py}


__________________________________________________________________


This module encodes and decodes files in uuencode format, allowing arbitrary
binary data to be transferred over ASCII-only connections. Wherever a file
argument is expected, the methods accept a file-like object.  For backwards
compatibility, a string containing a pathname is also accepted, and the
corresponding file will be opened for reading and writing; the pathname @code{'-'}
is understood to mean the standard input or output.  However, this interface is
deprecated; it’s better for the caller to open the file itself, and be sure
that, when required, the mode is @code{'rb'} or @code{'wb'} on Windows.

@geindex Jansen@comma{} Jack
@geindex Ellinghouse@comma{} Lance

This code was contributed by Lance Ellinghouse, and modified by Jack Jansen.

The @ref{12a,,uu} module defines the following functions:

@geindex encode() (in module uu)
@anchor{library/uu uu encode}@anchor{6d6}
@deffn {Function} uu.encode (in_file, out_file, name=None, mode=None, *, backtick=False)

Uuencode file @emph{in_file} into file @emph{out_file}.  The uuencoded file will have
the header specifying @emph{name} and @emph{mode} as the defaults for the results of
decoding the file. The default defaults are taken from @emph{in_file}, or @code{'-'}
and @code{0o666} respectively.  If @emph{backtick} is true, zeros are represented by
@code{'`'} instead of spaces.


Changed in version 3.7: Added the @emph{backtick} parameter.

@end deffn

@geindex decode() (in module uu)
@anchor{library/uu uu decode}@anchor{1cfa}
@deffn {Function} uu.decode (in_file, out_file=None, mode=None, quiet=False)

This call decodes uuencoded file @emph{in_file} placing the result on file
@emph{out_file}. If @emph{out_file} is a pathname, @emph{mode} is used to set the permission
bits if the file must be created. Defaults for @emph{out_file} and @emph{mode} are taken
from the uuencode header.  However, if the file specified in the header already
exists, a @ref{2e4e,,uu.Error} is raised.

@ref{1cfa,,decode()} may print a warning to standard error if the input was produced
by an incorrect uuencoder and Python could recover from that error.  Setting
@emph{quiet} to a true value silences this warning.
@end deffn

@geindex Error
@anchor{library/uu uu Error}@anchor{2e4e}
@deffn {Exception} uu.Error

Subclass of @ref{4d8,,Exception}, this can be raised by @ref{1cfa,,uu.decode()} under
various situations, such as described above, but also including a badly
formatted header, or truncated input file.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{f,,binascii}

Support module containing ASCII-to-binary and binary-to-ASCII conversions.
@end table


@node Structured Markup Processing Tools,Internet Protocols and Support,Internet Data Handling,The Python Standard Library
@anchor{library/markup doc}@anchor{2e4f}@anchor{library/markup markup}@anchor{2e50}@anchor{library/markup structured-markup-processing-tools}@anchor{2e51}
@section Structured Markup Processing Tools


Python supports a variety of modules to work with various forms of structured
data markup.  This includes modules to work with the Standard Generalized Markup
Language (SGML) and the Hypertext Markup Language (HTML), and several interfaces
for working with the Extensible Markup Language (XML).

@menu
* html — HyperText Markup Language support:: 
* html.parser — Simple HTML and XHTML parser: html parser — Simple HTML and XHTML parser. 
* html.entities — Definitions of HTML general entities: html entities — Definitions of HTML general entities. 
* XML Processing Modules:: 
* xml.etree.ElementTree — The ElementTree XML API: xml etree ElementTree — The ElementTree XML API. 
* xml.dom — The Document Object Model API: xml dom — The Document Object Model API. 
* xml.dom.minidom — Minimal DOM implementation: xml dom minidom — Minimal DOM implementation. 
* xml.dom.pulldom — Support for building partial DOM trees: xml dom pulldom — Support for building partial DOM trees. 
* xml.sax — Support for SAX2 parsers: xml sax — Support for SAX2 parsers. 
* xml.sax.handler — Base classes for SAX handlers: xml sax handler — Base classes for SAX handlers. 
* xml.sax.saxutils — SAX Utilities: xml sax saxutils — SAX Utilities. 
* xml.sax.xmlreader — Interface for XML parsers: xml sax xmlreader — Interface for XML parsers. 
* xml.parsers.expat — Fast XML parsing using Expat: xml parsers expat — Fast XML parsing using Expat. 

@end menu

@node html — HyperText Markup Language support,html parser — Simple HTML and XHTML parser,,Structured Markup Processing Tools
@anchor{library/html doc}@anchor{2e52}@anchor{library/html html-hypertext-markup-language-support}@anchor{2e53}@anchor{library/html module-html}@anchor{8b}
@subsection @code{html} — HyperText Markup Language support


@geindex module; html

@strong{Source code:} Lib/html/__init__.py@footnote{https://github.com/python/cpython/tree/main/Lib/html/__init__.py}


__________________________________________________________________


This module defines utilities to manipulate HTML.

@geindex escape() (in module html)
@anchor{library/html html escape}@anchor{b95}
@deffn {Function} html.escape (s, quote=True)

Convert the characters @code{&}, @code{<} and @code{>} in string @emph{s} to HTML-safe
sequences.  Use this if you need to display text that might contain such
characters in HTML.  If the optional flag @emph{quote} is true, the characters
(@code{"}) and (@code{'}) are also translated; this helps for inclusion in an HTML
attribute value delimited by quotes, as in @code{<a href="...">}.


New in version 3.2.

@end deffn

@geindex unescape() (in module html)
@anchor{library/html html unescape}@anchor{45a}
@deffn {Function} html.unescape (s)

Convert all named and numeric character references (e.g. @code{&gt;},
@code{&#62;}, @code{&#x3e;}) in the string @emph{s} to the corresponding Unicode
characters.  This function uses the rules defined by the HTML 5 standard
for both valid and invalid character references, and the @ref{c64,,list of HTML 5 named character references}.


New in version 3.4.

@end deffn


__________________________________________________________________


Submodules in the @code{html} package are:


@itemize *

@item 
@ref{8d,,html.parser} – HTML/XHTML parser with lenient parsing mode

@item 
@ref{8c,,html.entities} – HTML entity definitions
@end itemize

@node html parser — Simple HTML and XHTML parser,html entities — Definitions of HTML general entities,html — HyperText Markup Language support,Structured Markup Processing Tools
@anchor{library/html parser doc}@anchor{2e54}@anchor{library/html parser html-parser-simple-html-and-xhtml-parser}@anchor{2e55}@anchor{library/html parser module-html parser}@anchor{8d}
@subsection @code{html.parser} — Simple HTML and XHTML parser


@geindex module; html.parser

@strong{Source code:} Lib/html/parser.py@footnote{https://github.com/python/cpython/tree/main/Lib/html/parser.py}

@geindex HTML
@geindex XHTML


__________________________________________________________________


This module defines a class @ref{33c,,HTMLParser} which serves as the basis for
parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML.

@geindex HTMLParser (class in html.parser)
@anchor{library/html parser html parser HTMLParser}@anchor{33c}
@deffn {Class} html.parser.HTMLParser (*, convert_charrefs=True)

Create a parser instance able to parse invalid markup.

If @emph{convert_charrefs} is @code{True} (the default), all character
references (except the ones in @code{script}/@code{style} elements) are
automatically converted to the corresponding Unicode characters.

An @ref{33c,,HTMLParser} instance is fed HTML data and calls handler methods
when start tags, end tags, text, comments, and other markup elements are
encountered.  The user should subclass @ref{33c,,HTMLParser} and override its
methods to implement the desired behavior.

This parser does not check that end tags match start tags or call the end-tag
handler for elements which are closed implicitly by closing an outer element.


Changed in version 3.4: @emph{convert_charrefs} keyword argument added.



Changed in version 3.5: The default value for argument @emph{convert_charrefs} is now @code{True}.

@end deffn

@menu
* Example HTML Parser Application:: 
* HTMLParser Methods:: 
* Examples: Examples<22>. 

@end menu

@node Example HTML Parser Application,HTMLParser Methods,,html parser — Simple HTML and XHTML parser
@anchor{library/html parser example-html-parser-application}@anchor{2e56}
@subsubsection Example HTML Parser Application


As a basic example, below is a simple HTML parser that uses the
@ref{33c,,HTMLParser} class to print out start tags, end tags, and data
as they are encountered:

@example
from html.parser import HTMLParser

class MyHTMLParser(HTMLParser):
    def handle_starttag(self, tag, attrs):
        print("Encountered a start tag:", tag)

    def handle_endtag(self, tag):
        print("Encountered an end tag :", tag)

    def handle_data(self, data):
        print("Encountered some data  :", data)

parser = MyHTMLParser()
parser.feed('<html><head><title>Test</title></head>'
            '<body><h1>Parse me!</h1></body></html>')
@end example

The output will then be:

@example
Encountered a start tag: html
Encountered a start tag: head
Encountered a start tag: title
Encountered some data  : Test
Encountered an end tag : title
Encountered an end tag : head
Encountered a start tag: body
Encountered a start tag: h1
Encountered some data  : Parse me!
Encountered an end tag : h1
Encountered an end tag : body
Encountered an end tag : html
@end example

@node HTMLParser Methods,Examples<22>,Example HTML Parser Application,html parser — Simple HTML and XHTML parser
@anchor{library/html parser htmlparser-methods}@anchor{2e57}
@subsubsection @code{HTMLParser} Methods


@ref{33c,,HTMLParser} instances have the following methods:

@geindex feed() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser feed}@anchor{2e58}
@deffn {Method} HTMLParser.feed (data)

Feed some text to the parser.  It is processed insofar as it consists of
complete elements; incomplete data is buffered until more data is fed or
@ref{2e59,,close()} is called.  @emph{data} must be @ref{2ef,,str}.
@end deffn

@geindex close() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser close}@anchor{2e59}
@deffn {Method} HTMLParser.close ()

Force processing of all buffered data as if it were followed by an end-of-file
mark.  This method may be redefined by a derived class to define additional
processing at the end of the input, but the redefined version should always call
the @ref{33c,,HTMLParser} base class method @ref{2e59,,close()}.
@end deffn

@geindex reset() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser reset}@anchor{2e5a}
@deffn {Method} HTMLParser.reset ()

Reset the instance.  Loses all unprocessed data.  This is called implicitly at
instantiation time.
@end deffn

@geindex getpos() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser getpos}@anchor{2e5b}
@deffn {Method} HTMLParser.getpos ()

Return current line number and offset.
@end deffn

@geindex get_starttag_text() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser get_starttag_text}@anchor{2e5c}
@deffn {Method} HTMLParser.get_starttag_text ()

Return the text of the most recently opened start tag.  This should not normally
be needed for structured processing, but may be useful in dealing with HTML “as
deployed” or for re-generating input with minimal changes (whitespace between
attributes can be preserved, etc.).
@end deffn

The following methods are called when data or markup elements are encountered
and they are meant to be overridden in a subclass.  The base class
implementations do nothing (except for @ref{2e5d,,handle_startendtag()}):

@geindex handle_starttag() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser handle_starttag}@anchor{2e5e}
@deffn {Method} HTMLParser.handle_starttag (tag, attrs)

This method is called to handle the start tag of an element (e.g. @code{<div id="main">}).

The @emph{tag} argument is the name of the tag converted to lower case. The @emph{attrs}
argument is a list of @code{(name, value)} pairs containing the attributes found
inside the tag’s @code{<>} brackets.  The @emph{name} will be translated to lower case,
and quotes in the @emph{value} have been removed, and character and entity references
have been replaced.

For instance, for the tag @code{<A HREF="https://www.cwi.nl/">}, this method
would be called as @code{handle_starttag('a', [('href', 'https://www.cwi.nl/')])}.

All entity references from @ref{8c,,html.entities} are replaced in the attribute
values.
@end deffn

@geindex handle_endtag() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser handle_endtag}@anchor{2e5f}
@deffn {Method} HTMLParser.handle_endtag (tag)

This method is called to handle the end tag of an element (e.g. @code{</div>}).

The @emph{tag} argument is the name of the tag converted to lower case.
@end deffn

@geindex handle_startendtag() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser handle_startendtag}@anchor{2e5d}
@deffn {Method} HTMLParser.handle_startendtag (tag, attrs)

Similar to @ref{2e5e,,handle_starttag()}, but called when the parser encounters an
XHTML-style empty tag (@code{<img ... />}).  This method may be overridden by
subclasses which require this particular lexical information; the default
implementation simply calls @ref{2e5e,,handle_starttag()} and @ref{2e5f,,handle_endtag()}.
@end deffn

@geindex handle_data() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser handle_data}@anchor{2e60}
@deffn {Method} HTMLParser.handle_data (data)

This method is called to process arbitrary data (e.g. text nodes and the
content of @code{<script>...</script>} and @code{<style>...</style>}).
@end deffn

@geindex handle_entityref() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser handle_entityref}@anchor{2e61}
@deffn {Method} HTMLParser.handle_entityref (name)

This method is called to process a named character reference of the form
@code{&name;} (e.g. @code{&gt;}), where @emph{name} is a general entity reference
(e.g. @code{'gt'}).  This method is never called if @emph{convert_charrefs} is
@code{True}.
@end deffn

@geindex handle_charref() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser handle_charref}@anchor{2e62}
@deffn {Method} HTMLParser.handle_charref (name)

This method is called to process decimal and hexadecimal numeric character
references of the form @code{&#NNN;} and @code{&#xNNN;}.  For example, the decimal
equivalent for @code{&gt;} is @code{&#62;}, whereas the hexadecimal is @code{&#x3E;};
in this case the method will receive @code{'62'} or @code{'x3E'}.  This method
is never called if @emph{convert_charrefs} is @code{True}.
@end deffn

@geindex handle_comment() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser handle_comment}@anchor{2e63}
@deffn {Method} HTMLParser.handle_comment (data)

This method is called when a comment is encountered (e.g. @code{<!--comment-->}).

For example, the comment @code{<!-- comment -->} will cause this method to be
called with the argument @code{' comment '}.

The content of Internet Explorer conditional comments (condcoms) will also be
sent to this method, so, for @code{<!--[if IE 9]>IE9-specific content<![endif]-->},
this method will receive @code{'[if IE 9]>IE9-specific content<![endif]'}.
@end deffn

@geindex handle_decl() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser handle_decl}@anchor{2e64}
@deffn {Method} HTMLParser.handle_decl (decl)

This method is called to handle an HTML doctype declaration (e.g.
@code{<!DOCTYPE html>}).

The @emph{decl} parameter will be the entire contents of the declaration inside
the @code{<!...>} markup (e.g. @code{'DOCTYPE html'}).
@end deffn

@geindex handle_pi() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser handle_pi}@anchor{2e65}
@deffn {Method} HTMLParser.handle_pi (data)

Method called when a processing instruction is encountered.  The @emph{data}
parameter will contain the entire processing instruction. For example, for the
processing instruction @code{<?proc color='red'>}, this method would be called as
@code{handle_pi("proc color='red'")}.  It is intended to be overridden by a derived
class; the base class implementation does nothing.

@cartouche
@quotation Note 
The @ref{33c,,HTMLParser} class uses the SGML syntactic rules for processing
instructions.  An XHTML processing instruction using the trailing @code{'?'} will
cause the @code{'?'} to be included in @emph{data}.
@end quotation
@end cartouche
@end deffn

@geindex unknown_decl() (html.parser.HTMLParser method)
@anchor{library/html parser html parser HTMLParser unknown_decl}@anchor{2e66}
@deffn {Method} HTMLParser.unknown_decl (data)

This method is called when an unrecognized declaration is read by the parser.

The @emph{data} parameter will be the entire contents of the declaration inside
the @code{<![...]>} markup.  It is sometimes useful to be overridden by a
derived class.  The base class implementation does nothing.
@end deffn

@node Examples<22>,,HTMLParser Methods,html parser — Simple HTML and XHTML parser
@anchor{library/html parser examples}@anchor{2e67}@anchor{library/html parser htmlparser-examples}@anchor{2e68}
@subsubsection Examples


The following class implements a parser that will be used to illustrate more
examples:

@example
from html.parser import HTMLParser
from html.entities import name2codepoint

class MyHTMLParser(HTMLParser):
    def handle_starttag(self, tag, attrs):
        print("Start tag:", tag)
        for attr in attrs:
            print("     attr:", attr)

    def handle_endtag(self, tag):
        print("End tag  :", tag)

    def handle_data(self, data):
        print("Data     :", data)

    def handle_comment(self, data):
        print("Comment  :", data)

    def handle_entityref(self, name):
        c = chr(name2codepoint[name])
        print("Named ent:", c)

    def handle_charref(self, name):
        if name.startswith('x'):
            c = chr(int(name[1:], 16))
        else:
            c = chr(int(name))
        print("Num ent  :", c)

    def handle_decl(self, data):
        print("Decl     :", data)

parser = MyHTMLParser()
@end example

Parsing a doctype:

@example
>>> parser.feed('<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" '
...             '"http://www.w3.org/TR/html4/strict.dtd">')
Decl     : DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"
@end example

Parsing an element with a few attributes and a title:

@example
>>> parser.feed('<img src="python-logo.png" alt="The Python logo">')
Start tag: img
     attr: ('src', 'python-logo.png')
     attr: ('alt', 'The Python logo')
>>>
>>> parser.feed('<h1>Python</h1>')
Start tag: h1
Data     : Python
End tag  : h1
@end example

The content of @code{script} and @code{style} elements is returned as is, without
further parsing:

@example
>>> parser.feed('<style type="text/css">#python @{ color: green @}</style>')
Start tag: style
     attr: ('type', 'text/css')
Data     : #python @{ color: green @}
End tag  : style

>>> parser.feed('<script type="text/javascript">'
...             'alert("<strong>hello!</strong>");</script>')
Start tag: script
     attr: ('type', 'text/javascript')
Data     : alert("<strong>hello!</strong>");
End tag  : script
@end example

Parsing comments:

@example
>>> parser.feed('<!-- a comment -->'
...             '<!--[if IE 9]>IE-specific content<![endif]-->')
Comment  :  a comment
Comment  : [if IE 9]>IE-specific content<![endif]
@end example

Parsing named and numeric character references and converting them to the
correct char (note: these 3 references are all equivalent to @code{'>'}):

@example
>>> parser.feed('&gt;&#62;&#x3E;')
Named ent: >
Num ent  : >
Num ent  : >
@end example

Feeding incomplete chunks to @ref{2e58,,feed()} works, but
@ref{2e60,,handle_data()} might be called more than once
(unless @emph{convert_charrefs} is set to @code{True}):

@example
>>> for chunk in ['<sp', 'an>buff', 'ered ', 'text</s', 'pan>']:
...     parser.feed(chunk)
...
Start tag: span
Data     : buff
Data     : ered
Data     : text
End tag  : span
@end example

Parsing invalid HTML (e.g. unquoted attributes) also works:

@example
>>> parser.feed('<p><a class=link href=#main>tag soup</p ></a>')
Start tag: p
Start tag: a
     attr: ('class', 'link')
     attr: ('href', '#main')
Data     : tag soup
End tag  : p
End tag  : a
@end example

@node html entities — Definitions of HTML general entities,XML Processing Modules,html parser — Simple HTML and XHTML parser,Structured Markup Processing Tools
@anchor{library/html entities doc}@anchor{2e69}@anchor{library/html entities html-entities-definitions-of-html-general-entities}@anchor{2e6a}@anchor{library/html entities module-html entities}@anchor{8c}
@subsection @code{html.entities} — Definitions of HTML general entities


@geindex module; html.entities

@strong{Source code:} Lib/html/entities.py@footnote{https://github.com/python/cpython/tree/main/Lib/html/entities.py}


__________________________________________________________________


This module defines four dictionaries, @ref{c64,,html5},
@ref{2e6b,,name2codepoint}, @ref{2e6c,,codepoint2name}, and @ref{2e6d,,entitydefs}.

@geindex html5 (in module html.entities)
@anchor{library/html entities html entities html5}@anchor{c64}
@deffn {Data} html.entities.html5

A dictionary that maps HTML5 named character references @footnote{@w{(1)} 
See @indicateurl{https://html.spec.whatwg.org/multipage/syntax.html#named-character-references}
} to the
equivalent Unicode character(s), e.g. @code{html5['gt;'] == '>'}.
Note that the trailing semicolon is included in the name (e.g. @code{'gt;'}),
however some of the names are accepted by the standard even without the
semicolon: in this case the name is present with and without the @code{';'}.
See also @ref{45a,,html.unescape()}.


New in version 3.3.

@end deffn

@geindex entitydefs (in module html.entities)
@anchor{library/html entities html entities entitydefs}@anchor{2e6d}
@deffn {Data} html.entities.entitydefs

A dictionary mapping XHTML 1.0 entity definitions to their replacement text in
ISO Latin-1.
@end deffn

@geindex name2codepoint (in module html.entities)
@anchor{library/html entities html entities name2codepoint}@anchor{2e6b}
@deffn {Data} html.entities.name2codepoint

A dictionary that maps HTML entity names to the Unicode code points.
@end deffn

@geindex codepoint2name (in module html.entities)
@anchor{library/html entities html entities codepoint2name}@anchor{2e6c}
@deffn {Data} html.entities.codepoint2name

A dictionary that maps Unicode code points to HTML entity names.
@end deffn

@node XML Processing Modules,xml etree ElementTree — The ElementTree XML API,html entities — Definitions of HTML general entities,Structured Markup Processing Tools
@anchor{library/xml doc}@anchor{2e6e}@anchor{library/xml module-xml}@anchor{13a}@anchor{library/xml xml}@anchor{2e6f}@anchor{library/xml xml-processing-modules}@anchor{2e70}
@subsection XML Processing Modules


@geindex module; xml

@strong{Source code:} Lib/xml/@footnote{https://github.com/python/cpython/tree/main/Lib/xml/}


__________________________________________________________________


Python’s interfaces for processing XML are grouped in the @code{xml} package.

@cartouche
@quotation Warning 
The XML modules are not secure against erroneous or maliciously
constructed data.  If you need to parse untrusted or
unauthenticated data see the @ref{2e71,,XML vulnerabilities} and
@ref{2e72,,The defusedxml Package} sections.
@end quotation
@end cartouche

It is important to note that modules in the @ref{13a,,xml} package require that
there be at least one SAX-compliant XML parser available. The Expat parser is
included with Python, so the @ref{13f,,xml.parsers.expat} module will always be
available.

The documentation for the @ref{13b,,xml.dom} and @ref{142,,xml.sax} packages are the
definition of the Python bindings for the DOM and SAX interfaces.

The XML handling submodules are:


@itemize *

@item 
@ref{13e,,xml.etree.ElementTree}: the ElementTree API, a simple and lightweight
XML processor
@end itemize



@itemize *

@item 
@ref{13b,,xml.dom}: the DOM API definition

@item 
@ref{13c,,xml.dom.minidom}: a minimal DOM implementation

@item 
@ref{13d,,xml.dom.pulldom}: support for building partial DOM trees
@end itemize



@itemize *

@item 
@ref{142,,xml.sax}: SAX2 base classes and convenience functions

@item 
@ref{13f,,xml.parsers.expat}: the Expat parser binding
@end itemize

@menu
* XML vulnerabilities:: 
* The defusedxml Package:: 

@end menu

@node XML vulnerabilities,The defusedxml Package,,XML Processing Modules
@anchor{library/xml id1}@anchor{2e73}@anchor{library/xml xml-vulnerabilities}@anchor{2e71}
@subsubsection XML vulnerabilities


The XML processing modules are not secure against maliciously constructed data.
An attacker can abuse XML features to carry out denial of service attacks,
access local files, generate network connections to other machines, or
circumvent firewalls.

The following table gives an overview of the known attacks and whether
the various modules are vulnerable to them.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} 
@headitem

kind

@tab

sax

@tab

etree

@tab

minidom

@tab

pulldom

@tab

xmlrpc

@item

billion laughs

@tab

@strong{Vulnerable} (1)

@tab

@strong{Vulnerable} (1)

@tab

@strong{Vulnerable} (1)

@tab

@strong{Vulnerable} (1)

@tab

@strong{Vulnerable} (1)

@item

quadratic blowup

@tab

@strong{Vulnerable} (1)

@tab

@strong{Vulnerable} (1)

@tab

@strong{Vulnerable} (1)

@tab

@strong{Vulnerable} (1)

@tab

@strong{Vulnerable} (1)

@item

external entity expansion

@tab

Safe (5)

@tab

Safe (2)

@tab

Safe (3)

@tab

Safe (5)

@tab

Safe (4)

@item

DTD@footnote{https://en.wikipedia.org/wiki/Document_type_definition} retrieval

@tab

Safe (5)

@tab

Safe

@tab

Safe

@tab

Safe (5)

@tab

Safe

@item

decompression bomb

@tab

Safe

@tab

Safe

@tab

Safe

@tab

Safe

@tab

@strong{Vulnerable}

@end multitable



@enumerate 

@item 
Expat 2.4.1 and newer is not vulnerable to the “billion laughs” and
“quadratic blowup” vulnerabilities. Items still listed as vulnerable due to
potential reliance on system-provided libraries. Check
@code{pyexpat.EXPAT_VERSION}.

@item 
@ref{13e,,xml.etree.ElementTree} doesn’t expand external entities and raises a
@code{ParserError} when an entity occurs.

@item 
@ref{13c,,xml.dom.minidom} doesn’t expand external entities and simply returns
the unexpanded entity verbatim.

@item 
@code{xmlrpclib} doesn’t expand external entities and omits them.

@item 
Since Python 3.7.1, external general entities are no longer processed by
default.
@end enumerate


@table @asis

@item billion laughs / exponential entity expansion

The Billion Laughs@footnote{https://en.wikipedia.org/wiki/Billion_laughs} attack – also known as exponential entity expansion –
uses multiple levels of nested entities. Each entity refers to another entity
several times, and the final entity definition contains a small string.
The exponential expansion results in several gigabytes of text and
consumes lots of memory and CPU time.

@item quadratic blowup entity expansion

A quadratic blowup attack is similar to a Billion Laughs@footnote{https://en.wikipedia.org/wiki/Billion_laughs} attack; it abuses
entity expansion, too. Instead of nested entities it repeats one large entity
with a couple of thousand chars over and over again. The attack isn’t as
efficient as the exponential case but it avoids triggering parser countermeasures
that forbid deeply-nested entities.

@item external entity expansion

Entity declarations can contain more than just text for replacement. They can
also point to external resources or local files. The XML
parser accesses the resource and embeds the content into the XML document.

@item DTD@footnote{https://en.wikipedia.org/wiki/Document_type_definition} retrieval

Some XML libraries like Python’s @ref{13d,,xml.dom.pulldom} retrieve document type
definitions from remote or local locations. The feature has similar
implications as the external entity expansion issue.

@item decompression bomb

Decompression bombs (aka ZIP bomb@footnote{https://en.wikipedia.org/wiki/Zip_bomb}) apply to all XML libraries
that can parse compressed XML streams such as gzipped HTTP streams or
LZMA-compressed
files. For an attacker it can reduce the amount of transmitted data by three
magnitudes or more.
@end table

The documentation for defusedxml@footnote{https://pypi.org/project/defusedxml/} on PyPI has further information about
all known attack vectors with examples and references.

@node The defusedxml Package,,XML vulnerabilities,XML Processing Modules
@anchor{library/xml defusedxml-package}@anchor{2e72}@anchor{library/xml the-defusedxml-package}@anchor{2e74}
@subsubsection The @code{defusedxml} Package


defusedxml@footnote{https://pypi.org/project/defusedxml/} is a pure Python package with modified subclasses of all stdlib
XML parsers that prevent any potentially malicious operation. Use of this
package is recommended for any server code that parses untrusted XML data. The
package also ships with example exploits and extended documentation on more
XML exploits such as XPath injection.

@node xml etree ElementTree — The ElementTree XML API,xml dom — The Document Object Model API,XML Processing Modules,Structured Markup Processing Tools
@anchor{library/xml etree elementtree doc}@anchor{2e75}@anchor{library/xml etree elementtree dtd}@anchor{2e76}@anchor{library/xml etree elementtree module-xml etree ElementTree}@anchor{13e}@anchor{library/xml etree elementtree xml-etree-elementtree-the-elementtree-xml-api}@anchor{2e77}
@subsection @code{xml.etree.ElementTree} — The ElementTree XML API


@geindex module; xml.etree.ElementTree

@strong{Source code:} Lib/xml/etree/ElementTree.py@footnote{https://github.com/python/cpython/tree/main/Lib/xml/etree/ElementTree.py}


__________________________________________________________________


The @ref{13e,,xml.etree.ElementTree} module implements a simple and efficient API
for parsing and creating XML data.


Changed in version 3.3: This module will use a fast implementation whenever available.



Deprecated since version 3.3: The @code{xml.etree.cElementTree} module is deprecated.


@cartouche
@quotation Warning 
The @ref{13e,,xml.etree.ElementTree} module is not secure against
maliciously constructed data.  If you need to parse untrusted or
unauthenticated data see @ref{2e71,,XML vulnerabilities}.
@end quotation
@end cartouche

@menu
* Tutorial:: 
* XPath support:: 
* Reference: Reference<2>. 
* XInclude support:: 
* Reference: Reference<3>. 

@end menu

@node Tutorial,XPath support,,xml etree ElementTree — The ElementTree XML API
@anchor{library/xml etree elementtree tutorial}@anchor{2e78}
@subsubsection Tutorial


This is a short tutorial for using @ref{13e,,xml.etree.ElementTree} (@code{ET} in
short).  The goal is to demonstrate some of the building blocks and basic
concepts of the module.

@menu
* XML tree and elements:: 
* Parsing XML:: 
* Pull API for non-blocking parsing:: 
* Finding interesting elements:: 
* Modifying an XML File:: 
* Building XML documents:: 
* Parsing XML with Namespaces:: 

@end menu

@node XML tree and elements,Parsing XML,,Tutorial
@anchor{library/xml etree elementtree xml-tree-and-elements}@anchor{2e79}
@subsubsection XML tree and elements


XML is an inherently hierarchical data format, and the most natural way to
represent it is with a tree.  @code{ET} has two classes for this purpose -
@ref{44d,,ElementTree} represents the whole XML document as a tree, and
@ref{44e,,Element} represents a single node in this tree.  Interactions with
the whole document (reading and writing to/from files) are usually done
on the @ref{44d,,ElementTree} level.  Interactions with a single XML element
and its sub-elements are done on the @ref{44e,,Element} level.

@node Parsing XML,Pull API for non-blocking parsing,XML tree and elements,Tutorial
@anchor{library/xml etree elementtree elementtree-parsing-xml}@anchor{2e7a}@anchor{library/xml etree elementtree parsing-xml}@anchor{2e7b}
@subsubsection Parsing XML


We’ll be using the following XML document as the sample data for this section:

@example
<?xml version="1.0"?>
<data>
    <country name="Liechtenstein">
        <rank>1</rank>
        <year>2008</year>
        <gdppc>141100</gdppc>
        <neighbor name="Austria" direction="E"/>
        <neighbor name="Switzerland" direction="W"/>
    </country>
    <country name="Singapore">
        <rank>4</rank>
        <year>2011</year>
        <gdppc>59900</gdppc>
        <neighbor name="Malaysia" direction="N"/>
    </country>
    <country name="Panama">
        <rank>68</rank>
        <year>2011</year>
        <gdppc>13600</gdppc>
        <neighbor name="Costa Rica" direction="W"/>
        <neighbor name="Colombia" direction="E"/>
    </country>
</data>
@end example

We can import this data by reading from a file:

@example
import xml.etree.ElementTree as ET
tree = ET.parse('country_data.xml')
root = tree.getroot()
@end example

Or directly from a string:

@example
root = ET.fromstring(country_data_as_string)
@end example

@ref{2e7c,,fromstring()} parses XML from a string directly into an @ref{44e,,Element},
which is the root element of the parsed tree.  Other parsing functions may
create an @ref{44d,,ElementTree}.  Check the documentation to be sure.

As an @ref{44e,,Element}, @code{root} has a tag and a dictionary of attributes:

@example
>>> root.tag
'data'
>>> root.attrib
@{@}
@end example

It also has children nodes over which we can iterate:

@example
>>> for child in root:
...     print(child.tag, child.attrib)
...
country @{'name': 'Liechtenstein'@}
country @{'name': 'Singapore'@}
country @{'name': 'Panama'@}
@end example

Children are nested, and we can access specific child nodes by index:

@example
>>> root[0][1].text
'2008'
@end example

@cartouche
@quotation Note 
Not all elements of the XML input will end up as elements of the
parsed tree. Currently, this module skips over any XML comments,
processing instructions, and document type declarations in the
input. Nevertheless, trees built using this module’s API rather
than parsing from XML text can have comments and processing
instructions in them; they will be included when generating XML
output. A document type declaration may be accessed by passing a
custom @ref{56a,,TreeBuilder} instance to the @ref{569,,XMLParser}
constructor.
@end quotation
@end cartouche

@node Pull API for non-blocking parsing,Finding interesting elements,Parsing XML,Tutorial
@anchor{library/xml etree elementtree elementtree-pull-parsing}@anchor{b7a}@anchor{library/xml etree elementtree pull-api-for-non-blocking-parsing}@anchor{2e7d}
@subsubsection Pull API for non-blocking parsing


Most parsing functions provided by this module require the whole document
to be read at once before returning any result.  It is possible to use an
@ref{569,,XMLParser} and feed data into it incrementally, but it is a push API that
calls methods on a callback target, which is too low-level and inconvenient for
most needs.  Sometimes what the user really wants is to be able to parse XML
incrementally, without blocking operations, while enjoying the convenience of
fully constructed @ref{44e,,Element} objects.

The most powerful tool for doing this is @ref{b79,,XMLPullParser}.  It does not
require a blocking read to obtain the XML data, and is instead fed with data
incrementally with @ref{2e7e,,XMLPullParser.feed()} calls.  To get the parsed XML
elements, call @ref{2e7f,,XMLPullParser.read_events()}.  Here is an example:

@example
>>> parser = ET.XMLPullParser(['start', 'end'])
>>> parser.feed('<mytag>sometext')
>>> list(parser.read_events())
[('start', <Element 'mytag' at 0x7fa66db2be58>)]
>>> parser.feed(' more text</mytag>')
>>> for event, elem in parser.read_events():
...     print(event)
...     print(elem.tag, 'text=', elem.text)
...
end
@end example

The obvious use case is applications that operate in a non-blocking fashion
where the XML data is being received from a socket or read incrementally from
some storage device.  In such cases, blocking reads are unacceptable.

Because it’s so flexible, @ref{b79,,XMLPullParser} can be inconvenient to use for
simpler use-cases.  If you don’t mind your application blocking on reading XML
data but would still like to have incremental parsing capabilities, take a look
at @ref{ba1,,iterparse()}.  It can be useful when you’re reading a large XML document
and don’t want to hold it wholly in memory.

@node Finding interesting elements,Modifying an XML File,Pull API for non-blocking parsing,Tutorial
@anchor{library/xml etree elementtree finding-interesting-elements}@anchor{2e80}
@subsubsection Finding interesting elements


@ref{44e,,Element} has some useful methods that help iterate recursively over all
the sub-tree below it (its children, their children, and so on).  For example,
@ref{ee9,,Element.iter()}:

@example
>>> for neighbor in root.iter('neighbor'):
...     print(neighbor.attrib)
...
@{'name': 'Austria', 'direction': 'E'@}
@{'name': 'Switzerland', 'direction': 'W'@}
@{'name': 'Malaysia', 'direction': 'N'@}
@{'name': 'Costa Rica', 'direction': 'W'@}
@{'name': 'Colombia', 'direction': 'E'@}
@end example

@ref{2e81,,Element.findall()} finds only elements with a tag which are direct
children of the current element.  @ref{2e82,,Element.find()} finds the @emph{first} child
with a particular tag, and @ref{2e83,,Element.text} accesses the element’s text
content.  @ref{2e84,,Element.get()} accesses the element’s attributes:

@example
>>> for country in root.findall('country'):
...     rank = country.find('rank').text
...     name = country.get('name')
...     print(name, rank)
...
Liechtenstein 1
Singapore 4
Panama 68
@end example

More sophisticated specification of which elements to look for is possible by
using @ref{6e0,,XPath}.

@node Modifying an XML File,Building XML documents,Finding interesting elements,Tutorial
@anchor{library/xml etree elementtree modifying-an-xml-file}@anchor{2e85}
@subsubsection Modifying an XML File


@ref{44d,,ElementTree} provides a simple way to build XML documents and write them to files.
The @ref{b7d,,ElementTree.write()} method serves this purpose.

Once created, an @ref{44e,,Element} object may be manipulated by directly changing
its fields (such as @ref{2e83,,Element.text}), adding and modifying attributes
(@ref{2e86,,Element.set()} method), as well as adding new children (for example
with @ref{2e87,,Element.append()}).

Let’s say we want to add one to each country’s rank, and add an @code{updated}
attribute to the rank element:

@example
>>> for rank in root.iter('rank'):
...     new_rank = int(rank.text) + 1
...     rank.text = str(new_rank)
...     rank.set('updated', 'yes')
...
>>> tree.write('output.xml')
@end example

Our XML now looks like this:

@example
<?xml version="1.0"?>
<data>
    <country name="Liechtenstein">
        <rank updated="yes">2</rank>
        <year>2008</year>
        <gdppc>141100</gdppc>
        <neighbor name="Austria" direction="E"/>
        <neighbor name="Switzerland" direction="W"/>
    </country>
    <country name="Singapore">
        <rank updated="yes">5</rank>
        <year>2011</year>
        <gdppc>59900</gdppc>
        <neighbor name="Malaysia" direction="N"/>
    </country>
    <country name="Panama">
        <rank updated="yes">69</rank>
        <year>2011</year>
        <gdppc>13600</gdppc>
        <neighbor name="Costa Rica" direction="W"/>
        <neighbor name="Colombia" direction="E"/>
    </country>
</data>
@end example

We can remove elements using @ref{2e88,,Element.remove()}.  Let’s say we want to
remove all countries with a rank higher than 50:

@example
>>> for country in root.findall('country'):
...     # using root.findall() to avoid removal during traversal
...     rank = int(country.find('rank').text)
...     if rank > 50:
...         root.remove(country)
...
>>> tree.write('output.xml')
@end example

Note that concurrent modification while iterating can lead to problems,
just like when iterating and modifying Python lists or dicts.
Therefore, the example first collects all matching elements with
@code{root.findall()}, and only then iterates over the list of matches.

Our XML now looks like this:

@example
<?xml version="1.0"?>
<data>
    <country name="Liechtenstein">
        <rank updated="yes">2</rank>
        <year>2008</year>
        <gdppc>141100</gdppc>
        <neighbor name="Austria" direction="E"/>
        <neighbor name="Switzerland" direction="W"/>
    </country>
    <country name="Singapore">
        <rank updated="yes">5</rank>
        <year>2011</year>
        <gdppc>59900</gdppc>
        <neighbor name="Malaysia" direction="N"/>
    </country>
</data>
@end example

@node Building XML documents,Parsing XML with Namespaces,Modifying an XML File,Tutorial
@anchor{library/xml etree elementtree building-xml-documents}@anchor{2e89}
@subsubsection Building XML documents


The @ref{2e8a,,SubElement()} function also provides a convenient way to create new
sub-elements for a given element:

@example
>>> a = ET.Element('a')
>>> b = ET.SubElement(a, 'b')
>>> c = ET.SubElement(a, 'c')
>>> d = ET.SubElement(c, 'd')
>>> ET.dump(a)
<a><b /><c><d /></c></a>
@end example

@node Parsing XML with Namespaces,,Building XML documents,Tutorial
@anchor{library/xml etree elementtree parsing-xml-with-namespaces}@anchor{2e8b}
@subsubsection Parsing XML with Namespaces


If the XML input has namespaces@footnote{https://en.wikipedia.org/wiki/XML_namespace}, tags and attributes
with prefixes in the form @code{prefix:sometag} get expanded to
@code{@{uri@}sometag} where the @emph{prefix} is replaced by the full @emph{URI}.
Also, if there is a default namespace@footnote{https://www.w3.org/TR/xml-names/#defaulting},
that full URI gets prepended to all of the non-prefixed tags.

Here is an XML example that incorporates two namespaces, one with the
prefix “fictional” and the other serving as the default namespace:

@example
<?xml version="1.0"?>
<actors xmlns:fictional="http://characters.example.com"
        xmlns="http://people.example.com">
    <actor>
        <name>John Cleese</name>
        <fictional:character>Lancelot</fictional:character>
        <fictional:character>Archie Leach</fictional:character>
    </actor>
    <actor>
        <name>Eric Idle</name>
        <fictional:character>Sir Robin</fictional:character>
        <fictional:character>Gunther</fictional:character>
        <fictional:character>Commander Clement</fictional:character>
    </actor>
</actors>
@end example

One way to search and explore this XML example is to manually add the
URI to every tag or attribute in the xpath of a
@ref{2e82,,find()} or @ref{2e81,,findall()}:

@example
root = fromstring(xml_text)
for actor in root.findall('@{http://people.example.com@}actor'):
    name = actor.find('@{http://people.example.com@}name')
    print(name.text)
    for char in actor.findall('@{http://characters.example.com@}character'):
        print(' |-->', char.text)
@end example

A better way to search the namespaced XML example is to create a
dictionary with your own prefixes and use those in the search functions:

@example
ns = @{'real_person': 'http://people.example.com',
      'role': 'http://characters.example.com'@}

for actor in root.findall('real_person:actor', ns):
    name = actor.find('real_person:name', ns)
    print(name.text)
    for char in actor.findall('role:character', ns):
        print(' |-->', char.text)
@end example

These two approaches both output:

@example
John Cleese
 |--> Lancelot
 |--> Archie Leach
Eric Idle
 |--> Sir Robin
 |--> Gunther
 |--> Commander Clement
@end example

@node XPath support,Reference<2>,Tutorial,xml etree ElementTree — The ElementTree XML API
@anchor{library/xml etree elementtree elementtree-xpath}@anchor{6e0}@anchor{library/xml etree elementtree xpath-support}@anchor{2e8c}
@subsubsection XPath support


This module provides limited support for
XPath expressions@footnote{https://www.w3.org/TR/xpath} for locating elements in a
tree.  The goal is to support a small subset of the abbreviated syntax; a full
XPath engine is outside the scope of the module.

@menu
* Example: Example<9>. 
* Supported XPath syntax:: 

@end menu

@node Example<9>,Supported XPath syntax,,XPath support
@anchor{library/xml etree elementtree example}@anchor{2e8d}
@subsubsection Example


Here’s an example that demonstrates some of the XPath capabilities of the
module.  We’ll be using the @code{countrydata} XML document from the
@ref{2e7a,,Parsing XML} section:

@example
import xml.etree.ElementTree as ET

root = ET.fromstring(countrydata)

# Top-level elements
root.findall(".")

# All 'neighbor' grand-children of 'country' children of the top-level
# elements
root.findall("./country/neighbor")

# Nodes with name='Singapore' that have a 'year' child
root.findall(".//year/..[@@name='Singapore']")

# 'year' nodes that are children of nodes with name='Singapore'
root.findall(".//*[@@name='Singapore']/year")

# All 'neighbor' nodes that are the second child of their parent
root.findall(".//neighbor[2]")
@end example

For XML with namespaces, use the usual qualified @code{@{namespace@}tag} notation:

@example
# All dublin-core "title" tags in the document
root.findall(".//@{http://purl.org/dc/elements/1.1/@}title")
@end example

@node Supported XPath syntax,,Example<9>,XPath support
@anchor{library/xml etree elementtree supported-xpath-syntax}@anchor{2e8e}
@subsubsection Supported XPath syntax



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Syntax

@tab

Meaning

@item

@code{tag}

@tab

Selects all child elements with the given tag.
For example, @code{spam} selects all child elements
named @code{spam}, and @code{spam/egg} selects all
grandchildren named @code{egg} in all children named
@code{spam}.  @code{@{namespace@}*} selects all tags in the
given namespace, @code{@{*@}spam} selects tags named
@code{spam} in any (or no) namespace, and @code{@{@}*}
only selects tags that are not in a namespace.


Changed in version 3.8: Support for star-wildcards was added.


@item

@code{*}

@tab

Selects all child elements, including comments and
processing instructions.  For example, @code{*/egg}
selects all grandchildren named @code{egg}.

@item

@code{.}

@tab

Selects the current node.  This is mostly useful
at the beginning of the path, to indicate that it’s
a relative path.

@item

@code{//}

@tab

Selects all subelements, on all levels beneath the
current  element.  For example, @code{.//egg} selects
all @code{egg} elements in the entire tree.

@item

@code{..}

@tab

Selects the parent element.  Returns @code{None} if the
path attempts to reach the ancestors of the start
element (the element @code{find} was called on).

@item

@code{[@@attrib]}

@tab

Selects all elements that have the given attribute.

@item

@code{[@@attrib='value']}

@tab

Selects all elements for which the given attribute
has the given value.  The value cannot contain
quotes.

@item

@code{[@@attrib!='value']}

@tab

Selects all elements for which the given attribute
does not have the given value. The value cannot
contain quotes.


New in version 3.10.


@item

@code{[tag]}

@tab

Selects all elements that have a child named
@code{tag}.  Only immediate children are supported.

@item

@code{[.='text']}

@tab

Selects all elements whose complete text content,
including descendants, equals the given @code{text}.


New in version 3.7.


@item

@code{[.!='text']}

@tab

Selects all elements whose complete text content,
including descendants, does not equal the given
@code{text}.


New in version 3.10.


@item

@code{[tag='text']}

@tab

Selects all elements that have a child named
@code{tag} whose complete text content, including
descendants, equals the given @code{text}.

@item

@code{[tag!='text']}

@tab

Selects all elements that have a child named
@code{tag} whose complete text content, including
descendants, does not equal the given @code{text}.


New in version 3.10.


@item

@code{[position]}

@tab

Selects all elements that are located at the given
position.  The position can be either an integer
(1 is the first position), the expression @code{last()}
(for the last position), or a position relative to
the last position (e.g. @code{last()-1}).

@end multitable


Predicates (expressions within square brackets) must be preceded by a tag
name, an asterisk, or another predicate.  @code{position} predicates must be
preceded by a tag name.

@node Reference<2>,XInclude support,XPath support,xml etree ElementTree — The ElementTree XML API
@anchor{library/xml etree elementtree reference}@anchor{2e8f}
@subsubsection Reference


@menu
* Functions: Functions<7>. 

@end menu

@node Functions<7>,,,Reference<2>
@anchor{library/xml etree elementtree elementtree-functions}@anchor{2e90}@anchor{library/xml etree elementtree functions}@anchor{2e91}
@subsubsection Functions


@geindex canonicalize() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree canonicalize}@anchor{2e92}
@deffn {Function} xml.etree.ElementTree.canonicalize (xml_data=None, *, out=None, from_file=None, **options)

C14N 2.0@footnote{https://www.w3.org/TR/xml-c14n2/} transformation function.

Canonicalization is a way to normalise XML output in a way that allows
byte-by-byte comparisons and digital signatures.  It reduced the freedom
that XML serializers have and instead generates a more constrained XML
representation.  The main restrictions regard the placement of namespace
declarations, the ordering of attributes, and ignorable whitespace.

This function takes an XML data string (@emph{xml_data}) or a file path or
file-like object (@emph{from_file}) as input, converts it to the canonical
form, and writes it out using the @emph{out} file(-like) object, if provided,
or returns it as a text string if not.  The output file receives text,
not bytes.  It should therefore be opened in text mode with @code{utf-8}
encoding.

Typical uses:

@example
xml_data = "<root>...</root>"
print(canonicalize(xml_data))

with open("c14n_output.xml", mode='w', encoding='utf-8') as out_file:
    canonicalize(xml_data, out=out_file)

with open("c14n_output.xml", mode='w', encoding='utf-8') as out_file:
    canonicalize(from_file="inputfile.xml", out=out_file)
@end example

The configuration @emph{options} are as follows:


@itemize -

@item 
@emph{with_comments}: set to true to include comments (default: false)

@item 

@table @asis

@item @emph{strip_text}: set to true to strip whitespace before and after text content

(default: false)
@end table

@item 

@table @asis

@item @emph{rewrite_prefixes}: set to true to replace namespace prefixes by “n@{number@}”

(default: false)
@end table

@item 

@table @asis

@item @emph{qname_aware_tags}: a set of qname aware tag names in which prefixes

should be replaced in text content (default: empty)
@end table

@item 

@table @asis

@item @emph{qname_aware_attrs}: a set of qname aware attribute names in which prefixes

should be replaced in text content (default: empty)
@end table

@item 
@emph{exclude_attrs}: a set of attribute names that should not be serialised

@item 
@emph{exclude_tags}: a set of tag names that should not be serialised
@end itemize

In the option list above, “a set” refers to any collection or iterable of
strings, no ordering is expected.


New in version 3.8.

@end deffn

@geindex Comment() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree Comment}@anchor{2e93}
@deffn {Function} xml.etree.ElementTree.Comment (text=None)

Comment element factory.  This factory function creates a special element
that will be serialized as an XML comment by the standard serializer.  The
comment string can be either a bytestring or a Unicode string.  @emph{text} is a
string containing the comment string.  Returns an element instance
representing a comment.

Note that @ref{569,,XMLParser} skips over comments in the input
instead of creating comment objects for them. An @ref{44d,,ElementTree} will
only contain comment nodes if they have been inserted into to
the tree using one of the @ref{44e,,Element} methods.
@end deffn

@geindex dump() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree dump}@anchor{2e94}
@deffn {Function} xml.etree.ElementTree.dump (elem)

Writes an element tree or element structure to sys.stdout.  This function
should be used for debugging only.

The exact output format is implementation dependent.  In this version, it’s
written as an ordinary XML file.

@emph{elem} is an element tree or an individual element.


Changed in version 3.8: The @ref{2e94,,dump()} function now preserves the attribute order specified
by the user.

@end deffn

@geindex fromstring() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree fromstring}@anchor{2e7c}
@deffn {Function} xml.etree.ElementTree.fromstring (text, parser=None)

Parses an XML section from a string constant.  Same as @ref{2e95,,XML()}.  @emph{text}
is a string containing XML data.  @emph{parser} is an optional parser instance.
If not given, the standard @ref{569,,XMLParser} parser is used.
Returns an @ref{44e,,Element} instance.
@end deffn

@geindex fromstringlist() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree fromstringlist}@anchor{d75}
@deffn {Function} xml.etree.ElementTree.fromstringlist (sequence, parser=None)

Parses an XML document from a sequence of string fragments.  @emph{sequence} is a
list or other sequence containing XML data fragments.  @emph{parser} is an
optional parser instance.  If not given, the standard @ref{569,,XMLParser}
parser is used.  Returns an @ref{44e,,Element} instance.


New in version 3.2.

@end deffn

@geindex indent() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree indent}@anchor{2e96}
@deffn {Function} xml.etree.ElementTree.indent (tree, space='  ', level=0)

Appends whitespace to the subtree to indent the tree visually.
This can be used to generate pretty-printed XML output.
@emph{tree} can be an Element or ElementTree.  @emph{space} is the whitespace
string that will be inserted for each indentation level, two space
characters by default.  For indenting partial subtrees inside of an
already indented tree, pass the initial indentation level as @emph{level}.


New in version 3.9.

@end deffn

@geindex iselement() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree iselement}@anchor{2e97}
@deffn {Function} xml.etree.ElementTree.iselement (element)

Check if an object appears to be a valid element object.  @emph{element} is an
element instance.  Return @code{True} if this is an element object.
@end deffn

@geindex iterparse() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree iterparse}@anchor{ba1}
@deffn {Function} xml.etree.ElementTree.iterparse (source, events=None, parser=None)

Parses an XML section into an element tree incrementally, and reports what’s
going on to the user.  @emph{source} is a filename or @ref{d68,,file object}
containing XML data.  @emph{events} is a sequence of events to report back.  The
supported events are the strings @code{"start"}, @code{"end"}, @code{"comment"},
@code{"pi"}, @code{"start-ns"} and @code{"end-ns"}
(the “ns” events are used to get detailed namespace
information).  If @emph{events} is omitted, only @code{"end"} events are reported.
@emph{parser} is an optional parser instance.  If not given, the standard
@ref{569,,XMLParser} parser is used.  @emph{parser} must be a subclass of
@ref{569,,XMLParser} and can only use the default @ref{56a,,TreeBuilder} as a
target.  Returns an @ref{11c3,,iterator} providing @code{(event, elem)} pairs.

Note that while @ref{ba1,,iterparse()} builds the tree incrementally, it issues
blocking reads on @emph{source} (or the file it names).  As such, it’s unsuitable
for applications where blocking reads can’t be made.  For fully non-blocking
parsing, see @ref{b79,,XMLPullParser}.

@cartouche
@quotation Note 
@ref{ba1,,iterparse()} only guarantees that it has seen the “>” character of a
starting tag when it emits a “start” event, so the attributes are defined,
but the contents of the text and tail attributes are undefined at that
point.  The same applies to the element children; they may or may not be
present.

If you need a fully populated element, look for “end” events instead.
@end quotation
@end cartouche


Deprecated since version 3.4: The @emph{parser} argument.



Changed in version 3.8: The @code{comment} and @code{pi} events were added.

@end deffn

@geindex parse() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree parse}@anchor{2e98}
@deffn {Function} xml.etree.ElementTree.parse (source, parser=None)

Parses an XML section into an element tree.  @emph{source} is a filename or file
object containing XML data.  @emph{parser} is an optional parser instance.  If
not given, the standard @ref{569,,XMLParser} parser is used.  Returns an
@ref{44d,,ElementTree} instance.
@end deffn

@geindex ProcessingInstruction() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree ProcessingInstruction}@anchor{2e99}
@deffn {Function} xml.etree.ElementTree.ProcessingInstruction (target, text=None)

PI element factory.  This factory function creates a special element that
will be serialized as an XML processing instruction.  @emph{target} is a string
containing the PI target.  @emph{text} is a string containing the PI contents, if
given.  Returns an element instance, representing a processing instruction.

Note that @ref{569,,XMLParser} skips over processing instructions
in the input instead of creating comment objects for them. An
@ref{44d,,ElementTree} will only contain processing instruction nodes if
they have been inserted into to the tree using one of the
@ref{44e,,Element} methods.
@end deffn

@geindex register_namespace() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree register_namespace}@anchor{d76}
@deffn {Function} xml.etree.ElementTree.register_namespace (prefix, uri)

Registers a namespace prefix.  The registry is global, and any existing
mapping for either the given prefix or the namespace URI will be removed.
@emph{prefix} is a namespace prefix.  @emph{uri} is a namespace uri.  Tags and
attributes in this namespace will be serialized with the given prefix, if at
all possible.


New in version 3.2.

@end deffn

@geindex SubElement() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree SubElement}@anchor{2e8a}
@deffn {Function} xml.etree.ElementTree.SubElement (parent, tag, attrib=@{@}, **extra)

Subelement factory.  This function creates an element instance, and appends
it to an existing element.

The element name, attribute names, and attribute values can be either
bytestrings or Unicode strings.  @emph{parent} is the parent element.  @emph{tag} is
the subelement name.  @emph{attrib} is an optional dictionary, containing element
attributes.  @emph{extra} contains additional attributes, given as keyword
arguments.  Returns an element instance.
@end deffn

@geindex tostring() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree tostring}@anchor{b7b}
@deffn {Function} xml.etree.ElementTree.tostring (element, encoding='us-ascii', method='xml', *, xml_declaration=None, default_namespace=None, short_empty_elements=True)

Generates a string representation of an XML element, including all
subelements.  @emph{element} is an @ref{44e,,Element} instance.  @emph{encoding} @footnote{@w{(1)} 
The encoding string included in XML output should conform to the
appropriate standards.  For example, “UTF-8” is valid, but “UTF8” is
not.  See @indicateurl{https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl}
and @indicateurl{https://www.iana.org/assignments/character-sets/character-sets.xhtml}.
} is
the output encoding (default is US-ASCII).  Use @code{encoding="unicode"} to
generate a Unicode string (otherwise, a bytestring is generated).  @emph{method}
is either @code{"xml"}, @code{"html"} or @code{"text"} (default is @code{"xml"}).
@emph{xml_declaration}, @emph{default_namespace} and @emph{short_empty_elements} has the same
meaning as in @ref{b7d,,ElementTree.write()}. Returns an (optionally) encoded string
containing the XML data.


New in version 3.4: The @emph{short_empty_elements} parameter.



New in version 3.8: The @emph{xml_declaration} and @emph{default_namespace} parameters.



Changed in version 3.8: The @ref{b7b,,tostring()} function now preserves the attribute order
specified by the user.

@end deffn

@geindex tostringlist() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree tostringlist}@anchor{b7c}
@deffn {Function} xml.etree.ElementTree.tostringlist (element, encoding='us-ascii', method='xml', *, xml_declaration=None, default_namespace=None, short_empty_elements=True)

Generates a string representation of an XML element, including all
subelements.  @emph{element} is an @ref{44e,,Element} instance.  @emph{encoding} @footnote{@w{(1)} 
The encoding string included in XML output should conform to the
appropriate standards.  For example, “UTF-8” is valid, but “UTF8” is
not.  See @indicateurl{https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl}
and @indicateurl{https://www.iana.org/assignments/character-sets/character-sets.xhtml}.
} is
the output encoding (default is US-ASCII).  Use @code{encoding="unicode"} to
generate a Unicode string (otherwise, a bytestring is generated).  @emph{method}
is either @code{"xml"}, @code{"html"} or @code{"text"} (default is @code{"xml"}).
@emph{xml_declaration}, @emph{default_namespace} and @emph{short_empty_elements} has the same
meaning as in @ref{b7d,,ElementTree.write()}. Returns a list of (optionally) encoded
strings containing the XML data. It does not guarantee any specific sequence,
except that @code{b"".join(tostringlist(element)) == tostring(element)}.


New in version 3.2.



New in version 3.4: The @emph{short_empty_elements} parameter.



New in version 3.8: The @emph{xml_declaration} and @emph{default_namespace} parameters.



Changed in version 3.8: The @ref{b7c,,tostringlist()} function now preserves the attribute order
specified by the user.

@end deffn

@geindex XML() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree XML}@anchor{2e95}
@deffn {Function} xml.etree.ElementTree.XML (text, parser=None)

Parses an XML section from a string constant.  This function can be used to
embed “XML literals” in Python code.  @emph{text} is a string containing XML
data.  @emph{parser} is an optional parser instance.  If not given, the standard
@ref{569,,XMLParser} parser is used.  Returns an @ref{44e,,Element} instance.
@end deffn

@geindex XMLID() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree XMLID}@anchor{2e9a}
@deffn {Function} xml.etree.ElementTree.XMLID (text, parser=None)

Parses an XML section from a string constant, and also returns a dictionary
which maps from element id:s to elements.  @emph{text} is a string containing XML
data.  @emph{parser} is an optional parser instance.  If not given, the standard
@ref{569,,XMLParser} parser is used.  Returns a tuple containing an
@ref{44e,,Element} instance and a dictionary.
@end deffn

@node XInclude support,Reference<3>,Reference<2>,xml etree ElementTree — The ElementTree XML API
@anchor{library/xml etree elementtree elementtree-xinclude}@anchor{2e9b}@anchor{library/xml etree elementtree xinclude-support}@anchor{2e9c}
@subsubsection XInclude support


This module provides limited support for
XInclude directives@footnote{https://www.w3.org/TR/xinclude/}, via the @code{xml.etree.ElementInclude} helper module.  This module can be used to insert subtrees and text strings into element trees, based on information in the tree.

@menu
* Example: Example<10>. 

@end menu

@node Example<10>,,,XInclude support
@anchor{library/xml etree elementtree id3}@anchor{2e9d}
@subsubsection Example


Here’s an example that demonstrates use of the XInclude module. To include an XML document in the current document, use the @code{@{http://www.w3.org/2001/XInclude@}include} element and set the @strong{parse} attribute to @code{"xml"}, and use the @strong{href} attribute to specify the document to include.

@example
<?xml version="1.0"?>
<document xmlns:xi="http://www.w3.org/2001/XInclude">
  <xi:include href="source.xml" parse="xml" />
</document>
@end example

By default, the @strong{href} attribute is treated as a file name. You can use custom loaders to override this behaviour. Also note that the standard helper does not support XPointer syntax.

To process this file, load it as usual, and pass the root element to the @ref{13e,,xml.etree.ElementTree} module:

@example
from xml.etree import ElementTree, ElementInclude

tree = ElementTree.parse("document.xml")
root = tree.getroot()

ElementInclude.include(root)
@end example

The ElementInclude module replaces the @code{@{http://www.w3.org/2001/XInclude@}include} element with the root element from the @strong{source.xml} document. The result might look something like this:

@example
<document xmlns:xi="http://www.w3.org/2001/XInclude">
  <para>This is a paragraph.</para>
</document>
@end example

If the @strong{parse} attribute is omitted, it defaults to “xml”. The href attribute is required.

To include a text document, use the @code{@{http://www.w3.org/2001/XInclude@}include} element, and set the @strong{parse} attribute to “text”:

@example
<?xml version="1.0"?>
<document xmlns:xi="http://www.w3.org/2001/XInclude">
  Copyright (c) <xi:include href="year.txt" parse="text" />.
</document>
@end example

The result might look something like:

@example
<document xmlns:xi="http://www.w3.org/2001/XInclude">
  Copyright (c) 2003.
</document>
@end example

@node Reference<3>,,XInclude support,xml etree ElementTree — The ElementTree XML API
@anchor{library/xml etree elementtree id4}@anchor{2e9e}
@subsubsection Reference


@menu
* Functions: Functions<8>. 
* Element Objects:: 
* ElementTree Objects:: 
* QName Objects:: 
* TreeBuilder Objects:: 
* XMLParser Objects:: 
* XMLPullParser Objects:: 
* Exceptions: Exceptions<16>. 

@end menu

@node Functions<8>,Element Objects,,Reference<3>
@anchor{library/xml etree elementtree elementinclude-functions}@anchor{2e9f}@anchor{library/xml etree elementtree id5}@anchor{2ea0}
@subsubsection Functions


@geindex xml.etree.ElementInclude.default_loader() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree xml etree ElementInclude default_loader}@anchor{2ea1}
@deffn {Function} xml.etree.ElementInclude.default_loader (href, parse, encoding=None)

Default loader. This default loader reads an included resource from disk.  @emph{href} is a URL.
@emph{parse} is for parse mode either “xml” or “text”.  @emph{encoding}
is an optional text encoding.  If not given, encoding is @code{utf-8}.  Returns the
expanded resource.  If the parse mode is @code{"xml"}, this is an ElementTree
instance.  If the parse mode is “text”, this is a Unicode string.  If the
loader fails, it can return None or raise an exception.
@end deffn

@geindex xml.etree.ElementInclude.include() (in module xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree xml etree ElementInclude include}@anchor{2ea2}
@deffn {Function} xml.etree.ElementInclude.include (elem, loader=None, base_url=None, max_depth=6)

This function expands XInclude directives.  @emph{elem} is the root element.  @emph{loader} is
an optional resource loader.  If omitted, it defaults to @ref{2ea1,,default_loader()}.
If given, it should be a callable that implements the same interface as
@ref{2ea1,,default_loader()}.  @emph{base_url} is base URL of the original file, to resolve
relative include file references.  @emph{max_depth} is the maximum number of recursive
inclusions.  Limited to reduce the risk of malicious content explosion. Pass a
negative value to disable the limitation.

Returns the expanded resource.  If the parse mode is
@code{"xml"}, this is an ElementTree instance.  If the parse mode is “text”,
this is a Unicode string.  If the loader fails, it can return None or
raise an exception.


New in version 3.9: The @emph{base_url} and @emph{max_depth} parameters.

@end deffn

@node Element Objects,ElementTree Objects,Functions<8>,Reference<3>
@anchor{library/xml etree elementtree element-objects}@anchor{2ea3}@anchor{library/xml etree elementtree elementtree-element-objects}@anchor{2ea4}
@subsubsection Element Objects


@geindex Element (class in xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree Element}@anchor{44e}
@deffn {Class} xml.etree.ElementTree.Element (tag, attrib=@{@}, **extra)

Element class.  This class defines the Element interface, and provides a
reference implementation of this interface.

The element name, attribute names, and attribute values can be either
bytestrings or Unicode strings.  @emph{tag} is the element name.  @emph{attrib} is
an optional dictionary, containing element attributes.  @emph{extra} contains
additional attributes, given as keyword arguments.

@geindex tag (xml.etree.ElementTree.Element attribute)
@anchor{library/xml etree elementtree xml etree ElementTree Element tag}@anchor{2ea5}
@deffn {Attribute} tag

A string identifying what kind of data this element represents (the
element type, in other words).
@end deffn

@geindex text (xml.etree.ElementTree.Element attribute)
@geindex tail (xml.etree.ElementTree.Element attribute)
@anchor{library/xml etree elementtree xml etree ElementTree Element text}@anchor{2e83}
@deffn {Attribute} text
@anchor{library/xml etree elementtree xml etree ElementTree Element tail}@anchor{2ea6}
@deffnx {Attribute} tail

These attributes can be used to hold additional data associated with
the element.  Their values are usually strings but may be any
application-specific object.  If the element is created from
an XML file, the @emph{text} attribute holds either the text between
the element’s start tag and its first child or end tag, or @code{None}, and
the @emph{tail} attribute holds either the text between the element’s
end tag and the next tag, or @code{None}.  For the XML data

@example
<a><b>1<c>2<d/>3</c></b>4</a>
@end example

the @emph{a} element has @code{None} for both @emph{text} and @emph{tail} attributes,
the @emph{b} element has @emph{text} @code{"1"} and @emph{tail} @code{"4"},
the @emph{c} element has @emph{text} @code{"2"} and @emph{tail} @code{None},
and the @emph{d} element has @emph{text} @code{None} and @emph{tail} @code{"3"}.

To collect the inner text of an element, see @ref{d79,,itertext()}, for
example @code{"".join(element.itertext())}.

Applications may store arbitrary objects in these attributes.
@end deffn

@geindex attrib (xml.etree.ElementTree.Element attribute)
@anchor{library/xml etree elementtree xml etree ElementTree Element attrib}@anchor{2ea7}
@deffn {Attribute} attrib

A dictionary containing the element’s attributes.  Note that while the
@emph{attrib} value is always a real mutable Python dictionary, an ElementTree
implementation may choose to use another internal representation, and
create the dictionary only if someone asks for it.  To take advantage of
such implementations, use the dictionary methods below whenever possible.
@end deffn

The following dictionary-like methods work on the element attributes.

@geindex clear() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element clear}@anchor{2ea8}
@deffn {Method} clear ()

Resets an element.  This function removes all subelements, clears all
attributes, and sets the text and tail attributes to @code{None}.
@end deffn

@geindex get() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element get}@anchor{2e84}
@deffn {Method} get (key, default=None)

Gets the element attribute named @emph{key}.

Returns the attribute value, or @emph{default} if the attribute was not found.
@end deffn

@geindex items() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element items}@anchor{2ea9}
@deffn {Method} items ()

Returns the element attributes as a sequence of (name, value) pairs.  The
attributes are returned in an arbitrary order.
@end deffn

@geindex keys() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element keys}@anchor{2eaa}
@deffn {Method} keys ()

Returns the elements attribute names as a list.  The names are returned
in an arbitrary order.
@end deffn

@geindex set() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element set}@anchor{2e86}
@deffn {Method} set (key, value)

Set the attribute @emph{key} on the element to @emph{value}.
@end deffn

The following methods work on the element’s children (subelements).

@geindex append() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element append}@anchor{2e87}
@deffn {Method} append (subelement)

Adds the element @emph{subelement} to the end of this element’s internal list
of subelements.  Raises @ref{162,,TypeError} if @emph{subelement} is not an
@ref{44e,,Element}.
@end deffn

@geindex extend() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element extend}@anchor{d77}
@deffn {Method} extend (subelements)

Appends @emph{subelements} from a sequence object with zero or more elements.
Raises @ref{162,,TypeError} if a subelement is not an @ref{44e,,Element}.


New in version 3.2.

@end deffn

@geindex find() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element find}@anchor{2e82}
@deffn {Method} find (match, namespaces=None)

Finds the first subelement matching @emph{match}.  @emph{match} may be a tag name
or a @ref{6e0,,path}.  Returns an element instance
or @code{None}.  @emph{namespaces} is an optional mapping from namespace prefix
to full name.  Pass @code{'@w{'}} as prefix to move all unprefixed tag names
in the expression into the given namespace.
@end deffn

@geindex findall() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element findall}@anchor{2e81}
@deffn {Method} findall (match, namespaces=None)

Finds all matching subelements, by tag name or
@ref{6e0,,path}.  Returns a list containing all matching
elements in document order.  @emph{namespaces} is an optional mapping from
namespace prefix to full name.  Pass @code{'@w{'}} as prefix to move all
unprefixed tag names in the expression into the given namespace.
@end deffn

@geindex findtext() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element findtext}@anchor{2eab}
@deffn {Method} findtext (match, default=None, namespaces=None)

Finds text for the first subelement matching @emph{match}.  @emph{match} may be
a tag name or a @ref{6e0,,path}.  Returns the text content
of the first matching element, or @emph{default} if no element was found.
Note that if the matching element has no text content an empty string
is returned. @emph{namespaces} is an optional mapping from namespace prefix
to full name.  Pass @code{'@w{'}} as prefix to move all unprefixed tag names
in the expression into the given namespace.
@end deffn

@geindex insert() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element insert}@anchor{2eac}
@deffn {Method} insert (index, subelement)

Inserts @emph{subelement} at the given position in this element.  Raises
@ref{162,,TypeError} if @emph{subelement} is not an @ref{44e,,Element}.
@end deffn

@geindex iter() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element iter}@anchor{ee9}
@deffn {Method} iter (tag=None)

Creates a tree @ref{11c3,,iterator} with the current element as the root.
The iterator iterates over this element and all elements below it, in
document (depth first) order.  If @emph{tag} is not @code{None} or @code{'*'}, only
elements whose tag equals @emph{tag} are returned from the iterator.  If the
tree structure is modified during iteration, the result is undefined.


New in version 3.2.

@end deffn

@geindex iterfind() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element iterfind}@anchor{d78}
@deffn {Method} iterfind (match, namespaces=None)

Finds all matching subelements, by tag name or
@ref{6e0,,path}.  Returns an iterable yielding all
matching elements in document order. @emph{namespaces} is an optional mapping
from namespace prefix to full name.


New in version 3.2.

@end deffn

@geindex itertext() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element itertext}@anchor{d79}
@deffn {Method} itertext ()

Creates a text iterator.  The iterator loops over this element and all
subelements, in document order, and returns all inner text.


New in version 3.2.

@end deffn

@geindex makeelement() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element makeelement}@anchor{2ead}
@deffn {Method} makeelement (tag, attrib)

Creates a new element object of the same type as this element.  Do not
call this method, use the @ref{2e8a,,SubElement()} factory function instead.
@end deffn

@geindex remove() (xml.etree.ElementTree.Element method)
@anchor{library/xml etree elementtree xml etree ElementTree Element remove}@anchor{2e88}
@deffn {Method} remove (subelement)

Removes @emph{subelement} from the element.  Unlike the find* methods this
method compares elements based on the instance identity, not on tag value
or contents.
@end deffn

@ref{44e,,Element} objects also support the following sequence type methods
for working with subelements: @ref{17d9,,__delitem__()},
@ref{179c,,__getitem__()}, @ref{17d8,,__setitem__()},
@ref{17ad,,__len__()}.

Caution: Elements with no subelements will test as @code{False}.  This behavior
will change in future versions.  Use specific @code{len(elem)} or @code{elem is
None} test instead.

@example
element = root.find('foo')

if not element:  # careful!
    print("element not found, or element has no subelements")

if element is None:
    print("element not found")
@end example

Prior to Python 3.8, the serialisation order of the XML attributes of
elements was artificially made predictable by sorting the attributes by
their name. Based on the now guaranteed ordering of dicts, this arbitrary
reordering was removed in Python 3.8 to preserve the order in which
attributes were originally parsed or created by user code.

In general, user code should try not to depend on a specific ordering of
attributes, given that the XML Information Set@footnote{https://www.w3.org/TR/xml-infoset/} explicitly excludes the attribute
order from conveying information. Code should be prepared to deal with
any ordering on input. In cases where deterministic XML output is required,
e.g. for cryptographic signing or test data sets, canonical serialisation
is available with the @ref{2e92,,canonicalize()} function.

In cases where canonical output is not applicable but a specific attribute
order is still desirable on output, code should aim for creating the
attributes directly in the desired order, to avoid perceptual mismatches
for readers of the code. In cases where this is difficult to achieve, a
recipe like the following can be applied prior to serialisation to enforce
an order independently from the Element creation:

@example
def reorder_attributes(root):
    for el in root.iter():
        attrib = el.attrib
        if len(attrib) > 1:
            # adjust attribute order, e.g. by sorting
            attribs = sorted(attrib.items())
            attrib.clear()
            attrib.update(attribs)
@end example
@end deffn

@node ElementTree Objects,QName Objects,Element Objects,Reference<3>
@anchor{library/xml etree elementtree elementtree-elementtree-objects}@anchor{2eae}@anchor{library/xml etree elementtree elementtree-objects}@anchor{2eaf}
@subsubsection ElementTree Objects


@geindex ElementTree (class in xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree ElementTree}@anchor{44d}
@deffn {Class} xml.etree.ElementTree.ElementTree (element=None, file=None)

ElementTree wrapper class.  This class represents an entire element
hierarchy, and adds some extra support for serialization to and from
standard XML.

@emph{element} is the root element.  The tree is initialized with the contents
of the XML @emph{file} if given.

@geindex _setroot() (xml.etree.ElementTree.ElementTree method)
@anchor{library/xml etree elementtree xml etree ElementTree ElementTree _setroot}@anchor{2eb0}
@deffn {Method} _setroot (element)

Replaces the root element for this tree.  This discards the current
contents of the tree, and replaces it with the given element.  Use with
care.  @emph{element} is an element instance.
@end deffn

@geindex find() (xml.etree.ElementTree.ElementTree method)
@anchor{library/xml etree elementtree xml etree ElementTree ElementTree find}@anchor{2eb1}
@deffn {Method} find (match, namespaces=None)

Same as @ref{2e82,,Element.find()}, starting at the root of the tree.
@end deffn

@geindex findall() (xml.etree.ElementTree.ElementTree method)
@anchor{library/xml etree elementtree xml etree ElementTree ElementTree findall}@anchor{2eb2}
@deffn {Method} findall (match, namespaces=None)

Same as @ref{2e81,,Element.findall()}, starting at the root of the tree.
@end deffn

@geindex findtext() (xml.etree.ElementTree.ElementTree method)
@anchor{library/xml etree elementtree xml etree ElementTree ElementTree findtext}@anchor{2eb3}
@deffn {Method} findtext (match, default=None, namespaces=None)

Same as @ref{2eab,,Element.findtext()}, starting at the root of the tree.
@end deffn

@geindex getroot() (xml.etree.ElementTree.ElementTree method)
@anchor{library/xml etree elementtree xml etree ElementTree ElementTree getroot}@anchor{2eb4}
@deffn {Method} getroot ()

Returns the root element for this tree.
@end deffn

@geindex iter() (xml.etree.ElementTree.ElementTree method)
@anchor{library/xml etree elementtree xml etree ElementTree ElementTree iter}@anchor{2eb5}
@deffn {Method} iter (tag=None)

Creates and returns a tree iterator for the root element.  The iterator
loops over all elements in this tree, in section order.  @emph{tag} is the tag
to look for (default is to return all elements).
@end deffn

@geindex iterfind() (xml.etree.ElementTree.ElementTree method)
@anchor{library/xml etree elementtree xml etree ElementTree ElementTree iterfind}@anchor{2eb6}
@deffn {Method} iterfind (match, namespaces=None)

Same as @ref{d78,,Element.iterfind()}, starting at the root of the tree.


New in version 3.2.

@end deffn

@geindex parse() (xml.etree.ElementTree.ElementTree method)
@anchor{library/xml etree elementtree xml etree ElementTree ElementTree parse}@anchor{2eb7}
@deffn {Method} parse (source, parser=None)

Loads an external XML section into this element tree.  @emph{source} is a file
name or @ref{d68,,file object}.  @emph{parser} is an optional parser instance.
If not given, the standard @ref{569,,XMLParser} parser is used.  Returns the
section root element.
@end deffn

@geindex write() (xml.etree.ElementTree.ElementTree method)
@anchor{library/xml etree elementtree xml etree ElementTree ElementTree write}@anchor{b7d}
@deffn {Method} write (file, encoding='us-ascii', xml_declaration=None, default_namespace=None, method='xml', *, short_empty_elements=True)

Writes the element tree to a file, as XML.  @emph{file} is a file name, or a
@ref{d68,,file object} opened for writing.  @emph{encoding} @footnote{@w{(1)} 
The encoding string included in XML output should conform to the
appropriate standards.  For example, “UTF-8” is valid, but “UTF8” is
not.  See @indicateurl{https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl}
and @indicateurl{https://www.iana.org/assignments/character-sets/character-sets.xhtml}.
} is the output
encoding (default is US-ASCII).
@emph{xml_declaration} controls if an XML declaration should be added to the
file.  Use @code{False} for never, @code{True} for always, @code{None}
for only if not US-ASCII or UTF-8 or Unicode (default is @code{None}).
@emph{default_namespace} sets the default XML namespace (for “xmlns”).
@emph{method} is either @code{"xml"}, @code{"html"} or @code{"text"} (default is
@code{"xml"}).
The keyword-only @emph{short_empty_elements} parameter controls the formatting
of elements that contain no content.  If @code{True} (the default), they are
emitted as a single self-closed tag, otherwise they are emitted as a pair
of start/end tags.

The output is either a string (@ref{2ef,,str}) or binary (@ref{2f0,,bytes}).
This is controlled by the @emph{encoding} argument.  If @emph{encoding} is
@code{"unicode"}, the output is a string; otherwise, it’s binary.  Note that
this may conflict with the type of @emph{file} if it’s an open
@ref{d68,,file object}; make sure you do not try to write a string to a
binary stream and vice versa.


New in version 3.4: The @emph{short_empty_elements} parameter.



Changed in version 3.8: The @ref{b7d,,write()} method now preserves the attribute order specified
by the user.

@end deffn
@end deffn

This is the XML file that is going to be manipulated:

@example
<html>
    <head>
        <title>Example page</title>
    </head>
    <body>
        <p>Moved to <a href="http://example.org/">example.org</a>
        or <a href="http://example.com/">example.com</a>.</p>
    </body>
</html>
@end example

Example of changing the attribute “target” of every link in first paragraph:

@example
>>> from xml.etree.ElementTree import ElementTree
>>> tree = ElementTree()
>>> tree.parse("index.xhtml")
<Element 'html' at 0xb77e6fac>
>>> p = tree.find("body/p")     # Finds first occurrence of tag p in body
>>> p
<Element 'p' at 0xb77ec26c>
>>> links = list(p.iter("a"))   # Returns list of all links
>>> links
[<Element 'a' at 0xb77ec2ac>, <Element 'a' at 0xb77ec1cc>]
>>> for i in links:             # Iterates through all found links
...     i.attrib["target"] = "blank"
>>> tree.write("output.xhtml")
@end example

@node QName Objects,TreeBuilder Objects,ElementTree Objects,Reference<3>
@anchor{library/xml etree elementtree elementtree-qname-objects}@anchor{2eb8}@anchor{library/xml etree elementtree qname-objects}@anchor{2eb9}
@subsubsection QName Objects


@geindex QName (class in xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree QName}@anchor{2eba}
@deffn {Class} xml.etree.ElementTree.QName (text_or_uri, tag=None)

QName wrapper.  This can be used to wrap a QName attribute value, in order
to get proper namespace handling on output.  @emph{text_or_uri} is a string
containing the QName value, in the form @{uri@}local, or, if the tag argument
is given, the URI part of a QName.  If @emph{tag} is given, the first argument is
interpreted as a URI, and this argument is interpreted as a local name.
@ref{2eba,,QName} instances are opaque.
@end deffn

@node TreeBuilder Objects,XMLParser Objects,QName Objects,Reference<3>
@anchor{library/xml etree elementtree elementtree-treebuilder-objects}@anchor{2ebb}@anchor{library/xml etree elementtree treebuilder-objects}@anchor{2ebc}
@subsubsection TreeBuilder Objects


@geindex TreeBuilder (class in xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree TreeBuilder}@anchor{56a}
@deffn {Class} xml.etree.ElementTree.TreeBuilder (element_factory=None, *, comment_factory=None, pi_factory=None, insert_comments=False, insert_pis=False)

Generic element structure builder.  This builder converts a sequence of
start, data, end, comment and pi method calls to a well-formed element
structure.  You can use this class to build an element structure using
a custom XML parser, or a parser for some other XML-like format.

@emph{element_factory}, when given, must be a callable accepting two positional
arguments: a tag and a dict of attributes.  It is expected to return a new
element instance.

The @emph{comment_factory} and @emph{pi_factory} functions, when given, should behave
like the @ref{2e93,,Comment()} and @ref{2e99,,ProcessingInstruction()} functions to
create comments and processing instructions.  When not given, the default
factories will be used.  When @emph{insert_comments} and/or @emph{insert_pis} is true,
comments/pis will be inserted into the tree if they appear within the root
element (but not outside of it).

@geindex close() (xml.etree.ElementTree.TreeBuilder method)
@anchor{library/xml etree elementtree xml etree ElementTree TreeBuilder close}@anchor{2ebd}
@deffn {Method} close ()

Flushes the builder buffers, and returns the toplevel document
element.  Returns an @ref{44e,,Element} instance.
@end deffn

@geindex data() (xml.etree.ElementTree.TreeBuilder method)
@anchor{library/xml etree elementtree xml etree ElementTree TreeBuilder data}@anchor{2ebe}
@deffn {Method} data (data)

Adds text to the current element.  @emph{data} is a string.  This should be
either a bytestring, or a Unicode string.
@end deffn

@geindex end() (xml.etree.ElementTree.TreeBuilder method)
@anchor{library/xml etree elementtree xml etree ElementTree TreeBuilder end}@anchor{d7a}
@deffn {Method} end (tag)

Closes the current element.  @emph{tag} is the element name.  Returns the
closed element.
@end deffn

@geindex start() (xml.etree.ElementTree.TreeBuilder method)
@anchor{library/xml etree elementtree xml etree ElementTree TreeBuilder start}@anchor{1212}
@deffn {Method} start (tag, attrs)

Opens a new element.  @emph{tag} is the element name.  @emph{attrs} is a dictionary
containing element attributes.  Returns the opened element.
@end deffn

@geindex comment() (xml.etree.ElementTree.TreeBuilder method)
@anchor{library/xml etree elementtree xml etree ElementTree TreeBuilder comment}@anchor{2ebf}
@deffn {Method} comment (text)

Creates a comment with the given @emph{text}.  If @code{insert_comments} is true,
this will also add it to the tree.


New in version 3.8.

@end deffn

@geindex pi() (xml.etree.ElementTree.TreeBuilder method)
@anchor{library/xml etree elementtree xml etree ElementTree TreeBuilder pi}@anchor{2ec0}
@deffn {Method} pi (target, text)

Creates a comment with the given @emph{target} name and @emph{text}.  If
@code{insert_pis} is true, this will also add it to the tree.


New in version 3.8.

@end deffn

In addition, a custom @ref{56a,,TreeBuilder} object can provide the
following methods:

@geindex doctype() (xml.etree.ElementTree.TreeBuilder method)
@anchor{library/xml etree elementtree xml etree ElementTree TreeBuilder doctype}@anchor{5b6}
@deffn {Method} doctype (name, pubid, system)

Handles a doctype declaration.  @emph{name} is the doctype name.  @emph{pubid} is
the public identifier.  @emph{system} is the system identifier.  This method
does not exist on the default @ref{56a,,TreeBuilder} class.


New in version 3.2.

@end deffn

@geindex start_ns() (xml.etree.ElementTree.TreeBuilder method)
@anchor{library/xml etree elementtree xml etree ElementTree TreeBuilder start_ns}@anchor{2ec1}
@deffn {Method} start_ns (prefix, uri)

Is called whenever the parser encounters a new namespace declaration,
before the @code{start()} callback for the opening element that defines it.
@emph{prefix} is @code{'@w{'}} for the default namespace and the declared
namespace prefix name otherwise.  @emph{uri} is the namespace URI.


New in version 3.8.

@end deffn

@geindex end_ns() (xml.etree.ElementTree.TreeBuilder method)
@anchor{library/xml etree elementtree xml etree ElementTree TreeBuilder end_ns}@anchor{2ec2}
@deffn {Method} end_ns (prefix)

Is called after the @code{end()} callback of an element that declared
a namespace prefix mapping, with the name of the @emph{prefix} that went
out of scope.


New in version 3.8.

@end deffn
@end deffn

@geindex C14NWriterTarget (class in xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree C14NWriterTarget}@anchor{2ec3}
@deffn {Class} xml.etree.ElementTree.C14NWriterTarget (write, *, with_comments=False, strip_text=False, rewrite_prefixes=False, qname_aware_tags=None, qname_aware_attrs=None, exclude_attrs=None, exclude_tags=None)

A C14N 2.0@footnote{https://www.w3.org/TR/xml-c14n2/} writer.  Arguments are the
same as for the @ref{2e92,,canonicalize()} function.  This class does not build a
tree but translates the callback events directly into a serialised form
using the @emph{write} function.


New in version 3.8.

@end deffn

@node XMLParser Objects,XMLPullParser Objects,TreeBuilder Objects,Reference<3>
@anchor{library/xml etree elementtree elementtree-xmlparser-objects}@anchor{2ec4}@anchor{library/xml etree elementtree xmlparser-objects}@anchor{2ec5}
@subsubsection XMLParser Objects


@geindex XMLParser (class in xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree XMLParser}@anchor{569}
@deffn {Class} xml.etree.ElementTree.XMLParser (*, target=None, encoding=None)

This class is the low-level building block of the module.  It uses
@ref{13f,,xml.parsers.expat} for efficient, event-based parsing of XML.  It can
be fed XML data incrementally with the @ref{2ec6,,feed()} method, and parsing
events are translated to a push API - by invoking callbacks on the @emph{target}
object.  If @emph{target} is omitted, the standard @ref{56a,,TreeBuilder} is used.
If @emph{encoding} @footnote{@w{(1)} 
The encoding string included in XML output should conform to the
appropriate standards.  For example, “UTF-8” is valid, but “UTF8” is
not.  See @indicateurl{https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl}
and @indicateurl{https://www.iana.org/assignments/character-sets/character-sets.xhtml}.
} is given, the value overrides the
encoding specified in the XML file.


Changed in version 3.8: Parameters are now @ref{5a2,,keyword-only}.
The @emph{html} argument no longer supported.


@geindex close() (xml.etree.ElementTree.XMLParser method)
@anchor{library/xml etree elementtree xml etree ElementTree XMLParser close}@anchor{2ec7}
@deffn {Method} close ()

Finishes feeding data to the parser.  Returns the result of calling the
@code{close()} method of the @emph{target} passed during construction; by default,
this is the toplevel document element.
@end deffn

@geindex feed() (xml.etree.ElementTree.XMLParser method)
@anchor{library/xml etree elementtree xml etree ElementTree XMLParser feed}@anchor{2ec6}
@deffn {Method} feed (data)

Feeds data to the parser.  @emph{data} is encoded data.
@end deffn

@ref{2ec6,,XMLParser.feed()} calls @emph{target}'s @code{start(tag, attrs_dict)} method
for each opening tag, its @code{end(tag)} method for each closing tag, and data
is processed by method @code{data(data)}.  For further supported callback
methods, see the @ref{56a,,TreeBuilder} class.  @ref{2ec7,,XMLParser.close()} calls
@emph{target}'s method @code{close()}. @ref{569,,XMLParser} can be used not only for
building a tree structure. This is an example of counting the maximum depth
of an XML file:

@example
>>> from xml.etree.ElementTree import XMLParser
>>> class MaxDepth:                     # The target object of the parser
...     maxDepth = 0
...     depth = 0
...     def start(self, tag, attrib):   # Called for each opening tag.
...         self.depth += 1
...         if self.depth > self.maxDepth:
...             self.maxDepth = self.depth
...     def end(self, tag):             # Called for each closing tag.
...         self.depth -= 1
...     def data(self, data):
...         pass            # We do not need to do anything with data.
...     def close(self):    # Called when all data has been parsed.
...         return self.maxDepth
...
>>> target = MaxDepth()
>>> parser = XMLParser(target=target)
>>> exampleXml = """
... <a>
...   <b>
...   </b>
...   <b>
...     <c>
...       <d>
...       </d>
...     </c>
...   </b>
... </a>"""
>>> parser.feed(exampleXml)
>>> parser.close()
4
@end example
@end deffn

@node XMLPullParser Objects,Exceptions<16>,XMLParser Objects,Reference<3>
@anchor{library/xml etree elementtree elementtree-xmlpullparser-objects}@anchor{2ec8}@anchor{library/xml etree elementtree xmlpullparser-objects}@anchor{2ec9}
@subsubsection XMLPullParser Objects


@geindex XMLPullParser (class in xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree XMLPullParser}@anchor{b79}
@deffn {Class} xml.etree.ElementTree.XMLPullParser (events=None)

A pull parser suitable for non-blocking applications.  Its input-side API is
similar to that of @ref{569,,XMLParser}, but instead of pushing calls to a
callback target, @ref{b79,,XMLPullParser} collects an internal list of parsing
events and lets the user read from it. @emph{events} is a sequence of events to
report back.  The supported events are the strings @code{"start"}, @code{"end"},
@code{"comment"}, @code{"pi"}, @code{"start-ns"} and @code{"end-ns"} (the “ns” events
are used to get detailed namespace information).  If @emph{events} is omitted,
only @code{"end"} events are reported.

@geindex feed() (xml.etree.ElementTree.XMLPullParser method)
@anchor{library/xml etree elementtree xml etree ElementTree XMLPullParser feed}@anchor{2e7e}
@deffn {Method} feed (data)

Feed the given bytes data to the parser.
@end deffn

@geindex close() (xml.etree.ElementTree.XMLPullParser method)
@anchor{library/xml etree elementtree xml etree ElementTree XMLPullParser close}@anchor{2eca}
@deffn {Method} close ()

Signal the parser that the data stream is terminated. Unlike
@ref{2ec7,,XMLParser.close()}, this method always returns @ref{182,,None}.
Any events not yet retrieved when the parser is closed can still be
read with @ref{2e7f,,read_events()}.
@end deffn

@geindex read_events() (xml.etree.ElementTree.XMLPullParser method)
@anchor{library/xml etree elementtree xml etree ElementTree XMLPullParser read_events}@anchor{2e7f}
@deffn {Method} read_events ()

Return an iterator over the events which have been encountered in the
data fed to the
parser.  The iterator yields @code{(event, elem)} pairs, where @emph{event} is a
string representing the type of event (e.g. @code{"end"}) and @emph{elem} is the
encountered @ref{44e,,Element} object, or other context value as follows.


@itemize *

@item 
@code{start}, @code{end}: the current Element.

@item 
@code{comment}, @code{pi}: the current comment / processing instruction

@item 
@code{start-ns}: a tuple @code{(prefix, uri)} naming the declared namespace
mapping.

@item 
@code{end-ns}: @ref{182,,None} (this may change in a future version)
@end itemize

Events provided in a previous call to @ref{2e7f,,read_events()} will not be
yielded again.  Events are consumed from the internal queue only when
they are retrieved from the iterator, so multiple readers iterating in
parallel over iterators obtained from @ref{2e7f,,read_events()} will have
unpredictable results.
@end deffn

@cartouche
@quotation Note 
@ref{b79,,XMLPullParser} only guarantees that it has seen the “>”
character of a starting tag when it emits a “start” event, so the
attributes are defined, but the contents of the text and tail attributes
are undefined at that point.  The same applies to the element children;
they may or may not be present.

If you need a fully populated element, look for “end” events instead.
@end quotation
@end cartouche


New in version 3.4.



Changed in version 3.8: The @code{comment} and @code{pi} events were added.

@end deffn

@node Exceptions<16>,,XMLPullParser Objects,Reference<3>
@anchor{library/xml etree elementtree exceptions}@anchor{2ecb}
@subsubsection Exceptions


@geindex ParseError (class in xml.etree.ElementTree)
@anchor{library/xml etree elementtree xml etree ElementTree ParseError}@anchor{e26}
@deffn {Class} xml.etree.ElementTree.ParseError

XML parse error, raised by the various parsing methods in this module when
parsing fails.  The string representation of an instance of this exception
will contain a user-friendly error message.  In addition, it will have
the following attributes available:

@geindex code (xml.etree.ElementTree.ParseError attribute)
@anchor{library/xml etree elementtree xml etree ElementTree ParseError code}@anchor{2ecc}
@deffn {Attribute} code

A numeric error code from the expat parser. See the documentation of
@ref{13f,,xml.parsers.expat} for the list of error codes and their meanings.
@end deffn

@geindex position (xml.etree.ElementTree.ParseError attribute)
@anchor{library/xml etree elementtree xml etree ElementTree ParseError position}@anchor{2ecd}
@deffn {Attribute} position

A tuple of @emph{line}, @emph{column} numbers, specifying where the error occurred.
@end deffn
@end deffn

@node xml dom — The Document Object Model API,xml dom minidom — Minimal DOM implementation,xml etree ElementTree — The ElementTree XML API,Structured Markup Processing Tools
@anchor{library/xml dom doc}@anchor{2ece}@anchor{library/xml dom module-xml dom}@anchor{13b}@anchor{library/xml dom xml-dom-the-document-object-model-api}@anchor{2ecf}
@subsection @code{xml.dom} — The Document Object Model API


@geindex module; xml.dom

@strong{Source code:} Lib/xml/dom/__init__.py@footnote{https://github.com/python/cpython/tree/main/Lib/xml/dom/__init__.py}


__________________________________________________________________


The Document Object Model, or “DOM,” is a cross-language API from the World Wide
Web Consortium (W3C) for accessing and modifying XML documents.  A DOM
implementation presents an XML document as a tree structure, or allows client
code to build such a structure from scratch.  It then gives access to the
structure through a set of objects which provided well-known interfaces.

The DOM is extremely useful for random-access applications.  SAX only allows you
a view of one bit of the document at a time.  If you are looking at one SAX
element, you have no access to another.  If you are looking at a text node, you
have no access to a containing element. When you write a SAX application, you
need to keep track of your program’s position in the document somewhere in your
own code.  SAX does not do it for you.  Also, if you need to look ahead in the
XML document, you are just out of luck.

Some applications are simply impossible in an event driven model with no access
to a tree.  Of course you could build some sort of tree yourself in SAX events,
but the DOM allows you to avoid writing that code.  The DOM is a standard tree
representation for XML data.

The Document Object Model is being defined by the W3C in stages, or “levels” in
their terminology.  The Python mapping of the API is substantially based on the
DOM Level 2 recommendation.

@c What if your needs are somewhere between SAX and the DOM?  Perhaps
@c you cannot afford to load the entire tree in memory but you find the
@c SAX model somewhat cumbersome and low-level.  There is also a module
@c called xml.dom.pulldom that allows you to build trees of only the
@c parts of a document that you need structured access to.  It also has
@c features that allow you to find your way around the DOM.
@c See http://www.prescod.net/python/pulldom

DOM applications typically start by parsing some XML into a DOM.  How this is
accomplished is not covered at all by DOM Level 1, and Level 2 provides only
limited improvements: There is a @code{DOMImplementation} object class which
provides access to @code{Document} creation methods, but no way to access an
XML reader/parser/Document builder in an implementation-independent way. There
is also no well-defined way to access these methods without an existing
@code{Document} object.  In Python, each DOM implementation will provide a
function @ref{2ed0,,getDOMImplementation()}. DOM Level 3 adds a Load/Store
specification, which defines an interface to the reader, but this is not yet
available in the Python standard library.

Once you have a DOM document object, you can access the parts of your XML
document through its properties and methods.  These properties are defined in
the DOM specification; this portion of the reference manual describes the
interpretation of the specification in Python.

The specification provided by the W3C defines the DOM API for Java, ECMAScript,
and OMG IDL.  The Python mapping defined here is based in large part on the IDL
version of the specification, but strict compliance is not required (though
implementations are free to support the strict mapping from IDL).  See section
@ref{2ed1,,Conformance} for a detailed discussion of mapping requirements.


@subsubheading See also



@table @asis

@item Document Object Model (DOM) Level 2 Specification@footnote{https://www.w3.org/TR/2000/REC-DOM-Level-2-Core-20001113/}

The W3C recommendation upon which the Python DOM API is based.

@item Document Object Model (DOM) Level 1 Specification@footnote{https://www.w3.org/TR/REC-DOM-Level-1/}

The W3C recommendation for the DOM supported by @ref{13c,,xml.dom.minidom}.

@item Python Language Mapping Specification@footnote{https://www.omg.org/spec/PYTH/1.2/PDF}

This specifies the mapping from OMG IDL to Python.
@end table


@menu
* Module Contents: Module Contents<4>. 
* Objects in the DOM:: 
* Conformance:: 

@end menu

@node Module Contents<4>,Objects in the DOM,,xml dom — The Document Object Model API
@anchor{library/xml dom module-contents}@anchor{2ed2}
@subsubsection Module Contents


The @ref{13b,,xml.dom} contains the following functions:

@geindex registerDOMImplementation() (in module xml.dom)
@anchor{library/xml dom xml dom registerDOMImplementation}@anchor{2ed3}
@deffn {Function} xml.dom.registerDOMImplementation (name, factory)

Register the @emph{factory} function with the name @emph{name}.  The factory function
should return an object which implements the @code{DOMImplementation}
interface.  The factory function can return the same object every time, or a new
one for each call, as appropriate for the specific implementation (e.g. if that
implementation supports some customization).
@end deffn

@geindex getDOMImplementation() (in module xml.dom)
@anchor{library/xml dom xml dom getDOMImplementation}@anchor{2ed0}
@deffn {Function} xml.dom.getDOMImplementation (name=None, features=())

Return a suitable DOM implementation. The @emph{name} is either well-known, the
module name of a DOM implementation, or @code{None}. If it is not @code{None}, imports
the corresponding module and returns a @code{DOMImplementation} object if the
import succeeds.  If no name is given, and if the environment variable
@geindex PYTHON_DOM
@geindex environment variable; PYTHON_DOM
@code{PYTHON_DOM} is set, this variable is used to find the implementation.

If name is not given, this examines the available implementations to find one
with the required feature set.  If no implementation can be found, raise an
@ref{3b8,,ImportError}.  The features list must be a sequence of @code{(feature,
version)} pairs which are passed to the @code{hasFeature()} method on available
@code{DOMImplementation} objects.
@end deffn

Some convenience constants are also provided:

@geindex EMPTY_NAMESPACE (in module xml.dom)
@anchor{library/xml dom xml dom EMPTY_NAMESPACE}@anchor{2ed4}
@deffn {Data} xml.dom.EMPTY_NAMESPACE

The value used to indicate that no namespace is associated with a node in the
DOM.  This is typically found as the @code{namespaceURI} of a node, or used as
the @emph{namespaceURI} parameter to a namespaces-specific method.
@end deffn

@geindex XML_NAMESPACE (in module xml.dom)
@anchor{library/xml dom xml dom XML_NAMESPACE}@anchor{2ed5}
@deffn {Data} xml.dom.XML_NAMESPACE

The namespace URI associated with the reserved prefix @code{xml}, as defined by
Namespaces in XML@footnote{https://www.w3.org/TR/REC-xml-names/} (section 4).
@end deffn

@geindex XMLNS_NAMESPACE (in module xml.dom)
@anchor{library/xml dom xml dom XMLNS_NAMESPACE}@anchor{2ed6}
@deffn {Data} xml.dom.XMLNS_NAMESPACE

The namespace URI for namespace declarations, as defined by Document Object Model (DOM) Level 2 Core Specification@footnote{https://www.w3.org/TR/DOM-Level-2-Core/core.html} (section 1.1.8).
@end deffn

@geindex XHTML_NAMESPACE (in module xml.dom)
@anchor{library/xml dom xml dom XHTML_NAMESPACE}@anchor{2ed7}
@deffn {Data} xml.dom.XHTML_NAMESPACE

The URI of the XHTML namespace as defined by XHTML 1.0: The Extensible HyperText Markup Language@footnote{https://www.w3.org/TR/xhtml1/} (section 3.1.1).
@end deffn

In addition, @ref{13b,,xml.dom} contains a base @code{Node} class and the DOM
exception classes.  The @code{Node} class provided by this module does not
implement any of the methods or attributes defined by the DOM specification;
concrete DOM implementations must provide those.  The @code{Node} class
provided as part of this module does provide the constants used for the
@code{nodeType} attribute on concrete @code{Node} objects; they are located
within the class rather than at the module level to conform with the DOM
specifications.

@c Should the Node documentation go here?

@node Objects in the DOM,Conformance,Module Contents<4>,xml dom — The Document Object Model API
@anchor{library/xml dom dom-objects}@anchor{2ed8}@anchor{library/xml dom objects-in-the-dom}@anchor{2ed9}
@subsubsection Objects in the DOM


The definitive documentation for the DOM is the DOM specification from the W3C.

Note that DOM attributes may also be manipulated as nodes instead of as simple
strings.  It is fairly rare that you must do this, however, so this usage is not
yet documented.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Interface

@tab

Section

@tab

Purpose

@item

@code{DOMImplementation}

@tab

@ref{2eda,,DOMImplementation Objects}

@tab

Interface to the underlying
implementation.

@item

@code{Node}

@tab

@ref{2edb,,Node Objects}

@tab

Base interface for most objects
in a document.

@item

@code{NodeList}

@tab

@ref{2edc,,NodeList Objects}

@tab

Interface for a sequence of
nodes.

@item

@code{DocumentType}

@tab

@ref{2edd,,DocumentType Objects}

@tab

Information about the
declarations needed to process
a document.

@item

@code{Document}

@tab

@ref{2ede,,Document Objects}

@tab

Object which represents an
entire document.

@item

@code{Element}

@tab

@ref{2edf,,Element Objects}

@tab

Element nodes in the document
hierarchy.

@item

@code{Attr}

@tab

@ref{2ee0,,Attr Objects}

@tab

Attribute value nodes on
element nodes.

@item

@code{Comment}

@tab

@ref{2ee1,,Comment Objects}

@tab

Representation of comments in
the source document.

@item

@code{Text}

@tab

@ref{2ee2,,Text and CDATASection Objects}

@tab

Nodes containing textual
content from the document.

@item

@code{ProcessingInstruction}

@tab

@ref{2ee3,,ProcessingInstruction Objects}

@tab

Processing instruction
representation.

@end multitable


An additional section describes the exceptions defined for working with the DOM
in Python.

@menu
* DOMImplementation Objects:: 
* Node Objects:: 
* NodeList Objects:: 
* DocumentType Objects:: 
* Document Objects:: 
* Element Objects: Element Objects<2>. 
* Attr Objects:: 
* NamedNodeMap Objects:: 
* Comment Objects:: 
* Text and CDATASection Objects:: 
* ProcessingInstruction Objects:: 
* Exceptions: Exceptions<17>. 

@end menu

@node DOMImplementation Objects,Node Objects,,Objects in the DOM
@anchor{library/xml dom dom-implementation-objects}@anchor{2eda}@anchor{library/xml dom domimplementation-objects}@anchor{2ee4}
@subsubsection DOMImplementation Objects


The @code{DOMImplementation} interface provides a way for applications to
determine the availability of particular features in the DOM they are using.
DOM Level 2 added the ability to create new @code{Document} and
@code{DocumentType} objects using the @code{DOMImplementation} as well.

@geindex hasFeature() (xml.dom.DOMImplementation method)
@anchor{library/xml dom xml dom DOMImplementation hasFeature}@anchor{2ee5}
@deffn {Method} DOMImplementation.hasFeature (feature, version)

Return @code{True} if the feature identified by the pair of strings @emph{feature} and
@emph{version} is implemented.
@end deffn

@geindex createDocument() (xml.dom.DOMImplementation method)
@anchor{library/xml dom xml dom DOMImplementation createDocument}@anchor{2ee6}
@deffn {Method} DOMImplementation.createDocument (namespaceUri, qualifiedName, doctype)

Return a new @code{Document} object (the root of the DOM), with a child
@code{Element} object having the given @emph{namespaceUri} and @emph{qualifiedName}. The
@emph{doctype} must be a @code{DocumentType} object created by
@ref{2ee7,,createDocumentType()}, or @code{None}. In the Python DOM API, the first two
arguments can also be @code{None} in order to indicate that no @code{Element}
child is to be created.
@end deffn

@geindex createDocumentType() (xml.dom.DOMImplementation method)
@anchor{library/xml dom xml dom DOMImplementation createDocumentType}@anchor{2ee7}
@deffn {Method} DOMImplementation.createDocumentType (qualifiedName, publicId, systemId)

Return a new @code{DocumentType} object that encapsulates the given
@emph{qualifiedName}, @emph{publicId}, and @emph{systemId} strings, representing the
information contained in an XML document type declaration.
@end deffn

@node Node Objects,NodeList Objects,DOMImplementation Objects,Objects in the DOM
@anchor{library/xml dom dom-node-objects}@anchor{2edb}@anchor{library/xml dom node-objects}@anchor{2ee8}
@subsubsection Node Objects


All of the components of an XML document are subclasses of @code{Node}.

@geindex nodeType (xml.dom.Node attribute)
@anchor{library/xml dom xml dom Node nodeType}@anchor{2ee9}
@deffn {Attribute} Node.nodeType

An integer representing the node type.  Symbolic constants for the types are on
the @code{Node} object: @code{ELEMENT_NODE}, @code{ATTRIBUTE_NODE},
@code{TEXT_NODE}, @code{CDATA_SECTION_NODE}, @code{ENTITY_NODE},
@code{PROCESSING_INSTRUCTION_NODE}, @code{COMMENT_NODE},
@code{DOCUMENT_NODE}, @code{DOCUMENT_TYPE_NODE}, @code{NOTATION_NODE}.
This is a read-only attribute.
@end deffn

@geindex parentNode (xml.dom.Node attribute)
@anchor{library/xml dom xml dom Node parentNode}@anchor{2eea}
@deffn {Attribute} Node.parentNode

The parent of the current node, or @code{None} for the document node. The value is
always a @code{Node} object or @code{None}.  For @code{Element} nodes, this
will be the parent element, except for the root element, in which case it will
be the @code{Document} object. For @code{Attr} nodes, this is always
@code{None}. This is a read-only attribute.
@end deffn

@geindex attributes (xml.dom.Node attribute)
@anchor{library/xml dom xml dom Node attributes}@anchor{2eeb}
@deffn {Attribute} Node.attributes

A @code{NamedNodeMap} of attribute objects.  Only elements have actual values
for this; others provide @code{None} for this attribute. This is a read-only
attribute.
@end deffn

@geindex previousSibling (xml.dom.Node attribute)
@anchor{library/xml dom xml dom Node previousSibling}@anchor{2eec}
@deffn {Attribute} Node.previousSibling

The node that immediately precedes this one with the same parent.  For
instance the element with an end-tag that comes just before the @emph{self}
element’s start-tag.  Of course, XML documents are made up of more than just
elements so the previous sibling could be text, a comment, or something else.
If this node is the first child of the parent, this attribute will be
@code{None}. This is a read-only attribute.
@end deffn

@geindex nextSibling (xml.dom.Node attribute)
@anchor{library/xml dom xml dom Node nextSibling}@anchor{2eed}
@deffn {Attribute} Node.nextSibling

The node that immediately follows this one with the same parent.  See also
@ref{2eec,,previousSibling}.  If this is the last child of the parent, this
attribute will be @code{None}. This is a read-only attribute.
@end deffn

@geindex childNodes (xml.dom.Node attribute)
@anchor{library/xml dom xml dom Node childNodes}@anchor{2eee}
@deffn {Attribute} Node.childNodes

A list of nodes contained within this node. This is a read-only attribute.
@end deffn

@geindex firstChild (xml.dom.Node attribute)
@anchor{library/xml dom xml dom Node firstChild}@anchor{2eef}
@deffn {Attribute} Node.firstChild

The first child of the node, if there are any, or @code{None}. This is a read-only
attribute.
@end deffn

@geindex lastChild (xml.dom.Node attribute)
@anchor{library/xml dom xml dom Node lastChild}@anchor{2ef0}
@deffn {Attribute} Node.lastChild

The last child of the node, if there are any, or @code{None}. This is a read-only
attribute.
@end deffn

@geindex localName (xml.dom.Node attribute)
@anchor{library/xml dom xml dom Node localName}@anchor{2ef1}
@deffn {Attribute} Node.localName

The part of the @code{tagName} following the colon if there is one, else the
entire @code{tagName}.  The value is a string.
@end deffn

@geindex prefix (xml.dom.Node attribute)
@anchor{library/xml dom xml dom Node prefix}@anchor{2ef2}
@deffn {Attribute} Node.prefix

The part of the @code{tagName} preceding the colon if there is one, else the
empty string.  The value is a string, or @code{None}.
@end deffn

@geindex namespaceURI (xml.dom.Node attribute)
@anchor{library/xml dom xml dom Node namespaceURI}@anchor{2ef3}
@deffn {Attribute} Node.namespaceURI

The namespace associated with the element name.  This will be a string or
@code{None}.  This is a read-only attribute.
@end deffn

@geindex nodeName (xml.dom.Node attribute)
@anchor{library/xml dom xml dom Node nodeName}@anchor{2ef4}
@deffn {Attribute} Node.nodeName

This has a different meaning for each node type; see the DOM specification for
details.  You can always get the information you would get here from another
property such as the @code{tagName} property for elements or the @code{name}
property for attributes. For all node types, the value of this attribute will be
either a string or @code{None}.  This is a read-only attribute.
@end deffn

@geindex nodeValue (xml.dom.Node attribute)
@anchor{library/xml dom xml dom Node nodeValue}@anchor{2ef5}
@deffn {Attribute} Node.nodeValue

This has a different meaning for each node type; see the DOM specification for
details.  The situation is similar to that with @ref{2ef4,,nodeName}.  The value is
a string or @code{None}.
@end deffn

@geindex hasAttributes() (xml.dom.Node method)
@anchor{library/xml dom xml dom Node hasAttributes}@anchor{2ef6}
@deffn {Method} Node.hasAttributes ()

Return @code{True} if the node has any attributes.
@end deffn

@geindex hasChildNodes() (xml.dom.Node method)
@anchor{library/xml dom xml dom Node hasChildNodes}@anchor{2ef7}
@deffn {Method} Node.hasChildNodes ()

Return @code{True} if the node has any child nodes.
@end deffn

@geindex isSameNode() (xml.dom.Node method)
@anchor{library/xml dom xml dom Node isSameNode}@anchor{2ef8}
@deffn {Method} Node.isSameNode (other)

Return @code{True} if @emph{other} refers to the same node as this node. This is especially
useful for DOM implementations which use any sort of proxy architecture (because
more than one object can refer to the same node).

@cartouche
@quotation Note 
This is based on a proposed DOM Level 3 API which is still in the “working
draft” stage, but this particular interface appears uncontroversial.  Changes
from the W3C will not necessarily affect this method in the Python DOM interface
(though any new W3C API for this would also be supported).
@end quotation
@end cartouche
@end deffn

@geindex appendChild() (xml.dom.Node method)
@anchor{library/xml dom xml dom Node appendChild}@anchor{2ef9}
@deffn {Method} Node.appendChild (newChild)

Add a new child node to this node at the end of the list of
children, returning @emph{newChild}. If the node was already in
the tree, it is removed first.
@end deffn

@geindex insertBefore() (xml.dom.Node method)
@anchor{library/xml dom xml dom Node insertBefore}@anchor{2efa}
@deffn {Method} Node.insertBefore (newChild, refChild)

Insert a new child node before an existing child.  It must be the case that
@emph{refChild} is a child of this node; if not, @ref{17c,,ValueError} is raised.
@emph{newChild} is returned. If @emph{refChild} is @code{None}, it inserts @emph{newChild} at the
end of the children’s list.
@end deffn

@geindex removeChild() (xml.dom.Node method)
@anchor{library/xml dom xml dom Node removeChild}@anchor{2efb}
@deffn {Method} Node.removeChild (oldChild)

Remove a child node.  @emph{oldChild} must be a child of this node; if not,
@ref{17c,,ValueError} is raised.  @emph{oldChild} is returned on success.  If @emph{oldChild}
will not be used further, its @code{unlink()} method should be called.
@end deffn

@geindex replaceChild() (xml.dom.Node method)
@anchor{library/xml dom xml dom Node replaceChild}@anchor{2efc}
@deffn {Method} Node.replaceChild (newChild, oldChild)

Replace an existing node with a new node. It must be the case that  @emph{oldChild}
is a child of this node; if not, @ref{17c,,ValueError} is raised.
@end deffn

@geindex normalize() (xml.dom.Node method)
@anchor{library/xml dom xml dom Node normalize}@anchor{2efd}
@deffn {Method} Node.normalize ()

Join adjacent text nodes so that all stretches of text are stored as single
@code{Text} instances.  This simplifies processing text from a DOM tree for
many applications.
@end deffn

@geindex cloneNode() (xml.dom.Node method)
@anchor{library/xml dom xml dom Node cloneNode}@anchor{2efe}
@deffn {Method} Node.cloneNode (deep)

Clone this node.  Setting @emph{deep} means to clone all child nodes as well.  This
returns the clone.
@end deffn

@node NodeList Objects,DocumentType Objects,Node Objects,Objects in the DOM
@anchor{library/xml dom dom-nodelist-objects}@anchor{2edc}@anchor{library/xml dom nodelist-objects}@anchor{2eff}
@subsubsection NodeList Objects


A @code{NodeList} represents a sequence of nodes.  These objects are used in
two ways in the DOM Core recommendation:  an @code{Element} object provides
one as its list of child nodes, and the @code{getElementsByTagName()} and
@code{getElementsByTagNameNS()} methods of @code{Node} return objects with this
interface to represent query results.

The DOM Level 2 recommendation defines one method and one attribute for these
objects:

@geindex item() (xml.dom.NodeList method)
@anchor{library/xml dom xml dom NodeList item}@anchor{2f00}
@deffn {Method} NodeList.item (i)

Return the @emph{i}’th item from the sequence, if there is one, or @code{None}.  The
index @emph{i} is not allowed to be less than zero or greater than or equal to the
length of the sequence.
@end deffn

@geindex length (xml.dom.NodeList attribute)
@anchor{library/xml dom xml dom NodeList length}@anchor{2f01}
@deffn {Attribute} NodeList.length

The number of nodes in the sequence.
@end deffn

In addition, the Python DOM interface requires that some additional support is
provided to allow @code{NodeList} objects to be used as Python sequences.  All
@code{NodeList} implementations must include support for
@ref{17ad,,__len__()} and
@ref{179c,,__getitem__()}; this allows iteration over the @code{NodeList} in
@ref{161,,for} statements and proper support for the @ref{4a6,,len()} built-in
function.

If a DOM implementation supports modification of the document, the
@code{NodeList} implementation must also support the
@ref{17d8,,__setitem__()} and @ref{17d9,,__delitem__()} methods.

@node DocumentType Objects,Document Objects,NodeList Objects,Objects in the DOM
@anchor{library/xml dom documenttype-objects}@anchor{2f02}@anchor{library/xml dom dom-documenttype-objects}@anchor{2edd}
@subsubsection DocumentType Objects


Information about the notations and entities declared by a document (including
the external subset if the parser uses it and can provide the information) is
available from a @code{DocumentType} object.  The @code{DocumentType} for a
document is available from the @code{Document} object’s @code{doctype}
attribute; if there is no @code{DOCTYPE} declaration for the document, the
document’s @code{doctype} attribute will be set to @code{None} instead of an
instance of this interface.

@code{DocumentType} is a specialization of @code{Node}, and adds the
following attributes:

@geindex publicId (xml.dom.DocumentType attribute)
@anchor{library/xml dom xml dom DocumentType publicId}@anchor{2f03}
@deffn {Attribute} DocumentType.publicId

The public identifier for the external subset of the document type definition.
This will be a string or @code{None}.
@end deffn

@geindex systemId (xml.dom.DocumentType attribute)
@anchor{library/xml dom xml dom DocumentType systemId}@anchor{2f04}
@deffn {Attribute} DocumentType.systemId

The system identifier for the external subset of the document type definition.
This will be a URI as a string, or @code{None}.
@end deffn

@geindex internalSubset (xml.dom.DocumentType attribute)
@anchor{library/xml dom xml dom DocumentType internalSubset}@anchor{2f05}
@deffn {Attribute} DocumentType.internalSubset

A string giving the complete internal subset from the document. This does not
include the brackets which enclose the subset.  If the document has no internal
subset, this should be @code{None}.
@end deffn

@geindex name (xml.dom.DocumentType attribute)
@anchor{library/xml dom xml dom DocumentType name}@anchor{2f06}
@deffn {Attribute} DocumentType.name

The name of the root element as given in the @code{DOCTYPE} declaration, if
present.
@end deffn

@geindex entities (xml.dom.DocumentType attribute)
@anchor{library/xml dom xml dom DocumentType entities}@anchor{2f07}
@deffn {Attribute} DocumentType.entities

This is a @code{NamedNodeMap} giving the definitions of external entities.
For entity names defined more than once, only the first definition is provided
(others are ignored as required by the XML recommendation).  This may be
@code{None} if the information is not provided by the parser, or if no entities are
defined.
@end deffn

@geindex notations (xml.dom.DocumentType attribute)
@anchor{library/xml dom xml dom DocumentType notations}@anchor{2f08}
@deffn {Attribute} DocumentType.notations

This is a @code{NamedNodeMap} giving the definitions of notations. For
notation names defined more than once, only the first definition is provided
(others are ignored as required by the XML recommendation).  This may be
@code{None} if the information is not provided by the parser, or if no notations
are defined.
@end deffn

@node Document Objects,Element Objects<2>,DocumentType Objects,Objects in the DOM
@anchor{library/xml dom document-objects}@anchor{2f09}@anchor{library/xml dom dom-document-objects}@anchor{2ede}
@subsubsection Document Objects


A @code{Document} represents an entire XML document, including its constituent
elements, attributes, processing instructions, comments etc.  Remember that it
inherits properties from @code{Node}.

@geindex documentElement (xml.dom.Document attribute)
@anchor{library/xml dom xml dom Document documentElement}@anchor{2f0a}
@deffn {Attribute} Document.documentElement

The one and only root element of the document.
@end deffn

@geindex createElement() (xml.dom.Document method)
@anchor{library/xml dom xml dom Document createElement}@anchor{2f0b}
@deffn {Method} Document.createElement (tagName)

Create and return a new element node.  The element is not inserted into the
document when it is created.  You need to explicitly insert it with one of the
other methods such as @code{insertBefore()} or @code{appendChild()}.
@end deffn

@geindex createElementNS() (xml.dom.Document method)
@anchor{library/xml dom xml dom Document createElementNS}@anchor{2f0c}
@deffn {Method} Document.createElementNS (namespaceURI, tagName)

Create and return a new element with a namespace.  The @emph{tagName} may have a
prefix.  The element is not inserted into the document when it is created.  You
need to explicitly insert it with one of the other methods such as
@code{insertBefore()} or @code{appendChild()}.
@end deffn

@geindex createTextNode() (xml.dom.Document method)
@anchor{library/xml dom xml dom Document createTextNode}@anchor{2f0d}
@deffn {Method} Document.createTextNode (data)

Create and return a text node containing the data passed as a parameter.  As
with the other creation methods, this one does not insert the node into the
tree.
@end deffn

@geindex createComment() (xml.dom.Document method)
@anchor{library/xml dom xml dom Document createComment}@anchor{2f0e}
@deffn {Method} Document.createComment (data)

Create and return a comment node containing the data passed as a parameter.  As
with the other creation methods, this one does not insert the node into the
tree.
@end deffn

@geindex createProcessingInstruction() (xml.dom.Document method)
@anchor{library/xml dom xml dom Document createProcessingInstruction}@anchor{2f0f}
@deffn {Method} Document.createProcessingInstruction (target, data)

Create and return a processing instruction node containing the @emph{target} and
@emph{data} passed as parameters.  As with the other creation methods, this one does
not insert the node into the tree.
@end deffn

@geindex createAttribute() (xml.dom.Document method)
@anchor{library/xml dom xml dom Document createAttribute}@anchor{2f10}
@deffn {Method} Document.createAttribute (name)

Create and return an attribute node.  This method does not associate the
attribute node with any particular element.  You must use
@code{setAttributeNode()} on the appropriate @code{Element} object to use the
newly created attribute instance.
@end deffn

@geindex createAttributeNS() (xml.dom.Document method)
@anchor{library/xml dom xml dom Document createAttributeNS}@anchor{2f11}
@deffn {Method} Document.createAttributeNS (namespaceURI, qualifiedName)

Create and return an attribute node with a namespace.  The @emph{tagName} may have a
prefix.  This method does not associate the attribute node with any particular
element.  You must use @code{setAttributeNode()} on the appropriate
@code{Element} object to use the newly created attribute instance.
@end deffn

@geindex getElementsByTagName() (xml.dom.Document method)
@anchor{library/xml dom xml dom Document getElementsByTagName}@anchor{2f12}
@deffn {Method} Document.getElementsByTagName (tagName)

Search for all descendants (direct children, children’s children, etc.) with a
particular element type name.
@end deffn

@geindex getElementsByTagNameNS() (xml.dom.Document method)
@anchor{library/xml dom xml dom Document getElementsByTagNameNS}@anchor{2f13}
@deffn {Method} Document.getElementsByTagNameNS (namespaceURI, localName)

Search for all descendants (direct children, children’s children, etc.) with a
particular namespace URI and localname.  The localname is the part of the
namespace after the prefix.
@end deffn

@node Element Objects<2>,Attr Objects,Document Objects,Objects in the DOM
@anchor{library/xml dom dom-element-objects}@anchor{2edf}@anchor{library/xml dom element-objects}@anchor{2f14}
@subsubsection Element Objects


@code{Element} is a subclass of @code{Node}, so inherits all the attributes
of that class.

@geindex tagName (xml.dom.Element attribute)
@anchor{library/xml dom xml dom Element tagName}@anchor{2f15}
@deffn {Attribute} Element.tagName

The element type name.  In a namespace-using document it may have colons in it.
The value is a string.
@end deffn

@geindex getElementsByTagName() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element getElementsByTagName}@anchor{2f16}
@deffn {Method} Element.getElementsByTagName (tagName)

Same as equivalent method in the @code{Document} class.
@end deffn

@geindex getElementsByTagNameNS() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element getElementsByTagNameNS}@anchor{2f17}
@deffn {Method} Element.getElementsByTagNameNS (namespaceURI, localName)

Same as equivalent method in the @code{Document} class.
@end deffn

@geindex hasAttribute() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element hasAttribute}@anchor{2f18}
@deffn {Method} Element.hasAttribute (name)

Return @code{True} if the element has an attribute named by @emph{name}.
@end deffn

@geindex hasAttributeNS() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element hasAttributeNS}@anchor{2f19}
@deffn {Method} Element.hasAttributeNS (namespaceURI, localName)

Return @code{True} if the element has an attribute named by @emph{namespaceURI} and
@emph{localName}.
@end deffn

@geindex getAttribute() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element getAttribute}@anchor{2f1a}
@deffn {Method} Element.getAttribute (name)

Return the value of the attribute named by @emph{name} as a string. If no such
attribute exists, an empty string is returned, as if the attribute had no value.
@end deffn

@geindex getAttributeNode() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element getAttributeNode}@anchor{2f1b}
@deffn {Method} Element.getAttributeNode (attrname)

Return the @code{Attr} node for the attribute named by @emph{attrname}.
@end deffn

@geindex getAttributeNS() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element getAttributeNS}@anchor{2f1c}
@deffn {Method} Element.getAttributeNS (namespaceURI, localName)

Return the value of the attribute named by @emph{namespaceURI} and @emph{localName} as a
string. If no such attribute exists, an empty string is returned, as if the
attribute had no value.
@end deffn

@geindex getAttributeNodeNS() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element getAttributeNodeNS}@anchor{2f1d}
@deffn {Method} Element.getAttributeNodeNS (namespaceURI, localName)

Return an attribute value as a node, given a @emph{namespaceURI} and @emph{localName}.
@end deffn

@geindex removeAttribute() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element removeAttribute}@anchor{2f1e}
@deffn {Method} Element.removeAttribute (name)

Remove an attribute by name.  If there is no matching attribute, a
@ref{2f1f,,NotFoundErr} is raised.
@end deffn

@geindex removeAttributeNode() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element removeAttributeNode}@anchor{2f20}
@deffn {Method} Element.removeAttributeNode (oldAttr)

Remove and return @emph{oldAttr} from the attribute list, if present. If @emph{oldAttr} is
not present, @ref{2f1f,,NotFoundErr} is raised.
@end deffn

@geindex removeAttributeNS() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element removeAttributeNS}@anchor{2f21}
@deffn {Method} Element.removeAttributeNS (namespaceURI, localName)

Remove an attribute by name.  Note that it uses a localName, not a qname.  No
exception is raised if there is no matching attribute.
@end deffn

@geindex setAttribute() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element setAttribute}@anchor{2f22}
@deffn {Method} Element.setAttribute (name, value)

Set an attribute value from a string.
@end deffn

@geindex setAttributeNode() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element setAttributeNode}@anchor{2f23}
@deffn {Method} Element.setAttributeNode (newAttr)

Add a new attribute node to the element, replacing an existing attribute if
necessary if the @code{name} attribute matches.  If a replacement occurs, the
old attribute node will be returned.  If @emph{newAttr} is already in use,
@ref{2f24,,InuseAttributeErr} will be raised.
@end deffn

@geindex setAttributeNodeNS() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element setAttributeNodeNS}@anchor{2f25}
@deffn {Method} Element.setAttributeNodeNS (newAttr)

Add a new attribute node to the element, replacing an existing attribute if
necessary if the @code{namespaceURI} and @code{localName} attributes match.
If a replacement occurs, the old attribute node will be returned.  If @emph{newAttr}
is already in use, @ref{2f24,,InuseAttributeErr} will be raised.
@end deffn

@geindex setAttributeNS() (xml.dom.Element method)
@anchor{library/xml dom xml dom Element setAttributeNS}@anchor{2f26}
@deffn {Method} Element.setAttributeNS (namespaceURI, qname, value)

Set an attribute value from a string, given a @emph{namespaceURI} and a @emph{qname}.
Note that a qname is the whole attribute name.  This is different than above.
@end deffn

@node Attr Objects,NamedNodeMap Objects,Element Objects<2>,Objects in the DOM
@anchor{library/xml dom attr-objects}@anchor{2f27}@anchor{library/xml dom dom-attr-objects}@anchor{2ee0}
@subsubsection Attr Objects


@code{Attr} inherits from @code{Node}, so inherits all its attributes.

@geindex name (xml.dom.Attr attribute)
@anchor{library/xml dom xml dom Attr name}@anchor{2f28}
@deffn {Attribute} Attr.name

The attribute name.
In a namespace-using document it may include a colon.
@end deffn

@geindex localName (xml.dom.Attr attribute)
@anchor{library/xml dom xml dom Attr localName}@anchor{2f29}
@deffn {Attribute} Attr.localName

The part of the name following the colon if there is one, else the
entire name.
This is a read-only attribute.
@end deffn

@geindex prefix (xml.dom.Attr attribute)
@anchor{library/xml dom xml dom Attr prefix}@anchor{2f2a}
@deffn {Attribute} Attr.prefix

The part of the name preceding the colon if there is one, else the
empty string.
@end deffn

@geindex value (xml.dom.Attr attribute)
@anchor{library/xml dom xml dom Attr value}@anchor{2f2b}
@deffn {Attribute} Attr.value

The text value of the attribute.  This is a synonym for the
@code{nodeValue} attribute.
@end deffn

@node NamedNodeMap Objects,Comment Objects,Attr Objects,Objects in the DOM
@anchor{library/xml dom dom-attributelist-objects}@anchor{2f2c}@anchor{library/xml dom namednodemap-objects}@anchor{2f2d}
@subsubsection NamedNodeMap Objects


@code{NamedNodeMap} does @emph{not} inherit from @code{Node}.

@geindex length (xml.dom.NamedNodeMap attribute)
@anchor{library/xml dom xml dom NamedNodeMap length}@anchor{2f2e}
@deffn {Attribute} NamedNodeMap.length

The length of the attribute list.
@end deffn

@geindex item() (xml.dom.NamedNodeMap method)
@anchor{library/xml dom xml dom NamedNodeMap item}@anchor{2f2f}
@deffn {Method} NamedNodeMap.item (index)

Return an attribute with a particular index.  The order you get the attributes
in is arbitrary but will be consistent for the life of a DOM.  Each item is an
attribute node.  Get its value with the @code{value} attribute.
@end deffn

There are also experimental methods that give this class more mapping behavior.
You can use them or you can use the standardized @code{getAttribute*()} family
of methods on the @code{Element} objects.

@node Comment Objects,Text and CDATASection Objects,NamedNodeMap Objects,Objects in the DOM
@anchor{library/xml dom comment-objects}@anchor{2f30}@anchor{library/xml dom dom-comment-objects}@anchor{2ee1}
@subsubsection Comment Objects


@code{Comment} represents a comment in the XML document.  It is a subclass of
@code{Node}, but cannot have child nodes.

@geindex data (xml.dom.Comment attribute)
@anchor{library/xml dom xml dom Comment data}@anchor{2f31}
@deffn {Attribute} Comment.data

The content of the comment as a string.  The attribute contains all characters
between the leading @code{<!-}@code{-} and trailing @code{-}@code{->}, but does not
include them.
@end deffn

@node Text and CDATASection Objects,ProcessingInstruction Objects,Comment Objects,Objects in the DOM
@anchor{library/xml dom dom-text-objects}@anchor{2ee2}@anchor{library/xml dom text-and-cdatasection-objects}@anchor{2f32}
@subsubsection Text and CDATASection Objects


The @code{Text} interface represents text in the XML document.  If the parser
and DOM implementation support the DOM’s XML extension, portions of the text
enclosed in CDATA marked sections are stored in @code{CDATASection} objects.
These two interfaces are identical, but provide different values for the
@code{nodeType} attribute.

These interfaces extend the @code{Node} interface.  They cannot have child
nodes.

@geindex data (xml.dom.Text attribute)
@anchor{library/xml dom xml dom Text data}@anchor{2f33}
@deffn {Attribute} Text.data

The content of the text node as a string.
@end deffn

@cartouche
@quotation Note 
The use of a @code{CDATASection} node does not indicate that the node
represents a complete CDATA marked section, only that the content of the node
was part of a CDATA section.  A single CDATA section may be represented by more
than one node in the document tree.  There is no way to determine whether two
adjacent @code{CDATASection} nodes represent different CDATA marked sections.
@end quotation
@end cartouche

@node ProcessingInstruction Objects,Exceptions<17>,Text and CDATASection Objects,Objects in the DOM
@anchor{library/xml dom dom-pi-objects}@anchor{2ee3}@anchor{library/xml dom processinginstruction-objects}@anchor{2f34}
@subsubsection ProcessingInstruction Objects


Represents a processing instruction in the XML document; this inherits from the
@code{Node} interface and cannot have child nodes.

@geindex target (xml.dom.ProcessingInstruction attribute)
@anchor{library/xml dom xml dom ProcessingInstruction target}@anchor{2f35}
@deffn {Attribute} ProcessingInstruction.target

The content of the processing instruction up to the first whitespace character.
This is a read-only attribute.
@end deffn

@geindex data (xml.dom.ProcessingInstruction attribute)
@anchor{library/xml dom xml dom ProcessingInstruction data}@anchor{2f36}
@deffn {Attribute} ProcessingInstruction.data

The content of the processing instruction following the first whitespace
character.
@end deffn

@node Exceptions<17>,,ProcessingInstruction Objects,Objects in the DOM
@anchor{library/xml dom dom-exceptions}@anchor{2f37}@anchor{library/xml dom exceptions}@anchor{2f38}
@subsubsection Exceptions


The DOM Level 2 recommendation defines a single exception, @ref{2f39,,DOMException},
and a number of constants that allow applications to determine what sort of
error occurred. @ref{2f39,,DOMException} instances carry a @ref{19,,code} attribute
that provides the appropriate value for the specific exception.

The Python DOM interface provides the constants, but also expands the set of
exceptions so that a specific exception exists for each of the exception codes
defined by the DOM.  The implementations must raise the appropriate specific
exception, each of which carries the appropriate value for the @ref{19,,code}
attribute.

@geindex DOMException
@anchor{library/xml dom xml dom DOMException}@anchor{2f39}
@deffn {Exception} xml.dom.DOMException

Base exception class used for all specific DOM exceptions.  This exception class
cannot be directly instantiated.
@end deffn

@geindex DomstringSizeErr
@anchor{library/xml dom xml dom DomstringSizeErr}@anchor{2f3a}
@deffn {Exception} xml.dom.DomstringSizeErr

Raised when a specified range of text does not fit into a string. This is not
known to be used in the Python DOM implementations, but may be received from DOM
implementations not written in Python.
@end deffn

@geindex HierarchyRequestErr
@anchor{library/xml dom xml dom HierarchyRequestErr}@anchor{2f3b}
@deffn {Exception} xml.dom.HierarchyRequestErr

Raised when an attempt is made to insert a node where the node type is not
allowed.
@end deffn

@geindex IndexSizeErr
@anchor{library/xml dom xml dom IndexSizeErr}@anchor{2f3c}
@deffn {Exception} xml.dom.IndexSizeErr

Raised when an index or size parameter to a method is negative or exceeds the
allowed values.
@end deffn

@geindex InuseAttributeErr
@anchor{library/xml dom xml dom InuseAttributeErr}@anchor{2f24}
@deffn {Exception} xml.dom.InuseAttributeErr

Raised when an attempt is made to insert an @code{Attr} node that is already
present elsewhere in the document.
@end deffn

@geindex InvalidAccessErr
@anchor{library/xml dom xml dom InvalidAccessErr}@anchor{2f3d}
@deffn {Exception} xml.dom.InvalidAccessErr

Raised if a parameter or an operation is not supported on the underlying object.
@end deffn

@geindex InvalidCharacterErr
@anchor{library/xml dom xml dom InvalidCharacterErr}@anchor{2f3e}
@deffn {Exception} xml.dom.InvalidCharacterErr

This exception is raised when a string parameter contains a character that is
not permitted in the context it’s being used in by the XML 1.0 recommendation.
For example, attempting to create an @code{Element} node with a space in the
element type name will cause this error to be raised.
@end deffn

@geindex InvalidModificationErr
@anchor{library/xml dom xml dom InvalidModificationErr}@anchor{2f3f}
@deffn {Exception} xml.dom.InvalidModificationErr

Raised when an attempt is made to modify the type of a node.
@end deffn

@geindex InvalidStateErr
@anchor{library/xml dom xml dom InvalidStateErr}@anchor{2f40}
@deffn {Exception} xml.dom.InvalidStateErr

Raised when an attempt is made to use an object that is not defined or is no
longer usable.
@end deffn

@geindex NamespaceErr
@anchor{library/xml dom xml dom NamespaceErr}@anchor{2f41}
@deffn {Exception} xml.dom.NamespaceErr

If an attempt is made to change any object in a way that is not permitted with
regard to the Namespaces in XML@footnote{https://www.w3.org/TR/REC-xml-names/}
recommendation, this exception is raised.
@end deffn

@geindex NotFoundErr
@anchor{library/xml dom xml dom NotFoundErr}@anchor{2f1f}
@deffn {Exception} xml.dom.NotFoundErr

Exception when a node does not exist in the referenced context.  For example,
@code{NamedNodeMap.removeNamedItem()} will raise this if the node passed in does
not exist in the map.
@end deffn

@geindex NotSupportedErr
@anchor{library/xml dom xml dom NotSupportedErr}@anchor{2f42}
@deffn {Exception} xml.dom.NotSupportedErr

Raised when the implementation does not support the requested type of object or
operation.
@end deffn

@geindex NoDataAllowedErr
@anchor{library/xml dom xml dom NoDataAllowedErr}@anchor{2f43}
@deffn {Exception} xml.dom.NoDataAllowedErr

This is raised if data is specified for a node which does not support data.

@c XXX  a better explanation is needed!
@end deffn

@geindex NoModificationAllowedErr
@anchor{library/xml dom xml dom NoModificationAllowedErr}@anchor{2f44}
@deffn {Exception} xml.dom.NoModificationAllowedErr

Raised on attempts to modify an object where modifications are not allowed (such
as for read-only nodes).
@end deffn

@geindex SyntaxErr
@anchor{library/xml dom xml dom SyntaxErr}@anchor{2f45}
@deffn {Exception} xml.dom.SyntaxErr

Raised when an invalid or illegal string is specified.

@c XXX  how is this different from InvalidCharacterErr?
@end deffn

@geindex WrongDocumentErr
@anchor{library/xml dom xml dom WrongDocumentErr}@anchor{2f46}
@deffn {Exception} xml.dom.WrongDocumentErr

Raised when a node is inserted in a different document than it currently belongs
to, and the implementation does not support migrating the node from one document
to the other.
@end deffn

The exception codes defined in the DOM recommendation map to the exceptions
described above according to this table:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Exception

@item

@code{DOMSTRING_SIZE_ERR}

@tab

@ref{2f3a,,DomstringSizeErr}

@item

@code{HIERARCHY_REQUEST_ERR}

@tab

@ref{2f3b,,HierarchyRequestErr}

@item

@code{INDEX_SIZE_ERR}

@tab

@ref{2f3c,,IndexSizeErr}

@item

@code{INUSE_ATTRIBUTE_ERR}

@tab

@ref{2f24,,InuseAttributeErr}

@item

@code{INVALID_ACCESS_ERR}

@tab

@ref{2f3d,,InvalidAccessErr}

@item

@code{INVALID_CHARACTER_ERR}

@tab

@ref{2f3e,,InvalidCharacterErr}

@item

@code{INVALID_MODIFICATION_ERR}

@tab

@ref{2f3f,,InvalidModificationErr}

@item

@code{INVALID_STATE_ERR}

@tab

@ref{2f40,,InvalidStateErr}

@item

@code{NAMESPACE_ERR}

@tab

@ref{2f41,,NamespaceErr}

@item

@code{NOT_FOUND_ERR}

@tab

@ref{2f1f,,NotFoundErr}

@item

@code{NOT_SUPPORTED_ERR}

@tab

@ref{2f42,,NotSupportedErr}

@item

@code{NO_DATA_ALLOWED_ERR}

@tab

@ref{2f43,,NoDataAllowedErr}

@item

@code{NO_MODIFICATION_ALLOWED_ERR}

@tab

@ref{2f44,,NoModificationAllowedErr}

@item

@code{SYNTAX_ERR}

@tab

@ref{2f45,,SyntaxErr}

@item

@code{WRONG_DOCUMENT_ERR}

@tab

@ref{2f46,,WrongDocumentErr}

@end multitable


@node Conformance,,Objects in the DOM,xml dom — The Document Object Model API
@anchor{library/xml dom conformance}@anchor{2f47}@anchor{library/xml dom dom-conformance}@anchor{2ed1}
@subsubsection Conformance


This section describes the conformance requirements and relationships between
the Python DOM API, the W3C DOM recommendations, and the OMG IDL mapping for
Python.

@menu
* Type Mapping:: 
* Accessor Methods:: 

@end menu

@node Type Mapping,Accessor Methods,,Conformance
@anchor{library/xml dom dom-type-mapping}@anchor{2f48}@anchor{library/xml dom type-mapping}@anchor{2f49}
@subsubsection Type Mapping


The IDL types used in the DOM specification are mapped to Python types
according to the following table.


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

IDL Type

@tab

Python Type

@item

@code{boolean}

@tab

@code{bool} or @code{int}

@item

@code{int}

@tab

@code{int}

@item

@code{long int}

@tab

@code{int}

@item

@code{unsigned int}

@tab

@code{int}

@item

@code{DOMString}

@tab

@code{str} or @code{bytes}

@item

@code{null}

@tab

@code{None}

@end multitable


@node Accessor Methods,,Type Mapping,Conformance
@anchor{library/xml dom accessor-methods}@anchor{2f4a}@anchor{library/xml dom dom-accessor-methods}@anchor{2f4b}
@subsubsection Accessor Methods


The mapping from OMG IDL to Python defines accessor functions for IDL
@code{attribute} declarations in much the way the Java mapping does.
Mapping the IDL declarations

@example
readonly attribute string someValue;
         attribute string anotherValue;
@end example

yields three accessor functions:  a “get” method for @code{someValue}
(@code{_get_someValue()}), and “get” and “set” methods for @code{anotherValue}
(@code{_get_anotherValue()} and @code{_set_anotherValue()}).  The mapping, in
particular, does not require that the IDL attributes are accessible as normal
Python attributes:  @code{object.someValue} is @emph{not} required to work, and may
raise an @ref{163,,AttributeError}.

The Python DOM API, however, @emph{does} require that normal attribute access work.
This means that the typical surrogates generated by Python IDL compilers are not
likely to work, and wrapper objects may be needed on the client if the DOM
objects are accessed via CORBA. While this does require some additional
consideration for CORBA DOM clients, the implementers with experience using DOM
over CORBA from Python do not consider this a problem.  Attributes that are
declared @code{readonly} may not restrict write access in all DOM
implementations.

In the Python DOM API, accessor functions are not required.  If provided, they
should take the form defined by the Python IDL mapping, but these methods are
considered unnecessary since the attributes are accessible directly from Python.
“Set” accessors should never be provided for @code{readonly} attributes.

The IDL definitions do not fully embody the requirements of the W3C DOM API,
such as the notion of certain objects, such as the return value of
@code{getElementsByTagName()}, being “live”.  The Python DOM API does not require
implementations to enforce such requirements.

@node xml dom minidom — Minimal DOM implementation,xml dom pulldom — Support for building partial DOM trees,xml dom — The Document Object Model API,Structured Markup Processing Tools
@anchor{library/xml dom minidom doc}@anchor{2f4c}@anchor{library/xml dom minidom module-xml dom minidom}@anchor{13c}@anchor{library/xml dom minidom xml-dom-minidom-minimal-dom-implementation}@anchor{2f4d}
@subsection @code{xml.dom.minidom} — Minimal DOM implementation


@geindex module; xml.dom.minidom

@strong{Source code:} Lib/xml/dom/minidom.py@footnote{https://github.com/python/cpython/tree/main/Lib/xml/dom/minidom.py}


__________________________________________________________________


@ref{13c,,xml.dom.minidom} is a minimal implementation of the Document Object
Model interface, with an API similar to that in other languages.  It is intended
to be simpler than the full DOM and also significantly smaller.  Users who are
not already proficient with the DOM should consider using the
@ref{13e,,xml.etree.ElementTree} module for their XML processing instead.

@cartouche
@quotation Warning 
The @ref{13c,,xml.dom.minidom} module is not secure against
maliciously constructed data.  If you need to parse untrusted or
unauthenticated data see @ref{2e71,,XML vulnerabilities}.
@end quotation
@end cartouche

DOM applications typically start by parsing some XML into a DOM.  With
@ref{13c,,xml.dom.minidom}, this is done through the parse functions:

@example
from xml.dom.minidom import parse, parseString

dom1 = parse('c:\\temp\\mydata.xml')  # parse an XML file by name

datasource = open('c:\\temp\\mydata.xml')
dom2 = parse(datasource)  # parse an open file

dom3 = parseString('<myxml>Some data<empty/> some more data</myxml>')
@end example

The @ref{2f4e,,parse()} function can take either a filename or an open file object.

@geindex parse() (in module xml.dom.minidom)
@anchor{library/xml dom minidom xml dom minidom parse}@anchor{2f4e}
@deffn {Function} xml.dom.minidom.parse (filename_or_file, parser=None, bufsize=None)

Return a @code{Document} from the given input. @emph{filename_or_file} may be
either a file name, or a file-like object. @emph{parser}, if given, must be a SAX2
parser object. This function will change the document handler of the parser and
activate namespace support; other parser configuration (like setting an entity
resolver) must have been done in advance.
@end deffn

If you have XML in a string, you can use the @ref{2f4f,,parseString()} function
instead:

@geindex parseString() (in module xml.dom.minidom)
@anchor{library/xml dom minidom xml dom minidom parseString}@anchor{2f4f}
@deffn {Function} xml.dom.minidom.parseString (string, parser=None)

Return a @code{Document} that represents the @emph{string}. This method creates an
@ref{aa1,,io.StringIO} object for the string and passes that on to @ref{2f4e,,parse()}.
@end deffn

Both functions return a @code{Document} object representing the content of the
document.

What the @ref{2f4e,,parse()} and @ref{2f4f,,parseString()} functions do is connect an XML
parser with a “DOM builder” that can accept parse events from any SAX parser and
convert them into a DOM tree.  The name of the functions are perhaps misleading,
but are easy to grasp when learning the interfaces.  The parsing of the document
will be completed before these functions return; it’s simply that these
functions do not provide a parser implementation themselves.

You can also create a @code{Document} by calling a method on a “DOM
Implementation” object.  You can get this object either by calling the
@code{getDOMImplementation()} function in the @ref{13b,,xml.dom} package or the
@ref{13c,,xml.dom.minidom} module.  Once you have a @code{Document}, you
can add child nodes to it to populate the DOM:

@example
from xml.dom.minidom import getDOMImplementation

impl = getDOMImplementation()

newdoc = impl.createDocument(None, "some_tag", None)
top_element = newdoc.documentElement
text = newdoc.createTextNode('Some textual content.')
top_element.appendChild(text)
@end example

Once you have a DOM document object, you can access the parts of your XML
document through its properties and methods.  These properties are defined in
the DOM specification.  The main property of the document object is the
@code{documentElement} property.  It gives you the main element in the XML
document: the one that holds all others.  Here is an example program:

@example
dom3 = parseString("<myxml>Some data</myxml>")
assert dom3.documentElement.tagName == "myxml"
@end example

When you are finished with a DOM tree, you may optionally call the
@code{unlink()} method to encourage early cleanup of the now-unneeded
objects.  @code{unlink()} is an @ref{13c,,xml.dom.minidom}-specific
extension to the DOM API that renders the node and its descendants are
essentially useless.  Otherwise, Python’s garbage collector will
eventually take care of the objects in the tree.


@subsubheading See also



@table @asis

@item Document Object Model (DOM) Level 1 Specification@footnote{https://www.w3.org/TR/REC-DOM-Level-1/}

The W3C recommendation for the DOM supported by @ref{13c,,xml.dom.minidom}.
@end table


@menu
* DOM Objects:: 
* DOM Example:: 
* minidom and the DOM standard:: 

@end menu

@node DOM Objects,DOM Example,,xml dom minidom — Minimal DOM implementation
@anchor{library/xml dom minidom dom-objects}@anchor{2f50}@anchor{library/xml dom minidom minidom-objects}@anchor{2f51}
@subsubsection DOM Objects


The definition of the DOM API for Python is given as part of the @ref{13b,,xml.dom}
module documentation.  This section lists the differences between the API and
@ref{13c,,xml.dom.minidom}.

@geindex unlink() (xml.dom.minidom.Node method)
@anchor{library/xml dom minidom xml dom minidom Node unlink}@anchor{2f52}
@deffn {Method} Node.unlink ()

Break internal references within the DOM so that it will be garbage collected on
versions of Python without cyclic GC.  Even when cyclic GC is available, using
this can make large amounts of memory available sooner, so calling this on DOM
objects as soon as they are no longer needed is good practice.  This only needs
to be called on the @code{Document} object, but may be called on child nodes
to discard children of that node.

You can avoid calling this method explicitly by using the @ref{168,,with}
statement. The following code will automatically unlink @emph{dom} when the
@code{with} block is exited:

@example
with xml.dom.minidom.parse(datasource) as dom:
    ... # Work with dom.
@end example
@end deffn

@geindex writexml() (xml.dom.minidom.Node method)
@anchor{library/xml dom minidom xml dom minidom Node writexml}@anchor{2f53}
@deffn {Method} Node.writexml (writer, indent='@w{'}, addindent='@w{'}, newl='@w{'}, encoding=None, standalone=None)

Write XML to the writer object.  The writer receives texts but not bytes as input,
it should have a @code{write()} method which matches that of the file object
interface.  The @emph{indent} parameter is the indentation of the current node.
The @emph{addindent} parameter is the incremental indentation to use for subnodes
of the current one.  The @emph{newl} parameter specifies the string to use to
terminate newlines.

For the @code{Document} node, an additional keyword argument @emph{encoding} can
be used to specify the encoding field of the XML header.

Similarly, explicitly stating the @emph{standalone} argument causes the
standalone document declarations to be added to the prologue of the XML
document.
If the value is set to @cite{True}, @cite{standalone=”yes”} is added,
otherwise it is set to @cite{“no”}.
Not stating the argument will omit the declaration from the document.


Changed in version 3.8: The @ref{2f53,,writexml()} method now preserves the attribute order specified
by the user.



Changed in version 3.9: The @emph{standalone} parameter was added.

@end deffn

@geindex toxml() (xml.dom.minidom.Node method)
@anchor{library/xml dom minidom xml dom minidom Node toxml}@anchor{2f54}
@deffn {Method} Node.toxml (encoding=None, standalone=None)

Return a string or byte string containing the XML represented by
the DOM node.

With an explicit @emph{encoding} @footnote{@w{(1)} 
The encoding name included in the XML output should conform to
the appropriate standards. For example, “UTF-8” is valid, but
“UTF8” is not valid in an XML document’s declaration, even though
Python accepts it as an encoding name.
See @indicateurl{https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl}
and @indicateurl{https://www.iana.org/assignments/character-sets/character-sets.xhtml}.
} argument, the result is a byte
string in the specified encoding.
With no @emph{encoding} argument, the result is a Unicode string, and the
XML declaration in the resulting string does not specify an
encoding. Encoding this string in an encoding other than UTF-8 is
likely incorrect, since UTF-8 is the default encoding of XML.

The @emph{standalone} argument behaves exactly as in @ref{2f53,,writexml()}.


Changed in version 3.8: The @ref{2f54,,toxml()} method now preserves the attribute order specified
by the user.



Changed in version 3.9: The @emph{standalone} parameter was added.

@end deffn

@geindex toprettyxml() (xml.dom.minidom.Node method)
@anchor{library/xml dom minidom xml dom minidom Node toprettyxml}@anchor{2f55}
@deffn {Method} Node.toprettyxml (indent='\t', newl='\n', encoding=None, standalone=None)

Return a pretty-printed version of the document. @emph{indent} specifies the
indentation string and defaults to a tabulator; @emph{newl} specifies the string
emitted at the end of each line and defaults to @code{\n}.

The @emph{encoding} argument behaves like the corresponding argument of
@ref{2f54,,toxml()}.

The @emph{standalone} argument behaves exactly as in @ref{2f53,,writexml()}.


Changed in version 3.8: The @ref{2f55,,toprettyxml()} method now preserves the attribute order specified
by the user.



Changed in version 3.9: The @emph{standalone} parameter was added.

@end deffn

@node DOM Example,minidom and the DOM standard,DOM Objects,xml dom minidom — Minimal DOM implementation
@anchor{library/xml dom minidom dom-example}@anchor{2f56}@anchor{library/xml dom minidom id2}@anchor{2f57}
@subsubsection DOM Example


This example program is a fairly realistic example of a simple program. In this
particular case, we do not take much advantage of the flexibility of the DOM.

@example
import xml.dom.minidom

document = """\
<slideshow>
<title>Demo slideshow</title>
<slide><title>Slide title</title>
<point>This is a demo</point>
<point>Of a program for processing slides</point>
</slide>

<slide><title>Another demo slide</title>
<point>It is important</point>
<point>To have more than</point>
<point>one slide</point>
</slide>
</slideshow>
"""

dom = xml.dom.minidom.parseString(document)

def getText(nodelist):
    rc = []
    for node in nodelist:
        if node.nodeType == node.TEXT_NODE:
            rc.append(node.data)
    return '@w{'}.join(rc)

def handleSlideshow(slideshow):
    print("<html>")
    handleSlideshowTitle(slideshow.getElementsByTagName("title")[0])
    slides = slideshow.getElementsByTagName("slide")
    handleToc(slides)
    handleSlides(slides)
    print("</html>")

def handleSlides(slides):
    for slide in slides:
        handleSlide(slide)

def handleSlide(slide):
    handleSlideTitle(slide.getElementsByTagName("title")[0])
    handlePoints(slide.getElementsByTagName("point"))

def handleSlideshowTitle(title):
    print(f"<title>@{getText(title.childNodes)@}</title>")

def handleSlideTitle(title):
    print(f"<h2>@{getText(title.childNodes)@}</h2>")

def handlePoints(points):
    print("<ul>")
    for point in points:
        handlePoint(point)
    print("</ul>")

def handlePoint(point):
    print(f"<li>@{getText(point.childNodes)@}</li>")

def handleToc(slides):
    for slide in slides:
        title = slide.getElementsByTagName("title")[0]
        print(f"<p>@{getText(title.childNodes)@}</p>")

handleSlideshow(dom)
@end example

@node minidom and the DOM standard,,DOM Example,xml dom minidom — Minimal DOM implementation
@anchor{library/xml dom minidom minidom-and-dom}@anchor{2f58}@anchor{library/xml dom minidom minidom-and-the-dom-standard}@anchor{2f59}
@subsubsection minidom and the DOM standard


The @ref{13c,,xml.dom.minidom} module is essentially a DOM 1.0-compatible DOM with
some DOM 2 features (primarily namespace features).

Usage of the DOM interface in Python is straight-forward.  The following mapping
rules apply:


@itemize *

@item 
Interfaces are accessed through instance objects. Applications should not
instantiate the classes themselves; they should use the creator functions
available on the @code{Document} object. Derived interfaces support all
operations (and attributes) from the base interfaces, plus any new operations.

@item 
Operations are used as methods. Since the DOM uses only @ref{2fc,,in}
parameters, the arguments are passed in normal order (from left to right).
There are no optional arguments. @code{void} operations return @code{None}.

@item 
IDL attributes map to instance attributes. For compatibility with the OMG IDL
language mapping for Python, an attribute @code{foo} can also be accessed through
accessor methods @code{_get_foo()} and @code{_set_foo()}.  @code{readonly}
attributes must not be changed; this is not enforced at runtime.

@item 
The types @code{short int}, @code{unsigned int}, @code{unsigned long long}, and
@code{boolean} all map to Python integer objects.

@item 
The type @code{DOMString} maps to Python strings. @ref{13c,,xml.dom.minidom} supports
either bytes or strings, but will normally produce strings.
Values of type @code{DOMString} may also be @code{None} where allowed to have the IDL
@code{null} value by the DOM specification from the W3C.

@item 
@code{const} declarations map to variables in their respective scope (e.g.
@code{xml.dom.minidom.Node.PROCESSING_INSTRUCTION_NODE}); they must not be changed.

@item 
@code{DOMException} is currently not supported in @ref{13c,,xml.dom.minidom}.
Instead, @ref{13c,,xml.dom.minidom} uses standard Python exceptions such as
@ref{162,,TypeError} and @ref{163,,AttributeError}.

@item 
@code{NodeList} objects are implemented using Python’s built-in list type.
These objects provide the interface defined in the DOM specification, but with
earlier versions of Python they do not support the official API.  They are,
however, much more “Pythonic” than the interface defined in the W3C
recommendations.
@end itemize

The following interfaces have no implementation in @ref{13c,,xml.dom.minidom}:


@itemize *

@item 
@code{DOMTimeStamp}

@item 
@code{EntityReference}
@end itemize

Most of these reflect information in the XML document that is not of general
utility to most DOM users.

@node xml dom pulldom — Support for building partial DOM trees,xml sax — Support for SAX2 parsers,xml dom minidom — Minimal DOM implementation,Structured Markup Processing Tools
@anchor{library/xml dom pulldom doc}@anchor{2f5a}@anchor{library/xml dom pulldom module-xml dom pulldom}@anchor{13d}@anchor{library/xml dom pulldom xml-dom-pulldom-support-for-building-partial-dom-trees}@anchor{2f5b}
@subsection @code{xml.dom.pulldom} — Support for building partial DOM trees


@geindex module; xml.dom.pulldom

@strong{Source code:} Lib/xml/dom/pulldom.py@footnote{https://github.com/python/cpython/tree/main/Lib/xml/dom/pulldom.py}


__________________________________________________________________


The @ref{13d,,xml.dom.pulldom} module provides a “pull parser” which can also be
asked to produce DOM-accessible fragments of the document where necessary. The
basic concept involves pulling “events” from a stream of incoming XML and
processing them. In contrast to SAX which also employs an event-driven
processing model together with callbacks, the user of a pull parser is
responsible for explicitly pulling events from the stream, looping over those
events until either processing is finished or an error condition occurs.

@cartouche
@quotation Warning 
The @ref{13d,,xml.dom.pulldom} module is not secure against
maliciously constructed data.  If you need to parse untrusted or
unauthenticated data see @ref{2e71,,XML vulnerabilities}.
@end quotation
@end cartouche


Changed in version 3.7.1: The SAX parser no longer processes general external entities by default to
increase security by default. To enable processing of external entities,
pass a custom parser instance in:

@example
from xml.dom.pulldom import parse
from xml.sax import make_parser
from xml.sax.handler import feature_external_ges

parser = make_parser()
parser.setFeature(feature_external_ges, True)
parse(filename, parser=parser)
@end example


Example:

@example
from xml.dom import pulldom

doc = pulldom.parse('sales_items.xml')
for event, node in doc:
    if event == pulldom.START_ELEMENT and node.tagName == 'item':
        if int(node.getAttribute('price')) > 50:
            doc.expandNode(node)
            print(node.toxml())
@end example

@code{event} is a constant and can be one of:


@itemize *

@item 
@code{START_ELEMENT}

@item 
@code{END_ELEMENT}

@item 
@code{COMMENT}

@item 
@code{START_DOCUMENT}

@item 
@code{END_DOCUMENT}

@item 
@code{CHARACTERS}

@item 
@code{PROCESSING_INSTRUCTION}

@item 
@code{IGNORABLE_WHITESPACE}
@end itemize

@code{node} is an object of type @code{xml.dom.minidom.Document},
@code{xml.dom.minidom.Element} or @code{xml.dom.minidom.Text}.

Since the document is treated as a “flat” stream of events, the document “tree”
is implicitly traversed and the desired elements are found regardless of their
depth in the tree. In other words, one does not need to consider hierarchical
issues such as recursive searching of the document nodes, although if the
context of elements were important, one would either need to maintain some
context-related state (i.e. remembering where one is in the document at any
given point) or to make use of the @ref{2f5c,,DOMEventStream.expandNode()} method
and switch to DOM-related processing.

@geindex PullDom (class in xml.dom.pulldom)
@anchor{library/xml dom pulldom xml dom pulldom PullDom}@anchor{2f5d}
@deffn {Class} xml.dom.pulldom.PullDom (documentFactory=None)

Subclass of @ref{2f5e,,xml.sax.handler.ContentHandler}.
@end deffn

@geindex SAX2DOM (class in xml.dom.pulldom)
@anchor{library/xml dom pulldom xml dom pulldom SAX2DOM}@anchor{2f5f}
@deffn {Class} xml.dom.pulldom.SAX2DOM (documentFactory=None)

Subclass of @ref{2f5e,,xml.sax.handler.ContentHandler}.
@end deffn

@geindex parse() (in module xml.dom.pulldom)
@anchor{library/xml dom pulldom xml dom pulldom parse}@anchor{2f60}
@deffn {Function} xml.dom.pulldom.parse (stream_or_string, parser=None, bufsize=None)

Return a @ref{1ba,,DOMEventStream} from the given input. @emph{stream_or_string} may be
either a file name, or a file-like object. @emph{parser}, if given, must be an
@ref{2f61,,XMLReader} object. This function will change the
document handler of the
parser and activate namespace support; other parser configuration (like
setting an entity resolver) must have been done in advance.
@end deffn

If you have XML in a string, you can use the @ref{2f62,,parseString()} function instead:

@geindex parseString() (in module xml.dom.pulldom)
@anchor{library/xml dom pulldom xml dom pulldom parseString}@anchor{2f62}
@deffn {Function} xml.dom.pulldom.parseString (string, parser=None)

Return a @ref{1ba,,DOMEventStream} that represents the (Unicode) @emph{string}.
@end deffn

@geindex default_bufsize (in module xml.dom.pulldom)
@anchor{library/xml dom pulldom xml dom pulldom default_bufsize}@anchor{2f63}
@deffn {Data} xml.dom.pulldom.default_bufsize

Default value for the @emph{bufsize} parameter to @ref{2f60,,parse()}.

The value of this variable can be changed before calling @ref{2f60,,parse()} and
the new value will take effect.
@end deffn

@menu
* DOMEventStream Objects:: 

@end menu

@node DOMEventStream Objects,,,xml dom pulldom — Support for building partial DOM trees
@anchor{library/xml dom pulldom domeventstream-objects}@anchor{2f64}@anchor{library/xml dom pulldom id1}@anchor{2f65}
@subsubsection DOMEventStream Objects


@geindex DOMEventStream (class in xml.dom.pulldom)
@anchor{library/xml dom pulldom xml dom pulldom DOMEventStream}@anchor{1ba}
@deffn {Class} xml.dom.pulldom.DOMEventStream (stream, parser, bufsize)


Changed in version 3.11: Support for @code{__getitem__()} method has been removed.


@geindex getEvent() (xml.dom.pulldom.DOMEventStream method)
@anchor{library/xml dom pulldom xml dom pulldom DOMEventStream getEvent}@anchor{2f66}
@deffn {Method} getEvent ()

Return a tuple containing @emph{event} and the current @emph{node} as
@code{xml.dom.minidom.Document} if event equals @code{START_DOCUMENT},
@code{xml.dom.minidom.Element} if event equals @code{START_ELEMENT} or
@code{END_ELEMENT} or @code{xml.dom.minidom.Text} if event equals
@code{CHARACTERS}.
The current node does not contain information about its children, unless
@ref{2f5c,,expandNode()} is called.
@end deffn

@geindex expandNode() (xml.dom.pulldom.DOMEventStream method)
@anchor{library/xml dom pulldom xml dom pulldom DOMEventStream expandNode}@anchor{2f5c}
@deffn {Method} expandNode (node)

Expands all children of @emph{node} into @emph{node}. Example:

@example
from xml.dom import pulldom

xml = '<html><title>Foo</title> <p>Some text <div>and more</div></p> </html>'
doc = pulldom.parseString(xml)
for event, node in doc:
    if event == pulldom.START_ELEMENT and node.tagName == 'p':
        # Following statement only prints '<p/>'
        print(node.toxml())
        doc.expandNode(node)
        # Following statement prints node with all its children '<p>Some text <div>and more</div></p>'
        print(node.toxml())
@end example
@end deffn

@geindex reset() (xml.dom.pulldom.DOMEventStream method)
@anchor{library/xml dom pulldom xml dom pulldom DOMEventStream reset}@anchor{2f67}
@deffn {Method} reset ()
@end deffn
@end deffn

@node xml sax — Support for SAX2 parsers,xml sax handler — Base classes for SAX handlers,xml dom pulldom — Support for building partial DOM trees,Structured Markup Processing Tools
@anchor{library/xml sax doc}@anchor{2f68}@anchor{library/xml sax module-xml sax}@anchor{142}@anchor{library/xml sax xml-sax-support-for-sax2-parsers}@anchor{2f69}
@subsection @code{xml.sax} — Support for SAX2 parsers


@geindex module; xml.sax

@strong{Source code:} Lib/xml/sax/__init__.py@footnote{https://github.com/python/cpython/tree/main/Lib/xml/sax/__init__.py}


__________________________________________________________________


The @ref{142,,xml.sax} package provides a number of modules which implement the
Simple API for XML (SAX) interface for Python.  The package itself provides the
SAX exceptions and the convenience functions which will be most used by users of
the SAX API.

@cartouche
@quotation Warning 
The @ref{142,,xml.sax} module is not secure against maliciously
constructed data.  If you need to parse untrusted or unauthenticated data see
@ref{2e71,,XML vulnerabilities}.
@end quotation
@end cartouche


Changed in version 3.7.1: The SAX parser no longer processes general external entities by default
to increase security. Before, the parser created network connections
to fetch remote files or loaded local files from the file
system for DTD and entities. The feature can be enabled again with method
@ref{2f6a,,setFeature()} on the parser object
and argument @ref{2f6b,,feature_external_ges}.


The convenience functions are:

@geindex make_parser() (in module xml.sax)
@anchor{library/xml sax xml sax make_parser}@anchor{12ef}
@deffn {Function} xml.sax.make_parser (parser_list=[])

Create and return a SAX @ref{2f61,,XMLReader} object.  The
first parser found will
be used.  If @emph{parser_list} is provided, it must be an iterable of strings which
name modules that have a function named @code{create_parser()}.  Modules listed
in @emph{parser_list} will be used before modules in the default list of parsers.


Changed in version 3.8: The @emph{parser_list} argument can be any iterable, not just a list.

@end deffn

@geindex parse() (in module xml.sax)
@anchor{library/xml sax xml sax parse}@anchor{12af}
@deffn {Function} xml.sax.parse (filename_or_stream, handler, error_handler=handler.ErrorHandler())

Create a SAX parser and use it to parse a document.  The document, passed in as
@emph{filename_or_stream}, can be a filename or a file object.  The @emph{handler}
parameter needs to be a SAX @ref{2f5e,,ContentHandler} instance.  If
@emph{error_handler} is given, it must be a SAX @ref{2f6c,,ErrorHandler}
instance; if
omitted,  @ref{2f6d,,SAXParseException} will be raised on all errors.  There is no
return value; all work must be done by the @emph{handler} passed in.
@end deffn

@geindex parseString() (in module xml.sax)
@anchor{library/xml sax xml sax parseString}@anchor{a0d}
@deffn {Function} xml.sax.parseString (string, handler, error_handler=handler.ErrorHandler())

Similar to @ref{12af,,parse()}, but parses from a buffer @emph{string} received as a
parameter.  @emph{string} must be a @ref{2ef,,str} instance or a
@ref{87e,,bytes-like object}.


Changed in version 3.5: Added support of @ref{2ef,,str} instances.

@end deffn

A typical SAX application uses three kinds of objects: readers, handlers and
input sources.  “Reader” in this context is another term for parser, i.e. some
piece of code that reads the bytes or characters from the input source, and
produces a sequence of events. The events then get distributed to the handler
objects, i.e. the reader invokes a method on the handler.  A SAX application
must therefore obtain a reader object, create or open the input sources, create
the handlers, and connect these objects all together.  As the final step of
preparation, the reader is called to parse the input. During parsing, methods on
the handler objects are called based on structural and syntactic events from the
input data.

For these objects, only the interfaces are relevant; they are normally not
instantiated by the application itself.  Since Python does not have an explicit
notion of interface, they are formally introduced as classes, but applications
may use implementations which do not inherit from the provided classes.  The
@ref{a0c,,InputSource}, @ref{2f6e,,Locator},
@code{Attributes}, @code{AttributesNS},
and @ref{2f61,,XMLReader} interfaces are defined in the
module @ref{145,,xml.sax.xmlreader}.  The handler interfaces are defined in
@ref{143,,xml.sax.handler}.  For convenience,
@ref{a0c,,InputSource} (which is often
instantiated directly) and the handler classes are also available from
@ref{142,,xml.sax}.  These interfaces are described below.

In addition to these classes, @ref{142,,xml.sax} provides the following exception
classes.

@geindex SAXException
@anchor{library/xml sax xml sax SAXException}@anchor{2f6f}
@deffn {Exception} xml.sax.SAXException (msg, exception=None)

Encapsulate an XML error or warning.  This class can contain basic error or
warning information from either the XML parser or the application: it can be
subclassed to provide additional functionality or to add localization.  Note
that although the handlers defined in the
@ref{2f6c,,ErrorHandler} interface
receive instances of this exception, it is not required to actually raise the
exception — it is also useful as a container for information.

When instantiated, @emph{msg} should be a human-readable description of the error.
The optional @emph{exception} parameter, if given, should be @code{None} or an exception
that was caught by the parsing code and is being passed along as information.

This is the base class for the other SAX exception classes.
@end deffn

@geindex SAXParseException
@anchor{library/xml sax xml sax SAXParseException}@anchor{2f6d}
@deffn {Exception} xml.sax.SAXParseException (msg, exception, locator)

Subclass of @ref{2f6f,,SAXException} raised on parse errors. Instances of this
class are passed to the methods of the SAX
@ref{2f6c,,ErrorHandler} interface to provide information
about the parse error.  This class supports the SAX
@ref{2f6e,,Locator} interface as well as the
@ref{2f6f,,SAXException} interface.
@end deffn

@geindex SAXNotRecognizedException
@anchor{library/xml sax xml sax SAXNotRecognizedException}@anchor{2f70}
@deffn {Exception} xml.sax.SAXNotRecognizedException (msg, exception=None)

Subclass of @ref{2f6f,,SAXException} raised when a SAX
@ref{2f61,,XMLReader} is
confronted with an unrecognized feature or property.  SAX applications and
extensions may use this class for similar purposes.
@end deffn

@geindex SAXNotSupportedException
@anchor{library/xml sax xml sax SAXNotSupportedException}@anchor{2f71}
@deffn {Exception} xml.sax.SAXNotSupportedException (msg, exception=None)

Subclass of @ref{2f6f,,SAXException} raised when a SAX
@ref{2f61,,XMLReader} is asked to
enable a feature that is not supported, or to set a property to a value that the
implementation does not support.  SAX applications and extensions may use this
class for similar purposes.
@end deffn


@subsubheading See also



@table @asis

@item SAX: The Simple API for XML@footnote{http://www.saxproject.org/}

This site is the focal point for the definition of the SAX API.  It provides a
Java implementation and online documentation.  Links to implementations and
historical information are also available.

@item Module @ref{143,,xml.sax.handler}

Definitions of the interfaces for application-provided objects.

@item Module @ref{144,,xml.sax.saxutils}

Convenience functions for use in SAX applications.

@item Module @ref{145,,xml.sax.xmlreader}

Definitions of the interfaces for parser-provided objects.
@end table


@menu
* SAXException Objects:: 

@end menu

@node SAXException Objects,,,xml sax — Support for SAX2 parsers
@anchor{library/xml sax sax-exception-objects}@anchor{2f72}@anchor{library/xml sax saxexception-objects}@anchor{2f73}
@subsubsection SAXException Objects


The @ref{2f6f,,SAXException} exception class supports the following methods:

@geindex getMessage() (xml.sax.SAXException method)
@anchor{library/xml sax xml sax SAXException getMessage}@anchor{2f74}
@deffn {Method} SAXException.getMessage ()

Return a human-readable message describing the error condition.
@end deffn

@geindex getException() (xml.sax.SAXException method)
@anchor{library/xml sax xml sax SAXException getException}@anchor{2f75}
@deffn {Method} SAXException.getException ()

Return an encapsulated exception object, or @code{None}.
@end deffn

@node xml sax handler — Base classes for SAX handlers,xml sax saxutils — SAX Utilities,xml sax — Support for SAX2 parsers,Structured Markup Processing Tools
@anchor{library/xml sax handler doc}@anchor{2f76}@anchor{library/xml sax handler module-xml sax handler}@anchor{143}@anchor{library/xml sax handler xml-sax-handler-base-classes-for-sax-handlers}@anchor{2f77}
@subsection @code{xml.sax.handler} — Base classes for SAX handlers


@geindex module; xml.sax.handler

@strong{Source code:} Lib/xml/sax/handler.py@footnote{https://github.com/python/cpython/tree/main/Lib/xml/sax/handler.py}


__________________________________________________________________


The SAX API defines five kinds of handlers: content handlers, DTD handlers,
error handlers, entity resolvers and lexical handlers. Applications normally
only need to implement those interfaces whose events they are interested in;
they can implement the interfaces in a single object or in multiple objects.
Handler implementations should inherit from the base classes provided in the
module @ref{143,,xml.sax.handler}, so that all methods get default implementations.

@geindex ContentHandler (class in xml.sax.handler)
@anchor{library/xml sax handler xml sax handler ContentHandler}@anchor{2f5e}
@deffn {Class} xml.sax.handler.ContentHandler

This is the main callback interface in SAX, and the one most important to
applications. The order of events in this interface mirrors the order of the
information in the document.
@end deffn

@geindex DTDHandler (class in xml.sax.handler)
@anchor{library/xml sax handler xml sax handler DTDHandler}@anchor{2f78}
@deffn {Class} xml.sax.handler.DTDHandler

Handle DTD events.

This interface specifies only those DTD events required for basic parsing
(unparsed entities and attributes).
@end deffn

@geindex EntityResolver (class in xml.sax.handler)
@anchor{library/xml sax handler xml sax handler EntityResolver}@anchor{2f79}
@deffn {Class} xml.sax.handler.EntityResolver

Basic interface for resolving entities. If you create an object implementing
this interface, then register the object with your Parser, the parser will call
the method in your object to resolve all external entities.
@end deffn

@geindex ErrorHandler (class in xml.sax.handler)
@anchor{library/xml sax handler xml sax handler ErrorHandler}@anchor{2f6c}
@deffn {Class} xml.sax.handler.ErrorHandler

Interface used by the parser to present error and warning messages to the
application.  The methods of this object control whether errors are immediately
converted to exceptions or are handled in some other way.
@end deffn

@geindex LexicalHandler (class in xml.sax.handler)
@anchor{library/xml sax handler xml sax handler LexicalHandler}@anchor{2e8}
@deffn {Class} xml.sax.handler.LexicalHandler

Interface used by the parser to represent low frequency events which may not
be of interest to many applications.
@end deffn

In addition to these classes, @ref{143,,xml.sax.handler} provides symbolic constants
for the feature and property names.

@geindex feature_namespaces (in module xml.sax.handler)
@anchor{library/xml sax handler xml sax handler feature_namespaces}@anchor{2f7a}
@deffn {Data} xml.sax.handler.feature_namespaces


@display
value: @code{"http://xml.org/sax/features/namespaces"}@w{ }
true: Perform Namespace processing.@w{ }
false: Optionally do not perform Namespace processing (implies namespace-prefixes; default).@w{ }
access: (parsing) read-only; (not parsing) read/write@w{ }
@end display


@end deffn

@geindex feature_namespace_prefixes (in module xml.sax.handler)
@anchor{library/xml sax handler xml sax handler feature_namespace_prefixes}@anchor{2f7b}
@deffn {Data} xml.sax.handler.feature_namespace_prefixes


@display
value: @code{"http://xml.org/sax/features/namespace-prefixes"}@w{ }
true: Report the original prefixed names and attributes used for Namespace declarations.@w{ }
false: Do not report attributes used for Namespace declarations, and optionally do not report original prefixed names (default).@w{ }
access: (parsing) read-only; (not parsing) read/write@w{ }
@end display


@end deffn

@geindex feature_string_interning (in module xml.sax.handler)
@anchor{library/xml sax handler xml sax handler feature_string_interning}@anchor{2f7c}
@deffn {Data} xml.sax.handler.feature_string_interning


@display
value: @code{"http://xml.org/sax/features/string-interning"}@w{ }
true: All element names, prefixes, attribute names, Namespace URIs, and local names are interned using the built-in intern function.@w{ }
false: Names are not necessarily interned, although they may be (default).@w{ }
access: (parsing) read-only; (not parsing) read/write@w{ }
@end display


@end deffn

@geindex feature_validation (in module xml.sax.handler)
@anchor{library/xml sax handler xml sax handler feature_validation}@anchor{2f7d}
@deffn {Data} xml.sax.handler.feature_validation


@display
value: @code{"http://xml.org/sax/features/validation"}@w{ }
true: Report all validation errors (implies external-general-entities and external-parameter-entities).@w{ }
false: Do not report validation errors.@w{ }
access: (parsing) read-only; (not parsing) read/write@w{ }
@end display


@end deffn

@geindex feature_external_ges (in module xml.sax.handler)
@anchor{library/xml sax handler xml sax handler feature_external_ges}@anchor{2f6b}
@deffn {Data} xml.sax.handler.feature_external_ges


@display
value: @code{"http://xml.org/sax/features/external-general-entities"}@w{ }
true: Include all external general (text) entities.@w{ }
false: Do not include external general entities.@w{ }
access: (parsing) read-only; (not parsing) read/write@w{ }
@end display


@end deffn

@geindex feature_external_pes (in module xml.sax.handler)
@anchor{library/xml sax handler xml sax handler feature_external_pes}@anchor{2f7e}
@deffn {Data} xml.sax.handler.feature_external_pes


@display
value: @code{"http://xml.org/sax/features/external-parameter-entities"}@w{ }
true: Include all external parameter entities, including the external DTD subset.@w{ }
false: Do not include any external parameter entities, even the external DTD subset.@w{ }
access: (parsing) read-only; (not parsing) read/write@w{ }
@end display


@end deffn

@geindex all_features (in module xml.sax.handler)
@anchor{library/xml sax handler xml sax handler all_features}@anchor{2f7f}
@deffn {Data} xml.sax.handler.all_features

List of all features.
@end deffn

@geindex property_lexical_handler (in module xml.sax.handler)
@anchor{library/xml sax handler xml sax handler property_lexical_handler}@anchor{2f80}
@deffn {Data} xml.sax.handler.property_lexical_handler


@display
value: @code{"http://xml.org/sax/properties/lexical-handler"}@w{ }
data type: xml.sax.handler.LexicalHandler (not supported in Python 2)@w{ }
description: An optional extension handler for lexical events like comments.@w{ }
access: read/write@w{ }
@end display


@end deffn

@geindex property_declaration_handler (in module xml.sax.handler)
@anchor{library/xml sax handler xml sax handler property_declaration_handler}@anchor{2f81}
@deffn {Data} xml.sax.handler.property_declaration_handler


@display
value: @code{"http://xml.org/sax/properties/declaration-handler"}@w{ }
data type: xml.sax.sax2lib.DeclHandler (not supported in Python 2)@w{ }
description: An optional extension handler for DTD-related events other than notations and unparsed entities.@w{ }
access: read/write@w{ }
@end display


@end deffn

@geindex property_dom_node (in module xml.sax.handler)
@anchor{library/xml sax handler xml sax handler property_dom_node}@anchor{2f82}
@deffn {Data} xml.sax.handler.property_dom_node


@display
value: @code{"http://xml.org/sax/properties/dom-node"}@w{ }
data type: org.w3c.dom.Node (not supported in Python 2)@w{ }
description: When parsing, the current DOM node being visited if this is a DOM iterator; when not parsing, the root DOM node for iteration.@w{ }
access: (parsing) read-only; (not parsing) read/write@w{ }
@end display


@end deffn

@geindex property_xml_string (in module xml.sax.handler)
@anchor{library/xml sax handler xml sax handler property_xml_string}@anchor{2f83}
@deffn {Data} xml.sax.handler.property_xml_string


@display
value: @code{"http://xml.org/sax/properties/xml-string"}@w{ }
data type: String@w{ }
description: The literal string of characters that was the source for the current event.@w{ }
access: read-only@w{ }
@end display


@end deffn

@geindex all_properties (in module xml.sax.handler)
@anchor{library/xml sax handler xml sax handler all_properties}@anchor{2f84}
@deffn {Data} xml.sax.handler.all_properties

List of all known property names.
@end deffn

@menu
* ContentHandler Objects:: 
* DTDHandler Objects:: 
* EntityResolver Objects:: 
* ErrorHandler Objects:: 
* LexicalHandler Objects:: 

@end menu

@node ContentHandler Objects,DTDHandler Objects,,xml sax handler — Base classes for SAX handlers
@anchor{library/xml sax handler content-handler-objects}@anchor{2f85}@anchor{library/xml sax handler contenthandler-objects}@anchor{2f86}
@subsubsection ContentHandler Objects


Users are expected to subclass @ref{2f5e,,ContentHandler} to support their
application.  The following methods are called by the parser on the appropriate
events in the input document:

@geindex setDocumentLocator() (xml.sax.handler.ContentHandler method)
@anchor{library/xml sax handler xml sax handler ContentHandler setDocumentLocator}@anchor{2f87}
@deffn {Method} ContentHandler.setDocumentLocator (locator)

Called by the parser to give the application a locator for locating the origin
of document events.

SAX parsers are strongly encouraged (though not absolutely required) to supply a
locator: if it does so, it must supply the locator to the application by
invoking this method before invoking any of the other methods in the
DocumentHandler interface.

The locator allows the application to determine the end position of any
document-related event, even if the parser is not reporting an error. Typically,
the application will use this information for reporting its own errors (such as
character content that does not match an application’s business rules). The
information returned by the locator is probably not sufficient for use with a
search engine.

Note that the locator will return correct information only during the invocation
of the events in this interface. The application should not attempt to use it at
any other time.
@end deffn

@geindex startDocument() (xml.sax.handler.ContentHandler method)
@anchor{library/xml sax handler xml sax handler ContentHandler startDocument}@anchor{2f88}
@deffn {Method} ContentHandler.startDocument ()

Receive notification of the beginning of a document.

The SAX parser will invoke this method only once, before any other methods in
this interface or in DTDHandler (except for @ref{2f87,,setDocumentLocator()}).
@end deffn

@geindex endDocument() (xml.sax.handler.ContentHandler method)
@anchor{library/xml sax handler xml sax handler ContentHandler endDocument}@anchor{2f89}
@deffn {Method} ContentHandler.endDocument ()

Receive notification of the end of a document.

The SAX parser will invoke this method only once, and it will be the last method
invoked during the parse. The parser shall not invoke this method until it has
either abandoned parsing (because of an unrecoverable error) or reached the end
of input.
@end deffn

@geindex startPrefixMapping() (xml.sax.handler.ContentHandler method)
@anchor{library/xml sax handler xml sax handler ContentHandler startPrefixMapping}@anchor{2f8a}
@deffn {Method} ContentHandler.startPrefixMapping (prefix, uri)

Begin the scope of a prefix-URI Namespace mapping.

The information from this event is not necessary for normal Namespace
processing: the SAX XML reader will automatically replace prefixes for element
and attribute names when the @code{feature_namespaces} feature is enabled (the
default).

There are cases, however, when applications need to use prefixes in character
data or in attribute values, where they cannot safely be expanded automatically;
the @ref{2f8a,,startPrefixMapping()} and @ref{2f8b,,endPrefixMapping()} events supply the
information to the application to expand prefixes in those contexts itself, if
necessary.

@c XXX This is not really the default, is it? MvL

Note that @ref{2f8a,,startPrefixMapping()} and @ref{2f8b,,endPrefixMapping()} events are not
guaranteed to be properly nested relative to each-other: all
@ref{2f8a,,startPrefixMapping()} events will occur before the corresponding
@ref{2f8c,,startElement()} event, and all @ref{2f8b,,endPrefixMapping()} events will occur
after the corresponding @ref{2f8d,,endElement()} event, but their order is not
guaranteed.
@end deffn

@geindex endPrefixMapping() (xml.sax.handler.ContentHandler method)
@anchor{library/xml sax handler xml sax handler ContentHandler endPrefixMapping}@anchor{2f8b}
@deffn {Method} ContentHandler.endPrefixMapping (prefix)

End the scope of a prefix-URI mapping.

See @ref{2f8a,,startPrefixMapping()} for details. This event will always occur after
the corresponding @ref{2f8d,,endElement()} event, but the order of
@ref{2f8b,,endPrefixMapping()} events is not otherwise guaranteed.
@end deffn

@geindex startElement() (xml.sax.handler.ContentHandler method)
@anchor{library/xml sax handler xml sax handler ContentHandler startElement}@anchor{2f8c}
@deffn {Method} ContentHandler.startElement (name, attrs)

Signals the start of an element in non-namespace mode.

The @emph{name} parameter contains the raw XML 1.0 name of the element type as a
string and the @emph{attrs} parameter holds an object of the
@code{Attributes}
interface (see @ref{2f8e,,The Attributes Interface}) containing the attributes of
the element.  The object passed as @emph{attrs} may be re-used by the parser; holding
on to a reference to it is not a reliable way to keep a copy of the attributes.
To keep a copy of the attributes, use the @ref{24,,copy()} method of the @emph{attrs}
object.
@end deffn

@geindex endElement() (xml.sax.handler.ContentHandler method)
@anchor{library/xml sax handler xml sax handler ContentHandler endElement}@anchor{2f8d}
@deffn {Method} ContentHandler.endElement (name)

Signals the end of an element in non-namespace mode.

The @emph{name} parameter contains the name of the element type, just as with the
@ref{2f8c,,startElement()} event.
@end deffn

@geindex startElementNS() (xml.sax.handler.ContentHandler method)
@anchor{library/xml sax handler xml sax handler ContentHandler startElementNS}@anchor{2f8f}
@deffn {Method} ContentHandler.startElementNS (name, qname, attrs)

Signals the start of an element in namespace mode.

The @emph{name} parameter contains the name of the element type as a @code{(uri,
localname)} tuple, the @emph{qname} parameter contains the raw XML 1.0 name used in
the source document, and the @emph{attrs} parameter holds an instance of the
@code{AttributesNS} interface (see
@ref{2f90,,The AttributesNS Interface})
containing the attributes of the element.  If no namespace is associated with
the element, the @emph{uri} component of @emph{name} will be @code{None}.  The object passed
as @emph{attrs} may be re-used by the parser; holding on to a reference to it is not
a reliable way to keep a copy of the attributes.  To keep a copy of the
attributes, use the @ref{24,,copy()} method of the @emph{attrs} object.

Parsers may set the @emph{qname} parameter to @code{None}, unless the
@code{feature_namespace_prefixes} feature is activated.
@end deffn

@geindex endElementNS() (xml.sax.handler.ContentHandler method)
@anchor{library/xml sax handler xml sax handler ContentHandler endElementNS}@anchor{2f91}
@deffn {Method} ContentHandler.endElementNS (name, qname)

Signals the end of an element in namespace mode.

The @emph{name} parameter contains the name of the element type, just as with the
@ref{2f8f,,startElementNS()} method, likewise the @emph{qname} parameter.
@end deffn

@geindex characters() (xml.sax.handler.ContentHandler method)
@anchor{library/xml sax handler xml sax handler ContentHandler characters}@anchor{2f92}
@deffn {Method} ContentHandler.characters (content)

Receive notification of character data.

The Parser will call this method to report each chunk of character data. SAX
parsers may return all contiguous character data in a single chunk, or they may
split it into several chunks; however, all of the characters in any single event
must come from the same external entity so that the Locator provides useful
information.

@emph{content} may be a string or bytes instance; the @code{expat} reader module
always produces strings.

@cartouche
@quotation Note 
The earlier SAX 1 interface provided by the Python XML Special Interest Group
used a more Java-like interface for this method.  Since most parsers used from
Python did not take advantage of the older interface, the simpler signature was
chosen to replace it.  To convert old code to the new interface, use @emph{content}
instead of slicing content with the old @emph{offset} and @emph{length} parameters.
@end quotation
@end cartouche
@end deffn

@geindex ignorableWhitespace() (xml.sax.handler.ContentHandler method)
@anchor{library/xml sax handler xml sax handler ContentHandler ignorableWhitespace}@anchor{2f93}
@deffn {Method} ContentHandler.ignorableWhitespace (whitespace)

Receive notification of ignorable whitespace in element content.

Validating Parsers must use this method to report each chunk of ignorable
whitespace (see the W3C XML 1.0 recommendation, section 2.10): non-validating
parsers may also use this method if they are capable of parsing and using
content models.

SAX parsers may return all contiguous whitespace in a single chunk, or they may
split it into several chunks; however, all of the characters in any single event
must come from the same external entity, so that the Locator provides useful
information.
@end deffn

@geindex processingInstruction() (xml.sax.handler.ContentHandler method)
@anchor{library/xml sax handler xml sax handler ContentHandler processingInstruction}@anchor{2f94}
@deffn {Method} ContentHandler.processingInstruction (target, data)

Receive notification of a processing instruction.

The Parser will invoke this method once for each processing instruction found:
note that processing instructions may occur before or after the main document
element.

A SAX parser should never report an XML declaration (XML 1.0, section 2.8) or a
text declaration (XML 1.0, section 4.3.1) using this method.
@end deffn

@geindex skippedEntity() (xml.sax.handler.ContentHandler method)
@anchor{library/xml sax handler xml sax handler ContentHandler skippedEntity}@anchor{2f95}
@deffn {Method} ContentHandler.skippedEntity (name)

Receive notification of a skipped entity.

The Parser will invoke this method once for each entity skipped. Non-validating
processors may skip entities if they have not seen the declarations (because,
for example, the entity was declared in an external DTD subset). All processors
may skip external entities, depending on the values of the
@code{feature_external_ges} and the @code{feature_external_pes} properties.
@end deffn

@node DTDHandler Objects,EntityResolver Objects,ContentHandler Objects,xml sax handler — Base classes for SAX handlers
@anchor{library/xml sax handler dtd-handler-objects}@anchor{2f96}@anchor{library/xml sax handler dtdhandler-objects}@anchor{2f97}
@subsubsection DTDHandler Objects


@ref{2f78,,DTDHandler} instances provide the following methods:

@geindex notationDecl() (xml.sax.handler.DTDHandler method)
@anchor{library/xml sax handler xml sax handler DTDHandler notationDecl}@anchor{2f98}
@deffn {Method} DTDHandler.notationDecl (name, publicId, systemId)

Handle a notation declaration event.
@end deffn

@geindex unparsedEntityDecl() (xml.sax.handler.DTDHandler method)
@anchor{library/xml sax handler xml sax handler DTDHandler unparsedEntityDecl}@anchor{2f99}
@deffn {Method} DTDHandler.unparsedEntityDecl (name, publicId, systemId, ndata)

Handle an unparsed entity declaration event.
@end deffn

@node EntityResolver Objects,ErrorHandler Objects,DTDHandler Objects,xml sax handler — Base classes for SAX handlers
@anchor{library/xml sax handler entity-resolver-objects}@anchor{2f9a}@anchor{library/xml sax handler entityresolver-objects}@anchor{2f9b}
@subsubsection EntityResolver Objects


@geindex resolveEntity() (xml.sax.handler.EntityResolver method)
@anchor{library/xml sax handler xml sax handler EntityResolver resolveEntity}@anchor{2f9c}
@deffn {Method} EntityResolver.resolveEntity (publicId, systemId)

Resolve the system identifier of an entity and return either the system
identifier to read from as a string, or an InputSource to read from. The default
implementation returns @emph{systemId}.
@end deffn

@node ErrorHandler Objects,LexicalHandler Objects,EntityResolver Objects,xml sax handler — Base classes for SAX handlers
@anchor{library/xml sax handler errorhandler-objects}@anchor{2f9d}@anchor{library/xml sax handler sax-error-handler}@anchor{2f9e}
@subsubsection ErrorHandler Objects


Objects with this interface are used to receive error and warning information
from the @ref{2f61,,XMLReader}.  If you create an object that
implements this interface, then register the object with your
@ref{2f61,,XMLReader}, the parser
will call the methods in your object to report all warnings and errors. There
are three levels of errors available: warnings, (possibly) recoverable errors,
and unrecoverable errors.  All methods take a @code{SAXParseException} as the
only parameter.  Errors and warnings may be converted to an exception by raising
the passed-in exception object.

@geindex error() (xml.sax.handler.ErrorHandler method)
@anchor{library/xml sax handler xml sax handler ErrorHandler error}@anchor{2f9f}
@deffn {Method} ErrorHandler.error (exception)

Called when the parser encounters a recoverable error.  If this method does not
raise an exception, parsing may continue, but further document information
should not be expected by the application.  Allowing the parser to continue may
allow additional errors to be discovered in the input document.
@end deffn

@geindex fatalError() (xml.sax.handler.ErrorHandler method)
@anchor{library/xml sax handler xml sax handler ErrorHandler fatalError}@anchor{2fa0}
@deffn {Method} ErrorHandler.fatalError (exception)

Called when the parser encounters an error it cannot recover from; parsing is
expected to terminate when this method returns.
@end deffn

@geindex warning() (xml.sax.handler.ErrorHandler method)
@anchor{library/xml sax handler xml sax handler ErrorHandler warning}@anchor{2fa1}
@deffn {Method} ErrorHandler.warning (exception)

Called when the parser presents minor warning information to the application.
Parsing is expected to continue when this method returns, and document
information will continue to be passed to the application. Raising an exception
in this method will cause parsing to end.
@end deffn

@node LexicalHandler Objects,,ErrorHandler Objects,xml sax handler — Base classes for SAX handlers
@anchor{library/xml sax handler lexical-handler-objects}@anchor{2fa2}@anchor{library/xml sax handler lexicalhandler-objects}@anchor{2fa3}
@subsubsection LexicalHandler Objects


Optional SAX2 handler for lexical events.

This handler is used to obtain lexical information about an XML
document. Lexical information includes information describing the
document encoding used and XML comments embedded in the document, as
well as section boundaries for the DTD and for any CDATA sections.
The lexical handlers are used in the same manner as content handlers.

Set the LexicalHandler of an XMLReader by using the setProperty method
with the property identifier
@code{'http://xml.org/sax/properties/lexical-handler'}.

@geindex comment() (xml.sax.handler.LexicalHandler method)
@anchor{library/xml sax handler xml sax handler LexicalHandler comment}@anchor{2fa4}
@deffn {Method} LexicalHandler.comment (content)

Reports a comment anywhere in the document (including the DTD and
outside the document element).
@end deffn

@geindex startDTD() (xml.sax.handler.LexicalHandler method)
@anchor{library/xml sax handler xml sax handler LexicalHandler startDTD}@anchor{2fa5}
@deffn {Method} LexicalHandler.startDTD (name, public_id, system_id)

Reports the start of the DTD declarations if the document has an
associated DTD.
@end deffn

@geindex endDTD() (xml.sax.handler.LexicalHandler method)
@anchor{library/xml sax handler xml sax handler LexicalHandler endDTD}@anchor{2fa6}
@deffn {Method} LexicalHandler.endDTD ()

Reports the end of DTD declaration.
@end deffn

@geindex startCDATA() (xml.sax.handler.LexicalHandler method)
@anchor{library/xml sax handler xml sax handler LexicalHandler startCDATA}@anchor{2fa7}
@deffn {Method} LexicalHandler.startCDATA ()

Reports the start of a CDATA marked section.

The contents of the CDATA marked section will be reported through
the characters handler.
@end deffn

@geindex endCDATA() (xml.sax.handler.LexicalHandler method)
@anchor{library/xml sax handler xml sax handler LexicalHandler endCDATA}@anchor{2fa8}
@deffn {Method} LexicalHandler.endCDATA ()

Reports the end of a CDATA marked section.
@end deffn

@node xml sax saxutils — SAX Utilities,xml sax xmlreader — Interface for XML parsers,xml sax handler — Base classes for SAX handlers,Structured Markup Processing Tools
@anchor{library/xml sax utils doc}@anchor{2fa9}@anchor{library/xml sax utils module-xml sax saxutils}@anchor{144}@anchor{library/xml sax utils xml-sax-saxutils-sax-utilities}@anchor{2faa}
@subsection @code{xml.sax.saxutils} — SAX Utilities


@geindex module; xml.sax.saxutils

@strong{Source code:} Lib/xml/sax/saxutils.py@footnote{https://github.com/python/cpython/tree/main/Lib/xml/sax/saxutils.py}


__________________________________________________________________


The module @ref{144,,xml.sax.saxutils} contains a number of classes and functions
that are commonly useful when creating SAX applications, either in direct use,
or as base classes.

@geindex escape() (in module xml.sax.saxutils)
@anchor{library/xml sax utils xml sax saxutils escape}@anchor{2fab}
@deffn {Function} xml.sax.saxutils.escape (data, entities=@{@})

Escape @code{'&'}, @code{'<'}, and @code{'>'} in a string of data.

You can escape other strings of data by passing a dictionary as the optional
@emph{entities} parameter.  The keys and values must all be strings; each key will be
replaced with its corresponding value.  The characters @code{'&'}, @code{'<'} and
@code{'>'} are always escaped, even if @emph{entities} is provided.
@end deffn

@geindex unescape() (in module xml.sax.saxutils)
@anchor{library/xml sax utils xml sax saxutils unescape}@anchor{2fac}
@deffn {Function} xml.sax.saxutils.unescape (data, entities=@{@})

Unescape @code{'&amp;'}, @code{'&lt;'}, and @code{'&gt;'} in a string of data.

You can unescape other strings of data by passing a dictionary as the optional
@emph{entities} parameter.  The keys and values must all be strings; each key will be
replaced with its corresponding value.  @code{'&amp'}, @code{'&lt;'}, and @code{'&gt;'}
are always unescaped, even if @emph{entities} is provided.
@end deffn

@geindex quoteattr() (in module xml.sax.saxutils)
@anchor{library/xml sax utils xml sax saxutils quoteattr}@anchor{2fad}
@deffn {Function} xml.sax.saxutils.quoteattr (data, entities=@{@})

Similar to @ref{2fab,,escape()}, but also prepares @emph{data} to be used as an
attribute value.  The return value is a quoted version of @emph{data} with any
additional required replacements. @ref{2fad,,quoteattr()} will select a quote
character based on the content of @emph{data}, attempting to avoid encoding any
quote characters in the string.  If both single- and double-quote characters
are already in @emph{data}, the double-quote characters will be encoded and @emph{data}
will be wrapped in double-quotes.  The resulting string can be used directly
as an attribute value:

@example
>>> print("<element attr=%s>" % quoteattr("ab ' cd \" ef"))
<element attr="ab ' cd &quot; ef">
@end example

This function is useful when generating attribute values for HTML or any SGML
using the reference concrete syntax.
@end deffn

@geindex XMLGenerator (class in xml.sax.saxutils)
@anchor{library/xml sax utils xml sax saxutils XMLGenerator}@anchor{2fae}
@deffn {Class} xml.sax.saxutils.XMLGenerator (out=None, encoding='iso-8859-1', short_empty_elements=False)

This class implements the @ref{2f5e,,ContentHandler} interface
by writing SAX
events back into an XML document. In other words, using an @ref{2fae,,XMLGenerator}
as the content handler will reproduce the original document being parsed. @emph{out}
should be a file-like object which will default to @emph{sys.stdout}. @emph{encoding} is
the encoding of the output stream which defaults to @code{'iso-8859-1'}.
@emph{short_empty_elements} controls the formatting of elements that contain no
content:  if @code{False} (the default) they are emitted as a pair of start/end
tags, if set to @code{True} they are emitted as a single self-closed tag.


New in version 3.2: The @emph{short_empty_elements} parameter.

@end deffn

@geindex XMLFilterBase (class in xml.sax.saxutils)
@anchor{library/xml sax utils xml sax saxutils XMLFilterBase}@anchor{2faf}
@deffn {Class} xml.sax.saxutils.XMLFilterBase (base)

This class is designed to sit between an
@ref{2f61,,XMLReader} and the client
application’s event handlers.  By default, it does nothing but pass requests up
to the reader and events on to the handlers unmodified, but subclasses can
override specific methods to modify the event stream or the configuration
requests as they pass through.
@end deffn

@geindex prepare_input_source() (in module xml.sax.saxutils)
@anchor{library/xml sax utils xml sax saxutils prepare_input_source}@anchor{2fb0}
@deffn {Function} xml.sax.saxutils.prepare_input_source (source, base='@w{'})

This function takes an input source and an optional base URL and returns a
fully resolved @ref{a0c,,InputSource} object ready for
reading.  The input source can be given as a string, a file-like object, or
an @ref{a0c,,InputSource} object; parsers will use this
function to implement the polymorphic @emph{source} argument to their
@code{parse()} method.
@end deffn

@node xml sax xmlreader — Interface for XML parsers,xml parsers expat — Fast XML parsing using Expat,xml sax saxutils — SAX Utilities,Structured Markup Processing Tools
@anchor{library/xml sax reader doc}@anchor{2fb1}@anchor{library/xml sax reader module-xml sax xmlreader}@anchor{145}@anchor{library/xml sax reader xml-sax-xmlreader-interface-for-xml-parsers}@anchor{2fb2}
@subsection @code{xml.sax.xmlreader} — Interface for XML parsers


@geindex module; xml.sax.xmlreader

@strong{Source code:} Lib/xml/sax/xmlreader.py@footnote{https://github.com/python/cpython/tree/main/Lib/xml/sax/xmlreader.py}


__________________________________________________________________


SAX parsers implement the @ref{2f61,,XMLReader} interface. They are implemented in
a Python module, which must provide a function @code{create_parser()}. This
function is invoked by  @ref{12ef,,xml.sax.make_parser()} with no arguments to create
a new  parser object.

@geindex XMLReader (class in xml.sax.xmlreader)
@anchor{library/xml sax reader xml sax xmlreader XMLReader}@anchor{2f61}
@deffn {Class} xml.sax.xmlreader.XMLReader

Base class which can be inherited by SAX parsers.
@end deffn

@geindex IncrementalParser (class in xml.sax.xmlreader)
@anchor{library/xml sax reader xml sax xmlreader IncrementalParser}@anchor{2fb3}
@deffn {Class} xml.sax.xmlreader.IncrementalParser

In some cases, it is desirable not to parse an input source at once, but to feed
chunks of the document as they get available. Note that the reader will normally
not read the entire file, but read it in chunks as well; still @code{parse()}
won’t return until the entire document is processed. So these interfaces should
be used if the blocking behaviour of @code{parse()} is not desirable.

When the parser is instantiated it is ready to begin accepting data from the
feed method immediately. After parsing has been finished with a call to close
the reset method must be called to make the parser ready to accept new data,
either from feed or using the parse method.

Note that these methods must @emph{not} be called during parsing, that is, after
parse has been called and before it returns.

By default, the class also implements the parse method of the XMLReader
interface using the feed, close and reset methods of the IncrementalParser
interface as a convenience to SAX 2.0 driver writers.
@end deffn

@geindex Locator (class in xml.sax.xmlreader)
@anchor{library/xml sax reader xml sax xmlreader Locator}@anchor{2f6e}
@deffn {Class} xml.sax.xmlreader.Locator

Interface for associating a SAX event with a document location. A locator object
will return valid results only during calls to DocumentHandler methods; at any
other time, the results are unpredictable. If information is not available,
methods may return @code{None}.
@end deffn

@geindex InputSource (class in xml.sax.xmlreader)
@anchor{library/xml sax reader xml sax xmlreader InputSource}@anchor{a0c}
@deffn {Class} xml.sax.xmlreader.InputSource (system_id=None)

Encapsulation of the information needed by the @ref{2f61,,XMLReader} to read
entities.

This class may include information about the public identifier, system
identifier, byte stream (possibly with character encoding information) and/or
the character stream of an entity.

Applications will create objects of this class for use in the
@ref{2fb4,,XMLReader.parse()} method and for returning from
EntityResolver.resolveEntity.

An @ref{a0c,,InputSource} belongs to the application, the @ref{2f61,,XMLReader} is
not allowed to modify @ref{a0c,,InputSource} objects passed to it from the
application, although it may make copies and modify those.
@end deffn

@geindex AttributesImpl (class in xml.sax.xmlreader)
@anchor{library/xml sax reader xml sax xmlreader AttributesImpl}@anchor{2fb5}
@deffn {Class} xml.sax.xmlreader.AttributesImpl (attrs)

This is an implementation of the @code{Attributes} interface (see section
@ref{2f8e,,The Attributes Interface}).  This is a dictionary-like object which
represents the element attributes in a @code{startElement()} call. In addition
to the most useful dictionary operations, it supports a number of other
methods as described by the interface. Objects of this class should be
instantiated by readers; @emph{attrs} must be a dictionary-like object containing
a mapping from attribute names to attribute values.
@end deffn

@geindex AttributesNSImpl (class in xml.sax.xmlreader)
@anchor{library/xml sax reader xml sax xmlreader AttributesNSImpl}@anchor{2fb6}
@deffn {Class} xml.sax.xmlreader.AttributesNSImpl (attrs, qnames)

Namespace-aware variant of @ref{2fb5,,AttributesImpl}, which will be passed to
@code{startElementNS()}. It is derived from @ref{2fb5,,AttributesImpl}, but
understands attribute names as two-tuples of @emph{namespaceURI} and
@emph{localname}. In addition, it provides a number of methods expecting qualified
names as they appear in the original document.  This class implements the
@code{AttributesNS} interface (see section @ref{2f90,,The AttributesNS Interface}).
@end deffn

@menu
* XMLReader Objects:: 
* IncrementalParser Objects:: 
* Locator Objects:: 
* InputSource Objects:: 
* The Attributes Interface:: 
* The AttributesNS Interface:: 

@end menu

@node XMLReader Objects,IncrementalParser Objects,,xml sax xmlreader — Interface for XML parsers
@anchor{library/xml sax reader id1}@anchor{2fb7}@anchor{library/xml sax reader xmlreader-objects}@anchor{2fb8}
@subsubsection XMLReader Objects


The @ref{2f61,,XMLReader} interface supports the following methods:

@geindex parse() (xml.sax.xmlreader.XMLReader method)
@anchor{library/xml sax reader xml sax xmlreader XMLReader parse}@anchor{2fb4}
@deffn {Method} XMLReader.parse (source)

Process an input source, producing SAX events. The @emph{source} object can be a
system identifier (a string identifying the input source – typically a file
name or a URL), a @ref{522,,pathlib.Path} or @ref{63d,,path-like}
object, or an @ref{a0c,,InputSource} object. When
@ref{2fb4,,parse()} returns, the input is completely processed, and the parser object
can be discarded or reset.


Changed in version 3.5: Added support of character streams.



Changed in version 3.8: Added support of path-like objects.

@end deffn

@geindex getContentHandler() (xml.sax.xmlreader.XMLReader method)
@anchor{library/xml sax reader xml sax xmlreader XMLReader getContentHandler}@anchor{2fb9}
@deffn {Method} XMLReader.getContentHandler ()

Return the current @ref{2f5e,,ContentHandler}.
@end deffn

@geindex setContentHandler() (xml.sax.xmlreader.XMLReader method)
@anchor{library/xml sax reader xml sax xmlreader XMLReader setContentHandler}@anchor{2fba}
@deffn {Method} XMLReader.setContentHandler (handler)

Set the current @ref{2f5e,,ContentHandler}.  If no
@ref{2f5e,,ContentHandler} is set, content events will be
discarded.
@end deffn

@geindex getDTDHandler() (xml.sax.xmlreader.XMLReader method)
@anchor{library/xml sax reader xml sax xmlreader XMLReader getDTDHandler}@anchor{2fbb}
@deffn {Method} XMLReader.getDTDHandler ()

Return the current @ref{2f78,,DTDHandler}.
@end deffn

@geindex setDTDHandler() (xml.sax.xmlreader.XMLReader method)
@anchor{library/xml sax reader xml sax xmlreader XMLReader setDTDHandler}@anchor{2fbc}
@deffn {Method} XMLReader.setDTDHandler (handler)

Set the current @ref{2f78,,DTDHandler}.  If no
@ref{2f78,,DTDHandler} is set, DTD
events will be discarded.
@end deffn

@geindex getEntityResolver() (xml.sax.xmlreader.XMLReader method)
@anchor{library/xml sax reader xml sax xmlreader XMLReader getEntityResolver}@anchor{2fbd}
@deffn {Method} XMLReader.getEntityResolver ()

Return the current @ref{2f79,,EntityResolver}.
@end deffn

@geindex setEntityResolver() (xml.sax.xmlreader.XMLReader method)
@anchor{library/xml sax reader xml sax xmlreader XMLReader setEntityResolver}@anchor{2fbe}
@deffn {Method} XMLReader.setEntityResolver (handler)

Set the current @ref{2f79,,EntityResolver}.  If no
@ref{2f79,,EntityResolver} is set,
attempts to resolve an external entity will result in opening the system
identifier for the entity, and fail if it is not available.
@end deffn

@geindex getErrorHandler() (xml.sax.xmlreader.XMLReader method)
@anchor{library/xml sax reader xml sax xmlreader XMLReader getErrorHandler}@anchor{2fbf}
@deffn {Method} XMLReader.getErrorHandler ()

Return the current @ref{2f6c,,ErrorHandler}.
@end deffn

@geindex setErrorHandler() (xml.sax.xmlreader.XMLReader method)
@anchor{library/xml sax reader xml sax xmlreader XMLReader setErrorHandler}@anchor{2fc0}
@deffn {Method} XMLReader.setErrorHandler (handler)

Set the current error handler.  If no @ref{2f6c,,ErrorHandler}
is set, errors will be raised as exceptions, and warnings will be printed.
@end deffn

@geindex setLocale() (xml.sax.xmlreader.XMLReader method)
@anchor{library/xml sax reader xml sax xmlreader XMLReader setLocale}@anchor{2fc1}
@deffn {Method} XMLReader.setLocale (locale)

Allow an application to set the locale for errors and warnings.

SAX parsers are not required to provide localization for errors and warnings; if
they cannot support the requested locale, however, they must raise a SAX
exception.  Applications may request a locale change in the middle of a parse.
@end deffn

@geindex getFeature() (xml.sax.xmlreader.XMLReader method)
@anchor{library/xml sax reader xml sax xmlreader XMLReader getFeature}@anchor{2fc2}
@deffn {Method} XMLReader.getFeature (featurename)

Return the current setting for feature @emph{featurename}.  If the feature is not
recognized, @code{SAXNotRecognizedException} is raised. The well-known
featurenames are listed in the module @ref{143,,xml.sax.handler}.
@end deffn

@geindex setFeature() (xml.sax.xmlreader.XMLReader method)
@anchor{library/xml sax reader xml sax xmlreader XMLReader setFeature}@anchor{2f6a}
@deffn {Method} XMLReader.setFeature (featurename, value)

Set the @emph{featurename} to @emph{value}. If the feature is not recognized,
@code{SAXNotRecognizedException} is raised. If the feature or its setting is not
supported by the parser, @emph{SAXNotSupportedException} is raised.
@end deffn

@geindex getProperty() (xml.sax.xmlreader.XMLReader method)
@anchor{library/xml sax reader xml sax xmlreader XMLReader getProperty}@anchor{2fc3}
@deffn {Method} XMLReader.getProperty (propertyname)

Return the current setting for property @emph{propertyname}. If the property is not
recognized, a @code{SAXNotRecognizedException} is raised. The well-known
propertynames are listed in the module @ref{143,,xml.sax.handler}.
@end deffn

@geindex setProperty() (xml.sax.xmlreader.XMLReader method)
@anchor{library/xml sax reader xml sax xmlreader XMLReader setProperty}@anchor{2fc4}
@deffn {Method} XMLReader.setProperty (propertyname, value)

Set the @emph{propertyname} to @emph{value}. If the property is not recognized,
@code{SAXNotRecognizedException} is raised. If the property or its setting is
not supported by the parser, @emph{SAXNotSupportedException} is raised.
@end deffn

@node IncrementalParser Objects,Locator Objects,XMLReader Objects,xml sax xmlreader — Interface for XML parsers
@anchor{library/xml sax reader incremental-parser-objects}@anchor{2fc5}@anchor{library/xml sax reader incrementalparser-objects}@anchor{2fc6}
@subsubsection IncrementalParser Objects


Instances of @ref{2fb3,,IncrementalParser} offer the following additional methods:

@geindex feed() (xml.sax.xmlreader.IncrementalParser method)
@anchor{library/xml sax reader xml sax xmlreader IncrementalParser feed}@anchor{2fc7}
@deffn {Method} IncrementalParser.feed (data)

Process a chunk of @emph{data}.
@end deffn

@geindex close() (xml.sax.xmlreader.IncrementalParser method)
@anchor{library/xml sax reader xml sax xmlreader IncrementalParser close}@anchor{2fc8}
@deffn {Method} IncrementalParser.close ()

Assume the end of the document. That will check well-formedness conditions that
can be checked only at the end, invoke handlers, and may clean up resources
allocated during parsing.
@end deffn

@geindex reset() (xml.sax.xmlreader.IncrementalParser method)
@anchor{library/xml sax reader xml sax xmlreader IncrementalParser reset}@anchor{2fc9}
@deffn {Method} IncrementalParser.reset ()

This method is called after close has been called to reset the parser so that it
is ready to parse new documents. The results of calling parse or feed after
close without calling reset are undefined.
@end deffn

@node Locator Objects,InputSource Objects,IncrementalParser Objects,xml sax xmlreader — Interface for XML parsers
@anchor{library/xml sax reader id2}@anchor{2fca}@anchor{library/xml sax reader locator-objects}@anchor{2fcb}
@subsubsection Locator Objects


Instances of @ref{2f6e,,Locator} provide these methods:

@geindex getColumnNumber() (xml.sax.xmlreader.Locator method)
@anchor{library/xml sax reader xml sax xmlreader Locator getColumnNumber}@anchor{2fcc}
@deffn {Method} Locator.getColumnNumber ()

Return the column number where the current event begins.
@end deffn

@geindex getLineNumber() (xml.sax.xmlreader.Locator method)
@anchor{library/xml sax reader xml sax xmlreader Locator getLineNumber}@anchor{2fcd}
@deffn {Method} Locator.getLineNumber ()

Return the line number where the current event begins.
@end deffn

@geindex getPublicId() (xml.sax.xmlreader.Locator method)
@anchor{library/xml sax reader xml sax xmlreader Locator getPublicId}@anchor{2fce}
@deffn {Method} Locator.getPublicId ()

Return the public identifier for the current event.
@end deffn

@geindex getSystemId() (xml.sax.xmlreader.Locator method)
@anchor{library/xml sax reader xml sax xmlreader Locator getSystemId}@anchor{2fcf}
@deffn {Method} Locator.getSystemId ()

Return the system identifier for the current event.
@end deffn

@node InputSource Objects,The Attributes Interface,Locator Objects,xml sax xmlreader — Interface for XML parsers
@anchor{library/xml sax reader input-source-objects}@anchor{2fd0}@anchor{library/xml sax reader inputsource-objects}@anchor{2fd1}
@subsubsection InputSource Objects


@geindex setPublicId() (xml.sax.xmlreader.InputSource method)
@anchor{library/xml sax reader xml sax xmlreader InputSource setPublicId}@anchor{2fd2}
@deffn {Method} InputSource.setPublicId (id)

Sets the public identifier of this @ref{a0c,,InputSource}.
@end deffn

@geindex getPublicId() (xml.sax.xmlreader.InputSource method)
@anchor{library/xml sax reader xml sax xmlreader InputSource getPublicId}@anchor{2fd3}
@deffn {Method} InputSource.getPublicId ()

Returns the public identifier of this @ref{a0c,,InputSource}.
@end deffn

@geindex setSystemId() (xml.sax.xmlreader.InputSource method)
@anchor{library/xml sax reader xml sax xmlreader InputSource setSystemId}@anchor{2fd4}
@deffn {Method} InputSource.setSystemId (id)

Sets the system identifier of this @ref{a0c,,InputSource}.
@end deffn

@geindex getSystemId() (xml.sax.xmlreader.InputSource method)
@anchor{library/xml sax reader xml sax xmlreader InputSource getSystemId}@anchor{2fd5}
@deffn {Method} InputSource.getSystemId ()

Returns the system identifier of this @ref{a0c,,InputSource}.
@end deffn

@geindex setEncoding() (xml.sax.xmlreader.InputSource method)
@anchor{library/xml sax reader xml sax xmlreader InputSource setEncoding}@anchor{2fd6}
@deffn {Method} InputSource.setEncoding (encoding)

Sets the character encoding of this @ref{a0c,,InputSource}.

The encoding must be a string acceptable for an XML encoding declaration (see
section 4.3.3 of the XML recommendation).

The encoding attribute of the @ref{a0c,,InputSource} is ignored if the
@ref{a0c,,InputSource} also contains a character stream.
@end deffn

@geindex getEncoding() (xml.sax.xmlreader.InputSource method)
@anchor{library/xml sax reader xml sax xmlreader InputSource getEncoding}@anchor{2fd7}
@deffn {Method} InputSource.getEncoding ()

Get the character encoding of this InputSource.
@end deffn

@geindex setByteStream() (xml.sax.xmlreader.InputSource method)
@anchor{library/xml sax reader xml sax xmlreader InputSource setByteStream}@anchor{2fd8}
@deffn {Method} InputSource.setByteStream (bytefile)

Set the byte stream (a @ref{2138,,binary file}) for this input source.

The SAX parser will ignore this if there is also a character stream specified,
but it will use a byte stream in preference to opening a URI connection itself.

If the application knows the character encoding of the byte stream, it should
set it with the setEncoding method.
@end deffn

@geindex getByteStream() (xml.sax.xmlreader.InputSource method)
@anchor{library/xml sax reader xml sax xmlreader InputSource getByteStream}@anchor{2fd9}
@deffn {Method} InputSource.getByteStream ()

Get the byte stream for this input source.

The getEncoding method will return the character encoding for this byte stream,
or @code{None} if unknown.
@end deffn

@geindex setCharacterStream() (xml.sax.xmlreader.InputSource method)
@anchor{library/xml sax reader xml sax xmlreader InputSource setCharacterStream}@anchor{2fda}
@deffn {Method} InputSource.setCharacterStream (charfile)

Set the character stream (a @ref{1566,,text file}) for this input source.

If there is a character stream specified, the SAX parser will ignore any byte
stream and will not attempt to open a URI connection to the system identifier.
@end deffn

@geindex getCharacterStream() (xml.sax.xmlreader.InputSource method)
@anchor{library/xml sax reader xml sax xmlreader InputSource getCharacterStream}@anchor{2fdb}
@deffn {Method} InputSource.getCharacterStream ()

Get the character stream for this input source.
@end deffn

@node The Attributes Interface,The AttributesNS Interface,InputSource Objects,xml sax xmlreader — Interface for XML parsers
@anchor{library/xml sax reader attributes-objects}@anchor{2f8e}@anchor{library/xml sax reader the-attributes-interface}@anchor{2fdc}
@subsubsection The @code{Attributes} Interface


@code{Attributes} objects implement a portion of the @ref{d62,,mapping protocol}, including the methods @code{copy()},
@code{get()}, @ref{17a0,,__contains__()},
@code{items()}, @code{keys()},
and @code{values()}.  The following methods
are also provided:

@geindex getLength() (xml.sax.xmlreader.Attributes method)
@anchor{library/xml sax reader xml sax xmlreader Attributes getLength}@anchor{2fdd}
@deffn {Method} Attributes.getLength ()

Return the number of attributes.
@end deffn

@geindex getNames() (xml.sax.xmlreader.Attributes method)
@anchor{library/xml sax reader xml sax xmlreader Attributes getNames}@anchor{2fde}
@deffn {Method} Attributes.getNames ()

Return the names of the attributes.
@end deffn

@geindex getType() (xml.sax.xmlreader.Attributes method)
@anchor{library/xml sax reader xml sax xmlreader Attributes getType}@anchor{2fdf}
@deffn {Method} Attributes.getType (name)

Returns the type of the attribute @emph{name}, which is normally @code{'CDATA'}.
@end deffn

@geindex getValue() (xml.sax.xmlreader.Attributes method)
@anchor{library/xml sax reader xml sax xmlreader Attributes getValue}@anchor{2fe0}
@deffn {Method} Attributes.getValue (name)

Return the value of attribute @emph{name}.
@end deffn

@c getValueByQName, getNameByQName, getQNameByName, getQNames available

@c here already, but documented only for derived class.

@node The AttributesNS Interface,,The Attributes Interface,xml sax xmlreader — Interface for XML parsers
@anchor{library/xml sax reader attributes-ns-objects}@anchor{2f90}@anchor{library/xml sax reader the-attributesns-interface}@anchor{2fe1}
@subsubsection The @code{AttributesNS} Interface


This interface is a subtype of the @code{Attributes} interface (see section
@ref{2f8e,,The Attributes Interface}).  All methods supported by that interface are also
available on @code{AttributesNS} objects.

The following methods are also available:

@geindex getValueByQName() (xml.sax.xmlreader.AttributesNS method)
@anchor{library/xml sax reader xml sax xmlreader AttributesNS getValueByQName}@anchor{2fe2}
@deffn {Method} AttributesNS.getValueByQName (name)

Return the value for a qualified name.
@end deffn

@geindex getNameByQName() (xml.sax.xmlreader.AttributesNS method)
@anchor{library/xml sax reader xml sax xmlreader AttributesNS getNameByQName}@anchor{2fe3}
@deffn {Method} AttributesNS.getNameByQName (name)

Return the @code{(namespace, localname)} pair for a qualified @emph{name}.
@end deffn

@geindex getQNameByName() (xml.sax.xmlreader.AttributesNS method)
@anchor{library/xml sax reader xml sax xmlreader AttributesNS getQNameByName}@anchor{2fe4}
@deffn {Method} AttributesNS.getQNameByName (name)

Return the qualified name for a @code{(namespace, localname)} pair.
@end deffn

@geindex getQNames() (xml.sax.xmlreader.AttributesNS method)
@anchor{library/xml sax reader xml sax xmlreader AttributesNS getQNames}@anchor{2fe5}
@deffn {Method} AttributesNS.getQNames ()

Return the qualified names of all attributes.
@end deffn

@node xml parsers expat — Fast XML parsing using Expat,,xml sax xmlreader — Interface for XML parsers,Structured Markup Processing Tools
@anchor{library/pyexpat doc}@anchor{2fe6}@anchor{library/pyexpat module-xml parsers expat}@anchor{13f}@anchor{library/pyexpat xml-parsers-expat-fast-xml-parsing-using-expat}@anchor{2fe7}
@subsection @code{xml.parsers.expat} — Fast XML parsing using Expat


@geindex module; xml.parsers.expat


__________________________________________________________________


@c Markup notes:
@c 
@c Many of the attributes of the XMLParser objects are callbacks.  Since
@c signature information must be presented, these are described using the method
@c directive.  Since they are attributes which are set by client code, in-text
@c references to these attributes should be marked using the :member: role.

@cartouche
@quotation Warning 
The @code{pyexpat} module is not secure against maliciously
constructed data.  If you need to parse untrusted or unauthenticated data see
@ref{2e71,,XML vulnerabilities}.
@end quotation
@end cartouche

@geindex Expat

The @ref{13f,,xml.parsers.expat} module is a Python interface to the Expat
non-validating XML parser. The module provides a single extension type,
@code{xmlparser}, that represents the current state of an XML parser.  After
an @code{xmlparser} object has been created, various attributes of the object
can be set to handler functions.  When an XML document is then fed to the
parser, the handler functions are called for the character data and markup in
the XML document.

@geindex module; pyexpat

This module uses the @code{pyexpat} module to provide access to the Expat
parser.  Direct use of the @code{pyexpat} module is deprecated.

This module provides one exception and one type object:

@geindex ExpatError
@anchor{library/pyexpat xml parsers expat ExpatError}@anchor{e27}
@deffn {Exception} xml.parsers.expat.ExpatError

The exception raised when Expat reports an error.  See section
@ref{2fe8,,ExpatError Exceptions} for more information on interpreting Expat errors.
@end deffn

@geindex error
@anchor{library/pyexpat xml parsers expat error}@anchor{2fe9}
@deffn {Exception} xml.parsers.expat.error

Alias for @ref{e27,,ExpatError}.
@end deffn

@geindex XMLParserType (in module xml.parsers.expat)
@anchor{library/pyexpat xml parsers expat XMLParserType}@anchor{2fea}
@deffn {Data} xml.parsers.expat.XMLParserType

The type of the return values from the @ref{2feb,,ParserCreate()} function.
@end deffn

The @ref{13f,,xml.parsers.expat} module contains two functions:

@geindex ErrorString() (in module xml.parsers.expat)
@anchor{library/pyexpat xml parsers expat ErrorString}@anchor{2fec}
@deffn {Function} xml.parsers.expat.ErrorString (errno)

Returns an explanatory string for a given error number @emph{errno}.
@end deffn

@geindex ParserCreate() (in module xml.parsers.expat)
@anchor{library/pyexpat xml parsers expat ParserCreate}@anchor{2feb}
@deffn {Function} xml.parsers.expat.ParserCreate (encoding=None, namespace_separator=None)

Creates and returns a new @code{xmlparser} object.   @emph{encoding}, if specified,
must be a string naming the encoding  used by the XML data.  Expat doesn’t
support as many encodings as Python does, and its repertoire of encodings can’t
be extended; it supports UTF-8, UTF-16, ISO-8859-1 (Latin1), and ASCII.  If
@emph{encoding} @footnote{@w{(1)} 
The encoding string included in XML output should conform to the
appropriate standards. For example, “UTF-8” is valid, but “UTF8” is
not. See @indicateurl{https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl}
and @indicateurl{https://www.iana.org/assignments/character-sets/character-sets.xhtml}.
} is given it will override the implicit or explicit encoding of the
document.

Expat can optionally do XML namespace processing for you, enabled by providing a
value for @emph{namespace_separator}.  The value must be a one-character string; a
@ref{17c,,ValueError} will be raised if the string has an illegal length (@code{None}
is considered the same as omission).  When namespace processing is enabled,
element type names and attribute names that belong to a namespace will be
expanded.  The element name passed to the element handlers
@code{StartElementHandler} and @code{EndElementHandler} will be the
concatenation of the namespace URI, the namespace separator character, and the
local part of the name.  If the namespace separator is a zero byte (@code{chr(0)})
then the namespace URI and the local part will be concatenated without any
separator.

For example, if @emph{namespace_separator} is set to a space character (@code{' '}) and
the following document is parsed:

@example
<?xml version="1.0"?>
<root xmlns    = "http://default-namespace.org/"
      xmlns:py = "http://www.python.org/ns/">
  <py:elem1 />
  <elem2 xmlns="" />
</root>
@end example

@code{StartElementHandler} will receive the following strings for each
element:

@example
http://default-namespace.org/ root
http://www.python.org/ns/ elem1
elem2
@end example

Due to limitations in the @code{Expat} library used by @code{pyexpat},
the @code{xmlparser} instance returned can only be used to parse a single
XML document.  Call @code{ParserCreate} for each document to provide unique
parser instances.
@end deffn


@subsubheading See also



@table @asis

@item The Expat XML Parser@footnote{http://www.libexpat.org/}

Home page of the Expat project.
@end table


@menu
* XMLParser Objects: XMLParser Objects<2>. 
* ExpatError Exceptions:: 
* Example: Example<11>. 
* Content Model Descriptions:: 
* Expat error constants:: 

@end menu

@node XMLParser Objects<2>,ExpatError Exceptions,,xml parsers expat — Fast XML parsing using Expat
@anchor{library/pyexpat id2}@anchor{2fed}@anchor{library/pyexpat xmlparser-objects}@anchor{2fee}
@subsubsection XMLParser Objects


@code{xmlparser} objects have the following methods:

@geindex Parse() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser Parse}@anchor{2fef}
@deffn {Method} xmlparser.Parse (data[, isfinal])

Parses the contents of the string @emph{data}, calling the appropriate handler
functions to process the parsed data.  @emph{isfinal} must be true on the final call
to this method; it allows the parsing of a single file in fragments,
not the submission of multiple files.
@emph{data} can be the empty string at any time.
@end deffn

@geindex ParseFile() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser ParseFile}@anchor{2ff0}
@deffn {Method} xmlparser.ParseFile (file)

Parse XML data reading from the object @emph{file}.  @emph{file} only needs to provide
the @code{read(nbytes)} method, returning the empty string when there’s no more
data.
@end deffn

@geindex SetBase() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser SetBase}@anchor{2ff1}
@deffn {Method} xmlparser.SetBase (base)

Sets the base to be used for resolving relative URIs in system identifiers in
declarations.  Resolving relative identifiers is left to the application: this
value will be passed through as the @emph{base} argument to the
@ref{2ff2,,ExternalEntityRefHandler()}, @ref{2ff3,,NotationDeclHandler()}, and
@ref{2ff4,,UnparsedEntityDeclHandler()} functions.
@end deffn

@geindex GetBase() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser GetBase}@anchor{2ff5}
@deffn {Method} xmlparser.GetBase ()

Returns a string containing the base set by a previous call to @ref{2ff1,,SetBase()},
or @code{None} if  @ref{2ff1,,SetBase()} hasn’t been called.
@end deffn

@geindex GetInputContext() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser GetInputContext}@anchor{2ff6}
@deffn {Method} xmlparser.GetInputContext ()

Returns the input data that generated the current event as a string. The data is
in the encoding of the entity which contains the text. When called while an
event handler is not active, the return value is @code{None}.
@end deffn

@geindex ExternalEntityParserCreate() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser ExternalEntityParserCreate}@anchor{2ff7}
@deffn {Method} xmlparser.ExternalEntityParserCreate (context[, encoding])

Create a “child” parser which can be used to parse an external parsed entity
referred to by content parsed by the parent parser.  The @emph{context} parameter
should be the string passed to the @ref{2ff2,,ExternalEntityRefHandler()} handler
function, described below. The child parser is created with the
@ref{2ff8,,ordered_attributes} and @ref{2ff9,,specified_attributes} set to the values of
this parser.
@end deffn

@geindex SetParamEntityParsing() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser SetParamEntityParsing}@anchor{2ffa}
@deffn {Method} xmlparser.SetParamEntityParsing (flag)

Control parsing of parameter entities (including the external DTD subset).
Possible @emph{flag} values are @code{XML_PARAM_ENTITY_PARSING_NEVER},
@code{XML_PARAM_ENTITY_PARSING_UNLESS_STANDALONE} and
@code{XML_PARAM_ENTITY_PARSING_ALWAYS}.  Return true if setting the flag
was successful.
@end deffn

@geindex UseForeignDTD() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser UseForeignDTD}@anchor{2ffb}
@deffn {Method} xmlparser.UseForeignDTD ([flag])

Calling this with a true value for @emph{flag} (the default) will cause Expat to call
the @ref{2ff2,,ExternalEntityRefHandler} with @ref{182,,None} for all arguments to
allow an alternate DTD to be loaded.  If the document does not contain a
document type declaration, the @ref{2ff2,,ExternalEntityRefHandler} will still be
called, but the @ref{2ffc,,StartDoctypeDeclHandler} and
@ref{2ffd,,EndDoctypeDeclHandler} will not be called.

Passing a false value for @emph{flag} will cancel a previous call that passed a true
value, but otherwise has no effect.

This method can only be called before the @ref{2fef,,Parse()} or @ref{2ff0,,ParseFile()}
methods are called; calling it after either of those have been called causes
@ref{e27,,ExpatError} to be raised with the @ref{19,,code} attribute set to
@code{errors.codes[errors.XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING]}.
@end deffn

@code{xmlparser} objects have the following attributes:

@geindex buffer_size (xml.parsers.expat.xmlparser attribute)
@anchor{library/pyexpat xml parsers expat xmlparser buffer_size}@anchor{2ffe}
@deffn {Attribute} xmlparser.buffer_size

The size of the buffer used when @ref{2fff,,buffer_text} is true.
A new buffer size can be set by assigning a new integer value
to this attribute.
When the size is changed, the buffer will be flushed.
@end deffn

@geindex buffer_text (xml.parsers.expat.xmlparser attribute)
@anchor{library/pyexpat xml parsers expat xmlparser buffer_text}@anchor{2fff}
@deffn {Attribute} xmlparser.buffer_text

Setting this to true causes the @code{xmlparser} object to buffer textual
content returned by Expat to avoid multiple calls to the
@ref{3000,,CharacterDataHandler()} callback whenever possible.  This can improve
performance substantially since Expat normally breaks character data into chunks
at every line ending.  This attribute is false by default, and may be changed at
any time.
@end deffn

@geindex buffer_used (xml.parsers.expat.xmlparser attribute)
@anchor{library/pyexpat xml parsers expat xmlparser buffer_used}@anchor{3001}
@deffn {Attribute} xmlparser.buffer_used

If @ref{2fff,,buffer_text} is enabled, the number of bytes stored in the buffer.
These bytes represent UTF-8 encoded text.  This attribute has no meaningful
interpretation when @ref{2fff,,buffer_text} is false.
@end deffn

@geindex ordered_attributes (xml.parsers.expat.xmlparser attribute)
@anchor{library/pyexpat xml parsers expat xmlparser ordered_attributes}@anchor{2ff8}
@deffn {Attribute} xmlparser.ordered_attributes

Setting this attribute to a non-zero integer causes the attributes to be
reported as a list rather than a dictionary.  The attributes are presented in
the order found in the document text.  For each attribute, two list entries are
presented: the attribute name and the attribute value.  (Older versions of this
module also used this format.)  By default, this attribute is false; it may be
changed at any time.
@end deffn

@geindex specified_attributes (xml.parsers.expat.xmlparser attribute)
@anchor{library/pyexpat xml parsers expat xmlparser specified_attributes}@anchor{2ff9}
@deffn {Attribute} xmlparser.specified_attributes

If set to a non-zero integer, the parser will report only those attributes which
were specified in the document instance and not those which were derived from
attribute declarations.  Applications which set this need to be especially
careful to use what additional information is available from the declarations as
needed to comply with the standards for the behavior of XML processors.  By
default, this attribute is false; it may be changed at any time.
@end deffn

The following attributes contain values relating to the most recent error
encountered by an @code{xmlparser} object, and will only have correct values
once a call to @code{Parse()} or @code{ParseFile()} has raised an
@ref{e27,,xml.parsers.expat.ExpatError} exception.

@geindex ErrorByteIndex (xml.parsers.expat.xmlparser attribute)
@anchor{library/pyexpat xml parsers expat xmlparser ErrorByteIndex}@anchor{3002}
@deffn {Attribute} xmlparser.ErrorByteIndex

Byte index at which an error occurred.
@end deffn

@geindex ErrorCode (xml.parsers.expat.xmlparser attribute)
@anchor{library/pyexpat xml parsers expat xmlparser ErrorCode}@anchor{3003}
@deffn {Attribute} xmlparser.ErrorCode

Numeric code specifying the problem.  This value can be passed to the
@ref{2fec,,ErrorString()} function, or compared to one of the constants defined in the
@code{errors} object.
@end deffn

@geindex ErrorColumnNumber (xml.parsers.expat.xmlparser attribute)
@anchor{library/pyexpat xml parsers expat xmlparser ErrorColumnNumber}@anchor{3004}
@deffn {Attribute} xmlparser.ErrorColumnNumber

Column number at which an error occurred.
@end deffn

@geindex ErrorLineNumber (xml.parsers.expat.xmlparser attribute)
@anchor{library/pyexpat xml parsers expat xmlparser ErrorLineNumber}@anchor{3005}
@deffn {Attribute} xmlparser.ErrorLineNumber

Line number at which an error occurred.
@end deffn

The following attributes contain values relating to the current parse location
in an @code{xmlparser} object.  During a callback reporting a parse event they
indicate the location of the first of the sequence of characters that generated
the event.  When called outside of a callback, the position indicated will be
just past the last parse event (regardless of whether there was an associated
callback).

@geindex CurrentByteIndex (xml.parsers.expat.xmlparser attribute)
@anchor{library/pyexpat xml parsers expat xmlparser CurrentByteIndex}@anchor{3006}
@deffn {Attribute} xmlparser.CurrentByteIndex

Current byte index in the parser input.
@end deffn

@geindex CurrentColumnNumber (xml.parsers.expat.xmlparser attribute)
@anchor{library/pyexpat xml parsers expat xmlparser CurrentColumnNumber}@anchor{3007}
@deffn {Attribute} xmlparser.CurrentColumnNumber

Current column number in the parser input.
@end deffn

@geindex CurrentLineNumber (xml.parsers.expat.xmlparser attribute)
@anchor{library/pyexpat xml parsers expat xmlparser CurrentLineNumber}@anchor{3008}
@deffn {Attribute} xmlparser.CurrentLineNumber

Current line number in the parser input.
@end deffn

Here is the list of handlers that can be set.  To set a handler on an
@code{xmlparser} object @emph{o}, use @code{o.handlername = func}.  @emph{handlername} must
be taken from the following list, and @emph{func} must be a callable object accepting
the correct number of arguments.  The arguments are all strings, unless
otherwise stated.

@geindex XmlDeclHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser XmlDeclHandler}@anchor{3009}
@deffn {Method} xmlparser.XmlDeclHandler (version, encoding, standalone)

Called when the XML declaration is parsed.  The XML declaration is the
(optional) declaration of the applicable version of the XML recommendation, the
encoding of the document text, and an optional “standalone” declaration.
@emph{version} and @emph{encoding} will be strings, and @emph{standalone} will be @code{1} if the
document is declared standalone, @code{0} if it is declared not to be standalone,
or @code{-1} if the standalone clause was omitted. This is only available with
Expat version 1.95.0 or newer.
@end deffn

@geindex StartDoctypeDeclHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser StartDoctypeDeclHandler}@anchor{2ffc}
@deffn {Method} xmlparser.StartDoctypeDeclHandler (doctypeName, systemId, publicId, has_internal_subset)

Called when Expat begins parsing the document type declaration (@code{<!DOCTYPE
...}).  The @emph{doctypeName} is provided exactly as presented.  The @emph{systemId} and
@emph{publicId} parameters give the system and public identifiers if specified, or
@code{None} if omitted.  @emph{has_internal_subset} will be true if the document
contains and internal document declaration subset. This requires Expat version
1.2 or newer.
@end deffn

@geindex EndDoctypeDeclHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser EndDoctypeDeclHandler}@anchor{2ffd}
@deffn {Method} xmlparser.EndDoctypeDeclHandler ()

Called when Expat is done parsing the document type declaration. This requires
Expat version 1.2 or newer.
@end deffn

@geindex ElementDeclHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser ElementDeclHandler}@anchor{300a}
@deffn {Method} xmlparser.ElementDeclHandler (name, model)

Called once for each element type declaration.  @emph{name} is the name of the
element type, and @emph{model} is a representation of the content model.
@end deffn

@geindex AttlistDeclHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser AttlistDeclHandler}@anchor{300b}
@deffn {Method} xmlparser.AttlistDeclHandler (elname, attname, type, default, required)

Called for each declared attribute for an element type.  If an attribute list
declaration declares three attributes, this handler is called three times, once
for each attribute.  @emph{elname} is the name of the element to which the
declaration applies and @emph{attname} is the name of the attribute declared.  The
attribute type is a string passed as @emph{type}; the possible values are
@code{'CDATA'}, @code{'ID'}, @code{'IDREF'}, … @emph{default} gives the default value for
the attribute used when the attribute is not specified by the document instance,
or @code{None} if there is no default value (@code{#IMPLIED} values).  If the
attribute is required to be given in the document instance, @emph{required} will be
true. This requires Expat version 1.95.0 or newer.
@end deffn

@geindex StartElementHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser StartElementHandler}@anchor{300c}
@deffn {Method} xmlparser.StartElementHandler (name, attributes)

Called for the start of every element.  @emph{name} is a string containing the
element name, and @emph{attributes} is the element attributes. If
@ref{2ff8,,ordered_attributes} is true, this is a list (see
@ref{2ff8,,ordered_attributes} for a full description). Otherwise it’s a
dictionary mapping names to values.
@end deffn

@geindex EndElementHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser EndElementHandler}@anchor{300d}
@deffn {Method} xmlparser.EndElementHandler (name)

Called for the end of every element.
@end deffn

@geindex ProcessingInstructionHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser ProcessingInstructionHandler}@anchor{300e}
@deffn {Method} xmlparser.ProcessingInstructionHandler (target, data)

Called for every processing instruction.
@end deffn

@geindex CharacterDataHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser CharacterDataHandler}@anchor{3000}
@deffn {Method} xmlparser.CharacterDataHandler (data)

Called for character data.  This will be called for normal character data, CDATA
marked content, and ignorable whitespace.  Applications which must distinguish
these cases can use the @ref{300f,,StartCdataSectionHandler},
@ref{3010,,EndCdataSectionHandler}, and @ref{300a,,ElementDeclHandler} callbacks to
collect the required information.
@end deffn

@geindex UnparsedEntityDeclHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser UnparsedEntityDeclHandler}@anchor{2ff4}
@deffn {Method} xmlparser.UnparsedEntityDeclHandler (entityName, base, systemId, publicId, notationName)

Called for unparsed (NDATA) entity declarations.  This is only present for
version 1.2 of the Expat library; for more recent versions, use
@ref{3011,,EntityDeclHandler} instead.  (The underlying function in the Expat
library has been declared obsolete.)
@end deffn

@geindex EntityDeclHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser EntityDeclHandler}@anchor{3011}
@deffn {Method} xmlparser.EntityDeclHandler (entityName, is_parameter_entity, value, base, systemId, publicId, notationName)

Called for all entity declarations.  For parameter and internal entities,
@emph{value} will be a string giving the declared contents of the entity; this will
be @code{None} for external entities.  The @emph{notationName} parameter will be
@code{None} for parsed entities, and the name of the notation for unparsed
entities. @emph{is_parameter_entity} will be true if the entity is a parameter entity
or false for general entities (most applications only need to be concerned with
general entities). This is only available starting with version 1.95.0 of the
Expat library.
@end deffn

@geindex NotationDeclHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser NotationDeclHandler}@anchor{2ff3}
@deffn {Method} xmlparser.NotationDeclHandler (notationName, base, systemId, publicId)

Called for notation declarations.  @emph{notationName}, @emph{base}, and @emph{systemId}, and
@emph{publicId} are strings if given.  If the public identifier is omitted,
@emph{publicId} will be @code{None}.
@end deffn

@geindex StartNamespaceDeclHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser StartNamespaceDeclHandler}@anchor{3012}
@deffn {Method} xmlparser.StartNamespaceDeclHandler (prefix, uri)

Called when an element contains a namespace declaration.  Namespace declarations
are processed before the @ref{300c,,StartElementHandler} is called for the element
on which declarations are placed.
@end deffn

@geindex EndNamespaceDeclHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser EndNamespaceDeclHandler}@anchor{3013}
@deffn {Method} xmlparser.EndNamespaceDeclHandler (prefix)

Called when the closing tag is reached for an element  that contained a
namespace declaration.  This is called once for each namespace declaration on
the element in the reverse of the order for which the
@ref{3012,,StartNamespaceDeclHandler} was called to indicate the start of each
namespace declaration’s scope.  Calls to this handler are made after the
corresponding @ref{300d,,EndElementHandler} for the end of the element.
@end deffn

@geindex CommentHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser CommentHandler}@anchor{3014}
@deffn {Method} xmlparser.CommentHandler (data)

Called for comments.  @emph{data} is the text of the comment, excluding the leading
@code{'<!-}@code{-'} and trailing @code{'-}@code{->'}.
@end deffn

@geindex StartCdataSectionHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser StartCdataSectionHandler}@anchor{300f}
@deffn {Method} xmlparser.StartCdataSectionHandler ()

Called at the start of a CDATA section.  This and @ref{3010,,EndCdataSectionHandler}
are needed to be able to identify the syntactical start and end for CDATA
sections.
@end deffn

@geindex EndCdataSectionHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser EndCdataSectionHandler}@anchor{3010}
@deffn {Method} xmlparser.EndCdataSectionHandler ()

Called at the end of a CDATA section.
@end deffn

@geindex DefaultHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser DefaultHandler}@anchor{3015}
@deffn {Method} xmlparser.DefaultHandler (data)

Called for any characters in the XML document for which no applicable handler
has been specified.  This means characters that are part of a construct which
could be reported, but for which no handler has been supplied.
@end deffn

@geindex DefaultHandlerExpand() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser DefaultHandlerExpand}@anchor{3016}
@deffn {Method} xmlparser.DefaultHandlerExpand (data)

This is the same as the @ref{3015,,DefaultHandler()},  but doesn’t inhibit expansion
of internal entities. The entity reference will not be passed to the default
handler.
@end deffn

@geindex NotStandaloneHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser NotStandaloneHandler}@anchor{3017}
@deffn {Method} xmlparser.NotStandaloneHandler ()

Called if the XML document hasn’t been declared as being a standalone document.
This happens when there is an external subset or a reference to a parameter
entity, but the XML declaration does not set standalone to @code{yes} in an XML
declaration.  If this handler returns @code{0}, then the parser will raise an
@code{XML_ERROR_NOT_STANDALONE} error.  If this handler is not set, no
exception is raised by the parser for this condition.
@end deffn

@geindex ExternalEntityRefHandler() (xml.parsers.expat.xmlparser method)
@anchor{library/pyexpat xml parsers expat xmlparser ExternalEntityRefHandler}@anchor{2ff2}
@deffn {Method} xmlparser.ExternalEntityRefHandler (context, base, systemId, publicId)

Called for references to external entities.  @emph{base} is the current base, as set
by a previous call to @ref{2ff1,,SetBase()}.  The public and system identifiers,
@emph{systemId} and @emph{publicId}, are strings if given; if the public identifier is not
given, @emph{publicId} will be @code{None}.  The @emph{context} value is opaque and should
only be used as described below.

For external entities to be parsed, this handler must be implemented. It is
responsible for creating the sub-parser using
@code{ExternalEntityParserCreate(context)}, initializing it with the appropriate
callbacks, and parsing the entity.  This handler should return an integer; if it
returns @code{0}, the parser will raise an
@code{XML_ERROR_EXTERNAL_ENTITY_HANDLING} error, otherwise parsing will
continue.

If this handler is not provided, external entities are reported by the
@ref{3015,,DefaultHandler} callback, if provided.
@end deffn

@node ExpatError Exceptions,Example<11>,XMLParser Objects<2>,xml parsers expat — Fast XML parsing using Expat
@anchor{library/pyexpat expaterror-exceptions}@anchor{3018}@anchor{library/pyexpat expaterror-objects}@anchor{2fe8}
@subsubsection ExpatError Exceptions


@ref{e27,,ExpatError} exceptions have a number of interesting attributes:

@geindex code (xml.parsers.expat.ExpatError attribute)
@anchor{library/pyexpat xml parsers expat ExpatError code}@anchor{3019}
@deffn {Attribute} ExpatError.code

Expat’s internal error number for the specific error.  The
@ref{301a,,errors.messages} dictionary maps
these error numbers to Expat’s error messages.  For example:

@example
from xml.parsers.expat import ParserCreate, ExpatError, errors

p = ParserCreate()
try:
    p.Parse(some_xml_document)
except ExpatError as err:
    print("Error:", errors.messages[err.code])
@end example

The @ref{140,,errors} module also provides error message
constants and a dictionary @ref{301b,,codes} mapping
these messages back to the error codes, see below.
@end deffn

@geindex lineno (xml.parsers.expat.ExpatError attribute)
@anchor{library/pyexpat xml parsers expat ExpatError lineno}@anchor{301c}
@deffn {Attribute} ExpatError.lineno

Line number on which the error was detected.  The first line is numbered @code{1}.
@end deffn

@geindex offset (xml.parsers.expat.ExpatError attribute)
@anchor{library/pyexpat xml parsers expat ExpatError offset}@anchor{301d}
@deffn {Attribute} ExpatError.offset

Character offset into the line where the error occurred.  The first column is
numbered @code{0}.
@end deffn

@node Example<11>,Content Model Descriptions,ExpatError Exceptions,xml parsers expat — Fast XML parsing using Expat
@anchor{library/pyexpat example}@anchor{301e}@anchor{library/pyexpat expat-example}@anchor{301f}
@subsubsection Example


The following program defines three handlers that just print out their
arguments.

@example
import xml.parsers.expat

# 3 handler functions
def start_element(name, attrs):
    print('Start element:', name, attrs)
def end_element(name):
    print('End element:', name)
def char_data(data):
    print('Character data:', repr(data))

p = xml.parsers.expat.ParserCreate()

p.StartElementHandler = start_element
p.EndElementHandler = end_element
p.CharacterDataHandler = char_data

p.Parse("""<?xml version="1.0"?>
<parent id="top"><child1 name="paul">Text goes here</child1>
<child2 name="fred">More text</child2>
</parent>""", 1)
@end example

The output from this program is:

@example
Start element: parent @{'id': 'top'@}
Start element: child1 @{'name': 'paul'@}
Character data: 'Text goes here'
End element: child1
Character data: '\n'
Start element: child2 @{'name': 'fred'@}
Character data: 'More text'
End element: child2
Character data: '\n'
End element: parent
@end example

@node Content Model Descriptions,Expat error constants,Example<11>,xml parsers expat — Fast XML parsing using Expat
@anchor{library/pyexpat content-model-descriptions}@anchor{3020}@anchor{library/pyexpat expat-content-models}@anchor{3021}@anchor{library/pyexpat module-xml parsers expat model}@anchor{141}
@subsubsection Content Model Descriptions


@geindex module; xml.parsers.expat.model

Content models are described using nested tuples.  Each tuple contains four
values: the type, the quantifier, the name, and a tuple of children.  Children
are simply additional content model descriptions.

The values of the first two fields are constants defined in the
@ref{141,,xml.parsers.expat.model} module.  These constants can be collected in two
groups: the model type group and the quantifier group.

The constants in the model type group are:


@deffn {Data} xml.parsers.expat.model.XML_CTYPE_ANY

The element named by the model name was declared to have a content model of
@code{ANY}.
@end deffn


@deffn {Data} xml.parsers.expat.model.XML_CTYPE_CHOICE

The named element allows a choice from a number of options; this is used for
content models such as @code{(A | B | C)}.
@end deffn


@deffn {Data} xml.parsers.expat.model.XML_CTYPE_EMPTY

Elements which are declared to be @code{EMPTY} have this model type.
@end deffn


@deffn {Data} xml.parsers.expat.model.XML_CTYPE_MIXED
@end deffn


@deffn {Data} xml.parsers.expat.model.XML_CTYPE_NAME
@end deffn


@deffn {Data} xml.parsers.expat.model.XML_CTYPE_SEQ

Models which represent a series of models which follow one after the other are
indicated with this model type.  This is used for models such as @code{(A, B, C)}.
@end deffn

The constants in the quantifier group are:


@deffn {Data} xml.parsers.expat.model.XML_CQUANT_NONE

No modifier is given, so it can appear exactly once, as for @code{A}.
@end deffn


@deffn {Data} xml.parsers.expat.model.XML_CQUANT_OPT

The model is optional: it can appear once or not at all, as for @code{A?}.
@end deffn


@deffn {Data} xml.parsers.expat.model.XML_CQUANT_PLUS

The model must occur one or more times (like @code{A+}).
@end deffn


@deffn {Data} xml.parsers.expat.model.XML_CQUANT_REP

The model must occur zero or more times, as for @code{A*}.
@end deffn

@node Expat error constants,,Content Model Descriptions,xml parsers expat — Fast XML parsing using Expat
@anchor{library/pyexpat expat-error-constants}@anchor{3022}@anchor{library/pyexpat expat-errors}@anchor{3023}@anchor{library/pyexpat module-xml parsers expat errors}@anchor{140}
@subsubsection Expat error constants


@geindex module; xml.parsers.expat.errors

The following constants are provided in the @ref{140,,xml.parsers.expat.errors}
module.  These constants are useful in interpreting some of the attributes of
the @code{ExpatError} exception objects raised when an error has occurred.
Since for backwards compatibility reasons, the constants’ value is the error
@emph{message} and not the numeric error @emph{code}, you do this by comparing its
@ref{19,,code} attribute with
@code{errors.codes[errors.XML_ERROR_@emph{CONSTANT_NAME}]}.

The @code{errors} module has the following attributes:

@geindex codes (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors codes}@anchor{301b}
@deffn {Data} xml.parsers.expat.errors.codes

A dictionary mapping string descriptions to their error codes.


New in version 3.2.

@end deffn

@geindex messages (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors messages}@anchor{301a}
@deffn {Data} xml.parsers.expat.errors.messages

A dictionary mapping numeric error codes to their string descriptions.


New in version 3.2.

@end deffn

@geindex XML_ERROR_ASYNC_ENTITY (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_ASYNC_ENTITY}@anchor{3024}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_ASYNC_ENTITY
@end deffn

@geindex XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF}@anchor{3025}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF

An entity reference in an attribute value referred to an external entity instead
of an internal entity.
@end deffn

@geindex XML_ERROR_BAD_CHAR_REF (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_BAD_CHAR_REF}@anchor{3026}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_BAD_CHAR_REF

A character reference referred to a character which is illegal in XML (for
example, character @code{0}, or ‘@code{&#0;}’).
@end deffn

@geindex XML_ERROR_BINARY_ENTITY_REF (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_BINARY_ENTITY_REF}@anchor{3027}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_BINARY_ENTITY_REF

An entity reference referred to an entity which was declared with a notation, so
cannot be parsed.
@end deffn

@geindex XML_ERROR_DUPLICATE_ATTRIBUTE (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_DUPLICATE_ATTRIBUTE}@anchor{3028}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_DUPLICATE_ATTRIBUTE

An attribute was used more than once in a start tag.
@end deffn

@geindex XML_ERROR_INCORRECT_ENCODING (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_INCORRECT_ENCODING}@anchor{3029}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_INCORRECT_ENCODING
@end deffn

@geindex XML_ERROR_INVALID_TOKEN (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_INVALID_TOKEN}@anchor{302a}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_INVALID_TOKEN

Raised when an input byte could not properly be assigned to a character; for
example, a NUL byte (value @code{0}) in a UTF-8 input stream.
@end deffn

@geindex XML_ERROR_JUNK_AFTER_DOC_ELEMENT (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_JUNK_AFTER_DOC_ELEMENT}@anchor{302b}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_JUNK_AFTER_DOC_ELEMENT

Something other than whitespace occurred after the document element.
@end deffn

@geindex XML_ERROR_MISPLACED_XML_PI (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_MISPLACED_XML_PI}@anchor{302c}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_MISPLACED_XML_PI

An XML declaration was found somewhere other than the start of the input data.
@end deffn

@geindex XML_ERROR_NO_ELEMENTS (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_NO_ELEMENTS}@anchor{302d}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_NO_ELEMENTS

The document contains no elements (XML requires all documents to contain exactly
one top-level element)..
@end deffn

@geindex XML_ERROR_NO_MEMORY (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_NO_MEMORY}@anchor{302e}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_NO_MEMORY

Expat was not able to allocate memory internally.
@end deffn

@geindex XML_ERROR_PARAM_ENTITY_REF (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_PARAM_ENTITY_REF}@anchor{302f}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_PARAM_ENTITY_REF

A parameter entity reference was found where it was not allowed.
@end deffn

@geindex XML_ERROR_PARTIAL_CHAR (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_PARTIAL_CHAR}@anchor{3030}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_PARTIAL_CHAR

An incomplete character was found in the input.
@end deffn

@geindex XML_ERROR_RECURSIVE_ENTITY_REF (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_RECURSIVE_ENTITY_REF}@anchor{3031}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_RECURSIVE_ENTITY_REF

An entity reference contained another reference to the same entity; possibly via
a different name, and possibly indirectly.
@end deffn

@geindex XML_ERROR_SYNTAX (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_SYNTAX}@anchor{3032}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_SYNTAX

Some unspecified syntax error was encountered.
@end deffn

@geindex XML_ERROR_TAG_MISMATCH (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_TAG_MISMATCH}@anchor{3033}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_TAG_MISMATCH

An end tag did not match the innermost open start tag.
@end deffn

@geindex XML_ERROR_UNCLOSED_TOKEN (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_UNCLOSED_TOKEN}@anchor{3034}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_UNCLOSED_TOKEN

Some token (such as a start tag) was not closed before the end of the stream or
the next token was encountered.
@end deffn

@geindex XML_ERROR_UNDEFINED_ENTITY (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_UNDEFINED_ENTITY}@anchor{3035}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_UNDEFINED_ENTITY

A reference was made to an entity which was not defined.
@end deffn

@geindex XML_ERROR_UNKNOWN_ENCODING (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_UNKNOWN_ENCODING}@anchor{3036}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_UNKNOWN_ENCODING

The document encoding is not supported by Expat.
@end deffn

@geindex XML_ERROR_UNCLOSED_CDATA_SECTION (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_UNCLOSED_CDATA_SECTION}@anchor{3037}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_UNCLOSED_CDATA_SECTION

A CDATA marked section was not closed.
@end deffn

@geindex XML_ERROR_EXTERNAL_ENTITY_HANDLING (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_EXTERNAL_ENTITY_HANDLING}@anchor{3038}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_EXTERNAL_ENTITY_HANDLING
@end deffn

@geindex XML_ERROR_NOT_STANDALONE (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_NOT_STANDALONE}@anchor{3039}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_NOT_STANDALONE

The parser determined that the document was not “standalone” though it declared
itself to be in the XML declaration, and the @code{NotStandaloneHandler} was
set and returned @code{0}.
@end deffn

@geindex XML_ERROR_UNEXPECTED_STATE (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_UNEXPECTED_STATE}@anchor{303a}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_UNEXPECTED_STATE
@end deffn

@geindex XML_ERROR_ENTITY_DECLARED_IN_PE (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_ENTITY_DECLARED_IN_PE}@anchor{303b}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_ENTITY_DECLARED_IN_PE
@end deffn

@geindex XML_ERROR_FEATURE_REQUIRES_XML_DTD (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_FEATURE_REQUIRES_XML_DTD}@anchor{303c}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_FEATURE_REQUIRES_XML_DTD

An operation was requested that requires DTD support to be compiled in, but
Expat was configured without DTD support.  This should never be reported by a
standard build of the @ref{13f,,xml.parsers.expat} module.
@end deffn

@geindex XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING}@anchor{303d}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING

A behavioral change was requested after parsing started that can only be changed
before parsing has started.  This is (currently) only raised by
@code{UseForeignDTD()}.
@end deffn

@geindex XML_ERROR_UNBOUND_PREFIX (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_UNBOUND_PREFIX}@anchor{303e}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_UNBOUND_PREFIX

An undeclared prefix was found when namespace processing was enabled.
@end deffn

@geindex XML_ERROR_UNDECLARING_PREFIX (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_UNDECLARING_PREFIX}@anchor{303f}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_UNDECLARING_PREFIX

The document attempted to remove the namespace declaration associated with a
prefix.
@end deffn

@geindex XML_ERROR_INCOMPLETE_PE (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_INCOMPLETE_PE}@anchor{3040}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_INCOMPLETE_PE

A parameter entity contained incomplete markup.
@end deffn

@geindex XML_ERROR_XML_DECL (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_XML_DECL}@anchor{3041}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_XML_DECL

The document contained no document element at all.
@end deffn

@geindex XML_ERROR_TEXT_DECL (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_TEXT_DECL}@anchor{3042}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_TEXT_DECL

There was an error parsing a text declaration in an external entity.
@end deffn

@geindex XML_ERROR_PUBLICID (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_PUBLICID}@anchor{3043}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_PUBLICID

Characters were found in the public id that are not allowed.
@end deffn

@geindex XML_ERROR_SUSPENDED (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_SUSPENDED}@anchor{3044}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_SUSPENDED

The requested operation was made on a suspended parser, but isn’t allowed.  This
includes attempts to provide additional input or to stop the parser.
@end deffn

@geindex XML_ERROR_NOT_SUSPENDED (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_NOT_SUSPENDED}@anchor{3045}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_NOT_SUSPENDED

An attempt to resume the parser was made when the parser had not been suspended.
@end deffn

@geindex XML_ERROR_ABORTED (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_ABORTED}@anchor{3046}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_ABORTED

This should not be reported to Python applications.
@end deffn

@geindex XML_ERROR_FINISHED (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_FINISHED}@anchor{3047}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_FINISHED

The requested operation was made on a parser which was finished parsing input,
but isn’t allowed.  This includes attempts to provide additional input or to
stop the parser.
@end deffn

@geindex XML_ERROR_SUSPEND_PE (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_SUSPEND_PE}@anchor{3048}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_SUSPEND_PE
@end deffn

@geindex XML_ERROR_RESERVED_PREFIX_XML (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_RESERVED_PREFIX_XML}@anchor{3049}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_RESERVED_PREFIX_XML

An attempt was made to
undeclare reserved namespace prefix @code{xml}
or to bind it to another namespace URI.
@end deffn

@geindex XML_ERROR_RESERVED_PREFIX_XMLNS (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_RESERVED_PREFIX_XMLNS}@anchor{304a}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_RESERVED_PREFIX_XMLNS

An attempt was made to declare or undeclare reserved namespace prefix @code{xmlns}.
@end deffn

@geindex XML_ERROR_RESERVED_NAMESPACE_URI (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_RESERVED_NAMESPACE_URI}@anchor{304b}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_RESERVED_NAMESPACE_URI

An attempt was made to bind the URI of one the reserved namespace
prefixes @code{xml} and @code{xmlns} to another namespace prefix.
@end deffn

@geindex XML_ERROR_INVALID_ARGUMENT (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_INVALID_ARGUMENT}@anchor{304c}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_INVALID_ARGUMENT

This should not be reported to Python applications.
@end deffn

@geindex XML_ERROR_NO_BUFFER (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_NO_BUFFER}@anchor{304d}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_NO_BUFFER

This should not be reported to Python applications.
@end deffn

@geindex XML_ERROR_AMPLIFICATION_LIMIT_BREACH (in module xml.parsers.expat.errors)
@anchor{library/pyexpat xml parsers expat errors XML_ERROR_AMPLIFICATION_LIMIT_BREACH}@anchor{304e}
@deffn {Data} xml.parsers.expat.errors.XML_ERROR_AMPLIFICATION_LIMIT_BREACH

The limit on input amplification factor (from DTD and entities)
has been breached.
@end deffn

@node Internet Protocols and Support,Multimedia Services,Structured Markup Processing Tools,The Python Standard Library
@anchor{library/internet doc}@anchor{304f}@anchor{library/internet internet}@anchor{3050}@anchor{library/internet internet-protocols-and-support}@anchor{3051}
@section Internet Protocols and Support


@geindex WWW
@geindex Internet
@geindex World Wide Web

@geindex module; socket

The modules described in this chapter implement internet protocols and  support
for related technology.  They are all implemented in Python. Most of these
modules require the presence of the system-dependent module @ref{ea,,socket}, which
is currently supported on most popular platforms.  Here is an overview:

@menu
* webbrowser — Convenient web-browser controller:: 
* cgi — Common Gateway Interface support:: 
* cgitb — Traceback manager for CGI scripts:: 
* wsgiref — WSGI Utilities and Reference Implementation:: 
* urllib — URL handling modules:: 
* urllib.request — Extensible library for opening URLs: urllib request — Extensible library for opening URLs. 
* urllib.response — Response classes used by urllib: urllib response — Response classes used by urllib. 
* urllib.parse — Parse URLs into components: urllib parse — Parse URLs into components. 
* urllib.error — Exception classes raised by urllib.request: urllib error — Exception classes raised by urllib request. 
* urllib.robotparser — Parser for robots.txt: urllib robotparser — Parser for robots txt. 
* http — HTTP modules:: 
* http.client — HTTP protocol client: http client — HTTP protocol client. 
* ftplib — FTP protocol client:: 
* poplib — POP3 protocol client:: 
* imaplib — IMAP4 protocol client:: 
* nntplib — NNTP protocol client:: 
* smtplib — SMTP protocol client:: 
* smtpd — SMTP Server:: 
* telnetlib — Telnet client:: 
* uuid — UUID objects according to RFC 4122:: 
* socketserver — A framework for network servers:: 
* http.server — HTTP servers: http server — HTTP servers. 
* http.cookies — HTTP state management: http cookies — HTTP state management. 
* http.cookiejar — Cookie handling for HTTP clients: http cookiejar — Cookie handling for HTTP clients. 
* xmlrpc — XMLRPC server and client modules:: 
* xmlrpc.client — XML-RPC client access: xmlrpc client — XML-RPC client access. 
* xmlrpc.server — Basic XML-RPC servers: xmlrpc server — Basic XML-RPC servers. 
* ipaddress — IPv4/IPv6 manipulation library:: 

@end menu

@node webbrowser — Convenient web-browser controller,cgi — Common Gateway Interface support,,Internet Protocols and Support
@anchor{library/webbrowser doc}@anchor{3052}@anchor{library/webbrowser module-webbrowser}@anchor{130}@anchor{library/webbrowser webbrowser-convenient-web-browser-controller}@anchor{3053}
@subsection @code{webbrowser} — Convenient web-browser controller


@geindex module; webbrowser

@strong{Source code:} Lib/webbrowser.py@footnote{https://github.com/python/cpython/tree/main/Lib/webbrowser.py}


__________________________________________________________________


The @ref{130,,webbrowser} module provides a high-level interface to allow displaying
web-based documents to users. Under most circumstances, simply calling the
@ref{3054,,open()} function from this module will do the right thing.

Under Unix, graphical browsers are preferred under X11, but text-mode browsers
will be used if graphical browsers are not available or an X11 display isn’t
available.  If text-mode browsers are used, the calling process will block until
the user exits the browser.

If the environment variable 
@geindex BROWSER
@geindex environment variable; BROWSER
@code{BROWSER} exists, it is interpreted as the
@ref{161d,,os.pathsep}-separated list of browsers to try ahead of the platform
defaults.  When the value of a list part contains the string @code{%s}, then it is
interpreted as a literal browser command line to be used with the argument URL
substituted for @code{%s}; if the part does not contain @code{%s}, it is simply
interpreted as the name of the browser to launch. @footnote{@w{(1)} 
Executables named here without a full path will be searched in the
directories given in the 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} environment variable.
}

For non-Unix platforms, or when a remote browser is available on Unix, the
controlling process will not wait for the user to finish with the browser, but
allow the remote browser to maintain its own windows on the display.  If remote
browsers are not available on Unix, the controlling process will launch a new
browser and wait.

The script @code{webbrowser} can be used as a command-line interface for the
module. It accepts a URL as the argument. It accepts the following optional
parameters: @code{-n} opens the URL in a new browser window, if possible;
@code{-t} opens the URL in a new browser page (“tab”). The options are,
naturally, mutually exclusive.  Usage example:

@example
python -m webbrowser -t "https://www.python.org"
@end example

The following exception is defined:

@geindex Error
@anchor{library/webbrowser webbrowser Error}@anchor{3055}
@deffn {Exception} webbrowser.Error

Exception raised when a browser control error occurs.
@end deffn

The following functions are defined:

@geindex open() (in module webbrowser)
@anchor{library/webbrowser webbrowser open}@anchor{3054}
@deffn {Function} webbrowser.open (url, new=0, autoraise=True)

Display @emph{url} using the default browser. If @emph{new} is 0, the @emph{url} is opened
in the same browser window if possible.  If @emph{new} is 1, a new browser window
is opened if possible.  If @emph{new} is 2, a new browser page (“tab”) is opened
if possible.  If @emph{autoraise} is @code{True}, the window is raised if possible
(note that under many window managers this will occur regardless of the
setting of this variable).

Note that on some platforms, trying to open a filename using this function,
may work and start the operating system’s associated program.  However, this
is neither supported nor portable.

Raises an @ref{1130,,auditing event} @code{webbrowser.open} with argument @code{url}.
@end deffn

@geindex open_new() (in module webbrowser)
@anchor{library/webbrowser webbrowser open_new}@anchor{3056}
@deffn {Function} webbrowser.open_new (url)

Open @emph{url} in a new window of the default browser, if possible, otherwise, open
@emph{url} in the only browser window.
@end deffn

@geindex open_new_tab() (in module webbrowser)
@anchor{library/webbrowser webbrowser open_new_tab}@anchor{3057}
@deffn {Function} webbrowser.open_new_tab (url)

Open @emph{url} in a new page (“tab”) of the default browser, if possible, otherwise
equivalent to @ref{3056,,open_new()}.
@end deffn

@geindex get() (in module webbrowser)
@anchor{library/webbrowser webbrowser get}@anchor{3058}
@deffn {Function} webbrowser.get (using=None)

Return a controller object for the browser type @emph{using}.  If @emph{using} is
@code{None}, return a controller for a default browser appropriate to the
caller’s environment.
@end deffn

@geindex register() (in module webbrowser)
@anchor{library/webbrowser webbrowser register}@anchor{3059}
@deffn {Function} webbrowser.register (name, constructor, instance=None, *, preferred=False)

Register the browser type @emph{name}.  Once a browser type is registered, the
@ref{3058,,get()} function can return a controller for that browser type.  If
@emph{instance} is not provided, or is @code{None}, @emph{constructor} will be called without
parameters to create an instance when needed.  If @emph{instance} is provided,
@emph{constructor} will never be called, and may be @code{None}.

Setting @emph{preferred} to @code{True} makes this browser a preferred result for
a @ref{3058,,get()} call with no argument.  Otherwise, this entry point is only
useful if you plan to either set the 
@geindex BROWSER
@geindex environment variable; BROWSER
@code{BROWSER} variable or call
@ref{3058,,get()} with a nonempty argument matching the name of a handler you
declare.


Changed in version 3.7: @emph{preferred} keyword-only parameter was added.

@end deffn

A number of browser types are predefined.  This table gives the type names that
may be passed to the @ref{3058,,get()} function and the corresponding instantiations
for the controller classes, all defined in this module.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} 
@headitem

Type Name

@tab

Class Name

@tab

Notes

@item

@code{'mozilla'}

@tab

@code{Mozilla('mozilla')}

@tab

@item

@code{'firefox'}

@tab

@code{Mozilla('mozilla')}

@tab

@item

@code{'netscape'}

@tab

@code{Mozilla('netscape')}

@tab

@item

@code{'galeon'}

@tab

@code{Galeon('galeon')}

@tab

@item

@code{'epiphany'}

@tab

@code{Galeon('epiphany')}

@tab

@item

@code{'skipstone'}

@tab

@code{BackgroundBrowser('skipstone')}

@tab

@item

@code{'kfmclient'}

@tab

@code{Konqueror()}

@tab

(1)

@item

@code{'konqueror'}

@tab

@code{Konqueror()}

@tab

(1)

@item

@code{'kfm'}

@tab

@code{Konqueror()}

@tab

(1)

@item

@code{'mosaic'}

@tab

@code{BackgroundBrowser('mosaic')}

@tab

@item

@code{'opera'}

@tab

@code{Opera()}

@tab

@item

@code{'grail'}

@tab

@code{Grail()}

@tab

@item

@code{'links'}

@tab

@code{GenericBrowser('links')}

@tab

@item

@code{'elinks'}

@tab

@code{Elinks('elinks')}

@tab

@item

@code{'lynx'}

@tab

@code{GenericBrowser('lynx')}

@tab

@item

@code{'w3m'}

@tab

@code{GenericBrowser('w3m')}

@tab

@item

@code{'windows-default'}

@tab

@code{WindowsDefault}

@tab

(2)

@item

@code{'macosx'}

@tab

@code{MacOSXOSAScript('default')}

@tab

(3)

@item

@code{'safari'}

@tab

@code{MacOSXOSAScript('safari')}

@tab

(3)

@item

@code{'google-chrome'}

@tab

@code{Chrome('google-chrome')}

@tab

@item

@code{'chrome'}

@tab

@code{Chrome('chrome')}

@tab

@item

@code{'chromium'}

@tab

@code{Chromium('chromium')}

@tab

@item

@code{'chromium-browser'}

@tab

@code{Chromium('chromium-browser')}

@tab

@end multitable


Notes:


@enumerate 

@item 
“Konqueror” is the file manager for the KDE desktop environment for Unix, and
only makes sense to use if KDE is running.  Some way of reliably detecting KDE
would be nice; the 
@geindex KDEDIR
@geindex environment variable; KDEDIR
@code{KDEDIR} variable is not sufficient.  Note also that
the name “kfm” is used even when using the @code{konqueror} command with KDE
2 — the implementation selects the best strategy for running Konqueror.

@item 
Only on Windows platforms.

@item 
Only on macOS platform.
@end enumerate


New in version 3.3: Support for Chrome/Chromium has been added.



Deprecated since version 3.11, will be removed in version 3.13: @code{MacOSX} is deprecated, use @code{MacOSXOSAScript} instead.


Here are some simple examples:

@example
url = 'https://docs.python.org/'

# Open URL in a new tab, if a browser window is already open.
webbrowser.open_new_tab(url)

# Open URL in new window, raising the window if possible.
webbrowser.open_new(url)
@end example

@menu
* Browser Controller Objects:: 

@end menu

@node Browser Controller Objects,,,webbrowser — Convenient web-browser controller
@anchor{library/webbrowser browser-controller-objects}@anchor{305a}@anchor{library/webbrowser browser-controllers}@anchor{305b}
@subsubsection Browser Controller Objects


Browser controllers provide these methods which parallel three of the
module-level convenience functions:

@geindex name (in module webbrowser)
@anchor{library/webbrowser webbrowser name}@anchor{305c}
@deffn {Attribute} webbrowser.name

System-dependent name for the browser.
@end deffn

@geindex open() (webbrowser.controller method)
@anchor{library/webbrowser webbrowser controller open}@anchor{305d}
@deffn {Method} controller.open (url, new=0, autoraise=True)

Display @emph{url} using the browser handled by this controller. If @emph{new} is 1, a new
browser window is opened if possible. If @emph{new} is 2, a new browser page (“tab”)
is opened if possible.
@end deffn

@geindex open_new() (webbrowser.controller method)
@anchor{library/webbrowser webbrowser controller open_new}@anchor{305e}
@deffn {Method} controller.open_new (url)

Open @emph{url} in a new window of the browser handled by this controller, if
possible, otherwise, open @emph{url} in the only browser window.  Alias
@ref{3056,,open_new()}.
@end deffn

@geindex open_new_tab() (webbrowser.controller method)
@anchor{library/webbrowser webbrowser controller open_new_tab}@anchor{305f}
@deffn {Method} controller.open_new_tab (url)

Open @emph{url} in a new page (“tab”) of the browser handled by this controller, if
possible, otherwise equivalent to @ref{3056,,open_new()}.
@end deffn

@node cgi — Common Gateway Interface support,cgitb — Traceback manager for CGI scripts,webbrowser — Convenient web-browser controller,Internet Protocols and Support
@anchor{library/cgi doc}@anchor{3060}@anchor{library/cgi cgi-common-gateway-interface-support}@anchor{3061}@anchor{library/cgi module-cgi}@anchor{14}
@subsection @code{cgi} — Common Gateway Interface support


@geindex module; cgi

@strong{Source code:} Lib/cgi.py@footnote{https://github.com/python/cpython/tree/main/Lib/cgi.py}

@geindex WWW; server
@geindex CGI; protocol
@geindex HTTP; protocol
@geindex MIME; headers
@geindex URL
@geindex Common Gateway Interface


__________________________________________________________________


Support module for Common Gateway Interface (CGI) scripts.

This module defines a number of utilities for use by CGI scripts written in
Python.

The global variable @code{maxlen} can be set to an integer indicating the maximum
size of a POST request. POST requests larger than this size will result in a
@ref{17c,,ValueError} being raised during parsing. The default value of this
variable is @code{0}, meaning the request size is unlimited.

@menu
* Introduction: Introduction<10>. 
* Using the cgi module:: 
* Higher Level Interface:: 
* Functions: Functions<9>. 
* Caring about security:: 
* Installing your CGI script on a Unix system:: 
* Testing your CGI script:: 
* Debugging CGI scripts:: 
* Common problems and solutions:: 

@end menu

@node Introduction<10>,Using the cgi module,,cgi — Common Gateway Interface support
@anchor{library/cgi introduction}@anchor{3062}
@subsubsection Introduction

@anchor{library/cgi cgi-intro}@anchor{3063}
A CGI script is invoked by an HTTP server, usually to process user input
submitted through an HTML @code{<FORM>} or @code{<ISINDEX>} element.

Most often, CGI scripts live in the server’s special @code{cgi-bin} directory.
The HTTP server places all sorts of information about the request (such as the
client’s hostname, the requested URL, the query string, and lots of other
goodies) in the script’s shell environment, executes the script, and sends the
script’s output back to the client.

The script’s input is connected to the client too, and sometimes the form data
is read this way; at other times the form data is passed via the “query string”
part of the URL.  This module is intended to take care of the different cases
and provide a simpler interface to the Python script.  It also provides a number
of utilities that help in debugging scripts, and the latest addition is support
for file uploads from a form (if your browser supports it).

The output of a CGI script should consist of two sections, separated by a blank
line.  The first section contains a number of headers, telling the client what
kind of data is following.  Python code to generate a minimal header section
looks like this:

@example
print("Content-Type: text/html")    # HTML is following
print()                             # blank line, end of headers
@end example

The second section is usually HTML, which allows the client software to display
nicely formatted text with header, in-line images, etc. Here’s Python code that
prints a simple piece of HTML:

@example
print("<TITLE>CGI script output</TITLE>")
print("<H1>This is my first CGI script</H1>")
print("Hello, world!")
@end example

@node Using the cgi module,Higher Level Interface,Introduction<10>,cgi — Common Gateway Interface support
@anchor{library/cgi id1}@anchor{3064}@anchor{library/cgi using-the-cgi-module}@anchor{3065}
@subsubsection Using the cgi module


Begin by writing @code{import cgi}.

When you write a new script, consider adding these lines:

@example
import cgitb
cgitb.enable()
@end example

This activates a special exception handler that will display detailed reports in
the web browser if any errors occur.  If you’d rather not show the guts of your
program to users of your script, you can have the reports saved to files
instead, with code like this:

@example
import cgitb
cgitb.enable(display=0, logdir="/path/to/logdir")
@end example

It’s very helpful to use this feature during script development. The reports
produced by @ref{15,,cgitb} provide information that can save you a lot of time in
tracking down bugs.  You can always remove the @code{cgitb} line later when you
have tested your script and are confident that it works correctly.

To get at submitted form data, use the @code{FieldStorage} class. If the form
contains non-ASCII characters, use the @emph{encoding} keyword parameter set to the
value of the encoding defined for the document. It is usually contained in the
META tag in the HEAD section of the HTML document or by the
@code{Content-Type} header.  This reads the form contents from the
standard input or the environment (depending on the value of various
environment variables set according to the CGI standard).  Since it may consume
standard input, it should be instantiated only once.

The @code{FieldStorage} instance can be indexed like a Python dictionary.
It allows membership testing with the @ref{2fc,,in} operator, and also supports
the standard dictionary method @ref{243,,keys()} and the built-in function
@ref{4a6,,len()}.  Form fields containing empty strings are ignored and do not appear
in the dictionary; to keep such values, provide a true value for the optional
@emph{keep_blank_values} keyword parameter when creating the @code{FieldStorage}
instance.

For instance, the following code (which assumes that the
@code{Content-Type} header and blank line have already been printed)
checks that the fields @code{name} and @code{addr} are both set to a non-empty
string:

@example
form = cgi.FieldStorage()
if "name" not in form or "addr" not in form:
    print("<H1>Error</H1>")
    print("Please fill in the name and addr fields.")
    return
print("<p>name:", form["name"].value)
print("<p>addr:", form["addr"].value)
...further form processing here...
@end example

Here the fields, accessed through @code{form[key]}, are themselves instances of
@code{FieldStorage} (or @code{MiniFieldStorage}, depending on the form
encoding). The @code{value} attribute of the instance yields
the string value of the field.  The @code{getvalue()} method
returns this string value directly; it also accepts an optional second argument
as a default to return if the requested key is not present.

If the submitted form data contains more than one field with the same name, the
object retrieved by @code{form[key]} is not a @code{FieldStorage} or
@code{MiniFieldStorage} instance but a list of such instances.  Similarly, in
this situation, @code{form.getvalue(key)} would return a list of strings. If you
expect this possibility (when your HTML form contains multiple fields with the
same name), use the @ref{3066,,getlist()} method, which always returns
a list of values (so that you do not need to special-case the single item
case).  For example, this code concatenates any number of username fields,
separated by commas:

@example
value = form.getlist("username")
usernames = ",".join(value)
@end example

If a field represents an uploaded file, accessing the value via the
@code{value} attribute or the @code{getvalue()}
method reads the entire file in memory as bytes.  This may not be what you
want.  You can test for an uploaded file by testing either the
@code{filename} attribute or the @code{file}
attribute.  You can then read the data from the @code{file}
attribute before it is automatically closed as part of the garbage collection of
the @code{FieldStorage} instance
(the @ref{983,,read()} and @ref{1b74,,readline()} methods will
return bytes):

@example
fileitem = form["userfile"]
if fileitem.file:
    # It's an uploaded file; count lines
    linecount = 0
    while True:
        line = fileitem.file.readline()
        if not line: break
        linecount = linecount + 1
@end example

@code{FieldStorage} objects also support being used in a @ref{168,,with}
statement, which will automatically close them when done.

If an error is encountered when obtaining the contents of an uploaded file
(for example, when the user interrupts the form submission by clicking on
a Back or Cancel button) the @code{done} attribute of the
object for the field will be set to the value -1.

The file upload draft standard entertains the possibility of uploading multiple
files from one field (using a recursive @code{multipart/*} encoding).
When this occurs, the item will be a dictionary-like @code{FieldStorage} item.
This can be determined by testing its @code{type} attribute, which should be
@code{multipart/form-data} (or perhaps another MIME type matching
@code{multipart/*}).  In this case, it can be iterated over recursively
just like the top-level form object.

When a form is submitted in the “old” format (as the query string or as a single
data part of type @code{application/x-www-form-urlencoded}), the items will
actually be instances of the class @code{MiniFieldStorage}.  In this case, the
@code{list}, @code{file}, and @code{filename} attributes are always @code{None}.

A form submitted via POST that also has a query string will contain both
@code{FieldStorage} and @code{MiniFieldStorage} items.


Changed in version 3.4: The @code{file} attribute is automatically closed upon the
garbage collection of the creating @code{FieldStorage} instance.



Changed in version 3.5: Added support for the context management protocol to the
@code{FieldStorage} class.


@node Higher Level Interface,Functions<9>,Using the cgi module,cgi — Common Gateway Interface support
@anchor{library/cgi higher-level-interface}@anchor{3067}
@subsubsection Higher Level Interface


The previous section explains how to read CGI form data using the
@code{FieldStorage} class.  This section describes a higher level interface
which was added to this class to allow one to do it in a more readable and
intuitive way.  The interface doesn’t make the techniques described in previous
sections obsolete — they are still useful to process file uploads efficiently,
for example.

@c XXX: Is this true ?

The interface consists of two simple methods. Using the methods you can process
form data in a generic way, without the need to worry whether only one or more
values were posted under one name.

In the previous section, you learned to write following code anytime you
expected a user to post more than one value under one name:

@example
item = form.getvalue("item")
if isinstance(item, list):
    # The user is requesting more than one item.
else:
    # The user is requesting only one item.
@end example

This situation is common for example when a form contains a group of multiple
checkboxes with the same name:

@example
<input type="checkbox" name="item" value="1" />
<input type="checkbox" name="item" value="2" />
@end example

In most situations, however, there’s only one form control with a particular
name in a form and then you expect and need only one value associated with this
name.  So you write a script containing for example this code:

@example
user = form.getvalue("user").upper()
@end example

The problem with the code is that you should never expect that a client will
provide valid input to your scripts.  For example, if a curious user appends
another @code{user=foo} pair to the query string, then the script would crash,
because in this situation the @code{getvalue("user")} method call returns a list
instead of a string.  Calling the @ref{1ab6,,upper()} method on a list is not valid
(since lists do not have a method of this name) and results in an
@ref{163,,AttributeError} exception.

Therefore, the appropriate way to read form data values was to always use the
code which checks whether the obtained value is a single value or a list of
values.  That’s annoying and leads to less readable scripts.

A more convenient approach is to use the methods @ref{3068,,getfirst()}
and @ref{3066,,getlist()} provided by this higher level interface.

@geindex getfirst() (cgi.FieldStorage method)
@anchor{library/cgi cgi FieldStorage getfirst}@anchor{3068}
@deffn {Method} FieldStorage.getfirst (name, default=None)

This method always returns only one value associated with form field @emph{name}.
The method returns only the first value in case that more values were posted
under such name.  Please note that the order in which the values are received
may vary from browser to browser and should not be counted on. @footnote{@w{(1)} 
Note that some recent versions of the HTML specification do state what
order the field values should be supplied in, but knowing whether a request
was received from a conforming browser, or even from a browser at all, is
tedious and error-prone.
}  If no such
form field or value exists then the method returns the value specified by the
optional parameter @emph{default}.  This parameter defaults to @code{None} if not
specified.
@end deffn

@geindex getlist() (cgi.FieldStorage method)
@anchor{library/cgi cgi FieldStorage getlist}@anchor{3066}
@deffn {Method} FieldStorage.getlist (name)

This method always returns a list of values associated with form field @emph{name}.
The method returns an empty list if no such form field or value exists for
@emph{name}.  It returns a list consisting of one item if only one such value exists.
@end deffn

Using these methods you can write nice compact code:

@example
import cgi
form = cgi.FieldStorage()
user = form.getfirst("user", "").upper()    # This way it's safe.
for item in form.getlist("item"):
    do_something(item)
@end example

@node Functions<9>,Caring about security,Higher Level Interface,cgi — Common Gateway Interface support
@anchor{library/cgi functions}@anchor{3069}@anchor{library/cgi functions-in-cgi-module}@anchor{306a}
@subsubsection Functions


These are useful if you want more control, or if you want to employ some of the
algorithms implemented in this module in other circumstances.

@geindex parse() (in module cgi)
@anchor{library/cgi cgi parse}@anchor{2e5}
@deffn {Function} cgi.parse (fp=None, environ=os.environ, keep_blank_values=False, strict_parsing=False, separator='&')

Parse a query in the environment or from a file (the file defaults to
@code{sys.stdin}).  The @emph{keep_blank_values}, @emph{strict_parsing} and @emph{separator} parameters are
passed to @ref{2e3,,urllib.parse.parse_qs()} unchanged.
@end deffn

@geindex parse_multipart() (in module cgi)
@anchor{library/cgi cgi parse_multipart}@anchor{2e6}
@deffn {Function} cgi.parse_multipart (fp, pdict, encoding='utf-8', errors='replace', separator='&')

Parse input of type @code{multipart/form-data} (for  file uploads).
Arguments are @emph{fp} for the input file, @emph{pdict} for a dictionary containing
other parameters in the @code{Content-Type} header, and @emph{encoding},
the request encoding.

Returns a dictionary just like @ref{2e3,,urllib.parse.parse_qs()}: keys are the
field names, each value is a list of values for that field. For non-file
fields, the value is a list of strings.

This is easy to use but not much good if you are expecting megabytes to be
uploaded — in that case, use the @code{FieldStorage} class instead
which is much more flexible.


Changed in version 3.7: Added the @emph{encoding} and @emph{errors} parameters.  For non-file fields, the
value is now a list of strings, not bytes.



Changed in version 3.10: Added the @emph{separator} parameter.

@end deffn

@geindex parse_header() (in module cgi)
@anchor{library/cgi cgi parse_header}@anchor{306b}
@deffn {Function} cgi.parse_header (string)

Parse a MIME header (such as @code{Content-Type}) into a main value and a
dictionary of parameters.
@end deffn

@geindex test() (in module cgi)
@anchor{library/cgi cgi test}@anchor{306c}
@deffn {Function} cgi.test ()

Robust test CGI script, usable as main program. Writes minimal HTTP headers and
formats all information provided to the script in HTML format.
@end deffn

@geindex print_environ() (in module cgi)
@anchor{library/cgi cgi print_environ}@anchor{306d}
@deffn {Function} cgi.print_environ ()

Format the shell environment in HTML.
@end deffn

@geindex print_form() (in module cgi)
@anchor{library/cgi cgi print_form}@anchor{306e}
@deffn {Function} cgi.print_form (form)

Format a form in HTML.
@end deffn

@geindex print_directory() (in module cgi)
@anchor{library/cgi cgi print_directory}@anchor{306f}
@deffn {Function} cgi.print_directory ()

Format the current directory in HTML.
@end deffn

@geindex print_environ_usage() (in module cgi)
@anchor{library/cgi cgi print_environ_usage}@anchor{3070}
@deffn {Function} cgi.print_environ_usage ()

Print a list of useful (used by CGI) environment variables in HTML.
@end deffn

@node Caring about security,Installing your CGI script on a Unix system,Functions<9>,cgi — Common Gateway Interface support
@anchor{library/cgi caring-about-security}@anchor{3071}@anchor{library/cgi cgi-security}@anchor{3072}
@subsubsection Caring about security


@geindex CGI; security

There’s one important rule: if you invoke an external program (via
@ref{fbb,,os.system()}, @ref{59f,,os.popen()} or other functions with similar
functionality), make very sure you don’t pass arbitrary strings received from
the client to the shell.  This is a well-known security hole whereby clever
hackers anywhere on the web can exploit a gullible CGI script to invoke
arbitrary shell commands.  Even parts of the URL or field names cannot be
trusted, since the request doesn’t have to come from your form!

To be on the safe side, if you must pass a string gotten from a form to a shell
command, you should make sure the string contains only alphanumeric characters,
dashes, underscores, and periods.

@node Installing your CGI script on a Unix system,Testing your CGI script,Caring about security,cgi — Common Gateway Interface support
@anchor{library/cgi installing-your-cgi-script-on-a-unix-system}@anchor{3073}
@subsubsection Installing your CGI script on a Unix system


Read the documentation for your HTTP server and check with your local system
administrator to find the directory where CGI scripts should be installed;
usually this is in a directory @code{cgi-bin} in the server tree.

Make sure that your script is readable and executable by “others”; the Unix file
mode should be @code{0o755} octal (use @code{chmod 0755 filename}).  Make sure that the
first line of the script contains @code{#!} starting in column 1 followed by the
pathname of the Python interpreter, for instance:

@example
#!/usr/local/bin/python
@end example

Make sure the Python interpreter exists and is executable by “others”.

Make sure that any files your script needs to read or write are readable or
writable, respectively, by “others” — their mode should be @code{0o644} for
readable and @code{0o666} for writable.  This is because, for security reasons, the
HTTP server executes your script as user “nobody”, without any special
privileges.  It can only read (write, execute) files that everybody can read
(write, execute).  The current directory at execution time is also different (it
is usually the server’s cgi-bin directory) and the set of environment variables
is also different from what you get when you log in.  In particular, don’t count
on the shell’s search path for executables (
@geindex PATH
@geindex environment variable; PATH
@code{PATH}) or the Python module
search path (
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH}) to be set to anything interesting.

If you need to load modules from a directory which is not on Python’s default
module search path, you can change the path in your script, before importing
other modules.  For example:

@example
import sys
sys.path.insert(0, "/usr/home/joe/lib/python")
sys.path.insert(0, "/usr/local/lib/python")
@end example

(This way, the directory inserted last will be searched first!)

Instructions for non-Unix systems will vary; check your HTTP server’s
documentation (it will usually have a section on CGI scripts).

@node Testing your CGI script,Debugging CGI scripts,Installing your CGI script on a Unix system,cgi — Common Gateway Interface support
@anchor{library/cgi testing-your-cgi-script}@anchor{3074}
@subsubsection Testing your CGI script


Unfortunately, a CGI script will generally not run when you try it from the
command line, and a script that works perfectly from the command line may fail
mysteriously when run from the server.  There’s one reason why you should still
test your script from the command line: if it contains a syntax error, the
Python interpreter won’t execute it at all, and the HTTP server will most likely
send a cryptic error to the client.

Assuming your script has no syntax errors, yet it does not work, you have no
choice but to read the next section.

@node Debugging CGI scripts,Common problems and solutions,Testing your CGI script,cgi — Common Gateway Interface support
@anchor{library/cgi debugging-cgi-scripts}@anchor{3075}
@subsubsection Debugging CGI scripts


@geindex CGI; debugging

First of all, check for trivial installation errors — reading the section
above on installing your CGI script carefully can save you a lot of time.  If
you wonder whether you have understood the installation procedure correctly, try
installing a copy of this module file (@code{cgi.py}) as a CGI script.  When
invoked as a script, the file will dump its environment and the contents of the
form in HTML format. Give it the right mode etc., and send it a request.  If it’s
installed in the standard @code{cgi-bin} directory, it should be possible to
send it a request by entering a URL into your browser of the form:

@example
http://yourhostname/cgi-bin/cgi.py?name=Joe+Blow&addr=At+Home
@end example

If this gives an error of type 404, the server cannot find the script – perhaps
you need to install it in a different directory.  If it gives another error,
there’s an installation problem that you should fix before trying to go any
further.  If you get a nicely formatted listing of the environment and form
content (in this example, the fields should be listed as “addr” with value “At
Home” and “name” with value “Joe Blow”), the @code{cgi.py} script has been
installed correctly.  If you follow the same procedure for your own script, you
should now be able to debug it.

The next step could be to call the @ref{14,,cgi} module’s @ref{ff,,test()} function
from your script: replace its main code with the single statement

@example
cgi.test()
@end example

This should produce the same results as those gotten from installing the
@code{cgi.py} file itself.

When an ordinary Python script raises an unhandled exception (for whatever
reason: of a typo in a module name, a file that can’t be opened, etc.), the
Python interpreter prints a nice traceback and exits.  While the Python
interpreter will still do this when your CGI script raises an exception, most
likely the traceback will end up in one of the HTTP server’s log files, or be
discarded altogether.

Fortunately, once you have managed to get your script to execute @emph{some} code,
you can easily send tracebacks to the web browser using the @ref{15,,cgitb} module.
If you haven’t done so already, just add the lines:

@example
import cgitb
cgitb.enable()
@end example

to the top of your script.  Then try running it again; when a problem occurs,
you should see a detailed report that will likely make apparent the cause of the
crash.

If you suspect that there may be a problem in importing the @ref{15,,cgitb} module,
you can use an even more robust approach (which only uses built-in modules):

@example
import sys
sys.stderr = sys.stdout
print("Content-Type: text/plain")
print()
...your code here...
@end example

This relies on the Python interpreter to print the traceback.  The content type
of the output is set to plain text, which disables all HTML processing.  If your
script works, the raw HTML will be displayed by your client.  If it raises an
exception, most likely after the first two lines have been printed, a traceback
will be displayed. Because no HTML interpretation is going on, the traceback
will be readable.

@node Common problems and solutions,,Debugging CGI scripts,cgi — Common Gateway Interface support
@anchor{library/cgi common-problems-and-solutions}@anchor{3076}
@subsubsection Common problems and solutions



@itemize *

@item 
Most HTTP servers buffer the output from CGI scripts until the script is
completed.  This means that it is not possible to display a progress report on
the client’s display while the script is running.

@item 
Check the installation instructions above.

@item 
Check the HTTP server’s log files.  (@code{tail -f logfile} in a separate window
may be useful!)

@item 
Always check a script for syntax errors first, by doing something like
@code{python script.py}.

@item 
If your script does not have any syntax errors, try adding @code{import cgitb;
cgitb.enable()} to the top of the script.

@item 
When invoking external programs, make sure they can be found. Usually, this
means using absolute path names — 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} is usually not set to a very
useful value in a CGI script.

@item 
When reading or writing external files, make sure they can be read or written
by the userid under which your CGI script will be running: this is typically the
userid under which the web server is running, or some explicitly specified
userid for a web server’s @code{suexec} feature.

@item 
Don’t try to give a CGI script a set-uid mode.  This doesn’t work on most
systems, and is a security liability as well.
@end itemize

@node cgitb — Traceback manager for CGI scripts,wsgiref — WSGI Utilities and Reference Implementation,cgi — Common Gateway Interface support,Internet Protocols and Support
@anchor{library/cgitb doc}@anchor{3077}@anchor{library/cgitb cgitb-traceback-manager-for-cgi-scripts}@anchor{3078}@anchor{library/cgitb module-cgitb}@anchor{15}
@subsection @code{cgitb} — Traceback manager for CGI scripts


@geindex module; cgitb

@strong{Source code:} Lib/cgitb.py@footnote{https://github.com/python/cpython/tree/main/Lib/cgitb.py}

@geindex CGI; exceptions
@geindex CGI; tracebacks
@geindex exceptions; in CGI scripts
@geindex tracebacks; in CGI scripts


__________________________________________________________________


The @ref{15,,cgitb} module provides a special exception handler for Python scripts.
(Its name is a bit misleading.  It was originally designed to display extensive
traceback information in HTML for CGI scripts.  It was later generalized to also
display this information in plain text.)  After this module is activated, if an
uncaught exception occurs, a detailed, formatted report will be displayed.  The
report includes a traceback showing excerpts of the source code for each level,
as well as the values of the arguments and local variables to currently running
functions, to help you debug the problem.  Optionally, you can save this
information to a file instead of sending it to the browser.

To enable this feature, simply add this to the top of your CGI script:

@example
import cgitb
cgitb.enable()
@end example

The options to the @ref{3079,,enable()} function control whether the report is
displayed in the browser and whether the report is logged to a file for later
analysis.

@geindex enable() (in module cgitb)
@anchor{library/cgitb cgitb enable}@anchor{3079}
@deffn {Function} cgitb.enable (display=1, logdir=None, context=5, format='html')

@geindex excepthook() (in module sys)

This function causes the @ref{15,,cgitb} module to take over the interpreter’s
default handling for exceptions by setting the value of @ref{28e,,sys.excepthook}.

The optional argument @emph{display} defaults to @code{1} and can be set to @code{0} to
suppress sending the traceback to the browser. If the argument @emph{logdir} is
present, the traceback reports are written to files.  The value of @emph{logdir}
should be a directory where these files will be placed. The optional argument
@emph{context} is the number of lines of context to display around the current line
of source code in the traceback; this defaults to @code{5}. If the optional
argument @emph{format} is @code{"html"}, the output is formatted as HTML.  Any other
value forces plain text output.  The default value is @code{"html"}.
@end deffn

@geindex text() (in module cgitb)
@anchor{library/cgitb cgitb text}@anchor{307a}
@deffn {Function} cgitb.text (info, context=5)

This function handles the exception described by @emph{info} (a 3-tuple containing
the result of @ref{18f,,sys.exc_info()}), formatting its traceback as text and
returning the result as a string. The optional argument @emph{context} is the
number of lines of context to display around the current line of source code
in the traceback; this defaults to @code{5}.
@end deffn

@geindex html() (in module cgitb)
@anchor{library/cgitb cgitb html}@anchor{307b}
@deffn {Function} cgitb.html (info, context=5)

This function handles the exception described by @emph{info} (a 3-tuple containing
the result of @ref{18f,,sys.exc_info()}), formatting its traceback as HTML and
returning the result as a string. The optional argument @emph{context} is the
number of lines of context to display around the current line of source code
in the traceback; this defaults to @code{5}.
@end deffn

@geindex handler() (in module cgitb)
@anchor{library/cgitb cgitb handler}@anchor{307c}
@deffn {Function} cgitb.handler (info=None)

This function handles an exception using the default settings (that is, show a
report in the browser, but don’t log to a file). This can be used when you’ve
caught an exception and want to report it using @ref{15,,cgitb}.  The optional
@emph{info} argument should be a 3-tuple containing an exception type, exception
value, and traceback object, exactly like the tuple returned by
@ref{18f,,sys.exc_info()}.  If the @emph{info} argument is not supplied, the current
exception is obtained from @ref{18f,,sys.exc_info()}.
@end deffn

@node wsgiref — WSGI Utilities and Reference Implementation,urllib — URL handling modules,cgitb — Traceback manager for CGI scripts,Internet Protocols and Support
@anchor{library/wsgiref doc}@anchor{307d}@anchor{library/wsgiref module-wsgiref}@anchor{133}@anchor{library/wsgiref wsgiref-wsgi-utilities-and-reference-implementation}@anchor{307e}
@subsection @code{wsgiref} — WSGI Utilities and Reference Implementation


@geindex module; wsgiref


__________________________________________________________________


The Web Server Gateway Interface (WSGI) is a standard interface between web
server software and web applications written in Python. Having a standard
interface makes it easy to use an application that supports WSGI with a number
of different web servers.

Only authors of web servers and programming frameworks need to know every detail
and corner case of the WSGI design.  You don’t need to understand every detail
of WSGI just to install a WSGI application or to write a web application using
an existing framework.

@ref{133,,wsgiref} is a reference implementation of the WSGI specification that can
be used to add WSGI support to a web server or framework.  It provides utilities
for manipulating WSGI environment variables and response headers, base classes
for implementing WSGI servers, a demo HTTP server that serves WSGI applications,
and a validation tool that checks WSGI servers and applications for conformance
to the WSGI specification (
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333}).

See wsgi.readthedocs.io@footnote{https://wsgi.readthedocs.io/} for more information about WSGI, and links
to tutorials and other resources.

@c XXX If you're just trying to write a web application...

@menu
* wsgiref.util – WSGI environment utilities: wsgiref util – WSGI environment utilities. 
* wsgiref.headers – WSGI response header tools: wsgiref headers – WSGI response header tools. 
* wsgiref.simple_server – a simple WSGI HTTP server: wsgiref simple_server – a simple WSGI HTTP server. 
* wsgiref.validate — WSGI conformance checker: wsgiref validate — WSGI conformance checker. 
* wsgiref.handlers – server/gateway base classes: wsgiref handlers – server/gateway base classes. 
* Examples: Examples<23>. 

@end menu

@node wsgiref util – WSGI environment utilities,wsgiref headers – WSGI response header tools,,wsgiref — WSGI Utilities and Reference Implementation
@anchor{library/wsgiref module-wsgiref util}@anchor{137}@anchor{library/wsgiref wsgiref-util-wsgi-environment-utilities}@anchor{307f}
@subsubsection @code{wsgiref.util} – WSGI environment utilities


@geindex module; wsgiref.util

This module provides a variety of utility functions for working with WSGI
environments.  A WSGI environment is a dictionary containing HTTP request
variables as described in 
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333}.  All of the functions taking an @emph{environ}
parameter expect a WSGI-compliant dictionary to be supplied; please see
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333} for a detailed specification.

@geindex guess_scheme() (in module wsgiref.util)
@anchor{library/wsgiref wsgiref util guess_scheme}@anchor{3080}
@deffn {Function} wsgiref.util.guess_scheme (environ)

Return a guess for whether @code{wsgi.url_scheme} should be “http” or “https”, by
checking for a @code{HTTPS} environment variable in the @emph{environ} dictionary.  The
return value is a string.

This function is useful when creating a gateway that wraps CGI or a CGI-like
protocol such as FastCGI.  Typically, servers providing such protocols will
include a @code{HTTPS} variable with a value of “1”, “yes”, or “on” when a request
is received via SSL.  So, this function returns “https” if such a value is
found, and “http” otherwise.
@end deffn

@geindex request_uri() (in module wsgiref.util)
@anchor{library/wsgiref wsgiref util request_uri}@anchor{3081}
@deffn {Function} wsgiref.util.request_uri (environ, include_query=True)

Return the full request URI, optionally including the query string, using the
algorithm found in the “URL Reconstruction” section of 
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333}.  If
@emph{include_query} is false, the query string is not included in the resulting URI.
@end deffn

@geindex application_uri() (in module wsgiref.util)
@anchor{library/wsgiref wsgiref util application_uri}@anchor{3082}
@deffn {Function} wsgiref.util.application_uri (environ)

Similar to @ref{3081,,request_uri()}, except that the @code{PATH_INFO} and
@code{QUERY_STRING} variables are ignored.  The result is the base URI of the
application object addressed by the request.
@end deffn

@geindex shift_path_info() (in module wsgiref.util)
@anchor{library/wsgiref wsgiref util shift_path_info}@anchor{3083}
@deffn {Function} wsgiref.util.shift_path_info (environ)

Shift a single name from @code{PATH_INFO} to @code{SCRIPT_NAME} and return the name.
The @emph{environ} dictionary is @emph{modified} in-place; use a copy if you need to keep
the original @code{PATH_INFO} or @code{SCRIPT_NAME} intact.

If there are no remaining path segments in @code{PATH_INFO}, @code{None} is returned.

Typically, this routine is used to process each portion of a request URI path,
for example to treat the path as a series of dictionary keys. This routine
modifies the passed-in environment to make it suitable for invoking another WSGI
application that is located at the target URI. For example, if there is a WSGI
application at @code{/foo}, and the request URI path is @code{/foo/bar/baz}, and the
WSGI application at @code{/foo} calls @ref{3083,,shift_path_info()}, it will receive the
string “bar”, and the environment will be updated to be suitable for passing to
a WSGI application at @code{/foo/bar}.  That is, @code{SCRIPT_NAME} will change from
@code{/foo} to @code{/foo/bar}, and @code{PATH_INFO} will change from @code{/bar/baz} to
@code{/baz}.

When @code{PATH_INFO} is just a “/”, this routine returns an empty string and
appends a trailing slash to @code{SCRIPT_NAME}, even though empty path segments are
normally ignored, and @code{SCRIPT_NAME} doesn’t normally end in a slash.  This is
intentional behavior, to ensure that an application can tell the difference
between URIs ending in @code{/x} from ones ending in @code{/x/} when using this
routine to do object traversal.
@end deffn

@geindex setup_testing_defaults() (in module wsgiref.util)
@anchor{library/wsgiref wsgiref util setup_testing_defaults}@anchor{3084}
@deffn {Function} wsgiref.util.setup_testing_defaults (environ)

Update @emph{environ} with trivial defaults for testing purposes.

This routine adds various parameters required for WSGI, including @code{HTTP_HOST},
@code{SERVER_NAME}, @code{SERVER_PORT}, @code{REQUEST_METHOD}, @code{SCRIPT_NAME},
@code{PATH_INFO}, and all of the 
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333}-defined @code{wsgi.*} variables.  It
only supplies default values, and does not replace any existing settings for
these variables.

This routine is intended to make it easier for unit tests of WSGI servers and
applications to set up dummy environments.  It should NOT be used by actual WSGI
servers or applications, since the data is fake!

Example usage:

@example
from wsgiref.util import setup_testing_defaults
from wsgiref.simple_server import make_server

# A relatively simple WSGI application. It's going to print out the
# environment dictionary after being updated by setup_testing_defaults
def simple_app(environ, start_response):
    setup_testing_defaults(environ)

    status = '200 OK'
    headers = [('Content-type', 'text/plain; charset=utf-8')]

    start_response(status, headers)

    ret = [("%s: %s\n" % (key, value)).encode("utf-8")
           for key, value in environ.items()]
    return ret

with make_server('@w{'}, 8000, simple_app) as httpd:
    print("Serving on port 8000...")
    httpd.serve_forever()
@end example
@end deffn

In addition to the environment functions above, the @ref{137,,wsgiref.util} module
also provides these miscellaneous utilities:

@geindex is_hop_by_hop() (in module wsgiref.util)
@anchor{library/wsgiref wsgiref util is_hop_by_hop}@anchor{3085}
@deffn {Function} wsgiref.util.is_hop_by_hop (header_name)

Return @code{True} if ‘header_name’ is an HTTP/1.1 “Hop-by-Hop” header, as defined by
@geindex RFC; RFC 2616
RFC 2616@footnote{https://tools.ietf.org/html/rfc2616.html}.
@end deffn

@geindex FileWrapper (class in wsgiref.util)
@anchor{library/wsgiref wsgiref util FileWrapper}@anchor{1bb}
@deffn {Class} wsgiref.util.FileWrapper (filelike, blksize=8192)

A wrapper to convert a file-like object to an @ref{11c3,,iterator}.  The resulting objects
are @ref{dcc,,iterable}s. As the object is iterated over, the
optional @emph{blksize} parameter will be repeatedly passed to the @emph{filelike}
object’s @code{read()} method to obtain bytestrings to yield.  When @code{read()}
returns an empty bytestring, iteration is ended and is not resumable.

If @emph{filelike} has a @code{close()} method, the returned object will also have a
@code{close()} method, and it will invoke the @emph{filelike} object’s @code{close()}
method when called.

Example usage:

@example
from io import StringIO
from wsgiref.util import FileWrapper

# We're using a StringIO-buffer for as the file-like object
filelike = StringIO("This is an example file-like object"*10)
wrapper = FileWrapper(filelike, blksize=5)

for chunk in wrapper:
    print(chunk)
@end example


Changed in version 3.11: Support for @code{__getitem__()} method has been removed.

@end deffn

@node wsgiref headers – WSGI response header tools,wsgiref simple_server – a simple WSGI HTTP server,wsgiref util – WSGI environment utilities,wsgiref — WSGI Utilities and Reference Implementation
@anchor{library/wsgiref module-wsgiref headers}@anchor{135}@anchor{library/wsgiref wsgiref-headers-wsgi-response-header-tools}@anchor{3086}
@subsubsection @code{wsgiref.headers} – WSGI response header tools


@geindex module; wsgiref.headers

This module provides a single class, @ref{a09,,Headers}, for convenient
manipulation of WSGI response headers using a mapping-like interface.

@geindex Headers (class in wsgiref.headers)
@anchor{library/wsgiref wsgiref headers Headers}@anchor{a09}
@deffn {Class} wsgiref.headers.Headers ([headers])

Create a mapping-like object wrapping @emph{headers}, which must be a list of header
name/value tuples as described in 
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333}. The default value of @emph{headers} is
an empty list.

@ref{a09,,Headers} objects support typical mapping operations including
@code{__getitem__()}, @code{get()}, @code{__setitem__()}, @code{setdefault()},
@code{__delitem__()} and @code{__contains__()}.  For each of
these methods, the key is the header name (treated case-insensitively), and the
value is the first value associated with that header name.  Setting a header
deletes any existing values for that header, then adds a new value at the end of
the wrapped header list.  Headers’ existing order is generally maintained, with
new headers added to the end of the wrapped list.

Unlike a dictionary, @ref{a09,,Headers} objects do not raise an error when you try
to get or delete a key that isn’t in the wrapped header list. Getting a
nonexistent header just returns @code{None}, and deleting a nonexistent header does
nothing.

@ref{a09,,Headers} objects also support @code{keys()}, @code{values()}, and
@code{items()} methods.  The lists returned by @code{keys()} and @code{items()} can
include the same key more than once if there is a multi-valued header.  The
@code{len()} of a @ref{a09,,Headers} object is the same as the length of its
@code{items()}, which is the same as the length of the wrapped header list.  In
fact, the @code{items()} method just returns a copy of the wrapped header list.

Calling @code{bytes()} on a @ref{a09,,Headers} object returns a formatted bytestring
suitable for transmission as HTTP response headers.  Each header is placed on a
line with its value, separated by a colon and a space. Each line is terminated
by a carriage return and line feed, and the bytestring is terminated with a
blank line.

In addition to their mapping interface and formatting features, @ref{a09,,Headers}
objects also have the following methods for querying and adding multi-valued
headers, and for adding headers with MIME parameters:

@geindex get_all() (wsgiref.headers.Headers method)
@anchor{library/wsgiref wsgiref headers Headers get_all}@anchor{3087}
@deffn {Method} get_all (name)

Return a list of all the values for the named header.

The returned list will be sorted in the order they appeared in the original
header list or were added to this instance, and may contain duplicates.  Any
fields deleted and re-inserted are always appended to the header list.  If no
fields exist with the given name, returns an empty list.
@end deffn

@geindex add_header() (wsgiref.headers.Headers method)
@anchor{library/wsgiref wsgiref headers Headers add_header}@anchor{3088}
@deffn {Method} add_header (name, value, **_params)

Add a (possibly multi-valued) header, with optional MIME parameters specified
via keyword arguments.

@emph{name} is the header field to add.  Keyword arguments can be used to set MIME
parameters for the header field.  Each parameter must be a string or @code{None}.
Underscores in parameter names are converted to dashes, since dashes are illegal
in Python identifiers, but many MIME parameter names include dashes.  If the
parameter value is a string, it is added to the header value parameters in the
form @code{name="value"}. If it is @code{None}, only the parameter name is added.
(This is used for MIME parameters without a value.)  Example usage:

@example
h.add_header('content-disposition', 'attachment', filename='bud.gif')
@end example

The above will add a header that looks like this:

@example
Content-Disposition: attachment; filename="bud.gif"
@end example
@end deffn


Changed in version 3.5: @emph{headers} parameter is optional.

@end deffn

@node wsgiref simple_server – a simple WSGI HTTP server,wsgiref validate — WSGI conformance checker,wsgiref headers – WSGI response header tools,wsgiref — WSGI Utilities and Reference Implementation
@anchor{library/wsgiref module-wsgiref simple_server}@anchor{136}@anchor{library/wsgiref wsgiref-simple-server-a-simple-wsgi-http-server}@anchor{3089}
@subsubsection @code{wsgiref.simple_server} – a simple WSGI HTTP server


@geindex module; wsgiref.simple_server

This module implements a simple HTTP server (based on @ref{92,,http.server})
that serves WSGI applications.  Each server instance serves a single WSGI
application on a given host and port.  If you want to serve multiple
applications on a single host and port, you should create a WSGI application
that parses @code{PATH_INFO} to select which application to invoke for each
request.  (E.g., using the @code{shift_path_info()} function from
@ref{137,,wsgiref.util}.)

@geindex make_server() (in module wsgiref.simple_server)
@anchor{library/wsgiref wsgiref simple_server make_server}@anchor{308a}
@deffn {Function} wsgiref.simple_server.make_server (host, port, app, server_class=WSGIServer, handler_class=WSGIRequestHandler)

Create a new WSGI server listening on @emph{host} and @emph{port}, accepting connections
for @emph{app}.  The return value is an instance of the supplied @emph{server_class}, and
will process requests using the specified @emph{handler_class}.  @emph{app} must be a WSGI
application object, as defined by 
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333}.

Example usage:

@example
from wsgiref.simple_server import make_server, demo_app

with make_server('@w{'}, 8000, demo_app) as httpd:
    print("Serving HTTP on port 8000...")

    # Respond to requests until process is killed
    httpd.serve_forever()

    # Alternative: serve one request, then exit
    httpd.handle_request()
@end example
@end deffn

@geindex demo_app() (in module wsgiref.simple_server)
@anchor{library/wsgiref wsgiref simple_server demo_app}@anchor{308b}
@deffn {Function} wsgiref.simple_server.demo_app (environ, start_response)

This function is a small but complete WSGI application that returns a text page
containing the message “Hello world!” and a list of the key/value pairs provided
in the @emph{environ} parameter.  It’s useful for verifying that a WSGI server (such
as @ref{136,,wsgiref.simple_server}) is able to run a simple WSGI application
correctly.
@end deffn

@geindex WSGIServer (class in wsgiref.simple_server)
@anchor{library/wsgiref wsgiref simple_server WSGIServer}@anchor{308c}
@deffn {Class} wsgiref.simple_server.WSGIServer (server_address, RequestHandlerClass)

Create a @ref{308c,,WSGIServer} instance.  @emph{server_address} should be a
@code{(host,port)} tuple, and @emph{RequestHandlerClass} should be the subclass of
@ref{c5d,,http.server.BaseHTTPRequestHandler} that will be used to process
requests.

You do not normally need to call this constructor, as the @ref{308a,,make_server()}
function can handle all the details for you.

@ref{308c,,WSGIServer} is a subclass of @ref{308d,,http.server.HTTPServer}, so all
of its methods (such as @code{serve_forever()} and @code{handle_request()}) are
available. @ref{308c,,WSGIServer} also provides these WSGI-specific methods:

@geindex set_app() (wsgiref.simple_server.WSGIServer method)
@anchor{library/wsgiref wsgiref simple_server WSGIServer set_app}@anchor{308e}
@deffn {Method} set_app (application)

Sets the callable @emph{application} as the WSGI application that will receive
requests.
@end deffn

@geindex get_app() (wsgiref.simple_server.WSGIServer method)
@anchor{library/wsgiref wsgiref simple_server WSGIServer get_app}@anchor{308f}
@deffn {Method} get_app ()

Returns the currently-set application callable.
@end deffn

Normally, however, you do not need to use these additional methods, as
@ref{308e,,set_app()} is normally called by @ref{308a,,make_server()}, and the
@ref{308f,,get_app()} exists mainly for the benefit of request handler instances.
@end deffn

@geindex WSGIRequestHandler (class in wsgiref.simple_server)
@anchor{library/wsgiref wsgiref simple_server WSGIRequestHandler}@anchor{3090}
@deffn {Class} wsgiref.simple_server.WSGIRequestHandler (request, client_address, server)

Create an HTTP handler for the given @emph{request} (i.e. a socket), @emph{client_address}
(a @code{(host,port)} tuple), and @emph{server} (@ref{308c,,WSGIServer} instance).

You do not need to create instances of this class directly; they are
automatically created as needed by @ref{308c,,WSGIServer} objects.  You can,
however, subclass this class and supply it as a @emph{handler_class} to the
@ref{308a,,make_server()} function.  Some possibly relevant methods for overriding in
subclasses:

@geindex get_environ() (wsgiref.simple_server.WSGIRequestHandler method)
@anchor{library/wsgiref wsgiref simple_server WSGIRequestHandler get_environ}@anchor{3091}
@deffn {Method} get_environ ()

Returns a dictionary containing the WSGI environment for a request.  The default
implementation copies the contents of the @ref{308c,,WSGIServer} object’s
@code{base_environ} dictionary attribute and then adds various headers derived
from the HTTP request.  Each call to this method should return a new dictionary
containing all of the relevant CGI environment variables as specified in
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333}.
@end deffn

@geindex get_stderr() (wsgiref.simple_server.WSGIRequestHandler method)
@anchor{library/wsgiref wsgiref simple_server WSGIRequestHandler get_stderr}@anchor{3092}
@deffn {Method} get_stderr ()

Return the object that should be used as the @code{wsgi.errors} stream. The default
implementation just returns @code{sys.stderr}.
@end deffn

@geindex handle() (wsgiref.simple_server.WSGIRequestHandler method)
@anchor{library/wsgiref wsgiref simple_server WSGIRequestHandler handle}@anchor{3093}
@deffn {Method} handle ()

Process the HTTP request.  The default implementation creates a handler instance
using a @ref{134,,wsgiref.handlers} class to implement the actual WSGI application
interface.
@end deffn
@end deffn

@node wsgiref validate — WSGI conformance checker,wsgiref handlers – server/gateway base classes,wsgiref simple_server – a simple WSGI HTTP server,wsgiref — WSGI Utilities and Reference Implementation
@anchor{library/wsgiref module-wsgiref validate}@anchor{138}@anchor{library/wsgiref wsgiref-validate-wsgi-conformance-checker}@anchor{3094}
@subsubsection @code{wsgiref.validate} — WSGI conformance checker


@geindex module; wsgiref.validate

When creating new WSGI application objects, frameworks, servers, or middleware,
it can be useful to validate the new code’s conformance using
@ref{138,,wsgiref.validate}.  This module provides a function that creates WSGI
application objects that validate communications between a WSGI server or
gateway and a WSGI application object, to check both sides for protocol
conformance.

Note that this utility does not guarantee complete 
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333} compliance; an
absence of errors from this module does not necessarily mean that errors do not
exist.  However, if this module does produce an error, then it is virtually
certain that either the server or application is not 100% compliant.

This module is based on the @code{paste.lint} module from Ian Bicking’s “Python
Paste” library.

@geindex validator() (in module wsgiref.validate)
@anchor{library/wsgiref wsgiref validate validator}@anchor{3095}
@deffn {Function} wsgiref.validate.validator (application)

Wrap @emph{application} and return a new WSGI application object.  The returned
application will forward all requests to the original @emph{application}, and will
check that both the @emph{application} and the server invoking it are conforming to
the WSGI specification and to 
@geindex RFC; RFC 2616
RFC 2616@footnote{https://tools.ietf.org/html/rfc2616.html}.

Any detected nonconformance results in an @ref{46e,,AssertionError} being raised;
note, however, that how these errors are handled is server-dependent.  For
example, @ref{136,,wsgiref.simple_server} and other servers based on
@ref{134,,wsgiref.handlers} (that don’t override the error handling methods to do
something else) will simply output a message that an error has occurred, and
dump the traceback to @code{sys.stderr} or some other error stream.

This wrapper may also generate output using the @ref{12d,,warnings} module to
indicate behaviors that are questionable but which may not actually be
prohibited by 
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333}.  Unless they are suppressed using Python command-line
options or the @ref{12d,,warnings} API, any such warnings will be written to
@code{sys.stderr} (@emph{not} @code{wsgi.errors}, unless they happen to be the same
object).

Example usage:

@example
from wsgiref.validate import validator
from wsgiref.simple_server import make_server

# Our callable object which is intentionally not compliant to the
# standard, so the validator is going to break
def simple_app(environ, start_response):
    status = '200 OK'  # HTTP Status
    headers = [('Content-type', 'text/plain')]  # HTTP Headers
    start_response(status, headers)

    # This is going to break because we need to return a list, and
    # the validator is going to inform us
    return b"Hello World"

# This is the application wrapped in a validator
validator_app = validator(simple_app)

with make_server('@w{'}, 8000, validator_app) as httpd:
    print("Listening on port 8000....")
    httpd.serve_forever()
@end example
@end deffn

@node wsgiref handlers – server/gateway base classes,Examples<23>,wsgiref validate — WSGI conformance checker,wsgiref — WSGI Utilities and Reference Implementation
@anchor{library/wsgiref module-wsgiref handlers}@anchor{134}@anchor{library/wsgiref wsgiref-handlers-server-gateway-base-classes}@anchor{3096}
@subsubsection @code{wsgiref.handlers} – server/gateway base classes


@geindex module; wsgiref.handlers

This module provides base handler classes for implementing WSGI servers and
gateways.  These base classes handle most of the work of communicating with a
WSGI application, as long as they are given a CGI-like environment, along with
input, output, and error streams.

@geindex CGIHandler (class in wsgiref.handlers)
@anchor{library/wsgiref wsgiref handlers CGIHandler}@anchor{3097}
@deffn {Class} wsgiref.handlers.CGIHandler

CGI-based invocation via @code{sys.stdin}, @code{sys.stdout}, @code{sys.stderr} and
@code{os.environ}.  This is useful when you have a WSGI application and want to run
it as a CGI script.  Simply invoke @code{CGIHandler().run(app)}, where @code{app} is
the WSGI application object you wish to invoke.

This class is a subclass of @ref{3098,,BaseCGIHandler} that sets @code{wsgi.run_once}
to true, @code{wsgi.multithread} to false, and @code{wsgi.multiprocess} to true, and
always uses @ref{f7,,sys} and @ref{c0,,os} to obtain the necessary CGI streams and
environment.
@end deffn

@geindex IISCGIHandler (class in wsgiref.handlers)
@anchor{library/wsgiref wsgiref handlers IISCGIHandler}@anchor{3099}
@deffn {Class} wsgiref.handlers.IISCGIHandler

A specialized alternative to @ref{3097,,CGIHandler}, for use when deploying on
Microsoft’s IIS web server, without having set the config allowPathInfo
option (IIS>=7) or metabase allowPathInfoForScriptMappings (IIS<7).

By default, IIS gives a @code{PATH_INFO} that duplicates the @code{SCRIPT_NAME} at
the front, causing problems for WSGI applications that wish to implement
routing. This handler strips any such duplicated path.

IIS can be configured to pass the correct @code{PATH_INFO}, but this causes
another bug where @code{PATH_TRANSLATED} is wrong. Luckily this variable is
rarely used and is not guaranteed by WSGI. On IIS<7, though, the
setting can only be made on a vhost level, affecting all other script
mappings, many of which break when exposed to the @code{PATH_TRANSLATED} bug.
For this reason IIS<7 is almost never deployed with the fix (Even IIS7
rarely uses it because there is still no UI for it.).

There is no way for CGI code to tell whether the option was set, so a
separate handler class is provided.  It is used in the same way as
@ref{3097,,CGIHandler}, i.e., by calling @code{IISCGIHandler().run(app)}, where
@code{app} is the WSGI application object you wish to invoke.


New in version 3.2.

@end deffn

@geindex BaseCGIHandler (class in wsgiref.handlers)
@anchor{library/wsgiref wsgiref handlers BaseCGIHandler}@anchor{3098}
@deffn {Class} wsgiref.handlers.BaseCGIHandler (stdin, stdout, stderr, environ, multithread=True, multiprocess=False)

Similar to @ref{3097,,CGIHandler}, but instead of using the @ref{f7,,sys} and
@ref{c0,,os} modules, the CGI environment and I/O streams are specified explicitly.
The @emph{multithread} and @emph{multiprocess} values are used to set the
@code{wsgi.multithread} and @code{wsgi.multiprocess} flags for any applications run by
the handler instance.

This class is a subclass of @ref{309a,,SimpleHandler} intended for use with
software other than HTTP “origin servers”.  If you are writing a gateway
protocol implementation (such as CGI, FastCGI, SCGI, etc.) that uses a
@code{Status:} header to send an HTTP status, you probably want to subclass this
instead of @ref{309a,,SimpleHandler}.
@end deffn

@geindex SimpleHandler (class in wsgiref.handlers)
@anchor{library/wsgiref wsgiref handlers SimpleHandler}@anchor{309a}
@deffn {Class} wsgiref.handlers.SimpleHandler (stdin, stdout, stderr, environ, multithread=True, multiprocess=False)

Similar to @ref{3098,,BaseCGIHandler}, but designed for use with HTTP origin
servers.  If you are writing an HTTP server implementation, you will probably
want to subclass this instead of @ref{3098,,BaseCGIHandler}.

This class is a subclass of @ref{129a,,BaseHandler}.  It overrides the
@code{__init__()}, @code{get_stdin()}, @code{get_stderr()}, @code{add_cgi_vars()},
@code{_write()}, and @code{_flush()} methods to support explicitly setting the
environment and streams via the constructor.  The supplied environment and
streams are stored in the @code{stdin}, @code{stdout}, @code{stderr}, and
@code{environ} attributes.

The @ref{824,,write()} method of @emph{stdout} should write
each chunk in full, like @ref{823,,io.BufferedIOBase}.
@end deffn

@geindex BaseHandler (class in wsgiref.handlers)
@anchor{library/wsgiref wsgiref handlers BaseHandler}@anchor{129a}
@deffn {Class} wsgiref.handlers.BaseHandler

This is an abstract base class for running WSGI applications.  Each instance
will handle a single HTTP request, although in principle you could create a
subclass that was reusable for multiple requests.

@ref{129a,,BaseHandler} instances have only one method intended for external use:

@geindex run() (wsgiref.handlers.BaseHandler method)
@anchor{library/wsgiref wsgiref handlers BaseHandler run}@anchor{309b}
@deffn {Method} run (app)

Run the specified WSGI application, @emph{app}.
@end deffn

All of the other @ref{129a,,BaseHandler} methods are invoked by this method in the
process of running the application, and thus exist primarily to allow
customizing the process.

The following methods MUST be overridden in a subclass:

@geindex _write() (wsgiref.handlers.BaseHandler method)
@anchor{library/wsgiref wsgiref handlers BaseHandler _write}@anchor{309c}
@deffn {Method} _write (data)

Buffer the bytes @emph{data} for transmission to the client.  It’s okay if this
method actually transmits the data; @ref{129a,,BaseHandler} just separates write
and flush operations for greater efficiency when the underlying system actually
has such a distinction.
@end deffn

@geindex _flush() (wsgiref.handlers.BaseHandler method)
@anchor{library/wsgiref wsgiref handlers BaseHandler _flush}@anchor{309d}
@deffn {Method} _flush ()

Force buffered data to be transmitted to the client.  It’s okay if this method
is a no-op (i.e., if @ref{309c,,_write()} actually sends the data).
@end deffn

@geindex get_stdin() (wsgiref.handlers.BaseHandler method)
@anchor{library/wsgiref wsgiref handlers BaseHandler get_stdin}@anchor{309e}
@deffn {Method} get_stdin ()

Return an input stream object suitable for use as the @code{wsgi.input} of the
request currently being processed.
@end deffn

@geindex get_stderr() (wsgiref.handlers.BaseHandler method)
@anchor{library/wsgiref wsgiref handlers BaseHandler get_stderr}@anchor{309f}
@deffn {Method} get_stderr ()

Return an output stream object suitable for use as the @code{wsgi.errors} of the
request currently being processed.
@end deffn

@geindex add_cgi_vars() (wsgiref.handlers.BaseHandler method)
@anchor{library/wsgiref wsgiref handlers BaseHandler add_cgi_vars}@anchor{30a0}
@deffn {Method} add_cgi_vars ()

Insert CGI variables for the current request into the @code{environ} attribute.
@end deffn

Here are some other methods and attributes you may wish to override. This list
is only a summary, however, and does not include every method that can be
overridden.  You should consult the docstrings and source code for additional
information before attempting to create a customized @ref{129a,,BaseHandler}
subclass.

Attributes and methods for customizing the WSGI environment:

@geindex wsgi_multithread (wsgiref.handlers.BaseHandler attribute)
@anchor{library/wsgiref wsgiref handlers BaseHandler wsgi_multithread}@anchor{30a1}
@deffn {Attribute} wsgi_multithread

The value to be used for the @code{wsgi.multithread} environment variable.  It
defaults to true in @ref{129a,,BaseHandler}, but may have a different default (or
be set by the constructor) in the other subclasses.
@end deffn

@geindex wsgi_multiprocess (wsgiref.handlers.BaseHandler attribute)
@anchor{library/wsgiref wsgiref handlers BaseHandler wsgi_multiprocess}@anchor{30a2}
@deffn {Attribute} wsgi_multiprocess

The value to be used for the @code{wsgi.multiprocess} environment variable.  It
defaults to true in @ref{129a,,BaseHandler}, but may have a different default (or
be set by the constructor) in the other subclasses.
@end deffn

@geindex wsgi_run_once (wsgiref.handlers.BaseHandler attribute)
@anchor{library/wsgiref wsgiref handlers BaseHandler wsgi_run_once}@anchor{30a3}
@deffn {Attribute} wsgi_run_once

The value to be used for the @code{wsgi.run_once} environment variable.  It
defaults to false in @ref{129a,,BaseHandler}, but @ref{3097,,CGIHandler} sets it to
true by default.
@end deffn

@geindex os_environ (wsgiref.handlers.BaseHandler attribute)
@anchor{library/wsgiref wsgiref handlers BaseHandler os_environ}@anchor{30a4}
@deffn {Attribute} os_environ

The default environment variables to be included in every request’s WSGI
environment.  By default, this is a copy of @code{os.environ} at the time that
@ref{134,,wsgiref.handlers} was imported, but subclasses can either create their own
at the class or instance level.  Note that the dictionary should be considered
read-only, since the default value is shared between multiple classes and
instances.
@end deffn

@geindex server_software (wsgiref.handlers.BaseHandler attribute)
@anchor{library/wsgiref wsgiref handlers BaseHandler server_software}@anchor{30a5}
@deffn {Attribute} server_software

If the @ref{30a6,,origin_server} attribute is set, this attribute’s value is used to
set the default @code{SERVER_SOFTWARE} WSGI environment variable, and also to set a
default @code{Server:} header in HTTP responses.  It is ignored for handlers (such
as @ref{3098,,BaseCGIHandler} and @ref{3097,,CGIHandler}) that are not HTTP origin
servers.


Changed in version 3.3: The term “Python” is replaced with implementation specific term like
“CPython”, “Jython” etc.

@end deffn

@geindex get_scheme() (wsgiref.handlers.BaseHandler method)
@anchor{library/wsgiref wsgiref handlers BaseHandler get_scheme}@anchor{30a7}
@deffn {Method} get_scheme ()

Return the URL scheme being used for the current request.  The default
implementation uses the @code{guess_scheme()} function from @ref{137,,wsgiref.util}
to guess whether the scheme should be “http” or “https”, based on the current
request’s @code{environ} variables.
@end deffn

@geindex setup_environ() (wsgiref.handlers.BaseHandler method)
@anchor{library/wsgiref wsgiref handlers BaseHandler setup_environ}@anchor{30a8}
@deffn {Method} setup_environ ()

Set the @code{environ} attribute to a fully-populated WSGI environment.  The
default implementation uses all of the above methods and attributes, plus the
@ref{309e,,get_stdin()}, @ref{309f,,get_stderr()}, and @ref{30a0,,add_cgi_vars()} methods and the
@ref{30a9,,wsgi_file_wrapper} attribute.  It also inserts a @code{SERVER_SOFTWARE} key
if not present, as long as the @ref{30a6,,origin_server} attribute is a true value
and the @ref{30a5,,server_software} attribute is set.
@end deffn

Methods and attributes for customizing exception handling:

@geindex log_exception() (wsgiref.handlers.BaseHandler method)
@anchor{library/wsgiref wsgiref handlers BaseHandler log_exception}@anchor{30aa}
@deffn {Method} log_exception (exc_info)

Log the @emph{exc_info} tuple in the server log.  @emph{exc_info} is a @code{(type, value,
traceback)} tuple.  The default implementation simply writes the traceback to
the request’s @code{wsgi.errors} stream and flushes it.  Subclasses can override
this method to change the format or retarget the output, mail the traceback to
an administrator, or whatever other action may be deemed suitable.
@end deffn

@geindex traceback_limit (wsgiref.handlers.BaseHandler attribute)
@anchor{library/wsgiref wsgiref handlers BaseHandler traceback_limit}@anchor{30ab}
@deffn {Attribute} traceback_limit

The maximum number of frames to include in tracebacks output by the default
@ref{30aa,,log_exception()} method.  If @code{None}, all frames are included.
@end deffn

@geindex error_output() (wsgiref.handlers.BaseHandler method)
@anchor{library/wsgiref wsgiref handlers BaseHandler error_output}@anchor{30ac}
@deffn {Method} error_output (environ, start_response)

This method is a WSGI application to generate an error page for the user.  It is
only invoked if an error occurs before headers are sent to the client.

This method can access the current error information using @code{sys.exc_info()},
and should pass that information to @emph{start_response} when calling it (as
described in the “Error Handling” section of 
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333}).

The default implementation just uses the @ref{30ad,,error_status},
@ref{30ae,,error_headers}, and @ref{30af,,error_body} attributes to generate an output
page.  Subclasses can override this to produce more dynamic error output.

Note, however, that it’s not recommended from a security perspective to spit out
diagnostics to any old user; ideally, you should have to do something special to
enable diagnostic output, which is why the default implementation doesn’t
include any.
@end deffn

@geindex error_status (wsgiref.handlers.BaseHandler attribute)
@anchor{library/wsgiref wsgiref handlers BaseHandler error_status}@anchor{30ad}
@deffn {Attribute} error_status

The HTTP status used for error responses.  This should be a status string as
defined in 
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333}; it defaults to a 500 code and message.
@end deffn

@geindex error_headers (wsgiref.handlers.BaseHandler attribute)
@anchor{library/wsgiref wsgiref handlers BaseHandler error_headers}@anchor{30ae}
@deffn {Attribute} error_headers

The HTTP headers used for error responses.  This should be a list of WSGI
response headers (@code{(name, value)} tuples), as described in 
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333}.  The
default list just sets the content type to @code{text/plain}.
@end deffn

@geindex error_body (wsgiref.handlers.BaseHandler attribute)
@anchor{library/wsgiref wsgiref handlers BaseHandler error_body}@anchor{30af}
@deffn {Attribute} error_body

The error response body.  This should be an HTTP response body bytestring. It
defaults to the plain text, “A server error occurred.  Please contact the
administrator.”
@end deffn

Methods and attributes for 
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333}’s “Optional Platform-Specific File
Handling” feature:

@geindex wsgi_file_wrapper (wsgiref.handlers.BaseHandler attribute)
@anchor{library/wsgiref wsgiref handlers BaseHandler wsgi_file_wrapper}@anchor{30a9}
@deffn {Attribute} wsgi_file_wrapper

A @code{wsgi.file_wrapper} factory, or @code{None}.  The default value of this
attribute is the @ref{1bb,,wsgiref.util.FileWrapper} class.
@end deffn

@geindex sendfile() (wsgiref.handlers.BaseHandler method)
@anchor{library/wsgiref wsgiref handlers BaseHandler sendfile}@anchor{30b0}
@deffn {Method} sendfile ()

Override to implement platform-specific file transmission.  This method is
called only if the application’s return value is an instance of the class
specified by the @ref{30a9,,wsgi_file_wrapper} attribute.  It should return a true
value if it was able to successfully transmit the file, so that the default
transmission code will not be executed. The default implementation of this
method just returns a false value.
@end deffn

Miscellaneous methods and attributes:

@geindex origin_server (wsgiref.handlers.BaseHandler attribute)
@anchor{library/wsgiref wsgiref handlers BaseHandler origin_server}@anchor{30a6}
@deffn {Attribute} origin_server

This attribute should be set to a true value if the handler’s @ref{309c,,_write()} and
@ref{309d,,_flush()} are being used to communicate directly to the client, rather than
via a CGI-like gateway protocol that wants the HTTP status in a special
@code{Status:} header.

This attribute’s default value is true in @ref{129a,,BaseHandler}, but false in
@ref{3098,,BaseCGIHandler} and @ref{3097,,CGIHandler}.
@end deffn

@geindex http_version (wsgiref.handlers.BaseHandler attribute)
@anchor{library/wsgiref wsgiref handlers BaseHandler http_version}@anchor{30b1}
@deffn {Attribute} http_version

If @ref{30a6,,origin_server} is true, this string attribute is used to set the HTTP
version of the response set to the client.  It defaults to @code{"1.0"}.
@end deffn
@end deffn

@geindex read_environ() (in module wsgiref.handlers)
@anchor{library/wsgiref wsgiref handlers read_environ}@anchor{d5f}
@deffn {Function} wsgiref.handlers.read_environ ()

Transcode CGI variables from @code{os.environ} to 
@geindex Python Enhancement Proposals; PEP 3333
PEP 3333@footnote{https://www.python.org/dev/peps/pep-3333} “bytes in unicode”
strings, returning a new dictionary.  This function is used by
@ref{3097,,CGIHandler} and @ref{3099,,IISCGIHandler} in place of directly using
@code{os.environ}, which is not necessarily WSGI-compliant on all platforms
and web servers using Python 3 – specifically, ones where the OS’s
actual environment is Unicode (i.e. Windows), or ones where the environment
is bytes, but the system encoding used by Python to decode it is anything
other than ISO-8859-1 (e.g. Unix systems using UTF-8).

If you are implementing a CGI-based handler of your own, you probably want
to use this routine instead of just copying values out of @code{os.environ}
directly.


New in version 3.2.

@end deffn

@node Examples<23>,,wsgiref handlers – server/gateway base classes,wsgiref — WSGI Utilities and Reference Implementation
@anchor{library/wsgiref examples}@anchor{30b2}
@subsubsection Examples


This is a working “Hello World” WSGI application:

@example
from wsgiref.simple_server import make_server

# Every WSGI application must have an application object - a callable
# object that accepts two arguments. For that purpose, we're going to
# use a function (note that you're not limited to a function, you can
# use a class for example). The first argument passed to the function
# is a dictionary containing CGI-style environment variables and the
# second variable is the callable object.
def hello_world_app(environ, start_response):
    status = '200 OK'  # HTTP Status
    headers = [('Content-type', 'text/plain; charset=utf-8')]  # HTTP Headers
    start_response(status, headers)

    # The returned object is going to be printed
    return [b"Hello World"]

with make_server('@w{'}, 8000, hello_world_app) as httpd:
    print("Serving on port 8000...")

    # Serve until process is killed
    httpd.serve_forever()
@end example

Example of a WSGI application serving the current directory, accept optional
directory and port number (default: 8000) on the command line:

@example
#!/usr/bin/env python3
'@w{'}'
Small wsgiref based web server. Takes a path to serve from and an
optional port number (defaults to 8000), then tries to serve files.
Mime types are guessed from the file names, 404 errors are raised
if the file is not found. Used for the make serve target in Doc.
'@w{'}'
import sys
import os
import mimetypes
from wsgiref import simple_server, util

def app(environ, respond):

    fn = os.path.join(path, environ['PATH_INFO'][1:])
    if '.' not in fn.split(os.path.sep)[-1]:
        fn = os.path.join(fn, 'index.html')
    type = mimetypes.guess_type(fn)[0]

    if os.path.exists(fn):
        respond('200 OK', [('Content-Type', type)])
        return util.FileWrapper(open(fn, "rb"))
    else:
        respond('404 Not Found', [('Content-Type', 'text/plain')])
        return [b'not found']

if __name__ == '__main__':
    path = sys.argv[1] if len(sys.argv) > 1 else os.getcwd()
    port = int(sys.argv[2]) if len(sys.argv) > 2 else 8000
    httpd = simple_server.make_server('@w{'}, port, app)
    print("Serving @{@} on port @{@}, control-C to stop".format(path, port))
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        print("Shutting down.")
        httpd.server_close()
@end example

@node urllib — URL handling modules,urllib request — Extensible library for opening URLs,wsgiref — WSGI Utilities and Reference Implementation,Internet Protocols and Support
@anchor{library/urllib doc}@anchor{30b3}@anchor{library/urllib module-urllib}@anchor{124}@anchor{library/urllib urllib-url-handling-modules}@anchor{30b4}
@subsection @code{urllib} — URL handling modules


@geindex module; urllib

@strong{Source code:} Lib/urllib/@footnote{https://github.com/python/cpython/tree/main/Lib/urllib/}


__________________________________________________________________


@code{urllib} is a package that collects several modules for working with URLs:


@itemize *

@item 
@ref{127,,urllib.request} for opening and reading URLs

@item 
@ref{125,,urllib.error} containing the exceptions raised by @ref{127,,urllib.request}

@item 
@ref{126,,urllib.parse} for parsing URLs

@item 
@ref{129,,urllib.robotparser} for parsing @code{robots.txt} files
@end itemize

@node urllib request — Extensible library for opening URLs,urllib response — Response classes used by urllib,urllib — URL handling modules,Internet Protocols and Support
@anchor{library/urllib request doc}@anchor{30b5}@anchor{library/urllib request module-urllib request}@anchor{127}@anchor{library/urllib request urllib-request-extensible-library-for-opening-urls}@anchor{30b6}
@subsection @code{urllib.request} — Extensible library for opening URLs


@geindex module; urllib.request

@strong{Source code:} Lib/urllib/request.py@footnote{https://github.com/python/cpython/tree/main/Lib/urllib/request.py}


__________________________________________________________________


The @ref{127,,urllib.request} module defines functions and classes which help in
opening URLs (mostly HTTP) in a complex world — basic and digest
authentication, redirections, cookies and more.


@subsubheading See also


The Requests package@footnote{https://requests.readthedocs.io/en/master/}
is recommended for a higher-level HTTP client interface.


The @ref{127,,urllib.request} module defines the following functions:

@geindex urlopen() (in module urllib.request)
@anchor{library/urllib request urllib request urlopen}@anchor{a06}
@deffn {Function} urllib.request.urlopen (url, data=None[, timeout], *, cafile=None, capath=None, cadefault=False, context=None)

Open the URL @emph{url}, which can be either a string or a
@ref{b5e,,Request} object.

@emph{data} must be an object specifying additional data to be sent to the
server, or @code{None} if no such data is needed.  See @ref{b5e,,Request}
for details.

urllib.request module uses HTTP/1.1 and includes @code{Connection:close} header
in its HTTP requests.

The optional @emph{timeout} parameter specifies a timeout in seconds for
blocking operations like the connection attempt (if not specified,
the global default timeout setting will be used).  This actually
only works for HTTP, HTTPS and FTP connections.

If @emph{context} is specified, it must be a @ref{2bf,,ssl.SSLContext} instance
describing the various SSL options. See @ref{664,,HTTPSConnection}
for more details.

The optional @emph{cafile} and @emph{capath} parameters specify a set of trusted
CA certificates for HTTPS requests.  @emph{cafile} should point to a single
file containing a bundle of CA certificates, whereas @emph{capath} should
point to a directory of hashed certificate files.  More information can
be found in @ref{a5f,,ssl.SSLContext.load_verify_locations()}.

The @emph{cadefault} parameter is ignored.

This function always returns an object which can work as a
@ref{166,,context manager} and has the properties @emph{url}, @emph{headers}, and @emph{status}.
See @ref{30b7,,urllib.response.addinfourl} for more detail on these properties.

For HTTP and HTTPS URLs, this function returns a
@ref{c60,,http.client.HTTPResponse} object slightly modified. In addition
to the three new methods above, the msg attribute contains the
same information as the @ref{30b8,,reason}
attribute — the reason phrase returned by server — instead of
the response headers as it is specified in the documentation for
@ref{c60,,HTTPResponse}.

For FTP, file, and data URLs and requests explicitly handled by legacy
@ref{30b9,,URLopener} and @ref{30ba,,FancyURLopener} classes, this function
returns a @ref{30b7,,urllib.response.addinfourl} object.

Raises @ref{30bb,,URLError} on protocol errors.

Note that @code{None} may be returned if no handler handles the request (though
the default installed global @ref{30bc,,OpenerDirector} uses
@ref{30bd,,UnknownHandler} to ensure this never happens).

In addition, if proxy settings are detected (for example, when a @code{*_proxy}
environment variable like 
@geindex http_proxy
@geindex environment variable; http_proxy
@code{http_proxy} is set),
@ref{1270,,ProxyHandler} is default installed and makes sure the requests are
handled through the proxy.

The legacy @code{urllib.urlopen} function from Python 2.6 and earlier has been
discontinued; @ref{a06,,urllib.request.urlopen()} corresponds to the old
@code{urllib2.urlopen}.  Proxy handling, which was done by passing a dictionary
parameter to @code{urllib.urlopen}, can be obtained by using
@ref{1270,,ProxyHandler} objects.


The default opener raises an @ref{1130,,auditing event}
@code{urllib.Request} with arguments @code{fullurl}, @code{data}, @code{headers},
@code{method} taken from the request object.



Changed in version 3.2: @emph{cafile} and @emph{capath} were added.



Changed in version 3.2: HTTPS virtual hosts are now supported if possible (that is, if
@ref{2b4f,,ssl.HAS_SNI} is true).



New in version 3.2: @emph{data} can be an iterable object.



Changed in version 3.3: @emph{cadefault} was added.



Changed in version 3.4.3: @emph{context} was added.



Changed in version 3.10: HTTPS connection now send an ALPN extension with protocol indicator
@code{http/1.1} when no @emph{context} is given. Custom @emph{context} should set
ALPN protocols with @code{set_alpn_protocol()}.



Deprecated since version 3.6: @emph{cafile}, @emph{capath} and @emph{cadefault} are deprecated in favor of @emph{context}.
Please use @ref{cd9,,ssl.SSLContext.load_cert_chain()} instead, or let
@ref{b29,,ssl.create_default_context()} select the system’s trusted CA
certificates for you.

@end deffn

@geindex install_opener() (in module urllib.request)
@anchor{library/urllib request urllib request install_opener}@anchor{30be}
@deffn {Function} urllib.request.install_opener (opener)

Install an @ref{30bc,,OpenerDirector} instance as the default global opener.
Installing an opener is only necessary if you want urlopen to use that
opener; otherwise, simply call @ref{b62,,OpenerDirector.open()} instead of
@ref{a06,,urlopen()}.  The code does not check for a real
@ref{30bc,,OpenerDirector}, and any class with the appropriate interface will
work.
@end deffn

@geindex build_opener() (in module urllib.request)
@anchor{library/urllib request urllib request build_opener}@anchor{30bf}
@deffn {Function} urllib.request.build_opener ([handler, ...])

Return an @ref{30bc,,OpenerDirector} instance, which chains the handlers in the
order given. @emph{handler}s can be either instances of @ref{30c0,,BaseHandler}, or
subclasses of @ref{30c0,,BaseHandler} (in which case it must be possible to call
the constructor without any parameters).  Instances of the following classes
will be in front of the @emph{handler}s, unless the @emph{handler}s contain them,
instances of them or subclasses of them: @ref{1270,,ProxyHandler} (if proxy
settings are detected), @ref{30bd,,UnknownHandler}, @ref{30c1,,HTTPHandler},
@ref{30c2,,HTTPDefaultErrorHandler}, @ref{30c3,,HTTPRedirectHandler},
@ref{30c4,,FTPHandler}, @ref{30c5,,FileHandler}, @ref{30c6,,HTTPErrorProcessor}.

If the Python installation has SSL support (i.e., if the @ref{ee,,ssl} module
can be imported), @ref{dc8,,HTTPSHandler} will also be added.

A @ref{30c0,,BaseHandler} subclass may also change its @code{handler_order}
attribute to modify its position in the handlers list.
@end deffn

@geindex pathname2url() (in module urllib.request)
@anchor{library/urllib request urllib request pathname2url}@anchor{30c7}
@deffn {Function} urllib.request.pathname2url (path)

Convert the pathname @emph{path} from the local syntax for a path to the form used in
the path component of a URL.  This does not produce a complete URL.  The return
value will already be quoted using the @ref{6d4,,quote()} function.
@end deffn

@geindex url2pathname() (in module urllib.request)
@anchor{library/urllib request urllib request url2pathname}@anchor{30c8}
@deffn {Function} urllib.request.url2pathname (path)

Convert the path component @emph{path} from a percent-encoded URL to the local syntax for a
path.  This does not accept a complete URL.  This function uses
@ref{1273,,unquote()} to decode @emph{path}.
@end deffn

@geindex getproxies() (in module urllib.request)
@anchor{library/urllib request urllib request getproxies}@anchor{30c9}
@deffn {Function} urllib.request.getproxies ()

This helper function returns a dictionary of scheme to proxy server URL
mappings. It scans the environment for variables named @code{<scheme>_proxy},
in a case insensitive approach, for all operating systems first, and when it
cannot find it, looks for proxy information from System
Configuration for macOS and Windows Systems Registry for Windows.
If both lowercase and uppercase environment variables exist (and disagree),
lowercase is preferred.

@cartouche
@quotation Note 
If the environment variable @code{REQUEST_METHOD} is set, which usually
indicates your script is running in a CGI environment, the environment
variable @code{HTTP_PROXY} (uppercase @code{_PROXY}) will be ignored. This is
because that variable can be injected by a client using the “Proxy:” HTTP
header. If you need to use an HTTP proxy in a CGI environment, either use
@code{ProxyHandler} explicitly, or make sure the variable name is in
lowercase (or at least the @code{_proxy} suffix).
@end quotation
@end cartouche
@end deffn

The following classes are provided:

@geindex Request (class in urllib.request)
@anchor{library/urllib request urllib request Request}@anchor{b5e}
@deffn {Class} urllib.request.Request (url, data=None, headers=@{@}, origin_req_host=None, unverifiable=False, method=None)

This class is an abstraction of a URL request.

@emph{url} should be a string containing a valid URL.

@emph{data} must be an object specifying additional data to send to the
server, or @code{None} if no such data is needed.  Currently HTTP
requests are the only ones that use @emph{data}.  The supported object
types include bytes, file-like objects, and iterables of bytes-like objects.
If no @code{Content-Length} nor @code{Transfer-Encoding} header field
has been provided, @ref{30c1,,HTTPHandler} will set these headers according
to the type of @emph{data}.  @code{Content-Length} will be used to send
bytes objects, while @code{Transfer-Encoding: chunked} as specified in
@geindex RFC; RFC 7230
RFC 7230@footnote{https://tools.ietf.org/html/rfc7230.html}, Section 3.3.1 will be used to send files and other iterables.

For an HTTP POST request method, @emph{data} should be a buffer in the
standard @code{application/x-www-form-urlencoded} format.  The
@ref{a05,,urllib.parse.urlencode()} function takes a mapping or sequence
of 2-tuples and returns an ASCII string in this format. It should
be encoded to bytes before being used as the @emph{data} parameter.

@emph{headers} should be a dictionary, and will be treated as if
@ref{30ca,,add_header()} was called with each key and value as arguments.
This is often used to “spoof” the @code{User-Agent} header value, which is
used by a browser to identify itself – some HTTP servers only
allow requests coming from common browsers as opposed to scripts.
For example, Mozilla Firefox may identify itself as @code{"Mozilla/5.0
(X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"}, while
@ref{124,,urllib}’s default user agent string is
@code{"Python-urllib/2.6"} (on Python 2.6).

An appropriate @code{Content-Type} header should be included if the @emph{data}
argument is present.  If this header has not been provided and @emph{data}
is not None, @code{Content-Type: application/x-www-form-urlencoded} will
be added as a default.

The next two arguments are only of interest for correct handling
of third-party HTTP cookies:

@emph{origin_req_host} should be the request-host of the origin
transaction, as defined by 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html}.  It defaults to
@code{http.cookiejar.request_host(self)}.  This is the host name or IP
address of the original request that was initiated by the user.
For example, if the request is for an image in an HTML document,
this should be the request-host of the request for the page
containing the image.

@emph{unverifiable} should indicate whether the request is unverifiable,
as defined by 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html}.  It defaults to @code{False}.  An unverifiable
request is one whose URL the user did not have the option to
approve.  For example, if the request is for an image in an HTML
document, and the user had no option to approve the automatic
fetching of the image, this should be true.

@emph{method} should be a string that indicates the HTTP request method that
will be used (e.g. @code{'HEAD'}).  If provided, its value is stored in the
@ref{b5f,,method} attribute and is used by @ref{d06,,get_method()}.
The default is @code{'GET'} if @emph{data} is @code{None} or @code{'POST'} otherwise.
Subclasses may indicate a different default method by setting the
@ref{b5f,,method} attribute in the class itself.

@cartouche
@quotation Note 
The request will not work as expected if the data object is unable
to deliver its content more than once (e.g. a file or an iterable
that can produce the content only once) and the request is retried
for HTTP redirects or authentication.  The @emph{data} is sent to the
HTTP server right away after the headers.  There is no support for
a 100-continue expectation in the library.
@end quotation
@end cartouche


Changed in version 3.3: @ref{b5f,,Request.method} argument is added to the Request class.



Changed in version 3.4: Default @ref{b5f,,Request.method} may be indicated at the class level.



Changed in version 3.6: Do not raise an error if the @code{Content-Length} has not been
provided and @emph{data} is neither @code{None} nor a bytes object.
Fall back to use chunked transfer encoding instead.

@end deffn

@geindex OpenerDirector (class in urllib.request)
@anchor{library/urllib request urllib request OpenerDirector}@anchor{30bc}
@deffn {Class} urllib.request.OpenerDirector

The @ref{30bc,,OpenerDirector} class opens URLs via @ref{30c0,,BaseHandler}s chained
together. It manages the chaining of handlers, and recovery from errors.
@end deffn

@geindex BaseHandler (class in urllib.request)
@anchor{library/urllib request urllib request BaseHandler}@anchor{30c0}
@deffn {Class} urllib.request.BaseHandler

This is the base class for all registered handlers — and handles only the
simple mechanics of registration.
@end deffn

@geindex HTTPDefaultErrorHandler (class in urllib.request)
@anchor{library/urllib request urllib request HTTPDefaultErrorHandler}@anchor{30c2}
@deffn {Class} urllib.request.HTTPDefaultErrorHandler

A class which defines a default handler for HTTP error responses; all responses
are turned into @ref{b64,,HTTPError} exceptions.
@end deffn

@geindex HTTPRedirectHandler (class in urllib.request)
@anchor{library/urllib request urllib request HTTPRedirectHandler}@anchor{30c3}
@deffn {Class} urllib.request.HTTPRedirectHandler

A class to handle redirections.
@end deffn

@geindex HTTPCookieProcessor (class in urllib.request)
@anchor{library/urllib request urllib request HTTPCookieProcessor}@anchor{30cb}
@deffn {Class} urllib.request.HTTPCookieProcessor (cookiejar=None)

A class to handle HTTP Cookies.
@end deffn

@geindex ProxyHandler (class in urllib.request)
@anchor{library/urllib request urllib request ProxyHandler}@anchor{1270}
@deffn {Class} urllib.request.ProxyHandler (proxies=None)

Cause requests to go through a proxy. If @emph{proxies} is given, it must be a
dictionary mapping protocol names to URLs of proxies. The default is to read
the list of proxies from the environment variables
@code{<protocol>_proxy}.  If no proxy environment variables are set, then
in a Windows environment proxy settings are obtained from the registry’s
Internet Settings section, and in a macOS environment proxy information
is retrieved from the System Configuration Framework.

To disable autodetected proxy pass an empty dictionary.

The 
@geindex no_proxy
@geindex environment variable; no_proxy
@code{no_proxy} environment variable can be used to specify hosts
which shouldn’t be reached via proxy; if set, it should be a comma-separated
list of hostname suffixes, optionally with @code{:port} appended, for example
@code{cern.ch,ncsa.uiuc.edu,some.host:8080}.

@quotation

@cartouche
@quotation Note 
@code{HTTP_PROXY} will be ignored if a variable @code{REQUEST_METHOD} is set;
see the documentation on @ref{30c9,,getproxies()}.
@end quotation
@end cartouche
@end quotation
@end deffn

@geindex HTTPPasswordMgr (class in urllib.request)
@anchor{library/urllib request urllib request HTTPPasswordMgr}@anchor{30cc}
@deffn {Class} urllib.request.HTTPPasswordMgr

Keep a database of  @code{(realm, uri) -> (user, password)} mappings.
@end deffn

@geindex HTTPPasswordMgrWithDefaultRealm (class in urllib.request)
@anchor{library/urllib request urllib request HTTPPasswordMgrWithDefaultRealm}@anchor{30cd}
@deffn {Class} urllib.request.HTTPPasswordMgrWithDefaultRealm

Keep a database of  @code{(realm, uri) -> (user, password)} mappings. A realm of
@code{None} is considered a catch-all realm, which is searched if no other realm
fits.
@end deffn

@geindex HTTPPasswordMgrWithPriorAuth (class in urllib.request)
@anchor{library/urllib request urllib request HTTPPasswordMgrWithPriorAuth}@anchor{a04}
@deffn {Class} urllib.request.HTTPPasswordMgrWithPriorAuth

A variant of @ref{30cd,,HTTPPasswordMgrWithDefaultRealm} that also has a
database of @code{uri -> is_authenticated} mappings.  Can be used by a
BasicAuth handler to determine when to send authentication credentials
immediately instead of waiting for a @code{401} response first.


New in version 3.5.

@end deffn

@geindex AbstractBasicAuthHandler (class in urllib.request)
@anchor{library/urllib request urllib request AbstractBasicAuthHandler}@anchor{1131}
@deffn {Class} urllib.request.AbstractBasicAuthHandler (password_mgr=None)

This is a mixin class that helps with HTTP authentication, both to the remote
host and to a proxy. @emph{password_mgr}, if given, should be something that is
compatible with @ref{30cc,,HTTPPasswordMgr}; refer to section
@ref{30ce,,HTTPPasswordMgr Objects} for information on the interface that must be
supported.  If @emph{passwd_mgr} also provides @code{is_authenticated} and
@code{update_authenticated} methods (see
@ref{30cf,,HTTPPasswordMgrWithPriorAuth Objects}), then the handler will use the
@code{is_authenticated} result for a given URI to determine whether or not to
send authentication credentials with the request.  If @code{is_authenticated}
returns @code{True} for the URI, credentials are sent.  If @code{is_authenticated}
is @code{False}, credentials are not sent, and then if a @code{401} response is
received the request is re-sent with the authentication credentials.  If
authentication succeeds, @code{update_authenticated} is called to set
@code{is_authenticated} @code{True} for the URI, so that subsequent requests to
the URI or any of its super-URIs will automatically include the
authentication credentials.


New in version 3.5: Added @code{is_authenticated} support.

@end deffn

@geindex HTTPBasicAuthHandler (class in urllib.request)
@anchor{library/urllib request urllib request HTTPBasicAuthHandler}@anchor{30d0}
@deffn {Class} urllib.request.HTTPBasicAuthHandler (password_mgr=None)

Handle authentication with the remote host. @emph{password_mgr}, if given, should
be something that is compatible with @ref{30cc,,HTTPPasswordMgr}; refer to
section @ref{30ce,,HTTPPasswordMgr Objects} for information on the interface that must
be supported. HTTPBasicAuthHandler will raise a @ref{17c,,ValueError} when
presented with a wrong Authentication scheme.
@end deffn

@geindex ProxyBasicAuthHandler (class in urllib.request)
@anchor{library/urllib request urllib request ProxyBasicAuthHandler}@anchor{30d1}
@deffn {Class} urllib.request.ProxyBasicAuthHandler (password_mgr=None)

Handle authentication with the proxy. @emph{password_mgr}, if given, should be
something that is compatible with @ref{30cc,,HTTPPasswordMgr}; refer to section
@ref{30ce,,HTTPPasswordMgr Objects} for information on the interface that must be
supported.
@end deffn

@geindex AbstractDigestAuthHandler (class in urllib.request)
@anchor{library/urllib request urllib request AbstractDigestAuthHandler}@anchor{1211}
@deffn {Class} urllib.request.AbstractDigestAuthHandler (password_mgr=None)

This is a mixin class that helps with HTTP authentication, both to the remote
host and to a proxy. @emph{password_mgr}, if given, should be something that is
compatible with @ref{30cc,,HTTPPasswordMgr}; refer to section
@ref{30ce,,HTTPPasswordMgr Objects} for information on the interface that must be
supported.
@end deffn

@geindex HTTPDigestAuthHandler (class in urllib.request)
@anchor{library/urllib request urllib request HTTPDigestAuthHandler}@anchor{30d2}
@deffn {Class} urllib.request.HTTPDigestAuthHandler (password_mgr=None)

Handle authentication with the remote host. @emph{password_mgr}, if given, should
be something that is compatible with @ref{30cc,,HTTPPasswordMgr}; refer to
section @ref{30ce,,HTTPPasswordMgr Objects} for information on the interface that must
be supported. When both Digest Authentication Handler and Basic
Authentication Handler are both added, Digest Authentication is always tried
first. If the Digest Authentication returns a 40x response again, it is sent
to Basic Authentication handler to Handle.  This Handler method will raise a
@ref{17c,,ValueError} when presented with an authentication scheme other than
Digest or Basic.


Changed in version 3.3: Raise @ref{17c,,ValueError} on unsupported Authentication Scheme.

@end deffn

@geindex ProxyDigestAuthHandler (class in urllib.request)
@anchor{library/urllib request urllib request ProxyDigestAuthHandler}@anchor{30d3}
@deffn {Class} urllib.request.ProxyDigestAuthHandler (password_mgr=None)

Handle authentication with the proxy. @emph{password_mgr}, if given, should be
something that is compatible with @ref{30cc,,HTTPPasswordMgr}; refer to section
@ref{30ce,,HTTPPasswordMgr Objects} for information on the interface that must be
supported.
@end deffn

@geindex HTTPHandler (class in urllib.request)
@anchor{library/urllib request urllib request HTTPHandler}@anchor{30c1}
@deffn {Class} urllib.request.HTTPHandler

A class to handle opening of HTTP URLs.
@end deffn

@geindex HTTPSHandler (class in urllib.request)
@anchor{library/urllib request urllib request HTTPSHandler}@anchor{dc8}
@deffn {Class} urllib.request.HTTPSHandler (debuglevel=0, context=None, check_hostname=None)

A class to handle opening of HTTPS URLs.  @emph{context} and @emph{check_hostname}
have the same meaning as in @ref{664,,http.client.HTTPSConnection}.


Changed in version 3.2: @emph{context} and @emph{check_hostname} were added.

@end deffn

@geindex FileHandler (class in urllib.request)
@anchor{library/urllib request urllib request FileHandler}@anchor{30c5}
@deffn {Class} urllib.request.FileHandler

Open local files.
@end deffn

@geindex DataHandler (class in urllib.request)
@anchor{library/urllib request urllib request DataHandler}@anchor{b5d}
@deffn {Class} urllib.request.DataHandler

Open data URLs.


New in version 3.4.

@end deffn

@geindex FTPHandler (class in urllib.request)
@anchor{library/urllib request urllib request FTPHandler}@anchor{30c4}
@deffn {Class} urllib.request.FTPHandler

Open FTP URLs.
@end deffn

@geindex CacheFTPHandler (class in urllib.request)
@anchor{library/urllib request urllib request CacheFTPHandler}@anchor{30d4}
@deffn {Class} urllib.request.CacheFTPHandler

Open FTP URLs, keeping a cache of open FTP connections to minimize delays.
@end deffn

@geindex UnknownHandler (class in urllib.request)
@anchor{library/urllib request urllib request UnknownHandler}@anchor{30bd}
@deffn {Class} urllib.request.UnknownHandler

A catch-all class to handle unknown URLs.
@end deffn

@geindex HTTPErrorProcessor (class in urllib.request)
@anchor{library/urllib request urllib request HTTPErrorProcessor}@anchor{30c6}
@deffn {Class} urllib.request.HTTPErrorProcessor

Process HTTP error responses.
@end deffn

@menu
* Request Objects:: 
* OpenerDirector Objects:: 
* BaseHandler Objects:: 
* HTTPRedirectHandler Objects:: 
* HTTPCookieProcessor Objects:: 
* ProxyHandler Objects:: 
* HTTPPasswordMgr Objects:: 
* HTTPPasswordMgrWithPriorAuth Objects:: 
* AbstractBasicAuthHandler Objects:: 
* HTTPBasicAuthHandler Objects:: 
* ProxyBasicAuthHandler Objects:: 
* AbstractDigestAuthHandler Objects:: 
* HTTPDigestAuthHandler Objects:: 
* ProxyDigestAuthHandler Objects:: 
* HTTPHandler Objects:: 
* HTTPSHandler Objects:: 
* FileHandler Objects:: 
* DataHandler Objects:: 
* FTPHandler Objects:: 
* CacheFTPHandler Objects:: 
* UnknownHandler Objects:: 
* HTTPErrorProcessor Objects:: 
* Examples: Examples<24>. 
* Legacy interface:: 
* urllib.request Restrictions: urllib request Restrictions. 

@end menu

@node Request Objects,OpenerDirector Objects,,urllib request — Extensible library for opening URLs
@anchor{library/urllib request id1}@anchor{30d5}@anchor{library/urllib request request-objects}@anchor{30d6}
@subsubsection Request Objects


The following methods describe @ref{b5e,,Request}’s public interface,
and so all may be overridden in subclasses.  It also defines several
public attributes that can be used by clients to inspect the parsed
request.

@geindex full_url (urllib.request.Request attribute)
@anchor{library/urllib request urllib request Request full_url}@anchor{b60}
@deffn {Attribute} Request.full_url

The original URL passed to the constructor.


Changed in version 3.4.


Request.full_url is a property with setter, getter and a deleter. Getting
@ref{b60,,full_url} returns the original request URL with the
fragment, if it was present.
@end deffn

@geindex type (urllib.request.Request attribute)
@anchor{library/urllib request urllib request Request type}@anchor{30d7}
@deffn {Attribute} Request.type

The URI scheme.
@end deffn

@geindex host (urllib.request.Request attribute)
@anchor{library/urllib request urllib request Request host}@anchor{30d8}
@deffn {Attribute} Request.host

The URI authority, typically a host, but may also contain a port
separated by a colon.
@end deffn

@geindex origin_req_host (urllib.request.Request attribute)
@anchor{library/urllib request urllib request Request origin_req_host}@anchor{30d9}
@deffn {Attribute} Request.origin_req_host

The original host for the request, without port.
@end deffn

@geindex selector (urllib.request.Request attribute)
@anchor{library/urllib request urllib request Request selector}@anchor{30da}
@deffn {Attribute} Request.selector

The URI path.  If the @ref{b5e,,Request} uses a proxy, then selector
will be the full URL that is passed to the proxy.
@end deffn

@geindex data (urllib.request.Request attribute)
@anchor{library/urllib request urllib request Request data}@anchor{b61}
@deffn {Attribute} Request.data

The entity body for the request, or @code{None} if not specified.


Changed in version 3.4: Changing value of @ref{b61,,Request.data} now deletes “Content-Length”
header if it was previously set or calculated.

@end deffn

@geindex unverifiable (urllib.request.Request attribute)
@anchor{library/urllib request urllib request Request unverifiable}@anchor{30db}
@deffn {Attribute} Request.unverifiable

boolean, indicates whether the request is unverifiable as defined
by 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html}.
@end deffn

@geindex method (urllib.request.Request attribute)
@anchor{library/urllib request urllib request Request method}@anchor{b5f}
@deffn {Attribute} Request.method

The HTTP request method to use.  By default its value is @ref{182,,None},
which means that @ref{d06,,get_method()} will do its normal computation
of the method to be used.  Its value can be set (thus overriding the default
computation in @ref{d06,,get_method()}) either by providing a default
value by setting it at the class level in a @ref{b5e,,Request} subclass, or by
passing a value in to the @ref{b5e,,Request} constructor via the @emph{method}
argument.


New in version 3.3.



Changed in version 3.4: A default value can now be set in subclasses; previously it could only
be set via the constructor argument.

@end deffn

@geindex get_method() (urllib.request.Request method)
@anchor{library/urllib request urllib request Request get_method}@anchor{d06}
@deffn {Method} Request.get_method ()

Return a string indicating the HTTP request method.  If
@ref{b5f,,Request.method} is not @code{None}, return its value, otherwise return
@code{'GET'} if @ref{b61,,Request.data} is @code{None}, or @code{'POST'} if it’s not.
This is only meaningful for HTTP requests.


Changed in version 3.3: get_method now looks at the value of @ref{b5f,,Request.method}.

@end deffn

@geindex add_header() (urllib.request.Request method)
@anchor{library/urllib request urllib request Request add_header}@anchor{30ca}
@deffn {Method} Request.add_header (key, val)

Add another header to the request.  Headers are currently ignored by all
handlers except HTTP handlers, where they are added to the list of headers sent
to the server.  Note that there cannot be more than one header with the same
name, and later calls will overwrite previous calls in case the @emph{key} collides.
Currently, this is no loss of HTTP functionality, since all headers which have
meaning when used more than once have a (header-specific) way of gaining the
same functionality using only one header.
@end deffn

@geindex add_unredirected_header() (urllib.request.Request method)
@anchor{library/urllib request urllib request Request add_unredirected_header}@anchor{30dc}
@deffn {Method} Request.add_unredirected_header (key, header)

Add a header that will not be added to a redirected request.
@end deffn

@geindex has_header() (urllib.request.Request method)
@anchor{library/urllib request urllib request Request has_header}@anchor{30dd}
@deffn {Method} Request.has_header (header)

Return whether the instance has the named header (checks both regular and
unredirected).
@end deffn

@geindex remove_header() (urllib.request.Request method)
@anchor{library/urllib request urllib request Request remove_header}@anchor{b63}
@deffn {Method} Request.remove_header (header)

Remove named header from the request instance (both from regular and
unredirected headers).


New in version 3.4.

@end deffn

@geindex get_full_url() (urllib.request.Request method)
@anchor{library/urllib request urllib request Request get_full_url}@anchor{30de}
@deffn {Method} Request.get_full_url ()

Return the URL given in the constructor.


Changed in version 3.4.


Returns @ref{b60,,Request.full_url}
@end deffn

@geindex set_proxy() (urllib.request.Request method)
@anchor{library/urllib request urllib request Request set_proxy}@anchor{30df}
@deffn {Method} Request.set_proxy (host, type)

Prepare the request by connecting to a proxy server. The @emph{host} and @emph{type} will
replace those of the instance, and the instance’s selector will be the original
URL given in the constructor.
@end deffn

@geindex get_header() (urllib.request.Request method)
@anchor{library/urllib request urllib request Request get_header}@anchor{30e0}
@deffn {Method} Request.get_header (header_name, default=None)

Return the value of the given header. If the header is not present, return
the default value.
@end deffn

@geindex header_items() (urllib.request.Request method)
@anchor{library/urllib request urllib request Request header_items}@anchor{30e1}
@deffn {Method} Request.header_items ()

Return a list of tuples (header_name, header_value) of the Request headers.
@end deffn


Changed in version 3.4: The request methods add_data, has_data, get_data, get_type, get_host,
get_selector, get_origin_req_host and is_unverifiable that were deprecated
since 3.3 have been removed.


@node OpenerDirector Objects,BaseHandler Objects,Request Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request opener-director-objects}@anchor{30e2}@anchor{library/urllib request openerdirector-objects}@anchor{30e3}
@subsubsection OpenerDirector Objects


@ref{30bc,,OpenerDirector} instances have the following methods:

@geindex add_handler() (urllib.request.OpenerDirector method)
@anchor{library/urllib request urllib request OpenerDirector add_handler}@anchor{30e4}
@deffn {Method} OpenerDirector.add_handler (handler)

@emph{handler} should be an instance of @ref{30c0,,BaseHandler}.  The following methods
are searched, and added to the possible chains (note that HTTP errors are a
special case).  Note that, in the following, @emph{protocol} should be replaced
with the actual protocol to handle, for example @code{http_response()} would
be the HTTP protocol response handler.  Also @emph{type} should be replaced with
the actual HTTP code, for example @code{http_error_404()} would handle HTTP
404 errors.


@itemize *

@item 
@code{<protocol>_open()} — signal that the handler knows how to open @emph{protocol}
URLs.

See @ref{30e5,,BaseHandler.<protocol>_open()} for more information.

@item 
@code{http_error_<type>()} — signal that the handler knows how to handle HTTP
errors with HTTP error code @emph{type}.

See @ref{30e6,,BaseHandler.http_error_<nnn>()} for more information.

@item 
@code{<protocol>_error()} — signal that the handler knows how to handle errors
from (non-@code{http}) @emph{protocol}.

@item 
@code{<protocol>_request()} — signal that the handler knows how to pre-process
@emph{protocol} requests.

See @ref{30e7,,BaseHandler.<protocol>_request()} for more information.

@item 
@code{<protocol>_response()} — signal that the handler knows how to
post-process @emph{protocol} responses.

See @ref{30e8,,BaseHandler.<protocol>_response()} for more information.
@end itemize
@end deffn

@geindex open() (urllib.request.OpenerDirector method)
@anchor{library/urllib request urllib request OpenerDirector open}@anchor{b62}
@deffn {Method} OpenerDirector.open (url, data=None[, timeout])

Open the given @emph{url} (which can be a request object or a string), optionally
passing the given @emph{data}. Arguments, return values and exceptions raised are
the same as those of @ref{a06,,urlopen()} (which simply calls the @ref{1c8,,open()}
method on the currently installed global @ref{30bc,,OpenerDirector}).  The
optional @emph{timeout} parameter specifies a timeout in seconds for blocking
operations like the connection attempt (if not specified, the global default
timeout setting will be used). The timeout feature actually works only for
HTTP, HTTPS and FTP connections.
@end deffn

@geindex error() (urllib.request.OpenerDirector method)
@anchor{library/urllib request urllib request OpenerDirector error}@anchor{30e9}
@deffn {Method} OpenerDirector.error (proto, *args)

Handle an error of the given protocol.  This will call the registered error
handlers for the given protocol with the given arguments (which are protocol
specific).  The HTTP protocol is a special case which uses the HTTP response
code to determine the specific error handler; refer to the @code{http_error_<type>()}
methods of the handler classes.

Return values and exceptions raised are the same as those of @ref{a06,,urlopen()}.
@end deffn

OpenerDirector objects open URLs in three stages:

The order in which these methods are called within each stage is determined by
sorting the handler instances.


@enumerate 

@item 
Every handler with a method named like @code{<protocol>_request()} has that
method called to pre-process the request.

@item 
Handlers with a method named like @code{<protocol>_open()} are called to handle
the request. This stage ends when a handler either returns a non-@ref{182,,None}
value (ie. a response), or raises an exception (usually
@ref{30bb,,URLError}).  Exceptions are allowed to propagate.

In fact, the above algorithm is first tried for methods named
@code{default_open()}.  If all such methods return @ref{182,,None}, the algorithm
is repeated for methods named like @code{<protocol>_open()}.  If all such methods
return @ref{182,,None}, the algorithm is repeated for methods named
@code{unknown_open()}.

Note that the implementation of these methods may involve calls of the parent
@ref{30bc,,OpenerDirector} instance’s @ref{b62,,open()} and
@ref{30e9,,error()} methods.

@item 
Every handler with a method named like @code{<protocol>_response()} has that
method called to post-process the response.
@end enumerate

@node BaseHandler Objects,HTTPRedirectHandler Objects,OpenerDirector Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request base-handler-objects}@anchor{30ea}@anchor{library/urllib request basehandler-objects}@anchor{30eb}
@subsubsection BaseHandler Objects


@ref{30c0,,BaseHandler} objects provide a couple of methods that are directly
useful, and others that are meant to be used by derived classes.  These are
intended for direct use:

@geindex add_parent() (urllib.request.BaseHandler method)
@anchor{library/urllib request urllib request BaseHandler add_parent}@anchor{30ec}
@deffn {Method} BaseHandler.add_parent (director)

Add a director as parent.
@end deffn

@geindex close() (urllib.request.BaseHandler method)
@anchor{library/urllib request urllib request BaseHandler close}@anchor{30ed}
@deffn {Method} BaseHandler.close ()

Remove any parents.
@end deffn

The following attribute and methods should only be used by classes derived from
@ref{30c0,,BaseHandler}.

@cartouche
@quotation Note 
The convention has been adopted that subclasses defining
@code{<protocol>_request()} or @code{<protocol>_response()} methods are named
@code{*Processor}; all others are named @code{*Handler}.
@end quotation
@end cartouche

@geindex parent (urllib.request.BaseHandler attribute)
@anchor{library/urllib request urllib request BaseHandler parent}@anchor{30ee}
@deffn {Attribute} BaseHandler.parent

A valid @ref{30bc,,OpenerDirector}, which can be used to open using a different
protocol, or handle errors.
@end deffn

@geindex default_open() (urllib.request.BaseHandler method)
@anchor{library/urllib request urllib request BaseHandler default_open}@anchor{30ef}
@deffn {Method} BaseHandler.default_open (req)

This method is @emph{not} defined in @ref{30c0,,BaseHandler}, but subclasses should
define it if they want to catch all URLs.

This method, if implemented, will be called by the parent
@ref{30bc,,OpenerDirector}.  It should return a file-like object as described in
the return value of the @ref{1c8,,open()} of @ref{30bc,,OpenerDirector}, or @code{None}.
It should raise @ref{30bb,,URLError}, unless a truly exceptional
thing happens (for example, @ref{10de,,MemoryError} should not be mapped to
@code{URLError}).

This method will be called before any protocol-specific open method.
@end deffn
@anchor{library/urllib request protocol-open}@anchor{30e5}

@deffn {Method} BaseHandler.<protocol>_open(req)

This method is @emph{not} defined in @ref{30c0,,BaseHandler}, but subclasses should
define it if they want to handle URLs with the given protocol.

This method, if defined, will be called by the parent @ref{30bc,,OpenerDirector}.
Return values should be the same as for  @code{default_open()}.
@end deffn

@geindex unknown_open() (urllib.request.BaseHandler method)
@anchor{library/urllib request urllib request BaseHandler unknown_open}@anchor{30f0}
@deffn {Method} BaseHandler.unknown_open (req)

This method is @emph{not} defined in @ref{30c0,,BaseHandler}, but subclasses should
define it if they want to catch all URLs with no specific registered handler to
open it.

This method, if implemented, will be called by the @ref{30ee,,parent}
@ref{30bc,,OpenerDirector}.  Return values should be the same as for
@ref{30ef,,default_open()}.
@end deffn

@geindex http_error_default() (urllib.request.BaseHandler method)
@anchor{library/urllib request urllib request BaseHandler http_error_default}@anchor{30f1}
@deffn {Method} BaseHandler.http_error_default (req, fp, code, msg, hdrs)

This method is @emph{not} defined in @ref{30c0,,BaseHandler}, but subclasses should
override it if they intend to provide a catch-all for otherwise unhandled HTTP
errors.  It will be called automatically by the  @ref{30bc,,OpenerDirector} getting
the error, and should not normally be called in other circumstances.

@emph{req} will be a @ref{b5e,,Request} object, @emph{fp} will be a file-like object with
the HTTP error body, @emph{code} will be the three-digit code of the error, @emph{msg}
will be the user-visible explanation of the code and @emph{hdrs} will be a mapping
object with the headers of the error.

Return values and exceptions raised should be the same as those of
@ref{a06,,urlopen()}.
@end deffn
@anchor{library/urllib request http-error-nnn}@anchor{30e6}

@deffn {Method} BaseHandler.http_error_<nnn>(req, fp, code, msg, hdrs)

@emph{nnn} should be a three-digit HTTP error code.  This method is also not defined
in @ref{30c0,,BaseHandler}, but will be called, if it exists, on an instance of a
subclass, when an HTTP error with code @emph{nnn} occurs.

Subclasses should override this method to handle specific HTTP errors.

Arguments, return values and exceptions raised should be the same as for
@code{http_error_default()}.
@end deffn
@anchor{library/urllib request protocol-request}@anchor{30e7}

@deffn {Method} BaseHandler.<protocol>_request(req)

This method is @emph{not} defined in @ref{30c0,,BaseHandler}, but subclasses should
define it if they want to pre-process requests of the given protocol.

This method, if defined, will be called by the parent @ref{30bc,,OpenerDirector}.
@emph{req} will be a @ref{b5e,,Request} object. The return value should be a
@ref{b5e,,Request} object.
@end deffn
@anchor{library/urllib request protocol-response}@anchor{30e8}

@deffn {Method} BaseHandler.<protocol>_response(req, response)

This method is @emph{not} defined in @ref{30c0,,BaseHandler}, but subclasses should
define it if they want to post-process responses of the given protocol.

This method, if defined, will be called by the parent @ref{30bc,,OpenerDirector}.
@emph{req} will be a @ref{b5e,,Request} object. @emph{response} will be an object
implementing the same interface as the return value of @ref{a06,,urlopen()}.  The
return value should implement the same interface as the return value of
@ref{a06,,urlopen()}.
@end deffn

@node HTTPRedirectHandler Objects,HTTPCookieProcessor Objects,BaseHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request http-redirect-handler}@anchor{30f2}@anchor{library/urllib request httpredirecthandler-objects}@anchor{30f3}
@subsubsection HTTPRedirectHandler Objects


@cartouche
@quotation Note 
Some HTTP redirections require action from this module’s client code.  If this
is the case, @ref{b64,,HTTPError} is raised.  See 
@geindex RFC; RFC 2616
RFC 2616@footnote{https://tools.ietf.org/html/rfc2616.html} for
details of the precise meanings of the various redirection codes.

An @code{HTTPError} exception raised as a security consideration if the
HTTPRedirectHandler is presented with a redirected URL which is not an HTTP,
HTTPS or FTP URL.
@end quotation
@end cartouche

@geindex redirect_request() (urllib.request.HTTPRedirectHandler method)
@anchor{library/urllib request urllib request HTTPRedirectHandler redirect_request}@anchor{30f4}
@deffn {Method} HTTPRedirectHandler.redirect_request (req, fp, code, msg, hdrs, newurl)

Return a @ref{b5e,,Request} or @code{None} in response to a redirect. This is called
by the default implementations of the @code{http_error_30*()} methods when a
redirection is received from the server.  If a redirection should take place,
return a new @ref{b5e,,Request} to allow @code{http_error_30*()} to perform the
redirect to @emph{newurl}.  Otherwise, raise @ref{b64,,HTTPError} if
no other handler should try to handle this URL, or return @code{None} if you
can’t but another handler might.

@cartouche
@quotation Note 
The default implementation of this method does not strictly follow 
@geindex RFC; RFC 2616
RFC 2616@footnote{https://tools.ietf.org/html/rfc2616.html},
which says that 301 and 302 responses to @code{POST} requests must not be
automatically redirected without confirmation by the user.  In reality, browsers
do allow automatic redirection of these responses, changing the POST to a
@code{GET}, and the default implementation reproduces this behavior.
@end quotation
@end cartouche
@end deffn

@geindex http_error_301() (urllib.request.HTTPRedirectHandler method)
@anchor{library/urllib request urllib request HTTPRedirectHandler http_error_301}@anchor{30f5}
@deffn {Method} HTTPRedirectHandler.http_error_301 (req, fp, code, msg, hdrs)

Redirect to the @code{Location:} or @code{URI:} URL.  This method is called by the
parent @ref{30bc,,OpenerDirector} when getting an HTTP ‘moved permanently’ response.
@end deffn

@geindex http_error_302() (urllib.request.HTTPRedirectHandler method)
@anchor{library/urllib request urllib request HTTPRedirectHandler http_error_302}@anchor{30f6}
@deffn {Method} HTTPRedirectHandler.http_error_302 (req, fp, code, msg, hdrs)

The same as @ref{30f5,,http_error_301()}, but called for the ‘found’ response.
@end deffn

@geindex http_error_303() (urllib.request.HTTPRedirectHandler method)
@anchor{library/urllib request urllib request HTTPRedirectHandler http_error_303}@anchor{30f7}
@deffn {Method} HTTPRedirectHandler.http_error_303 (req, fp, code, msg, hdrs)

The same as @ref{30f5,,http_error_301()}, but called for the ‘see other’ response.
@end deffn

@geindex http_error_307() (urllib.request.HTTPRedirectHandler method)
@anchor{library/urllib request urllib request HTTPRedirectHandler http_error_307}@anchor{30f8}
@deffn {Method} HTTPRedirectHandler.http_error_307 (req, fp, code, msg, hdrs)

The same as @ref{30f5,,http_error_301()}, but called for the ‘temporary redirect’
response. It does not allow changing the request method from @code{POST}
to @code{GET}.
@end deffn

@geindex http_error_308() (urllib.request.HTTPRedirectHandler method)
@anchor{library/urllib request urllib request HTTPRedirectHandler http_error_308}@anchor{30f9}
@deffn {Method} HTTPRedirectHandler.http_error_308 (req, fp, code, msg, hdrs)

The same as @ref{30f5,,http_error_301()}, but called for the ‘permanent redirect’
response. It does not allow changing the request method from @code{POST}
to @code{GET}.


New in version 3.11.

@end deffn

@node HTTPCookieProcessor Objects,ProxyHandler Objects,HTTPRedirectHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request http-cookie-processor}@anchor{30fa}@anchor{library/urllib request httpcookieprocessor-objects}@anchor{30fb}
@subsubsection HTTPCookieProcessor Objects


@ref{30cb,,HTTPCookieProcessor} instances have one attribute:

@geindex cookiejar (urllib.request.HTTPCookieProcessor attribute)
@anchor{library/urllib request urllib request HTTPCookieProcessor cookiejar}@anchor{30fc}
@deffn {Attribute} HTTPCookieProcessor.cookiejar

The @ref{30fd,,http.cookiejar.CookieJar} in which cookies are stored.
@end deffn

@node ProxyHandler Objects,HTTPPasswordMgr Objects,HTTPCookieProcessor Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request proxy-handler}@anchor{30fe}@anchor{library/urllib request proxyhandler-objects}@anchor{30ff}
@subsubsection ProxyHandler Objects



@deffn {Method} ProxyHandler.<protocol>_open(request)

The @ref{1270,,ProxyHandler} will have a method @code{<protocol>_open()} for every
@emph{protocol} which has a proxy in the @emph{proxies} dictionary given in the
constructor.  The method will modify requests to go through the proxy, by
calling @code{request.set_proxy()}, and call the next handler in the chain to
actually execute the protocol.
@end deffn

@node HTTPPasswordMgr Objects,HTTPPasswordMgrWithPriorAuth Objects,ProxyHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request http-password-mgr}@anchor{30ce}@anchor{library/urllib request httppasswordmgr-objects}@anchor{3100}
@subsubsection HTTPPasswordMgr Objects


These methods are available on @ref{30cc,,HTTPPasswordMgr} and
@ref{30cd,,HTTPPasswordMgrWithDefaultRealm} objects.

@geindex add_password() (urllib.request.HTTPPasswordMgr method)
@anchor{library/urllib request urllib request HTTPPasswordMgr add_password}@anchor{3101}
@deffn {Method} HTTPPasswordMgr.add_password (realm, uri, user, passwd)

@emph{uri} can be either a single URI, or a sequence of URIs. @emph{realm}, @emph{user} and
@emph{passwd} must be strings. This causes @code{(user, passwd)} to be used as
authentication tokens when authentication for @emph{realm} and a super-URI of any of
the given URIs is given.
@end deffn

@geindex find_user_password() (urllib.request.HTTPPasswordMgr method)
@anchor{library/urllib request urllib request HTTPPasswordMgr find_user_password}@anchor{108c}
@deffn {Method} HTTPPasswordMgr.find_user_password (realm, authuri)

Get user/password for given realm and URI, if any.  This method will return
@code{(None, None)} if there is no matching user/password.

For @ref{30cd,,HTTPPasswordMgrWithDefaultRealm} objects, the realm @code{None} will be
searched if the given @emph{realm} has no matching user/password.
@end deffn

@node HTTPPasswordMgrWithPriorAuth Objects,AbstractBasicAuthHandler Objects,HTTPPasswordMgr Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request http-password-mgr-with-prior-auth}@anchor{30cf}@anchor{library/urllib request httppasswordmgrwithpriorauth-objects}@anchor{3102}
@subsubsection HTTPPasswordMgrWithPriorAuth Objects


This password manager extends @ref{30cd,,HTTPPasswordMgrWithDefaultRealm} to support
tracking URIs for which authentication credentials should always be sent.

@geindex add_password() (urllib.request.HTTPPasswordMgrWithPriorAuth method)
@anchor{library/urllib request urllib request HTTPPasswordMgrWithPriorAuth add_password}@anchor{3103}
@deffn {Method} HTTPPasswordMgrWithPriorAuth.add_password (realm, uri, user, passwd, is_authenticated=False)

@emph{realm}, @emph{uri}, @emph{user}, @emph{passwd} are as for
@ref{3101,,HTTPPasswordMgr.add_password()}.  @emph{is_authenticated} sets the initial
value of the @code{is_authenticated} flag for the given URI or list of URIs.
If @emph{is_authenticated} is specified as @code{True}, @emph{realm} is ignored.
@end deffn

@geindex find_user_password() (urllib.request.HTTPPasswordMgrWithPriorAuth method)
@anchor{library/urllib request urllib request HTTPPasswordMgrWithPriorAuth find_user_password}@anchor{3104}
@deffn {Method} HTTPPasswordMgrWithPriorAuth.find_user_password (realm, authuri)

Same as for @ref{30cd,,HTTPPasswordMgrWithDefaultRealm} objects
@end deffn

@geindex update_authenticated() (urllib.request.HTTPPasswordMgrWithPriorAuth method)
@anchor{library/urllib request urllib request HTTPPasswordMgrWithPriorAuth update_authenticated}@anchor{3105}
@deffn {Method} HTTPPasswordMgrWithPriorAuth.update_authenticated (self, uri, is_authenticated=False)

Update the @code{is_authenticated} flag for the given @emph{uri} or list
of URIs.
@end deffn

@geindex is_authenticated() (urllib.request.HTTPPasswordMgrWithPriorAuth method)
@anchor{library/urllib request urllib request HTTPPasswordMgrWithPriorAuth is_authenticated}@anchor{108d}
@deffn {Method} HTTPPasswordMgrWithPriorAuth.is_authenticated (self, authuri)

Returns the current state of the @code{is_authenticated} flag for
the given URI.
@end deffn

@node AbstractBasicAuthHandler Objects,HTTPBasicAuthHandler Objects,HTTPPasswordMgrWithPriorAuth Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request abstract-basic-auth-handler}@anchor{3106}@anchor{library/urllib request abstractbasicauthhandler-objects}@anchor{3107}
@subsubsection AbstractBasicAuthHandler Objects


@geindex http_error_auth_reqed() (urllib.request.AbstractBasicAuthHandler method)
@anchor{library/urllib request urllib request AbstractBasicAuthHandler http_error_auth_reqed}@anchor{3108}
@deffn {Method} AbstractBasicAuthHandler.http_error_auth_reqed (authreq, host, req, headers)

Handle an authentication request by getting a user/password pair, and re-trying
the request.  @emph{authreq} should be the name of the header where the information
about the realm is included in the request, @emph{host} specifies the URL and path to
authenticate for, @emph{req} should be the (failed) @ref{b5e,,Request} object, and
@emph{headers} should be the error headers.

@emph{host} is either an authority (e.g. @code{"python.org"}) or a URL containing an
authority component (e.g. @code{"http://python.org/"}). In either case, the
authority must not contain a userinfo component (so, @code{"python.org"} and
@code{"python.org:80"} are fine, @code{"joe:password@@python.org"} is not).
@end deffn

@node HTTPBasicAuthHandler Objects,ProxyBasicAuthHandler Objects,AbstractBasicAuthHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request http-basic-auth-handler}@anchor{3109}@anchor{library/urllib request httpbasicauthhandler-objects}@anchor{310a}
@subsubsection HTTPBasicAuthHandler Objects


@geindex http_error_401() (urllib.request.HTTPBasicAuthHandler method)
@anchor{library/urllib request urllib request HTTPBasicAuthHandler http_error_401}@anchor{310b}
@deffn {Method} HTTPBasicAuthHandler.http_error_401 (req, fp, code, msg, hdrs)

Retry the request with authentication information, if available.
@end deffn

@node ProxyBasicAuthHandler Objects,AbstractDigestAuthHandler Objects,HTTPBasicAuthHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request proxy-basic-auth-handler}@anchor{310c}@anchor{library/urllib request proxybasicauthhandler-objects}@anchor{310d}
@subsubsection ProxyBasicAuthHandler Objects


@geindex http_error_407() (urllib.request.ProxyBasicAuthHandler method)
@anchor{library/urllib request urllib request ProxyBasicAuthHandler http_error_407}@anchor{310e}
@deffn {Method} ProxyBasicAuthHandler.http_error_407 (req, fp, code, msg, hdrs)

Retry the request with authentication information, if available.
@end deffn

@node AbstractDigestAuthHandler Objects,HTTPDigestAuthHandler Objects,ProxyBasicAuthHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request abstract-digest-auth-handler}@anchor{310f}@anchor{library/urllib request abstractdigestauthhandler-objects}@anchor{3110}
@subsubsection AbstractDigestAuthHandler Objects


@geindex http_error_auth_reqed() (urllib.request.AbstractDigestAuthHandler method)
@anchor{library/urllib request urllib request AbstractDigestAuthHandler http_error_auth_reqed}@anchor{3111}
@deffn {Method} AbstractDigestAuthHandler.http_error_auth_reqed (authreq, host, req, headers)

@emph{authreq} should be the name of the header where the information about the realm
is included in the request, @emph{host} should be the host to authenticate to, @emph{req}
should be the (failed) @ref{b5e,,Request} object, and @emph{headers} should be the
error headers.
@end deffn

@node HTTPDigestAuthHandler Objects,ProxyDigestAuthHandler Objects,AbstractDigestAuthHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request http-digest-auth-handler}@anchor{3112}@anchor{library/urllib request httpdigestauthhandler-objects}@anchor{3113}
@subsubsection HTTPDigestAuthHandler Objects


@geindex http_error_401() (urllib.request.HTTPDigestAuthHandler method)
@anchor{library/urllib request urllib request HTTPDigestAuthHandler http_error_401}@anchor{3114}
@deffn {Method} HTTPDigestAuthHandler.http_error_401 (req, fp, code, msg, hdrs)

Retry the request with authentication information, if available.
@end deffn

@node ProxyDigestAuthHandler Objects,HTTPHandler Objects,HTTPDigestAuthHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request proxy-digest-auth-handler}@anchor{3115}@anchor{library/urllib request proxydigestauthhandler-objects}@anchor{3116}
@subsubsection ProxyDigestAuthHandler Objects


@geindex http_error_407() (urllib.request.ProxyDigestAuthHandler method)
@anchor{library/urllib request urllib request ProxyDigestAuthHandler http_error_407}@anchor{3117}
@deffn {Method} ProxyDigestAuthHandler.http_error_407 (req, fp, code, msg, hdrs)

Retry the request with authentication information, if available.
@end deffn

@node HTTPHandler Objects,HTTPSHandler Objects,ProxyDigestAuthHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request http-handler-objects}@anchor{3118}@anchor{library/urllib request httphandler-objects}@anchor{3119}
@subsubsection HTTPHandler Objects


@geindex http_open() (urllib.request.HTTPHandler method)
@anchor{library/urllib request urllib request HTTPHandler http_open}@anchor{311a}
@deffn {Method} HTTPHandler.http_open (req)

Send an HTTP request, which can be either GET or POST, depending on
@code{req.has_data()}.
@end deffn

@node HTTPSHandler Objects,FileHandler Objects,HTTPHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request https-handler-objects}@anchor{311b}@anchor{library/urllib request httpshandler-objects}@anchor{311c}
@subsubsection HTTPSHandler Objects


@geindex https_open() (urllib.request.HTTPSHandler method)
@anchor{library/urllib request urllib request HTTPSHandler https_open}@anchor{311d}
@deffn {Method} HTTPSHandler.https_open (req)

Send an HTTPS request, which can be either GET or POST, depending on
@code{req.has_data()}.
@end deffn

@node FileHandler Objects,DataHandler Objects,HTTPSHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request file-handler-objects}@anchor{311e}@anchor{library/urllib request filehandler-objects}@anchor{311f}
@subsubsection FileHandler Objects


@geindex file_open() (urllib.request.FileHandler method)
@anchor{library/urllib request urllib request FileHandler file_open}@anchor{3120}
@deffn {Method} FileHandler.file_open (req)

Open the file locally, if there is no host name, or the host name is
@code{'localhost'}.


Changed in version 3.2: This method is applicable only for local hostnames.  When a remote
hostname is given, an @ref{30bb,,URLError} is raised.

@end deffn

@node DataHandler Objects,FTPHandler Objects,FileHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request data-handler-objects}@anchor{3121}@anchor{library/urllib request datahandler-objects}@anchor{3122}
@subsubsection DataHandler Objects


@geindex data_open() (urllib.request.DataHandler method)
@anchor{library/urllib request urllib request DataHandler data_open}@anchor{3123}
@deffn {Method} DataHandler.data_open (req)

Read a data URL. This kind of URL contains the content encoded in the URL
itself. The data URL syntax is specified in 
@geindex RFC; RFC 2397
RFC 2397@footnote{https://tools.ietf.org/html/rfc2397.html}. This implementation
ignores white spaces in base64 encoded data URLs so the URL may be wrapped
in whatever source file it comes from. But even though some browsers don’t
mind about a missing padding at the end of a base64 encoded data URL, this
implementation will raise an @ref{17c,,ValueError} in that case.
@end deffn

@node FTPHandler Objects,CacheFTPHandler Objects,DataHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request ftp-handler-objects}@anchor{3124}@anchor{library/urllib request ftphandler-objects}@anchor{3125}
@subsubsection FTPHandler Objects


@geindex ftp_open() (urllib.request.FTPHandler method)
@anchor{library/urllib request urllib request FTPHandler ftp_open}@anchor{3126}
@deffn {Method} FTPHandler.ftp_open (req)

Open the FTP file indicated by @emph{req}. The login is always done with empty
username and password.
@end deffn

@node CacheFTPHandler Objects,UnknownHandler Objects,FTPHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request cacheftp-handler-objects}@anchor{3127}@anchor{library/urllib request cacheftphandler-objects}@anchor{3128}
@subsubsection CacheFTPHandler Objects


@ref{30d4,,CacheFTPHandler} objects are @ref{30c4,,FTPHandler} objects with the
following additional methods:

@geindex setTimeout() (urllib.request.CacheFTPHandler method)
@anchor{library/urllib request urllib request CacheFTPHandler setTimeout}@anchor{3129}
@deffn {Method} CacheFTPHandler.setTimeout (t)

Set timeout of connections to @emph{t} seconds.
@end deffn

@geindex setMaxConns() (urllib.request.CacheFTPHandler method)
@anchor{library/urllib request urllib request CacheFTPHandler setMaxConns}@anchor{312a}
@deffn {Method} CacheFTPHandler.setMaxConns (m)

Set maximum number of cached connections to @emph{m}.
@end deffn

@node UnknownHandler Objects,HTTPErrorProcessor Objects,CacheFTPHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request unknown-handler-objects}@anchor{312b}@anchor{library/urllib request unknownhandler-objects}@anchor{312c}
@subsubsection UnknownHandler Objects


@geindex unknown_open() (urllib.request.UnknownHandler method)
@anchor{library/urllib request urllib request UnknownHandler unknown_open}@anchor{312d}
@deffn {Method} UnknownHandler.unknown_open ()

Raise a @ref{30bb,,URLError} exception.
@end deffn

@node HTTPErrorProcessor Objects,Examples<24>,UnknownHandler Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request http-error-processor-objects}@anchor{312e}@anchor{library/urllib request httperrorprocessor-objects}@anchor{312f}
@subsubsection HTTPErrorProcessor Objects


@geindex http_response() (urllib.request.HTTPErrorProcessor method)
@anchor{library/urllib request urllib request HTTPErrorProcessor http_response}@anchor{3130}
@deffn {Method} HTTPErrorProcessor.http_response (request, response)

Process HTTP error responses.

For 200 error codes, the response object is returned immediately.

For non-200 error codes, this simply passes the job on to the
@code{http_error_<type>()} handler methods, via @ref{30e9,,OpenerDirector.error()}.
Eventually, @ref{30c2,,HTTPDefaultErrorHandler} will raise an
@ref{b64,,HTTPError} if no other handler handles the error.
@end deffn

@geindex https_response() (urllib.request.HTTPErrorProcessor method)
@anchor{library/urllib request urllib request HTTPErrorProcessor https_response}@anchor{3131}
@deffn {Method} HTTPErrorProcessor.https_response (request, response)

Process HTTPS error responses.

The behavior is same as @ref{3130,,http_response()}.
@end deffn

@node Examples<24>,Legacy interface,HTTPErrorProcessor Objects,urllib request — Extensible library for opening URLs
@anchor{library/urllib request examples}@anchor{3132}@anchor{library/urllib request urllib-request-examples}@anchor{3133}
@subsubsection Examples


In addition to the examples below, more examples are given in
@ref{3134,,HOWTO Fetch Internet Resources Using The urllib Package}.

This example gets the python.org main page and displays the first 300 bytes of
it.

@example
>>> import urllib.request
>>> with urllib.request.urlopen('http://www.python.org/') as f:
...     print(f.read(300))
...
b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n\n\n<html
xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">\n\n<head>\n
<meta http-equiv="content-type" content="text/html; charset=utf-8" />\n
<title>Python Programming '
@end example

Note that urlopen returns a bytes object.  This is because there is no way
for urlopen to automatically determine the encoding of the byte stream
it receives from the HTTP server. In general, a program will decode
the returned bytes object to string once it determines or guesses
the appropriate encoding.

The following W3C document, @indicateurl{https://www.w3.org/International/O-charset}, lists
the various ways in which an (X)HTML or an XML document could have specified its
encoding information.

As the python.org website uses @emph{utf-8} encoding as specified in its meta tag, we
will use the same for decoding the bytes object.

@example
>>> with urllib.request.urlopen('http://www.python.org/') as f:
...     print(f.read(100).decode('utf-8'))
...
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtm
@end example

It is also possible to achieve the same result without using the
@ref{166,,context manager} approach.

@example
>>> import urllib.request
>>> f = urllib.request.urlopen('http://www.python.org/')
>>> print(f.read(100).decode('utf-8'))
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtm
@end example

In the following example, we are sending a data-stream to the stdin of a CGI
and reading the data it returns to us. Note that this example will only work
when the Python installation supports SSL.

@example
>>> import urllib.request
>>> req = urllib.request.Request(url='https://localhost/cgi-bin/test.cgi',
...                       data=b'This data is passed to stdin of the CGI')
>>> with urllib.request.urlopen(req) as f:
...     print(f.read().decode('utf-8'))
...
Got Data: "This data is passed to stdin of the CGI"
@end example

The code for the sample CGI used in the above example is:

@example
#!/usr/bin/env python
import sys
data = sys.stdin.read()
print('Content-type: text/plain\n\nGot Data: "%s"' % data)
@end example

Here is an example of doing a @code{PUT} request using @ref{b5e,,Request}:

@example
import urllib.request
DATA = b'some data'
req = urllib.request.Request(url='http://localhost:8080', data=DATA, method='PUT')
with urllib.request.urlopen(req) as f:
    pass
print(f.status)
print(f.reason)
@end example

Use of Basic HTTP Authentication:

@example
import urllib.request
# Create an OpenerDirector with support for Basic HTTP Authentication...
auth_handler = urllib.request.HTTPBasicAuthHandler()
auth_handler.add_password(realm='PDQ Application',
                          uri='https://mahler:8092/site-updates.py',
                          user='klem',
                          passwd='kadidd!ehopper')
opener = urllib.request.build_opener(auth_handler)
# ...and install it globally so it can be used with urlopen.
urllib.request.install_opener(opener)
urllib.request.urlopen('http://www.example.com/login.html')
@end example

@ref{30bf,,build_opener()} provides many handlers by default, including a
@ref{1270,,ProxyHandler}.  By default, @ref{1270,,ProxyHandler} uses the environment
variables named @code{<scheme>_proxy}, where @code{<scheme>} is the URL scheme
involved.  For example, the 
@geindex http_proxy
@geindex environment variable; http_proxy
@code{http_proxy} environment variable is read to
obtain the HTTP proxy’s URL.

This example replaces the default @ref{1270,,ProxyHandler} with one that uses
programmatically-supplied proxy URLs, and adds proxy authorization support with
@ref{30d1,,ProxyBasicAuthHandler}.

@example
proxy_handler = urllib.request.ProxyHandler(@{'http': 'http://www.example.com:3128/'@})
proxy_auth_handler = urllib.request.ProxyBasicAuthHandler()
proxy_auth_handler.add_password('realm', 'host', 'username', 'password')

opener = urllib.request.build_opener(proxy_handler, proxy_auth_handler)
# This time, rather than install the OpenerDirector, we use it directly:
opener.open('http://www.example.com/login.html')
@end example

Adding HTTP headers:

Use the @emph{headers} argument to the @ref{b5e,,Request} constructor, or:

@example
import urllib.request
req = urllib.request.Request('http://www.example.com/')
req.add_header('Referer', 'http://www.python.org/')
# Customize the default User-Agent header value:
req.add_header('User-Agent', 'urllib-example/0.1 (Contact: . . .)')
r = urllib.request.urlopen(req)
@end example

@ref{30bc,,OpenerDirector} automatically adds a @code{User-Agent} header to
every @ref{b5e,,Request}.  To change this:

@example
import urllib.request
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
opener.open('http://www.example.com/')
@end example

Also, remember that a few standard headers (@code{Content-Length},
@code{Content-Type} and @code{Host})
are added when the @ref{b5e,,Request} is passed to @ref{a06,,urlopen()} (or
@ref{b62,,OpenerDirector.open()}).
@anchor{library/urllib request urllib-examples}@anchor{3135}
Here is an example session that uses the @code{GET} method to retrieve a URL
containing parameters:

@example
>>> import urllib.request
>>> import urllib.parse
>>> params = urllib.parse.urlencode(@{'spam': 1, 'eggs': 2, 'bacon': 0@})
>>> url = "http://www.musi-cal.com/cgi-bin/query?%s" % params
>>> with urllib.request.urlopen(url) as f:
...     print(f.read().decode('utf-8'))
...
@end example

The following example uses the @code{POST} method instead. Note that params output
from urlencode is encoded to bytes before it is sent to urlopen as data:

@example
>>> import urllib.request
>>> import urllib.parse
>>> data = urllib.parse.urlencode(@{'spam': 1, 'eggs': 2, 'bacon': 0@})
>>> data = data.encode('ascii')
>>> with urllib.request.urlopen("http://requestb.in/xrbl82xr", data) as f:
...     print(f.read().decode('utf-8'))
...
@end example

The following example uses an explicitly specified HTTP proxy, overriding
environment settings:

@example
>>> import urllib.request
>>> proxies = @{'http': 'http://proxy.example.com:8080/'@}
>>> opener = urllib.request.FancyURLopener(proxies)
>>> with opener.open("http://www.python.org") as f:
...     f.read().decode('utf-8')
...
@end example

The following example uses no proxies at all, overriding environment settings:

@example
>>> import urllib.request
>>> opener = urllib.request.FancyURLopener(@{@})
>>> with opener.open("http://www.python.org/") as f:
...     f.read().decode('utf-8')
...
@end example

@node Legacy interface,urllib request Restrictions,Examples<24>,urllib request — Extensible library for opening URLs
@anchor{library/urllib request legacy-interface}@anchor{3136}
@subsubsection Legacy interface


The following functions and classes are ported from the Python 2 module
@code{urllib} (as opposed to @code{urllib2}).  They might become deprecated at
some point in the future.

@geindex urlretrieve() (in module urllib.request)
@anchor{library/urllib request urllib request urlretrieve}@anchor{3137}
@deffn {Function} urllib.request.urlretrieve (url, filename=None, reporthook=None, data=None)

Copy a network object denoted by a URL to a local file. If the URL
points to a local file, the object will not be copied unless filename is supplied.
Return a tuple @code{(filename, headers)} where @emph{filename} is the
local file name under which the object can be found, and @emph{headers} is whatever
the @code{info()} method of the object returned by @ref{a06,,urlopen()} returned (for
a remote object). Exceptions are the same as for @ref{a06,,urlopen()}.

The second argument, if present, specifies the file location to copy to (if
absent, the location will be a tempfile with a generated name). The third
argument, if present, is a callable that will be called once on
establishment of the network connection and once after each block read
thereafter.  The callable will be passed three arguments; a count of blocks
transferred so far, a block size in bytes, and the total size of the file.  The
third argument may be @code{-1} on older FTP servers which do not return a file
size in response to a retrieval request.

The following example illustrates the most common usage scenario:

@example
>>> import urllib.request
>>> local_filename, headers = urllib.request.urlretrieve('http://python.org/')
>>> html = open(local_filename)
>>> html.close()
@end example

If the @emph{url} uses the @code{http:} scheme identifier, the optional @emph{data}
argument may be given to specify a @code{POST} request (normally the request
type is @code{GET}).  The @emph{data} argument must be a bytes object in standard
@code{application/x-www-form-urlencoded} format; see the
@ref{a05,,urllib.parse.urlencode()} function.

@ref{3137,,urlretrieve()} will raise @code{ContentTooShortError} when it detects that
the amount of data available  was less than the expected amount (which is the
size reported by a  @emph{Content-Length} header). This can occur, for example, when
the  download is interrupted.

The @emph{Content-Length} is treated as a lower bound: if there’s more data  to read,
urlretrieve reads more data, but if less data is available,  it raises the
exception.

You can still retrieve the downloaded data in this case, it is stored  in the
@code{content} attribute of the exception instance.

If no @emph{Content-Length} header was supplied, urlretrieve can not check the size
of the data it has downloaded, and just returns it.  In this case you just have
to assume that the download was successful.
@end deffn

@geindex urlcleanup() (in module urllib.request)
@anchor{library/urllib request urllib request urlcleanup}@anchor{1279}
@deffn {Function} urllib.request.urlcleanup ()

Cleans up temporary files that may have been left behind by previous
calls to @ref{3137,,urlretrieve()}.
@end deffn

@geindex URLopener (class in urllib.request)
@anchor{library/urllib request urllib request URLopener}@anchor{30b9}
@deffn {Class} urllib.request.URLopener (proxies=None, **x509)


Deprecated since version 3.3.


Base class for opening and reading URLs.  Unless you need to support opening
objects using schemes other than @code{http:}, @code{ftp:}, or @code{file:},
you probably want to use @ref{30ba,,FancyURLopener}.

By default, the @ref{30b9,,URLopener} class sends a @code{User-Agent} header
of @code{urllib/VVV}, where @emph{VVV} is the @ref{124,,urllib} version number.
Applications can define their own @code{User-Agent} header by subclassing
@ref{30b9,,URLopener} or @ref{30ba,,FancyURLopener} and setting the class attribute
@ref{3138,,version} to an appropriate string value in the subclass definition.

The optional @emph{proxies} parameter should be a dictionary mapping scheme names to
proxy URLs, where an empty dictionary turns proxies off completely.  Its default
value is @code{None}, in which case environmental proxy settings will be used if
present, as discussed in the definition of @ref{a06,,urlopen()}, above.

Additional keyword parameters, collected in @emph{x509}, may be used for
authentication of the client when using the @code{https:} scheme.  The keywords
@emph{key_file} and @emph{cert_file} are supported to provide an  SSL key and certificate;
both are needed to support client authentication.

@ref{30b9,,URLopener} objects will raise an @ref{2a1,,OSError} exception if the server
returns an error code.

@geindex open() (urllib.request.URLopener method)
@anchor{library/urllib request urllib request URLopener open}@anchor{3139}
@deffn {Method} open (fullurl, data=None)

Open @emph{fullurl} using the appropriate protocol.  This method sets up cache and
proxy information, then calls the appropriate open method with its input
arguments.  If the scheme is not recognized, @ref{313a,,open_unknown()} is called.
The @emph{data} argument has the same meaning as the @emph{data} argument of
@ref{a06,,urlopen()}.

This method always quotes @emph{fullurl} using @ref{6d4,,quote()}.
@end deffn

@geindex open_unknown() (urllib.request.URLopener method)
@anchor{library/urllib request urllib request URLopener open_unknown}@anchor{313a}
@deffn {Method} open_unknown (fullurl, data=None)

Overridable interface to open unknown URL types.
@end deffn

@geindex retrieve() (urllib.request.URLopener method)
@anchor{library/urllib request urllib request URLopener retrieve}@anchor{1297}
@deffn {Method} retrieve (url, filename=None, reporthook=None, data=None)

Retrieves the contents of @emph{url} and places it in @emph{filename}.  The return value
is a tuple consisting of a local filename and either an
@ref{7e3,,email.message.Message} object containing the response headers (for remote
URLs) or @code{None} (for local URLs).  The caller must then open and read the
contents of @emph{filename}.  If @emph{filename} is not given and the URL refers to a
local file, the input filename is returned.  If the URL is non-local and
@emph{filename} is not given, the filename is the output of @ref{10f2,,tempfile.mktemp()}
with a suffix that matches the suffix of the last path component of the input
URL.  If @emph{reporthook} is given, it must be a function accepting three numeric
parameters: A chunk number, the maximum size chunks are read in and the total size of the download
(-1 if unknown).  It will be called once at the start and after each chunk of data is read from the
network.  @emph{reporthook} is ignored for local URLs.

If the @emph{url} uses the @code{http:} scheme identifier, the optional @emph{data}
argument may be given to specify a @code{POST} request (normally the request type
is @code{GET}).  The @emph{data} argument must in standard
@code{application/x-www-form-urlencoded} format; see the
@ref{a05,,urllib.parse.urlencode()} function.
@end deffn

@geindex version (urllib.request.URLopener attribute)
@anchor{library/urllib request urllib request URLopener version}@anchor{3138}
@deffn {Attribute} version

Variable that specifies the user agent of the opener object.  To get
@ref{124,,urllib} to tell servers that it is a particular user agent, set this in a
subclass as a class variable or in the constructor before calling the base
constructor.
@end deffn
@end deffn

@geindex FancyURLopener (class in urllib.request)
@anchor{library/urllib request urllib request FancyURLopener}@anchor{30ba}
@deffn {Class} urllib.request.FancyURLopener (...)


Deprecated since version 3.3.


@ref{30ba,,FancyURLopener} subclasses @ref{30b9,,URLopener} providing default handling
for the following HTTP response codes: 301, 302, 303, 307 and 401.  For the 30x
response codes listed above, the @code{Location} header is used to fetch
the actual URL.  For 401 response codes (authentication required), basic HTTP
authentication is performed.  For the 30x response codes, recursion is bounded
by the value of the @emph{maxtries} attribute, which defaults to 10.

For all other response codes, the method @code{http_error_default()} is called
which you can override in subclasses to handle the error appropriately.

@cartouche
@quotation Note 
According to the letter of 
@geindex RFC; RFC 2616
RFC 2616@footnote{https://tools.ietf.org/html/rfc2616.html}, 301 and 302 responses to POST requests
must not be automatically redirected without confirmation by the user.  In
reality, browsers do allow automatic redirection of these responses, changing
the POST to a GET, and @ref{124,,urllib} reproduces this behaviour.
@end quotation
@end cartouche

The parameters to the constructor are the same as those for @ref{30b9,,URLopener}.

@cartouche
@quotation Note 
When performing basic authentication, a @ref{30ba,,FancyURLopener} instance calls
its @ref{313b,,prompt_user_passwd()} method.  The default implementation asks the
users for the required information on the controlling terminal.  A subclass may
override this method to support more appropriate behavior if needed.
@end quotation
@end cartouche

The @ref{30ba,,FancyURLopener} class offers one additional method that should be
overloaded to provide the appropriate behavior:

@geindex prompt_user_passwd() (urllib.request.FancyURLopener method)
@anchor{library/urllib request urllib request FancyURLopener prompt_user_passwd}@anchor{313b}
@deffn {Method} prompt_user_passwd (host, realm)

Return information needed to authenticate the user at the given host in the
specified security realm.  The return value should be a tuple, @code{(user,
password)}, which can be used for basic authentication.

The implementation prompts for this information on the terminal; an application
should override this method to use an appropriate interaction model in the local
environment.
@end deffn
@end deffn

@node urllib request Restrictions,,Legacy interface,urllib request — Extensible library for opening URLs
@anchor{library/urllib request urllib-request-restrictions}@anchor{313c}
@subsubsection @code{urllib.request} Restrictions


@quotation

@geindex HTTP; protocol
@geindex FTP; protocol
@end quotation


@itemize *

@item 
Currently, only the following protocols are supported: HTTP (versions 0.9 and
1.0), FTP, local files, and data URLs.


Changed in version 3.4: Added support for data URLs.


@item 
The caching feature of @ref{3137,,urlretrieve()} has been disabled until someone
finds the time to hack proper processing of Expiration time headers.

@item 
There should be a function to query whether a particular URL is in the cache.

@item 
For backward compatibility, if a URL appears to point to a local file but the
file can’t be opened, the URL is re-interpreted using the FTP protocol.  This
can sometimes cause confusing error messages.

@item 
The @ref{a06,,urlopen()} and @ref{3137,,urlretrieve()} functions can cause arbitrarily
long delays while waiting for a network connection to be set up.  This means
that it is difficult to build an interactive web client using these functions
without using threads.

@geindex HTML
@geindex HTTP; protocol

@item 
The data returned by @ref{a06,,urlopen()} or @ref{3137,,urlretrieve()} is the raw data
returned by the server.  This may be binary data (such as an image), plain text
or (for example) HTML.  The HTTP protocol provides type information in the reply
header, which can be inspected by looking at the @code{Content-Type}
header.  If the returned data is HTML, you can use the module
@ref{8d,,html.parser} to parse it.

@geindex FTP

@item 
The code handling the FTP protocol cannot differentiate between a file and a
directory.  This can lead to unexpected behavior when attempting to read a URL
that points to a file that is not accessible.  If the URL ends in a @code{/}, it is
assumed to refer to a directory and will be handled accordingly.  But if an
attempt to read a file leads to a 550 error (meaning the URL cannot be found or
is not accessible, often for permission reasons), then the path is treated as a
directory in order to handle the case when a directory is specified by a URL but
the trailing @code{/} has been left off.  This can cause misleading results when
you try to fetch a file whose read permissions make it inaccessible; the FTP
code will try to read it, fail with a 550 error, and then perform a directory
listing for the unreadable file. If fine-grained control is needed, consider
using the @ref{7e,,ftplib} module, subclassing @ref{30ba,,FancyURLopener}, or changing
@emph{_urlopener} to meet your needs.
@end itemize

@node urllib response — Response classes used by urllib,urllib parse — Parse URLs into components,urllib request — Extensible library for opening URLs,Internet Protocols and Support
@anchor{library/urllib request module-urllib response}@anchor{128}@anchor{library/urllib request urllib-response-response-classes-used-by-urllib}@anchor{313d}
@subsection @code{urllib.response} — Response classes used by urllib


@geindex module; urllib.response

The @ref{128,,urllib.response} module defines functions and classes which define a
minimal file-like interface, including @code{read()} and @code{readline()}.
Functions defined by this module are used internally by the @ref{127,,urllib.request} module.
The typical response object is a @ref{30b7,,urllib.response.addinfourl} instance:

@geindex addinfourl (class in urllib.response)
@anchor{library/urllib request urllib response addinfourl}@anchor{30b7}
@deffn {Class} urllib.response.addinfourl

@geindex url (urllib.response.addinfourl attribute)
@anchor{library/urllib request urllib response addinfourl url}@anchor{313e}
@deffn {Attribute} url

URL of the resource retrieved, commonly used to determine if a redirect was followed.
@end deffn

@geindex headers (urllib.response.addinfourl attribute)
@anchor{library/urllib request urllib response addinfourl headers}@anchor{313f}
@deffn {Attribute} headers

Returns the headers of the response in the form of an @ref{7e4,,EmailMessage} instance.
@end deffn

@geindex status (urllib.response.addinfourl attribute)
@anchor{library/urllib request urllib response addinfourl status}@anchor{3140}
@deffn {Attribute} status


New in version 3.9.


Status code returned by server.
@end deffn

@geindex geturl() (urllib.response.addinfourl method)
@anchor{library/urllib request urllib response addinfourl geturl}@anchor{3141}
@deffn {Method} geturl ()


Deprecated since version 3.9: Deprecated in favor of @ref{313e,,url}.

@end deffn

@geindex info() (urllib.response.addinfourl method)
@anchor{library/urllib request urllib response addinfourl info}@anchor{3142}
@deffn {Method} info ()


Deprecated since version 3.9: Deprecated in favor of @ref{313f,,headers}.

@end deffn

@geindex code (urllib.response.addinfourl attribute)
@anchor{library/urllib request urllib response addinfourl code}@anchor{3143}
@deffn {Attribute} code


Deprecated since version 3.9: Deprecated in favor of @ref{3140,,status}.

@end deffn

@geindex getstatus() (urllib.response.addinfourl method)
@anchor{library/urllib request urllib response addinfourl getstatus}@anchor{3144}
@deffn {Method} getstatus ()


Deprecated since version 3.9: Deprecated in favor of @ref{3140,,status}.

@end deffn
@end deffn

@node urllib parse — Parse URLs into components,urllib error — Exception classes raised by urllib request,urllib response — Response classes used by urllib,Internet Protocols and Support
@anchor{library/urllib parse doc}@anchor{3145}@anchor{library/urllib parse module-urllib parse}@anchor{126}@anchor{library/urllib parse urllib-parse-parse-urls-into-components}@anchor{3146}
@subsection @code{urllib.parse} — Parse URLs into components


@geindex module; urllib.parse

@strong{Source code:} Lib/urllib/parse.py@footnote{https://github.com/python/cpython/tree/main/Lib/urllib/parse.py}

@geindex WWW
@geindex World Wide Web
@geindex URL
@geindex URL; parsing
@geindex relative; URL


__________________________________________________________________


This module defines a standard interface to break Uniform Resource Locator (URL)
strings up in components (addressing scheme, network location, path etc.), to
combine the components back into a URL string, and to convert a “relative URL”
to an absolute URL given a “base URL.”

The module has been designed to match the internet RFC on Relative Uniform
Resource Locators. It supports the following URL schemes: @code{file}, @code{ftp},
@code{gopher}, @code{hdl}, @code{http}, @code{https}, @code{imap}, @code{mailto}, @code{mms},
@code{news}, @code{nntp}, @code{prospero}, @code{rsync}, @code{rtsp}, @code{rtspu}, @code{sftp},
@code{shttp}, @code{sip}, @code{sips}, @code{snews}, @code{svn}, @code{svn+ssh}, @code{telnet},
@code{wais}, @code{ws}, @code{wss}.

The @ref{126,,urllib.parse} module defines functions that fall into two broad
categories: URL parsing and URL quoting. These are covered in detail in
the following sections.

@menu
* URL Parsing:: 
* Parsing ASCII Encoded Bytes:: 
* Structured Parse Results:: 
* URL Quoting:: 

@end menu

@node URL Parsing,Parsing ASCII Encoded Bytes,,urllib parse — Parse URLs into components
@anchor{library/urllib parse url-parsing}@anchor{3147}
@subsubsection URL Parsing


The URL parsing functions focus on splitting a URL string into its components,
or on combining URL components into a URL string.

@geindex urlparse() (in module urllib.parse)
@anchor{library/urllib parse urllib parse urlparse}@anchor{890}
@deffn {Function} urllib.parse.urlparse (urlstring, scheme='@w{'}, allow_fragments=True)

Parse a URL into six components, returning a 6-item @ref{ce9,,named tuple}.  This
corresponds to the general structure of a URL:
@code{scheme://netloc/path;parameters?query#fragment}.
Each tuple item is a string, possibly empty. The components are not broken up
into smaller parts (for example, the network location is a single string), and %
escapes are not expanded. The delimiters as shown above are not part of the
result, except for a leading slash in the @emph{path} component, which is retained if
present.  For example:

@example
>>> from urllib.parse import urlparse
>>> urlparse("scheme://netloc/path;parameters?query#fragment")
ParseResult(scheme='scheme', netloc='netloc', path='/path;parameters', params='@w{'},
            query='query', fragment='fragment')
>>> o = urlparse("http://docs.python.org:80/3/library/urllib.parse.html?"
...              "highlight=params#url-parsing")
>>> o
ParseResult(scheme='http', netloc='docs.python.org:80',
            path='/3/library/urllib.parse.html', params='@w{'},
            query='highlight=params', fragment='url-parsing')
>>> o.scheme
'http'
>>> o.netloc
'docs.python.org:80'
>>> o.hostname
'docs.python.org'
>>> o.port
80
>>> o._replace(fragment="").geturl()
'http://docs.python.org:80/3/library/urllib.parse.html?highlight=params'
@end example

Following the syntax specifications in 
@geindex RFC; RFC 1808
RFC 1808@footnote{https://tools.ietf.org/html/rfc1808.html}, urlparse recognizes
a netloc only if it is properly introduced by ‘//’.  Otherwise the
input is presumed to be a relative URL and thus to start with
a path component.

@example
>>> from urllib.parse import urlparse
>>> urlparse('//www.cwi.nl:80/%7Eguido/Python.html')
ParseResult(scheme='@w{'}, netloc='www.cwi.nl:80', path='/%7Eguido/Python.html',
            params='@w{'}, query='@w{'}, fragment='@w{'})
>>> urlparse('www.cwi.nl/%7Eguido/Python.html')
ParseResult(scheme='@w{'}, netloc='@w{'}, path='www.cwi.nl/%7Eguido/Python.html',
            params='@w{'}, query='@w{'}, fragment='@w{'})
>>> urlparse('help/Python.html')
ParseResult(scheme='@w{'}, netloc='@w{'}, path='help/Python.html', params='@w{'},
            query='@w{'}, fragment='@w{'})
@end example

The @emph{scheme} argument gives the default addressing scheme, to be
used only if the URL does not specify one.  It should be the same type
(text or bytes) as @emph{urlstring}, except that the default value @code{'@w{'}} is
always allowed, and is automatically converted to @code{b'@w{'}} if appropriate.

If the @emph{allow_fragments} argument is false, fragment identifiers are not
recognized.  Instead, they are parsed as part of the path, parameters
or query component, and @code{fragment} is set to the empty string in
the return value.

The return value is a @ref{ce9,,named tuple}, which means that its items can
be accessed by index or as named attributes, which are:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Attribute

@tab

Index

@tab

Value

@tab

Value if not present

@item

@code{scheme}

@tab

0

@tab

URL scheme specifier

@tab

@emph{scheme} parameter

@item

@code{netloc}

@tab

1

@tab

Network location part

@tab

empty string

@item

@code{path}

@tab

2

@tab

Hierarchical path

@tab

empty string

@item

@code{params}

@tab

3

@tab

No longer used

@tab

always an empty string

@item

@code{query}

@tab

4

@tab

Query component

@tab

empty string

@item

@code{fragment}

@tab

5

@tab

Fragment identifier

@tab

empty string

@item

@code{username}

@tab

@tab

User name

@tab

@ref{182,,None}

@item

@code{password}

@tab

@tab

Password

@tab

@ref{182,,None}

@item

@code{hostname}

@tab

@tab

Host name (lower case)

@tab

@ref{182,,None}

@item

@code{port}

@tab

@tab

Port number as integer,
if present

@tab

@ref{182,,None}

@end multitable


Reading the @code{port} attribute will raise a @ref{17c,,ValueError} if
an invalid port is specified in the URL.  See section
@ref{3148,,Structured Parse Results} for more information on the result object.

Unmatched square brackets in the @code{netloc} attribute will raise a
@ref{17c,,ValueError}.

Characters in the @code{netloc} attribute that decompose under NFKC
normalization (as used by the IDNA encoding) into any of @code{/}, @code{?},
@code{#}, @code{@@}, or @code{:} will raise a @ref{17c,,ValueError}. If the URL is
decomposed before parsing, no error will be raised.

As is the case with all named tuples, the subclass has a few additional methods
and attributes that are particularly useful. One such method is @code{_replace()}.
The @code{_replace()} method will return a new ParseResult object replacing specified
fields with new values.

@example
>>> from urllib.parse import urlparse
>>> u = urlparse('//www.cwi.nl:80/%7Eguido/Python.html')
>>> u
ParseResult(scheme='@w{'}, netloc='www.cwi.nl:80', path='/%7Eguido/Python.html',
            params='@w{'}, query='@w{'}, fragment='@w{'})
>>> u._replace(scheme='http')
ParseResult(scheme='http', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html',
            params='@w{'}, query='@w{'}, fragment='@w{'})
@end example


Changed in version 3.2: Added IPv6 URL parsing capabilities.



Changed in version 3.3: The fragment is now parsed for all URL schemes (unless @emph{allow_fragment} is
false), in accordance with 
@geindex RFC; RFC 3986
RFC 3986@footnote{https://tools.ietf.org/html/rfc3986.html}.  Previously, an allowlist of
schemes that support fragments existed.



Changed in version 3.6: Out-of-range port numbers now raise @ref{17c,,ValueError}, instead of
returning @ref{182,,None}.



Changed in version 3.8: Characters that affect netloc parsing under NFKC normalization will
now raise @ref{17c,,ValueError}.

@end deffn

@geindex parse_qs() (in module urllib.parse)
@anchor{library/urllib parse urllib parse parse_qs}@anchor{2e3}
@deffn {Function} urllib.parse.parse_qs (qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace', max_num_fields=None, separator='&')

Parse a query string given as a string argument (data of type
@code{application/x-www-form-urlencoded}).  Data are returned as a
dictionary.  The dictionary keys are the unique query variable names and the
values are lists of values for each name.

The optional argument @emph{keep_blank_values} is a flag indicating whether blank
values in percent-encoded queries should be treated as blank strings. A true value
indicates that blanks should be retained as  blank strings.  The default false
value indicates that blank values are to be ignored and treated as if they were
not included.

The optional argument @emph{strict_parsing} is a flag indicating what to do with
parsing errors.  If false (the default), errors are silently ignored.  If true,
errors raise a @ref{17c,,ValueError} exception.

The optional @emph{encoding} and @emph{errors} parameters specify how to decode
percent-encoded sequences into Unicode characters, as accepted by the
@ref{3bb,,bytes.decode()} method.

The optional argument @emph{max_num_fields} is the maximum number of fields to
read. If set, then throws a @ref{17c,,ValueError} if there are more than
@emph{max_num_fields} fields read.

The optional argument @emph{separator} is the symbol to use for separating the
query arguments. It defaults to @code{&}.

Use the @ref{a05,,urllib.parse.urlencode()} function (with the @code{doseq}
parameter set to @code{True}) to convert such dictionaries into query
strings.


Changed in version 3.2: Add @emph{encoding} and @emph{errors} parameters.



Changed in version 3.8: Added @emph{max_num_fields} parameter.



Changed in version 3.10: Added @emph{separator} parameter with the default value of @code{&}. Python
versions earlier than Python 3.10 allowed using both @code{;} and @code{&} as
query parameter separator. This has been changed to allow only a single
separator key, with @code{&} as the default separator.

@end deffn

@geindex parse_qsl() (in module urllib.parse)
@anchor{library/urllib parse urllib parse parse_qsl}@anchor{2e4}
@deffn {Function} urllib.parse.parse_qsl (qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace', max_num_fields=None, separator='&')

Parse a query string given as a string argument (data of type
@code{application/x-www-form-urlencoded}).  Data are returned as a list of
name, value pairs.

The optional argument @emph{keep_blank_values} is a flag indicating whether blank
values in percent-encoded queries should be treated as blank strings. A true value
indicates that blanks should be retained as  blank strings.  The default false
value indicates that blank values are to be ignored and treated as if they were
not included.

The optional argument @emph{strict_parsing} is a flag indicating what to do with
parsing errors.  If false (the default), errors are silently ignored.  If true,
errors raise a @ref{17c,,ValueError} exception.

The optional @emph{encoding} and @emph{errors} parameters specify how to decode
percent-encoded sequences into Unicode characters, as accepted by the
@ref{3bb,,bytes.decode()} method.

The optional argument @emph{max_num_fields} is the maximum number of fields to
read. If set, then throws a @ref{17c,,ValueError} if there are more than
@emph{max_num_fields} fields read.

The optional argument @emph{separator} is the symbol to use for separating the
query arguments. It defaults to @code{&}.

Use the @ref{a05,,urllib.parse.urlencode()} function to convert such lists of pairs into
query strings.


Changed in version 3.2: Add @emph{encoding} and @emph{errors} parameters.



Changed in version 3.8: Added @emph{max_num_fields} parameter.



Changed in version 3.10: Added @emph{separator} parameter with the default value of @code{&}. Python
versions earlier than Python 3.10 allowed using both @code{;} and @code{&} as
query parameter separator. This has been changed to allow only a single
separator key, with @code{&} as the default separator.

@end deffn

@geindex urlunparse() (in module urllib.parse)
@anchor{library/urllib parse urllib parse urlunparse}@anchor{3149}
@deffn {Function} urllib.parse.urlunparse (parts)

Construct a URL from a tuple as returned by @code{urlparse()}. The @emph{parts}
argument can be any six-item iterable. This may result in a slightly
different, but equivalent URL, if the URL that was parsed originally had
unnecessary delimiters (for example, a @code{?} with an empty query; the RFC
states that these are equivalent).
@end deffn

@geindex urlsplit() (in module urllib.parse)
@anchor{library/urllib parse urllib parse urlsplit}@anchor{88f}
@deffn {Function} urllib.parse.urlsplit (urlstring, scheme='@w{'}, allow_fragments=True)

This is similar to @ref{890,,urlparse()}, but does not split the params from the URL.
This should generally be used instead of @ref{890,,urlparse()} if the more recent URL
syntax allowing parameters to be applied to each segment of the @emph{path} portion
of the URL (see 
@geindex RFC; RFC 2396
RFC 2396@footnote{https://tools.ietf.org/html/rfc2396.html}) is wanted.  A separate function is needed to
separate the path segments and parameters.  This function returns a 5-item
@ref{ce9,,named tuple}:

@example
(addressing scheme, network location, path, query, fragment identifier).
@end example

The return value is a @ref{ce9,,named tuple}, its items can be accessed by index
or as named attributes:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Attribute

@tab

Index

@tab

Value

@tab

Value if not present

@item

@code{scheme}

@tab

0

@tab

URL scheme specifier

@tab

@emph{scheme} parameter

@item

@code{netloc}

@tab

1

@tab

Network location part

@tab

empty string

@item

@code{path}

@tab

2

@tab

Hierarchical path

@tab

empty string

@item

@code{query}

@tab

3

@tab

Query component

@tab

empty string

@item

@code{fragment}

@tab

4

@tab

Fragment identifier

@tab

empty string

@item

@code{username}

@tab

@tab

User name

@tab

@ref{182,,None}

@item

@code{password}

@tab

@tab

Password

@tab

@ref{182,,None}

@item

@code{hostname}

@tab

@tab

Host name (lower case)

@tab

@ref{182,,None}

@item

@code{port}

@tab

@tab

Port number as integer,
if present

@tab

@ref{182,,None}

@end multitable


Reading the @code{port} attribute will raise a @ref{17c,,ValueError} if
an invalid port is specified in the URL.  See section
@ref{3148,,Structured Parse Results} for more information on the result object.

Unmatched square brackets in the @code{netloc} attribute will raise a
@ref{17c,,ValueError}.

Characters in the @code{netloc} attribute that decompose under NFKC
normalization (as used by the IDNA encoding) into any of @code{/}, @code{?},
@code{#}, @code{@@}, or @code{:} will raise a @ref{17c,,ValueError}. If the URL is
decomposed before parsing, no error will be raised.

Following the WHATWG spec@footnote{https://url.spec.whatwg.org/#concept-basic-url-parser} that updates RFC 3986, ASCII newline
@code{\n}, @code{\r} and tab @code{\t} characters are stripped from the URL.


Changed in version 3.6: Out-of-range port numbers now raise @ref{17c,,ValueError}, instead of
returning @ref{182,,None}.



Changed in version 3.8: Characters that affect netloc parsing under NFKC normalization will
now raise @ref{17c,,ValueError}.



Changed in version 3.10: ASCII newline and tab characters are stripped from the URL.

@end deffn

@geindex urlunsplit() (in module urllib.parse)
@anchor{library/urllib parse urllib parse urlunsplit}@anchor{314a}
@deffn {Function} urllib.parse.urlunsplit (parts)

Combine the elements of a tuple as returned by @ref{88f,,urlsplit()} into a
complete URL as a string. The @emph{parts} argument can be any five-item
iterable. This may result in a slightly different, but equivalent URL, if the
URL that was parsed originally had unnecessary delimiters (for example, a ?
with an empty query; the RFC states that these are equivalent).
@end deffn

@geindex urljoin() (in module urllib.parse)
@anchor{library/urllib parse urllib parse urljoin}@anchor{a07}
@deffn {Function} urllib.parse.urljoin (base, url, allow_fragments=True)

Construct a full (“absolute”) URL by combining a “base URL” (@emph{base}) with
another URL (@emph{url}).  Informally, this uses components of the base URL, in
particular the addressing scheme, the network location and (part of) the
path, to provide missing components in the relative URL.  For example:

@example
>>> from urllib.parse import urljoin
>>> urljoin('http://www.cwi.nl/%7Eguido/Python.html', 'FAQ.html')
'http://www.cwi.nl/%7Eguido/FAQ.html'
@end example

The @emph{allow_fragments} argument has the same meaning and default as for
@ref{890,,urlparse()}.

@cartouche
@quotation Note 
If @emph{url} is an absolute URL (that is, it starts with @code{//} or @code{scheme://}),
the @emph{url}’s hostname and/or scheme will be present in the result.  For example:

@example
>>> urljoin('http://www.cwi.nl/%7Eguido/Python.html',
...         '//www.python.org/%7Eguido')
'http://www.python.org/%7Eguido'
@end example

If you do not want that behavior, preprocess the @emph{url} with @ref{88f,,urlsplit()} and
@ref{314a,,urlunsplit()}, removing possible @emph{scheme} and @emph{netloc} parts.
@end quotation
@end cartouche


Changed in version 3.5: Behavior updated to match the semantics defined in 
@geindex RFC; RFC 3986
RFC 3986@footnote{https://tools.ietf.org/html/rfc3986.html}.

@end deffn

@geindex urldefrag() (in module urllib.parse)
@anchor{library/urllib parse urllib parse urldefrag}@anchor{dfb}
@deffn {Function} urllib.parse.urldefrag (url)

If @emph{url} contains a fragment identifier, return a modified version of @emph{url}
with no fragment identifier, and the fragment identifier as a separate
string.  If there is no fragment identifier in @emph{url}, return @emph{url} unmodified
and an empty string.

The return value is a @ref{ce9,,named tuple}, its items can be accessed by index
or as named attributes:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Attribute

@tab

Index

@tab

Value

@tab

Value if not present

@item

@code{url}

@tab

0

@tab

URL with no fragment

@tab

empty string

@item

@code{fragment}

@tab

1

@tab

Fragment identifier

@tab

empty string

@end multitable


See section @ref{3148,,Structured Parse Results} for more information on the result
object.


Changed in version 3.2: Result is a structured object rather than a simple 2-tuple.

@end deffn

@geindex unwrap() (in module urllib.parse)
@anchor{library/urllib parse urllib parse unwrap}@anchor{314b}
@deffn {Function} urllib.parse.unwrap (url)

Extract the url from a wrapped URL (that is, a string formatted as
@code{<URL:scheme://host/path>}, @code{<scheme://host/path>}, @code{URL:scheme://host/path}
or @code{scheme://host/path}). If @emph{url} is not a wrapped URL, it is returned
without changes.
@end deffn

@node Parsing ASCII Encoded Bytes,Structured Parse Results,URL Parsing,urllib parse — Parse URLs into components
@anchor{library/urllib parse id1}@anchor{314c}@anchor{library/urllib parse parsing-ascii-encoded-bytes}@anchor{dfd}
@subsubsection Parsing ASCII Encoded Bytes


The URL parsing functions were originally designed to operate on character
strings only. In practice, it is useful to be able to manipulate properly
quoted and encoded URLs as sequences of ASCII bytes. Accordingly, the
URL parsing functions in this module all operate on @ref{2f0,,bytes} and
@ref{2f1,,bytearray} objects in addition to @ref{2ef,,str} objects.

If @ref{2ef,,str} data is passed in, the result will also contain only
@ref{2ef,,str} data. If @ref{2f0,,bytes} or @ref{2f1,,bytearray} data is
passed in, the result will contain only @ref{2f0,,bytes} data.

Attempting to mix @ref{2ef,,str} data with @ref{2f0,,bytes} or
@ref{2f1,,bytearray} in a single function call will result in a
@ref{162,,TypeError} being raised, while attempting to pass in non-ASCII
byte values will trigger @ref{51f,,UnicodeDecodeError}.

To support easier conversion of result objects between @ref{2ef,,str} and
@ref{2f0,,bytes}, all return values from URL parsing functions provide
either an @code{encode()} method (when the result contains @ref{2ef,,str}
data) or a @code{decode()} method (when the result contains @ref{2f0,,bytes}
data). The signatures of these methods match those of the corresponding
@ref{2ef,,str} and @ref{2f0,,bytes} methods (except that the default encoding
is @code{'ascii'} rather than @code{'utf-8'}). Each produces a value of a
corresponding type that contains either @ref{2f0,,bytes} data (for
@code{encode()} methods) or @ref{2ef,,str} data (for
@code{decode()} methods).

Applications that need to operate on potentially improperly quoted URLs
that may contain non-ASCII data will need to do their own decoding from
bytes to characters before invoking the URL parsing methods.

The behaviour described in this section applies only to the URL parsing
functions. The URL quoting functions use their own rules when producing
or consuming byte sequences as detailed in the documentation of the
individual URL quoting functions.


Changed in version 3.2: URL parsing functions now accept ASCII encoded byte sequences


@node Structured Parse Results,URL Quoting,Parsing ASCII Encoded Bytes,urllib parse — Parse URLs into components
@anchor{library/urllib parse structured-parse-results}@anchor{314d}@anchor{library/urllib parse urlparse-result-object}@anchor{3148}
@subsubsection Structured Parse Results


The result objects from the @ref{890,,urlparse()}, @ref{88f,,urlsplit()}  and
@ref{dfb,,urldefrag()} functions are subclasses of the @ref{3e5,,tuple} type.
These subclasses add the attributes listed in the documentation for
those functions, the encoding and decoding support described in the
previous section, as well as an additional method:

@geindex geturl() (urllib.parse.urllib.parse.SplitResult method)
@anchor{library/urllib parse urllib parse urllib parse SplitResult geturl}@anchor{314e}
@deffn {Method} urllib.parse.SplitResult.geturl ()

Return the re-combined version of the original URL as a string. This may
differ from the original URL in that the scheme may be normalized to lower
case and empty components may be dropped. Specifically, empty parameters,
queries, and fragment identifiers will be removed.

For @ref{dfb,,urldefrag()} results, only empty fragment identifiers will be removed.
For @ref{88f,,urlsplit()} and @ref{890,,urlparse()} results, all noted changes will be
made to the URL returned by this method.

The result of this method remains unchanged if passed back through the original
parsing function:

@example
>>> from urllib.parse import urlsplit
>>> url = 'HTTP://www.Python.org/doc/#'
>>> r1 = urlsplit(url)
>>> r1.geturl()
'http://www.Python.org/doc/'
>>> r2 = urlsplit(r1.geturl())
>>> r2.geturl()
'http://www.Python.org/doc/'
@end example
@end deffn

The following classes provide the implementations of the structured parse
results when operating on @ref{2ef,,str} objects:

@geindex DefragResult (class in urllib.parse)
@anchor{library/urllib parse urllib parse DefragResult}@anchor{314f}
@deffn {Class} urllib.parse.DefragResult (url, fragment)

Concrete class for @ref{dfb,,urldefrag()} results containing @ref{2ef,,str}
data. The @code{encode()} method returns a @ref{3150,,DefragResultBytes}
instance.


New in version 3.2.

@end deffn

@geindex ParseResult (class in urllib.parse)
@anchor{library/urllib parse urllib parse ParseResult}@anchor{3151}
@deffn {Class} urllib.parse.ParseResult (scheme, netloc, path, params, query, fragment)

Concrete class for @ref{890,,urlparse()} results containing @ref{2ef,,str}
data. The @code{encode()} method returns a @ref{3152,,ParseResultBytes}
instance.
@end deffn

@geindex SplitResult (class in urllib.parse)
@anchor{library/urllib parse urllib parse SplitResult}@anchor{3153}
@deffn {Class} urllib.parse.SplitResult (scheme, netloc, path, query, fragment)

Concrete class for @ref{88f,,urlsplit()} results containing @ref{2ef,,str}
data. The @code{encode()} method returns a @ref{3154,,SplitResultBytes}
instance.
@end deffn

The following classes provide the implementations of the parse results when
operating on @ref{2f0,,bytes} or @ref{2f1,,bytearray} objects:

@geindex DefragResultBytes (class in urllib.parse)
@anchor{library/urllib parse urllib parse DefragResultBytes}@anchor{3150}
@deffn {Class} urllib.parse.DefragResultBytes (url, fragment)

Concrete class for @ref{dfb,,urldefrag()} results containing @ref{2f0,,bytes}
data. The @code{decode()} method returns a @ref{314f,,DefragResult}
instance.


New in version 3.2.

@end deffn

@geindex ParseResultBytes (class in urllib.parse)
@anchor{library/urllib parse urllib parse ParseResultBytes}@anchor{3152}
@deffn {Class} urllib.parse.ParseResultBytes (scheme, netloc, path, params, query, fragment)

Concrete class for @ref{890,,urlparse()} results containing @ref{2f0,,bytes}
data. The @code{decode()} method returns a @ref{3151,,ParseResult}
instance.


New in version 3.2.

@end deffn

@geindex SplitResultBytes (class in urllib.parse)
@anchor{library/urllib parse urllib parse SplitResultBytes}@anchor{3154}
@deffn {Class} urllib.parse.SplitResultBytes (scheme, netloc, path, query, fragment)

Concrete class for @ref{88f,,urlsplit()} results containing @ref{2f0,,bytes}
data. The @code{decode()} method returns a @ref{3153,,SplitResult}
instance.


New in version 3.2.

@end deffn

@node URL Quoting,,Structured Parse Results,urllib parse — Parse URLs into components
@anchor{library/urllib parse url-quoting}@anchor{3155}
@subsubsection URL Quoting


The URL quoting functions focus on taking program data and making it safe
for use as URL components by quoting special characters and appropriately
encoding non-ASCII text. They also support reversing these operations to
recreate the original data from the contents of a URL component if that
task isn’t already covered by the URL parsing functions above.

@geindex quote() (in module urllib.parse)
@anchor{library/urllib parse urllib parse quote}@anchor{6d4}
@deffn {Function} urllib.parse.quote (string, safe='/', encoding=None, errors=None)

Replace special characters in @emph{string} using the @code{%xx} escape. Letters,
digits, and the characters @code{'_.-~'} are never quoted. By default, this
function is intended for quoting the path section of a URL. The optional
@emph{safe} parameter specifies additional ASCII characters that should not be
quoted — its default value is @code{'/'}.

@emph{string} may be either a @ref{2ef,,str} or a @ref{2f0,,bytes} object.


Changed in version 3.7: Moved from 
@geindex RFC; RFC 2396
RFC 2396@footnote{https://tools.ietf.org/html/rfc2396.html} to 
@geindex RFC; RFC 3986
RFC 3986@footnote{https://tools.ietf.org/html/rfc3986.html} for quoting URL strings. “~” is now
included in the set of unreserved characters.


The optional @emph{encoding} and @emph{errors} parameters specify how to deal with
non-ASCII characters, as accepted by the @ref{3ba,,str.encode()} method.
@emph{encoding} defaults to @code{'utf-8'}.
@emph{errors} defaults to @code{'strict'}, meaning unsupported characters raise a
@ref{185,,UnicodeEncodeError}.
@emph{encoding} and @emph{errors} must not be supplied if @emph{string} is a
@ref{2f0,,bytes}, or a @ref{162,,TypeError} is raised.

Note that @code{quote(string, safe, encoding, errors)} is equivalent to
@code{quote_from_bytes(string.encode(encoding, errors), safe)}.

Example: @code{quote('/El Niño/')} yields @code{'/El%20Ni%C3%B1o/'}.
@end deffn

@geindex quote_plus() (in module urllib.parse)
@anchor{library/urllib parse urllib parse quote_plus}@anchor{dfc}
@deffn {Function} urllib.parse.quote_plus (string, safe='@w{'}, encoding=None, errors=None)

Like @ref{6d4,,quote()}, but also replace spaces with plus signs, as required for
quoting HTML form values when building up a query string to go into a URL.
Plus signs in the original string are escaped unless they are included in
@emph{safe}.  It also does not have @emph{safe} default to @code{'/'}.

Example: @code{quote_plus('/El Niño/')} yields @code{'%2FEl+Ni%C3%B1o%2F'}.
@end deffn

@geindex quote_from_bytes() (in module urllib.parse)
@anchor{library/urllib parse urllib parse quote_from_bytes}@anchor{3156}
@deffn {Function} urllib.parse.quote_from_bytes (bytes, safe='/')

Like @ref{6d4,,quote()}, but accepts a @ref{2f0,,bytes} object rather than a
@ref{2ef,,str}, and does not perform string-to-bytes encoding.

Example: @code{quote_from_bytes(b'a&\xef')} yields
@code{'a%26%EF'}.
@end deffn

@geindex unquote() (in module urllib.parse)
@anchor{library/urllib parse urllib parse unquote}@anchor{1273}
@deffn {Function} urllib.parse.unquote (string, encoding='utf-8', errors='replace')

Replace @code{%xx} escapes with their single-character equivalent.
The optional @emph{encoding} and @emph{errors} parameters specify how to decode
percent-encoded sequences into Unicode characters, as accepted by the
@ref{3bb,,bytes.decode()} method.

@emph{string} may be either a @ref{2ef,,str} or a @ref{2f0,,bytes} object.

@emph{encoding} defaults to @code{'utf-8'}.
@emph{errors} defaults to @code{'replace'}, meaning invalid sequences are replaced
by a placeholder character.

Example: @code{unquote('/El%20Ni%C3%B1o/')} yields @code{'/El Niño/'}.


Changed in version 3.9: @emph{string} parameter supports bytes and str objects (previously only str).

@end deffn

@geindex unquote_plus() (in module urllib.parse)
@anchor{library/urllib parse urllib parse unquote_plus}@anchor{3157}
@deffn {Function} urllib.parse.unquote_plus (string, encoding='utf-8', errors='replace')

Like @ref{1273,,unquote()}, but also replace plus signs with spaces, as required
for unquoting HTML form values.

@emph{string} must be a @ref{2ef,,str}.

Example: @code{unquote_plus('/El+Ni%C3%B1o/')} yields @code{'/El Niño/'}.
@end deffn

@geindex unquote_to_bytes() (in module urllib.parse)
@anchor{library/urllib parse urllib parse unquote_to_bytes}@anchor{3158}
@deffn {Function} urllib.parse.unquote_to_bytes (string)

Replace @code{%xx} escapes with their single-octet equivalent, and return a
@ref{2f0,,bytes} object.

@emph{string} may be either a @ref{2ef,,str} or a @ref{2f0,,bytes} object.

If it is a @ref{2ef,,str}, unescaped non-ASCII characters in @emph{string}
are encoded into UTF-8 bytes.

Example: @code{unquote_to_bytes('a%26%EF')} yields @code{b'a&\xef'}.
@end deffn

@geindex urlencode() (in module urllib.parse)
@anchor{library/urllib parse urllib parse urlencode}@anchor{a05}
@deffn {Function} urllib.parse.urlencode (query, doseq=False, safe='@w{'}, encoding=None, errors=None, quote_via=quote_plus)

Convert a mapping object or a sequence of two-element tuples, which may
contain @ref{2ef,,str} or @ref{2f0,,bytes} objects, to a percent-encoded ASCII
text string.  If the resultant string is to be used as a @emph{data} for POST
operation with the @ref{a06,,urlopen()} function, then
it should be encoded to bytes, otherwise it would result in a
@ref{162,,TypeError}.

The resulting string is a series of @code{key=value} pairs separated by @code{'&'}
characters, where both @emph{key} and @emph{value} are quoted using the @emph{quote_via}
function.  By default, @ref{dfc,,quote_plus()} is used to quote the values, which
means spaces are quoted as a @code{'+'} character and ‘/’ characters are
encoded as @code{%2F}, which follows the standard for GET requests
(@code{application/x-www-form-urlencoded}).  An alternate function that can be
passed as @emph{quote_via} is @ref{6d4,,quote()}, which will encode spaces as @code{%20}
and not encode ‘/’ characters.  For maximum control of what is quoted, use
@code{quote} and specify a value for @emph{safe}.

When a sequence of two-element tuples is used as the @emph{query}
argument, the first element of each tuple is a key and the second is a
value. The value element in itself can be a sequence and in that case, if
the optional parameter @emph{doseq} evaluates to @code{True}, individual
@code{key=value} pairs separated by @code{'&'} are generated for each element of
the value sequence for the key.  The order of parameters in the encoded
string will match the order of parameter tuples in the sequence.

The @emph{safe}, @emph{encoding}, and @emph{errors} parameters are passed down to
@emph{quote_via} (the @emph{encoding} and @emph{errors} parameters are only passed
when a query element is a @ref{2ef,,str}).

To reverse this encoding process, @ref{2e3,,parse_qs()} and @ref{2e4,,parse_qsl()} are
provided in this module to parse query strings into Python data structures.

Refer to @ref{3135,,urllib examples} to find out how the
@ref{a05,,urllib.parse.urlencode()} method can be used for generating the query
string of a URL or data for a POST request.


Changed in version 3.2: @emph{query} supports bytes and string objects.



New in version 3.5: @emph{quote_via} parameter.

@end deffn


@subsubheading See also



@table @asis

@item WHATWG@footnote{https://url.spec.whatwg.org/} -  URL Living standard

Working Group for the URL Standard that defines URLs, domains, IP addresses, the
application/x-www-form-urlencoded format, and their API.

@geindex RFC; RFC 3986

@item RFC 3986@footnote{https://tools.ietf.org/html/rfc3986.html} - Uniform Resource Identifiers

This is the current standard (STD66). Any changes to urllib.parse module
should conform to this. Certain deviations could be observed, which are
mostly for backward compatibility purposes and for certain de-facto
parsing requirements as commonly observed in major browsers.

@geindex RFC; RFC 2732

@item RFC 2732@footnote{https://tools.ietf.org/html/rfc2732.html} - Format for Literal IPv6 Addresses in URL’s.

This specifies the parsing requirements of IPv6 URLs.

@geindex RFC; RFC 2396

@item RFC 2396@footnote{https://tools.ietf.org/html/rfc2396.html} - Uniform Resource Identifiers (URI): Generic Syntax

Document describing the generic syntactic requirements for both Uniform Resource
Names (URNs) and Uniform Resource Locators (URLs).

@geindex RFC; RFC 2368

@item RFC 2368@footnote{https://tools.ietf.org/html/rfc2368.html} - The mailto URL scheme.

Parsing requirements for mailto URL schemes.

@geindex RFC; RFC 1808

@item RFC 1808@footnote{https://tools.ietf.org/html/rfc1808.html} - Relative Uniform Resource Locators

This Request For Comments includes the rules for joining an absolute and a
relative URL, including a fair number of “Abnormal Examples” which govern the
treatment of border cases.

@geindex RFC; RFC 1738

@item RFC 1738@footnote{https://tools.ietf.org/html/rfc1738.html} - Uniform Resource Locators (URL)

This specifies the formal syntax and semantics of absolute URLs.
@end table


@node urllib error — Exception classes raised by urllib request,urllib robotparser — Parser for robots txt,urllib parse — Parse URLs into components,Internet Protocols and Support
@anchor{library/urllib error doc}@anchor{3159}@anchor{library/urllib error module-urllib error}@anchor{125}@anchor{library/urllib error urllib-error-exception-classes-raised-by-urllib-request}@anchor{315a}@anchor{library/urllib error whatwg}@anchor{315b}
@subsection @code{urllib.error} — Exception classes raised by urllib.request


@geindex module; urllib.error

@strong{Source code:} Lib/urllib/error.py@footnote{https://github.com/python/cpython/tree/main/Lib/urllib/error.py}


__________________________________________________________________


The @ref{125,,urllib.error} module defines the exception classes for exceptions
raised by @ref{127,,urllib.request}.  The base exception class is @ref{30bb,,URLError}.

The following exceptions are raised by @ref{125,,urllib.error} as appropriate:

@geindex URLError
@anchor{library/urllib error urllib error URLError}@anchor{30bb}
@deffn {Exception} urllib.error.URLError

The handlers raise this exception (or derived exceptions) when they run into
a problem.  It is a subclass of @ref{2a1,,OSError}.

@geindex reason (urllib.error.URLError attribute)
@anchor{library/urllib error urllib error URLError reason}@anchor{315c}
@deffn {Attribute} reason

The reason for this error.  It can be a message string or another
exception instance.
@end deffn


Changed in version 3.3: @ref{30bb,,URLError} has been made a subclass of @ref{2a1,,OSError} instead
of @ref{be8,,IOError}.

@end deffn

@geindex HTTPError
@anchor{library/urllib error urllib error HTTPError}@anchor{b64}
@deffn {Exception} urllib.error.HTTPError

Though being an exception (a subclass of @ref{30bb,,URLError}), an
@ref{b64,,HTTPError} can also function as a non-exceptional file-like return
value (the same thing that @ref{a06,,urlopen()} returns).  This
is useful when handling exotic HTTP errors, such as requests for
authentication.

@geindex code (urllib.error.HTTPError attribute)
@anchor{library/urllib error urllib error HTTPError code}@anchor{315d}
@deffn {Attribute} code

An HTTP status code as defined in 
@geindex RFC; RFC 2616
RFC 2616@footnote{https://tools.ietf.org/html/rfc2616.html}.  This numeric value corresponds
to a value found in the dictionary of codes as found in
@ref{315e,,http.server.BaseHTTPRequestHandler.responses}.
@end deffn

@geindex reason (urllib.error.HTTPError attribute)
@anchor{library/urllib error urllib error HTTPError reason}@anchor{315f}
@deffn {Attribute} reason

This is usually a string explaining the reason for this error.
@end deffn

@geindex headers (urllib.error.HTTPError attribute)
@anchor{library/urllib error urllib error HTTPError headers}@anchor{b65}
@deffn {Attribute} headers

The HTTP response headers for the HTTP request that caused the
@ref{b64,,HTTPError}.


New in version 3.4.

@end deffn
@end deffn

@geindex ContentTooShortError
@anchor{library/urllib error urllib error ContentTooShortError}@anchor{3160}
@deffn {Exception} urllib.error.ContentTooShortError (msg, content)

This exception is raised when the @ref{3137,,urlretrieve()}
function detects that
the amount of the downloaded data is less than the expected amount (given by
the @emph{Content-Length} header).  The @code{content} attribute stores the
downloaded (and supposedly truncated) data.
@end deffn

@node urllib robotparser — Parser for robots txt,http — HTTP modules,urllib error — Exception classes raised by urllib request,Internet Protocols and Support
@anchor{library/urllib robotparser doc}@anchor{3161}@anchor{library/urllib robotparser module-urllib robotparser}@anchor{129}@anchor{library/urllib robotparser urllib-robotparser-parser-for-robots-txt}@anchor{3162}
@subsection @code{urllib.robotparser} —  Parser for robots.txt


@geindex module; urllib.robotparser

@strong{Source code:} Lib/urllib/robotparser.py@footnote{https://github.com/python/cpython/tree/main/Lib/urllib/robotparser.py}

@geindex WWW
@geindex World Wide Web
@geindex URL
@geindex robots.txt


__________________________________________________________________


This module provides a single class, @ref{848,,RobotFileParser}, which answers
questions about whether or not a particular user agent can fetch a URL on the
web site that published the @code{robots.txt} file.  For more details on the
structure of @code{robots.txt} files, see @indicateurl{http://www.robotstxt.org/orig.html}.

@geindex RobotFileParser (class in urllib.robotparser)
@anchor{library/urllib robotparser urllib robotparser RobotFileParser}@anchor{848}
@deffn {Class} urllib.robotparser.RobotFileParser (url='@w{'})

This class provides methods to read, parse and answer questions about the
@code{robots.txt} file at @emph{url}.

@geindex set_url() (urllib.robotparser.RobotFileParser method)
@anchor{library/urllib robotparser urllib robotparser RobotFileParser set_url}@anchor{3163}
@deffn {Method} set_url (url)

Sets the URL referring to a @code{robots.txt} file.
@end deffn

@geindex read() (urllib.robotparser.RobotFileParser method)
@anchor{library/urllib robotparser urllib robotparser RobotFileParser read}@anchor{3164}
@deffn {Method} read ()

Reads the @code{robots.txt} URL and feeds it to the parser.
@end deffn

@geindex parse() (urllib.robotparser.RobotFileParser method)
@anchor{library/urllib robotparser urllib robotparser RobotFileParser parse}@anchor{1363}
@deffn {Method} parse (lines)

Parses the lines argument.
@end deffn

@geindex can_fetch() (urllib.robotparser.RobotFileParser method)
@anchor{library/urllib robotparser urllib robotparser RobotFileParser can_fetch}@anchor{3165}
@deffn {Method} can_fetch (useragent, url)

Returns @code{True} if the @emph{useragent} is allowed to fetch the @emph{url}
according to the rules contained in the parsed @code{robots.txt}
file.
@end deffn

@geindex mtime() (urllib.robotparser.RobotFileParser method)
@anchor{library/urllib robotparser urllib robotparser RobotFileParser mtime}@anchor{3166}
@deffn {Method} mtime ()

Returns the time the @code{robots.txt} file was last fetched.  This is
useful for long-running web spiders that need to check for new
@code{robots.txt} files periodically.
@end deffn

@geindex modified() (urllib.robotparser.RobotFileParser method)
@anchor{library/urllib robotparser urllib robotparser RobotFileParser modified}@anchor{3167}
@deffn {Method} modified ()

Sets the time the @code{robots.txt} file was last fetched to the current
time.
@end deffn

@geindex crawl_delay() (urllib.robotparser.RobotFileParser method)
@anchor{library/urllib robotparser urllib robotparser RobotFileParser crawl_delay}@anchor{3168}
@deffn {Method} crawl_delay (useragent)

Returns the value of the @code{Crawl-delay} parameter from @code{robots.txt}
for the @emph{useragent} in question.  If there is no such parameter or it
doesn’t apply to the @emph{useragent} specified or the @code{robots.txt} entry
for this parameter has invalid syntax, return @code{None}.


New in version 3.6.

@end deffn

@geindex request_rate() (urllib.robotparser.RobotFileParser method)
@anchor{library/urllib robotparser urllib robotparser RobotFileParser request_rate}@anchor{3169}
@deffn {Method} request_rate (useragent)

Returns the contents of the @code{Request-rate} parameter from
@code{robots.txt} as a @ref{ce9,,named tuple} @code{RequestRate(requests, seconds)}.
If there is no such parameter or it doesn’t apply to the @emph{useragent}
specified or the @code{robots.txt} entry for this parameter has invalid
syntax, return @code{None}.


New in version 3.6.

@end deffn

@geindex site_maps() (urllib.robotparser.RobotFileParser method)
@anchor{library/urllib robotparser urllib robotparser RobotFileParser site_maps}@anchor{12fb}
@deffn {Method} site_maps ()

Returns the contents of the @code{Sitemap} parameter from
@code{robots.txt} in the form of a @ref{43a,,list()}. If there is no such
parameter or the @code{robots.txt} entry for this parameter has
invalid syntax, return @code{None}.


New in version 3.8.

@end deffn
@end deffn

The following example demonstrates basic use of the @ref{848,,RobotFileParser}
class:

@example
>>> import urllib.robotparser
>>> rp = urllib.robotparser.RobotFileParser()
>>> rp.set_url("http://www.musi-cal.com/robots.txt")
>>> rp.read()
>>> rrate = rp.request_rate("*")
>>> rrate.requests
3
>>> rrate.seconds
20
>>> rp.crawl_delay("*")
6
>>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco")
False
>>> rp.can_fetch("*", "http://www.musi-cal.com/")
True
@end example

@node http — HTTP modules,http client — HTTP protocol client,urllib robotparser — Parser for robots txt,Internet Protocols and Support
@anchor{library/http doc}@anchor{316a}@anchor{library/http http-http-modules}@anchor{316b}@anchor{library/http module-http}@anchor{8e}
@subsection @code{http} — HTTP modules


@geindex module; http

@strong{Source code:} Lib/http/__init__.py@footnote{https://github.com/python/cpython/tree/main/Lib/http/__init__.py}

@geindex HTTP; protocol
@geindex HTTP; http (standard module)


__________________________________________________________________


@ref{8e,,http} is a package that collects several modules for working with the
HyperText Transfer Protocol:


@itemize *

@item 
@ref{8f,,http.client} is a low-level HTTP protocol client; for high-level URL
opening use @ref{127,,urllib.request}

@item 
@ref{92,,http.server} contains basic HTTP server classes based on @ref{eb,,socketserver}

@item 
@ref{91,,http.cookies} has utilities for implementing state management with cookies

@item 
@ref{90,,http.cookiejar} provides persistence of cookies
@end itemize

@ref{8e,,http} is also a module that defines a number of HTTP status codes and
associated messages through the @ref{3ef,,http.HTTPStatus} enum:

@geindex HTTPStatus (class in http)
@anchor{library/http http HTTPStatus}@anchor{3ef}
@deffn {Class} http.HTTPStatus


New in version 3.5.


A subclass of @ref{828,,enum.IntEnum} that defines a set of HTTP status codes,
reason phrases and long descriptions written in English.

Usage:

@example
>>> from http import HTTPStatus
>>> HTTPStatus.OK
HTTPStatus.OK
>>> HTTPStatus.OK == 200
True
>>> HTTPStatus.OK.value
200
>>> HTTPStatus.OK.phrase
'OK'
>>> HTTPStatus.OK.description
'Request fulfilled, document follows'
>>> list(HTTPStatus)
[HTTPStatus.CONTINUE, HTTPStatus.SWITCHING_PROTOCOLS, ...]
@end example
@end deffn

@menu
* HTTP status codes:: 

@end menu

@node HTTP status codes,,,http — HTTP modules
@anchor{library/http http-status-codes}@anchor{316c}@anchor{library/http id1}@anchor{316d}
@subsubsection HTTP status codes


Supported,
IANA-registered@footnote{https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml}
status codes available in @ref{3ef,,http.HTTPStatus} are:


@multitable {xxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Code

@tab

Enum Name

@tab

Details

@item

@code{100}

@tab

@code{CONTINUE}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.2.1

@item

@code{101}

@tab

@code{SWITCHING_PROTOCOLS}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.2.2

@item

@code{102}

@tab

@code{PROCESSING}

@tab

WebDAV 
@geindex RFC; RFC 2518
RFC 2518@footnote{https://tools.ietf.org/html/rfc2518.html}, Section 10.1

@item

@code{103}

@tab

@code{EARLY_HINTS}

@tab

An HTTP Status Code for Indicating Hints 
@geindex RFC; RFC 8297
RFC 8297@footnote{https://tools.ietf.org/html/rfc8297.html}

@item

@code{200}

@tab

@code{OK}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.3.1

@item

@code{201}

@tab

@code{CREATED}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.3.2

@item

@code{202}

@tab

@code{ACCEPTED}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.3.3

@item

@code{203}

@tab

@code{NON_AUTHORITATIVE_INFORMATION}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.3.4

@item

@code{204}

@tab

@code{NO_CONTENT}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.3.5

@item

@code{205}

@tab

@code{RESET_CONTENT}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.3.6

@item

@code{206}

@tab

@code{PARTIAL_CONTENT}

@tab

HTTP/1.1 
@geindex RFC; RFC 7233
RFC 7233@footnote{https://tools.ietf.org/html/rfc7233.html}, Section 4.1

@item

@code{207}

@tab

@code{MULTI_STATUS}

@tab

WebDAV 
@geindex RFC; RFC 4918
RFC 4918@footnote{https://tools.ietf.org/html/rfc4918.html}, Section 11.1

@item

@code{208}

@tab

@code{ALREADY_REPORTED}

@tab

WebDAV Binding Extensions 
@geindex RFC; RFC 5842
RFC 5842@footnote{https://tools.ietf.org/html/rfc5842.html}, Section 7.1 (Experimental)

@item

@code{226}

@tab

@code{IM_USED}

@tab

Delta Encoding in HTTP 
@geindex RFC; RFC 3229
RFC 3229@footnote{https://tools.ietf.org/html/rfc3229.html}, Section 10.4.1

@item

@code{300}

@tab

@code{MULTIPLE_CHOICES}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.4.1

@item

@code{301}

@tab

@code{MOVED_PERMANENTLY}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.4.2

@item

@code{302}

@tab

@code{FOUND}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.4.3

@item

@code{303}

@tab

@code{SEE_OTHER}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.4.4

@item

@code{304}

@tab

@code{NOT_MODIFIED}

@tab

HTTP/1.1 
@geindex RFC; RFC 7232
RFC 7232@footnote{https://tools.ietf.org/html/rfc7232.html}, Section 4.1

@item

@code{305}

@tab

@code{USE_PROXY}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.4.5

@item

@code{307}

@tab

@code{TEMPORARY_REDIRECT}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.4.7

@item

@code{308}

@tab

@code{PERMANENT_REDIRECT}

@tab

Permanent Redirect 
@geindex RFC; RFC 7238
RFC 7238@footnote{https://tools.ietf.org/html/rfc7238.html}, Section 3 (Experimental)

@item

@code{400}

@tab

@code{BAD_REQUEST}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.1

@item

@code{401}

@tab

@code{UNAUTHORIZED}

@tab

HTTP/1.1 Authentication 
@geindex RFC; RFC 7235
RFC 7235@footnote{https://tools.ietf.org/html/rfc7235.html}, Section 3.1

@item

@code{402}

@tab

@code{PAYMENT_REQUIRED}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.2

@item

@code{403}

@tab

@code{FORBIDDEN}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.3

@item

@code{404}

@tab

@code{NOT_FOUND}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.4

@item

@code{405}

@tab

@code{METHOD_NOT_ALLOWED}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.5

@item

@code{406}

@tab

@code{NOT_ACCEPTABLE}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.6

@item

@code{407}

@tab

@code{PROXY_AUTHENTICATION_REQUIRED}

@tab

HTTP/1.1 Authentication 
@geindex RFC; RFC 7235
RFC 7235@footnote{https://tools.ietf.org/html/rfc7235.html}, Section 3.2

@item

@code{408}

@tab

@code{REQUEST_TIMEOUT}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.7

@item

@code{409}

@tab

@code{CONFLICT}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.8

@item

@code{410}

@tab

@code{GONE}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.9

@item

@code{411}

@tab

@code{LENGTH_REQUIRED}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.10

@item

@code{412}

@tab

@code{PRECONDITION_FAILED}

@tab

HTTP/1.1 
@geindex RFC; RFC 7232
RFC 7232@footnote{https://tools.ietf.org/html/rfc7232.html}, Section 4.2

@item

@code{413}

@tab

@code{REQUEST_ENTITY_TOO_LARGE}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.11

@item

@code{414}

@tab

@code{REQUEST_URI_TOO_LONG}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.12

@item

@code{415}

@tab

@code{UNSUPPORTED_MEDIA_TYPE}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.13

@item

@code{416}

@tab

@code{REQUESTED_RANGE_NOT_SATISFIABLE}

@tab

HTTP/1.1 Range Requests 
@geindex RFC; RFC 7233
RFC 7233@footnote{https://tools.ietf.org/html/rfc7233.html}, Section 4.4

@item

@code{417}

@tab

@code{EXPECTATION_FAILED}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.14

@item

@code{418}

@tab

@code{IM_A_TEAPOT}

@tab

HTCPCP/1.0 
@geindex RFC; RFC 2324
RFC 2324@footnote{https://tools.ietf.org/html/rfc2324.html}, Section 2.3.2

@item

@code{421}

@tab

@code{MISDIRECTED_REQUEST}

@tab

HTTP/2 
@geindex RFC; RFC 7540
RFC 7540@footnote{https://tools.ietf.org/html/rfc7540.html}, Section 9.1.2

@item

@code{422}

@tab

@code{UNPROCESSABLE_ENTITY}

@tab

WebDAV 
@geindex RFC; RFC 4918
RFC 4918@footnote{https://tools.ietf.org/html/rfc4918.html}, Section 11.2

@item

@code{423}

@tab

@code{LOCKED}

@tab

WebDAV 
@geindex RFC; RFC 4918
RFC 4918@footnote{https://tools.ietf.org/html/rfc4918.html}, Section 11.3

@item

@code{424}

@tab

@code{FAILED_DEPENDENCY}

@tab

WebDAV 
@geindex RFC; RFC 4918
RFC 4918@footnote{https://tools.ietf.org/html/rfc4918.html}, Section 11.4

@item

@code{425}

@tab

@code{TOO_EARLY}

@tab

Using Early Data in HTTP 
@geindex RFC; RFC 8470
RFC 8470@footnote{https://tools.ietf.org/html/rfc8470.html}

@item

@code{426}

@tab

@code{UPGRADE_REQUIRED}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.5.15

@item

@code{428}

@tab

@code{PRECONDITION_REQUIRED}

@tab

Additional HTTP Status Codes 
@geindex RFC; RFC 6585
RFC 6585@footnote{https://tools.ietf.org/html/rfc6585.html}

@item

@code{429}

@tab

@code{TOO_MANY_REQUESTS}

@tab

Additional HTTP Status Codes 
@geindex RFC; RFC 6585
RFC 6585@footnote{https://tools.ietf.org/html/rfc6585.html}

@item

@code{431}

@tab

@code{REQUEST_HEADER_FIELDS_TOO_LARGE}

@tab

Additional HTTP Status Codes 
@geindex RFC; RFC 6585
RFC 6585@footnote{https://tools.ietf.org/html/rfc6585.html}

@item

@code{451}

@tab

@code{UNAVAILABLE_FOR_LEGAL_REASONS}

@tab

An HTTP Status Code to Report Legal Obstacles 
@geindex RFC; RFC 7725
RFC 7725@footnote{https://tools.ietf.org/html/rfc7725.html}

@item

@code{500}

@tab

@code{INTERNAL_SERVER_ERROR}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.6.1

@item

@code{501}

@tab

@code{NOT_IMPLEMENTED}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.6.2

@item

@code{502}

@tab

@code{BAD_GATEWAY}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.6.3

@item

@code{503}

@tab

@code{SERVICE_UNAVAILABLE}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.6.4

@item

@code{504}

@tab

@code{GATEWAY_TIMEOUT}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.6.5

@item

@code{505}

@tab

@code{HTTP_VERSION_NOT_SUPPORTED}

@tab

HTTP/1.1 
@geindex RFC; RFC 7231
RFC 7231@footnote{https://tools.ietf.org/html/rfc7231.html}, Section 6.6.6

@item

@code{506}

@tab

@code{VARIANT_ALSO_NEGOTIATES}

@tab

Transparent Content Negotiation in HTTP 
@geindex RFC; RFC 2295
RFC 2295@footnote{https://tools.ietf.org/html/rfc2295.html}, Section 8.1 (Experimental)

@item

@code{507}

@tab

@code{INSUFFICIENT_STORAGE}

@tab

WebDAV 
@geindex RFC; RFC 4918
RFC 4918@footnote{https://tools.ietf.org/html/rfc4918.html}, Section 11.5

@item

@code{508}

@tab

@code{LOOP_DETECTED}

@tab

WebDAV Binding Extensions 
@geindex RFC; RFC 5842
RFC 5842@footnote{https://tools.ietf.org/html/rfc5842.html}, Section 7.2 (Experimental)

@item

@code{510}

@tab

@code{NOT_EXTENDED}

@tab

An HTTP Extension Framework 
@geindex RFC; RFC 2774
RFC 2774@footnote{https://tools.ietf.org/html/rfc2774.html}, Section 7 (Experimental)

@item

@code{511}

@tab

@code{NETWORK_AUTHENTICATION_REQUIRED}

@tab

Additional HTTP Status Codes 
@geindex RFC; RFC 6585
RFC 6585@footnote{https://tools.ietf.org/html/rfc6585.html}, Section 6

@end multitable


In order to preserve backwards compatibility, enum values are also present
in the @ref{8f,,http.client} module in the form of constants. The enum name is
equal to the constant name (i.e. @code{http.HTTPStatus.OK} is also available as
@code{http.client.OK}).


Changed in version 3.7: Added @code{421 MISDIRECTED_REQUEST} status code.



New in version 3.8: Added @code{451 UNAVAILABLE_FOR_LEGAL_REASONS} status code.



New in version 3.9: Added @code{103 EARLY_HINTS}, @code{418 IM_A_TEAPOT} and @code{425 TOO_EARLY} status codes.


@node http client — HTTP protocol client,ftplib — FTP protocol client,http — HTTP modules,Internet Protocols and Support
@anchor{library/http client doc}@anchor{316e}@anchor{library/http client http-client-http-protocol-client}@anchor{316f}@anchor{library/http client module-http client}@anchor{8f}
@subsection @code{http.client} — HTTP protocol client


@geindex module; http.client

@strong{Source code:} Lib/http/client.py@footnote{https://github.com/python/cpython/tree/main/Lib/http/client.py}

@geindex HTTP; protocol
@geindex HTTP; http.client (standard module)

@geindex module; urllib.request


__________________________________________________________________


This module defines classes which implement the client side of the HTTP and
HTTPS protocols.  It is normally not used directly — the module
@ref{127,,urllib.request} uses it to handle URLs that use HTTP and HTTPS.


@subsubheading See also


The Requests package@footnote{https://requests.readthedocs.io/en/master/}
is recommended for a higher-level HTTP client interface.


@cartouche
@quotation Note 
HTTPS support is only available if Python was compiled with SSL support
(through the @ref{ee,,ssl} module).
@end quotation
@end cartouche

The module provides the following classes:

@geindex HTTPConnection (class in http.client)
@anchor{library/http client http client HTTPConnection}@anchor{663}
@deffn {Class} http.client.HTTPConnection (host, port=None[, timeout], source_address=None, blocksize=8192)

An @ref{663,,HTTPConnection} instance represents one transaction with an HTTP
server.  It should be instantiated passing it a host and optional port
number.  If no port number is passed, the port is extracted from the host
string if it has the form @code{host:port}, else the default HTTP port (80) is
used.  If the optional @emph{timeout} parameter is given, blocking
operations (like connection attempts) will timeout after that many seconds
(if it is not given, the global default timeout setting is used).
The optional @emph{source_address} parameter may be a tuple of a (host, port)
to use as the source address the HTTP connection is made from.
The optional @emph{blocksize} parameter sets the buffer size in bytes for
sending a file-like message body.

For example, the following calls all create instances that connect to the server
at the same host and port:

@example
>>> h1 = http.client.HTTPConnection('www.python.org')
>>> h2 = http.client.HTTPConnection('www.python.org:80')
>>> h3 = http.client.HTTPConnection('www.python.org', 80)
>>> h4 = http.client.HTTPConnection('www.python.org', 80, timeout=10)
@end example


Changed in version 3.2: @emph{source_address} was added.



Changed in version 3.4: The  @emph{strict} parameter was removed. HTTP 0.9-style “Simple Responses” are
not longer supported.



Changed in version 3.7: @emph{blocksize} parameter was added.

@end deffn

@geindex HTTPSConnection (class in http.client)
@anchor{library/http client http client HTTPSConnection}@anchor{664}
@deffn {Class} http.client.HTTPSConnection (host, port=None, key_file=None, cert_file=None[, timeout], source_address=None, *, context=None, check_hostname=None, blocksize=8192)

A subclass of @ref{663,,HTTPConnection} that uses SSL for communication with
secure servers.  Default port is @code{443}.  If @emph{context} is specified, it
must be a @ref{2bf,,ssl.SSLContext} instance describing the various SSL
options.

Please read @ref{2a39,,Security considerations} for more information on best practices.


Changed in version 3.2: @emph{source_address}, @emph{context} and @emph{check_hostname} were added.



Changed in version 3.2: This class now supports HTTPS virtual hosts if possible (that is,
if @ref{2b4f,,ssl.HAS_SNI} is true).



Changed in version 3.4: The @emph{strict} parameter was removed. HTTP 0.9-style “Simple Responses” are
no longer supported.



Changed in version 3.4.3: This class now performs all the necessary certificate and hostname checks
by default. To revert to the previous, unverified, behavior
@code{ssl._create_unverified_context()} can be passed to the @emph{context}
parameter.



Changed in version 3.8: This class now enables TLS 1.3
@ref{542,,ssl.SSLContext.post_handshake_auth} for the default @emph{context} or
when @emph{cert_file} is passed with a custom @emph{context}.



Changed in version 3.10: This class now sends an ALPN extension with protocol indicator
@code{http/1.1} when no @emph{context} is given. Custom @emph{context} should set
ALPN protocols with @code{set_alpn_protocol()}.



Deprecated since version 3.6: @emph{key_file} and @emph{cert_file} are deprecated in favor of @emph{context}.
Please use @ref{cd9,,ssl.SSLContext.load_cert_chain()} instead, or let
@ref{b29,,ssl.create_default_context()} select the system’s trusted CA
certificates for you.

The @emph{check_hostname} parameter is also deprecated; the
@ref{2b42,,ssl.SSLContext.check_hostname} attribute of @emph{context} should
be used instead.

@end deffn

@geindex HTTPResponse (class in http.client)
@anchor{library/http client http client HTTPResponse}@anchor{c60}
@deffn {Class} http.client.HTTPResponse (sock, debuglevel=0, method=None, url=None)

Class whose instances are returned upon successful connection.  Not
instantiated directly by user.


Changed in version 3.4: The @emph{strict} parameter was removed. HTTP 0.9 style “Simple Responses” are
no longer supported.

@end deffn

This module provides the following function:

@geindex parse_headers() (in module http.client)
@anchor{library/http client http client parse_headers}@anchor{1304}
@deffn {Function} http.client.parse_headers (fp)

Parse the headers from a file pointer @emph{fp} representing a HTTP
request/response. The file has to be a @code{BufferedIOBase} reader
(i.e. not text) and must provide a valid 
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html} style header.

This function returns an instance of @code{http.client.HTTPMessage}
that holds the header fields, but no payload
(the same as @ref{3170,,HTTPResponse.msg}
and @ref{3171,,http.server.BaseHTTPRequestHandler.headers}).
After returning, the file pointer @emph{fp} is ready to read the HTTP body.

@cartouche
@quotation Note 
@ref{1304,,parse_headers()} does not parse the start-line of a HTTP message;
it only parses the @code{Name: value} lines. The file has to be ready to
read these field lines, so the first line should already be consumed
before calling the function.
@end quotation
@end cartouche
@end deffn

The following exceptions are raised as appropriate:

@geindex HTTPException
@anchor{library/http client http client HTTPException}@anchor{3172}
@deffn {Exception} http.client.HTTPException

The base class of the other exceptions in this module.  It is a subclass of
@ref{4d8,,Exception}.
@end deffn

@geindex NotConnected
@anchor{library/http client http client NotConnected}@anchor{3173}
@deffn {Exception} http.client.NotConnected

A subclass of @ref{3172,,HTTPException}.
@end deffn

@geindex InvalidURL
@anchor{library/http client http client InvalidURL}@anchor{3174}
@deffn {Exception} http.client.InvalidURL

A subclass of @ref{3172,,HTTPException}, raised if a port is given and is either
non-numeric or empty.
@end deffn

@geindex UnknownProtocol
@anchor{library/http client http client UnknownProtocol}@anchor{3175}
@deffn {Exception} http.client.UnknownProtocol

A subclass of @ref{3172,,HTTPException}.
@end deffn

@geindex UnknownTransferEncoding
@anchor{library/http client http client UnknownTransferEncoding}@anchor{3176}
@deffn {Exception} http.client.UnknownTransferEncoding

A subclass of @ref{3172,,HTTPException}.
@end deffn

@geindex UnimplementedFileMode
@anchor{library/http client http client UnimplementedFileMode}@anchor{3177}
@deffn {Exception} http.client.UnimplementedFileMode

A subclass of @ref{3172,,HTTPException}.
@end deffn

@geindex IncompleteRead
@anchor{library/http client http client IncompleteRead}@anchor{3178}
@deffn {Exception} http.client.IncompleteRead

A subclass of @ref{3172,,HTTPException}.
@end deffn

@geindex ImproperConnectionState
@anchor{library/http client http client ImproperConnectionState}@anchor{3179}
@deffn {Exception} http.client.ImproperConnectionState

A subclass of @ref{3172,,HTTPException}.
@end deffn

@geindex CannotSendRequest
@anchor{library/http client http client CannotSendRequest}@anchor{317a}
@deffn {Exception} http.client.CannotSendRequest

A subclass of @ref{3179,,ImproperConnectionState}.
@end deffn

@geindex CannotSendHeader
@anchor{library/http client http client CannotSendHeader}@anchor{317b}
@deffn {Exception} http.client.CannotSendHeader

A subclass of @ref{3179,,ImproperConnectionState}.
@end deffn

@geindex ResponseNotReady
@anchor{library/http client http client ResponseNotReady}@anchor{317c}
@deffn {Exception} http.client.ResponseNotReady

A subclass of @ref{3179,,ImproperConnectionState}.
@end deffn

@geindex BadStatusLine
@anchor{library/http client http client BadStatusLine}@anchor{317d}
@deffn {Exception} http.client.BadStatusLine

A subclass of @ref{3172,,HTTPException}.  Raised if a server responds with a HTTP
status code that we don’t understand.
@end deffn

@geindex LineTooLong
@anchor{library/http client http client LineTooLong}@anchor{317e}
@deffn {Exception} http.client.LineTooLong

A subclass of @ref{3172,,HTTPException}.  Raised if an excessively long line
is received in the HTTP protocol from the server.
@end deffn

@geindex RemoteDisconnected
@anchor{library/http client http client RemoteDisconnected}@anchor{968}
@deffn {Exception} http.client.RemoteDisconnected

A subclass of @ref{bf5,,ConnectionResetError} and @ref{317d,,BadStatusLine}.  Raised
by @ref{967,,HTTPConnection.getresponse()} when the attempt to read the response
results in no data read from the connection, indicating that the remote end
has closed the connection.


New in version 3.5: Previously, @ref{317d,,BadStatusLine}@code{('@w{'})} was raised.

@end deffn

The constants defined in this module are:

@geindex HTTP_PORT (in module http.client)
@anchor{library/http client http client HTTP_PORT}@anchor{317f}
@deffn {Data} http.client.HTTP_PORT

The default port for the HTTP protocol (always @code{80}).
@end deffn

@geindex HTTPS_PORT (in module http.client)
@anchor{library/http client http client HTTPS_PORT}@anchor{3180}
@deffn {Data} http.client.HTTPS_PORT

The default port for the HTTPS protocol (always @code{443}).
@end deffn

@geindex responses (in module http.client)
@anchor{library/http client http client responses}@anchor{3181}
@deffn {Data} http.client.responses

This dictionary maps the HTTP 1.1 status codes to the W3C names.

Example: @code{http.client.responses[http.client.NOT_FOUND]} is @code{'Not Found'}.
@end deffn

See @ref{316c,,HTTP status codes} for a list of HTTP status codes that are
available in this module as constants.

@menu
* HTTPConnection Objects:: 
* HTTPResponse Objects:: 
* Examples: Examples<25>. 
* HTTPMessage Objects:: 

@end menu

@node HTTPConnection Objects,HTTPResponse Objects,,http client — HTTP protocol client
@anchor{library/http client httpconnection-objects}@anchor{3182}@anchor{library/http client id1}@anchor{3183}
@subsubsection HTTPConnection Objects


@ref{663,,HTTPConnection} instances have the following methods:

@geindex request() (http.client.HTTPConnection method)
@anchor{library/http client http client HTTPConnection request}@anchor{7f1}
@deffn {Method} HTTPConnection.request (method, url, body=None, headers=@{@}, *, encode_chunked=False)

This will send a request to the server using the HTTP request
method @emph{method} and the selector @emph{url}.

If @emph{body} is specified, the specified data is sent after the headers are
finished.  It may be a @ref{2ef,,str}, a @ref{87e,,bytes-like object}, an
open @ref{d68,,file object}, or an iterable of @ref{2f0,,bytes}.  If @emph{body}
is a string, it is encoded as ISO-8859-1, the default for HTTP.  If it
is a bytes-like object, the bytes are sent as is.  If it is a @ref{d68,,file object}, the contents of the file is sent; this file object should
support at least the @code{read()} method.  If the file object is an
instance of @ref{e47,,io.TextIOBase}, the data returned by the @code{read()}
method will be encoded as ISO-8859-1, otherwise the data returned by
@code{read()} is sent as is.  If @emph{body} is an iterable, the elements of the
iterable are sent as is until the iterable is exhausted.

The @emph{headers} argument should be a mapping of extra HTTP headers to send
with the request.

If @emph{headers} contains neither Content-Length nor Transfer-Encoding,
but there is a request body, one of those
header fields will be added automatically.  If
@emph{body} is @code{None}, the Content-Length header is set to @code{0} for
methods that expect a body (@code{PUT}, @code{POST}, and @code{PATCH}).  If
@emph{body} is a string or a bytes-like object that is not also a
@ref{d68,,file}, the Content-Length header is
set to its length.  Any other type of @emph{body} (files
and iterables in general) will be chunk-encoded, and the
Transfer-Encoding header will automatically be set instead of
Content-Length.

The @emph{encode_chunked} argument is only relevant if Transfer-Encoding is
specified in @emph{headers}.  If @emph{encode_chunked} is @code{False}, the
HTTPConnection object assumes that all encoding is handled by the
calling code.  If it is @code{True}, the body will be chunk-encoded.

@cartouche
@quotation Note 
Chunked transfer encoding has been added to the HTTP protocol
version 1.1.  Unless the HTTP server is known to handle HTTP 1.1,
the caller must either specify the Content-Length, or must pass a
@ref{2ef,,str} or bytes-like object that is not also a file as the
body representation.
@end quotation
@end cartouche


New in version 3.2: @emph{body} can now be an iterable.



Changed in version 3.6: If neither Content-Length nor Transfer-Encoding are set in
@emph{headers}, file and iterable @emph{body} objects are now chunk-encoded.
The @emph{encode_chunked} argument was added.
No attempt is made to determine the Content-Length for file
objects.

@end deffn

@geindex getresponse() (http.client.HTTPConnection method)
@anchor{library/http client http client HTTPConnection getresponse}@anchor{967}
@deffn {Method} HTTPConnection.getresponse ()

Should be called after a request is sent to get the response from the server.
Returns an @ref{c60,,HTTPResponse} instance.

@cartouche
@quotation Note 
Note that you must have read the whole response before you can send a new
request to the server.
@end quotation
@end cartouche


Changed in version 3.5: If a @ref{969,,ConnectionError} or subclass is raised, the
@ref{663,,HTTPConnection} object will be ready to reconnect when
a new request is sent.

@end deffn

@geindex set_debuglevel() (http.client.HTTPConnection method)
@anchor{library/http client http client HTTPConnection set_debuglevel}@anchor{3184}
@deffn {Method} HTTPConnection.set_debuglevel (level)

Set the debugging level.  The default debug level is @code{0}, meaning no
debugging output is printed.  Any value greater than @code{0} will cause all
currently defined debug output to be printed to stdout.  The @code{debuglevel}
is passed to any new @ref{c60,,HTTPResponse} objects that are created.


New in version 3.1.

@end deffn

@geindex set_tunnel() (http.client.HTTPConnection method)
@anchor{library/http client http client HTTPConnection set_tunnel}@anchor{dcd}
@deffn {Method} HTTPConnection.set_tunnel (host, port=None, headers=None)

Set the host and the port for HTTP Connect Tunnelling. This allows running
the connection through a proxy server.

The host and port arguments specify the endpoint of the tunneled connection
(i.e. the address included in the CONNECT request, @emph{not} the address of the
proxy server).

The headers argument should be a mapping of extra HTTP headers to send with
the CONNECT request.

For example, to tunnel through a HTTPS proxy server running locally on port
8080, we would pass the address of the proxy to the @ref{664,,HTTPSConnection}
constructor, and the address of the host that we eventually want to reach to
the @ref{dcd,,set_tunnel()} method:

@example
>>> import http.client
>>> conn = http.client.HTTPSConnection("localhost", 8080)
>>> conn.set_tunnel("www.python.org")
>>> conn.request("HEAD","/index.html")
@end example


New in version 3.2.

@end deffn

@geindex connect() (http.client.HTTPConnection method)
@anchor{library/http client http client HTTPConnection connect}@anchor{3185}
@deffn {Method} HTTPConnection.connect ()

Connect to the server specified when the object was created.  By default,
this is called automatically when making a request if the client does not
already have a connection.

Raises an @ref{1130,,auditing event} @code{http.client.connect} with arguments @code{self}, @code{host}, @code{port}.
@end deffn

@geindex close() (http.client.HTTPConnection method)
@anchor{library/http client http client HTTPConnection close}@anchor{3186}
@deffn {Method} HTTPConnection.close ()

Close the connection to the server.
@end deffn

@geindex blocksize (http.client.HTTPConnection attribute)
@anchor{library/http client http client HTTPConnection blocksize}@anchor{3187}
@deffn {Attribute} HTTPConnection.blocksize

Buffer size in bytes for sending a file-like message body.


New in version 3.7.

@end deffn

As an alternative to using the @code{request()} method described above, you can
also send your request step by step, by using the four functions below.

@geindex putrequest() (http.client.HTTPConnection method)
@anchor{library/http client http client HTTPConnection putrequest}@anchor{3188}
@deffn {Method} HTTPConnection.putrequest (method, url, skip_host=False, skip_accept_encoding=False)

This should be the first call after the connection to the server has been
made. It sends a line to the server consisting of the @emph{method} string,
the @emph{url} string, and the HTTP version (@code{HTTP/1.1}).  To disable automatic
sending of @code{Host:} or @code{Accept-Encoding:} headers (for example to accept
additional content encodings), specify @emph{skip_host} or @emph{skip_accept_encoding}
with non-False values.
@end deffn

@geindex putheader() (http.client.HTTPConnection method)
@anchor{library/http client http client HTTPConnection putheader}@anchor{3189}
@deffn {Method} HTTPConnection.putheader (header, argument[, ...])

Send an 
@geindex RFC; RFC 822
RFC 822@footnote{https://tools.ietf.org/html/rfc822.html}-style header to the server.  It sends a line to the server
consisting of the header, a colon and a space, and the first argument.  If more
arguments are given, continuation lines are sent, each consisting of a tab and
an argument.
@end deffn

@geindex endheaders() (http.client.HTTPConnection method)
@anchor{library/http client http client HTTPConnection endheaders}@anchor{7f2}
@deffn {Method} HTTPConnection.endheaders (message_body=None, *, encode_chunked=False)

Send a blank line to the server, signalling the end of the headers. The
optional @emph{message_body} argument can be used to pass a message body
associated with the request.

If @emph{encode_chunked} is @code{True}, the result of each iteration of
@emph{message_body} will be chunk-encoded as specified in 
@geindex RFC; RFC 7230
RFC 7230@footnote{https://tools.ietf.org/html/rfc7230.html},
Section 3.3.1.  How the data is encoded is dependent on the type of
@emph{message_body}.  If @emph{message_body} implements the @ref{1a3e,,buffer interface} the encoding will result in a single chunk.
If @emph{message_body} is a @ref{1b3c,,collections.abc.Iterable}, each iteration
of @emph{message_body} will result in a chunk.  If @emph{message_body} is a
@ref{d68,,file object}, each call to @code{.read()} will result in a chunk.
The method automatically signals the end of the chunk-encoded data
immediately after @emph{message_body}.

@cartouche
@quotation Note 
Due to the chunked encoding specification, empty chunks
yielded by an iterator body will be ignored by the chunk-encoder.
This is to avoid premature termination of the read of the request by
the target server due to malformed encoding.
@end quotation
@end cartouche


New in version 3.6: Chunked encoding support.  The @emph{encode_chunked} parameter was
added.

@end deffn

@geindex send() (http.client.HTTPConnection method)
@anchor{library/http client http client HTTPConnection send}@anchor{318a}
@deffn {Method} HTTPConnection.send (data)

Send data to the server.  This should be used directly only after the
@ref{7f2,,endheaders()} method has been called and before @ref{967,,getresponse()} is
called.

Raises an @ref{1130,,auditing event} @code{http.client.send} with arguments @code{self}, @code{data}.
@end deffn

@node HTTPResponse Objects,Examples<25>,HTTPConnection Objects,http client — HTTP protocol client
@anchor{library/http client httpresponse-objects}@anchor{318b}@anchor{library/http client id2}@anchor{318c}
@subsubsection HTTPResponse Objects


An @ref{c60,,HTTPResponse} instance wraps the HTTP response from the
server.  It provides access to the request headers and the entity
body.  The response is an iterable object and can be used in a with
statement.


Changed in version 3.5: The @ref{823,,io.BufferedIOBase} interface is now implemented and
all of its reader operations are supported.


@geindex read() (http.client.HTTPResponse method)
@anchor{library/http client http client HTTPResponse read}@anchor{318d}
@deffn {Method} HTTPResponse.read ([amt])

Reads and returns the response body, or up to the next @emph{amt} bytes.
@end deffn

@geindex readinto() (http.client.HTTPResponse method)
@anchor{library/http client http client HTTPResponse readinto}@anchor{c61}
@deffn {Method} HTTPResponse.readinto (b)

Reads up to the next len(b) bytes of the response body into the buffer @emph{b}.
Returns the number of bytes read.


New in version 3.3.

@end deffn

@geindex getheader() (http.client.HTTPResponse method)
@anchor{library/http client http client HTTPResponse getheader}@anchor{318e}
@deffn {Method} HTTPResponse.getheader (name, default=None)

Return the value of the header @emph{name}, or @emph{default} if there is no header
matching @emph{name}.  If there is more than one  header with the name @emph{name},
return all of the values joined by ‘, ‘.  If ‘default’ is any iterable other
than a single string, its elements are similarly returned joined by commas.
@end deffn

@geindex getheaders() (http.client.HTTPResponse method)
@anchor{library/http client http client HTTPResponse getheaders}@anchor{318f}
@deffn {Method} HTTPResponse.getheaders ()

Return a list of (header, value) tuples.
@end deffn

@geindex fileno() (http.client.HTTPResponse method)
@anchor{library/http client http client HTTPResponse fileno}@anchor{3190}
@deffn {Method} HTTPResponse.fileno ()

Return the @code{fileno} of the underlying socket.
@end deffn

@geindex msg (http.client.HTTPResponse attribute)
@anchor{library/http client http client HTTPResponse msg}@anchor{3170}
@deffn {Attribute} HTTPResponse.msg

A @code{http.client.HTTPMessage} instance containing the response
headers.  @code{http.client.HTTPMessage} is a subclass of
@ref{7e3,,email.message.Message}.
@end deffn

@geindex version (http.client.HTTPResponse attribute)
@anchor{library/http client http client HTTPResponse version}@anchor{3191}
@deffn {Attribute} HTTPResponse.version

HTTP protocol version used by server.  10 for HTTP/1.0, 11 for HTTP/1.1.
@end deffn

@geindex url (http.client.HTTPResponse attribute)
@anchor{library/http client http client HTTPResponse url}@anchor{3192}
@deffn {Attribute} HTTPResponse.url

URL of the resource retrieved, commonly used to determine if a redirect was followed.
@end deffn

@geindex headers (http.client.HTTPResponse attribute)
@anchor{library/http client http client HTTPResponse headers}@anchor{3193}
@deffn {Attribute} HTTPResponse.headers

Headers of the response in the form of an @ref{7e4,,email.message.EmailMessage} instance.
@end deffn

@geindex status (http.client.HTTPResponse attribute)
@anchor{library/http client http client HTTPResponse status}@anchor{3194}
@deffn {Attribute} HTTPResponse.status

Status code returned by server.
@end deffn

@geindex reason (http.client.HTTPResponse attribute)
@anchor{library/http client http client HTTPResponse reason}@anchor{30b8}
@deffn {Attribute} HTTPResponse.reason

Reason phrase returned by server.
@end deffn

@geindex debuglevel (http.client.HTTPResponse attribute)
@anchor{library/http client http client HTTPResponse debuglevel}@anchor{3195}
@deffn {Attribute} HTTPResponse.debuglevel

A debugging hook.  If @ref{3195,,debuglevel} is greater than zero, messages
will be printed to stdout as the response is read and parsed.
@end deffn

@geindex closed (http.client.HTTPResponse attribute)
@anchor{library/http client http client HTTPResponse closed}@anchor{3196}
@deffn {Attribute} HTTPResponse.closed

Is @code{True} if the stream is closed.
@end deffn

@geindex geturl() (http.client.HTTPResponse method)
@anchor{library/http client http client HTTPResponse geturl}@anchor{3197}
@deffn {Method} HTTPResponse.geturl ()


Deprecated since version 3.9: Deprecated in favor of @ref{3192,,url}.

@end deffn

@geindex info() (http.client.HTTPResponse method)
@anchor{library/http client http client HTTPResponse info}@anchor{3198}
@deffn {Method} HTTPResponse.info ()


Deprecated since version 3.9: Deprecated in favor of @ref{3193,,headers}.

@end deffn

@geindex getstatus() (http.client.HTTPResponse method)
@anchor{library/http client http client HTTPResponse getstatus}@anchor{3199}
@deffn {Method} HTTPResponse.getstatus ()


Deprecated since version 3.9: Deprecated in favor of @ref{3194,,status}.

@end deffn

@node Examples<25>,HTTPMessage Objects,HTTPResponse Objects,http client — HTTP protocol client
@anchor{library/http client examples}@anchor{319a}
@subsubsection Examples


Here is an example session that uses the @code{GET} method:

@example
>>> import http.client
>>> conn = http.client.HTTPSConnection("www.python.org")
>>> conn.request("GET", "/")
>>> r1 = conn.getresponse()
>>> print(r1.status, r1.reason)
200 OK
>>> data1 = r1.read()  # This will return entire content.
>>> # The following example demonstrates reading data in chunks.
>>> conn.request("GET", "/")
>>> r1 = conn.getresponse()
>>> while chunk := r1.read(200):
...     print(repr(chunk))
b'<!doctype html>\n<!--[if"...
...
>>> # Example of an invalid request
>>> conn = http.client.HTTPSConnection("docs.python.org")
>>> conn.request("GET", "/parrot.spam")
>>> r2 = conn.getresponse()
>>> print(r2.status, r2.reason)
404 Not Found
>>> data2 = r2.read()
>>> conn.close()
@end example

Here is an example session that uses the @code{HEAD} method.  Note that the
@code{HEAD} method never returns any data.

@example
>>> import http.client
>>> conn = http.client.HTTPSConnection("www.python.org")
>>> conn.request("HEAD", "/")
>>> res = conn.getresponse()
>>> print(res.status, res.reason)
200 OK
>>> data = res.read()
>>> print(len(data))
0
>>> data == b'@w{'}
True
@end example

Here is an example session that shows how to @code{POST} requests:

@example
>>> import http.client, urllib.parse
>>> params = urllib.parse.urlencode(@{'@@number': 12524, '@@type': 'issue', '@@action': 'show'@})
>>> headers = @{"Content-type": "application/x-www-form-urlencoded",
...            "Accept": "text/plain"@}
>>> conn = http.client.HTTPConnection("bugs.python.org")
>>> conn.request("POST", "", params, headers)
>>> response = conn.getresponse()
>>> print(response.status, response.reason)
302 Found
>>> data = response.read()
>>> data
b'Redirecting to <a href="http://bugs.python.org/issue12524">http://bugs.python.org/issue12524</a>'
>>> conn.close()
@end example

Client side @code{HTTP PUT} requests are very similar to @code{POST} requests. The
difference lies only the server side where HTTP server will allow resources to
be created via @code{PUT} request. It should be noted that custom HTTP methods
are also handled in @ref{b5e,,urllib.request.Request} by setting the appropriate
method attribute. Here is an example session that shows how to send a @code{PUT}
request using http.client:

@example
>>> # This creates an HTTP message
>>> # with the content of BODY as the enclosed representation
>>> # for the resource http://localhost:8080/file
...
>>> import http.client
>>> BODY = "***filecontents***"
>>> conn = http.client.HTTPConnection("localhost", 8080)
>>> conn.request("PUT", "/file", BODY)
>>> response = conn.getresponse()
>>> print(response.status, response.reason)
200, OK
@end example

@node HTTPMessage Objects,,Examples<25>,http client — HTTP protocol client
@anchor{library/http client httpmessage-objects}@anchor{319b}@anchor{library/http client id3}@anchor{319c}
@subsubsection HTTPMessage Objects


An @code{http.client.HTTPMessage} instance holds the headers from an HTTP
response.  It is implemented using the @ref{7e3,,email.message.Message} class.

@c XXX Define the methods that clients can depend upon between versions.

@node ftplib — FTP protocol client,poplib — POP3 protocol client,http client — HTTP protocol client,Internet Protocols and Support
@anchor{library/ftplib doc}@anchor{319d}@anchor{library/ftplib ftplib-ftp-protocol-client}@anchor{319e}@anchor{library/ftplib module-ftplib}@anchor{7e}
@subsection @code{ftplib} — FTP protocol client


@geindex module; ftplib

@strong{Source code:} Lib/ftplib.py@footnote{https://github.com/python/cpython/tree/main/Lib/ftplib.py}

@geindex FTP; protocol
@geindex FTP; ftplib (standard module)


__________________________________________________________________


This module defines the class @ref{3e9,,FTP} and a few related items. The
@ref{3e9,,FTP} class implements the client side of the FTP protocol.  You can use
this to write Python programs that perform a variety of automated FTP jobs, such
as mirroring other FTP servers.  It is also used by the module
@ref{127,,urllib.request} to handle URLs that use FTP.  For more information on FTP
(File Transfer Protocol), see internet 
@geindex RFC; RFC 959
RFC 959@footnote{https://tools.ietf.org/html/rfc959.html}.

The default encoding is UTF-8, following 
@geindex RFC; RFC 2640
RFC 2640@footnote{https://tools.ietf.org/html/rfc2640.html}.

Here’s a sample session using the @ref{7e,,ftplib} module:

@example
>>> from ftplib import FTP
>>> ftp = FTP('ftp.us.debian.org')  # connect to host, default port
>>> ftp.login()                     # user anonymous, passwd anonymous@@
'230 Login successful.'
>>> ftp.cwd('debian')               # change into "debian" directory
'250 Directory successfully changed.'
>>> ftp.retrlines('LIST')           # list directory contents
-rw-rw-r--    1 1176     1176         1063 Jun 15 10:18 README
...
drwxr-sr-x    5 1176     1176         4096 Dec 19  2000 pool
drwxr-sr-x    4 1176     1176         4096 Nov 17  2008 project
drwxr-xr-x    3 1176     1176         4096 Oct 10  2012 tools
'226 Directory send OK.'
>>> with open('README', 'wb') as fp:
>>>     ftp.retrbinary('RETR README', fp.write)
'226 Transfer complete.'
>>> ftp.quit()
'221 Goodbye.'
@end example

The module defines the following items:

@geindex FTP (class in ftplib)
@anchor{library/ftplib ftplib FTP}@anchor{3e9}
@deffn {Class} ftplib.FTP (host='@w{'}, user='@w{'}, passwd='@w{'}, acct='@w{'}, timeout=None, source_address=None, *, encoding='utf-8')

Return a new instance of the @ref{3e9,,FTP} class.  When @emph{host} is given, the
method call @code{connect(host)} is made.  When @emph{user} is given, additionally
the method call @code{login(user, passwd, acct)} is made (where @emph{passwd} and
@emph{acct} default to the empty string when not given).  The optional @emph{timeout}
parameter specifies a timeout in seconds for blocking operations like the
connection attempt (if is not specified, the global default timeout setting
will be used). @emph{source_address} is a 2-tuple @code{(host, port)} for the socket
to bind to as its source address before connecting. The @emph{encoding} parameter
specifies the encoding for directories and filenames.

The @ref{3e9,,FTP} class supports the @ref{168,,with} statement, e.g.:

@example
>>> from ftplib import FTP
>>> with FTP("ftp1.at.proftpd.org") as ftp:
...     ftp.login()
...     ftp.dir()
... 
'230 Anonymous login ok, restrictions apply.'
dr-xr-xr-x   9 ftp      ftp           154 May  6 10:43 .
dr-xr-xr-x   9 ftp      ftp           154 May  6 10:43 ..
dr-xr-xr-x   5 ftp      ftp          4096 May  6 10:43 CentOS
dr-xr-xr-x   3 ftp      ftp            18 Jul 10  2008 Fedora
>>>
@end example


Changed in version 3.2: Support for the @ref{168,,with} statement was added.



Changed in version 3.3: @emph{source_address} parameter was added.



Changed in version 3.9: If the @emph{timeout} parameter is set to be zero, it will raise a
@ref{17c,,ValueError} to prevent the creation of a non-blocking socket.
The @emph{encoding} parameter was added, and the default was changed from
Latin-1 to UTF-8 to follow 
@geindex RFC; RFC 2640
RFC 2640@footnote{https://tools.ietf.org/html/rfc2640.html}.

@end deffn

@geindex FTP_TLS (class in ftplib)
@anchor{library/ftplib ftplib FTP_TLS}@anchor{3ea}
@deffn {Class} ftplib.FTP_TLS (host='@w{'}, user='@w{'}, passwd='@w{'}, acct='@w{'}, keyfile=None, certfile=None, context=None, timeout=None, source_address=None, *, encoding='utf-8')

A @ref{3e9,,FTP} subclass which adds TLS support to FTP as described in
@geindex RFC; RFC 4217
RFC 4217@footnote{https://tools.ietf.org/html/rfc4217.html}.
Connect as usual to port 21 implicitly securing the FTP control connection
before authenticating. Securing the data connection requires the user to
explicitly ask for it by calling the @ref{319f,,prot_p()} method.  @emph{context}
is a @ref{2bf,,ssl.SSLContext} object which allows bundling SSL configuration
options, certificates and private keys into a single (potentially
long-lived) structure.  Please read @ref{2a39,,Security considerations} for best practices.

@emph{keyfile} and @emph{certfile} are a legacy alternative to @emph{context} – they
can point to PEM-formatted private key and certificate chain files
(respectively) for the SSL connection.


New in version 3.2.



Changed in version 3.3: @emph{source_address} parameter was added.



Changed in version 3.4: The class now supports hostname check with
@ref{2b42,,ssl.SSLContext.check_hostname} and @emph{Server Name Indication} (see
@ref{2b4f,,ssl.HAS_SNI}).



Deprecated since version 3.6: @emph{keyfile} and @emph{certfile} are deprecated in favor of @emph{context}.
Please use @ref{cd9,,ssl.SSLContext.load_cert_chain()} instead, or let
@ref{b29,,ssl.create_default_context()} select the system’s trusted CA
certificates for you.



Changed in version 3.9: If the @emph{timeout} parameter is set to be zero, it will raise a
@ref{17c,,ValueError} to prevent the creation of a non-blocking socket.
The @emph{encoding} parameter was added, and the default was changed from
Latin-1 to UTF-8 to follow 
@geindex RFC; RFC 2640
RFC 2640@footnote{https://tools.ietf.org/html/rfc2640.html}.


Here’s a sample session using the @ref{3ea,,FTP_TLS} class:

@example
>>> ftps = FTP_TLS('ftp.pureftpd.org')
>>> ftps.login()
'230 Anonymous user logged in'
>>> ftps.prot_p()
'200 Data protection level set to "private"'
>>> ftps.nlst()
['6jack', 'OpenBSD', 'antilink', 'blogbench', 'bsdcam', 'clockspeed', 'djbdns-jedi', 'docs', 'eaccelerator-jedi', 'favicon.ico', 'francotone', 'fugu', 'ignore', 'libpuzzle', 'metalog', 'minidentd', 'misc', 'mysql-udf-global-user-variables', 'php-jenkins-hash', 'php-skein-hash', 'php-webdav', 'phpaudit', 'phpbench', 'pincaster', 'ping', 'posto', 'pub', 'public', 'public_keys', 'pure-ftpd', 'qscan', 'qtc', 'sharedance', 'skycache', 'sound', 'tmp', 'ucarp']
@end example
@end deffn

@geindex error_reply
@anchor{library/ftplib ftplib error_reply}@anchor{31a0}
@deffn {Exception} ftplib.error_reply

Exception raised when an unexpected reply is received from the server.
@end deffn

@geindex error_temp
@anchor{library/ftplib ftplib error_temp}@anchor{31a1}
@deffn {Exception} ftplib.error_temp

Exception raised when an error code signifying a temporary error (response
codes in the range 400–499) is received.
@end deffn

@geindex error_perm
@anchor{library/ftplib ftplib error_perm}@anchor{31a2}
@deffn {Exception} ftplib.error_perm

Exception raised when an error code signifying a permanent error (response
codes in the range 500–599) is received.
@end deffn

@geindex error_proto
@anchor{library/ftplib ftplib error_proto}@anchor{31a3}
@deffn {Exception} ftplib.error_proto

Exception raised when a reply is received from the server that does not fit
the response specifications of the File Transfer Protocol, i.e. begin with a
digit in the range 1–5.
@end deffn

@geindex all_errors (in module ftplib)
@anchor{library/ftplib ftplib all_errors}@anchor{31a4}
@deffn {Data} ftplib.all_errors

The set of all exceptions (as a tuple) that methods of @ref{3e9,,FTP}
instances may raise as a result of problems with the FTP connection (as
opposed to programming errors made by the caller).  This set includes the
four exceptions listed above as well as @ref{2a1,,OSError} and @ref{e74,,EOFError}.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{ba,,netrc}

Parser for the @code{.netrc} file format.  The file @code{.netrc} is
typically used by FTP clients to load user authentication information
before prompting the user.
@end table


@menu
* FTP Objects:: 
* FTP_TLS Objects:: 

@end menu

@node FTP Objects,FTP_TLS Objects,,ftplib — FTP protocol client
@anchor{library/ftplib ftp-objects}@anchor{31a5}@anchor{library/ftplib id1}@anchor{31a6}
@subsubsection FTP Objects


Several methods are available in two flavors: one for handling text files and
another for binary files.  These are named for the command which is used
followed by @code{lines} for the text version or @code{binary} for the binary version.

@ref{3e9,,FTP} instances have the following methods:

@geindex set_debuglevel() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP set_debuglevel}@anchor{31a7}
@deffn {Method} FTP.set_debuglevel (level)

Set the instance’s debugging level.  This controls the amount of debugging
output printed.  The default, @code{0}, produces no debugging output.  A value of
@code{1} produces a moderate amount of debugging output, generally a single line
per request.  A value of @code{2} or higher produces the maximum amount of
debugging output, logging each line sent and received on the control connection.
@end deffn

@geindex connect() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP connect}@anchor{31a8}
@deffn {Method} FTP.connect (host='@w{'}, port=0, timeout=None, source_address=None)

Connect to the given host and port.  The default port number is @code{21}, as
specified by the FTP protocol specification.  It is rarely needed to specify a
different port number.  This function should be called only once for each
instance; it should not be called at all if a host was given when the instance
was created.  All other methods can only be used after a connection has been
made.
The optional @emph{timeout} parameter specifies a timeout in seconds for the
connection attempt. If no @emph{timeout} is passed, the global default timeout
setting will be used.
@emph{source_address} is a 2-tuple @code{(host, port)} for the socket to bind to as
its source address before connecting.

Raises an @ref{1130,,auditing event} @code{ftplib.connect} with arguments @code{self}, @code{host}, @code{port}.


Changed in version 3.3: @emph{source_address} parameter was added.

@end deffn

@geindex getwelcome() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP getwelcome}@anchor{31a9}
@deffn {Method} FTP.getwelcome ()

Return the welcome message sent by the server in reply to the initial
connection.  (This message sometimes contains disclaimers or help information
that may be relevant to the user.)
@end deffn

@geindex login() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP login}@anchor{31aa}
@deffn {Method} FTP.login (user='anonymous', passwd='@w{'}, acct='@w{'})

Log in as the given @emph{user}.  The @emph{passwd} and @emph{acct} parameters are optional and
default to the empty string.  If no @emph{user} is specified, it defaults to
@code{'anonymous'}.  If @emph{user} is @code{'anonymous'}, the default @emph{passwd} is
@code{'anonymous@@'}.  This function should be called only once for each instance,
after a connection has been established; it should not be called at all if a
host and user were given when the instance was created.  Most FTP commands are
only allowed after the client has logged in.  The @emph{acct} parameter supplies
“accounting information”; few systems implement this.
@end deffn

@geindex abort() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP abort}@anchor{31ab}
@deffn {Method} FTP.abort ()

Abort a file transfer that is in progress.  Using this does not always work, but
it’s worth a try.
@end deffn

@geindex sendcmd() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP sendcmd}@anchor{31ac}
@deffn {Method} FTP.sendcmd (cmd)

Send a simple command string to the server and return the response string.

Raises an @ref{1130,,auditing event} @code{ftplib.sendcmd} with arguments @code{self}, @code{cmd}.
@end deffn

@geindex voidcmd() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP voidcmd}@anchor{31ad}
@deffn {Method} FTP.voidcmd (cmd)

Send a simple command string to the server and handle the response.  Return
nothing if a response code corresponding to success (codes in the range
200–299) is received.  Raise @ref{31a0,,error_reply} otherwise.

Raises an @ref{1130,,auditing event} @code{ftplib.sendcmd} with arguments @code{self}, @code{cmd}.
@end deffn

@geindex retrbinary() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP retrbinary}@anchor{31ae}
@deffn {Method} FTP.retrbinary (cmd, callback, blocksize=8192, rest=None)

Retrieve a file in binary transfer mode.  @emph{cmd} should be an appropriate
@code{RETR} command: @code{'RETR filename'}. The @emph{callback} function is called for
each block of data received, with a single bytes argument giving the data
block. The optional @emph{blocksize} argument specifies the maximum chunk size to
read on the low-level socket object created to do the actual transfer (which
will also be the largest size of the data blocks passed to @emph{callback}).  A
reasonable default is chosen. @emph{rest} means the same thing as in the
@ref{31af,,transfercmd()} method.
@end deffn

@geindex retrlines() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP retrlines}@anchor{31b0}
@deffn {Method} FTP.retrlines (cmd, callback=None)

Retrieve a file or directory listing in the encoding specified by the
@emph{encoding} parameter at initialization.
@emph{cmd} should be an appropriate @code{RETR} command (see @ref{31ae,,retrbinary()}) or
a command such as @code{LIST} or @code{NLST} (usually just the string @code{'LIST'}).
@code{LIST} retrieves a list of files and information about those files.
@code{NLST} retrieves a list of file names.
The @emph{callback} function is called for each line with a string argument
containing the line with the trailing CRLF stripped.  The default @emph{callback}
prints the line to @code{sys.stdout}.
@end deffn

@geindex set_pasv() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP set_pasv}@anchor{31b1}
@deffn {Method} FTP.set_pasv (val)

Enable “passive” mode if @emph{val} is true, otherwise disable passive mode.
Passive mode is on by default.
@end deffn

@geindex storbinary() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP storbinary}@anchor{ea4}
@deffn {Method} FTP.storbinary (cmd, fp, blocksize=8192, callback=None, rest=None)

Store a file in binary transfer mode.  @emph{cmd} should be an appropriate
@code{STOR} command: @code{"STOR filename"}. @emph{fp} is a @ref{d68,,file object}
(opened in binary mode) which is read until EOF using its @code{read()}
method in blocks of size @emph{blocksize} to provide the data to be stored.
The @emph{blocksize} argument defaults to 8192.  @emph{callback} is an optional single
parameter callable that is called on each block of data after it is sent.
@emph{rest} means the same thing as in the @ref{31af,,transfercmd()} method.


Changed in version 3.2: @emph{rest} parameter added.

@end deffn

@geindex storlines() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP storlines}@anchor{31b2}
@deffn {Method} FTP.storlines (cmd, fp, callback=None)

Store a file in line mode.  @emph{cmd} should be an appropriate
@code{STOR} command (see @ref{ea4,,storbinary()}).  Lines are read until EOF from the
@ref{d68,,file object} @emph{fp} (opened in binary mode) using its @ref{1b74,,readline()}
method to provide the data to be stored.  @emph{callback} is an optional single
parameter callable that is called on each line after it is sent.
@end deffn

@geindex transfercmd() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP transfercmd}@anchor{31af}
@deffn {Method} FTP.transfercmd (cmd, rest=None)

Initiate a transfer over the data connection.  If the transfer is active, send an
@code{EPRT} or  @code{PORT} command and the transfer command specified by @emph{cmd}, and
accept the connection.  If the server is passive, send an @code{EPSV} or @code{PASV}
command, connect to it, and start the transfer command.  Either way, return the
socket for the connection.

If optional @emph{rest} is given, a @code{REST} command is sent to the server, passing
@emph{rest} as an argument.  @emph{rest} is usually a byte offset into the requested file,
telling the server to restart sending the file’s bytes at the requested offset,
skipping over the initial bytes.  Note however that the @ref{31af,,transfercmd()}
method converts @emph{rest} to a string with the @emph{encoding} parameter specified
at initialization, but no check is performed on the string’s contents.  If the
server does not recognize the @code{REST} command, an @ref{31a0,,error_reply} exception
will be raised.  If this happens, simply call @ref{31af,,transfercmd()} without a
@emph{rest} argument.
@end deffn

@geindex ntransfercmd() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP ntransfercmd}@anchor{31b3}
@deffn {Method} FTP.ntransfercmd (cmd, rest=None)

Like @ref{31af,,transfercmd()}, but returns a tuple of the data connection and the
expected size of the data.  If the expected size could not be computed, @code{None}
will be returned as the expected size.  @emph{cmd} and @emph{rest} means the same thing as
in @ref{31af,,transfercmd()}.
@end deffn

@geindex mlsd() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP mlsd}@anchor{c54}
@deffn {Method} FTP.mlsd (path='@w{'}, facts=[])

List a directory in a standardized format by using @code{MLSD} command
(
@geindex RFC; RFC 3659
RFC 3659@footnote{https://tools.ietf.org/html/rfc3659.html}).  If @emph{path} is omitted the current directory is assumed.
@emph{facts} is a list of strings representing the type of information desired
(e.g. @code{["type", "size", "perm"]}).  Return a generator object yielding a
tuple of two elements for every file found in path.  First element is the
file name, the second one is a dictionary containing facts about the file
name.  Content of this dictionary might be limited by the @emph{facts} argument
but server is not guaranteed to return all requested facts.


New in version 3.3.

@end deffn

@geindex nlst() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP nlst}@anchor{c55}
@deffn {Method} FTP.nlst (argument[, ...])

Return a list of file names as returned by the @code{NLST} command.  The
optional @emph{argument} is a directory to list (default is the current server
directory).  Multiple arguments can be used to pass non-standard options to
the @code{NLST} command.

@cartouche
@quotation Note 
If your server supports the command, @ref{c54,,mlsd()} offers a better API.
@end quotation
@end cartouche
@end deffn

@geindex dir() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP dir}@anchor{c56}
@deffn {Method} FTP.dir (argument[, ...])

Produce a directory listing as returned by the @code{LIST} command, printing it to
standard output.  The optional @emph{argument} is a directory to list (default is the
current server directory).  Multiple arguments can be used to pass non-standard
options to the @code{LIST} command.  If the last argument is a function, it is used
as a @emph{callback} function as for @ref{31b0,,retrlines()}; the default prints to
@code{sys.stdout}.  This method returns @code{None}.

@cartouche
@quotation Note 
If your server supports the command, @ref{c54,,mlsd()} offers a better API.
@end quotation
@end cartouche
@end deffn

@geindex rename() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP rename}@anchor{31b4}
@deffn {Method} FTP.rename (fromname, toname)

Rename file @emph{fromname} on the server to @emph{toname}.
@end deffn

@geindex delete() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP delete}@anchor{31b5}
@deffn {Method} FTP.delete (filename)

Remove the file named @emph{filename} from the server.  If successful, returns the
text of the response, otherwise raises @ref{31a2,,error_perm} on permission errors or
@ref{31a0,,error_reply} on other errors.
@end deffn

@geindex cwd() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP cwd}@anchor{31b6}
@deffn {Method} FTP.cwd (pathname)

Set the current directory on the server.
@end deffn

@geindex mkd() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP mkd}@anchor{31b7}
@deffn {Method} FTP.mkd (pathname)

Create a new directory on the server.
@end deffn

@geindex pwd() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP pwd}@anchor{31b8}
@deffn {Method} FTP.pwd ()

Return the pathname of the current directory on the server.
@end deffn

@geindex rmd() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP rmd}@anchor{31b9}
@deffn {Method} FTP.rmd (dirname)

Remove the directory named @emph{dirname} on the server.
@end deffn

@geindex size() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP size}@anchor{31ba}
@deffn {Method} FTP.size (filename)

Request the size of the file named @emph{filename} on the server.  On success, the
size of the file is returned as an integer, otherwise @code{None} is returned.
Note that the @code{SIZE} command is not  standardized, but is supported by many
common server implementations.
@end deffn

@geindex quit() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP quit}@anchor{31bb}
@deffn {Method} FTP.quit ()

Send a @code{QUIT} command to the server and close the connection. This is the
“polite” way to close a connection, but it may raise an exception if the server
responds with an error to the @code{QUIT} command.  This implies a call to the
@ref{31bc,,close()} method which renders the @ref{3e9,,FTP} instance useless for
subsequent calls (see below).
@end deffn

@geindex close() (ftplib.FTP method)
@anchor{library/ftplib ftplib FTP close}@anchor{31bc}
@deffn {Method} FTP.close ()

Close the connection unilaterally.  This should not be applied to an already
closed connection such as after a successful call to @ref{31bb,,quit()}.
After this call the @ref{3e9,,FTP} instance should not be used any more (after
a call to @ref{31bc,,close()} or @ref{31bb,,quit()} you cannot reopen the
connection by issuing another @ref{31aa,,login()} method).
@end deffn

@node FTP_TLS Objects,,FTP Objects,ftplib — FTP protocol client
@anchor{library/ftplib ftp-tls-objects}@anchor{31bd}
@subsubsection FTP_TLS Objects


@ref{3ea,,FTP_TLS} class inherits from @ref{3e9,,FTP}, defining these additional objects:

@geindex ssl_version (ftplib.FTP_TLS attribute)
@anchor{library/ftplib ftplib FTP_TLS ssl_version}@anchor{31be}
@deffn {Attribute} FTP_TLS.ssl_version

The SSL version to use (defaults to @ref{32a,,ssl.PROTOCOL_SSLv23}).
@end deffn

@geindex auth() (ftplib.FTP_TLS method)
@anchor{library/ftplib ftplib FTP_TLS auth}@anchor{31bf}
@deffn {Method} FTP_TLS.auth ()

Set up a secure control connection by using TLS or SSL, depending on what
is specified in the @ref{31be,,ssl_version} attribute.


Changed in version 3.4: The method now supports hostname check with
@ref{2b42,,ssl.SSLContext.check_hostname} and @emph{Server Name Indication} (see
@ref{2b4f,,ssl.HAS_SNI}).

@end deffn

@geindex ccc() (ftplib.FTP_TLS method)
@anchor{library/ftplib ftplib FTP_TLS ccc}@anchor{c53}
@deffn {Method} FTP_TLS.ccc ()

Revert control channel back to plaintext.  This can be useful to take
advantage of firewalls that know how to handle NAT with non-secure FTP
without opening fixed ports.


New in version 3.3.

@end deffn

@geindex prot_p() (ftplib.FTP_TLS method)
@anchor{library/ftplib ftplib FTP_TLS prot_p}@anchor{319f}
@deffn {Method} FTP_TLS.prot_p ()

Set up secure data connection.
@end deffn

@geindex prot_c() (ftplib.FTP_TLS method)
@anchor{library/ftplib ftplib FTP_TLS prot_c}@anchor{31c0}
@deffn {Method} FTP_TLS.prot_c ()

Set up clear text data connection.
@end deffn

@node poplib — POP3 protocol client,imaplib — IMAP4 protocol client,ftplib — FTP protocol client,Internet Protocols and Support
@anchor{library/poplib doc}@anchor{31c1}@anchor{library/poplib module-poplib}@anchor{cb}@anchor{library/poplib poplib-pop3-protocol-client}@anchor{31c2}
@subsection @code{poplib} — POP3 protocol client


@geindex module; poplib

@c revised by ESR, January 2000

@strong{Source code:} Lib/poplib.py@footnote{https://github.com/python/cpython/tree/main/Lib/poplib.py}

@geindex POP3; protocol


__________________________________________________________________


This module defines a class, @ref{418,,POP3}, which encapsulates a connection to a
POP3 server and implements the protocol as defined in 
@geindex RFC; RFC 1939
RFC 1939@footnote{https://tools.ietf.org/html/rfc1939.html}. The
@ref{418,,POP3} class supports both the minimal and optional command sets from
@geindex RFC; RFC 1939
RFC 1939@footnote{https://tools.ietf.org/html/rfc1939.html}. The @ref{418,,POP3} class also supports the @code{STLS} command introduced
in 
@geindex RFC; RFC 2595
RFC 2595@footnote{https://tools.ietf.org/html/rfc2595.html} to enable encrypted communication on an already established connection.

Additionally, this module provides a class @ref{419,,POP3_SSL}, which provides
support for connecting to POP3 servers that use SSL as an underlying protocol
layer.

Note that POP3, though widely supported, is obsolescent.  The implementation
quality of POP3 servers varies widely, and too many are quite poor. If your
mailserver supports IMAP, you would be better off using the
@ref{3f2,,imaplib.IMAP4} class, as IMAP servers tend to be better implemented.

The @ref{cb,,poplib} module provides two classes:

@geindex POP3 (class in poplib)
@anchor{library/poplib poplib POP3}@anchor{418}
@deffn {Class} poplib.POP3 (host, port=POP3_PORT[, timeout])

This class implements the actual POP3 protocol.  The connection is created when
the instance is initialized. If @emph{port} is omitted, the standard POP3 port (110)
is used. The optional @emph{timeout} parameter specifies a timeout in seconds for the
connection attempt (if not specified, the global default timeout setting will
be used).

Raises an @ref{1130,,auditing event} @code{poplib.connect} with arguments @code{self}, @code{host}, @code{port}.


All commands will raise an @ref{1130,,auditing event}
@code{poplib.putline} with arguments @code{self} and @code{line},
where @code{line} is the bytes about to be sent to the remote host.



Changed in version 3.9: If the @emph{timeout} parameter is set to be zero, it will raise a
@ref{17c,,ValueError} to prevent the creation of a non-blocking socket.

@end deffn

@geindex POP3_SSL (class in poplib)
@anchor{library/poplib poplib POP3_SSL}@anchor{419}
@deffn {Class} poplib.POP3_SSL (host, port=POP3_SSL_PORT, keyfile=None, certfile=None, timeout=None, context=None)

This is a subclass of @ref{418,,POP3} that connects to the server over an SSL
encrypted socket.  If @emph{port} is not specified, 995, the standard POP3-over-SSL
port is used.  @emph{timeout} works as in the @ref{418,,POP3} constructor.
@emph{context} is an optional @ref{2bf,,ssl.SSLContext} object which allows
bundling SSL configuration options, certificates and private keys into a
single (potentially long-lived) structure.  Please read @ref{2a39,,Security considerations}
for best practices.

@emph{keyfile} and @emph{certfile} are a legacy alternative to @emph{context} - they can
point to PEM-formatted private key and certificate chain files,
respectively, for the SSL connection.

Raises an @ref{1130,,auditing event} @code{poplib.connect} with arguments @code{self}, @code{host}, @code{port}.


All commands will raise an @ref{1130,,auditing event}
@code{poplib.putline} with arguments @code{self} and @code{line},
where @code{line} is the bytes about to be sent to the remote host.



Changed in version 3.2: @emph{context} parameter added.



Changed in version 3.4: The class now supports hostname check with
@ref{2b42,,ssl.SSLContext.check_hostname} and @emph{Server Name Indication} (see
@ref{2b4f,,ssl.HAS_SNI}).



Deprecated since version 3.6: @emph{keyfile} and @emph{certfile} are deprecated in favor of @emph{context}.
Please use @ref{cd9,,ssl.SSLContext.load_cert_chain()} instead, or let
@ref{b29,,ssl.create_default_context()} select the system’s trusted CA
certificates for you.



Changed in version 3.9: If the @emph{timeout} parameter is set to be zero, it will raise a
@ref{17c,,ValueError} to prevent the creation of a non-blocking socket.

@end deffn

One exception is defined as an attribute of the @ref{cb,,poplib} module:

@geindex error_proto
@anchor{library/poplib poplib error_proto}@anchor{d42}
@deffn {Exception} poplib.error_proto

Exception raised on any errors from this module (errors from @ref{ea,,socket}
module are not caught). The reason for the exception is passed to the
constructor as a string.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{93,,imaplib}

The standard Python IMAP module.

@item Frequently Asked Questions About Fetchmail@footnote{http://www.catb.org/~esr/fetchmail/fetchmail-FAQ.html}

The FAQ for the @code{fetchmail} POP/IMAP client collects information on
POP3 server variations and RFC noncompliance that may be useful if you need to
write an application based on the POP protocol.
@end table


@menu
* POP3 Objects:: 
* POP3 Example:: 

@end menu

@node POP3 Objects,POP3 Example,,poplib — POP3 protocol client
@anchor{library/poplib id1}@anchor{31c3}@anchor{library/poplib pop3-objects}@anchor{31c4}
@subsubsection POP3 Objects


All POP3 commands are represented by methods of the same name, in lowercase;
most return the response text sent by the server.

A @ref{418,,POP3} instance has the following methods:

@geindex set_debuglevel() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 set_debuglevel}@anchor{31c5}
@deffn {Method} POP3.set_debuglevel (level)

Set the instance’s debugging level.  This controls the amount of debugging
output printed.  The default, @code{0}, produces no debugging output.  A value of
@code{1} produces a moderate amount of debugging output, generally a single line
per request.  A value of @code{2} or higher produces the maximum amount of
debugging output, logging each line sent and received on the control connection.
@end deffn

@geindex getwelcome() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 getwelcome}@anchor{31c6}
@deffn {Method} POP3.getwelcome ()

Returns the greeting string sent by the POP3 server.
@end deffn

@geindex capa() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 capa}@anchor{afd}
@deffn {Method} POP3.capa ()

Query the server’s capabilities as specified in 
@geindex RFC; RFC 2449
RFC 2449@footnote{https://tools.ietf.org/html/rfc2449.html}.
Returns a dictionary in the form @code{@{'name': ['param'...]@}}.


New in version 3.4.

@end deffn

@geindex user() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 user}@anchor{31c7}
@deffn {Method} POP3.user (username)

Send user command, response should indicate that a password is required.
@end deffn

@geindex pass_() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 pass_}@anchor{31c8}
@deffn {Method} POP3.pass_ (password)

Send password, response includes message count and mailbox size. Note: the
mailbox on the server is locked until @code{quit()} is called.
@end deffn

@geindex apop() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 apop}@anchor{31c9}
@deffn {Method} POP3.apop (user, secret)

Use the more secure APOP authentication to log into the POP3 server.
@end deffn

@geindex rpop() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 rpop}@anchor{31ca}
@deffn {Method} POP3.rpop (user)

Use RPOP authentication (similar to UNIX r-commands) to log into POP3 server.
@end deffn

@geindex stat() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 stat}@anchor{31cb}
@deffn {Method} POP3.stat ()

Get mailbox status.  The result is a tuple of 2 integers: @code{(message count,
mailbox size)}.
@end deffn

@geindex list() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 list}@anchor{31cc}
@deffn {Method} POP3.list ([which])

Request message list, result is in the form @code{(response, ['mesg_num octets',
...], octets)}. If @emph{which} is set, it is the message to list.
@end deffn

@geindex retr() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 retr}@anchor{31cd}
@deffn {Method} POP3.retr (which)

Retrieve whole message number @emph{which}, and set its seen flag. Result is in form
@code{(response, ['line', ...], octets)}.
@end deffn

@geindex dele() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 dele}@anchor{31ce}
@deffn {Method} POP3.dele (which)

Flag message number @emph{which} for deletion.  On most servers deletions are not
actually performed until QUIT (the major exception is Eudora QPOP, which
deliberately violates the RFCs by doing pending deletes on any disconnect).
@end deffn

@geindex rset() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 rset}@anchor{31cf}
@deffn {Method} POP3.rset ()

Remove any deletion marks for the mailbox.
@end deffn

@geindex noop() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 noop}@anchor{31d0}
@deffn {Method} POP3.noop ()

Do nothing.  Might be used as a keep-alive.
@end deffn

@geindex quit() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 quit}@anchor{d41}
@deffn {Method} POP3.quit ()

Signoff:  commit changes, unlock mailbox, drop connection.
@end deffn

@geindex top() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 top}@anchor{31d1}
@deffn {Method} POP3.top (which, howmuch)

Retrieves the message header plus @emph{howmuch} lines of the message after the
header of message number @emph{which}. Result is in form @code{(response, ['line', ...],
octets)}.

The POP3 TOP command this method uses, unlike the RETR command, doesn’t set the
message’s seen flag; unfortunately, TOP is poorly specified in the RFCs and is
frequently broken in off-brand servers. Test this method by hand against the
POP3 servers you will use before trusting it.
@end deffn

@geindex uidl() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 uidl}@anchor{31d2}
@deffn {Method} POP3.uidl (which=None)

Return message digest (unique id) list. If @emph{which} is specified, result contains
the unique id for that message in the form @code{'response mesgnum uid}, otherwise
result is list @code{(response, ['mesgnum uid', ...], octets)}.
@end deffn

@geindex utf8() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 utf8}@anchor{9b1}
@deffn {Method} POP3.utf8 ()

Try to switch to UTF-8 mode. Returns the server response if successful,
raises @ref{d42,,error_proto} if not. Specified in 
@geindex RFC; RFC 6856
RFC 6856@footnote{https://tools.ietf.org/html/rfc6856.html}.


New in version 3.5.

@end deffn

@geindex stls() (poplib.POP3 method)
@anchor{library/poplib poplib POP3 stls}@anchor{afe}
@deffn {Method} POP3.stls (context=None)

Start a TLS session on the active connection as specified in 
@geindex RFC; RFC 2595
RFC 2595@footnote{https://tools.ietf.org/html/rfc2595.html}.
This is only allowed before user authentication

@emph{context} parameter is a @ref{2bf,,ssl.SSLContext} object which allows
bundling SSL configuration options, certificates and private keys into
a single (potentially long-lived) structure.  Please read @ref{2a39,,Security considerations}
for best practices.

This method supports hostname checking via
@ref{2b42,,ssl.SSLContext.check_hostname} and @emph{Server Name Indication} (see
@ref{2b4f,,ssl.HAS_SNI}).


New in version 3.4.

@end deffn

Instances of @ref{419,,POP3_SSL} have no additional methods. The interface of this
subclass is identical to its parent.

@node POP3 Example,,POP3 Objects,poplib — POP3 protocol client
@anchor{library/poplib id2}@anchor{31d3}@anchor{library/poplib pop3-example}@anchor{31d4}
@subsubsection POP3 Example


Here is a minimal example (without error checking) that opens a mailbox and
retrieves and prints all messages:

@example
import getpass, poplib

M = poplib.POP3('localhost')
M.user(getpass.getuser())
M.pass_(getpass.getpass())
numMessages = len(M.list()[1])
for i in range(numMessages):
    for j in M.retr(i+1)[1]:
        print(j)
@end example

At the end of the module, there is a test section that contains a more extensive
example of usage.

@node imaplib — IMAP4 protocol client,nntplib — NNTP protocol client,poplib — POP3 protocol client,Internet Protocols and Support
@anchor{library/imaplib doc}@anchor{31d5}@anchor{library/imaplib imaplib-imap4-protocol-client}@anchor{31d6}@anchor{library/imaplib module-imaplib}@anchor{93}
@subsection @code{imaplib} — IMAP4 protocol client


@geindex module; imaplib

@c revised by ESR, January 2000

@c changes for IMAP4_SSL by Tino Lange <Tino.Lange@isg.de>, March 2002

@c changes for IMAP4_stream by Piers Lauder <piers@communitysolutions.com.au>,
@c November 2002

@strong{Source code:} Lib/imaplib.py@footnote{https://github.com/python/cpython/tree/main/Lib/imaplib.py}

@geindex IMAP4; protocol
@geindex IMAP4_SSL; protocol
@geindex IMAP4_stream; protocol


__________________________________________________________________


This module defines three classes, @ref{3f2,,IMAP4}, @ref{3f3,,IMAP4_SSL} and
@ref{3f5,,IMAP4_stream}, which encapsulate a connection to an IMAP4 server and
implement a large subset of the IMAP4rev1 client protocol as defined in
@geindex RFC; RFC 2060
RFC 2060@footnote{https://tools.ietf.org/html/rfc2060.html}. It is backward compatible with IMAP4 (
@geindex RFC; RFC 1730
RFC 1730@footnote{https://tools.ietf.org/html/rfc1730.html}) servers, but
note that the @code{STATUS} command is not supported in IMAP4.

Three classes are provided by the @ref{93,,imaplib} module, @ref{3f2,,IMAP4} is the
base class:

@geindex IMAP4 (class in imaplib)
@anchor{library/imaplib imaplib IMAP4}@anchor{3f2}
@deffn {Class} imaplib.IMAP4 (host='@w{'}, port=IMAP4_PORT, timeout=None)

This class implements the actual IMAP4 protocol.  The connection is created and
protocol version (IMAP4 or IMAP4rev1) is determined when the instance is
initialized. If @emph{host} is not specified, @code{'@w{'}} (the local host) is used. If
@emph{port} is omitted, the standard IMAP4 port (143) is used. The optional @emph{timeout}
parameter specifies a timeout in seconds for the connection attempt.
If timeout is not given or is None, the global default socket timeout is used.

The @ref{3f2,,IMAP4} class supports the @ref{168,,with} statement.  When used
like this, the IMAP4 @code{LOGOUT} command is issued automatically when the
@code{with} statement exits.  E.g.:

@example
>>> from imaplib import IMAP4
>>> with IMAP4("domain.org") as M:
...     M.noop()
...
('OK', [b'Nothing Accomplished. d25if65hy903weo.87'])
@end example


Changed in version 3.5: Support for the @ref{168,,with} statement was added.



Changed in version 3.9: The optional @emph{timeout} parameter was added.

@end deffn

Three exceptions are defined as attributes of the @ref{3f2,,IMAP4} class:

@geindex IMAP4.error
@anchor{library/imaplib imaplib IMAP4 error}@anchor{31d7}
@deffn {Exception} IMAP4.error

Exception raised on any errors.  The reason for the exception is passed to the
constructor as a string.
@end deffn

@geindex IMAP4.abort
@anchor{library/imaplib imaplib IMAP4 abort}@anchor{31d8}
@deffn {Exception} IMAP4.abort

IMAP4 server errors cause this exception to be raised.  This is a sub-class of
@ref{31d7,,IMAP4.error}.  Note that closing the instance and instantiating a new one
will usually allow recovery from this exception.
@end deffn

@geindex IMAP4.readonly
@anchor{library/imaplib imaplib IMAP4 readonly}@anchor{31d9}
@deffn {Exception} IMAP4.readonly

This exception is raised when a writable mailbox has its status changed by the
server.  This is a sub-class of @ref{31d7,,IMAP4.error}.  Some other client now has
write permission, and the mailbox will need to be re-opened to re-obtain write
permission.
@end deffn

There’s also a subclass for secure connections:

@geindex IMAP4_SSL (class in imaplib)
@anchor{library/imaplib imaplib IMAP4_SSL}@anchor{3f3}
@deffn {Class} imaplib.IMAP4_SSL (host='@w{'}, port=IMAP4_SSL_PORT, keyfile=None, certfile=None, ssl_context=None, timeout=None)

This is a subclass derived from @ref{3f2,,IMAP4} that connects over an SSL
encrypted socket (to use this class you need a socket module that was compiled
with SSL support).  If @emph{host} is not specified, @code{'@w{'}} (the local host) is used.
If @emph{port} is omitted, the standard IMAP4-over-SSL port (993) is used.
@emph{ssl_context} is a @ref{2bf,,ssl.SSLContext} object which allows bundling
SSL configuration options, certificates and private keys into a single
(potentially long-lived) structure.  Please read @ref{2a39,,Security considerations} for
best practices.

@emph{keyfile} and @emph{certfile} are a legacy alternative to @emph{ssl_context} - they
can point to PEM-formatted private key and certificate chain files for
the SSL connection.  Note that the @emph{keyfile}/@emph{certfile} parameters are
mutually exclusive with @emph{ssl_context}, a @ref{17c,,ValueError} is raised
if @emph{keyfile}/@emph{certfile} is provided along with @emph{ssl_context}.

The optional @emph{timeout} parameter specifies a timeout in seconds for the
connection attempt. If timeout is not given or is None, the global default
socket timeout is used.


Changed in version 3.3: @emph{ssl_context} parameter was added.



Changed in version 3.4: The class now supports hostname check with
@ref{2b42,,ssl.SSLContext.check_hostname} and @emph{Server Name Indication} (see
@ref{2b4f,,ssl.HAS_SNI}).



Deprecated since version 3.6: @emph{keyfile} and @emph{certfile} are deprecated in favor of @emph{ssl_context}.
Please use @ref{cd9,,ssl.SSLContext.load_cert_chain()} instead, or let
@ref{b29,,ssl.create_default_context()} select the system’s trusted CA
certificates for you.



Changed in version 3.9: The optional @emph{timeout} parameter was added.

@end deffn

The second subclass allows for connections created by a child process:

@geindex IMAP4_stream (class in imaplib)
@anchor{library/imaplib imaplib IMAP4_stream}@anchor{3f5}
@deffn {Class} imaplib.IMAP4_stream (command)

This is a subclass derived from @ref{3f2,,IMAP4} that connects to the
@code{stdin/stdout} file descriptors created by passing @emph{command} to
@code{subprocess.Popen()}.
@end deffn

The following utility functions are defined:

@geindex Internaldate2tuple() (in module imaplib)
@anchor{library/imaplib imaplib Internaldate2tuple}@anchor{31da}
@deffn {Function} imaplib.Internaldate2tuple (datestr)

Parse an IMAP4 @code{INTERNALDATE} string and return corresponding local
time.  The return value is a @ref{834,,time.struct_time} tuple or
@code{None} if the string has wrong format.
@end deffn

@geindex Int2AP() (in module imaplib)
@anchor{library/imaplib imaplib Int2AP}@anchor{31db}
@deffn {Function} imaplib.Int2AP (num)

Converts an integer into a bytes representation using characters from the set
[@code{A} .. @code{P}].
@end deffn

@geindex ParseFlags() (in module imaplib)
@anchor{library/imaplib imaplib ParseFlags}@anchor{31dc}
@deffn {Function} imaplib.ParseFlags (flagstr)

Converts an IMAP4 @code{FLAGS} response to a tuple of individual flags.
@end deffn

@geindex Time2Internaldate() (in module imaplib)
@anchor{library/imaplib imaplib Time2Internaldate}@anchor{31dd}
@deffn {Function} imaplib.Time2Internaldate (date_time)

Convert @emph{date_time} to an IMAP4 @code{INTERNALDATE} representation.
The return value is a string in the form: @code{"DD-Mmm-YYYY HH:MM:SS
+HHMM"} (including double-quotes).  The @emph{date_time} argument can
be a number (int or float) representing seconds since epoch (as
returned by @ref{5fc,,time.time()}), a 9-tuple representing local time
an instance of @ref{834,,time.struct_time} (as returned by
@ref{1224,,time.localtime()}), an aware instance of
@ref{363,,datetime.datetime}, or a double-quoted string.  In the last
case, it is assumed to already be in the correct format.
@end deffn

Note that IMAP4 message numbers change as the mailbox changes; in particular,
after an @code{EXPUNGE} command performs deletions the remaining messages are
renumbered. So it is highly advisable to use UIDs instead, with the UID command.

At the end of the module, there is a test section that contains a more extensive
example of usage.


@subsubheading See also


Documents describing the protocol, sources for servers
implementing it, by the University of Washington’s IMAP Information Center
can all be found at (@strong{Source Code}) @indicateurl{https://github.com/uw-imap/imap} (@strong{Not Maintained}).


@menu
* IMAP4 Objects:: 
* IMAP4 Example:: 

@end menu

@node IMAP4 Objects,IMAP4 Example,,imaplib — IMAP4 protocol client
@anchor{library/imaplib id1}@anchor{31de}@anchor{library/imaplib imap4-objects}@anchor{31df}
@subsubsection IMAP4 Objects


All IMAP4rev1 commands are represented by methods of the same name, either
upper-case or lower-case.

All arguments to commands are converted to strings, except for @code{AUTHENTICATE},
and the last argument to @code{APPEND} which is passed as an IMAP4 literal.  If
necessary (the string contains IMAP4 protocol-sensitive characters and isn’t
enclosed with either parentheses or double quotes) each string is quoted.
However, the @emph{password} argument to the @code{LOGIN} command is always quoted. If
you want to avoid having an argument string quoted (eg: the @emph{flags} argument to
@code{STORE}) then enclose the string in parentheses (eg: @code{r'(\Deleted)'}).

Each command returns a tuple: @code{(type, [data, ...])} where @emph{type} is usually
@code{'OK'} or @code{'NO'}, and @emph{data} is either the text from the command response,
or mandated results from the command. Each @emph{data} is either a @code{bytes}, or a
tuple. If a tuple, then the first part is the header of the response, and the
second part contains the data (ie: ‘literal’ value).

The @emph{message_set} options to commands below is a string specifying one or more
messages to be acted upon.  It may be a simple message number (@code{'1'}), a range
of message numbers (@code{'2:4'}), or a group of non-contiguous ranges separated by
commas (@code{'1:3,6:9'}).  A range can contain an asterisk to indicate an infinite
upper bound (@code{'3:*'}).

An @ref{3f2,,IMAP4} instance has the following methods:

@geindex append() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 append}@anchor{31e0}
@deffn {Method} IMAP4.append (mailbox, flags, date_time, message)

Append @emph{message} to named mailbox.
@end deffn

@geindex authenticate() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 authenticate}@anchor{31e1}
@deffn {Method} IMAP4.authenticate (mechanism, authobject)

Authenticate command — requires response processing.

@emph{mechanism} specifies which authentication mechanism is to be used - it should
appear in the instance variable @code{capabilities} in the form @code{AUTH=mechanism}.

@emph{authobject} must be a callable object:

@example
data = authobject(response)
@end example

It will be called to process server continuation responses; the @emph{response}
argument it is passed will be @code{bytes}.  It should return @code{bytes} @emph{data}
that will be base64 encoded and sent to the server.  It should return
@code{None} if the client abort response @code{*} should be sent instead.


Changed in version 3.5: string usernames and passwords are now encoded to @code{utf-8} instead of
being limited to ASCII.

@end deffn

@geindex check() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 check}@anchor{31e2}
@deffn {Method} IMAP4.check ()

Checkpoint mailbox on server.
@end deffn

@geindex close() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 close}@anchor{3f7}
@deffn {Method} IMAP4.close ()

Close currently selected mailbox. Deleted messages are removed from writable
mailbox. This is the recommended command before @code{LOGOUT}.
@end deffn

@geindex copy() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 copy}@anchor{31e3}
@deffn {Method} IMAP4.copy (message_set, new_mailbox)

Copy @emph{message_set} messages onto end of @emph{new_mailbox}.
@end deffn

@geindex create() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 create}@anchor{31e4}
@deffn {Method} IMAP4.create (mailbox)

Create new mailbox named @emph{mailbox}.
@end deffn

@geindex delete() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 delete}@anchor{31e5}
@deffn {Method} IMAP4.delete (mailbox)

Delete old mailbox named @emph{mailbox}.
@end deffn

@geindex deleteacl() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 deleteacl}@anchor{31e6}
@deffn {Method} IMAP4.deleteacl (mailbox, who)

Delete the ACLs (remove any rights) set for who on mailbox.
@end deffn

@geindex enable() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 enable}@anchor{96c}
@deffn {Method} IMAP4.enable (capability)

Enable @emph{capability} (see 
@geindex RFC; RFC 5161
RFC 5161@footnote{https://tools.ietf.org/html/rfc5161.html}).  Most capabilities do not need to be
enabled.  Currently only the @code{UTF8=ACCEPT} capability is supported
(see 
@geindex RFC; RFC 6855
RFC 6855@footnote{https://tools.ietf.org/html/rfc6855.html}).


New in version 3.5: The @ref{96c,,enable()} method itself, and 
@geindex RFC; RFC 6855
RFC 6855@footnote{https://tools.ietf.org/html/rfc6855.html} support.

@end deffn

@geindex expunge() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 expunge}@anchor{31e7}
@deffn {Method} IMAP4.expunge ()

Permanently remove deleted items from selected mailbox. Generates an @code{EXPUNGE}
response for each deleted message. Returned data contains a list of @code{EXPUNGE}
message numbers in order received.
@end deffn

@geindex fetch() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 fetch}@anchor{31e8}
@deffn {Method} IMAP4.fetch (message_set, message_parts)

Fetch (parts of) messages.  @emph{message_parts} should be a string of message part
names enclosed within parentheses, eg: @code{"(UID BODY[TEXT])"}.  Returned data
are tuples of message part envelope and data.
@end deffn

@geindex getacl() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 getacl}@anchor{31e9}
@deffn {Method} IMAP4.getacl (mailbox)

Get the @code{ACL}s for @emph{mailbox}. The method is non-standard, but is supported
by the @code{Cyrus} server.
@end deffn

@geindex getannotation() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 getannotation}@anchor{31ea}
@deffn {Method} IMAP4.getannotation (mailbox, entry, attribute)

Retrieve the specified @code{ANNOTATION}s for @emph{mailbox}. The method is
non-standard, but is supported by the @code{Cyrus} server.
@end deffn

@geindex getquota() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 getquota}@anchor{31eb}
@deffn {Method} IMAP4.getquota (root)

Get the @code{quota} @emph{root}’s resource usage and limits. This method is part of the
IMAP4 QUOTA extension defined in rfc2087.
@end deffn

@geindex getquotaroot() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 getquotaroot}@anchor{31ec}
@deffn {Method} IMAP4.getquotaroot (mailbox)

Get the list of @code{quota} @code{roots} for the named @emph{mailbox}. This method is part
of the IMAP4 QUOTA extension defined in rfc2087.
@end deffn

@geindex list() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 list}@anchor{31ed}
@deffn {Method} IMAP4.list ([directory[, pattern]])

List mailbox names in @emph{directory} matching @emph{pattern}.  @emph{directory} defaults to
the top-level mail folder, and @emph{pattern} defaults to match anything.  Returned
data contains a list of @code{LIST} responses.
@end deffn

@geindex login() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 login}@anchor{31ee}
@deffn {Method} IMAP4.login (user, password)

Identify the client using a plaintext password. The @emph{password} will be quoted.
@end deffn

@geindex login_cram_md5() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 login_cram_md5}@anchor{31ef}
@deffn {Method} IMAP4.login_cram_md5 (user, password)

Force use of @code{CRAM-MD5} authentication when identifying the client to protect
the password.  Will only work if the server @code{CAPABILITY} response includes the
phrase @code{AUTH=CRAM-MD5}.
@end deffn

@geindex logout() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 logout}@anchor{31f0}
@deffn {Method} IMAP4.logout ()

Shutdown connection to server. Returns server @code{BYE} response.


Changed in version 3.8: The method no longer ignores silently arbitrary exceptions.

@end deffn

@geindex lsub() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 lsub}@anchor{31f1}
@deffn {Method} IMAP4.lsub (directory='""', pattern='*')

List subscribed mailbox names in directory matching pattern. @emph{directory}
defaults to the top level directory and @emph{pattern} defaults to match any mailbox.
Returned data are tuples of message part envelope and data.
@end deffn

@geindex myrights() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 myrights}@anchor{31f2}
@deffn {Method} IMAP4.myrights (mailbox)

Show my ACLs for a mailbox (i.e. the rights that I have on mailbox).
@end deffn

@geindex namespace() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 namespace}@anchor{31f3}
@deffn {Method} IMAP4.namespace ()

Returns IMAP namespaces as defined in 
@geindex RFC; RFC 2342
RFC 2342@footnote{https://tools.ietf.org/html/rfc2342.html}.
@end deffn

@geindex noop() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 noop}@anchor{31f4}
@deffn {Method} IMAP4.noop ()

Send @code{NOOP} to server.
@end deffn

@geindex open() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 open}@anchor{3f4}
@deffn {Method} IMAP4.open (host, port, timeout=None)

Opens socket to @emph{port} at @emph{host}. The optional @emph{timeout} parameter
specifies a timeout in seconds for the connection attempt.
If timeout is not given or is None, the global default socket timeout
is used. Also note that if the @emph{timeout} parameter is set to be zero,
it will raise a @ref{17c,,ValueError} to reject creating a non-blocking socket.
This method is implicitly called by the @ref{3f2,,IMAP4} constructor.
The connection objects established by this method will be used in
the @ref{31f5,,IMAP4.read()}, @ref{31f6,,IMAP4.readline()}, @ref{31f7,,IMAP4.send()},
and @ref{31f8,,IMAP4.shutdown()} methods. You may override this method.

Raises an @ref{1130,,auditing event} @code{imaplib.open} with arguments @code{self}, @code{host}, @code{port}.


Changed in version 3.9: The @emph{timeout} parameter was added.

@end deffn

@geindex partial() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 partial}@anchor{31f9}
@deffn {Method} IMAP4.partial (message_num, message_part, start, length)

Fetch truncated part of a message. Returned data is a tuple of message part
envelope and data.
@end deffn

@geindex proxyauth() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 proxyauth}@anchor{31fa}
@deffn {Method} IMAP4.proxyauth (user)

Assume authentication as @emph{user}. Allows an authorised administrator to proxy
into any user’s mailbox.
@end deffn

@geindex read() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 read}@anchor{31f5}
@deffn {Method} IMAP4.read (size)

Reads @emph{size} bytes from the remote server. You may override this method.
@end deffn

@geindex readline() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 readline}@anchor{31f6}
@deffn {Method} IMAP4.readline ()

Reads one line from the remote server. You may override this method.
@end deffn

@geindex recent() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 recent}@anchor{31fb}
@deffn {Method} IMAP4.recent ()

Prompt server for an update. Returned data is @code{None} if no new messages, else
value of @code{RECENT} response.
@end deffn

@geindex rename() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 rename}@anchor{31fc}
@deffn {Method} IMAP4.rename (oldmailbox, newmailbox)

Rename mailbox named @emph{oldmailbox} to @emph{newmailbox}.
@end deffn

@geindex response() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 response}@anchor{31fd}
@deffn {Method} IMAP4.response (code)

Return data for response @emph{code} if received, or @code{None}. Returns the given
code, instead of the usual type.
@end deffn

@geindex search() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 search}@anchor{31fe}
@deffn {Method} IMAP4.search (charset, criterion[, ...])

Search mailbox for matching messages.  @emph{charset} may be @code{None}, in which case
no @code{CHARSET} will be specified in the request to the server.  The IMAP
protocol requires that at least one criterion be specified; an exception will be
raised when the server returns an error.  @emph{charset} must be @code{None} if
the @code{UTF8=ACCEPT} capability was enabled using the @ref{96c,,enable()}
command.

Example:

@example
# M is a connected IMAP4 instance...
typ, msgnums = M.search(None, 'FROM', '"LDJ"')

# or:
typ, msgnums = M.search(None, '(FROM "LDJ")')
@end example
@end deffn

@geindex select() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 select}@anchor{31ff}
@deffn {Method} IMAP4.select (mailbox='INBOX', readonly=False)

Select a mailbox. Returned data is the count of messages in @emph{mailbox}
(@code{EXISTS} response).  The default @emph{mailbox} is @code{'INBOX'}.  If the @emph{readonly}
flag is set, modifications to the mailbox are not allowed.
@end deffn

@geindex send() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 send}@anchor{31f7}
@deffn {Method} IMAP4.send (data)

Sends @code{data} to the remote server. You may override this method.

Raises an @ref{1130,,auditing event} @code{imaplib.send} with arguments @code{self}, @code{data}.
@end deffn

@geindex setacl() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 setacl}@anchor{3200}
@deffn {Method} IMAP4.setacl (mailbox, who, what)

Set an @code{ACL} for @emph{mailbox}. The method is non-standard, but is supported by
the @code{Cyrus} server.
@end deffn

@geindex setannotation() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 setannotation}@anchor{3201}
@deffn {Method} IMAP4.setannotation (mailbox, entry, attribute[, ...])

Set @code{ANNOTATION}s for @emph{mailbox}. The method is non-standard, but is
supported by the @code{Cyrus} server.
@end deffn

@geindex setquota() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 setquota}@anchor{3202}
@deffn {Method} IMAP4.setquota (root, limits)

Set the @code{quota} @emph{root}’s resource @emph{limits}. This method is part of the IMAP4
QUOTA extension defined in rfc2087.
@end deffn

@geindex shutdown() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 shutdown}@anchor{31f8}
@deffn {Method} IMAP4.shutdown ()

Close connection established in @code{open}.  This method is implicitly
called by @ref{31f0,,IMAP4.logout()}.  You may override this method.
@end deffn

@geindex socket() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 socket}@anchor{3203}
@deffn {Method} IMAP4.socket ()

Returns socket instance used to connect to server.
@end deffn

@geindex sort() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 sort}@anchor{3204}
@deffn {Method} IMAP4.sort (sort_criteria, charset, search_criterion[, ...])

The @code{sort} command is a variant of @code{search} with sorting semantics for the
results.  Returned data contains a space separated list of matching message
numbers.

Sort has two arguments before the @emph{search_criterion} argument(s); a
parenthesized list of @emph{sort_criteria}, and the searching @emph{charset}.  Note that
unlike @code{search}, the searching @emph{charset} argument is mandatory.  There is also
a @code{uid sort} command which corresponds to @code{sort} the way that @code{uid search}
corresponds to @code{search}.  The @code{sort} command first searches the mailbox for
messages that match the given searching criteria using the charset argument for
the interpretation of strings in the searching criteria.  It then returns the
numbers of matching messages.

This is an @code{IMAP4rev1} extension command.
@end deffn

@geindex starttls() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 starttls}@anchor{dca}
@deffn {Method} IMAP4.starttls (ssl_context=None)

Send a @code{STARTTLS} command.  The @emph{ssl_context} argument is optional
and should be a @ref{2bf,,ssl.SSLContext} object.  This will enable
encryption on the IMAP connection.  Please read @ref{2a39,,Security considerations} for
best practices.


New in version 3.2.



Changed in version 3.4: The method now supports hostname check with
@ref{2b42,,ssl.SSLContext.check_hostname} and @emph{Server Name Indication} (see
@ref{2b4f,,ssl.HAS_SNI}).

@end deffn

@geindex status() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 status}@anchor{3205}
@deffn {Method} IMAP4.status (mailbox, names)

Request named status conditions for @emph{mailbox}.
@end deffn

@geindex store() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 store}@anchor{3206}
@deffn {Method} IMAP4.store (message_set, command, flag_list)

Alters flag dispositions for messages in mailbox.  @emph{command} is specified by
section 6.4.6 of 
@geindex RFC; RFC 2060
RFC 2060@footnote{https://tools.ietf.org/html/rfc2060.html} as being one of “FLAGS”, “+FLAGS”, or “-FLAGS”,
optionally with a suffix of “.SILENT”.

For example, to set the delete flag on all messages:

@example
typ, data = M.search(None, 'ALL')
for num in data[0].split():
   M.store(num, '+FLAGS', '\\Deleted')
M.expunge()
@end example

@cartouche
@quotation Note 
Creating flags containing ‘]’ (for example: “[test]”) violates
@geindex RFC; RFC 3501
RFC 3501@footnote{https://tools.ietf.org/html/rfc3501.html} (the IMAP protocol).  However, imaplib has historically
allowed creation of such tags, and popular IMAP servers, such as Gmail,
accept and produce such flags.  There are non-Python programs which also
create such tags.  Although it is an RFC violation and IMAP clients and
servers are supposed to be strict, imaplib nonetheless continues to allow
such tags to be created for backward compatibility reasons, and as of
Python 3.6, handles them if they are sent from the server, since this
improves real-world compatibility.
@end quotation
@end cartouche
@end deffn

@geindex subscribe() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 subscribe}@anchor{3207}
@deffn {Method} IMAP4.subscribe (mailbox)

Subscribe to new mailbox.
@end deffn

@geindex thread() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 thread}@anchor{3208}
@deffn {Method} IMAP4.thread (threading_algorithm, charset, search_criterion[, ...])

The @code{thread} command is a variant of @code{search} with threading semantics for
the results.  Returned data contains a space separated list of thread members.

Thread members consist of zero or more messages numbers, delimited by spaces,
indicating successive parent and child.

Thread has two arguments before the @emph{search_criterion} argument(s); a
@emph{threading_algorithm}, and the searching @emph{charset}.  Note that unlike
@code{search}, the searching @emph{charset} argument is mandatory.  There is also a
@code{uid thread} command which corresponds to @code{thread} the way that @code{uid
search} corresponds to @code{search}.  The @code{thread} command first searches the
mailbox for messages that match the given searching criteria using the charset
argument for the interpretation of strings in the searching criteria. It then
returns the matching messages threaded according to the specified threading
algorithm.

This is an @code{IMAP4rev1} extension command.
@end deffn

@geindex uid() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 uid}@anchor{3209}
@deffn {Method} IMAP4.uid (command, arg[, ...])

Execute command args with messages identified by UID, rather than message
number.  Returns response appropriate to command.  At least one argument must be
supplied; if none are provided, the server will return an error and an exception
will be raised.
@end deffn

@geindex unsubscribe() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 unsubscribe}@anchor{320a}
@deffn {Method} IMAP4.unsubscribe (mailbox)

Unsubscribe from old mailbox.
@end deffn

@geindex unselect() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 unselect}@anchor{3f6}
@deffn {Method} IMAP4.unselect ()

@ref{3f6,,imaplib.IMAP4.unselect()} frees server’s resources associated with the
selected mailbox and returns the server to the authenticated
state. This command performs the same actions as @ref{3f7,,imaplib.IMAP4.close()}, except
that no messages are permanently removed from the currently
selected mailbox.


New in version 3.9.

@end deffn

@geindex xatom() (imaplib.IMAP4 method)
@anchor{library/imaplib imaplib IMAP4 xatom}@anchor{320b}
@deffn {Method} IMAP4.xatom (name[, ...])

Allow simple extension commands notified by server in @code{CAPABILITY} response.
@end deffn

The following attributes are defined on instances of @ref{3f2,,IMAP4}:

@geindex PROTOCOL_VERSION (imaplib.IMAP4 attribute)
@anchor{library/imaplib imaplib IMAP4 PROTOCOL_VERSION}@anchor{320c}
@deffn {Attribute} IMAP4.PROTOCOL_VERSION

The most recent supported protocol in the @code{CAPABILITY} response from the
server.
@end deffn

@geindex debug (imaplib.IMAP4 attribute)
@anchor{library/imaplib imaplib IMAP4 debug}@anchor{320d}
@deffn {Attribute} IMAP4.debug

Integer value to control debugging output.  The initialize value is taken from
the module variable @code{Debug}.  Values greater than three trace each command.
@end deffn

@geindex utf8_enabled (imaplib.IMAP4 attribute)
@anchor{library/imaplib imaplib IMAP4 utf8_enabled}@anchor{96d}
@deffn {Attribute} IMAP4.utf8_enabled

Boolean value that is normally @code{False}, but is set to @code{True} if an
@ref{96c,,enable()} command is successfully issued for the @code{UTF8=ACCEPT}
capability.


New in version 3.5.

@end deffn

@node IMAP4 Example,,IMAP4 Objects,imaplib — IMAP4 protocol client
@anchor{library/imaplib id2}@anchor{320e}@anchor{library/imaplib imap4-example}@anchor{320f}
@subsubsection IMAP4 Example


Here is a minimal example (without error checking) that opens a mailbox and
retrieves and prints all messages:

@example
import getpass, imaplib

M = imaplib.IMAP4()
M.login(getpass.getuser(), getpass.getpass())
M.select()
typ, data = M.search(None, 'ALL')
for num in data[0].split():
    typ, data = M.fetch(num, '(RFC822)')
    print('Message %s\n%s\n' % (num, data[0][1]))
M.close()
M.logout()
@end example

@node nntplib — NNTP protocol client,smtplib — SMTP protocol client,imaplib — IMAP4 protocol client,Internet Protocols and Support
@anchor{library/nntplib doc}@anchor{3210}@anchor{library/nntplib module-nntplib}@anchor{bc}@anchor{library/nntplib nntplib-nntp-protocol-client}@anchor{3211}
@subsection @code{nntplib} — NNTP protocol client


@geindex module; nntplib

@strong{Source code:} Lib/nntplib.py@footnote{https://github.com/python/cpython/tree/main/Lib/nntplib.py}

@geindex NNTP; protocol
@geindex Network News Transfer Protocol


__________________________________________________________________


This module defines the class @ref{409,,NNTP} which implements the client side of
the Network News Transfer Protocol.  It can be used to implement a news reader
or poster, or automated news processors.  It is compatible with 
@geindex RFC; RFC 3977
RFC 3977@footnote{https://tools.ietf.org/html/rfc3977.html}
as well as the older 
@geindex RFC; RFC 977
RFC 977@footnote{https://tools.ietf.org/html/rfc977.html} and 
@geindex RFC; RFC 2980
RFC 2980@footnote{https://tools.ietf.org/html/rfc2980.html}.

Here are two small examples of how it can be used.  To list some statistics
about a newsgroup and print the subjects of the last 10 articles:

@example
>>> s = nntplib.NNTP('news.gmane.io')
>>> resp, count, first, last, name = s.group('gmane.comp.python.committers')
>>> print('Group', name, 'has', count, 'articles, range', first, 'to', last)
Group gmane.comp.python.committers has 1096 articles, range 1 to 1096
>>> resp, overviews = s.over((last - 9, last))
>>> for id, over in overviews:
...     print(id, nntplib.decode_header(over['subject']))
...
1087 Re: Commit privileges for Łukasz Langa
1088 Re: 3.2 alpha 2 freeze
1089 Re: 3.2 alpha 2 freeze
1090 Re: Commit privileges for Łukasz Langa
1091 Re: Commit privileges for Łukasz Langa
1092 Updated ssh key
1093 Re: Updated ssh key
1094 Re: Updated ssh key
1095 Hello fellow committers!
1096 Re: Hello fellow committers!
>>> s.quit()
'205 Bye!'
@end example

To post an article from a binary file (this assumes that the article has valid
headers, and that you have right to post on the particular newsgroup):

@example
>>> s = nntplib.NNTP('news.gmane.io')
>>> f = open('article.txt', 'rb')
>>> s.post(f)
'240 Article posted successfully.'
>>> s.quit()
'205 Bye!'
@end example

The module itself defines the following classes:

@geindex NNTP (class in nntplib)
@anchor{library/nntplib nntplib NNTP}@anchor{409}
@deffn {Class} nntplib.NNTP (host, port=119, user=None, password=None, readermode=None, usenetrc=False[, timeout])

Return a new @ref{409,,NNTP} object, representing a connection
to the NNTP server running on host @emph{host}, listening at port @emph{port}.
An optional @emph{timeout} can be specified for the socket connection.
If the optional @emph{user} and @emph{password} are provided, or if suitable
credentials are present in @code{/.netrc} and the optional flag @emph{usenetrc}
is true, the @code{AUTHINFO USER} and @code{AUTHINFO PASS} commands are used
to identify and authenticate the user to the server.  If the optional
flag @emph{readermode} is true, then a @code{mode reader} command is sent before
authentication is performed.  Reader mode is sometimes necessary if you are
connecting to an NNTP server on the local machine and intend to call
reader-specific commands, such as @code{group}.  If you get unexpected
@ref{3212,,NNTPPermanentError}s, you might need to set @emph{readermode}.
The @ref{409,,NNTP} class supports the @ref{168,,with} statement to
unconditionally consume @ref{2a1,,OSError} exceptions and to close the NNTP
connection when done, e.g.:

@example
>>> from nntplib import NNTP
>>> with NNTP('news.gmane.io') as n:
...     n.group('gmane.comp.python.committers')
... 
('211 1755 1 1755 gmane.comp.python.committers', 1755, 1, 1755, 'gmane.comp.python.committers')
>>>
@end example

Raises an @ref{1130,,auditing event} @code{nntplib.connect} with arguments @code{self}, @code{host}, @code{port}.


All commands will raise an @ref{1130,,auditing event}
@code{nntplib.putline} with arguments @code{self} and @code{line},
where @code{line} is the bytes about to be sent to the remote host.



Changed in version 3.2: @emph{usenetrc} is now @code{False} by default.



Changed in version 3.3: Support for the @ref{168,,with} statement was added.



Changed in version 3.9: If the @emph{timeout} parameter is set to be zero, it will raise a
@ref{17c,,ValueError} to prevent the creation of a non-blocking socket.

@end deffn

@geindex NNTP_SSL (class in nntplib)
@anchor{library/nntplib nntplib NNTP_SSL}@anchor{40a}
@deffn {Class} nntplib.NNTP_SSL (host, port=563, user=None, password=None, ssl_context=None, readermode=None, usenetrc=False[, timeout])

Return a new @ref{40a,,NNTP_SSL} object, representing an encrypted
connection to the NNTP server running on host @emph{host}, listening at
port @emph{port}.  @ref{40a,,NNTP_SSL} objects have the same methods as
@ref{409,,NNTP} objects.  If @emph{port} is omitted, port 563 (NNTPS) is used.
@emph{ssl_context} is also optional, and is a @ref{2bf,,SSLContext} object.
Please read @ref{2a39,,Security considerations} for best practices.
All other parameters behave the same as for @ref{409,,NNTP}.

Note that SSL-on-563 is discouraged per 
@geindex RFC; RFC 4642
RFC 4642@footnote{https://tools.ietf.org/html/rfc4642.html}, in favor of
STARTTLS as described below.  However, some servers only support the
former.

Raises an @ref{1130,,auditing event} @code{nntplib.connect} with arguments @code{self}, @code{host}, @code{port}.


All commands will raise an @ref{1130,,auditing event}
@code{nntplib.putline} with arguments @code{self} and @code{line},
where @code{line} is the bytes about to be sent to the remote host.



New in version 3.2.



Changed in version 3.4: The class now supports hostname check with
@ref{2b42,,ssl.SSLContext.check_hostname} and @emph{Server Name Indication} (see
@ref{2b4f,,ssl.HAS_SNI}).



Changed in version 3.9: If the @emph{timeout} parameter is set to be zero, it will raise a
@ref{17c,,ValueError} to prevent the creation of a non-blocking socket.

@end deffn

@geindex NNTPError
@anchor{library/nntplib nntplib NNTPError}@anchor{3213}
@deffn {Exception} nntplib.NNTPError

Derived from the standard exception @ref{4d8,,Exception}, this is the base
class for all exceptions raised by the @ref{bc,,nntplib} module.  Instances
of this class have the following attribute:

@geindex response (nntplib.NNTPError attribute)
@anchor{library/nntplib nntplib NNTPError response}@anchor{3214}
@deffn {Attribute} response

The response of the server if available, as a @ref{2ef,,str} object.
@end deffn
@end deffn

@geindex NNTPReplyError
@anchor{library/nntplib nntplib NNTPReplyError}@anchor{3215}
@deffn {Exception} nntplib.NNTPReplyError

Exception raised when an unexpected reply is received from the server.
@end deffn

@geindex NNTPTemporaryError
@anchor{library/nntplib nntplib NNTPTemporaryError}@anchor{3216}
@deffn {Exception} nntplib.NNTPTemporaryError

Exception raised when a response code in the range 400–499 is received.
@end deffn

@geindex NNTPPermanentError
@anchor{library/nntplib nntplib NNTPPermanentError}@anchor{3212}
@deffn {Exception} nntplib.NNTPPermanentError

Exception raised when a response code in the range 500–599 is received.
@end deffn

@geindex NNTPProtocolError
@anchor{library/nntplib nntplib NNTPProtocolError}@anchor{3217}
@deffn {Exception} nntplib.NNTPProtocolError

Exception raised when a reply is received from the server that does not begin
with a digit in the range 1–5.
@end deffn

@geindex NNTPDataError
@anchor{library/nntplib nntplib NNTPDataError}@anchor{3218}
@deffn {Exception} nntplib.NNTPDataError

Exception raised when there is some error in the response data.
@end deffn

@menu
* NNTP Objects:: 
* Utility functions: Utility functions<2>. 

@end menu

@node NNTP Objects,Utility functions<2>,,nntplib — NNTP protocol client
@anchor{library/nntplib id1}@anchor{3219}@anchor{library/nntplib nntp-objects}@anchor{321a}
@subsubsection NNTP Objects


When connected, @ref{409,,NNTP} and @ref{40a,,NNTP_SSL} objects support the
following methods and attributes.

@menu
* Attributes:: 
* Methods: Methods<3>. 

@end menu

@node Attributes,Methods<3>,,NNTP Objects
@anchor{library/nntplib attributes}@anchor{321b}
@subsubsection Attributes


@geindex nntp_version (nntplib.NNTP attribute)
@anchor{library/nntplib nntplib NNTP nntp_version}@anchor{321c}
@deffn {Attribute} NNTP.nntp_version

An integer representing the version of the NNTP protocol supported by the
server.  In practice, this should be @code{2} for servers advertising
@geindex RFC; RFC 3977
RFC 3977@footnote{https://tools.ietf.org/html/rfc3977.html} compliance and @code{1} for others.


New in version 3.2.

@end deffn

@geindex nntp_implementation (nntplib.NNTP attribute)
@anchor{library/nntplib nntplib NNTP nntp_implementation}@anchor{321d}
@deffn {Attribute} NNTP.nntp_implementation

A string describing the software name and version of the NNTP server,
or @ref{182,,None} if not advertised by the server.


New in version 3.2.

@end deffn

@node Methods<3>,,Attributes,NNTP Objects
@anchor{library/nntplib methods}@anchor{321e}
@subsubsection Methods


The @emph{response} that is returned as the first item in the return tuple of almost
all methods is the server’s response: a string beginning with a three-digit
code.  If the server’s response indicates an error, the method raises one of
the above exceptions.

Many of the following methods take an optional keyword-only argument @emph{file}.
When the @emph{file} argument is supplied, it must be either a @ref{d68,,file object}
opened for binary writing, or the name of an on-disk file to be written to.
The method will then write any data returned by the server (except for the
response line and the terminating dot) to the file; any list of lines,
tuples or objects that the method normally returns will be empty.


Changed in version 3.2: Many of the following methods have been reworked and fixed, which makes
them incompatible with their 3.1 counterparts.


@geindex quit() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP quit}@anchor{321f}
@deffn {Method} NNTP.quit ()

Send a @code{QUIT} command and close the connection.  Once this method has been
called, no other methods of the NNTP object should be called.
@end deffn

@geindex getwelcome() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP getwelcome}@anchor{3220}
@deffn {Method} NNTP.getwelcome ()

Return the welcome message sent by the server in reply to the initial
connection.  (This message sometimes contains disclaimers or help information
that may be relevant to the user.)
@end deffn

@geindex getcapabilities() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP getcapabilities}@anchor{3221}
@deffn {Method} NNTP.getcapabilities ()

Return the 
@geindex RFC; RFC 3977
RFC 3977@footnote{https://tools.ietf.org/html/rfc3977.html} capabilities advertised by the server, as a
@ref{3b0,,dict} instance mapping capability names to (possibly empty) lists
of values. On legacy servers which don’t understand the @code{CAPABILITIES}
command, an empty dictionary is returned instead.

@example
>>> s = NNTP('news.gmane.io')
>>> 'POST' in s.getcapabilities()
True
@end example


New in version 3.2.

@end deffn

@geindex login() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP login}@anchor{3222}
@deffn {Method} NNTP.login (user=None, password=None, usenetrc=True)

Send @code{AUTHINFO} commands with the user name and password.  If @emph{user}
and @emph{password} are @code{None} and @emph{usenetrc} is true, credentials from
@code{~/.netrc} will be used if possible.

Unless intentionally delayed, login is normally performed during the
@ref{409,,NNTP} object initialization and separately calling this function
is unnecessary.  To force authentication to be delayed, you must not set
@emph{user} or @emph{password} when creating the object, and must set @emph{usenetrc} to
False.


New in version 3.2.

@end deffn

@geindex starttls() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP starttls}@anchor{dc6}
@deffn {Method} NNTP.starttls (context=None)

Send a @code{STARTTLS} command.  This will enable encryption on the NNTP
connection.  The @emph{context} argument is optional and should be a
@ref{2bf,,ssl.SSLContext} object.  Please read @ref{2a39,,Security considerations} for best
practices.

Note that this may not be done after authentication information has
been transmitted, and authentication occurs by default if possible during a
@ref{409,,NNTP} object initialization.  See @ref{3222,,NNTP.login()} for information
on suppressing this behavior.


New in version 3.2.



Changed in version 3.4: The method now supports hostname check with
@ref{2b42,,ssl.SSLContext.check_hostname} and @emph{Server Name Indication} (see
@ref{2b4f,,ssl.HAS_SNI}).

@end deffn

@geindex newgroups() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP newgroups}@anchor{3223}
@deffn {Method} NNTP.newgroups (date, *, file=None)

Send a @code{NEWGROUPS} command.  The @emph{date} argument should be a
@ref{3e2,,datetime.date} or @ref{363,,datetime.datetime} object.
Return a pair @code{(response, groups)} where @emph{groups} is a list representing
the groups that are new since the given @emph{date}. If @emph{file} is supplied,
though, then @emph{groups} will be empty.

@example
>>> from datetime import date, timedelta
>>> resp, groups = s.newgroups(date.today() - timedelta(days=3))
>>> len(groups) 
85
>>> groups[0] 
GroupInfo(group='gmane.network.tor.devel', last='4', first='1', flag='m')
@end example
@end deffn

@geindex newnews() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP newnews}@anchor{3224}
@deffn {Method} NNTP.newnews (group, date, *, file=None)

Send a @code{NEWNEWS} command.  Here, @emph{group} is a group name or @code{'*'}, and
@emph{date} has the same meaning as for @ref{3223,,newgroups()}.  Return a pair
@code{(response, articles)} where @emph{articles} is a list of message ids.

This command is frequently disabled by NNTP server administrators.
@end deffn

@geindex list() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP list}@anchor{3225}
@deffn {Method} NNTP.list (group_pattern=None, *, file=None)

Send a @code{LIST} or @code{LIST ACTIVE} command.  Return a pair
@code{(response, list)} where @emph{list} is a list of tuples representing all
the groups available from this NNTP server, optionally matching the
pattern string @emph{group_pattern}.  Each tuple has the form
@code{(group, last, first, flag)}, where @emph{group} is a group name, @emph{last}
and @emph{first} are the last and first article numbers, and @emph{flag} usually
takes one of these values:


@itemize *

@item 
@code{y}: Local postings and articles from peers are allowed.

@item 
@code{m}: The group is moderated and all postings must be approved.

@item 
@code{n}: No local postings are allowed, only articles from peers.

@item 
@code{j}: Articles from peers are filed in the junk group instead.

@item 
@code{x}: No local postings, and articles from peers are ignored.

@item 
@code{=foo.bar}: Articles are filed in the @code{foo.bar} group instead.
@end itemize

If @emph{flag} has another value, then the status of the newsgroup should be
considered unknown.

This command can return very large results, especially if @emph{group_pattern}
is not specified.  It is best to cache the results offline unless you
really need to refresh them.


Changed in version 3.2: @emph{group_pattern} was added.

@end deffn

@geindex descriptions() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP descriptions}@anchor{446}
@deffn {Method} NNTP.descriptions (grouppattern)

Send a @code{LIST NEWSGROUPS} command, where @emph{grouppattern} is a wildmat string as
specified in 
@geindex RFC; RFC 3977
RFC 3977@footnote{https://tools.ietf.org/html/rfc3977.html} (it’s essentially the same as DOS or UNIX shell wildcard
strings).  Return a pair @code{(response, descriptions)}, where @emph{descriptions}
is a dictionary mapping group names to textual descriptions.

@example
>>> resp, descs = s.descriptions('gmane.comp.python.*')
>>> len(descs) 
295
>>> descs.popitem() 
('gmane.comp.python.bio.general', 'BioPython discussion list (Moderated)')
@end example
@end deffn

@geindex description() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP description}@anchor{447}
@deffn {Method} NNTP.description (group)

Get a description for a single group @emph{group}.  If more than one group matches
(if ‘group’ is a real wildmat string), return the first match.   If no group
matches, return an empty string.

This elides the response code from the server.  If the response code is needed,
use @ref{446,,descriptions()}.
@end deffn

@geindex group() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP group}@anchor{3226}
@deffn {Method} NNTP.group (name)

Send a @code{GROUP} command, where @emph{name} is the group name.  The group is
selected as the current group, if it exists.  Return a tuple
@code{(response, count, first, last, name)} where @emph{count} is the (estimated)
number of articles in the group, @emph{first} is the first article number in
the group, @emph{last} is the last article number in the group, and @emph{name}
is the group name.
@end deffn

@geindex over() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP over}@anchor{3227}
@deffn {Method} NNTP.over (message_spec, *, file=None)

Send an @code{OVER} command, or an @code{XOVER} command on legacy servers.
@emph{message_spec} can be either a string representing a message id, or
a @code{(first, last)} tuple of numbers indicating a range of articles in
the current group, or a @code{(first, None)} tuple indicating a range of
articles starting from @emph{first} to the last article in the current group,
or @ref{182,,None} to select the current article in the current group.

Return a pair @code{(response, overviews)}.  @emph{overviews} is a list of
@code{(article_number, overview)} tuples, one for each article selected
by @emph{message_spec}.  Each @emph{overview} is a dictionary with the same number
of items, but this number depends on the server.  These items are either
message headers (the key is then the lower-cased header name) or metadata
items (the key is then the metadata name prepended with @code{":"}).  The
following items are guaranteed to be present by the NNTP specification:


@itemize *

@item 
the @code{subject}, @code{from}, @code{date}, @code{message-id} and @code{references}
headers

@item 
the @code{:bytes} metadata: the number of bytes in the entire raw article
(including headers and body)

@item 
the @code{:lines} metadata: the number of lines in the article body
@end itemize

The value of each item is either a string, or @ref{182,,None} if not present.

It is advisable to use the @ref{3228,,decode_header()} function on header
values when they may contain non-ASCII characters:

@example
>>> _, _, first, last, _ = s.group('gmane.comp.python.devel')
>>> resp, overviews = s.over((last, last))
>>> art_num, over = overviews[0]
>>> art_num
117216
>>> list(over.keys())
['xref', 'from', ':lines', ':bytes', 'references', 'date', 'message-id', 'subject']
>>> over['from']
'=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?= <martin@@v.loewis.de>'
>>> nntplib.decode_header(over['from'])
'"Martin v. Löwis" <martin@@v.loewis.de>'
@end example


New in version 3.2.

@end deffn

@geindex help() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP help}@anchor{3229}
@deffn {Method} NNTP.help (*, file=None)

Send a @code{HELP} command.  Return a pair @code{(response, list)} where @emph{list} is a
list of help strings.
@end deffn

@geindex stat() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP stat}@anchor{322a}
@deffn {Method} NNTP.stat (message_spec=None)

Send a @code{STAT} command, where @emph{message_spec} is either a message id
(enclosed in @code{'<'} and @code{'>'}) or an article number in the current group.
If @emph{message_spec} is omitted or @ref{182,,None}, the current article in the
current group is considered.  Return a triple @code{(response, number, id)}
where @emph{number} is the article number and @emph{id} is the message id.

@example
>>> _, _, first, last, _ = s.group('gmane.comp.python.devel')
>>> resp, number, message_id = s.stat(first)
>>> number, message_id
(9099, '<20030112190404.GE29873@@epoch.metaslash.com>')
@end example
@end deffn

@geindex next() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP next}@anchor{322b}
@deffn {Method} NNTP.next ()

Send a @code{NEXT} command.  Return as for @ref{322a,,stat()}.
@end deffn

@geindex last() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP last}@anchor{322c}
@deffn {Method} NNTP.last ()

Send a @code{LAST} command.  Return as for @ref{322a,,stat()}.
@end deffn

@geindex article() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP article}@anchor{322d}
@deffn {Method} NNTP.article (message_spec=None, *, file=None)

Send an @code{ARTICLE} command, where @emph{message_spec} has the same meaning as
for @ref{322a,,stat()}.  Return a tuple @code{(response, info)} where @emph{info}
is a @ref{3e4,,namedtuple} with three attributes @emph{number},
@emph{message_id} and @emph{lines} (in that order).  @emph{number} is the article number
in the group (or 0 if the information is not available), @emph{message_id} the
message id as a string, and @emph{lines} a list of lines (without terminating
newlines) comprising the raw message including headers and body.

@example
>>> resp, info = s.article('<20030112190404.GE29873@@epoch.metaslash.com>')
>>> info.number
0
>>> info.message_id
'<20030112190404.GE29873@@epoch.metaslash.com>'
>>> len(info.lines)
65
>>> info.lines[0]
b'Path: main.gmane.org!not-for-mail'
>>> info.lines[1]
b'From: Neal Norwitz <neal@@metaslash.com>'
>>> info.lines[-3:]
[b'There is a patch for 2.3 as well as 2.2.', b'@w{'}, b'Neal']
@end example
@end deffn

@geindex head() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP head}@anchor{322e}
@deffn {Method} NNTP.head (message_spec=None, *, file=None)

Same as @ref{322d,,article()}, but sends a @code{HEAD} command.  The @emph{lines}
returned (or written to @emph{file}) will only contain the message headers, not
the body.
@end deffn

@geindex body() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP body}@anchor{322f}
@deffn {Method} NNTP.body (message_spec=None, *, file=None)

Same as @ref{322d,,article()}, but sends a @code{BODY} command.  The @emph{lines}
returned (or written to @emph{file}) will only contain the message body, not the
headers.
@end deffn

@geindex post() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP post}@anchor{3230}
@deffn {Method} NNTP.post (data)

Post an article using the @code{POST} command.  The @emph{data} argument is either
a @ref{d68,,file object} opened for binary reading, or any iterable of bytes
objects (representing raw lines of the article to be posted).  It should
represent a well-formed news article, including the required headers.  The
@ref{3230,,post()} method automatically escapes lines beginning with @code{.} and
appends the termination line.

If the method succeeds, the server’s response is returned.  If the server
refuses posting, a @ref{3215,,NNTPReplyError} is raised.
@end deffn

@geindex ihave() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP ihave}@anchor{3231}
@deffn {Method} NNTP.ihave (message_id, data)

Send an @code{IHAVE} command. @emph{message_id} is the id of the message to send
to the server (enclosed in  @code{'<'} and @code{'>'}).  The @emph{data} parameter
and the return value are the same as for @ref{3230,,post()}.
@end deffn

@geindex date() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP date}@anchor{3232}
@deffn {Method} NNTP.date ()

Return a pair @code{(response, date)}.  @emph{date} is a @ref{363,,datetime}
object containing the current date and time of the server.
@end deffn

@geindex slave() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP slave}@anchor{3233}
@deffn {Method} NNTP.slave ()

Send a @code{SLAVE} command.  Return the server’s @emph{response}.
@end deffn

@geindex set_debuglevel() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP set_debuglevel}@anchor{3234}
@deffn {Method} NNTP.set_debuglevel (level)

Set the instance’s debugging level.  This controls the amount of debugging
output printed.  The default, @code{0}, produces no debugging output.  A value of
@code{1} produces a moderate amount of debugging output, generally a single line
per request or response.  A value of @code{2} or higher produces the maximum amount
of debugging output, logging each line sent and received on the connection
(including message text).
@end deffn

The following are optional NNTP extensions defined in 
@geindex RFC; RFC 2980
RFC 2980@footnote{https://tools.ietf.org/html/rfc2980.html}.  Some of
them have been superseded by newer commands in 
@geindex RFC; RFC 3977
RFC 3977@footnote{https://tools.ietf.org/html/rfc3977.html}.

@geindex xhdr() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP xhdr}@anchor{3235}
@deffn {Method} NNTP.xhdr (hdr, str, *, file=None)

Send an @code{XHDR} command.  The @emph{hdr} argument is a header keyword, e.g.
@code{'subject'}.  The @emph{str} argument should have the form @code{'first-last'}
where @emph{first} and @emph{last} are the first and last article numbers to search.
Return a pair @code{(response, list)}, where @emph{list} is a list of pairs @code{(id,
text)}, where @emph{id} is an article number (as a string) and @emph{text} is the text of
the requested header for that article. If the @emph{file} parameter is supplied, then
the output of the  @code{XHDR} command is stored in a file.  If @emph{file} is a string,
then the method will open a file with that name, write to it  then close it.
If @emph{file} is a @ref{d68,,file object}, then it will start calling @code{write()} on
it to store the lines of the command output. If @emph{file} is supplied, then the
returned @emph{list} is an empty list.
@end deffn

@geindex xover() (nntplib.NNTP method)
@anchor{library/nntplib nntplib NNTP xover}@anchor{3236}
@deffn {Method} NNTP.xover (start, end, *, file=None)

Send an @code{XOVER} command.  @emph{start} and @emph{end} are article numbers
delimiting the range of articles to select.  The return value is the
same of for @ref{3227,,over()}.  It is recommended to use @ref{3227,,over()}
instead, since it will automatically use the newer @code{OVER} command
if available.
@end deffn

@node Utility functions<2>,,NNTP Objects,nntplib — NNTP protocol client
@anchor{library/nntplib utility-functions}@anchor{3237}
@subsubsection Utility functions


The module also defines the following utility function:

@geindex decode_header() (in module nntplib)
@anchor{library/nntplib nntplib decode_header}@anchor{3228}
@deffn {Function} nntplib.decode_header (header_str)

Decode a header value, un-escaping any escaped non-ASCII characters.
@emph{header_str} must be a @ref{2ef,,str} object.  The unescaped value is
returned.  Using this function is recommended to display some headers
in a human readable form:

@example
>>> decode_header("Some subject")
'Some subject'
>>> decode_header("=?ISO-8859-15?Q?D=E9buter_en_Python?=")
'Débuter en Python'
>>> decode_header("Re: =?UTF-8?B?cHJvYmzDqG1lIGRlIG1hdHJpY2U=?=")
'Re: problème de matrice'
@end example
@end deffn

@node smtplib — SMTP protocol client,smtpd — SMTP Server,nntplib — NNTP protocol client,Internet Protocols and Support
@anchor{library/smtplib doc}@anchor{3238}@anchor{library/smtplib module-smtplib}@anchor{e8}@anchor{library/smtplib smtplib-smtp-protocol-client}@anchor{3239}
@subsection @code{smtplib} — SMTP protocol client


@geindex module; smtplib

@strong{Source code:} Lib/smtplib.py@footnote{https://github.com/python/cpython/tree/main/Lib/smtplib.py}

@geindex SMTP; protocol
@geindex Simple Mail Transfer Protocol


__________________________________________________________________


The @ref{e8,,smtplib} module defines an SMTP client session object that can be used
to send mail to any internet machine with an SMTP or ESMTP listener daemon.  For
details of SMTP and ESMTP operation, consult 
@geindex RFC; RFC 821
RFC 821@footnote{https://tools.ietf.org/html/rfc821.html} (Simple Mail Transfer
Protocol) and 
@geindex RFC; RFC 1869
RFC 1869@footnote{https://tools.ietf.org/html/rfc1869.html} (SMTP Service Extensions).

@geindex SMTP (class in smtplib)
@anchor{library/smtplib smtplib SMTP}@anchor{421}
@deffn {Class} smtplib.SMTP (host='@w{'}, port=0, local_hostname=None[, timeout], source_address=None)

An @ref{421,,SMTP} instance encapsulates an SMTP connection.  It has methods
that support a full repertoire of SMTP and ESMTP operations. If the optional
host and port parameters are given, the SMTP @ref{1266,,connect()} method is
called with those parameters during initialization.  If specified,
@emph{local_hostname} is used as the FQDN of the local host in the HELO/EHLO
command.  Otherwise, the local hostname is found using
@ref{2b11,,socket.getfqdn()}.  If the @ref{1266,,connect()} call returns anything other
than a success code, an @ref{323a,,SMTPConnectError} is raised. The optional
@emph{timeout} parameter specifies a timeout in seconds for blocking operations
like the connection attempt (if not specified, the global default timeout
setting will be used).  If the timeout expires, @ref{2b8,,TimeoutError} is
raised.  The optional source_address parameter allows binding
to some specific source address in a machine with multiple network
interfaces, and/or to some specific source TCP port. It takes a 2-tuple
(host, port), for the socket to bind to as its source address before
connecting. If omitted (or if host or port are @code{'@w{'}} and/or 0 respectively)
the OS default behavior will be used.

For normal use, you should only require the initialization/connect,
@ref{9c3,,sendmail()}, and @ref{323b,,SMTP.quit()} methods.
An example is included below.

The @ref{421,,SMTP} class supports the @ref{168,,with} statement.  When used
like this, the SMTP @code{QUIT} command is issued automatically when the
@code{with} statement exits.  E.g.:

@example
>>> from smtplib import SMTP
>>> with SMTP("domain.org") as smtp:
...     smtp.noop()
...
(250, b'Ok')
>>>
@end example


All commands will raise an @ref{1130,,auditing event}
@code{smtplib.SMTP.send} with arguments @code{self} and @code{data},
where @code{data} is the bytes about to be sent to the remote host.



Changed in version 3.3: Support for the @ref{168,,with} statement was added.



Changed in version 3.3: source_address argument was added.



New in version 3.5: The SMTPUTF8 extension (
@geindex RFC; RFC 6531
RFC 6531@footnote{https://tools.ietf.org/html/rfc6531.html}) is now supported.



Changed in version 3.9: If the @emph{timeout} parameter is set to be zero, it will raise a
@ref{17c,,ValueError} to prevent the creation of a non-blocking socket

@end deffn

@geindex SMTP_SSL (class in smtplib)
@anchor{library/smtplib smtplib SMTP_SSL}@anchor{422}
@deffn {Class} smtplib.SMTP_SSL (host='@w{'}, port=0, local_hostname=None, keyfile=None, certfile=None[, timeout], context=None, source_address=None)

An @ref{422,,SMTP_SSL} instance behaves exactly the same as instances of
@ref{421,,SMTP}. @ref{422,,SMTP_SSL} should be used for situations where SSL is
required from the beginning of the connection and using @code{starttls()} is
not appropriate. If @emph{host} is not specified, the local host is used. If
@emph{port} is zero, the standard SMTP-over-SSL port (465) is used.  The optional
arguments @emph{local_hostname}, @emph{timeout} and @emph{source_address} have the same
meaning as they do in the @ref{421,,SMTP} class.  @emph{context}, also optional,
can contain a @ref{2bf,,SSLContext} and allows configuring various
aspects of the secure connection.  Please read @ref{2a39,,Security considerations} for
best practices.

@emph{keyfile} and @emph{certfile} are a legacy alternative to @emph{context}, and can
point to a PEM formatted private key and certificate chain file for the
SSL connection.


Changed in version 3.3: @emph{context} was added.



Changed in version 3.3: source_address argument was added.



Changed in version 3.4: The class now supports hostname check with
@ref{2b42,,ssl.SSLContext.check_hostname} and @emph{Server Name Indication} (see
@ref{2b4f,,ssl.HAS_SNI}).



Deprecated since version 3.6: @emph{keyfile} and @emph{certfile} are deprecated in favor of @emph{context}.
Please use @ref{cd9,,ssl.SSLContext.load_cert_chain()} instead, or let
@ref{b29,,ssl.create_default_context()} select the system’s trusted CA
certificates for you.



Changed in version 3.9: If the @emph{timeout} parameter is set to be zero, it will raise a
@ref{17c,,ValueError} to prevent the creation of a non-blocking socket

@end deffn

@geindex LMTP (class in smtplib)
@anchor{library/smtplib smtplib LMTP}@anchor{423}
@deffn {Class} smtplib.LMTP (host='@w{'}, port=LMTP_PORT, local_hostname=None, source_address=None[, timeout])

The LMTP protocol, which is very similar to ESMTP, is heavily based on the
standard SMTP client. It’s common to use Unix sockets for LMTP, so our
@code{connect()} method must support that as well as a regular host:port
server. The optional arguments local_hostname and source_address have the
same meaning as they do in the @ref{421,,SMTP} class. To specify a Unix
socket, you must use an absolute path for @emph{host}, starting with a ‘/’.

Authentication is supported, using the regular SMTP mechanism. When using a
Unix socket, LMTP generally don’t support or require any authentication, but
your mileage might vary.


Changed in version 3.9: The optional @emph{timeout} parameter was added.

@end deffn

A nice selection of exceptions is defined as well:

@geindex SMTPException
@anchor{library/smtplib smtplib SMTPException}@anchor{b22}
@deffn {Exception} smtplib.SMTPException

Subclass of @ref{2a1,,OSError} that is the base exception class for all
the other exceptions provided by this module.


Changed in version 3.4: SMTPException became subclass of @ref{2a1,,OSError}

@end deffn

@geindex SMTPServerDisconnected
@anchor{library/smtplib smtplib SMTPServerDisconnected}@anchor{323c}
@deffn {Exception} smtplib.SMTPServerDisconnected

This exception is raised when the server unexpectedly disconnects, or when an
attempt is made to use the @ref{421,,SMTP} instance before connecting it to a
server.
@end deffn

@geindex SMTPResponseException
@anchor{library/smtplib smtplib SMTPResponseException}@anchor{323d}
@deffn {Exception} smtplib.SMTPResponseException

Base class for all exceptions that include an SMTP error code. These exceptions
are generated in some instances when the SMTP server returns an error code.  The
error code is stored in the @code{smtp_code} attribute of the error, and the
@code{smtp_error} attribute is set to the error message.
@end deffn

@geindex SMTPSenderRefused
@anchor{library/smtplib smtplib SMTPSenderRefused}@anchor{323e}
@deffn {Exception} smtplib.SMTPSenderRefused

Sender address refused.  In addition to the attributes set by on all
@ref{323d,,SMTPResponseException} exceptions, this sets ‘sender’ to the string that
the SMTP server refused.
@end deffn

@geindex SMTPRecipientsRefused
@anchor{library/smtplib smtplib SMTPRecipientsRefused}@anchor{323f}
@deffn {Exception} smtplib.SMTPRecipientsRefused

All recipient addresses refused.  The errors for each recipient are accessible
through the attribute @code{recipients}, which is a dictionary of exactly the
same sort as @ref{9c3,,SMTP.sendmail()} returns.
@end deffn

@geindex SMTPDataError
@anchor{library/smtplib smtplib SMTPDataError}@anchor{3240}
@deffn {Exception} smtplib.SMTPDataError

The SMTP server refused to accept the message data.
@end deffn

@geindex SMTPConnectError
@anchor{library/smtplib smtplib SMTPConnectError}@anchor{323a}
@deffn {Exception} smtplib.SMTPConnectError

Error occurred during establishment of a connection  with the server.
@end deffn

@geindex SMTPHeloError
@anchor{library/smtplib smtplib SMTPHeloError}@anchor{3241}
@deffn {Exception} smtplib.SMTPHeloError

The server refused our @code{HELO} message.
@end deffn

@geindex SMTPNotSupportedError
@anchor{library/smtplib smtplib SMTPNotSupportedError}@anchor{1264}
@deffn {Exception} smtplib.SMTPNotSupportedError

The command or option attempted is not supported by the server.


New in version 3.5.

@end deffn

@geindex SMTPAuthenticationError
@anchor{library/smtplib smtplib SMTPAuthenticationError}@anchor{3242}
@deffn {Exception} smtplib.SMTPAuthenticationError

SMTP authentication went wrong.  Most probably the server didn’t accept the
username/password combination provided.
@end deffn


@subsubheading See also



@table @asis

@geindex RFC; RFC 821

@item RFC 821@footnote{https://tools.ietf.org/html/rfc821.html} - Simple Mail Transfer Protocol

Protocol definition for SMTP.  This document covers the model, operating
procedure, and protocol details for SMTP.

@geindex RFC; RFC 1869

@item RFC 1869@footnote{https://tools.ietf.org/html/rfc1869.html} - SMTP Service Extensions

Definition of the ESMTP extensions for SMTP.  This describes a framework for
extending SMTP with new commands, supporting dynamic discovery of the commands
provided by the server, and defines a few additional commands.
@end table


@menu
* SMTP Objects:: 
* SMTP Example:: 

@end menu

@node SMTP Objects,SMTP Example,,smtplib — SMTP protocol client
@anchor{library/smtplib id1}@anchor{3243}@anchor{library/smtplib smtp-objects}@anchor{3244}
@subsubsection SMTP Objects


An @ref{421,,SMTP} instance has the following methods:

@geindex set_debuglevel() (smtplib.SMTP method)
@anchor{library/smtplib smtplib SMTP set_debuglevel}@anchor{9c2}
@deffn {Method} SMTP.set_debuglevel (level)

Set the debug output level.  A value of 1 or @code{True} for @emph{level} results in
debug messages for connection and for all messages sent to and received from
the server.  A value of 2 for @emph{level} results in these messages being
timestamped.


Changed in version 3.5: Added debuglevel 2.

@end deffn

@geindex docmd() (smtplib.SMTP method)
@anchor{library/smtplib smtplib SMTP docmd}@anchor{3245}
@deffn {Method} SMTP.docmd (cmd, args='@w{'})

Send a command @emph{cmd} to the server.  The optional argument @emph{args} is simply
concatenated to the command, separated by a space.

This returns a 2-tuple composed of a numeric response code and the actual
response line (multiline responses are joined into one long line.)

In normal operation it should not be necessary to call this method explicitly.
It is used to implement other methods and may be useful for testing private
extensions.

If the connection to the server is lost while waiting for the reply,
@ref{323c,,SMTPServerDisconnected} will be raised.
@end deffn

@geindex connect() (smtplib.SMTP method)
@anchor{library/smtplib smtplib SMTP connect}@anchor{1266}
@deffn {Method} SMTP.connect (host='localhost', port=0)

Connect to a host on a given port.  The defaults are to connect to the local
host at the standard SMTP port (25). If the hostname ends with a colon (@code{':'})
followed by a number, that suffix will be stripped off and the number
interpreted as the port number to use. This method is automatically invoked by
the constructor if a host is specified during instantiation.  Returns a
2-tuple of the response code and message sent by the server in its
connection response.

Raises an @ref{1130,,auditing event} @code{smtplib.connect} with arguments @code{self}, @code{host}, @code{port}.
@end deffn

@geindex helo() (smtplib.SMTP method)
@anchor{library/smtplib smtplib SMTP helo}@anchor{3246}
@deffn {Method} SMTP.helo (name='@w{'})

Identify yourself to the SMTP server using @code{HELO}.  The hostname argument
defaults to the fully qualified domain name of the local host.
The message returned by the server is stored as the @code{helo_resp} attribute
of the object.

In normal operation it should not be necessary to call this method explicitly.
It will be implicitly called by the @ref{9c3,,sendmail()} when necessary.
@end deffn

@geindex ehlo() (smtplib.SMTP method)
@anchor{library/smtplib smtplib SMTP ehlo}@anchor{3247}
@deffn {Method} SMTP.ehlo (name='@w{'})

Identify yourself to an ESMTP server using @code{EHLO}.  The hostname argument
defaults to the fully qualified domain name of the local host.  Examine the
response for ESMTP option and store them for use by @ref{3248,,has_extn()}.
Also sets several informational attributes: the message returned by
the server is stored as the @code{ehlo_resp} attribute, @code{does_esmtp}
is set to @code{True} or @code{False} depending on whether the server supports
ESMTP, and @code{esmtp_features} will be a dictionary containing the names
of the SMTP service extensions this server supports, and their parameters
(if any).

Unless you wish to use @ref{3248,,has_extn()} before sending mail, it should not be
necessary to call this method explicitly.  It will be implicitly called by
@ref{9c3,,sendmail()} when necessary.
@end deffn

@geindex ehlo_or_helo_if_needed() (smtplib.SMTP method)
@anchor{library/smtplib smtplib SMTP ehlo_or_helo_if_needed}@anchor{3249}
@deffn {Method} SMTP.ehlo_or_helo_if_needed ()

This method calls @ref{3247,,ehlo()} and/or @ref{3246,,helo()} if there has been no
previous @code{EHLO} or @code{HELO} command this session.  It tries ESMTP @code{EHLO}
first.


@table @asis

@item @ref{3241,,SMTPHeloError}

The server didn’t reply properly to the @code{HELO} greeting.
@end table
@end deffn

@geindex has_extn() (smtplib.SMTP method)
@anchor{library/smtplib smtplib SMTP has_extn}@anchor{3248}
@deffn {Method} SMTP.has_extn (name)

Return @ref{748,,True} if @emph{name} is in the set of SMTP service extensions returned
by the server, @ref{659,,False} otherwise. Case is ignored.
@end deffn

@geindex verify() (smtplib.SMTP method)
@anchor{library/smtplib smtplib SMTP verify}@anchor{324a}
@deffn {Method} SMTP.verify (address)

Check the validity of an address on this server using SMTP @code{VRFY}. Returns a
tuple consisting of code 250 and a full 
@geindex RFC; RFC 822
RFC 822@footnote{https://tools.ietf.org/html/rfc822.html} address (including human
name) if the user address is valid. Otherwise returns an SMTP error code of 400
or greater and an error string.

@cartouche
@quotation Note 
Many sites disable SMTP @code{VRFY} in order to foil spammers.
@end quotation
@end cartouche
@end deffn

@geindex login() (smtplib.SMTP method)
@anchor{library/smtplib smtplib SMTP login}@anchor{324b}
@deffn {Method} SMTP.login (user, password, *, initial_response_ok=True)

Log in on an SMTP server that requires authentication. The arguments are the
username and the password to authenticate with. If there has been no previous
@code{EHLO} or @code{HELO} command this session, this method tries ESMTP @code{EHLO}
first. This method will return normally if the authentication was successful, or
may raise the following exceptions:


@table @asis

@item @ref{3241,,SMTPHeloError}

The server didn’t reply properly to the @code{HELO} greeting.

@item @ref{3242,,SMTPAuthenticationError}

The server didn’t accept the username/password combination.

@item @ref{1264,,SMTPNotSupportedError}

The @code{AUTH} command is not supported by the server.

@item @ref{b22,,SMTPException}

No suitable authentication method was found.
@end table

Each of the authentication methods supported by @ref{e8,,smtplib} are tried in
turn if they are advertised as supported by the server.  See @ref{9c1,,auth()}
for a list of supported authentication methods.  @emph{initial_response_ok} is
passed through to @ref{9c1,,auth()}.

Optional keyword argument @emph{initial_response_ok} specifies whether, for
authentication methods that support it, an “initial response” as specified
in 
@geindex RFC; RFC 4954
RFC 4954@footnote{https://tools.ietf.org/html/rfc4954.html} can be sent along with the @code{AUTH} command, rather than
requiring a challenge/response.


Changed in version 3.5: @ref{1264,,SMTPNotSupportedError} may be raised, and the
@emph{initial_response_ok} parameter was added.

@end deffn

@geindex auth() (smtplib.SMTP method)
@anchor{library/smtplib smtplib SMTP auth}@anchor{9c1}
@deffn {Method} SMTP.auth (mechanism, authobject, *, initial_response_ok=True)

Issue an @code{SMTP} @code{AUTH} command for the specified authentication
@emph{mechanism}, and handle the challenge response via @emph{authobject}.

@emph{mechanism} specifies which authentication mechanism is to
be used as argument to the @code{AUTH} command; the valid values are
those listed in the @code{auth} element of @code{esmtp_features}.

@emph{authobject} must be a callable object taking an optional single argument:

@quotation

data = authobject(challenge=None)
@end quotation

If optional keyword argument @emph{initial_response_ok} is true,
@code{authobject()} will be called first with no argument.  It can return the
@geindex RFC; RFC 4954
RFC 4954@footnote{https://tools.ietf.org/html/rfc4954.html} “initial response” ASCII @code{str} which will be encoded and sent with
the @code{AUTH} command as below.  If the @code{authobject()} does not support an
initial response (e.g. because it requires a challenge), it should return
@code{None} when called with @code{challenge=None}.  If @emph{initial_response_ok} is
false, then @code{authobject()} will not be called first with @code{None}.

If the initial response check returns @code{None}, or if @emph{initial_response_ok} is
false, @code{authobject()} will be called to process the server’s challenge
response; the @emph{challenge} argument it is passed will be a @code{bytes}.  It
should return ASCII @code{str} @emph{data} that will be base64 encoded and sent to the
server.

The @code{SMTP} class provides @code{authobjects} for the @code{CRAM-MD5}, @code{PLAIN},
and @code{LOGIN} mechanisms; they are named @code{SMTP.auth_cram_md5},
@code{SMTP.auth_plain}, and @code{SMTP.auth_login} respectively.  They all require
that the @code{user} and @code{password} properties of the @code{SMTP} instance are
set to appropriate values.

User code does not normally need to call @code{auth} directly, but can instead
call the @ref{324b,,login()} method, which will try each of the above mechanisms
in turn, in the order listed.  @code{auth} is exposed to facilitate the
implementation of authentication methods not (or not yet) supported
directly by @ref{e8,,smtplib}.


New in version 3.5.

@end deffn

@geindex starttls() (smtplib.SMTP method)
@anchor{library/smtplib smtplib SMTP starttls}@anchor{ccd}
@deffn {Method} SMTP.starttls (keyfile=None, certfile=None, context=None)

Put the SMTP connection in TLS (Transport Layer Security) mode.  All SMTP
commands that follow will be encrypted.  You should then call @ref{3247,,ehlo()}
again.

If @emph{keyfile} and @emph{certfile} are provided, they are used to create an
@ref{2bf,,ssl.SSLContext}.

Optional @emph{context} parameter is an @ref{2bf,,ssl.SSLContext} object; This is
an alternative to using a keyfile and a certfile and if specified both
@emph{keyfile} and @emph{certfile} should be @code{None}.

If there has been no previous @code{EHLO} or @code{HELO} command this session,
this method tries ESMTP @code{EHLO} first.


Deprecated since version 3.6: @emph{keyfile} and @emph{certfile} are deprecated in favor of @emph{context}.
Please use @ref{cd9,,ssl.SSLContext.load_cert_chain()} instead, or let
@ref{b29,,ssl.create_default_context()} select the system’s trusted CA
certificates for you.



@table @asis

@item @ref{3241,,SMTPHeloError}

The server didn’t reply properly to the @code{HELO} greeting.

@item @ref{1264,,SMTPNotSupportedError}

The server does not support the STARTTLS extension.

@item @ref{5af,,RuntimeError}

SSL/TLS support is not available to your Python interpreter.
@end table


Changed in version 3.3: @emph{context} was added.



Changed in version 3.4: The method now supports hostname check with
@code{SSLContext.check_hostname} and @emph{Server Name Indicator} (see
@ref{2b4f,,HAS_SNI}).



Changed in version 3.5: The error raised for lack of STARTTLS support is now the
@ref{1264,,SMTPNotSupportedError} subclass instead of the base
@ref{b22,,SMTPException}.

@end deffn

@geindex sendmail() (smtplib.SMTP method)
@anchor{library/smtplib smtplib SMTP sendmail}@anchor{9c3}
@deffn {Method} SMTP.sendmail (from_addr, to_addrs, msg, mail_options=(), rcpt_options=())

Send mail.  The required arguments are an 
@geindex RFC; RFC 822
RFC 822@footnote{https://tools.ietf.org/html/rfc822.html} from-address string, a list
of 
@geindex RFC; RFC 822
RFC 822@footnote{https://tools.ietf.org/html/rfc822.html} to-address strings (a bare string will be treated as a list with 1
address), and a message string.  The caller may pass a list of ESMTP options
(such as @code{8bitmime}) to be used in @code{MAIL FROM} commands as @emph{mail_options}.
ESMTP options (such as @code{DSN} commands) that should be used with all @code{RCPT}
commands can be passed as @emph{rcpt_options}.  (If you need to use different ESMTP
options to different recipients you have to use the low-level methods such as
@code{mail()}, @code{rcpt()} and @code{data()} to send the message.)

@cartouche
@quotation Note 
The @emph{from_addr} and @emph{to_addrs} parameters are used to construct the message
envelope used by the transport agents.  @code{sendmail} does not modify the
message headers in any way.
@end quotation
@end cartouche

@emph{msg} may be a string containing characters in the ASCII range, or a byte
string.  A string is encoded to bytes using the ascii codec, and lone @code{\r}
and @code{\n} characters are converted to @code{\r\n} characters.  A byte string is
not modified.

If there has been no previous @code{EHLO} or @code{HELO} command this session, this
method tries ESMTP @code{EHLO} first. If the server does ESMTP, message size and
each of the specified options will be passed to it (if the option is in the
feature set the server advertises).  If @code{EHLO} fails, @code{HELO} will be tried
and ESMTP options suppressed.

This method will return normally if the mail is accepted for at least one
recipient. Otherwise it will raise an exception.  That is, if this method does
not raise an exception, then someone should get your mail. If this method does
not raise an exception, it returns a dictionary, with one entry for each
recipient that was refused.  Each entry contains a tuple of the SMTP error code
and the accompanying error message sent by the server.

If @code{SMTPUTF8} is included in @emph{mail_options}, and the server supports it,
@emph{from_addr} and @emph{to_addrs} may contain non-ASCII characters.

This method may raise the following exceptions:


@table @asis

@item @ref{323f,,SMTPRecipientsRefused}

All recipients were refused.  Nobody got the mail.  The @code{recipients}
attribute of the exception object is a dictionary with information about the
refused recipients (like the one returned when at least one recipient was
accepted).

@item @ref{3241,,SMTPHeloError}

The server didn’t reply properly to the @code{HELO} greeting.

@item @ref{323e,,SMTPSenderRefused}

The server didn’t accept the @emph{from_addr}.

@item @ref{3240,,SMTPDataError}

The server replied with an unexpected error code (other than a refusal of a
recipient).

@item @ref{1264,,SMTPNotSupportedError}

@code{SMTPUTF8} was given in the @emph{mail_options} but is not supported by the
server.
@end table

Unless otherwise noted, the connection will be open even after an exception is
raised.


Changed in version 3.2: @emph{msg} may be a byte string.



Changed in version 3.5: @code{SMTPUTF8} support added, and @ref{1264,,SMTPNotSupportedError} may be
raised if @code{SMTPUTF8} is specified but the server does not support it.

@end deffn

@geindex send_message() (smtplib.SMTP method)
@anchor{library/smtplib smtplib SMTP send_message}@anchor{9c4}
@deffn {Method} SMTP.send_message (msg, from_addr=None, to_addrs=None, mail_options=(), rcpt_options=())

This is a convenience method for calling @ref{9c3,,sendmail()} with the message
represented by an @ref{7e3,,email.message.Message} object.  The arguments have
the same meaning as for @ref{9c3,,sendmail()}, except that @emph{msg} is a @code{Message}
object.

If @emph{from_addr} is @code{None} or @emph{to_addrs} is @code{None}, @code{send_message} fills
those arguments with addresses extracted from the headers of @emph{msg} as
specified in 
@geindex RFC; RFC 5322
RFC 5322@footnote{https://tools.ietf.org/html/rfc5322.html}: @emph{from_addr} is set to the @code{Sender}
field if it is present, and otherwise to the @code{From} field.
@emph{to_addrs} combines the values (if any) of the @code{To},
@code{Cc}, and @code{Bcc} fields from @emph{msg}.  If exactly one
set of @code{Resent-*} headers appear in the message, the regular
headers are ignored and the @code{Resent-*} headers are used instead.
If the message contains more than one set of @code{Resent-*} headers,
a @ref{17c,,ValueError} is raised, since there is no way to unambiguously detect
the most recent set of @code{Resent-} headers.

@code{send_message} serializes @emph{msg} using
@ref{d73,,BytesGenerator} with @code{\r\n} as the @emph{linesep}, and
calls @ref{9c3,,sendmail()} to transmit the resulting message.  Regardless of the
values of @emph{from_addr} and @emph{to_addrs}, @code{send_message} does not transmit any
@code{Bcc} or @code{Resent-Bcc} headers that may appear
in @emph{msg}.  If any of the addresses in @emph{from_addr} and @emph{to_addrs} contain
non-ASCII characters and the server does not advertise @code{SMTPUTF8} support,
an @code{SMTPNotSupported} error is raised.  Otherwise the @code{Message} is
serialized with a clone of its @ref{70,,policy} with the
@ref{957,,utf8} attribute set to @code{True}, and
@code{SMTPUTF8} and @code{BODY=8BITMIME} are added to @emph{mail_options}.


New in version 3.2.



New in version 3.5: Support for internationalized addresses (@code{SMTPUTF8}).

@end deffn

@geindex quit() (smtplib.SMTP method)
@anchor{library/smtplib smtplib SMTP quit}@anchor{323b}
@deffn {Method} SMTP.quit ()

Terminate the SMTP session and close the connection.  Return the result of
the SMTP @code{QUIT} command.
@end deffn

Low-level methods corresponding to the standard SMTP/ESMTP commands @code{HELP},
@code{RSET}, @code{NOOP}, @code{MAIL}, @code{RCPT}, and @code{DATA} are also supported.
Normally these do not need to be called directly, so they are not documented
here.  For details, consult the module code.

@node SMTP Example,,SMTP Objects,smtplib — SMTP protocol client
@anchor{library/smtplib id2}@anchor{324c}@anchor{library/smtplib smtp-example}@anchor{324d}
@subsubsection SMTP Example


This example prompts the user for addresses needed in the message envelope (‘To’
and ‘From’ addresses), and the message to be delivered.  Note that the headers
to be included with the message must be included in the message as entered; this
example doesn’t do any processing of the 
@geindex RFC; RFC 822
RFC 822@footnote{https://tools.ietf.org/html/rfc822.html} headers.  In particular, the
‘To’ and ‘From’ addresses must be included in the message headers explicitly.

@example
import smtplib

def prompt(prompt):
    return input(prompt).strip()

fromaddr = prompt("From: ")
toaddrs  = prompt("To: ").split()
print("Enter message, end with ^D (Unix) or ^Z (Windows):")

# Add the From: and To: headers at the start!
msg = ("From: %s\r\nTo: %s\r\n\r\n"
       % (fromaddr, ", ".join(toaddrs)))
while True:
    try:
        line = input()
    except EOFError:
        break
    if not line:
        break
    msg = msg + line

print("Message length is", len(msg))

server = smtplib.SMTP('localhost')
server.set_debuglevel(1)
server.sendmail(fromaddr, toaddrs, msg)
server.quit()
@end example

@cartouche
@quotation Note 
In general, you will want to use the @ref{64,,email} package’s features to
construct an email message, which you can then send
via @ref{9c4,,send_message()}; see @ref{ab9,,email; Examples}.
@end quotation
@end cartouche

@node smtpd — SMTP Server,telnetlib — Telnet client,smtplib — SMTP protocol client,Internet Protocols and Support
@anchor{library/smtpd doc}@anchor{324e}@anchor{library/smtpd module-smtpd}@anchor{e7}@anchor{library/smtpd smtpd-smtp-server}@anchor{324f}
@subsection @code{smtpd} — SMTP Server


@geindex module; smtpd

@strong{Source code:} Lib/smtpd.py@footnote{https://github.com/python/cpython/tree/main/Lib/smtpd.py}


__________________________________________________________________


This module offers several classes to implement SMTP (email) servers.


Deprecated since version 3.6: The aiosmtpd@footnote{https://aiosmtpd.readthedocs.io/} package is a recommended
replacement for this module.  It is based on @ref{9,,asyncio} and provides a
more straightforward API.


Several server implementations are present; one is a generic
do-nothing implementation, which can be overridden, while the other two offer
specific mail-sending strategies.

Additionally the SMTPChannel may be extended to implement very specific
interaction behaviour with SMTP clients.

The code supports 
@geindex RFC; RFC 5321
RFC 5321@footnote{https://tools.ietf.org/html/rfc5321.html}, plus the 
@geindex RFC; RFC 1870
RFC 1870@footnote{https://tools.ietf.org/html/rfc1870.html} SIZE and 
@geindex RFC; RFC 6531
RFC 6531@footnote{https://tools.ietf.org/html/rfc6531.html}
SMTPUTF8 extensions.

@menu
* SMTPServer Objects:: 
* DebuggingServer Objects:: 
* PureProxy Objects:: 
* SMTPChannel Objects:: 

@end menu

@node SMTPServer Objects,DebuggingServer Objects,,smtpd — SMTP Server
@anchor{library/smtpd smtpserver-objects}@anchor{3250}
@subsubsection SMTPServer Objects


@geindex SMTPServer (class in smtpd)
@anchor{library/smtpd smtpd SMTPServer}@anchor{898}
@deffn {Class} smtpd.SMTPServer (localaddr, remoteaddr, data_size_limit=33554432, map=None, enable_SMTPUTF8=False, decode_data=False)

Create a new @ref{898,,SMTPServer} object, which binds to local address
@emph{localaddr}.  It will treat @emph{remoteaddr} as an upstream SMTP relayer.  Both
@emph{localaddr} and @emph{remoteaddr} should be a @ref{2aec,,(host@comma{} port)}
tuple.  The object inherits from @ref{de0,,asyncore.dispatcher}, and so will
insert itself into @ref{a,,asyncore}’s event loop on instantiation.

@emph{data_size_limit} specifies the maximum number of bytes that will be
accepted in a @code{DATA} command.  A value of @code{None} or @code{0} means no
limit.

@emph{map} is the socket map to use for connections (an initially empty
dictionary is a suitable value).  If not specified the @ref{a,,asyncore}
global socket map is used.

@emph{enable_SMTPUTF8} determines whether the @code{SMTPUTF8} extension (as defined
in 
@geindex RFC; RFC 6531
RFC 6531@footnote{https://tools.ietf.org/html/rfc6531.html}) should be enabled.  The default is @code{False}.
When @code{True}, @code{SMTPUTF8} is accepted as a parameter to the @code{MAIL}
command and when present is passed to @ref{899,,process_message()} in the
@code{kwargs['mail_options']} list.  @emph{decode_data} and @emph{enable_SMTPUTF8}
cannot be set to @code{True} at the same time.

@emph{decode_data} specifies whether the data portion of the SMTP transaction
should be decoded using UTF-8.  When @emph{decode_data} is @code{False} (the
default), the server advertises the @code{8BITMIME}
extension (
@geindex RFC; RFC 6152
RFC 6152@footnote{https://tools.ietf.org/html/rfc6152.html}), accepts the @code{BODY=8BITMIME} parameter to
the @code{MAIL} command, and when present passes it to @ref{899,,process_message()}
in the @code{kwargs['mail_options']} list. @emph{decode_data} and @emph{enable_SMTPUTF8}
cannot be set to @code{True} at the same time.

@geindex process_message() (smtpd.SMTPServer method)
@anchor{library/smtpd smtpd SMTPServer process_message}@anchor{899}
@deffn {Method} process_message (peer, mailfrom, rcpttos, data, **kwargs)

Raise a @ref{8a3,,NotImplementedError} exception. Override this in subclasses to
do something useful with this message. Whatever was passed in the
constructor as @emph{remoteaddr} will be available as the @code{_remoteaddr}
attribute. @emph{peer} is the remote host’s address, @emph{mailfrom} is the envelope
originator, @emph{rcpttos} are the envelope recipients and @emph{data} is a string
containing the contents of the e-mail (which should be in 
@geindex RFC; RFC 5321
RFC 5321@footnote{https://tools.ietf.org/html/rfc5321.html}
format).

If the @emph{decode_data} constructor keyword is set to @code{True}, the @emph{data}
argument will be a unicode string.  If it is set to @code{False}, it
will be a bytes object.

@emph{kwargs} is a dictionary containing additional information. It is empty
if @code{decode_data=True} was given as an init argument, otherwise
it contains the following keys:

@quotation


@table @asis

@item @emph{mail_options}:

a list of all received parameters to the @code{MAIL}
command (the elements are uppercase strings; example:
@code{['BODY=8BITMIME', 'SMTPUTF8']}).

@item @emph{rcpt_options}:

same as @emph{mail_options} but for the @code{RCPT} command.
Currently no @code{RCPT TO} options are supported, so for now
this will always be an empty list.
@end table
@end quotation

Implementations of @code{process_message} should use the @code{**kwargs}
signature to accept arbitrary keyword arguments, since future feature
enhancements may add keys to the kwargs dictionary.

Return @code{None} to request a normal @code{250 Ok} response; otherwise
return the desired response string in 
@geindex RFC; RFC 5321
RFC 5321@footnote{https://tools.ietf.org/html/rfc5321.html} format.
@end deffn

@geindex channel_class (smtpd.SMTPServer attribute)
@anchor{library/smtpd smtpd SMTPServer channel_class}@anchor{3251}
@deffn {Attribute} channel_class

Override this in subclasses to use a custom @ref{897,,SMTPChannel} for
managing SMTP clients.
@end deffn


New in version 3.4: The @emph{map} constructor argument.



Changed in version 3.5: @emph{localaddr} and @emph{remoteaddr} may now contain IPv6 addresses.



New in version 3.5: The @emph{decode_data} and @emph{enable_SMTPUTF8} constructor parameters, and the
@emph{kwargs} parameter to @ref{899,,process_message()} when @emph{decode_data} is
@code{False}.



Changed in version 3.6: @emph{decode_data} is now @code{False} by default.

@end deffn

@node DebuggingServer Objects,PureProxy Objects,SMTPServer Objects,smtpd — SMTP Server
@anchor{library/smtpd debuggingserver-objects}@anchor{3252}
@subsubsection DebuggingServer Objects


@geindex DebuggingServer (class in smtpd)
@anchor{library/smtpd smtpd DebuggingServer}@anchor{3253}
@deffn {Class} smtpd.DebuggingServer (localaddr, remoteaddr)

Create a new debugging server.  Arguments are as per @ref{898,,SMTPServer}.
Messages will be discarded, and printed on stdout.
@end deffn

@node PureProxy Objects,SMTPChannel Objects,DebuggingServer Objects,smtpd — SMTP Server
@anchor{library/smtpd pureproxy-objects}@anchor{3254}
@subsubsection PureProxy Objects


@geindex PureProxy (class in smtpd)
@anchor{library/smtpd smtpd PureProxy}@anchor{3255}
@deffn {Class} smtpd.PureProxy (localaddr, remoteaddr)

Create a new pure proxy server. Arguments are as per @ref{898,,SMTPServer}.
Everything will be relayed to @emph{remoteaddr}.  Note that running this has a good
chance to make you into an open relay, so please be careful.
@end deffn

@node SMTPChannel Objects,,PureProxy Objects,smtpd — SMTP Server
@anchor{library/smtpd smtpchannel-objects}@anchor{3256}
@subsubsection SMTPChannel Objects


@geindex SMTPChannel (class in smtpd)
@anchor{library/smtpd smtpd SMTPChannel}@anchor{897}
@deffn {Class} smtpd.SMTPChannel (server, conn, addr, data_size_limit=33554432, map=None, enable_SMTPUTF8=False, decode_data=False)

Create a new @ref{897,,SMTPChannel} object which manages the communication
between the server and a single SMTP client.

@emph{conn} and @emph{addr} are as per the instance variables described below.

@emph{data_size_limit} specifies the maximum number of bytes that will be
accepted in a @code{DATA} command.  A value of @code{None} or @code{0} means no
limit.

@emph{enable_SMTPUTF8} determines whether the @code{SMTPUTF8} extension (as defined
in 
@geindex RFC; RFC 6531
RFC 6531@footnote{https://tools.ietf.org/html/rfc6531.html}) should be enabled.  The default is @code{False}.
@emph{decode_data} and @emph{enable_SMTPUTF8} cannot be set to @code{True} at the same
time.

A dictionary can be specified in @emph{map} to avoid using a global socket map.

@emph{decode_data} specifies whether the data portion of the SMTP transaction
should be decoded using UTF-8.  The default is @code{False}.
@emph{decode_data} and @emph{enable_SMTPUTF8} cannot be set to @code{True} at the same
time.

To use a custom SMTPChannel implementation you need to override the
@ref{3251,,SMTPServer.channel_class} of your @ref{898,,SMTPServer}.


Changed in version 3.5: The @emph{decode_data} and @emph{enable_SMTPUTF8} parameters were added.



Changed in version 3.6: @emph{decode_data} is now @code{False} by default.


The @ref{897,,SMTPChannel} has the following instance variables:

@geindex smtp_server (smtpd.SMTPChannel attribute)
@anchor{library/smtpd smtpd SMTPChannel smtp_server}@anchor{3257}
@deffn {Attribute} smtp_server

Holds the @ref{898,,SMTPServer} that spawned this channel.
@end deffn

@geindex conn (smtpd.SMTPChannel attribute)
@anchor{library/smtpd smtpd SMTPChannel conn}@anchor{3258}
@deffn {Attribute} conn

Holds the socket object connecting to the client.
@end deffn

@geindex addr (smtpd.SMTPChannel attribute)
@anchor{library/smtpd smtpd SMTPChannel addr}@anchor{3259}
@deffn {Attribute} addr

Holds the address of the client, the second value returned by
@ref{8fe,,socket.accept}
@end deffn

@geindex received_lines (smtpd.SMTPChannel attribute)
@anchor{library/smtpd smtpd SMTPChannel received_lines}@anchor{325a}
@deffn {Attribute} received_lines

Holds a list of the line strings (decoded using UTF-8) received from
the client. The lines have their @code{"\r\n"} line ending translated to
@code{"\n"}.
@end deffn

@geindex smtp_state (smtpd.SMTPChannel attribute)
@anchor{library/smtpd smtpd SMTPChannel smtp_state}@anchor{325b}
@deffn {Attribute} smtp_state

Holds the current state of the channel. This will be either
@code{COMMAND} initially and then @code{DATA} after the client sends
a “DATA” line.
@end deffn

@geindex seen_greeting (smtpd.SMTPChannel attribute)
@anchor{library/smtpd smtpd SMTPChannel seen_greeting}@anchor{325c}
@deffn {Attribute} seen_greeting

Holds a string containing the greeting sent by the client in its “HELO”.
@end deffn

@geindex mailfrom (smtpd.SMTPChannel attribute)
@anchor{library/smtpd smtpd SMTPChannel mailfrom}@anchor{325d}
@deffn {Attribute} mailfrom

Holds a string containing the address identified in the “MAIL FROM:” line
from the client.
@end deffn

@geindex rcpttos (smtpd.SMTPChannel attribute)
@anchor{library/smtpd smtpd SMTPChannel rcpttos}@anchor{325e}
@deffn {Attribute} rcpttos

Holds a list of strings containing the addresses identified in the
“RCPT TO:” lines from the client.
@end deffn

@geindex received_data (smtpd.SMTPChannel attribute)
@anchor{library/smtpd smtpd SMTPChannel received_data}@anchor{325f}
@deffn {Attribute} received_data

Holds a string containing all of the data sent by the client during the
DATA state, up to but not including the terminating @code{"\r\n.\r\n"}.
@end deffn

@geindex fqdn (smtpd.SMTPChannel attribute)
@anchor{library/smtpd smtpd SMTPChannel fqdn}@anchor{3260}
@deffn {Attribute} fqdn

Holds the fully-qualified domain name of the server as returned by
@ref{2b11,,socket.getfqdn()}.
@end deffn

@geindex peer (smtpd.SMTPChannel attribute)
@anchor{library/smtpd smtpd SMTPChannel peer}@anchor{3261}
@deffn {Attribute} peer

Holds the name of the client peer as returned by @code{conn.getpeername()}
where @code{conn} is @ref{3258,,conn}.
@end deffn

The @ref{897,,SMTPChannel} operates by invoking methods named @code{smtp_<command>}
upon reception of a command line from the client. Built into the base
@ref{897,,SMTPChannel} class are methods for handling the following commands
(and responding to them appropriately):


@multitable {xxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Command

@tab

Action taken

@item

HELO

@tab

Accepts the greeting from the client and stores it in
@ref{325c,,seen_greeting}.  Sets server to base command mode.

@item

EHLO

@tab

Accepts the greeting from the client and stores it in
@ref{325c,,seen_greeting}.  Sets server to extended command mode.

@item

NOOP

@tab

Takes no action.

@item

QUIT

@tab

Closes the connection cleanly.

@item

MAIL

@tab

Accepts the “MAIL FROM:” syntax and stores the supplied address as
@ref{325d,,mailfrom}.  In extended command mode, accepts the
@geindex RFC; RFC 1870
RFC 1870@footnote{https://tools.ietf.org/html/rfc1870.html} SIZE attribute and responds appropriately based on the
value of @emph{data_size_limit}.

@item

RCPT

@tab

Accepts the “RCPT TO:” syntax and stores the supplied addresses in
the @ref{325e,,rcpttos} list.

@item

RSET

@tab

Resets the @ref{325d,,mailfrom}, @ref{325e,,rcpttos}, and
@ref{325f,,received_data}, but not the greeting.

@item

DATA

@tab

Sets the internal state to @code{DATA} and stores remaining lines
from the client in @ref{325f,,received_data} until the terminator
@code{"\r\n.\r\n"} is received.

@item

HELP

@tab

Returns minimal information on command syntax

@item

VRFY

@tab

Returns code 252 (the server doesn’t know if the address is valid)

@item

EXPN

@tab

Reports that the command is not implemented.

@end multitable

@end deffn

@node telnetlib — Telnet client,uuid — UUID objects according to RFC 4122,smtpd — SMTP Server,Internet Protocols and Support
@anchor{library/telnetlib doc}@anchor{3262}@anchor{library/telnetlib module-telnetlib}@anchor{fc}@anchor{library/telnetlib telnetlib-telnet-client}@anchor{3263}
@subsection @code{telnetlib} — Telnet client


@geindex module; telnetlib

@strong{Source code:} Lib/telnetlib.py@footnote{https://github.com/python/cpython/tree/main/Lib/telnetlib.py}

@geindex protocol; Telnet


__________________________________________________________________


The @ref{fc,,telnetlib} module provides a @ref{832,,Telnet} class that implements the
Telnet protocol.  See 
@geindex RFC; RFC 854
RFC 854@footnote{https://tools.ietf.org/html/rfc854.html} for details about the protocol. In addition, it
provides symbolic constants for the protocol characters (see below), and for the
telnet options. The symbolic names of the telnet options follow the definitions
in @code{arpa/telnet.h}, with the leading @code{TELOPT_} removed. For symbolic names
of options which are traditionally not included in @code{arpa/telnet.h}, see the
module source itself.

The symbolic constants for the telnet commands are: IAC, DONT, DO, WONT, WILL,
SE (Subnegotiation End), NOP (No Operation), DM (Data Mark), BRK (Break), IP
(Interrupt process), AO (Abort output), AYT (Are You There), EC (Erase
Character), EL (Erase Line), GA (Go Ahead), SB (Subnegotiation Begin).

@geindex Telnet (class in telnetlib)
@anchor{library/telnetlib telnetlib Telnet}@anchor{832}
@deffn {Class} telnetlib.Telnet (host=None, port=0[, timeout])

@ref{832,,Telnet} represents a connection to a Telnet server. The instance is
initially not connected by default; the @ref{3264,,open()} method must be used to
establish a connection.  Alternatively, the host name and optional port
number can be passed to the constructor too, in which case the connection to
the server will be established before the constructor returns.  The optional
@emph{timeout} parameter specifies a timeout in seconds for blocking operations
like the connection attempt (if not specified, the global default timeout
setting will be used).

Do not reopen an already connected instance.

This class has many @code{read_*()} methods.  Note that some of them  raise
@ref{e74,,EOFError} when the end of the connection is read, because they can return
an empty string for other reasons.  See the individual descriptions below.

A @ref{832,,Telnet} object is a context manager and can be used in a
@ref{168,,with} statement.  When the @code{with} block ends, the
@ref{3265,,close()} method is called:

@example
>>> from telnetlib import Telnet
>>> with Telnet('localhost', 23) as tn:
...     tn.interact()
...
@end example


Changed in version 3.6: Context manager support added

@end deffn


@subsubheading See also



@table @asis

@geindex RFC; RFC 854

@item RFC 854@footnote{https://tools.ietf.org/html/rfc854.html} - Telnet Protocol Specification

Definition of the Telnet protocol.
@end table


@menu
* Telnet Objects:: 
* Telnet Example:: 

@end menu

@node Telnet Objects,Telnet Example,,telnetlib — Telnet client
@anchor{library/telnetlib id1}@anchor{3266}@anchor{library/telnetlib telnet-objects}@anchor{3267}
@subsubsection Telnet Objects


@ref{832,,Telnet} instances have the following methods:

@geindex read_until() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet read_until}@anchor{3268}
@deffn {Method} Telnet.read_until (expected, timeout=None)

Read until a given byte string, @emph{expected}, is encountered or until @emph{timeout}
seconds have passed.

When no match is found, return whatever is available instead, possibly empty
bytes.  Raise @ref{e74,,EOFError} if the connection is closed and no cooked data
is available.
@end deffn

@geindex read_all() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet read_all}@anchor{3269}
@deffn {Method} Telnet.read_all ()

Read all data until EOF as bytes; block until connection closed.
@end deffn

@geindex read_some() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet read_some}@anchor{326a}
@deffn {Method} Telnet.read_some ()

Read at least one byte of cooked data unless EOF is hit. Return @code{b'@w{'}} if
EOF is hit.  Block if no data is immediately available.
@end deffn

@geindex read_very_eager() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet read_very_eager}@anchor{326b}
@deffn {Method} Telnet.read_very_eager ()

Read everything that can be without blocking in I/O (eager).

Raise @ref{e74,,EOFError} if connection closed and no cooked data available.
Return @code{b'@w{'}} if no cooked data available otherwise. Do not block unless in
the midst of an IAC sequence.
@end deffn

@geindex read_eager() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet read_eager}@anchor{326c}
@deffn {Method} Telnet.read_eager ()

Read readily available data.

Raise @ref{e74,,EOFError} if connection closed and no cooked data available.
Return @code{b'@w{'}} if no cooked data available otherwise. Do not block unless in
the midst of an IAC sequence.
@end deffn

@geindex read_lazy() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet read_lazy}@anchor{326d}
@deffn {Method} Telnet.read_lazy ()

Process and return data already in the queues (lazy).

Raise @ref{e74,,EOFError} if connection closed and no data available. Return
@code{b'@w{'}} if no cooked data available otherwise.  Do not block unless in the
midst of an IAC sequence.
@end deffn

@geindex read_very_lazy() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet read_very_lazy}@anchor{326e}
@deffn {Method} Telnet.read_very_lazy ()

Return any data available in the cooked queue (very lazy).

Raise @ref{e74,,EOFError} if connection closed and no data available. Return
@code{b'@w{'}} if no cooked data available otherwise.  This method never blocks.
@end deffn

@geindex read_sb_data() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet read_sb_data}@anchor{326f}
@deffn {Method} Telnet.read_sb_data ()

Return the data collected between a SB/SE pair (suboption begin/end). The
callback should access these data when it was invoked with a @code{SE} command.
This method never blocks.
@end deffn

@geindex open() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet open}@anchor{3264}
@deffn {Method} Telnet.open (host, port=0[, timeout])

Connect to a host. The optional second argument is the port number, which
defaults to the standard Telnet port (23). The optional @emph{timeout} parameter
specifies a timeout in seconds for blocking operations like the connection
attempt (if not specified, the global default timeout setting will be used).

Do not try to reopen an already connected instance.

Raises an @ref{1130,,auditing event} @code{telnetlib.Telnet.open} with arguments @code{self}, @code{host}, @code{port}.
@end deffn

@geindex msg() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet msg}@anchor{3270}
@deffn {Method} Telnet.msg (msg, *args)

Print a debug message when the debug level is @code{>} 0. If extra arguments are
present, they are substituted in the message using the standard string
formatting operator.
@end deffn

@geindex set_debuglevel() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet set_debuglevel}@anchor{3271}
@deffn {Method} Telnet.set_debuglevel (debuglevel)

Set the debug level.  The higher the value of @emph{debuglevel}, the more debug
output you get (on @code{sys.stdout}).
@end deffn

@geindex close() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet close}@anchor{3265}
@deffn {Method} Telnet.close ()

Close the connection.
@end deffn

@geindex get_socket() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet get_socket}@anchor{3272}
@deffn {Method} Telnet.get_socket ()

Return the socket object used internally.
@end deffn

@geindex fileno() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet fileno}@anchor{3273}
@deffn {Method} Telnet.fileno ()

Return the file descriptor of the socket object used internally.
@end deffn

@geindex write() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet write}@anchor{3274}
@deffn {Method} Telnet.write (buffer)

Write a byte string to the socket, doubling any IAC characters. This can
block if the connection is blocked.  May raise @ref{2a1,,OSError} if the
connection is closed.

Raises an @ref{1130,,auditing event} @code{telnetlib.Telnet.write} with arguments @code{self}, @code{buffer}.


Changed in version 3.3: This method used to raise @ref{beb,,socket.error}, which is now an alias
of @ref{2a1,,OSError}.

@end deffn

@geindex interact() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet interact}@anchor{3275}
@deffn {Method} Telnet.interact ()

Interaction function, emulates a very dumb Telnet client.
@end deffn

@geindex mt_interact() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet mt_interact}@anchor{3276}
@deffn {Method} Telnet.mt_interact ()

Multithreaded version of @ref{3275,,interact()}.
@end deffn

@geindex expect() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet expect}@anchor{3277}
@deffn {Method} Telnet.expect (list, timeout=None)

Read until one from a list of a regular expressions matches.

The first argument is a list of regular expressions, either compiled
(@ref{b07,,regex objects}) or uncompiled (byte strings). The
optional second argument is a timeout, in seconds; the default is to block
indefinitely.

Return a tuple of three items: the index in the list of the first regular
expression that matches; the match object returned; and the bytes read up
till and including the match.

If end of file is found and no bytes were read, raise @ref{e74,,EOFError}.
Otherwise, when nothing matches, return @code{(-1, None, data)} where @emph{data} is
the bytes received so far (may be empty bytes if a timeout happened).

If a regular expression ends with a greedy match (such as @code{.*}) or if more
than one expression can match the same input, the results are
non-deterministic, and may depend on the I/O timing.
@end deffn

@geindex set_option_negotiation_callback() (telnetlib.Telnet method)
@anchor{library/telnetlib telnetlib Telnet set_option_negotiation_callback}@anchor{3278}
@deffn {Method} Telnet.set_option_negotiation_callback (callback)

Each time a telnet option is read on the input flow, this @emph{callback} (if set) is
called with the following parameters: callback(telnet socket, command
(DO/DONT/WILL/WONT), option).  No other action is done afterwards by telnetlib.
@end deffn

@node Telnet Example,,Telnet Objects,telnetlib — Telnet client
@anchor{library/telnetlib id2}@anchor{3279}@anchor{library/telnetlib telnet-example}@anchor{327a}
@subsubsection Telnet Example


A simple example illustrating typical use:

@example
import getpass
import telnetlib

HOST = "localhost"
user = input("Enter your remote account: ")
password = getpass.getpass()

tn = telnetlib.Telnet(HOST)

tn.read_until(b"login: ")
tn.write(user.encode('ascii') + b"\n")
if password:
    tn.read_until(b"Password: ")
    tn.write(password.encode('ascii') + b"\n")

tn.write(b"ls\n")
tn.write(b"exit\n")

print(tn.read_all().decode('ascii'))
@end example

@node uuid — UUID objects according to RFC 4122,socketserver — A framework for network servers,telnetlib — Telnet client,Internet Protocols and Support
@anchor{library/uuid doc}@anchor{327b}@anchor{library/uuid module-uuid}@anchor{12b}@anchor{library/uuid uuid-uuid-objects-according-to-rfc-4122}@anchor{327c}
@subsection @code{uuid} — UUID objects according to 
@geindex RFC; RFC 4122
@strong{RFC 4122}


@geindex module; uuid

@strong{Source code:} Lib/uuid.py@footnote{https://github.com/python/cpython/tree/main/Lib/uuid.py}


__________________________________________________________________


This module provides immutable @ref{577,,UUID} objects (the @ref{577,,UUID} class)
and the functions @ref{6da,,uuid1()}, @ref{327d,,uuid3()}, @ref{327e,,uuid4()}, @ref{327f,,uuid5()} for
generating version 1, 3, 4, and 5 UUIDs as specified in 
@geindex RFC; RFC 4122
RFC 4122@footnote{https://tools.ietf.org/html/rfc4122.html}.

If all you want is a unique ID, you should probably call @ref{6da,,uuid1()} or
@ref{327e,,uuid4()}.  Note that @ref{6da,,uuid1()} may compromise privacy since it creates
a UUID containing the computer’s network address.  @ref{327e,,uuid4()} creates a
random UUID.

Depending on support from the underlying platform, @ref{6da,,uuid1()} may or may
not return a “safe” UUID.  A safe UUID is one which is generated using
synchronization methods that ensure no two processes can obtain the same
UUID.  All instances of @ref{577,,UUID} have an @code{is_safe} attribute
which relays any information about the UUID’s safety, using this enumeration:

@geindex SafeUUID (class in uuid)
@anchor{library/uuid uuid SafeUUID}@anchor{3280}
@deffn {Class} uuid.SafeUUID


New in version 3.7.


@geindex safe (uuid.SafeUUID attribute)
@anchor{library/uuid uuid SafeUUID safe}@anchor{3281}
@deffn {Attribute} safe

The UUID was generated by the platform in a multiprocessing-safe way.
@end deffn

@geindex unsafe (uuid.SafeUUID attribute)
@anchor{library/uuid uuid SafeUUID unsafe}@anchor{3282}
@deffn {Attribute} unsafe

The UUID was not generated in a multiprocessing-safe way.
@end deffn

@geindex unknown (uuid.SafeUUID attribute)
@anchor{library/uuid uuid SafeUUID unknown}@anchor{3283}
@deffn {Attribute} unknown

The platform does not provide information on whether the UUID was
generated safely or not.
@end deffn
@end deffn

@geindex UUID (class in uuid)
@anchor{library/uuid uuid UUID}@anchor{577}
@deffn {Class} uuid.UUID (hex=None, bytes=None, bytes_le=None, fields=None, int=None, version=None, *, is_safe=SafeUUID.unknown)

Create a UUID from either a string of 32 hexadecimal digits, a string of 16
bytes in big-endian order as the @emph{bytes} argument, a string of 16 bytes in
little-endian order as the @emph{bytes_le} argument, a tuple of six integers
(32-bit @emph{time_low}, 16-bit @emph{time_mid}, 16-bit @emph{time_hi_version},
8-bit @emph{clock_seq_hi_variant}, 8-bit @emph{clock_seq_low}, 48-bit @emph{node}) as the
@emph{fields} argument, or a single 128-bit integer as the @emph{int} argument.
When a string of hex digits is given, curly braces, hyphens,
and a URN prefix are all optional.  For example, these
expressions all yield the same UUID:

@example
UUID('@{12345678-1234-5678-1234-567812345678@}')
UUID('12345678123456781234567812345678')
UUID('urn:uuid:12345678-1234-5678-1234-567812345678')
UUID(bytes=b'\x12\x34\x56\x78'*4)
UUID(bytes_le=b'\x78\x56\x34\x12\x34\x12\x78\x56' +
              b'\x12\x34\x56\x78\x12\x34\x56\x78')
UUID(fields=(0x12345678, 0x1234, 0x5678, 0x12, 0x34, 0x567812345678))
UUID(int=0x12345678123456781234567812345678)
@end example

Exactly one of @emph{hex}, @emph{bytes}, @emph{bytes_le}, @emph{fields}, or @emph{int} must be given.
The @emph{version} argument is optional; if given, the resulting UUID will have its
variant and version number set according to 
@geindex RFC; RFC 4122
RFC 4122@footnote{https://tools.ietf.org/html/rfc4122.html}, overriding bits in the
given @emph{hex}, @emph{bytes}, @emph{bytes_le}, @emph{fields}, or @emph{int}.

Comparison of UUID objects are made by way of comparing their
@ref{3284,,UUID.int} attributes.  Comparison with a non-UUID object
raises a @ref{162,,TypeError}.

@code{str(uuid)} returns a string in the form
@code{12345678-1234-5678-1234-567812345678} where the 32 hexadecimal digits
represent the UUID.
@end deffn

@ref{577,,UUID} instances have these read-only attributes:

@geindex bytes (uuid.UUID attribute)
@anchor{library/uuid uuid UUID bytes}@anchor{3285}
@deffn {Attribute} UUID.bytes

The UUID as a 16-byte string (containing the six integer fields in big-endian
byte order).
@end deffn

@geindex bytes_le (uuid.UUID attribute)
@anchor{library/uuid uuid UUID bytes_le}@anchor{3286}
@deffn {Attribute} UUID.bytes_le

The UUID as a 16-byte string (with @emph{time_low}, @emph{time_mid}, and @emph{time_hi_version}
in little-endian byte order).
@end deffn

@geindex fields (uuid.UUID attribute)
@anchor{library/uuid uuid UUID fields}@anchor{3287}
@deffn {Attribute} UUID.fields

A tuple of the six integer fields of the UUID, which are also available as six
individual attributes and two derived attributes:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Field

@tab

Meaning

@item

@code{time_low}

@tab

the first 32 bits of the UUID

@item

@code{time_mid}

@tab

the next 16 bits of the UUID

@item

@code{time_hi_version}

@tab

the next 16 bits of the UUID

@item

@code{clock_seq_hi_variant}

@tab

the next 8 bits of the UUID

@item

@code{clock_seq_low}

@tab

the next 8 bits of the UUID

@item

@code{node}

@tab

the last 48 bits of the UUID

@item

@ref{10a,,time}

@tab

the 60-bit timestamp

@item

@code{clock_seq}

@tab

the 14-bit sequence number

@end multitable

@end deffn

@geindex hex (uuid.UUID attribute)
@anchor{library/uuid uuid UUID hex}@anchor{3288}
@deffn {Attribute} UUID.hex

The UUID as a 32-character lowercase hexadecimal string.
@end deffn

@geindex int (uuid.UUID attribute)
@anchor{library/uuid uuid UUID int}@anchor{3284}
@deffn {Attribute} UUID.int

The UUID as a 128-bit integer.
@end deffn

@geindex urn (uuid.UUID attribute)
@anchor{library/uuid uuid UUID urn}@anchor{3289}
@deffn {Attribute} UUID.urn

The UUID as a URN as specified in 
@geindex RFC; RFC 4122
RFC 4122@footnote{https://tools.ietf.org/html/rfc4122.html}.
@end deffn

@geindex variant (uuid.UUID attribute)
@anchor{library/uuid uuid UUID variant}@anchor{328a}
@deffn {Attribute} UUID.variant

The UUID variant, which determines the internal layout of the UUID. This will be
one of the constants @ref{328b,,RESERVED_NCS}, @ref{328c,,RFC_4122},
@ref{328d,,RESERVED_MICROSOFT}, or @ref{328e,,RESERVED_FUTURE}.
@end deffn

@geindex version (uuid.UUID attribute)
@anchor{library/uuid uuid UUID version}@anchor{328f}
@deffn {Attribute} UUID.version

The UUID version number (1 through 5, meaningful only when the variant is
@ref{328c,,RFC_4122}).
@end deffn

@geindex is_safe (uuid.UUID attribute)
@anchor{library/uuid uuid UUID is_safe}@anchor{6d8}
@deffn {Attribute} UUID.is_safe

An enumeration of @ref{3280,,SafeUUID} which indicates whether the platform
generated the UUID in a multiprocessing-safe way.


New in version 3.7.

@end deffn

The @ref{12b,,uuid} module defines the following functions:

@geindex getnode() (in module uuid)
@anchor{library/uuid uuid getnode}@anchor{6d9}
@deffn {Function} uuid.getnode ()

Get the hardware address as a 48-bit positive integer.  The first time this
runs, it may launch a separate program, which could be quite slow.  If all
attempts to obtain the hardware address fail, we choose a random 48-bit
number with the multicast bit (least significant bit of the first octet)
set to 1 as recommended in 
@geindex RFC; RFC 4122
RFC 4122@footnote{https://tools.ietf.org/html/rfc4122.html}.  “Hardware address” means the MAC
address of a network interface.  On a machine with multiple network
interfaces, universally administered MAC addresses (i.e. where the second
least significant bit of the first octet is @emph{unset}) will be preferred over
locally administered MAC addresses, but with no other ordering guarantees.


Changed in version 3.7: Universally administered MAC addresses are preferred over locally
administered MAC addresses, since the former are guaranteed to be
globally unique, while the latter are not.

@end deffn

@geindex getnode

@geindex uuid1() (in module uuid)
@anchor{library/uuid uuid uuid1}@anchor{6da}
@deffn {Function} uuid.uuid1 (node=None, clock_seq=None)

Generate a UUID from a host ID, sequence number, and the current time. If @emph{node}
is not given, @ref{6d9,,getnode()} is used to obtain the hardware address. If
@emph{clock_seq} is given, it is used as the sequence number; otherwise a random
14-bit sequence number is chosen.
@end deffn

@geindex uuid1

@geindex uuid3() (in module uuid)
@anchor{library/uuid uuid uuid3}@anchor{327d}
@deffn {Function} uuid.uuid3 (namespace, name)

Generate a UUID based on the MD5 hash of a namespace identifier (which is a
UUID) and a name (which is a string).
@end deffn

@geindex uuid3

@geindex uuid4() (in module uuid)
@anchor{library/uuid uuid uuid4}@anchor{327e}
@deffn {Function} uuid.uuid4 ()

Generate a random UUID.
@end deffn

@geindex uuid4

@geindex uuid5() (in module uuid)
@anchor{library/uuid uuid uuid5}@anchor{327f}
@deffn {Function} uuid.uuid5 (namespace, name)

Generate a UUID based on the SHA-1 hash of a namespace identifier (which is a
UUID) and a name (which is a string).
@end deffn

@geindex uuid5

The @ref{12b,,uuid} module defines the following namespace identifiers for use with
@ref{327d,,uuid3()} or @ref{327f,,uuid5()}.

@geindex NAMESPACE_DNS (in module uuid)
@anchor{library/uuid uuid NAMESPACE_DNS}@anchor{3290}
@deffn {Data} uuid.NAMESPACE_DNS

When this namespace is specified, the @emph{name} string is a fully-qualified domain
name.
@end deffn

@geindex NAMESPACE_URL (in module uuid)
@anchor{library/uuid uuid NAMESPACE_URL}@anchor{3291}
@deffn {Data} uuid.NAMESPACE_URL

When this namespace is specified, the @emph{name} string is a URL.
@end deffn

@geindex NAMESPACE_OID (in module uuid)
@anchor{library/uuid uuid NAMESPACE_OID}@anchor{3292}
@deffn {Data} uuid.NAMESPACE_OID

When this namespace is specified, the @emph{name} string is an ISO OID.
@end deffn

@geindex NAMESPACE_X500 (in module uuid)
@anchor{library/uuid uuid NAMESPACE_X500}@anchor{3293}
@deffn {Data} uuid.NAMESPACE_X500

When this namespace is specified, the @emph{name} string is an X.500 DN in DER or a
text output format.
@end deffn

The @ref{12b,,uuid} module defines the following constants for the possible values
of the @code{variant} attribute:

@geindex RESERVED_NCS (in module uuid)
@anchor{library/uuid uuid RESERVED_NCS}@anchor{328b}
@deffn {Data} uuid.RESERVED_NCS

Reserved for NCS compatibility.
@end deffn

@geindex RFC_4122 (in module uuid)
@anchor{library/uuid uuid RFC_4122}@anchor{328c}
@deffn {Data} uuid.RFC_4122

Specifies the UUID layout given in 
@geindex RFC; RFC 4122
RFC 4122@footnote{https://tools.ietf.org/html/rfc4122.html}.
@end deffn

@geindex RESERVED_MICROSOFT (in module uuid)
@anchor{library/uuid uuid RESERVED_MICROSOFT}@anchor{328d}
@deffn {Data} uuid.RESERVED_MICROSOFT

Reserved for Microsoft compatibility.
@end deffn

@geindex RESERVED_FUTURE (in module uuid)
@anchor{library/uuid uuid RESERVED_FUTURE}@anchor{328e}
@deffn {Data} uuid.RESERVED_FUTURE

Reserved for future definition.
@end deffn


@subsubheading See also



@table @asis

@geindex RFC; RFC 4122

@item RFC 4122@footnote{https://tools.ietf.org/html/rfc4122.html} - A Universally Unique IDentifier (UUID) URN Namespace

This specification defines a Uniform Resource Name namespace for UUIDs, the
internal format of UUIDs, and methods of generating UUIDs.
@end table


@menu
* Example: Example<12>. 

@end menu

@node Example<12>,,,uuid — UUID objects according to RFC 4122
@anchor{library/uuid example}@anchor{3294}@anchor{library/uuid uuid-example}@anchor{3295}
@subsubsection Example


Here are some examples of typical usage of the @ref{12b,,uuid} module:

@example
>>> import uuid

>>> # make a UUID based on the host ID and current time
>>> uuid.uuid1()
UUID('a8098c1a-f86e-11da-bd1a-00112444be1e')

>>> # make a UUID using an MD5 hash of a namespace UUID and a name
>>> uuid.uuid3(uuid.NAMESPACE_DNS, 'python.org')
UUID('6fa459ea-ee8a-3ca4-894e-db77e160355e')

>>> # make a random UUID
>>> uuid.uuid4()
UUID('16fd2706-8baf-433b-82eb-8c7fada847da')

>>> # make a UUID using a SHA-1 hash of a namespace UUID and a name
>>> uuid.uuid5(uuid.NAMESPACE_DNS, 'python.org')
UUID('886313e1-3b8a-5372-9b90-0c9aee199e5d')

>>> # make a UUID from a string of hex digits (braces and hyphens ignored)
>>> x = uuid.UUID('@{00010203-0405-0607-0809-0a0b0c0d0e0f@}')

>>> # convert a UUID to a string of hex digits in standard form
>>> str(x)
'00010203-0405-0607-0809-0a0b0c0d0e0f'

>>> # get the raw 16 bytes of the UUID
>>> x.bytes
b'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f'

>>> # make a UUID from a 16-byte string
>>> uuid.UUID(bytes=x.bytes)
UUID('00010203-0405-0607-0809-0a0b0c0d0e0f')
@end example

@node socketserver — A framework for network servers,http server — HTTP servers,uuid — UUID objects according to RFC 4122,Internet Protocols and Support
@anchor{library/socketserver doc}@anchor{3296}@anchor{library/socketserver module-socketserver}@anchor{eb}@anchor{library/socketserver socketserver-a-framework-for-network-servers}@anchor{3297}
@subsection @code{socketserver} — A framework for network servers


@geindex module; socketserver

@strong{Source code:} Lib/socketserver.py@footnote{https://github.com/python/cpython/tree/main/Lib/socketserver.py}


__________________________________________________________________


The @ref{eb,,socketserver} module simplifies the task of writing network servers.

There are four basic concrete server classes:

@geindex TCPServer (class in socketserver)
@anchor{library/socketserver socketserver TCPServer}@anchor{3298}
@deffn {Class} socketserver.TCPServer (server_address, RequestHandlerClass, bind_and_activate=True)

This uses the internet TCP protocol, which provides for
continuous streams of data between the client and server.
If @emph{bind_and_activate} is true, the constructor automatically attempts to
invoke @ref{3299,,server_bind()} and
@ref{329a,,server_activate()}.  The other parameters are passed to
the @ref{cd2,,BaseServer} base class.
@end deffn

@geindex UDPServer (class in socketserver)
@anchor{library/socketserver socketserver UDPServer}@anchor{329b}
@deffn {Class} socketserver.UDPServer (server_address, RequestHandlerClass, bind_and_activate=True)

This uses datagrams, which are discrete packets of information that may
arrive out of order or be lost while in transit.  The parameters are
the same as for @ref{3298,,TCPServer}.
@end deffn

@geindex UnixStreamServer (class in socketserver)
@geindex UnixDatagramServer (class in socketserver)
@anchor{library/socketserver socketserver UnixStreamServer}@anchor{329c}
@deffn {Class} socketserver.UnixStreamServer (server_address, RequestHandlerClass, bind_and_activate=True)
@anchor{library/socketserver socketserver UnixDatagramServer}@anchor{329d}
@deffnx {Class} socketserver.UnixDatagramServer (server_address, RequestHandlerClass, bind_and_activate=True)

These more infrequently used classes are similar to the TCP and
UDP classes, but use Unix domain sockets; they’re not available on
non-Unix platforms.  The parameters are the same as for
@ref{3298,,TCPServer}.
@end deffn

These four classes process requests @emph{synchronously}; each request must be
completed before the next request can be started.  This isn’t suitable if each
request takes a long time to complete, because it requires a lot of computation,
or because it returns a lot of data which the client is slow to process.  The
solution is to create a separate process or thread to handle each request; the
@ref{6a4,,ForkingMixIn} and @ref{6a5,,ThreadingMixIn} mix-in classes can be used to
support asynchronous behaviour.

Creating a server requires several steps.  First, you must create a request
handler class by subclassing the @ref{329e,,BaseRequestHandler} class and
overriding its @ref{329f,,handle()} method;
this method will process incoming
requests.  Second, you must instantiate one of the server classes, passing it
the server’s address and the request handler class. It is recommended to use
the server in a @ref{168,,with} statement. Then call the
@ref{32a0,,handle_request()} or
@ref{cd4,,serve_forever()} method of the server object to
process one or many requests.  Finally, call @ref{32a1,,server_close()}
to close the socket (unless you used a @code{with} statement).

When inheriting from @ref{6a5,,ThreadingMixIn} for threaded connection behavior,
you should explicitly declare how you want your threads to behave on an abrupt
shutdown.  The @ref{6a5,,ThreadingMixIn} class defines an attribute
@emph{daemon_threads}, which indicates whether or not the server should wait for
thread termination.  You should set the flag explicitly if you would like
threads to behave autonomously; the default is @ref{659,,False}, meaning that
Python will not exit until all threads created by @ref{6a5,,ThreadingMixIn} have
exited.

Server classes have the same external methods and attributes, no matter what
network protocol they use.

@menu
* Server Creation Notes:: 
* Server Objects: Server Objects<2>. 
* Request Handler Objects:: 
* Examples: Examples<26>. 

@end menu

@node Server Creation Notes,Server Objects<2>,,socketserver — A framework for network servers
@anchor{library/socketserver server-creation-notes}@anchor{32a2}
@subsubsection Server Creation Notes


There are five classes in an inheritance diagram, four of which represent
synchronous servers of four types:

@example
+------------+
| BaseServer |
+------------+
      |
      v
+-----------+        +------------------+
| TCPServer |------->| UnixStreamServer |
+-----------+        +------------------+
      |
      v
+-----------+        +--------------------+
| UDPServer |------->| UnixDatagramServer |
+-----------+        +--------------------+
@end example

Note that @ref{329d,,UnixDatagramServer} derives from @ref{329b,,UDPServer}, not from
@ref{329c,,UnixStreamServer} — the only difference between an IP and a Unix
stream server is the address family, which is simply repeated in both Unix
server classes.

@geindex ForkingMixIn (class in socketserver)
@geindex ThreadingMixIn (class in socketserver)
@anchor{library/socketserver socketserver ForkingMixIn}@anchor{6a4}
@deffn {Class} socketserver.ForkingMixIn
@anchor{library/socketserver socketserver ThreadingMixIn}@anchor{6a5}
@deffnx {Class} socketserver.ThreadingMixIn

Forking and threading versions of each type of server can be created
using these mix-in classes.  For instance, @ref{32a3,,ThreadingUDPServer}
is created as follows:

@example
class ThreadingUDPServer(ThreadingMixIn, UDPServer):
    pass
@end example

The mix-in class comes first, since it overrides a method defined in
@ref{329b,,UDPServer}.  Setting the various attributes also changes the
behavior of the underlying server mechanism.

@ref{6a4,,ForkingMixIn} and the Forking classes mentioned below are
only available on POSIX platforms that support @ref{bb9,,fork()}.

@code{socketserver.ForkingMixIn.server_close()} waits until all child
processes complete, except if
@code{socketserver.ForkingMixIn.block_on_close} attribute is false.

@code{socketserver.ThreadingMixIn.server_close()} waits until all non-daemon
threads complete, except if
@code{socketserver.ThreadingMixIn.block_on_close} attribute is false. Use
daemonic threads by setting
@code{ThreadingMixIn.daemon_threads} to @code{True} to not wait until threads
complete.


Changed in version 3.7: @code{socketserver.ForkingMixIn.server_close()} and
@code{socketserver.ThreadingMixIn.server_close()} now waits until all
child processes and non-daemonic threads complete.
Add a new @code{socketserver.ForkingMixIn.block_on_close} class
attribute to opt-in for the pre-3.7 behaviour.

@end deffn

@geindex ForkingTCPServer (class in socketserver)
@geindex ForkingUDPServer (class in socketserver)
@geindex ThreadingTCPServer (class in socketserver)
@geindex ThreadingUDPServer (class in socketserver)
@anchor{library/socketserver socketserver ForkingTCPServer}@anchor{32a4}
@deffn {Class} socketserver.ForkingTCPServer
@anchor{library/socketserver socketserver ForkingUDPServer}@anchor{32a5}
@deffnx {Class} socketserver.ForkingUDPServer
@anchor{library/socketserver socketserver ThreadingTCPServer}@anchor{32a6}
@deffnx {Class} socketserver.ThreadingTCPServer
@anchor{library/socketserver socketserver ThreadingUDPServer}@anchor{32a3}
@deffnx {Class} socketserver.ThreadingUDPServer

These classes are pre-defined using the mix-in classes.
@end deffn

To implement a service, you must derive a class from @ref{329e,,BaseRequestHandler}
and redefine its @ref{329f,,handle()} method.
You can then run various versions of
the service by combining one of the server classes with your request handler
class.  The request handler class must be different for datagram or stream
services.  This can be hidden by using the handler subclasses
@ref{822,,StreamRequestHandler} or @ref{32a7,,DatagramRequestHandler}.

Of course, you still have to use your head!  For instance, it makes no sense to
use a forking server if the service contains state in memory that can be
modified by different requests, since the modifications in the child process
would never reach the initial state kept in the parent process and passed to
each child.  In this case, you can use a threading server, but you will probably
have to use locks to protect the integrity of the shared data.

On the other hand, if you are building an HTTP server where all data is stored
externally (for instance, in the file system), a synchronous class will
essentially render the service “deaf” while one request is being handled –
which may be for a very long time if a client is slow to receive all the data it
has requested.  Here a threading or forking server is appropriate.

In some cases, it may be appropriate to process part of a request synchronously,
but to finish processing in a forked child depending on the request data.  This
can be implemented by using a synchronous server and doing an explicit fork in
the request handler class @ref{329f,,handle()} method.

Another approach to handling multiple simultaneous requests in an environment
that supports neither threads nor @ref{bb9,,fork()} (or where these are too
expensive or inappropriate for the service) is to maintain an explicit table of
partially finished requests and to use @ref{e1,,selectors} to decide which
request to work on next (or whether to handle a new incoming request).  This is
particularly important for stream services where each client can potentially be
connected for a long time (if threads or subprocesses cannot be used).  See
@ref{a,,asyncore} for another way to manage this.

@c XXX should data and methods be intermingled, or separate?
@c how should the distinction between class and instance variables be drawn?

@node Server Objects<2>,Request Handler Objects,Server Creation Notes,socketserver — A framework for network servers
@anchor{library/socketserver server-objects}@anchor{32a8}
@subsubsection Server Objects


@geindex BaseServer (class in socketserver)
@anchor{library/socketserver socketserver BaseServer}@anchor{cd2}
@deffn {Class} socketserver.BaseServer (server_address, RequestHandlerClass)

This is the superclass of all Server objects in the module.  It defines the
interface, given below, but does not implement most of the methods, which is
done in subclasses.  The two parameters are stored in the respective
@ref{32a9,,server_address} and @ref{32aa,,RequestHandlerClass} attributes.

@geindex fileno() (socketserver.BaseServer method)
@anchor{library/socketserver socketserver BaseServer fileno}@anchor{32ab}
@deffn {Method} fileno ()

Return an integer file descriptor for the socket on which the server is
listening.  This function is most commonly passed to @ref{e1,,selectors}, to
allow monitoring multiple servers in the same process.
@end deffn

@geindex handle_request() (socketserver.BaseServer method)
@anchor{library/socketserver socketserver BaseServer handle_request}@anchor{32a0}
@deffn {Method} handle_request ()

Process a single request.  This function calls the following methods in
order: @ref{32ac,,get_request()}, @ref{32ad,,verify_request()}, and
@ref{32ae,,process_request()}.  If the user-provided
@ref{329f,,handle()} method of the
handler class raises an exception, the server’s @ref{893,,handle_error()} method
will be called.  If no request is received within @ref{32af,,timeout}
seconds, @ref{32b0,,handle_timeout()} will be called and @ref{32a0,,handle_request()}
will return.
@end deffn

@geindex serve_forever() (socketserver.BaseServer method)
@anchor{library/socketserver socketserver BaseServer serve_forever}@anchor{cd4}
@deffn {Method} serve_forever (poll_interval=0.5)

Handle requests until an explicit @ref{12e5,,shutdown()} request.  Poll for
shutdown every @emph{poll_interval} seconds.
Ignores the @ref{32af,,timeout} attribute.  It
also calls @ref{cd3,,service_actions()}, which may be used by a subclass or mixin
to provide actions specific to a given service.  For example, the
@ref{6a4,,ForkingMixIn} class uses @ref{cd3,,service_actions()} to clean up zombie
child processes.


Changed in version 3.3: Added @code{service_actions} call to the @code{serve_forever} method.

@end deffn

@geindex service_actions() (socketserver.BaseServer method)
@anchor{library/socketserver socketserver BaseServer service_actions}@anchor{cd3}
@deffn {Method} service_actions ()

This is called in the @ref{cd4,,serve_forever()} loop. This method can be
overridden by subclasses or mixin classes to perform actions specific to
a given service, such as cleanup actions.


New in version 3.3.

@end deffn

@geindex shutdown() (socketserver.BaseServer method)
@anchor{library/socketserver socketserver BaseServer shutdown}@anchor{12e5}
@deffn {Method} shutdown ()

Tell the @ref{cd4,,serve_forever()} loop to stop and wait until it does.
@ref{12e5,,shutdown()} must be called while @ref{cd4,,serve_forever()} is running in a
different thread otherwise it will deadlock.
@end deffn

@geindex server_close() (socketserver.BaseServer method)
@anchor{library/socketserver socketserver BaseServer server_close}@anchor{32a1}
@deffn {Method} server_close ()

Clean up the server. May be overridden.
@end deffn

@geindex address_family (socketserver.BaseServer attribute)
@anchor{library/socketserver socketserver BaseServer address_family}@anchor{32b1}
@deffn {Attribute} address_family

The family of protocols to which the server’s socket belongs.
Common examples are @ref{2a34,,socket.AF_INET} and @ref{2a3a,,socket.AF_UNIX}.
@end deffn

@geindex RequestHandlerClass (socketserver.BaseServer attribute)
@anchor{library/socketserver socketserver BaseServer RequestHandlerClass}@anchor{32aa}
@deffn {Attribute} RequestHandlerClass

The user-provided request handler class; an instance of this class is created
for each request.
@end deffn

@geindex server_address (socketserver.BaseServer attribute)
@anchor{library/socketserver socketserver BaseServer server_address}@anchor{32a9}
@deffn {Attribute} server_address

The address on which the server is listening.  The format of addresses varies
depending on the protocol family;
see the documentation for the @ref{ea,,socket} module
for details.  For internet protocols, this is a tuple containing a string giving
the address, and an integer port number: @code{('127.0.0.1', 80)}, for example.
@end deffn

@geindex socket (socketserver.BaseServer attribute)
@anchor{library/socketserver socketserver BaseServer socket}@anchor{32b2}
@deffn {Attribute} socket

The socket object on which the server will listen for incoming requests.
@end deffn

The server classes support the following class variables:

@c XXX should class variables be covered before instance variables, or vice versa?

@geindex allow_reuse_address (socketserver.BaseServer attribute)
@anchor{library/socketserver socketserver BaseServer allow_reuse_address}@anchor{32b3}
@deffn {Attribute} allow_reuse_address

Whether the server will allow the reuse of an address.  This defaults to
@ref{659,,False}, and can be set in subclasses to change the policy.
@end deffn

@geindex request_queue_size (socketserver.BaseServer attribute)
@anchor{library/socketserver socketserver BaseServer request_queue_size}@anchor{32b4}
@deffn {Attribute} request_queue_size

The size of the request queue.  If it takes a long time to process a single
request, any requests that arrive while the server is busy are placed into a
queue, up to @ref{32b4,,request_queue_size} requests.  Once the queue is full,
further requests from clients will get a “Connection denied” error.  The default
value is usually 5, but this can be overridden by subclasses.
@end deffn

@geindex socket_type (socketserver.BaseServer attribute)
@anchor{library/socketserver socketserver BaseServer socket_type}@anchor{32b5}
@deffn {Attribute} socket_type

The type of socket used by the server; @ref{e91,,socket.SOCK_STREAM} and
@ref{e90,,socket.SOCK_DGRAM} are two common values.
@end deffn

@geindex timeout (socketserver.BaseServer attribute)
@anchor{library/socketserver socketserver BaseServer timeout}@anchor{32af}
@deffn {Attribute} timeout

Timeout duration, measured in seconds, or @ref{182,,None} if no timeout is
desired.  If @ref{32a0,,handle_request()} receives no incoming requests within the
timeout period, the @ref{32b0,,handle_timeout()} method is called.
@end deffn

There are various server methods that can be overridden by subclasses of base
server classes like @ref{3298,,TCPServer}; these methods aren’t useful to external
users of the server object.

@c XXX should the default implementations of these be documented, or should
@c it be assumed that the user will look at socketserver.py?

@geindex finish_request() (socketserver.BaseServer method)
@anchor{library/socketserver socketserver BaseServer finish_request}@anchor{32b6}
@deffn {Method} finish_request (request, client_address)

Actually processes the request by instantiating @ref{32aa,,RequestHandlerClass} and
calling its @ref{329f,,handle()} method.
@end deffn

@geindex get_request() (socketserver.BaseServer method)
@anchor{library/socketserver socketserver BaseServer get_request}@anchor{32ac}
@deffn {Method} get_request ()

Must accept a request from the socket, and return a 2-tuple containing the @emph{new}
socket object to be used to communicate with the client, and the client’s
address.
@end deffn

@geindex handle_error() (socketserver.BaseServer method)
@anchor{library/socketserver socketserver BaseServer handle_error}@anchor{893}
@deffn {Method} handle_error (request, client_address)

This function is called if the @ref{329f,,handle()}
method of a @ref{32aa,,RequestHandlerClass} instance raises
an exception.  The default action is to print the traceback to
standard error and continue handling further requests.


Changed in version 3.6: Now only called for exceptions derived from the @ref{4d8,,Exception}
class.

@end deffn

@geindex handle_timeout() (socketserver.BaseServer method)
@anchor{library/socketserver socketserver BaseServer handle_timeout}@anchor{32b0}
@deffn {Method} handle_timeout ()

This function is called when the @ref{32af,,timeout} attribute has been set to a
value other than @ref{182,,None} and the timeout period has passed with no
requests being received.  The default action for forking servers is
to collect the status of any child processes that have exited, while
in threading servers this method does nothing.
@end deffn

@geindex process_request() (socketserver.BaseServer method)
@anchor{library/socketserver socketserver BaseServer process_request}@anchor{32ae}
@deffn {Method} process_request (request, client_address)

Calls @ref{32b6,,finish_request()} to create an instance of the
@ref{32aa,,RequestHandlerClass}.  If desired, this function can create a new process
or thread to handle the request; the @ref{6a4,,ForkingMixIn} and
@ref{6a5,,ThreadingMixIn} classes do this.
@end deffn

@c Is there any point in documenting the following two functions?
@c What would the purpose of overriding them be: initializing server
@c instance variables, adding new network families?

@geindex server_activate() (socketserver.BaseServer method)
@anchor{library/socketserver socketserver BaseServer server_activate}@anchor{329a}
@deffn {Method} server_activate ()

Called by the server’s constructor to activate the server.  The default behavior
for a TCP server just invokes @ref{9ca,,listen()}
on the server’s socket.  May be overridden.
@end deffn

@geindex server_bind() (socketserver.BaseServer method)
@anchor{library/socketserver socketserver BaseServer server_bind}@anchor{3299}
@deffn {Method} server_bind ()

Called by the server’s constructor to bind the socket to the desired address.
May be overridden.
@end deffn

@geindex verify_request() (socketserver.BaseServer method)
@anchor{library/socketserver socketserver BaseServer verify_request}@anchor{32ad}
@deffn {Method} verify_request (request, client_address)

Must return a Boolean value; if the value is @ref{748,,True}, the request will
be processed, and if it’s @ref{659,,False}, the request will be denied.  This
function can be overridden to implement access controls for a server. The
default implementation always returns @ref{748,,True}.
@end deffn


Changed in version 3.6: Support for the @ref{166,,context manager} protocol was added.  Exiting the
context manager is equivalent to calling @ref{32a1,,server_close()}.

@end deffn

@node Request Handler Objects,Examples<26>,Server Objects<2>,socketserver — A framework for network servers
@anchor{library/socketserver request-handler-objects}@anchor{32b7}
@subsubsection Request Handler Objects


@geindex BaseRequestHandler (class in socketserver)
@anchor{library/socketserver socketserver BaseRequestHandler}@anchor{329e}
@deffn {Class} socketserver.BaseRequestHandler

This is the superclass of all request handler objects.  It defines
the interface, given below.  A concrete request handler subclass must
define a new @ref{329f,,handle()} method, and can override any of
the other methods.  A new instance of the subclass is created for each
request.

@geindex setup() (socketserver.BaseRequestHandler method)
@anchor{library/socketserver socketserver BaseRequestHandler setup}@anchor{32b8}
@deffn {Method} setup ()

Called before the @ref{329f,,handle()} method to perform any initialization actions
required.  The default implementation does nothing.
@end deffn

@geindex handle() (socketserver.BaseRequestHandler method)
@anchor{library/socketserver socketserver BaseRequestHandler handle}@anchor{329f}
@deffn {Method} handle ()

This function must do all the work required to service a request.  The
default implementation does nothing.  Several instance attributes are
available to it; the request is available as @code{self.request}; the client
address as @code{self.client_address}; and the server instance as
@code{self.server}, in case it needs access to per-server information.

The type of @code{self.request} is different for datagram or stream
services.  For stream services, @code{self.request} is a socket object; for
datagram services, @code{self.request} is a pair of string and socket.
@end deffn

@geindex finish() (socketserver.BaseRequestHandler method)
@anchor{library/socketserver socketserver BaseRequestHandler finish}@anchor{32b9}
@deffn {Method} finish ()

Called after the @ref{329f,,handle()} method to perform any clean-up actions
required.  The default implementation does nothing.  If @ref{32b8,,setup()}
raises an exception, this function will not be called.
@end deffn
@end deffn

@geindex StreamRequestHandler (class in socketserver)
@geindex DatagramRequestHandler (class in socketserver)
@anchor{library/socketserver socketserver StreamRequestHandler}@anchor{822}
@deffn {Class} socketserver.StreamRequestHandler
@anchor{library/socketserver socketserver DatagramRequestHandler}@anchor{32a7}
@deffnx {Class} socketserver.DatagramRequestHandler

These @ref{329e,,BaseRequestHandler} subclasses override the
@ref{32b8,,setup()} and @ref{32b9,,finish()}
methods, and provide @code{self.rfile} and @code{self.wfile} attributes.
The @code{self.rfile} and @code{self.wfile} attributes can be
read or written, respectively, to get the request data or return data
to the client.

The @code{rfile} attributes of both classes support the
@ref{823,,io.BufferedIOBase} readable interface, and
@code{DatagramRequestHandler.wfile} supports the
@ref{823,,io.BufferedIOBase} writable interface.


Changed in version 3.6: @code{StreamRequestHandler.wfile} also supports the
@ref{823,,io.BufferedIOBase} writable interface.

@end deffn

@node Examples<26>,,Request Handler Objects,socketserver — A framework for network servers
@anchor{library/socketserver examples}@anchor{32ba}
@subsubsection Examples


@menu
* socketserver.TCPServer Example: socketserver TCPServer Example. 
* socketserver.UDPServer Example: socketserver UDPServer Example. 
* Asynchronous Mixins:: 

@end menu

@node socketserver TCPServer Example,socketserver UDPServer Example,,Examples<26>
@anchor{library/socketserver socketserver-tcpserver-example}@anchor{32bb}
@subsubsection @code{socketserver.TCPServer} Example


This is the server side:

@example
import socketserver

class MyTCPHandler(socketserver.BaseRequestHandler):
    """
    The request handler class for our server.

    It is instantiated once per connection to the server, and must
    override the handle() method to implement communication to the
    client.
    """

    def handle(self):
        # self.request is the TCP socket connected to the client
        self.data = self.request.recv(1024).strip()
        print("@{@} wrote:".format(self.client_address[0]))
        print(self.data)
        # just send back the same data, but upper-cased
        self.request.sendall(self.data.upper())

if __name__ == "__main__":
    HOST, PORT = "localhost", 9999

    # Create the server, binding to localhost on port 9999
    with socketserver.TCPServer((HOST, PORT), MyTCPHandler) as server:
        # Activate the server; this will keep running until you
        # interrupt the program with Ctrl-C
        server.serve_forever()
@end example

An alternative request handler class that makes use of streams (file-like
objects that simplify communication by providing the standard file interface):

@example
class MyTCPHandler(socketserver.StreamRequestHandler):

    def handle(self):
        # self.rfile is a file-like object created by the handler;
        # we can now use e.g. readline() instead of raw recv() calls
        self.data = self.rfile.readline().strip()
        print("@{@} wrote:".format(self.client_address[0]))
        print(self.data)
        # Likewise, self.wfile is a file-like object used to write back
        # to the client
        self.wfile.write(self.data.upper())
@end example

The difference is that the @code{readline()} call in the second handler will call
@code{recv()} multiple times until it encounters a newline character, while the
single @code{recv()} call in the first handler will just return what has been sent
from the client in one @code{sendall()} call.

This is the client side:

@example
import socket
import sys

HOST, PORT = "localhost", 9999
data = " ".join(sys.argv[1:])

# Create a socket (SOCK_STREAM means a TCP socket)
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
    # Connect to server and send data
    sock.connect((HOST, PORT))
    sock.sendall(bytes(data + "\n", "utf-8"))

    # Receive data from the server and shut down
    received = str(sock.recv(1024), "utf-8")

print("Sent:     @{@}".format(data))
print("Received: @{@}".format(received))
@end example

The output of the example should look something like this:

Server:

@example
$ python TCPServer.py
127.0.0.1 wrote:
b'hello world with TCP'
127.0.0.1 wrote:
b'python is nice'
@end example

Client:

@example
$ python TCPClient.py hello world with TCP
Sent:     hello world with TCP
Received: HELLO WORLD WITH TCP
$ python TCPClient.py python is nice
Sent:     python is nice
Received: PYTHON IS NICE
@end example

@node socketserver UDPServer Example,Asynchronous Mixins,socketserver TCPServer Example,Examples<26>
@anchor{library/socketserver socketserver-udpserver-example}@anchor{32bc}
@subsubsection @code{socketserver.UDPServer} Example


This is the server side:

@example
import socketserver

class MyUDPHandler(socketserver.BaseRequestHandler):
    """
    This class works similar to the TCP handler class, except that
    self.request consists of a pair of data and client socket, and since
    there is no connection the client address must be given explicitly
    when sending data back via sendto().
    """

    def handle(self):
        data = self.request[0].strip()
        socket = self.request[1]
        print("@{@} wrote:".format(self.client_address[0]))
        print(data)
        socket.sendto(data.upper(), self.client_address)

if __name__ == "__main__":
    HOST, PORT = "localhost", 9999
    with socketserver.UDPServer((HOST, PORT), MyUDPHandler) as server:
        server.serve_forever()
@end example

This is the client side:

@example
import socket
import sys

HOST, PORT = "localhost", 9999
data = " ".join(sys.argv[1:])

# SOCK_DGRAM is the socket type to use for UDP sockets
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

# As you can see, there is no connect() call; UDP has no connections.
# Instead, data is directly sent to the recipient via sendto().
sock.sendto(bytes(data + "\n", "utf-8"), (HOST, PORT))
received = str(sock.recv(1024), "utf-8")

print("Sent:     @{@}".format(data))
print("Received: @{@}".format(received))
@end example

The output of the example should look exactly like for the TCP server example.

@node Asynchronous Mixins,,socketserver UDPServer Example,Examples<26>
@anchor{library/socketserver asynchronous-mixins}@anchor{32bd}
@subsubsection Asynchronous Mixins


To build asynchronous handlers, use the @ref{6a5,,ThreadingMixIn} and
@ref{6a4,,ForkingMixIn} classes.

An example for the @ref{6a5,,ThreadingMixIn} class:

@example
import socket
import threading
import socketserver

class ThreadedTCPRequestHandler(socketserver.BaseRequestHandler):

    def handle(self):
        data = str(self.request.recv(1024), 'ascii')
        cur_thread = threading.current_thread()
        response = bytes("@{@}: @{@}".format(cur_thread.name, data), 'ascii')
        self.request.sendall(response)

class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
    pass

def client(ip, port, message):
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
        sock.connect((ip, port))
        sock.sendall(bytes(message, 'ascii'))
        response = str(sock.recv(1024), 'ascii')
        print("Received: @{@}".format(response))

if __name__ == "__main__":
    # Port 0 means to select an arbitrary unused port
    HOST, PORT = "localhost", 0

    server = ThreadedTCPServer((HOST, PORT), ThreadedTCPRequestHandler)
    with server:
        ip, port = server.server_address

        # Start a thread with the server -- that thread will then start one
        # more thread for each request
        server_thread = threading.Thread(target=server.serve_forever)
        # Exit the server thread when the main thread terminates
        server_thread.daemon = True
        server_thread.start()
        print("Server loop running in thread:", server_thread.name)

        client(ip, port, "Hello World 1")
        client(ip, port, "Hello World 2")
        client(ip, port, "Hello World 3")

        server.shutdown()
@end example

The output of the example should look something like this:

@example
$ python ThreadedTCPServer.py
Server loop running in thread: Thread-1
Received: Thread-2: Hello World 1
Received: Thread-3: Hello World 2
Received: Thread-4: Hello World 3
@end example

The @ref{6a4,,ForkingMixIn} class is used in the same way, except that the server
will spawn a new process for each request.
Available only on POSIX platforms that support @ref{bb9,,fork()}.

@node http server — HTTP servers,http cookies — HTTP state management,socketserver — A framework for network servers,Internet Protocols and Support
@anchor{library/http server doc}@anchor{32be}@anchor{library/http server http-server-http-servers}@anchor{32bf}@anchor{library/http server module-http server}@anchor{92}
@subsection @code{http.server} — HTTP servers


@geindex module; http.server

@strong{Source code:} Lib/http/server.py@footnote{https://github.com/python/cpython/tree/main/Lib/http/server.py}

@geindex WWW; server
@geindex HTTP; protocol
@geindex URL
@geindex httpd


__________________________________________________________________


This module defines classes for implementing HTTP servers.

@cartouche
@quotation Warning 
@ref{92,,http.server} is not recommended for production. It only implements
basic security checks.
@end quotation
@end cartouche

One class, @ref{308d,,HTTPServer}, is a @ref{3298,,socketserver.TCPServer} subclass.
It creates and listens at the HTTP socket, dispatching the requests to a
handler.  Code to create and run the server looks like this:

@example
def run(server_class=HTTPServer, handler_class=BaseHTTPRequestHandler):
    server_address = ('@w{'}, 8000)
    httpd = server_class(server_address, handler_class)
    httpd.serve_forever()
@end example

@geindex HTTPServer (class in http.server)
@anchor{library/http server http server HTTPServer}@anchor{308d}
@deffn {Class} http.server.HTTPServer (server_address, RequestHandlerClass)

This class builds on the @ref{3298,,TCPServer} class by storing
the server address as instance variables named @code{server_name} and
@code{server_port}. The server is accessible by the handler, typically
through the handler’s @code{server} instance variable.
@end deffn

@geindex ThreadingHTTPServer (class in http.server)
@anchor{library/http server http server ThreadingHTTPServer}@anchor{667}
@deffn {Class} http.server.ThreadingHTTPServer (server_address, RequestHandlerClass)

This class is identical to HTTPServer but uses threads to handle
requests by using the @ref{6a5,,ThreadingMixIn}. This
is useful to handle web browsers pre-opening sockets, on which
@ref{308d,,HTTPServer} would wait indefinitely.


New in version 3.7.

@end deffn

The @ref{308d,,HTTPServer} and @ref{667,,ThreadingHTTPServer} must be given
a @emph{RequestHandlerClass} on instantiation, of which this module
provides three different variants:

@geindex BaseHTTPRequestHandler (class in http.server)
@anchor{library/http server http server BaseHTTPRequestHandler}@anchor{c5d}
@deffn {Class} http.server.BaseHTTPRequestHandler (request, client_address, server)

This class is used to handle the HTTP requests that arrive at the server.  By
itself, it cannot respond to any actual HTTP requests; it must be subclassed
to handle each request method (e.g. GET or POST).
@ref{c5d,,BaseHTTPRequestHandler} provides a number of class and instance
variables, and methods for use by subclasses.

The handler will parse the request and the headers, then call a method
specific to the request type. The method name is constructed from the
request. For example, for the request method @code{SPAM}, the @code{do_SPAM()}
method will be called with no arguments. All of the relevant information is
stored in instance variables of the handler.  Subclasses should not need to
override or extend the @code{__init__()} method.

@ref{c5d,,BaseHTTPRequestHandler} has the following instance variables:

@geindex client_address (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler client_address}@anchor{32c0}
@deffn {Attribute} client_address

Contains a tuple of the form @code{(host, port)} referring to the client’s
address.
@end deffn

@geindex server (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler server}@anchor{32c1}
@deffn {Attribute} server

Contains the server instance.
@end deffn

@geindex close_connection (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler close_connection}@anchor{32c2}
@deffn {Attribute} close_connection

Boolean that should be set before @ref{32c3,,handle_one_request()} returns,
indicating if another request may be expected, or if the connection should
be shut down.
@end deffn

@geindex requestline (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler requestline}@anchor{32c4}
@deffn {Attribute} requestline

Contains the string representation of the HTTP request line. The
terminating CRLF is stripped. This attribute should be set by
@ref{32c3,,handle_one_request()}. If no valid request line was processed, it
should be set to the empty string.
@end deffn

@geindex command (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler command}@anchor{32c5}
@deffn {Attribute} command

Contains the command (request type). For example, @code{'GET'}.
@end deffn

@geindex path (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler path}@anchor{32c6}
@deffn {Attribute} path

Contains the request path. If query component of the URL is present,
then @code{path} includes the query. Using the terminology of 
@geindex RFC; RFC 3986
RFC 3986@footnote{https://tools.ietf.org/html/rfc3986.html},
@code{path} here includes @code{hier-part} and the @code{query}.
@end deffn

@geindex request_version (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler request_version}@anchor{32c7}
@deffn {Attribute} request_version

Contains the version string from the request. For example, @code{'HTTP/1.0'}.
@end deffn

@geindex headers (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler headers}@anchor{3171}
@deffn {Attribute} headers

Holds an instance of the class specified by the @ref{32c8,,MessageClass} class
variable. This instance parses and manages the headers in the HTTP
request. The @ref{1304,,parse_headers()} function from
@ref{8f,,http.client} is used to parse the headers and it requires that the
HTTP request provide a valid 
@geindex RFC; RFC 2822
RFC 2822@footnote{https://tools.ietf.org/html/rfc2822.html} style header.
@end deffn

@geindex rfile (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler rfile}@anchor{32c9}
@deffn {Attribute} rfile

An @ref{823,,io.BufferedIOBase} input stream, ready to read from
the start of the optional input data.
@end deffn

@geindex wfile (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler wfile}@anchor{32ca}
@deffn {Attribute} wfile

Contains the output stream for writing a response back to the
client. Proper adherence to the HTTP protocol must be used when writing to
this stream in order to achieve successful interoperation with HTTP
clients.


Changed in version 3.6: This is an @ref{823,,io.BufferedIOBase} stream.

@end deffn

@ref{c5d,,BaseHTTPRequestHandler} has the following attributes:

@geindex server_version (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler server_version}@anchor{32cb}
@deffn {Attribute} server_version

Specifies the server software version.  You may want to override this. The
format is multiple whitespace-separated strings, where each string is of
the form name[/version]. For example, @code{'BaseHTTP/0.2'}.
@end deffn

@geindex sys_version (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler sys_version}@anchor{32cc}
@deffn {Attribute} sys_version

Contains the Python system version, in a form usable by the
@ref{32cd,,version_string} method and the @ref{32cb,,server_version} class
variable. For example, @code{'Python/1.4'}.
@end deffn

@geindex error_message_format (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler error_message_format}@anchor{32ce}
@deffn {Attribute} error_message_format

Specifies a format string that should be used by @ref{ace,,send_error()} method
for building an error response to the client. The string is filled by
default with variables from @ref{315e,,responses} based on the status code
that passed to @ref{ace,,send_error()}.
@end deffn

@geindex error_content_type (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler error_content_type}@anchor{32cf}
@deffn {Attribute} error_content_type

Specifies the Content-Type HTTP header of error responses sent to the
client.  The default value is @code{'text/html'}.
@end deffn

@geindex protocol_version (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler protocol_version}@anchor{32d0}
@deffn {Attribute} protocol_version

This specifies the HTTP protocol version used in responses.  If set to
@code{'HTTP/1.1'}, the server will permit HTTP persistent connections;
however, your server @emph{must} then include an accurate @code{Content-Length}
header (using @ref{32d1,,send_header()}) in all of its responses to clients.
For backwards compatibility, the setting defaults to @code{'HTTP/1.0'}.
@end deffn

@geindex MessageClass (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler MessageClass}@anchor{32c8}
@deffn {Attribute} MessageClass

Specifies an @ref{7e3,,email.message.Message}-like class to parse HTTP
headers.  Typically, this is not overridden, and it defaults to
@code{http.client.HTTPMessage}.
@end deffn

@geindex responses (http.server.BaseHTTPRequestHandler attribute)
@anchor{library/http server http server BaseHTTPRequestHandler responses}@anchor{315e}
@deffn {Attribute} responses

This attribute contains a mapping of error code integers to two-element tuples
containing a short and long message. For example, @code{@{code: (shortmessage,
longmessage)@}}. The @emph{shortmessage} is usually used as the @emph{message} key in an
error response, and @emph{longmessage} as the @emph{explain} key.  It is used by
@ref{32d2,,send_response_only()} and @ref{ace,,send_error()} methods.
@end deffn

A @ref{c5d,,BaseHTTPRequestHandler} instance has the following methods:

@geindex handle() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler handle}@anchor{32d3}
@deffn {Method} handle ()

Calls @ref{32c3,,handle_one_request()} once (or, if persistent connections are
enabled, multiple times) to handle incoming HTTP requests. You should
never need to override it; instead, implement appropriate @code{do_*()}
methods.
@end deffn

@geindex handle_one_request() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler handle_one_request}@anchor{32c3}
@deffn {Method} handle_one_request ()

This method will parse and dispatch the request to the appropriate
@code{do_*()} method.  You should never need to override it.
@end deffn

@geindex handle_expect_100() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler handle_expect_100}@anchor{32d4}
@deffn {Method} handle_expect_100 ()

When a HTTP/1.1 compliant server receives an @code{Expect: 100-continue}
request header it responds back with a @code{100 Continue} followed by @code{200
OK} headers.
This method can be overridden to raise an error if the server does not
want the client to continue.  For e.g. server can choose to send @code{417
Expectation Failed} as a response header and @code{return False}.


New in version 3.2.

@end deffn

@geindex send_error() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler send_error}@anchor{ace}
@deffn {Method} send_error (code, message=None, explain=None)

Sends and logs a complete error reply to the client. The numeric @emph{code}
specifies the HTTP error code, with @emph{message} as an optional, short, human
readable description of the error.  The @emph{explain} argument can be used to
provide more detailed information about the error; it will be formatted
using the @ref{32ce,,error_message_format} attribute and emitted, after
a complete set of headers, as the response body.  The @ref{315e,,responses}
attribute holds the default values for @emph{message} and @emph{explain} that
will be used if no value is provided; for unknown codes the default value
for both is the string @code{???}. The body will be empty if the method is
HEAD or the response code is one of the following: @code{1xx},
@code{204 No Content}, @code{205 Reset Content}, @code{304 Not Modified}.


Changed in version 3.4: The error response includes a Content-Length header.
Added the @emph{explain} argument.

@end deffn

@geindex send_response() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler send_response}@anchor{32d5}
@deffn {Method} send_response (code, message=None)

Adds a response header to the headers buffer and logs the accepted
request. The HTTP response line is written to the internal buffer,
followed by @emph{Server} and @emph{Date} headers. The values for these two headers
are picked up from the @ref{32cd,,version_string()} and
@ref{32d6,,date_time_string()} methods, respectively. If the server does not
intend to send any other headers using the @ref{32d1,,send_header()} method,
then @ref{32d5,,send_response()} should be followed by an @ref{c5e,,end_headers()}
call.


Changed in version 3.3: Headers are stored to an internal buffer and @ref{c5e,,end_headers()}
needs to be called explicitly.

@end deffn

@geindex send_header() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler send_header}@anchor{32d1}
@deffn {Method} send_header (keyword, value)

Adds the HTTP header to an internal buffer which will be written to the
output stream when either @ref{c5e,,end_headers()} or @ref{c5f,,flush_headers()} is
invoked. @emph{keyword} should specify the header keyword, with @emph{value}
specifying its value. Note that, after the send_header calls are done,
@ref{c5e,,end_headers()} MUST BE called in order to complete the operation.


Changed in version 3.2: Headers are stored in an internal buffer.

@end deffn

@geindex send_response_only() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler send_response_only}@anchor{32d2}
@deffn {Method} send_response_only (code, message=None)

Sends the response header only, used for the purposes when @code{100
Continue} response is sent by the server to the client. The headers not
buffered and sent directly the output stream.If the @emph{message} is not
specified, the HTTP message corresponding the response @emph{code}  is sent.


New in version 3.2.

@end deffn

@geindex end_headers() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler end_headers}@anchor{c5e}
@deffn {Method} end_headers ()

Adds a blank line
(indicating the end of the HTTP headers in the response)
to the headers buffer and calls @ref{c5f,,flush_headers()}.


Changed in version 3.2: The buffered headers are written to the output stream.

@end deffn

@geindex flush_headers() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler flush_headers}@anchor{c5f}
@deffn {Method} flush_headers ()

Finally send the headers to the output stream and flush the internal
headers buffer.


New in version 3.3.

@end deffn

@geindex log_request() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler log_request}@anchor{32d7}
@deffn {Method} log_request (code='-', size='-')

Logs an accepted (successful) request. @emph{code} should specify the numeric
HTTP code associated with the response. If a size of the response is
available, then it should be passed as the @emph{size} parameter.
@end deffn

@geindex log_error() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler log_error}@anchor{32d8}
@deffn {Method} log_error (...)

Logs an error when a request cannot be fulfilled. By default, it passes
the message to @ref{32d9,,log_message()}, so it takes the same arguments
(@emph{format} and additional values).
@end deffn

@geindex log_message() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler log_message}@anchor{32d9}
@deffn {Method} log_message (format, ...)

Logs an arbitrary message to @code{sys.stderr}. This is typically overridden
to create custom error logging mechanisms. The @emph{format} argument is a
standard printf-style format string, where the additional arguments to
@ref{32d9,,log_message()} are applied as inputs to the formatting. The client
ip address and current date and time are prefixed to every message logged.
@end deffn

@geindex version_string() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler version_string}@anchor{32cd}
@deffn {Method} version_string ()

Returns the server software’s version string. This is a combination of the
@ref{32cb,,server_version} and @ref{32cc,,sys_version} attributes.
@end deffn

@geindex date_time_string() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler date_time_string}@anchor{32d6}
@deffn {Method} date_time_string (timestamp=None)

Returns the date and time given by @emph{timestamp} (which must be @code{None} or in
the format returned by @ref{5fc,,time.time()}), formatted for a message
header. If @emph{timestamp} is omitted, it uses the current date and time.

The result looks like @code{'Sun, 06 Nov 1994 08:49:37 GMT'}.
@end deffn

@geindex log_date_time_string() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler log_date_time_string}@anchor{32da}
@deffn {Method} log_date_time_string ()

Returns the current date and time, formatted for logging.
@end deffn

@geindex address_string() (http.server.BaseHTTPRequestHandler method)
@anchor{library/http server http server BaseHTTPRequestHandler address_string}@anchor{32db}
@deffn {Method} address_string ()

Returns the client address.


Changed in version 3.3: Previously, a name lookup was performed. To avoid name resolution
delays, it now always returns the IP address.

@end deffn
@end deffn

@geindex SimpleHTTPRequestHandler (class in http.server)
@anchor{library/http server http server SimpleHTTPRequestHandler}@anchor{666}
@deffn {Class} http.server.SimpleHTTPRequestHandler (request, client_address, server, directory=None)

This class serves files from the directory @emph{directory} and below,
or the current directory if @emph{directory} is not provided, directly
mapping the directory structure to HTTP requests.


New in version 3.7: The @emph{directory} parameter.



Changed in version 3.9: The @emph{directory} parameter accepts a @ref{63d,,path-like object}.


A lot of the work, such as parsing the request, is done by the base class
@ref{c5d,,BaseHTTPRequestHandler}.  This class implements the @ref{32dc,,do_GET()}
and @ref{32dd,,do_HEAD()} functions.

The following are defined as class-level attributes of
@ref{666,,SimpleHTTPRequestHandler}:

@geindex server_version (http.server.SimpleHTTPRequestHandler attribute)
@anchor{library/http server http server SimpleHTTPRequestHandler server_version}@anchor{32de}
@deffn {Attribute} server_version

This will be @code{"SimpleHTTP/" + __version__}, where @code{__version__} is
defined at the module level.
@end deffn

@geindex extensions_map (http.server.SimpleHTTPRequestHandler attribute)
@anchor{library/http server http server SimpleHTTPRequestHandler extensions_map}@anchor{32df}
@deffn {Attribute} extensions_map

A dictionary mapping suffixes into MIME types, contains custom overrides
for the default system mappings. The mapping is used case-insensitively,
and so should contain only lower-cased keys.


Changed in version 3.9: This dictionary is no longer filled with the default system mappings,
but only contains overrides.

@end deffn

The @ref{666,,SimpleHTTPRequestHandler} class defines the following methods:

@geindex do_HEAD() (http.server.SimpleHTTPRequestHandler method)
@anchor{library/http server http server SimpleHTTPRequestHandler do_HEAD}@anchor{32dd}
@deffn {Method} do_HEAD ()

This method serves the @code{'HEAD'} request type: it sends the headers it
would send for the equivalent @code{GET} request. See the @ref{32dc,,do_GET()}
method for a more complete explanation of the possible headers.
@end deffn

@geindex do_GET() (http.server.SimpleHTTPRequestHandler method)
@anchor{library/http server http server SimpleHTTPRequestHandler do_GET}@anchor{32dc}
@deffn {Method} do_GET ()

The request is mapped to a local file by interpreting the request as a
path relative to the current working directory.

If the request was mapped to a directory, the directory is checked for a
file named @code{index.html} or @code{index.htm} (in that order). If found, the
file’s contents are returned; otherwise a directory listing is generated
by calling the @code{list_directory()} method. This method uses
@ref{c8d,,os.listdir()} to scan the directory, and returns a @code{404} error
response if the @ref{c8d,,listdir()} fails.

If the request was mapped to a file, it is opened. Any @ref{2a1,,OSError}
exception in opening the requested file is mapped to a @code{404},
@code{'File not found'} error. If there was a @code{'If-Modified-Since'}
header in the request, and the file was not modified after this time,
a @code{304}, @code{'Not Modified'} response is sent. Otherwise, the content
type is guessed by calling the @code{guess_type()} method, which in turn
uses the @emph{extensions_map} variable, and the file contents are returned.

A @code{'Content-type:'} header with the guessed content type is output,
followed by a @code{'Content-Length:'} header with the file’s size and a
@code{'Last-Modified:'} header with the file’s modification time.

Then follows a blank line signifying the end of the headers, and then the
contents of the file are output. If the file’s MIME type starts with
@code{text/} the file is opened in text mode; otherwise binary mode is used.

For example usage, see the implementation of the @ref{ff,,test()} function
invocation in the @ref{92,,http.server} module.


Changed in version 3.7: Support of the @code{'If-Modified-Since'} header.

@end deffn
@end deffn

The @ref{666,,SimpleHTTPRequestHandler} class can be used in the following
manner in order to create a very basic webserver serving files relative to
the current directory:

@example
import http.server
import socketserver

PORT = 8000

Handler = http.server.SimpleHTTPRequestHandler

with socketserver.TCPServer(("", PORT), Handler) as httpd:
    print("serving at port", PORT)
    httpd.serve_forever()
@end example
@anchor{library/http server http-server-cli}@anchor{acf}
@ref{92,,http.server} can also be invoked directly using the @ref{611,,-m}
switch of the interpreter.  Similar to
the previous example, this serves files relative to the current directory:

@example
python -m http.server
@end example

The server listens to port 8000 by default. The default can be overridden
by passing the desired port number as an argument:

@example
python -m http.server 9000
@end example

By default, the server binds itself to all interfaces.  The option @code{-b/--bind}
specifies a specific address to which it should bind. Both IPv4 and IPv6
addresses are supported. For example, the following command causes the server
to bind to localhost only:

@example
python -m http.server --bind 127.0.0.1
@end example


New in version 3.4: @code{--bind} argument was introduced.



New in version 3.8: @code{--bind} argument enhanced to support IPv6


By default, the server uses the current directory. The option @code{-d/--directory}
specifies a directory to which it should serve the files. For example,
the following command uses a specific directory:

@example
python -m http.server --directory /tmp/
@end example


New in version 3.7: @code{--directory} argument was introduced.


@geindex CGIHTTPRequestHandler (class in http.server)
@anchor{library/http server http server CGIHTTPRequestHandler}@anchor{32e0}
@deffn {Class} http.server.CGIHTTPRequestHandler (request, client_address, server)

This class is used to serve either files or output of CGI scripts from the
current directory and below. Note that mapping HTTP hierarchic structure to
local directory structure is exactly as in @ref{666,,SimpleHTTPRequestHandler}.

@cartouche
@quotation Note 
CGI scripts run by the @ref{32e0,,CGIHTTPRequestHandler} class cannot execute
redirects (HTTP code 302), because code 200 (script output follows) is
sent prior to execution of the CGI script.  This pre-empts the status
code.
@end quotation
@end cartouche

The class will however, run the CGI script, instead of serving it as a file,
if it guesses it to be a CGI script.  Only directory-based CGI are used —
the other common server configuration is to treat special extensions as
denoting CGI scripts.

The @code{do_GET()} and @code{do_HEAD()} functions are modified to run CGI scripts
and serve the output, instead of serving files, if the request leads to
somewhere below the @code{cgi_directories} path.

The @ref{32e0,,CGIHTTPRequestHandler} defines the following data member:

@geindex cgi_directories (http.server.CGIHTTPRequestHandler attribute)
@anchor{library/http server http server CGIHTTPRequestHandler cgi_directories}@anchor{32e1}
@deffn {Attribute} cgi_directories

This defaults to @code{['/cgi-bin', '/htbin']} and describes directories to
treat as containing CGI scripts.
@end deffn

The @ref{32e0,,CGIHTTPRequestHandler} defines the following method:

@geindex do_POST() (http.server.CGIHTTPRequestHandler method)
@anchor{library/http server http server CGIHTTPRequestHandler do_POST}@anchor{32e2}
@deffn {Method} do_POST ()

This method serves the @code{'POST'} request type, only allowed for CGI
scripts.  Error 501, “Can only POST to CGI scripts”, is output when trying
to POST to a non-CGI url.
@end deffn

Note that CGI scripts will be run with UID of user nobody, for security
reasons.  Problems with the CGI script will be translated to error 403.
@end deffn

@ref{32e0,,CGIHTTPRequestHandler} can be enabled in the command line by passing
the @code{--cgi} option:

@example
python -m http.server --cgi
@end example

@node http cookies — HTTP state management,http cookiejar — Cookie handling for HTTP clients,http server — HTTP servers,Internet Protocols and Support
@anchor{library/http cookies doc}@anchor{32e3}@anchor{library/http cookies http-cookies-http-state-management}@anchor{32e4}@anchor{library/http cookies module-http cookies}@anchor{91}
@subsection @code{http.cookies} — HTTP state management


@geindex module; http.cookies

@strong{Source code:} Lib/http/cookies.py@footnote{https://github.com/python/cpython/tree/main/Lib/http/cookies.py}


__________________________________________________________________


The @ref{91,,http.cookies} module defines classes for abstracting the concept of
cookies, an HTTP state management mechanism. It supports both simple string-only
cookies, and provides an abstraction for having any serializable data-type as
cookie value.

The module formerly strictly applied the parsing rules described in the
@geindex RFC; RFC 2109
RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} and 
@geindex RFC; RFC 2068
RFC 2068@footnote{https://tools.ietf.org/html/rfc2068.html} specifications.  It has since been discovered that
MSIE 3.0x doesn’t follow the character rules outlined in those specs and also
many current day browsers and servers have relaxed parsing rules when comes to
Cookie handling.  As a result, the parsing rules used are a bit less strict.

The character set, @ref{e68,,string.ascii_letters}, @ref{1b9e,,string.digits} and
@code{!#$%&'*+-.^_`|~:} denote the set of valid characters allowed by this module
in Cookie name (as @ref{73c,,key}).


Changed in version 3.3: Allowed ‘:’ as a valid Cookie name character.


@cartouche
@quotation Note 
On encountering an invalid cookie, @ref{32e5,,CookieError} is raised, so if your
cookie data comes from a browser you should always prepare for invalid data
and catch @ref{32e5,,CookieError} on parsing.
@end quotation
@end cartouche

@geindex CookieError
@anchor{library/http cookies http cookies CookieError}@anchor{32e5}
@deffn {Exception} http.cookies.CookieError

Exception failing because of 
@geindex RFC; RFC 2109
RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} invalidity: incorrect attributes,
incorrect @code{Set-Cookie} header, etc.
@end deffn

@geindex BaseCookie (class in http.cookies)
@anchor{library/http cookies http cookies BaseCookie}@anchor{32e6}
@deffn {Class} http.cookies.BaseCookie ([input])

This class is a dictionary-like object whose keys are strings and whose values
are @ref{73f,,Morsel} instances. Note that upon setting a key to a value, the
value is first converted to a @ref{73f,,Morsel} containing the key and the value.

If @emph{input} is given, it is passed to the @ref{32e7,,load()} method.
@end deffn

@geindex SimpleCookie (class in http.cookies)
@anchor{library/http cookies http cookies SimpleCookie}@anchor{12fd}
@deffn {Class} http.cookies.SimpleCookie ([input])

This class derives from @ref{32e6,,BaseCookie} and overrides @code{value_decode()}
and @code{value_encode()}. SimpleCookie supports strings as cookie values.
When setting the value, SimpleCookie calls the builtin @ref{2ef,,str()} to convert
the value to a string. Values received from HTTP are kept as strings.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{90,,http.cookiejar}

HTTP cookie handling for web @emph{clients}.  The @ref{90,,http.cookiejar} and
@ref{91,,http.cookies} modules do not depend on each other.

@geindex RFC; RFC 2109

@item RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} - HTTP State Management Mechanism

This is the state management specification implemented by this module.
@end table


@menu
* Cookie Objects:: 
* Morsel Objects:: 
* Example: Example<13>. 

@end menu

@node Cookie Objects,Morsel Objects,,http cookies — HTTP state management
@anchor{library/http cookies cookie-objects}@anchor{32e8}@anchor{library/http cookies id1}@anchor{32e9}
@subsubsection Cookie Objects


@geindex value_decode() (http.cookies.BaseCookie method)
@anchor{library/http cookies http cookies BaseCookie value_decode}@anchor{32ea}
@deffn {Method} BaseCookie.value_decode (val)

Return a tuple @code{(real_value, coded_value)} from a string representation.
@code{real_value} can be any type. This method does no decoding in
@ref{32e6,,BaseCookie} — it exists so it can be overridden.
@end deffn

@geindex value_encode() (http.cookies.BaseCookie method)
@anchor{library/http cookies http cookies BaseCookie value_encode}@anchor{32eb}
@deffn {Method} BaseCookie.value_encode (val)

Return a tuple @code{(real_value, coded_value)}. @emph{val} can be any type, but
@code{coded_value} will always be converted to a string.
This method does no encoding in @ref{32e6,,BaseCookie} — it exists so it can
be overridden.

In general, it should be the case that @ref{32eb,,value_encode()} and
@ref{32ea,,value_decode()} are inverses on the range of @emph{value_decode}.
@end deffn

@geindex output() (http.cookies.BaseCookie method)
@anchor{library/http cookies http cookies BaseCookie output}@anchor{32ec}
@deffn {Method} BaseCookie.output (attrs=None, header='Set-Cookie:', sep='\r\n')

Return a string representation suitable to be sent as HTTP headers. @emph{attrs} and
@emph{header} are sent to each @ref{73f,,Morsel}’s @ref{32ec,,output()} method. @emph{sep} is used
to join the headers together, and is by default the combination @code{'\r\n'}
(CRLF).
@end deffn

@geindex js_output() (http.cookies.BaseCookie method)
@anchor{library/http cookies http cookies BaseCookie js_output}@anchor{32ed}
@deffn {Method} BaseCookie.js_output (attrs=None)

Return an embeddable JavaScript snippet, which, if run on a browser which
supports JavaScript, will act the same as if the HTTP headers was sent.

The meaning for @emph{attrs} is the same as in @ref{32ec,,output()}.
@end deffn

@geindex load() (http.cookies.BaseCookie method)
@anchor{library/http cookies http cookies BaseCookie load}@anchor{32e7}
@deffn {Method} BaseCookie.load (rawdata)

If @emph{rawdata} is a string, parse it as an @code{HTTP_COOKIE} and add the values
found there as @ref{73f,,Morsel}s. If it is a dictionary, it is equivalent to:

@example
for k, v in rawdata.items():
    cookie[k] = v
@end example
@end deffn

@node Morsel Objects,Example<13>,Cookie Objects,http cookies — HTTP state management
@anchor{library/http cookies id2}@anchor{32ee}@anchor{library/http cookies morsel-objects}@anchor{32ef}
@subsubsection Morsel Objects


@geindex Morsel (class in http.cookies)
@anchor{library/http cookies http cookies Morsel}@anchor{73f}
@deffn {Class} http.cookies.Morsel

Abstract a key/value pair, which has some 
@geindex RFC; RFC 2109
RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} attributes.

Morsels are dictionary-like objects, whose set of keys is constant — the valid
@geindex RFC; RFC 2109
RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} attributes, which are


@itemize *

@item 
@code{expires}

@item 
@code{path}

@item 
@code{comment}

@item 
@code{domain}

@item 
@code{max-age}

@item 
@code{secure}

@item 
@code{version}

@item 
@code{httponly}

@item 
@code{samesite}
@end itemize

The attribute @code{httponly} specifies that the cookie is only transferred
in HTTP requests, and is not accessible through JavaScript. This is intended
to mitigate some forms of cross-site scripting.

The attribute @code{samesite} specifies that the browser is not allowed to
send the cookie along with cross-site requests. This helps to mitigate CSRF
attacks. Valid values for this attribute are “Strict” and “Lax”.

The keys are case-insensitive and their default value is @code{'@w{'}}.


Changed in version 3.5: @code{__eq__()} now takes @ref{73c,,key} and @ref{73d,,value}
into account.



Changed in version 3.7: Attributes @ref{73c,,key}, @ref{73d,,value} and
@ref{73e,,coded_value} are read-only.  Use @ref{740,,set()} for
setting them.



Changed in version 3.8: Added support for the @code{samesite} attribute.

@end deffn

@geindex value (http.cookies.Morsel attribute)
@anchor{library/http cookies http cookies Morsel value}@anchor{73d}
@deffn {Attribute} Morsel.value

The value of the cookie.
@end deffn

@geindex coded_value (http.cookies.Morsel attribute)
@anchor{library/http cookies http cookies Morsel coded_value}@anchor{73e}
@deffn {Attribute} Morsel.coded_value

The encoded value of the cookie — this is what should be sent.
@end deffn

@geindex key (http.cookies.Morsel attribute)
@anchor{library/http cookies http cookies Morsel key}@anchor{73c}
@deffn {Attribute} Morsel.key

The name of the cookie.
@end deffn

@geindex set() (http.cookies.Morsel method)
@anchor{library/http cookies http cookies Morsel set}@anchor{740}
@deffn {Method} Morsel.set (key, value, coded_value)

Set the @emph{key}, @emph{value} and @emph{coded_value} attributes.
@end deffn

@geindex isReservedKey() (http.cookies.Morsel method)
@anchor{library/http cookies http cookies Morsel isReservedKey}@anchor{32f0}
@deffn {Method} Morsel.isReservedKey (K)

Whether @emph{K} is a member of the set of keys of a @ref{73f,,Morsel}.
@end deffn

@geindex output() (http.cookies.Morsel method)
@anchor{library/http cookies http cookies Morsel output}@anchor{32f1}
@deffn {Method} Morsel.output (attrs=None, header='Set-Cookie:')

Return a string representation of the Morsel, suitable to be sent as an HTTP
header. By default, all the attributes are included, unless @emph{attrs} is given, in
which case it should be a list of attributes to use. @emph{header} is by default
@code{"Set-Cookie:"}.
@end deffn

@geindex js_output() (http.cookies.Morsel method)
@anchor{library/http cookies http cookies Morsel js_output}@anchor{32f2}
@deffn {Method} Morsel.js_output (attrs=None)

Return an embeddable JavaScript snippet, which, if run on a browser which
supports JavaScript, will act the same as if the HTTP header was sent.

The meaning for @emph{attrs} is the same as in @ref{32f1,,output()}.
@end deffn

@geindex OutputString() (http.cookies.Morsel method)
@anchor{library/http cookies http cookies Morsel OutputString}@anchor{32f3}
@deffn {Method} Morsel.OutputString (attrs=None)

Return a string representing the Morsel, without any surrounding HTTP or
JavaScript.

The meaning for @emph{attrs} is the same as in @ref{32f1,,output()}.
@end deffn

@geindex update() (http.cookies.Morsel method)
@anchor{library/http cookies http cookies Morsel update}@anchor{a3b}
@deffn {Method} Morsel.update (values)

Update the values in the Morsel dictionary with the values in the dictionary
@emph{values}.  Raise an error if any of the keys in the @emph{values} dict is not a
valid 
@geindex RFC; RFC 2109
RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} attribute.


Changed in version 3.5: an error is raised for invalid keys.

@end deffn

@geindex copy() (http.cookies.Morsel method)
@anchor{library/http cookies http cookies Morsel copy}@anchor{a3a}
@deffn {Method} Morsel.copy (value)

Return a shallow copy of the Morsel object.


Changed in version 3.5: return a Morsel object instead of a dict.

@end deffn

@geindex setdefault() (http.cookies.Morsel method)
@anchor{library/http cookies http cookies Morsel setdefault}@anchor{32f4}
@deffn {Method} Morsel.setdefault (key, value=None)

Raise an error if key is not a valid 
@geindex RFC; RFC 2109
RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} attribute, otherwise
behave the same as @ref{c0f,,dict.setdefault()}.
@end deffn

@node Example<13>,,Morsel Objects,http cookies — HTTP state management
@anchor{library/http cookies cookie-example}@anchor{32f5}@anchor{library/http cookies example}@anchor{32f6}
@subsubsection Example


The following example demonstrates how to use the @ref{91,,http.cookies} module.

@example
>>> from http import cookies
>>> C = cookies.SimpleCookie()
>>> C["fig"] = "newton"
>>> C["sugar"] = "wafer"
>>> print(C) # generate HTTP headers
Set-Cookie: fig=newton
Set-Cookie: sugar=wafer
>>> print(C.output()) # same thing
Set-Cookie: fig=newton
Set-Cookie: sugar=wafer
>>> C = cookies.SimpleCookie()
>>> C["rocky"] = "road"
>>> C["rocky"]["path"] = "/cookie"
>>> print(C.output(header="Cookie:"))
Cookie: rocky=road; Path=/cookie
>>> print(C.output(attrs=[], header="Cookie:"))
Cookie: rocky=road
>>> C = cookies.SimpleCookie()
>>> C.load("chips=ahoy; vienna=finger") # load from a string (HTTP header)
>>> print(C)
Set-Cookie: chips=ahoy
Set-Cookie: vienna=finger
>>> C = cookies.SimpleCookie()
>>> C.load('keebler="E=everybody; L=\\"Loves\\"; fudge=\\012;";')
>>> print(C)
Set-Cookie: keebler="E=everybody; L=\"Loves\"; fudge=\012;"
>>> C = cookies.SimpleCookie()
>>> C["oreo"] = "doublestuff"
>>> C["oreo"]["path"] = "/"
>>> print(C)
Set-Cookie: oreo=doublestuff; Path=/
>>> C = cookies.SimpleCookie()
>>> C["twix"] = "none for you"
>>> C["twix"].value
'none for you'
>>> C = cookies.SimpleCookie()
>>> C["number"] = 7 # equivalent to C["number"] = str(7)
>>> C["string"] = "seven"
>>> C["number"].value
'7'
>>> C["string"].value
'seven'
>>> print(C)
Set-Cookie: number=7
Set-Cookie: string=seven
@end example

@node http cookiejar — Cookie handling for HTTP clients,xmlrpc — XMLRPC server and client modules,http cookies — HTTP state management,Internet Protocols and Support
@anchor{library/http cookiejar doc}@anchor{32f7}@anchor{library/http cookiejar http-cookiejar-cookie-handling-for-http-clients}@anchor{32f8}@anchor{library/http cookiejar module-http cookiejar}@anchor{90}
@subsection @code{http.cookiejar} — Cookie handling for HTTP clients


@geindex module; http.cookiejar

@strong{Source code:} Lib/http/cookiejar.py@footnote{https://github.com/python/cpython/tree/main/Lib/http/cookiejar.py}


__________________________________________________________________


The @ref{90,,http.cookiejar} module defines classes for automatic handling of HTTP
cookies.  It is useful for accessing web sites that require small pieces of data
– @emph{cookies} – to be set on the client machine by an HTTP response from a
web server, and then returned to the server in later HTTP requests.

Both the regular Netscape cookie protocol and the protocol defined by
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} are handled.  RFC 2965 handling is switched off by default.
@geindex RFC; RFC 2109
RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} cookies are parsed as Netscape cookies and subsequently treated
either as Netscape or RFC 2965 cookies according to the ‘policy’ in effect.
Note that the great majority of cookies on the internet are Netscape cookies.
@ref{90,,http.cookiejar} attempts to follow the de-facto Netscape cookie protocol (which
differs substantially from that set out in the original Netscape specification),
including taking note of the @code{max-age} and @code{port} cookie-attributes
introduced with RFC 2965.

@cartouche
@quotation Note 
The various named parameters found in @code{Set-Cookie} and
@code{Set-Cookie2} headers (eg. @code{domain} and @code{expires}) are
conventionally referred to as @emph{attributes}.  To distinguish them from
Python attributes, the documentation for this module uses the term
@emph{cookie-attribute} instead.
@end quotation
@end cartouche

The module defines the following exception:

@geindex LoadError
@anchor{library/http cookiejar http cookiejar LoadError}@anchor{32f9}
@deffn {Exception} http.cookiejar.LoadError

Instances of @ref{32fa,,FileCookieJar} raise this exception on failure to load
cookies from a file.  @ref{32f9,,LoadError} is a subclass of @ref{2a1,,OSError}.


Changed in version 3.3: LoadError was made a subclass of @ref{2a1,,OSError} instead of
@ref{be8,,IOError}.

@end deffn

The following classes are provided:

@geindex CookieJar (class in http.cookiejar)
@anchor{library/http cookiejar http cookiejar CookieJar}@anchor{30fd}
@deffn {Class} http.cookiejar.CookieJar (policy=None)

@emph{policy} is an object implementing the @ref{32fb,,CookiePolicy} interface.

The @ref{30fd,,CookieJar} class stores HTTP cookies.  It extracts cookies from HTTP
requests, and returns them in HTTP responses. @ref{30fd,,CookieJar} instances
automatically expire contained cookies when necessary.  Subclasses are also
responsible for storing and retrieving cookies from a file or database.
@end deffn

@geindex FileCookieJar (class in http.cookiejar)
@anchor{library/http cookiejar http cookiejar FileCookieJar}@anchor{32fa}
@deffn {Class} http.cookiejar.FileCookieJar (filename, delayload=None, policy=None)

@emph{policy} is an object implementing the @ref{32fb,,CookiePolicy} interface.  For the
other arguments, see the documentation for the corresponding attributes.

A @ref{30fd,,CookieJar} which can load cookies from, and perhaps save cookies to, a
file on disk.  Cookies are @strong{NOT} loaded from the named file until either the
@ref{32fc,,load()} or @ref{32fd,,revert()} method is called.  Subclasses of this class are
documented in section @ref{32fe,,FileCookieJar subclasses and co-operation with web browsers}.


Changed in version 3.8: The filename parameter supports a @ref{63d,,path-like object}.

@end deffn

@geindex CookiePolicy (class in http.cookiejar)
@anchor{library/http cookiejar http cookiejar CookiePolicy}@anchor{32fb}
@deffn {Class} http.cookiejar.CookiePolicy

This class is responsible for deciding whether each cookie should be accepted
from / returned to the server.
@end deffn

@geindex DefaultCookiePolicy (class in http.cookiejar)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy}@anchor{12bc}
@deffn {Class} http.cookiejar.DefaultCookiePolicy (blocked_domains=None, allowed_domains=None, netscape=True, rfc2965=False, rfc2109_as_netscape=None, hide_cookie2=False, strict_domain=False, strict_rfc2965_unverifiable=True, strict_ns_unverifiable=False, strict_ns_domain=DefaultCookiePolicy.DomainLiberal, strict_ns_set_initial_dollar=False, strict_ns_set_path=False, secure_protocols=('https', 'wss'))

Constructor arguments should be passed as keyword arguments only.
@emph{blocked_domains} is a sequence of domain names that we never accept cookies
from, nor return cookies to. @emph{allowed_domains} if not @ref{182,,None}, this is a
sequence of the only domains for which we accept and return cookies.
@emph{secure_protocols} is a sequence of protocols for which secure cookies can be
added to. By default @emph{https} and @emph{wss} (secure websocket) are considered
secure protocols. For all other arguments, see the documentation for
@ref{32fb,,CookiePolicy} and @ref{12bc,,DefaultCookiePolicy} objects.

@ref{12bc,,DefaultCookiePolicy} implements the standard accept / reject rules for
Netscape and 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} cookies.  By default, 
@geindex RFC; RFC 2109
RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} cookies (ie. cookies
received in a @code{Set-Cookie} header with a version cookie-attribute of
1) are treated according to the RFC 2965 rules.  However, if RFC 2965 handling
is turned off or @ref{32ff,,rfc2109_as_netscape} is @code{True}, RFC 2109 cookies are
‘downgraded’ by the @ref{30fd,,CookieJar} instance to Netscape cookies, by
setting the @code{version} attribute of the @ref{3300,,Cookie} instance to 0.
@ref{12bc,,DefaultCookiePolicy} also provides some parameters to allow some
fine-tuning of policy.
@end deffn

@geindex Cookie (class in http.cookiejar)
@anchor{library/http cookiejar http cookiejar Cookie}@anchor{3300}
@deffn {Class} http.cookiejar.Cookie

This class represents Netscape, 
@geindex RFC; RFC 2109
RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} and 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} cookies.  It is not
expected that users of @ref{90,,http.cookiejar} construct their own @ref{3300,,Cookie}
instances.  Instead, if necessary, call @code{make_cookies()} on a
@ref{30fd,,CookieJar} instance.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{127,,urllib.request}

URL opening with automatic cookie handling.

@item Module @ref{91,,http.cookies}

HTTP cookie classes, principally useful for server-side code.  The
@ref{90,,http.cookiejar} and @ref{91,,http.cookies} modules do not depend on each
other.

@item @indicateurl{https://curl.se/rfc/cookie_spec.html}

The specification of the original Netscape cookie protocol.  Though this is
still the dominant protocol, the ‘Netscape cookie protocol’ implemented by all
the major browsers (and @ref{90,,http.cookiejar}) only bears a passing resemblance to
the one sketched out in @code{cookie_spec.html}.

@geindex RFC; RFC 2109

@item RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} - HTTP State Management Mechanism

Obsoleted by 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html}. Uses @code{Set-Cookie} with version=1.

@geindex RFC; RFC 2965

@item RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} - HTTP State Management Mechanism

The Netscape protocol with the bugs fixed.  Uses @code{Set-Cookie2} in
place of @code{Set-Cookie}.  Not widely used.

@item @indicateurl{http://kristol.org/cookie/errata.html}

Unfinished errata to 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html}.
@end table

@geindex RFC; RFC 2964
RFC 2964@footnote{https://tools.ietf.org/html/rfc2964.html} - Use of HTTP State Management


@menu
* CookieJar and FileCookieJar Objects:: 
* FileCookieJar subclasses and co-operation with web browsers:: 
* CookiePolicy Objects:: 
* DefaultCookiePolicy Objects:: 
* Cookie Objects: Cookie Objects<2>. 
* Examples: Examples<27>. 

@end menu

@node CookieJar and FileCookieJar Objects,FileCookieJar subclasses and co-operation with web browsers,,http cookiejar — Cookie handling for HTTP clients
@anchor{library/http cookiejar cookie-jar-objects}@anchor{3301}@anchor{library/http cookiejar cookiejar-and-filecookiejar-objects}@anchor{3302}
@subsubsection CookieJar and FileCookieJar Objects


@ref{30fd,,CookieJar} objects support the @ref{11c3,,iterator} protocol for iterating over
contained @ref{3300,,Cookie} objects.

@ref{30fd,,CookieJar} has the following methods:

@geindex add_cookie_header() (http.cookiejar.CookieJar method)
@anchor{library/http cookiejar http cookiejar CookieJar add_cookie_header}@anchor{3303}
@deffn {Method} CookieJar.add_cookie_header (request)

Add correct @code{Cookie} header to @emph{request}.

If policy allows (ie. the @code{rfc2965} and @code{hide_cookie2} attributes of
the @ref{30fd,,CookieJar}’s @ref{32fb,,CookiePolicy} instance are true and false
respectively), the @code{Cookie2} header is also added when appropriate.

The @emph{request} object (usually a @ref{b5e,,urllib.request.Request} instance)
must support the methods @code{get_full_url()}, @code{get_host()},
@code{get_type()}, @code{unverifiable()}, @code{has_header()},
@code{get_header()}, @code{header_items()}, @code{add_unredirected_header()}
and @code{origin_req_host} attribute as documented by
@ref{127,,urllib.request}.


Changed in version 3.3: @emph{request} object needs @code{origin_req_host} attribute. Dependency on a
deprecated method @code{get_origin_req_host()} has been removed.

@end deffn

@geindex extract_cookies() (http.cookiejar.CookieJar method)
@anchor{library/http cookiejar http cookiejar CookieJar extract_cookies}@anchor{3304}
@deffn {Method} CookieJar.extract_cookies (response, request)

Extract cookies from HTTP @emph{response} and store them in the @ref{30fd,,CookieJar},
where allowed by policy.

The @ref{30fd,,CookieJar} will look for allowable @code{Set-Cookie} and
@code{Set-Cookie2} headers in the @emph{response} argument, and store cookies
as appropriate (subject to the @ref{3305,,CookiePolicy.set_ok()} method’s approval).

The @emph{response} object (usually the result of a call to
@ref{a06,,urllib.request.urlopen()}, or similar) should support an @code{info()}
method, which returns an @ref{7e3,,email.message.Message} instance.

The @emph{request} object (usually a @ref{b5e,,urllib.request.Request} instance)
must support the methods @code{get_full_url()}, @code{get_host()},
@code{unverifiable()}, and @code{origin_req_host} attribute, as documented
by @ref{127,,urllib.request}.  The request is used to set default values for
cookie-attributes as well as for checking that the cookie is allowed to be
set.


Changed in version 3.3: @emph{request} object needs @code{origin_req_host} attribute. Dependency on a
deprecated method @code{get_origin_req_host()} has been removed.

@end deffn

@geindex set_policy() (http.cookiejar.CookieJar method)
@anchor{library/http cookiejar http cookiejar CookieJar set_policy}@anchor{3306}
@deffn {Method} CookieJar.set_policy (policy)

Set the @ref{32fb,,CookiePolicy} instance to be used.
@end deffn

@geindex make_cookies() (http.cookiejar.CookieJar method)
@anchor{library/http cookiejar http cookiejar CookieJar make_cookies}@anchor{3307}
@deffn {Method} CookieJar.make_cookies (response, request)

Return sequence of @ref{3300,,Cookie} objects extracted from @emph{response} object.

See the documentation for @ref{3304,,extract_cookies()} for the interfaces required of
the @emph{response} and @emph{request} arguments.
@end deffn

@geindex set_cookie_if_ok() (http.cookiejar.CookieJar method)
@anchor{library/http cookiejar http cookiejar CookieJar set_cookie_if_ok}@anchor{3308}
@deffn {Method} CookieJar.set_cookie_if_ok (cookie, request)

Set a @ref{3300,,Cookie} if policy says it’s OK to do so.
@end deffn

@geindex set_cookie() (http.cookiejar.CookieJar method)
@anchor{library/http cookiejar http cookiejar CookieJar set_cookie}@anchor{3309}
@deffn {Method} CookieJar.set_cookie (cookie)

Set a @ref{3300,,Cookie}, without checking with policy to see whether or not it
should be set.
@end deffn

@geindex clear() (http.cookiejar.CookieJar method)
@anchor{library/http cookiejar http cookiejar CookieJar clear}@anchor{330a}
@deffn {Method} CookieJar.clear ([domain[, path[, name]]])

Clear some cookies.

If invoked without arguments, clear all cookies.  If given a single argument,
only cookies belonging to that @emph{domain} will be removed. If given two arguments,
cookies belonging to the specified @emph{domain} and URL @emph{path} are removed.  If
given three arguments, then the cookie with the specified @emph{domain}, @emph{path} and
@emph{name} is removed.

Raises @ref{5bb,,KeyError} if no matching cookie exists.
@end deffn

@geindex clear_session_cookies() (http.cookiejar.CookieJar method)
@anchor{library/http cookiejar http cookiejar CookieJar clear_session_cookies}@anchor{330b}
@deffn {Method} CookieJar.clear_session_cookies ()

Discard all session cookies.

Discards all contained cookies that have a true @code{discard} attribute
(usually because they had either no @code{max-age} or @code{expires} cookie-attribute,
or an explicit @code{discard} cookie-attribute).  For interactive browsers, the end
of a session usually corresponds to closing the browser window.

Note that the @code{save()} method won’t save session cookies anyway, unless you
ask otherwise by passing a true @emph{ignore_discard} argument.
@end deffn

@ref{32fa,,FileCookieJar} implements the following additional methods:

@geindex save() (http.cookiejar.FileCookieJar method)
@anchor{library/http cookiejar http cookiejar FileCookieJar save}@anchor{330c}
@deffn {Method} FileCookieJar.save (filename=None, ignore_discard=False, ignore_expires=False)

Save cookies to a file.

This base class raises @ref{8a3,,NotImplementedError}.  Subclasses may leave this
method unimplemented.

@emph{filename} is the name of file in which to save cookies.  If @emph{filename} is not
specified, @code{self.filename} is used (whose default is the value passed to
the constructor, if any); if @code{self.filename} is @ref{182,,None},
@ref{17c,,ValueError} is raised.

@emph{ignore_discard}: save even cookies set to be discarded. @emph{ignore_expires}: save
even cookies that have expired

The file is overwritten if it already exists, thus wiping all the cookies it
contains.  Saved cookies can be restored later using the @ref{32fc,,load()} or
@ref{32fd,,revert()} methods.
@end deffn

@geindex load() (http.cookiejar.FileCookieJar method)
@anchor{library/http cookiejar http cookiejar FileCookieJar load}@anchor{32fc}
@deffn {Method} FileCookieJar.load (filename=None, ignore_discard=False, ignore_expires=False)

Load cookies from a file.

Old cookies are kept unless overwritten by newly loaded ones.

Arguments are as for @ref{330c,,save()}.

The named file must be in the format understood by the class, or
@ref{32f9,,LoadError} will be raised.  Also, @ref{2a1,,OSError} may be raised, for
example if the file does not exist.


Changed in version 3.3: @ref{be8,,IOError} used to be raised, it is now an alias of @ref{2a1,,OSError}.

@end deffn

@geindex revert() (http.cookiejar.FileCookieJar method)
@anchor{library/http cookiejar http cookiejar FileCookieJar revert}@anchor{32fd}
@deffn {Method} FileCookieJar.revert (filename=None, ignore_discard=False, ignore_expires=False)

Clear all cookies and reload cookies from a saved file.

@ref{32fd,,revert()} can raise the same exceptions as @ref{32fc,,load()}. If there is a
failure, the object’s state will not be altered.
@end deffn

@ref{32fa,,FileCookieJar} instances have the following public attributes:

@geindex filename (http.cookiejar.FileCookieJar attribute)
@anchor{library/http cookiejar http cookiejar FileCookieJar filename}@anchor{330d}
@deffn {Attribute} FileCookieJar.filename

Filename of default file in which to keep cookies.  This attribute may be
assigned to.
@end deffn

@geindex delayload (http.cookiejar.FileCookieJar attribute)
@anchor{library/http cookiejar http cookiejar FileCookieJar delayload}@anchor{330e}
@deffn {Attribute} FileCookieJar.delayload

If true, load cookies lazily from disk.  This attribute should not be assigned
to.  This is only a hint, since this only affects performance, not behaviour
(unless the cookies on disk are changing). A @ref{30fd,,CookieJar} object may
ignore it.  None of the @ref{32fa,,FileCookieJar} classes included in the standard
library lazily loads cookies.
@end deffn

@node FileCookieJar subclasses and co-operation with web browsers,CookiePolicy Objects,CookieJar and FileCookieJar Objects,http cookiejar — Cookie handling for HTTP clients
@anchor{library/http cookiejar file-cookie-jar-classes}@anchor{32fe}@anchor{library/http cookiejar filecookiejar-subclasses-and-co-operation-with-web-browsers}@anchor{330f}
@subsubsection FileCookieJar subclasses and co-operation with web browsers


The following @ref{30fd,,CookieJar} subclasses are provided for reading and
writing.

@geindex MozillaCookieJar (class in http.cookiejar)
@anchor{library/http cookiejar http cookiejar MozillaCookieJar}@anchor{3310}
@deffn {Class} http.cookiejar.MozillaCookieJar (filename, delayload=None, policy=None)

A @ref{32fa,,FileCookieJar} that can load from and save cookies to disk in the
Mozilla @code{cookies.txt} file format (which is also used by the Lynx and Netscape
browsers).

@cartouche
@quotation Note 
This loses information about 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} cookies, and also about newer or
non-standard cookie-attributes such as @code{port}.
@end quotation
@end cartouche

@cartouche
@quotation Warning 
Back up your cookies before saving if you have cookies whose loss / corruption
would be inconvenient (there are some subtleties which may lead to slight
changes in the file over a load / save round-trip).
@end quotation
@end cartouche

Also note that cookies saved while Mozilla is running will get clobbered by
Mozilla.
@end deffn

@geindex LWPCookieJar (class in http.cookiejar)
@anchor{library/http cookiejar http cookiejar LWPCookieJar}@anchor{3311}
@deffn {Class} http.cookiejar.LWPCookieJar (filename, delayload=None, policy=None)

A @ref{32fa,,FileCookieJar} that can load from and save cookies to disk in format
compatible with the libwww-perl library’s @code{Set-Cookie3} file format.  This is
convenient if you want to store cookies in a human-readable file.


Changed in version 3.8: The filename parameter supports a @ref{63d,,path-like object}.

@end deffn

@node CookiePolicy Objects,DefaultCookiePolicy Objects,FileCookieJar subclasses and co-operation with web browsers,http cookiejar — Cookie handling for HTTP clients
@anchor{library/http cookiejar cookie-policy-objects}@anchor{3312}@anchor{library/http cookiejar cookiepolicy-objects}@anchor{3313}
@subsubsection CookiePolicy Objects


Objects implementing the @ref{32fb,,CookiePolicy} interface have the following
methods:

@geindex set_ok() (http.cookiejar.CookiePolicy method)
@anchor{library/http cookiejar http cookiejar CookiePolicy set_ok}@anchor{3305}
@deffn {Method} CookiePolicy.set_ok (cookie, request)

Return boolean value indicating whether cookie should be accepted from server.

@emph{cookie} is a @ref{3300,,Cookie} instance.  @emph{request} is an object
implementing the interface defined by the documentation for
@ref{3304,,CookieJar.extract_cookies()}.
@end deffn

@geindex return_ok() (http.cookiejar.CookiePolicy method)
@anchor{library/http cookiejar http cookiejar CookiePolicy return_ok}@anchor{3314}
@deffn {Method} CookiePolicy.return_ok (cookie, request)

Return boolean value indicating whether cookie should be returned to server.

@emph{cookie} is a @ref{3300,,Cookie} instance.  @emph{request} is an object
implementing the interface defined by the documentation for
@ref{3303,,CookieJar.add_cookie_header()}.
@end deffn

@geindex domain_return_ok() (http.cookiejar.CookiePolicy method)
@anchor{library/http cookiejar http cookiejar CookiePolicy domain_return_ok}@anchor{3315}
@deffn {Method} CookiePolicy.domain_return_ok (domain, request)

Return @code{False} if cookies should not be returned, given cookie domain.

This method is an optimization.  It removes the need for checking every cookie
with a particular domain (which might involve reading many files).  Returning
true from @ref{3315,,domain_return_ok()} and @ref{3316,,path_return_ok()} leaves all the
work to @ref{3314,,return_ok()}.

If @ref{3315,,domain_return_ok()} returns true for the cookie domain,
@ref{3316,,path_return_ok()} is called for the cookie path.  Otherwise,
@ref{3316,,path_return_ok()} and @ref{3314,,return_ok()} are never called for that cookie
domain.  If @ref{3316,,path_return_ok()} returns true, @ref{3314,,return_ok()} is called
with the @ref{3300,,Cookie} object itself for a full check.  Otherwise,
@ref{3314,,return_ok()} is never called for that cookie path.

Note that @ref{3315,,domain_return_ok()} is called for every @emph{cookie} domain, not just
for the @emph{request} domain.  For example, the function might be called with both
@code{".example.com"} and @code{"www.example.com"} if the request domain is
@code{"www.example.com"}.  The same goes for @ref{3316,,path_return_ok()}.

The @emph{request} argument is as documented for @ref{3314,,return_ok()}.
@end deffn

@geindex path_return_ok() (http.cookiejar.CookiePolicy method)
@anchor{library/http cookiejar http cookiejar CookiePolicy path_return_ok}@anchor{3316}
@deffn {Method} CookiePolicy.path_return_ok (path, request)

Return @code{False} if cookies should not be returned, given cookie path.

See the documentation for @ref{3315,,domain_return_ok()}.
@end deffn

In addition to implementing the methods above, implementations of the
@ref{32fb,,CookiePolicy} interface must also supply the following attributes,
indicating which protocols should be used, and how.  All of these attributes may
be assigned to.

@geindex netscape (http.cookiejar.CookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar CookiePolicy netscape}@anchor{3317}
@deffn {Attribute} CookiePolicy.netscape

Implement Netscape protocol.
@end deffn

@geindex rfc2965 (http.cookiejar.CookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar CookiePolicy rfc2965}@anchor{3318}
@deffn {Attribute} CookiePolicy.rfc2965

Implement 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} protocol.
@end deffn

@geindex hide_cookie2 (http.cookiejar.CookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar CookiePolicy hide_cookie2}@anchor{3319}
@deffn {Attribute} CookiePolicy.hide_cookie2

Don’t add @code{Cookie2} header to requests (the presence of this header
indicates to the server that we understand 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} cookies).
@end deffn

The most useful way to define a @ref{32fb,,CookiePolicy} class is by subclassing
from @ref{12bc,,DefaultCookiePolicy} and overriding some or all of the methods
above.  @ref{32fb,,CookiePolicy} itself may be used as a ‘null policy’ to allow
setting and receiving any and all cookies (this is unlikely to be useful).

@node DefaultCookiePolicy Objects,Cookie Objects<2>,CookiePolicy Objects,http cookiejar — Cookie handling for HTTP clients
@anchor{library/http cookiejar default-cookie-policy-objects}@anchor{331a}@anchor{library/http cookiejar defaultcookiepolicy-objects}@anchor{331b}
@subsubsection DefaultCookiePolicy Objects


Implements the standard rules for accepting and returning cookies.

Both 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} and Netscape cookies are covered.  RFC 2965 handling is switched
off by default.

The easiest way to provide your own policy is to override this class and call
its methods in your overridden implementations before adding your own additional
checks:

@example
import http.cookiejar
class MyCookiePolicy(http.cookiejar.DefaultCookiePolicy):
    def set_ok(self, cookie, request):
        if not http.cookiejar.DefaultCookiePolicy.set_ok(self, cookie, request):
            return False
        if i_dont_want_to_store_this_cookie(cookie):
            return False
        return True
@end example

In addition to the features required to implement the @ref{32fb,,CookiePolicy}
interface, this class allows you to block and allow domains from setting and
receiving cookies.  There are also some strictness switches that allow you to
tighten up the rather loose Netscape protocol rules a little bit (at the cost of
blocking some benign cookies).

A domain blocklist and allowlist is provided (both off by default). Only domains
not in the blocklist and present in the allowlist (if the allowlist is active)
participate in cookie setting and returning.  Use the @emph{blocked_domains}
constructor argument, and @code{blocked_domains()} and
@code{set_blocked_domains()} methods (and the corresponding argument and methods
for @emph{allowed_domains}).  If you set an allowlist, you can turn it off again by
setting it to @ref{182,,None}.

Domains in block or allow lists that do not start with a dot must equal the
cookie domain to be matched.  For example, @code{"example.com"} matches a blocklist
entry of @code{"example.com"}, but @code{"www.example.com"} does not.  Domains that do
start with a dot are matched by more specific domains too. For example, both
@code{"www.example.com"} and @code{"www.coyote.example.com"} match @code{".example.com"}
(but @code{"example.com"} itself does not).  IP addresses are an exception, and
must match exactly.  For example, if blocked_domains contains @code{"192.168.1.2"}
and @code{".168.1.2"}, 192.168.1.2 is blocked, but 193.168.1.2 is not.

@ref{12bc,,DefaultCookiePolicy} implements the following additional methods:

@geindex blocked_domains() (http.cookiejar.DefaultCookiePolicy method)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy blocked_domains}@anchor{331c}
@deffn {Method} DefaultCookiePolicy.blocked_domains ()

Return the sequence of blocked domains (as a tuple).
@end deffn

@geindex set_blocked_domains() (http.cookiejar.DefaultCookiePolicy method)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy set_blocked_domains}@anchor{331d}
@deffn {Method} DefaultCookiePolicy.set_blocked_domains (blocked_domains)

Set the sequence of blocked domains.
@end deffn

@geindex is_blocked() (http.cookiejar.DefaultCookiePolicy method)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy is_blocked}@anchor{331e}
@deffn {Method} DefaultCookiePolicy.is_blocked (domain)

Return @code{True} if @emph{domain} is on the blocklist for setting or receiving
cookies.
@end deffn

@geindex allowed_domains() (http.cookiejar.DefaultCookiePolicy method)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy allowed_domains}@anchor{331f}
@deffn {Method} DefaultCookiePolicy.allowed_domains ()

Return @ref{182,,None}, or the sequence of allowed domains (as a tuple).
@end deffn

@geindex set_allowed_domains() (http.cookiejar.DefaultCookiePolicy method)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy set_allowed_domains}@anchor{3320}
@deffn {Method} DefaultCookiePolicy.set_allowed_domains (allowed_domains)

Set the sequence of allowed domains, or @ref{182,,None}.
@end deffn

@geindex is_not_allowed() (http.cookiejar.DefaultCookiePolicy method)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy is_not_allowed}@anchor{3321}
@deffn {Method} DefaultCookiePolicy.is_not_allowed (domain)

Return @code{True} if @emph{domain} is not on the allowlist for setting or receiving
cookies.
@end deffn

@ref{12bc,,DefaultCookiePolicy} instances have the following attributes, which are
all initialised from the constructor arguments of the same name, and which may
all be assigned to.

@geindex rfc2109_as_netscape (http.cookiejar.DefaultCookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy rfc2109_as_netscape}@anchor{32ff}
@deffn {Attribute} DefaultCookiePolicy.rfc2109_as_netscape

If true, request that the @ref{30fd,,CookieJar} instance downgrade 
@geindex RFC; RFC 2109
RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} cookies
(ie. cookies received in a @code{Set-Cookie} header with a version
cookie-attribute of 1) to Netscape cookies by setting the version attribute of
the @ref{3300,,Cookie} instance to 0.  The default value is @ref{182,,None}, in which
case RFC 2109 cookies are downgraded if and only if 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} handling is turned
off.  Therefore, RFC 2109 cookies are downgraded by default.
@end deffn

General strictness switches:

@geindex strict_domain (http.cookiejar.DefaultCookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy strict_domain}@anchor{3322}
@deffn {Attribute} DefaultCookiePolicy.strict_domain

Don’t allow sites to set two-component domains with country-code top-level
domains like @code{.co.uk}, @code{.gov.uk}, @code{.co.nz}.etc.  This is far from perfect
and isn’t guaranteed to work!
@end deffn

@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} protocol strictness switches:

@geindex strict_rfc2965_unverifiable (http.cookiejar.DefaultCookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy strict_rfc2965_unverifiable}@anchor{3323}
@deffn {Attribute} DefaultCookiePolicy.strict_rfc2965_unverifiable

Follow 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} rules on unverifiable transactions (usually, an unverifiable
transaction is one resulting from a redirect or a request for an image hosted on
another site).  If this is false, cookies are @emph{never} blocked on the basis of
verifiability
@end deffn

Netscape protocol strictness switches:

@geindex strict_ns_unverifiable (http.cookiejar.DefaultCookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy strict_ns_unverifiable}@anchor{3324}
@deffn {Attribute} DefaultCookiePolicy.strict_ns_unverifiable

Apply 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} rules on unverifiable transactions even to Netscape cookies.
@end deffn

@geindex strict_ns_domain (http.cookiejar.DefaultCookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy strict_ns_domain}@anchor{3325}
@deffn {Attribute} DefaultCookiePolicy.strict_ns_domain

Flags indicating how strict to be with domain-matching rules for Netscape
cookies.  See below for acceptable values.
@end deffn

@geindex strict_ns_set_initial_dollar (http.cookiejar.DefaultCookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy strict_ns_set_initial_dollar}@anchor{3326}
@deffn {Attribute} DefaultCookiePolicy.strict_ns_set_initial_dollar

Ignore cookies in Set-Cookie: headers that have names starting with @code{'$'}.
@end deffn

@geindex strict_ns_set_path (http.cookiejar.DefaultCookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy strict_ns_set_path}@anchor{3327}
@deffn {Attribute} DefaultCookiePolicy.strict_ns_set_path

Don’t allow setting cookies whose path doesn’t path-match request URI.
@end deffn

@code{strict_ns_domain} is a collection of flags.  Its value is constructed by
or-ing together (for example, @code{DomainStrictNoDots|DomainStrictNonDomain} means
both flags are set).

@geindex DomainStrictNoDots (http.cookiejar.DefaultCookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy DomainStrictNoDots}@anchor{3328}
@deffn {Attribute} DefaultCookiePolicy.DomainStrictNoDots

When setting cookies, the ‘host prefix’ must not contain a dot (eg.
@code{www.foo.bar.com} can’t set a cookie for @code{.bar.com}, because @code{www.foo}
contains a dot).
@end deffn

@geindex DomainStrictNonDomain (http.cookiejar.DefaultCookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy DomainStrictNonDomain}@anchor{3329}
@deffn {Attribute} DefaultCookiePolicy.DomainStrictNonDomain

Cookies that did not explicitly specify a @code{domain} cookie-attribute can only
be returned to a domain equal to the domain that set the cookie (eg.
@code{spam.example.com} won’t be returned cookies from @code{example.com} that had no
@code{domain} cookie-attribute).
@end deffn

@geindex DomainRFC2965Match (http.cookiejar.DefaultCookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy DomainRFC2965Match}@anchor{332a}
@deffn {Attribute} DefaultCookiePolicy.DomainRFC2965Match

When setting cookies, require a full 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} domain-match.
@end deffn

The following attributes are provided for convenience, and are the most useful
combinations of the above flags:

@geindex DomainLiberal (http.cookiejar.DefaultCookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy DomainLiberal}@anchor{332b}
@deffn {Attribute} DefaultCookiePolicy.DomainLiberal

Equivalent to 0 (ie. all of the above Netscape domain strictness flags switched
off).
@end deffn

@geindex DomainStrict (http.cookiejar.DefaultCookiePolicy attribute)
@anchor{library/http cookiejar http cookiejar DefaultCookiePolicy DomainStrict}@anchor{332c}
@deffn {Attribute} DefaultCookiePolicy.DomainStrict

Equivalent to @code{DomainStrictNoDots|DomainStrictNonDomain}.
@end deffn

@node Cookie Objects<2>,Examples<27>,DefaultCookiePolicy Objects,http cookiejar — Cookie handling for HTTP clients
@anchor{library/http cookiejar cookie-objects}@anchor{332d}
@subsubsection Cookie Objects


@ref{3300,,Cookie} instances have Python attributes roughly corresponding to the
standard cookie-attributes specified in the various cookie standards.  The
correspondence is not one-to-one, because there are complicated rules for
assigning default values, because the @code{max-age} and @code{expires}
cookie-attributes contain equivalent information, and because 
@geindex RFC; RFC 2109
RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} cookies
may be ‘downgraded’ by @ref{90,,http.cookiejar} from version 1 to version 0 (Netscape)
cookies.

Assignment to these attributes should not be necessary other than in rare
circumstances in a @ref{32fb,,CookiePolicy} method.  The class does not enforce
internal consistency, so you should know what you’re doing if you do that.

@geindex version (http.cookiejar.Cookie attribute)
@anchor{library/http cookiejar http cookiejar Cookie version}@anchor{332e}
@deffn {Attribute} Cookie.version

Integer or @ref{182,,None}.  Netscape cookies have @ref{332e,,version} 0. 
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} and
@geindex RFC; RFC 2109
RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} cookies have a @code{version} cookie-attribute of 1.  However, note that
@ref{90,,http.cookiejar} may ‘downgrade’ RFC 2109 cookies to Netscape cookies, in which
case @ref{332e,,version} is 0.
@end deffn

@geindex name (http.cookiejar.Cookie attribute)
@anchor{library/http cookiejar http cookiejar Cookie name}@anchor{332f}
@deffn {Attribute} Cookie.name

Cookie name (a string).
@end deffn

@geindex value (http.cookiejar.Cookie attribute)
@anchor{library/http cookiejar http cookiejar Cookie value}@anchor{3330}
@deffn {Attribute} Cookie.value

Cookie value (a string), or @ref{182,,None}.
@end deffn

@geindex port (http.cookiejar.Cookie attribute)
@anchor{library/http cookiejar http cookiejar Cookie port}@anchor{3331}
@deffn {Attribute} Cookie.port

String representing a port or a set of ports (eg. ‘80’, or ‘80,8080’), or
@ref{182,,None}.
@end deffn

@geindex path (http.cookiejar.Cookie attribute)
@anchor{library/http cookiejar http cookiejar Cookie path}@anchor{3332}
@deffn {Attribute} Cookie.path

Cookie path (a string, eg. @code{'/acme/rocket_launchers'}).
@end deffn

@geindex secure (http.cookiejar.Cookie attribute)
@anchor{library/http cookiejar http cookiejar Cookie secure}@anchor{3333}
@deffn {Attribute} Cookie.secure

@code{True} if cookie should only be returned over a secure connection.
@end deffn

@geindex expires (http.cookiejar.Cookie attribute)
@anchor{library/http cookiejar http cookiejar Cookie expires}@anchor{3334}
@deffn {Attribute} Cookie.expires

Integer expiry date in seconds since epoch, or @ref{182,,None}.  See also the
@ref{3335,,is_expired()} method.
@end deffn

@geindex discard (http.cookiejar.Cookie attribute)
@anchor{library/http cookiejar http cookiejar Cookie discard}@anchor{3336}
@deffn {Attribute} Cookie.discard

@code{True} if this is a session cookie.
@end deffn

@geindex comment (http.cookiejar.Cookie attribute)
@anchor{library/http cookiejar http cookiejar Cookie comment}@anchor{3337}
@deffn {Attribute} Cookie.comment

String comment from the server explaining the function of this cookie, or
@ref{182,,None}.
@end deffn

@geindex comment_url (http.cookiejar.Cookie attribute)
@anchor{library/http cookiejar http cookiejar Cookie comment_url}@anchor{3338}
@deffn {Attribute} Cookie.comment_url

URL linking to a comment from the server explaining the function of this cookie,
or @ref{182,,None}.
@end deffn

@geindex rfc2109 (http.cookiejar.Cookie attribute)
@anchor{library/http cookiejar http cookiejar Cookie rfc2109}@anchor{3339}
@deffn {Attribute} Cookie.rfc2109

@code{True} if this cookie was received as an 
@geindex RFC; RFC 2109
RFC 2109@footnote{https://tools.ietf.org/html/rfc2109.html} cookie (ie. the cookie
arrived in a @code{Set-Cookie} header, and the value of the Version
cookie-attribute in that header was 1).  This attribute is provided because
@ref{90,,http.cookiejar} may ‘downgrade’ RFC 2109 cookies to Netscape cookies, in
which case @ref{332e,,version} is 0.
@end deffn

@geindex port_specified (http.cookiejar.Cookie attribute)
@anchor{library/http cookiejar http cookiejar Cookie port_specified}@anchor{333a}
@deffn {Attribute} Cookie.port_specified

@code{True} if a port or set of ports was explicitly specified by the server (in the
@code{Set-Cookie} / @code{Set-Cookie2} header).
@end deffn

@geindex domain_specified (http.cookiejar.Cookie attribute)
@anchor{library/http cookiejar http cookiejar Cookie domain_specified}@anchor{333b}
@deffn {Attribute} Cookie.domain_specified

@code{True} if a domain was explicitly specified by the server.
@end deffn

@geindex domain_initial_dot (http.cookiejar.Cookie attribute)
@anchor{library/http cookiejar http cookiejar Cookie domain_initial_dot}@anchor{333c}
@deffn {Attribute} Cookie.domain_initial_dot

@code{True} if the domain explicitly specified by the server began with a dot
(@code{'.'}).
@end deffn

Cookies may have additional non-standard cookie-attributes.  These may be
accessed using the following methods:

@geindex has_nonstandard_attr() (http.cookiejar.Cookie method)
@anchor{library/http cookiejar http cookiejar Cookie has_nonstandard_attr}@anchor{333d}
@deffn {Method} Cookie.has_nonstandard_attr (name)

Return @code{True} if cookie has the named cookie-attribute.
@end deffn

@geindex get_nonstandard_attr() (http.cookiejar.Cookie method)
@anchor{library/http cookiejar http cookiejar Cookie get_nonstandard_attr}@anchor{333e}
@deffn {Method} Cookie.get_nonstandard_attr (name, default=None)

If cookie has the named cookie-attribute, return its value. Otherwise, return
@emph{default}.
@end deffn

@geindex set_nonstandard_attr() (http.cookiejar.Cookie method)
@anchor{library/http cookiejar http cookiejar Cookie set_nonstandard_attr}@anchor{333f}
@deffn {Method} Cookie.set_nonstandard_attr (name, value)

Set the value of the named cookie-attribute.
@end deffn

The @ref{3300,,Cookie} class also defines the following method:

@geindex is_expired() (http.cookiejar.Cookie method)
@anchor{library/http cookiejar http cookiejar Cookie is_expired}@anchor{3335}
@deffn {Method} Cookie.is_expired (now=None)

@code{True} if cookie has passed the time at which the server requested it should
expire.  If @emph{now} is given (in seconds since the epoch), return whether the
cookie has expired at the specified time.
@end deffn

@node Examples<27>,,Cookie Objects<2>,http cookiejar — Cookie handling for HTTP clients
@anchor{library/http cookiejar examples}@anchor{3340}
@subsubsection Examples


The first example shows the most common usage of @ref{90,,http.cookiejar}:

@example
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://example.com/")
@end example

This example illustrates how to open a URL using your Netscape, Mozilla, or Lynx
cookies (assumes Unix/Netscape convention for location of the cookies file):

@example
import os, http.cookiejar, urllib.request
cj = http.cookiejar.MozillaCookieJar()
cj.load(os.path.join(os.path.expanduser("~"), ".netscape", "cookies.txt"))
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://example.com/")
@end example

The next example illustrates the use of @ref{12bc,,DefaultCookiePolicy}. Turn on
@geindex RFC; RFC 2965
RFC 2965@footnote{https://tools.ietf.org/html/rfc2965.html} cookies, be more strict about domains when setting and returning
Netscape cookies, and block some domains from setting cookies or having them
returned:

@example
import urllib.request
from http.cookiejar import CookieJar, DefaultCookiePolicy
policy = DefaultCookiePolicy(
    rfc2965=True, strict_ns_domain=Policy.DomainStrict,
    blocked_domains=["ads.net", ".ads.net"])
cj = CookieJar(policy)
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://example.com/")
@end example

@node xmlrpc — XMLRPC server and client modules,xmlrpc client — XML-RPC client access,http cookiejar — Cookie handling for HTTP clients,Internet Protocols and Support
@anchor{library/xmlrpc doc}@anchor{3341}@anchor{library/xmlrpc xmlrpc-xmlrpc-server-and-client-modules}@anchor{3342}
@subsection @code{xmlrpc} — XMLRPC server and client modules


XML-RPC is a Remote Procedure Call method that uses XML passed via HTTP as a
transport.  With it, a client can call methods with parameters on a remote
server (the server is named by a URI) and get back structured data.

@code{xmlrpc} is a package that collects server and client modules implementing
XML-RPC.  The modules are:


@itemize *

@item 
@ref{146,,xmlrpc.client}

@item 
@ref{147,,xmlrpc.server}
@end itemize

@node xmlrpc client — XML-RPC client access,xmlrpc server — Basic XML-RPC servers,xmlrpc — XMLRPC server and client modules,Internet Protocols and Support
@anchor{library/xmlrpc client doc}@anchor{3343}@anchor{library/xmlrpc client module-xmlrpc client}@anchor{146}@anchor{library/xmlrpc client xmlrpc-client-xml-rpc-client-access}@anchor{3344}
@subsection @code{xmlrpc.client} — XML-RPC client access


@geindex module; xmlrpc.client

@strong{Source code:} Lib/xmlrpc/client.py@footnote{https://github.com/python/cpython/tree/main/Lib/xmlrpc/client.py}

@c XXX Not everything is documented yet.  It might be good to describe
@c Marshaller, Unmarshaller, getparser and Transport.


__________________________________________________________________


XML-RPC is a Remote Procedure Call method that uses XML passed via HTTP(S) as a
transport.  With it, a client can call methods with parameters on a remote
server (the server is named by a URI) and get back structured data.  This module
supports writing XML-RPC client code; it handles all the details of translating
between conformable Python objects and XML on the wire.

@cartouche
@quotation Warning 
The @ref{146,,xmlrpc.client} module is not secure against maliciously
constructed data.  If you need to parse untrusted or unauthenticated data see
@ref{2e71,,XML vulnerabilities}.
@end quotation
@end cartouche


Changed in version 3.5: For HTTPS URIs, @ref{146,,xmlrpc.client} now performs all the necessary
certificate and hostname checks by default.


@geindex ServerProxy (class in xmlrpc.client)
@anchor{library/xmlrpc client xmlrpc client ServerProxy}@anchor{56c}
@deffn {Class} xmlrpc.client.ServerProxy (uri, transport=None, encoding=None, verbose=False, allow_none=False, use_datetime=False, use_builtin_types=False, *, headers=(), context=None)

A @ref{56c,,ServerProxy} instance is an object that manages communication with a
remote XML-RPC server.  The required first argument is a URI (Uniform Resource
Indicator), and will normally be the URL of the server.  The optional second
argument is a transport factory instance; by default it is an internal
@code{SafeTransport} instance for https: URLs and an internal HTTP
@code{Transport} instance otherwise.  The optional third argument is an
encoding, by default UTF-8. The optional fourth argument is a debugging flag.

The following parameters govern the use of the returned proxy instance.
If @emph{allow_none} is true,  the Python constant @code{None} will be translated into
XML; the default behaviour is for @code{None} to raise a @ref{162,,TypeError}. This is
a commonly-used extension to the XML-RPC specification, but isn’t supported by
all clients and servers; see http://ontosys.com/xml-rpc/extensions.php@footnote{https://web.archive.org/web/20130120074804/http://ontosys.com/xml-rpc/extensions.php}
for a description.
The @emph{use_builtin_types} flag can be used to cause date/time values
to be presented as @ref{363,,datetime.datetime} objects and binary data to be
presented as @ref{2f0,,bytes} objects; this flag is false by default.
@ref{363,,datetime.datetime}, @ref{2f0,,bytes} and @ref{2f1,,bytearray} objects
may be passed to calls.
The @emph{headers} parameter is an optional sequence of HTTP headers to send with
each request, expressed as a sequence of 2-tuples representing the header
name and value. (e.g. @cite{[(‘Header-Name’@comma{} ‘value’)]}).
The obsolete @emph{use_datetime} flag is similar to @emph{use_builtin_types} but it
applies only to date/time values.
@end deffn


Changed in version 3.3: The @emph{use_builtin_types} flag was added.



Changed in version 3.8: The @emph{headers} parameter was added.

Both the HTTP and HTTPS transports support the URL syntax extension for HTTP
Basic Authentication: @code{http://user:pass@@host:port/path}.  The  @code{user:pass}
portion will be base64-encoded as an HTTP ‘Authorization’ header, and sent to
the remote server as part of the connection process when invoking an XML-RPC
method.  You only need to use this if the remote server requires a Basic
Authentication user and password. If an HTTPS URL is provided, @emph{context} may
be @ref{2bf,,ssl.SSLContext} and configures the SSL settings of the underlying
HTTPS connection.

The returned instance is a proxy object with methods that can be used to invoke
corresponding RPC calls on the remote server.  If the remote server supports the
introspection API, the proxy can also be used to query the remote server for the
methods it supports (service discovery) and fetch other server-associated
metadata.

Types that are conformable (e.g. that can be marshalled through XML),
include the following (and except where noted, they are unmarshalled
as the same Python type):


@multitable {xxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

XML-RPC type

@tab

Python type

@item

@code{boolean}

@tab

@ref{2f7,,bool}

@item

@code{int}, @code{i1},
@code{i2},  @code{i4},
@code{i8} or
@code{biginteger}

@tab

@ref{1b1,,int} in range from -2147483648 to 2147483647.
Values get the @code{<int>} tag.

@item

@code{double} or
@code{float}

@tab

@ref{256,,float}.  Values get the @code{<double>} tag.

@item

@code{string}

@tab

@ref{2ef,,str}

@item

@code{array}

@tab

@ref{43a,,list} or @ref{3e5,,tuple} containing
conformable elements.  Arrays are returned as
@ref{43a,,lists}.

@item

@code{struct}

@tab

@ref{3b0,,dict}.  Keys must be strings, values may be
any conformable type.  Objects of user-defined
classes can be passed in; only their
@ref{7a4,,__dict__} attribute is transmitted.

@item

@code{dateTime.iso8601}

@tab

@ref{3345,,DateTime} or @ref{363,,datetime.datetime}.
Returned type depends on values of
@emph{use_builtin_types} and @emph{use_datetime} flags.

@item

@code{base64}

@tab

@ref{3346,,Binary}, @ref{2f0,,bytes} or
@ref{2f1,,bytearray}.  Returned type depends on the
value of the @emph{use_builtin_types} flag.

@item

@code{nil}

@tab

The @code{None} constant.  Passing is allowed only if
@emph{allow_none} is true.

@item

@code{bigdecimal}

@tab

@ref{248,,decimal.Decimal}.  Returned type only.

@end multitable


This is the full set of data types supported by XML-RPC.  Method calls may also
raise a special @ref{3347,,Fault} instance, used to signal XML-RPC server errors, or
@ref{3348,,ProtocolError} used to signal an error in the HTTP/HTTPS transport layer.
Both @ref{3347,,Fault} and @ref{3348,,ProtocolError} derive from a base class called
@code{Error}.  Note that the xmlrpc client module currently does not marshal
instances of subclasses of built-in types.

When passing strings, characters special to XML such as @code{<}, @code{>}, and @code{&}
will be automatically escaped.  However, it’s the caller’s responsibility to
ensure that the string is free of characters that aren’t allowed in XML, such as
the control characters with ASCII values between 0 and 31 (except, of course,
tab, newline and carriage return); failing to do this will result in an XML-RPC
request that isn’t well-formed XML.  If you have to pass arbitrary bytes
via XML-RPC, use @ref{2f0,,bytes} or @ref{2f1,,bytearray} classes or the
@ref{3346,,Binary} wrapper class described below.

@code{Server} is retained as an alias for @ref{56c,,ServerProxy} for backwards
compatibility.  New code should use @ref{56c,,ServerProxy}.


Changed in version 3.5: Added the @emph{context} argument.



Changed in version 3.6: Added support of type tags with prefixes (e.g. @code{ex:nil}).
Added support of unmarshalling additional types used by Apache XML-RPC
implementation for numerics: @code{i1}, @code{i2}, @code{i8}, @code{biginteger},
@code{float} and @code{bigdecimal}.
See @indicateurl{http://ws.apache.org/xmlrpc/types.html} for a description.




@subsubheading See also



@table @asis

@item XML-RPC HOWTO@footnote{http://www.tldp.org/HOWTO/XML-RPC-HOWTO/index.html}

A good description of XML-RPC operation and client software in several languages.
Contains pretty much everything an XML-RPC client developer needs to know.

@item XML-RPC Introspection@footnote{http://xmlrpc-c.sourceforge.net/introspection.html}

Describes the XML-RPC protocol extension for introspection.

@item XML-RPC Specification@footnote{http://xmlrpc.scripting.com/spec.html}

The official specification.
@end table


@menu
* ServerProxy Objects:: 
* DateTime Objects:: 
* Binary Objects:: 
* Fault Objects:: 
* ProtocolError Objects:: 
* MultiCall Objects:: 
* Convenience Functions:: 
* Example of Client Usage:: 
* Example of Client and Server Usage:: 

@end menu

@node ServerProxy Objects,DateTime Objects,,xmlrpc client — XML-RPC client access
@anchor{library/xmlrpc client id1}@anchor{3349}@anchor{library/xmlrpc client serverproxy-objects}@anchor{334a}
@subsubsection ServerProxy Objects


A @ref{56c,,ServerProxy} instance has a method corresponding to each remote
procedure call accepted by the XML-RPC server.  Calling the method performs an
RPC, dispatched by both name and argument signature (e.g. the same method name
can be overloaded with multiple argument signatures).  The RPC finishes by
returning a value, which may be either returned data in a conformant type or a
@ref{3347,,Fault} or @ref{3348,,ProtocolError} object indicating an error.

Servers that support the XML introspection API support some common methods
grouped under the reserved @code{system} attribute:

@geindex listMethods() (xmlrpc.client.ServerProxy.system method)
@anchor{library/xmlrpc client xmlrpc client ServerProxy system listMethods}@anchor{334b}
@deffn {Method} ServerProxy.system.listMethods ()

This method returns a list of strings, one for each (non-system) method
supported by the XML-RPC server.
@end deffn

@geindex methodSignature() (xmlrpc.client.ServerProxy.system method)
@anchor{library/xmlrpc client xmlrpc client ServerProxy system methodSignature}@anchor{334c}
@deffn {Method} ServerProxy.system.methodSignature (name)

This method takes one parameter, the name of a method implemented by the XML-RPC
server. It returns an array of possible signatures for this method. A signature
is an array of types. The first of these types is the return type of the method,
the rest are parameters.

Because multiple signatures (ie. overloading) is permitted, this method returns
a list of signatures rather than a singleton.

Signatures themselves are restricted to the top level parameters expected by a
method. For instance if a method expects one array of structs as a parameter,
and it returns a string, its signature is simply “string, array”. If it expects
three integers and returns a string, its signature is “string, int, int, int”.

If no signature is defined for the method, a non-array value is returned. In
Python this means that the type of the returned  value will be something other
than list.
@end deffn

@geindex methodHelp() (xmlrpc.client.ServerProxy.system method)
@anchor{library/xmlrpc client xmlrpc client ServerProxy system methodHelp}@anchor{334d}
@deffn {Method} ServerProxy.system.methodHelp (name)

This method takes one parameter, the name of a method implemented by the XML-RPC
server.  It returns a documentation string describing the use of that method. If
no such string is available, an empty string is returned. The documentation
string may contain HTML markup.
@end deffn


Changed in version 3.5: Instances of @ref{56c,,ServerProxy} support the @ref{166,,context manager} protocol
for closing the underlying transport.


A working example follows. The server code:

@example
from xmlrpc.server import SimpleXMLRPCServer

def is_even(n):
    return n % 2 == 0

server = SimpleXMLRPCServer(("localhost", 8000))
print("Listening on port 8000...")
server.register_function(is_even, "is_even")
server.serve_forever()
@end example

The client code for the preceding server:

@example
import xmlrpc.client

with xmlrpc.client.ServerProxy("http://localhost:8000/") as proxy:
    print("3 is even: %s" % str(proxy.is_even(3)))
    print("100 is even: %s" % str(proxy.is_even(100)))
@end example

@node DateTime Objects,Binary Objects,ServerProxy Objects,xmlrpc client — XML-RPC client access
@anchor{library/xmlrpc client datetime-objects}@anchor{334e}@anchor{library/xmlrpc client id2}@anchor{334f}
@subsubsection DateTime Objects


@geindex DateTime (class in xmlrpc.client)
@anchor{library/xmlrpc client xmlrpc client DateTime}@anchor{3345}
@deffn {Class} xmlrpc.client.DateTime

This class may be initialized with seconds since the epoch, a time
tuple, an ISO 8601 time/date string, or a @ref{363,,datetime.datetime}
instance.  It has the following methods, supported mainly for internal
use by the marshalling/unmarshalling code:

@geindex decode() (xmlrpc.client.DateTime method)
@anchor{library/xmlrpc client xmlrpc client DateTime decode}@anchor{3350}
@deffn {Method} decode (string)

Accept a string as the instance’s new time value.
@end deffn

@geindex encode() (xmlrpc.client.DateTime method)
@anchor{library/xmlrpc client xmlrpc client DateTime encode}@anchor{3351}
@deffn {Method} encode (out)

Write the XML-RPC encoding of this @ref{3345,,DateTime} item to the @emph{out} stream
object.
@end deffn

It also supports certain of Python’s built-in operators through rich comparison
and @code{__repr__()} methods.
@end deffn

A working example follows. The server code:

@example
import datetime
from xmlrpc.server import SimpleXMLRPCServer
import xmlrpc.client

def today():
    today = datetime.datetime.today()
    return xmlrpc.client.DateTime(today)

server = SimpleXMLRPCServer(("localhost", 8000))
print("Listening on port 8000...")
server.register_function(today, "today")
server.serve_forever()
@end example

The client code for the preceding server:

@example
import xmlrpc.client
import datetime

proxy = xmlrpc.client.ServerProxy("http://localhost:8000/")

today = proxy.today()
# convert the ISO8601 string to a datetime object
converted = datetime.datetime.strptime(today.value, "%Y%m%dT%H:%M:%S")
print("Today: %s" % converted.strftime("%d.%m.%Y, %H:%M"))
@end example

@node Binary Objects,Fault Objects,DateTime Objects,xmlrpc client — XML-RPC client access
@anchor{library/xmlrpc client binary-objects}@anchor{3352}@anchor{library/xmlrpc client id3}@anchor{3353}
@subsubsection Binary Objects


@geindex Binary (class in xmlrpc.client)
@anchor{library/xmlrpc client xmlrpc client Binary}@anchor{3346}
@deffn {Class} xmlrpc.client.Binary

This class may be initialized from bytes data (which may include NULs). The
primary access to the content of a @ref{3346,,Binary} object is provided by an
attribute:

@geindex data (xmlrpc.client.Binary attribute)
@anchor{library/xmlrpc client xmlrpc client Binary data}@anchor{3354}
@deffn {Attribute} data

The binary data encapsulated by the @ref{3346,,Binary} instance.  The data is
provided as a @ref{2f0,,bytes} object.
@end deffn

@ref{3346,,Binary} objects have the following methods, supported mainly for
internal use by the marshalling/unmarshalling code:

@geindex decode() (xmlrpc.client.Binary method)
@anchor{library/xmlrpc client xmlrpc client Binary decode}@anchor{3355}
@deffn {Method} decode (bytes)

Accept a base64 @ref{2f0,,bytes} object and decode it as the instance’s new data.
@end deffn

@geindex encode() (xmlrpc.client.Binary method)
@anchor{library/xmlrpc client xmlrpc client Binary encode}@anchor{3356}
@deffn {Method} encode (out)

Write the XML-RPC base 64 encoding of this binary item to the @emph{out} stream object.

The encoded data will have newlines every 76 characters as per
@geindex RFC; RFC 2045#section-6.8
RFC 2045 section 6.8@footnote{https://tools.ietf.org/html/rfc2045.html#section-6.8},
which was the de facto standard base64 specification when the
XML-RPC spec was written.
@end deffn

It also supports certain of Python’s built-in operators through @code{__eq__()}
and @code{__ne__()} methods.
@end deffn

Example usage of the binary objects.  We’re going to transfer an image over
XMLRPC:

@example
from xmlrpc.server import SimpleXMLRPCServer
import xmlrpc.client

def python_logo():
    with open("python_logo.jpg", "rb") as handle:
        return xmlrpc.client.Binary(handle.read())

server = SimpleXMLRPCServer(("localhost", 8000))
print("Listening on port 8000...")
server.register_function(python_logo, 'python_logo')

server.serve_forever()
@end example

The client gets the image and saves it to a file:

@example
import xmlrpc.client

proxy = xmlrpc.client.ServerProxy("http://localhost:8000/")
with open("fetched_python_logo.jpg", "wb") as handle:
    handle.write(proxy.python_logo().data)
@end example

@node Fault Objects,ProtocolError Objects,Binary Objects,xmlrpc client — XML-RPC client access
@anchor{library/xmlrpc client fault-objects}@anchor{3357}@anchor{library/xmlrpc client id4}@anchor{3358}
@subsubsection Fault Objects


@geindex Fault (class in xmlrpc.client)
@anchor{library/xmlrpc client xmlrpc client Fault}@anchor{3347}
@deffn {Class} xmlrpc.client.Fault

A @ref{3347,,Fault} object encapsulates the content of an XML-RPC fault tag. Fault
objects have the following attributes:

@geindex faultCode (xmlrpc.client.Fault attribute)
@anchor{library/xmlrpc client xmlrpc client Fault faultCode}@anchor{3359}
@deffn {Attribute} faultCode

An int indicating the fault type.
@end deffn

@geindex faultString (xmlrpc.client.Fault attribute)
@anchor{library/xmlrpc client xmlrpc client Fault faultString}@anchor{335a}
@deffn {Attribute} faultString

A string containing a diagnostic message associated with the fault.
@end deffn
@end deffn

In the following example we’re going to intentionally cause a @ref{3347,,Fault} by
returning a complex type object.  The server code:

@example
from xmlrpc.server import SimpleXMLRPCServer

# A marshalling error is going to occur because we're returning a
# complex number
def add(x, y):
    return x+y+0j

server = SimpleXMLRPCServer(("localhost", 8000))
print("Listening on port 8000...")
server.register_function(add, 'add')

server.serve_forever()
@end example

The client code for the preceding server:

@example
import xmlrpc.client

proxy = xmlrpc.client.ServerProxy("http://localhost:8000/")
try:
    proxy.add(2, 5)
except xmlrpc.client.Fault as err:
    print("A fault occurred")
    print("Fault code: %d" % err.faultCode)
    print("Fault string: %s" % err.faultString)
@end example

@node ProtocolError Objects,MultiCall Objects,Fault Objects,xmlrpc client — XML-RPC client access
@anchor{library/xmlrpc client protocol-error-objects}@anchor{335b}@anchor{library/xmlrpc client protocolerror-objects}@anchor{335c}
@subsubsection ProtocolError Objects


@geindex ProtocolError (class in xmlrpc.client)
@anchor{library/xmlrpc client xmlrpc client ProtocolError}@anchor{3348}
@deffn {Class} xmlrpc.client.ProtocolError

A @ref{3348,,ProtocolError} object describes a protocol error in the underlying
transport layer (such as a 404 ‘not found’ error if the server named by the URI
does not exist).  It has the following attributes:

@geindex url (xmlrpc.client.ProtocolError attribute)
@anchor{library/xmlrpc client xmlrpc client ProtocolError url}@anchor{335d}
@deffn {Attribute} url

The URI or URL that triggered the error.
@end deffn

@geindex errcode (xmlrpc.client.ProtocolError attribute)
@anchor{library/xmlrpc client xmlrpc client ProtocolError errcode}@anchor{335e}
@deffn {Attribute} errcode

The error code.
@end deffn

@geindex errmsg (xmlrpc.client.ProtocolError attribute)
@anchor{library/xmlrpc client xmlrpc client ProtocolError errmsg}@anchor{335f}
@deffn {Attribute} errmsg

The error message or diagnostic string.
@end deffn

@geindex headers (xmlrpc.client.ProtocolError attribute)
@anchor{library/xmlrpc client xmlrpc client ProtocolError headers}@anchor{3360}
@deffn {Attribute} headers

A dict containing the headers of the HTTP/HTTPS request that triggered the
error.
@end deffn
@end deffn

In the following example we’re going to intentionally cause a @ref{3348,,ProtocolError}
by providing an invalid URI:

@example
import xmlrpc.client

# create a ServerProxy with a URI that doesn't respond to XMLRPC requests
proxy = xmlrpc.client.ServerProxy("http://google.com/")

try:
    proxy.some_method()
except xmlrpc.client.ProtocolError as err:
    print("A protocol error occurred")
    print("URL: %s" % err.url)
    print("HTTP/HTTPS headers: %s" % err.headers)
    print("Error code: %d" % err.errcode)
    print("Error message: %s" % err.errmsg)
@end example

@node MultiCall Objects,Convenience Functions,ProtocolError Objects,xmlrpc client — XML-RPC client access
@anchor{library/xmlrpc client multicall-objects}@anchor{3361}
@subsubsection MultiCall Objects


The @ref{3362,,MultiCall} object provides a way to encapsulate multiple calls to a
remote server into a single request @footnote{@w{(1)} 
This approach has been first presented in @uref{https://web.archive.org/web/20060624230303/http://www.xmlrpc.com/discuss/msgReader$1208?mode=topic,a discussion on xmlrpc.com}.
}.

@geindex MultiCall (class in xmlrpc.client)
@anchor{library/xmlrpc client xmlrpc client MultiCall}@anchor{3362}
@deffn {Class} xmlrpc.client.MultiCall (server)

Create an object used to boxcar method calls. @emph{server} is the eventual target of
the call. Calls can be made to the result object, but they will immediately
return @code{None}, and only store the call name and parameters in the
@ref{3362,,MultiCall} object. Calling the object itself causes all stored calls to
be transmitted as a single @code{system.multicall} request. The result of this call
is a @ref{bf7,,generator}; iterating over this generator yields the individual
results.
@end deffn

A usage example of this class follows.  The server code:

@example
from xmlrpc.server import SimpleXMLRPCServer

def add(x, y):
    return x + y

def subtract(x, y):
    return x - y

def multiply(x, y):
    return x * y

def divide(x, y):
    return x // y

# A simple server with simple arithmetic functions
server = SimpleXMLRPCServer(("localhost", 8000))
print("Listening on port 8000...")
server.register_multicall_functions()
server.register_function(add, 'add')
server.register_function(subtract, 'subtract')
server.register_function(multiply, 'multiply')
server.register_function(divide, 'divide')
server.serve_forever()
@end example

The client code for the preceding server:

@example
import xmlrpc.client

proxy = xmlrpc.client.ServerProxy("http://localhost:8000/")
multicall = xmlrpc.client.MultiCall(proxy)
multicall.add(7, 3)
multicall.subtract(7, 3)
multicall.multiply(7, 3)
multicall.divide(7, 3)
result = multicall()

print("7+3=%d, 7-3=%d, 7*3=%d, 7//3=%d" % tuple(result))
@end example

@node Convenience Functions,Example of Client Usage,MultiCall Objects,xmlrpc client — XML-RPC client access
@anchor{library/xmlrpc client convenience-functions}@anchor{3363}
@subsubsection Convenience Functions


@geindex dumps() (in module xmlrpc.client)
@anchor{library/xmlrpc client xmlrpc client dumps}@anchor{3364}
@deffn {Function} xmlrpc.client.dumps (params, methodname=None, methodresponse=None, encoding=None, allow_none=False)

Convert @emph{params} into an XML-RPC request. or into a response if @emph{methodresponse}
is true. @emph{params} can be either a tuple of arguments or an instance of the
@ref{3347,,Fault} exception class.  If @emph{methodresponse} is true, only a single value
can be returned, meaning that @emph{params} must be of length 1. @emph{encoding}, if
supplied, is the encoding to use in the generated XML; the default is UTF-8.
Python’s @ref{182,,None} value cannot be used in standard XML-RPC; to allow using
it via an extension,  provide a true value for @emph{allow_none}.
@end deffn

@geindex loads() (in module xmlrpc.client)
@anchor{library/xmlrpc client xmlrpc client loads}@anchor{3365}
@deffn {Function} xmlrpc.client.loads (data, use_datetime=False, use_builtin_types=False)

Convert an XML-RPC request or response into Python objects, a @code{(params,
methodname)}.  @emph{params} is a tuple of argument; @emph{methodname} is a string, or
@code{None} if no method name is present in the packet. If the XML-RPC packet
represents a fault condition, this function will raise a @ref{3347,,Fault} exception.
The @emph{use_builtin_types} flag can be used to cause date/time values to be
presented as @ref{363,,datetime.datetime} objects and binary data to be
presented as @ref{2f0,,bytes} objects; this flag is false by default.

The obsolete @emph{use_datetime} flag is similar to @emph{use_builtin_types} but it
applies only to date/time values.


Changed in version 3.3: The @emph{use_builtin_types} flag was added.

@end deffn

@node Example of Client Usage,Example of Client and Server Usage,Convenience Functions,xmlrpc client — XML-RPC client access
@anchor{library/xmlrpc client example-of-client-usage}@anchor{3366}@anchor{library/xmlrpc client xmlrpc-client-example}@anchor{3367}
@subsubsection Example of Client Usage


@example
# simple test program (from the XML-RPC specification)
from xmlrpc.client import ServerProxy, Error

# server = ServerProxy("http://localhost:8000") # local server
with ServerProxy("http://betty.userland.com") as proxy:

    print(proxy)

    try:
        print(proxy.examples.getStateName(41))
    except Error as v:
        print("ERROR", v)
@end example

To access an XML-RPC server through a HTTP proxy, you need to define a custom
transport.  The following example shows how:

@example
import http.client
import xmlrpc.client

class ProxiedTransport(xmlrpc.client.Transport):

    def set_proxy(self, host, port=None, headers=None):
        self.proxy = host, port
        self.proxy_headers = headers

    def make_connection(self, host):
        connection = http.client.HTTPConnection(*self.proxy)
        connection.set_tunnel(host, headers=self.proxy_headers)
        self._connection = host, connection
        return connection

transport = ProxiedTransport()
transport.set_proxy('proxy-server', 8080)
server = xmlrpc.client.ServerProxy('http://betty.userland.com', transport=transport)
print(server.examples.getStateName(41))
@end example

@node Example of Client and Server Usage,,Example of Client Usage,xmlrpc client — XML-RPC client access
@anchor{library/xmlrpc client example-of-client-and-server-usage}@anchor{3368}
@subsubsection Example of Client and Server Usage


See @ref{3369,,SimpleXMLRPCServer Example}.

@c the link now points to webarchive since the one at

@c http://www.xmlrpc.com/discuss/msgReader%241208 is broken (and webadmin

@c doesn't reply)

@node xmlrpc server — Basic XML-RPC servers,ipaddress — IPv4/IPv6 manipulation library,xmlrpc client — XML-RPC client access,Internet Protocols and Support
@anchor{library/xmlrpc server doc}@anchor{336a}@anchor{library/xmlrpc server module-xmlrpc server}@anchor{147}@anchor{library/xmlrpc server xmlrpc-server-basic-xml-rpc-servers}@anchor{336b}
@subsection @code{xmlrpc.server} — Basic XML-RPC servers


@geindex module; xmlrpc.server

@strong{Source code:} Lib/xmlrpc/server.py@footnote{https://github.com/python/cpython/tree/main/Lib/xmlrpc/server.py}


__________________________________________________________________


The @ref{147,,xmlrpc.server} module provides a basic server framework for XML-RPC
servers written in Python.  Servers can either be free standing, using
@ref{336c,,SimpleXMLRPCServer}, or embedded in a CGI environment, using
@ref{336d,,CGIXMLRPCRequestHandler}.

@cartouche
@quotation Warning 
The @ref{147,,xmlrpc.server} module is not secure against maliciously
constructed data.  If you need to parse untrusted or unauthenticated data see
@ref{2e71,,XML vulnerabilities}.
@end quotation
@end cartouche

@geindex SimpleXMLRPCServer (class in xmlrpc.server)
@anchor{library/xmlrpc server xmlrpc server SimpleXMLRPCServer}@anchor{336c}
@deffn {Class} xmlrpc.server.SimpleXMLRPCServer (addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True, use_builtin_types=False)

Create a new server instance.  This class provides methods for registration of
functions that can be called by the XML-RPC protocol.  The @emph{requestHandler}
parameter should be a factory for request handler instances; it defaults to
@ref{336e,,SimpleXMLRPCRequestHandler}.  The @emph{addr} and @emph{requestHandler} parameters
are passed to the @ref{3298,,socketserver.TCPServer} constructor.  If @emph{logRequests}
is true (the default), requests will be logged; setting this parameter to false
will turn off logging.   The @emph{allow_none} and @emph{encoding} parameters are passed
on to @ref{146,,xmlrpc.client} and control the XML-RPC responses that will be returned
from the server. The @emph{bind_and_activate} parameter controls whether
@code{server_bind()} and @code{server_activate()} are called immediately by the
constructor; it defaults to true. Setting it to false allows code to manipulate
the @emph{allow_reuse_address} class variable before the address is bound.
The @emph{use_builtin_types} parameter is passed to the
@ref{3365,,loads()} function and controls which types are processed
when date/times values or binary data are received; it defaults to false.


Changed in version 3.3: The @emph{use_builtin_types} flag was added.

@end deffn

@geindex CGIXMLRPCRequestHandler (class in xmlrpc.server)
@anchor{library/xmlrpc server xmlrpc server CGIXMLRPCRequestHandler}@anchor{336d}
@deffn {Class} xmlrpc.server.CGIXMLRPCRequestHandler (allow_none=False, encoding=None, use_builtin_types=False)

Create a new instance to handle XML-RPC requests in a CGI environment.  The
@emph{allow_none} and @emph{encoding} parameters are passed on to @ref{146,,xmlrpc.client}
and control the XML-RPC responses that will be returned from the server.
The @emph{use_builtin_types} parameter is passed to the
@ref{3365,,loads()} function and controls which types are processed
when date/times values or binary data are received; it defaults to false.


Changed in version 3.3: The @emph{use_builtin_types} flag was added.

@end deffn

@geindex SimpleXMLRPCRequestHandler (class in xmlrpc.server)
@anchor{library/xmlrpc server xmlrpc server SimpleXMLRPCRequestHandler}@anchor{336e}
@deffn {Class} xmlrpc.server.SimpleXMLRPCRequestHandler

Create a new request handler instance.  This request handler supports @code{POST}
requests and modifies logging so that the @emph{logRequests} parameter to the
@ref{336c,,SimpleXMLRPCServer} constructor parameter is honored.
@end deffn

@menu
* SimpleXMLRPCServer Objects:: 
* CGIXMLRPCRequestHandler:: 
* Documenting XMLRPC server:: 
* DocXMLRPCServer Objects:: 
* DocCGIXMLRPCRequestHandler:: 

@end menu

@node SimpleXMLRPCServer Objects,CGIXMLRPCRequestHandler,,xmlrpc server — Basic XML-RPC servers
@anchor{library/xmlrpc server simple-xmlrpc-servers}@anchor{336f}@anchor{library/xmlrpc server simplexmlrpcserver-objects}@anchor{3370}
@subsubsection SimpleXMLRPCServer Objects


The @ref{336c,,SimpleXMLRPCServer} class is based on
@ref{3298,,socketserver.TCPServer} and provides a means of creating simple, stand
alone XML-RPC servers.

@geindex register_function() (xmlrpc.server.SimpleXMLRPCServer method)
@anchor{library/xmlrpc server xmlrpc server SimpleXMLRPCServer register_function}@anchor{3371}
@deffn {Method} SimpleXMLRPCServer.register_function (function=None, name=None)

Register a function that can respond to XML-RPC requests.  If @emph{name} is given,
it will be the method name associated with @emph{function}, otherwise
@code{function.__name__} will be used.  @emph{name} is a string, and may contain
characters not legal in Python identifiers, including the period character.

This method can also be used as a decorator.  When used as a decorator,
@emph{name} can only be given as a keyword argument to register @emph{function} under
@emph{name}.  If no @emph{name} is given, @code{function.__name__} will be used.


Changed in version 3.7: @ref{3371,,register_function()} can be used as a decorator.

@end deffn

@geindex register_instance() (xmlrpc.server.SimpleXMLRPCServer method)
@anchor{library/xmlrpc server xmlrpc server SimpleXMLRPCServer register_instance}@anchor{3372}
@deffn {Method} SimpleXMLRPCServer.register_instance (instance, allow_dotted_names=False)

Register an object which is used to expose method names which have not been
registered using @ref{3371,,register_function()}.  If @emph{instance} contains a
@code{_dispatch()} method, it is called with the requested method name and the
parameters from the request.  Its API is @code{def _dispatch(self, method, params)}
(note that @emph{params} does not represent a variable argument list).  If it calls
an underlying function to perform its task, that function is called as
@code{func(*params)}, expanding the parameter list. The return value from
@code{_dispatch()} is returned to the client as the result.  If @emph{instance} does
not have a @code{_dispatch()} method, it is searched for an attribute matching
the name of the requested method.

If the optional @emph{allow_dotted_names} argument is true and the instance does not
have a @code{_dispatch()} method, then if the requested method name contains
periods, each component of the method name is searched for individually, with
the effect that a simple hierarchical search is performed.  The value found from
this search is then called with the parameters from the request, and the return
value is passed back to the client.

@cartouche
@quotation Warning 
Enabling the @emph{allow_dotted_names} option allows intruders to access your
module’s global variables and may allow intruders to execute arbitrary code on
your machine.  Only use this option on a secure, closed network.
@end quotation
@end cartouche
@end deffn

@geindex register_introspection_functions() (xmlrpc.server.SimpleXMLRPCServer method)
@anchor{library/xmlrpc server xmlrpc server SimpleXMLRPCServer register_introspection_functions}@anchor{3373}
@deffn {Method} SimpleXMLRPCServer.register_introspection_functions ()

Registers the XML-RPC introspection functions @code{system.listMethods},
@code{system.methodHelp} and @code{system.methodSignature}.
@end deffn

@geindex register_multicall_functions() (xmlrpc.server.SimpleXMLRPCServer method)
@anchor{library/xmlrpc server xmlrpc server SimpleXMLRPCServer register_multicall_functions}@anchor{3374}
@deffn {Method} SimpleXMLRPCServer.register_multicall_functions ()

Registers the XML-RPC multicall function system.multicall.
@end deffn

@geindex rpc_paths (xmlrpc.server.SimpleXMLRPCRequestHandler attribute)
@anchor{library/xmlrpc server xmlrpc server SimpleXMLRPCRequestHandler rpc_paths}@anchor{3375}
@deffn {Attribute} SimpleXMLRPCRequestHandler.rpc_paths

An attribute value that must be a tuple listing valid path portions of the URL
for receiving XML-RPC requests.  Requests posted to other paths will result in a
404 “no such page” HTTP error.  If this tuple is empty, all paths will be
considered valid. The default value is @code{('/', '/RPC2')}.
@end deffn

@menu
* SimpleXMLRPCServer Example:: 

@end menu

@node SimpleXMLRPCServer Example,,,SimpleXMLRPCServer Objects
@anchor{library/xmlrpc server id1}@anchor{3376}@anchor{library/xmlrpc server simplexmlrpcserver-example}@anchor{3369}
@subsubsection SimpleXMLRPCServer Example


Server code:

@example
from xmlrpc.server import SimpleXMLRPCServer
from xmlrpc.server import SimpleXMLRPCRequestHandler

# Restrict to a particular path.
class RequestHandler(SimpleXMLRPCRequestHandler):
    rpc_paths = ('/RPC2',)

# Create server
with SimpleXMLRPCServer(('localhost', 8000),
                        requestHandler=RequestHandler) as server:
    server.register_introspection_functions()

    # Register pow() function; this will use the value of
    # pow.__name__ as the name, which is just 'pow'.
    server.register_function(pow)

    # Register a function under a different name
    def adder_function(x, y):
        return x + y
    server.register_function(adder_function, 'add')

    # Register an instance; all the methods of the instance are
    # published as XML-RPC methods (in this case, just 'mul').
    class MyFuncs:
        def mul(self, x, y):
            return x * y

    server.register_instance(MyFuncs())

    # Run the server's main loop
    server.serve_forever()
@end example

The following client code will call the methods made available by the preceding
server:

@example
import xmlrpc.client

s = xmlrpc.client.ServerProxy('http://localhost:8000')
print(s.pow(2,3))  # Returns 2**3 = 8
print(s.add(2,3))  # Returns 5
print(s.mul(5,2))  # Returns 5*2 = 10

# Print list of available methods
print(s.system.listMethods())
@end example

@code{register_function()} can also be used as a decorator. The previous server
example can register functions in a decorator way:

@example
from xmlrpc.server import SimpleXMLRPCServer
from xmlrpc.server import SimpleXMLRPCRequestHandler

class RequestHandler(SimpleXMLRPCRequestHandler):
    rpc_paths = ('/RPC2',)

with SimpleXMLRPCServer(('localhost', 8000),
                        requestHandler=RequestHandler) as server:
    server.register_introspection_functions()

    # Register pow() function; this will use the value of
    # pow.__name__ as the name, which is just 'pow'.
    server.register_function(pow)

    # Register a function under a different name, using
    # register_function as a decorator. *name* can only be given
    # as a keyword argument.
    @@server.register_function(name='add')
    def adder_function(x, y):
        return x + y

    # Register a function under function.__name__.
    @@server.register_function
    def mul(x, y):
        return x * y

    server.serve_forever()
@end example

The following example included in the @code{Lib/xmlrpc/server.py} module shows
a server allowing dotted names and registering a multicall function.

@cartouche
@quotation Warning 
Enabling the @emph{allow_dotted_names} option allows intruders to access your
module’s global variables and may allow intruders to execute arbitrary code on
your machine.  Only use this example only within a secure, closed network.
@end quotation
@end cartouche

@example
import datetime

class ExampleService:
    def getData(self):
        return '42'

    class currentTime:
        @@staticmethod
        def getCurrentTime():
            return datetime.datetime.now()

with SimpleXMLRPCServer(("localhost", 8000)) as server:
    server.register_function(pow)
    server.register_function(lambda x,y: x+y, 'add')
    server.register_instance(ExampleService(), allow_dotted_names=True)
    server.register_multicall_functions()
    print('Serving XML-RPC on localhost port 8000')
    try:
        server.serve_forever()
    except KeyboardInterrupt:
        print("\nKeyboard interrupt received, exiting.")
        sys.exit(0)
@end example

This ExampleService demo can be invoked from the command line:

@example
python -m xmlrpc.server
@end example

The client that interacts with the above server is included in
@cite{Lib/xmlrpc/client.py}:

@example
server = ServerProxy("http://localhost:8000")

try:
    print(server.currentTime.getCurrentTime())
except Error as v:
    print("ERROR", v)

multi = MultiCall(server)
multi.getData()
multi.pow(2,9)
multi.add(1,2)
try:
    for response in multi():
        print(response)
except Error as v:
    print("ERROR", v)
@end example

This client which interacts with the demo XMLRPC server can be invoked as:

@example
python -m xmlrpc.client
@end example

@node CGIXMLRPCRequestHandler,Documenting XMLRPC server,SimpleXMLRPCServer Objects,xmlrpc server — Basic XML-RPC servers
@anchor{library/xmlrpc server cgixmlrpcrequesthandler}@anchor{3377}
@subsubsection CGIXMLRPCRequestHandler


The @ref{336d,,CGIXMLRPCRequestHandler} class can be used to handle XML-RPC
requests sent to Python CGI scripts.

@geindex register_function() (xmlrpc.server.CGIXMLRPCRequestHandler method)
@anchor{library/xmlrpc server xmlrpc server CGIXMLRPCRequestHandler register_function}@anchor{3378}
@deffn {Method} CGIXMLRPCRequestHandler.register_function (function=None, name=None)

Register a function that can respond to XML-RPC requests.  If @emph{name} is given,
it will be the method name associated with @emph{function}, otherwise
@code{function.__name__} will be used.  @emph{name} is a string, and may contain
characters not legal in Python identifiers, including the period character.

This method can also be used as a decorator.  When used as a decorator,
@emph{name} can only be given as a keyword argument to register @emph{function} under
@emph{name}.  If no @emph{name} is given, @code{function.__name__} will be used.


Changed in version 3.7: @ref{3378,,register_function()} can be used as a decorator.

@end deffn

@geindex register_instance() (xmlrpc.server.CGIXMLRPCRequestHandler method)
@anchor{library/xmlrpc server xmlrpc server CGIXMLRPCRequestHandler register_instance}@anchor{3379}
@deffn {Method} CGIXMLRPCRequestHandler.register_instance (instance)

Register an object which is used to expose method names  which have not been
registered using @ref{3378,,register_function()}. If  instance contains a
@code{_dispatch()} method, it is called with the  requested method name and the
parameters from the  request; the return value is returned to the client as the
result. If instance does not have a @code{_dispatch()} method, it is searched
for an attribute matching the name of the requested method; if  the requested
method name contains periods, each  component of the method name is searched for
individually,  with the effect that a simple hierarchical search is performed.
The value found from this search is then called with the  parameters from the
request, and the return value is passed  back to the client.
@end deffn

@geindex register_introspection_functions() (xmlrpc.server.CGIXMLRPCRequestHandler method)
@anchor{library/xmlrpc server xmlrpc server CGIXMLRPCRequestHandler register_introspection_functions}@anchor{337a}
@deffn {Method} CGIXMLRPCRequestHandler.register_introspection_functions ()

Register the XML-RPC introspection functions  @code{system.listMethods},
@code{system.methodHelp} and  @code{system.methodSignature}.
@end deffn

@geindex register_multicall_functions() (xmlrpc.server.CGIXMLRPCRequestHandler method)
@anchor{library/xmlrpc server xmlrpc server CGIXMLRPCRequestHandler register_multicall_functions}@anchor{337b}
@deffn {Method} CGIXMLRPCRequestHandler.register_multicall_functions ()

Register the XML-RPC multicall function @code{system.multicall}.
@end deffn

@geindex handle_request() (xmlrpc.server.CGIXMLRPCRequestHandler method)
@anchor{library/xmlrpc server xmlrpc server CGIXMLRPCRequestHandler handle_request}@anchor{337c}
@deffn {Method} CGIXMLRPCRequestHandler.handle_request (request_text=None)

Handle an XML-RPC request. If @emph{request_text} is given, it should be the POST
data provided by the HTTP server,  otherwise the contents of stdin will be used.
@end deffn

Example:

@example
class MyFuncs:
    def mul(self, x, y):
        return x * y


handler = CGIXMLRPCRequestHandler()
handler.register_function(pow)
handler.register_function(lambda x,y: x+y, 'add')
handler.register_introspection_functions()
handler.register_instance(MyFuncs())
handler.handle_request()
@end example

@node Documenting XMLRPC server,DocXMLRPCServer Objects,CGIXMLRPCRequestHandler,xmlrpc server — Basic XML-RPC servers
@anchor{library/xmlrpc server documenting-xmlrpc-server}@anchor{337d}
@subsubsection Documenting XMLRPC server


These classes extend the above classes to serve HTML documentation in response
to HTTP GET requests.  Servers can either be free standing, using
@ref{1258,,DocXMLRPCServer}, or embedded in a CGI environment, using
@ref{337e,,DocCGIXMLRPCRequestHandler}.

@geindex DocXMLRPCServer (class in xmlrpc.server)
@anchor{library/xmlrpc server xmlrpc server DocXMLRPCServer}@anchor{1258}
@deffn {Class} xmlrpc.server.DocXMLRPCServer (addr, requestHandler=DocXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True, use_builtin_types=True)

Create a new server instance. All parameters have the same meaning as for
@ref{336c,,SimpleXMLRPCServer}; @emph{requestHandler} defaults to
@ref{337f,,DocXMLRPCRequestHandler}.


Changed in version 3.3: The @emph{use_builtin_types} flag was added.

@end deffn

@geindex DocCGIXMLRPCRequestHandler (class in xmlrpc.server)
@anchor{library/xmlrpc server xmlrpc server DocCGIXMLRPCRequestHandler}@anchor{337e}
@deffn {Class} xmlrpc.server.DocCGIXMLRPCRequestHandler

Create a new instance to handle XML-RPC requests in a CGI environment.
@end deffn

@geindex DocXMLRPCRequestHandler (class in xmlrpc.server)
@anchor{library/xmlrpc server xmlrpc server DocXMLRPCRequestHandler}@anchor{337f}
@deffn {Class} xmlrpc.server.DocXMLRPCRequestHandler

Create a new request handler instance. This request handler supports XML-RPC
POST requests, documentation GET requests, and modifies logging so that the
@emph{logRequests} parameter to the @ref{1258,,DocXMLRPCServer} constructor parameter is
honored.
@end deffn

@node DocXMLRPCServer Objects,DocCGIXMLRPCRequestHandler,Documenting XMLRPC server,xmlrpc server — Basic XML-RPC servers
@anchor{library/xmlrpc server doc-xmlrpc-servers}@anchor{3380}@anchor{library/xmlrpc server docxmlrpcserver-objects}@anchor{3381}
@subsubsection DocXMLRPCServer Objects


The @ref{1258,,DocXMLRPCServer} class is derived from @ref{336c,,SimpleXMLRPCServer}
and provides a means of creating self-documenting, stand alone XML-RPC
servers. HTTP POST requests are handled as XML-RPC method calls. HTTP GET
requests are handled by generating pydoc-style HTML documentation. This allows a
server to provide its own web-based documentation.

@geindex set_server_title() (xmlrpc.server.DocXMLRPCServer method)
@anchor{library/xmlrpc server xmlrpc server DocXMLRPCServer set_server_title}@anchor{3382}
@deffn {Method} DocXMLRPCServer.set_server_title (server_title)

Set the title used in the generated HTML documentation. This title will be used
inside the HTML “title” element.
@end deffn

@geindex set_server_name() (xmlrpc.server.DocXMLRPCServer method)
@anchor{library/xmlrpc server xmlrpc server DocXMLRPCServer set_server_name}@anchor{3383}
@deffn {Method} DocXMLRPCServer.set_server_name (server_name)

Set the name used in the generated HTML documentation. This name will appear at
the top of the generated documentation inside a “h1” element.
@end deffn

@geindex set_server_documentation() (xmlrpc.server.DocXMLRPCServer method)
@anchor{library/xmlrpc server xmlrpc server DocXMLRPCServer set_server_documentation}@anchor{3384}
@deffn {Method} DocXMLRPCServer.set_server_documentation (server_documentation)

Set the description used in the generated HTML documentation. This description
will appear as a paragraph, below the server name, in the documentation.
@end deffn

@node DocCGIXMLRPCRequestHandler,,DocXMLRPCServer Objects,xmlrpc server — Basic XML-RPC servers
@anchor{library/xmlrpc server doccgixmlrpcrequesthandler}@anchor{3385}
@subsubsection DocCGIXMLRPCRequestHandler


The @ref{337e,,DocCGIXMLRPCRequestHandler} class is derived from
@ref{336d,,CGIXMLRPCRequestHandler} and provides a means of creating
self-documenting, XML-RPC CGI scripts. HTTP POST requests are handled as XML-RPC
method calls. HTTP GET requests are handled by generating pydoc-style HTML
documentation. This allows a server to provide its own web-based documentation.

@geindex set_server_title() (xmlrpc.server.DocCGIXMLRPCRequestHandler method)
@anchor{library/xmlrpc server xmlrpc server DocCGIXMLRPCRequestHandler set_server_title}@anchor{3386}
@deffn {Method} DocCGIXMLRPCRequestHandler.set_server_title (server_title)

Set the title used in the generated HTML documentation. This title will be used
inside the HTML “title” element.
@end deffn

@geindex set_server_name() (xmlrpc.server.DocCGIXMLRPCRequestHandler method)
@anchor{library/xmlrpc server xmlrpc server DocCGIXMLRPCRequestHandler set_server_name}@anchor{3387}
@deffn {Method} DocCGIXMLRPCRequestHandler.set_server_name (server_name)

Set the name used in the generated HTML documentation. This name will appear at
the top of the generated documentation inside a “h1” element.
@end deffn

@geindex set_server_documentation() (xmlrpc.server.DocCGIXMLRPCRequestHandler method)
@anchor{library/xmlrpc server xmlrpc server DocCGIXMLRPCRequestHandler set_server_documentation}@anchor{3388}
@deffn {Method} DocCGIXMLRPCRequestHandler.set_server_documentation (server_documentation)

Set the description used in the generated HTML documentation. This description
will appear as a paragraph, below the server name, in the documentation.
@end deffn

@node ipaddress — IPv4/IPv6 manipulation library,,xmlrpc server — Basic XML-RPC servers,Internet Protocols and Support
@anchor{library/ipaddress doc}@anchor{3389}@anchor{library/ipaddress ipaddress-ipv4-ipv6-manipulation-library}@anchor{338a}@anchor{library/ipaddress module-ipaddress}@anchor{9e}
@subsection @code{ipaddress} — IPv4/IPv6 manipulation library


@geindex module; ipaddress

@strong{Source code:} Lib/ipaddress.py@footnote{https://github.com/python/cpython/tree/main/Lib/ipaddress.py}


__________________________________________________________________


@ref{9e,,ipaddress} provides the capabilities to create, manipulate and
operate on IPv4 and IPv6 addresses and networks.

The functions and classes in this module make it straightforward to handle
various tasks related to IP addresses, including checking whether or not two
hosts are on the same subnet, iterating over all hosts in a particular
subnet, checking whether or not a string represents a valid IP address or
network definition, and so on.

This is the full module API reference—for an overview and introduction, see
@ref{338b,,An introduction to the ipaddress module}.


New in version 3.3.


@c import ipaddress
@c from ipaddress import (
@c     ip_network, IPv4Address, IPv4Interface, IPv4Network,
@c )

@menu
* Convenience factory functions:: 
* IP Addresses:: 
* IP Network definitions:: 
* Interface objects:: 
* Other Module Level Functions:: 
* Custom Exceptions:: 

@end menu

@node Convenience factory functions,IP Addresses,,ipaddress — IPv4/IPv6 manipulation library
@anchor{library/ipaddress convenience-factory-functions}@anchor{338c}
@subsubsection Convenience factory functions


The @ref{9e,,ipaddress} module provides factory functions to conveniently create
IP addresses, networks and interfaces:

@geindex ip_address() (in module ipaddress)
@anchor{library/ipaddress ipaddress ip_address}@anchor{338d}
@deffn {Function} ipaddress.ip_address (address)

Return an @ref{338e,,IPv4Address} or @ref{3fe,,IPv6Address} object depending on
the IP address passed as argument.  Either IPv4 or IPv6 addresses may be
supplied; integers less than @code{2**32} will be considered to be IPv4 by default.
A @ref{17c,,ValueError} is raised if @emph{address} does not represent a valid IPv4
or IPv6 address.

@example
>>> ipaddress.ip_address('192.168.0.1')
IPv4Address('192.168.0.1')
>>> ipaddress.ip_address('2001:db8::')
IPv6Address('2001:db8::')
@end example
@end deffn

@geindex ip_network() (in module ipaddress)
@anchor{library/ipaddress ipaddress ip_network}@anchor{338f}
@deffn {Function} ipaddress.ip_network (address, strict=True)

Return an @ref{670,,IPv4Network} or @ref{66f,,IPv6Network} object depending on
the IP address passed as argument.  @emph{address} is a string or integer
representing the IP network.  Either IPv4 or IPv6 networks may be supplied;
integers less than @code{2**32} will be considered to be IPv4 by default.  @emph{strict}
is passed to @ref{670,,IPv4Network} or @ref{66f,,IPv6Network} constructor.  A
@ref{17c,,ValueError} is raised if @emph{address} does not represent a valid IPv4 or
IPv6 address, or if the network has host bits set.

@example
>>> ipaddress.ip_network('192.168.0.0/28')
IPv4Network('192.168.0.0/28')
@end example
@end deffn

@geindex ip_interface() (in module ipaddress)
@anchor{library/ipaddress ipaddress ip_interface}@anchor{3390}
@deffn {Function} ipaddress.ip_interface (address)

Return an @ref{3391,,IPv4Interface} or @ref{3392,,IPv6Interface} object depending
on the IP address passed as argument.  @emph{address} is a string or integer
representing the IP address.  Either IPv4 or IPv6 addresses may be supplied;
integers less than @code{2**32} will be considered to be IPv4 by default.  A
@ref{17c,,ValueError} is raised if @emph{address} does not represent a valid IPv4 or
IPv6 address.
@end deffn

One downside of these convenience functions is that the need to handle both
IPv4 and IPv6 formats means that error messages provide minimal
information on the precise error, as the functions don’t know whether the
IPv4 or IPv6 format was intended. More detailed error reporting can be
obtained by calling the appropriate version specific class constructors
directly.

@node IP Addresses,IP Network definitions,Convenience factory functions,ipaddress — IPv4/IPv6 manipulation library
@anchor{library/ipaddress ip-addresses}@anchor{3393}
@subsubsection IP Addresses


@menu
* Address objects:: 
* Conversion to Strings and Integers:: 
* Operators: Operators<3>. 

@end menu

@node Address objects,Conversion to Strings and Integers,,IP Addresses
@anchor{library/ipaddress address-objects}@anchor{3394}
@subsubsection Address objects


The @ref{338e,,IPv4Address} and @ref{3fe,,IPv6Address} objects share a lot of common
attributes.  Some attributes that are only meaningful for IPv6 addresses are
also implemented by @ref{338e,,IPv4Address} objects, in order to make it easier to
write code that handles both IP versions correctly.  Address objects are
@ref{2dc,,hashable}, so they can be used as keys in dictionaries.

@geindex IPv4Address (class in ipaddress)
@anchor{library/ipaddress ipaddress IPv4Address}@anchor{338e}
@deffn {Class} ipaddress.IPv4Address (address)

Construct an IPv4 address.  An @ref{3395,,AddressValueError} is raised if
@emph{address} is not a valid IPv4 address.

The following constitutes a valid IPv4 address:


@enumerate 

@item 
A string in decimal-dot notation, consisting of four decimal integers in
the inclusive range 0–255, separated by dots (e.g. @code{192.168.0.1}). Each
integer represents an octet (byte) in the address. Leading zeroes are
not tolerated to prevent confusion with octal notation.

@item 
An integer that fits into 32 bits.

@item 
An integer packed into a @ref{2f0,,bytes} object of length 4 (most
significant octet first).
@end enumerate

@example
>>> ipaddress.IPv4Address('192.168.0.1')
IPv4Address('192.168.0.1')
>>> ipaddress.IPv4Address(3232235521)
IPv4Address('192.168.0.1')
>>> ipaddress.IPv4Address(b'\xC0\xA8\x00\x01')
IPv4Address('192.168.0.1')
@end example


Changed in version 3.8: Leading zeros are tolerated, even in ambiguous cases that look like
octal notation.



Changed in version 3.10: Leading zeros are no longer tolerated and are treated as an error.
IPv4 address strings are now parsed as strict as glibc
@ref{465,,inet_pton()}.



Changed in version 3.9.5: The above change was also included in Python 3.9 starting with
version 3.9.5.



Changed in version 3.8.12: The above change was also included in Python 3.8 starting with
version 3.8.12.


@geindex version (ipaddress.IPv4Address attribute)
@anchor{library/ipaddress ipaddress IPv4Address version}@anchor{3396}
@deffn {Attribute} version

The appropriate version number: @code{4} for IPv4, @code{6} for IPv6.
@end deffn

@geindex max_prefixlen (ipaddress.IPv4Address attribute)
@anchor{library/ipaddress ipaddress IPv4Address max_prefixlen}@anchor{3397}
@deffn {Attribute} max_prefixlen

The total number of bits in the address representation for this
version: @code{32} for IPv4, @code{128} for IPv6.

The prefix defines the number of leading bits in an  address that
are compared to determine whether or not an address is part of a
network.
@end deffn

@geindex compressed (ipaddress.IPv4Address attribute)
@anchor{library/ipaddress ipaddress IPv4Address compressed}@anchor{3398}
@deffn {Attribute} compressed
@end deffn

@geindex exploded (ipaddress.IPv4Address attribute)
@anchor{library/ipaddress ipaddress IPv4Address exploded}@anchor{3399}
@deffn {Attribute} exploded

The string representation in dotted decimal notation. Leading zeroes
are never included in the representation.

As IPv4 does not define a shorthand notation for addresses with octets
set to zero, these two attributes are always the same as @code{str(addr)}
for IPv4 addresses. Exposing these attributes makes it easier to
write display code that can handle both IPv4 and IPv6 addresses.
@end deffn

@geindex packed (ipaddress.IPv4Address attribute)
@anchor{library/ipaddress ipaddress IPv4Address packed}@anchor{339a}
@deffn {Attribute} packed

The binary representation of this address - a @ref{2f0,,bytes} object of
the appropriate length (most significant octet first). This is 4 bytes
for IPv4 and 16 bytes for IPv6.
@end deffn

@geindex reverse_pointer (ipaddress.IPv4Address attribute)
@anchor{library/ipaddress ipaddress IPv4Address reverse_pointer}@anchor{339b}
@deffn {Attribute} reverse_pointer

The name of the reverse DNS PTR record for the IP address, e.g.:

@example
>>> ipaddress.ip_address("127.0.0.1").reverse_pointer
'1.0.0.127.in-addr.arpa'
>>> ipaddress.ip_address("2001:db8::1").reverse_pointer
'1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa'
@end example

This is the name that could be used for performing a PTR lookup, not the
resolved hostname itself.


New in version 3.5.

@end deffn

@geindex is_multicast (ipaddress.IPv4Address attribute)
@anchor{library/ipaddress ipaddress IPv4Address is_multicast}@anchor{339c}
@deffn {Attribute} is_multicast

@code{True} if the address is reserved for multicast use.  See
@geindex RFC; RFC 3171
RFC 3171@footnote{https://tools.ietf.org/html/rfc3171.html} (for IPv4) or 
@geindex RFC; RFC 2373
RFC 2373@footnote{https://tools.ietf.org/html/rfc2373.html} (for IPv6).
@end deffn

@geindex is_private (ipaddress.IPv4Address attribute)
@anchor{library/ipaddress ipaddress IPv4Address is_private}@anchor{339d}
@deffn {Attribute} is_private

@code{True} if the address is allocated for private networks.  See
iana-ipv4-special-registry@footnote{https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml} (for IPv4) or iana-ipv6-special-registry@footnote{https://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml}
(for IPv6).
@end deffn

@geindex is_global (ipaddress.IPv4Address attribute)
@anchor{library/ipaddress ipaddress IPv4Address is_global}@anchor{add}
@deffn {Attribute} is_global

@code{True} if the address is allocated for public networks.  See
iana-ipv4-special-registry@footnote{https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml} (for IPv4) or iana-ipv6-special-registry@footnote{https://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml}
(for IPv6).


New in version 3.4.

@end deffn

@geindex is_unspecified (ipaddress.IPv4Address attribute)
@anchor{library/ipaddress ipaddress IPv4Address is_unspecified}@anchor{339e}
@deffn {Attribute} is_unspecified

@code{True} if the address is unspecified.  See 
@geindex RFC; RFC 5735
RFC 5735@footnote{https://tools.ietf.org/html/rfc5735.html} (for IPv4)
or 
@geindex RFC; RFC 2373
RFC 2373@footnote{https://tools.ietf.org/html/rfc2373.html} (for IPv6).
@end deffn

@geindex is_reserved (ipaddress.IPv4Address attribute)
@anchor{library/ipaddress ipaddress IPv4Address is_reserved}@anchor{339f}
@deffn {Attribute} is_reserved

@code{True} if the address is otherwise IETF reserved.
@end deffn

@geindex is_loopback (ipaddress.IPv4Address attribute)
@anchor{library/ipaddress ipaddress IPv4Address is_loopback}@anchor{33a0}
@deffn {Attribute} is_loopback

@code{True} if this is a loopback address.  See 
@geindex RFC; RFC 3330
RFC 3330@footnote{https://tools.ietf.org/html/rfc3330.html} (for IPv4)
or 
@geindex RFC; RFC 2373
RFC 2373@footnote{https://tools.ietf.org/html/rfc2373.html} (for IPv6).
@end deffn

@geindex is_link_local (ipaddress.IPv4Address attribute)
@anchor{library/ipaddress ipaddress IPv4Address is_link_local}@anchor{33a1}
@deffn {Attribute} is_link_local

@code{True} if the address is reserved for link-local usage.  See
@geindex RFC; RFC 3927
RFC 3927@footnote{https://tools.ietf.org/html/rfc3927.html}.
@end deffn
@end deffn

@geindex __format__() (ipaddress.IPv4Address method)
@anchor{library/ipaddress ipaddress IPv4Address __format__}@anchor{33a2}
@deffn {Method} IPv4Address.__format__ (fmt)

Returns a string representation of the IP address, controlled by
an explicit format string.
@emph{fmt} can be one of the following: @code{'s'}, the default option,
equivalent to @ref{2ef,,str()}, @code{'b'} for a zero-padded binary string,
@code{'X'} or @code{'x'} for an uppercase or lowercase hexadecimal
representation, or @code{'n'}, which is equivalent to @code{'b'} for IPv4
addresses and @code{'x'} for IPv6. For binary and hexadecimal
representations, the form specifier @code{'#'} and the grouping option
@code{'_'} are available. @code{__format__} is used by @code{format}, @code{str.format}
and f-strings.

@example
>>> format(ipaddress.IPv4Address('192.168.0.1'))
'192.168.0.1'
>>> '@{:#b@}'.format(ipaddress.IPv4Address('192.168.0.1'))
'0b11000000101010000000000000000001'
>>> f'@{ipaddress.IPv6Address("2001:db8::1000"):s@}'
'2001:db8::1000'
>>> format(ipaddress.IPv6Address('2001:db8::1000'), '_X')
'2001_0DB8_0000_0000_0000_0000_0000_1000'
>>> '@{:#_n@}'.format(ipaddress.IPv6Address('2001:db8::1000'))
'0x2001_0db8_0000_0000_0000_0000_0000_1000'
@end example


New in version 3.9.

@end deffn

@geindex IPv6Address (class in ipaddress)
@anchor{library/ipaddress ipaddress IPv6Address}@anchor{3fe}
@deffn {Class} ipaddress.IPv6Address (address)

Construct an IPv6 address.  An @ref{3395,,AddressValueError} is raised if
@emph{address} is not a valid IPv6 address.

The following constitutes a valid IPv6 address:


@enumerate 

@item 
A string consisting of eight groups of four hexadecimal digits, each
group representing 16 bits.  The groups are separated by colons.
This describes an @emph{exploded} (longhand) notation.  The string can
also be @emph{compressed} (shorthand notation) by various means.  See
@geindex RFC; RFC 4291
RFC 4291@footnote{https://tools.ietf.org/html/rfc4291.html} for details.  For example,
@code{"0000:0000:0000:0000:0000:0abc:0007:0def"} can be compressed to
@code{"::abc:7:def"}.

Optionally, the string may also have a scope zone ID, expressed
with a suffix @code{%scope_id}. If present, the scope ID must be non-empty,
and may not contain @code{%}.
See 
@geindex RFC; RFC 4007
RFC 4007@footnote{https://tools.ietf.org/html/rfc4007.html} for details.
For example, @code{fe80::1234%1} might identify address @code{fe80::1234} on the first link of the node.

@item 
An integer that fits into 128 bits.

@item 
An integer packed into a @ref{2f0,,bytes} object of length 16, big-endian.
@end enumerate

@example
>>> ipaddress.IPv6Address('2001:db8::1000')
IPv6Address('2001:db8::1000')
>>> ipaddress.IPv6Address('ff02::5678%1')
IPv6Address('ff02::5678%1')
@end example

@geindex compressed (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address compressed}@anchor{33a3}
@deffn {Attribute} compressed
@end deffn

The short form of the address representation, with leading zeroes in
groups omitted and the longest sequence of groups consisting entirely of
zeroes collapsed to a single empty group.

This is also the value returned by @code{str(addr)} for IPv6 addresses.

@geindex exploded (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address exploded}@anchor{33a4}
@deffn {Attribute} exploded
@end deffn

The long form of the address representation, with all leading zeroes and
groups consisting entirely of zeroes included.

For the following attributes and methods, see the corresponding
documentation of the @ref{338e,,IPv4Address} class:

@geindex packed (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address packed}@anchor{33a5}
@deffn {Attribute} packed
@end deffn

@geindex reverse_pointer (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address reverse_pointer}@anchor{33a6}
@deffn {Attribute} reverse_pointer
@end deffn

@geindex version (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address version}@anchor{33a7}
@deffn {Attribute} version
@end deffn

@geindex max_prefixlen (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address max_prefixlen}@anchor{33a8}
@deffn {Attribute} max_prefixlen
@end deffn

@geindex is_multicast (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address is_multicast}@anchor{33a9}
@deffn {Attribute} is_multicast
@end deffn

@geindex is_private (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address is_private}@anchor{1110}
@deffn {Attribute} is_private
@end deffn

@geindex is_global (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address is_global}@anchor{33aa}
@deffn {Attribute} is_global
@end deffn

@geindex is_unspecified (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address is_unspecified}@anchor{33ab}
@deffn {Attribute} is_unspecified
@end deffn

@geindex is_reserved (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address is_reserved}@anchor{33ac}
@deffn {Attribute} is_reserved
@end deffn

@geindex is_loopback (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address is_loopback}@anchor{33ad}
@deffn {Attribute} is_loopback
@end deffn

@geindex is_link_local (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address is_link_local}@anchor{33ae}
@deffn {Attribute} is_link_local


New in version 3.4: is_global

@end deffn

@geindex is_site_local (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address is_site_local}@anchor{33af}
@deffn {Attribute} is_site_local

@code{True} if the address is reserved for site-local usage.  Note that
the site-local address space has been deprecated by 
@geindex RFC; RFC 3879
RFC 3879@footnote{https://tools.ietf.org/html/rfc3879.html}. Use
@ref{339d,,is_private} to test if this address is in the
space of unique local addresses as defined by 
@geindex RFC; RFC 4193
RFC 4193@footnote{https://tools.ietf.org/html/rfc4193.html}.
@end deffn

@geindex ipv4_mapped (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address ipv4_mapped}@anchor{33b0}
@deffn {Attribute} ipv4_mapped

For addresses that appear to be IPv4 mapped addresses (starting with
@code{::FFFF/96}), this property will report the embedded IPv4 address.
For any other address, this property will be @code{None}.
@end deffn

@geindex scope_id (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address scope_id}@anchor{3ff}
@deffn {Attribute} scope_id

For scoped addresses as defined by 
@geindex RFC; RFC 4007
RFC 4007@footnote{https://tools.ietf.org/html/rfc4007.html}, this property identifies
the particular zone of the address’s scope that the address belongs to,
as a string. When no scope zone is specified, this property will be @code{None}.
@end deffn

@geindex sixtofour (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address sixtofour}@anchor{33b1}
@deffn {Attribute} sixtofour

For addresses that appear to be 6to4 addresses  (starting with
@code{2002::/16}) as defined by 
@geindex RFC; RFC 3056
RFC 3056@footnote{https://tools.ietf.org/html/rfc3056.html}, this property will report
the embedded IPv4 address.  For any other address, this property will
be @code{None}.
@end deffn

@geindex teredo (ipaddress.IPv6Address attribute)
@anchor{library/ipaddress ipaddress IPv6Address teredo}@anchor{33b2}
@deffn {Attribute} teredo

For addresses that appear to be Teredo addresses (starting with
@code{2001::/32}) as defined by 
@geindex RFC; RFC 4380
RFC 4380@footnote{https://tools.ietf.org/html/rfc4380.html}, this property will report
the embedded @code{(server, client)} IP address pair.  For any other
address, this property will be @code{None}.
@end deffn
@end deffn

@geindex __format__() (ipaddress.IPv6Address method)
@anchor{library/ipaddress ipaddress IPv6Address __format__}@anchor{33b3}
@deffn {Method} IPv6Address.__format__ (fmt)

Refer to the corresponding method documentation in
@ref{338e,,IPv4Address}.


New in version 3.9.

@end deffn

@node Conversion to Strings and Integers,Operators<3>,Address objects,IP Addresses
@anchor{library/ipaddress conversion-to-strings-and-integers}@anchor{33b4}
@subsubsection Conversion to Strings and Integers


To interoperate with networking interfaces such as the socket module,
addresses must be converted to strings or integers. This is handled using
the @ref{2ef,,str()} and @ref{1b1,,int()} builtin functions:

@example
>>> str(ipaddress.IPv4Address('192.168.0.1'))
'192.168.0.1'
>>> int(ipaddress.IPv4Address('192.168.0.1'))
3232235521
>>> str(ipaddress.IPv6Address('::1'))
'::1'
>>> int(ipaddress.IPv6Address('::1'))
1
@end example

Note that IPv6 scoped addresses are converted to integers without scope zone ID.

@node Operators<3>,,Conversion to Strings and Integers,IP Addresses
@anchor{library/ipaddress operators}@anchor{33b5}
@subsubsection Operators


Address objects support some operators.  Unless stated otherwise, operators can
only be applied between compatible objects (i.e. IPv4 with IPv4, IPv6 with
IPv6).

@menu
* Comparison operators:: 
* Arithmetic operators:: 

@end menu

@node Comparison operators,Arithmetic operators,,Operators<3>
@anchor{library/ipaddress comparison-operators}@anchor{33b6}
@subsubsection Comparison operators


Address objects can be compared with the usual set of comparison operators.
Same IPv6 addresses with different scope zone IDs are not equal.
Some examples:

@example
>>> IPv4Address('127.0.0.2') > IPv4Address('127.0.0.1')
True
>>> IPv4Address('127.0.0.2') == IPv4Address('127.0.0.1')
False
>>> IPv4Address('127.0.0.2') != IPv4Address('127.0.0.1')
True
>>> IPv6Address('fe80::1234') == IPv6Address('fe80::1234%1')
False
>>> IPv6Address('fe80::1234%1') != IPv6Address('fe80::1234%2')
True
@end example

@node Arithmetic operators,,Comparison operators,Operators<3>
@anchor{library/ipaddress arithmetic-operators}@anchor{33b7}
@subsubsection Arithmetic operators


Integers can be added to or subtracted from address objects.  Some examples:

@example
>>> IPv4Address('127.0.0.2') + 3
IPv4Address('127.0.0.5')
>>> IPv4Address('127.0.0.2') - 3
IPv4Address('126.255.255.255')
>>> IPv4Address('255.255.255.255') + 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ipaddress.AddressValueError: 4294967296 (>= 2**32) is not permitted as an IPv4 address
@end example

@node IP Network definitions,Interface objects,IP Addresses,ipaddress — IPv4/IPv6 manipulation library
@anchor{library/ipaddress ip-network-definitions}@anchor{33b8}
@subsubsection IP Network definitions


The @ref{670,,IPv4Network} and @ref{66f,,IPv6Network} objects provide a mechanism
for defining and inspecting IP network definitions.  A network definition
consists of a @emph{mask} and a @emph{network address}, and as such defines a range of
IP addresses that equal the network address when masked (binary AND) with the
mask.  For example, a network definition with the mask @code{255.255.255.0} and
the network address @code{192.168.1.0} consists of IP addresses in the inclusive
range @code{192.168.1.0} to @code{192.168.1.255}.

@menu
* Prefix@comma{} net mask and host mask: Prefix net mask and host mask. 
* Network objects:: 
* Operators: Operators<4>. 

@end menu

@node Prefix net mask and host mask,Network objects,,IP Network definitions
@anchor{library/ipaddress prefix-net-mask-and-host-mask}@anchor{33b9}
@subsubsection Prefix, net mask and host mask


There are several equivalent ways to specify IP network masks.  A @emph{prefix}
@code{/<nbits>} is a notation that denotes how many high-order bits are set in
the network mask.  A @emph{net mask} is an IP address with some number of
high-order bits set.  Thus the prefix @code{/24} is equivalent to the net mask
@code{255.255.255.0} in IPv4, or @code{ffff:ff00::} in IPv6.  In addition, a
@emph{host mask} is the logical inverse of a @emph{net mask}, and is sometimes used
(for example in Cisco access control lists) to denote a network mask.  The
host mask equivalent to @code{/24} in IPv4 is @code{0.0.0.255}.

@node Network objects,Operators<4>,Prefix net mask and host mask,IP Network definitions
@anchor{library/ipaddress network-objects}@anchor{33ba}
@subsubsection Network objects


All attributes implemented by address objects are implemented by network
objects as well.  In addition, network objects implement additional attributes.
All of these are common between @ref{670,,IPv4Network} and @ref{66f,,IPv6Network},
so to avoid duplication they are only documented for @ref{670,,IPv4Network}.
Network objects are @ref{2dc,,hashable}, so they can be used as keys in
dictionaries.

@geindex IPv4Network (class in ipaddress)
@anchor{library/ipaddress ipaddress IPv4Network}@anchor{670}
@deffn {Class} ipaddress.IPv4Network (address, strict=True)

Construct an IPv4 network definition.  @emph{address} can be one of the following:


@enumerate 

@item 
A string consisting of an IP address and an optional mask, separated by
a slash (@code{/}).  The IP address is the network address, and the mask
can be either a single number, which means it’s a @emph{prefix}, or a string
representation of an IPv4 address.  If it’s the latter, the mask is
interpreted as a @emph{net mask} if it starts with a non-zero field, or as a
@emph{host mask} if it starts with a zero field, with the single exception of
an all-zero mask which is treated as a @emph{net mask}.  If no mask is provided,
it’s considered to be @code{/32}.

For example, the following @emph{address} specifications are equivalent:
@code{192.168.1.0/24}, @code{192.168.1.0/255.255.255.0} and
@code{192.168.1.0/0.0.0.255}.

@item 
An integer that fits into 32 bits.  This is equivalent to a
single-address network, with the network address being @emph{address} and
the mask being @code{/32}.

@item 
An integer packed into a @ref{2f0,,bytes} object of length 4, big-endian.
The interpretation is similar to an integer @emph{address}.

@item 
A two-tuple of an address description and a netmask, where the address
description is either a string, a 32-bits integer, a 4-bytes packed
integer, or an existing IPv4Address object; and the netmask is either
an integer representing the prefix length (e.g. @code{24}) or a string
representing the prefix mask (e.g. @code{255.255.255.0}).
@end enumerate

An @ref{3395,,AddressValueError} is raised if @emph{address} is not a valid IPv4
address.  A @ref{33bb,,NetmaskValueError} is raised if the mask is not valid for
an IPv4 address.

If @emph{strict} is @code{True} and host bits are set in the supplied address,
then @ref{17c,,ValueError} is raised.  Otherwise, the host bits are masked out
to determine the appropriate network address.

Unless stated otherwise, all network methods accepting other network/address
objects will raise @ref{162,,TypeError} if the argument’s IP version is
incompatible to @code{self}.


Changed in version 3.5: Added the two-tuple form for the @emph{address} constructor parameter.


@geindex version (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network version}@anchor{33bc}
@deffn {Attribute} version
@end deffn

@geindex max_prefixlen (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network max_prefixlen}@anchor{33bd}
@deffn {Attribute} max_prefixlen

Refer to the corresponding attribute documentation in
@ref{338e,,IPv4Address}.
@end deffn

@geindex is_multicast (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network is_multicast}@anchor{33be}
@deffn {Attribute} is_multicast
@end deffn

@geindex is_private (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network is_private}@anchor{33bf}
@deffn {Attribute} is_private
@end deffn

@geindex is_unspecified (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network is_unspecified}@anchor{33c0}
@deffn {Attribute} is_unspecified
@end deffn

@geindex is_reserved (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network is_reserved}@anchor{33c1}
@deffn {Attribute} is_reserved
@end deffn

@geindex is_loopback (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network is_loopback}@anchor{33c2}
@deffn {Attribute} is_loopback
@end deffn

@geindex is_link_local (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network is_link_local}@anchor{33c3}
@deffn {Attribute} is_link_local

These attributes are true for the network as a whole if they are true
for both the network address and the broadcast address.
@end deffn

@geindex network_address (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network network_address}@anchor{33c4}
@deffn {Attribute} network_address

The network address for the network. The network address and the
prefix length together uniquely define a network.
@end deffn

@geindex broadcast_address (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network broadcast_address}@anchor{33c5}
@deffn {Attribute} broadcast_address

The broadcast address for the network. Packets sent to the broadcast
address should be received by every host on the network.
@end deffn

@geindex hostmask (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network hostmask}@anchor{33c6}
@deffn {Attribute} hostmask

The host mask, as an @ref{338e,,IPv4Address} object.
@end deffn

@geindex netmask (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network netmask}@anchor{33c7}
@deffn {Attribute} netmask

The net mask, as an @ref{338e,,IPv4Address} object.
@end deffn

@geindex with_prefixlen (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network with_prefixlen}@anchor{33c8}
@deffn {Attribute} with_prefixlen
@end deffn

@geindex compressed (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network compressed}@anchor{33c9}
@deffn {Attribute} compressed
@end deffn

@geindex exploded (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network exploded}@anchor{33ca}
@deffn {Attribute} exploded

A string representation of the network, with the mask in prefix
notation.

@code{with_prefixlen} and @code{compressed} are always the same as
@code{str(network)}.
@code{exploded} uses the exploded form the network address.
@end deffn

@geindex with_netmask (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network with_netmask}@anchor{33cb}
@deffn {Attribute} with_netmask

A string representation of the network, with the mask in net mask
notation.
@end deffn

@geindex with_hostmask (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network with_hostmask}@anchor{33cc}
@deffn {Attribute} with_hostmask

A string representation of the network, with the mask in host mask
notation.
@end deffn

@geindex num_addresses (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network num_addresses}@anchor{33cd}
@deffn {Attribute} num_addresses

The total number of addresses in the network.
@end deffn

@geindex prefixlen (ipaddress.IPv4Network attribute)
@anchor{library/ipaddress ipaddress IPv4Network prefixlen}@anchor{33ce}
@deffn {Attribute} prefixlen

Length of the network prefix, in bits.
@end deffn

@geindex hosts() (ipaddress.IPv4Network method)
@anchor{library/ipaddress ipaddress IPv4Network hosts}@anchor{33cf}
@deffn {Method} hosts ()

Returns an iterator over the usable hosts in the network.  The usable
hosts are all the IP addresses that belong to the network, except the
network address itself and the network broadcast address.  For networks
with a mask length of 31, the network address and network broadcast
address are also included in the result. Networks with a mask of 32
will return a list containing the single host address.

@example
>>> list(ip_network('192.0.2.0/29').hosts())  
[IPv4Address('192.0.2.1'), IPv4Address('192.0.2.2'),
 IPv4Address('192.0.2.3'), IPv4Address('192.0.2.4'),
 IPv4Address('192.0.2.5'), IPv4Address('192.0.2.6')]
>>> list(ip_network('192.0.2.0/31').hosts())
[IPv4Address('192.0.2.0'), IPv4Address('192.0.2.1')]
>>> list(ip_network('192.0.2.1/32').hosts())
[IPv4Address('192.0.2.1')]
@end example
@end deffn

@geindex overlaps() (ipaddress.IPv4Network method)
@anchor{library/ipaddress ipaddress IPv4Network overlaps}@anchor{33d0}
@deffn {Method} overlaps (other)

@code{True} if this network is partly or wholly contained in @emph{other} or
@emph{other} is wholly contained in this network.
@end deffn

@geindex address_exclude() (ipaddress.IPv4Network method)
@anchor{library/ipaddress ipaddress IPv4Network address_exclude}@anchor{33d1}
@deffn {Method} address_exclude (network)

Computes the network definitions resulting from removing the given
@emph{network} from this one.  Returns an iterator of network objects.
Raises @ref{17c,,ValueError} if @emph{network} is not completely contained in
this network.

@example
>>> n1 = ip_network('192.0.2.0/28')
>>> n2 = ip_network('192.0.2.1/32')
>>> list(n1.address_exclude(n2))  
[IPv4Network('192.0.2.8/29'), IPv4Network('192.0.2.4/30'),
 IPv4Network('192.0.2.2/31'), IPv4Network('192.0.2.0/32')]
@end example
@end deffn

@geindex subnets() (ipaddress.IPv4Network method)
@anchor{library/ipaddress ipaddress IPv4Network subnets}@anchor{a11}
@deffn {Method} subnets (prefixlen_diff=1, new_prefix=None)

The subnets that join to make the current network definition, depending
on the argument values.  @emph{prefixlen_diff} is the amount our prefix
length should be increased by.  @emph{new_prefix} is the desired new
prefix of the subnets; it must be larger than our prefix.  One and
only one of @emph{prefixlen_diff} and @emph{new_prefix} must be set.  Returns an
iterator of network objects.

@example
>>> list(ip_network('192.0.2.0/24').subnets())
[IPv4Network('192.0.2.0/25'), IPv4Network('192.0.2.128/25')]
>>> list(ip_network('192.0.2.0/24').subnets(prefixlen_diff=2))  
[IPv4Network('192.0.2.0/26'), IPv4Network('192.0.2.64/26'),
 IPv4Network('192.0.2.128/26'), IPv4Network('192.0.2.192/26')]
>>> list(ip_network('192.0.2.0/24').subnets(new_prefix=26))  
[IPv4Network('192.0.2.0/26'), IPv4Network('192.0.2.64/26'),
 IPv4Network('192.0.2.128/26'), IPv4Network('192.0.2.192/26')]
>>> list(ip_network('192.0.2.0/24').subnets(new_prefix=23))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    raise ValueError('new prefix must be longer')
ValueError: new prefix must be longer
>>> list(ip_network('192.0.2.0/24').subnets(new_prefix=25))
[IPv4Network('192.0.2.0/25'), IPv4Network('192.0.2.128/25')]
@end example
@end deffn

@geindex supernet() (ipaddress.IPv4Network method)
@anchor{library/ipaddress ipaddress IPv4Network supernet}@anchor{a12}
@deffn {Method} supernet (prefixlen_diff=1, new_prefix=None)

The supernet containing this network definition, depending on the
argument values.  @emph{prefixlen_diff} is the amount our prefix length
should be decreased by.  @emph{new_prefix} is the desired new prefix of
the supernet; it must be smaller than our prefix.  One and only one
of @emph{prefixlen_diff} and @emph{new_prefix} must be set.  Returns a single
network object.

@example
>>> ip_network('192.0.2.0/24').supernet()
IPv4Network('192.0.2.0/23')
>>> ip_network('192.0.2.0/24').supernet(prefixlen_diff=2)
IPv4Network('192.0.0.0/22')
>>> ip_network('192.0.2.0/24').supernet(new_prefix=20)
IPv4Network('192.0.0.0/20')
@end example
@end deffn

@geindex subnet_of() (ipaddress.IPv4Network method)
@anchor{library/ipaddress ipaddress IPv4Network subnet_of}@anchor{33d2}
@deffn {Method} subnet_of (other)

Return @code{True} if this network is a subnet of @emph{other}.

@example
>>> a = ip_network('192.168.1.0/24')
>>> b = ip_network('192.168.1.128/30')
>>> b.subnet_of(a)
True
@end example


New in version 3.7.

@end deffn

@geindex supernet_of() (ipaddress.IPv4Network method)
@anchor{library/ipaddress ipaddress IPv4Network supernet_of}@anchor{33d3}
@deffn {Method} supernet_of (other)

Return @code{True} if this network is a supernet of @emph{other}.

@example
>>> a = ip_network('192.168.1.0/24')
>>> b = ip_network('192.168.1.128/30')
>>> a.supernet_of(b)
True
@end example


New in version 3.7.

@end deffn

@geindex compare_networks() (ipaddress.IPv4Network method)
@anchor{library/ipaddress ipaddress IPv4Network compare_networks}@anchor{33d4}
@deffn {Method} compare_networks (other)

Compare this network to @emph{other}.  In this comparison only the network
addresses are considered; host bits aren’t.  Returns either @code{-1},
@code{0} or @code{1}.

@example
>>> ip_network('192.0.2.1/32').compare_networks(ip_network('192.0.2.2/32'))
-1
>>> ip_network('192.0.2.1/32').compare_networks(ip_network('192.0.2.0/32'))
1
>>> ip_network('192.0.2.1/32').compare_networks(ip_network('192.0.2.1/32'))
0
@end example


Deprecated since version 3.7: It uses the same ordering and comparison algorithm as “<”, “==”, and “>”

@end deffn
@end deffn

@geindex IPv6Network (class in ipaddress)
@anchor{library/ipaddress ipaddress IPv6Network}@anchor{66f}
@deffn {Class} ipaddress.IPv6Network (address, strict=True)

Construct an IPv6 network definition.  @emph{address} can be one of the following:


@enumerate 

@item 
A string consisting of an IP address and an optional prefix length,
separated by a slash (@code{/}).  The IP address is the network address,
and the prefix length must be a single number, the @emph{prefix}.  If no
prefix length is provided, it’s considered to be @code{/128}.

Note that currently expanded netmasks are not supported.  That means
@code{2001:db00::0/24} is a valid argument while @code{2001:db00::0/ffff:ff00::}
is not.

@item 
An integer that fits into 128 bits.  This is equivalent to a
single-address network, with the network address being @emph{address} and
the mask being @code{/128}.

@item 
An integer packed into a @ref{2f0,,bytes} object of length 16, big-endian.
The interpretation is similar to an integer @emph{address}.

@item 
A two-tuple of an address description and a netmask, where the address
description is either a string, a 128-bits integer, a 16-bytes packed
integer, or an existing IPv6Address object; and the netmask is an
integer representing the prefix length.
@end enumerate

An @ref{3395,,AddressValueError} is raised if @emph{address} is not a valid IPv6
address.  A @ref{33bb,,NetmaskValueError} is raised if the mask is not valid for
an IPv6 address.

If @emph{strict} is @code{True} and host bits are set in the supplied address,
then @ref{17c,,ValueError} is raised.  Otherwise, the host bits are masked out
to determine the appropriate network address.


Changed in version 3.5: Added the two-tuple form for the @emph{address} constructor parameter.


@geindex version (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network version}@anchor{33d5}
@deffn {Attribute} version
@end deffn

@geindex max_prefixlen (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network max_prefixlen}@anchor{33d6}
@deffn {Attribute} max_prefixlen
@end deffn

@geindex is_multicast (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network is_multicast}@anchor{33d7}
@deffn {Attribute} is_multicast
@end deffn

@geindex is_private (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network is_private}@anchor{33d8}
@deffn {Attribute} is_private
@end deffn

@geindex is_unspecified (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network is_unspecified}@anchor{33d9}
@deffn {Attribute} is_unspecified
@end deffn

@geindex is_reserved (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network is_reserved}@anchor{33da}
@deffn {Attribute} is_reserved
@end deffn

@geindex is_loopback (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network is_loopback}@anchor{33db}
@deffn {Attribute} is_loopback
@end deffn

@geindex is_link_local (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network is_link_local}@anchor{33dc}
@deffn {Attribute} is_link_local
@end deffn

@geindex network_address (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network network_address}@anchor{33dd}
@deffn {Attribute} network_address
@end deffn

@geindex broadcast_address (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network broadcast_address}@anchor{33de}
@deffn {Attribute} broadcast_address
@end deffn

@geindex hostmask (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network hostmask}@anchor{33df}
@deffn {Attribute} hostmask
@end deffn

@geindex netmask (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network netmask}@anchor{33e0}
@deffn {Attribute} netmask
@end deffn

@geindex with_prefixlen (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network with_prefixlen}@anchor{33e1}
@deffn {Attribute} with_prefixlen
@end deffn

@geindex compressed (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network compressed}@anchor{33e2}
@deffn {Attribute} compressed
@end deffn

@geindex exploded (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network exploded}@anchor{33e3}
@deffn {Attribute} exploded
@end deffn

@geindex with_netmask (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network with_netmask}@anchor{33e4}
@deffn {Attribute} with_netmask
@end deffn

@geindex with_hostmask (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network with_hostmask}@anchor{33e5}
@deffn {Attribute} with_hostmask
@end deffn

@geindex num_addresses (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network num_addresses}@anchor{33e6}
@deffn {Attribute} num_addresses
@end deffn

@geindex prefixlen (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network prefixlen}@anchor{33e7}
@deffn {Attribute} prefixlen
@end deffn

@geindex hosts() (ipaddress.IPv6Network method)
@anchor{library/ipaddress ipaddress IPv6Network hosts}@anchor{33e8}
@deffn {Method} hosts ()

Returns an iterator over the usable hosts in the network.  The usable
hosts are all the IP addresses that belong to the network, except the
Subnet-Router anycast address.  For networks with a mask length of 127,
the Subnet-Router anycast address is also included in the result.
Networks with a mask of 128 will return a list containing the
single host address.
@end deffn

@geindex overlaps() (ipaddress.IPv6Network method)
@anchor{library/ipaddress ipaddress IPv6Network overlaps}@anchor{33e9}
@deffn {Method} overlaps (other)
@end deffn

@geindex address_exclude() (ipaddress.IPv6Network method)
@anchor{library/ipaddress ipaddress IPv6Network address_exclude}@anchor{33ea}
@deffn {Method} address_exclude (network)
@end deffn

@geindex subnets() (ipaddress.IPv6Network method)
@anchor{library/ipaddress ipaddress IPv6Network subnets}@anchor{33eb}
@deffn {Method} subnets (prefixlen_diff=1, new_prefix=None)
@end deffn

@geindex supernet() (ipaddress.IPv6Network method)
@anchor{library/ipaddress ipaddress IPv6Network supernet}@anchor{33ec}
@deffn {Method} supernet (prefixlen_diff=1, new_prefix=None)
@end deffn

@geindex subnet_of() (ipaddress.IPv6Network method)
@anchor{library/ipaddress ipaddress IPv6Network subnet_of}@anchor{33ed}
@deffn {Method} subnet_of (other)
@end deffn

@geindex supernet_of() (ipaddress.IPv6Network method)
@anchor{library/ipaddress ipaddress IPv6Network supernet_of}@anchor{33ee}
@deffn {Method} supernet_of (other)
@end deffn

@geindex compare_networks() (ipaddress.IPv6Network method)
@anchor{library/ipaddress ipaddress IPv6Network compare_networks}@anchor{33ef}
@deffn {Method} compare_networks (other)

Refer to the corresponding attribute documentation in
@ref{670,,IPv4Network}.
@end deffn

@geindex is_site_local (ipaddress.IPv6Network attribute)
@anchor{library/ipaddress ipaddress IPv6Network is_site_local}@anchor{33f0}
@deffn {Attribute} is_site_local

These attribute is true for the network as a whole if it is true
for both the network address and the broadcast address.
@end deffn
@end deffn

@node Operators<4>,,Network objects,IP Network definitions
@anchor{library/ipaddress id1}@anchor{33f1}
@subsubsection Operators


Network objects support some operators.  Unless stated otherwise, operators can
only be applied between compatible objects (i.e. IPv4 with IPv4, IPv6 with
IPv6).

@menu
* Logical operators:: 
* Iteration:: 
* Networks as containers of addresses:: 

@end menu

@node Logical operators,Iteration,,Operators<4>
@anchor{library/ipaddress logical-operators}@anchor{33f2}
@subsubsection Logical operators


Network objects can be compared with the usual set of logical operators.
Network objects are ordered first by network address, then by net mask.

@node Iteration,Networks as containers of addresses,Logical operators,Operators<4>
@anchor{library/ipaddress iteration}@anchor{33f3}
@subsubsection Iteration


Network objects can be iterated to list all the addresses belonging to the
network.  For iteration, @emph{all} hosts are returned, including unusable hosts
(for usable hosts, use the @ref{33cf,,hosts()} method).  An
example:

@example
>>> for addr in IPv4Network('192.0.2.0/28'):
...     addr
...
IPv4Address('192.0.2.0')
IPv4Address('192.0.2.1')
IPv4Address('192.0.2.2')
IPv4Address('192.0.2.3')
IPv4Address('192.0.2.4')
IPv4Address('192.0.2.5')
IPv4Address('192.0.2.6')
IPv4Address('192.0.2.7')
IPv4Address('192.0.2.8')
IPv4Address('192.0.2.9')
IPv4Address('192.0.2.10')
IPv4Address('192.0.2.11')
IPv4Address('192.0.2.12')
IPv4Address('192.0.2.13')
IPv4Address('192.0.2.14')
IPv4Address('192.0.2.15')
@end example

@node Networks as containers of addresses,,Iteration,Operators<4>
@anchor{library/ipaddress networks-as-containers-of-addresses}@anchor{33f4}
@subsubsection Networks as containers of addresses


Network objects can act as containers of addresses.  Some examples:

@example
>>> IPv4Network('192.0.2.0/28')[0]
IPv4Address('192.0.2.0')
>>> IPv4Network('192.0.2.0/28')[15]
IPv4Address('192.0.2.15')
>>> IPv4Address('192.0.2.6') in IPv4Network('192.0.2.0/28')
True
>>> IPv4Address('192.0.3.6') in IPv4Network('192.0.2.0/28')
False
@end example

@node Interface objects,Other Module Level Functions,IP Network definitions,ipaddress — IPv4/IPv6 manipulation library
@anchor{library/ipaddress interface-objects}@anchor{33f5}
@subsubsection Interface objects


Interface objects are @ref{2dc,,hashable}, so they can be used as keys in
dictionaries.

@geindex IPv4Interface (class in ipaddress)
@anchor{library/ipaddress ipaddress IPv4Interface}@anchor{3391}
@deffn {Class} ipaddress.IPv4Interface (address)

Construct an IPv4 interface.  The meaning of @emph{address} is as in the
constructor of @ref{670,,IPv4Network}, except that arbitrary host addresses
are always accepted.

@ref{3391,,IPv4Interface} is a subclass of @ref{338e,,IPv4Address}, so it inherits
all the attributes from that class.  In addition, the following attributes
are available:

@geindex ip (ipaddress.IPv4Interface attribute)
@anchor{library/ipaddress ipaddress IPv4Interface ip}@anchor{33f6}
@deffn {Attribute} ip

The address (@ref{338e,,IPv4Address}) without network information.

@example
>>> interface = IPv4Interface('192.0.2.5/24')
>>> interface.ip
IPv4Address('192.0.2.5')
@end example
@end deffn

@geindex network (ipaddress.IPv4Interface attribute)
@anchor{library/ipaddress ipaddress IPv4Interface network}@anchor{33f7}
@deffn {Attribute} network

The network (@ref{670,,IPv4Network}) this interface belongs to.

@example
>>> interface = IPv4Interface('192.0.2.5/24')
>>> interface.network
IPv4Network('192.0.2.0/24')
@end example
@end deffn

@geindex with_prefixlen (ipaddress.IPv4Interface attribute)
@anchor{library/ipaddress ipaddress IPv4Interface with_prefixlen}@anchor{33f8}
@deffn {Attribute} with_prefixlen

A string representation of the interface with the mask in prefix notation.

@example
>>> interface = IPv4Interface('192.0.2.5/24')
>>> interface.with_prefixlen
'192.0.2.5/24'
@end example
@end deffn

@geindex with_netmask (ipaddress.IPv4Interface attribute)
@anchor{library/ipaddress ipaddress IPv4Interface with_netmask}@anchor{33f9}
@deffn {Attribute} with_netmask

A string representation of the interface with the network as a net mask.

@example
>>> interface = IPv4Interface('192.0.2.5/24')
>>> interface.with_netmask
'192.0.2.5/255.255.255.0'
@end example
@end deffn

@geindex with_hostmask (ipaddress.IPv4Interface attribute)
@anchor{library/ipaddress ipaddress IPv4Interface with_hostmask}@anchor{33fa}
@deffn {Attribute} with_hostmask

A string representation of the interface with the network as a host mask.

@example
>>> interface = IPv4Interface('192.0.2.5/24')
>>> interface.with_hostmask
'192.0.2.5/0.0.0.255'
@end example
@end deffn
@end deffn

@geindex IPv6Interface (class in ipaddress)
@anchor{library/ipaddress ipaddress IPv6Interface}@anchor{3392}
@deffn {Class} ipaddress.IPv6Interface (address)

Construct an IPv6 interface.  The meaning of @emph{address} is as in the
constructor of @ref{66f,,IPv6Network}, except that arbitrary host addresses
are always accepted.

@ref{3392,,IPv6Interface} is a subclass of @ref{3fe,,IPv6Address}, so it inherits
all the attributes from that class.  In addition, the following attributes
are available:

@geindex ip (ipaddress.IPv6Interface attribute)
@anchor{library/ipaddress ipaddress IPv6Interface ip}@anchor{33fb}
@deffn {Attribute} ip
@end deffn

@geindex network (ipaddress.IPv6Interface attribute)
@anchor{library/ipaddress ipaddress IPv6Interface network}@anchor{33fc}
@deffn {Attribute} network
@end deffn

@geindex with_prefixlen (ipaddress.IPv6Interface attribute)
@anchor{library/ipaddress ipaddress IPv6Interface with_prefixlen}@anchor{33fd}
@deffn {Attribute} with_prefixlen
@end deffn

@geindex with_netmask (ipaddress.IPv6Interface attribute)
@anchor{library/ipaddress ipaddress IPv6Interface with_netmask}@anchor{33fe}
@deffn {Attribute} with_netmask
@end deffn

@geindex with_hostmask (ipaddress.IPv6Interface attribute)
@anchor{library/ipaddress ipaddress IPv6Interface with_hostmask}@anchor{33ff}
@deffn {Attribute} with_hostmask

Refer to the corresponding attribute documentation in
@ref{3391,,IPv4Interface}.
@end deffn
@end deffn

@menu
* Operators: Operators<5>. 

@end menu

@node Operators<5>,,,Interface objects
@anchor{library/ipaddress id2}@anchor{3400}
@subsubsection Operators


Interface objects support some operators.  Unless stated otherwise, operators
can only be applied between compatible objects (i.e. IPv4 with IPv4, IPv6 with
IPv6).

@menu
* Logical operators: Logical operators<2>. 

@end menu

@node Logical operators<2>,,,Operators<5>
@anchor{library/ipaddress id3}@anchor{3401}
@subsubsection Logical operators


Interface objects can be compared with the usual set of logical operators.

For equality comparison (@code{==} and @code{!=}), both the IP address and network
must be the same for the objects to be equal.  An interface will not compare
equal to any address or network object.

For ordering (@code{<}, @code{>}, etc) the rules are different.  Interface and
address objects with the same IP version can be compared, and the address
objects will always sort before the interface objects.  Two interface objects
are first compared by their networks and, if those are the same, then by their
IP addresses.

@node Other Module Level Functions,Custom Exceptions,Interface objects,ipaddress — IPv4/IPv6 manipulation library
@anchor{library/ipaddress other-module-level-functions}@anchor{3402}
@subsubsection Other Module Level Functions


The module also provides the following module level functions:

@geindex v4_int_to_packed() (in module ipaddress)
@anchor{library/ipaddress ipaddress v4_int_to_packed}@anchor{3403}
@deffn {Function} ipaddress.v4_int_to_packed (address)

Represent an address as 4 packed bytes in network (big-endian) order.
@emph{address} is an integer representation of an IPv4 IP address.  A
@ref{17c,,ValueError} is raised if the integer is negative or too large to be an
IPv4 IP address.

@example
>>> ipaddress.ip_address(3221225985)
IPv4Address('192.0.2.1')
>>> ipaddress.v4_int_to_packed(3221225985)
b'\xc0\x00\x02\x01'
@end example
@end deffn

@geindex v6_int_to_packed() (in module ipaddress)
@anchor{library/ipaddress ipaddress v6_int_to_packed}@anchor{3404}
@deffn {Function} ipaddress.v6_int_to_packed (address)

Represent an address as 16 packed bytes in network (big-endian) order.
@emph{address} is an integer representation of an IPv6 IP address.  A
@ref{17c,,ValueError} is raised if the integer is negative or too large to be an
IPv6 IP address.
@end deffn

@geindex summarize_address_range() (in module ipaddress)
@anchor{library/ipaddress ipaddress summarize_address_range}@anchor{a13}
@deffn {Function} ipaddress.summarize_address_range (first, last)

Return an iterator of the summarized network range given the first and last
IP addresses.  @emph{first} is the first @ref{338e,,IPv4Address} or
@ref{3fe,,IPv6Address} in the range and @emph{last} is the last @ref{338e,,IPv4Address}
or @ref{3fe,,IPv6Address} in the range.  A @ref{162,,TypeError} is raised if
@emph{first} or @emph{last} are not IP addresses or are not of the same version.  A
@ref{17c,,ValueError} is raised if @emph{last} is not greater than @emph{first} or if
@emph{first} address version is not 4 or 6.

@example
>>> [ipaddr for ipaddr in ipaddress.summarize_address_range(
...    ipaddress.IPv4Address('192.0.2.0'),
...    ipaddress.IPv4Address('192.0.2.130'))]
[IPv4Network('192.0.2.0/25'), IPv4Network('192.0.2.128/31'), IPv4Network('192.0.2.130/32')]
@end example
@end deffn

@geindex collapse_addresses() (in module ipaddress)
@anchor{library/ipaddress ipaddress collapse_addresses}@anchor{a14}
@deffn {Function} ipaddress.collapse_addresses (addresses)

Return an iterator of the collapsed @ref{670,,IPv4Network} or
@ref{66f,,IPv6Network} objects.  @emph{addresses} is an iterator of
@ref{670,,IPv4Network} or @ref{66f,,IPv6Network} objects.  A @ref{162,,TypeError} is
raised if @emph{addresses} contains mixed version objects.

@example
>>> [ipaddr for ipaddr in
... ipaddress.collapse_addresses([ipaddress.IPv4Network('192.0.2.0/25'),
... ipaddress.IPv4Network('192.0.2.128/25')])]
[IPv4Network('192.0.2.0/24')]
@end example
@end deffn

@geindex get_mixed_type_key() (in module ipaddress)
@anchor{library/ipaddress ipaddress get_mixed_type_key}@anchor{3405}
@deffn {Function} ipaddress.get_mixed_type_key (obj)

Return a key suitable for sorting between networks and addresses.  Address
and Network objects are not sortable by default; they’re fundamentally
different, so the expression:

@example
IPv4Address('192.0.2.0') <= IPv4Network('192.0.2.0/24')
@end example

doesn’t make sense.  There are some times however, where you may wish to
have @ref{9e,,ipaddress} sort these anyway.  If you need to do this, you can use
this function as the @emph{key} argument to @ref{704,,sorted()}.

@emph{obj} is either a network or address object.
@end deffn

@node Custom Exceptions,,Other Module Level Functions,ipaddress — IPv4/IPv6 manipulation library
@anchor{library/ipaddress custom-exceptions}@anchor{3406}
@subsubsection Custom Exceptions


To support more specific error reporting from class constructors, the
module defines the following exceptions:

@geindex AddressValueError
@anchor{library/ipaddress ipaddress AddressValueError}@anchor{3395}
@deffn {Exception} ipaddress.AddressValueError (ValueError)

Any value error related to the address.
@end deffn

@geindex NetmaskValueError
@anchor{library/ipaddress ipaddress NetmaskValueError}@anchor{33bb}
@deffn {Exception} ipaddress.NetmaskValueError (ValueError)

Any value error related to the net mask.
@end deffn

@node Multimedia Services,Internationalization,Internet Protocols and Support,The Python Standard Library
@anchor{library/mm doc}@anchor{3407}@anchor{library/mm mmedia}@anchor{3408}@anchor{library/mm multimedia-services}@anchor{3409}
@section Multimedia Services


The modules described in this chapter implement various algorithms or interfaces
that are mainly useful for multimedia applications.  They are available at the
discretion of the installation.  Here’s an overview:

@menu
* audioop — Manipulate raw audio data:: 
* aifc — Read and write AIFF and AIFC files:: 
* sunau — Read and write Sun AU files:: 
* wave — Read and write WAV files:: 
* chunk — Read IFF chunked data:: 
* colorsys — Conversions between color systems:: 
* imghdr — Determine the type of an image:: 
* sndhdr — Determine type of sound file:: 
* ossaudiodev — Access to OSS-compatible audio devices:: 

@end menu

@node audioop — Manipulate raw audio data,aifc — Read and write AIFF and AIFC files,,Multimedia Services
@anchor{library/audioop doc}@anchor{340a}@anchor{library/audioop audioop-manipulate-raw-audio-data}@anchor{340b}@anchor{library/audioop module-audioop}@anchor{c}
@subsection @code{audioop} — Manipulate raw audio data


@geindex module; audioop


__________________________________________________________________


The @ref{c,,audioop} module contains some useful operations on sound fragments.
It operates on sound fragments consisting of signed integer samples 8, 16, 24
or 32 bits wide, stored in @ref{87e,,bytes-like objects}.  All scalar items are
integers, unless specified otherwise.


Changed in version 3.4: Support for 24-bit samples was added.
All functions now accept any @ref{87e,,bytes-like object}.
String input now results in an immediate error.


@geindex Intel/DVI ADPCM
@geindex ADPCM@comma{} Intel/DVI
@geindex a-LAW
@geindex u-LAW

This module provides support for a-LAW, u-LAW and Intel/DVI ADPCM encodings.

@c This para is mostly here to provide an excuse for the index entries...

A few of the more complicated operations only take 16-bit samples, otherwise the
sample size (in bytes) is always a parameter of the operation.

The module defines the following variables and functions:

@geindex error
@anchor{library/audioop audioop error}@anchor{340c}
@deffn {Exception} audioop.error

This exception is raised on all errors, such as unknown number of bytes per
sample, etc.
@end deffn

@geindex add() (in module audioop)
@anchor{library/audioop audioop add}@anchor{340d}
@deffn {Function} audioop.add (fragment1, fragment2, width)

Return a fragment which is the addition of the two samples passed as parameters.
@emph{width} is the sample width in bytes, either @code{1}, @code{2}, @code{3} or @code{4}.  Both
fragments should have the same length.  Samples are truncated in case of overflow.
@end deffn

@geindex adpcm2lin() (in module audioop)
@anchor{library/audioop audioop adpcm2lin}@anchor{340e}
@deffn {Function} audioop.adpcm2lin (adpcmfragment, width, state)

Decode an Intel/DVI ADPCM coded fragment to a linear fragment.  See the
description of @ref{340f,,lin2adpcm()} for details on ADPCM coding. Return a tuple
@code{(sample, newstate)} where the sample has the width specified in @emph{width}.
@end deffn

@geindex alaw2lin() (in module audioop)
@anchor{library/audioop audioop alaw2lin}@anchor{3410}
@deffn {Function} audioop.alaw2lin (fragment, width)

Convert sound fragments in a-LAW encoding to linearly encoded sound fragments.
a-LAW encoding always uses 8 bits samples, so @emph{width} refers only to the sample
width of the output fragment here.
@end deffn

@geindex avg() (in module audioop)
@anchor{library/audioop audioop avg}@anchor{3411}
@deffn {Function} audioop.avg (fragment, width)

Return the average over all samples in the fragment.
@end deffn

@geindex avgpp() (in module audioop)
@anchor{library/audioop audioop avgpp}@anchor{3412}
@deffn {Function} audioop.avgpp (fragment, width)

Return the average peak-peak value over all samples in the fragment. No
filtering is done, so the usefulness of this routine is questionable.
@end deffn

@geindex bias() (in module audioop)
@anchor{library/audioop audioop bias}@anchor{3413}
@deffn {Function} audioop.bias (fragment, width, bias)

Return a fragment that is the original fragment with a bias added to each
sample.  Samples wrap around in case of overflow.
@end deffn

@geindex byteswap() (in module audioop)
@anchor{library/audioop audioop byteswap}@anchor{a96}
@deffn {Function} audioop.byteswap (fragment, width)

“Byteswap” all samples in a fragment and returns the modified fragment.
Converts big-endian samples to little-endian and vice versa.


New in version 3.4.

@end deffn

@geindex cross() (in module audioop)
@anchor{library/audioop audioop cross}@anchor{3414}
@deffn {Function} audioop.cross (fragment, width)

Return the number of zero crossings in the fragment passed as an argument.
@end deffn

@geindex findfactor() (in module audioop)
@anchor{library/audioop audioop findfactor}@anchor{3415}
@deffn {Function} audioop.findfactor (fragment, reference)

Return a factor @emph{F} such that @code{rms(add(fragment, mul(reference, -F)))} is
minimal, i.e., return the factor with which you should multiply @emph{reference} to
make it match as well as possible to @emph{fragment}.  The fragments should both
contain 2-byte samples.

The time taken by this routine is proportional to @code{len(fragment)}.
@end deffn

@geindex findfit() (in module audioop)
@anchor{library/audioop audioop findfit}@anchor{3416}
@deffn {Function} audioop.findfit (fragment, reference)

Try to match @emph{reference} as well as possible to a portion of @emph{fragment} (which
should be the longer fragment).  This is (conceptually) done by taking slices
out of @emph{fragment}, using @ref{3415,,findfactor()} to compute the best match, and
minimizing the result.  The fragments should both contain 2-byte samples.
Return a tuple @code{(offset, factor)} where @emph{offset} is the (integer) offset into
@emph{fragment} where the optimal match started and @emph{factor} is the (floating-point)
factor as per @ref{3415,,findfactor()}.
@end deffn

@geindex findmax() (in module audioop)
@anchor{library/audioop audioop findmax}@anchor{3417}
@deffn {Function} audioop.findmax (fragment, length)

Search @emph{fragment} for a slice of length @emph{length} samples (not bytes!) with
maximum energy, i.e., return @emph{i} for which @code{rms(fragment[i*2:(i+length)*2])}
is maximal.  The fragments should both contain 2-byte samples.

The routine takes time proportional to @code{len(fragment)}.
@end deffn

@geindex getsample() (in module audioop)
@anchor{library/audioop audioop getsample}@anchor{3418}
@deffn {Function} audioop.getsample (fragment, width, index)

Return the value of sample @emph{index} from the fragment.
@end deffn

@geindex lin2adpcm() (in module audioop)
@anchor{library/audioop audioop lin2adpcm}@anchor{340f}
@deffn {Function} audioop.lin2adpcm (fragment, width, state)

Convert samples to 4 bit Intel/DVI ADPCM encoding.  ADPCM coding is an adaptive
coding scheme, whereby each 4 bit number is the difference between one sample
and the next, divided by a (varying) step.  The Intel/DVI ADPCM algorithm has
been selected for use by the IMA, so it may well become a standard.

@emph{state} is a tuple containing the state of the coder.  The coder returns a tuple
@code{(adpcmfrag, newstate)}, and the @emph{newstate} should be passed to the next call
of @ref{340f,,lin2adpcm()}.  In the initial call, @code{None} can be passed as the state.
@emph{adpcmfrag} is the ADPCM coded fragment packed 2 4-bit values per byte.
@end deffn

@geindex lin2alaw() (in module audioop)
@anchor{library/audioop audioop lin2alaw}@anchor{3419}
@deffn {Function} audioop.lin2alaw (fragment, width)

Convert samples in the audio fragment to a-LAW encoding and return this as a
bytes object.  a-LAW is an audio encoding format whereby you get a dynamic
range of about 13 bits using only 8 bit samples.  It is used by the Sun audio
hardware, among others.
@end deffn

@geindex lin2lin() (in module audioop)
@anchor{library/audioop audioop lin2lin}@anchor{341a}
@deffn {Function} audioop.lin2lin (fragment, width, newwidth)

Convert samples between 1-, 2-, 3- and 4-byte formats.

@cartouche
@quotation Note 
In some audio formats, such as .WAV files, 16, 24 and 32 bit samples are
signed, but 8 bit samples are unsigned.  So when converting to 8 bit wide
samples for these formats, you need to also add 128 to the result:

@example
new_frames = audioop.lin2lin(frames, old_width, 1)
new_frames = audioop.bias(new_frames, 1, 128)
@end example

The same, in reverse, has to be applied when converting from 8 to 16, 24
or 32 bit width samples.
@end quotation
@end cartouche
@end deffn

@geindex lin2ulaw() (in module audioop)
@anchor{library/audioop audioop lin2ulaw}@anchor{341b}
@deffn {Function} audioop.lin2ulaw (fragment, width)

Convert samples in the audio fragment to u-LAW encoding and return this as a
bytes object.  u-LAW is an audio encoding format whereby you get a dynamic
range of about 14 bits using only 8 bit samples.  It is used by the Sun audio
hardware, among others.
@end deffn

@geindex max() (in module audioop)
@anchor{library/audioop audioop max}@anchor{341c}
@deffn {Function} audioop.max (fragment, width)

Return the maximum of the @emph{absolute value} of all samples in a fragment.
@end deffn

@geindex maxpp() (in module audioop)
@anchor{library/audioop audioop maxpp}@anchor{341d}
@deffn {Function} audioop.maxpp (fragment, width)

Return the maximum peak-peak value in the sound fragment.
@end deffn

@geindex minmax() (in module audioop)
@anchor{library/audioop audioop minmax}@anchor{341e}
@deffn {Function} audioop.minmax (fragment, width)

Return a tuple consisting of the minimum and maximum values of all samples in
the sound fragment.
@end deffn

@geindex mul() (in module audioop)
@anchor{library/audioop audioop mul}@anchor{341f}
@deffn {Function} audioop.mul (fragment, width, factor)

Return a fragment that has all samples in the original fragment multiplied by
the floating-point value @emph{factor}.  Samples are truncated in case of overflow.
@end deffn

@geindex ratecv() (in module audioop)
@anchor{library/audioop audioop ratecv}@anchor{3420}
@deffn {Function} audioop.ratecv (fragment, width, nchannels, inrate, outrate, state[, weightA[, weightB]])

Convert the frame rate of the input fragment.

@emph{state} is a tuple containing the state of the converter.  The converter returns
a tuple @code{(newfragment, newstate)}, and @emph{newstate} should be passed to the next
call of @ref{3420,,ratecv()}.  The initial call should pass @code{None} as the state.

The @emph{weightA} and @emph{weightB} arguments are parameters for a simple digital filter
and default to @code{1} and @code{0} respectively.
@end deffn

@geindex reverse() (in module audioop)
@anchor{library/audioop audioop reverse}@anchor{3421}
@deffn {Function} audioop.reverse (fragment, width)

Reverse the samples in a fragment and returns the modified fragment.
@end deffn

@geindex rms() (in module audioop)
@anchor{library/audioop audioop rms}@anchor{3422}
@deffn {Function} audioop.rms (fragment, width)

Return the root-mean-square of the fragment, i.e. @code{sqrt(sum(S_i^2)/n)}.

This is a measure of the power in an audio signal.
@end deffn

@geindex tomono() (in module audioop)
@anchor{library/audioop audioop tomono}@anchor{3423}
@deffn {Function} audioop.tomono (fragment, width, lfactor, rfactor)

Convert a stereo fragment to a mono fragment.  The left channel is multiplied by
@emph{lfactor} and the right channel by @emph{rfactor} before adding the two channels to
give a mono signal.
@end deffn

@geindex tostereo() (in module audioop)
@anchor{library/audioop audioop tostereo}@anchor{3424}
@deffn {Function} audioop.tostereo (fragment, width, lfactor, rfactor)

Generate a stereo fragment from a mono fragment.  Each pair of samples in the
stereo fragment are computed from the mono sample, whereby left channel samples
are multiplied by @emph{lfactor} and right channel samples by @emph{rfactor}.
@end deffn

@geindex ulaw2lin() (in module audioop)
@anchor{library/audioop audioop ulaw2lin}@anchor{3425}
@deffn {Function} audioop.ulaw2lin (fragment, width)

Convert sound fragments in u-LAW encoding to linearly encoded sound fragments.
u-LAW encoding always uses 8 bits samples, so @emph{width} refers only to the sample
width of the output fragment here.
@end deffn

Note that operations such as @ref{341f,,mul()} or @ref{341c,,max()} make no distinction
between mono and stereo fragments, i.e. all samples are treated equal.  If this
is a problem the stereo fragment should be split into two mono fragments first
and recombined later.  Here is an example of how to do that:

@example
def mul_stereo(sample, width, lfactor, rfactor):
    lsample = audioop.tomono(sample, width, 1, 0)
    rsample = audioop.tomono(sample, width, 0, 1)
    lsample = audioop.mul(lsample, width, lfactor)
    rsample = audioop.mul(rsample, width, rfactor)
    lsample = audioop.tostereo(lsample, width, 1, 0)
    rsample = audioop.tostereo(rsample, width, 0, 1)
    return audioop.add(lsample, rsample, width)
@end example

If you use the ADPCM coder to build network packets and you want your protocol
to be stateless (i.e. to be able to tolerate packet loss) you should not only
transmit the data but also the state.  Note that you should send the @emph{initial}
state (the one you passed to @ref{340f,,lin2adpcm()}) along to the decoder, not the
final state (as returned by the coder).  If you want to use
@ref{1c8f,,struct.Struct} to store the state in binary you can code the first
element (the predicted value) in 16 bits and the second (the delta index) in 8.

The ADPCM coders have never been tried against other ADPCM coders, only against
themselves.  It could well be that I misinterpreted the standards in which case
they will not be interoperable with the respective standards.

The @code{find*()} routines might look a bit funny at first sight. They are
primarily meant to do echo cancellation.  A reasonably fast way to do this is to
pick the most energetic piece of the output sample, locate that in the input
sample and subtract the whole output sample from the input sample:

@example
def echocancel(outputdata, inputdata):
    pos = audioop.findmax(outputdata, 800)    # one tenth second
    out_test = outputdata[pos*2:]
    in_test = inputdata[pos*2:]
    ipos, factor = audioop.findfit(in_test, out_test)
    # Optional (for better cancellation):
    # factor = audioop.findfactor(in_test[ipos*2:ipos*2+len(out_test)],
    #              out_test)
    prefill = '\0'*(pos+ipos)*2
    postfill = '\0'*(len(inputdata)-len(prefill)-len(outputdata))
    outputdata = prefill + audioop.mul(outputdata, 2, -factor) + postfill
    return audioop.add(inputdata, outputdata, 2)
@end example

@node aifc — Read and write AIFF and AIFC files,sunau — Read and write Sun AU files,audioop — Manipulate raw audio data,Multimedia Services
@anchor{library/aifc doc}@anchor{3426}@anchor{library/aifc aifc-read-and-write-aiff-and-aifc-files}@anchor{3427}@anchor{library/aifc module-aifc}@anchor{4}
@subsection @code{aifc} — Read and write AIFF and AIFC files


@geindex module; aifc

@strong{Source code:} Lib/aifc.py@footnote{https://github.com/python/cpython/tree/main/Lib/aifc.py}

@geindex Audio Interchange File Format
@geindex AIFF
@geindex AIFF-C


__________________________________________________________________


This module provides support for reading and writing AIFF and AIFF-C files.
AIFF is Audio Interchange File Format, a format for storing digital audio
samples in a file.  AIFF-C is a newer version of the format that includes the
ability to compress the audio data.

Audio files have a number of parameters that describe the audio data. The
sampling rate or frame rate is the number of times per second the sound is
sampled.  The number of channels indicate if the audio is mono, stereo, or
quadro.  Each frame consists of one sample per channel.  The sample size is the
size in bytes of each sample.  Thus a frame consists of
@code{nchannels * samplesize} bytes, and a second’s worth of audio consists of
@code{nchannels * samplesize * framerate} bytes.

For example, CD quality audio has a sample size of two bytes (16 bits), uses two
channels (stereo) and has a frame rate of 44,100 frames/second.  This gives a
frame size of 4 bytes (2*2), and a second’s worth occupies 2*2*44100 bytes
(176,400 bytes).

Module @ref{4,,aifc} defines the following function:

@geindex open() (in module aifc)
@anchor{library/aifc aifc open}@anchor{718}
@deffn {Function} aifc.open (file, mode=None)

Open an AIFF or AIFF-C file and return an object instance with methods that are
described below.  The argument @emph{file} is either a string naming a file or a
@ref{d68,,file object}.  @emph{mode} must be @code{'r'} or @code{'rb'} when the file must be
opened for reading, or @code{'w'}  or @code{'wb'} when the file must be opened for writing.
If omitted, @code{file.mode} is used if it exists, otherwise @code{'rb'} is used.  When
used for writing, the file object should be seekable, unless you know ahead of
time how many samples you are going to write in total and use
@code{writeframesraw()} and @code{setnframes()}.
The @ref{718,,open()} function may be used in a @ref{168,,with} statement.  When
the @code{with} block completes, the @ref{a90,,close()} method is called.


Changed in version 3.4: Support for the @ref{168,,with} statement was added.

@end deffn

Objects returned by @ref{718,,open()} when a file is opened for reading have the
following methods:

@geindex getnchannels() (aifc.aifc method)
@anchor{library/aifc aifc aifc getnchannels}@anchor{3428}
@deffn {Method} aifc.getnchannels ()

Return the number of audio channels (1 for mono, 2 for stereo).
@end deffn

@geindex getsampwidth() (aifc.aifc method)
@anchor{library/aifc aifc aifc getsampwidth}@anchor{3429}
@deffn {Method} aifc.getsampwidth ()

Return the size in bytes of individual samples.
@end deffn

@geindex getframerate() (aifc.aifc method)
@anchor{library/aifc aifc aifc getframerate}@anchor{342a}
@deffn {Method} aifc.getframerate ()

Return the sampling rate (number of audio frames per second).
@end deffn

@geindex getnframes() (aifc.aifc method)
@anchor{library/aifc aifc aifc getnframes}@anchor{342b}
@deffn {Method} aifc.getnframes ()

Return the number of audio frames in the file.
@end deffn

@geindex getcomptype() (aifc.aifc method)
@anchor{library/aifc aifc aifc getcomptype}@anchor{342c}
@deffn {Method} aifc.getcomptype ()

Return a bytes array of length 4 describing the type of compression
used in the audio file.  For AIFF files, the returned value is
@code{b'NONE'}.
@end deffn

@geindex getcompname() (aifc.aifc method)
@anchor{library/aifc aifc aifc getcompname}@anchor{342d}
@deffn {Method} aifc.getcompname ()

Return a bytes array convertible to a human-readable description
of the type of compression used in the audio file.  For AIFF files,
the returned value is @code{b'not compressed'}.
@end deffn

@geindex getparams() (aifc.aifc method)
@anchor{library/aifc aifc aifc getparams}@anchor{a8f}
@deffn {Method} aifc.getparams ()

Returns a @ref{3e4,,namedtuple()} @code{(nchannels, sampwidth,
framerate, nframes, comptype, compname)}, equivalent to output of the
@code{get*()} methods.
@end deffn

@geindex getmarkers() (aifc.aifc method)
@anchor{library/aifc aifc aifc getmarkers}@anchor{342e}
@deffn {Method} aifc.getmarkers ()

Return a list of markers in the audio file.  A marker consists of a tuple of
three elements.  The first is the mark ID (an integer), the second is the mark
position in frames from the beginning of the data (an integer), the third is the
name of the mark (a string).
@end deffn

@geindex getmark() (aifc.aifc method)
@anchor{library/aifc aifc aifc getmark}@anchor{342f}
@deffn {Method} aifc.getmark (id)

Return the tuple as described in @ref{342e,,getmarkers()} for the mark with the given
@emph{id}.
@end deffn

@geindex readframes() (aifc.aifc method)
@anchor{library/aifc aifc aifc readframes}@anchor{3430}
@deffn {Method} aifc.readframes (nframes)

Read and return the next @emph{nframes} frames from the audio file.  The returned
data is a string containing for each frame the uncompressed samples of all
channels.
@end deffn

@geindex rewind() (aifc.aifc method)
@anchor{library/aifc aifc aifc rewind}@anchor{3431}
@deffn {Method} aifc.rewind ()

Rewind the read pointer.  The next @ref{3430,,readframes()} will start from the
beginning.
@end deffn

@geindex setpos() (aifc.aifc method)
@anchor{library/aifc aifc aifc setpos}@anchor{3432}
@deffn {Method} aifc.setpos (pos)

Seek to the specified frame number.
@end deffn

@geindex tell() (aifc.aifc method)
@anchor{library/aifc aifc aifc tell}@anchor{3433}
@deffn {Method} aifc.tell ()

Return the current frame number.
@end deffn

@geindex close() (aifc.aifc method)
@anchor{library/aifc aifc aifc close}@anchor{a90}
@deffn {Method} aifc.close ()

Close the AIFF file.  After calling this method, the object can no longer be
used.
@end deffn

Objects returned by @ref{718,,open()} when a file is opened for writing have all the
above methods, except for @code{readframes()} and @code{setpos()}.  In addition
the following methods exist.  The @code{get*()} methods can only be called after
the corresponding @code{set*()} methods have been called.  Before the first
@code{writeframes()} or @code{writeframesraw()}, all parameters except for the
number of frames must be filled in.

@geindex aiff() (aifc.aifc method)
@anchor{library/aifc aifc aifc aiff}@anchor{3434}
@deffn {Method} aifc.aiff ()

Create an AIFF file.  The default is that an AIFF-C file is created, unless the
name of the file ends in @code{'.aiff'} in which case the default is an AIFF file.
@end deffn

@geindex aifc() (aifc.aifc method)
@anchor{library/aifc aifc aifc aifc}@anchor{3435}
@deffn {Method} aifc.aifc ()

Create an AIFF-C file.  The default is that an AIFF-C file is created, unless
the name of the file ends in @code{'.aiff'} in which case the default is an AIFF
file.
@end deffn

@geindex setnchannels() (aifc.aifc method)
@anchor{library/aifc aifc aifc setnchannels}@anchor{3436}
@deffn {Method} aifc.setnchannels (nchannels)

Specify the number of channels in the audio file.
@end deffn

@geindex setsampwidth() (aifc.aifc method)
@anchor{library/aifc aifc aifc setsampwidth}@anchor{3437}
@deffn {Method} aifc.setsampwidth (width)

Specify the size in bytes of audio samples.
@end deffn

@geindex setframerate() (aifc.aifc method)
@anchor{library/aifc aifc aifc setframerate}@anchor{3438}
@deffn {Method} aifc.setframerate (rate)

Specify the sampling frequency in frames per second.
@end deffn

@geindex setnframes() (aifc.aifc method)
@anchor{library/aifc aifc aifc setnframes}@anchor{3439}
@deffn {Method} aifc.setnframes (nframes)

Specify the number of frames that are to be written to the audio file. If this
parameter is not set, or not set correctly, the file needs to support seeking.
@end deffn

@geindex setcomptype() (aifc.aifc method)
@anchor{library/aifc aifc aifc setcomptype}@anchor{343a}
@deffn {Method} aifc.setcomptype (type, name)

@geindex u-LAW
@geindex A-LAW
@geindex G.722

Specify the compression type.  If not specified, the audio data will
not be compressed.  In AIFF files, compression is not possible.
The name parameter should be a human-readable description of the
compression type as a bytes array, the type parameter should be a
bytes array of length 4.  Currently the following compression types
are supported: @code{b'NONE'}, @code{b'ULAW'}, @code{b'ALAW'}, @code{b'G722'}.
@end deffn

@geindex setparams() (aifc.aifc method)
@anchor{library/aifc aifc aifc setparams}@anchor{343b}
@deffn {Method} aifc.setparams (nchannels, sampwidth, framerate, comptype, compname)

Set all the above parameters at once.  The argument is a tuple consisting of the
various parameters.  This means that it is possible to use the result of a
@ref{a8f,,getparams()} call as argument to @ref{343b,,setparams()}.
@end deffn

@geindex setmark() (aifc.aifc method)
@anchor{library/aifc aifc aifc setmark}@anchor{343c}
@deffn {Method} aifc.setmark (id, pos, name)

Add a mark with the given id (larger than 0), and the given name at the given
position.  This method can be called at any time before @ref{a90,,close()}.
@end deffn


@deffn {Method} aifc.tell ()

Return the current write position in the output file.  Useful in combination
with @ref{343c,,setmark()}.
@end deffn

@geindex writeframes() (aifc.aifc method)
@anchor{library/aifc aifc aifc writeframes}@anchor{a92}
@deffn {Method} aifc.writeframes (data)

Write data to the output file.  This method can only be called after the audio
file parameters have been set.


Changed in version 3.4: Any @ref{87e,,bytes-like object} is now accepted.

@end deffn

@geindex writeframesraw() (aifc.aifc method)
@anchor{library/aifc aifc aifc writeframesraw}@anchor{a91}
@deffn {Method} aifc.writeframesraw (data)

Like @ref{a92,,writeframes()}, except that the header of the audio file is not
updated.


Changed in version 3.4: Any @ref{87e,,bytes-like object} is now accepted.

@end deffn


@deffn {Method} aifc.close ()

Close the AIFF file.  The header of the file is updated to reflect the actual
size of the audio data. After calling this method, the object can no longer be
used.
@end deffn

@node sunau — Read and write Sun AU files,wave — Read and write WAV files,aifc — Read and write AIFF and AIFC files,Multimedia Services
@anchor{library/sunau doc}@anchor{343d}@anchor{library/sunau module-sunau}@anchor{f5}@anchor{library/sunau sunau-read-and-write-sun-au-files}@anchor{343e}
@subsection @code{sunau} — Read and write Sun AU files


@geindex module; sunau

@strong{Source code:} Lib/sunau.py@footnote{https://github.com/python/cpython/tree/main/Lib/sunau.py}


__________________________________________________________________


The @ref{f5,,sunau} module provides a convenient interface to the Sun AU sound
format.  Note that this module is interface-compatible with the modules
@ref{4,,aifc} and @ref{12e,,wave}.

An audio file consists of a header followed by the data.  The fields of the
header are:


@multitable {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Field

@tab

Contents

@item

magic word

@tab

The four bytes @code{.snd}.

@item

header size

@tab

Size of the header, including info, in bytes.

@item

data size

@tab

Physical size of the data, in bytes.

@item

encoding

@tab

Indicates how the audio samples are encoded.

@item

sample rate

@tab

The sampling rate.

@item

# of channels

@tab

The number of channels in the samples.

@item

info

@tab

ASCII string giving a description of the
audio file (padded with null bytes).

@end multitable


Apart from the info field, all header fields are 4 bytes in size. They are all
32-bit unsigned integers encoded in big-endian byte order.

The @ref{f5,,sunau} module defines the following functions:

@geindex open() (in module sunau)
@anchor{library/sunau sunau open}@anchor{727}
@deffn {Function} sunau.open (file, mode)

If @emph{file} is a string, open the file by that name, otherwise treat it as a
seekable file-like object. @emph{mode} can be any of


@table @asis

@item @code{'r'}

Read only mode.

@item @code{'w'}

Write only mode.
@end table

Note that it does not allow read/write files.

A @emph{mode} of @code{'r'} returns an @code{AU_read} object, while a @emph{mode} of @code{'w'}
or @code{'wb'} returns an @code{AU_write} object.
@end deffn

The @ref{f5,,sunau} module defines the following exception:

@geindex Error
@anchor{library/sunau sunau Error}@anchor{343f}
@deffn {Exception} sunau.Error

An error raised when something is impossible because of Sun AU specs or
implementation deficiency.
@end deffn

The @ref{f5,,sunau} module defines the following data items:

@geindex AUDIO_FILE_MAGIC (in module sunau)
@anchor{library/sunau sunau AUDIO_FILE_MAGIC}@anchor{3440}
@deffn {Data} sunau.AUDIO_FILE_MAGIC

An integer every valid Sun AU file begins with, stored in big-endian form.  This
is the string @code{.snd} interpreted as an integer.
@end deffn

@geindex AUDIO_FILE_ENCODING_MULAW_8 (in module sunau)
@geindex AUDIO_FILE_ENCODING_LINEAR_8 (in module sunau)
@geindex AUDIO_FILE_ENCODING_LINEAR_16 (in module sunau)
@geindex AUDIO_FILE_ENCODING_LINEAR_24 (in module sunau)
@geindex AUDIO_FILE_ENCODING_LINEAR_32 (in module sunau)
@geindex AUDIO_FILE_ENCODING_ALAW_8 (in module sunau)
@anchor{library/sunau sunau AUDIO_FILE_ENCODING_MULAW_8}@anchor{3441}
@deffn {Data} sunau.AUDIO_FILE_ENCODING_MULAW_8
@anchor{library/sunau sunau AUDIO_FILE_ENCODING_LINEAR_8}@anchor{3442}
@deffnx {Data} sunau.AUDIO_FILE_ENCODING_LINEAR_8
@anchor{library/sunau sunau AUDIO_FILE_ENCODING_LINEAR_16}@anchor{3443}
@deffnx {Data} sunau.AUDIO_FILE_ENCODING_LINEAR_16
@anchor{library/sunau sunau AUDIO_FILE_ENCODING_LINEAR_24}@anchor{3444}
@deffnx {Data} sunau.AUDIO_FILE_ENCODING_LINEAR_24
@anchor{library/sunau sunau AUDIO_FILE_ENCODING_LINEAR_32}@anchor{3445}
@deffnx {Data} sunau.AUDIO_FILE_ENCODING_LINEAR_32
@anchor{library/sunau sunau AUDIO_FILE_ENCODING_ALAW_8}@anchor{3446}
@deffnx {Data} sunau.AUDIO_FILE_ENCODING_ALAW_8

Values of the encoding field from the AU header which are supported by this
module.
@end deffn

@geindex AUDIO_FILE_ENCODING_FLOAT (in module sunau)
@geindex AUDIO_FILE_ENCODING_DOUBLE (in module sunau)
@geindex AUDIO_FILE_ENCODING_ADPCM_G721 (in module sunau)
@geindex AUDIO_FILE_ENCODING_ADPCM_G722 (in module sunau)
@geindex AUDIO_FILE_ENCODING_ADPCM_G723_3 (in module sunau)
@geindex AUDIO_FILE_ENCODING_ADPCM_G723_5 (in module sunau)
@anchor{library/sunau sunau AUDIO_FILE_ENCODING_FLOAT}@anchor{3447}
@deffn {Data} sunau.AUDIO_FILE_ENCODING_FLOAT
@anchor{library/sunau sunau AUDIO_FILE_ENCODING_DOUBLE}@anchor{3448}
@deffnx {Data} sunau.AUDIO_FILE_ENCODING_DOUBLE
@anchor{library/sunau sunau AUDIO_FILE_ENCODING_ADPCM_G721}@anchor{3449}
@deffnx {Data} sunau.AUDIO_FILE_ENCODING_ADPCM_G721
@anchor{library/sunau sunau AUDIO_FILE_ENCODING_ADPCM_G722}@anchor{344a}
@deffnx {Data} sunau.AUDIO_FILE_ENCODING_ADPCM_G722
@anchor{library/sunau sunau AUDIO_FILE_ENCODING_ADPCM_G723_3}@anchor{344b}
@deffnx {Data} sunau.AUDIO_FILE_ENCODING_ADPCM_G723_3
@anchor{library/sunau sunau AUDIO_FILE_ENCODING_ADPCM_G723_5}@anchor{344c}
@deffnx {Data} sunau.AUDIO_FILE_ENCODING_ADPCM_G723_5

Additional known values of the encoding field from the AU header, but which are
not supported by this module.
@end deffn

@menu
* AU_read Objects:: 
* AU_write Objects:: 

@end menu

@node AU_read Objects,AU_write Objects,,sunau — Read and write Sun AU files
@anchor{library/sunau au-read-objects}@anchor{344d}@anchor{library/sunau id1}@anchor{344e}
@subsubsection AU_read Objects


AU_read objects, as returned by @ref{727,,open()} above, have the following methods:

@geindex close() (sunau.AU_read method)
@anchor{library/sunau sunau AU_read close}@anchor{344f}
@deffn {Method} AU_read.close ()

Close the stream, and make the instance unusable. (This is  called automatically
on deletion.)
@end deffn

@geindex getnchannels() (sunau.AU_read method)
@anchor{library/sunau sunau AU_read getnchannels}@anchor{3450}
@deffn {Method} AU_read.getnchannels ()

Returns number of audio channels (1 for mono, 2 for stereo).
@end deffn

@geindex getsampwidth() (sunau.AU_read method)
@anchor{library/sunau sunau AU_read getsampwidth}@anchor{3451}
@deffn {Method} AU_read.getsampwidth ()

Returns sample width in bytes.
@end deffn

@geindex getframerate() (sunau.AU_read method)
@anchor{library/sunau sunau AU_read getframerate}@anchor{3452}
@deffn {Method} AU_read.getframerate ()

Returns sampling frequency.
@end deffn

@geindex getnframes() (sunau.AU_read method)
@anchor{library/sunau sunau AU_read getnframes}@anchor{3453}
@deffn {Method} AU_read.getnframes ()

Returns number of audio frames.
@end deffn

@geindex getcomptype() (sunau.AU_read method)
@anchor{library/sunau sunau AU_read getcomptype}@anchor{3454}
@deffn {Method} AU_read.getcomptype ()

Returns compression type. Supported compression types are @code{'ULAW'}, @code{'ALAW'}
and @code{'NONE'}.
@end deffn

@geindex getcompname() (sunau.AU_read method)
@anchor{library/sunau sunau AU_read getcompname}@anchor{3455}
@deffn {Method} AU_read.getcompname ()

Human-readable version of @ref{3454,,getcomptype()}.  The supported types have the
respective names @code{'CCITT G.711 u-law'}, @code{'CCITT G.711 A-law'} and @code{'not
compressed'}.
@end deffn

@geindex getparams() (sunau.AU_read method)
@anchor{library/sunau sunau AU_read getparams}@anchor{3456}
@deffn {Method} AU_read.getparams ()

Returns a @ref{3e4,,namedtuple()} @code{(nchannels, sampwidth,
framerate, nframes, comptype, compname)}, equivalent to output of the
@code{get*()} methods.
@end deffn

@geindex readframes() (sunau.AU_read method)
@anchor{library/sunau sunau AU_read readframes}@anchor{3457}
@deffn {Method} AU_read.readframes (n)

Reads and returns at most @emph{n} frames of audio, as a @ref{2f0,,bytes} object.  The data
will be returned in linear format.  If the original data is in u-LAW format, it
will be converted.
@end deffn

@geindex rewind() (sunau.AU_read method)
@anchor{library/sunau sunau AU_read rewind}@anchor{3458}
@deffn {Method} AU_read.rewind ()

Rewind the file pointer to the beginning of the audio stream.
@end deffn

The following two methods define a term “position” which is compatible between
them, and is otherwise implementation dependent.

@geindex setpos() (sunau.AU_read method)
@anchor{library/sunau sunau AU_read setpos}@anchor{3459}
@deffn {Method} AU_read.setpos (pos)

Set the file pointer to the specified position.  Only values returned from
@ref{345a,,tell()} should be used for @emph{pos}.
@end deffn

@geindex tell() (sunau.AU_read method)
@anchor{library/sunau sunau AU_read tell}@anchor{345a}
@deffn {Method} AU_read.tell ()

Return current file pointer position.  Note that the returned value has nothing
to do with the actual position in the file.
@end deffn

The following two functions are defined for compatibility with the  @ref{4,,aifc},
and don’t do anything interesting.

@geindex getmarkers() (sunau.AU_read method)
@anchor{library/sunau sunau AU_read getmarkers}@anchor{345b}
@deffn {Method} AU_read.getmarkers ()

Returns @code{None}.
@end deffn

@geindex getmark() (sunau.AU_read method)
@anchor{library/sunau sunau AU_read getmark}@anchor{345c}
@deffn {Method} AU_read.getmark (id)

Raise an error.
@end deffn

@node AU_write Objects,,AU_read Objects,sunau — Read and write Sun AU files
@anchor{library/sunau au-write-objects}@anchor{345d}@anchor{library/sunau id2}@anchor{345e}
@subsubsection AU_write Objects


AU_write objects, as returned by @ref{727,,open()} above, have the following methods:

@geindex setnchannels() (sunau.AU_write method)
@anchor{library/sunau sunau AU_write setnchannels}@anchor{345f}
@deffn {Method} AU_write.setnchannels (n)

Set the number of channels.
@end deffn

@geindex setsampwidth() (sunau.AU_write method)
@anchor{library/sunau sunau AU_write setsampwidth}@anchor{b46}
@deffn {Method} AU_write.setsampwidth (n)

Set the sample width (in bytes.)


Changed in version 3.4: Added support for 24-bit samples.

@end deffn

@geindex setframerate() (sunau.AU_write method)
@anchor{library/sunau sunau AU_write setframerate}@anchor{3460}
@deffn {Method} AU_write.setframerate (n)

Set the frame rate.
@end deffn

@geindex setnframes() (sunau.AU_write method)
@anchor{library/sunau sunau AU_write setnframes}@anchor{3461}
@deffn {Method} AU_write.setnframes (n)

Set the number of frames. This can be later changed, when and if more  frames
are written.
@end deffn

@geindex setcomptype() (sunau.AU_write method)
@anchor{library/sunau sunau AU_write setcomptype}@anchor{3462}
@deffn {Method} AU_write.setcomptype (type, name)

Set the compression type and description. Only @code{'NONE'} and @code{'ULAW'} are
supported on output.
@end deffn

@geindex setparams() (sunau.AU_write method)
@anchor{library/sunau sunau AU_write setparams}@anchor{3463}
@deffn {Method} AU_write.setparams (tuple)

The @emph{tuple} should be @code{(nchannels, sampwidth, framerate, nframes, comptype,
compname)}, with values valid for the @code{set*()} methods.  Set all
parameters.
@end deffn

@geindex tell() (sunau.AU_write method)
@anchor{library/sunau sunau AU_write tell}@anchor{3464}
@deffn {Method} AU_write.tell ()

Return current position in the file, with the same disclaimer for the
@ref{345a,,AU_read.tell()} and @ref{3459,,AU_read.setpos()} methods.
@end deffn

@geindex writeframesraw() (sunau.AU_write method)
@anchor{library/sunau sunau AU_write writeframesraw}@anchor{b47}
@deffn {Method} AU_write.writeframesraw (data)

Write audio frames, without correcting @emph{nframes}.


Changed in version 3.4: Any @ref{87e,,bytes-like object} is now accepted.

@end deffn

@geindex writeframes() (sunau.AU_write method)
@anchor{library/sunau sunau AU_write writeframes}@anchor{b48}
@deffn {Method} AU_write.writeframes (data)

Write audio frames and make sure @emph{nframes} is correct.


Changed in version 3.4: Any @ref{87e,,bytes-like object} is now accepted.

@end deffn

@geindex close() (sunau.AU_write method)
@anchor{library/sunau sunau AU_write close}@anchor{3465}
@deffn {Method} AU_write.close ()

Make sure @emph{nframes} is correct, and close the file.

This method is called upon deletion.
@end deffn

Note that it is invalid to set any parameters after calling  @code{writeframes()}
or @code{writeframesraw()}.

@node wave — Read and write WAV files,chunk — Read IFF chunked data,sunau — Read and write Sun AU files,Multimedia Services
@anchor{library/wave doc}@anchor{3466}@anchor{library/wave module-wave}@anchor{12e}@anchor{library/wave wave-read-and-write-wav-files}@anchor{3467}
@subsection @code{wave} — Read and write WAV files


@geindex module; wave

@c Documentations stolen from comments in file.

@strong{Source code:} Lib/wave.py@footnote{https://github.com/python/cpython/tree/main/Lib/wave.py}


__________________________________________________________________


The @ref{12e,,wave} module provides a convenient interface to the WAV sound format.
It does not support compression/decompression, but it does support mono/stereo.

The @ref{12e,,wave} module defines the following function and exception:

@geindex open() (in module wave)
@anchor{library/wave wave open}@anchor{44a}
@deffn {Function} wave.open (file, mode=None)

If @emph{file} is a string, open the file by that name, otherwise treat it as a
file-like object.  @emph{mode} can be:


@table @asis

@item @code{'rb'}

Read only mode.

@item @code{'wb'}

Write only mode.
@end table

Note that it does not allow read/write WAV files.

A @emph{mode} of @code{'rb'} returns a @code{Wave_read} object, while a @emph{mode} of
@code{'wb'} returns a @code{Wave_write} object.  If @emph{mode} is omitted and a
file-like object is passed as @emph{file}, @code{file.mode} is used as the default
value for @emph{mode}.

If you pass in a file-like object, the wave object will not close it when its
@code{close()} method is called; it is the caller’s responsibility to close
the file object.

The @ref{44a,,open()} function may be used in a @ref{168,,with} statement.  When
the @code{with} block completes, the @ref{3468,,Wave_read.close()} or @ref{3469,,Wave_write.close()} method is called.


Changed in version 3.4: Added support for unseekable files.

@end deffn

@geindex Error
@anchor{library/wave wave Error}@anchor{346a}
@deffn {Exception} wave.Error

An error raised when something is impossible because it violates the WAV
specification or hits an implementation deficiency.
@end deffn

@menu
* Wave_read Objects:: 
* Wave_write Objects:: 

@end menu

@node Wave_read Objects,Wave_write Objects,,wave — Read and write WAV files
@anchor{library/wave id1}@anchor{346b}@anchor{library/wave wave-read-objects}@anchor{346c}
@subsubsection Wave_read Objects


Wave_read objects, as returned by @ref{44a,,open()}, have the following methods:

@geindex close() (wave.Wave_read method)
@anchor{library/wave wave Wave_read close}@anchor{3468}
@deffn {Method} Wave_read.close ()

Close the stream if it was opened by @ref{12e,,wave}, and make the instance
unusable.  This is called automatically on object collection.
@end deffn

@geindex getnchannels() (wave.Wave_read method)
@anchor{library/wave wave Wave_read getnchannels}@anchor{346d}
@deffn {Method} Wave_read.getnchannels ()

Returns number of audio channels (@code{1} for mono, @code{2} for stereo).
@end deffn

@geindex getsampwidth() (wave.Wave_read method)
@anchor{library/wave wave Wave_read getsampwidth}@anchor{346e}
@deffn {Method} Wave_read.getsampwidth ()

Returns sample width in bytes.
@end deffn

@geindex getframerate() (wave.Wave_read method)
@anchor{library/wave wave Wave_read getframerate}@anchor{346f}
@deffn {Method} Wave_read.getframerate ()

Returns sampling frequency.
@end deffn

@geindex getnframes() (wave.Wave_read method)
@anchor{library/wave wave Wave_read getnframes}@anchor{3470}
@deffn {Method} Wave_read.getnframes ()

Returns number of audio frames.
@end deffn

@geindex getcomptype() (wave.Wave_read method)
@anchor{library/wave wave Wave_read getcomptype}@anchor{3471}
@deffn {Method} Wave_read.getcomptype ()

Returns compression type (@code{'NONE'} is the only supported type).
@end deffn

@geindex getcompname() (wave.Wave_read method)
@anchor{library/wave wave Wave_read getcompname}@anchor{3472}
@deffn {Method} Wave_read.getcompname ()

Human-readable version of @ref{3471,,getcomptype()}. Usually @code{'not compressed'}
parallels @code{'NONE'}.
@end deffn

@geindex getparams() (wave.Wave_read method)
@anchor{library/wave wave Wave_read getparams}@anchor{3473}
@deffn {Method} Wave_read.getparams ()

Returns a @ref{3e4,,namedtuple()} @code{(nchannels, sampwidth,
framerate, nframes, comptype, compname)}, equivalent to output of the
@code{get*()} methods.
@end deffn

@geindex readframes() (wave.Wave_read method)
@anchor{library/wave wave Wave_read readframes}@anchor{3474}
@deffn {Method} Wave_read.readframes (n)

Reads and returns at most @emph{n} frames of audio, as a @ref{2f0,,bytes} object.
@end deffn

@geindex rewind() (wave.Wave_read method)
@anchor{library/wave wave Wave_read rewind}@anchor{3475}
@deffn {Method} Wave_read.rewind ()

Rewind the file pointer to the beginning of the audio stream.
@end deffn

The following two methods are defined for compatibility with the @ref{4,,aifc}
module, and don’t do anything interesting.

@geindex getmarkers() (wave.Wave_read method)
@anchor{library/wave wave Wave_read getmarkers}@anchor{3476}
@deffn {Method} Wave_read.getmarkers ()

Returns @code{None}.
@end deffn

@geindex getmark() (wave.Wave_read method)
@anchor{library/wave wave Wave_read getmark}@anchor{3477}
@deffn {Method} Wave_read.getmark (id)

Raise an error.
@end deffn

The following two methods define a term “position” which is compatible between
them, and is otherwise implementation dependent.

@geindex setpos() (wave.Wave_read method)
@anchor{library/wave wave Wave_read setpos}@anchor{3478}
@deffn {Method} Wave_read.setpos (pos)

Set the file pointer to the specified position.
@end deffn

@geindex tell() (wave.Wave_read method)
@anchor{library/wave wave Wave_read tell}@anchor{3479}
@deffn {Method} Wave_read.tell ()

Return current file pointer position.
@end deffn

@node Wave_write Objects,,Wave_read Objects,wave — Read and write WAV files
@anchor{library/wave id2}@anchor{347a}@anchor{library/wave wave-write-objects}@anchor{b71}
@subsubsection Wave_write Objects


For seekable output streams, the @code{wave} header will automatically be updated
to reflect the number of frames actually written.  For unseekable streams, the
@emph{nframes} value must be accurate when the first frame data is written.  An
accurate @emph{nframes} value can be achieved either by calling
@ref{347b,,setnframes()} or @ref{347c,,setparams()} with the number
of frames that will be written before @ref{3469,,close()} is called and
then using @ref{b72,,writeframesraw()} to write the frame data, or by
calling @ref{b73,,writeframes()} with all of the frame data to be
written.  In the latter case @ref{b73,,writeframes()} will calculate
the number of frames in the data and set @emph{nframes} accordingly before writing
the frame data.

Wave_write objects, as returned by @ref{44a,,open()}, have the following methods:


Changed in version 3.4: Added support for unseekable files.


@geindex close() (wave.Wave_write method)
@anchor{library/wave wave Wave_write close}@anchor{3469}
@deffn {Method} Wave_write.close ()

Make sure @emph{nframes} is correct, and close the file if it was opened by
@ref{12e,,wave}.  This method is called upon object collection.  It will raise
an exception if the output stream is not seekable and @emph{nframes} does not
match the number of frames actually written.
@end deffn

@geindex setnchannels() (wave.Wave_write method)
@anchor{library/wave wave Wave_write setnchannels}@anchor{347d}
@deffn {Method} Wave_write.setnchannels (n)

Set the number of channels.
@end deffn

@geindex setsampwidth() (wave.Wave_write method)
@anchor{library/wave wave Wave_write setsampwidth}@anchor{347e}
@deffn {Method} Wave_write.setsampwidth (n)

Set the sample width to @emph{n} bytes.
@end deffn

@geindex setframerate() (wave.Wave_write method)
@anchor{library/wave wave Wave_write setframerate}@anchor{347f}
@deffn {Method} Wave_write.setframerate (n)

Set the frame rate to @emph{n}.


Changed in version 3.2: A non-integral input to this method is rounded to the nearest
integer.

@end deffn

@geindex setnframes() (wave.Wave_write method)
@anchor{library/wave wave Wave_write setnframes}@anchor{347b}
@deffn {Method} Wave_write.setnframes (n)

Set the number of frames to @emph{n}.  This will be changed later if the number
of frames actually written is different (this update attempt will
raise an error if the output stream is not seekable).
@end deffn

@geindex setcomptype() (wave.Wave_write method)
@anchor{library/wave wave Wave_write setcomptype}@anchor{3480}
@deffn {Method} Wave_write.setcomptype (type, name)

Set the compression type and description. At the moment, only compression type
@code{NONE} is supported, meaning no compression.
@end deffn

@geindex setparams() (wave.Wave_write method)
@anchor{library/wave wave Wave_write setparams}@anchor{347c}
@deffn {Method} Wave_write.setparams (tuple)

The @emph{tuple} should be @code{(nchannels, sampwidth, framerate, nframes, comptype,
compname)}, with values valid for the @code{set*()} methods.  Sets all
parameters.
@end deffn

@geindex tell() (wave.Wave_write method)
@anchor{library/wave wave Wave_write tell}@anchor{3481}
@deffn {Method} Wave_write.tell ()

Return current position in the file, with the same disclaimer for the
@ref{3479,,Wave_read.tell()} and @ref{3478,,Wave_read.setpos()} methods.
@end deffn

@geindex writeframesraw() (wave.Wave_write method)
@anchor{library/wave wave Wave_write writeframesraw}@anchor{b72}
@deffn {Method} Wave_write.writeframesraw (data)

Write audio frames, without correcting @emph{nframes}.


Changed in version 3.4: Any @ref{87e,,bytes-like object} is now accepted.

@end deffn

@geindex writeframes() (wave.Wave_write method)
@anchor{library/wave wave Wave_write writeframes}@anchor{b73}
@deffn {Method} Wave_write.writeframes (data)

Write audio frames and make sure @emph{nframes} is correct.  It will raise an
error if the output stream is not seekable and the total number of frames
that have been written after @emph{data} has been written does not match the
previously set value for @emph{nframes}.


Changed in version 3.4: Any @ref{87e,,bytes-like object} is now accepted.

@end deffn

Note that it is invalid to set any parameters after calling @code{writeframes()}
or @code{writeframesraw()}, and any attempt to do so will raise
@ref{346a,,wave.Error}.

@node chunk — Read IFF chunked data,colorsys — Conversions between color systems,wave — Read and write WAV files,Multimedia Services
@anchor{library/chunk doc}@anchor{3482}@anchor{library/chunk chunk-read-iff-chunked-data}@anchor{3483}@anchor{library/chunk module-chunk}@anchor{16}
@subsection @code{chunk} — Read IFF chunked data


@geindex module; chunk

@strong{Source code:} Lib/chunk.py@footnote{https://github.com/python/cpython/tree/main/Lib/chunk.py}

@geindex Audio Interchange File Format
@geindex AIFF
@geindex AIFF-C
@geindex Real Media File Format
@geindex RMFF


__________________________________________________________________


This module provides an interface for reading files that use EA IFF 85 chunks.
@footnote{@w{(1)} 
“EA IFF 85” Standard for Interchange Format Files, Jerry Morrison, Electronic
Arts, January 1985.
}  This format is used in at least the Audio Interchange File Format
(AIFF/AIFF-C) and the Real Media File Format (RMFF).  The WAVE audio file format
is closely related and can also be read using this module.

A chunk has the following structure:


@multitable {xxxxxxxxxxx} {xxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Offset

@tab

Length

@tab

Contents

@item

0

@tab

4

@tab

Chunk ID

@item

4

@tab

4

@tab

Size of chunk in big-endian
byte order, not including the
header

@item

8

@tab

@emph{n}

@tab

Data bytes, where @emph{n} is the
size given in the preceding
field

@item

8 + @emph{n}

@tab

0 or 1

@tab

Pad byte needed if @emph{n} is odd
and chunk alignment is used

@end multitable


The ID is a 4-byte string which identifies the type of chunk.

The size field (a 32-bit value, encoded using big-endian byte order) gives the
size of the chunk data, not including the 8-byte header.

Usually an IFF-type file consists of one or more chunks.  The proposed usage of
the @ref{3484,,Chunk} class defined here is to instantiate an instance at the start
of each chunk and read from the instance until it reaches the end, after which a
new instance can be instantiated. At the end of the file, creating a new
instance will fail with an @ref{e74,,EOFError} exception.

@geindex Chunk (class in chunk)
@anchor{library/chunk chunk Chunk}@anchor{3484}
@deffn {Class} chunk.Chunk (file, align=True, bigendian=True, inclheader=False)

Class which represents a chunk.  The @emph{file} argument is expected to be a
file-like object.  An instance of this class is specifically allowed.  The
only method that is needed is @code{read()}.  If the methods
@ref{2231,,seek()} and @ref{2232,,tell()} are present and don’t
raise an exception, they are also used.
If these methods are present and raise an exception, they are expected to not
have altered the object.  If the optional argument @emph{align} is true, chunks
are assumed to be aligned on 2-byte boundaries.  If @emph{align} is false, no
alignment is assumed.  The default value is true.  If the optional argument
@emph{bigendian} is false, the chunk size is assumed to be in little-endian order.
This is needed for WAVE audio files. The default value is true.  If the
optional argument @emph{inclheader} is true, the size given in the chunk header
includes the size of the header.  The default value is false.

A @ref{3484,,Chunk} object supports the following methods:

@geindex getname() (chunk.Chunk method)
@anchor{library/chunk chunk Chunk getname}@anchor{3485}
@deffn {Method} getname ()

Returns the name (ID) of the chunk.  This is the first 4 bytes of the
chunk.
@end deffn

@geindex getsize() (chunk.Chunk method)
@anchor{library/chunk chunk Chunk getsize}@anchor{3486}
@deffn {Method} getsize ()

Returns the size of the chunk.
@end deffn

@geindex close() (chunk.Chunk method)
@anchor{library/chunk chunk Chunk close}@anchor{3487}
@deffn {Method} close ()

Close and skip to the end of the chunk.  This does not close the
underlying file.
@end deffn

The remaining methods will raise @ref{2a1,,OSError} if called after the
@ref{3487,,close()} method has been called.  Before Python 3.3, they used to
raise @ref{be8,,IOError}, now an alias of @ref{2a1,,OSError}.

@geindex isatty() (chunk.Chunk method)
@anchor{library/chunk chunk Chunk isatty}@anchor{3488}
@deffn {Method} isatty ()

Returns @code{False}.
@end deffn

@geindex seek() (chunk.Chunk method)
@anchor{library/chunk chunk Chunk seek}@anchor{3489}
@deffn {Method} seek (pos, whence=0)

Set the chunk’s current position.  The @emph{whence} argument is optional and
defaults to @code{0} (absolute file positioning); other values are @code{1}
(seek relative to the current position) and @code{2} (seek relative to the
file’s end).  There is no return value. If the underlying file does not
allow seek, only forward seeks are allowed.
@end deffn

@geindex tell() (chunk.Chunk method)
@anchor{library/chunk chunk Chunk tell}@anchor{348a}
@deffn {Method} tell ()

Return the current position into the chunk.
@end deffn

@geindex read() (chunk.Chunk method)
@anchor{library/chunk chunk Chunk read}@anchor{348b}
@deffn {Method} read (size=- 1)

Read at most @emph{size} bytes from the chunk (less if the read hits the end of
the chunk before obtaining @emph{size} bytes).  If the @emph{size} argument is
negative or omitted, read all data until the end of the chunk.  An empty
bytes object is returned when the end of the chunk is encountered
immediately.
@end deffn

@geindex skip() (chunk.Chunk method)
@anchor{library/chunk chunk Chunk skip}@anchor{348c}
@deffn {Method} skip ()

Skip to the end of the chunk.  All further calls to @ref{348b,,read()} for the
chunk will return @code{b'@w{'}}.  If you are not interested in the contents of
the chunk, this method should be called so that the file points to the
start of the next chunk.
@end deffn
@end deffn

@node colorsys — Conversions between color systems,imghdr — Determine the type of an image,chunk — Read IFF chunked data,Multimedia Services
@anchor{library/colorsys doc}@anchor{348d}@anchor{library/colorsys colorsys-conversions-between-color-systems}@anchor{348e}@anchor{library/colorsys module-colorsys}@anchor{1e}
@subsection @code{colorsys} — Conversions between color systems


@geindex module; colorsys

@strong{Source code:} Lib/colorsys.py@footnote{https://github.com/python/cpython/tree/main/Lib/colorsys.py}


__________________________________________________________________


The @ref{1e,,colorsys} module defines bidirectional conversions of color values
between colors expressed in the RGB (Red Green Blue) color space used in
computer monitors and three other coordinate systems: YIQ, HLS (Hue Lightness
Saturation) and HSV (Hue Saturation Value).  Coordinates in all of these color
spaces are floating point values.  In the YIQ space, the Y coordinate is between
0 and 1, but the I and Q coordinates can be positive or negative.  In all other
spaces, the coordinates are all between 0 and 1.


@subsubheading See also


More information about color spaces can be found at
@indicateurl{https://poynton.ca/ColorFAQ.html} and
@indicateurl{https://www.cambridgeincolour.com/tutorials/color-spaces.htm}.


The @ref{1e,,colorsys} module defines the following functions:

@geindex rgb_to_yiq() (in module colorsys)
@anchor{library/colorsys colorsys rgb_to_yiq}@anchor{348f}
@deffn {Function} colorsys.rgb_to_yiq (r, g, b)

Convert the color from RGB coordinates to YIQ coordinates.
@end deffn

@geindex yiq_to_rgb() (in module colorsys)
@anchor{library/colorsys colorsys yiq_to_rgb}@anchor{3490}
@deffn {Function} colorsys.yiq_to_rgb (y, i, q)

Convert the color from YIQ coordinates to RGB coordinates.
@end deffn

@geindex rgb_to_hls() (in module colorsys)
@anchor{library/colorsys colorsys rgb_to_hls}@anchor{3491}
@deffn {Function} colorsys.rgb_to_hls (r, g, b)

Convert the color from RGB coordinates to HLS coordinates.
@end deffn

@geindex hls_to_rgb() (in module colorsys)
@anchor{library/colorsys colorsys hls_to_rgb}@anchor{3492}
@deffn {Function} colorsys.hls_to_rgb (h, l, s)

Convert the color from HLS coordinates to RGB coordinates.
@end deffn

@geindex rgb_to_hsv() (in module colorsys)
@anchor{library/colorsys colorsys rgb_to_hsv}@anchor{3493}
@deffn {Function} colorsys.rgb_to_hsv (r, g, b)

Convert the color from RGB coordinates to HSV coordinates.
@end deffn

@geindex hsv_to_rgb() (in module colorsys)
@anchor{library/colorsys colorsys hsv_to_rgb}@anchor{3494}
@deffn {Function} colorsys.hsv_to_rgb (h, s, v)

Convert the color from HSV coordinates to RGB coordinates.
@end deffn

Example:

@example
>>> import colorsys
>>> colorsys.rgb_to_hsv(0.2, 0.4, 0.4)
(0.5, 0.5, 0.4)
>>> colorsys.hsv_to_rgb(0.5, 0.5, 0.4)
(0.2, 0.4, 0.4)
@end example

@node imghdr — Determine the type of an image,sndhdr — Determine type of sound file,colorsys — Conversions between color systems,Multimedia Services
@anchor{library/imghdr doc}@anchor{3495}@anchor{library/imghdr imghdr-determine-the-type-of-an-image}@anchor{3496}@anchor{library/imghdr module-imghdr}@anchor{94}
@subsection @code{imghdr} — Determine the type of an image


@geindex module; imghdr

@strong{Source code:} Lib/imghdr.py@footnote{https://github.com/python/cpython/tree/main/Lib/imghdr.py}


__________________________________________________________________


The @ref{94,,imghdr} module determines the type of image contained in a file or
byte stream.

The @ref{94,,imghdr} module defines the following function:

@geindex what() (in module imghdr)
@anchor{library/imghdr imghdr what}@anchor{96f}
@deffn {Function} imghdr.what (file, h=None)

Tests the image data contained in the file named by @emph{file}, and returns a
string describing the image type.  If optional @emph{h} is provided, the @emph{file}
argument is ignored and @emph{h} is assumed to contain the byte stream to test.


Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

The following image types are recognized, as listed below with the return value
from @ref{96f,,what()}:


@multitable {xxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Value

@tab

Image format

@item

@code{'rgb'}

@tab

SGI ImgLib Files

@item

@code{'gif'}

@tab

GIF 87a and 89a Files

@item

@code{'pbm'}

@tab

Portable Bitmap Files

@item

@code{'pgm'}

@tab

Portable Graymap Files

@item

@code{'ppm'}

@tab

Portable Pixmap Files

@item

@code{'tiff'}

@tab

TIFF Files

@item

@code{'rast'}

@tab

Sun Raster Files

@item

@code{'xbm'}

@tab

X Bitmap Files

@item

@code{'jpeg'}

@tab

JPEG data in JFIF or Exif formats

@item

@code{'bmp'}

@tab

BMP files

@item

@code{'png'}

@tab

Portable Network Graphics

@item

@code{'webp'}

@tab

WebP files

@item

@code{'exr'}

@tab

OpenEXR Files

@end multitable



New in version 3.5: The @emph{exr} and @emph{webp} formats were added.


You can extend the list of file types @ref{94,,imghdr} can recognize by appending
to this variable:

@geindex tests (in module imghdr)
@anchor{library/imghdr imghdr tests}@anchor{3497}
@deffn {Data} imghdr.tests

A list of functions performing the individual tests.  Each function takes two
arguments: the byte-stream and an open file-like object. When @ref{96f,,what()} is
called with a byte-stream, the file-like object will be @code{None}.

The test function should return a string describing the image type if the test
succeeded, or @code{None} if it failed.
@end deffn

Example:

@example
>>> import imghdr
>>> imghdr.what('bass.gif')
'gif'
@end example

@node sndhdr — Determine type of sound file,ossaudiodev — Access to OSS-compatible audio devices,imghdr — Determine the type of an image,Multimedia Services
@anchor{library/sndhdr doc}@anchor{3498}@anchor{library/sndhdr module-sndhdr}@anchor{e9}@anchor{library/sndhdr sndhdr-determine-type-of-sound-file}@anchor{3499}
@subsection @code{sndhdr} — Determine type of sound file


@geindex module; sndhdr

@c Based on comments in the module source file.

@strong{Source code:} Lib/sndhdr.py@footnote{https://github.com/python/cpython/tree/main/Lib/sndhdr.py}

@geindex A-LAW
@geindex u-LAW


__________________________________________________________________


The @ref{e9,,sndhdr} provides utility functions which attempt to determine the type
of sound data which is in a file.  When these functions are able to determine
what type of sound data is stored in a file, they return a
@ref{3e4,,namedtuple()}, containing five attributes: (@code{filetype},
@code{framerate}, @code{nchannels}, @code{nframes}, @code{sampwidth}). The value for @emph{type}
indicates the data type and will be one of the strings @code{'aifc'}, @code{'aiff'},
@code{'au'}, @code{'hcom'}, @code{'sndr'}, @code{'sndt'}, @code{'voc'}, @code{'wav'}, @code{'8svx'},
@code{'sb'}, @code{'ub'}, or @code{'ul'}.  The @emph{sampling_rate} will be either the actual
value or @code{0} if unknown or difficult to decode.  Similarly, @emph{channels} will be
either the number of channels or @code{0} if it cannot be determined or if the
value is difficult to decode.  The value for @emph{frames} will be either the number
of frames or @code{-1}.  The last item in the tuple, @emph{bits_per_sample}, will either
be the sample size in bits or @code{'A'} for A-LAW or @code{'U'} for u-LAW.

@geindex what() (in module sndhdr)
@anchor{library/sndhdr sndhdr what}@anchor{9c6}
@deffn {Function} sndhdr.what (filename)

Determines the type of sound data stored in the file @emph{filename} using
@ref{9c7,,whathdr()}.  If it succeeds, returns a namedtuple as described above, otherwise
@code{None} is returned.


Changed in version 3.5: Result changed from a tuple to a namedtuple.

@end deffn

@geindex whathdr() (in module sndhdr)
@anchor{library/sndhdr sndhdr whathdr}@anchor{9c7}
@deffn {Function} sndhdr.whathdr (filename)

Determines the type of sound data stored in a file based on the file  header.
The name of the file is given by @emph{filename}.  This function returns a namedtuple as
described above on success, or @code{None}.


Changed in version 3.5: Result changed from a tuple to a namedtuple.

@end deffn

@node ossaudiodev — Access to OSS-compatible audio devices,,sndhdr — Determine type of sound file,Multimedia Services
@anchor{library/ossaudiodev doc}@anchor{349a}@anchor{library/ossaudiodev module-ossaudiodev}@anchor{c2}@anchor{library/ossaudiodev ossaudiodev-access-to-oss-compatible-audio-devices}@anchor{349b}
@subsection @code{ossaudiodev} — Access to OSS-compatible audio devices


@geindex module; ossaudiodev


__________________________________________________________________


This module allows you to access the OSS (Open Sound System) audio interface.
OSS is available for a wide range of open-source and commercial Unices, and is
the standard audio interface for Linux and recent versions of FreeBSD.

@c Things will get more complicated for future Linux versions, since
@c ALSA is in the standard kernel as of 2.5.x.  Presumably if you
@c use ALSA, you'll have to make sure its OSS compatibility layer
@c is active to use ossaudiodev, but you're going to need it for the vast
@c majority of Linux audio apps anyway.
@c 
@c Sounds like things are also complicated for other BSDs.  In response
@c to my python-dev query, Thomas Wouters said:
@c 
@c > Likewise, googling shows OpenBSD also uses OSS/Free -- the commercial
@c > OSS installation manual tells you to remove references to OSS/Free from the
@c > kernel :)
@c 
@c but Aleksander Piotrowsk actually has an OpenBSD box, and he quotes
@c from its <soundcard.h>:
@c >  * WARNING!  WARNING!
@c >  * This is an OSS (Linux) audio emulator.
@c >  * Use the Native NetBSD API for developing new code, and this
@c >  * only for compiling Linux programs.
@c 
@c There's also an ossaudio manpage on OpenBSD that explains things
@c further.  Presumably NetBSD and OpenBSD have a different standard
@c audio interface.  That's the great thing about standards, there are so
@c many to choose from ... ;-)
@c 
@c This probably all warrants a footnote or two, but I don't understand
@c things well enough right now to write it!   --GPW


Changed in version 3.3: Operations in this module now raise @ref{2a1,,OSError} where @ref{be8,,IOError}
was raised.



@subsubheading See also



@table @asis

@item Open Sound System Programmer's Guide@footnote{http://www.opensound.com/pguide/oss.pdf}

the official documentation for the OSS C API
@end table

The module defines a large number of constants supplied by the OSS device
driver; see @code{<sys/soundcard.h>} on either Linux or FreeBSD for a listing.


@ref{c2,,ossaudiodev} defines the following variables and functions:

@geindex OSSAudioError
@anchor{library/ossaudiodev ossaudiodev OSSAudioError}@anchor{349c}
@deffn {Exception} ossaudiodev.OSSAudioError

This exception is raised on certain errors.  The argument is a string describing
what went wrong.

(If @ref{c2,,ossaudiodev} receives an error from a system call such as
@code{open()}, @code{write()}, or @code{ioctl()}, it raises @ref{2a1,,OSError}.
Errors detected directly by @ref{c2,,ossaudiodev} result in @ref{349c,,OSSAudioError}.)

(For backwards compatibility, the exception class is also available as
@code{ossaudiodev.error}.)
@end deffn

@geindex open() (in module ossaudiodev)
@anchor{library/ossaudiodev ossaudiodev open}@anchor{349d}
@deffn {Function} ossaudiodev.open (mode)

@deffnx {Function} ossaudiodev.open (device, mode)

Open an audio device and return an OSS audio device object.  This object
supports many file-like methods, such as @code{read()}, @code{write()}, and
@code{fileno()} (although there are subtle differences between conventional Unix
read/write semantics and those of OSS audio devices).  It also supports a number
of audio-specific methods; see below for the complete list of methods.

@emph{device} is the audio device filename to use.  If it is not specified, this
module first looks in the environment variable 
@geindex AUDIODEV
@geindex environment variable; AUDIODEV
@code{AUDIODEV} for a device
to use.  If not found, it falls back to @code{/dev/dsp}.

@emph{mode} is one of @code{'r'} for read-only (record) access, @code{'w'} for
write-only (playback) access and @code{'rw'} for both. Since many sound cards
only allow one process to have the recorder or player open at a time, it is a
good idea to open the device only for the activity needed.  Further, some
sound cards are half-duplex: they can be opened for reading or writing, but
not both at once.

Note the unusual calling syntax: the @emph{first} argument is optional, and the
second is required.  This is a historical artifact for compatibility with the
older @code{linuxaudiodev} module which @ref{c2,,ossaudiodev} supersedes.

@c XXX it might also be motivated
@c by my unfounded-but-still-possibly-true belief that the default
@c audio device varies unpredictably across operating systems.  -GW
@end deffn

@geindex openmixer() (in module ossaudiodev)
@anchor{library/ossaudiodev ossaudiodev openmixer}@anchor{349e}
@deffn {Function} ossaudiodev.openmixer ([device])

Open a mixer device and return an OSS mixer device object.   @emph{device} is the
mixer device filename to use.  If it is not specified, this module first looks
in the environment variable 
@geindex MIXERDEV
@geindex environment variable; MIXERDEV
@code{MIXERDEV} for a device to use.  If not
found, it falls back to @code{/dev/mixer}.
@end deffn

@menu
* Audio Device Objects:: 
* Mixer Device Objects:: 

@end menu

@node Audio Device Objects,Mixer Device Objects,,ossaudiodev — Access to OSS-compatible audio devices
@anchor{library/ossaudiodev audio-device-objects}@anchor{349f}@anchor{library/ossaudiodev ossaudio-device-objects}@anchor{34a0}
@subsubsection Audio Device Objects


Before you can write to or read from an audio device, you must call three
methods in the correct order:


@enumerate 

@item 
@code{setfmt()} to set the output format

@item 
@code{channels()} to set the number of channels

@item 
@code{speed()} to set the sample rate
@end enumerate

Alternately, you can use the @code{setparameters()} method to set all three audio
parameters at once.  This is more convenient, but may not be as flexible in all
cases.

The audio device objects returned by @ref{349d,,open()} define the following methods
and (read-only) attributes:

@geindex close() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device close}@anchor{34a1}
@deffn {Method} oss_audio_device.close ()

Explicitly close the audio device.  When you are done writing to or reading from
an audio device, you should explicitly close it.  A closed device cannot be used
again.
@end deffn

@geindex fileno() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device fileno}@anchor{34a2}
@deffn {Method} oss_audio_device.fileno ()

Return the file descriptor associated with the device.
@end deffn

@geindex read() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device read}@anchor{34a3}
@deffn {Method} oss_audio_device.read (size)

Read @emph{size} bytes from the audio input and return them as a Python string.
Unlike most Unix device drivers, OSS audio devices in blocking mode (the
default) will block @ref{34a3,,read()} until the entire requested amount of data is
available.
@end deffn

@geindex write() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device write}@anchor{34a4}
@deffn {Method} oss_audio_device.write (data)

Write a @ref{87e,,bytes-like object} @emph{data} to the audio device and return the
number of bytes written.  If the audio device is in blocking mode (the
default), the entire data is always written (again, this is different from
usual Unix device semantics).  If the device is in non-blocking mode, some
data may not be written—see @ref{34a5,,writeall()}.


Changed in version 3.5: Writable @ref{87e,,bytes-like object} is now accepted.

@end deffn

@geindex writeall() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device writeall}@anchor{34a5}
@deffn {Method} oss_audio_device.writeall (data)

Write a @ref{87e,,bytes-like object} @emph{data} to the audio device: waits until
the audio device is able to accept data, writes as much data as it will
accept, and repeats until @emph{data} has been completely written. If the device
is in blocking mode (the default), this has the same effect as
@ref{34a4,,write()}; @ref{34a5,,writeall()} is only useful in non-blocking mode.  Has
no return value, since the amount of data written is always equal to the
amount of data supplied.


Changed in version 3.5: Writable @ref{87e,,bytes-like object} is now accepted.

@end deffn


Changed in version 3.2: Audio device objects also support the context management protocol, i.e. they can
be used in a @ref{168,,with} statement.


The following methods each map to exactly one @code{ioctl()} system call.  The
correspondence is obvious: for example, @code{setfmt()} corresponds to the
@code{SNDCTL_DSP_SETFMT} ioctl, and @code{sync()} to @code{SNDCTL_DSP_SYNC} (this can
be useful when consulting the OSS documentation).  If the underlying
@code{ioctl()} fails, they all raise @ref{2a1,,OSError}.

@geindex nonblock() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device nonblock}@anchor{34a6}
@deffn {Method} oss_audio_device.nonblock ()

Put the device into non-blocking mode.  Once in non-blocking mode, there is no
way to return it to blocking mode.
@end deffn

@geindex getfmts() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device getfmts}@anchor{34a7}
@deffn {Method} oss_audio_device.getfmts ()

Return a bitmask of the audio output formats supported by the soundcard.  Some
of the formats supported by OSS are:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Format

@tab

Description

@item

@code{AFMT_MU_LAW}

@tab

a logarithmic encoding (used by Sun @code{.au}
files and @code{/dev/audio})

@item

@code{AFMT_A_LAW}

@tab

a logarithmic encoding

@item

@code{AFMT_IMA_ADPCM}

@tab

a 4:1 compressed format defined by the
Interactive Multimedia Association

@item

@code{AFMT_U8}

@tab

Unsigned, 8-bit audio

@item

@code{AFMT_S16_LE}

@tab

Signed, 16-bit audio, little-endian byte
order (as used by Intel processors)

@item

@code{AFMT_S16_BE}

@tab

Signed, 16-bit audio, big-endian byte order
(as used by 68k, PowerPC, Sparc)

@item

@code{AFMT_S8}

@tab

Signed, 8 bit audio

@item

@code{AFMT_U16_LE}

@tab

Unsigned, 16-bit little-endian audio

@item

@code{AFMT_U16_BE}

@tab

Unsigned, 16-bit big-endian audio

@end multitable


Consult the OSS documentation for a full list of audio formats, and note that
most devices support only a subset of these formats.  Some older devices only
support @code{AFMT_U8}; the most common format used today is
@code{AFMT_S16_LE}.
@end deffn

@geindex setfmt() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device setfmt}@anchor{34a8}
@deffn {Method} oss_audio_device.setfmt (format)

Try to set the current audio format to @emph{format}—see @ref{34a7,,getfmts()} for a
list.  Returns the audio format that the device was set to, which may not be the
requested format.  May also be used to return the current audio format—do this
by passing an “audio format” of @code{AFMT_QUERY}.
@end deffn

@geindex channels() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device channels}@anchor{34a9}
@deffn {Method} oss_audio_device.channels (nchannels)

Set the number of output channels to @emph{nchannels}.  A value of 1 indicates
monophonic sound, 2 stereophonic.  Some devices may have more than 2 channels,
and some high-end devices may not support mono. Returns the number of channels
the device was set to.
@end deffn

@geindex speed() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device speed}@anchor{34aa}
@deffn {Method} oss_audio_device.speed (samplerate)

Try to set the audio sampling rate to @emph{samplerate} samples per second.  Returns
the rate actually set.  Most sound devices don’t support arbitrary sampling
rates.  Common rates are:


@multitable {xxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Rate

@tab

Description

@item

8000

@tab

default rate for @code{/dev/audio}

@item

11025

@tab

speech recording

@item

22050

@tab

@item

44100

@tab

CD quality audio (at 16 bits/sample and 2
channels)

@item

96000

@tab

DVD quality audio (at 24 bits/sample)

@end multitable

@end deffn

@geindex sync() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device sync}@anchor{34ab}
@deffn {Method} oss_audio_device.sync ()

Wait until the sound device has played every byte in its buffer.  (This happens
implicitly when the device is closed.)  The OSS documentation recommends closing
and re-opening the device rather than using @ref{34ab,,sync()}.
@end deffn

@geindex reset() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device reset}@anchor{34ac}
@deffn {Method} oss_audio_device.reset ()

Immediately stop playing or recording and return the device to a state where it
can accept commands.  The OSS documentation recommends closing and re-opening
the device after calling @ref{34ac,,reset()}.
@end deffn

@geindex post() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device post}@anchor{34ad}
@deffn {Method} oss_audio_device.post ()

Tell the driver that there is likely to be a pause in the output, making it
possible for the device to handle the pause more intelligently.  You might use
this after playing a spot sound effect, before waiting for user input, or before
doing disk I/O.
@end deffn

The following convenience methods combine several ioctls, or one ioctl and some
simple calculations.

@geindex setparameters() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device setparameters}@anchor{34ae}
@deffn {Method} oss_audio_device.setparameters (format, nchannels, samplerate[, strict=False])

Set the key audio sampling parameters—sample format, number of channels, and
sampling rate—in one method call.  @emph{format},  @emph{nchannels}, and @emph{samplerate}
should be as specified in the @ref{34a8,,setfmt()}, @ref{34a9,,channels()}, and
@ref{34aa,,speed()}  methods.  If @emph{strict} is true, @ref{34ae,,setparameters()} checks to
see if each parameter was actually set to the requested value, and raises
@ref{349c,,OSSAudioError} if not.  Returns a tuple (@emph{format}, @emph{nchannels},
@emph{samplerate}) indicating the parameter values that were actually set by the
device driver (i.e., the same as the return values of @ref{34a8,,setfmt()},
@ref{34a9,,channels()}, and @ref{34aa,,speed()}).

For example,

@example
(fmt, channels, rate) = dsp.setparameters(fmt, channels, rate)
@end example

is equivalent to

@example
fmt = dsp.setfmt(fmt)
channels = dsp.channels(channels)
rate = dsp.rate(rate)
@end example
@end deffn

@geindex bufsize() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device bufsize}@anchor{34af}
@deffn {Method} oss_audio_device.bufsize ()

Returns the size of the hardware buffer, in samples.
@end deffn

@geindex obufcount() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device obufcount}@anchor{34b0}
@deffn {Method} oss_audio_device.obufcount ()

Returns the number of samples that are in the hardware buffer yet to be played.
@end deffn

@geindex obuffree() (ossaudiodev.oss_audio_device method)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device obuffree}@anchor{34b1}
@deffn {Method} oss_audio_device.obuffree ()

Returns the number of samples that could be queued into the hardware buffer to
be played without blocking.
@end deffn

Audio device objects also support several read-only attributes:

@geindex closed (ossaudiodev.oss_audio_device attribute)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device closed}@anchor{34b2}
@deffn {Attribute} oss_audio_device.closed

Boolean indicating whether the device has been closed.
@end deffn

@geindex name (ossaudiodev.oss_audio_device attribute)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device name}@anchor{34b3}
@deffn {Attribute} oss_audio_device.name

String containing the name of the device file.
@end deffn

@geindex mode (ossaudiodev.oss_audio_device attribute)
@anchor{library/ossaudiodev ossaudiodev oss_audio_device mode}@anchor{34b4}
@deffn {Attribute} oss_audio_device.mode

The I/O mode for the file, either @code{"r"}, @code{"rw"}, or @code{"w"}.
@end deffn

@node Mixer Device Objects,,Audio Device Objects,ossaudiodev — Access to OSS-compatible audio devices
@anchor{library/ossaudiodev id1}@anchor{34b5}@anchor{library/ossaudiodev mixer-device-objects}@anchor{34b6}
@subsubsection Mixer Device Objects


The mixer object provides two file-like methods:

@geindex close() (ossaudiodev.oss_mixer_device method)
@anchor{library/ossaudiodev ossaudiodev oss_mixer_device close}@anchor{34b7}
@deffn {Method} oss_mixer_device.close ()

This method closes the open mixer device file.  Any further attempts to use the
mixer after this file is closed will raise an @ref{2a1,,OSError}.
@end deffn

@geindex fileno() (ossaudiodev.oss_mixer_device method)
@anchor{library/ossaudiodev ossaudiodev oss_mixer_device fileno}@anchor{34b8}
@deffn {Method} oss_mixer_device.fileno ()

Returns the file handle number of the open mixer device file.
@end deffn


Changed in version 3.2: Mixer objects also support the context management protocol.


The remaining methods are specific to audio mixing:

@geindex controls() (ossaudiodev.oss_mixer_device method)
@anchor{library/ossaudiodev ossaudiodev oss_mixer_device controls}@anchor{34b9}
@deffn {Method} oss_mixer_device.controls ()

This method returns a bitmask specifying the available mixer controls (“Control”
being a specific mixable “channel”, such as @code{SOUND_MIXER_PCM} or
@code{SOUND_MIXER_SYNTH}).  This bitmask indicates a subset of all available
mixer controls—the @code{SOUND_MIXER_*} constants defined at module level.
To determine if, for example, the current mixer object supports a PCM mixer, use
the following Python code:

@example
mixer=ossaudiodev.openmixer()
if mixer.controls() & (1 << ossaudiodev.SOUND_MIXER_PCM):
    # PCM is supported
    ... code ...
@end example

For most purposes, the @code{SOUND_MIXER_VOLUME} (master volume) and
@code{SOUND_MIXER_PCM} controls should suffice—but code that uses the mixer
should be flexible when it comes to choosing mixer controls.  On the Gravis
Ultrasound, for example, @code{SOUND_MIXER_VOLUME} does not exist.
@end deffn

@geindex stereocontrols() (ossaudiodev.oss_mixer_device method)
@anchor{library/ossaudiodev ossaudiodev oss_mixer_device stereocontrols}@anchor{34ba}
@deffn {Method} oss_mixer_device.stereocontrols ()

Returns a bitmask indicating stereo mixer controls.  If a bit is set, the
corresponding control is stereo; if it is unset, the control is either
monophonic or not supported by the mixer (use in combination with
@ref{34b9,,controls()} to determine which).

See the code example for the @ref{34b9,,controls()} function for an example of getting
data from a bitmask.
@end deffn

@geindex reccontrols() (ossaudiodev.oss_mixer_device method)
@anchor{library/ossaudiodev ossaudiodev oss_mixer_device reccontrols}@anchor{34bb}
@deffn {Method} oss_mixer_device.reccontrols ()

Returns a bitmask specifying the mixer controls that may be used to record.  See
the code example for @ref{34b9,,controls()} for an example of reading from a bitmask.
@end deffn

@geindex get() (ossaudiodev.oss_mixer_device method)
@anchor{library/ossaudiodev ossaudiodev oss_mixer_device get}@anchor{34bc}
@deffn {Method} oss_mixer_device.get (control)

Returns the volume of a given mixer control.  The returned volume is a 2-tuple
@code{(left_volume,right_volume)}.  Volumes are specified as numbers from 0
(silent) to 100 (full volume).  If the control is monophonic, a 2-tuple is still
returned, but both volumes are the same.

Raises @ref{349c,,OSSAudioError} if an invalid control is specified, or
@ref{2a1,,OSError} if an unsupported control is specified.
@end deffn

@geindex set() (ossaudiodev.oss_mixer_device method)
@anchor{library/ossaudiodev ossaudiodev oss_mixer_device set}@anchor{34bd}
@deffn {Method} oss_mixer_device.set (control, (left, right))

Sets the volume for a given mixer control to @code{(left,right)}. @code{left} and
@code{right} must be ints and between 0 (silent) and 100 (full volume).  On
success, the new volume is returned as a 2-tuple. Note that this may not be
exactly the same as the volume specified, because of the limited resolution of
some soundcard’s mixers.

Raises @ref{349c,,OSSAudioError} if an invalid mixer control was specified, or if the
specified volumes were out-of-range.
@end deffn

@geindex get_recsrc() (ossaudiodev.oss_mixer_device method)
@anchor{library/ossaudiodev ossaudiodev oss_mixer_device get_recsrc}@anchor{34be}
@deffn {Method} oss_mixer_device.get_recsrc ()

This method returns a bitmask indicating which control(s) are currently being
used as a recording source.
@end deffn

@geindex set_recsrc() (ossaudiodev.oss_mixer_device method)
@anchor{library/ossaudiodev ossaudiodev oss_mixer_device set_recsrc}@anchor{34bf}
@deffn {Method} oss_mixer_device.set_recsrc (bitmask)

Call this function to specify a recording source.  Returns a bitmask indicating
the new recording source (or sources) if successful; raises @ref{2a1,,OSError} if an
invalid source was specified.  To set the current recording source to the
microphone input:

@example
mixer.setrecsrc (1 << ossaudiodev.SOUND_MIXER_MIC)
@end example
@end deffn

@node Internationalization,Program Frameworks,Multimedia Services,The Python Standard Library
@anchor{library/i18n doc}@anchor{34c0}@anchor{library/i18n i18n}@anchor{34c1}@anchor{library/i18n internationalization}@anchor{34c2}
@section Internationalization


The modules described in this chapter help you write software that is
independent of language and locale by providing mechanisms for selecting a
language to be used in  program messages or by tailoring output to match local
conventions.

The list of modules described in this chapter is:

@menu
* gettext — Multilingual internationalization services:: 
* locale — Internationalization services:: 

@end menu

@node gettext — Multilingual internationalization services,locale — Internationalization services,,Internationalization
@anchor{library/gettext doc}@anchor{34c3}@anchor{library/gettext gettext-multilingual-internationalization-services}@anchor{34c4}@anchor{library/gettext module-gettext}@anchor{83}
@subsection @code{gettext} — Multilingual internationalization services


@geindex module; gettext

@strong{Source code:} Lib/gettext.py@footnote{https://github.com/python/cpython/tree/main/Lib/gettext.py}


__________________________________________________________________


The @ref{83,,gettext} module provides internationalization (I18N) and localization
(L10N) services for your Python modules and applications. It supports both the
GNU @code{gettext} message catalog API and a higher level, class-based API that may
be more appropriate for Python files.  The interface described below allows you
to write your module and application messages in one natural language, and
provide a catalog of translated messages for running under different natural
languages.

Some hints on localizing your Python modules and applications are also given.

@menu
* GNU gettext API:: 
* Class-based API:: 
* Internationalizing your programs and modules:: 
* Acknowledgements: Acknowledgements<9>. 

@end menu

@node GNU gettext API,Class-based API,,gettext — Multilingual internationalization services
@anchor{library/gettext gnu-gettext-api}@anchor{34c5}
@subsubsection GNU @code{gettext} API


The @ref{83,,gettext} module defines the following API, which is very similar to
the GNU @code{gettext} API.  If you use this API you will affect the
translation of your entire application globally.  Often this is what you want if
your application is monolingual, with the choice of language dependent on the
locale of your user.  If you are localizing a Python module, or if your
application needs to switch languages on the fly, you probably want to use the
class-based API instead.

@geindex bindtextdomain() (in module gettext)
@anchor{library/gettext gettext bindtextdomain}@anchor{34c6}
@deffn {Function} gettext.bindtextdomain (domain, localedir=None)

Bind the @emph{domain} to the locale directory @emph{localedir}.  More concretely,
@ref{83,,gettext} will look for binary @code{.mo} files for the given domain using
the path (on Unix): @code{@emph{localedir}/@emph{language}/LC_MESSAGES/@emph{domain}.mo}, where
@emph{language} is searched for in the environment variables 
@geindex LANGUAGE
@geindex environment variable; LANGUAGE
@code{LANGUAGE},
@geindex LC_ALL
@geindex environment variable; LC_ALL
@code{LC_ALL}, 
@geindex LC_MESSAGES
@geindex environment variable; LC_MESSAGES
@code{LC_MESSAGES}, and 
@geindex LANG
@geindex environment variable; LANG
@code{LANG} respectively.

If @emph{localedir} is omitted or @code{None}, then the current binding for @emph{domain} is
returned. @footnote{@w{(1)} 
The default locale directory is system dependent; for example, on RedHat Linux
it is @code{/usr/share/locale}, but on Solaris it is @code{/usr/lib/locale}.
The @ref{83,,gettext} module does not try to support these system dependent
defaults; instead its default is @code{@emph{sys.base_prefix}/share/locale} (see
@ref{34c7,,sys.base_prefix}). For this reason, it is always best to call
@ref{34c6,,bindtextdomain()} with an explicit absolute path at the start of your
application.
}
@end deffn

@geindex textdomain() (in module gettext)
@anchor{library/gettext gettext textdomain}@anchor{34c8}
@deffn {Function} gettext.textdomain (domain=None)

Change or query the current global domain.  If @emph{domain} is @code{None}, then the
current global domain is returned, otherwise the global domain is set to
@emph{domain}, which is returned.
@end deffn

@geindex _ (underscore); gettext

@geindex gettext() (in module gettext)
@anchor{library/gettext gettext gettext}@anchor{fc6}
@deffn {Function} gettext.gettext (message)

Return the localized translation of @emph{message}, based on the current global
domain, language, and locale directory.  This function is usually aliased as
@code{_()} in the local namespace (see examples below).
@end deffn

@geindex dgettext() (in module gettext)
@anchor{library/gettext gettext dgettext}@anchor{34c9}
@deffn {Function} gettext.dgettext (domain, message)

Like @ref{fc6,,gettext()}, but look the message up in the specified @emph{domain}.
@end deffn

@geindex ngettext() (in module gettext)
@anchor{library/gettext gettext ngettext}@anchor{34ca}
@deffn {Function} gettext.ngettext (singular, plural, n)

Like @ref{fc6,,gettext()}, but consider plural forms. If a translation is found,
apply the plural formula to @emph{n}, and return the resulting message (some
languages have more than two plural forms). If no translation is found, return
@emph{singular} if @emph{n} is 1; return @emph{plural} otherwise.

The Plural formula is taken from the catalog header. It is a C or Python
expression that has a free variable @emph{n}; the expression evaluates to the index
of the plural in the catalog. See
the GNU gettext documentation@footnote{https://www.gnu.org/software/gettext/manual/gettext.html}
for the precise syntax to be used in @code{.po} files and the
formulas for a variety of languages.
@end deffn

@geindex dngettext() (in module gettext)
@anchor{library/gettext gettext dngettext}@anchor{34cb}
@deffn {Function} gettext.dngettext (domain, singular, plural, n)

Like @ref{34ca,,ngettext()}, but look the message up in the specified @emph{domain}.
@end deffn

@geindex pgettext() (in module gettext)
@anchor{library/gettext gettext pgettext}@anchor{4fa}
@deffn {Function} gettext.pgettext (context, message)
@end deffn

@geindex dpgettext() (in module gettext)
@anchor{library/gettext gettext dpgettext}@anchor{34cc}
@deffn {Function} gettext.dpgettext (domain, context, message)
@end deffn

@geindex npgettext() (in module gettext)
@anchor{library/gettext gettext npgettext}@anchor{34cd}
@deffn {Function} gettext.npgettext (context, singular, plural, n)
@end deffn

@geindex dnpgettext() (in module gettext)
@anchor{library/gettext gettext dnpgettext}@anchor{34ce}
@deffn {Function} gettext.dnpgettext (domain, context, singular, plural, n)

Similar to the corresponding functions without the @code{p} in the prefix (that
is, @ref{83,,gettext()}, @ref{34c9,,dgettext()}, @ref{34ca,,ngettext()}, @ref{34cb,,dngettext()}),
but the translation is restricted to the given message @emph{context}.


New in version 3.8.

@end deffn

Note that GNU @code{gettext} also defines a @code{dcgettext()} method, but
this was deemed not useful and so it is currently unimplemented.

Here’s an example of typical usage for this API:

@example
import gettext
gettext.bindtextdomain('myapplication', '/path/to/my/language/directory')
gettext.textdomain('myapplication')
_ = gettext.gettext
# ...
print(_('This is a translatable string.'))
@end example

@node Class-based API,Internationalizing your programs and modules,GNU gettext API,gettext — Multilingual internationalization services
@anchor{library/gettext class-based-api}@anchor{34cf}
@subsubsection Class-based API


The class-based API of the @ref{83,,gettext} module gives you more flexibility and
greater convenience than the GNU @code{gettext} API.  It is the recommended
way of localizing your Python applications and modules.  @code{gettext} defines
a @ref{34d0,,GNUTranslations} class which implements the parsing of GNU @code{.mo} format
files, and has methods for returning strings. Instances of this class can also
install themselves in the built-in namespace as the function @code{_()}.

@geindex find() (in module gettext)
@anchor{library/gettext gettext find}@anchor{34d1}
@deffn {Function} gettext.find (domain, localedir=None, languages=None, all=False)

This function implements the standard @code{.mo} file search algorithm.  It
takes a @emph{domain}, identical to what @ref{34c8,,textdomain()} takes.  Optional
@emph{localedir} is as in @ref{34c6,,bindtextdomain()}. Optional @emph{languages} is a list of
strings, where each string is a language code.

If @emph{localedir} is not given, then the default system locale directory is used.
@footnote{@w{(2)} 
See the footnote for @ref{34c6,,bindtextdomain()} above.
}  If @emph{languages} is not given, then the following environment variables are
searched: 
@geindex LANGUAGE
@geindex environment variable; LANGUAGE
@code{LANGUAGE}, 
@geindex LC_ALL
@geindex environment variable; LC_ALL
@code{LC_ALL}, 
@geindex LC_MESSAGES
@geindex environment variable; LC_MESSAGES
@code{LC_MESSAGES}, and
@geindex LANG
@geindex environment variable; LANG
@code{LANG}.  The first one returning a non-empty value is used for the
@emph{languages} variable. The environment variables should contain a colon separated
list of languages, which will be split on the colon to produce the expected list
of language code strings.

@ref{34d1,,find()} then expands and normalizes the languages, and then iterates
through them, searching for an existing file built of these components:

@code{@emph{localedir}/@emph{language}/LC_MESSAGES/@emph{domain}.mo}

The first such file name that exists is returned by @ref{34d1,,find()}. If no such
file is found, then @code{None} is returned. If @emph{all} is given, it returns a list
of all file names, in the order in which they appear in the languages list or
the environment variables.
@end deffn

@geindex translation() (in module gettext)
@anchor{library/gettext gettext translation}@anchor{1bd}
@deffn {Function} gettext.translation (domain, localedir=None, languages=None, class_=None, fallback=False)

Return a @code{*Translations} instance based on the @emph{domain}, @emph{localedir},
and @emph{languages}, which are first passed to @ref{34d1,,find()} to get a list of the
associated @code{.mo} file paths.  Instances with identical @code{.mo} file
names are cached.  The actual class instantiated is @emph{class_} if
provided, otherwise @ref{34d0,,GNUTranslations}.  The class’s constructor must
take a single @ref{d68,,file object} argument.  If provided, @emph{codeset} will change
the charset used to encode translated strings in the
@code{lgettext()} and @code{lngettext()}
methods.

If multiple files are found, later files are used as fallbacks for earlier ones.
To allow setting the fallback, @ref{69a,,copy.copy()} is used to clone each
translation object from the cache; the actual instance data is still shared with
the cache.

If no @code{.mo} file is found, this function raises @ref{2a1,,OSError} if
@emph{fallback} is false (which is the default), and returns a
@ref{34d2,,NullTranslations} instance if @emph{fallback} is true.


Changed in version 3.3: @ref{be8,,IOError} used to be raised instead of @ref{2a1,,OSError}.



Changed in version 3.11: @emph{codeset} parameter is removed.

@end deffn

@geindex install() (in module gettext)
@anchor{library/gettext gettext install}@anchor{1be}
@deffn {Function} gettext.install (domain, localedir=None, *, names=None)

This installs the function @code{_()} in Python’s builtins namespace, based on
@emph{domain} and @emph{localedir} which are passed to the function @ref{1bd,,translation()}.

For the @emph{names} parameter, please see the description of the translation
object’s @ref{34d3,,install()} method.

As seen below, you usually mark the strings in your application that are
candidates for translation, by wrapping them in a call to the @code{_()}
function, like this:

@example
print(_('This string will be translated.'))
@end example

For convenience, you want the @code{_()} function to be installed in Python’s
builtins namespace, so it is easily accessible in all modules of your
application.


Changed in version 3.11: @emph{names} is now a keyword-only parameter.

@end deffn

@menu
* The NullTranslations class:: 
* The GNUTranslations class:: 
* Solaris message catalog support:: 
* The Catalog constructor:: 

@end menu

@node The NullTranslations class,The GNUTranslations class,,Class-based API
@anchor{library/gettext the-nulltranslations-class}@anchor{34d4}
@subsubsection The @code{NullTranslations} class


Translation classes are what actually implement the translation of original
source file message strings to translated message strings. The base class used
by all translation classes is @ref{34d2,,NullTranslations}; this provides the basic
interface you can use to write your own specialized translation classes.  Here
are the methods of @code{NullTranslations}:

@geindex NullTranslations (class in gettext)
@anchor{library/gettext gettext NullTranslations}@anchor{34d2}
@deffn {Class} gettext.NullTranslations (fp=None)

Takes an optional @ref{d68,,file object} @emph{fp}, which is ignored by the base class.
Initializes “protected” instance variables @emph{_info} and @emph{_charset} which are set
by derived classes, as well as @emph{_fallback}, which is set through
@ref{34d5,,add_fallback()}.  It then calls @code{self._parse(fp)} if @emph{fp} is not
@code{None}.

@geindex _parse() (gettext.NullTranslations method)
@anchor{library/gettext gettext NullTranslations _parse}@anchor{34d6}
@deffn {Method} _parse (fp)

No-op in the base class, this method takes file object @emph{fp}, and reads
the data from the file, initializing its message catalog.  If you have an
unsupported message catalog file format, you should override this method
to parse your format.
@end deffn

@geindex add_fallback() (gettext.NullTranslations method)
@anchor{library/gettext gettext NullTranslations add_fallback}@anchor{34d5}
@deffn {Method} add_fallback (fallback)

Add @emph{fallback} as the fallback object for the current translation object.
A translation object should consult the fallback if it cannot provide a
translation for a given message.
@end deffn

@geindex gettext() (gettext.NullTranslations method)
@anchor{library/gettext gettext NullTranslations gettext}@anchor{34d7}
@deffn {Method} gettext (message)

If a fallback has been set, forward @code{gettext()} to the fallback.
Otherwise, return @emph{message}.  Overridden in derived classes.
@end deffn

@geindex ngettext() (gettext.NullTranslations method)
@anchor{library/gettext gettext NullTranslations ngettext}@anchor{34d8}
@deffn {Method} ngettext (singular, plural, n)

If a fallback has been set, forward @code{ngettext()} to the fallback.
Otherwise, return @emph{singular} if @emph{n} is 1; return @emph{plural} otherwise.
Overridden in derived classes.
@end deffn

@geindex pgettext() (gettext.NullTranslations method)
@anchor{library/gettext gettext NullTranslations pgettext}@anchor{34d9}
@deffn {Method} pgettext (context, message)

If a fallback has been set, forward @ref{4fa,,pgettext()} to the fallback.
Otherwise, return the translated message.  Overridden in derived classes.


New in version 3.8.

@end deffn

@geindex npgettext() (gettext.NullTranslations method)
@anchor{library/gettext gettext NullTranslations npgettext}@anchor{34da}
@deffn {Method} npgettext (context, singular, plural, n)

If a fallback has been set, forward @ref{34cd,,npgettext()} to the fallback.
Otherwise, return the translated message.  Overridden in derived classes.


New in version 3.8.

@end deffn

@geindex info() (gettext.NullTranslations method)
@anchor{library/gettext gettext NullTranslations info}@anchor{34db}
@deffn {Method} info ()

Return the “protected” @code{_info} variable, a dictionary containing
the metadata found in the message catalog file.
@end deffn

@geindex charset() (gettext.NullTranslations method)
@anchor{library/gettext gettext NullTranslations charset}@anchor{34dc}
@deffn {Method} charset ()

Return the encoding of the message catalog file.
@end deffn

@geindex install() (gettext.NullTranslations method)
@anchor{library/gettext gettext NullTranslations install}@anchor{34d3}
@deffn {Method} install (names=None)

This method installs @ref{34d7,,gettext()} into the built-in namespace,
binding it to @code{_}.

If the @emph{names} parameter is given, it must be a sequence containing the
names of functions you want to install in the builtins namespace in
addition to @code{_()}.  Supported names are @code{'gettext'}, @code{'ngettext'},
@code{'pgettext'}, @code{'npgettext'}, @code{'lgettext'}, and @code{'lngettext'}.

Note that this is only one way, albeit the most convenient way, to make
the @code{_()} function available to your application.  Because it affects
the entire application globally, and specifically the built-in namespace,
localized modules should never install @code{_()}. Instead, they should use
this code to make @code{_()} available to their module:

@example
import gettext
t = gettext.translation('mymodule', ...)
_ = t.gettext
@end example

This puts @code{_()} only in the module’s global namespace and so only
affects calls within this module.


Changed in version 3.8: Added @code{'pgettext'} and @code{'npgettext'}.

@end deffn
@end deffn

@node The GNUTranslations class,Solaris message catalog support,The NullTranslations class,Class-based API
@anchor{library/gettext the-gnutranslations-class}@anchor{34dd}
@subsubsection The @code{GNUTranslations} class


The @ref{83,,gettext} module provides one additional class derived from
@ref{34d2,,NullTranslations}: @ref{34d0,,GNUTranslations}.  This class overrides
@code{_parse()} to enable reading GNU @code{gettext} format @code{.mo} files
in both big-endian and little-endian format.

@ref{34d0,,GNUTranslations} parses optional metadata out of the translation
catalog. It is convention with GNU @code{gettext} to include metadata as
the translation for the empty string. This metadata is in 
@geindex RFC; RFC 822
RFC 822@footnote{https://tools.ietf.org/html/rfc822.html}-style
@code{key: value} pairs, and should contain the @code{Project-Id-Version} key.  If the
key @code{Content-Type} is found, then the @code{charset} property is used to
initialize the “protected” @code{_charset} instance variable, defaulting to
@code{None} if not found.  If the charset encoding is specified, then all message
ids and message strings read from the catalog are converted to Unicode using
this encoding, else ASCII is assumed.

Since message ids are read as Unicode strings too, all @code{*gettext()} methods
will assume message ids as Unicode strings, not byte strings.

The entire set of key/value pairs are placed into a dictionary and set as the
“protected” @code{_info} instance variable.

If the @code{.mo} file’s magic number is invalid, the major version number is
unexpected, or if other problems occur while reading the file, instantiating a
@ref{34d0,,GNUTranslations} class can raise @ref{2a1,,OSError}.

@geindex GNUTranslations (class in gettext)
@anchor{library/gettext gettext GNUTranslations}@anchor{34d0}
@deffn {Class} gettext.GNUTranslations

The following methods are overridden from the base class implementation:

@geindex gettext() (gettext.GNUTranslations method)
@anchor{library/gettext gettext GNUTranslations gettext}@anchor{34de}
@deffn {Method} gettext (message)

Look up the @emph{message} id in the catalog and return the corresponding message
string, as a Unicode string.  If there is no entry in the catalog for the
@emph{message} id, and a fallback has been set, the look up is forwarded to the
fallback’s @ref{34d7,,gettext()} method.  Otherwise, the
@emph{message} id is returned.
@end deffn

@geindex ngettext() (gettext.GNUTranslations method)
@anchor{library/gettext gettext GNUTranslations ngettext}@anchor{34df}
@deffn {Method} ngettext (singular, plural, n)

Do a plural-forms lookup of a message id.  @emph{singular} is used as the message id
for purposes of lookup in the catalog, while @emph{n} is used to determine which
plural form to use.  The returned message string is a Unicode string.

If the message id is not found in the catalog, and a fallback is specified,
the request is forwarded to the fallback’s @ref{34d8,,ngettext()}
method.  Otherwise, when @emph{n} is 1 @emph{singular} is returned, and @emph{plural} is
returned in all other cases.

Here is an example:

@example
n = len(os.listdir('.'))
cat = GNUTranslations(somefile)
message = cat.ngettext(
    'There is %(num)d file in this directory',
    'There are %(num)d files in this directory',
    n) % @{'num': n@}
@end example
@end deffn

@geindex pgettext() (gettext.GNUTranslations method)
@anchor{library/gettext gettext GNUTranslations pgettext}@anchor{34e0}
@deffn {Method} pgettext (context, message)

Look up the @emph{context} and @emph{message} id in the catalog and return the
corresponding message string, as a Unicode string.  If there is no
entry in the catalog for the @emph{message} id and @emph{context}, and a fallback
has been set, the look up is forwarded to the fallback’s
@ref{4fa,,pgettext()} method.  Otherwise, the @emph{message} id is returned.


New in version 3.8.

@end deffn

@geindex npgettext() (gettext.GNUTranslations method)
@anchor{library/gettext gettext GNUTranslations npgettext}@anchor{34e1}
@deffn {Method} npgettext (context, singular, plural, n)

Do a plural-forms lookup of a message id.  @emph{singular} is used as the
message id for purposes of lookup in the catalog, while @emph{n} is used to
determine which plural form to use.

If the message id for @emph{context} is not found in the catalog, and a
fallback is specified, the request is forwarded to the fallback’s
@ref{34cd,,npgettext()} method.  Otherwise, when @emph{n} is 1 @emph{singular} is
returned, and @emph{plural} is returned in all other cases.


New in version 3.8.

@end deffn
@end deffn

@node Solaris message catalog support,The Catalog constructor,The GNUTranslations class,Class-based API
@anchor{library/gettext solaris-message-catalog-support}@anchor{34e2}
@subsubsection Solaris message catalog support


The Solaris operating system defines its own binary @code{.mo} file format, but
since no documentation can be found on this format, it is not supported at this
time.

@node The Catalog constructor,,Solaris message catalog support,Class-based API
@anchor{library/gettext the-catalog-constructor}@anchor{34e3}
@subsubsection The Catalog constructor


@geindex GNOME

GNOME uses a version of the @ref{83,,gettext} module by James Henstridge, but this
version has a slightly different API.  Its documented usage was:

@example
import gettext
cat = gettext.Catalog(domain, localedir)
_ = cat.gettext
print(_('hello world'))
@end example

For compatibility with this older module, the function @code{Catalog()} is an
alias for the @ref{1bd,,translation()} function described above.

One difference between this module and Henstridge’s: his catalog objects
supported access through a mapping API, but this appears to be unused and so is
not currently supported.

@node Internationalizing your programs and modules,Acknowledgements<9>,Class-based API,gettext — Multilingual internationalization services
@anchor{library/gettext internationalizing-your-programs-and-modules}@anchor{34e4}
@subsubsection Internationalizing your programs and modules


Internationalization (I18N) refers to the operation by which a program is made
aware of multiple languages.  Localization (L10N) refers to the adaptation of
your program, once internationalized, to the local language and cultural habits.
In order to provide multilingual messages for your Python programs, you need to
take the following steps:


@enumerate 

@item 
prepare your program or module by specially marking translatable strings

@item 
run a suite of tools over your marked files to generate raw messages catalogs

@item 
create language-specific translations of the message catalogs

@item 
use the @ref{83,,gettext} module so that message strings are properly translated
@end enumerate

In order to prepare your code for I18N, you need to look at all the strings in
your files.  Any string that needs to be translated should be marked by wrapping
it in @code{_('...')} — that is, a call to the function @code{_()}.  For example:

@example
filename = 'mylog.txt'
message = _('writing a log message')
with open(filename, 'w') as fp:
    fp.write(message)
@end example

In this example, the string @code{'writing a log message'} is marked as a candidate
for translation, while the strings @code{'mylog.txt'} and @code{'w'} are not.

There are a few tools to extract the strings meant for translation.
The original GNU @code{gettext} only supported C or C++ source
code but its extended version @code{xgettext} scans code written
in a number of languages, including Python, to find strings marked as
translatable.  Babel@footnote{http://babel.pocoo.org/} is a Python
internationalization library that includes a @code{pybabel} script to
extract and compile message catalogs.  François Pinard’s program
called @code{xpot} does a similar job and is available as part of
his po-utils package@footnote{https://github.com/pinard/po-utils}.

(Python also includes pure-Python versions of these programs, called
@code{pygettext.py} and @code{msgfmt.py}; some Python distributions
will install them for you.  @code{pygettext.py} is similar to
@code{xgettext}, but only understands Python source code and
cannot handle other programming languages such as C or C++.
@code{pygettext.py} supports a command-line interface similar to
@code{xgettext}; for details on its use, run @code{pygettext.py
--help}.  @code{msgfmt.py} is binary compatible with GNU
@code{msgfmt}.  With these two programs, you may not need the GNU
@code{gettext} package to internationalize your Python
applications.)

@code{xgettext}, @code{pygettext}, and similar tools generate
@code{.po} files that are message catalogs.  They are structured
human-readable files that contain every marked string in the source
code, along with a placeholder for the translated versions of these
strings.

Copies of these @code{.po} files are then handed over to the
individual human translators who write translations for every
supported natural language.  They send back the completed
language-specific versions as a @code{<language-name>.po} file that’s
compiled into a machine-readable @code{.mo} binary catalog file using
the @code{msgfmt} program.  The @code{.mo} files are used by the
@ref{83,,gettext} module for the actual translation processing at
run-time.

How you use the @ref{83,,gettext} module in your code depends on whether you are
internationalizing a single module or your entire application. The next two
sections will discuss each case.

@menu
* Localizing your module:: 
* Localizing your application:: 
* Changing languages on the fly:: 
* Deferred translations:: 

@end menu

@node Localizing your module,Localizing your application,,Internationalizing your programs and modules
@anchor{library/gettext localizing-your-module}@anchor{34e5}
@subsubsection Localizing your module


If you are localizing your module, you must take care not to make global
changes, e.g. to the built-in namespace. You should not use the GNU @code{gettext}
API but instead the class-based API.

Let’s say your module is called “spam” and the module’s various natural language
translation @code{.mo} files reside in @code{/usr/share/locale} in GNU
@code{gettext} format.  Here’s what you would put at the top of your
module:

@example
import gettext
t = gettext.translation('spam', '/usr/share/locale')
_ = t.gettext
@end example

@node Localizing your application,Changing languages on the fly,Localizing your module,Internationalizing your programs and modules
@anchor{library/gettext localizing-your-application}@anchor{34e6}
@subsubsection Localizing your application


If you are localizing your application, you can install the @code{_()} function
globally into the built-in namespace, usually in the main driver file of your
application.  This will let all your application-specific files just use
@code{_('...')} without having to explicitly install it in each file.

In the simple case then, you need only add the following bit of code to the main
driver file of your application:

@example
import gettext
gettext.install('myapplication')
@end example

If you need to set the locale directory, you can pass it into the
@ref{1be,,install()} function:

@example
import gettext
gettext.install('myapplication', '/usr/share/locale')
@end example

@node Changing languages on the fly,Deferred translations,Localizing your application,Internationalizing your programs and modules
@anchor{library/gettext changing-languages-on-the-fly}@anchor{34e7}
@subsubsection Changing languages on the fly


If your program needs to support many languages at the same time, you may want
to create multiple translation instances and then switch between them
explicitly, like so:

@example
import gettext

lang1 = gettext.translation('myapplication', languages=['en'])
lang2 = gettext.translation('myapplication', languages=['fr'])
lang3 = gettext.translation('myapplication', languages=['de'])

# start by using language1
lang1.install()

# ... time goes by, user selects language 2
lang2.install()

# ... more time goes by, user selects language 3
lang3.install()
@end example

@node Deferred translations,,Changing languages on the fly,Internationalizing your programs and modules
@anchor{library/gettext deferred-translations}@anchor{34e8}
@subsubsection Deferred translations


In most coding situations, strings are translated where they are coded.
Occasionally however, you need to mark strings for translation, but defer actual
translation until later.  A classic example is:

@example
animals = ['mollusk',
           'albatross',
           'rat',
           'penguin',
           'python', ]
# ...
for a in animals:
    print(a)
@end example

Here, you want to mark the strings in the @code{animals} list as being
translatable, but you don’t actually want to translate them until they are
printed.

Here is one way you can handle this situation:

@example
def _(message): return message

animals = [_('mollusk'),
           _('albatross'),
           _('rat'),
           _('penguin'),
           _('python'), ]

del _

# ...
for a in animals:
    print(_(a))
@end example

This works because the dummy definition of @code{_()} simply returns the string
unchanged.  And this dummy definition will temporarily override any definition
of @code{_()} in the built-in namespace (until the @ref{152e,,del} command). Take
care, though if you have a previous definition of @code{_()} in the local
namespace.

Note that the second use of @code{_()} will not identify “a” as being
translatable to the @code{gettext} program, because the parameter
is not a string literal.

Another way to handle this is with the following example:

@example
def N_(message): return message

animals = [N_('mollusk'),
           N_('albatross'),
           N_('rat'),
           N_('penguin'),
           N_('python'), ]

# ...
for a in animals:
    print(_(a))
@end example

In this case, you are marking translatable strings with the function
@code{N_()}, which won’t conflict with any definition of @code{_()}.
However, you will need to teach your message extraction program to
look for translatable strings marked with @code{N_()}. @code{xgettext},
@code{pygettext}, @code{pybabel extract}, and @code{xpot} all
support this through the use of the @code{-k} command-line switch.
The choice of @code{N_()} here is totally arbitrary; it could have just
as easily been @code{MarkThisStringForTranslation()}.

@node Acknowledgements<9>,,Internationalizing your programs and modules,gettext — Multilingual internationalization services
@anchor{library/gettext acknowledgements}@anchor{34e9}
@subsubsection Acknowledgements


The following people contributed code, feedback, design suggestions, previous
implementations, and valuable experience to the creation of this module:


@itemize *

@item 
Peter Funk

@item 
James Henstridge

@item 
Juan David Ibáñez Palomar

@item 
Marc-André Lemburg

@item 
Martin von Löwis

@item 
François Pinard

@item 
Barry Warsaw

@item 
Gustavo Niemeyer
@end itemize

@node locale — Internationalization services,,gettext — Multilingual internationalization services,Internationalization
@anchor{library/locale doc}@anchor{34ea}@anchor{library/locale locale-internationalization-services}@anchor{34eb}@anchor{library/locale module-locale}@anchor{a5}
@subsection @code{locale} — Internationalization services


@geindex module; locale

@strong{Source code:} Lib/locale.py@footnote{https://github.com/python/cpython/tree/main/Lib/locale.py}


__________________________________________________________________


The @ref{a5,,locale} module opens access to the POSIX locale database and
functionality. The POSIX locale mechanism allows programmers to deal with
certain cultural issues in an application, without requiring the programmer to
know all the specifics of each country where the software is executed.

@geindex module; _locale

The @ref{a5,,locale} module is implemented on top of the @code{_locale} module,
which in turn uses an ANSI C locale implementation if available.

The @ref{a5,,locale} module defines the following exception and functions:

@geindex Error
@anchor{library/locale locale Error}@anchor{34ec}
@deffn {Exception} locale.Error

Exception raised when the locale passed to @ref{1b4,,setlocale()} is not
recognized.
@end deffn

@geindex setlocale() (in module locale)
@anchor{library/locale locale setlocale}@anchor{1b4}
@deffn {Function} locale.setlocale (category, locale=None)

If @emph{locale} is given and not @code{None}, @ref{1b4,,setlocale()} modifies the locale
setting for the @emph{category}. The available categories are listed in the data
description below. @emph{locale} may be a string, or an iterable of two strings
(language code and encoding). If it’s an iterable, it’s converted to a locale
name using the locale aliasing engine. An empty string specifies the user’s
default settings. If the modification of the locale fails, the exception
@ref{34ec,,Error} is raised. If successful, the new locale setting is returned.

If @emph{locale} is omitted or @code{None}, the current setting for @emph{category} is
returned.

@ref{1b4,,setlocale()} is not thread-safe on most systems. Applications typically
start with a call of

@example
import locale
locale.setlocale(locale.LC_ALL, '@w{'})
@end example

This sets the locale for all categories to the user’s default setting (typically
specified in the 
@geindex LANG
@geindex environment variable; LANG
@code{LANG} environment variable).  If the locale is not
changed thereafter, using multithreading should not cause problems.
@end deffn

@geindex localeconv() (in module locale)
@anchor{library/locale locale localeconv}@anchor{739}
@deffn {Function} locale.localeconv ()

Returns the database of the local conventions as a dictionary. This dictionary
has the following strings as keys:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Category

@tab

Key

@tab

Meaning

@item

@ref{34ed,,LC_NUMERIC}

@tab

@code{'decimal_point'}

@tab

Decimal point character.

@item

@tab

@code{'grouping'}

@tab

Sequence of numbers specifying
which relative positions the
@code{'thousands_sep'} is
expected.  If the sequence is
terminated with
@ref{34ee,,CHAR_MAX}, no further
grouping is performed. If the
sequence terminates with a
@code{0},  the last group size is
repeatedly used.

@item

@tab

@code{'thousands_sep'}

@tab

Character used between groups.

@item

@ref{34ef,,LC_MONETARY}

@tab

@code{'int_curr_symbol'}

@tab

International currency symbol.

@item

@tab

@code{'currency_symbol'}

@tab

Local currency symbol.

@item

@tab

@code{'p_cs_precedes/n_cs_precedes'}

@tab

Whether the currency symbol
precedes the value (for
positive resp. negative
values).

@item

@tab

@code{'p_sep_by_space/n_sep_by_space'}

@tab

Whether the currency symbol is
separated from the value  by a
space (for positive resp.
negative values).

@item

@tab

@code{'mon_decimal_point'}

@tab

Decimal point used for
monetary values.

@item

@tab

@code{'frac_digits'}

@tab

Number of fractional digits
used in local formatting of
monetary values.

@item

@tab

@code{'int_frac_digits'}

@tab

Number of fractional digits
used in international
formatting of monetary values.

@item

@tab

@code{'mon_thousands_sep'}

@tab

Group separator used for
monetary values.

@item

@tab

@code{'mon_grouping'}

@tab

Equivalent to @code{'grouping'},
used for monetary values.

@item

@tab

@code{'positive_sign'}

@tab

Symbol used to annotate a
positive monetary value.

@item

@tab

@code{'negative_sign'}

@tab

Symbol used to annotate a
negative monetary value.

@item

@tab

@code{'p_sign_posn/n_sign_posn'}

@tab

The position of the sign (for
positive resp. negative
values), see below.

@end multitable


All numeric values can be set to @ref{34ee,,CHAR_MAX} to indicate that there is no
value specified in this locale.

The possible values for @code{'p_sign_posn'} and @code{'n_sign_posn'} are given below.


@multitable {xxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Value

@tab

Explanation

@item

@code{0}

@tab

Currency and value are surrounded by
parentheses.

@item

@code{1}

@tab

The sign should precede the value and
currency symbol.

@item

@code{2}

@tab

The sign should follow the value and
currency symbol.

@item

@code{3}

@tab

The sign should immediately precede the
value.

@item

@code{4}

@tab

The sign should immediately follow the
value.

@item

@code{CHAR_MAX}

@tab

Nothing is specified in this locale.

@end multitable


The function sets temporarily the @code{LC_CTYPE} locale to the @code{LC_NUMERIC}
locale or the @code{LC_MONETARY} locale if locales are different and numeric or
monetary strings are non-ASCII. This temporary change affects other threads.


Changed in version 3.7: The function now sets temporarily the @code{LC_CTYPE} locale to the
@code{LC_NUMERIC} locale in some cases.

@end deffn

@geindex nl_langinfo() (in module locale)
@anchor{library/locale locale nl_langinfo}@anchor{34f0}
@deffn {Function} locale.nl_langinfo (option)

Return some locale-specific information as a string.  This function is not
available on all systems, and the set of possible options might also vary
across platforms.  The possible argument values are numbers, for which
symbolic constants are available in the locale module.

The @ref{34f0,,nl_langinfo()} function accepts one of the following keys.  Most
descriptions are taken from the corresponding description in the GNU C
library.

@geindex CODESET (in module locale)
@anchor{library/locale locale CODESET}@anchor{34f1}
@deffn {Data} locale.CODESET

Get a string with the name of the character encoding used in the
selected locale.
@end deffn

@geindex D_T_FMT (in module locale)
@anchor{library/locale locale D_T_FMT}@anchor{34f2}
@deffn {Data} locale.D_T_FMT

Get a string that can be used as a format string for @ref{d8b,,time.strftime()} to
represent date and time in a locale-specific way.
@end deffn

@geindex D_FMT (in module locale)
@anchor{library/locale locale D_FMT}@anchor{34f3}
@deffn {Data} locale.D_FMT

Get a string that can be used as a format string for @ref{d8b,,time.strftime()} to
represent a date in a locale-specific way.
@end deffn

@geindex T_FMT (in module locale)
@anchor{library/locale locale T_FMT}@anchor{34f4}
@deffn {Data} locale.T_FMT

Get a string that can be used as a format string for @ref{d8b,,time.strftime()} to
represent a time in a locale-specific way.
@end deffn

@geindex T_FMT_AMPM (in module locale)
@anchor{library/locale locale T_FMT_AMPM}@anchor{34f5}
@deffn {Data} locale.T_FMT_AMPM

Get a format string for @ref{d8b,,time.strftime()} to represent time in the am/pm
format.
@end deffn


@deffn {Data} DAY_1 ... DAY_7

Get the name of the n-th day of the week.

@cartouche
@quotation Note 
This follows the US convention of @code{DAY_1} being Sunday, not the
international convention (ISO 8601) that Monday is the first day of the
week.
@end quotation
@end cartouche
@end deffn


@deffn {Data} ABDAY_1 ... ABDAY_7

Get the abbreviated name of the n-th day of the week.
@end deffn


@deffn {Data} MON_1 ... MON_12

Get the name of the n-th month.
@end deffn


@deffn {Data} ABMON_1 ... ABMON_12

Get the abbreviated name of the n-th month.
@end deffn

@geindex RADIXCHAR (in module locale)
@anchor{library/locale locale RADIXCHAR}@anchor{34f6}
@deffn {Data} locale.RADIXCHAR

Get the radix character (decimal dot, decimal comma, etc.).
@end deffn

@geindex THOUSEP (in module locale)
@anchor{library/locale locale THOUSEP}@anchor{34f7}
@deffn {Data} locale.THOUSEP

Get the separator character for thousands (groups of three digits).
@end deffn

@geindex YESEXPR (in module locale)
@anchor{library/locale locale YESEXPR}@anchor{34f8}
@deffn {Data} locale.YESEXPR

Get a regular expression that can be used with the regex function to
recognize a positive response to a yes/no question.

@cartouche
@quotation Note 
The expression is in the syntax suitable for the @code{regex()} function
from the C library, which might differ from the syntax used in @ref{d8,,re}.
@end quotation
@end cartouche
@end deffn

@geindex NOEXPR (in module locale)
@anchor{library/locale locale NOEXPR}@anchor{34f9}
@deffn {Data} locale.NOEXPR

Get a regular expression that can be used with the regex(3) function to
recognize a negative response to a yes/no question.
@end deffn

@geindex CRNCYSTR (in module locale)
@anchor{library/locale locale CRNCYSTR}@anchor{34fa}
@deffn {Data} locale.CRNCYSTR

Get the currency symbol, preceded by “-” if the symbol should appear before
the value, “+” if the symbol should appear after the value, or “.” if the
symbol should replace the radix character.
@end deffn

@geindex ERA (in module locale)
@anchor{library/locale locale ERA}@anchor{34fb}
@deffn {Data} locale.ERA

Get a string that represents the era used in the current locale.

Most locales do not define this value.  An example of a locale which does
define this value is the Japanese one.  In Japan, the traditional
representation of dates includes the name of the era corresponding to the
then-emperor’s reign.

Normally it should not be necessary to use this value directly. Specifying
the @code{E} modifier in their format strings causes the @ref{d8b,,time.strftime()}
function to use this information.  The format of the returned string is not
specified, and therefore you should not assume knowledge of it on different
systems.
@end deffn

@geindex ERA_D_T_FMT (in module locale)
@anchor{library/locale locale ERA_D_T_FMT}@anchor{34fc}
@deffn {Data} locale.ERA_D_T_FMT

Get a format string for @ref{d8b,,time.strftime()} to represent date and time in a
locale-specific era-based way.
@end deffn

@geindex ERA_D_FMT (in module locale)
@anchor{library/locale locale ERA_D_FMT}@anchor{34fd}
@deffn {Data} locale.ERA_D_FMT

Get a format string for @ref{d8b,,time.strftime()} to represent a date in a
locale-specific era-based way.
@end deffn

@geindex ERA_T_FMT (in module locale)
@anchor{library/locale locale ERA_T_FMT}@anchor{34fe}
@deffn {Data} locale.ERA_T_FMT

Get a format string for @ref{d8b,,time.strftime()} to represent a time in a
locale-specific era-based way.
@end deffn

@geindex ALT_DIGITS (in module locale)
@anchor{library/locale locale ALT_DIGITS}@anchor{34ff}
@deffn {Data} locale.ALT_DIGITS

Get a representation of up to 100 values used to represent the values
0 to 99.
@end deffn
@end deffn

@geindex getdefaultlocale() (in module locale)
@anchor{library/locale locale getdefaultlocale}@anchor{1b3}
@deffn {Function} locale.getdefaultlocale ([envvars])

Tries to determine the default locale settings and returns them as a tuple of
the form @code{(language code, encoding)}.

According to POSIX, a program which has not called @code{setlocale(LC_ALL, '@w{'})}
runs using the portable @code{'C'} locale.  Calling @code{setlocale(LC_ALL, '@w{'})} lets
it use the default locale as defined by the 
@geindex LANG
@geindex environment variable; LANG
@code{LANG} variable.  Since we
do not want to interfere with the current locale setting we thus emulate the
behavior in the way described above.

To maintain compatibility with other platforms, not only the 
@geindex LANG
@geindex environment variable; LANG
@code{LANG}
variable is tested, but a list of variables given as envvars parameter.  The
first found to be defined will be used.  @emph{envvars} defaults to the search
path used in GNU gettext; it must always contain the variable name
@code{'LANG'}.  The GNU gettext search path contains @code{'LC_ALL'},
@code{'LC_CTYPE'}, @code{'LANG'} and @code{'LANGUAGE'}, in that order.

Except for the code @code{'C'}, the language code corresponds to 
@geindex RFC; RFC 1766
RFC 1766@footnote{https://tools.ietf.org/html/rfc1766.html}.
@emph{language code} and @emph{encoding} may be @code{None} if their values cannot be
determined.


Deprecated since version 3.11: 3.13

@end deffn

@geindex getlocale() (in module locale)
@anchor{library/locale locale getlocale}@anchor{1b6}
@deffn {Function} locale.getlocale (category=LC_CTYPE)

Returns the current setting for the given locale category as sequence containing
@emph{language code}, @emph{encoding}. @emph{category} may be one of the @code{LC_*} values
except @ref{3500,,LC_ALL}.  It defaults to @ref{3501,,LC_CTYPE}.

Except for the code @code{'C'}, the language code corresponds to 
@geindex RFC; RFC 1766
RFC 1766@footnote{https://tools.ietf.org/html/rfc1766.html}.
@emph{language code} and @emph{encoding} may be @code{None} if their values cannot be
determined.
@end deffn

@geindex getpreferredencoding() (in module locale)
@anchor{library/locale locale getpreferredencoding}@anchor{1b5}
@deffn {Function} locale.getpreferredencoding (do_setlocale=True)

Return the @ref{237a,,locale encoding} used for text data, according to user
preferences.  User preferences are expressed differently on different
systems, and might not be available programmatically on some systems, so
this function only returns a guess.

On some systems, it is necessary to invoke @ref{1b4,,setlocale()} to obtain the
user preferences, so this function is not thread-safe. If invoking setlocale
is not necessary or desired, @emph{do_setlocale} should be set to @code{False}.

On Android or if the @ref{5f2,,Python UTF-8 Mode} is enabled, always
return @code{'UTF-8'}, the @ref{237a,,locale encoding} and the @emph{do_setlocale}
argument are ignored.

The @ref{3502,,Python preinitialization} configures the LC_CTYPE
locale. See also the @ref{1628,,filesystem encoding and error handler}.


Changed in version 3.7: The function now always returns @code{UTF-8} on Android or if the
@ref{5f2,,Python UTF-8 Mode} is enabled.

@end deffn

@geindex normalize() (in module locale)
@anchor{library/locale locale normalize}@anchor{3503}
@deffn {Function} locale.normalize (localename)

Returns a normalized locale code for the given locale name.  The returned locale
code is formatted for use with @ref{1b4,,setlocale()}.  If normalization fails, the
original name is returned unchanged.

If the given encoding is not known, the function defaults to the default
encoding for the locale code just like @ref{1b4,,setlocale()}.
@end deffn

@geindex resetlocale() (in module locale)
@anchor{library/locale locale resetlocale}@anchor{3504}
@deffn {Function} locale.resetlocale (category=LC_ALL)

Sets the locale for @emph{category} to the default setting.

The default setting is determined by calling @ref{1b3,,getdefaultlocale()}.
@emph{category} defaults to @ref{3500,,LC_ALL}.
@end deffn

@geindex strcoll() (in module locale)
@anchor{library/locale locale strcoll}@anchor{3505}
@deffn {Function} locale.strcoll (string1, string2)

Compares two strings according to the current @ref{3506,,LC_COLLATE} setting. As
any other compare function, returns a negative, or a positive value, or @code{0},
depending on whether @emph{string1} collates before or after @emph{string2} or is equal to
it.
@end deffn

@geindex strxfrm() (in module locale)
@anchor{library/locale locale strxfrm}@anchor{3507}
@deffn {Function} locale.strxfrm (string)

Transforms a string to one that can be used in locale-aware
comparisons.  For example, @code{strxfrm(s1) < strxfrm(s2)} is
equivalent to @code{strcoll(s1, s2) < 0}.  This function can be used
when the same string is compared repeatedly, e.g. when collating a
sequence of strings.
@end deffn

@geindex format_string() (in module locale)
@anchor{library/locale locale format_string}@anchor{674}
@deffn {Function} locale.format_string (format, val, grouping=False, monetary=False)

Formats a number @emph{val} according to the current @ref{34ed,,LC_NUMERIC} setting.
The format follows the conventions of the @code{%} operator.  For floating point
values, the decimal point is modified if appropriate.  If @emph{grouping} is true,
also takes the grouping into account.

If @emph{monetary} is true, the conversion uses monetary thousands separator and
grouping strings.

Processes formatting specifiers as in @code{format % val}, but takes the current
locale settings into account.


Changed in version 3.7: The @emph{monetary} keyword parameter was added.

@end deffn

@geindex format() (in module locale)
@anchor{library/locale locale format}@anchor{721}
@deffn {Function} locale.format (format, val, grouping=False, monetary=False)

Please note that this function works like @ref{674,,format_string()} but will
only work for exactly one @code{%char} specifier.  For example, @code{'%f'} and
@code{'%.0f'} are both valid specifiers, but @code{'%f KiB'} is not.

For whole format strings, use @ref{674,,format_string()}.


Deprecated since version 3.7: Use @ref{674,,format_string()} instead.

@end deffn

@geindex currency() (in module locale)
@anchor{library/locale locale currency}@anchor{3508}
@deffn {Function} locale.currency (val, symbol=True, grouping=False, international=False)

Formats a number @emph{val} according to the current @ref{34ef,,LC_MONETARY} settings.

The returned string includes the currency symbol if @emph{symbol} is true, which is
the default. If @emph{grouping} is true (which is not the default), grouping is done
with the value. If @emph{international} is true (which is not the default), the
international currency symbol is used.

Note that this function will not work with the ‘C’ locale, so you have to set a
locale via @ref{1b4,,setlocale()} first.
@end deffn

@geindex str() (in module locale)
@anchor{library/locale locale str}@anchor{3509}
@deffn {Function} locale.str (float)

Formats a floating point number using the same format as the built-in function
@code{str(float)}, but takes the decimal point into account.
@end deffn

@geindex delocalize() (in module locale)
@anchor{library/locale locale delocalize}@anchor{98c}
@deffn {Function} locale.delocalize (string)

Converts a string into a normalized number string, following the
@ref{34ed,,LC_NUMERIC} settings.


New in version 3.5.

@end deffn

@geindex localize() (in module locale)
@anchor{library/locale locale localize}@anchor{350a}
@deffn {Function} locale.localize (string, grouping=False, monetary=False)

Converts a normalized number string into a formatted string following the
@ref{34ed,,LC_NUMERIC} settings.


New in version 3.10.

@end deffn

@geindex atof() (in module locale)
@anchor{library/locale locale atof}@anchor{350b}
@deffn {Function} locale.atof (string)

Converts a string to a floating point number, following the @ref{34ed,,LC_NUMERIC}
settings.
@end deffn

@geindex atoi() (in module locale)
@anchor{library/locale locale atoi}@anchor{350c}
@deffn {Function} locale.atoi (string)

Converts a string to an integer, following the @ref{34ed,,LC_NUMERIC} conventions.
@end deffn

@geindex LC_CTYPE (in module locale)
@anchor{library/locale locale LC_CTYPE}@anchor{3501}
@deffn {Data} locale.LC_CTYPE

@geindex module; string

Locale category for the character type functions.  Depending on the settings of
this category, the functions of module @ref{f1,,string} dealing with case change
their behaviour.
@end deffn

@geindex LC_COLLATE (in module locale)
@anchor{library/locale locale LC_COLLATE}@anchor{3506}
@deffn {Data} locale.LC_COLLATE

Locale category for sorting strings.  The functions @ref{3505,,strcoll()} and
@ref{3507,,strxfrm()} of the @ref{a5,,locale} module are affected.
@end deffn

@geindex LC_TIME (in module locale)
@anchor{library/locale locale LC_TIME}@anchor{350d}
@deffn {Data} locale.LC_TIME

Locale category for the formatting of time.  The function @ref{d8b,,time.strftime()}
follows these conventions.
@end deffn

@geindex LC_MONETARY (in module locale)
@anchor{library/locale locale LC_MONETARY}@anchor{34ef}
@deffn {Data} locale.LC_MONETARY

Locale category for formatting of monetary values.  The available options are
available from the @ref{739,,localeconv()} function.
@end deffn

@geindex LC_MESSAGES (in module locale)
@anchor{library/locale locale LC_MESSAGES}@anchor{350e}
@deffn {Data} locale.LC_MESSAGES

Locale category for message display. Python currently does not support
application specific locale-aware messages.  Messages displayed by the operating
system, like those returned by @ref{239b,,os.strerror()} might be affected by this
category.
@end deffn

@geindex LC_NUMERIC (in module locale)
@anchor{library/locale locale LC_NUMERIC}@anchor{34ed}
@deffn {Data} locale.LC_NUMERIC

Locale category for formatting numbers.  The functions @ref{721,,format()},
@ref{350c,,atoi()}, @ref{350b,,atof()} and @ref{3509,,str()} of the @ref{a5,,locale} module are
affected by that category.  All other numeric formatting operations are not
affected.
@end deffn

@geindex LC_ALL (in module locale)
@anchor{library/locale locale LC_ALL}@anchor{3500}
@deffn {Data} locale.LC_ALL

Combination of all locale settings.  If this flag is used when the locale is
changed, setting the locale for all categories is attempted. If that fails for
any category, no category is changed at all.  When the locale is retrieved using
this flag, a string indicating the setting for all categories is returned. This
string can be later used to restore the settings.
@end deffn

@geindex CHAR_MAX (in module locale)
@anchor{library/locale locale CHAR_MAX}@anchor{34ee}
@deffn {Data} locale.CHAR_MAX

This is a symbolic constant used for different values returned by
@ref{739,,localeconv()}.
@end deffn

Example:

@example
>>> import locale
>>> loc = locale.getlocale()  # get current locale
# use German locale; name might vary with platform
>>> locale.setlocale(locale.LC_ALL, 'de_DE')
>>> locale.strcoll('f\xe4n', 'foo')  # compare a string containing an umlaut
>>> locale.setlocale(locale.LC_ALL, '@w{'})   # use user's preferred locale
>>> locale.setlocale(locale.LC_ALL, 'C')  # use default (C) locale
>>> locale.setlocale(locale.LC_ALL, loc)  # restore saved locale
@end example

@menu
* Background@comma{} details@comma{} hints@comma{} tips and caveats: Background details hints tips and caveats. 
* For extension writers and programs that embed Python:: 
* Access to message catalogs:: 

@end menu

@node Background details hints tips and caveats,For extension writers and programs that embed Python,,locale — Internationalization services
@anchor{library/locale background-details-hints-tips-and-caveats}@anchor{350f}
@subsubsection Background, details, hints, tips and caveats


The C standard defines the locale as a program-wide property that may be
relatively expensive to change.  On top of that, some implementations are broken
in such a way that frequent locale changes may cause core dumps.  This makes the
locale somewhat painful to use correctly.

Initially, when a program is started, the locale is the @code{C} locale, no matter
what the user’s preferred locale is.  There is one exception: the
@ref{3501,,LC_CTYPE} category is changed at startup to set the current locale
encoding to the user’s preferred locale encoding. The program must explicitly
say that it wants the user’s preferred locale settings for other categories by
calling @code{setlocale(LC_ALL, '@w{'})}.

It is generally a bad idea to call @ref{1b4,,setlocale()} in some library routine,
since as a side effect it affects the entire program.  Saving and restoring it
is almost as bad: it is expensive and affects other threads that happen to run
before the settings have been restored.

If, when coding a module for general use, you need a locale independent version
of an operation that is affected by the locale (such as
certain formats used with @ref{d8b,,time.strftime()}), you will have to find a way to
do it without using the standard library routine.  Even better is convincing
yourself that using locale settings is okay.  Only as a last resort should you
document that your module is not compatible with non-@code{C} locale settings.

The only way to perform numeric operations according to the locale is to use the
special functions defined by this module: @ref{350b,,atof()}, @ref{350c,,atoi()},
@ref{721,,format()}, @ref{3509,,str()}.

There is no way to perform case conversions and character classifications
according to the locale.  For (Unicode) text strings these are done according
to the character value only, while for byte strings, the conversions and
classifications are done according to the ASCII value of the byte, and bytes
whose high bit is set (i.e., non-ASCII bytes) are never converted or considered
part of a character class such as letter or whitespace.

@node For extension writers and programs that embed Python,Access to message catalogs,Background details hints tips and caveats,locale — Internationalization services
@anchor{library/locale embedding-locale}@anchor{3510}@anchor{library/locale for-extension-writers-and-programs-that-embed-python}@anchor{3511}
@subsubsection For extension writers and programs that embed Python


Extension modules should never call @ref{1b4,,setlocale()}, except to find out what
the current locale is.  But since the return value can only be used portably to
restore it, that is not very useful (except perhaps to find out whether or not
the locale is @code{C}).

When Python code uses the @ref{a5,,locale} module to change the locale, this also
affects the embedding application.  If the embedding application doesn’t want
this to happen, it should remove the @code{_locale} extension module (which does
all the work) from the table of built-in modules in the @code{config.c} file,
and make sure that the @code{_locale} module is not accessible as a shared
library.

@node Access to message catalogs,,For extension writers and programs that embed Python,locale — Internationalization services
@anchor{library/locale access-to-message-catalogs}@anchor{3512}@anchor{library/locale locale-gettext}@anchor{3513}
@subsubsection Access to message catalogs


@geindex gettext() (in module locale)
@anchor{library/locale locale gettext}@anchor{3514}
@deffn {Function} locale.gettext (msg)
@end deffn

@geindex dgettext() (in module locale)
@anchor{library/locale locale dgettext}@anchor{3515}
@deffn {Function} locale.dgettext (domain, msg)
@end deffn

@geindex dcgettext() (in module locale)
@anchor{library/locale locale dcgettext}@anchor{3516}
@deffn {Function} locale.dcgettext (domain, msg, category)
@end deffn

@geindex textdomain() (in module locale)
@anchor{library/locale locale textdomain}@anchor{3517}
@deffn {Function} locale.textdomain (domain)
@end deffn

@geindex bindtextdomain() (in module locale)
@anchor{library/locale locale bindtextdomain}@anchor{3518}
@deffn {Function} locale.bindtextdomain (domain, dir)
@end deffn

The locale module exposes the C library’s gettext interface on systems that
provide this interface.  It consists of the functions @code{gettext()},
@code{dgettext()}, @code{dcgettext()}, @code{textdomain()}, @code{bindtextdomain()},
and @code{bind_textdomain_codeset()}.  These are similar to the same functions in
the @ref{83,,gettext} module, but use the C library’s binary format for message
catalogs, and the C library’s search algorithms for locating message catalogs.

Python applications should normally find no need to invoke these functions, and
should use @ref{83,,gettext} instead.  A known exception to this rule are
applications that link with additional C libraries which internally invoke
@code{gettext()} or @code{dcgettext()}.  For these applications, it may be
necessary to bind the text domain, so that the libraries can properly locate
their message catalogs.

@node Program Frameworks,Graphical User Interfaces with Tk,Internationalization,The Python Standard Library
@anchor{library/frameworks doc}@anchor{3519}@anchor{library/frameworks frameworks}@anchor{351a}@anchor{library/frameworks program-frameworks}@anchor{351b}
@section Program Frameworks


The modules described in this chapter are frameworks that will largely dictate
the structure of your program.  Currently the modules described  here are all
oriented toward writing command-line interfaces.

The full list of modules described in this chapter is:

@menu
* turtle — Turtle graphics:: 
* cmd — Support for line-oriented command interpreters:: 
* shlex — Simple lexical analysis:: 

@end menu

@node turtle — Turtle graphics,cmd — Support for line-oriented command interpreters,,Program Frameworks
@anchor{library/turtle doc}@anchor{351c}@anchor{library/turtle module-turtle}@anchor{11d}@anchor{library/turtle turtle-turtle-graphics}@anchor{351d}
@subsection @code{turtle} — Turtle graphics


@geindex module; turtle

@strong{Source code:} Lib/turtle.py@footnote{https://github.com/python/cpython/tree/main/Lib/turtle.py}

@c from turtle import *
@c turtle = Turtle()


__________________________________________________________________


@menu
* Introduction: Introduction<11>. 
* Overview of available Turtle and Screen methods:: 
* Methods of RawTurtle/Turtle and corresponding functions:: 
* Methods of TurtleScreen/Screen and corresponding functions:: 
* Public classes:: 
* Help and configuration:: 
* turtledemo — Demo scripts:: 
* Changes since Python 2.6: Changes since Python 2 6. 
* Changes since Python 3.0: Changes since Python 3 0. 

@end menu

@node Introduction<11>,Overview of available Turtle and Screen methods,,turtle — Turtle graphics
@anchor{library/turtle introduction}@anchor{351e}
@subsubsection Introduction


Turtle graphics is a popular way for introducing programming to kids.  It was
part of the original Logo programming language developed by Wally Feurzeig,
Seymour Papert and Cynthia Solomon in 1967.

Imagine a robotic turtle starting at (0, 0) in the x-y plane.  After an @code{import turtle}, give it the
command @code{turtle.forward(15)}, and it moves (on-screen!) 15 pixels in the
direction it is facing, drawing a line as it moves.  Give it the command
@code{turtle.right(25)}, and it rotates in-place 25 degrees clockwise.

@subsubheading Turtle star



Turtle can draw intricate shapes using programs that repeat simple
moves.

@image{python-figures/turtle-star,,,,png}

@example
from turtle import *
color('red', 'yellow')
begin_fill()
while True:
    forward(200)
    left(170)
    if abs(pos()) < 1:
        break
end_fill()
done()
@end example

By combining together these and similar commands, intricate shapes and pictures
can easily be drawn.

The @ref{11d,,turtle} module is an extended reimplementation of the same-named
module from the Python standard distribution up to version Python 2.5.

It tries to keep the merits of the old turtle module and to be (nearly) 100%
compatible with it.  This means in the first place to enable the learning
programmer to use all the commands, classes and methods interactively when using
the module from within IDLE run with the @code{-n} switch.

The turtle module provides turtle graphics primitives, in both object-oriented
and procedure-oriented ways.  Because it uses @ref{10c,,tkinter} for the underlying
graphics, it needs a version of Python installed with Tk support.

The object-oriented interface uses essentially two+two classes:


@enumerate 

@item 
The @ref{351f,,TurtleScreen} class defines graphics windows as a playground for
the drawing turtles.  Its constructor needs a @code{tkinter.Canvas} or a
@ref{3520,,ScrolledCanvas} as argument.  It should be used when @ref{11d,,turtle} is
used as part of some application.

The function @ref{3521,,Screen()} returns a singleton object of a
@ref{351f,,TurtleScreen} subclass. This function should be used when
@ref{11d,,turtle} is used as a standalone tool for doing graphics.
As a singleton object, inheriting from its class is not possible.

All methods of TurtleScreen/Screen also exist as functions, i.e. as part of
the procedure-oriented interface.

@item 
@ref{3522,,RawTurtle} (alias: @ref{3523,,RawPen}) defines Turtle objects which draw
on a @ref{351f,,TurtleScreen}.  Its constructor needs a Canvas, ScrolledCanvas
or TurtleScreen as argument, so the RawTurtle objects know where to draw.

Derived from RawTurtle is the subclass @ref{3524,,Turtle} (alias: @code{Pen}),
which draws on “the” @ref{3521,,Screen} instance which is automatically
created, if not already present.

All methods of RawTurtle/Turtle also exist as functions, i.e. part of the
procedure-oriented interface.
@end enumerate

The procedural interface provides functions which are derived from the methods
of the classes @ref{3521,,Screen} and @ref{3524,,Turtle}.  They have the same names as
the corresponding methods.  A screen object is automatically created whenever a
function derived from a Screen method is called.  An (unnamed) turtle object is
automatically created whenever any of the functions derived from a Turtle method
is called.

To use multiple turtles on a screen one has to use the object-oriented interface.

@cartouche
@quotation Note 
In the following documentation the argument list for functions is given.
Methods, of course, have the additional first argument @emph{self} which is
omitted here.
@end quotation
@end cartouche

@node Overview of available Turtle and Screen methods,Methods of RawTurtle/Turtle and corresponding functions,Introduction<11>,turtle — Turtle graphics
@anchor{library/turtle overview-of-available-turtle-and-screen-methods}@anchor{3525}
@subsubsection Overview of available Turtle and Screen methods


@menu
* Turtle methods:: 
* Methods of TurtleScreen/Screen:: 

@end menu

@node Turtle methods,Methods of TurtleScreen/Screen,,Overview of available Turtle and Screen methods
@anchor{library/turtle turtle-methods}@anchor{3526}
@subsubsection Turtle methods



@table @asis

@item Turtle motion


@table @asis

@item Move and draw


@display
@ref{3527,,forward()} | @ref{3528,,fd()}@w{ }
@ref{3529,,backward()} | @ref{352a,,bk()} | @ref{352b,,back()}@w{ }
@ref{352c,,right()} | @ref{352d,,rt()}@w{ }
@ref{352e,,left()} | @ref{352f,,lt()}@w{ }
@ref{3530,,goto()} | @ref{3531,,setpos()} | @ref{3532,,setposition()}@w{ }
@ref{3533,,setx()}@w{ }
@ref{3534,,sety()}@w{ }
@ref{3535,,setheading()} | @ref{3536,,seth()}@w{ }
@ref{3537,,home()}@w{ }
@ref{3538,,circle()}@w{ }
@ref{3539,,dot()}@w{ }
@ref{353a,,stamp()}@w{ }
@ref{353b,,clearstamp()}@w{ }
@ref{353c,,clearstamps()}@w{ }
@ref{353d,,undo()}@w{ }
@ref{353e,,speed()}@w{ }
@end display



@item Tell Turtle’s state


@display
@ref{353f,,position()} | @ref{3540,,pos()}@w{ }
@ref{3541,,towards()}@w{ }
@ref{3542,,xcor()}@w{ }
@ref{3543,,ycor()}@w{ }
@ref{3544,,heading()}@w{ }
@ref{3545,,distance()}@w{ }
@end display



@item Setting and measurement


@display
@ref{3546,,degrees()}@w{ }
@ref{3547,,radians()}@w{ }
@end display


@end table

@item Pen control


@table @asis

@item Drawing state


@display
@ref{3548,,pendown()} | @ref{3549,,pd()} | @ref{354a,,down()}@w{ }
@ref{354b,,penup()} | @ref{354c,,pu()} | @ref{354d,,up()}@w{ }
@ref{354e,,pensize()} | @ref{354f,,width()}@w{ }
@ref{3550,,pen()}@w{ }
@ref{3551,,isdown()}@w{ }
@end display



@item Color control


@display
@ref{3552,,color()}@w{ }
@ref{3553,,pencolor()}@w{ }
@ref{3554,,fillcolor()}@w{ }
@end display



@item Filling


@display
@ref{3555,,filling()}@w{ }
@ref{3556,,begin_fill()}@w{ }
@ref{3557,,end_fill()}@w{ }
@end display



@item More drawing control


@display
@ref{3558,,reset()}@w{ }
@ref{3559,,clear()}@w{ }
@ref{355a,,write()}@w{ }
@end display


@end table

@item Turtle state


@table @asis

@item Visibility


@display
@ref{355b,,showturtle()} | @ref{355c,,st()}@w{ }
@ref{355d,,hideturtle()} | @ref{355e,,ht()}@w{ }
@ref{355f,,isvisible()}@w{ }
@end display



@item Appearance


@display
@ref{3560,,shape()}@w{ }
@ref{3561,,resizemode()}@w{ }
@ref{3562,,shapesize()} | @ref{3563,,turtlesize()}@w{ }
@ref{3564,,shearfactor()}@w{ }
@ref{3565,,settiltangle()}@w{ }
@ref{3566,,tiltangle()}@w{ }
@ref{3567,,tilt()}@w{ }
@ref{3568,,shapetransform()}@w{ }
@ref{3569,,get_shapepoly()}@w{ }
@end display


@end table

@item Using events


@display
@ref{356a,,onclick()}@w{ }
@ref{356b,,onrelease()}@w{ }
@ref{356c,,ondrag()}@w{ }
@end display



@item Special Turtle methods


@display
@ref{356d,,begin_poly()}@w{ }
@ref{356e,,end_poly()}@w{ }
@ref{356f,,get_poly()}@w{ }
@ref{3570,,clone()}@w{ }
@ref{3571,,getturtle()} | @ref{3572,,getpen()}@w{ }
@ref{3573,,getscreen()}@w{ }
@ref{3574,,setundobuffer()}@w{ }
@ref{3575,,undobufferentries()}@w{ }
@end display


@end table

@node Methods of TurtleScreen/Screen,,Turtle methods,Overview of available Turtle and Screen methods
@anchor{library/turtle methods-of-turtlescreen-screen}@anchor{3576}
@subsubsection Methods of TurtleScreen/Screen



@table @asis

@item Window control


@display
@ref{3577,,bgcolor()}@w{ }
@ref{3578,,bgpic()}@w{ }
@ref{3579,,clearscreen()}@w{ }
@ref{357a,,resetscreen()}@w{ }
@ref{357b,,screensize()}@w{ }
@ref{357c,,setworldcoordinates()}@w{ }
@end display



@item Animation control


@display
@ref{357d,,delay()}@w{ }
@ref{357e,,tracer()}@w{ }
@ref{357f,,update()}@w{ }
@end display



@item Using screen events


@display
@ref{3580,,listen()}@w{ }
@ref{3581,,onkey()} | @ref{3582,,onkeyrelease()}@w{ }
@ref{3583,,onkeypress()}@w{ }
@ref{356a,,onclick()} | @ref{3584,,onscreenclick()}@w{ }
@ref{3585,,ontimer()}@w{ }
@ref{3586,,mainloop()} | @ref{3587,,done()}@w{ }
@end display



@item Settings and special methods


@display
@ref{3588,,mode()}@w{ }
@ref{3589,,colormode()}@w{ }
@ref{358a,,getcanvas()}@w{ }
@ref{358b,,getshapes()}@w{ }
@ref{358c,,register_shape()} | @ref{358d,,addshape()}@w{ }
@ref{358e,,turtles()}@w{ }
@ref{358f,,window_height()}@w{ }
@ref{3590,,window_width()}@w{ }
@end display



@item Input methods


@display
@ref{1138,,textinput()}@w{ }
@ref{1139,,numinput()}@w{ }
@end display



@item Methods specific to Screen


@display
@ref{3591,,bye()}@w{ }
@ref{3592,,exitonclick()}@w{ }
@ref{3593,,setup()}@w{ }
@ref{3594,,title()}@w{ }
@end display


@end table

@node Methods of RawTurtle/Turtle and corresponding functions,Methods of TurtleScreen/Screen and corresponding functions,Overview of available Turtle and Screen methods,turtle — Turtle graphics
@anchor{library/turtle methods-of-rawturtle-turtle-and-corresponding-functions}@anchor{3595}
@subsubsection Methods of RawTurtle/Turtle and corresponding functions


Most of the examples in this section refer to a Turtle instance called
@code{turtle}.

@menu
* Turtle motion:: 
* Tell Turtle’s state:: 
* Settings for measurement:: 
* Pen control:: 
* Turtle state:: 
* Using events:: 
* Special Turtle methods:: 
* Compound shapes:: 

@end menu

@node Turtle motion,Tell Turtle’s state,,Methods of RawTurtle/Turtle and corresponding functions
@anchor{library/turtle turtle-motion}@anchor{3596}
@subsubsection Turtle motion


@geindex forward() (in module turtle)
@geindex fd() (in module turtle)
@anchor{library/turtle turtle forward}@anchor{3527}
@deffn {Function} turtle.forward (distance)
@anchor{library/turtle turtle fd}@anchor{3528}
@deffnx {Function} turtle.fd (distance)

@*Parameters: 
@code{distance} – a number (integer or float)


Move the turtle forward by the specified @emph{distance}, in the direction the
turtle is headed.

@example
>>> turtle.position()
(0.00,0.00)
>>> turtle.forward(25)
>>> turtle.position()
(25.00,0.00)
>>> turtle.forward(-75)
>>> turtle.position()
(-50.00,0.00)
@end example
@end deffn

@geindex back() (in module turtle)
@geindex bk() (in module turtle)
@geindex backward() (in module turtle)
@anchor{library/turtle turtle back}@anchor{352b}
@deffn {Function} turtle.back (distance)
@anchor{library/turtle turtle bk}@anchor{352a}
@deffnx {Function} turtle.bk (distance)
@anchor{library/turtle turtle backward}@anchor{3529}
@deffnx {Function} turtle.backward (distance)

@*Parameters: 
@code{distance} – a number


Move the turtle backward by @emph{distance}, opposite to the direction the
turtle is headed.  Do not change the turtle’s heading.

@c >>> turtle.goto(0, 0)

@example
>>> turtle.position()
(0.00,0.00)
>>> turtle.backward(30)
>>> turtle.position()
(-30.00,0.00)
@end example
@end deffn

@geindex right() (in module turtle)
@geindex rt() (in module turtle)
@anchor{library/turtle turtle right}@anchor{352c}
@deffn {Function} turtle.right (angle)
@anchor{library/turtle turtle rt}@anchor{352d}
@deffnx {Function} turtle.rt (angle)

@*Parameters: 
@code{angle} – a number (integer or float)


Turn turtle right by @emph{angle} units.  (Units are by default degrees, but
can be set via the @ref{3546,,degrees()} and @ref{3547,,radians()} functions.)  Angle
orientation depends on the turtle mode, see @ref{3588,,mode()}.

@c >>> turtle.setheading(22)

@example
>>> turtle.heading()
22.0
>>> turtle.right(45)
>>> turtle.heading()
337.0
@end example
@end deffn

@geindex left() (in module turtle)
@geindex lt() (in module turtle)
@anchor{library/turtle turtle left}@anchor{352e}
@deffn {Function} turtle.left (angle)
@anchor{library/turtle turtle lt}@anchor{352f}
@deffnx {Function} turtle.lt (angle)

@*Parameters: 
@code{angle} – a number (integer or float)


Turn turtle left by @emph{angle} units.  (Units are by default degrees, but
can be set via the @ref{3546,,degrees()} and @ref{3547,,radians()} functions.)  Angle
orientation depends on the turtle mode, see @ref{3588,,mode()}.

@c >>> turtle.setheading(22)

@example
>>> turtle.heading()
22.0
>>> turtle.left(45)
>>> turtle.heading()
67.0
@end example
@end deffn

@geindex goto() (in module turtle)
@geindex setpos() (in module turtle)
@geindex setposition() (in module turtle)
@anchor{library/turtle turtle goto}@anchor{3530}
@deffn {Function} turtle.goto (x, y=None)
@anchor{library/turtle turtle setpos}@anchor{3531}
@deffnx {Function} turtle.setpos (x, y=None)
@anchor{library/turtle turtle setposition}@anchor{3532}
@deffnx {Function} turtle.setposition (x, y=None)

@*Parameters: 

@itemize *

@item 
@code{x} – a number or a pair/vector of numbers

@item 
@code{y} – a number or @code{None}
@end itemize


If @emph{y} is @code{None}, @emph{x} must be a pair of coordinates or a @ref{3597,,Vec2D}
(e.g. as returned by @ref{3540,,pos()}).

Move turtle to an absolute position.  If the pen is down, draw line.  Do
not change the turtle’s orientation.

@c >>> turtle.goto(0, 0)

@example
 >>> tp = turtle.pos()
 >>> tp
 (0.00,0.00)
 >>> turtle.setpos(60,30)
 >>> turtle.pos()
 (60.00,30.00)
 >>> turtle.setpos((20,80))
 >>> turtle.pos()
 (20.00,80.00)
 >>> turtle.setpos(tp)
 >>> turtle.pos()
 (0.00,0.00)
@end example
@end deffn

@geindex setx() (in module turtle)
@anchor{library/turtle turtle setx}@anchor{3533}
@deffn {Function} turtle.setx (x)

@*Parameters: 
@code{x} – a number (integer or float)


Set the turtle’s first coordinate to @emph{x}, leave second coordinate
unchanged.

@c >>> turtle.goto(0, 240)

@example
>>> turtle.position()
(0.00,240.00)
>>> turtle.setx(10)
>>> turtle.position()
(10.00,240.00)
@end example
@end deffn

@geindex sety() (in module turtle)
@anchor{library/turtle turtle sety}@anchor{3534}
@deffn {Function} turtle.sety (y)

@*Parameters: 
@code{y} – a number (integer or float)


Set the turtle’s second coordinate to @emph{y}, leave first coordinate unchanged.

@c >>> turtle.goto(0, 40)

@example
>>> turtle.position()
(0.00,40.00)
>>> turtle.sety(-10)
>>> turtle.position()
(0.00,-10.00)
@end example
@end deffn

@geindex setheading() (in module turtle)
@geindex seth() (in module turtle)
@anchor{library/turtle turtle setheading}@anchor{3535}
@deffn {Function} turtle.setheading (to_angle)
@anchor{library/turtle turtle seth}@anchor{3536}
@deffnx {Function} turtle.seth (to_angle)

@*Parameters: 
@code{to_angle} – a number (integer or float)


Set the orientation of the turtle to @emph{to_angle}.  Here are some common
directions in degrees:


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxx} 
@headitem

standard mode

@tab

logo mode

@item

0 - east

@tab

0 - north

@item

90 - north

@tab

90 - east

@item

180 - west

@tab

180 - south

@item

270 - south

@tab

270 - west

@end multitable


@example
>>> turtle.setheading(90)
>>> turtle.heading()
90.0
@end example
@end deffn

@geindex home() (in module turtle)
@anchor{library/turtle turtle home}@anchor{3537}
@deffn {Function} turtle.home ()

Move turtle to the origin – coordinates (0,0) – and set its heading to
its start-orientation (which depends on the mode, see @ref{3588,,mode()}).

@c >>> turtle.setheading(90)
@c >>> turtle.goto(0, -10)

@example
>>> turtle.heading()
90.0
>>> turtle.position()
(0.00,-10.00)
>>> turtle.home()
>>> turtle.position()
(0.00,0.00)
>>> turtle.heading()
0.0
@end example
@end deffn

@geindex circle() (in module turtle)
@anchor{library/turtle turtle circle}@anchor{3538}
@deffn {Function} turtle.circle (radius, extent=None, steps=None)

@*Parameters: 

@itemize *

@item 
@code{radius} – a number

@item 
@code{extent} – a number (or @code{None})

@item 
@code{steps} – an integer (or @code{None})
@end itemize


Draw a circle with given @emph{radius}.  The center is @emph{radius} units left of
the turtle; @emph{extent} – an angle – determines which part of the circle
is drawn.  If @emph{extent} is not given, draw the entire circle.  If @emph{extent}
is not a full circle, one endpoint of the arc is the current pen
position.  Draw the arc in counterclockwise direction if @emph{radius} is
positive, otherwise in clockwise direction.  Finally the direction of the
turtle is changed by the amount of @emph{extent}.

As the circle is approximated by an inscribed regular polygon, @emph{steps}
determines the number of steps to use.  If not given, it will be
calculated automatically.  May be used to draw regular polygons.

@example
>>> turtle.home()
>>> turtle.position()
(0.00,0.00)
>>> turtle.heading()
0.0
>>> turtle.circle(50)
>>> turtle.position()
(-0.00,0.00)
>>> turtle.heading()
0.0
>>> turtle.circle(120, 180)  # draw a semicircle
>>> turtle.position()
(0.00,240.00)
>>> turtle.heading()
180.0
@end example
@end deffn

@geindex dot() (in module turtle)
@anchor{library/turtle turtle dot}@anchor{3539}
@deffn {Function} turtle.dot (size=None, *color)

@*Parameters: 

@itemize *

@item 
@code{size} – an integer >= 1 (if given)

@item 
@code{color} – a colorstring or a numeric color tuple
@end itemize


Draw a circular dot with diameter @emph{size}, using @emph{color}.  If @emph{size} is
not given, the maximum of pensize+4 and 2*pensize is used.

@example
>>> turtle.home()
>>> turtle.dot()
>>> turtle.fd(50); turtle.dot(20, "blue"); turtle.fd(50)
>>> turtle.position()
(100.00,-0.00)
>>> turtle.heading()
0.0
@end example
@end deffn

@geindex stamp() (in module turtle)
@anchor{library/turtle turtle stamp}@anchor{353a}
@deffn {Function} turtle.stamp ()

Stamp a copy of the turtle shape onto the canvas at the current turtle
position.  Return a stamp_id for that stamp, which can be used to delete
it by calling @code{clearstamp(stamp_id)}.

@example
>>> turtle.color("blue")
>>> turtle.stamp()
11
>>> turtle.fd(50)
@end example
@end deffn

@geindex clearstamp() (in module turtle)
@anchor{library/turtle turtle clearstamp}@anchor{353b}
@deffn {Function} turtle.clearstamp (stampid)

@*Parameters: 
@code{stampid} – an integer, must be return value of previous
@ref{353a,,stamp()} call


Delete stamp with given @emph{stampid}.

@example
>>> turtle.position()
(150.00,-0.00)
>>> turtle.color("blue")
>>> astamp = turtle.stamp()
>>> turtle.fd(50)
>>> turtle.position()
(200.00,-0.00)
>>> turtle.clearstamp(astamp)
>>> turtle.position()
(200.00,-0.00)
@end example
@end deffn

@geindex clearstamps() (in module turtle)
@anchor{library/turtle turtle clearstamps}@anchor{353c}
@deffn {Function} turtle.clearstamps (n=None)

@*Parameters: 
@code{n} – an integer (or @code{None})


Delete all or first/last @emph{n} of turtle’s stamps.  If @emph{n} is @code{None}, delete
all stamps, if @emph{n} > 0 delete first @emph{n} stamps, else if @emph{n} < 0 delete
last @emph{n} stamps.

@example
>>> for i in range(8):
...     turtle.stamp(); turtle.fd(30)
13
14
15
16
17
18
19
20
>>> turtle.clearstamps(2)
>>> turtle.clearstamps(-2)
>>> turtle.clearstamps()
@end example
@end deffn

@geindex undo() (in module turtle)
@anchor{library/turtle turtle undo}@anchor{353d}
@deffn {Function} turtle.undo ()

Undo (repeatedly) the last turtle action(s).  Number of available
undo actions is determined by the size of the undobuffer.

@example
>>> for i in range(4):
...     turtle.fd(50); turtle.lt(80)
...
>>> for i in range(8):
...     turtle.undo()
@end example
@end deffn

@geindex speed() (in module turtle)
@anchor{library/turtle turtle speed}@anchor{353e}
@deffn {Function} turtle.speed (speed=None)

@*Parameters: 
@code{speed} – an integer in the range 0..10 or a speedstring (see below)


Set the turtle’s speed to an integer value in the range 0..10.  If no
argument is given, return current speed.

If input is a number greater than 10 or smaller than 0.5, speed is set
to 0.  Speedstrings are mapped to speedvalues as follows:


@itemize *

@item 
“fastest”:  0

@item 
“fast”:  10

@item 
“normal”:  6

@item 
“slow”:  3

@item 
“slowest”:  1
@end itemize

Speeds from 1 to 10 enforce increasingly faster animation of line drawing
and turtle turning.

Attention: @emph{speed} = 0 means that @emph{no} animation takes
place. forward/back makes turtle jump and likewise left/right make the
turtle turn instantly.

@example
>>> turtle.speed()
3
>>> turtle.speed('normal')
>>> turtle.speed()
6
>>> turtle.speed(9)
>>> turtle.speed()
9
@end example
@end deffn

@node Tell Turtle’s state,Settings for measurement,Turtle motion,Methods of RawTurtle/Turtle and corresponding functions
@anchor{library/turtle tell-turtle-s-state}@anchor{3598}
@subsubsection Tell Turtle’s state


@geindex position() (in module turtle)
@geindex pos() (in module turtle)
@anchor{library/turtle turtle position}@anchor{353f}
@deffn {Function} turtle.position ()
@anchor{library/turtle turtle pos}@anchor{3540}
@deffnx {Function} turtle.pos ()

Return the turtle’s current location (x,y) (as a @ref{3597,,Vec2D} vector).

@example
>>> turtle.pos()
(440.00,-0.00)
@end example
@end deffn

@geindex towards() (in module turtle)
@anchor{library/turtle turtle towards}@anchor{3541}
@deffn {Function} turtle.towards (x, y=None)

@*Parameters: 

@itemize *

@item 
@code{x} – a number or a pair/vector of numbers or a turtle instance

@item 
@code{y} – a number if @emph{x} is a number, else @code{None}
@end itemize


Return the angle between the line from turtle position to position specified
by (x,y), the vector or the other turtle.  This depends on the turtle’s start
orientation which depends on the mode - “standard”/”world” or “logo”.

@example
>>> turtle.goto(10, 10)
>>> turtle.towards(0,0)
225.0
@end example
@end deffn

@geindex xcor() (in module turtle)
@anchor{library/turtle turtle xcor}@anchor{3542}
@deffn {Function} turtle.xcor ()

Return the turtle’s x coordinate.

@example
>>> turtle.home()
>>> turtle.left(50)
>>> turtle.forward(100)
>>> turtle.pos()
(64.28,76.60)
>>> print(round(turtle.xcor(), 5))
64.27876
@end example
@end deffn

@geindex ycor() (in module turtle)
@anchor{library/turtle turtle ycor}@anchor{3543}
@deffn {Function} turtle.ycor ()

Return the turtle’s y coordinate.

@example
>>> turtle.home()
>>> turtle.left(60)
>>> turtle.forward(100)
>>> print(turtle.pos())
(50.00,86.60)
>>> print(round(turtle.ycor(), 5))
86.60254
@end example
@end deffn

@geindex heading() (in module turtle)
@anchor{library/turtle turtle heading}@anchor{3544}
@deffn {Function} turtle.heading ()

Return the turtle’s current heading (value depends on the turtle mode, see
@ref{3588,,mode()}).

@example
>>> turtle.home()
>>> turtle.left(67)
>>> turtle.heading()
67.0
@end example
@end deffn

@geindex distance() (in module turtle)
@anchor{library/turtle turtle distance}@anchor{3545}
@deffn {Function} turtle.distance (x, y=None)

@*Parameters: 

@itemize *

@item 
@code{x} – a number or a pair/vector of numbers or a turtle instance

@item 
@code{y} – a number if @emph{x} is a number, else @code{None}
@end itemize


Return the distance from the turtle to (x,y), the given vector, or the given
other turtle, in turtle step units.

@example
>>> turtle.home()
>>> turtle.distance(30,40)
50.0
>>> turtle.distance((30,40))
50.0
>>> joe = Turtle()
>>> joe.forward(77)
>>> turtle.distance(joe)
77.0
@end example
@end deffn

@node Settings for measurement,Pen control,Tell Turtle’s state,Methods of RawTurtle/Turtle and corresponding functions
@anchor{library/turtle settings-for-measurement}@anchor{3599}
@subsubsection Settings for measurement


@geindex degrees() (in module turtle)
@anchor{library/turtle turtle degrees}@anchor{3546}
@deffn {Function} turtle.degrees (fullcircle=360.0)

@*Parameters: 
@code{fullcircle} – a number


Set angle measurement units, i.e. set number of “degrees” for a full circle.
Default value is 360 degrees.

@example
>>> turtle.home()
>>> turtle.left(90)
>>> turtle.heading()
90.0

Change angle measurement unit to grad (also known as gon,
grade, or gradian and equals 1/100-th of the right angle.)
>>> turtle.degrees(400.0)
>>> turtle.heading()
100.0
>>> turtle.degrees(360)
>>> turtle.heading()
90.0
@end example
@end deffn

@geindex radians() (in module turtle)
@anchor{library/turtle turtle radians}@anchor{3547}
@deffn {Function} turtle.radians ()

Set the angle measurement units to radians.  Equivalent to
@code{degrees(2*math.pi)}.

@example
>>> turtle.home()
>>> turtle.left(90)
>>> turtle.heading()
90.0
>>> turtle.radians()
>>> turtle.heading()
1.5707963267948966
@end example

@c >>> turtle.degrees(360)
@end deffn

@node Pen control,Turtle state,Settings for measurement,Methods of RawTurtle/Turtle and corresponding functions
@anchor{library/turtle pen-control}@anchor{359a}
@subsubsection Pen control


@menu
* Drawing state:: 
* Color control:: 
* Filling:: 
* More drawing control:: 

@end menu

@node Drawing state,Color control,,Pen control
@anchor{library/turtle drawing-state}@anchor{359b}
@subsubsection Drawing state


@geindex pendown() (in module turtle)
@geindex pd() (in module turtle)
@geindex down() (in module turtle)
@anchor{library/turtle turtle pendown}@anchor{3548}
@deffn {Function} turtle.pendown ()
@anchor{library/turtle turtle pd}@anchor{3549}
@deffnx {Function} turtle.pd ()
@anchor{library/turtle turtle down}@anchor{354a}
@deffnx {Function} turtle.down ()

Pull the pen down – drawing when moving.
@end deffn

@geindex penup() (in module turtle)
@geindex pu() (in module turtle)
@geindex up() (in module turtle)
@anchor{library/turtle turtle penup}@anchor{354b}
@deffn {Function} turtle.penup ()
@anchor{library/turtle turtle pu}@anchor{354c}
@deffnx {Function} turtle.pu ()
@anchor{library/turtle turtle up}@anchor{354d}
@deffnx {Function} turtle.up ()

Pull the pen up – no drawing when moving.
@end deffn

@geindex pensize() (in module turtle)
@geindex width() (in module turtle)
@anchor{library/turtle turtle pensize}@anchor{354e}
@deffn {Function} turtle.pensize (width=None)
@anchor{library/turtle turtle width}@anchor{354f}
@deffnx {Function} turtle.width (width=None)

@*Parameters: 
@code{width} – a positive number


Set the line thickness to @emph{width} or return it.  If resizemode is set to
“auto” and turtleshape is a polygon, that polygon is drawn with the same line
thickness.  If no argument is given, the current pensize is returned.

@example
>>> turtle.pensize()
1
>>> turtle.pensize(10)   # from here on lines of width 10 are drawn
@end example
@end deffn

@geindex pen() (in module turtle)
@anchor{library/turtle turtle pen}@anchor{3550}
@deffn {Function} turtle.pen (pen=None, **pendict)

@*Parameters: 

@itemize *

@item 
@code{pen} – a dictionary with some or all of the below listed keys

@item 
@code{pendict} – one or more keyword-arguments with the below listed keys as keywords
@end itemize


Return or set the pen’s attributes in a “pen-dictionary” with the following
key/value pairs:


@itemize *

@item 
“shown”: True/False

@item 
“pendown”: True/False

@item 
“pencolor”: color-string or color-tuple

@item 
“fillcolor”: color-string or color-tuple

@item 
“pensize”: positive number

@item 
“speed”: number in range 0..10

@item 
“resizemode”: “auto” or “user” or “noresize”

@item 
“stretchfactor”: (positive number, positive number)

@item 
“outline”: positive number

@item 
“tilt”: number
@end itemize

This dictionary can be used as argument for a subsequent call to @ref{3550,,pen()}
to restore the former pen-state.  Moreover one or more of these attributes
can be provided as keyword-arguments.  This can be used to set several pen
attributes in one statement.

@example
>>> turtle.pen(fillcolor="black", pencolor="red", pensize=10)
>>> sorted(turtle.pen().items())
[('fillcolor', 'black'), ('outline', 1), ('pencolor', 'red'),
 ('pendown', True), ('pensize', 10), ('resizemode', 'noresize'),
 ('shearfactor', 0.0), ('shown', True), ('speed', 9),
 ('stretchfactor', (1.0, 1.0)), ('tilt', 0.0)]
>>> penstate=turtle.pen()
>>> turtle.color("yellow", "")
>>> turtle.penup()
>>> sorted(turtle.pen().items())[:3]
[('fillcolor', '@w{'}), ('outline', 1), ('pencolor', 'yellow')]
>>> turtle.pen(penstate, fillcolor="green")
>>> sorted(turtle.pen().items())[:3]
[('fillcolor', 'green'), ('outline', 1), ('pencolor', 'red')]
@end example
@end deffn

@geindex isdown() (in module turtle)
@anchor{library/turtle turtle isdown}@anchor{3551}
@deffn {Function} turtle.isdown ()

Return @code{True} if pen is down, @code{False} if it’s up.

@example
>>> turtle.penup()
>>> turtle.isdown()
False
>>> turtle.pendown()
>>> turtle.isdown()
True
@end example
@end deffn

@node Color control,Filling,Drawing state,Pen control
@anchor{library/turtle color-control}@anchor{359c}
@subsubsection Color control


@geindex pencolor() (in module turtle)
@anchor{library/turtle turtle pencolor}@anchor{3553}
@deffn {Function} turtle.pencolor (*args)

Return or set the pencolor.

Four input formats are allowed:


@table @asis

@item @code{pencolor()}

Return the current pencolor as color specification string or
as a tuple (see example).  May be used as input to another
color/pencolor/fillcolor call.

@item @code{pencolor(colorstring)}

Set pencolor to @emph{colorstring}, which is a Tk color specification string,
such as @code{"red"}, @code{"yellow"}, or @code{"#33cc8c"}.

@item @code{pencolor((r, g, b))}

Set pencolor to the RGB color represented by the tuple of @emph{r}, @emph{g}, and
@emph{b}.  Each of @emph{r}, @emph{g}, and @emph{b} must be in the range 0..colormode, where
colormode is either 1.0 or 255 (see @ref{3589,,colormode()}).

@item @code{pencolor(r, g, b)}

Set pencolor to the RGB color represented by @emph{r}, @emph{g}, and @emph{b}.  Each of
@emph{r}, @emph{g}, and @emph{b} must be in the range 0..colormode.
@end table

If turtleshape is a polygon, the outline of that polygon is drawn with the
newly set pencolor.

@example
 >>> colormode()
 1.0
 >>> turtle.pencolor()
 'red'
 >>> turtle.pencolor("brown")
 >>> turtle.pencolor()
 'brown'
 >>> tup = (0.2, 0.8, 0.55)
 >>> turtle.pencolor(tup)
 >>> turtle.pencolor()
 (0.2, 0.8, 0.5490196078431373)
 >>> colormode(255)
 >>> turtle.pencolor()
 (51.0, 204.0, 140.0)
 >>> turtle.pencolor('#32c18f')
 >>> turtle.pencolor()
 (50.0, 193.0, 143.0)
@end example
@end deffn

@geindex fillcolor() (in module turtle)
@anchor{library/turtle turtle fillcolor}@anchor{3554}
@deffn {Function} turtle.fillcolor (*args)

Return or set the fillcolor.

Four input formats are allowed:


@table @asis

@item @code{fillcolor()}

Return the current fillcolor as color specification string, possibly
in tuple format (see example).  May be used as input to another
color/pencolor/fillcolor call.

@item @code{fillcolor(colorstring)}

Set fillcolor to @emph{colorstring}, which is a Tk color specification string,
such as @code{"red"}, @code{"yellow"}, or @code{"#33cc8c"}.

@item @code{fillcolor((r, g, b))}

Set fillcolor to the RGB color represented by the tuple of @emph{r}, @emph{g}, and
@emph{b}.  Each of @emph{r}, @emph{g}, and @emph{b} must be in the range 0..colormode, where
colormode is either 1.0 or 255 (see @ref{3589,,colormode()}).

@item @code{fillcolor(r, g, b)}

Set fillcolor to the RGB color represented by @emph{r}, @emph{g}, and @emph{b}.  Each of
@emph{r}, @emph{g}, and @emph{b} must be in the range 0..colormode.
@end table

If turtleshape is a polygon, the interior of that polygon is drawn
with the newly set fillcolor.

@example
 >>> turtle.fillcolor("violet")
 >>> turtle.fillcolor()
 'violet'
 >>> turtle.pencolor()
 (50.0, 193.0, 143.0)
 >>> turtle.fillcolor((50, 193, 143))  # Integers, not floats
 >>> turtle.fillcolor()
 (50.0, 193.0, 143.0)
 >>> turtle.fillcolor('#ffffff')
 >>> turtle.fillcolor()
 (255.0, 255.0, 255.0)
@end example
@end deffn

@geindex color() (in module turtle)
@anchor{library/turtle turtle color}@anchor{3552}
@deffn {Function} turtle.color (*args)

Return or set pencolor and fillcolor.

Several input formats are allowed.  They use 0 to 3 arguments as
follows:


@table @asis

@item @code{color()}

Return the current pencolor and the current fillcolor as a pair of color
specification strings or tuples as returned by @ref{3553,,pencolor()} and
@ref{3554,,fillcolor()}.

@item @code{color(colorstring)}, @code{color((r,g,b))}, @code{color(r,g,b)}

Inputs as in @ref{3553,,pencolor()}, set both, fillcolor and pencolor, to the
given value.

@item @code{color(colorstring1, colorstring2)}, @code{color((r1,g1,b1), (r2,g2,b2))}

Equivalent to @code{pencolor(colorstring1)} and @code{fillcolor(colorstring2)}
and analogously if the other input format is used.
@end table

If turtleshape is a polygon, outline and interior of that polygon is drawn
with the newly set colors.

@example
 >>> turtle.color("red", "green")
 >>> turtle.color()
 ('red', 'green')
 >>> color("#285078", "#a0c8f0")
 >>> color()
 ((40.0, 80.0, 120.0), (160.0, 200.0, 240.0))
@end example
@end deffn

See also: Screen method @ref{3589,,colormode()}.

@node Filling,More drawing control,Color control,Pen control
@anchor{library/turtle filling}@anchor{359d}
@subsubsection Filling


@c >>> turtle.home()

@geindex filling() (in module turtle)
@anchor{library/turtle turtle filling}@anchor{3555}
@deffn {Function} turtle.filling ()

Return fillstate (@code{True} if filling, @code{False} else).

@example
 >>> turtle.begin_fill()
 >>> if turtle.filling():
 ...    turtle.pensize(5)
 ... else:
 ...    turtle.pensize(3)
@end example
@end deffn

@geindex begin_fill() (in module turtle)
@anchor{library/turtle turtle begin_fill}@anchor{3556}
@deffn {Function} turtle.begin_fill ()

To be called just before drawing a shape to be filled.
@end deffn

@geindex end_fill() (in module turtle)
@anchor{library/turtle turtle end_fill}@anchor{3557}
@deffn {Function} turtle.end_fill ()

Fill the shape drawn after the last call to @ref{3556,,begin_fill()}.

Whether or not overlap regions for self-intersecting polygons
or multiple shapes are filled depends on the operating system graphics,
type of overlap, and number of overlaps.  For example, the Turtle star
above may be either all yellow or have some white regions.

@example
>>> turtle.color("black", "red")
>>> turtle.begin_fill()
>>> turtle.circle(80)
>>> turtle.end_fill()
@end example
@end deffn

@node More drawing control,,Filling,Pen control
@anchor{library/turtle more-drawing-control}@anchor{359e}
@subsubsection More drawing control


@geindex reset() (in module turtle)
@anchor{library/turtle turtle reset}@anchor{3558}
@deffn {Function} turtle.reset ()

Delete the turtle’s drawings from the screen, re-center the turtle and set
variables to the default values.

@example
>>> turtle.goto(0,-22)
>>> turtle.left(100)
>>> turtle.position()
(0.00,-22.00)
>>> turtle.heading()
100.0
>>> turtle.reset()
>>> turtle.position()
(0.00,0.00)
>>> turtle.heading()
0.0
@end example
@end deffn

@geindex clear() (in module turtle)
@anchor{library/turtle turtle clear}@anchor{3559}
@deffn {Function} turtle.clear ()

Delete the turtle’s drawings from the screen.  Do not move turtle.  State and
position of the turtle as well as drawings of other turtles are not affected.
@end deffn

@geindex write() (in module turtle)
@anchor{library/turtle turtle write}@anchor{355a}
@deffn {Function} turtle.write (arg, move=False, align='left', font=('Arial', 8, 'normal'))

@*Parameters: 

@itemize *

@item 
@code{arg} – object to be written to the TurtleScreen

@item 
@code{move} – True/False

@item 
@code{align} – one of the strings “left”, “center” or right”

@item 
@code{font} – a triple (fontname, fontsize, fonttype)
@end itemize


Write text - the string representation of @emph{arg} - at the current turtle
position according to @emph{align} (“left”, “center” or “right”) and with the given
font.  If @emph{move} is true, the pen is moved to the bottom-right corner of the
text.  By default, @emph{move} is @code{False}.

@example
>>> turtle.write("Home = ", True, align="center")
>>> turtle.write((0,0), True)
@end example
@end deffn

@node Turtle state,Using events,Pen control,Methods of RawTurtle/Turtle and corresponding functions
@anchor{library/turtle turtle-state}@anchor{359f}
@subsubsection Turtle state


@menu
* Visibility:: 
* Appearance:: 

@end menu

@node Visibility,Appearance,,Turtle state
@anchor{library/turtle visibility}@anchor{35a0}
@subsubsection Visibility


@geindex hideturtle() (in module turtle)
@geindex ht() (in module turtle)
@anchor{library/turtle turtle hideturtle}@anchor{355d}
@deffn {Function} turtle.hideturtle ()
@anchor{library/turtle turtle ht}@anchor{355e}
@deffnx {Function} turtle.ht ()

Make the turtle invisible.  It’s a good idea to do this while you’re in the
middle of doing some complex drawing, because hiding the turtle speeds up the
drawing observably.

@example
>>> turtle.hideturtle()
@end example
@end deffn

@geindex showturtle() (in module turtle)
@geindex st() (in module turtle)
@anchor{library/turtle turtle showturtle}@anchor{355b}
@deffn {Function} turtle.showturtle ()
@anchor{library/turtle turtle st}@anchor{355c}
@deffnx {Function} turtle.st ()

Make the turtle visible.

@example
>>> turtle.showturtle()
@end example
@end deffn

@geindex isvisible() (in module turtle)
@anchor{library/turtle turtle isvisible}@anchor{355f}
@deffn {Function} turtle.isvisible ()

Return @code{True} if the Turtle is shown, @code{False} if it’s hidden.

@example
>>> turtle.hideturtle()
>>> turtle.isvisible()
False
>>> turtle.showturtle()
>>> turtle.isvisible()
True
@end example
@end deffn

@node Appearance,,Visibility,Turtle state
@anchor{library/turtle appearance}@anchor{35a1}
@subsubsection Appearance


@geindex shape() (in module turtle)
@anchor{library/turtle turtle shape}@anchor{3560}
@deffn {Function} turtle.shape (name=None)

@*Parameters: 
@code{name} – a string which is a valid shapename


Set turtle shape to shape with given @emph{name} or, if name is not given, return
name of current shape.  Shape with @emph{name} must exist in the TurtleScreen’s
shape dictionary.  Initially there are the following polygon shapes: “arrow”,
“turtle”, “circle”, “square”, “triangle”, “classic”.  To learn about how to
deal with shapes see Screen method @ref{358c,,register_shape()}.

@example
>>> turtle.shape()
'classic'
>>> turtle.shape("turtle")
>>> turtle.shape()
'turtle'
@end example
@end deffn

@geindex resizemode() (in module turtle)
@anchor{library/turtle turtle resizemode}@anchor{3561}
@deffn {Function} turtle.resizemode (rmode=None)

@*Parameters: 
@code{rmode} – one of the strings “auto”, “user”, “noresize”


Set resizemode to one of the values: “auto”, “user”, “noresize”.  If @emph{rmode}
is not given, return current resizemode.  Different resizemodes have the
following effects:


@itemize -

@item 
“auto”: adapts the appearance of the turtle corresponding to the value of pensize.

@item 
“user”: adapts the appearance of the turtle according to the values of
stretchfactor and outlinewidth (outline), which are set by
@ref{3562,,shapesize()}.

@item 
“noresize”: no adaption of the turtle’s appearance takes place.
@end itemize

@code{resizemode("user")} is called by @ref{3562,,shapesize()} when used with arguments.

@example
>>> turtle.resizemode()
'noresize'
>>> turtle.resizemode("auto")
>>> turtle.resizemode()
'auto'
@end example
@end deffn

@geindex shapesize() (in module turtle)
@geindex turtlesize() (in module turtle)
@anchor{library/turtle turtle shapesize}@anchor{3562}
@deffn {Function} turtle.shapesize (stretch_wid=None, stretch_len=None, outline=None)
@anchor{library/turtle turtle turtlesize}@anchor{3563}
@deffnx {Function} turtle.turtlesize (stretch_wid=None, stretch_len=None, outline=None)

@*Parameters: 

@itemize *

@item 
@code{stretch_wid} – positive number

@item 
@code{stretch_len} – positive number

@item 
@code{outline} – positive number
@end itemize


Return or set the pen’s attributes x/y-stretchfactors and/or outline.  Set
resizemode to “user”.  If and only if resizemode is set to “user”, the turtle
will be displayed stretched according to its stretchfactors: @emph{stretch_wid} is
stretchfactor perpendicular to its orientation, @emph{stretch_len} is
stretchfactor in direction of its orientation, @emph{outline} determines the width
of the shapes’s outline.

@example
>>> turtle.shapesize()
(1.0, 1.0, 1)
>>> turtle.resizemode("user")
>>> turtle.shapesize(5, 5, 12)
>>> turtle.shapesize()
(5, 5, 12)
>>> turtle.shapesize(outline=8)
>>> turtle.shapesize()
(5, 5, 8)
@end example
@end deffn

@geindex shearfactor() (in module turtle)
@anchor{library/turtle turtle shearfactor}@anchor{3564}
@deffn {Function} turtle.shearfactor (shear=None)

@*Parameters: 
@code{shear} – number (optional)


Set or return the current shearfactor. Shear the turtleshape according to
the given shearfactor shear, which is the tangent of the shear angle.
Do @emph{not} change the turtle’s heading (direction of movement).
If shear is not given: return the current shearfactor, i. e. the
tangent of the shear angle, by which lines parallel to the
heading of the turtle are sheared.

@example
 >>> turtle.shape("circle")
 >>> turtle.shapesize(5,2)
 >>> turtle.shearfactor(0.5)
 >>> turtle.shearfactor()
 0.5
@end example
@end deffn

@geindex tilt() (in module turtle)
@anchor{library/turtle turtle tilt}@anchor{3567}
@deffn {Function} turtle.tilt (angle)

@*Parameters: 
@code{angle} – a number


Rotate the turtleshape by @emph{angle} from its current tilt-angle, but do @emph{not}
change the turtle’s heading (direction of movement).

@example
>>> turtle.reset()
>>> turtle.shape("circle")
>>> turtle.shapesize(5,2)
>>> turtle.tilt(30)
>>> turtle.fd(50)
>>> turtle.tilt(30)
>>> turtle.fd(50)
@end example
@end deffn

@geindex settiltangle() (in module turtle)
@anchor{library/turtle turtle settiltangle}@anchor{3565}
@deffn {Function} turtle.settiltangle (angle)

@*Parameters: 
@code{angle} – a number


Rotate the turtleshape to point in the direction specified by @emph{angle},
regardless of its current tilt-angle.  @emph{Do not} change the turtle’s heading
(direction of movement).

@example
>>> turtle.reset()
>>> turtle.shape("circle")
>>> turtle.shapesize(5,2)
>>> turtle.settiltangle(45)
>>> turtle.fd(50)
>>> turtle.settiltangle(-45)
>>> turtle.fd(50)
@end example


Deprecated since version 3.1.

@end deffn

@geindex tiltangle() (in module turtle)
@anchor{library/turtle turtle tiltangle}@anchor{3566}
@deffn {Function} turtle.tiltangle (angle=None)

@*Parameters: 
@code{angle} – a number (optional)


Set or return the current tilt-angle. If angle is given, rotate the
turtleshape to point in the direction specified by angle,
regardless of its current tilt-angle. Do @emph{not} change the turtle’s
heading (direction of movement).
If angle is not given: return the current tilt-angle, i. e. the angle
between the orientation of the turtleshape and the heading of the
turtle (its direction of movement).

@example
>>> turtle.reset()
>>> turtle.shape("circle")
>>> turtle.shapesize(5,2)
>>> turtle.tilt(45)
>>> turtle.tiltangle()
45.0
@end example
@end deffn

@geindex shapetransform() (in module turtle)
@anchor{library/turtle turtle shapetransform}@anchor{3568}
@deffn {Function} turtle.shapetransform (t11=None, t12=None, t21=None, t22=None)

@*Parameters: 

@itemize *

@item 
@code{t11} – a number (optional)

@item 
@code{t12} – a number (optional)

@item 
@code{t21} – a number (optional)

@item 
@code{t12} – a number (optional)
@end itemize


Set or return the current transformation matrix of the turtle shape.

If none of the matrix elements are given, return the transformation
matrix as a tuple of 4 elements.
Otherwise set the given elements and transform the turtleshape
according to the matrix consisting of first row t11, t12 and
second row t21, t22. The determinant t11 * t22 - t12 * t21 must not be
zero, otherwise an error is raised.
Modify stretchfactor, shearfactor and tiltangle according to the
given matrix.

@example
>>> turtle = Turtle()
>>> turtle.shape("square")
>>> turtle.shapesize(4,2)
>>> turtle.shearfactor(-0.5)
>>> turtle.shapetransform()
(4.0, -1.0, -0.0, 2.0)
@end example
@end deffn

@geindex get_shapepoly() (in module turtle)
@anchor{library/turtle turtle get_shapepoly}@anchor{3569}
@deffn {Function} turtle.get_shapepoly ()

Return the current shape polygon as tuple of coordinate pairs. This
can be used to define a new shape or components of a compound shape.

@example
>>> turtle.shape("square")
>>> turtle.shapetransform(4, -1, 0, 2)
>>> turtle.get_shapepoly()
((50, -20), (30, 20), (-50, 20), (-30, -20))
@end example
@end deffn

@node Using events,Special Turtle methods,Turtle state,Methods of RawTurtle/Turtle and corresponding functions
@anchor{library/turtle using-events}@anchor{35a2}
@subsubsection Using events



@deffn {Function} turtle.onclick (fun, btn=1, add=None)

@*Parameters: 

@itemize *

@item 
@code{fun} – a function with two arguments which will be called with the
coordinates of the clicked point on the canvas

@item 
@code{btn} – number of the mouse-button, defaults to 1 (left mouse button)

@item 
@code{add} – @code{True} or @code{False} – if @code{True}, a new binding will be
added, otherwise it will replace a former binding
@end itemize


Bind @emph{fun} to mouse-click events on this turtle.  If @emph{fun} is @code{None},
existing bindings are removed.  Example for the anonymous turtle, i.e. the
procedural way:

@example
>>> def turn(x, y):
...     left(180)
...
>>> onclick(turn)  # Now clicking into the turtle will turn it.
>>> onclick(None)  # event-binding will be removed
@end example
@end deffn

@geindex onrelease() (in module turtle)
@anchor{library/turtle turtle onrelease}@anchor{356b}
@deffn {Function} turtle.onrelease (fun, btn=1, add=None)

@*Parameters: 

@itemize *

@item 
@code{fun} – a function with two arguments which will be called with the
coordinates of the clicked point on the canvas

@item 
@code{btn} – number of the mouse-button, defaults to 1 (left mouse button)

@item 
@code{add} – @code{True} or @code{False} – if @code{True}, a new binding will be
added, otherwise it will replace a former binding
@end itemize


Bind @emph{fun} to mouse-button-release events on this turtle.  If @emph{fun} is
@code{None}, existing bindings are removed.

@example
>>> class MyTurtle(Turtle):
...     def glow(self,x,y):
...         self.fillcolor("red")
...     def unglow(self,x,y):
...         self.fillcolor("")
...
>>> turtle = MyTurtle()
>>> turtle.onclick(turtle.glow)     # clicking on turtle turns fillcolor red,
>>> turtle.onrelease(turtle.unglow) # releasing turns it to transparent.
@end example
@end deffn

@geindex ondrag() (in module turtle)
@anchor{library/turtle turtle ondrag}@anchor{356c}
@deffn {Function} turtle.ondrag (fun, btn=1, add=None)

@*Parameters: 

@itemize *

@item 
@code{fun} – a function with two arguments which will be called with the
coordinates of the clicked point on the canvas

@item 
@code{btn} – number of the mouse-button, defaults to 1 (left mouse button)

@item 
@code{add} – @code{True} or @code{False} – if @code{True}, a new binding will be
added, otherwise it will replace a former binding
@end itemize


Bind @emph{fun} to mouse-move events on this turtle.  If @emph{fun} is @code{None},
existing bindings are removed.

Remark: Every sequence of mouse-move-events on a turtle is preceded by a
mouse-click event on that turtle.

@example
>>> turtle.ondrag(turtle.goto)
@end example

Subsequently, clicking and dragging the Turtle will move it across
the screen thereby producing handdrawings (if pen is down).
@end deffn

@node Special Turtle methods,Compound shapes,Using events,Methods of RawTurtle/Turtle and corresponding functions
@anchor{library/turtle special-turtle-methods}@anchor{35a3}
@subsubsection Special Turtle methods


@geindex begin_poly() (in module turtle)
@anchor{library/turtle turtle begin_poly}@anchor{356d}
@deffn {Function} turtle.begin_poly ()

Start recording the vertices of a polygon.  Current turtle position is first
vertex of polygon.
@end deffn

@geindex end_poly() (in module turtle)
@anchor{library/turtle turtle end_poly}@anchor{356e}
@deffn {Function} turtle.end_poly ()

Stop recording the vertices of a polygon.  Current turtle position is last
vertex of polygon.  This will be connected with the first vertex.
@end deffn

@geindex get_poly() (in module turtle)
@anchor{library/turtle turtle get_poly}@anchor{356f}
@deffn {Function} turtle.get_poly ()

Return the last recorded polygon.

@example
>>> turtle.home()
>>> turtle.begin_poly()
>>> turtle.fd(100)
>>> turtle.left(20)
>>> turtle.fd(30)
>>> turtle.left(60)
>>> turtle.fd(50)
>>> turtle.end_poly()
>>> p = turtle.get_poly()
>>> register_shape("myFavouriteShape", p)
@end example
@end deffn

@geindex clone() (in module turtle)
@anchor{library/turtle turtle clone}@anchor{3570}
@deffn {Function} turtle.clone ()

Create and return a clone of the turtle with same position, heading and
turtle properties.

@example
>>> mick = Turtle()
>>> joe = mick.clone()
@end example
@end deffn

@geindex getturtle() (in module turtle)
@geindex getpen() (in module turtle)
@anchor{library/turtle turtle getturtle}@anchor{3571}
@deffn {Function} turtle.getturtle ()
@anchor{library/turtle turtle getpen}@anchor{3572}
@deffnx {Function} turtle.getpen ()

Return the Turtle object itself.  Only reasonable use: as a function to
return the “anonymous turtle”:

@example
>>> pet = getturtle()
>>> pet.fd(50)
>>> pet
<turtle.Turtle object at 0x...>
@end example
@end deffn

@geindex getscreen() (in module turtle)
@anchor{library/turtle turtle getscreen}@anchor{3573}
@deffn {Function} turtle.getscreen ()

Return the @ref{351f,,TurtleScreen} object the turtle is drawing on.
TurtleScreen methods can then be called for that object.

@example
>>> ts = turtle.getscreen()
>>> ts
<turtle._Screen object at 0x...>
>>> ts.bgcolor("pink")
@end example
@end deffn

@geindex setundobuffer() (in module turtle)
@anchor{library/turtle turtle setundobuffer}@anchor{3574}
@deffn {Function} turtle.setundobuffer (size)

@*Parameters: 
@code{size} – an integer or @code{None}


Set or disable undobuffer.  If @emph{size} is an integer, an empty undobuffer of
given size is installed.  @emph{size} gives the maximum number of turtle actions
that can be undone by the @ref{353d,,undo()} method/function.  If @emph{size} is
@code{None}, the undobuffer is disabled.

@example
>>> turtle.setundobuffer(42)
@end example
@end deffn

@geindex undobufferentries() (in module turtle)
@anchor{library/turtle turtle undobufferentries}@anchor{3575}
@deffn {Function} turtle.undobufferentries ()

Return number of entries in the undobuffer.

@example
>>> while undobufferentries():
...     undo()
@end example
@end deffn

@node Compound shapes,,Special Turtle methods,Methods of RawTurtle/Turtle and corresponding functions
@anchor{library/turtle compound-shapes}@anchor{35a4}@anchor{library/turtle compoundshapes}@anchor{35a5}
@subsubsection Compound shapes


To use compound turtle shapes, which consist of several polygons of different
color, you must use the helper class @ref{35a6,,Shape} explicitly as described
below:


@enumerate 

@item 
Create an empty Shape object of type “compound”.

@item 
Add as many components to this object as desired, using the
@code{addcomponent()} method.

For example:

@example
>>> s = Shape("compound")
>>> poly1 = ((0,0),(10,-5),(0,10),(-10,-5))
>>> s.addcomponent(poly1, "red", "blue")
>>> poly2 = ((0,0),(10,-5),(-10,-5))
>>> s.addcomponent(poly2, "blue", "red")
@end example

@item 
Now add the Shape to the Screen’s shapelist and use it:

@example
>>> register_shape("myshape", s)
>>> shape("myshape")
@end example
@end enumerate

@cartouche
@quotation Note 
The @ref{35a6,,Shape} class is used internally by the @ref{358c,,register_shape()}
method in different ways.  The application programmer has to deal with the
Shape class @emph{only} when using compound shapes like shown above!
@end quotation
@end cartouche

@node Methods of TurtleScreen/Screen and corresponding functions,Public classes,Methods of RawTurtle/Turtle and corresponding functions,turtle — Turtle graphics
@anchor{library/turtle methods-of-turtlescreen-screen-and-corresponding-functions}@anchor{35a7}
@subsubsection Methods of TurtleScreen/Screen and corresponding functions


Most of the examples in this section refer to a TurtleScreen instance called
@code{screen}.

@c >>> screen = Screen()

@menu
* Window control:: 
* Animation control:: 
* Using screen events:: 
* Input methods:: 
* Settings and special methods:: 
* Methods specific to Screen@comma{} not inherited from TurtleScreen: Methods specific to Screen not inherited from TurtleScreen. 

@end menu

@node Window control,Animation control,,Methods of TurtleScreen/Screen and corresponding functions
@anchor{library/turtle window-control}@anchor{35a8}
@subsubsection Window control


@geindex bgcolor() (in module turtle)
@anchor{library/turtle turtle bgcolor}@anchor{3577}
@deffn {Function} turtle.bgcolor (*args)

@*Parameters: 
@code{args} – a color string or three numbers in the range 0..colormode or a
3-tuple of such numbers


Set or return background color of the TurtleScreen.

@example
>>> screen.bgcolor("orange")
>>> screen.bgcolor()
'orange'
>>> screen.bgcolor("#800080")
>>> screen.bgcolor()
(128.0, 0.0, 128.0)
@end example
@end deffn

@geindex bgpic() (in module turtle)
@anchor{library/turtle turtle bgpic}@anchor{3578}
@deffn {Function} turtle.bgpic (picname=None)

@*Parameters: 
@code{picname} – a string, name of a gif-file or @code{"nopic"}, or @code{None}


Set background image or return name of current backgroundimage.  If @emph{picname}
is a filename, set the corresponding image as background.  If @emph{picname} is
@code{"nopic"}, delete background image, if present.  If @emph{picname} is @code{None},
return the filename of the current backgroundimage.

@example
>>> screen.bgpic()
'nopic'
>>> screen.bgpic("landscape.gif")
>>> screen.bgpic()
"landscape.gif"
@end example
@end deffn


@deffn {Function} turtle.clear ()

@cartouche
@quotation Note 
This TurtleScreen method is available as a global function only under the
name @code{clearscreen}.  The global function @code{clear} is a different one
derived from the Turtle method @code{clear}.
@end quotation
@end cartouche
@end deffn

@geindex clearscreen() (in module turtle)
@anchor{library/turtle turtle clearscreen}@anchor{3579}
@deffn {Function} turtle.clearscreen ()

Delete all drawings and all turtles from the TurtleScreen.  Reset the now
empty TurtleScreen to its initial state: white background, no background
image, no event bindings and tracing on.
@end deffn


@deffn {Function} turtle.reset ()

@cartouche
@quotation Note 
This TurtleScreen method is available as a global function only under the
name @code{resetscreen}.  The global function @code{reset} is another one
derived from the Turtle method @code{reset}.
@end quotation
@end cartouche
@end deffn

@geindex resetscreen() (in module turtle)
@anchor{library/turtle turtle resetscreen}@anchor{357a}
@deffn {Function} turtle.resetscreen ()

Reset all Turtles on the Screen to their initial state.
@end deffn

@geindex screensize() (in module turtle)
@anchor{library/turtle turtle screensize}@anchor{357b}
@deffn {Function} turtle.screensize (canvwidth=None, canvheight=None, bg=None)

@*Parameters: 

@itemize *

@item 
@code{canvwidth} – positive integer, new width of canvas in pixels

@item 
@code{canvheight} – positive integer, new height of canvas in pixels

@item 
@code{bg} – colorstring or color-tuple, new background color
@end itemize


If no arguments are given, return current (canvaswidth, canvasheight).  Else
resize the canvas the turtles are drawing on.  Do not alter the drawing
window.  To observe hidden parts of the canvas, use the scrollbars. With this
method, one can make visible those parts of a drawing which were outside the
canvas before.

@example
>>> screen.screensize()
(400, 300)
>>> screen.screensize(2000,1500)
>>> screen.screensize()
(2000, 1500)
@end example

e.g. to search for an erroneously escaped turtle ;-)
@end deffn

@geindex setworldcoordinates() (in module turtle)
@anchor{library/turtle turtle setworldcoordinates}@anchor{357c}
@deffn {Function} turtle.setworldcoordinates (llx, lly, urx, ury)

@*Parameters: 

@itemize *

@item 
@code{llx} – a number, x-coordinate of lower left corner of canvas

@item 
@code{lly} – a number, y-coordinate of lower left corner of canvas

@item 
@code{urx} – a number, x-coordinate of upper right corner of canvas

@item 
@code{ury} – a number, y-coordinate of upper right corner of canvas
@end itemize


Set up user-defined coordinate system and switch to mode “world” if
necessary.  This performs a @code{screen.reset()}.  If mode “world” is already
active, all drawings are redrawn according to the new coordinates.

@strong{ATTENTION}: in user-defined coordinate systems angles may appear
distorted.

@example
>>> screen.reset()
>>> screen.setworldcoordinates(-50,-7.5,50,7.5)
>>> for _ in range(72):
...     left(10)
...
>>> for _ in range(8):
...     left(45); fd(2)   # a regular octagon
@end example

@c >>> screen.reset()
@c >>> for t in turtles():
@c ...      t.reset()
@end deffn

@node Animation control,Using screen events,Window control,Methods of TurtleScreen/Screen and corresponding functions
@anchor{library/turtle animation-control}@anchor{35a9}
@subsubsection Animation control


@geindex delay() (in module turtle)
@anchor{library/turtle turtle delay}@anchor{357d}
@deffn {Function} turtle.delay (delay=None)

@*Parameters: 
@code{delay} – positive integer


Set or return the drawing @emph{delay} in milliseconds.  (This is approximately
the time interval between two consecutive canvas updates.)  The longer the
drawing delay, the slower the animation.

Optional argument:

@example
>>> screen.delay()
10
>>> screen.delay(5)
>>> screen.delay()
5
@end example
@end deffn

@geindex tracer() (in module turtle)
@anchor{library/turtle turtle tracer}@anchor{357e}
@deffn {Function} turtle.tracer (n=None, delay=None)

@*Parameters: 

@itemize *

@item 
@code{n} – nonnegative integer

@item 
@code{delay} – nonnegative integer
@end itemize


Turn turtle animation on/off and set delay for update drawings.  If
@emph{n} is given, only each n-th regular screen update is really
performed.  (Can be used to accelerate the drawing of complex
graphics.)  When called without arguments, returns the currently
stored value of n. Second argument sets delay value (see
@ref{357d,,delay()}).

@example
>>> screen.tracer(8, 25)
>>> dist = 2
>>> for i in range(200):
...     fd(dist)
...     rt(90)
...     dist += 2
@end example
@end deffn

@geindex update() (in module turtle)
@anchor{library/turtle turtle update}@anchor{357f}
@deffn {Function} turtle.update ()

Perform a TurtleScreen update. To be used when tracer is turned off.
@end deffn

See also the RawTurtle/Turtle method @ref{353e,,speed()}.

@node Using screen events,Input methods,Animation control,Methods of TurtleScreen/Screen and corresponding functions
@anchor{library/turtle using-screen-events}@anchor{35aa}
@subsubsection Using screen events


@geindex listen() (in module turtle)
@anchor{library/turtle turtle listen}@anchor{3580}
@deffn {Function} turtle.listen (xdummy=None, ydummy=None)

Set focus on TurtleScreen (in order to collect key-events).  Dummy arguments
are provided in order to be able to pass @ref{3580,,listen()} to the onclick method.
@end deffn

@geindex onkey() (in module turtle)
@geindex onkeyrelease() (in module turtle)
@anchor{library/turtle turtle onkey}@anchor{3581}
@deffn {Function} turtle.onkey (fun, key)
@anchor{library/turtle turtle onkeyrelease}@anchor{3582}
@deffnx {Function} turtle.onkeyrelease (fun, key)

@*Parameters: 

@itemize *

@item 
@code{fun} – a function with no arguments or @code{None}

@item 
@code{key} – a string: key (e.g. “a”) or key-symbol (e.g. “space”)
@end itemize


Bind @emph{fun} to key-release event of key.  If @emph{fun} is @code{None}, event bindings
are removed. Remark: in order to be able to register key-events, TurtleScreen
must have the focus. (See method @ref{3580,,listen()}.)

@example
>>> def f():
...     fd(50)
...     lt(60)
...
>>> screen.onkey(f, "Up")
>>> screen.listen()
@end example
@end deffn

@geindex onkeypress() (in module turtle)
@anchor{library/turtle turtle onkeypress}@anchor{3583}
@deffn {Function} turtle.onkeypress (fun, key=None)

@*Parameters: 

@itemize *

@item 
@code{fun} – a function with no arguments or @code{None}

@item 
@code{key} – a string: key (e.g. “a”) or key-symbol (e.g. “space”)
@end itemize


Bind @emph{fun} to key-press event of key if key is given,
or to any key-press-event if no key is given.
Remark: in order to be able to register key-events, TurtleScreen
must have focus. (See method @ref{3580,,listen()}.)

@example
>>> def f():
...     fd(50)
...
>>> screen.onkey(f, "Up")
>>> screen.listen()
@end example
@end deffn

@geindex onclick() (in module turtle)
@geindex onscreenclick() (in module turtle)
@anchor{library/turtle turtle onclick}@anchor{356a}
@deffn {Function} turtle.onclick (fun, btn=1, add=None)
@anchor{library/turtle turtle onscreenclick}@anchor{3584}
@deffnx {Function} turtle.onscreenclick (fun, btn=1, add=None)

@*Parameters: 

@itemize *

@item 
@code{fun} – a function with two arguments which will be called with the
coordinates of the clicked point on the canvas

@item 
@code{btn} – number of the mouse-button, defaults to 1 (left mouse button)

@item 
@code{add} – @code{True} or @code{False} – if @code{True}, a new binding will be
added, otherwise it will replace a former binding
@end itemize


Bind @emph{fun} to mouse-click events on this screen.  If @emph{fun} is @code{None},
existing bindings are removed.

Example for a TurtleScreen instance named @code{screen} and a Turtle instance
named @code{turtle}:

@example
>>> screen.onclick(turtle.goto) # Subsequently clicking into the TurtleScreen will
>>>                             # make the turtle move to the clicked point.
>>> screen.onclick(None)        # remove event binding again
@end example

@cartouche
@quotation Note 
This TurtleScreen method is available as a global function only under the
name @code{onscreenclick}.  The global function @code{onclick} is another one
derived from the Turtle method @code{onclick}.
@end quotation
@end cartouche
@end deffn

@geindex ontimer() (in module turtle)
@anchor{library/turtle turtle ontimer}@anchor{3585}
@deffn {Function} turtle.ontimer (fun, t=0)

@*Parameters: 

@itemize *

@item 
@code{fun} – a function with no arguments

@item 
@code{t} – a number >= 0
@end itemize


Install a timer that calls @emph{fun} after @emph{t} milliseconds.

@example
>>> running = True
>>> def f():
...     if running:
...         fd(50)
...         lt(60)
...         screen.ontimer(f, 250)
>>> f()   ### makes the turtle march around
>>> running = False
@end example
@end deffn

@geindex mainloop() (in module turtle)
@geindex done() (in module turtle)
@anchor{library/turtle turtle mainloop}@anchor{3586}
@deffn {Function} turtle.mainloop ()
@anchor{library/turtle turtle done}@anchor{3587}
@deffnx {Function} turtle.done ()

Starts event loop - calling Tkinter’s mainloop function.
Must be the last statement in a turtle graphics program.
Must @emph{not} be used if a script is run from within IDLE in -n mode
(No subprocess) - for interactive use of turtle graphics.

@example
>>> screen.mainloop()
@end example
@end deffn

@node Input methods,Settings and special methods,Using screen events,Methods of TurtleScreen/Screen and corresponding functions
@anchor{library/turtle input-methods}@anchor{35ab}
@subsubsection Input methods


@geindex textinput() (in module turtle)
@anchor{library/turtle turtle textinput}@anchor{1138}
@deffn {Function} turtle.textinput (title, prompt)

@*Parameters: 

@itemize *

@item 
@code{title} – string

@item 
@code{prompt} – string
@end itemize


Pop up a dialog window for input of a string. Parameter title is
the title of the dialog window, prompt is a text mostly describing
what information to input.
Return the string input. If the dialog is canceled, return @code{None}.

@example
>>> screen.textinput("NIM", "Name of first player:")
@end example
@end deffn

@geindex numinput() (in module turtle)
@anchor{library/turtle turtle numinput}@anchor{1139}
@deffn {Function} turtle.numinput (title, prompt, default=None, minval=None, maxval=None)

@*Parameters: 

@itemize *

@item 
@code{title} – string

@item 
@code{prompt} – string

@item 
@code{default} – number (optional)

@item 
@code{minval} – number (optional)

@item 
@code{maxval} – number (optional)
@end itemize


Pop up a dialog window for input of a number. title is the title of the
dialog window, prompt is a text mostly describing what numerical information
to input. default: default value, minval: minimum value for input,
maxval: maximum value for input.
The number input must be in the range minval .. maxval if these are
given. If not, a hint is issued and the dialog remains open for
correction.
Return the number input. If the dialog is canceled,  return @code{None}.

@example
>>> screen.numinput("Poker", "Your stakes:", 1000, minval=10, maxval=10000)
@end example
@end deffn

@node Settings and special methods,Methods specific to Screen not inherited from TurtleScreen,Input methods,Methods of TurtleScreen/Screen and corresponding functions
@anchor{library/turtle settings-and-special-methods}@anchor{35ac}
@subsubsection Settings and special methods


@geindex mode() (in module turtle)
@anchor{library/turtle turtle mode}@anchor{3588}
@deffn {Function} turtle.mode (mode=None)

@*Parameters: 
@code{mode} – one of the strings “standard”, “logo” or “world”


Set turtle mode (“standard”, “logo” or “world”) and perform reset.  If mode
is not given, current mode is returned.

Mode “standard” is compatible with old @ref{11d,,turtle}.  Mode “logo” is
compatible with most Logo turtle graphics.  Mode “world” uses user-defined
“world coordinates”. @strong{Attention}: in this mode angles appear distorted if
@code{x/y} unit-ratio doesn’t equal 1.


@multitable {xxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxx} 
@headitem

Mode

@tab

Initial turtle heading

@tab

positive angles

@item

“standard”

@tab

to the right (east)

@tab

counterclockwise

@item

“logo”

@tab

upward    (north)

@tab

clockwise

@end multitable


@example
>>> mode("logo")   # resets turtle heading to north
>>> mode()
'logo'
@end example
@end deffn

@geindex colormode() (in module turtle)
@anchor{library/turtle turtle colormode}@anchor{3589}
@deffn {Function} turtle.colormode (cmode=None)

@*Parameters: 
@code{cmode} – one of the values 1.0 or 255


Return the colormode or set it to 1.0 or 255.  Subsequently @emph{r}, @emph{g}, @emph{b}
values of color triples have to be in the range 0..*cmode*.

@example
>>> screen.colormode(1)
>>> turtle.pencolor(240, 160, 80)
Traceback (most recent call last):
     ...
TurtleGraphicsError: bad color sequence: (240, 160, 80)
>>> screen.colormode()
1.0
>>> screen.colormode(255)
>>> screen.colormode()
255
>>> turtle.pencolor(240,160,80)
@end example
@end deffn

@geindex getcanvas() (in module turtle)
@anchor{library/turtle turtle getcanvas}@anchor{358a}
@deffn {Function} turtle.getcanvas ()

Return the Canvas of this TurtleScreen.  Useful for insiders who know what to
do with a Tkinter Canvas.

@example
>>> cv = screen.getcanvas()
>>> cv
<turtle.ScrolledCanvas object ...>
@end example
@end deffn

@geindex getshapes() (in module turtle)
@anchor{library/turtle turtle getshapes}@anchor{358b}
@deffn {Function} turtle.getshapes ()

Return a list of names of all currently available turtle shapes.

@example
>>> screen.getshapes()
['arrow', 'blank', 'circle', ..., 'turtle']
@end example
@end deffn

@geindex register_shape() (in module turtle)
@geindex addshape() (in module turtle)
@anchor{library/turtle turtle register_shape}@anchor{358c}
@deffn {Function} turtle.register_shape (name, shape=None)
@anchor{library/turtle turtle addshape}@anchor{358d}
@deffnx {Function} turtle.addshape (name, shape=None)

There are three different ways to call this function:


@enumerate 

@item 
@emph{name} is the name of a gif-file and @emph{shape} is @code{None}: Install the
corresponding image shape.

@example
>>> screen.register_shape("turtle.gif")
@end example

@cartouche
@quotation Note 
Image shapes @emph{do not} rotate when turning the turtle, so they do not
display the heading of the turtle!
@end quotation
@end cartouche

@item 
@emph{name} is an arbitrary string and @emph{shape} is a tuple of pairs of
coordinates: Install the corresponding polygon shape.

@example
>>> screen.register_shape("triangle", ((5,-3), (0,5), (-5,-3)))
@end example

@item 
@emph{name} is an arbitrary string and @emph{shape} is a (compound) @ref{35a6,,Shape}
object: Install the corresponding compound shape.
@end enumerate

Add a turtle shape to TurtleScreen’s shapelist.  Only thusly registered
shapes can be used by issuing the command @code{shape(shapename)}.
@end deffn

@geindex turtles() (in module turtle)
@anchor{library/turtle turtle turtles}@anchor{358e}
@deffn {Function} turtle.turtles ()

Return the list of turtles on the screen.

@example
>>> for turtle in screen.turtles():
...     turtle.color("red")
@end example
@end deffn

@geindex window_height() (in module turtle)
@anchor{library/turtle turtle window_height}@anchor{358f}
@deffn {Function} turtle.window_height ()

Return the height of the turtle window.

@example
>>> screen.window_height()
480
@end example
@end deffn

@geindex window_width() (in module turtle)
@anchor{library/turtle turtle window_width}@anchor{3590}
@deffn {Function} turtle.window_width ()

Return the width of the turtle window.

@example
>>> screen.window_width()
640
@end example
@end deffn

@node Methods specific to Screen not inherited from TurtleScreen,,Settings and special methods,Methods of TurtleScreen/Screen and corresponding functions
@anchor{library/turtle methods-specific-to-screen-not-inherited-from-turtlescreen}@anchor{35ad}@anchor{library/turtle screenspecific}@anchor{35ae}
@subsubsection Methods specific to Screen, not inherited from TurtleScreen


@geindex bye() (in module turtle)
@anchor{library/turtle turtle bye}@anchor{3591}
@deffn {Function} turtle.bye ()

Shut the turtlegraphics window.
@end deffn

@geindex exitonclick() (in module turtle)
@anchor{library/turtle turtle exitonclick}@anchor{3592}
@deffn {Function} turtle.exitonclick ()

Bind @code{bye()} method to mouse clicks on the Screen.

If the value “using_IDLE” in the configuration dictionary is @code{False}
(default value), also enter mainloop.  Remark: If IDLE with the @code{-n} switch
(no subprocess) is used, this value should be set to @code{True} in
@code{turtle.cfg}.  In this case IDLE’s own mainloop is active also for the
client script.
@end deffn

@geindex setup() (in module turtle)
@anchor{library/turtle turtle setup}@anchor{3593}
@deffn {Function} turtle.setup (width=_CFG['width'], height=_CFG['height'], startx=_CFG['leftright'], starty=_CFG['topbottom'])

Set the size and position of the main window.  Default values of arguments
are stored in the configuration dictionary and can be changed via a
@code{turtle.cfg} file.

@*Parameters: 

@itemize *

@item 
@code{width} – if an integer, a size in pixels, if a float, a fraction of the
screen; default is 50% of screen

@item 
@code{height} – if an integer, the height in pixels, if a float, a fraction of
the screen; default is 75% of screen

@item 
@code{startx} – if positive, starting position in pixels from the left
edge of the screen, if negative from the right edge, if @code{None},
center window horizontally

@item 
@code{starty} – if positive, starting position in pixels from the top
edge of the screen, if negative from the bottom edge, if @code{None},
center window vertically
@end itemize


@example
>>> screen.setup (width=200, height=200, startx=0, starty=0)
>>>              # sets window to 200x200 pixels, in upper left of screen
>>> screen.setup(width=.75, height=0.5, startx=None, starty=None)
>>>              # sets window to 75% of screen by 50% of screen and centers
@end example
@end deffn

@geindex title() (in module turtle)
@anchor{library/turtle turtle title}@anchor{3594}
@deffn {Function} turtle.title (titlestring)

@*Parameters: 
@code{titlestring} – a string that is shown in the titlebar of the turtle
graphics window


Set title of turtle window to @emph{titlestring}.

@example
>>> screen.title("Welcome to the turtle zoo!")
@end example
@end deffn

@node Public classes,Help and configuration,Methods of TurtleScreen/Screen and corresponding functions,turtle — Turtle graphics
@anchor{library/turtle public-classes}@anchor{35af}
@subsubsection Public classes


@geindex RawTurtle (class in turtle)
@geindex RawPen (class in turtle)
@anchor{library/turtle turtle RawTurtle}@anchor{3522}
@deffn {Class} turtle.RawTurtle (canvas)
@anchor{library/turtle turtle RawPen}@anchor{3523}
@deffnx {Class} turtle.RawPen (canvas)

@*Parameters: 
@code{canvas} – a @code{tkinter.Canvas}, a @ref{3520,,ScrolledCanvas} or a
@ref{351f,,TurtleScreen}


Create a turtle.  The turtle has all methods described above as “methods of
Turtle/RawTurtle”.
@end deffn

@geindex Turtle (class in turtle)
@anchor{library/turtle turtle Turtle}@anchor{3524}
@deffn {Class} turtle.Turtle

Subclass of RawTurtle, has the same interface but draws on a default
@ref{3521,,Screen} object created automatically when needed for the first time.
@end deffn

@geindex TurtleScreen (class in turtle)
@anchor{library/turtle turtle TurtleScreen}@anchor{351f}
@deffn {Class} turtle.TurtleScreen (cv)

@*Parameters: 
@code{cv} – a @code{tkinter.Canvas}


Provides screen oriented methods like @code{setbg()} etc. that are described
above.
@end deffn

@geindex Screen (class in turtle)
@anchor{library/turtle turtle Screen}@anchor{3521}
@deffn {Class} turtle.Screen

Subclass of TurtleScreen, with @ref{35ae,,four methods added}.
@end deffn

@geindex ScrolledCanvas (class in turtle)
@anchor{library/turtle turtle ScrolledCanvas}@anchor{3520}
@deffn {Class} turtle.ScrolledCanvas (master)

@*Parameters: 
@code{master} – some Tkinter widget to contain the ScrolledCanvas, i.e.
a Tkinter-canvas with scrollbars added


Used by class Screen, which thus automatically provides a ScrolledCanvas as
playground for the turtles.
@end deffn

@geindex Shape (class in turtle)
@anchor{library/turtle turtle Shape}@anchor{35a6}
@deffn {Class} turtle.Shape (type_, data)

@*Parameters: 
@code{type_} – one of the strings “polygon”, “image”, “compound”


Data structure modeling shapes.  The pair @code{(type_, data)} must follow this
specification:


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

@emph{type_}

@tab

@emph{data}

@item

“polygon”

@tab

a polygon-tuple, i.e. a tuple of pairs of coordinates

@item

“image”

@tab

an image  (in this form only used internally!)

@item

“compound”

@tab

@code{None} (a compound shape has to be constructed using the
@ref{35b0,,addcomponent()} method)

@end multitable


@geindex addcomponent() (turtle.Shape method)
@anchor{library/turtle turtle Shape addcomponent}@anchor{35b0}
@deffn {Method} addcomponent (poly, fill, outline=None)

@*Parameters: 

@itemize *

@item 
@code{poly} – a polygon, i.e. a tuple of pairs of numbers

@item 
@code{fill} – a color the @emph{poly} will be filled with

@item 
@code{outline} – a color for the poly’s outline (if given)
@end itemize


Example:

@example
>>> poly = ((0,0),(10,-5),(0,10),(-10,-5))
>>> s = Shape("compound")
>>> s.addcomponent(poly, "red", "blue")
>>> # ... add more components and then use register_shape()
@end example

See @ref{35a5,,Compound shapes}.
@end deffn
@end deffn

@geindex Vec2D (class in turtle)
@anchor{library/turtle turtle Vec2D}@anchor{3597}
@deffn {Class} turtle.Vec2D (x, y)

A two-dimensional vector class, used as a helper class for implementing
turtle graphics.  May be useful for turtle graphics programs too.  Derived
from tuple, so a vector is a tuple!

Provides (for @emph{a}, @emph{b} vectors, @emph{k} number):


@itemize *

@item 
@code{a + b} vector addition

@item 
@code{a - b} vector subtraction

@item 
@code{a * b} inner product

@item 
@code{k * a} and @code{a * k} multiplication with scalar

@item 
@code{abs(a)} absolute value of a

@item 
@code{a.rotate(angle)} rotation
@end itemize
@end deffn

@node Help and configuration,turtledemo — Demo scripts,Public classes,turtle — Turtle graphics
@anchor{library/turtle help-and-configuration}@anchor{35b1}
@subsubsection Help and configuration


@menu
* How to use help:: 
* Translation of docstrings into different languages:: 
* How to configure Screen and Turtles:: 

@end menu

@node How to use help,Translation of docstrings into different languages,,Help and configuration
@anchor{library/turtle how-to-use-help}@anchor{35b2}
@subsubsection How to use help


The public methods of the Screen and Turtle classes are documented extensively
via docstrings.  So these can be used as online-help via the Python help
facilities:


@itemize -

@item 
When using IDLE, tooltips show the signatures and first lines of the
docstrings of typed in function-/method calls.

@item 
Calling @ref{3bc,,help()} on methods or functions displays the docstrings:

@example
>>> help(Screen.bgcolor)
Help on method bgcolor in module turtle:

bgcolor(self, *args) unbound turtle.Screen method
    Set or return backgroundcolor of the TurtleScreen.

    Arguments (if given): a color string or three numbers
    in the range 0..colormode or a 3-tuple of such numbers.


      >>> screen.bgcolor("orange")
      >>> screen.bgcolor()
      "orange"
      >>> screen.bgcolor(0.5,0,0.5)
      >>> screen.bgcolor()
      "#800080"

>>> help(Turtle.penup)
Help on method penup in module turtle:

penup(self) unbound turtle.Turtle method
    Pull the pen up -- no drawing when moving.

    Aliases: penup | pu | up

    No argument

    >>> turtle.penup()
@end example

@item 
The docstrings of the functions which are derived from methods have a modified
form:

@example
>>> help(bgcolor)
Help on function bgcolor in module turtle:

bgcolor(*args)
    Set or return backgroundcolor of the TurtleScreen.

    Arguments (if given): a color string or three numbers
    in the range 0..colormode or a 3-tuple of such numbers.

    Example::

      >>> bgcolor("orange")
      >>> bgcolor()
      "orange"
      >>> bgcolor(0.5,0,0.5)
      >>> bgcolor()
      "#800080"

>>> help(penup)
Help on function penup in module turtle:

penup()
    Pull the pen up -- no drawing when moving.

    Aliases: penup | pu | up

    No argument

    Example:
    >>> penup()
@end example
@end itemize

These modified docstrings are created automatically together with the function
definitions that are derived from the methods at import time.

@node Translation of docstrings into different languages,How to configure Screen and Turtles,How to use help,Help and configuration
@anchor{library/turtle translation-of-docstrings-into-different-languages}@anchor{35b3}
@subsubsection Translation of docstrings into different languages


There is a utility to create a dictionary the keys of which are the method names
and the values of which are the docstrings of the public methods of the classes
Screen and Turtle.

@geindex write_docstringdict() (in module turtle)
@anchor{library/turtle turtle write_docstringdict}@anchor{35b4}
@deffn {Function} turtle.write_docstringdict (filename='turtle_docstringdict')

@*Parameters: 
@code{filename} – a string, used as filename


Create and write docstring-dictionary to a Python script with the given
filename.  This function has to be called explicitly (it is not used by the
turtle graphics classes).  The docstring dictionary will be written to the
Python script @code{@emph{filename}.py}.  It is intended to serve as a template
for translation of the docstrings into different languages.
@end deffn

If you (or your students) want to use @ref{11d,,turtle} with online help in your
native language, you have to translate the docstrings and save the resulting
file as e.g. @code{turtle_docstringdict_german.py}.

If you have an appropriate entry in your @code{turtle.cfg} file this dictionary
will be read in at import time and will replace the original English docstrings.

At the time of this writing there are docstring dictionaries in German and in
Italian.  (Requests please to @email{glingl@@aon.at}.)

@node How to configure Screen and Turtles,,Translation of docstrings into different languages,Help and configuration
@anchor{library/turtle how-to-configure-screen-and-turtles}@anchor{35b5}
@subsubsection How to configure Screen and Turtles


The built-in default configuration mimics the appearance and behaviour of the
old turtle module in order to retain best possible compatibility with it.

If you want to use a different configuration which better reflects the features
of this module or which better fits to your needs, e.g. for use in a classroom,
you can prepare a configuration file @code{turtle.cfg} which will be read at import
time and modify the configuration according to its settings.

The built in configuration would correspond to the following turtle.cfg:

@example
width = 0.5
height = 0.75
leftright = None
topbottom = None
canvwidth = 400
canvheight = 300
mode = standard
colormode = 1.0
delay = 10
undobuffersize = 1000
shape = classic
pencolor = black
fillcolor = black
resizemode = noresize
visible = True
language = english
exampleturtle = turtle
examplescreen = screen
title = Python Turtle Graphics
using_IDLE = False
@end example

Short explanation of selected entries:


@itemize -

@item 
The first four lines correspond to the arguments of the @code{Screen.setup()}
method.

@item 
Line 5 and 6 correspond to the arguments of the method
@code{Screen.screensize()}.

@item 
@emph{shape} can be any of the built-in shapes, e.g: arrow, turtle, etc.  For more
info try @code{help(shape)}.

@item 
If you want to use no fillcolor (i.e. make the turtle transparent), you have
to write @code{fillcolor = ""} (but all nonempty strings must not have quotes in
the cfg-file).

@item 
If you want to reflect the turtle its state, you have to use @code{resizemode =
auto}.

@item 
If you set e.g. @code{language = italian} the docstringdict
@code{turtle_docstringdict_italian.py} will be loaded at import time (if
present on the import path, e.g. in the same directory as @ref{11d,,turtle}).

@item 
The entries @emph{exampleturtle} and @emph{examplescreen} define the names of these
objects as they occur in the docstrings.  The transformation of
method-docstrings to function-docstrings will delete these names from the
docstrings.

@item 
@emph{using_IDLE}: Set this to @code{True} if you regularly work with IDLE and its @code{-n}
switch (“no subprocess”).  This will prevent @ref{3592,,exitonclick()} to enter the
mainloop.
@end itemize

There can be a @code{turtle.cfg} file in the directory where @ref{11d,,turtle} is
stored and an additional one in the current working directory.  The latter will
override the settings of the first one.

The @code{Lib/turtledemo} directory contains a @code{turtle.cfg} file.  You can
study it as an example and see its effects when running the demos (preferably
not from within the demo-viewer).

@node turtledemo — Demo scripts,Changes since Python 2 6,Help and configuration,turtle — Turtle graphics
@anchor{library/turtle module-turtledemo}@anchor{11e}@anchor{library/turtle turtledemo-demo-scripts}@anchor{35b6}
@subsubsection @code{turtledemo} — Demo scripts


@geindex module; turtledemo

The @ref{11e,,turtledemo} package includes a set of demo scripts.  These
scripts can be run and viewed using the supplied demo viewer as follows:

@example
python -m turtledemo
@end example

Alternatively, you can run the demo scripts individually.  For example,

@example
python -m turtledemo.bytedesign
@end example

The @ref{11e,,turtledemo} package directory contains:


@itemize -

@item 
A demo viewer @code{__main__.py} which can be used to view the sourcecode
of the scripts and run them at the same time.

@item 
Multiple scripts demonstrating different features of the @ref{11d,,turtle}
module.  Examples can be accessed via the Examples menu.  They can also
be run standalone.

@item 
A @code{turtle.cfg} file which serves as an example of how to write
and use such files.
@end itemize

The demo scripts are:


@multitable {xxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Name

@tab

Description

@tab

Features

@item

bytedesign

@tab

complex classical
turtle graphics pattern

@tab

@code{tracer()}, delay,
@code{update()}

@item

chaos

@tab

graphs Verhulst dynamics,
shows that computer’s
computations can generate
results sometimes against the
common sense expectations

@tab

world coordinates

@item

clock

@tab

analog clock showing time
of your computer

@tab

turtles as clock’s
hands, ontimer

@item

colormixer

@tab

experiment with r, g, b

@tab

@code{ondrag()}

@item

forest

@tab

3 breadth-first trees

@tab

randomization

@item

fractalcurves

@tab

Hilbert & Koch curves

@tab

recursion

@item

lindenmayer

@tab

ethnomathematics
(indian kolams)

@tab

L-System

@item

minimal_hanoi

@tab

Towers of Hanoi

@tab

Rectangular Turtles
as Hanoi discs
(shape, shapesize)

@item

nim

@tab

play the classical nim game
with three heaps of sticks
against the computer.

@tab

turtles as nimsticks,
event driven (mouse,
keyboard)

@item

paint

@tab

super minimalistic
drawing program

@tab

@code{onclick()}

@item

peace

@tab

elementary

@tab

turtle: appearance
and animation

@item

penrose

@tab

aperiodic tiling with
kites and darts

@tab

@code{stamp()}

@item

planet_and_moon

@tab

simulation of
gravitational system

@tab

compound shapes,
@code{Vec2D}

@item

round_dance

@tab

dancing turtles rotating
pairwise in opposite
direction

@tab

compound shapes, clone
shapesize, tilt,
get_shapepoly, update

@item

sorting_animate

@tab

visual demonstration of
different sorting methods

@tab

simple alignment,
randomization

@item

tree

@tab

a (graphical) breadth
first tree (using generators)

@tab

@code{clone()}

@item

two_canvases

@tab

simple design

@tab

turtles on two
canvases

@item

wikipedia

@tab

a pattern from the wikipedia
article on turtle graphics

@tab

@code{clone()},
@code{undo()}

@item

yinyang

@tab

another elementary example

@tab

@code{circle()}

@end multitable


Have fun!

@node Changes since Python 2 6,Changes since Python 3 0,turtledemo — Demo scripts,turtle — Turtle graphics
@anchor{library/turtle changes-since-python-2-6}@anchor{35b7}
@subsubsection Changes since Python 2.6



@itemize -

@item 
The methods @code{Turtle.tracer()}, @code{Turtle.window_width()} and
@code{Turtle.window_height()} have been eliminated.
Methods with these names and functionality are now available only
as methods of @code{Screen}. The functions derived from these remain
available. (In fact already in Python 2.6 these methods were merely
duplications of the corresponding
@code{TurtleScreen}/@code{Screen}-methods.)

@item 
The method @code{Turtle.fill()} has been eliminated.
The behaviour of @code{begin_fill()} and @code{end_fill()}
have changed slightly: now  every filling-process must be completed with an
@code{end_fill()} call.

@item 
A method @code{Turtle.filling()} has been added. It returns a boolean
value: @code{True} if a filling process is under way, @code{False} otherwise.
This behaviour corresponds to a @code{fill()} call without arguments in
Python 2.6.
@end itemize

@node Changes since Python 3 0,,Changes since Python 2 6,turtle — Turtle graphics
@anchor{library/turtle changes-since-python-3-0}@anchor{35b8}
@subsubsection Changes since Python 3.0



@itemize -

@item 
The methods @code{Turtle.shearfactor()}, @code{Turtle.shapetransform()} and
@code{Turtle.get_shapepoly()} have been added. Thus the full range of
regular linear transforms is now available for transforming turtle shapes.
@code{Turtle.tiltangle()} has been enhanced in functionality: it now can
be used to get or set the tiltangle. @code{Turtle.settiltangle()} has been
deprecated.

@item 
The method @code{Screen.onkeypress()} has been added as a complement to
@code{Screen.onkey()} which in fact binds actions to the keyrelease event.
Accordingly the latter has got an alias: @code{Screen.onkeyrelease()}.

@item 
The method  @code{Screen.mainloop()} has been added. So when working only
with Screen and Turtle objects one must not additionally import
@code{mainloop()} anymore.

@item 
Two input methods has been added @code{Screen.textinput()} and
@code{Screen.numinput()}. These popup input dialogs and return
strings and numbers respectively.

@item 
Two example scripts @code{tdemo_nim.py} and @code{tdemo_round_dance.py}
have been added to the @code{Lib/turtledemo} directory.
@end itemize

@c >>> for turtle in turtles():
@c ...      turtle.reset()
@c >>> turtle.penup()
@c >>> turtle.goto(-200,25)
@c >>> turtle.pendown()
@c >>> turtle.write("No one expects the Spanish Inquisition!",
@c ...      font=("Arial", 20, "normal"))
@c >>> turtle.penup()
@c >>> turtle.goto(-100,-50)
@c >>> turtle.pendown()
@c >>> turtle.write("Our two chief Turtles are...",
@c ...      font=("Arial", 16, "normal"))
@c >>> turtle.penup()
@c >>> turtle.goto(-450,-75)
@c >>> turtle.write(str(turtles()))

@node cmd — Support for line-oriented command interpreters,shlex — Simple lexical analysis,turtle — Turtle graphics,Program Frameworks
@anchor{library/cmd doc}@anchor{35b9}@anchor{library/cmd cmd-support-for-line-oriented-command-interpreters}@anchor{35ba}@anchor{library/cmd module-cmd}@anchor{18}
@subsection @code{cmd} — Support for line-oriented command interpreters


@geindex module; cmd

@strong{Source code:} Lib/cmd.py@footnote{https://github.com/python/cpython/tree/main/Lib/cmd.py}


__________________________________________________________________


The @ref{35bb,,Cmd} class provides a simple framework for writing line-oriented
command interpreters.  These are often useful for test harnesses, administrative
tools, and prototypes that will later be wrapped in a more sophisticated
interface.

@geindex Cmd (class in cmd)
@anchor{library/cmd cmd Cmd}@anchor{35bb}
@deffn {Class} cmd.Cmd (completekey='tab', stdin=None, stdout=None)

A @ref{35bb,,Cmd} instance or subclass instance is a line-oriented interpreter
framework.  There is no good reason to instantiate @ref{35bb,,Cmd} itself; rather,
it’s useful as a superclass of an interpreter class you define yourself in order
to inherit @ref{35bb,,Cmd}’s methods and encapsulate action methods.

The optional argument @emph{completekey} is the @ref{d9,,readline} name of a completion
key; it defaults to @code{Tab}. If @emph{completekey} is not @ref{182,,None} and
@ref{d9,,readline} is available, command completion is done automatically.

The optional arguments @emph{stdin} and @emph{stdout} specify the  input and output file
objects that the Cmd instance or subclass  instance will use for input and
output. If not specified, they will default to @ref{5ee,,sys.stdin} and
@ref{5ef,,sys.stdout}.

If you want a given @emph{stdin} to be used, make sure to set the instance’s
@ref{35bc,,use_rawinput} attribute to @code{False}, otherwise @emph{stdin} will be
ignored.
@end deffn

@menu
* Cmd Objects:: 
* Cmd Example:: 

@end menu

@node Cmd Objects,Cmd Example,,cmd — Support for line-oriented command interpreters
@anchor{library/cmd cmd-objects}@anchor{35bd}@anchor{library/cmd id1}@anchor{35be}
@subsubsection Cmd Objects


A @ref{35bb,,Cmd} instance has the following methods:

@geindex cmdloop() (cmd.Cmd method)
@anchor{library/cmd cmd Cmd cmdloop}@anchor{35bf}
@deffn {Method} Cmd.cmdloop (intro=None)

Repeatedly issue a prompt, accept input, parse an initial prefix off the
received input, and dispatch to action methods, passing them the remainder of
the line as argument.

The optional argument is a banner or intro string to be issued before the first
prompt (this overrides the @ref{35c0,,intro} class attribute).

If the @ref{d9,,readline} module is loaded, input will automatically inherit
@code{bash}-like history-list editing (e.g. @code{Control-P} scrolls back
to the last command, @code{Control-N} forward to the next one, @code{Control-F}
moves the cursor to the right non-destructively, @code{Control-B} moves the
cursor to the left non-destructively, etc.).

An end-of-file on input is passed back as the string @code{'EOF'}.

@geindex ? (question mark); in a command interpreter
@geindex ! (exclamation); in a command interpreter

An interpreter instance will recognize a command name @code{foo} if and only if it
has a method @code{do_foo()}.  As a special case, a line beginning with the
character @code{'?'} is dispatched to the method @code{do_help()}.  As another
special case, a line beginning with the character @code{'!'} is dispatched to the
method @code{do_shell()} (if such a method is defined).

This method will return when the @ref{35c1,,postcmd()} method returns a true value.
The @emph{stop} argument to @ref{35c1,,postcmd()} is the return value from the command’s
corresponding @code{do_*()} method.

If completion is enabled, completing commands will be done automatically, and
completing of commands args is done by calling @code{complete_foo()} with
arguments @emph{text}, @emph{line}, @emph{begidx}, and @emph{endidx}.  @emph{text} is the string prefix
we are attempting to match: all returned matches must begin with it. @emph{line} is
the current input line with leading whitespace removed, @emph{begidx} and @emph{endidx}
are the beginning and ending indexes of the prefix text, which could be used to
provide different completion depending upon which position the argument is in.

All subclasses of @ref{35bb,,Cmd} inherit a predefined @code{do_help()}.  This
method, called with an argument @code{'bar'}, invokes the corresponding method
@code{help_bar()}, and if that is not present, prints the docstring of
@code{do_bar()}, if available.  With no argument, @code{do_help()} lists all
available help topics (that is, all commands with corresponding
@code{help_*()} methods or commands that have docstrings), and also lists any
undocumented commands.
@end deffn

@geindex onecmd() (cmd.Cmd method)
@anchor{library/cmd cmd Cmd onecmd}@anchor{35c2}
@deffn {Method} Cmd.onecmd (str)

Interpret the argument as though it had been typed in response to the prompt.
This may be overridden, but should not normally need to be; see the
@ref{35c3,,precmd()} and @ref{35c1,,postcmd()} methods for useful execution hooks.  The
return value is a flag indicating whether interpretation of commands by the
interpreter should stop.  If there is a @code{do_*()} method for the command
@emph{str}, the return value of that method is returned, otherwise the return value
from the @ref{35c4,,default()} method is returned.
@end deffn

@geindex emptyline() (cmd.Cmd method)
@anchor{library/cmd cmd Cmd emptyline}@anchor{35c5}
@deffn {Method} Cmd.emptyline ()

Method called when an empty line is entered in response to the prompt. If this
method is not overridden, it repeats the last nonempty command entered.
@end deffn

@geindex default() (cmd.Cmd method)
@anchor{library/cmd cmd Cmd default}@anchor{35c4}
@deffn {Method} Cmd.default (line)

Method called on an input line when the command prefix is not recognized. If
this method is not overridden, it prints an error message and returns.
@end deffn

@geindex completedefault() (cmd.Cmd method)
@anchor{library/cmd cmd Cmd completedefault}@anchor{35c6}
@deffn {Method} Cmd.completedefault (text, line, begidx, endidx)

Method called to complete an input line when no command-specific
@code{complete_*()} method is available.  By default, it returns an empty list.
@end deffn

@geindex columnize() (cmd.Cmd method)
@anchor{library/cmd cmd Cmd columnize}@anchor{10c1}
@deffn {Method} Cmd.columnize (list, displaywidth=80)

Method called to display a list of strings as a compact set of columns.
Each column is only as wide as necessary.
Columns are separated by two spaces for readability.
@end deffn

@geindex precmd() (cmd.Cmd method)
@anchor{library/cmd cmd Cmd precmd}@anchor{35c3}
@deffn {Method} Cmd.precmd (line)

Hook method executed just before the command line @emph{line} is interpreted, but
after the input prompt is generated and issued.  This method is a stub in
@ref{35bb,,Cmd}; it exists to be overridden by subclasses.  The return value is
used as the command which will be executed by the @ref{35c2,,onecmd()} method; the
@ref{35c3,,precmd()} implementation may re-write the command or simply return @emph{line}
unchanged.
@end deffn

@geindex postcmd() (cmd.Cmd method)
@anchor{library/cmd cmd Cmd postcmd}@anchor{35c1}
@deffn {Method} Cmd.postcmd (stop, line)

Hook method executed just after a command dispatch is finished.  This method is
a stub in @ref{35bb,,Cmd}; it exists to be overridden by subclasses.  @emph{line} is the
command line which was executed, and @emph{stop} is a flag which indicates whether
execution will be terminated after the call to @ref{35c1,,postcmd()}; this will be the
return value of the @ref{35c2,,onecmd()} method.  The return value of this method will
be used as the new value for the internal flag which corresponds to @emph{stop};
returning false will cause interpretation to continue.
@end deffn

@geindex preloop() (cmd.Cmd method)
@anchor{library/cmd cmd Cmd preloop}@anchor{35c7}
@deffn {Method} Cmd.preloop ()

Hook method executed once when @ref{35bf,,cmdloop()} is called.  This method is a stub
in @ref{35bb,,Cmd}; it exists to be overridden by subclasses.
@end deffn

@geindex postloop() (cmd.Cmd method)
@anchor{library/cmd cmd Cmd postloop}@anchor{35c8}
@deffn {Method} Cmd.postloop ()

Hook method executed once when @ref{35bf,,cmdloop()} is about to return. This method
is a stub in @ref{35bb,,Cmd}; it exists to be overridden by subclasses.
@end deffn

Instances of @ref{35bb,,Cmd} subclasses have some public instance variables:

@geindex prompt (cmd.Cmd attribute)
@anchor{library/cmd cmd Cmd prompt}@anchor{35c9}
@deffn {Attribute} Cmd.prompt

The prompt issued to solicit input.
@end deffn

@geindex identchars (cmd.Cmd attribute)
@anchor{library/cmd cmd Cmd identchars}@anchor{35ca}
@deffn {Attribute} Cmd.identchars

The string of characters accepted for the command prefix.
@end deffn

@geindex lastcmd (cmd.Cmd attribute)
@anchor{library/cmd cmd Cmd lastcmd}@anchor{35cb}
@deffn {Attribute} Cmd.lastcmd

The last nonempty command prefix seen.
@end deffn

@geindex cmdqueue (cmd.Cmd attribute)
@anchor{library/cmd cmd Cmd cmdqueue}@anchor{35cc}
@deffn {Attribute} Cmd.cmdqueue

A list of queued input lines.  The cmdqueue list is checked in
@ref{35bf,,cmdloop()} when new input is needed; if it is nonempty, its elements
will be processed in order, as if entered at the prompt.
@end deffn

@geindex intro (cmd.Cmd attribute)
@anchor{library/cmd cmd Cmd intro}@anchor{35c0}
@deffn {Attribute} Cmd.intro

A string to issue as an intro or banner.  May be overridden by giving the
@ref{35bf,,cmdloop()} method an argument.
@end deffn

@geindex doc_header (cmd.Cmd attribute)
@anchor{library/cmd cmd Cmd doc_header}@anchor{35cd}
@deffn {Attribute} Cmd.doc_header

The header to issue if the help output has a section for documented commands.
@end deffn

@geindex misc_header (cmd.Cmd attribute)
@anchor{library/cmd cmd Cmd misc_header}@anchor{35ce}
@deffn {Attribute} Cmd.misc_header

The header to issue if the help output has a section for miscellaneous  help
topics (that is, there are @code{help_*()} methods without corresponding
@code{do_*()} methods).
@end deffn

@geindex undoc_header (cmd.Cmd attribute)
@anchor{library/cmd cmd Cmd undoc_header}@anchor{35cf}
@deffn {Attribute} Cmd.undoc_header

The header to issue if the help output has a section for undocumented  commands
(that is, there are @code{do_*()} methods without corresponding @code{help_*()}
methods).
@end deffn

@geindex ruler (cmd.Cmd attribute)
@anchor{library/cmd cmd Cmd ruler}@anchor{35d0}
@deffn {Attribute} Cmd.ruler

The character used to draw separator lines under the help-message headers.  If
empty, no ruler line is drawn.  It defaults to @code{'='}.
@end deffn

@geindex use_rawinput (cmd.Cmd attribute)
@anchor{library/cmd cmd Cmd use_rawinput}@anchor{35bc}
@deffn {Attribute} Cmd.use_rawinput

A flag, defaulting to true.  If true, @ref{35bf,,cmdloop()} uses @ref{e73,,input()} to
display a prompt and read the next command; if false, @code{sys.stdout.write()}
and @code{sys.stdin.readline()} are used. (This means that by importing
@ref{d9,,readline}, on systems that support it, the interpreter will automatically
support @code{Emacs}-like line editing  and command-history keystrokes.)
@end deffn

@node Cmd Example,,Cmd Objects,cmd — Support for line-oriented command interpreters
@anchor{library/cmd cmd-example}@anchor{35d1}@anchor{library/cmd id2}@anchor{35d2}
@subsubsection Cmd Example


The @ref{18,,cmd} module is mainly useful for building custom shells that let a
user work with a program interactively.

This section presents a simple example of how to build a shell around a few of
the commands in the @ref{11d,,turtle} module.

Basic turtle commands such as @ref{3527,,forward()} are added to a
@ref{35bb,,Cmd} subclass with method named @code{do_forward()}.  The argument is
converted to a number and dispatched to the turtle module.  The docstring is
used in the help utility provided by the shell.

The example also includes a basic record and playback facility implemented with
the @ref{35c3,,precmd()} method which is responsible for converting the input to
lowercase and writing the commands to a file.  The @code{do_playback()} method
reads the file and adds the recorded commands to the @code{cmdqueue} for
immediate playback:

@example
import cmd, sys
from turtle import *

class TurtleShell(cmd.Cmd):
    intro = 'Welcome to the turtle shell.   Type help or ? to list commands.\n'
    prompt = '(turtle) '
    file = None

    # ----- basic turtle commands -----
    def do_forward(self, arg):
        'Move the turtle forward by the specified distance:  FORWARD 10'
        forward(*parse(arg))
    def do_right(self, arg):
        'Turn turtle right by given number of degrees:  RIGHT 20'
        right(*parse(arg))
    def do_left(self, arg):
        'Turn turtle left by given number of degrees:  LEFT 90'
        left(*parse(arg))
    def do_goto(self, arg):
        'Move turtle to an absolute position with changing orientation.  GOTO 100 200'
        goto(*parse(arg))
    def do_home(self, arg):
        'Return turtle to the home position:  HOME'
        home()
    def do_circle(self, arg):
        'Draw circle with given radius an options extent and steps:  CIRCLE 50'
        circle(*parse(arg))
    def do_position(self, arg):
        'Print the current turtle position:  POSITION'
        print('Current position is %d %d\n' % position())
    def do_heading(self, arg):
        'Print the current turtle heading in degrees:  HEADING'
        print('Current heading is %d\n' % (heading(),))
    def do_color(self, arg):
        'Set the color:  COLOR BLUE'
        color(arg.lower())
    def do_undo(self, arg):
        'Undo (repeatedly) the last turtle action(s):  UNDO'
    def do_reset(self, arg):
        'Clear the screen and return turtle to center:  RESET'
        reset()
    def do_bye(self, arg):
        'Stop recording, close the turtle window, and exit:  BYE'
        print('Thank you for using Turtle')
        self.close()
        bye()
        return True

    # ----- record and playback -----
    def do_record(self, arg):
        'Save future commands to filename:  RECORD rose.cmd'
        self.file = open(arg, 'w')
    def do_playback(self, arg):
        'Playback commands from a file:  PLAYBACK rose.cmd'
        self.close()
        with open(arg) as f:
            self.cmdqueue.extend(f.read().splitlines())
    def precmd(self, line):
        line = line.lower()
        if self.file and 'playback' not in line:
            print(line, file=self.file)
        return line
    def close(self):
        if self.file:
            self.file.close()
            self.file = None

def parse(arg):
    'Convert a series of zero or more numbers to an argument tuple'
    return tuple(map(int, arg.split()))

if __name__ == '__main__':
    TurtleShell().cmdloop()
@end example

Here is a sample session with the turtle shell showing the help functions, using
blank lines to repeat commands, and the simple record and playback facility:

@example
Welcome to the turtle shell.   Type help or ? to list commands.

(turtle) ?

Documented commands (type help <topic>):
========================================
bye     color    goto     home  playback  record  right
circle  forward  heading  left  position  reset   undo

(turtle) help forward
Move the turtle forward by the specified distance:  FORWARD 10
(turtle) record spiral.cmd
(turtle) position
Current position is 0 0

(turtle) heading
Current heading is 0

(turtle) reset
(turtle) circle 20
(turtle) right 30
(turtle) circle 40
(turtle) right 30
(turtle) circle 60
(turtle) right 30
(turtle) circle 80
(turtle) right 30
(turtle) circle 100
(turtle) right 30
(turtle) circle 120
(turtle) right 30
(turtle) circle 120
(turtle) heading
Current heading is 180

(turtle) forward 100
(turtle)
(turtle) right 90
(turtle) forward 100
(turtle)
(turtle) right 90
(turtle) forward 400
(turtle) right 90
(turtle) forward 500
(turtle) right 90
(turtle) forward 400
(turtle) right 90
(turtle) forward 300
(turtle) playback spiral.cmd
Current position is 0 0

Current heading is 0

Current heading is 180

(turtle) bye
Thank you for using Turtle
@end example

@node shlex — Simple lexical analysis,,cmd — Support for line-oriented command interpreters,Program Frameworks
@anchor{library/shlex doc}@anchor{35d3}@anchor{library/shlex module-shlex}@anchor{e3}@anchor{library/shlex shlex-simple-lexical-analysis}@anchor{35d4}
@subsection @code{shlex} — Simple lexical analysis


@geindex module; shlex

@strong{Source code:} Lib/shlex.py@footnote{https://github.com/python/cpython/tree/main/Lib/shlex.py}


__________________________________________________________________


The @ref{815,,shlex} class makes it easy to write lexical analyzers for
simple syntaxes resembling that of the Unix shell.  This will often be useful
for writing minilanguages, (for example, in run control files for Python
applications) or for parsing quoted strings.

The @ref{e3,,shlex} module defines the following functions:

@geindex split() (in module shlex)
@anchor{library/shlex shlex split}@anchor{443}
@deffn {Function} shlex.split (s, comments=False, posix=True)

Split the string @emph{s} using shell-like syntax. If @emph{comments} is @ref{659,,False}
(the default), the parsing of comments in the given string will be disabled
(setting the @ref{35d5,,commenters} attribute of the
@ref{815,,shlex} instance to the empty string).  This function operates
in POSIX mode by default, but uses non-POSIX mode if the @emph{posix} argument is
false.

@cartouche
@quotation Note 
Since the @ref{443,,split()} function instantiates a @ref{815,,shlex}
instance, passing @code{None} for @emph{s} will read the string to split from
standard input.
@end quotation
@end cartouche


Deprecated since version 3.9: Passing @code{None} for @emph{s} will raise an exception in future Python
versions.

@end deffn

@geindex join() (in module shlex)
@anchor{library/shlex shlex join}@anchor{536}
@deffn {Function} shlex.join (split_command)

Concatenate the tokens of the list @emph{split_command} and return a string.
This function is the inverse of @ref{443,,split()}.

@example
>>> from shlex import join
>>> print(join(['echo', '-n', 'Multiple words']))
echo -n 'Multiple words'
@end example

The returned value is shell-escaped to protect against injection
vulnerabilities (see @ref{cbf,,quote()}).


New in version 3.8.

@end deffn

@geindex quote() (in module shlex)
@anchor{library/shlex shlex quote}@anchor{cbf}
@deffn {Function} shlex.quote (s)

Return a shell-escaped version of the string @emph{s}.  The returned value is a
string that can safely be used as one token in a shell command line, for
cases where you cannot use a list.
@anchor{library/shlex shlex-quote-warning}@anchor{2907}
@cartouche
@quotation Warning 
The @code{shlex} module is @strong{only designed for Unix shells}.

The @ref{cbf,,quote()} function is not guaranteed to be correct on non-POSIX
compliant shells or shells from other operating systems such as Windows.
Executing commands quoted by this module on such shells can open up the
possibility of a command injection vulnerability.

Consider using functions that pass command arguments with lists such as
@ref{6b7,,subprocess.run()} with @code{shell=False}.
@end quotation
@end cartouche

This idiom would be unsafe:

@example
>>> filename = 'somefile; rm -rf ~'
>>> command = 'ls -l @{@}'.format(filename)
>>> print(command)  # executed by a shell: boom!
ls -l somefile; rm -rf ~
@end example

@ref{cbf,,quote()} lets you plug the security hole:

@example
>>> from shlex import quote
>>> command = 'ls -l @{@}'.format(quote(filename))
>>> print(command)
ls -l 'somefile; rm -rf ~'
>>> remote_command = 'ssh home @{@}'.format(quote(command))
>>> print(remote_command)
ssh home 'ls -l '"'"'somefile; rm -rf ~'"'"'@w{'}
@end example

The quoting is compatible with UNIX shells and with @ref{443,,split()}:

@example
>>> from shlex import split
>>> remote_command = split(remote_command)
>>> remote_command
['ssh', 'home', "ls -l 'somefile; rm -rf ~'"]
>>> command = split(remote_command[-1])
>>> command
['ls', '-l', 'somefile; rm -rf ~']
@end example


New in version 3.3.

@end deffn

The @ref{e3,,shlex} module defines the following class:

@geindex shlex (class in shlex)
@anchor{library/shlex shlex shlex}@anchor{815}
@deffn {Class} shlex.shlex (instream=None, infile=None, posix=False, punctuation_chars=False)

A @ref{815,,shlex} instance or subclass instance is a lexical analyzer
object.  The initialization argument, if present, specifies where to read
characters from.  It must be a file-/stream-like object with
@ref{24a6,,read()} and @ref{209b,,readline()} methods, or
a string.  If no argument is given, input will be taken from @code{sys.stdin}.
The second optional argument is a filename string, which sets the initial
value of the @ref{35d6,,infile} attribute.  If the @emph{instream}
argument is omitted or equal to @code{sys.stdin}, this second argument
defaults to “stdin”.  The @emph{posix} argument defines the operational mode:
when @emph{posix} is not true (default), the @ref{815,,shlex} instance will
operate in compatibility mode.  When operating in POSIX mode,
@ref{815,,shlex} will try to be as close as possible to the POSIX shell
parsing rules.  The @emph{punctuation_chars} argument provides a way to make the
behaviour even closer to how real shells parse.  This can take a number of
values: the default value, @code{False}, preserves the behaviour seen under
Python 3.5 and earlier.  If set to @code{True}, then parsing of the characters
@code{();<>|&} is changed: any run of these characters (considered punctuation
characters) is returned as a single token.  If set to a non-empty string of
characters, those characters will be used as the punctuation characters.  Any
characters in the @ref{35d7,,wordchars} attribute that appear in
@emph{punctuation_chars} will be removed from @ref{35d7,,wordchars}.  See
@ref{816,,Improved Compatibility with Shells} for more information. @emph{punctuation_chars}
can be set only upon @ref{815,,shlex} instance creation and can’t be
modified later.


Changed in version 3.6: The @emph{punctuation_chars} parameter was added.

@end deffn


@subsubheading See also



@table @asis

@item Module @ref{21,,configparser}

Parser for configuration files similar to the Windows @code{.ini} files.
@end table


@menu
* shlex Objects:: 
* Parsing Rules:: 
* Improved Compatibility with Shells:: 

@end menu

@node shlex Objects,Parsing Rules,,shlex — Simple lexical analysis
@anchor{library/shlex id1}@anchor{35d8}@anchor{library/shlex shlex-objects}@anchor{35d9}
@subsubsection shlex Objects


A @ref{815,,shlex} instance has the following methods:

@geindex get_token() (shlex.shlex method)
@anchor{library/shlex shlex shlex get_token}@anchor{35da}
@deffn {Method} shlex.get_token ()

Return a token.  If tokens have been stacked using @ref{35db,,push_token()}, pop a
token off the stack.  Otherwise, read one from the input stream.  If reading
encounters an immediate end-of-file, @ref{35dc,,eof} is returned (the empty
string (@code{'@w{'}}) in non-POSIX mode, and @code{None} in POSIX mode).
@end deffn

@geindex push_token() (shlex.shlex method)
@anchor{library/shlex shlex shlex push_token}@anchor{35db}
@deffn {Method} shlex.push_token (str)

Push the argument onto the token stack.
@end deffn

@geindex read_token() (shlex.shlex method)
@anchor{library/shlex shlex shlex read_token}@anchor{35dd}
@deffn {Method} shlex.read_token ()

Read a raw token.  Ignore the pushback stack, and do not interpret source
requests.  (This is not ordinarily a useful entry point, and is documented here
only for the sake of completeness.)
@end deffn

@geindex sourcehook() (shlex.shlex method)
@anchor{library/shlex shlex shlex sourcehook}@anchor{35de}
@deffn {Method} shlex.sourcehook (filename)

When @ref{815,,shlex} detects a source request (see @ref{35df,,source}
below) this method is given the following token as argument, and expected
to return a tuple consisting of a filename and an open file-like object.

Normally, this method first strips any quotes off the argument.  If the result
is an absolute pathname, or there was no previous source request in effect, or
the previous source was a stream (such as @code{sys.stdin}), the result is left
alone.  Otherwise, if the result is a relative pathname, the directory part of
the name of the file immediately before it on the source inclusion stack is
prepended (this behavior is like the way the C preprocessor handles @code{#include
"file.h"}).

The result of the manipulations is treated as a filename, and returned as the
first component of the tuple, with @ref{1c8,,open()} called on it to yield the second
component. (Note: this is the reverse of the order of arguments in instance
initialization!)

This hook is exposed so that you can use it to implement directory search paths,
addition of file extensions, and other namespace hacks. There is no
corresponding ‘close’ hook, but a shlex instance will call the
@ref{12d8,,close()} method of the sourced input stream when it returns
EOF.

For more explicit control of source stacking, use the @ref{35e0,,push_source()} and
@ref{35e1,,pop_source()} methods.
@end deffn

@geindex push_source() (shlex.shlex method)
@anchor{library/shlex shlex shlex push_source}@anchor{35e0}
@deffn {Method} shlex.push_source (newstream, newfile=None)

Push an input source stream onto the input stack.  If the filename argument is
specified it will later be available for use in error messages.  This is the
same method used internally by the @ref{35de,,sourcehook()} method.
@end deffn

@geindex pop_source() (shlex.shlex method)
@anchor{library/shlex shlex shlex pop_source}@anchor{35e1}
@deffn {Method} shlex.pop_source ()

Pop the last-pushed input source from the input stack. This is the same method
used internally when the lexer reaches EOF on a stacked input stream.
@end deffn

@geindex error_leader() (shlex.shlex method)
@anchor{library/shlex shlex shlex error_leader}@anchor{35e2}
@deffn {Method} shlex.error_leader (infile=None, lineno=None)

This method generates an error message leader in the format of a Unix C compiler
error label; the format is @code{'"%s", line %d: '}, where the @code{%s} is replaced
with the name of the current source file and the @code{%d} with the current input
line number (the optional arguments can be used to override these).

This convenience is provided to encourage @ref{e3,,shlex} users to generate error
messages in the standard, parseable format understood by Emacs and other Unix
tools.
@end deffn

Instances of @ref{815,,shlex} subclasses have some public instance
variables which either control lexical analysis or can be used for debugging:

@geindex commenters (shlex.shlex attribute)
@anchor{library/shlex shlex shlex commenters}@anchor{35d5}
@deffn {Attribute} shlex.commenters

The string of characters that are recognized as comment beginners. All
characters from the comment beginner to end of line are ignored. Includes just
@code{'#'} by default.
@end deffn

@geindex wordchars (shlex.shlex attribute)
@anchor{library/shlex shlex shlex wordchars}@anchor{35d7}
@deffn {Attribute} shlex.wordchars

The string of characters that will accumulate into multi-character tokens.  By
default, includes all ASCII alphanumerics and underscore.  In POSIX mode, the
accented characters in the Latin-1 set are also included.  If
@ref{1271,,punctuation_chars} is not empty, the characters @code{~-./*?=}, which can
appear in filename specifications and command line parameters, will also be
included in this attribute, and any characters which appear in
@code{punctuation_chars} will be removed from @code{wordchars} if they are present
there. If @ref{35e3,,whitespace_split} is set to @code{True}, this will have no
effect.
@end deffn

@geindex whitespace (shlex.shlex attribute)
@anchor{library/shlex shlex shlex whitespace}@anchor{35e4}
@deffn {Attribute} shlex.whitespace

Characters that will be considered whitespace and skipped.  Whitespace bounds
tokens.  By default, includes space, tab, linefeed and carriage-return.
@end deffn

@geindex escape (shlex.shlex attribute)
@anchor{library/shlex shlex shlex escape}@anchor{35e5}
@deffn {Attribute} shlex.escape

Characters that will be considered as escape. This will be only used in POSIX
mode, and includes just @code{'\'} by default.
@end deffn

@geindex quotes (shlex.shlex attribute)
@anchor{library/shlex shlex shlex quotes}@anchor{35e6}
@deffn {Attribute} shlex.quotes

Characters that will be considered string quotes.  The token accumulates until
the same quote is encountered again (thus, different quote types protect each
other as in the shell.)  By default, includes ASCII single and double quotes.
@end deffn

@geindex escapedquotes (shlex.shlex attribute)
@anchor{library/shlex shlex shlex escapedquotes}@anchor{35e7}
@deffn {Attribute} shlex.escapedquotes

Characters in @ref{35e6,,quotes} that will interpret escape characters defined in
@ref{35e5,,escape}.  This is only used in POSIX mode, and includes just @code{'"'} by
default.
@end deffn

@geindex whitespace_split (shlex.shlex attribute)
@anchor{library/shlex shlex shlex whitespace_split}@anchor{35e3}
@deffn {Attribute} shlex.whitespace_split

If @code{True}, tokens will only be split in whitespaces.  This is useful, for
example, for parsing command lines with @ref{815,,shlex}, getting
tokens in a similar way to shell arguments.  When used in combination with
@ref{1271,,punctuation_chars}, tokens will be split on whitespace in addition to
those characters.


Changed in version 3.8: The @ref{1271,,punctuation_chars} attribute was made compatible with the
@ref{35e3,,whitespace_split} attribute.

@end deffn

@geindex infile (shlex.shlex attribute)
@anchor{library/shlex shlex shlex infile}@anchor{35d6}
@deffn {Attribute} shlex.infile

The name of the current input file, as initially set at class instantiation time
or stacked by later source requests.  It may be useful to examine this when
constructing error messages.
@end deffn

@geindex instream (shlex.shlex attribute)
@anchor{library/shlex shlex shlex instream}@anchor{35e8}
@deffn {Attribute} shlex.instream

The input stream from which this @ref{815,,shlex} instance is reading
characters.
@end deffn

@geindex source (shlex.shlex attribute)
@anchor{library/shlex shlex shlex source}@anchor{35df}
@deffn {Attribute} shlex.source

This attribute is @code{None} by default.  If you assign a string to it, that
string will be recognized as a lexical-level inclusion request similar to the
@code{source} keyword in various shells.  That is, the immediately following token
will be opened as a filename and input will be taken from that stream until
EOF, at which point the @ref{12d8,,close()} method of that stream will be
called and the input source will again become the original input stream.  Source
requests may be stacked any number of levels deep.
@end deffn

@geindex debug (shlex.shlex attribute)
@anchor{library/shlex shlex shlex debug}@anchor{35e9}
@deffn {Attribute} shlex.debug

If this attribute is numeric and @code{1} or more, a @ref{815,,shlex}
instance will print verbose progress output on its behavior.  If you need
to use this, you can read the module source code to learn the details.
@end deffn

@geindex lineno (shlex.shlex attribute)
@anchor{library/shlex shlex shlex lineno}@anchor{35ea}
@deffn {Attribute} shlex.lineno

Source line number (count of newlines seen so far plus one).
@end deffn

@geindex token (shlex.shlex attribute)
@anchor{library/shlex shlex shlex token}@anchor{35eb}
@deffn {Attribute} shlex.token

The token buffer.  It may be useful to examine this when catching exceptions.
@end deffn

@geindex eof (shlex.shlex attribute)
@anchor{library/shlex shlex shlex eof}@anchor{35dc}
@deffn {Attribute} shlex.eof

Token used to determine end of file. This will be set to the empty string
(@code{'@w{'}}), in non-POSIX mode, and to @code{None} in POSIX mode.
@end deffn

@geindex punctuation_chars (shlex.shlex attribute)
@anchor{library/shlex shlex shlex punctuation_chars}@anchor{1271}
@deffn {Attribute} shlex.punctuation_chars

A read-only property. Characters that will be considered punctuation. Runs of
punctuation characters will be returned as a single token. However, note that no
semantic validity checking will be performed: for example, ‘>>>’ could be
returned as a token, even though it may not be recognised as such by shells.


New in version 3.6.

@end deffn

@node Parsing Rules,Improved Compatibility with Shells,shlex Objects,shlex — Simple lexical analysis
@anchor{library/shlex parsing-rules}@anchor{35ec}@anchor{library/shlex shlex-parsing-rules}@anchor{35ed}
@subsubsection Parsing Rules


When operating in non-POSIX mode, @ref{815,,shlex} will try to obey to the
following rules.


@itemize *

@item 
Quote characters are not recognized within words (@code{Do"Not"Separate} is
parsed as the single word @code{Do"Not"Separate});

@item 
Escape characters are not recognized;

@item 
Enclosing characters in quotes preserve the literal value of all characters
within the quotes;

@item 
Closing quotes separate words (@code{"Do"Separate} is parsed as @code{"Do"} and
@code{Separate});

@item 
If @ref{35e3,,whitespace_split} is @code{False}, any character not
declared to be a word character, whitespace, or a quote will be returned as
a single-character token. If it is @code{True}, @ref{815,,shlex} will only
split words in whitespaces;

@item 
EOF is signaled with an empty string (@code{'@w{'}});

@item 
It’s not possible to parse empty strings, even if quoted.
@end itemize

When operating in POSIX mode, @ref{815,,shlex} will try to obey to the
following parsing rules.


@itemize *

@item 
Quotes are stripped out, and do not separate words (@code{"Do"Not"Separate"} is
parsed as the single word @code{DoNotSeparate});

@item 
Non-quoted escape characters (e.g. @code{'\'}) preserve the literal value of the
next character that follows;

@item 
Enclosing characters in quotes which are not part of
@ref{35e7,,escapedquotes} (e.g. @code{"'"}) preserve the literal value
of all characters within the quotes;

@item 
Enclosing characters in quotes which are part of
@ref{35e7,,escapedquotes} (e.g. @code{'"'}) preserves the literal value
of all characters within the quotes, with the exception of the characters
mentioned in @ref{35e5,,escape}.  The escape characters retain its
special meaning only when followed by the quote in use, or the escape
character itself. Otherwise the escape character will be considered a
normal character.

@item 
EOF is signaled with a @ref{182,,None} value;

@item 
Quoted empty strings (@code{'@w{'}}) are allowed.
@end itemize

@node Improved Compatibility with Shells,,Parsing Rules,shlex — Simple lexical analysis
@anchor{library/shlex improved-compatibility-with-shells}@anchor{35ee}@anchor{library/shlex improved-shell-compatibility}@anchor{816}
@subsubsection Improved Compatibility with Shells



New in version 3.6.


The @ref{e3,,shlex} class provides compatibility with the parsing performed by
common Unix shells like @code{bash}, @code{dash}, and @code{sh}.  To take advantage of
this compatibility, specify the @code{punctuation_chars} argument in the
constructor.  This defaults to @code{False}, which preserves pre-3.6 behaviour.
However, if it is set to @code{True}, then parsing of the characters @code{();<>|&}
is changed: any run of these characters is returned as a single token.  While
this is short of a full parser for shells (which would be out of scope for the
standard library, given the multiplicity of shells out there), it does allow
you to perform processing of command lines more easily than you could
otherwise.  To illustrate, you can see the difference in the following snippet:

@example
 >>> import shlex
 >>> text = "a && b; c && d || e; f >'abc'; (def \"ghi\")"
 >>> s = shlex.shlex(text, posix=True)
 >>> s.whitespace_split = True
 >>> list(s)
 ['a', '&&', 'b;', 'c', '&&', 'd', '||', 'e;', 'f', '>abc;', '(def', 'ghi)']
 >>> s = shlex.shlex(text, posix=True, punctuation_chars=True)
 >>> s.whitespace_split = True
 >>> list(s)
 ['a', '&&', 'b', ';', 'c', '&&', 'd', '||', 'e', ';', 'f', '>', 'abc', ';',
 '(', 'def', 'ghi', ')']
@end example

Of course, tokens will be returned which are not valid for shells, and you’ll
need to implement your own error checks on the returned tokens.

Instead of passing @code{True} as the value for the punctuation_chars parameter,
you can pass a string with specific characters, which will be used to determine
which characters constitute punctuation. For example:

@example
>>> import shlex
>>> s = shlex.shlex("a && b || c", punctuation_chars="|")
>>> list(s)
['a', '&', '&', 'b', '||', 'c']
@end example

@cartouche
@quotation Note 
When @code{punctuation_chars} is specified, the @ref{35d7,,wordchars}
attribute is augmented with the characters @code{~-./*?=}.  That is because these
characters can appear in file names (including wildcards) and command-line
arguments (e.g. @code{--color=auto}). Hence:

@example
>>> import shlex
>>> s = shlex.shlex('~/a && b-c --color=auto || d *.py?',
...                 punctuation_chars=True)
>>> list(s)
['~/a', '&&', 'b-c', '--color=auto', '||', 'd', '*.py?']
@end example

However, to match the shell as closely as possible, it is recommended to
always use @code{posix} and @ref{35e3,,whitespace_split} when using
@ref{1271,,punctuation_chars}, which will negate
@ref{35d7,,wordchars} entirely.
@end quotation
@end cartouche

For best effect, @code{punctuation_chars} should be set in conjunction with
@code{posix=True}. (Note that @code{posix=False} is the default for
@ref{815,,shlex}.)

@node Graphical User Interfaces with Tk,Development Tools,Program Frameworks,The Python Standard Library
@anchor{library/tk doc}@anchor{35ef}@anchor{library/tk graphical-user-interfaces-with-tk}@anchor{35f0}@anchor{library/tk tkinter}@anchor{35f1}
@section Graphical User Interfaces with Tk


@geindex GUI
@geindex Graphical User Interface
@geindex Tkinter
@geindex Tk

Tk/Tcl has long been an integral part of Python.  It provides a robust and
platform independent windowing toolkit, that is available to Python programmers
using the @ref{10c,,tkinter} package, and its extension, the @ref{115,,tkinter.tix} and
the @ref{116,,tkinter.ttk} modules.

The @ref{10c,,tkinter} package is a thin object-oriented layer on top of Tcl/Tk. To
use @ref{10c,,tkinter}, you don’t need to write Tcl code, but you will need to
consult the Tk documentation, and occasionally the Tcl documentation.
@ref{10c,,tkinter} is a set of wrappers that implement the Tk widgets as Python
classes.

@ref{10c,,tkinter}’s chief virtues are that it is fast, and that it usually comes
bundled with Python. Although its standard documentation is weak, good
material is available, which includes: references, tutorials, a book and
others. @ref{10c,,tkinter} is also famous for having an outdated look and feel,
which has been vastly improved in Tk 8.5. Nevertheless, there are many other
GUI libraries that you could be interested in. The Python wiki lists several
alternative GUI frameworks and tools@footnote{https://wiki.python.org/moin/GuiProgramming}.

@menu
* tkinter — Python interface to Tcl/Tk:: 
* tkinter.colorchooser — Color choosing dialog: tkinter colorchooser — Color choosing dialog. 
* tkinter.font — Tkinter font wrapper: tkinter font — Tkinter font wrapper. 
* Tkinter Dialogs:: 
* tkinter.messagebox — Tkinter message prompts: tkinter messagebox — Tkinter message prompts. 
* tkinter.scrolledtext — Scrolled Text Widget: tkinter scrolledtext — Scrolled Text Widget. 
* tkinter.dnd — Drag and drop support: tkinter dnd — Drag and drop support. 
* tkinter.ttk — Tk themed widgets: tkinter ttk — Tk themed widgets. 
* tkinter.tix — Extension widgets for Tk: tkinter tix — Extension widgets for Tk. 
* IDLE: IDLE<51>. 

@end menu

@node tkinter — Python interface to Tcl/Tk,tkinter colorchooser — Color choosing dialog,,Graphical User Interfaces with Tk
@anchor{library/tkinter doc}@anchor{35f2}@anchor{library/tkinter module-tkinter}@anchor{10c}@anchor{library/tkinter tkinter-python-interface-to-tcl-tk}@anchor{35f3}
@subsection @code{tkinter} — Python interface to Tcl/Tk


@geindex module; tkinter

@strong{Source code:} Lib/tkinter/__init__.py@footnote{https://github.com/python/cpython/tree/main/Lib/tkinter/__init__.py}


__________________________________________________________________


The @ref{10c,,tkinter} package (“Tk interface”) is the standard Python interface to
the Tcl/Tk GUI toolkit.  Both Tk and @ref{10c,,tkinter} are available on most Unix
platforms, including macOS, as well as on Windows systems.

Running @code{python -m tkinter} from the command line should open a window
demonstrating a simple Tk interface, letting you know that @ref{10c,,tkinter} is
properly installed on your system, and also showing what version of Tcl/Tk is
installed, so you can read the Tcl/Tk documentation specific to that version.

Tkinter supports a range of Tcl/Tk versions, built either with or
without thread support. The official Python binary release bundles Tcl/Tk 8.6
threaded. See the source code for the @code{_tkinter} module
for more information about supported versions.

Tkinter is not a thin wrapper, but adds a fair amount of its own logic to
make the experience more pythonic. This documentation will concentrate on these
additions and changes, and refer to the official Tcl/Tk documentation for
details that are unchanged.

@cartouche
@quotation Note 
Tcl/Tk 8.5 (2007) introduced a modern set of themed user interface components
along with a new API to use them. Both old and new APIs are still available.
Most documentation you will find online still uses the old API and
can be woefully outdated.
@end quotation
@end cartouche


@subsubheading See also



@itemize *

@item 

@table @asis

@item TkDocs@footnote{http://tkdocs.com/}

Extensive tutorial on creating user interfaces with Tkinter.  Explains key concepts,
and illustrates recommended approaches using the modern API.
@end table

@item 

@table @asis

@item Tkinter 8.5 reference: a GUI for Python@footnote{https://www.tkdocs.com/shipman/}

Reference documentation for Tkinter 8.5 detailing available classes, methods, and options.
@end table
@end itemize

Tcl/Tk Resources:


@itemize *

@item 

@table @asis

@item Tk commands@footnote{https://www.tcl.tk/man/tcl8.6/TkCmd/contents.htm}

Comprehensive reference to each of the underlying Tcl/Tk commands used by Tkinter.
@end table

@item 

@table @asis

@item Tcl/Tk Home Page@footnote{https://www.tcl.tk}

Additional documentation, and links to Tcl/Tk core development.
@end table
@end itemize

Books:


@itemize *

@item 

@table @asis

@item Modern Tkinter for Busy Python Developers@footnote{https://tkdocs.com/book.html}

By Mark Roseman. (ISBN 978-1999149567)
@end table

@item 

@table @asis

@item Python and Tkinter Programming@footnote{https://www.packtpub.com/product/python-gui-programming-with-tkinter/9781788835886}

By Alan Moore. (ISBN 978-1788835886)
@end table

@item 

@table @asis

@item Programming Python@footnote{http://learning-python.com/about-pp4e.html}

By Mark Lutz; has excellent coverage of Tkinter. (ISBN 978-0596158101)
@end table

@item 

@table @asis

@item Tcl and the Tk Toolkit (2nd edition)@footnote{https://www.amazon.com/exec/obidos/ASIN/032133633X}

By John Ousterhout, inventor of Tcl/Tk, and Ken Jones; does not cover Tkinter. (ISBN 978-0321336330)
@end table
@end itemize


@menu
* Architecture:: 
* Tkinter Modules:: 
* Tkinter Life Preserver:: 
* Threading model:: 
* Handy Reference:: 
* File Handlers:: 

@end menu

@node Architecture,Tkinter Modules,,tkinter — Python interface to Tcl/Tk
@anchor{library/tkinter architecture}@anchor{35f4}
@subsubsection Architecture


Tcl/Tk is not a single library but rather consists of a few distinct
modules, each with separate functionality and its own official
documentation. Python’s binary releases also ship an add-on module
together with it.


@table @asis

@item Tcl

Tcl is a dynamic interpreted programming language, just like Python. Though
it can be used on its own as a general-purpose programming language, it is
most commonly embedded into C applications as a scripting engine or an
interface to the Tk toolkit. The Tcl library has a C interface to
create and manage one or more instances of a Tcl interpreter, run Tcl
commands and scripts in those instances, and add custom commands
implemented in either Tcl or C. Each interpreter has an event queue,
and there are facilities to send events to it and process them.
Unlike Python, Tcl’s execution model is designed around cooperative
multitasking, and Tkinter bridges this difference
(see @ref{35f5,,Threading model} for details).

@item Tk

Tk is a Tcl package@footnote{http://wiki.tcl.tk/37432} implemented in C
that adds custom commands to create and manipulate GUI widgets. Each
@ref{35f6,,Tk} object embeds its own Tcl interpreter instance with Tk loaded into
it. Tk’s widgets are very customizable, though at the cost of a dated appearance.
Tk uses Tcl’s event queue to generate and process GUI events.

@item Ttk

Themed Tk (Ttk) is a newer family of Tk widgets that provide a much better
appearance on different platforms than many of the classic Tk widgets.
Ttk is distributed as part of Tk, starting with Tk version 8.5. Python
bindings are provided in a separate module, @ref{116,,tkinter.ttk}.
@end table

Internally, Tk and Ttk use facilities of the underlying operating system,
i.e., Xlib on Unix/X11, Cocoa on macOS, GDI on Windows.

When your Python application uses a class in Tkinter, e.g., to create a widget,
the @ref{10c,,tkinter} module first assembles a Tcl/Tk command string. It passes that
Tcl command string to an internal @code{_tkinter} binary module, which then
calls the Tcl interpreter to evaluate it. The Tcl interpreter will then call into the
Tk and/or Ttk packages, which will in turn make calls to Xlib, Cocoa, or GDI.

@node Tkinter Modules,Tkinter Life Preserver,Architecture,tkinter — Python interface to Tcl/Tk
@anchor{library/tkinter tkinter-modules}@anchor{35f7}
@subsubsection Tkinter Modules


Support for Tkinter is spread across several modules. Most applications will need the
main @ref{10c,,tkinter} module, as well as the @ref{116,,tkinter.ttk} module, which provides
the modern themed widget set and API:

@example
from tkinter import *
from tkinter import ttk
@end example

@geindex Tk (class in tkinter)
@anchor{library/tkinter tkinter Tk}@anchor{35f6}
@deffn {Class} tkinter.Tk (screenName=None, baseName=None, className='Tk', useTk=1)

The @ref{35f6,,Tk} class is instantiated without arguments. This creates a toplevel
widget of Tk which usually is the main window of an application. Each instance
has its own associated Tcl interpreter.

@c FIXME: The following keyword arguments are currently recognized:
@end deffn

@geindex Tcl() (in module tkinter)
@anchor{library/tkinter tkinter Tcl}@anchor{35f8}
@deffn {Function} tkinter.Tcl (screenName=None, baseName=None, className='Tk', useTk=0)

The @ref{35f8,,Tcl()} function is a factory function which creates an object much like
that created by the @ref{35f6,,Tk} class, except that it does not initialize the Tk
subsystem.  This is most often useful when driving the Tcl interpreter in an
environment where one doesn’t want to create extraneous toplevel windows, or
where one cannot (such as Unix/Linux systems without an X server).  An object
created by the @ref{35f8,,Tcl()} object can have a Toplevel window created (and the Tk
subsystem initialized) by calling its @code{loadtk()} method.
@end deffn

The modules that provide Tk support include:


@table @asis

@item @ref{10c,,tkinter}

Main Tkinter module.

@item @ref{10d,,tkinter.colorchooser}

Dialog to let the user choose a color.

@item @ref{10e,,tkinter.commondialog}

Base class for the dialogs defined in the other modules listed here.

@item @ref{110,,tkinter.filedialog}

Common dialogs to allow the user to specify a file to open or save.

@item @ref{111,,tkinter.font}

Utilities to help work with fonts.

@item @ref{112,,tkinter.messagebox}

Access to standard Tk dialog boxes.

@item @ref{113,,tkinter.scrolledtext}

Text widget with a vertical scroll bar built in.

@item @ref{114,,tkinter.simpledialog}

Basic dialogs and convenience functions.

@item @ref{116,,tkinter.ttk}

Themed widget set introduced in Tk 8.5, providing modern alternatives
for many of the classic widgets in the main @ref{10c,,tkinter} module.
@end table

Additional modules:


@table @asis

@item @code{_tkinter}

A binary module that contains the low-level interface to Tcl/Tk.
It is automatically imported by the main @ref{10c,,tkinter} module,
and should never be used directly by application programmers.
It is usually a shared library (or DLL), but might in some cases be
statically linked with the Python interpreter.

@item @code{idlelib}

Python’s Integrated Development and Learning Environment (IDLE). Based
on @ref{10c,,tkinter}.

@item @code{tkinter.constants}

Symbolic constants that can be used in place of strings when passing
various parameters to Tkinter calls. Automatically imported by the
main @ref{10c,,tkinter} module.

@item @ref{10f,,tkinter.dnd}

(experimental) Drag-and-drop support for @ref{10c,,tkinter}. This will
become deprecated when it is replaced with the Tk DND.

@item @ref{115,,tkinter.tix}

(deprecated) An older third-party Tcl/Tk package that adds several new
widgets. Better alternatives for most can be found in @ref{116,,tkinter.ttk}.

@item @ref{11d,,turtle}

Turtle graphics in a Tk window.
@end table

@node Tkinter Life Preserver,Threading model,Tkinter Modules,tkinter — Python interface to Tcl/Tk
@anchor{library/tkinter tkinter-life-preserver}@anchor{35f9}
@subsubsection Tkinter Life Preserver


This section is not designed to be an exhaustive tutorial on either Tk or
Tkinter.  For that, refer to one of the external resources noted earlier.
Instead, this section provides a very quick orientation to what a Tkinter
application looks like, identifies foundational Tk concepts, and
explains how the Tkinter wrapper is structured.

The remainder of this section will help you to identify the classes,
methods, and options you’ll need in your Tkinter application, and where to
find more detailed documentation on them, including in the official Tcl/Tk
reference manual.

@menu
* A Hello World Program:: 
* Important Tk Concepts:: 
* Understanding How Tkinter Wraps Tcl/Tk:: 
* How do I…? What option does…?:: 
* Navigating the Tcl/Tk Reference Manual:: 

@end menu

@node A Hello World Program,Important Tk Concepts,,Tkinter Life Preserver
@anchor{library/tkinter a-hello-world-program}@anchor{35fa}
@subsubsection A Hello World Program


We’ll start by walking through a “Hello World” application in Tkinter. This
isn’t the smallest one we could write, but has enough to illustrate some
key concepts you’ll need to know.

@example
from tkinter import *
from tkinter import ttk
root = Tk()
frm = ttk.Frame(root, padding=10)
frm.grid()
ttk.Label(frm, text="Hello World!").grid(column=0, row=0)
ttk.Button(frm, text="Quit", command=root.destroy).grid(column=1, row=0)
root.mainloop()
@end example

After the imports, the next line creates an instance of the @ref{35f6,,Tk} class,
which initializes Tk and creates its associated Tcl interpreter. It also
creates a toplevel window, known as the root window, which serves as the main
window of the application.

The following line creates a frame widget, which in this case will contain
a label and a button we’ll create next. The frame is fit inside the root
window.

The next line creates a label widget holding a static text string. The
@code{grid()} method is used to specify the relative layout (position) of the
label within its containing frame widget, similar to how tables in HTML work.

A button widget is then created, and placed to the right of the label. When
pressed, it will call the @code{destroy()} method of the root window.

Finally, the @code{mainloop()} method puts everything on the display, and
responds to user input until the program terminates.

@node Important Tk Concepts,Understanding How Tkinter Wraps Tcl/Tk,A Hello World Program,Tkinter Life Preserver
@anchor{library/tkinter important-tk-concepts}@anchor{35fb}
@subsubsection Important Tk Concepts


Even this simple program illustrates the following key Tk concepts:


@table @asis

@item widgets

A Tkinter user interface is made up of individual @emph{widgets}. Each widget is
represented as a Python object, instantiated from classes like
@code{ttk.Frame}, @code{ttk.Label}, and @code{ttk.Button}.

@item widget hierarchy

Widgets are arranged in a @emph{hierarchy}. The label and button were contained
within a frame, which in turn was contained within the root window. When
creating each @emph{child} widget, its @emph{parent} widget is passed as the first
argument to the widget constructor.

@item configuration options

Widgets have @emph{configuration options}, which modify their appearance and
behavior, such as the text to display in a label or button. Different
classes of widgets will have different sets of options.

@item geometry management

Widgets aren’t automatically added to the user interface when they are
created. A @emph{geometry manager} like @code{grid} controls where in the
user interface they are placed.

@item event loop

Tkinter reacts to user input, changes from your program, and even refreshes
the display only when actively running an @emph{event loop}. If your program
isn’t running the event loop, your user interface won’t update.
@end table

@node Understanding How Tkinter Wraps Tcl/Tk,How do I…? What option does…?,Important Tk Concepts,Tkinter Life Preserver
@anchor{library/tkinter understanding-how-tkinter-wraps-tcl-tk}@anchor{35fc}
@subsubsection Understanding How Tkinter Wraps Tcl/Tk


When your application uses Tkinter’s classes and methods, internally Tkinter
is assembling strings representing Tcl/Tk commands, and executing those
commands in the Tcl interpreter attached to your applicaton’s @ref{35f6,,Tk}
instance.

Whether it’s trying to navigate reference documentation, trying to find
the right method or option, adapting some existing code, or debugging your
Tkinter application, there are times that it will be useful to understand
what those underlying Tcl/Tk commands look like.

To illustrate, here is the Tcl/Tk equivalent of the main part of the Tkinter
script above.

@example
ttk::frame .frm -padding 10
grid .frm
grid [ttk::label .frm.lbl -text "Hello World!"] -column 0 -row 0
grid [ttk::button .frm.btn -text "Quit" -command "destroy ."] -column 1 -row 0
@end example

Tcl’s syntax is similar to many shell languages, where the first word is the
command to be executed, with arguments to that command following it, separated
by spaces. Without getting into too many details, notice the following:


@itemize *

@item 
The commands used to create widgets (like @code{ttk::frame}) correspond to
widget classes in Tkinter.

@item 
Tcl widget options (like @code{-text}) correspond to keyword arguments in
Tkinter.

@item 
Widgets are referred to by a @emph{pathname} in Tcl (like @code{.frm.btn}),
whereas Tkinter doesn’t use names but object references.

@item 
A widget’s place in the widget hierarchy is encoded in its (hierarchical)
pathname, which uses a @code{.} (dot) as a path separator. The pathname for
the root window is just @code{.} (dot). In Tkinter, the hierarchy is defined
not by pathname but by specifying the parent widget when creating each
child widget.

@item 
Operations which are implemented as separate @emph{commands} in Tcl (like
@code{grid} or @code{destroy}) are represented as @emph{methods} on Tkinter widget
objects. As you’ll see shortly, at other times Tcl uses what appear to be
method calls on widget objects, which more closely mirror what would is
used in Tkinter.
@end itemize

@node How do I…? What option does…?,Navigating the Tcl/Tk Reference Manual,Understanding How Tkinter Wraps Tcl/Tk,Tkinter Life Preserver
@anchor{library/tkinter how-do-i-what-option-does}@anchor{35fd}
@subsubsection How do I…? What option does…?


If you’re not sure how to do something in Tkinter, and you can’t immediately
find it in the tutorial or reference documentation you’re using, there are a
few strategies that can be helpful.

First, remember that the details of how individual widgets work may vary
across different versions of both Tkinter and Tcl/Tk. If you’re searching
documentation, make sure it corresponds to the Python and Tcl/Tk versions
installed on your system.

When searching for how to use an API, it helps to know the exact name of the
class, option, or method that you’re using. Introspection, either in an
interactive Python shell or with @ref{af6,,print()}, can help you identify what
you need.

To find out what configuration options are available on any widget, call its
@code{configure()} method, which returns a dictionary containing a variety of
information about each object, including its default and current values. Use
@code{keys()} to get just the names of each option.

@example
btn = ttk.Button(frm, ...)
print(btn.configure().keys())
@end example

As most widgets have many configuration options in common, it can be useful
to find out which are specific to a particular widget class. Comparing the
list of options to that of a simpler widget, like a frame, is one way to
do that.

@example
print(set(btn.configure().keys()) - set(frm.configure().keys()))
@end example

Similarly, you can find the available methods for a widget object using the
standard @ref{f2f,,dir()} function. If you try it, you’ll see there are over 200
common widget methods, so again identifying those specific to a widget class
is helpful.

@example
print(dir(btn))
print(set(dir(btn)) - set(dir(frm)))
@end example

@node Navigating the Tcl/Tk Reference Manual,,How do I…? What option does…?,Tkinter Life Preserver
@anchor{library/tkinter navigating-the-tcl-tk-reference-manual}@anchor{35fe}
@subsubsection Navigating the Tcl/Tk Reference Manual


As noted, the official Tk commands@footnote{https://www.tcl.tk/man/tcl8.6/TkCmd/contents.htm}
reference manual (man pages) is often the most accurate description of what
specific operations on widgets do. Even when you know the name of the option
or method that you need, you may still have a few places to look.

While all operations in Tkinter are implemented as method calls on widget
objects, you’ve seen that many Tcl/Tk operations appear as commands that
take a widget pathname as its first parameter, followed by optional
parameters, e.g.

@example
destroy .
grid .frm.btn -column 0 -row 0
@end example

Others, however, look more like methods called on a widget object (in fact,
when you create a widget in Tcl/Tk, it creates a Tcl command with the name
of the widget pathname, with the first parameter to that command being the
name of a method to call).

@example
.frm.btn invoke
.frm.lbl configure -text "Goodbye"
@end example

In the official Tcl/Tk reference documentation, you’ll find most operations
that look like method calls on the man page for a specific widget (e.g.,
you’ll find the @code{invoke()} method on the
ttk::button@footnote{https://www.tcl.tk/man/tcl8.6/TkCmd/ttk_button.htm}
man page), while functions that take a widget as a parameter often have
their own man page (e.g.,
grid@footnote{https://www.tcl.tk/man/tcl8.6/TkCmd/grid.htm}).

You’ll find many common options and methods in the
options@footnote{https://www.tcl.tk/man/tcl8.6/TkCmd/options.htm} or
ttk::widget@footnote{https://www.tcl.tk/man/tcl8.6/TkCmd/ttk_widget.htm} man
pages, while others are found in the man page for a specific widget class.

You’ll also find that many Tkinter methods have compound names, e.g.,
@code{winfo_x()}, @code{winfo_height()}, @code{winfo_viewable()}. You’d find
documentation for all of these in the
winfo@footnote{https://www.tcl.tk/man/tcl8.6/TkCmd/winfo.htm} man page.

@cartouche
@quotation Note 
Somewhat confusingly, there are also methods on all Tkinter widgets
that don’t actually operate on the widget, but operate at a global
scope, independent of any widget. Examples are methods for accessing
the clipboard or the system bell. (They happen to be implemented as
methods in the base @code{Widget} class that all Tkinter widgets
inherit from).
@end quotation
@end cartouche

@node Threading model,Handy Reference,Tkinter Life Preserver,tkinter — Python interface to Tcl/Tk
@anchor{library/tkinter threading-model}@anchor{35f5}
@subsubsection Threading model


Python and Tcl/Tk have very different threading models, which @ref{10c,,tkinter}
tries to bridge. If you use threads, you may need to be aware of this.

A Python interpreter may have many threads associated with it. In Tcl, multiple
threads can be created, but each thread has a separate Tcl interpreter instance
associated with it. Threads can also create more than one interpreter instance,
though each interpreter instance can be used only by the one thread that created it.

Each @ref{35f6,,Tk} object created by @ref{10c,,tkinter} contains a Tcl interpreter.
It also keeps track of which thread created that interpreter. Calls to
@ref{10c,,tkinter} can be made from any Python thread. Internally, if a call comes
from a thread other than the one that created the @ref{35f6,,Tk} object, an event
is posted to the interpreter’s event queue, and when executed, the result is
returned to the calling Python thread.

Tcl/Tk applications are normally event-driven, meaning that after initialization,
the interpreter runs an event loop (i.e. @code{Tk.mainloop()}) and responds to events.
Because it is single-threaded, event handlers must respond quickly, otherwise they
will block other events from being processed. To avoid this, any long-running
computations should not run in an event handler, but are either broken into smaller
pieces using timers, or run in another thread. This is different from many GUI
toolkits where the GUI runs in a completely separate thread from all application
code including event handlers.

If the Tcl interpreter is not running the event loop and processing events, any
@ref{10c,,tkinter} calls made from threads other than the one running the Tcl
interpreter will fail.

A number of special cases exist:

@quotation


@itemize *

@item 
Tcl/Tk libraries can be built so they are not thread-aware. In this case,
@ref{10c,,tkinter} calls the library from the originating Python thread, even
if this is different than the thread that created the Tcl interpreter. A global
lock ensures only one call occurs at a time.

@item 
While @ref{10c,,tkinter} allows you to create more than one instance of a @ref{35f6,,Tk}
object (with its own interpreter), all interpreters that are part of the same
thread share a common event queue, which gets ugly fast. In practice, don’t create
more than one instance of @ref{35f6,,Tk} at a time. Otherwise, it’s best to create
them in separate threads and ensure you’re running a thread-aware Tcl/Tk build.

@item 
Blocking event handlers are not the only way to prevent the Tcl interpreter from
reentering the event loop. It is even possible to run multiple nested event loops
or abandon the event loop entirely. If you’re doing anything tricky when it comes
to events or threads, be aware of these possibilities.

@item 
There are a few select @ref{10c,,tkinter} functions that presently work only when
called from the thread that created the Tcl interpreter.
@end itemize
@end quotation

@node Handy Reference,File Handlers,Threading model,tkinter — Python interface to Tcl/Tk
@anchor{library/tkinter handy-reference}@anchor{35ff}
@subsubsection Handy Reference


@menu
* Setting Options:: 
* The Packer:: 
* Packer Options:: 
* Coupling Widget Variables:: 
* The Window Manager:: 
* Tk Option Data Types:: 
* Bindings and Events:: 
* The index Parameter:: 
* Images:: 

@end menu

@node Setting Options,The Packer,,Handy Reference
@anchor{library/tkinter setting-options}@anchor{3600}@anchor{library/tkinter tkinter-setting-options}@anchor{3601}
@subsubsection Setting Options


Options control things like the color and border width of a widget. Options can
be set in three ways:


@table @asis

@item At object creation time, using keyword arguments

@example
fred = Button(self, fg="red", bg="blue")
@end example

@item After object creation, treating the option name like a dictionary index

@example
fred["fg"] = "red"
fred["bg"] = "blue"
@end example

@item Use the config() method to update multiple attrs subsequent to object creation

@example
fred.config(fg="red", bg="blue")
@end example
@end table

For a complete explanation of a given option and its behavior, see the Tk man
pages for the widget in question.

Note that the man pages list “STANDARD OPTIONS” and “WIDGET SPECIFIC OPTIONS”
for each widget.  The former is a list of options that are common to many
widgets, the latter are the options that are idiosyncratic to that particular
widget.  The Standard Options are documented on the @code{options(3)} man
page.

No distinction between standard and widget-specific options is made in this
document.  Some options don’t apply to some kinds of widgets. Whether a given
widget responds to a particular option depends on the class of the widget;
buttons have a @code{command} option, labels do not.

The options supported by a given widget are listed in that widget’s man page, or
can be queried at runtime by calling the @code{config()} method without
arguments, or by calling the @code{keys()} method on that widget.  The return
value of these calls is a dictionary whose key is the name of the option as a
string (for example, @code{'relief'}) and whose values are 5-tuples.

Some options, like @code{bg} are synonyms for common options with long names
(@code{bg} is shorthand for “background”). Passing the @code{config()} method the name
of a shorthand option will return a 2-tuple, not 5-tuple. The 2-tuple passed
back will contain the name of the synonym and the “real” option (such as
@code{('bg', 'background')}).


@multitable {xxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxx} 
@headitem

Index

@tab

Meaning

@tab

Example

@item

0

@tab

option name

@tab

@code{'relief'}

@item

1

@tab

option name for database lookup

@tab

@code{'relief'}

@item

2

@tab

option class for database
lookup

@tab

@code{'Relief'}

@item

3

@tab

default value

@tab

@code{'raised'}

@item

4

@tab

current value

@tab

@code{'groove'}

@end multitable


Example:

@example
>>> print(fred.config())
@{'relief': ('relief', 'relief', 'Relief', 'raised', 'groove')@}
@end example

Of course, the dictionary printed will include all the options available and
their values.  This is meant only as an example.

@node The Packer,Packer Options,Setting Options,Handy Reference
@anchor{library/tkinter the-packer}@anchor{3602}
@subsubsection The Packer


@geindex packing (widgets)

The packer is one of Tk’s geometry-management mechanisms.    Geometry managers
are used to specify the relative positioning of widgets within their container -
their mutual @emph{master}.  In contrast to the more cumbersome @emph{placer} (which is
used less commonly, and we do not cover here), the packer takes qualitative
relationship specification - @emph{above}, @emph{to the left of}, @emph{filling}, etc - and
works everything out to determine the exact placement coordinates for you.

The size of any @emph{master} widget is determined by the size of the “slave widgets”
inside.  The packer is used to control where slave widgets appear inside the
master into which they are packed.  You can pack widgets into frames, and frames
into other frames, in order to achieve the kind of layout you desire.
Additionally, the arrangement is dynamically adjusted to accommodate incremental
changes to the configuration, once it is packed.

Note that widgets do not appear until they have had their geometry specified
with a geometry manager.  It’s a common early mistake to leave out the geometry
specification, and then be surprised when the widget is created but nothing
appears.  A widget will appear only after it has had, for example, the packer’s
@code{pack()} method applied to it.

The pack() method can be called with keyword-option/value pairs that control
where the widget is to appear within its container, and how it is to behave when
the main application window is resized.  Here are some examples:

@example
fred.pack()                     # defaults to side = "top"
fred.pack(side="left")
fred.pack(expand=1)
@end example

@node Packer Options,Coupling Widget Variables,The Packer,Handy Reference
@anchor{library/tkinter packer-options}@anchor{3603}
@subsubsection Packer Options


For more extensive information on the packer and the options that it can take,
see the man pages and page 183 of John Ousterhout’s book.


@table @asis

@item anchor

Anchor type.  Denotes where the packer is to place each slave in its parcel.

@item expand

Boolean, @code{0} or @code{1}.

@item fill

Legal values: @code{'x'}, @code{'y'}, @code{'both'}, @code{'none'}.

@item ipadx and ipady

A distance - designating internal padding on each side of the slave widget.

@item padx and pady

A distance - designating external padding on each side of the slave widget.

@item side

Legal values are: @code{'left'}, @code{'right'}, @code{'top'}, @code{'bottom'}.
@end table

@node Coupling Widget Variables,The Window Manager,Packer Options,Handy Reference
@anchor{library/tkinter coupling-widget-variables}@anchor{3604}
@subsubsection Coupling Widget Variables


The current-value setting of some widgets (like text entry widgets) can be
connected directly to application variables by using special options.  These
options are @code{variable}, @code{textvariable}, @code{onvalue}, @code{offvalue}, and
@code{value}.  This connection works both ways: if the variable changes for any
reason, the widget it’s connected to will be updated to reflect the new value.

Unfortunately, in the current implementation of @ref{10c,,tkinter} it is not
possible to hand over an arbitrary Python variable to a widget through a
@code{variable} or @code{textvariable} option.  The only kinds of variables for which
this works are variables that are subclassed from a class called Variable,
defined in @ref{10c,,tkinter}.

There are many useful subclasses of Variable already defined:
@code{StringVar}, @code{IntVar}, @code{DoubleVar}, and
@code{BooleanVar}.  To read the current value of such a variable, call the
@code{get()} method on it, and to change its value you call the @code{set()}
method.  If you follow this protocol, the widget will always track the value of
the variable, with no further intervention on your part.

For example:

@example
import tkinter as tk

class App(tk.Frame):
    def __init__(self, master):
        super().__init__(master)
        self.pack()

        self.entrythingy = tk.Entry()
        self.entrythingy.pack()

        # Create the application variable.
        self.contents = tk.StringVar()
        # Set it to some value.
        self.contents.set("this is a variable")
        # Tell the entry widget to watch this variable.
        self.entrythingy["textvariable"] = self.contents

        # Define a callback for when the user hits return.
        # It prints the current value of the variable.
        self.entrythingy.bind('<Key-Return>',
                             self.print_contents)

    def print_contents(self, event):
        print("Hi. The current entry content is:",
              self.contents.get())

root = tk.Tk()
myapp = App(root)
myapp.mainloop()
@end example

@node The Window Manager,Tk Option Data Types,Coupling Widget Variables,Handy Reference
@anchor{library/tkinter the-window-manager}@anchor{3605}
@subsubsection The Window Manager


@geindex window manager (widgets)

In Tk, there is a utility command, @code{wm}, for interacting with the window
manager.  Options to the @code{wm} command allow you to control things like titles,
placement, icon bitmaps, and the like.  In @ref{10c,,tkinter}, these commands have
been implemented as methods on the @code{Wm} class.  Toplevel widgets are
subclassed from the @code{Wm} class, and so can call the @code{Wm} methods
directly.

To get at the toplevel window that contains a given widget, you can often just
refer to the widget’s master.  Of course if the widget has been packed inside of
a frame, the master won’t represent a toplevel window.  To get at the toplevel
window that contains an arbitrary widget, you can call the @code{_root()} method.
This method begins with an underscore to denote the fact that this function is
part of the implementation, and not an interface to Tk functionality.

Here are some examples of typical usage:

@example
import tkinter as tk

class App(tk.Frame):
    def __init__(self, master=None):
        super().__init__(master)
        self.pack()

# create the application
myapp = App()

#
# here are method calls to the window manager class
#
myapp.master.title("My Do-Nothing Application")
myapp.master.maxsize(1000, 400)

# start the program
myapp.mainloop()
@end example

@node Tk Option Data Types,Bindings and Events,The Window Manager,Handy Reference
@anchor{library/tkinter tk-option-data-types}@anchor{3606}
@subsubsection Tk Option Data Types


@geindex Tk Option Data Types


@table @asis

@item anchor

Legal values are points of the compass: @code{"n"}, @code{"ne"}, @code{"e"}, @code{"se"},
@code{"s"}, @code{"sw"}, @code{"w"}, @code{"nw"}, and also @code{"center"}.

@item bitmap

There are eight built-in, named bitmaps: @code{'error'}, @code{'gray25'},
@code{'gray50'}, @code{'hourglass'}, @code{'info'}, @code{'questhead'}, @code{'question'},
@code{'warning'}.  To specify an X bitmap filename, give the full path to the file,
preceded with an @code{@@}, as in @code{"@@/usr/contrib/bitmap/gumby.bit"}.

@item boolean

You can pass integers 0 or 1 or the strings @code{"yes"} or @code{"no"}.

@item callback

This is any Python function that takes no arguments.  For example:

@example
def print_it():
    print("hi there")
fred["command"] = print_it
@end example

@item color

Colors can be given as the names of X colors in the rgb.txt file, or as strings
representing RGB values in 4 bit: @code{"#RGB"}, 8 bit: @code{"#RRGGBB"}, 12 bit:
@code{"#RRRGGGBBB"}, or 16 bit: @code{"#RRRRGGGGBBBB"} ranges, where R,G,B here
represent any legal hex digit.  See page 160 of Ousterhout’s book for details.

@item cursor

The standard X cursor names from @code{cursorfont.h} can be used, without the
@code{XC_} prefix.  For example to get a hand cursor (@code{XC_hand2}), use the
string @code{"hand2"}.  You can also specify a bitmap and mask file of your own.
See page 179 of Ousterhout’s book.

@item distance

Screen distances can be specified in either pixels or absolute distances.
Pixels are given as numbers and absolute distances as strings, with the trailing
character denoting units: @code{c} for centimetres, @code{i} for inches, @code{m} for
millimetres, @code{p} for printer’s points.  For example, 3.5 inches is expressed
as @code{"3.5i"}.

@item font

Tk uses a list font name format, such as @code{@{courier 10 bold@}}. Font sizes with
positive numbers are measured in points; sizes with negative numbers are
measured in pixels.

@item geometry

This is a string of the form @code{widthxheight}, where width and height are
measured in pixels for most widgets (in characters for widgets displaying text).
For example: @code{fred["geometry"] = "200x100"}.

@item justify

Legal values are the strings: @code{"left"}, @code{"center"}, @code{"right"}, and
@code{"fill"}.

@item region

This is a string with four space-delimited elements, each of which is a legal
distance (see above).  For example: @code{"2 3 4 5"} and @code{"3i 2i 4.5i 2i"} and
@code{"3c 2c 4c 10.43c"}  are all legal regions.

@item relief

Determines what the border style of a widget will be.  Legal values are:
@code{"raised"}, @code{"sunken"}, @code{"flat"}, @code{"groove"}, and @code{"ridge"}.

@item scrollcommand

This is almost always the @code{set()} method of some scrollbar widget, but can
be any widget method that takes a single argument.

@item wrap

Must be one of: @code{"none"}, @code{"char"}, or @code{"word"}.
@end table

@node Bindings and Events,The index Parameter,Tk Option Data Types,Handy Reference
@anchor{library/tkinter bindings-and-events}@anchor{3607}@anchor{library/tkinter id2}@anchor{3608}
@subsubsection Bindings and Events


@geindex bind (widgets)
@geindex events (widgets)

The bind method from the widget command allows you to watch for certain events
and to have a callback function trigger when that event type occurs.  The form
of the bind method is:

@example
def bind(self, sequence, func, add='@w{'}):
@end example

where:


@table @asis

@item sequence

is a string that denotes the target kind of event.  (See the bind man page and
page 201 of John Ousterhout’s book for details).

@item func

is a Python function, taking one argument, to be invoked when the event occurs.
An Event instance will be passed as the argument. (Functions deployed this way
are commonly known as @emph{callbacks}.)

@item add

is optional, either @code{'@w{'}} or @code{'+'}.  Passing an empty string denotes that
this binding is to replace any other bindings that this event is associated
with.  Passing a @code{'+'} means that this function is to be added to the list
of functions bound to this event type.
@end table

For example:

@example
def turn_red(self, event):
    event.widget["activeforeground"] = "red"

self.button.bind("<Enter>", self.turn_red)
@end example

Notice how the widget field of the event is being accessed in the
@code{turn_red()} callback.  This field contains the widget that caught the X
event.  The following table lists the other event fields you can access, and how
they are denoted in Tk, which can be useful when referring to the Tk man pages.


@multitable {xxxxxx} {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxx} {xxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Tk

@tab

Tkinter Event Field

@tab

Tk

@tab

Tkinter Event Field

@item

%f

@tab

focus

@tab

%A

@tab

char

@item

%h

@tab

height

@tab

%E

@tab

send_event

@item

%k

@tab

keycode

@tab

%K

@tab

keysym

@item

%s

@tab

state

@tab

%N

@tab

keysym_num

@item

%t

@tab

time

@tab

%T

@tab

type

@item

%w

@tab

width

@tab

%W

@tab

widget

@item

%x

@tab

x

@tab

%X

@tab

x_root

@item

%y

@tab

y

@tab

%Y

@tab

y_root

@end multitable


@node The index Parameter,Images,Bindings and Events,Handy Reference
@anchor{library/tkinter the-index-parameter}@anchor{3609}
@subsubsection The index Parameter


A number of widgets require “index” parameters to be passed.  These are used to
point at a specific place in a Text widget, or to particular characters in an
Entry widget, or to particular menu items in a Menu widget.


@table @asis

@item Entry widget indexes (index, view index, etc.)

Entry widgets have options that refer to character positions in the text being
displayed.  You can use these @ref{10c,,tkinter} functions to access these special
points in text widgets:

@item Text widget indexes

The index notation for Text widgets is very rich and is best described in the Tk
man pages.

@item Menu indexes (menu.invoke(), menu.entryconfig(), etc.)

Some options and methods for menus manipulate specific menu entries. Anytime a
menu index is needed for an option or a parameter, you may pass in:


@itemize *

@item 
an integer which refers to the numeric position of the entry in the widget,
counted from the top, starting with 0;

@item 
the string @code{"active"}, which refers to the menu position that is currently
under the cursor;

@item 
the string @code{"last"} which refers to the last menu item;

@item 
An integer preceded by @code{@@}, as in @code{@@6}, where the integer is interpreted
as a y pixel coordinate in the menu’s coordinate system;

@item 
the string @code{"none"}, which indicates no menu entry at all, most often used
with menu.activate() to deactivate all entries, and finally,

@item 
a text string that is pattern matched against the label of the menu entry, as
scanned from the top of the menu to the bottom.  Note that this index type is
considered after all the others, which means that matches for menu items
labelled @code{last}, @code{active}, or @code{none} may be interpreted as the above
literals, instead.
@end itemize
@end table

@node Images,,The index Parameter,Handy Reference
@anchor{library/tkinter images}@anchor{360a}
@subsubsection Images


Images of different formats can be created through the corresponding subclass
of @code{tkinter.Image}:


@itemize *

@item 
@code{BitmapImage} for images in XBM format.

@item 
@code{PhotoImage} for images in PGM, PPM, GIF and PNG formats. The latter
is supported starting with Tk 8.6.
@end itemize

Either type of image is created through either the @code{file} or the @code{data}
option (other options are available as well).

The image object can then be used wherever an @code{image} option is supported by
some widget (e.g. labels, buttons, menus). In these cases, Tk will not keep a
reference to the image. When the last Python reference to the image object is
deleted, the image data is deleted as well, and Tk will display an empty box
wherever the image was used.


@subsubheading See also


The Pillow@footnote{http://python-pillow.org/} package adds support for
formats such as BMP, JPEG, TIFF, and WebP, among others.


@node File Handlers,,Handy Reference,tkinter — Python interface to Tcl/Tk
@anchor{library/tkinter file-handlers}@anchor{360b}@anchor{library/tkinter tkinter-file-handlers}@anchor{360c}
@subsubsection File Handlers


Tk allows you to register and unregister a callback function which will be
called from the Tk mainloop when I/O is possible on a file descriptor.
Only one handler may be registered per file descriptor. Example code:

@example
import tkinter
widget = tkinter.Tk()
mask = tkinter.READABLE | tkinter.WRITABLE
widget.tk.createfilehandler(file, mask, callback)
...
widget.tk.deletefilehandler(file)
@end example

This feature is not available on Windows.

Since you don’t know how many bytes are available for reading, you may not
want to use the @ref{823,,BufferedIOBase} or @ref{e47,,TextIOBase}
@ref{21ef,,read()} or @ref{1b74,,readline()} methods,
since these will insist on reading a predefined number of bytes.
For sockets, the @ref{900,,recv()} or
@ref{901,,recvfrom()} methods will work fine; for other files,
use raw reads or @code{os.read(file.fileno(), maxbytecount)}.

@geindex createfilehandler() (tkinter.Widget.tk method)
@anchor{library/tkinter tkinter Widget tk createfilehandler}@anchor{360d}
@deffn {Method} Widget.tk.createfilehandler (file, mask, func)

Registers the file handler callback function @emph{func}. The @emph{file} argument
may either be an object with a @ref{23a2,,fileno()} method (such as
a file or socket object), or an integer file descriptor. The @emph{mask}
argument is an ORed combination of any of the three constants below.
The callback is called as follows:

@example
callback(file, mask)
@end example
@end deffn

@geindex deletefilehandler() (tkinter.Widget.tk method)
@anchor{library/tkinter tkinter Widget tk deletefilehandler}@anchor{360e}
@deffn {Method} Widget.tk.deletefilehandler (file)

Unregisters a file handler.
@end deffn

@geindex READABLE (in module tkinter)
@geindex WRITABLE (in module tkinter)
@geindex EXCEPTION (in module tkinter)
@anchor{library/tkinter tkinter READABLE}@anchor{360f}
@deffn {Data} tkinter.READABLE
@anchor{library/tkinter tkinter WRITABLE}@anchor{3610}
@deffnx {Data} tkinter.WRITABLE
@anchor{library/tkinter tkinter EXCEPTION}@anchor{3611}
@deffnx {Data} tkinter.EXCEPTION

Constants used in the @emph{mask} arguments.
@end deffn

@node tkinter colorchooser — Color choosing dialog,tkinter font — Tkinter font wrapper,tkinter — Python interface to Tcl/Tk,Graphical User Interfaces with Tk
@anchor{library/tkinter colorchooser doc}@anchor{3612}@anchor{library/tkinter colorchooser module-tkinter colorchooser}@anchor{10d}@anchor{library/tkinter colorchooser tkinter-colorchooser-color-choosing-dialog}@anchor{3613}
@subsection @code{tkinter.colorchooser} — Color choosing dialog


@geindex module; tkinter.colorchooser

@strong{Source code:} Lib/tkinter/colorchooser.py@footnote{https://github.com/python/cpython/tree/main/Lib/tkinter/colorchooser.py}


__________________________________________________________________


The @ref{10d,,tkinter.colorchooser} module provides the @ref{3614,,Chooser} class
as an interface to the native color picker dialog. @code{Chooser} implements
a modal color choosing dialog window. The @code{Chooser} class inherits from
the @ref{3615,,Dialog} class.

@geindex Chooser (class in tkinter.colorchooser)
@anchor{library/tkinter colorchooser tkinter colorchooser Chooser}@anchor{3614}
@deffn {Class} tkinter.colorchooser.Chooser (master=None, **options)
@end deffn

@geindex askcolor() (in module tkinter.colorchooser)
@anchor{library/tkinter colorchooser tkinter colorchooser askcolor}@anchor{3616}
@deffn {Function} tkinter.colorchooser.askcolor (color=None, **options)

Create a color choosing dialog. A call to this method will show the window,
wait for the user to make a selection, and return the selected color (or
@code{None}) to the caller.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{10e,,tkinter.commondialog}

Tkinter standard dialog module
@end table


@node tkinter font — Tkinter font wrapper,Tkinter Dialogs,tkinter colorchooser — Color choosing dialog,Graphical User Interfaces with Tk
@anchor{library/tkinter font doc}@anchor{3617}@anchor{library/tkinter font module-tkinter font}@anchor{111}@anchor{library/tkinter font tkinter-font-tkinter-font-wrapper}@anchor{3618}
@subsection @code{tkinter.font} — Tkinter font wrapper


@geindex module; tkinter.font

@strong{Source code:} Lib/tkinter/font.py@footnote{https://github.com/python/cpython/tree/main/Lib/tkinter/font.py}


__________________________________________________________________


The @ref{111,,tkinter.font} module provides the @ref{1183,,Font} class for creating
and using named fonts.

The different font weights and slants are:

@geindex NORMAL (in module tkinter.font)
@geindex BOLD (in module tkinter.font)
@geindex ITALIC (in module tkinter.font)
@geindex ROMAN (in module tkinter.font)
@anchor{library/tkinter font tkinter font NORMAL}@anchor{3619}
@deffn {Data} tkinter.font.NORMAL
@anchor{library/tkinter font tkinter font BOLD}@anchor{361a}
@deffnx {Data} tkinter.font.BOLD
@anchor{library/tkinter font tkinter font ITALIC}@anchor{361b}
@deffnx {Data} tkinter.font.ITALIC
@anchor{library/tkinter font tkinter font ROMAN}@anchor{361c}
@deffnx {Data} tkinter.font.ROMAN
@end deffn

@geindex Font (class in tkinter.font)
@anchor{library/tkinter font tkinter font Font}@anchor{1183}
@deffn {Class} tkinter.font.Font (root=None, font=None, name=None, exists=False, **options)

The @ref{1183,,Font} class represents a named font. @emph{Font} instances are given
unique names and can be specified by their family, size, and style
configuration. Named fonts are Tk’s method of creating and identifying
fonts as a single object, rather than specifying a font by its attributes
with each occurrence.

@quotation

arguments:

@quotation


@display
@emph{font} - font specifier tuple (family, size, options)@w{ }
@emph{name} - unique font name@w{ }
@emph{exists} - self points to existing named font if true@w{ }
@end display


@end quotation

additional keyword options (ignored if @emph{font} is specified):

@quotation


@display
@emph{family} - font family i.e. Courier, Times@w{ }
@emph{size} - font size@w{ }
@display
If @emph{size} is positive it is interpreted as size in points.@w{ }
If @emph{size} is a negative number its absolute value is treated@w{ }
as size in pixels.@w{ }
@end display
@emph{weight} - font emphasis (NORMAL, BOLD)@w{ }
@emph{slant} - ROMAN, ITALIC@w{ }
@emph{underline} - font underlining (0 - none, 1 - underline)@w{ }
@emph{overstrike} - font strikeout (0 - none, 1 - strikeout)@w{ }
@end display


@end quotation
@end quotation

@geindex actual() (tkinter.font.Font method)
@anchor{library/tkinter font tkinter font Font actual}@anchor{361d}
@deffn {Method} actual (option=None, displayof=None)

Return the attributes of the font.
@end deffn

@geindex cget() (tkinter.font.Font method)
@anchor{library/tkinter font tkinter font Font cget}@anchor{361e}
@deffn {Method} cget (option)

Retrieve an attribute of the font.
@end deffn

@geindex config() (tkinter.font.Font method)
@anchor{library/tkinter font tkinter font Font config}@anchor{361f}
@deffn {Method} config (**options)

Modify attributes of the font.
@end deffn

@geindex copy() (tkinter.font.Font method)
@anchor{library/tkinter font tkinter font Font copy}@anchor{3620}
@deffn {Method} copy ()

Return new instance of the current font.
@end deffn

@geindex measure() (tkinter.font.Font method)
@anchor{library/tkinter font tkinter font Font measure}@anchor{3621}
@deffn {Method} measure (text, displayof=None)

Return amount of space the text would occupy on the specified display
when formatted in the current font. If no display is specified then the
main application window is assumed.
@end deffn

@geindex metrics() (tkinter.font.Font method)
@anchor{library/tkinter font tkinter font Font metrics}@anchor{3622}
@deffn {Method} metrics (*options, **kw)

Return font-specific data.
Options include:


@table @asis

@item @emph{ascent} - distance between baseline and highest point that a

character of the font can occupy

@item @emph{descent} - distance between baseline and lowest point that a

character of the font can occupy

@item @emph{linespace} - minimum vertical separation necessary between any two

characters of the font that ensures no vertical overlap between lines.
@end table

@emph{fixed} - 1 if font is fixed-width else 0
@end deffn
@end deffn

@geindex families() (in module tkinter.font)
@anchor{library/tkinter font tkinter font families}@anchor{3623}
@deffn {Function} tkinter.font.families (root=None, displayof=None)

Return the different font families.
@end deffn

@geindex names() (in module tkinter.font)
@anchor{library/tkinter font tkinter font names}@anchor{3624}
@deffn {Function} tkinter.font.names (root=None)

Return the names of defined fonts.
@end deffn

@geindex nametofont() (in module tkinter.font)
@anchor{library/tkinter font tkinter font nametofont}@anchor{1185}
@deffn {Function} tkinter.font.nametofont (name, root=None)

Return a @ref{1183,,Font} representation of a tk named font.


Changed in version 3.10: The @emph{root} parameter was added.

@end deffn

@node Tkinter Dialogs,tkinter messagebox — Tkinter message prompts,tkinter font — Tkinter font wrapper,Graphical User Interfaces with Tk
@anchor{library/dialog doc}@anchor{3625}@anchor{library/dialog tkinter-dialogs}@anchor{3626}
@subsection Tkinter Dialogs


@menu
* tkinter.simpledialog — Standard Tkinter input dialogs: tkinter simpledialog — Standard Tkinter input dialogs. 
* tkinter.filedialog — File selection dialogs: tkinter filedialog — File selection dialogs. 
* tkinter.commondialog — Dialog window templates: tkinter commondialog — Dialog window templates. 

@end menu

@node tkinter simpledialog — Standard Tkinter input dialogs,tkinter filedialog — File selection dialogs,,Tkinter Dialogs
@anchor{library/dialog module-tkinter simpledialog}@anchor{114}@anchor{library/dialog tkinter-simpledialog-standard-tkinter-input-dialogs}@anchor{3627}
@subsubsection @code{tkinter.simpledialog} — Standard Tkinter input dialogs


@geindex module; tkinter.simpledialog

@strong{Source code:} Lib/tkinter/simpledialog.py@footnote{https://github.com/python/cpython/tree/main/Lib/tkinter/simpledialog.py}


__________________________________________________________________


The @ref{114,,tkinter.simpledialog} module contains convenience classes and
functions for creating simple modal dialogs to get a value from the user.

@geindex askfloat() (in module tkinter.simpledialog)
@geindex askinteger() (in module tkinter.simpledialog)
@geindex askstring() (in module tkinter.simpledialog)
@anchor{library/dialog tkinter simpledialog askfloat}@anchor{3628}
@deffn {Function} tkinter.simpledialog.askfloat (title, prompt, **kw)
@anchor{library/dialog tkinter simpledialog askinteger}@anchor{1186}
@deffnx {Function} tkinter.simpledialog.askinteger (title, prompt, **kw)
@anchor{library/dialog tkinter simpledialog askstring}@anchor{3629}
@deffnx {Function} tkinter.simpledialog.askstring (title, prompt, **kw)

The above three functions provide dialogs that prompt the user to enter a value
of the desired type.
@end deffn

@geindex Dialog (class in tkinter.simpledialog)
@anchor{library/dialog tkinter simpledialog Dialog}@anchor{362a}
@deffn {Class} tkinter.simpledialog.Dialog (parent, title=None)

The base class for custom dialogs.

@quotation

@geindex body() (tkinter.simpledialog.Dialog method)
@anchor{library/dialog tkinter simpledialog Dialog body}@anchor{362b}
@deffn {Method} body (master)

Override to construct the dialog’s interface and return the widget that
should have initial focus.
@end deffn

@geindex buttonbox() (tkinter.simpledialog.Dialog method)
@anchor{library/dialog tkinter simpledialog Dialog buttonbox}@anchor{362c}
@deffn {Method} buttonbox ()

Default behaviour adds OK and Cancel buttons. Override for custom button
layouts.
@end deffn
@end quotation
@end deffn

@node tkinter filedialog — File selection dialogs,tkinter commondialog — Dialog window templates,tkinter simpledialog — Standard Tkinter input dialogs,Tkinter Dialogs
@anchor{library/dialog module-tkinter filedialog}@anchor{110}@anchor{library/dialog tkinter-filedialog-file-selection-dialogs}@anchor{362d}
@subsubsection @code{tkinter.filedialog} — File selection dialogs


@geindex module; tkinter.filedialog

@strong{Source code:} Lib/tkinter/filedialog.py@footnote{https://github.com/python/cpython/tree/main/Lib/tkinter/filedialog.py}


__________________________________________________________________


The @ref{110,,tkinter.filedialog} module provides classes and factory functions for
creating file/directory selection windows.

@menu
* Native Load/Save Dialogs:: 

@end menu

@node Native Load/Save Dialogs,,,tkinter filedialog — File selection dialogs
@anchor{library/dialog native-load-save-dialogs}@anchor{362e}
@subsubsection Native Load/Save Dialogs


The following classes and functions provide file dialog windows that combine a
native look-and-feel with configuration options to customize behaviour.
The following keyword arguments are applicable to the classes and functions
listed below:

@quotation


@display
@emph{parent} - the window to place the dialog on top of@w{ }
@end display




@display
@emph{title} - the title of the window@w{ }
@end display




@display
@emph{initialdir} - the directory that the dialog starts in@w{ }
@end display




@display
@emph{initialfile} - the file selected upon opening of the dialog@w{ }
@end display




@display
@emph{filetypes} - a sequence of (label, pattern) tuples, ‘*’ wildcard is allowed@w{ }
@end display




@display
@emph{defaultextension} - default extension to append to file (save dialogs)@w{ }
@end display




@display
@emph{multiple} - when true, selection of multiple items is allowed@w{ }
@end display


@end quotation

@strong{Static factory functions}

The below functions when called create a modal, native look-and-feel dialog,
wait for the user’s selection, then return the selected value(s) or @code{None} to the
caller.

@geindex askopenfile() (in module tkinter.filedialog)
@geindex askopenfiles() (in module tkinter.filedialog)
@anchor{library/dialog tkinter filedialog askopenfile}@anchor{362f}
@deffn {Function} tkinter.filedialog.askopenfile (mode='r', **options)
@anchor{library/dialog tkinter filedialog askopenfiles}@anchor{3630}
@deffnx {Function} tkinter.filedialog.askopenfiles (mode='r', **options)

The above two functions create an @ref{3631,,Open} dialog and return the opened
file object(s) in read-only mode.
@end deffn

@geindex asksaveasfile() (in module tkinter.filedialog)
@anchor{library/dialog tkinter filedialog asksaveasfile}@anchor{3632}
@deffn {Function} tkinter.filedialog.asksaveasfile (mode='w', **options)

Create a @ref{3633,,SaveAs} dialog and return a file object opened in write-only mode.
@end deffn

@geindex askopenfilename() (in module tkinter.filedialog)
@geindex askopenfilenames() (in module tkinter.filedialog)
@anchor{library/dialog tkinter filedialog askopenfilename}@anchor{3634}
@deffn {Function} tkinter.filedialog.askopenfilename (**options)
@anchor{library/dialog tkinter filedialog askopenfilenames}@anchor{3635}
@deffnx {Function} tkinter.filedialog.askopenfilenames (**options)

The above two functions create an @ref{3631,,Open} dialog and return the
selected filename(s) that correspond to existing file(s).
@end deffn

@geindex asksaveasfilename() (in module tkinter.filedialog)
@anchor{library/dialog tkinter filedialog asksaveasfilename}@anchor{3636}
@deffn {Function} tkinter.filedialog.asksaveasfilename (**options)

Create a @ref{3633,,SaveAs} dialog and return the selected filename.
@end deffn

@geindex askdirectory() (in module tkinter.filedialog)
@anchor{library/dialog tkinter filedialog askdirectory}@anchor{3637}
@deffn {Function} tkinter.filedialog.askdirectory (**options)


@display
Prompt user to select a directory.@w{ }
Additional keyword option:@w{ }
@display
@emph{mustexist} - determines if selection must be an existing directory.@w{ }
@end display
@end display


@end deffn

@geindex Open (class in tkinter.filedialog)
@geindex SaveAs (class in tkinter.filedialog)
@anchor{library/dialog tkinter filedialog Open}@anchor{3631}
@deffn {Class} tkinter.filedialog.Open (master=None, **options)
@anchor{library/dialog tkinter filedialog SaveAs}@anchor{3633}
@deffnx {Class} tkinter.filedialog.SaveAs (master=None, **options)

The above two classes provide native dialog windows for saving and loading
files.
@end deffn

@strong{Convenience classes}

The below classes are used for creating file/directory windows from scratch.
These do not emulate the native look-and-feel of the platform.

@geindex Directory (class in tkinter.filedialog)
@anchor{library/dialog tkinter filedialog Directory}@anchor{3638}
@deffn {Class} tkinter.filedialog.Directory (master=None, **options)

Create a dialog prompting the user to select a directory.
@end deffn

@cartouche
@quotation Note 
The @emph{FileDialog} class should be subclassed for custom event
handling and behaviour.
@end quotation
@end cartouche

@geindex FileDialog (class in tkinter.filedialog)
@anchor{library/dialog tkinter filedialog FileDialog}@anchor{3639}
@deffn {Class} tkinter.filedialog.FileDialog (master, title=None)

Create a basic file selection dialog.

@geindex cancel_command() (tkinter.filedialog.FileDialog method)
@anchor{library/dialog tkinter filedialog FileDialog cancel_command}@anchor{363a}
@deffn {Method} cancel_command (event=None)

Trigger the termination of the dialog window.
@end deffn

@geindex dirs_double_event() (tkinter.filedialog.FileDialog method)
@anchor{library/dialog tkinter filedialog FileDialog dirs_double_event}@anchor{363b}
@deffn {Method} dirs_double_event (event)

Event handler for double-click event on directory.
@end deffn

@geindex dirs_select_event() (tkinter.filedialog.FileDialog method)
@anchor{library/dialog tkinter filedialog FileDialog dirs_select_event}@anchor{363c}
@deffn {Method} dirs_select_event (event)

Event handler for click event on directory.
@end deffn

@geindex files_double_event() (tkinter.filedialog.FileDialog method)
@anchor{library/dialog tkinter filedialog FileDialog files_double_event}@anchor{363d}
@deffn {Method} files_double_event (event)

Event handler for double-click event on file.
@end deffn

@geindex files_select_event() (tkinter.filedialog.FileDialog method)
@anchor{library/dialog tkinter filedialog FileDialog files_select_event}@anchor{363e}
@deffn {Method} files_select_event (event)

Event handler for single-click event on file.
@end deffn

@geindex filter_command() (tkinter.filedialog.FileDialog method)
@anchor{library/dialog tkinter filedialog FileDialog filter_command}@anchor{363f}
@deffn {Method} filter_command (event=None)

Filter the files by directory.
@end deffn

@geindex get_filter() (tkinter.filedialog.FileDialog method)
@anchor{library/dialog tkinter filedialog FileDialog get_filter}@anchor{3640}
@deffn {Method} get_filter ()

Retrieve the file filter currently in use.
@end deffn

@geindex get_selection() (tkinter.filedialog.FileDialog method)
@anchor{library/dialog tkinter filedialog FileDialog get_selection}@anchor{3641}
@deffn {Method} get_selection ()

Retrieve the currently selected item.
@end deffn

@geindex go() (tkinter.filedialog.FileDialog method)
@anchor{library/dialog tkinter filedialog FileDialog go}@anchor{3642}
@deffn {Method} go (dir_or_file=os.curdir, pattern='*', default='@w{'}, key=None)

Render dialog and start event loop.
@end deffn

@geindex ok_event() (tkinter.filedialog.FileDialog method)
@anchor{library/dialog tkinter filedialog FileDialog ok_event}@anchor{3643}
@deffn {Method} ok_event (event)

Exit dialog returning current selection.
@end deffn

@geindex quit() (tkinter.filedialog.FileDialog method)
@anchor{library/dialog tkinter filedialog FileDialog quit}@anchor{3644}
@deffn {Method} quit (how=None)

Exit dialog returning filename, if any.
@end deffn

@geindex set_filter() (tkinter.filedialog.FileDialog method)
@anchor{library/dialog tkinter filedialog FileDialog set_filter}@anchor{3645}
@deffn {Method} set_filter (dir, pat)

Set the file filter.
@end deffn

@geindex set_selection() (tkinter.filedialog.FileDialog method)
@anchor{library/dialog tkinter filedialog FileDialog set_selection}@anchor{3646}
@deffn {Method} set_selection (file)

Update the current file selection to @emph{file}.
@end deffn
@end deffn

@geindex LoadFileDialog (class in tkinter.filedialog)
@anchor{library/dialog tkinter filedialog LoadFileDialog}@anchor{3647}
@deffn {Class} tkinter.filedialog.LoadFileDialog (master, title=None)

A subclass of FileDialog that creates a dialog window for selecting an
existing file.

@geindex ok_command() (tkinter.filedialog.LoadFileDialog method)
@anchor{library/dialog tkinter filedialog LoadFileDialog ok_command}@anchor{3648}
@deffn {Method} ok_command ()

Test that a file is provided and that the selection indicates an
already existing file.
@end deffn
@end deffn

@geindex SaveFileDialog (class in tkinter.filedialog)
@anchor{library/dialog tkinter filedialog SaveFileDialog}@anchor{3649}
@deffn {Class} tkinter.filedialog.SaveFileDialog (master, title=None)

A subclass of FileDialog that creates a dialog window for selecting a
destination file.

@geindex ok_command() (tkinter.filedialog.SaveFileDialog method)
@anchor{library/dialog tkinter filedialog SaveFileDialog ok_command}@anchor{364a}
@deffn {Method} ok_command ()

Test whether or not the selection points to a valid file that is not a
directory. Confirmation is required if an already existing file is
selected.
@end deffn
@end deffn

@node tkinter commondialog — Dialog window templates,,tkinter filedialog — File selection dialogs,Tkinter Dialogs
@anchor{library/dialog module-tkinter commondialog}@anchor{10e}@anchor{library/dialog tkinter-commondialog-dialog-window-templates}@anchor{364b}
@subsubsection @code{tkinter.commondialog} — Dialog window templates


@geindex module; tkinter.commondialog

@strong{Source code:} Lib/tkinter/commondialog.py@footnote{https://github.com/python/cpython/tree/main/Lib/tkinter/commondialog.py}


__________________________________________________________________


The @ref{10e,,tkinter.commondialog} module provides the @ref{3615,,Dialog} class that
is the base class for dialogs defined in other supporting modules.

@geindex Dialog (class in tkinter.commondialog)
@anchor{library/dialog tkinter commondialog Dialog}@anchor{3615}
@deffn {Class} tkinter.commondialog.Dialog (master=None, **options)

@geindex show() (tkinter.commondialog.Dialog method)
@anchor{library/dialog tkinter commondialog Dialog show}@anchor{364c}
@deffn {Method} show (color=None, **options)

Render the Dialog window.
@end deffn
@end deffn


@subsubheading See also


Modules @ref{112,,tkinter.messagebox}, @ref{1561,,Reading and Writing Files}


@node tkinter messagebox — Tkinter message prompts,tkinter scrolledtext — Scrolled Text Widget,Tkinter Dialogs,Graphical User Interfaces with Tk
@anchor{library/tkinter messagebox doc}@anchor{364d}@anchor{library/tkinter messagebox module-tkinter messagebox}@anchor{112}@anchor{library/tkinter messagebox tkinter-messagebox-tkinter-message-prompts}@anchor{364e}
@subsection @code{tkinter.messagebox} — Tkinter message prompts


@geindex module; tkinter.messagebox

@strong{Source code:} Lib/tkinter/messagebox.py@footnote{https://github.com/python/cpython/tree/main/Lib/tkinter/messagebox.py}


__________________________________________________________________


The @ref{112,,tkinter.messagebox} module provides a template base class as well as
a variety of convenience methods for commonly used configurations. The message
boxes are modal and will return a subset of (True, False, OK, None, Yes, No) based on
the user’s selection. Common message box styles and layouts include but are not
limited to:


@float Figure

@image{python-figures/tk_msg,,,,png}

@end float


@geindex Message (class in tkinter.messagebox)
@anchor{library/tkinter messagebox tkinter messagebox Message}@anchor{364f}
@deffn {Class} tkinter.messagebox.Message (master=None, **options)

Create a default information message box.
@end deffn

@strong{Information message box}

@geindex showinfo() (in module tkinter.messagebox)
@anchor{library/tkinter messagebox tkinter messagebox showinfo}@anchor{3650}
@deffn {Method} tkinter.messagebox.showinfo (title=None, message=None, **options)
@end deffn

@strong{Warning message boxes}

@geindex showwarning() (in module tkinter.messagebox)
@geindex showerror() (in module tkinter.messagebox)
@anchor{library/tkinter messagebox tkinter messagebox showwarning}@anchor{3651}
@deffn {Method} tkinter.messagebox.showwarning (title=None, message=None, **options)
@anchor{library/tkinter messagebox tkinter messagebox showerror}@anchor{3652}
@deffnx {Method} tkinter.messagebox.showerror (title=None, message=None, **options)
@end deffn

@strong{Question message boxes}

@geindex askquestion() (in module tkinter.messagebox)
@geindex askokcancel() (in module tkinter.messagebox)
@geindex askretrycancel() (in module tkinter.messagebox)
@geindex askyesno() (in module tkinter.messagebox)
@geindex askyesnocancel() (in module tkinter.messagebox)
@anchor{library/tkinter messagebox tkinter messagebox askquestion}@anchor{3653}
@deffn {Method} tkinter.messagebox.askquestion (title=None, message=None, **options)
@anchor{library/tkinter messagebox tkinter messagebox askokcancel}@anchor{3654}
@deffnx {Method} tkinter.messagebox.askokcancel (title=None, message=None, **options)
@anchor{library/tkinter messagebox tkinter messagebox askretrycancel}@anchor{3655}
@deffnx {Method} tkinter.messagebox.askretrycancel (title=None, message=None, **options)
@anchor{library/tkinter messagebox tkinter messagebox askyesno}@anchor{3656}
@deffnx {Method} tkinter.messagebox.askyesno (title=None, message=None, **options)
@anchor{library/tkinter messagebox tkinter messagebox askyesnocancel}@anchor{3657}
@deffnx {Method} tkinter.messagebox.askyesnocancel (title=None, message=None, **options)
@end deffn

@node tkinter scrolledtext — Scrolled Text Widget,tkinter dnd — Drag and drop support,tkinter messagebox — Tkinter message prompts,Graphical User Interfaces with Tk
@anchor{library/tkinter scrolledtext doc}@anchor{3658}@anchor{library/tkinter scrolledtext module-tkinter scrolledtext}@anchor{113}@anchor{library/tkinter scrolledtext tkinter-scrolledtext-scrolled-text-widget}@anchor{3659}
@subsection @code{tkinter.scrolledtext} — Scrolled Text Widget


@geindex module; tkinter.scrolledtext

@strong{Source code:} Lib/tkinter/scrolledtext.py@footnote{https://github.com/python/cpython/tree/main/Lib/tkinter/scrolledtext.py}


__________________________________________________________________


The @ref{113,,tkinter.scrolledtext} module provides a class of the same name which
implements a basic text widget which has a vertical scroll bar configured to do
the “right thing.”  Using the @ref{365a,,ScrolledText} class is a lot easier than
setting up a text widget and scroll bar directly.

The text widget and scrollbar are packed together in a @code{Frame}, and the
methods of the @code{Grid} and @code{Pack} geometry managers are acquired
from the @code{Frame} object.  This allows the @ref{365a,,ScrolledText} widget to
be used directly to achieve most normal geometry management behavior.

Should more specific control be necessary, the following attributes are
available:

@geindex ScrolledText (class in tkinter.scrolledtext)
@anchor{library/tkinter scrolledtext tkinter scrolledtext ScrolledText}@anchor{365a}
@deffn {Class} tkinter.scrolledtext.ScrolledText (master=None, **kw)

@geindex frame (tkinter.scrolledtext.ScrolledText attribute)
@anchor{library/tkinter scrolledtext tkinter scrolledtext ScrolledText frame}@anchor{365b}
@deffn {Attribute} frame

The frame which surrounds the text and scroll bar widgets.
@end deffn

@geindex vbar (tkinter.scrolledtext.ScrolledText attribute)
@anchor{library/tkinter scrolledtext tkinter scrolledtext ScrolledText vbar}@anchor{365c}
@deffn {Attribute} vbar

The scroll bar widget.
@end deffn
@end deffn

@node tkinter dnd — Drag and drop support,tkinter ttk — Tk themed widgets,tkinter scrolledtext — Scrolled Text Widget,Graphical User Interfaces with Tk
@anchor{library/tkinter dnd doc}@anchor{365d}@anchor{library/tkinter dnd module-tkinter dnd}@anchor{10f}@anchor{library/tkinter dnd tkinter-dnd-drag-and-drop-support}@anchor{365e}
@subsection @code{tkinter.dnd} — Drag and drop support


@geindex module; tkinter.dnd

@strong{Source code:} Lib/tkinter/dnd.py@footnote{https://github.com/python/cpython/tree/main/Lib/tkinter/dnd.py}


__________________________________________________________________


@cartouche
@quotation Note 
This is experimental and due to be deprecated when it is replaced
with the Tk DND.
@end quotation
@end cartouche

The @ref{10f,,tkinter.dnd} module provides drag-and-drop support for objects within
a single application, within the same window or between windows. To enable an
object to be dragged, you must create an event binding for it that starts the
drag-and-drop process. Typically, you bind a ButtonPress event to a callback
function that you write (see @ref{3607,,Bindings and Events}). The function should
call @ref{365f,,dnd_start()}, where ‘source’ is the object to be dragged, and ‘event’
is the event that invoked the call (the argument to your callback function).

Selection of a target object occurs as follows:


@enumerate 

@item 
Top-down search of area under mouse for target widget
@end enumerate

@quotation


@itemize *

@item 
Target widget should have a callable @emph{dnd_accept} attribute

@item 
If @emph{dnd_accept} is not present or returns None, search moves to parent widget

@item 
If no target widget is found, then the target object is None
@end itemize
@end quotation


@enumerate 2

@item 
Call to @emph{<old_target>.dnd_leave(source, event)}

@item 
Call to @emph{<new_target>.dnd_enter(source, event)}

@item 
Call to @emph{<target>.dnd_commit(source, event)} to notify of drop

@item 
Call to @emph{<source>.dnd_end(target, event)} to signal end of drag-and-drop
@end enumerate

@geindex DndHandler (class in tkinter.dnd)
@anchor{library/tkinter dnd tkinter dnd DndHandler}@anchor{3660}
@deffn {Class} tkinter.dnd.DndHandler (source, event)

The @emph{DndHandler} class handles drag-and-drop events tracking Motion and
ButtonRelease events on the root of the event widget.

@geindex cancel() (tkinter.dnd.DndHandler method)
@anchor{library/tkinter dnd tkinter dnd DndHandler cancel}@anchor{3661}
@deffn {Method} cancel (event=None)

Cancel the drag-and-drop process.
@end deffn

@geindex finish() (tkinter.dnd.DndHandler method)
@anchor{library/tkinter dnd tkinter dnd DndHandler finish}@anchor{3662}
@deffn {Method} finish (event, commit=0)

Execute end of drag-and-drop functions.
@end deffn

@geindex on_motion() (tkinter.dnd.DndHandler method)
@anchor{library/tkinter dnd tkinter dnd DndHandler on_motion}@anchor{3663}
@deffn {Method} on_motion (event)

Inspect area below mouse for target objects while drag is performed.
@end deffn

@geindex on_release() (tkinter.dnd.DndHandler method)
@anchor{library/tkinter dnd tkinter dnd DndHandler on_release}@anchor{3664}
@deffn {Method} on_release (event)

Signal end of drag when the release pattern is triggered.
@end deffn
@end deffn

@geindex dnd_start() (in module tkinter.dnd)
@anchor{library/tkinter dnd tkinter dnd dnd_start}@anchor{365f}
@deffn {Function} tkinter.dnd.dnd_start (source, event)

Factory function for drag-and-drop process.
@end deffn


@subsubheading See also


@ref{3607,,Bindings and Events}


@node tkinter ttk — Tk themed widgets,tkinter tix — Extension widgets for Tk,tkinter dnd — Drag and drop support,Graphical User Interfaces with Tk
@anchor{library/tkinter ttk doc}@anchor{3665}@anchor{library/tkinter ttk module-tkinter ttk}@anchor{116}@anchor{library/tkinter ttk tkinter-ttk-tk-themed-widgets}@anchor{3666}
@subsection @code{tkinter.ttk} — Tk themed widgets


@geindex module; tkinter.ttk

@strong{Source code:} Lib/tkinter/ttk.py@footnote{https://github.com/python/cpython/tree/main/Lib/tkinter/ttk.py}

@geindex ttk


__________________________________________________________________


The @ref{116,,tkinter.ttk} module provides access to the Tk themed widget set,
introduced in Tk 8.5. If Python has not been compiled against Tk 8.5, this
module can still be accessed if @emph{Tile} has been installed.  The former
method using Tk 8.5 provides additional benefits including anti-aliased font
rendering under X11 and window transparency (requiring a composition
window manager on X11).

The basic idea for @ref{116,,tkinter.ttk} is to separate, to the extent possible,
the code implementing a widget’s behavior from the code implementing its
appearance.


@subsubheading See also



@table @asis

@item Tk Widget Styling Support@footnote{https://core.tcl.tk/tips/doc/trunk/tip/48.md}

A document introducing theming support for Tk
@end table


@menu
* Using Ttk:: 
* Ttk Widgets:: 
* Widget:: 
* Combobox:: 
* Spinbox:: 
* Notebook:: 
* Progressbar:: 
* Separator:: 
* Sizegrip:: 
* Treeview:: 
* Ttk Styling:: 

@end menu

@node Using Ttk,Ttk Widgets,,tkinter ttk — Tk themed widgets
@anchor{library/tkinter ttk using-ttk}@anchor{3667}
@subsubsection Using Ttk


To start using Ttk, import its module:

@example
from tkinter import ttk
@end example

To override the basic Tk widgets, the import should follow the Tk import:

@example
from tkinter import *
from tkinter.ttk import *
@end example

That code causes several @ref{116,,tkinter.ttk} widgets (@code{Button},
@code{Checkbutton}, @code{Entry}, @code{Frame}, @code{Label},
@code{LabelFrame}, @code{Menubutton}, @code{PanedWindow},
@code{Radiobutton}, @code{Scale} and @code{Scrollbar}) to
automatically replace the Tk widgets.

This has the direct benefit of using the new widgets which gives a better
look and feel across platforms; however, the replacement widgets are not
completely compatible. The main difference is that widget options such as
“fg”, “bg” and others related to widget styling are no
longer present in Ttk widgets.  Instead, use  the @code{ttk.Style} class
for improved styling effects.


@subsubheading See also



@table @asis

@item Converting existing applications to use Tile widgets@footnote{http://tktable.sourceforge.net/tile/doc/converting.txt}

A monograph (using Tcl terminology) about differences typically
encountered when moving applications to use the new widgets.
@end table


@node Ttk Widgets,Widget,Using Ttk,tkinter ttk — Tk themed widgets
@anchor{library/tkinter ttk ttk-widgets}@anchor{3668}
@subsubsection Ttk Widgets


Ttk comes with 18 widgets, twelve of which already existed in tkinter:
@code{Button}, @code{Checkbutton}, @code{Entry}, @code{Frame},
@code{Label}, @code{LabelFrame}, @code{Menubutton}, @code{PanedWindow},
@code{Radiobutton}, @code{Scale}, @code{Scrollbar}, and @ref{6c5,,Spinbox}.
The other six are new: @ref{3669,,Combobox}, @ref{366a,,Notebook},
@ref{366b,,Progressbar}, @code{Separator}, @code{Sizegrip} and
@ref{5b2,,Treeview}. And all them are subclasses of @ref{366c,,Widget}.

Using the Ttk widgets gives the application an improved look and feel.
As discussed above, there are differences in how the styling is coded.

Tk code:

@example
l1 = tkinter.Label(text="Test", fg="black", bg="white")
l2 = tkinter.Label(text="Test", fg="black", bg="white")
@end example

Ttk code:

@example
style = ttk.Style()
style.configure("BW.TLabel", foreground="black", background="white")

l1 = ttk.Label(text="Test", style="BW.TLabel")
l2 = ttk.Label(text="Test", style="BW.TLabel")
@end example

For more information about @ref{366d,,TtkStyling}, see the @ref{366e,,Style} class
documentation.

@node Widget,Combobox,Ttk Widgets,tkinter ttk — Tk themed widgets
@anchor{library/tkinter ttk widget}@anchor{366f}
@subsubsection Widget


@code{ttk.Widget} defines standard options and methods supported by Tk
themed widgets and is not supposed to be directly instantiated.

@menu
* Standard Options:: 
* Scrollable Widget Options:: 
* Label Options:: 
* Compatibility Options:: 
* Widget States:: 
* ttk.Widget: ttk Widget. 

@end menu

@node Standard Options,Scrollable Widget Options,,Widget
@anchor{library/tkinter ttk standard-options}@anchor{3670}
@subsubsection Standard Options


All the @code{ttk} Widgets accepts the following options:

@quotation


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Description

@item

class

@tab

Specifies the window class. The class is used when querying
the option database for the window’s other options, to
determine the default bindtags for the window, and to select
the widget’s default layout and style. This option is
read-only, and may only be specified when the window is
created.

@item

cursor

@tab

Specifies the mouse cursor to be used for the widget. If set
to the empty string (the default), the cursor is inherited
for the parent widget.

@item

takefocus

@tab

Determines whether the window accepts the focus during
keyboard traversal. 0, 1 or an empty string is returned.
If 0 is returned, it means that the window should be skipped
entirely during keyboard traversal. If 1, it means that the
window should receive the input focus as long as it is
viewable. And an empty string means that the traversal
scripts make the decision about whether or not to focus
on the window.

@item

style

@tab

May be used to specify a custom widget style.

@end multitable

@end quotation

@node Scrollable Widget Options,Label Options,Standard Options,Widget
@anchor{library/tkinter ttk scrollable-widget-options}@anchor{3671}
@subsubsection Scrollable Widget Options


The following options are supported by widgets that are controlled by a
scrollbar.

@quotation


@multitable {xxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Description

@item

xscrollcommand

@tab

Used to communicate with horizontal scrollbars.

When the view in the widget’s window change, the widget
will generate a Tcl command based on the scrollcommand.

Usually this option consists of the method
@code{Scrollbar.set()} of some scrollbar. This will cause
the scrollbar to be updated whenever the view in the
window changes.

@item

yscrollcommand

@tab

Used to communicate with vertical scrollbars.
For some more information, see above.

@end multitable

@end quotation

@node Label Options,Compatibility Options,Scrollable Widget Options,Widget
@anchor{library/tkinter ttk label-options}@anchor{3672}
@subsubsection Label Options


The following options are supported by labels, buttons and other button-like
widgets.

@quotation


@multitable {xxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Description

@item

text

@tab

Specifies a text string to be displayed inside the widget.

@item

textvariable

@tab

Specifies a name whose value will be used in place of the
text option resource.

@item

underline

@tab

If set, specifies the index (0-based) of a character to
underline in the text string. The underline character is
used for mnemonic activation.

@item

image

@tab

Specifies an image to display. This is a list of 1 or more
elements. The first element is the default image name. The
rest of the list if a sequence of statespec/value pairs as
defined by @ref{1196,,Style.map()}, specifying different images
to use when the widget is in a particular state or a
combination of states. All images in the list should have
the same size.

@item

compound

@tab

Specifies how to display the image relative to the text,
in the case both text and images options are present.
Valid values are:


@itemize *

@item 
text: display text only

@item 
image: display image only

@item 
top, bottom, left, right: display image above, below,
left of, or right of the text, respectively.

@item 
none: the default. display the image if present,
otherwise the text.
@end itemize

@item

width

@tab

If greater than zero, specifies how much space, in
character widths, to allocate for the text label, if less
than zero, specifies a minimum width. If zero or
unspecified, the natural width of the text label is used.

@end multitable

@end quotation

@node Compatibility Options,Widget States,Label Options,Widget
@anchor{library/tkinter ttk compatibility-options}@anchor{3673}
@subsubsection Compatibility Options


@quotation


@multitable {xxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Description

@item

state

@tab

May be set to “normal” or “disabled” to control the “disabled”
state bit. This is a write-only option: setting it changes the
widget state, but the @ref{3674,,Widget.state()} method does not
affect this option.

@end multitable

@end quotation

@node Widget States,ttk Widget,Compatibility Options,Widget
@anchor{library/tkinter ttk widget-states}@anchor{3675}
@subsubsection Widget States


The widget state is a bitmap of independent state flags.

@quotation


@multitable {xxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Flag

@tab

Description

@item

active

@tab

The mouse cursor is over the widget and pressing a mouse
button will cause some action to occur

@item

disabled

@tab

Widget is disabled under program control

@item

focus

@tab

Widget has keyboard focus

@item

pressed

@tab

Widget is being pressed

@item

selected

@tab

“On”, “true”, or “current” for things like Checkbuttons and
radiobuttons

@item

background

@tab

Windows and Mac have a notion of an “active” or foreground
window. The @emph{background} state is set for widgets in a
background window, and cleared for those in the foreground
window

@item

readonly

@tab

Widget should not allow user modification

@item

alternate

@tab

A widget-specific alternate display format

@item

invalid

@tab

The widget’s value is invalid

@end multitable

@end quotation

A state specification is a sequence of state names, optionally prefixed with
an exclamation point indicating that the bit is off.

@node ttk Widget,,Widget States,Widget
@anchor{library/tkinter ttk ttk-widget}@anchor{3676}
@subsubsection ttk.Widget


Besides the methods described below, the @code{ttk.Widget} supports the
methods @code{tkinter.Widget.cget()} and @code{tkinter.Widget.configure()}.

@geindex Widget (class in tkinter.ttk)
@anchor{library/tkinter ttk tkinter ttk Widget}@anchor{366c}
@deffn {Class} tkinter.ttk.Widget

@geindex identify() (tkinter.ttk.Widget method)
@anchor{library/tkinter ttk tkinter ttk Widget identify}@anchor{3677}
@deffn {Method} identify (x, y)

Returns the name of the element at position @emph{x} @emph{y}, or the empty string
if the point does not lie within any element.

@emph{x} and @emph{y} are pixel coordinates relative to the widget.
@end deffn

@geindex instate() (tkinter.ttk.Widget method)
@anchor{library/tkinter ttk tkinter ttk Widget instate}@anchor{3678}
@deffn {Method} instate (statespec, callback=None, *args, **kw)

Test the widget’s state. If a callback is not specified, returns @code{True}
if the widget state matches @emph{statespec} and @code{False} otherwise. If callback
is specified then it is called with args if widget state matches
@emph{statespec}.
@end deffn

@geindex state() (tkinter.ttk.Widget method)
@anchor{library/tkinter ttk tkinter ttk Widget state}@anchor{3674}
@deffn {Method} state (statespec=None)

Modify or inquire widget state. If @emph{statespec} is specified, sets the
widget state according to it and return a new @emph{statespec} indicating
which flags were changed. If @emph{statespec} is not specified, returns
the currently-enabled state flags.
@end deffn

@emph{statespec} will usually be a list or a tuple.
@end deffn

@node Combobox,Spinbox,Widget,tkinter ttk — Tk themed widgets
@anchor{library/tkinter ttk combobox}@anchor{3679}
@subsubsection Combobox


The @code{ttk.Combobox} widget combines a text field with a pop-down list of
values. This widget is a subclass of @code{Entry}.

Besides the methods inherited from @ref{366c,,Widget}: @code{Widget.cget()},
@code{Widget.configure()}, @ref{3677,,Widget.identify()}, @ref{3678,,Widget.instate()}
and @ref{3674,,Widget.state()}, and the following inherited from @code{Entry}:
@code{Entry.bbox()}, @code{Entry.delete()}, @code{Entry.icursor()},
@code{Entry.index()}, @code{Entry.insert()}, @code{Entry.selection()},
@code{Entry.xview()}, it has some other methods, described at
@code{ttk.Combobox}.

@menu
* Options:: 
* Virtual events:: 
* ttk.Combobox: ttk Combobox. 

@end menu

@node Options,Virtual events,,Combobox
@anchor{library/tkinter ttk options}@anchor{367a}
@subsubsection Options


This widget accepts the following specific options:

@quotation


@multitable {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Description

@item

exportselection

@tab

Boolean value. If set, the widget selection is linked
to the Window Manager selection (which can be returned
by invoking Misc.selection_get, for example).

@item

justify

@tab

Specifies how the text is aligned within the widget.
One of “left”, “center”, or “right”.

@item

height

@tab

Specifies the height of the pop-down listbox, in rows.

@item

postcommand

@tab

A script (possibly registered with Misc.register) that
is called immediately before displaying the values. It
may specify which values to display.

@item

state

@tab

One of “normal”, “readonly”, or “disabled”. In the
“readonly” state, the value may not be edited directly,
and the user can only selection of the values from the
dropdown list. In the “normal” state, the text field is
directly editable. In the “disabled” state, no
interaction is possible.

@item

textvariable

@tab

Specifies a name whose value is linked to the widget
value. Whenever the value associated with that name
changes, the widget value is updated, and vice versa.
See @code{tkinter.StringVar}.

@item

values

@tab

Specifies the list of values to display in the
drop-down listbox.

@item

width

@tab

Specifies an integer value indicating the desired width
of the entry window, in average-size characters of the
widget’s font.

@end multitable

@end quotation

@node Virtual events,ttk Combobox,Options,Combobox
@anchor{library/tkinter ttk virtual-events}@anchor{367b}
@subsubsection Virtual events


The combobox widgets generates a @strong{<<ComboboxSelected>>} virtual event
when the user selects an element from the list of values.

@node ttk Combobox,,Virtual events,Combobox
@anchor{library/tkinter ttk ttk-combobox}@anchor{367c}
@subsubsection ttk.Combobox


@geindex Combobox (class in tkinter.ttk)
@anchor{library/tkinter ttk tkinter ttk Combobox}@anchor{3669}
@deffn {Class} tkinter.ttk.Combobox

@geindex current() (tkinter.ttk.Combobox method)
@anchor{library/tkinter ttk tkinter ttk Combobox current}@anchor{367d}
@deffn {Method} current (newindex=None)

If @emph{newindex} is specified, sets the combobox value to the element
position @emph{newindex}. Otherwise, returns the index of the current value or
-1 if the current value is not in the values list.
@end deffn

@geindex get() (tkinter.ttk.Combobox method)
@anchor{library/tkinter ttk tkinter ttk Combobox get}@anchor{367e}
@deffn {Method} get ()

Returns the current value of the combobox.
@end deffn

@geindex set() (tkinter.ttk.Combobox method)
@anchor{library/tkinter ttk tkinter ttk Combobox set}@anchor{367f}
@deffn {Method} set (value)

Sets the value of the combobox to @emph{value}.
@end deffn
@end deffn

@node Spinbox,Notebook,Combobox,tkinter ttk — Tk themed widgets
@anchor{library/tkinter ttk spinbox}@anchor{3680}
@subsubsection Spinbox


The @code{ttk.Spinbox} widget is a @code{ttk.Entry} enhanced with increment
and decrement arrows.  It can be used for numbers or lists of string values.
This widget is a subclass of @code{Entry}.

Besides the methods inherited from @ref{366c,,Widget}: @code{Widget.cget()},
@code{Widget.configure()}, @ref{3677,,Widget.identify()}, @ref{3678,,Widget.instate()}
and @ref{3674,,Widget.state()}, and the following inherited from @code{Entry}:
@code{Entry.bbox()}, @code{Entry.delete()}, @code{Entry.icursor()},
@code{Entry.index()}, @code{Entry.insert()}, @code{Entry.xview()},
it has some other methods, described at @code{ttk.Spinbox}.

@menu
* Options: Options<2>. 
* Virtual events: Virtual events<2>. 
* ttk.Spinbox: ttk Spinbox. 

@end menu

@node Options<2>,Virtual events<2>,,Spinbox
@anchor{library/tkinter ttk id1}@anchor{3681}
@subsubsection Options


This widget accepts the following specific options:

@quotation
@end quotation


@multitable {xxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Description

@item

from

@tab

Float value.  If set, this is the minimum value to
which the decrement button will decrement.  Must be
spelled as @code{from_} when used as an argument, since
@code{from} is a Python keyword.

@item

to

@tab

Float value.  If set, this is the maximum value to
which the increment button will increment.

@item

increment

@tab

Float value.  Specifies the amount which the
increment/decrement buttons change the
value. Defaults to 1.0.

@item

values

@tab

Sequence of string or float values.  If specified,
the increment/decrement buttons will cycle through
the items in this sequence rather than incrementing
or decrementing numbers.

@item

wrap

@tab

Boolean value.  If @code{True}, increment and decrement
buttons will cycle from the @code{to} value to the
@code{from} value or the @code{from} value to the @code{to}
value, respectively.

@item

format

@tab

String value.  This specifies the format of numbers
set by the increment/decrement buttons.  It must be
in the form “%W.Pf”, where W is the padded width of
the value, P is the precision, and ‘%’ and ‘f’ are
literal.

@item

command

@tab

Python callable.  Will be called with no arguments
whenever either of the increment or decrement buttons
are pressed.

@end multitable


@node Virtual events<2>,ttk Spinbox,Options<2>,Spinbox
@anchor{library/tkinter ttk id2}@anchor{3682}
@subsubsection Virtual events


The spinbox widget generates an @strong{<<Increment>>} virtual event when the
user presses <Up>, and a @strong{<<Decrement>>} virtual event when the user
presses <Down>.

@node ttk Spinbox,,Virtual events<2>,Spinbox
@anchor{library/tkinter ttk ttk-spinbox}@anchor{3683}
@subsubsection ttk.Spinbox


@geindex Spinbox (class in tkinter.ttk)
@anchor{library/tkinter ttk tkinter ttk Spinbox}@anchor{6c5}
@deffn {Class} tkinter.ttk.Spinbox

@geindex get() (tkinter.ttk.Spinbox method)
@anchor{library/tkinter ttk tkinter ttk Spinbox get}@anchor{3684}
@deffn {Method} get ()

Returns the current value of the spinbox.
@end deffn

@geindex set() (tkinter.ttk.Spinbox method)
@anchor{library/tkinter ttk tkinter ttk Spinbox set}@anchor{3685}
@deffn {Method} set (value)

Sets the value of the spinbox to @emph{value}.
@end deffn
@end deffn

@node Notebook,Progressbar,Spinbox,tkinter ttk — Tk themed widgets
@anchor{library/tkinter ttk notebook}@anchor{3686}
@subsubsection Notebook


Ttk Notebook widget manages a collection of windows and displays a single
one at a time. Each child window is associated with a tab, which the user
may select to change the currently-displayed window.

@menu
* Options: Options<3>. 
* Tab Options:: 
* Tab Identifiers:: 
* Virtual Events:: 
* ttk.Notebook: ttk Notebook. 

@end menu

@node Options<3>,Tab Options,,Notebook
@anchor{library/tkinter ttk id3}@anchor{3687}
@subsubsection Options


This widget accepts the following specific options:

@quotation


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Description

@item

height

@tab

If present and greater than zero, specifies the desired height
of the pane area (not including internal padding or tabs).
Otherwise, the maximum height of all panes is used.

@item

padding

@tab

Specifies the amount of extra space to add around the outside
of the notebook. The padding is a list up to four length
specifications left top right bottom. If fewer than four
elements are specified, bottom defaults to top, right defaults
to left, and top defaults to left.

@item

width

@tab

If present and greater than zero, specified the desired width
of the pane area (not including internal padding). Otherwise,
the maximum width of all panes is used.

@end multitable

@end quotation

@node Tab Options,Tab Identifiers,Options<3>,Notebook
@anchor{library/tkinter ttk tab-options}@anchor{3688}
@subsubsection Tab Options


There are also specific options for tabs:

@quotation


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Description

@item

state

@tab

Either “normal”, “disabled” or “hidden”. If “disabled”, then
the tab is not selectable. If “hidden”, then the tab is not
shown.

@item

sticky

@tab

Specifies how the child window is positioned within the pane
area. Value is a string containing zero or more of the
characters “n”, “s”, “e” or “w”. Each letter refers to a
side (north, south, east or west) that the child window will
stick to, as per the @code{grid()} geometry manager.

@item

padding

@tab

Specifies the amount of extra space to add between the
notebook and this pane. Syntax is the same as for the option
padding used by this widget.

@item

text

@tab

Specifies a text to be displayed in the tab.

@item

image

@tab

Specifies an image to display in the tab. See the option
image described in @ref{366c,,Widget}.

@item

compound

@tab

Specifies how to display the image relative to the text, in
the case both options text and image are present. See
@ref{3672,,Label Options} for legal values.

@item

underline

@tab

Specifies the index (0-based) of a character to underline in
the text string. The underlined character is used for
mnemonic activation if @ref{3689,,Notebook.enable_traversal()} is
called.

@end multitable

@end quotation

@node Tab Identifiers,Virtual Events,Tab Options,Notebook
@anchor{library/tkinter ttk tab-identifiers}@anchor{368a}
@subsubsection Tab Identifiers


The tab_id present in several methods of @code{ttk.Notebook} may take any
of the following forms:


@itemize *

@item 
An integer between zero and the number of tabs

@item 
The name of a child window

@item 
A positional specification of the form “@@x,y”, which identifies the tab

@item 
The literal string “current”, which identifies the currently-selected tab

@item 
The literal string “end”, which returns the number of tabs (only valid for
@ref{368b,,Notebook.index()})
@end itemize

@node Virtual Events,ttk Notebook,Tab Identifiers,Notebook
@anchor{library/tkinter ttk id4}@anchor{368c}
@subsubsection Virtual Events


This widget generates a @strong{<<NotebookTabChanged>>} virtual event after a new
tab is selected.

@node ttk Notebook,,Virtual Events,Notebook
@anchor{library/tkinter ttk ttk-notebook}@anchor{368d}
@subsubsection ttk.Notebook


@geindex Notebook (class in tkinter.ttk)
@anchor{library/tkinter ttk tkinter ttk Notebook}@anchor{366a}
@deffn {Class} tkinter.ttk.Notebook

@geindex add() (tkinter.ttk.Notebook method)
@anchor{library/tkinter ttk tkinter ttk Notebook add}@anchor{368e}
@deffn {Method} add (child, **kw)

Adds a new tab to the notebook.

If window is currently managed by the notebook but hidden, it is
restored to its previous position.

See @ref{3688,,Tab Options} for the list of available options.
@end deffn

@geindex forget() (tkinter.ttk.Notebook method)
@anchor{library/tkinter ttk tkinter ttk Notebook forget}@anchor{368f}
@deffn {Method} forget (tab_id)

Removes the tab specified by @emph{tab_id}, unmaps and unmanages the
associated window.
@end deffn

@geindex hide() (tkinter.ttk.Notebook method)
@anchor{library/tkinter ttk tkinter ttk Notebook hide}@anchor{3690}
@deffn {Method} hide (tab_id)

Hides the tab specified by @emph{tab_id}.

The tab will not be displayed, but the associated window remains
managed by the notebook and its configuration remembered. Hidden tabs
may be restored with the @ref{368e,,add()} command.
@end deffn

@geindex identify() (tkinter.ttk.Notebook method)
@anchor{library/tkinter ttk tkinter ttk Notebook identify}@anchor{3691}
@deffn {Method} identify (x, y)

Returns the name of the tab element at position @emph{x}, @emph{y}, or the empty
string if none.
@end deffn

@geindex index() (tkinter.ttk.Notebook method)
@anchor{library/tkinter ttk tkinter ttk Notebook index}@anchor{368b}
@deffn {Method} index (tab_id)

Returns the numeric index of the tab specified by @emph{tab_id}, or the total
number of tabs if @emph{tab_id} is the string “end”.
@end deffn

@geindex insert() (tkinter.ttk.Notebook method)
@anchor{library/tkinter ttk tkinter ttk Notebook insert}@anchor{3692}
@deffn {Method} insert (pos, child, **kw)

Inserts a pane at the specified position.

@emph{pos} is either the string “end”, an integer index, or the name of a
managed child. If @emph{child} is already managed by the notebook, moves it to
the specified position.

See @ref{3688,,Tab Options} for the list of available options.
@end deffn

@geindex select() (tkinter.ttk.Notebook method)
@anchor{library/tkinter ttk tkinter ttk Notebook select}@anchor{3693}
@deffn {Method} select (tab_id=None)

Selects the specified @emph{tab_id}.

The associated child window will be displayed, and the
previously-selected window (if different) is unmapped. If @emph{tab_id} is
omitted, returns the widget name of the currently selected pane.
@end deffn

@geindex tab() (tkinter.ttk.Notebook method)
@anchor{library/tkinter ttk tkinter ttk Notebook tab}@anchor{3694}
@deffn {Method} tab (tab_id, option=None, **kw)

Query or modify the options of the specific @emph{tab_id}.

If @emph{kw} is not given, returns a dictionary of the tab option values. If
@emph{option} is specified, returns the value of that @emph{option}. Otherwise,
sets the options to the corresponding values.
@end deffn

@geindex tabs() (tkinter.ttk.Notebook method)
@anchor{library/tkinter ttk tkinter ttk Notebook tabs}@anchor{3695}
@deffn {Method} tabs ()

Returns a list of windows managed by the notebook.
@end deffn

@geindex enable_traversal() (tkinter.ttk.Notebook method)
@anchor{library/tkinter ttk tkinter ttk Notebook enable_traversal}@anchor{3689}
@deffn {Method} enable_traversal ()

Enable keyboard traversal for a toplevel window containing this notebook.

This will extend the bindings for the toplevel window containing the
notebook as follows:


@itemize *

@item 
@code{Control-Tab}: selects the tab following the currently selected one.

@item 
@code{Shift-Control-Tab}: selects the tab preceding the currently selected one.

@item 
@code{Alt-K}: where @emph{K} is the mnemonic (underlined) character of any tab, will
select that tab.
@end itemize

Multiple notebooks in a single toplevel may be enabled for traversal,
including nested notebooks. However, notebook traversal only works
properly if all panes have the notebook they are in as master.
@end deffn
@end deffn

@node Progressbar,Separator,Notebook,tkinter ttk — Tk themed widgets
@anchor{library/tkinter ttk progressbar}@anchor{3696}
@subsubsection Progressbar


The @code{ttk.Progressbar} widget shows the status of a long-running
operation. It can operate in two modes:  1) the determinate mode which shows the
amount completed relative to the total amount of work to be done and 2) the
indeterminate mode which provides an animated display to let the user know that
work is progressing.

@menu
* Options: Options<4>. 
* ttk.Progressbar: ttk Progressbar. 

@end menu

@node Options<4>,ttk Progressbar,,Progressbar
@anchor{library/tkinter ttk id5}@anchor{3697}
@subsubsection Options


This widget accepts the following specific options:

@quotation


@multitable {xxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Description

@item

orient

@tab

One of “horizontal” or “vertical”. Specifies the orientation
of the progress bar.

@item

length

@tab

Specifies the length of the long axis of the progress bar
(width if horizontal, height if vertical).

@item

mode

@tab

One of “determinate” or “indeterminate”.

@item

maximum

@tab

A number specifying the maximum value. Defaults to 100.

@item

value

@tab

The current value of the progress bar. In “determinate” mode,
this represents the amount of work completed. In
“indeterminate” mode, it is interpreted as modulo @emph{maximum};
that is, the progress bar completes one “cycle” when its value
increases by @emph{maximum}.

@item

variable

@tab

A name which is linked to the option value. If specified, the
value of the progress bar is automatically set to the value of
this name whenever the latter is modified.

@item

phase

@tab

Read-only option. The widget periodically increments the value
of this option whenever its value is greater than 0 and, in
determinate mode, less than maximum. This option may be used
by the current theme to provide additional animation effects.

@end multitable

@end quotation

@node ttk Progressbar,,Options<4>,Progressbar
@anchor{library/tkinter ttk ttk-progressbar}@anchor{3698}
@subsubsection ttk.Progressbar


@geindex Progressbar (class in tkinter.ttk)
@anchor{library/tkinter ttk tkinter ttk Progressbar}@anchor{366b}
@deffn {Class} tkinter.ttk.Progressbar

@geindex start() (tkinter.ttk.Progressbar method)
@anchor{library/tkinter ttk tkinter ttk Progressbar start}@anchor{3699}
@deffn {Method} start (interval=None)

Begin autoincrement mode: schedules a recurring timer event that calls
@ref{369a,,Progressbar.step()} every @emph{interval} milliseconds. If omitted,
@emph{interval} defaults to 50 milliseconds.
@end deffn

@geindex step() (tkinter.ttk.Progressbar method)
@anchor{library/tkinter ttk tkinter ttk Progressbar step}@anchor{369a}
@deffn {Method} step (amount=None)

Increments the progress bar’s value by @emph{amount}.

@emph{amount} defaults to 1.0 if omitted.
@end deffn

@geindex stop() (tkinter.ttk.Progressbar method)
@anchor{library/tkinter ttk tkinter ttk Progressbar stop}@anchor{369b}
@deffn {Method} stop ()

Stop autoincrement mode: cancels any recurring timer event initiated by
@ref{3699,,Progressbar.start()} for this progress bar.
@end deffn
@end deffn

@node Separator,Sizegrip,Progressbar,tkinter ttk — Tk themed widgets
@anchor{library/tkinter ttk separator}@anchor{369c}
@subsubsection Separator


The @code{ttk.Separator} widget displays a horizontal or vertical separator
bar.

It has no other methods besides the ones inherited from @code{ttk.Widget}.

@menu
* Options: Options<5>. 

@end menu

@node Options<5>,,,Separator
@anchor{library/tkinter ttk id6}@anchor{369d}
@subsubsection Options


This widget accepts the following specific option:

@quotation


@multitable {xxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Description

@item

orient

@tab

One of “horizontal” or “vertical”. Specifies the orientation of
the separator.

@end multitable

@end quotation

@node Sizegrip,Treeview,Separator,tkinter ttk — Tk themed widgets
@anchor{library/tkinter ttk sizegrip}@anchor{369e}
@subsubsection Sizegrip


The @code{ttk.Sizegrip} widget (also known as a grow box) allows the user to
resize the containing toplevel window by pressing and dragging the grip.

This widget has neither specific options nor specific methods, besides the
ones inherited from @code{ttk.Widget}.

@menu
* Platform-specific notes:: 
* Bugs:: 

@end menu

@node Platform-specific notes,Bugs,,Sizegrip
@anchor{library/tkinter ttk platform-specific-notes}@anchor{369f}
@subsubsection Platform-specific notes



@itemize *

@item 
On macOS, toplevel windows automatically include a built-in size grip
by default. Adding a @code{Sizegrip} is harmless, since the built-in
grip will just mask the widget.
@end itemize

@node Bugs,,Platform-specific notes,Sizegrip
@anchor{library/tkinter ttk bugs}@anchor{36a0}
@subsubsection Bugs



@itemize *

@item 
If the containing toplevel’s position was specified relative to the right
or bottom of the screen (e.g. ….), the @code{Sizegrip} widget will
not resize the window.

@item 
This widget supports only “southeast” resizing.
@end itemize

@node Treeview,Ttk Styling,Sizegrip,tkinter ttk — Tk themed widgets
@anchor{library/tkinter ttk treeview}@anchor{36a1}
@subsubsection Treeview


The @code{ttk.Treeview} widget displays a hierarchical collection of items.
Each item has a textual label, an optional image, and an optional list of data
values. The data values are displayed in successive columns after the tree
label.

The order in which data values are displayed may be controlled by setting
the widget option @code{displaycolumns}. The tree widget can also display column
headings. Columns may be accessed by number or symbolic names listed in the
widget option columns. See @ref{36a2,,Column Identifiers}.

Each item is identified by a unique name. The widget will generate item IDs
if they are not supplied by the caller. There is a distinguished root item,
named @code{@{@}}. The root item itself is not displayed; its children appear at the
top level of the hierarchy.

Each item also has a list of tags, which can be used to associate event bindings
with individual items and control the appearance of the item.

The Treeview widget supports horizontal and vertical scrolling, according to
the options described in @ref{3671,,Scrollable Widget Options} and the methods
@ref{36a3,,Treeview.xview()} and @ref{36a4,,Treeview.yview()}.

@menu
* Options: Options<6>. 
* Item Options:: 
* Tag Options:: 
* Column Identifiers:: 
* Virtual Events: Virtual Events<2>. 
* ttk.Treeview: ttk Treeview. 

@end menu

@node Options<6>,Item Options,,Treeview
@anchor{library/tkinter ttk id7}@anchor{36a5}
@subsubsection Options


This widget accepts the following specific options:

@quotation


@multitable {xxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Description

@item

columns

@tab

A list of column identifiers, specifying the number of
columns and their names.

@item

displaycolumns

@tab

A list of column identifiers (either symbolic or
integer indices) specifying which data columns are
displayed and the order in which they appear, or the
string “#all”.

@item

height

@tab

Specifies the number of rows which should be visible.
Note: the requested width is determined from the sum
of the column widths.

@item

padding

@tab

Specifies the internal padding for the widget. The
padding is a list of up to four length specifications.

@item

selectmode

@tab

Controls how the built-in class bindings manage the
selection. One of “extended”, “browse” or “none”.
If set to “extended” (the default), multiple items may
be selected. If “browse”, only a single item will be
selected at a time. If “none”, the selection will not
be changed.

Note that the application code and tag bindings can set
the selection however they wish, regardless of the
value  of this option.

@item

show

@tab

A list containing zero or more of the following values,
specifying which elements of the tree to display.


@itemize *

@item 
tree: display tree labels in column #0.

@item 
headings: display the heading row.
@end itemize

The default is “tree headings”, i.e., show all
elements.

@strong{Note}: Column #0 always refers to the tree column,
even if show=”tree” is not specified.

@end multitable

@end quotation

@node Item Options,Tag Options,Options<6>,Treeview
@anchor{library/tkinter ttk item-options}@anchor{36a6}
@subsubsection Item Options


The following item options may be specified for items in the insert and item
widget commands.

@quotation


@multitable {xxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Description

@item

text

@tab

The textual label to display for the item.

@item

image

@tab

A Tk Image, displayed to the left of the label.

@item

values

@tab

The list of values associated with the item.

Each item should have the same number of values as the widget
option columns. If there are fewer values than columns, the
remaining values are assumed empty. If there are more values
than columns, the extra values are ignored.

@item

open

@tab

@code{True}/@code{False} value indicating whether the item’s
children should be displayed or hidden.

@item

tags

@tab

A list of tags associated with this item.

@end multitable

@end quotation

@node Tag Options,Column Identifiers,Item Options,Treeview
@anchor{library/tkinter ttk tag-options}@anchor{36a7}
@subsubsection Tag Options


The following options may be specified on tags:

@quotation


@multitable {xxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Option

@tab

Description

@item

foreground

@tab

Specifies the text foreground color.

@item

background

@tab

Specifies the cell or item background color.

@item

font

@tab

Specifies the font to use when drawing text.

@item

image

@tab

Specifies the item image, in case the item’s image option
is empty.

@end multitable

@end quotation

@node Column Identifiers,Virtual Events<2>,Tag Options,Treeview
@anchor{library/tkinter ttk column-identifiers}@anchor{36a2}
@subsubsection Column Identifiers


Column identifiers take any of the following forms:


@itemize *

@item 
A symbolic name from the list of columns option.

@item 
An integer n, specifying the nth data column.

@item 
A string of the form #n, where n is an integer, specifying the nth display
column.
@end itemize

Notes:


@itemize *

@item 
Item’s option values may be displayed in a different order than the order
in which they are stored.

@item 
Column #0 always refers to the tree column, even if show=”tree” is not
specified.
@end itemize

A data column number is an index into an item’s option values list; a display
column number is the column number in the tree where the values are displayed.
Tree labels are displayed in column #0. If option displaycolumns is not set,
then data column n is displayed in column #n+1. Again, @strong{column #0 always
refers to the tree column}.

@node Virtual Events<2>,ttk Treeview,Column Identifiers,Treeview
@anchor{library/tkinter ttk id8}@anchor{36a8}
@subsubsection Virtual Events


The Treeview widget generates the following virtual events.

@quotation


@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Event

@tab

Description

@item

<<TreeviewSelect>>

@tab

Generated whenever the selection changes.

@item

<<TreeviewOpen>>

@tab

Generated just before settings the focus item to
open=True.

@item

<<TreeviewClose>>

@tab

Generated just after setting the focus item to
open=False.

@end multitable

@end quotation

The @ref{36a9,,Treeview.focus()} and @ref{5b1,,Treeview.selection()} methods can be used
to determine the affected item or items.

@node ttk Treeview,,Virtual Events<2>,Treeview
@anchor{library/tkinter ttk ttk-treeview}@anchor{36aa}
@subsubsection ttk.Treeview


@geindex Treeview (class in tkinter.ttk)
@anchor{library/tkinter ttk tkinter ttk Treeview}@anchor{5b2}
@deffn {Class} tkinter.ttk.Treeview

@geindex bbox() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview bbox}@anchor{36ab}
@deffn {Method} bbox (item, column=None)

Returns the bounding box (relative to the treeview widget’s window) of
the specified @emph{item} in the form (x, y, width, height).

If @emph{column} is specified, returns the bounding box of that cell. If the
@emph{item} is not visible (i.e., if it is a descendant of a closed item or is
scrolled offscreen), returns an empty string.
@end deffn

@geindex get_children() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview get_children}@anchor{36ac}
@deffn {Method} get_children (item=None)

Returns the list of children belonging to @emph{item}.

If @emph{item} is not specified, returns root children.
@end deffn

@geindex set_children() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview set_children}@anchor{36ad}
@deffn {Method} set_children (item, *newchildren)

Replaces @emph{item}’s child with @emph{newchildren}.

Children present in @emph{item} that are not present in @emph{newchildren} are
detached from the tree. No items in @emph{newchildren} may be an ancestor of
@emph{item}. Note that not specifying @emph{newchildren} results in detaching
@emph{item}’s children.
@end deffn

@geindex column() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview column}@anchor{36ae}
@deffn {Method} column (column, option=None, **kw)

Query or modify the options for the specified @emph{column}.

If @emph{kw} is not given, returns a dict of the column option values. If
@emph{option} is specified then the value for that @emph{option} is returned.
Otherwise, sets the options to the corresponding values.

The valid options/values are:


@itemize *

@item 

@table @asis

@item id

Returns the column name. This is a read-only option.
@end table

@item 

@table @asis

@item anchor: One of the standard Tk anchor values.

Specifies how the text in this column should be aligned with respect
to the cell.
@end table

@item 

@table @asis

@item minwidth: width

The minimum width of the column in pixels. The treeview widget will
not make the column any smaller than specified by this option when
the widget is resized or the user drags a column.
@end table

@item 

@table @asis

@item stretch: @code{True}/@code{False}

Specifies whether the column’s width should be adjusted when
the widget is resized.
@end table

@item 

@table @asis

@item width: width

The width of the column in pixels.
@end table
@end itemize

To configure the tree column, call this with column = “#0”
@end deffn

@geindex delete() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview delete}@anchor{36af}
@deffn {Method} delete (*items)

Delete all specified @emph{items} and all their descendants.

The root item may not be deleted.
@end deffn

@geindex detach() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview detach}@anchor{36b0}
@deffn {Method} detach (*items)

Unlinks all of the specified @emph{items} from the tree.

The items and all of their descendants are still present, and may be
reinserted at another point in the tree, but will not be displayed.

The root item may not be detached.
@end deffn

@geindex exists() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview exists}@anchor{36b1}
@deffn {Method} exists (item)

Returns @code{True} if the specified @emph{item} is present in the tree.
@end deffn

@geindex focus() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview focus}@anchor{36a9}
@deffn {Method} focus (item=None)

If @emph{item} is specified, sets the focus item to @emph{item}. Otherwise, returns
the current focus item, or ‘’ if there is none.
@end deffn

@geindex heading() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview heading}@anchor{36b2}
@deffn {Method} heading (column, option=None, **kw)

Query or modify the heading options for the specified @emph{column}.

If @emph{kw} is not given, returns a dict of the heading option values. If
@emph{option} is specified then the value for that @emph{option} is returned.
Otherwise, sets the options to the corresponding values.

The valid options/values are:


@itemize *

@item 

@table @asis

@item text: text

The text to display in the column heading.
@end table

@item 

@table @asis

@item image: imageName

Specifies an image to display to the right of the column heading.
@end table

@item 

@table @asis

@item anchor: anchor

Specifies how the heading text should be aligned. One of the standard
Tk anchor values.
@end table

@item 

@table @asis

@item command: callback

A callback to be invoked when the heading label is pressed.
@end table
@end itemize

To configure the tree column heading, call this with column = “#0”.
@end deffn

@geindex identify() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview identify}@anchor{36b3}
@deffn {Method} identify (component, x, y)

Returns a description of the specified @emph{component} under the point given
by @emph{x} and @emph{y}, or the empty string if no such @emph{component} is present at
that position.
@end deffn

@geindex identify_row() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview identify_row}@anchor{36b4}
@deffn {Method} identify_row (y)

Returns the item ID of the item at position @emph{y}.
@end deffn

@geindex identify_column() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview identify_column}@anchor{36b5}
@deffn {Method} identify_column (x)

Returns the data column identifier of the cell at position @emph{x}.

The tree column has ID #0.
@end deffn

@geindex identify_region() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview identify_region}@anchor{36b6}
@deffn {Method} identify_region (x, y)

Returns one of:


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

region

@tab

meaning

@item

heading

@tab

Tree heading area.

@item

separator

@tab

Space between two columns headings.

@item

tree

@tab

The tree area.

@item

cell

@tab

A data cell.

@end multitable


Availability: Tk 8.6.
@end deffn

@geindex identify_element() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview identify_element}@anchor{36b7}
@deffn {Method} identify_element (x, y)

Returns the element at position @emph{x}, @emph{y}.

Availability: Tk 8.6.
@end deffn

@geindex index() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview index}@anchor{36b8}
@deffn {Method} index (item)

Returns the integer index of @emph{item} within its parent’s list of children.
@end deffn

@geindex insert() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview insert}@anchor{36b9}
@deffn {Method} insert (parent, index, iid=None, **kw)

Creates a new item and returns the item identifier of the newly created
item.

@emph{parent} is the item ID of the parent item, or the empty string to create
a new top-level item. @emph{index} is an integer, or the value “end”,
specifying where in the list of parent’s children to insert the new item.
If @emph{index} is less than or equal to zero, the new node is inserted at
the beginning; if @emph{index} is greater than or equal to the current number
of children, it is inserted at the end. If @emph{iid} is specified, it is used
as the item identifier; @emph{iid} must not already exist in the tree.
Otherwise, a new unique identifier is generated.

See @ref{36a6,,Item Options} for the list of available points.
@end deffn

@geindex item() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview item}@anchor{36ba}
@deffn {Method} item (item, option=None, **kw)

Query or modify the options for the specified @emph{item}.

If no options are given, a dict with options/values for the item is
returned.
If @emph{option} is specified then the value for that option is returned.
Otherwise, sets the options to the corresponding values as given by @emph{kw}.
@end deffn

@geindex move() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview move}@anchor{36bb}
@deffn {Method} move (item, parent, index)

Moves @emph{item} to position @emph{index} in @emph{parent}’s list of children.

It is illegal to move an item under one of its descendants. If @emph{index} is
less than or equal to zero, @emph{item} is moved to the beginning; if greater
than or equal to the number of children, it is moved to the end. If @emph{item}
was detached it is reattached.
@end deffn

@geindex next() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview next}@anchor{36bc}
@deffn {Method} next (item)

Returns the identifier of @emph{item}’s next sibling, or ‘’ if @emph{item} is the
last child of its parent.
@end deffn

@geindex parent() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview parent}@anchor{36bd}
@deffn {Method} parent (item)

Returns the ID of the parent of @emph{item}, or ‘’ if @emph{item} is at the top
level of the hierarchy.
@end deffn

@geindex prev() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview prev}@anchor{36be}
@deffn {Method} prev (item)

Returns the identifier of @emph{item}’s previous sibling, or ‘’ if @emph{item} is
the first child of its parent.
@end deffn

@geindex reattach() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview reattach}@anchor{36bf}
@deffn {Method} reattach (item, parent, index)

An alias for @ref{36bb,,Treeview.move()}.
@end deffn

@geindex see() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview see}@anchor{36c0}
@deffn {Method} see (item)

Ensure that @emph{item} is visible.

Sets all of @emph{item}’s ancestors open option to @code{True}, and scrolls the
widget if necessary so that @emph{item} is within the visible portion of
the tree.
@end deffn

@geindex selection() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview selection}@anchor{5b1}
@deffn {Method} selection ()

Returns a tuple of selected items.


Changed in version 3.8: @code{selection()} no longer takes arguments.  For changing the selection
state use the following selection methods.

@end deffn

@geindex selection_set() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview selection_set}@anchor{5b3}
@deffn {Method} selection_set (*items)

@emph{items} becomes the new selection.


Changed in version 3.6: @emph{items} can be passed as separate arguments, not just as a single tuple.

@end deffn

@geindex selection_add() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview selection_add}@anchor{36c1}
@deffn {Method} selection_add (*items)

Add @emph{items} to the selection.


Changed in version 3.6: @emph{items} can be passed as separate arguments, not just as a single tuple.

@end deffn

@geindex selection_remove() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview selection_remove}@anchor{36c2}
@deffn {Method} selection_remove (*items)

Remove @emph{items} from the selection.


Changed in version 3.6: @emph{items} can be passed as separate arguments, not just as a single tuple.

@end deffn

@geindex selection_toggle() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview selection_toggle}@anchor{36c3}
@deffn {Method} selection_toggle (*items)

Toggle the selection state of each item in @emph{items}.


Changed in version 3.6: @emph{items} can be passed as separate arguments, not just as a single tuple.

@end deffn

@geindex set() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview set}@anchor{36c4}
@deffn {Method} set (item, column=None, value=None)

With one argument, returns a dictionary of column/value pairs for the
specified @emph{item}. With two arguments, returns the current value of the
specified @emph{column}. With three arguments, sets the value of given
@emph{column} in given @emph{item} to the specified @emph{value}.
@end deffn

@geindex tag_bind() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview tag_bind}@anchor{36c5}
@deffn {Method} tag_bind (tagname, sequence=None, callback=None)

Bind a callback for the given event @emph{sequence} to the tag @emph{tagname}.
When an event is delivered to an item, the callbacks for each of the
item’s tags option are called.
@end deffn

@geindex tag_configure() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview tag_configure}@anchor{36c6}
@deffn {Method} tag_configure (tagname, option=None, **kw)

Query or modify the options for the specified @emph{tagname}.

If @emph{kw} is not given, returns a dict of the option settings for
@emph{tagname}. If @emph{option} is specified, returns the value for that @emph{option}
for the specified @emph{tagname}. Otherwise, sets the options to the
corresponding values for the given @emph{tagname}.
@end deffn

@geindex tag_has() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview tag_has}@anchor{36c7}
@deffn {Method} tag_has (tagname, item=None)

If @emph{item} is specified, returns 1 or 0 depending on whether the specified
@emph{item} has the given @emph{tagname}. Otherwise, returns a list of all items
that have the specified tag.

Availability: Tk 8.6
@end deffn

@geindex xview() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview xview}@anchor{36a3}
@deffn {Method} xview (*args)

Query or modify horizontal position of the treeview.
@end deffn

@geindex yview() (tkinter.ttk.Treeview method)
@anchor{library/tkinter ttk tkinter ttk Treeview yview}@anchor{36a4}
@deffn {Method} yview (*args)

Query or modify vertical position of the treeview.
@end deffn
@end deffn

@node Ttk Styling,,Treeview,tkinter ttk — Tk themed widgets
@anchor{library/tkinter ttk ttk-styling}@anchor{36c8}@anchor{library/tkinter ttk ttkstyling}@anchor{366d}
@subsubsection Ttk Styling


Each widget in @code{ttk} is assigned a style, which specifies the set of
elements making up the widget and how they are arranged, along with dynamic
and default settings for element options. By default the style name is the
same as the widget’s class name, but it may be overridden by the widget’s style
option. If you don’t know the class name of a widget, use the method
@code{Misc.winfo_class()} (somewidget.winfo_class()).


@subsubheading See also



@table @asis

@item Tcl'2004 conference presentation@footnote{http://tktable.sourceforge.net/tile/tile-tcl2004.pdf}

This document explains how the theme engine works
@end table


@geindex Style (class in tkinter.ttk)
@anchor{library/tkinter ttk tkinter ttk Style}@anchor{366e}
@deffn {Class} tkinter.ttk.Style

This class is used to manipulate the style database.

@geindex configure() (tkinter.ttk.Style method)
@anchor{library/tkinter ttk tkinter ttk Style configure}@anchor{36c9}
@deffn {Method} configure (style, query_opt=None, **kw)

Query or set the default value of the specified option(s) in @emph{style}.

Each key in @emph{kw} is an option and each value is a string identifying
the value for that option.

For example, to change every default button to be a flat button with
some padding and a different background color:

@example
from tkinter import ttk
import tkinter

root = tkinter.Tk()

ttk.Style().configure("TButton", padding=6, relief="flat",
   background="#ccc")

btn = ttk.Button(text="Sample")
btn.pack()

root.mainloop()
@end example
@end deffn

@geindex map() (tkinter.ttk.Style method)
@anchor{library/tkinter ttk tkinter ttk Style map}@anchor{1196}
@deffn {Method} map (style, query_opt=None, **kw)

Query or sets dynamic values of the specified option(s) in @emph{style}.

Each key in @emph{kw} is an option and each value should be a list or a
tuple (usually) containing statespecs grouped in tuples, lists, or
some other preference. A statespec is a compound of one
or more states and then a value.

An example may make it more understandable:

@example
import tkinter
from tkinter import ttk

root = tkinter.Tk()

style = ttk.Style()
style.map("C.TButton",
    foreground=[('pressed', 'red'), ('active', 'blue')],
    background=[('pressed', '!disabled', 'black'), ('active', 'white')]
    )

colored_btn = ttk.Button(text="Test", style="C.TButton").pack()

root.mainloop()
@end example

Note that the order of the (states, value) sequences for an option does
matter, if the order is changed to @code{[('active', 'blue'), ('pressed',
'red')]} in the foreground option, for example, the result would be a
blue foreground when the widget were in active or pressed states.
@end deffn

@geindex lookup() (tkinter.ttk.Style method)
@anchor{library/tkinter ttk tkinter ttk Style lookup}@anchor{36ca}
@deffn {Method} lookup (style, option, state=None, default=None)

Returns the value specified for @emph{option} in @emph{style}.

If @emph{state} is specified, it is expected to be a sequence of one or more
states. If the @emph{default} argument is set, it is used as a fallback value
in case no specification for option is found.

To check what font a Button uses by default:

@example
from tkinter import ttk

print(ttk.Style().lookup("TButton", "font"))
@end example
@end deffn

@geindex layout() (tkinter.ttk.Style method)
@anchor{library/tkinter ttk tkinter ttk Style layout}@anchor{36cb}
@deffn {Method} layout (style, layoutspec=None)

Define the widget layout for given @emph{style}. If @emph{layoutspec} is omitted,
return the layout specification for given style.

@emph{layoutspec}, if specified, is expected to be a list or some other
sequence type (excluding strings), where each item should be a tuple and
the first item is the layout name and the second item should have the
format described in @ref{36cc,,Layouts}.

To understand the format, see the following example (it is not
intended to do anything useful):

@example
from tkinter import ttk
import tkinter

root = tkinter.Tk()

style = ttk.Style()
style.layout("TMenubutton", [
   ("Menubutton.background", None),
   ("Menubutton.button", @{"children":
       [("Menubutton.focus", @{"children":
           [("Menubutton.padding", @{"children":
               [("Menubutton.label", @{"side": "left", "expand": 1@})]
           @})]
       @})]
   @}),
])

mbtn = ttk.Menubutton(text='Text')
mbtn.pack()
root.mainloop()
@end example
@end deffn

@geindex element_create() (tkinter.ttk.Style method)
@anchor{library/tkinter ttk tkinter ttk Style element_create}@anchor{36cd}
@deffn {Method} element_create (elementname, etype, *args, **kw)

Create a new element in the current theme, of the given @emph{etype} which is
expected to be either “image”, “from” or “vsapi”. The latter is only
available in Tk 8.6a for Windows XP and Vista and is not described here.

If “image” is used, @emph{args} should contain the default image name followed
by statespec/value pairs (this is the imagespec), and @emph{kw} may have the
following options:

@quotation


@itemize *

@item 

@table @asis

@item border=padding

padding is a list of up to four integers, specifying the left, top,
right, and bottom borders, respectively.
@end table

@item 

@table @asis

@item height=height

Specifies a minimum height for the element. If less than zero, the
base image’s height is used as a default.
@end table

@item 

@table @asis

@item padding=padding

Specifies the element’s interior padding. Defaults to border’s value
if not specified.
@end table

@item 

@table @asis

@item sticky=spec

Specifies how the image is placed within the final parcel. spec
contains zero or more characters “n”, “s”, “w”, or “e”.
@end table

@item 

@table @asis

@item width=width

Specifies a minimum width for the element. If less than zero, the
base image’s width is used as a default.
@end table
@end itemize
@end quotation

If “from” is used as the value of @emph{etype},
@ref{36cd,,element_create()} will clone an existing
element. @emph{args} is expected to contain a themename, from which
the element will be cloned, and optionally an element to clone from.
If this element to clone from is not specified, an empty element will
be used. @emph{kw} is discarded.
@end deffn

@geindex element_names() (tkinter.ttk.Style method)
@anchor{library/tkinter ttk tkinter ttk Style element_names}@anchor{36ce}
@deffn {Method} element_names ()

Returns the list of elements defined in the current theme.
@end deffn

@geindex element_options() (tkinter.ttk.Style method)
@anchor{library/tkinter ttk tkinter ttk Style element_options}@anchor{36cf}
@deffn {Method} element_options (elementname)

Returns the list of @emph{elementname}’s options.
@end deffn

@geindex theme_create() (tkinter.ttk.Style method)
@anchor{library/tkinter ttk tkinter ttk Style theme_create}@anchor{36d0}
@deffn {Method} theme_create (themename, parent=None, settings=None)

Create a new theme.

It is an error if @emph{themename} already exists. If @emph{parent} is specified,
the new theme will inherit styles, elements and layouts from the parent
theme. If @emph{settings} are present they are expected to have the same
syntax used for @ref{36d1,,theme_settings()}.
@end deffn

@geindex theme_settings() (tkinter.ttk.Style method)
@anchor{library/tkinter ttk tkinter ttk Style theme_settings}@anchor{36d1}
@deffn {Method} theme_settings (themename, settings)

Temporarily sets the current theme to @emph{themename}, apply specified
@emph{settings} and then restore the previous theme.

Each key in @emph{settings} is a style and each value may contain the keys
‘configure’, ‘map’, ‘layout’ and ‘element create’ and they are expected
to have the same format as specified by the methods
@ref{36c9,,Style.configure()}, @ref{1196,,Style.map()}, @ref{36cb,,Style.layout()} and
@ref{36cd,,Style.element_create()} respectively.

As an example, let’s change the Combobox for the default theme a bit:

@example
from tkinter import ttk
import tkinter

root = tkinter.Tk()

style = ttk.Style()
style.theme_settings("default", @{
   "TCombobox": @{
       "configure": @{"padding": 5@},
       "map": @{
           "background": [("active", "green2"),
                          ("!disabled", "green4")],
           "fieldbackground": [("!disabled", "green3")],
           "foreground": [("focus", "OliveDrab1"),
                          ("!disabled", "OliveDrab2")]
       @}
   @}
@})

combo = ttk.Combobox().pack()

root.mainloop()
@end example
@end deffn

@geindex theme_names() (tkinter.ttk.Style method)
@anchor{library/tkinter ttk tkinter ttk Style theme_names}@anchor{36d2}
@deffn {Method} theme_names ()

Returns a list of all known themes.
@end deffn

@geindex theme_use() (tkinter.ttk.Style method)
@anchor{library/tkinter ttk tkinter ttk Style theme_use}@anchor{36d3}
@deffn {Method} theme_use (themename=None)

If @emph{themename} is not given, returns the theme in use.  Otherwise, sets
the current theme to @emph{themename}, refreshes all widgets and emits a
<<ThemeChanged>> event.
@end deffn
@end deffn

@menu
* Layouts:: 

@end menu

@node Layouts,,,Ttk Styling
@anchor{library/tkinter ttk layouts}@anchor{36cc}
@subsubsection Layouts


A layout can be just @code{None}, if it takes no options, or a dict of
options specifying how to arrange the element. The layout mechanism
uses a simplified version of the pack geometry manager: given an
initial cavity, each element is allocated a parcel. Valid
options/values are:

@quotation


@itemize *

@item 

@table @asis

@item side: whichside

Specifies which side of the cavity to place the element; one of
top, right, bottom or left. If omitted, the element occupies the
entire cavity.
@end table

@item 

@table @asis

@item sticky: nswe

Specifies where the element is placed inside its allocated parcel.
@end table

@item 

@table @asis

@item unit: 0 or 1

If set to 1, causes the element and all of its descendants to be treated as
a single element for the purposes of @ref{3677,,Widget.identify()} et al. It’s
used for things like scrollbar thumbs with grips.
@end table

@item 

@table @asis

@item children: [sublayout… ]

Specifies a list of elements to place inside the element. Each
element is a tuple (or other sequence type) where the first item is
the layout name, and the other is a @ref{36cc,,Layout}.
@end table
@end itemize
@end quotation

@node tkinter tix — Extension widgets for Tk,IDLE<51>,tkinter ttk — Tk themed widgets,Graphical User Interfaces with Tk
@anchor{library/tkinter tix doc}@anchor{36d4}@anchor{library/tkinter tix layout}@anchor{36d5}@anchor{library/tkinter tix layouts}@anchor{36d6}@anchor{library/tkinter tix module-tkinter tix}@anchor{115}@anchor{library/tkinter tix tkinter-tix-extension-widgets-for-tk}@anchor{36d7}
@subsection @code{tkinter.tix} — Extension widgets for Tk


@geindex module; tkinter.tix

@strong{Source code:} Lib/tkinter/tix.py@footnote{https://github.com/python/cpython/tree/main/Lib/tkinter/tix.py}

@geindex Tix


Deprecated since version 3.6: This Tk extension is unmaintained and should not be used in new code.  Use
@ref{116,,tkinter.ttk} instead.



__________________________________________________________________


The @ref{115,,tkinter.tix} (Tk Interface Extension) module provides an additional
rich set of widgets. Although the standard Tk library has many useful widgets,
they are far from complete. The @ref{115,,tkinter.tix} library provides most of the
commonly needed widgets that are missing from standard Tk: @ref{36d8,,HList},
@ref{36d9,,ComboBox}, @ref{36da,,Control} (a.k.a. SpinBox) and an assortment of
scrollable widgets.
@ref{115,,tkinter.tix} also includes many more widgets that are generally useful in
a wide range of applications: @ref{36db,,NoteBook}, @ref{36dc,,FileEntry},
@ref{36dd,,PanedWindow}, etc; there are more than 40 of them.

With all these new widgets, you can introduce new interaction techniques into
applications, creating more useful and more intuitive user interfaces. You can
design your application by choosing the most appropriate widgets to match the
special needs of your application and users.


@subsubheading See also



@table @asis

@item Tix Homepage@footnote{http://tix.sourceforge.net/}

The home page for @code{Tix}.  This includes links to additional documentation
and downloads.

@item Tix Man Pages@footnote{http://tix.sourceforge.net/dist/current/man/}

On-line version of the man pages and reference material.

@item Tix Programming Guide@footnote{http://tix.sourceforge.net/dist/current/docs/tix-book/tix.book.html}

On-line version of the programmer’s reference material.

@item Tix Development Applications@footnote{http://tix.sourceforge.net/Tixapps/src/Tide.html}

Tix applications for development of Tix and Tkinter programs. Tide applications
work under Tk or Tkinter, and include @code{TixInspect}, an inspector to
remotely modify and debug Tix/Tk/Tkinter applications.
@end table


@menu
* Using Tix:: 
* Tix Widgets:: 
* Tix Commands:: 

@end menu

@node Using Tix,Tix Widgets,,tkinter tix — Extension widgets for Tk
@anchor{library/tkinter tix using-tix}@anchor{36de}
@subsubsection Using Tix


@geindex Tk (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix Tk}@anchor{36df}
@deffn {Class} tkinter.tix.Tk (screenName=None, baseName=None, className='Tix')

Toplevel widget of Tix which represents mostly the main window of an
application. It has an associated Tcl interpreter.

Classes in the @ref{115,,tkinter.tix} module subclasses the classes in the
@ref{10c,,tkinter}. The former imports the latter, so to use @ref{115,,tkinter.tix}
with Tkinter, all you need to do is to import one module. In general, you
can just import @ref{115,,tkinter.tix}, and replace the toplevel call to
@ref{35f6,,tkinter.Tk} with @code{tix.Tk}:

@example
from tkinter import tix
from tkinter.constants import *
root = tix.Tk()
@end example
@end deffn

To use @ref{115,,tkinter.tix}, you must have the Tix widgets installed, usually
alongside your installation of the Tk widgets. To test your installation, try
the following:

@example
from tkinter import tix
root = tix.Tk()
root.tk.eval('package require Tix')
@end example

@node Tix Widgets,Tix Commands,Using Tix,tkinter tix — Extension widgets for Tk
@anchor{library/tkinter tix tix-widgets}@anchor{36e0}
@subsubsection Tix Widgets


Tix@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/TixIntro.htm}
introduces over 40 widget classes to the @ref{10c,,tkinter} repertoire.

@menu
* Basic Widgets:: 
* File Selectors:: 
* Hierarchical ListBox:: 
* Tabular ListBox:: 
* Manager Widgets:: 
* Image Types:: 
* Miscellaneous Widgets:: 
* Form Geometry Manager:: 

@end menu

@node Basic Widgets,File Selectors,,Tix Widgets
@anchor{library/tkinter tix basic-widgets}@anchor{36e1}
@subsubsection Basic Widgets


@geindex Balloon (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix Balloon}@anchor{36e2}
@deffn {Class} tkinter.tix.Balloon

A Balloon@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixBalloon.htm} that
pops up over a widget to provide help.  When the user moves the cursor inside a
widget to which a Balloon widget has been bound, a small pop-up window with a
descriptive message will be shown on the screen.
@end deffn

@c Python Demo of:

@c \ulink{Balloon}{http://tix.sourceforge.net/dist/current/demos/samples/Balloon.tcl}

@geindex ButtonBox (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix ButtonBox}@anchor{36e3}
@deffn {Class} tkinter.tix.ButtonBox

The ButtonBox@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixButtonBox.htm}
widget creates a box of buttons, such as is commonly used for @code{Ok Cancel}.
@end deffn

@c Python Demo of:

@c \ulink{ButtonBox}{http://tix.sourceforge.net/dist/current/demos/samples/BtnBox.tcl}

@geindex ComboBox (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix ComboBox}@anchor{36d9}
@deffn {Class} tkinter.tix.ComboBox

The ComboBox@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixComboBox.htm}
widget is similar to the combo box control in MS Windows. The user can select a
choice by either typing in the entry subwidget or selecting from the listbox
subwidget.
@end deffn

@c Python Demo of:

@c \ulink{ComboBox}{http://tix.sourceforge.net/dist/current/demos/samples/ComboBox.tcl}

@geindex Control (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix Control}@anchor{36da}
@deffn {Class} tkinter.tix.Control

The Control@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixControl.htm}
widget is also known as the @code{SpinBox} widget. The user can adjust the
value by pressing the two arrow buttons or by entering the value directly into
the entry. The new value will be checked against the user-defined upper and
lower limits.
@end deffn

@c Python Demo of:

@c \ulink{Control}{http://tix.sourceforge.net/dist/current/demos/samples/Control.tcl}

@geindex LabelEntry (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix LabelEntry}@anchor{36e4}
@deffn {Class} tkinter.tix.LabelEntry

The LabelEntry@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixLabelEntry.htm}
widget packages an entry widget and a label into one mega widget. It can
be used to simplify the creation of “entry-form” type of interface.
@end deffn

@c Python Demo of:

@c \ulink{LabelEntry}{http://tix.sourceforge.net/dist/current/demos/samples/LabEntry.tcl}

@geindex LabelFrame (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix LabelFrame}@anchor{36e5}
@deffn {Class} tkinter.tix.LabelFrame

The LabelFrame@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixLabelFrame.htm}
widget packages a frame widget and a label into one mega widget.  To create
widgets inside a LabelFrame widget, one creates the new widgets relative to the
@code{frame} subwidget and manage them inside the @code{frame} subwidget.
@end deffn

@c Python Demo of:

@c \ulink{LabelFrame}{http://tix.sourceforge.net/dist/current/demos/samples/LabFrame.tcl}

@geindex Meter (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix Meter}@anchor{36e6}
@deffn {Class} tkinter.tix.Meter

The Meter@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixMeter.htm} widget
can be used to show the progress of a background job which may take a long time
to execute.
@end deffn

@c Python Demo of:

@c \ulink{Meter}{http://tix.sourceforge.net/dist/current/demos/samples/Meter.tcl}

@geindex OptionMenu (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix OptionMenu}@anchor{36e7}
@deffn {Class} tkinter.tix.OptionMenu

The OptionMenu@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixOptionMenu.htm}
creates a menu button of options.
@end deffn

@c Python Demo of:

@c \ulink{OptionMenu}{http://tix.sourceforge.net/dist/current/demos/samples/OptMenu.tcl}

@geindex PopupMenu (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix PopupMenu}@anchor{36e8}
@deffn {Class} tkinter.tix.PopupMenu

The PopupMenu@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixPopupMenu.htm}
widget can be used as a replacement of the @code{tk_popup} command. The advantage
of the @code{Tix} @ref{36e8,,PopupMenu} widget is it requires less application code
to manipulate.
@end deffn

@c Python Demo of:

@c \ulink{PopupMenu}{http://tix.sourceforge.net/dist/current/demos/samples/PopMenu.tcl}

@geindex Select (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix Select}@anchor{36e9}
@deffn {Class} tkinter.tix.Select

The Select@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixSelect.htm} widget
is a container of button subwidgets. It can be used to provide radio-box or
check-box style of selection options for the user.
@end deffn

@c Python Demo of:

@c \ulink{Select}{http://tix.sourceforge.net/dist/current/demos/samples/Select.tcl}

@geindex StdButtonBox (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix StdButtonBox}@anchor{36ea}
@deffn {Class} tkinter.tix.StdButtonBox

The StdButtonBox@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixStdButtonBox.htm}
widget is a group of standard buttons for Motif-like dialog boxes.
@end deffn

@c Python Demo of:

@c \ulink{StdButtonBox}{http://tix.sourceforge.net/dist/current/demos/samples/StdBBox.tcl}

@node File Selectors,Hierarchical ListBox,Basic Widgets,Tix Widgets
@anchor{library/tkinter tix file-selectors}@anchor{36eb}
@subsubsection File Selectors


@geindex DirList (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix DirList}@anchor{36ec}
@deffn {Class} tkinter.tix.DirList

The DirList@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixDirList.htm}
widget displays a list view of a directory, its previous directories and its
sub-directories. The user can choose one of the directories displayed in the
list or change to another directory.
@end deffn

@c Python Demo of:

@c \ulink{DirList}{http://tix.sourceforge.net/dist/current/demos/samples/DirList.tcl}

@geindex DirTree (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix DirTree}@anchor{36ed}
@deffn {Class} tkinter.tix.DirTree

The DirTree@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixDirTree.htm}
widget displays a tree view of a directory, its previous directories and its
sub-directories. The user can choose one of the directories displayed in the
list or change to another directory.
@end deffn

@c Python Demo of:

@c \ulink{DirTree}{http://tix.sourceforge.net/dist/current/demos/samples/DirTree.tcl}

@geindex DirSelectDialog (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix DirSelectDialog}@anchor{36ee}
@deffn {Class} tkinter.tix.DirSelectDialog

The DirSelectDialog@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixDirSelectDialog.htm}
widget presents the directories in the file system in a dialog window.  The user
can use this dialog window to navigate through the file system to select the
desired directory.
@end deffn

@c Python Demo of:

@c \ulink{DirSelectDialog}{http://tix.sourceforge.net/dist/current/demos/samples/DirDlg.tcl}

@geindex DirSelectBox (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix DirSelectBox}@anchor{36ef}
@deffn {Class} tkinter.tix.DirSelectBox

The @ref{36ef,,DirSelectBox} is similar to the standard Motif(TM)
directory-selection box. It is generally used for the user to choose a
directory.  DirSelectBox stores the directories mostly recently selected into
a ComboBox widget so that they can be quickly selected again.
@end deffn

@geindex ExFileSelectBox (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix ExFileSelectBox}@anchor{36f0}
@deffn {Class} tkinter.tix.ExFileSelectBox

The ExFileSelectBox@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixExFileSelectBox.htm}
widget is usually embedded in a tixExFileSelectDialog widget. It provides a
convenient method for the user to select files. The style of the
@ref{36f0,,ExFileSelectBox} widget is very similar to the standard file dialog on
MS Windows 3.1.
@end deffn

@c Python Demo of:

@c \ulink{ExFileSelectDialog}{http://tix.sourceforge.net/dist/current/demos/samples/EFileDlg.tcl}

@geindex FileSelectBox (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix FileSelectBox}@anchor{36f1}
@deffn {Class} tkinter.tix.FileSelectBox

The FileSelectBox@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixFileSelectBox.htm}
is similar to the standard Motif(TM) file-selection box. It is generally used
for the user to choose a file. FileSelectBox stores the files mostly recently
selected into a @ref{36d9,,ComboBox} widget so that they can be quickly selected
again.
@end deffn

@c Python Demo of:

@c \ulink{FileSelectDialog}{http://tix.sourceforge.net/dist/current/demos/samples/FileDlg.tcl}

@geindex FileEntry (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix FileEntry}@anchor{36dc}
@deffn {Class} tkinter.tix.FileEntry

The FileEntry@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixFileEntry.htm}
widget can be used to input a filename. The user can type in the filename
manually. Alternatively, the user can press the button widget that sits next to
the entry, which will bring up a file selection dialog.
@end deffn

@c Python Demo of:

@c \ulink{FileEntry}{http://tix.sourceforge.net/dist/current/demos/samples/FileEnt.tcl}

@node Hierarchical ListBox,Tabular ListBox,File Selectors,Tix Widgets
@anchor{library/tkinter tix hierarchical-listbox}@anchor{36f2}
@subsubsection Hierarchical ListBox


@geindex HList (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix HList}@anchor{36d8}
@deffn {Class} tkinter.tix.HList

The HList@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixHList.htm} widget
can be used to display any data that have a hierarchical structure, for example,
file system directory trees. The list entries are indented and connected by
branch lines according to their places in the hierarchy.
@end deffn

@c Python Demo of:

@c \ulink{HList}{http://tix.sourceforge.net/dist/current/demos/samples/HList1.tcl}

@geindex CheckList (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix CheckList}@anchor{36f3}
@deffn {Class} tkinter.tix.CheckList

The CheckList@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixCheckList.htm}
widget displays a list of items to be selected by the user. CheckList acts
similarly to the Tk checkbutton or radiobutton widgets, except it is capable of
handling many more items than checkbuttons or radiobuttons.
@end deffn

@c Python Demo of:

@c \ulink{ CheckList}{http://tix.sourceforge.net/dist/current/demos/samples/ChkList.tcl}

@c Python Demo of:

@c \ulink{ScrolledHList (1)}{http://tix.sourceforge.net/dist/current/demos/samples/SHList.tcl}

@c Python Demo of:

@c \ulink{ScrolledHList (2)}{http://tix.sourceforge.net/dist/current/demos/samples/SHList2.tcl}

@geindex Tree (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix Tree}@anchor{36f4}
@deffn {Class} tkinter.tix.Tree

The Tree@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixTree.htm} widget
can be used to display hierarchical data in a tree form. The user can adjust the
view of the tree by opening or closing parts of the tree.
@end deffn

@c Python Demo of:

@c \ulink{Tree}{http://tix.sourceforge.net/dist/current/demos/samples/Tree.tcl}

@c Python Demo of:

@c \ulink{Tree (Dynamic)}{http://tix.sourceforge.net/dist/current/demos/samples/DynTree.tcl}

@node Tabular ListBox,Manager Widgets,Hierarchical ListBox,Tix Widgets
@anchor{library/tkinter tix tabular-listbox}@anchor{36f5}
@subsubsection Tabular ListBox


@geindex TList (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix TList}@anchor{36f6}
@deffn {Class} tkinter.tix.TList

The TList@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixTList.htm} widget
can be used to display data in a tabular format. The list entries of a
@ref{36f6,,TList} widget are similar to the entries in the Tk listbox widget.  The
main differences are (1) the @ref{36f6,,TList} widget can display the list entries
in a two dimensional format and (2) you can use graphical images as well as
multiple colors and fonts for the list entries.
@end deffn

@c Python Demo of:

@c \ulink{ScrolledTList (1)}{http://tix.sourceforge.net/dist/current/demos/samples/STList1.tcl}

@c Python Demo of:

@c \ulink{ScrolledTList (2)}{http://tix.sourceforge.net/dist/current/demos/samples/STList2.tcl}

@c Grid has yet to be added to Python

@c \subsubsection{Grid Widget}

@c Python Demo of:

@c \ulink{Simple Grid}{http://tix.sourceforge.net/dist/current/demos/samples/SGrid0.tcl}

@c Python Demo of:

@c \ulink{ScrolledGrid}{http://tix.sourceforge.net/dist/current/demos/samples/SGrid1.tcl}

@c Python Demo of:

@c \ulink{Editable Grid}{http://tix.sourceforge.net/dist/current/demos/samples/EditGrid.tcl}

@node Manager Widgets,Image Types,Tabular ListBox,Tix Widgets
@anchor{library/tkinter tix manager-widgets}@anchor{36f7}
@subsubsection Manager Widgets


@geindex PanedWindow (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix PanedWindow}@anchor{36dd}
@deffn {Class} tkinter.tix.PanedWindow

The PanedWindow@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixPanedWindow.htm}
widget allows the user to interactively manipulate the sizes of several panes.
The panes can be arranged either vertically or horizontally.  The user changes
the sizes of the panes by dragging the resize handle between two panes.
@end deffn

@c Python Demo of:

@c \ulink{PanedWindow}{http://tix.sourceforge.net/dist/current/demos/samples/PanedWin.tcl}

@geindex ListNoteBook (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix ListNoteBook}@anchor{36f8}
@deffn {Class} tkinter.tix.ListNoteBook

The ListNoteBook@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixListNoteBook.htm}
widget is very similar to the @code{TixNoteBook} widget: it can be used to
display many windows in a limited space using a notebook metaphor. The notebook
is divided into a stack of pages (windows). At one time only one of these pages
can be shown. The user can navigate through these pages by choosing the name of
the desired page in the @code{hlist} subwidget.
@end deffn

@c Python Demo of:

@c \ulink{ListNoteBook}{http://tix.sourceforge.net/dist/current/demos/samples/ListNBK.tcl}

@geindex NoteBook (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix NoteBook}@anchor{36db}
@deffn {Class} tkinter.tix.NoteBook

The NoteBook@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixNoteBook.htm}
widget can be used to display many windows in a limited space using a notebook
metaphor. The notebook is divided into a stack of pages. At one time only one of
these pages can be shown. The user can navigate through these pages by choosing
the visual “tabs” at the top of the NoteBook widget.
@end deffn

@c Python Demo of:

@c \ulink{NoteBook}{http://tix.sourceforge.net/dist/current/demos/samples/NoteBook.tcl}

@c \subsubsection{Scrolled Widgets}

@c Python Demo of:

@c \ulink{ScrolledListBox}{http://tix.sourceforge.net/dist/current/demos/samples/SListBox.tcl}

@c Python Demo of:

@c \ulink{ScrolledText}{http://tix.sourceforge.net/dist/current/demos/samples/SText.tcl}

@c Python Demo of:

@c \ulink{ScrolledWindow}{http://tix.sourceforge.net/dist/current/demos/samples/SWindow.tcl}

@c Python Demo of:

@c \ulink{Canvas Object View}{http://tix.sourceforge.net/dist/current/demos/samples/CObjView.tcl}

@node Image Types,Miscellaneous Widgets,Manager Widgets,Tix Widgets
@anchor{library/tkinter tix image-types}@anchor{36f9}
@subsubsection Image Types


The @ref{115,,tkinter.tix} module adds:


@itemize *

@item 
pixmap@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/pixmap.htm}
capabilities to all @ref{115,,tkinter.tix} and @ref{10c,,tkinter} widgets to create
color images from XPM files.

@c Python Demo of:

@c \ulink{XPM Image In Button}{http://tix.sourceforge.net/dist/current/demos/samples/Xpm.tcl}

@c Python Demo of:

@c \ulink{XPM Image In Menu}{http://tix.sourceforge.net/dist/current/demos/samples/Xpm1.tcl}

@item 
Compound@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/compound.htm} image
types can be used to create images that consists of multiple horizontal lines;
each line is composed of a series of items (texts, bitmaps, images or spaces)
arranged from left to right. For example, a compound image can be used to
display a bitmap and a text string simultaneously in a Tk @code{Button}
widget.

@c Python Demo of:

@c \ulink{Compound Image In Buttons}{http://tix.sourceforge.net/dist/current/demos/samples/CmpImg.tcl}

@c Python Demo of:

@c \ulink{Compound Image In NoteBook}{http://tix.sourceforge.net/dist/current/demos/samples/CmpImg2.tcl}

@c Python Demo of:

@c \ulink{Compound Image Notebook Color Tabs}{http://tix.sourceforge.net/dist/current/demos/samples/CmpImg4.tcl}

@c Python Demo of:

@c \ulink{Compound Image Icons}{http://tix.sourceforge.net/dist/current/demos/samples/CmpImg3.tcl}
@end itemize

@node Miscellaneous Widgets,Form Geometry Manager,Image Types,Tix Widgets
@anchor{library/tkinter tix miscellaneous-widgets}@anchor{36fa}
@subsubsection Miscellaneous Widgets


@geindex InputOnly (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix InputOnly}@anchor{36fb}
@deffn {Class} tkinter.tix.InputOnly

The InputOnly@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixInputOnly.htm}
widgets are to accept inputs from the user, which can be done with the @code{bind}
command (Unix only).
@end deffn

@node Form Geometry Manager,,Miscellaneous Widgets,Tix Widgets
@anchor{library/tkinter tix form-geometry-manager}@anchor{36fc}
@subsubsection Form Geometry Manager


In addition, @ref{115,,tkinter.tix} augments @ref{10c,,tkinter} by providing:

@geindex Form (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix Form}@anchor{36fd}
@deffn {Class} tkinter.tix.Form

The Form@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixForm.htm} geometry
manager based on attachment rules for all Tk widgets.
@end deffn

@node Tix Commands,,Tix Widgets,tkinter tix — Extension widgets for Tk
@anchor{library/tkinter tix tix-commands}@anchor{36fe}
@subsubsection Tix Commands


@geindex tixCommand (class in tkinter.tix)
@anchor{library/tkinter tix tkinter tix tixCommand}@anchor{36ff}
@deffn {Class} tkinter.tix.tixCommand

The tix commands@footnote{http://tix.sourceforge.net/dist/current/man/html/TixCmd/tix.htm} provide
access to miscellaneous elements of @code{Tix}’s internal state and the
@code{Tix} application context.  Most of the information manipulated by these
methods pertains to the application as a whole, or to a screen or display,
rather than to a particular window.

To view the current settings, the common usage is:

@example
from tkinter import tix
root = tix.Tk()
print(root.tix_configure())
@end example
@end deffn

@geindex tix_configure() (tkinter.tix.tixCommand method)
@anchor{library/tkinter tix tkinter tix tixCommand tix_configure}@anchor{3700}
@deffn {Method} tixCommand.tix_configure (cnf=None, **kw)

Query or modify the configuration options of the Tix application context. If no
option is specified, returns a dictionary all of the available options.  If
option is specified with no value, then the method returns a list describing the
one named option (this list will be identical to the corresponding sublist of
the value returned if no option is specified).  If one or more option-value
pairs are specified, then the method modifies the given option(s) to have the
given value(s); in this case the method returns an empty string. Option may be
any of the configuration options.
@end deffn

@geindex tix_cget() (tkinter.tix.tixCommand method)
@anchor{library/tkinter tix tkinter tix tixCommand tix_cget}@anchor{3701}
@deffn {Method} tixCommand.tix_cget (option)

Returns the current value of the configuration option given by @emph{option}. Option
may be any of the configuration options.
@end deffn

@geindex tix_getbitmap() (tkinter.tix.tixCommand method)
@anchor{library/tkinter tix tkinter tix tixCommand tix_getbitmap}@anchor{3702}
@deffn {Method} tixCommand.tix_getbitmap (name)

Locates a bitmap file of the name @code{name.xpm} or @code{name} in one of the bitmap
directories (see the @ref{3703,,tix_addbitmapdir()} method).  By using
@ref{3702,,tix_getbitmap()}, you can avoid hard coding the pathnames of the bitmap
files in your application. When successful, it returns the complete pathname of
the bitmap file, prefixed with the character @code{@@}.  The returned value can be
used to configure the @code{bitmap} option of the Tk and Tix widgets.
@end deffn

@geindex tix_addbitmapdir() (tkinter.tix.tixCommand method)
@anchor{library/tkinter tix tkinter tix tixCommand tix_addbitmapdir}@anchor{3703}
@deffn {Method} tixCommand.tix_addbitmapdir (directory)

Tix maintains a list of directories under which the @ref{3704,,tix_getimage()} and
@ref{3702,,tix_getbitmap()} methods will search for image files.  The standard bitmap
directory is @code{$TIX_LIBRARY/bitmaps}. The @ref{3703,,tix_addbitmapdir()} method
adds @emph{directory} into this list. By using this method, the image files of an
applications can also be located using the @ref{3704,,tix_getimage()} or
@ref{3702,,tix_getbitmap()} method.
@end deffn

@geindex tix_filedialog() (tkinter.tix.tixCommand method)
@anchor{library/tkinter tix tkinter tix tixCommand tix_filedialog}@anchor{3705}
@deffn {Method} tixCommand.tix_filedialog ([dlgclass])

Returns the file selection dialog that may be shared among different calls from
this application.  This method will create a file selection dialog widget when
it is called the first time. This dialog will be returned by all subsequent
calls to @ref{3705,,tix_filedialog()}.  An optional dlgclass parameter can be passed
as a string to specified what type of file selection dialog widget is desired.
Possible options are @code{tix}, @code{FileSelectDialog} or @code{tixExFileSelectDialog}.
@end deffn

@geindex tix_getimage() (tkinter.tix.tixCommand method)
@anchor{library/tkinter tix tkinter tix tixCommand tix_getimage}@anchor{3704}
@deffn {Method} tixCommand.tix_getimage (self, name)

Locates an image file of the name @code{name.xpm}, @code{name.xbm} or
@code{name.ppm} in one of the bitmap directories (see the
@ref{3703,,tix_addbitmapdir()} method above). If more than one file with the same name
(but different extensions) exist, then the image type is chosen according to the
depth of the X display: xbm images are chosen on monochrome displays and color
images are chosen on color displays. By using @ref{3704,,tix_getimage()}, you can
avoid hard coding the pathnames of the image files in your application. When
successful, this method returns the name of the newly created image, which can
be used to configure the @code{image} option of the Tk and Tix widgets.
@end deffn

@geindex tix_option_get() (tkinter.tix.tixCommand method)
@anchor{library/tkinter tix tkinter tix tixCommand tix_option_get}@anchor{3706}
@deffn {Method} tixCommand.tix_option_get (name)

Gets the options maintained by the Tix scheme mechanism.
@end deffn

@geindex tix_resetoptions() (tkinter.tix.tixCommand method)
@anchor{library/tkinter tix tkinter tix tixCommand tix_resetoptions}@anchor{3707}
@deffn {Method} tixCommand.tix_resetoptions (newScheme, newFontSet[, newScmPrio])

Resets the scheme and fontset of the Tix application to @emph{newScheme} and
@emph{newFontSet}, respectively.  This affects only those widgets created after this
call.  Therefore, it is best to call the resetoptions method before the creation
of any widgets in a Tix application.

The optional parameter @emph{newScmPrio} can be given to reset the priority level of
the Tk options set by the Tix schemes.

Because of the way Tk handles the X option database, after Tix has been has
imported and inited, it is not possible to reset the color schemes and font sets
using the @code{tix_config()} method. Instead, the @ref{3707,,tix_resetoptions()}
method must be used.
@end deffn

@node IDLE<51>,,tkinter tix — Extension widgets for Tk,Graphical User Interfaces with Tk
@anchor{library/idle doc}@anchor{3708}@anchor{library/idle id1}@anchor{3709}@anchor{library/idle idle}@anchor{ba3}
@subsection IDLE


@strong{Source code:} Lib/idlelib/@footnote{https://github.com/python/cpython/tree/main/Lib/idlelib/}

@geindex IDLE
@geindex Python Editor
@geindex Integrated Development Environment


__________________________________________________________________


IDLE is Python’s Integrated Development and Learning Environment.

IDLE has the following features:


@itemize *

@item 
coded in 100% pure Python, using the @ref{10c,,tkinter} GUI toolkit

@item 
cross-platform: works mostly the same on Windows, Unix, and macOS

@item 
Python shell window (interactive interpreter) with colorizing
of code input, output, and error messages

@item 
multi-window text editor with multiple undo, Python colorizing,
smart indent, call tips, auto completion, and other features

@item 
search within any window, replace within editor windows, and search
through multiple files (grep)

@item 
debugger with persistent breakpoints, stepping, and viewing
of global and local namespaces

@item 
configuration, browsers, and other dialogs
@end itemize

@menu
* Menus:: 
* Editing and navigation:: 
* Startup and code execution:: 
* Help and preferences:: 

@end menu

@node Menus,Editing and navigation,,IDLE<51>
@anchor{library/idle menus}@anchor{370a}
@subsubsection Menus


IDLE has two main window types, the Shell window and the Editor window.  It is
possible to have multiple editor windows simultaneously.  On Windows and
Linux, each has its own top menu.  Each menu documented below indicates
which window type it is associated with.

Output windows, such as used for Edit => Find in Files, are a subtype of editor
window.  They currently have the same top menu but a different
default title and context menu.

On macOS, there is one application menu.  It dynamically changes according
to the window currently selected.  It has an IDLE menu, and some entries
described below are moved around to conform to Apple guidelines.

@menu
* File menu (Shell and Editor): File menu Shell and Editor. 
* Edit menu (Shell and Editor): Edit menu Shell and Editor. 
* Format menu (Editor window only): Format menu Editor window only. 
* Run menu (Editor window only): Run menu Editor window only. 
* Shell menu (Shell window only): Shell menu Shell window only. 
* Debug menu (Shell window only): Debug menu Shell window only. 
* Options menu (Shell and Editor): Options menu Shell and Editor. 
* Window menu (Shell and Editor): Window menu Shell and Editor. 
* Help menu (Shell and Editor): Help menu Shell and Editor. 
* Context Menus:: 

@end menu

@node File menu Shell and Editor,Edit menu Shell and Editor,,Menus
@anchor{library/idle file-menu-shell-and-editor}@anchor{370b}
@subsubsection File menu (Shell and Editor)



@table @asis

@item New File

Create a new file editing window.

@item Open…

Open an existing file with an Open dialog.

@item Recent Files

Open a list of recent files.  Click one to open it.

@item Open Module…

Open an existing module (searches sys.path).
@end table

@geindex Class browser
@geindex Path browser


@table @asis

@item Class Browser

Show functions, classes, and methods in the current Editor file in a
tree structure.  In the shell, open a module first.

@item Path Browser

Show sys.path directories, modules, functions, classes and methods in a
tree structure.

@item Save

Save the current window to the associated file, if there is one.  Windows
that have been changed since being opened or last saved have a * before
and after the window title.  If there is no associated file,
do Save As instead.

@item Save As…

Save the current window with a Save As dialog.  The file saved becomes the
new associated file for the window.

@item Save Copy As…

Save the current window to different file without changing the associated
file.

@item Print Window

Print the current window to the default printer.

@item Close Window

Close the current window (if an unsaved editor, ask to save; if an unsaved
Shell, ask to quit execution).  Calling @code{exit()} or @code{close()} in the Shell
window also closes Shell.  If this is the only window, also exit IDLE.

@item Exit IDLE

Close all windows and quit IDLE (ask to save unsaved edit windows).
@end table

@node Edit menu Shell and Editor,Format menu Editor window only,File menu Shell and Editor,Menus
@anchor{library/idle edit-menu-shell-and-editor}@anchor{370c}
@subsubsection Edit menu (Shell and Editor)



@table @asis

@item Undo

Undo the last change to the current window.  A maximum of 1000 changes may
be undone.

@item Redo

Redo the last undone change to the current window.

@item Cut

Copy selection into the system-wide clipboard; then delete the selection.

@item Copy

Copy selection into the system-wide clipboard.

@item Paste

Insert contents of the system-wide clipboard into the current window.
@end table

The clipboard functions are also available in context menus.


@table @asis

@item Select All

Select the entire contents of the current window.

@item Find…

Open a search dialog with many options

@item Find Again

Repeat the last search, if there is one.

@item Find Selection

Search for the currently selected string, if there is one.

@item Find in Files…

Open a file search dialog.  Put results in a new output window.

@item Replace…

Open a search-and-replace dialog.

@item Go to Line

Move the cursor to the beginning of the line requested and make that
line visible.  A request past the end of the file goes to the end.
Clear any selection and update the line and column status.

@item Show Completions

Open a scrollable list allowing selection of existing names. See
@ref{370d,,Completions} in the Editing and navigation section below.

@item Expand Word

Expand a prefix you have typed to match a full word in the same window;
repeat to get a different expansion.

@item Show call tip

After an unclosed parenthesis for a function, open a small window with
function parameter hints.  See @ref{370e,,Calltips} in the
Editing and navigation section below.

@item Show surrounding parens

Highlight the surrounding parenthesis.
@end table

@node Format menu Editor window only,Run menu Editor window only,Edit menu Shell and Editor,Menus
@anchor{library/idle format-menu}@anchor{370f}@anchor{library/idle format-menu-editor-window-only}@anchor{3710}
@subsubsection Format menu (Editor window only)



@table @asis

@item Indent Region

Shift selected lines right by the indent width (default 4 spaces).

@item Dedent Region

Shift selected lines left by the indent width (default 4 spaces).

@item Comment Out Region

Insert ## in front of selected lines.

@item Uncomment Region

Remove leading # or ## from selected lines.

@item Tabify Region

Turn @emph{leading} stretches of spaces into tabs. (Note: We recommend using
4 space blocks to indent Python code.)

@item Untabify Region

Turn @emph{all} tabs into the correct number of spaces.

@item Toggle Tabs

Open a dialog to switch between indenting with spaces and tabs.

@item New Indent Width

Open a dialog to change indent width. The accepted default by the Python
community is 4 spaces.

@item Format Paragraph

Reformat the current blank-line-delimited paragraph in comment block or
multiline string or selected line in a string.  All lines in the
paragraph will be formatted to less than N columns, where N defaults to 72.

@item Strip trailing whitespace

Remove trailing space and other whitespace characters after the last
non-whitespace character of a line by applying str.rstrip to each line,
including lines within multiline strings.  Except for Shell windows,
remove extra newlines at the end of the file.
@end table

@geindex Run script

@node Run menu Editor window only,Shell menu Shell window only,Format menu Editor window only,Menus
@anchor{library/idle run-menu-editor-window-only}@anchor{3711}
@subsubsection Run menu (Editor window only)

@anchor{library/idle run-module}@anchor{3712}

@table @asis

@item Run Module

Do @ref{3713,,Check Module}.  If no error, restart the shell to clean the
environment, then execute the module.  Output is displayed in the Shell
window.  Note that output requires use of @code{print} or @code{write}.
When execution is complete, the Shell retains focus and displays a prompt.
At this point, one may interactively explore the result of execution.
This is similar to executing a file with @code{python -i file} at a command
line.
@end table
@anchor{library/idle run-custom}@anchor{3714}

@table @asis

@item Run… Customized

Same as @ref{3712,,Run Module}, but run the module with customized
settings.  @emph{Command Line Arguments} extend @ref{e15,,sys.argv} as if passed
on a command line. The module can be run in the Shell without restarting.
@end table
@anchor{library/idle check-module}@anchor{3713}

@table @asis

@item Check Module

Check the syntax of the module currently open in the Editor window. If the
module has not been saved IDLE will either prompt the user to save or
autosave, as selected in the General tab of the Idle Settings dialog.  If
there is a syntax error, the approximate location is indicated in the
Editor window.
@end table
@anchor{library/idle python-shell}@anchor{3715}

@table @asis

@item Python Shell

Open or wake up the Python Shell window.
@end table

@node Shell menu Shell window only,Debug menu Shell window only,Run menu Editor window only,Menus
@anchor{library/idle shell-menu-shell-window-only}@anchor{3716}
@subsubsection Shell menu (Shell window only)



@table @asis

@item View Last Restart

Scroll the shell window to the last Shell restart.

@item Restart Shell

Restart the shell to clean the environment and reset display and exception handling.

@item Previous History

Cycle through earlier commands in history which match the current entry.

@item Next History

Cycle through later commands in history which match the current entry.

@item Interrupt Execution

Stop a running program.
@end table

@node Debug menu Shell window only,Options menu Shell and Editor,Shell menu Shell window only,Menus
@anchor{library/idle debug-menu-shell-window-only}@anchor{3717}
@subsubsection Debug menu (Shell window only)



@table @asis

@item Go to File/Line

Look on the current line. with the cursor, and the line above for a filename
and line number.  If found, open the file if not already open, and show the
line.  Use this to view source lines referenced in an exception traceback
and lines found by Find in Files. Also available in the context menu of
the Shell window and Output windows.
@end table

@geindex debugger
@geindex stack viewer


@table @asis

@item Debugger (toggle)

When activated, code entered in the Shell or run from an Editor will run
under the debugger.  In the Editor, breakpoints can be set with the context
menu.  This feature is still incomplete and somewhat experimental.

@item Stack Viewer

Show the stack traceback of the last exception in a tree widget, with
access to locals and globals.

@item Auto-open Stack Viewer

Toggle automatically opening the stack viewer on an unhandled exception.
@end table

@node Options menu Shell and Editor,Window menu Shell and Editor,Debug menu Shell window only,Menus
@anchor{library/idle options-menu-shell-and-editor}@anchor{3718}
@subsubsection Options menu (Shell and Editor)



@table @asis

@item Configure IDLE

Open a configuration dialog and change preferences for the following:
fonts, indentation, keybindings, text color themes, startup windows and
size, additional help sources, and extensions.  On macOS, open the
configuration dialog by selecting Preferences in the application
menu. For more details, see
@ref{3719,,Setting preferences} under Help and preferences.
@end table

Most configuration options apply to all windows or all future windows.
The option items below only apply to the active window.


@table @asis

@item Show/Hide Code Context (Editor Window only)

Open a pane at the top of the edit window which shows the block context
of the code which has scrolled above the top of the window.  See
@ref{371a,,Code Context} in the Editing and Navigation section
below.

@item Show/Hide Line Numbers (Editor Window only)

Open a column to the left of the edit window which shows the number
of each line of text.  The default is off, which may be changed in the
preferences (see @ref{3719,,Setting preferences}).

@item Zoom/Restore Height

Toggles the window between normal size and maximum height. The initial size
defaults to 40 lines by 80 chars unless changed on the General tab of the
Configure IDLE dialog.  The maximum height for a screen is determined by
momentarily maximizing a window the first time one is zoomed on the screen.
Changing screen settings may invalidate the saved height.  This toggle has
no effect when a window is maximized.
@end table

@node Window menu Shell and Editor,Help menu Shell and Editor,Options menu Shell and Editor,Menus
@anchor{library/idle window-menu-shell-and-editor}@anchor{371b}
@subsubsection Window menu (Shell and Editor)


Lists the names of all open windows; select one to bring it to the foreground
(deiconifying it if necessary).

@node Help menu Shell and Editor,Context Menus,Window menu Shell and Editor,Menus
@anchor{library/idle help-menu-shell-and-editor}@anchor{371c}
@subsubsection Help menu (Shell and Editor)



@table @asis

@item About IDLE

Display version, copyright, license, credits, and more.

@item IDLE Help

Display this IDLE document, detailing the menu options, basic editing and
navigation, and other tips.

@item Python Docs

Access local Python documentation, if installed, or start a web browser
and open docs.python.org showing the latest Python documentation.

@item Turtle Demo

Run the turtledemo module with example Python code and turtle drawings.
@end table

Additional help sources may be added here with the Configure IDLE dialog under
the General tab. See the @ref{371d,,Help sources} subsection below
for more on Help menu choices.

@geindex Cut
@geindex Copy
@geindex Paste
@geindex Set Breakpoint
@geindex Clear Breakpoint
@geindex breakpoints

@node Context Menus,,Help menu Shell and Editor,Menus
@anchor{library/idle context-menus}@anchor{371e}
@subsubsection Context Menus


Open a context menu by right-clicking in a window (Control-click on macOS).
Context menus have the standard clipboard functions also on the Edit menu.


@table @asis

@item Cut

Copy selection into the system-wide clipboard; then delete the selection.

@item Copy

Copy selection into the system-wide clipboard.

@item Paste

Insert contents of the system-wide clipboard into the current window.
@end table

Editor windows also have breakpoint functions.  Lines with a breakpoint set are
specially marked.  Breakpoints only have an effect when running under the
debugger.  Breakpoints for a file are saved in the user’s @code{.idlerc}
directory.


@table @asis

@item Set Breakpoint

Set a breakpoint on the current line.

@item Clear Breakpoint

Clear the breakpoint on that line.
@end table

Shell and Output windows also have the following.


@table @asis

@item Go to file/line

Same as in Debug menu.
@end table

The Shell window also has an output squeezing facility explained in the @emph{Python
Shell window} subsection below.


@table @asis

@item Squeeze

If the cursor is over an output line, squeeze all the output between
the code above and the prompt below down to a ‘Squeezed text’ label.
@end table

@node Editing and navigation,Startup and code execution,Menus,IDLE<51>
@anchor{library/idle editing-and-navigation}@anchor{371f}@anchor{library/idle id2}@anchor{3720}
@subsubsection Editing and navigation


@menu
* Editor windows:: 
* Key bindings:: 
* Automatic indentation:: 
* Completions:: 
* Calltips:: 
* Code Context:: 
* Python Shell window:: 
* Text colors:: 

@end menu

@node Editor windows,Key bindings,,Editing and navigation
@anchor{library/idle editor-windows}@anchor{3721}
@subsubsection Editor windows


IDLE may open editor windows when it starts, depending on settings
and how you start IDLE.  Thereafter, use the File menu.  There can be only
one open editor window for a given file.

The title bar contains the name of the file, the full path, and the version
of Python and IDLE running the window.  The status bar contains the line
number (‘Ln’) and column number (‘Col’).  Line numbers start with 1;
column numbers with 0.

IDLE assumes that files with a known .py* extension contain Python code
and that other files do not.  Run Python code with the Run menu.

@node Key bindings,Automatic indentation,Editor windows,Editing and navigation
@anchor{library/idle key-bindings}@anchor{3722}
@subsubsection Key bindings


In this section, ‘C’ refers to the @code{Control} key on Windows and Unix and
the @code{Command} key on macOS.


@itemize *

@item 
@code{Backspace} deletes to the left; @code{Del} deletes to the right

@item 
@code{C-Backspace} delete word left; @code{C-Del} delete word to the right

@item 
Arrow keys and @code{Page Up}/@code{Page Down} to move around

@item 
@code{C-LeftArrow} and @code{C-RightArrow} moves by words

@item 
@code{Home}/@code{End} go to begin/end of line

@item 
@code{C-Home}/@code{C-End} go to begin/end of file

@item 
Some useful Emacs bindings are inherited from Tcl/Tk:

@quotation


@itemize *

@item 
@code{C-a} beginning of line

@item 
@code{C-e} end of line

@item 
@code{C-k} kill line (but doesn’t put it in clipboard)

@item 
@code{C-l} center window around the insertion point

@item 
@code{C-b} go backward one character without deleting (usually you can
also use the cursor key for this)

@item 
@code{C-f} go forward one character without deleting (usually you can
also use the cursor key for this)

@item 
@code{C-p} go up one line (usually you can also use the cursor key for
this)

@item 
@code{C-d} delete next character
@end itemize
@end quotation
@end itemize

Standard keybindings (like @code{C-c} to copy and @code{C-v} to paste)
may work.  Keybindings are selected in the Configure IDLE dialog.

@node Automatic indentation,Completions,Key bindings,Editing and navigation
@anchor{library/idle automatic-indentation}@anchor{3723}
@subsubsection Automatic indentation


After a block-opening statement, the next line is indented by 4 spaces (in the
Python Shell window by one tab).  After certain keywords (break, return etc.)
the next line is dedented.  In leading indentation, @code{Backspace} deletes up
to 4 spaces if they are there. @code{Tab} inserts spaces (in the Python
Shell window one tab), number depends on Indent width. Currently, tabs
are restricted to four spaces due to Tcl/Tk limitations.

See also the indent/dedent region commands on the
@ref{370f,,Format menu}.

@node Completions,Calltips,Automatic indentation,Editing and navigation
@anchor{library/idle completions}@anchor{370d}@anchor{library/idle id3}@anchor{3724}
@subsubsection Completions


Completions are supplied, when requested and available, for module
names, attributes of classes or functions, or filenames.  Each request
method displays a completion box with existing names.  (See tab
completions below for an exception.) For any box, change the name
being completed and the item highlighted in the box by
typing and deleting characters; by hitting @code{Up}, @code{Down},
@code{PageUp}, @code{PageDown}, @code{Home}, and @code{End} keys;
and by a single click within the box.  Close the box with @code{Escape},
@code{Enter}, and double @code{Tab} keys or clicks outside the box.
A double click within the box selects and closes.

One way to open a box is to type a key character and wait for a
predefined interval.  This defaults to 2 seconds; customize it
in the settings dialog.  (To prevent auto popups, set the delay to a
large number of milliseconds, such as 100000000.) For imported module
names or class or function attributes, type ‘.’.
For filenames in the root directory, type @ref{210d,,os.sep} or
@ref{210e,,os.altsep} immediately after an opening quote.  (On Windows,
one can specify a drive first.)  Move into subdirectories by typing a
directory name and a separator.

Instead of waiting, or after a box is closed, open a completion box
immediately with Show Completions on the Edit menu.  The default hot
key is @code{C-space}.  If one types a prefix for the desired name
before opening the box, the first match or near miss is made visible.
The result is the same as if one enters a prefix
after the box is displayed.  Show Completions after a quote completes
filenames in the current directory instead of a root directory.

Hitting @code{Tab} after a prefix usually has the same effect as Show
Completions.  (With no prefix, it indents.)  However, if there is only
one match to the prefix, that match is immediately added to the editor
text without opening a box.

Invoking ‘Show Completions’, or hitting @code{Tab} after a prefix,
outside of a string and without a preceding ‘.’ opens a box with
keywords, builtin names, and available module-level names.

When editing code in an editor (as oppose to Shell), increase the
available module-level names by running your code
and not restarting the Shell thereafter.  This is especially useful
after adding imports at the top of a file.  This also increases
possible attribute completions.

Completion boxes initially exclude names beginning with ‘_’ or, for
modules, not included in ‘__all__’.  The hidden names can be accessed
by typing ‘_’ after ‘.’, either before or after the box is opened.

@node Calltips,Code Context,Completions,Editing and navigation
@anchor{library/idle calltips}@anchor{370e}@anchor{library/idle id4}@anchor{3725}
@subsubsection Calltips


A calltip is shown automatically when one types @code{(} after the name
of an @emph{accessible} function.  A function name expression may include
dots and subscripts.  A calltip remains until it is clicked, the cursor
is moved out of the argument area, or @code{)} is typed.  Whenever the
cursor is in the argument part of a definition, select Edit and “Show
Call Tip” on the menu or enter its shortcut to display a calltip.

The calltip consists of the function’s signature and docstring up to
the latter’s first blank line or the fifth non-blank line.  (Some builtin
functions lack an accessible signature.)  A ‘/’ or ‘*’ in the signature
indicates that the preceding or following arguments are passed by
position or name (keyword) only.  Details are subject to change.

In Shell, the accessible functions depends on what modules have been
imported into the user process, including those imported by Idle itself,
and which definitions have been run, all since the last restart.

For example, restart the Shell and enter @code{itertools.count(}.  A calltip
appears because Idle imports itertools into the user process for its own
use.  (This could change.)  Enter @code{turtle.write(} and nothing appears.
Idle does not itself import turtle.  The menu entry and shortcut also do
nothing.  Enter @code{import turtle}.  Thereafter, @code{turtle.write(}
will display a calltip.

In an editor, import statements have no effect until one runs the file.
One might want to run a file after writing import statements, after
adding function definitions, or after opening an existing file.

@node Code Context,Python Shell window,Calltips,Editing and navigation
@anchor{library/idle code-context}@anchor{371a}@anchor{library/idle id5}@anchor{3726}
@subsubsection Code Context


Within an editor window containing Python code, code context can be toggled
in order to show or hide a pane at the top of the window.  When shown, this
pane freezes the opening lines for block code, such as those beginning with
@code{class}, @code{def}, or @code{if} keywords, that would have otherwise scrolled
out of view.  The size of the pane will be expanded and contracted as needed
to show the all current levels of context, up to the maximum number of
lines defined in the Configure IDLE dialog (which defaults to 15).  If there
are no current context lines and the feature is toggled on, a single blank
line will display.  Clicking on a line in the context pane will move that
line to the top of the editor.

The text and background colors for the context pane can be configured under
the Highlights tab in the Configure IDLE dialog.

@node Python Shell window,Text colors,Code Context,Editing and navigation
@anchor{library/idle python-shell-window}@anchor{3727}
@subsubsection Python Shell window


With IDLE’s Shell, one enters, edits, and recalls complete statements.
Most consoles and terminals only work with a single physical line at a time.

When one pastes code into Shell, it is not compiled and possibly executed
until one hits @code{Return}.  One may edit pasted code first.
If one pastes more that one statement into Shell, the result will be a
@ref{21a,,SyntaxError} when multiple statements are compiled as if they were one.

The editing features described in previous subsections work when entering
code interactively.  IDLE’s Shell window also responds to the following keys.


@itemize *

@item 
@code{C-c} interrupts executing command

@item 
@code{C-d} sends end-of-file; closes window if typed at a @code{>>>} prompt

@item 
@code{Alt-/} (Expand word) is also useful to reduce typing

Command history


@itemize *

@item 
@code{Alt-p} retrieves previous command matching what you have typed. On
macOS use @code{C-p}.

@item 
@code{Alt-n} retrieves next. On macOS use @code{C-n}.

@item 
@code{Return} while on any previous command retrieves that command
@end itemize
@end itemize

@node Text colors,,Python Shell window,Editing and navigation
@anchor{library/idle text-colors}@anchor{3728}
@subsubsection Text colors


Idle defaults to black on white text, but colors text with special meanings.
For the shell, these are shell output, shell error, user output, and
user error.  For Python code, at the shell prompt or in an editor, these are
keywords, builtin class and function names, names following @code{class} and
@code{def}, strings, and comments. For any text window, these are the cursor (when
present), found text (when possible), and selected text.

IDLE also highlights the @ref{28f,,soft keywords} @ref{290,,match},
@ref{290,,case}, and @ref{291,,_} in
pattern-matching statements. However, this highlighting is not perfect and
will be incorrect in some rare cases, including some @code{_}-s in @code{case}
patterns.

Text coloring is done in the background, so uncolorized text is occasionally
visible.  To change the color scheme, use the Configure IDLE dialog
Highlighting tab.  The marking of debugger breakpoint lines in the editor and
text in popups and dialogs is not user-configurable.

@node Startup and code execution,Help and preferences,Editing and navigation,IDLE<51>
@anchor{library/idle startup-and-code-execution}@anchor{3729}
@subsubsection Startup and code execution


Upon startup with the @code{-s} option, IDLE will execute the file referenced by
the environment variables 
@geindex IDLESTARTUP
@geindex environment variable; IDLESTARTUP
@code{IDLESTARTUP} or 
@geindex PYTHONSTARTUP
@geindex environment variable; PYTHONSTARTUP
@ref{b4d,,PYTHONSTARTUP}.
IDLE first checks for @code{IDLESTARTUP}; if @code{IDLESTARTUP} is present the file
referenced is run.  If @code{IDLESTARTUP} is not present, IDLE checks for
@code{PYTHONSTARTUP}.  Files referenced by these environment variables are
convenient places to store functions that are used frequently from the IDLE
shell, or for executing import statements to import common modules.

In addition, @code{Tk} also loads a startup file if it is present.  Note that the
Tk file is loaded unconditionally.  This additional file is @code{.Idle.py} and is
looked for in the user’s home directory.  Statements in this file will be
executed in the Tk namespace, so this file is not useful for importing
functions to be used from IDLE’s Python shell.

@menu
* Command line usage:: 
* Startup failure:: 
* Running user code:: 
* User output in Shell:: 
* Developing tkinter applications:: 
* Running without a subprocess:: 

@end menu

@node Command line usage,Startup failure,,Startup and code execution
@anchor{library/idle command-line-usage}@anchor{372a}
@subsubsection Command line usage


@example
idle.py [-c command] [-d] [-e] [-h] [-i] [-r file] [-s] [-t title] [-] [arg] ...

-c command  run command in the shell window
-d          enable debugger and open shell window
-e          open editor window
-h          print help message with legal combinations and exit
-i          open shell window
-r file     run file in shell window
-s          run $IDLESTARTUP or $PYTHONSTARTUP first, in shell window
-t title    set title of shell window
-           run stdin in shell (- must be last option before args)
@end example

If there are arguments:


@itemize *

@item 
If @code{-}, @code{-c}, or @code{r} is used, all arguments are placed in
@code{sys.argv[1:...]} and @code{sys.argv[0]} is set to @code{'@w{'}}, @code{'-c'},
or @code{'-r'}.  No editor window is opened, even if that is the default
set in the Options dialog.

@item 
Otherwise, arguments are files opened for editing and
@code{sys.argv} reflects the arguments passed to IDLE itself.
@end itemize

@node Startup failure,Running user code,Command line usage,Startup and code execution
@anchor{library/idle startup-failure}@anchor{372b}
@subsubsection Startup failure


IDLE uses a socket to communicate between the IDLE GUI process and the user
code execution process.  A connection must be established whenever the Shell
starts or restarts.  (The latter is indicated by a divider line that says
‘RESTART’). If the user process fails to connect to the GUI process, it
usually displays a @code{Tk} error box with a ‘cannot connect’ message
that directs the user here.  It then exits.

One specific connection failure on Unix systems results from
misconfigured masquerading rules somewhere in a system’s network setup.
When IDLE is started from a terminal, one will see a message starting
with @code{** Invalid host:}.
The valid value is @code{127.0.0.1 (idlelib.rpc.LOCALHOST)}.
One can diagnose with @code{tcpconnect -irv 127.0.0.1 6543} in one
terminal window and @code{tcplisten <same args>} in another.

A common cause of failure is a user-written file with the same name as a
standard library module, such as @emph{random.py} and @emph{tkinter.py}. When such a
file is located in the same directory as a file that is about to be run,
IDLE cannot import the stdlib file.  The current fix is to rename the
user file.

Though less common than in the past, an antivirus or firewall program may
stop the connection.  If the program cannot be taught to allow the
connection, then it must be turned off for IDLE to work.  It is safe to
allow this internal connection because no data is visible on external
ports.  A similar problem is a network mis-configuration that blocks
connections.

Python installation issues occasionally stop IDLE: multiple versions can
clash, or a single installation might need admin access.  If one undo the
clash, or cannot or does not want to run as admin, it might be easiest to
completely remove Python and start over.

A zombie pythonw.exe process could be a problem.  On Windows, use Task
Manager to check for one and stop it if there is.  Sometimes a restart
initiated by a program crash or Keyboard Interrupt (control-C) may fail
to connect.  Dismissing the error box or using Restart Shell on the Shell
menu may fix a temporary problem.

When IDLE first starts, it attempts to read user configuration files in
@code{~/.idlerc/} (~ is one’s home directory).  If there is a problem, an error
message should be displayed.  Leaving aside random disk glitches, this can
be prevented by never editing the files by hand.  Instead, use the
configuration dialog, under Options.  Once there is an error in a user
configuration file, the best solution may be to delete it and start over
with the settings dialog.

If IDLE quits with no message, and it was not started from a console, try
starting it from a console or terminal (@code{python -m idlelib}) and see if
this results in an error message.

On Unix-based systems with tcl/tk older than @code{8.6.11} (see
@code{About IDLE}) certain characters of certain fonts can cause
a tk failure with a message to the terminal.  This can happen either
if one starts IDLE to edit a file with such a character or later
when entering such a character.  If one cannot upgrade tcl/tk,
then re-configure IDLE to use a font that works better.

@node Running user code,User output in Shell,Startup failure,Startup and code execution
@anchor{library/idle running-user-code}@anchor{372c}
@subsubsection Running user code


With rare exceptions, the result of executing Python code with IDLE is
intended to be the same as executing the same code by the default method,
directly with Python in a text-mode system console or terminal window.
However, the different interface and operation occasionally affect
visible results.  For instance, @code{sys.modules} starts with more entries,
and @code{threading.active_count()} returns 2 instead of 1.

By default, IDLE runs user code in a separate OS process rather than in
the user interface process that runs the shell and editor.  In the execution
process, it replaces @code{sys.stdin}, @code{sys.stdout}, and @code{sys.stderr}
with objects that get input from and send output to the Shell window.
The original values stored in @code{sys.__stdin__}, @code{sys.__stdout__}, and
@code{sys.__stderr__} are not touched, but may be @code{None}.

Sending print output from one process to a text widget in another is
slower than printing to a system terminal in the same process.
This has the most effect when printing multiple arguments, as the string
for each argument, each separator, the newline are sent separately.
For development, this is usually not a problem, but if one wants to
print faster in IDLE, format and join together everything one wants
displayed together and then print a single string.  Both format strings
and @ref{1a8b,,str.join()} can help combine fields and lines.

IDLE’s standard stream replacements are not inherited by subprocesses
created in the execution process, whether directly by user code or by
modules such as multiprocessing.  If such subprocess use @code{input} from
sys.stdin or @code{print} or @code{write} to sys.stdout or sys.stderr,
IDLE should be started in a command line window.  The secondary subprocess
will then be attached to that window for input and output.

If @code{sys} is reset by user code, such as with @code{importlib.reload(sys)},
IDLE’s changes are lost and input from the keyboard and output to the screen
will not work correctly.

When Shell has the focus, it controls the keyboard and screen.  This is
normally transparent, but functions that directly access the keyboard
and screen will not work.  These include system-specific functions that
determine whether a key has been pressed and if so, which.

The IDLE code running in the execution process adds frames to the call stack
that would not be there otherwise.  IDLE wraps @code{sys.getrecursionlimit} and
@code{sys.setrecursionlimit} to reduce the effect of the additional stack
frames.

When user code raises SystemExit either directly or by calling sys.exit,
IDLE returns to a Shell prompt instead of exiting.

@node User output in Shell,Developing tkinter applications,Running user code,Startup and code execution
@anchor{library/idle user-output-in-shell}@anchor{372d}
@subsubsection User output in Shell


When a program outputs text, the result is determined by the
corresponding output device.  When IDLE executes user code, @code{sys.stdout}
and @code{sys.stderr} are connected to the display area of IDLE’s Shell.  Some of
its features are inherited from the underlying Tk Text widget.  Others
are programmed additions.  Where it matters, Shell is designed for development
rather than production runs.

For instance, Shell never throws away output.  A program that sends unlimited
output to Shell will eventually fill memory, resulting in a memory error.
In contrast, some system text windows only keep the last n lines of output.
A Windows console, for instance, keeps a user-settable 1 to 9999 lines,
with 300 the default.

A Tk Text widget, and hence IDLE’s Shell, displays characters (codepoints) in
the BMP (Basic Multilingual Plane) subset of Unicode.  Which characters are
displayed with a proper glyph and which with a replacement box depends on the
operating system and installed fonts.  Tab characters cause the following text
to begin after the next tab stop. (They occur every 8 ‘characters’).  Newline
characters cause following text to appear on a new line.  Other control
characters are ignored or displayed as a space, box, or something else,
depending on the operating system and font.  (Moving the text cursor through
such output with arrow keys may exhibit some surprising spacing behavior.)

@example
>>> s = 'a\tb\a<\x02><\r>\bc\nd'  # Enter 22 chars.
>>> len(s)
14
>>> s  # Display repr(s)
'a\tb\x07<\x02><\r>\x08c\nd'
>>> print(s, end='@w{'})  # Display s as is.
# Result varies by OS and font.  Try it.
@end example

The @code{repr} function is used for interactive echo of expression
values.  It returns an altered version of the input string in which
control codes, some BMP codepoints, and all non-BMP codepoints are
replaced with escape codes. As demonstrated above, it allows one to
identify the characters in a string, regardless of how they are displayed.

Normal and error output are generally kept separate (on separate lines)
from code input and each other.  They each get different highlight colors.

For SyntaxError tracebacks, the normal ‘^’ marking where the error was
detected is replaced by coloring the text with an error highlight.
When code run from a file causes other exceptions, one may right click
on a traceback line to jump to the corresponding line in an IDLE editor.
The file will be opened if necessary.

Shell has a special facility for squeezing output lines down to a
‘Squeezed text’ label.  This is done automatically
for output over N lines (N = 50 by default).
N can be changed in the PyShell section of the General
page of the Settings dialog.  Output with fewer lines can be squeezed by
right clicking on the output.  This can be useful lines long enough to slow
down scrolling.

Squeezed output is expanded in place by double-clicking the label.
It can also be sent to the clipboard or a separate view window by
right-clicking the label.

@node Developing tkinter applications,Running without a subprocess,User output in Shell,Startup and code execution
@anchor{library/idle developing-tkinter-applications}@anchor{372e}
@subsubsection Developing tkinter applications


IDLE is intentionally different from standard Python in order to
facilitate development of tkinter programs.  Enter @code{import tkinter as tk;
root = tk.Tk()} in standard Python and nothing appears.  Enter the same
in IDLE and a tk window appears.  In standard Python, one must also enter
@code{root.update()} to see the window.  IDLE does the equivalent in the
background, about 20 times a second, which is about every 50 milliseconds.
Next enter @code{b = tk.Button(root, text='button'); b.pack()}.  Again,
nothing visibly changes in standard Python until one enters @code{root.update()}.

Most tkinter programs run @code{root.mainloop()}, which usually does not
return until the tk app is destroyed.  If the program is run with
@code{python -i} or from an IDLE editor, a @code{>>>} shell prompt does not
appear until @code{mainloop()} returns, at which time there is nothing left
to interact with.

When running a tkinter program from an IDLE editor, one can comment out
the mainloop call.  One then gets a shell prompt immediately and can
interact with the live application.  One just has to remember to
re-enable the mainloop call when running in standard Python.

@node Running without a subprocess,,Developing tkinter applications,Startup and code execution
@anchor{library/idle running-without-a-subprocess}@anchor{372f}
@subsubsection Running without a subprocess


By default, IDLE executes user code in a separate subprocess via a socket,
which uses the internal loopback interface.  This connection is not
externally visible and no data is sent to or received from the internet.
If firewall software complains anyway, you can ignore it.

If the attempt to make the socket connection fails, Idle will notify you.
Such failures are sometimes transient, but if persistent, the problem
may be either a firewall blocking the connection or misconfiguration of
a particular system.  Until the problem is fixed, one can run Idle with
the -n command line switch.

If IDLE is started with the -n command line switch it will run in a
single process and will not create the subprocess which runs the RPC
Python execution server.  This can be useful if Python cannot create
the subprocess or the RPC socket interface on your platform.  However,
in this mode user code is not isolated from IDLE itself.  Also, the
environment is not restarted when Run/Run Module (F5) is selected.  If
your code has been modified, you must reload() the affected modules and
re-import any specific items (e.g. from foo import baz) if the changes
are to take effect.  For these reasons, it is preferable to run IDLE
with the default subprocess if at all possible.


Deprecated since version 3.4.


@node Help and preferences,,Startup and code execution,IDLE<51>
@anchor{library/idle help-and-preferences}@anchor{3730}
@subsubsection Help and preferences


@menu
* Help sources:: 
* Setting preferences:: 
* IDLE on macOS:: 
* Extensions:: 

@end menu

@node Help sources,Setting preferences,,Help and preferences
@anchor{library/idle help-sources}@anchor{371d}@anchor{library/idle id6}@anchor{3731}
@subsubsection Help sources


Help menu entry “IDLE Help” displays a formatted html version of the
IDLE chapter of the Library Reference.  The result, in a read-only
tkinter text window, is close to what one sees in a web browser.
Navigate through the text with a mousewheel,
the scrollbar, or up and down arrow keys held down.
Or click the TOC (Table of Contents) button and select a section
header in the opened box.

Help menu entry “Python Docs” opens the extensive sources of help,
including tutorials, available at @code{docs.python.org/x.y}, where ‘x.y’
is the currently running Python version.  If your system
has an off-line copy of the docs (this may be an installation option),
that will be opened instead.

Selected URLs can be added or removed from the help menu at any time using the
General tab of the Configure IDLE dialog.

@node Setting preferences,IDLE on macOS,Help sources,Help and preferences
@anchor{library/idle preferences}@anchor{3719}@anchor{library/idle setting-preferences}@anchor{3732}
@subsubsection Setting preferences


The font preferences, highlighting, keys, and general preferences can be
changed via Configure IDLE on the Option menu.
Non-default user settings are saved in a @code{.idlerc} directory in the user’s
home directory.  Problems caused by bad user configuration files are solved
by editing or deleting one or more of the files in @code{.idlerc}.

On the Font tab, see the text sample for the effect of font face and size
on multiple characters in multiple languages.  Edit the sample to add
other characters of personal interest.  Use the sample to select
monospaced fonts.  If particular characters have problems in Shell or an
editor, add them to the top of the sample and try changing first size
and then font.

On the Highlights and Keys tab, select a built-in or custom color theme
and key set.  To use a newer built-in color theme or key set with older
IDLEs, save it as a new custom theme or key set and it well be accessible
to older IDLEs.

@node IDLE on macOS,Extensions,Setting preferences,Help and preferences
@anchor{library/idle idle-on-macos}@anchor{3733}
@subsubsection IDLE on macOS


Under System Preferences: Dock, one can set “Prefer tabs when opening
documents” to “Always”.  This setting is not compatible with the tk/tkinter
GUI framework used by IDLE, and it breaks a few IDLE features.

@node Extensions,,IDLE on macOS,Help and preferences
@anchor{library/idle extensions}@anchor{3734}
@subsubsection Extensions


IDLE contains an extension facility.  Preferences for extensions can be
changed with the Extensions tab of the preferences dialog. See the
beginning of config-extensions.def in the idlelib directory for further
information.  The only current default extension is zzdummy, an example
also used for testing.

@c Other sections I have in mind are
@c Tkinter internals
@c Freezing Tkinter applications

@node Development Tools,Debugging and Profiling,Graphical User Interfaces with Tk,The Python Standard Library
@anchor{library/development doc}@anchor{3735}@anchor{library/development development}@anchor{3736}@anchor{library/development development-tools}@anchor{3737}
@section Development Tools


The modules described in this chapter help you write software.  For example, the
@ref{d4,,pydoc} module takes a module and generates documentation based on the
module’s contents.  The @ref{63,,doctest} and @ref{122,,unittest} modules contains
frameworks for writing unit tests that automatically exercise code and verify
that the expected output is produced.  @code{2to3} can translate Python 2.x
source code into valid Python 3.x code.

The list of modules described in this chapter is:

@menu
* typing — Support for type hints:: 
* pydoc — Documentation generator and online help system:: 
* Python Development Mode:: 
* Effects of the Python Development Mode:: 
* ResourceWarning Example:: 
* Bad file descriptor error example:: 
* doctest — Test interactive Python examples:: 
* unittest — Unit testing framework:: 
* unittest.mock — mock object library: unittest mock — mock object library. 
* unittest.mock — getting started: unittest mock — getting started. 
* 2to3 - Automated Python 2 to 3 code translation:: 
* test — Regression tests package for Python:: 
* test.support — Utilities for the Python test suite: test support — Utilities for the Python test suite. 
* test.support.socket_helper — Utilities for socket tests: test support socket_helper — Utilities for socket tests. 
* test.support.script_helper — Utilities for the Python execution tests: test support script_helper — Utilities for the Python execution tests. 
* test.support.bytecode_helper — Support tools for testing correct bytecode generation: test support bytecode_helper — Support tools for testing correct bytecode generation. 
* test.support.threading_helper — Utilities for threading tests: test support threading_helper — Utilities for threading tests. 
* test.support.os_helper — Utilities for os tests: test support os_helper — Utilities for os tests. 
* test.support.import_helper — Utilities for import tests: test support import_helper — Utilities for import tests. 
* test.support.warnings_helper — Utilities for warnings tests: test support warnings_helper — Utilities for warnings tests. 

@end menu

@node typing — Support for type hints,pydoc — Documentation generator and online help system,,Development Tools
@anchor{library/typing doc}@anchor{3738}@anchor{library/typing module-typing}@anchor{120}@anchor{library/typing typing-support-for-type-hints}@anchor{3739}
@subsection @code{typing} — Support for type hints


@geindex module; typing


New in version 3.5.


@strong{Source code:} Lib/typing.py@footnote{https://github.com/python/cpython/tree/main/Lib/typing.py}

@cartouche
@quotation Note 
The Python runtime does not enforce function and variable type annotations.
They can be used by third party tools such as type checkers, IDEs, linters,
etc.
@end quotation
@end cartouche


__________________________________________________________________


This module provides runtime support for type hints. The most fundamental
support consists of the types @ref{8dc,,Any}, @ref{235,,Union}, @ref{23c,,Callable},
@ref{10ba,,TypeVar}, and @ref{11c4,,Generic}. For a full specification, please see
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484}. For a simplified introduction to type hints, see 
@geindex Python Enhancement Proposals; PEP 483
PEP 483@footnote{https://www.python.org/dev/peps/pep-0483}.

The function below takes and returns a string and is annotated as follows:

@example
def greeting(name: str) -> str:
    return 'Hello ' + name
@end example

In the function @code{greeting}, the argument @code{name} is expected to be of type
@ref{2ef,,str} and the return type @ref{2ef,,str}. Subtypes are accepted as
arguments.

@menu
* Relevant PEPs:: 
* Type aliases:: 
* NewType:: 
* Callable:: 
* Generics:: 
* User-defined generic types:: 
* The Any type:: 
* Nominal vs structural subtyping:: 
* Module contents: Module contents<3>. 

@end menu

@node Relevant PEPs,Type aliases,,typing — Support for type hints
@anchor{library/typing id1}@anchor{373a}@anchor{library/typing relevant-peps}@anchor{373b}
@subsubsection Relevant PEPs


Since the initial introduction of type hints in 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} and 
@geindex Python Enhancement Proposals; PEP 483
PEP 483@footnote{https://www.python.org/dev/peps/pep-0483}, a
number of PEPs have modified and enhanced Python’s framework for type
annotations. These include:


@itemize *

@item 

@table @asis

@geindex Python Enhancement Proposals; PEP 526

@item PEP 526@footnote{https://www.python.org/dev/peps/pep-0526}: Syntax for Variable Annotations

@emph{Introducing} syntax for annotating variables outside of function
definitions, and @ref{83e,,ClassVar}
@end table

@item 

@table @asis

@geindex Python Enhancement Proposals; PEP 544

@item PEP 544@footnote{https://www.python.org/dev/peps/pep-0544}: Protocols: Structural subtyping (static duck typing)

@emph{Introducing} @ref{558,,Protocol} and the
@ref{559,,@@runtime_checkable} decorator
@end table

@item 

@table @asis

@geindex Python Enhancement Proposals; PEP 585

@item PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}: Type Hinting Generics In Standard Collections

@emph{Introducing} @ref{26a,,types.GenericAlias} and the ability to use standard
library classes as @ref{268,,generic types}
@end table

@item 

@table @asis

@geindex Python Enhancement Proposals; PEP 586

@item PEP 586@footnote{https://www.python.org/dev/peps/pep-0586}: Literal Types

@emph{Introducing} @ref{2db,,Literal}
@end table

@item 

@table @asis

@geindex Python Enhancement Proposals; PEP 589

@item PEP 589@footnote{https://www.python.org/dev/peps/pep-0589}: TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys

@emph{Introducing} @ref{2de,,TypedDict}
@end table

@item 

@table @asis

@geindex Python Enhancement Proposals; PEP 591

@item PEP 591@footnote{https://www.python.org/dev/peps/pep-0591}: Adding a final qualifier to typing

@emph{Introducing} @ref{556,,Final} and the @ref{557,,@@final} decorator
@end table

@item 

@table @asis

@geindex Python Enhancement Proposals; PEP 593

@item PEP 593@footnote{https://www.python.org/dev/peps/pep-0593}: Flexible function and variable annotations

@emph{Introducing} @ref{431,,Annotated}
@end table

@item 

@table @asis

@geindex Python Enhancement Proposals; PEP 604

@item PEP 604@footnote{https://www.python.org/dev/peps/pep-0604}: Allow writing union types as @code{X | Y}

@emph{Introducing} @ref{1b4e,,types.UnionType} and the ability to use
the binary-or operator @code{|} to signify a
@ref{238,,union of types}
@end table

@item 

@table @asis

@geindex Python Enhancement Proposals; PEP 612

@item PEP 612@footnote{https://www.python.org/dev/peps/pep-0612}: Parameter Specification Variables

@emph{Introducing} @ref{23a,,ParamSpec} and @ref{23b,,Concatenate}
@end table

@item 

@table @asis

@geindex Python Enhancement Proposals; PEP 613

@item PEP 613@footnote{https://www.python.org/dev/peps/pep-0613}: Explicit Type Aliases

@emph{Introducing} @ref{11ab,,TypeAlias}
@end table

@item 

@table @asis

@geindex Python Enhancement Proposals; PEP 647

@item PEP 647@footnote{https://www.python.org/dev/peps/pep-0647}: User-Defined Type Guards

@emph{Introducing} @ref{373c,,TypeGuard}
@end table

@item 

@table @asis

@geindex Python Enhancement Proposals; PEP 673

@item PEP 673@footnote{https://www.python.org/dev/peps/pep-0673}: Self type

@emph{Introducing} @ref{1092,,Self}
@end table
@end itemize

@node Type aliases,NewType,Relevant PEPs,typing — Support for type hints
@anchor{library/typing id2}@anchor{373d}@anchor{library/typing type-aliases}@anchor{373e}
@subsubsection Type aliases


A type alias is defined by assigning the type to the alias. In this example,
@code{Vector} and @code{list[float]} will be treated as interchangeable synonyms:

@example
Vector = list[float]

def scale(scalar: float, vector: Vector) -> Vector:
    return [scalar * num for num in vector]

# typechecks; a list of floats qualifies as a Vector.
new_vector = scale(2.0, [1.0, -4.2, 5.4])
@end example

Type aliases are useful for simplifying complex type signatures. For example:

@example
from collections.abc import Sequence

ConnectionOptions = dict[str, str]
Address = tuple[str, int]
Server = tuple[Address, ConnectionOptions]

def broadcast_message(message: str, servers: Sequence[Server]) -> None:
    ...

# The static type checker will treat the previous type signature as
# being exactly equivalent to this one.
def broadcast_message(
        message: str,
        servers: Sequence[tuple[tuple[str, int], dict[str, str]]]) -> None:
    ...
@end example

Note that @code{None} as a type hint is a special case and is replaced by
@code{type(None)}.

@node NewType,Callable,Type aliases,typing — Support for type hints
@anchor{library/typing distinct}@anchor{373f}@anchor{library/typing newtype}@anchor{3740}
@subsubsection NewType


Use the @ref{840,,NewType} helper class to create distinct types:

@example
from typing import NewType

UserId = NewType('UserId', int)
some_id = UserId(524313)
@end example

The static type checker will treat the new type as if it were a subclass
of the original type. This is useful in helping catch logical errors:

@example
def get_user_name(user_id: UserId) -> str:
    ...

# typechecks
user_a = get_user_name(UserId(42351))

# does not typecheck; an int is not a UserId
user_b = get_user_name(-1)
@end example

You may still perform all @code{int} operations on a variable of type @code{UserId},
but the result will always be of type @code{int}. This lets you pass in a
@code{UserId} wherever an @code{int} might be expected, but will prevent you from
accidentally creating a @code{UserId} in an invalid way:

@example
# 'output' is of type 'int', not 'UserId'
output = UserId(23413) + UserId(54341)
@end example

Note that these checks are enforced only by the static type checker. At runtime,
the statement @code{Derived = NewType('Derived', Base)} will make @code{Derived} a
class that immediately returns whatever parameter you pass it. That means
the expression @code{Derived(some_value)} does not create a new class or introduce
much overhead beyond that of a regular function call.

More precisely, the expression @code{some_value is Derived(some_value)} is always
true at runtime.

It is invalid to create a subtype of @code{Derived}:

@example
from typing import NewType

UserId = NewType('UserId', int)

# Fails at runtime and does not typecheck
class AdminUserId(UserId): pass
@end example

However, it is possible to create a @ref{840,,NewType} based on a ‘derived’ @code{NewType}:

@example
from typing import NewType

UserId = NewType('UserId', int)

ProUserId = NewType('ProUserId', UserId)
@end example

and typechecking for @code{ProUserId} will work as expected.

See 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} for more details.

@cartouche
@quotation Note 
Recall that the use of a type alias declares two types to be @emph{equivalent} to
one another. Doing @code{Alias = Original} will make the static type checker
treat @code{Alias} as being @emph{exactly equivalent} to @code{Original} in all cases.
This is useful when you want to simplify complex type signatures.

In contrast, @code{NewType} declares one type to be a @emph{subtype} of another.
Doing @code{Derived = NewType('Derived', Original)} will make the static type
checker treat @code{Derived} as a @emph{subclass} of @code{Original}, which means a
value of type @code{Original} cannot be used in places where a value of type
@code{Derived} is expected. This is useful when you want to prevent logic
errors with minimal runtime cost.
@end quotation
@end cartouche


New in version 3.5.2.



Changed in version 3.10: @code{NewType} is now a class rather than a function.  There is some additional
runtime cost when calling @code{NewType} over a regular function.  However, this
cost will be reduced in 3.11.0.


@node Callable,Generics,NewType,typing — Support for type hints
@anchor{library/typing callable}@anchor{3741}
@subsubsection Callable


Frameworks expecting callback functions of specific signatures might be
type hinted using @code{Callable[[Arg1Type, Arg2Type], ReturnType]}.

For example:

@example
from collections.abc import Callable

def feeder(get_next_item: Callable[[], str]) -> None:
    # Body

def async_query(on_success: Callable[[int], None],
                on_error: Callable[[int, Exception], None]) -> None:
    # Body
@end example

It is possible to declare the return type of a callable without specifying
the call signature by substituting a literal ellipsis
for the list of arguments in the type hint: @code{Callable[..., ReturnType]}.

Callables which take other callables as arguments may indicate that their
parameter types are dependent on each other using @ref{23a,,ParamSpec}.
Additionally, if that callable adds or removes arguments from other
callables, the @ref{23b,,Concatenate} operator may be used.  They
take the form @code{Callable[ParamSpecVariable, ReturnType]} and
@code{Callable[Concatenate[Arg1Type, Arg2Type, ..., ParamSpecVariable], ReturnType]}
respectively.


Changed in version 3.10: @code{Callable} now supports @ref{23a,,ParamSpec} and @ref{23b,,Concatenate}.
See 
@geindex Python Enhancement Proposals; PEP 612
PEP 612@footnote{https://www.python.org/dev/peps/pep-0612} for more information.



@subsubheading See also


The documentation for @ref{23a,,ParamSpec} and @ref{23b,,Concatenate} provide
examples of usage in @code{Callable}.


@node Generics,User-defined generic types,Callable,typing — Support for type hints
@anchor{library/typing generics}@anchor{17cb}@anchor{library/typing id3}@anchor{3742}
@subsubsection Generics


Since type information about objects kept in containers cannot be statically
inferred in a generic way, abstract base classes have been extended to support
subscription to denote expected types for container elements.

@example
from collections.abc import Mapping, Sequence

def notify_by_email(employees: Sequence[Employee],
                    overrides: Mapping[str, str]) -> None: ...
@end example

Generics can be parameterized by using a new factory available in typing
called @ref{10ba,,TypeVar}.

@example
from collections.abc import Sequence
from typing import TypeVar

T = TypeVar('T')      # Declare type variable

def first(l: Sequence[T]) -> T:   # Generic function
    return l[0]
@end example

@node User-defined generic types,The Any type,Generics,typing — Support for type hints
@anchor{library/typing user-defined-generic-types}@anchor{3743}@anchor{library/typing user-defined-generics}@anchor{17cc}
@subsubsection User-defined generic types


A user-defined class can be defined as a generic class.

@example
from typing import TypeVar, Generic
from logging import Logger

T = TypeVar('T')

class LoggedVar(Generic[T]):
    def __init__(self, value: T, name: str, logger: Logger) -> None:
        self.name = name
        self.logger = logger
        self.value = value

    def set(self, new: T) -> None:
        self.log('Set ' + repr(self.value))
        self.value = new

    def get(self) -> T:
        self.log('Get ' + repr(self.value))
        return self.value

    def log(self, message: str) -> None:
        self.logger.info('%s: %s', self.name, message)
@end example

@code{Generic[T]} as a base class defines that the class @code{LoggedVar} takes a
single type parameter @code{T} . This also makes @code{T} valid as a type within the
class body.

The @ref{11c4,,Generic} base class defines @ref{10a5,,__class_getitem__()} so
that @code{LoggedVar[t]} is valid as a type:

@example
from collections.abc import Iterable

def zero_all_vars(vars: Iterable[LoggedVar[int]]) -> None:
    for var in vars:
        var.set(0)
@end example

A generic type can have any number of type variables, and type variables may
be constrained:

@example
from typing import TypeVar, Generic
...

T = TypeVar('T')
S = TypeVar('S', int, str)

class StrangePair(Generic[T, S]):
    ...
@end example

Each type variable argument to @ref{11c4,,Generic} must be distinct.
This is thus invalid:

@example
from typing import TypeVar, Generic
...

T = TypeVar('T')

class Pair(Generic[T, T]):   # INVALID
    ...
@end example

You can use multiple inheritance with @ref{11c4,,Generic}:

@example
from collections.abc import Sized
from typing import TypeVar, Generic

T = TypeVar('T')

class LinkedList(Sized, Generic[T]):
    ...
@end example

When inheriting from generic classes, some type variables could be fixed:

@example
from collections.abc import Mapping
from typing import TypeVar

T = TypeVar('T')

class MyDict(Mapping[str, T]):
    ...
@end example

In this case @code{MyDict} has a single parameter, @code{T}.

Using a generic class without specifying type parameters assumes
@ref{8dc,,Any} for each position. In the following example, @code{MyIterable} is
not generic but implicitly inherits from @code{Iterable[Any]}:

@example
from collections.abc import Iterable

class MyIterable(Iterable): # Same as Iterable[Any]
@end example

User defined generic type aliases are also supported. Examples:

@example
from collections.abc import Iterable
from typing import TypeVar
S = TypeVar('S')
Response = Iterable[S] | int

# Return type here is same as Iterable[str] | int
def response(query: str) -> Response[str]:
    ...

T = TypeVar('T', int, float, complex)
Vec = Iterable[tuple[T, T]]

def inproduct(v: Vec[T]) -> T: # Same as Iterable[tuple[T, T]]
    return sum(x*y for x, y in v)
@end example


Changed in version 3.7: @ref{11c4,,Generic} no longer has a custom metaclass.


User-defined generics for parameter expressions are also supported via parameter
specification variables in the form @code{Generic[P]}.  The behavior is consistent
with type variables’ described above as parameter specification variables are
treated by the typing module as a specialized type variable.  The one exception
to this is that a list of types can be used to substitute a @ref{23a,,ParamSpec}:

@example
>>> from typing import Generic, ParamSpec, TypeVar

>>> T = TypeVar('T')
>>> P = ParamSpec('P')

>>> class Z(Generic[T, P]): ...
...
>>> Z[int, [dict, float]]
__main__.Z[int, (<class 'dict'>, <class 'float'>)]
@end example

Furthermore, a generic with only one parameter specification variable will accept
parameter lists in the forms @code{X[[Type1, Type2, ...]]} and also
@code{X[Type1, Type2, ...]} for aesthetic reasons.  Internally, the latter is converted
to the former and are thus equivalent:

@example
>>> class X(Generic[P]): ...
...
>>> X[int, str]
__main__.X[(<class 'int'>, <class 'str'>)]
>>> X[[int, str]]
__main__.X[(<class 'int'>, <class 'str'>)]
@end example

Do note that generics with @ref{23a,,ParamSpec} may not have correct
@code{__parameters__} after substitution in some cases because they
are intended primarily for static type checking.


Changed in version 3.10: @ref{11c4,,Generic} can now be parameterized over parameter expressions.
See @ref{23a,,ParamSpec} and 
@geindex Python Enhancement Proposals; PEP 612
PEP 612@footnote{https://www.python.org/dev/peps/pep-0612} for more details.


A user-defined generic class can have ABCs as base classes without a metaclass
conflict. Generic metaclasses are not supported. The outcome of parameterizing
generics is cached, and most types in the typing module are hashable and
comparable for equality.

@node The Any type,Nominal vs structural subtyping,User-defined generic types,typing — Support for type hints
@anchor{library/typing the-any-type}@anchor{3744}
@subsubsection The @code{Any} type


A special kind of type is @ref{8dc,,Any}. A static type checker will treat
every type as being compatible with @ref{8dc,,Any} and @ref{8dc,,Any} as being
compatible with every type.

This means that it is possible to perform any operation or method call on a
value of type @ref{8dc,,Any} and assign it to any variable:

@example
from typing import Any

a: Any = None
a = []          # OK
a = 2           # OK

s: str = '@w{'}
s = a           # OK

def foo(item: Any) -> int:
    # Typechecks; 'item' could be any type,
    # and that type might have a 'bar' method
    item.bar()
    ...
@end example

Notice that no typechecking is performed when assigning a value of type
@ref{8dc,,Any} to a more precise type. For example, the static type checker did
not report an error when assigning @code{a} to @code{s} even though @code{s} was
declared to be of type @ref{2ef,,str} and receives an @ref{1b1,,int} value at
runtime!

Furthermore, all functions without a return type or parameter types will
implicitly default to using @ref{8dc,,Any}:

@example
def legacy_parser(text):
    ...
    return data

# A static type checker will treat the above
# as having the same signature as:
def legacy_parser(text: Any) -> Any:
    ...
    return data
@end example

This behavior allows @ref{8dc,,Any} to be used as an @emph{escape hatch} when you
need to mix dynamically and statically typed code.

Contrast the behavior of @ref{8dc,,Any} with the behavior of @ref{5a5,,object}.
Similar to @ref{8dc,,Any}, every type is a subtype of @ref{5a5,,object}. However,
unlike @ref{8dc,,Any}, the reverse is not true: @ref{5a5,,object} is @emph{not} a
subtype of every other type.

That means when the type of a value is @ref{5a5,,object}, a type checker will
reject almost all operations on it, and assigning it to a variable (or using
it as a return value) of a more specialized type is a type error. For example:

@example
def hash_a(item: object) -> int:
    # Fails; an object does not have a 'magic' method.
    item.magic()
    ...

def hash_b(item: Any) -> int:
    # Typechecks
    item.magic()
    ...

# Typechecks, since ints and strs are subclasses of object
hash_a(42)
hash_a("foo")

# Typechecks, since Any is compatible with all types
hash_b(42)
hash_b("foo")
@end example

Use @ref{5a5,,object} to indicate that a value could be any type in a typesafe
manner. Use @ref{8dc,,Any} to indicate that a value is dynamically typed.

@node Nominal vs structural subtyping,Module contents<3>,The Any type,typing — Support for type hints
@anchor{library/typing nominal-vs-structural-subtyping}@anchor{3745}
@subsubsection Nominal vs structural subtyping


Initially 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} defined Python static type system as using
@emph{nominal subtyping}. This means that a class @code{A} is allowed where
a class @code{B} is expected if and only if @code{A} is a subclass of @code{B}.

This requirement previously also applied to abstract base classes, such as
@ref{1b3c,,Iterable}. The problem with this approach is that a class had
to be explicitly marked to support them, which is unpythonic and unlike
what one would normally do in idiomatic dynamically typed Python code.
For example, this conforms to 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484}:

@example
from collections.abc import Sized, Iterable, Iterator

class Bucket(Sized, Iterable[int]):
    ...
    def __len__(self) -> int: ...
    def __iter__(self) -> Iterator[int]: ...
@end example

@geindex Python Enhancement Proposals; PEP 544
PEP 544@footnote{https://www.python.org/dev/peps/pep-0544} allows to solve this problem by allowing users to write
the above code without explicit base classes in the class definition,
allowing @code{Bucket} to be implicitly considered a subtype of both @code{Sized}
and @code{Iterable[int]} by static type checkers. This is known as
@emph{structural subtyping} (or static duck-typing):

@example
from collections.abc import Iterator, Iterable

class Bucket:  # Note: no base classes
    ...
    def __len__(self) -> int: ...
    def __iter__(self) -> Iterator[int]: ...

def collect(items: Iterable[int]) -> int: ...
result = collect(Bucket())  # Passes type check
@end example

Moreover, by subclassing a special class @ref{558,,Protocol}, a user
can define new custom protocols to fully enjoy structural subtyping
(see examples below).

@node Module contents<3>,,Nominal vs structural subtyping,typing — Support for type hints
@anchor{library/typing module-contents}@anchor{3746}
@subsubsection Module contents


The module defines the following classes, functions and decorators.

@cartouche
@quotation Note 
This module defines several types that are subclasses of pre-existing
standard library classes which also extend @ref{11c4,,Generic}
to support type variables inside @code{[]}.
These types became redundant in Python 3.9 when the
corresponding pre-existing classes were enhanced to support @code{[]}.

The redundant types are deprecated as of Python 3.9 but no
deprecation warnings will be issued by the interpreter.
It is expected that type checkers will flag the deprecated types
when the checked program targets Python 3.9 or newer.

The deprecated types will be removed from the @ref{120,,typing} module
in the first Python version released 5 years after the release of Python 3.9.0.
See details in 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}—@emph{Type Hinting Generics In Standard Collections}.
@end quotation
@end cartouche

@menu
* Special typing primitives:: 
* Generic concrete collections:: 
* Abstract Base Classes:: 
* Protocols: Protocols<3>. 
* Functions and decorators:: 
* Introspection helpers:: 
* Constant:: 

@end menu

@node Special typing primitives,Generic concrete collections,,Module contents<3>
@anchor{library/typing special-typing-primitives}@anchor{3747}
@subsubsection Special typing primitives


@menu
* Special types:: 
* Special forms:: 
* Building generic types:: 
* Other special directives:: 

@end menu

@node Special types,Special forms,,Special typing primitives
@anchor{library/typing special-types}@anchor{3748}
@subsubsection Special types


These can be used as types in annotations and do not support @code{[]}.

@geindex Any (in module typing)
@anchor{library/typing typing Any}@anchor{8dc}
@deffn {Data} typing.Any

Special type indicating an unconstrained type.


@itemize *

@item 
Every type is compatible with @ref{8dc,,Any}.

@item 
@ref{8dc,,Any} is compatible with every type.
@end itemize
@end deffn

@geindex Never (in module typing)
@anchor{library/typing typing Never}@anchor{1093}
@deffn {Data} typing.Never

The bottom type@footnote{https://en.wikipedia.org/wiki/Bottom_type},
a type that has no members.

This can be used to define a function that should never be
called, or a function that never returns:

@example
from typing import Never

def never_call_me(arg: Never) -> None:
    pass

def int_or_str(arg: int | str) -> None:
    never_call_me(arg)  # type checker error
    match arg:
        case int():
            print("It's an int")
        case str():
            print("It's a str")
        case _:
            never_call_me(arg)  # ok, arg is of type Never
@end example


New in version 3.11: On older Python versions, @ref{3749,,NoReturn} may be used to express the
same concept. @code{Never} was added to make the intended meaning more explicit.

@end deffn

@geindex NoReturn (in module typing)
@anchor{library/typing typing NoReturn}@anchor{3749}
@deffn {Data} typing.NoReturn

Special type indicating that a function never returns.
For example:

@example
from typing import NoReturn

def stop() -> NoReturn:
    raise RuntimeError('no way')
@end example

@code{NoReturn} can also be used as a
bottom type@footnote{https://en.wikipedia.org/wiki/Bottom_type}, a type that
has no values. Starting in Python 3.11, the @ref{1093,,Never} type should
be used for this concept instead. Type checkers should treat the two
equivalently.


New in version 3.5.4.



New in version 3.6.2.

@end deffn

@geindex Self (in module typing)
@anchor{library/typing typing Self}@anchor{1092}
@deffn {Data} typing.Self

Special type to represent the current enclosed class.
For example:

@example
from typing import Self

class Foo:
   def returns_self(self) -> Self:
      ...
      return self
@end example

This annotation is semantically equivalent to the following,
albeit in a more succinct fashion:

@example
from typing import TypeVar

Self = TypeVar("Self", bound="Foo")

class Foo:
   def returns_self(self: Self) -> Self:
      ...
      return self
@end example

In general if something currently follows the pattern of:

@example
class Foo:
   def return_self(self) -> "Foo":
      ...
      return self
@end example

You should use use @ref{1092,,Self} as calls to @code{SubclassOfFoo.returns_self} would have
@code{Foo} as the return type and not @code{SubclassOfFoo}.

Other common use cases include:

@quotation


@itemize -

@item 
@ref{254,,classmethod}s that are used as alternative constructors and return instances
of the @code{cls} parameter.

@item 
Annotating an @ref{1811,,object.__enter__()} method which returns self.
@end itemize
@end quotation

For more information, see 
@geindex Python Enhancement Proposals; PEP 673
PEP 673@footnote{https://www.python.org/dev/peps/pep-0673}.


New in version 3.11.

@end deffn

@geindex TypeAlias (in module typing)
@anchor{library/typing typing TypeAlias}@anchor{11ab}
@deffn {Data} typing.TypeAlias

Special annotation for explicitly declaring a @ref{373e,,type alias}.
For example:

@example
from typing import TypeAlias

Factors: TypeAlias = list[int]
@end example

See 
@geindex Python Enhancement Proposals; PEP 613
PEP 613@footnote{https://www.python.org/dev/peps/pep-0613} for more details about explicit type aliases.


New in version 3.10.

@end deffn

@node Special forms,Building generic types,Special types,Special typing primitives
@anchor{library/typing special-forms}@anchor{374a}
@subsubsection Special forms


These can be used as types in annotations using @code{[]}, each having a unique syntax.

@geindex Tuple (in module typing)
@anchor{library/typing typing Tuple}@anchor{374b}
@deffn {Data} typing.Tuple

Tuple type; @code{Tuple[X, Y]} is the type of a tuple of two items
with the first item of type X and the second of type Y. The type of
the empty tuple can be written as @code{Tuple[()]}.

Example: @code{Tuple[T1, T2]} is a tuple of two elements corresponding
to type variables T1 and T2.  @code{Tuple[int, float, str]} is a tuple
of an int, a float and a string.

To specify a variable-length tuple of homogeneous type,
use literal ellipsis, e.g. @code{Tuple[int, ...]}. A plain @ref{374b,,Tuple}
is equivalent to @code{Tuple[Any, ...]}, and in turn to @ref{3e5,,tuple}.


Deprecated since version 3.9: @ref{3e5,,builtins.tuple} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and
@ref{268,,Generic Alias Type}.

@end deffn

@geindex Union (in module typing)
@anchor{library/typing typing Union}@anchor{235}
@deffn {Data} typing.Union

Union type; @code{Union[X, Y]} is equivalent to @code{X | Y} and means either X or Y.

To define a union, use e.g. @code{Union[int, str]} or the shorthand @code{int | str}. Using that shorthand is recommended. Details:


@itemize *

@item 
The arguments must be types and there must be at least one.

@item 
Unions of unions are flattened, e.g.:

@example
Union[Union[int, str], float] == Union[int, str, float]
@end example

@item 
Unions of a single argument vanish, e.g.:

@example
Union[int] == int  # The constructor actually returns int
@end example

@item 
Redundant arguments are skipped, e.g.:

@example
Union[int, str, int] == Union[int, str] == int | str
@end example

@item 
When comparing unions, the argument order is ignored, e.g.:

@example
Union[int, str] == Union[str, int]
@end example

@item 
You cannot subclass or instantiate a @code{Union}.

@item 
You cannot write @code{Union[X][Y]}.
@end itemize


Changed in version 3.7: Don’t remove explicit subclasses from unions at runtime.



Changed in version 3.10: Unions can now be written as @code{X | Y}. See
@ref{238,,union type expressions}.

@end deffn

@geindex Optional (in module typing)
@anchor{library/typing typing Optional}@anchor{374c}
@deffn {Data} typing.Optional

Optional type.

@code{Optional[X]} is equivalent to @code{X | None} (or @code{Union[X, None]}).

Note that this is not the same concept as an optional argument,
which is one that has a default.  An optional argument with a
default does not require the @code{Optional} qualifier on its type
annotation just because it is optional. For example:

@example
def foo(arg: int = 0) -> None:
    ...
@end example

On the other hand, if an explicit value of @code{None} is allowed, the
use of @code{Optional} is appropriate, whether the argument is optional
or not. For example:

@example
def foo(arg: Optional[int] = None) -> None:
    ...
@end example


Changed in version 3.10: Optional can now be written as @code{X | None}. See
@ref{238,,union type expressions}.

@end deffn

@geindex Callable (in module typing)
@anchor{library/typing typing Callable}@anchor{23c}
@deffn {Data} typing.Callable

Callable type; @code{Callable[[int], str]} is a function of (int) -> str.

The subscription syntax must always be used with exactly two
values: the argument list and the return type.  The argument list
must be a list of types or an ellipsis; the return type must be
a single type.

There is no syntax to indicate optional or keyword arguments;
such function types are rarely used as callback types.
@code{Callable[..., ReturnType]} (literal ellipsis) can be used to
type hint a callable taking any number of arguments and returning
@code{ReturnType}.  A plain @ref{23c,,Callable} is equivalent to
@code{Callable[..., Any]}, and in turn to
@ref{269,,collections.abc.Callable}.

Callables which take other callables as arguments may indicate that their
parameter types are dependent on each other using @ref{23a,,ParamSpec}.
Additionally, if that callable adds or removes arguments from other
callables, the @ref{23b,,Concatenate} operator may be used.  They
take the form @code{Callable[ParamSpecVariable, ReturnType]} and
@code{Callable[Concatenate[Arg1Type, Arg2Type, ..., ParamSpecVariable], ReturnType]}
respectively.


Deprecated since version 3.9: @ref{269,,collections.abc.Callable} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and
@ref{268,,Generic Alias Type}.



Changed in version 3.10: @code{Callable} now supports @ref{23a,,ParamSpec} and @ref{23b,,Concatenate}.
See 
@geindex Python Enhancement Proposals; PEP 612
PEP 612@footnote{https://www.python.org/dev/peps/pep-0612} for more information.



@subsubheading See also


The documentation for @ref{23a,,ParamSpec} and @ref{23b,,Concatenate} provide
examples of usage with @code{Callable}.

@end deffn

@geindex Concatenate (in module typing)
@anchor{library/typing typing Concatenate}@anchor{23b}
@deffn {Data} typing.Concatenate

Used with @ref{23c,,Callable} and @ref{23a,,ParamSpec} to type annotate a higher
order callable which adds, removes, or transforms parameters of another
callable.  Usage is in the form
@code{Concatenate[Arg1Type, Arg2Type, ..., ParamSpecVariable]}. @code{Concatenate}
is currently only valid when used as the first argument to a @ref{23c,,Callable}.
The last parameter to @code{Concatenate} must be a @ref{23a,,ParamSpec}.

For example, to annotate a decorator @code{with_lock} which provides a
@ref{27f5,,threading.Lock} to the decorated function,  @code{Concatenate} can be
used to indicate that @code{with_lock} expects a callable which takes in a
@code{Lock} as the first argument, and returns a callable with a different type
signature.  In this case, the @ref{23a,,ParamSpec} indicates that the returned
callable’s parameter types are dependent on the parameter types of the
callable being passed in:

@example
from collections.abc import Callable
from threading import Lock
from typing import Concatenate, ParamSpec, TypeVar

P = ParamSpec('P')
R = TypeVar('R')

# Use this lock to ensure that only one thread is executing a function
# at any time.
my_lock = Lock()

def with_lock(f: Callable[Concatenate[Lock, P], R]) -> Callable[P, R]:
    '@w{'}'A type-safe decorator which provides a lock.'@w{'}'
    global my_lock
    def inner(*args: P.args, **kwargs: P.kwargs) -> R:
        # Provide the lock as the first argument.
        return f(my_lock, *args, **kwargs)
    return inner

@@with_lock
def sum_threadsafe(lock: Lock, numbers: list[float]) -> float:
    '@w{'}'Add a list of numbers together in a thread-safe manner.'@w{'}'
    with lock:
        return sum(numbers)

# We don't need to pass in the lock ourselves thanks to the decorator.
sum_threadsafe([1.1, 2.2, 3.3])
@end example
@end deffn


New in version 3.10.



@subsubheading See also



@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 612
PEP 612@footnote{https://www.python.org/dev/peps/pep-0612} – Parameter Specification Variables (the PEP which introduced
@code{ParamSpec} and @code{Concatenate}).

@item 
@ref{23a,,ParamSpec} and @ref{23c,,Callable}.
@end itemize


@geindex Type (class in typing)
@anchor{library/typing typing Type}@anchor{374d}
@deffn {Class} typing.Type (Generic[CT_co])

A variable annotated with @code{C} may accept a value of type @code{C}. In
contrast, a variable annotated with @code{Type[C]} may accept values that are
classes themselves – specifically, it will accept the @emph{class object} of
@code{C}. For example:

@example
a = 3         # Has type 'int'
b = int       # Has type 'Type[int]'
c = type(a)   # Also has type 'Type[int]'
@end example

Note that @code{Type[C]} is covariant:

@example
class User: ...
class BasicUser(User): ...
class ProUser(User): ...
class TeamUser(User): ...

# Accepts User, BasicUser, ProUser, TeamUser, ...
def make_new_user(user_class: Type[User]) -> User:
    # ...
    return user_class()
@end example

The fact that @code{Type[C]} is covariant implies that all subclasses of
@code{C} should implement the same constructor signature and class method
signatures as @code{C}. The type checker should flag violations of this,
but should also allow constructor calls in subclasses that match the
constructor calls in the indicated base class. How the type checker is
required to handle this particular case may change in future revisions of
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484}.

The only legal parameters for @ref{374d,,Type} are classes, @ref{8dc,,Any},
@ref{17cb,,type variables}, and unions of any of these types.
For example:

@example
def new_non_team_user(user_class: Type[BasicUser | ProUser]): ...
@end example

@code{Type[Any]} is equivalent to @code{Type} which in turn is equivalent
to @code{type}, which is the root of Python’s metaclass hierarchy.


New in version 3.5.2.



Deprecated since version 3.9: @ref{89e,,builtins.type} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and
@ref{268,,Generic Alias Type}.

@end deffn

@geindex Literal (in module typing)
@anchor{library/typing typing Literal}@anchor{2db}
@deffn {Data} typing.Literal

A type that can be used to indicate to type checkers that the
corresponding variable or function parameter has a value equivalent to
the provided literal (or one of several literals). For example:

@example
def validate_simple(data: Any) -> Literal[True]:  # always returns True
    ...

MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
    ...

open_helper('/some/path', 'r')  # Passes type check
open_helper('/other/path', 'typo')  # Error in type checker
@end example

@code{Literal[...]} cannot be subclassed. At runtime, an arbitrary value
is allowed as type argument to @code{Literal[...]}, but type checkers may
impose restrictions. See 
@geindex Python Enhancement Proposals; PEP 586
PEP 586@footnote{https://www.python.org/dev/peps/pep-0586} for more details about literal types.


New in version 3.8.



Changed in version 3.9.1: @code{Literal} now de-duplicates parameters.  Equality comparisons of
@code{Literal} objects are no longer order dependent. @code{Literal} objects
will now raise a @ref{162,,TypeError} exception during equality comparisons
if one of their parameters are not @ref{2dc,,hashable}.

@end deffn

@geindex ClassVar (in module typing)
@anchor{library/typing typing ClassVar}@anchor{83e}
@deffn {Data} typing.ClassVar

Special type construct to mark class variables.

As introduced in 
@geindex Python Enhancement Proposals; PEP 526
PEP 526@footnote{https://www.python.org/dev/peps/pep-0526}, a variable annotation wrapped in ClassVar
indicates that a given attribute is intended to be used as a class variable
and should not be set on instances of that class. Usage:

@example
class Starship:
    stats: ClassVar[dict[str, int]] = @{@} # class variable
    damage: int = 10                     # instance variable
@end example

@ref{83e,,ClassVar} accepts only types and cannot be further subscribed.

@ref{83e,,ClassVar} is not a class itself, and should not
be used with @ref{236,,isinstance()} or @ref{237,,issubclass()}.
@ref{83e,,ClassVar} does not change Python runtime behavior, but
it can be used by third-party type checkers. For example, a type checker
might flag the following code as an error:

@example
enterprise_d = Starship(3000)
enterprise_d.stats = @{@} # Error, setting class variable on instance
Starship.stats = @{@}     # This is OK
@end example


New in version 3.5.3.

@end deffn

@geindex Final (in module typing)
@anchor{library/typing typing Final}@anchor{556}
@deffn {Data} typing.Final

A special typing construct to indicate to type checkers that a name
cannot be re-assigned or overridden in a subclass. For example:

@example
MAX_SIZE: Final = 9000
MAX_SIZE += 1  # Error reported by type checker

class Connection:
    TIMEOUT: Final[int] = 10

class FastConnector(Connection):
    TIMEOUT = 1  # Error reported by type checker
@end example

There is no runtime checking of these properties. See 
@geindex Python Enhancement Proposals; PEP 591
PEP 591@footnote{https://www.python.org/dev/peps/pep-0591} for
more details.


New in version 3.8.

@end deffn

@geindex Annotated (in module typing)
@anchor{library/typing typing Annotated}@anchor{431}
@deffn {Data} typing.Annotated

A type, introduced in 
@geindex Python Enhancement Proposals; PEP 593
PEP 593@footnote{https://www.python.org/dev/peps/pep-0593} (@code{Flexible function and variable
annotations}), to decorate existing types with context-specific metadata
(possibly multiple pieces of it, as @code{Annotated} is variadic).
Specifically, a type @code{T} can be annotated with metadata @code{x} via the
typehint @code{Annotated[T, x]}. This metadata can be used for either static
analysis or at runtime. If a library (or tool) encounters a typehint
@code{Annotated[T, x]} and has no special logic for metadata @code{x}, it
should ignore it and simply treat the type as @code{T}. Unlike the
@code{no_type_check} functionality that currently exists in the @code{typing}
module which completely disables typechecking annotations on a function
or a class, the @code{Annotated} type allows for both static typechecking
of @code{T} (e.g., via mypy or Pyre, which can safely ignore @code{x})
together with runtime access to @code{x} within a specific application.

Ultimately, the responsibility of how to interpret the annotations (if
at all) is the responsibility of the tool or library encountering the
@code{Annotated} type. A tool or library encountering an @code{Annotated} type
can scan through the annotations to determine if they are of interest
(e.g., using @code{isinstance()}).

When a tool or a library does not support annotations or encounters an
unknown annotation it should just ignore it and treat annotated type as
the underlying type.

It’s up to the tool consuming the annotations to decide whether the
client is allowed to have several annotations on one type and how to
merge those annotations.

Since the @code{Annotated} type allows you to put several annotations of
the same (or different) type(s) on any node, the tools or libraries
consuming those annotations are in charge of dealing with potential
duplicates. For example, if you are doing value range analysis you might
allow this:

@example
T1 = Annotated[int, ValueRange(-10, 5)]
T2 = Annotated[T1, ValueRange(-20, 3)]
@end example

Passing @code{include_extras=True} to @ref{432,,get_type_hints()} lets one
access the extra annotations at runtime.

The details of the syntax:


@itemize *

@item 
The first argument to @code{Annotated} must be a valid type

@item 
Multiple type annotations are supported (@code{Annotated} supports variadic
arguments):

@example
Annotated[int, ValueRange(3, 10), ctype("char")]
@end example

@item 
@code{Annotated} must be called with at least two arguments (
@code{Annotated[int]} is not valid)

@item 
The order of the annotations is preserved and matters for equality
checks:

@example
Annotated[int, ValueRange(3, 10), ctype("char")] != Annotated[
    int, ctype("char"), ValueRange(3, 10)
]
@end example

@item 
Nested @code{Annotated} types are flattened, with metadata ordered
starting with the innermost annotation:

@example
Annotated[Annotated[int, ValueRange(3, 10)], ctype("char")] == Annotated[
    int, ValueRange(3, 10), ctype("char")
]
@end example

@item 
Duplicated annotations are not removed:

@example
Annotated[int, ValueRange(3, 10)] != Annotated[
    int, ValueRange(3, 10), ValueRange(3, 10)
]
@end example

@item 
@code{Annotated} can be used with nested and generic aliases:

@example
T = TypeVar('T')
Vec = Annotated[list[tuple[T, T]], MaxLen(10)]
V = Vec[int]

V == Annotated[list[tuple[int, int]], MaxLen(10)]
@end example
@end itemize


New in version 3.9.

@end deffn

@geindex TypeGuard (in module typing)
@anchor{library/typing typing TypeGuard}@anchor{373c}
@deffn {Data} typing.TypeGuard

Special typing form used to annotate the return type of a user-defined
type guard function.  @code{TypeGuard} only accepts a single type argument.
At runtime, functions marked this way should return a boolean.

@code{TypeGuard} aims to benefit @emph{type narrowing} – a technique used by static
type checkers to determine a more precise type of an expression within a
program’s code flow.  Usually type narrowing is done by analyzing
conditional code flow and applying the narrowing to a block of code.  The
conditional expression here is sometimes referred to as a “type guard”:

@example
def is_str(val: str | float):
    # "isinstance" type guard
    if isinstance(val, str):
        # Type of `@w{`}val`@w{`} is narrowed to `@w{`}str`@w{`}
        ...
    else:
        # Else, type of `@w{`}val`@w{`} is narrowed to `@w{`}float`@w{`}.
        ...
@end example

Sometimes it would be convenient to use a user-defined boolean function
as a type guard.  Such a function should use @code{TypeGuard[...]} as its
return type to alert static type checkers to this intention.

Using  @code{-> TypeGuard} tells the static type checker that for a given
function:


@enumerate 

@item 
The return value is a boolean.

@item 
If the return value is @code{True}, the type of its argument
is the type inside @code{TypeGuard}.

For example:

@example
def is_str_list(val: list[object]) -> TypeGuard[list[str]]:
    '@w{'}'Determines whether all objects in the list are strings'@w{'}'
    return all(isinstance(x, str) for x in val)

def func1(val: list[object]):
    if is_str_list(val):
        # Type of `@w{`}val`@w{`} is narrowed to `@w{`}list[str]`@w{`}.
        print(" ".join(val))
    else:
        # Type of `@w{`}val`@w{`} remains as `@w{`}list[object]`@w{`}.
        print("Not a list of strings!")
@end example
@end enumerate

If @code{is_str_list} is a class or instance method, then the type in
@code{TypeGuard} maps to the type of the second parameter after @code{cls} or
@code{self}.

In short, the form @code{def foo(arg: TypeA) -> TypeGuard[TypeB]: ...},
means that if @code{foo(arg)} returns @code{True}, then @code{arg} narrows from
@code{TypeA} to @code{TypeB}.

@cartouche
@quotation Note 
@code{TypeB} need not be a narrower form of @code{TypeA} – it can even be a
wider form. The main reason is to allow for things like
narrowing @code{list[object]} to @code{list[str]} even though the latter
is not a subtype of the former, since @code{list} is invariant.
The responsibility of writing type-safe type guards is left to the user.
@end quotation
@end cartouche

@code{TypeGuard} also works with type variables.  For more information, see
@geindex Python Enhancement Proposals; PEP 647
PEP 647@footnote{https://www.python.org/dev/peps/pep-0647} (User-Defined Type Guards).


New in version 3.10.

@end deffn

@node Building generic types,Other special directives,Special forms,Special typing primitives
@anchor{library/typing building-generic-types}@anchor{374e}
@subsubsection Building generic types


These are not used in annotations. They are building blocks for creating generic types.

@geindex Generic (class in typing)
@anchor{library/typing typing Generic}@anchor{11c4}
@deffn {Class} typing.Generic

Abstract base class for generic types.

A generic type is typically declared by inheriting from an
instantiation of this class with one or more type variables.
For example, a generic mapping type might be defined as:

@example
class Mapping(Generic[KT, VT]):
    def __getitem__(self, key: KT) -> VT:
        ...
        # Etc.
@end example

This class can then be used as follows:

@example
X = TypeVar('X')
Y = TypeVar('Y')

def lookup_name(mapping: Mapping[X, Y], key: X, default: Y) -> Y:
    try:
        return mapping[key]
    except KeyError:
        return default
@end example
@end deffn

@geindex TypeVar (class in typing)
@anchor{library/typing typing TypeVar}@anchor{10ba}
@deffn {Class} typing.TypeVar

Type variable.

Usage:

@example
T = TypeVar('T')  # Can be anything
A = TypeVar('A', str, bytes)  # Must be str or bytes
@end example

Type variables exist primarily for the benefit of static type
checkers.  They serve as the parameters for generic types as well
as for generic function definitions.  See @ref{11c4,,Generic} for more
information on generic types.  Generic functions work as follows:

@example
def repeat(x: T, n: int) -> Sequence[T]:
    """Return a list containing n references to x."""
    return [x]*n

def longest(x: A, y: A) -> A:
    """Return the longest of two strings."""
    return x if len(x) >= len(y) else y
@end example

The latter example’s signature is essentially the overloading
of @code{(str, str) -> str} and @code{(bytes, bytes) -> bytes}.  Also note
that if the arguments are instances of some subclass of @ref{2ef,,str},
the return type is still plain @ref{2ef,,str}.

At runtime, @code{isinstance(x, T)} will raise @ref{162,,TypeError}.  In general,
@ref{236,,isinstance()} and @ref{237,,issubclass()} should not be used with types.

Type variables may be marked covariant or contravariant by passing
@code{covariant=True} or @code{contravariant=True}.  See 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} for more
details.  By default type variables are invariant.  Alternatively,
a type variable may specify an upper bound using @code{bound=<type>}.
This means that an actual type substituted (explicitly or implicitly)
for the type variable must be a subclass of the boundary type,
see 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484}.
@end deffn

@geindex ParamSpec (class in typing)
@anchor{library/typing typing ParamSpec}@anchor{23a}
@deffn {Class} typing.ParamSpec (name, *, bound=None, covariant=False, contravariant=False)

Parameter specification variable.  A specialized version of
@ref{10ba,,type variables}.

Usage:

@example
P = ParamSpec('P')
@end example

Parameter specification variables exist primarily for the benefit of static
type checkers.  They are used to forward the parameter types of one
callable to another callable – a pattern commonly found in higher order
functions and decorators.  They are only valid when used in @code{Concatenate},
or as the first argument to @code{Callable}, or as parameters for user-defined
Generics.  See @ref{11c4,,Generic} for more information on generic types.

For example, to add basic logging to a function, one can create a decorator
@code{add_logging} to log function calls.  The parameter specification variable
tells the type checker that the callable passed into the decorator and the
new callable returned by it have inter-dependent type parameters:

@example
from collections.abc import Callable
from typing import TypeVar, ParamSpec
import logging

T = TypeVar('T')
P = ParamSpec('P')

def add_logging(f: Callable[P, T]) -> Callable[P, T]:
    '@w{'}'A type-safe decorator to add logging to a function.'@w{'}'
    def inner(*args: P.args, **kwargs: P.kwargs) -> T:
        logging.info(f'@{f.__name__@} was called')
        return f(*args, **kwargs)
    return inner

@@add_logging
def add_two(x: float, y: float) -> float:
    '@w{'}'Add two numbers together.'@w{'}'
    return x + y
@end example

Without @code{ParamSpec}, the simplest way to annotate this previously was to
use a @ref{10ba,,TypeVar} with bound @code{Callable[..., Any]}.  However this
causes two problems:

@quotation


@enumerate 

@item 
The type checker can’t type check the @code{inner} function because
@code{*args} and @code{**kwargs} have to be typed @ref{8dc,,Any}.

@item 
@ref{374f,,cast()} may be required in the body of the @code{add_logging}
decorator when returning the @code{inner} function, or the static type
checker must be told to ignore the @code{return inner}.
@end enumerate
@end quotation

@geindex args (typing.ParamSpec attribute)
@anchor{library/typing typing ParamSpec args}@anchor{3750}
@deffn {Attribute} args
@end deffn

@geindex kwargs (typing.ParamSpec attribute)
@anchor{library/typing typing ParamSpec kwargs}@anchor{3751}
@deffn {Attribute} kwargs

Since @code{ParamSpec} captures both positional and keyword parameters,
@code{P.args} and @code{P.kwargs} can be used to split a @code{ParamSpec} into its
components.  @code{P.args} represents the tuple of positional parameters in a
given call and should only be used to annotate @code{*args}.  @code{P.kwargs}
represents the mapping of keyword parameters to their values in a given call,
and should be only be used to annotate @code{**kwargs}.  Both
attributes require the annotated parameter to be in scope. At runtime,
@code{P.args} and @code{P.kwargs} are instances respectively of
@ref{23d,,ParamSpecArgs} and @ref{23e,,ParamSpecKwargs}.
@end deffn

Parameter specification variables created with @code{covariant=True} or
@code{contravariant=True} can be used to declare covariant or contravariant
generic types.  The @code{bound} argument is also accepted, similar to
@ref{10ba,,TypeVar}.  However the actual semantics of these keywords are yet to
be decided.


New in version 3.10.


@cartouche
@quotation Note 
Only parameter specification variables defined in global scope can
be pickled.
@end quotation
@end cartouche


@subsubheading See also



@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 612
PEP 612@footnote{https://www.python.org/dev/peps/pep-0612} – Parameter Specification Variables (the PEP which introduced
@code{ParamSpec} and @code{Concatenate}).

@item 
@ref{23c,,Callable} and @ref{23b,,Concatenate}.
@end itemize

@end deffn

@geindex ParamSpecArgs (in module typing)
@anchor{library/typing typing ParamSpecArgs}@anchor{23d}
@deffn {Data} typing.ParamSpecArgs
@end deffn

@geindex ParamSpecKwargs (in module typing)
@anchor{library/typing typing ParamSpecKwargs}@anchor{23e}
@deffn {Data} typing.ParamSpecKwargs

Arguments and keyword arguments attributes of a @ref{23a,,ParamSpec}. The
@code{P.args} attribute of a @code{ParamSpec} is an instance of @code{ParamSpecArgs},
and @code{P.kwargs} is an instance of @code{ParamSpecKwargs}. They are intended
for runtime introspection and have no special meaning to static type checkers.

Calling @ref{55c,,get_origin()} on either of these objects will return the
original @code{ParamSpec}:

@example
P = ParamSpec("P")
get_origin(P.args)  # returns P
get_origin(P.kwargs)  # returns P
@end example


New in version 3.10.

@end deffn

@geindex AnyStr (in module typing)
@anchor{library/typing typing AnyStr}@anchor{3752}
@deffn {Data} typing.AnyStr

@code{AnyStr} is a type variable defined as
@code{AnyStr = TypeVar('AnyStr', str, bytes)}.

It is meant to be used for functions that may accept any kind of string
without allowing different kinds of strings to mix. For example:

@example
def concat(a: AnyStr, b: AnyStr) -> AnyStr:
    return a + b

concat(u"foo", u"bar")  # Ok, output has type 'unicode'
concat(b"foo", b"bar")  # Ok, output has type 'bytes'
concat(u"foo", b"bar")  # Error, cannot mix unicode and bytes
@end example
@end deffn

@geindex Protocol (class in typing)
@anchor{library/typing typing Protocol}@anchor{558}
@deffn {Class} typing.Protocol (Generic)

Base class for protocol classes. Protocol classes are defined like this:

@example
class Proto(Protocol):
    def meth(self) -> int:
        ...
@end example

Such classes are primarily used with static type checkers that recognize
structural subtyping (static duck-typing), for example:

@example
class C:
    def meth(self) -> int:
        return 0

def func(x: Proto) -> int:
    return x.meth()

func(C())  # Passes static type check
@end example

See 
@geindex Python Enhancement Proposals; PEP 544
PEP 544@footnote{https://www.python.org/dev/peps/pep-0544} for details. Protocol classes decorated with
@ref{559,,runtime_checkable()} (described later) act as simple-minded runtime
protocols that check only the presence of given attributes, ignoring their
type signatures.

Protocol classes can be generic, for example:

@example
class GenProto(Protocol[T]):
    def meth(self) -> T:
        ...
@end example


New in version 3.8.

@end deffn

@geindex runtime_checkable() (in module typing)
@anchor{library/typing typing runtime_checkable}@anchor{559}
@deffn {Function} @@typing.runtime_checkable

Mark a protocol class as a runtime protocol.

Such a protocol can be used with @ref{236,,isinstance()} and @ref{237,,issubclass()}.
This raises @ref{162,,TypeError} when applied to a non-protocol class.  This
allows a simple-minded structural check, very similar to “one trick ponies”
in @ref{1d,,collections.abc} such as @ref{1b3c,,Iterable}.  For example:

@example
@@runtime_checkable
class Closable(Protocol):
    def close(self): ...

assert isinstance(open('/some/file'), Closable)
@end example

@cartouche
@quotation Note 
@ref{559,,runtime_checkable()} will check only the presence of the required
methods, not their type signatures. For example, @ref{6af,,ssl.SSLObject}
is a class, therefore it passes an @ref{237,,issubclass()}
check against @ref{23c,,Callable}.  However, the
@code{ssl.SSLObject.__init__()} method exists only to raise a
@ref{162,,TypeError} with a more informative message, therefore making
it impossible to call (instantiate) @ref{6af,,ssl.SSLObject}.
@end quotation
@end cartouche


New in version 3.8.

@end deffn

@node Other special directives,,Building generic types,Special typing primitives
@anchor{library/typing other-special-directives}@anchor{3753}
@subsubsection Other special directives


These are not used in annotations. They are building blocks for declaring types.

@geindex NamedTuple (class in typing)
@anchor{library/typing typing NamedTuple}@anchor{457}
@deffn {Class} typing.NamedTuple

Typed version of @ref{3e4,,collections.namedtuple()}.

Usage:

@example
class Employee(NamedTuple):
    name: str
    id: int
@end example

This is equivalent to:

@example
Employee = collections.namedtuple('Employee', ['name', 'id'])
@end example

To give a field a default value, you can assign to it in the class body:

@example
class Employee(NamedTuple):
    name: str
    id: int = 3

employee = Employee('Guido')
assert employee.id == 3
@end example

Fields with a default value must come after any fields without a default.

The resulting class has an extra attribute @code{__annotations__} giving a
dict that maps the field names to the field types.  (The field names are in
the @code{_fields} attribute and the default values are in the
@code{_field_defaults} attribute both of which are part of the namedtuple
API.)

@code{NamedTuple} subclasses can also have docstrings and methods:

@example
class Employee(NamedTuple):
    """Represents an employee."""
    name: str
    id: int = 3

    def __repr__(self) -> str:
        return f'<Employee @{self.name@}, id=@{self.id@}>'
@end example

Backward-compatible usage:

@example
Employee = NamedTuple('Employee', [('name', str), ('id', int)])
@end example


Changed in version 3.6: Added support for 
@geindex Python Enhancement Proposals; PEP 526
PEP 526@footnote{https://www.python.org/dev/peps/pep-0526} variable annotation syntax.



Changed in version 3.6.1: Added support for default values, methods, and docstrings.



Changed in version 3.8: The @code{_field_types} and @code{__annotations__} attributes are
now regular dictionaries instead of instances of @code{OrderedDict}.



Changed in version 3.9: Removed the @code{_field_types} attribute in favor of the more
standard @code{__annotations__} attribute which has the same information.

@end deffn

@geindex NewType (class in typing)
@anchor{library/typing typing NewType}@anchor{840}
@deffn {Class} typing.NewType (name, tp)

A helper class to indicate a distinct type to a typechecker,
see @ref{373f,,NewType}. At runtime it returns an object that returns
its argument when called.
Usage:

@example
UserId = NewType('UserId', int)
first_user = UserId(1)
@end example


New in version 3.5.2.



Changed in version 3.10: @code{NewType} is now a class rather than a function.

@end deffn

@geindex TypedDict (class in typing)
@anchor{library/typing typing TypedDict}@anchor{2de}
@deffn {Class} typing.TypedDict (dict)

Special construct to add type hints to a dictionary.
At runtime it is a plain @ref{3b0,,dict}.

@code{TypedDict} declares a dictionary type that expects all of its
instances to have a certain set of keys, where each key is
associated with a value of a consistent type. This expectation
is not checked at runtime but is only enforced by type checkers.
Usage:

@example
class Point2D(TypedDict):
    x: int
    y: int
    label: str

a: Point2D = @{'x': 1, 'y': 2, 'label': 'good'@}  # OK
b: Point2D = @{'z': 3, 'label': 'bad'@}           # Fails type check

assert Point2D(x=1, y=2, label='first') == dict(x=1, y=2, label='first')
@end example

The type info for introspection can be accessed via @code{Point2D.__annotations__},
@code{Point2D.__total__}, @code{Point2D.__required_keys__}, and
@code{Point2D.__optional_keys__}.
To allow using this feature with older versions of Python that do not
support 
@geindex Python Enhancement Proposals; PEP 526
PEP 526@footnote{https://www.python.org/dev/peps/pep-0526}, @code{TypedDict} supports two additional equivalent
syntactic forms:


@itemize *

@item 
Using a literal @ref{3b0,,dict} as the second argument:

@example
Point2D = TypedDict('Point2D', @{'x': int, 'y': int, 'label': str@})
@end example

@item 
Using keyword arguments:

@example
Point2D = TypedDict('Point2D', x=int, y=int, label=str)
@end example
@end itemize


Deprecated since version 3.11, will be removed in version 3.13: The keyword-argument syntax is deprecated in 3.11 and will be removed
in 3.13. It may also be unsupported by static type checkers.


By default, all keys must be present in a @code{TypedDict}. It is possible to
override this by specifying totality.
Usage:

@example
class Point2D(TypedDict, total=False):
    x: int
    y: int

# Alternative syntax
Point2D = TypedDict('Point2D', @{'x': int, 'y': int@}, total=False)
@end example

This means that a @code{Point2D} @code{TypedDict} can have any of the keys
omitted. A type checker is only expected to support a literal @code{False} or
@code{True} as the value of the @code{total} argument. @code{True} is the default,
and makes all items defined in the class body required.

See 
@geindex Python Enhancement Proposals; PEP 589
PEP 589@footnote{https://www.python.org/dev/peps/pep-0589} for more examples and detailed rules of using @code{TypedDict}.


New in version 3.8.

@end deffn

@node Generic concrete collections,Abstract Base Classes,Special typing primitives,Module contents<3>
@anchor{library/typing generic-concrete-collections}@anchor{3754}
@subsubsection Generic concrete collections


@menu
* Corresponding to built-in types:: 
* Corresponding to types in collections:: 
* Other concrete types:: 

@end menu

@node Corresponding to built-in types,Corresponding to types in collections,,Generic concrete collections
@anchor{library/typing corresponding-to-built-in-types}@anchor{3755}
@subsubsection Corresponding to built-in types


@geindex Dict (class in typing)
@anchor{library/typing typing Dict}@anchor{3756}
@deffn {Class} typing.Dict (dict, MutableMapping[KT, VT])

A generic version of @ref{3b0,,dict}.
Useful for annotating return types. To annotate arguments it is preferred
to use an abstract collection type such as @ref{3757,,Mapping}.

This type can be used as follows:

@example
def count_words(text: str) -> Dict[str, int]:
    ...
@end example


Deprecated since version 3.9: @ref{3b0,,builtins.dict} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and
@ref{268,,Generic Alias Type}.

@end deffn

@geindex List (class in typing)
@anchor{library/typing typing List}@anchor{3758}
@deffn {Class} typing.List (list, MutableSequence[T])

Generic version of @ref{43a,,list}.
Useful for annotating return types. To annotate arguments it is preferred
to use an abstract collection type such as @ref{3759,,Sequence} or
@ref{375a,,Iterable}.

This type may be used as follows:

@example
T = TypeVar('T', int, float)

def vec2(x: T, y: T) -> List[T]:
    return [x, y]

def keep_positives(vector: Sequence[T]) -> List[T]:
    return [item for item in vector if item > 0]
@end example


Deprecated since version 3.9: @ref{43a,,builtins.list} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and
@ref{268,,Generic Alias Type}.

@end deffn

@geindex Set (class in typing)
@anchor{library/typing typing Set}@anchor{375b}
@deffn {Class} typing.Set (set, MutableSet[T])

A generic version of @ref{370,,builtins.set}.
Useful for annotating return types. To annotate arguments it is preferred
to use an abstract collection type such as @ref{375c,,AbstractSet}.


Deprecated since version 3.9: @ref{370,,builtins.set} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and
@ref{268,,Generic Alias Type}.

@end deffn

@geindex FrozenSet (class in typing)
@anchor{library/typing typing FrozenSet}@anchor{375d}
@deffn {Class} typing.FrozenSet (frozenset, AbstractSet[T_co])

A generic version of @ref{439,,builtins.frozenset}.


Deprecated since version 3.9: @ref{439,,builtins.frozenset} now supports @code{[]}. See
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and @ref{268,,Generic Alias Type}.

@end deffn

@cartouche
@quotation Note 
@ref{374b,,Tuple} is a special form.
@end quotation
@end cartouche

@node Corresponding to types in collections,Other concrete types,Corresponding to built-in types,Generic concrete collections
@anchor{library/typing corresponding-to-types-in-collections}@anchor{375e}
@subsubsection Corresponding to types in @code{collections}


@geindex DefaultDict (class in typing)
@anchor{library/typing typing DefaultDict}@anchor{375f}
@deffn {Class} typing.DefaultDict (collections.defaultdict, MutableMapping[KT, VT])

A generic version of @ref{d63,,collections.defaultdict}.


New in version 3.5.2.



Deprecated since version 3.9: @ref{d63,,collections.defaultdict} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and
@ref{268,,Generic Alias Type}.

@end deffn

@geindex OrderedDict (class in typing)
@anchor{library/typing typing OrderedDict}@anchor{3760}
@deffn {Class} typing.OrderedDict (collections.OrderedDict, MutableMapping[KT, VT])

A generic version of @ref{4e5,,collections.OrderedDict}.


New in version 3.7.2.



Deprecated since version 3.9: @ref{4e5,,collections.OrderedDict} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and
@ref{268,,Generic Alias Type}.

@end deffn

@geindex ChainMap (class in typing)
@anchor{library/typing typing ChainMap}@anchor{3761}
@deffn {Class} typing.ChainMap (collections.ChainMap, MutableMapping[KT, VT])

A generic version of @ref{758,,collections.ChainMap}.


New in version 3.5.4.



New in version 3.6.1.



Deprecated since version 3.9: @ref{758,,collections.ChainMap} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and
@ref{268,,Generic Alias Type}.

@end deffn

@geindex Counter (class in typing)
@anchor{library/typing typing Counter}@anchor{3762}
@deffn {Class} typing.Counter (collections.Counter, Dict[T, int])

A generic version of @ref{c2a,,collections.Counter}.


New in version 3.5.4.



New in version 3.6.1.



Deprecated since version 3.9: @ref{c2a,,collections.Counter} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and
@ref{268,,Generic Alias Type}.

@end deffn

@geindex Deque (class in typing)
@anchor{library/typing typing Deque}@anchor{3763}
@deffn {Class} typing.Deque (deque, MutableSequence[T])

A generic version of @ref{7d3,,collections.deque}.


New in version 3.5.4.



New in version 3.6.1.



Deprecated since version 3.9: @ref{7d3,,collections.deque} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and
@ref{268,,Generic Alias Type}.

@end deffn

@node Other concrete types,,Corresponding to types in collections,Generic concrete collections
@anchor{library/typing other-concrete-types}@anchor{3764}
@subsubsection Other concrete types


@geindex IO (class in typing)
@geindex TextIO (class in typing)
@geindex BinaryIO (class in typing)
@anchor{library/typing typing IO}@anchor{3765}
@deffn {Class} typing.IO
@anchor{library/typing typing TextIO}@anchor{3766}
@deffnx {Class} typing.TextIO
@anchor{library/typing typing BinaryIO}@anchor{3767}
@deffnx {Class} typing.BinaryIO

Generic type @code{IO[AnyStr]} and its subclasses @code{TextIO(IO[str])}
and @code{BinaryIO(IO[bytes])}
represent the types of I/O streams such as returned by
@ref{1c8,,open()}.


Deprecated since version 3.8, will be removed in version 3.12: The @code{typing.io} namespace is deprecated and will be removed.
These types should be directly imported from @code{typing} instead.

@end deffn

@geindex Pattern (class in typing)
@geindex Match (class in typing)
@anchor{library/typing typing Pattern}@anchor{3768}
@deffn {Class} typing.Pattern
@anchor{library/typing typing Match}@anchor{3769}
@deffnx {Class} typing.Match

These type aliases
correspond to the return types from @ref{70b,,re.compile()} and
@ref{dd3,,re.match()}.  These types (and the corresponding functions)
are generic in @code{AnyStr} and can be made specific by writing
@code{Pattern[str]}, @code{Pattern[bytes]}, @code{Match[str]}, or
@code{Match[bytes]}.


Deprecated since version 3.8, will be removed in version 3.12: The @code{typing.re} namespace is deprecated and will be removed.
These types should be directly imported from @code{typing} instead.



Deprecated since version 3.9: Classes @code{Pattern} and @code{Match} from @ref{d8,,re} now support @code{[]}.
See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and @ref{268,,Generic Alias Type}.

@end deffn

@geindex Text (class in typing)
@anchor{library/typing typing Text}@anchor{376a}
@deffn {Class} typing.Text

@code{Text} is an alias for @code{str}. It is provided to supply a forward
compatible path for Python 2 code: in Python 2, @code{Text} is an alias for
@code{unicode}.

Use @code{Text} to indicate that a value must contain a unicode string in
a manner that is compatible with both Python 2 and Python 3:

@example
def add_unicode_checkmark(text: Text) -> Text:
    return text + u' \u2713'
@end example


New in version 3.5.2.

@end deffn

@node Abstract Base Classes,Protocols<3>,Generic concrete collections,Module contents<3>
@anchor{library/typing abstract-base-classes}@anchor{376b}
@subsubsection Abstract Base Classes


@menu
* Corresponding to collections in collections.abc: Corresponding to collections in collections abc. 
* Corresponding to other types in collections.abc: Corresponding to other types in collections abc. 
* Asynchronous programming:: 
* Context manager types:: 

@end menu

@node Corresponding to collections in collections abc,Corresponding to other types in collections abc,,Abstract Base Classes
@anchor{library/typing corresponding-to-collections-in-collections-abc}@anchor{376c}
@subsubsection Corresponding to collections in @code{collections.abc}


@geindex AbstractSet (class in typing)
@anchor{library/typing typing AbstractSet}@anchor{375c}
@deffn {Class} typing.AbstractSet (Sized, Collection[T_co])

A generic version of @ref{1b33,,collections.abc.Set}.


Deprecated since version 3.9: @ref{1b33,,collections.abc.Set} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and
@ref{268,,Generic Alias Type}.

@end deffn

@geindex ByteString (class in typing)
@anchor{library/typing typing ByteString}@anchor{376d}
@deffn {Class} typing.ByteString (Sequence[int])

A generic version of @ref{1b40,,collections.abc.ByteString}.

This type represents the types @ref{2f0,,bytes}, @ref{2f1,,bytearray},
and @ref{573,,memoryview} of byte sequences.

As a shorthand for this type, @ref{2f0,,bytes} can be used to
annotate arguments of any of the types mentioned above.


Deprecated since version 3.9: @ref{1b40,,collections.abc.ByteString} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex Collection (class in typing)
@anchor{library/typing typing Collection}@anchor{83d}
@deffn {Class} typing.Collection (Sized, Iterable[T_co], Container[T_co])

A generic version of @ref{7d0,,collections.abc.Collection}


New in version 3.6.0.



Deprecated since version 3.9: @ref{7d0,,collections.abc.Collection} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex Container (class in typing)
@anchor{library/typing typing Container}@anchor{376e}
@deffn {Class} typing.Container (Generic[T_co])

A generic version of @ref{1b3e,,collections.abc.Container}.


Deprecated since version 3.9: @ref{1b3e,,collections.abc.Container} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex ItemsView (class in typing)
@anchor{library/typing typing ItemsView}@anchor{376f}
@deffn {Class} typing.ItemsView (MappingView, Generic[KT_co, VT_co])

A generic version of @ref{1b43,,collections.abc.ItemsView}.


Deprecated since version 3.9: @ref{1b43,,collections.abc.ItemsView} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex KeysView (class in typing)
@anchor{library/typing typing KeysView}@anchor{3770}
@deffn {Class} typing.KeysView (MappingView[KT_co], AbstractSet[KT_co])

A generic version of @ref{1b42,,collections.abc.KeysView}.


Deprecated since version 3.9: @ref{1b42,,collections.abc.KeysView} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex Mapping (class in typing)
@anchor{library/typing typing Mapping}@anchor{3757}
@deffn {Class} typing.Mapping (Sized, Collection[KT], Generic[VT_co])

A generic version of @ref{3ab,,collections.abc.Mapping}.
This type can be used as follows:

@example
def get_position_in_index(word_list: Mapping[str, int], word: str) -> int:
    return word_list[word]
@end example


Deprecated since version 3.9: @ref{3ab,,collections.abc.Mapping} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex MappingView (class in typing)
@anchor{library/typing typing MappingView}@anchor{3771}
@deffn {Class} typing.MappingView (Sized, Iterable[T_co])

A generic version of @ref{1b41,,collections.abc.MappingView}.


Deprecated since version 3.9: @ref{1b41,,collections.abc.MappingView} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex MutableMapping (class in typing)
@anchor{library/typing typing MutableMapping}@anchor{3772}
@deffn {Class} typing.MutableMapping (Mapping[KT, VT])

A generic version of @ref{c3f,,collections.abc.MutableMapping}.


Deprecated since version 3.9: @ref{c3f,,collections.abc.MutableMapping} now supports @code{[]}. See
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and @ref{268,,Generic Alias Type}.

@end deffn

@geindex MutableSequence (class in typing)
@anchor{library/typing typing MutableSequence}@anchor{3773}
@deffn {Class} typing.MutableSequence (Sequence[T])

A generic version of @ref{933,,collections.abc.MutableSequence}.


Deprecated since version 3.9: @ref{933,,collections.abc.MutableSequence} now supports @code{[]}. See
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and @ref{268,,Generic Alias Type}.

@end deffn

@geindex MutableSet (class in typing)
@anchor{library/typing typing MutableSet}@anchor{3774}
@deffn {Class} typing.MutableSet (AbstractSet[T])

A generic version of @ref{1b3f,,collections.abc.MutableSet}.


Deprecated since version 3.9: @ref{1b3f,,collections.abc.MutableSet} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex Sequence (class in typing)
@anchor{library/typing typing Sequence}@anchor{3759}
@deffn {Class} typing.Sequence (Reversible[T_co], Collection[T_co])

A generic version of @ref{110c,,collections.abc.Sequence}.


Deprecated since version 3.9: @ref{110c,,collections.abc.Sequence} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex ValuesView (class in typing)
@anchor{library/typing typing ValuesView}@anchor{3775}
@deffn {Class} typing.ValuesView (MappingView[VT_co])

A generic version of @ref{1b44,,collections.abc.ValuesView}.


Deprecated since version 3.9: @ref{1b44,,collections.abc.ValuesView} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@node Corresponding to other types in collections abc,Asynchronous programming,Corresponding to collections in collections abc,Abstract Base Classes
@anchor{library/typing corresponding-to-other-types-in-collections-abc}@anchor{3776}
@subsubsection Corresponding to other types in @code{collections.abc}


@geindex Iterable (class in typing)
@anchor{library/typing typing Iterable}@anchor{375a}
@deffn {Class} typing.Iterable (Generic[T_co])

A generic version of @ref{1b3c,,collections.abc.Iterable}.


Deprecated since version 3.9: @ref{1b3c,,collections.abc.Iterable} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex Iterator (class in typing)
@anchor{library/typing typing Iterator}@anchor{3777}
@deffn {Class} typing.Iterator (Iterable[T_co])

A generic version of @ref{1b3d,,collections.abc.Iterator}.


Deprecated since version 3.9: @ref{1b3d,,collections.abc.Iterator} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex Generator (class in typing)
@anchor{library/typing typing Generator}@anchor{3778}
@deffn {Class} typing.Generator (Iterator[T_co], Generic[T_co, T_contra, V_co])

A generator can be annotated by the generic type
@code{Generator[YieldType, SendType, ReturnType]}. For example:

@example
def echo_round() -> Generator[int, float, str]:
    sent = yield 0
    while sent >= 0:
        sent = yield round(sent)
    return 'Done'
@end example

Note that unlike many other generics in the typing module, the @code{SendType}
of @ref{3778,,Generator} behaves contravariantly, not covariantly or
invariantly.

If your generator will only yield values, set the @code{SendType} and
@code{ReturnType} to @code{None}:

@example
def infinite_stream(start: int) -> Generator[int, None, None]:
    while True:
        yield start
        start += 1
@end example

Alternatively, annotate your generator as having a return type of
either @code{Iterable[YieldType]} or @code{Iterator[YieldType]}:

@example
def infinite_stream(start: int) -> Iterator[int]:
    while True:
        yield start
        start += 1
@end example


Deprecated since version 3.9: @ref{93a,,collections.abc.Generator} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex Hashable (class in typing)
@anchor{library/typing typing Hashable}@anchor{3779}
@deffn {Class} typing.Hashable

An alias to @ref{1de9,,collections.abc.Hashable}
@end deffn

@geindex Reversible (class in typing)
@anchor{library/typing typing Reversible}@anchor{377a}
@deffn {Class} typing.Reversible (Iterable[T_co])

A generic version of @ref{7d1,,collections.abc.Reversible}.


Deprecated since version 3.9: @ref{7d1,,collections.abc.Reversible} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex Sized (class in typing)
@anchor{library/typing typing Sized}@anchor{377b}
@deffn {Class} typing.Sized

An alias to @ref{1dea,,collections.abc.Sized}
@end deffn

@node Asynchronous programming,Context manager types,Corresponding to other types in collections abc,Abstract Base Classes
@anchor{library/typing asynchronous-programming}@anchor{377c}
@subsubsection Asynchronous programming


@geindex Coroutine (class in typing)
@anchor{library/typing typing Coroutine}@anchor{11cf}
@deffn {Class} typing.Coroutine (Awaitable[V_co], Generic[T_co, T_contra, V_co])

A generic version of @ref{93c,,collections.abc.Coroutine}.
The variance and order of type variables
correspond to those of @ref{3778,,Generator}, for example:

@example
from collections.abc import Coroutine
c: Coroutine[list[str], str, int]  # Some coroutine defined elsewhere
x = c.send('hi')                   # Inferred type of 'x' is list[str]
async def bar() -> None:
    y = await c                    # Inferred type of 'y' is int
@end example


New in version 3.5.3.



Deprecated since version 3.9: @ref{93c,,collections.abc.Coroutine} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex AsyncGenerator (class in typing)
@anchor{library/typing typing AsyncGenerator}@anchor{377d}
@deffn {Class} typing.AsyncGenerator (AsyncIterator[T_co], Generic[T_co, T_contra])

An async generator can be annotated by the generic type
@code{AsyncGenerator[YieldType, SendType]}. For example:

@example
async def echo_round() -> AsyncGenerator[int, float]:
    sent = yield 0
    while sent >= 0.0:
        rounded = await round(sent)
        sent = yield rounded
@end example

Unlike normal generators, async generators cannot return a value, so there
is no @code{ReturnType} type parameter. As with @ref{3778,,Generator}, the
@code{SendType} behaves contravariantly.

If your generator will only yield values, set the @code{SendType} to
@code{None}:

@example
async def infinite_stream(start: int) -> AsyncGenerator[int, None]:
    while True:
        yield start
        start = await increment(start)
@end example

Alternatively, annotate your generator as having a return type of
either @code{AsyncIterable[YieldType]} or @code{AsyncIterator[YieldType]}:

@example
async def infinite_stream(start: int) -> AsyncIterator[int]:
    while True:
        yield start
        start = await increment(start)
@end example


New in version 3.6.1.



Deprecated since version 3.9: @ref{7d2,,collections.abc.AsyncGenerator} now supports @code{[]}. See
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and @ref{268,,Generic Alias Type}.

@end deffn

@geindex AsyncIterable (class in typing)
@anchor{library/typing typing AsyncIterable}@anchor{377e}
@deffn {Class} typing.AsyncIterable (Generic[T_co])

A generic version of @ref{93e,,collections.abc.AsyncIterable}.


New in version 3.5.2.



Deprecated since version 3.9: @ref{93e,,collections.abc.AsyncIterable} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex AsyncIterator (class in typing)
@anchor{library/typing typing AsyncIterator}@anchor{377f}
@deffn {Class} typing.AsyncIterator (AsyncIterable[T_co])

A generic version of @ref{93d,,collections.abc.AsyncIterator}.


New in version 3.5.2.



Deprecated since version 3.9: @ref{93d,,collections.abc.AsyncIterator} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@geindex Awaitable (class in typing)
@anchor{library/typing typing Awaitable}@anchor{3780}
@deffn {Class} typing.Awaitable (Generic[T_co])

A generic version of @ref{93b,,collections.abc.Awaitable}.


New in version 3.5.2.



Deprecated since version 3.9: @ref{93b,,collections.abc.Awaitable} now supports @code{[]}. See 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}
and @ref{268,,Generic Alias Type}.

@end deffn

@node Context manager types,,Asynchronous programming,Abstract Base Classes
@anchor{library/typing context-manager-types}@anchor{3781}
@subsubsection Context manager types


@geindex ContextManager (class in typing)
@anchor{library/typing typing ContextManager}@anchor{7d7}
@deffn {Class} typing.ContextManager (Generic[T_co])

A generic version of @ref{7d6,,contextlib.AbstractContextManager}.


New in version 3.5.4.



New in version 3.6.0.



Deprecated since version 3.9: @ref{7d6,,contextlib.AbstractContextManager} now supports @code{[]}. See
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and @ref{268,,Generic Alias Type}.

@end deffn

@geindex AsyncContextManager (class in typing)
@anchor{library/typing typing AsyncContextManager}@anchor{3782}
@deffn {Class} typing.AsyncContextManager (Generic[T_co])

A generic version of @ref{64a,,contextlib.AbstractAsyncContextManager}.


New in version 3.5.4.



New in version 3.6.2.



Deprecated since version 3.9: @ref{64a,,contextlib.AbstractAsyncContextManager} now supports @code{[]}. See
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} and @ref{268,,Generic Alias Type}.

@end deffn

@node Protocols<3>,Functions and decorators,Abstract Base Classes,Module contents<3>
@anchor{library/typing protocols}@anchor{3783}
@subsubsection Protocols


These protocols are decorated with @ref{559,,runtime_checkable()}.

@geindex SupportsAbs (class in typing)
@anchor{library/typing typing SupportsAbs}@anchor{3784}
@deffn {Class} typing.SupportsAbs

An ABC with one abstract method @code{__abs__} that is covariant
in its return type.
@end deffn

@geindex SupportsBytes (class in typing)
@anchor{library/typing typing SupportsBytes}@anchor{16c}
@deffn {Class} typing.SupportsBytes

An ABC with one abstract method @code{__bytes__}.
@end deffn

@geindex SupportsComplex (class in typing)
@anchor{library/typing typing SupportsComplex}@anchor{16b}
@deffn {Class} typing.SupportsComplex

An ABC with one abstract method @code{__complex__}.
@end deffn

@geindex SupportsFloat (class in typing)
@anchor{library/typing typing SupportsFloat}@anchor{3785}
@deffn {Class} typing.SupportsFloat

An ABC with one abstract method @code{__float__}.
@end deffn

@geindex SupportsIndex (class in typing)
@anchor{library/typing typing SupportsIndex}@anchor{55b}
@deffn {Class} typing.SupportsIndex

An ABC with one abstract method @code{__index__}.


New in version 3.8.

@end deffn

@geindex SupportsInt (class in typing)
@anchor{library/typing typing SupportsInt}@anchor{55a}
@deffn {Class} typing.SupportsInt

An ABC with one abstract method @code{__int__}.
@end deffn

@geindex SupportsRound (class in typing)
@anchor{library/typing typing SupportsRound}@anchor{3786}
@deffn {Class} typing.SupportsRound

An ABC with one abstract method @code{__round__}
that is covariant in its return type.
@end deffn

@node Functions and decorators,Introspection helpers,Protocols<3>,Module contents<3>
@anchor{library/typing functions-and-decorators}@anchor{3787}
@subsubsection Functions and decorators


@geindex cast() (in module typing)
@anchor{library/typing typing cast}@anchor{374f}
@deffn {Function} typing.cast (typ, val)

Cast a value to a type.

This returns the value unchanged.  To the type checker this
signals that the return value has the designated type, but at
runtime we intentionally don’t check anything (we want this
to be as fast as possible).
@end deffn

@geindex assert_never() (in module typing)
@anchor{library/typing typing assert_never}@anchor{1094}
@deffn {Function} typing.assert_never (arg, /)

Assert to the type checker that a line of code is unreachable.

Example:

@example
def int_or_str(arg: int | str) -> None:
    match arg:
        case int():
            print("It's an int")
        case str():
            print("It's a str")
        case _ as unreachable:
            assert_never(unreachable)
@end example

If a type checker finds that a call to @code{assert_never()} is
reachable, it will emit an error.

At runtime, this throws an exception when called.


New in version 3.11.

@end deffn

@geindex reveal_type() (in module typing)
@anchor{library/typing typing reveal_type}@anchor{10a6}
@deffn {Function} typing.reveal_type (obj)

Reveal the inferred static type of an expression.

When a static type checker encounters a call to this function,
it emits a diagnostic with the type of the argument. For example:

@example
x: int = 1
reveal_type(x)  # Revealed type is "builtins.int"
@end example

This can be useful when you want to debug how your type checker
handles a particular piece of code.

The function returns its argument unchanged, which allows using
it within an expression:

@example
x = reveal_type(1)  # Revealed type is "builtins.int"
@end example

Most type checkers support @code{reveal_type()} anywhere, even if the
name is not imported from @code{typing}. Importing the name from
@code{typing} allows your code to run without runtime errors and
communicates intent more clearly.

At runtime, this function prints the runtime type of its argument to stderr
and returns it unchanged:

@example
x = reveal_type(1)  # prints "Runtime type is int"
print(x)  # prints "1"
@end example


New in version 3.11.

@end deffn

@geindex overload() (in module typing)
@anchor{library/typing typing overload}@anchor{3788}
@deffn {Function} @@typing.overload

The @code{@@overload} decorator allows describing functions and methods
that support multiple different combinations of argument types. A series
of @code{@@overload}-decorated definitions must be followed by exactly one
non-@code{@@overload}-decorated definition (for the same function/method).
The @code{@@overload}-decorated definitions are for the benefit of the
type checker only, since they will be overwritten by the
non-@code{@@overload}-decorated definition, while the latter is used at
runtime but should be ignored by a type checker.  At runtime, calling
a @code{@@overload}-decorated function directly will raise
@ref{8a3,,NotImplementedError}. An example of overload that gives a more
precise type than can be expressed using a union or a type variable:

@example
@@overload
def process(response: None) -> None:
    ...
@@overload
def process(response: int) -> tuple[int, str]:
    ...
@@overload
def process(response: bytes) -> str:
    ...
def process(response):
    <actual implementation>
@end example

See 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} for details and comparison with other typing semantics.
@end deffn

@geindex final() (in module typing)
@anchor{library/typing typing final}@anchor{557}
@deffn {Function} @@typing.final

A decorator to indicate to type checkers that the decorated method
cannot be overridden, and the decorated class cannot be subclassed.
For example:

@example
class Base:
    @@final
    def done(self) -> None:
        ...
class Sub(Base):
    def done(self) -> None:  # Error reported by type checker
          ...

@@final
class Leaf:
    ...
class Other(Leaf):  # Error reported by type checker
    ...
@end example

There is no runtime checking of these properties. See 
@geindex Python Enhancement Proposals; PEP 591
PEP 591@footnote{https://www.python.org/dev/peps/pep-0591} for
more details.


New in version 3.8.



Changed in version 3.11: The decorator will now set the @code{__final__} attribute to @code{True}
on the decorated object. Thus, a check like
@code{if getattr(obj, "__final__", False)} can be used at runtime
to determine whether an object @code{obj} has been marked as final.
If the decorated object does not support setting attributes,
the decorator returns the object unchanged without raising an exception.

@end deffn

@geindex no_type_check() (in module typing)
@anchor{library/typing typing no_type_check}@anchor{1091}
@deffn {Function} @@typing.no_type_check

Decorator to indicate that annotations are not type hints.

This works as class or function @ref{1bf,,decorator}.  With a class, it
applies recursively to all methods and classes defined in that class
(but not to methods defined in its superclasses or subclasses).

This mutates the function(s) in place.
@end deffn

@geindex no_type_check_decorator() (in module typing)
@anchor{library/typing typing no_type_check_decorator}@anchor{3789}
@deffn {Function} @@typing.no_type_check_decorator

Decorator to give another decorator the @ref{1091,,no_type_check()} effect.

This wraps the decorator with something that wraps the decorated
function in @ref{1091,,no_type_check()}.
@end deffn

@geindex type_check_only() (in module typing)
@anchor{library/typing typing type_check_only}@anchor{378a}
@deffn {Function} @@typing.type_check_only

Decorator to mark a class or function to be unavailable at runtime.

This decorator is itself not available at runtime. It is mainly
intended to mark classes that are defined in type stub files if
an implementation returns an instance of a private class:

@example
@@type_check_only
class Response:  # private or not available at runtime
    code: int
    def get_header(self, name: str) -> str: ...

def fetch_response() -> Response: ...
@end example

Note that returning instances of private classes is not recommended.
It is usually preferable to make such classes public.
@end deffn

@node Introspection helpers,Constant,Functions and decorators,Module contents<3>
@anchor{library/typing introspection-helpers}@anchor{378b}
@subsubsection Introspection helpers


@geindex get_type_hints() (in module typing)
@anchor{library/typing typing get_type_hints}@anchor{432}
@deffn {Function} typing.get_type_hints (obj, globalns=None, localns=None, include_extras=False)

Return a dictionary containing type hints for a function, method, module
or class object.

This is often the same as @code{obj.__annotations__}. In addition,
forward references encoded as string literals are handled by evaluating
them in @code{globals} and @code{locals} namespaces. For a class @code{C}, return
a dictionary constructed by merging all the @code{__annotations__} along
@code{C.__mro__} in reverse order.

The function recursively replaces all @code{Annotated[T, ...]} with @code{T},
unless @code{include_extras} is set to @code{True} (see @ref{431,,Annotated} for
more information). For example:

@example
class Student(NamedTuple):
    name: Annotated[str, 'some marker']

get_type_hints(Student) == @{'name': str@}
get_type_hints(Student, include_extras=False) == @{'name': str@}
get_type_hints(Student, include_extras=True) == @{
    'name': Annotated[str, 'some marker']
@}
@end example

@cartouche
@quotation Note 
@ref{432,,get_type_hints()} does not work with imported
@ref{373e,,type aliases} that include forward references.
Enabling postponed evaluation of annotations (
@geindex Python Enhancement Proposals; PEP 563
PEP 563@footnote{https://www.python.org/dev/peps/pep-0563}) may remove
the need for most forward references.
@end quotation
@end cartouche


Changed in version 3.9: Added @code{include_extras} parameter as part of 
@geindex Python Enhancement Proposals; PEP 593
PEP 593@footnote{https://www.python.org/dev/peps/pep-0593}.



Changed in version 3.11: Previously, @code{Optional[t]} was added for function and method annotations
if a default value equal to @code{None} was set.
Now the annotation is returned unchanged.

@end deffn

@geindex get_args() (in module typing)
@anchor{library/typing typing get_args}@anchor{344}
@deffn {Function} typing.get_args (tp)
@end deffn

@geindex get_origin() (in module typing)
@anchor{library/typing typing get_origin}@anchor{55c}
@deffn {Function} typing.get_origin (tp)

Provide basic introspection for generic types and special typing forms.

For a typing object of the form @code{X[Y, Z, ...]} these functions return
@code{X} and @code{(Y, Z, ...)}. If @code{X} is a generic alias for a builtin or
@ref{1c,,collections} class, it gets normalized to the original class.
If @code{X} is a union or @ref{2db,,Literal} contained in another
generic type, the order of @code{(Y, Z, ...)} may be different from the order
of the original arguments @code{[Y, Z, ...]} due to type caching.
For unsupported objects return @code{None} and @code{()} correspondingly.
Examples:

@example
assert get_origin(Dict[str, int]) is dict
assert get_args(Dict[int, str]) == (int, str)

assert get_origin(Union[int, str]) is Union
assert get_args(Union[int, str]) == (int, str)
@end example


New in version 3.8.

@end deffn

@geindex is_typeddict() (in module typing)
@anchor{library/typing typing is_typeddict}@anchor{2dd}
@deffn {Function} typing.is_typeddict (tp)

Check if a type is a @ref{2de,,TypedDict}.

For example:

@example
class Film(TypedDict):
    title: str
    year: int

is_typeddict(Film)  # => True
is_typeddict(list | str)  # => False
@end example


New in version 3.10.

@end deffn

@geindex ForwardRef (class in typing)
@anchor{library/typing typing ForwardRef}@anchor{1090}
@deffn {Class} typing.ForwardRef

A class used for internal typing representation of string forward references.
For example, @code{list["SomeClass"]} is implicitly transformed into
@code{list[ForwardRef("SomeClass")]}.  This class should not be instantiated by
a user, but may be used by introspection tools.

@cartouche
@quotation Note 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585} generic types such as @code{list["SomeClass"]} will not be
implicitly transformed into @code{list[ForwardRef("SomeClass")]} and thus
will not automatically resolve to @code{list[SomeClass]}.
@end quotation
@end cartouche


New in version 3.7.4.

@end deffn

@node Constant,,Introspection helpers,Module contents<3>
@anchor{library/typing constant}@anchor{378c}
@subsubsection Constant


@geindex TYPE_CHECKING (in module typing)
@anchor{library/typing typing TYPE_CHECKING}@anchor{83f}
@deffn {Data} typing.TYPE_CHECKING

A special constant that is assumed to be @code{True} by 3rd party static
type checkers. It is @code{False} at runtime. Usage:

@example
if TYPE_CHECKING:
    import expensive_mod

def fun(arg: 'expensive_mod.SomeType') -> None:
    local_var: expensive_mod.AnotherType = other_fun()
@end example

The first type annotation must be enclosed in quotes, making it a
“forward reference”, to hide the @code{expensive_mod} reference from the
interpreter runtime.  Type annotations for local variables are not
evaluated, so the second annotation does not need to be enclosed in quotes.

@cartouche
@quotation Note 
If @code{from __future__ import annotations} is used in Python 3.7 or later,
annotations are not evaluated at function definition time.
Instead, they are stored as strings in @code{__annotations__}.
This makes it unnecessary to use quotes around the annotation.
(see 
@geindex Python Enhancement Proposals; PEP 563
PEP 563@footnote{https://www.python.org/dev/peps/pep-0563}).
@end quotation
@end cartouche


New in version 3.5.2.

@end deffn

@node pydoc — Documentation generator and online help system,Python Development Mode,typing — Support for type hints,Development Tools
@anchor{library/pydoc doc}@anchor{378d}@anchor{library/pydoc module-pydoc}@anchor{d4}@anchor{library/pydoc pydoc-documentation-generator-and-online-help-system}@anchor{378e}
@subsection @code{pydoc} — Documentation generator and online help system


@geindex module; pydoc

@strong{Source code:} Lib/pydoc.py@footnote{https://github.com/python/cpython/tree/main/Lib/pydoc.py}

@geindex documentation; generation
@geindex documentation; online
@geindex help; online


__________________________________________________________________


The @ref{d4,,pydoc} module automatically generates documentation from Python
modules.  The documentation can be presented as pages of text on the console,
served to a web browser, or saved to HTML files.

For modules, classes, functions and methods, the displayed documentation is
derived from the docstring (i.e. the @code{__doc__} attribute) of the object,
and recursively of its documentable members.  If there is no docstring,
@ref{d4,,pydoc} tries to obtain a description from the block of comment lines just
above the definition of the class, function or method in the source file, or at
the top of the module (see @ref{378f,,inspect.getcomments()}).

The built-in function @ref{3bc,,help()} invokes the online help system in the
interactive interpreter, which uses @ref{d4,,pydoc} to generate its documentation
as text on the console.  The same text documentation can also be viewed from
outside the Python interpreter by running @code{pydoc} as a script at the
operating system’s command prompt. For example, running

@example
pydoc sys
@end example

at a shell prompt will display documentation on the @ref{f7,,sys} module, in a
style similar to the manual pages shown by the Unix @code{man} command.  The
argument to @code{pydoc} can be the name of a function, module, or package,
or a dotted reference to a class, method, or function within a module or module
in a package.  If the argument to @code{pydoc} looks like a path (that is,
it contains the path separator for your operating system, such as a slash in
Unix), and refers to an existing Python source file, then documentation is
produced for that file.

@cartouche
@quotation Note 
In order to find objects and their documentation, @ref{d4,,pydoc} imports the
module(s) to be documented.  Therefore, any code on module level will be
executed on that occasion.  Use an @code{if __name__ == '__main__':} guard to
only execute code when a file is invoked as a script and not just imported.
@end quotation
@end cartouche

When printing output to the console, @code{pydoc} attempts to paginate the
output for easier reading.  If the 
@geindex PAGER
@geindex environment variable; PAGER
@code{PAGER} environment variable is set,
@code{pydoc} will use its value as a pagination program.

Specifying a @code{-w} flag before the argument will cause HTML documentation
to be written out to a file in the current directory, instead of displaying text
on the console.

Specifying a @code{-k} flag before the argument will search the synopsis
lines of all available modules for the keyword given as the argument, again in a
manner similar to the Unix @code{man} command.  The synopsis line of a
module is the first line of its documentation string.

You can also use @code{pydoc} to start an HTTP server on the local machine
that will serve documentation to visiting web browsers.  @code{pydoc -p 1234}
will start a HTTP server on port 1234, allowing you to browse the
documentation at @code{http://localhost:1234/} in your preferred web browser.
Specifying @code{0} as the port number will select an arbitrary unused port.

@code{pydoc -n <hostname>} will start the server listening at the given
hostname.  By default the hostname is ‘localhost’ but if you want the server to
be reached from other machines, you may want to change the host name that the
server responds to.  During development this is especially useful if you want
to run pydoc from within a container.

@code{pydoc -b} will start the server and additionally open a web
browser to a module index page.  Each served page has a navigation bar at the
top where you can @emph{Get} help on an individual item, @emph{Search} all modules with a
keyword in their synopsis line, and go to the @emph{Module index}, @emph{Topics} and
@emph{Keywords} pages.

When @code{pydoc} generates documentation, it uses the current environment
and path to locate modules.  Thus, invoking @code{pydoc spam}
documents precisely the version of the module you would get if you started the
Python interpreter and typed @code{import spam}.

Module docs for core modules are assumed to reside in
@code{https://docs.python.org/X.Y/library/} where @code{X} and @code{Y} are the
major and minor version numbers of the Python interpreter.  This can
be overridden by setting the 
@geindex PYTHONDOCS
@geindex environment variable; PYTHONDOCS
@code{PYTHONDOCS} environment variable
to a different URL or to a local directory containing the Library
Reference Manual pages.


Changed in version 3.2: Added the @code{-b} option.



Changed in version 3.3: The @code{-g} command line option was removed.



Changed in version 3.4: @ref{d4,,pydoc} now uses @ref{1c1,,inspect.signature()} rather than
@ref{1c2,,inspect.getfullargspec()} to extract signature information from
callables.



Changed in version 3.7: Added the @code{-n} option.


@node Python Development Mode,Effects of the Python Development Mode,pydoc — Documentation generator and online help system,Development Tools
@anchor{library/devmode doc}@anchor{3790}@anchor{library/devmode devmode}@anchor{3ac}@anchor{library/devmode python-development-mode}@anchor{3791}
@subsection Python Development Mode



New in version 3.7.


The Python Development Mode introduces additional runtime checks that are too
expensive to be enabled by default. It should not be more verbose than the
default if the code is correct; new warnings are only emitted when an issue is
detected.

It can be enabled using the @ref{159,,-X dev} command line option or by
setting the 
@geindex PYTHONDEVMODE
@geindex environment variable; PYTHONDEVMODE
@ref{60a,,PYTHONDEVMODE} environment variable to @code{1}.

See also @ref{338,,Python debug build}.

@node Effects of the Python Development Mode,ResourceWarning Example,Python Development Mode,Development Tools
@anchor{library/devmode effects-of-the-python-development-mode}@anchor{3792}
@subsection Effects of the Python Development Mode


Enabling the Python Development Mode is similar to the following command, but
with additional effects described below:

@example
PYTHONMALLOC=debug PYTHONASYNCIODEBUG=1 python3 -W default -X faulthandler
@end example

Effects of the Python Development Mode:


@itemize *

@item 
Add @code{default} @ref{1614,,warning filter}. The
following warnings are shown:


@itemize *

@item 
@ref{1b2,,DeprecationWarning}

@item 
@ref{304,,ImportWarning}

@item 
@ref{3a9,,PendingDeprecationWarning}

@item 
@ref{77b,,ResourceWarning}
@end itemize

Normally, the above warnings are filtered by the default @ref{1614,,warning filters}.

It behaves as if the @ref{3a8,,-W default} command line option is used.

Use the @ref{3a8,,-W error} command line option or set the
@geindex PYTHONWARNINGS
@geindex environment variable; PYTHONWARNINGS
@ref{6de,,PYTHONWARNINGS} environment variable to @code{error} to treat warnings
as errors.

@item 
Install debug hooks on memory allocators to check for:


@itemize *

@item 
Buffer underflow

@item 
Buffer overflow

@item 
Memory allocator API violation

@item 
Unsafe usage of the GIL
@end itemize

See the @ref{7b0,,PyMem_SetupDebugHooks()} C function.

It behaves as if the 
@geindex PYTHONMALLOC
@geindex environment variable; PYTHONMALLOC
@ref{7ab,,PYTHONMALLOC} environment variable is set to
@code{debug}.

To enable the Python Development Mode without installing debug hooks on
memory allocators, set the 
@geindex PYTHONMALLOC
@geindex environment variable; PYTHONMALLOC
@ref{7ab,,PYTHONMALLOC} environment variable to
@code{default}.

@item 
Call @ref{7ea,,faulthandler.enable()} at Python startup to install handlers for
the @code{SIGSEGV}, @code{SIGFPE}, @code{SIGABRT}, @code{SIGBUS} and
@code{SIGILL} signals to dump the Python traceback on a crash.

It behaves as if the @ref{159,,-X faulthandler} command line option is
used or if the 
@geindex PYTHONFAULTHANDLER
@geindex environment variable; PYTHONFAULTHANDLER
@ref{c18,,PYTHONFAULTHANDLER} environment variable is set to
@code{1}.

@item 
Enable @ref{1625,,asyncio debug mode}. For example,
@ref{9,,asyncio} checks for coroutines that were not awaited and logs them.

It behaves as if the 
@geindex PYTHONASYNCIODEBUG
@geindex environment variable; PYTHONASYNCIODEBUG
@ref{1624,,PYTHONASYNCIODEBUG} environment variable is set
to @code{1}.

@item 
Check the @emph{encoding} and @emph{errors} arguments for string encoding and decoding
operations. Examples: @ref{1c8,,open()}, @ref{3ba,,str.encode()} and
@ref{3bb,,bytes.decode()}.

By default, for best performance, the @emph{errors} argument is only checked at
the first encoding/decoding error and the @emph{encoding} argument is sometimes
ignored for empty strings.

@item 
The @ref{503,,io.IOBase} destructor logs @code{close()} exceptions.

@item 
Set the @code{dev_mode} attribute of @ref{22f,,sys.flags} to
@code{True}.
@end itemize

The Python Development Mode does not enable the @ref{11b,,tracemalloc} module by
default, because the overhead cost (to performance and memory) would be too
large. Enabling the @ref{11b,,tracemalloc} module provides additional information
on the origin of some errors. For example, @ref{77b,,ResourceWarning} logs the
traceback where the resource was allocated, and a buffer overflow error logs
the traceback where the memory block was allocated.

The Python Development Mode does not prevent the @ref{912,,-O} command line
option from removing @ref{46d,,assert} statements nor from setting
@ref{257,,__debug__} to @code{False}.

The Python Development Mode can only be enabled at the Python startup. Its
value can be read from @ref{22f,,sys.flags.dev_mode}.


Changed in version 3.8: The @ref{503,,io.IOBase} destructor now logs @code{close()} exceptions.



Changed in version 3.9: The @emph{encoding} and @emph{errors} arguments are now checked for string encoding
and decoding operations.


@node ResourceWarning Example,Bad file descriptor error example,Effects of the Python Development Mode,Development Tools
@anchor{library/devmode resourcewarning-example}@anchor{3793}
@subsection ResourceWarning Example


Example of a script counting the number of lines of the text file specified in
the command line:

@example
import sys

def main():
    fp = open(sys.argv[1])
    nlines = len(fp.readlines())
    print(nlines)
    # The file is closed implicitly

if __name__ == "__main__":
    main()
@end example

The script does not close the file explicitly. By default, Python does not emit
any warning. Example using README.txt, which has 269 lines:

@example
$ python3 script.py README.txt
269
@end example

Enabling the Python Development Mode displays a @ref{77b,,ResourceWarning} warning:

@example
$ python3 -X dev script.py README.txt
269
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'>
  main()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
@end example

In addition, enabling @ref{11b,,tracemalloc} shows the line where the file was
opened:

@example
$ python3 -X dev -X tracemalloc=5 script.py README.rst
269
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'>
  main()
Object allocated at (most recent call last):
  File "script.py", lineno 10
    main()
  File "script.py", lineno 4
    fp = open(sys.argv[1])
@end example

The fix is to close explicitly the file. Example using a context manager:

@example
def main():
    # Close the file explicitly when exiting the with block
    with open(sys.argv[1]) as fp:
        nlines = len(fp.readlines())
    print(nlines)
@end example

Not closing a resource explicitly can leave a resource open for way longer than
expected; it can cause severe issues upon exiting Python. It is bad in
CPython, but it is even worse in PyPy. Closing resources explicitly makes an
application more deterministic and more reliable.

@node Bad file descriptor error example,doctest — Test interactive Python examples,ResourceWarning Example,Development Tools
@anchor{library/devmode bad-file-descriptor-error-example}@anchor{3794}
@subsection Bad file descriptor error example


Script displaying the first line of itself:

@example
import os

def main():
    fp = open(__file__)
    firstline = fp.readline()
    print(firstline.rstrip())
    os.close(fp.fileno())
    # The file is closed implicitly

main()
@end example

By default, Python does not emit any warning:

@example
$ python3 script.py
import os
@end example

The Python Development Mode shows a @ref{77b,,ResourceWarning} and logs a “Bad file
descriptor” error when finalizing the file object:

@example
$ python3 script.py
import os
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'>
  main()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Exception ignored in: <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'>
Traceback (most recent call last):
  File "script.py", line 10, in <module>
    main()
OSError: [Errno 9] Bad file descriptor
@end example

@code{os.close(fp.fileno())} closes the file descriptor. When the file object
finalizer tries to close the file descriptor again, it fails with the @code{Bad
file descriptor} error. A file descriptor must be closed only once. In the
worst case scenario, closing it twice can lead to a crash (see bpo-18748@footnote{https://bugs.python.org/issue18748}
for an example).

The fix is to remove the @code{os.close(fp.fileno())} line, or open the file with
@code{closefd=False}.

@node doctest — Test interactive Python examples,unittest — Unit testing framework,Bad file descriptor error example,Development Tools
@anchor{library/doctest doc}@anchor{3795}@anchor{library/doctest doctest-test-interactive-python-examples}@anchor{3796}@anchor{library/doctest module-doctest}@anchor{63}
@subsection @code{doctest} — Test interactive Python examples


@geindex module; doctest

@strong{Source code:} Lib/doctest.py@footnote{https://github.com/python/cpython/tree/main/Lib/doctest.py}


__________________________________________________________________


The @ref{63,,doctest} module searches for pieces of text that look like interactive
Python sessions, and then executes those sessions to verify that they work
exactly as shown.  There are several common ways to use doctest:


@itemize *

@item 
To check that a module’s docstrings are up-to-date by verifying that all
interactive examples still work as documented.

@item 
To perform regression testing by verifying that interactive examples from a
test file or a test object work as expected.

@item 
To write tutorial documentation for a package, liberally illustrated with
input-output examples.  Depending on whether the examples or the expository text
are emphasized, this has the flavor of “literate testing” or “executable
documentation”.
@end itemize

Here’s a complete but small example module:

@example
"""
This is the "example" module.

The example module supplies one function, factorial().  For example,

>>> factorial(5)
120
"""

def factorial(n):
    """Return the factorial of n, an exact integer >= 0.

    >>> [factorial(n) for n in range(6)]
    [1, 1, 2, 6, 24, 120]
    >>> factorial(30)
    265252859812191058636308480000000
    >>> factorial(-1)
    Traceback (most recent call last):
        ...
    ValueError: n must be >= 0

    Factorials of floats are OK, but the float must be an exact integer:
    >>> factorial(30.1)
    Traceback (most recent call last):
        ...
    ValueError: n must be exact integer
    >>> factorial(30.0)
    265252859812191058636308480000000

    It must also not be ridiculously large:
    >>> factorial(1e100)
    Traceback (most recent call last):
        ...
    OverflowError: n too large
    """

    import math
    if not n >= 0:
        raise ValueError("n must be >= 0")
    if math.floor(n) != n:
        raise ValueError("n must be exact integer")
    if n+1 == n:  # catch a value like 1e300
        raise OverflowError("n too large")
    result = 1
    factor = 2
    while factor <= n:
        result *= factor
        factor += 1
    return result


if __name__ == "__main__":
    import doctest
    doctest.testmod()
@end example

If you run @code{example.py} directly from the command line, @ref{63,,doctest}
works its magic:

@example
$ python example.py
$
@end example

There’s no output!  That’s normal, and it means all the examples worked.  Pass
@code{-v} to the script, and @ref{63,,doctest} prints a detailed log of what
it’s trying, and prints a summary at the end:

@example
$ python example.py -v
Trying:
    factorial(5)
Expecting:
    120
ok
Trying:
    [factorial(n) for n in range(6)]
Expecting:
    [1, 1, 2, 6, 24, 120]
ok
@end example

And so on, eventually ending with:

@example
Trying:
    factorial(1e100)
Expecting:
    Traceback (most recent call last):
        ...
    OverflowError: n too large
ok
2 items passed all tests:
   1 tests in __main__
   8 tests in __main__.factorial
9 tests in 2 items.
9 passed and 0 failed.
Test passed.
$
@end example

That’s all you need to know to start making productive use of @ref{63,,doctest}!
Jump in.  The following sections provide full details.  Note that there are many
examples of doctests in the standard Python test suite and libraries.
Especially useful examples can be found in the standard test file
@code{Lib/test/test_doctest.py}.

@menu
* Simple Usage; Checking Examples in Docstrings: Simple Usage Checking Examples in Docstrings. 
* Simple Usage; Checking Examples in a Text File: Simple Usage Checking Examples in a Text File. 
* How It Works:: 
* Basic API:: 
* Unittest API:: 
* Advanced API:: 
* Debugging:: 
* Soapbox:: 

@end menu

@node Simple Usage Checking Examples in Docstrings,Simple Usage Checking Examples in a Text File,,doctest — Test interactive Python examples
@anchor{library/doctest doctest-simple-testmod}@anchor{3797}@anchor{library/doctest simple-usage-checking-examples-in-docstrings}@anchor{3798}
@subsubsection Simple Usage: Checking Examples in Docstrings


The simplest way to start using doctest (but not necessarily the way you’ll
continue to do it) is to end each module @code{M} with:

@example
if __name__ == "__main__":
    import doctest
    doctest.testmod()
@end example

@ref{63,,doctest} then examines docstrings in module @code{M}.

Running the module as a script causes the examples in the docstrings to get
executed and verified:

@example
python M.py
@end example

This won’t display anything unless an example fails, in which case the failing
example(s) and the cause(s) of the failure(s) are printed to stdout, and the
final line of output is @code{***Test Failed*** N failures.}, where @emph{N} is the
number of examples that failed.

Run it with the @code{-v} switch instead:

@example
python M.py -v
@end example

and a detailed report of all examples tried is printed to standard output, along
with assorted summaries at the end.

You can force verbose mode by passing @code{verbose=True} to @ref{fc9,,testmod()}, or
prohibit it by passing @code{verbose=False}.  In either of those cases,
@code{sys.argv} is not examined by @ref{fc9,,testmod()} (so passing @code{-v} or not
has no effect).

There is also a command line shortcut for running @ref{fc9,,testmod()}.  You can
instruct the Python interpreter to run the doctest module directly from the
standard library and pass the module name(s) on the command line:

@example
python -m doctest -v example.py
@end example

This will import @code{example.py} as a standalone module and run
@ref{fc9,,testmod()} on it.  Note that this may not work correctly if the file is
part of a package and imports other submodules from that package.

For more information on @ref{fc9,,testmod()}, see section @ref{3799,,Basic API}.

@node Simple Usage Checking Examples in a Text File,How It Works,Simple Usage Checking Examples in Docstrings,doctest — Test interactive Python examples
@anchor{library/doctest doctest-simple-testfile}@anchor{379a}@anchor{library/doctest simple-usage-checking-examples-in-a-text-file}@anchor{379b}
@subsubsection Simple Usage: Checking Examples in a Text File


Another simple application of doctest is testing interactive examples in a text
file.  This can be done with the @ref{379c,,testfile()} function:

@example
import doctest
doctest.testfile("example.txt")
@end example

That short script executes and verifies any interactive Python examples
contained in the file @code{example.txt}.  The file content is treated as if it
were a single giant docstring; the file doesn’t need to contain a Python
program!   For example, perhaps @code{example.txt} contains this:

@example
The `@w{`}example`@w{`} module
======================

Using `@w{`}factorial`@w{`}
-------------------

This is an example text file in reStructuredText format.  First import
`@w{`}factorial`@w{`} from the `@w{`}example`@w{`} module:

    >>> from example import factorial

Now use it:

    >>> factorial(6)
    120
@end example

Running @code{doctest.testfile("example.txt")} then finds the error in this
documentation:

@example
File "./example.txt", line 14, in example.txt
Failed example:
    factorial(6)
Expected:
    120
Got:
    720
@end example

As with @ref{fc9,,testmod()}, @ref{379c,,testfile()} won’t display anything unless an
example fails.  If an example does fail, then the failing example(s) and the
cause(s) of the failure(s) are printed to stdout, using the same format as
@ref{fc9,,testmod()}.

By default, @ref{379c,,testfile()} looks for files in the calling module’s directory.
See section @ref{3799,,Basic API} for a description of the optional arguments
that can be used to tell it to look for files in other locations.

Like @ref{fc9,,testmod()}, @ref{379c,,testfile()}’s verbosity can be set with the
@code{-v} command-line switch or with the optional keyword argument
@emph{verbose}.

There is also a command line shortcut for running @ref{379c,,testfile()}.  You can
instruct the Python interpreter to run the doctest module directly from the
standard library and pass the file name(s) on the command line:

@example
python -m doctest -v example.txt
@end example

Because the file name does not end with @code{.py}, @ref{63,,doctest} infers that
it must be run with @ref{379c,,testfile()}, not @ref{fc9,,testmod()}.

For more information on @ref{379c,,testfile()}, see section @ref{3799,,Basic API}.

@node How It Works,Basic API,Simple Usage Checking Examples in a Text File,doctest — Test interactive Python examples
@anchor{library/doctest doctest-how-it-works}@anchor{379d}@anchor{library/doctest how-it-works}@anchor{379e}
@subsubsection How It Works


This section examines in detail how doctest works: which docstrings it looks at,
how it finds interactive examples, what execution context it uses, how it
handles exceptions, and how option flags can be used to control its behavior.
This is the information that you need to know to write doctest examples; for
information about actually running doctest on these examples, see the following
sections.

@menu
* Which Docstrings Are Examined?:: 
* How are Docstring Examples Recognized?:: 
* What’s the Execution Context?:: 
* What About Exceptions?:: 
* Option Flags:: 
* Directives:: 
* Warnings: Warnings<2>. 

@end menu

@node Which Docstrings Are Examined?,How are Docstring Examples Recognized?,,How It Works
@anchor{library/doctest doctest-which-docstrings}@anchor{379f}@anchor{library/doctest which-docstrings-are-examined}@anchor{37a0}
@subsubsection Which Docstrings Are Examined?


The module docstring, and all function, class and method docstrings are
searched.  Objects imported into the module are not searched.

In addition, if @code{M.__test__} exists and “is true”, it must be a dict, and each
entry maps a (string) name to a function object, class object, or string.
Function and class object docstrings found from @code{M.__test__} are searched, and
strings are treated as if they were docstrings.  In output, a key @code{K} in
@code{M.__test__} appears with name

@example
<name of M>.__test__.K
@end example

Any classes found are recursively searched similarly, to test docstrings in
their contained methods and nested classes.

@strong{CPython implementation detail:} Prior to version 3.4, extension modules written in C were not fully
searched by doctest.

@node How are Docstring Examples Recognized?,What’s the Execution Context?,Which Docstrings Are Examined?,How It Works
@anchor{library/doctest doctest-finding-examples}@anchor{37a1}@anchor{library/doctest how-are-docstring-examples-recognized}@anchor{37a2}
@subsubsection How are Docstring Examples Recognized?


In most cases a copy-and-paste of an interactive console session works fine,
but doctest isn’t trying to do an exact emulation of any specific Python shell.

@example
>>> # comments are ignored
>>> x = 12
>>> x
12
>>> if x == 13:
...     print("yes")
... else:
...     print("no")
...     print("NO")
...     print("NO!!!")
...
no
NO
NO!!!
>>>
@end example

@geindex >>>; interpreter prompt
@geindex ...; interpreter prompt

Any expected output must immediately follow the final @code{'>>> '} or @code{'... '}
line containing the code, and the expected output (if any) extends to the next
@code{'>>> '} or all-whitespace line.

The fine print:


@itemize *

@item 
Expected output cannot contain an all-whitespace line, since such a line is
taken to signal the end of expected output.  If expected output does contain a
blank line, put @code{<BLANKLINE>} in your doctest example each place a blank line
is expected.

@item 
All hard tab characters are expanded to spaces, using 8-column tab stops.
Tabs in output generated by the tested code are not modified.  Because any
hard tabs in the sample output @emph{are} expanded, this means that if the code
output includes hard tabs, the only way the doctest can pass is if the
@ref{37a3,,NORMALIZE_WHITESPACE} option or @ref{37a4,,directive}
is in effect.
Alternatively, the test can be rewritten to capture the output and compare it
to an expected value as part of the test.  This handling of tabs in the
source was arrived at through trial and error, and has proven to be the least
error prone way of handling them.  It is possible to use a different
algorithm for handling tabs by writing a custom @ref{37a5,,DocTestParser} class.

@item 
Output to stdout is captured, but not output to stderr (exception tracebacks
are captured via a different means).

@item 
If you continue a line via backslashing in an interactive session, or for any
other reason use a backslash, you should use a raw docstring, which will
preserve your backslashes exactly as you type them:

@example
>>> def f(x):
...     r'@w{'}'Backslashes in a raw docstring: m\n'@w{'}'
>>> print(f.__doc__)
Backslashes in a raw docstring: m\n
@end example

Otherwise, the backslash will be interpreted as part of the string. For example,
the @code{\n} above would be interpreted as a newline character.  Alternatively, you
can double each backslash in the doctest version (and not use a raw string):

@example
>>> def f(x):
...     '@w{'}'Backslashes in a raw docstring: m\\n'@w{'}'
>>> print(f.__doc__)
Backslashes in a raw docstring: m\n
@end example

@item 
The starting column doesn’t matter:

@example
>>> assert "Easy!"
      >>> import math
          >>> math.floor(1.9)
          1
@end example

and as many leading whitespace characters are stripped from the expected output
as appeared in the initial @code{'>>> '} line that started the example.
@end itemize

@node What’s the Execution Context?,What About Exceptions?,How are Docstring Examples Recognized?,How It Works
@anchor{library/doctest doctest-execution-context}@anchor{37a6}@anchor{library/doctest what-s-the-execution-context}@anchor{37a7}
@subsubsection What’s the Execution Context?


By default, each time @ref{63,,doctest} finds a docstring to test, it uses a
@emph{shallow copy} of @code{M}’s globals, so that running tests doesn’t change the
module’s real globals, and so that one test in @code{M} can’t leave behind
crumbs that accidentally allow another test to work.  This means examples can
freely use any names defined at top-level in @code{M}, and names defined earlier
in the docstring being run. Examples cannot see names defined in other
docstrings.

You can force use of your own dict as the execution context by passing
@code{globs=your_dict} to @ref{fc9,,testmod()} or @ref{379c,,testfile()} instead.

@node What About Exceptions?,Option Flags,What’s the Execution Context?,How It Works
@anchor{library/doctest doctest-exceptions}@anchor{37a8}@anchor{library/doctest what-about-exceptions}@anchor{37a9}
@subsubsection What About Exceptions?


No problem, provided that the traceback is the only output produced by the
example:  just paste in the traceback. @footnote{@w{(1)} 
Examples containing both expected output and an exception are not supported.
Trying to guess where one ends and the other begins is too error-prone, and that
also makes for a confusing test.
} Since tracebacks contain details
that are likely to change rapidly (for example, exact file paths and line
numbers), this is one case where doctest works hard to be flexible in what it
accepts.

Simple example:

@example
>>> [1, 2, 3].remove(42)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: list.remove(x): x not in list
@end example

That doctest succeeds if @ref{17c,,ValueError} is raised, with the @code{list.remove(x):
x not in list} detail as shown.

The expected output for an exception must start with a traceback header, which
may be either of the following two lines, indented the same as the first line of
the example:

@example
Traceback (most recent call last):
Traceback (innermost last):
@end example

The traceback header is followed by an optional traceback stack, whose contents
are ignored by doctest.  The traceback stack is typically omitted, or copied
verbatim from an interactive session.

The traceback stack is followed by the most interesting part: the line(s)
containing the exception type and detail.  This is usually the last line of a
traceback, but can extend across multiple lines if the exception has a
multi-line detail:

@example
>>> raise ValueError('multi\n    line\ndetail')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: multi
    line
detail
@end example

The last three lines (starting with @ref{17c,,ValueError}) are compared against the
exception’s type and detail, and the rest are ignored.

Best practice is to omit the traceback stack, unless it adds significant
documentation value to the example.  So the last example is probably better as:

@example
>>> raise ValueError('multi\n    line\ndetail')
Traceback (most recent call last):
    ...
ValueError: multi
    line
detail
@end example

Note that tracebacks are treated very specially.  In particular, in the
rewritten example, the use of @code{...} is independent of doctest’s
@ref{fca,,ELLIPSIS} option.  The ellipsis in that example could be left out, or
could just as well be three (or three hundred) commas or digits, or an indented
transcript of a Monty Python skit.

Some details you should read once, but won’t need to remember:


@itemize *

@item 
Doctest can’t guess whether your expected output came from an exception
traceback or from ordinary printing.  So, e.g., an example that expects
@code{ValueError: 42 is prime} will pass whether @ref{17c,,ValueError} is actually
raised or if the example merely prints that traceback text.  In practice,
ordinary output rarely begins with a traceback header line, so this doesn’t
create real problems.

@item 
Each line of the traceback stack (if present) must be indented further than
the first line of the example, @emph{or} start with a non-alphanumeric character.
The first line following the traceback header indented the same and starting
with an alphanumeric is taken to be the start of the exception detail.  Of
course this does the right thing for genuine tracebacks.

@item 
When the @ref{37aa,,IGNORE_EXCEPTION_DETAIL} doctest option is specified,
everything following the leftmost colon and any module information in the
exception name is ignored.

@item 
The interactive shell omits the traceback header line for some
@ref{21a,,SyntaxError}s.  But doctest uses the traceback header line to
distinguish exceptions from non-exceptions.  So in the rare case where you need
to test a @ref{21a,,SyntaxError} that omits the traceback header, you will need to
manually add the traceback header line to your test example.
@end itemize

@geindex ^ (caret); marker


@itemize *

@item 
For some exceptions, Python displays the position of the error using @code{^}
markers and tildes:

@example
>>> 1 + None
  File "<stdin>", line 1
    1 + None
    ~~^~~~~~
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
@end example

Since the lines showing the position of the error come before the exception type
and detail, they are not checked by doctest.  For example, the following test
would pass, even though it puts the @code{^} marker in the wrong location:

@example
>>> 1 + None
  File "<stdin>", line 1
    1 + None
    ^~~~~~~~
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
@end example
@end itemize
@anchor{library/doctest option-flags-and-directives}@anchor{37ab}
@node Option Flags,Directives,What About Exceptions?,How It Works
@anchor{library/doctest doctest-options}@anchor{ab0}@anchor{library/doctest option-flags}@anchor{37ac}
@subsubsection Option Flags


A number of option flags control various aspects of doctest’s behavior.
Symbolic names for the flags are supplied as module constants, which can be
@ref{18f9,,bitwise ORed} together and passed to various functions.
The names can also be used in @ref{37a4,,doctest directives},
and may be passed to the doctest command line interface via the @code{-o} option.


New in version 3.4: The @code{-o} command line option.


The first group of options define test semantics, controlling aspects of how
doctest decides whether actual output matches an example’s expected output:

@geindex DONT_ACCEPT_TRUE_FOR_1 (in module doctest)
@anchor{library/doctest doctest DONT_ACCEPT_TRUE_FOR_1}@anchor{37ad}
@deffn {Data} doctest.DONT_ACCEPT_TRUE_FOR_1

By default, if an expected output block contains just @code{1}, an actual output
block containing just @code{1} or just @code{True} is considered to be a match, and
similarly for @code{0} versus @code{False}.  When @ref{37ad,,DONT_ACCEPT_TRUE_FOR_1} is
specified, neither substitution is allowed.  The default behavior caters to that
Python changed the return type of many functions from integer to boolean;
doctests expecting “little integer” output still work in these cases.  This
option will probably go away, but not for several years.
@end deffn

@geindex <BLANKLINE>

@geindex DONT_ACCEPT_BLANKLINE (in module doctest)
@anchor{library/doctest doctest DONT_ACCEPT_BLANKLINE}@anchor{37ae}
@deffn {Data} doctest.DONT_ACCEPT_BLANKLINE

By default, if an expected output block contains a line containing only the
string @code{<BLANKLINE>}, then that line will match a blank line in the actual
output.  Because a genuinely blank line delimits the expected output, this is
the only way to communicate that a blank line is expected.  When
@ref{37ae,,DONT_ACCEPT_BLANKLINE} is specified, this substitution is not allowed.
@end deffn

@geindex NORMALIZE_WHITESPACE (in module doctest)
@anchor{library/doctest doctest NORMALIZE_WHITESPACE}@anchor{37a3}
@deffn {Data} doctest.NORMALIZE_WHITESPACE

When specified, all sequences of whitespace (blanks and newlines) are treated as
equal.  Any sequence of whitespace within the expected output will match any
sequence of whitespace within the actual output. By default, whitespace must
match exactly. @ref{37a3,,NORMALIZE_WHITESPACE} is especially useful when a line of
expected output is very long, and you want to wrap it across multiple lines in
your source.
@end deffn

@geindex ...; in doctests

@geindex ELLIPSIS (in module doctest)
@anchor{library/doctest doctest ELLIPSIS}@anchor{fca}
@deffn {Data} doctest.ELLIPSIS

When specified, an ellipsis marker (@code{...}) in the expected output can match
any substring in the actual output.  This includes substrings that span line
boundaries, and empty substrings, so it’s best to keep usage of this simple.
Complicated uses can lead to the same kinds of “oops, it matched too much!”
surprises that @code{.*} is prone to in regular expressions.
@end deffn

@geindex IGNORE_EXCEPTION_DETAIL (in module doctest)
@anchor{library/doctest doctest IGNORE_EXCEPTION_DETAIL}@anchor{37aa}
@deffn {Data} doctest.IGNORE_EXCEPTION_DETAIL

When specified, an example that expects an exception passes if an exception of
the expected type is raised, even if the exception detail does not match.  For
example, an example expecting @code{ValueError: 42} will pass if the actual
exception raised is @code{ValueError: 3*14}, but will fail, e.g., if
@ref{162,,TypeError} is raised.

It will also ignore the module name used in Python 3 doctest reports. Hence
both of these variations will work with the flag specified, regardless of
whether the test is run under Python 2.7 or Python 3.2 (or later versions):

@example
>>> raise CustomError('message')
Traceback (most recent call last):
CustomError: message

>>> raise CustomError('message')
Traceback (most recent call last):
my_module.CustomError: message
@end example

Note that @ref{fca,,ELLIPSIS} can also be used to ignore the
details of the exception message, but such a test may still fail based
on whether or not the module details are printed as part of the
exception name. Using @ref{37aa,,IGNORE_EXCEPTION_DETAIL} and the details
from Python 2.3 is also the only clear way to write a doctest that doesn’t
care about the exception detail yet continues to pass under Python 2.3 or
earlier (those releases do not support @ref{37a4,,doctest directives} and ignore them as irrelevant comments). For example:

@example
>>> (1, 2)[3] = 'moo'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object doesn't support item assignment
@end example

passes under Python 2.3 and later Python versions with the flag specified,
even though the detail
changed in Python 2.4 to say “does not” instead of “doesn’t”.


Changed in version 3.2: @ref{37aa,,IGNORE_EXCEPTION_DETAIL} now also ignores any information relating
to the module containing the exception under test.

@end deffn

@geindex SKIP (in module doctest)
@anchor{library/doctest doctest SKIP}@anchor{37af}
@deffn {Data} doctest.SKIP

When specified, do not run the example at all.  This can be useful in contexts
where doctest examples serve as both documentation and test cases, and an
example should be included for documentation purposes, but should not be
checked.  E.g., the example’s output might be random; or the example might
depend on resources which would be unavailable to the test driver.

The SKIP flag can also be used for temporarily “commenting out” examples.
@end deffn

@geindex COMPARISON_FLAGS (in module doctest)
@anchor{library/doctest doctest COMPARISON_FLAGS}@anchor{37b0}
@deffn {Data} doctest.COMPARISON_FLAGS

A bitmask or’ing together all the comparison flags above.
@end deffn

The second group of options controls how test failures are reported:

@geindex REPORT_UDIFF (in module doctest)
@anchor{library/doctest doctest REPORT_UDIFF}@anchor{fcb}
@deffn {Data} doctest.REPORT_UDIFF

When specified, failures that involve multi-line expected and actual outputs are
displayed using a unified diff.
@end deffn

@geindex REPORT_CDIFF (in module doctest)
@anchor{library/doctest doctest REPORT_CDIFF}@anchor{fcc}
@deffn {Data} doctest.REPORT_CDIFF

When specified, failures that involve multi-line expected and actual outputs
will be displayed using a context diff.
@end deffn

@geindex REPORT_NDIFF (in module doctest)
@anchor{library/doctest doctest REPORT_NDIFF}@anchor{fcd}
@deffn {Data} doctest.REPORT_NDIFF

When specified, differences are computed by @code{difflib.Differ}, using the same
algorithm as the popular @code{ndiff.py} utility. This is the only method that
marks differences within lines as well as across lines.  For example, if a line
of expected output contains digit @code{1} where actual output contains letter
@code{l}, a line is inserted with a caret marking the mismatching column positions.
@end deffn

@geindex REPORT_ONLY_FIRST_FAILURE (in module doctest)
@anchor{library/doctest doctest REPORT_ONLY_FIRST_FAILURE}@anchor{37b1}
@deffn {Data} doctest.REPORT_ONLY_FIRST_FAILURE

When specified, display the first failing example in each doctest, but suppress
output for all remaining examples.  This will prevent doctest from reporting
correct examples that break because of earlier failures; but it might also hide
incorrect examples that fail independently of the first failure.  When
@ref{37b1,,REPORT_ONLY_FIRST_FAILURE} is specified, the remaining examples are
still run, and still count towards the total number of failures reported; only
the output is suppressed.
@end deffn

@geindex FAIL_FAST (in module doctest)
@anchor{library/doctest doctest FAIL_FAST}@anchor{ab1}
@deffn {Data} doctest.FAIL_FAST

When specified, exit after the first failing example and don’t attempt to run
the remaining examples. Thus, the number of failures reported will be at most
1.  This flag may be useful during debugging, since examples after the first
failure won’t even produce debugging output.

The doctest command line accepts the option @code{-f} as a shorthand for @code{-o
FAIL_FAST}.


New in version 3.4.

@end deffn

@geindex REPORTING_FLAGS (in module doctest)
@anchor{library/doctest doctest REPORTING_FLAGS}@anchor{37b2}
@deffn {Data} doctest.REPORTING_FLAGS

A bitmask or’ing together all the reporting flags above.
@end deffn

There is also a way to register new option flag names, though this isn’t
useful unless you intend to extend @ref{63,,doctest} internals via subclassing:

@geindex register_optionflag() (in module doctest)
@anchor{library/doctest doctest register_optionflag}@anchor{37b3}
@deffn {Function} doctest.register_optionflag (name)

Create a new option flag with a given name, and return the new flag’s integer
value.  @ref{37b3,,register_optionflag()} can be used when subclassing
@ref{37b4,,OutputChecker} or @ref{37b5,,DocTestRunner} to create new options that are
supported by your subclasses.  @ref{37b3,,register_optionflag()} should always be
called using the following idiom:

@example
MY_FLAG = register_optionflag('MY_FLAG')
@end example
@end deffn

@geindex # (hash); in doctests
@geindex + (plus); in doctests
@geindex - (minus); in doctests

@node Directives,Warnings<2>,Option Flags,How It Works
@anchor{library/doctest directives}@anchor{37b6}@anchor{library/doctest doctest-directives}@anchor{37a4}
@subsubsection Directives


Doctest directives may be used to modify the @ref{ab0,,option flags} for an individual example.  Doctest directives are
special Python comments following an example’s source code:

@example
@anchor{library/doctest grammar-token-doctest-directive}@anchor{37b7}@anchor{library/doctest grammar-token-directive}@anchor{37b8}directive             ::= "#" "doctest:" directive_options
@anchor{library/doctest grammar-token-doctest-directive_options}@anchor{37b9}@anchor{library/doctest grammar-token-directive-options}@anchor{37ba}directive_options     ::= directive_option ("," directive_option)\*
@anchor{library/doctest grammar-token-doctest-directive_option}@anchor{37bb}@anchor{library/doctest grammar-token-directive-option}@anchor{37bc}directive_option      ::= on_or_off directive_option_name
@anchor{library/doctest grammar-token-doctest-on_or_off}@anchor{37bd}@anchor{library/doctest grammar-token-on-or-off}@anchor{37be}on_or_off             ::= "+" \| "-"
@anchor{library/doctest grammar-token-doctest-directive_option_name}@anchor{37bf}@anchor{library/doctest grammar-token-directive-option-name}@anchor{37c0}directive_option_name ::= "DONT_ACCEPT_BLANKLINE" \| "NORMALIZE_WHITESPACE" \| ...
@end example

Whitespace is not allowed between the @code{+} or @code{-} and the directive option
name.  The directive option name can be any of the option flag names explained
above.

An example’s doctest directives modify doctest’s behavior for that single
example.  Use @code{+} to enable the named behavior, or @code{-} to disable it.

For example, this test passes:

@example
>>> print(list(range(20))) 
[0,   1,  2,  3,  4,  5,  6,  7,  8,  9,
10,  11, 12, 13, 14, 15, 16, 17, 18, 19]
@end example

Without the directive it would fail, both because the actual output doesn’t have
two blanks before the single-digit list elements, and because the actual output
is on a single line.  This test also passes, and also requires a directive to do
so:

@example
>>> print(list(range(20))) 
[0, 1, ..., 18, 19]
@end example

Multiple directives can be used on a single physical line, separated by
commas:

@example
>>> print(list(range(20))) 
[0,    1, ...,   18,    19]
@end example

If multiple directive comments are used for a single example, then they are
combined:

@example
>>> print(list(range(20))) 
...                        
[0,    1, ...,   18,    19]
@end example

As the previous example shows, you can add @code{...} lines to your example
containing only directives.  This can be useful when an example is too long for
a directive to comfortably fit on the same line:

@example
>>> print(list(range(5)) + list(range(10, 20)) + list(range(30, 40)))
... 
[0, ..., 4, 10, ..., 19, 30, ..., 39]
@end example

Note that since all options are disabled by default, and directives apply only
to the example they appear in, enabling options (via @code{+} in a directive) is
usually the only meaningful choice.  However, option flags can also be passed to
functions that run doctests, establishing different defaults.  In such cases,
disabling an option via @code{-} in a directive can be useful.

@node Warnings<2>,,Directives,How It Works
@anchor{library/doctest doctest-warnings}@anchor{37c1}@anchor{library/doctest warnings}@anchor{37c2}
@subsubsection Warnings


@ref{63,,doctest} is serious about requiring exact matches in expected output.  If
even a single character doesn’t match, the test fails.  This will probably
surprise you a few times, as you learn exactly what Python does and doesn’t
guarantee about output.  For example, when printing a set, Python doesn’t
guarantee that the element is printed in any particular order, so a test like

@example
>>> foo()
@{"Hermione", "Harry"@}
@end example

is vulnerable!  One workaround is to do

@example
>>> foo() == @{"Hermione", "Harry"@}
True
@end example

instead.  Another is to do

@example
>>> d = sorted(foo())
>>> d
['Harry', 'Hermione']
@end example

@cartouche
@quotation Note 
Before Python 3.6, when printing a dict, Python did not guarantee that
the key-value pairs was printed in any particular order.
@end quotation
@end cartouche

There are others, but you get the idea.

Another bad idea is to print things that embed an object address, like

@example
>>> id(1.0) # certain to fail some of the time
7948648
>>> class C: pass
>>> C()   # the default repr() for instances embeds an address
<__main__.C instance at 0x00AC18F0>
@end example

The @ref{fca,,ELLIPSIS} directive gives a nice approach for the last example:

@example
>>> C() 
<__main__.C instance at 0x...>
@end example

Floating-point numbers are also subject to small output variations across
platforms, because Python defers to the platform C library for float formatting,
and C libraries vary widely in quality here.

@example
>>> 1./7  # risky
0.14285714285714285
>>> print(1./7) # safer
0.142857142857
>>> print(round(1./7, 6)) # much safer
0.142857
@end example

Numbers of the form @code{I/2.**J} are safe across all platforms, and I often
contrive doctest examples to produce numbers of that form:

@example
>>> 3./4  # utterly safe
0.75
@end example

Simple fractions are also easier for people to understand, and that makes for
better documentation.

@node Basic API,Unittest API,How It Works,doctest — Test interactive Python examples
@anchor{library/doctest basic-api}@anchor{37c3}@anchor{library/doctest doctest-basic-api}@anchor{3799}
@subsubsection Basic API


The functions @ref{fc9,,testmod()} and @ref{379c,,testfile()} provide a simple interface to
doctest that should be sufficient for most basic uses.  For a less formal
introduction to these two functions, see sections @ref{3797,,Simple Usage; Checking Examples in Docstrings}
and @ref{379a,,Simple Usage; Checking Examples in a Text File}.

@geindex testfile() (in module doctest)
@anchor{library/doctest doctest testfile}@anchor{379c}
@deffn {Function} doctest.testfile (filename, module_relative=True, name=None, package=None, globs=None, verbose=None, report=True, optionflags=0, extraglobs=None, raise_on_error=False, parser=DocTestParser(), encoding=None)

All arguments except @emph{filename} are optional, and should be specified in keyword
form.

Test examples in the file named @emph{filename}.  Return @code{(failure_count,
test_count)}.

Optional argument @emph{module_relative} specifies how the filename should be
interpreted:


@itemize *

@item 
If @emph{module_relative} is @code{True} (the default), then @emph{filename} specifies an
OS-independent module-relative path.  By default, this path is relative to the
calling module’s directory; but if the @emph{package} argument is specified, then it
is relative to that package.  To ensure OS-independence, @emph{filename} should use
@code{/} characters to separate path segments, and may not be an absolute path
(i.e., it may not begin with @code{/}).

@item 
If @emph{module_relative} is @code{False}, then @emph{filename} specifies an OS-specific
path.  The path may be absolute or relative; relative paths are resolved with
respect to the current working directory.
@end itemize

Optional argument @emph{name} gives the name of the test; by default, or if @code{None},
@code{os.path.basename(filename)} is used.

Optional argument @emph{package} is a Python package or the name of a Python package
whose directory should be used as the base directory for a module-relative
filename.  If no package is specified, then the calling module’s directory is
used as the base directory for module-relative filenames.  It is an error to
specify @emph{package} if @emph{module_relative} is @code{False}.

Optional argument @emph{globs} gives a dict to be used as the globals when executing
examples.  A new shallow copy of this dict is created for the doctest, so its
examples start with a clean slate. By default, or if @code{None}, a new empty dict
is used.

Optional argument @emph{extraglobs} gives a dict merged into the globals used to
execute examples.  This works like @ref{fc3,,dict.update()}:  if @emph{globs} and
@emph{extraglobs} have a common key, the associated value in @emph{extraglobs} appears in
the combined dict.  By default, or if @code{None}, no extra globals are used.  This
is an advanced feature that allows parameterization of doctests.  For example, a
doctest can be written for a base class, using a generic name for the class,
then reused to test any number of subclasses by passing an @emph{extraglobs} dict
mapping the generic name to the subclass to be tested.

Optional argument @emph{verbose} prints lots of stuff if true, and prints only
failures if false; by default, or if @code{None}, it’s true if and only if @code{'-v'}
is in @code{sys.argv}.

Optional argument @emph{report} prints a summary at the end when true, else prints
nothing at the end.  In verbose mode, the summary is detailed, else the summary
is very brief (in fact, empty if all tests passed).

Optional argument @emph{optionflags} (default value 0) takes the
@ref{18f9,,bitwise OR} of option flags.
See section @ref{ab0,,Option Flags}.

Optional argument @emph{raise_on_error} defaults to false.  If true, an exception is
raised upon the first failure or unexpected exception in an example.  This
allows failures to be post-mortem debugged. Default behavior is to continue
running examples.

Optional argument @emph{parser} specifies a @ref{37a5,,DocTestParser} (or subclass) that
should be used to extract tests from the files.  It defaults to a normal parser
(i.e., @code{DocTestParser()}).

Optional argument @emph{encoding} specifies an encoding that should be used to
convert the file to unicode.
@end deffn

@geindex testmod() (in module doctest)
@anchor{library/doctest doctest testmod}@anchor{fc9}
@deffn {Function} doctest.testmod (m=None, name=None, globs=None, verbose=None, report=True, optionflags=0, extraglobs=None, raise_on_error=False, exclude_empty=False)

All arguments are optional, and all except for @emph{m} should be specified in
keyword form.

Test examples in docstrings in functions and classes reachable from module @emph{m}
(or module @ref{1,,__main__} if @emph{m} is not supplied or is @code{None}), starting with
@code{m.__doc__}.

Also test examples reachable from dict @code{m.__test__}, if it exists and is not
@code{None}.  @code{m.__test__} maps names (strings) to functions, classes and
strings; function and class docstrings are searched for examples; strings are
searched directly, as if they were docstrings.

Only docstrings attached to objects belonging to module @emph{m} are searched.

Return @code{(failure_count, test_count)}.

Optional argument @emph{name} gives the name of the module; by default, or if
@code{None}, @code{m.__name__} is used.

Optional argument @emph{exclude_empty} defaults to false.  If true, objects for which
no doctests are found are excluded from consideration. The default is a backward
compatibility hack, so that code still using @code{doctest.master.summarize()} in
conjunction with @ref{fc9,,testmod()} continues to get output for objects with no
tests. The @emph{exclude_empty} argument to the newer @ref{37c4,,DocTestFinder}
constructor defaults to true.

Optional arguments @emph{extraglobs}, @emph{verbose}, @emph{report}, @emph{optionflags},
@emph{raise_on_error}, and @emph{globs} are the same as for function @ref{379c,,testfile()}
above, except that @emph{globs} defaults to @code{m.__dict__}.
@end deffn

@geindex run_docstring_examples() (in module doctest)
@anchor{library/doctest doctest run_docstring_examples}@anchor{37c5}
@deffn {Function} doctest.run_docstring_examples (f, globs, verbose=False, name='NoName', compileflags=None, optionflags=0)

Test examples associated with object @emph{f}; for example, @emph{f} may be a string,
a module, a function, or a class object.

A shallow copy of dictionary argument @emph{globs} is used for the execution context.

Optional argument @emph{name} is used in failure messages, and defaults to
@code{"NoName"}.

If optional argument @emph{verbose} is true, output is generated even if there are no
failures.  By default, output is generated only in case of an example failure.

Optional argument @emph{compileflags} gives the set of flags that should be used by
the Python compiler when running the examples.  By default, or if @code{None},
flags are deduced corresponding to the set of future features found in @emph{globs}.

Optional argument @emph{optionflags} works as for function @ref{379c,,testfile()} above.
@end deffn

@node Unittest API,Advanced API,Basic API,doctest — Test interactive Python examples
@anchor{library/doctest doctest-unittest-api}@anchor{37c6}@anchor{library/doctest unittest-api}@anchor{37c7}
@subsubsection Unittest API


As your collection of doctest’ed modules grows, you’ll want a way to run all
their doctests systematically.  @ref{63,,doctest} provides two functions that can
be used to create @ref{122,,unittest} test suites from modules and text files
containing doctests.  To integrate with @ref{122,,unittest} test discovery, include
a @code{load_tests()} function in your test module:

@example
import unittest
import doctest
import my_module_with_doctests

def load_tests(loader, tests, ignore):
    tests.addTests(doctest.DocTestSuite(my_module_with_doctests))
    return tests
@end example

There are two main functions for creating @ref{953,,unittest.TestSuite} instances
from text files and modules with doctests:

@geindex DocFileSuite() (in module doctest)
@anchor{library/doctest doctest DocFileSuite}@anchor{37c8}
@deffn {Function} doctest.DocFileSuite (*paths, module_relative=True, package=None, setUp=None, tearDown=None, globs=None, optionflags=0, parser=DocTestParser(), encoding=None)

Convert doctest tests from one or more text files to a
@ref{953,,unittest.TestSuite}.

The returned @ref{953,,unittest.TestSuite} is to be run by the unittest framework
and runs the interactive examples in each file.  If an example in any file
fails, then the synthesized unit test fails, and a @code{failureException}
exception is raised showing the name of the file containing the test and a
(sometimes approximate) line number.

Pass one or more paths (as strings) to text files to be examined.

Options may be provided as keyword arguments:

Optional argument @emph{module_relative} specifies how the filenames in @emph{paths}
should be interpreted:


@itemize *

@item 
If @emph{module_relative} is @code{True} (the default), then each filename in
@emph{paths} specifies an OS-independent module-relative path.  By default, this
path is relative to the calling module’s directory; but if the @emph{package}
argument is specified, then it is relative to that package.  To ensure
OS-independence, each filename should use @code{/} characters to separate path
segments, and may not be an absolute path (i.e., it may not begin with
@code{/}).

@item 
If @emph{module_relative} is @code{False}, then each filename in @emph{paths} specifies
an OS-specific path.  The path may be absolute or relative; relative paths
are resolved with respect to the current working directory.
@end itemize

Optional argument @emph{package} is a Python package or the name of a Python
package whose directory should be used as the base directory for
module-relative filenames in @emph{paths}.  If no package is specified, then the
calling module’s directory is used as the base directory for module-relative
filenames.  It is an error to specify @emph{package} if @emph{module_relative} is
@code{False}.

Optional argument @emph{setUp} specifies a set-up function for the test suite.
This is called before running the tests in each file.  The @emph{setUp} function
will be passed a @ref{37c9,,DocTest} object.  The setUp function can access the
test globals as the @emph{globs} attribute of the test passed.

Optional argument @emph{tearDown} specifies a tear-down function for the test
suite.  This is called after running the tests in each file.  The @emph{tearDown}
function will be passed a @ref{37c9,,DocTest} object.  The setUp function can
access the test globals as the @emph{globs} attribute of the test passed.

Optional argument @emph{globs} is a dictionary containing the initial global
variables for the tests.  A new copy of this dictionary is created for each
test.  By default, @emph{globs} is a new empty dictionary.

Optional argument @emph{optionflags} specifies the default doctest options for the
tests, created by or-ing together individual option flags.  See section
@ref{ab0,,Option Flags}. See function @ref{37ca,,set_unittest_reportflags()} below
for a better way to set reporting options.

Optional argument @emph{parser} specifies a @ref{37a5,,DocTestParser} (or subclass)
that should be used to extract tests from the files.  It defaults to a normal
parser (i.e., @code{DocTestParser()}).

Optional argument @emph{encoding} specifies an encoding that should be used to
convert the file to unicode.

The global @code{__file__} is added to the globals provided to doctests loaded
from a text file using @ref{37c8,,DocFileSuite()}.
@end deffn

@geindex DocTestSuite() (in module doctest)
@anchor{library/doctest doctest DocTestSuite}@anchor{952}
@deffn {Function} doctest.DocTestSuite (module=None, globs=None, extraglobs=None, test_finder=None, setUp=None, tearDown=None, checker=None)

Convert doctest tests for a module to a @ref{953,,unittest.TestSuite}.

The returned @ref{953,,unittest.TestSuite} is to be run by the unittest framework
and runs each doctest in the module.  If any of the doctests fail, then the
synthesized unit test fails, and a @code{failureException} exception is raised
showing the name of the file containing the test and a (sometimes approximate)
line number.

Optional argument @emph{module} provides the module to be tested.  It can be a module
object or a (possibly dotted) module name.  If not specified, the module calling
this function is used.

Optional argument @emph{globs} is a dictionary containing the initial global
variables for the tests.  A new copy of this dictionary is created for each
test.  By default, @emph{globs} is a new empty dictionary.

Optional argument @emph{extraglobs} specifies an extra set of global variables, which
is merged into @emph{globs}.  By default, no extra globals are used.

Optional argument @emph{test_finder} is the @ref{37c4,,DocTestFinder} object (or a
drop-in replacement) that is used to extract doctests from the module.

Optional arguments @emph{setUp}, @emph{tearDown}, and @emph{optionflags} are the same as for
function @ref{37c8,,DocFileSuite()} above.

This function uses the same search technique as @ref{fc9,,testmod()}.


Changed in version 3.5: @ref{952,,DocTestSuite()} returns an empty @ref{953,,unittest.TestSuite} if @emph{module}
contains no docstrings instead of raising @ref{17c,,ValueError}.

@end deffn

Under the covers, @ref{952,,DocTestSuite()} creates a @ref{953,,unittest.TestSuite} out
of @code{doctest.DocTestCase} instances, and @code{DocTestCase} is a
subclass of @ref{1ab,,unittest.TestCase}. @code{DocTestCase} isn’t documented
here (it’s an internal detail), but studying its code can answer questions about
the exact details of @ref{122,,unittest} integration.

Similarly, @ref{37c8,,DocFileSuite()} creates a @ref{953,,unittest.TestSuite} out of
@code{doctest.DocFileCase} instances, and @code{DocFileCase} is a subclass
of @code{DocTestCase}.

So both ways of creating a @ref{953,,unittest.TestSuite} run instances of
@code{DocTestCase}.  This is important for a subtle reason: when you run
@ref{63,,doctest} functions yourself, you can control the @ref{63,,doctest} options in
use directly, by passing option flags to @ref{63,,doctest} functions.  However, if
you’re writing a @ref{122,,unittest} framework, @ref{122,,unittest} ultimately controls
when and how tests get run.  The framework author typically wants to control
@ref{63,,doctest} reporting options (perhaps, e.g., specified by command line
options), but there’s no way to pass options through @ref{122,,unittest} to
@ref{63,,doctest} test runners.

For this reason, @ref{63,,doctest} also supports a notion of @ref{63,,doctest}
reporting flags specific to @ref{122,,unittest} support, via this function:

@geindex set_unittest_reportflags() (in module doctest)
@anchor{library/doctest doctest set_unittest_reportflags}@anchor{37ca}
@deffn {Function} doctest.set_unittest_reportflags (flags)

Set the @ref{63,,doctest} reporting flags to use.

Argument @emph{flags} takes the @ref{18f9,,bitwise OR} of option flags.  See
section @ref{ab0,,Option Flags}.  Only “reporting flags” can be used.

This is a module-global setting, and affects all future doctests run by module
@ref{122,,unittest}:  the @code{runTest()} method of @code{DocTestCase} looks at
the option flags specified for the test case when the @code{DocTestCase}
instance was constructed.  If no reporting flags were specified (which is the
typical and expected case), @ref{63,,doctest}’s @ref{122,,unittest} reporting flags are
@ref{18f9,,bitwise ORed} into the option flags, and the option flags
so augmented are passed to the @ref{37b5,,DocTestRunner} instance created to
run the doctest.  If any reporting flags were specified when the
@code{DocTestCase} instance was constructed, @ref{63,,doctest}’s
@ref{122,,unittest} reporting flags are ignored.

The value of the @ref{122,,unittest} reporting flags in effect before the function
was called is returned by the function.
@end deffn

@node Advanced API,Debugging,Unittest API,doctest — Test interactive Python examples
@anchor{library/doctest advanced-api}@anchor{37cb}@anchor{library/doctest doctest-advanced-api}@anchor{37cc}
@subsubsection Advanced API


The basic API is a simple wrapper that’s intended to make doctest easy to use.
It is fairly flexible, and should meet most users’ needs; however, if you
require more fine-grained control over testing, or wish to extend doctest’s
capabilities, then you should use the advanced API.

The advanced API revolves around two container classes, which are used to store
the interactive examples extracted from doctest cases:


@itemize *

@item 
@ref{37cd,,Example}: A single Python @ref{2034,,statement}, paired with its expected
output.

@item 
@ref{37c9,,DocTest}: A collection of @ref{37cd,,Example}s, typically extracted
from a single docstring or text file.
@end itemize

Additional processing classes are defined to find, parse, and run, and check
doctest examples:


@itemize *

@item 
@ref{37c4,,DocTestFinder}: Finds all docstrings in a given module, and uses a
@ref{37a5,,DocTestParser} to create a @ref{37c9,,DocTest} from every docstring that
contains interactive examples.

@item 
@ref{37a5,,DocTestParser}: Creates a @ref{37c9,,DocTest} object from a string (such
as an object’s docstring).

@item 
@ref{37b5,,DocTestRunner}: Executes the examples in a @ref{37c9,,DocTest}, and uses
an @ref{37b4,,OutputChecker} to verify their output.

@item 
@ref{37b4,,OutputChecker}: Compares the actual output from a doctest example with
the expected output, and decides whether they match.
@end itemize

The relationships among these processing classes are summarized in the following
diagram:

@example
                            list of:
+------+                   +---------+
|module| --DocTestFinder-> | DocTest | --DocTestRunner-> results
+------+    |        ^     +---------+     |       ^    (printed)
            |        |     | Example |     |       |
            v        |     |   ...   |     v       |
           DocTestParser   | Example |   OutputChecker
                           +---------+
@end example

@menu
* DocTest Objects:: 
* Example Objects:: 
* DocTestFinder objects:: 
* DocTestParser objects:: 
* DocTestRunner objects:: 
* OutputChecker objects:: 

@end menu

@node DocTest Objects,Example Objects,,Advanced API
@anchor{library/doctest doctest-doctest}@anchor{37ce}@anchor{library/doctest doctest-objects}@anchor{37cf}
@subsubsection DocTest Objects


@geindex DocTest (class in doctest)
@anchor{library/doctest doctest DocTest}@anchor{37c9}
@deffn {Class} doctest.DocTest (examples, globs, name, filename, lineno, docstring)

A collection of doctest examples that should be run in a single namespace.  The
constructor arguments are used to initialize the attributes of the same names.

@ref{37c9,,DocTest} defines the following attributes.  They are initialized by
the constructor, and should not be modified directly.

@geindex examples (doctest.DocTest attribute)
@anchor{library/doctest doctest DocTest examples}@anchor{37d0}
@deffn {Attribute} examples

A list of @ref{37cd,,Example} objects encoding the individual interactive Python
examples that should be run by this test.
@end deffn

@geindex globs (doctest.DocTest attribute)
@anchor{library/doctest doctest DocTest globs}@anchor{37d1}
@deffn {Attribute} globs

The namespace (aka globals) that the examples should be run in. This is a
dictionary mapping names to values.  Any changes to the namespace made by the
examples (such as binding new variables) will be reflected in @ref{37d1,,globs}
after the test is run.
@end deffn

@geindex name (doctest.DocTest attribute)
@anchor{library/doctest doctest DocTest name}@anchor{37d2}
@deffn {Attribute} name

A string name identifying the @ref{37c9,,DocTest}.  Typically, this is the name
of the object or file that the test was extracted from.
@end deffn

@geindex filename (doctest.DocTest attribute)
@anchor{library/doctest doctest DocTest filename}@anchor{37d3}
@deffn {Attribute} filename

The name of the file that this @ref{37c9,,DocTest} was extracted from; or
@code{None} if the filename is unknown, or if the @ref{37c9,,DocTest} was not
extracted from a file.
@end deffn

@geindex lineno (doctest.DocTest attribute)
@anchor{library/doctest doctest DocTest lineno}@anchor{37d4}
@deffn {Attribute} lineno

The line number within @ref{37d3,,filename} where this @ref{37c9,,DocTest} begins, or
@code{None} if the line number is unavailable.  This line number is zero-based
with respect to the beginning of the file.
@end deffn

@geindex docstring (doctest.DocTest attribute)
@anchor{library/doctest doctest DocTest docstring}@anchor{37d5}
@deffn {Attribute} docstring

The string that the test was extracted from, or @code{None} if the string is
unavailable, or if the test was not extracted from a string.
@end deffn
@end deffn

@node Example Objects,DocTestFinder objects,DocTest Objects,Advanced API
@anchor{library/doctest doctest-example}@anchor{37d6}@anchor{library/doctest example-objects}@anchor{37d7}
@subsubsection Example Objects


@geindex Example (class in doctest)
@anchor{library/doctest doctest Example}@anchor{37cd}
@deffn {Class} doctest.Example (source, want, exc_msg=None, lineno=0, indent=0, options=None)

A single interactive example, consisting of a Python statement and its expected
output.  The constructor arguments are used to initialize the attributes of
the same names.

@ref{37cd,,Example} defines the following attributes.  They are initialized by
the constructor, and should not be modified directly.

@geindex source (doctest.Example attribute)
@anchor{library/doctest doctest Example source}@anchor{37d8}
@deffn {Attribute} source

A string containing the example’s source code.  This source code consists of a
single Python statement, and always ends with a newline; the constructor adds
a newline when necessary.
@end deffn

@geindex want (doctest.Example attribute)
@anchor{library/doctest doctest Example want}@anchor{37d9}
@deffn {Attribute} want

The expected output from running the example’s source code (either from
stdout, or a traceback in case of exception).  @ref{37d9,,want} ends with a
newline unless no output is expected, in which case it’s an empty string.  The
constructor adds a newline when necessary.
@end deffn

@geindex exc_msg (doctest.Example attribute)
@anchor{library/doctest doctest Example exc_msg}@anchor{37da}
@deffn {Attribute} exc_msg

The exception message generated by the example, if the example is expected to
generate an exception; or @code{None} if it is not expected to generate an
exception.  This exception message is compared against the return value of
@ref{2d4,,traceback.format_exception_only()}.  @ref{37da,,exc_msg} ends with a newline
unless it’s @code{None}.  The constructor adds a newline if needed.
@end deffn

@geindex lineno (doctest.Example attribute)
@anchor{library/doctest doctest Example lineno}@anchor{37db}
@deffn {Attribute} lineno

The line number within the string containing this example where the example
begins.  This line number is zero-based with respect to the beginning of the
containing string.
@end deffn

@geindex indent (doctest.Example attribute)
@anchor{library/doctest doctest Example indent}@anchor{37dc}
@deffn {Attribute} indent

The example’s indentation in the containing string, i.e., the number of space
characters that precede the example’s first prompt.
@end deffn

@geindex options (doctest.Example attribute)
@anchor{library/doctest doctest Example options}@anchor{37dd}
@deffn {Attribute} options

A dictionary mapping from option flags to @code{True} or @code{False}, which is used
to override default options for this example.  Any option flags not contained
in this dictionary are left at their default value (as specified by the
@ref{37b5,,DocTestRunner}’s @code{optionflags}). By default, no options are set.
@end deffn
@end deffn

@node DocTestFinder objects,DocTestParser objects,Example Objects,Advanced API
@anchor{library/doctest doctest-doctestfinder}@anchor{37de}@anchor{library/doctest doctestfinder-objects}@anchor{37df}
@subsubsection DocTestFinder objects


@geindex DocTestFinder (class in doctest)
@anchor{library/doctest doctest DocTestFinder}@anchor{37c4}
@deffn {Class} doctest.DocTestFinder (verbose=False, parser=DocTestParser(), recurse=True, exclude_empty=True)

A processing class used to extract the @ref{37c9,,DocTest}s that are relevant to
a given object, from its docstring and the docstrings of its contained objects.
@ref{37c9,,DocTest}s can be extracted from modules, classes, functions,
methods, staticmethods, classmethods, and properties.

The optional argument @emph{verbose} can be used to display the objects searched by
the finder.  It defaults to @code{False} (no output).

The optional argument @emph{parser} specifies the @ref{37a5,,DocTestParser} object (or a
drop-in replacement) that is used to extract doctests from docstrings.

If the optional argument @emph{recurse} is false, then @ref{37e0,,DocTestFinder.find()}
will only examine the given object, and not any contained objects.

If the optional argument @emph{exclude_empty} is false, then
@ref{37e0,,DocTestFinder.find()} will include tests for objects with empty docstrings.

@ref{37c4,,DocTestFinder} defines the following method:

@geindex find() (doctest.DocTestFinder method)
@anchor{library/doctest doctest DocTestFinder find}@anchor{37e0}
@deffn {Method} find (obj[, name][, module][, globs][, extraglobs])

Return a list of the @ref{37c9,,DocTest}s that are defined by @emph{obj}’s
docstring, or by any of its contained objects’ docstrings.

The optional argument @emph{name} specifies the object’s name; this name will be
used to construct names for the returned @ref{37c9,,DocTest}s.  If @emph{name} is
not specified, then @code{obj.__name__} is used.

The optional parameter @emph{module} is the module that contains the given object.
If the module is not specified or is @code{None}, then the test finder will attempt
to automatically determine the correct module.  The object’s module is used:


@itemize *

@item 
As a default namespace, if @emph{globs} is not specified.

@item 
To prevent the DocTestFinder from extracting DocTests from objects that are
imported from other modules.  (Contained objects with modules other than
@emph{module} are ignored.)

@item 
To find the name of the file containing the object.

@item 
To help find the line number of the object within its file.
@end itemize

If @emph{module} is @code{False}, no attempt to find the module will be made.  This is
obscure, of use mostly in testing doctest itself: if @emph{module} is @code{False}, or
is @code{None} but cannot be found automatically, then all objects are considered
to belong to the (non-existent) module, so all contained objects will
(recursively) be searched for doctests.

The globals for each @ref{37c9,,DocTest} is formed by combining @emph{globs} and
@emph{extraglobs} (bindings in @emph{extraglobs} override bindings in @emph{globs}).  A new
shallow copy of the globals dictionary is created for each @ref{37c9,,DocTest}.
If @emph{globs} is not specified, then it defaults to the module’s @emph{__dict__}, if
specified, or @code{@{@}} otherwise.  If @emph{extraglobs} is not specified, then it
defaults to @code{@{@}}.
@end deffn
@end deffn

@node DocTestParser objects,DocTestRunner objects,DocTestFinder objects,Advanced API
@anchor{library/doctest doctest-doctestparser}@anchor{37e1}@anchor{library/doctest doctestparser-objects}@anchor{37e2}
@subsubsection DocTestParser objects


@geindex DocTestParser (class in doctest)
@anchor{library/doctest doctest DocTestParser}@anchor{37a5}
@deffn {Class} doctest.DocTestParser

A processing class used to extract interactive examples from a string, and use
them to create a @ref{37c9,,DocTest} object.

@ref{37a5,,DocTestParser} defines the following methods:

@geindex get_doctest() (doctest.DocTestParser method)
@anchor{library/doctest doctest DocTestParser get_doctest}@anchor{37e3}
@deffn {Method} get_doctest (string, globs, name, filename, lineno)

Extract all doctest examples from the given string, and collect them into a
@ref{37c9,,DocTest} object.

@emph{globs}, @emph{name}, @emph{filename}, and @emph{lineno} are attributes for the new
@ref{37c9,,DocTest} object.  See the documentation for @ref{37c9,,DocTest} for more
information.
@end deffn

@geindex get_examples() (doctest.DocTestParser method)
@anchor{library/doctest doctest DocTestParser get_examples}@anchor{37e4}
@deffn {Method} get_examples (string, name='<string>')

Extract all doctest examples from the given string, and return them as a list
of @ref{37cd,,Example} objects.  Line numbers are 0-based.  The optional argument
@emph{name} is a name identifying this string, and is only used for error messages.
@end deffn

@geindex parse() (doctest.DocTestParser method)
@anchor{library/doctest doctest DocTestParser parse}@anchor{37e5}
@deffn {Method} parse (string, name='<string>')

Divide the given string into examples and intervening text, and return them as
a list of alternating @ref{37cd,,Example}s and strings. Line numbers for the
@ref{37cd,,Example}s are 0-based.  The optional argument @emph{name} is a name
identifying this string, and is only used for error messages.
@end deffn
@end deffn

@node DocTestRunner objects,OutputChecker objects,DocTestParser objects,Advanced API
@anchor{library/doctest doctest-doctestrunner}@anchor{37e6}@anchor{library/doctest doctestrunner-objects}@anchor{37e7}
@subsubsection DocTestRunner objects


@geindex DocTestRunner (class in doctest)
@anchor{library/doctest doctest DocTestRunner}@anchor{37b5}
@deffn {Class} doctest.DocTestRunner (checker=None, verbose=None, optionflags=0)

A processing class used to execute and verify the interactive examples in a
@ref{37c9,,DocTest}.

The comparison between expected outputs and actual outputs is done by an
@ref{37b4,,OutputChecker}.  This comparison may be customized with a number of
option flags; see section @ref{ab0,,Option Flags} for more information.  If the
option flags are insufficient, then the comparison may also be customized by
passing a subclass of @ref{37b4,,OutputChecker} to the constructor.

The test runner’s display output can be controlled in two ways. First, an output
function can be passed to @code{TestRunner.run()}; this function will be called
with strings that should be displayed.  It defaults to @code{sys.stdout.write}.  If
capturing the output is not sufficient, then the display output can be also
customized by subclassing DocTestRunner, and overriding the methods
@ref{37e8,,report_start()}, @ref{37e9,,report_success()},
@ref{37ea,,report_unexpected_exception()}, and @ref{37eb,,report_failure()}.

The optional keyword argument @emph{checker} specifies the @ref{37b4,,OutputChecker}
object (or drop-in replacement) that should be used to compare the expected
outputs to the actual outputs of doctest examples.

The optional keyword argument @emph{verbose} controls the @ref{37b5,,DocTestRunner}’s
verbosity.  If @emph{verbose} is @code{True}, then information is printed about each
example, as it is run.  If @emph{verbose} is @code{False}, then only failures are
printed.  If @emph{verbose} is unspecified, or @code{None}, then verbose output is used
iff the command-line switch @code{-v} is used.

The optional keyword argument @emph{optionflags} can be used to control how the test
runner compares expected output to actual output, and how it displays failures.
For more information, see section @ref{ab0,,Option Flags}.

@ref{37a5,,DocTestParser} defines the following methods:

@geindex report_start() (doctest.DocTestRunner method)
@anchor{library/doctest doctest DocTestRunner report_start}@anchor{37e8}
@deffn {Method} report_start (out, test, example)

Report that the test runner is about to process the given example. This method
is provided to allow subclasses of @ref{37b5,,DocTestRunner} to customize their
output; it should not be called directly.

@emph{example} is the example about to be processed.  @emph{test} is the test
@emph{containing example}.  @emph{out} is the output function that was passed to
@ref{37ec,,DocTestRunner.run()}.
@end deffn

@geindex report_success() (doctest.DocTestRunner method)
@anchor{library/doctest doctest DocTestRunner report_success}@anchor{37e9}
@deffn {Method} report_success (out, test, example, got)

Report that the given example ran successfully.  This method is provided to
allow subclasses of @ref{37b5,,DocTestRunner} to customize their output; it
should not be called directly.

@emph{example} is the example about to be processed.  @emph{got} is the actual output
from the example.  @emph{test} is the test containing @emph{example}.  @emph{out} is the
output function that was passed to @ref{37ec,,DocTestRunner.run()}.
@end deffn

@geindex report_failure() (doctest.DocTestRunner method)
@anchor{library/doctest doctest DocTestRunner report_failure}@anchor{37eb}
@deffn {Method} report_failure (out, test, example, got)

Report that the given example failed.  This method is provided to allow
subclasses of @ref{37b5,,DocTestRunner} to customize their output; it should not
be called directly.

@emph{example} is the example about to be processed.  @emph{got} is the actual output
from the example.  @emph{test} is the test containing @emph{example}.  @emph{out} is the
output function that was passed to @ref{37ec,,DocTestRunner.run()}.
@end deffn

@geindex report_unexpected_exception() (doctest.DocTestRunner method)
@anchor{library/doctest doctest DocTestRunner report_unexpected_exception}@anchor{37ea}
@deffn {Method} report_unexpected_exception (out, test, example, exc_info)

Report that the given example raised an unexpected exception. This method is
provided to allow subclasses of @ref{37b5,,DocTestRunner} to customize their
output; it should not be called directly.

@emph{example} is the example about to be processed. @emph{exc_info} is a tuple
containing information about the unexpected exception (as returned by
@ref{18f,,sys.exc_info()}). @emph{test} is the test containing @emph{example}.  @emph{out} is the
output function that was passed to @ref{37ec,,DocTestRunner.run()}.
@end deffn

@geindex run() (doctest.DocTestRunner method)
@anchor{library/doctest doctest DocTestRunner run}@anchor{37ec}
@deffn {Method} run (test, compileflags=None, out=None, clear_globs=True)

Run the examples in @emph{test} (a @ref{37c9,,DocTest} object), and display the
results using the writer function @emph{out}.

The examples are run in the namespace @code{test.globs}.  If @emph{clear_globs} is
true (the default), then this namespace will be cleared after the test runs,
to help with garbage collection. If you would like to examine the namespace
after the test completes, then use @emph{clear_globs=False}.

@emph{compileflags} gives the set of flags that should be used by the Python
compiler when running the examples.  If not specified, then it will default to
the set of future-import flags that apply to @emph{globs}.

The output of each example is checked using the @ref{37b5,,DocTestRunner}’s
output checker, and the results are formatted by the
@code{DocTestRunner.report_*()} methods.
@end deffn

@geindex summarize() (doctest.DocTestRunner method)
@anchor{library/doctest doctest DocTestRunner summarize}@anchor{37ed}
@deffn {Method} summarize (verbose=None)

Print a summary of all the test cases that have been run by this DocTestRunner,
and return a @ref{ce9,,named tuple} @code{TestResults(failed, attempted)}.

The optional @emph{verbose} argument controls how detailed the summary is.  If the
verbosity is not specified, then the @ref{37b5,,DocTestRunner}’s verbosity is
used.
@end deffn
@end deffn

@node OutputChecker objects,,DocTestRunner objects,Advanced API
@anchor{library/doctest doctest-outputchecker}@anchor{37ee}@anchor{library/doctest outputchecker-objects}@anchor{37ef}
@subsubsection OutputChecker objects


@geindex OutputChecker (class in doctest)
@anchor{library/doctest doctest OutputChecker}@anchor{37b4}
@deffn {Class} doctest.OutputChecker

A class used to check the whether the actual output from a doctest example
matches the expected output.  @ref{37b4,,OutputChecker} defines two methods:
@ref{37f0,,check_output()}, which compares a given pair of outputs, and returns @code{True}
if they match; and @ref{37f1,,output_difference()}, which returns a string describing
the differences between two outputs.

@ref{37b4,,OutputChecker} defines the following methods:

@geindex check_output() (doctest.OutputChecker method)
@anchor{library/doctest doctest OutputChecker check_output}@anchor{37f0}
@deffn {Method} check_output (want, got, optionflags)

Return @code{True} iff the actual output from an example (@emph{got}) matches the
expected output (@emph{want}).  These strings are always considered to match if
they are identical; but depending on what option flags the test runner is
using, several non-exact match types are also possible.  See section
@ref{ab0,,Option Flags} for more information about option flags.
@end deffn

@geindex output_difference() (doctest.OutputChecker method)
@anchor{library/doctest doctest OutputChecker output_difference}@anchor{37f1}
@deffn {Method} output_difference (example, got, optionflags)

Return a string describing the differences between the expected output for a
given example (@emph{example}) and the actual output (@emph{got}).  @emph{optionflags} is the
set of option flags used to compare @emph{want} and @emph{got}.
@end deffn
@end deffn

@node Debugging,Soapbox,Advanced API,doctest — Test interactive Python examples
@anchor{library/doctest debugging}@anchor{37f2}@anchor{library/doctest doctest-debugging}@anchor{37f3}
@subsubsection Debugging


Doctest provides several mechanisms for debugging doctest examples:


@itemize *

@item 
Several functions convert doctests to executable Python programs, which can be
run under the Python debugger, @ref{c4,,pdb}.

@item 
The @ref{37f4,,DebugRunner} class is a subclass of @ref{37b5,,DocTestRunner} that
raises an exception for the first failing example, containing information about
that example. This information can be used to perform post-mortem debugging on
the example.

@item 
The @ref{122,,unittest} cases generated by @ref{952,,DocTestSuite()} support the
@ref{37f5,,debug()} method defined by @ref{1ab,,unittest.TestCase}.

@item 
You can add a call to @ref{691,,pdb.set_trace()} in a doctest example, and you’ll
drop into the Python debugger when that line is executed.  Then you can inspect
current values of variables, and so on.  For example, suppose @code{a.py}
contains just this module docstring:

@example
"""
>>> def f(x):
...     g(x*2)
>>> def g(x):
...     print(x+3)
...     import pdb; pdb.set_trace()
>>> f(3)
9
"""
@end example

Then an interactive Python session may look like this:

@example
>>> import a, doctest
>>> doctest.testmod(a)
--Return--
> <doctest a[1]>(3)g()->None
-> import pdb; pdb.set_trace()
(Pdb) list
  1     def g(x):
  2         print(x+3)
  3  ->     import pdb; pdb.set_trace()
[EOF]
(Pdb) p x
6
(Pdb) step
--Return--
> <doctest a[0]>(2)f()->None
-> g(x*2)
(Pdb) list
  1     def f(x):
  2  ->     g(x*2)
[EOF]
(Pdb) p x
3
(Pdb) step
--Return--
> <doctest a[2]>(1)?()->None
-> f(3)
(Pdb) cont
(0, 3)
>>>
@end example
@end itemize

Functions that convert doctests to Python code, and possibly run the synthesized
code under the debugger:

@geindex script_from_examples() (in module doctest)
@anchor{library/doctest doctest script_from_examples}@anchor{37f6}
@deffn {Function} doctest.script_from_examples (s)

Convert text with examples to a script.

Argument @emph{s} is a string containing doctest examples.  The string is converted
to a Python script, where doctest examples in @emph{s} are converted to regular code,
and everything else is converted to Python comments.  The generated script is
returned as a string. For example,

@example
import doctest
print(doctest.script_from_examples(r"""
    Set x and y to 1 and 2.
    >>> x, y = 1, 2

    Print their sum:
    >>> print(x+y)
    3
"""))
@end example

displays:

@example
# Set x and y to 1 and 2.
x, y = 1, 2
#
# Print their sum:
print(x+y)
# Expected:
## 3
@end example

This function is used internally by other functions (see below), but can also be
useful when you want to transform an interactive Python session into a Python
script.
@end deffn

@geindex testsource() (in module doctest)
@anchor{library/doctest doctest testsource}@anchor{37f7}
@deffn {Function} doctest.testsource (module, name)

Convert the doctest for an object to a script.

Argument @emph{module} is a module object, or dotted name of a module, containing the
object whose doctests are of interest.  Argument @emph{name} is the name (within the
module) of the object with the doctests of interest.  The result is a string,
containing the object’s docstring converted to a Python script, as described for
@ref{37f6,,script_from_examples()} above.  For example, if module @code{a.py}
contains a top-level function @code{f()}, then

@example
import a, doctest
print(doctest.testsource(a, "a.f"))
@end example

prints a script version of function @code{f()}’s docstring, with doctests
converted to code, and the rest placed in comments.
@end deffn

@geindex debug() (in module doctest)
@anchor{library/doctest doctest debug}@anchor{37f5}
@deffn {Function} doctest.debug (module, name, pm=False)

Debug the doctests for an object.

The @emph{module} and @emph{name} arguments are the same as for function
@ref{37f7,,testsource()} above.  The synthesized Python script for the named object’s
docstring is written to a temporary file, and then that file is run under the
control of the Python debugger, @ref{c4,,pdb}.

A shallow copy of @code{module.__dict__} is used for both local and global
execution context.

Optional argument @emph{pm} controls whether post-mortem debugging is used.  If @emph{pm}
has a true value, the script file is run directly, and the debugger gets
involved only if the script terminates via raising an unhandled exception.  If
it does, then post-mortem debugging is invoked, via @ref{f40,,pdb.post_mortem()},
passing the traceback object from the unhandled exception.  If @emph{pm} is not
specified, or is false, the script is run under the debugger from the start, via
passing an appropriate @ref{34b,,exec()} call to @ref{37f8,,pdb.run()}.
@end deffn

@geindex debug_src() (in module doctest)
@anchor{library/doctest doctest debug_src}@anchor{37f9}
@deffn {Function} doctest.debug_src (src, pm=False, globs=None)

Debug the doctests in a string.

This is like function @ref{37f5,,debug()} above, except that a string containing
doctest examples is specified directly, via the @emph{src} argument.

Optional argument @emph{pm} has the same meaning as in function @ref{37f5,,debug()} above.

Optional argument @emph{globs} gives a dictionary to use as both local and global
execution context.  If not specified, or @code{None}, an empty dictionary is used.
If specified, a shallow copy of the dictionary is used.
@end deffn

The @ref{37f4,,DebugRunner} class, and the special exceptions it may raise, are of
most interest to testing framework authors, and will only be sketched here.  See
the source code, and especially @ref{37f4,,DebugRunner}’s docstring (which is a
doctest!) for more details:

@geindex DebugRunner (class in doctest)
@anchor{library/doctest doctest DebugRunner}@anchor{37f4}
@deffn {Class} doctest.DebugRunner (checker=None, verbose=None, optionflags=0)

A subclass of @ref{37b5,,DocTestRunner} that raises an exception as soon as a
failure is encountered.  If an unexpected exception occurs, an
@ref{37fa,,UnexpectedException} exception is raised, containing the test, the
example, and the original exception.  If the output doesn’t match, then a
@ref{37fb,,DocTestFailure} exception is raised, containing the test, the example, and
the actual output.

For information about the constructor parameters and methods, see the
documentation for @ref{37b5,,DocTestRunner} in section @ref{37cc,,Advanced API}.
@end deffn

There are two exceptions that may be raised by @ref{37f4,,DebugRunner} instances:

@geindex DocTestFailure
@anchor{library/doctest doctest DocTestFailure}@anchor{37fb}
@deffn {Exception} doctest.DocTestFailure (test, example, got)

An exception raised by @ref{37b5,,DocTestRunner} to signal that a doctest example’s
actual output did not match its expected output. The constructor arguments are
used to initialize the attributes of the same names.
@end deffn

@ref{37fb,,DocTestFailure} defines the following attributes:

@geindex test (doctest.DocTestFailure attribute)
@anchor{library/doctest doctest DocTestFailure test}@anchor{37fc}
@deffn {Attribute} DocTestFailure.test

The @ref{37c9,,DocTest} object that was being run when the example failed.
@end deffn

@geindex example (doctest.DocTestFailure attribute)
@anchor{library/doctest doctest DocTestFailure example}@anchor{37fd}
@deffn {Attribute} DocTestFailure.example

The @ref{37cd,,Example} that failed.
@end deffn

@geindex got (doctest.DocTestFailure attribute)
@anchor{library/doctest doctest DocTestFailure got}@anchor{37fe}
@deffn {Attribute} DocTestFailure.got

The example’s actual output.
@end deffn

@geindex UnexpectedException
@anchor{library/doctest doctest UnexpectedException}@anchor{37fa}
@deffn {Exception} doctest.UnexpectedException (test, example, exc_info)

An exception raised by @ref{37b5,,DocTestRunner} to signal that a doctest
example raised an unexpected exception.  The constructor arguments are used
to initialize the attributes of the same names.
@end deffn

@ref{37fa,,UnexpectedException} defines the following attributes:

@geindex test (doctest.UnexpectedException attribute)
@anchor{library/doctest doctest UnexpectedException test}@anchor{37ff}
@deffn {Attribute} UnexpectedException.test

The @ref{37c9,,DocTest} object that was being run when the example failed.
@end deffn

@geindex example (doctest.UnexpectedException attribute)
@anchor{library/doctest doctest UnexpectedException example}@anchor{3800}
@deffn {Attribute} UnexpectedException.example

The @ref{37cd,,Example} that failed.
@end deffn

@geindex exc_info (doctest.UnexpectedException attribute)
@anchor{library/doctest doctest UnexpectedException exc_info}@anchor{3801}
@deffn {Attribute} UnexpectedException.exc_info

A tuple containing information about the unexpected exception, as returned by
@ref{18f,,sys.exc_info()}.
@end deffn

@node Soapbox,,Debugging,doctest — Test interactive Python examples
@anchor{library/doctest doctest-soapbox}@anchor{3802}@anchor{library/doctest soapbox}@anchor{3803}
@subsubsection Soapbox


As mentioned in the introduction, @ref{63,,doctest} has grown to have three primary
uses:


@enumerate 

@item 
Checking examples in docstrings.

@item 
Regression testing.

@item 
Executable documentation / literate testing.
@end enumerate

These uses have different requirements, and it is important to distinguish them.
In particular, filling your docstrings with obscure test cases makes for bad
documentation.

When writing a docstring, choose docstring examples with care. There’s an art to
this that needs to be learned—it may not be natural at first.  Examples should
add genuine value to the documentation.  A good example can often be worth many
words. If done with care, the examples will be invaluable for your users, and
will pay back the time it takes to collect them many times over as the years go
by and things change.  I’m still amazed at how often one of my @ref{63,,doctest}
examples stops working after a “harmless” change.

Doctest also makes an excellent tool for regression testing, especially if you
don’t skimp on explanatory text.  By interleaving prose and examples, it becomes
much easier to keep track of what’s actually being tested, and why.  When a test
fails, good prose can make it much easier to figure out what the problem is, and
how it should be fixed.  It’s true that you could write extensive comments in
code-based testing, but few programmers do. Many have found that using doctest
approaches instead leads to much clearer tests.  Perhaps this is simply because
doctest makes writing prose a little easier than writing code, while writing
comments in code is a little harder.  I think it goes deeper than just that:
the natural attitude when writing a doctest-based test is that you want to
explain the fine points of your software, and illustrate them with examples.
This in turn naturally leads to test files that start with the simplest
features, and logically progress to complications and edge cases.  A coherent
narrative is the result, instead of a collection of isolated functions that test
isolated bits of functionality seemingly at random.  It’s a different attitude,
and produces different results, blurring the distinction between testing and
explaining.

Regression testing is best confined to dedicated objects or files.  There are
several options for organizing tests:


@itemize *

@item 
Write text files containing test cases as interactive examples, and test the
files using @ref{379c,,testfile()} or @ref{37c8,,DocFileSuite()}.  This is recommended,
although is easiest to do for new projects, designed from the start to use
doctest.

@item 
Define functions named @code{_regrtest_topic} that consist of single docstrings,
containing test cases for the named topics.  These functions can be included in
the same file as the module, or separated out into a separate test file.

@item 
Define a @code{__test__} dictionary mapping from regression test topics to
docstrings containing test cases.
@end itemize

When you have placed your tests in a module, the module can itself be the test
runner.  When a test fails, you can arrange for your test runner to re-run only
the failing doctest while you debug the problem.  Here is a minimal example of
such a test runner:

@example
if __name__ == '__main__':
    import doctest
    flags = doctest.REPORT_NDIFF|doctest.FAIL_FAST
    if len(sys.argv) > 1:
        name = sys.argv[1]
        if name in globals():
            obj = globals()[name]
        else:
            obj = __test__[name]
        doctest.run_docstring_examples(obj, globals(), name=name,
                                       optionflags=flags)
    else:
        fail, total = doctest.testmod(optionflags=flags)
        print("@{@} failures out of @{@} tests".format(fail, total))
@end example

@node unittest — Unit testing framework,unittest mock — mock object library,doctest — Test interactive Python examples,Development Tools
@anchor{library/unittest doc}@anchor{3804}@anchor{library/unittest module-unittest}@anchor{122}@anchor{library/unittest unittest-unit-testing-framework}@anchor{3805}
@subsection @code{unittest} — Unit testing framework


@geindex module; unittest

@strong{Source code:} Lib/unittest/__init__.py@footnote{https://github.com/python/cpython/tree/main/Lib/unittest/__init__.py}


__________________________________________________________________


(If you are already familiar with the basic concepts of testing, you might want
to skip to @ref{3806,,the list of assert methods}.)

The @ref{122,,unittest} unit testing framework was originally inspired by JUnit
and has a similar flavor as major unit testing frameworks in other
languages.  It supports test automation, sharing of setup and shutdown code
for tests, aggregation of tests into collections, and independence of the
tests from the reporting framework.

To achieve this, @ref{122,,unittest} supports some important concepts in an
object-oriented way:


@table @asis

@item test fixture

A @emph{test fixture} represents the preparation needed to perform one or more
tests, and any associated cleanup actions.  This may involve, for example,
creating temporary or proxy databases, directories, or starting a server
process.

@item test case

A @emph{test case} is the individual unit of testing.  It checks for a specific
response to a particular set of inputs.  @ref{122,,unittest} provides a base class,
@ref{1ab,,TestCase}, which may be used to create new test cases.

@item test suite

A @emph{test suite} is a collection of test cases, test suites, or both.  It is
used to aggregate tests that should be executed together.

@item test runner

A @emph{test runner} is a component which orchestrates the execution of tests
and provides the outcome to the user.  The runner may use a graphical interface,
a textual interface, or return a special value to indicate the results of
executing the tests.
@end table


@subsubheading See also



@table @asis

@item Module @ref{63,,doctest}

Another test-support module with a very different flavor.

@item Simple Smalltalk Testing: With Patterns@footnote{https://web.archive.org/web/20150315073817/http://www.xprogramming.com/testfram.htm}

Kent Beck’s original paper on testing frameworks using the pattern shared
by @ref{122,,unittest}.

@item pytest@footnote{https://docs.pytest.org/}

Third-party unittest framework with a lighter-weight syntax for writing
tests.  For example, @code{assert func(10) == 42}.

@item The Python Testing Tools Taxonomy@footnote{https://wiki.python.org/moin/PythonTestingToolsTaxonomy}

An extensive list of Python testing tools including functional testing
frameworks and mock object libraries.

@item Testing in Python Mailing List@footnote{http://lists.idyll.org/listinfo/testing-in-python}

A special-interest-group for discussion of testing, and testing tools,
in Python.
@end table

The script @code{Tools/unittestgui/unittestgui.py} in the Python source distribution is
a GUI tool for test discovery and execution.  This is intended largely for ease of use
for those new to unit testing.  For production environments it is
recommended that tests be driven by a continuous integration system such as
Buildbot@footnote{https://buildbot.net/}, Jenkins@footnote{https://jenkins.io/},
GitHub Actions@footnote{https://github.com/features/actions}, or
AppVeyor@footnote{https://www.appveyor.com/}.


@menu
* Basic example:: 
* Command-Line Interface: Command-Line Interface<3>. 
* Test Discovery:: 
* Organizing test code:: 
* Re-using old test code:: 
* Skipping tests and expected failures:: 
* Distinguishing test iterations using subtests:: 
* Classes and functions:: 
* Class and Module Fixtures:: 
* Signal Handling:: 

@end menu

@node Basic example,Command-Line Interface<3>,,unittest — Unit testing framework
@anchor{library/unittest basic-example}@anchor{3807}@anchor{library/unittest unittest-minimal-example}@anchor{3808}
@subsubsection Basic example


The @ref{122,,unittest} module provides a rich set of tools for constructing and
running tests.  This section demonstrates that a small subset of the tools
suffice to meet the needs of most users.

Here is a short script to test three string methods:

@example
import unittest

class TestStringMethods(unittest.TestCase):

    def test_upper(self):
        self.assertEqual('foo'.upper(), 'FOO')

    def test_isupper(self):
        self.assertTrue('FOO'.isupper())
        self.assertFalse('Foo'.isupper())

    def test_split(self):
        s = 'hello world'
        self.assertEqual(s.split(), ['hello', 'world'])
        # check that s.split fails when the separator is not a string
        with self.assertRaises(TypeError):
            s.split(2)

if __name__ == '__main__':
    unittest.main()
@end example

A testcase is created by subclassing @ref{1ab,,unittest.TestCase}.  The three
individual tests are defined with methods whose names start with the letters
@code{test}.  This naming convention informs the test runner about which methods
represent tests.

The crux of each test is a call to @ref{dd5,,assertEqual()} to check for an
expected result; @ref{dd4,,assertTrue()} or @ref{ec9,,assertFalse()}
to verify a condition; or @ref{cff,,assertRaises()} to verify that a
specific exception gets raised.  These methods are used instead of the
@ref{46d,,assert} statement so the test runner can accumulate all test results
and produce a report.

The @ref{ecd,,setUp()} and @ref{ece,,tearDown()} methods allow you
to define instructions that will be executed before and after each test method.
They are covered in more detail in the section @ref{3809,,Organizing test code}.

The final block shows a simple way to run the tests. @ref{b69,,unittest.main()}
provides a command-line interface to the test script.  When run from the command
line, the above script produces an output that looks like this:

@example
...
----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK
@end example

Passing the @code{-v} option to your test script will instruct @ref{b69,,unittest.main()}
to enable a higher level of verbosity, and produce the following output:

@example
test_isupper (__main__.TestStringMethods) ... ok
test_split (__main__.TestStringMethods) ... ok
test_upper (__main__.TestStringMethods) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.001s

OK
@end example

The above examples show the most commonly used @ref{122,,unittest} features which
are sufficient to meet many everyday testing needs.  The remainder of the
documentation explores the full feature set from first principles.


Changed in version 3.11: The behavior of returning a value from a test method (other than the default
@code{None} value), is now deprecated.


@node Command-Line Interface<3>,Test Discovery,Basic example,unittest — Unit testing framework
@anchor{library/unittest command-line-interface}@anchor{380a}@anchor{library/unittest unittest-command-line-interface}@anchor{380b}
@subsubsection Command-Line Interface


The unittest module can be used from the command line to run tests from
modules, classes or even individual test methods:

@example
python -m unittest test_module1 test_module2
python -m unittest test_module.TestClass
python -m unittest test_module.TestClass.test_method
@end example

You can pass in a list with any combination of module names, and fully
qualified class or method names.

Test modules can be specified by file path as well:

@example
python -m unittest tests/test_something.py
@end example

This allows you to use the shell filename completion to specify the test module.
The file specified must still be importable as a module. The path is converted
to a module name by removing the ‘.py’ and converting path separators into ‘.’.
If you want to execute a test file that isn’t importable as a module you should
execute the file directly instead.

You can run tests with more detail (higher verbosity) by passing in the -v flag:

@example
python -m unittest -v test_module
@end example

When executed without arguments @ref{380c,,Test Discovery} is started:

@example
python -m unittest
@end example

For a list of all the command-line options:

@example
python -m unittest -h
@end example


Changed in version 3.2: In earlier versions it was only possible to run individual test methods and
not modules or classes.


@menu
* Command-line options: Command-line options<3>. 

@end menu

@node Command-line options<3>,,,Command-Line Interface<3>
@anchor{library/unittest command-line-options}@anchor{380d}
@subsubsection Command-line options


@code{unittest} supports these command-line options:

@geindex unittest command line option; -b
@geindex unittest command line option; --buffer
@anchor{library/unittest cmdoption-unittest-b}@anchor{ec5}@anchor{library/unittest cmdoption-unittest-buffer}@anchor{380e}
@deffn {Program Option} @w{-}b, @w{-}@w{-}buffer

The standard output and standard error streams are buffered during the test
run. Output during a passing test is discarded. Output is echoed normally
on test fail or error and is added to the failure messages.
@end deffn

@geindex unittest command line option; -c
@geindex unittest command line option; --catch
@anchor{library/unittest cmdoption-unittest-c}@anchor{ec6}@anchor{library/unittest cmdoption-unittest-catch}@anchor{380f}
@deffn {Program Option} @w{-}c, @w{-}@w{-}catch

@code{Control-C} during the test run waits for the current test to end and then
reports all the results so far. A second @code{Control-C} raises the normal
@ref{4ca,,KeyboardInterrupt} exception.

See @ref{3810,,Signal Handling} for the functions that provide this functionality.
@end deffn

@geindex unittest command line option; -f
@geindex unittest command line option; --failfast
@anchor{library/unittest cmdoption-unittest-f}@anchor{ec8}@anchor{library/unittest cmdoption-unittest-failfast}@anchor{3811}
@deffn {Program Option} @w{-}f, @w{-}@w{-}failfast

Stop the test run on the first error or failure.
@end deffn

@geindex unittest command line option; -k
@anchor{library/unittest cmdoption-unittest-k}@anchor{3812}
@deffn {Program Option} @w{-}k

Only run test methods and classes that match the pattern or substring.
This option may be used multiple times, in which case all test cases that
match any of the given patterns are included.

Patterns that contain a wildcard character (@code{*}) are matched against the
test name using @ref{1107,,fnmatch.fnmatchcase()}; otherwise simple case-sensitive
substring matching is used.

Patterns are matched against the fully qualified test method name as
imported by the test loader.

For example, @code{-k foo} matches @code{foo_tests.SomeTest.test_something},
@code{bar_tests.SomeTest.test_foo}, but not @code{bar_tests.FooTest.test_something}.
@end deffn

@geindex unittest command line option; --locals
@anchor{library/unittest cmdoption-unittest-locals}@anchor{3813}
@deffn {Program Option} @w{-}@w{-}locals

Show local variables in tracebacks.
@end deffn


New in version 3.2: The command-line options @code{-b}, @code{-c} and @code{-f} were added.



New in version 3.5: The command-line option @code{--locals}.



New in version 3.7: The command-line option @code{-k}.


The command line can also be used for test discovery, for running all of the
tests in a project or just a subset.

@node Test Discovery,Organizing test code,Command-Line Interface<3>,unittest — Unit testing framework
@anchor{library/unittest test-discovery}@anchor{3814}@anchor{library/unittest unittest-test-discovery}@anchor{380c}
@subsubsection Test Discovery



New in version 3.2.


Unittest supports simple test discovery. In order to be compatible with test
discovery, all of the test files must be @ref{153d,,modules} or
@ref{154a,,packages} importable from the top-level directory of
the project (this means that their filenames must be valid @ref{170b,,identifiers}).

Test discovery is implemented in @ref{b6b,,TestLoader.discover()}, but can also be
used from the command line. The basic command-line usage is:

@example
cd project_directory
python -m unittest discover
@end example

@cartouche
@quotation Note 
As a shortcut, @code{python -m unittest} is the equivalent of
@code{python -m unittest discover}. If you want to pass arguments to test
discovery the @code{discover} sub-command must be used explicitly.
@end quotation
@end cartouche

The @code{discover} sub-command has the following options:

@geindex unittest-discover command line option; -v
@geindex unittest-discover command line option; --verbose
@anchor{library/unittest cmdoption-unittest-discover-v}@anchor{3815}@anchor{library/unittest cmdoption-unittest-discover-verbose}@anchor{3816}
@deffn {Program Option} @w{-}v, @w{-}@w{-}verbose

Verbose output
@end deffn

@geindex unittest-discover command line option; -s
@geindex unittest-discover command line option; --start-directory directory
@anchor{library/unittest cmdoption-unittest-discover-s}@anchor{3817}@anchor{library/unittest cmdoption-unittest-discover-start-directory}@anchor{3818}
@deffn {Program Option} @w{-}s, @w{-}@w{-}start@w{-}directory directory

Directory to start discovery (@code{.} default)
@end deffn

@geindex unittest-discover command line option; -p
@geindex unittest-discover command line option; --pattern pattern
@anchor{library/unittest cmdoption-unittest-discover-p}@anchor{3819}@anchor{library/unittest cmdoption-unittest-discover-pattern}@anchor{381a}
@deffn {Program Option} @w{-}p, @w{-}@w{-}pattern pattern

Pattern to match test files (@code{test*.py} default)
@end deffn

@geindex unittest-discover command line option; -t
@geindex unittest-discover command line option; --top-level-directory directory
@anchor{library/unittest cmdoption-unittest-discover-t}@anchor{381b}@anchor{library/unittest cmdoption-unittest-discover-top-level-directory}@anchor{381c}
@deffn {Program Option} @w{-}t, @w{-}@w{-}top@w{-}level@w{-}directory directory

Top level directory of project (defaults to start directory)
@end deffn

The @ref{3817,,-s}, @ref{3819,,-p}, and @ref{381b,,-t} options can be passed in
as positional arguments in that order. The following two command lines
are equivalent:

@example
python -m unittest discover -s project_directory -p "*_test.py"
python -m unittest discover project_directory "*_test.py"
@end example

As well as being a path it is possible to pass a package name, for example
@code{myproject.subpackage.test}, as the start directory. The package name you
supply will then be imported and its location on the filesystem will be used
as the start directory.

@cartouche
@quotation Caution 
Test discovery loads tests by importing them. Once test discovery has found
all the test files from the start directory you specify it turns the paths
into package names to import. For example @code{foo/bar/baz.py} will be
imported as @code{foo.bar.baz}.

If you have a package installed globally and attempt test discovery on
a different copy of the package then the import @emph{could} happen from the
wrong place. If this happens test discovery will warn you and exit.

If you supply the start directory as a package name rather than a
path to a directory then discover assumes that whichever location it
imports from is the location you intended, so you will not get the
warning.
@end quotation
@end cartouche

Test modules and packages can customize test loading and discovery by through
the @ref{381d,,load_tests protocol}.


Changed in version 3.4: Test discovery supports @ref{1840,,namespace packages}
for the start directory. Note that you need to specify the top level
directory too (e.g.
@code{python -m unittest discover -s root/namespace -t root}).



Changed in version 3.11: Python 3.11 dropped the @ref{1840,,namespace packages}
support. It has been broken since Python 3.7. Start directory and
subdirectories containing tests must be regular package that have
@code{__init__.py} file.

Directories containing start directory still can be a namespace package.
In this case, you need to specify start directory as dotted package name,
and target directory explicitly. For example:

@example
# proj/  <-- current directory
#   namespace/
#     mypkg/
#       __init__.py
#       test_mypkg.py

python -m unittest discover -s namespace.mypkg -t .
@end example


@node Organizing test code,Re-using old test code,Test Discovery,unittest — Unit testing framework
@anchor{library/unittest organizing-test-code}@anchor{381e}@anchor{library/unittest organizing-tests}@anchor{3809}
@subsubsection Organizing test code


The basic building blocks of unit testing are @emph{test cases} — single
scenarios that must be set up and checked for correctness.  In @ref{122,,unittest},
test cases are represented by @ref{1ab,,unittest.TestCase} instances.
To make your own test cases you must write subclasses of
@ref{1ab,,TestCase} or use @ref{381f,,FunctionTestCase}.

The testing code of a @ref{1ab,,TestCase} instance should be entirely self
contained, such that it can be run either in isolation or in arbitrary
combination with any number of other test cases.

The simplest @ref{1ab,,TestCase} subclass will simply implement a test method
(i.e. a method whose name starts with @code{test}) in order to perform specific
testing code:

@example
import unittest

class DefaultWidgetSizeTestCase(unittest.TestCase):
    def test_default_widget_size(self):
        widget = Widget('The widget')
        self.assertEqual(widget.size(), (50, 50))
@end example

Note that in order to test something, we use one of the @code{assert*()}
methods provided by the @ref{1ab,,TestCase} base class.  If the test fails, an
exception will be raised with an explanatory message, and @ref{122,,unittest}
will identify the test case as a @emph{failure}.  Any other exceptions will be
treated as @emph{errors}.

Tests can be numerous, and their set-up can be repetitive.  Luckily, we
can factor out set-up code by implementing a method called
@ref{ecd,,setUp()}, which the testing framework will automatically
call for every single test we run:

@example
import unittest

class WidgetTestCase(unittest.TestCase):
    def setUp(self):
        self.widget = Widget('The widget')

    def test_default_widget_size(self):
        self.assertEqual(self.widget.size(), (50,50),
                         'incorrect default size')

    def test_widget_resize(self):
        self.widget.resize(100,150)
        self.assertEqual(self.widget.size(), (100,150),
                         'wrong size after resize')
@end example

@cartouche
@quotation Note 
The order in which the various tests will be run is determined
by sorting the test method names with respect to the built-in
ordering for strings.
@end quotation
@end cartouche

If the @ref{ecd,,setUp()} method raises an exception while the test is
running, the framework will consider the test to have suffered an error, and
the test method will not be executed.

Similarly, we can provide a @ref{ece,,tearDown()} method that tidies up
after the test method has been run:

@example
import unittest

class WidgetTestCase(unittest.TestCase):
    def setUp(self):
        self.widget = Widget('The widget')

    def tearDown(self):
        self.widget.dispose()
@end example

If @ref{ecd,,setUp()} succeeded, @ref{ece,,tearDown()} will be
run whether the test method succeeded or not.

Such a working environment for the testing code is called a
@emph{test fixture}.  A new TestCase instance is created as a unique
test fixture used to execute each individual test method.  Thus
@ref{ecd,,setUp()}, @ref{ece,,tearDown()}, and @code{__init__()}
will be called once per test.

It is recommended that you use TestCase implementations to group tests together
according to the features they test.  @ref{122,,unittest} provides a mechanism for
this: the @emph{test suite}, represented by @ref{122,,unittest}’s
@ref{953,,TestSuite} class.  In most cases, calling @ref{b69,,unittest.main()} will do
the right thing and collect all the module’s test cases for you and execute
them.

However, should you want to customize the building of your test suite,
you can do it yourself:

@example
def suite():
    suite = unittest.TestSuite()
    suite.addTest(WidgetTestCase('test_default_widget_size'))
    suite.addTest(WidgetTestCase('test_widget_resize'))
    return suite

if __name__ == '__main__':
    runner = unittest.TextTestRunner()
    runner.run(suite())
@end example

You can place the definitions of test cases and test suites in the same modules
as the code they are to test (such as @code{widget.py}), but there are several
advantages to placing the test code in a separate module, such as
@code{test_widget.py}:


@itemize *

@item 
The test module can be run standalone from the command line.

@item 
The test code can more easily be separated from shipped code.

@item 
There is less temptation to change test code to fit the code it tests without
a good reason.

@item 
Test code should be modified much less frequently than the code it tests.

@item 
Tested code can be refactored more easily.

@item 
Tests for modules written in C must be in separate modules anyway, so why not
be consistent?

@item 
If the testing strategy changes, there is no need to change the source code.
@end itemize

@node Re-using old test code,Skipping tests and expected failures,Organizing test code,unittest — Unit testing framework
@anchor{library/unittest legacy-unit-tests}@anchor{3820}@anchor{library/unittest re-using-old-test-code}@anchor{3821}
@subsubsection Re-using old test code


Some users will find that they have existing test code that they would like to
run from @ref{122,,unittest}, without converting every old test function to a
@ref{1ab,,TestCase} subclass.

For this reason, @ref{122,,unittest} provides a @ref{381f,,FunctionTestCase} class.
This subclass of @ref{1ab,,TestCase} can be used to wrap an existing test
function.  Set-up and tear-down functions can also be provided.

Given the following test function:

@example
def testSomething():
    something = makeSomething()
    assert something.name is not None
    # ...
@end example

one can create an equivalent test case instance as follows, with optional
set-up and tear-down methods:

@example
testcase = unittest.FunctionTestCase(testSomething,
                                     setUp=makeSomethingDB,
                                     tearDown=deleteSomethingDB)
@end example

@cartouche
@quotation Note 
Even though @ref{381f,,FunctionTestCase} can be used to quickly convert an
existing test base over to a @ref{122,,unittest}-based system, this approach is
not recommended.  Taking the time to set up proper @ref{1ab,,TestCase}
subclasses will make future test refactorings infinitely easier.
@end quotation
@end cartouche

In some cases, the existing tests may have been written using the @ref{63,,doctest}
module.  If so, @ref{63,,doctest} provides a @code{DocTestSuite} class that can
automatically build @ref{953,,unittest.TestSuite} instances from the existing
@ref{63,,doctest}-based tests.

@node Skipping tests and expected failures,Distinguishing test iterations using subtests,Re-using old test code,unittest — Unit testing framework
@anchor{library/unittest skipping-tests-and-expected-failures}@anchor{3822}@anchor{library/unittest unittest-skipping}@anchor{3823}
@subsubsection Skipping tests and expected failures



New in version 3.1.


Unittest supports skipping individual test methods and even whole classes of
tests.  In addition, it supports marking a test as an “expected failure,” a test
that is broken and will fail, but shouldn’t be counted as a failure on a
@ref{d04,,TestResult}.

Skipping a test is simply a matter of using the @ref{127b,,skip()} @ref{1bf,,decorator}
or one of its conditional variants, calling @ref{3824,,TestCase.skipTest()} within a
@ref{ecd,,setUp()} or test method, or raising @ref{b6a,,SkipTest} directly.

Basic skipping looks like this:

@example
class MyTestCase(unittest.TestCase):

    @@unittest.skip("demonstrating skipping")
    def test_nothing(self):
        self.fail("shouldn't happen")

    @@unittest.skipIf(mylib.__version__ < (1, 3),
                     "not supported in this library version")
    def test_format(self):
        # Tests that work for only a certain version of the library.
        pass

    @@unittest.skipUnless(sys.platform.startswith("win"), "requires Windows")
    def test_windows_support(self):
        # windows specific testing code
        pass

    def test_maybe_skipped(self):
        if not external_resource_available():
            self.skipTest("external resource not available")
        # test code that depends on the external resource
        pass
@end example

This is the output of running the example above in verbose mode:

@example
test_format (__main__.MyTestCase) ... skipped 'not supported in this library version'
test_nothing (__main__.MyTestCase) ... skipped 'demonstrating skipping'
test_maybe_skipped (__main__.MyTestCase) ... skipped 'external resource not available'
test_windows_support (__main__.MyTestCase) ... skipped 'requires Windows'

----------------------------------------------------------------------
Ran 4 tests in 0.005s

OK (skipped=4)
@end example

Classes can be skipped just like methods:

@example
@@unittest.skip("showing class skipping")
class MySkippedTestCase(unittest.TestCase):
    def test_not_run(self):
        pass
@end example

@ref{ecd,,TestCase.setUp()} can also skip the test.  This is useful when a resource
that needs to be set up is not available.

Expected failures use the @ref{3825,,expectedFailure()} decorator.

@example
class ExpectedFailureTestCase(unittest.TestCase):
    @@unittest.expectedFailure
    def test_fail(self):
        self.assertEqual(1, 0, "broken")
@end example

It’s easy to roll your own skipping decorators by making a decorator that calls
@ref{127b,,skip()} on the test when it wants it to be skipped.  This decorator skips
the test unless the passed object has a certain attribute:

@example
def skipUnlessHasattr(obj, attr):
    if hasattr(obj, attr):
        return lambda func: func
    return unittest.skip("@{!r@} doesn't have @{!r@}".format(obj, attr))
@end example

The following decorators and exception implement test skipping and expected failures:

@geindex skip() (in module unittest)
@anchor{library/unittest unittest skip}@anchor{127b}
@deffn {Function} @@unittest.skip (reason)

Unconditionally skip the decorated test.  @emph{reason} should describe why the
test is being skipped.
@end deffn

@geindex skipIf() (in module unittest)
@anchor{library/unittest unittest skipIf}@anchor{3826}
@deffn {Function} @@unittest.skipIf (condition, reason)

Skip the decorated test if @emph{condition} is true.
@end deffn

@geindex skipUnless() (in module unittest)
@anchor{library/unittest unittest skipUnless}@anchor{3827}
@deffn {Function} @@unittest.skipUnless (condition, reason)

Skip the decorated test unless @emph{condition} is true.
@end deffn

@geindex expectedFailure() (in module unittest)
@anchor{library/unittest unittest expectedFailure}@anchor{3825}
@deffn {Function} @@unittest.expectedFailure

Mark the test as an expected failure or error.  If the test fails or errors
in the test function itself (rather than in one of the @emph{test fixture}
methods) then it will be considered a success.  If the test passes, it will
be considered a failure.
@end deffn

@geindex SkipTest
@anchor{library/unittest unittest SkipTest}@anchor{b6a}
@deffn {Exception} unittest.SkipTest (reason)

This exception is raised to skip a test.

Usually you can use @ref{3824,,TestCase.skipTest()} or one of the skipping
decorators instead of raising this directly.
@end deffn

Skipped tests will not have @ref{ecd,,setUp()} or @ref{ece,,tearDown()} run around them.
Skipped classes will not have @ref{564,,setUpClass()} or @ref{ecb,,tearDownClass()} run.
Skipped modules will not have @code{setUpModule()} or @code{tearDownModule()} run.

@node Distinguishing test iterations using subtests,Classes and functions,Skipping tests and expected failures,unittest — Unit testing framework
@anchor{library/unittest distinguishing-test-iterations-using-subtests}@anchor{3828}@anchor{library/unittest subtests}@anchor{b68}
@subsubsection Distinguishing test iterations using subtests



New in version 3.4.


When there are very small differences among your tests, for
instance some parameters, unittest allows you to distinguish them inside
the body of a test method using the @ref{b67,,subTest()} context manager.

For example, the following test:

@example
class NumbersTest(unittest.TestCase):

    def test_even(self):
        """
        Test that numbers between 0 and 5 are all even.
        """
        for i in range(0, 6):
            with self.subTest(i=i):
                self.assertEqual(i % 2, 0)
@end example

will produce the following output:

@example
======================================================================
FAIL: test_even (__main__.NumbersTest) (i=1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "subtests.py", line 32, in test_even
    self.assertEqual(i % 2, 0)
AssertionError: 1 != 0

======================================================================
FAIL: test_even (__main__.NumbersTest) (i=3)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "subtests.py", line 32, in test_even
    self.assertEqual(i % 2, 0)
AssertionError: 1 != 0

======================================================================
FAIL: test_even (__main__.NumbersTest) (i=5)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "subtests.py", line 32, in test_even
    self.assertEqual(i % 2, 0)
AssertionError: 1 != 0
@end example

Without using a subtest, execution would stop after the first failure,
and the error would be less easy to diagnose because the value of @code{i}
wouldn’t be displayed:

@example
======================================================================
FAIL: test_even (__main__.NumbersTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "subtests.py", line 32, in test_even
    self.assertEqual(i % 2, 0)
AssertionError: 1 != 0
@end example

@node Classes and functions,Class and Module Fixtures,Distinguishing test iterations using subtests,unittest — Unit testing framework
@anchor{library/unittest classes-and-functions}@anchor{3829}@anchor{library/unittest unittest-contents}@anchor{382a}
@subsubsection Classes and functions


This section describes in depth the API of @ref{122,,unittest}.

@menu
* Test cases:: 
* Grouping tests:: 
* Loading and running tests:: 

@end menu

@node Test cases,Grouping tests,,Classes and functions
@anchor{library/unittest test-cases}@anchor{382b}@anchor{library/unittest testcase-objects}@anchor{382c}
@subsubsection Test cases


@geindex TestCase (class in unittest)
@anchor{library/unittest unittest TestCase}@anchor{1ab}
@deffn {Class} unittest.TestCase (methodName='runTest')

Instances of the @ref{1ab,,TestCase} class represent the logical test units
in the @ref{122,,unittest} universe.  This class is intended to be used as a base
class, with specific tests being implemented by concrete subclasses.  This class
implements the interface needed by the test runner to allow it to drive the
tests, and methods that the test code can use to check for and report various
kinds of failure.

Each instance of @ref{1ab,,TestCase} will run a single base method: the method
named @emph{methodName}.
In most uses of @ref{1ab,,TestCase}, you will neither change
the @emph{methodName} nor reimplement the default @code{runTest()} method.


Changed in version 3.2: @ref{1ab,,TestCase} can be instantiated successfully without providing a
@emph{methodName}. This makes it easier to experiment with @ref{1ab,,TestCase}
from the interactive interpreter.


@ref{1ab,,TestCase} instances provide three groups of methods: one group used
to run the test, another used by the test implementation to check conditions
and report failures, and some inquiry methods allowing information about the
test itself to be gathered.

Methods in the first group (running the test) are:

@geindex setUp() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase setUp}@anchor{ecd}
@deffn {Method} setUp ()

Method called to prepare the test fixture.  This is called immediately
before calling the test method; other than @ref{46e,,AssertionError} or @ref{b6a,,SkipTest},
any exception raised by this method will be considered an error rather than
a test failure. The default implementation does nothing.
@end deffn

@geindex tearDown() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase tearDown}@anchor{ece}
@deffn {Method} tearDown ()

Method called immediately after the test method has been called and the
result recorded.  This is called even if the test method raised an
exception, so the implementation in subclasses may need to be particularly
careful about checking internal state.  Any exception, other than
@ref{46e,,AssertionError} or @ref{b6a,,SkipTest}, raised by this method will be
considered an additional error rather than a test failure (thus increasing
the total number of reported errors). This method will only be called if
the @ref{ecd,,setUp()} succeeds, regardless of the outcome of the test method.
The default implementation does nothing.
@end deffn

@geindex setUpClass() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase setUpClass}@anchor{564}
@deffn {Method} setUpClass ()

A class method called before tests in an individual class are run.
@code{setUpClass} is called with the class as the only argument
and must be decorated as a @ref{254,,classmethod()}:

@example
@@classmethod
def setUpClass(cls):
    ...
@end example

See @ref{382d,,Class and Module Fixtures} for more details.


New in version 3.2.

@end deffn

@geindex tearDownClass() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase tearDownClass}@anchor{ecb}
@deffn {Method} tearDownClass ()

A class method called after tests in an individual class have run.
@code{tearDownClass} is called with the class as the only argument
and must be decorated as a @ref{254,,classmethod()}:

@example
@@classmethod
def tearDownClass(cls):
    ...
@end example

See @ref{382d,,Class and Module Fixtures} for more details.


New in version 3.2.

@end deffn

@geindex run() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase run}@anchor{d03}
@deffn {Method} run (result=None)

Run the test, collecting the result into the @ref{d04,,TestResult} object
passed as @emph{result}.  If @emph{result} is omitted or @code{None}, a temporary
result object is created (by calling the @ref{382e,,defaultTestResult()}
method) and used. The result object is returned to @ref{d03,,run()}’s
caller.

The same effect may be had by simply calling the @ref{1ab,,TestCase}
instance.


Changed in version 3.3: Previous versions of @code{run} did not return the result. Neither did
calling an instance.

@end deffn

@geindex skipTest() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase skipTest}@anchor{3824}
@deffn {Method} skipTest (reason)

Calling this during a test method or @ref{ecd,,setUp()} skips the current
test.  See @ref{3823,,Skipping tests and expected failures} for more information.


New in version 3.1.

@end deffn

@geindex subTest() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase subTest}@anchor{b67}
@deffn {Method} subTest (msg=None, **params)

Return a context manager which executes the enclosed code block as a
subtest.  @emph{msg} and @emph{params} are optional, arbitrary values which are
displayed whenever a subtest fails, allowing you to identify them
clearly.

A test case can contain any number of subtest declarations, and
they can be arbitrarily nested.

See @ref{b68,,Distinguishing test iterations using subtests} for more information.


New in version 3.4.

@end deffn

@geindex debug() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase debug}@anchor{10f9}
@deffn {Method} debug ()

Run the test without collecting the result.  This allows exceptions raised
by the test to be propagated to the caller, and can be used to support
running tests under a debugger.
@end deffn
@anchor{library/unittest assert-methods}@anchor{3806}
The @ref{1ab,,TestCase} class provides several assert methods to check for and
report failures.  The following table lists the most commonly used methods
(see the tables below for more assert methods):


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxx} 
@headitem

Method

@tab

Checks that

@tab

New in

@item

@ref{dd5,,assertEqual(a@comma{} b)}

@tab

@code{a == b}

@tab

@item

@ref{dd6,,assertNotEqual(a@comma{} b)}

@tab

@code{a != b}

@tab

@item

@ref{dd4,,assertTrue(x)}

@tab

@code{bool(x) is True}

@tab

@item

@ref{ec9,,assertFalse(x)}

@tab

@code{bool(x) is False}

@tab

@item

@ref{ed1,,assertIs(a@comma{} b)}

@tab

@code{a is b}

@tab

3.1

@item

@ref{ed2,,assertIsNot(a@comma{} b)}

@tab

@code{a is not b}

@tab

3.1

@item

@ref{ecf,,assertIsNone(x)}

@tab

@code{x is None}

@tab

3.1

@item

@ref{ed0,,assertIsNotNone(x)}

@tab

@code{x is not None}

@tab

3.1

@item

@ref{eda,,assertIn(a@comma{} b)}

@tab

@code{a in b}

@tab

3.1

@item

@ref{edb,,assertNotIn(a@comma{} b)}

@tab

@code{a not in b}

@tab

3.1

@item

@ref{ed3,,assertIsInstance(a@comma{} b)}

@tab

@code{isinstance(a, b)}

@tab

3.2

@item

@ref{ed4,,assertNotIsInstance(a@comma{} b)}

@tab

@code{not isinstance(a, b)}

@tab

3.2

@end multitable


All the assert methods accept a @emph{msg} argument that, if specified, is used
as the error message on failure (see also @ref{eca,,longMessage}).
Note that the @emph{msg} keyword argument can be passed to @ref{cff,,assertRaises()},
@ref{d00,,assertRaisesRegex()}, @ref{d01,,assertWarns()}, @ref{d02,,assertWarnsRegex()}
only when they are used as a context manager.

@geindex assertEqual() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertEqual}@anchor{dd5}
@deffn {Method} assertEqual (first, second, msg=None)

Test that @emph{first} and @emph{second} are equal.  If the values do not
compare equal, the test will fail.

In addition, if @emph{first} and @emph{second} are the exact same type and one of
list, tuple, dict, set, frozenset or str or any type that a subclass
registers with @ref{ee3,,addTypeEqualityFunc()} the type-specific equality
function will be called in order to generate a more useful default
error message (see also the @ref{382f,,list of type-specific methods}).


Changed in version 3.1: Added the automatic calling of type-specific equality function.



Changed in version 3.2: @ref{ed9,,assertMultiLineEqual()} added as the default type equality
function for comparing strings.

@end deffn

@geindex assertNotEqual() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertNotEqual}@anchor{dd6}
@deffn {Method} assertNotEqual (first, second, msg=None)

Test that @emph{first} and @emph{second} are not equal.  If the values do
compare equal, the test will fail.
@end deffn

@geindex assertTrue() (unittest.TestCase method)
@geindex assertFalse() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertTrue}@anchor{dd4}
@deffn {Method} assertTrue (expr, msg=None)
@anchor{library/unittest unittest TestCase assertFalse}@anchor{ec9}
@deffnx {Method} assertFalse (expr, msg=None)

Test that @emph{expr} is true (or false).

Note that this is equivalent to @code{bool(expr) is True} and not to @code{expr
is True} (use @code{assertIs(expr, True)} for the latter).  This method
should also be avoided when more specific methods are available (e.g.
@code{assertEqual(a, b)} instead of @code{assertTrue(a == b)}), because they
provide a better error message in case of failure.
@end deffn

@geindex assertIs() (unittest.TestCase method)
@geindex assertIsNot() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertIs}@anchor{ed1}
@deffn {Method} assertIs (first, second, msg=None)
@anchor{library/unittest unittest TestCase assertIsNot}@anchor{ed2}
@deffnx {Method} assertIsNot (first, second, msg=None)

Test that @emph{first} and @emph{second} are (or are not) the same object.


New in version 3.1.

@end deffn

@geindex assertIsNone() (unittest.TestCase method)
@geindex assertIsNotNone() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertIsNone}@anchor{ecf}
@deffn {Method} assertIsNone (expr, msg=None)
@anchor{library/unittest unittest TestCase assertIsNotNone}@anchor{ed0}
@deffnx {Method} assertIsNotNone (expr, msg=None)

Test that @emph{expr} is (or is not) @code{None}.


New in version 3.1.

@end deffn

@geindex assertIn() (unittest.TestCase method)
@geindex assertNotIn() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertIn}@anchor{eda}
@deffn {Method} assertIn (member, container, msg=None)
@anchor{library/unittest unittest TestCase assertNotIn}@anchor{edb}
@deffnx {Method} assertNotIn (member, container, msg=None)

Test that @emph{member} is (or is not) in @emph{container}.


New in version 3.1.

@end deffn

@geindex assertIsInstance() (unittest.TestCase method)
@geindex assertNotIsInstance() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertIsInstance}@anchor{ed3}
@deffn {Method} assertIsInstance (obj, cls, msg=None)
@anchor{library/unittest unittest TestCase assertNotIsInstance}@anchor{ed4}
@deffnx {Method} assertNotIsInstance (obj, cls, msg=None)

Test that @emph{obj} is (or is not) an instance of @emph{cls} (which can be a
class or a tuple of classes, as supported by @ref{236,,isinstance()}).
To check for the exact type, use @ref{ed1,,assertIs(type(obj)@comma{} cls)}.


New in version 3.2.

@end deffn

It is also possible to check the production of exceptions, warnings, and
log messages using the following methods:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxx} 
@headitem

Method

@tab

Checks that

@tab

New in

@item

@ref{cff,,assertRaises(exc@comma{} fun@comma{} *args@comma{} **kwds)}

@tab

@code{fun(*args, **kwds)} raises @emph{exc}

@tab

@item

@ref{d00,,assertRaisesRegex(exc@comma{} r@comma{} fun@comma{} *args@comma{} **kwds)}

@tab

@code{fun(*args, **kwds)} raises @emph{exc}
and the message matches regex @emph{r}

@tab

3.1

@item

@ref{d01,,assertWarns(warn@comma{} fun@comma{} *args@comma{} **kwds)}

@tab

@code{fun(*args, **kwds)} raises @emph{warn}

@tab

3.2

@item

@ref{d02,,assertWarnsRegex(warn@comma{} r@comma{} fun@comma{} *args@comma{} **kwds)}

@tab

@code{fun(*args, **kwds)} raises @emph{warn}
and the message matches regex @emph{r}

@tab

3.2

@item

@ref{2e1,,assertLogs(logger@comma{} level)}

@tab

The @code{with} block logs on @emph{logger}
with minimum @emph{level}

@tab

3.4

@item

@ref{2e0,,assertNoLogs(logger@comma{} level)}

@tab


@table @asis

@item The @code{with} block does not log on

@emph{logger} with minimum @emph{level}
@end table

@tab

3.10

@end multitable


@geindex assertRaises() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertRaises}@anchor{cff}
@deffn {Method} assertRaises (exception, callable, *args, **kwds)

@deffnx {Method} assertRaises (exception, *, msg=None)

Test that an exception is raised when @emph{callable} is called with any
positional or keyword arguments that are also passed to
@ref{cff,,assertRaises()}.  The test passes if @emph{exception} is raised, is an
error if another exception is raised, or fails if no exception is raised.
To catch any of a group of exceptions, a tuple containing the exception
classes may be passed as @emph{exception}.

If only the @emph{exception} and possibly the @emph{msg} arguments are given,
return a context manager so that the code under test can be written
inline rather than as a function:

@example
with self.assertRaises(SomeException):
    do_something()
@end example

When used as a context manager, @ref{cff,,assertRaises()} accepts the
additional keyword argument @emph{msg}.

The context manager will store the caught exception object in its
@code{exception} attribute.  This can be useful if the intention
is to perform additional checks on the exception raised:

@example
with self.assertRaises(SomeException) as cm:
    do_something()

the_exception = cm.exception
self.assertEqual(the_exception.error_code, 3)
@end example


Changed in version 3.1: Added the ability to use @ref{cff,,assertRaises()} as a context manager.



Changed in version 3.2: Added the @code{exception} attribute.



Changed in version 3.3: Added the @emph{msg} keyword argument when used as a context manager.

@end deffn

@geindex assertRaisesRegex() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertRaisesRegex}@anchor{d00}
@deffn {Method} assertRaisesRegex (exception, regex, callable, *args, **kwds)

@deffnx {Method} assertRaisesRegex (exception, regex, *, msg=None)

Like @ref{cff,,assertRaises()} but also tests that @emph{regex} matches
on the string representation of the raised exception.  @emph{regex} may be
a regular expression object or a string containing a regular expression
suitable for use by @ref{dd2,,re.search()}.  Examples:

@example
self.assertRaisesRegex(ValueError, "invalid literal for.*XYZ'$",
                       int, 'XYZ')
@end example

or:

@example
with self.assertRaisesRegex(ValueError, 'literal'):
   int('XYZ')
@end example


New in version 3.1: Added under the name @code{assertRaisesRegexp}.



Changed in version 3.2: Renamed to @ref{d00,,assertRaisesRegex()}.



Changed in version 3.3: Added the @emph{msg} keyword argument when used as a context manager.

@end deffn

@geindex assertWarns() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertWarns}@anchor{d01}
@deffn {Method} assertWarns (warning, callable, *args, **kwds)

@deffnx {Method} assertWarns (warning, *, msg=None)

Test that a warning is triggered when @emph{callable} is called with any
positional or keyword arguments that are also passed to
@ref{d01,,assertWarns()}.  The test passes if @emph{warning} is triggered and
fails if it isn’t.  Any exception is an error.
To catch any of a group of warnings, a tuple containing the warning
classes may be passed as @emph{warnings}.

If only the @emph{warning} and possibly the @emph{msg} arguments are given,
return a context manager so that the code under test can be written
inline rather than as a function:

@example
with self.assertWarns(SomeWarning):
    do_something()
@end example

When used as a context manager, @ref{d01,,assertWarns()} accepts the
additional keyword argument @emph{msg}.

The context manager will store the caught warning object in its
@code{warning} attribute, and the source line which triggered the
warnings in the @code{filename} and @code{lineno} attributes.
This can be useful if the intention is to perform additional checks
on the warning caught:

@example
with self.assertWarns(SomeWarning) as cm:
    do_something()

self.assertIn('myfile.py', cm.filename)
self.assertEqual(320, cm.lineno)
@end example

This method works regardless of the warning filters in place when it
is called.


New in version 3.2.



Changed in version 3.3: Added the @emph{msg} keyword argument when used as a context manager.

@end deffn

@geindex assertWarnsRegex() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertWarnsRegex}@anchor{d02}
@deffn {Method} assertWarnsRegex (warning, regex, callable, *args, **kwds)

@deffnx {Method} assertWarnsRegex (warning, regex, *, msg=None)

Like @ref{d01,,assertWarns()} but also tests that @emph{regex} matches on the
message of the triggered warning.  @emph{regex} may be a regular expression
object or a string containing a regular expression suitable for use
by @ref{dd2,,re.search()}.  Example:

@example
self.assertWarnsRegex(DeprecationWarning,
                      r'legacy_function\(\) is deprecated',
                      legacy_function, 'XYZ')
@end example

or:

@example
with self.assertWarnsRegex(RuntimeWarning, 'unsafe frobnicating'):
    frobnicate('/etc/passwd')
@end example


New in version 3.2.



Changed in version 3.3: Added the @emph{msg} keyword argument when used as a context manager.

@end deffn

@geindex assertLogs() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertLogs}@anchor{2e1}
@deffn {Method} assertLogs (logger=None, level=None)

A context manager to test that at least one message is logged on
the @emph{logger} or one of its children, with at least the given
@emph{level}.

If given, @emph{logger} should be a @ref{676,,logging.Logger} object or a
@ref{2ef,,str} giving the name of a logger.  The default is the root
logger, which will catch all messages that were not blocked by a
non-propagating descendent logger.

If given, @emph{level} should be either a numeric logging level or
its string equivalent (for example either @code{"ERROR"} or
@code{logging.ERROR}).  The default is @code{logging.INFO}.

The test passes if at least one message emitted inside the @code{with}
block matches the @emph{logger} and @emph{level} conditions, otherwise it fails.

The object returned by the context manager is a recording helper
which keeps tracks of the matching log messages.  It has two
attributes:

@geindex records (unittest.TestCase attribute)
@anchor{library/unittest unittest TestCase records}@anchor{3830}
@deffn {Attribute} records

A list of @ref{b6c,,logging.LogRecord} objects of the matching
log messages.
@end deffn

@geindex output (unittest.TestCase attribute)
@anchor{library/unittest unittest TestCase output}@anchor{3831}
@deffn {Attribute} output

A list of @ref{2ef,,str} objects with the formatted output of
matching messages.
@end deffn

Example:

@example
with self.assertLogs('foo', level='INFO') as cm:
   logging.getLogger('foo').info('first message')
   logging.getLogger('foo.bar').error('second message')
self.assertEqual(cm.output, ['INFO:foo:first message',
                             'ERROR:foo.bar:second message'])
@end example


New in version 3.4.

@end deffn

@geindex assertNoLogs() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertNoLogs}@anchor{2e0}
@deffn {Method} assertNoLogs (logger=None, level=None)

A context manager to test that no messages are logged on
the @emph{logger} or one of its children, with at least the given
@emph{level}.

If given, @emph{logger} should be a @ref{676,,logging.Logger} object or a
@ref{2ef,,str} giving the name of a logger.  The default is the root
logger, which will catch all messages.

If given, @emph{level} should be either a numeric logging level or
its string equivalent (for example either @code{"ERROR"} or
@code{logging.ERROR}).  The default is @code{logging.INFO}.

Unlike @ref{2e1,,assertLogs()}, nothing will be returned by the context
manager.


New in version 3.10.

@end deffn

There are also other methods used to perform more specific checks, such as:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxx} 
@headitem

Method

@tab

Checks that

@tab

New in

@item

@ref{dd7,,assertAlmostEqual(a@comma{} b)}

@tab

@code{round(a-b, 7) == 0}

@tab

@item

@ref{dd8,,assertNotAlmostEqual(a@comma{} b)}

@tab

@code{round(a-b, 7) != 0}

@tab

@item

@ref{ed5,,assertGreater(a@comma{} b)}

@tab

@code{a > b}

@tab

3.1

@item

@ref{ed6,,assertGreaterEqual(a@comma{} b)}

@tab

@code{a >= b}

@tab

3.1

@item

@ref{ed7,,assertLess(a@comma{} b)}

@tab

@code{a < b}

@tab

3.1

@item

@ref{ed8,,assertLessEqual(a@comma{} b)}

@tab

@code{a <= b}

@tab

3.1

@item

@ref{dd1,,assertRegex(s@comma{} r)}

@tab

@code{r.search(s)}

@tab

3.1

@item

@ref{3832,,assertNotRegex(s@comma{} r)}

@tab

@code{not r.search(s)}

@tab

3.2

@item

@ref{dcf,,assertCountEqual(a@comma{} b)}

@tab

@emph{a} and @emph{b} have the same
elements in the same number,
regardless of their order.

@tab

3.2

@end multitable


@geindex assertAlmostEqual() (unittest.TestCase method)
@geindex assertNotAlmostEqual() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertAlmostEqual}@anchor{dd7}
@deffn {Method} assertAlmostEqual (first, second, places=7, msg=None, delta=None)
@anchor{library/unittest unittest TestCase assertNotAlmostEqual}@anchor{dd8}
@deffnx {Method} assertNotAlmostEqual (first, second, places=7, msg=None, delta=None)

Test that @emph{first} and @emph{second} are approximately (or not approximately)
equal by computing the difference, rounding to the given number of
decimal @emph{places} (default 7), and comparing to zero.  Note that these
methods round the values to the given number of @emph{decimal places} (i.e.
like the @ref{e75,,round()} function) and not @emph{significant digits}.

If @emph{delta} is supplied instead of @emph{places} then the difference
between @emph{first} and @emph{second} must be less or equal to (or greater than) @emph{delta}.

Supplying both @emph{delta} and @emph{places} raises a @ref{162,,TypeError}.


Changed in version 3.2: @ref{dd7,,assertAlmostEqual()} automatically considers almost equal objects
that compare equal.  @ref{dd8,,assertNotAlmostEqual()} automatically fails
if the objects compare equal.  Added the @emph{delta} keyword argument.

@end deffn

@geindex assertGreater() (unittest.TestCase method)
@geindex assertGreaterEqual() (unittest.TestCase method)
@geindex assertLess() (unittest.TestCase method)
@geindex assertLessEqual() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertGreater}@anchor{ed5}
@deffn {Method} assertGreater (first, second, msg=None)
@anchor{library/unittest unittest TestCase assertGreaterEqual}@anchor{ed6}
@deffnx {Method} assertGreaterEqual (first, second, msg=None)
@anchor{library/unittest unittest TestCase assertLess}@anchor{ed7}
@deffnx {Method} assertLess (first, second, msg=None)
@anchor{library/unittest unittest TestCase assertLessEqual}@anchor{ed8}
@deffnx {Method} assertLessEqual (first, second, msg=None)

Test that @emph{first} is respectively >, >=, < or <= than @emph{second} depending
on the method name.  If not, the test will fail:

@example
>>> self.assertGreaterEqual(3, 4)
AssertionError: "3" unexpectedly not greater than or equal to "4"
@end example


New in version 3.1.

@end deffn

@geindex assertRegex() (unittest.TestCase method)
@geindex assertNotRegex() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertRegex}@anchor{dd1}
@deffn {Method} assertRegex (text, regex, msg=None)
@anchor{library/unittest unittest TestCase assertNotRegex}@anchor{3832}
@deffnx {Method} assertNotRegex (text, regex, msg=None)

Test that a @emph{regex} search matches (or does not match) @emph{text}.  In case
of failure, the error message will include the pattern and the @emph{text} (or
the pattern and the part of @emph{text} that unexpectedly matched).  @emph{regex}
may be a regular expression object or a string containing a regular
expression suitable for use by @ref{dd2,,re.search()}.


New in version 3.1: Added under the name @code{assertRegexpMatches}.



Changed in version 3.2: The method @code{assertRegexpMatches()} has been renamed to
@ref{dd1,,assertRegex()}.



New in version 3.2: @ref{3832,,assertNotRegex()}.



New in version 3.5: The name @code{assertNotRegexpMatches} is a deprecated alias
for @ref{3832,,assertNotRegex()}.

@end deffn

@geindex assertCountEqual() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertCountEqual}@anchor{dcf}
@deffn {Method} assertCountEqual (first, second, msg=None)

Test that sequence @emph{first} contains the same elements as @emph{second},
regardless of their order. When they don’t, an error message listing the
differences between the sequences will be generated.

Duplicate elements are @emph{not} ignored when comparing @emph{first} and
@emph{second}. It verifies whether each element has the same count in both
sequences. Equivalent to:
@code{assertEqual(Counter(list(first)), Counter(list(second)))}
but works with sequences of unhashable objects as well.


New in version 3.2.

@end deffn
@anchor{library/unittest type-specific-methods}@anchor{382f}
The @ref{dd5,,assertEqual()} method dispatches the equality check for objects of
the same type to different type-specific methods.  These methods are already
implemented for most of the built-in types, but it’s also possible to
register new methods using @ref{ee3,,addTypeEqualityFunc()}:

@geindex addTypeEqualityFunc() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase addTypeEqualityFunc}@anchor{ee3}
@deffn {Method} addTypeEqualityFunc (typeobj, function)

Registers a type-specific method called by @ref{dd5,,assertEqual()} to check
if two objects of exactly the same @emph{typeobj} (not subclasses) compare
equal.  @emph{function} must take two positional arguments and a third msg=None
keyword argument just as @ref{dd5,,assertEqual()} does.  It must raise
@ref{3833,,self.failureException(msg)} when inequality
between the first two parameters is detected – possibly providing useful
information and explaining the inequalities in details in the error
message.


New in version 3.1.

@end deffn

The list of type-specific methods automatically used by
@ref{dd5,,assertEqual()} are summarized in the following table.  Note
that it’s usually not necessary to invoke these methods directly.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxx} 
@headitem

Method

@tab

Used to compare

@tab

New in

@item

@ref{ed9,,assertMultiLineEqual(a@comma{} b)}

@tab

strings

@tab

3.1

@item

@ref{edf,,assertSequenceEqual(a@comma{} b)}

@tab

sequences

@tab

3.1

@item

@ref{edd,,assertListEqual(a@comma{} b)}

@tab

lists

@tab

3.1

@item

@ref{ede,,assertTupleEqual(a@comma{} b)}

@tab

tuples

@tab

3.1

@item

@ref{edc,,assertSetEqual(a@comma{} b)}

@tab

sets or frozensets

@tab

3.1

@item

@ref{ee0,,assertDictEqual(a@comma{} b)}

@tab

dicts

@tab

3.1

@end multitable


@geindex assertMultiLineEqual() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertMultiLineEqual}@anchor{ed9}
@deffn {Method} assertMultiLineEqual (first, second, msg=None)

Test that the multiline string @emph{first} is equal to the string @emph{second}.
When not equal a diff of the two strings highlighting the differences
will be included in the error message. This method is used by default
when comparing strings with @ref{dd5,,assertEqual()}.


New in version 3.1.

@end deffn

@geindex assertSequenceEqual() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertSequenceEqual}@anchor{edf}
@deffn {Method} assertSequenceEqual (first, second, msg=None, seq_type=None)

Tests that two sequences are equal.  If a @emph{seq_type} is supplied, both
@emph{first} and @emph{second} must be instances of @emph{seq_type} or a failure will
be raised.  If the sequences are different an error message is
constructed that shows the difference between the two.

This method is not called directly by @ref{dd5,,assertEqual()}, but
it’s used to implement @ref{edd,,assertListEqual()} and
@ref{ede,,assertTupleEqual()}.


New in version 3.1.

@end deffn

@geindex assertListEqual() (unittest.TestCase method)
@geindex assertTupleEqual() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertListEqual}@anchor{edd}
@deffn {Method} assertListEqual (first, second, msg=None)
@anchor{library/unittest unittest TestCase assertTupleEqual}@anchor{ede}
@deffnx {Method} assertTupleEqual (first, second, msg=None)

Tests that two lists or tuples are equal.  If not, an error message is
constructed that shows only the differences between the two.  An error
is also raised if either of the parameters are of the wrong type.
These methods are used by default when comparing lists or tuples with
@ref{dd5,,assertEqual()}.


New in version 3.1.

@end deffn

@geindex assertSetEqual() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertSetEqual}@anchor{edc}
@deffn {Method} assertSetEqual (first, second, msg=None)

Tests that two sets are equal.  If not, an error message is constructed
that lists the differences between the sets.  This method is used by
default when comparing sets or frozensets with @ref{dd5,,assertEqual()}.

Fails if either of @emph{first} or @emph{second} does not have a @code{set.difference()}
method.


New in version 3.1.

@end deffn

@geindex assertDictEqual() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase assertDictEqual}@anchor{ee0}
@deffn {Method} assertDictEqual (first, second, msg=None)

Test that two dictionaries are equal.  If not, an error message is
constructed that shows the differences in the dictionaries. This
method will be used by default to compare dictionaries in
calls to @ref{dd5,,assertEqual()}.


New in version 3.1.

@end deffn
@anchor{library/unittest other-methods-and-attrs}@anchor{3834}
Finally the @ref{1ab,,TestCase} provides the following methods and attributes:

@geindex fail() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase fail}@anchor{3835}
@deffn {Method} fail (msg=None)

Signals a test failure unconditionally, with @emph{msg} or @code{None} for
the error message.
@end deffn

@geindex failureException (unittest.TestCase attribute)
@anchor{library/unittest unittest TestCase failureException}@anchor{3833}
@deffn {Attribute} failureException

This class attribute gives the exception raised by the test method.  If a
test framework needs to use a specialized exception, possibly to carry
additional information, it must subclass this exception in order to “play
fair” with the framework.  The initial value of this attribute is
@ref{46e,,AssertionError}.
@end deffn

@geindex longMessage (unittest.TestCase attribute)
@anchor{library/unittest unittest TestCase longMessage}@anchor{eca}
@deffn {Attribute} longMessage

This class attribute determines what happens when a custom failure message
is passed as the msg argument to an assertXYY call that fails.
@code{True} is the default value. In this case, the custom message is appended
to the end of the standard failure message.
When set to @code{False}, the custom message replaces the standard message.

The class setting can be overridden in individual test methods by assigning
an instance attribute, self.longMessage, to @code{True} or @code{False} before
calling the assert methods.

The class setting gets reset before each test call.


New in version 3.1.

@end deffn

@geindex maxDiff (unittest.TestCase attribute)
@anchor{library/unittest unittest TestCase maxDiff}@anchor{dd0}
@deffn {Attribute} maxDiff

This attribute controls the maximum length of diffs output by assert
methods that report diffs on failure. It defaults to 80*8 characters.
Assert methods affected by this attribute are
@ref{edf,,assertSequenceEqual()} (including all the sequence comparison
methods that delegate to it), @ref{ee0,,assertDictEqual()} and
@ref{ed9,,assertMultiLineEqual()}.

Setting @code{maxDiff} to @code{None} means that there is no maximum length of
diffs.


New in version 3.2.

@end deffn

Testing frameworks can use the following methods to collect information on
the test:

@geindex countTestCases() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase countTestCases}@anchor{3836}
@deffn {Method} countTestCases ()

Return the number of tests represented by this test object.  For
@ref{1ab,,TestCase} instances, this will always be @code{1}.
@end deffn

@geindex defaultTestResult() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase defaultTestResult}@anchor{382e}
@deffn {Method} defaultTestResult ()

Return an instance of the test result class that should be used for this
test case class (if no other result instance is provided to the
@ref{d03,,run()} method).

For @ref{1ab,,TestCase} instances, this will always be an instance of
@ref{d04,,TestResult}; subclasses of @ref{1ab,,TestCase} should override this
as necessary.
@end deffn

@geindex id() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase id}@anchor{3837}
@deffn {Method} id ()

Return a string identifying the specific test case.  This is usually the
full name of the test method, including the module and class name.
@end deffn

@geindex shortDescription() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase shortDescription}@anchor{3838}
@deffn {Method} shortDescription ()

Returns a description of the test, or @code{None} if no description
has been provided.  The default implementation of this method
returns the first line of the test method’s docstring, if available,
or @code{None}.


Changed in version 3.1: In 3.1 this was changed to add the test name to the short description
even in the presence of a docstring.  This caused compatibility issues
with unittest extensions and adding the test name was moved to the
@ref{10bd,,TextTestResult} in Python 3.2.

@end deffn

@geindex addCleanup() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase addCleanup}@anchor{599}
@deffn {Method} addCleanup (function, /, *args, **kwargs)

Add a function to be called after @ref{ece,,tearDown()} to cleanup resources
used during the test. Functions will be called in reverse order to the
order they are added (@abbr{LIFO}).  They
are called with any arguments and keyword arguments passed into
@ref{599,,addCleanup()} when they are added.

If @ref{ecd,,setUp()} fails, meaning that @ref{ece,,tearDown()} is not called,
then any cleanup functions added will still be called.


New in version 3.1.

@end deffn

@geindex doCleanups() (unittest.TestCase method)
@anchor{library/unittest unittest TestCase doCleanups}@anchor{ecc}
@deffn {Method} doCleanups ()

This method is called unconditionally after @ref{ece,,tearDown()}, or
after @ref{ecd,,setUp()} if @ref{ecd,,setUp()} raises an exception.

It is responsible for calling all the cleanup functions added by
@ref{599,,addCleanup()}. If you need cleanup functions to be called
@emph{prior} to @ref{ece,,tearDown()} then you can call @ref{ecc,,doCleanups()}
yourself.

@ref{ecc,,doCleanups()} pops methods off the stack of cleanup
functions one at a time, so it can be called at any time.


New in version 3.1.

@end deffn

@geindex addClassCleanup() (unittest.TestCase class method)
@anchor{library/unittest unittest TestCase addClassCleanup}@anchor{563}
@deffn {Method} classmethod addClassCleanup (function, /, *args, **kwargs)

Add a function to be called after @ref{ecb,,tearDownClass()} to cleanup
resources used during the test class. Functions will be called in reverse
order to the order they are added (@abbr{LIFO}).
They are called with any arguments and keyword arguments passed into
@ref{563,,addClassCleanup()} when they are added.

If @ref{564,,setUpClass()} fails, meaning that @ref{ecb,,tearDownClass()} is not
called, then any cleanup functions added will still be called.


New in version 3.8.

@end deffn

@geindex doClassCleanups() (unittest.TestCase class method)
@anchor{library/unittest unittest TestCase doClassCleanups}@anchor{3839}
@deffn {Method} classmethod doClassCleanups ()

This method is called unconditionally after @ref{ecb,,tearDownClass()}, or
after @ref{564,,setUpClass()} if @ref{564,,setUpClass()} raises an exception.

It is responsible for calling all the cleanup functions added by
@ref{563,,addClassCleanup()}. If you need cleanup functions to be called
@emph{prior} to @ref{ecb,,tearDownClass()} then you can call
@ref{3839,,doClassCleanups()} yourself.

@ref{3839,,doClassCleanups()} pops methods off the stack of cleanup
functions one at a time, so it can be called at any time.


New in version 3.8.

@end deffn
@end deffn

@geindex IsolatedAsyncioTestCase (class in unittest)
@anchor{library/unittest unittest IsolatedAsyncioTestCase}@anchor{1ac}
@deffn {Class} unittest.IsolatedAsyncioTestCase (methodName='runTest')

This class provides an API similar to @ref{1ab,,TestCase} and also accepts
coroutines as test functions.


New in version 3.8.


@geindex asyncSetUp() (unittest.IsolatedAsyncioTestCase method)
@anchor{library/unittest unittest IsolatedAsyncioTestCase asyncSetUp}@anchor{383a}
@deffn {Method} coroutine asyncSetUp ()

Method called to prepare the test fixture. This is called after @code{setUp()}.
This is called immediately before calling the test method; other than
@ref{46e,,AssertionError} or @ref{b6a,,SkipTest}, any exception raised by this method
will be considered an error rather than a test failure. The default implementation
does nothing.
@end deffn

@geindex asyncTearDown() (unittest.IsolatedAsyncioTestCase method)
@anchor{library/unittest unittest IsolatedAsyncioTestCase asyncTearDown}@anchor{383b}
@deffn {Method} coroutine asyncTearDown ()

Method called immediately after the test method has been called and the
result recorded.  This is called before @code{tearDown()}. This is called even if
the test method raised an exception, so the implementation in subclasses may need
to be particularly careful about checking internal state.  Any exception, other than
@ref{46e,,AssertionError} or @ref{b6a,,SkipTest}, raised by this method will be
considered an additional error rather than a test failure (thus increasing
the total number of reported errors). This method will only be called if
the @ref{383a,,asyncSetUp()} succeeds, regardless of the outcome of the test method.
The default implementation does nothing.
@end deffn

@geindex addAsyncCleanup() (unittest.IsolatedAsyncioTestCase method)
@anchor{library/unittest unittest IsolatedAsyncioTestCase addAsyncCleanup}@anchor{383c}
@deffn {Method} addAsyncCleanup (function, /, *args, **kwargs)

This method accepts a coroutine that can be used as a cleanup function.
@end deffn

@geindex run() (unittest.IsolatedAsyncioTestCase method)
@anchor{library/unittest unittest IsolatedAsyncioTestCase run}@anchor{383d}
@deffn {Method} run (result=None)

Sets up a new event loop to run the test, collecting the result into
the @ref{d04,,TestResult} object passed as @emph{result}.  If @emph{result} is
omitted or @code{None}, a temporary result object is created (by calling
the @code{defaultTestResult()} method) and used. The result object is
returned to @ref{383d,,run()}’s caller. At the end of the test all the tasks
in the event loop are cancelled.
@end deffn

An example illustrating the order:

@example
from unittest import IsolatedAsyncioTestCase

events = []


class Test(IsolatedAsyncioTestCase):


    def setUp(self):
        events.append("setUp")

    async def asyncSetUp(self):
        self._async_connection = await AsyncConnection()
        events.append("asyncSetUp")

    async def test_response(self):
        events.append("test_response")
        response = await self._async_connection.get("https://example.com")
        self.assertEqual(response.status_code, 200)
        self.addAsyncCleanup(self.on_cleanup)

    def tearDown(self):
        events.append("tearDown")

    async def asyncTearDown(self):
        await self._async_connection.close()
        events.append("asyncTearDown")

    async def on_cleanup(self):
        events.append("cleanup")

if __name__ == "__main__":
    unittest.main()
@end example

After running the test, @code{events} would contain @code{["setUp", "asyncSetUp", "test_response", "asyncTearDown", "tearDown", "cleanup"]}.
@end deffn

@geindex FunctionTestCase (class in unittest)
@anchor{library/unittest unittest FunctionTestCase}@anchor{381f}
@deffn {Class} unittest.FunctionTestCase (testFunc, setUp=None, tearDown=None, description=None)

This class implements the portion of the @ref{1ab,,TestCase} interface which
allows the test runner to drive the test, but does not provide the methods
which test code can use to check and report errors.  This is used to create
test cases using legacy test code, allowing it to be integrated into a
@ref{122,,unittest}-based test framework.
@end deffn

@menu
* Deprecated aliases:: 

@end menu

@node Deprecated aliases,,,Test cases
@anchor{library/unittest deprecated-aliases}@anchor{dd9}@anchor{library/unittest id1}@anchor{383e}
@subsubsection Deprecated aliases


For historical reasons, some of the @ref{1ab,,TestCase} methods had one or more
aliases that are now deprecated.  The following table lists the correct names
along with their deprecated aliases:

@quotation


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Method Name

@tab

Deprecated alias

@tab

Deprecated alias

@item

@ref{dd5,,assertEqual()}

@tab

failUnlessEqual

@tab

assertEquals

@item

@ref{dd6,,assertNotEqual()}

@tab

failIfEqual

@tab

assertNotEquals

@item

@ref{dd4,,assertTrue()}

@tab

failUnless

@tab

assert_

@item

@ref{ec9,,assertFalse()}

@tab

failIf

@tab

@item

@ref{cff,,assertRaises()}

@tab

failUnlessRaises

@tab

@item

@ref{dd7,,assertAlmostEqual()}

@tab

failUnlessAlmostEqual

@tab

assertAlmostEquals

@item

@ref{dd8,,assertNotAlmostEqual()}

@tab

failIfAlmostEqual

@tab

assertNotAlmostEquals

@item

@ref{dd1,,assertRegex()}

@tab

@tab

assertRegexpMatches

@item

@ref{3832,,assertNotRegex()}

@tab

@tab

assertNotRegexpMatches

@item

@ref{d00,,assertRaisesRegex()}

@tab

@tab

assertRaisesRegexp

@end multitable



Deprecated since version 3.1: The fail* aliases listed in the second column have been deprecated.



Deprecated since version 3.2: The assert* aliases listed in the third column have been deprecated.



Deprecated since version 3.2: @code{assertRegexpMatches} and @code{assertRaisesRegexp} have been renamed to
@ref{dd1,,assertRegex()} and @ref{d00,,assertRaisesRegex()}.



Deprecated since version 3.5: The @code{assertNotRegexpMatches} name is deprecated in favor of @ref{3832,,assertNotRegex()}.

@end quotation

@node Grouping tests,Loading and running tests,Test cases,Classes and functions
@anchor{library/unittest grouping-tests}@anchor{383f}@anchor{library/unittest testsuite-objects}@anchor{3840}
@subsubsection Grouping tests


@geindex TestSuite (class in unittest)
@anchor{library/unittest unittest TestSuite}@anchor{953}
@deffn {Class} unittest.TestSuite (tests=())

This class represents an aggregation of individual test cases and test suites.
The class presents the interface needed by the test runner to allow it to be run
as any other test case.  Running a @ref{953,,TestSuite} instance is the same as
iterating over the suite, running each test individually.

If @emph{tests} is given, it must be an iterable of individual test cases or other
test suites that will be used to build the suite initially. Additional methods
are provided to add test cases and suites to the collection later on.

@ref{953,,TestSuite} objects behave much like @ref{1ab,,TestCase} objects, except
they do not actually implement a test.  Instead, they are used to aggregate
tests into groups of tests that should be run together. Some additional
methods are available to add tests to @ref{953,,TestSuite} instances:

@geindex addTest() (unittest.TestSuite method)
@anchor{library/unittest unittest TestSuite addTest}@anchor{3841}
@deffn {Method} addTest (test)

Add a @ref{1ab,,TestCase} or @ref{953,,TestSuite} to the suite.
@end deffn

@geindex addTests() (unittest.TestSuite method)
@anchor{library/unittest unittest TestSuite addTests}@anchor{3842}
@deffn {Method} addTests (tests)

Add all the tests from an iterable of @ref{1ab,,TestCase} and @ref{953,,TestSuite}
instances to this test suite.

This is equivalent to iterating over @emph{tests}, calling @ref{3841,,addTest()} for
each element.
@end deffn

@ref{953,,TestSuite} shares the following methods with @ref{1ab,,TestCase}:

@geindex run() (unittest.TestSuite method)
@anchor{library/unittest unittest TestSuite run}@anchor{3843}
@deffn {Method} run (result)

Run the tests associated with this suite, collecting the result into the
test result object passed as @emph{result}.  Note that unlike
@ref{d03,,TestCase.run()}, @ref{3843,,TestSuite.run()} requires the result object to
be passed in.
@end deffn

@geindex debug() (unittest.TestSuite method)
@anchor{library/unittest unittest TestSuite debug}@anchor{3844}
@deffn {Method} debug ()

Run the tests associated with this suite without collecting the
result. This allows exceptions raised by the test to be propagated to the
caller and can be used to support running tests under a debugger.
@end deffn

@geindex countTestCases() (unittest.TestSuite method)
@anchor{library/unittest unittest TestSuite countTestCases}@anchor{3845}
@deffn {Method} countTestCases ()

Return the number of tests represented by this test object, including all
individual tests and sub-suites.
@end deffn

@geindex __iter__() (unittest.TestSuite method)
@anchor{library/unittest unittest TestSuite __iter__}@anchor{bba}
@deffn {Method} __iter__ ()

Tests grouped by a @ref{953,,TestSuite} are always accessed by iteration.
Subclasses can lazily provide tests by overriding @ref{bba,,__iter__()}. Note
that this method may be called several times on a single suite (for
example when counting tests or comparing for equality) so the tests
returned by repeated iterations before @ref{3843,,TestSuite.run()} must be the
same for each call iteration. After @ref{3843,,TestSuite.run()}, callers should
not rely on the tests returned by this method unless the caller uses a
subclass that overrides @code{TestSuite._removeTestAtIndex()} to preserve
test references.


Changed in version 3.2: In earlier versions the @ref{953,,TestSuite} accessed tests directly rather
than through iteration, so overriding @ref{bba,,__iter__()} wasn’t sufficient
for providing tests.



Changed in version 3.4: In earlier versions the @ref{953,,TestSuite} held references to each
@ref{1ab,,TestCase} after @ref{3843,,TestSuite.run()}. Subclasses can restore
that behavior by overriding @code{TestSuite._removeTestAtIndex()}.

@end deffn

In the typical usage of a @ref{953,,TestSuite} object, the @ref{3843,,run()} method
is invoked by a @code{TestRunner} rather than by the end-user test harness.
@end deffn

@node Loading and running tests,,Grouping tests,Classes and functions
@anchor{library/unittest loading-and-running-tests}@anchor{3846}
@subsubsection Loading and running tests


@geindex TestLoader (class in unittest)
@anchor{library/unittest unittest TestLoader}@anchor{1ad}
@deffn {Class} unittest.TestLoader

The @ref{1ad,,TestLoader} class is used to create test suites from classes and
modules.  Normally, there is no need to create an instance of this class; the
@ref{122,,unittest} module provides an instance that can be shared as
@ref{3847,,unittest.defaultTestLoader}.  Using a subclass or instance, however,
allows customization of some configurable properties.

@ref{1ad,,TestLoader} objects have the following attributes:

@geindex errors (unittest.TestLoader attribute)
@anchor{library/unittest unittest TestLoader errors}@anchor{9fe}
@deffn {Attribute} errors

A list of the non-fatal errors encountered while loading tests. Not reset
by the loader at any point. Fatal errors are signalled by the relevant
a method raising an exception to the caller. Non-fatal errors are also
indicated by a synthetic test that will raise the original error when
run.


New in version 3.5.

@end deffn

@ref{1ad,,TestLoader} objects have the following methods:

@geindex loadTestsFromTestCase() (unittest.TestLoader method)
@anchor{library/unittest unittest TestLoader loadTestsFromTestCase}@anchor{1af}
@deffn {Method} loadTestsFromTestCase (testCaseClass)

Return a suite of all test cases contained in the @ref{1ab,,TestCase}-derived
@code{testCaseClass}.

A test case instance is created for each method named by
@ref{1b0,,getTestCaseNames()}. By default these are the method names
beginning with @code{test}. If @ref{1b0,,getTestCaseNames()} returns no
methods, but the @code{runTest()} method is implemented, a single test
case is created for that method instead.
@end deffn

@geindex loadTestsFromModule() (unittest.TestLoader method)
@anchor{library/unittest unittest TestLoader loadTestsFromModule}@anchor{1ae}
@deffn {Method} loadTestsFromModule (module, pattern=None)

Return a suite of all test cases contained in the given module. This
method searches @emph{module} for classes derived from @ref{1ab,,TestCase} and
creates an instance of the class for each test method defined for the
class.

@cartouche
@quotation Note 
While using a hierarchy of @ref{1ab,,TestCase}-derived classes can be
convenient in sharing fixtures and helper functions, defining test
methods on base classes that are not intended to be instantiated
directly does not play well with this method.  Doing so, however, can
be useful when the fixtures are different and defined in subclasses.
@end quotation
@end cartouche

If a module provides a @code{load_tests} function it will be called to
load the tests. This allows modules to customize test loading.
This is the @ref{381d,,load_tests protocol}.  The @emph{pattern} argument is passed as
the third argument to @code{load_tests}.


Changed in version 3.2: Support for @code{load_tests} added.



Changed in version 3.5: The undocumented and unofficial @emph{use_load_tests} default argument is
deprecated and ignored, although it is still accepted for backward
compatibility.  The method also now accepts a keyword-only argument
@emph{pattern} which is passed to @code{load_tests} as the third argument.

@end deffn

@geindex loadTestsFromName() (unittest.TestLoader method)
@anchor{library/unittest unittest TestLoader loadTestsFromName}@anchor{ee1}
@deffn {Method} loadTestsFromName (name, module=None)

Return a suite of all test cases given a string specifier.

The specifier @emph{name} is a “dotted name” that may resolve either to a
module, a test case class, a test method within a test case class, a
@ref{953,,TestSuite} instance, or a callable object which returns a
@ref{1ab,,TestCase} or @ref{953,,TestSuite} instance.  These checks are
applied in the order listed here; that is, a method on a possible test
case class will be picked up as “a test method within a test case class”,
rather than “a callable object”.

For example, if you have a module @code{SampleTests} containing a
@ref{1ab,,TestCase}-derived class @code{SampleTestCase} with three test
methods (@code{test_one()}, @code{test_two()}, and @code{test_three()}), the
specifier @code{'SampleTests.SampleTestCase'} would cause this method to
return a suite which will run all three test methods. Using the specifier
@code{'SampleTests.SampleTestCase.test_two'} would cause it to return a test
suite which will run only the @code{test_two()} test method. The specifier
can refer to modules and packages which have not been imported; they will
be imported as a side-effect.

The method optionally resolves @emph{name} relative to the given @emph{module}.


Changed in version 3.5: If an @ref{3b8,,ImportError} or @ref{163,,AttributeError} occurs while traversing
@emph{name} then a synthetic test that raises that error when run will be
returned. These errors are included in the errors accumulated by
self.errors.

@end deffn

@geindex loadTestsFromNames() (unittest.TestLoader method)
@anchor{library/unittest unittest TestLoader loadTestsFromNames}@anchor{3848}
@deffn {Method} loadTestsFromNames (names, module=None)

Similar to @ref{ee1,,loadTestsFromName()}, but takes a sequence of names rather
than a single name.  The return value is a test suite which supports all
the tests defined for each name.
@end deffn

@geindex getTestCaseNames() (unittest.TestLoader method)
@anchor{library/unittest unittest TestLoader getTestCaseNames}@anchor{1b0}
@deffn {Method} getTestCaseNames (testCaseClass)

Return a sorted sequence of method names found within @emph{testCaseClass};
this should be a subclass of @ref{1ab,,TestCase}.
@end deffn

@geindex discover() (unittest.TestLoader method)
@anchor{library/unittest unittest TestLoader discover}@anchor{b6b}
@deffn {Method} discover (start_dir, pattern='test*.py', top_level_dir=None)

Find all the test modules by recursing into subdirectories from the
specified start directory, and return a TestSuite object containing them.
Only test files that match @emph{pattern} will be loaded. (Using shell style
pattern matching.) Only module names that are importable (i.e. are valid
Python identifiers) will be loaded.

All test modules must be importable from the top level of the project. If
the start directory is not the top level directory then the top level
directory must be specified separately.

If importing a module fails, for example due to a syntax error, then
this will be recorded as a single error and discovery will continue.  If
the import failure is due to @ref{b6a,,SkipTest} being raised, it will be
recorded as a skip instead of an error.

If a package (a directory containing a file named @code{__init__.py}) is
found, the package will be checked for a @code{load_tests} function. If this
exists then it will be called
@code{package.load_tests(loader, tests, pattern)}. Test discovery takes care
to ensure that a package is only checked for tests once during an
invocation, even if the load_tests function itself calls
@code{loader.discover}.

If @code{load_tests} exists then discovery does @emph{not} recurse into the
package, @code{load_tests} is responsible for loading all tests in the
package.

The pattern is deliberately not stored as a loader attribute so that
packages can continue discovery themselves. @emph{top_level_dir} is stored so
@code{load_tests} does not need to pass this argument in to
@code{loader.discover()}.

@emph{start_dir} can be a dotted module name as well as a directory.


New in version 3.2.



Changed in version 3.4: Modules that raise @ref{b6a,,SkipTest} on import are recorded as skips,
not errors.



Changed in version 3.4: @emph{start_dir} can be a @ref{1840,,namespace packages}.



Changed in version 3.4: Paths are sorted before being imported so that execution order is the
same even if the underlying file system’s ordering is not dependent
on file name.



Changed in version 3.5: Found packages are now checked for @code{load_tests} regardless of
whether their path matches @emph{pattern}, because it is impossible for
a package name to match the default pattern.



Changed in version 3.11: @emph{start_dir} can not be a @ref{1840,,namespace packages}.
It has been broken since Python 3.7 and Python 3.11 officially remove it.

@end deffn

The following attributes of a @ref{1ad,,TestLoader} can be configured either by
subclassing or assignment on an instance:

@geindex testMethodPrefix (unittest.TestLoader attribute)
@anchor{library/unittest unittest TestLoader testMethodPrefix}@anchor{3849}
@deffn {Attribute} testMethodPrefix

String giving the prefix of method names which will be interpreted as test
methods.  The default value is @code{'test'}.

This affects @ref{1b0,,getTestCaseNames()} and all the @code{loadTestsFrom*()}
methods.
@end deffn

@geindex sortTestMethodsUsing (unittest.TestLoader attribute)
@anchor{library/unittest unittest TestLoader sortTestMethodsUsing}@anchor{384a}
@deffn {Attribute} sortTestMethodsUsing

Function to be used to compare method names when sorting them in
@ref{1b0,,getTestCaseNames()} and all the @code{loadTestsFrom*()} methods.
@end deffn

@geindex suiteClass (unittest.TestLoader attribute)
@anchor{library/unittest unittest TestLoader suiteClass}@anchor{ee2}
@deffn {Attribute} suiteClass

Callable object that constructs a test suite from a list of tests. No
methods on the resulting object are needed.  The default value is the
@ref{953,,TestSuite} class.

This affects all the @code{loadTestsFrom*()} methods.
@end deffn

@geindex testNamePatterns (unittest.TestLoader attribute)
@anchor{library/unittest unittest TestLoader testNamePatterns}@anchor{384b}
@deffn {Attribute} testNamePatterns

List of Unix shell-style wildcard test name patterns that test methods
have to match to be included in test suites (see @code{-v} option).

If this attribute is not @code{None} (the default), all test methods to be
included in test suites must match one of the patterns in this list.
Note that matches are always performed using @ref{1107,,fnmatch.fnmatchcase()},
so unlike patterns passed to the @code{-v} option, simple substring patterns
will have to be converted using @code{*} wildcards.

This affects all the @code{loadTestsFrom*()} methods.


New in version 3.7.

@end deffn
@end deffn

@geindex TestResult (class in unittest)
@anchor{library/unittest unittest TestResult}@anchor{d04}
@deffn {Class} unittest.TestResult

This class is used to compile information about which tests have succeeded
and which have failed.

A @ref{d04,,TestResult} object stores the results of a set of tests.  The
@ref{1ab,,TestCase} and @ref{953,,TestSuite} classes ensure that results are
properly recorded; test authors do not need to worry about recording the
outcome of tests.

Testing frameworks built on top of @ref{122,,unittest} may want access to the
@ref{d04,,TestResult} object generated by running a set of tests for reporting
purposes; a @ref{d04,,TestResult} instance is returned by the
@code{TestRunner.run()} method for this purpose.

@ref{d04,,TestResult} instances have the following attributes that will be of
interest when inspecting the results of running a set of tests:

@geindex errors (unittest.TestResult attribute)
@anchor{library/unittest unittest TestResult errors}@anchor{384c}
@deffn {Attribute} errors

A list containing 2-tuples of @ref{1ab,,TestCase} instances and strings
holding formatted tracebacks. Each tuple represents a test which raised an
unexpected exception.
@end deffn

@geindex failures (unittest.TestResult attribute)
@anchor{library/unittest unittest TestResult failures}@anchor{384d}
@deffn {Attribute} failures

A list containing 2-tuples of @ref{1ab,,TestCase} instances and strings
holding formatted tracebacks. Each tuple represents a test where a failure
was explicitly signalled using the @code{TestCase.assert*()} methods.
@end deffn

@geindex skipped (unittest.TestResult attribute)
@anchor{library/unittest unittest TestResult skipped}@anchor{384e}
@deffn {Attribute} skipped

A list containing 2-tuples of @ref{1ab,,TestCase} instances and strings
holding the reason for skipping the test.


New in version 3.1.

@end deffn

@geindex expectedFailures (unittest.TestResult attribute)
@anchor{library/unittest unittest TestResult expectedFailures}@anchor{384f}
@deffn {Attribute} expectedFailures

A list containing 2-tuples of @ref{1ab,,TestCase} instances and strings
holding formatted tracebacks.  Each tuple represents an expected failure
or error of the test case.
@end deffn

@geindex unexpectedSuccesses (unittest.TestResult attribute)
@anchor{library/unittest unittest TestResult unexpectedSuccesses}@anchor{3850}
@deffn {Attribute} unexpectedSuccesses

A list containing @ref{1ab,,TestCase} instances that were marked as expected
failures, but succeeded.
@end deffn

@geindex shouldStop (unittest.TestResult attribute)
@anchor{library/unittest unittest TestResult shouldStop}@anchor{3851}
@deffn {Attribute} shouldStop

Set to @code{True} when the execution of tests should stop by @ref{3852,,stop()}.
@end deffn

@geindex testsRun (unittest.TestResult attribute)
@anchor{library/unittest unittest TestResult testsRun}@anchor{3853}
@deffn {Attribute} testsRun

The total number of tests run so far.
@end deffn

@geindex buffer (unittest.TestResult attribute)
@anchor{library/unittest unittest TestResult buffer}@anchor{3854}
@deffn {Attribute} buffer

If set to true, @code{sys.stdout} and @code{sys.stderr} will be buffered in between
@ref{3855,,startTest()} and @ref{3856,,stopTest()} being called. Collected output will
only be echoed onto the real @code{sys.stdout} and @code{sys.stderr} if the test
fails or errors. Any output is also attached to the failure / error message.


New in version 3.2.

@end deffn

@geindex failfast (unittest.TestResult attribute)
@anchor{library/unittest unittest TestResult failfast}@anchor{3857}
@deffn {Attribute} failfast

If set to true @ref{3852,,stop()} will be called on the first failure or error,
halting the test run.


New in version 3.2.

@end deffn

@geindex tb_locals (unittest.TestResult attribute)
@anchor{library/unittest unittest TestResult tb_locals}@anchor{3858}
@deffn {Attribute} tb_locals

If set to true then local variables will be shown in tracebacks.


New in version 3.5.

@end deffn

@geindex wasSuccessful() (unittest.TestResult method)
@anchor{library/unittest unittest TestResult wasSuccessful}@anchor{3859}
@deffn {Method} wasSuccessful ()

Return @code{True} if all tests run so far have passed, otherwise returns
@code{False}.


Changed in version 3.4: Returns @code{False} if there were any @ref{3850,,unexpectedSuccesses}
from tests marked with the @ref{3825,,expectedFailure()} decorator.

@end deffn

@geindex stop() (unittest.TestResult method)
@anchor{library/unittest unittest TestResult stop}@anchor{3852}
@deffn {Method} stop ()

This method can be called to signal that the set of tests being run should
be aborted by setting the @ref{3851,,shouldStop} attribute to @code{True}.
@code{TestRunner} objects should respect this flag and return without
running any additional tests.

For example, this feature is used by the @ref{10cf,,TextTestRunner} class to
stop the test framework when the user signals an interrupt from the
keyboard.  Interactive tools which provide @code{TestRunner}
implementations can use this in a similar manner.
@end deffn

The following methods of the @ref{d04,,TestResult} class are used to maintain
the internal data structures, and may be extended in subclasses to support
additional reporting requirements.  This is particularly useful in building
tools which support interactive reporting while tests are being run.

@geindex startTest() (unittest.TestResult method)
@anchor{library/unittest unittest TestResult startTest}@anchor{3855}
@deffn {Method} startTest (test)

Called when the test case @emph{test} is about to be run.
@end deffn

@geindex stopTest() (unittest.TestResult method)
@anchor{library/unittest unittest TestResult stopTest}@anchor{3856}
@deffn {Method} stopTest (test)

Called after the test case @emph{test} has been executed, regardless of the
outcome.
@end deffn

@geindex startTestRun() (unittest.TestResult method)
@anchor{library/unittest unittest TestResult startTestRun}@anchor{ee5}
@deffn {Method} startTestRun ()

Called once before any tests are executed.


New in version 3.1.

@end deffn

@geindex stopTestRun() (unittest.TestResult method)
@anchor{library/unittest unittest TestResult stopTestRun}@anchor{ee6}
@deffn {Method} stopTestRun ()

Called once after all tests are executed.


New in version 3.1.

@end deffn

@geindex addError() (unittest.TestResult method)
@anchor{library/unittest unittest TestResult addError}@anchor{10fc}
@deffn {Method} addError (test, err)

Called when the test case @emph{test} raises an unexpected exception. @emph{err} is a
tuple of the form returned by @ref{18f,,sys.exc_info()}: @code{(type, value,
traceback)}.

The default implementation appends a tuple @code{(test, formatted_err)} to
the instance’s @ref{384c,,errors} attribute, where @emph{formatted_err} is a
formatted traceback derived from @emph{err}.
@end deffn

@geindex addFailure() (unittest.TestResult method)
@anchor{library/unittest unittest TestResult addFailure}@anchor{10fb}
@deffn {Method} addFailure (test, err)

Called when the test case @emph{test} signals a failure. @emph{err} is a tuple of
the form returned by @ref{18f,,sys.exc_info()}: @code{(type, value, traceback)}.

The default implementation appends a tuple @code{(test, formatted_err)} to
the instance’s @ref{384d,,failures} attribute, where @emph{formatted_err} is a
formatted traceback derived from @emph{err}.
@end deffn

@geindex addSuccess() (unittest.TestResult method)
@anchor{library/unittest unittest TestResult addSuccess}@anchor{385a}
@deffn {Method} addSuccess (test)

Called when the test case @emph{test} succeeds.

The default implementation does nothing.
@end deffn

@geindex addSkip() (unittest.TestResult method)
@anchor{library/unittest unittest TestResult addSkip}@anchor{10fd}
@deffn {Method} addSkip (test, reason)

Called when the test case @emph{test} is skipped.  @emph{reason} is the reason the
test gave for skipping.

The default implementation appends a tuple @code{(test, reason)} to the
instance’s @ref{384e,,skipped} attribute.
@end deffn

@geindex addExpectedFailure() (unittest.TestResult method)
@anchor{library/unittest unittest TestResult addExpectedFailure}@anchor{385b}
@deffn {Method} addExpectedFailure (test, err)

Called when the test case @emph{test} fails or errors, but was marked with
the @ref{3825,,expectedFailure()} decorator.

The default implementation appends a tuple @code{(test, formatted_err)} to
the instance’s @ref{384f,,expectedFailures} attribute, where @emph{formatted_err}
is a formatted traceback derived from @emph{err}.
@end deffn

@geindex addUnexpectedSuccess() (unittest.TestResult method)
@anchor{library/unittest unittest TestResult addUnexpectedSuccess}@anchor{385c}
@deffn {Method} addUnexpectedSuccess (test)

Called when the test case @emph{test} was marked with the
@ref{3825,,expectedFailure()} decorator, but succeeded.

The default implementation appends the test to the instance’s
@ref{3850,,unexpectedSuccesses} attribute.
@end deffn

@geindex addSubTest() (unittest.TestResult method)
@anchor{library/unittest unittest TestResult addSubTest}@anchor{10fe}
@deffn {Method} addSubTest (test, subtest, outcome)

Called when a subtest finishes.  @emph{test} is the test case
corresponding to the test method.  @emph{subtest} is a custom
@ref{1ab,,TestCase} instance describing the subtest.

If @emph{outcome} is @ref{182,,None}, the subtest succeeded.  Otherwise,
it failed with an exception where @emph{outcome} is a tuple of the form
returned by @ref{18f,,sys.exc_info()}: @code{(type, value, traceback)}.

The default implementation does nothing when the outcome is a
success, and records subtest failures as normal failures.


New in version 3.4.

@end deffn
@end deffn

@geindex TextTestResult (class in unittest)
@anchor{library/unittest unittest TextTestResult}@anchor{10bd}
@deffn {Class} unittest.TextTestResult (stream, descriptions, verbosity)

A concrete implementation of @ref{d04,,TestResult} used by the
@ref{10cf,,TextTestRunner}.


New in version 3.2: This class was previously named @code{_TextTestResult}. The old name still
exists as an alias but is deprecated.

@end deffn

@geindex defaultTestLoader (in module unittest)
@anchor{library/unittest unittest defaultTestLoader}@anchor{3847}
@deffn {Data} unittest.defaultTestLoader

Instance of the @ref{1ad,,TestLoader} class intended to be shared.  If no
customization of the @ref{1ad,,TestLoader} is needed, this instance can be used
instead of repeatedly creating new instances.
@end deffn

@geindex TextTestRunner (class in unittest)
@anchor{library/unittest unittest TextTestRunner}@anchor{10cf}
@deffn {Class} unittest.TextTestRunner (stream=None, descriptions=True, verbosity=1, failfast=False, buffer=False, resultclass=None, warnings=None, *, tb_locals=False)

A basic test runner implementation that outputs results to a stream. If @emph{stream}
is @code{None}, the default, @ref{42d,,sys.stderr} is used as the output stream. This class
has a few configurable parameters, but is essentially very simple.  Graphical
applications which run test suites should provide alternate implementations. Such
implementations should accept @code{**kwargs} as the interface to construct runners
changes when features are added to unittest.

By default this runner shows @ref{1b2,,DeprecationWarning},
@ref{3a9,,PendingDeprecationWarning}, @ref{77b,,ResourceWarning} and
@ref{304,,ImportWarning} even if they are @ref{385d,,ignored by default}. Deprecation warnings caused by @ref{dd9,,deprecated unittest methods} are also special-cased and, when the warning
filters are @code{'default'} or @code{'always'}, they will appear only once
per-module, in order to avoid too many warning messages.  This behavior can
be overridden using Python’s @code{-Wd} or @code{-Wa} options
(see @ref{1612,,Warning control}) and leaving
@emph{warnings} to @code{None}.


Changed in version 3.2: Added the @code{warnings} argument.



Changed in version 3.2: The default stream is set to @ref{42d,,sys.stderr} at instantiation time rather
than import time.



Changed in version 3.5: Added the tb_locals parameter.


@geindex _makeResult() (unittest.TextTestRunner method)
@anchor{library/unittest unittest TextTestRunner _makeResult}@anchor{385e}
@deffn {Method} _makeResult ()

This method returns the instance of @code{TestResult} used by @ref{385f,,run()}.
It is not intended to be called directly, but can be overridden in
subclasses to provide a custom @code{TestResult}.

@code{_makeResult()} instantiates the class or callable passed in the
@code{TextTestRunner} constructor as the @code{resultclass} argument. It
defaults to @ref{10bd,,TextTestResult} if no @code{resultclass} is provided.
The result class is instantiated with the following arguments:

@example
stream, descriptions, verbosity
@end example
@end deffn

@geindex run() (unittest.TextTestRunner method)
@anchor{library/unittest unittest TextTestRunner run}@anchor{385f}
@deffn {Method} run (test)

This method is the main public interface to the @code{TextTestRunner}. This
method takes a @ref{953,,TestSuite} or @ref{1ab,,TestCase} instance. A
@ref{d04,,TestResult} is created by calling
@ref{385e,,_makeResult()} and the test(s) are run and the
results printed to stdout.
@end deffn
@end deffn

@geindex main() (in module unittest)
@anchor{library/unittest unittest main}@anchor{b69}
@deffn {Function} unittest.main (module='__main__', defaultTest=None, argv=None, testRunner=None, testLoader=unittest.defaultTestLoader, exit=True, verbosity=1, failfast=None, catchbreak=None, buffer=None, warnings=None)

A command-line program that loads a set of tests from @emph{module} and runs them;
this is primarily for making test modules conveniently executable.
The simplest use for this function is to include the following line at the
end of a test script:

@example
if __name__ == '__main__':
    unittest.main()
@end example

You can run tests with more detailed information by passing in the verbosity
argument:

@example
if __name__ == '__main__':
    unittest.main(verbosity=2)
@end example

The @emph{defaultTest} argument is either the name of a single test or an
iterable of test names to run if no test names are specified via @emph{argv}.  If
not specified or @code{None} and no test names are provided via @emph{argv}, all
tests found in @emph{module} are run.

The @emph{argv} argument can be a list of options passed to the program, with the
first element being the program name.  If not specified or @code{None},
the values of @ref{e15,,sys.argv} are used.

The @emph{testRunner} argument can either be a test runner class or an already
created instance of it. By default @code{main} calls @ref{ee4,,sys.exit()} with
an exit code indicating success or failure of the tests run.

The @emph{testLoader} argument has to be a @ref{1ad,,TestLoader} instance,
and defaults to @ref{3847,,defaultTestLoader}.

@code{main} supports being used from the interactive interpreter by passing in the
argument @code{exit=False}. This displays the result on standard output without
calling @ref{ee4,,sys.exit()}:

@example
>>> from unittest import main
>>> main(module='test_module', exit=False)
@end example

The @emph{failfast}, @emph{catchbreak} and @emph{buffer} parameters have the same
effect as the same-name @ref{380d,,command-line options}.

The @emph{warnings} argument specifies the @ref{3aa,,warning filter}
that should be used while running the tests.  If it’s not specified, it will
remain @code{None} if a @code{-W} option is passed to @code{python}
(see @ref{1612,,Warning control}),
otherwise it will be set to @code{'default'}.

Calling @code{main} actually returns an instance of the @code{TestProgram} class.
This stores the result of the tests run as the @code{result} attribute.


Changed in version 3.1: The @emph{exit} parameter was added.



Changed in version 3.2: The @emph{verbosity}, @emph{failfast}, @emph{catchbreak}, @emph{buffer}
and @emph{warnings} parameters were added.



Changed in version 3.4: The @emph{defaultTest} parameter was changed to also accept an iterable of
test names.

@end deffn

@menu
* load_tests Protocol:: 

@end menu

@node load_tests Protocol,,,Loading and running tests
@anchor{library/unittest load-tests-protocol}@anchor{381d}
@subsubsection load_tests Protocol



New in version 3.2.


Modules or packages can customize how tests are loaded from them during normal
test runs or test discovery by implementing a function called @code{load_tests}.

If a test module defines @code{load_tests} it will be called by
@ref{1ae,,TestLoader.loadTestsFromModule()} with the following arguments:

@example
load_tests(loader, standard_tests, pattern)
@end example

where @emph{pattern} is passed straight through from @code{loadTestsFromModule}.  It
defaults to @code{None}.

It should return a @ref{953,,TestSuite}.

@emph{loader} is the instance of @ref{1ad,,TestLoader} doing the loading.
@emph{standard_tests} are the tests that would be loaded by default from the
module. It is common for test modules to only want to add or remove tests
from the standard set of tests.
The third argument is used when loading packages as part of test discovery.

A typical @code{load_tests} function that loads tests from a specific set of
@ref{1ab,,TestCase} classes may look like:

@example
test_cases = (TestCase1, TestCase2, TestCase3)

def load_tests(loader, tests, pattern):
    suite = TestSuite()
    for test_class in test_cases:
        tests = loader.loadTestsFromTestCase(test_class)
        suite.addTests(tests)
    return suite
@end example

If discovery is started in a directory containing a package, either from the
command line or by calling @ref{b6b,,TestLoader.discover()}, then the package
@code{__init__.py} will be checked for @code{load_tests}.  If that function does
not exist, discovery will recurse into the package as though it were just
another directory.  Otherwise, discovery of the package’s tests will be left up
to @code{load_tests} which is called with the following arguments:

@example
load_tests(loader, standard_tests, pattern)
@end example

This should return a @ref{953,,TestSuite} representing all the tests
from the package. (@code{standard_tests} will only contain tests
collected from @code{__init__.py}.)

Because the pattern is passed into @code{load_tests} the package is free to
continue (and potentially modify) test discovery. A ‘do nothing’
@code{load_tests} function for a test package would look like:

@example
def load_tests(loader, standard_tests, pattern):
    # top level directory cached on loader instance
    this_dir = os.path.dirname(__file__)
    package_tests = loader.discover(start_dir=this_dir, pattern=pattern)
    standard_tests.addTests(package_tests)
    return standard_tests
@end example


Changed in version 3.5: Discovery no longer checks package names for matching @emph{pattern} due to the
impossibility of package names matching the default pattern.


@node Class and Module Fixtures,Signal Handling,Classes and functions,unittest — Unit testing framework
@anchor{library/unittest class-and-module-fixtures}@anchor{382d}
@subsubsection Class and Module Fixtures


Class and module level fixtures are implemented in @ref{953,,TestSuite}. When
the test suite encounters a test from a new class then @code{tearDownClass()}
from the previous class (if there is one) is called, followed by
@code{setUpClass()} from the new class.

Similarly if a test is from a different module from the previous test then
@code{tearDownModule} from the previous module is run, followed by
@code{setUpModule} from the new module.

After all the tests have run the final @code{tearDownClass} and
@code{tearDownModule} are run.

Note that shared fixtures do not play well with [potential] features like test
parallelization and they break test isolation. They should be used with care.

The default ordering of tests created by the unittest test loaders is to group
all tests from the same modules and classes together. This will lead to
@code{setUpClass} / @code{setUpModule} (etc) being called exactly once per class and
module. If you randomize the order, so that tests from different modules and
classes are adjacent to each other, then these shared fixture functions may be
called multiple times in a single test run.

Shared fixtures are not intended to work with suites with non-standard
ordering. A @code{BaseTestSuite} still exists for frameworks that don’t want to
support shared fixtures.

If there are any exceptions raised during one of the shared fixture functions
the test is reported as an error. Because there is no corresponding test
instance an @code{_ErrorHolder} object (that has the same interface as a
@ref{1ab,,TestCase}) is created to represent the error. If you are just using
the standard unittest test runner then this detail doesn’t matter, but if you
are a framework author it may be relevant.

@menu
* setUpClass and tearDownClass:: 
* setUpModule and tearDownModule:: 

@end menu

@node setUpClass and tearDownClass,setUpModule and tearDownModule,,Class and Module Fixtures
@anchor{library/unittest setupclass-and-teardownclass}@anchor{3860}
@subsubsection setUpClass and tearDownClass


These must be implemented as class methods:

@example
import unittest

class Test(unittest.TestCase):
    @@classmethod
    def setUpClass(cls):
        cls._connection = createExpensiveConnectionObject()

    @@classmethod
    def tearDownClass(cls):
        cls._connection.destroy()
@end example

If you want the @code{setUpClass} and @code{tearDownClass} on base classes called
then you must call up to them yourself. The implementations in
@ref{1ab,,TestCase} are empty.

If an exception is raised during a @code{setUpClass} then the tests in the class
are not run and the @code{tearDownClass} is not run. Skipped classes will not
have @code{setUpClass} or @code{tearDownClass} run. If the exception is a
@ref{b6a,,SkipTest} exception then the class will be reported as having been skipped
instead of as an error.

@node setUpModule and tearDownModule,,setUpClass and tearDownClass,Class and Module Fixtures
@anchor{library/unittest setupmodule-and-teardownmodule}@anchor{3861}
@subsubsection setUpModule and tearDownModule


These should be implemented as functions:

@example
def setUpModule():
    createConnection()

def tearDownModule():
    closeConnection()
@end example

If an exception is raised in a @code{setUpModule} then none of the tests in the
module will be run and the @code{tearDownModule} will not be run. If the exception is a
@ref{b6a,,SkipTest} exception then the module will be reported as having been skipped
instead of as an error.

To add cleanup code that must be run even in the case of an exception, use
@code{addModuleCleanup}:

@geindex addModuleCleanup() (in module unittest)
@anchor{library/unittest unittest addModuleCleanup}@anchor{562}
@deffn {Function} unittest.addModuleCleanup (function, /, *args, **kwargs)

Add a function to be called after @code{tearDownModule()} to cleanup
resources used during the test class. Functions will be called in reverse
order to the order they are added (@abbr{LIFO}).
They are called with any arguments and keyword arguments passed into
@ref{562,,addModuleCleanup()} when they are added.

If @code{setUpModule()} fails, meaning that @code{tearDownModule()} is not
called, then any cleanup functions added will still be called.


New in version 3.8.

@end deffn

@geindex doModuleCleanups() (in module unittest)
@anchor{library/unittest unittest doModuleCleanups}@anchor{1088}
@deffn {Function} unittest.doModuleCleanups ()

This function is called unconditionally after @code{tearDownModule()}, or
after @code{setUpModule()} if @code{setUpModule()} raises an exception.

It is responsible for calling all the cleanup functions added by
@code{addCleanupModule()}. If you need cleanup functions to be called
@emph{prior} to @code{tearDownModule()} then you can call
@ref{1088,,doModuleCleanups()} yourself.

@ref{1088,,doModuleCleanups()} pops methods off the stack of cleanup
functions one at a time, so it can be called at any time.


New in version 3.8.

@end deffn

@node Signal Handling,,Class and Module Fixtures,unittest — Unit testing framework
@anchor{library/unittest signal-handling}@anchor{3810}
@subsubsection Signal Handling



New in version 3.2.


The @ref{ec6,,-c/--catch} command-line option to unittest,
along with the @code{catchbreak} parameter to @ref{b69,,unittest.main()}, provide
more friendly handling of control-C during a test run. With catch break
behavior enabled control-C will allow the currently running test to complete,
and the test run will then end and report all the results so far. A second
control-c will raise a @ref{4ca,,KeyboardInterrupt} in the usual way.

The control-c handling signal handler attempts to remain compatible with code or
tests that install their own @ref{2ca,,signal.SIGINT} handler. If the @code{unittest}
handler is called but @emph{isn’t} the installed @ref{2ca,,signal.SIGINT} handler,
i.e. it has been replaced by the system under test and delegated to, then it
calls the default handler. This will normally be the expected behavior by code
that replaces an installed handler and delegates to it. For individual tests
that need @code{unittest} control-c handling disabled the @ref{ec7,,removeHandler()}
decorator can be used.

There are a few utility functions for framework authors to enable control-c
handling functionality within test frameworks.

@geindex installHandler() (in module unittest)
@anchor{library/unittest unittest installHandler}@anchor{3862}
@deffn {Function} unittest.installHandler ()

Install the control-c handler. When a @ref{2ca,,signal.SIGINT} is received
(usually in response to the user pressing control-c) all registered results
have @ref{3852,,stop()} called.
@end deffn

@geindex registerResult() (in module unittest)
@anchor{library/unittest unittest registerResult}@anchor{3863}
@deffn {Function} unittest.registerResult (result)

Register a @ref{d04,,TestResult} object for control-c handling. Registering a
result stores a weak reference to it, so it doesn’t prevent the result from
being garbage collected.

Registering a @ref{d04,,TestResult} object has no side-effects if control-c
handling is not enabled, so test frameworks can unconditionally register
all results they create independently of whether or not handling is enabled.
@end deffn

@geindex removeResult() (in module unittest)
@anchor{library/unittest unittest removeResult}@anchor{3864}
@deffn {Function} unittest.removeResult (result)

Remove a registered result. Once a result has been removed then
@ref{3852,,stop()} will no longer be called on that result object in
response to a control-c.
@end deffn

@geindex removeHandler() (in module unittest)
@anchor{library/unittest unittest removeHandler}@anchor{ec7}
@deffn {Function} unittest.removeHandler (function=None)

When called without arguments this function removes the control-c handler
if it has been installed. This function can also be used as a test decorator
to temporarily remove the handler while the test is being executed:

@example
@@unittest.removeHandler
def test_signal_handling(self):
    ...
@end example
@end deffn

@node unittest mock — mock object library,unittest mock — getting started,unittest — Unit testing framework,Development Tools
@anchor{library/unittest mock doc}@anchor{3865}@anchor{library/unittest mock module-unittest mock}@anchor{123}@anchor{library/unittest mock unittest-mock-mock-object-library}@anchor{3866}
@subsection @code{unittest.mock} — mock object library


@geindex module; unittest.mock


New in version 3.3.


@strong{Source code:} Lib/unittest/mock.py@footnote{https://github.com/python/cpython/tree/main/Lib/unittest/mock.py}


__________________________________________________________________


@ref{123,,unittest.mock} is a library for testing in Python. It allows you to
replace parts of your system under test with mock objects and make assertions
about how they have been used.

@ref{123,,unittest.mock} provides a core @ref{561,,Mock} class removing the need to
create a host of stubs throughout your test suite. After performing an
action, you can make assertions about which methods / attributes were used
and arguments they were called with. You can also specify return values and
set needed attributes in the normal way.

Additionally, mock provides a @ref{a02,,patch()} decorator that handles patching
module and class level attributes within the scope of a test, along with
@ref{6d1,,sentinel} for creating unique objects. See the @ref{3867,,quick guide} for
some examples of how to use @ref{561,,Mock}, @ref{a01,,MagicMock} and
@ref{a02,,patch()}.

Mock is designed for use with @ref{122,,unittest} and
is based on the ‘action -> assertion’ pattern instead of ‘record -> replay’
used by many mocking frameworks.

There is a backport of @ref{123,,unittest.mock} for earlier versions of Python,
available as mock on PyPI@footnote{https://pypi.org/project/mock}.

@menu
* Quick Guide:: 
* The Mock Class:: 
* The patchers:: 
* MagicMock and magic method support:: 
* Helpers:: 

@end menu

@node Quick Guide,The Mock Class,,unittest mock — mock object library
@anchor{library/unittest mock quick-guide}@anchor{3867}
@subsubsection Quick Guide


@c class ProductionClass:
@c     def method(self, a, b, c):
@c         pass
@c 
@c class SomeClass:
@c     @staticmethod
@c     def static_method(args):
@c         return args
@c 
@c     @classmethod
@c     def class_method(cls, args):
@c         return args

@ref{561,,Mock} and @ref{a01,,MagicMock} objects create all attributes and
methods as you access them and store details of how they have been used. You
can configure them, to specify return values or limit what attributes are
available, and then make assertions about how they have been used:

@example
>>> from unittest.mock import MagicMock
>>> thing = ProductionClass()
>>> thing.method = MagicMock(return_value=3)
>>> thing.method(3, 4, 5, key='value')
3
>>> thing.method.assert_called_with(3, 4, 5, key='value')
@end example

@code{side_effect} allows you to perform side effects, including raising an
exception when a mock is called:

@example
>>> mock = Mock(side_effect=KeyError('foo'))
>>> mock()
Traceback (most recent call last):
 ...
KeyError: 'foo'
@end example

@example
>>> values = @{'a': 1, 'b': 2, 'c': 3@}
>>> def side_effect(arg):
...     return values[arg]
...
>>> mock.side_effect = side_effect
>>> mock('a'), mock('b'), mock('c')
(1, 2, 3)
>>> mock.side_effect = [5, 4, 3, 2, 1]
>>> mock(), mock(), mock()
(5, 4, 3)
@end example

Mock has many other ways you can configure it and control its behaviour. For
example the @emph{spec} argument configures the mock to take its specification
from another object. Attempting to access attributes or methods on the mock
that don’t exist on the spec will fail with an @ref{163,,AttributeError}.

The @ref{a02,,patch()} decorator / context manager makes it easy to mock classes or
objects in a module under test. The object you specify will be replaced with a
mock (or other object) during the test and restored when the test ends:

@example
>>> from unittest.mock import patch
>>> @@patch('module.ClassName2')
... @@patch('module.ClassName1')
... def test(MockClass1, MockClass2):
...     module.ClassName1()
...     module.ClassName2()
...     assert MockClass1 is module.ClassName1
...     assert MockClass2 is module.ClassName2
...     assert MockClass1.called
...     assert MockClass2.called
...
>>> test()
@end example

@cartouche
@quotation Note 
When you nest patch decorators the mocks are passed in to the decorated
function in the same order they applied (the normal @emph{Python} order that
decorators are applied). This means from the bottom up, so in the example
above the mock for @code{module.ClassName1} is passed in first.

With @ref{a02,,patch()} it matters that you patch objects in the namespace where they
are looked up. This is normally straightforward, but for a quick guide
read @ref{3868,,where to patch}.
@end quotation
@end cartouche

As well as a decorator @ref{a02,,patch()} can be used as a context manager in a with
statement:

@example
>>> with patch.object(ProductionClass, 'method', return_value=None) as mock_method:
...     thing = ProductionClass()
...     thing.method(1, 2, 3)
...
>>> mock_method.assert_called_once_with(1, 2, 3)
@end example

There is also @ref{129d,,patch.dict()} for setting values in a dictionary just
during a scope and restoring the dictionary to its original state when the test
ends:

@example
>>> foo = @{'key': 'value'@}
>>> original = foo.copy()
>>> with patch.dict(foo, @{'newkey': 'newvalue'@}, clear=True):
...     assert foo == @{'newkey': 'newvalue'@}
...
>>> assert foo == original
@end example

Mock supports the mocking of Python @ref{3869,,magic methods}. The
easiest way of using magic methods is with the @ref{a01,,MagicMock} class. It
allows you to do things like:

@example
>>> mock = MagicMock()
>>> mock.__str__.return_value = 'foobarbaz'
>>> str(mock)
'foobarbaz'
>>> mock.__str__.assert_called_with()
@end example

Mock allows you to assign functions (or other Mock instances) to magic methods
and they will be called appropriately. The @ref{a01,,MagicMock} class is just a Mock
variant that has all of the magic methods pre-created for you (well, all the
useful ones anyway).

The following is an example of using magic methods with the ordinary Mock
class:

@example
>>> mock = Mock()
>>> mock.__str__ = Mock(return_value='wheeeeee')
>>> str(mock)
'wheeeeee'
@end example

For ensuring that the mock objects in your tests have the same api as the
objects they are replacing, you can use @ref{386a,,auto-speccing}.
Auto-speccing can be done through the @emph{autospec} argument to patch, or the
@ref{122a,,create_autospec()} function. Auto-speccing creates mock objects that
have the same attributes and methods as the objects they are replacing, and
any functions and methods (including constructors) have the same call
signature as the real object.

This ensures that your mocks will fail in the same way as your production
code if they are used incorrectly:

@example
>>> from unittest.mock import create_autospec
>>> def function(a, b, c):
...     pass
...
>>> mock_function = create_autospec(function, return_value='fishy')
>>> mock_function(1, 2, 3)
'fishy'
>>> mock_function.assert_called_once_with(1, 2, 3)
>>> mock_function('wrong arguments')
Traceback (most recent call last):
 ...
TypeError: <lambda>() takes exactly 3 arguments (1 given)
@end example

@ref{122a,,create_autospec()} can also be used on classes, where it copies the signature of
the @code{__init__} method, and on callable objects where it copies the signature of
the @code{__call__} method.

@node The Mock Class,The patchers,Quick Guide,unittest mock — mock object library
@anchor{library/unittest mock the-mock-class}@anchor{386b}
@subsubsection The Mock Class


@c import asyncio
@c import inspect
@c import unittest
@c from unittest.mock import sentinel, DEFAULT, ANY
@c from unittest.mock import patch, call, Mock, MagicMock, PropertyMock, AsyncMock
@c from unittest.mock import mock_open

@ref{561,,Mock} is a flexible mock object intended to replace the use of stubs and
test doubles throughout your code. Mocks are callable and create attributes as
new mocks when you access them @footnote{@w{(1)} 
The only exceptions are magic methods and attributes (those that have
leading and trailing double underscores). Mock doesn’t create these but
instead raises an @ref{163,,AttributeError}. This is because the interpreter
will often implicitly request these methods, and gets @emph{very} confused to
get a new Mock object when it expects a magic method. If you need magic
method support see @ref{3869,,magic methods}.
}. Accessing the same attribute will always
return the same mock. Mocks record how you use them, allowing you to make
assertions about what your code has done to them.

@ref{a01,,MagicMock} is a subclass of @ref{561,,Mock} with all the magic methods
pre-created and ready to use. There are also non-callable variants, useful
when you are mocking out objects that aren’t callable:
@ref{386c,,NonCallableMock} and @ref{386d,,NonCallableMagicMock}

The @ref{a02,,patch()} decorators makes it easy to temporarily replace classes
in a particular module with a @ref{561,,Mock} object. By default @ref{a02,,patch()} will create
a @ref{a01,,MagicMock} for you. You can specify an alternative class of @ref{561,,Mock} using
the @emph{new_callable} argument to @ref{a02,,patch()}.

@geindex Mock (class in unittest.mock)
@anchor{library/unittest mock unittest mock Mock}@anchor{561}
@deffn {Class} unittest.mock.Mock (spec=None, side_effect=None, return_value=DEFAULT, wraps=None, name=None, spec_set=None, unsafe=False, **kwargs)

Create a new @ref{561,,Mock} object. @ref{561,,Mock} takes several optional arguments
that specify the behaviour of the Mock object:


@itemize *

@item 
@emph{spec}: This can be either a list of strings or an existing object (a
class or instance) that acts as the specification for the mock object. If
you pass in an object then a list of strings is formed by calling dir on
the object (excluding unsupported magic attributes and methods).
Accessing any attribute not in this list will raise an @ref{163,,AttributeError}.

If @emph{spec} is an object (rather than a list of strings) then
@ref{fff,,__class__} returns the class of the spec object. This
allows mocks to pass @ref{236,,isinstance()} tests.

@item 
@emph{spec_set}: A stricter variant of @emph{spec}. If used, attempting to @emph{set}
or get an attribute on the mock that isn’t on the object passed as
@emph{spec_set} will raise an @ref{163,,AttributeError}.

@item 
@emph{side_effect}: A function to be called whenever the Mock is called. See
the @ref{386e,,side_effect} attribute. Useful for raising exceptions or
dynamically changing return values. The function is called with the same
arguments as the mock, and unless it returns @ref{386f,,DEFAULT}, the return
value of this function is used as the return value.

Alternatively @emph{side_effect} can be an exception class or instance. In
this case the exception will be raised when the mock is called.

If @emph{side_effect} is an iterable then each call to the mock will return
the next value from the iterable.

A @emph{side_effect} can be cleared by setting it to @code{None}.

@item 
@emph{return_value}: The value returned when the mock is called. By default
this is a new Mock (created on first access). See the
@ref{3870,,return_value} attribute.

@item 
@emph{unsafe}: By default, accessing any attribute whose name starts with
@emph{assert}, @emph{assret}, @emph{asert}, @emph{aseert} or @emph{assrt} will raise an
@ref{163,,AttributeError}. Passing @code{unsafe=True} will allow access to
these attributes.


New in version 3.5.


@item 
@emph{wraps}: Item for the mock object to wrap. If @emph{wraps} is not @code{None} then
calling the Mock will pass the call through to the wrapped object
(returning the real result). Attribute access on the mock will return a
Mock object that wraps the corresponding attribute of the wrapped
object (so attempting to access an attribute that doesn’t exist will
raise an @ref{163,,AttributeError}).

If the mock has an explicit @emph{return_value} set then calls are not passed
to the wrapped object and the @emph{return_value} is returned instead.

@item 
@emph{name}: If the mock has a name then it will be used in the repr of the
mock. This can be useful for debugging. The name is propagated to child
mocks.
@end itemize

Mocks can also be called with arbitrary keyword arguments. These will be
used to set attributes on the mock after it is created. See the
@ref{3871,,configure_mock()} method for details.

@geindex assert_called() (unittest.mock.Mock method)
@anchor{library/unittest mock unittest mock Mock assert_called}@anchor{843}
@deffn {Method} assert_called ()

Assert that the mock was called at least once.

@example
>>> mock = Mock()
>>> mock.method()
<Mock name='mock.method()' id='...'>
>>> mock.method.assert_called()
@end example


New in version 3.6.

@end deffn

@geindex assert_called_once() (unittest.mock.Mock method)
@anchor{library/unittest mock unittest mock Mock assert_called_once}@anchor{844}
@deffn {Method} assert_called_once ()

Assert that the mock was called exactly once.

@example
>>> mock = Mock()
>>> mock.method()
<Mock name='mock.method()' id='...'>
>>> mock.method.assert_called_once()
>>> mock.method()
<Mock name='mock.method()' id='...'>
>>> mock.method.assert_called_once()
Traceback (most recent call last):
...
AssertionError: Expected 'method' to have been called once. Called 2 times.
@end example


New in version 3.6.

@end deffn

@geindex assert_called_with() (unittest.mock.Mock method)
@anchor{library/unittest mock unittest mock Mock assert_called_with}@anchor{12cc}
@deffn {Method} assert_called_with (*args, **kwargs)

This method is a convenient way of asserting that the last call has been
made in a particular way:

@example
>>> mock = Mock()
>>> mock.method(1, 2, 3, test='wow')
<Mock name='mock.method()' id='...'>
>>> mock.method.assert_called_with(1, 2, 3, test='wow')
@end example
@end deffn

@geindex assert_called_once_with() (unittest.mock.Mock method)
@anchor{library/unittest mock unittest mock Mock assert_called_once_with}@anchor{3872}
@deffn {Method} assert_called_once_with (*args, **kwargs)

Assert that the mock was called exactly once and that call was with the
specified arguments.

@example
>>> mock = Mock(return_value=None)
>>> mock('foo', bar='baz')
>>> mock.assert_called_once_with('foo', bar='baz')
>>> mock('other', bar='values')
>>> mock.assert_called_once_with('other', bar='values')
Traceback (most recent call last):
  ...
AssertionError: Expected 'mock' to be called once. Called 2 times.
@end example
@end deffn

@geindex assert_any_call() (unittest.mock.Mock method)
@anchor{library/unittest mock unittest mock Mock assert_any_call}@anchor{3873}
@deffn {Method} assert_any_call (*args, **kwargs)

assert the mock has been called with the specified arguments.

The assert passes if the mock has @emph{ever} been called, unlike
@ref{12cc,,assert_called_with()} and @ref{3872,,assert_called_once_with()} that
only pass if the call is the most recent one, and in the case of
@ref{3872,,assert_called_once_with()} it must also be the only call.

@example
>>> mock = Mock(return_value=None)
>>> mock(1, 2, arg='thing')
>>> mock('some', 'thing', 'else')
>>> mock.assert_any_call(1, 2, arg='thing')
@end example
@end deffn

@geindex assert_has_calls() (unittest.mock.Mock method)
@anchor{library/unittest mock unittest mock Mock assert_has_calls}@anchor{3874}
@deffn {Method} assert_has_calls (calls, any_order=False)

assert the mock has been called with the specified calls.
The @ref{3875,,mock_calls} list is checked for the calls.

If @emph{any_order} is false then the calls must be
sequential. There can be extra calls before or after the
specified calls.

If @emph{any_order} is true then the calls can be in any order, but
they must all appear in @ref{3875,,mock_calls}.

@example
>>> mock = Mock(return_value=None)
>>> mock(1)
>>> mock(2)
>>> mock(3)
>>> mock(4)
>>> calls = [call(2), call(3)]
>>> mock.assert_has_calls(calls)
>>> calls = [call(4), call(2), call(3)]
>>> mock.assert_has_calls(calls, any_order=True)
@end example
@end deffn

@geindex assert_not_called() (unittest.mock.Mock method)
@anchor{library/unittest mock unittest mock Mock assert_not_called}@anchor{a00}
@deffn {Method} assert_not_called ()

Assert the mock was never called.

@example
>>> m = Mock()
>>> m.hello.assert_not_called()
>>> obj = m.hello()
>>> m.hello.assert_not_called()
Traceback (most recent call last):
  ...
AssertionError: Expected 'hello' to not have been called. Called 1 times.
@end example


New in version 3.5.

@end deffn

@geindex reset_mock() (unittest.mock.Mock method)
@anchor{library/unittest mock unittest mock Mock reset_mock}@anchor{845}
@deffn {Method} reset_mock (*, return_value=False, side_effect=False)

The reset_mock method resets all the call attributes on a mock object:

@example
>>> mock = Mock(return_value=None)
>>> mock('hello')
>>> mock.called
True
>>> mock.reset_mock()
>>> mock.called
False
@end example


Changed in version 3.6: Added two keyword only argument to the reset_mock function.


This can be useful where you want to make a series of assertions that
reuse the same object. Note that @ref{845,,reset_mock()} @emph{doesn’t} clear the
return value, @ref{386e,,side_effect} or any child attributes you have
set using normal assignment by default. In case you want to reset
@emph{return_value} or @ref{386e,,side_effect}, then pass the corresponding
parameter as @code{True}. Child mocks and the return value mock
(if any) are reset as well.

@cartouche
@quotation Note 
@emph{return_value}, and @ref{386e,,side_effect} are keyword only
argument.
@end quotation
@end cartouche
@end deffn

@geindex mock_add_spec() (unittest.mock.Mock method)
@anchor{library/unittest mock unittest mock Mock mock_add_spec}@anchor{3876}
@deffn {Method} mock_add_spec (spec, spec_set=False)

Add a spec to a mock. @emph{spec} can either be an object or a
list of strings. Only attributes on the @emph{spec} can be fetched as
attributes from the mock.

If @emph{spec_set} is true then only attributes on the spec can be set.
@end deffn

@geindex attach_mock() (unittest.mock.Mock method)
@anchor{library/unittest mock unittest mock Mock attach_mock}@anchor{3877}
@deffn {Method} attach_mock (mock, attribute)

Attach a mock as an attribute of this one, replacing its name and
parent. Calls to the attached mock will be recorded in the
@ref{3878,,method_calls} and @ref{3875,,mock_calls} attributes of this one.
@end deffn

@geindex configure_mock() (unittest.mock.Mock method)
@anchor{library/unittest mock unittest mock Mock configure_mock}@anchor{3871}
@deffn {Method} configure_mock (**kwargs)

Set attributes on the mock through keyword arguments.

Attributes plus return values and side effects can be set on child
mocks using standard dot notation and unpacking a dictionary in the
method call:

@example
>>> mock = Mock()
>>> attrs = @{'method.return_value': 3, 'other.side_effect': KeyError@}
>>> mock.configure_mock(**attrs)
>>> mock.method()
3
>>> mock.other()
Traceback (most recent call last):
  ...
KeyError
@end example

The same thing can be achieved in the constructor call to mocks:

@example
>>> attrs = @{'method.return_value': 3, 'other.side_effect': KeyError@}
>>> mock = Mock(some_attribute='eggs', **attrs)
>>> mock.some_attribute
'eggs'
>>> mock.method()
3
>>> mock.other()
Traceback (most recent call last):
  ...
KeyError
@end example

@ref{3871,,configure_mock()} exists to make it easier to do configuration
after the mock has been created.
@end deffn

@geindex __dir__() (unittest.mock.Mock method)
@anchor{library/unittest mock unittest mock Mock __dir__}@anchor{3879}
@deffn {Method} __dir__ ()

@ref{561,,Mock} objects limit the results of @code{dir(some_mock)} to useful results.
For mocks with a @emph{spec} this includes all the permitted attributes
for the mock.

See @ref{387a,,FILTER_DIR} for what this filtering does, and how to
switch it off.
@end deffn

@geindex _get_child_mock() (unittest.mock.Mock method)
@anchor{library/unittest mock unittest mock Mock _get_child_mock}@anchor{387b}
@deffn {Method} _get_child_mock (**kw)

Create the child mocks for attributes and return value.
By default child mocks will be the same type as the parent.
Subclasses of Mock may want to override this to customize the way
child mocks are made.

For non-callable mocks the callable variant will be used (rather than
any custom subclass).
@end deffn

@geindex called (unittest.mock.Mock attribute)
@anchor{library/unittest mock unittest mock Mock called}@anchor{387c}
@deffn {Attribute} called

A boolean representing whether or not the mock object has been called:

@example
>>> mock = Mock(return_value=None)
>>> mock.called
False
>>> mock()
>>> mock.called
True
@end example
@end deffn

@geindex call_count (unittest.mock.Mock attribute)
@anchor{library/unittest mock unittest mock Mock call_count}@anchor{387d}
@deffn {Attribute} call_count

An integer telling you how many times the mock object has been called:

@example
>>> mock = Mock(return_value=None)
>>> mock.call_count
0
>>> mock()
>>> mock()
>>> mock.call_count
2
@end example
@end deffn

@geindex return_value (unittest.mock.Mock attribute)
@anchor{library/unittest mock unittest mock Mock return_value}@anchor{3870}
@deffn {Attribute} return_value

Set this to configure the value returned by calling the mock:

@example
>>> mock = Mock()
>>> mock.return_value = 'fish'
>>> mock()
'fish'
@end example

The default return value is a mock object and you can configure it in
the normal way:

@example
>>> mock = Mock()
>>> mock.return_value.attribute = sentinel.Attribute
>>> mock.return_value()
<Mock name='mock()()' id='...'>
>>> mock.return_value.assert_called_with()
@end example

@ref{3870,,return_value} can also be set in the constructor:

@example
>>> mock = Mock(return_value=3)
>>> mock.return_value
3
>>> mock()
3
@end example
@end deffn

@geindex side_effect (unittest.mock.Mock attribute)
@anchor{library/unittest mock unittest mock Mock side_effect}@anchor{386e}
@deffn {Attribute} side_effect

This can either be a function to be called when the mock is called,
an iterable or an exception (class or instance) to be raised.

If you pass in a function it will be called with same arguments as the
mock and unless the function returns the @ref{386f,,DEFAULT} singleton the
call to the mock will then return whatever the function returns. If the
function returns @ref{386f,,DEFAULT} then the mock will return its normal
value (from the @ref{3870,,return_value}).

If you pass in an iterable, it is used to retrieve an iterator which
must yield a value on every call.  This value can either be an exception
instance to be raised, or a value to be returned from the call to the
mock (@ref{386f,,DEFAULT} handling is identical to the function case).

An example of a mock that raises an exception (to test exception
handling of an API):

@example
>>> mock = Mock()
>>> mock.side_effect = Exception('Boom!')
>>> mock()
Traceback (most recent call last):
  ...
Exception: Boom!
@end example

Using @ref{386e,,side_effect} to return a sequence of values:

@example
>>> mock = Mock()
>>> mock.side_effect = [3, 2, 1]
>>> mock(), mock(), mock()
(3, 2, 1)
@end example

Using a callable:

@example
>>> mock = Mock(return_value=3)
>>> def side_effect(*args, **kwargs):
...     return DEFAULT
...
>>> mock.side_effect = side_effect
>>> mock()
3
@end example

@ref{386e,,side_effect} can be set in the constructor. Here’s an example that
adds one to the value the mock is called with and returns it:

@example
>>> side_effect = lambda value: value + 1
>>> mock = Mock(side_effect=side_effect)
>>> mock(3)
4
>>> mock(-8)
-7
@end example

Setting @ref{386e,,side_effect} to @code{None} clears it:

@example
>>> m = Mock(side_effect=KeyError, return_value=3)
>>> m()
Traceback (most recent call last):
 ...
KeyError
>>> m.side_effect = None
>>> m()
3
@end example
@end deffn

@geindex call_args (unittest.mock.Mock attribute)
@anchor{library/unittest mock unittest mock Mock call_args}@anchor{120f}
@deffn {Attribute} call_args

This is either @code{None} (if the mock hasn’t been called), or the
arguments that the mock was last called with. This will be in the
form of a tuple: the first member, which can also be accessed through
the @code{args} property, is any ordered arguments the mock was
called with (or an empty tuple) and the second member, which can
also be accessed through the @code{kwargs} property, is any keyword
arguments (or an empty dictionary).

@example
>>> mock = Mock(return_value=None)
>>> print(mock.call_args)
None
>>> mock()
>>> mock.call_args
call()
>>> mock.call_args == ()
True
>>> mock(3, 4)
>>> mock.call_args
call(3, 4)
>>> mock.call_args == ((3, 4),)
True
>>> mock.call_args.args
(3, 4)
>>> mock.call_args.kwargs
@{@}
>>> mock(3, 4, 5, key='fish', next='w00t!')
>>> mock.call_args
call(3, 4, 5, key='fish', next='w00t!')
>>> mock.call_args.args
(3, 4, 5)
>>> mock.call_args.kwargs
@{'key': 'fish', 'next': 'w00t!'@}
@end example

@ref{120f,,call_args}, along with members of the lists @ref{387e,,call_args_list},
@ref{3878,,method_calls} and @ref{3875,,mock_calls} are @ref{1265,,call} objects.
These are tuples, so they can be unpacked to get at the individual
arguments and make more complex assertions. See
@ref{387f,,calls as tuples}.


Changed in version 3.8: Added @code{args} and @code{kwargs} properties.

@end deffn

@geindex call_args_list (unittest.mock.Mock attribute)
@anchor{library/unittest mock unittest mock Mock call_args_list}@anchor{387e}
@deffn {Attribute} call_args_list

This is a list of all the calls made to the mock object in sequence
(so the length of the list is the number of times it has been
called). Before any calls have been made it is an empty list. The
@ref{1265,,call} object can be used for conveniently constructing lists of
calls to compare with @ref{387e,,call_args_list}.

@example
>>> mock = Mock(return_value=None)
>>> mock()
>>> mock(3, 4)
>>> mock(key='fish', next='w00t!')
>>> mock.call_args_list
[call(), call(3, 4), call(key='fish', next='w00t!')]
>>> expected = [(), ((3, 4),), (@{'key': 'fish', 'next': 'w00t!'@},)]
>>> mock.call_args_list == expected
True
@end example

Members of @ref{387e,,call_args_list} are @ref{1265,,call} objects. These can be
unpacked as tuples to get at the individual arguments. See
@ref{387f,,calls as tuples}.
@end deffn

@geindex method_calls (unittest.mock.Mock attribute)
@anchor{library/unittest mock unittest mock Mock method_calls}@anchor{3878}
@deffn {Attribute} method_calls

As well as tracking calls to themselves, mocks also track calls to
methods and attributes, and @emph{their} methods and attributes:

@example
>>> mock = Mock()
>>> mock.method()
<Mock name='mock.method()' id='...'>
>>> mock.property.method.attribute()
<Mock name='mock.property.method.attribute()' id='...'>
>>> mock.method_calls
[call.method(), call.property.method.attribute()]
@end example

Members of @ref{3878,,method_calls} are @ref{1265,,call} objects. These can be
unpacked as tuples to get at the individual arguments. See
@ref{387f,,calls as tuples}.
@end deffn

@geindex mock_calls (unittest.mock.Mock attribute)
@anchor{library/unittest mock unittest mock Mock mock_calls}@anchor{3875}
@deffn {Attribute} mock_calls

@ref{3875,,mock_calls} records @emph{all} calls to the mock object, its methods,
magic methods @emph{and} return value mocks.

@example
>>> mock = MagicMock()
>>> result = mock(1, 2, 3)
>>> mock.first(a=3)
<MagicMock name='mock.first()' id='...'>
>>> mock.second()
<MagicMock name='mock.second()' id='...'>
>>> int(mock)
1
>>> result(1)
<MagicMock name='mock()()' id='...'>
>>> expected = [call(1, 2, 3), call.first(a=3), call.second(),
... call.__int__(), call()(1)]
>>> mock.mock_calls == expected
True
@end example

Members of @ref{3875,,mock_calls} are @ref{1265,,call} objects. These can be
unpacked as tuples to get at the individual arguments. See
@ref{387f,,calls as tuples}.

@cartouche
@quotation Note 
The way @ref{3875,,mock_calls} are recorded means that where nested
calls are made, the parameters of ancestor calls are not recorded
and so will always compare equal:

@example
>>> mock = MagicMock()
>>> mock.top(a=3).bottom()
<MagicMock name='mock.top().bottom()' id='...'>
>>> mock.mock_calls
[call.top(a=3), call.top().bottom()]
>>> mock.mock_calls[-1] == call.top(a=-1).bottom()
True
@end example
@end quotation
@end cartouche
@end deffn

@geindex __class__ (unittest.mock.Mock attribute)
@anchor{library/unittest mock unittest mock Mock __class__}@anchor{3880}
@deffn {Attribute} __class__

Normally the @ref{3880,,__class__} attribute of an object will return its type.
For a mock object with a @code{spec}, @code{__class__} returns the spec class
instead. This allows mock objects to pass @ref{236,,isinstance()} tests for the
object they are replacing / masquerading as:

@example
>>> mock = Mock(spec=3)
>>> isinstance(mock, int)
True
@end example

@ref{3880,,__class__} is assignable to, this allows a mock to pass an
@ref{236,,isinstance()} check without forcing you to use a spec:

@example
>>> mock = Mock()
>>> mock.__class__ = dict
>>> isinstance(mock, dict)
True
@end example
@end deffn
@end deffn

@geindex NonCallableMock (class in unittest.mock)
@anchor{library/unittest mock unittest mock NonCallableMock}@anchor{386c}
@deffn {Class} unittest.mock.NonCallableMock (spec=None, wraps=None, name=None, spec_set=None, **kwargs)

A non-callable version of @ref{561,,Mock}. The constructor parameters have the same
meaning of @ref{561,,Mock}, with the exception of @emph{return_value} and @emph{side_effect}
which have no meaning on a non-callable mock.
@end deffn

Mock objects that use a class or an instance as a @code{spec} or
@code{spec_set} are able to pass @ref{236,,isinstance()} tests:

@example
>>> mock = Mock(spec=SomeClass)
>>> isinstance(mock, SomeClass)
True
>>> mock = Mock(spec_set=SomeClass())
>>> isinstance(mock, SomeClass)
True
@end example

The @ref{561,,Mock} classes have support for mocking magic methods. See @ref{3869,,magic methods} for the full details.

The mock classes and the @ref{a02,,patch()} decorators all take arbitrary keyword
arguments for configuration. For the @ref{a02,,patch()} decorators the keywords are
passed to the constructor of the mock being created. The keyword arguments
are for configuring attributes of the mock:

@example
>>> m = MagicMock(attribute=3, other='fish')
>>> m.attribute
3
>>> m.other
'fish'
@end example

The return value and side effect of child mocks can be set in the same way,
using dotted notation. As you can’t use dotted names directly in a call you
have to create a dictionary and unpack it using @code{**}:

@example
>>> attrs = @{'method.return_value': 3, 'other.side_effect': KeyError@}
>>> mock = Mock(some_attribute='eggs', **attrs)
>>> mock.some_attribute
'eggs'
>>> mock.method()
3
>>> mock.other()
Traceback (most recent call last):
  ...
KeyError
@end example

A callable mock which was created with a @emph{spec} (or a @emph{spec_set}) will
introspect the specification object’s signature when matching calls to
the mock.  Therefore, it can match the actual call’s arguments regardless
of whether they were passed positionally or by name:

@example
>>> def f(a, b, c): pass
...
>>> mock = Mock(spec=f)
>>> mock(1, 2, c=3)
<Mock name='mock()' id='140161580456576'>
>>> mock.assert_called_with(1, 2, 3)
>>> mock.assert_called_with(a=1, b=2, c=3)
@end example

This applies to @ref{12cc,,assert_called_with()},
@ref{3872,,assert_called_once_with()}, @ref{3874,,assert_has_calls()} and
@ref{3873,,assert_any_call()}.  When @ref{386a,,Autospeccing}, it will also
apply to method calls on the mock object.

@quotation


Changed in version 3.4: Added signature introspection on specced and autospecced mock objects.

@end quotation

@geindex PropertyMock (class in unittest.mock)
@anchor{library/unittest mock unittest mock PropertyMock}@anchor{3881}
@deffn {Class} unittest.mock.PropertyMock (*args, **kwargs)

A mock intended to be used as a property, or other descriptor, on a class.
@ref{3881,,PropertyMock} provides @code{__get__()} and @code{__set__()} methods
so you can specify a return value when it is fetched.

Fetching a @ref{3881,,PropertyMock} instance from an object calls the mock, with
no args. Setting it calls the mock with the value being set.

@example
>>> class Foo:
...     @@property
...     def foo(self):
...         return 'something'
...     @@foo.setter
...     def foo(self, value):
...         pass
...
>>> with patch('__main__.Foo.foo', new_callable=PropertyMock) as mock_foo:
...     mock_foo.return_value = 'mockity-mock'
...     this_foo = Foo()
...     print(this_foo.foo)
...     this_foo.foo = 6
...
mockity-mock
>>> mock_foo.mock_calls
[call(), call(6)]
@end example
@end deffn

Because of the way mock attributes are stored you can’t directly attach a
@ref{3881,,PropertyMock} to a mock object. Instead you can attach it to the mock type
object:

@example
>>> m = MagicMock()
>>> p = PropertyMock(return_value=3)
>>> type(m).foo = p
>>> m.foo
3
>>> p.assert_called_once_with()
@end example

@geindex AsyncMock (class in unittest.mock)
@anchor{library/unittest mock unittest mock AsyncMock}@anchor{560}
@deffn {Class} unittest.mock.AsyncMock (spec=None, side_effect=None, return_value=DEFAULT, wraps=None, name=None, spec_set=None, unsafe=False, **kwargs)

An asynchronous version of @ref{a01,,MagicMock}. The @ref{560,,AsyncMock} object will
behave so the object is recognized as an async function, and the result of a
call is an awaitable.

@example
>>> mock = AsyncMock()
>>> asyncio.iscoroutinefunction(mock)
True
>>> inspect.isawaitable(mock())  
True
@end example

The result of @code{mock()} is an async function which will have the outcome
of @code{side_effect} or @code{return_value} after it has been awaited:


@itemize -

@item 
if @code{side_effect} is a function, the async function will return the
result of that function,

@item 
if @code{side_effect} is an exception, the async function will raise the
exception,

@item 
if @code{side_effect} is an iterable, the async function will return the
next value of the iterable, however, if the sequence of result is
exhausted, @code{StopAsyncIteration} is raised immediately,

@item 
if @code{side_effect} is not defined, the async function will return the
value defined by @code{return_value}, hence, by default, the async function
returns a new @ref{560,,AsyncMock} object.
@end itemize

Setting the @emph{spec} of a @ref{561,,Mock} or @ref{a01,,MagicMock} to an async function
will result in a coroutine object being returned after calling.

@example
>>> async def async_func(): pass
...
>>> mock = MagicMock(async_func)
>>> mock
<MagicMock spec='function' id='...'>
>>> mock()  
<coroutine object AsyncMockMixin._mock_call at ...>
@end example

Setting the @emph{spec} of a @ref{561,,Mock}, @ref{a01,,MagicMock}, or @ref{560,,AsyncMock}
to a class with asynchronous and synchronous functions will automatically
detect the synchronous functions and set them as @ref{a01,,MagicMock} (if the
parent mock is @ref{560,,AsyncMock} or @ref{a01,,MagicMock}) or @ref{561,,Mock} (if
the parent mock is @ref{561,,Mock}). All asynchronous functions will be
@ref{560,,AsyncMock}.

@example
>>> class ExampleClass:
...     def sync_foo():
...         pass
...     async def async_foo():
...         pass
...
>>> a_mock = AsyncMock(ExampleClass)
>>> a_mock.sync_foo
<MagicMock name='mock.sync_foo' id='...'>
>>> a_mock.async_foo
<AsyncMock name='mock.async_foo' id='...'>
>>> mock = Mock(ExampleClass)
>>> mock.sync_foo
<Mock name='mock.sync_foo' id='...'>
>>> mock.async_foo
<AsyncMock name='mock.async_foo' id='...'>
@end example


New in version 3.8.


@geindex assert_awaited() (unittest.mock.AsyncMock method)
@anchor{library/unittest mock unittest mock AsyncMock assert_awaited}@anchor{3882}
@deffn {Method} assert_awaited ()

Assert that the mock was awaited at least once. Note that this is separate
from the object having been called, the @code{await} keyword must be used:

@example
>>> mock = AsyncMock()
>>> async def main(coroutine_mock):
...     await coroutine_mock
...
>>> coroutine_mock = mock()
>>> mock.called
True
>>> mock.assert_awaited()
Traceback (most recent call last):
...
AssertionError: Expected mock to have been awaited.
>>> asyncio.run(main(coroutine_mock))
>>> mock.assert_awaited()
@end example
@end deffn

@geindex assert_awaited_once() (unittest.mock.AsyncMock method)
@anchor{library/unittest mock unittest mock AsyncMock assert_awaited_once}@anchor{3883}
@deffn {Method} assert_awaited_once ()

Assert that the mock was awaited exactly once.

@example
>>> mock = AsyncMock()
>>> async def main():
...     await mock()
...
>>> asyncio.run(main())
>>> mock.assert_awaited_once()
>>> asyncio.run(main())
>>> mock.method.assert_awaited_once()
Traceback (most recent call last):
...
AssertionError: Expected mock to have been awaited once. Awaited 2 times.
@end example
@end deffn

@geindex assert_awaited_with() (unittest.mock.AsyncMock method)
@anchor{library/unittest mock unittest mock AsyncMock assert_awaited_with}@anchor{3884}
@deffn {Method} assert_awaited_with (*args, **kwargs)

Assert that the last await was with the specified arguments.

@example
>>> mock = AsyncMock()
>>> async def main(*args, **kwargs):
...     await mock(*args, **kwargs)
...
>>> asyncio.run(main('foo', bar='bar'))
>>> mock.assert_awaited_with('foo', bar='bar')
>>> mock.assert_awaited_with('other')
Traceback (most recent call last):
...
AssertionError: expected call not found.
Expected: mock('other')
Actual: mock('foo', bar='bar')
@end example
@end deffn

@geindex assert_awaited_once_with() (unittest.mock.AsyncMock method)
@anchor{library/unittest mock unittest mock AsyncMock assert_awaited_once_with}@anchor{3885}
@deffn {Method} assert_awaited_once_with (*args, **kwargs)

Assert that the mock was awaited exactly once and with the specified
arguments.

@example
>>> mock = AsyncMock()
>>> async def main(*args, **kwargs):
...     await mock(*args, **kwargs)
...
>>> asyncio.run(main('foo', bar='bar'))
>>> mock.assert_awaited_once_with('foo', bar='bar')
>>> asyncio.run(main('foo', bar='bar'))
>>> mock.assert_awaited_once_with('foo', bar='bar')
Traceback (most recent call last):
...
AssertionError: Expected mock to have been awaited once. Awaited 2 times.
@end example
@end deffn

@geindex assert_any_await() (unittest.mock.AsyncMock method)
@anchor{library/unittest mock unittest mock AsyncMock assert_any_await}@anchor{3886}
@deffn {Method} assert_any_await (*args, **kwargs)

Assert the mock has ever been awaited with the specified arguments.

@example
>>> mock = AsyncMock()
>>> async def main(*args, **kwargs):
...     await mock(*args, **kwargs)
...
>>> asyncio.run(main('foo', bar='bar'))
>>> asyncio.run(main('hello'))
>>> mock.assert_any_await('foo', bar='bar')
>>> mock.assert_any_await('other')
Traceback (most recent call last):
...
AssertionError: mock('other') await not found
@end example
@end deffn

@geindex assert_has_awaits() (unittest.mock.AsyncMock method)
@anchor{library/unittest mock unittest mock AsyncMock assert_has_awaits}@anchor{3887}
@deffn {Method} assert_has_awaits (calls, any_order=False)

Assert the mock has been awaited with the specified calls.
The @ref{120e,,await_args_list} list is checked for the awaits.

If @emph{any_order} is false then the awaits must be
sequential. There can be extra calls before or after the
specified awaits.

If @emph{any_order} is true then the awaits can be in any order, but
they must all appear in @ref{120e,,await_args_list}.

@example
>>> mock = AsyncMock()
>>> async def main(*args, **kwargs):
...     await mock(*args, **kwargs)
...
>>> calls = [call("foo"), call("bar")]
>>> mock.assert_has_awaits(calls)
Traceback (most recent call last):
...
AssertionError: Awaits not found.
Expected: [call('foo'), call('bar')]
Actual: []
>>> asyncio.run(main('foo'))
>>> asyncio.run(main('bar'))
>>> mock.assert_has_awaits(calls)
@end example
@end deffn

@geindex assert_not_awaited() (unittest.mock.AsyncMock method)
@anchor{library/unittest mock unittest mock AsyncMock assert_not_awaited}@anchor{3888}
@deffn {Method} assert_not_awaited ()

Assert that the mock was never awaited.

@example
>>> mock = AsyncMock()
>>> mock.assert_not_awaited()
@end example
@end deffn

@geindex reset_mock() (unittest.mock.AsyncMock method)
@anchor{library/unittest mock unittest mock AsyncMock reset_mock}@anchor{3889}
@deffn {Method} reset_mock (*args, **kwargs)

See @ref{845,,Mock.reset_mock()}. Also sets @ref{388a,,await_count} to 0,
@ref{388b,,await_args} to None, and clears the @ref{120e,,await_args_list}.
@end deffn

@geindex await_count (unittest.mock.AsyncMock attribute)
@anchor{library/unittest mock unittest mock AsyncMock await_count}@anchor{388a}
@deffn {Attribute} await_count

An integer keeping track of how many times the mock object has been awaited.

@example
>>> mock = AsyncMock()
>>> async def main():
...     await mock()
...
>>> asyncio.run(main())
>>> mock.await_count
1
>>> asyncio.run(main())
>>> mock.await_count
2
@end example
@end deffn

@geindex await_args (unittest.mock.AsyncMock attribute)
@anchor{library/unittest mock unittest mock AsyncMock await_args}@anchor{388b}
@deffn {Attribute} await_args

This is either @code{None} (if the mock hasn’t been awaited), or the arguments that
the mock was last awaited with. Functions the same as @ref{120f,,Mock.call_args}.

@example
>>> mock = AsyncMock()
>>> async def main(*args):
...     await mock(*args)
...
>>> mock.await_args
>>> asyncio.run(main('foo'))
>>> mock.await_args
call('foo')
>>> asyncio.run(main('bar'))
>>> mock.await_args
call('bar')
@end example
@end deffn

@geindex await_args_list (unittest.mock.AsyncMock attribute)
@anchor{library/unittest mock unittest mock AsyncMock await_args_list}@anchor{120e}
@deffn {Attribute} await_args_list

This is a list of all the awaits made to the mock object in sequence (so the
length of the list is the number of times it has been awaited). Before any
awaits have been made it is an empty list.

@example
>>> mock = AsyncMock()
>>> async def main(*args):
...     await mock(*args)
...
>>> mock.await_args_list
[]
>>> asyncio.run(main('foo'))
>>> mock.await_args_list
[call('foo')]
>>> asyncio.run(main('bar'))
>>> mock.await_args_list
[call('foo'), call('bar')]
@end example
@end deffn
@end deffn

@menu
* Calling:: 
* Deleting Attributes:: 
* Mock names and the name attribute:: 
* Attaching Mocks as Attributes:: 

@end menu

@node Calling,Deleting Attributes,,The Mock Class
@anchor{library/unittest mock calling}@anchor{388c}
@subsubsection Calling


Mock objects are callable. The call will return the value set as the
@ref{3870,,return_value} attribute. The default return value is a new Mock
object; it is created the first time the return value is accessed (either
explicitly or by calling the Mock) - but it is stored and the same one
returned each time.

Calls made to the object will be recorded in the attributes
like @ref{120f,,call_args} and @ref{387e,,call_args_list}.

If @ref{386e,,side_effect} is set then it will be called after the call has
been recorded, so if @code{side_effect} raises an exception the call is still
recorded.

The simplest way to make a mock raise an exception when called is to make
@ref{386e,,side_effect} an exception class or instance:

@example
>>> m = MagicMock(side_effect=IndexError)
>>> m(1, 2, 3)
Traceback (most recent call last):
  ...
IndexError
>>> m.mock_calls
[call(1, 2, 3)]
>>> m.side_effect = KeyError('Bang!')
>>> m('two', 'three', 'four')
Traceback (most recent call last):
  ...
KeyError: 'Bang!'
>>> m.mock_calls
[call(1, 2, 3), call('two', 'three', 'four')]
@end example

If @code{side_effect} is a function then whatever that function returns is what
calls to the mock return. The @code{side_effect} function is called with the
same arguments as the mock. This allows you to vary the return value of the
call dynamically, based on the input:

@example
>>> def side_effect(value):
...     return value + 1
...
>>> m = MagicMock(side_effect=side_effect)
>>> m(1)
2
>>> m(2)
3
>>> m.mock_calls
[call(1), call(2)]
@end example

If you want the mock to still return the default return value (a new mock), or
any set return value, then there are two ways of doing this. Either return
@code{mock.return_value} from inside @code{side_effect}, or return @ref{386f,,DEFAULT}:

@example
>>> m = MagicMock()
>>> def side_effect(*args, **kwargs):
...     return m.return_value
...
>>> m.side_effect = side_effect
>>> m.return_value = 3
>>> m()
3
>>> def side_effect(*args, **kwargs):
...     return DEFAULT
...
>>> m.side_effect = side_effect
>>> m()
3
@end example

To remove a @code{side_effect}, and return to the default behaviour, set the
@code{side_effect} to @code{None}:

@example
>>> m = MagicMock(return_value=6)
>>> def side_effect(*args, **kwargs):
...     return 3
...
>>> m.side_effect = side_effect
>>> m()
3
>>> m.side_effect = None
>>> m()
6
@end example

The @code{side_effect} can also be any iterable object. Repeated calls to the mock
will return values from the iterable (until the iterable is exhausted and
a @ref{736,,StopIteration} is raised):

@example
>>> m = MagicMock(side_effect=[1, 2, 3])
>>> m()
1
>>> m()
2
>>> m()
3
>>> m()
Traceback (most recent call last):
  ...
StopIteration
@end example

If any members of the iterable are exceptions they will be raised instead of
returned:

@example
>>> iterable = (33, ValueError, 66)
>>> m = MagicMock(side_effect=iterable)
>>> m()
33
>>> m()
Traceback (most recent call last):
 ...
ValueError
>>> m()
66
@end example

@node Deleting Attributes,Mock names and the name attribute,Calling,The Mock Class
@anchor{library/unittest mock deleting-attributes}@anchor{388d}@anchor{library/unittest mock id2}@anchor{388e}
@subsubsection Deleting Attributes


Mock objects create attributes on demand. This allows them to pretend to be
objects of any type.

You may want a mock object to return @code{False} to a @ref{707,,hasattr()} call, or raise an
@ref{163,,AttributeError} when an attribute is fetched. You can do this by providing
an object as a @code{spec} for a mock, but that isn’t always convenient.

You “block” attributes by deleting them. Once deleted, accessing an attribute
will raise an @ref{163,,AttributeError}.

@example
>>> mock = MagicMock()
>>> hasattr(mock, 'm')
True
>>> del mock.m
>>> hasattr(mock, 'm')
False
>>> del mock.f
>>> mock.f
Traceback (most recent call last):
    ...
AttributeError: f
@end example

@node Mock names and the name attribute,Attaching Mocks as Attributes,Deleting Attributes,The Mock Class
@anchor{library/unittest mock mock-names-and-the-name-attribute}@anchor{388f}
@subsubsection Mock names and the name attribute


Since “name” is an argument to the @ref{561,,Mock} constructor, if you want your
mock object to have a “name” attribute you can’t just pass it in at creation
time. There are two alternatives. One option is to use
@ref{3871,,configure_mock()}:

@example
>>> mock = MagicMock()
>>> mock.configure_mock(name='my_name')
>>> mock.name
'my_name'
@end example

A simpler option is to simply set the “name” attribute after mock creation:

@example
>>> mock = MagicMock()
>>> mock.name = "foo"
@end example

@node Attaching Mocks as Attributes,,Mock names and the name attribute,The Mock Class
@anchor{library/unittest mock attaching-mocks-as-attributes}@anchor{3890}
@subsubsection Attaching Mocks as Attributes


When you attach a mock as an attribute of another mock (or as the return
value) it becomes a “child” of that mock. Calls to the child are recorded in
the @ref{3878,,method_calls} and @ref{3875,,mock_calls} attributes of the
parent. This is useful for configuring child mocks and then attaching them to
the parent, or for attaching mocks to a parent that records all calls to the
children and allows you to make assertions about the order of calls between
mocks:

@example
>>> parent = MagicMock()
>>> child1 = MagicMock(return_value=None)
>>> child2 = MagicMock(return_value=None)
>>> parent.child1 = child1
>>> parent.child2 = child2
>>> child1(1)
>>> child2(2)
>>> parent.mock_calls
[call.child1(1), call.child2(2)]
@end example

The exception to this is if the mock has a name. This allows you to prevent
the “parenting” if for some reason you don’t want it to happen.

@example
>>> mock = MagicMock()
>>> not_a_child = MagicMock(name='not-a-child')
>>> mock.attribute = not_a_child
>>> mock.attribute()
<MagicMock name='not-a-child()' id='...'>
>>> mock.mock_calls
[]
@end example

Mocks created for you by @ref{a02,,patch()} are automatically given names. To
attach mocks that have names to a parent you use the @ref{3877,,attach_mock()}
method:

@example
>>> thing1 = object()
>>> thing2 = object()
>>> parent = MagicMock()
>>> with patch('__main__.thing1', return_value=None) as child1:
...     with patch('__main__.thing2', return_value=None) as child2:
...         parent.attach_mock(child1, 'child1')
...         parent.attach_mock(child2, 'child2')
...         child1('one')
...         child2('two')
...
>>> parent.mock_calls
[call.child1('one'), call.child2('two')]
@end example

@node The patchers,MagicMock and magic method support,The Mock Class,unittest mock — mock object library
@anchor{library/unittest mock the-patchers}@anchor{3891}
@subsubsection The patchers


The patch decorators are used for patching objects only within the scope of
the function they decorate. They automatically handle the unpatching for you,
even if exceptions are raised. All of these functions can also be used in with
statements or as class decorators.

@menu
* patch:: 
* patch.object: patch object. 
* patch.dict: patch dict. 
* patch.multiple: patch multiple. 
* patch methods; start and stop: patch methods start and stop. 
* patch builtins:: 
* TEST_PREFIX:: 
* Nesting Patch Decorators:: 
* Where to patch:: 
* Patching Descriptors and Proxy Objects:: 

@end menu

@node patch,patch object,,The patchers
@anchor{library/unittest mock patch}@anchor{3892}
@subsubsection patch


@cartouche
@quotation Note 
The key is to do the patching in the right namespace. See the section @ref{3893,,where to patch}.
@end quotation
@end cartouche

@geindex patch() (in module unittest.mock)
@anchor{library/unittest mock unittest mock patch}@anchor{a02}
@deffn {Function} unittest.mock.patch (target, new=DEFAULT, spec=None, create=False, spec_set=None, autospec=None, new_callable=None, **kwargs)

@ref{a02,,patch()} acts as a function decorator, class decorator or a context
manager. Inside the body of the function or with statement, the @emph{target}
is patched with a @emph{new} object. When the function/with statement exits
the patch is undone.

If @emph{new} is omitted, then the target is replaced with an
@ref{560,,AsyncMock} if the patched object is an async function or
a @ref{a01,,MagicMock} otherwise.
If @ref{a02,,patch()} is used as a decorator and @emph{new} is
omitted, the created mock is passed in as an extra argument to the
decorated function. If @ref{a02,,patch()} is used as a context manager the created
mock is returned by the context manager.

@emph{target} should be a string in the form @code{'package.module.ClassName'}. The
@emph{target} is imported and the specified object replaced with the @emph{new}
object, so the @emph{target} must be importable from the environment you are
calling @ref{a02,,patch()} from. The target is imported when the decorated function
is executed, not at decoration time.

The @emph{spec} and @emph{spec_set} keyword arguments are passed to the @ref{a01,,MagicMock}
if patch is creating one for you.

In addition you can pass @code{spec=True} or @code{spec_set=True}, which causes
patch to pass in the object being mocked as the spec/spec_set object.

@emph{new_callable} allows you to specify a different class, or callable object,
that will be called to create the @emph{new} object. By default @ref{560,,AsyncMock}
is used for async functions and @ref{a01,,MagicMock} for the rest.

A more powerful form of @emph{spec} is @emph{autospec}. If you set @code{autospec=True}
then the mock will be created with a spec from the object being replaced.
All attributes of the mock will also have the spec of the corresponding
attribute of the object being replaced. Methods and functions being mocked
will have their arguments checked and will raise a @ref{162,,TypeError} if they are
called with the wrong signature. For mocks
replacing a class, their return value (the ‘instance’) will have the same
spec as the class. See the @ref{122a,,create_autospec()} function and
@ref{386a,,Autospeccing}.

Instead of @code{autospec=True} you can pass @code{autospec=some_object} to use an
arbitrary object as the spec instead of the one being replaced.

By default @ref{a02,,patch()} will fail to replace attributes that don’t exist.
If you pass in @code{create=True}, and the attribute doesn’t exist, patch will
create the attribute for you when the patched function is called, and delete
it again after the patched function has exited. This is useful for writing
tests against attributes that your production code creates at runtime. It is
off by default because it can be dangerous. With it switched on you can
write passing tests against APIs that don’t actually exist!

@cartouche
@quotation Note 

Changed in version 3.5: If you are patching builtins in a module then you don’t
need to pass @code{create=True}, it will be added by default.

@end quotation
@end cartouche

Patch can be used as a @code{TestCase} class decorator. It works by
decorating each test method in the class. This reduces the boilerplate
code when your test methods share a common patchings set. @ref{a02,,patch()} finds
tests by looking for method names that start with @code{patch.TEST_PREFIX}.
By default this is @code{'test'}, which matches the way @ref{122,,unittest} finds tests.
You can specify an alternative prefix by setting @code{patch.TEST_PREFIX}.

Patch can be used as a context manager, with the with statement. Here the
patching applies to the indented block after the with statement. If you
use “as” then the patched object will be bound to the name after the
“as”; very useful if @ref{a02,,patch()} is creating a mock object for you.

@ref{a02,,patch()} takes arbitrary keyword arguments. These will be passed to
@ref{560,,AsyncMock} if the patched object is asynchronous, to
@ref{a01,,MagicMock} otherwise or to @emph{new_callable} if specified.

@code{patch.dict(...)}, @code{patch.multiple(...)} and @code{patch.object(...)} are
available for alternate use-cases.
@end deffn

@ref{a02,,patch()} as function decorator, creating the mock for you and passing it into
the decorated function:

@example
>>> @@patch('__main__.SomeClass')
... def function(normal_argument, mock_class):
...     print(mock_class is SomeClass)
...
>>> function(None)
True
@end example

Patching a class replaces the class with a @ref{a01,,MagicMock} @emph{instance}. If the
class is instantiated in the code under test then it will be the
@ref{3870,,return_value} of the mock that will be used.

If the class is instantiated multiple times you could use
@ref{386e,,side_effect} to return a new mock each time. Alternatively you
can set the @emph{return_value} to be anything you want.

To configure return values on methods of @emph{instances} on the patched class
you must do this on the @code{return_value}. For example:

@example
>>> class Class:
...     def method(self):
...         pass
...
>>> with patch('__main__.Class') as MockClass:
...     instance = MockClass.return_value
...     instance.method.return_value = 'foo'
...     assert Class() is instance
...     assert Class().method() == 'foo'
...
@end example

If you use @emph{spec} or @emph{spec_set} and @ref{a02,,patch()} is replacing a @emph{class}, then the
return value of the created mock will have the same spec.

@example
>>> Original = Class
>>> patcher = patch('__main__.Class', spec=True)
>>> MockClass = patcher.start()
>>> instance = MockClass()
>>> assert isinstance(instance, Original)
>>> patcher.stop()
@end example

The @emph{new_callable} argument is useful where you want to use an alternative
class to the default @ref{a01,,MagicMock} for the created mock. For example, if
you wanted a @ref{386c,,NonCallableMock} to be used:

@example
>>> thing = object()
>>> with patch('__main__.thing', new_callable=NonCallableMock) as mock_thing:
...     assert thing is mock_thing
...     thing()
...
Traceback (most recent call last):
  ...
TypeError: 'NonCallableMock' object is not callable
@end example

Another use case might be to replace an object with an @ref{aa1,,io.StringIO} instance:

@example
>>> from io import StringIO
>>> def foo():
...     print('Something')
...
>>> @@patch('sys.stdout', new_callable=StringIO)
... def test(mock_stdout):
...     foo()
...     assert mock_stdout.getvalue() == 'Something\n'
...
>>> test()
@end example

When @ref{a02,,patch()} is creating a mock for you, it is common that the first thing
you need to do is to configure the mock. Some of that configuration can be done
in the call to patch. Any arbitrary keywords you pass into the call will be
used to set attributes on the created mock:

@example
>>> patcher = patch('__main__.thing', first='one', second='two')
>>> mock_thing = patcher.start()
>>> mock_thing.first
'one'
>>> mock_thing.second
'two'
@end example

As well as attributes on the created mock attributes, like the
@ref{3870,,return_value} and @ref{386e,,side_effect}, of child mocks can
also be configured. These aren’t syntactically valid to pass in directly as
keyword arguments, but a dictionary with these as keys can still be expanded
into a @ref{a02,,patch()} call using @code{**}:

@example
>>> config = @{'method.return_value': 3, 'other.side_effect': KeyError@}
>>> patcher = patch('__main__.thing', **config)
>>> mock_thing = patcher.start()
>>> mock_thing.method()
3
>>> mock_thing.other()
Traceback (most recent call last):
  ...
KeyError
@end example

By default, attempting to patch a function in a module (or a method or an
attribute in a class) that does not exist will fail with @ref{163,,AttributeError}:

@example
>>> @@patch('sys.non_existing_attribute', 42)
... def test():
...     assert sys.non_existing_attribute == 42
...
>>> test()
Traceback (most recent call last):
  ...
AttributeError: <module 'sys' (built-in)> does not have the attribute 'non_existing_attribute'
@end example

but adding @code{create=True} in the call to @ref{a02,,patch()} will make the previous example
work as expected:

@example
>>> @@patch('sys.non_existing_attribute', 42, create=True)
... def test(mock_stdout):
...     assert sys.non_existing_attribute == 42
...
>>> test()
@end example


Changed in version 3.8: @ref{a02,,patch()} now returns an @ref{560,,AsyncMock} if the target is an async function.


@node patch object,patch dict,patch,The patchers
@anchor{library/unittest mock patch-object}@anchor{3894}
@subsubsection patch.object


@geindex patch.object() (in module unittest.mock)
@anchor{library/unittest mock unittest mock patch object}@anchor{1250}
@deffn {Function} patch.object (target, attribute, new=DEFAULT, spec=None, create=False, spec_set=None, autospec=None, new_callable=None, **kwargs)

patch the named member (@emph{attribute}) on an object (@emph{target}) with a mock
object.

@ref{1250,,patch.object()} can be used as a decorator, class decorator or a context
manager. Arguments @emph{new}, @emph{spec}, @emph{create}, @emph{spec_set}, @emph{autospec} and
@emph{new_callable} have the same meaning as for @ref{a02,,patch()}. Like @ref{a02,,patch()},
@ref{1250,,patch.object()} takes arbitrary keyword arguments for configuring the mock
object it creates.

When used as a class decorator @ref{1250,,patch.object()} honours @code{patch.TEST_PREFIX}
for choosing which methods to wrap.
@end deffn

You can either call @ref{1250,,patch.object()} with three arguments or two arguments. The
three argument form takes the object to be patched, the attribute name and the
object to replace the attribute with.

When calling with the two argument form you omit the replacement object, and a
mock is created for you and passed in as an extra argument to the decorated
function:

@example
>>> @@patch.object(SomeClass, 'class_method')
... def test(mock_method):
...     SomeClass.class_method(3)
...     mock_method.assert_called_with(3)
...
>>> test()
@end example

@emph{spec}, @emph{create} and the other arguments to @ref{1250,,patch.object()} have the same
meaning as they do for @ref{a02,,patch()}.

@node patch dict,patch multiple,patch object,The patchers
@anchor{library/unittest mock patch-dict}@anchor{3895}
@subsubsection patch.dict


@geindex patch.dict() (in module unittest.mock)
@anchor{library/unittest mock unittest mock patch dict}@anchor{129d}
@deffn {Function} patch.dict (in_dict, values=(), clear=False, **kwargs)

Patch a dictionary, or dictionary like object, and restore the dictionary
to its original state after the test.

@emph{in_dict} can be a dictionary or a mapping like container. If it is a
mapping then it must at least support getting, setting and deleting items
plus iterating over keys.

@emph{in_dict} can also be a string specifying the name of the dictionary, which
will then be fetched by importing it.

@emph{values} can be a dictionary of values to set in the dictionary. @emph{values}
can also be an iterable of @code{(key, value)} pairs.

If @emph{clear} is true then the dictionary will be cleared before the new
values are set.

@ref{129d,,patch.dict()} can also be called with arbitrary keyword arguments to set
values in the dictionary.


Changed in version 3.8: @ref{129d,,patch.dict()} now returns the patched dictionary when used as a context
manager.

@end deffn

@ref{129d,,patch.dict()} can be used as a context manager, decorator or class
decorator:

@example
>>> foo = @{@}
>>> @@patch.dict(foo, @{'newkey': 'newvalue'@})
... def test():
...     assert foo == @{'newkey': 'newvalue'@}
>>> test()
>>> assert foo == @{@}
@end example

When used as a class decorator @ref{129d,,patch.dict()} honours
@code{patch.TEST_PREFIX} (default to @code{'test'}) for choosing which methods to wrap:

@example
>>> import os
>>> import unittest
>>> from unittest.mock import patch
>>> @@patch.dict('os.environ', @{'newkey': 'newvalue'@})
... class TestSample(unittest.TestCase):
...     def test_sample(self):
...         self.assertEqual(os.environ['newkey'], 'newvalue')
@end example

If you want to use a different prefix for your test, you can inform the
patchers of the different prefix by setting @code{patch.TEST_PREFIX}. For
more details about how to change the value of see @ref{3896,,TEST_PREFIX}.

@ref{129d,,patch.dict()} can be used to add members to a dictionary, or simply let a test
change a dictionary, and ensure the dictionary is restored when the test
ends.

@example
>>> foo = @{@}
>>> with patch.dict(foo, @{'newkey': 'newvalue'@}) as patched_foo:
...     assert foo == @{'newkey': 'newvalue'@}
...     assert patched_foo == @{'newkey': 'newvalue'@}
...     # You can add, update or delete keys of foo (or patched_foo, it's the same dict)
...     patched_foo['spam'] = 'eggs'
...
>>> assert foo == @{@}
>>> assert patched_foo == @{@}
@end example

@example
>>> import os
>>> with patch.dict('os.environ', @{'newkey': 'newvalue'@}):
...     print(os.environ['newkey'])
...
newvalue
>>> assert 'newkey' not in os.environ
@end example

Keywords can be used in the @ref{129d,,patch.dict()} call to set values in the dictionary:

@example
>>> mymodule = MagicMock()
>>> mymodule.function.return_value = 'fish'
>>> with patch.dict('sys.modules', mymodule=mymodule):
...     import mymodule
...     mymodule.function('some', 'args')
...
'fish'
@end example

@ref{129d,,patch.dict()} can be used with dictionary like objects that aren’t actually
dictionaries. At the very minimum they must support item getting, setting,
deleting and either iteration or membership test. This corresponds to the
magic methods @code{__getitem__()}, @code{__setitem__()}, @code{__delitem__()} and either
@code{__iter__()} or @code{__contains__()}.

@example
>>> class Container:
...     def __init__(self):
...         self.values = @{@}
...     def __getitem__(self, name):
...         return self.values[name]
...     def __setitem__(self, name, value):
...         self.values[name] = value
...     def __delitem__(self, name):
...         del self.values[name]
...     def __iter__(self):
...         return iter(self.values)
...
>>> thing = Container()
>>> thing['one'] = 1
>>> with patch.dict(thing, one=2, two=3):
...     assert thing['one'] == 2
...     assert thing['two'] == 3
...
>>> assert thing['one'] == 1
>>> assert list(thing) == ['one']
@end example

@node patch multiple,patch methods start and stop,patch dict,The patchers
@anchor{library/unittest mock patch-multiple}@anchor{3897}
@subsubsection patch.multiple


@geindex patch.multiple() (in module unittest.mock)
@anchor{library/unittest mock unittest mock patch multiple}@anchor{3898}
@deffn {Function} patch.multiple (target, spec=None, create=False, spec_set=None, autospec=None, new_callable=None, **kwargs)

Perform multiple patches in a single call. It takes the object to be
patched (either as an object or a string to fetch the object by importing)
and keyword arguments for the patches:

@example
with patch.multiple(settings, FIRST_PATCH='one', SECOND_PATCH='two'):
    ...
@end example

Use @ref{386f,,DEFAULT} as the value if you want @ref{3898,,patch.multiple()} to create
mocks for you. In this case the created mocks are passed into a decorated
function by keyword, and a dictionary is returned when @ref{3898,,patch.multiple()} is
used as a context manager.

@ref{3898,,patch.multiple()} can be used as a decorator, class decorator or a context
manager. The arguments @emph{spec}, @emph{spec_set}, @emph{create}, @emph{autospec} and
@emph{new_callable} have the same meaning as for @ref{a02,,patch()}. These arguments will
be applied to @emph{all} patches done by @ref{3898,,patch.multiple()}.

When used as a class decorator @ref{3898,,patch.multiple()} honours @code{patch.TEST_PREFIX}
for choosing which methods to wrap.
@end deffn

If you want @ref{3898,,patch.multiple()} to create mocks for you, then you can use
@ref{386f,,DEFAULT} as the value. If you use @ref{3898,,patch.multiple()} as a decorator
then the created mocks are passed into the decorated function by keyword.

@example
>>> thing = object()
>>> other = object()

>>> @@patch.multiple('__main__', thing=DEFAULT, other=DEFAULT)
... def test_function(thing, other):
...     assert isinstance(thing, MagicMock)
...     assert isinstance(other, MagicMock)
...
>>> test_function()
@end example

@ref{3898,,patch.multiple()} can be nested with other @code{patch} decorators, but put arguments
passed by keyword @emph{after} any of the standard arguments created by @ref{a02,,patch()}:

@example
>>> @@patch('sys.exit')
... @@patch.multiple('__main__', thing=DEFAULT, other=DEFAULT)
... def test_function(mock_exit, other, thing):
...     assert 'other' in repr(other)
...     assert 'thing' in repr(thing)
...     assert 'exit' in repr(mock_exit)
...
>>> test_function()
@end example

If @ref{3898,,patch.multiple()} is used as a context manager, the value returned by the
context manager is a dictionary where created mocks are keyed by name:

@example
>>> with patch.multiple('__main__', thing=DEFAULT, other=DEFAULT) as values:
...     assert 'other' in repr(values['other'])
...     assert 'thing' in repr(values['thing'])
...     assert values['thing'] is thing
...     assert values['other'] is other
...
@end example

@node patch methods start and stop,patch builtins,patch multiple,The patchers
@anchor{library/unittest mock patch-methods-start-and-stop}@anchor{3899}@anchor{library/unittest mock start-and-stop}@anchor{389a}
@subsubsection patch methods: start and stop


All the patchers have @code{start()} and @code{stop()} methods. These make it simpler to do
patching in @code{setUp} methods or where you want to do multiple patches without
nesting decorators or with statements.

To use them call @ref{a02,,patch()}, @ref{1250,,patch.object()} or @ref{129d,,patch.dict()} as
normal and keep a reference to the returned @code{patcher} object. You can then
call @code{start()} to put the patch in place and @code{stop()} to undo it.

If you are using @ref{a02,,patch()} to create a mock for you then it will be returned by
the call to @code{patcher.start}.

@example
>>> patcher = patch('package.module.ClassName')
>>> from package import module
>>> original = module.ClassName
>>> new_mock = patcher.start()
>>> assert module.ClassName is not original
>>> assert module.ClassName is new_mock
>>> patcher.stop()
>>> assert module.ClassName is original
>>> assert module.ClassName is not new_mock
@end example

A typical use case for this might be for doing multiple patches in the @code{setUp}
method of a @code{TestCase}:

@example
>>> class MyTest(unittest.TestCase):
...     def setUp(self):
...         self.patcher1 = patch('package.module.Class1')
...         self.patcher2 = patch('package.module.Class2')
...         self.MockClass1 = self.patcher1.start()
...         self.MockClass2 = self.patcher2.start()
...
...     def tearDown(self):
...         self.patcher1.stop()
...         self.patcher2.stop()
...
...     def test_something(self):
...         assert package.module.Class1 is self.MockClass1
...         assert package.module.Class2 is self.MockClass2
...
>>> MyTest('test_something').run()
@end example

@cartouche
@quotation Caution 
If you use this technique you must ensure that the patching is “undone” by
calling @code{stop}. This can be fiddlier than you might think, because if an
exception is raised in the @code{setUp} then @code{tearDown} is not called.
@ref{599,,unittest.TestCase.addCleanup()} makes this easier:

@example
>>> class MyTest(unittest.TestCase):
...     def setUp(self):
...         patcher = patch('package.module.Class')
...         self.MockClass = patcher.start()
...         self.addCleanup(patcher.stop)
...
...     def test_something(self):
...         assert package.module.Class is self.MockClass
...
@end example

As an added bonus you no longer need to keep a reference to the @code{patcher}
object.
@end quotation
@end cartouche

It is also possible to stop all patches which have been started by using
@ref{389b,,patch.stopall()}.

@geindex patch.stopall() (in module unittest.mock)
@anchor{library/unittest mock unittest mock patch stopall}@anchor{389b}
@deffn {Function} patch.stopall ()

Stop all active patches. Only stops patches started with @code{start}.
@end deffn

@node patch builtins,TEST_PREFIX,patch methods start and stop,The patchers
@anchor{library/unittest mock id4}@anchor{389c}@anchor{library/unittest mock patch-builtins}@anchor{389d}
@subsubsection patch builtins


You can patch any builtins within a module. The following example patches
builtin @ref{178e,,ord()}:

@example
>>> @@patch('__main__.ord')
... def test(mock_ord):
...     mock_ord.return_value = 101
...     print(ord('c'))
...
>>> test()
101
@end example

@node TEST_PREFIX,Nesting Patch Decorators,patch builtins,The patchers
@anchor{library/unittest mock id5}@anchor{389e}@anchor{library/unittest mock test-prefix}@anchor{3896}
@subsubsection TEST_PREFIX


All of the patchers can be used as class decorators. When used in this way
they wrap every test method on the class. The patchers recognise methods that
start with @code{'test'} as being test methods. This is the same way that the
@ref{1ad,,unittest.TestLoader} finds test methods by default.

It is possible that you want to use a different prefix for your tests. You can
inform the patchers of the different prefix by setting @code{patch.TEST_PREFIX}:

@example
>>> patch.TEST_PREFIX = 'foo'
>>> value = 3
>>>
>>> @@patch('__main__.value', 'not three')
... class Thing:
...     def foo_one(self):
...         print(value)
...     def foo_two(self):
...         print(value)
...
>>>
>>> Thing().foo_one()
not three
>>> Thing().foo_two()
not three
>>> value
3
@end example

@node Nesting Patch Decorators,Where to patch,TEST_PREFIX,The patchers
@anchor{library/unittest mock nesting-patch-decorators}@anchor{389f}
@subsubsection Nesting Patch Decorators


If you want to perform multiple patches then you can simply stack up the
decorators.

You can stack up multiple patch decorators using this pattern:

@example
>>> @@patch.object(SomeClass, 'class_method')
... @@patch.object(SomeClass, 'static_method')
... def test(mock1, mock2):
...     assert SomeClass.static_method is mock1
...     assert SomeClass.class_method is mock2
...     SomeClass.static_method('foo')
...     SomeClass.class_method('bar')
...     return mock1, mock2
...
>>> mock1, mock2 = test()
>>> mock1.assert_called_once_with('foo')
>>> mock2.assert_called_once_with('bar')
@end example

Note that the decorators are applied from the bottom upwards. This is the
standard way that Python applies decorators. The order of the created mocks
passed into your test function matches this order.

@node Where to patch,Patching Descriptors and Proxy Objects,Nesting Patch Decorators,The patchers
@anchor{library/unittest mock id6}@anchor{3893}@anchor{library/unittest mock where-to-patch}@anchor{3868}
@subsubsection Where to patch


@ref{a02,,patch()} works by (temporarily) changing the object that a @emph{name} points to with
another one. There can be many names pointing to any individual object, so
for patching to work you must ensure that you patch the name used by the system
under test.

The basic principle is that you patch where an object is @emph{looked up}, which
is not necessarily the same place as where it is defined. A couple of
examples will help to clarify this.

Imagine we have a project that we want to test with the following structure:

@example
a.py
    -> Defines SomeClass

b.py
    -> from a import SomeClass
    -> some_function instantiates SomeClass
@end example

Now we want to test @code{some_function} but we want to mock out @code{SomeClass} using
@ref{a02,,patch()}. The problem is that when we import module b, which we will have to
do then it imports @code{SomeClass} from module a. If we use @ref{a02,,patch()} to mock out
@code{a.SomeClass} then it will have no effect on our test; module b already has a
reference to the @emph{real} @code{SomeClass} and it looks like our patching had no
effect.

The key is to patch out @code{SomeClass} where it is used (or where it is looked up).
In this case @code{some_function} will actually look up @code{SomeClass} in module b,
where we have imported it. The patching should look like:

@example
@@patch('b.SomeClass')
@end example

However, consider the alternative scenario where instead of @code{from a import
SomeClass} module b does @code{import a} and @code{some_function} uses @code{a.SomeClass}. Both
of these import forms are common. In this case the class we want to patch is
being looked up in the module and so we have to patch @code{a.SomeClass} instead:

@example
@@patch('a.SomeClass')
@end example

@node Patching Descriptors and Proxy Objects,,Where to patch,The patchers
@anchor{library/unittest mock patching-descriptors-and-proxy-objects}@anchor{38a0}
@subsubsection Patching Descriptors and Proxy Objects


Both @ref{3892,,patch} and @ref{3894,,patch.object} correctly patch and restore descriptors: class
methods, static methods and properties. You should patch these on the @emph{class}
rather than an instance. They also work with @emph{some} objects
that proxy attribute access, like the django settings object@footnote{http://www.voidspace.org.uk/python/weblog/arch_d7_2010_12_04.shtml#e1198}.

@node MagicMock and magic method support,Helpers,The patchers,unittest mock — mock object library
@anchor{library/unittest mock magicmock-and-magic-method-support}@anchor{38a1}
@subsubsection MagicMock and magic method support


@menu
* Mocking Magic Methods:: 
* Magic Mock:: 

@end menu

@node Mocking Magic Methods,Magic Mock,,MagicMock and magic method support
@anchor{library/unittest mock magic-methods}@anchor{3869}@anchor{library/unittest mock mocking-magic-methods}@anchor{38a2}
@subsubsection Mocking Magic Methods


@ref{561,,Mock} supports mocking the Python protocol methods, also known as
“magic methods”. This allows mock objects to replace containers or other
objects that implement Python protocols.

Because magic methods are looked up differently from normal methods @footnote{@w{(2)} 
Magic methods @emph{should} be looked up on the class rather than the
instance. Different versions of Python are inconsistent about applying this
rule. The supported protocol methods should work with all supported versions
of Python.
}, this
support has been specially implemented. This means that only specific magic
methods are supported. The supported list includes @emph{almost} all of them. If
there are any missing that you need please let us know.

You mock magic methods by setting the method you are interested in to a function
or a mock instance. If you are using a function then it @emph{must} take @code{self} as
the first argument @footnote{@w{(3)} 
The function is basically hooked up to the class, but each @code{Mock}
instance is kept isolated from the others.
}.

@example
>>> def __str__(self):
...     return 'fooble'
...
>>> mock = Mock()
>>> mock.__str__ = __str__
>>> str(mock)
'fooble'
@end example

@example
>>> mock = Mock()
>>> mock.__str__ = Mock()
>>> mock.__str__.return_value = 'fooble'
>>> str(mock)
'fooble'
@end example

@example
>>> mock = Mock()
>>> mock.__iter__ = Mock(return_value=iter([]))
>>> list(mock)
[]
@end example

One use case for this is for mocking objects used as context managers in a
@ref{168,,with} statement:

@example
>>> mock = Mock()
>>> mock.__enter__ = Mock(return_value='foo')
>>> mock.__exit__ = Mock(return_value=False)
>>> with mock as m:
...     assert m == 'foo'
...
>>> mock.__enter__.assert_called_with()
>>> mock.__exit__.assert_called_with(None, None, None)
@end example

Calls to magic methods do not appear in @ref{3878,,method_calls}, but they
are recorded in @ref{3875,,mock_calls}.

@cartouche
@quotation Note 
If you use the @emph{spec} keyword argument to create a mock then attempting to
set a magic method that isn’t in the spec will raise an @ref{163,,AttributeError}.
@end quotation
@end cartouche

The full list of supported magic methods is:


@itemize *

@item 
@code{__hash__}, @code{__sizeof__}, @code{__repr__} and @code{__str__}

@item 
@code{__dir__}, @code{__format__} and @code{__subclasses__}

@item 
@code{__round__}, @code{__floor__}, @code{__trunc__} and @code{__ceil__}

@item 
Comparisons: @code{__lt__}, @code{__gt__}, @code{__le__}, @code{__ge__},
@code{__eq__} and @code{__ne__}

@item 
Container methods: @code{__getitem__}, @code{__setitem__}, @code{__delitem__},
@code{__contains__}, @code{__len__}, @code{__iter__}, @code{__reversed__}
and @code{__missing__}

@item 
Context manager: @code{__enter__}, @code{__exit__}, @code{__aenter__} and @code{__aexit__}

@item 
Unary numeric methods: @code{__neg__}, @code{__pos__} and @code{__invert__}

@item 
The numeric methods (including right hand and in-place variants):
@code{__add__}, @code{__sub__}, @code{__mul__}, @code{__matmul__}, @code{__truediv__},
@code{__floordiv__}, @code{__mod__}, @code{__divmod__}, @code{__lshift__},
@code{__rshift__}, @code{__and__}, @code{__xor__}, @code{__or__}, and @code{__pow__}

@item 
Numeric conversion methods: @code{__complex__}, @code{__int__}, @code{__float__}
and @code{__index__}

@item 
Descriptor methods: @code{__get__}, @code{__set__} and @code{__delete__}

@item 
Pickling: @code{__reduce__}, @code{__reduce_ex__}, @code{__getinitargs__},
@code{__getnewargs__}, @code{__getstate__} and @code{__setstate__}

@item 
File system path representation: @code{__fspath__}

@item 
Asynchronous iteration methods: @code{__aiter__} and @code{__anext__}
@end itemize


Changed in version 3.8: Added support for @ref{796,,os.PathLike.__fspath__()}.



Changed in version 3.8: Added support for @code{__aenter__}, @code{__aexit__}, @code{__aiter__} and @code{__anext__}.


The following methods exist but are @emph{not} supported as they are either in use
by mock, can’t be set dynamically, or can cause problems:


@itemize *

@item 
@code{__getattr__}, @code{__setattr__}, @code{__init__} and @code{__new__}

@item 
@code{__prepare__}, @code{__instancecheck__}, @code{__subclasscheck__}, @code{__del__}
@end itemize

@node Magic Mock,,Mocking Magic Methods,MagicMock and magic method support
@anchor{library/unittest mock magic-mock}@anchor{38a3}
@subsubsection Magic Mock


There are two @code{MagicMock} variants: @ref{a01,,MagicMock} and @ref{386d,,NonCallableMagicMock}.

@geindex MagicMock (class in unittest.mock)
@anchor{library/unittest mock unittest mock MagicMock}@anchor{a01}
@deffn {Class} unittest.mock.MagicMock (*args, **kw)

@code{MagicMock} is a subclass of @ref{561,,Mock} with default implementations
of most of the magic methods. You can use @code{MagicMock} without having to
configure the magic methods yourself.

The constructor parameters have the same meaning as for @ref{561,,Mock}.

If you use the @emph{spec} or @emph{spec_set} arguments then @emph{only} magic methods
that exist in the spec will be created.
@end deffn

@geindex NonCallableMagicMock (class in unittest.mock)
@anchor{library/unittest mock unittest mock NonCallableMagicMock}@anchor{386d}
@deffn {Class} unittest.mock.NonCallableMagicMock (*args, **kw)

A non-callable version of @ref{a01,,MagicMock}.

The constructor parameters have the same meaning as for
@ref{a01,,MagicMock}, with the exception of @emph{return_value} and
@emph{side_effect} which have no meaning on a non-callable mock.
@end deffn

The magic methods are setup with @ref{a01,,MagicMock} objects, so you can configure them
and use them in the usual way:

@example
>>> mock = MagicMock()
>>> mock[3] = 'fish'
>>> mock.__setitem__.assert_called_with(3, 'fish')
>>> mock.__getitem__.return_value = 'result'
>>> mock[2]
'result'
@end example

By default many of the protocol methods are required to return objects of a
specific type. These methods are preconfigured with a default return value, so
that they can be used without you having to do anything if you aren’t interested
in the return value. You can still @emph{set} the return value manually if you want
to change the default.

Methods and their defaults:


@itemize *

@item 
@code{__lt__}: @code{NotImplemented}

@item 
@code{__gt__}: @code{NotImplemented}

@item 
@code{__le__}: @code{NotImplemented}

@item 
@code{__ge__}: @code{NotImplemented}

@item 
@code{__int__}: @code{1}

@item 
@code{__contains__}: @code{False}

@item 
@code{__len__}: @code{0}

@item 
@code{__iter__}: @code{iter([])}

@item 
@code{__exit__}: @code{False}

@item 
@code{__aexit__}: @code{False}

@item 
@code{__complex__}: @code{1j}

@item 
@code{__float__}: @code{1.0}

@item 
@code{__bool__}: @code{True}

@item 
@code{__index__}: @code{1}

@item 
@code{__hash__}: default hash for the mock

@item 
@code{__str__}: default str for the mock

@item 
@code{__sizeof__}: default sizeof for the mock
@end itemize

For example:

@example
>>> mock = MagicMock()
>>> int(mock)
1
>>> len(mock)
0
>>> list(mock)
[]
>>> object() in mock
False
@end example

The two equality methods, @code{__eq__()} and @code{__ne__()}, are special.
They do the default equality comparison on identity, using the
@ref{386e,,side_effect} attribute, unless you change their return value to
return something else:

@example
>>> MagicMock() == 3
False
>>> MagicMock() != 3
True
>>> mock = MagicMock()
>>> mock.__eq__.return_value = True
>>> mock == 3
True
@end example

The return value of @code{MagicMock.__iter__()} can be any iterable object and isn’t
required to be an iterator:

@example
>>> mock = MagicMock()
>>> mock.__iter__.return_value = ['a', 'b', 'c']
>>> list(mock)
['a', 'b', 'c']
>>> list(mock)
['a', 'b', 'c']
@end example

If the return value @emph{is} an iterator, then iterating over it once will consume
it and subsequent iterations will result in an empty list:

@example
>>> mock.__iter__.return_value = iter(['a', 'b', 'c'])
>>> list(mock)
['a', 'b', 'c']
>>> list(mock)
[]
@end example

@code{MagicMock} has all of the supported magic methods configured except for some
of the obscure and obsolete ones. You can still set these up if you want.

Magic methods that are supported but not setup by default in @code{MagicMock} are:


@itemize *

@item 
@code{__subclasses__}

@item 
@code{__dir__}

@item 
@code{__format__}

@item 
@code{__get__}, @code{__set__} and @code{__delete__}

@item 
@code{__reversed__} and @code{__missing__}

@item 
@code{__reduce__}, @code{__reduce_ex__}, @code{__getinitargs__}, @code{__getnewargs__},
@code{__getstate__} and @code{__setstate__}

@item 
@code{__getformat__}
@end itemize

@node Helpers,,MagicMock and magic method support,unittest mock — mock object library
@anchor{library/unittest mock helpers}@anchor{38a4}
@subsubsection Helpers


@menu
* sentinel:: 
* DEFAULT:: 
* call:: 
* create_autospec:: 
* ANY:: 
* FILTER_DIR:: 
* mock_open:: 
* Autospeccing:: 
* Sealing mocks:: 

@end menu

@node sentinel,DEFAULT,,Helpers
@anchor{library/unittest mock sentinel}@anchor{38a5}
@subsubsection sentinel


@geindex sentinel (in module unittest.mock)
@anchor{library/unittest mock unittest mock sentinel}@anchor{6d1}
@deffn {Data} unittest.mock.sentinel

The @code{sentinel} object provides a convenient way of providing unique
objects for your tests.

Attributes are created on demand when you access them by name. Accessing
the same attribute will always return the same object. The objects
returned have a sensible repr so that test failure messages are readable.


Changed in version 3.7: The @code{sentinel} attributes now preserve their identity when they are
@ref{24,,copied} or @ref{c5,,pickled}.

@end deffn

Sometimes when testing you need to test that a specific object is passed as an
argument to another method, or returned. It can be common to create named
sentinel objects to test this. @ref{6d1,,sentinel} provides a convenient way of
creating and testing the identity of objects like this.

In this example we monkey patch @code{method} to return @code{sentinel.some_object}:

@example
>>> real = ProductionClass()
>>> real.method = Mock(name="method")
>>> real.method.return_value = sentinel.some_object
>>> result = real.method()
>>> assert result is sentinel.some_object
>>> result
sentinel.some_object
@end example

@node DEFAULT,call,sentinel,Helpers
@anchor{library/unittest mock default}@anchor{38a6}
@subsubsection DEFAULT


@geindex DEFAULT (in module unittest.mock)
@anchor{library/unittest mock unittest mock DEFAULT}@anchor{386f}
@deffn {Data} unittest.mock.DEFAULT

The @ref{386f,,DEFAULT} object is a pre-created sentinel (actually
@code{sentinel.DEFAULT}). It can be used by @ref{386e,,side_effect}
functions to indicate that the normal return value should be used.
@end deffn

@node call,create_autospec,DEFAULT,Helpers
@anchor{library/unittest mock call}@anchor{38a7}
@subsubsection call


@geindex call() (in module unittest.mock)
@anchor{library/unittest mock unittest mock call}@anchor{1265}
@deffn {Function} unittest.mock.call (*args, **kwargs)

@ref{1265,,call()} is a helper object for making simpler assertions, for comparing with
@ref{120f,,call_args}, @ref{387e,,call_args_list},
@ref{3875,,mock_calls} and @ref{3878,,method_calls}. @ref{1265,,call()} can also be
used with @ref{3874,,assert_has_calls()}.

@example
>>> m = MagicMock(return_value=None)
>>> m(1, 2, a='foo', b='bar')
>>> m()
>>> m.call_args_list == [call(1, 2, a='foo', b='bar'), call()]
True
@end example
@end deffn

@geindex call_list() (unittest.mock.call method)
@anchor{library/unittest mock unittest mock call call_list}@anchor{38a8}
@deffn {Method} call.call_list ()

For a call object that represents multiple calls, @ref{38a8,,call_list()}
returns a list of all the intermediate calls as well as the
final call.
@end deffn

@code{call_list} is particularly useful for making assertions on “chained calls”. A
chained call is multiple calls on a single line of code. This results in
multiple entries in @ref{3875,,mock_calls} on a mock. Manually constructing
the sequence of calls can be tedious.

@ref{38a8,,call_list()} can construct the sequence of calls from the same
chained call:

@example
>>> m = MagicMock()
>>> m(1).method(arg='foo').other('bar')(2.0)
<MagicMock name='mock().method().other()()' id='...'>
>>> kall = call(1).method(arg='foo').other('bar')(2.0)
>>> kall.call_list()
[call(1),
 call().method(arg='foo'),
 call().method().other('bar'),
 call().method().other()(2.0)]
>>> m.mock_calls == kall.call_list()
True
@end example
@anchor{library/unittest mock calls-as-tuples}@anchor{387f}
A @code{call} object is either a tuple of (positional args, keyword args) or
(name, positional args, keyword args) depending on how it was constructed. When
you construct them yourself this isn’t particularly interesting, but the @code{call}
objects that are in the @ref{120f,,Mock.call_args}, @ref{387e,,Mock.call_args_list} and
@ref{3875,,Mock.mock_calls} attributes can be introspected to get at the individual
arguments they contain.

The @code{call} objects in @ref{120f,,Mock.call_args} and @ref{387e,,Mock.call_args_list}
are two-tuples of (positional args, keyword args) whereas the @code{call} objects
in @ref{3875,,Mock.mock_calls}, along with ones you construct yourself, are
three-tuples of (name, positional args, keyword args).

You can use their “tupleness” to pull out the individual arguments for more
complex introspection and assertions. The positional arguments are a tuple
(an empty tuple if there are no positional arguments) and the keyword
arguments are a dictionary:

@example
>>> m = MagicMock(return_value=None)
>>> m(1, 2, 3, arg='one', arg2='two')
>>> kall = m.call_args
>>> kall.args
(1, 2, 3)
>>> kall.kwargs
@{'arg': 'one', 'arg2': 'two'@}
>>> kall.args is kall[0]
True
>>> kall.kwargs is kall[1]
True
@end example

@example
>>> m = MagicMock()
>>> m.foo(4, 5, 6, arg='two', arg2='three')
<MagicMock name='mock.foo()' id='...'>
>>> kall = m.mock_calls[0]
>>> name, args, kwargs = kall
>>> name
'foo'
>>> args
(4, 5, 6)
>>> kwargs
@{'arg': 'two', 'arg2': 'three'@}
>>> name is m.mock_calls[0][0]
True
@end example

@node create_autospec,ANY,call,Helpers
@anchor{library/unittest mock create-autospec}@anchor{38a9}
@subsubsection create_autospec


@geindex create_autospec() (in module unittest.mock)
@anchor{library/unittest mock unittest mock create_autospec}@anchor{122a}
@deffn {Function} unittest.mock.create_autospec (spec, spec_set=False, instance=False, **kwargs)

Create a mock object using another object as a spec. Attributes on the
mock will use the corresponding attribute on the @emph{spec} object as their
spec.

Functions or methods being mocked will have their arguments checked to
ensure that they are called with the correct signature.

If @emph{spec_set} is @code{True} then attempting to set attributes that don’t exist
on the spec object will raise an @ref{163,,AttributeError}.

If a class is used as a spec then the return value of the mock (the
instance of the class) will have the same spec. You can use a class as the
spec for an instance object by passing @code{instance=True}. The returned mock
will only be callable if instances of the mock are callable.

@ref{122a,,create_autospec()} also takes arbitrary keyword arguments that are passed to
the constructor of the created mock.
@end deffn

See @ref{386a,,Autospeccing} for examples of how to use auto-speccing with
@ref{122a,,create_autospec()} and the @emph{autospec} argument to @ref{a02,,patch()}.


Changed in version 3.8: @ref{122a,,create_autospec()} now returns an @ref{560,,AsyncMock} if the target is
an async function.


@node ANY,FILTER_DIR,create_autospec,Helpers
@anchor{library/unittest mock any}@anchor{38aa}
@subsubsection ANY


@geindex ANY (in module unittest.mock)
@anchor{library/unittest mock unittest mock ANY}@anchor{126a}
@deffn {Data} unittest.mock.ANY
@end deffn

Sometimes you may need to make assertions about @emph{some} of the arguments in a
call to mock, but either not care about some of the arguments or want to pull
them individually out of @ref{120f,,call_args} and make more complex
assertions on them.

To ignore certain arguments you can pass in objects that compare equal to
@emph{everything}. Calls to @ref{12cc,,assert_called_with()} and
@ref{3872,,assert_called_once_with()} will then succeed no matter what was
passed in.

@example
>>> mock = Mock(return_value=None)
>>> mock('foo', bar=object())
>>> mock.assert_called_once_with('foo', bar=ANY)
@end example

@ref{126a,,ANY} can also be used in comparisons with call lists like
@ref{3875,,mock_calls}:

@example
>>> m = MagicMock(return_value=None)
>>> m(1)
>>> m(1, 2)
>>> m(object())
>>> m.mock_calls == [call(1), call(1, 2), ANY]
True
@end example

@node FILTER_DIR,mock_open,ANY,Helpers
@anchor{library/unittest mock filter-dir}@anchor{38ab}
@subsubsection FILTER_DIR


@geindex FILTER_DIR (in module unittest.mock)
@anchor{library/unittest mock unittest mock FILTER_DIR}@anchor{387a}
@deffn {Data} unittest.mock.FILTER_DIR
@end deffn

@ref{387a,,FILTER_DIR} is a module level variable that controls the way mock objects
respond to @ref{f2f,,dir()} (only for Python 2.6 or more recent). The default is @code{True},
which uses the filtering described below, to only show useful members. If you
dislike this filtering, or need to switch it off for diagnostic purposes, then
set @code{mock.FILTER_DIR = False}.

With filtering on, @code{dir(some_mock)} shows only useful attributes and will
include any dynamically created attributes that wouldn’t normally be shown.
If the mock was created with a @emph{spec} (or @emph{autospec} of course) then all the
attributes from the original are shown, even if they haven’t been accessed
yet:

@example
>>> dir(Mock())
['assert_any_call',
 'assert_called',
 'assert_called_once',
 'assert_called_once_with',
 'assert_called_with',
 'assert_has_calls',
 'assert_not_called',
 'attach_mock',
 ...
>>> from urllib import request
>>> dir(Mock(spec=request))
['AbstractBasicAuthHandler',
 'AbstractDigestAuthHandler',
 'AbstractHTTPHandler',
 'BaseHandler',
 ...
@end example

Many of the not-very-useful (private to @ref{561,,Mock} rather than the thing being
mocked) underscore and double underscore prefixed attributes have been
filtered from the result of calling @ref{f2f,,dir()} on a @ref{561,,Mock}. If you dislike this
behaviour you can switch it off by setting the module level switch
@ref{387a,,FILTER_DIR}:

@example
>>> from unittest import mock
>>> mock.FILTER_DIR = False
>>> dir(mock.Mock())
['_NonCallableMock__get_return_value',
 '_NonCallableMock__get_side_effect',
 '_NonCallableMock__return_value_doc',
 '_NonCallableMock__set_return_value',
 '_NonCallableMock__set_side_effect',
 '__call__',
 '__class__',
 ...
@end example

Alternatively you can just use @code{vars(my_mock)} (instance members) and
@code{dir(type(my_mock))} (type members) to bypass the filtering irrespective of
@code{mock.FILTER_DIR}.

@node mock_open,Autospeccing,FILTER_DIR,Helpers
@anchor{library/unittest mock mock-open}@anchor{38ac}
@subsubsection mock_open


@geindex mock_open() (in module unittest.mock)
@anchor{library/unittest mock unittest mock mock_open}@anchor{1296}
@deffn {Function} unittest.mock.mock_open (mock=None, read_data=None)

A helper function to create a mock to replace the use of @ref{1c8,,open()}. It works
for @ref{1c8,,open()} called directly or used as a context manager.

The @emph{mock} argument is the mock object to configure. If @code{None} (the
default) then a @ref{a01,,MagicMock} will be created for you, with the API limited
to methods or attributes available on standard file handles.

@emph{read_data} is a string for the @code{read()},
@ref{1b74,,readline()}, and @ref{1c17,,readlines()} methods
of the file handle to return.  Calls to those methods will take data from
@emph{read_data} until it is depleted.  The mock of these methods is pretty
simplistic: every time the @emph{mock} is called, the @emph{read_data} is rewound to
the start.  If you need more control over the data that you are feeding to
the tested code you will need to customize this mock for yourself.  When that
is insufficient, one of the in-memory filesystem packages on PyPI@footnote{https://pypi.org} can offer a realistic filesystem for testing.


Changed in version 3.4: Added @ref{1b74,,readline()} and @ref{1c17,,readlines()} support.
The mock of @code{read()} changed to consume @emph{read_data} rather
than returning it on each call.



Changed in version 3.5: @emph{read_data} is now reset on each call to the @emph{mock}.



Changed in version 3.8: Added @code{__iter__()} to implementation so that iteration (such as in for
loops) correctly consumes @emph{read_data}.

@end deffn

Using @ref{1c8,,open()} as a context manager is a great way to ensure your file handles
are closed properly and is becoming common:

@example
with open('/some/path', 'w') as f:
    f.write('something')
@end example

The issue is that even if you mock out the call to @ref{1c8,,open()} it is the
@emph{returned object} that is used as a context manager (and has @code{__enter__()} and
@code{__exit__()} called).

Mocking context managers with a @ref{a01,,MagicMock} is common enough and fiddly
enough that a helper function is useful.

@example
>>> m = mock_open()
>>> with patch('__main__.open', m):
...     with open('foo', 'w') as h:
...         h.write('some stuff')
...
>>> m.mock_calls
[call('foo', 'w'),
 call().__enter__(),
 call().write('some stuff'),
 call().__exit__(None, None, None)]
>>> m.assert_called_once_with('foo', 'w')
>>> handle = m()
>>> handle.write.assert_called_once_with('some stuff')
@end example

And for reading files:

@example
>>> with patch('__main__.open', mock_open(read_data='bibble')) as m:
...     with open('foo') as h:
...         result = h.read()
...
>>> m.assert_called_once_with('foo')
>>> assert result == 'bibble'
@end example

@node Autospeccing,Sealing mocks,mock_open,Helpers
@anchor{library/unittest mock auto-speccing}@anchor{386a}@anchor{library/unittest mock autospeccing}@anchor{38ad}
@subsubsection Autospeccing


Autospeccing is based on the existing @code{spec} feature of mock. It limits the
api of mocks to the api of an original object (the spec), but it is recursive
(implemented lazily) so that attributes of mocks only have the same api as
the attributes of the spec. In addition mocked functions / methods have the
same call signature as the original so they raise a @ref{162,,TypeError} if they are
called incorrectly.

Before I explain how auto-speccing works, here’s why it is needed.

@ref{561,,Mock} is a very powerful and flexible object, but it suffers from two flaws
when used to mock out objects from a system under test. One of these flaws is
specific to the @ref{561,,Mock} api and the other is a more general problem with using
mock objects.

First the problem specific to @ref{561,,Mock}. @ref{561,,Mock} has two assert methods that are
extremely handy: @ref{12cc,,assert_called_with()} and
@ref{3872,,assert_called_once_with()}.

@example
>>> mock = Mock(name='Thing', return_value=None)
>>> mock(1, 2, 3)
>>> mock.assert_called_once_with(1, 2, 3)
>>> mock(1, 2, 3)
>>> mock.assert_called_once_with(1, 2, 3)
Traceback (most recent call last):
 ...
AssertionError: Expected 'mock' to be called once. Called 2 times.
@end example

Because mocks auto-create attributes on demand, and allow you to call them
with arbitrary arguments, if you misspell one of these assert methods then
your assertion is gone:

@example
>>> mock = Mock(name='Thing', return_value=None)
>>> mock(1, 2, 3)
>>> mock.assret_called_once_with(4, 5, 6)
@end example

Your tests can pass silently and incorrectly because of the typo.

The second issue is more general to mocking. If you refactor some of your
code, rename members and so on, any tests for code that is still using the
@emph{old api} but uses mocks instead of the real objects will still pass. This
means your tests can all pass even though your code is broken.

Note that this is another reason why you need integration tests as well as
unit tests. Testing everything in isolation is all fine and dandy, but if you
don’t test how your units are “wired together” there is still lots of room
for bugs that tests might have caught.

@code{mock} already provides a feature to help with this, called speccing. If you
use a class or instance as the @code{spec} for a mock then you can only access
attributes on the mock that exist on the real class:

@example
>>> from urllib import request
>>> mock = Mock(spec=request.Request)
>>> mock.assret_called_with
Traceback (most recent call last):
 ...
AttributeError: Mock object has no attribute 'assret_called_with'
@end example

The spec only applies to the mock itself, so we still have the same issue
with any methods on the mock:

@example
>>> mock.has_data()
<mock.Mock object at 0x...>
>>> mock.has_data.assret_called_with()
@end example

Auto-speccing solves this problem. You can either pass @code{autospec=True} to
@ref{a02,,patch()} / @ref{1250,,patch.object()} or use the @ref{122a,,create_autospec()} function to create a
mock with a spec. If you use the @code{autospec=True} argument to @ref{a02,,patch()} then the
object that is being replaced will be used as the spec object. Because the
speccing is done “lazily” (the spec is created as attributes on the mock are
accessed) you can use it with very complex or deeply nested objects (like
modules that import modules that import modules) without a big performance
hit.

Here’s an example of it in use:

@example
>>> from urllib import request
>>> patcher = patch('__main__.request', autospec=True)
>>> mock_request = patcher.start()
>>> request is mock_request
True
>>> mock_request.Request
<MagicMock name='request.Request' spec='Request' id='...'>
@end example

You can see that @code{request.Request} has a spec. @code{request.Request} takes two
arguments in the constructor (one of which is @emph{self}). Here’s what happens if
we try to call it incorrectly:

@example
>>> req = request.Request()
Traceback (most recent call last):
 ...
TypeError: <lambda>() takes at least 2 arguments (1 given)
@end example

The spec also applies to instantiated classes (i.e. the return value of
specced mocks):

@example
>>> req = request.Request('foo')
>>> req
<NonCallableMagicMock name='request.Request()' spec='Request' id='...'>
@end example

@code{Request} objects are not callable, so the return value of instantiating our
mocked out @code{request.Request} is a non-callable mock. With the spec in place
any typos in our asserts will raise the correct error:

@example
>>> req.add_header('spam', 'eggs')
<MagicMock name='request.Request().add_header()' id='...'>
>>> req.add_header.assret_called_with
Traceback (most recent call last):
 ...
AttributeError: Mock object has no attribute 'assret_called_with'
>>> req.add_header.assert_called_with('spam', 'eggs')
@end example

In many cases you will just be able to add @code{autospec=True} to your existing
@ref{a02,,patch()} calls and then be protected against bugs due to typos and api
changes.

As well as using @emph{autospec} through @ref{a02,,patch()} there is a
@ref{122a,,create_autospec()} for creating autospecced mocks directly:

@example
>>> from urllib import request
>>> mock_request = create_autospec(request)
>>> mock_request.Request('foo', 'bar')
<NonCallableMagicMock name='mock.Request()' spec='Request' id='...'>
@end example

This isn’t without caveats and limitations however, which is why it is not
the default behaviour. In order to know what attributes are available on the
spec object, autospec has to introspect (access attributes) the spec. As you
traverse attributes on the mock a corresponding traversal of the original
object is happening under the hood. If any of your specced objects have
properties or descriptors that can trigger code execution then you may not be
able to use autospec. On the other hand it is much better to design your
objects so that introspection is safe @footnote{@w{(4)} 
This only applies to classes or already instantiated objects. Calling
a mocked class to create a mock instance @emph{does not} create a real instance.
It is only attribute lookups - along with calls to @ref{f2f,,dir()} - that are done.
}.

A more serious problem is that it is common for instance attributes to be
created in the @code{__init__()} method and not to exist on the class at all.
@emph{autospec} can’t know about any dynamically created attributes and restricts
the api to visible attributes.

@example
>>> class Something:
...   def __init__(self):
...     self.a = 33
...
>>> with patch('__main__.Something', autospec=True):
...   thing = Something()
...   thing.a
...
Traceback (most recent call last):
  ...
AttributeError: Mock object has no attribute 'a'
@end example

There are a few different ways of resolving this problem. The easiest, but
not necessarily the least annoying, way is to simply set the required
attributes on the mock after creation. Just because @emph{autospec} doesn’t allow
you to fetch attributes that don’t exist on the spec it doesn’t prevent you
setting them:

@example
>>> with patch('__main__.Something', autospec=True):
...   thing = Something()
...   thing.a = 33
...
@end example

There is a more aggressive version of both @emph{spec} and @emph{autospec} that @emph{does}
prevent you setting non-existent attributes. This is useful if you want to
ensure your code only @emph{sets} valid attributes too, but obviously it prevents
this particular scenario:

@example
>>> with patch('__main__.Something', autospec=True, spec_set=True):
...   thing = Something()
...   thing.a = 33
...
Traceback (most recent call last):
 ...
AttributeError: Mock object has no attribute 'a'
@end example

Probably the best way of solving the problem is to add class attributes as
default values for instance members initialised in @code{__init__()}. Note that if
you are only setting default attributes in @code{__init__()} then providing them via
class attributes (shared between instances of course) is faster too. e.g.

@example
class Something:
    a = 33
@end example

This brings up another issue. It is relatively common to provide a default
value of @code{None} for members that will later be an object of a different type.
@code{None} would be useless as a spec because it wouldn’t let you access @emph{any}
attributes or methods on it. As @code{None} is @emph{never} going to be useful as a
spec, and probably indicates a member that will normally of some other type,
autospec doesn’t use a spec for members that are set to @code{None}. These will
just be ordinary mocks (well - MagicMocks):

@example
>>> class Something:
...     member = None
...
>>> mock = create_autospec(Something)
>>> mock.member.foo.bar.baz()
<MagicMock name='mock.member.foo.bar.baz()' id='...'>
@end example

If modifying your production classes to add defaults isn’t to your liking
then there are more options. One of these is simply to use an instance as the
spec rather than the class. The other is to create a subclass of the
production class and add the defaults to the subclass without affecting the
production class. Both of these require you to use an alternative object as
the spec. Thankfully @ref{a02,,patch()} supports this - you can simply pass the
alternative object as the @emph{autospec} argument:

@example
>>> class Something:
...   def __init__(self):
...     self.a = 33
...
>>> class SomethingForTest(Something):
...   a = 33
...
>>> p = patch('__main__.Something', autospec=SomethingForTest)
>>> mock = p.start()
>>> mock.a
<NonCallableMagicMock name='Something.a' spec='int' id='...'>
@end example

@node Sealing mocks,,Autospeccing,Helpers
@anchor{library/unittest mock sealing-mocks}@anchor{38ae}
@subsubsection Sealing mocks


@c from unittest.mock import seal

@geindex seal() (in module unittest.mock)
@anchor{library/unittest mock unittest mock seal}@anchor{6d2}
@deffn {Function} unittest.mock.seal (mock)

Seal will disable the automatic creation of mocks when accessing an attribute of
the mock being sealed or any of its attributes that are already mocks recursively.

If a mock instance with a name or a spec is assigned to an attribute
it won’t be considered in the sealing chain. This allows one to prevent seal from
fixing part of the mock object.

@example
>>> mock = Mock()
>>> mock.submock.attribute1 = 2
>>> mock.not_submock = mock.Mock(name="sample_name")
>>> seal(mock)
>>> mock.new_attribute  # This will raise AttributeError.
>>> mock.submock.attribute2  # This will raise AttributeError.
>>> mock.not_submock.attribute2  # This won't raise.
@end example


New in version 3.7.

@end deffn

@node unittest mock — getting started,2to3 - Automated Python 2 to 3 code translation,unittest mock — mock object library,Development Tools
@anchor{library/unittest mock-examples doc}@anchor{38af}@anchor{library/unittest mock-examples unittest-mock-getting-started}@anchor{38b0}
@subsection @code{unittest.mock} — getting started



New in version 3.3.

@anchor{library/unittest mock-examples getting-started}@anchor{38b1}
@c import asyncio
@c import unittest
@c from unittest.mock import Mock, MagicMock, AsyncMock, patch, call, sentinel
@c 
@c class SomeClass:
@c     attribute = 'this is a doctest'
@c 
@c     @staticmethod
@c     def static_method():
@c         pass

@menu
* Using Mock:: 
* Patch Decorators:: 
* Further Examples:: 

@end menu

@node Using Mock,Patch Decorators,,unittest mock — getting started
@anchor{library/unittest mock-examples using-mock}@anchor{38b2}
@subsubsection Using Mock


@menu
* Mock Patching Methods:: 
* Mock for Method Calls on an Object:: 
* Mocking Classes:: 
* Naming your mocks:: 
* Tracking all Calls:: 
* Setting Return Values and Attributes:: 
* Raising exceptions with mocks:: 
* Side effect functions and iterables:: 
* Mocking asynchronous iterators:: 
* Mocking asynchronous context manager:: 
* Creating a Mock from an Existing Object:: 

@end menu

@node Mock Patching Methods,Mock for Method Calls on an Object,,Using Mock
@anchor{library/unittest mock-examples mock-patching-methods}@anchor{38b3}
@subsubsection Mock Patching Methods


Common uses for @ref{561,,Mock} objects include:


@itemize *

@item 
Patching methods

@item 
Recording method calls on objects
@end itemize

You might want to replace a method on an object to check that
it is called with the correct arguments by another part of the system:

@example
>>> real = SomeClass()
>>> real.method = MagicMock(name='method')
>>> real.method(3, 4, 5, key='value')
<MagicMock name='method()' id='...'>
@end example

Once our mock has been used (@code{real.method} in this example) it has methods
and attributes that allow you to make assertions about how it has been used.

@cartouche
@quotation Note 
In most of these examples the @ref{561,,Mock} and @ref{a01,,MagicMock} classes
are interchangeable. As the @code{MagicMock} is the more capable class it makes
a sensible one to use by default.
@end quotation
@end cartouche

Once the mock has been called its @ref{387c,,called} attribute is set to
@code{True}. More importantly we can use the @ref{12cc,,assert_called_with()} or
@ref{3872,,assert_called_once_with()} method to check that it was called with
the correct arguments.

This example tests that calling @code{ProductionClass().method} results in a call to
the @code{something} method:

@example
>>> class ProductionClass:
...     def method(self):
...         self.something(1, 2, 3)
...     def something(self, a, b, c):
...         pass
...
>>> real = ProductionClass()
>>> real.something = MagicMock()
>>> real.method()
>>> real.something.assert_called_once_with(1, 2, 3)
@end example

@node Mock for Method Calls on an Object,Mocking Classes,Mock Patching Methods,Using Mock
@anchor{library/unittest mock-examples mock-for-method-calls-on-an-object}@anchor{38b4}
@subsubsection Mock for Method Calls on an Object


In the last example we patched a method directly on an object to check that it
was called correctly. Another common use case is to pass an object into a
method (or some part of the system under test) and then check that it is used
in the correct way.

The simple @code{ProductionClass} below has a @code{closer} method. If it is called with
an object then it calls @code{close} on it.

@example
>>> class ProductionClass:
...     def closer(self, something):
...         something.close()
...
@end example

So to test it we need to pass in an object with a @code{close} method and check
that it was called correctly.

@example
>>> real = ProductionClass()
>>> mock = Mock()
>>> real.closer(mock)
>>> mock.close.assert_called_with()
@end example

We don’t have to do any work to provide the ‘close’ method on our mock.
Accessing close creates it. So, if ‘close’ hasn’t already been called then
accessing it in the test will create it, but @ref{12cc,,assert_called_with()}
will raise a failure exception.

@node Mocking Classes,Naming your mocks,Mock for Method Calls on an Object,Using Mock
@anchor{library/unittest mock-examples mocking-classes}@anchor{38b5}
@subsubsection Mocking Classes


A common use case is to mock out classes instantiated by your code under test.
When you patch a class, then that class is replaced with a mock. Instances
are created by @emph{calling the class}. This means you access the “mock instance”
by looking at the return value of the mocked class.

In the example below we have a function @code{some_function} that instantiates @code{Foo}
and calls a method on it. The call to @ref{a02,,patch()} replaces the class @code{Foo} with a
mock. The @code{Foo} instance is the result of calling the mock, so it is configured
by modifying the mock @ref{3870,,return_value}.

@example
>>> def some_function():
...     instance = module.Foo()
...     return instance.method()
...
>>> with patch('module.Foo') as mock:
...     instance = mock.return_value
...     instance.method.return_value = 'the result'
...     result = some_function()
...     assert result == 'the result'
@end example

@node Naming your mocks,Tracking all Calls,Mocking Classes,Using Mock
@anchor{library/unittest mock-examples naming-your-mocks}@anchor{38b6}
@subsubsection Naming your mocks


It can be useful to give your mocks a name. The name is shown in the repr of
the mock and can be helpful when the mock appears in test failure messages. The
name is also propagated to attributes or methods of the mock:

@example
>>> mock = MagicMock(name='foo')
>>> mock
<MagicMock name='foo' id='...'>
>>> mock.method
<MagicMock name='foo.method' id='...'>
@end example

@node Tracking all Calls,Setting Return Values and Attributes,Naming your mocks,Using Mock
@anchor{library/unittest mock-examples tracking-all-calls}@anchor{38b7}
@subsubsection Tracking all Calls


Often you want to track more than a single call to a method. The
@ref{3875,,mock_calls} attribute records all calls
to child attributes of the mock - and also to their children.

@example
>>> mock = MagicMock()
>>> mock.method()
<MagicMock name='mock.method()' id='...'>
>>> mock.attribute.method(10, x=53)
<MagicMock name='mock.attribute.method()' id='...'>
>>> mock.mock_calls
[call.method(), call.attribute.method(10, x=53)]
@end example

If you make an assertion about @code{mock_calls} and any unexpected methods
have been called, then the assertion will fail. This is useful because as well
as asserting that the calls you expected have been made, you are also checking
that they were made in the right order and with no additional calls:

You use the @ref{1265,,call} object to construct lists for comparing with
@code{mock_calls}:

@example
>>> expected = [call.method(), call.attribute.method(10, x=53)]
>>> mock.mock_calls == expected
True
@end example

However, parameters to calls that return mocks are not recorded, which means it is not
possible to track nested calls where the parameters used to create ancestors are important:

@example
>>> m = Mock()
>>> m.factory(important=True).deliver()
<Mock name='mock.factory().deliver()' id='...'>
>>> m.mock_calls[-1] == call.factory(important=False).deliver()
True
@end example

@node Setting Return Values and Attributes,Raising exceptions with mocks,Tracking all Calls,Using Mock
@anchor{library/unittest mock-examples setting-return-values-and-attributes}@anchor{38b8}
@subsubsection Setting Return Values and Attributes


Setting the return values on a mock object is trivially easy:

@example
>>> mock = Mock()
>>> mock.return_value = 3
>>> mock()
3
@end example

Of course you can do the same for methods on the mock:

@example
>>> mock = Mock()
>>> mock.method.return_value = 3
>>> mock.method()
3
@end example

The return value can also be set in the constructor:

@example
>>> mock = Mock(return_value=3)
>>> mock()
3
@end example

If you need an attribute setting on your mock, just do it:

@example
>>> mock = Mock()
>>> mock.x = 3
>>> mock.x
3
@end example

Sometimes you want to mock up a more complex situation, like for example
@code{mock.connection.cursor().execute("SELECT 1")}. If we wanted this call to
return a list, then we have to configure the result of the nested call.

We can use @ref{1265,,call} to construct the set of calls in a “chained call” like
this for easy assertion afterwards:

@example
>>> mock = Mock()
>>> cursor = mock.connection.cursor.return_value
>>> cursor.execute.return_value = ['foo']
>>> mock.connection.cursor().execute("SELECT 1")
['foo']
>>> expected = call.connection.cursor().execute("SELECT 1").call_list()
>>> mock.mock_calls
[call.connection.cursor(), call.connection.cursor().execute('SELECT 1')]
>>> mock.mock_calls == expected
True
@end example

It is the call to @code{.call_list()} that turns our call object into a list of
calls representing the chained calls.

@node Raising exceptions with mocks,Side effect functions and iterables,Setting Return Values and Attributes,Using Mock
@anchor{library/unittest mock-examples raising-exceptions-with-mocks}@anchor{38b9}
@subsubsection Raising exceptions with mocks


A useful attribute is @ref{386e,,side_effect}. If you set this to an
exception class or instance then the exception will be raised when the mock
is called.

@example
>>> mock = Mock(side_effect=Exception('Boom!'))
>>> mock()
Traceback (most recent call last):
  ...
Exception: Boom!
@end example

@node Side effect functions and iterables,Mocking asynchronous iterators,Raising exceptions with mocks,Using Mock
@anchor{library/unittest mock-examples side-effect-functions-and-iterables}@anchor{38ba}
@subsubsection Side effect functions and iterables


@code{side_effect} can also be set to a function or an iterable. The use case for
@code{side_effect} as an iterable is where your mock is going to be called several
times, and you want each call to return a different value. When you set
@code{side_effect} to an iterable every call to the mock returns the next value
from the iterable:

@example
>>> mock = MagicMock(side_effect=[4, 5, 6])
>>> mock()
4
>>> mock()
5
>>> mock()
6
@end example

For more advanced use cases, like dynamically varying the return values
depending on what the mock is called with, @code{side_effect} can be a function.
The function will be called with the same arguments as the mock. Whatever the
function returns is what the call returns:

@example
>>> vals = @{(1, 2): 1, (2, 3): 2@}
>>> def side_effect(*args):
...     return vals[args]
...
>>> mock = MagicMock(side_effect=side_effect)
>>> mock(1, 2)
1
>>> mock(2, 3)
2
@end example

@node Mocking asynchronous iterators,Mocking asynchronous context manager,Side effect functions and iterables,Using Mock
@anchor{library/unittest mock-examples mocking-asynchronous-iterators}@anchor{38bb}
@subsubsection Mocking asynchronous iterators


Since Python 3.8, @code{AsyncMock} and @code{MagicMock} have support to mock
@ref{8d3,,Asynchronous Iterators} through @code{__aiter__}. The @ref{3870,,return_value}
attribute of @code{__aiter__} can be used to set the return values to be used for
iteration.

@example
>>> mock = MagicMock()  # AsyncMock also works here
>>> mock.__aiter__.return_value = [1, 2, 3]
>>> async def main():
...     return [i async for i in mock]
...
>>> asyncio.run(main())
[1, 2, 3]
@end example

@node Mocking asynchronous context manager,Creating a Mock from an Existing Object,Mocking asynchronous iterators,Using Mock
@anchor{library/unittest mock-examples mocking-asynchronous-context-manager}@anchor{38bc}
@subsubsection Mocking asynchronous context manager


Since Python 3.8, @code{AsyncMock} and @code{MagicMock} have support to mock
@ref{1821,,Asynchronous Context Managers} through @code{__aenter__} and @code{__aexit__}.
By default, @code{__aenter__} and @code{__aexit__} are @code{AsyncMock} instances that
return an async function.

@example
>>> class AsyncContextManager:
...     async def __aenter__(self):
...         return self
...     async def __aexit__(self, exc_type, exc, tb):
...         pass
...
>>> mock_instance = MagicMock(AsyncContextManager())  # AsyncMock also works here
>>> async def main():
...     async with mock_instance as result:
...         pass
...
>>> asyncio.run(main())
>>> mock_instance.__aenter__.assert_awaited_once()
>>> mock_instance.__aexit__.assert_awaited_once()
@end example

@node Creating a Mock from an Existing Object,,Mocking asynchronous context manager,Using Mock
@anchor{library/unittest mock-examples creating-a-mock-from-an-existing-object}@anchor{38bd}
@subsubsection Creating a Mock from an Existing Object


One problem with over use of mocking is that it couples your tests to the
implementation of your mocks rather than your real code. Suppose you have a
class that implements @code{some_method}. In a test for another class, you
provide a mock of this object that @emph{also} provides @code{some_method}. If later
you refactor the first class, so that it no longer has @code{some_method} - then
your tests will continue to pass even though your code is now broken!

@ref{561,,Mock} allows you to provide an object as a specification for the mock,
using the @emph{spec} keyword argument. Accessing methods / attributes on the
mock that don’t exist on your specification object will immediately raise an
attribute error. If you change the implementation of your specification, then
tests that use that class will start failing immediately without you having to
instantiate the class in those tests.

@example
>>> mock = Mock(spec=SomeClass)
>>> mock.old_method()
Traceback (most recent call last):
   ...
AttributeError: object has no attribute 'old_method'
@end example

Using a specification also enables a smarter matching of calls made to the
mock, regardless of whether some parameters were passed as positional or
named arguments:

@example
>>> def f(a, b, c): pass
...
>>> mock = Mock(spec=f)
>>> mock(1, 2, 3)
<Mock name='mock()' id='140161580456576'>
>>> mock.assert_called_with(a=1, b=2, c=3)
@end example

If you want this smarter matching to also work with method calls on the mock,
you can use @ref{386a,,auto-speccing}.

If you want a stronger form of specification that prevents the setting
of arbitrary attributes as well as the getting of them then you can use
@emph{spec_set} instead of @emph{spec}.

@node Patch Decorators,Further Examples,Using Mock,unittest mock — getting started
@anchor{library/unittest mock-examples patch-decorators}@anchor{38be}
@subsubsection Patch Decorators


@cartouche
@quotation Note 
With @ref{a02,,patch()} it matters that you patch objects in the namespace where
they are looked up. This is normally straightforward, but for a quick guide
read @ref{3868,,where to patch}.
@end quotation
@end cartouche

A common need in tests is to patch a class attribute or a module attribute,
for example patching a builtin or patching a class in a module to test that it
is instantiated. Modules and classes are effectively global, so patching on
them has to be undone after the test or the patch will persist into other
tests and cause hard to diagnose problems.

mock provides three convenient decorators for this: @ref{a02,,patch()}, @ref{1250,,patch.object()} and
@ref{129d,,patch.dict()}. @code{patch} takes a single string, of the form
@code{package.module.Class.attribute} to specify the attribute you are patching. It
also optionally takes a value that you want the attribute (or class or
whatever) to be replaced with. ‘patch.object’ takes an object and the name of
the attribute you would like patched, plus optionally the value to patch it
with.

@code{patch.object}:

@example
>>> original = SomeClass.attribute
>>> @@patch.object(SomeClass, 'attribute', sentinel.attribute)
... def test():
...     assert SomeClass.attribute == sentinel.attribute
...
>>> test()
>>> assert SomeClass.attribute == original

>>> @@patch('package.module.attribute', sentinel.attribute)
... def test():
...     from package.module import attribute
...     assert attribute is sentinel.attribute
...
>>> test()
@end example

If you are patching a module (including @ref{11,,builtins}) then use @ref{a02,,patch()}
instead of @ref{1250,,patch.object()}:

@example
>>> mock = MagicMock(return_value=sentinel.file_handle)
>>> with patch('builtins.open', mock):
...     handle = open('filename', 'r')
...
>>> mock.assert_called_with('filename', 'r')
>>> assert handle == sentinel.file_handle, "incorrect file handle returned"
@end example

The module name can be ‘dotted’, in the form @code{package.module} if needed:

@example
>>> @@patch('package.module.ClassName.attribute', sentinel.attribute)
... def test():
...     from package.module import ClassName
...     assert ClassName.attribute == sentinel.attribute
...
>>> test()
@end example

A nice pattern is to actually decorate test methods themselves:

@example
>>> class MyTest(unittest.TestCase):
...     @@patch.object(SomeClass, 'attribute', sentinel.attribute)
...     def test_something(self):
...         self.assertEqual(SomeClass.attribute, sentinel.attribute)
...
>>> original = SomeClass.attribute
>>> MyTest('test_something').test_something()
>>> assert SomeClass.attribute == original
@end example

If you want to patch with a Mock, you can use @ref{a02,,patch()} with only one argument
(or @ref{1250,,patch.object()} with two arguments). The mock will be created for you and
passed into the test function / method:

@example
>>> class MyTest(unittest.TestCase):
...     @@patch.object(SomeClass, 'static_method')
...     def test_something(self, mock_method):
...         SomeClass.static_method()
...         mock_method.assert_called_with()
...
>>> MyTest('test_something').test_something()
@end example

You can stack up multiple patch decorators using this pattern:

@example
>>> class MyTest(unittest.TestCase):
...     @@patch('package.module.ClassName1')
...     @@patch('package.module.ClassName2')
...     def test_something(self, MockClass2, MockClass1):
...         self.assertIs(package.module.ClassName1, MockClass1)
...         self.assertIs(package.module.ClassName2, MockClass2)
...
>>> MyTest('test_something').test_something()
@end example

When you nest patch decorators the mocks are passed in to the decorated
function in the same order they applied (the normal @emph{Python} order that
decorators are applied). This means from the bottom up, so in the example
above the mock for @code{test_module.ClassName2} is passed in first.

There is also @ref{129d,,patch.dict()} for setting values in a dictionary just
during a scope and restoring the dictionary to its original state when the test
ends:

@example
>>> foo = @{'key': 'value'@}
>>> original = foo.copy()
>>> with patch.dict(foo, @{'newkey': 'newvalue'@}, clear=True):
...     assert foo == @{'newkey': 'newvalue'@}
...
>>> assert foo == original
@end example

@code{patch}, @code{patch.object} and @code{patch.dict} can all be used as context managers.

Where you use @ref{a02,,patch()} to create a mock for you, you can get a reference to the
mock using the “as” form of the with statement:

@example
>>> class ProductionClass:
...     def method(self):
...         pass
...
>>> with patch.object(ProductionClass, 'method') as mock_method:
...     mock_method.return_value = None
...     real = ProductionClass()
...     real.method(1, 2, 3)
...
>>> mock_method.assert_called_with(1, 2, 3)
@end example

As an alternative @code{patch}, @code{patch.object} and @code{patch.dict} can be used as
class decorators. When used in this way it is the same as applying the
decorator individually to every method whose name starts with “test”.

@node Further Examples,,Patch Decorators,unittest mock — getting started
@anchor{library/unittest mock-examples further-examples}@anchor{38bf}@anchor{library/unittest mock-examples id1}@anchor{38c0}
@subsubsection Further Examples


Here are some more examples for some slightly more advanced scenarios.

@menu
* Mocking chained calls:: 
* Partial mocking:: 
* Mocking a Generator Method:: 
* Applying the same patch to every test method:: 
* Mocking Unbound Methods:: 
* Checking multiple calls with mock:: 
* Coping with mutable arguments:: 
* Nesting Patches:: 
* Mocking a dictionary with MagicMock:: 
* Mock subclasses and their attributes:: 
* Mocking imports with patch.dict: Mocking imports with patch dict. 
* Tracking order of calls and less verbose call assertions:: 
* More complex argument matching:: 

@end menu

@node Mocking chained calls,Partial mocking,,Further Examples
@anchor{library/unittest mock-examples mocking-chained-calls}@anchor{38c1}
@subsubsection Mocking chained calls


Mocking chained calls is actually straightforward with mock once you
understand the @ref{3870,,return_value} attribute. When a mock is called for
the first time, or you fetch its @code{return_value} before it has been called, a
new @ref{561,,Mock} is created.

This means that you can see how the object returned from a call to a mocked
object has been used by interrogating the @code{return_value} mock:

@example
>>> mock = Mock()
>>> mock().foo(a=2, b=3)
<Mock name='mock().foo()' id='...'>
>>> mock.return_value.foo.assert_called_with(a=2, b=3)
@end example

From here it is a simple step to configure and then make assertions about
chained calls. Of course another alternative is writing your code in a more
testable way in the first place…

So, suppose we have some code that looks a little bit like this:

@example
>>> class Something:
...     def __init__(self):
...         self.backend = BackendProvider()
...     def method(self):
...         response = self.backend.get_endpoint('foobar').create_call('spam', 'eggs').start_call()
...         # more code
@end example

Assuming that @code{BackendProvider} is already well tested, how do we test
@code{method()}? Specifically, we want to test that the code section @code{# more
code} uses the response object in the correct way.

As this chain of calls is made from an instance attribute we can monkey patch
the @code{backend} attribute on a @code{Something} instance. In this particular case
we are only interested in the return value from the final call to
@code{start_call} so we don’t have much configuration to do. Let’s assume the
object it returns is ‘file-like’, so we’ll ensure that our response object
uses the builtin @ref{1c8,,open()} as its @code{spec}.

To do this we create a mock instance as our mock backend and create a mock
response object for it. To set the response as the return value for that final
@code{start_call} we could do this:

@example
mock_backend.get_endpoint.return_value.create_call.return_value.start_call.return_value = mock_response
@end example

We can do that in a slightly nicer way using the @ref{3871,,configure_mock()}
method to directly set the return value for us:

@example
>>> something = Something()
>>> mock_response = Mock(spec=open)
>>> mock_backend = Mock()
>>> config = @{'get_endpoint.return_value.create_call.return_value.start_call.return_value': mock_response@}
>>> mock_backend.configure_mock(**config)
@end example

With these we monkey patch the “mock backend” in place and can make the real
call:

@example
>>> something.backend = mock_backend
>>> something.method()
@end example

Using @ref{3875,,mock_calls} we can check the chained call with a single
assert. A chained call is several calls in one line of code, so there will be
several entries in @code{mock_calls}. We can use @ref{38a8,,call.call_list()} to create
this list of calls for us:

@example
>>> chained = call.get_endpoint('foobar').create_call('spam', 'eggs').start_call()
>>> call_list = chained.call_list()
>>> assert mock_backend.mock_calls == call_list
@end example

@node Partial mocking,Mocking a Generator Method,Mocking chained calls,Further Examples
@anchor{library/unittest mock-examples partial-mocking}@anchor{38c2}
@subsubsection Partial mocking


In some tests I wanted to mock out a call to @ref{1d27,,datetime.date.today()}
to return a known date, but I didn’t want to prevent the code under test from
creating new date objects. Unfortunately @ref{3e2,,datetime.date} is written in C, and
so I couldn’t just monkey-patch out the static @code{date.today()} method.

I found a simple way of doing this that involved effectively wrapping the date
class with a mock, but passing through calls to the constructor to the real
class (and returning real instances).

The @ref{a02,,patch decorator} is used here to
mock out the @code{date} class in the module under test. The @code{side_effect}
attribute on the mock date class is then set to a lambda function that returns
a real date. When the mock date class is called a real date will be
constructed and returned by @code{side_effect}.

@example
>>> from datetime import date
>>> with patch('mymodule.date') as mock_date:
...     mock_date.today.return_value = date(2010, 10, 8)
...     mock_date.side_effect = lambda *args, **kw: date(*args, **kw)
...
...     assert mymodule.date.today() == date(2010, 10, 8)
...     assert mymodule.date(2009, 6, 8) == date(2009, 6, 8)
@end example

Note that we don’t patch @ref{3e2,,datetime.date} globally, we patch @code{date} in the
module that @emph{uses} it. See @ref{3868,,where to patch}.

When @code{date.today()} is called a known date is returned, but calls to the
@code{date(...)} constructor still return normal dates. Without this you can find
yourself having to calculate an expected result using exactly the same
algorithm as the code under test, which is a classic testing anti-pattern.

Calls to the date constructor are recorded in the @code{mock_date} attributes
(@code{call_count} and friends) which may also be useful for your tests.

An alternative way of dealing with mocking dates, or other builtin classes,
is discussed in this blog entry@footnote{https://williambert.online/2011/07/how-to-unit-testing-in-django-with-mocking-and-patching/}.

@node Mocking a Generator Method,Applying the same patch to every test method,Partial mocking,Further Examples
@anchor{library/unittest mock-examples mocking-a-generator-method}@anchor{38c3}
@subsubsection Mocking a Generator Method


A Python generator is a function or method that uses the @ref{4c5,,yield} statement
to return a series of values when iterated over @footnote{@w{(1)} 
There are also generator expressions and more @uref{http://www.dabeaz.com/coroutines/index.html,advanced uses} of generators, but we aren’t
concerned about them here. A very good introduction to generators and how
powerful they are is: @uref{http://www.dabeaz.com/generators/,Generator Tricks for Systems Programmers}.
}.

A generator method / function is called to return the generator object. It is
the generator object that is then iterated over. The protocol method for
iteration is @ref{1a83,,__iter__()}, so we can
mock this using a @ref{a01,,MagicMock}.

Here’s an example class with an “iter” method implemented as a generator:

@example
>>> class Foo:
...     def iter(self):
...         for i in [1, 2, 3]:
...             yield i
...
>>> foo = Foo()
>>> list(foo.iter())
[1, 2, 3]
@end example

How would we mock this class, and in particular its “iter” method?

To configure the values returned from the iteration (implicit in the call to
@ref{43a,,list}), we need to configure the object returned by the call to @code{foo.iter()}.

@example
>>> mock_foo = MagicMock()
>>> mock_foo.iter.return_value = iter([1, 2, 3])
>>> list(mock_foo.iter())
[1, 2, 3]
@end example

@node Applying the same patch to every test method,Mocking Unbound Methods,Mocking a Generator Method,Further Examples
@anchor{library/unittest mock-examples applying-the-same-patch-to-every-test-method}@anchor{38c4}
@subsubsection Applying the same patch to every test method


If you want several patches in place for multiple test methods the obvious way
is to apply the patch decorators to every method. This can feel like unnecessary
repetition. For Python 2.6 or more recent you can use @ref{a02,,patch()} (in all its
various forms) as a class decorator. This applies the patches to all test
methods on the class. A test method is identified by methods whose names start
with @code{test}:

@example
>>> @@patch('mymodule.SomeClass')
... class MyTest(unittest.TestCase):
...
...     def test_one(self, MockSomeClass):
...         self.assertIs(mymodule.SomeClass, MockSomeClass)
...
...     def test_two(self, MockSomeClass):
...         self.assertIs(mymodule.SomeClass, MockSomeClass)
...
...     def not_a_test(self):
...         return 'something'
...
>>> MyTest('test_one').test_one()
>>> MyTest('test_two').test_two()
>>> MyTest('test_two').not_a_test()
'something'
@end example

An alternative way of managing patches is to use the @ref{389a,,patch methods; start and stop}.
These allow you to move the patching into your @code{setUp} and @code{tearDown} methods.

@example
>>> class MyTest(unittest.TestCase):
...     def setUp(self):
...         self.patcher = patch('mymodule.foo')
...         self.mock_foo = self.patcher.start()
...
...     def test_foo(self):
...         self.assertIs(mymodule.foo, self.mock_foo)
...
...     def tearDown(self):
...         self.patcher.stop()
...
>>> MyTest('test_foo').run()
@end example

If you use this technique you must ensure that the patching is “undone” by
calling @code{stop}. This can be fiddlier than you might think, because if an
exception is raised in the setUp then tearDown is not called.
@ref{599,,unittest.TestCase.addCleanup()} makes this easier:

@example
>>> class MyTest(unittest.TestCase):
...     def setUp(self):
...         patcher = patch('mymodule.foo')
...         self.addCleanup(patcher.stop)
...         self.mock_foo = patcher.start()
...
...     def test_foo(self):
...         self.assertIs(mymodule.foo, self.mock_foo)
...
>>> MyTest('test_foo').run()
@end example

@node Mocking Unbound Methods,Checking multiple calls with mock,Applying the same patch to every test method,Further Examples
@anchor{library/unittest mock-examples mocking-unbound-methods}@anchor{38c5}
@subsubsection Mocking Unbound Methods


Whilst writing tests today I needed to patch an @emph{unbound method} (patching the
method on the class rather than on the instance). I needed self to be passed
in as the first argument because I want to make asserts about which objects
were calling this particular method. The issue is that you can’t patch with a
mock for this, because if you replace an unbound method with a mock it doesn’t
become a bound method when fetched from the instance, and so it doesn’t get
self passed in. The workaround is to patch the unbound method with a real
function instead. The @ref{a02,,patch()} decorator makes it so simple to
patch out methods with a mock that having to create a real function becomes a
nuisance.

If you pass @code{autospec=True} to patch then it does the patching with a
@emph{real} function object. This function object has the same signature as the one
it is replacing, but delegates to a mock under the hood. You still get your
mock auto-created in exactly the same way as before. What it means though, is
that if you use it to patch out an unbound method on a class the mocked
function will be turned into a bound method if it is fetched from an instance.
It will have @code{self} passed in as the first argument, which is exactly what I
wanted:

@example
>>> class Foo:
...   def foo(self):
...     pass
...
>>> with patch.object(Foo, 'foo', autospec=True) as mock_foo:
...   mock_foo.return_value = 'foo'
...   foo = Foo()
...   foo.foo()
...
'foo'
>>> mock_foo.assert_called_once_with(foo)
@end example

If we don’t use @code{autospec=True} then the unbound method is patched out
with a Mock instance instead, and isn’t called with @code{self}.

@node Checking multiple calls with mock,Coping with mutable arguments,Mocking Unbound Methods,Further Examples
@anchor{library/unittest mock-examples checking-multiple-calls-with-mock}@anchor{38c6}
@subsubsection Checking multiple calls with mock


mock has a nice API for making assertions about how your mock objects are used.

@example
>>> mock = Mock()
>>> mock.foo_bar.return_value = None
>>> mock.foo_bar('baz', spam='eggs')
>>> mock.foo_bar.assert_called_with('baz', spam='eggs')
@end example

If your mock is only being called once you can use the
@code{assert_called_once_with()} method that also asserts that the
@code{call_count} is one.

@example
>>> mock.foo_bar.assert_called_once_with('baz', spam='eggs')
>>> mock.foo_bar()
>>> mock.foo_bar.assert_called_once_with('baz', spam='eggs')
Traceback (most recent call last):
    ...
AssertionError: Expected to be called once. Called 2 times.
@end example

Both @code{assert_called_with} and @code{assert_called_once_with} make assertions about
the @emph{most recent} call. If your mock is going to be called several times, and
you want to make assertions about @emph{all} those calls you can use
@ref{387e,,call_args_list}:

@example
>>> mock = Mock(return_value=None)
>>> mock(1, 2, 3)
>>> mock(4, 5, 6)
>>> mock()
>>> mock.call_args_list
[call(1, 2, 3), call(4, 5, 6), call()]
@end example

The @ref{1265,,call} helper makes it easy to make assertions about these calls. You
can build up a list of expected calls and compare it to @code{call_args_list}. This
looks remarkably similar to the repr of the @code{call_args_list}:

@example
>>> expected = [call(1, 2, 3), call(4, 5, 6), call()]
>>> mock.call_args_list == expected
True
@end example

@node Coping with mutable arguments,Nesting Patches,Checking multiple calls with mock,Further Examples
@anchor{library/unittest mock-examples coping-with-mutable-arguments}@anchor{38c7}
@subsubsection Coping with mutable arguments


Another situation is rare, but can bite you, is when your mock is called with
mutable arguments. @code{call_args} and @code{call_args_list} store @emph{references} to the
arguments. If the arguments are mutated by the code under test then you can no
longer make assertions about what the values were when the mock was called.

Here’s some example code that shows the problem. Imagine the following functions
defined in ‘mymodule’:

@example
def frob(val):
    pass

def grob(val):
    "First frob and then clear val"
    frob(val)
    val.clear()
@end example

When we try to test that @code{grob} calls @code{frob} with the correct argument look
what happens:

@example
>>> with patch('mymodule.frob') as mock_frob:
...     val = @{6@}
...     mymodule.grob(val)
...
>>> val
set()
>>> mock_frob.assert_called_with(@{6@})
Traceback (most recent call last):
    ...
AssertionError: Expected: ((@{6@},), @{@})
Called with: ((set(),), @{@})
@end example

One possibility would be for mock to copy the arguments you pass in. This
could then cause problems if you do assertions that rely on object identity
for equality.

Here’s one solution that uses the @code{side_effect}
functionality. If you provide a @code{side_effect} function for a mock then
@code{side_effect} will be called with the same args as the mock. This gives us an
opportunity to copy the arguments and store them for later assertions. In this
example I’m using @emph{another} mock to store the arguments so that I can use the
mock methods for doing the assertion. Again a helper function sets this up for
me.

@example
>>> from copy import deepcopy
>>> from unittest.mock import Mock, patch, DEFAULT
>>> def copy_call_args(mock):
...     new_mock = Mock()
...     def side_effect(*args, **kwargs):
...         args = deepcopy(args)
...         kwargs = deepcopy(kwargs)
...         new_mock(*args, **kwargs)
...         return DEFAULT
...     mock.side_effect = side_effect
...     return new_mock
...
>>> with patch('mymodule.frob') as mock_frob:
...     new_mock = copy_call_args(mock_frob)
...     val = @{6@}
...     mymodule.grob(val)
...
>>> new_mock.assert_called_with(@{6@})
>>> new_mock.call_args
call(@{6@})
@end example

@code{copy_call_args} is called with the mock that will be called. It returns a new
mock that we do the assertion on. The @code{side_effect} function makes a copy of
the args and calls our @code{new_mock} with the copy.

@cartouche
@quotation Note 
If your mock is only going to be used once there is an easier way of
checking arguments at the point they are called. You can simply do the
checking inside a @code{side_effect} function.

@example
>>> def side_effect(arg):
...     assert arg == @{6@}
...
>>> mock = Mock(side_effect=side_effect)
>>> mock(@{6@})
>>> mock(set())
Traceback (most recent call last):
    ...
AssertionError
@end example
@end quotation
@end cartouche

An alternative approach is to create a subclass of @ref{561,,Mock} or
@ref{a01,,MagicMock} that copies (using @ref{69b,,copy.deepcopy()}) the arguments.
Here’s an example implementation:

@example
>>> from copy import deepcopy
>>> class CopyingMock(MagicMock):
...     def __call__(self, /, *args, **kwargs):
...         args = deepcopy(args)
...         kwargs = deepcopy(kwargs)
...         return super().__call__(*args, **kwargs)
...
>>> c = CopyingMock(return_value=None)
>>> arg = set()
>>> c(arg)
>>> arg.add(1)
>>> c.assert_called_with(set())
>>> c.assert_called_with(arg)
Traceback (most recent call last):
    ...
AssertionError: Expected call: mock(@{1@})
Actual call: mock(set())
>>> c.foo
<CopyingMock name='mock.foo' id='...'>
@end example

When you subclass @code{Mock} or @code{MagicMock} all dynamically created attributes,
and the @code{return_value} will use your subclass automatically. That means all
children of a @code{CopyingMock} will also have the type @code{CopyingMock}.

@node Nesting Patches,Mocking a dictionary with MagicMock,Coping with mutable arguments,Further Examples
@anchor{library/unittest mock-examples nesting-patches}@anchor{38c8}
@subsubsection Nesting Patches


Using patch as a context manager is nice, but if you do multiple patches you
can end up with nested with statements indenting further and further to the
right:

@example
>>> class MyTest(unittest.TestCase):
...
...     def test_foo(self):
...         with patch('mymodule.Foo') as mock_foo:
...             with patch('mymodule.Bar') as mock_bar:
...                 with patch('mymodule.Spam') as mock_spam:
...                     assert mymodule.Foo is mock_foo
...                     assert mymodule.Bar is mock_bar
...                     assert mymodule.Spam is mock_spam
...
>>> original = mymodule.Foo
>>> MyTest('test_foo').test_foo()
>>> assert mymodule.Foo is original
@end example

With unittest @code{cleanup} functions and the @ref{389a,,patch methods; start and stop} we can
achieve the same effect without the nested indentation. A simple helper
method, @code{create_patch}, puts the patch in place and returns the created mock
for us:

@example
>>> class MyTest(unittest.TestCase):
...
...     def create_patch(self, name):
...         patcher = patch(name)
...         thing = patcher.start()
...         self.addCleanup(patcher.stop)
...         return thing
...
...     def test_foo(self):
...         mock_foo = self.create_patch('mymodule.Foo')
...         mock_bar = self.create_patch('mymodule.Bar')
...         mock_spam = self.create_patch('mymodule.Spam')
...
...         assert mymodule.Foo is mock_foo
...         assert mymodule.Bar is mock_bar
...         assert mymodule.Spam is mock_spam
...
>>> original = mymodule.Foo
>>> MyTest('test_foo').run()
>>> assert mymodule.Foo is original
@end example

@node Mocking a dictionary with MagicMock,Mock subclasses and their attributes,Nesting Patches,Further Examples
@anchor{library/unittest mock-examples mocking-a-dictionary-with-magicmock}@anchor{38c9}
@subsubsection Mocking a dictionary with MagicMock


You may want to mock a dictionary, or other container object, recording all
access to it whilst having it still behave like a dictionary.

We can do this with @ref{a01,,MagicMock}, which will behave like a dictionary,
and using @ref{386e,,side_effect} to delegate dictionary access to a real
underlying dictionary that is under our control.

When the @code{__getitem__()} and @code{__setitem__()} methods of our @code{MagicMock} are called
(normal dictionary access) then @code{side_effect} is called with the key (and in
the case of @code{__setitem__} the value too). We can also control what is returned.

After the @code{MagicMock} has been used we can use attributes like
@ref{387e,,call_args_list} to assert about how the dictionary was used:

@example
>>> my_dict = @{'a': 1, 'b': 2, 'c': 3@}
>>> def getitem(name):
...      return my_dict[name]
...
>>> def setitem(name, val):
...     my_dict[name] = val
...
>>> mock = MagicMock()
>>> mock.__getitem__.side_effect = getitem
>>> mock.__setitem__.side_effect = setitem
@end example

@cartouche
@quotation Note 
An alternative to using @code{MagicMock} is to use @code{Mock} and @emph{only} provide
the magic methods you specifically want:

@example
>>> mock = Mock()
>>> mock.__getitem__ = Mock(side_effect=getitem)
>>> mock.__setitem__ = Mock(side_effect=setitem)
@end example

A @emph{third} option is to use @code{MagicMock} but passing in @code{dict} as the @emph{spec}
(or @emph{spec_set}) argument so that the @code{MagicMock} created only has
dictionary magic methods available:

@example
>>> mock = MagicMock(spec_set=dict)
>>> mock.__getitem__.side_effect = getitem
>>> mock.__setitem__.side_effect = setitem
@end example
@end quotation
@end cartouche

With these side effect functions in place, the @code{mock} will behave like a normal
dictionary but recording the access. It even raises a @ref{5bb,,KeyError} if you try
to access a key that doesn’t exist.

@example
>>> mock['a']
1
>>> mock['c']
3
>>> mock['d']
Traceback (most recent call last):
    ...
KeyError: 'd'
>>> mock['b'] = 'fish'
>>> mock['d'] = 'eggs'
>>> mock['b']
'fish'
>>> mock['d']
'eggs'
@end example

After it has been used you can make assertions about the access using the normal
mock methods and attributes:

@example
>>> mock.__getitem__.call_args_list
[call('a'), call('c'), call('d'), call('b'), call('d')]
>>> mock.__setitem__.call_args_list
[call('b', 'fish'), call('d', 'eggs')]
>>> my_dict
@{'a': 1, 'b': 'fish', 'c': 3, 'd': 'eggs'@}
@end example

@node Mock subclasses and their attributes,Mocking imports with patch dict,Mocking a dictionary with MagicMock,Further Examples
@anchor{library/unittest mock-examples mock-subclasses-and-their-attributes}@anchor{38ca}
@subsubsection Mock subclasses and their attributes


There are various reasons why you might want to subclass @ref{561,,Mock}. One
reason might be to add helper methods. Here’s a silly example:

@example
>>> class MyMock(MagicMock):
...     def has_been_called(self):
...         return self.called
...
>>> mymock = MyMock(return_value=None)
>>> mymock
<MyMock id='...'>
>>> mymock.has_been_called()
False
>>> mymock()
>>> mymock.has_been_called()
True
@end example

The standard behaviour for @code{Mock} instances is that attributes and the return
value mocks are of the same type as the mock they are accessed on. This ensures
that @code{Mock} attributes are @code{Mocks} and @code{MagicMock} attributes are @code{MagicMocks}
@footnote{@w{(2)} 
An exception to this rule are the non-callable mocks. Attributes use the
callable variant because otherwise non-callable mocks couldn’t have callable
methods.
}. So if you’re subclassing to add helper methods then they’ll also be
available on the attributes and return value mock of instances of your
subclass.

@example
>>> mymock.foo
<MyMock name='mock.foo' id='...'>
>>> mymock.foo.has_been_called()
False
>>> mymock.foo()
<MyMock name='mock.foo()' id='...'>
>>> mymock.foo.has_been_called()
True
@end example

Sometimes this is inconvenient. For example, one user@footnote{https://code.google.com/archive/p/mock/issues/105} is subclassing mock to
created a Twisted adaptor@footnote{https://twistedmatrix.com/documents/11.0.0/api/twisted.python.components.html}.
Having this applied to attributes too actually causes errors.

@code{Mock} (in all its flavours) uses a method called @code{_get_child_mock} to create
these “sub-mocks” for attributes and return values. You can prevent your
subclass being used for attributes by overriding this method. The signature is
that it takes arbitrary keyword arguments (@code{**kwargs}) which are then passed
onto the mock constructor:

@example
>>> class Subclass(MagicMock):
...     def _get_child_mock(self, /, **kwargs):
...         return MagicMock(**kwargs)
...
>>> mymock = Subclass()
>>> mymock.foo
<MagicMock name='mock.foo' id='...'>
>>> assert isinstance(mymock, Subclass)
>>> assert not isinstance(mymock.foo, Subclass)
>>> assert not isinstance(mymock(), Subclass)
@end example

@node Mocking imports with patch dict,Tracking order of calls and less verbose call assertions,Mock subclasses and their attributes,Further Examples
@anchor{library/unittest mock-examples mocking-imports-with-patch-dict}@anchor{38cb}
@subsubsection Mocking imports with patch.dict


One situation where mocking can be hard is where you have a local import inside
a function. These are harder to mock because they aren’t using an object from
the module namespace that we can patch out.

Generally local imports are to be avoided. They are sometimes done to prevent
circular dependencies, for which there is @emph{usually} a much better way to solve
the problem (refactor the code) or to prevent “up front costs” by delaying the
import. This can also be solved in better ways than an unconditional local
import (store the module as a class or module attribute and only do the import
on first use).

That aside there is a way to use @code{mock} to affect the results of an import.
Importing fetches an @emph{object} from the @ref{11f7,,sys.modules} dictionary. Note that it
fetches an @emph{object}, which need not be a module. Importing a module for the
first time results in a module object being put in @cite{sys.modules}, so usually
when you import something you get a module back. This need not be the case
however.

This means you can use @ref{129d,,patch.dict()} to @emph{temporarily} put a mock in place
in @ref{11f7,,sys.modules}. Any imports whilst this patch is active will fetch the mock.
When the patch is complete (the decorated function exits, the with statement
body is complete or @code{patcher.stop()} is called) then whatever was there
previously will be restored safely.

Here’s an example that mocks out the ‘fooble’ module.

@example
>>> import sys
>>> mock = Mock()
>>> with patch.dict('sys.modules', @{'fooble': mock@}):
...    import fooble
...    fooble.blob()
...
<Mock name='mock.blob()' id='...'>
>>> assert 'fooble' not in sys.modules
>>> mock.blob.assert_called_once_with()
@end example

As you can see the @code{import fooble} succeeds, but on exit there is no ‘fooble’
left in @ref{11f7,,sys.modules}.

This also works for the @code{from module import name} form:

@example
>>> mock = Mock()
>>> with patch.dict('sys.modules', @{'fooble': mock@}):
...    from fooble import blob
...    blob.blip()
...
<Mock name='mock.blob.blip()' id='...'>
>>> mock.blob.blip.assert_called_once_with()
@end example

With slightly more work you can also mock package imports:

@example
>>> mock = Mock()
>>> modules = @{'package': mock, 'package.module': mock.module@}
>>> with patch.dict('sys.modules', modules):
...    from package.module import fooble
...    fooble()
...
<Mock name='mock.module.fooble()' id='...'>
>>> mock.module.fooble.assert_called_once_with()
@end example

@node Tracking order of calls and less verbose call assertions,More complex argument matching,Mocking imports with patch dict,Further Examples
@anchor{library/unittest mock-examples tracking-order-of-calls-and-less-verbose-call-assertions}@anchor{38cc}
@subsubsection Tracking order of calls and less verbose call assertions


The @ref{561,,Mock} class allows you to track the @emph{order} of method calls on
your mock objects through the @ref{3878,,method_calls} attribute. This
doesn’t allow you to track the order of calls between separate mock objects,
however we can use @ref{3875,,mock_calls} to achieve the same effect.

Because mocks track calls to child mocks in @code{mock_calls}, and accessing an
arbitrary attribute of a mock creates a child mock, we can create our separate
mocks from a parent one. Calls to those child mock will then all be recorded,
in order, in the @code{mock_calls} of the parent:

@example
>>> manager = Mock()
>>> mock_foo = manager.foo
>>> mock_bar = manager.bar
@end example

@example
>>> mock_foo.something()
<Mock name='mock.foo.something()' id='...'>
>>> mock_bar.other.thing()
<Mock name='mock.bar.other.thing()' id='...'>
@end example

@example
>>> manager.mock_calls
[call.foo.something(), call.bar.other.thing()]
@end example

We can then assert about the calls, including the order, by comparing with
the @code{mock_calls} attribute on the manager mock:

@example
>>> expected_calls = [call.foo.something(), call.bar.other.thing()]
>>> manager.mock_calls == expected_calls
True
@end example

If @code{patch} is creating, and putting in place, your mocks then you can attach
them to a manager mock using the @ref{3877,,attach_mock()} method. After
attaching calls will be recorded in @code{mock_calls} of the manager.

@example
>>> manager = MagicMock()
>>> with patch('mymodule.Class1') as MockClass1:
...     with patch('mymodule.Class2') as MockClass2:
...         manager.attach_mock(MockClass1, 'MockClass1')
...         manager.attach_mock(MockClass2, 'MockClass2')
...         MockClass1().foo()
...         MockClass2().bar()
<MagicMock name='mock.MockClass1().foo()' id='...'>
<MagicMock name='mock.MockClass2().bar()' id='...'>
>>> manager.mock_calls
[call.MockClass1(),
call.MockClass1().foo(),
call.MockClass2(),
call.MockClass2().bar()]
@end example

If many calls have been made, but you’re only interested in a particular
sequence of them then an alternative is to use the
@ref{3874,,assert_has_calls()} method. This takes a list of calls (constructed
with the @ref{1265,,call} object). If that sequence of calls are in
@ref{3875,,mock_calls} then the assert succeeds.

@example
>>> m = MagicMock()
>>> m().foo().bar().baz()
<MagicMock name='mock().foo().bar().baz()' id='...'>
>>> m.one().two().three()
<MagicMock name='mock.one().two().three()' id='...'>
>>> calls = call.one().two().three().call_list()
>>> m.assert_has_calls(calls)
@end example

Even though the chained call @code{m.one().two().three()} aren’t the only calls that
have been made to the mock, the assert still succeeds.

Sometimes a mock may have several calls made to it, and you are only interested
in asserting about @emph{some} of those calls. You may not even care about the
order. In this case you can pass @code{any_order=True} to @code{assert_has_calls}:

@example
>>> m = MagicMock()
>>> m(1), m.two(2, 3), m.seven(7), m.fifty('50')
(...)
>>> calls = [call.fifty('50'), call(1), call.seven(7)]
>>> m.assert_has_calls(calls, any_order=True)
@end example

@node More complex argument matching,,Tracking order of calls and less verbose call assertions,Further Examples
@anchor{library/unittest mock-examples more-complex-argument-matching}@anchor{38cd}
@subsubsection More complex argument matching


Using the same basic concept as @ref{126a,,ANY} we can implement matchers to do more
complex assertions on objects used as arguments to mocks.

Suppose we expect some object to be passed to a mock that by default
compares equal based on object identity (which is the Python default for user
defined classes). To use @ref{12cc,,assert_called_with()} we would need to pass
in the exact same object. If we are only interested in some of the attributes
of this object then we can create a matcher that will check these attributes
for us.

You can see in this example how a ‘standard’ call to @code{assert_called_with} isn’t
sufficient:

@example
>>> class Foo:
...     def __init__(self, a, b):
...         self.a, self.b = a, b
...
>>> mock = Mock(return_value=None)
>>> mock(Foo(1, 2))
>>> mock.assert_called_with(Foo(1, 2))
Traceback (most recent call last):
    ...
AssertionError: Expected: call(<__main__.Foo object at 0x...>)
Actual call: call(<__main__.Foo object at 0x...>)
@end example

A comparison function for our @code{Foo} class might look something like this:

@example
>>> def compare(self, other):
...     if not type(self) == type(other):
...         return False
...     if self.a != other.a:
...         return False
...     if self.b != other.b:
...         return False
...     return True
...
@end example

And a matcher object that can use comparison functions like this for its
equality operation would look something like this:

@example
>>> class Matcher:
...     def __init__(self, compare, some_obj):
...         self.compare = compare
...         self.some_obj = some_obj
...     def __eq__(self, other):
...         return self.compare(self.some_obj, other)
...
@end example

Putting all this together:

@example
>>> match_foo = Matcher(compare, Foo(1, 2))
>>> mock.assert_called_with(match_foo)
@end example

The @code{Matcher} is instantiated with our compare function and the @code{Foo} object
we want to compare against. In @code{assert_called_with} the @code{Matcher} equality
method will be called, which compares the object the mock was called with
against the one we created our matcher with. If they match then
@code{assert_called_with} passes, and if they don’t an @ref{46e,,AssertionError} is raised:

@example
>>> match_wrong = Matcher(compare, Foo(3, 4))
>>> mock.assert_called_with(match_wrong)
Traceback (most recent call last):
    ...
AssertionError: Expected: ((<Matcher object at 0x...>,), @{@})
Called with: ((<Foo object at 0x...>,), @{@})
@end example

With a bit of tweaking you could have the comparison function raise the
@ref{46e,,AssertionError} directly and provide a more useful failure message.

As of version 1.5, the Python testing library PyHamcrest@footnote{https://pyhamcrest.readthedocs.io/} provides similar functionality,
that may be useful here, in the form of its equality matcher
(hamcrest.library.integration.match_equality@footnote{https://pyhamcrest.readthedocs.io/en/release-1.8/integration/#module-hamcrest.library.integration.match_equality}).

@node 2to3 - Automated Python 2 to 3 code translation,test — Regression tests package for Python,unittest mock — getting started,Development Tools
@anchor{library/2to3 doc}@anchor{38ce}@anchor{library/2to3 to3-automated-python-2-to-3-code-translation}@anchor{38cf}@anchor{library/2to3 to3-reference}@anchor{e7d}
@subsection 2to3 - Automated Python 2 to 3 code translation


2to3 is a Python program that reads Python 2.x source code and applies a series
of @emph{fixers} to transform it into valid Python 3.x code.  The standard library
contains a rich set of fixers that will handle almost all code.  2to3 supporting
library @ref{a3,,lib2to3} is, however, a flexible and generic library, so it is
possible to write your own fixers for 2to3.


Deprecated since version 3.11, will be removed in version 3.13: The @code{lib2to3} module was marked pending for deprecation in Python 3.9
(raising @ref{3a9,,PendingDeprecationWarning} on import) and fully deprecated
in Python 3.11 (raising @ref{1b2,,DeprecationWarning}).  The @code{2to3} tool is
part of that.  It will be removed in Python 3.13.


@menu
* Using 2to3:: 
* Fixers:: 
* lib2to3 - 2to3’s library:: 

@end menu

@node Using 2to3,Fixers,,2to3 - Automated Python 2 to 3 code translation
@anchor{library/2to3 to3-using}@anchor{38d0}@anchor{library/2to3 using-2to3}@anchor{38d1}
@subsubsection Using 2to3


2to3 will usually be installed with the Python interpreter as a script.  It is
also located in the @code{Tools/scripts} directory of the Python root.

2to3’s basic arguments are a list of files or directories to transform.  The
directories are recursively traversed for Python sources.

Here is a sample Python 2.x source file, @code{example.py}:

@example
def greet(name):
    print "Hello, @{0@}!".format(name)
print "What's your name?"
name = raw_input()
greet(name)
@end example

It can be converted to Python 3.x code via 2to3 on the command line:

@example
$ 2to3 example.py
@end example

A diff against the original source file is printed.  2to3 can also write the
needed modifications right back to the source file.  (A backup of the original
file is made unless @code{-n} is also given.)  Writing the changes back is
enabled with the @code{-w} flag:

@example
$ 2to3 -w example.py
@end example

After transformation, @code{example.py} looks like this:

@example
def greet(name):
    print("Hello, @{0@}!".format(name))
print("What's your name?")
name = input()
greet(name)
@end example

Comments and exact indentation are preserved throughout the translation process.

By default, 2to3 runs a set of @ref{38d2,,predefined fixers}.  The
@code{-l} flag lists all available fixers.  An explicit set of fixers to run
can be given with @code{-f}.  Likewise the @code{-x} explicitly disables a
fixer.  The following example runs only the @code{imports} and @code{has_key} fixers:

@example
$ 2to3 -f imports -f has_key example.py
@end example

This command runs every fixer except the @code{apply} fixer:

@example
$ 2to3 -x apply example.py
@end example

Some fixers are @emph{explicit}, meaning they aren’t run by default and must be
listed on the command line to be run.  Here, in addition to the default fixers,
the @code{idioms} fixer is run:

@example
$ 2to3 -f all -f idioms example.py
@end example

Notice how passing @code{all} enables all default fixers.

Sometimes 2to3 will find a place in your source code that needs to be changed,
but 2to3 cannot fix automatically.  In this case, 2to3 will print a warning
beneath the diff for a file.  You should address the warning in order to have
compliant 3.x code.

2to3 can also refactor doctests.  To enable this mode, use the @code{-d}
flag.  Note that @emph{only} doctests will be refactored.  This also doesn’t require
the module to be valid Python.  For example, doctest like examples in a reST
document could also be refactored with this option.

The @code{-v} option enables output of more information on the translation
process.

Since some print statements can be parsed as function calls or statements, 2to3
cannot always read files containing the print function.  When 2to3 detects the
presence of the @code{from __future__ import print_function} compiler directive, it
modifies its internal grammar to interpret @ref{af6,,print()} as a function.  This
change can also be enabled manually with the @code{-p} flag.  Use
@code{-p} to run fixers on code that already has had its print statements
converted. Also @code{-e} can be used to make @ref{34b,,exec()} a function.

The @code{-o} or @code{--output-dir} option allows specification of an
alternate directory for processed output files to be written to.  The
@code{-n} flag is required when using this as backup files do not make sense
when not overwriting the input files.


New in version 3.2.3: The @code{-o} option was added.


The @code{-W} or @code{--write-unchanged-files} flag tells 2to3 to always
write output files even if no changes were required to the file.  This is most
useful with @code{-o} so that an entire Python source tree is copied with
translation from one directory to another.
This option implies the @code{-w} flag as it would not make sense otherwise.


New in version 3.2.3: The @code{-W} flag was added.


The @code{--add-suffix} option specifies a string to append to all output
filenames.  The @code{-n} flag is required when specifying this as backups
are not necessary when writing to different filenames.  Example:

@example
$ 2to3 -n -W --add-suffix=3 example.py
@end example

Will cause a converted file named @code{example.py3} to be written.


New in version 3.2.3: The @code{--add-suffix} option was added.


To translate an entire project from one directory tree to another use:

@example
$ 2to3 --output-dir=python3-version/mycode -W -n python2-version/mycode
@end example

@node Fixers,lib2to3 - 2to3’s library,Using 2to3,2to3 - Automated Python 2 to 3 code translation
@anchor{library/2to3 fixers}@anchor{38d3}@anchor{library/2to3 to3-fixers}@anchor{38d2}
@subsubsection Fixers


Each step of transforming code is encapsulated in a fixer.  The command @code{2to3
-l} lists them.  As @ref{38d0,,documented above}, each can be turned on
and off individually.  They are described here in more detail.

@geindex apply (2to3 fixer)
@anchor{library/2to3 to3fixer-apply}@anchor{38d4}@anchor{library/2to3 2to3fixer-apply}@anchor{38d5}
@deffn {2to3fixer} apply

Removes usage of @code{apply()}.  For example @code{apply(function, *args,
**kwargs)} is converted to @code{function(*args, **kwargs)}.
@end deffn

@geindex asserts (2to3 fixer)
@anchor{library/2to3 to3fixer-asserts}@anchor{38d6}@anchor{library/2to3 2to3fixer-asserts}@anchor{38d7}
@deffn {2to3fixer} asserts

Replaces deprecated @ref{122,,unittest} method names with the correct ones.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

From

@tab

To

@item

@code{failUnlessEqual(a, b)}

@tab

@ref{dd5,,assertEqual(a@comma{} b)}

@item

@code{assertEquals(a, b)}

@tab

@ref{dd5,,assertEqual(a@comma{} b)}

@item

@code{failIfEqual(a, b)}

@tab

@ref{dd6,,assertNotEqual(a@comma{} b)}

@item

@code{assertNotEquals(a, b)}

@tab

@ref{dd6,,assertNotEqual(a@comma{} b)}

@item

@code{failUnless(a)}

@tab

@ref{dd4,,assertTrue(a)}

@item

@code{assert_(a)}

@tab

@ref{dd4,,assertTrue(a)}

@item

@code{failIf(a)}

@tab

@ref{ec9,,assertFalse(a)}

@item

@code{failUnlessRaises(exc, cal)}

@tab

@ref{cff,,assertRaises(exc@comma{} cal)}

@item

@code{failUnlessAlmostEqual(a, b)}

@tab

@ref{dd7,,assertAlmostEqual(a@comma{} b)}

@item

@code{assertAlmostEquals(a, b)}

@tab

@ref{dd7,,assertAlmostEqual(a@comma{} b)}

@item

@code{failIfAlmostEqual(a, b)}

@tab

@ref{dd8,,assertNotAlmostEqual(a@comma{} b)}

@item

@code{assertNotAlmostEquals(a, b)}

@tab

@ref{dd8,,assertNotAlmostEqual(a@comma{} b)}

@end multitable

@end deffn

@geindex basestring (2to3 fixer)
@anchor{library/2to3 to3fixer-basestring}@anchor{38d8}@anchor{library/2to3 2to3fixer-basestring}@anchor{38d9}
@deffn {2to3fixer} basestring

Converts @code{basestring} to @ref{2ef,,str}.
@end deffn

@geindex buffer (2to3 fixer)
@anchor{library/2to3 to3fixer-buffer}@anchor{38da}@anchor{library/2to3 2to3fixer-buffer}@anchor{38db}
@deffn {2to3fixer} buffer

Converts @code{buffer} to @ref{573,,memoryview}.  This fixer is optional
because the @ref{573,,memoryview} API is similar but not exactly the same as
that of @code{buffer}.
@end deffn

@geindex dict (2to3 fixer)
@anchor{library/2to3 to3fixer-dict}@anchor{38dc}@anchor{library/2to3 2to3fixer-dict}@anchor{38dd}
@deffn {2to3fixer} dict

Fixes dictionary iteration methods.  @code{dict.iteritems()} is converted to
@ref{245,,dict.items()}, @code{dict.iterkeys()} to @ref{243,,dict.keys()}, and
@code{dict.itervalues()} to @ref{244,,dict.values()}.  Similarly,
@code{dict.viewitems()}, @code{dict.viewkeys()} and @code{dict.viewvalues()} are
converted respectively to @ref{245,,dict.items()}, @ref{243,,dict.keys()} and
@ref{244,,dict.values()}.  It also wraps existing usages of @ref{245,,dict.items()},
@ref{243,,dict.keys()}, and @ref{244,,dict.values()} in a call to @ref{43a,,list}.
@end deffn

@geindex except (2to3 fixer)
@anchor{library/2to3 to3fixer-except}@anchor{38de}@anchor{library/2to3 2to3fixer-except}@anchor{38df}
@deffn {2to3fixer} except

Converts @code{except X, T} to @code{except X as T}.
@end deffn

@geindex exec (2to3 fixer)
@anchor{library/2to3 to3fixer-exec}@anchor{38e0}@anchor{library/2to3 2to3fixer-exec}@anchor{38e1}
@deffn {2to3fixer} exec

Converts the @code{exec} statement to the @ref{34b,,exec()} function.
@end deffn

@geindex execfile (2to3 fixer)
@anchor{library/2to3 to3fixer-execfile}@anchor{12eb}@anchor{library/2to3 2to3fixer-execfile}@anchor{38e2}
@deffn {2to3fixer} execfile

Removes usage of @code{execfile()}.  The argument to @code{execfile()} is
wrapped in calls to @ref{1c8,,open()}, @ref{4e2,,compile()}, and @ref{34b,,exec()}.
@end deffn

@geindex exitfunc (2to3 fixer)
@anchor{library/2to3 to3fixer-exitfunc}@anchor{38e3}@anchor{library/2to3 2to3fixer-exitfunc}@anchor{38e4}
@deffn {2to3fixer} exitfunc

Changes assignment of @code{sys.exitfunc} to use of the @ref{b,,atexit}
module.
@end deffn

@geindex filter (2to3 fixer)
@anchor{library/2to3 to3fixer-filter}@anchor{38e5}@anchor{library/2to3 2to3fixer-filter}@anchor{38e6}
@deffn {2to3fixer} filter

Wraps @ref{2f5,,filter()} usage in a @ref{43a,,list} call.
@end deffn

@geindex funcattrs (2to3 fixer)
@anchor{library/2to3 to3fixer-funcattrs}@anchor{38e7}@anchor{library/2to3 2to3fixer-funcattrs}@anchor{38e8}
@deffn {2to3fixer} funcattrs

Fixes function attributes that have been renamed.  For example,
@code{my_function.func_closure} is converted to @code{my_function.__closure__}.
@end deffn

@geindex future (2to3 fixer)
@anchor{library/2to3 to3fixer-future}@anchor{38e9}@anchor{library/2to3 2to3fixer-future}@anchor{38ea}
@deffn {2to3fixer} future

Removes @code{from __future__ import new_feature} statements.
@end deffn

@geindex getcwdu (2to3 fixer)
@anchor{library/2to3 to3fixer-getcwdu}@anchor{38eb}@anchor{library/2to3 2to3fixer-getcwdu}@anchor{38ec}
@deffn {2to3fixer} getcwdu

Renames @code{os.getcwdu()} to @ref{12ac,,os.getcwd()}.
@end deffn

@geindex has_key (2to3 fixer)
@anchor{library/2to3 to3fixer-has_key}@anchor{38ed}@anchor{library/2to3 2to3fixer-has_key}@anchor{38ee}
@deffn {2to3fixer} has_key

Changes @code{dict.has_key(key)} to @code{key in dict}.
@end deffn

@geindex idioms (2to3 fixer)
@anchor{library/2to3 to3fixer-idioms}@anchor{38ef}@anchor{library/2to3 2to3fixer-idioms}@anchor{38f0}
@deffn {2to3fixer} idioms

This optional fixer performs several transformations that make Python code
more idiomatic.  Type comparisons like @code{type(x) is SomeClass} and
@code{type(x) == SomeClass} are converted to @code{isinstance(x, SomeClass)}.
@code{while 1} becomes @code{while True}.  This fixer also tries to make use of
@ref{704,,sorted()} in appropriate places.  For example, this block

@example
L = list(some_iterable)
L.sort()
@end example

is changed to

@example
L = sorted(some_iterable)
@end example
@end deffn

@geindex import (2to3 fixer)
@anchor{library/2to3 to3fixer-import}@anchor{38f1}@anchor{library/2to3 2to3fixer-import}@anchor{38f2}
@deffn {2to3fixer} import

Detects sibling imports and converts them to relative imports.
@end deffn

@geindex imports (2to3 fixer)
@anchor{library/2to3 to3fixer-imports}@anchor{38f3}@anchor{library/2to3 2to3fixer-imports}@anchor{38f4}
@deffn {2to3fixer} imports

Handles module renames in the standard library.
@end deffn

@geindex imports2 (2to3 fixer)
@anchor{library/2to3 to3fixer-imports2}@anchor{38f5}@anchor{library/2to3 2to3fixer-imports2}@anchor{38f6}
@deffn {2to3fixer} imports2

Handles other modules renames in the standard library.  It is separate from
the @ref{38f3,,imports} fixer only because of technical limitations.
@end deffn

@geindex input (2to3 fixer)
@anchor{library/2to3 to3fixer-input}@anchor{38f7}@anchor{library/2to3 2to3fixer-input}@anchor{38f8}
@deffn {2to3fixer} input

Converts @code{input(prompt)} to @code{eval(input(prompt))}.
@end deffn

@geindex intern (2to3 fixer)
@anchor{library/2to3 to3fixer-intern}@anchor{38f9}@anchor{library/2to3 2to3fixer-intern}@anchor{38fa}
@deffn {2to3fixer} intern

Converts @code{intern()} to @ref{e76,,sys.intern()}.
@end deffn

@geindex isinstance (2to3 fixer)
@anchor{library/2to3 to3fixer-isinstance}@anchor{38fb}@anchor{library/2to3 2to3fixer-isinstance}@anchor{38fc}
@deffn {2to3fixer} isinstance

Fixes duplicate types in the second argument of @ref{236,,isinstance()}.  For
example, @code{isinstance(x, (int, int))} is converted to @code{isinstance(x,
int)} and @code{isinstance(x, (int, float, int))} is converted to
@code{isinstance(x, (int, float))}.
@end deffn

@geindex itertools_imports (2to3 fixer)
@anchor{library/2to3 to3fixer-itertools_imports}@anchor{38fd}@anchor{library/2to3 2to3fixer-itertools_imports}@anchor{38fe}
@deffn {2to3fixer} itertools_imports

Removes imports of @code{itertools.ifilter()}, @code{itertools.izip()}, and
@code{itertools.imap()}.  Imports of @code{itertools.ifilterfalse()} are also
changed to @ref{1a44,,itertools.filterfalse()}.
@end deffn

@geindex itertools (2to3 fixer)
@anchor{library/2to3 to3fixer-itertools}@anchor{38ff}@anchor{library/2to3 2to3fixer-itertools}@anchor{3900}
@deffn {2to3fixer} itertools

Changes usage of @code{itertools.ifilter()}, @code{itertools.izip()}, and
@code{itertools.imap()} to their built-in equivalents.
@code{itertools.ifilterfalse()} is changed to @ref{1a44,,itertools.filterfalse()}.
@end deffn

@geindex long (2to3 fixer)
@anchor{library/2to3 to3fixer-long}@anchor{3901}@anchor{library/2to3 2to3fixer-long}@anchor{3902}
@deffn {2to3fixer} long

Renames @code{long} to @ref{1b1,,int}.
@end deffn

@geindex map (2to3 fixer)
@anchor{library/2to3 to3fixer-map}@anchor{3903}@anchor{library/2to3 2to3fixer-map}@anchor{3904}
@deffn {2to3fixer} map

Wraps @ref{2f4,,map()} in a @ref{43a,,list} call.  It also changes @code{map(None, x)}
to @code{list(x)}.  Using @code{from future_builtins import map} disables this
fixer.
@end deffn

@geindex metaclass (2to3 fixer)
@anchor{library/2to3 to3fixer-metaclass}@anchor{3905}@anchor{library/2to3 2to3fixer-metaclass}@anchor{3906}
@deffn {2to3fixer} metaclass

Converts the old metaclass syntax (@code{__metaclass__ = Meta} in the class
body) to the new (@code{class X(metaclass=Meta)}).
@end deffn

@geindex methodattrs (2to3 fixer)
@anchor{library/2to3 to3fixer-methodattrs}@anchor{3907}@anchor{library/2to3 2to3fixer-methodattrs}@anchor{3908}
@deffn {2to3fixer} methodattrs

Fixes old method attribute names.  For example, @code{meth.im_func} is converted
to @code{meth.__func__}.
@end deffn

@geindex ne (2to3 fixer)
@anchor{library/2to3 to3fixer-ne}@anchor{3909}@anchor{library/2to3 2to3fixer-ne}@anchor{390a}
@deffn {2to3fixer} ne

Converts the old not-equal syntax, @code{<>}, to @code{!=}.
@end deffn

@geindex next (2to3 fixer)
@anchor{library/2to3 to3fixer-next}@anchor{390b}@anchor{library/2to3 2to3fixer-next}@anchor{390c}
@deffn {2to3fixer} next

Converts the use of iterator’s @code{next()} methods to the
@ref{252,,next()} function.  It also renames @ref{252,,next()} methods to
@ref{e6d,,__next__()}.
@end deffn

@geindex nonzero (2to3 fixer)
@anchor{library/2to3 to3fixer-nonzero}@anchor{390d}@anchor{library/2to3 2to3fixer-nonzero}@anchor{390e}
@deffn {2to3fixer} nonzero

Renames definitions of methods called @code{__nonzero__()}
to @ref{17ac,,__bool__()}.
@end deffn

@geindex numliterals (2to3 fixer)
@anchor{library/2to3 to3fixer-numliterals}@anchor{390f}@anchor{library/2to3 2to3fixer-numliterals}@anchor{3910}
@deffn {2to3fixer} numliterals

Converts octal literals into the new syntax.
@end deffn

@geindex operator (2to3 fixer)
@anchor{library/2to3 to3fixer-operator}@anchor{3911}@anchor{library/2to3 2to3fixer-operator}@anchor{3912}
@deffn {2to3fixer} operator

Converts calls to various functions in the @ref{be,,operator} module to other,
but equivalent, function calls.  When needed, the appropriate @code{import}
statements are added, e.g. @code{import collections.abc}.  The following mapping
are made:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

From

@tab

To

@item

@code{operator.isCallable(obj)}

@tab

@code{callable(obj)}

@item

@code{operator.sequenceIncludes(obj)}

@tab

@code{operator.contains(obj)}

@item

@code{operator.isSequenceType(obj)}

@tab

@code{isinstance(obj, collections.abc.Sequence)}

@item

@code{operator.isMappingType(obj)}

@tab

@code{isinstance(obj, collections.abc.Mapping)}

@item

@code{operator.isNumberType(obj)}

@tab

@code{isinstance(obj, numbers.Number)}

@item

@code{operator.repeat(obj, n)}

@tab

@code{operator.mul(obj, n)}

@item

@code{operator.irepeat(obj, n)}

@tab

@code{operator.imul(obj, n)}

@end multitable

@end deffn

@geindex paren (2to3 fixer)
@anchor{library/2to3 to3fixer-paren}@anchor{3913}@anchor{library/2to3 2to3fixer-paren}@anchor{3914}
@deffn {2to3fixer} paren

Add extra parenthesis where they are required in list comprehensions.  For
example, @code{[x for x in 1, 2]} becomes @code{[x for x in (1, 2)]}.
@end deffn

@geindex print (2to3 fixer)
@anchor{library/2to3 to3fixer-print}@anchor{3915}@anchor{library/2to3 2to3fixer-print}@anchor{3916}
@deffn {2to3fixer} print

Converts the @code{print} statement to the @ref{af6,,print()} function.
@end deffn

@geindex raise (2to3 fixer)
@anchor{library/2to3 to3fixer-raise}@anchor{3917}@anchor{library/2to3 2to3fixer-raise}@anchor{3918}
@deffn {2to3fixer} raise

Converts @code{raise E, V} to @code{raise E(V)}, and @code{raise E, V, T} to @code{raise
E(V).with_traceback(T)}.  If @code{E} is a tuple, the translation will be
incorrect because substituting tuples for exceptions has been removed in 3.0.
@end deffn

@geindex raw_input (2to3 fixer)
@anchor{library/2to3 to3fixer-raw_input}@anchor{3919}@anchor{library/2to3 2to3fixer-raw_input}@anchor{391a}
@deffn {2to3fixer} raw_input

Converts @code{raw_input()} to @ref{e73,,input()}.
@end deffn

@geindex reduce (2to3 fixer)
@anchor{library/2to3 to3fixer-reduce}@anchor{391b}@anchor{library/2to3 2to3fixer-reduce}@anchor{391c}
@deffn {2to3fixer} reduce

Handles the move of @code{reduce()} to @ref{e77,,functools.reduce()}.
@end deffn

@geindex reload (2to3 fixer)
@anchor{library/2to3 to3fixer-reload}@anchor{12f9}@anchor{library/2to3 2to3fixer-reload}@anchor{391d}
@deffn {2to3fixer} reload

Converts @code{reload()} to @ref{66a,,importlib.reload()}.
@end deffn

@geindex renames (2to3 fixer)
@anchor{library/2to3 to3fixer-renames}@anchor{391e}@anchor{library/2to3 2to3fixer-renames}@anchor{391f}
@deffn {2to3fixer} renames

Changes @code{sys.maxint} to @ref{d69,,sys.maxsize}.
@end deffn

@geindex repr (2to3 fixer)
@anchor{library/2to3 to3fixer-repr}@anchor{3920}@anchor{library/2to3 2to3fixer-repr}@anchor{3921}
@deffn {2to3fixer} repr

Replaces backtick repr with the @ref{27d,,repr()} function.
@end deffn

@geindex set_literal (2to3 fixer)
@anchor{library/2to3 to3fixer-set_literal}@anchor{3922}@anchor{library/2to3 2to3fixer-set_literal}@anchor{3923}
@deffn {2to3fixer} set_literal

Replaces use of the @ref{370,,set} constructor with set literals.  This fixer
is optional.
@end deffn

@geindex standarderror (2to3 fixer)
@anchor{library/2to3 to3fixer-standarderror}@anchor{3924}@anchor{library/2to3 2to3fixer-standarderror}@anchor{3925}
@deffn {2to3fixer} standarderror

Renames @code{StandardError} to @ref{4d8,,Exception}.
@end deffn

@geindex sys_exc (2to3 fixer)
@anchor{library/2to3 to3fixer-sys_exc}@anchor{3926}@anchor{library/2to3 2to3fixer-sys_exc}@anchor{3927}
@deffn {2to3fixer} sys_exc

Changes the deprecated @code{sys.exc_value}, @code{sys.exc_type},
@code{sys.exc_traceback} to use @ref{18f,,sys.exc_info()}.
@end deffn

@geindex throw (2to3 fixer)
@anchor{library/2to3 to3fixer-throw}@anchor{3928}@anchor{library/2to3 2to3fixer-throw}@anchor{3929}
@deffn {2to3fixer} throw

Fixes the API change in generator’s @code{throw()} method.
@end deffn

@geindex tuple_params (2to3 fixer)
@anchor{library/2to3 to3fixer-tuple_params}@anchor{392a}@anchor{library/2to3 2to3fixer-tuple_params}@anchor{392b}
@deffn {2to3fixer} tuple_params

Removes implicit tuple parameter unpacking.  This fixer inserts temporary
variables.
@end deffn

@geindex types (2to3 fixer)
@anchor{library/2to3 to3fixer-types}@anchor{392c}@anchor{library/2to3 2to3fixer-types}@anchor{392d}
@deffn {2to3fixer} types

Fixes code broken from the removal of some members in the @ref{11f,,types}
module.
@end deffn

@geindex unicode (2to3 fixer)
@anchor{library/2to3 to3fixer-unicode}@anchor{392e}@anchor{library/2to3 2to3fixer-unicode}@anchor{392f}
@deffn {2to3fixer} unicode

Renames @code{unicode} to @ref{2ef,,str}.
@end deffn

@geindex urllib (2to3 fixer)
@anchor{library/2to3 to3fixer-urllib}@anchor{3930}@anchor{library/2to3 2to3fixer-urllib}@anchor{3931}
@deffn {2to3fixer} urllib

Handles the rename of @ref{124,,urllib} and @code{urllib2} to the @ref{124,,urllib}
package.
@end deffn

@geindex ws_comma (2to3 fixer)
@anchor{library/2to3 to3fixer-ws_comma}@anchor{3932}@anchor{library/2to3 2to3fixer-ws_comma}@anchor{3933}
@deffn {2to3fixer} ws_comma

Removes excess whitespace from comma separated items.  This fixer is
optional.
@end deffn

@geindex xrange (2to3 fixer)
@anchor{library/2to3 to3fixer-xrange}@anchor{3934}@anchor{library/2to3 2to3fixer-xrange}@anchor{3935}
@deffn {2to3fixer} xrange

Renames @code{xrange()} to @ref{438,,range()} and wraps existing @ref{438,,range()}
calls with @ref{43a,,list}.
@end deffn

@geindex xreadlines (2to3 fixer)
@anchor{library/2to3 to3fixer-xreadlines}@anchor{3936}@anchor{library/2to3 2to3fixer-xreadlines}@anchor{3937}
@deffn {2to3fixer} xreadlines

Changes @code{for x in file.xreadlines()} to @code{for x in file}.
@end deffn

@geindex zip (2to3 fixer)
@anchor{library/2to3 to3fixer-zip}@anchor{3938}@anchor{library/2to3 2to3fixer-zip}@anchor{3939}
@deffn {2to3fixer} zip

Wraps @ref{247,,zip()} usage in a @ref{43a,,list} call.  This is disabled when
@code{from future_builtins import zip} appears.
@end deffn

@node lib2to3 - 2to3’s library,,Fixers,2to3 - Automated Python 2 to 3 code translation
@anchor{library/2to3 lib2to3-2to3-s-library}@anchor{393a}@anchor{library/2to3 module-lib2to3}@anchor{a3}
@subsubsection @code{lib2to3} - 2to3’s library


@geindex module; lib2to3

@strong{Source code:} Lib/lib2to3/@footnote{https://github.com/python/cpython/tree/main/Lib/lib2to3/}


__________________________________________________________________



Deprecated since version 3.11, will be removed in version 3.13: Python 3.9 switched to a PEG parser (see 
@geindex Python Enhancement Proposals; PEP 617
PEP 617@footnote{https://www.python.org/dev/peps/pep-0617}) while lib2to3 is
using a less flexible LL(1) parser.  Python 3.10 includes new language
syntax that is not parsable by lib2to3’s LL(1) parser (see 
@geindex Python Enhancement Proposals; PEP 634
PEP 634@footnote{https://www.python.org/dev/peps/pep-0634}).
The @code{lib2to3} module was marked pending for deprecation in Python 3.9
(raising @ref{3a9,,PendingDeprecationWarning} on import) and fully deprecated
in Python 3.11 (raising @ref{1b2,,DeprecationWarning}).
It will be removed from the standard library in Python 3.13.
Consider third-party alternatives such as LibCST@footnote{https://libcst.readthedocs.io/} or parso@footnote{https://parso.readthedocs.io/}.


@cartouche
@quotation Note 
The @ref{a3,,lib2to3} API should be considered unstable and may change
drastically in the future.
@end quotation
@end cartouche

@node test — Regression tests package for Python,test support — Utilities for the Python test suite,2to3 - Automated Python 2 to 3 code translation,Development Tools
@anchor{library/test doc}@anchor{393b}@anchor{library/test module-test}@anchor{ff}@anchor{library/test parso}@anchor{393c}@anchor{library/test test-regression-tests-package-for-python}@anchor{393d}
@subsection @code{test} — Regression tests package for Python


@geindex module; test

@cartouche
@quotation Note 
The @ref{ff,,test} package is meant for internal use by Python only. It is
documented for the benefit of the core developers of Python. Any use of
this package outside of Python’s standard library is discouraged as code
mentioned here can change or be removed without notice between releases of
Python.
@end quotation
@end cartouche


__________________________________________________________________


The @ref{ff,,test} package contains all regression tests for Python as well as the
modules @ref{100,,test.support} and @code{test.regrtest}.
@ref{100,,test.support} is used to enhance your tests while
@code{test.regrtest} drives the testing suite.

Each module in the @ref{ff,,test} package whose name starts with @code{test_} is a
testing suite for a specific module or feature. All new tests should be written
using the @ref{122,,unittest} or @ref{63,,doctest} module.  Some older tests are
written using a “traditional” testing style that compares output printed to
@code{sys.stdout}; this style of test is considered deprecated.


@subsubheading See also



@table @asis

@item Module @ref{122,,unittest}

Writing PyUnit regression tests.

@item Module @ref{63,,doctest}

Tests embedded in documentation strings.
@end table


@menu
* Writing Unit Tests for the test package:: 
* Running tests using the command-line interface:: 

@end menu

@node Writing Unit Tests for the test package,Running tests using the command-line interface,,test — Regression tests package for Python
@anchor{library/test writing-tests}@anchor{393e}@anchor{library/test writing-unit-tests-for-the-test-package}@anchor{393f}
@subsubsection Writing Unit Tests for the @code{test} package


It is preferred that tests that use the @ref{122,,unittest} module follow a few
guidelines. One is to name the test module by starting it with @code{test_} and end
it with the name of the module being tested. The test methods in the test module
should start with @code{test_} and end with a description of what the method is
testing. This is needed so that the methods are recognized by the test driver as
test methods. Also, no documentation string for the method should be included. A
comment (such as @code{# Tests function returns only True or False}) should be used
to provide documentation for test methods. This is done because documentation
strings get printed out if they exist and thus what test is being run is not
stated.

A basic boilerplate is often used:

@example
import unittest
from test import support

class MyTestCase1(unittest.TestCase):

    # Only use setUp() and tearDown() if necessary

    def setUp(self):
        ... code to execute in preparation for tests ...

    def tearDown(self):
        ... code to execute to clean up after tests ...

    def test_feature_one(self):
        # Test feature one.
        ... testing code ...

    def test_feature_two(self):
        # Test feature two.
        ... testing code ...

    ... more test methods ...

class MyTestCase2(unittest.TestCase):
    ... same structure as MyTestCase1 ...

... more test classes ...

if __name__ == '__main__':
    unittest.main()
@end example

This code pattern allows the testing suite to be run by @code{test.regrtest},
on its own as a script that supports the @ref{122,,unittest} CLI, or via the
@code{python -m unittest} CLI.

The goal for regression testing is to try to break code. This leads to a few
guidelines to be followed:


@itemize *

@item 
The testing suite should exercise all classes, functions, and constants. This
includes not just the external API that is to be presented to the outside
world but also “private” code.

@item 
Whitebox testing (examining the code being tested when the tests are being
written) is preferred. Blackbox testing (testing only the published user
interface) is not complete enough to make sure all boundary and edge cases
are tested.

@item 
Make sure all possible values are tested including invalid ones. This makes
sure that not only all valid values are acceptable but also that improper
values are handled correctly.

@item 
Exhaust as many code paths as possible. Test where branching occurs and thus
tailor input to make sure as many different paths through the code are taken.

@item 
Add an explicit test for any bugs discovered for the tested code. This will
make sure that the error does not crop up again if the code is changed in the
future.

@item 
Make sure to clean up after your tests (such as close and remove all temporary
files).

@item 
If a test is dependent on a specific condition of the operating system then
verify the condition already exists before attempting the test.

@item 
Import as few modules as possible and do it as soon as possible. This
minimizes external dependencies of tests and also minimizes possible anomalous
behavior from side-effects of importing a module.

@item 
Try to maximize code reuse. On occasion, tests will vary by something as small
as what type of input is used. Minimize code duplication by subclassing a
basic test class with a class that specifies the input:

@example
class TestFuncAcceptsSequencesMixin:

    func = mySuperWhammyFunction

    def test_func(self):
        self.func(self.arg)

class AcceptLists(TestFuncAcceptsSequencesMixin, unittest.TestCase):
    arg = [1, 2, 3]

class AcceptStrings(TestFuncAcceptsSequencesMixin, unittest.TestCase):
    arg = 'abc'

class AcceptTuples(TestFuncAcceptsSequencesMixin, unittest.TestCase):
    arg = (1, 2, 3)
@end example

When using this pattern, remember that all classes that inherit from
@ref{1ab,,unittest.TestCase} are run as tests.  The @code{Mixin} class in the example above
does not have any data and so can’t be run by itself, thus it does not
inherit from @ref{1ab,,unittest.TestCase}.
@end itemize


@subsubheading See also



@table @asis

@item Test Driven Development

A book by Kent Beck on writing tests before code.
@end table


@node Running tests using the command-line interface,,Writing Unit Tests for the test package,test — Regression tests package for Python
@anchor{library/test regrtest}@anchor{b8f}@anchor{library/test running-tests-using-the-command-line-interface}@anchor{3940}
@subsubsection Running tests using the command-line interface


The @ref{ff,,test} package can be run as a script to drive Python’s regression
test suite, thanks to the @ref{611,,-m} option: @code{python -m test}. Under
the hood, it uses @code{test.regrtest}; the call @code{python -m
test.regrtest} used in previous Python versions still works.  Running the
script by itself automatically starts running all regression tests in the
@ref{ff,,test} package. It does this by finding all modules in the package whose
name starts with @code{test_}, importing them, and executing the function
@code{test_main()} if present or loading the tests via
unittest.TestLoader.loadTestsFromModule if @code{test_main} does not exist.  The
names of tests to execute may also be passed to the script. Specifying a single
regression test (@code{python -m test test_spam}) will minimize output and
only print whether the test passed or failed.

Running @ref{ff,,test} directly allows what resources are available for
tests to use to be set. You do this by using the @code{-u} command-line
option. Specifying @code{all} as the value for the @code{-u} option enables all
possible resources: @code{python -m test -uall}.
If all but one resource is desired (a more common case), a
comma-separated list of resources that are not desired may be listed after
@code{all}. The command @code{python -m test -uall,-audio,-largefile}
will run @ref{ff,,test} with all resources except the @code{audio} and
@code{largefile} resources. For a list of all resources and more command-line
options, run @code{python -m test -h}.

Some other ways to execute the regression tests depend on what platform the
tests are being executed on. On Unix, you can run @code{make test} at the
top-level directory where Python was built. On Windows,
executing @code{rt.bat} from your @code{PCbuild} directory will run all
regression tests.

@node test support — Utilities for the Python test suite,test support socket_helper — Utilities for socket tests,test — Regression tests package for Python,Development Tools
@anchor{library/test module-test support}@anchor{100}@anchor{library/test test-support-utilities-for-the-python-test-suite}@anchor{3941}
@subsection @code{test.support} — Utilities for the Python test suite


@geindex module; test.support

The @ref{100,,test.support} module provides support for Python’s regression
test suite.

@cartouche
@quotation Note 
@ref{100,,test.support} is not a public module.  It is documented here to help
Python developers write tests.  The API of this module is subject to change
without backwards compatibility concerns between releases.
@end quotation
@end cartouche

This module defines the following exceptions:

@geindex TestFailed
@anchor{library/test test support TestFailed}@anchor{3942}
@deffn {Exception} test.support.TestFailed

Exception to be raised when a test fails. This is deprecated in favor of
@ref{122,,unittest}-based tests and @ref{1ab,,unittest.TestCase}’s assertion
methods.
@end deffn

@geindex ResourceDenied
@anchor{library/test test support ResourceDenied}@anchor{3943}
@deffn {Exception} test.support.ResourceDenied

Subclass of @ref{b6a,,unittest.SkipTest}. Raised when a resource (such as a
network connection) is not available. Raised by the @ref{3944,,requires()}
function.
@end deffn

The @ref{100,,test.support} module defines the following constants:

@geindex verbose (in module test.support)
@anchor{library/test test support verbose}@anchor{3945}
@deffn {Data} test.support.verbose

@code{True} when verbose output is enabled. Should be checked when more
detailed information is desired about a running test. @emph{verbose} is set by
@code{test.regrtest}.
@end deffn

@geindex is_jython (in module test.support)
@anchor{library/test test support is_jython}@anchor{3946}
@deffn {Data} test.support.is_jython

@code{True} if the running interpreter is Jython.
@end deffn

@geindex is_android (in module test.support)
@anchor{library/test test support is_android}@anchor{3947}
@deffn {Data} test.support.is_android

@code{True} if the system is Android.
@end deffn

@geindex unix_shell (in module test.support)
@anchor{library/test test support unix_shell}@anchor{3948}
@deffn {Data} test.support.unix_shell

Path for shell if not on Windows; otherwise @code{None}.
@end deffn

@geindex LOOPBACK_TIMEOUT (in module test.support)
@anchor{library/test test support LOOPBACK_TIMEOUT}@anchor{124d}
@deffn {Data} test.support.LOOPBACK_TIMEOUT

Timeout in seconds for tests using a network server listening on the network
local loopback interface like @code{127.0.0.1}.

The timeout is long enough to prevent test failure: it takes into account
that the client and the server can run in different threads or even
different processes.

The timeout should be long enough for @ref{8ff,,connect()},
@ref{900,,recv()} and @ref{903,,send()} methods of
@ref{8fd,,socket.socket}.

Its default value is 5 seconds.

See also @ref{124e,,INTERNET_TIMEOUT}.
@end deffn

@geindex INTERNET_TIMEOUT (in module test.support)
@anchor{library/test test support INTERNET_TIMEOUT}@anchor{124e}
@deffn {Data} test.support.INTERNET_TIMEOUT

Timeout in seconds for network requests going to the internet.

The timeout is short enough to prevent a test to wait for too long if the
internet request is blocked for whatever reason.

Usually, a timeout using @ref{124e,,INTERNET_TIMEOUT} should not mark a test as
failed, but skip the test instead: see
@ref{3949,,transient_internet()}.

Its default value is 1 minute.

See also @ref{124d,,LOOPBACK_TIMEOUT}.
@end deffn

@geindex SHORT_TIMEOUT (in module test.support)
@anchor{library/test test support SHORT_TIMEOUT}@anchor{1120}
@deffn {Data} test.support.SHORT_TIMEOUT

Timeout in seconds to mark a test as failed if the test takes “too long”.

The timeout value depends on the regrtest @code{--timeout} command line option.

If a test using @ref{1120,,SHORT_TIMEOUT} starts to fail randomly on slow
buildbots, use @ref{124f,,LONG_TIMEOUT} instead.

Its default value is 30 seconds.
@end deffn

@geindex LONG_TIMEOUT (in module test.support)
@anchor{library/test test support LONG_TIMEOUT}@anchor{124f}
@deffn {Data} test.support.LONG_TIMEOUT

Timeout in seconds to detect when a test hangs.

It is long enough to reduce the risk of test failure on the slowest Python
buildbots. It should not be used to mark a test as failed if the test takes
“too long”.  The timeout value depends on the regrtest @code{--timeout} command
line option.

Its default value is 5 minutes.

See also @ref{124d,,LOOPBACK_TIMEOUT}, @ref{124e,,INTERNET_TIMEOUT} and
@ref{1120,,SHORT_TIMEOUT}.
@end deffn

@geindex PGO (in module test.support)
@anchor{library/test test support PGO}@anchor{394a}
@deffn {Data} test.support.PGO

Set when tests can be skipped when they are not useful for PGO.
@end deffn

@geindex PIPE_MAX_SIZE (in module test.support)
@anchor{library/test test support PIPE_MAX_SIZE}@anchor{394b}
@deffn {Data} test.support.PIPE_MAX_SIZE

A constant that is likely larger than the underlying OS pipe buffer size,
to make writes blocking.
@end deffn

@geindex SOCK_MAX_SIZE (in module test.support)
@anchor{library/test test support SOCK_MAX_SIZE}@anchor{394c}
@deffn {Data} test.support.SOCK_MAX_SIZE

A constant that is likely larger than the underlying OS socket buffer size,
to make writes blocking.
@end deffn

@geindex TEST_SUPPORT_DIR (in module test.support)
@anchor{library/test test support TEST_SUPPORT_DIR}@anchor{394d}
@deffn {Data} test.support.TEST_SUPPORT_DIR

Set to the top level directory that contains @ref{100,,test.support}.
@end deffn

@geindex TEST_HOME_DIR (in module test.support)
@anchor{library/test test support TEST_HOME_DIR}@anchor{394e}
@deffn {Data} test.support.TEST_HOME_DIR

Set to the top level directory for the test package.
@end deffn

@geindex TEST_DATA_DIR (in module test.support)
@anchor{library/test test support TEST_DATA_DIR}@anchor{394f}
@deffn {Data} test.support.TEST_DATA_DIR

Set to the @code{data} directory within the test package.
@end deffn

@geindex MAX_Py_ssize_t (in module test.support)
@anchor{library/test test support MAX_Py_ssize_t}@anchor{3950}
@deffn {Data} test.support.MAX_Py_ssize_t

Set to @ref{d69,,sys.maxsize} for big memory tests.
@end deffn

@geindex max_memuse (in module test.support)
@anchor{library/test test support max_memuse}@anchor{3951}
@deffn {Data} test.support.max_memuse

Set by @ref{3952,,set_memlimit()} as the memory limit for big memory tests.
Limited by @ref{3950,,MAX_Py_ssize_t}.
@end deffn

@geindex real_max_memuse (in module test.support)
@anchor{library/test test support real_max_memuse}@anchor{3953}
@deffn {Data} test.support.real_max_memuse

Set by @ref{3952,,set_memlimit()} as the memory limit for big memory tests.  Not
limited by @ref{3950,,MAX_Py_ssize_t}.
@end deffn

@geindex MISSING_C_DOCSTRINGS (in module test.support)
@anchor{library/test test support MISSING_C_DOCSTRINGS}@anchor{3954}
@deffn {Data} test.support.MISSING_C_DOCSTRINGS

Return @code{True} if running on CPython, not on Windows, and configuration
not set with @code{WITH_DOC_STRINGS}.
@end deffn

@geindex HAVE_DOCSTRINGS (in module test.support)
@anchor{library/test test support HAVE_DOCSTRINGS}@anchor{3955}
@deffn {Data} test.support.HAVE_DOCSTRINGS

Check for presence of docstrings.
@end deffn

@geindex TEST_HTTP_URL (in module test.support)
@anchor{library/test test support TEST_HTTP_URL}@anchor{3956}
@deffn {Data} test.support.TEST_HTTP_URL

Define the URL of a dedicated HTTP server for the network tests.
@end deffn

@geindex ALWAYS_EQ (in module test.support)
@anchor{library/test test support ALWAYS_EQ}@anchor{3957}
@deffn {Data} test.support.ALWAYS_EQ

Object that is equal to anything.  Used to test mixed type comparison.
@end deffn

@geindex NEVER_EQ (in module test.support)
@anchor{library/test test support NEVER_EQ}@anchor{3958}
@deffn {Data} test.support.NEVER_EQ

Object that is not equal to anything (even to @ref{3957,,ALWAYS_EQ}).
Used to test mixed type comparison.
@end deffn

@geindex LARGEST (in module test.support)
@anchor{library/test test support LARGEST}@anchor{3959}
@deffn {Data} test.support.LARGEST

Object that is greater than anything (except itself).
Used to test mixed type comparison.
@end deffn

@geindex SMALLEST (in module test.support)
@anchor{library/test test support SMALLEST}@anchor{395a}
@deffn {Data} test.support.SMALLEST

Object that is less than anything (except itself).
Used to test mixed type comparison.
@end deffn

The @ref{100,,test.support} module defines the following functions:

@geindex is_resource_enabled() (in module test.support)
@anchor{library/test test support is_resource_enabled}@anchor{395b}
@deffn {Function} test.support.is_resource_enabled (resource)

Return @code{True} if @emph{resource} is enabled and available. The list of
available resources is only set when @code{test.regrtest} is executing the
tests.
@end deffn

@geindex python_is_optimized() (in module test.support)
@anchor{library/test test support python_is_optimized}@anchor{395c}
@deffn {Function} test.support.python_is_optimized ()

Return @code{True} if Python was not built with @code{-O0} or @code{-Og}.
@end deffn

@geindex with_pymalloc() (in module test.support)
@anchor{library/test test support with_pymalloc}@anchor{395d}
@deffn {Function} test.support.with_pymalloc ()

Return @code{_testcapi.WITH_PYMALLOC}.
@end deffn

@geindex requires() (in module test.support)
@anchor{library/test test support requires}@anchor{3944}
@deffn {Function} test.support.requires (resource, msg=None)

Raise @ref{3943,,ResourceDenied} if @emph{resource} is not available. @emph{msg} is the
argument to @ref{3943,,ResourceDenied} if it is raised. Always returns
@code{True} if called by a function whose @code{__name__} is @code{'__main__'}.
Used when tests are executed by @code{test.regrtest}.
@end deffn

@geindex system_must_validate_cert() (in module test.support)
@anchor{library/test test support system_must_validate_cert}@anchor{395e}
@deffn {Function} test.support.system_must_validate_cert (f)

Raise @ref{b6a,,unittest.SkipTest} on TLS certification validation failures.
@end deffn

@geindex sortdict() (in module test.support)
@anchor{library/test test support sortdict}@anchor{395f}
@deffn {Function} test.support.sortdict (dict)

Return a repr of @emph{dict} with keys sorted.
@end deffn

@geindex findfile() (in module test.support)
@anchor{library/test test support findfile}@anchor{3960}
@deffn {Function} test.support.findfile (filename, subdir=None)

Return the path to the file named @emph{filename}. If no match is found
@emph{filename} is returned. This does not equal a failure since it could be the
path to the file.

Setting @emph{subdir} indicates a relative path to use to find the file
rather than looking directly in the path directories.
@end deffn

@geindex match_test() (in module test.support)
@anchor{library/test test support match_test}@anchor{3961}
@deffn {Function} test.support.match_test (test)

Match @emph{test} to patterns set in @ref{3962,,set_match_tests()}.
@end deffn

@geindex set_match_tests() (in module test.support)
@anchor{library/test test support set_match_tests}@anchor{3962}
@deffn {Function} test.support.set_match_tests (patterns)

Define match test with regular expression @emph{patterns}.
@end deffn

@geindex run_unittest() (in module test.support)
@anchor{library/test test support run_unittest}@anchor{1306}
@deffn {Function} test.support.run_unittest (*classes)

Execute @ref{1ab,,unittest.TestCase} subclasses passed to the function. The
function scans the classes for methods starting with the prefix @code{test_}
and executes the tests individually.

It is also legal to pass strings as parameters; these should be keys in
@code{sys.modules}. Each associated module will be scanned by
@code{unittest.TestLoader.loadTestsFromModule()}. This is usually seen in the
following @code{test_main()} function:

@example
def test_main():
    support.run_unittest(__name__)
@end example

This will run all tests defined in the named module.
@end deffn

@geindex run_doctest() (in module test.support)
@anchor{library/test test support run_doctest}@anchor{3963}
@deffn {Function} test.support.run_doctest (module, verbosity=None, optionflags=0)

Run @ref{fc9,,doctest.testmod()} on the given @emph{module}.  Return
@code{(failure_count, test_count)}.

If @emph{verbosity} is @code{None}, @ref{fc9,,doctest.testmod()} is run with verbosity
set to @ref{3945,,verbose}.  Otherwise, it is run with verbosity set to
@code{None}.  @emph{optionflags} is passed as @code{optionflags} to
@ref{fc9,,doctest.testmod()}.
@end deffn

@geindex setswitchinterval() (in module test.support)
@anchor{library/test test support setswitchinterval}@anchor{3964}
@deffn {Function} test.support.setswitchinterval (interval)

Set the @ref{449,,sys.setswitchinterval()} to the given @emph{interval}.  Defines
a minimum interval for Android systems to prevent the system from hanging.
@end deffn

@geindex check_impl_detail() (in module test.support)
@anchor{library/test test support check_impl_detail}@anchor{3965}
@deffn {Function} test.support.check_impl_detail (**guards)

Use this check to guard CPython’s implementation-specific tests or to
run them only on the implementations guarded by the arguments:

@example
check_impl_detail()               # Only on CPython (default).
check_impl_detail(jython=True)    # Only on Jython.
check_impl_detail(cpython=False)  # Everywhere except CPython.
@end example
@end deffn

@geindex set_memlimit() (in module test.support)
@anchor{library/test test support set_memlimit}@anchor{3952}
@deffn {Function} test.support.set_memlimit (limit)

Set the values for @ref{3951,,max_memuse} and @ref{3953,,real_max_memuse} for big
memory tests.
@end deffn

@geindex record_original_stdout() (in module test.support)
@anchor{library/test test support record_original_stdout}@anchor{3966}
@deffn {Function} test.support.record_original_stdout (stdout)

Store the value from @emph{stdout}.  It is meant to hold the stdout at the
time the regrtest began.
@end deffn

@geindex get_original_stdout() (in module test.support)
@anchor{library/test test support get_original_stdout}@anchor{3967}
@deffn {Function} test.support.get_original_stdout ()

Return the original stdout set by @ref{3966,,record_original_stdout()} or
@code{sys.stdout} if it’s not set.
@end deffn

@geindex args_from_interpreter_flags() (in module test.support)
@anchor{library/test test support args_from_interpreter_flags}@anchor{3968}
@deffn {Function} test.support.args_from_interpreter_flags ()

Return a list of command line arguments reproducing the current settings
in @code{sys.flags} and @code{sys.warnoptions}.
@end deffn

@geindex optim_args_from_interpreter_flags() (in module test.support)
@anchor{library/test test support optim_args_from_interpreter_flags}@anchor{3969}
@deffn {Function} test.support.optim_args_from_interpreter_flags ()

Return a list of command line arguments reproducing the current
optimization settings in @code{sys.flags}.
@end deffn

@geindex captured_stdin() (in module test.support)
@geindex captured_stdout() (in module test.support)
@geindex captured_stderr() (in module test.support)
@anchor{library/test test support captured_stdin}@anchor{396a}
@deffn {Function} test.support.captured_stdin ()
@anchor{library/test test support captured_stdout}@anchor{396b}
@deffnx {Function} test.support.captured_stdout ()
@anchor{library/test test support captured_stderr}@anchor{12a0}
@deffnx {Function} test.support.captured_stderr ()

A context managers that temporarily replaces the named stream with
@ref{aa1,,io.StringIO} object.

Example use with output streams:

@example
with captured_stdout() as stdout, captured_stderr() as stderr:
    print("hello")
    print("error", file=sys.stderr)
assert stdout.getvalue() == "hello\n"
assert stderr.getvalue() == "error\n"
@end example

Example use with input stream:

@example
with captured_stdin() as stdin:
    stdin.write('hello\n')
    stdin.seek(0)
    # call test code that consumes from sys.stdin
    captured = input()
self.assertEqual(captured, "hello")
@end example
@end deffn

@geindex disable_faulthandler() (in module test.support)
@anchor{library/test test support disable_faulthandler}@anchor{396c}
@deffn {Function} test.support.disable_faulthandler ()

A context manager that replaces @code{sys.stderr} with @code{sys.__stderr__}.
@end deffn

@geindex gc_collect() (in module test.support)
@anchor{library/test test support gc_collect}@anchor{396d}
@deffn {Function} test.support.gc_collect ()

Force as many objects as possible to be collected.  This is needed because
timely deallocation is not guaranteed by the garbage collector.  This means
that @code{__del__} methods may be called later than expected and weakrefs
may remain alive for longer than expected.
@end deffn

@geindex disable_gc() (in module test.support)
@anchor{library/test test support disable_gc}@anchor{396e}
@deffn {Function} test.support.disable_gc ()

A context manager that disables the garbage collector upon entry and
reenables it upon exit.
@end deffn

@geindex swap_attr() (in module test.support)
@anchor{library/test test support swap_attr}@anchor{396f}
@deffn {Function} test.support.swap_attr (obj, attr, new_val)

Context manager to swap out an attribute with a new object.

Usage:

@example
with swap_attr(obj, "attr", 5):
    ...
@end example

This will set @code{obj.attr} to 5 for the duration of the @code{with} block,
restoring the old value at the end of the block.  If @code{attr} doesn’t
exist on @code{obj}, it will be created and then deleted at the end of the
block.

The old value (or @code{None} if it doesn’t exist) will be assigned to the
target of the “as” clause, if there is one.
@end deffn

@geindex swap_item() (in module test.support)
@anchor{library/test test support swap_item}@anchor{3970}
@deffn {Function} test.support.swap_item (obj, attr, new_val)

Context manager to swap out an item with a new object.

Usage:

@example
with swap_item(obj, "item", 5):
    ...
@end example

This will set @code{obj["item"]} to 5 for the duration of the @code{with} block,
restoring the old value at the end of the block. If @code{item} doesn’t
exist on @code{obj}, it will be created and then deleted at the end of the
block.

The old value (or @code{None} if it doesn’t exist) will be assigned to the
target of the “as” clause, if there is one.
@end deffn

@geindex flush_std_streams() (in module test.support)
@anchor{library/test test support flush_std_streams}@anchor{3971}
@deffn {Function} test.support.flush_std_streams ()

Call the @code{flush()} method on @ref{5ef,,sys.stdout} and then on
@ref{42d,,sys.stderr}. It can be used to make sure that the logs order is
consistent before writing into stderr.


New in version 3.11.

@end deffn

@geindex print_warning() (in module test.support)
@anchor{library/test test support print_warning}@anchor{3972}
@deffn {Function} test.support.print_warning (msg)

Print a warning into @ref{3973,,sys.__stderr__}. Format the message as:
@code{f"Warning -- @{msg@}"}. If @emph{msg} is made of multiple lines, add
@code{"Warning -- "} prefix to each line.


New in version 3.9.

@end deffn

@geindex wait_process() (in module test.support)
@anchor{library/test test support wait_process}@anchor{1202}
@deffn {Function} test.support.wait_process (pid, *, exitcode, timeout=None)

Wait until process @emph{pid} completes and check that the process exit code is
@emph{exitcode}.

Raise an @ref{46e,,AssertionError} if the process exit code is not equal to
@emph{exitcode}.

If the process runs longer than @emph{timeout} seconds (@ref{1120,,SHORT_TIMEOUT} by
default), kill the process and raise an @ref{46e,,AssertionError}. The timeout
feature is not available on Windows.


New in version 3.9.

@end deffn

@geindex calcobjsize() (in module test.support)
@anchor{library/test test support calcobjsize}@anchor{3974}
@deffn {Function} test.support.calcobjsize (fmt)

Return @ref{125e,,struct.calcsize()} for @code{nP@{fmt@}0n} or, if @code{gettotalrefcount}
exists, @code{2PnP@{fmt@}0P}.
@end deffn

@geindex calcvobjsize() (in module test.support)
@anchor{library/test test support calcvobjsize}@anchor{3975}
@deffn {Function} test.support.calcvobjsize (fmt)

Return @ref{125e,,struct.calcsize()} for @code{nPn@{fmt@}0n} or, if @code{gettotalrefcount}
exists, @code{2PnPn@{fmt@}0P}.
@end deffn

@geindex checksizeof() (in module test.support)
@anchor{library/test test support checksizeof}@anchor{3976}
@deffn {Function} test.support.checksizeof (test, o, size)

For testcase @emph{test}, assert that the @code{sys.getsizeof} for @emph{o} plus the GC
header size equals @emph{size}.
@end deffn

@geindex anticipate_failure() (in module test.support)
@anchor{library/test test support anticipate_failure}@anchor{3977}
@deffn {Function} @@test.support.anticipate_failure (condition)

A decorator to conditionally mark tests with
@ref{3825,,unittest.expectedFailure()}. Any use of this decorator should
have an associated comment identifying the relevant tracker issue.
@end deffn

@geindex run_with_locale() (in module test.support)
@anchor{library/test test support run_with_locale}@anchor{3978}
@deffn {Function} @@test.support.run_with_locale (catstr, *locales)

A decorator for running a function in a different locale, correctly
resetting it after it has finished.  @emph{catstr} is the locale category as
a string (for example @code{"LC_ALL"}).  The @emph{locales} passed will be tried
sequentially, and the first valid locale will be used.
@end deffn

@geindex run_with_tz() (in module test.support)
@anchor{library/test test support run_with_tz}@anchor{3979}
@deffn {Function} @@test.support.run_with_tz (tz)

A decorator for running a function in a specific timezone, correctly
resetting it after it has finished.
@end deffn

@geindex requires_freebsd_version() (in module test.support)
@anchor{library/test test support requires_freebsd_version}@anchor{397a}
@deffn {Function} @@test.support.requires_freebsd_version (*min_version)

Decorator for the minimum version when running test on FreeBSD.  If the
FreeBSD version is less than the minimum, raise @ref{b6a,,unittest.SkipTest}.
@end deffn

@geindex requires_linux_version() (in module test.support)
@anchor{library/test test support requires_linux_version}@anchor{397b}
@deffn {Function} @@test.support.requires_linux_version (*min_version)

Decorator for the minimum version when running test on Linux.  If the
Linux version is less than the minimum, raise @ref{b6a,,unittest.SkipTest}.
@end deffn

@geindex requires_mac_version() (in module test.support)
@anchor{library/test test support requires_mac_version}@anchor{397c}
@deffn {Function} @@test.support.requires_mac_version (*min_version)

Decorator for the minimum version when running test on macOS.  If the
macOS version is less than the minimum, raise @ref{b6a,,unittest.SkipTest}.
@end deffn

@geindex requires_IEEE_754() (in module test.support)
@anchor{library/test test support requires_IEEE_754}@anchor{397d}
@deffn {Function} @@test.support.requires_IEEE_754

Decorator for skipping tests on non-IEEE 754 platforms.
@end deffn

@geindex requires_zlib() (in module test.support)
@anchor{library/test test support requires_zlib}@anchor{397e}
@deffn {Function} @@test.support.requires_zlib

Decorator for skipping tests if @ref{14b,,zlib} doesn’t exist.
@end deffn

@geindex requires_gzip() (in module test.support)
@anchor{library/test test support requires_gzip}@anchor{397f}
@deffn {Function} @@test.support.requires_gzip

Decorator for skipping tests if @ref{87,,gzip} doesn’t exist.
@end deffn

@geindex requires_bz2() (in module test.support)
@anchor{library/test test support requires_bz2}@anchor{3980}
@deffn {Function} @@test.support.requires_bz2

Decorator for skipping tests if @ref{12,,bz2} doesn’t exist.
@end deffn

@geindex requires_lzma() (in module test.support)
@anchor{library/test test support requires_lzma}@anchor{3981}
@deffn {Function} @@test.support.requires_lzma

Decorator for skipping tests if @ref{a9,,lzma} doesn’t exist.
@end deffn

@geindex requires_resource() (in module test.support)
@anchor{library/test test support requires_resource}@anchor{3982}
@deffn {Function} @@test.support.requires_resource (resource)

Decorator for skipping tests if @emph{resource} is not available.
@end deffn

@geindex requires_docstrings() (in module test.support)
@anchor{library/test test support requires_docstrings}@anchor{3983}
@deffn {Function} @@test.support.requires_docstrings

Decorator for only running the test if @ref{3955,,HAVE_DOCSTRINGS}.
@end deffn

@geindex cpython_only() (in module test.support)
@anchor{library/test test support cpython_only}@anchor{3984}
@deffn {Function} @@test.support.cpython_only (test)

Decorator for tests only applicable to CPython.
@end deffn

@geindex impl_detail() (in module test.support)
@anchor{library/test test support impl_detail}@anchor{3985}
@deffn {Function} @@test.support.impl_detail (msg=None, **guards)

Decorator for invoking @ref{3965,,check_impl_detail()} on @emph{guards}.  If that
returns @code{False}, then uses @emph{msg} as the reason for skipping the test.
@end deffn

@geindex no_tracing() (in module test.support)
@anchor{library/test test support no_tracing}@anchor{3986}
@deffn {Function} @@test.support.no_tracing (func)

Decorator to temporarily turn off tracing for the duration of the test.
@end deffn

@geindex refcount_test() (in module test.support)
@anchor{library/test test support refcount_test}@anchor{3987}
@deffn {Function} @@test.support.refcount_test (test)

Decorator for tests which involve reference counting.  The decorator does
not run the test if it is not run by CPython.  Any trace function is unset
for the duration of the test to prevent unexpected refcounts caused by
the trace function.
@end deffn

@geindex bigmemtest() (in module test.support)
@anchor{library/test test support bigmemtest}@anchor{3988}
@deffn {Function} @@test.support.bigmemtest (size, memuse, dry_run=True)

Decorator for bigmem tests.

@emph{size} is a requested size for the test (in arbitrary, test-interpreted
units.)  @emph{memuse} is the number of bytes per unit for the test, or a good
estimate of it.  For example, a test that needs two byte buffers, of 4 GiB
each, could be decorated with @code{@@bigmemtest(size=_4G, memuse=2)}.

The @emph{size} argument is normally passed to the decorated test method as an
extra argument.  If @emph{dry_run} is @code{True}, the value passed to the test
method may be less than the requested value.  If @emph{dry_run} is @code{False}, it
means the test doesn’t support dummy runs when @code{-M} is not specified.
@end deffn

@geindex bigaddrspacetest() (in module test.support)
@anchor{library/test test support bigaddrspacetest}@anchor{3989}
@deffn {Function} @@test.support.bigaddrspacetest (f)

Decorator for tests that fill the address space.  @emph{f} is the function to
wrap.
@end deffn

@geindex check_syntax_error() (in module test.support)
@anchor{library/test test support check_syntax_error}@anchor{398a}
@deffn {Function} test.support.check_syntax_error (testcase, statement, errtext='@w{'}, *, lineno=None, offset=None)

Test for syntax errors in @emph{statement} by attempting to compile @emph{statement}.
@emph{testcase} is the @ref{122,,unittest} instance for the test.  @emph{errtext} is the
regular expression which should match the string representation of the
raised @ref{21a,,SyntaxError}.  If @emph{lineno} is not @code{None}, compares to
the line of the exception.  If @emph{offset} is not @code{None}, compares to
the offset of the exception.
@end deffn

@geindex open_urlresource() (in module test.support)
@anchor{library/test test support open_urlresource}@anchor{398b}
@deffn {Function} test.support.open_urlresource (url, *args, **kw)

Open @emph{url}.  If open fails, raises @ref{3942,,TestFailed}.
@end deffn

@geindex reap_children() (in module test.support)
@anchor{library/test test support reap_children}@anchor{398c}
@deffn {Function} test.support.reap_children ()

Use this at the end of @code{test_main} whenever sub-processes are started.
This will help ensure that no extra children (zombies) stick around to
hog resources and create problems when looking for refleaks.
@end deffn

@geindex get_attribute() (in module test.support)
@anchor{library/test test support get_attribute}@anchor{398d}
@deffn {Function} test.support.get_attribute (obj, name)

Get an attribute, raising @ref{b6a,,unittest.SkipTest} if @ref{163,,AttributeError}
is raised.
@end deffn

@geindex catch_unraisable_exception() (in module test.support)
@anchor{library/test test support catch_unraisable_exception}@anchor{127a}
@deffn {Function} test.support.catch_unraisable_exception ()

Context manager catching unraisable exception using
@ref{18d,,sys.unraisablehook()}.

Storing the exception value (@code{cm.unraisable.exc_value}) creates a
reference cycle. The reference cycle is broken explicitly when the context
manager exits.

Storing the object (@code{cm.unraisable.object}) can resurrect it if it is set
to an object which is being finalized. Exiting the context manager clears
the stored object.

Usage:

@example
with support.catch_unraisable_exception() as cm:
    # code creating an "unraisable exception"
    ...

    # check the unraisable exception: use cm.unraisable
    ...

# cm.unraisable attribute no longer exists at this point
# (to break a reference cycle)
@end example


New in version 3.8.

@end deffn

@geindex load_package_tests() (in module test.support)
@anchor{library/test test support load_package_tests}@anchor{398e}
@deffn {Function} test.support.load_package_tests (pkg_dir, loader, standard_tests, pattern)

Generic implementation of the @ref{122,,unittest} @code{load_tests} protocol for
use in test packages.  @emph{pkg_dir} is the root directory of the package;
@emph{loader}, @emph{standard_tests}, and @emph{pattern} are the arguments expected by
@code{load_tests}.  In simple cases, the test package’s @code{__init__.py}
can be the following:

@example
import os
from test.support import load_package_tests

def load_tests(*args):
    return load_package_tests(os.path.dirname(__file__), *args)
@end example
@end deffn

@geindex detect_api_mismatch() (in module test.support)
@anchor{library/test test support detect_api_mismatch}@anchor{398f}
@deffn {Function} test.support.detect_api_mismatch (ref_api, other_api, *, ignore=())

Returns the set of attributes, functions or methods of @emph{ref_api} not
found on @emph{other_api}, except for a defined list of items to be
ignored in this check specified in @emph{ignore}.

By default this skips private attributes beginning with ‘_’ but
includes all magic methods, i.e. those starting and ending in ‘__’.


New in version 3.5.

@end deffn

@geindex patch() (in module test.support)
@anchor{library/test test support patch}@anchor{3990}
@deffn {Function} test.support.patch (test_instance, object_to_patch, attr_name, new_value)

Override @emph{object_to_patch.attr_name} with @emph{new_value}.  Also add
cleanup procedure to @emph{test_instance} to restore @emph{object_to_patch} for
@emph{attr_name}.  The @emph{attr_name} should be a valid attribute for
@emph{object_to_patch}.
@end deffn

@geindex run_in_subinterp() (in module test.support)
@anchor{library/test test support run_in_subinterp}@anchor{3991}
@deffn {Function} test.support.run_in_subinterp (code)

Run @emph{code} in subinterpreter.  Raise @ref{b6a,,unittest.SkipTest} if
@ref{11b,,tracemalloc} is enabled.
@end deffn

@geindex check_free_after_iterating() (in module test.support)
@anchor{library/test test support check_free_after_iterating}@anchor{3992}
@deffn {Function} test.support.check_free_after_iterating (test, iter, cls, args=())

Assert that @emph{iter} is deallocated after iterating.
@end deffn

@geindex missing_compiler_executable() (in module test.support)
@anchor{library/test test support missing_compiler_executable}@anchor{3993}
@deffn {Function} test.support.missing_compiler_executable (cmd_names=[])

Check for the existence of the compiler executables whose names are listed
in @emph{cmd_names} or all the compiler executables when @emph{cmd_names} is empty
and return the first missing executable or @code{None} when none is found
missing.
@end deffn

@geindex check__all__() (in module test.support)
@anchor{library/test test support check__all__}@anchor{11d1}
@deffn {Function} test.support.check__all__ (test_case, module, name_of_module=None, extra=(), not_exported=())

Assert that the @code{__all__} variable of @emph{module} contains all public names.

The module’s public names (its API) are detected automatically
based on whether they match the public name convention and were defined in
@emph{module}.

The @emph{name_of_module} argument can specify (as a string or tuple thereof) what
module(s) an API could be defined in order to be detected as a public
API. One case for this is when @emph{module} imports part of its public API from
other modules, possibly a C backend (like @code{csv} and its @code{_csv}).

The @emph{extra} argument can be a set of names that wouldn’t otherwise be automatically
detected as “public”, like objects without a proper @code{__module__}
attribute. If provided, it will be added to the automatically detected ones.

The @emph{not_exported} argument can be a set of names that must not be treated
as part of the public API even though their names indicate otherwise.

Example use:

@example
import bar
import foo
import unittest
from test import support

class MiscTestCase(unittest.TestCase):
    def test__all__(self):
        support.check__all__(self, foo)

class OtherTestCase(unittest.TestCase):
    def test__all__(self):
        extra = @{'BAR_CONST', 'FOO_CONST'@}
        not_exported = @{'baz'@}  # Undocumented name.
        # bar imports part of its API from _bar.
        support.check__all__(self, bar, ('bar', '_bar'),
                             extra=extra, not_exported=not_exported)
@end example


New in version 3.6.

@end deffn

@geindex skip_if_broken_multiprocessing_synchronize() (in module test.support)
@anchor{library/test test support skip_if_broken_multiprocessing_synchronize}@anchor{11d2}
@deffn {Function} test.support.skip_if_broken_multiprocessing_synchronize ()

Skip tests if the @code{multiprocessing.synchronize} module is missing, if
there is no available semaphore implementation, or if creating a lock raises
an @ref{2a1,,OSError}.


New in version 3.10.

@end deffn

@geindex check_disallow_instantiation() (in module test.support)
@anchor{library/test test support check_disallow_instantiation}@anchor{3994}
@deffn {Function} test.support.check_disallow_instantiation (test_case, tp, *args, **kwds)

Assert that type @emph{tp} cannot be instantiated using @emph{args} and @emph{kwds}.


New in version 3.10.

@end deffn

The @ref{100,,test.support} module defines the following classes:

@geindex SuppressCrashReport (class in test.support)
@anchor{library/test test support SuppressCrashReport}@anchor{3995}
@deffn {Class} test.support.SuppressCrashReport

A context manager used to try to prevent crash dialog popups on tests that
are expected to crash a subprocess.

On Windows, it disables Windows Error Reporting dialogs using
SetErrorMode@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/ms680621.aspx}.

On UNIX, @ref{3996,,resource.setrlimit()} is used to set
@ref{3997,,resource.RLIMIT_CORE}’s soft limit to 0 to prevent coredump file
creation.

On both platforms, the old value is restored by @code{__exit__()}.
@end deffn

@geindex SaveSignals (class in test.support)
@anchor{library/test test support SaveSignals}@anchor{3998}
@deffn {Class} test.support.SaveSignals

Class to save and restore signal handlers registered by the Python signal
handler.
@end deffn

@geindex Matcher (class in test.support)
@anchor{library/test test support Matcher}@anchor{3999}
@deffn {Class} test.support.Matcher

@geindex matches() (test.support.Matcher method)
@anchor{library/test test support Matcher matches}@anchor{399a}
@deffn {Method} matches (self, d, **kwargs)

Try to match a single dict with the supplied arguments.
@end deffn

@geindex match_value() (test.support.Matcher method)
@anchor{library/test test support Matcher match_value}@anchor{399b}
@deffn {Method} match_value (self, k, dv, v)

Try to match a single stored value (@emph{dv}) with a supplied value (@emph{v}).
@end deffn
@end deffn

@geindex BasicTestRunner (class in test.support)
@anchor{library/test test support BasicTestRunner}@anchor{399c}
@deffn {Class} test.support.BasicTestRunner

@geindex run() (test.support.BasicTestRunner method)
@anchor{library/test test support BasicTestRunner run}@anchor{399d}
@deffn {Method} run (test)

Run @emph{test} and return the result.
@end deffn
@end deffn

@node test support socket_helper — Utilities for socket tests,test support script_helper — Utilities for the Python execution tests,test support — Utilities for the Python test suite,Development Tools
@anchor{library/test module-test support socket_helper}@anchor{105}@anchor{library/test test-support-socket-helper-utilities-for-socket-tests}@anchor{399e}
@subsection @code{test.support.socket_helper} — Utilities for socket tests


@geindex module; test.support.socket_helper

The @ref{105,,test.support.socket_helper} module provides support for socket tests.


New in version 3.9.


@geindex IPV6_ENABLED (in module test.support.socket_helper)
@anchor{library/test test support socket_helper IPV6_ENABLED}@anchor{399f}
@deffn {Data} test.support.socket_helper.IPV6_ENABLED

Set to @code{True} if IPv6 is enabled on this host, @code{False} otherwise.
@end deffn

@geindex find_unused_port() (in module test.support.socket_helper)
@anchor{library/test test support socket_helper find_unused_port}@anchor{39a0}
@deffn {Function} test.support.socket_helper.find_unused_port (family=socket.AF_INET, socktype=socket.SOCK_STREAM)

Returns an unused port that should be suitable for binding.  This is
achieved by creating a temporary socket with the same family and type as
the @code{sock} parameter (default is @ref{2a34,,AF_INET},
@ref{e91,,SOCK_STREAM}),
and binding it to the specified host address (defaults to @code{0.0.0.0})
with the port set to 0, eliciting an unused ephemeral port from the OS.
The temporary socket is then closed and deleted, and the ephemeral port is
returned.

Either this method or @ref{39a1,,bind_port()} should be used for any tests
where a server socket needs to be bound to a particular port for the
duration of the test.
Which one to use depends on whether the calling code is creating a Python
socket, or if an unused port needs to be provided in a constructor
or passed to an external program (i.e. the @code{-accept} argument to
openssl’s s_server mode).  Always prefer @ref{39a1,,bind_port()} over
@ref{39a0,,find_unused_port()} where possible.  Using a hard coded port is
discouraged since it can make multiple instances of the test impossible to
run simultaneously, which is a problem for buildbots.
@end deffn

@geindex bind_port() (in module test.support.socket_helper)
@anchor{library/test test support socket_helper bind_port}@anchor{39a1}
@deffn {Function} test.support.socket_helper.bind_port (sock, host=HOST)

Bind the socket to a free port and return the port number.  Relies on
ephemeral ports in order to ensure we are using an unbound port.  This is
important as many tests may be running simultaneously, especially in a
buildbot environment.  This method raises an exception if the
@code{sock.family} is @ref{2a34,,AF_INET} and @code{sock.type} is
@ref{e91,,SOCK_STREAM}, and the socket has
@code{SO_REUSEADDR} or @code{SO_REUSEPORT} set on it.
Tests should never set these socket options for TCP/IP sockets.
The only case for setting these options is testing multicasting via
multiple UDP sockets.

Additionally, if the @code{SO_EXCLUSIVEADDRUSE} socket option is
available (i.e. on Windows), it will be set on the socket.  This will
prevent anyone else from binding to our host/port for the duration of the
test.
@end deffn

@geindex bind_unix_socket() (in module test.support.socket_helper)
@anchor{library/test test support socket_helper bind_unix_socket}@anchor{39a2}
@deffn {Function} test.support.socket_helper.bind_unix_socket (sock, addr)

Bind a unix socket, raising @ref{b6a,,unittest.SkipTest} if
@ref{895,,PermissionError} is raised.
@end deffn

@geindex skip_unless_bind_unix_socket() (in module test.support.socket_helper)
@anchor{library/test test support socket_helper skip_unless_bind_unix_socket}@anchor{39a3}
@deffn {Function} @@test.support.socket_helper.skip_unless_bind_unix_socket

A decorator for running tests that require a functional @code{bind()} for Unix
sockets.
@end deffn

@geindex transient_internet() (in module test.support.socket_helper)
@anchor{library/test test support socket_helper transient_internet}@anchor{3949}
@deffn {Function} test.support.socket_helper.transient_internet (resource_name, *, timeout=30.0, errnos=())

A context manager that raises @ref{3943,,ResourceDenied} when
various issues with the internet connection manifest themselves as
exceptions.
@end deffn

@node test support script_helper — Utilities for the Python execution tests,test support bytecode_helper — Support tools for testing correct bytecode generation,test support socket_helper — Utilities for socket tests,Development Tools
@anchor{library/test module-test support script_helper}@anchor{104}@anchor{library/test test-support-script-helper-utilities-for-the-python-execution-tests}@anchor{39a4}
@subsection @code{test.support.script_helper} — Utilities for the Python execution tests


@geindex module; test.support.script_helper

The @ref{104,,test.support.script_helper} module provides support for Python’s
script execution tests.

@geindex interpreter_requires_environment() (in module test.support.script_helper)
@anchor{library/test test support script_helper interpreter_requires_environment}@anchor{39a5}
@deffn {Function} test.support.script_helper.interpreter_requires_environment ()

Return @code{True} if @code{sys.executable interpreter} requires environment
variables in order to be able to run at all.

This is designed to be used with @code{@@unittest.skipIf()} to annotate tests
that need to use an @code{assert_python*()} function to launch an isolated
mode (@code{-I}) or no environment mode (@code{-E}) sub-interpreter process.

A normal build & test does not run into this situation but it can happen
when trying to run the standard library test suite from an interpreter that
doesn’t have an obvious home with Python’s current home finding logic.

Setting 
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME} is one way to get most of the testsuite to run
in that situation.  
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} or 
@geindex PYTHONUSERSITE
@geindex environment variable; PYTHONUSERSITE
@code{PYTHONUSERSITE} are
other common environment variables that might impact whether or not the
interpreter can start.
@end deffn

@geindex run_python_until_end() (in module test.support.script_helper)
@anchor{library/test test support script_helper run_python_until_end}@anchor{39a6}
@deffn {Function} test.support.script_helper.run_python_until_end (*args, **env_vars)

Set up the environment based on @emph{env_vars} for running the interpreter
in a subprocess.  The values can include @code{__isolated}, @code{__cleanenv},
@code{__cwd}, and @code{TERM}.


Changed in version 3.9: The function no longer strips whitespaces from @emph{stderr}.

@end deffn

@geindex assert_python_ok() (in module test.support.script_helper)
@anchor{library/test test support script_helper assert_python_ok}@anchor{39a7}
@deffn {Function} test.support.script_helper.assert_python_ok (*args, **env_vars)

Assert that running the interpreter with @emph{args} and optional environment
variables @emph{env_vars} succeeds (@code{rc == 0}) and return a @code{(return code,
stdout, stderr)} tuple.

If the @code{__cleanenv} keyword is set, @emph{env_vars} is used as a fresh
environment.

Python is started in isolated mode (command line option @code{-I}),
except if the @code{__isolated} keyword is set to @code{False}.


Changed in version 3.9: The function no longer strips whitespaces from @emph{stderr}.

@end deffn

@geindex assert_python_failure() (in module test.support.script_helper)
@anchor{library/test test support script_helper assert_python_failure}@anchor{39a8}
@deffn {Function} test.support.script_helper.assert_python_failure (*args, **env_vars)

Assert that running the interpreter with @emph{args} and optional environment
variables @emph{env_vars} fails (@code{rc != 0}) and return a @code{(return code,
stdout, stderr)} tuple.

See @ref{39a7,,assert_python_ok()} for more options.


Changed in version 3.9: The function no longer strips whitespaces from @emph{stderr}.

@end deffn

@geindex spawn_python() (in module test.support.script_helper)
@anchor{library/test test support script_helper spawn_python}@anchor{39a9}
@deffn {Function} test.support.script_helper.spawn_python (*args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, **kw)

Run a Python subprocess with the given arguments.

@emph{kw} is extra keyword args to pass to @ref{5ae,,subprocess.Popen()}. Returns a
@ref{5ae,,subprocess.Popen} object.
@end deffn

@geindex kill_python() (in module test.support.script_helper)
@anchor{library/test test support script_helper kill_python}@anchor{39aa}
@deffn {Function} test.support.script_helper.kill_python (p)

Run the given @ref{5ae,,subprocess.Popen} process until completion and return
stdout.
@end deffn

@geindex make_script() (in module test.support.script_helper)
@anchor{library/test test support script_helper make_script}@anchor{39ab}
@deffn {Function} test.support.script_helper.make_script (script_dir, script_basename, source, omit_suffix=False)

Create script containing @emph{source} in path @emph{script_dir} and @emph{script_basename}.
If @emph{omit_suffix} is @code{False}, append @code{.py} to the name.  Return the full
script path.
@end deffn

@geindex make_zip_script() (in module test.support.script_helper)
@anchor{library/test test support script_helper make_zip_script}@anchor{39ac}
@deffn {Function} test.support.script_helper.make_zip_script (zip_dir, zip_basename, script_name, name_in_zip=None)

Create zip file at @emph{zip_dir} and @emph{zip_basename} with extension @code{zip} which
contains the files in @emph{script_name}. @emph{name_in_zip} is the archive name.
Return a tuple containing @code{(full path, full path of archive name)}.
@end deffn

@geindex make_pkg() (in module test.support.script_helper)
@anchor{library/test test support script_helper make_pkg}@anchor{39ad}
@deffn {Function} test.support.script_helper.make_pkg (pkg_dir, init_source='@w{'})

Create a directory named @emph{pkg_dir} containing an @code{__init__} file with
@emph{init_source} as its contents.
@end deffn

@geindex make_zip_pkg() (in module test.support.script_helper)
@anchor{library/test test support script_helper make_zip_pkg}@anchor{39ae}
@deffn {Function} test.support.script_helper.make_zip_pkg (zip_dir, zip_basename, pkg_name, script_basename, source, depth=1, compiled=False)

Create a zip package directory with a path of @emph{zip_dir} and @emph{zip_basename}
containing an empty @code{__init__} file and a file @emph{script_basename}
containing the @emph{source}.  If @emph{compiled} is @code{True}, both source files will
be compiled and added to the zip package.  Return a tuple of the full zip
path and the archive name for the zip file.
@end deffn

@node test support bytecode_helper — Support tools for testing correct bytecode generation,test support threading_helper — Utilities for threading tests,test support script_helper — Utilities for the Python execution tests,Development Tools
@anchor{library/test module-test support bytecode_helper}@anchor{101}@anchor{library/test test-support-bytecode-helper-support-tools-for-testing-correct-bytecode-generation}@anchor{39af}
@subsection @code{test.support.bytecode_helper} — Support tools for testing correct bytecode generation


@geindex module; test.support.bytecode_helper

The @ref{101,,test.support.bytecode_helper} module provides support for testing
and inspecting bytecode generation.


New in version 3.9.


The module defines the following class:

@geindex BytecodeTestCase (class in test.support.bytecode_helper)
@anchor{library/test test support bytecode_helper BytecodeTestCase}@anchor{39b0}
@deffn {Class} test.support.bytecode_helper.BytecodeTestCase (unittest.TestCase)

This class has custom assertion methods for inspecting bytecode.
@end deffn

@geindex get_disassembly_as_string() (test.support.bytecode_helper.BytecodeTestCase method)
@anchor{library/test test support bytecode_helper BytecodeTestCase get_disassembly_as_string}@anchor{39b1}
@deffn {Method} BytecodeTestCase.get_disassembly_as_string (co)

Return the disassembly of @emph{co} as string.
@end deffn

@geindex assertInBytecode() (test.support.bytecode_helper.BytecodeTestCase method)
@anchor{library/test test support bytecode_helper BytecodeTestCase assertInBytecode}@anchor{39b2}
@deffn {Method} BytecodeTestCase.assertInBytecode (x, opname, argval=_UNSPECIFIED)

Return instr if @emph{opname} is found, otherwise throws @ref{46e,,AssertionError}.
@end deffn

@geindex assertNotInBytecode() (test.support.bytecode_helper.BytecodeTestCase method)
@anchor{library/test test support bytecode_helper BytecodeTestCase assertNotInBytecode}@anchor{39b3}
@deffn {Method} BytecodeTestCase.assertNotInBytecode (x, opname, argval=_UNSPECIFIED)

Throws @ref{46e,,AssertionError} if @emph{opname} is found.
@end deffn

@node test support threading_helper — Utilities for threading tests,test support os_helper — Utilities for os tests,test support bytecode_helper — Support tools for testing correct bytecode generation,Development Tools
@anchor{library/test module-test support threading_helper}@anchor{106}@anchor{library/test test-support-threading-helper-utilities-for-threading-tests}@anchor{39b4}
@subsection @code{test.support.threading_helper} — Utilities for threading tests


@geindex module; test.support.threading_helper

The @ref{106,,test.support.threading_helper} module provides support for threading tests.


New in version 3.10.


@geindex join_thread() (in module test.support.threading_helper)
@anchor{library/test test support threading_helper join_thread}@anchor{39b5}
@deffn {Function} test.support.threading_helper.join_thread (thread, timeout=None)

Join a @emph{thread} within @emph{timeout}.  Raise an @ref{46e,,AssertionError} if thread
is still alive after @emph{timeout} seconds.
@end deffn

@geindex reap_threads() (in module test.support.threading_helper)
@anchor{library/test test support threading_helper reap_threads}@anchor{39b6}
@deffn {Function} @@test.support.threading_helper.reap_threads (func)

Decorator to ensure the threads are cleaned up even if the test fails.
@end deffn

@geindex start_threads() (in module test.support.threading_helper)
@anchor{library/test test support threading_helper start_threads}@anchor{39b7}
@deffn {Function} test.support.threading_helper.start_threads (threads, unlock=None)

Context manager to start @emph{threads}.  It attempts to join the threads upon
exit.
@end deffn

@geindex threading_cleanup() (in module test.support.threading_helper)
@anchor{library/test test support threading_helper threading_cleanup}@anchor{39b8}
@deffn {Function} test.support.threading_helper.threading_cleanup (*original_values)

Cleanup up threads not specified in @emph{original_values}.  Designed to emit
a warning if a test leaves running threads in the background.
@end deffn

@geindex threading_setup() (in module test.support.threading_helper)
@anchor{library/test test support threading_helper threading_setup}@anchor{39b9}
@deffn {Function} test.support.threading_helper.threading_setup ()

Return current thread count and copy of dangling threads.
@end deffn

@geindex wait_threads_exit() (in module test.support.threading_helper)
@anchor{library/test test support threading_helper wait_threads_exit}@anchor{39ba}
@deffn {Function} test.support.threading_helper.wait_threads_exit (timeout=None)

Context manager to wait until all threads created in the @code{with} statement
exit.
@end deffn

@geindex catch_threading_exception() (in module test.support.threading_helper)
@anchor{library/test test support threading_helper catch_threading_exception}@anchor{39bb}
@deffn {Function} test.support.threading_helper.catch_threading_exception ()

Context manager catching @ref{44b,,threading.Thread} exception using
@ref{2d1,,threading.excepthook()}.

Attributes set when an exception is caught:


@itemize *

@item 
@code{exc_type}

@item 
@code{exc_value}

@item 
@code{exc_traceback}

@item 
@code{thread}
@end itemize

See @ref{2d1,,threading.excepthook()} documentation.

These attributes are deleted at the context manager exit.

Usage:

@example
with threading_helper.catch_threading_exception() as cm:
    # code spawning a thread which raises an exception
    ...

    # check the thread exception, use cm attributes:
    # exc_type, exc_value, exc_traceback, thread
    ...

# exc_type, exc_value, exc_traceback, thread attributes of cm no longer
# exists at this point
# (to avoid reference cycles)
@end example


New in version 3.8.

@end deffn

@node test support os_helper — Utilities for os tests,test support import_helper — Utilities for import tests,test support threading_helper — Utilities for threading tests,Development Tools
@anchor{library/test module-test support os_helper}@anchor{103}@anchor{library/test test-support-os-helper-utilities-for-os-tests}@anchor{39bc}
@subsection @code{test.support.os_helper} — Utilities for os tests


@geindex module; test.support.os_helper

The @ref{103,,test.support.os_helper} module provides support for os tests.


New in version 3.10.


@geindex FS_NONASCII (in module test.support.os_helper)
@anchor{library/test test support os_helper FS_NONASCII}@anchor{39bd}
@deffn {Data} test.support.os_helper.FS_NONASCII

A non-ASCII character encodable by @ref{799,,os.fsencode()}.
@end deffn

@geindex SAVEDCWD (in module test.support.os_helper)
@anchor{library/test test support os_helper SAVEDCWD}@anchor{39be}
@deffn {Data} test.support.os_helper.SAVEDCWD

Set to @ref{12ac,,os.getcwd()}.
@end deffn

@geindex TESTFN (in module test.support.os_helper)
@anchor{library/test test support os_helper TESTFN}@anchor{39bf}
@deffn {Data} test.support.os_helper.TESTFN

Set to a name that is safe to use as the name of a temporary file.  Any
temporary file that is created should be closed and unlinked (removed).
@end deffn

@geindex TESTFN_NONASCII (in module test.support.os_helper)
@anchor{library/test test support os_helper TESTFN_NONASCII}@anchor{39c0}
@deffn {Data} test.support.os_helper.TESTFN_NONASCII

Set to a filename containing the @ref{39bd,,FS_NONASCII} character.
@end deffn

@geindex TESTFN_UNENCODABLE (in module test.support.os_helper)
@anchor{library/test test support os_helper TESTFN_UNENCODABLE}@anchor{39c1}
@deffn {Data} test.support.os_helper.TESTFN_UNENCODABLE

Set to a filename (str type) that should not be able to be encoded by file
system encoding in strict mode.  It may be @code{None} if it’s not possible to
generate such a filename.
@end deffn

@geindex TESTFN_UNDECODABLE (in module test.support.os_helper)
@anchor{library/test test support os_helper TESTFN_UNDECODABLE}@anchor{39c2}
@deffn {Data} test.support.os_helper.TESTFN_UNDECODABLE

Set to a filename (bytes type) that should not be able to be decoded by
file system encoding in strict mode.  It may be @code{None} if it’s not
possible to generate such a filename.
@end deffn

@geindex TESTFN_UNICODE (in module test.support.os_helper)
@anchor{library/test test support os_helper TESTFN_UNICODE}@anchor{39c3}
@deffn {Data} test.support.os_helper.TESTFN_UNICODE

Set to a non-ASCII name for a temporary file.
@end deffn

@geindex EnvironmentVarGuard (class in test.support.os_helper)
@anchor{library/test test support os_helper EnvironmentVarGuard}@anchor{39c4}
@deffn {Class} test.support.os_helper.EnvironmentVarGuard

Class used to temporarily set or unset environment variables.  Instances can
be used as a context manager and have a complete dictionary interface for
querying/modifying the underlying @code{os.environ}. After exit from the
context manager all changes to environment variables done through this
instance will be rolled back.


Changed in version 3.1: Added dictionary interface.

@end deffn

@geindex FakePath (class in test.support.os_helper)
@anchor{library/test test support os_helper FakePath}@anchor{39c5}
@deffn {Class} test.support.os_helper.FakePath (path)

Simple @ref{63d,,path-like object}.  It implements the @code{__fspath__()}
method which just returns the @emph{path} argument.  If @emph{path} is an exception,
it will be raised in @code{__fspath__()}.
@end deffn

@geindex set() (test.support.os_helper.EnvironmentVarGuard method)
@anchor{library/test test support os_helper EnvironmentVarGuard set}@anchor{39c6}
@deffn {Method} EnvironmentVarGuard.set (envvar, value)

Temporarily set the environment variable @code{envvar} to the value of
@code{value}.
@end deffn

@geindex unset() (test.support.os_helper.EnvironmentVarGuard method)
@anchor{library/test test support os_helper EnvironmentVarGuard unset}@anchor{39c7}
@deffn {Method} EnvironmentVarGuard.unset (envvar)

Temporarily unset the environment variable @code{envvar}.
@end deffn

@geindex can_symlink() (in module test.support.os_helper)
@anchor{library/test test support os_helper can_symlink}@anchor{39c8}
@deffn {Function} test.support.os_helper.can_symlink ()

Return @code{True} if the OS supports symbolic links, @code{False}
otherwise.
@end deffn

@geindex can_xattr() (in module test.support.os_helper)
@anchor{library/test test support os_helper can_xattr}@anchor{39c9}
@deffn {Function} test.support.os_helper.can_xattr ()

Return @code{True} if the OS supports xattr, @code{False}
otherwise.
@end deffn

@geindex change_cwd() (in module test.support.os_helper)
@anchor{library/test test support os_helper change_cwd}@anchor{39ca}
@deffn {Function} test.support.os_helper.change_cwd (path, quiet=False)

A context manager that temporarily changes the current working
directory to @emph{path} and yields the directory.

If @emph{quiet} is @code{False}, the context manager raises an exception
on error.  Otherwise, it issues only a warning and keeps the current
working directory the same.
@end deffn

@geindex create_empty_file() (in module test.support.os_helper)
@anchor{library/test test support os_helper create_empty_file}@anchor{39cb}
@deffn {Function} test.support.os_helper.create_empty_file (filename)

Create an empty file with @emph{filename}.  If it already exists, truncate it.
@end deffn

@geindex fd_count() (in module test.support.os_helper)
@anchor{library/test test support os_helper fd_count}@anchor{39cc}
@deffn {Function} test.support.os_helper.fd_count ()

Count the number of open file descriptors.
@end deffn

@geindex fs_is_case_insensitive() (in module test.support.os_helper)
@anchor{library/test test support os_helper fs_is_case_insensitive}@anchor{39cd}
@deffn {Function} test.support.os_helper.fs_is_case_insensitive (directory)

Return @code{True} if the file system for @emph{directory} is case-insensitive.
@end deffn

@geindex make_bad_fd() (in module test.support.os_helper)
@anchor{library/test test support os_helper make_bad_fd}@anchor{39ce}
@deffn {Function} test.support.os_helper.make_bad_fd ()

Create an invalid file descriptor by opening and closing a temporary file,
and returning its descriptor.
@end deffn

@geindex rmdir() (in module test.support.os_helper)
@anchor{library/test test support os_helper rmdir}@anchor{39cf}
@deffn {Function} test.support.os_helper.rmdir (filename)

Call @ref{c87,,os.rmdir()} on @emph{filename}.  On Windows platforms, this is
wrapped with a wait loop that checks for the existence of the file.
@end deffn

@geindex rmtree() (in module test.support.os_helper)
@anchor{library/test test support os_helper rmtree}@anchor{39d0}
@deffn {Function} test.support.os_helper.rmtree (path)

Call @ref{53a,,shutil.rmtree()} on @emph{path} or call @ref{517,,os.lstat()} and
@ref{c87,,os.rmdir()} to remove a path and its contents.  On Windows platforms,
this is wrapped with a wait loop that checks for the existence of the files.
@end deffn

@geindex skip_unless_symlink() (in module test.support.os_helper)
@anchor{library/test test support os_helper skip_unless_symlink}@anchor{39d1}
@deffn {Function} @@test.support.os_helper.skip_unless_symlink

A decorator for running tests that require support for symbolic links.
@end deffn

@geindex skip_unless_xattr() (in module test.support.os_helper)
@anchor{library/test test support os_helper skip_unless_xattr}@anchor{39d2}
@deffn {Function} @@test.support.os_helper.skip_unless_xattr

A decorator for running tests that require support for xattr.
@end deffn

@geindex temp_cwd() (in module test.support.os_helper)
@anchor{library/test test support os_helper temp_cwd}@anchor{39d3}
@deffn {Function} test.support.os_helper.temp_cwd (name='tempcwd', quiet=False)

A context manager that temporarily creates a new directory and
changes the current working directory (CWD).

The context manager creates a temporary directory in the current
directory with name @emph{name} before temporarily changing the current
working directory.  If @emph{name} is @code{None}, the temporary directory is
created using @ref{2103,,tempfile.mkdtemp()}.

If @emph{quiet} is @code{False} and it is not possible to create or change
the CWD, an error is raised.  Otherwise, only a warning is raised
and the original CWD is used.
@end deffn

@geindex temp_dir() (in module test.support.os_helper)
@anchor{library/test test support os_helper temp_dir}@anchor{39d4}
@deffn {Function} test.support.os_helper.temp_dir (path=None, quiet=False)

A context manager that creates a temporary directory at @emph{path} and
yields the directory.

If @emph{path} is @code{None}, the temporary directory is created using
@ref{2103,,tempfile.mkdtemp()}.  If @emph{quiet} is @code{False}, the context manager
raises an exception on error.  Otherwise, if @emph{path} is specified and
cannot be created, only a warning is issued.
@end deffn

@geindex temp_umask() (in module test.support.os_helper)
@anchor{library/test test support os_helper temp_umask}@anchor{39d5}
@deffn {Function} test.support.os_helper.temp_umask (umask)

A context manager that temporarily sets the process umask.
@end deffn

@geindex unlink() (in module test.support.os_helper)
@anchor{library/test test support os_helper unlink}@anchor{39d6}
@deffn {Function} test.support.os_helper.unlink (filename)

Call @ref{c89,,os.unlink()} on @emph{filename}.  On Windows platforms, this is
wrapped with a wait loop that checks for the existence of the file.
@end deffn

@node test support import_helper — Utilities for import tests,test support warnings_helper — Utilities for warnings tests,test support os_helper — Utilities for os tests,Development Tools
@anchor{library/test module-test support import_helper}@anchor{102}@anchor{library/test test-support-import-helper-utilities-for-import-tests}@anchor{39d7}
@subsection @code{test.support.import_helper} — Utilities for import tests


@geindex module; test.support.import_helper

The @ref{102,,test.support.import_helper} module provides support for import tests.


New in version 3.10.


@geindex forget() (in module test.support.import_helper)
@anchor{library/test test support import_helper forget}@anchor{39d8}
@deffn {Function} test.support.import_helper.forget (module_name)

Remove the module named @emph{module_name} from @code{sys.modules} and delete any
byte-compiled files of the module.
@end deffn

@geindex import_fresh_module() (in module test.support.import_helper)
@anchor{library/test test support import_helper import_fresh_module}@anchor{111f}
@deffn {Function} test.support.import_helper.import_fresh_module (name, fresh=(), blocked=(), deprecated=False)

This function imports and returns a fresh copy of the named Python module
by removing the named module from @code{sys.modules} before doing the import.
Note that unlike @code{reload()}, the original module is not affected by
this operation.

@emph{fresh} is an iterable of additional module names that are also removed
from the @code{sys.modules} cache before doing the import.

@emph{blocked} is an iterable of module names that are replaced with @code{None}
in the module cache during the import to ensure that attempts to import
them raise @ref{3b8,,ImportError}.

The named module and any modules named in the @emph{fresh} and @emph{blocked}
parameters are saved before starting the import and then reinserted into
@code{sys.modules} when the fresh import is complete.

Module and package deprecation messages are suppressed during this import
if @emph{deprecated} is @code{True}.

This function will raise @ref{3b8,,ImportError} if the named module cannot be
imported.

Example use:

@example
# Get copies of the warnings module for testing without affecting the
# version being used by the rest of the test suite. One copy uses the
# C implementation, the other is forced to use the pure Python fallback
# implementation
py_warnings = import_fresh_module('warnings', blocked=['_warnings'])
c_warnings = import_fresh_module('warnings', fresh=['_warnings'])
@end example


New in version 3.1.

@end deffn

@geindex import_module() (in module test.support.import_helper)
@anchor{library/test test support import_helper import_module}@anchor{39d9}
@deffn {Function} test.support.import_helper.import_module (name, deprecated=False, *, required_on())

This function imports and returns the named module. Unlike a normal
import, this function raises @ref{b6a,,unittest.SkipTest} if the module
cannot be imported.

Module and package deprecation messages are suppressed during this import
if @emph{deprecated} is @code{True}.  If a module is required on a platform but
optional for others, set @emph{required_on} to an iterable of platform prefixes
which will be compared against @ref{5a6,,sys.platform}.


New in version 3.1.

@end deffn

@geindex modules_setup() (in module test.support.import_helper)
@anchor{library/test test support import_helper modules_setup}@anchor{39da}
@deffn {Function} test.support.import_helper.modules_setup ()

Return a copy of @ref{11f7,,sys.modules}.
@end deffn

@geindex modules_cleanup() (in module test.support.import_helper)
@anchor{library/test test support import_helper modules_cleanup}@anchor{39db}
@deffn {Function} test.support.import_helper.modules_cleanup (oldmodules)

Remove modules except for @emph{oldmodules} and @code{encodings} in order to
preserve internal cache.
@end deffn

@geindex unload() (in module test.support.import_helper)
@anchor{library/test test support import_helper unload}@anchor{39dc}
@deffn {Function} test.support.import_helper.unload (name)

Delete @emph{name} from @code{sys.modules}.
@end deffn

@geindex make_legacy_pyc() (in module test.support.import_helper)
@anchor{library/test test support import_helper make_legacy_pyc}@anchor{39dd}
@deffn {Function} test.support.import_helper.make_legacy_pyc (source)

Move a 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147}/
@geindex Python Enhancement Proposals; PEP 488
PEP 488@footnote{https://www.python.org/dev/peps/pep-0488} pyc file to its legacy pyc location and return the file
system path to the legacy pyc file.  The @emph{source} value is the file system
path to the source file.  It does not need to exist, however the PEP
3147/488 pyc file must exist.
@end deffn

@geindex CleanImport (class in test.support.import_helper)
@anchor{library/test test support import_helper CleanImport}@anchor{39de}
@deffn {Class} test.support.import_helper.CleanImport (*module_names)

A context manager to force import to return a new module reference.  This
is useful for testing module-level behaviors, such as the emission of a
DeprecationWarning on import.  Example usage:

@example
with CleanImport('foo'):
    importlib.import_module('foo')  # New reference.
@end example
@end deffn

@geindex DirsOnSysPath (class in test.support.import_helper)
@anchor{library/test test support import_helper DirsOnSysPath}@anchor{39df}
@deffn {Class} test.support.import_helper.DirsOnSysPath (*paths)

A context manager to temporarily add directories to sys.path.

This makes a copy of @ref{489,,sys.path}, appends any directories given
as positional arguments, then reverts @ref{489,,sys.path} to the copied
settings when the context ends.

Note that @emph{all} @ref{489,,sys.path} modifications in the body of the
context manager, including replacement of the object,
will be reverted at the end of the block.
@end deffn

@node test support warnings_helper — Utilities for warnings tests,,test support import_helper — Utilities for import tests,Development Tools
@anchor{library/test module-test support warnings_helper}@anchor{107}@anchor{library/test test-support-warnings-helper-utilities-for-warnings-tests}@anchor{39e0}
@subsection @code{test.support.warnings_helper} — Utilities for warnings tests


@geindex module; test.support.warnings_helper

The @ref{107,,test.support.warnings_helper} module provides support for warnings tests.


New in version 3.10.


@geindex check_no_resource_warning() (in module test.support.warnings_helper)
@anchor{library/test test support warnings_helper check_no_resource_warning}@anchor{39e1}
@deffn {Function} test.support.warnings_helper.check_no_resource_warning (testcase)

Context manager to check that no @ref{77b,,ResourceWarning} was raised.  You
must remove the object which may emit @ref{77b,,ResourceWarning} before the
end of the context manager.
@end deffn

@geindex check_syntax_warning() (in module test.support.warnings_helper)
@anchor{library/test test support warnings_helper check_syntax_warning}@anchor{39e2}
@deffn {Function} test.support.warnings_helper.check_syntax_warning (testcase, statement, errtext='@w{'}, *, lineno=1, offset=None)

Test for syntax warning in @emph{statement} by attempting to compile @emph{statement}.
Test also that the @ref{4c7,,SyntaxWarning} is emitted only once, and that it
will be converted to a @ref{21a,,SyntaxError} when turned into error.
@emph{testcase} is the @ref{122,,unittest} instance for the test.  @emph{errtext} is the
regular expression which should match the string representation of the
emitted @ref{4c7,,SyntaxWarning} and raised @ref{21a,,SyntaxError}.  If @emph{lineno}
is not @code{None}, compares to the line of the warning and exception.
If @emph{offset} is not @code{None}, compares to the offset of the exception.


New in version 3.8.

@end deffn

@geindex check_warnings() (in module test.support.warnings_helper)
@anchor{library/test test support warnings_helper check_warnings}@anchor{39e3}
@deffn {Function} test.support.warnings_helper.check_warnings (*filters, quiet=True)

A convenience wrapper for @ref{39e4,,warnings.catch_warnings()} that makes it
easier to test that a warning was correctly raised.  It is approximately
equivalent to calling @code{warnings.catch_warnings(record=True)} with
@ref{39e5,,warnings.simplefilter()} set to @code{always} and with the option to
automatically validate the results that are recorded.

@code{check_warnings} accepts 2-tuples of the form @code{("message regexp",
WarningCategory)} as positional arguments. If one or more @emph{filters} are
provided, or if the optional keyword argument @emph{quiet} is @code{False},
it checks to make sure the warnings are as expected:  each specified filter
must match at least one of the warnings raised by the enclosed code or the
test fails, and if any warnings are raised that do not match any of the
specified filters the test fails.  To disable the first of these checks,
set @emph{quiet} to @code{True}.

If no arguments are specified, it defaults to:

@example
check_warnings(("", Warning), quiet=True)
@end example

In this case all warnings are caught and no errors are raised.

On entry to the context manager, a @code{WarningRecorder} instance is
returned. The underlying warnings list from
@ref{39e4,,catch_warnings()} is available via the recorder object’s
@ref{12d,,warnings} attribute.  As a convenience, the attributes of the object
representing the most recent warning can also be accessed directly through
the recorder object (see example below).  If no warning has been raised,
then any of the attributes that would otherwise be expected on an object
representing a warning will return @code{None}.

The recorder object also has a @code{reset()} method, which clears the
warnings list.

The context manager is designed to be used like this:

@example
with check_warnings(("assertion is always true", SyntaxWarning),
                    ("", UserWarning)):
    exec('assert(False, "Hey!")')
    warnings.warn(UserWarning("Hide me!"))
@end example

In this case if either warning was not raised, or some other warning was
raised, @ref{39e3,,check_warnings()} would raise an error.

When a test needs to look more deeply into the warnings, rather than
just checking whether or not they occurred, code like this can be used:

@example
with check_warnings(quiet=True) as w:
    warnings.warn("foo")
    assert str(w.args[0]) == "foo"
    warnings.warn("bar")
    assert str(w.args[0]) == "bar"
    assert str(w.warnings[0].args[0]) == "foo"
    assert str(w.warnings[1].args[0]) == "bar"
    w.reset()
    assert len(w.warnings) == 0
@end example

Here all warnings will be caught, and the test code tests the captured
warnings directly.


Changed in version 3.2: New optional arguments @emph{filters} and @emph{quiet}.

@end deffn

@geindex WarningsRecorder (class in test.support.warnings_helper)
@anchor{library/test test support warnings_helper WarningsRecorder}@anchor{39e6}
@deffn {Class} test.support.warnings_helper.WarningsRecorder

Class used to record warnings for unit tests. See documentation of
@ref{39e3,,check_warnings()} above for more details.
@end deffn

@node Debugging and Profiling,Software Packaging and Distribution,Development Tools,The Python Standard Library
@anchor{library/debug doc}@anchor{39e7}@anchor{library/debug debugging-and-profiling}@anchor{39e8}
@section Debugging and Profiling


These libraries help you with Python development: the debugger enables you to
step through code, analyze stack frames and set breakpoints etc., and the
profilers run code and give you a detailed breakdown of execution times,
allowing you to identify bottlenecks in your programs. Auditing events
provide visibility into runtime behaviors that would otherwise require
intrusive debugging or patching.
@anchor{library/audit_events audit-events}@anchor{10cc}
@geindex audit events

@menu
* Audit events table:: 
* bdb — Debugger framework:: 
* faulthandler — Dump the Python traceback:: 
* pdb — The Python Debugger:: 
* The Python Profilers:: 
* timeit — Measure execution time of small code snippets:: 
* trace — Trace or track Python statement execution:: 
* tracemalloc — Trace memory allocations:: 

@end menu

@node Audit events table,bdb — Debugger framework,,Debugging and Profiling
@anchor{library/audit_events doc}@anchor{39e9}@anchor{library/audit_events audit-events-table}@anchor{39ea}
@subsection Audit events table


This table contains all events raised by @ref{11fc,,sys.audit()} or
@ref{120d,,PySys_Audit()} calls throughout the CPython runtime and the
standard library.  These calls were added in 3.8.0 or later (see 
@geindex Python Enhancement Proposals; PEP 578
PEP 578@footnote{https://www.python.org/dev/peps/pep-0578}).

See @ref{39eb,,sys.addaudithook()} and @ref{39ec,,PySys_AddAuditHook()} for
information on handling these events.

@strong{CPython implementation detail:} This table is generated from the CPython documentation, and may not
represent events raised by other implementations. See your runtime
specific documentation for actual events raised.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxx} 
@headitem

Audit event

@tab

Arguments

@tab

References

@item

array.__new__

@tab

@code{typecode}, @code{initializer}

@tab

@ref{25e,,[1]}

@item

builtins.breakpoint

@tab

@code{breakpointhook}

@tab

@ref{5db,,[1]}

@item

builtins.id

@tab

@code{id}

@tab

@ref{f80,,[1]}

@item

builtins.input

@tab

@code{prompt}

@tab

@ref{e73,,[1]}

@item

builtins.input/result

@tab

@code{result}

@tab

@ref{e73,,[1]}

@item

code.__new__

@tab

@code{code}, @code{filename}, @code{name}, @code{argcount}, @code{posonlyargcount}, @code{kwonlyargcount}, @code{nlocals}, @code{stacksize}, @code{flags}

@tab

@ref{4cb,,[1]}

@item

compile

@tab

@code{source}, @code{filename}

@tab

@ref{4e2,,[1]}

@item

cpython.PyInterpreterState_Clear

@tab



@tab

@ref{39ed,,[1]}

@item

cpython.PyInterpreterState_New

@tab



@tab

@ref{39ee,,[1]}

@item

cpython._PySys_ClearAuditHooks

@tab



@tab

@ref{862,,[1]}

@item

cpython.run_command

@tab

@code{command}

@tab

@ref{1180,,[1]}

@item

cpython.run_file

@tab

@code{filename}

@tab

@ref{39ef,,[1]}

@item

cpython.run_interactivehook

@tab

@code{hook}

@tab

@ref{b4c,,[1]}

@item

cpython.run_module

@tab

@code{module-name}

@tab

@ref{611,,[1]}

@item

cpython.run_startup

@tab

@code{filename}

@tab

@ref{b4d,,[1]}

@item

cpython.run_stdin

@tab



@tab

@ref{39f0,,[1]}

@item

ctypes.addressof

@tab

@code{obj}

@tab

@ref{27b5,,[1]}

@item

ctypes.call_function

@tab

@code{func_pointer}, @code{arguments}

@tab

@ref{27aa,,[1]}

@item

ctypes.cdata

@tab

@code{address}

@tab

@ref{27c6,,[1]}

@item

ctypes.cdata/buffer

@tab

@code{pointer}, @code{size}, @code{offset}

@tab

@ref{27c4,,[1]}@ref{27c5,,[2]}

@item

ctypes.create_string_buffer

@tab

@code{init}, @code{size}

@tab

@ref{2767,,[1]}

@item

ctypes.create_unicode_buffer

@tab

@code{init}, @code{size}

@tab

@ref{126f,,[1]}

@item

ctypes.dlopen

@tab

@code{name}

@tab

@ref{27a7,,[1]}

@item

ctypes.dlsym

@tab

@code{library}, @code{name}

@tab

@ref{27a7,,[1]}

@item

ctypes.dlsym/handle

@tab

@code{handle}, @code{name}

@tab

@ref{27a7,,[1]}

@item

ctypes.get_errno

@tab



@tab

@ref{27a1,,[1]}

@item

ctypes.get_last_error

@tab



@tab

@ref{27a3,,[1]}

@item

ctypes.seh_exception

@tab

@code{code}

@tab

@ref{27aa,,[1]}

@item

ctypes.set_errno

@tab

@code{errno}

@tab

@ref{27a2,,[1]}

@item

ctypes.set_last_error

@tab

@code{error}

@tab

@ref{27a4,,[1]}

@item

ctypes.string_at

@tab

@code{address}, @code{size}

@tab

@ref{27be,,[1]}

@item

ctypes.wstring_at

@tab

@code{address}, @code{size}

@tab

@ref{27c0,,[1]}

@item

ensurepip.bootstrap

@tab

@code{root}

@tab

@ref{39f1,,[1]}

@item

exec

@tab

@code{code_object}

@tab

@ref{34a,,[1]}@ref{34b,,[2]}

@item

fcntl.fcntl

@tab

@code{fd}, @code{cmd}, @code{arg}

@tab

@ref{2b20,,[1]}

@item

fcntl.flock

@tab

@code{fd}, @code{operation}

@tab

@ref{39f2,,[1]}

@item

fcntl.ioctl

@tab

@code{fd}, @code{request}, @code{arg}

@tab

@ref{fd7,,[1]}

@item

fcntl.lockf

@tab

@code{fd}, @code{cmd}, @code{len}, @code{start}, @code{whence}

@tab

@ref{f4d,,[1]}

@item

ftplib.connect

@tab

@code{self}, @code{host}, @code{port}

@tab

@ref{31a8,,[1]}

@item

ftplib.sendcmd

@tab

@code{self}, @code{cmd}

@tab

@ref{31ac,,[1]}@ref{31ad,,[2]}

@item

function.__new__

@tab

@code{code}

@tab

@ref{349,,[1]}

@item

gc.get_objects

@tab

@code{generation}

@tab

@ref{284,,[1]}

@item

gc.get_referents

@tab

@code{objs}

@tab

@ref{286,,[1]}

@item

gc.get_referrers

@tab

@code{objs}

@tab

@ref{285,,[1]}

@item

glob.glob

@tab

@code{pathname}, @code{recursive}

@tab

@ref{288,,[1]}@ref{289,,[2]}

@item

glob.glob/2

@tab

@code{pathname}, @code{recursive}, @code{root_dir}, @code{dir_fd}

@tab

@ref{288,,[1]}@ref{289,,[2]}

@item

http.client.connect

@tab

@code{self}, @code{host}, @code{port}

@tab

@ref{3185,,[1]}

@item

http.client.send

@tab

@code{self}, @code{data}

@tab

@ref{318a,,[1]}

@item

imaplib.open

@tab

@code{self}, @code{host}, @code{port}

@tab

@ref{3f4,,[1]}

@item

imaplib.send

@tab

@code{self}, @code{data}

@tab

@ref{31f7,,[1]}

@item

import

@tab

@code{module}, @code{filename}, @code{sys.path}, @code{sys.meta_path}, @code{sys.path_hooks}

@tab

@ref{e35,,[1]}

@item

marshal.dumps

@tab

@code{value}, @code{version}

@tab

@ref{217a,,[1]}

@item

marshal.load

@tab



@tab

@ref{12be,,[1]}

@item

marshal.loads

@tab

@code{bytes}

@tab

@ref{12be,,[1]}

@item

mmap.__new__

@tab

@code{fileno}, @code{length}, @code{access}, @code{offset}

@tab

@ref{511,,[1]}

@item

msvcrt.get_osfhandle

@tab

@code{fd}

@tab

@ref{39f3,,[1]}

@item

msvcrt.locking

@tab

@code{fd}, @code{mode}, @code{nbytes}

@tab

@ref{39f4,,[1]}

@item

msvcrt.open_osfhandle

@tab

@code{handle}, @code{flags}

@tab

@ref{39f5,,[1]}

@item

nntplib.connect

@tab

@code{self}, @code{host}, @code{port}

@tab

@ref{409,,[1]}@ref{40a,,[2]}

@item

nntplib.putline

@tab

@code{self}, @code{line}

@tab

@ref{409,,[1]}@ref{40a,,[2]}

@item

object.__delattr__

@tab

@code{obj}, @code{name}

@tab

@ref{1799,,[1]}

@item

object.__getattr__

@tab

@code{obj}, @code{name}

@tab

@ref{70a,,[1]}

@item

object.__setattr__

@tab

@code{obj}, @code{name}, @code{value}

@tab

@ref{1798,,[1]}

@item

open

@tab

@code{path}, @code{mode}, @code{flags}

@tab

@ref{1c8,,[1]}@ref{1c9,,[2]}@ref{8ee,,[3]}

@item

os.add_dll_directory

@tab

@code{path}

@tab

@ref{4ee,,[1]}

@item

os.chdir

@tab

@code{path}

@tab

@ref{3b9,,[1]}@ref{8e4,,[2]}

@item

os.chflags

@tab

@code{path}, @code{flags}

@tab

@ref{c7f,,[1]}@ref{23e7,,[2]}

@item

os.chmod

@tab

@code{path}, @code{mode}, @code{dir_fd}

@tab

@ref{c80,,[1]}@ref{8e5,,[2]}@ref{12f0,,[3]}

@item

os.chown

@tab

@code{path}, @code{uid}, @code{gid}, @code{dir_fd}

@tab

@ref{c81,,[1]}@ref{8e6,,[2]}@ref{23e8,,[3]}

@item

os.exec

@tab

@code{path}, @code{args}, @code{env}

@tab

@ref{2429,,[1]}

@item

os.fork

@tab



@tab

@ref{bb9,,[1]}

@item

os.forkpty

@tab



@tab

@ref{2441,,[1]}

@item

os.fwalk

@tab

@code{top}, @code{topdown}, @code{onerror}, @code{follow_symlinks}, @code{dir_fd}

@tab

@ref{683,,[1]}

@item

os.getxattr

@tab

@code{path}, @code{attribute}

@tab

@ref{c97,,[1]}

@item

os.kill

@tab

@code{pid}, @code{sig}

@tab

@ref{ef3,,[1]}

@item

os.killpg

@tab

@code{pgid}, @code{sig}

@tab

@ref{1278,,[1]}

@item

os.link

@tab

@code{src}, @code{dst}, @code{src_dir_fd}, @code{dst_dir_fd}

@tab

@ref{c82,,[1]}

@item

os.listdir

@tab

@code{path}

@tab

@ref{c8d,,[1]}

@item

os.listxattr

@tab

@code{path}

@tab

@ref{c98,,[1]}

@item

os.lockf

@tab

@code{fd}, @code{cmd}, @code{len}

@tab

@ref{ca6,,[1]}

@item

os.mkdir

@tab

@code{path}, @code{mode}, @code{dir_fd}

@tab

@ref{68c,,[1]}@ref{c83,,[2]}

@item

os.posix_spawn

@tab

@code{path}, @code{argv}, @code{env}

@tab

@ref{56e,,[1]}@ref{12dd,,[2]}

@item

os.putenv

@tab

@code{key}, @code{value}

@tab

@ref{410,,[1]}

@item

os.remove

@tab

@code{path}, @code{dir_fd}

@tab

@ref{c84,,[1]}@ref{23ef,,[2]}@ref{c89,,[3]}

@item

os.removexattr

@tab

@code{path}, @code{attribute}

@tab

@ref{c99,,[1]}

@item

os.rename

@tab

@code{src}, @code{dst}, @code{src_dir_fd}, @code{dst_dir_fd}

@tab

@ref{c85,,[1]}@ref{23f0,,[2]}@ref{c86,,[3]}

@item

os.rmdir

@tab

@code{path}, @code{dir_fd}

@tab

@ref{c87,,[1]}

@item

os.scandir

@tab

@code{path}

@tab

@ref{576,,[1]}

@item

os.setxattr

@tab

@code{path}, @code{attribute}, @code{value}, @code{flags}

@tab

@ref{c9a,,[1]}

@item

os.spawn

@tab

@code{mode}, @code{path}, @code{args}, @code{env}

@tab

@ref{23e0,,[1]}

@item

os.startfile

@tab

@code{path}, @code{operation}

@tab

@ref{1143,,[1]}

@item

os.startfile/2

@tab

@code{path}, @code{operation}, @code{arguments}, @code{cwd}, @code{show_cmd}

@tab

@ref{1143,,[1]}

@item

os.symlink

@tab

@code{src}, @code{dst}, @code{dir_fd}

@tab

@ref{c88,,[1]}

@item

os.system

@tab

@code{command}

@tab

@ref{fbb,,[1]}

@item

os.truncate

@tab

@code{fd}, @code{length}

@tab

@ref{8eb,,[1]}@ref{9a3,,[2]}

@item

os.unsetenv

@tab

@code{key}

@tab

@ref{40f,,[1]}

@item

os.utime

@tab

@code{path}, @code{times}, @code{ns}, @code{dir_fd}

@tab

@ref{c8a,,[1]}

@item

os.walk

@tab

@code{top}, @code{topdown}, @code{onerror}, @code{followlinks}

@tab

@ref{8de,,[1]}

@item

pathlib.Path.glob

@tab

@code{self}, @code{pattern}

@tab

@ref{1228,,[1]}

@item

pathlib.Path.rglob

@tab

@code{self}, @code{pattern}

@tab

@ref{1229,,[1]}

@item

pdb.Pdb

@tab



@tab

@ref{416,,[1]}

@item

pickle.find_class

@tab

@code{module}, @code{name}

@tab

@ref{2153,,[1]}

@item

poplib.connect

@tab

@code{self}, @code{host}, @code{port}

@tab

@ref{418,,[1]}@ref{419,,[2]}

@item

poplib.putline

@tab

@code{self}, @code{line}

@tab

@ref{418,,[1]}@ref{419,,[2]}

@item

pty.spawn

@tab

@code{argv}

@tab

@ref{b03,,[1]}

@item

resource.prlimit

@tab

@code{pid}, @code{resource}, @code{limits}

@tab

@ref{b0a,,[1]}

@item

resource.setrlimit

@tab

@code{resource}, @code{limits}

@tab

@ref{3996,,[1]}

@item

setopencodehook

@tab



@tab

@ref{2488,,[1]}

@item

shutil.chown

@tab

@code{path}, @code{user}, @code{group}

@tab

@ref{cc2,,[1]}

@item

shutil.copyfile

@tab

@code{src}, @code{dst}

@tab

@ref{570,,[1]}@ref{571,,[2]}@ref{56f,,[3]}

@item

shutil.copymode

@tab

@code{src}, @code{dst}

@tab

@ref{570,,[1]}@ref{211b,,[2]}

@item

shutil.copystat

@tab

@code{src}, @code{dst}

@tab

@ref{571,,[1]}@ref{cc3,,[2]}

@item

shutil.copytree

@tab

@code{src}, @code{dst}

@tab

@ref{538,,[1]}

@item

shutil.make_archive

@tab

@code{base_name}, @code{format}, @code{root_dir}, @code{base_dir}

@tab

@ref{539,,[1]}

@item

shutil.move

@tab

@code{src}, @code{dst}

@tab

@ref{572,,[1]}

@item

shutil.rmtree

@tab

@code{path}

@tab

@ref{53a,,[1]}

@item

shutil.unpack_archive

@tab

@code{filename}, @code{extract_dir}, @code{format}

@tab

@ref{dbb,,[1]}

@item

signal.pthread_kill

@tab

@code{thread_id}, @code{signalnum}

@tab

@ref{cc6,,[1]}

@item

smtplib.connect

@tab

@code{self}, @code{host}, @code{port}

@tab

@ref{1266,,[1]}

@item

smtplib.send

@tab

@code{self}, @code{data}

@tab

@ref{421,,[1]}

@item

socket.__new__

@tab

@code{self}, @code{family}, @code{type}, @code{protocol}

@tab

@ref{8fd,,[1]}

@item

socket.bind

@tab

@code{self}, @code{address}

@tab

@ref{2b1a,,[1]}

@item

socket.connect

@tab

@code{self}, @code{address}

@tab

@ref{8ff,,[1]}@ref{2b1c,,[2]}

@item

socket.getaddrinfo

@tab

@code{host}, @code{port}, @code{family}, @code{type}, @code{protocol}

@tab

@ref{11cc,,[1]}

@item

socket.gethostbyaddr

@tab

@code{ip_address}

@tab

@ref{1144,,[1]}

@item

socket.gethostbyname

@tab

@code{hostname}

@tab

@ref{2b12,,[1]}@ref{1145,,[2]}

@item

socket.gethostname

@tab



@tab

@ref{239d,,[1]}

@item

socket.getnameinfo

@tab

@code{sockaddr}

@tab

@ref{2a48,,[1]}

@item

socket.getservbyname

@tab

@code{servicename}, @code{protocolname}

@tab

@ref{2b14,,[1]}

@item

socket.getservbyport

@tab

@code{port}, @code{protocolname}

@tab

@ref{2b15,,[1]}

@item

socket.sendmsg

@tab

@code{self}, @code{address}

@tab

@ref{905,,[1]}

@item

socket.sendto

@tab

@code{self}, @code{address}

@tab

@ref{906,,[1]}

@item

socket.sethostname

@tab

@code{name}

@tab

@ref{cd0,,[1]}

@item

sqlite3.connect

@tab

@code{database}

@tab

@ref{6a8,,[1]}

@item

sqlite3.connect/handle

@tab

@code{connection_handle}

@tab

@ref{6a8,,[1]}

@item

sqlite3.enable_load_extension

@tab

@code{connection}, @code{enabled}

@tab

@ref{39f6,,[1]}

@item

sqlite3.load_extension

@tab

@code{connection}, @code{path}

@tab

@ref{39f7,,[1]}

@item

subprocess.Popen

@tab

@code{executable}, @code{args}, @code{cwd}, @code{env}

@tab

@ref{5ae,,[1]}

@item

sys._current_exceptions

@tab



@tab

@ref{11a8,,[1]}

@item

sys._current_frames

@tab



@tab

@ref{f95,,[1]}

@item

sys._getframe

@tab



@tab

@ref{1057,,[1]}

@item

sys.addaudithook

@tab



@tab

@ref{39ec,,[1]}@ref{39eb,,[2]}

@item

sys.excepthook

@tab

@code{hook}, @code{type}, @code{value}, @code{traceback}

@tab

@ref{28e,,[1]}

@item

sys.set_asyncgen_hooks_finalizer

@tab



@tab

@ref{11fd,,[1]}

@item

sys.set_asyncgen_hooks_firstiter

@tab



@tab

@ref{11fd,,[1]}

@item

sys.setprofile

@tab



@tab

@ref{1033,,[1]}

@item

sys.settrace

@tab



@tab

@ref{1034,,[1]}

@item

sys.unraisablehook

@tab

@code{hook}, @code{unraisable}

@tab

@ref{18d,,[1]}

@item

syslog.closelog

@tab



@tab

@ref{39f8,,[1]}

@item

syslog.openlog

@tab

@code{ident}, @code{logoption}, @code{facility}

@tab

@ref{1242,,[1]}

@item

syslog.setlogmask

@tab

@code{maskpri}

@tab

@ref{39f9,,[1]}

@item

syslog.syslog

@tab

@code{priority}, @code{message}

@tab

@ref{39fa,,[1]}

@item

telnetlib.Telnet.open

@tab

@code{self}, @code{host}, @code{port}

@tab

@ref{3264,,[1]}

@item

telnetlib.Telnet.write

@tab

@code{self}, @code{buffer}

@tab

@ref{3274,,[1]}

@item

tempfile.mkdtemp

@tab

@code{fullpath}

@tab

@ref{de5,,[1]}@ref{2103,,[2]}

@item

tempfile.mkstemp

@tab

@code{fullpath}

@tab

@ref{f44,,[1]}@ref{2101,,[2]}@ref{2102,,[3]}

@item

urllib.Request

@tab

@code{fullurl}, @code{data}, @code{headers}, @code{method}

@tab

@ref{a06,,[1]}

@item

webbrowser.open

@tab

@code{url}

@tab

@ref{3054,,[1]}

@item

winreg.ConnectRegistry

@tab

@code{computer_name}, @code{key}

@tab

@ref{39fb,,[1]}

@item

winreg.CreateKey

@tab

@code{key}, @code{sub_key}, @code{access}

@tab

@ref{39fc,,[1]}@ref{39fd,,[2]}

@item

winreg.DeleteKey

@tab

@code{key}, @code{sub_key}, @code{access}

@tab

@ref{39fe,,[1]}@ref{39ff,,[2]}

@item

winreg.DeleteValue

@tab

@code{key}, @code{value}

@tab

@ref{3a00,,[1]}

@item

winreg.DisableReflectionKey

@tab

@code{key}

@tab

@ref{3a01,,[1]}

@item

winreg.EnableReflectionKey

@tab

@code{key}

@tab

@ref{3a02,,[1]}

@item

winreg.EnumKey

@tab

@code{key}, @code{index}

@tab

@ref{3a03,,[1]}

@item

winreg.EnumValue

@tab

@code{key}, @code{index}

@tab

@ref{3a04,,[1]}

@item

winreg.ExpandEnvironmentStrings

@tab

@code{str}

@tab

@ref{3a05,,[1]}

@item

winreg.LoadKey

@tab

@code{key}, @code{sub_key}, @code{file_name}

@tab

@ref{3a06,,[1]}

@item

winreg.OpenKey

@tab

@code{key}, @code{sub_key}, @code{access}

@tab

@ref{3a07,,[1]}

@item

winreg.OpenKey/result

@tab

@code{key}

@tab

@ref{39fc,,[1]}@ref{39fd,,[2]}@ref{3a07,,[3]}

@item

winreg.PyHKEY.Detach

@tab

@code{key}

@tab

@ref{3a08,,[1]}

@item

winreg.QueryInfoKey

@tab

@code{key}

@tab

@ref{3a09,,[1]}

@item

winreg.QueryReflectionKey

@tab

@code{key}

@tab

@ref{3a0a,,[1]}

@item

winreg.QueryValue

@tab

@code{key}, @code{sub_key}, @code{value_name}

@tab

@ref{3a0b,,[1]}@ref{3a0c,,[2]}

@item

winreg.SaveKey

@tab

@code{key}, @code{file_name}

@tab

@ref{3a0d,,[1]}

@item

winreg.SetValue

@tab

@code{key}, @code{sub_key}, @code{type}, @code{value}

@tab

@ref{3a0e,,[1]}@ref{3a0f,,[2]}

@end multitable


The following events are raised internally and do not correspond to any
public API of CPython:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Audit event

@tab

Arguments

@item

_winapi.CreateFile

@tab

@code{file_name}, @code{desired_access},
@code{share_mode}, @code{creation_disposition},
@code{flags_and_attributes}

@item

_winapi.CreateJunction

@tab

@code{src_path}, @code{dst_path}

@item

_winapi.CreateNamedPipe

@tab

@code{name}, @code{open_mode}, @code{pipe_mode}

@item

_winapi.CreatePipe

@tab

@item

_winapi.CreateProcess

@tab

@code{application_name}, @code{command_line},
@code{current_directory}

@item

_winapi.OpenProcess

@tab

@code{process_id}, @code{desired_access}

@item

_winapi.TerminateProcess

@tab

@code{handle}, @code{exit_code}

@item

ctypes.PyObj_FromPtr

@tab

@code{obj}

@end multitable


@node bdb — Debugger framework,faulthandler — Dump the Python traceback,Audit events table,Debugging and Profiling
@anchor{library/bdb doc}@anchor{3a10}@anchor{library/bdb bdb-debugger-framework}@anchor{3a11}@anchor{library/bdb module-bdb}@anchor{e}
@subsection @code{bdb} — Debugger framework


@geindex module; bdb

@strong{Source code:} Lib/bdb.py@footnote{https://github.com/python/cpython/tree/main/Lib/bdb.py}


__________________________________________________________________


The @ref{e,,bdb} module handles basic debugger functions, like setting breakpoints
or managing execution via the debugger.

The following exception is defined:

@geindex BdbQuit
@anchor{library/bdb bdb BdbQuit}@anchor{3a12}
@deffn {Exception} bdb.BdbQuit

Exception raised by the @ref{e9d,,Bdb} class for quitting the debugger.
@end deffn

The @ref{e,,bdb} module also defines two classes:

@geindex Breakpoint (class in bdb)
@anchor{library/bdb bdb Breakpoint}@anchor{3a13}
@deffn {Class} bdb.Breakpoint (self, file, line, temporary=0, cond=None, funcname=None)

This class implements temporary breakpoints, ignore counts, disabling and
(re-)enabling, and conditionals.

Breakpoints are indexed by number through a list called @code{bpbynumber}
and by @code{(file, line)} pairs through @code{bplist}.  The former points to a
single instance of class @ref{3a13,,Breakpoint}.  The latter points to a list of
such instances since there may be more than one breakpoint per line.

When creating a breakpoint, its associated filename should be in canonical
form.  If a @emph{funcname} is defined, a breakpoint hit will be counted when the
first line of that function is executed.  A conditional breakpoint always
counts a hit.

@ref{3a13,,Breakpoint} instances have the following methods:

@geindex deleteMe() (bdb.Breakpoint method)
@anchor{library/bdb bdb Breakpoint deleteMe}@anchor{3a14}
@deffn {Method} deleteMe ()

Delete the breakpoint from the list associated to a file/line.  If it is
the last breakpoint in that position, it also deletes the entry for the
file/line.
@end deffn

@geindex enable() (bdb.Breakpoint method)
@anchor{library/bdb bdb Breakpoint enable}@anchor{3a15}
@deffn {Method} enable ()

Mark the breakpoint as enabled.
@end deffn

@geindex disable() (bdb.Breakpoint method)
@anchor{library/bdb bdb Breakpoint disable}@anchor{3a16}
@deffn {Method} disable ()

Mark the breakpoint as disabled.
@end deffn

@geindex bpformat() (bdb.Breakpoint method)
@anchor{library/bdb bdb Breakpoint bpformat}@anchor{3a17}
@deffn {Method} bpformat ()

Return a string with all the information about the breakpoint, nicely
formatted:


@itemize *

@item 
The breakpoint number.

@item 
If it is temporary or not.

@item 
Its file,line position.

@item 
The condition that causes a break.

@item 
If it must be ignored the next N times.

@item 
The breakpoint hit count.
@end itemize


New in version 3.2.

@end deffn

@geindex bpprint() (bdb.Breakpoint method)
@anchor{library/bdb bdb Breakpoint bpprint}@anchor{3a18}
@deffn {Method} bpprint (out=None)

Print the output of @ref{3a17,,bpformat()} to the file @emph{out}, or if it is
@code{None}, to standard output.
@end deffn
@end deffn

@geindex Bdb (class in bdb)
@anchor{library/bdb bdb Bdb}@anchor{e9d}
@deffn {Class} bdb.Bdb (skip=None)

The @ref{e9d,,Bdb} class acts as a generic Python debugger base class.

This class takes care of the details of the trace facility; a derived class
should implement user interaction.  The standard debugger class
(@ref{416,,pdb.Pdb}) is an example.

The @emph{skip} argument, if given, must be an iterable of glob-style
module name patterns.  The debugger will not step into frames that
originate in a module that matches one of these patterns. Whether a
frame is considered to originate in a certain module is determined
by the @code{__name__} in the frame globals.


New in version 3.1: The @emph{skip} argument.


The following methods of @ref{e9d,,Bdb} normally don’t need to be overridden.

@geindex canonic() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb canonic}@anchor{3a19}
@deffn {Method} canonic (filename)

Auxiliary method for getting a filename in a canonical form, that is, as a
case-normalized (on case-insensitive filesystems) absolute path, stripped
of surrounding angle brackets.
@end deffn

@geindex reset() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb reset}@anchor{3a1a}
@deffn {Method} reset ()

Set the @code{botframe}, @code{stopframe}, @code{returnframe} and
@code{quitting} attributes with values ready to start debugging.
@end deffn

@geindex trace_dispatch() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb trace_dispatch}@anchor{3a1b}
@deffn {Method} trace_dispatch (frame, event, arg)

This function is installed as the trace function of debugged frames.  Its
return value is the new trace function (in most cases, that is, itself).

The default implementation decides how to dispatch a frame, depending on
the type of event (passed as a string) that is about to be executed.
@emph{event} can be one of the following:


@itemize *

@item 
@code{"line"}: A new line of code is going to be executed.

@item 
@code{"call"}: A function is about to be called, or another code block
entered.

@item 
@code{"return"}: A function or other code block is about to return.

@item 
@code{"exception"}: An exception has occurred.

@item 
@code{"c_call"}: A C function is about to be called.

@item 
@code{"c_return"}: A C function has returned.

@item 
@code{"c_exception"}: A C function has raised an exception.
@end itemize

For the Python events, specialized functions (see below) are called.  For
the C events, no action is taken.

The @emph{arg} parameter depends on the previous event.

See the documentation for @ref{1034,,sys.settrace()} for more information on the
trace function.  For more information on code and frame objects, refer to
@ref{1787,,The standard type hierarchy}.
@end deffn

@geindex dispatch_line() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb dispatch_line}@anchor{3a1c}
@deffn {Method} dispatch_line (frame)

If the debugger should stop on the current line, invoke the
@ref{3a1d,,user_line()} method (which should be overridden in subclasses).
Raise a @ref{3a12,,BdbQuit} exception if the @code{Bdb.quitting} flag is set
(which can be set from @ref{3a1d,,user_line()}).  Return a reference to the
@ref{3a1b,,trace_dispatch()} method for further tracing in that scope.
@end deffn

@geindex dispatch_call() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb dispatch_call}@anchor{3a1e}
@deffn {Method} dispatch_call (frame, arg)

If the debugger should stop on this function call, invoke the
@ref{3a1f,,user_call()} method (which should be overridden in subclasses).
Raise a @ref{3a12,,BdbQuit} exception if the @code{Bdb.quitting} flag is set
(which can be set from @ref{3a1f,,user_call()}).  Return a reference to the
@ref{3a1b,,trace_dispatch()} method for further tracing in that scope.
@end deffn

@geindex dispatch_return() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb dispatch_return}@anchor{3a20}
@deffn {Method} dispatch_return (frame, arg)

If the debugger should stop on this function return, invoke the
@ref{3a21,,user_return()} method (which should be overridden in subclasses).
Raise a @ref{3a12,,BdbQuit} exception if the @code{Bdb.quitting} flag is set
(which can be set from @ref{3a21,,user_return()}).  Return a reference to the
@ref{3a1b,,trace_dispatch()} method for further tracing in that scope.
@end deffn

@geindex dispatch_exception() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb dispatch_exception}@anchor{3a22}
@deffn {Method} dispatch_exception (frame, arg)

If the debugger should stop at this exception, invokes the
@ref{3a23,,user_exception()} method (which should be overridden in subclasses).
Raise a @ref{3a12,,BdbQuit} exception if the @code{Bdb.quitting} flag is set
(which can be set from @ref{3a23,,user_exception()}).  Return a reference to the
@ref{3a1b,,trace_dispatch()} method for further tracing in that scope.
@end deffn

Normally derived classes don’t override the following methods, but they may
if they want to redefine the definition of stopping and breakpoints.

@geindex stop_here() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb stop_here}@anchor{3a24}
@deffn {Method} stop_here (frame)

This method checks if the @emph{frame} is somewhere below @code{botframe} in
the call stack.  @code{botframe} is the frame in which debugging started.
@end deffn

@geindex break_here() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb break_here}@anchor{3a25}
@deffn {Method} break_here (frame)

This method checks if there is a breakpoint in the filename and line
belonging to @emph{frame} or, at least, in the current function.  If the
breakpoint is a temporary one, this method deletes it.
@end deffn

@geindex break_anywhere() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb break_anywhere}@anchor{3a26}
@deffn {Method} break_anywhere (frame)

This method checks if there is a breakpoint in the filename of the current
frame.
@end deffn

Derived classes should override these methods to gain control over debugger
operation.

@geindex user_call() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb user_call}@anchor{3a1f}
@deffn {Method} user_call (frame, argument_list)

This method is called from @ref{3a1e,,dispatch_call()} when there is the
possibility that a break might be necessary anywhere inside the called
function.
@end deffn

@geindex user_line() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb user_line}@anchor{3a1d}
@deffn {Method} user_line (frame)

This method is called from @ref{3a1c,,dispatch_line()} when either
@ref{3a24,,stop_here()} or @ref{3a25,,break_here()} yields @code{True}.
@end deffn

@geindex user_return() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb user_return}@anchor{3a21}
@deffn {Method} user_return (frame, return_value)

This method is called from @ref{3a20,,dispatch_return()} when @ref{3a24,,stop_here()}
yields @code{True}.
@end deffn

@geindex user_exception() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb user_exception}@anchor{3a23}
@deffn {Method} user_exception (frame, exc_info)

This method is called from @ref{3a22,,dispatch_exception()} when
@ref{3a24,,stop_here()} yields @code{True}.
@end deffn

@geindex do_clear() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb do_clear}@anchor{3a27}
@deffn {Method} do_clear (arg)

Handle how a breakpoint must be removed when it is a temporary one.

This method must be implemented by derived classes.
@end deffn

Derived classes and clients can call the following methods to affect the
stepping state.

@geindex set_step() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb set_step}@anchor{3a28}
@deffn {Method} set_step ()

Stop after one line of code.
@end deffn

@geindex set_next() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb set_next}@anchor{3a29}
@deffn {Method} set_next (frame)

Stop on the next line in or below the given frame.
@end deffn

@geindex set_return() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb set_return}@anchor{3a2a}
@deffn {Method} set_return (frame)

Stop when returning from the given frame.
@end deffn

@geindex set_until() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb set_until}@anchor{3a2b}
@deffn {Method} set_until (frame)

Stop when the line with the line no greater than the current one is
reached or when returning from current frame.
@end deffn

@geindex set_trace() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb set_trace}@anchor{3a2c}
@deffn {Method} set_trace ([frame])

Start debugging from @emph{frame}.  If @emph{frame} is not specified, debugging
starts from caller’s frame.
@end deffn

@geindex set_continue() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb set_continue}@anchor{3a2d}
@deffn {Method} set_continue ()

Stop only at breakpoints or when finished.  If there are no breakpoints,
set the system trace function to @code{None}.
@end deffn

@geindex set_quit() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb set_quit}@anchor{3a2e}
@deffn {Method} set_quit ()

Set the @code{quitting} attribute to @code{True}.  This raises @ref{3a12,,BdbQuit} in
the next call to one of the @code{dispatch_*()} methods.
@end deffn

Derived classes and clients can call the following methods to manipulate
breakpoints.  These methods return a string containing an error message if
something went wrong, or @code{None} if all is well.

@geindex set_break() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb set_break}@anchor{3a2f}
@deffn {Method} set_break (filename, lineno, temporary=0, cond, funcname)

Set a new breakpoint.  If the @emph{lineno} line doesn’t exist for the
@emph{filename} passed as argument, return an error message.  The @emph{filename}
should be in canonical form, as described in the @ref{3a19,,canonic()} method.
@end deffn

@geindex clear_break() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb clear_break}@anchor{3a30}
@deffn {Method} clear_break (filename, lineno)

Delete the breakpoints in @emph{filename} and @emph{lineno}.  If none were set, an
error message is returned.
@end deffn

@geindex clear_bpbynumber() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb clear_bpbynumber}@anchor{3a31}
@deffn {Method} clear_bpbynumber (arg)

Delete the breakpoint which has the index @emph{arg} in the
@code{Breakpoint.bpbynumber}.  If @emph{arg} is not numeric or out of range,
return an error message.
@end deffn

@geindex clear_all_file_breaks() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb clear_all_file_breaks}@anchor{3a32}
@deffn {Method} clear_all_file_breaks (filename)

Delete all breakpoints in @emph{filename}.  If none were set, an error message
is returned.
@end deffn

@geindex clear_all_breaks() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb clear_all_breaks}@anchor{3a33}
@deffn {Method} clear_all_breaks ()

Delete all existing breakpoints.
@end deffn

@geindex get_bpbynumber() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb get_bpbynumber}@anchor{3a34}
@deffn {Method} get_bpbynumber (arg)

Return a breakpoint specified by the given number.  If @emph{arg} is a string,
it will be converted to a number.  If @emph{arg} is a non-numeric string, if
the given breakpoint never existed or has been deleted, a
@ref{17c,,ValueError} is raised.


New in version 3.2.

@end deffn

@geindex get_break() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb get_break}@anchor{3a35}
@deffn {Method} get_break (filename, lineno)

Check if there is a breakpoint for @emph{lineno} of @emph{filename}.
@end deffn

@geindex get_breaks() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb get_breaks}@anchor{3a36}
@deffn {Method} get_breaks (filename, lineno)

Return all breakpoints for @emph{lineno} in @emph{filename}, or an empty list if
none are set.
@end deffn

@geindex get_file_breaks() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb get_file_breaks}@anchor{3a37}
@deffn {Method} get_file_breaks (filename)

Return all breakpoints in @emph{filename}, or an empty list if none are set.
@end deffn

@geindex get_all_breaks() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb get_all_breaks}@anchor{3a38}
@deffn {Method} get_all_breaks ()

Return all breakpoints that are set.
@end deffn

Derived classes and clients can call the following methods to get a data
structure representing a stack trace.

@geindex get_stack() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb get_stack}@anchor{3a39}
@deffn {Method} get_stack (f, t)

Get a list of records for a frame and all higher (calling) and lower
frames, and the size of the higher part.
@end deffn

@geindex format_stack_entry() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb format_stack_entry}@anchor{3a3a}
@deffn {Method} format_stack_entry (frame_lineno, lprefix=': ')

Return a string with information about a stack entry, identified by a
@code{(frame, lineno)} tuple:


@itemize *

@item 
The canonical form of the filename which contains the frame.

@item 
The function name, or @code{"<lambda>"}.

@item 
The input arguments.

@item 
The return value.

@item 
The line of code (if it exists).
@end itemize
@end deffn

The following two methods can be called by clients to use a debugger to debug
a @ref{2034,,statement}, given as a string.

@geindex run() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb run}@anchor{3a3b}
@deffn {Method} run (cmd, globals=None, locals=None)

Debug a statement executed via the @ref{34b,,exec()} function.  @emph{globals}
defaults to @code{__main__.__dict__}, @emph{locals} defaults to @emph{globals}.
@end deffn

@geindex runeval() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb runeval}@anchor{3a3c}
@deffn {Method} runeval (expr, globals=None, locals=None)

Debug an expression executed via the @ref{34a,,eval()} function.  @emph{globals} and
@emph{locals} have the same meaning as in @ref{3a3b,,run()}.
@end deffn

@geindex runctx() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb runctx}@anchor{3a3d}
@deffn {Method} runctx (cmd, globals, locals)

For backwards compatibility.  Calls the @ref{3a3b,,run()} method.
@end deffn

@geindex runcall() (bdb.Bdb method)
@anchor{library/bdb bdb Bdb runcall}@anchor{596}
@deffn {Method} runcall (func, /, *args, **kwds)

Debug a single function call, and return its result.
@end deffn
@end deffn

Finally, the module defines the following functions:

@geindex checkfuncname() (in module bdb)
@anchor{library/bdb bdb checkfuncname}@anchor{3a3e}
@deffn {Function} bdb.checkfuncname (b, frame)

Check whether we should break here, depending on the way the breakpoint @emph{b}
was set.

If it was set via line number, it checks if @code{b.line} is the same as the one
in the frame also passed as argument.  If the breakpoint was set via function
name, we have to check we are in the right frame (the right function) and if
we are in its first executable line.
@end deffn

@geindex effective() (in module bdb)
@anchor{library/bdb bdb effective}@anchor{3a3f}
@deffn {Function} bdb.effective (file, line, frame)

Determine if there is an effective (active) breakpoint at this line of code.
Return a tuple of the breakpoint and a boolean that indicates if it is ok
to delete a temporary breakpoint.  Return @code{(None, None)} if there is no
matching breakpoint.
@end deffn

@geindex set_trace() (in module bdb)
@anchor{library/bdb bdb set_trace}@anchor{3a40}
@deffn {Function} bdb.set_trace ()

Start debugging with a @ref{e9d,,Bdb} instance from caller’s frame.
@end deffn

@node faulthandler — Dump the Python traceback,pdb — The Python Debugger,bdb — Debugger framework,Debugging and Profiling
@anchor{library/faulthandler doc}@anchor{3a41}@anchor{library/faulthandler faulthandler-dump-the-python-traceback}@anchor{3a42}@anchor{library/faulthandler module-faulthandler}@anchor{78}
@subsection @code{faulthandler} — Dump the Python traceback


@geindex module; faulthandler


New in version 3.3.



__________________________________________________________________


This module contains functions to dump Python tracebacks explicitly, on a fault,
after a timeout, or on a user signal. Call @ref{7ea,,faulthandler.enable()} to
install fault handlers for the @code{SIGSEGV}, @code{SIGFPE},
@code{SIGABRT}, @code{SIGBUS}, and @code{SIGILL} signals. You can also
enable them at startup by setting the 
@geindex PYTHONFAULTHANDLER
@geindex environment variable; PYTHONFAULTHANDLER
@ref{c18,,PYTHONFAULTHANDLER} environment
variable or by using the @ref{159,,-X} @code{faulthandler} command line option.

The fault handler is compatible with system fault handlers like Apport or the
Windows fault handler. The module uses an alternative stack for signal handlers
if the @code{sigaltstack()} function is available. This allows it to dump the
traceback even on a stack overflow.

The fault handler is called on catastrophic cases and therefore can only use
signal-safe functions (e.g. it cannot allocate memory on the heap). Because of
this limitation traceback dumping is minimal compared to normal Python
tracebacks:


@itemize *

@item 
Only ASCII is supported. The @code{backslashreplace} error handler is used on
encoding.

@item 
Each string is limited to 500 characters.

@item 
Only the filename, the function name and the line number are
displayed. (no source code)

@item 
It is limited to 100 frames and 100 threads.

@item 
The order is reversed: the most recent call is shown first.
@end itemize

By default, the Python traceback is written to @ref{42d,,sys.stderr}. To see
tracebacks, applications must be run in the terminal. A log file can
alternatively be passed to @ref{7ea,,faulthandler.enable()}.

The module is implemented in C, so tracebacks can be dumped on a crash or when
Python is deadlocked.

The @ref{3ac,,Python Development Mode} calls @ref{7ea,,faulthandler.enable()}
at Python startup.

@menu
* Dumping the traceback:: 
* Fault handler state:: 
* Dumping the tracebacks after a timeout:: 
* Dumping the traceback on a user signal:: 
* Issue with file descriptors:: 
* Example: Example<14>. 

@end menu

@node Dumping the traceback,Fault handler state,,faulthandler — Dump the Python traceback
@anchor{library/faulthandler dumping-the-traceback}@anchor{3a43}
@subsubsection Dumping the traceback


@geindex dump_traceback() (in module faulthandler)
@anchor{library/faulthandler faulthandler dump_traceback}@anchor{95d}
@deffn {Function} faulthandler.dump_traceback (file=sys.stderr, all_threads=True)

Dump the tracebacks of all threads into @emph{file}. If @emph{all_threads} is
@code{False}, dump only the current thread.


Changed in version 3.5: Added support for passing file descriptor to this function.

@end deffn

@node Fault handler state,Dumping the tracebacks after a timeout,Dumping the traceback,faulthandler — Dump the Python traceback
@anchor{library/faulthandler fault-handler-state}@anchor{3a44}
@subsubsection Fault handler state


@geindex enable() (in module faulthandler)
@anchor{library/faulthandler faulthandler enable}@anchor{7ea}
@deffn {Function} faulthandler.enable (file=sys.stderr, all_threads=True)

Enable the fault handler: install handlers for the @code{SIGSEGV},
@code{SIGFPE}, @code{SIGABRT}, @code{SIGBUS} and @code{SIGILL}
signals to dump the Python traceback. If @emph{all_threads} is @code{True},
produce tracebacks for every running thread. Otherwise, dump only the current
thread.

The @emph{file} must be kept open until the fault handler is disabled: see
@ref{3a45,,issue with file descriptors}.


Changed in version 3.5: Added support for passing file descriptor to this function.



Changed in version 3.6: On Windows, a handler for Windows exception is also installed.



Changed in version 3.10: The dump now mentions if a garbage collector collection is running
if @emph{all_threads} is true.

@end deffn

@geindex disable() (in module faulthandler)
@anchor{library/faulthandler faulthandler disable}@anchor{3a46}
@deffn {Function} faulthandler.disable ()

Disable the fault handler: uninstall the signal handlers installed by
@ref{7ea,,enable()}.
@end deffn

@geindex is_enabled() (in module faulthandler)
@anchor{library/faulthandler faulthandler is_enabled}@anchor{3a47}
@deffn {Function} faulthandler.is_enabled ()

Check if the fault handler is enabled.
@end deffn

@node Dumping the tracebacks after a timeout,Dumping the traceback on a user signal,Fault handler state,faulthandler — Dump the Python traceback
@anchor{library/faulthandler dumping-the-tracebacks-after-a-timeout}@anchor{3a48}
@subsubsection Dumping the tracebacks after a timeout


@geindex dump_traceback_later() (in module faulthandler)
@anchor{library/faulthandler faulthandler dump_traceback_later}@anchor{95e}
@deffn {Function} faulthandler.dump_traceback_later (timeout, repeat=False, file=sys.stderr, exit=False)

Dump the tracebacks of all threads, after a timeout of @emph{timeout} seconds, or
every @emph{timeout} seconds if @emph{repeat} is @code{True}.  If @emph{exit} is @code{True}, call
@code{_exit()} with status=1 after dumping the tracebacks.  (Note
@code{_exit()} exits the process immediately, which means it doesn’t do any
cleanup like flushing file buffers.) If the function is called twice, the new
call replaces previous parameters and resets the timeout. The timer has a
sub-second resolution.

The @emph{file} must be kept open until the traceback is dumped or
@ref{3a49,,cancel_dump_traceback_later()} is called: see @ref{3a45,,issue with file descriptors}.

This function is implemented using a watchdog thread.


Changed in version 3.7: This function is now always available.



Changed in version 3.5: Added support for passing file descriptor to this function.

@end deffn

@geindex cancel_dump_traceback_later() (in module faulthandler)
@anchor{library/faulthandler faulthandler cancel_dump_traceback_later}@anchor{3a49}
@deffn {Function} faulthandler.cancel_dump_traceback_later ()

Cancel the last call to @ref{95e,,dump_traceback_later()}.
@end deffn

@node Dumping the traceback on a user signal,Issue with file descriptors,Dumping the tracebacks after a timeout,faulthandler — Dump the Python traceback
@anchor{library/faulthandler dumping-the-traceback-on-a-user-signal}@anchor{3a4a}
@subsubsection Dumping the traceback on a user signal


@geindex register() (in module faulthandler)
@anchor{library/faulthandler faulthandler register}@anchor{95c}
@deffn {Function} faulthandler.register (signum, file=sys.stderr, all_threads=True, chain=False)

Register a user signal: install a handler for the @emph{signum} signal to dump
the traceback of all threads, or of the current thread if @emph{all_threads} is
@code{False}, into @emph{file}. Call the previous handler if chain is @code{True}.

The @emph{file} must be kept open until the signal is unregistered by
@ref{3a4b,,unregister()}: see @ref{3a45,,issue with file descriptors}.

Not available on Windows.


Changed in version 3.5: Added support for passing file descriptor to this function.

@end deffn

@geindex unregister() (in module faulthandler)
@anchor{library/faulthandler faulthandler unregister}@anchor{3a4b}
@deffn {Function} faulthandler.unregister (signum)

Unregister a user signal: uninstall the handler of the @emph{signum} signal
installed by @ref{95c,,register()}. Return @code{True} if the signal was registered,
@code{False} otherwise.

Not available on Windows.
@end deffn

@node Issue with file descriptors,Example<14>,Dumping the traceback on a user signal,faulthandler — Dump the Python traceback
@anchor{library/faulthandler faulthandler-fd}@anchor{3a45}@anchor{library/faulthandler issue-with-file-descriptors}@anchor{3a4c}
@subsubsection Issue with file descriptors


@ref{7ea,,enable()}, @ref{95e,,dump_traceback_later()} and @ref{95c,,register()} keep the
file descriptor of their @emph{file} argument. If the file is closed and its file
descriptor is reused by a new file, or if @ref{68d,,os.dup2()} is used to replace
the file descriptor, the traceback will be written into a different file. Call
these functions again each time that the file is replaced.

@node Example<14>,,Issue with file descriptors,faulthandler — Dump the Python traceback
@anchor{library/faulthandler example}@anchor{3a4d}
@subsubsection Example


Example of a segmentation fault on Linux with and without enabling the fault
handler:

@example
$ python3 -c "import ctypes; ctypes.string_at(0)"
Segmentation fault

$ python3 -q -X faulthandler
>>> import ctypes
>>> ctypes.string_at(0)
Fatal Python error: Segmentation fault

Current thread 0x00007fb899f39700 (most recent call first):
  File "/home/python/cpython/Lib/ctypes/__init__.py", line 486 in string_at
  File "<stdin>", line 1 in <module>
Segmentation fault
@end example

@node pdb — The Python Debugger,The Python Profilers,faulthandler — Dump the Python traceback,Debugging and Profiling
@anchor{library/pdb doc}@anchor{3a4e}@anchor{library/pdb debugger}@anchor{3a4f}@anchor{library/pdb module-pdb}@anchor{c4}@anchor{library/pdb pdb-the-python-debugger}@anchor{3a50}
@subsection @code{pdb} — The Python Debugger


@geindex module; pdb

@strong{Source code:} Lib/pdb.py@footnote{https://github.com/python/cpython/tree/main/Lib/pdb.py}

@geindex debugging


__________________________________________________________________


The module @ref{c4,,pdb} defines an interactive source code debugger for Python
programs.  It supports setting (conditional) breakpoints and single stepping at
the source line level, inspection of stack frames, source code listing, and
evaluation of arbitrary Python code in the context of any stack frame.  It also
supports post-mortem debugging and can be called under program control.

@geindex Pdb (class in pdb)
@geindex module; bdb
@geindex module; cmd

The debugger is extensible – it is actually defined as the class @ref{416,,Pdb}.
This is currently undocumented but easily understood by reading the source.  The
extension interface uses the modules @ref{e,,bdb} and @ref{18,,cmd}.

The debugger’s prompt is @code{(Pdb)}. Typical usage to run a program under control
of the debugger is:

@example
>>> import pdb
>>> import mymodule
>>> pdb.run('mymodule.test()')
> <string>(0)?()
(Pdb) continue
> <string>(1)?()
(Pdb) continue
NameError: 'spam'
> <string>(1)?()
(Pdb)
@end example


Changed in version 3.3: Tab-completion via the @ref{d9,,readline} module is available for commands and
command arguments, e.g. the current global and local names are offered as
arguments of the @code{p} command.


@code{pdb.py} can also be invoked as a script to debug other scripts.  For
example:

@example
python3 -m pdb myscript.py
@end example

When invoked as a script, pdb will automatically enter post-mortem debugging if
the program being debugged exits abnormally.  After post-mortem debugging (or
after normal exit of the program), pdb will restart the program.  Automatic
restarting preserves pdb’s state (such as breakpoints) and in most cases is more
useful than quitting the debugger upon program’s exit.


New in version 3.2: @code{pdb.py} now accepts a @code{-c} option that executes commands as if given
in a @code{.pdbrc} file, see @ref{3a51,,Debugger Commands}.



New in version 3.7: @code{pdb.py} now accepts a @code{-m} option that execute modules similar to the way
@code{python3 -m} does. As with a script, the debugger will pause execution just
before the first line of the module.


The typical usage to break into the debugger is to insert:

@example
import pdb; pdb.set_trace()
@end example

at the location you want to break into the debugger, and then run the program.
You can then step through the code following this statement, and continue
running without the debugger using the @ref{3a52,,continue} command.


New in version 3.7: The built-in @ref{5db,,breakpoint()}, when called with defaults, can be used
instead of @code{import pdb; pdb.set_trace()}.


The typical usage to inspect a crashed program is:

@example
>>> import pdb
>>> import mymodule
>>> mymodule.test()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "./mymodule.py", line 4, in test
    test2()
  File "./mymodule.py", line 3, in test2
    print(spam)
NameError: spam
>>> pdb.pm()
> ./mymodule.py(3)test2()
-> print(spam)
(Pdb)
@end example

The module defines the following functions; each enters the debugger in a
slightly different way:

@geindex run() (in module pdb)
@anchor{library/pdb pdb run}@anchor{37f8}
@deffn {Function} pdb.run (statement, globals=None, locals=None)

Execute the @emph{statement} (given as a string or a code object) under debugger
control.  The debugger prompt appears before any code is executed; you can
set breakpoints and type @ref{3a52,,continue}, or you can step through the
statement using @ref{3a53,,step} or @ref{3a54,,next} (all these commands are
explained below).  The optional @emph{globals} and @emph{locals} arguments specify the
environment in which the code is executed; by default the dictionary of the
module @ref{1,,__main__} is used.  (See the explanation of the built-in
@ref{34b,,exec()} or @ref{34a,,eval()} functions.)
@end deffn

@geindex runeval() (in module pdb)
@anchor{library/pdb pdb runeval}@anchor{3a55}
@deffn {Function} pdb.runeval (expression, globals=None, locals=None)

Evaluate the @emph{expression} (given as a string or a code object) under debugger
control.  When @ref{3a55,,runeval()} returns, it returns the value of the
expression.  Otherwise this function is similar to @ref{37f8,,run()}.
@end deffn

@geindex runcall() (in module pdb)
@anchor{library/pdb pdb runcall}@anchor{3a56}
@deffn {Function} pdb.runcall (function, *args, **kwds)

Call the @emph{function} (a function or method object, not a string) with the
given arguments.  When @ref{3a56,,runcall()} returns, it returns whatever the
function call returned.  The debugger prompt appears as soon as the function
is entered.
@end deffn

@geindex set_trace() (in module pdb)
@anchor{library/pdb pdb set_trace}@anchor{691}
@deffn {Function} pdb.set_trace (*, header=None)

Enter the debugger at the calling stack frame.  This is useful to hard-code
a breakpoint at a given point in a program, even if the code is not
otherwise being debugged (e.g. when an assertion fails).  If given,
@emph{header} is printed to the console just before debugging begins.


Changed in version 3.7: The keyword-only argument @emph{header}.

@end deffn

@geindex post_mortem() (in module pdb)
@anchor{library/pdb pdb post_mortem}@anchor{f40}
@deffn {Function} pdb.post_mortem (traceback=None)

Enter post-mortem debugging of the given @emph{traceback} object.  If no
@emph{traceback} is given, it uses the one of the exception that is currently
being handled (an exception must be being handled if the default is to be
used).
@end deffn

@geindex pm() (in module pdb)
@anchor{library/pdb pdb pm}@anchor{3a57}
@deffn {Function} pdb.pm ()

Enter post-mortem debugging of the traceback found in
@ref{3a58,,sys.last_traceback}.
@end deffn

The @code{run*} functions and @ref{691,,set_trace()} are aliases for instantiating the
@ref{416,,Pdb} class and calling the method of the same name.  If you want to
access further features, you have to do this yourself:

@geindex Pdb (class in pdb)
@anchor{library/pdb pdb Pdb}@anchor{416}
@deffn {Class} pdb.Pdb (completekey='tab', stdin=None, stdout=None, skip=None, nosigint=False, readrc=True)

@ref{416,,Pdb} is the debugger class.

The @emph{completekey}, @emph{stdin} and @emph{stdout} arguments are passed to the
underlying @ref{35bb,,cmd.Cmd} class; see the description there.

The @emph{skip} argument, if given, must be an iterable of glob-style module name
patterns.  The debugger will not step into frames that originate in a module
that matches one of these patterns. @footnote{@w{(1)} 
Whether a frame is considered to originate in a certain module
is determined by the @code{__name__} in the frame globals.
}

By default, Pdb sets a handler for the SIGINT signal (which is sent when the
user presses @code{Ctrl-C} on the console) when you give a @code{continue} command.
This allows you to break into the debugger again by pressing @code{Ctrl-C}.  If you
want Pdb not to touch the SIGINT handler, set @emph{nosigint} to true.

The @emph{readrc} argument defaults to true and controls whether Pdb will load
.pdbrc files from the filesystem.

Example call to enable tracing with @emph{skip}:

@example
import pdb; pdb.Pdb(skip=['django.*']).set_trace()
@end example

Raises an @ref{1130,,auditing event} @code{pdb.Pdb} with no arguments.


New in version 3.1: The @emph{skip} argument.



New in version 3.2: The @emph{nosigint} argument.  Previously, a SIGINT handler was never set by
Pdb.



Changed in version 3.6: The @emph{readrc} argument.


@geindex run() (pdb.Pdb method)
@geindex runeval() (pdb.Pdb method)
@geindex runcall() (pdb.Pdb method)
@geindex set_trace() (pdb.Pdb method)
@anchor{library/pdb pdb Pdb run}@anchor{3a59}
@deffn {Method} run (statement, globals=None, locals=None)
@anchor{library/pdb pdb Pdb runeval}@anchor{3a5a}
@deffnx {Method} runeval (expression, globals=None, locals=None)
@anchor{library/pdb pdb Pdb runcall}@anchor{3a5b}
@deffnx {Method} runcall (function, *args, **kwds)
@anchor{library/pdb pdb Pdb set_trace}@anchor{3a5c}
@deffnx {Method} set_trace ()

See the documentation for the functions explained above.
@end deffn
@end deffn

@menu
* Debugger Commands:: 

@end menu

@node Debugger Commands,,,pdb — The Python Debugger
@anchor{library/pdb debugger-commands}@anchor{3a51}@anchor{library/pdb id2}@anchor{3a5d}
@subsubsection Debugger Commands


The commands recognized by the debugger are listed below.  Most commands can be
abbreviated to one or two letters as indicated; e.g. @code{h(elp)} means that
either @code{h} or @code{help} can be used to enter the help command (but not @code{he}
or @code{hel}, nor @code{H} or @code{Help} or @code{HELP}).  Arguments to commands must be
separated by whitespace (spaces or tabs).  Optional arguments are enclosed in
square brackets (@code{[]}) in the command syntax; the square brackets must not be
typed.  Alternatives in the command syntax are separated by a vertical bar
(@code{|}).

Entering a blank line repeats the last command entered.  Exception: if the last
command was a @ref{3a5e,,list} command, the next 11 lines are listed.

Commands that the debugger doesn’t recognize are assumed to be Python statements
and are executed in the context of the program being debugged.  Python
statements can also be prefixed with an exclamation point (@code{!}).  This is a
powerful way to inspect the program being debugged; it is even possible to
change a variable or call a function.  When an exception occurs in such a
statement, the exception name is printed but the debugger’s state is not
changed.

The debugger supports @ref{3a5f,,aliases}.  Aliases can have
parameters which allows one a certain level of adaptability to the context under
examination.

Multiple commands may be entered on a single line, separated by @code{;;}.  (A
single @code{;} is not used as it is the separator for multiple commands in a line
that is passed to the Python parser.)  No intelligence is applied to separating
the commands; the input is split at the first @code{;;} pair, even if it is in the
middle of a quoted string.

@geindex .pdbrc; file
@geindex debugger; configuration; file

If a file @code{.pdbrc} exists in the user’s home directory or in the current
directory, it is read with @code{'utf-8'} encoding and executed as if it had been
typed at the debugger prompt.  This is particularly useful for aliases.  If both
files exist, the one in the home directory is read first and aliases defined there
can be overridden by the local file.


Changed in version 3.11: @code{.pdbrc} is now read with @code{'utf-8'} encoding. Previously, it was read
with the system locale encoding.



Changed in version 3.2: @code{.pdbrc} can now contain commands that continue debugging, such as
@ref{3a52,,continue} or @ref{3a54,,next}.  Previously, these commands had no
effect.


@geindex help (pdb command)
@anchor{library/pdb pdbcommand-help}@anchor{3a60}
@deffn {Pdbcommand} h(elp) [command]

Without argument, print the list of available commands.  With a @emph{command} as
argument, print help about that command.  @code{help pdb} displays the full
documentation (the docstring of the @ref{c4,,pdb} module).  Since the @emph{command}
argument must be an identifier, @code{help exec} must be entered to get help on
the @code{!} command.
@end deffn

@geindex where (pdb command)
@anchor{library/pdb pdbcommand-where}@anchor{3a61}
@deffn {Pdbcommand} w(here)

Print a stack trace, with the most recent frame at the bottom.  An arrow
indicates the current frame, which determines the context of most commands.
@end deffn

@geindex down (pdb command)
@anchor{library/pdb pdbcommand-down}@anchor{3a62}
@deffn {Pdbcommand} d(own) [count]

Move the current frame @emph{count} (default one) levels down in the stack trace
(to a newer frame).
@end deffn

@geindex up (pdb command)
@anchor{library/pdb pdbcommand-up}@anchor{3a63}
@deffn {Pdbcommand} u(p) [count]

Move the current frame @emph{count} (default one) levels up in the stack trace (to
an older frame).
@end deffn

@geindex break (pdb command)
@anchor{library/pdb pdbcommand-break}@anchor{3a64}
@deffn {Pdbcommand} b(reak) [([filename:]lineno | function) [, condition]]

With a @emph{lineno} argument, set a break there in the current file.  With a
@emph{function} argument, set a break at the first executable statement within
that function.  The line number may be prefixed with a filename and a colon,
to specify a breakpoint in another file (probably one that hasn’t been loaded
yet).  The file is searched on @ref{489,,sys.path}.  Note that each breakpoint
is assigned a number to which all the other breakpoint commands refer.

If a second argument is present, it is an expression which must evaluate to
true before the breakpoint is honored.

Without argument, list all breaks, including for each breakpoint, the number
of times that breakpoint has been hit, the current ignore count, and the
associated condition if any.
@end deffn

@geindex tbreak (pdb command)
@anchor{library/pdb pdbcommand-tbreak}@anchor{3a65}
@deffn {Pdbcommand} tbreak [([filename:]lineno | function) [, condition]]

Temporary breakpoint, which is removed automatically when it is first hit.
The arguments are the same as for @ref{3a64,,break}.
@end deffn

@geindex clear (pdb command)
@anchor{library/pdb pdbcommand-clear}@anchor{3a66}
@deffn {Pdbcommand} cl(ear) [filename:lineno | bpnumber ...]

With a @emph{filename:lineno} argument, clear all the breakpoints at this line.
With a space separated list of breakpoint numbers, clear those breakpoints.
Without argument, clear all breaks (but first ask confirmation).
@end deffn

@geindex disable (pdb command)
@anchor{library/pdb pdbcommand-disable}@anchor{3a67}
@deffn {Pdbcommand} disable [bpnumber ...]

Disable the breakpoints given as a space separated list of breakpoint
numbers.  Disabling a breakpoint means it cannot cause the program to stop
execution, but unlike clearing a breakpoint, it remains in the list of
breakpoints and can be (re-)enabled.
@end deffn

@geindex enable (pdb command)
@anchor{library/pdb pdbcommand-enable}@anchor{3a68}
@deffn {Pdbcommand} enable [bpnumber ...]

Enable the breakpoints specified.
@end deffn

@geindex ignore (pdb command)
@anchor{library/pdb pdbcommand-ignore}@anchor{3a69}
@deffn {Pdbcommand} ignore bpnumber [count]

Set the ignore count for the given breakpoint number.  If count is omitted,
the ignore count is set to 0.  A breakpoint becomes active when the ignore
count is zero.  When non-zero, the count is decremented each time the
breakpoint is reached and the breakpoint is not disabled and any associated
condition evaluates to true.
@end deffn

@geindex condition (pdb command)
@anchor{library/pdb pdbcommand-condition}@anchor{3a6a}
@deffn {Pdbcommand} condition bpnumber [condition]

Set a new @emph{condition} for the breakpoint, an expression which must evaluate
to true before the breakpoint is honored.  If @emph{condition} is absent, any
existing condition is removed; i.e., the breakpoint is made unconditional.
@end deffn

@geindex commands (pdb command)
@anchor{library/pdb pdbcommand-commands}@anchor{3a6b}
@deffn {Pdbcommand} commands [bpnumber]

Specify a list of commands for breakpoint number @emph{bpnumber}.  The commands
themselves appear on the following lines.  Type a line containing just
@code{end} to terminate the commands. An example:

@example
(Pdb) commands 1
(com) p some_variable
(com) end
(Pdb)
@end example

To remove all commands from a breakpoint, type @code{commands} and follow it
immediately with @code{end}; that is, give no commands.

With no @emph{bpnumber} argument, @code{commands} refers to the last breakpoint set.

You can use breakpoint commands to start your program up again.  Simply use
the @ref{3a52,,continue} command, or @ref{3a53,,step},
or any other command that resumes execution.

Specifying any command resuming execution
(currently @ref{3a52,,continue}, @ref{3a53,,step}, @ref{3a54,,next},
@ref{3a6c,,return}, @ref{3a6d,,jump}, @ref{3a6e,,quit} and their abbreviations)
terminates the command list (as if
that command was immediately followed by end). This is because any time you
resume execution (even with a simple next or step), you may encounter another
breakpoint—which could have its own command list, leading to ambiguities about
which list to execute.

If you use the ‘silent’ command in the command list, the usual message about
stopping at a breakpoint is not printed.  This may be desirable for breakpoints
that are to print a specific message and then continue.  If none of the other
commands print anything, you see no sign that the breakpoint was reached.
@end deffn

@geindex step (pdb command)
@anchor{library/pdb pdbcommand-step}@anchor{3a53}
@deffn {Pdbcommand} s(tep)

Execute the current line, stop at the first possible occasion (either in a
function that is called or on the next line in the current function).
@end deffn

@geindex next (pdb command)
@anchor{library/pdb pdbcommand-next}@anchor{3a54}
@deffn {Pdbcommand} n(ext)

Continue execution until the next line in the current function is reached or
it returns.  (The difference between @ref{3a54,,next} and @ref{3a53,,step} is
that @ref{3a53,,step} stops inside a called function, while @ref{3a54,,next}
executes called functions at (nearly) full speed, only stopping at the next
line in the current function.)
@end deffn

@geindex until (pdb command)
@anchor{library/pdb pdbcommand-until}@anchor{3a6f}
@deffn {Pdbcommand} unt(il) [lineno]

Without argument, continue execution until the line with a number greater
than the current one is reached.

With a line number, continue execution until a line with a number greater or
equal to that is reached.  In both cases, also stop when the current frame
returns.


Changed in version 3.2: Allow giving an explicit line number.

@end deffn

@geindex return (pdb command)
@anchor{library/pdb pdbcommand-return}@anchor{3a6c}
@deffn {Pdbcommand} r(eturn)

Continue execution until the current function returns.
@end deffn

@geindex continue (pdb command)
@anchor{library/pdb pdbcommand-continue}@anchor{3a52}
@deffn {Pdbcommand} c(ont(inue))

Continue execution, only stop when a breakpoint is encountered.
@end deffn

@geindex jump (pdb command)
@anchor{library/pdb pdbcommand-jump}@anchor{3a6d}
@deffn {Pdbcommand} j(ump) lineno

Set the next line that will be executed.  Only available in the bottom-most
frame.  This lets you jump back and execute code again, or jump forward to
skip code that you don’t want to run.

It should be noted that not all jumps are allowed – for instance it is not
possible to jump into the middle of a @ref{161,,for} loop or out of a
@ref{4c1,,finally} clause.
@end deffn

@geindex list (pdb command)
@anchor{library/pdb pdbcommand-list}@anchor{3a5e}
@deffn {Pdbcommand} l(ist) [first[, last]]

List source code for the current file.  Without arguments, list 11 lines
around the current line or continue the previous listing.  With @code{.} as
argument, list 11 lines around the current line.  With one argument,
list 11 lines around at that line.  With two arguments, list the given range;
if the second argument is less than the first, it is interpreted as a count.

The current line in the current frame is indicated by @code{->}.  If an
exception is being debugged, the line where the exception was originally
raised or propagated is indicated by @code{>>}, if it differs from the current
line.


New in version 3.2: The @code{>>} marker.

@end deffn

@geindex ll (pdb command)
@anchor{library/pdb pdbcommand-ll}@anchor{3a70}
@deffn {Pdbcommand} ll | longlist

List all source code for the current function or frame.  Interesting lines
are marked as for @ref{3a5e,,list}.


New in version 3.2.

@end deffn

@geindex args (pdb command)
@anchor{library/pdb pdbcommand-args}@anchor{3a71}
@deffn {Pdbcommand} a(rgs)

Print the argument list of the current function.
@end deffn

@geindex p (pdb command)
@anchor{library/pdb pdbcommand-p}@anchor{af7}
@deffn {Pdbcommand} p expression

Evaluate the @emph{expression} in the current context and print its value.

@cartouche
@quotation Note 
@code{print()} can also be used, but is not a debugger command — this executes the
Python @ref{af6,,print()} function.
@end quotation
@end cartouche
@end deffn

@geindex pp (pdb command)
@anchor{library/pdb pdbcommand-pp}@anchor{3a72}
@deffn {Pdbcommand} pp expression

Like the @ref{af7,,p} command, except the value of the expression is
pretty-printed using the @ref{cd,,pprint} module.
@end deffn

@geindex whatis (pdb command)
@anchor{library/pdb pdbcommand-whatis}@anchor{3a73}
@deffn {Pdbcommand} whatis expression

Print the type of the @emph{expression}.
@end deffn

@geindex source (pdb command)
@anchor{library/pdb pdbcommand-source}@anchor{3a74}
@deffn {Pdbcommand} source expression

Try to get source code for the given object and display it.


New in version 3.2.

@end deffn

@geindex display (pdb command)
@anchor{library/pdb pdbcommand-display}@anchor{3a75}
@deffn {Pdbcommand} display [expression]

Display the value of the expression if it changed, each time execution stops
in the current frame.

Without expression, list all display expressions for the current frame.


New in version 3.2.

@end deffn

@geindex undisplay (pdb command)
@anchor{library/pdb pdbcommand-undisplay}@anchor{3a76}
@deffn {Pdbcommand} undisplay [expression]

Do not display the expression any more in the current frame.  Without
expression, clear all display expressions for the current frame.


New in version 3.2.

@end deffn

@geindex interact (pdb command)
@anchor{library/pdb pdbcommand-interact}@anchor{3a77}
@deffn {Pdbcommand} interact

Start an interactive interpreter (using the @ref{19,,code} module) whose global
namespace contains all the (global and local) names found in the current
scope.


New in version 3.2.

@end deffn
@anchor{library/pdb debugger-aliases}@anchor{3a5f}
@geindex alias (pdb command)
@anchor{library/pdb pdbcommand-alias}@anchor{3a78}
@deffn {Pdbcommand} alias [name [command]]

Create an alias called @emph{name} that executes @emph{command}.  The command must
@emph{not} be enclosed in quotes.  Replaceable parameters can be indicated by
@code{%1}, @code{%2}, and so on, while @code{%*} is replaced by all the parameters.
If no command is given, the current alias for @emph{name} is shown. If no
arguments are given, all aliases are listed.

Aliases may be nested and can contain anything that can be legally typed at
the pdb prompt.  Note that internal pdb commands @emph{can} be overridden by
aliases.  Such a command is then hidden until the alias is removed.  Aliasing
is recursively applied to the first word of the command line; all other words
in the line are left alone.

As an example, here are two useful aliases (especially when placed in the
@code{.pdbrc} file):

@example
# Print instance variables (usage "pi classInst")
alias pi for k in %1.__dict__.keys(): print("%1.",k,"=",%1.__dict__[k])
# Print instance variables in self
alias ps pi self
@end example
@end deffn

@geindex unalias (pdb command)
@anchor{library/pdb pdbcommand-unalias}@anchor{3a79}
@deffn {Pdbcommand} unalias name

Delete the specified alias.
@end deffn

@geindex ! (pdb command)
@anchor{library/pdb pdbcommand-0}@anchor{3a7a}@anchor{library/pdb pdbcommand-!}@anchor{3a7b}
@deffn {Pdbcommand} ! statement

Execute the (one-line) @emph{statement} in the context of the current stack frame.
The exclamation point can be omitted unless the first word of the statement
resembles a debugger command.  To set a global variable, you can prefix the
assignment command with a @ref{1504,,global} statement on the same line,
e.g.:

@example
(Pdb) global list_options; list_options = ['-l']
(Pdb)
@end example
@end deffn

@geindex run (pdb command)
@geindex restart (pdb command)
@anchor{library/pdb pdbcommand-run}@anchor{3a7c}
@deffn {Pdbcommand} run [args ...]
@anchor{library/pdb pdbcommand-restart}@anchor{3a7d}
@deffnx {Pdbcommand} restart [args ...]

Restart the debugged Python program.  If an argument is supplied, it is split
with @ref{e3,,shlex} and the result is used as the new @ref{e15,,sys.argv}.
History, breakpoints, actions and debugger options are preserved.
@ref{3a7d,,restart} is an alias for @ref{3a7c,,run}.
@end deffn

@geindex quit (pdb command)
@anchor{library/pdb pdbcommand-quit}@anchor{3a6e}
@deffn {Pdbcommand} q(uit)

Quit from the debugger.  The program being executed is aborted.
@end deffn

@geindex debug (pdb command)
@anchor{library/pdb pdbcommand-debug}@anchor{3a7e}
@deffn {Pdbcommand} debug code

Enter a recursive debugger that steps through the code
argument (which is an arbitrary expression or statement to be
executed in the current environment).
@end deffn

@geindex retval (pdb command)
@anchor{library/pdb pdbcommand-retval}@anchor{3a7f}
@deffn {Pdbcommand} retval

Print the return value for the last return of a function.
@end deffn

@node The Python Profilers,timeit — Measure execution time of small code snippets,pdb — The Python Debugger,Debugging and Profiling
@anchor{library/profile doc}@anchor{3a80}@anchor{library/profile profile}@anchor{3a81}@anchor{library/profile the-python-profilers}@anchor{3a82}
@subsection The Python Profilers


@strong{Source code:} Lib/profile.py@footnote{https://github.com/python/cpython/tree/main/Lib/profile.py} and Lib/pstats.py@footnote{https://github.com/python/cpython/tree/main/Lib/pstats.py}


__________________________________________________________________


@menu
* Introduction to the profilers:: 
* Instant User’s Manual:: 
* profile and cProfile Module Reference:: 
* The Stats Class:: 
* What Is Deterministic Profiling?:: 
* Limitations:: 
* Calibration:: 
* Using a custom timer:: 

@end menu

@node Introduction to the profilers,Instant User’s Manual,,The Python Profilers
@anchor{library/profile introduction-to-the-profilers}@anchor{3a83}@anchor{library/profile profiler-introduction}@anchor{3a84}
@subsubsection Introduction to the profilers


@geindex deterministic profiling
@geindex profiling@comma{} deterministic

@ref{26,,cProfile} and @ref{ce,,profile} provide @emph{deterministic profiling} of
Python programs. A @emph{profile} is a set of statistics that describes how
often and for how long various parts of the program executed. These statistics
can be formatted into reports via the @ref{cf,,pstats} module.

The Python standard library provides two different implementations of the same
profiling interface:


@enumerate 

@item 
@ref{26,,cProfile} is recommended for most users; it’s a C extension with
reasonable overhead that makes it suitable for profiling long-running
programs.  Based on @code{lsprof}, contributed by Brett Rosen and Ted
Czotter.

@item 
@ref{ce,,profile}, a pure Python module whose interface is imitated by
@ref{26,,cProfile}, but which adds significant overhead to profiled programs.
If you’re trying to extend the profiler in some way, the task might be easier
with this module.  Originally designed and written by Jim Roskind.
@end enumerate

@cartouche
@quotation Note 
The profiler modules are designed to provide an execution profile for a given
program, not for benchmarking purposes (for that, there is @ref{10b,,timeit} for
reasonably accurate results).  This particularly applies to benchmarking
Python code against C code: the profilers introduce overhead for Python code,
but not for C-level functions, and so the C code would seem faster than any
Python one.
@end quotation
@end cartouche

@node Instant User’s Manual,profile and cProfile Module Reference,Introduction to the profilers,The Python Profilers
@anchor{library/profile instant-user-s-manual}@anchor{3a85}@anchor{library/profile profile-instant}@anchor{3a86}
@subsubsection Instant User’s Manual


This section is provided for users that “don’t want to read the manual.” It
provides a very brief overview, and allows a user to rapidly perform profiling
on an existing application.

To profile a function that takes a single argument, you can do:

@example
import cProfile
import re
cProfile.run('re.compile("foo|bar")')
@end example

(Use @ref{ce,,profile} instead of @ref{26,,cProfile} if the latter is not available on
your system.)

The above action would run @ref{70b,,re.compile()} and print profile results like
the following:

@example
      197 function calls (192 primitive calls) in 0.002 seconds

Ordered by: standard name

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.000    0.000    0.001    0.001 <string>:1(<module>)
     1    0.000    0.000    0.001    0.001 re.py:212(compile)
     1    0.000    0.000    0.001    0.001 re.py:268(_compile)
     1    0.000    0.000    0.000    0.000 sre_compile.py:172(_compile_charset)
     1    0.000    0.000    0.000    0.000 sre_compile.py:201(_optimize_charset)
     4    0.000    0.000    0.000    0.000 sre_compile.py:25(_identityfunction)
   3/1    0.000    0.000    0.000    0.000 sre_compile.py:33(_compile)
@end example

The first line indicates that 197 calls were monitored.  Of those calls, 192
were @emph{primitive}, meaning that the call was not induced via recursion. The
next line: @code{Ordered by: standard name}, indicates that the text string in the
far right column was used to sort the output. The column headings include:


@table @asis

@item ncalls

for the number of calls.

@item tottime

for the total time spent in the given function (and excluding time made in
calls to sub-functions)

@item percall

is the quotient of @code{tottime} divided by @code{ncalls}

@item cumtime

is the cumulative time spent in this and all subfunctions (from invocation
till exit). This figure is accurate @emph{even} for recursive functions.

@item percall

is the quotient of @code{cumtime} divided by primitive calls

@item filename:lineno(function)

provides the respective data of each function
@end table

When there are two numbers in the first column (for example @code{3/1}), it means
that the function recursed.  The second value is the number of primitive calls
and the former is the total number of calls.  Note that when the function does
not recurse, these two values are the same, and only the single figure is
printed.

Instead of printing the output at the end of the profile run, you can save the
results to a file by specifying a filename to the @code{run()} function:

@example
import cProfile
import re
cProfile.run('re.compile("foo|bar")', 'restats')
@end example

The @ref{3a87,,pstats.Stats} class reads profile results from a file and formats
them in various ways.

The files @ref{26,,cProfile} and @ref{ce,,profile} can also be invoked as a script to
profile another script.  For example:

@example
python -m cProfile [-o output_file] [-s sort_order] (-m module | myscript.py)
@end example

@code{-o} writes the profile results to a file instead of to stdout

@code{-s} specifies one of the @ref{3a88,,sort_stats()} sort values to sort
the output by. This only applies when @code{-o} is not supplied.

@code{-m} specifies that a module is being profiled instead of a script.

@quotation


New in version 3.7: Added the @code{-m} option to @ref{26,,cProfile}.



New in version 3.8: Added the @code{-m} option to @ref{ce,,profile}.

@end quotation

The @ref{cf,,pstats} module’s @ref{3a87,,Stats} class has a variety of methods
for manipulating and printing the data saved into a profile results file:

@example
import pstats
from pstats import SortKey
p = pstats.Stats('restats')
p.strip_dirs().sort_stats(-1).print_stats()
@end example

The @ref{3a89,,strip_dirs()} method removed the extraneous path from all
the module names. The @ref{3a88,,sort_stats()} method sorted all the
entries according to the standard module/line/name string that is printed. The
@ref{3a8a,,print_stats()} method printed out all the statistics.  You
might try the following sort calls:

@example
p.sort_stats(SortKey.NAME)
p.print_stats()
@end example

The first call will actually sort the list by function name, and the second call
will print out the statistics.  The following are some interesting calls to
experiment with:

@example
p.sort_stats(SortKey.CUMULATIVE).print_stats(10)
@end example

This sorts the profile by cumulative time in a function, and then only prints
the ten most significant lines.  If you want to understand what algorithms are
taking time, the above line is what you would use.

If you were looking to see what functions were looping a lot, and taking a lot
of time, you would do:

@example
p.sort_stats(SortKey.TIME).print_stats(10)
@end example

to sort according to time spent within each function, and then print the
statistics for the top ten functions.

You might also try:

@example
p.sort_stats(SortKey.FILENAME).print_stats('__init__')
@end example

This will sort all the statistics by file name, and then print out statistics
for only the class init methods (since they are spelled with @code{__init__} in
them).  As one final example, you could try:

@example
p.sort_stats(SortKey.TIME, SortKey.CUMULATIVE).print_stats(.5, 'init')
@end example

This line sorts statistics with a primary key of time, and a secondary key of
cumulative time, and then prints out some of the statistics. To be specific, the
list is first culled down to 50% (re: @code{.5}) of its original size, then only
lines containing @code{init} are maintained, and that sub-sub-list is printed.

If you wondered what functions called the above functions, you could now (@code{p}
is still sorted according to the last criteria) do:

@example
p.print_callers(.5, 'init')
@end example

and you would get a list of callers for each of the listed functions.

If you want more functionality, you’re going to have to read the manual, or
guess what the following functions do:

@example
p.print_callees()
p.add('restats')
@end example

Invoked as a script, the @ref{cf,,pstats} module is a statistics browser for
reading and examining profile dumps.  It has a simple line-oriented interface
(implemented using @ref{18,,cmd}) and interactive help.

@node profile and cProfile Module Reference,The Stats Class,Instant User’s Manual,The Python Profilers
@anchor{library/profile module-cProfile}@anchor{26}@anchor{library/profile profile-and-cprofile-module-reference}@anchor{3a8b}
@subsubsection @code{profile} and @code{cProfile} Module Reference


@geindex module; cProfile
@anchor{library/profile module-profile}@anchor{ce}
@geindex module; profile

Both the @ref{ce,,profile} and @ref{26,,cProfile} modules provide the following
functions:

@geindex run() (in module profile)
@anchor{library/profile profile run}@anchor{3a8c}
@deffn {Function} profile.run (command, filename=None, sort=- 1)

This function takes a single argument that can be passed to the @ref{34b,,exec()}
function, and an optional file name.  In all cases this routine executes:

@example
exec(command, __main__.__dict__, __main__.__dict__)
@end example

and gathers profiling statistics from the execution. If no file name is
present, then this function automatically creates a @ref{3a87,,Stats}
instance and prints a simple profiling report. If the sort value is specified,
it is passed to this @ref{3a87,,Stats} instance to control how the
results are sorted.
@end deffn

@geindex runctx() (in module profile)
@anchor{library/profile profile runctx}@anchor{3a8d}
@deffn {Function} profile.runctx (command, globals, locals, filename=None, sort=- 1)

This function is similar to @ref{3a8c,,run()}, with added arguments to supply the
globals and locals dictionaries for the @emph{command} string. This routine
executes:

@example
exec(command, globals, locals)
@end example

and gathers profiling statistics as in the @ref{3a8c,,run()} function above.
@end deffn

@geindex Profile (class in profile)
@anchor{library/profile profile Profile}@anchor{4e7}
@deffn {Class} profile.Profile (timer=None, timeunit=0.0, subcalls=True, builtins=True)

This class is normally only used if more precise control over profiling is
needed than what the @code{cProfile.run()} function provides.

A custom timer can be supplied for measuring how long code takes to run via
the @emph{timer} argument. This must be a function that returns a single number
representing the current time. If the number is an integer, the @emph{timeunit}
specifies a multiplier that specifies the duration of each unit of time. For
example, if the timer returns times measured in thousands of seconds, the
time unit would be @code{.001}.

Directly using the @ref{4e7,,Profile} class allows formatting profile results
without writing the profile data to a file:

@example
import cProfile, pstats, io
from pstats import SortKey
pr = cProfile.Profile()
pr.enable()
# ... do something ...
pr.disable()
s = io.StringIO()
sortby = SortKey.CUMULATIVE
ps = pstats.Stats(pr, stream=s).sort_stats(sortby)
ps.print_stats()
print(s.getvalue())
@end example

The @ref{4e7,,Profile} class can also be used as a context manager (supported
only in @ref{26,,cProfile} module. see @ref{1810,,Context Manager Types}):

@example
import cProfile

with cProfile.Profile() as pr:
    # ... do something ...

pr.print_stats()
@end example


Changed in version 3.8: Added context manager support.


@geindex enable() (profile.Profile method)
@anchor{library/profile profile Profile enable}@anchor{3a8e}
@deffn {Method} enable ()

Start collecting profiling data. Only in @ref{26,,cProfile}.
@end deffn

@geindex disable() (profile.Profile method)
@anchor{library/profile profile Profile disable}@anchor{3a8f}
@deffn {Method} disable ()

Stop collecting profiling data. Only in @ref{26,,cProfile}.
@end deffn

@geindex create_stats() (profile.Profile method)
@anchor{library/profile profile Profile create_stats}@anchor{3a90}
@deffn {Method} create_stats ()

Stop collecting profiling data and record the results internally
as the current profile.
@end deffn

@geindex print_stats() (profile.Profile method)
@anchor{library/profile profile Profile print_stats}@anchor{3a91}
@deffn {Method} print_stats (sort=- 1)

Create a @ref{3a87,,Stats} object based on the current
profile and print the results to stdout.
@end deffn

@geindex dump_stats() (profile.Profile method)
@anchor{library/profile profile Profile dump_stats}@anchor{3a92}
@deffn {Method} dump_stats (filename)

Write the results of the current profile to @emph{filename}.
@end deffn

@geindex run() (profile.Profile method)
@anchor{library/profile profile Profile run}@anchor{3a93}
@deffn {Method} run (cmd)

Profile the cmd via @ref{34b,,exec()}.
@end deffn

@geindex runctx() (profile.Profile method)
@anchor{library/profile profile Profile runctx}@anchor{3a94}
@deffn {Method} runctx (cmd, globals, locals)

Profile the cmd via @ref{34b,,exec()} with the specified global and
local environment.
@end deffn

@geindex runcall() (profile.Profile method)
@anchor{library/profile profile Profile runcall}@anchor{595}
@deffn {Method} runcall (func, /, *args, **kwargs)

Profile @code{func(*args, **kwargs)}
@end deffn
@end deffn

Note that profiling will only work if the called command/function actually
returns.  If the interpreter is terminated (e.g. via a @ref{ee4,,sys.exit()} call
during the called command/function execution) no profiling results will be
printed.

@node The Stats Class,What Is Deterministic Profiling?,profile and cProfile Module Reference,The Python Profilers
@anchor{library/profile profile-stats}@anchor{3a95}@anchor{library/profile the-stats-class}@anchor{3a96}
@subsubsection The @code{Stats} Class


Analysis of the profiler data is done using the @ref{3a87,,Stats} class.
@anchor{library/profile module-pstats}@anchor{cf}
@geindex module; pstats

@geindex Stats (class in pstats)
@anchor{library/profile pstats Stats}@anchor{3a87}
@deffn {Class} pstats.Stats (*filenames or profile, stream=sys.stdout)

This class constructor creates an instance of a “statistics object” from a
@emph{filename} (or list of filenames) or from a @code{Profile} instance. Output
will be printed to the stream specified by @emph{stream}.

The file selected by the above constructor must have been created by the
corresponding version of @ref{ce,,profile} or @ref{26,,cProfile}.  To be specific,
there is @emph{no} file compatibility guaranteed with future versions of this
profiler, and there is no compatibility with files produced by other
profilers, or the same profiler run on a different operating system.  If
several files are provided, all the statistics for identical functions will
be coalesced, so that an overall view of several processes can be considered
in a single report.  If additional files need to be combined with data in an
existing @ref{3a87,,Stats} object, the @ref{3a97,,add()} method
can be used.

Instead of reading the profile data from a file, a @code{cProfile.Profile}
or @ref{4e7,,profile.Profile} object can be used as the profile data source.

@ref{3a87,,Stats} objects have the following methods:

@geindex strip_dirs() (pstats.Stats method)
@anchor{library/profile pstats Stats strip_dirs}@anchor{3a89}
@deffn {Method} strip_dirs ()

This method for the @ref{3a87,,Stats} class removes all leading path
information from file names.  It is very useful in reducing the size of
the printout to fit within (close to) 80 columns.  This method modifies
the object, and the stripped information is lost.  After performing a
strip operation, the object is considered to have its entries in a
“random” order, as it was just after object initialization and loading.
If @ref{3a89,,strip_dirs()} causes two function names to be
indistinguishable (they are on the same line of the same filename, and
have the same function name), then the statistics for these two entries
are accumulated into a single entry.
@end deffn

@geindex add() (pstats.Stats method)
@anchor{library/profile pstats Stats add}@anchor{3a97}
@deffn {Method} add (*filenames)

This method of the @ref{3a87,,Stats} class accumulates additional profiling
information into the current profiling object.  Its arguments should refer
to filenames created by the corresponding version of @ref{3a8c,,profile.run()}
or @code{cProfile.run()}. Statistics for identically named (re: file, line,
name) functions are automatically accumulated into single function
statistics.
@end deffn

@geindex dump_stats() (pstats.Stats method)
@anchor{library/profile pstats Stats dump_stats}@anchor{3a98}
@deffn {Method} dump_stats (filename)

Save the data loaded into the @ref{3a87,,Stats} object to a file named
@emph{filename}.  The file is created if it does not exist, and is overwritten
if it already exists.  This is equivalent to the method of the same name
on the @ref{4e7,,profile.Profile} and @code{cProfile.Profile} classes.
@end deffn

@geindex sort_stats() (pstats.Stats method)
@anchor{library/profile pstats Stats sort_stats}@anchor{3a88}
@deffn {Method} sort_stats (*keys)

This method modifies the @ref{3a87,,Stats} object by sorting it according to
the supplied criteria.  The argument can be either a string or a SortKey
enum identifying the basis of a sort (example: @code{'time'}, @code{'name'},
@code{SortKey.TIME} or @code{SortKey.NAME}). The SortKey enums argument have
advantage over the string argument in that it is more robust and less
error prone.

When more than one key is provided, then additional keys are used as
secondary criteria when there is equality in all keys selected before
them.  For example, @code{sort_stats(SortKey.NAME, SortKey.FILE)} will sort
all the entries according to their function name, and resolve all ties
(identical function names) by sorting by file name.

For the string argument, abbreviations can be used for any key names, as
long as the abbreviation is unambiguous.

The following are the valid string and SortKey:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Valid String Arg

@tab

Valid enum Arg

@tab

Meaning

@item

@code{'calls'}

@tab

SortKey.CALLS

@tab

call count

@item

@code{'cumulative'}

@tab

SortKey.CUMULATIVE

@tab

cumulative time

@item

@code{'cumtime'}

@tab

N/A

@tab

cumulative time

@item

@code{'file'}

@tab

N/A

@tab

file name

@item

@code{'filename'}

@tab

SortKey.FILENAME

@tab

file name

@item

@code{'module'}

@tab

N/A

@tab

file name

@item

@code{'ncalls'}

@tab

N/A

@tab

call count

@item

@code{'pcalls'}

@tab

SortKey.PCALLS

@tab

primitive call count

@item

@code{'line'}

@tab

SortKey.LINE

@tab

line number

@item

@code{'name'}

@tab

SortKey.NAME

@tab

function name

@item

@code{'nfl'}

@tab

SortKey.NFL

@tab

name/file/line

@item

@code{'stdname'}

@tab

SortKey.STDNAME

@tab

standard name

@item

@code{'time'}

@tab

SortKey.TIME

@tab

internal time

@item

@code{'tottime'}

@tab

N/A

@tab

internal time

@end multitable


Note that all sorts on statistics are in descending order (placing most
time consuming items first), where as name, file, and line number searches
are in ascending order (alphabetical). The subtle distinction between
@code{SortKey.NFL} and @code{SortKey.STDNAME} is that the standard name is a
sort of the name as printed, which means that the embedded line numbers
get compared in an odd way.  For example, lines 3, 20, and 40 would (if
the file names were the same) appear in the string order 20, 3 and 40.
In contrast, @code{SortKey.NFL} does a numeric compare of the line numbers.
In fact, @code{sort_stats(SortKey.NFL)} is the same as
@code{sort_stats(SortKey.NAME, SortKey.FILENAME, SortKey.LINE)}.

For backward-compatibility reasons, the numeric arguments @code{-1}, @code{0},
@code{1}, and @code{2} are permitted.  They are interpreted as @code{'stdname'},
@code{'calls'}, @code{'time'}, and @code{'cumulative'} respectively.  If this old
style format (numeric) is used, only one sort key (the numeric key) will
be used, and additional arguments will be silently ignored.

@c For compatibility with the old profiler.


New in version 3.7: Added the SortKey enum.

@end deffn

@geindex reverse_order() (pstats.Stats method)
@anchor{library/profile pstats Stats reverse_order}@anchor{3a99}
@deffn {Method} reverse_order ()

This method for the @ref{3a87,,Stats} class reverses the ordering of the
basic list within the object.  Note that by default ascending vs
descending order is properly selected based on the sort key of choice.

@c This method is provided primarily for compatibility with the old
@c profiler.
@end deffn

@geindex print_stats() (pstats.Stats method)
@anchor{library/profile pstats Stats print_stats}@anchor{3a8a}
@deffn {Method} print_stats (*restrictions)

This method for the @ref{3a87,,Stats} class prints out a report as described
in the @ref{3a8c,,profile.run()} definition.

The order of the printing is based on the last
@ref{3a88,,sort_stats()} operation done on the object (subject to
caveats in @ref{3a97,,add()} and
@ref{3a89,,strip_dirs()}).

The arguments provided (if any) can be used to limit the list down to the
significant entries.  Initially, the list is taken to be the complete set
of profiled functions.  Each restriction is either an integer (to select a
count of lines), or a decimal fraction between 0.0 and 1.0 inclusive (to
select a percentage of lines), or a string that will interpreted as a
regular expression (to pattern match the standard name that is printed).
If several restrictions are provided, then they are applied sequentially.
For example:

@example
print_stats(.1, 'foo:')
@end example

would first limit the printing to first 10% of list, and then only print
functions that were part of filename @code{.*foo:}.  In contrast, the
command:

@example
print_stats('foo:', .1)
@end example

would limit the list to all functions having file names @code{.*foo:},
and then proceed to only print the first 10% of them.
@end deffn

@geindex print_callers() (pstats.Stats method)
@anchor{library/profile pstats Stats print_callers}@anchor{3a9a}
@deffn {Method} print_callers (*restrictions)

This method for the @ref{3a87,,Stats} class prints a list of all functions
that called each function in the profiled database.  The ordering is
identical to that provided by @ref{3a8a,,print_stats()}, and the
definition of the restricting argument is also identical.  Each caller is
reported on its own line.  The format differs slightly depending on the
profiler that produced the stats:


@itemize *

@item 
With @ref{ce,,profile}, a number is shown in parentheses after each caller
to show how many times this specific call was made.  For convenience, a
second non-parenthesized number repeats the cumulative time spent in the
function at the right.

@item 
With @ref{26,,cProfile}, each caller is preceded by three numbers: the
number of times this specific call was made, and the total and
cumulative times spent in the current function while it was invoked by
this specific caller.
@end itemize
@end deffn

@geindex print_callees() (pstats.Stats method)
@anchor{library/profile pstats Stats print_callees}@anchor{3a9b}
@deffn {Method} print_callees (*restrictions)

This method for the @ref{3a87,,Stats} class prints a list of all function
that were called by the indicated function.  Aside from this reversal of
direction of calls (re: called vs was called by), the arguments and
ordering are identical to the @ref{3a9a,,print_callers()} method.
@end deffn

@geindex get_stats_profile() (pstats.Stats method)
@anchor{library/profile pstats Stats get_stats_profile}@anchor{3a9c}
@deffn {Method} get_stats_profile ()

This method returns an instance of StatsProfile, which contains a mapping
of function names to instances of FunctionProfile. Each FunctionProfile
instance holds information related to the function’s profile such as how
long the function took to run, how many times it was called, etc…


New in version 3.9: Added the following dataclasses: StatsProfile, FunctionProfile.
Added the following function: get_stats_profile.

@end deffn
@end deffn

@node What Is Deterministic Profiling?,Limitations,The Stats Class,The Python Profilers
@anchor{library/profile deterministic-profiling}@anchor{3a9d}@anchor{library/profile what-is-deterministic-profiling}@anchor{3a9e}
@subsubsection What Is Deterministic Profiling?


@emph{Deterministic profiling} is meant to reflect the fact that all @emph{function
call}, @emph{function return}, and @emph{exception} events are monitored, and precise
timings are made for the intervals between these events (during which time the
user’s code is executing).  In contrast, @emph{statistical profiling} (which is
not done by this module) randomly samples the effective instruction pointer, and
deduces where time is being spent.  The latter technique traditionally involves
less overhead (as the code does not need to be instrumented), but provides only
relative indications of where time is being spent.

In Python, since there is an interpreter active during execution, the presence
of instrumented code is not required in order to do deterministic profiling.
Python automatically provides a @emph{hook} (optional callback) for each event.
In addition, the interpreted nature of Python tends to add so much overhead to
execution, that deterministic profiling tends to only add small processing
overhead in typical applications.  The result is that deterministic profiling is
not that expensive, yet provides extensive run time statistics about the
execution of a Python program.

Call count statistics can be used to identify bugs in code (surprising counts),
and to identify possible inline-expansion points (high call counts).  Internal
time statistics can be used to identify “hot loops” that should be carefully
optimized.  Cumulative time statistics should be used to identify high level
errors in the selection of algorithms.  Note that the unusual handling of
cumulative times in this profiler allows statistics for recursive
implementations of algorithms to be directly compared to iterative
implementations.

@node Limitations,Calibration,What Is Deterministic Profiling?,The Python Profilers
@anchor{library/profile limitations}@anchor{3a9f}@anchor{library/profile profile-limitations}@anchor{3aa0}
@subsubsection Limitations


One limitation has to do with accuracy of timing information. There is a
fundamental problem with deterministic profilers involving accuracy.  The most
obvious restriction is that the underlying “clock” is only ticking at a rate
(typically) of about .001 seconds.  Hence no measurements will be more accurate
than the underlying clock.  If enough measurements are taken, then the “error”
will tend to average out. Unfortunately, removing this first error induces a
second source of error.

The second problem is that it “takes a while” from when an event is dispatched
until the profiler’s call to get the time actually @emph{gets} the state of the
clock.  Similarly, there is a certain lag when exiting the profiler event
handler from the time that the clock’s value was obtained (and then squirreled
away), until the user’s code is once again executing.  As a result, functions
that are called many times, or call many functions, will typically accumulate
this error. The error that accumulates in this fashion is typically less than
the accuracy of the clock (less than one clock tick), but it @emph{can} accumulate
and become very significant.

The problem is more important with @ref{ce,,profile} than with the lower-overhead
@ref{26,,cProfile}.  For this reason, @ref{ce,,profile} provides a means of
calibrating itself for a given platform so that this error can be
probabilistically (on the average) removed. After the profiler is calibrated, it
will be more accurate (in a least square sense), but it will sometimes produce
negative numbers (when call counts are exceptionally low, and the gods of
probability work against you :-). )  Do @emph{not} be alarmed by negative numbers in
the profile.  They should @emph{only} appear if you have calibrated your profiler,
and the results are actually better than without calibration.

@node Calibration,Using a custom timer,Limitations,The Python Profilers
@anchor{library/profile calibration}@anchor{3aa1}@anchor{library/profile profile-calibration}@anchor{3aa2}
@subsubsection Calibration


The profiler of the @ref{ce,,profile} module subtracts a constant from each event
handling time to compensate for the overhead of calling the time function, and
socking away the results.  By default, the constant is 0. The following
procedure can be used to obtain a better constant for a given platform (see
@ref{3aa0,,Limitations}).

@example
import profile
pr = profile.Profile()
for i in range(5):
    print(pr.calibrate(10000))
@end example

The method executes the number of Python calls given by the argument, directly
and again under the profiler, measuring the time for both. It then computes the
hidden overhead per profiler event, and returns that as a float.  For example,
on a 1.8Ghz Intel Core i5 running macOS, and using Python’s time.process_time() as
the timer, the magical number is about 4.04e-6.

The object of this exercise is to get a fairly consistent result. If your
computer is @emph{very} fast, or your timer function has poor resolution, you might
have to pass 100000, or even 1000000, to get consistent results.

When you have a consistent answer, there are three ways you can use it:

@example
import profile

# 1. Apply computed bias to all Profile instances created hereafter.
profile.Profile.bias = your_computed_bias

# 2. Apply computed bias to a specific Profile instance.
pr = profile.Profile()
pr.bias = your_computed_bias

# 3. Specify computed bias in instance constructor.
pr = profile.Profile(bias=your_computed_bias)
@end example

If you have a choice, you are better off choosing a smaller constant, and then
your results will “less often” show up as negative in profile statistics.

@node Using a custom timer,,Calibration,The Python Profilers
@anchor{library/profile profile-timers}@anchor{3aa3}@anchor{library/profile using-a-custom-timer}@anchor{3aa4}
@subsubsection Using a custom timer


If you want to change how current time is determined (for example, to force use
of wall-clock time or elapsed process time), pass the timing function you want
to the @code{Profile} class constructor:

@example
pr = profile.Profile(your_time_func)
@end example

The resulting profiler will then call @code{your_time_func}. Depending on whether
you are using @ref{4e7,,profile.Profile} or @code{cProfile.Profile},
@code{your_time_func}’s return value will be interpreted differently:


@table @asis

@item @ref{4e7,,profile.Profile}

@code{your_time_func} should return a single number, or a list of numbers whose
sum is the current time (like what @ref{ca8,,os.times()} returns).  If the
function returns a single time number, or the list of returned numbers has
length 2, then you will get an especially fast version of the dispatch
routine.

Be warned that you should calibrate the profiler class for the timer function
that you choose (see @ref{3aa2,,Calibration}).  For most machines, a timer
that returns a lone integer value will provide the best results in terms of
low overhead during profiling.  (@ref{ca8,,os.times()} is @emph{pretty} bad, as it
returns a tuple of floating point values).  If you want to substitute a
better timer in the cleanest fashion, derive a class and hardwire a
replacement dispatch method that best handles your timer call, along with the
appropriate calibration constant.

@item @code{cProfile.Profile}

@code{your_time_func} should return a single number.  If it returns integers,
you can also invoke the class constructor with a second argument specifying
the real duration of one unit of time.  For example, if
@code{your_integer_time_func} returns times measured in thousands of seconds,
you would construct the @code{Profile} instance as follows:

@example
pr = cProfile.Profile(your_integer_time_func, 0.001)
@end example

As the @code{cProfile.Profile} class cannot be calibrated, custom timer
functions should be used with care and should be as fast as possible.  For
the best results with a custom timer, it might be necessary to hard-code it
in the C source of the internal @code{_lsprof} module.
@end table

Python 3.3 adds several new functions in @ref{10a,,time} that can be used to make
precise measurements of process or wall-clock time. For example, see
@ref{5a0,,time.perf_counter()}.

@node timeit — Measure execution time of small code snippets,trace — Trace or track Python statement execution,The Python Profilers,Debugging and Profiling
@anchor{library/timeit doc}@anchor{3aa5}@anchor{library/timeit module-timeit}@anchor{10b}@anchor{library/timeit timeit-measure-execution-time-of-small-code-snippets}@anchor{3aa6}
@subsection @code{timeit} — Measure execution time of small code snippets


@geindex module; timeit

@strong{Source code:} Lib/timeit.py@footnote{https://github.com/python/cpython/tree/main/Lib/timeit.py}

@geindex Benchmarking
@geindex Performance


__________________________________________________________________


This module provides a simple way to time small bits of Python code. It has both
a @ref{3aa7,,Command-Line Interface} as well as a @ref{3aa8,,callable}
one.  It avoids a number of common traps for measuring execution times.
See also Tim Peters’ introduction to the “Algorithms” chapter in the second
edition of @emph{Python Cookbook}, published by O’Reilly.

@menu
* Basic Examples: Basic Examples<2>. 
* Python Interface:: 
* Command-Line Interface: Command-Line Interface<4>. 
* Examples: Examples<28>. 

@end menu

@node Basic Examples<2>,Python Interface,,timeit — Measure execution time of small code snippets
@anchor{library/timeit basic-examples}@anchor{3aa9}
@subsubsection Basic Examples


The following example shows how the @ref{3aa7,,Command-Line Interface}
can be used to compare three different expressions:

@example
$ python3 -m timeit '"-".join(str(n) for n in range(100))'
10000 loops, best of 5: 30.2 usec per loop
$ python3 -m timeit '"-".join([str(n) for n in range(100)])'
10000 loops, best of 5: 27.5 usec per loop
$ python3 -m timeit '"-".join(map(str, range(100)))'
10000 loops, best of 5: 23.2 usec per loop
@end example

This can be achieved from the @ref{3aa8,,Python Interface} with:

@example
>>> import timeit
>>> timeit.timeit('"-".join(str(n) for n in range(100))', number=10000)
0.3018611848820001
>>> timeit.timeit('"-".join([str(n) for n in range(100)])', number=10000)
0.2727368790656328
>>> timeit.timeit('"-".join(map(str, range(100)))', number=10000)
0.23702679807320237
@end example

A callable can also be passed from the @ref{3aa8,,Python Interface}:

@example
>>> timeit.timeit(lambda: "-".join(map(str, range(100))), number=10000)
0.19665591977536678
@end example

Note however that @ref{9ef,,timeit()} will automatically determine the number of
repetitions only when the command-line interface is used.  In the
@ref{3aaa,,Examples} section you can find more advanced examples.

@node Python Interface,Command-Line Interface<4>,Basic Examples<2>,timeit — Measure execution time of small code snippets
@anchor{library/timeit id1}@anchor{3aab}@anchor{library/timeit python-interface}@anchor{3aa8}
@subsubsection Python Interface


The module defines three convenience functions and a public class:

@geindex timeit() (in module timeit)
@anchor{library/timeit timeit timeit}@anchor{9ef}
@deffn {Function} timeit.timeit (stmt='pass', setup='pass', timer=<default timer>, number=1000000, globals=None)

Create a @ref{11c1,,Timer} instance with the given statement, @emph{setup} code and
@emph{timer} function and run its @ref{837,,timeit()} method with @emph{number} executions.
The optional @emph{globals} argument specifies a namespace in which to execute the
code.


Changed in version 3.5: The optional @emph{globals} parameter was added.

@end deffn

@geindex repeat() (in module timeit)
@anchor{library/timeit timeit repeat}@anchor{3aac}
@deffn {Function} timeit.repeat (stmt='pass', setup='pass', timer=<default timer>, repeat=5, number=1000000, globals=None)

Create a @ref{11c1,,Timer} instance with the given statement, @emph{setup} code and
@emph{timer} function and run its @ref{3aad,,repeat()} method with the given @emph{repeat}
count and @emph{number} executions.  The optional @emph{globals} argument specifies a
namespace in which to execute the code.


Changed in version 3.5: The optional @emph{globals} parameter was added.



Changed in version 3.7: Default value of @emph{repeat} changed from 3 to 5.

@end deffn

@geindex default_timer() (in module timeit)
@anchor{library/timeit timeit default_timer}@anchor{3aae}
@deffn {Function} timeit.default_timer ()

The default timer, which is always @ref{5a0,,time.perf_counter()}.


Changed in version 3.3: @ref{5a0,,time.perf_counter()} is now the default timer.

@end deffn

@geindex Timer (class in timeit)
@anchor{library/timeit timeit Timer}@anchor{11c1}
@deffn {Class} timeit.Timer (stmt='pass', setup='pass', timer=<timer function>, globals=None)

Class for timing execution speed of small code snippets.

The constructor takes a statement to be timed, an additional statement used
for setup, and a timer function.  Both statements default to @code{'pass'};
the timer function is platform-dependent (see the module doc string).
@emph{stmt} and @emph{setup} may also contain multiple statements separated by @code{;}
or newlines, as long as they don’t contain multi-line string literals.  The
statement will by default be executed within timeit’s namespace; this behavior
can be controlled by passing a namespace to @emph{globals}.

To measure the execution time of the first statement, use the @ref{837,,timeit()}
method.  The @ref{3aad,,repeat()} and @ref{836,,autorange()} methods are convenience
methods to call @ref{837,,timeit()} multiple times.

The execution time of @emph{setup} is excluded from the overall timed execution run.

The @emph{stmt} and @emph{setup} parameters can also take objects that are callable
without arguments.  This will embed calls to them in a timer function that
will then be executed by @ref{837,,timeit()}.  Note that the timing overhead is a
little larger in this case because of the extra function calls.


Changed in version 3.5: The optional @emph{globals} parameter was added.


@geindex timeit() (timeit.Timer method)
@anchor{library/timeit timeit Timer timeit}@anchor{837}
@deffn {Method} timeit (number=1000000)

Time @emph{number} executions of the main statement.  This executes the setup
statement once, and then returns the time it takes to execute the main
statement a number of times, measured in seconds as a float.
The argument is the number of times through the loop, defaulting to one
million.  The main statement, the setup statement and the timer function
to be used are passed to the constructor.

@cartouche
@quotation Note 
By default, @ref{837,,timeit()} temporarily turns off @ref{1192,,garbage collection} during the timing.  The advantage of this approach is that
it makes independent timings more comparable.  The disadvantage is
that GC may be an important component of the performance of the
function being measured.  If so, GC can be re-enabled as the first
statement in the @emph{setup} string.  For example:

@example
timeit.Timer('for i in range(10): oct(i)', 'gc.enable()').timeit()
@end example
@end quotation
@end cartouche
@end deffn

@geindex autorange() (timeit.Timer method)
@anchor{library/timeit timeit Timer autorange}@anchor{836}
@deffn {Method} autorange (callback=None)

Automatically determine how many times to call @ref{837,,timeit()}.

This is a convenience function that calls @ref{837,,timeit()} repeatedly
so that the total time >= 0.2 second, returning the eventual
(number of loops, time taken for that number of loops). It calls
@ref{837,,timeit()} with increasing numbers from the sequence 1, 2, 5,
10, 20, 50, … until the time taken is at least 0.2 second.

If @emph{callback} is given and is not @code{None}, it will be called after
each trial with two arguments: @code{callback(number, time_taken)}.


New in version 3.6.

@end deffn

@geindex repeat() (timeit.Timer method)
@anchor{library/timeit timeit Timer repeat}@anchor{3aad}
@deffn {Method} repeat (repeat=5, number=1000000)

Call @ref{837,,timeit()} a few times.

This is a convenience function that calls the @ref{837,,timeit()} repeatedly,
returning a list of results.  The first argument specifies how many times
to call @ref{837,,timeit()}.  The second argument specifies the @emph{number}
argument for @ref{837,,timeit()}.

@cartouche
@quotation Note 
It’s tempting to calculate mean and standard deviation from the result
vector and report these.  However, this is not very useful.
In a typical case, the lowest value gives a lower bound for how fast
your machine can run the given code snippet; higher values in the
result vector are typically not caused by variability in Python’s
speed, but by other processes interfering with your timing accuracy.
So the @ref{a7d,,min()} of the result is probably the only number you
should be interested in.  After that, you should look at the entire
vector and apply common sense rather than statistics.
@end quotation
@end cartouche


Changed in version 3.7: Default value of @emph{repeat} changed from 3 to 5.

@end deffn

@geindex print_exc() (timeit.Timer method)
@anchor{library/timeit timeit Timer print_exc}@anchor{3aaf}
@deffn {Method} print_exc (file=None)

Helper to print a traceback from the timed code.

Typical use:

@example
t = Timer(...)       # outside the try/except
try:
    t.timeit(...)    # or t.repeat(...)
except Exception:
    t.print_exc()
@end example

The advantage over the standard traceback is that source lines in the
compiled template will be displayed.  The optional @emph{file} argument directs
where the traceback is sent; it defaults to @ref{42d,,sys.stderr}.
@end deffn
@end deffn

@node Command-Line Interface<4>,Examples<28>,Python Interface,timeit — Measure execution time of small code snippets
@anchor{library/timeit command-line-interface}@anchor{3ab0}@anchor{library/timeit timeit-command-line-interface}@anchor{3aa7}
@subsubsection Command-Line Interface


When called as a program from the command line, the following form is used:

@example
python -m timeit [-n N] [-r N] [-u U] [-s S] [-h] [statement ...]
@end example

Where the following options are understood:

@geindex timeit command line option; -n N
@geindex timeit command line option; --number=N
@anchor{library/timeit cmdoption-timeit-n}@anchor{3ab1}@anchor{library/timeit cmdoption-timeit-number}@anchor{3ab2}
@deffn {Program Option} @w{-}n N, @w{-}@w{-}number=N

how many times to execute ‘statement’
@end deffn

@geindex timeit command line option; -r N
@geindex timeit command line option; --repeat=N
@anchor{library/timeit cmdoption-timeit-r}@anchor{3ab3}@anchor{library/timeit cmdoption-timeit-repeat}@anchor{3ab4}
@deffn {Program Option} @w{-}r N, @w{-}@w{-}repeat=N

how many times to repeat the timer (default 5)
@end deffn

@geindex timeit command line option; -s S
@geindex timeit command line option; --setup=S
@anchor{library/timeit cmdoption-timeit-s}@anchor{3ab5}@anchor{library/timeit cmdoption-timeit-setup}@anchor{3ab6}
@deffn {Program Option} @w{-}s S, @w{-}@w{-}setup=S

statement to be executed once initially (default @code{pass})
@end deffn

@geindex timeit command line option; -p
@geindex timeit command line option; --process
@anchor{library/timeit cmdoption-timeit-p}@anchor{3ab7}@anchor{library/timeit cmdoption-timeit-process}@anchor{3ab8}
@deffn {Program Option} @w{-}p, @w{-}@w{-}process

measure process time, not wallclock time, using @ref{5a1,,time.process_time()}
instead of @ref{5a0,,time.perf_counter()}, which is the default


New in version 3.3.

@end deffn

@geindex timeit command line option; -u
@geindex timeit command line option; --unit=U
@anchor{library/timeit cmdoption-timeit-u}@anchor{3ab9}@anchor{library/timeit cmdoption-timeit-unit}@anchor{3aba}
@deffn {Program Option} @w{-}u, @w{-}@w{-}unit=U

specify a time unit for timer output; can select @code{nsec}, @code{usec}, @code{msec}, or @code{sec}


New in version 3.5.

@end deffn

@geindex timeit command line option; -v
@geindex timeit command line option; --verbose
@anchor{library/timeit cmdoption-timeit-v}@anchor{3abb}@anchor{library/timeit cmdoption-timeit-verbose}@anchor{3abc}
@deffn {Program Option} @w{-}v, @w{-}@w{-}verbose

print raw timing results; repeat for more digits precision
@end deffn

@geindex timeit command line option; -h
@geindex timeit command line option; --help
@anchor{library/timeit cmdoption-timeit-h}@anchor{3abd}@anchor{library/timeit cmdoption-timeit-help}@anchor{3abe}
@deffn {Program Option} @w{-}h, @w{-}@w{-}help

print a short usage message and exit
@end deffn

A multi-line statement may be given by specifying each line as a separate
statement argument; indented lines are possible by enclosing an argument in
quotes and using leading spaces.  Multiple @ref{3ab5,,-s} options are treated
similarly.

If @ref{3ab1,,-n} is not given, a suitable number of loops is calculated by trying
increasing numbers from the sequence 1, 2, 5, 10, 20, 50, … until the total
time is at least 0.2 seconds.

@ref{3aae,,default_timer()} measurements can be affected by other programs running on
the same machine, so the best thing to do when accurate timing is necessary is
to repeat the timing a few times and use the best time.  The @ref{3ab3,,-r}
option is good for this; the default of 5 repetitions is probably enough in
most cases.  You can use @ref{5a1,,time.process_time()} to measure CPU time.

@cartouche
@quotation Note 
There is a certain baseline overhead associated with executing a pass statement.
The code here doesn’t try to hide it, but you should be aware of it.  The
baseline overhead can be measured by invoking the program without arguments,
and it might differ between Python versions.
@end quotation
@end cartouche

@node Examples<28>,,Command-Line Interface<4>,timeit — Measure execution time of small code snippets
@anchor{library/timeit examples}@anchor{3abf}@anchor{library/timeit timeit-examples}@anchor{3aaa}
@subsubsection Examples


It is possible to provide a setup statement that is executed only once at the beginning:

@example
$ python -m timeit -s 'text = "sample string"; char = "g"'  'char in text'
5000000 loops, best of 5: 0.0877 usec per loop
$ python -m timeit -s 'text = "sample string"; char = "g"'  'text.find(char)'
1000000 loops, best of 5: 0.342 usec per loop
@end example

In the output, there are three fields. The loop count, which tells you how many
times the statement body was run per timing loop repetition. The repetition
count (‘best of 5’) which tells you how many times the timing loop was
repeated, and finally the time the statement body took on average within the
best repetition of the timing loop. That is, the time the fastest repetition
took divided by the loop count.

@example
>>> import timeit
>>> timeit.timeit('char in text', setup='text = "sample string"; char = "g"')
0.41440500499993504
>>> timeit.timeit('text.find(char)', setup='text = "sample string"; char = "g"')
1.7246671520006203
@end example

The same can be done using the @ref{11c1,,Timer} class and its methods:

@example
>>> import timeit
>>> t = timeit.Timer('char in text', setup='text = "sample string"; char = "g"')
>>> t.timeit()
0.3955516149999312
>>> t.repeat()
[0.40183617287970225, 0.37027556854118704, 0.38344867356679524, 0.3712595970846668, 0.37866875250654886]
@end example

The following examples show how to time expressions that contain multiple lines.
Here we compare the cost of using @ref{707,,hasattr()} vs. @ref{f6d,,try}/@ref{16e,,except}
to test for missing and present object attributes:

@example
$ python -m timeit 'try:' '  str.__bool__' 'except AttributeError:' '  pass'
20000 loops, best of 5: 15.7 usec per loop
$ python -m timeit 'if hasattr(str, "__bool__"): pass'
50000 loops, best of 5: 4.26 usec per loop

$ python -m timeit 'try:' '  int.__bool__' 'except AttributeError:' '  pass'
200000 loops, best of 5: 1.43 usec per loop
$ python -m timeit 'if hasattr(int, "__bool__"): pass'
100000 loops, best of 5: 2.23 usec per loop
@end example

@example
>>> import timeit
>>> # attribute is missing
>>> s = """\
... try:
...     str.__bool__
... except AttributeError:
...     pass
... """
>>> timeit.timeit(stmt=s, number=100000)
0.9138244460009446
>>> s = "if hasattr(str, '__bool__'): pass"
>>> timeit.timeit(stmt=s, number=100000)
0.5829014980008651
>>>
>>> # attribute is present
>>> s = """\
... try:
...     int.__bool__
... except AttributeError:
...     pass
... """
>>> timeit.timeit(stmt=s, number=100000)
0.04215312199994514
>>> s = "if hasattr(int, '__bool__'): pass"
>>> timeit.timeit(stmt=s, number=100000)
0.08588060699912603
@end example

To give the @ref{10b,,timeit} module access to functions you define, you can pass a
@emph{setup} parameter which contains an import statement:

@example
def test():
    """Stupid test function"""
    L = [i for i in range(100)]

if __name__ == '__main__':
    import timeit
    print(timeit.timeit("test()", setup="from __main__ import test"))
@end example

Another option is to pass @ref{10c2,,globals()} to the  @emph{globals} parameter, which will cause the code
to be executed within your current global namespace.  This can be more convenient
than individually specifying imports:

@example
def f(x):
    return x**2
def g(x):
    return x**4
def h(x):
    return x**8

import timeit
print(timeit.timeit('[func(42) for func in (f,g,h)]', globals=globals()))
@end example

@node trace — Trace or track Python statement execution,tracemalloc — Trace memory allocations,timeit — Measure execution time of small code snippets,Debugging and Profiling
@anchor{library/trace doc}@anchor{3ac0}@anchor{library/trace module-trace}@anchor{119}@anchor{library/trace trace-trace-or-track-python-statement-execution}@anchor{3ac1}
@subsection @code{trace} — Trace or track Python statement execution


@geindex module; trace

@strong{Source code:} Lib/trace.py@footnote{https://github.com/python/cpython/tree/main/Lib/trace.py}


__________________________________________________________________


The @ref{119,,trace} module allows you to trace program execution, generate
annotated statement coverage listings, print caller/callee relationships and
list functions executed during a program run.  It can be used in another program
or from the command line.


@subsubheading See also



@table @asis

@item Coverage.py@footnote{https://coverage.readthedocs.io/}

A popular third-party coverage tool that provides HTML
output along with advanced features such as branch coverage.
@end table


@menu
* Command-Line Usage:: 
* Programmatic Interface:: 

@end menu

@node Command-Line Usage,Programmatic Interface,,trace — Trace or track Python statement execution
@anchor{library/trace command-line-usage}@anchor{3ac2}@anchor{library/trace trace-cli}@anchor{3ac3}
@subsubsection Command-Line Usage


The @ref{119,,trace} module can be invoked from the command line.  It can be as
simple as

@example
python -m trace --count -C . somefile.py ...
@end example

The above will execute @code{somefile.py} and generate annotated listings of
all Python modules imported during the execution into the current directory.

@geindex trace command line option; --help
@anchor{library/trace cmdoption-trace-help}@anchor{3ac4}
@deffn {Program Option} @w{-}@w{-}help

Display usage and exit.
@end deffn

@geindex trace command line option; --version
@anchor{library/trace cmdoption-trace-version}@anchor{3ac5}
@deffn {Program Option} @w{-}@w{-}version

Display the version of the module and exit.
@end deffn


New in version 3.8: Added @code{--module} option that allows to run an executable module.


@menu
* Main options:: 
* Modifiers:: 
* Filters:: 

@end menu

@node Main options,Modifiers,,Command-Line Usage
@anchor{library/trace main-options}@anchor{3ac6}
@subsubsection Main options


At least one of the following options must be specified when invoking
@ref{119,,trace}.  The @ref{3ac7,,--listfuncs} option is mutually exclusive with
the @ref{3ac8,,--trace} and @ref{3ac9,,--count} options. When
@ref{3ac7,,--listfuncs} is provided, neither @ref{3ac9,,--count} nor
@ref{3ac8,,--trace} are accepted, and vice versa.

@geindex trace command line option; -c
@geindex trace command line option; --count
@anchor{library/trace cmdoption-trace-c}@anchor{3ac9}@anchor{library/trace cmdoption-trace-count}@anchor{3aca}
@deffn {Program Option} @w{-}c, @w{-}@w{-}count

Produce a set of annotated listing files upon program completion that shows
how many times each statement was executed.  See also
@ref{3acb,,--coverdir}, @ref{3acc,,--file} and
@ref{3acd,,--no-report} below.
@end deffn

@geindex trace command line option; -t
@geindex trace command line option; --trace
@anchor{library/trace cmdoption-trace-t}@anchor{3ac8}@anchor{library/trace cmdoption-trace-trace}@anchor{3ace}
@deffn {Program Option} @w{-}t, @w{-}@w{-}trace

Display lines as they are executed.
@end deffn

@geindex trace command line option; -l
@geindex trace command line option; --listfuncs
@anchor{library/trace cmdoption-trace-l}@anchor{3ac7}@anchor{library/trace cmdoption-trace-listfuncs}@anchor{3acf}
@deffn {Program Option} @w{-}l, @w{-}@w{-}listfuncs

Display the functions executed by running the program.
@end deffn

@geindex trace command line option; -r
@geindex trace command line option; --report
@anchor{library/trace cmdoption-trace-r}@anchor{3ad0}@anchor{library/trace cmdoption-trace-report}@anchor{3ad1}
@deffn {Program Option} @w{-}r, @w{-}@w{-}report

Produce an annotated list from an earlier program run that used the
@ref{3ac9,,--count} and @ref{3acc,,--file} option.  This does not
execute any code.
@end deffn

@geindex trace command line option; -T
@geindex trace command line option; --trackcalls
@anchor{library/trace cmdoption-trace-T}@anchor{3ad2}@anchor{library/trace cmdoption-trace-trackcalls}@anchor{3ad3}
@deffn {Program Option} @w{-}T, @w{-}@w{-}trackcalls

Display the calling relationships exposed by running the program.
@end deffn

@node Modifiers,Filters,Main options,Command-Line Usage
@anchor{library/trace modifiers}@anchor{3ad4}
@subsubsection Modifiers


@geindex trace command line option; -f
@geindex trace command line option; --file=<file>
@anchor{library/trace cmdoption-trace-f}@anchor{3acc}@anchor{library/trace cmdoption-trace-file}@anchor{3ad5}
@deffn {Program Option} @w{-}f, @w{-}@w{-}file=<file>

Name of a file to accumulate counts over several tracing runs.  Should be
used with the @ref{3ac9,,--count} option.
@end deffn

@geindex trace command line option; -C
@geindex trace command line option; --coverdir=<dir>
@anchor{library/trace cmdoption-trace-C}@anchor{3acb}@anchor{library/trace cmdoption-trace-coverdir}@anchor{3ad6}
@deffn {Program Option} @w{-}C, @w{-}@w{-}coverdir=<dir>

Directory where the report files go.  The coverage report for
@code{package.module} is written to file @code{@emph{dir}/@emph{package}/@emph{module}.cover}.
@end deffn

@geindex trace command line option; -m
@geindex trace command line option; --missing
@anchor{library/trace cmdoption-trace-m}@anchor{3ad7}@anchor{library/trace cmdoption-trace-missing}@anchor{3ad8}
@deffn {Program Option} @w{-}m, @w{-}@w{-}missing

When generating annotated listings, mark lines which were not executed with
@code{>>>>>>}.
@end deffn

@geindex trace command line option; -s
@geindex trace command line option; --summary
@anchor{library/trace cmdoption-trace-s}@anchor{3ad9}@anchor{library/trace cmdoption-trace-summary}@anchor{3ada}
@deffn {Program Option} @w{-}s, @w{-}@w{-}summary

When using @ref{3ac9,,--count} or @ref{3ad0,,--report}, write a brief
summary to stdout for each file processed.
@end deffn

@geindex trace command line option; -R
@geindex trace command line option; --no-report
@anchor{library/trace cmdoption-trace-R}@anchor{3acd}@anchor{library/trace cmdoption-trace-no-report}@anchor{3adb}
@deffn {Program Option} @w{-}R, @w{-}@w{-}no@w{-}report

Do not generate annotated listings.  This is useful if you intend to make
several runs with @ref{3ac9,,--count}, and then produce a single set of
annotated listings at the end.
@end deffn

@geindex trace command line option; -g
@geindex trace command line option; --timing
@anchor{library/trace cmdoption-trace-g}@anchor{3adc}@anchor{library/trace cmdoption-trace-timing}@anchor{3add}
@deffn {Program Option} @w{-}g, @w{-}@w{-}timing

Prefix each line with the time since the program started.  Only used while
tracing.
@end deffn

@node Filters,,Modifiers,Command-Line Usage
@anchor{library/trace filters}@anchor{3ade}
@subsubsection Filters


These options may be repeated multiple times.

@geindex trace command line option; --ignore-module=<mod>
@anchor{library/trace cmdoption-trace-ignore-module}@anchor{3adf}
@deffn {Program Option} @w{-}@w{-}ignore@w{-}module=<mod>

Ignore each of the given module names and its submodules (if it is a
package).  The argument can be a list of names separated by a comma.
@end deffn

@geindex trace command line option; --ignore-dir=<dir>
@anchor{library/trace cmdoption-trace-ignore-dir}@anchor{3ae0}
@deffn {Program Option} @w{-}@w{-}ignore@w{-}dir=<dir>

Ignore all modules and packages in the named directory and subdirectories.
The argument can be a list of directories separated by @ref{161d,,os.pathsep}.
@end deffn

@node Programmatic Interface,,Command-Line Usage,trace — Trace or track Python statement execution
@anchor{library/trace programmatic-interface}@anchor{3ae1}@anchor{library/trace trace-api}@anchor{3ae2}
@subsubsection Programmatic Interface


@geindex Trace (class in trace)
@anchor{library/trace trace Trace}@anchor{3ae3}
@deffn {Class} trace.Trace (count=1, trace=1, countfuncs=0, countcallers=0, ignoremods=(), ignoredirs=(), infile=None, outfile=None, timing=False)

Create an object to trace execution of a single statement or expression.  All
parameters are optional.  @emph{count} enables counting of line numbers.  @emph{trace}
enables line execution tracing.  @emph{countfuncs} enables listing of the
functions called during the run.  @emph{countcallers} enables call relationship
tracking.  @emph{ignoremods} is a list of modules or packages to ignore.
@emph{ignoredirs} is a list of directories whose modules or packages should be
ignored.  @emph{infile} is the name of the file from which to read stored count
information.  @emph{outfile} is the name of the file in which to write updated
count information.  @emph{timing} enables a timestamp relative to when tracing was
started to be displayed.

@geindex run() (trace.Trace method)
@anchor{library/trace trace Trace run}@anchor{3ae4}
@deffn {Method} run (cmd)

Execute the command and gather statistics from the execution with
the current tracing parameters.  @emph{cmd} must be a string or code object,
suitable for passing into @ref{34b,,exec()}.
@end deffn

@geindex runctx() (trace.Trace method)
@anchor{library/trace trace Trace runctx}@anchor{3ae5}
@deffn {Method} runctx (cmd, globals=None, locals=None)

Execute the command and gather statistics from the execution with the
current tracing parameters, in the defined global and local
environments.  If not defined, @emph{globals} and @emph{locals} default to empty
dictionaries.
@end deffn

@geindex runfunc() (trace.Trace method)
@anchor{library/trace trace Trace runfunc}@anchor{597}
@deffn {Method} runfunc (func, /, *args, **kwds)

Call @emph{func} with the given arguments under control of the @ref{3ae3,,Trace}
object with the current tracing parameters.
@end deffn

@geindex results() (trace.Trace method)
@anchor{library/trace trace Trace results}@anchor{3ae6}
@deffn {Method} results ()

Return a @ref{3ae7,,CoverageResults} object that contains the cumulative
results of all previous calls to @code{run}, @code{runctx} and @code{runfunc}
for the given @ref{3ae3,,Trace} instance.  Does not reset the accumulated
trace results.
@end deffn
@end deffn

@geindex CoverageResults (class in trace)
@anchor{library/trace trace CoverageResults}@anchor{3ae7}
@deffn {Class} trace.CoverageResults

A container for coverage results, created by @ref{3ae6,,Trace.results()}.  Should
not be created directly by the user.

@geindex update() (trace.CoverageResults method)
@anchor{library/trace trace CoverageResults update}@anchor{3ae8}
@deffn {Method} update (other)

Merge in data from another @ref{3ae7,,CoverageResults} object.
@end deffn

@geindex write_results() (trace.CoverageResults method)
@anchor{library/trace trace CoverageResults write_results}@anchor{3ae9}
@deffn {Method} write_results (show_missing=True, summary=False, coverdir=None)

Write coverage results.  Set @emph{show_missing} to show lines that had no
hits.  Set @emph{summary} to include in the output the coverage summary per
module.  @emph{coverdir} specifies the directory into which the coverage
result files will be output.  If @code{None}, the results for each source
file are placed in its directory.
@end deffn
@end deffn

A simple example demonstrating the use of the programmatic interface:

@example
import sys
import trace

# create a Trace object, telling it what to ignore, and whether to
# do tracing or line-counting or both.
tracer = trace.Trace(
    ignoredirs=[sys.prefix, sys.exec_prefix],
    trace=0,
    count=1)

# run the new command using the given tracer
tracer.run('main()')

# make a report, placing output in the current directory
r = tracer.results()
r.write_results(show_missing=True, coverdir=".")
@end example

@node tracemalloc — Trace memory allocations,,trace — Trace or track Python statement execution,Debugging and Profiling
@anchor{library/tracemalloc doc}@anchor{3aea}@anchor{library/tracemalloc module-tracemalloc}@anchor{11b}@anchor{library/tracemalloc tracemalloc-trace-memory-allocations}@anchor{3aeb}
@subsection @code{tracemalloc} — Trace memory allocations


@geindex module; tracemalloc


New in version 3.4.


@strong{Source code:} Lib/tracemalloc.py@footnote{https://github.com/python/cpython/tree/main/Lib/tracemalloc.py}


__________________________________________________________________


The tracemalloc module is a debug tool to trace memory blocks allocated by
Python. It provides the following information:


@itemize *

@item 
Traceback where an object was allocated

@item 
Statistics on allocated memory blocks per filename and per line number:
total size, number and average size of allocated memory blocks

@item 
Compute the differences between two snapshots to detect memory leaks
@end itemize

To trace most memory blocks allocated by Python, the module should be started
as early as possible by setting the 
@geindex PYTHONTRACEMALLOC
@geindex environment variable; PYTHONTRACEMALLOC
@ref{1623,,PYTHONTRACEMALLOC} environment
variable to @code{1}, or by using @ref{159,,-X} @code{tracemalloc} command line
option. The @ref{1616,,tracemalloc.start()} function can be called at runtime to
start tracing Python memory allocations.

By default, a trace of an allocated memory block only stores the most recent
frame (1 frame). To store 25 frames at startup: set the
@geindex PYTHONTRACEMALLOC
@geindex environment variable; PYTHONTRACEMALLOC
@ref{1623,,PYTHONTRACEMALLOC} environment variable to @code{25}, or use the
@ref{159,,-X} @code{tracemalloc=25} command line option.

@menu
* Examples: Examples<29>. 
* API:: 

@end menu

@node Examples<29>,API,,tracemalloc — Trace memory allocations
@anchor{library/tracemalloc examples}@anchor{3aec}
@subsubsection Examples


@menu
* Display the top 10:: 
* Compute differences:: 
* Get the traceback of a memory block:: 
* Pretty top:: 

@end menu

@node Display the top 10,Compute differences,,Examples<29>
@anchor{library/tracemalloc display-the-top-10}@anchor{3aed}
@subsubsection Display the top 10


Display the 10 files allocating the most memory:

@example
import tracemalloc

tracemalloc.start()

# ... run your application ...

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')

print("[ Top 10 ]")
for stat in top_stats[:10]:
    print(stat)
@end example

Example of output of the Python test suite:

@example
[ Top 10 ]
<frozen importlib._bootstrap>:716: size=4855 KiB, count=39328, average=126 B
<frozen importlib._bootstrap>:284: size=521 KiB, count=3199, average=167 B
/usr/lib/python3.4/collections/__init__.py:368: size=244 KiB, count=2315, average=108 B
/usr/lib/python3.4/unittest/case.py:381: size=185 KiB, count=779, average=243 B
/usr/lib/python3.4/unittest/case.py:402: size=154 KiB, count=378, average=416 B
/usr/lib/python3.4/abc.py:133: size=88.7 KiB, count=347, average=262 B
<frozen importlib._bootstrap>:1446: size=70.4 KiB, count=911, average=79 B
<frozen importlib._bootstrap>:1454: size=52.0 KiB, count=25, average=2131 B
<string>:5: size=49.7 KiB, count=148, average=344 B
/usr/lib/python3.4/sysconfig.py:411: size=48.0 KiB, count=1, average=48.0 KiB
@end example

We can see that Python loaded @code{4855 KiB} data (bytecode and constants) from
modules and that the @ref{1c,,collections} module allocated @code{244 KiB} to build
@ref{3e4,,namedtuple} types.

See @ref{3aee,,Snapshot.statistics()} for more options.

@node Compute differences,Get the traceback of a memory block,Display the top 10,Examples<29>
@anchor{library/tracemalloc compute-differences}@anchor{3aef}
@subsubsection Compute differences


Take two snapshots and display the differences:

@example
import tracemalloc
tracemalloc.start()
# ... start your application ...

snapshot1 = tracemalloc.take_snapshot()
# ... call the function leaking memory ...
snapshot2 = tracemalloc.take_snapshot()

top_stats = snapshot2.compare_to(snapshot1, 'lineno')

print("[ Top 10 differences ]")
for stat in top_stats[:10]:
    print(stat)
@end example

Example of output before/after running some tests of the Python test suite:

@example
[ Top 10 differences ]
<frozen importlib._bootstrap>:716: size=8173 KiB (+4428 KiB), count=71332 (+39369), average=117 B
/usr/lib/python3.4/linecache.py:127: size=940 KiB (+940 KiB), count=8106 (+8106), average=119 B
/usr/lib/python3.4/unittest/case.py:571: size=298 KiB (+298 KiB), count=589 (+589), average=519 B
<frozen importlib._bootstrap>:284: size=1005 KiB (+166 KiB), count=7423 (+1526), average=139 B
/usr/lib/python3.4/mimetypes.py:217: size=112 KiB (+112 KiB), count=1334 (+1334), average=86 B
/usr/lib/python3.4/http/server.py:848: size=96.0 KiB (+96.0 KiB), count=1 (+1), average=96.0 KiB
/usr/lib/python3.4/inspect.py:1465: size=83.5 KiB (+83.5 KiB), count=109 (+109), average=784 B
/usr/lib/python3.4/unittest/mock.py:491: size=77.7 KiB (+77.7 KiB), count=143 (+143), average=557 B
/usr/lib/python3.4/urllib/parse.py:476: size=71.8 KiB (+71.8 KiB), count=969 (+969), average=76 B
/usr/lib/python3.4/contextlib.py:38: size=67.2 KiB (+67.2 KiB), count=126 (+126), average=546 B
@end example

We can see that Python has loaded @code{8173 KiB} of module data (bytecode and
constants), and that this is @code{4428 KiB} more than had been loaded before the
tests, when the previous snapshot was taken. Similarly, the @ref{a4,,linecache}
module has cached @code{940 KiB} of Python source code to format tracebacks, all
of it since the previous snapshot.

If the system has little free memory, snapshots can be written on disk using
the @ref{3af0,,Snapshot.dump()} method to analyze the snapshot offline. Then use the
@ref{3af1,,Snapshot.load()} method reload the snapshot.

@node Get the traceback of a memory block,Pretty top,Compute differences,Examples<29>
@anchor{library/tracemalloc get-the-traceback-of-a-memory-block}@anchor{3af2}
@subsubsection Get the traceback of a memory block


Code to display the traceback of the biggest memory block:

@example
import tracemalloc

# Store 25 frames
tracemalloc.start(25)

# ... run your application ...

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('traceback')

# pick the biggest memory block
stat = top_stats[0]
print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024))
for line in stat.traceback.format():
    print(line)
@end example

Example of output of the Python test suite (traceback limited to 25 frames):

@example
903 memory blocks: 870.1 KiB
  File "<frozen importlib._bootstrap>", line 716
  File "<frozen importlib._bootstrap>", line 1036
  File "<frozen importlib._bootstrap>", line 934
  File "<frozen importlib._bootstrap>", line 1068
  File "<frozen importlib._bootstrap>", line 619
  File "<frozen importlib._bootstrap>", line 1581
  File "<frozen importlib._bootstrap>", line 1614
  File "/usr/lib/python3.4/doctest.py", line 101
    import pdb
  File "<frozen importlib._bootstrap>", line 284
  File "<frozen importlib._bootstrap>", line 938
  File "<frozen importlib._bootstrap>", line 1068
  File "<frozen importlib._bootstrap>", line 619
  File "<frozen importlib._bootstrap>", line 1581
  File "<frozen importlib._bootstrap>", line 1614
  File "/usr/lib/python3.4/test/support/__init__.py", line 1728
    import doctest
  File "/usr/lib/python3.4/test/test_pickletools.py", line 21
    support.run_doctest(pickletools)
  File "/usr/lib/python3.4/test/regrtest.py", line 1276
    test_runner()
  File "/usr/lib/python3.4/test/regrtest.py", line 976
    display_failure=not verbose)
  File "/usr/lib/python3.4/test/regrtest.py", line 761
    match_tests=ns.match_tests)
  File "/usr/lib/python3.4/test/regrtest.py", line 1563
    main()
  File "/usr/lib/python3.4/test/__main__.py", line 3
    regrtest.main_in_temp_cwd()
  File "/usr/lib/python3.4/runpy.py", line 73
    exec(code, run_globals)
  File "/usr/lib/python3.4/runpy.py", line 160
    "__main__", fname, loader, pkg_name)
@end example

We can see that the most memory was allocated in the @ref{96,,importlib} module to
load data (bytecode and constants) from modules: @code{870.1 KiB}. The traceback is
where the @ref{96,,importlib} loaded data most recently: on the @code{import pdb}
line of the @ref{63,,doctest} module. The traceback may change if a new module is
loaded.

@node Pretty top,,Get the traceback of a memory block,Examples<29>
@anchor{library/tracemalloc pretty-top}@anchor{3af3}
@subsubsection Pretty top


Code to display the 10 lines allocating the most memory with a pretty output,
ignoring @code{<frozen importlib._bootstrap>} and @code{<unknown>} files:

@example
import linecache
import os
import tracemalloc

def display_top(snapshot, key_type='lineno', limit=10):
    snapshot = snapshot.filter_traces((
        tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
        tracemalloc.Filter(False, "<unknown>"),
    ))
    top_stats = snapshot.statistics(key_type)

    print("Top %s lines" % limit)
    for index, stat in enumerate(top_stats[:limit], 1):
        frame = stat.traceback[0]
        print("#%s: %s:%s: %.1f KiB"
              % (index, frame.filename, frame.lineno, stat.size / 1024))
        line = linecache.getline(frame.filename, frame.lineno).strip()
        if line:
            print('    %s' % line)

    other = top_stats[limit:]
    if other:
        size = sum(stat.size for stat in other)
        print("%s other: %.1f KiB" % (len(other), size / 1024))
    total = sum(stat.size for stat in top_stats)
    print("Total allocated size: %.1f KiB" % (total / 1024))

tracemalloc.start()

# ... run your application ...

snapshot = tracemalloc.take_snapshot()
display_top(snapshot)
@end example

Example of output of the Python test suite:

@example
Top 10 lines
#1: Lib/base64.py:414: 419.8 KiB
    _b85chars2 = [(a + b) for a in _b85chars for b in _b85chars]
#2: Lib/base64.py:306: 419.8 KiB
    _a85chars2 = [(a + b) for a in _a85chars for b in _a85chars]
#3: collections/__init__.py:368: 293.6 KiB
    exec(class_definition, namespace)
#4: Lib/abc.py:133: 115.2 KiB
    cls = super().__new__(mcls, name, bases, namespace)
#5: unittest/case.py:574: 103.1 KiB
    testMethod()
#6: Lib/linecache.py:127: 95.4 KiB
    lines = fp.readlines()
#7: urllib/parse.py:476: 71.8 KiB
    for a in _hexdig for b in _hexdig@}
#8: <string>:5: 62.0 KiB
#9: Lib/_weakrefset.py:37: 60.0 KiB
    self.data = set()
#10: Lib/base64.py:142: 59.8 KiB
    _b32tab2 = [a + b for a in _b32tab for b in _b32tab]
6220 other: 3602.8 KiB
Total allocated size: 5303.1 KiB
@end example

See @ref{3aee,,Snapshot.statistics()} for more options.

@menu
* Record the current and peak size of all traced memory blocks:: 

@end menu

@node Record the current and peak size of all traced memory blocks,,,Pretty top
@anchor{library/tracemalloc record-the-current-and-peak-size-of-all-traced-memory-blocks}@anchor{3af4}
@subsubsection Record the current and peak size of all traced memory blocks


The following code computes two sums like @code{0 + 1 + 2 + ...} inefficiently, by
creating a list of those numbers. This list consumes a lot of memory
temporarily. We can use @ref{3af5,,get_traced_memory()} and @ref{42f,,reset_peak()} to
observe the small memory usage after the sum is computed as well as the peak
memory usage during the computations:

@example
import tracemalloc

tracemalloc.start()

# Example code: compute a sum with a large temporary list
large_sum = sum(list(range(100000)))

first_size, first_peak = tracemalloc.get_traced_memory()

tracemalloc.reset_peak()

# Example code: compute a sum with a small temporary list
small_sum = sum(list(range(1000)))

second_size, second_peak = tracemalloc.get_traced_memory()

print(f"@{first_size=@}, @{first_peak=@}")
print(f"@{second_size=@}, @{second_peak=@}")
@end example

Output:

@example
first_size=664, first_peak=3592984
second_size=804, second_peak=29704
@end example

Using @ref{42f,,reset_peak()} ensured we could accurately record the peak during the
computation of @code{small_sum}, even though it is much smaller than the overall
peak size of memory blocks since the @ref{1616,,start()} call. Without the call to
@ref{42f,,reset_peak()}, @code{second_peak} would still be the peak from the
computation @code{large_sum} (that is, equal to @code{first_peak}). In this case,
both peaks are much higher than the final memory usage, and which suggests we
could optimise (by removing the unnecessary call to @ref{43a,,list}, and writing
@code{sum(range(...))}).

@node API,,Examples<29>,tracemalloc — Trace memory allocations
@anchor{library/tracemalloc api}@anchor{3af6}
@subsubsection API


@menu
* Functions: Functions<10>. 
* DomainFilter:: 
* Filter:: 
* Frame:: 
* Snapshot:: 
* Statistic:: 
* StatisticDiff:: 
* Trace:: 
* Traceback:: 

@end menu

@node Functions<10>,DomainFilter,,API
@anchor{library/tracemalloc functions}@anchor{3af7}
@subsubsection Functions


@geindex clear_traces() (in module tracemalloc)
@anchor{library/tracemalloc tracemalloc clear_traces}@anchor{3af8}
@deffn {Function} tracemalloc.clear_traces ()

Clear traces of memory blocks allocated by Python.

See also @ref{3af9,,stop()}.
@end deffn

@geindex get_object_traceback() (in module tracemalloc)
@anchor{library/tracemalloc tracemalloc get_object_traceback}@anchor{1427}
@deffn {Function} tracemalloc.get_object_traceback (obj)

Get the traceback where the Python object @emph{obj} was allocated.
Return a @ref{6c7,,Traceback} instance, or @code{None} if the @ref{11b,,tracemalloc}
module is not tracing memory allocations or did not trace the allocation of
the object.

See also @ref{285,,gc.get_referrers()} and @ref{3afa,,sys.getsizeof()} functions.
@end deffn

@geindex get_traceback_limit() (in module tracemalloc)
@anchor{library/tracemalloc tracemalloc get_traceback_limit}@anchor{3afb}
@deffn {Function} tracemalloc.get_traceback_limit ()

Get the maximum number of frames stored in the traceback of a trace.

The @ref{11b,,tracemalloc} module must be tracing memory allocations to
get the limit, otherwise an exception is raised.

The limit is set by the @ref{1616,,start()} function.
@end deffn

@geindex get_traced_memory() (in module tracemalloc)
@anchor{library/tracemalloc tracemalloc get_traced_memory}@anchor{3af5}
@deffn {Function} tracemalloc.get_traced_memory ()

Get the current size and peak size of memory blocks traced by the
@ref{11b,,tracemalloc} module as a tuple: @code{(current: int, peak: int)}.
@end deffn

@geindex reset_peak() (in module tracemalloc)
@anchor{library/tracemalloc tracemalloc reset_peak}@anchor{42f}
@deffn {Function} tracemalloc.reset_peak ()

Set the peak size of memory blocks traced by the @ref{11b,,tracemalloc} module
to the current size.

Do nothing if the @ref{11b,,tracemalloc} module is not tracing memory
allocations.

This function only modifies the recorded peak size, and does not modify or
clear any traces, unlike @ref{3af8,,clear_traces()}. Snapshots taken with
@ref{3afc,,take_snapshot()} before a call to @ref{42f,,reset_peak()} can be
meaningfully compared to snapshots taken after the call.

See also @ref{3af5,,get_traced_memory()}.


New in version 3.9.

@end deffn

@geindex get_tracemalloc_memory() (in module tracemalloc)
@anchor{library/tracemalloc tracemalloc get_tracemalloc_memory}@anchor{3afd}
@deffn {Function} tracemalloc.get_tracemalloc_memory ()

Get the memory usage in bytes of the @ref{11b,,tracemalloc} module used to store
traces of memory blocks.
Return an @ref{1b1,,int}.
@end deffn

@geindex is_tracing() (in module tracemalloc)
@anchor{library/tracemalloc tracemalloc is_tracing}@anchor{3afe}
@deffn {Function} tracemalloc.is_tracing ()

@code{True} if the @ref{11b,,tracemalloc} module is tracing Python memory
allocations, @code{False} otherwise.

See also @ref{1616,,start()} and @ref{3af9,,stop()} functions.
@end deffn

@geindex start() (in module tracemalloc)
@anchor{library/tracemalloc tracemalloc start}@anchor{1616}
@deffn {Function} tracemalloc.start (nframe: int = 1)

Start tracing Python memory allocations: install hooks on Python memory
allocators. Collected tracebacks of traces will be limited to @emph{nframe}
frames. By default, a trace of a memory block only stores the most recent
frame: the limit is @code{1}. @emph{nframe} must be greater or equal to @code{1}.

You can still read the original number of total frames that composed the
traceback by looking at the @ref{3aff,,Traceback.total_nframe} attribute.

Storing more than @code{1} frame is only useful to compute statistics grouped
by @code{'traceback'} or to compute cumulative statistics: see the
@ref{3b00,,Snapshot.compare_to()} and @ref{3aee,,Snapshot.statistics()} methods.

Storing more frames increases the memory and CPU overhead of the
@ref{11b,,tracemalloc} module. Use the @ref{3afd,,get_tracemalloc_memory()} function
to measure how much memory is used by the @ref{11b,,tracemalloc} module.

The 
@geindex PYTHONTRACEMALLOC
@geindex environment variable; PYTHONTRACEMALLOC
@ref{1623,,PYTHONTRACEMALLOC} environment variable
(@code{PYTHONTRACEMALLOC=NFRAME}) and the @ref{159,,-X} @code{tracemalloc=NFRAME}
command line option can be used to start tracing at startup.

See also @ref{3af9,,stop()}, @ref{3afe,,is_tracing()} and @ref{3afb,,get_traceback_limit()}
functions.
@end deffn

@geindex stop() (in module tracemalloc)
@anchor{library/tracemalloc tracemalloc stop}@anchor{3af9}
@deffn {Function} tracemalloc.stop ()

Stop tracing Python memory allocations: uninstall hooks on Python memory
allocators. Also clears all previously collected traces of memory blocks
allocated by Python.

Call @ref{3afc,,take_snapshot()} function to take a snapshot of traces before
clearing them.

See also @ref{1616,,start()}, @ref{3afe,,is_tracing()} and @ref{3af8,,clear_traces()}
functions.
@end deffn

@geindex take_snapshot() (in module tracemalloc)
@anchor{library/tracemalloc tracemalloc take_snapshot}@anchor{3afc}
@deffn {Function} tracemalloc.take_snapshot ()

Take a snapshot of traces of memory blocks allocated by Python. Return a new
@ref{3b01,,Snapshot} instance.

The snapshot does not include memory blocks allocated before the
@ref{11b,,tracemalloc} module started to trace memory allocations.

Tracebacks of traces are limited to @ref{3afb,,get_traceback_limit()} frames. Use
the @emph{nframe} parameter of the @ref{1616,,start()} function to store more frames.

The @ref{11b,,tracemalloc} module must be tracing memory allocations to take a
snapshot, see the @ref{1616,,start()} function.

See also the @ref{1427,,get_object_traceback()} function.
@end deffn

@node DomainFilter,Filter,Functions<10>,API
@anchor{library/tracemalloc domainfilter}@anchor{3b02}
@subsubsection DomainFilter


@geindex DomainFilter (class in tracemalloc)
@anchor{library/tracemalloc tracemalloc DomainFilter}@anchor{83b}
@deffn {Class} tracemalloc.DomainFilter (inclusive: bool, domain: int)

Filter traces of memory blocks by their address space (domain).


New in version 3.6.


@geindex inclusive (tracemalloc.DomainFilter attribute)
@anchor{library/tracemalloc tracemalloc DomainFilter inclusive}@anchor{3b03}
@deffn {Attribute} inclusive

If @emph{inclusive} is @code{True} (include), match memory blocks allocated
in the address space @ref{3b04,,domain}.

If @emph{inclusive} is @code{False} (exclude), match memory blocks not allocated
in the address space @ref{3b04,,domain}.
@end deffn

@geindex domain (tracemalloc.DomainFilter attribute)
@anchor{library/tracemalloc tracemalloc DomainFilter domain}@anchor{3b04}
@deffn {Attribute} domain

Address space of a memory block (@code{int}). Read-only property.
@end deffn
@end deffn

@node Filter,Frame,DomainFilter,API
@anchor{library/tracemalloc filter}@anchor{3b05}
@subsubsection Filter


@geindex Filter (class in tracemalloc)
@anchor{library/tracemalloc tracemalloc Filter}@anchor{3b06}
@deffn {Class} tracemalloc.Filter (inclusive: bool, filename_pattern: str, lineno: int = None, all_frames: bool = False, domain: int = None)

Filter on traces of memory blocks.

See the @ref{1106,,fnmatch.fnmatch()} function for the syntax of
@emph{filename_pattern}. The @code{'.pyc'} file extension is
replaced with @code{'.py'}.

Examples:


@itemize *

@item 
@code{Filter(True, subprocess.__file__)} only includes traces of the
@ref{f4,,subprocess} module

@item 
@code{Filter(False, tracemalloc.__file__)} excludes traces of the
@ref{11b,,tracemalloc} module

@item 
@code{Filter(False, "<unknown>")} excludes empty tracebacks
@end itemize


Changed in version 3.5: The @code{'.pyo'} file extension is no longer replaced with @code{'.py'}.



Changed in version 3.6: Added the @ref{3b07,,domain} attribute.


@geindex domain (tracemalloc.Filter attribute)
@anchor{library/tracemalloc tracemalloc Filter domain}@anchor{3b07}
@deffn {Attribute} domain

Address space of a memory block (@code{int} or @code{None}).

tracemalloc uses the domain @code{0} to trace memory allocations made by
Python. C extensions can use other domains to trace other resources.
@end deffn

@geindex inclusive (tracemalloc.Filter attribute)
@anchor{library/tracemalloc tracemalloc Filter inclusive}@anchor{3b08}
@deffn {Attribute} inclusive

If @emph{inclusive} is @code{True} (include), only match memory blocks allocated
in a file with a name matching @ref{3b09,,filename_pattern} at line number
@ref{3b0a,,lineno}.

If @emph{inclusive} is @code{False} (exclude), ignore memory blocks allocated in
a file with a name matching @ref{3b09,,filename_pattern} at line number
@ref{3b0a,,lineno}.
@end deffn

@geindex lineno (tracemalloc.Filter attribute)
@anchor{library/tracemalloc tracemalloc Filter lineno}@anchor{3b0a}
@deffn {Attribute} lineno

Line number (@code{int}) of the filter. If @emph{lineno} is @code{None}, the filter
matches any line number.
@end deffn

@geindex filename_pattern (tracemalloc.Filter attribute)
@anchor{library/tracemalloc tracemalloc Filter filename_pattern}@anchor{3b09}
@deffn {Attribute} filename_pattern

Filename pattern of the filter (@code{str}). Read-only property.
@end deffn

@geindex all_frames (tracemalloc.Filter attribute)
@anchor{library/tracemalloc tracemalloc Filter all_frames}@anchor{3b0b}
@deffn {Attribute} all_frames

If @emph{all_frames} is @code{True}, all frames of the traceback are checked. If
@emph{all_frames} is @code{False}, only the most recent frame is checked.

This attribute has no effect if the traceback limit is @code{1}.  See the
@ref{3afb,,get_traceback_limit()} function and @ref{3b0c,,Snapshot.traceback_limit}
attribute.
@end deffn
@end deffn

@node Frame,Snapshot,Filter,API
@anchor{library/tracemalloc frame}@anchor{3b0d}
@subsubsection Frame


@geindex Frame (class in tracemalloc)
@anchor{library/tracemalloc tracemalloc Frame}@anchor{3b0e}
@deffn {Class} tracemalloc.Frame

Frame of a traceback.

The @ref{6c7,,Traceback} class is a sequence of @ref{3b0e,,Frame} instances.

@geindex filename (tracemalloc.Frame attribute)
@anchor{library/tracemalloc tracemalloc Frame filename}@anchor{3b0f}
@deffn {Attribute} filename

Filename (@code{str}).
@end deffn

@geindex lineno (tracemalloc.Frame attribute)
@anchor{library/tracemalloc tracemalloc Frame lineno}@anchor{3b10}
@deffn {Attribute} lineno

Line number (@code{int}).
@end deffn
@end deffn

@node Snapshot,Statistic,Frame,API
@anchor{library/tracemalloc snapshot}@anchor{3b11}
@subsubsection Snapshot


@geindex Snapshot (class in tracemalloc)
@anchor{library/tracemalloc tracemalloc Snapshot}@anchor{3b01}
@deffn {Class} tracemalloc.Snapshot

Snapshot of traces of memory blocks allocated by Python.

The @ref{3afc,,take_snapshot()} function creates a snapshot instance.

@geindex compare_to() (tracemalloc.Snapshot method)
@anchor{library/tracemalloc tracemalloc Snapshot compare_to}@anchor{3b00}
@deffn {Method} compare_to (old_snapshot: Snapshot, key_type: str, cumulative: bool = False)

Compute the differences with an old snapshot. Get statistics as a sorted
list of @ref{3b12,,StatisticDiff} instances grouped by @emph{key_type}.

See the @ref{3aee,,Snapshot.statistics()} method for @emph{key_type} and @emph{cumulative}
parameters.

The result is sorted from the biggest to the smallest by: absolute value
of @ref{3b13,,StatisticDiff.size_diff}, @ref{3b14,,StatisticDiff.size}, absolute
value of @ref{3b15,,StatisticDiff.count_diff}, @ref{3b16,,Statistic.count} and
then by @ref{3b17,,StatisticDiff.traceback}.
@end deffn

@geindex dump() (tracemalloc.Snapshot method)
@anchor{library/tracemalloc tracemalloc Snapshot dump}@anchor{3af0}
@deffn {Method} dump (filename)

Write the snapshot into a file.

Use @ref{3af1,,load()} to reload the snapshot.
@end deffn

@geindex filter_traces() (tracemalloc.Snapshot method)
@anchor{library/tracemalloc tracemalloc Snapshot filter_traces}@anchor{3b18}
@deffn {Method} filter_traces (filters)

Create a new @ref{3b01,,Snapshot} instance with a filtered @ref{3b19,,traces}
sequence, @emph{filters} is a list of @ref{83b,,DomainFilter} and
@ref{3b06,,Filter} instances.  If @emph{filters} is an empty list, return a new
@ref{3b01,,Snapshot} instance with a copy of the traces.

All inclusive filters are applied at once, a trace is ignored if no
inclusive filters match it. A trace is ignored if at least one exclusive
filter matches it.


Changed in version 3.6: @ref{83b,,DomainFilter} instances are now also accepted in @emph{filters}.

@end deffn

@geindex load() (tracemalloc.Snapshot class method)
@anchor{library/tracemalloc tracemalloc Snapshot load}@anchor{3af1}
@deffn {Method} classmethod load (filename)

Load a snapshot from a file.

See also @ref{3af0,,dump()}.
@end deffn

@geindex statistics() (tracemalloc.Snapshot method)
@anchor{library/tracemalloc tracemalloc Snapshot statistics}@anchor{3aee}
@deffn {Method} statistics (key_type: str, cumulative: bool = False)

Get statistics as a sorted list of @ref{3b1a,,Statistic} instances grouped
by @emph{key_type}:


@multitable {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

key_type

@tab

description

@item

@code{'filename'}

@tab

filename

@item

@code{'lineno'}

@tab

filename and line number

@item

@code{'traceback'}

@tab

traceback

@end multitable


If @emph{cumulative} is @code{True}, cumulate size and count of memory blocks of
all frames of the traceback of a trace, not only the most recent frame.
The cumulative mode can only be used with @emph{key_type} equals to
@code{'filename'} and @code{'lineno'}.

The result is sorted from the biggest to the smallest by:
@ref{3b1b,,Statistic.size}, @ref{3b16,,Statistic.count} and then by
@ref{3b1c,,Statistic.traceback}.
@end deffn

@geindex traceback_limit (tracemalloc.Snapshot attribute)
@anchor{library/tracemalloc tracemalloc Snapshot traceback_limit}@anchor{3b0c}
@deffn {Attribute} traceback_limit

Maximum number of frames stored in the traceback of @ref{3b19,,traces}:
result of the @ref{3afb,,get_traceback_limit()} when the snapshot was taken.
@end deffn

@geindex traces (tracemalloc.Snapshot attribute)
@anchor{library/tracemalloc tracemalloc Snapshot traces}@anchor{3b19}
@deffn {Attribute} traces

Traces of all memory blocks allocated by Python: sequence of
@ref{3b1d,,Trace} instances.

The sequence has an undefined order. Use the @ref{3aee,,Snapshot.statistics()}
method to get a sorted list of statistics.
@end deffn
@end deffn

@node Statistic,StatisticDiff,Snapshot,API
@anchor{library/tracemalloc statistic}@anchor{3b1e}
@subsubsection Statistic


@geindex Statistic (class in tracemalloc)
@anchor{library/tracemalloc tracemalloc Statistic}@anchor{3b1a}
@deffn {Class} tracemalloc.Statistic

Statistic on memory allocations.

@ref{3aee,,Snapshot.statistics()} returns a list of @ref{3b1a,,Statistic} instances.

See also the @ref{3b12,,StatisticDiff} class.

@geindex count (tracemalloc.Statistic attribute)
@anchor{library/tracemalloc tracemalloc Statistic count}@anchor{3b16}
@deffn {Attribute} count

Number of memory blocks (@code{int}).
@end deffn

@geindex size (tracemalloc.Statistic attribute)
@anchor{library/tracemalloc tracemalloc Statistic size}@anchor{3b1b}
@deffn {Attribute} size

Total size of memory blocks in bytes (@code{int}).
@end deffn

@geindex traceback (tracemalloc.Statistic attribute)
@anchor{library/tracemalloc tracemalloc Statistic traceback}@anchor{3b1c}
@deffn {Attribute} traceback

Traceback where the memory block was allocated, @ref{6c7,,Traceback}
instance.
@end deffn
@end deffn

@node StatisticDiff,Trace,Statistic,API
@anchor{library/tracemalloc statisticdiff}@anchor{3b1f}
@subsubsection StatisticDiff


@geindex StatisticDiff (class in tracemalloc)
@anchor{library/tracemalloc tracemalloc StatisticDiff}@anchor{3b12}
@deffn {Class} tracemalloc.StatisticDiff

Statistic difference on memory allocations between an old and a new
@ref{3b01,,Snapshot} instance.

@ref{3b00,,Snapshot.compare_to()} returns a list of @ref{3b12,,StatisticDiff}
instances. See also the @ref{3b1a,,Statistic} class.

@geindex count (tracemalloc.StatisticDiff attribute)
@anchor{library/tracemalloc tracemalloc StatisticDiff count}@anchor{3b20}
@deffn {Attribute} count

Number of memory blocks in the new snapshot (@code{int}): @code{0} if
the memory blocks have been released in the new snapshot.
@end deffn

@geindex count_diff (tracemalloc.StatisticDiff attribute)
@anchor{library/tracemalloc tracemalloc StatisticDiff count_diff}@anchor{3b15}
@deffn {Attribute} count_diff

Difference of number of memory blocks between the old and the new
snapshots (@code{int}): @code{0} if the memory blocks have been allocated in
the new snapshot.
@end deffn

@geindex size (tracemalloc.StatisticDiff attribute)
@anchor{library/tracemalloc tracemalloc StatisticDiff size}@anchor{3b14}
@deffn {Attribute} size

Total size of memory blocks in bytes in the new snapshot (@code{int}):
@code{0} if the memory blocks have been released in the new snapshot.
@end deffn

@geindex size_diff (tracemalloc.StatisticDiff attribute)
@anchor{library/tracemalloc tracemalloc StatisticDiff size_diff}@anchor{3b13}
@deffn {Attribute} size_diff

Difference of total size of memory blocks in bytes between the old and
the new snapshots (@code{int}): @code{0} if the memory blocks have been
allocated in the new snapshot.
@end deffn

@geindex traceback (tracemalloc.StatisticDiff attribute)
@anchor{library/tracemalloc tracemalloc StatisticDiff traceback}@anchor{3b17}
@deffn {Attribute} traceback

Traceback where the memory blocks were allocated, @ref{6c7,,Traceback}
instance.
@end deffn
@end deffn

@node Trace,Traceback,StatisticDiff,API
@anchor{library/tracemalloc trace}@anchor{3b21}
@subsubsection Trace


@geindex Trace (class in tracemalloc)
@anchor{library/tracemalloc tracemalloc Trace}@anchor{3b1d}
@deffn {Class} tracemalloc.Trace

Trace of a memory block.

The @ref{3b19,,Snapshot.traces} attribute is a sequence of @ref{3b1d,,Trace}
instances.


Changed in version 3.6: Added the @ref{3b22,,domain} attribute.


@geindex domain (tracemalloc.Trace attribute)
@anchor{library/tracemalloc tracemalloc Trace domain}@anchor{3b22}
@deffn {Attribute} domain

Address space of a memory block (@code{int}). Read-only property.

tracemalloc uses the domain @code{0} to trace memory allocations made by
Python. C extensions can use other domains to trace other resources.
@end deffn

@geindex size (tracemalloc.Trace attribute)
@anchor{library/tracemalloc tracemalloc Trace size}@anchor{3b23}
@deffn {Attribute} size

Size of the memory block in bytes (@code{int}).
@end deffn

@geindex traceback (tracemalloc.Trace attribute)
@anchor{library/tracemalloc tracemalloc Trace traceback}@anchor{3b24}
@deffn {Attribute} traceback

Traceback where the memory block was allocated, @ref{6c7,,Traceback}
instance.
@end deffn
@end deffn

@node Traceback,,Trace,API
@anchor{library/tracemalloc traceback}@anchor{3b25}
@subsubsection Traceback


@geindex Traceback (class in tracemalloc)
@anchor{library/tracemalloc tracemalloc Traceback}@anchor{6c7}
@deffn {Class} tracemalloc.Traceback

Sequence of @ref{3b0e,,Frame} instances sorted from the oldest frame to the
most recent frame.

A traceback contains at least @code{1} frame. If the @code{tracemalloc} module
failed to get a frame, the filename @code{"<unknown>"} at line number @code{0} is
used.

When a snapshot is taken, tracebacks of traces are limited to
@ref{3afb,,get_traceback_limit()} frames. See the @ref{3afc,,take_snapshot()} function.
The original number of frames of the traceback is stored in the
@ref{3aff,,Traceback.total_nframe} attribute. That allows to know if a traceback
has been truncated by the traceback limit.

The @ref{3b24,,Trace.traceback} attribute is an instance of @ref{6c7,,Traceback}
instance.


Changed in version 3.7: Frames are now sorted from the oldest to the most recent, instead of most recent to oldest.


@geindex total_nframe (tracemalloc.Traceback attribute)
@anchor{library/tracemalloc tracemalloc Traceback total_nframe}@anchor{3aff}
@deffn {Attribute} total_nframe

Total number of frames that composed the traceback before truncation.
This attribute can be set to @code{None} if the information is not
available.
@end deffn


Changed in version 3.9: The @ref{3aff,,Traceback.total_nframe} attribute was added.


@geindex format() (tracemalloc.Traceback method)
@anchor{library/tracemalloc tracemalloc Traceback format}@anchor{6c8}
@deffn {Method} format (limit=None, most_recent_first=False)

Format the traceback as a list of lines. Use the @ref{a4,,linecache} module to
retrieve lines from the source code. If @emph{limit} is set, format the @emph{limit}
most recent frames if @emph{limit} is positive. Otherwise, format the
@code{abs(limit)} oldest frames. If @emph{most_recent_first} is @code{True}, the order
of the formatted frames is reversed, returning the most recent frame first
instead of last.

Similar to the @ref{3b26,,traceback.format_tb()} function, except that
@ref{6c8,,format()} does not include newlines.

Example:

@example
print("Traceback (most recent call first):")
for line in traceback:
    print(line)
@end example

Output:

@example
Traceback (most recent call first):
  File "test.py", line 9
    obj = Object()
  File "test.py", line 12
    tb = tracemalloc.get_object_traceback(f())
@end example
@end deffn
@end deffn

@node Software Packaging and Distribution,Python Runtime Services,Debugging and Profiling,The Python Standard Library
@anchor{library/distribution doc}@anchor{3b27}@anchor{library/distribution software-packaging-and-distribution}@anchor{3b28}
@section Software Packaging and Distribution


These libraries help you with publishing and installing Python software.
While these modules are designed to work in conjunction with the
Python Package Index@footnote{https://pypi.org}, they can also be used
with a local index server, or without any index server at all.

@menu
* distutils — Building and installing Python modules:: 
* ensurepip — Bootstrapping the pip installer:: 
* venv — Creation of virtual environments:: 
* zipapp — Manage executable Python zip archives:: 

@end menu

@node distutils — Building and installing Python modules,ensurepip — Bootstrapping the pip installer,,Software Packaging and Distribution
@anchor{library/distutils doc}@anchor{3b29}@anchor{library/distutils distutils-building-and-installing-python-modules}@anchor{3b2a}@anchor{library/distutils module-distutils}@anchor{37}
@subsection @code{distutils} — Building and installing Python modules


@geindex module; distutils


__________________________________________________________________


@ref{37,,distutils} is deprecated with removal planned for Python 3.12.
See the @ref{279,,What’s New} entry for more information.


__________________________________________________________________


The @ref{37,,distutils} package provides support for building and installing
additional modules into a Python installation.  The new modules may be either
100%-pure Python, or may be extension modules written in C, or may be
collections of Python packages which include modules coded in both Python and C.

Most Python users will @emph{not} want to use this module directly, but instead
use the cross-version tools maintained by the Python Packaging Authority. In
particular,
setuptools@footnote{https://setuptools.readthedocs.io/en/latest/} is an
enhanced alternative to @ref{37,,distutils} that provides:


@itemize *

@item 
support for declaring project dependencies

@item 
additional mechanisms for configuring which files to include in source
releases (including plugins for integration with version control systems)

@item 
the ability to declare project “entry points”, which can be used as the
basis for application plugin systems

@item 
the ability to automatically generate Windows command line executables at
installation time rather than needing to prebuild them

@item 
consistent behaviour across all supported Python versions
@end itemize

The recommended pip@footnote{https://pip.pypa.io/} installer runs all
@code{setup.py} scripts with @code{setuptools}, even if the script itself only
imports @code{distutils}. Refer to the
Python Packaging User Guide@footnote{https://packaging.python.org} for more
information.

For the benefits of packaging tool authors and users seeking a deeper
understanding of the details of the current packaging and distribution
system, the legacy @ref{37,,distutils} based user documentation and API
reference remain available:


@itemize *

@item 
@ref{a6b,,Installing Python Modules (Legacy version)}

@item 
@ref{a6c,,Distributing Python Modules (Legacy version)}
@end itemize

@node ensurepip — Bootstrapping the pip installer,venv — Creation of virtual environments,distutils — Building and installing Python modules,Software Packaging and Distribution
@anchor{library/ensurepip doc}@anchor{3b2b}@anchor{library/ensurepip ensurepip-bootstrapping-the-pip-installer}@anchor{3b2c}@anchor{library/ensurepip module-ensurepip}@anchor{75}
@subsection @code{ensurepip} — Bootstrapping the @code{pip} installer


@geindex module; ensurepip


New in version 3.4.



__________________________________________________________________


The @ref{75,,ensurepip} package provides support for bootstrapping the @code{pip}
installer into an existing Python installation or virtual environment. This
bootstrapping approach reflects the fact that @code{pip} is an independent
project with its own release cycle, and the latest available stable version
is bundled with maintenance and feature releases of the CPython reference
interpreter.

In most cases, end users of Python shouldn’t need to invoke this module
directly (as @code{pip} should be bootstrapped by default), but it may be
needed if installing @code{pip} was skipped when installing Python (or
when creating a virtual environment) or after explicitly uninstalling
@code{pip}.

@cartouche
@quotation Note 
This module @emph{does not} access the internet. All of the components
needed to bootstrap @code{pip} are included as internal parts of the
package.
@end quotation
@end cartouche


@subsubheading See also



@table @asis

@item @ref{a69,,Installing Python Modules}

The end user guide for installing Python packages

@geindex Python Enhancement Proposals; PEP 453

@item PEP 453@footnote{https://www.python.org/dev/peps/pep-0453}: Explicit bootstrapping of pip in Python installations

The original rationale and specification for this module.
@end table


@menu
* Command line interface:: 
* Module API:: 

@end menu

@node Command line interface,Module API,,ensurepip — Bootstrapping the pip installer
@anchor{library/ensurepip command-line-interface}@anchor{3b2d}
@subsubsection Command line interface


The command line interface is invoked using the interpreter’s @code{-m} switch.

The simplest possible invocation is:

@example
python -m ensurepip
@end example

This invocation will install @code{pip} if it is not already installed,
but otherwise does nothing. To ensure the installed version of @code{pip}
is at least as recent as the one available in @code{ensurepip}, pass the
@code{--upgrade} option:

@example
python -m ensurepip --upgrade
@end example

By default, @code{pip} is installed into the current virtual environment
(if one is active) or into the system site packages (if there is no
active virtual environment). The installation location can be controlled
through two additional command line options:


@itemize *

@item 
@code{--root <dir>}: Installs @code{pip} relative to the given root directory
rather than the root of the currently active virtual environment (if any)
or the default root for the current Python installation.

@item 
@code{--user}: Installs @code{pip} into the user site packages directory rather
than globally for the current Python installation (this option is not
permitted inside an active virtual environment).
@end itemize

By default, the scripts @code{pipX} and @code{pipX.Y} will be installed (where
X.Y stands for the version of Python used to invoke @code{ensurepip}). The
scripts installed can be controlled through two additional command line
options:


@itemize *

@item 
@code{--altinstall}: if an alternate installation is requested, the @code{pipX}
script will @emph{not} be installed.

@item 
@code{--default-pip}: if a “default pip” installation is requested, the
@code{pip} script will be installed in addition to the two regular scripts.
@end itemize

Providing both of the script selection options will trigger an exception.

@node Module API,,Command line interface,ensurepip — Bootstrapping the pip installer
@anchor{library/ensurepip module-api}@anchor{3b2e}
@subsubsection Module API


@ref{75,,ensurepip} exposes two functions for programmatic use:

@geindex version() (in module ensurepip)
@anchor{library/ensurepip ensurepip version}@anchor{3b2f}
@deffn {Function} ensurepip.version ()

Returns a string specifying the available version of pip that will be
installed when bootstrapping an environment.
@end deffn

@geindex bootstrap() (in module ensurepip)
@anchor{library/ensurepip ensurepip bootstrap}@anchor{39f1}
@deffn {Function} ensurepip.bootstrap (root=None, upgrade=False, user=False, altinstall=False, default_pip=False, verbosity=0)

Bootstraps @code{pip} into the current or designated environment.

@emph{root} specifies an alternative root directory to install relative to.
If @emph{root} is @code{None}, then installation uses the default install location
for the current environment.

@emph{upgrade} indicates whether or not to upgrade an existing installation
of an earlier version of @code{pip} to the available version.

@emph{user} indicates whether to use the user scheme rather than installing
globally.

By default, the scripts @code{pipX} and @code{pipX.Y} will be installed (where
X.Y stands for the current version of Python).

If @emph{altinstall} is set, then @code{pipX} will @emph{not} be installed.

If @emph{default_pip} is set, then @code{pip} will be installed in addition to
the two regular scripts.

Setting both @emph{altinstall} and @emph{default_pip} will trigger
@ref{17c,,ValueError}.

@emph{verbosity} controls the level of output to @ref{5ef,,sys.stdout} from the
bootstrapping operation.

Raises an @ref{1130,,auditing event} @code{ensurepip.bootstrap} with argument @code{root}.

@cartouche
@quotation Note 
The bootstrapping process has side effects on both @code{sys.path} and
@code{os.environ}. Invoking the command line interface in a subprocess
instead allows these side effects to be avoided.
@end quotation
@end cartouche

@cartouche
@quotation Note 
The bootstrapping process may install additional modules required by
@code{pip}, but other software should not assume those dependencies will
always be present by default (as the dependencies may be removed in a
future version of @code{pip}).
@end quotation
@end cartouche
@end deffn

@node venv — Creation of virtual environments,zipapp — Manage executable Python zip archives,ensurepip — Bootstrapping the pip installer,Software Packaging and Distribution
@anchor{library/venv doc}@anchor{3b30}@anchor{library/venv module-venv}@anchor{12c}@anchor{library/venv venv-creation-of-virtual-environments}@anchor{3b31}
@subsection @code{venv} — Creation of virtual environments


@geindex module; venv


New in version 3.3.


@strong{Source code:} Lib/venv/@footnote{https://github.com/python/cpython/tree/main/Lib/venv/}

@geindex Environments; virtual


__________________________________________________________________


The @ref{12c,,venv} module provides support for creating lightweight “virtual
environments” with their own site directories, optionally isolated from system
site directories.  Each virtual environment has its own Python binary (which
matches the version of the binary that was used to create this environment) and
can have its own independent set of installed Python packages in its site
directories.

See 
@geindex Python Enhancement Proposals; PEP 405
PEP 405@footnote{https://www.python.org/dev/peps/pep-0405} for more information about Python virtual environments.


@subsubheading See also


Python Packaging User Guide: Creating and using virtual environments@footnote{https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment}


@menu
* Creating virtual environments:: 
* API: API<2>. 
* An example of extending EnvBuilder:: 

@end menu

@node Creating virtual environments,API<2>,,venv — Creation of virtual environments
@anchor{library/venv creating-virtual-environments}@anchor{3b32}
@subsubsection Creating virtual environments


Creation of @ref{3b33,,virtual environments} is done by executing the
command @code{venv}:

@example
python3 -m venv /path/to/new/virtual/environment
@end example

Running this command creates the target directory (creating any parent
directories that don’t exist already) and places a @code{pyvenv.cfg} file in it
with a @code{home} key pointing to the Python installation from which the command
was run (a common name for the target directory is @code{.venv}).  It also creates
a @code{bin} (or @code{Scripts} on Windows) subdirectory containing a copy/symlink
of the Python binary/binaries (as appropriate for the platform or arguments
used at environment creation time). It also creates an (initially empty)
@code{lib/pythonX.Y/site-packages} subdirectory (on Windows, this is
@code{Lib\site-packages}). If an existing directory is specified, it will be
re-used.


Deprecated since version 3.6: @code{pyvenv} was the recommended tool for creating virtual environments for
Python 3.3 and 3.4, and is deprecated in Python 3.6@footnote{https://docs.python.org/dev/whatsnew/3.6.html#deprecated-features}.



Changed in version 3.5: The use of @code{venv} is now recommended for creating virtual environments.


On Windows, invoke the @code{venv} command as follows:

@example
c:\>c:\Python35\python -m venv c:\path\to\myenv
@end example

Alternatively, if you configured the @code{PATH} and @code{PATHEXT} variables for
your @ref{5c9,,Python installation}:

@example
c:\>python -m venv c:\path\to\myenv
@end example

The command, if run with @code{-h}, will show the available options:

@example
usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear]
            [--upgrade] [--without-pip] [--prompt PROMPT] [--upgrade-deps]
            ENV_DIR [ENV_DIR ...]

Creates virtual Python environments in one or more target directories.

positional arguments:
  ENV_DIR               A directory to create the environment in.

optional arguments:
  -h, --help            show this help message and exit
  --system-site-packages
                        Give the virtual environment access to the system
                        site-packages dir.
  --symlinks            Try to use symlinks rather than copies, when symlinks
                        are not the default for the platform.
  --copies              Try to use copies rather than symlinks, even when
                        symlinks are the default for the platform.
  --clear               Delete the contents of the environment directory if it
                        already exists, before environment creation.
  --upgrade             Upgrade the environment directory to use this version
                        of Python, assuming Python has been upgraded in-place.
  --without-pip         Skips installing or upgrading pip in the virtual
                        environment (pip is bootstrapped by default)
  --prompt PROMPT       Provides an alternative prompt prefix for this
                        environment.
  --upgrade-deps        Upgrade core dependencies: pip setuptools to the
                        latest version in PyPI

Once an environment has been created, you may wish to activate it, e.g. by
sourcing an activate script in its bin directory.
@end example


Changed in version 3.9: Add @code{--upgrade-deps} option to upgrade pip + setuptools to the latest on PyPI



Changed in version 3.4: Installs pip by default, added the @code{--without-pip}  and @code{--copies}
options



Changed in version 3.4: In earlier versions, if the target directory already existed, an error was
raised, unless the @code{--clear} or @code{--upgrade} option was provided.


@cartouche
@quotation Note 
While symlinks are supported on Windows, they are not recommended. Of
particular note is that double-clicking @code{python.exe} in File Explorer
will resolve the symlink eagerly and ignore the virtual environment.
@end quotation
@end cartouche

@cartouche
@quotation Note 
On Microsoft Windows, it may be required to enable the @code{Activate.ps1}
script by setting the execution policy for the user. You can do this by
issuing the following PowerShell command:

PS C:> Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

See About Execution Policies@footnote{https://go.microsoft.com/fwlink/?LinkID=135170}
for more information.
@end quotation
@end cartouche

The created @code{pyvenv.cfg} file also includes the
@code{include-system-site-packages} key, set to @code{true} if @code{venv} is
run with the @code{--system-site-packages} option, @code{false} otherwise.

Unless the @code{--without-pip} option is given, @ref{75,,ensurepip} will be
invoked to bootstrap @code{pip} into the virtual environment.

Multiple paths can be given to @code{venv}, in which case an identical virtual
environment will be created, according to the given options, at each provided
path.

Once a virtual environment has been created, it can be “activated” using a
script in the virtual environment’s binary directory. The invocation of the
script is platform-specific (@cite{<venv>} must be replaced by the path of the
directory containing the virtual environment):


@multitable {xxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Platform

@tab

Shell

@tab

Command to activate virtual environment

@item

POSIX

@tab

bash/zsh

@tab

$ source <venv>/bin/activate

@item

@tab

fish

@tab

$ source <venv>/bin/activate.fish

@item

@tab

csh/tcsh

@tab

$ source <venv>/bin/activate.csh

@item

@tab

PowerShell Core

@tab

$ <venv>/bin/Activate.ps1

@item

Windows

@tab

cmd.exe

@tab

C:\> <venv>\Scripts\activate.bat

@item

@tab

PowerShell

@tab

PS C:\> <venv>\Scripts\Activate.ps1

@end multitable


When a virtual environment is active, the 
@geindex VIRTUAL_ENV
@geindex environment variable; VIRTUAL_ENV
@code{VIRTUAL_ENV} environment
variable is set to the path of the virtual environment. This can be used to
check if one is running inside a virtual environment.

You don’t specifically @emph{need} to activate an environment; activation just
prepends the virtual environment’s binary directory to your path, so that
“python” invokes the virtual environment’s Python interpreter and you can run
installed scripts without having to use their full path. However, all scripts
installed in a virtual environment should be runnable without activating it,
and run with the virtual environment’s Python automatically.

You can deactivate a virtual environment by typing “deactivate” in your shell.
The exact mechanism is platform-specific and is an internal implementation
detail (typically a script or shell function will be used).


New in version 3.4: @code{fish} and @code{csh} activation scripts.



New in version 3.8: PowerShell activation scripts installed under POSIX for PowerShell Core
support.

@anchor{library/venv venv-def}@anchor{3b33}
@cartouche
@quotation Note 
A virtual environment is a Python environment such that the Python
interpreter, libraries and scripts installed into it are isolated from those
installed in other virtual environments, and (by default) any libraries
installed in a “system” Python, i.e., one which is installed as part of your
operating system.

A virtual environment is a directory tree which contains Python executable
files and other files which indicate that it is a virtual environment.

Common installation tools such as setuptools@footnote{https://pypi.org/project/setuptools/} and pip@footnote{https://pypi.org/project/pip/} work as
expected with virtual environments. In other words, when a virtual
environment is active, they install Python packages into the virtual
environment without needing to be told to do so explicitly.

When a virtual environment is active (i.e., the virtual environment’s Python
interpreter is running), the attributes @ref{3b34,,sys.prefix} and
@ref{3b35,,sys.exec_prefix} point to the base directory of the virtual
environment, whereas @ref{34c7,,sys.base_prefix} and
@ref{3b36,,sys.base_exec_prefix} point to the non-virtual environment Python
installation which was used to create the virtual environment. If a virtual
environment is not active, then @ref{3b34,,sys.prefix} is the same as
@ref{34c7,,sys.base_prefix} and @ref{3b35,,sys.exec_prefix} is the same as
@ref{3b36,,sys.base_exec_prefix} (they all point to a non-virtual environment
Python installation).

When a virtual environment is active, any options that change the
installation path will be ignored from all @ref{37,,distutils} configuration
files to prevent projects being inadvertently installed outside of the
virtual environment.

When working in a command shell, users can make a virtual environment active
by running an @code{activate} script in the virtual environment’s executables
directory (the precise filename and command to use the file is
shell-dependent), which prepends the virtual environment’s directory for
executables to the @code{PATH} environment variable for the running shell. There
should be no need in other circumstances to activate a virtual
environment; scripts installed into virtual environments have a “shebang”
line which points to the virtual environment’s Python interpreter. This means
that the script will run with that interpreter regardless of the value of
@code{PATH}. On Windows, “shebang” line processing is supported if you have the
Python Launcher for Windows installed (this was added to Python in 3.3 - see
@geindex Python Enhancement Proposals; PEP 397
PEP 397@footnote{https://www.python.org/dev/peps/pep-0397} for more details). Thus, double-clicking an installed script in a
Windows Explorer window should run the script with the correct interpreter
without there needing to be any reference to its virtual environment in
@code{PATH}.
@end quotation
@end cartouche

@node API<2>,An example of extending EnvBuilder,Creating virtual environments,venv — Creation of virtual environments
@anchor{library/venv api}@anchor{3b37}@anchor{library/venv venv-api}@anchor{a66}
@subsubsection API


The high-level method described above makes use of a simple API which provides
mechanisms for third-party virtual environment creators to customize environment
creation according to their needs, the @ref{b6e,,EnvBuilder} class.

@geindex EnvBuilder (class in venv)
@anchor{library/venv venv EnvBuilder}@anchor{b6e}
@deffn {Class} venv.EnvBuilder (system_site_packages=False, clear=False, symlinks=False, upgrade=False, with_pip=False, prompt=None, upgrade_deps=False)

The @ref{b6e,,EnvBuilder} class accepts the following keyword arguments on
instantiation:


@itemize *

@item 
@code{system_site_packages} – a Boolean value indicating that the system Python
site-packages should be available to the environment (defaults to @code{False}).

@item 
@code{clear} – a Boolean value which, if true, will delete the contents of
any existing target directory, before creating the environment.

@item 
@code{symlinks} – a Boolean value indicating whether to attempt to symlink the
Python binary rather than copying.

@item 
@code{upgrade} – a Boolean value which, if true, will upgrade an existing
environment with the running Python - for use when that Python has been
upgraded in-place (defaults to @code{False}).

@item 
@code{with_pip} – a Boolean value which, if true, ensures pip is
installed in the virtual environment. This uses @ref{75,,ensurepip} with
the @code{--default-pip} option.

@item 
@code{prompt} – a String to be used after virtual environment is activated
(defaults to @code{None} which means directory name of the environment would
be used). If the special string @code{"."} is provided, the basename of the
current directory is used as the prompt.

@item 
@code{upgrade_deps} – Update the base venv modules to the latest on PyPI
@end itemize


Changed in version 3.4: Added the @code{with_pip} parameter



New in version 3.6: Added the @code{prompt} parameter



New in version 3.9: Added the @code{upgrade_deps} parameter


Creators of third-party virtual environment tools will be free to use the
provided @ref{b6e,,EnvBuilder} class as a base class.

The returned env-builder is an object which has a method, @code{create}:

@geindex create() (venv.EnvBuilder method)
@anchor{library/venv venv EnvBuilder create}@anchor{3b38}
@deffn {Method} create (env_dir)

Create a virtual environment by specifying the target directory
(absolute or relative to the current directory) which is to contain the
virtual environment.  The @code{create} method will either create the
environment in the specified directory, or raise an appropriate
exception.

The @code{create} method of the @ref{b6e,,EnvBuilder} class illustrates the
hooks available for subclass customization:

@example
def create(self, env_dir):
    """
    Create a virtualized Python environment in a directory.
    env_dir is the target directory to create an environment in.
    """
    env_dir = os.path.abspath(env_dir)
    context = self.ensure_directories(env_dir)
    self.create_configuration(context)
    self.setup_python(context)
    self.setup_scripts(context)
    self.post_setup(context)
@end example

Each of the methods @ref{3b39,,ensure_directories()},
@ref{3b3a,,create_configuration()}, @ref{3b3b,,setup_python()},
@ref{3b3c,,setup_scripts()} and @ref{3b3d,,post_setup()} can be overridden.
@end deffn

@geindex ensure_directories() (venv.EnvBuilder method)
@anchor{library/venv venv EnvBuilder ensure_directories}@anchor{3b39}
@deffn {Method} ensure_directories (env_dir)

Creates the environment directory and all necessary subdirectories that
don’t already exist, and returns a context object.  This context object
is just a holder for attributes (such as paths) for use by the other
methods.  If the @ref{b6e,,EnvBuilder} is created with the arg
@code{clear=True}, contents of the environment directory will be cleared
and then all necessary subdirectories will be recreated.
@end deffn

@geindex create_configuration() (venv.EnvBuilder method)
@anchor{library/venv venv EnvBuilder create_configuration}@anchor{3b3a}
@deffn {Method} create_configuration (context)

Creates the @code{pyvenv.cfg} configuration file in the environment.
@end deffn

@geindex setup_python() (venv.EnvBuilder method)
@anchor{library/venv venv EnvBuilder setup_python}@anchor{3b3b}
@deffn {Method} setup_python (context)

Creates a copy or symlink to the Python executable in the environment.
On POSIX systems, if a specific executable @code{python3.x} was used,
symlinks to @code{python} and @code{python3} will be created pointing to that
executable, unless files with those names already exist.
@end deffn

@geindex setup_scripts() (venv.EnvBuilder method)
@anchor{library/venv venv EnvBuilder setup_scripts}@anchor{3b3c}
@deffn {Method} setup_scripts (context)

Installs activation scripts appropriate to the platform into the virtual
environment.
@end deffn

@geindex upgrade_dependencies() (venv.EnvBuilder method)
@anchor{library/venv venv EnvBuilder upgrade_dependencies}@anchor{3b3e}
@deffn {Method} upgrade_dependencies (context)

Upgrades the core venv dependency packages (currently @code{pip} and
@code{setuptools}) in the environment. This is done by shelling out to the
@code{pip} executable in the environment.


New in version 3.9.

@end deffn

@geindex post_setup() (venv.EnvBuilder method)
@anchor{library/venv venv EnvBuilder post_setup}@anchor{3b3d}
@deffn {Method} post_setup (context)

A placeholder method which can be overridden in third party
implementations to pre-install packages in the virtual environment or
perform other post-creation steps.
@end deffn


Changed in version 3.7.2: Windows now uses redirector scripts for @code{python[w].exe} instead of
copying the actual binaries. In 3.7.2 only @ref{3b3b,,setup_python()} does
nothing unless running from a build in the source tree.



Changed in version 3.7.3: Windows copies the redirector scripts as part of @ref{3b3b,,setup_python()}
instead of @ref{3b3c,,setup_scripts()}. This was not the case in 3.7.2.
When using symlinks, the original executables will be linked.


In addition, @ref{b6e,,EnvBuilder} provides this utility method that can be
called from @ref{3b3c,,setup_scripts()} or @ref{3b3d,,post_setup()} in subclasses to
assist in installing custom scripts into the virtual environment.

@geindex install_scripts() (venv.EnvBuilder method)
@anchor{library/venv venv EnvBuilder install_scripts}@anchor{3b3f}
@deffn {Method} install_scripts (context, path)

@emph{path} is the path to a directory that should contain subdirectories
“common”, “posix”, “nt”, each containing scripts destined for the bin
directory in the environment.  The contents of “common” and the
directory corresponding to @ref{2375,,os.name} are copied after some text
replacement of placeholders:


@itemize *

@item 
@code{__VENV_DIR__} is replaced with the absolute path of the environment
directory.

@item 
@code{__VENV_NAME__} is replaced with the environment name (final path
segment of environment directory).

@item 
@code{__VENV_PROMPT__} is replaced with the prompt (the environment
name surrounded by parentheses and with a following space)

@item 
@code{__VENV_BIN_NAME__} is replaced with the name of the bin directory
(either @code{bin} or @code{Scripts}).

@item 
@code{__VENV_PYTHON__} is replaced with the absolute path of the
environment’s executable.
@end itemize

The directories are allowed to exist (for when an existing environment
is being upgraded).
@end deffn
@end deffn

There is also a module-level convenience function:

@geindex create() (in module venv)
@anchor{library/venv venv create}@anchor{b6f}
@deffn {Function} venv.create (env_dir, system_site_packages=False, clear=False, symlinks=False, with_pip=False, prompt=None, upgrade_deps=False)

Create an @ref{b6e,,EnvBuilder} with the given keyword arguments, and call its
@ref{3b38,,create()} method with the @emph{env_dir} argument.


New in version 3.3.



Changed in version 3.4: Added the @code{with_pip} parameter



Changed in version 3.6: Added the @code{prompt} parameter



Changed in version 3.9: Added the @code{upgrade_deps} parameter

@end deffn

@node An example of extending EnvBuilder,,API<2>,venv — Creation of virtual environments
@anchor{library/venv an-example-of-extending-envbuilder}@anchor{3b40}
@subsubsection An example of extending @code{EnvBuilder}


The following script shows how to extend @ref{b6e,,EnvBuilder} by implementing a
subclass which installs setuptools and pip into a created virtual environment:

@example
import os
import os.path
from subprocess import Popen, PIPE
import sys
from threading import Thread
from urllib.parse import urlparse
from urllib.request import urlretrieve
import venv

class ExtendedEnvBuilder(venv.EnvBuilder):
    """
    This builder installs setuptools and pip so that you can pip or
    easy_install other packages into the created virtual environment.

    :param nodist: If true, setuptools and pip are not installed into the
                   created virtual environment.
    :param nopip: If true, pip is not installed into the created
                  virtual environment.
    :param progress: If setuptools or pip are installed, the progress of the
                     installation can be monitored by passing a progress
                     callable. If specified, it is called with two
                     arguments: a string indicating some progress, and a
                     context indicating where the string is coming from.
                     The context argument can have one of three values:
                     'main', indicating that it is called from virtualize()
                     itself, and 'stdout' and 'stderr', which are obtained
                     by reading lines from the output streams of a subprocess
                     which is used to install the app.

                     If a callable is not specified, default progress
                     information is output to sys.stderr.
    """

    def __init__(self, *args, **kwargs):
        self.nodist = kwargs.pop('nodist', False)
        self.nopip = kwargs.pop('nopip', False)
        self.progress = kwargs.pop('progress', None)
        self.verbose = kwargs.pop('verbose', False)
        super().__init__(*args, **kwargs)

    def post_setup(self, context):
        """
        Set up any packages which need to be pre-installed into the
        virtual environment being created.

        :param context: The information for the virtual environment
                        creation request being processed.
        """
        os.environ['VIRTUAL_ENV'] = context.env_dir
        if not self.nodist:
            self.install_setuptools(context)
        # Can't install pip without setuptools
        if not self.nopip and not self.nodist:
            self.install_pip(context)

    def reader(self, stream, context):
        """
        Read lines from a subprocess' output stream and either pass to a progress
        callable (if specified) or write progress information to sys.stderr.
        """
        progress = self.progress
        while True:
            s = stream.readline()
            if not s:
                break
            if progress is not None:
                progress(s, context)
            else:
                if not self.verbose:
                    sys.stderr.write('.')
                else:
                    sys.stderr.write(s.decode('utf-8'))
                sys.stderr.flush()
        stream.close()

    def install_script(self, context, name, url):
        _, _, path, _, _, _ = urlparse(url)
        fn = os.path.split(path)[-1]
        binpath = context.bin_path
        distpath = os.path.join(binpath, fn)
        # Download script into the virtual environment's binaries folder
        urlretrieve(url, distpath)
        progress = self.progress
        if self.verbose:
            term = '\n'
        else:
            term = '@w{'}
        if progress is not None:
            progress('Installing %s ...%s' % (name, term), 'main')
        else:
            sys.stderr.write('Installing %s ...%s' % (name, term))
            sys.stderr.flush()
        # Install in the virtual environment
        args = [context.env_exe, fn]
        p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath)
        t1 = Thread(target=self.reader, args=(p.stdout, 'stdout'))
        t1.start()
        t2 = Thread(target=self.reader, args=(p.stderr, 'stderr'))
        t2.start()
        p.wait()
        t1.join()
        t2.join()
        if progress is not None:
            progress('done.', 'main')
        else:
            sys.stderr.write('done.\n')
        # Clean up - no longer needed
        os.unlink(distpath)

    def install_setuptools(self, context):
        """
        Install setuptools in the virtual environment.

        :param context: The information for the virtual environment
                        creation request being processed.
        """
        url = 'https://bitbucket.org/pypa/setuptools/downloads/ez_setup.py'
        self.install_script(context, 'setuptools', url)
        # clear up the setuptools archive which gets downloaded
        pred = lambda o: o.startswith('setuptools-') and o.endswith('.tar.gz')
        files = filter(pred, os.listdir(context.bin_path))
        for f in files:
            f = os.path.join(context.bin_path, f)
            os.unlink(f)

    def install_pip(self, context):
        """
        Install pip in the virtual environment.

        :param context: The information for the virtual environment
                        creation request being processed.
        """
        url = 'https://bootstrap.pypa.io/get-pip.py'
        self.install_script(context, 'pip', url)

def main(args=None):
    compatible = True
    if sys.version_info < (3, 3):
        compatible = False
    elif not hasattr(sys, 'base_prefix'):
        compatible = False
    if not compatible:
        raise ValueError('This script is only for use with '
                         'Python 3.3 or later')
    else:
        import argparse

        parser = argparse.ArgumentParser(prog=__name__,
                                         description='Creates virtual Python '
                                                     'environments in one or '
                                                     'more target '
                                                     'directories.')
        parser.add_argument('dirs', metavar='ENV_DIR', nargs='+',
                            help='A directory in which to create the '
                                 'virtual environment.')
        parser.add_argument('--no-setuptools', default=False,
                            action='store_true', dest='nodist',
                            help="Don't install setuptools or pip in the "
                                 "virtual environment.")
        parser.add_argument('--no-pip', default=False,
                            action='store_true', dest='nopip',
                            help="Don't install pip in the virtual "
                                 "environment.")
        parser.add_argument('--system-site-packages', default=False,
                            action='store_true', dest='system_site',
                            help='Give the virtual environment access to the '
                                 'system site-packages dir.')
        if os.name == 'nt':
            use_symlinks = False
        else:
            use_symlinks = True
        parser.add_argument('--symlinks', default=use_symlinks,
                            action='store_true', dest='symlinks',
                            help='Try to use symlinks rather than copies, '
                                 'when symlinks are not the default for '
                                 'the platform.')
        parser.add_argument('--clear', default=False, action='store_true',
                            dest='clear', help='Delete the contents of the '
                                               'virtual environment '
                                               'directory if it already '
                                               'exists, before virtual '
                                               'environment creation.')
        parser.add_argument('--upgrade', default=False, action='store_true',
                            dest='upgrade', help='Upgrade the virtual '
                                                 'environment directory to '
                                                 'use this version of '
                                                 'Python, assuming Python '
                                                 'has been upgraded '
                                                 'in-place.')
        parser.add_argument('--verbose', default=False, action='store_true',
                            dest='verbose', help='Display the output '
                                               'from the scripts which '
                                               'install setuptools and pip.')
        options = parser.parse_args(args)
        if options.upgrade and options.clear:
            raise ValueError('you cannot supply --upgrade and --clear together.')
        builder = ExtendedEnvBuilder(system_site_packages=options.system_site,
                                       clear=options.clear,
                                       symlinks=options.symlinks,
                                       upgrade=options.upgrade,
                                       nodist=options.nodist,
                                       nopip=options.nopip,
                                       verbose=options.verbose)
        for d in options.dirs:
            builder.create(d)

if __name__ == '__main__':
    rc = 1
    try:
        main()
        rc = 0
    except Exception as e:
        print('Error: %s' % e, file=sys.stderr)
    sys.exit(rc)
@end example

This script is also available for download online@footnote{https://gist.github.com/vsajip/4673395}.

@node zipapp — Manage executable Python zip archives,,venv — Creation of virtual environments,Software Packaging and Distribution
@anchor{library/zipapp doc}@anchor{3b41}@anchor{library/zipapp module-zipapp}@anchor{148}@anchor{library/zipapp pip}@anchor{3b42}@anchor{library/zipapp zipapp-manage-executable-python-zip-archives}@anchor{3b43}
@subsection @code{zipapp} — Manage executable Python zip archives


@geindex module; zipapp


New in version 3.5.


@strong{Source code:} Lib/zipapp.py@footnote{https://github.com/python/cpython/tree/main/Lib/zipapp.py}

@geindex Executable Zip Files


__________________________________________________________________


This module provides tools to manage the creation of zip files containing
Python code, which can be  @ref{15fd,,executed directly by the Python interpreter}.  The module provides both a
@ref{3b44,,Command-Line Interface} and a @ref{3b45,,Python API}.

@menu
* Basic Example:: 
* Command-Line Interface: Command-Line Interface<5>. 
* Python API:: 
* Examples: Examples<30>. 
* Specifying the Interpreter:: 
* Creating Standalone Applications with zipapp:: 
* The Python Zip Application Archive Format:: 

@end menu

@node Basic Example,Command-Line Interface<5>,,zipapp — Manage executable Python zip archives
@anchor{library/zipapp basic-example}@anchor{3b46}
@subsubsection Basic Example


The following example shows how the @ref{3b44,,Command-Line Interface}
can be used to create an executable archive from a directory containing
Python code.  When run, the archive will execute the @code{main} function from
the module @code{myapp} in the archive.

@example
$ python -m zipapp myapp -m "myapp:main"
$ python myapp.pyz
<output from myapp>
@end example

@node Command-Line Interface<5>,Python API,Basic Example,zipapp — Manage executable Python zip archives
@anchor{library/zipapp command-line-interface}@anchor{3b47}@anchor{library/zipapp zipapp-command-line-interface}@anchor{3b44}
@subsubsection Command-Line Interface


When called as a program from the command line, the following form is used:

@example
$ python -m zipapp source [options]
@end example

If @emph{source} is a directory, this will create an archive from the contents of
@emph{source}.  If @emph{source} is a file, it should be an archive, and it will be
copied to the target archive (or the contents of its shebang line will be
displayed if the –info option is specified).

The following options are understood:

@geindex zipapp command line option; -o <output>
@geindex zipapp command line option; --output=<output>
@anchor{library/zipapp cmdoption-zipapp-o}@anchor{3b48}@anchor{library/zipapp cmdoption-zipapp-output}@anchor{3b49}
@deffn {Program Option} @w{-}o <output>, @w{-}@w{-}output=<output>

Write the output to a file named @emph{output}.  If this option is not specified,
the output filename will be the same as the input @emph{source}, with the
extension @code{.pyz} added.  If an explicit filename is given, it is used as
is (so a @code{.pyz} extension should be included if required).

An output filename must be specified if the @emph{source} is an archive (and in
that case, @emph{output} must not be the same as @emph{source}).
@end deffn

@geindex zipapp command line option; -p <interpreter>
@geindex zipapp command line option; --python=<interpreter>
@anchor{library/zipapp cmdoption-zipapp-p}@anchor{3b4a}@anchor{library/zipapp cmdoption-zipapp-python}@anchor{3b4b}
@deffn {Program Option} @w{-}p <interpreter>, @w{-}@w{-}python=<interpreter>

Add a @code{#!} line to the archive specifying @emph{interpreter} as the command
to run.  Also, on POSIX, make the archive executable.  The default is to
write no @code{#!} line, and not make the file executable.
@end deffn

@geindex zipapp command line option; -m <mainfn>
@geindex zipapp command line option; --main=<mainfn>
@anchor{library/zipapp cmdoption-zipapp-m}@anchor{3b4c}@anchor{library/zipapp cmdoption-zipapp-main}@anchor{3b4d}
@deffn {Program Option} @w{-}m <mainfn>, @w{-}@w{-}main=<mainfn>

Write a @code{__main__.py} file to the archive that executes @emph{mainfn}.  The
@emph{mainfn} argument should have the form “pkg.mod:fn”, where “pkg.mod” is a
package/module in the archive, and “fn” is a callable in the given module.
The @code{__main__.py} file will execute that callable.

@ref{3b4c,,--main} cannot be specified when copying an archive.
@end deffn

@geindex zipapp command line option; -c
@geindex zipapp command line option; --compress
@anchor{library/zipapp cmdoption-zipapp-c}@anchor{3b4e}@anchor{library/zipapp cmdoption-zipapp-compress}@anchor{3b4f}
@deffn {Program Option} @w{-}c, @w{-}@w{-}compress

Compress files with the deflate method, reducing the size of the output
file. By default, files are stored uncompressed in the archive.

@ref{3b4e,,--compress} has no effect when copying an archive.


New in version 3.7.

@end deffn

@geindex zipapp command line option; --info
@anchor{library/zipapp cmdoption-zipapp-info}@anchor{3b50}
@deffn {Program Option} @w{-}@w{-}info

Display the interpreter embedded in the archive, for diagnostic purposes.  In
this case, any other options are ignored and SOURCE must be an archive, not a
directory.
@end deffn

@geindex zipapp command line option; -h
@geindex zipapp command line option; --help
@anchor{library/zipapp cmdoption-zipapp-h}@anchor{3b51}@anchor{library/zipapp cmdoption-zipapp-help}@anchor{3b52}
@deffn {Program Option} @w{-}h, @w{-}@w{-}help

Print a short usage message and exit.
@end deffn

@node Python API,Examples<30>,Command-Line Interface<5>,zipapp — Manage executable Python zip archives
@anchor{library/zipapp python-api}@anchor{3b53}@anchor{library/zipapp zipapp-python-api}@anchor{3b45}
@subsubsection Python API


The module defines two convenience functions:

@geindex create_archive() (in module zipapp)
@anchor{library/zipapp zipapp create_archive}@anchor{6e3}
@deffn {Function} zipapp.create_archive (source, target=None, interpreter=None, main=None, filter=None, compressed=False)

Create an application archive from @emph{source}.  The source can be any
of the following:


@itemize *

@item 
The name of a directory, or a @ref{63d,,path-like object} referring
to a directory, in which case a new application archive will be
created from the content of that directory.

@item 
The name of an existing application archive file, or a @ref{63d,,path-like object}
referring to such a file, in which case the file is copied to
the target (modifying it to reflect the value given for the @emph{interpreter}
argument).  The file name should include the @code{.pyz} extension, if required.

@item 
A file object open for reading in bytes mode.  The content of the
file should be an application archive, and the file object is
assumed to be positioned at the start of the archive.
@end itemize

The @emph{target} argument determines where the resulting archive will be
written:


@itemize *

@item 
If it is the name of a file, or a @ref{63d,,path-like object},
the archive will be written to that file.

@item 
If it is an open file object, the archive will be written to that
file object, which must be open for writing in bytes mode.

@item 
If the target is omitted (or @code{None}), the source must be a directory
and the target will be a file with the same name as the source, with
a @code{.pyz} extension added.
@end itemize

The @emph{interpreter} argument specifies the name of the Python
interpreter with which the archive will be executed.  It is written as
a “shebang” line at the start of the archive.  On POSIX, this will be
interpreted by the OS, and on Windows it will be handled by the Python
launcher.  Omitting the @emph{interpreter} results in no shebang line being
written.  If an interpreter is specified, and the target is a
filename, the executable bit of the target file will be set.

The @emph{main} argument specifies the name of a callable which will be
used as the main program for the archive.  It can only be specified if
the source is a directory, and the source does not already contain a
@code{__main__.py} file.  The @emph{main} argument should take the form
“pkg.module:callable” and the archive will be run by importing
“pkg.module” and executing the given callable with no arguments.  It
is an error to omit @emph{main} if the source is a directory and does not
contain a @code{__main__.py} file, as otherwise the resulting archive
would not be executable.

The optional @emph{filter} argument specifies a callback function that
is passed a Path object representing the path to the file being added
(relative to the source directory).  It should return @code{True} if the
file is to be added.

The optional @emph{compressed} argument determines whether files are
compressed.  If set to @code{True}, files in the archive are compressed
with the deflate method; otherwise, files are stored uncompressed.
This argument has no effect when copying an existing archive.

If a file object is specified for @emph{source} or @emph{target}, it is the
caller’s responsibility to close it after calling create_archive.

When copying an existing archive, file objects supplied only need
@code{read} and @code{readline}, or @code{write} methods.  When creating an
archive from a directory, if the target is a file object it will be
passed to the @code{zipfile.ZipFile} class, and must supply the methods
needed by that class.


New in version 3.7: Added the @emph{filter} and @emph{compressed} arguments.

@end deffn

@geindex get_interpreter() (in module zipapp)
@anchor{library/zipapp zipapp get_interpreter}@anchor{3b54}
@deffn {Function} zipapp.get_interpreter (archive)

Return the interpreter specified in the @code{#!} line at the start of the
archive.  If there is no @code{#!} line, return @ref{182,,None}.
The @emph{archive} argument can be a filename or a file-like object open
for reading in bytes mode.  It is assumed to be at the start of the archive.
@end deffn

@node Examples<30>,Specifying the Interpreter,Python API,zipapp — Manage executable Python zip archives
@anchor{library/zipapp examples}@anchor{3b55}@anchor{library/zipapp zipapp-examples}@anchor{3b56}
@subsubsection Examples


Pack up a directory into an archive, and run it.

@example
$ python -m zipapp myapp
$ python myapp.pyz
<output from myapp>
@end example

The same can be done using the @ref{6e3,,create_archive()} function:

@example
>>> import zipapp
>>> zipapp.create_archive('myapp', 'myapp.pyz')
@end example

To make the application directly executable on POSIX, specify an interpreter
to use.

@example
$ python -m zipapp myapp -p "/usr/bin/env python"
$ ./myapp.pyz
<output from myapp>
@end example

To replace the shebang line on an existing archive, create a modified archive
using the @ref{6e3,,create_archive()} function:

@example
>>> import zipapp
>>> zipapp.create_archive('old_archive.pyz', 'new_archive.pyz', '/usr/bin/python3')
@end example

To update the file in place, do the replacement in memory using a @code{BytesIO}
object, and then overwrite the source afterwards.  Note that there is a risk
when overwriting a file in place that an error will result in the loss of
the original file.  This code does not protect against such errors, but
production code should do so.  Also, this method will only work if the archive
fits in memory:

@example
>>> import zipapp
>>> import io
>>> temp = io.BytesIO()
>>> zipapp.create_archive('myapp.pyz', temp, '/usr/bin/python2')
>>> with open('myapp.pyz', 'wb') as f:
>>>     f.write(temp.getvalue())
@end example

@node Specifying the Interpreter,Creating Standalone Applications with zipapp,Examples<30>,zipapp — Manage executable Python zip archives
@anchor{library/zipapp specifying-the-interpreter}@anchor{3b57}@anchor{library/zipapp zipapp-specifying-the-interpreter}@anchor{3b58}
@subsubsection Specifying the Interpreter


Note that if you specify an interpreter and then distribute your application
archive, you need to ensure that the interpreter used is portable.  The Python
launcher for Windows supports most common forms of POSIX @code{#!} line, but there
are other issues to consider:


@itemize *

@item 
If you use “/usr/bin/env python” (or other forms of the “python” command,
such as “/usr/bin/python”), you need to consider that your users may have
either Python 2 or Python 3 as their default, and write your code to work
under both versions.

@item 
If you use an explicit version, for example “/usr/bin/env python3” your
application will not work for users who do not have that version.  (This
may be what you want if you have not made your code Python 2 compatible).

@item 
There is no way to say “python X.Y or later”, so be careful of using an
exact version like “/usr/bin/env python3.4” as you will need to change your
shebang line for users of Python 3.5, for example.
@end itemize

Typically, you should use an “/usr/bin/env python2” or “/usr/bin/env python3”,
depending on whether your code is written for Python 2 or 3.

@node Creating Standalone Applications with zipapp,The Python Zip Application Archive Format,Specifying the Interpreter,zipapp — Manage executable Python zip archives
@anchor{library/zipapp creating-standalone-applications-with-zipapp}@anchor{3b59}
@subsubsection Creating Standalone Applications with zipapp


Using the @ref{148,,zipapp} module, it is possible to create self-contained Python
programs, which can be distributed to end users who only need to have a
suitable version of Python installed on their system.  The key to doing this
is to bundle all of the application’s dependencies into the archive, along
with the application code.

The steps to create a standalone archive are as follows:


@enumerate 

@item 
Create your application in a directory as normal, so you have a @code{myapp}
directory containing a @code{__main__.py} file, and any supporting application
code.

@item 
Install all of your application’s dependencies into the @code{myapp} directory,
using pip:

@example
$ python -m pip install -r requirements.txt --target myapp
@end example

(this assumes you have your project requirements in a @code{requirements.txt}
file - if not, you can just list the dependencies manually on the pip command
line).

@item 
Optionally, delete the @code{.dist-info} directories created by pip in the
@code{myapp} directory. These hold metadata for pip to manage the packages, and
as you won’t be making any further use of pip they aren’t required -
although it won’t do any harm if you leave them.

@item 
Package the application using:

@example
$ python -m zipapp -p "interpreter" myapp
@end example
@end enumerate

This will produce a standalone executable, which can be run on any machine with
the appropriate interpreter available. See @ref{3b58,,Specifying the Interpreter}
for details. It can be shipped to users as a single file.

On Unix, the @code{myapp.pyz} file is executable as it stands.  You can rename the
file to remove the @code{.pyz} extension if you prefer a “plain” command name.  On
Windows, the @code{myapp.pyz[w]} file is executable by virtue of the fact that
the Python interpreter registers the @code{.pyz} and @code{.pyzw} file extensions
when installed.

@menu
* Making a Windows executable:: 
* Caveats:: 

@end menu

@node Making a Windows executable,Caveats,,Creating Standalone Applications with zipapp
@anchor{library/zipapp making-a-windows-executable}@anchor{3b5a}
@subsubsection Making a Windows executable


On Windows, registration of the @code{.pyz} extension is optional, and
furthermore, there are certain places that don’t recognise registered
extensions “transparently” (the simplest example is that
@code{subprocess.run(['myapp'])} won’t find your application - you need to
explicitly specify the extension).

On Windows, therefore, it is often preferable to create an executable from the
zipapp.  This is relatively easy, although it does require a C compiler.  The
basic approach relies on the fact that zipfiles can have arbitrary data
prepended, and Windows exe files can have arbitrary data appended.  So by
creating a suitable launcher and tacking the @code{.pyz} file onto the end of it,
you end up with a single-file executable that runs your application.

A suitable launcher can be as simple as the following:

@example
#define Py_LIMITED_API 1
#include "Python.h"

#define WIN32_LEAN_AND_MEAN
#include <windows.h>

#ifdef WINDOWS
int WINAPI wWinMain(
    HINSTANCE hInstance,      /* handle to current instance */
    HINSTANCE hPrevInstance,  /* handle to previous instance */
    LPWSTR lpCmdLine,         /* pointer to command line */
    int nCmdShow              /* show state of window */
)
#else
int wmain()
#endif
@{
    wchar_t **myargv = _alloca((__argc + 1) * sizeof(wchar_t*));
    myargv[0] = __wargv[0];
    memcpy(myargv + 1, __wargv, __argc * sizeof(wchar_t *));
    return Py_Main(__argc+1, myargv);
@}
@end example

If you define the @code{WINDOWS} preprocessor symbol, this will generate a
GUI executable, and without it, a console executable.

To compile the executable, you can either just use the standard MSVC
command line tools, or you can take advantage of the fact that distutils
knows how to compile Python source:

@example
>>> from distutils.ccompiler import new_compiler
>>> import distutils.sysconfig
>>> import sys
>>> import os
>>> from pathlib import Path

>>> def compile(src):
>>>     src = Path(src)
>>>     cc = new_compiler()
>>>     exe = src.stem
>>>     cc.add_include_dir(distutils.sysconfig.get_python_inc())
>>>     cc.add_library_dir(os.path.join(sys.base_exec_prefix, 'libs'))
>>>     # First the CLI executable
>>>     objs = cc.compile([str(src)])
>>>     cc.link_executable(objs, exe)
>>>     # Now the GUI executable
>>>     cc.define_macro('WINDOWS')
>>>     objs = cc.compile([str(src)])
>>>     cc.link_executable(objs, exe + 'w')

>>> if __name__ == "__main__":
>>>     compile("zastub.c")
@end example

The resulting launcher uses the “Limited ABI”, so it will run unchanged with
any version of Python 3.x.  All it needs is for Python (@code{python3.dll}) to be
on the user’s @code{PATH}.

For a fully standalone distribution, you can distribute the launcher with your
application appended, bundled with the Python “embedded” distribution.  This
will run on any PC with the appropriate architecture (32 bit or 64 bit).

@node Caveats,,Making a Windows executable,Creating Standalone Applications with zipapp
@anchor{library/zipapp caveats}@anchor{3b5b}
@subsubsection Caveats


There are some limitations to the process of bundling your application into
a single file.  In most, if not all, cases they can be addressed without
needing major changes to your application.


@enumerate 

@item 
If your application depends on a package that includes a C extension, that
package cannot be run from a zip file (this is an OS limitation, as executable
code must be present in the filesystem for the OS loader to load it). In this
case, you can exclude that dependency from the zipfile, and either require
your users to have it installed, or ship it alongside your zipfile and add code
to your @code{__main__.py} to include the directory containing the unzipped
module in @code{sys.path}. In this case, you will need to make sure to ship
appropriate binaries for your target architecture(s) (and potentially pick the
correct version to add to @code{sys.path} at runtime, based on the user’s machine).

@item 
If you are shipping a Windows executable as described above, you either need to
ensure that your users have @code{python3.dll} on their PATH (which is not the
default behaviour of the installer) or you should bundle your application with
the embedded distribution.

@item 
The suggested launcher above uses the Python embedding API.  This means that in
your application, @code{sys.executable} will be your application, and @emph{not} a
conventional Python interpreter.  Your code and its dependencies need to be
prepared for this possibility.  For example, if your application uses the
@ref{b3,,multiprocessing} module, it will need to call
@ref{2856,,multiprocessing.set_executable()} to let the module know where to find the
standard Python interpreter.
@end enumerate

@node The Python Zip Application Archive Format,,Creating Standalone Applications with zipapp,zipapp — Manage executable Python zip archives
@anchor{library/zipapp the-python-zip-application-archive-format}@anchor{3b5c}
@subsubsection The Python Zip Application Archive Format


Python has been able to execute zip files which contain a @code{__main__.py} file
since version 2.6.  In order to be executed by Python, an application archive
simply has to be a standard zip file containing a @code{__main__.py} file which
will be run as the entry point for the application.  As usual for any Python
script, the parent of the script (in this case the zip file) will be placed on
@ref{489,,sys.path} and thus further modules can be imported from the zip file.

The zip file format allows arbitrary data to be prepended to a zip file.  The
zip application format uses this ability to prepend a standard POSIX “shebang”
line to the file (@code{#!/path/to/interpreter}).

Formally, the Python zip application format is therefore:


@enumerate 

@item 
An optional shebang line, containing the characters @code{b'#!'} followed by an
interpreter name, and then a newline (@code{b'\n'}) character.  The interpreter
name can be anything acceptable to the OS “shebang” processing, or the Python
launcher on Windows.  The interpreter should be encoded in UTF-8 on Windows,
and in @ref{79e,,sys.getfilesystemencoding()} on POSIX.

@item 
Standard zipfile data, as generated by the @ref{149,,zipfile} module.  The
zipfile content @emph{must} include a file called @code{__main__.py} (which must be
in the “root” of the zipfile - i.e., it cannot be in a subdirectory).  The
zipfile data can be compressed or uncompressed.
@end enumerate

If an application archive has a shebang line, it may have the executable bit set
on POSIX systems, to allow it to be executed directly.

There is no requirement that the tools in this module are used to create
application archives - the module is a convenience, but archives in the above
format created by any means are acceptable to Python.

@node Python Runtime Services,Custom Python Interpreters,Software Packaging and Distribution,The Python Standard Library
@anchor{library/python doc}@anchor{3b5d}@anchor{library/python python}@anchor{3b5e}@anchor{library/python python-runtime-services}@anchor{3b5f}
@section Python Runtime Services


The modules described in this chapter provide a wide range of services related
to the Python interpreter and its interaction with its environment.  Here’s an
overview:

@menu
* sys — System-specific parameters and functions:: 
* sysconfig — Provide access to Python’s configuration information:: 
* builtins — Built-in objects:: 
* __main__ — Top-level code environment:: 
* warnings — Warning control:: 
* dataclasses — Data Classes:: 
* contextlib — Utilities for with-statement contexts:: 
* abc — Abstract Base Classes:: 
* atexit — Exit handlers:: 
* traceback — Print or retrieve a stack traceback:: 
* __future__ — Future statement definitions:: 
* gc — Garbage Collector interface:: 
* inspect — Inspect live objects:: 
* site — Site-specific configuration hook:: 

@end menu

@node sys — System-specific parameters and functions,sysconfig — Provide access to Python’s configuration information,,Python Runtime Services
@anchor{library/sys doc}@anchor{3b60}@anchor{library/sys module-sys}@anchor{f7}@anchor{library/sys sys-system-specific-parameters-and-functions}@anchor{3b61}
@subsection @code{sys} — System-specific parameters and functions


@geindex module; sys


__________________________________________________________________


This module provides access to some variables used or maintained by the
interpreter and to functions that interact strongly with the interpreter. It is
always available.

@geindex abiflags (in module sys)
@anchor{library/sys sys abiflags}@anchor{57a}
@deffn {Data} sys.abiflags

On POSIX systems where Python was built with the standard @code{configure}
script, this contains the ABI flags as specified by 
@geindex Python Enhancement Proposals; PEP 3149
PEP 3149@footnote{https://www.python.org/dev/peps/pep-3149}.


Changed in version 3.8: Default flags became an empty string (@code{m} flag for pymalloc has been
removed).



New in version 3.2.

@end deffn

@geindex addaudithook() (in module sys)
@anchor{library/sys sys addaudithook}@anchor{39eb}
@deffn {Function} sys.addaudithook (hook)

Append the callable @emph{hook} to the list of active auditing hooks for the
current (sub)interpreter.

When an auditing event is raised through the @ref{11fc,,sys.audit()} function, each
hook will be called in the order it was added with the event name and the
tuple of arguments. Native hooks added by @ref{39ec,,PySys_AddAuditHook()} are
called first, followed by hooks added in the current (sub)interpreter.  Hooks
can then log the event, raise an exception to abort the operation,
or terminate the process entirely.


Calling @ref{39eb,,sys.addaudithook()} will itself raise an auditing event
named @code{sys.addaudithook} with no arguments. If any
existing hooks raise an exception derived from @ref{5af,,RuntimeError}, the
new hook will not be added and the exception suppressed. As a result,
callers cannot assume that their hook has been added unless they control
all existing hooks.


See the @ref{10cc,,audit events table} for all events raised by
CPython, and 
@geindex Python Enhancement Proposals; PEP 578
PEP 578@footnote{https://www.python.org/dev/peps/pep-0578} for the original design discussion.


New in version 3.8.



Changed in version 3.8.1: Exceptions derived from @ref{4d8,,Exception} but not @ref{5af,,RuntimeError}
are no longer suppressed.


@strong{CPython implementation detail:} When tracing is enabled (see @ref{1034,,settrace()}), Python hooks are only
traced if the callable has a @code{__cantrace__} member that is set to a
true value. Otherwise, trace functions will skip the hook.
@end deffn

@geindex argv (in module sys)
@anchor{library/sys sys argv}@anchor{e15}
@deffn {Data} sys.argv

The list of command line arguments passed to a Python script. @code{argv[0]} is the
script name (it is operating system dependent whether this is a full pathname or
not).  If the command was executed using the @ref{1180,,-c} command line option to
the interpreter, @code{argv[0]} is set to the string @code{'-c'}.  If no script name
was passed to the Python interpreter, @code{argv[0]} is the empty string.

To loop over the standard input, or the list of files given on the
command line, see the @ref{7b,,fileinput} module.

See also @ref{2c6,,sys.orig_argv}.

@cartouche
@quotation Note 
On Unix, command line arguments are passed by bytes from OS.  Python decodes
them with filesystem encoding and “surrogateescape” error handler.
When you need original bytes, you can get it by
@code{[os.fsencode(arg) for arg in sys.argv]}.
@end quotation
@end cartouche
@end deffn
@anchor{library/sys auditing}@anchor{1130}
@geindex audit() (in module sys)
@anchor{library/sys sys audit}@anchor{11fc}
@deffn {Function} sys.audit (event, *args)

@geindex auditing

Raise an auditing event and trigger any active auditing hooks.
@emph{event} is a string identifying the event, and @emph{args} may contain
optional arguments with more information about the event.  The
number and types of arguments for a given event are considered a
public and stable API and should not be modified between releases.

For example, one auditing event is named @code{os.chdir}. This event has
one argument called @emph{path} that will contain the requested new
working directory.

@ref{11fc,,sys.audit()} will call the existing auditing hooks, passing
the event name and arguments, and will re-raise the first exception
from any hook. In general, if an exception is raised, it should not
be handled and the process should be terminated as quickly as
possible. This allows hook implementations to decide how to respond
to particular events: they can merely log the event or abort the
operation by raising an exception.

Hooks are added using the @ref{39eb,,sys.addaudithook()} or
@ref{39ec,,PySys_AddAuditHook()} functions.

The native equivalent of this function is @ref{120d,,PySys_Audit()}. Using the
native function is preferred when possible.

See the @ref{10cc,,audit events table} for all events raised by
CPython.


New in version 3.8.

@end deffn

@geindex base_exec_prefix (in module sys)
@anchor{library/sys sys base_exec_prefix}@anchor{3b36}
@deffn {Data} sys.base_exec_prefix

Set during Python startup, before @code{site.py} is run, to the same value as
@ref{3b35,,exec_prefix}. If not running in a
@ref{3b33,,virtual environment}, the values will stay the same; if
@code{site.py} finds that a virtual environment is in use, the values of
@ref{3b34,,prefix} and @ref{3b35,,exec_prefix} will be changed to point to the
virtual environment, whereas @ref{34c7,,base_prefix} and
@ref{3b36,,base_exec_prefix} will remain pointing to the base Python
installation (the one which the virtual environment was created from).


New in version 3.3.

@end deffn

@geindex base_prefix (in module sys)
@anchor{library/sys sys base_prefix}@anchor{34c7}
@deffn {Data} sys.base_prefix

Set during Python startup, before @code{site.py} is run, to the same value as
@ref{3b34,,prefix}. If not running in a @ref{3b33,,virtual environment}, the values
will stay the same; if @code{site.py} finds that a virtual environment is in
use, the values of @ref{3b34,,prefix} and @ref{3b35,,exec_prefix} will be changed to
point to the virtual environment, whereas @ref{34c7,,base_prefix} and
@ref{3b36,,base_exec_prefix} will remain pointing to the base Python
installation (the one which the virtual environment was created from).


New in version 3.3.

@end deffn

@geindex byteorder (in module sys)
@anchor{library/sys sys byteorder}@anchor{1a7c}
@deffn {Data} sys.byteorder

An indicator of the native byte order.  This will have the value @code{'big'} on
big-endian (most-significant byte first) platforms, and @code{'little'} on
little-endian (least-significant byte first) platforms.
@end deffn

@geindex builtin_module_names (in module sys)
@anchor{library/sys sys builtin_module_names}@anchor{3b62}
@deffn {Data} sys.builtin_module_names

A tuple of strings containing the names of all modules that are compiled into this
Python interpreter.  (This information is not available in any other way —
@code{modules.keys()} only lists the imported modules.)

See also the @ref{2c7,,sys.stdlib_module_names} list.
@end deffn

@geindex call_tracing() (in module sys)
@anchor{library/sys sys call_tracing}@anchor{3b63}
@deffn {Function} sys.call_tracing (func, args)

Call @code{func(*args)}, while tracing is enabled.  The tracing state is saved,
and restored afterwards.  This is intended to be called from a debugger from
a checkpoint, to recursively debug some other code.
@end deffn

@geindex copyright (in module sys)
@anchor{library/sys sys copyright}@anchor{3b64}
@deffn {Data} sys.copyright

A string containing the copyright pertaining to the Python interpreter.
@end deffn

@geindex _clear_type_cache() (in module sys)
@anchor{library/sys sys _clear_type_cache}@anchor{3b65}
@deffn {Function} sys._clear_type_cache ()

Clear the internal type cache. The type cache is used to speed up attribute
and method lookups. Use the function @emph{only} to drop unnecessary references
during reference leak debugging.

This function should be used for internal and specialized purposes only.
@end deffn

@geindex _current_frames() (in module sys)
@anchor{library/sys sys _current_frames}@anchor{f95}
@deffn {Function} sys._current_frames ()

Return a dictionary mapping each thread’s identifier to the topmost stack frame
currently active in that thread at the time the function is called. Note that
functions in the @ref{11a,,traceback} module can build the call stack given such a
frame.

This is most useful for debugging deadlock:  this function does not require the
deadlocked threads’ cooperation, and such threads’ call stacks are frozen for as
long as they remain deadlocked.  The frame returned for a non-deadlocked thread
may bear no relationship to that thread’s current activity by the time calling
code examines the frame.

This function should be used for internal and specialized purposes only.

Raises an @ref{1130,,auditing event} @code{sys._current_frames} with no arguments.
@end deffn

@geindex _current_exceptions() (in module sys)
@anchor{library/sys sys _current_exceptions}@anchor{11a8}
@deffn {Function} sys._current_exceptions ()

Return a dictionary mapping each thread’s identifier to the topmost exception
currently active in that thread at the time the function is called.
If a thread is not currently handling an exception, it is not included in
the result dictionary.

This is most useful for statistical profiling.

This function should be used for internal and specialized purposes only.

Raises an @ref{1130,,auditing event} @code{sys._current_exceptions} with no arguments.
@end deffn

@geindex breakpointhook() (in module sys)
@anchor{library/sys sys breakpointhook}@anchor{5f4}
@deffn {Function} sys.breakpointhook ()

This hook function is called by built-in @ref{5db,,breakpoint()}.  By default,
it drops you into the @ref{c4,,pdb} debugger, but it can be set to any other
function so that you can choose which debugger gets used.

The signature of this function is dependent on what it calls.  For example,
the default binding (e.g. @code{pdb.set_trace()}) expects no arguments, but
you might bind it to a function that expects additional arguments
(positional and/or keyword).  The built-in @code{breakpoint()} function passes
its @code{*args} and @code{**kws} straight through.  Whatever
@code{breakpointhooks()} returns is returned from @code{breakpoint()}.

The default implementation first consults the environment variable
@geindex PYTHONBREAKPOINT
@geindex environment variable; PYTHONBREAKPOINT
@ref{5f5,,PYTHONBREAKPOINT}.  If that is set to @code{"0"} then this function
returns immediately; i.e. it is a no-op.  If the environment variable is
not set, or is set to the empty string, @code{pdb.set_trace()} is called.
Otherwise this variable should name a function to run, using Python’s
dotted-import nomenclature, e.g. @code{package.subpackage.module.function}.
In this case, @code{package.subpackage.module} would be imported and the
resulting module must have a callable named @code{function()}.  This is run,
passing in @code{*args} and @code{**kws}, and whatever @code{function()} returns,
@code{sys.breakpointhook()} returns to the built-in @ref{5db,,breakpoint()}
function.

Note that if anything goes wrong while importing the callable named by
@geindex PYTHONBREAKPOINT
@geindex environment variable; PYTHONBREAKPOINT
@ref{5f5,,PYTHONBREAKPOINT}, a @ref{5b5,,RuntimeWarning} is reported and the
breakpoint is ignored.

Also note that if @code{sys.breakpointhook()} is overridden programmatically,
@geindex PYTHONBREAKPOINT
@geindex environment variable; PYTHONBREAKPOINT
@ref{5f5,,PYTHONBREAKPOINT} is @emph{not} consulted.


New in version 3.7.

@end deffn

@geindex _debugmallocstats() (in module sys)
@anchor{library/sys sys _debugmallocstats}@anchor{3b66}
@deffn {Function} sys._debugmallocstats ()

Print low-level information to stderr about the state of CPython’s memory
allocator.

If Python is @cite{built in debug mode <debug-build>} (@ref{165a,,configure --with-pydebug option}), it also performs some expensive
internal consistency checks.


New in version 3.3.


@strong{CPython implementation detail:} This function is specific to CPython.  The exact output format is not
defined here, and may change.
@end deffn

@geindex dllhandle (in module sys)
@anchor{library/sys sys dllhandle}@anchor{3b67}
@deffn {Data} sys.dllhandle

Integer specifying the handle of the Python DLL.

@ref{1629,,Availability}: Windows.
@end deffn

@geindex displayhook() (in module sys)
@anchor{library/sys sys displayhook}@anchor{1053}
@deffn {Function} sys.displayhook (value)

If @emph{value} is not @code{None}, this function prints @code{repr(value)} to
@code{sys.stdout}, and saves @emph{value} in @code{builtins._}. If @code{repr(value)} is
not encodable to @code{sys.stdout.encoding} with @code{sys.stdout.errors} error
handler (which is probably @code{'strict'}), encode it to
@code{sys.stdout.encoding} with @code{'backslashreplace'} error handler.

@code{sys.displayhook} is called on the result of evaluating an @ref{17d2,,expression}
entered in an interactive Python session.  The display of these values can be
customized by assigning another one-argument function to @code{sys.displayhook}.

Pseudo-code:

@example
def displayhook(value):
    if value is None:
        return
    # Set '_' to None to avoid recursion
    builtins._ = None
    text = repr(value)
    try:
        sys.stdout.write(text)
    except UnicodeEncodeError:
        bytes = text.encode(sys.stdout.encoding, 'backslashreplace')
        if hasattr(sys.stdout, 'buffer'):
            sys.stdout.buffer.write(bytes)
        else:
            text = bytes.decode(sys.stdout.encoding, 'strict')
            sys.stdout.write(text)
    sys.stdout.write("\n")
    builtins._ = value
@end example


Changed in version 3.2: Use @code{'backslashreplace'} error handler on @ref{185,,UnicodeEncodeError}.

@end deffn

@geindex dont_write_bytecode (in module sys)
@anchor{library/sys sys dont_write_bytecode}@anchor{3b68}
@deffn {Data} sys.dont_write_bytecode

If this is true, Python won’t try to write @code{.pyc} files on the
import of source modules.  This value is initially set to @code{True} or
@code{False} depending on the @ref{f34,,-B} command line option and the
@geindex PYTHONDONTWRITEBYTECODE
@geindex environment variable; PYTHONDONTWRITEBYTECODE
@ref{f35,,PYTHONDONTWRITEBYTECODE} environment variable, but you can set it
yourself to control bytecode file generation.
@end deffn

@geindex pycache_prefix (in module sys)
@anchor{library/sys sys pycache_prefix}@anchor{4ab}
@deffn {Data} sys.pycache_prefix

If this is set (not @code{None}), Python will write bytecode-cache @code{.pyc}
files to (and read them from) a parallel directory tree rooted at this
directory, rather than from @code{__pycache__} directories in the source code
tree. Any @code{__pycache__} directories in the source code tree will be ignored
and new @cite{.pyc} files written within the pycache prefix. Thus if you use
@ref{1f,,compileall} as a pre-build step, you must ensure you run it with the
same pycache prefix (if any) that you will use at runtime.

A relative path is interpreted relative to the current working directory.

This value is initially set based on the value of the @ref{159,,-X}
@code{pycache_prefix=PATH} command-line option or the
@geindex PYTHONPYCACHEPREFIX
@geindex environment variable; PYTHONPYCACHEPREFIX
@ref{4aa,,PYTHONPYCACHEPREFIX} environment variable (command-line takes
precedence). If neither are set, it is @code{None}.


New in version 3.8.

@end deffn

@geindex excepthook() (in module sys)
@anchor{library/sys sys excepthook}@anchor{28e}
@deffn {Function} sys.excepthook (type, value, traceback)

This function prints out a given traceback and exception to @code{sys.stderr}.

When an exception is raised and uncaught, the interpreter calls
@code{sys.excepthook} with three arguments, the exception class, exception
instance, and a traceback object.  In an interactive session this happens just
before control is returned to the prompt; in a Python program this happens just
before the program exits.  The handling of such top-level exceptions can be
customized by assigning another three-argument function to @code{sys.excepthook}.


Raise an auditing event @code{sys.excepthook} with arguments @code{hook},
@code{type}, @code{value}, @code{traceback} when an uncaught exception occurs.
If no hook has been set, @code{hook} may be @code{None}. If any hook raises
an exception derived from @ref{5af,,RuntimeError} the call to the hook will
be suppressed. Otherwise, the audit hook exception will be reported as
unraisable and @code{sys.excepthook} will be called.



@subsubheading See also


The @ref{18d,,sys.unraisablehook()} function handles unraisable exceptions
and the @ref{2d1,,threading.excepthook()} function handles exception raised
by @ref{54e,,threading.Thread.run()}.

@end deffn

@geindex __breakpointhook__ (in module sys)
@geindex __displayhook__ (in module sys)
@geindex __excepthook__ (in module sys)
@geindex __unraisablehook__ (in module sys)
@anchor{library/sys sys __breakpointhook__}@anchor{3b69}
@deffn {Data} sys.__breakpointhook__
@anchor{library/sys sys __displayhook__}@anchor{3b6a}
@deffnx {Data} sys.__displayhook__
@anchor{library/sys sys __excepthook__}@anchor{3b6b}
@deffnx {Data} sys.__excepthook__
@anchor{library/sys sys __unraisablehook__}@anchor{3b6c}
@deffnx {Data} sys.__unraisablehook__

These objects contain the original values of @code{breakpointhook},
@code{displayhook}, @code{excepthook}, and @code{unraisablehook} at the start of the
program.  They are saved so that @code{breakpointhook}, @code{displayhook} and
@code{excepthook}, @code{unraisablehook} can be restored in case they happen to
get replaced with broken or alternative objects.


New in version 3.7: __breakpointhook__



New in version 3.8: __unraisablehook__

@end deffn

@geindex exception() (in module sys)
@anchor{library/sys sys exception}@anchor{190}
@deffn {Function} sys.exception ()

This function returns the exception instance that is currently being
handled.  This exception is specific both to the current thread and
to the current stack frame.  If the current stack frame is not handling
an exception, the exception is taken from the calling stack frame, or its
caller, and so on until a stack frame is found that is handling an
exception.  Here, “handling an exception” is defined as “executing an
except clause.” For any stack frame, only the exception being currently
handled is accessible.

@geindex object; traceback

If no exception is being handled anywhere on the stack, @code{None} is
returned.


New in version 3.11.

@end deffn

@geindex exc_info() (in module sys)
@anchor{library/sys sys exc_info}@anchor{18f}
@deffn {Function} sys.exc_info ()

This function returns the old-style representation of the handled
exception. If an exception @code{e} is currently handled (so
@ref{190,,exception()} would return @code{e}), @ref{18f,,exc_info()} returns the
tuple @code{(type(e), e, e.__traceback__)}.
That is, a tuple containing the type of the exception (a subclass of
@ref{15f,,BaseException}), the exception itself, and a @ref{610,,traceback object} which typically encapsulates the call
stack at the point where the exception last occurred.

@geindex object; traceback

If no exception is being handled anywhere on the stack, this function
return a tuple containing three @code{None} values.


Changed in version 3.11: The @code{type} and @code{traceback} fields are now derived from the @code{value}
(the exception instance), so when an exception is modified while it is
being handled, the changes are reflected in the results of subsequent
calls to @ref{18f,,exc_info()}.

@end deffn

@geindex exec_prefix (in module sys)
@anchor{library/sys sys exec_prefix}@anchor{3b35}
@deffn {Data} sys.exec_prefix

A string giving the site-specific directory prefix where the platform-dependent
Python files are installed; by default, this is also @code{'/usr/local'}.  This can
be set at build time with the @code{--exec-prefix} argument to the
@code{configure} script.  Specifically, all configuration files (e.g. the
@code{pyconfig.h} header file) are installed in the directory
@code{@emph{exec_prefix}/lib/python@emph{X.Y}/config}, and shared library modules are
installed in @code{@emph{exec_prefix}/lib/python@emph{X.Y}/lib-dynload}, where @emph{X.Y}
is the version number of Python, for example @code{3.2}.

@cartouche
@quotation Note 
If a @ref{3b33,,virtual environment} is in effect, this
value will be changed in @code{site.py} to point to the virtual environment.
The value for the Python installation will still be available, via
@ref{3b36,,base_exec_prefix}.
@end quotation
@end cartouche
@end deffn

@geindex executable (in module sys)
@anchor{library/sys sys executable}@anchor{585}
@deffn {Data} sys.executable

A string giving the absolute path of the executable binary for the Python
interpreter, on systems where this makes sense. If Python is unable to retrieve
the real path to its executable, @ref{585,,sys.executable} will be an empty string
or @code{None}.
@end deffn

@geindex exit() (in module sys)
@anchor{library/sys sys exit}@anchor{ee4}
@deffn {Function} sys.exit ([arg])

Raise a @ref{892,,SystemExit} exception, signaling an intention to exit the interpreter.

The optional argument @emph{arg} can be an integer giving the exit status
(defaulting to zero), or another type of object.  If it is an integer, zero
is considered “successful termination” and any nonzero value is considered
“abnormal termination” by shells and the like.  Most systems require it to be
in the range 0–127, and produce undefined results otherwise.  Some systems
have a convention for assigning specific meanings to specific exit codes, but
these are generally underdeveloped; Unix programs generally use 2 for command
line syntax errors and 1 for all other kind of errors.  If another type of
object is passed, @code{None} is equivalent to passing zero, and any other
object is printed to @ref{42d,,stderr} and results in an exit code of 1.  In
particular, @code{sys.exit("some error message")} is a quick way to exit a
program when an error occurs.

Since @ref{1a6b,,exit()} ultimately “only” raises an exception, it will only exit
the process when called from the main thread, and the exception is not
intercepted. Cleanup actions specified by finally clauses of @ref{f6d,,try} statements
are honored, and it is possible to intercept the exit attempt at an outer level.


Changed in version 3.6: If an error occurs in the cleanup after the Python interpreter
has caught @ref{892,,SystemExit} (such as an error flushing buffered data
in the standard streams), the exit status is changed to 120.

@end deffn

@geindex flags (in module sys)
@anchor{library/sys sys flags}@anchor{22f}
@deffn {Data} sys.flags

The @ref{ce9,,named tuple} @emph{flags} exposes the status of command line
flags. The attributes are read only.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

attribute

@tab

flag

@item

@code{debug}

@tab

@ref{1607,,-d}

@item

@ref{9c,,inspect}

@tab

@ref{1018,,-i}

@item

@code{interactive}

@tab

@ref{1018,,-i}

@item

@code{isolated}

@tab

@ref{461,,-I}

@item

@code{optimize}

@tab

@ref{912,,-O} or @ref{913,,-OO}

@item

@ref{3b68,,dont_write_bytecode}

@tab

@ref{f34,,-B}

@item

@code{no_user_site}

@tab

@ref{f15,,-s}

@item

@code{no_site}

@tab

@ref{d4e,,-S}

@item

@code{ignore_environment}

@tab

@ref{460,,-E}

@item

@code{verbose}

@tab

@ref{f82,,-v}

@item

@code{bytes_warning}

@tab

@ref{6dc,,-b}

@item

@code{quiet}

@tab

@ref{160d,,-q}

@item

@code{hash_randomization}

@tab

@ref{1357,,-R}

@item

@code{dev_mode}

@tab

@ref{159,,-X dev} (@ref{3ac,,Python Development Mode})

@item

@code{utf8_mode}

@tab

@ref{159,,-X utf8}

@end multitable



Changed in version 3.2: Added @code{quiet} attribute for the new @ref{160d,,-q} flag.



New in version 3.2.3: The @code{hash_randomization} attribute.



Changed in version 3.3: Removed obsolete @code{division_warning} attribute.



Changed in version 3.4: Added @code{isolated} attribute for @ref{461,,-I} @code{isolated} flag.



Changed in version 3.7: Added the @code{dev_mode} attribute for the new @ref{3ac,,Python Development Mode} and the @code{utf8_mode} attribute for the new  @ref{159,,-X}
@code{utf8} flag.

@end deffn

@geindex float_info (in module sys)
@anchor{library/sys sys float_info}@anchor{1290}
@deffn {Data} sys.float_info

A @ref{ce9,,named tuple} holding information about the float type. It
contains low level information about the precision and internal
representation.  The values correspond to the various floating-point
constants defined in the standard header file @code{float.h} for the ‘C’
programming language; see section 5.2.4.2.2 of the 1999 ISO/IEC C standard
@ref{3b6d,,[C99]}, ‘Characteristics of floating types’, for details.


@multitable {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

attribute

@tab

float.h macro

@tab

explanation

@item

@code{epsilon}

@tab

DBL_EPSILON

@tab

difference between 1.0 and the least value
greater than 1.0 that is representable as a float

See also @ref{404,,math.ulp()}.

@item

@code{dig}

@tab

DBL_DIG

@tab

maximum number of decimal digits that can be
faithfully represented in a float;  see below

@item

@code{mant_dig}

@tab

DBL_MANT_DIG

@tab

float precision: the number of base-@code{radix}
digits in the significand of a float

@item

@ref{a7e,,max}

@tab

DBL_MAX

@tab

maximum representable positive finite float

@item

@code{max_exp}

@tab

DBL_MAX_EXP

@tab

maximum integer @emph{e} such that @code{radix**(e-1)} is
a representable finite float

@item

@code{max_10_exp}

@tab

DBL_MAX_10_EXP

@tab

maximum integer @emph{e} such that @code{10**e} is in the
range of representable finite floats

@item

@ref{a7d,,min}

@tab

DBL_MIN

@tab

minimum representable positive @emph{normalized} float

Use @ref{404,,math.ulp(0.0)} to get the
smallest positive @emph{denormalized} representable
float.

@item

@code{min_exp}

@tab

DBL_MIN_EXP

@tab

minimum integer @emph{e} such that @code{radix**(e-1)} is
a normalized float

@item

@code{min_10_exp}

@tab

DBL_MIN_10_EXP

@tab

minimum integer @emph{e} such that @code{10**e} is a
normalized float

@item

@code{radix}

@tab

FLT_RADIX

@tab

radix of exponent representation

@item

@code{rounds}

@tab

FLT_ROUNDS

@tab

integer constant representing the rounding mode
used for arithmetic operations.  This reflects
the value of the system FLT_ROUNDS macro at
interpreter startup time.  See section 5.2.4.2.2
of the C99 standard for an explanation of the
possible values and their meanings.

@end multitable


The attribute @code{sys.float_info.dig} needs further explanation.  If
@code{s} is any string representing a decimal number with at most
@code{sys.float_info.dig} significant digits, then converting @code{s} to a
float and back again will recover a string representing the same decimal
value:

@example
>>> import sys
>>> sys.float_info.dig
15
>>> s = '3.14159265358979'    # decimal string with 15 significant digits
>>> format(float(s), '.15g')  # convert to float and back -> same value
'3.14159265358979'
@end example

But for strings with more than @code{sys.float_info.dig} significant digits,
this isn’t always true:

@example
>>> s = '9876543211234567'    # 16 significant digits is too many!
>>> format(float(s), '.16g')  # conversion changes value
'9876543211234568'
@end example
@end deffn

@geindex float_repr_style (in module sys)
@anchor{library/sys sys float_repr_style}@anchor{e97}
@deffn {Data} sys.float_repr_style

A string indicating how the @ref{27d,,repr()} function behaves for
floats.  If the string has value @code{'short'} then for a finite
float @code{x}, @code{repr(x)} aims to produce a short string with the
property that @code{float(repr(x)) == x}.  This is the usual behaviour
in Python 3.1 and later.  Otherwise, @code{float_repr_style} has value
@code{'legacy'} and @code{repr(x)} behaves in the same way as it did in
versions of Python prior to 3.1.


New in version 3.1.

@end deffn

@geindex getallocatedblocks() (in module sys)
@anchor{library/sys sys getallocatedblocks}@anchor{b4a}
@deffn {Function} sys.getallocatedblocks ()

Return the number of memory blocks currently allocated by the interpreter,
regardless of their size.  This function is mainly useful for tracking
and debugging memory leaks.  Because of the interpreter’s internal
caches, the result can vary from call to call; you may have to call
@ref{3b65,,_clear_type_cache()} and @ref{54b,,gc.collect()} to get more
predictable results.

If a Python build or implementation cannot reasonably compute this
information, @ref{b4a,,getallocatedblocks()} is allowed to return 0 instead.


New in version 3.4.

@end deffn

@geindex getandroidapilevel() (in module sys)
@anchor{library/sys sys getandroidapilevel}@anchor{6bb}
@deffn {Function} sys.getandroidapilevel ()

Return the build time API version of Android as an integer.

@ref{1629,,Availability}: Android.


New in version 3.7.

@end deffn

@geindex getdefaultencoding() (in module sys)
@anchor{library/sys sys getdefaultencoding}@anchor{3b6e}
@deffn {Function} sys.getdefaultencoding ()

Return the name of the current default string encoding used by the Unicode
implementation.
@end deffn

@geindex getdlopenflags() (in module sys)
@anchor{library/sys sys getdlopenflags}@anchor{1044}
@deffn {Function} sys.getdlopenflags ()

Return the current value of the flags that are used for
@code{dlopen()} calls.  Symbolic names for the flag values can be
found in the @ref{c0,,os} module (@code{RTLD_xxx} constants, e.g.
@ref{cab,,os.RTLD_LAZY}).

@ref{1629,,Availability}: Unix.
@end deffn

@geindex getfilesystemencoding() (in module sys)
@anchor{library/sys sys getfilesystemencoding}@anchor{79e}
@deffn {Function} sys.getfilesystemencoding ()

Get the @ref{1628,,filesystem encoding}:
the encoding used with the @ref{1628,,filesystem error handler} to convert between Unicode filenames and bytes
filenames. The filesystem error handler is returned from
@ref{79e,,getfilesystemencoding()}.

For best compatibility, str should be used for filenames in all cases,
although representing filenames as bytes is also supported. Functions
accepting or returning filenames should support either str or bytes and
internally convert to the system’s preferred representation.

@ref{799,,os.fsencode()} and @ref{798,,os.fsdecode()} should be used to ensure that
the correct encoding and errors mode are used.

The @ref{1628,,filesystem encoding and error handler} are configured at Python
startup by the @code{PyConfig_Read()} function: see
@ref{2378,,filesystem_encoding} and
@ref{2379,,filesystem_errors} members of @ref{206,,PyConfig}.


Changed in version 3.2: @ref{79e,,getfilesystemencoding()} result cannot be @code{None} anymore.



Changed in version 3.6: Windows is no longer guaranteed to return @code{'mbcs'}. See 
@geindex Python Enhancement Proposals; PEP 529
PEP 529@footnote{https://www.python.org/dev/peps/pep-0529}
and @ref{7a0,,_enablelegacywindowsfsencoding()} for more information.



Changed in version 3.7: Return @code{'utf-8'} if the @ref{5f2,,Python UTF-8 Mode} is
enabled.

@end deffn

@geindex getfilesystemencodeerrors() (in module sys)
@anchor{library/sys sys getfilesystemencodeerrors}@anchor{82f}
@deffn {Function} sys.getfilesystemencodeerrors ()

Get the @ref{1628,,filesystem error handler}: the error handler used with the @ref{1628,,filesystem encoding} to convert between Unicode
filenames and bytes filenames. The filesystem encoding is returned from
@ref{79e,,getfilesystemencoding()}.

@ref{799,,os.fsencode()} and @ref{798,,os.fsdecode()} should be used to ensure that
the correct encoding and errors mode are used.

The @ref{1628,,filesystem encoding and error handler} are configured at Python
startup by the @code{PyConfig_Read()} function: see
@ref{2378,,filesystem_encoding} and
@ref{2379,,filesystem_errors} members of @ref{206,,PyConfig}.


New in version 3.6.

@end deffn

@geindex getrefcount() (in module sys)
@anchor{library/sys sys getrefcount}@anchor{3b6f}
@deffn {Function} sys.getrefcount (object)

Return the reference count of the @emph{object}.  The count returned is generally one
higher than you might expect, because it includes the (temporary) reference as
an argument to @ref{3b6f,,getrefcount()}.
@end deffn

@geindex getrecursionlimit() (in module sys)
@anchor{library/sys sys getrecursionlimit}@anchor{106a}
@deffn {Function} sys.getrecursionlimit ()

Return the current value of the recursion limit, the maximum depth of the Python
interpreter stack.  This limit prevents infinite recursion from causing an
overflow of the C stack and crashing Python.  It can be set by
@ref{106b,,setrecursionlimit()}.
@end deffn

@geindex getsizeof() (in module sys)
@anchor{library/sys sys getsizeof}@anchor{3afa}
@deffn {Function} sys.getsizeof (object[, default])

Return the size of an object in bytes. The object can be any type of
object. All built-in objects will return correct results, but this
does not have to hold true for third-party extensions as it is implementation
specific.

Only the memory consumption directly attributed to the object is
accounted for, not the memory consumption of objects it refers to.

If given, @emph{default} will be returned if the object does not provide means to
retrieve the size.  Otherwise a @ref{162,,TypeError} will be raised.

@ref{3afa,,getsizeof()} calls the object’s @code{__sizeof__} method and adds an
additional garbage collector overhead if the object is managed by the garbage
collector.

See recursive sizeof recipe@footnote{https://code.activestate.com/recipes/577504}
for an example of using @ref{3afa,,getsizeof()} recursively to find the size of
containers and all their contents.
@end deffn

@geindex getswitchinterval() (in module sys)
@anchor{library/sys sys getswitchinterval}@anchor{448}
@deffn {Function} sys.getswitchinterval ()

Return the interpreter’s “thread switch interval”; see
@ref{449,,setswitchinterval()}.


New in version 3.2.

@end deffn

@geindex _getframe() (in module sys)
@anchor{library/sys sys _getframe}@anchor{1057}
@deffn {Function} sys._getframe ([depth])

Return a frame object from the call stack.  If optional integer @emph{depth} is
given, return the frame object that many calls below the top of the stack.  If
that is deeper than the call stack, @ref{17c,,ValueError} is raised.  The default
for @emph{depth} is zero, returning the frame at the top of the call stack.

Raises an @ref{1130,,auditing event} @code{sys._getframe} with no arguments.

@strong{CPython implementation detail:} This function should be used for internal and specialized purposes only.
It is not guaranteed to exist in all implementations of Python.
@end deffn

@geindex getprofile() (in module sys)
@anchor{library/sys sys getprofile}@anchor{f42}
@deffn {Function} sys.getprofile ()

@geindex profile function
@geindex profiler

Get the profiler function as set by @ref{1033,,setprofile()}.
@end deffn

@geindex gettrace() (in module sys)
@anchor{library/sys sys gettrace}@anchor{f43}
@deffn {Function} sys.gettrace ()

@geindex trace function
@geindex debugger

Get the trace function as set by @ref{1034,,settrace()}.

@strong{CPython implementation detail:} The @ref{f43,,gettrace()} function is intended only for implementing debuggers,
profilers, coverage tools and the like.  Its behavior is part of the
implementation platform, rather than part of the language definition, and
thus may not be available in all Python implementations.
@end deffn

@geindex getwindowsversion() (in module sys)
@anchor{library/sys sys getwindowsversion}@anchor{830}
@deffn {Function} sys.getwindowsversion ()

Return a named tuple describing the Windows version
currently running.  The named elements are @emph{major}, @emph{minor},
@emph{build}, @emph{platform}, @emph{service_pack}, @emph{service_pack_minor},
@emph{service_pack_major}, @emph{suite_mask}, @emph{product_type} and
@emph{platform_version}. @emph{service_pack} contains a string,
@emph{platform_version} a 3-tuple and all other values are
integers. The components can also be accessed by name, so
@code{sys.getwindowsversion()[0]} is equivalent to
@code{sys.getwindowsversion().major}. For compatibility with prior
versions, only the first 5 elements are retrievable by indexing.

@emph{platform} will be @code{2 (VER_PLATFORM_WIN32_NT)}.

@emph{product_type} may be one of the following values:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Meaning

@item

@code{1 (VER_NT_WORKSTATION)}

@tab

The system is a workstation.

@item

@code{2 (VER_NT_DOMAIN_CONTROLLER)}

@tab

The system is a domain
controller.

@item

@code{3 (VER_NT_SERVER)}

@tab

The system is a server, but not
a domain controller.

@end multitable


This function wraps the Win32 @code{GetVersionEx()} function; see the
Microsoft documentation on @code{OSVERSIONINFOEX()} for more information
about these fields.

@emph{platform_version} returns the major version, minor version and
build number of the current operating system, rather than the version that
is being emulated for the process. It is intended for use in logging rather
than for feature detection.

@cartouche
@quotation Note 
@emph{platform_version} derives the version from kernel32.dll which can be of a different
version than the OS version. Please use @ref{c9,,platform} module for achieving accurate
OS version.
@end quotation
@end cartouche

@ref{1629,,Availability}: Windows.


Changed in version 3.2: Changed to a named tuple and added @emph{service_pack_minor},
@emph{service_pack_major}, @emph{suite_mask}, and @emph{product_type}.



Changed in version 3.6: Added @emph{platform_version}

@end deffn

@geindex get_asyncgen_hooks() (in module sys)
@anchor{library/sys sys get_asyncgen_hooks}@anchor{3b70}
@deffn {Function} sys.get_asyncgen_hooks ()

Returns an @emph{asyncgen_hooks} object, which is similar to a
@ref{3e4,,namedtuple} of the form @cite{(firstiter@comma{} finalizer)},
where @emph{firstiter} and @emph{finalizer} are expected to be either @code{None} or
functions which take an @ref{3b71,,asynchronous generator iterator} as an
argument, and are used to schedule finalization of an asynchronous
generator by an event loop.


New in version 3.6: See 
@geindex Python Enhancement Proposals; PEP 525
PEP 525@footnote{https://www.python.org/dev/peps/pep-0525} for more details.


@cartouche
@quotation Note 
This function has been added on a provisional basis (see 
@geindex Python Enhancement Proposals; PEP 411
PEP 411@footnote{https://www.python.org/dev/peps/pep-0411}
for details.)
@end quotation
@end cartouche
@end deffn

@geindex get_coroutine_origin_tracking_depth() (in module sys)
@anchor{library/sys sys get_coroutine_origin_tracking_depth}@anchor{6bc}
@deffn {Function} sys.get_coroutine_origin_tracking_depth ()

Get the current coroutine origin tracking depth, as set by
@ref{6bd,,set_coroutine_origin_tracking_depth()}.


New in version 3.7.


@cartouche
@quotation Note 
This function has been added on a provisional basis (see 
@geindex Python Enhancement Proposals; PEP 411
PEP 411@footnote{https://www.python.org/dev/peps/pep-0411}
for details.)  Use it only for debugging purposes.
@end quotation
@end cartouche
@end deffn

@geindex hash_info (in module sys)
@anchor{library/sys sys hash_info}@anchor{b85}
@deffn {Data} sys.hash_info

A @ref{ce9,,named tuple} giving parameters of the numeric hash
implementation.  For more details about hashing of numeric types, see
@ref{1a81,,Hashing of numeric types}.


@multitable {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

attribute

@tab

explanation

@item

@code{width}

@tab

width in bits used for hash values

@item

@code{modulus}

@tab

prime modulus P used for numeric hash scheme

@item

@code{inf}

@tab

hash value returned for a positive infinity

@item

@code{nan}

@tab

(this attribute is no longer used)

@item

@code{imag}

@tab

multiplier used for the imaginary part of a
complex number

@item

@code{algorithm}

@tab

name of the algorithm for hashing of str, bytes,
and memoryview

@item

@code{hash_bits}

@tab

internal output size of the hash algorithm

@item

@code{seed_bits}

@tab

size of the seed key of the hash algorithm

@end multitable



New in version 3.2.



Changed in version 3.4: Added @emph{algorithm}, @emph{hash_bits} and @emph{seed_bits}

@end deffn

@geindex hexversion (in module sys)
@anchor{library/sys sys hexversion}@anchor{3b72}
@deffn {Data} sys.hexversion

The version number encoded as a single integer.  This is guaranteed to increase
with each version, including proper support for non-production releases.  For
example, to test that the Python interpreter is at least version 1.5.2, use:

@example
if sys.hexversion >= 0x010502F0:
    # use some advanced feature
    ...
else:
    # use an alternative implementation or warn the user
    ...
@end example

This is called @code{hexversion} since it only really looks meaningful when viewed
as the result of passing it to the built-in @ref{e6f,,hex()} function.  The
@ref{ce9,,named tuple}  @ref{d45,,sys.version_info} may be used for a more
human-friendly encoding of the same information.

More details of @code{hexversion} can be found at @ref{3b73,,API and ABI Versioning}.
@end deffn

@geindex implementation (in module sys)
@anchor{library/sys sys implementation}@anchor{bff}
@deffn {Data} sys.implementation

An object containing information about the implementation of the
currently running Python interpreter.  The following attributes are
required to exist in all Python implementations.

@emph{name} is the implementation’s identifier, e.g. @code{'cpython'}.  The actual
string is defined by the Python implementation, but it is guaranteed to be
lower case.

@emph{version} is a named tuple, in the same format as
@ref{d45,,sys.version_info}.  It represents the version of the Python
@emph{implementation}.  This has a distinct meaning from the specific
version of the Python @emph{language} to which the currently running
interpreter conforms, which @code{sys.version_info} represents.  For
example, for PyPy 1.8 @code{sys.implementation.version} might be
@code{sys.version_info(1, 8, 0, 'final', 0)}, whereas @code{sys.version_info}
would be @code{sys.version_info(2, 7, 2, 'final', 0)}.  For CPython they
are the same value, since it is the reference implementation.

@emph{hexversion} is the implementation version in hexadecimal format, like
@ref{3b72,,sys.hexversion}.

@emph{cache_tag} is the tag used by the import machinery in the filenames of
cached modules.  By convention, it would be a composite of the
implementation’s name and version, like @code{'cpython-33'}.  However, a
Python implementation may use some other value if appropriate.  If
@code{cache_tag} is set to @code{None}, it indicates that module caching should
be disabled.

@ref{bff,,sys.implementation} may contain additional attributes specific to
the Python implementation.  These non-standard attributes must start with
an underscore, and are not described here.  Regardless of its contents,
@ref{bff,,sys.implementation} will not change during a run of the interpreter,
nor between implementation versions.  (It may change between Python
language versions, however.)  See 
@geindex Python Enhancement Proposals; PEP 421
PEP 421@footnote{https://www.python.org/dev/peps/pep-0421} for more information.


New in version 3.3.


@cartouche
@quotation Note 
The addition of new required attributes must go through the normal PEP
process. See 
@geindex Python Enhancement Proposals; PEP 421
PEP 421@footnote{https://www.python.org/dev/peps/pep-0421} for more information.
@end quotation
@end cartouche
@end deffn

@geindex int_info (in module sys)
@anchor{library/sys sys int_info}@anchor{e39}
@deffn {Data} sys.int_info

A @ref{ce9,,named tuple} that holds information about Python’s internal
representation of integers.  The attributes are read only.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Attribute

@tab

Explanation

@item

@code{bits_per_digit}

@tab

number of bits held in each digit.  Python
integers are stored internally in base
@code{2**int_info.bits_per_digit}

@item

@code{sizeof_digit}

@tab

size in bytes of the C type used to
represent a digit

@end multitable



New in version 3.1.

@end deffn

@geindex __interactivehook__ (in module sys)
@anchor{library/sys sys __interactivehook__}@anchor{b4c}
@deffn {Data} sys.__interactivehook__

When this attribute exists, its value is automatically called (with no
arguments) when the interpreter is launched in @ref{b4b,,interactive mode}.  This is done after the 
@geindex PYTHONSTARTUP
@geindex environment variable; PYTHONSTARTUP
@ref{b4d,,PYTHONSTARTUP} file is
read, so that you can set this hook there.  The @ref{e6,,site} module
@ref{b4e,,sets this}.


Raises an @ref{1130,,auditing event}
@code{cpython.run_interactivehook} with the hook object as the argument when
the hook is called on startup.



New in version 3.4.

@end deffn

@geindex intern() (in module sys)
@anchor{library/sys sys intern}@anchor{e76}
@deffn {Function} sys.intern (string)

Enter @emph{string} in the table of “interned” strings and return the interned string
– which is @emph{string} itself or a copy. Interning strings is useful to gain a
little performance on dictionary lookup – if the keys in a dictionary are
interned, and the lookup key is interned, the key comparisons (after hashing)
can be done by a pointer compare instead of a string compare.  Normally, the
names used in Python programs are automatically interned, and the dictionaries
used to hold module, class or instance attributes have interned keys.

Interned strings are not immortal; you must keep a reference to the return
value of @ref{e76,,intern()} around to benefit from it.
@end deffn

@geindex is_finalizing() (in module sys)
@anchor{library/sys sys is_finalizing}@anchor{9e1}
@deffn {Function} sys.is_finalizing ()

Return @ref{748,,True} if the Python interpreter is
@ref{9e2,,shutting down}, @ref{659,,False} otherwise.


New in version 3.5.

@end deffn

@geindex last_type (in module sys)
@geindex last_value (in module sys)
@geindex last_traceback (in module sys)
@anchor{library/sys sys last_type}@anchor{e66}
@deffn {Data} sys.last_type
@anchor{library/sys sys last_value}@anchor{3b74}
@deffnx {Data} sys.last_value
@anchor{library/sys sys last_traceback}@anchor{3a58}
@deffnx {Data} sys.last_traceback

These three variables are not always defined; they are set when an exception is
not handled and the interpreter prints an error message and a stack traceback.
Their intended use is to allow an interactive user to import a debugger module
and engage in post-mortem debugging without having to re-execute the command
that caused the error.  (Typical use is @code{import pdb; pdb.pm()} to enter the
post-mortem debugger; see @ref{c4,,pdb} module for
more information.)

The meaning of the variables is the same as that of the return values from
@ref{18f,,exc_info()} above.
@end deffn

@geindex maxsize (in module sys)
@anchor{library/sys sys maxsize}@anchor{d69}
@deffn {Data} sys.maxsize

An integer giving the maximum value a variable of type @code{Py_ssize_t} can
take.  It’s usually @code{2**31 - 1} on a 32-bit platform and @code{2**63 - 1} on a
64-bit platform.
@end deffn

@geindex maxunicode (in module sys)
@anchor{library/sys sys maxunicode}@anchor{be1}
@deffn {Data} sys.maxunicode

An integer giving the value of the largest Unicode code point,
i.e. @code{1114111} (@code{0x10FFFF} in hexadecimal).


Changed in version 3.3: Before 
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393}, @code{sys.maxunicode} used to be either @code{0xFFFF}
or @code{0x10FFFF}, depending on the configuration option that specified
whether Unicode characters were stored as UCS-2 or UCS-4.

@end deffn

@geindex meta_path (in module sys)
@anchor{library/sys sys meta_path}@anchor{87c}
@deffn {Data} sys.meta_path

A list of @ref{c05,,meta path finder} objects that have their
@ref{308,,find_spec()} methods called to see if one
of the objects can find the module to be imported. The
@ref{308,,find_spec()} method is called with at
least the absolute name of the module being imported. If the module to be
imported is contained in a package, then the parent package’s @ref{1550,,__path__}
attribute is passed in as a second argument. The method returns a
@ref{3b75,,module spec}, or @code{None} if the module cannot be found.


@subsubheading See also



@table @asis

@item @ref{30e,,importlib.abc.MetaPathFinder}

The abstract base class defining the interface of finder objects on
@ref{87c,,meta_path}.

@item @ref{1853,,importlib.machinery.ModuleSpec}

The concrete class which
@ref{308,,find_spec()} should return
instances of.
@end table



Changed in version 3.4: @ref{3b75,,Module specs} were introduced in Python 3.4, by
@geindex Python Enhancement Proposals; PEP 451
PEP 451@footnote{https://www.python.org/dev/peps/pep-0451}. Earlier versions of Python looked for a method called
@ref{306,,find_module()}.
This is still called as a fallback if a @ref{87c,,meta_path} entry doesn’t
have a @ref{308,,find_spec()} method.

@end deffn

@geindex modules (in module sys)
@anchor{library/sys sys modules}@anchor{11f7}
@deffn {Data} sys.modules

This is a dictionary that maps module names to modules which have already been
loaded.  This can be manipulated to force reloading of modules and other tricks.
However, replacing the dictionary will not necessarily work as expected and
deleting essential items from the dictionary may cause Python to fail.  If
you want to iterate over this global dictionary always use
@code{sys.modules.copy()} or @code{tuple(sys.modules)} to avoid exceptions as its
size may change during iteration as a side effect of code or activity in
other threads.
@end deffn

@geindex orig_argv (in module sys)
@anchor{library/sys sys orig_argv}@anchor{2c6}
@deffn {Data} sys.orig_argv

The list of the original command line arguments passed to the Python
executable.

See also @ref{e15,,sys.argv}.


New in version 3.10.

@end deffn

@geindex path (in module sys)
@anchor{library/sys sys path}@anchor{489}
@deffn {Data} sys.path

@geindex module; search; path

A list of strings that specifies the search path for modules. Initialized from
the environment variable 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH}, plus an installation-dependent
default.

As initialized upon program startup, the first item of this list, @code{path[0]},
is the directory containing the script that was used to invoke the Python
interpreter.  If the script directory is not available (e.g.  if the interpreter
is invoked interactively or if the script is read from standard input),
@code{path[0]} is the empty string, which directs Python to search modules in the
current directory first.  Notice that the script directory is inserted @emph{before}
the entries inserted as a result of 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH}.

A program is free to modify this list for its own purposes.  Only strings
and bytes should be added to @ref{489,,sys.path}; all other data types are
ignored during import.


@subsubheading See also


Module @ref{e6,,site} This describes how to use .pth files to extend
@ref{489,,sys.path}.

@end deffn

@geindex path_hooks (in module sys)
@anchor{library/sys sys path_hooks}@anchor{bb5}
@deffn {Data} sys.path_hooks

A list of callables that take a path argument to try to create a
@ref{1847,,finder} for the path. If a finder can be created, it is to be
returned by the callable, else raise @ref{3b8,,ImportError}.

Originally specified in 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302}.
@end deffn

@geindex path_importer_cache (in module sys)
@anchor{library/sys sys path_importer_cache}@anchor{74c}
@deffn {Data} sys.path_importer_cache

A dictionary acting as a cache for @ref{1847,,finder} objects. The keys are
paths that have been passed to @ref{bb5,,sys.path_hooks} and the values are
the finders that are found. If a path is a valid file system path but no
finder is found on @ref{bb5,,sys.path_hooks} then @code{None} is
stored.

Originally specified in 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302}.


Changed in version 3.3: @code{None} is stored instead of @ref{c0c,,imp.NullImporter} when no finder
is found.

@end deffn

@geindex platform (in module sys)
@anchor{library/sys sys platform}@anchor{5a6}
@deffn {Data} sys.platform

This string contains a platform identifier that can be used to append
platform-specific components to @ref{489,,sys.path}, for instance.

For Unix systems, except on Linux and AIX, this is the lowercased OS name as
returned by @code{uname -s} with the first part of the version as returned by
@code{uname -r} appended, e.g. @code{'sunos5'} or @code{'freebsd8'}, @emph{at the time
when Python was built}.  Unless you want to test for a specific system
version, it is therefore recommended to use the following idiom:

@example
if sys.platform.startswith('freebsd'):
    # FreeBSD-specific code here...
elif sys.platform.startswith('linux'):
    # Linux-specific code here...
elif sys.platform.startswith('aix'):
    # AIX-specific code here...
@end example

For other systems, the values are:


@multitable {xxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

System

@tab

@code{platform} value

@item

AIX

@tab

@code{'aix'}

@item

Linux

@tab

@code{'linux'}

@item

Windows

@tab

@code{'win32'}

@item

Windows/Cygwin

@tab

@code{'cygwin'}

@item

macOS

@tab

@code{'darwin'}

@end multitable



Changed in version 3.3: On Linux, @ref{5a6,,sys.platform} doesn’t contain the major version anymore.
It is always @code{'linux'}, instead of @code{'linux2'} or @code{'linux3'}.  Since
older Python versions include the version number, it is recommended to
always use the @code{startswith} idiom presented above.



Changed in version 3.8: On AIX, @ref{5a6,,sys.platform} doesn’t contain the major version anymore.
It is always @code{'aix'}, instead of @code{'aix5'} or @code{'aix7'}.  Since
older Python versions include the version number, it is recommended to
always use the @code{startswith} idiom presented above.



@subsubheading See also


@ref{2375,,os.name} has a coarser granularity.  @ref{ca9,,os.uname()} gives
system-dependent version information.

The @ref{c9,,platform} module provides detailed checks for the
system’s identity.

@end deffn

@geindex platlibdir (in module sys)
@anchor{library/sys sys platlibdir}@anchor{42c}
@deffn {Data} sys.platlibdir

Name of the platform-specific library directory. It is used to build the
path of standard library and the paths of installed extension modules.

It is equal to @code{"lib"} on most platforms. On Fedora and SuSE, it is equal
to @code{"lib64"} on 64-bit platforms which gives the following @code{sys.path}
paths (where @code{X.Y} is the Python @code{major.minor} version):


@itemize *

@item 
@code{/usr/lib64/pythonX.Y/}:
Standard library (like @code{os.py} of the @ref{c0,,os} module)

@item 
@code{/usr/lib64/pythonX.Y/lib-dynload/}:
C extension modules of the standard library (like the @ref{77,,errno} module,
the exact filename is platform specific)

@item 
@code{/usr/lib/pythonX.Y/site-packages/} (always use @code{lib}, not
@ref{42c,,sys.platlibdir}): Third-party modules

@item 
@code{/usr/lib64/pythonX.Y/site-packages/}:
C extension modules of third-party packages
@end itemize


New in version 3.9.

@end deffn

@geindex prefix (in module sys)
@anchor{library/sys sys prefix}@anchor{3b34}
@deffn {Data} sys.prefix

A string giving the site-specific directory prefix where the platform
independent Python files are installed; on Unix, the default is
@code{'/usr/local'}.  This can be set at build time with the @code{--prefix}
argument to the @code{configure} script.  See
@ref{3b76,,Installation paths} for derived paths.

@cartouche
@quotation Note 
If a @ref{3b33,,virtual environment} is in effect, this
value will be changed in @code{site.py} to point to the virtual
environment. The value for the Python installation will still be
available, via @ref{34c7,,base_prefix}.
@end quotation
@end cartouche
@end deffn

@geindex ps1 (in module sys)
@geindex ps2 (in module sys)
@anchor{library/sys sys ps1}@anchor{161e}
@deffn {Data} sys.ps1
@anchor{library/sys sys ps2}@anchor{161f}
@deffnx {Data} sys.ps2

@geindex interpreter prompts
@geindex prompts@comma{} interpreter
@geindex >>>; interpreter prompt
@geindex ...; interpreter prompt

Strings specifying the primary and secondary prompt of the interpreter.  These
are only defined if the interpreter is in interactive mode.  Their initial
values in this case are @code{'>>> '} and @code{'... '}.  If a non-string object is
assigned to either variable, its @ref{2ef,,str()} is re-evaluated each time the
interpreter prepares to read a new interactive command; this can be used to
implement a dynamic prompt.
@end deffn

@geindex setdlopenflags() (in module sys)
@anchor{library/sys sys setdlopenflags}@anchor{cb2}
@deffn {Function} sys.setdlopenflags (n)

Set the flags used by the interpreter for @code{dlopen()} calls, such as when
the interpreter loads extension modules.  Among other things, this will enable a
lazy resolving of symbols when importing a module, if called as
@code{sys.setdlopenflags(0)}.  To share symbols across extension modules, call as
@code{sys.setdlopenflags(os.RTLD_GLOBAL)}.  Symbolic names for the flag values
can be found in the @ref{c0,,os} module (@code{RTLD_xxx} constants, e.g.
@ref{cab,,os.RTLD_LAZY}).

@ref{1629,,Availability}: Unix.
@end deffn

@geindex setprofile() (in module sys)
@anchor{library/sys sys setprofile}@anchor{1033}
@deffn {Function} sys.setprofile (profilefunc)

@geindex profile function
@geindex profiler

Set the system’s profile function, which allows you to implement a Python source
code profiler in Python.  See chapter @ref{3a81,,The Python Profilers} for more information on the
Python profiler.  The system’s profile function is called similarly to the
system’s trace function (see @ref{1034,,settrace()}), but it is called with different events,
for example it isn’t called for each executed line of code (only on call and return,
but the return event is reported even when an exception has been set). The function is
thread-specific, but there is no way for the profiler to know about context switches between
threads, so it does not make sense to use this in the presence of multiple threads. Also,
its return value is not used, so it can simply return @code{None}.  Error in the profile
function will cause itself unset.

Profile functions should have three arguments: @emph{frame}, @emph{event}, and
@emph{arg}. @emph{frame} is the current stack frame.  @emph{event} is a string: @code{'call'},
@code{'return'}, @code{'c_call'}, @code{'c_return'}, or @code{'c_exception'}. @emph{arg} depends
on the event type.

Raises an @ref{1130,,auditing event} @code{sys.setprofile} with no arguments.

The events have the following meaning:


@table @asis

@item @code{'call'}

A function is called (or some other code block entered).  The
profile function is called; @emph{arg} is @code{None}.

@item @code{'return'}

A function (or other code block) is about to return.  The profile
function is called; @emph{arg} is the value that will be returned, or @code{None}
if the event is caused by an exception being raised.

@item @code{'c_call'}

A C function is about to be called.  This may be an extension function or
a built-in.  @emph{arg} is the C function object.

@item @code{'c_return'}

A C function has returned. @emph{arg} is the C function object.

@item @code{'c_exception'}

A C function has raised an exception.  @emph{arg} is the C function object.
@end table
@end deffn

@geindex setrecursionlimit() (in module sys)
@anchor{library/sys sys setrecursionlimit}@anchor{106b}
@deffn {Function} sys.setrecursionlimit (limit)

Set the maximum depth of the Python interpreter stack to @emph{limit}.  This limit
prevents infinite recursion from causing an overflow of the C stack and crashing
Python.

The highest possible limit is platform-dependent.  A user may need to set the
limit higher when they have a program that requires deep recursion and a platform
that supports a higher limit.  This should be done with care, because a too-high
limit can lead to a crash.

If the new limit is too low at the current recursion depth, a
@ref{8c4,,RecursionError} exception is raised.


Changed in version 3.5.1: A @ref{8c4,,RecursionError} exception is now raised if the new limit is too
low at the current recursion depth.

@end deffn

@geindex setswitchinterval() (in module sys)
@anchor{library/sys sys setswitchinterval}@anchor{449}
@deffn {Function} sys.setswitchinterval (interval)

Set the interpreter’s thread switch interval (in seconds).  This floating-point
value determines the ideal duration of the “timeslices” allocated to
concurrently running Python threads.  Please note that the actual value
can be higher, especially if long-running internal functions or methods
are used.  Also, which thread becomes scheduled at the end of the interval
is the operating system’s decision.  The interpreter doesn’t have its
own scheduler.


New in version 3.2.

@end deffn

@geindex settrace() (in module sys)
@anchor{library/sys sys settrace}@anchor{1034}
@deffn {Function} sys.settrace (tracefunc)

@geindex trace function
@geindex debugger

Set the system’s trace function, which allows you to implement a Python
source code debugger in Python.  The function is thread-specific; for a
debugger to support multiple threads, it must register a trace function using
@ref{1034,,settrace()} for each thread being debugged or use @ref{2ce,,threading.settrace()}.

Trace functions should have three arguments: @emph{frame}, @emph{event}, and
@emph{arg}. @emph{frame} is the current stack frame.  @emph{event} is a string: @code{'call'},
@code{'line'}, @code{'return'}, @code{'exception'} or @code{'opcode'}.  @emph{arg} depends on
the event type.

The trace function is invoked (with @emph{event} set to @code{'call'}) whenever a new
local scope is entered; it should return a reference to a local trace
function to be used for the new scope, or @code{None} if the scope shouldn’t be
traced.

The local trace function should return a reference to itself (or to another
function for further tracing in that scope), or @code{None} to turn off tracing
in that scope.

If there is any error occurred in the trace function, it will be unset, just
like @code{settrace(None)} is called.

The events have the following meaning:


@table @asis

@item @code{'call'}

A function is called (or some other code block entered).  The
global trace function is called; @emph{arg} is @code{None}; the return value
specifies the local trace function.

@item @code{'line'}

The interpreter is about to execute a new line of code or re-execute the
condition of a loop.  The local trace function is called; @emph{arg} is
@code{None}; the return value specifies the new local trace function.  See
@code{Objects/lnotab_notes.txt} for a detailed explanation of how this
works.
Per-line events may be disabled for a frame by setting
@code{f_trace_lines} to @ref{659,,False} on that frame.

@item @code{'return'}

A function (or other code block) is about to return.  The local trace
function is called; @emph{arg} is the value that will be returned, or @code{None}
if the event is caused by an exception being raised.  The trace function’s
return value is ignored.

@item @code{'exception'}

An exception has occurred.  The local trace function is called; @emph{arg} is a
tuple @code{(exception, value, traceback)}; the return value specifies the
new local trace function.

@item @code{'opcode'}

The interpreter is about to execute a new opcode (see @ref{36,,dis} for
opcode details).  The local trace function is called; @emph{arg} is
@code{None}; the return value specifies the new local trace function.
Per-opcode events are not emitted by default: they must be explicitly
requested by setting @code{f_trace_opcodes} to @ref{748,,True} on the
frame.
@end table

Note that as an exception is propagated down the chain of callers, an
@code{'exception'} event is generated at each level.

For more fine-grained usage, it’s possible to set a trace function by
assigning @code{frame.f_trace = tracefunc} explicitly, rather than relying on
it being set indirectly via the return value from an already installed
trace function. This is also required for activating the trace function on
the current frame, which @ref{1034,,settrace()} doesn’t do. Note that in order
for this to work, a global tracing function must have been installed
with @ref{1034,,settrace()} in order to enable the runtime tracing machinery,
but it doesn’t need to be the same tracing function (e.g. it could be a
low overhead tracing function that simply returns @code{None} to disable
itself immediately on each frame).

For more information on code and frame objects, refer to @ref{1787,,The standard type hierarchy}.

Raises an @ref{1130,,auditing event} @code{sys.settrace} with no arguments.

@strong{CPython implementation detail:} The @ref{1034,,settrace()} function is intended only for implementing debuggers,
profilers, coverage tools and the like.  Its behavior is part of the
implementation platform, rather than part of the language definition, and
thus may not be available in all Python implementations.


Changed in version 3.7: @code{'opcode'} event type added; @code{f_trace_lines} and
@code{f_trace_opcodes} attributes added to frames

@end deffn

@geindex set_asyncgen_hooks() (in module sys)
@anchor{library/sys sys set_asyncgen_hooks}@anchor{11fd}
@deffn {Function} sys.set_asyncgen_hooks (firstiter, finalizer)

Accepts two optional keyword arguments which are callables that accept an
@ref{3b71,,asynchronous generator iterator} as an argument. The @emph{firstiter}
callable will be called when an asynchronous generator is iterated for the
first time. The @emph{finalizer} will be called when an asynchronous generator
is about to be garbage collected.

Raises an @ref{1130,,auditing event} @code{sys.set_asyncgen_hooks_firstiter} with no arguments.

Raises an @ref{1130,,auditing event} @code{sys.set_asyncgen_hooks_finalizer} with no arguments.

Two auditing events are raised because the underlying API consists of two
calls, each of which must raise its own event.


New in version 3.6: See 
@geindex Python Enhancement Proposals; PEP 525
PEP 525@footnote{https://www.python.org/dev/peps/pep-0525} for more details, and for a reference example of a
@emph{finalizer} method see the implementation of
@code{asyncio.Loop.shutdown_asyncgens} in
Lib/asyncio/base_events.py@footnote{https://github.com/python/cpython/tree/main/Lib/asyncio/base_events.py}


@cartouche
@quotation Note 
This function has been added on a provisional basis (see 
@geindex Python Enhancement Proposals; PEP 411
PEP 411@footnote{https://www.python.org/dev/peps/pep-0411}
for details.)
@end quotation
@end cartouche
@end deffn

@geindex set_coroutine_origin_tracking_depth() (in module sys)
@anchor{library/sys sys set_coroutine_origin_tracking_depth}@anchor{6bd}
@deffn {Function} sys.set_coroutine_origin_tracking_depth (depth)

Allows enabling or disabling coroutine origin tracking. When
enabled, the @code{cr_origin} attribute on coroutine objects will
contain a tuple of (filename, line number, function name) tuples
describing the traceback where the coroutine object was created,
with the most recent call first. When disabled, @code{cr_origin} will
be None.

To enable, pass a @emph{depth} value greater than zero; this sets the
number of frames whose information will be captured. To disable,
pass set @emph{depth} to zero.

This setting is thread-specific.


New in version 3.7.


@cartouche
@quotation Note 
This function has been added on a provisional basis (see 
@geindex Python Enhancement Proposals; PEP 411
PEP 411@footnote{https://www.python.org/dev/peps/pep-0411}
for details.)  Use it only for debugging purposes.
@end quotation
@end cartouche
@end deffn

@geindex _enablelegacywindowsfsencoding() (in module sys)
@anchor{library/sys sys _enablelegacywindowsfsencoding}@anchor{7a0}
@deffn {Function} sys._enablelegacywindowsfsencoding ()

Changes the @ref{1628,,filesystem encoding and error handler} to ‘mbcs’ and
‘replace’ respectively, for consistency with versions of Python prior to
3.6.

This is equivalent to defining the 
@geindex PYTHONLEGACYWINDOWSFSENCODING
@geindex environment variable; PYTHONLEGACYWINDOWSFSENCODING
@ref{79f,,PYTHONLEGACYWINDOWSFSENCODING}
environment variable before launching Python.

See also @ref{79e,,sys.getfilesystemencoding()} and
@ref{82f,,sys.getfilesystemencodeerrors()}.

@ref{1629,,Availability}: Windows.


New in version 3.6: See 
@geindex Python Enhancement Proposals; PEP 529
PEP 529@footnote{https://www.python.org/dev/peps/pep-0529} for more details.

@end deffn

@geindex stdin (in module sys)
@geindex stdout (in module sys)
@geindex stderr (in module sys)
@anchor{library/sys sys stdin}@anchor{5ee}
@deffn {Data} sys.stdin
@anchor{library/sys sys stdout}@anchor{5ef}
@deffnx {Data} sys.stdout
@anchor{library/sys sys stderr}@anchor{42d}
@deffnx {Data} sys.stderr

@ref{d68,,File objects} used by the interpreter for standard
input, output and errors:


@itemize *

@item 
@code{stdin} is used for all interactive input (including calls to
@ref{e73,,input()});

@item 
@code{stdout} is used for the output of @ref{af6,,print()} and @ref{17d2,,expression}
statements and for the prompts of @ref{e73,,input()};

@item 
The interpreter’s own prompts and its error messages go to @code{stderr}.
@end itemize

These streams are regular @ref{1566,,text files} like those
returned by the @ref{1c8,,open()} function.  Their parameters are chosen as
follows:


@itemize *

@item 
The encoding and error handling are is initialized from
@ref{3b77,,PyConfig.stdio_encoding} and @ref{3b78,,PyConfig.stdio_errors}.

On Windows, UTF-8 is used for the console device.  Non-character
devices such as disk files and pipes use the system locale
encoding (i.e. the ANSI codepage).  Non-console character
devices such as NUL (i.e. where @code{isatty()} returns @code{True}) use the
value of the console input and output codepages at startup,
respectively for stdin and stdout/stderr. This defaults to the
system @ref{237a,,locale encoding} if the process is not initially attached
to a console.

The special behaviour of the console can be overridden
by setting the environment variable PYTHONLEGACYWINDOWSSTDIO
before starting Python. In that case, the console codepages are
used as for any other character device.

Under all platforms, you can override the character encoding by
setting the 
@geindex PYTHONIOENCODING
@geindex environment variable; PYTHONIOENCODING
@ref{b91,,PYTHONIOENCODING} environment variable before
starting Python or by using the new @ref{159,,-X} @code{utf8} command
line option and 
@geindex PYTHONUTF8
@geindex environment variable; PYTHONUTF8
@ref{5f1,,PYTHONUTF8} environment variable.  However,
for the Windows console, this only applies when
@geindex PYTHONLEGACYWINDOWSSTDIO
@geindex environment variable; PYTHONLEGACYWINDOWSSTDIO
@ref{7a2,,PYTHONLEGACYWINDOWSSTDIO} is also set.

@item 
When interactive, the @code{stdout} stream is line-buffered. Otherwise,
it is block-buffered like regular text files.  The @code{stderr} stream
is line-buffered in both cases.  You can make both streams unbuffered
by passing the @ref{123a,,-u} command-line option or setting the
@geindex PYTHONUNBUFFERED
@geindex environment variable; PYTHONUNBUFFERED
@ref{123b,,PYTHONUNBUFFERED} environment variable.
@end itemize


Changed in version 3.9: Non-interactive @code{stderr} is now line-buffered instead of fully
buffered.


@cartouche
@quotation Note 
To write or read binary data from/to the standard streams, use the
underlying binary @ref{e48,,buffer} object.  For example, to
write bytes to @ref{5ef,,stdout}, use @code{sys.stdout.buffer.write(b'abc')}.

However, if you are writing a library (and do not control in which
context its code will be executed), be aware that the standard streams
may be replaced with file-like objects like @ref{aa1,,io.StringIO} which
do not support the @code{buffer} attribute.
@end quotation
@end cartouche
@end deffn

@geindex __stdin__ (in module sys)
@geindex __stdout__ (in module sys)
@geindex __stderr__ (in module sys)
@anchor{library/sys sys __stdin__}@anchor{3b79}
@deffn {Data} sys.__stdin__
@anchor{library/sys sys __stdout__}@anchor{2130}
@deffnx {Data} sys.__stdout__
@anchor{library/sys sys __stderr__}@anchor{3973}
@deffnx {Data} sys.__stderr__

These objects contain the original values of @code{stdin}, @code{stderr} and
@code{stdout} at the start of the program.  They are used during finalization,
and could be useful to print to the actual standard stream no matter if the
@code{sys.std*} object has been redirected.

It can also be used to restore the actual files to known working file objects
in case they have been overwritten with a broken object.  However, the
preferred way to do this is to explicitly save the previous stream before
replacing it, and restore the saved object.

@cartouche
@quotation Note 
Under some conditions @code{stdin}, @code{stdout} and @code{stderr} as well as the
original values @code{__stdin__}, @code{__stdout__} and @code{__stderr__} can be
@code{None}. It is usually the case for Windows GUI apps that aren’t connected
to a console and Python apps started with @code{pythonw}.
@end quotation
@end cartouche
@end deffn

@geindex stdlib_module_names (in module sys)
@anchor{library/sys sys stdlib_module_names}@anchor{2c7}
@deffn {Data} sys.stdlib_module_names

A frozenset of strings containing the names of standard library modules.

It is the same on all platforms. Modules which are not available on
some platforms and modules disabled at Python build are also listed.
All module kinds are listed: pure Python, built-in, frozen and extension
modules. Test modules are excluded.

For packages, only the main package is listed: sub-packages and sub-modules
are not listed. For example, the @code{email} package is listed, but the
@code{email.mime} sub-package and the @code{email.message} sub-module are not
listed.

See also the @ref{3b62,,sys.builtin_module_names} list.


New in version 3.10.

@end deffn

@geindex thread_info (in module sys)
@anchor{library/sys sys thread_info}@anchor{ce8}
@deffn {Data} sys.thread_info

A @ref{ce9,,named tuple} holding information about the thread
implementation.


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Attribute

@tab

Explanation

@item

@code{name}

@tab

Name of the thread implementation:

@quotation


@itemize *

@item 
@code{'nt'}: Windows threads

@item 
@code{'pthread'}: POSIX threads

@item 
@code{'solaris'}: Solaris threads
@end itemize
@end quotation

@item

@code{lock}

@tab

Name of the lock implementation:

@quotation


@itemize *

@item 
@code{'semaphore'}: a lock uses a semaphore

@item 
@code{'mutex+cond'}: a lock uses a mutex
and a condition variable

@item 
@code{None} if this information is unknown
@end itemize
@end quotation

@item

@ref{86c,,version}

@tab

Name and version of the thread library. It is a string,
or @code{None} if this information is unknown.

@end multitable



New in version 3.3.

@end deffn

@geindex tracebacklimit (in module sys)
@anchor{library/sys sys tracebacklimit}@anchor{3b7a}
@deffn {Data} sys.tracebacklimit

When this variable is set to an integer value, it determines the maximum number
of levels of traceback information printed when an unhandled exception occurs.
The default is @code{1000}.  When set to @code{0} or less, all traceback information
is suppressed and only the exception type and value are printed.
@end deffn

@geindex unraisablehook() (in module sys)
@anchor{library/sys sys unraisablehook}@anchor{18d}
@deffn {Function} sys.unraisablehook (unraisable, /)

Handle an unraisable exception.

Called when an exception has occurred but there is no way for Python to
handle it. For example, when a destructor raises an exception or during
garbage collection (@ref{54b,,gc.collect()}).

The @emph{unraisable} argument has the following attributes:


@itemize *

@item 
@emph{exc_type}: Exception type.

@item 
@emph{exc_value}: Exception value, can be @code{None}.

@item 
@emph{exc_traceback}: Exception traceback, can be @code{None}.

@item 
@emph{err_msg}: Error message, can be @code{None}.

@item 
@emph{object}: Object causing the exception, can be @code{None}.
@end itemize

The default hook formats @emph{err_msg} and @emph{object} as:
@code{f'@{err_msg@}: @{object!r@}'}; use “Exception ignored in” error message
if @emph{err_msg} is @code{None}.

@ref{18d,,sys.unraisablehook()} can be overridden to control how unraisable
exceptions are handled.

Storing @emph{exc_value} using a custom hook can create a reference cycle. It
should be cleared explicitly to break the reference cycle when the
exception is no longer needed.

Storing @emph{object} using a custom hook can resurrect it if it is set to an
object which is being finalized. Avoid storing @emph{object} after the custom
hook completes to avoid resurrecting objects.

See also @ref{28e,,excepthook()} which handles uncaught exceptions.


Raise an auditing event @code{sys.unraisablehook} with arguments
@code{hook}, @code{unraisable} when an exception that cannot be handled occurs.
The @code{unraisable} object is the same as what will be passed to the hook.
If no hook has been set, @code{hook} may be @code{None}.



New in version 3.8.

@end deffn

@geindex version (in module sys)
@anchor{library/sys sys version}@anchor{86c}
@deffn {Data} sys.version

A string containing the version number of the Python interpreter plus additional
information on the build number and compiler used.  This string is displayed
when the interactive interpreter is started.  Do not extract version information
out of it, rather, use @ref{d45,,version_info} and the functions provided by the
@ref{c9,,platform} module.
@end deffn

@geindex api_version (in module sys)
@anchor{library/sys sys api_version}@anchor{3b7b}
@deffn {Data} sys.api_version

The C API version for this interpreter.  Programmers may find this useful when
debugging version conflicts between Python and extension modules.
@end deffn

@geindex version_info (in module sys)
@anchor{library/sys sys version_info}@anchor{d45}
@deffn {Data} sys.version_info

A tuple containing the five components of the version number: @emph{major}, @emph{minor},
@emph{micro}, @emph{releaselevel}, and @emph{serial}.  All values except @emph{releaselevel} are
integers; the release level is @code{'alpha'}, @code{'beta'}, @code{'candidate'}, or
@code{'final'}.  The @code{version_info} value corresponding to the Python version 2.0
is @code{(2, 0, 0, 'final', 0)}.  The components can also be accessed by name,
so @code{sys.version_info[0]} is equivalent to @code{sys.version_info.major}
and so on.


Changed in version 3.1: Added named component attributes.

@end deffn

@geindex warnoptions (in module sys)
@anchor{library/sys sys warnoptions}@anchor{6dd}
@deffn {Data} sys.warnoptions

This is an implementation detail of the warnings framework; do not modify this
value.  Refer to the @ref{12d,,warnings} module for more information on the warnings
framework.
@end deffn

@geindex winver (in module sys)
@anchor{library/sys sys winver}@anchor{3b7c}
@deffn {Data} sys.winver

The version number used to form registry keys on Windows platforms. This is
stored as string resource 1000 in the Python DLL.  The value is normally the
first three characters of @ref{86c,,version}.  It is provided in the @ref{f7,,sys}
module for informational purposes; modifying this value has no effect on the
registry keys used by Python.

@ref{1629,,Availability}: Windows.
@end deffn

@geindex _xoptions (in module sys)
@anchor{library/sys sys _xoptions}@anchor{1618}
@deffn {Data} sys._xoptions

A dictionary of the various implementation-specific flags passed through
the @ref{159,,-X} command-line option.  Option names are either mapped to
their values, if given explicitly, or to @ref{748,,True}.  Example:

@example
$ ./python -Xpycache_prefix=some_path -Xdev
Python 3.2a3+ (py3k, Oct 16 2010, 20:14:50)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._xoptions
@{'pycache_prefix': 'some_path', 'dev': True@}
@end example

@strong{CPython implementation detail:} This is a CPython-specific way of accessing options passed through
@ref{159,,-X}.  Other implementations may export them through other
means, or not at all.


New in version 3.2.

@end deffn

@subsubheading Citations


@anchor{library/sys c99}@anchor{3b6d}@w{(C99)} 
ISO/IEC 9899:1999.  “Programming languages – C.”  A public draft of this standard is available at @indicateurl{http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf}.

@node sysconfig — Provide access to Python’s configuration information,builtins — Built-in objects,sys — System-specific parameters and functions,Python Runtime Services
@anchor{library/sysconfig doc}@anchor{3b7d}@anchor{library/sysconfig module-sysconfig}@anchor{f8}@anchor{library/sysconfig sysconfig-provide-access-to-python-s-configuration-information}@anchor{3b7e}
@subsection @code{sysconfig} — Provide access to Python’s configuration information


@geindex module; sysconfig


New in version 3.2.


@strong{Source code:} Lib/sysconfig.py@footnote{https://github.com/python/cpython/tree/main/Lib/sysconfig.py}

@geindex configuration information


__________________________________________________________________


The @ref{f8,,sysconfig} module provides access to Python’s configuration
information like the list of installation paths and the configuration variables
relevant for the current platform.

@menu
* Configuration variables:: 
* Installation paths:: 
* Other functions: Other functions<3>. 
* Using sysconfig as a script:: 

@end menu

@node Configuration variables,Installation paths,,sysconfig — Provide access to Python’s configuration information
@anchor{library/sysconfig configuration-variables}@anchor{3b7f}
@subsubsection Configuration variables


A Python distribution contains a @code{Makefile} and a @code{pyconfig.h}
header file that are necessary to build both the Python binary itself and
third-party C extensions compiled using @ref{37,,distutils}.

@ref{f8,,sysconfig} puts all variables found in these files in a dictionary that
can be accessed using @ref{bbd,,get_config_vars()} or @ref{bbc,,get_config_var()}.

Notice that on Windows, it’s a much smaller set.

@geindex get_config_vars() (in module sysconfig)
@anchor{library/sysconfig sysconfig get_config_vars}@anchor{bbd}
@deffn {Function} sysconfig.get_config_vars (*args)

With no arguments, return a dictionary of all configuration variables
relevant for the current platform.

With arguments, return a list of values that result from looking up each
argument in the configuration variable dictionary.

For each argument, if the value is not found, return @code{None}.
@end deffn

@geindex get_config_var() (in module sysconfig)
@anchor{library/sysconfig sysconfig get_config_var}@anchor{bbc}
@deffn {Function} sysconfig.get_config_var (name)

Return the value of a single variable @emph{name}. Equivalent to
@code{get_config_vars().get(name)}.

If @emph{name} is not found, return @code{None}.
@end deffn

Example of usage:

@example
>>> import sysconfig
>>> sysconfig.get_config_var('Py_ENABLE_SHARED')
0
>>> sysconfig.get_config_var('LIBDIR')
'/usr/local/lib'
>>> sysconfig.get_config_vars('AR', 'CXX')
['ar', 'g++']
@end example

@node Installation paths,Other functions<3>,Configuration variables,sysconfig — Provide access to Python’s configuration information
@anchor{library/sysconfig id1}@anchor{3b80}@anchor{library/sysconfig installation-paths}@anchor{3b76}
@subsubsection Installation paths


Python uses an installation scheme that differs depending on the platform and on
the installation options.  These schemes are stored in @ref{f8,,sysconfig} under
unique identifiers based on the value returned by @ref{2375,,os.name}.

Every new component that is installed using @ref{37,,distutils} or a
Distutils-based system will follow the same scheme to copy its file in the right
places.

Python currently supports six schemes:


@itemize -

@item 
@emph{posix_prefix}: scheme for POSIX platforms like Linux or macOS.  This is
the default scheme used when Python or a component is installed.

@item 
@emph{posix_home}: scheme for POSIX platforms used when a @emph{home} option is used
upon installation.  This scheme is used when a component is installed through
Distutils with a specific home prefix.

@item 
@emph{posix_user}: scheme for POSIX platforms used when a component is installed
through Distutils and the @emph{user} option is used.  This scheme defines paths
located under the user home directory.

@item 
@emph{nt}: scheme for NT platforms like Windows.

@item 
@emph{nt_user}: scheme for NT platforms, when the @emph{user} option is used.

@item 
@emph{osx_framework_user}: scheme for macOS, when the @emph{user} option is used.
@end itemize

Each scheme is itself composed of a series of paths and each path has a unique
identifier.  Python currently uses eight paths:


@itemize -

@item 
@emph{stdlib}: directory containing the standard Python library files that are not
platform-specific.

@item 
@emph{platstdlib}: directory containing the standard Python library files that are
platform-specific.

@item 
@emph{platlib}: directory for site-specific, platform-specific files.

@item 
@emph{purelib}: directory for site-specific, non-platform-specific files.

@item 
@emph{include}: directory for non-platform-specific header files for
the Python C-API.

@item 
@emph{platinclude}: directory for platform-specific header files for
the Python C-API.

@item 
@emph{scripts}: directory for script files.

@item 
@emph{data}: directory for data files.
@end itemize

@ref{f8,,sysconfig} provides some functions to determine these paths.

@geindex get_scheme_names() (in module sysconfig)
@anchor{library/sysconfig sysconfig get_scheme_names}@anchor{3b81}
@deffn {Function} sysconfig.get_scheme_names ()

Return a tuple containing all schemes currently supported in
@ref{f8,,sysconfig}.
@end deffn

@geindex get_default_scheme() (in module sysconfig)
@anchor{library/sysconfig sysconfig get_default_scheme}@anchor{113b}
@deffn {Function} sysconfig.get_default_scheme ()

Return the default scheme name for the current platform.


Changed in version 3.10: This function was previously named @code{_get_default_scheme()} and
considered an implementation detail.

@end deffn

@geindex get_preferred_scheme() (in module sysconfig)
@anchor{library/sysconfig sysconfig get_preferred_scheme}@anchor{113a}
@deffn {Function} sysconfig.get_preferred_scheme (key)

Return a preferred scheme name for an installation layout specified by @emph{key}.

@emph{key} must be either @code{"prefix"}, @code{"home"}, or @code{"user"}.

The return value is a scheme name listed in @ref{3b81,,get_scheme_names()}. It
can be passed to @ref{f8,,sysconfig} functions that take a @emph{scheme} argument,
such as @ref{df6,,get_paths()}.


New in version 3.10.

@end deffn

@geindex _get_preferred_schemes() (in module sysconfig)
@anchor{library/sysconfig sysconfig _get_preferred_schemes}@anchor{3b82}
@deffn {Function} sysconfig._get_preferred_schemes ()

Return a dict containing preferred scheme names on the current platform.
Python implementers and redistributors may add their preferred schemes to
the @code{_INSTALL_SCHEMES} module-level global value, and modify this function
to return those scheme names, to e.g. provide different schemes for system
and language package managers to use, so packages installed by either do not
mix with those by the other.

End users should not use this function, but @ref{113b,,get_default_scheme()} and
@ref{113a,,get_preferred_scheme()} instead.


New in version 3.10.

@end deffn

@geindex get_path_names() (in module sysconfig)
@anchor{library/sysconfig sysconfig get_path_names}@anchor{3b83}
@deffn {Function} sysconfig.get_path_names ()

Return a tuple containing all path names currently supported in
@ref{f8,,sysconfig}.
@end deffn

@geindex get_path() (in module sysconfig)
@anchor{library/sysconfig sysconfig get_path}@anchor{ec0}
@deffn {Function} sysconfig.get_path (name[, scheme[, vars[, expand]]])

Return an installation path corresponding to the path @emph{name}, from the
install scheme named @emph{scheme}.

@emph{name} has to be a value from the list returned by @ref{3b83,,get_path_names()}.

@ref{f8,,sysconfig} stores installation paths corresponding to each path name,
for each platform, with variables to be expanded.  For instance the @emph{stdlib}
path for the @emph{nt} scheme is: @code{@{base@}/Lib}.

@ref{ec0,,get_path()} will use the variables returned by @ref{bbd,,get_config_vars()}
to expand the path.  All variables have default values for each platform so
one may call this function and get the default value.

If @emph{scheme} is provided, it must be a value from the list returned by
@ref{3b81,,get_scheme_names()}.  Otherwise, the default scheme for the current
platform is used.

If @emph{vars} is provided, it must be a dictionary of variables that will update
the dictionary return by @ref{bbd,,get_config_vars()}.

If @emph{expand} is set to @code{False}, the path will not be expanded using the
variables.

If @emph{name} is not found, raise a @ref{5bb,,KeyError}.
@end deffn

@geindex get_paths() (in module sysconfig)
@anchor{library/sysconfig sysconfig get_paths}@anchor{df6}
@deffn {Function} sysconfig.get_paths ([scheme[, vars[, expand]]])

Return a dictionary containing all installation paths corresponding to an
installation scheme. See @ref{ec0,,get_path()} for more information.

If @emph{scheme} is not provided, will use the default scheme for the current
platform.

If @emph{vars} is provided, it must be a dictionary of variables that will
update the dictionary used to expand the paths.

If @emph{expand} is set to false, the paths will not be expanded.

If @emph{scheme} is not an existing scheme, @ref{df6,,get_paths()} will raise a
@ref{5bb,,KeyError}.
@end deffn

@node Other functions<3>,Using sysconfig as a script,Installation paths,sysconfig — Provide access to Python’s configuration information
@anchor{library/sysconfig other-functions}@anchor{3b84}
@subsubsection Other functions


@geindex get_python_version() (in module sysconfig)
@anchor{library/sysconfig sysconfig get_python_version}@anchor{df5}
@deffn {Function} sysconfig.get_python_version ()

Return the @code{MAJOR.MINOR} Python version number as a string.  Similar to
@code{'%d.%d' % sys.version_info[:2]}.
@end deffn

@geindex get_platform() (in module sysconfig)
@anchor{library/sysconfig sysconfig get_platform}@anchor{df4}
@deffn {Function} sysconfig.get_platform ()

Return a string that identifies the current platform.

This is used mainly to distinguish platform-specific build directories and
platform-specific built distributions.  Typically includes the OS name and
version and the architecture (as supplied by ‘os.uname()’), although the
exact information included depends on the OS; e.g., on Linux, the kernel
version isn’t particularly important.

Examples of returned values:


@itemize -

@item 
linux-i586

@item 
linux-alpha (?)

@item 
solaris-2.6-sun4u
@end itemize

Windows will return one of:


@itemize -

@item 
win-amd64 (64bit Windows on AMD64, aka x86_64, Intel64, and EM64T)

@item 
win32 (all others - specifically, sys.platform is returned)
@end itemize

macOS can return:


@itemize -

@item 
macosx-10.6-ppc

@item 
macosx-10.4-ppc64

@item 
macosx-10.3-i386

@item 
macosx-10.4-fat
@end itemize

For other non-POSIX platforms, currently just returns @ref{5a6,,sys.platform}.
@end deffn

@geindex is_python_build() (in module sysconfig)
@anchor{library/sysconfig sysconfig is_python_build}@anchor{ec1}
@deffn {Function} sysconfig.is_python_build ()

Return @code{True} if the running Python interpreter was built from source and
is being run from its built location, and not from a location resulting from
e.g. running @code{make install} or installing via a binary installer.
@end deffn

@geindex parse_config_h() (in module sysconfig)
@anchor{library/sysconfig sysconfig parse_config_h}@anchor{3b85}
@deffn {Function} sysconfig.parse_config_h (fp[, vars])

Parse a @code{config.h}-style file.

@emph{fp} is a file-like object pointing to the @code{config.h}-like file.

A dictionary containing name/value pairs is returned.  If an optional
dictionary is passed in as the second argument, it is used instead of a new
dictionary, and updated with the values read in the file.
@end deffn

@geindex get_config_h_filename() (in module sysconfig)
@anchor{library/sysconfig sysconfig get_config_h_filename}@anchor{3b86}
@deffn {Function} sysconfig.get_config_h_filename ()

Return the path of @code{pyconfig.h}.
@end deffn

@geindex get_makefile_filename() (in module sysconfig)
@anchor{library/sysconfig sysconfig get_makefile_filename}@anchor{3b87}
@deffn {Function} sysconfig.get_makefile_filename ()

Return the path of @code{Makefile}.
@end deffn

@node Using sysconfig as a script,,Other functions<3>,sysconfig — Provide access to Python’s configuration information
@anchor{library/sysconfig using-sysconfig-as-a-script}@anchor{3b88}
@subsubsection Using @code{sysconfig} as a script


You can use @ref{f8,,sysconfig} as a script with Python’s @emph{-m} option:

@example
$ python -m sysconfig
Platform: "macosx-10.4-i386"
Python version: "3.2"
Current installation scheme: "posix_prefix"

Paths:
        data = "/usr/local"
        include = "/Users/tarek/Dev/svn.python.org/py3k/Include"
        platinclude = "."
        platlib = "/usr/local/lib/python3.2/site-packages"
        platstdlib = "/usr/local/lib/python3.2"
        purelib = "/usr/local/lib/python3.2/site-packages"
        scripts = "/usr/local/bin"
        stdlib = "/usr/local/lib/python3.2"

Variables:
        AC_APPLE_UNIVERSAL_BUILD = "0"
        AIX_GENUINE_CPLUSPLUS = "0"
        AR = "ar"
        ARFLAGS = "rc"
        ...
@end example

This call will print in the standard output the information returned by
@ref{df4,,get_platform()}, @ref{df5,,get_python_version()}, @ref{ec0,,get_path()} and
@ref{bbd,,get_config_vars()}.

@node builtins — Built-in objects,__main__ — Top-level code environment,sysconfig — Provide access to Python’s configuration information,Python Runtime Services
@anchor{library/builtins doc}@anchor{3b89}@anchor{library/builtins builtins-built-in-objects}@anchor{3b8a}@anchor{library/builtins module-builtins}@anchor{11}
@subsection @code{builtins} — Built-in objects


@geindex module; builtins


__________________________________________________________________


This module provides direct access to all ‘built-in’ identifiers of Python; for
example, @code{builtins.open} is the full name for the built-in function
@ref{1c8,,open()}.  See @ref{e0d,,Built-in Functions} and @ref{1a66,,Built-in Constants} for
documentation.

This module is not normally accessed explicitly by most applications, but can be
useful in modules that provide objects with the same name as a built-in value,
but in which the built-in of that name is also needed.  For example, in a module
that wants to implement an @ref{1c8,,open()} function that wraps the built-in
@ref{1c8,,open()}, this module can be used directly:

@example
import builtins

def open(path):
    f = builtins.open(path, 'r')
    return UpperCaser(f)

class UpperCaser:
    '@w{'}'Wrapper around a file that converts output to uppercase.'@w{'}'

    def __init__(self, f):
        self._f = f

    def read(self, count=-1):
        return self._f.read(count).upper()

    # ...
@end example

As an implementation detail, most modules have the name @code{__builtins__} made
available as part of their globals.  The value of @code{__builtins__} is normally
either this module or the value of this module’s @ref{7a4,,__dict__} attribute.
Since this is an implementation detail, it may not be used by alternate
implementations of Python.

@node __main__ — Top-level code environment,warnings — Warning control,builtins — Built-in objects,Python Runtime Services
@anchor{library/__main__ doc}@anchor{3b8b}@anchor{library/__main__ main-top-level-code-environment}@anchor{3b8c}@anchor{library/__main__ module-__main__}@anchor{1}
@subsection @code{__main__} — Top-level code environment


@geindex module; __main__


__________________________________________________________________


In Python, the special name @code{__main__} is used for two important constructs:


@enumerate 

@item 
the name of the top-level environment of the program, which can be
checked using the @code{__name__ == '__main__'} expression; and

@item 
the @code{__main__.py} file in Python packages.
@end enumerate

Both of these mechanisms are related to Python modules; how users interact with
them and how they interact with each other.  They are explained in detail
below.  If you’re new to Python modules, see the tutorial section
@ref{153d,,Modules} for an introduction.

@menu
* __name__ == '__main__':: 
* __main__.py in Python Packages: __main__ py in Python Packages. 
* import __main__:: 

@end menu

@node __name__ == '__main__',__main__ py in Python Packages,,__main__ — Top-level code environment
@anchor{library/__main__ name-equals-main}@anchor{3b8d}@anchor{library/__main__ name-main}@anchor{3b8e}
@subsubsection @code{__name__ == '__main__'}


When a Python module or package is imported, @code{__name__} is set to the
module’s name.  Usually, this is the name of the Python file itself without the
@code{.py} extension:

@example
>>> import configparser
>>> configparser.__name__
'configparser'
@end example

If the file is part of a package, @code{__name__} will also include the parent
package’s path:

@example
>>> from concurrent.futures import process
>>> process.__name__
'concurrent.futures.process'
@end example

However, if the module is executed in the top-level code environment,
its @code{__name__} is set to the string @code{'__main__'}.

@menu
* What is the “top-level code environment”?:: 
* Idiomatic Usage:: 
* Packaging Considerations:: 

@end menu

@node What is the “top-level code environment”?,Idiomatic Usage,,__name__ == '__main__'
@anchor{library/__main__ what-is-the-top-level-code-environment}@anchor{3b8f}
@subsubsection What is the “top-level code environment”?


@code{__main__} is the name of the environment where top-level code is run.
“Top-level code” is the first user-specified Python module that starts running.
It’s “top-level” because it imports all other modules that the program needs.
Sometimes “top-level code” is called an @emph{entry point} to the application.

The top-level code environment can be:


@itemize *

@item 
the scope of an interactive prompt:

@example
>>> __name__
'__main__'
@end example

@item 
the Python module passed to the Python interpreter as a file argument:

@quotation

@example
$ python3 helloworld.py
Hello, world!
@end example
@end quotation

@item 
the Python module or package passed to the Python interpreter with the
@ref{611,,-m} argument:

@quotation

@example
$ python3 -m tarfile
usage: tarfile.py [-h] [-v] (...)
@end example
@end quotation

@item 
Python code read by the Python interpreter from standard input:

@quotation

@example
$ echo "import this" | python3
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
...
@end example
@end quotation

@item 
Python code passed to the Python interpreter with the @ref{1180,,-c} argument:

@quotation

@example
$ python3 -c "import this"
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
...
@end example
@end quotation
@end itemize

In each of these situations, the top-level module’s @code{__name__} is set to
@code{'__main__'}.

As a result, a module can discover whether or not it is running in the
top-level environment by checking its own @code{__name__}, which allows a common
idiom for conditionally executing code when the module is not initialized from
an import statement:

@example
if __name__ == '__main__':
    # Execute when the module is not initialized from an import statement.
    ...
@end example


@subsubheading See also


For a more detailed look at how @code{__name__} is set in all situations, see
the tutorial section @ref{153d,,Modules}.


@node Idiomatic Usage,Packaging Considerations,What is the “top-level code environment”?,__name__ == '__main__'
@anchor{library/__main__ idiomatic-usage}@anchor{3b90}
@subsubsection Idiomatic Usage


Some modules contain code that is intended for script use only, like parsing
command-line arguments or fetching data from standard input.  If a module
like this was imported from a different module, for example to unit test
it, the script code would unintentionally execute as well.

This is where using the @code{if __name__ == '__main__'} code block comes in
handy. Code within this block won’t run unless the module is executed in the
top-level environment.

Putting as few statements as possible in the block below @code{if __name___ ==
'__main__'} can improve code clarity and correctness. Most often, a function
named @code{main} encapsulates the program’s primary behavior:

@example
# echo.py

import shlex
import sys

def echo(phrase: str) -> None:
   """A dummy wrapper around print."""
   # for demonstration purposes, you can imagine that there is some
   # valuable and reusable logic inside this function
   print(phrase)

def main() -> int:
    """Echo the input arguments to standard output"""
    phrase = shlex.join(sys.argv)
    echo(phrase)
    return 0

if __name__ == '__main__':
    sys.exit(main())  # next section explains the use of sys.exit
@end example

Note that if the module didn’t encapsulate code inside the @code{main} function
but instead put it directly within the @code{if __name__ == '__main__'} block,
the @code{phrase} variable would be global to the entire module.  This is
error-prone as other functions within the module could be unintentionally using
the global variable instead of a local name.  A @code{main} function solves this
problem.

Using a @code{main} function has the added benefit of the @code{echo} function itself
being isolated and importable elsewhere. When @code{echo.py} is imported, the
@code{echo} and @code{main} functions will be defined, but neither of them will be
called, because @code{__name__ != '__main__'}.

@node Packaging Considerations,,Idiomatic Usage,__name__ == '__main__'
@anchor{library/__main__ packaging-considerations}@anchor{3b91}
@subsubsection Packaging Considerations


@code{main} functions are often used to create command-line tools by specifying
them as entry points for console scripts.  When this is done,
pip@footnote{https://pip.pypa.io/} inserts the function call into a template script,
where the return value of @code{main} is passed into @ref{ee4,,sys.exit()}.
For example:

@example
sys.exit(main())
@end example

Since the call to @code{main} is wrapped in @ref{ee4,,sys.exit()}, the expectation is
that your function will return some value acceptable as an input to
@ref{ee4,,sys.exit()}; typically, an integer or @code{None} (which is implicitly
returned if your function does not have a return statement).

By proactively following this convention ourselves, our module will have the
same behavior when run directly (i.e. @code{python3 echo.py}) as it will have if
we later package it as a console script entry-point in a pip-installable
package.

In particular, be careful about returning strings from your @code{main} function.
@ref{ee4,,sys.exit()} will interpret a string argument as a failure message, so
your program will have an exit code of @code{1}, indicating failure, and the
string will be written to @ref{42d,,sys.stderr}.  The @code{echo.py} example from
earlier exemplifies using the @code{sys.exit(main())} convention.


@subsubheading See also


Python Packaging User Guide@footnote{https://packaging.python.org/}
contains a collection of tutorials and references on how to distribute and
install Python packages with modern tools.


@node __main__ py in Python Packages,import __main__,__name__ == '__main__',__main__ — Top-level code environment
@anchor{library/__main__ main-py-in-python-packages}@anchor{3b92}
@subsubsection @code{__main__.py} in Python Packages


If you are not familiar with Python packages, see section @ref{154a,,Packages}
of the tutorial.  Most commonly, the @code{__main__.py} file is used to provide
a command-line interface for a package. Consider the following hypothetical
package, “bandclass”:

@example
bandclass
  ├── __init__.py
  ├── __main__.py
  └── student.py
@end example

@code{__main__.py} will be executed when the package itself is invoked
directly from the command line using the @ref{611,,-m} flag. For example:

@example
$ python3 -m bandclass
@end example

This command will cause @code{__main__.py} to run. How you utilize this mechanism
will depend on the nature of the package you are writing, but in this
hypothetical case, it might make sense to allow the teacher to search for
students:

@example
# bandclass/__main__.py

import sys
from .student import search_students

student_name = sys.argv[2] if len(sys.argv) >= 2 else '@w{'}
print(f'Found student: @{search_students(student_name)@}')
@end example

Note that @code{from .student import search_students} is an example of a relative
import.  This import style can be used when referencing modules within a
package.  For more details, see @ref{154e,,Intra-package References} in the
@ref{153d,,Modules} section of the tutorial.

@menu
* Idiomatic Usage: Idiomatic Usage<2>. 

@end menu

@node Idiomatic Usage<2>,,,__main__ py in Python Packages
@anchor{library/__main__ id1}@anchor{3b93}
@subsubsection Idiomatic Usage


The contents of @code{__main__.py} typically isn’t fenced with
@code{if __name__ == '__main__'} blocks.  Instead, those files are kept short,
functions to execute from other modules.  Those other modules can then be
easily unit-tested and are properly reusable.

If used, an @code{if __name__ == '__main__'} block will still work as expected
for a @code{__main__.py} file within a package, because its @code{__name__}
attribute will include the package’s path if imported:

@example
>>> import asyncio.__main__
>>> asyncio.__main__.__name__
'asyncio.__main__'
@end example

This won’t work for @code{__main__.py} files in the root directory of a .zip file
though.  Hence, for consistency, minimal @code{__main__.py} like the @ref{12c,,venv}
one mentioned below are preferred.


@subsubheading See also


See @ref{12c,,venv} for an example of a package with a minimal @code{__main__.py}
in the standard library. It doesn’t contain a @code{if __name__ == '__main__'}
block. You can invoke it with @code{python3 -m venv [directory]}.

See @ref{dd,,runpy} for more details on the @ref{611,,-m} flag to the
interpreter executable.

See @ref{148,,zipapp} for how to run applications packaged as @emph{.zip} files. In
this case Python looks for a @code{__main__.py} file in the root directory of
the archive.


@node import __main__,,__main__ py in Python Packages,__main__ — Top-level code environment
@anchor{library/__main__ import-main}@anchor{3b94}
@subsubsection @code{import __main__}


Regardless of which module a Python program was started with, other modules
running within that same program can import the top-level environment’s scope
(@ref{3b95,,namespace}) by importing the @code{__main__} module.  This doesn’t import
a @code{__main__.py} file but rather whichever module that received the special
name @code{'__main__'}.

Here is an example module that consumes the @code{__main__} namespace:

@example
# namely.py

import __main__

def did_user_define_their_name():
    return 'my_name' in dir(__main__)

def print_user_name():
    if not did_user_define_their_name():
        raise ValueError('Define the variable `my_name`!')

    if '__file__' in dir(__main__):
        print(__main__.my_name, "found in file", __main__.__file__)
    else:
        print(__main__.my_name)
@end example

Example usage of this module could be as follows:

@example
# start.py

import sys

from namely import print_user_name

# my_name = "Dinsdale"

def main():
    try:
        print_user_name()
    except ValueError as ve:
        return str(ve)

if __name__ == "__main__":
    sys.exit(main())
@end example

Now, if we started our program, the result would look like this:

@example
$ python3 start.py
Define the variable `my_name`!
@end example

The exit code of the program would be 1, indicating an error. Uncommenting the
line with @code{my_name = "Dinsdale"} fixes the program and now it exits with
status code 0, indicating success:

@example
$ python3 start.py
Dinsdale found in file /path/to/start.py
@end example

Note that importing @code{__main__} doesn’t cause any issues with unintentionally
running top-level code meant for script use which is put in the
@code{if __name__ == "__main__"} block of the @code{start} module. Why does this work?

Python inserts an empty @code{__main__} module in @ref{11f7,,sys.modules} at
interpreter startup, and populates it by running top-level code. In our example
this is the @code{start} module which runs line by line and imports @code{namely}.
In turn, @code{namely} imports @code{__main__} (which is really @code{start}). That’s an
import cycle! Fortunately, since the partially populated @code{__main__}
module is present in @ref{11f7,,sys.modules}, Python passes that to @code{namely}.
See @ref{186b,,Special considerations for __main__} in the
import system’s reference for details on how this works.

The Python REPL is another example of a “top-level environment”, so anything
defined in the REPL becomes part of the @code{__main__} scope:

@example
>>> import namely
>>> namely.did_user_define_their_name()
False
>>> namely.print_user_name()
Traceback (most recent call last):
...
ValueError: Define the variable `my_name`!
>>> my_name = 'Jabberwocky'
>>> namely.did_user_define_their_name()
True
>>> namely.print_user_name()
Jabberwocky
@end example

Note that in this case the @code{__main__} scope doesn’t contain a @code{__file__}
attribute as it’s interactive.

The @code{__main__} scope is used in the implementation of @ref{c4,,pdb} and
@ref{dc,,rlcompleter}.

@node warnings — Warning control,dataclasses — Data Classes,__main__ — Top-level code environment,Python Runtime Services
@anchor{library/warnings doc}@anchor{3b96}@anchor{library/warnings module-warnings}@anchor{12d}@anchor{library/warnings warnings-warning-control}@anchor{3b97}
@subsection @code{warnings} — Warning control


@geindex module; warnings

@strong{Source code:} Lib/warnings.py@footnote{https://github.com/python/cpython/tree/main/Lib/warnings.py}

@geindex warnings


__________________________________________________________________


Warning messages are typically issued in situations where it is useful to alert
the user of some condition in a program, where that condition (normally) doesn’t
warrant raising an exception and terminating the program.  For example, one
might want to issue a warning when a program uses an obsolete module.

Python programmers issue warnings by calling the @ref{104d,,warn()} function defined
in this module.  (C programmers use @ref{fab,,PyErr_WarnEx()}; see
@ref{3b98,,Exception Handling} for details).

Warning messages are normally written to @ref{42d,,sys.stderr}, but their disposition
can be changed flexibly, from ignoring all warnings to turning them into
exceptions.  The disposition of warnings can vary based on the @ref{1b8c,,warning category}, the text of the warning message, and the source location where it
is issued.  Repetitions of a particular warning for the same source location are
typically suppressed.

There are two stages in warning control: first, each time a warning is issued, a
determination is made whether a message should be issued or not; next, if a
message is to be issued, it is formatted and printed using a user-settable hook.

The determination whether to issue a warning message is controlled by the
@ref{3aa,,warning filter}, which is a sequence of matching rules and actions. Rules can be
added to the filter by calling @ref{ffd,,filterwarnings()} and reset to its default
state by calling @ref{3b99,,resetwarnings()}.

The printing of warning messages is done by calling @ref{3b9a,,showwarning()}, which
may be overridden; the default implementation of this function formats the
message by calling @ref{12c0,,formatwarning()}, which is also available for use by
custom implementations.


@subsubheading See also


@ref{256a,,logging.captureWarnings()} allows you to handle all warnings with
the standard logging infrastructure.


@menu
* Warning Categories:: 
* The Warnings Filter:: 
* Temporarily Suppressing Warnings:: 
* Testing Warnings:: 
* Updating Code For New Versions of Dependencies:: 
* Available Functions:: 
* Available Context Managers:: 

@end menu

@node Warning Categories,The Warnings Filter,,warnings — Warning control
@anchor{library/warnings id1}@anchor{3b9b}@anchor{library/warnings warning-categories}@anchor{1b8c}
@subsubsection Warning Categories


There are a number of built-in exceptions that represent warning categories.
This categorization is useful to be able to filter out groups of warnings.

While these are technically
@ref{1b8a,,built-in exceptions}, they are
documented here, because conceptually they belong to the warnings mechanism.

User code can define additional warning categories by subclassing one of the
standard warning categories.  A warning category must always be a subclass of
the @ref{1b8d,,Warning} class.

The following warnings category classes are currently defined:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Class

@tab

Description

@item

@ref{1b8d,,Warning}

@tab

This is the base class of all warning
category classes.  It is a subclass of
@ref{4d8,,Exception}.

@item

@ref{1b8e,,UserWarning}

@tab

The default category for @ref{104d,,warn()}.

@item

@ref{1b2,,DeprecationWarning}

@tab

Base category for warnings about deprecated
features when those warnings are intended for
other Python developers (ignored by default,
unless triggered by code in @code{__main__}).

@item

@ref{4c7,,SyntaxWarning}

@tab

Base category for warnings about dubious
syntactic features.

@item

@ref{5b5,,RuntimeWarning}

@tab

Base category for warnings about dubious
runtime features.

@item

@ref{604,,FutureWarning}

@tab

Base category for warnings about deprecated
features when those warnings are intended for
end users of applications that are written in
Python.

@item

@ref{3a9,,PendingDeprecationWarning}

@tab

Base category for warnings about features
that will be deprecated in the future
(ignored by default).

@item

@ref{304,,ImportWarning}

@tab

Base category for warnings triggered during
the process of importing a module (ignored by
default).

@item

@ref{f81,,UnicodeWarning}

@tab

Base category for warnings related to
Unicode.

@item

@ref{761,,BytesWarning}

@tab

Base category for warnings related to
@ref{2f0,,bytes} and @ref{2f1,,bytearray}.

@item

@ref{77b,,ResourceWarning}

@tab

Base category for warnings related to
resource usage (ignored by default).

@end multitable



Changed in version 3.7: Previously @ref{1b2,,DeprecationWarning} and @ref{604,,FutureWarning} were
distinguished based on whether a feature was being removed entirely or
changing its behaviour. They are now distinguished based on their
intended audience and the way they’re handled by the default warnings
filters.


@node The Warnings Filter,Temporarily Suppressing Warnings,Warning Categories,warnings — Warning control
@anchor{library/warnings the-warnings-filter}@anchor{3b9c}@anchor{library/warnings warning-filter}@anchor{3aa}
@subsubsection The Warnings Filter


The warnings filter controls whether warnings are ignored, displayed, or turned
into errors (raising an exception).

Conceptually, the warnings filter maintains an ordered list of filter
specifications; any specific warning is matched against each filter
specification in the list in turn until a match is found; the filter determines
the disposition of the match.  Each entry is a tuple of the form (@emph{action},
@emph{message}, @emph{category}, @emph{module}, @emph{lineno}), where:


@itemize *

@item 
@emph{action} is one of the following strings:


@multitable {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Value

@tab

Disposition

@item

@code{"default"}

@tab

print the first occurrence of matching
warnings for each location (module +
line number) where the warning is issued

@item

@code{"error"}

@tab

turn matching warnings into exceptions

@item

@code{"ignore"}

@tab

never print matching warnings

@item

@code{"always"}

@tab

always print matching warnings

@item

@code{"module"}

@tab

print the first occurrence of matching
warnings for each module where the warning
is issued (regardless of line number)

@item

@code{"once"}

@tab

print only the first occurrence of matching
warnings, regardless of location

@end multitable


@item 
@emph{message} is a string containing a regular expression that the start of
the warning message must match.  The expression is compiled to always be
case-insensitive.

@item 
@emph{category} is a class (a subclass of @ref{1b8d,,Warning}) of which the warning
category must be a subclass in order to match.

@item 
@emph{module} is a string containing a regular expression that the module name must
match.  The expression is compiled to be case-sensitive.

@item 
@emph{lineno} is an integer that the line number where the warning occurred must
match, or @code{0} to match all line numbers.
@end itemize

Since the @ref{1b8d,,Warning} class is derived from the built-in @ref{4d8,,Exception}
class, to turn a warning into an error we simply raise @code{category(message)}.

If a warning is reported and doesn’t match any registered filter then the
“default” action is applied (hence its name).

@menu
* Describing Warning Filters:: 
* Default Warning Filter:: 
* Overriding the default filter:: 

@end menu

@node Describing Warning Filters,Default Warning Filter,,The Warnings Filter
@anchor{library/warnings describing-warning-filters}@anchor{1614}@anchor{library/warnings id2}@anchor{3b9d}
@subsubsection Describing Warning Filters


The warnings filter is initialized by @ref{3a8,,-W} options passed to the Python
interpreter command line and the 
@geindex PYTHONWARNINGS
@geindex environment variable; PYTHONWARNINGS
@ref{6de,,PYTHONWARNINGS} environment variable.
The interpreter saves the arguments for all supplied entries without
interpretation in @ref{6dd,,sys.warnoptions}; the @ref{12d,,warnings} module parses these
when it is first imported (invalid options are ignored, after printing a
message to @ref{42d,,sys.stderr}).

Individual warnings filters are specified as a sequence of fields separated by
colons:

@example
action:message:category:module:line
@end example

The meaning of each of these fields is as described in @ref{3aa,,The Warnings Filter}.
When listing multiple filters on a single line (as for
@geindex PYTHONWARNINGS
@geindex environment variable; PYTHONWARNINGS
@ref{6de,,PYTHONWARNINGS}), the individual filters are separated by commas and
the filters listed later take precedence over those listed before them (as
they’re applied left-to-right, and the most recently applied filters take
precedence over earlier ones).

Commonly used warning filters apply to either all warnings, warnings in a
particular category, or warnings raised by particular modules or packages.
Some examples:

@example
default                      # Show all warnings (even those ignored by default)
ignore                       # Ignore all warnings
error                        # Convert all warnings to errors
error::ResourceWarning       # Treat ResourceWarning messages as errors
default::DeprecationWarning  # Show DeprecationWarning messages
ignore,default:::mymodule    # Only report warnings triggered by "mymodule"
error:::mymodule[.*]         # Convert warnings to errors in "mymodule"
                             # and any subpackages of "mymodule"
@end example

@node Default Warning Filter,Overriding the default filter,Describing Warning Filters,The Warnings Filter
@anchor{library/warnings default-warning-filter}@anchor{3b9e}@anchor{library/warnings id3}@anchor{3b9f}
@subsubsection Default Warning Filter


By default, Python installs several warning filters, which can be overridden by
the @ref{3a8,,-W} command-line option, the 
@geindex PYTHONWARNINGS
@geindex environment variable; PYTHONWARNINGS
@ref{6de,,PYTHONWARNINGS} environment
variable and calls to @ref{ffd,,filterwarnings()}.

In regular release builds, the default warning filter has the following entries
(in order of precedence):

@example
default::DeprecationWarning:__main__
ignore::DeprecationWarning
ignore::PendingDeprecationWarning
ignore::ImportWarning
ignore::ResourceWarning
@end example

In a @ref{338,,debug build}, the list of default warning filters is empty.


Changed in version 3.2: @ref{1b2,,DeprecationWarning} is now ignored by default in addition to
@ref{3a9,,PendingDeprecationWarning}.



Changed in version 3.7: @ref{1b2,,DeprecationWarning} is once again shown by default when triggered
directly by code in @code{__main__}.



Changed in version 3.7: @ref{761,,BytesWarning} no longer appears in the default filter list and is
instead configured via @ref{6dd,,sys.warnoptions} when @ref{6dc,,-b} is specified
twice.


@node Overriding the default filter,,Default Warning Filter,The Warnings Filter
@anchor{library/warnings overriding-the-default-filter}@anchor{3ba0}@anchor{library/warnings warning-disable}@anchor{3ba1}
@subsubsection Overriding the default filter


Developers of applications written in Python may wish to hide @emph{all} Python level
warnings from their users by default, and only display them when running tests
or otherwise working on the application. The @ref{6dd,,sys.warnoptions} attribute
used to pass filter configurations to the interpreter can be used as a marker to
indicate whether or not warnings should be disabled:

@example
import sys

if not sys.warnoptions:
    import warnings
    warnings.simplefilter("ignore")
@end example

Developers of test runners for Python code are advised to instead ensure that
@emph{all} warnings are displayed by default for the code under test, using code
like:

@example
import sys

if not sys.warnoptions:
    import os, warnings
    warnings.simplefilter("default") # Change the filter in this process
    os.environ["PYTHONWARNINGS"] = "default" # Also affect subprocesses
@end example

Finally, developers of interactive shells that run user code in a namespace
other than @code{__main__} are advised to ensure that @ref{1b2,,DeprecationWarning}
messages are made visible by default, using code like the following (where
@code{user_ns} is the module used to execute code entered interactively):

@example
import warnings
warnings.filterwarnings("default", category=DeprecationWarning,
                                   module=user_ns.get("__name__"))
@end example

@node Temporarily Suppressing Warnings,Testing Warnings,The Warnings Filter,warnings — Warning control
@anchor{library/warnings temporarily-suppressing-warnings}@anchor{3ba2}@anchor{library/warnings warning-suppress}@anchor{3ba3}
@subsubsection Temporarily Suppressing Warnings


If you are using code that you know will raise a warning, such as a deprecated
function, but do not want to see the warning (even when warnings have been
explicitly configured via the command line), then it is possible to suppress
the warning using the @ref{39e4,,catch_warnings} context manager:

@example
import warnings

def fxn():
    warnings.warn("deprecated", DeprecationWarning)

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    fxn()
@end example

While within the context manager all warnings will simply be ignored. This
allows you to use known-deprecated code without having to see the warning while
not suppressing the warning for other code that might not be aware of its use
of deprecated code.  Note: this can only be guaranteed in a single-threaded
application. If two or more threads use the @ref{39e4,,catch_warnings} context
manager at the same time, the behavior is undefined.

@node Testing Warnings,Updating Code For New Versions of Dependencies,Temporarily Suppressing Warnings,warnings — Warning control
@anchor{library/warnings testing-warnings}@anchor{3ba4}@anchor{library/warnings warning-testing}@anchor{3ba5}
@subsubsection Testing Warnings


To test warnings raised by code, use the @ref{39e4,,catch_warnings} context
manager. With it you can temporarily mutate the warnings filter to facilitate
your testing. For instance, do the following to capture all raised warnings to
check:

@example
import warnings

def fxn():
    warnings.warn("deprecated", DeprecationWarning)

with warnings.catch_warnings(record=True) as w:
    # Cause all warnings to always be triggered.
    warnings.simplefilter("always")
    # Trigger a warning.
    fxn()
    # Verify some things
    assert len(w) == 1
    assert issubclass(w[-1].category, DeprecationWarning)
    assert "deprecated" in str(w[-1].message)
@end example

One can also cause all warnings to be exceptions by using @code{error} instead of
@code{always}. One thing to be aware of is that if a warning has already been
raised because of a @code{once}/@code{default} rule, then no matter what filters are
set the warning will not be seen again unless the warnings registry related to
the warning has been cleared.

Once the context manager exits, the warnings filter is restored to its state
when the context was entered. This prevents tests from changing the warnings
filter in unexpected ways between tests and leading to indeterminate test
results. The @ref{3b9a,,showwarning()} function in the module is also restored to
its original value.  Note: this can only be guaranteed in a single-threaded
application. If two or more threads use the @ref{39e4,,catch_warnings} context
manager at the same time, the behavior is undefined.

When testing multiple operations that raise the same kind of warning, it
is important to test them in a manner that confirms each operation is raising
a new warning (e.g. set warnings to be raised as exceptions and check the
operations raise exceptions, check that the length of the warning list
continues to increase after each operation, or else delete the previous
entries from the warnings list before each new operation).

@node Updating Code For New Versions of Dependencies,Available Functions,Testing Warnings,warnings — Warning control
@anchor{library/warnings updating-code-for-new-versions-of-dependencies}@anchor{3ba6}@anchor{library/warnings warning-ignored}@anchor{385d}
@subsubsection Updating Code For New Versions of Dependencies


Warning categories that are primarily of interest to Python developers (rather
than end users of applications written in Python) are ignored by default.

Notably, this “ignored by default” list includes @ref{1b2,,DeprecationWarning}
(for every module except @code{__main__}), which means developers should make sure
to test their code with typically ignored warnings made visible in order to
receive timely notifications of future breaking API changes (whether in the
standard library or third party packages).

In the ideal case, the code will have a suitable test suite, and the test runner
will take care of implicitly enabling all warnings when running tests
(the test runner provided by the @ref{122,,unittest} module does this).

In less ideal cases, applications can be checked for use of deprecated
interfaces by passing @ref{3a8,,-Wd} to the Python interpreter (this is
shorthand for @code{-W default}) or setting @code{PYTHONWARNINGS=default} in
the environment. This enables default handling for all warnings, including those
that are ignored by default. To change what action is taken for encountered
warnings you can change what argument is passed to @ref{3a8,,-W} (e.g.
@code{-W error}). See the @ref{3a8,,-W} flag for more details on what is
possible.

@node Available Functions,Available Context Managers,Updating Code For New Versions of Dependencies,warnings — Warning control
@anchor{library/warnings available-functions}@anchor{3ba7}@anchor{library/warnings warning-functions}@anchor{3ba8}
@subsubsection Available Functions


@geindex warn() (in module warnings)
@anchor{library/warnings warnings warn}@anchor{104d}
@deffn {Function} warnings.warn (message, category=None, stacklevel=1, source=None)

Issue a warning, or maybe ignore it or raise an exception.  The @emph{category}
argument, if given, must be a @ref{1b8c,,warning category class}; it
defaults to @ref{1b8e,,UserWarning}.  Alternatively, @emph{message} can be a @ref{1b8d,,Warning} instance,
in which case @emph{category} will be ignored and @code{message.__class__} will be used.
In this case, the message text will be @code{str(message)}. This function raises an
exception if the particular warning issued is changed into an error by the
@ref{3aa,,warnings filter}.  The @emph{stacklevel} argument can be used by wrapper
functions written in Python, like this:

@example
def deprecation(message):
    warnings.warn(message, DeprecationWarning, stacklevel=2)
@end example

This makes the warning refer to @code{deprecation()}’s caller, rather than to the
source of @code{deprecation()} itself (since the latter would defeat the purpose
of the warning message).

@emph{source}, if supplied, is the destroyed object which emitted a
@ref{77b,,ResourceWarning}.


Changed in version 3.6: Added @emph{source} parameter.

@end deffn

@geindex warn_explicit() (in module warnings)
@anchor{library/warnings warnings warn_explicit}@anchor{84b}
@deffn {Function} warnings.warn_explicit (message, category, filename, lineno, module=None, registry=None, module_globals=None, source=None)

This is a low-level interface to the functionality of @ref{104d,,warn()}, passing in
explicitly the message, category, filename and line number, and optionally the
module name and the registry (which should be the @code{__warningregistry__}
dictionary of the module).  The module name defaults to the filename with
@code{.py} stripped; if no registry is passed, the warning is never suppressed.
@emph{message} must be a string and @emph{category} a subclass of @ref{1b8d,,Warning} or
@emph{message} may be a @ref{1b8d,,Warning} instance, in which case @emph{category} will be
ignored.

@emph{module_globals}, if supplied, should be the global namespace in use by the code
for which the warning is issued.  (This argument is used to support displaying
source for modules found in zipfiles or other non-filesystem import
sources).

@emph{source}, if supplied, is the destroyed object which emitted a
@ref{77b,,ResourceWarning}.


Changed in version 3.6: Add the @emph{source} parameter.

@end deffn

@geindex showwarning() (in module warnings)
@anchor{library/warnings warnings showwarning}@anchor{3b9a}
@deffn {Function} warnings.showwarning (message, category, filename, lineno, file=None, line=None)

Write a warning to a file.  The default implementation calls
@code{formatwarning(message, category, filename, lineno, line)} and writes the
resulting string to @emph{file}, which defaults to @ref{42d,,sys.stderr}.  You may replace
this function with any callable by assigning to @code{warnings.showwarning}.
@emph{line} is a line of source code to be included in the warning
message; if @emph{line} is not supplied, @ref{3b9a,,showwarning()} will
try to read the line specified by @emph{filename} and @emph{lineno}.
@end deffn

@geindex formatwarning() (in module warnings)
@anchor{library/warnings warnings formatwarning}@anchor{12c0}
@deffn {Function} warnings.formatwarning (message, category, filename, lineno, line=None)

Format a warning the standard way.  This returns a string which may contain
embedded newlines and ends in a newline.  @emph{line} is a line of source code to
be included in the warning message; if @emph{line} is not supplied,
@ref{12c0,,formatwarning()} will try to read the line specified by @emph{filename} and
@emph{lineno}.
@end deffn

@geindex filterwarnings() (in module warnings)
@anchor{library/warnings warnings filterwarnings}@anchor{ffd}
@deffn {Function} warnings.filterwarnings (action, message='@w{'}, category=Warning, module='@w{'}, lineno=0, append=False)

Insert an entry into the list of @ref{3aa,,warnings filter specifications}.  The entry is inserted at the front by default; if
@emph{append} is true, it is inserted at the end.  This checks the types of the
arguments, compiles the @emph{message} and @emph{module} regular expressions, and
inserts them as a tuple in the list of warnings filters.  Entries closer to
the front of the list override entries later in the list, if both match a
particular warning.  Omitted arguments default to a value that matches
everything.
@end deffn

@geindex simplefilter() (in module warnings)
@anchor{library/warnings warnings simplefilter}@anchor{39e5}
@deffn {Function} warnings.simplefilter (action, category=Warning, lineno=0, append=False)

Insert a simple entry into the list of @ref{3aa,,warnings filter specifications}.  The meaning of the function parameters is as for
@ref{ffd,,filterwarnings()}, but regular expressions are not needed as the filter
inserted always matches any message in any module as long as the category and
line number match.
@end deffn

@geindex resetwarnings() (in module warnings)
@anchor{library/warnings warnings resetwarnings}@anchor{3b99}
@deffn {Function} warnings.resetwarnings ()

Reset the warnings filter.  This discards the effect of all previous calls to
@ref{ffd,,filterwarnings()}, including that of the @ref{3a8,,-W} command line options
and calls to @ref{39e5,,simplefilter()}.
@end deffn

@node Available Context Managers,,Available Functions,warnings — Warning control
@anchor{library/warnings available-context-managers}@anchor{3ba9}
@subsubsection Available Context Managers


@geindex catch_warnings (class in warnings)
@anchor{library/warnings warnings catch_warnings}@anchor{39e4}
@deffn {Class} warnings.catch_warnings (*, record=False, module=None)

A context manager that copies and, upon exit, restores the warnings filter
and the @ref{3b9a,,showwarning()} function.
If the @emph{record} argument is @ref{659,,False} (the default) the context manager
returns @ref{182,,None} on entry. If @emph{record} is @ref{748,,True}, a list is
returned that is progressively populated with objects as seen by a custom
@ref{3b9a,,showwarning()} function (which also suppresses output to @code{sys.stdout}).
Each object in the list has attributes with the same names as the arguments to
@ref{3b9a,,showwarning()}.

The @emph{module} argument takes a module that will be used instead of the
module returned when you import @ref{12d,,warnings} whose filter will be
protected. This argument exists primarily for testing the @ref{12d,,warnings}
module itself.

@cartouche
@quotation Note 
The @ref{39e4,,catch_warnings} manager works by replacing and
then later restoring the module’s
@ref{3b9a,,showwarning()} function and internal list of filter
specifications.  This means the context manager is modifying
global state and therefore is not thread-safe.
@end quotation
@end cartouche
@end deffn

@node dataclasses — Data Classes,contextlib — Utilities for with-statement contexts,warnings — Warning control,Python Runtime Services
@anchor{library/dataclasses doc}@anchor{3baa}@anchor{library/dataclasses dataclasses-data-classes}@anchor{3bab}@anchor{library/dataclasses module-dataclasses}@anchor{2e}
@subsection @code{dataclasses} — Data Classes


@geindex module; dataclasses

@strong{Source code:} Lib/dataclasses.py@footnote{https://github.com/python/cpython/tree/main/Lib/dataclasses.py}


__________________________________________________________________


This module provides a decorator and functions for automatically
adding generated @ref{111a,,special method}s such as @code{__init__()} and
@code{__repr__()} to user-defined classes.  It was originally described
in 
@geindex Python Enhancement Proposals; PEP 557
PEP 557@footnote{https://www.python.org/dev/peps/pep-0557}.

The member variables to use in these generated methods are defined
using 
@geindex Python Enhancement Proposals; PEP 526
PEP 526@footnote{https://www.python.org/dev/peps/pep-0526} type annotations.  For example, this code:

@example
from dataclasses import dataclass

@@dataclass
class InventoryItem:
    """Class for keeping track of an item in inventory."""
    name: str
    unit_price: float
    quantity_on_hand: int = 0

    def total_cost(self) -> float:
        return self.unit_price * self.quantity_on_hand
@end example

will add, among other things, a @code{__init__()} that looks like:

@example
def __init__(self, name: str, unit_price: float, quantity_on_hand: int = 0):
    self.name = name
    self.unit_price = unit_price
    self.quantity_on_hand = quantity_on_hand
@end example

Note that this method is automatically added to the class: it is not
directly specified in the @code{InventoryItem} definition shown above.


New in version 3.7.


@menu
* Module contents: Module contents<4>. 
* Post-init processing:: 
* Class variables:: 
* Init-only variables:: 
* Frozen instances:: 
* Inheritance: Inheritance<2>. 
* Re-ordering of keyword-only parameters in __init__(): Re-ordering of keyword-only parameters in __init__. 
* Default factory functions:: 
* Mutable default values:: 

@end menu

@node Module contents<4>,Post-init processing,,dataclasses — Data Classes
@anchor{library/dataclasses module-contents}@anchor{3bac}
@subsubsection Module contents


@geindex dataclass() (in module dataclasses)
@anchor{library/dataclasses dataclasses dataclass}@anchor{276}
@deffn {Function} @@dataclasses.dataclass (*, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False, match_args=True, kw_only=False, slots=False)

This function is a @ref{1bf,,decorator} that is used to add generated
@ref{111a,,special method}s to classes, as described below.

The @ref{276,,dataclass()} decorator examines the class to find
@code{field}s.  A @code{field} is defined as a class variable that has a
@ref{8af,,type annotation}.  With two
exceptions described below, nothing in @ref{276,,dataclass()}
examines the type specified in the variable annotation.

The order of the fields in all of the generated methods is the
order in which they appear in the class definition.

The @ref{276,,dataclass()} decorator will add various “dunder” methods to
the class, described below.  If any of the added methods already
exist in the class, the behavior depends on the parameter, as documented
below. The decorator returns the same class that it is called on; no new
class is created.

If @ref{276,,dataclass()} is used just as a simple decorator with no parameters,
it acts as if it has the default values documented in this
signature.  That is, these three uses of @ref{276,,dataclass()} are
equivalent:

@example
@@dataclass
class C:
    ...

@@dataclass()
class C:
    ...

@@dataclass(init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False, match_args=True, kw_only=False, slots=False)
class C:
   ...
@end example

The parameters to @ref{276,,dataclass()} are:


@itemize -

@item 
@code{init}: If true (the default), a @code{__init__()} method will be
generated.

If the class already defines @code{__init__()}, this parameter is
ignored.

@item 
@code{repr}: If true (the default), a @code{__repr__()} method will be
generated.  The generated repr string will have the class name and
the name and repr of each field, in the order they are defined in
the class.  Fields that are marked as being excluded from the repr
are not included.  For example:
@code{InventoryItem(name='widget', unit_price=3.0, quantity_on_hand=10)}.

If the class already defines @code{__repr__()}, this parameter is
ignored.

@item 
@code{eq}: If true (the default), an @code{__eq__()} method will be
generated.  This method compares the class as if it were a tuple
of its fields, in order.  Both instances in the comparison must
be of the identical type.

If the class already defines @code{__eq__()}, this parameter is
ignored.

@item 
@code{order}: If true (the default is @code{False}), @code{__lt__()},
@code{__le__()}, @code{__gt__()}, and @code{__ge__()} methods will be
generated.  These compare the class as if it were a tuple of its
fields, in order.  Both instances in the comparison must be of the
identical type.  If @code{order} is true and @code{eq} is false, a
@ref{17c,,ValueError} is raised.

If the class already defines any of @code{__lt__()},
@code{__le__()}, @code{__gt__()}, or @code{__ge__()}, then
@ref{162,,TypeError} is raised.

@item 
@code{unsafe_hash}: If @code{False} (the default), a @code{__hash__()} method
is generated according to how @code{eq} and @code{frozen} are set.

@code{__hash__()} is used by built-in @ref{c14,,hash()}, and when objects are
added to hashed collections such as dictionaries and sets.  Having a
@code{__hash__()} implies that instances of the class are immutable.
Mutability is a complicated property that depends on the programmer’s
intent, the existence and behavior of @code{__eq__()}, and the values of
the @code{eq} and @code{frozen} flags in the @ref{276,,dataclass()} decorator.

By default, @ref{276,,dataclass()} will not implicitly add a @code{__hash__()}
method unless it is safe to do so.  Neither will it add or change an
existing explicitly defined @code{__hash__()} method.  Setting the class
attribute @code{__hash__ = None} has a specific meaning to Python, as
described in the @code{__hash__()} documentation.

If @code{__hash__()} is not explicitly defined, or if it is set to @code{None},
then @ref{276,,dataclass()} @emph{may} add an implicit @code{__hash__()} method.
Although not recommended, you can force @ref{276,,dataclass()} to create a
@code{__hash__()} method with @code{unsafe_hash=True}. This might be the case
if your class is logically immutable but can nonetheless be mutated.
This is a specialized use case and should be considered carefully.

Here are the rules governing implicit creation of a @code{__hash__()}
method.  Note that you cannot both have an explicit @code{__hash__()}
method in your dataclass and set @code{unsafe_hash=True}; this will result
in a @ref{162,,TypeError}.

If @code{eq} and @code{frozen} are both true, by default @ref{276,,dataclass()} will
generate a @code{__hash__()} method for you.  If @code{eq} is true and
@code{frozen} is false, @code{__hash__()} will be set to @code{None}, marking it
unhashable (which it is, since it is mutable).  If @code{eq} is false,
@code{__hash__()} will be left untouched meaning the @code{__hash__()}
method of the superclass will be used (if the superclass is
@ref{5a5,,object}, this means it will fall back to id-based hashing).

@item 
@code{frozen}: If true (the default is @code{False}), assigning to fields will
generate an exception.  This emulates read-only frozen instances.  If
@code{__setattr__()} or @code{__delattr__()} is defined in the class, then
@ref{162,,TypeError} is raised.  See the discussion below.

@item 
@code{match_args}: If true (the default is @code{True}), the
@code{__match_args__} tuple will be created from the list of
parameters to the generated @code{__init__()} method (even if
@code{__init__()} is not generated, see above).  If false, or if
@code{__match_args__} is already defined in the class, then
@code{__match_args__} will not be generated.
@end itemize

@quotation


New in version 3.10.

@end quotation


@itemize -

@item 
@code{kw_only}: If true (the default value is @code{False}), then all
fields will be marked as keyword-only.  If a field is marked as
keyword-only, then the only effect is that the @code{__init__()}
parameter generated from a keyword-only field must be specified
with a keyword when @code{__init__()} is called.  There is no
effect on any other aspect of dataclasses.  See the
@ref{18e1,,parameter} glossary entry for details.  Also see the
@ref{3bad,,KW_ONLY} section.
@end itemize

@quotation


New in version 3.10.

@end quotation


@itemize -

@item 
@code{slots}: If true (the default is @code{False}), @code{__slots__} attribute
will be generated and new class will be returned instead of the original one.
If @code{__slots__} is already defined in the class, then @ref{162,,TypeError}
is raised.
@end itemize

@quotation


New in version 3.10.

@end quotation

@code{field}s may optionally specify a default value, using normal
Python syntax:

@example
@@dataclass
class C:
    a: int       # 'a' has no default value
    b: int = 0   # assign a default value for 'b'
@end example

In this example, both @code{a} and @code{b} will be included in the added
@code{__init__()} method, which will be defined as:

@example
def __init__(self, a: int, b: int = 0):
@end example

@ref{162,,TypeError} will be raised if a field without a default value
follows a field with a default value.  This is true whether this
occurs in a single class, or as a result of class inheritance.
@end deffn

@geindex field() (in module dataclasses)
@anchor{library/dataclasses dataclasses field}@anchor{12cb}
@deffn {Function} dataclasses.field (*, default=MISSING, default_factory=MISSING, init=True, repr=True, hash=None, compare=True, metadata=None, kw_only=MISSING)

For common and simple use cases, no other functionality is
required.  There are, however, some dataclass features that
require additional per-field information.  To satisfy this need for
additional information, you can replace the default field value
with a call to the provided @ref{12cb,,field()} function.  For example:

@example
@@dataclass
class C:
    mylist: list[int] = field(default_factory=list)

c = C()
c.mylist += [1, 2, 3]
@end example

As shown above, the @ref{3bae,,MISSING} value is a sentinel object used to
detect if some parameters are provided by the user. This sentinel is
used because @code{None} is a valid value for some parameters with
a distinct meaning.  No code should directly use the @ref{3bae,,MISSING} value.

The parameters to @ref{12cb,,field()} are:


@itemize -

@item 
@code{default}: If provided, this will be the default value for this
field.  This is needed because the @ref{12cb,,field()} call itself
replaces the normal position of the default value.

@item 
@code{default_factory}: If provided, it must be a zero-argument
callable that will be called when a default value is needed for
this field.  Among other purposes, this can be used to specify
fields with mutable default values, as discussed below.  It is an
error to specify both @code{default} and @code{default_factory}.

@item 
@code{init}: If true (the default), this field is included as a
parameter to the generated @code{__init__()} method.

@item 
@code{repr}: If true (the default), this field is included in the
string returned by the generated @code{__repr__()} method.

@item 
@code{hash}: This can be a bool or @code{None}.  If true, this field is
included in the generated @code{__hash__()} method.  If @code{None} (the
default), use the value of @code{compare}: this would normally be
the expected behavior.  A field should be considered in the hash
if it’s used for comparisons.  Setting this value to anything
other than @code{None} is discouraged.

One possible reason to set @code{hash=False} but @code{compare=True}
would be if a field is expensive to compute a hash value for,
that field is needed for equality testing, and there are other
fields that contribute to the type’s hash value.  Even if a field
is excluded from the hash, it will still be used for comparisons.

@item 
@code{compare}: If true (the default), this field is included in the
generated equality and comparison methods (@code{__eq__()},
@code{__gt__()}, et al.).

@item 
@code{metadata}: This can be a mapping or None. None is treated as
an empty dict.  This value is wrapped in
@ref{246,,MappingProxyType()} to make it read-only, and exposed
on the @ref{1b45,,Field} object. It is not used at all by Data
Classes, and is provided as a third-party extension mechanism.
Multiple third-parties can each have their own key, to use as a
namespace in the metadata.

@item 
@code{kw_only}: If true, this field will be marked as keyword-only.
This is used when the generated @code{__init__()} method’s
parameters are computed.
@end itemize

@quotation


New in version 3.10.

@end quotation

If the default value of a field is specified by a call to
@ref{12cb,,field()}, then the class attribute for this field will be
replaced by the specified @code{default} value.  If no @code{default} is
provided, then the class attribute will be deleted.  The intent is
that after the @ref{276,,dataclass()} decorator runs, the class
attributes will all contain the default values for the fields, just
as if the default value itself were specified.  For example,
after:

@example
@@dataclass
class C:
    x: int
    y: int = field(repr=False)
    z: int = field(repr=False, default=10)
    t: int = 20
@end example

The class attribute @code{C.z} will be @code{10}, the class attribute
@code{C.t} will be @code{20}, and the class attributes @code{C.x} and
@code{C.y} will not be set.
@end deffn

@geindex Field (class in dataclasses)
@anchor{library/dataclasses dataclasses Field}@anchor{1b45}
@deffn {Class} dataclasses.Field

@ref{1b45,,Field} objects describe each defined field. These objects
are created internally, and are returned by the @ref{3baf,,fields()}
module-level method (see below).  Users should never instantiate a
@ref{1b45,,Field} object directly.  Its documented attributes are:

@quotation


@itemize -

@item 
@code{name}: The name of the field.

@item 
@code{type}: The type of the field.

@item 
@code{default}, @code{default_factory}, @code{init}, @code{repr}, @code{hash},
@code{compare}, @code{metadata}, and @code{kw_only} have the identical
meaning and values as they do in the @ref{12cb,,field()} function.
@end itemize
@end quotation

Other attributes may exist, but they are private and must not be
inspected or relied on.
@end deffn

@geindex fields() (in module dataclasses)
@anchor{library/dataclasses dataclasses fields}@anchor{3baf}
@deffn {Function} dataclasses.fields (class_or_instance)

Returns a tuple of @ref{1b45,,Field} objects that define the fields for this
dataclass.  Accepts either a dataclass, or an instance of a dataclass.
Raises @ref{162,,TypeError} if not passed a dataclass or instance of one.
Does not return pseudo-fields which are @code{ClassVar} or @code{InitVar}.
@end deffn

@geindex asdict() (in module dataclasses)
@anchor{library/dataclasses dataclasses asdict}@anchor{10d3}
@deffn {Function} dataclasses.asdict (obj, *, dict_factory=dict)

Converts the dataclass @code{obj} to a dict (by using the
factory function @code{dict_factory}).  Each dataclass is converted
to a dict of its fields, as @code{name: value} pairs.  dataclasses, dicts,
lists, and tuples are recursed into.  Other objects are copied with
@ref{69b,,copy.deepcopy()}.

Example of using @ref{10d3,,asdict()} on nested dataclasses:

@example
@@dataclass
class Point:
     x: int
     y: int

@@dataclass
class C:
     mylist: list[Point]

p = Point(10, 20)
assert asdict(p) == @{'x': 10, 'y': 20@}

c = C([Point(0, 0), Point(10, 4)])
assert asdict(c) == @{'mylist': [@{'x': 0, 'y': 0@}, @{'x': 10, 'y': 4@}]@}
@end example

To create a shallow copy, the following workaround may be used:

@example
dict((field.name, getattr(obj, field.name)) for field in fields(obj))
@end example

@ref{10d3,,asdict()} raises @ref{162,,TypeError} if @code{obj} is not a dataclass
instance.
@end deffn

@geindex astuple() (in module dataclasses)
@anchor{library/dataclasses dataclasses astuple}@anchor{10d2}
@deffn {Function} dataclasses.astuple (obj, *, tuple_factory=tuple)

Converts the dataclass @code{obj} to a tuple (by using the
factory function @code{tuple_factory}).  Each dataclass is converted
to a tuple of its field values.  dataclasses, dicts, lists, and
tuples are recursed into. Other objects are copied with
@ref{69b,,copy.deepcopy()}.

Continuing from the previous example:

@example
assert astuple(p) == (10, 20)
assert astuple(c) == ([(0, 0), (10, 4)],)
@end example

To create a shallow copy, the following workaround may be used:

@example
tuple(getattr(obj, field.name) for field in dataclasses.fields(obj))
@end example

@ref{10d2,,astuple()} raises @ref{162,,TypeError} if @code{obj} is not a dataclass
instance.
@end deffn

@geindex make_dataclass() (in module dataclasses)
@anchor{library/dataclasses dataclasses make_dataclass}@anchor{3bb0}
@deffn {Function} dataclasses.make_dataclass (cls_name, fields, *, bases=(), namespace=None, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False, match_args=True, kw_only=False, slots=False)

Creates a new dataclass with name @code{cls_name}, fields as defined
in @code{fields}, base classes as given in @code{bases}, and initialized
with a namespace as given in @code{namespace}.  @code{fields} is an
iterable whose elements are each either @code{name}, @code{(name, type)},
or @code{(name, type, Field)}.  If just @code{name} is supplied,
@code{typing.Any} is used for @code{type}.  The values of @code{init},
@code{repr}, @code{eq}, @code{order}, @code{unsafe_hash}, @code{frozen},
@code{match_args}, @code{kw_only}, and  @code{slots} have the same meaning as
they do in @ref{276,,dataclass()}.

This function is not strictly required, because any Python
mechanism for creating a new class with @code{__annotations__} can
then apply the @ref{276,,dataclass()} function to convert that class to
a dataclass.  This function is provided as a convenience.  For
example:

@example
C = make_dataclass('C',
                   [('x', int),
                     'y',
                    ('z', int, field(default=5))],
                   namespace=@{'add_one': lambda self: self.x + 1@})
@end example

Is equivalent to:

@example
@@dataclass
class C:
    x: int
    y: 'typing.Any'
    z: int = 5

    def add_one(self):
        return self.x + 1
@end example
@end deffn

@geindex replace() (in module dataclasses)
@anchor{library/dataclasses dataclasses replace}@anchor{113c}
@deffn {Function} dataclasses.replace (obj, /, **changes)

Creates a new object of the same type as @code{obj}, replacing
fields with values from @code{changes}.  If @code{obj} is not a Data
Class, raises @ref{162,,TypeError}.  If values in @code{changes} do not
specify fields, raises @ref{162,,TypeError}.

The newly returned object is created by calling the @code{__init__()}
method of the dataclass.  This ensures that
@code{__post_init__()}, if present, is also called.

Init-only variables without default values, if any exist, must be
specified on the call to @ref{113c,,replace()} so that they can be passed to
@code{__init__()} and @code{__post_init__()}.

It is an error for @code{changes} to contain any fields that are
defined as having @code{init=False}.  A @ref{17c,,ValueError} will be raised
in this case.

Be forewarned about how @code{init=False} fields work during a call to
@ref{113c,,replace()}.  They are not copied from the source object, but
rather are initialized in @code{__post_init__()}, if they’re
initialized at all.  It is expected that @code{init=False} fields will
be rarely and judiciously used.  If they are used, it might be wise
to have alternate class constructors, or perhaps a custom
@code{replace()} (or similarly named) method which handles instance
copying.
@end deffn

@geindex is_dataclass() (in module dataclasses)
@anchor{library/dataclasses dataclasses is_dataclass}@anchor{10d0}
@deffn {Function} dataclasses.is_dataclass (obj)

Return @code{True} if its parameter is a dataclass or an instance of one,
otherwise return @code{False}.

If you need to know if a class is an instance of a dataclass (and
not a dataclass itself), then add a further check for @code{not
isinstance(obj, type)}:

@example
def is_dataclass_instance(obj):
    return is_dataclass(obj) and not isinstance(obj, type)
@end example
@end deffn

@geindex MISSING (in module dataclasses)
@anchor{library/dataclasses dataclasses MISSING}@anchor{3bae}
@deffn {Data} dataclasses.MISSING

A sentinel value signifying a missing default or default_factory.
@end deffn

@geindex KW_ONLY (in module dataclasses)
@anchor{library/dataclasses dataclasses KW_ONLY}@anchor{3bad}
@deffn {Data} dataclasses.KW_ONLY

A sentinel value used as a type annotation.  Any fields after a
pseudo-field with the type of @ref{3bad,,KW_ONLY} are marked as
keyword-only fields.  Note that a pseudo-field of type
@ref{3bad,,KW_ONLY} is otherwise completely ignored.  This includes the
name of such a field.  By convention, a name of @code{_} is used for a
@ref{3bad,,KW_ONLY} field.  Keyword-only fields signify
@code{__init__()} parameters that must be specified as keywords when
the class is instantiated.

In this example, the fields @code{y} and @code{z} will be marked as keyword-only fields:

@example
@@dataclass
class Point:
  x: float
  _: KW_ONLY
  y: float
  z: float

p = Point(0, y=1.5, z=2.0)
@end example

In a single dataclass, it is an error to specify more than one
field whose type is @ref{3bad,,KW_ONLY}.


New in version 3.10.

@end deffn

@geindex FrozenInstanceError
@anchor{library/dataclasses dataclasses FrozenInstanceError}@anchor{3bb1}
@deffn {Exception} dataclasses.FrozenInstanceError

Raised when an implicitly defined @code{__setattr__()} or
@code{__delattr__()} is called on a dataclass which was defined with
@code{frozen=True}. It is a subclass of @ref{163,,AttributeError}.
@end deffn

@node Post-init processing,Class variables,Module contents<4>,dataclasses — Data Classes
@anchor{library/dataclasses post-init-processing}@anchor{3bb2}
@subsubsection Post-init processing


The generated @code{__init__()} code will call a method named
@code{__post_init__()}, if @code{__post_init__()} is defined on the
class.  It will normally be called as @code{self.__post_init__()}.
However, if any @code{InitVar} fields are defined, they will also be
passed to @code{__post_init__()} in the order they were defined in the
class.  If no @code{__init__()} method is generated, then
@code{__post_init__()} will not automatically be called.

Among other uses, this allows for initializing field values that
depend on one or more other fields.  For example:

@example
@@dataclass
class C:
    a: float
    b: float
    c: float = field(init=False)

    def __post_init__(self):
        self.c = self.a + self.b
@end example

The @code{__init__()} method generated by @ref{276,,dataclass()} does not call base
class @code{__init__()} methods. If the base class has an @code{__init__()} method
that has to be called, it is common to call this method in a
@code{__post_init__()} method:

@example
@@dataclass
class Rectangle:
    height: float
    width: float

@@dataclass
class Square(Rectangle):
    side: float

    def __post_init__(self):
        super().__init__(self.side, self.side)
@end example

Note, however, that in general the dataclass-generated @code{__init__()} methods
don’t need to be called, since the derived dataclass will take care of
initializing all fields of any base class that is a dataclass itself.

See the section below on init-only variables for ways to pass
parameters to @code{__post_init__()}.  Also see the warning about how
@ref{113c,,replace()} handles @code{init=False} fields.

@node Class variables,Init-only variables,Post-init processing,dataclasses — Data Classes
@anchor{library/dataclasses class-variables}@anchor{3bb3}
@subsubsection Class variables


One of two places where @ref{276,,dataclass()} actually inspects the type
of a field is to determine if a field is a class variable as defined
in 
@geindex Python Enhancement Proposals; PEP 526
PEP 526@footnote{https://www.python.org/dev/peps/pep-0526}.  It does this by checking if the type of the field is
@code{typing.ClassVar}.  If a field is a @code{ClassVar}, it is excluded
from consideration as a field and is ignored by the dataclass
mechanisms.  Such @code{ClassVar} pseudo-fields are not returned by the
module-level @ref{3baf,,fields()} function.

@node Init-only variables,Frozen instances,Class variables,dataclasses — Data Classes
@anchor{library/dataclasses init-only-variables}@anchor{3bb4}
@subsubsection Init-only variables


The other place where @ref{276,,dataclass()} inspects a type annotation is to
determine if a field is an init-only variable.  It does this by seeing
if the type of a field is of type @code{dataclasses.InitVar}.  If a field
is an @code{InitVar}, it is considered a pseudo-field called an init-only
field.  As it is not a true field, it is not returned by the
module-level @ref{3baf,,fields()} function.  Init-only fields are added as
parameters to the generated @code{__init__()} method, and are passed to
the optional @code{__post_init__()} method.  They are not otherwise used
by dataclasses.

For example, suppose a field will be initialized from a database, if a
value is not provided when creating the class:

@example
@@dataclass
class C:
    i: int
    j: int = None
    database: InitVar[DatabaseType] = None

    def __post_init__(self, database):
        if self.j is None and database is not None:
            self.j = database.lookup('j')

c = C(10, database=my_database)
@end example

In this case, @ref{3baf,,fields()} will return @ref{1b45,,Field} objects for @code{i} and
@code{j}, but not for @code{database}.

@node Frozen instances,Inheritance<2>,Init-only variables,dataclasses — Data Classes
@anchor{library/dataclasses frozen-instances}@anchor{3bb5}
@subsubsection Frozen instances


It is not possible to create truly immutable Python objects.  However,
by passing @code{frozen=True} to the @ref{276,,dataclass()} decorator you can
emulate immutability.  In that case, dataclasses will add
@code{__setattr__()} and @code{__delattr__()} methods to the class.  These
methods will raise a @ref{3bb1,,FrozenInstanceError} when invoked.

There is a tiny performance penalty when using @code{frozen=True}:
@code{__init__()} cannot use simple assignment to initialize fields, and
must use @ref{1798,,object.__setattr__()}.

@node Inheritance<2>,Re-ordering of keyword-only parameters in __init__,Frozen instances,dataclasses — Data Classes
@anchor{library/dataclasses inheritance}@anchor{3bb6}
@subsubsection Inheritance


When the dataclass is being created by the @ref{276,,dataclass()} decorator,
it looks through all of the class’s base classes in reverse MRO (that
is, starting at @ref{5a5,,object}) and, for each dataclass that it finds,
adds the fields from that base class to an ordered mapping of fields.
After all of the base class fields are added, it adds its own fields
to the ordered mapping.  All of the generated methods will use this
combined, calculated ordered mapping of fields.  Because the fields
are in insertion order, derived classes override base classes.  An
example:

@example
@@dataclass
class Base:
    x: Any = 15.0
    y: int = 0

@@dataclass
class C(Base):
    z: int = 10
    x: int = 15
@end example

The final list of fields is, in order, @code{x}, @code{y}, @code{z}.  The final
type of @code{x} is @code{int}, as specified in class @code{C}.

The generated @code{__init__()} method for @code{C} will look like:

@example
def __init__(self, x: int = 15, y: int = 0, z: int = 10):
@end example

@node Re-ordering of keyword-only parameters in __init__,Default factory functions,Inheritance<2>,dataclasses — Data Classes
@anchor{library/dataclasses re-ordering-of-keyword-only-parameters-in-init}@anchor{3bb7}
@subsubsection Re-ordering of keyword-only parameters in @code{__init__()}


After the parameters needed for @code{__init__()} are computed, any
keyword-only parameters are moved to come after all regular
(non-keyword-only) parameters.  This is a requirement of how
keyword-only parameters are implemented in Python: they must come
after non-keyword-only parameters.

In this example, @code{Base.y}, @code{Base.w}, and @code{D.t} are keyword-only
fields, and @code{Base.x} and @code{D.z} are regular fields:

@example
@@dataclass
class Base:
    x: Any = 15.0
    _: KW_ONLY
    y: int = 0
    w: int = 1

@@dataclass
class D(Base):
    z: int = 10
    t: int = field(kw_only=True, default=0)
@end example

The generated @code{__init__()} method for @code{D} will look like:

@example
def __init__(self, x: Any = 15.0, z: int = 10, *, y: int = 0, w: int = 1, t: int = 0):
@end example

Note that the parameters have been re-ordered from how they appear in
the list of fields: parameters derived from regular fields are
followed by parameters derived from keyword-only fields.

The relative ordering of keyword-only parameters is maintained in the
re-ordered @code{__init__()} parameter list.

@node Default factory functions,Mutable default values,Re-ordering of keyword-only parameters in __init__,dataclasses — Data Classes
@anchor{library/dataclasses default-factory-functions}@anchor{3bb8}
@subsubsection Default factory functions


If a @ref{12cb,,field()} specifies a @code{default_factory}, it is called with
zero arguments when a default value for the field is needed.  For
example, to create a new instance of a list, use:

@example
mylist: list = field(default_factory=list)
@end example

If a field is excluded from @code{__init__()} (using @code{init=False})
and the field also specifies @code{default_factory}, then the default
factory function will always be called from the generated
@code{__init__()} function.  This happens because there is no other
way to give the field an initial value.

@node Mutable default values,,Default factory functions,dataclasses — Data Classes
@anchor{library/dataclasses mutable-default-values}@anchor{3bb9}
@subsubsection Mutable default values


Python stores default member variable values in class attributes.
Consider this example, not using dataclasses:

@example
class C:
    x = []
    def add(self, element):
        self.x.append(element)

o1 = C()
o2 = C()
o1.add(1)
o2.add(2)
assert o1.x == [1, 2]
assert o1.x is o2.x
@end example

Note that the two instances of class @code{C} share the same class
variable @code{x}, as expected.

Using dataclasses, @emph{if} this code was valid:

@example
@@dataclass
class D:
    x: List = []
    def add(self, element):
        self.x += element
@end example

it would generate code similar to:

@example
class D:
    x = []
    def __init__(self, x=x):
        self.x = x
    def add(self, element):
        self.x += element

assert D().x is D().x
@end example

This has the same issue as the original example using class @code{C}.
That is, two instances of class @code{D} that do not specify a value
for @code{x} when creating a class instance will share the same copy
of @code{x}.  Because dataclasses just use normal Python class
creation they also share this behavior.  There is no general way
for Data Classes to detect this condition.  Instead, the
@ref{276,,dataclass()} decorator will raise a @ref{162,,TypeError} if it
detects an unhashable default parameter.  The assumption is that if
a value is unhashable, it is mutable.  This is a partial solution,
but it does protect against many common errors.

Using default factory functions is a way to create new instances of
mutable types as default values for fields:

@example
@@dataclass
class D:
    x: list = field(default_factory=list)

assert D().x is not D().x
@end example


Changed in version 3.11: Instead of looking for and disallowing objects of type @code{list},
@code{dict}, or @code{set}, unhashable objects are now not allowed as
default values.  Unhashability is used to approximate
mutability.


@node contextlib — Utilities for with-statement contexts,abc — Abstract Base Classes,dataclasses — Data Classes,Python Runtime Services
@anchor{library/contextlib doc}@anchor{3bba}@anchor{library/contextlib contextlib-utilities-for-with-statement-contexts}@anchor{3bbb}@anchor{library/contextlib module-contextlib}@anchor{22}
@subsection @code{contextlib} — Utilities for @code{with}-statement contexts


@geindex module; contextlib

@strong{Source code:} Lib/contextlib.py@footnote{https://github.com/python/cpython/tree/main/Lib/contextlib.py}


__________________________________________________________________


This module provides utilities for common tasks involving the @ref{168,,with}
statement. For more information see also @ref{1810,,Context Manager Types} and
@ref{180e,,With Statement Context Managers}.

@menu
* Utilities:: 
* Examples and Recipes: Examples and Recipes<2>. 
* Single use@comma{} reusable and reentrant context managers: Single use reusable and reentrant context managers. 

@end menu

@node Utilities,Examples and Recipes<2>,,contextlib — Utilities for with-statement contexts
@anchor{library/contextlib utilities}@anchor{3bbc}
@subsubsection Utilities


Functions and classes provided:

@geindex AbstractContextManager (class in contextlib)
@anchor{library/contextlib contextlib AbstractContextManager}@anchor{7d6}
@deffn {Class} contextlib.AbstractContextManager

An @ref{d5c,,abstract base class} for classes that implement
@ref{1811,,object.__enter__()} and @ref{1812,,object.__exit__()}. A default
implementation for @ref{1811,,object.__enter__()} is provided which returns
@code{self} while @ref{1812,,object.__exit__()} is an abstract method which by default
returns @code{None}. See also the definition of @ref{1810,,Context Manager Types}.


New in version 3.6.

@end deffn

@geindex AbstractAsyncContextManager (class in contextlib)
@anchor{library/contextlib contextlib AbstractAsyncContextManager}@anchor{64a}
@deffn {Class} contextlib.AbstractAsyncContextManager

An @ref{d5c,,abstract base class} for classes that implement
@ref{1823,,object.__aenter__()} and @ref{1824,,object.__aexit__()}. A default
implementation for @ref{1823,,object.__aenter__()} is provided which returns
@code{self} while @ref{1824,,object.__aexit__()} is an abstract method which by default
returns @code{None}. See also the definition of
@ref{1821,,Asynchronous Context Managers}.


New in version 3.7.

@end deffn

@geindex contextmanager() (in module contextlib)
@anchor{library/contextlib contextlib contextmanager}@anchor{da3}
@deffn {Function} @@contextlib.contextmanager

This function is a @ref{1bf,,decorator} that can be used to define a factory
function for @ref{168,,with} statement context managers, without needing to
create a class or separate @code{__enter__()} and @code{__exit__()} methods.

While many objects natively support use in with statements, sometimes a
resource needs to be managed that isn’t a context manager in its own right,
and doesn’t implement a @code{close()} method for use with @code{contextlib.closing}

An abstract example would be the following to ensure correct resource
management:

@example
from contextlib import contextmanager

@@contextmanager
def managed_resource(*args, **kwds):
    # Code to acquire resource, e.g.:
    resource = acquire_resource(*args, **kwds)
    try:
        yield resource
    finally:
        # Code to release resource, e.g.:
        release_resource(resource)

>>> with managed_resource(timeout=3600) as resource:
...     # Resource is released at the end of this block,
...     # even if code in the block raises an exception
@end example

The function being decorated must return a @ref{bf7,,generator}-iterator when
called. This iterator must yield exactly one value, which will be bound to
the targets in the @ref{168,,with} statement’s @code{as} clause, if any.

At the point where the generator yields, the block nested in the @ref{168,,with}
statement is executed.  The generator is then resumed after the block is exited.
If an unhandled exception occurs in the block, it is reraised inside the
generator at the point where the yield occurred.  Thus, you can use a
@ref{f6d,,try}…@ref{16e,,except}…@ref{4c1,,finally} statement to trap
the error (if any), or ensure that some cleanup takes place. If an exception is
trapped merely in order to log it or to perform some action (rather than to
suppress it entirely), the generator must reraise that exception. Otherwise the
generator context manager will indicate to the @code{with} statement that
the exception has been handled, and execution will resume with the statement
immediately following the @code{with} statement.

@ref{da3,,contextmanager()} uses @ref{da2,,ContextDecorator} so the context managers
it creates can be used as decorators as well as in @ref{168,,with} statements.
When used as a decorator, a new generator instance is implicitly created on
each function call (this allows the otherwise “one-shot” context managers
created by @ref{da3,,contextmanager()} to meet the requirement that context
managers support multiple invocations in order to be used as decorators).


Changed in version 3.2: Use of @ref{da2,,ContextDecorator}.

@end deffn

@geindex asynccontextmanager() (in module contextlib)
@anchor{library/contextlib contextlib asynccontextmanager}@anchor{649}
@deffn {Function} @@contextlib.asynccontextmanager

@quotation

Similar to @ref{da3,,contextmanager()}, but creates an
@ref{1821,,asynchronous context manager}.

This function is a @ref{1bf,,decorator} that can be used to define a factory
function for @ref{169,,async with} statement asynchronous context managers,
without needing to create a class or separate @code{__aenter__()} and
@code{__aexit__()} methods. It must be applied to an @ref{18ac,,asynchronous generator} function.

A simple example:

@example
from contextlib import asynccontextmanager

@@asynccontextmanager
async def get_connection():
    conn = await acquire_db_connection()
    try:
        yield conn
    finally:
        await release_db_connection(conn)

async def get_all_users():
    async with get_connection() as conn:
        return conn.query('SELECT ...')
@end example


New in version 3.7.


Context managers defined with @ref{649,,asynccontextmanager()} can be used
either as decorators or with @ref{169,,async with} statements:

@example
import time
from contextlib import asynccontextmanager

@@asynccontextmanager
async def timeit():
    now = time.monotonic()
    try:
        yield
    finally:
        print(f'it took @{time.monotonic() - now@}s to run')

 @@timeit()
 async def main():
     # ... async code ...
@end example

When used as a decorator, a new generator instance is implicitly created on
each function call. This allows the otherwise “one-shot” context managers
created by @ref{649,,asynccontextmanager()} to meet the requirement that context
managers support multiple invocations in order to be used as decorators.
@end quotation


Changed in version 3.10: Async context managers created with @ref{649,,asynccontextmanager()} can
be used as decorators.

@end deffn

@geindex closing() (in module contextlib)
@anchor{library/contextlib contextlib closing}@anchor{3bbd}
@deffn {Function} contextlib.closing (thing)

Return a context manager that closes @emph{thing} upon completion of the block.  This
is basically equivalent to:

@example
from contextlib import contextmanager

@@contextmanager
def closing(thing):
    try:
        yield thing
    finally:
        thing.close()
@end example

And lets you write code like this:

@example
from contextlib import closing
from urllib.request import urlopen

with closing(urlopen('https://www.python.org')) as page:
    for line in page:
        print(line)
@end example

without needing to explicitly close @code{page}.  Even if an error occurs,
@code{page.close()} will be called when the @ref{168,,with} block is exited.
@end deffn

@geindex aclosing (class in contextlib)
@anchor{library/contextlib contextlib aclosing}@anchor{26c}
@deffn {Class} contextlib.aclosing (thing)

Return an async context manager that calls the @code{aclose()} method of @emph{thing}
upon completion of the block.  This is basically equivalent to:

@example
from contextlib import asynccontextmanager

@@asynccontextmanager
async def aclosing(thing):
    try:
        yield thing
    finally:
        await thing.aclose()
@end example

Significantly, @code{aclosing()} supports deterministic cleanup of async
generators when they happen to exit early by @ref{5cb,,break} or an
exception.  For example:

@example
from contextlib import aclosing

async with aclosing(my_generator()) as values:
    async for value in values:
        if value == 42:
            break
@end example

This pattern ensures that the generator’s async exit code is executed in
the same context as its iterations (so that exceptions and context
variables work as expected, and the exit code isn’t run after the
lifetime of some task it depends on).


New in version 3.10.

@end deffn
@anchor{library/contextlib simplifying-support-for-single-optional-context-managers}@anchor{3bbe}
@geindex nullcontext() (in module contextlib)
@anchor{library/contextlib contextlib nullcontext}@anchor{26d}
@deffn {Function} contextlib.nullcontext (enter_result=None)

Return a context manager that returns @emph{enter_result} from @code{__enter__}, but
otherwise does nothing. It is intended to be used as a stand-in for an
optional context manager, for example:

@example
def myfunction(arg, ignore_exceptions=False):
    if ignore_exceptions:
        # Use suppress to ignore all exceptions.
        cm = contextlib.suppress(Exception)
    else:
        # Do not ignore any exceptions, cm has no effect.
        cm = contextlib.nullcontext()
    with cm:
        # Do something
@end example

An example using @emph{enter_result}:

@example
def process_file(file_or_path):
    if isinstance(file_or_path, str):
        # If string, open file
        cm = open(file_or_path)
    else:
        # Caller is responsible for closing file
        cm = nullcontext(file_or_path)

    with cm as file:
        # Perform processing on the file
@end example

It can also be used as a stand-in for
@ref{1821,,asynchronous context managers}:

@example
async def send_http(session=None):
   if not session:
       # If no http session, create it with aiohttp
       cm = aiohttp.ClientSession()
   else:
       # Caller is responsible for closing the session
       cm = nullcontext(session)

   async with cm as session:
       # Send http requests with session
@end example


New in version 3.7.



Changed in version 3.10: @ref{167,,asynchronous context manager} support was added.

@end deffn

@geindex suppress() (in module contextlib)
@anchor{library/contextlib contextlib suppress}@anchor{aa0}
@deffn {Function} contextlib.suppress (*exceptions)

Return a context manager that suppresses any of the specified exceptions
if they occur in the body of a @code{with} statement and then
resumes execution with the first statement following the end of the
@code{with} statement.

As with any other mechanism that completely suppresses exceptions, this
context manager should be used only to cover very specific errors where
silently continuing with program execution is known to be the right
thing to do.

For example:

@example
from contextlib import suppress

with suppress(FileNotFoundError):
    os.remove('somefile.tmp')

with suppress(FileNotFoundError):
    os.remove('someotherfile.tmp')
@end example

This code is equivalent to:

@example
try:
    os.remove('somefile.tmp')
except FileNotFoundError:
    pass

try:
    os.remove('someotherfile.tmp')
except FileNotFoundError:
    pass
@end example

This context manager is @ref{3bbf,,reentrant}.


New in version 3.4.

@end deffn

@geindex redirect_stdout() (in module contextlib)
@anchor{library/contextlib contextlib redirect_stdout}@anchor{947}
@deffn {Function} contextlib.redirect_stdout (new_target)

Context manager for temporarily redirecting @ref{5ef,,sys.stdout} to
another file or file-like object.

This tool adds flexibility to existing functions or classes whose output
is hardwired to stdout.

For example, the output of @ref{3bc,,help()} normally is sent to @emph{sys.stdout}.
You can capture that output in a string by redirecting the output to an
@ref{aa1,,io.StringIO} object. The replacement stream is returned from the
@code{__enter__} method and so is available as the target of the
@ref{168,,with} statement:

@example
with redirect_stdout(io.StringIO()) as f:
    help(pow)
s = f.getvalue()
@end example

To send the output of @ref{3bc,,help()} to a file on disk, redirect the output
to a regular file:

@example
with open('help.txt', 'w') as f:
    with redirect_stdout(f):
        help(pow)
@end example

To send the output of @ref{3bc,,help()} to @emph{sys.stderr}:

@example
with redirect_stdout(sys.stderr):
    help(pow)
@end example

Note that the global side effect on @ref{5ef,,sys.stdout} means that this
context manager is not suitable for use in library code and most threaded
applications. It also has no effect on the output of subprocesses.
However, it is still a useful approach for many utility scripts.

This context manager is @ref{3bbf,,reentrant}.


New in version 3.4.

@end deffn

@geindex redirect_stderr() (in module contextlib)
@anchor{library/contextlib contextlib redirect_stderr}@anchor{946}
@deffn {Function} contextlib.redirect_stderr (new_target)

Similar to @ref{947,,redirect_stdout()} but redirecting
@ref{42d,,sys.stderr} to another file or file-like object.

This context manager is @ref{3bbf,,reentrant}.


New in version 3.5.

@end deffn

@geindex chdir() (in module contextlib)
@anchor{library/contextlib contextlib chdir}@anchor{10e4}
@deffn {Function} contextlib.chdir (path)

Non parallel-safe context manager to change the current working directory.
As this changes a global state, the working directory, it is not suitable
for use in most threaded or async contexts. It is also not suitable for most
non-linear code execution, like generators, where the program execution is
temporarily relinquished – unless explicitely desired, you should not yield
when this context manager is active.

This is a simple wrapper around @ref{3b9,,chdir()}, it changes the current
working directory upon entering and restores the old one on exit.

This context manager is @ref{3bbf,,reentrant}.


New in version 3.11.

@end deffn

@geindex ContextDecorator (class in contextlib)
@anchor{library/contextlib contextlib ContextDecorator}@anchor{da2}
@deffn {Class} contextlib.ContextDecorator

A base class that enables a context manager to also be used as a decorator.

Context managers inheriting from @code{ContextDecorator} have to implement
@code{__enter__} and @code{__exit__} as normal. @code{__exit__} retains its optional
exception handling even when used as a decorator.

@code{ContextDecorator} is used by @ref{da3,,contextmanager()}, so you get this
functionality automatically.

Example of @code{ContextDecorator}:

@example
from contextlib import ContextDecorator

class mycontext(ContextDecorator):
    def __enter__(self):
        print('Starting')
        return self

    def __exit__(self, *exc):
        print('Finishing')
        return False

>>> @@mycontext()
... def function():
...     print('The bit in the middle')
...
>>> function()
Starting
The bit in the middle
Finishing

>>> with mycontext():
...     print('The bit in the middle')
...
Starting
The bit in the middle
Finishing
@end example

This change is just syntactic sugar for any construct of the following form:

@example
def f():
    with cm():
        # Do stuff
@end example

@code{ContextDecorator} lets you instead write:

@example
@@cm()
def f():
    # Do stuff
@end example

It makes it clear that the @code{cm} applies to the whole function, rather than
just a piece of it (and saving an indentation level is nice, too).

Existing context managers that already have a base class can be extended by
using @code{ContextDecorator} as a mixin class:

@example
from contextlib import ContextDecorator

class mycontext(ContextBaseClass, ContextDecorator):
    def __enter__(self):
        return self

    def __exit__(self, *exc):
        return False
@end example

@cartouche
@quotation Note 
As the decorated function must be able to be called multiple times, the
underlying context manager must support use in multiple @ref{168,,with}
statements. If this is not the case, then the original construct with the
explicit @code{with} statement inside the function should be used.
@end quotation
@end cartouche


New in version 3.2.

@end deffn

@geindex AsyncContextDecorator (class in contextlib)
@anchor{library/contextlib contextlib AsyncContextDecorator}@anchor{3bc0}
@deffn {Class} contextlib.AsyncContextDecorator

Similar to @ref{da2,,ContextDecorator} but only for asynchronous functions.

Example of @code{AsyncContextDecorator}:

@example
from asyncio import run
from contextlib import AsyncContextDecorator

class mycontext(AsyncContextDecorator):
    async def __aenter__(self):
        print('Starting')
        return self

    async def __aexit__(self, *exc):
        print('Finishing')
        return False

>>> @@mycontext()
... async def function():
...     print('The bit in the middle')
...
>>> run(function())
Starting
The bit in the middle
Finishing

>>> async def function():
...    async with mycontext():
...         print('The bit in the middle')
...
>>> run(function())
Starting
The bit in the middle
Finishing
@end example


New in version 3.10.

@end deffn

@geindex ExitStack (class in contextlib)
@anchor{library/contextlib contextlib ExitStack}@anchor{648}
@deffn {Class} contextlib.ExitStack

A context manager that is designed to make it easy to programmatically
combine other context managers and cleanup functions, especially those
that are optional or otherwise driven by input data.

For example, a set of files may easily be handled in a single with
statement as follows:

@example
with ExitStack() as stack:
    files = [stack.enter_context(open(fname)) for fname in filenames]
    # All opened files will automatically be closed at the end of
    # the with statement, even if attempts to open files later
    # in the list raise an exception
@end example

Each instance maintains a stack of registered callbacks that are called in
reverse order when the instance is closed (either explicitly or implicitly
at the end of a @ref{168,,with} statement). Note that callbacks are @emph{not}
invoked implicitly when the context stack instance is garbage collected.

This stack model is used so that context managers that acquire their
resources in their @code{__init__} method (such as file objects) can be
handled correctly.

Since registered callbacks are invoked in the reverse order of
registration, this ends up behaving as if multiple nested @ref{168,,with}
statements had been used with the registered set of callbacks. This even
extends to exception handling - if an inner callback suppresses or replaces
an exception, then outer callbacks will be passed arguments based on that
updated state.

This is a relatively low level API that takes care of the details of
correctly unwinding the stack of exit callbacks. It provides a suitable
foundation for higher level context managers that manipulate the exit
stack in application specific ways.


New in version 3.3.


@geindex enter_context() (contextlib.ExitStack method)
@anchor{library/contextlib contextlib ExitStack enter_context}@anchor{164}
@deffn {Method} enter_context (cm)

Enters a new context manager and adds its @code{__exit__()} method to
the callback stack. The return value is the result of the context
manager’s own @code{__enter__()} method.

These context managers may suppress exceptions just as they normally
would if used directly as part of a @ref{168,,with} statement.


Changed in version 3.11: Raises @ref{162,,TypeError} instead of @ref{163,,AttributeError} if @emph{cm}
is not a context manager.

@end deffn

@geindex push() (contextlib.ExitStack method)
@anchor{library/contextlib contextlib ExitStack push}@anchor{3bc1}
@deffn {Method} push (exit)

Adds a context manager’s @code{__exit__()} method to the callback stack.

As @code{__enter__} is @emph{not} invoked, this method can be used to cover
part of an @code{__enter__()} implementation with a context manager’s own
@code{__exit__()} method.

If passed an object that is not a context manager, this method assumes
it is a callback with the same signature as a context manager’s
@code{__exit__()} method and adds it directly to the callback stack.

By returning true values, these callbacks can suppress exceptions the
same way context manager @code{__exit__()} methods can.

The passed in object is returned from the function, allowing this
method to be used as a function decorator.
@end deffn

@geindex callback() (contextlib.ExitStack method)
@anchor{library/contextlib contextlib ExitStack callback}@anchor{59b}
@deffn {Method} callback (callback, /, *args, **kwds)

Accepts an arbitrary callback function and arguments and adds it to
the callback stack.

Unlike the other methods, callbacks added this way cannot suppress
exceptions (as they are never passed the exception details).

The passed in callback is returned from the function, allowing this
method to be used as a function decorator.
@end deffn

@geindex pop_all() (contextlib.ExitStack method)
@anchor{library/contextlib contextlib ExitStack pop_all}@anchor{3bc2}
@deffn {Method} pop_all ()

Transfers the callback stack to a fresh @ref{648,,ExitStack} instance
and returns it. No callbacks are invoked by this operation - instead,
they will now be invoked when the new stack is closed (either
explicitly or implicitly at the end of a @ref{168,,with} statement).

For example, a group of files can be opened as an “all or nothing”
operation as follows:

@example
with ExitStack() as stack:
    files = [stack.enter_context(open(fname)) for fname in filenames]
    # Hold onto the close method, but don't call it yet.
    close_files = stack.pop_all().close
    # If opening any file fails, all previously opened files will be
    # closed automatically. If all files are opened successfully,
    # they will remain open even after the with statement ends.
    # close_files() can then be invoked explicitly to close them all.
@end example
@end deffn

@geindex close() (contextlib.ExitStack method)
@anchor{library/contextlib contextlib ExitStack close}@anchor{3bc3}
@deffn {Method} close ()

Immediately unwinds the callback stack, invoking callbacks in the
reverse order of registration. For any context managers and exit
callbacks registered, the arguments passed in will indicate that no
exception occurred.
@end deffn
@end deffn

@geindex AsyncExitStack (class in contextlib)
@anchor{library/contextlib contextlib AsyncExitStack}@anchor{64b}
@deffn {Class} contextlib.AsyncExitStack

An @ref{1821,,asynchronous context manager}, similar
to @ref{648,,ExitStack}, that supports combining both synchronous and
asynchronous context managers, as well as having coroutines for
cleanup logic.

The @code{close()} method is not implemented, @ref{3bc4,,aclose()} must be used
instead.

@geindex enter_async_context() (contextlib.AsyncExitStack method)
@anchor{library/contextlib contextlib AsyncExitStack enter_async_context}@anchor{165}
@deffn {Method} coroutine enter_async_context (cm)

Similar to @code{enter_context()} but expects an asynchronous context
manager.


Changed in version 3.11: Raises @ref{162,,TypeError} instead of @ref{163,,AttributeError} if @emph{cm}
is not an asynchronous context manager.

@end deffn

@geindex push_async_exit() (contextlib.AsyncExitStack method)
@anchor{library/contextlib contextlib AsyncExitStack push_async_exit}@anchor{3bc5}
@deffn {Method} push_async_exit (exit)

Similar to @code{push()} but expects either an asynchronous context manager
or a coroutine function.
@end deffn

@geindex push_async_callback() (contextlib.AsyncExitStack method)
@anchor{library/contextlib contextlib AsyncExitStack push_async_callback}@anchor{59c}
@deffn {Method} push_async_callback (callback, /, *args, **kwds)

Similar to @code{callback()} but expects a coroutine function.
@end deffn

@geindex aclose() (contextlib.AsyncExitStack method)
@anchor{library/contextlib contextlib AsyncExitStack aclose}@anchor{3bc4}
@deffn {Method} coroutine aclose ()

Similar to @code{close()} but properly handles awaitables.
@end deffn

Continuing the example for @ref{649,,asynccontextmanager()}:

@example
async with AsyncExitStack() as stack:
    connections = [await stack.enter_async_context(get_connection())
        for i in range(5)]
    # All opened connections will automatically be released at the end of
    # the async with statement, even if attempts to open a connection
    # later in the list raise an exception.
@end example


New in version 3.7.

@end deffn

@node Examples and Recipes<2>,Single use reusable and reentrant context managers,Utilities,contextlib — Utilities for with-statement contexts
@anchor{library/contextlib examples-and-recipes}@anchor{3bc6}
@subsubsection Examples and Recipes


This section describes some examples and recipes for making effective use of
the tools provided by @ref{22,,contextlib}.

@menu
* Supporting a variable number of context managers:: 
* Catching exceptions from __enter__ methods:: 
* Cleaning up in an __enter__ implementation:: 
* Replacing any use of try-finally and flag variables:: 
* Using a context manager as a function decorator:: 

@end menu

@node Supporting a variable number of context managers,Catching exceptions from __enter__ methods,,Examples and Recipes<2>
@anchor{library/contextlib supporting-a-variable-number-of-context-managers}@anchor{3bc7}
@subsubsection Supporting a variable number of context managers


The primary use case for @ref{648,,ExitStack} is the one given in the class
documentation: supporting a variable number of context managers and other
cleanup operations in a single @ref{168,,with} statement. The variability
may come from the number of context managers needed being driven by user
input (such as opening a user specified collection of files), or from
some of the context managers being optional:

@example
with ExitStack() as stack:
    for resource in resources:
        stack.enter_context(resource)
    if need_special_resource():
        special = acquire_special_resource()
        stack.callback(release_special_resource, special)
    # Perform operations that use the acquired resources
@end example

As shown, @ref{648,,ExitStack} also makes it quite easy to use @ref{168,,with}
statements to manage arbitrary resources that don’t natively support the
context management protocol.

@node Catching exceptions from __enter__ methods,Cleaning up in an __enter__ implementation,Supporting a variable number of context managers,Examples and Recipes<2>
@anchor{library/contextlib catching-exceptions-from-enter-methods}@anchor{3bc8}
@subsubsection Catching exceptions from @code{__enter__} methods


It is occasionally desirable to catch exceptions from an @code{__enter__}
method implementation, @emph{without} inadvertently catching exceptions from
the @ref{168,,with} statement body or the context manager’s @code{__exit__}
method. By using @ref{648,,ExitStack} the steps in the context management
protocol can be separated slightly in order to allow this:

@example
stack = ExitStack()
try:
    x = stack.enter_context(cm)
except Exception:
    # handle __enter__ exception
else:
    with stack:
        # Handle normal case
@end example

Actually needing to do this is likely to indicate that the underlying API
should be providing a direct resource management interface for use with
@ref{f6d,,try}/@ref{16e,,except}/@ref{4c1,,finally} statements, but not
all APIs are well designed in that regard. When a context manager is the
only resource management API provided, then @ref{648,,ExitStack} can make it
easier to handle various situations that can’t be handled directly in a
@ref{168,,with} statement.

@node Cleaning up in an __enter__ implementation,Replacing any use of try-finally and flag variables,Catching exceptions from __enter__ methods,Examples and Recipes<2>
@anchor{library/contextlib cleaning-up-in-an-enter-implementation}@anchor{3bc9}
@subsubsection Cleaning up in an @code{__enter__} implementation


As noted in the documentation of @ref{3bc1,,ExitStack.push()}, this
method can be useful in cleaning up an already allocated resource if later
steps in the @code{__enter__()} implementation fail.

Here’s an example of doing this for a context manager that accepts resource
acquisition and release functions, along with an optional validation function,
and maps them to the context management protocol:

@example
from contextlib import contextmanager, AbstractContextManager, ExitStack

class ResourceManager(AbstractContextManager):

    def __init__(self, acquire_resource, release_resource, check_resource_ok=None):
        self.acquire_resource = acquire_resource
        self.release_resource = release_resource
        if check_resource_ok is None:
            def check_resource_ok(resource):
                return True
        self.check_resource_ok = check_resource_ok

    @@contextmanager
    def _cleanup_on_error(self):
        with ExitStack() as stack:
            stack.push(self)
            yield
            # The validation check passed and didn't raise an exception
            # Accordingly, we want to keep the resource, and pass it
            # back to our caller
            stack.pop_all()

    def __enter__(self):
        resource = self.acquire_resource()
        with self._cleanup_on_error():
            if not self.check_resource_ok(resource):
                msg = "Failed validation for @{!r@}"
                raise RuntimeError(msg.format(resource))
        return resource

    def __exit__(self, *exc_details):
        # We don't need to duplicate any of our resource release logic
        self.release_resource()
@end example

@node Replacing any use of try-finally and flag variables,Using a context manager as a function decorator,Cleaning up in an __enter__ implementation,Examples and Recipes<2>
@anchor{library/contextlib replacing-any-use-of-try-finally-and-flag-variables}@anchor{3bca}
@subsubsection Replacing any use of @code{try-finally} and flag variables


A pattern you will sometimes see is a @code{try-finally} statement with a flag
variable to indicate whether or not the body of the @code{finally} clause should
be executed. In its simplest form (that can’t already be handled just by
using an @code{except} clause instead), it looks something like this:

@example
cleanup_needed = True
try:
    result = perform_operation()
    if result:
        cleanup_needed = False
finally:
    if cleanup_needed:
        cleanup_resources()
@end example

As with any @code{try} statement based code, this can cause problems for
development and review, because the setup code and the cleanup code can end
up being separated by arbitrarily long sections of code.

@ref{648,,ExitStack} makes it possible to instead register a callback for
execution at the end of a @code{with} statement, and then later decide to skip
executing that callback:

@example
from contextlib import ExitStack

with ExitStack() as stack:
    stack.callback(cleanup_resources)
    result = perform_operation()
    if result:
        stack.pop_all()
@end example

This allows the intended cleanup up behaviour to be made explicit up front,
rather than requiring a separate flag variable.

If a particular application uses this pattern a lot, it can be simplified
even further by means of a small helper class:

@example
from contextlib import ExitStack

class Callback(ExitStack):
    def __init__(self, callback, /, *args, **kwds):
        super().__init__()
        self.callback(callback, *args, **kwds)

    def cancel(self):
        self.pop_all()

with Callback(cleanup_resources) as cb:
    result = perform_operation()
    if result:
        cb.cancel()
@end example

If the resource cleanup isn’t already neatly bundled into a standalone
function, then it is still possible to use the decorator form of
@ref{59b,,ExitStack.callback()} to declare the resource cleanup in
advance:

@example
from contextlib import ExitStack

with ExitStack() as stack:
    @@stack.callback
    def cleanup_resources():
        ...
    result = perform_operation()
    if result:
        stack.pop_all()
@end example

Due to the way the decorator protocol works, a callback function
declared this way cannot take any parameters. Instead, any resources to
be released must be accessed as closure variables.

@node Using a context manager as a function decorator,,Replacing any use of try-finally and flag variables,Examples and Recipes<2>
@anchor{library/contextlib using-a-context-manager-as-a-function-decorator}@anchor{3bcb}
@subsubsection Using a context manager as a function decorator


@ref{da2,,ContextDecorator} makes it possible to use a context manager in
both an ordinary @code{with} statement and also as a function decorator.

For example, it is sometimes useful to wrap functions or groups of statements
with a logger that can track the time of entry and time of exit.  Rather than
writing both a function decorator and a context manager for the task,
inheriting from @ref{da2,,ContextDecorator} provides both capabilities in a
single definition:

@example
from contextlib import ContextDecorator
import logging

logging.basicConfig(level=logging.INFO)

class track_entry_and_exit(ContextDecorator):
    def __init__(self, name):
        self.name = name

    def __enter__(self):
        logging.info('Entering: %s', self.name)

    def __exit__(self, exc_type, exc, exc_tb):
        logging.info('Exiting: %s', self.name)
@end example

Instances of this class can be used as both a context manager:

@example
with track_entry_and_exit('widget loader'):
    print('Some time consuming activity goes here')
    load_widget()
@end example

And also as a function decorator:

@example
@@track_entry_and_exit('widget loader')
def activity():
    print('Some time consuming activity goes here')
    load_widget()
@end example

Note that there is one additional limitation when using context managers
as function decorators: there’s no way to access the return value of
@code{__enter__()}. If that value is needed, then it is still necessary to use
an explicit @code{with} statement.


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 343

@item PEP 343@footnote{https://www.python.org/dev/peps/pep-0343} - The “with” statement

The specification, background, and examples for the Python @ref{168,,with}
statement.
@end table


@node Single use reusable and reentrant context managers,,Examples and Recipes<2>,contextlib — Utilities for with-statement contexts
@anchor{library/contextlib single-use-reusable-and-reentrant-cms}@anchor{aa2}@anchor{library/contextlib single-use-reusable-and-reentrant-context-managers}@anchor{3bcc}
@subsubsection Single use, reusable and reentrant context managers


Most context managers are written in a way that means they can only be
used effectively in a @ref{168,,with} statement once. These single use
context managers must be created afresh each time they’re used -
attempting to use them a second time will trigger an exception or
otherwise not work correctly.

This common limitation means that it is generally advisable to create
context managers directly in the header of the @ref{168,,with} statement
where they are used (as shown in all of the usage examples above).

Files are an example of effectively single use context managers, since
the first @ref{168,,with} statement will close the file, preventing any
further IO operations using that file object.

Context managers created using @ref{da3,,contextmanager()} are also single use
context managers, and will complain about the underlying generator failing
to yield if an attempt is made to use them a second time:

@example
>>> from contextlib import contextmanager
>>> @@contextmanager
... def singleuse():
...     print("Before")
...     yield
...     print("After")
...
>>> cm = singleuse()
>>> with cm:
...     pass
...
Before
After
>>> with cm:
...     pass
...
Traceback (most recent call last):
    ...
RuntimeError: generator didn't yield
@end example

@menu
* Reentrant context managers:: 
* Reusable context managers:: 

@end menu

@node Reentrant context managers,Reusable context managers,,Single use reusable and reentrant context managers
@anchor{library/contextlib reentrant-cms}@anchor{3bbf}@anchor{library/contextlib reentrant-context-managers}@anchor{3bcd}
@subsubsection Reentrant context managers


More sophisticated context managers may be “reentrant”. These context
managers can not only be used in multiple @ref{168,,with} statements,
but may also be used @emph{inside} a @code{with} statement that is already
using the same context manager.

@ref{e09,,threading.RLock} is an example of a reentrant context manager, as are
@ref{aa0,,suppress()}, @ref{947,,redirect_stdout()}, and @ref{10e4,,chdir()}. Here’s a very
simple example of reentrant use:

@example
>>> from contextlib import redirect_stdout
>>> from io import StringIO
>>> stream = StringIO()
>>> write_to_stream = redirect_stdout(stream)
>>> with write_to_stream:
...     print("This is written to the stream rather than stdout")
...     with write_to_stream:
...         print("This is also written to the stream")
...
>>> print("This is written directly to stdout")
This is written directly to stdout
>>> print(stream.getvalue())
This is written to the stream rather than stdout
This is also written to the stream
@end example

Real world examples of reentrancy are more likely to involve multiple
functions calling each other and hence be far more complicated than this
example.

Note also that being reentrant is @emph{not} the same thing as being thread safe.
@ref{947,,redirect_stdout()}, for example, is definitely not thread safe, as it
makes a global modification to the system state by binding @ref{5ef,,sys.stdout}
to a different stream.

@node Reusable context managers,,Reentrant context managers,Single use reusable and reentrant context managers
@anchor{library/contextlib reusable-cms}@anchor{3bce}@anchor{library/contextlib reusable-context-managers}@anchor{3bcf}
@subsubsection Reusable context managers


Distinct from both single use and reentrant context managers are “reusable”
context managers (or, to be completely explicit, “reusable, but not
reentrant” context managers, since reentrant context managers are also
reusable). These context managers support being used multiple times, but
will fail (or otherwise not work correctly) if the specific context manager
instance has already been used in a containing with statement.

@ref{27f5,,threading.Lock} is an example of a reusable, but not reentrant,
context manager (for a reentrant lock, it is necessary to use
@ref{e09,,threading.RLock} instead).

Another example of a reusable, but not reentrant, context manager is
@ref{648,,ExitStack}, as it invokes @emph{all} currently registered callbacks
when leaving any with statement, regardless of where those callbacks
were added:

@example
>>> from contextlib import ExitStack
>>> stack = ExitStack()
>>> with stack:
...     stack.callback(print, "Callback: from first context")
...     print("Leaving first context")
...
Leaving first context
Callback: from first context
>>> with stack:
...     stack.callback(print, "Callback: from second context")
...     print("Leaving second context")
...
Leaving second context
Callback: from second context
>>> with stack:
...     stack.callback(print, "Callback: from outer context")
...     with stack:
...         stack.callback(print, "Callback: from inner context")
...         print("Leaving inner context")
...     print("Leaving outer context")
...
Leaving inner context
Callback: from inner context
Callback: from outer context
Leaving outer context
@end example

As the output from the example shows, reusing a single stack object across
multiple with statements works correctly, but attempting to nest them
will cause the stack to be cleared at the end of the innermost with
statement, which is unlikely to be desirable behaviour.

Using separate @ref{648,,ExitStack} instances instead of reusing a single
instance avoids that problem:

@example
>>> from contextlib import ExitStack
>>> with ExitStack() as outer_stack:
...     outer_stack.callback(print, "Callback: from outer context")
...     with ExitStack() as inner_stack:
...         inner_stack.callback(print, "Callback: from inner context")
...         print("Leaving inner context")
...     print("Leaving outer context")
...
Leaving inner context
Callback: from inner context
Leaving outer context
Callback: from outer context
@end example

@node abc — Abstract Base Classes,atexit — Exit handlers,contextlib — Utilities for with-statement contexts,Python Runtime Services
@anchor{library/abc doc}@anchor{3bd0}@anchor{library/abc abc-abstract-base-classes}@anchor{3bd1}@anchor{library/abc module-abc}@anchor{3}
@subsection @code{abc} — Abstract Base Classes


@geindex module; abc

@c much of the content adapted from docstrings

@strong{Source code:} Lib/abc.py@footnote{https://github.com/python/cpython/tree/main/Lib/abc.py}


__________________________________________________________________


This module provides the infrastructure for defining @ref{d5c,,abstract base classes} (ABCs) in Python, as outlined in 
@geindex Python Enhancement Proposals; PEP 3119
PEP 3119@footnote{https://www.python.org/dev/peps/pep-3119};
see the PEP for why this was added to Python. (See also 
@geindex Python Enhancement Proposals; PEP 3141
PEP 3141@footnote{https://www.python.org/dev/peps/pep-3141} and the
@ref{bd,,numbers} module regarding a type hierarchy for numbers based on ABCs.)

The @ref{1c,,collections} module has some concrete classes that derive from
ABCs; these can, of course, be further derived. In addition, the
@ref{1d,,collections.abc} submodule has some ABCs that can be used to test whether
a class or instance provides a particular interface, for example, if it is
hashable or if it is a mapping.

This module provides the metaclass @ref{a8d,,ABCMeta} for defining ABCs and
a helper class @ref{a8c,,ABC} to alternatively define ABCs through inheritance:

@geindex ABC (class in abc)
@anchor{library/abc abc ABC}@anchor{a8c}
@deffn {Class} abc.ABC

A helper class that has @ref{a8d,,ABCMeta} as its metaclass.  With this class,
an abstract base class can be created by simply deriving from @ref{a8c,,ABC}
avoiding sometimes confusing metaclass usage, for example:

@example
from abc import ABC

class MyABC(ABC):
    pass
@end example

Note that the type of @ref{a8c,,ABC} is still @ref{a8d,,ABCMeta}, therefore
inheriting from @ref{a8c,,ABC} requires the usual precautions regarding
metaclass usage, as multiple inheritance may lead to metaclass conflicts.
One may also define an abstract base class by passing the metaclass
keyword and using @ref{a8d,,ABCMeta} directly, for example:

@example
from abc import ABCMeta

class MyABC(metaclass=ABCMeta):
    pass
@end example


New in version 3.4.

@end deffn

@geindex ABCMeta (class in abc)
@anchor{library/abc abc ABCMeta}@anchor{a8d}
@deffn {Class} abc.ABCMeta

Metaclass for defining Abstract Base Classes (ABCs).

Use this metaclass to create an ABC.  An ABC can be subclassed directly, and
then acts as a mix-in class.  You can also register unrelated concrete
classes (even built-in classes) and unrelated ABCs as “virtual subclasses” –
these and their descendants will be considered subclasses of the registering
ABC by the built-in @ref{237,,issubclass()} function, but the registering ABC
won’t show up in their MRO (Method Resolution Order) nor will method
implementations defined by the registering ABC be callable (not even via
@ref{78c,,super()}). @footnote{@w{(1)} 
C++ programmers should note that Python’s virtual base class
concept is not the same as C++’s.
}

Classes created with a metaclass of @ref{a8d,,ABCMeta} have the following method:

@geindex register() (abc.ABCMeta method)
@anchor{library/abc abc ABCMeta register}@anchor{c21}
@deffn {Method} register (subclass)

Register @emph{subclass} as a “virtual subclass” of this ABC. For
example:

@example
from abc import ABC

class MyABC(ABC):
    pass

MyABC.register(tuple)

assert issubclass(tuple, MyABC)
assert isinstance((), MyABC)
@end example


Changed in version 3.3: Returns the registered subclass, to allow usage as a class decorator.



Changed in version 3.4: To detect calls to @ref{c21,,register()}, you can use the
@ref{a8b,,get_cache_token()} function.

@end deffn

You can also override this method in an abstract base class:

@geindex __subclasshook__() (abc.ABCMeta method)
@anchor{library/abc abc ABCMeta __subclasshook__}@anchor{3bd2}
@deffn {Method} __subclasshook__ (subclass)

(Must be defined as a class method.)

Check whether @emph{subclass} is considered a subclass of this ABC.  This means
that you can customize the behavior of @code{issubclass} further without the
need to call @ref{c21,,register()} on every class you want to consider a
subclass of the ABC.  (This class method is called from the
@code{__subclasscheck__()} method of the ABC.)

This method should return @code{True}, @code{False} or @code{NotImplemented}.  If
it returns @code{True}, the @emph{subclass} is considered a subclass of this ABC.
If it returns @code{False}, the @emph{subclass} is not considered a subclass of
this ABC, even if it would normally be one.  If it returns
@code{NotImplemented}, the subclass check is continued with the usual
mechanism.

@c XXX explain the "usual mechanism"
@end deffn

For a demonstration of these concepts, look at this example ABC definition:

@example
class Foo:
    def __getitem__(self, index):
        ...
    def __len__(self):
        ...
    def get_iterator(self):
        return iter(self)

class MyIterable(ABC):

    @@abstractmethod
    def __iter__(self):
        while False:
            yield None

    def get_iterator(self):
        return self.__iter__()

    @@classmethod
    def __subclasshook__(cls, C):
        if cls is MyIterable:
            if any("__iter__" in B.__dict__ for B in C.__mro__):
                return True
        return NotImplemented

MyIterable.register(Foo)
@end example

The ABC @code{MyIterable} defines the standard iterable method,
@ref{1a84,,__iter__()}, as an abstract method.  The implementation given
here can still be called from subclasses.  The @code{get_iterator()} method
is also part of the @code{MyIterable} abstract base class, but it does not have
to be overridden in non-abstract derived classes.

The @ref{3bd2,,__subclasshook__()} class method defined here says that any class
that has an @ref{1a84,,__iter__()} method in its
@ref{7a4,,__dict__} (or in that of one of its base classes, accessed
via the @ref{1a61,,__mro__} list) is considered a @code{MyIterable} too.

Finally, the last line makes @code{Foo} a virtual subclass of @code{MyIterable},
even though it does not define an @ref{1a84,,__iter__()} method (it uses
the old-style iterable protocol, defined in terms of @code{__len__()} and
@code{__getitem__()}).  Note that this will not make @code{get_iterator}
available as a method of @code{Foo}, so it is provided separately.
@end deffn

The @ref{3,,abc} module also provides the following decorator:

@geindex abstractmethod() (in module abc)
@anchor{library/abc abc abstractmethod}@anchor{c1e}
@deffn {Function} @@abc.abstractmethod

A decorator indicating abstract methods.

Using this decorator requires that the class’s metaclass is @ref{a8d,,ABCMeta}
or is derived from it.  A class that has a metaclass derived from
@ref{a8d,,ABCMeta} cannot be instantiated unless all of its abstract methods
and properties are overridden.  The abstract methods can be called using any
of the normal ‘super’ call mechanisms.  @ref{c1e,,abstractmethod()} may be used
to declare abstract methods for properties and descriptors.

Dynamically adding abstract methods to a class, or attempting to modify the
abstraction status of a method or class once it is created, are only
supported using the @ref{3bd3,,update_abstractmethods()} function.  The
@ref{c1e,,abstractmethod()} only affects subclasses derived using regular
inheritance; “virtual subclasses” registered with the ABC’s @code{register()}
method are not affected.

When @ref{c1e,,abstractmethod()} is applied in combination with other method
descriptors, it should be applied as the innermost decorator, as shown in
the following usage examples:

@example
class C(ABC):
    @@abstractmethod
    def my_abstract_method(self, arg1):
        ...
    @@classmethod
    @@abstractmethod
    def my_abstract_classmethod(cls, arg2):
        ...
    @@staticmethod
    @@abstractmethod
    def my_abstract_staticmethod(arg3):
        ...

    @@property
    @@abstractmethod
    def my_abstract_property(self):
        ...
    @@my_abstract_property.setter
    @@abstractmethod
    def my_abstract_property(self, val):
        ...

    @@abstractmethod
    def _get_x(self):
        ...
    @@abstractmethod
    def _set_x(self, val):
        ...
    x = property(_get_x, _set_x)
@end example

In order to correctly interoperate with the abstract base class machinery,
the descriptor must identify itself as abstract using
@code{__isabstractmethod__}. In general, this attribute should be @code{True}
if any of the methods used to compose the descriptor are abstract. For
example, Python’s built-in @ref{501,,property} does the equivalent of:

@example
class Descriptor:
    ...
    @@property
    def __isabstractmethod__(self):
        return any(getattr(f, '__isabstractmethod__', False) for
                   f in (self._fget, self._fset, self._fdel))
@end example

@cartouche
@quotation Note 
Unlike Java abstract methods, these abstract
methods may have an implementation. This implementation can be
called via the @ref{78c,,super()} mechanism from the class that
overrides it.  This could be useful as an end-point for a
super-call in a framework that uses cooperative
multiple-inheritance.
@end quotation
@end cartouche
@end deffn

The @ref{3,,abc} module also supports the following legacy decorators:

@geindex abstractclassmethod() (in module abc)
@anchor{library/abc abc abstractclassmethod}@anchor{c1f}
@deffn {Function} @@abc.abstractclassmethod


New in version 3.2.



Deprecated since version 3.3: It is now possible to use @ref{254,,classmethod} with
@ref{c1e,,abstractmethod()}, making this decorator redundant.


A subclass of the built-in @ref{254,,classmethod()}, indicating an abstract
classmethod. Otherwise it is similar to @ref{c1e,,abstractmethod()}.

This special case is deprecated, as the @ref{254,,classmethod()} decorator
is now correctly identified as abstract when applied to an abstract
method:

@example
class C(ABC):
    @@classmethod
    @@abstractmethod
    def my_abstract_classmethod(cls, arg):
        ...
@end example
@end deffn

@geindex abstractstaticmethod() (in module abc)
@anchor{library/abc abc abstractstaticmethod}@anchor{c20}
@deffn {Function} @@abc.abstractstaticmethod


New in version 3.2.



Deprecated since version 3.3: It is now possible to use @ref{253,,staticmethod} with
@ref{c1e,,abstractmethod()}, making this decorator redundant.


A subclass of the built-in @ref{253,,staticmethod()}, indicating an abstract
staticmethod. Otherwise it is similar to @ref{c1e,,abstractmethod()}.

This special case is deprecated, as the @ref{253,,staticmethod()} decorator
is now correctly identified as abstract when applied to an abstract
method:

@example
class C(ABC):
    @@staticmethod
    @@abstractmethod
    def my_abstract_staticmethod(arg):
        ...
@end example
@end deffn

@geindex abstractproperty() (in module abc)
@anchor{library/abc abc abstractproperty}@anchor{c1d}
@deffn {Function} @@abc.abstractproperty


Deprecated since version 3.3: It is now possible to use @ref{501,,property}, @code{property.getter()},
@code{property.setter()} and @code{property.deleter()} with
@ref{c1e,,abstractmethod()}, making this decorator redundant.


A subclass of the built-in @ref{501,,property()}, indicating an abstract
property.

This special case is deprecated, as the @ref{501,,property()} decorator
is now correctly identified as abstract when applied to an abstract
method:

@example
class C(ABC):
    @@property
    @@abstractmethod
    def my_abstract_property(self):
        ...
@end example

The above example defines a read-only property; you can also define a
read-write abstract property by appropriately marking one or more of the
underlying methods as abstract:

@example
class C(ABC):
    @@property
    def x(self):
        ...

    @@x.setter
    @@abstractmethod
    def x(self, val):
        ...
@end example

If only some components are abstract, only those components need to be
updated to create a concrete property in a subclass:

@example
class D(C):
    @@C.x.setter
    def x(self, val):
        ...
@end example
@end deffn

The @ref{3,,abc} module also provides the following functions:

@geindex get_cache_token() (in module abc)
@anchor{library/abc abc get_cache_token}@anchor{a8b}
@deffn {Function} abc.get_cache_token ()

Returns the current abstract base class cache token.

The token is an opaque object (that supports equality testing) identifying
the current version of the abstract base class cache for virtual subclasses.
The token changes with every call to @ref{c21,,ABCMeta.register()} on any ABC.


New in version 3.4.

@end deffn

@geindex update_abstractmethods() (in module abc)
@anchor{library/abc abc update_abstractmethods}@anchor{3bd3}
@deffn {Function} abc.update_abstractmethods (cls)

A function to recalculate an abstract class’s abstraction status. This
function should be called if a class’s abstract methods have been
implemented or changed after it was created. Usually, this function should
be called from within a class decorator.

Returns @emph{cls}, to allow usage as a class decorator.

If @emph{cls} is not an instance of @ref{a8d,,ABCMeta}, does nothing.

@cartouche
@quotation Note 
This function assumes that @emph{cls}’s superclasses are already updated.
It does not update any subclasses.
@end quotation
@end cartouche


New in version 3.10.

@end deffn

@node atexit — Exit handlers,traceback — Print or retrieve a stack traceback,abc — Abstract Base Classes,Python Runtime Services
@anchor{library/atexit doc}@anchor{3bd4}@anchor{library/atexit atexit-exit-handlers}@anchor{3bd5}@anchor{library/atexit module-atexit}@anchor{b}
@subsection @code{atexit} — Exit handlers


@geindex module; atexit


__________________________________________________________________


The @ref{b,,atexit} module defines functions to register and unregister cleanup
functions.  Functions thus registered are automatically executed upon normal
interpreter termination.  @ref{b,,atexit} runs these functions in the @emph{reverse}
order in which they were registered; if you register @code{A}, @code{B}, and @code{C},
at interpreter termination time they will be run in the order @code{C}, @code{B},
@code{A}.

@strong{Note:} The functions registered via this module are not called when the
program is killed by a signal not handled by Python, when a Python fatal
internal error is detected, or when @ref{1b81,,os._exit()} is called.


Changed in version 3.7: When used with C-API subinterpreters, registered functions
are local to the interpreter they were registered in.


@geindex register() (in module atexit)
@anchor{library/atexit atexit register}@anchor{343}
@deffn {Function} atexit.register (func, *args, **kwargs)

Register @emph{func} as a function to be executed at termination.  Any optional
arguments that are to be passed to @emph{func} must be passed as arguments to
@ref{343,,register()}.  It is possible to register the same function and arguments
more than once.

At normal program termination (for instance, if @ref{ee4,,sys.exit()} is called or
the main module’s execution completes), all functions registered are called in
last in, first out order.  The assumption is that lower level modules will
normally be imported before higher level modules and thus must be cleaned up
later.

If an exception is raised during execution of the exit handlers, a traceback is
printed (unless @ref{892,,SystemExit} is raised) and the exception information is
saved.  After all exit handlers have had a chance to run, the last exception to
be raised is re-raised.

This function returns @emph{func}, which makes it possible to use it as a
decorator.
@end deffn

@geindex unregister() (in module atexit)
@anchor{library/atexit atexit unregister}@anchor{3bd6}
@deffn {Function} atexit.unregister (func)

Remove @emph{func} from the list of functions to be run at interpreter shutdown.
@ref{3bd6,,unregister()} silently does nothing if @emph{func} was not previously
registered.  If @emph{func} has been registered more than once, every occurrence
of that function in the @ref{b,,atexit} call stack will be removed.  Equality
comparisons (@code{==}) are used internally during unregistration, so function
references do not need to have matching identities.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{d9,,readline}

Useful example of @ref{b,,atexit} to read and write @ref{d9,,readline} history
files.
@end table


@menu
* atexit Example:: 

@end menu

@node atexit Example,,,atexit — Exit handlers
@anchor{library/atexit atexit-example}@anchor{3bd7}@anchor{library/atexit id1}@anchor{3bd8}
@subsubsection @code{atexit} Example


The following simple example demonstrates how a module can initialize a counter
from a file when it is imported and save the counter’s updated value
automatically when the program terminates without relying on the application
making an explicit call into this module at termination.

@example
try:
    with open('counterfile') as infile:
        _count = int(infile.read())
except FileNotFoundError:
    _count = 0

def incrcounter(n):
    global _count
    _count = _count + n

def savecounter():
    with open('counterfile', 'w') as outfile:
        outfile.write('%d' % _count)

import atexit

atexit.register(savecounter)
@end example

Positional and keyword arguments may also be passed to @ref{343,,register()} to be
passed along to the registered function when it is called:

@example
def goodbye(name, adjective):
    print('Goodbye %s, it was %s to meet you.' % (name, adjective))

import atexit

atexit.register(goodbye, 'Donny', 'nice')
# or:
atexit.register(goodbye, adjective='nice', name='Donny')
@end example

Usage as a @ref{1bf,,decorator}:

@example
import atexit

@@atexit.register
def goodbye():
    print('You are now leaving the Python sector.')
@end example

This only works with functions that can be called without arguments.

@node traceback — Print or retrieve a stack traceback,__future__ — Future statement definitions,atexit — Exit handlers,Python Runtime Services
@anchor{library/traceback doc}@anchor{3bd9}@anchor{library/traceback module-traceback}@anchor{11a}@anchor{library/traceback traceback-print-or-retrieve-a-stack-traceback}@anchor{3bda}
@subsection @code{traceback} — Print or retrieve a stack traceback


@geindex module; traceback

@strong{Source code:} Lib/traceback.py@footnote{https://github.com/python/cpython/tree/main/Lib/traceback.py}


__________________________________________________________________


This module provides a standard interface to extract, format and print stack
traces of Python programs.  It exactly mimics the behavior of the Python
interpreter when it prints a stack trace.  This is useful when you want to print
stack traces under program control, such as in a “wrapper” around the
interpreter.

@geindex object; traceback

The module uses traceback objects — this is the object type that is stored in
the @ref{3a58,,sys.last_traceback} variable and returned as the third item from
@ref{18f,,sys.exc_info()}.

The module defines the following functions:

@geindex print_tb() (in module traceback)
@anchor{library/traceback traceback print_tb}@anchor{9f7}
@deffn {Function} traceback.print_tb (tb, limit=None, file=None)

Print up to @emph{limit} stack trace entries from traceback object @emph{tb} (starting
from the caller’s frame) if @emph{limit} is positive.  Otherwise, print the last
@code{abs(limit)} entries.  If @emph{limit} is omitted or @code{None}, all entries are
printed.  If @emph{file} is omitted or @code{None}, the output goes to
@code{sys.stderr}; otherwise it should be an open file or file-like object to
receive the output.


Changed in version 3.5: Added negative @emph{limit} support.

@end deffn

@geindex print_exception() (in module traceback)
@anchor{library/traceback traceback print_exception}@anchor{2d5}
@deffn {Function} traceback.print_exception (exc, /[, value, tb], limit=None, file=None, chain=True)

Print exception information and stack trace entries from traceback object
@emph{tb} to @emph{file}. This differs from @ref{9f7,,print_tb()} in the following
ways:


@itemize *

@item 
if @emph{tb} is not @code{None}, it prints a header @code{Traceback (most recent
call last):}

@item 
it prints the exception type and @emph{value} after the stack trace
@end itemize

@geindex ^ (caret); marker


@itemize *

@item 
if @emph{type(value)} is @ref{21a,,SyntaxError} and @emph{value} has the appropriate
format, it prints the line where the syntax error occurred with a caret
indicating the approximate position of the error.
@end itemize

Since Python 3.10, instead of passing @emph{value} and @emph{tb}, an exception object
can be passed as the first argument. If @emph{value} and @emph{tb} are provided, the
first argument is ignored in order to provide backwards compatibility.

The optional @emph{limit} argument has the same meaning as for @ref{9f7,,print_tb()}.
If @emph{chain} is true (the default), then chained exceptions (the
@code{__cause__} or @code{__context__} attributes of the exception) will be
printed as well, like the interpreter itself does when printing an unhandled
exception.


Changed in version 3.5: The @emph{etype} argument is ignored and inferred from the type of @emph{value}.



Changed in version 3.10: The @emph{etype} parameter has been renamed to @emph{exc} and is now
positional-only.

@end deffn

@geindex print_exc() (in module traceback)
@anchor{library/traceback traceback print_exc}@anchor{10e1}
@deffn {Function} traceback.print_exc (limit=None, file=None, chain=True)

This is a shorthand for @code{print_exception(*sys.exc_info(), limit, file,
chain)}.
@end deffn

@geindex print_last() (in module traceback)
@anchor{library/traceback traceback print_last}@anchor{3bdb}
@deffn {Function} traceback.print_last (limit=None, file=None, chain=True)

This is a shorthand for @code{print_exception(sys.last_type, sys.last_value,
sys.last_traceback, limit, file, chain)}.  In general it will work only
after an exception has reached an interactive prompt (see
@ref{e66,,sys.last_type}).
@end deffn

@geindex print_stack() (in module traceback)
@anchor{library/traceback traceback print_stack}@anchor{9f8}
@deffn {Function} traceback.print_stack (f=None, limit=None, file=None)

Print up to @emph{limit} stack trace entries (starting from the invocation
point) if @emph{limit} is positive.  Otherwise, print the last @code{abs(limit)}
entries.  If @emph{limit} is omitted or @code{None}, all entries are printed.
The optional @emph{f} argument can be used to specify an alternate stack frame
to start.  The optional @emph{file} argument has the same meaning as for
@ref{9f7,,print_tb()}.


Changed in version 3.5: Added negative @emph{limit} support.

@end deffn

@geindex extract_tb() (in module traceback)
@anchor{library/traceback traceback extract_tb}@anchor{3bdc}
@deffn {Function} traceback.extract_tb (tb, limit=None)

Return a @ref{9f5,,StackSummary} object representing a list of “pre-processed”
stack trace entries extracted from the traceback object @emph{tb}.  It is useful
for alternate formatting of stack traces.  The optional @emph{limit} argument has
the same meaning as for @ref{9f7,,print_tb()}.  A “pre-processed” stack trace
entry is a @ref{9f6,,FrameSummary} object containing attributes
@code{filename}, @code{lineno},
@code{name}, and @code{line} representing the
information that is usually printed for a stack trace.  The
@code{line} is a string with leading and trailing
whitespace stripped; if the source is not available it is @code{None}.
@end deffn

@geindex extract_stack() (in module traceback)
@anchor{library/traceback traceback extract_stack}@anchor{3bdd}
@deffn {Function} traceback.extract_stack (f=None, limit=None)

Extract the raw traceback from the current stack frame.  The return value has
the same format as for @ref{3bdc,,extract_tb()}.  The optional @emph{f} and @emph{limit}
arguments have the same meaning as for @ref{9f8,,print_stack()}.
@end deffn

@geindex format_list() (in module traceback)
@anchor{library/traceback traceback format_list}@anchor{3bde}
@deffn {Function} traceback.format_list (extracted_list)

Given a list of tuples or @ref{9f6,,FrameSummary} objects as returned by
@ref{3bdc,,extract_tb()} or @ref{3bdd,,extract_stack()}, return a list of strings ready
for printing.  Each string in the resulting list corresponds to the item with
the same index in the argument list.  Each string ends in a newline; the
strings may contain internal newlines as well, for those items whose source
text line is not @code{None}.
@end deffn

@geindex format_exception_only() (in module traceback)
@anchor{library/traceback traceback format_exception_only}@anchor{2d4}
@deffn {Function} traceback.format_exception_only (exc, /[, value])

Format the exception part of a traceback using an exception value such as
given by @code{sys.last_value}.  The return value is a list of strings, each
ending in a newline.  Normally, the list contains a single string; however,
for @ref{21a,,SyntaxError} exceptions, it contains several lines that (when
printed) display detailed information about where the syntax error occurred.
The message indicating which exception occurred is the always last string in
the list.

Since Python 3.10, instead of passing @emph{value}, an exception object
can be passed as the first argument.  If @emph{value} is provided, the first
argument is ignored in order to provide backwards compatibility.


Changed in version 3.10: The @emph{etype} parameter has been renamed to @emph{exc} and is now
positional-only.

@end deffn

@geindex format_exception() (in module traceback)
@anchor{library/traceback traceback format_exception}@anchor{2d3}
@deffn {Function} traceback.format_exception (exc, /[, value, tb], limit=None, chain=True)

Format a stack trace and the exception information.  The arguments  have the
same meaning as the corresponding arguments to @ref{2d5,,print_exception()}.  The
return value is a list of strings, each ending in a newline and some
containing internal newlines.  When these lines are concatenated and printed,
exactly the same text is printed as does @ref{2d5,,print_exception()}.


Changed in version 3.5: The @emph{etype} argument is ignored and inferred from the type of @emph{value}.



Changed in version 3.10: This function’s behavior and signature were modified to match
@ref{2d5,,print_exception()}.

@end deffn

@geindex format_exc() (in module traceback)
@anchor{library/traceback traceback format_exc}@anchor{3bdf}
@deffn {Function} traceback.format_exc (limit=None, chain=True)

This is like @code{print_exc(limit)} but returns a string instead of printing to
a file.
@end deffn

@geindex format_tb() (in module traceback)
@anchor{library/traceback traceback format_tb}@anchor{3b26}
@deffn {Function} traceback.format_tb (tb, limit=None)

A shorthand for @code{format_list(extract_tb(tb, limit))}.
@end deffn

@geindex format_stack() (in module traceback)
@anchor{library/traceback traceback format_stack}@anchor{3be0}
@deffn {Function} traceback.format_stack (f=None, limit=None)

A shorthand for @code{format_list(extract_stack(f, limit))}.
@end deffn

@geindex clear_frames() (in module traceback)
@anchor{library/traceback traceback clear_frames}@anchor{b59}
@deffn {Function} traceback.clear_frames (tb)

Clears the local variables of all the stack frames in a traceback @emph{tb}
by calling the @code{clear()} method of each frame object.


New in version 3.4.

@end deffn

@geindex walk_stack() (in module traceback)
@anchor{library/traceback traceback walk_stack}@anchor{9f2}
@deffn {Function} traceback.walk_stack (f)

Walk a stack following @code{f.f_back} from the given frame, yielding the frame
and line number for each frame. If @emph{f} is @code{None}, the current stack is
used. This helper is used with @ref{3be1,,StackSummary.extract()}.


New in version 3.5.

@end deffn

@geindex walk_tb() (in module traceback)
@anchor{library/traceback traceback walk_tb}@anchor{9f3}
@deffn {Function} traceback.walk_tb (tb)

Walk a traceback following @code{tb_next} yielding the frame and line number
for each frame. This helper is used with @ref{3be1,,StackSummary.extract()}.


New in version 3.5.

@end deffn

The module also defines the following classes:

@menu
* TracebackException Objects:: 
* StackSummary Objects:: 
* FrameSummary Objects:: 
* Traceback Examples:: 

@end menu

@node TracebackException Objects,StackSummary Objects,,traceback — Print or retrieve a stack traceback
@anchor{library/traceback tracebackexception-objects}@anchor{3be2}
@subsubsection @code{TracebackException} Objects



New in version 3.5.


@ref{9f4,,TracebackException} objects are created from actual exceptions to
capture data for later printing in a lightweight fashion.

@geindex TracebackException (class in traceback)
@anchor{library/traceback traceback TracebackException}@anchor{9f4}
@deffn {Class} traceback.TracebackException (exc_type, exc_value, exc_traceback, *, limit=None, lookup_lines=True, capture_locals=False, compact=False)

Capture an exception for later rendering. @emph{limit}, @emph{lookup_lines} and
@emph{capture_locals} are as for the @ref{9f5,,StackSummary} class.

If @emph{compact} is true, only data that is required by @ref{9f4,,TracebackException}’s
@code{format} method is saved in the class attributes. In particular, the
@code{__context__} field is calculated only if @code{__cause__} is @code{None} and
@code{__suppress_context__} is false.

Note that when locals are captured, they are also shown in the traceback.

@geindex __cause__ (traceback.TracebackException attribute)
@anchor{library/traceback traceback TracebackException __cause__}@anchor{3be3}
@deffn {Attribute} __cause__

A @ref{9f4,,TracebackException} of the original @code{__cause__}.
@end deffn

@geindex __context__ (traceback.TracebackException attribute)
@anchor{library/traceback traceback TracebackException __context__}@anchor{3be4}
@deffn {Attribute} __context__

A @ref{9f4,,TracebackException} of the original @code{__context__}.
@end deffn

@geindex __suppress_context__ (traceback.TracebackException attribute)
@anchor{library/traceback traceback TracebackException __suppress_context__}@anchor{3be5}
@deffn {Attribute} __suppress_context__

The @code{__suppress_context__} value from the original exception.
@end deffn

@geindex stack (traceback.TracebackException attribute)
@anchor{library/traceback traceback TracebackException stack}@anchor{3be6}
@deffn {Attribute} stack

A @ref{9f5,,StackSummary} representing the traceback.
@end deffn

@geindex exc_type (traceback.TracebackException attribute)
@anchor{library/traceback traceback TracebackException exc_type}@anchor{3be7}
@deffn {Attribute} exc_type

The class of the original traceback.
@end deffn

@geindex filename (traceback.TracebackException attribute)
@anchor{library/traceback traceback TracebackException filename}@anchor{3be8}
@deffn {Attribute} filename

For syntax errors - the file name where the error occurred.
@end deffn

@geindex lineno (traceback.TracebackException attribute)
@anchor{library/traceback traceback TracebackException lineno}@anchor{3be9}
@deffn {Attribute} lineno

For syntax errors - the line number where the error occurred.
@end deffn

@geindex text (traceback.TracebackException attribute)
@anchor{library/traceback traceback TracebackException text}@anchor{3bea}
@deffn {Attribute} text

For syntax errors - the text where the error occurred.
@end deffn

@geindex offset (traceback.TracebackException attribute)
@anchor{library/traceback traceback TracebackException offset}@anchor{3beb}
@deffn {Attribute} offset

For syntax errors - the offset into the text where the error occurred.
@end deffn

@geindex msg (traceback.TracebackException attribute)
@anchor{library/traceback traceback TracebackException msg}@anchor{3bec}
@deffn {Attribute} msg

For syntax errors - the compiler error message.
@end deffn

@geindex from_exception() (traceback.TracebackException class method)
@anchor{library/traceback traceback TracebackException from_exception}@anchor{3bed}
@deffn {Method} classmethod from_exception (exc, *, limit=None, lookup_lines=True, capture_locals=False)

Capture an exception for later rendering. @emph{limit}, @emph{lookup_lines} and
@emph{capture_locals} are as for the @ref{9f5,,StackSummary} class.

Note that when locals are captured, they are also shown in the traceback.
@end deffn

@geindex print() (traceback.TracebackException method)
@anchor{library/traceback traceback TracebackException print}@anchor{1115}
@deffn {Method} print (*, file=None, chain=True)

Print to @emph{file} (default @code{sys.stderr}) the exception information returned by
@ref{785,,format()}.


New in version 3.11.

@end deffn

@geindex format() (traceback.TracebackException method)
@anchor{library/traceback traceback TracebackException format}@anchor{3bee}
@deffn {Method} format (*, chain=True)

Format the exception.

If @emph{chain} is not @code{True}, @code{__cause__} and @code{__context__} will not
be formatted.

The return value is a generator of strings, each ending in a newline and
some containing internal newlines. @ref{2d5,,print_exception()}
is a wrapper around this method which just prints the lines to a file.

The message indicating which exception occurred is always the last
string in the output.
@end deffn

@geindex format_exception_only() (traceback.TracebackException method)
@anchor{library/traceback traceback TracebackException format_exception_only}@anchor{3bef}
@deffn {Method} format_exception_only ()

Format the exception part of the traceback.

The return value is a generator of strings, each ending in a newline.

Normally, the generator emits a single string; however, for
@ref{21a,,SyntaxError} exceptions, it emits several lines that (when
printed) display detailed information about where the syntax
error occurred.

The message indicating which exception occurred is always the last
string in the output.
@end deffn


Changed in version 3.10: Added the @emph{compact} parameter.

@end deffn

@node StackSummary Objects,FrameSummary Objects,TracebackException Objects,traceback — Print or retrieve a stack traceback
@anchor{library/traceback stacksummary-objects}@anchor{3bf0}
@subsubsection @code{StackSummary} Objects



New in version 3.5.


@ref{9f5,,StackSummary} objects represent a call stack ready for formatting.

@geindex StackSummary (class in traceback)
@anchor{library/traceback traceback StackSummary}@anchor{9f5}
@deffn {Class} traceback.StackSummary

@geindex extract() (traceback.StackSummary class method)
@anchor{library/traceback traceback StackSummary extract}@anchor{3be1}
@deffn {Method} classmethod extract (frame_gen, *, limit=None, lookup_lines=True, capture_locals=False)

Construct a @ref{9f5,,StackSummary} object from a frame generator (such as
is returned by @ref{9f2,,walk_stack()} or
@ref{9f3,,walk_tb()}).

If @emph{limit} is supplied, only this many frames are taken from @emph{frame_gen}.
If @emph{lookup_lines} is @code{False}, the returned @ref{9f6,,FrameSummary}
objects will not have read their lines in yet, making the cost of
creating the @ref{9f5,,StackSummary} cheaper (which may be valuable if it
may not actually get formatted). If @emph{capture_locals} is @code{True} the
local variables in each @ref{9f6,,FrameSummary} are captured as object
representations.
@end deffn

@geindex from_list() (traceback.StackSummary class method)
@anchor{library/traceback traceback StackSummary from_list}@anchor{3bf1}
@deffn {Method} classmethod from_list (a_list)

Construct a @ref{9f5,,StackSummary} object from a supplied list of
@ref{9f6,,FrameSummary} objects or old-style list of tuples.  Each tuple
should be a 4-tuple with filename, lineno, name, line as the elements.
@end deffn

@geindex format() (traceback.StackSummary method)
@anchor{library/traceback traceback StackSummary format}@anchor{3bf2}
@deffn {Method} format ()

Returns a list of strings ready for printing.  Each string in the
resulting list corresponds to a single frame from the stack.
Each string ends in a newline; the strings may contain internal
newlines as well, for those items with source text lines.

For long sequences of the same frame and line, the first few
repetitions are shown, followed by a summary line stating the exact
number of further repetitions.


Changed in version 3.6: Long sequences of repeated frames are now abbreviated.

@end deffn

@geindex format_frame_summary() (traceback.StackSummary method)
@anchor{library/traceback traceback StackSummary format_frame_summary}@anchor{10ff}
@deffn {Method} format_frame_summary (frame_summary)

Returns a string for printing one of the frames involved in the stack.
This method is called for each @ref{9f6,,FrameSummary} object to be
printed by @ref{3bf2,,StackSummary.format()}. If it returns @code{None}, the
frame is omitted from the output.


New in version 3.11.

@end deffn
@end deffn

@node FrameSummary Objects,Traceback Examples,StackSummary Objects,traceback — Print or retrieve a stack traceback
@anchor{library/traceback framesummary-objects}@anchor{3bf3}
@subsubsection @code{FrameSummary} Objects



New in version 3.5.


A @ref{9f6,,FrameSummary} object represents a single frame in a traceback.

@geindex FrameSummary (class in traceback)
@anchor{library/traceback traceback FrameSummary}@anchor{9f6}
@deffn {Class} traceback.FrameSummary (filename, lineno, name, lookup_line=True, locals=None, line=None)

Represent a single frame in the traceback or stack that is being formatted
or printed. It may optionally have a stringified version of the frames
locals included in it. If @emph{lookup_line} is @code{False}, the source code is not
looked up until the @ref{9f6,,FrameSummary} has the @code{line}
attribute accessed (which also happens when casting it to a tuple).
@code{line} may be directly provided, and will prevent line
lookups happening at all. @emph{locals} is an optional local variable
dictionary, and if supplied the variable representations are stored in the
summary for later display.
@end deffn

@node Traceback Examples,,FrameSummary Objects,traceback — Print or retrieve a stack traceback
@anchor{library/traceback traceback-example}@anchor{3bf4}@anchor{library/traceback traceback-examples}@anchor{3bf5}
@subsubsection Traceback Examples


This simple example implements a basic read-eval-print loop, similar to (but
less useful than) the standard Python interactive interpreter loop.  For a more
complete implementation of the interpreter loop, refer to the @ref{19,,code}
module.

@example
import sys, traceback

def run_user_code(envdir):
    source = input(">>> ")
    try:
        exec(source, envdir)
    except Exception:
        print("Exception in user code:")
        print("-"*60)
        traceback.print_exc(file=sys.stdout)
        print("-"*60)

envdir = @{@}
while True:
    run_user_code(envdir)
@end example

The following example demonstrates the different ways to print and format the
exception and traceback:

@example
import sys, traceback

def lumberjack():
    bright_side_of_death()

def bright_side_of_death():
    return tuple()[0]

try:
    lumberjack()
except IndexError:
    exc_type, exc_value, exc_traceback = sys.exc_info()
    print("*** print_tb:")
    traceback.print_tb(exc_traceback, limit=1, file=sys.stdout)
    print("*** print_exception:")
    # exc_type below is ignored on 3.5 and later
    traceback.print_exception(exc_type, exc_value, exc_traceback,
                              limit=2, file=sys.stdout)
    print("*** print_exc:")
    traceback.print_exc(limit=2, file=sys.stdout)
    print("*** format_exc, first and last line:")
    formatted_lines = traceback.format_exc().splitlines()
    print(formatted_lines[0])
    print(formatted_lines[-1])
    print("*** format_exception:")
    # exc_type below is ignored on 3.5 and later
    print(repr(traceback.format_exception(exc_type, exc_value,
                                          exc_traceback)))
    print("*** extract_tb:")
    print(repr(traceback.extract_tb(exc_traceback)))
    print("*** format_tb:")
    print(repr(traceback.format_tb(exc_traceback)))
    print("*** tb_lineno:", exc_traceback.tb_lineno)
@end example

The output for the example would look similar to this:

@example
*** print_tb:
  File "<doctest...>", line 10, in <module>
    lumberjack()
    ^^^^^^^^^^^^
*** print_exception:
Traceback (most recent call last):
  File "<doctest...>", line 10, in <module>
    lumberjack()
    ^^^^^^^^^^^^
  File "<doctest...>", line 4, in lumberjack
    bright_side_of_death()
    ^^^^^^^^^^^^^^^^^^^^^^
IndexError: tuple index out of range
*** print_exc:
Traceback (most recent call last):
  File "<doctest...>", line 10, in <module>
    lumberjack()
    ^^^^^^^^^^^^
  File "<doctest...>", line 4, in lumberjack
    bright_side_of_death()
    ^^^^^^^^^^^^^^^^^^^^^^
IndexError: tuple index out of range
*** format_exc, first and last line:
Traceback (most recent call last):
IndexError: tuple index out of range
*** format_exception:
['Traceback (most recent call last):\n',
 '  File "<doctest default[0]>", line 10, in <module>\n    lumberjack()\n    ^^^^^^^^^^^^\n',
 '  File "<doctest default[0]>", line 4, in lumberjack\n    bright_side_of_death()\n    ^^^^^^^^^^^^^^^^^^^^^^\n',
 '  File "<doctest default[0]>", line 7, in bright_side_of_death\n    return tuple()[0]\n           ~~~~~~~^^^\n',
 'IndexError: tuple index out of range\n']
*** extract_tb:
[<FrameSummary file <doctest...>, line 10 in <module>>,
 <FrameSummary file <doctest...>, line 4 in lumberjack>,
 <FrameSummary file <doctest...>, line 7 in bright_side_of_death>]
*** format_tb:
['  File "<doctest default[0]>", line 10, in <module>\n    lumberjack()\n    ^^^^^^^^^^^^\n',
 '  File "<doctest default[0]>", line 4, in lumberjack\n    bright_side_of_death()\n    ^^^^^^^^^^^^^^^^^^^^^^\n',
 '  File "<doctest default[0]>", line 7, in bright_side_of_death\n    return tuple()[0]\n           ~~~~~~~^^^\n']
*** tb_lineno: 10
@end example

The following example shows the different ways to print and format the stack:

@example
>>> import traceback
>>> def another_function():
...     lumberstack()
...
>>> def lumberstack():
...     traceback.print_stack()
...     print(repr(traceback.extract_stack()))
...     print(repr(traceback.format_stack()))
...
>>> another_function()
  File "<doctest>", line 10, in <module>
    another_function()
  File "<doctest>", line 3, in another_function
    lumberstack()
  File "<doctest>", line 6, in lumberstack
    traceback.print_stack()
[('<doctest>', 10, '<module>', 'another_function()'),
 ('<doctest>', 3, 'another_function', 'lumberstack()'),
 ('<doctest>', 7, 'lumberstack', 'print(repr(traceback.extract_stack()))')]
['  File "<doctest>", line 10, in <module>\n    another_function()\n',
 '  File "<doctest>", line 3, in another_function\n    lumberstack()\n',
 '  File "<doctest>", line 8, in lumberstack\n    print(repr(traceback.format_stack()))\n']
@end example

This last example demonstrates the final few formatting functions:

@example
>>> import traceback
>>> traceback.format_list([('spam.py', 3, '<module>', 'spam.eggs()'),
...                        ('eggs.py', 42, 'eggs', 'return "bacon"')])
['  File "spam.py", line 3, in <module>\n    spam.eggs()\n',
 '  File "eggs.py", line 42, in eggs\n    return "bacon"\n']
>>> an_error = IndexError('tuple index out of range')
>>> traceback.format_exception_only(type(an_error), an_error)
['IndexError: tuple index out of range\n']
@end example

@node __future__ — Future statement definitions,gc — Garbage Collector interface,traceback — Print or retrieve a stack traceback,Python Runtime Services
@anchor{library/__future__ doc}@anchor{3bf6}@anchor{library/__future__ future-future-statement-definitions}@anchor{3bf7}@anchor{library/__future__ module-__future__}@anchor{0}
@subsection @code{__future__} — Future statement definitions


@geindex module; __future__

@strong{Source code:} Lib/__future__.py@footnote{https://github.com/python/cpython/tree/main/Lib/__future__.py}


__________________________________________________________________


@ref{0,,__future__} is a real module, and serves three purposes:


@itemize *

@item 
To avoid confusing existing tools that analyze import statements and expect to
find the modules they’re importing.

@item 
To ensure that @ref{1969,,future statements} run under releases prior to
2.1 at least yield runtime exceptions (the import of @ref{0,,__future__} will
fail, because there was no module of that name prior to 2.1).

@item 
To document when incompatible changes were introduced, and when they will be
— or were — made mandatory.  This is a form of executable documentation, and
can be inspected programmatically via importing @ref{0,,__future__} and examining
its contents.
@end itemize

Each statement in @code{__future__.py} is of the form:

@example
FeatureName = _Feature(OptionalRelease, MandatoryRelease,
                       CompilerFlag)
@end example

where, normally, @emph{OptionalRelease} is less than @emph{MandatoryRelease}, and both are
5-tuples of the same form as @ref{d45,,sys.version_info}:

@example
(PY_MAJOR_VERSION, # the 2 in 2.1.0a3; an int
 PY_MINOR_VERSION, # the 1; an int
 PY_MICRO_VERSION, # the 0; an int
 PY_RELEASE_LEVEL, # "alpha", "beta", "candidate" or "final"; string
 PY_RELEASE_SERIAL # the 3; an int
)
@end example

@emph{OptionalRelease} records the first release in which the feature was accepted.

In the case of a @emph{MandatoryRelease} that has not yet occurred,
@emph{MandatoryRelease} predicts the release in which the feature will become part of
the language.

Else @emph{MandatoryRelease} records when the feature became part of the language; in
releases at or after that, modules no longer need a future statement to use the
feature in question, but may continue to use such imports.

@emph{MandatoryRelease} may also be @code{None}, meaning that a planned feature got
dropped.

Instances of class @code{_Feature} have two corresponding methods,
@code{getOptionalRelease()} and @code{getMandatoryRelease()}.

@emph{CompilerFlag} is the (bitfield) flag that should be passed in the fourth
argument to the built-in function @ref{4e2,,compile()} to enable the feature in
dynamically compiled code.  This flag is stored in the @code{compiler_flag}
attribute on @code{_Feature} instances.

No feature description will ever be deleted from @ref{0,,__future__}. Since its
introduction in Python 2.1 the following features have found their way into the
language using this mechanism:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

feature

@tab

optional in

@tab

mandatory in

@tab

effect

@item

nested_scopes

@tab

2.1.0b1

@tab

2.2

@tab

@geindex Python Enhancement Proposals; PEP 227
PEP 227@footnote{https://www.python.org/dev/peps/pep-0227}:
@emph{Statically Nested Scopes}

@item

generators

@tab

2.2.0a1

@tab

2.3

@tab

@geindex Python Enhancement Proposals; PEP 255
PEP 255@footnote{https://www.python.org/dev/peps/pep-0255}:
@emph{Simple Generators}

@item

division

@tab

2.2.0a2

@tab

3.0

@tab

@geindex Python Enhancement Proposals; PEP 238
PEP 238@footnote{https://www.python.org/dev/peps/pep-0238}:
@emph{Changing the Division Operator}

@item

absolute_import

@tab

2.5.0a1

@tab

3.0

@tab

@geindex Python Enhancement Proposals; PEP 328
PEP 328@footnote{https://www.python.org/dev/peps/pep-0328}:
@emph{Imports: Multi-Line and Absolute/Relative}

@item

with_statement

@tab

2.5.0a1

@tab

2.6

@tab

@geindex Python Enhancement Proposals; PEP 343
PEP 343@footnote{https://www.python.org/dev/peps/pep-0343}:
@emph{The “with” Statement}

@item

print_function

@tab

2.6.0a2

@tab

3.0

@tab

@geindex Python Enhancement Proposals; PEP 3105
PEP 3105@footnote{https://www.python.org/dev/peps/pep-3105}:
@emph{Make print a function}

@item

unicode_literals

@tab

2.6.0a2

@tab

3.0

@tab

@geindex Python Enhancement Proposals; PEP 3112
PEP 3112@footnote{https://www.python.org/dev/peps/pep-3112}:
@emph{Bytes literals in Python 3000}

@item

generator_stop

@tab

3.5.0b1

@tab

3.7

@tab

@geindex Python Enhancement Proposals; PEP 479
PEP 479@footnote{https://www.python.org/dev/peps/pep-0479}:
@emph{StopIteration handling inside generators}

@item

annotations

@tab

3.7.0b1

@tab

3.11

@tab

@geindex Python Enhancement Proposals; PEP 563
PEP 563@footnote{https://www.python.org/dev/peps/pep-0563}:
@emph{Postponed evaluation of annotations}

@end multitable


@c XXX Adding a new entry?  Remember to update simple_stmts.rst, too.


@subsubheading See also



@table @asis

@item @ref{1969,,Future statements}

How the compiler treats future imports.
@end table


@node gc — Garbage Collector interface,inspect — Inspect live objects,__future__ — Future statement definitions,Python Runtime Services
@anchor{library/gc doc}@anchor{3bf8}@anchor{library/gc gc-garbage-collector-interface}@anchor{3bf9}@anchor{library/gc module-gc}@anchor{80}
@subsection @code{gc} — Garbage Collector interface


@geindex module; gc


__________________________________________________________________


This module provides an interface to the optional garbage collector.  It
provides the ability to disable the collector, tune the collection frequency,
and set debugging options.  It also provides access to unreachable objects that
the collector found but cannot free.  Since the collector supplements the
reference counting already used in Python, you can disable the collector if you
are sure your program does not create reference cycles.  Automatic collection
can be disabled by calling @code{gc.disable()}.  To debug a leaking program call
@code{gc.set_debug(gc.DEBUG_LEAK)}. Notice that this includes
@code{gc.DEBUG_SAVEALL}, causing garbage-collected objects to be saved in
gc.garbage for inspection.

The @ref{80,,gc} module provides the following functions:

@geindex enable() (in module gc)
@anchor{library/gc gc enable}@anchor{3bfa}
@deffn {Function} gc.enable ()

Enable automatic garbage collection.
@end deffn

@geindex disable() (in module gc)
@anchor{library/gc gc disable}@anchor{3bfb}
@deffn {Function} gc.disable ()

Disable automatic garbage collection.
@end deffn

@geindex isenabled() (in module gc)
@anchor{library/gc gc isenabled}@anchor{3bfc}
@deffn {Function} gc.isenabled ()

Return @code{True} if automatic collection is enabled.
@end deffn

@geindex collect() (in module gc)
@anchor{library/gc gc collect}@anchor{54b}
@deffn {Function} gc.collect (generation=2)

With no arguments, run a full collection.  The optional argument @emph{generation}
may be an integer specifying which generation to collect (from 0 to 2).  A
@ref{17c,,ValueError} is raised if the generation number  is invalid. The number of
unreachable objects found is returned.

The free lists maintained for a number of built-in types are cleared
whenever a full collection or collection of the highest generation (2)
is run.  Not all items in some free lists may be freed due to the
particular implementation, in particular @ref{256,,float}.
@end deffn

@geindex set_debug() (in module gc)
@anchor{library/gc gc set_debug}@anchor{3bfd}
@deffn {Function} gc.set_debug (flags)

Set the garbage collection debugging flags. Debugging information will be
written to @code{sys.stderr}.  See below for a list of debugging flags which can be
combined using bit operations to control debugging.
@end deffn

@geindex get_debug() (in module gc)
@anchor{library/gc gc get_debug}@anchor{3bfe}
@deffn {Function} gc.get_debug ()

Return the debugging flags currently set.
@end deffn

@geindex get_objects() (in module gc)
@anchor{library/gc gc get_objects}@anchor{284}
@deffn {Function} gc.get_objects (generation=None)

Returns a list of all objects tracked by the collector, excluding the list
returned. If @emph{generation} is not None, return only the objects tracked by
the collector that are in that generation.


Changed in version 3.8: New @emph{generation} parameter.


Raises an @ref{1130,,auditing event} @code{gc.get_objects} with argument @code{generation}.
@end deffn

@geindex get_stats() (in module gc)
@anchor{library/gc gc get_stats}@anchor{ac1}
@deffn {Function} gc.get_stats ()

Return a list of three per-generation dictionaries containing collection
statistics since interpreter start.  The number of keys may change
in the future, but currently each dictionary will contain the following
items:


@itemize *

@item 
@code{collections} is the number of times this generation was collected;

@item 
@code{collected} is the total number of objects collected inside this
generation;

@item 
@code{uncollectable} is the total number of objects which were found
to be uncollectable (and were therefore moved to the @ref{d66,,garbage}
list) inside this generation.
@end itemize


New in version 3.4.

@end deffn

@geindex set_threshold() (in module gc)
@anchor{library/gc gc set_threshold}@anchor{3bff}
@deffn {Function} gc.set_threshold (threshold0[, threshold1[, threshold2]])

Set the garbage collection thresholds (the collection frequency). Setting
@emph{threshold0} to zero disables collection.

The GC classifies objects into three generations depending on how many
collection sweeps they have survived.  New objects are placed in the youngest
generation (generation @code{0}).  If an object survives a collection it is moved
into the next older generation.  Since generation @code{2} is the oldest
generation, objects in that generation remain there after a collection.  In
order to decide when to run, the collector keeps track of the number object
allocations and deallocations since the last collection.  When the number of
allocations minus the number of deallocations exceeds @emph{threshold0}, collection
starts.  Initially only generation @code{0} is examined.  If generation @code{0} has
been examined more than @emph{threshold1} times since generation @code{1} has been
examined, then generation @code{1} is examined as well.
With the third generation, things are a bit more complicated,
see Collecting the oldest generation@footnote{https://devguide.python.org/garbage_collector/#collecting-the-oldest-generation} for more information.
@end deffn

@geindex get_count() (in module gc)
@anchor{library/gc gc get_count}@anchor{3c00}
@deffn {Function} gc.get_count ()

Return the current collection  counts as a tuple of @code{(count0, count1,
count2)}.
@end deffn

@geindex get_threshold() (in module gc)
@anchor{library/gc gc get_threshold}@anchor{3c01}
@deffn {Function} gc.get_threshold ()

Return the current collection thresholds as a tuple of @code{(threshold0,
threshold1, threshold2)}.
@end deffn

@geindex get_referrers() (in module gc)
@anchor{library/gc gc get_referrers}@anchor{285}
@deffn {Function} gc.get_referrers (*objs)

Return the list of objects that directly refer to any of objs. This function
will only locate those containers which support garbage collection; extension
types which do refer to other objects but do not support garbage collection will
not be found.

Note that objects which have already been dereferenced, but which live in cycles
and have not yet been collected by the garbage collector can be listed among the
resulting referrers.  To get only currently live objects, call @ref{54b,,collect()}
before calling @ref{285,,get_referrers()}.

@cartouche
@quotation Warning 
Care must be taken when using objects returned by @ref{285,,get_referrers()} because
some of them could still be under construction and hence in a temporarily
invalid state. Avoid using @ref{285,,get_referrers()} for any purpose other than
debugging.
@end quotation
@end cartouche

Raises an @ref{1130,,auditing event} @code{gc.get_referrers} with argument @code{objs}.
@end deffn

@geindex get_referents() (in module gc)
@anchor{library/gc gc get_referents}@anchor{286}
@deffn {Function} gc.get_referents (*objs)

Return a list of objects directly referred to by any of the arguments. The
referents returned are those objects visited by the arguments’ C-level
@ref{1ee,,tp_traverse} methods (if any), and may not be all objects actually
directly reachable.  @ref{1ee,,tp_traverse} methods are supported only by objects
that support garbage collection, and are only required to visit objects that may
be involved in a cycle.  So, for example, if an integer is directly reachable
from an argument, that integer object may or may not appear in the result list.

Raises an @ref{1130,,auditing event} @code{gc.get_referents} with argument @code{objs}.
@end deffn

@geindex is_tracked() (in module gc)
@anchor{library/gc gc is_tracked}@anchor{ea5}
@deffn {Function} gc.is_tracked (obj)

Returns @code{True} if the object is currently tracked by the garbage collector,
@code{False} otherwise.  As a general rule, instances of atomic types aren’t
tracked and instances of non-atomic types (containers, user-defined
objects…) are.  However, some type-specific optimizations can be present
in order to suppress the garbage collector footprint of simple instances
(e.g. dicts containing only atomic keys and values):

@example
>>> gc.is_tracked(0)
False
>>> gc.is_tracked("a")
False
>>> gc.is_tracked([])
True
>>> gc.is_tracked(@{@})
False
>>> gc.is_tracked(@{"a": 1@})
False
>>> gc.is_tracked(@{"a": []@})
True
@end example


New in version 3.1.

@end deffn

@geindex is_finalized() (in module gc)
@anchor{library/gc gc is_finalized}@anchor{3ec}
@deffn {Function} gc.is_finalized (obj)

Returns @code{True} if the given object has been finalized by the
garbage collector, @code{False} otherwise.

@example
>>> x = None
>>> class Lazarus:
...     def __del__(self):
...         global x
...         x = self
...
>>> lazarus = Lazarus()
>>> gc.is_finalized(lazarus)
False
>>> del lazarus
>>> gc.is_finalized(x)
True
@end example


New in version 3.9.

@end deffn

@geindex freeze() (in module gc)
@anchor{library/gc gc freeze}@anchor{65d}
@deffn {Function} gc.freeze ()

Freeze all the objects tracked by gc - move them to a permanent generation
and ignore all the future collections. This can be used before a POSIX
fork() call to make the gc copy-on-write friendly or to speed up collection.
Also collection before a POSIX fork() call may free pages for future
allocation which can cause copy-on-write too so it’s advised to disable gc
in parent process and freeze before fork and enable gc in child process.


New in version 3.7.

@end deffn

@geindex unfreeze() (in module gc)
@anchor{library/gc gc unfreeze}@anchor{65e}
@deffn {Function} gc.unfreeze ()

Unfreeze the objects in the permanent generation, put them back into the
oldest generation.


New in version 3.7.

@end deffn

@geindex get_freeze_count() (in module gc)
@anchor{library/gc gc get_freeze_count}@anchor{65f}
@deffn {Function} gc.get_freeze_count ()

Return the number of objects in the permanent generation.


New in version 3.7.

@end deffn

The following variables are provided for read-only access (you can mutate the
values but should not rebind them):

@geindex garbage (in module gc)
@anchor{library/gc gc garbage}@anchor{d66}
@deffn {Data} gc.garbage

A list of objects which the collector found to be unreachable but could
not be freed (uncollectable objects).  Starting with Python 3.4, this
list should be empty most of the time, except when using instances of
C extension types with a non-@code{NULL} @code{tp_del} slot.

If @ref{3c02,,DEBUG_SAVEALL} is set, then all unreachable objects will be
added to this list rather than freed.


Changed in version 3.2: If this list is non-empty at @ref{9e2,,interpreter shutdown}, a
@ref{77b,,ResourceWarning} is emitted, which is silent by default.  If
@ref{d67,,DEBUG_UNCOLLECTABLE} is set, in addition all uncollectable objects
are printed.



Changed in version 3.4: Following 
@geindex Python Enhancement Proposals; PEP 442
PEP 442@footnote{https://www.python.org/dev/peps/pep-0442}, objects with a @code{__del__()} method don’t end
up in @ref{d66,,gc.garbage} anymore.

@end deffn

@geindex callbacks (in module gc)
@anchor{library/gc gc callbacks}@anchor{c59}
@deffn {Data} gc.callbacks

A list of callbacks that will be invoked by the garbage collector before and
after collection.  The callbacks will be called with two arguments,
@emph{phase} and @emph{info}.

@emph{phase} can be one of two values:

@quotation

“start”: The garbage collection is about to start.

“stop”: The garbage collection has finished.
@end quotation

@emph{info} is a dict providing more information for the callback.  The following
keys are currently defined:

@quotation

“generation”: The oldest generation being collected.

“collected”: When @emph{phase} is “stop”, the number of objects
successfully collected.

“uncollectable”: When @emph{phase} is “stop”, the number of objects
that could not be collected and were put in @ref{d66,,garbage}.
@end quotation

Applications can add their own callbacks to this list.  The primary
use cases are:

@quotation

Gathering statistics about garbage collection, such as how often
various generations are collected, and how long the collection
takes.

Allowing applications to identify and clear their own uncollectable
types when they appear in @ref{d66,,garbage}.
@end quotation


New in version 3.3.

@end deffn

The following constants are provided for use with @ref{3bfd,,set_debug()}:

@geindex DEBUG_STATS (in module gc)
@anchor{library/gc gc DEBUG_STATS}@anchor{3c03}
@deffn {Data} gc.DEBUG_STATS

Print statistics during collection.  This information can be useful when tuning
the collection frequency.
@end deffn

@geindex DEBUG_COLLECTABLE (in module gc)
@anchor{library/gc gc DEBUG_COLLECTABLE}@anchor{3c04}
@deffn {Data} gc.DEBUG_COLLECTABLE

Print information on collectable objects found.
@end deffn

@geindex DEBUG_UNCOLLECTABLE (in module gc)
@anchor{library/gc gc DEBUG_UNCOLLECTABLE}@anchor{d67}
@deffn {Data} gc.DEBUG_UNCOLLECTABLE

Print information of uncollectable objects found (objects which are not
reachable but cannot be freed by the collector).  These objects will be added
to the @code{garbage} list.


Changed in version 3.2: Also print the contents of the @ref{d66,,garbage} list at
@ref{9e2,,interpreter shutdown}, if it isn’t empty.

@end deffn

@geindex DEBUG_SAVEALL (in module gc)
@anchor{library/gc gc DEBUG_SAVEALL}@anchor{3c02}
@deffn {Data} gc.DEBUG_SAVEALL

When set, all unreachable objects found will be appended to @emph{garbage} rather
than being freed.  This can be useful for debugging a leaking program.
@end deffn

@geindex DEBUG_LEAK (in module gc)
@anchor{library/gc gc DEBUG_LEAK}@anchor{3c05}
@deffn {Data} gc.DEBUG_LEAK

The debugging flags necessary for the collector to print information about a
leaking program (equal to @code{DEBUG_COLLECTABLE | DEBUG_UNCOLLECTABLE |
DEBUG_SAVEALL}).
@end deffn

@node inspect — Inspect live objects,site — Site-specific configuration hook,gc — Garbage Collector interface,Python Runtime Services
@anchor{library/inspect doc}@anchor{3c06}@anchor{library/inspect inspect-inspect-live-objects}@anchor{3c07}@anchor{library/inspect module-inspect}@anchor{9c}
@subsection @code{inspect} — Inspect live objects


@geindex module; inspect

@strong{Source code:} Lib/inspect.py@footnote{https://github.com/python/cpython/tree/main/Lib/inspect.py}


__________________________________________________________________


The @ref{9c,,inspect} module provides several useful functions to help get
information about live objects such as modules, classes, methods, functions,
tracebacks, frame objects, and code objects.  For example, it can help you
examine the contents of a class, retrieve the source code of a method, extract
and format the argument list for a function, or get all the information you need
to display a detailed traceback.

There are four main kinds of services provided by this module: type checking,
getting source code, inspecting classes and functions, and examining the
interpreter stack.

@menu
* Types and members:: 
* Retrieving source code:: 
* Introspecting callables with the Signature object:: 
* Classes and functions: Classes and functions<2>. 
* The interpreter stack:: 
* Fetching attributes statically:: 
* Current State of Generators and Coroutines:: 
* Code Objects Bit Flags:: 
* Command Line Interface: Command Line Interface<3>. 

@end menu

@node Types and members,Retrieving source code,,inspect — Inspect live objects
@anchor{library/inspect inspect-types}@anchor{3c08}@anchor{library/inspect types-and-members}@anchor{3c09}
@subsubsection Types and members


The @ref{10ab,,getmembers()} function retrieves the members of an object such as a
class or module. The functions whose names begin with “is” are mainly
provided as convenient choices for the second argument to @ref{10ab,,getmembers()}.
They also help you determine when you can expect to find the following special
attributes:

@c this function name is too big to fit in the ascii-art table below


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Type

@tab

Attribute

@tab

Description

@item

module

@tab

__doc__

@tab

documentation string

@item

@tab

__file__

@tab

filename (missing for
built-in modules)

@item

class

@tab

__doc__

@tab

documentation string

@item

@tab

__name__

@tab

name with which this
class was defined

@item

@tab

__qualname__

@tab

qualified name

@item

@tab

__module__

@tab

name of module in which
this class was defined

@item

method

@tab

__doc__

@tab

documentation string

@item

@tab

__name__

@tab

name with which this
method was defined

@item

@tab

__qualname__

@tab

qualified name

@item

@tab

__func__

@tab

function object
containing implementation
of method

@item

@tab

__self__

@tab

instance to which this
method is bound, or
@code{None}

@item

@tab

__module__

@tab

name of module in which
this method was defined

@item

function

@tab

__doc__

@tab

documentation string

@item

@tab

__name__

@tab

name with which this
function was defined

@item

@tab

__qualname__

@tab

qualified name

@item

@tab

__code__

@tab

code object containing
compiled function
@ref{8a8,,bytecode}

@item

@tab

__defaults__

@tab

tuple of any default
values for positional or
keyword parameters

@item

@tab

__kwdefaults__

@tab

mapping of any default
values for keyword-only
parameters

@item

@tab

__globals__

@tab

global namespace in which
this function was defined

@item

@tab

__builtins__

@tab

builtins namespace

@item

@tab

__annotations__

@tab

mapping of parameters
names to annotations;
@code{"return"} key is
reserved for return
annotations.

@item

@tab

__module__

@tab

name of module in which
this function was defined

@item

traceback

@tab

tb_frame

@tab

frame object at this
level

@item

@tab

tb_lasti

@tab

index of last attempted
instruction in bytecode

@item

@tab

tb_lineno

@tab

current line number in
Python source code

@item

@tab

tb_next

@tab

next inner traceback
object (called by this
level)

@item

frame

@tab

f_back

@tab

next outer frame object
(this frame’s caller)

@item

@tab

f_builtins

@tab

builtins namespace seen
by this frame

@item

@tab

f_code

@tab

code object being
executed in this frame

@item

@tab

f_globals

@tab

global namespace seen by
this frame

@item

@tab

f_lasti

@tab

index of last attempted
instruction in bytecode

@item

@tab

f_lineno

@tab

current line number in
Python source code

@item

@tab

f_locals

@tab

local namespace seen by
this frame

@item

@tab

f_trace

@tab

tracing function for this
frame, or @code{None}

@item

code

@tab

co_argcount

@tab

number of arguments (not
including keyword only
arguments, * or **
args)

@item

@tab

co_code

@tab

string of raw compiled
bytecode

@item

@tab

co_cellvars

@tab

tuple of names of cell
variables (referenced by
containing scopes)

@item

@tab

co_consts

@tab

tuple of constants used
in the bytecode

@item

@tab

co_filename

@tab

name of file in which
this code object was
created

@item

@tab

co_firstlineno

@tab

number of first line in
Python source code

@item

@tab

co_flags

@tab

bitmap of @code{CO_*} flags,
read more @ref{3c0a,,here}

@item

@tab

co_lnotab

@tab

encoded mapping of line
numbers to bytecode
indices

@item

@tab

co_freevars

@tab

tuple of names of free
variables (referenced via
a function’s closure)

@item

@tab

co_posonlyargcount

@tab

number of positional only
arguments

@item

@tab

co_kwonlyargcount

@tab

number of keyword only
arguments (not including
** arg)

@item

@tab

co_name

@tab

name with which this code
object was defined

@item

@tab

co_qualname

@tab

fully-qualified name with
which this code object
was defined

@item

@tab

co_names

@tab

tuple of names other
than arguments and
function locals

@item

@tab

co_nlocals

@tab

number of local variables

@item

@tab

co_stacksize

@tab

virtual machine stack
space required

@item

@tab

co_varnames

@tab

tuple of names of
arguments and local
variables

@item

generator

@tab

__name__

@tab

name

@item

@tab

__qualname__

@tab

qualified name

@item

@tab

gi_frame

@tab

frame

@item

@tab

gi_running

@tab

is the generator running?

@item

@tab

gi_code

@tab

code

@item

@tab

gi_yieldfrom

@tab

object being iterated by
@code{yield from}, or
@code{None}

@item

coroutine

@tab

__name__

@tab

name

@item

@tab

__qualname__

@tab

qualified name

@item

@tab

cr_await

@tab

object being awaited on,
or @code{None}

@item

@tab

cr_frame

@tab

frame

@item

@tab

cr_running

@tab

is the coroutine running?

@item

@tab

cr_code

@tab

code

@item

@tab

cr_origin

@tab

where coroutine was
created, or @code{None}. See
@ref{6bd,,sys.set_coroutine_origin_tracking_depth()}

@item

builtin

@tab

__doc__

@tab

documentation string

@item

@tab

__name__

@tab

original name of this
function or method

@item

@tab

__qualname__

@tab

qualified name

@item

@tab

__self__

@tab

instance to which a
method is bound, or
@code{None}

@end multitable



Changed in version 3.5: Add @code{__qualname__} and @code{gi_yieldfrom} attributes to generators.

The @code{__name__} attribute of generators is now set from the function
name, instead of the code name, and it can now be modified.



Changed in version 3.7: Add @code{cr_origin} attribute to coroutines.



Changed in version 3.10: Add @code{__builtins__} attribute to functions.


@geindex getmembers() (in module inspect)
@anchor{library/inspect inspect getmembers}@anchor{10ab}
@deffn {Function} inspect.getmembers (object[, predicate])

Return all the members of an object in a list of @code{(name, value)}
pairs sorted by name. If the optional @emph{predicate} argument—which will be
called with the @code{value} object of each member—is supplied, only members
for which the predicate returns a true value are included.

@cartouche
@quotation Note 
@ref{10ab,,getmembers()} will only return class attributes defined in the
metaclass when the argument is a class and those attributes have been
listed in the metaclass’ custom @code{__dir__()}.
@end quotation
@end cartouche
@end deffn

@geindex getmembers_static() (in module inspect)
@anchor{library/inspect inspect getmembers_static}@anchor{175}
@deffn {Function} inspect.getmembers_static (object[, predicate])

Return all the members of an object in a list of @code{(name, value)}
pairs sorted by name without triggering dynamic lookup via the descriptor
protocol, __getattr__ or __getattribute__. Optionally, only return members
that satisfy a given predicate.

@cartouche
@quotation Note 
@ref{175,,getmembers_static()} may not be able to retrieve all members
that getmembers can fetch (like dynamically created attributes)
and may find members that getmembers can’t (like descriptors
that raise AttributeError). It can also return descriptor objects
instead of instance members in some cases.
@end quotation
@end cartouche


New in version 3.11.

@end deffn

@geindex getmodulename() (in module inspect)
@anchor{library/inspect inspect getmodulename}@anchor{888}
@deffn {Function} inspect.getmodulename (path)

Return the name of the module named by the file @emph{path}, without including the
names of enclosing packages. The file extension is checked against all of
the entries in @ref{3c0b,,importlib.machinery.all_suffixes()}. If it matches,
the final path component is returned with the extension removed.
Otherwise, @code{None} is returned.

Note that this function @emph{only} returns a meaningful name for actual
Python modules - paths that potentially refer to Python packages will
still return @code{None}.


Changed in version 3.3: The function is based directly on @ref{96,,importlib}.

@end deffn

@geindex ismodule() (in module inspect)
@anchor{library/inspect inspect ismodule}@anchor{3c0c}
@deffn {Function} inspect.ismodule (object)

Return @code{True} if the object is a module.
@end deffn

@geindex isclass() (in module inspect)
@anchor{library/inspect inspect isclass}@anchor{3c0d}
@deffn {Function} inspect.isclass (object)

Return @code{True} if the object is a class, whether built-in or created in Python
code.
@end deffn

@geindex ismethod() (in module inspect)
@anchor{library/inspect inspect ismethod}@anchor{3c0e}
@deffn {Function} inspect.ismethod (object)

Return @code{True} if the object is a bound method written in Python.
@end deffn

@geindex isfunction() (in module inspect)
@anchor{library/inspect inspect isfunction}@anchor{3c0f}
@deffn {Function} inspect.isfunction (object)

Return @code{True} if the object is a Python function, which includes functions
created by a @ref{1fec,,lambda} expression.
@end deffn

@geindex isgeneratorfunction() (in module inspect)
@anchor{library/inspect inspect isgeneratorfunction}@anchor{12e6}
@deffn {Function} inspect.isgeneratorfunction (object)

Return @code{True} if the object is a Python generator function.


Changed in version 3.8: Functions wrapped in @ref{a3f,,functools.partial()} now return @code{True} if the
wrapped function is a Python generator function.

@end deffn

@geindex isgenerator() (in module inspect)
@anchor{library/inspect inspect isgenerator}@anchor{3c10}
@deffn {Function} inspect.isgenerator (object)

Return @code{True} if the object is a generator.
@end deffn

@geindex iscoroutinefunction() (in module inspect)
@anchor{library/inspect inspect iscoroutinefunction}@anchor{979}
@deffn {Function} inspect.iscoroutinefunction (object)

Return @code{True} if the object is a @ref{8cf,,coroutine function}
(a function defined with an @ref{1c0,,async def} syntax).


New in version 3.5.



Changed in version 3.8: Functions wrapped in @ref{a3f,,functools.partial()} now return @code{True} if the
wrapped function is a @ref{8cf,,coroutine function}.

@end deffn

@geindex iscoroutine() (in module inspect)
@anchor{library/inspect inspect iscoroutine}@anchor{978}
@deffn {Function} inspect.iscoroutine (object)

Return @code{True} if the object is a @ref{3cf,,coroutine} created by an
@ref{1c0,,async def} function.


New in version 3.5.

@end deffn

@geindex isawaitable() (in module inspect)
@anchor{library/inspect inspect isawaitable}@anchor{97a}
@deffn {Function} inspect.isawaitable (object)

Return @code{True} if the object can be used in @ref{4d5,,await} expression.

Can also be used to distinguish generator-based coroutines from regular
generators:

@example
def gen():
    yield
@@types.coroutine
def gen_coro():
    yield

assert not isawaitable(gen())
assert isawaitable(gen_coro())
@end example


New in version 3.5.

@end deffn

@geindex isasyncgenfunction() (in module inspect)
@anchor{library/inspect inspect isasyncgenfunction}@anchor{12e7}
@deffn {Function} inspect.isasyncgenfunction (object)

Return @code{True} if the object is an @ref{18ac,,asynchronous generator} function,
for example:

@example
>>> async def agen():
...     yield 1
...
>>> inspect.isasyncgenfunction(agen)
True
@end example


New in version 3.6.



Changed in version 3.8: Functions wrapped in @ref{a3f,,functools.partial()} now return @code{True} if the
wrapped function is a @ref{18ac,,asynchronous generator} function.

@end deffn

@geindex isasyncgen() (in module inspect)
@anchor{library/inspect inspect isasyncgen}@anchor{3c11}
@deffn {Function} inspect.isasyncgen (object)

Return @code{True} if the object is an @ref{3b71,,asynchronous generator iterator}
created by an @ref{18ac,,asynchronous generator} function.


New in version 3.6.

@end deffn

@geindex istraceback() (in module inspect)
@anchor{library/inspect inspect istraceback}@anchor{3c12}
@deffn {Function} inspect.istraceback (object)

Return @code{True} if the object is a traceback.
@end deffn

@geindex isframe() (in module inspect)
@anchor{library/inspect inspect isframe}@anchor{3c13}
@deffn {Function} inspect.isframe (object)

Return @code{True} if the object is a frame.
@end deffn

@geindex iscode() (in module inspect)
@anchor{library/inspect inspect iscode}@anchor{3c14}
@deffn {Function} inspect.iscode (object)

Return @code{True} if the object is a code.
@end deffn

@geindex isbuiltin() (in module inspect)
@anchor{library/inspect inspect isbuiltin}@anchor{3c15}
@deffn {Function} inspect.isbuiltin (object)

Return @code{True} if the object is a built-in function or a bound built-in method.
@end deffn

@geindex ismethodwrapper() (in module inspect)
@anchor{library/inspect inspect ismethodwrapper}@anchor{176}
@deffn {Function} inspect.ismethodwrapper (object)

Return @code{True} if the type of object is a @ref{177,,MethodWrapperType}.

These are instances of @ref{177,,MethodWrapperType}, such as @code{__str__()},
@code{__eq__()} and @code{__repr__()}
@end deffn

@geindex isroutine() (in module inspect)
@anchor{library/inspect inspect isroutine}@anchor{1098}
@deffn {Function} inspect.isroutine (object)

Return @code{True} if the object is a user-defined or built-in function or method.
@end deffn

@geindex isabstract() (in module inspect)
@anchor{library/inspect inspect isabstract}@anchor{3c16}
@deffn {Function} inspect.isabstract (object)

Return @code{True} if the object is an abstract base class.
@end deffn

@geindex ismethoddescriptor() (in module inspect)
@anchor{library/inspect inspect ismethoddescriptor}@anchor{3c17}
@deffn {Function} inspect.ismethoddescriptor (object)

Return @code{True} if the object is a method descriptor, but not if
@ref{3c0e,,ismethod()}, @ref{3c0d,,isclass()}, @ref{3c0f,,isfunction()} or @ref{3c15,,isbuiltin()}
are true.

This, for example, is true of @code{int.__add__}.  An object passing this test
has a @ref{17b0,,__get__()} method but not a @ref{17b5,,__set__()}
method, but beyond that the set of attributes varies.  A
@ref{e70,,__name__} attribute is usually
sensible, and @code{__doc__} often is.

Methods implemented via descriptors that also pass one of the other tests
return @code{False} from the @ref{3c17,,ismethoddescriptor()} test, simply because the
other tests promise more – you can, e.g., count on having the
@code{__func__} attribute (etc) when an object passes @ref{3c0e,,ismethod()}.
@end deffn

@geindex isdatadescriptor() (in module inspect)
@anchor{library/inspect inspect isdatadescriptor}@anchor{3c18}
@deffn {Function} inspect.isdatadescriptor (object)

Return @code{True} if the object is a data descriptor.

Data descriptors have a @ref{17b5,,__set__} or a @ref{17b6,,__delete__} method.
Examples are properties (defined in Python), getsets, and members.  The
latter two are defined in C and there are more specific tests available for
those types, which is robust across Python implementations.  Typically, data
descriptors will also have @ref{e70,,__name__} and @code{__doc__} attributes
(properties, getsets, and members have both of these attributes), but this is
not guaranteed.
@end deffn

@geindex isgetsetdescriptor() (in module inspect)
@anchor{library/inspect inspect isgetsetdescriptor}@anchor{3c19}
@deffn {Function} inspect.isgetsetdescriptor (object)

Return @code{True} if the object is a getset descriptor.

@strong{CPython implementation detail:} getsets are attributes defined in extension modules via
@ref{6ec,,PyGetSetDef} structures.  For Python implementations without such
types, this method will always return @code{False}.
@end deffn

@geindex ismemberdescriptor() (in module inspect)
@anchor{library/inspect inspect ismemberdescriptor}@anchor{3c1a}
@deffn {Function} inspect.ismemberdescriptor (object)

Return @code{True} if the object is a member descriptor.

@strong{CPython implementation detail:} Member descriptors are attributes defined in extension modules via
@ref{6eb,,PyMemberDef} structures.  For Python implementations without such
types, this method will always return @code{False}.
@end deffn

@node Retrieving source code,Introspecting callables with the Signature object,Types and members,inspect — Inspect live objects
@anchor{library/inspect inspect-source}@anchor{3c1b}@anchor{library/inspect retrieving-source-code}@anchor{3c1c}
@subsubsection Retrieving source code


@geindex getdoc() (in module inspect)
@anchor{library/inspect inspect getdoc}@anchor{500}
@deffn {Function} inspect.getdoc (object)

Get the documentation string for an object, cleaned up with @ref{3c1d,,cleandoc()}.
If the documentation string for an object is not provided and the object is
a class, a method, a property or a descriptor, retrieve the documentation
string from the inheritance hierarchy.


Changed in version 3.5: Documentation strings are now inherited if not overridden.

@end deffn

@geindex getcomments() (in module inspect)
@anchor{library/inspect inspect getcomments}@anchor{378f}
@deffn {Function} inspect.getcomments (object)

Return in a single string any lines of comments immediately preceding the
object’s source code (for a class, function, or method), or at the top of the
Python source file (if the object is a module).  If the object’s source code
is unavailable, return @code{None}.  This could happen if the object has been
defined in C or the interactive shell.
@end deffn

@geindex getfile() (in module inspect)
@anchor{library/inspect inspect getfile}@anchor{1365}
@deffn {Function} inspect.getfile (object)

Return the name of the (text or binary) file in which an object was defined.
This will fail with a @ref{162,,TypeError} if the object is a built-in module,
class, or function.
@end deffn

@geindex getmodule() (in module inspect)
@anchor{library/inspect inspect getmodule}@anchor{10e3}
@deffn {Function} inspect.getmodule (object)

Try to guess which module an object was defined in.
@end deffn

@geindex getsourcefile() (in module inspect)
@anchor{library/inspect inspect getsourcefile}@anchor{3c1e}
@deffn {Function} inspect.getsourcefile (object)

Return the name of the Python source file in which an object was defined.  This
will fail with a @ref{162,,TypeError} if the object is a built-in module, class, or
function.
@end deffn

@geindex getsourcelines() (in module inspect)
@anchor{library/inspect inspect getsourcelines}@anchor{3c1f}
@deffn {Function} inspect.getsourcelines (object)

Return a list of source lines and starting line number for an object. The
argument may be a module, class, method, function, traceback, frame, or code
object.  The source code is returned as a list of the lines corresponding to the
object and the line number indicates where in the original source file the first
line of code was found.  An @ref{2a1,,OSError} is raised if the source code cannot
be retrieved.


Changed in version 3.3: @ref{2a1,,OSError} is raised instead of @ref{be8,,IOError}, now an alias of the
former.

@end deffn

@geindex getsource() (in module inspect)
@anchor{library/inspect inspect getsource}@anchor{1105}
@deffn {Function} inspect.getsource (object)

Return the text of the source code for an object. The argument may be a module,
class, method, function, traceback, frame, or code object.  The source code is
returned as a single string.  An @ref{2a1,,OSError} is raised if the source code
cannot be retrieved.


Changed in version 3.3: @ref{2a1,,OSError} is raised instead of @ref{be8,,IOError}, now an alias of the
former.

@end deffn

@geindex cleandoc() (in module inspect)
@anchor{library/inspect inspect cleandoc}@anchor{3c1d}
@deffn {Function} inspect.cleandoc (doc)

Clean up indentation from docstrings that are indented to line up with blocks
of code.

All leading whitespace is removed from the first line.  Any leading whitespace
that can be uniformly removed from the second line onwards is removed.  Empty
lines at the beginning and end are subsequently removed.  Also, all tabs are
expanded to spaces.
@end deffn

@node Introspecting callables with the Signature object,Classes and functions<2>,Retrieving source code,inspect — Inspect live objects
@anchor{library/inspect inspect-signature-object}@anchor{3c20}@anchor{library/inspect introspecting-callables-with-the-signature-object}@anchor{3c21}
@subsubsection Introspecting callables with the Signature object



New in version 3.3.


The Signature object represents the call signature of a callable object and its
return annotation.  To retrieve a Signature object, use the @ref{1c1,,signature()}
function.

@geindex signature() (in module inspect)
@anchor{library/inspect inspect signature}@anchor{1c1}
@deffn {Function} inspect.signature (callable, *, follow_wrapped=True, globals=None, locals=None, eval_str=False)

Return a @ref{975,,Signature} object for the given @code{callable}:

@example
>>> from inspect import signature
>>> def foo(a, *, b:int, **kwargs):
...     pass

>>> sig = signature(foo)

>>> str(sig)
'(a, *, b:int, **kwargs)'

>>> str(sig.parameters['b'])
'b:int'

>>> sig.parameters['b'].annotation
<class 'int'>
@end example

Accepts a wide range of Python callables, from plain functions and classes to
@ref{a3f,,functools.partial()} objects.

For objects defined in modules using stringized annotations
(@code{from __future__ import annotations}), @ref{1c1,,signature()} will
attempt to automatically un-stringize the annotations using
@ref{295,,inspect.get_annotations()}.  The
@code{global}, @code{locals}, and @code{eval_str} parameters are passed
into @ref{295,,inspect.get_annotations()} when resolving the
annotations; see the documentation for @ref{295,,inspect.get_annotations()}
for instructions on how to use these parameters.

Raises @ref{17c,,ValueError} if no signature can be provided, and
@ref{162,,TypeError} if that type of object is not supported.  Also,
if the annotations are stringized, and @code{eval_str} is not false,
the @code{eval()} call(s) to un-stringize the annotations could
potentially raise any kind of exception.

A slash(/) in the signature of a function denotes that the parameters prior
to it are positional-only. For more info, see
@ref{1a50,,the FAQ entry on positional-only parameters}.


New in version 3.5: @code{follow_wrapped} parameter. Pass @code{False} to get a signature of
@code{callable} specifically (@code{callable.__wrapped__} will not be used to
unwrap decorated callables.)



New in version 3.10: @code{globals}, @code{locals}, and @code{eval_str} parameters.


@cartouche
@quotation Note 
Some callables may not be introspectable in certain implementations of
Python.  For example, in CPython, some built-in functions defined in
C provide no metadata about their arguments.
@end quotation
@end cartouche
@end deffn

@geindex Signature (class in inspect)
@anchor{library/inspect inspect Signature}@anchor{975}
@deffn {Class} inspect.Signature (parameters=None, *, return_annotation=Signature.empty)

A Signature object represents the call signature of a function and its return
annotation.  For each parameter accepted by the function it stores a
@ref{976,,Parameter} object in its @ref{122b,,parameters} collection.

The optional @emph{parameters} argument is a sequence of @ref{976,,Parameter}
objects, which is validated to check that there are no parameters with
duplicate names, and that the parameters are in the right order, i.e.
positional-only first, then positional-or-keyword, and that parameters with
defaults follow parameters without defaults.

The optional @emph{return_annotation} argument, can be an arbitrary Python object,
is the “return” annotation of the callable.

Signature objects are @emph{immutable}.  Use @ref{3c22,,Signature.replace()} to make a
modified copy.


Changed in version 3.5: Signature objects are picklable and hashable.


@geindex empty (inspect.Signature attribute)
@anchor{library/inspect inspect Signature empty}@anchor{3c23}
@deffn {Attribute} empty

A special class-level marker to specify absence of a return annotation.
@end deffn

@geindex parameters (inspect.Signature attribute)
@anchor{library/inspect inspect Signature parameters}@anchor{122b}
@deffn {Attribute} parameters

An ordered mapping of parameters’ names to the corresponding
@ref{976,,Parameter} objects.  Parameters appear in strict definition
order, including keyword-only parameters.


Changed in version 3.7: Python only explicitly guaranteed that it preserved the declaration
order of keyword-only parameters as of version 3.7, although in practice
this order had always been preserved in Python 3.

@end deffn

@geindex return_annotation (inspect.Signature attribute)
@anchor{library/inspect inspect Signature return_annotation}@anchor{3c24}
@deffn {Attribute} return_annotation

The “return” annotation for the callable.  If the callable has no “return”
annotation, this attribute is set to @ref{3c23,,Signature.empty}.
@end deffn

@geindex bind() (inspect.Signature method)
@anchor{library/inspect inspect Signature bind}@anchor{3c25}
@deffn {Method} bind (*args, **kwargs)

Create a mapping from positional and keyword arguments to parameters.
Returns @ref{bfd,,BoundArguments} if @code{*args} and @code{**kwargs} match the
signature, or raises a @ref{162,,TypeError}.
@end deffn

@geindex bind_partial() (inspect.Signature method)
@anchor{library/inspect inspect Signature bind_partial}@anchor{3c26}
@deffn {Method} bind_partial (*args, **kwargs)

Works the same way as @ref{3c25,,Signature.bind()}, but allows the omission of
some required arguments (mimics @ref{a3f,,functools.partial()} behavior.)
Returns @ref{bfd,,BoundArguments}, or raises a @ref{162,,TypeError} if the
passed arguments do not match the signature.
@end deffn

@geindex replace() (inspect.Signature method)
@anchor{library/inspect inspect Signature replace}@anchor{3c22}
@deffn {Method} replace (*[, parameters][, return_annotation])

Create a new Signature instance based on the instance replace was invoked
on.  It is possible to pass different @code{parameters} and/or
@code{return_annotation} to override the corresponding properties of the base
signature.  To remove return_annotation from the copied Signature, pass in
@ref{3c23,,Signature.empty}.

@example
>>> def test(a, b):
...     pass
>>> sig = signature(test)
>>> new_sig = sig.replace(return_annotation="new return anno")
>>> str(new_sig)
"(a, b) -> 'new return anno'"
@end example
@end deffn

@geindex from_callable() (inspect.Signature class method)
@anchor{library/inspect inspect Signature from_callable}@anchor{1c3}
@deffn {Method} classmethod from_callable (obj, *, follow_wrapped=True, globalns=None, localns=None)

Return a @ref{975,,Signature} (or its subclass) object for a given callable
@code{obj}.  Pass @code{follow_wrapped=False} to get a signature of @code{obj}
without unwrapping its @code{__wrapped__} chain. @code{globalns} and
@code{localns} will be used as the namespaces when resolving annotations.

This method simplifies subclassing of @ref{975,,Signature}:

@example
class MySignature(Signature):
    pass
sig = MySignature.from_callable(min)
assert isinstance(sig, MySignature)
@end example


New in version 3.5.



New in version 3.10: @code{globalns} and @code{localns} parameters.

@end deffn
@end deffn

@geindex Parameter (class in inspect)
@anchor{library/inspect inspect Parameter}@anchor{976}
@deffn {Class} inspect.Parameter (name, kind, *, default=Parameter.empty, annotation=Parameter.empty)

Parameter objects are @emph{immutable}.  Instead of modifying a Parameter object,
you can use @ref{3c27,,Parameter.replace()} to create a modified copy.


Changed in version 3.5: Parameter objects are picklable and hashable.


@geindex empty (inspect.Parameter attribute)
@anchor{library/inspect inspect Parameter empty}@anchor{3c28}
@deffn {Attribute} empty

A special class-level marker to specify absence of default values and
annotations.
@end deffn

@geindex name (inspect.Parameter attribute)
@anchor{library/inspect inspect Parameter name}@anchor{3c29}
@deffn {Attribute} name

The name of the parameter as a string.  The name must be a valid
Python identifier.

@strong{CPython implementation detail:} CPython generates implicit parameter names of the form @code{.0} on the
code objects used to implement comprehensions and generator
expressions.


Changed in version 3.6: These parameter names are exposed by this module as names like
@code{implicit0}.

@end deffn

@geindex default (inspect.Parameter attribute)
@anchor{library/inspect inspect Parameter default}@anchor{3c2a}
@deffn {Attribute} default

The default value for the parameter.  If the parameter has no default
value, this attribute is set to @ref{3c28,,Parameter.empty}.
@end deffn

@geindex annotation (inspect.Parameter attribute)
@anchor{library/inspect inspect Parameter annotation}@anchor{3c2b}
@deffn {Attribute} annotation

The annotation for the parameter.  If the parameter has no annotation,
this attribute is set to @ref{3c28,,Parameter.empty}.
@end deffn

@geindex kind (inspect.Parameter attribute)
@anchor{library/inspect inspect Parameter kind}@anchor{3c2c}
@deffn {Attribute} kind

Describes how argument values are bound to the parameter.  Possible values
(accessible via @ref{976,,Parameter}, like @code{Parameter.KEYWORD_ONLY}):


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Name

@tab

Meaning

@item

@emph{POSITIONAL_ONLY}

@tab

Value must be supplied as a positional
argument. Positional only parameters are
those which appear before a @code{/} entry (if
present) in a Python function definition.

@item

@emph{POSITIONAL_OR_KEYWORD}

@tab

Value may be supplied as either a keyword or
positional argument (this is the standard
binding behaviour for functions implemented
in Python.)

@item

@emph{VAR_POSITIONAL}

@tab

A tuple of positional arguments that aren’t
bound to any other parameter. This
corresponds to a @code{*args} parameter in a
Python function definition.

@item

@emph{KEYWORD_ONLY}

@tab

Value must be supplied as a keyword argument.
Keyword only parameters are those which
appear after a @code{*} or @code{*args} entry in a
Python function definition.

@item

@emph{VAR_KEYWORD}

@tab

A dict of keyword arguments that aren’t bound
to any other parameter. This corresponds to a
@code{**kwargs} parameter in a Python function
definition.

@end multitable


Example: print all keyword-only arguments without default values:

@example
>>> def foo(a, b, *, c, d=10):
...     pass

>>> sig = signature(foo)
>>> for param in sig.parameters.values():
...     if (param.kind == param.KEYWORD_ONLY and
...                        param.default is param.empty):
...         print('Parameter:', param)
Parameter: c
@end example
@end deffn

@geindex description (inspect.Parameter.kind attribute)
@anchor{library/inspect inspect Parameter kind description}@anchor{3c2d}
@deffn {Attribute} kind.description

Describes a enum value of Parameter.kind.


New in version 3.8.


Example: print all descriptions of arguments:

@example
>>> def foo(a, b, *, c, d=10):
...     pass

>>> sig = signature(foo)
>>> for param in sig.parameters.values():
...     print(param.kind.description)
positional or keyword
positional or keyword
keyword-only
keyword-only
@end example
@end deffn

@geindex replace() (inspect.Parameter method)
@anchor{library/inspect inspect Parameter replace}@anchor{3c27}
@deffn {Method} replace (*[, name][, kind][, default][, annotation])

Create a new Parameter instance based on the instance replaced was invoked
on.  To override a @ref{976,,Parameter} attribute, pass the corresponding
argument.  To remove a default value or/and an annotation from a
Parameter, pass @ref{3c28,,Parameter.empty}.

@example
>>> from inspect import Parameter
>>> param = Parameter('foo', Parameter.KEYWORD_ONLY, default=42)
>>> str(param)
'foo=42'

>>> str(param.replace()) # Will create a shallow copy of 'param'
'foo=42'

>>> str(param.replace(default=Parameter.empty, annotation='spam'))
"foo:'spam'"
@end example
@end deffn


Changed in version 3.4: In Python 3.3 Parameter objects were allowed to have @code{name} set
to @code{None} if their @code{kind} was set to @code{POSITIONAL_ONLY}.
This is no longer permitted.

@end deffn

@geindex BoundArguments (class in inspect)
@anchor{library/inspect inspect BoundArguments}@anchor{bfd}
@deffn {Class} inspect.BoundArguments

Result of a @ref{3c25,,Signature.bind()} or @ref{3c26,,Signature.bind_partial()} call.
Holds the mapping of arguments to the function’s parameters.

@geindex arguments (inspect.BoundArguments attribute)
@anchor{library/inspect inspect BoundArguments arguments}@anchor{3fc}
@deffn {Attribute} arguments

A mutable mapping of parameters’ names to arguments’ values.
Contains only explicitly bound arguments.  Changes in @ref{3fc,,arguments}
will reflect in @ref{3c2e,,args} and @ref{3c2f,,kwargs}.

Should be used in conjunction with @ref{122b,,Signature.parameters} for any
argument processing purposes.

@cartouche
@quotation Note 
Arguments for which @ref{3c25,,Signature.bind()} or
@ref{3c26,,Signature.bind_partial()} relied on a default value are skipped.
However, if needed, use @ref{977,,BoundArguments.apply_defaults()} to add
them.
@end quotation
@end cartouche


Changed in version 3.9: @ref{3fc,,arguments} is now of type @ref{3b0,,dict}. Formerly, it was of
type @ref{4e5,,collections.OrderedDict}.

@end deffn

@geindex args (inspect.BoundArguments attribute)
@anchor{library/inspect inspect BoundArguments args}@anchor{3c2e}
@deffn {Attribute} args

A tuple of positional arguments values.  Dynamically computed from the
@ref{3fc,,arguments} attribute.
@end deffn

@geindex kwargs (inspect.BoundArguments attribute)
@anchor{library/inspect inspect BoundArguments kwargs}@anchor{3c2f}
@deffn {Attribute} kwargs

A dict of keyword arguments values.  Dynamically computed from the
@ref{3fc,,arguments} attribute.
@end deffn

@geindex signature (inspect.BoundArguments attribute)
@anchor{library/inspect inspect BoundArguments signature}@anchor{3c30}
@deffn {Attribute} signature

A reference to the parent @ref{975,,Signature} object.
@end deffn

@geindex apply_defaults() (inspect.BoundArguments method)
@anchor{library/inspect inspect BoundArguments apply_defaults}@anchor{977}
@deffn {Method} apply_defaults ()

Set default values for missing arguments.

For variable-positional arguments (@code{*args}) the default is an
empty tuple.

For variable-keyword arguments (@code{**kwargs}) the default is an
empty dict.

@example
>>> def foo(a, b='ham', *args): pass
>>> ba = inspect.signature(foo).bind('spam')
>>> ba.apply_defaults()
>>> ba.arguments
@{'a': 'spam', 'b': 'ham', 'args': ()@}
@end example


New in version 3.5.

@end deffn

The @ref{3c2e,,args} and @ref{3c2f,,kwargs} properties can be used to invoke
functions:

@example
def test(a, *, b):
    ...

sig = signature(test)
ba = sig.bind(10, b=20)
test(*ba.args, **ba.kwargs)
@end example
@end deffn


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 362

@item PEP 362@footnote{https://www.python.org/dev/peps/pep-0362} - Function Signature Object.

The detailed specification, implementation details and examples.
@end table


@node Classes and functions<2>,The interpreter stack,Introspecting callables with the Signature object,inspect — Inspect live objects
@anchor{library/inspect classes-and-functions}@anchor{3c31}@anchor{library/inspect inspect-classes-functions}@anchor{3c32}
@subsubsection Classes and functions


@geindex getclasstree() (in module inspect)
@anchor{library/inspect inspect getclasstree}@anchor{3c33}
@deffn {Function} inspect.getclasstree (classes, unique=False)

Arrange the given list of classes into a hierarchy of nested lists. Where a
nested list appears, it contains classes derived from the class whose entry
immediately precedes the list.  Each entry is a 2-tuple containing a class and a
tuple of its base classes.  If the @emph{unique} argument is true, exactly one entry
appears in the returned structure for each class in the given list.  Otherwise,
classes using multiple inheritance and their descendants will appear multiple
times.
@end deffn

@geindex getfullargspec() (in module inspect)
@anchor{library/inspect inspect getfullargspec}@anchor{1c2}
@deffn {Function} inspect.getfullargspec (func)

Get the names and default values of a Python function’s parameters.  A
@ref{ce9,,named tuple} is returned:

@code{FullArgSpec(args, varargs, varkw, defaults, kwonlyargs, kwonlydefaults,
annotations)}

@emph{args} is a list of the positional parameter names.
@emph{varargs} is the name of the @code{*} parameter or @code{None} if arbitrary
positional arguments are not accepted.
@emph{varkw} is the name of the @code{**} parameter or @code{None} if arbitrary
keyword arguments are not accepted.
@emph{defaults} is an @emph{n}-tuple of default argument values corresponding to the
last @emph{n} positional parameters, or @code{None} if there are no such defaults
defined.
@emph{kwonlyargs} is a list of keyword-only parameter names in declaration order.
@emph{kwonlydefaults} is a dictionary mapping parameter names from @emph{kwonlyargs}
to the default values used if no argument is supplied.
@emph{annotations} is a dictionary mapping parameter names to annotations.
The special key @code{"return"} is used to report the function return value
annotation (if any).

Note that @ref{1c1,,signature()} and
@ref{3c20,,Signature Object} provide the recommended
API for callable introspection, and support additional behaviours (like
positional-only arguments) that are sometimes encountered in extension module
APIs. This function is retained primarily for use in code that needs to
maintain compatibility with the Python 2 @code{inspect} module API.


Changed in version 3.4: This function is now based on @ref{1c1,,signature()}, but still ignores
@code{__wrapped__} attributes and includes the already bound first
parameter in the signature output for bound methods.



Changed in version 3.6: This method was previously documented as deprecated in favour of
@ref{1c1,,signature()} in Python 3.5, but that decision has been reversed
in order to restore a clearly supported standard interface for
single-source Python 2/3 code migrating away from the legacy
@code{getargspec()} API.



Changed in version 3.7: Python only explicitly guaranteed that it preserved the declaration
order of keyword-only parameters as of version 3.7, although in practice
this order had always been preserved in Python 3.

@end deffn

@geindex getargvalues() (in module inspect)
@anchor{library/inspect inspect getargvalues}@anchor{a30}
@deffn {Function} inspect.getargvalues (frame)

Get information about arguments passed into a particular frame.  A
@ref{ce9,,named tuple} @code{ArgInfo(args, varargs, keywords, locals)} is
returned. @emph{args} is a list of the argument names.  @emph{varargs} and @emph{keywords}
are the names of the @code{*} and @code{**} arguments or @code{None}.  @emph{locals} is the
locals dictionary of the given frame.

@cartouche
@quotation Note 
This function was inadvertently marked as deprecated in Python 3.5.
@end quotation
@end cartouche
@end deffn

@geindex formatargvalues() (in module inspect)
@anchor{library/inspect inspect formatargvalues}@anchor{a31}
@deffn {Function} inspect.formatargvalues (args[, varargs, varkw, locals, formatarg, formatvarargs, formatvarkw, formatvalue])

Format a pretty argument spec from the four values returned by
@ref{a30,,getargvalues()}.  The format* arguments are the corresponding optional
formatting functions that are called to turn names and values into strings.

@cartouche
@quotation Note 
This function was inadvertently marked as deprecated in Python 3.5.
@end quotation
@end cartouche
@end deffn

@geindex getmro() (in module inspect)
@anchor{library/inspect inspect getmro}@anchor{3c34}
@deffn {Function} inspect.getmro (cls)

Return a tuple of class cls’s base classes, including cls, in method resolution
order.  No class appears more than once in this tuple. Note that the method
resolution order depends on cls’s type.  Unless a very peculiar user-defined
metatype is in use, cls will be the first element of the tuple.
@end deffn

@geindex getcallargs() (in module inspect)
@anchor{library/inspect inspect getcallargs}@anchor{a2f}
@deffn {Function} inspect.getcallargs (func, /, *args, **kwds)

Bind the @emph{args} and @emph{kwds} to the argument names of the Python function or
method @emph{func}, as if it was called with them. For bound methods, bind also the
first argument (typically named @code{self}) to the associated instance. A dict
is returned, mapping the argument names (including the names of the @code{*} and
@code{**} arguments, if any) to their values from @emph{args} and @emph{kwds}. In case of
invoking @emph{func} incorrectly, i.e. whenever @code{func(*args, **kwds)} would raise
an exception because of incompatible signature, an exception of the same type
and the same or similar message is raised. For example:

@example
>>> from inspect import getcallargs
>>> def f(a, b=1, *pos, **named):
...     pass
>>> getcallargs(f, 1, 2, 3) == @{'a': 1, 'named': @{@}, 'b': 2, 'pos': (3,)@}
True
>>> getcallargs(f, a=2, x=4) == @{'a': 2, 'named': @{'x': 4@}, 'b': 1, 'pos': ()@}
True
>>> getcallargs(f)
Traceback (most recent call last):
...
TypeError: f() missing 1 required positional argument: 'a'
@end example


New in version 3.2.



Deprecated since version 3.5: Use @ref{3c25,,Signature.bind()} and @ref{3c26,,Signature.bind_partial()} instead.

@end deffn

@geindex getclosurevars() (in module inspect)
@anchor{library/inspect inspect getclosurevars}@anchor{c67}
@deffn {Function} inspect.getclosurevars (func)

Get the mapping of external name references in a Python function or
method @emph{func} to their current values. A
@ref{ce9,,named tuple} @code{ClosureVars(nonlocals, globals, builtins, unbound)}
is returned. @emph{nonlocals} maps referenced names to lexical closure
variables, @emph{globals} to the function’s module globals and @emph{builtins} to
the builtins visible from the function body. @emph{unbound} is the set of names
referenced in the function that could not be resolved at all given the
current module globals and builtins.

@ref{162,,TypeError} is raised if @emph{func} is not a Python function or method.


New in version 3.3.

@end deffn

@geindex unwrap() (in module inspect)
@anchor{library/inspect inspect unwrap}@anchor{ada}
@deffn {Function} inspect.unwrap (func, *, stop=None)

Get the object wrapped by @emph{func}. It follows the chain of @code{__wrapped__}
attributes returning the last object in the chain.

@emph{stop} is an optional callback accepting an object in the wrapper chain
as its sole argument that allows the unwrapping to be terminated early if
the callback returns a true value. If the callback never returns a true
value, the last object in the chain is returned as usual. For example,
@ref{1c1,,signature()} uses this to stop unwrapping if any object in the
chain has a @code{__signature__} attribute defined.

@ref{17c,,ValueError} is raised if a cycle is encountered.


New in version 3.4.

@end deffn

@geindex get_annotations() (in module inspect)
@anchor{library/inspect inspect get_annotations}@anchor{295}
@deffn {Function} inspect.get_annotations (obj, *, globals=None, locals=None, eval_str=False)

Compute the annotations dict for an object.

@code{obj} may be a callable, class, or module.
Passing in an object of any other type raises @ref{162,,TypeError}.

Returns a dict.  @code{get_annotations()} returns a new dict every time
it’s called; calling it twice on the same object will return two
different but equivalent dicts.

This function handles several details for you:


@itemize *

@item 
If @code{eval_str} is true, values of type @code{str} will
be un-stringized using @ref{34a,,eval()}.  This is intended
for use with stringized annotations
(@code{from __future__ import annotations}).

@item 
If @code{obj} doesn’t have an annotations dict, returns an
empty dict.  (Functions and methods always have an
annotations dict; classes, modules, and other types of
callables may not.)

@item 
Ignores inherited annotations on classes.  If a class
doesn’t have its own annotations dict, returns an empty dict.

@item 
All accesses to object members and dict values are done
using @code{getattr()} and @code{dict.get()} for safety.

@item 
Always, always, always returns a freshly-created dict.
@end itemize

@code{eval_str} controls whether or not values of type @code{str} are replaced
with the result of calling @ref{34a,,eval()} on those values:


@itemize *

@item 
If eval_str is true, @ref{34a,,eval()} is called on values of type @code{str}.
(Note that @code{get_annotations} doesn’t catch exceptions; if @ref{34a,,eval()}
raises an exception, it will unwind the stack past the @code{get_annotations}
call.)

@item 
If eval_str is false (the default), values of type @code{str} are unchanged.
@end itemize

@code{globals} and @code{locals} are passed in to @ref{34a,,eval()}; see the documentation
for @ref{34a,,eval()} for more information.  If @code{globals} or @code{locals}
is @code{None}, this function may replace that value with a context-specific
default, contingent on @code{type(obj)}:


@itemize *

@item 
If @code{obj} is a module, @code{globals} defaults to @code{obj.__dict__}.

@item 
If @code{obj} is a class, @code{globals} defaults to
@code{sys.modules[obj.__module__].__dict__} and @code{locals} defaults
to the @code{obj} class namespace.

@item 
If @code{obj} is a callable, @code{globals} defaults to @code{obj.__globals__},
although if @code{obj} is a wrapped function (using
@code{functools.update_wrapper()}) it is first unwrapped.
@end itemize

Calling @code{get_annotations} is best practice for accessing the
annotations dict of any object.  See @ref{255,,Annotations Best Practices} for
more information on annotations best practices.


New in version 3.10.

@end deffn

@node The interpreter stack,Fetching attributes statically,Classes and functions<2>,inspect — Inspect live objects
@anchor{library/inspect inspect-stack}@anchor{3c35}@anchor{library/inspect the-interpreter-stack}@anchor{3c36}
@subsubsection The interpreter stack


When the following functions return “frame records,” each record is a
@ref{ce9,,named tuple}
@code{FrameInfo(frame, filename, lineno, function, code_context, index)}.
The tuple contains the frame object, the filename, the line number of the
current line,
the function name, a list of lines of context from the source code, and the
index of the current line within that list.


Changed in version 3.5: Return a named tuple instead of a tuple.


@cartouche
@quotation Note 
Keeping references to frame objects, as found in the first element of the frame
records these functions return, can cause your program to create reference
cycles.  Once a reference cycle has been created, the lifespan of all objects
which can be accessed from the objects which form the cycle can become much
longer even if Python’s optional cycle detector is enabled.  If such cycles must
be created, it is important to ensure they are explicitly broken to avoid the
delayed destruction of objects and increased memory consumption which occurs.

Though the cycle detector will catch these, destruction of the frames (and local
variables) can be made deterministic by removing the cycle in a
@ref{4c1,,finally} clause.  This is also important if the cycle detector was
disabled when Python was compiled or using @ref{3bfb,,gc.disable()}.  For example:

@example
def handle_stackframe_without_leak():
    frame = inspect.currentframe()
    try:
        # do something with the frame
    finally:
        del frame
@end example

If you want to keep the frame around (for example to print a traceback
later), you can also break reference cycles by using the
@ref{a7f,,frame.clear()} method.
@end quotation
@end cartouche

The optional @emph{context} argument supported by most of these functions specifies
the number of lines of context to return, which are centered around the current
line.

@geindex getframeinfo() (in module inspect)
@anchor{library/inspect inspect getframeinfo}@anchor{3c37}
@deffn {Function} inspect.getframeinfo (frame, context=1)

Get information about a frame or traceback object.  A @ref{ce9,,named tuple}
@code{Traceback(filename, lineno, function, code_context, index)} is returned.
@end deffn

@geindex getouterframes() (in module inspect)
@anchor{library/inspect inspect getouterframes}@anchor{97f}
@deffn {Function} inspect.getouterframes (frame, context=1)

Get a list of frame records for a frame and all outer frames.  These frames
represent the calls that lead to the creation of @emph{frame}. The first entry in the
returned list represents @emph{frame}; the last entry represents the outermost call
on @emph{frame}’s stack.


Changed in version 3.5: A list of @ref{ce9,,named tuples}
@code{FrameInfo(frame, filename, lineno, function, code_context, index)}
is returned.

@end deffn

@geindex getinnerframes() (in module inspect)
@anchor{library/inspect inspect getinnerframes}@anchor{980}
@deffn {Function} inspect.getinnerframes (traceback, context=1)

Get a list of frame records for a traceback’s frame and all inner frames.  These
frames represent calls made as a consequence of @emph{frame}.  The first entry in the
list represents @emph{traceback}; the last entry represents where the exception was
raised.


Changed in version 3.5: A list of @ref{ce9,,named tuples}
@code{FrameInfo(frame, filename, lineno, function, code_context, index)}
is returned.

@end deffn

@geindex currentframe() (in module inspect)
@anchor{library/inspect inspect currentframe}@anchor{3c38}
@deffn {Function} inspect.currentframe ()

Return the frame object for the caller’s stack frame.

@strong{CPython implementation detail:} This function relies on Python stack frame support in the interpreter,
which isn’t guaranteed to exist in all implementations of Python.  If
running in an implementation without Python stack frame support this
function returns @code{None}.
@end deffn

@geindex stack() (in module inspect)
@anchor{library/inspect inspect stack}@anchor{97d}
@deffn {Function} inspect.stack (context=1)

Return a list of frame records for the caller’s stack.  The first entry in the
returned list represents the caller; the last entry represents the outermost
call on the stack.


Changed in version 3.5: A list of @ref{ce9,,named tuples}
@code{FrameInfo(frame, filename, lineno, function, code_context, index)}
is returned.

@end deffn

@geindex trace() (in module inspect)
@anchor{library/inspect inspect trace}@anchor{97e}
@deffn {Function} inspect.trace (context=1)

Return a list of frame records for the stack between the current frame and the
frame in which an exception currently being handled was raised in.  The first
entry in the list represents the caller; the last entry represents where the
exception was raised.


Changed in version 3.5: A list of @ref{ce9,,named tuples}
@code{FrameInfo(frame, filename, lineno, function, code_context, index)}
is returned.

@end deffn

@node Fetching attributes statically,Current State of Generators and Coroutines,The interpreter stack,inspect — Inspect live objects
@anchor{library/inspect fetching-attributes-statically}@anchor{3c39}
@subsubsection Fetching attributes statically


Both @ref{708,,getattr()} and @ref{707,,hasattr()} can trigger code execution when
fetching or checking for the existence of attributes. Descriptors, like
properties, will be invoked and @code{__getattr__()} and @code{__getattribute__()}
may be called.

For cases where you want passive introspection, like documentation tools, this
can be inconvenient. @ref{de8,,getattr_static()} has the same signature as @ref{708,,getattr()}
but avoids executing code when it fetches attributes.

@geindex getattr_static() (in module inspect)
@anchor{library/inspect inspect getattr_static}@anchor{de8}
@deffn {Function} inspect.getattr_static (obj, attr, default=None)

Retrieve attributes without triggering dynamic lookup via the
descriptor protocol, @code{__getattr__()} or @code{__getattribute__()}.

Note: this function may not be able to retrieve all attributes
that getattr can fetch (like dynamically created attributes)
and may find attributes that getattr can’t (like descriptors
that raise AttributeError). It can also return descriptors objects
instead of instance members.

If the instance @ref{7a4,,__dict__} is shadowed by another member (for
example a property) then this function will be unable to find instance
members.


New in version 3.2.

@end deffn

@ref{de8,,getattr_static()} does not resolve descriptors, for example slot descriptors or
getset descriptors on objects implemented in C. The descriptor object
is returned instead of the underlying attribute.

You can handle these with code like the following. Note that
for arbitrary getset descriptors invoking these may trigger
code execution:

@example
# example code for resolving the builtin descriptor types
class _foo:
    __slots__ = ['foo']

slot_descriptor = type(_foo.foo)
getset_descriptor = type(type(open(__file__)).name)
wrapper_descriptor = type(str.__dict__['__add__'])
descriptor_types = (slot_descriptor, getset_descriptor, wrapper_descriptor)

result = getattr_static(some_object, 'foo')
if type(result) in descriptor_types:
    try:
        result = result.__get__()
    except AttributeError:
        # descriptors can raise AttributeError to
        # indicate there is no underlying value
        # in which case the descriptor itself will
        # have to do
        pass
@end example

@node Current State of Generators and Coroutines,Code Objects Bit Flags,Fetching attributes statically,inspect — Inspect live objects
@anchor{library/inspect current-state-of-generators-and-coroutines}@anchor{3c3a}
@subsubsection Current State of Generators and Coroutines


When implementing coroutine schedulers and for other advanced uses of
generators, it is useful to determine whether a generator is currently
executing, is waiting to start or resume or execution, or has already
terminated. @ref{de7,,getgeneratorstate()} allows the current state of a
generator to be determined easily.

@geindex getgeneratorstate() (in module inspect)
@anchor{library/inspect inspect getgeneratorstate}@anchor{de7}
@deffn {Function} inspect.getgeneratorstate (generator)

Get current state of a generator-iterator.


@table @asis

@item Possible states are:


@itemize *

@item 
GEN_CREATED: Waiting to start execution.

@item 
GEN_RUNNING: Currently being executed by the interpreter.

@item 
GEN_SUSPENDED: Currently suspended at a yield expression.

@item 
GEN_CLOSED: Execution has completed.
@end itemize
@end table


New in version 3.2.

@end deffn

@geindex getcoroutinestate() (in module inspect)
@anchor{library/inspect inspect getcoroutinestate}@anchor{97c}
@deffn {Function} inspect.getcoroutinestate (coroutine)

Get current state of a coroutine object.  The function is intended to be
used with coroutine objects created by @ref{1c0,,async def} functions, but
will accept any coroutine-like object that has @code{cr_running} and
@code{cr_frame} attributes.


@table @asis

@item Possible states are:


@itemize *

@item 
CORO_CREATED: Waiting to start execution.

@item 
CORO_RUNNING: Currently being executed by the interpreter.

@item 
CORO_SUSPENDED: Currently suspended at an await expression.

@item 
CORO_CLOSED: Execution has completed.
@end itemize
@end table


New in version 3.5.

@end deffn

The current internal state of the generator can also be queried. This is
mostly useful for testing purposes, to ensure that internal state is being
updated as expected:

@geindex getgeneratorlocals() (in module inspect)
@anchor{library/inspect inspect getgeneratorlocals}@anchor{c68}
@deffn {Function} inspect.getgeneratorlocals (generator)

Get the mapping of live local variables in @emph{generator} to their current
values.  A dictionary is returned that maps from variable names to values.
This is the equivalent of calling @ref{713,,locals()} in the body of the
generator, and all the same caveats apply.

If @emph{generator} is a @ref{bf7,,generator} with no currently associated frame,
then an empty dictionary is returned.  @ref{162,,TypeError} is raised if
@emph{generator} is not a Python generator object.

@strong{CPython implementation detail:} This function relies on the generator exposing a Python stack frame
for introspection, which isn’t guaranteed to be the case in all
implementations of Python. In such cases, this function will always
return an empty dictionary.


New in version 3.3.

@end deffn

@geindex getcoroutinelocals() (in module inspect)
@anchor{library/inspect inspect getcoroutinelocals}@anchor{97b}
@deffn {Function} inspect.getcoroutinelocals (coroutine)

This function is analogous to @ref{c68,,getgeneratorlocals()}, but
works for coroutine objects created by @ref{1c0,,async def} functions.


New in version 3.5.

@end deffn

@node Code Objects Bit Flags,Command Line Interface<3>,Current State of Generators and Coroutines,inspect — Inspect live objects
@anchor{library/inspect code-objects-bit-flags}@anchor{3c3b}@anchor{library/inspect inspect-module-co-flags}@anchor{3c0a}
@subsubsection Code Objects Bit Flags


Python code objects have a @code{co_flags} attribute, which is a bitmap of
the following flags:

@geindex CO_OPTIMIZED (in module inspect)
@anchor{library/inspect inspect CO_OPTIMIZED}@anchor{3c3c}
@deffn {Data} inspect.CO_OPTIMIZED

The code object is optimized, using fast locals.
@end deffn

@geindex CO_NEWLOCALS (in module inspect)
@anchor{library/inspect inspect CO_NEWLOCALS}@anchor{3c3d}
@deffn {Data} inspect.CO_NEWLOCALS

If set, a new dict will be created for the frame’s @code{f_locals} when
the code object is executed.
@end deffn

@geindex CO_VARARGS (in module inspect)
@anchor{library/inspect inspect CO_VARARGS}@anchor{3c3e}
@deffn {Data} inspect.CO_VARARGS

The code object has a variable positional parameter (@code{*args}-like).
@end deffn

@geindex CO_VARKEYWORDS (in module inspect)
@anchor{library/inspect inspect CO_VARKEYWORDS}@anchor{3c3f}
@deffn {Data} inspect.CO_VARKEYWORDS

The code object has a variable keyword parameter (@code{**kwargs}-like).
@end deffn

@geindex CO_NESTED (in module inspect)
@anchor{library/inspect inspect CO_NESTED}@anchor{3c40}
@deffn {Data} inspect.CO_NESTED

The flag is set when the code object is a nested function.
@end deffn

@geindex CO_GENERATOR (in module inspect)
@anchor{library/inspect inspect CO_GENERATOR}@anchor{3c41}
@deffn {Data} inspect.CO_GENERATOR

The flag is set when the code object is a generator function, i.e.
a generator object is returned when the code object is executed.
@end deffn

@geindex CO_COROUTINE (in module inspect)
@anchor{library/inspect inspect CO_COROUTINE}@anchor{3c42}
@deffn {Data} inspect.CO_COROUTINE

The flag is set when the code object is a coroutine function.
When the code object is executed it returns a coroutine object.
See 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} for more details.


New in version 3.5.

@end deffn

@geindex CO_ITERABLE_COROUTINE (in module inspect)
@anchor{library/inspect inspect CO_ITERABLE_COROUTINE}@anchor{3c43}
@deffn {Data} inspect.CO_ITERABLE_COROUTINE

The flag is used to transform generators into generator-based
coroutines.  Generator objects with this flag can be used in
@code{await} expression, and can @code{yield from} coroutine objects.
See 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} for more details.


New in version 3.5.

@end deffn

@geindex CO_ASYNC_GENERATOR (in module inspect)
@anchor{library/inspect inspect CO_ASYNC_GENERATOR}@anchor{3c44}
@deffn {Data} inspect.CO_ASYNC_GENERATOR

The flag is set when the code object is an asynchronous generator
function.  When the code object is executed it returns an
asynchronous generator object.  See 
@geindex Python Enhancement Proposals; PEP 525
PEP 525@footnote{https://www.python.org/dev/peps/pep-0525} for more details.


New in version 3.6.

@end deffn

@cartouche
@quotation Note 
The flags are specific to CPython, and may not be defined in other
Python implementations.  Furthermore, the flags are an implementation
detail, and can be removed or deprecated in future Python releases.
It’s recommended to use public APIs from the @ref{9c,,inspect} module
for any introspection needs.
@end quotation
@end cartouche

@node Command Line Interface<3>,,Code Objects Bit Flags,inspect — Inspect live objects
@anchor{library/inspect command-line-interface}@anchor{3c45}@anchor{library/inspect inspect-module-cli}@anchor{ad9}
@subsubsection Command Line Interface


The @ref{9c,,inspect} module also provides a basic introspection capability
from the command line.

By default, accepts the name of a module and prints the source of that
module. A class or function within the module can be printed instead by
appended a colon and the qualified name of the target object.

@geindex inspect command line option; --details
@anchor{library/inspect cmdoption-inspect-details}@anchor{3c46}
@deffn {Program Option} @w{-}@w{-}details

Print information about the specified object rather than the source code
@end deffn

@node site — Site-specific configuration hook,,inspect — Inspect live objects,Python Runtime Services
@anchor{library/site doc}@anchor{3c47}@anchor{library/site module-site}@anchor{e6}@anchor{library/site site-site-specific-configuration-hook}@anchor{3c48}
@subsection @code{site} — Site-specific configuration hook


@geindex module; site

@strong{Source code:} Lib/site.py@footnote{https://github.com/python/cpython/tree/main/Lib/site.py}


__________________________________________________________________


@strong{This module is automatically imported during initialization.} The automatic
import can be suppressed using the interpreter’s @ref{d4e,,-S} option.

@geindex module; search; path

Importing this module will append site-specific paths to the module search path
and add a few builtins, unless @ref{d4e,,-S} was used.  In that case, this module
can be safely imported with no automatic modifications to the module search path
or additions to the builtins.  To explicitly trigger the usual site-specific
additions, call the @ref{1610,,site.main()} function.


Changed in version 3.3: Importing the module used to trigger paths manipulation even when using
@ref{d4e,,-S}.


@geindex site-packages; directory

It starts by constructing up to four directories from a head and a tail part.
For the head part, it uses @code{sys.prefix} and @code{sys.exec_prefix}; empty heads
are skipped.  For the tail part, it uses the empty string and then
@code{lib/site-packages} (on Windows) or
@code{lib/python@emph{X.Y}/site-packages} (on Unix and macOS).  For each
of the distinct head-tail combinations, it sees if it refers to an existing
directory, and if so, adds it to @code{sys.path} and also inspects the newly
added path for configuration files.


Changed in version 3.5: Support for the “site-python” directory has been removed.


If a file named “pyvenv.cfg” exists one directory above sys.executable,
sys.prefix and sys.exec_prefix are set to that directory and
it is also checked for site-packages (sys.base_prefix and
sys.base_exec_prefix will always be the “real” prefixes of the Python
installation). If “pyvenv.cfg” (a bootstrap configuration file) contains
the key “include-system-site-packages” set to anything other than “true”
(case-insensitive), the system-level prefixes will not be
searched for site-packages; otherwise they will.

@geindex # (hash); comment
@geindex statement; import

A path configuration file is a file whose name has the form @code{@emph{name}.pth}
and exists in one of the four directories mentioned above; its contents are
additional items (one per line) to be added to @code{sys.path}.  Non-existing items
are never added to @code{sys.path}, and no check is made that the item refers to a
directory rather than a file.  No item is added to @code{sys.path} more than
once.  Blank lines and lines beginning with @code{#} are skipped.  Lines starting
with @code{import} (followed by space or tab) are executed.

@cartouche
@quotation Note 
An executable line in a @code{.pth} file is run at every Python startup,
regardless of whether a particular module is actually going to be used.
Its impact should thus be kept to a minimum.
The primary intended purpose of executable lines is to make the
corresponding module(s) importable
(load 3rd-party import hooks, adjust 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} etc).
Any other initialization is supposed to be done upon a module’s
actual import, if and when it happens.
Limiting a code chunk to a single line is a deliberate measure
to discourage putting anything more complex here.
@end quotation
@end cartouche

@geindex package
@geindex path; configuration; file

For example, suppose @code{sys.prefix} and @code{sys.exec_prefix} are set to
@code{/usr/local}.  The Python X.Y library is then installed in
@code{/usr/local/lib/python@emph{X.Y}}.  Suppose this has
a subdirectory @code{/usr/local/lib/python@emph{X.Y}/site-packages} with three
subsubdirectories, @code{foo}, @code{bar} and @code{spam}, and two path
configuration files, @code{foo.pth} and @code{bar.pth}.  Assume
@code{foo.pth} contains the following:

@example
# foo package configuration

foo
bar
bletch
@end example

and @code{bar.pth} contains:

@example
# bar package configuration

bar
@end example

Then the following version-specific directories are added to
@code{sys.path}, in this order:

@example
/usr/local/lib/pythonX.Y/site-packages/bar
/usr/local/lib/pythonX.Y/site-packages/foo
@end example

Note that @code{bletch} is omitted because it doesn’t exist; the @code{bar}
directory precedes the @code{foo} directory because @code{bar.pth} comes
alphabetically before @code{foo.pth}; and @code{spam} is omitted because it is
not mentioned in either path configuration file.

@geindex module; sitecustomize

After these path manipulations, an attempt is made to import a module named
@code{sitecustomize}, which can perform arbitrary site-specific customizations.
It is typically created by a system administrator in the site-packages
directory.  If this import fails with an @ref{3b8,,ImportError} or its subclass
exception, and the exception’s @code{name} attribute equals to @code{'sitecustomize'},
it is silently ignored.  If Python is started without output streams available, as
with @code{pythonw.exe} on Windows (which is used by default to start IDLE),
attempted output from @code{sitecustomize} is ignored.  Any other exception
causes a silent and perhaps mysterious failure of the process.

@geindex module; usercustomize

After this, an attempt is made to import a module named @code{usercustomize},
which can perform arbitrary user-specific customizations, if
@ref{3c49,,ENABLE_USER_SITE} is true.  This file is intended to be created in the
user site-packages directory (see below), which is part of @code{sys.path} unless
disabled by @ref{f15,,-s}.  If this import fails with an @ref{3b8,,ImportError} or
its subclass exception, and the exception’s @code{name} attribute equals to
@code{'usercustomize'}, it is silently ignored.

Note that for some non-Unix systems, @code{sys.prefix} and @code{sys.exec_prefix} are
empty, and the path manipulations are skipped; however the import of
@code{sitecustomize} and @code{usercustomize} is still attempted.

@menu
* Readline configuration:: 
* Module contents: Module contents<5>. 
* Command Line Interface: Command Line Interface<4>. 

@end menu

@node Readline configuration,Module contents<5>,,site — Site-specific configuration hook
@anchor{library/site readline-configuration}@anchor{3c4a}@anchor{library/site rlcompleter-config}@anchor{b4e}
@subsubsection Readline configuration


On systems that support @ref{d9,,readline}, this module will also import and
configure the @ref{dc,,rlcompleter} module, if Python is started in
@ref{b4b,,interactive mode} and without the @ref{d4e,,-S} option.
The default behavior is enable tab-completion and to use
@code{~/.python_history} as the history save file.  To disable it, delete (or
override) the @ref{b4c,,sys.__interactivehook__} attribute in your
@code{sitecustomize} or @code{usercustomize} module or your
@geindex PYTHONSTARTUP
@geindex environment variable; PYTHONSTARTUP
@ref{b4d,,PYTHONSTARTUP} file.


Changed in version 3.4: Activation of rlcompleter and history was made automatic.


@node Module contents<5>,Command Line Interface<4>,Readline configuration,site — Site-specific configuration hook
@anchor{library/site module-contents}@anchor{3c4b}
@subsubsection Module contents


@geindex PREFIXES (in module site)
@anchor{library/site site PREFIXES}@anchor{3c4c}
@deffn {Data} site.PREFIXES

A list of prefixes for site-packages directories.
@end deffn

@geindex ENABLE_USER_SITE (in module site)
@anchor{library/site site ENABLE_USER_SITE}@anchor{3c49}
@deffn {Data} site.ENABLE_USER_SITE

Flag showing the status of the user site-packages directory.  @code{True} means
that it is enabled and was added to @code{sys.path}.  @code{False} means that it
was disabled by user request (with @ref{f15,,-s} or
@geindex PYTHONNOUSERSITE
@geindex environment variable; PYTHONNOUSERSITE
@ref{f16,,PYTHONNOUSERSITE}).  @code{None} means it was disabled for security
reasons (mismatch between user or group id and effective id) or by an
administrator.
@end deffn

@geindex USER_SITE (in module site)
@anchor{library/site site USER_SITE}@anchor{160f}
@deffn {Data} site.USER_SITE

Path to the user site-packages for the running Python.  Can be @code{None} if
@ref{df2,,getusersitepackages()} hasn’t been called yet.  Default value is
@code{~/.local/lib/python@emph{X.Y}/site-packages} for UNIX and non-framework
macOS builds, @code{~/Library/Python/@emph{X.Y}/lib/python/site-packages} for macOS
framework builds, and @code{@emph{%APPDATA%}\Python\Python@emph{XY}\site-packages}
on Windows.  This directory is a site directory, which means that
@code{.pth} files in it will be processed.
@end deffn

@geindex USER_BASE (in module site)
@anchor{library/site site USER_BASE}@anchor{1620}
@deffn {Data} site.USER_BASE

Path to the base directory for the user site-packages.  Can be @code{None} if
@ref{df1,,getuserbase()} hasn’t been called yet.  Default value is
@code{~/.local} for UNIX and macOS non-framework builds,
@code{~/Library/Python/@emph{X.Y}} for macOS framework builds, and
@code{@emph{%APPDATA%}\Python} for Windows.  This value is used by Distutils to
compute the installation directories for scripts, data files, Python modules,
etc. for the @ref{1621,,user installation scheme}.
See also 
@geindex PYTHONUSERBASE
@geindex environment variable; PYTHONUSERBASE
@ref{f14,,PYTHONUSERBASE}.
@end deffn

@geindex main() (in module site)
@anchor{library/site site main}@anchor{1610}
@deffn {Function} site.main ()

Adds all the standard site-specific directories to the module search
path.  This function is called automatically when this module is imported,
unless the Python interpreter was started with the @ref{d4e,,-S} flag.


Changed in version 3.3: This function used to be called unconditionally.

@end deffn

@geindex addsitedir() (in module site)
@anchor{library/site site addsitedir}@anchor{3c4d}
@deffn {Function} site.addsitedir (sitedir, known_paths=None)

Add a directory to sys.path and process its @code{.pth} files.  Typically
used in @code{sitecustomize} or @code{usercustomize} (see above).
@end deffn

@geindex getsitepackages() (in module site)
@anchor{library/site site getsitepackages}@anchor{df0}
@deffn {Function} site.getsitepackages ()

Return a list containing all global site-packages directories.


New in version 3.2.

@end deffn

@geindex getuserbase() (in module site)
@anchor{library/site site getuserbase}@anchor{df1}
@deffn {Function} site.getuserbase ()

Return the path of the user base directory, @ref{1620,,USER_BASE}.  If it is not
initialized yet, this function will also set it, respecting
@geindex PYTHONUSERBASE
@geindex environment variable; PYTHONUSERBASE
@ref{f14,,PYTHONUSERBASE}.


New in version 3.2.

@end deffn

@geindex getusersitepackages() (in module site)
@anchor{library/site site getusersitepackages}@anchor{df2}
@deffn {Function} site.getusersitepackages ()

Return the path of the user-specific site-packages directory,
@ref{160f,,USER_SITE}.  If it is not initialized yet, this function will also set
it, respecting @ref{1620,,USER_BASE}.  To determine if the user-specific
site-packages was added to @code{sys.path} @ref{3c49,,ENABLE_USER_SITE} should be
used.


New in version 3.2.

@end deffn

@node Command Line Interface<4>,,Module contents<5>,site — Site-specific configuration hook
@anchor{library/site command-line-interface}@anchor{3c4e}@anchor{library/site site-commandline}@anchor{3c4f}
@subsubsection Command Line Interface


The @ref{e6,,site} module also provides a way to get the user directories from the
command line:

@example
$ python3 -m site --user-site
/home/user/.local/lib/python3.3/site-packages
@end example

If it is called without arguments, it will print the contents of
@ref{489,,sys.path} on the standard output, followed by the value of
@ref{1620,,USER_BASE} and whether the directory exists, then the same thing for
@ref{160f,,USER_SITE}, and finally the value of @ref{3c49,,ENABLE_USER_SITE}.

@geindex site command line option; --user-base
@anchor{library/site cmdoption-site-user-base}@anchor{3c50}
@deffn {Program Option} @w{-}@w{-}user@w{-}base

Print the path to the user base directory.
@end deffn

@geindex site command line option; --user-site
@anchor{library/site cmdoption-site-user-site}@anchor{3c51}
@deffn {Program Option} @w{-}@w{-}user@w{-}site

Print the path to the user site-packages directory.
@end deffn

If both options are given, user base and user site will be printed (always in
this order), separated by @ref{161d,,os.pathsep}.

If any option is given, the script will exit with one of these values: @code{0} if
the user site-packages directory is enabled, @code{1} if it was disabled by the
user, @code{2} if it is disabled for security reasons or by an administrator, and a
value greater than 2 if there is an error.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 370
PEP 370@footnote{https://www.python.org/dev/peps/pep-0370} – Per user site-packages directory


@node Custom Python Interpreters,Importing Modules,Python Runtime Services,The Python Standard Library
@anchor{library/custominterp doc}@anchor{3c52}@anchor{library/custominterp custom-python-interpreters}@anchor{3c53}@anchor{library/custominterp custominterp}@anchor{3c54}
@section Custom Python Interpreters


The modules described in this chapter allow writing interfaces similar to
Python’s interactive interpreter.  If you want a Python interpreter that
supports some special feature in addition to the Python language, you should
look at the @ref{19,,code} module.  (The @ref{1b,,codeop} module is lower-level, used
to support compiling a possibly-incomplete chunk of Python code.)

The full list of modules described in this chapter is:

@menu
* code — Interpreter base classes:: 
* codeop — Compile Python code:: 

@end menu

@node code — Interpreter base classes,codeop — Compile Python code,,Custom Python Interpreters
@anchor{library/code doc}@anchor{3c55}@anchor{library/code code-interpreter-base-classes}@anchor{3c56}@anchor{library/code module-code}@anchor{19}
@subsection @code{code} — Interpreter base classes


@geindex module; code

@strong{Source code:} Lib/code.py@footnote{https://github.com/python/cpython/tree/main/Lib/code.py}


__________________________________________________________________


The @code{code} module provides facilities to implement read-eval-print loops in
Python.  Two classes and convenience functions are included which can be used to
build applications which provide an interactive interpreter prompt.

@geindex InteractiveInterpreter (class in code)
@anchor{library/code code InteractiveInterpreter}@anchor{3c57}
@deffn {Class} code.InteractiveInterpreter (locals=None)

This class deals with parsing and interpreter state (the user’s namespace); it
does not deal with input buffering or prompting or input file naming (the
filename is always passed in explicitly). The optional @emph{locals} argument
specifies the dictionary in which code will be executed; it defaults to a newly
created dictionary with key @code{'__name__'} set to @code{'__console__'} and key
@code{'__doc__'} set to @code{None}.
@end deffn

@geindex InteractiveConsole (class in code)
@anchor{library/code code InteractiveConsole}@anchor{1c83}
@deffn {Class} code.InteractiveConsole (locals=None, filename='<console>')

Closely emulate the behavior of the interactive Python interpreter. This class
builds on @ref{3c57,,InteractiveInterpreter} and adds prompting using the familiar
@code{sys.ps1} and @code{sys.ps2}, and input buffering.
@end deffn

@geindex interact() (in module code)
@anchor{library/code code interact}@anchor{3c58}
@deffn {Function} code.interact (banner=None, readfunc=None, local=None, exitmsg=None)

Convenience function to run a read-eval-print loop.  This creates a new
instance of @ref{1c83,,InteractiveConsole} and sets @emph{readfunc} to be used as
the @ref{3c59,,InteractiveConsole.raw_input()} method, if provided.  If @emph{local} is
provided, it is passed to the @ref{1c83,,InteractiveConsole} constructor for
use as the default namespace for the interpreter loop.  The @ref{3c58,,interact()}
method of the instance is then run with @emph{banner} and @emph{exitmsg} passed as the
banner and exit message to use, if provided.  The console object is discarded
after use.


Changed in version 3.6: Added @emph{exitmsg} parameter.

@end deffn

@geindex compile_command() (in module code)
@anchor{library/code code compile_command}@anchor{3c5a}
@deffn {Function} code.compile_command (source, filename='<input>', symbol='single')

This function is useful for programs that want to emulate Python’s interpreter
main loop (a.k.a. the read-eval-print loop).  The tricky part is to determine
when the user has entered an incomplete command that can be completed by
entering more text (as opposed to a complete command or a syntax error).  This
function @emph{almost} always makes the same decision as the real interpreter main
loop.

@emph{source} is the source string; @emph{filename} is the optional filename from which
source was read, defaulting to @code{'<input>'}; and @emph{symbol} is the optional
grammar start symbol, which should be @code{'single'} (the default), @code{'eval'}
or @code{'exec'}.

Returns a code object (the same as @code{compile(source, filename, symbol)}) if the
command is complete and valid; @code{None} if the command is incomplete; raises
@ref{21a,,SyntaxError} if the command is complete and contains a syntax error, or
raises @ref{347,,OverflowError} or @ref{17c,,ValueError} if the command contains an
invalid literal.
@end deffn

@menu
* Interactive Interpreter Objects:: 
* Interactive Console Objects:: 

@end menu

@node Interactive Interpreter Objects,Interactive Console Objects,,code — Interpreter base classes
@anchor{library/code interactive-interpreter-objects}@anchor{3c5b}@anchor{library/code interpreter-objects}@anchor{3c5c}
@subsubsection Interactive Interpreter Objects


@geindex runsource() (code.InteractiveInterpreter method)
@anchor{library/code code InteractiveInterpreter runsource}@anchor{3c5d}
@deffn {Method} InteractiveInterpreter.runsource (source, filename='<input>', symbol='single')

Compile and run some source in the interpreter. Arguments are the same as for
@ref{3c5a,,compile_command()}; the default for @emph{filename} is @code{'<input>'}, and for
@emph{symbol} is @code{'single'}.  One of several things can happen:


@itemize *

@item 
The input is incorrect; @ref{3c5a,,compile_command()} raised an exception
(@ref{21a,,SyntaxError} or @ref{347,,OverflowError}).  A syntax traceback will be
printed by calling the @ref{3c5e,,showsyntaxerror()} method.  @ref{3c5d,,runsource()}
returns @code{False}.

@item 
The input is incomplete, and more input is required; @ref{3c5a,,compile_command()}
returned @code{None}. @ref{3c5d,,runsource()} returns @code{True}.

@item 
The input is complete; @ref{3c5a,,compile_command()} returned a code object.  The
code is executed by calling the @ref{3c5f,,runcode()} (which also handles run-time
exceptions, except for @ref{892,,SystemExit}). @ref{3c5d,,runsource()} returns @code{False}.
@end itemize

The return value can be used to decide whether to use @code{sys.ps1} or @code{sys.ps2}
to prompt the next line.
@end deffn

@geindex runcode() (code.InteractiveInterpreter method)
@anchor{library/code code InteractiveInterpreter runcode}@anchor{3c5f}
@deffn {Method} InteractiveInterpreter.runcode (code)

Execute a code object. When an exception occurs, @ref{92e,,showtraceback()} is called
to display a traceback.  All exceptions are caught except @ref{892,,SystemExit},
which is allowed to propagate.

A note about @ref{4ca,,KeyboardInterrupt}: this exception may occur elsewhere in
this code, and may not always be caught.  The caller should be prepared to deal
with it.
@end deffn

@geindex showsyntaxerror() (code.InteractiveInterpreter method)
@anchor{library/code code InteractiveInterpreter showsyntaxerror}@anchor{3c5e}
@deffn {Method} InteractiveInterpreter.showsyntaxerror (filename=None)

Display the syntax error that just occurred.  This does not display a stack
trace because there isn’t one for syntax errors. If @emph{filename} is given, it is
stuffed into the exception instead of the default filename provided by Python’s
parser, because it always uses @code{'<string>'} when reading from a string. The
output is written by the @ref{3c60,,write()} method.
@end deffn

@geindex showtraceback() (code.InteractiveInterpreter method)
@anchor{library/code code InteractiveInterpreter showtraceback}@anchor{92e}
@deffn {Method} InteractiveInterpreter.showtraceback ()

Display the exception that just occurred.  We remove the first stack item
because it is within the interpreter object implementation. The output is
written by the @ref{3c60,,write()} method.


Changed in version 3.5: The full chained traceback is displayed instead
of just the primary traceback.

@end deffn

@geindex write() (code.InteractiveInterpreter method)
@anchor{library/code code InteractiveInterpreter write}@anchor{3c60}
@deffn {Method} InteractiveInterpreter.write (data)

Write a string to the standard error stream (@code{sys.stderr}). Derived classes
should override this to provide the appropriate output handling as needed.
@end deffn

@node Interactive Console Objects,,Interactive Interpreter Objects,code — Interpreter base classes
@anchor{library/code console-objects}@anchor{3c61}@anchor{library/code interactive-console-objects}@anchor{3c62}
@subsubsection Interactive Console Objects


The @ref{1c83,,InteractiveConsole} class is a subclass of
@ref{3c57,,InteractiveInterpreter}, and so offers all the methods of the
interpreter objects as well as the following additions.

@geindex interact() (code.InteractiveConsole method)
@anchor{library/code code InteractiveConsole interact}@anchor{3c63}
@deffn {Method} InteractiveConsole.interact (banner=None, exitmsg=None)

Closely emulate the interactive Python console. The optional @emph{banner} argument
specify the banner to print before the first interaction; by default it prints a
banner similar to the one printed by the standard Python interpreter, followed
by the class name of the console object in parentheses (so as not to confuse
this with the real interpreter – since it’s so close!).

The optional @emph{exitmsg} argument specifies an exit message printed when exiting.
Pass the empty string to suppress the exit message. If @emph{exitmsg} is not given or
@code{None}, a default message is printed.


Changed in version 3.4: To suppress printing any banner, pass an empty string.



Changed in version 3.6: Print an exit message when exiting.

@end deffn

@geindex push() (code.InteractiveConsole method)
@anchor{library/code code InteractiveConsole push}@anchor{3c64}
@deffn {Method} InteractiveConsole.push (line)

Push a line of source text to the interpreter. The line should not have a
trailing newline; it may have internal newlines.  The line is appended to a
buffer and the interpreter’s @code{runsource()} method is called with the
concatenated contents of the buffer as source.  If this indicates that the
command was executed or invalid, the buffer is reset; otherwise, the command is
incomplete, and the buffer is left as it was after the line was appended.  The
return value is @code{True} if more input is required, @code{False} if the line was
dealt with in some way (this is the same as @code{runsource()}).
@end deffn

@geindex resetbuffer() (code.InteractiveConsole method)
@anchor{library/code code InteractiveConsole resetbuffer}@anchor{3c65}
@deffn {Method} InteractiveConsole.resetbuffer ()

Remove any unhandled source text from the input buffer.
@end deffn

@geindex raw_input() (code.InteractiveConsole method)
@anchor{library/code code InteractiveConsole raw_input}@anchor{3c59}
@deffn {Method} InteractiveConsole.raw_input (prompt='@w{'})

Write a prompt and read a line.  The returned line does not include the trailing
newline.  When the user enters the EOF key sequence, @ref{e74,,EOFError} is raised.
The base implementation reads from @code{sys.stdin}; a subclass may replace this
with a different implementation.
@end deffn

@node codeop — Compile Python code,,code — Interpreter base classes,Custom Python Interpreters
@anchor{library/codeop doc}@anchor{3c66}@anchor{library/codeop codeop-compile-python-code}@anchor{3c67}@anchor{library/codeop module-codeop}@anchor{1b}
@subsection @code{codeop} — Compile Python code


@geindex module; codeop

@strong{Source code:} Lib/codeop.py@footnote{https://github.com/python/cpython/tree/main/Lib/codeop.py}


__________________________________________________________________


The @ref{1b,,codeop} module provides utilities upon which the Python
read-eval-print loop can be emulated, as is done in the @ref{19,,code} module.  As
a result, you probably don’t want to use the module directly; if you want to
include such a loop in your program you probably want to use the @ref{19,,code}
module instead.

There are two parts to this job:


@enumerate 

@item 
Being able to tell if a line of input completes a Python  statement: in
short, telling whether to print ‘@code{>>>}’ or ‘@code{...}’ next.

@item 
Remembering which future statements the user has entered, so  subsequent
input can be compiled with these in effect.
@end enumerate

The @ref{1b,,codeop} module provides a way of doing each of these things, and a way
of doing them both.

To do just the former:

@geindex compile_command() (in module codeop)
@anchor{library/codeop codeop compile_command}@anchor{11e3}
@deffn {Function} codeop.compile_command (source, filename='<input>', symbol='single')

Tries to compile @emph{source}, which should be a string of Python code and return a
code object if @emph{source} is valid Python code. In that case, the filename
attribute of the code object will be @emph{filename}, which defaults to
@code{'<input>'}. Returns @code{None} if @emph{source} is @emph{not} valid Python code, but is a
prefix of valid Python code.

If there is a problem with @emph{source}, an exception will be raised.
@ref{21a,,SyntaxError} is raised if there is invalid Python syntax, and
@ref{347,,OverflowError} or @ref{17c,,ValueError} if there is an invalid literal.

The @emph{symbol} argument determines whether @emph{source} is compiled as a statement
(@code{'single'}, the default), as a sequence of statements (@code{'exec'}) or
as an @ref{17d2,,expression} (@code{'eval'}).  Any other value will
cause @ref{17c,,ValueError} to  be raised.

@cartouche
@quotation Note 
It is possible (but not likely) that the parser stops parsing with a
successful outcome before reaching the end of the source; in this case,
trailing symbols may be ignored instead of causing an error.  For example,
a backslash followed by two newlines may be followed by arbitrary garbage.
This will be fixed once the API for the parser is better.
@end quotation
@end cartouche
@end deffn

@geindex Compile (class in codeop)
@anchor{library/codeop codeop Compile}@anchor{3c68}
@deffn {Class} codeop.Compile

Instances of this class have @code{__call__()} methods identical in signature to
the built-in function @ref{4e2,,compile()}, but with the difference that if the
instance compiles program text containing a @ref{0,,__future__} statement, the
instance ‘remembers’ and compiles all subsequent program texts with the
statement in force.
@end deffn

@geindex CommandCompiler (class in codeop)
@anchor{library/codeop codeop CommandCompiler}@anchor{3c69}
@deffn {Class} codeop.CommandCompiler

Instances of this class have @code{__call__()} methods identical in signature to
@ref{11e3,,compile_command()}; the difference is that if the instance compiles program
text containing a @code{__future__} statement, the instance ‘remembers’ and
compiles all subsequent program texts with the statement in force.
@end deffn

@node Importing Modules,Python Language Services,Custom Python Interpreters,The Python Standard Library
@anchor{library/modules doc}@anchor{3c6a}@anchor{library/modules importing-modules}@anchor{3c6b}@anchor{library/modules modules}@anchor{3c6c}
@section Importing Modules


The modules described in this chapter provide new ways to import other Python
modules and hooks for customizing the import process.

The full list of modules described in this chapter is:

@menu
* zipimport — Import modules from Zip archives:: 
* pkgutil — Package extension utility:: 
* modulefinder — Find modules used by a script:: 
* runpy — Locating and executing Python modules:: 
* importlib — The implementation of import:: 
* importlib.resources – Resources: importlib resources – Resources. 
* Using importlib.metadata: Using importlib metadata. 

@end menu

@node zipimport — Import modules from Zip archives,pkgutil — Package extension utility,,Importing Modules
@anchor{library/zipimport doc}@anchor{3c6d}@anchor{library/zipimport module-zipimport}@anchor{14a}@anchor{library/zipimport zipimport-import-modules-from-zip-archives}@anchor{3c6e}
@subsection @code{zipimport} — Import modules from Zip archives


@geindex module; zipimport

@strong{Source code:} Lib/zipimport.py@footnote{https://github.com/python/cpython/tree/main/Lib/zipimport.py}


__________________________________________________________________


This module adds the ability to import Python modules (@code{*.py},
@code{*.pyc}) and packages from ZIP-format archives. It is usually not
needed to use the @ref{14a,,zipimport} module explicitly; it is automatically used
by the built-in @ref{e35,,import} mechanism for @ref{489,,sys.path} items that are paths
to ZIP archives.

Typically, @ref{489,,sys.path} is a list of directory names as strings.  This module
also allows an item of @ref{489,,sys.path} to be a string naming a ZIP file archive.
The ZIP archive can contain a subdirectory structure to support package imports,
and a path within the archive can be specified to only import from a
subdirectory.  For example, the path @code{example.zip/lib/} would only
import from the @code{lib/} subdirectory within the archive.

Any files may be present in the ZIP archive, but importers are only invoked for
@code{.py} and @code{.pyc} files.  ZIP import of dynamic modules
(@code{.pyd}, @code{.so}) is disallowed. Note that if an archive only contains
@code{.py} files, Python will not attempt to modify the archive by adding the
corresponding @code{.pyc} file, meaning that if a ZIP archive
doesn’t contain @code{.pyc} files, importing may be rather slow.


Changed in version 3.8: Previously, ZIP archives with an archive comment were not supported.



@subsubheading See also



@table @asis

@item PKZIP Application Note@footnote{https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT}

Documentation on the ZIP file format by Phil Katz, the creator of the format and
algorithms used.

@geindex Python Enhancement Proposals; PEP 273

@item PEP 273@footnote{https://www.python.org/dev/peps/pep-0273} - Import Modules from Zip Archives

Written by James C. Ahlstrom, who also provided an implementation. Python 2.3
follows the specification in 
@geindex Python Enhancement Proposals; PEP 273
PEP 273@footnote{https://www.python.org/dev/peps/pep-0273}, but uses an implementation written by Just
van Rossum that uses the import hooks described in 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302}.

@item @ref{96,,importlib} - The implementation of the import machinery

Package providing the relevant protocols for all importers to
implement.
@end table


This module defines an exception:

@geindex ZipImportError
@anchor{library/zipimport zipimport ZipImportError}@anchor{3c6f}
@deffn {Exception} zipimport.ZipImportError

Exception raised by zipimporter objects. It’s a subclass of @ref{3b8,,ImportError},
so it can be caught as @ref{3b8,,ImportError}, too.
@end deffn

@menu
* zipimporter Objects:: 
* Examples: Examples<31>. 

@end menu

@node zipimporter Objects,Examples<31>,,zipimport — Import modules from Zip archives
@anchor{library/zipimport id1}@anchor{3c70}@anchor{library/zipimport zipimporter-objects}@anchor{3c71}
@subsubsection zipimporter Objects


@ref{3c72,,zipimporter} is the class for importing ZIP files.

@geindex zipimporter (class in zipimport)
@anchor{library/zipimport zipimport zipimporter}@anchor{3c72}
@deffn {Class} zipimport.zipimporter (archivepath)

Create a new zipimporter instance. @emph{archivepath} must be a path to a ZIP
file, or to a specific path within a ZIP file.  For example, an @emph{archivepath}
of @code{foo/bar.zip/lib} will look for modules in the @code{lib} directory
inside the ZIP file @code{foo/bar.zip} (provided that it exists).

@ref{3c6f,,ZipImportError} is raised if @emph{archivepath} doesn’t point to a valid ZIP
archive.

@geindex create_module() (zipimport.zipimporter method)
@anchor{library/zipimport zipimport zipimporter create_module}@anchor{2eb}
@deffn {Method} create_module (spec)

Implementation of @ref{7f6,,importlib.abc.Loader.create_module()} that returns
@ref{182,,None} to explicitly request the default semantics.


New in version 3.10.

@end deffn

@geindex exec_module() (zipimport.zipimporter method)
@anchor{library/zipimport zipimport zipimporter exec_module}@anchor{2ec}
@deffn {Method} exec_module (module)

Implementation of @ref{302,,importlib.abc.Loader.exec_module()}.


New in version 3.10.

@end deffn

@geindex find_loader() (zipimport.zipimporter method)
@anchor{library/zipimport zipimport zipimporter find_loader}@anchor{3c73}
@deffn {Method} find_loader (fullname, path=None)

An implementation of @ref{2ff,,importlib.abc.PathEntryFinder.find_loader()}.


Deprecated since version 3.10: Use @ref{2ea,,find_spec()} instead.

@end deffn

@geindex find_module() (zipimport.zipimporter method)
@anchor{library/zipimport zipimport zipimporter find_module}@anchor{3c74}
@deffn {Method} find_module (fullname, path=None)

Search for a module specified by @emph{fullname}. @emph{fullname} must be the fully
qualified (dotted) module name. It returns the zipimporter instance itself
if the module was found, or @ref{182,,None} if it wasn’t. The optional
@emph{path} argument is ignored—it’s there for compatibility with the
importer protocol.


Deprecated since version 3.10: Use @ref{2ea,,find_spec()} instead.

@end deffn

@geindex find_spec() (zipimport.zipimporter method)
@anchor{library/zipimport zipimport zipimporter find_spec}@anchor{2ea}
@deffn {Method} find_spec (fullname, target=None)

An implementation of @ref{309,,importlib.abc.PathEntryFinder.find_spec()}.


New in version 3.10.

@end deffn

@geindex get_code() (zipimport.zipimporter method)
@anchor{library/zipimport zipimport zipimporter get_code}@anchor{3c75}
@deffn {Method} get_code (fullname)

Return the code object for the specified module. Raise
@ref{3c6f,,ZipImportError} if the module couldn’t be imported.
@end deffn

@geindex get_data() (zipimport.zipimporter method)
@anchor{library/zipimport zipimport zipimporter get_data}@anchor{137b}
@deffn {Method} get_data (pathname)

Return the data associated with @emph{pathname}. Raise @ref{2a1,,OSError} if the
file wasn’t found.


Changed in version 3.3: @ref{be8,,IOError} used to be raised instead of @ref{2a1,,OSError}.

@end deffn

@geindex get_filename() (zipimport.zipimporter method)
@anchor{library/zipimport zipimport zipimporter get_filename}@anchor{3c76}
@deffn {Method} get_filename (fullname)

Return the value @code{__file__} would be set to if the specified module
was imported. Raise @ref{3c6f,,ZipImportError} if the module couldn’t be
imported.


New in version 3.1.

@end deffn

@geindex get_source() (zipimport.zipimporter method)
@anchor{library/zipimport zipimport zipimporter get_source}@anchor{3c77}
@deffn {Method} get_source (fullname)

Return the source code for the specified module. Raise
@ref{3c6f,,ZipImportError} if the module couldn’t be found, return
@ref{182,,None} if the archive does contain the module, but has no source
for it.
@end deffn

@geindex is_package() (zipimport.zipimporter method)
@anchor{library/zipimport zipimport zipimporter is_package}@anchor{3c78}
@deffn {Method} is_package (fullname)

Return @code{True} if the module specified by @emph{fullname} is a package. Raise
@ref{3c6f,,ZipImportError} if the module couldn’t be found.
@end deffn

@geindex load_module() (zipimport.zipimporter method)
@anchor{library/zipimport zipimport zipimporter load_module}@anchor{3c79}
@deffn {Method} load_module (fullname)

Load the module specified by @emph{fullname}. @emph{fullname} must be the fully
qualified (dotted) module name. Returns the imported module on success,
raises @ref{3c6f,,ZipImportError} on failure.


Deprecated since version 3.10: Use @ref{2ec,,exec_module()} instead.

@end deffn

@geindex invalidate_caches() (zipimport.zipimporter method)
@anchor{library/zipimport zipimport zipimporter invalidate_caches}@anchor{2ed}
@deffn {Method} invalidate_caches ()

Clear out the internal cache of information about files found within
the ZIP archive.


New in version 3.10.

@end deffn

@geindex archive (zipimport.zipimporter attribute)
@anchor{library/zipimport zipimport zipimporter archive}@anchor{3c7a}
@deffn {Attribute} archive

The file name of the importer’s associated ZIP file, without a possible
subpath.
@end deffn

@geindex prefix (zipimport.zipimporter attribute)
@anchor{library/zipimport zipimport zipimporter prefix}@anchor{3c7b}
@deffn {Attribute} prefix

The subpath within the ZIP file where modules are searched.  This is the
empty string for zipimporter objects which point to the root of the ZIP
file.
@end deffn

The @ref{3c7a,,archive} and @ref{3c7b,,prefix} attributes, when combined with a
slash, equal the original @emph{archivepath} argument given to the
@ref{3c72,,zipimporter} constructor.
@end deffn

@node Examples<31>,,zipimporter Objects,zipimport — Import modules from Zip archives
@anchor{library/zipimport examples}@anchor{3c7c}@anchor{library/zipimport zipimport-examples}@anchor{3c7d}
@subsubsection Examples


Here is an example that imports a module from a ZIP archive - note that the
@ref{14a,,zipimport} module is not explicitly used.

@example
$ unzip -l example.zip
Archive:  example.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
     8467  11-26-02 22:30   jwzthreading.py
 --------                   -------
     8467                   1 file
$ ./python
Python 2.3 (#1, Aug 1 2003, 19:54:32)
>>> import sys
>>> sys.path.insert(0, 'example.zip')  # Add .zip file to front of path
>>> import jwzthreading
>>> jwzthreading.__file__
'example.zip/jwzthreading.py'
@end example

@node pkgutil — Package extension utility,modulefinder — Find modules used by a script,zipimport — Import modules from Zip archives,Importing Modules
@anchor{library/pkgutil doc}@anchor{3c7e}@anchor{library/pkgutil module-pkgutil}@anchor{c8}@anchor{library/pkgutil pkgutil-package-extension-utility}@anchor{3c7f}
@subsection @code{pkgutil} — Package extension utility


@geindex module; pkgutil

@strong{Source code:} Lib/pkgutil.py@footnote{https://github.com/python/cpython/tree/main/Lib/pkgutil.py}


__________________________________________________________________


This module provides utilities for the import system, in particular package
support.

@geindex ModuleInfo (class in pkgutil)
@anchor{library/pkgutil pkgutil ModuleInfo}@anchor{8a2}
@deffn {Class} pkgutil.ModuleInfo (module_finder, name, ispkg)

A namedtuple that holds a brief summary of a module’s info.


New in version 3.6.

@end deffn

@geindex extend_path() (in module pkgutil)
@anchor{library/pkgutil pkgutil extend_path}@anchor{3c80}
@deffn {Function} pkgutil.extend_path (path, name)

Extend the search path for the modules which comprise a package.  Intended
use is to place the following code in a package’s @code{__init__.py}:

@example
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
@end example

This will add to the package’s @code{__path__} all subdirectories of directories
on @code{sys.path} named after the package.  This is useful if one wants to
distribute different parts of a single logical package as multiple
directories.

It also looks for @code{*.pkg} files beginning where @code{*} matches the
@emph{name} argument.  This feature is similar to @code{*.pth} files (see the
@ref{e6,,site} module for more information), except that it doesn’t special-case
lines starting with @code{import}.  A @code{*.pkg} file is trusted at face
value: apart from checking for duplicates, all entries found in a
@code{*.pkg} file are added to the path, regardless of whether they exist
on the filesystem.  (This is a feature.)

If the input path is not a list (as is the case for frozen packages) it is
returned unchanged.  The input path is not modified; an extended copy is
returned.  Items are only appended to the copy at the end.

It is assumed that @ref{489,,sys.path} is a sequence.  Items of @ref{489,,sys.path}
that are not strings referring to existing directories are ignored. Unicode
items on @ref{489,,sys.path} that cause errors when used as filenames may cause
this function to raise an exception (in line with @ref{51c,,os.path.isdir()}
behavior).
@end deffn

@geindex ImpImporter (class in pkgutil)
@anchor{library/pkgutil pkgutil ImpImporter}@anchor{312}
@deffn {Class} pkgutil.ImpImporter (dirname=None)

@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} Finder that wraps Python’s “classic” import algorithm.

If @emph{dirname} is a string, a 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} finder is created that searches that
directory.  If @emph{dirname} is @code{None}, a 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} finder is created that
searches the current @ref{489,,sys.path}, plus any modules that are frozen or
built-in.

Note that @ref{312,,ImpImporter} does not currently support being used by
placement on @ref{87c,,sys.meta_path}.


Deprecated since version 3.3: This emulation is no longer needed, as the standard import mechanism
is now fully 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} compliant and available in @ref{96,,importlib}.

@end deffn

@geindex ImpLoader (class in pkgutil)
@anchor{library/pkgutil pkgutil ImpLoader}@anchor{313}
@deffn {Class} pkgutil.ImpLoader (fullname, file, filename, etc)

@ref{1848,,Loader} that wraps Python’s “classic” import algorithm.


Deprecated since version 3.3: This emulation is no longer needed, as the standard import mechanism
is now fully 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} compliant and available in @ref{96,,importlib}.

@end deffn

@geindex find_loader() (in module pkgutil)
@anchor{library/pkgutil pkgutil find_loader}@anchor{3c81}
@deffn {Function} pkgutil.find_loader (fullname)

Retrieve a module @ref{1848,,loader} for the given @emph{fullname}.

This is a backwards compatibility wrapper around
@ref{b98,,importlib.util.find_spec()} that converts most failures to
@ref{3b8,,ImportError} and only returns the loader rather than the full
@code{ModuleSpec}.


Changed in version 3.3: Updated to be based directly on @ref{96,,importlib} rather than relying
on the package internal 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} import emulation.



Changed in version 3.4: Updated to be based on 
@geindex Python Enhancement Proposals; PEP 451
PEP 451@footnote{https://www.python.org/dev/peps/pep-0451}

@end deffn

@geindex get_importer() (in module pkgutil)
@anchor{library/pkgutil pkgutil get_importer}@anchor{3c82}
@deffn {Function} pkgutil.get_importer (path_item)

Retrieve a @ref{1847,,finder} for the given @emph{path_item}.

The returned finder is cached in @ref{74c,,sys.path_importer_cache} if it was
newly created by a path hook.

The cache (or part of it) can be cleared manually if a rescan of
@ref{bb5,,sys.path_hooks} is necessary.


Changed in version 3.3: Updated to be based directly on @ref{96,,importlib} rather than relying
on the package internal 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} import emulation.

@end deffn

@geindex get_loader() (in module pkgutil)
@anchor{library/pkgutil pkgutil get_loader}@anchor{3c83}
@deffn {Function} pkgutil.get_loader (module_or_name)

Get a @ref{1848,,loader} object for @emph{module_or_name}.

If the module or package is accessible via the normal import mechanism, a
wrapper around the relevant part of that machinery is returned.  Returns
@code{None} if the module cannot be found or imported.  If the named module is
not already imported, its containing package (if any) is imported, in order
to establish the package @code{__path__}.


Changed in version 3.3: Updated to be based directly on @ref{96,,importlib} rather than relying
on the package internal 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} import emulation.



Changed in version 3.4: Updated to be based on 
@geindex Python Enhancement Proposals; PEP 451
PEP 451@footnote{https://www.python.org/dev/peps/pep-0451}

@end deffn

@geindex iter_importers() (in module pkgutil)
@anchor{library/pkgutil pkgutil iter_importers}@anchor{d3f}
@deffn {Function} pkgutil.iter_importers (fullname='@w{'})

Yield @ref{1847,,finder} objects for the given module name.

If fullname contains a ‘.’, the finders will be for the package
containing fullname, otherwise they will be all registered top level
finders (i.e. those on both sys.meta_path and sys.path_hooks).

If the named module is in a package, that package is imported as a side
effect of invoking this function.

If no module name is specified, all top level finders are produced.


Changed in version 3.3: Updated to be based directly on @ref{96,,importlib} rather than relying
on the package internal 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} import emulation.

@end deffn

@geindex iter_modules() (in module pkgutil)
@anchor{library/pkgutil pkgutil iter_modules}@anchor{8a1}
@deffn {Function} pkgutil.iter_modules (path=None, prefix='@w{'})

Yields @ref{8a2,,ModuleInfo} for all submodules on @emph{path}, or, if
@emph{path} is @code{None}, all top-level modules on @code{sys.path}.

@emph{path} should be either @code{None} or a list of paths to look for modules in.

@emph{prefix} is a string to output on the front of every module name on output.

@cartouche
@quotation Note 
Only works for a @ref{1847,,finder} which defines an @code{iter_modules()}
method. This interface is non-standard, so the module also provides
implementations for @ref{c07,,importlib.machinery.FileFinder} and
@ref{3c72,,zipimport.zipimporter}.
@end quotation
@end cartouche


Changed in version 3.3: Updated to be based directly on @ref{96,,importlib} rather than relying
on the package internal 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} import emulation.

@end deffn

@geindex walk_packages() (in module pkgutil)
@anchor{library/pkgutil pkgutil walk_packages}@anchor{73a}
@deffn {Function} pkgutil.walk_packages (path=None, prefix='@w{'}, onerror=None)

Yields @ref{8a2,,ModuleInfo} for all modules recursively on
@emph{path}, or, if @emph{path} is @code{None}, all accessible modules.

@emph{path} should be either @code{None} or a list of paths to look for modules in.

@emph{prefix} is a string to output on the front of every module name on output.

Note that this function must import all @emph{packages} (@emph{not} all modules!) on
the given @emph{path}, in order to access the @code{__path__} attribute to find
submodules.

@emph{onerror} is a function which gets called with one argument (the name of the
package which was being imported) if any exception occurs while trying to
import a package.  If no @emph{onerror} function is supplied, @ref{3b8,,ImportError}s
are caught and ignored, while all other exceptions are propagated,
terminating the search.

Examples:

@example
# list all modules python can access
walk_packages()

# list all submodules of ctypes
walk_packages(ctypes.__path__, ctypes.__name__ + '.')
@end example

@cartouche
@quotation Note 
Only works for a @ref{1847,,finder} which defines an @code{iter_modules()}
method. This interface is non-standard, so the module also provides
implementations for @ref{c07,,importlib.machinery.FileFinder} and
@ref{3c72,,zipimport.zipimporter}.
@end quotation
@end cartouche


Changed in version 3.3: Updated to be based directly on @ref{96,,importlib} rather than relying
on the package internal 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} import emulation.

@end deffn

@geindex get_data() (in module pkgutil)
@anchor{library/pkgutil pkgutil get_data}@anchor{1268}
@deffn {Function} pkgutil.get_data (package, resource)

Get a resource from a package.

This is a wrapper for the @ref{1848,,loader}
@ref{3c84,,get_data} API.  The
@emph{package} argument should be the name of a package, in standard module format
(@code{foo.bar}).  The @emph{resource} argument should be in the form of a relative
filename, using @code{/} as the path separator.  The parent directory name
@code{..} is not allowed, and nor is a rooted name (starting with a @code{/}).

The function returns a binary string that is the contents of the specified
resource.

For packages located in the filesystem, which have already been imported,
this is the rough equivalent of:

@example
d = os.path.dirname(sys.modules[package].__file__)
data = open(os.path.join(d, resource), 'rb').read()
@end example

If the package cannot be located or loaded, or it uses a @ref{1848,,loader}
which does not support @ref{3c84,,get_data},
then @code{None} is returned.  In particular, the @ref{1848,,loader} for
@ref{1840,,namespace packages} does not support
@ref{3c84,,get_data}.
@end deffn

@geindex resolve_name() (in module pkgutil)
@anchor{library/pkgutil pkgutil resolve_name}@anchor{3c85}
@deffn {Function} pkgutil.resolve_name (name)

Resolve a name to an object.

This functionality is used in numerous places in the standard library (see
bpo-12915@footnote{https://bugs.python.org/issue12915}) - and equivalent functionality is also in widely used
third-party packages such as setuptools, Django and Pyramid.

It is expected that @emph{name} will be a string in one of the following
formats, where W is shorthand for a valid Python identifier and dot stands
for a literal period in these pseudo-regexes:


@itemize *

@item 
@code{W(.W)*}

@item 
@code{W(.W)*:(W(.W)*)?}
@end itemize

The first form is intended for backward compatibility only. It assumes that
some part of the dotted name is a package, and the rest is an object
somewhere within that package, possibly nested inside other objects.
Because the place where the package stops and the object hierarchy starts
can’t be inferred by inspection, repeated attempts to import must be done
with this form.

In the second form, the caller makes the division point clear through the
provision of a single colon: the dotted name to the left of the colon is a
package to be imported, and the dotted name to the right is the object
hierarchy within that package. Only one import is needed in this form. If
it ends with the colon, then a module object is returned.

The function will return an object (which might be a module), or raise one
of the following exceptions:

@ref{17c,,ValueError} – if @emph{name} isn’t in a recognised format.

@ref{3b8,,ImportError} – if an import failed when it shouldn’t have.

@ref{163,,AttributeError} – If a failure occurred when traversing the object
hierarchy within the imported package to get to the desired object.


New in version 3.9.

@end deffn

@node modulefinder — Find modules used by a script,runpy — Locating and executing Python modules,pkgutil — Package extension utility,Importing Modules
@anchor{library/modulefinder doc}@anchor{3c86}@anchor{library/modulefinder module-modulefinder}@anchor{b0}@anchor{library/modulefinder modulefinder-find-modules-used-by-a-script}@anchor{3c87}
@subsection @code{modulefinder} — Find modules used by a script


@geindex module; modulefinder

@strong{Source code:} Lib/modulefinder.py@footnote{https://github.com/python/cpython/tree/main/Lib/modulefinder.py}


__________________________________________________________________


This module provides a @ref{12ae,,ModuleFinder} class that can be used to determine
the set of modules imported by a script. @code{modulefinder.py} can also be run as
a script, giving the filename of a Python script as its argument, after which a
report of the imported modules will be printed.

@geindex AddPackagePath() (in module modulefinder)
@anchor{library/modulefinder modulefinder AddPackagePath}@anchor{3c88}
@deffn {Function} modulefinder.AddPackagePath (pkg_name, path)

Record that the package named @emph{pkg_name} can be found in the specified @emph{path}.
@end deffn

@geindex ReplacePackage() (in module modulefinder)
@anchor{library/modulefinder modulefinder ReplacePackage}@anchor{3c89}
@deffn {Function} modulefinder.ReplacePackage (oldname, newname)

Allows specifying that the module named @emph{oldname} is in fact the package named
@emph{newname}.
@end deffn

@geindex ModuleFinder (class in modulefinder)
@anchor{library/modulefinder modulefinder ModuleFinder}@anchor{12ae}
@deffn {Class} modulefinder.ModuleFinder (path=None, debug=0, excludes=[], replace_paths=[])

This class provides @ref{3c8a,,run_script()} and @ref{3c8b,,report()} methods to determine
the set of modules imported by a script. @emph{path} can be a list of directories to
search for modules; if not specified, @code{sys.path} is used.  @emph{debug} sets the
debugging level; higher values make the class print debugging messages about
what it’s doing. @emph{excludes} is a list of module names to exclude from the
analysis. @emph{replace_paths} is a list of @code{(oldpath, newpath)} tuples that will
be replaced in module paths.

@geindex report() (modulefinder.ModuleFinder method)
@anchor{library/modulefinder modulefinder ModuleFinder report}@anchor{3c8b}
@deffn {Method} report ()

Print a report to standard output that lists the modules imported by the
script and their paths, as well as modules that are missing or seem to be
missing.
@end deffn

@geindex run_script() (modulefinder.ModuleFinder method)
@anchor{library/modulefinder modulefinder ModuleFinder run_script}@anchor{3c8a}
@deffn {Method} run_script (pathname)

Analyze the contents of the @emph{pathname} file, which must contain Python
code.
@end deffn

@geindex modules (modulefinder.ModuleFinder attribute)
@anchor{library/modulefinder modulefinder ModuleFinder modules}@anchor{3c8c}
@deffn {Attribute} modules

A dictionary mapping module names to modules. See
@ref{3c8d,,Example usage of ModuleFinder}.
@end deffn
@end deffn

@menu
* Example usage of ModuleFinder:: 

@end menu

@node Example usage of ModuleFinder,,,modulefinder — Find modules used by a script
@anchor{library/modulefinder example-usage-of-modulefinder}@anchor{3c8e}@anchor{library/modulefinder modulefinder-example}@anchor{3c8d}
@subsubsection Example usage of @code{ModuleFinder}


The script that is going to get analyzed later on (bacon.py):

@example
import re, itertools

try:
    import baconhameggs
except ImportError:
    pass

try:
    import guido.python.ham
except ImportError:
    pass
@end example

The script that will output the report of bacon.py:

@example
from modulefinder import ModuleFinder

finder = ModuleFinder()
finder.run_script('bacon.py')

print('Loaded modules:')
for name, mod in finder.modules.items():
    print('%s: ' % name, end='@w{'})
    print(','.join(list(mod.globalnames.keys())[:3]))

print('-'*50)
print('Modules not imported:')
print('\n'.join(finder.badmodules.keys()))
@end example

Sample output (may vary depending on the architecture):

@example
Loaded modules:
_types:
copyreg:  _inverted_registry,_slotnames,__all__
sre_compile:  isstring,_sre,_optimize_unicode
_sre:
sre_constants:  REPEAT_ONE,makedict,AT_END_LINE
sys:
re:  __module__,finditer,_expand
itertools:
__main__:  re,itertools,baconhameggs
sre_parse:  _PATTERNENDERS,SRE_FLAG_UNICODE
array:
types:  __module__,IntType,TypeType
---------------------------------------------------
Modules not imported:
guido.python.ham
baconhameggs
@end example

@node runpy — Locating and executing Python modules,importlib — The implementation of import,modulefinder — Find modules used by a script,Importing Modules
@anchor{library/runpy doc}@anchor{3c8f}@anchor{library/runpy module-runpy}@anchor{dd}@anchor{library/runpy runpy-locating-and-executing-python-modules}@anchor{3c90}
@subsection @code{runpy} — Locating and executing Python modules


@geindex module; runpy

@strong{Source code:} Lib/runpy.py@footnote{https://github.com/python/cpython/tree/main/Lib/runpy.py}


__________________________________________________________________


The @ref{dd,,runpy} module is used to locate and run Python modules without
importing them first. Its main use is to implement the @ref{611,,-m} command
line switch that allows scripts to be located using the Python module
namespace rather than the filesystem.

Note that this is @emph{not} a sandbox module - all code is executed in the
current process, and any side effects (such as cached imports of other
modules) will remain in place after the functions have returned.

Furthermore, any functions and classes defined by the executed code are not
guaranteed to work correctly after a @ref{dd,,runpy} function has returned.
If that limitation is not acceptable for a given use case, @ref{96,,importlib}
is likely to be a more suitable choice than this module.

The @ref{dd,,runpy} module provides two functions:

@geindex run_module() (in module runpy)
@anchor{library/runpy runpy run_module}@anchor{15fe}
@deffn {Function} runpy.run_module (mod_name, init_globals=None, run_name=None, alter_sys=False)

@geindex module; __main__

Execute the code of the specified module and return the resulting module
globals dictionary. The module’s code is first located using the standard
import mechanism (refer to 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} for details) and then executed in a
fresh module namespace.

The @emph{mod_name} argument should be an absolute module name.
If the module name refers to a package rather than a normal
module, then that package is imported and the @code{__main__} submodule within
that package is then executed and the resulting module globals dictionary
returned.

The optional dictionary argument @emph{init_globals} may be used to pre-populate
the module’s globals dictionary before the code is executed. The supplied
dictionary will not be modified. If any of the special global variables
below are defined in the supplied dictionary, those definitions are
overridden by @ref{15fe,,run_module()}.

The special global variables @code{__name__}, @code{__spec__}, @code{__file__},
@code{__cached__}, @code{__loader__} and @code{__package__} are set in the globals
dictionary before the module code is executed (Note that this is a
minimal set of variables - other variables may be set implicitly as an
interpreter implementation detail).

@code{__name__} is set to @emph{run_name} if this optional argument is not
@ref{182,,None}, to @code{mod_name + '.__main__'} if the named module is a
package and to the @emph{mod_name} argument otherwise.

@code{__spec__} will be set appropriately for the @emph{actually} imported
module (that is, @code{__spec__.name} will always be @emph{mod_name} or
@code{mod_name + '.__main__}, never @emph{run_name}).

@code{__file__}, @code{__cached__}, @code{__loader__} and @code{__package__} are
@ref{184f,,set as normal} based on the module spec.

If the argument @emph{alter_sys} is supplied and evaluates to @ref{748,,True},
then @code{sys.argv[0]} is updated with the value of @code{__file__} and
@code{sys.modules[__name__]} is updated with a temporary module object for the
module being executed. Both @code{sys.argv[0]} and @code{sys.modules[__name__]}
are restored to their original values before the function returns.

Note that this manipulation of @ref{f7,,sys} is not thread-safe. Other threads
may see the partially initialised module, as well as the altered list of
arguments. It is recommended that the @ref{f7,,sys} module be left alone when
invoking this function from threaded code.


@subsubheading See also


The @ref{611,,-m} option offering equivalent functionality from the
command line.



Changed in version 3.1: Added ability to execute packages by looking for a @code{__main__} submodule.



Changed in version 3.2: Added @code{__cached__} global variable (see 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147}).



Changed in version 3.4: Updated to take advantage of the module spec feature added by
@geindex Python Enhancement Proposals; PEP 451
PEP 451@footnote{https://www.python.org/dev/peps/pep-0451}. This allows @code{__cached__} to be set correctly for modules
run this way, as well as ensuring the real module name is always
accessible as @code{__spec__.name}.

@end deffn

@geindex run_path() (in module runpy)
@anchor{library/runpy runpy run_path}@anchor{eb3}
@deffn {Function} runpy.run_path (file_path, init_globals=None, run_name=None)

@geindex module; __main__

Execute the code at the named filesystem location and return the resulting
module globals dictionary. As with a script name supplied to the CPython
command line, the supplied path may refer to a Python source file, a
compiled bytecode file or a valid sys.path entry containing a @code{__main__}
module (e.g. a zipfile containing a top-level @code{__main__.py} file).

For a simple script, the specified code is simply executed in a fresh
module namespace. For a valid sys.path entry (typically a zipfile or
directory), the entry is first added to the beginning of @code{sys.path}. The
function then looks for and executes a @ref{1,,__main__} module using the
updated path. Note that there is no special protection against invoking
an existing @ref{1,,__main__} entry located elsewhere on @code{sys.path} if
there is no such module at the specified location.

The optional dictionary argument @emph{init_globals} may be used to pre-populate
the module’s globals dictionary before the code is executed. The supplied
dictionary will not be modified. If any of the special global variables
below are defined in the supplied dictionary, those definitions are
overridden by @ref{eb3,,run_path()}.

The special global variables @code{__name__}, @code{__spec__}, @code{__file__},
@code{__cached__}, @code{__loader__} and @code{__package__} are set in the globals
dictionary before the module code is executed (Note that this is a
minimal set of variables - other variables may be set implicitly as an
interpreter implementation detail).

@code{__name__} is set to @emph{run_name} if this optional argument is not
@ref{182,,None} and to @code{'<run_path>'} otherwise.

If the supplied path directly references a script file (whether as source
or as precompiled byte code), then @code{__file__} will be set to the
supplied path, and @code{__spec__}, @code{__cached__}, @code{__loader__} and
@code{__package__} will all be set to @ref{182,,None}.

If the supplied path is a reference to a valid sys.path entry, then
@code{__spec__} will be set appropriately for the imported @code{__main__}
module (that is, @code{__spec__.name} will always be @code{__main__}).
@code{__file__}, @code{__cached__}, @code{__loader__} and @code{__package__} will be
@ref{184f,,set as normal} based on the module spec.

A number of alterations are also made to the @ref{f7,,sys} module. Firstly,
@code{sys.path} may be altered as described above. @code{sys.argv[0]} is updated
with the value of @code{file_path} and @code{sys.modules[__name__]} is updated
with a temporary module object for the module being executed. All
modifications to items in @ref{f7,,sys} are reverted before the function
returns.

Note that, unlike @ref{15fe,,run_module()}, the alterations made to @ref{f7,,sys}
are not optional in this function as these adjustments are essential to
allowing the execution of sys.path entries. As the thread-safety
limitations still apply, use of this function in threaded code should be
either serialised with the import lock or delegated to a separate process.


@subsubheading See also


@ref{15fd,,Interface options} for equivalent functionality on the
command line (@code{python path/to/script}).



New in version 3.2.



Changed in version 3.4: Updated to take advantage of the module spec feature added by
@geindex Python Enhancement Proposals; PEP 451
PEP 451@footnote{https://www.python.org/dev/peps/pep-0451}. This allows @code{__cached__} to be set correctly in the
case where @code{__main__} is imported from a valid sys.path entry rather
than being executed directly.

@end deffn


@subsubheading See also



@table @asis

@geindex Python Enhancement Proposals; PEP 338

@item PEP 338@footnote{https://www.python.org/dev/peps/pep-0338} – Executing modules as scripts

PEP written and implemented by Nick Coghlan.

@geindex Python Enhancement Proposals; PEP 366

@item PEP 366@footnote{https://www.python.org/dev/peps/pep-0366} – Main module explicit relative imports

PEP written and implemented by Nick Coghlan.

@geindex Python Enhancement Proposals; PEP 451

@item PEP 451@footnote{https://www.python.org/dev/peps/pep-0451} – A ModuleSpec Type for the Import System

PEP written and implemented by Eric Snow
@end table

@ref{1259,,Command line and environment} - CPython command line details

The @ref{d3e,,importlib.import_module()} function


@node importlib — The implementation of import,importlib resources – Resources,runpy — Locating and executing Python modules,Importing Modules
@anchor{library/importlib doc}@anchor{3c91}@anchor{library/importlib importlib-the-implementation-of-import}@anchor{3c92}@anchor{library/importlib module-importlib}@anchor{96}
@subsection @code{importlib} — The implementation of @code{import}


@geindex module; importlib


New in version 3.1.


@strong{Source code:} Lib/importlib/__init__.py@footnote{https://github.com/python/cpython/tree/main/Lib/importlib/__init__.py}


__________________________________________________________________


@menu
* Introduction: Introduction<12>. 
* Functions: Functions<11>. 
* importlib.abc – Abstract base classes related to import: importlib abc – Abstract base classes related to import. 
* importlib.machinery – Importers and path hooks: importlib machinery – Importers and path hooks. 
* importlib.util – Utility code for importers: importlib util – Utility code for importers. 
* Examples: Examples<32>. 

@end menu

@node Introduction<12>,Functions<11>,,importlib — The implementation of import
@anchor{library/importlib introduction}@anchor{3c93}
@subsubsection Introduction


The purpose of the @ref{96,,importlib} package is three-fold.

One is to provide the
implementation of the @ref{e35,,import} statement (and thus, by extension, the
@ref{3b7,,__import__()} function) in Python source code. This provides an
implementation of @code{import} which is portable to any Python
interpreter. This also provides an implementation which is easier to
comprehend than one implemented in a programming language other than Python.

Two, the components to implement @ref{e35,,import} are exposed in this
package, making it easier for users to create their own custom objects (known
generically as an @ref{1849,,importer}) to participate in the import process.

Three, the package contains modules exposing additional functionality for
managing aspects of Python packages:


@itemize *

@item 
@ref{99,,importlib.metadata} presents access to metadata from third-party
distributions.

@item 
@ref{9a,,importlib.resources} provides routines for accessing non-code
“resources” from Python packages.
@end itemize


@subsubheading See also



@table @asis

@item @ref{e35,,The import statement}

The language reference for the @ref{e35,,import} statement.

@item Packages specification@footnote{https://www.python.org/doc/essays/packages/}

Original specification of packages. Some semantics have changed since
the writing of this document (e.g. redirecting based on @code{None}
in @ref{11f7,,sys.modules}).

@item The @ref{c02,,__import__()} function

The @ref{e35,,import} statement is syntactic sugar for this function.

@geindex Python Enhancement Proposals; PEP 235

@item PEP 235@footnote{https://www.python.org/dev/peps/pep-0235}

Import on Case-Insensitive Platforms

@geindex Python Enhancement Proposals; PEP 263

@item PEP 263@footnote{https://www.python.org/dev/peps/pep-0263}

Defining Python Source Code Encodings

@geindex Python Enhancement Proposals; PEP 302

@item PEP 302@footnote{https://www.python.org/dev/peps/pep-0302}

New Import Hooks

@geindex Python Enhancement Proposals; PEP 328

@item PEP 328@footnote{https://www.python.org/dev/peps/pep-0328}

Imports: Multi-Line and Absolute/Relative

@geindex Python Enhancement Proposals; PEP 366

@item PEP 366@footnote{https://www.python.org/dev/peps/pep-0366}

Main module explicit relative imports

@geindex Python Enhancement Proposals; PEP 420

@item PEP 420@footnote{https://www.python.org/dev/peps/pep-0420}

Implicit namespace packages

@geindex Python Enhancement Proposals; PEP 451

@item PEP 451@footnote{https://www.python.org/dev/peps/pep-0451}

A ModuleSpec Type for the Import System

@geindex Python Enhancement Proposals; PEP 488

@item PEP 488@footnote{https://www.python.org/dev/peps/pep-0488}

Elimination of PYO files

@geindex Python Enhancement Proposals; PEP 489

@item PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}

Multi-phase extension module initialization

@geindex Python Enhancement Proposals; PEP 552

@item PEP 552@footnote{https://www.python.org/dev/peps/pep-0552}

Deterministic pycs

@geindex Python Enhancement Proposals; PEP 3120

@item PEP 3120@footnote{https://www.python.org/dev/peps/pep-3120}

Using UTF-8 as the Default Source Encoding

@geindex Python Enhancement Proposals; PEP 3147

@item PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147}

PYC Repository Directories
@end table


@node Functions<11>,importlib abc – Abstract base classes related to import,Introduction<12>,importlib — The implementation of import
@anchor{library/importlib functions}@anchor{3c94}
@subsubsection Functions


@geindex __import__() (in module importlib)
@anchor{library/importlib importlib __import__}@anchor{c02}
@deffn {Function} importlib.__import__ (name, globals=None, locals=None, fromlist=(), level=0)

An implementation of the built-in @ref{3b7,,__import__()} function.

@cartouche
@quotation Note 
Programmatic importing of modules should use @ref{d3e,,import_module()}
instead of this function.
@end quotation
@end cartouche
@end deffn

@geindex import_module() (in module importlib)
@anchor{library/importlib importlib import_module}@anchor{d3e}
@deffn {Function} importlib.import_module (name, package=None)

Import a module. The @emph{name} argument specifies what module to
import in absolute or relative terms
(e.g. either @code{pkg.mod} or @code{..mod}). If the name is
specified in relative terms, then the @emph{package} argument must be set to
the name of the package which is to act as the anchor for resolving the
package name (e.g. @code{import_module('..mod', 'pkg.subpkg')} will import
@code{pkg.mod}).

The @ref{d3e,,import_module()} function acts as a simplifying wrapper around
@ref{c02,,importlib.__import__()}. This means all semantics of the function are
derived from @ref{c02,,importlib.__import__()}. The most important difference
between these two functions is that @ref{d3e,,import_module()} returns the
specified package or module (e.g. @code{pkg.mod}), while @ref{3b7,,__import__()}
returns the top-level package or module (e.g. @code{pkg}).

If you are dynamically importing a module that was created since the
interpreter began execution (e.g., created a Python source file), you may
need to call @ref{74b,,invalidate_caches()} in order for the new module to be
noticed by the import system.


Changed in version 3.3: Parent packages are automatically imported.

@end deffn

@geindex find_loader() (in module importlib)
@anchor{library/importlib importlib find_loader}@anchor{310}
@deffn {Function} importlib.find_loader (name, path=None)

Find the loader for a module, optionally within the specified @emph{path}. If the
module is in @ref{11f7,,sys.modules}, then @code{sys.modules[name].__loader__} is
returned (unless the loader would be @code{None} or is not set, in which case
@ref{17c,,ValueError} is raised). Otherwise a search using @ref{87c,,sys.meta_path}
is done. @code{None} is returned if no loader is found.

A dotted name does not have its parents implicitly imported as that requires
loading them and that may not be desired. To properly import a submodule you
will need to import all parent packages of the submodule and use the correct
argument to @emph{path}.


New in version 3.3.



Changed in version 3.4: If @code{__loader__} is not set, raise @ref{17c,,ValueError}, just like when the
attribute is set to @code{None}.



Deprecated since version 3.4: Use @ref{b98,,importlib.util.find_spec()} instead.

@end deffn

@geindex invalidate_caches() (in module importlib)
@anchor{library/importlib importlib invalidate_caches}@anchor{74b}
@deffn {Function} importlib.invalidate_caches ()

Invalidate the internal caches of finders stored at
@ref{87c,,sys.meta_path}. If a finder implements @code{invalidate_caches()} then it
will be called to perform the invalidation.  This function should be called
if any modules are created/installed while your program is running to
guarantee all finders will notice the new module’s existence.


New in version 3.3.



Changed in version 3.10: Namespace packages created/installed in a different @ref{489,,sys.path}
location after the same namespace was already imported are noticed.

@end deffn

@geindex reload() (in module importlib)
@anchor{library/importlib importlib reload}@anchor{66a}
@deffn {Function} importlib.reload (module)

Reload a previously imported @emph{module}.  The argument must be a module object,
so it must have been successfully imported before.  This is useful if you
have edited the module source file using an external editor and want to try
out the new version without leaving the Python interpreter.  The return value
is the module object (which can be different if re-importing causes a
different object to be placed in @ref{11f7,,sys.modules}).

When @ref{66a,,reload()} is executed:


@itemize *

@item 
Python module’s code is recompiled and the module-level code re-executed,
defining a new set of objects which are bound to names in the module’s
dictionary by reusing the @ref{1848,,loader} which originally loaded the
module.  The @code{init} function of extension modules is not called a second
time.

@item 
As with all other objects in Python the old objects are only reclaimed
after their reference counts drop to zero.

@item 
The names in the module namespace are updated to point to any new or
changed objects.

@item 
Other references to the old objects (such as names external to the module) are
not rebound to refer to the new objects and must be updated in each namespace
where they occur if that is desired.
@end itemize

There are a number of other caveats:

When a module is reloaded, its dictionary (containing the module’s global
variables) is retained.  Redefinitions of names will override the old
definitions, so this is generally not a problem.  If the new version of a
module does not define a name that was defined by the old version, the old
definition remains.  This feature can be used to the module’s advantage if it
maintains a global table or cache of objects — with a @ref{f6d,,try}
statement it can test for the table’s presence and skip its initialization if
desired:

@example
try:
    cache
except NameError:
    cache = @{@}
@end example

It is generally not very useful to reload built-in or dynamically loaded
modules.  Reloading @ref{f7,,sys}, @ref{1,,__main__}, @ref{11,,builtins} and other
key modules is not recommended.  In many cases extension modules are not
designed to be initialized more than once, and may fail in arbitrary ways
when reloaded.

If a module imports objects from another module using @ref{e51,,from} …
@ref{e35,,import} …, calling @ref{66a,,reload()} for the other module does not
redefine the objects imported from it — one way around this is to
re-execute the @code{from} statement, another is to use @code{import}
and qualified names (@emph{module.name}) instead.

If a module instantiates instances of a class, reloading the module that
defines the class does not affect the method definitions of the instances —
they continue to use the old class definition.  The same is true for derived
classes.


New in version 3.4.



Changed in version 3.7: @ref{66b,,ModuleNotFoundError} is raised when the module being reloaded lacks
a @ref{1853,,ModuleSpec}.

@end deffn

@node importlib abc – Abstract base classes related to import,importlib machinery – Importers and path hooks,Functions<11>,importlib — The implementation of import
@anchor{library/importlib importlib-abc-abstract-base-classes-related-to-import}@anchor{3c95}@anchor{library/importlib module-importlib abc}@anchor{97}
@subsubsection @code{importlib.abc} – Abstract base classes related to import


@geindex module; importlib.abc

@strong{Source code:} Lib/importlib/abc.py@footnote{https://github.com/python/cpython/tree/main/Lib/importlib/abc.py}


__________________________________________________________________


The @ref{97,,importlib.abc} module contains all of the core abstract base classes
used by @ref{e35,,import}. Some subclasses of the core abstract base classes
are also provided to help in implementing the core ABCs.

ABC hierarchy:

@example
object
 +-- Finder (deprecated)
 |    +-- MetaPathFinder
 |    +-- PathEntryFinder
 +-- Loader
      +-- ResourceLoader --------+
      +-- InspectLoader          |
           +-- ExecutionLoader --+
                                 +-- FileLoader
                                 +-- SourceLoader
@end example

@geindex Finder (class in importlib.abc)
@anchor{library/importlib importlib abc Finder}@anchor{30d}
@deffn {Class} importlib.abc.Finder

An abstract base class representing a @ref{1847,,finder}.


Deprecated since version 3.3: Use @ref{30e,,MetaPathFinder} or @ref{30f,,PathEntryFinder} instead.


@geindex find_module() (importlib.abc.Finder method)
@anchor{library/importlib importlib abc Finder find_module}@anchor{300}
@deffn {Method} abstractmethod find_module (fullname, path=None)

An abstract method for finding a @ref{1848,,loader} for the specified
module.  Originally specified in 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302}, this method was meant
for use in @ref{87c,,sys.meta_path} and in the path-based import subsystem.


Changed in version 3.4: Returns @code{None} when called instead of raising
@ref{8a3,,NotImplementedError}.



Deprecated since version 3.10: Implement @ref{308,,MetaPathFinder.find_spec()} or
@ref{309,,PathEntryFinder.find_spec()} instead.

@end deffn
@end deffn

@geindex MetaPathFinder (class in importlib.abc)
@anchor{library/importlib importlib abc MetaPathFinder}@anchor{30e}
@deffn {Class} importlib.abc.MetaPathFinder

An abstract base class representing a @ref{c05,,meta path finder}. For
compatibility, this is a subclass of @ref{30d,,Finder}.


New in version 3.3.



Changed in version 3.10: No longer a subclass of @ref{30d,,Finder}.


@geindex find_spec() (importlib.abc.MetaPathFinder method)
@anchor{library/importlib importlib abc MetaPathFinder find_spec}@anchor{308}
@deffn {Method} find_spec (fullname, path, target=None)

An abstract method for finding a @ref{3b75,,spec} for
the specified module.  If this is a top-level import, @emph{path} will
be @code{None}.  Otherwise, this is a search for a subpackage or
module and @emph{path} will be the value of @ref{1550,,__path__} from the
parent package. If a spec cannot be found, @code{None} is returned.
When passed in, @code{target} is a module object that the finder may
use to make a more educated guess about what spec to return.
@ref{30a,,importlib.util.spec_from_loader()} may be useful for implementing
concrete @code{MetaPathFinders}.


New in version 3.4.

@end deffn

@geindex find_module() (importlib.abc.MetaPathFinder method)
@anchor{library/importlib importlib abc MetaPathFinder find_module}@anchor{306}
@deffn {Method} find_module (fullname, path)

A legacy method for finding a @ref{1848,,loader} for the specified
module.  If this is a top-level import, @emph{path} will be @code{None}.
Otherwise, this is a search for a subpackage or module and @emph{path}
will be the value of @ref{1550,,__path__} from the parent
package. If a loader cannot be found, @code{None} is returned.

If @ref{308,,find_spec()} is defined, backwards-compatible functionality is
provided.


Changed in version 3.4: Returns @code{None} when called instead of raising
@ref{8a3,,NotImplementedError}. Can use @ref{308,,find_spec()} to provide
functionality.



Deprecated since version 3.4: Use @ref{308,,find_spec()} instead.

@end deffn

@geindex invalidate_caches() (importlib.abc.MetaPathFinder method)
@anchor{library/importlib importlib abc MetaPathFinder invalidate_caches}@anchor{3c96}
@deffn {Method} invalidate_caches ()

An optional method which, when called, should invalidate any internal
cache used by the finder. Used by @ref{74b,,importlib.invalidate_caches()}
when invalidating the caches of all finders on @ref{87c,,sys.meta_path}.


Changed in version 3.4: Returns @code{None} when called instead of @code{NotImplemented}.

@end deffn
@end deffn

@geindex PathEntryFinder (class in importlib.abc)
@anchor{library/importlib importlib abc PathEntryFinder}@anchor{30f}
@deffn {Class} importlib.abc.PathEntryFinder

An abstract base class representing a @ref{c06,,path entry finder}.  Though
it bears some similarities to @ref{30e,,MetaPathFinder}, @code{PathEntryFinder}
is meant for use only within the path-based import subsystem provided
by @ref{bb4,,importlib.machinery.PathFinder}.


New in version 3.3.



Changed in version 3.10: No longer a subclass of @ref{30d,,Finder}.


@geindex find_spec() (importlib.abc.PathEntryFinder method)
@anchor{library/importlib importlib abc PathEntryFinder find_spec}@anchor{309}
@deffn {Method} find_spec (fullname, target=None)

An abstract method for finding a @ref{3b75,,spec} for
the specified module.  The finder will search for the module only
within the @ref{1864,,path entry} to which it is assigned.  If a spec
cannot be found, @code{None} is returned.  When passed in, @code{target}
is a module object that the finder may use to make a more educated
guess about what spec to return. @ref{30a,,importlib.util.spec_from_loader()}
may be useful for implementing concrete @code{PathEntryFinders}.


New in version 3.4.

@end deffn

@geindex find_loader() (importlib.abc.PathEntryFinder method)
@anchor{library/importlib importlib abc PathEntryFinder find_loader}@anchor{2ff}
@deffn {Method} find_loader (fullname)

A legacy method for finding a @ref{1848,,loader} for the specified
module.  Returns a 2-tuple of @code{(loader, portion)} where @code{portion}
is a sequence of file system locations contributing to part of a namespace
package. The loader may be @code{None} while specifying @code{portion} to
signify the contribution of the file system locations to a namespace
package. An empty list can be used for @code{portion} to signify the loader
is not part of a namespace package. If @code{loader} is @code{None} and
@code{portion} is the empty list then no loader or location for a namespace
package were found (i.e. failure to find anything for the module).

If @ref{309,,find_spec()} is defined then backwards-compatible functionality is
provided.


Changed in version 3.4: Returns @code{(None, [])} instead of raising @ref{8a3,,NotImplementedError}.
Uses @ref{309,,find_spec()} when available to provide functionality.



Deprecated since version 3.4: Use @ref{309,,find_spec()} instead.

@end deffn

@geindex find_module() (importlib.abc.PathEntryFinder method)
@anchor{library/importlib importlib abc PathEntryFinder find_module}@anchor{307}
@deffn {Method} find_module (fullname)

A concrete implementation of @ref{300,,Finder.find_module()} which is
equivalent to @code{self.find_loader(fullname)[0]}.


Deprecated since version 3.4: Use @ref{309,,find_spec()} instead.

@end deffn

@geindex invalidate_caches() (importlib.abc.PathEntryFinder method)
@anchor{library/importlib importlib abc PathEntryFinder invalidate_caches}@anchor{3c97}
@deffn {Method} invalidate_caches ()

An optional method which, when called, should invalidate any internal
cache used by the finder. Used by
@ref{74a,,importlib.machinery.PathFinder.invalidate_caches()}
when invalidating the caches of all cached finders.
@end deffn
@end deffn

@geindex Loader (class in importlib.abc)
@anchor{library/importlib importlib abc Loader}@anchor{a39}
@deffn {Class} importlib.abc.Loader

An abstract base class for a @ref{1848,,loader}.
See 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} for the exact definition for a loader.

Loaders that wish to support resource reading should implement a
@code{get_resource_reader()} method as specified by
@ref{61b,,importlib.abc.ResourceReader}.


Changed in version 3.7: Introduced the optional @code{get_resource_reader()} method.


@geindex create_module() (importlib.abc.Loader method)
@anchor{library/importlib importlib abc Loader create_module}@anchor{7f6}
@deffn {Method} create_module (spec)

A method that returns the module object to use when
importing a module.  This method may return @code{None},
indicating that default module creation semantics should take place.


New in version 3.4.



Changed in version 3.5: Starting in Python 3.6, this method will not be optional when
@ref{302,,exec_module()} is defined.

@end deffn

@geindex exec_module() (importlib.abc.Loader method)
@anchor{library/importlib importlib abc Loader exec_module}@anchor{302}
@deffn {Method} exec_module (module)

An abstract method that executes the module in its own namespace
when a module is imported or reloaded.  The module should already
be initialized when @ref{302,,exec_module()} is called.  When this method exists,
@ref{7f6,,create_module()} must be defined.


New in version 3.4.



Changed in version 3.6: @ref{7f6,,create_module()} must also be defined.

@end deffn

@geindex load_module() (importlib.abc.Loader method)
@anchor{library/importlib importlib abc Loader load_module}@anchor{301}
@deffn {Method} load_module (fullname)

A legacy method for loading a module.  If the module cannot be
loaded, @ref{3b8,,ImportError} is raised, otherwise the loaded module is
returned.

If the requested module already exists in @ref{11f7,,sys.modules}, that
module should be used and reloaded.
Otherwise the loader should create a new module and insert it into
@ref{11f7,,sys.modules} before any loading begins, to prevent recursion
from the import.  If the loader inserted a module and the load fails, it
must be removed by the loader from @ref{11f7,,sys.modules}; modules already
in @ref{11f7,,sys.modules} before the loader began execution should be left
alone (see @ref{311,,importlib.util.module_for_loader()}).

The loader should set several attributes on the module
(note that some of these attributes can change when a module is
reloaded):


@itemize -

@item 

@table @asis

@item @ref{f12,,__name__}

The module’s fully-qualified name.
It is @code{'__main__'} for an executed module.
@end table

@item 

@table @asis

@item @ref{1797,,__file__}

The location the @ref{1848,,loader} used to load the module.
For example, for modules loaded from a .py file this is the filename.
It is not set on all modules (e.g. built-in modules).
@end table

@item 

@table @asis

@item @ref{d5b,,__cached__}

The filename of a compiled version of the module’s code.
It is not set on all modules (e.g. built-in modules).
@end table

@item 

@table @asis

@item @ref{1550,,__path__}

The list of locations where the package’s submodules will be found.
Most of the time this is a single directory.
The import system passes this attribute to @code{__import__()} and to finders
in the same way as @ref{489,,sys.path} but just for the package.
It is not set on non-package modules so it can be used
as an indicator that the module is a package.
@end table

@item 

@table @asis

@item @ref{bae,,__package__}

The fully-qualified name of the package the module is in (or the
empty string for a top-level module).
If the module is a package then this is the same as @ref{f12,,__name__}.
@end table

@item 

@table @asis

@item @ref{baf,,__loader__}

The @ref{1848,,loader} used to load the module.
@end table
@end itemize

When @ref{302,,exec_module()} is available then backwards-compatible
functionality is provided.


Changed in version 3.4: Raise @ref{3b8,,ImportError} when called instead of
@ref{8a3,,NotImplementedError}.  Functionality provided when
@ref{302,,exec_module()} is available.



Deprecated since version 3.4: The recommended API for loading a module is @ref{302,,exec_module()}
(and @ref{7f6,,create_module()}).  Loaders should implement it instead of
@ref{301,,load_module()}.  The import machinery takes care of all the
other responsibilities of @ref{301,,load_module()} when
@ref{302,,exec_module()} is implemented.

@end deffn

@geindex module_repr() (importlib.abc.Loader method)
@anchor{library/importlib importlib abc Loader module_repr}@anchor{303}
@deffn {Method} module_repr (module)

A legacy method which when implemented calculates and returns the given
module’s representation, as a string.  The module type’s default
@code{__repr__()} will use the result of this method as appropriate.


New in version 3.3.



Changed in version 3.4: Made optional instead of an abstractmethod.



Deprecated since version 3.4: The import machinery now takes care of this automatically.

@end deffn
@end deffn

@geindex ResourceReader (class in importlib.abc)
@anchor{library/importlib importlib abc ResourceReader}@anchor{61b}
@deffn {Class} importlib.abc.ResourceReader

@emph{Superseded by TraversableResources}

An @ref{d5c,,abstract base class} to provide the ability to read
@emph{resources}.

From the perspective of this ABC, a @emph{resource} is a binary
artifact that is shipped within a package. Typically this is
something like a data file that lives next to the @code{__init__.py}
file of the package. The purpose of this class is to help abstract
out the accessing of such data files so that it does not matter if
the package and its data file(s) are stored in a e.g. zip file
versus on the file system.

For any of methods of this class, a @emph{resource} argument is
expected to be a @ref{63d,,path-like object} which represents
conceptually just a file name. This means that no subdirectory
paths should be included in the @emph{resource} argument. This is
because the location of the package the reader is for, acts as the
“directory”. Hence the metaphor for directories and file
names is packages and resources, respectively. This is also why
instances of this class are expected to directly correlate to
a specific package (instead of potentially representing multiple
packages or a module).

Loaders that wish to support resource reading are expected to
provide a method called @code{get_resource_reader(fullname)} which
returns an object implementing this ABC’s interface. If the module
specified by fullname is not a package, this method should return
@ref{182,,None}. An object compatible with this ABC should only be
returned when the specified module is a package.


New in version 3.7.


@geindex open_resource() (importlib.abc.ResourceReader method)
@anchor{library/importlib importlib abc ResourceReader open_resource}@anchor{3c98}
@deffn {Method} abstractmethod open_resource (resource)

Returns an opened, @ref{1e54,,file-like object} for binary reading
of the @emph{resource}.

If the resource cannot be found, @ref{a37,,FileNotFoundError} is
raised.
@end deffn

@geindex resource_path() (importlib.abc.ResourceReader method)
@anchor{library/importlib importlib abc ResourceReader resource_path}@anchor{3c99}
@deffn {Method} abstractmethod resource_path (resource)

Returns the file system path to the @emph{resource}.

If the resource does not concretely exist on the file system,
raise @ref{a37,,FileNotFoundError}.
@end deffn

@geindex is_resource() (importlib.abc.ResourceReader method)
@anchor{library/importlib importlib abc ResourceReader is_resource}@anchor{3c9a}
@deffn {Method} abstractmethod is_resource (name)

Returns @code{True} if the named @emph{name} is considered a resource.
@ref{a37,,FileNotFoundError} is raised if @emph{name} does not exist.
@end deffn

@geindex contents() (importlib.abc.ResourceReader method)
@anchor{library/importlib importlib abc ResourceReader contents}@anchor{12ff}
@deffn {Method} abstractmethod contents ()

Returns an @ref{dcc,,iterable} of strings over the contents of
the package. Do note that it is not required that all names
returned by the iterator be actual resources, e.g. it is
acceptable to return names for which @code{is_resource()} would
be false.

Allowing non-resource names to be returned is to allow for
situations where how a package and its resources are stored
are known a priori and the non-resource names would be useful.
For instance, returning subdirectory names is allowed so that
when it is known that the package and resources are stored on
the file system then those subdirectory names can be used
directly.

The abstract method returns an iterable of no items.
@end deffn
@end deffn

@geindex ResourceLoader (class in importlib.abc)
@anchor{library/importlib importlib abc ResourceLoader}@anchor{71f}
@deffn {Class} importlib.abc.ResourceLoader

An abstract base class for a @ref{1848,,loader} which implements the optional
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} protocol for loading arbitrary resources from the storage
back-end.


Deprecated since version 3.7: This ABC is deprecated in favour of supporting resource loading
through @ref{61b,,importlib.abc.ResourceReader}.


@geindex get_data() (importlib.abc.ResourceLoader method)
@anchor{library/importlib importlib abc ResourceLoader get_data}@anchor{3c84}
@deffn {Method} abstractmethod get_data (path)

An abstract method to return the bytes for the data located at @emph{path}.
Loaders that have a file-like storage back-end
that allows storing arbitrary data
can implement this abstract method to give direct access
to the data stored. @ref{2a1,,OSError} is to be raised if the @emph{path} cannot
be found. The @emph{path} is expected to be constructed using a module’s
@ref{1797,,__file__} attribute or an item from a package’s @ref{1550,,__path__}.


Changed in version 3.4: Raises @ref{2a1,,OSError} instead of @ref{8a3,,NotImplementedError}.

@end deffn
@end deffn

@geindex InspectLoader (class in importlib.abc)
@anchor{library/importlib importlib abc InspectLoader}@anchor{ad2}
@deffn {Class} importlib.abc.InspectLoader

An abstract base class for a @ref{1848,,loader} which implements the optional
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} protocol for loaders that inspect modules.

@geindex get_code() (importlib.abc.InspectLoader method)
@anchor{library/importlib importlib abc InspectLoader get_code}@anchor{ad3}
@deffn {Method} get_code (fullname)

Return the code object for a module, or @code{None} if the module does not
have a code object (as would be the case, for example, for a built-in
module).  Raise an @ref{3b8,,ImportError} if loader cannot find the
requested module.

@cartouche
@quotation Note 
While the method has a default implementation, it is suggested that
it be overridden if possible for performance.
@end quotation
@end cartouche

@geindex universal newlines; importlib.abc.InspectLoader.get_source method


Changed in version 3.4: No longer abstract and a concrete implementation is provided.

@end deffn

@geindex get_source() (importlib.abc.InspectLoader method)
@anchor{library/importlib importlib abc InspectLoader get_source}@anchor{3c9b}
@deffn {Method} abstractmethod get_source (fullname)

An abstract method to return the source of a module. It is returned as
a text string using @ref{88a,,universal newlines}, translating all
recognized line separators into @code{'\n'} characters.  Returns @code{None}
if no source is available (e.g. a built-in module). Raises
@ref{3b8,,ImportError} if the loader cannot find the module specified.


Changed in version 3.4: Raises @ref{3b8,,ImportError} instead of @ref{8a3,,NotImplementedError}.

@end deffn

@geindex is_package() (importlib.abc.InspectLoader method)
@anchor{library/importlib importlib abc InspectLoader is_package}@anchor{3c9c}
@deffn {Method} is_package (fullname)

An optional method to return a true value if the module is a package, a
false value otherwise. @ref{3b8,,ImportError} is raised if the
@ref{1848,,loader} cannot find the module.


Changed in version 3.4: Raises @ref{3b8,,ImportError} instead of @ref{8a3,,NotImplementedError}.

@end deffn

@geindex source_to_code() (importlib.abc.InspectLoader static method)
@anchor{library/importlib importlib abc InspectLoader source_to_code}@anchor{971}
@deffn {Method} static source_to_code (data, path='<string>')

Create a code object from Python source.

The @emph{data} argument can be whatever the @ref{4e2,,compile()} function
supports (i.e. string or bytes). The @emph{path} argument should be
the “path” to where the source code originated from, which can be an
abstract concept (e.g. location in a zip file).

With the subsequent code object one can execute it in a module by
running @code{exec(code, module.__dict__)}.


New in version 3.4.



Changed in version 3.5: Made the method static.

@end deffn

@geindex exec_module() (importlib.abc.InspectLoader method)
@anchor{library/importlib importlib abc InspectLoader exec_module}@anchor{b9d}
@deffn {Method} exec_module (module)

Implementation of @ref{302,,Loader.exec_module()}.


New in version 3.4.

@end deffn

@geindex load_module() (importlib.abc.InspectLoader method)
@anchor{library/importlib importlib abc InspectLoader load_module}@anchor{b9a}
@deffn {Method} load_module (fullname)

Implementation of @ref{301,,Loader.load_module()}.


Deprecated since version 3.4: use @code{exec_module()} instead.

@end deffn
@end deffn

@geindex ExecutionLoader (class in importlib.abc)
@anchor{library/importlib importlib abc ExecutionLoader}@anchor{3c9d}
@deffn {Class} importlib.abc.ExecutionLoader

An abstract base class which inherits from @code{InspectLoader} that,
when implemented, helps a module to be executed as a script. The ABC
represents an optional 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} protocol.

@geindex get_filename() (importlib.abc.ExecutionLoader method)
@anchor{library/importlib importlib abc ExecutionLoader get_filename}@anchor{3c9e}
@deffn {Method} abstractmethod get_filename (fullname)

An abstract method that is to return the value of @ref{1797,,__file__} for
the specified module. If no path is available, @ref{3b8,,ImportError} is
raised.

If source code is available, then the method should return the path to
the source file, regardless of whether a bytecode was used to load the
module.


Changed in version 3.4: Raises @ref{3b8,,ImportError} instead of @ref{8a3,,NotImplementedError}.

@end deffn
@end deffn

@geindex FileLoader (class in importlib.abc)
@anchor{library/importlib importlib abc FileLoader}@anchor{c08}
@deffn {Class} importlib.abc.FileLoader (fullname, path)

An abstract base class which inherits from @code{ResourceLoader} and
@code{ExecutionLoader}, providing concrete implementations of
@code{ResourceLoader.get_data()} and @code{ExecutionLoader.get_filename()}.

The @emph{fullname} argument is a fully resolved name of the module the loader is
to handle. The @emph{path} argument is the path to the file for the module.


New in version 3.3.


@geindex name (importlib.abc.FileLoader attribute)
@anchor{library/importlib importlib abc FileLoader name}@anchor{3c9f}
@deffn {Attribute} name

The name of the module the loader can handle.
@end deffn

@geindex path (importlib.abc.FileLoader attribute)
@anchor{library/importlib importlib abc FileLoader path}@anchor{3ca0}
@deffn {Attribute} path

Path to the file of the module.
@end deffn

@geindex load_module() (importlib.abc.FileLoader method)
@anchor{library/importlib importlib abc FileLoader load_module}@anchor{b9b}
@deffn {Method} load_module (fullname)

Calls super’s @code{load_module()}.


Deprecated since version 3.4: Use @ref{302,,Loader.exec_module()} instead.

@end deffn

@geindex get_filename() (importlib.abc.FileLoader method)
@anchor{library/importlib importlib abc FileLoader get_filename}@anchor{3ca1}
@deffn {Method} abstractmethod get_filename (fullname)

Returns @code{path}.
@end deffn

@geindex get_data() (importlib.abc.FileLoader method)
@anchor{library/importlib importlib abc FileLoader get_data}@anchor{3ca2}
@deffn {Method} abstractmethod get_data (path)

Reads @emph{path} as a binary file and returns the bytes from it.
@end deffn
@end deffn

@geindex SourceLoader (class in importlib.abc)
@anchor{library/importlib importlib abc SourceLoader}@anchor{3ca3}
@deffn {Class} importlib.abc.SourceLoader

An abstract base class for implementing source (and optionally bytecode)
file loading. The class inherits from both @code{ResourceLoader} and
@code{ExecutionLoader}, requiring the implementation of:


@itemize *

@item 
@code{ResourceLoader.get_data()}

@item 

@table @asis

@item @code{ExecutionLoader.get_filename()}

Should only return the path to the source file; sourceless
loading is not supported.
@end table
@end itemize

The abstract methods defined by this class are to add optional bytecode
file support. Not implementing these optional methods (or causing them to
raise @ref{8a3,,NotImplementedError}) causes the loader to
only work with source code. Implementing the methods allows the loader to
work with source @emph{and} bytecode files; it does not allow for @emph{sourceless}
loading where only bytecode is provided.  Bytecode files are an
optimization to speed up loading by removing the parsing step of Python’s
compiler, and so no bytecode-specific API is exposed.

@geindex path_stats() (importlib.abc.SourceLoader method)
@anchor{library/importlib importlib abc SourceLoader path_stats}@anchor{d2a}
@deffn {Method} path_stats (path)

Optional abstract method which returns a @ref{3b0,,dict} containing
metadata about the specified path.  Supported dictionary keys are:


@itemize -

@item 
@code{'mtime'} (mandatory): an integer or floating-point number
representing the modification time of the source code;

@item 
@code{'size'} (optional): the size in bytes of the source code.
@end itemize

Any other keys in the dictionary are ignored, to allow for future
extensions. If the path cannot be handled, @ref{2a1,,OSError} is raised.


New in version 3.3.



Changed in version 3.4: Raise @ref{2a1,,OSError} instead of @ref{8a3,,NotImplementedError}.

@end deffn

@geindex path_mtime() (importlib.abc.SourceLoader method)
@anchor{library/importlib importlib abc SourceLoader path_mtime}@anchor{d29}
@deffn {Method} path_mtime (path)

Optional abstract method which returns the modification time for the
specified path.


Deprecated since version 3.3: This method is deprecated in favour of @code{path_stats()}.  You don’t
have to implement it, but it is still available for compatibility
purposes. Raise @ref{2a1,,OSError} if the path cannot be handled.



Changed in version 3.4: Raise @ref{2a1,,OSError} instead of @ref{8a3,,NotImplementedError}.

@end deffn

@geindex set_data() (importlib.abc.SourceLoader method)
@anchor{library/importlib importlib abc SourceLoader set_data}@anchor{3ca4}
@deffn {Method} set_data (path, data)

Optional abstract method which writes the specified bytes to a file
path. Any intermediate directories which do not exist are to be created
automatically.

When writing to the path fails because the path is read-only
(@ref{26de,,errno.EACCES}/@ref{895,,PermissionError}), do not propagate the
exception.


Changed in version 3.4: No longer raises @ref{8a3,,NotImplementedError} when called.

@end deffn

@geindex get_code() (importlib.abc.SourceLoader method)
@anchor{library/importlib importlib abc SourceLoader get_code}@anchor{3ca5}
@deffn {Method} get_code (fullname)

Concrete implementation of @code{InspectLoader.get_code()}.
@end deffn

@geindex exec_module() (importlib.abc.SourceLoader method)
@anchor{library/importlib importlib abc SourceLoader exec_module}@anchor{b9e}
@deffn {Method} exec_module (module)

Concrete implementation of @ref{302,,Loader.exec_module()}.


New in version 3.4.

@end deffn

@geindex load_module() (importlib.abc.SourceLoader method)
@anchor{library/importlib importlib abc SourceLoader load_module}@anchor{b9c}
@deffn {Method} load_module (fullname)

Concrete implementation of @ref{301,,Loader.load_module()}.


Deprecated since version 3.4: Use @code{exec_module()} instead.

@end deffn

@geindex get_source() (importlib.abc.SourceLoader method)
@anchor{library/importlib importlib abc SourceLoader get_source}@anchor{bb2}
@deffn {Method} get_source (fullname)

Concrete implementation of @code{InspectLoader.get_source()}.
@end deffn

@geindex is_package() (importlib.abc.SourceLoader method)
@anchor{library/importlib importlib abc SourceLoader is_package}@anchor{3ca6}
@deffn {Method} is_package (fullname)

Concrete implementation of @code{InspectLoader.is_package()}. A module
is determined to be a package if its file path (as provided by
@code{ExecutionLoader.get_filename()}) is a file named
@code{__init__} when the file extension is removed @strong{and} the module name
itself does not end in @code{__init__}.
@end deffn
@end deffn

@geindex Traversable (class in importlib.abc)
@anchor{library/importlib importlib abc Traversable}@anchor{10e9}
@deffn {Class} importlib.abc.Traversable

An object with a subset of pathlib.Path methods suitable for
traversing directories and opening files.


New in version 3.9.


@geindex name (importlib.abc.Traversable attribute)
@anchor{library/importlib importlib abc Traversable name}@anchor{3ca7}
@deffn {Attribute} name

Abstract. The base name of this object without any parent references.
@end deffn

@geindex iterdir() (importlib.abc.Traversable method)
@anchor{library/importlib importlib abc Traversable iterdir}@anchor{3ca8}
@deffn {Method} abstractmethod iterdir ()

Yield Traversable objects in self.
@end deffn

@geindex is_dir() (importlib.abc.Traversable method)
@anchor{library/importlib importlib abc Traversable is_dir}@anchor{3ca9}
@deffn {Method} abstractmethod is_dir ()

Return True if self is a directory.
@end deffn

@geindex is_file() (importlib.abc.Traversable method)
@anchor{library/importlib importlib abc Traversable is_file}@anchor{3caa}
@deffn {Method} abstractmethod is_file ()

Return True if self is a file.
@end deffn

@geindex joinpath() (importlib.abc.Traversable method)
@anchor{library/importlib importlib abc Traversable joinpath}@anchor{3cab}
@deffn {Method} abstractmethod joinpath (child)

Return Traversable child in self.
@end deffn

@geindex __truediv__() (importlib.abc.Traversable method)
@anchor{library/importlib importlib abc Traversable __truediv__}@anchor{3cac}
@deffn {Method} abstractmethod __truediv__ (child)

Return Traversable child in self.
@end deffn

@geindex open() (importlib.abc.Traversable method)
@anchor{library/importlib importlib abc Traversable open}@anchor{3cad}
@deffn {Method} abstractmethod open (mode='r', *args, **kwargs)

@emph{mode} may be ‘r’ or ‘rb’ to open as text or binary. Return a handle
suitable for reading (same as @ref{207b,,pathlib.Path.open}).

When opening as text, accepts encoding parameters such as those
accepted by @ref{889,,io.TextIOWrapper}.
@end deffn

@geindex read_bytes() (importlib.abc.Traversable method)
@anchor{library/importlib importlib abc Traversable read_bytes}@anchor{3cae}
@deffn {Method} read_bytes ()

Read contents of self as bytes.
@end deffn

@geindex read_text() (importlib.abc.Traversable method)
@anchor{library/importlib importlib abc Traversable read_text}@anchor{3caf}
@deffn {Method} read_text (encoding=None)

Read contents of self as text.
@end deffn
@end deffn

@geindex TraversableResources (class in importlib.abc)
@anchor{library/importlib importlib abc TraversableResources}@anchor{10e7}
@deffn {Class} importlib.abc.TraversableResources

An abstract base class for resource readers capable of serving
the @ref{3fa,,importlib.resources.files()} interface. Subclasses
@ref{61b,,importlib.abc.ResourceReader} and provides
concrete implementations of the @ref{61b,,importlib.abc.ResourceReader}’s
abstract methods. Therefore, any loader supplying
@code{importlib.abc.TraversableReader} also supplies ResourceReader.

Loaders that wish to support resource reading are expected to
implement this interface.


New in version 3.9.


@geindex files() (importlib.abc.TraversableResources method)
@anchor{library/importlib importlib abc TraversableResources files}@anchor{3cb0}
@deffn {Method} abstractmethod files ()

Returns a @ref{10e9,,importlib.abc.Traversable} object for the loaded
package.
@end deffn
@end deffn

@node importlib machinery – Importers and path hooks,importlib util – Utility code for importers,importlib abc – Abstract base classes related to import,importlib — The implementation of import
@anchor{library/importlib importlib-machinery-importers-and-path-hooks}@anchor{3cb1}@anchor{library/importlib module-importlib machinery}@anchor{98}
@subsubsection @code{importlib.machinery} – Importers and path hooks


@geindex module; importlib.machinery

@strong{Source code:} Lib/importlib/machinery.py@footnote{https://github.com/python/cpython/tree/main/Lib/importlib/machinery.py}


__________________________________________________________________


This module contains the various objects that help @ref{e35,,import}
find and load modules.

@geindex SOURCE_SUFFIXES (in module importlib.machinery)
@anchor{library/importlib importlib machinery SOURCE_SUFFIXES}@anchor{3cb2}
@deffn {Attribute} importlib.machinery.SOURCE_SUFFIXES

A list of strings representing the recognized file suffixes for source
modules.


New in version 3.3.

@end deffn

@geindex DEBUG_BYTECODE_SUFFIXES (in module importlib.machinery)
@anchor{library/importlib importlib machinery DEBUG_BYTECODE_SUFFIXES}@anchor{3cb3}
@deffn {Attribute} importlib.machinery.DEBUG_BYTECODE_SUFFIXES

A list of strings representing the file suffixes for non-optimized bytecode
modules.


New in version 3.3.



Deprecated since version 3.5: Use @ref{3cb4,,BYTECODE_SUFFIXES} instead.

@end deffn

@geindex OPTIMIZED_BYTECODE_SUFFIXES (in module importlib.machinery)
@anchor{library/importlib importlib machinery OPTIMIZED_BYTECODE_SUFFIXES}@anchor{3cb5}
@deffn {Attribute} importlib.machinery.OPTIMIZED_BYTECODE_SUFFIXES

A list of strings representing the file suffixes for optimized bytecode
modules.


New in version 3.3.



Deprecated since version 3.5: Use @ref{3cb4,,BYTECODE_SUFFIXES} instead.

@end deffn

@geindex BYTECODE_SUFFIXES (in module importlib.machinery)
@anchor{library/importlib importlib machinery BYTECODE_SUFFIXES}@anchor{3cb4}
@deffn {Attribute} importlib.machinery.BYTECODE_SUFFIXES

A list of strings representing the recognized file suffixes for bytecode
modules (including the leading dot).


New in version 3.3.



Changed in version 3.5: The value is no longer dependent on @code{__debug__}.

@end deffn

@geindex EXTENSION_SUFFIXES (in module importlib.machinery)
@anchor{library/importlib importlib machinery EXTENSION_SUFFIXES}@anchor{3cb6}
@deffn {Attribute} importlib.machinery.EXTENSION_SUFFIXES

A list of strings representing the recognized file suffixes for
extension modules.


New in version 3.3.

@end deffn

@geindex all_suffixes() (in module importlib.machinery)
@anchor{library/importlib importlib machinery all_suffixes}@anchor{3c0b}
@deffn {Function} importlib.machinery.all_suffixes ()

Returns a combined list of strings representing all file suffixes for
modules recognized by the standard import machinery. This is a
helper for code which simply needs to know if a filesystem path
potentially refers to a module without needing any details on the kind
of module (for example, @ref{888,,inspect.getmodulename()}).


New in version 3.3.

@end deffn

@geindex BuiltinImporter (class in importlib.machinery)
@anchor{library/importlib importlib machinery BuiltinImporter}@anchor{7f7}
@deffn {Class} importlib.machinery.BuiltinImporter

An @ref{1849,,importer} for built-in modules. All known built-in modules are
listed in @ref{3b62,,sys.builtin_module_names}. This class implements the
@ref{30e,,importlib.abc.MetaPathFinder} and
@ref{ad2,,importlib.abc.InspectLoader} ABCs.

Only class methods are defined by this class to alleviate the need for
instantiation.


Changed in version 3.5: As part of 
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}, the builtin importer now implements
@code{Loader.create_module()} and @code{Loader.exec_module()}

@end deffn

@geindex FrozenImporter (class in importlib.machinery)
@anchor{library/importlib importlib machinery FrozenImporter}@anchor{bb0}
@deffn {Class} importlib.machinery.FrozenImporter

An @ref{1849,,importer} for frozen modules. This class implements the
@ref{30e,,importlib.abc.MetaPathFinder} and
@ref{ad2,,importlib.abc.InspectLoader} ABCs.

Only class methods are defined by this class to alleviate the need for
instantiation.


Changed in version 3.4: Gained @code{create_module()} and @code{exec_module()}
methods.

@end deffn

@geindex WindowsRegistryFinder (class in importlib.machinery)
@anchor{library/importlib importlib machinery WindowsRegistryFinder}@anchor{87b}
@deffn {Class} importlib.machinery.WindowsRegistryFinder

@ref{1847,,Finder} for modules declared in the Windows registry.  This class
implements the @ref{30e,,importlib.abc.MetaPathFinder} ABC.

Only class methods are defined by this class to alleviate the need for
instantiation.


New in version 3.3.



Deprecated since version 3.6: Use @ref{e6,,site} configuration instead. Future versions of Python may
not enable this finder by default.

@end deffn

@geindex PathFinder (class in importlib.machinery)
@anchor{library/importlib importlib machinery PathFinder}@anchor{bb4}
@deffn {Class} importlib.machinery.PathFinder

A @ref{1847,,Finder} for @ref{489,,sys.path} and package @code{__path__} attributes.
This class implements the @ref{30e,,importlib.abc.MetaPathFinder} ABC.

Only class methods are defined by this class to alleviate the need for
instantiation.

@geindex find_spec() (importlib.machinery.PathFinder class method)
@anchor{library/importlib importlib machinery PathFinder find_spec}@anchor{b99}
@deffn {Method} classmethod find_spec (fullname, path=None, target=None)

Class method that attempts to find a @ref{3b75,,spec}
for the module specified by @emph{fullname} on @ref{489,,sys.path} or, if
defined, on @emph{path}. For each path entry that is searched,
@ref{74c,,sys.path_importer_cache} is checked. If a non-false object
is found then it is used as the @ref{c06,,path entry finder} to look
for the module being searched for. If no entry is found in
@ref{74c,,sys.path_importer_cache}, then @ref{bb5,,sys.path_hooks} is
searched for a finder for the path entry and, if found, is stored
in @ref{74c,,sys.path_importer_cache} along with being queried about
the module. If no finder is ever found then @code{None} is both
stored in the cache and returned.


New in version 3.4.



Changed in version 3.5: If the current working directory – represented by an empty string –
is no longer valid then @code{None} is returned but no value is cached
in @ref{74c,,sys.path_importer_cache}.

@end deffn

@geindex find_module() (importlib.machinery.PathFinder class method)
@anchor{library/importlib importlib machinery PathFinder find_module}@anchor{30b}
@deffn {Method} classmethod find_module (fullname, path=None)

A legacy wrapper around @ref{b99,,find_spec()}.


Deprecated since version 3.4: Use @ref{b99,,find_spec()} instead.

@end deffn

@geindex invalidate_caches() (importlib.machinery.PathFinder class method)
@anchor{library/importlib importlib machinery PathFinder invalidate_caches}@anchor{74a}
@deffn {Method} classmethod invalidate_caches ()

Calls @ref{3c97,,importlib.abc.PathEntryFinder.invalidate_caches()} on all
finders stored in @ref{74c,,sys.path_importer_cache} that define the method.
Otherwise entries in @ref{74c,,sys.path_importer_cache} set to @code{None} are
deleted.


Changed in version 3.7: Entries of @code{None} in @ref{74c,,sys.path_importer_cache} are deleted.

@end deffn


Changed in version 3.4: Calls objects in @ref{bb5,,sys.path_hooks} with the current working
directory for @code{'@w{'}} (i.e. the empty string).

@end deffn

@geindex FileFinder (class in importlib.machinery)
@anchor{library/importlib importlib machinery FileFinder}@anchor{c07}
@deffn {Class} importlib.machinery.FileFinder (path, *loader_details)

A concrete implementation of @ref{30f,,importlib.abc.PathEntryFinder} which
caches results from the file system.

The @emph{path} argument is the directory for which the finder is in charge of
searching.

The @emph{loader_details} argument is a variable number of 2-item tuples each
containing a loader and a sequence of file suffixes the loader recognizes.
The loaders are expected to be callables which accept two arguments of
the module’s name and the path to the file found.

The finder will cache the directory contents as necessary, making stat calls
for each module search to verify the cache is not outdated. Because cache
staleness relies upon the granularity of the operating system’s state
information of the file system, there is a potential race condition of
searching for a module, creating a new file, and then searching for the
module the new file represents. If the operations happen fast enough to fit
within the granularity of stat calls, then the module search will fail. To
prevent this from happening, when you create a module dynamically, make sure
to call @ref{74b,,importlib.invalidate_caches()}.


New in version 3.3.


@geindex path (importlib.machinery.FileFinder attribute)
@anchor{library/importlib importlib machinery FileFinder path}@anchor{3cb7}
@deffn {Attribute} path

The path the finder will search in.
@end deffn

@geindex find_spec() (importlib.machinery.FileFinder method)
@anchor{library/importlib importlib machinery FileFinder find_spec}@anchor{a38}
@deffn {Method} find_spec (fullname, target=None)

Attempt to find the spec to handle @emph{fullname} within @ref{3cb7,,path}.


New in version 3.4.

@end deffn

@geindex find_loader() (importlib.machinery.FileFinder method)
@anchor{library/importlib importlib machinery FileFinder find_loader}@anchor{30c}
@deffn {Method} find_loader (fullname)

Attempt to find the loader to handle @emph{fullname} within @ref{3cb7,,path}.


Deprecated since version 3.10: Use @ref{a38,,find_spec()} instead.

@end deffn

@geindex invalidate_caches() (importlib.machinery.FileFinder method)
@anchor{library/importlib importlib machinery FileFinder invalidate_caches}@anchor{3cb8}
@deffn {Method} invalidate_caches ()

Clear out the internal cache.
@end deffn

@geindex path_hook() (importlib.machinery.FileFinder class method)
@anchor{library/importlib importlib machinery FileFinder path_hook}@anchor{3cb9}
@deffn {Method} classmethod path_hook (*loader_details)

A class method which returns a closure for use on @ref{bb5,,sys.path_hooks}.
An instance of @ref{c07,,FileFinder} is returned by the closure using the
path argument given to the closure directly and @emph{loader_details}
indirectly.

If the argument to the closure is not an existing directory,
@ref{3b8,,ImportError} is raised.
@end deffn
@end deffn

@geindex SourceFileLoader (class in importlib.machinery)
@anchor{library/importlib importlib machinery SourceFileLoader}@anchor{c09}
@deffn {Class} importlib.machinery.SourceFileLoader (fullname, path)

A concrete implementation of @ref{3ca3,,importlib.abc.SourceLoader} by
subclassing @ref{c08,,importlib.abc.FileLoader} and providing some concrete
implementations of other methods.


New in version 3.3.


@geindex name (importlib.machinery.SourceFileLoader attribute)
@anchor{library/importlib importlib machinery SourceFileLoader name}@anchor{3cba}
@deffn {Attribute} name

The name of the module that this loader will handle.
@end deffn

@geindex path (importlib.machinery.SourceFileLoader attribute)
@anchor{library/importlib importlib machinery SourceFileLoader path}@anchor{3cbb}
@deffn {Attribute} path

The path to the source file.
@end deffn

@geindex is_package() (importlib.machinery.SourceFileLoader method)
@anchor{library/importlib importlib machinery SourceFileLoader is_package}@anchor{3cbc}
@deffn {Method} is_package (fullname)

Return @code{True} if @ref{3cbb,,path} appears to be for a package.
@end deffn

@geindex path_stats() (importlib.machinery.SourceFileLoader method)
@anchor{library/importlib importlib machinery SourceFileLoader path_stats}@anchor{3cbd}
@deffn {Method} path_stats (path)

Concrete implementation of @ref{d2a,,importlib.abc.SourceLoader.path_stats()}.
@end deffn

@geindex set_data() (importlib.machinery.SourceFileLoader method)
@anchor{library/importlib importlib machinery SourceFileLoader set_data}@anchor{3cbe}
@deffn {Method} set_data (path, data)

Concrete implementation of @ref{3ca4,,importlib.abc.SourceLoader.set_data()}.
@end deffn

@geindex load_module() (importlib.machinery.SourceFileLoader method)
@anchor{library/importlib importlib machinery SourceFileLoader load_module}@anchor{879}
@deffn {Method} load_module (name=None)

Concrete implementation of @ref{301,,importlib.abc.Loader.load_module()} where
specifying the name of the module to load is optional.


Deprecated since version 3.6: Use @ref{302,,importlib.abc.Loader.exec_module()} instead.

@end deffn
@end deffn

@geindex SourcelessFileLoader (class in importlib.machinery)
@anchor{library/importlib importlib machinery SourcelessFileLoader}@anchor{c0a}
@deffn {Class} importlib.machinery.SourcelessFileLoader (fullname, path)

A concrete implementation of @ref{c08,,importlib.abc.FileLoader} which can
import bytecode files (i.e. no source code files exist).

Please note that direct use of bytecode files (and thus not source code
files) inhibits your modules from being usable by all Python
implementations or new versions of Python which change the bytecode
format.


New in version 3.3.


@geindex name (importlib.machinery.SourcelessFileLoader attribute)
@anchor{library/importlib importlib machinery SourcelessFileLoader name}@anchor{3cbf}
@deffn {Attribute} name

The name of the module the loader will handle.
@end deffn

@geindex path (importlib.machinery.SourcelessFileLoader attribute)
@anchor{library/importlib importlib machinery SourcelessFileLoader path}@anchor{3cc0}
@deffn {Attribute} path

The path to the bytecode file.
@end deffn

@geindex is_package() (importlib.machinery.SourcelessFileLoader method)
@anchor{library/importlib importlib machinery SourcelessFileLoader is_package}@anchor{3cc1}
@deffn {Method} is_package (fullname)

Determines if the module is a package based on @ref{3cc0,,path}.
@end deffn

@geindex get_code() (importlib.machinery.SourcelessFileLoader method)
@anchor{library/importlib importlib machinery SourcelessFileLoader get_code}@anchor{3cc2}
@deffn {Method} get_code (fullname)

Returns the code object for @ref{3cbf,,name} created from @ref{3cc0,,path}.
@end deffn

@geindex get_source() (importlib.machinery.SourcelessFileLoader method)
@anchor{library/importlib importlib machinery SourcelessFileLoader get_source}@anchor{3cc3}
@deffn {Method} get_source (fullname)

Returns @code{None} as bytecode files have no source when this loader is
used.
@end deffn

@geindex load_module() (importlib.machinery.SourcelessFileLoader method)
@anchor{library/importlib importlib machinery SourcelessFileLoader load_module}@anchor{87a}
@deffn {Method} load_module (name=None)
@end deffn

Concrete implementation of @ref{301,,importlib.abc.Loader.load_module()} where
specifying the name of the module to load is optional.


Deprecated since version 3.6: Use @ref{302,,importlib.abc.Loader.exec_module()} instead.

@end deffn

@geindex ExtensionFileLoader (class in importlib.machinery)
@anchor{library/importlib importlib machinery ExtensionFileLoader}@anchor{7f8}
@deffn {Class} importlib.machinery.ExtensionFileLoader (fullname, path)

A concrete implementation of @ref{3c9d,,importlib.abc.ExecutionLoader} for
extension modules.

The @emph{fullname} argument specifies the name of the module the loader is to
support. The @emph{path} argument is the path to the extension module’s file.


New in version 3.3.


@geindex name (importlib.machinery.ExtensionFileLoader attribute)
@anchor{library/importlib importlib machinery ExtensionFileLoader name}@anchor{3cc4}
@deffn {Attribute} name

Name of the module the loader supports.
@end deffn

@geindex path (importlib.machinery.ExtensionFileLoader attribute)
@anchor{library/importlib importlib machinery ExtensionFileLoader path}@anchor{3cc5}
@deffn {Attribute} path

Path to the extension module.
@end deffn

@geindex create_module() (importlib.machinery.ExtensionFileLoader method)
@anchor{library/importlib importlib machinery ExtensionFileLoader create_module}@anchor{3cc6}
@deffn {Method} create_module (spec)

Creates the module object from the given specification in accordance
with 
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}.


New in version 3.5.

@end deffn

@geindex exec_module() (importlib.machinery.ExtensionFileLoader method)
@anchor{library/importlib importlib machinery ExtensionFileLoader exec_module}@anchor{3cc7}
@deffn {Method} exec_module (module)

Initializes the given module object in accordance with 
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}.


New in version 3.5.

@end deffn

@geindex is_package() (importlib.machinery.ExtensionFileLoader method)
@anchor{library/importlib importlib machinery ExtensionFileLoader is_package}@anchor{3cc8}
@deffn {Method} is_package (fullname)

Returns @code{True} if the file path points to a package’s @code{__init__}
module based on @ref{3cb6,,EXTENSION_SUFFIXES}.
@end deffn

@geindex get_code() (importlib.machinery.ExtensionFileLoader method)
@anchor{library/importlib importlib machinery ExtensionFileLoader get_code}@anchor{3cc9}
@deffn {Method} get_code (fullname)

Returns @code{None} as extension modules lack a code object.
@end deffn

@geindex get_source() (importlib.machinery.ExtensionFileLoader method)
@anchor{library/importlib importlib machinery ExtensionFileLoader get_source}@anchor{3cca}
@deffn {Method} get_source (fullname)

Returns @code{None} as extension modules do not have source code.
@end deffn

@geindex get_filename() (importlib.machinery.ExtensionFileLoader method)
@anchor{library/importlib importlib machinery ExtensionFileLoader get_filename}@anchor{ad7}
@deffn {Method} get_filename (fullname)

Returns @ref{3cc5,,path}.


New in version 3.4.

@end deffn
@end deffn


@deffn {Class} NamespaceLoader(name, path, path_finder):

A concrete implementation of @ref{ad2,,importlib.abc.InspectLoader} for
namespace packages.  This is an alias for a private class and is only made
public for introspecting the @code{__loader__} attribute on namespace
packages:

@example
>>> from importlib.machinery import NamespaceLoader
>>> import my_namespace
>>> isinstance(my_namespace.__loader__, NamespaceLoader)
True
>>> import importlib.abc
>>> isinstance(my_namespace.__loader__, importlib.abc.Loader)
True
@end example


New in version 3.11.

@end deffn

@geindex ModuleSpec (class in importlib.machinery)
@anchor{library/importlib importlib machinery ModuleSpec}@anchor{1853}
@deffn {Class} importlib.machinery.ModuleSpec (name, loader, *, origin=None, loader_state=None, is_package=None)

A specification for a module’s import-system-related state.  This is
typically exposed as the module’s @ref{1858,,__spec__} attribute.  In the
descriptions below, the names in parentheses give the corresponding
attribute available directly on the module object,
e.g. @code{module.__spec__.origin == module.__file__}.  Note, however, that
while the @emph{values} are usually equivalent, they can differ since there is
no synchronization between the two objects.  For example, it is possible to update
the module’s @ref{1797,,__file__} at runtime and this will not be automatically
reflected in the module’s @code{__spec__.origin}, and vice versa.


New in version 3.4.


@geindex name (importlib.machinery.ModuleSpec attribute)
@anchor{library/importlib importlib machinery ModuleSpec name}@anchor{1e42}
@deffn {Attribute} name
@end deffn

(@ref{f12,,__name__})

The module’s fully-qualified name.
The @ref{1847,,finder} should always set this attribute to a non-empty string.

@geindex loader (importlib.machinery.ModuleSpec attribute)
@anchor{library/importlib importlib machinery ModuleSpec loader}@anchor{1e40}
@deffn {Attribute} loader
@end deffn

(@ref{baf,,__loader__})

The @ref{1848,,loader} used to load the module.
The @ref{1847,,finder} should always set this attribute.

@geindex origin (importlib.machinery.ModuleSpec attribute)
@anchor{library/importlib importlib machinery ModuleSpec origin}@anchor{3ccb}
@deffn {Attribute} origin
@end deffn

(@ref{1797,,__file__})

The location the @ref{1848,,loader} should use to load the module.
For example, for modules loaded from a .py file this is the filename.
The @ref{1847,,finder} should always set this attribute to a meaningful value
for the @ref{1848,,loader} to use.  In the uncommon case that there is not one
(like for namespace packages), it should be set to @code{None}.

@geindex submodule_search_locations (importlib.machinery.ModuleSpec attribute)
@anchor{library/importlib importlib machinery ModuleSpec submodule_search_locations}@anchor{3ccc}
@deffn {Attribute} submodule_search_locations
@end deffn

(@ref{1550,,__path__})

The list of locations where the package’s submodules will be found.
Most of the time this is a single directory.
The @ref{1847,,finder} should set this attribute to a list, even an empty one, to indicate
to the import system that the module is a package.  It should be set to @code{None} for
non-package modules.  It is set automatically later to a special object for
namespace packages.

@geindex loader_state (importlib.machinery.ModuleSpec attribute)
@anchor{library/importlib importlib machinery ModuleSpec loader_state}@anchor{3ccd}
@deffn {Attribute} loader_state
@end deffn

The @ref{1847,,finder} may set this attribute to an object containing additional,
module-specific data to use when loading the module.  Otherwise it should be
set to @code{None}.

@geindex cached (importlib.machinery.ModuleSpec attribute)
@anchor{library/importlib importlib machinery ModuleSpec cached}@anchor{3cce}
@deffn {Attribute} cached
@end deffn

(@ref{d5b,,__cached__})

The filename of a compiled version of the module’s code.
The @ref{1847,,finder} should always set this attribute but it may be @code{None}
for modules that do not need compiled code stored.

@geindex parent (importlib.machinery.ModuleSpec attribute)
@anchor{library/importlib importlib machinery ModuleSpec parent}@anchor{1e44}
@deffn {Attribute} parent
@end deffn

(@ref{bae,,__package__})

(Read-only) The fully-qualified name of the package the module is in (or the
empty string for a top-level module).
If the module is a package then this is the same as @ref{1e42,,name}.

@geindex has_location (importlib.machinery.ModuleSpec attribute)
@anchor{library/importlib importlib machinery ModuleSpec has_location}@anchor{3ccf}
@deffn {Attribute} has_location
@end deffn


@table @asis

@item @code{True} if the spec’s @ref{3ccb,,origin} refers to a loadable location,

@code{False} otherwise.  This value impacts how @ref{3ccb,,origin} is interpreted
and how the module’s @ref{1797,,__file__} is populated.
@end table
@end deffn

@node importlib util – Utility code for importers,Examples<32>,importlib machinery – Importers and path hooks,importlib — The implementation of import
@anchor{library/importlib importlib-util-utility-code-for-importers}@anchor{3cd0}@anchor{library/importlib module-importlib util}@anchor{9b}
@subsubsection @code{importlib.util} – Utility code for importers


@geindex module; importlib.util

@strong{Source code:} Lib/importlib/util.py@footnote{https://github.com/python/cpython/tree/main/Lib/importlib/util.py}


__________________________________________________________________


This module contains the various objects that help in the construction of
an @ref{1849,,importer}.

@geindex MAGIC_NUMBER (in module importlib.util)
@anchor{library/importlib importlib util MAGIC_NUMBER}@anchor{ad4}
@deffn {Attribute} importlib.util.MAGIC_NUMBER

The bytes which represent the bytecode version number. If you need help with
loading/writing bytecode then consider @ref{3ca3,,importlib.abc.SourceLoader}.


New in version 3.4.

@end deffn

@geindex cache_from_source() (in module importlib.util)
@anchor{library/importlib importlib util cache_from_source}@anchor{7f9}
@deffn {Function} importlib.util.cache_from_source (path, debug_override=None, *, optimization=None)

Return the 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147}/
@geindex Python Enhancement Proposals; PEP 488
PEP 488@footnote{https://www.python.org/dev/peps/pep-0488} path to the byte-compiled file associated
with the source @emph{path}.  For example, if @emph{path} is @code{/foo/bar/baz.py} the return
value would be @code{/foo/bar/__pycache__/baz.cpython-32.pyc} for Python 3.2.
The @code{cpython-32} string comes from the current magic tag (see
@code{get_tag()}; if @code{sys.implementation.cache_tag} is not defined then
@ref{8a3,,NotImplementedError} will be raised).

The @emph{optimization} parameter is used to specify the optimization level of the
bytecode file. An empty string represents no optimization, so
@code{/foo/bar/baz.py} with an @emph{optimization} of @code{'@w{'}} will result in a
bytecode path of @code{/foo/bar/__pycache__/baz.cpython-32.pyc}. @code{None} causes
the interpreter’s optimization level to be used. Any other value’s string
representation is used, so @code{/foo/bar/baz.py} with an @emph{optimization} of
@code{2} will lead to the bytecode path of
@code{/foo/bar/__pycache__/baz.cpython-32.opt-2.pyc}. The string representation
of @emph{optimization} can only be alphanumeric, else @ref{17c,,ValueError} is raised.

The @emph{debug_override} parameter is deprecated and can be used to override
the system’s value for @code{__debug__}. A @code{True} value is the equivalent of
setting @emph{optimization} to the empty string. A @code{False} value is the same as
setting @emph{optimization} to @code{1}. If both @emph{debug_override} an @emph{optimization}
are not @code{None} then @ref{162,,TypeError} is raised.


New in version 3.4.



Changed in version 3.5: The @emph{optimization} parameter was added and the @emph{debug_override} parameter
was deprecated.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex source_from_cache() (in module importlib.util)
@anchor{library/importlib importlib util source_from_cache}@anchor{7fa}
@deffn {Function} importlib.util.source_from_cache (path)

Given the @emph{path} to a 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} file name, return the associated source code
file path.  For example, if @emph{path} is
@code{/foo/bar/__pycache__/baz.cpython-32.pyc} the returned path would be
@code{/foo/bar/baz.py}.  @emph{path} need not exist, however if it does not conform
to 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} or 
@geindex Python Enhancement Proposals; PEP 488
PEP 488@footnote{https://www.python.org/dev/peps/pep-0488} format, a @ref{17c,,ValueError} is raised. If
@code{sys.implementation.cache_tag} is not defined,
@ref{8a3,,NotImplementedError} is raised.


New in version 3.4.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex decode_source() (in module importlib.util)
@anchor{library/importlib importlib util decode_source}@anchor{ad6}
@deffn {Function} importlib.util.decode_source (source_bytes)

Decode the given bytes representing source code and return it as a string
with universal newlines (as required by
@ref{3c9b,,importlib.abc.InspectLoader.get_source()}).


New in version 3.4.

@end deffn

@geindex resolve_name() (in module importlib.util)
@anchor{library/importlib importlib util resolve_name}@anchor{3f9}
@deffn {Function} importlib.util.resolve_name (name, package)

Resolve a relative module name to an absolute one.

If  @strong{name} has no leading dots, then @strong{name} is simply returned. This
allows for usage such as
@code{importlib.util.resolve_name('sys', __spec__.parent)} without doing a
check to see if the @strong{package} argument is needed.

@ref{3b8,,ImportError} is raised if @strong{name} is a relative module name but
@strong{package} is a false value (e.g. @code{None} or the empty string).
@ref{3b8,,ImportError} is also raised if a relative name would escape its
containing package (e.g. requesting @code{..bacon} from within the @code{spam}
package).


New in version 3.3.



Changed in version 3.9: To improve consistency with import statements, raise
@ref{3b8,,ImportError} instead of @ref{17c,,ValueError} for invalid relative
import attempts.

@end deffn

@geindex find_spec() (in module importlib.util)
@anchor{library/importlib importlib util find_spec}@anchor{b98}
@deffn {Function} importlib.util.find_spec (name, package=None)

Find the @ref{3b75,,spec} for a module, optionally relative to
the specified @strong{package} name. If the module is in @ref{11f7,,sys.modules},
then @code{sys.modules[name].__spec__} is returned (unless the spec would be
@code{None} or is not set, in which case @ref{17c,,ValueError} is raised).
Otherwise a search using @ref{87c,,sys.meta_path} is done. @code{None} is
returned if no spec is found.

If @strong{name} is for a submodule (contains a dot), the parent module is
automatically imported.

@strong{name} and @strong{package} work the same as for @code{import_module()}.


New in version 3.4.



Changed in version 3.7: Raises @ref{66b,,ModuleNotFoundError} instead of @ref{163,,AttributeError} if
@strong{package} is in fact not a package (i.e. lacks a @ref{1550,,__path__}
attribute).

@end deffn

@geindex module_from_spec() (in module importlib.util)
@anchor{library/importlib importlib util module_from_spec}@anchor{972}
@deffn {Function} importlib.util.module_from_spec (spec)

Create a new module based on @strong{spec} and
@ref{7f6,,spec.loader.create_module}.

If @ref{7f6,,spec.loader.create_module}
does not return @code{None}, then any pre-existing attributes will not be reset.
Also, no @ref{163,,AttributeError} will be raised if triggered while accessing
@strong{spec} or setting an attribute on the module.

This function is preferred over using @ref{973,,types.ModuleType} to create a
new module as @strong{spec} is used to set as many import-controlled attributes on
the module as possible.


New in version 3.5.

@end deffn

@geindex module_for_loader() (in module importlib.util)
@anchor{library/importlib importlib util module_for_loader}@anchor{311}
@deffn {Function} @@importlib.util.module_for_loader

A @ref{1bf,,decorator} for @ref{301,,importlib.abc.Loader.load_module()}
to handle selecting the proper
module object to load with. The decorated method is expected to have a call
signature taking two positional arguments
(e.g. @code{load_module(self, module)}) for which the second argument
will be the module @strong{object} to be used by the loader.
Note that the decorator will not work on static methods because of the
assumption of two arguments.

The decorated method will take in the @strong{name} of the module to be loaded
as expected for a @ref{1848,,loader}. If the module is not found in
@ref{11f7,,sys.modules} then a new one is constructed. Regardless of where the
module came from, @ref{baf,,__loader__} set to @strong{self} and @ref{bae,,__package__}
is set based on what @ref{3c9c,,importlib.abc.InspectLoader.is_package()} returns
(if available). These attributes are set unconditionally to support
reloading.

If an exception is raised by the decorated method and a module was added to
@ref{11f7,,sys.modules}, then the module will be removed to prevent a partially
initialized module from being in left in @ref{11f7,,sys.modules}. If the module
was already in @ref{11f7,,sys.modules} then it is left alone.


Changed in version 3.3: @ref{baf,,__loader__} and @ref{bae,,__package__} are automatically set
(when possible).



Changed in version 3.4: Set @ref{f12,,__name__}, @ref{baf,,__loader__} @ref{bae,,__package__}
unconditionally to support reloading.



Deprecated since version 3.4: The import machinery now directly performs all the functionality
provided by this function.

@end deffn

@geindex set_loader() (in module importlib.util)
@anchor{library/importlib importlib util set_loader}@anchor{b9f}
@deffn {Function} @@importlib.util.set_loader

A @ref{1bf,,decorator} for @ref{301,,importlib.abc.Loader.load_module()}
to set the @ref{baf,,__loader__}
attribute on the returned module. If the attribute is already set the
decorator does nothing. It is assumed that the first positional argument to
the wrapped method (i.e. @code{self}) is what @ref{baf,,__loader__} should be set
to.


Changed in version 3.4: Set @code{__loader__} if set to @code{None}, as if the attribute does not
exist.



Deprecated since version 3.4: The import machinery takes care of this automatically.

@end deffn

@geindex set_package() (in module importlib.util)
@anchor{library/importlib importlib util set_package}@anchor{ba0}
@deffn {Function} @@importlib.util.set_package

A @ref{1bf,,decorator} for @ref{301,,importlib.abc.Loader.load_module()} to set the
@ref{bae,,__package__} attribute on the returned module. If @ref{bae,,__package__}
is set and has a value other than @code{None} it will not be changed.


Deprecated since version 3.4: The import machinery takes care of this automatically.

@end deffn

@geindex spec_from_loader() (in module importlib.util)
@anchor{library/importlib importlib util spec_from_loader}@anchor{30a}
@deffn {Function} importlib.util.spec_from_loader (name, loader, *, origin=None, is_package=None)

A factory function for creating a @ref{1853,,ModuleSpec}
instance based on a loader.  The parameters have the same meaning as they do
for ModuleSpec.  The function uses available @ref{1848,,loader} APIs, such as
@code{InspectLoader.is_package()}, to fill in any missing
information on the spec.


New in version 3.4.

@end deffn

@geindex spec_from_file_location() (in module importlib.util)
@anchor{library/importlib importlib util spec_from_file_location}@anchor{7fb}
@deffn {Function} importlib.util.spec_from_file_location (name, location, *, loader=None, submodule_search_locations=None)

A factory function for creating a @ref{1853,,ModuleSpec}
instance based on the path to a file.  Missing information will be filled in
on the spec by making use of loader APIs and by the implication that the
module will be file-based.


New in version 3.4.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex source_hash() (in module importlib.util)
@anchor{library/importlib importlib util source_hash}@anchor{3cd1}
@deffn {Function} importlib.util.source_hash (source_bytes)

Return the hash of @emph{source_bytes} as bytes. A hash-based @code{.pyc} file embeds
the @ref{3cd1,,source_hash()} of the corresponding source file’s contents in its
header.


New in version 3.7.

@end deffn

@geindex LazyLoader (class in importlib.util)
@anchor{library/importlib importlib util LazyLoader}@anchor{7f5}
@deffn {Class} importlib.util.LazyLoader (loader)

A class which postpones the execution of the loader of a module until the
module has an attribute accessed.

This class @strong{only} works with loaders that define
@ref{302,,exec_module()} as control over what module type
is used for the module is required. For those same reasons, the loader’s
@ref{7f6,,create_module()} method must return @code{None} or a
type for which its @code{__class__} attribute can be mutated along with not
using @ref{3cd2,,slots}. Finally, modules which substitute the object
placed into @ref{11f7,,sys.modules} will not work as there is no way to properly
replace the module references throughout the interpreter safely;
@ref{17c,,ValueError} is raised if such a substitution is detected.

@cartouche
@quotation Note 
For projects where startup time is critical, this class allows for
potentially minimizing the cost of loading a module if it is never used.
For projects where startup time is not essential then use of this class is
@strong{heavily} discouraged due to error messages created during loading being
postponed and thus occurring out of context.
@end quotation
@end cartouche


New in version 3.5.



Changed in version 3.6: Began calling @ref{7f6,,create_module()}, removing the
compatibility warning for @ref{7f7,,importlib.machinery.BuiltinImporter} and
@ref{7f8,,importlib.machinery.ExtensionFileLoader}.


@geindex factory() (importlib.util.LazyLoader class method)
@anchor{library/importlib importlib util LazyLoader factory}@anchor{3cd3}
@deffn {Method} classmethod factory (loader)

A static method which returns a callable that creates a lazy loader. This
is meant to be used in situations where the loader is passed by class
instead of by instance.

@example
suffixes = importlib.machinery.SOURCE_SUFFIXES
loader = importlib.machinery.SourceFileLoader
lazy_loader = importlib.util.LazyLoader.factory(loader)
finder = importlib.machinery.FileFinder(path, (lazy_loader, suffixes))
@end example
@end deffn
@end deffn

@node Examples<32>,,importlib util – Utility code for importers,importlib — The implementation of import
@anchor{library/importlib examples}@anchor{3cd4}@anchor{library/importlib importlib-examples}@anchor{3cd5}
@subsubsection Examples


@menu
* Importing programmatically:: 
* Checking if a module can be imported:: 
* Importing a source file directly:: 
* Implementing lazy imports:: 
* Setting up an importer:: 
* Approximating importlib.import_module(): Approximating importlib import_module. 

@end menu

@node Importing programmatically,Checking if a module can be imported,,Examples<32>
@anchor{library/importlib importing-programmatically}@anchor{3cd6}
@subsubsection Importing programmatically


To programmatically import a module, use @ref{d3e,,importlib.import_module()}.

@example
import importlib

itertools = importlib.import_module('itertools')
@end example

@node Checking if a module can be imported,Importing a source file directly,Importing programmatically,Examples<32>
@anchor{library/importlib checking-if-a-module-can-be-imported}@anchor{3cd7}
@subsubsection Checking if a module can be imported


If you need to find out if a module can be imported without actually doing the
import, then you should use @ref{b98,,importlib.util.find_spec()}.

@example
import importlib.util
import sys

# For illustrative purposes.
name = 'itertools'

if name in sys.modules:
    print(f"@{name!r@} already in sys.modules")
elif (spec := importlib.util.find_spec(name)) is not None:
    # If you chose to perform the actual import ...
    module = importlib.util.module_from_spec(spec)
    sys.modules[name] = module
    spec.loader.exec_module(module)
    print(f"@{name!r@} has been imported")
else:
    print(f"can't find the @{name!r@} module")
@end example

@node Importing a source file directly,Implementing lazy imports,Checking if a module can be imported,Examples<32>
@anchor{library/importlib importing-a-source-file-directly}@anchor{3cd8}
@subsubsection Importing a source file directly


To import a Python source file directly, use the following recipe
(Python 3.5 and newer only):

@example
import importlib.util
import sys

# For illustrative purposes.
import tokenize
file_path = tokenize.__file__
module_name = tokenize.__name__

spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
sys.modules[module_name] = module
spec.loader.exec_module(module)
@end example

@node Implementing lazy imports,Setting up an importer,Importing a source file directly,Examples<32>
@anchor{library/importlib implementing-lazy-imports}@anchor{3cd9}
@subsubsection Implementing lazy imports


The example below shows how to implement lazy imports:

@example
>>> import importlib.util
>>> import sys
>>> def lazy_import(name):
...     spec = importlib.util.find_spec(name)
...     loader = importlib.util.LazyLoader(spec.loader)
...     spec.loader = loader
...     module = importlib.util.module_from_spec(spec)
...     sys.modules[name] = module
...     loader.exec_module(module)
...     return module
...
>>> lazy_typing = lazy_import("typing")
>>> #lazy_typing is a real module object,
>>> #but it is not loaded in memory yet.
>>> lazy_typing.TYPE_CHECKING
False
@end example

@node Setting up an importer,Approximating importlib import_module,Implementing lazy imports,Examples<32>
@anchor{library/importlib setting-up-an-importer}@anchor{3cda}
@subsubsection Setting up an importer


For deep customizations of import, you typically want to implement an
@ref{1849,,importer}. This means managing both the @ref{1847,,finder} and @ref{1848,,loader}
side of things. For finders there are two flavours to choose from depending on
your needs: a @ref{c05,,meta path finder} or a @ref{c06,,path entry finder}. The
former is what you would put on @ref{87c,,sys.meta_path} while the latter is what
you create using a @ref{1866,,path entry hook} on @ref{bb5,,sys.path_hooks} which works
with @ref{489,,sys.path} entries to potentially create a finder. This example will
show you how to register your own importers so that import will use them (for
creating an importer for yourself, read the documentation for the appropriate
classes defined within this package):

@example
import importlib.machinery
import sys

# For illustrative purposes only.
SpamMetaPathFinder = importlib.machinery.PathFinder
SpamPathEntryFinder = importlib.machinery.FileFinder
loader_details = (importlib.machinery.SourceFileLoader,
                  importlib.machinery.SOURCE_SUFFIXES)

# Setting up a meta path finder.
# Make sure to put the finder in the proper location in the list in terms of
# priority.
sys.meta_path.append(SpamMetaPathFinder)

# Setting up a path entry finder.
# Make sure to put the path hook in the proper location in the list in terms
# of priority.
sys.path_hooks.append(SpamPathEntryFinder.path_hook(loader_details))
@end example

@node Approximating importlib import_module,,Setting up an importer,Examples<32>
@anchor{library/importlib approximating-importlib-import-module}@anchor{3cdb}
@subsubsection Approximating @code{importlib.import_module()}


Import itself is implemented in Python code, making it possible to
expose most of the import machinery through importlib. The following
helps illustrate the various APIs that importlib exposes by providing an
approximate implementation of
@ref{d3e,,importlib.import_module()} (Python 3.4 and newer for the importlib usage,
Python 3.6 and newer for other parts of the code).

@example
import importlib.util
import sys

def import_module(name, package=None):
    """An approximate implementation of import."""
    absolute_name = importlib.util.resolve_name(name, package)
    try:
        return sys.modules[absolute_name]
    except KeyError:
        pass

    path = None
    if '.' in absolute_name:
        parent_name, _, child_name = absolute_name.rpartition('.')
        parent_module = import_module(parent_name)
        path = parent_module.__spec__.submodule_search_locations
    for finder in sys.meta_path:
        spec = finder.find_spec(absolute_name, path)
        if spec is not None:
            break
    else:
        msg = f'No module named @{absolute_name!r@}'
        raise ModuleNotFoundError(msg, name=absolute_name)
    module = importlib.util.module_from_spec(spec)
    sys.modules[absolute_name] = module
    spec.loader.exec_module(module)
    if path is not None:
        setattr(parent_module, child_name, module)
    return module
@end example

@node importlib resources – Resources,Using importlib metadata,importlib — The implementation of import,Importing Modules
@anchor{library/importlib resources doc}@anchor{3cdc}@anchor{library/importlib resources importlib-resources-resources}@anchor{3cdd}@anchor{library/importlib resources module-importlib resources}@anchor{9a}
@subsection @code{importlib.resources} – Resources


@geindex module; importlib.resources

@strong{Source code:} Lib/importlib/resources.py@footnote{https://github.com/python/cpython/tree/main/Lib/importlib/resources.py}


__________________________________________________________________



New in version 3.7.


This module leverages Python’s import system to provide access to @emph{resources}
within @emph{packages}.  If you can import a package, you can access resources
within that package.  Resources can be opened or read, in either binary or
text mode.

Resources are roughly akin to files inside directories, though it’s important
to keep in mind that this is just a metaphor.  Resources and packages @strong{do
not} have to exist as physical files and directories on the file system.

@cartouche
@quotation Note 
This module provides functionality similar to pkg_resources@footnote{https://setuptools.readthedocs.io/en/latest/pkg_resources.html} Basic Resource Access@footnote{http://setuptools.readthedocs.io/en/latest/pkg_resources.html#basic-resource-access}
without the performance overhead of that package.  This makes reading
resources included in packages easier, with more stable and consistent
semantics.

The standalone backport of this module provides more information
on using importlib.resources@footnote{http://importlib-resources.readthedocs.io/en/latest/using.html} and
migrating from pkg_resources to importlib.resources@footnote{http://importlib-resources.readthedocs.io/en/latest/migration.html}
and
migrating legacy usage@footnote{https://importlib-resources.readthedocs.io/en/latest/using.html#migrating-from-legacy}.
@end quotation
@end cartouche

Loaders that wish to support resource reading should implement a
@code{get_resource_reader(fullname)} method as specified by
@ref{61b,,importlib.abc.ResourceReader}.

The following types are defined.

@geindex Package (in module importlib.resources)
@anchor{library/importlib resources importlib resources Package}@anchor{3cde}
@deffn {Data} importlib.resources.Package

The @code{Package} type is defined as @code{Union[str, ModuleType]}.  This means
that where the function describes accepting a @code{Package}, you can pass in
either a string or a module.  Module objects must have a resolvable
@code{__spec__.submodule_search_locations} that is not @code{None}.
@end deffn

@geindex Resource (in module importlib.resources)
@anchor{library/importlib resources importlib resources Resource}@anchor{3cdf}
@deffn {Data} importlib.resources.Resource

This type describes the resource names passed into the various functions
in this package.  This is defined as @code{Union[str, os.PathLike]}.
@end deffn

The following functions are available.

@geindex files() (in module importlib.resources)
@anchor{library/importlib resources importlib resources files}@anchor{3fa}
@deffn {Function} importlib.resources.files (package)

Returns an @code{importlib.resources.abc.Traversable} object
representing the resource container for the package (think directory)
and its resources (think files). A Traversable may contain other
containers (think subdirectories).

@emph{package} is either a name or a module object which conforms to the
@code{Package} requirements.


New in version 3.9.

@end deffn

@geindex as_file() (in module importlib.resources)
@anchor{library/importlib resources importlib resources as_file}@anchor{3ce0}
@deffn {Function} importlib.resources.as_file (traversable)

Given a @code{importlib.resources.abc.Traversable} object representing
a file, typically from @ref{3fa,,importlib.resources.files()}, return
a context manager for use in a @ref{168,,with} statement.
The context manager provides a @ref{522,,pathlib.Path} object.

Exiting the context manager cleans up any temporary file created when the
resource was extracted from e.g. a zip file.

Use @code{as_file} when the Traversable methods
(@code{read_text}, etc) are insufficient and an actual file on
the file system is required.


New in version 3.9.

@end deffn

@geindex open_binary() (in module importlib.resources)
@anchor{library/importlib resources importlib resources open_binary}@anchor{3ce1}
@deffn {Function} importlib.resources.open_binary (package, resource)

@quotation

Open for binary reading the @emph{resource} within @emph{package}.

@emph{package} is either a name or a module object which conforms to the
@code{Package} requirements.  @emph{resource} is the name of the resource to open
within @emph{package}; it may not contain path separators and it may not have
sub-resources (i.e. it cannot be a directory).  This function returns a
@code{typing.BinaryIO} instance, a binary I/O stream open for reading.
@end quotation


Deprecated since version 3.11.

@end deffn

@geindex open_text() (in module importlib.resources)
@anchor{library/importlib resources importlib resources open_text}@anchor{3ce2}
@deffn {Function} importlib.resources.open_text (package, resource, encoding='utf-8', errors='strict')

@quotation

Open for text reading the @emph{resource} within @emph{package}.  By default, the
resource is opened for reading as UTF-8.

@emph{package} is either a name or a module object which conforms to the
@code{Package} requirements.  @emph{resource} is the name of the resource to open
within @emph{package}; it may not contain path separators and it may not have
sub-resources (i.e. it cannot be a directory).  @emph{encoding} and @emph{errors}
have the same meaning as with built-in @ref{1c8,,open()}.

This function returns a @code{typing.TextIO} instance, a text I/O stream open
for reading.
@end quotation


Deprecated since version 3.11.

@end deffn

@geindex read_binary() (in module importlib.resources)
@anchor{library/importlib resources importlib resources read_binary}@anchor{1267}
@deffn {Function} importlib.resources.read_binary (package, resource)

@quotation

Read and return the contents of the @emph{resource} within @emph{package} as
@code{bytes}.

@emph{package} is either a name or a module object which conforms to the
@code{Package} requirements.  @emph{resource} is the name of the resource to open
within @emph{package}; it may not contain path separators and it may not have
sub-resources (i.e. it cannot be a directory).  This function returns the
contents of the resource as @ref{2f0,,bytes}.
@end quotation


Deprecated since version 3.11.

@end deffn

@geindex read_text() (in module importlib.resources)
@anchor{library/importlib resources importlib resources read_text}@anchor{3ce3}
@deffn {Function} importlib.resources.read_text (package, resource, encoding='utf-8', errors='strict')

@quotation

Read and return the contents of @emph{resource} within @emph{package} as a @code{str}.
By default, the contents are read as strict UTF-8.

@emph{package} is either a name or a module object which conforms to the
@code{Package} requirements.  @emph{resource} is the name of the resource to open
within @emph{package}; it may not contain path separators and it may not have
sub-resources (i.e. it cannot be a directory).  @emph{encoding} and @emph{errors}
have the same meaning as with built-in @ref{1c8,,open()}.  This function
returns the contents of the resource as @ref{2ef,,str}.
@end quotation


Deprecated since version 3.11.

@end deffn

@geindex path() (in module importlib.resources)
@anchor{library/importlib resources importlib resources path}@anchor{3ce4}
@deffn {Function} importlib.resources.path (package, resource)

@quotation

Return the path to the @emph{resource} as an actual file system path.  This
function returns a context manager for use in a @ref{168,,with} statement.
The context manager provides a @ref{522,,pathlib.Path} object.

Exiting the context manager cleans up any temporary file created when the
resource needs to be extracted from e.g. a zip file.

@emph{package} is either a name or a module object which conforms to the
@code{Package} requirements.  @emph{resource} is the name of the resource to open
within @emph{package}; it may not contain path separators and it may not have
sub-resources (i.e. it cannot be a directory).
@end quotation


Deprecated since version 3.11.

@end deffn

@geindex is_resource() (in module importlib.resources)
@anchor{library/importlib resources importlib resources is_resource}@anchor{3ce5}
@deffn {Function} importlib.resources.is_resource (package, name)

@quotation

Return @code{True} if there is a resource named @emph{name} in the package,
otherwise @code{False}.  Remember that directories are @emph{not} resources!
@emph{package} is either a name or a module object which conforms to the
@code{Package} requirements.
@end quotation


Deprecated since version 3.11.

@end deffn

@geindex contents() (in module importlib.resources)
@anchor{library/importlib resources importlib resources contents}@anchor{12fe}
@deffn {Function} importlib.resources.contents (package)

@quotation

Return an iterable over the named items within the package.  The iterable
returns @ref{2ef,,str} resources (e.g. files) and non-resources
(e.g. directories).  The iterable does not recurse into subdirectories.

@emph{package} is either a name or a module object which conforms to the
@code{Package} requirements.
@end quotation


Deprecated since version 3.11.

@end deffn

@node Using importlib metadata,,importlib resources – Resources,Importing Modules
@anchor{library/importlib metadata doc}@anchor{3ce6}@anchor{library/importlib metadata module-importlib metadata}@anchor{99}@anchor{library/importlib metadata using}@anchor{3ce7}@anchor{library/importlib metadata using-importlib-metadata}@anchor{3ce8}
@subsection Using @code{importlib.metadata}


@geindex module; importlib.metadata


New in version 3.8.



Changed in version 3.10: @code{importlib.metadata} is no longer provisional.


@strong{Source code:} Lib/importlib/metadata/__init__.py@footnote{https://github.com/python/cpython/tree/main/Lib/importlib/metadata/__init__.py}

@code{importlib.metadata} is a library that provides for access to installed
package metadata.  Built in part on Python’s import system, this library
intends to replace similar functionality in the entry point API@footnote{https://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points} and metadata API@footnote{https://setuptools.readthedocs.io/en/latest/pkg_resources.html#metadata-api} of @code{pkg_resources}.  Along with
@ref{9a,,importlib.resources} in Python 3.7
and newer (backported as importlib_resources@footnote{https://importlib-resources.readthedocs.io/en/latest/index.html} for older versions of
Python), this can eliminate the need to use the older and less efficient
@code{pkg_resources} package.

By “installed package” we generally mean a third-party package installed into
Python’s @code{site-packages} directory via tools such as pip@footnote{https://pypi.org/project/pip/}.  Specifically,
it means a package with either a discoverable @code{dist-info} or @code{egg-info}
directory, and metadata defined by 
@geindex Python Enhancement Proposals; PEP 566
PEP 566@footnote{https://www.python.org/dev/peps/pep-0566} or its older specifications.
By default, package metadata can live on the file system or in zip archives on
@ref{489,,sys.path}.  Through an extension mechanism, the metadata can live almost
anywhere.

@menu
* Overview: Overview<3>. 
* Functional API:: 
* Distributions:: 
* Extending the search algorithm:: 

@end menu

@node Overview<3>,Functional API,,Using importlib metadata
@anchor{library/importlib metadata overview}@anchor{3ce9}
@subsubsection Overview


Let’s say you wanted to get the version string for a package you’ve installed
using @code{pip}.  We start by creating a virtual environment and installing
something into it:

@example
$ python3 -m venv example
$ source example/bin/activate
(example) $ pip install wheel
@end example

You can get the version string for @code{wheel} by running the following:

@example
(example) $ python
>>> from importlib.metadata import version  
>>> version('wheel')  
'0.32.3'
@end example

You can also get the set of entry points keyed by group, such as
@code{console_scripts}, @code{distutils.commands} and others.  Each group contains a
sequence of @ref{293,,EntryPoint} objects.

You can get the @ref{3cea,,metadata for a distribution}:

@example
>>> list(metadata('wheel'))  
['Metadata-Version', 'Name', 'Version', 'Summary', 'Home-page', 'Author', 'Author-email', 'Maintainer', 'Maintainer-email', 'License', 'Project-URL', 'Project-URL', 'Project-URL', 'Keywords', 'Platform', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Classifier', 'Requires-Python', 'Provides-Extra', 'Requires-Dist', 'Requires-Dist']
@end example

You can also get a @ref{3ceb,,distribution’s version number}, list its
@ref{3cec,,constituent files}, and get a list of the distribution’s
@ref{3ced,,Distribution requirements}.

@node Functional API,Distributions,Overview<3>,Using importlib metadata
@anchor{library/importlib metadata functional-api}@anchor{3cee}
@subsubsection Functional API


This package provides the following functionality via its public API.

@menu
* Entry points:: 
* Distribution metadata:: 
* Distribution versions:: 
* Distribution files:: 
* Distribution requirements:: 
* Package distributions:: 

@end menu

@node Entry points,Distribution metadata,,Functional API
@anchor{library/importlib metadata entry-points}@anchor{293}@anchor{library/importlib metadata id1}@anchor{3cef}
@subsubsection Entry points


The @code{entry_points()} function returns a collection of entry points.
Entry points are represented by @code{EntryPoint} instances;
each @code{EntryPoint} has a @code{.name}, @code{.group}, and @code{.value} attributes and
a @code{.load()} method to resolve the value.  There are also @code{.module},
@code{.attr}, and @code{.extras} attributes for getting the components of the
@code{.value} attribute.

Query all entry points:

@example
>>> eps = entry_points()  
@end example

The @code{entry_points()} function returns an @code{EntryPoints} object,
a sequence of all @code{EntryPoint} objects with @code{names} and @code{groups}
attributes for convenience:

@example
>>> sorted(eps.groups)  
['console_scripts', 'distutils.commands', 'distutils.setup_keywords', 'egg_info.writers', 'setuptools.installation']
@end example

@code{EntryPoints} has a @code{select} method to select entry points
matching specific properties. Select entry points in the
@code{console_scripts} group:

@example
>>> scripts = eps.select(group='console_scripts')  
@end example

Equivalently, since @code{entry_points} passes keyword arguments
through to select:

@example
>>> scripts = entry_points(group='console_scripts')  
@end example

Pick out a specific script named “wheel” (found in the wheel project):

@example
>>> 'wheel' in scripts.names  
True
>>> wheel = scripts['wheel']  
@end example

Equivalently, query for that entry point during selection:

@example
>>> (wheel,) = entry_points(group='console_scripts', name='wheel')  
>>> (wheel,) = entry_points().select(group='console_scripts', name='wheel')  
@end example

Inspect the resolved entry point:

@example
>>> wheel  
EntryPoint(name='wheel', value='wheel.cli:main', group='console_scripts')
>>> wheel.module  
'wheel.cli'
>>> wheel.attr  
'main'
>>> wheel.extras  
[]
>>> main = wheel.load()  
>>> main  
<function main at 0x103528488>
@end example

The @code{group} and @code{name} are arbitrary values defined by the package author
and usually a client will wish to resolve all entry points for a particular
group.  Read the setuptools docs@footnote{https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins}
for more information on entry points, their definition, and usage.

@emph{Compatibility Note}

The “selectable” entry points were introduced in @code{importlib_metadata}
3.6 and Python 3.10. Prior to those changes, @code{entry_points} accepted
no parameters and always returned a dictionary of entry points, keyed
by group. For compatibility, if no parameters are passed to entry_points,
a @code{SelectableGroups} object is returned, implementing that dict
interface. In the future, calling @code{entry_points} with no parameters
will return an @code{EntryPoints} object. Users should rely on the selection
interface to retrieve entry points by group.

@node Distribution metadata,Distribution versions,Entry points,Functional API
@anchor{library/importlib metadata distribution-metadata}@anchor{3cf0}@anchor{library/importlib metadata metadata}@anchor{3cea}
@subsubsection Distribution metadata


Every distribution includes some metadata, which you can extract using the
@code{metadata()} function:

@example
>>> wheel_metadata = metadata('wheel')  
@end example

The keys of the returned data structure, a @code{PackageMetadata},
name the metadata keywords, and
the values are returned unparsed from the distribution metadata:

@example
>>> wheel_metadata['Requires-Python']  
'>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*'
@end example

@code{PackageMetadata} also presents a @code{json} attribute that returns
all the metadata in a JSON-compatible form per 
@geindex Python Enhancement Proposals; PEP 566
PEP 566@footnote{https://www.python.org/dev/peps/pep-0566}:

@example
>>> wheel_metadata.json['requires_python']
'>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*'
@end example


Changed in version 3.10: The @code{Description} is now included in the metadata when presented
through the payload. Line continuation characters have been removed.



New in version 3.10: The @code{json} attribute was added.


@node Distribution versions,Distribution files,Distribution metadata,Functional API
@anchor{library/importlib metadata distribution-versions}@anchor{3cf1}@anchor{library/importlib metadata version}@anchor{3ceb}
@subsubsection Distribution versions


The @code{version()} function is the quickest way to get a distribution’s version
number, as a string:

@example
>>> version('wheel')  
'0.32.3'
@end example

@node Distribution files,Distribution requirements,Distribution versions,Functional API
@anchor{library/importlib metadata distribution-files}@anchor{3cf2}@anchor{library/importlib metadata files}@anchor{3cec}
@subsubsection Distribution files


You can also get the full set of files contained within a distribution.  The
@code{files()} function takes a distribution package name and returns all of the
files installed by this distribution.  Each file object returned is a
@code{PackagePath}, a @ref{1c4,,pathlib.PurePath} derived object with additional @code{dist},
@code{size}, and @code{hash} properties as indicated by the metadata.  For example:

@example
>>> util = [p for p in files('wheel') if 'util.py' in str(p)][0]  
>>> util  
PackagePath('wheel/util.py')
>>> util.size  
859
>>> util.dist  
<importlib.metadata._hooks.PathDistribution object at 0x101e0cef0>
>>> util.hash  
<FileHash mode: sha256 value: bYkw5oMccfazVCoYQwKkkemoVyMAFoR34mmKBx8R1NI>
@end example

Once you have the file, you can also read its contents:

@example
>>> print(util.read_text())  
import base64
import sys
...
def as_bytes(s):
    if isinstance(s, text_type):
        return s.encode('utf-8')
    return s
@end example

You can also use the @code{locate} method to get a the absolute path to the
file:

@example
>>> util.locate()  
PosixPath('/home/gustav/example/lib/site-packages/wheel/util.py')
@end example

In the case where the metadata file listing files
(RECORD or SOURCES.txt) is missing, @code{files()} will
return @code{None}. The caller may wish to wrap calls to
@code{files()} in always_iterable@footnote{https://more-itertools.readthedocs.io/en/stable/api.html#more_itertools.always_iterable}
or otherwise guard against this condition if the target
distribution is not known to have the metadata present.

@node Distribution requirements,Package distributions,Distribution files,Functional API
@anchor{library/importlib metadata distribution-requirements}@anchor{3cf3}@anchor{library/importlib metadata requirements}@anchor{3ced}
@subsubsection Distribution requirements


To get the full set of requirements for a distribution, use the @code{requires()}
function:

@example
>>> requires('wheel')  
["pytest (>=3.0.0) ; extra == 'test'", "pytest-cov ; extra == 'test'"]
@end example

@node Package distributions,,Distribution requirements,Functional API
@anchor{library/importlib metadata package-distributions}@anchor{3cf4}
@subsubsection Package distributions


A convenience method to resolve the distribution or
distributions (in the case of a namespace package) for top-level
Python packages or modules:

@example
>>> packages_distributions()
@{'importlib_metadata': ['importlib-metadata'], 'yaml': ['PyYAML'], 'jaraco': ['jaraco.classes', 'jaraco.functools'], ...@}
@end example


New in version 3.10.


@node Distributions,Extending the search algorithm,Functional API,Using importlib metadata
@anchor{library/importlib metadata distributions}@anchor{3cf5}
@subsubsection Distributions


While the above API is the most common and convenient usage, you can get all
of that information from the @code{Distribution} class.  A @code{Distribution} is an
abstract object that represents the metadata for a Python package.  You can
get the @code{Distribution} instance:

@example
>>> from importlib.metadata import distribution  
>>> dist = distribution('wheel')  
@end example

Thus, an alternative way to get the version number is through the
@code{Distribution} instance:

@example
>>> dist.version  
'0.32.3'
@end example

There are all kinds of additional metadata available on the @code{Distribution}
instance:

@example
>>> dist.metadata['Requires-Python']  
'>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*'
>>> dist.metadata['License']  
'MIT'
@end example

The full set of available metadata is not described here.  See 
@geindex Python Enhancement Proposals; PEP 566
PEP 566@footnote{https://www.python.org/dev/peps/pep-0566}
for additional details.

@node Extending the search algorithm,,Distributions,Using importlib metadata
@anchor{library/importlib metadata extending-the-search-algorithm}@anchor{3cf6}
@subsubsection Extending the search algorithm


Because package metadata is not available through @ref{489,,sys.path} searches, or
package loaders directly, the metadata for a package is found through import
system @ref{1845,,finders}.  To find a distribution package’s metadata,
@code{importlib.metadata} queries the list of @ref{c05,,meta path finders} on
@ref{87c,,sys.meta_path}.

The default @code{PathFinder} for Python includes a hook that calls into
@code{importlib.metadata.MetadataPathFinder} for finding distributions
loaded from typical file-system-based paths.

The abstract class @ref{30e,,importlib.abc.MetaPathFinder} defines the
interface expected of finders by Python’s import system.
@code{importlib.metadata} extends this protocol by looking for an optional
@code{find_distributions} callable on the finders from
@ref{87c,,sys.meta_path} and presents this extended interface as the
@code{DistributionFinder} abstract base class, which defines this abstract
method:

@example
@@abc.abstractmethod
def find_distributions(context=DistributionFinder.Context()):
    """Return an iterable of all Distribution instances capable of
    loading the metadata for packages for the indicated `@w{`}context`@w{`}.
    """
@end example

The @code{DistributionFinder.Context} object provides @code{.path} and @code{.name}
properties indicating the path to search and name to match and may
supply other relevant context.

What this means in practice is that to support finding distribution package
metadata in locations other than the file system, subclass
@code{Distribution} and implement the abstract methods. Then from
a custom finder, return instances of this derived @code{Distribution} in the
@code{find_distributions()} method.

@node Python Language Services,MS Windows Specific Services,Importing Modules,The Python Standard Library
@anchor{library/language doc}@anchor{3cf7}@anchor{library/language language}@anchor{3cf8}@anchor{library/language python-language-services}@anchor{3cf9}
@section Python Language Services


Python provides a number of modules to assist in working with the Python
language.  These modules support tokenizing, parsing, syntax analysis, bytecode
disassembly, and various other facilities.

These modules include:

@menu
* ast — Abstract Syntax Trees:: 
* symtable — Access to the compiler’s symbol tables:: 
* token — Constants used with Python parse trees:: 
* keyword — Testing for Python keywords:: 
* tokenize — Tokenizer for Python source:: 
* tabnanny — Detection of ambiguous indentation:: 
* pyclbr — Python module browser support:: 
* py_compile — Compile Python source files:: 
* compileall — Byte-compile Python libraries:: 
* dis — Disassembler for Python bytecode:: 
* pickletools — Tools for pickle developers:: 

@end menu

@node ast — Abstract Syntax Trees,symtable — Access to the compiler’s symbol tables,,Python Language Services
@anchor{library/ast doc}@anchor{3cfa}@anchor{library/ast ast-abstract-syntax-trees}@anchor{3cfb}@anchor{library/ast module-ast}@anchor{7}
@subsection @code{ast} — Abstract Syntax Trees


@geindex module; ast

@c import ast

@strong{Source code:} Lib/ast.py@footnote{https://github.com/python/cpython/tree/main/Lib/ast.py}


__________________________________________________________________


The @ref{7,,ast} module helps Python applications to process trees of the Python
abstract syntax grammar.  The abstract syntax itself might change with each
Python release; this module helps to find out programmatically what the current
grammar looks like.

An abstract syntax tree can be generated by passing @ref{3cfc,,ast.PyCF_ONLY_AST} as
a flag to the @ref{4e2,,compile()} built-in function, or using the @ref{4d3,,parse()}
helper provided in this module.  The result will be a tree of objects whose
classes all inherit from @ref{3cd,,ast.AST}.  An abstract syntax tree can be
compiled into a Python code object using the built-in @ref{4e2,,compile()} function.

@menu
* Abstract Grammar:: 
* Node classes:: 
* ast Helpers:: 
* Compiler Flags:: 
* Command-Line Usage: Command-Line Usage<2>. 

@end menu

@node Abstract Grammar,Node classes,,ast — Abstract Syntax Trees
@anchor{library/ast abstract-grammar}@anchor{3cfd}@anchor{library/ast id1}@anchor{3cfe}
@subsubsection Abstract Grammar


The abstract grammar is currently defined as follows:

@example
-- ASDL's 4 builtin types are:
-- identifier, int, string, constant

module Python
@{
    mod = Module(stmt* body, type_ignore* type_ignores)
        | Interactive(stmt* body)
        | Expression(expr body)
        | FunctionType(expr* argtypes, expr returns)

    stmt = FunctionDef(identifier name, arguments args,
                       stmt* body, expr* decorator_list, expr? returns,
                       string? type_comment)
          | AsyncFunctionDef(identifier name, arguments args,
                             stmt* body, expr* decorator_list, expr? returns,
                             string? type_comment)

          | ClassDef(identifier name,
             expr* bases,
             keyword* keywords,
             stmt* body,
             expr* decorator_list)
          | Return(expr? value)

          | Delete(expr* targets)
          | Assign(expr* targets, expr value, string? type_comment)
          | AugAssign(expr target, operator op, expr value)
          -- 'simple' indicates that we annotate simple name without parens
          | AnnAssign(expr target, expr annotation, expr? value, int simple)

          -- use 'orelse' because else is a keyword in target languages
          | For(expr target, expr iter, stmt* body, stmt* orelse, string? type_comment)
          | AsyncFor(expr target, expr iter, stmt* body, stmt* orelse, string? type_comment)
          | While(expr test, stmt* body, stmt* orelse)
          | If(expr test, stmt* body, stmt* orelse)
          | With(withitem* items, stmt* body, string? type_comment)
          | AsyncWith(withitem* items, stmt* body, string? type_comment)

          | Match(expr subject, match_case* cases)

          | Raise(expr? exc, expr? cause)
          | Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody)
          | TryStar(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody)
          | Assert(expr test, expr? msg)

          | Import(alias* names)
          | ImportFrom(identifier? module, alias* names, int? level)

          | Global(identifier* names)
          | Nonlocal(identifier* names)
          | Expr(expr value)
          | Pass | Break | Continue

          -- col_offset is the byte offset in the utf8 string the parser uses
          attributes (int lineno, int col_offset, int? end_lineno, int? end_col_offset)

          -- BoolOp() can use left & right?
    expr = BoolOp(boolop op, expr* values)
         | NamedExpr(expr target, expr value)
         | BinOp(expr left, operator op, expr right)
         | UnaryOp(unaryop op, expr operand)
         | Lambda(arguments args, expr body)
         | IfExp(expr test, expr body, expr orelse)
         | Dict(expr* keys, expr* values)
         | Set(expr* elts)
         | ListComp(expr elt, comprehension* generators)
         | SetComp(expr elt, comprehension* generators)
         | DictComp(expr key, expr value, comprehension* generators)
         | GeneratorExp(expr elt, comprehension* generators)
         -- the grammar constrains where yield expressions can occur
         | Await(expr value)
         | Yield(expr? value)
         | YieldFrom(expr value)
         -- need sequences for compare to distinguish between
         -- x < 4 < 3 and (x < 4) < 3
         | Compare(expr left, cmpop* ops, expr* comparators)
         | Call(expr func, expr* args, keyword* keywords)
         | FormattedValue(expr value, int conversion, expr? format_spec)
         | JoinedStr(expr* values)
         | Constant(constant value, string? kind)

         -- the following expression can appear in assignment context
         | Attribute(expr value, identifier attr, expr_context ctx)
         | Subscript(expr value, expr slice, expr_context ctx)
         | Starred(expr value, expr_context ctx)
         | Name(identifier id, expr_context ctx)
         | List(expr* elts, expr_context ctx)
         | Tuple(expr* elts, expr_context ctx)

         -- can appear only in Subscript
         | Slice(expr? lower, expr? upper, expr? step)

          -- col_offset is the byte offset in the utf8 string the parser uses
          attributes (int lineno, int col_offset, int? end_lineno, int? end_col_offset)

    expr_context = Load | Store | Del

    boolop = And | Or

    operator = Add | Sub | Mult | MatMult | Div | Mod | Pow | LShift
                 | RShift | BitOr | BitXor | BitAnd | FloorDiv

    unaryop = Invert | Not | UAdd | USub

    cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn

    comprehension = (expr target, expr iter, expr* ifs, int is_async)

    excepthandler = ExceptHandler(expr? type, identifier? name, stmt* body)
                    attributes (int lineno, int col_offset, int? end_lineno, int? end_col_offset)

    arguments = (arg* posonlyargs, arg* args, arg? vararg, arg* kwonlyargs,
                 expr* kw_defaults, arg? kwarg, expr* defaults)

    arg = (identifier arg, expr? annotation, string? type_comment)
           attributes (int lineno, int col_offset, int? end_lineno, int? end_col_offset)

    -- keyword arguments supplied to call (NULL identifier for **kwargs)
    keyword = (identifier? arg, expr value)
               attributes (int lineno, int col_offset, int? end_lineno, int? end_col_offset)

    -- import name with optional 'as' alias.
    alias = (identifier name, identifier? asname)
             attributes (int lineno, int col_offset, int? end_lineno, int? end_col_offset)

    withitem = (expr context_expr, expr? optional_vars)

    match_case = (pattern pattern, expr? guard, stmt* body)

    pattern = MatchValue(expr value)
            | MatchSingleton(constant value)
            | MatchSequence(pattern* patterns)
            | MatchMapping(expr* keys, pattern* patterns, identifier? rest)
            | MatchClass(expr cls, pattern* patterns, identifier* kwd_attrs, pattern* kwd_patterns)

            | MatchStar(identifier? name)
            -- The optional "rest" MatchMapping parameter handles capturing extra mapping keys

            | MatchAs(pattern? pattern, identifier? name)
            | MatchOr(pattern* patterns)

             attributes (int lineno, int col_offset, int end_lineno, int end_col_offset)

    type_ignore = TypeIgnore(int lineno, string tag)
@}
@end example

@node Node classes,ast Helpers,Abstract Grammar,ast — Abstract Syntax Trees
@anchor{library/ast node-classes}@anchor{3cff}
@subsubsection Node classes


@geindex AST (class in ast)
@anchor{library/ast ast AST}@anchor{3cd}
@deffn {Class} ast.AST

This is the base of all AST node classes.  The actual node classes are
derived from the @code{Parser/Python.asdl} file, which is reproduced
@ref{3cfd,,below}.  They are defined in the @code{_ast} C
module and re-exported in @ref{7,,ast}.

There is one class defined for each left-hand side symbol in the abstract
grammar (for example, @code{ast.stmt} or @code{ast.expr}).  In addition,
there is one class defined for each constructor on the right-hand side; these
classes inherit from the classes for the left-hand side trees.  For example,
@ref{126b,,ast.BinOp} inherits from @code{ast.expr}.  For production rules
with alternatives (aka “sums”), the left-hand side class is abstract: only
instances of specific constructor nodes are ever created.

@geindex ? (question mark); in AST grammar

@geindex * (asterisk); in AST grammar

@geindex _fields (ast.AST attribute)
@anchor{library/ast ast AST _fields}@anchor{3d00}
@deffn {Attribute} _fields

Each concrete class has an attribute @ref{3d00,,_fields} which gives the names
of all child nodes.

Each instance of a concrete class has one attribute for each child node,
of the type as defined in the grammar.  For example, @ref{126b,,ast.BinOp}
instances have an attribute @code{left} of type @code{ast.expr}.

If these attributes are marked as optional in the grammar (using a
question mark), the value might be @code{None}.  If the attributes can have
zero-or-more values (marked with an asterisk), the values are represented
as Python lists.  All possible attributes must be present and have valid
values when compiling an AST with @ref{4e2,,compile()}.
@end deffn

@geindex lineno (ast.AST attribute)
@geindex col_offset (ast.AST attribute)
@geindex end_lineno (ast.AST attribute)
@geindex end_col_offset (ast.AST attribute)
@anchor{library/ast ast AST lineno}@anchor{3d01}
@deffn {Attribute} lineno
@anchor{library/ast ast AST col_offset}@anchor{3d02}
@deffnx {Attribute} col_offset
@anchor{library/ast ast AST end_lineno}@anchor{3d03}
@deffnx {Attribute} end_lineno
@anchor{library/ast ast AST end_col_offset}@anchor{3d04}
@deffnx {Attribute} end_col_offset

Instances of @code{ast.expr} and @code{ast.stmt} subclasses have
@ref{3d01,,lineno}, @ref{3d02,,col_offset}, @ref{3d03,,end_lineno}, and
@ref{3d04,,end_col_offset} attributes.  The @ref{3d01,,lineno} and @ref{3d03,,end_lineno}
are the first and last line numbers of source text span (1-indexed so the
first line is line 1) and the @ref{3d02,,col_offset} and @ref{3d04,,end_col_offset}
are the corresponding UTF-8 byte offsets of the first and last tokens that
generated the node. The UTF-8 offset is recorded because the parser uses
UTF-8 internally.

Note that the end positions are not required by the compiler and are
therefore optional. The end offset is @emph{after} the last symbol, for example
one can get the source segment of a one-line expression node using
@code{source_line[node.col_offset : node.end_col_offset]}.
@end deffn

The constructor of a class @code{ast.T} parses its arguments as follows:


@itemize *

@item 
If there are positional arguments, there must be as many as there are items
in @code{T._fields}; they will be assigned as attributes of these names.

@item 
If there are keyword arguments, they will set the attributes of the same
names to the given values.
@end itemize

For example, to create and populate an @ref{3d05,,ast.UnaryOp} node, you could
use

@example
node = ast.UnaryOp()
node.op = ast.USub()
node.operand = ast.Constant()
node.operand.value = 5
node.operand.lineno = 0
node.operand.col_offset = 0
node.lineno = 0
node.col_offset = 0
@end example

or the more compact

@example
node = ast.UnaryOp(ast.USub(), ast.Constant(5, lineno=0, col_offset=0),
                   lineno=0, col_offset=0)
@end example
@end deffn


Changed in version 3.8: Class @ref{588,,ast.Constant} is now used for all constants.



Changed in version 3.9: Simple indices are represented by their value, extended slices are
represented as tuples.



Deprecated since version 3.8: Old classes @code{ast.Num}, @code{ast.Str}, @code{ast.Bytes},
@code{ast.NameConstant} and @code{ast.Ellipsis} are still available,
but they will be removed in future Python releases.  In the meantime,
instantiating them will return an instance of a different class.



Deprecated since version 3.9: Old classes @code{ast.Index} and @code{ast.ExtSlice} are still
available, but they will be removed in future Python releases.
In the meantime, instantiating them will return an instance of
a different class.


@cartouche
@quotation Note 
The descriptions of the specific node classes displayed here
were initially adapted from the fantastic Green Tree Snakes@footnote{https://greentreesnakes.readthedocs.io/en/latest/} project and
all its contributors.
@end quotation
@end cartouche

@menu
* Literals: Literals<3>. 
* Variables:: 
* Expressions: Expressions<2>. 
* Statements:: 
* Control flow:: 
* Pattern matching:: 
* Function and class definitions:: 
* Async and await:: 

@end menu

@node Literals<3>,Variables,,Node classes
@anchor{library/ast literals}@anchor{3d06}
@subsubsection Literals


@geindex Constant (class in ast)
@anchor{library/ast ast Constant}@anchor{588}
@deffn {Class} ast.Constant (value)

A constant value. The @code{value} attribute of the @code{Constant} literal contains the
Python object it represents. The values represented can be simple types
such as a number, string or @code{None}, but also immutable container types
(tuples and frozensets) if all of their elements are constant.

@example
>>> print(ast.dump(ast.parse('123', mode='eval'), indent=4))
Expression(
    body=Constant(value=123))
@end example
@end deffn

@geindex FormattedValue (class in ast)
@anchor{library/ast ast FormattedValue}@anchor{3d07}
@deffn {Class} ast.FormattedValue (value, conversion, format_spec)

Node representing a single formatting field in an f-string. If the string
contains a single formatting field and nothing else the node can be
isolated otherwise it appears in @ref{3d08,,JoinedStr}.


@itemize *

@item 
@code{value} is any expression node (such as a literal, a variable, or a
function call).

@item 
@code{conversion} is an integer:


@itemize *

@item 
-1: no formatting

@item 
115: @code{!s} string formatting

@item 
114: @code{!r} repr formatting

@item 
97: @code{!a} ascii formatting
@end itemize

@item 
@code{format_spec} is a @ref{3d08,,JoinedStr} node representing the formatting
of the value, or @code{None} if no format was specified. Both
@code{conversion} and @code{format_spec} can be set at the same time.
@end itemize
@end deffn

@geindex JoinedStr (class in ast)
@anchor{library/ast ast JoinedStr}@anchor{3d08}
@deffn {Class} ast.JoinedStr (values)

An f-string, comprising a series of @ref{3d07,,FormattedValue} and @ref{588,,Constant}
nodes.

@example
>>> print(ast.dump(ast.parse('f"sin(@{a@}) is @{sin(a):.3@}"', mode='eval'), indent=4))
Expression(
    body=JoinedStr(
        values=[
            Constant(value='sin('),
            FormattedValue(
                value=Name(id='a', ctx=Load()),
                conversion=-1),
            Constant(value=') is '),
            FormattedValue(
                value=Call(
                    func=Name(id='sin', ctx=Load()),
                    args=[
                        Name(id='a', ctx=Load())],
                    keywords=[]),
                conversion=-1,
                format_spec=JoinedStr(
                    values=[
                        Constant(value='.3')]))]))
@end example
@end deffn

@geindex List (class in ast)
@geindex Tuple (class in ast)
@anchor{library/ast ast List}@anchor{3d09}
@deffn {Class} ast.List (elts, ctx)
@anchor{library/ast ast Tuple}@anchor{3d0a}
@deffnx {Class} ast.Tuple (elts, ctx)

A list or tuple. @code{elts} holds a list of nodes representing the elements.
@code{ctx} is @ref{3d0b,,Store} if the container is an assignment target (i.e.
@code{(x,y)=something}), and @ref{3d0c,,Load} otherwise.

@example
>>> print(ast.dump(ast.parse('[1, 2, 3]', mode='eval'), indent=4))
Expression(
    body=List(
        elts=[
            Constant(value=1),
            Constant(value=2),
            Constant(value=3)],
        ctx=Load()))
>>> print(ast.dump(ast.parse('(1, 2, 3)', mode='eval'), indent=4))
Expression(
    body=Tuple(
        elts=[
            Constant(value=1),
            Constant(value=2),
            Constant(value=3)],
        ctx=Load()))
@end example
@end deffn

@geindex Set (class in ast)
@anchor{library/ast ast Set}@anchor{3d0d}
@deffn {Class} ast.Set (elts)

A set. @code{elts} holds a list of nodes representing the set’s elements.

@example
>>> print(ast.dump(ast.parse('@{1, 2, 3@}', mode='eval'), indent=4))
Expression(
    body=Set(
        elts=[
            Constant(value=1),
            Constant(value=2),
            Constant(value=3)]))
@end example
@end deffn

@geindex Dict (class in ast)
@anchor{library/ast ast Dict}@anchor{11e6}
@deffn {Class} ast.Dict (keys, values)

A dictionary. @code{keys} and @code{values} hold lists of nodes representing the
keys and the values respectively, in matching order (what would be returned
when calling @code{dictionary.keys()} and @code{dictionary.values()}).

When doing dictionary unpacking using dictionary literals the expression to be
expanded goes in the @code{values} list, with a @code{None} at the corresponding
position in @code{keys}.

@example
>>> print(ast.dump(ast.parse('@{"a":1, **d@}', mode='eval'), indent=4))
Expression(
    body=Dict(
        keys=[
            Constant(value='a'),
            None],
        values=[
            Constant(value=1),
            Name(id='d', ctx=Load())]))
@end example
@end deffn

@node Variables,Expressions<2>,Literals<3>,Node classes
@anchor{library/ast variables}@anchor{3d0e}
@subsubsection Variables


@geindex Name (class in ast)
@anchor{library/ast ast Name}@anchor{11bc}
@deffn {Class} ast.Name (id, ctx)

A variable name. @code{id} holds the name as a string, and @code{ctx} is one of
the following types.
@end deffn

@geindex Load (class in ast)
@geindex Store (class in ast)
@geindex Del (class in ast)
@anchor{library/ast ast Load}@anchor{3d0c}
@deffn {Class} ast.Load
@anchor{library/ast ast Store}@anchor{3d0b}
@deffnx {Class} ast.Store
@anchor{library/ast ast Del}@anchor{3d0f}
@deffnx {Class} ast.Del

Variable references can be used to load the value of a variable, to assign
a new value to it, or to delete it. Variable references are given a context
to distinguish these cases.

@example
>>> print(ast.dump(ast.parse('a'), indent=4))
Module(
    body=[
        Expr(
            value=Name(id='a', ctx=Load()))],
    type_ignores=[])

>>> print(ast.dump(ast.parse('a = 1'), indent=4))
Module(
    body=[
        Assign(
            targets=[
                Name(id='a', ctx=Store())],
            value=Constant(value=1))],
    type_ignores=[])

>>> print(ast.dump(ast.parse('del a'), indent=4))
Module(
    body=[
        Delete(
            targets=[
                Name(id='a', ctx=Del())])],
    type_ignores=[])
@end example
@end deffn

@geindex Starred (class in ast)
@anchor{library/ast ast Starred}@anchor{3d10}
@deffn {Class} ast.Starred (value, ctx)

A @code{*var} variable reference. @code{value} holds the variable, typically a
@ref{11bc,,Name} node. This type must be used when building a @ref{3d11,,Call}
node with @code{*args}.

@example
>>> print(ast.dump(ast.parse('a, *b = it'), indent=4))
Module(
    body=[
        Assign(
            targets=[
                Tuple(
                    elts=[
                        Name(id='a', ctx=Store()),
                        Starred(
                            value=Name(id='b', ctx=Store()),
                            ctx=Store())],
                    ctx=Store())],
            value=Name(id='it', ctx=Load()))],
    type_ignores=[])
@end example
@end deffn

@node Expressions<2>,Statements,Variables,Node classes
@anchor{library/ast expressions}@anchor{3d12}
@subsubsection Expressions


@geindex Expr (class in ast)
@anchor{library/ast ast Expr}@anchor{3d13}
@deffn {Class} ast.Expr (value)

When an expression, such as a function call, appears as a statement by itself
with its return value not used or stored, it is wrapped in this container.
@code{value} holds one of the other nodes in this section, a @ref{588,,Constant}, a
@ref{11bc,,Name}, a @ref{3d14,,Lambda}, a @ref{3d15,,Yield} or @ref{3d16,,YieldFrom} node.

@example
>>> print(ast.dump(ast.parse('-a'), indent=4))
Module(
    body=[
        Expr(
            value=UnaryOp(
                op=USub(),
                operand=Name(id='a', ctx=Load())))],
    type_ignores=[])
@end example
@end deffn

@geindex UnaryOp (class in ast)
@anchor{library/ast ast UnaryOp}@anchor{3d05}
@deffn {Class} ast.UnaryOp (op, operand)

A unary operation. @code{op} is the operator, and @code{operand} any expression
node.
@end deffn

@geindex UAdd (class in ast)
@geindex USub (class in ast)
@geindex Not (class in ast)
@geindex Invert (class in ast)
@anchor{library/ast ast UAdd}@anchor{3d17}
@deffn {Class} ast.UAdd
@anchor{library/ast ast USub}@anchor{3d18}
@deffnx {Class} ast.USub
@anchor{library/ast ast Not}@anchor{3d19}
@deffnx {Class} ast.Not
@anchor{library/ast ast Invert}@anchor{3d1a}
@deffnx {Class} ast.Invert

Unary operator tokens. @ref{3d19,,Not} is the @code{not} keyword, @ref{3d1a,,Invert}
is the @code{~} operator.

@example
>>> print(ast.dump(ast.parse('not x', mode='eval'), indent=4))
Expression(
    body=UnaryOp(
        op=Not(),
        operand=Name(id='x', ctx=Load())))
@end example
@end deffn

@geindex BinOp (class in ast)
@anchor{library/ast ast BinOp}@anchor{126b}
@deffn {Class} ast.BinOp (left, op, right)

A binary operation (like addition or division). @code{op} is the operator, and
@code{left} and @code{right} are any expression nodes.

@example
>>> print(ast.dump(ast.parse('x + y', mode='eval'), indent=4))
Expression(
    body=BinOp(
        left=Name(id='x', ctx=Load()),
        op=Add(),
        right=Name(id='y', ctx=Load())))
@end example
@end deffn

@geindex Add (class in ast)
@geindex Sub (class in ast)
@geindex Mult (class in ast)
@geindex Div (class in ast)
@geindex FloorDiv (class in ast)
@geindex Mod (class in ast)
@geindex Pow (class in ast)
@geindex LShift (class in ast)
@geindex RShift (class in ast)
@geindex BitOr (class in ast)
@geindex BitXor (class in ast)
@geindex BitAnd (class in ast)
@geindex MatMult (class in ast)
@anchor{library/ast ast Add}@anchor{3d1b}
@deffn {Class} ast.Add
@anchor{library/ast ast Sub}@anchor{3d1c}
@deffnx {Class} ast.Sub
@anchor{library/ast ast Mult}@anchor{3d1d}
@deffnx {Class} ast.Mult
@anchor{library/ast ast Div}@anchor{3d1e}
@deffnx {Class} ast.Div
@anchor{library/ast ast FloorDiv}@anchor{3d1f}
@deffnx {Class} ast.FloorDiv
@anchor{library/ast ast Mod}@anchor{3d20}
@deffnx {Class} ast.Mod
@anchor{library/ast ast Pow}@anchor{3d21}
@deffnx {Class} ast.Pow
@anchor{library/ast ast LShift}@anchor{3d22}
@deffnx {Class} ast.LShift
@anchor{library/ast ast RShift}@anchor{3d23}
@deffnx {Class} ast.RShift
@anchor{library/ast ast BitOr}@anchor{3d24}
@deffnx {Class} ast.BitOr
@anchor{library/ast ast BitXor}@anchor{3d25}
@deffnx {Class} ast.BitXor
@anchor{library/ast ast BitAnd}@anchor{3d26}
@deffnx {Class} ast.BitAnd
@anchor{library/ast ast MatMult}@anchor{3d27}
@deffnx {Class} ast.MatMult

Binary operator tokens.
@end deffn

@geindex BoolOp (class in ast)
@anchor{library/ast ast BoolOp}@anchor{3d28}
@deffn {Class} ast.BoolOp (op, values)

A boolean operation, ‘or’ or ‘and’. @code{op} is @ref{3d29,,Or} or @ref{3d2a,,And}.
@code{values} are the values involved. Consecutive operations with the same
operator, such as @code{a or b or c}, are collapsed into one node with several
values.

This doesn’t include @code{not}, which is a @ref{3d05,,UnaryOp}.

@example
>>> print(ast.dump(ast.parse('x or y', mode='eval'), indent=4))
Expression(
    body=BoolOp(
        op=Or(),
        values=[
            Name(id='x', ctx=Load()),
            Name(id='y', ctx=Load())]))
@end example
@end deffn

@geindex And (class in ast)
@geindex Or (class in ast)
@anchor{library/ast ast And}@anchor{3d2a}
@deffn {Class} ast.And
@anchor{library/ast ast Or}@anchor{3d29}
@deffnx {Class} ast.Or

Boolean operator tokens.
@end deffn

@geindex Compare (class in ast)
@anchor{library/ast ast Compare}@anchor{3d2b}
@deffn {Class} ast.Compare (left, ops, comparators)

A comparison of two or more values. @code{left} is the first value in the
comparison, @code{ops} the list of operators, and @code{comparators} the list
of values after the first element in the comparison.

@example
>>> print(ast.dump(ast.parse('1 <= a < 10', mode='eval'), indent=4))
Expression(
    body=Compare(
        left=Constant(value=1),
        ops=[
            LtE(),
            Lt()],
        comparators=[
            Name(id='a', ctx=Load()),
            Constant(value=10)]))
@end example
@end deffn

@geindex Eq (class in ast)
@geindex NotEq (class in ast)
@geindex Lt (class in ast)
@geindex LtE (class in ast)
@geindex Gt (class in ast)
@geindex GtE (class in ast)
@geindex Is (class in ast)
@geindex IsNot (class in ast)
@geindex In (class in ast)
@geindex NotIn (class in ast)
@anchor{library/ast ast Eq}@anchor{3d2c}
@deffn {Class} ast.Eq
@anchor{library/ast ast NotEq}@anchor{3d2d}
@deffnx {Class} ast.NotEq
@anchor{library/ast ast Lt}@anchor{3d2e}
@deffnx {Class} ast.Lt
@anchor{library/ast ast LtE}@anchor{3d2f}
@deffnx {Class} ast.LtE
@anchor{library/ast ast Gt}@anchor{3d30}
@deffnx {Class} ast.Gt
@anchor{library/ast ast GtE}@anchor{3d31}
@deffnx {Class} ast.GtE
@anchor{library/ast ast Is}@anchor{3d32}
@deffnx {Class} ast.Is
@anchor{library/ast ast IsNot}@anchor{3d33}
@deffnx {Class} ast.IsNot
@anchor{library/ast ast In}@anchor{3d34}
@deffnx {Class} ast.In
@anchor{library/ast ast NotIn}@anchor{3d35}
@deffnx {Class} ast.NotIn

Comparison operator tokens.
@end deffn

@geindex Call (class in ast)
@anchor{library/ast ast Call}@anchor{3d11}
@deffn {Class} ast.Call (func, args, keywords, starargs, kwargs)

A function call. @code{func} is the function, which will often be a
@ref{11bc,,Name} or @ref{3d36,,Attribute} object. Of the arguments:


@itemize *

@item 
@code{args} holds a list of the arguments passed by position.

@item 
@code{keywords} holds a list of @ref{a2,,keyword} objects representing
arguments passed by keyword.
@end itemize

When creating a @code{Call} node, @code{args} and @code{keywords} are required, but
they can be empty lists. @code{starargs} and @code{kwargs} are optional.

@example
>>> print(ast.dump(ast.parse('func(a, b=c, *d, **e)', mode='eval'), indent=4))
Expression(
    body=Call(
        func=Name(id='func', ctx=Load()),
        args=[
            Name(id='a', ctx=Load()),
            Starred(
                value=Name(id='d', ctx=Load()),
                ctx=Load())],
        keywords=[
            keyword(
                arg='b',
                value=Name(id='c', ctx=Load())),
            keyword(
                value=Name(id='e', ctx=Load()))]))
@end example
@end deffn

@geindex keyword (class in ast)
@anchor{library/ast ast keyword}@anchor{3d37}
@deffn {Class} ast.keyword (arg, value)

A keyword argument to a function call or class definition. @code{arg} is a raw
string of the parameter name, @code{value} is a node to pass in.
@end deffn

@geindex IfExp (class in ast)
@anchor{library/ast ast IfExp}@anchor{3d38}
@deffn {Class} ast.IfExp (test, body, orelse)

An expression such as @code{a if b else c}. Each field holds a single node, so
in the following example, all three are @ref{11bc,,Name} nodes.

@example
>>> print(ast.dump(ast.parse('a if b else c', mode='eval'), indent=4))
Expression(
    body=IfExp(
        test=Name(id='b', ctx=Load()),
        body=Name(id='a', ctx=Load()),
        orelse=Name(id='c', ctx=Load())))
@end example
@end deffn

@geindex Attribute (class in ast)
@anchor{library/ast ast Attribute}@anchor{3d36}
@deffn {Class} ast.Attribute (value, attr, ctx)

Attribute access, e.g. @code{d.keys}. @code{value} is a node, typically a
@ref{11bc,,Name}. @code{attr} is a bare string giving the name of the attribute,
and @code{ctx} is @ref{3d0c,,Load}, @ref{3d0b,,Store} or @ref{3d0f,,Del} according to how
the attribute is acted on.

@example
>>> print(ast.dump(ast.parse('snake.colour', mode='eval'), indent=4))
Expression(
    body=Attribute(
        value=Name(id='snake', ctx=Load()),
        attr='colour',
        ctx=Load()))
@end example
@end deffn

@geindex NamedExpr (class in ast)
@anchor{library/ast ast NamedExpr}@anchor{3d39}
@deffn {Class} ast.NamedExpr (target, value)

@quotation

A named expression. This AST node is produced by the assignment expressions
operator (also known as the walrus operator). As opposed to the @ref{3d3a,,Assign}
node in which the first argument can be multiple nodes, in this case both
@code{target} and @code{value} must be single nodes.
@end quotation

@example
>>> print(ast.dump(ast.parse('(x := 4)', mode='eval'), indent=4))
Expression(
    body=NamedExpr(
        target=Name(id='x', ctx=Store()),
        value=Constant(value=4)))
@end example
@end deffn

@menu
* Subscripting:: 
* Comprehensions:: 

@end menu

@node Subscripting,Comprehensions,,Expressions<2>
@anchor{library/ast subscripting}@anchor{3d3b}
@subsubsection Subscripting


@geindex Subscript (class in ast)
@anchor{library/ast ast Subscript}@anchor{3d3c}
@deffn {Class} ast.Subscript (value, slice, ctx)

A subscript, such as @code{l[1]}. @code{value} is the subscripted object
(usually sequence or mapping). @code{slice} is an index, slice or key.
It can be a @ref{3d0a,,Tuple} and contain a @ref{3d3d,,Slice}.
@code{ctx} is @ref{3d0c,,Load}, @ref{3d0b,,Store} or @ref{3d0f,,Del}
according to the action performed with the subscript.

@example
>>> print(ast.dump(ast.parse('l[1:2, 3]', mode='eval'), indent=4))
Expression(
    body=Subscript(
        value=Name(id='l', ctx=Load()),
        slice=Tuple(
            elts=[
                Slice(
                    lower=Constant(value=1),
                    upper=Constant(value=2)),
                Constant(value=3)],
            ctx=Load()),
        ctx=Load()))
@end example
@end deffn

@geindex Slice (class in ast)
@anchor{library/ast ast Slice}@anchor{3d3d}
@deffn {Class} ast.Slice (lower, upper, step)

Regular slicing (on the form @code{lower:upper} or @code{lower:upper:step}).
Can occur only inside the @emph{slice} field of @ref{3d3c,,Subscript}, either
directly or as an element of @ref{3d0a,,Tuple}.

@example
>>> print(ast.dump(ast.parse('l[1:2]', mode='eval'), indent=4))
Expression(
    body=Subscript(
        value=Name(id='l', ctx=Load()),
        slice=Slice(
            lower=Constant(value=1),
            upper=Constant(value=2)),
        ctx=Load()))
@end example
@end deffn

@node Comprehensions,,Subscripting,Expressions<2>
@anchor{library/ast comprehensions}@anchor{3d3e}
@subsubsection Comprehensions


@geindex ListComp (class in ast)
@geindex SetComp (class in ast)
@geindex GeneratorExp (class in ast)
@geindex DictComp (class in ast)
@anchor{library/ast ast ListComp}@anchor{3d3f}
@deffn {Class} ast.ListComp (elt, generators)
@anchor{library/ast ast SetComp}@anchor{3d40}
@deffnx {Class} ast.SetComp (elt, generators)
@anchor{library/ast ast GeneratorExp}@anchor{3d41}
@deffnx {Class} ast.GeneratorExp (elt, generators)
@anchor{library/ast ast DictComp}@anchor{3d42}
@deffnx {Class} ast.DictComp (key, value, generators)

List and set comprehensions, generator expressions, and dictionary
comprehensions. @code{elt} (or @code{key} and @code{value}) is a single node
representing the part that will be evaluated for each item.

@code{generators} is a list of @ref{3d43,,comprehension} nodes.

@example
>>> print(ast.dump(ast.parse('[x for x in numbers]', mode='eval'), indent=4))
Expression(
    body=ListComp(
        elt=Name(id='x', ctx=Load()),
        generators=[
            comprehension(
                target=Name(id='x', ctx=Store()),
                iter=Name(id='numbers', ctx=Load()),
                ifs=[],
                is_async=0)]))
>>> print(ast.dump(ast.parse('@{x: x**2 for x in numbers@}', mode='eval'), indent=4))
Expression(
    body=DictComp(
        key=Name(id='x', ctx=Load()),
        value=BinOp(
            left=Name(id='x', ctx=Load()),
            op=Pow(),
            right=Constant(value=2)),
        generators=[
            comprehension(
                target=Name(id='x', ctx=Store()),
                iter=Name(id='numbers', ctx=Load()),
                ifs=[],
                is_async=0)]))
>>> print(ast.dump(ast.parse('@{x for x in numbers@}', mode='eval'), indent=4))
Expression(
    body=SetComp(
        elt=Name(id='x', ctx=Load()),
        generators=[
            comprehension(
                target=Name(id='x', ctx=Store()),
                iter=Name(id='numbers', ctx=Load()),
                ifs=[],
                is_async=0)]))
@end example
@end deffn

@geindex comprehension (class in ast)
@anchor{library/ast ast comprehension}@anchor{3d43}
@deffn {Class} ast.comprehension (target, iter, ifs, is_async)

One @code{for} clause in a comprehension. @code{target} is the reference to use for
each element - typically a @ref{11bc,,Name} or @ref{3d0a,,Tuple} node. @code{iter}
is the object to iterate over. @code{ifs} is a list of test expressions: each
@code{for} clause can have multiple @code{ifs}.

@code{is_async} indicates a comprehension is asynchronous (using an
@code{async for} instead of @code{for}). The value is an integer (0 or 1).

@example
>>> print(ast.dump(ast.parse('[ord(c) for line in file for c in line]', mode='eval'),
...                indent=4)) # Multiple comprehensions in one.
Expression(
    body=ListComp(
        elt=Call(
            func=Name(id='ord', ctx=Load()),
            args=[
                Name(id='c', ctx=Load())],
            keywords=[]),
        generators=[
            comprehension(
                target=Name(id='line', ctx=Store()),
                iter=Name(id='file', ctx=Load()),
                ifs=[],
                is_async=0),
            comprehension(
                target=Name(id='c', ctx=Store()),
                iter=Name(id='line', ctx=Load()),
                ifs=[],
                is_async=0)]))

>>> print(ast.dump(ast.parse('(n**2 for n in it if n>5 if n<10)', mode='eval'),
...                indent=4)) # generator comprehension
Expression(
    body=GeneratorExp(
        elt=BinOp(
            left=Name(id='n', ctx=Load()),
            op=Pow(),
            right=Constant(value=2)),
        generators=[
            comprehension(
                target=Name(id='n', ctx=Store()),
                iter=Name(id='it', ctx=Load()),
                ifs=[
                    Compare(
                        left=Name(id='n', ctx=Load()),
                        ops=[
                            Gt()],
                        comparators=[
                            Constant(value=5)]),
                    Compare(
                        left=Name(id='n', ctx=Load()),
                        ops=[
                            Lt()],
                        comparators=[
                            Constant(value=10)])],
                is_async=0)]))

>>> print(ast.dump(ast.parse('[i async for i in soc]', mode='eval'),
...                indent=4)) # Async comprehension
Expression(
    body=ListComp(
        elt=Name(id='i', ctx=Load()),
        generators=[
            comprehension(
                target=Name(id='i', ctx=Store()),
                iter=Name(id='soc', ctx=Load()),
                ifs=[],
                is_async=1)]))
@end example
@end deffn

@node Statements,Control flow,Expressions<2>,Node classes
@anchor{library/ast statements}@anchor{3d44}
@subsubsection Statements


@geindex Assign (class in ast)
@anchor{library/ast ast Assign}@anchor{3d3a}
@deffn {Class} ast.Assign (targets, value, type_comment)

An assignment. @code{targets} is a list of nodes, and @code{value} is a single node.

Multiple nodes in @code{targets} represents assigning the same value to each.
Unpacking is represented by putting a @ref{3d0a,,Tuple} or @ref{3d09,,List}
within @code{targets}.

@geindex type_comment (ast.Assign attribute)
@anchor{library/ast ast Assign type_comment}@anchor{3d45}
@deffn {Attribute} type_comment

@code{type_comment} is an optional string with the type annotation as a comment.
@end deffn

@example
>>> print(ast.dump(ast.parse('a = b = 1'), indent=4)) # Multiple assignment
Module(
    body=[
        Assign(
            targets=[
                Name(id='a', ctx=Store()),
                Name(id='b', ctx=Store())],
            value=Constant(value=1))],
    type_ignores=[])

>>> print(ast.dump(ast.parse('a,b = c'), indent=4)) # Unpacking
Module(
    body=[
        Assign(
            targets=[
                Tuple(
                    elts=[
                        Name(id='a', ctx=Store()),
                        Name(id='b', ctx=Store())],
                    ctx=Store())],
            value=Name(id='c', ctx=Load()))],
    type_ignores=[])
@end example
@end deffn

@geindex AnnAssign (class in ast)
@anchor{library/ast ast AnnAssign}@anchor{3d46}
@deffn {Class} ast.AnnAssign (target, annotation, value, simple)

An assignment with a type annotation. @code{target} is a single node and can
be a @ref{11bc,,Name}, a @ref{3d36,,Attribute} or a @ref{3d3c,,Subscript}.
@code{annotation} is the annotation, such as a @ref{588,,Constant} or @ref{11bc,,Name}
node. @code{value} is a single optional node. @code{simple} is a boolean integer
set to True for a @ref{11bc,,Name} node in @code{target} that do not appear in
between parenthesis and are hence pure names and not expressions.

@example
>>> print(ast.dump(ast.parse('c: int'), indent=4))
Module(
    body=[
        AnnAssign(
            target=Name(id='c', ctx=Store()),
            annotation=Name(id='int', ctx=Load()),
            simple=1)],
    type_ignores=[])

>>> print(ast.dump(ast.parse('(a): int = 1'), indent=4)) # Annotation with parenthesis
Module(
    body=[
        AnnAssign(
            target=Name(id='a', ctx=Store()),
            annotation=Name(id='int', ctx=Load()),
            value=Constant(value=1),
            simple=0)],
    type_ignores=[])

>>> print(ast.dump(ast.parse('a.b: int'), indent=4)) # Attribute annotation
Module(
    body=[
        AnnAssign(
            target=Attribute(
                value=Name(id='a', ctx=Load()),
                attr='b',
                ctx=Store()),
            annotation=Name(id='int', ctx=Load()),
            simple=0)],
    type_ignores=[])

>>> print(ast.dump(ast.parse('a[1]: int'), indent=4)) # Subscript annotation
Module(
    body=[
        AnnAssign(
            target=Subscript(
                value=Name(id='a', ctx=Load()),
                slice=Constant(value=1),
                ctx=Store()),
            annotation=Name(id='int', ctx=Load()),
            simple=0)],
    type_ignores=[])
@end example
@end deffn

@geindex AugAssign (class in ast)
@anchor{library/ast ast AugAssign}@anchor{3d47}
@deffn {Class} ast.AugAssign (target, op, value)

Augmented assignment, such as @code{a += 1}. In the following example,
@code{target} is a @ref{11bc,,Name} node for @code{x} (with the @ref{3d0b,,Store}
context), @code{op} is @ref{3d1b,,Add}, and @code{value} is a @ref{588,,Constant} with
value for 1.

The @code{target} attribute cannot be of class @ref{3d0a,,Tuple} or @ref{3d09,,List},
unlike the targets of @ref{3d3a,,Assign}.

@example
>>> print(ast.dump(ast.parse('x += 2'), indent=4))
Module(
    body=[
        AugAssign(
            target=Name(id='x', ctx=Store()),
            op=Add(),
            value=Constant(value=2))],
    type_ignores=[])
@end example
@end deffn

@geindex Raise (class in ast)
@anchor{library/ast ast Raise}@anchor{3d48}
@deffn {Class} ast.Raise (exc, cause)

A @code{raise} statement. @code{exc} is the exception object to be raised, normally a
@ref{3d11,,Call} or @ref{11bc,,Name}, or @code{None} for a standalone @code{raise}.
@code{cause} is the optional part for @code{y} in @code{raise x from y}.

@example
>>> print(ast.dump(ast.parse('raise x from y'), indent=4))
Module(
    body=[
        Raise(
            exc=Name(id='x', ctx=Load()),
            cause=Name(id='y', ctx=Load()))],
    type_ignores=[])
@end example
@end deffn

@geindex Assert (class in ast)
@anchor{library/ast ast Assert}@anchor{3d49}
@deffn {Class} ast.Assert (test, msg)

An assertion. @code{test} holds the condition, such as a @ref{3d2b,,Compare} node.
@code{msg} holds the failure message.

@example
>>> print(ast.dump(ast.parse('assert x,y'), indent=4))
Module(
    body=[
        Assert(
            test=Name(id='x', ctx=Load()),
            msg=Name(id='y', ctx=Load()))],
    type_ignores=[])
@end example
@end deffn

@geindex Delete (class in ast)
@anchor{library/ast ast Delete}@anchor{3d4a}
@deffn {Class} ast.Delete (targets)

Represents a @code{del} statement. @code{targets} is a list of nodes, such as
@ref{11bc,,Name}, @ref{3d36,,Attribute} or @ref{3d3c,,Subscript} nodes.

@example
>>> print(ast.dump(ast.parse('del x,y,z'), indent=4))
Module(
    body=[
        Delete(
            targets=[
                Name(id='x', ctx=Del()),
                Name(id='y', ctx=Del()),
                Name(id='z', ctx=Del())])],
    type_ignores=[])
@end example
@end deffn

@geindex Pass (class in ast)
@anchor{library/ast ast Pass}@anchor{3d4b}
@deffn {Class} ast.Pass

A @code{pass} statement.

@example
>>> print(ast.dump(ast.parse('pass'), indent=4))
Module(
    body=[
        Pass()],
    type_ignores=[])
@end example
@end deffn

Other statements which are only applicable inside functions or loops are
described in other sections.

@menu
* Imports:: 

@end menu

@node Imports,,,Statements
@anchor{library/ast imports}@anchor{3d4c}
@subsubsection Imports


@geindex Import (class in ast)
@anchor{library/ast ast Import}@anchor{3d4d}
@deffn {Class} ast.Import (names)

An import statement. @code{names} is a list of @ref{1134,,alias} nodes.

@example
>>> print(ast.dump(ast.parse('import x,y,z'), indent=4))
Module(
    body=[
        Import(
            names=[
                alias(name='x'),
                alias(name='y'),
                alias(name='z')])],
    type_ignores=[])
@end example
@end deffn

@geindex ImportFrom (class in ast)
@anchor{library/ast ast ImportFrom}@anchor{3d4e}
@deffn {Class} ast.ImportFrom (module, names, level)

Represents @code{from x import y}. @code{module} is a raw string of the ‘from’ name,
without any leading dots, or @code{None} for statements such as @code{from . import foo}.
@code{level} is an integer holding the level of the relative import (0 means
absolute import).

@example
>>> print(ast.dump(ast.parse('from y import x,y,z'), indent=4))
Module(
    body=[
        ImportFrom(
            module='y',
            names=[
                alias(name='x'),
                alias(name='y'),
                alias(name='z')],
            level=0)],
    type_ignores=[])
@end example
@end deffn

@geindex alias (class in ast)
@anchor{library/ast ast alias}@anchor{1134}
@deffn {Class} ast.alias (name, asname)

Both parameters are raw strings of the names. @code{asname} can be @code{None} if
the regular name is to be used.

@example
>>> print(ast.dump(ast.parse('from ..foo.bar import a as b, c'), indent=4))
Module(
    body=[
        ImportFrom(
            module='foo.bar',
            names=[
                alias(name='a', asname='b'),
                alias(name='c')],
            level=2)],
    type_ignores=[])
@end example
@end deffn

@node Control flow,Pattern matching,Statements,Node classes
@anchor{library/ast control-flow}@anchor{3d4f}
@subsubsection Control flow


@cartouche
@quotation Note 
Optional clauses such as @code{else} are stored as an empty list if they’re
not present.
@end quotation
@end cartouche

@geindex If (class in ast)
@anchor{library/ast ast If}@anchor{3d50}
@deffn {Class} ast.If (test, body, orelse)

An @code{if} statement. @code{test} holds a single node, such as a @ref{3d2b,,Compare}
node. @code{body} and @code{orelse} each hold a list of nodes.

@code{elif} clauses don’t have a special representation in the AST, but rather
appear as extra @ref{3d50,,If} nodes within the @code{orelse} section of the
previous one.

@example
>>> print(ast.dump(ast.parse("""
... if x:
...    ...
... elif y:
...    ...
... else:
...    ...
... """), indent=4))
Module(
    body=[
        If(
            test=Name(id='x', ctx=Load()),
            body=[
                Expr(
                    value=Constant(value=Ellipsis))],
            orelse=[
                If(
                    test=Name(id='y', ctx=Load()),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))],
                    orelse=[
                        Expr(
                            value=Constant(value=Ellipsis))])])],
    type_ignores=[])
@end example
@end deffn

@geindex For (class in ast)
@anchor{library/ast ast For}@anchor{3d51}
@deffn {Class} ast.For (target, iter, body, orelse, type_comment)

A @code{for} loop. @code{target} holds the variable(s) the loop assigns to, as a
single @ref{11bc,,Name}, @ref{3d0a,,Tuple} or @ref{3d09,,List} node. @code{iter} holds
the item to be looped over, again as a single node. @code{body} and @code{orelse}
contain lists of nodes to execute. Those in @code{orelse} are executed if the
loop finishes normally, rather than via a @code{break} statement.

@geindex type_comment (ast.For attribute)
@anchor{library/ast ast For type_comment}@anchor{3d52}
@deffn {Attribute} type_comment

@code{type_comment} is an optional string with the type annotation as a comment.
@end deffn

@example
>>> print(ast.dump(ast.parse("""
... for x in y:
...     ...
... else:
...     ...
... """), indent=4))
Module(
    body=[
        For(
            target=Name(id='x', ctx=Store()),
            iter=Name(id='y', ctx=Load()),
            body=[
                Expr(
                    value=Constant(value=Ellipsis))],
            orelse=[
                Expr(
                    value=Constant(value=Ellipsis))])],
    type_ignores=[])
@end example
@end deffn

@geindex While (class in ast)
@anchor{library/ast ast While}@anchor{3d53}
@deffn {Class} ast.While (test, body, orelse)

A @code{while} loop. @code{test} holds the condition, such as a @ref{3d2b,,Compare}
node.

@example
>> print(ast.dump(ast.parse("""
... while x:
...    ...
... else:
...    ...
... """), indent=4))
Module(
    body=[
        While(
            test=Name(id='x', ctx=Load()),
            body=[
                Expr(
                    value=Constant(value=Ellipsis))],
            orelse=[
                Expr(
                    value=Constant(value=Ellipsis))])],
    type_ignores=[])
@end example
@end deffn

@geindex Break (class in ast)
@geindex Continue (class in ast)
@anchor{library/ast ast Break}@anchor{3d54}
@deffn {Class} ast.Break
@anchor{library/ast ast Continue}@anchor{3d55}
@deffnx {Class} ast.Continue

The @code{break} and @code{continue} statements.

@example
>>> print(ast.dump(ast.parse("""\
... for a in b:
...     if a > 5:
...         break
...     else:
...         continue
...
... """), indent=4))
Module(
    body=[
        For(
            target=Name(id='a', ctx=Store()),
            iter=Name(id='b', ctx=Load()),
            body=[
                If(
                    test=Compare(
                        left=Name(id='a', ctx=Load()),
                        ops=[
                            Gt()],
                        comparators=[
                            Constant(value=5)]),
                    body=[
                        Break()],
                    orelse=[
                        Continue()])],
            orelse=[])],
    type_ignores=[])
@end example
@end deffn

@geindex Try (class in ast)
@anchor{library/ast ast Try}@anchor{3d56}
@deffn {Class} ast.Try (body, handlers, orelse, finalbody)

@code{try} blocks. All attributes are list of nodes to execute, except for
@code{handlers}, which is a list of @ref{3d57,,ExceptHandler} nodes.

@example
>>> print(ast.dump(ast.parse("""
... try:
...    ...
... except Exception:
...    ...
... except OtherException as e:
...    ...
... else:
...    ...
... finally:
...    ...
... """), indent=4))
Module(
    body=[
        Try(
            body=[
                Expr(
                    value=Constant(value=Ellipsis))],
            handlers=[
                ExceptHandler(
                    type=Name(id='Exception', ctx=Load()),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))]),
                ExceptHandler(
                    type=Name(id='OtherException', ctx=Load()),
                    name='e',
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))])],
            orelse=[
                Expr(
                    value=Constant(value=Ellipsis))],
            finalbody=[
                Expr(
                    value=Constant(value=Ellipsis))])],
    type_ignores=[])
@end example
@end deffn

@geindex TryStar (class in ast)
@anchor{library/ast ast TryStar}@anchor{3d58}
@deffn {Class} ast.TryStar (body, handlers, orelse, finalbody)

@code{try} blocks which are followed by @code{except*} clauses. The attributes are the
same as for @ref{3d56,,Try} but the @ref{3d57,,ExceptHandler} nodes in @code{handlers}
are interpreted as @code{except*} blocks rather then @code{except}.

@example
>>> print(ast.dump(ast.parse("""
... try:
...    ...
... except* Exception:
...    ...
... """), indent=4))
Module(
    body=[
        TryStar(
            body=[
                Expr(
                    value=Constant(value=Ellipsis))],
            handlers=[
                ExceptHandler(
                    type=Name(id='Exception', ctx=Load()),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))])],
            orelse=[],
            finalbody=[])],
    type_ignores=[])
@end example
@end deffn

@geindex ExceptHandler (class in ast)
@anchor{library/ast ast ExceptHandler}@anchor{3d57}
@deffn {Class} ast.ExceptHandler (type, name, body)

A single @code{except} clause. @code{type} is the exception type it will match,
typically a @ref{11bc,,Name} node (or @code{None} for a catch-all @code{except:} clause).
@code{name} is a raw string for the name to hold the exception, or @code{None} if
the clause doesn’t have @code{as foo}. @code{body} is a list of nodes.

@example
>>> print(ast.dump(ast.parse("""\
... try:
...     a + 1
... except TypeError:
...     pass
... """), indent=4))
Module(
    body=[
        Try(
            body=[
                Expr(
                    value=BinOp(
                        left=Name(id='a', ctx=Load()),
                        op=Add(),
                        right=Constant(value=1)))],
            handlers=[
                ExceptHandler(
                    type=Name(id='TypeError', ctx=Load()),
                    body=[
                        Pass()])],
            orelse=[],
            finalbody=[])],
    type_ignores=[])
@end example
@end deffn

@geindex With (class in ast)
@anchor{library/ast ast With}@anchor{3d59}
@deffn {Class} ast.With (items, body, type_comment)

A @code{with} block. @code{items} is a list of @ref{3d5a,,withitem} nodes representing
the context managers, and @code{body} is the indented block inside the context.

@geindex type_comment (ast.With attribute)
@anchor{library/ast ast With type_comment}@anchor{3d5b}
@deffn {Attribute} type_comment

@code{type_comment} is an optional string with the type annotation as a comment.
@end deffn
@end deffn

@geindex withitem (class in ast)
@anchor{library/ast ast withitem}@anchor{3d5a}
@deffn {Class} ast.withitem (context_expr, optional_vars)

A single context manager in a @code{with} block. @code{context_expr} is the context
manager, often a @ref{3d11,,Call} node. @code{optional_vars} is a @ref{11bc,,Name},
@ref{3d0a,,Tuple} or @ref{3d09,,List} for the @code{as foo} part, or @code{None} if that
isn’t used.

@example
>>> print(ast.dump(ast.parse("""\
... with a as b, c as d:
...    something(b, d)
... """), indent=4))
Module(
    body=[
        With(
            items=[
                withitem(
                    context_expr=Name(id='a', ctx=Load()),
                    optional_vars=Name(id='b', ctx=Store())),
                withitem(
                    context_expr=Name(id='c', ctx=Load()),
                    optional_vars=Name(id='d', ctx=Store()))],
            body=[
                Expr(
                    value=Call(
                        func=Name(id='something', ctx=Load()),
                        args=[
                            Name(id='b', ctx=Load()),
                            Name(id='d', ctx=Load())],
                        keywords=[]))])],
    type_ignores=[])
@end example
@end deffn

@node Pattern matching,Function and class definitions,Control flow,Node classes
@anchor{library/ast pattern-matching}@anchor{3d5c}
@subsubsection Pattern matching


@geindex Match (class in ast)
@anchor{library/ast ast Match}@anchor{3d5d}
@deffn {Class} ast.Match (subject, cases)

A @code{match} statement. @code{subject} holds the subject of the match (the object
that is being matched against the cases) and @code{cases} contains an iterable of
@ref{3d5e,,match_case} nodes with the different cases.
@end deffn

@geindex match_case (class in ast)
@anchor{library/ast ast match_case}@anchor{3d5e}
@deffn {Class} ast.match_case (pattern, guard, body)

A single case pattern in a @code{match} statement. @code{pattern} contains the
match pattern that the subject will be matched against. Note that the
@ref{3cd,,AST} nodes produced for patterns differ from those produced for
expressions, even when they share the same syntax.

The @code{guard} attribute contains an expression that will be evaluated if
the pattern matches the subject.

@code{body} contains a list of nodes to execute if the pattern matches and
the result of evaluating the guard expression is true.

@example
>>> print(ast.dump(ast.parse("""
... match x:
...     case [x] if x>0:
...         ...
...     case tuple():
...         ...
... """), indent=4))
Module(
    body=[
        Match(
            subject=Name(id='x', ctx=Load()),
            cases=[
                match_case(
                    pattern=MatchSequence(
                        patterns=[
                            MatchAs(name='x')]),
                    guard=Compare(
                        left=Name(id='x', ctx=Load()),
                        ops=[
                            Gt()],
                        comparators=[
                            Constant(value=0)]),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))]),
                match_case(
                    pattern=MatchClass(
                        cls=Name(id='tuple', ctx=Load()),
                        patterns=[],
                        kwd_attrs=[],
                        kwd_patterns=[]),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))])])],
    type_ignores=[])
@end example
@end deffn

@geindex MatchValue (class in ast)
@anchor{library/ast ast MatchValue}@anchor{3d5f}
@deffn {Class} ast.MatchValue (value)

A match literal or value pattern that compares by equality. @code{value} is
an expression node. Permitted value nodes are restricted as described in
the match statement documentation. This pattern succeeds if the match
subject is equal to the evaluated value.

@example
>>> print(ast.dump(ast.parse("""
... match x:
...     case "Relevant":
...         ...
... """), indent=4))
Module(
    body=[
        Match(
            subject=Name(id='x', ctx=Load()),
            cases=[
                match_case(
                    pattern=MatchValue(
                        value=Constant(value='Relevant')),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))])])],
    type_ignores=[])
@end example
@end deffn

@geindex MatchSingleton (class in ast)
@anchor{library/ast ast MatchSingleton}@anchor{3d60}
@deffn {Class} ast.MatchSingleton (value)

A match literal pattern that compares by identity. @code{value} is the
singleton to be compared against: @code{None}, @code{True}, or @code{False}. This
pattern succeeds if the match subject is the given constant.

@example
>>> print(ast.dump(ast.parse("""
... match x:
...     case None:
...         ...
... """), indent=4))
Module(
    body=[
        Match(
            subject=Name(id='x', ctx=Load()),
            cases=[
                match_case(
                    pattern=MatchSingleton(value=None),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))])])],
    type_ignores=[])
@end example
@end deffn

@geindex MatchSequence (class in ast)
@anchor{library/ast ast MatchSequence}@anchor{3d61}
@deffn {Class} ast.MatchSequence (patterns)

A match sequence pattern. @code{patterns} contains the patterns to be matched
against the subject elements if the subject is a sequence. Matches a variable
length sequence if one of the subpatterns is a @code{MatchStar} node, otherwise
matches a fixed length sequence.

@example
>>> print(ast.dump(ast.parse("""
... match x:
...     case [1, 2]:
...         ...
... """), indent=4))
Module(
    body=[
        Match(
            subject=Name(id='x', ctx=Load()),
            cases=[
                match_case(
                    pattern=MatchSequence(
                        patterns=[
                            MatchValue(
                                value=Constant(value=1)),
                            MatchValue(
                                value=Constant(value=2))]),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))])])],
    type_ignores=[])
@end example
@end deffn

@geindex MatchStar (class in ast)
@anchor{library/ast ast MatchStar}@anchor{3d62}
@deffn {Class} ast.MatchStar (name)

Matches the rest of the sequence in a variable length match sequence pattern.
If @code{name} is not @code{None}, a list containing the remaining sequence
elements is bound to that name if the overall sequence pattern is successful.

@example
>>> print(ast.dump(ast.parse("""
... match x:
...     case [1, 2, *rest]:
...         ...
...     case [*_]:
...         ...
... """), indent=4))
Module(
    body=[
        Match(
            subject=Name(id='x', ctx=Load()),
            cases=[
                match_case(
                    pattern=MatchSequence(
                        patterns=[
                            MatchValue(
                                value=Constant(value=1)),
                            MatchValue(
                                value=Constant(value=2)),
                            MatchStar(name='rest')]),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))]),
                match_case(
                    pattern=MatchSequence(
                        patterns=[
                            MatchStar()]),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))])])],
    type_ignores=[])
@end example
@end deffn

@geindex MatchMapping (class in ast)
@anchor{library/ast ast MatchMapping}@anchor{3d63}
@deffn {Class} ast.MatchMapping (keys, patterns, rest)

A match mapping pattern. @code{keys} is a sequence of expression nodes.
@code{patterns} is a corresponding sequence of pattern nodes. @code{rest} is an
optional name that can be specified to capture the remaining mapping elements.
Permitted key expressions are restricted as described in the match statement
documentation.

This pattern succeeds if the subject is a mapping, all evaluated key
expressions are present in the mapping, and the value corresponding to each
key matches the corresponding subpattern. If @code{rest} is not @code{None}, a dict
containing the remaining mapping elements is bound to that name if the overall
mapping pattern is successful.

@example
>>> print(ast.dump(ast.parse("""
... match x:
...     case @{1: _, 2: _@}:
...         ...
...     case @{**rest@}:
...         ...
... """), indent=4))
Module(
    body=[
        Match(
            subject=Name(id='x', ctx=Load()),
            cases=[
                match_case(
                    pattern=MatchMapping(
                        keys=[
                            Constant(value=1),
                            Constant(value=2)],
                        patterns=[
                            MatchAs(),
                            MatchAs()]),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))]),
                match_case(
                    pattern=MatchMapping(keys=[], patterns=[], rest='rest'),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))])])],
    type_ignores=[])
@end example
@end deffn

@geindex MatchClass (class in ast)
@anchor{library/ast ast MatchClass}@anchor{3d64}
@deffn {Class} ast.MatchClass (cls, patterns, kwd_attrs, kwd_patterns)

A match class pattern. @code{cls} is an expression giving the nominal class to
be matched. @code{patterns} is a sequence of pattern nodes to be matched against
the class defined sequence of pattern matching attributes. @code{kwd_attrs} is a
sequence of additional attributes to be matched (specified as keyword arguments
in the class pattern), @code{kwd_patterns} are the corresponding patterns
(specified as keyword values in the class pattern).

This pattern succeeds if the subject is an instance of the nominated class,
all positional patterns match the corresponding class-defined attributes, and
any specified keyword attributes match their corresponding pattern.

Note: classes may define a property that returns self in order to match a
pattern node against the instance being matched. Several builtin types are
also matched that way, as described in the match statement documentation.

@example
>>> print(ast.dump(ast.parse("""
... match x:
...     case Point2D(0, 0):
...         ...
...     case Point3D(x=0, y=0, z=0):
...         ...
... """), indent=4))
Module(
    body=[
        Match(
            subject=Name(id='x', ctx=Load()),
            cases=[
                match_case(
                    pattern=MatchClass(
                        cls=Name(id='Point2D', ctx=Load()),
                        patterns=[
                            MatchValue(
                                value=Constant(value=0)),
                            MatchValue(
                                value=Constant(value=0))],
                        kwd_attrs=[],
                        kwd_patterns=[]),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))]),
                match_case(
                    pattern=MatchClass(
                        cls=Name(id='Point3D', ctx=Load()),
                        patterns=[],
                        kwd_attrs=[
                            'x',
                            'y',
                            'z'],
                        kwd_patterns=[
                            MatchValue(
                                value=Constant(value=0)),
                            MatchValue(
                                value=Constant(value=0)),
                            MatchValue(
                                value=Constant(value=0))]),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))])])],
    type_ignores=[])
@end example
@end deffn

@geindex MatchAs (class in ast)
@anchor{library/ast ast MatchAs}@anchor{3d65}
@deffn {Class} ast.MatchAs (pattern, name)

A match “as-pattern”, capture pattern or wildcard pattern. @code{pattern}
contains the match pattern that the subject will be matched against.
If the pattern is @code{None}, the node represents a capture pattern (i.e a
bare name) and will always succeed.

The @code{name} attribute contains the name that will be bound if the pattern
is successful. If @code{name} is @code{None}, @code{pattern} must also be @code{None}
and the node represents the wildcard pattern.

@example
>>> print(ast.dump(ast.parse("""
... match x:
...     case [x] as y:
...         ...
...     case _:
...         ...
... """), indent=4))
Module(
    body=[
        Match(
            subject=Name(id='x', ctx=Load()),
            cases=[
                match_case(
                    pattern=MatchAs(
                        pattern=MatchSequence(
                            patterns=[
                                MatchAs(name='x')]),
                        name='y'),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))]),
                match_case(
                    pattern=MatchAs(),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))])])],
    type_ignores=[])
@end example
@end deffn

@geindex MatchOr (class in ast)
@anchor{library/ast ast MatchOr}@anchor{3d66}
@deffn {Class} ast.MatchOr (patterns)

A match “or-pattern”. An or-pattern matches each of its subpatterns in turn
to the subject, until one succeeds. The or-pattern is then deemed to
succeed. If none of the subpatterns succeed the or-pattern fails. The
@code{patterns} attribute contains a list of match pattern nodes that will be
matched against the subject.

@example
>>> print(ast.dump(ast.parse("""
... match x:
...     case [x] | (y):
...         ...
... """), indent=4))
Module(
    body=[
        Match(
            subject=Name(id='x', ctx=Load()),
            cases=[
                match_case(
                    pattern=MatchOr(
                        patterns=[
                            MatchSequence(
                                patterns=[
                                    MatchAs(name='x')]),
                            MatchAs(name='y')]),
                    body=[
                        Expr(
                            value=Constant(value=Ellipsis))])])],
    type_ignores=[])
@end example
@end deffn

@node Function and class definitions,Async and await,Pattern matching,Node classes
@anchor{library/ast function-and-class-definitions}@anchor{3d67}
@subsubsection Function and class definitions


@geindex FunctionDef (class in ast)
@anchor{library/ast ast FunctionDef}@anchor{3d68}
@deffn {Class} ast.FunctionDef (name, args, body, decorator_list, returns, type_comment)

A function definition.


@itemize *

@item 
@code{name} is a raw string of the function name.

@item 
@code{args} is an @ref{3d69,,arguments} node.

@item 
@code{body} is the list of nodes inside the function.

@item 
@code{decorator_list} is the list of decorators to be applied, stored outermost
first (i.e. the first in the list will be applied last).

@item 
@code{returns} is the return annotation.
@end itemize

@geindex type_comment (ast.FunctionDef attribute)
@anchor{library/ast ast FunctionDef type_comment}@anchor{3d6a}
@deffn {Attribute} type_comment

@code{type_comment} is an optional string with the type annotation as a comment.
@end deffn
@end deffn

@geindex Lambda (class in ast)
@anchor{library/ast ast Lambda}@anchor{3d14}
@deffn {Class} ast.Lambda (args, body)

@code{lambda} is a minimal function definition that can be used inside an
expression. Unlike @ref{3d68,,FunctionDef}, @code{body} holds a single node.

@example
>>> print(ast.dump(ast.parse('lambda x,y: ...'), indent=4))
Module(
    body=[
        Expr(
            value=Lambda(
                args=arguments(
                    posonlyargs=[],
                    args=[
                        arg(arg='x'),
                        arg(arg='y')],
                    kwonlyargs=[],
                    kw_defaults=[],
                    defaults=[]),
                body=Constant(value=Ellipsis)))],
    type_ignores=[])
@end example
@end deffn

@geindex arguments (class in ast)
@anchor{library/ast ast arguments}@anchor{3d69}
@deffn {Class} ast.arguments (posonlyargs, args, vararg, kwonlyargs, kw_defaults, kwarg, defaults)

The arguments for a function.


@itemize *

@item 
@code{posonlyargs}, @code{args} and @code{kwonlyargs} are lists of @ref{3d6b,,arg} nodes.

@item 
@code{vararg} and @code{kwarg} are single @ref{3d6b,,arg} nodes, referring to the
@code{*args, **kwargs} parameters.

@item 
@code{kw_defaults} is a list of default values for keyword-only arguments. If
one is @code{None}, the corresponding argument is required.

@item 
@code{defaults} is a list of default values for arguments that can be passed
positionally. If there are fewer defaults, they correspond to the last n
arguments.
@end itemize
@end deffn

@geindex arg (class in ast)
@anchor{library/ast ast arg}@anchor{3d6b}
@deffn {Class} ast.arg (arg, annotation, type_comment)

A single argument in a list. @code{arg} is a raw string of the argument
name, @code{annotation} is its annotation, such as a @code{Str} or
@ref{11bc,,Name} node.

@geindex type_comment (ast.arg attribute)
@anchor{library/ast ast arg type_comment}@anchor{3d6c}
@deffn {Attribute} type_comment

@code{type_comment} is an optional string with the type annotation as a comment
@end deffn

@example
>>> print(ast.dump(ast.parse("""\
... @@decorator1
... @@decorator2
... def f(a: 'annotation', b=1, c=2, *d, e, f=3, **g) -> 'return annotation':
...     pass
... """), indent=4))
Module(
    body=[
        FunctionDef(
            name='f',
            args=arguments(
                posonlyargs=[],
                args=[
                    arg(
                        arg='a',
                        annotation=Constant(value='annotation')),
                    arg(arg='b'),
                    arg(arg='c')],
                vararg=arg(arg='d'),
                kwonlyargs=[
                    arg(arg='e'),
                    arg(arg='f')],
                kw_defaults=[
                    None,
                    Constant(value=3)],
                kwarg=arg(arg='g'),
                defaults=[
                    Constant(value=1),
                    Constant(value=2)]),
            body=[
                Pass()],
            decorator_list=[
                Name(id='decorator1', ctx=Load()),
                Name(id='decorator2', ctx=Load())],
            returns=Constant(value='return annotation'))],
    type_ignores=[])
@end example
@end deffn

@geindex Return (class in ast)
@anchor{library/ast ast Return}@anchor{3d6d}
@deffn {Class} ast.Return (value)

A @code{return} statement.

@example
>>> print(ast.dump(ast.parse('return 4'), indent=4))
Module(
    body=[
        Return(
            value=Constant(value=4))],
    type_ignores=[])
@end example
@end deffn

@geindex Yield (class in ast)
@geindex YieldFrom (class in ast)
@anchor{library/ast ast Yield}@anchor{3d15}
@deffn {Class} ast.Yield (value)
@anchor{library/ast ast YieldFrom}@anchor{3d16}
@deffnx {Class} ast.YieldFrom (value)

A @code{yield} or @code{yield from} expression. Because these are expressions, they
must be wrapped in a @ref{3d13,,Expr} node if the value sent back is not used.

@example
>>> print(ast.dump(ast.parse('yield x'), indent=4))
Module(
    body=[
        Expr(
            value=Yield(
                value=Name(id='x', ctx=Load())))],
    type_ignores=[])

>>> print(ast.dump(ast.parse('yield from x'), indent=4))
Module(
    body=[
        Expr(
            value=YieldFrom(
                value=Name(id='x', ctx=Load())))],
    type_ignores=[])
@end example
@end deffn

@geindex Global (class in ast)
@geindex Nonlocal (class in ast)
@anchor{library/ast ast Global}@anchor{3d6e}
@deffn {Class} ast.Global (names)
@anchor{library/ast ast Nonlocal}@anchor{3d6f}
@deffnx {Class} ast.Nonlocal (names)

@code{global} and @code{nonlocal} statements. @code{names} is a list of raw strings.

@example
>>> print(ast.dump(ast.parse('global x,y,z'), indent=4))
Module(
    body=[
        Global(
            names=[
                'x',
                'y',
                'z'])],
    type_ignores=[])

>>> print(ast.dump(ast.parse('nonlocal x,y,z'), indent=4))
Module(
    body=[
        Nonlocal(
            names=[
                'x',
                'y',
                'z'])],
    type_ignores=[])
@end example
@end deffn

@geindex ClassDef (class in ast)
@anchor{library/ast ast ClassDef}@anchor{3d70}
@deffn {Class} ast.ClassDef (name, bases, keywords, starargs, kwargs, body, decorator_list)

A class definition.


@itemize *

@item 
@code{name} is a raw string for the class name

@item 
@code{bases} is a list of nodes for explicitly specified base classes.

@item 
@code{keywords} is a list of @ref{a2,,keyword} nodes, principally for ‘metaclass’.
Other keywords will be passed to the metaclass, as per PEP-3115@footnote{https://www.python.org/dev/peps/pep-3115/}.

@item 
@code{starargs} and @code{kwargs} are each a single node, as in a function call.
starargs will be expanded to join the list of base classes, and kwargs will
be passed to the metaclass.

@item 
@code{body} is a list of nodes representing the code within the class
definition.

@item 
@code{decorator_list} is a list of nodes, as in @ref{3d68,,FunctionDef}.
@end itemize

@example
>>> print(ast.dump(ast.parse("""\
... @@decorator1
... @@decorator2
... class Foo(base1, base2, metaclass=meta):
...     pass
... """), indent=4))
Module(
    body=[
        ClassDef(
            name='Foo',
            bases=[
                Name(id='base1', ctx=Load()),
                Name(id='base2', ctx=Load())],
            keywords=[
                keyword(
                    arg='metaclass',
                    value=Name(id='meta', ctx=Load()))],
            body=[
                Pass()],
            decorator_list=[
                Name(id='decorator1', ctx=Load()),
                Name(id='decorator2', ctx=Load())])],
    type_ignores=[])
@end example
@end deffn

@node Async and await,,Function and class definitions,Node classes
@anchor{library/ast async-and-await}@anchor{3d71}
@subsubsection Async and await


@geindex AsyncFunctionDef (class in ast)
@anchor{library/ast ast AsyncFunctionDef}@anchor{3d72}
@deffn {Class} ast.AsyncFunctionDef (name, args, body, decorator_list, returns, type_comment)

An @code{async def} function definition. Has the same fields as
@ref{3d68,,FunctionDef}.
@end deffn

@geindex Await (class in ast)
@anchor{library/ast ast Await}@anchor{3d73}
@deffn {Class} ast.Await (value)

An @code{await} expression. @code{value} is what it waits for.
Only valid in the body of an @ref{3d72,,AsyncFunctionDef}.
@end deffn

@example
>>> print(ast.dump(ast.parse("""\
... async def f():
...     await other_func()
... """), indent=4))
Module(
    body=[
        AsyncFunctionDef(
            name='f',
            args=arguments(
                posonlyargs=[],
                args=[],
                kwonlyargs=[],
                kw_defaults=[],
                defaults=[]),
            body=[
                Expr(
                    value=Await(
                        value=Call(
                            func=Name(id='other_func', ctx=Load()),
                            args=[],
                            keywords=[])))],
            decorator_list=[])],
    type_ignores=[])
@end example

@geindex AsyncFor (class in ast)
@geindex AsyncWith (class in ast)
@anchor{library/ast ast AsyncFor}@anchor{3d74}
@deffn {Class} ast.AsyncFor (target, iter, body, orelse, type_comment)
@anchor{library/ast ast AsyncWith}@anchor{3d75}
@deffnx {Class} ast.AsyncWith (items, body, type_comment)

@code{async for} loops and @code{async with} context managers. They have the same
fields as @ref{3d51,,For} and @ref{3d59,,With}, respectively. Only valid in the
body of an @ref{3d72,,AsyncFunctionDef}.
@end deffn

@cartouche
@quotation Note 
When a string is parsed by @ref{4d3,,ast.parse()}, operator nodes (subclasses
of @code{ast.operator}, @code{ast.unaryop}, @code{ast.cmpop},
@code{ast.boolop} and @code{ast.expr_context}) on the returned tree
will be singletons. Changes to one will be reflected in all other
occurrences of the same value (e.g. @ref{3d1b,,ast.Add}).
@end quotation
@end cartouche

@node ast Helpers,Compiler Flags,Node classes,ast — Abstract Syntax Trees
@anchor{library/ast ast-helpers}@anchor{3d76}
@subsubsection @code{ast} Helpers


Apart from the node classes, the @ref{7,,ast} module defines these utility functions
and classes for traversing abstract syntax trees:

@geindex parse() (in module ast)
@anchor{library/ast ast parse}@anchor{4d3}
@deffn {Function} ast.parse (source, filename='<unknown>', mode='exec', *, type_comments=False, feature_version=None)

Parse the source into an AST node.  Equivalent to @code{compile(source,
filename, mode, ast.PyCF_ONLY_AST)}.

If @code{type_comments=True} is given, the parser is modified to check
and return type comments as specified by 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} and 
@geindex Python Enhancement Proposals; PEP 526
PEP 526@footnote{https://www.python.org/dev/peps/pep-0526}.
This is equivalent to adding @ref{3d77,,ast.PyCF_TYPE_COMMENTS} to the
flags passed to @ref{4e2,,compile()}.  This will report syntax errors
for misplaced type comments.  Without this flag, type comments will
be ignored, and the @code{type_comment} field on selected AST nodes
will always be @code{None}.  In addition, the locations of @code{# type:
ignore} comments will be returned as the @code{type_ignores}
attribute of @code{Module} (otherwise it is always an empty list).

In addition, if @code{mode} is @code{'func_type'}, the input syntax is
modified to correspond to 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} “signature type comments”,
e.g. @code{(str, int) -> List[str]}.

Also, setting @code{feature_version} to a tuple @code{(major, minor)}
will attempt to parse using that Python version’s grammar.
Currently @code{major} must equal to @code{3}.  For example, setting
@code{feature_version=(3, 4)} will allow the use of @code{async} and
@code{await} as variable names.  The lowest supported version is
@code{(3, 4)}; the highest is @code{sys.version_info[0:2]}.

If source contains a null character (’0’), @ref{17c,,ValueError} is raised.

@quotation

@cartouche
@quotation Warning 
Note that successfully parsing source code into an AST object doesn’t
guarantee that the source code provided is valid Python code that can
be executed as the compilation step can raise further @ref{21a,,SyntaxError}
exceptions. For instance, the source @code{return 42} generates a valid
AST node for a return statement, but it cannot be compiled alone (it needs
to be inside a function node).

In particular, @ref{4d3,,ast.parse()} won’t do any scoping checks, which the
compilation step does.
@end quotation
@end cartouche
@end quotation

@cartouche
@quotation Warning 
It is possible to crash the Python interpreter with a
sufficiently large/complex string due to stack depth limitations
in Python’s AST compiler.
@end quotation
@end cartouche


Changed in version 3.8: Added @code{type_comments}, @code{mode='func_type'} and @code{feature_version}.

@end deffn

@geindex unparse() (in module ast)
@anchor{library/ast ast unparse}@anchor{3cc}
@deffn {Function} ast.unparse (ast_obj)

Unparse an @ref{3cd,,ast.AST} object and generate a string with code
that would produce an equivalent @ref{3cd,,ast.AST} object if parsed
back with @ref{4d3,,ast.parse()}.

@cartouche
@quotation Warning 
The produced code string will not necessarily be equal to the original
code that generated the @ref{3cd,,ast.AST} object (without any compiler
optimizations, such as constant tuples/frozensets).
@end quotation
@end cartouche

@cartouche
@quotation Warning 
Trying to unparse a highly complex expression would result with
@ref{8c4,,RecursionError}.
@end quotation
@end cartouche


New in version 3.9.

@end deffn

@geindex literal_eval() (in module ast)
@anchor{library/ast ast literal_eval}@anchor{754}
@deffn {Function} ast.literal_eval (node_or_string)

Safely evaluate an expression node or a string containing a Python literal or
container display.  The string or node provided may only consist of the
following Python literal structures: strings, bytes, numbers, tuples, lists,
dicts, sets, booleans, @code{None} and @code{Ellipsis}.

This can be used for safely evaluating strings containing Python values from
untrusted sources without the need to parse the values oneself.  It is not
capable of evaluating arbitrarily complex expressions, for example involving
operators or indexing.

@cartouche
@quotation Warning 
It is possible to crash the Python interpreter with a
sufficiently large/complex string due to stack depth limitations
in Python’s AST compiler.

It can raise @ref{17c,,ValueError}, @ref{162,,TypeError}, @ref{21a,,SyntaxError},
@ref{10de,,MemoryError} and @ref{8c4,,RecursionError} depending on the malformed
input.
@end quotation
@end cartouche


Changed in version 3.2: Now allows bytes and set literals.



Changed in version 3.9: Now supports creating empty sets with @code{'set()'}.



Changed in version 3.10: For string inputs, leading spaces and tabs are now stripped.

@end deffn

@geindex get_docstring() (in module ast)
@anchor{library/ast ast get_docstring}@anchor{1318}
@deffn {Function} ast.get_docstring (node, clean=True)

Return the docstring of the given @emph{node} (which must be a
@ref{3d68,,FunctionDef}, @ref{3d72,,AsyncFunctionDef}, @ref{3d70,,ClassDef},
or @code{Module} node), or @code{None} if it has no docstring.
If @emph{clean} is true, clean up the docstring’s indentation with
@ref{3c1d,,inspect.cleandoc()}.


Changed in version 3.5: @ref{3d72,,AsyncFunctionDef} is now supported.

@end deffn

@geindex get_source_segment() (in module ast)
@anchor{library/ast ast get_source_segment}@anchor{4d2}
@deffn {Function} ast.get_source_segment (source, node, *, padded=False)

Get source code segment of the @emph{source} that generated @emph{node}.
If some location information (@code{lineno}, @code{end_lineno},
@code{col_offset}, or @code{end_col_offset}) is missing, return @code{None}.

If @emph{padded} is @code{True}, the first line of a multi-line statement will
be padded with spaces to match its original position.


New in version 3.8.

@end deffn

@geindex fix_missing_locations() (in module ast)
@anchor{library/ast ast fix_missing_locations}@anchor{3d78}
@deffn {Function} ast.fix_missing_locations (node)

When you compile a node tree with @ref{4e2,,compile()}, the compiler expects
@code{lineno} and @code{col_offset} attributes for every node that supports
them.  This is rather tedious to fill in for generated nodes, so this helper
adds these attributes recursively where not already set, by setting them to
the values of the parent node.  It works recursively starting at @emph{node}.
@end deffn

@geindex increment_lineno() (in module ast)
@anchor{library/ast ast increment_lineno}@anchor{11ca}
@deffn {Function} ast.increment_lineno (node, n=1)

Increment the line number and end line number of each node in the tree
starting at @emph{node} by @emph{n}. This is useful to “move code” to a different
location in a file.
@end deffn

@geindex copy_location() (in module ast)
@anchor{library/ast ast copy_location}@anchor{3d79}
@deffn {Function} ast.copy_location (new_node, old_node)

Copy source location (@code{lineno}, @code{col_offset}, @code{end_lineno},
and @code{end_col_offset}) from @emph{old_node} to @emph{new_node} if possible,
and return @emph{new_node}.
@end deffn

@geindex iter_fields() (in module ast)
@anchor{library/ast ast iter_fields}@anchor{3d7a}
@deffn {Function} ast.iter_fields (node)

Yield a tuple of @code{(fieldname, value)} for each field in @code{node._fields}
that is present on @emph{node}.
@end deffn

@geindex iter_child_nodes() (in module ast)
@anchor{library/ast ast iter_child_nodes}@anchor{3d7b}
@deffn {Function} ast.iter_child_nodes (node)

Yield all direct child nodes of @emph{node}, that is, all fields that are nodes
and all items of fields that are lists of nodes.
@end deffn

@geindex walk() (in module ast)
@anchor{library/ast ast walk}@anchor{3d7c}
@deffn {Function} ast.walk (node)

Recursively yield all descendant nodes in the tree starting at @emph{node}
(including @emph{node} itself), in no specified order.  This is useful if you only
want to modify nodes in place and don’t care about the context.
@end deffn

@geindex NodeVisitor (class in ast)
@anchor{library/ast ast NodeVisitor}@anchor{589}
@deffn {Class} ast.NodeVisitor

A node visitor base class that walks the abstract syntax tree and calls a
visitor function for every node found.  This function may return a value
which is forwarded by the @ref{3d7d,,visit()} method.

This class is meant to be subclassed, with the subclass adding visitor
methods.

@geindex visit() (ast.NodeVisitor method)
@anchor{library/ast ast NodeVisitor visit}@anchor{3d7d}
@deffn {Method} visit (node)

Visit a node.  The default implementation calls the method called
@code{self.visit_@emph{classname}} where @emph{classname} is the name of the node
class, or @ref{3d7e,,generic_visit()} if that method doesn’t exist.
@end deffn

@geindex generic_visit() (ast.NodeVisitor method)
@anchor{library/ast ast NodeVisitor generic_visit}@anchor{3d7e}
@deffn {Method} generic_visit (node)

This visitor calls @ref{3d7d,,visit()} on all children of the node.

Note that child nodes of nodes that have a custom visitor method won’t be
visited unless the visitor calls @ref{3d7e,,generic_visit()} or visits them
itself.
@end deffn

Don’t use the @ref{589,,NodeVisitor} if you want to apply changes to nodes
during traversal.  For this a special visitor exists
(@ref{3d7f,,NodeTransformer}) that allows modifications.


Deprecated since version 3.8: Methods @code{visit_Num()}, @code{visit_Str()}, @code{visit_Bytes()},
@code{visit_NameConstant()} and @code{visit_Ellipsis()} are deprecated
now and will not be called in future Python versions.  Add the
@code{visit_Constant()} method to handle all constant nodes.

@end deffn

@geindex NodeTransformer (class in ast)
@anchor{library/ast ast NodeTransformer}@anchor{3d7f}
@deffn {Class} ast.NodeTransformer

A @ref{589,,NodeVisitor} subclass that walks the abstract syntax tree and
allows modification of nodes.

The @ref{3d7f,,NodeTransformer} will walk the AST and use the return value of
the visitor methods to replace or remove the old node.  If the return value
of the visitor method is @code{None}, the node will be removed from its
location, otherwise it is replaced with the return value.  The return value
may be the original node in which case no replacement takes place.

Here is an example transformer that rewrites all occurrences of name lookups
(@code{foo}) to @code{data['foo']}:

@example
class RewriteName(NodeTransformer):

    def visit_Name(self, node):
        return Subscript(
            value=Name(id='data', ctx=Load()),
            slice=Constant(value=node.id),
            ctx=node.ctx
        )
@end example

Keep in mind that if the node you’re operating on has child nodes you must
either transform the child nodes yourself or call the @code{generic_visit()}
method for the node first.

For nodes that were part of a collection of statements (that applies to all
statement nodes), the visitor may also return a list of nodes rather than
just a single node.

If @ref{3d7f,,NodeTransformer} introduces new nodes (that weren’t part of
original tree) without giving them location information (such as
@code{lineno}), @ref{3d78,,fix_missing_locations()} should be called with
the new sub-tree to recalculate the location information:

@example
tree = ast.parse('foo', mode='eval')
new_tree = fix_missing_locations(RewriteName().visit(tree))
@end example

Usually you use the transformer like this:

@example
node = YourTransformer().visit(node)
@end example
@end deffn

@geindex dump() (in module ast)
@anchor{library/ast ast dump}@anchor{3cb}
@deffn {Function} ast.dump (node, annotate_fields=True, include_attributes=False, *, indent=None)

Return a formatted dump of the tree in @emph{node}.  This is mainly useful for
debugging purposes.  If @emph{annotate_fields} is true (by default),
the returned string will show the names and the values for fields.
If @emph{annotate_fields} is false, the result string will be more compact by
omitting unambiguous field names.  Attributes such as line
numbers and column offsets are not dumped by default.  If this is wanted,
@emph{include_attributes} can be set to true.

If @emph{indent} is a non-negative integer or string, then the tree will be
pretty-printed with that indent level.  An indent level
of 0, negative, or @code{""} will only insert newlines.  @code{None} (the default)
selects the single line representation. Using a positive integer indent
indents that many spaces per level.  If @emph{indent} is a string (such as @code{"\t"}),
that string is used to indent each level.


Changed in version 3.9: Added the @emph{indent} option.

@end deffn

@node Compiler Flags,Command-Line Usage<2>,ast Helpers,ast — Abstract Syntax Trees
@anchor{library/ast ast-compiler-flags}@anchor{1a42}@anchor{library/ast compiler-flags}@anchor{3d80}
@subsubsection Compiler Flags


The following flags may be passed to @ref{4e2,,compile()} in order to change
effects on the compilation of a program:

@geindex PyCF_ALLOW_TOP_LEVEL_AWAIT (in module ast)
@anchor{library/ast ast PyCF_ALLOW_TOP_LEVEL_AWAIT}@anchor{3d81}
@deffn {Data} ast.PyCF_ALLOW_TOP_LEVEL_AWAIT

Enables support for top-level @code{await}, @code{async for}, @code{async with}
and async comprehensions.


New in version 3.8.

@end deffn

@geindex PyCF_ONLY_AST (in module ast)
@anchor{library/ast ast PyCF_ONLY_AST}@anchor{3cfc}
@deffn {Data} ast.PyCF_ONLY_AST

Generates and returns an abstract syntax tree instead of returning a
compiled code object.
@end deffn

@geindex PyCF_TYPE_COMMENTS (in module ast)
@anchor{library/ast ast PyCF_TYPE_COMMENTS}@anchor{3d77}
@deffn {Data} ast.PyCF_TYPE_COMMENTS

Enables support for 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484} and 
@geindex Python Enhancement Proposals; PEP 526
PEP 526@footnote{https://www.python.org/dev/peps/pep-0526} style type comments
(@code{# type: <type>}, @code{# type: ignore <stuff>}).


New in version 3.8.

@end deffn

@node Command-Line Usage<2>,,Compiler Flags,ast — Abstract Syntax Trees
@anchor{library/ast ast-cli}@anchor{3d82}@anchor{library/ast command-line-usage}@anchor{3d83}
@subsubsection Command-Line Usage



New in version 3.9.


The @ref{7,,ast} module can be executed as a script from the command line.
It is as simple as:

@example
python -m ast [-m <mode>] [-a] [infile]
@end example

The following options are accepted:

@geindex ast command line option; -h
@geindex ast command line option; --help
@anchor{library/ast cmdoption-ast-h}@anchor{3d84}@anchor{library/ast cmdoption-ast-help}@anchor{3d85}
@deffn {Program Option} @w{-}h, @w{-}@w{-}help

Show the help message and exit.
@end deffn

@geindex ast command line option; -m <mode>
@geindex ast command line option; --mode <mode>
@anchor{library/ast cmdoption-ast-m}@anchor{3d86}
@deffn {Program Option} @w{-}m <mode>
@anchor{library/ast cmdoption-ast-mode}@anchor{3d87}
@deffnx {Program Option} @w{-}@w{-}mode <mode>

Specify what kind of code must be compiled, like the @emph{mode} argument
in @ref{4d3,,parse()}.
@end deffn

@geindex ast command line option; --no-type-comments
@anchor{library/ast cmdoption-ast-no-type-comments}@anchor{3d88}
@deffn {Program Option} @w{-}@w{-}no@w{-}type@w{-}comments

Don’t parse type comments.
@end deffn

@geindex ast command line option; -a
@geindex ast command line option; --include-attributes
@anchor{library/ast cmdoption-ast-a}@anchor{3d89}@anchor{library/ast cmdoption-ast-include-attributes}@anchor{3d8a}
@deffn {Program Option} @w{-}a, @w{-}@w{-}include@w{-}attributes

Include attributes such as line numbers and column offsets.
@end deffn

@geindex ast command line option; -i <indent>
@geindex ast command line option; --indent <indent>
@anchor{library/ast cmdoption-ast-i}@anchor{3d8b}
@deffn {Program Option} @w{-}i <indent>
@anchor{library/ast cmdoption-ast-indent}@anchor{3d8c}
@deffnx {Program Option} @w{-}@w{-}indent <indent>

Indentation of nodes in AST (number of spaces).
@end deffn

If @code{infile} is specified its contents are parsed to AST and dumped
to stdout.  Otherwise, the content is read from stdin.


@subsubheading See also


Green Tree Snakes@footnote{https://greentreesnakes.readthedocs.io/}, an external
documentation resource, has good details on working with Python ASTs.

ASTTokens@footnote{https://asttokens.readthedocs.io/en/latest/user-guide.html}
annotates Python ASTs with the positions of tokens and text in the source
code that generated them. This is helpful for tools that make source code
transformations.

leoAst.py@footnote{http://leoeditor.com/appendices.html#leoast-py} unifies the
token-based and parse-tree-based views of python programs by inserting
two-way links between tokens and ast nodes.

LibCST@footnote{https://libcst.readthedocs.io/} parses code as a Concrete Syntax
Tree that looks like an ast tree and keeps all formatting details. It’s
useful for building automated refactoring (codemod) applications and
linters.

Parso@footnote{https://parso.readthedocs.io} is a Python parser that supports
error recovery and round-trip parsing for different Python versions (in
multiple Python versions). Parso is also able to list multiple syntax errors
in your python file.


@node symtable — Access to the compiler’s symbol tables,token — Constants used with Python parse trees,ast — Abstract Syntax Trees,Python Language Services
@anchor{library/symtable doc}@anchor{3d8d}@anchor{library/symtable module-symtable}@anchor{f6}@anchor{library/symtable symtable-access-to-the-compiler-s-symbol-tables}@anchor{3d8e}
@subsection @code{symtable} — Access to the compiler’s symbol tables


@geindex module; symtable

@strong{Source code:} Lib/symtable.py@footnote{https://github.com/python/cpython/tree/main/Lib/symtable.py}


__________________________________________________________________


Symbol tables are generated by the compiler from AST just before bytecode is
generated.  The symbol table is responsible for calculating the scope of every
identifier in the code.  @ref{f6,,symtable} provides an interface to examine these
tables.

@menu
* Generating Symbol Tables:: 
* Examining Symbol Tables:: 

@end menu

@node Generating Symbol Tables,Examining Symbol Tables,,symtable — Access to the compiler’s symbol tables
@anchor{library/symtable generating-symbol-tables}@anchor{3d8f}
@subsubsection Generating Symbol Tables


@geindex symtable() (in module symtable)
@anchor{library/symtable symtable symtable}@anchor{1295}
@deffn {Function} symtable.symtable (code, filename, compile_type)

Return the toplevel @ref{3d90,,SymbolTable} for the Python source @emph{code}.
@emph{filename} is the name of the file containing the code.  @emph{compile_type} is
like the @emph{mode} argument to @ref{4e2,,compile()}.
@end deffn

@node Examining Symbol Tables,,Generating Symbol Tables,symtable — Access to the compiler’s symbol tables
@anchor{library/symtable examining-symbol-tables}@anchor{3d91}
@subsubsection Examining Symbol Tables


@geindex SymbolTable (class in symtable)
@anchor{library/symtable symtable SymbolTable}@anchor{3d90}
@deffn {Class} symtable.SymbolTable

A namespace table for a block.  The constructor is not public.

@geindex get_type() (symtable.SymbolTable method)
@anchor{library/symtable symtable SymbolTable get_type}@anchor{3d92}
@deffn {Method} get_type ()

Return the type of the symbol table.  Possible values are @code{'class'},
@code{'module'}, and @code{'function'}.
@end deffn

@geindex get_id() (symtable.SymbolTable method)
@anchor{library/symtable symtable SymbolTable get_id}@anchor{3d93}
@deffn {Method} get_id ()

Return the table’s identifier.
@end deffn

@geindex get_name() (symtable.SymbolTable method)
@anchor{library/symtable symtable SymbolTable get_name}@anchor{3d94}
@deffn {Method} get_name ()

Return the table’s name.  This is the name of the class if the table is
for a class, the name of the function if the table is for a function, or
@code{'top'} if the table is global (@ref{3d92,,get_type()} returns @code{'module'}).
@end deffn

@geindex get_lineno() (symtable.SymbolTable method)
@anchor{library/symtable symtable SymbolTable get_lineno}@anchor{3d95}
@deffn {Method} get_lineno ()

Return the number of the first line in the block this table represents.
@end deffn

@geindex is_optimized() (symtable.SymbolTable method)
@anchor{library/symtable symtable SymbolTable is_optimized}@anchor{3d96}
@deffn {Method} is_optimized ()

Return @code{True} if the locals in this table can be optimized.
@end deffn

@geindex is_nested() (symtable.SymbolTable method)
@anchor{library/symtable symtable SymbolTable is_nested}@anchor{3d97}
@deffn {Method} is_nested ()

Return @code{True} if the block is a nested class or function.
@end deffn

@geindex has_children() (symtable.SymbolTable method)
@anchor{library/symtable symtable SymbolTable has_children}@anchor{3d98}
@deffn {Method} has_children ()

Return @code{True} if the block has nested namespaces within it.  These can
be obtained with @ref{3d99,,get_children()}.
@end deffn

@geindex get_identifiers() (symtable.SymbolTable method)
@anchor{library/symtable symtable SymbolTable get_identifiers}@anchor{3d9a}
@deffn {Method} get_identifiers ()

Return a list of names of symbols in this table.
@end deffn

@geindex lookup() (symtable.SymbolTable method)
@anchor{library/symtable symtable SymbolTable lookup}@anchor{3d9b}
@deffn {Method} lookup (name)

Lookup @emph{name} in the table and return a @ref{3d9c,,Symbol} instance.
@end deffn

@geindex get_symbols() (symtable.SymbolTable method)
@anchor{library/symtable symtable SymbolTable get_symbols}@anchor{3d9d}
@deffn {Method} get_symbols ()

Return a list of @ref{3d9c,,Symbol} instances for names in the table.
@end deffn

@geindex get_children() (symtable.SymbolTable method)
@anchor{library/symtable symtable SymbolTable get_children}@anchor{3d99}
@deffn {Method} get_children ()

Return a list of the nested symbol tables.
@end deffn
@end deffn

@geindex Function (class in symtable)
@anchor{library/symtable symtable Function}@anchor{3d9e}
@deffn {Class} symtable.Function

A namespace for a function or method.  This class inherits
@ref{3d90,,SymbolTable}.

@geindex get_parameters() (symtable.Function method)
@anchor{library/symtable symtable Function get_parameters}@anchor{3d9f}
@deffn {Method} get_parameters ()

Return a tuple containing names of parameters to this function.
@end deffn

@geindex get_locals() (symtable.Function method)
@anchor{library/symtable symtable Function get_locals}@anchor{3da0}
@deffn {Method} get_locals ()

Return a tuple containing names of locals in this function.
@end deffn

@geindex get_globals() (symtable.Function method)
@anchor{library/symtable symtable Function get_globals}@anchor{3da1}
@deffn {Method} get_globals ()

Return a tuple containing names of globals in this function.
@end deffn

@geindex get_nonlocals() (symtable.Function method)
@anchor{library/symtable symtable Function get_nonlocals}@anchor{3da2}
@deffn {Method} get_nonlocals ()

Return a tuple containing names of nonlocals in this function.
@end deffn

@geindex get_frees() (symtable.Function method)
@anchor{library/symtable symtable Function get_frees}@anchor{3da3}
@deffn {Method} get_frees ()

Return a tuple containing names of free variables in this function.
@end deffn
@end deffn

@geindex Class (class in symtable)
@anchor{library/symtable symtable Class}@anchor{3da4}
@deffn {Class} symtable.Class

A namespace of a class.  This class inherits @ref{3d90,,SymbolTable}.

@geindex get_methods() (symtable.Class method)
@anchor{library/symtable symtable Class get_methods}@anchor{3da5}
@deffn {Method} get_methods ()

Return a tuple containing the names of methods declared in the class.
@end deffn
@end deffn

@geindex Symbol (class in symtable)
@anchor{library/symtable symtable Symbol}@anchor{3d9c}
@deffn {Class} symtable.Symbol

An entry in a @ref{3d90,,SymbolTable} corresponding to an identifier in the
source.  The constructor is not public.

@geindex get_name() (symtable.Symbol method)
@anchor{library/symtable symtable Symbol get_name}@anchor{3da6}
@deffn {Method} get_name ()

Return the symbol’s name.
@end deffn

@geindex is_referenced() (symtable.Symbol method)
@anchor{library/symtable symtable Symbol is_referenced}@anchor{3da7}
@deffn {Method} is_referenced ()

Return @code{True} if the symbol is used in its block.
@end deffn

@geindex is_imported() (symtable.Symbol method)
@anchor{library/symtable symtable Symbol is_imported}@anchor{3da8}
@deffn {Method} is_imported ()

Return @code{True} if the symbol is created from an import statement.
@end deffn

@geindex is_parameter() (symtable.Symbol method)
@anchor{library/symtable symtable Symbol is_parameter}@anchor{3da9}
@deffn {Method} is_parameter ()

Return @code{True} if the symbol is a parameter.
@end deffn

@geindex is_global() (symtable.Symbol method)
@anchor{library/symtable symtable Symbol is_global}@anchor{3daa}
@deffn {Method} is_global ()

Return @code{True} if the symbol is global.
@end deffn

@geindex is_nonlocal() (symtable.Symbol method)
@anchor{library/symtable symtable Symbol is_nonlocal}@anchor{12d7}
@deffn {Method} is_nonlocal ()

Return @code{True} if the symbol is nonlocal.
@end deffn

@geindex is_declared_global() (symtable.Symbol method)
@anchor{library/symtable symtable Symbol is_declared_global}@anchor{eb8}
@deffn {Method} is_declared_global ()

Return @code{True} if the symbol is declared global with a global statement.
@end deffn

@geindex is_local() (symtable.Symbol method)
@anchor{library/symtable symtable Symbol is_local}@anchor{3dab}
@deffn {Method} is_local ()

Return @code{True} if the symbol is local to its block.
@end deffn

@geindex is_annotated() (symtable.Symbol method)
@anchor{library/symtable symtable Symbol is_annotated}@anchor{3dac}
@deffn {Method} is_annotated ()

Return @code{True} if the symbol is annotated.


New in version 3.6.

@end deffn

@geindex is_free() (symtable.Symbol method)
@anchor{library/symtable symtable Symbol is_free}@anchor{3dad}
@deffn {Method} is_free ()

Return @code{True} if the symbol is referenced in its block, but not assigned
to.
@end deffn

@geindex is_assigned() (symtable.Symbol method)
@anchor{library/symtable symtable Symbol is_assigned}@anchor{3dae}
@deffn {Method} is_assigned ()

Return @code{True} if the symbol is assigned to in its block.
@end deffn

@geindex is_namespace() (symtable.Symbol method)
@anchor{library/symtable symtable Symbol is_namespace}@anchor{3daf}
@deffn {Method} is_namespace ()

Return @code{True} if name binding introduces new namespace.

If the name is used as the target of a function or class statement, this
will be true.

For example:

@example
>>> table = symtable.symtable("def some_func(): pass", "string", "exec")
>>> table.lookup("some_func").is_namespace()
True
@end example

Note that a single name can be bound to multiple objects.  If the result
is @code{True}, the name may also be bound to other objects, like an int or
list, that does not introduce a new namespace.
@end deffn

@geindex get_namespaces() (symtable.Symbol method)
@anchor{library/symtable symtable Symbol get_namespaces}@anchor{3db0}
@deffn {Method} get_namespaces ()

Return a list of namespaces bound to this name.
@end deffn

@geindex get_namespace() (symtable.Symbol method)
@anchor{library/symtable symtable Symbol get_namespace}@anchor{3db1}
@deffn {Method} get_namespace ()

Return the namespace bound to this name. If more than one or no namespace
is bound to this name, a @ref{17c,,ValueError} is raised.
@end deffn
@end deffn

@node token — Constants used with Python parse trees,keyword — Testing for Python keywords,symtable — Access to the compiler’s symbol tables,Python Language Services
@anchor{library/token doc}@anchor{3db2}@anchor{library/token module-token}@anchor{117}@anchor{library/token token-constants-used-with-python-parse-trees}@anchor{3db3}
@subsection @code{token} — Constants used with Python parse trees


@geindex module; token

@strong{Source code:} Lib/token.py@footnote{https://github.com/python/cpython/tree/main/Lib/token.py}


__________________________________________________________________


This module provides constants which represent the numeric values of leaf nodes
of the parse tree (terminal tokens).  Refer to the file @code{Grammar/Tokens}
in the Python distribution for the definitions of the names in the context of
the language grammar.  The specific numeric values which the names map to may
change between Python versions.

The module also provides a mapping from numeric codes to names and some
functions.  The functions mirror definitions in the Python C header files.

@geindex tok_name (in module token)
@anchor{library/token token tok_name}@anchor{3db4}
@deffn {Data} token.tok_name

Dictionary mapping the numeric values of the constants defined in this module
back to name strings, allowing more human-readable representation of parse trees
to be generated.
@end deffn

@geindex ISTERMINAL() (in module token)
@anchor{library/token token ISTERMINAL}@anchor{3db5}
@deffn {Function} token.ISTERMINAL (x)

Return @code{True} for terminal token values.
@end deffn

@geindex ISNONTERMINAL() (in module token)
@anchor{library/token token ISNONTERMINAL}@anchor{3db6}
@deffn {Function} token.ISNONTERMINAL (x)

Return @code{True} for non-terminal token values.
@end deffn

@geindex ISEOF() (in module token)
@anchor{library/token token ISEOF}@anchor{3db7}
@deffn {Function} token.ISEOF (x)

Return @code{True} if @emph{x} is the marker indicating the end of input.
@end deffn

The token constants are:

@c Auto-generated by Tools/scripts/generate_token.py

@geindex ENDMARKER (in module token)
@anchor{library/token token ENDMARKER}@anchor{3db8}
@deffn {Data} token.ENDMARKER
@end deffn

@geindex NAME (in module token)
@anchor{library/token token NAME}@anchor{3db9}
@deffn {Data} token.NAME
@end deffn

@geindex NUMBER (in module token)
@anchor{library/token token NUMBER}@anchor{3dba}
@deffn {Data} token.NUMBER
@end deffn

@geindex STRING (in module token)
@anchor{library/token token STRING}@anchor{3dbb}
@deffn {Data} token.STRING
@end deffn

@geindex NEWLINE (in module token)
@anchor{library/token token NEWLINE}@anchor{3dbc}
@deffn {Data} token.NEWLINE
@end deffn

@geindex INDENT (in module token)
@anchor{library/token token INDENT}@anchor{3dbd}
@deffn {Data} token.INDENT
@end deffn

@geindex DEDENT (in module token)
@anchor{library/token token DEDENT}@anchor{3dbe}
@deffn {Data} token.DEDENT
@end deffn

@geindex LPAR (in module token)
@anchor{library/token token LPAR}@anchor{3dbf}
@deffn {Data} token.LPAR

Token value for @code{"("}.
@end deffn

@geindex RPAR (in module token)
@anchor{library/token token RPAR}@anchor{3dc0}
@deffn {Data} token.RPAR

Token value for @code{")"}.
@end deffn

@geindex LSQB (in module token)
@anchor{library/token token LSQB}@anchor{3dc1}
@deffn {Data} token.LSQB

Token value for @code{"["}.
@end deffn

@geindex RSQB (in module token)
@anchor{library/token token RSQB}@anchor{3dc2}
@deffn {Data} token.RSQB

Token value for @code{"]"}.
@end deffn

@geindex COLON (in module token)
@anchor{library/token token COLON}@anchor{3dc3}
@deffn {Data} token.COLON

Token value for @code{":"}.
@end deffn

@geindex COMMA (in module token)
@anchor{library/token token COMMA}@anchor{3dc4}
@deffn {Data} token.COMMA

Token value for @code{","}.
@end deffn

@geindex SEMI (in module token)
@anchor{library/token token SEMI}@anchor{3dc5}
@deffn {Data} token.SEMI

Token value for @code{";"}.
@end deffn

@geindex PLUS (in module token)
@anchor{library/token token PLUS}@anchor{3dc6}
@deffn {Data} token.PLUS

Token value for @code{"+"}.
@end deffn

@geindex MINUS (in module token)
@anchor{library/token token MINUS}@anchor{3dc7}
@deffn {Data} token.MINUS

Token value for @code{"-"}.
@end deffn

@geindex STAR (in module token)
@anchor{library/token token STAR}@anchor{3dc8}
@deffn {Data} token.STAR

Token value for @code{"*"}.
@end deffn

@geindex SLASH (in module token)
@anchor{library/token token SLASH}@anchor{3dc9}
@deffn {Data} token.SLASH

Token value for @code{"/"}.
@end deffn

@geindex VBAR (in module token)
@anchor{library/token token VBAR}@anchor{3dca}
@deffn {Data} token.VBAR

Token value for @code{"|"}.
@end deffn

@geindex AMPER (in module token)
@anchor{library/token token AMPER}@anchor{3dcb}
@deffn {Data} token.AMPER

Token value for @code{"&"}.
@end deffn

@geindex LESS (in module token)
@anchor{library/token token LESS}@anchor{3dcc}
@deffn {Data} token.LESS

Token value for @code{"<"}.
@end deffn

@geindex GREATER (in module token)
@anchor{library/token token GREATER}@anchor{3dcd}
@deffn {Data} token.GREATER

Token value for @code{">"}.
@end deffn

@geindex EQUAL (in module token)
@anchor{library/token token EQUAL}@anchor{3dce}
@deffn {Data} token.EQUAL

Token value for @code{"="}.
@end deffn

@geindex DOT (in module token)
@anchor{library/token token DOT}@anchor{3dcf}
@deffn {Data} token.DOT

Token value for @code{"."}.
@end deffn

@geindex PERCENT (in module token)
@anchor{library/token token PERCENT}@anchor{3dd0}
@deffn {Data} token.PERCENT

Token value for @code{"%"}.
@end deffn

@geindex LBRACE (in module token)
@anchor{library/token token LBRACE}@anchor{3dd1}
@deffn {Data} token.LBRACE

Token value for @code{"@{"}.
@end deffn

@geindex RBRACE (in module token)
@anchor{library/token token RBRACE}@anchor{3dd2}
@deffn {Data} token.RBRACE

Token value for @code{"@}"}.
@end deffn

@geindex EQEQUAL (in module token)
@anchor{library/token token EQEQUAL}@anchor{3dd3}
@deffn {Data} token.EQEQUAL

Token value for @code{"=="}.
@end deffn

@geindex NOTEQUAL (in module token)
@anchor{library/token token NOTEQUAL}@anchor{3dd4}
@deffn {Data} token.NOTEQUAL

Token value for @code{"!="}.
@end deffn

@geindex LESSEQUAL (in module token)
@anchor{library/token token LESSEQUAL}@anchor{3dd5}
@deffn {Data} token.LESSEQUAL

Token value for @code{"<="}.
@end deffn

@geindex GREATEREQUAL (in module token)
@anchor{library/token token GREATEREQUAL}@anchor{3dd6}
@deffn {Data} token.GREATEREQUAL

Token value for @code{">="}.
@end deffn

@geindex TILDE (in module token)
@anchor{library/token token TILDE}@anchor{3dd7}
@deffn {Data} token.TILDE

Token value for @code{"~"}.
@end deffn

@geindex CIRCUMFLEX (in module token)
@anchor{library/token token CIRCUMFLEX}@anchor{3dd8}
@deffn {Data} token.CIRCUMFLEX

Token value for @code{"^"}.
@end deffn

@geindex LEFTSHIFT (in module token)
@anchor{library/token token LEFTSHIFT}@anchor{3dd9}
@deffn {Data} token.LEFTSHIFT

Token value for @code{"<<"}.
@end deffn

@geindex RIGHTSHIFT (in module token)
@anchor{library/token token RIGHTSHIFT}@anchor{3dda}
@deffn {Data} token.RIGHTSHIFT

Token value for @code{">>"}.
@end deffn

@geindex DOUBLESTAR (in module token)
@anchor{library/token token DOUBLESTAR}@anchor{3ddb}
@deffn {Data} token.DOUBLESTAR

Token value for @code{"**"}.
@end deffn

@geindex PLUSEQUAL (in module token)
@anchor{library/token token PLUSEQUAL}@anchor{3ddc}
@deffn {Data} token.PLUSEQUAL

Token value for @code{"+="}.
@end deffn

@geindex MINEQUAL (in module token)
@anchor{library/token token MINEQUAL}@anchor{3ddd}
@deffn {Data} token.MINEQUAL

Token value for @code{"-="}.
@end deffn

@geindex STAREQUAL (in module token)
@anchor{library/token token STAREQUAL}@anchor{3dde}
@deffn {Data} token.STAREQUAL

Token value for @code{"*="}.
@end deffn

@geindex SLASHEQUAL (in module token)
@anchor{library/token token SLASHEQUAL}@anchor{3ddf}
@deffn {Data} token.SLASHEQUAL

Token value for @code{"/="}.
@end deffn

@geindex PERCENTEQUAL (in module token)
@anchor{library/token token PERCENTEQUAL}@anchor{3de0}
@deffn {Data} token.PERCENTEQUAL

Token value for @code{"%="}.
@end deffn

@geindex AMPEREQUAL (in module token)
@anchor{library/token token AMPEREQUAL}@anchor{3de1}
@deffn {Data} token.AMPEREQUAL

Token value for @code{"&="}.
@end deffn

@geindex VBAREQUAL (in module token)
@anchor{library/token token VBAREQUAL}@anchor{3de2}
@deffn {Data} token.VBAREQUAL

Token value for @code{"|="}.
@end deffn

@geindex CIRCUMFLEXEQUAL (in module token)
@anchor{library/token token CIRCUMFLEXEQUAL}@anchor{3de3}
@deffn {Data} token.CIRCUMFLEXEQUAL

Token value for @code{"^="}.
@end deffn

@geindex LEFTSHIFTEQUAL (in module token)
@anchor{library/token token LEFTSHIFTEQUAL}@anchor{3de4}
@deffn {Data} token.LEFTSHIFTEQUAL

Token value for @code{"<<="}.
@end deffn

@geindex RIGHTSHIFTEQUAL (in module token)
@anchor{library/token token RIGHTSHIFTEQUAL}@anchor{3de5}
@deffn {Data} token.RIGHTSHIFTEQUAL

Token value for @code{">>="}.
@end deffn

@geindex DOUBLESTAREQUAL (in module token)
@anchor{library/token token DOUBLESTAREQUAL}@anchor{3de6}
@deffn {Data} token.DOUBLESTAREQUAL

Token value for @code{"**="}.
@end deffn

@geindex DOUBLESLASH (in module token)
@anchor{library/token token DOUBLESLASH}@anchor{3de7}
@deffn {Data} token.DOUBLESLASH

Token value for @code{"//"}.
@end deffn

@geindex DOUBLESLASHEQUAL (in module token)
@anchor{library/token token DOUBLESLASHEQUAL}@anchor{3de8}
@deffn {Data} token.DOUBLESLASHEQUAL

Token value for @code{"//="}.
@end deffn

@geindex AT (in module token)
@anchor{library/token token AT}@anchor{3de9}
@deffn {Data} token.AT

Token value for @code{"@@"}.
@end deffn

@geindex ATEQUAL (in module token)
@anchor{library/token token ATEQUAL}@anchor{3dea}
@deffn {Data} token.ATEQUAL

Token value for @code{"@@="}.
@end deffn

@geindex RARROW (in module token)
@anchor{library/token token RARROW}@anchor{3deb}
@deffn {Data} token.RARROW

Token value for @code{"->"}.
@end deffn

@geindex ELLIPSIS (in module token)
@anchor{library/token token ELLIPSIS}@anchor{3dec}
@deffn {Data} token.ELLIPSIS

Token value for @code{"..."}.
@end deffn

@geindex COLONEQUAL (in module token)
@anchor{library/token token COLONEQUAL}@anchor{3ded}
@deffn {Data} token.COLONEQUAL

Token value for @code{":="}.
@end deffn

@geindex OP (in module token)
@anchor{library/token token OP}@anchor{3dee}
@deffn {Data} token.OP
@end deffn

@geindex AWAIT (in module token)
@anchor{library/token token AWAIT}@anchor{3def}
@deffn {Data} token.AWAIT
@end deffn

@geindex ASYNC (in module token)
@anchor{library/token token ASYNC}@anchor{3df0}
@deffn {Data} token.ASYNC
@end deffn

@geindex TYPE_IGNORE (in module token)
@anchor{library/token token TYPE_IGNORE}@anchor{3df1}
@deffn {Data} token.TYPE_IGNORE
@end deffn

@geindex TYPE_COMMENT (in module token)
@anchor{library/token token TYPE_COMMENT}@anchor{3df2}
@deffn {Data} token.TYPE_COMMENT
@end deffn

@geindex SOFT_KEYWORD (in module token)
@anchor{library/token token SOFT_KEYWORD}@anchor{3df3}
@deffn {Data} token.SOFT_KEYWORD
@end deffn

@geindex ERRORTOKEN (in module token)
@anchor{library/token token ERRORTOKEN}@anchor{3df4}
@deffn {Data} token.ERRORTOKEN
@end deffn

@geindex N_TOKENS (in module token)
@anchor{library/token token N_TOKENS}@anchor{3df5}
@deffn {Data} token.N_TOKENS
@end deffn

@geindex NT_OFFSET (in module token)
@anchor{library/token token NT_OFFSET}@anchor{3df6}
@deffn {Data} token.NT_OFFSET
@end deffn

The following token type values aren’t used by the C tokenizer but are needed for
the @ref{118,,tokenize} module.

@geindex COMMENT (in module token)
@anchor{library/token token COMMENT}@anchor{3df7}
@deffn {Data} token.COMMENT

Token value used to indicate a comment.
@end deffn

@geindex NL (in module token)
@anchor{library/token token NL}@anchor{3df8}
@deffn {Data} token.NL

Token value used to indicate a non-terminating newline.  The
@ref{3dbc,,NEWLINE} token indicates the end of a logical line of Python code;
@code{NL} tokens are generated when a logical line of code is continued over
multiple physical lines.
@end deffn

@geindex ENCODING (in module token)
@anchor{library/token token ENCODING}@anchor{3df9}
@deffn {Data} token.ENCODING

Token value that indicates the encoding used to decode the source bytes
into text. The first token returned by @ref{e67,,tokenize.tokenize()} will
always be an @code{ENCODING} token.
@end deffn


@deffn {Data} token.TYPE_COMMENT

Token value indicating that a type comment was recognized.  Such
tokens are only produced when @ref{4d3,,ast.parse()} is invoked with
@code{type_comments=True}.
@end deffn


Changed in version 3.5: Added @ref{3def,,AWAIT} and @ref{3df0,,ASYNC} tokens.



Changed in version 3.7: Added @ref{3df7,,COMMENT}, @ref{3df8,,NL} and @ref{3df9,,ENCODING} tokens.



Changed in version 3.7: Removed @ref{3def,,AWAIT} and @ref{3df0,,ASYNC} tokens. “async” and “await” are
now tokenized as @ref{3db9,,NAME} tokens.



Changed in version 3.8: Added @ref{3df2,,TYPE_COMMENT}, @ref{3df1,,TYPE_IGNORE}, @ref{3ded,,COLONEQUAL}.
Added @ref{3def,,AWAIT} and @ref{3df0,,ASYNC} tokens back (they’re needed
to support parsing older Python versions for @ref{4d3,,ast.parse()} with
@code{feature_version} set to 6 or lower).


@node keyword — Testing for Python keywords,tokenize — Tokenizer for Python source,token — Constants used with Python parse trees,Python Language Services
@anchor{library/keyword doc}@anchor{3dfa}@anchor{library/keyword keyword-testing-for-python-keywords}@anchor{3dfb}@anchor{library/keyword module-keyword}@anchor{a2}
@subsection @code{keyword} — Testing for Python keywords


@geindex module; keyword

@strong{Source code:} Lib/keyword.py@footnote{https://github.com/python/cpython/tree/main/Lib/keyword.py}


__________________________________________________________________


This module allows a Python program to determine if a string is a
@ref{1718,,keyword} or @ref{28f,,soft keyword}.

@geindex iskeyword() (in module keyword)
@anchor{library/keyword keyword iskeyword}@anchor{1aa4}
@deffn {Function} keyword.iskeyword (s)

Return @code{True} if @emph{s} is a Python @ref{1718,,keyword}.
@end deffn

@geindex kwlist (in module keyword)
@anchor{library/keyword keyword kwlist}@anchor{3dfc}
@deffn {Data} keyword.kwlist

Sequence containing all the @ref{1718,,keywords} defined for the
interpreter.  If any keywords are defined to only be active when particular
@ref{0,,__future__} statements are in effect, these will be included as well.
@end deffn

@geindex issoftkeyword() (in module keyword)
@anchor{library/keyword keyword issoftkeyword}@anchor{3dfd}
@deffn {Function} keyword.issoftkeyword (s)

Return @code{True} if @emph{s} is a Python @ref{28f,,soft keyword}.


New in version 3.9.

@end deffn

@geindex softkwlist (in module keyword)
@anchor{library/keyword keyword softkwlist}@anchor{3dfe}
@deffn {Data} keyword.softkwlist

Sequence containing all the @ref{28f,,soft keywords} defined for the
interpreter.  If any soft keywords are defined to only be active when particular
@ref{0,,__future__} statements are in effect, these will be included as well.


New in version 3.9.

@end deffn

@node tokenize — Tokenizer for Python source,tabnanny — Detection of ambiguous indentation,keyword — Testing for Python keywords,Python Language Services
@anchor{library/tokenize doc}@anchor{3dff}@anchor{library/tokenize module-tokenize}@anchor{118}@anchor{library/tokenize tokenize-tokenizer-for-python-source}@anchor{3e00}
@subsection @code{tokenize} — Tokenizer for Python source


@geindex module; tokenize

@strong{Source code:} Lib/tokenize.py@footnote{https://github.com/python/cpython/tree/main/Lib/tokenize.py}


__________________________________________________________________


The @ref{118,,tokenize} module provides a lexical scanner for Python source code,
implemented in Python.  The scanner in this module returns comments as tokens
as well, making it useful for implementing “pretty-printers”, including
colorizers for on-screen displays.

To simplify token stream handling, all @ref{177f,,operator} and
@ref{1780,,delimiter} tokens and @ref{1a67,,Ellipsis} are returned using
the generic @ref{3dee,,OP} token type.  The exact
type can be determined by checking the @code{exact_type} property on the
@ref{ce9,,named tuple} returned from @ref{e67,,tokenize.tokenize()}.

@menu
* Tokenizing Input:: 
* Command-Line Usage: Command-Line Usage<3>. 
* Examples: Examples<33>. 

@end menu

@node Tokenizing Input,Command-Line Usage<3>,,tokenize — Tokenizer for Python source
@anchor{library/tokenize tokenizing-input}@anchor{3e01}
@subsubsection Tokenizing Input


The primary entry point is a @ref{bf7,,generator}:

@geindex tokenize() (in module tokenize)
@anchor{library/tokenize tokenize tokenize}@anchor{e67}
@deffn {Function} tokenize.tokenize (readline)

The @ref{e67,,tokenize()} generator requires one argument, @emph{readline}, which
must be a callable object which provides the same interface as the
@ref{1b74,,io.IOBase.readline()} method of file objects.  Each call to the
function should return one line of input as bytes.

The generator produces 5-tuples with these members: the token type; the
token string; a 2-tuple @code{(srow, scol)} of ints specifying the row and
column where the token begins in the source; a 2-tuple @code{(erow, ecol)} of
ints specifying the row and column where the token ends in the source; and
the line on which the token was found. The line passed (the last tuple item)
is the @emph{physical} line.  The 5 tuple is returned as a @ref{ce9,,named tuple}
with the field names:
@code{type string start end line}.

The returned @ref{ce9,,named tuple} has an additional property named
@code{exact_type} that contains the exact operator type for
@ref{3dee,,OP} tokens.  For all other token types @code{exact_type}
equals the named tuple @code{type} field.


Changed in version 3.1: Added support for named tuples.



Changed in version 3.3: Added support for @code{exact_type}.


@ref{e67,,tokenize()} determines the source encoding of the file by looking for a
UTF-8 BOM or encoding cookie, according to 
@geindex Python Enhancement Proposals; PEP 263
PEP 263@footnote{https://www.python.org/dev/peps/pep-0263}.
@end deffn

@geindex generate_tokens() (in module tokenize)
@anchor{library/tokenize tokenize generate_tokens}@anchor{12fa}
@deffn {Function} tokenize.generate_tokens (readline)

Tokenize a source reading unicode strings instead of bytes.

Like @ref{e67,,tokenize()}, the @emph{readline} argument is a callable returning
a single line of input. However, @ref{12fa,,generate_tokens()} expects @emph{readline}
to return a str object rather than bytes.

The result is an iterator yielding named tuples, exactly like
@ref{e67,,tokenize()}. It does not yield an @ref{3df9,,ENCODING} token.
@end deffn

All constants from the @ref{117,,token} module are also exported from
@ref{118,,tokenize}.

Another function is provided to reverse the tokenization process. This is
useful for creating tools that tokenize a script, modify the token stream, and
write back the modified script.

@geindex untokenize() (in module tokenize)
@anchor{library/tokenize tokenize untokenize}@anchor{3e02}
@deffn {Function} tokenize.untokenize (iterable)

Converts tokens back into Python source code.  The @emph{iterable} must return
sequences with at least two elements, the token type and the token string.
Any additional sequence elements are ignored.

The reconstructed script is returned as a single string.  The result is
guaranteed to tokenize back to match the input so that the conversion is
lossless and round-trips are assured.  The guarantee applies only to the
token type and token string as the spacing between tokens (column
positions) may change.

It returns bytes, encoded using the @ref{3df9,,ENCODING} token, which
is the first token sequence output by @ref{e67,,tokenize()}. If there is no
encoding token in the input, it returns a str instead.
@end deffn

@ref{e67,,tokenize()} needs to detect the encoding of source files it tokenizes. The
function it uses to do this is available:

@geindex detect_encoding() (in module tokenize)
@anchor{library/tokenize tokenize detect_encoding}@anchor{2114}
@deffn {Function} tokenize.detect_encoding (readline)

The @ref{2114,,detect_encoding()} function is used to detect the encoding that
should be used to decode a Python source file. It requires one argument,
readline, in the same way as the @ref{e67,,tokenize()} generator.

It will call readline a maximum of twice, and return the encoding used
(as a string) and a list of any lines (not decoded from bytes) it has read
in.

It detects the encoding from the presence of a UTF-8 BOM or an encoding
cookie as specified in 
@geindex Python Enhancement Proposals; PEP 263
PEP 263@footnote{https://www.python.org/dev/peps/pep-0263}. If both a BOM and a cookie are present,
but disagree, a @ref{21a,,SyntaxError} will be raised. Note that if the BOM is found,
@code{'utf-8-sig'} will be returned as an encoding.

If no encoding is specified, then the default of @code{'utf-8'} will be
returned.

Use @ref{2113,,open()} to open Python source files: it uses
@ref{2114,,detect_encoding()} to detect the file encoding.
@end deffn

@geindex open() (in module tokenize)
@anchor{library/tokenize tokenize open}@anchor{2113}
@deffn {Function} tokenize.open (filename)

Open a file in read only mode using the encoding detected by
@ref{2114,,detect_encoding()}.


New in version 3.2.

@end deffn

@geindex TokenError
@anchor{library/tokenize tokenize TokenError}@anchor{3e03}
@deffn {Exception} tokenize.TokenError

Raised when either a docstring or expression that may be split over several
lines is not completed anywhere in the file, for example:

@example
"""Beginning of
docstring
@end example

or:

@example
[1,
 2,
 3
@end example
@end deffn

Note that unclosed single-quoted strings do not cause an error to be
raised. They are tokenized as @ref{3df4,,ERRORTOKEN}, followed by the
tokenization of their contents.

@node Command-Line Usage<3>,Examples<33>,Tokenizing Input,tokenize — Tokenizer for Python source
@anchor{library/tokenize command-line-usage}@anchor{3e04}@anchor{library/tokenize tokenize-cli}@anchor{3e05}
@subsubsection Command-Line Usage



New in version 3.3.


The @ref{118,,tokenize} module can be executed as a script from the command line.
It is as simple as:

@example
python -m tokenize [-e] [filename.py]
@end example

The following options are accepted:

@geindex tokenize command line option; -h
@geindex tokenize command line option; --help
@anchor{library/tokenize cmdoption-tokenize-h}@anchor{3e06}@anchor{library/tokenize cmdoption-tokenize-help}@anchor{3e07}
@deffn {Program Option} @w{-}h, @w{-}@w{-}help

show this help message and exit
@end deffn

@geindex tokenize command line option; -e
@geindex tokenize command line option; --exact
@anchor{library/tokenize cmdoption-tokenize-e}@anchor{3e08}@anchor{library/tokenize cmdoption-tokenize-exact}@anchor{3e09}
@deffn {Program Option} @w{-}e, @w{-}@w{-}exact

display token names using the exact type
@end deffn

If @code{filename.py} is specified its contents are tokenized to stdout.
Otherwise, tokenization is performed on stdin.

@node Examples<33>,,Command-Line Usage<3>,tokenize — Tokenizer for Python source
@anchor{library/tokenize examples}@anchor{3e0a}
@subsubsection Examples


Example of a script rewriter that transforms float literals into Decimal
objects:

@example
from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP
from io import BytesIO

def decistmt(s):
    """Substitute Decimals for floats in a string of statements.

    >>> from decimal import Decimal
    >>> s = 'print(+21.3e-5*-.1234/81.7)'
    >>> decistmt(s)
    "print (+Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7'))"

    The format of the exponent is inherited from the platform C library.
    Known cases are "e-007" (Windows) and "e-07" (not Windows).  Since
    we're only showing 12 digits, and the 13th isn't close to 5, the
    rest of the output should be platform-independent.

    >>> exec(s)  #doctest: +ELLIPSIS
    -3.21716034272e-0...7

    Output from calculations with Decimal should be identical across all
    platforms.

    >>> exec(decistmt(s))
    -3.217160342717258261933904529E-7
    """
    result = []
    g = tokenize(BytesIO(s.encode('utf-8')).readline)  # tokenize the string
    for toknum, tokval, _, _, _ in g:
        if toknum == NUMBER and '.' in tokval:  # replace NUMBER tokens
            result.extend([
                (NAME, 'Decimal'),
                (OP, '('),
                (STRING, repr(tokval)),
                (OP, ')')
            ])
        else:
            result.append((toknum, tokval))
    return untokenize(result).decode('utf-8')
@end example

Example of tokenizing from the command line.  The script:

@example
def say_hello():
    print("Hello, World!")

say_hello()
@end example

will be tokenized to the following output where the first column is the range
of the line/column coordinates where the token is found, the second column is
the name of the token, and the final column is the value of the token (if any)

@example
$ python -m tokenize hello.py
0,0-0,0:            ENCODING       'utf-8'
1,0-1,3:            NAME           'def'
1,4-1,13:           NAME           'say_hello'
1,13-1,14:          OP             '('
1,14-1,15:          OP             ')'
1,15-1,16:          OP             ':'
1,16-1,17:          NEWLINE        '\n'
2,0-2,4:            INDENT         '    '
2,4-2,9:            NAME           'print'
2,9-2,10:           OP             '('
2,10-2,25:          STRING         '"Hello, World!"'
2,25-2,26:          OP             ')'
2,26-2,27:          NEWLINE        '\n'
3,0-3,1:            NL             '\n'
4,0-4,0:            DEDENT         '@w{'}
4,0-4,9:            NAME           'say_hello'
4,9-4,10:           OP             '('
4,10-4,11:          OP             ')'
4,11-4,12:          NEWLINE        '\n'
5,0-5,0:            ENDMARKER      '@w{'}
@end example

The exact token type names can be displayed using the @ref{3e08,,-e} option:

@example
$ python -m tokenize -e hello.py
0,0-0,0:            ENCODING       'utf-8'
1,0-1,3:            NAME           'def'
1,4-1,13:           NAME           'say_hello'
1,13-1,14:          LPAR           '('
1,14-1,15:          RPAR           ')'
1,15-1,16:          COLON          ':'
1,16-1,17:          NEWLINE        '\n'
2,0-2,4:            INDENT         '    '
2,4-2,9:            NAME           'print'
2,9-2,10:           LPAR           '('
2,10-2,25:          STRING         '"Hello, World!"'
2,25-2,26:          RPAR           ')'
2,26-2,27:          NEWLINE        '\n'
3,0-3,1:            NL             '\n'
4,0-4,0:            DEDENT         '@w{'}
4,0-4,9:            NAME           'say_hello'
4,9-4,10:           LPAR           '('
4,10-4,11:          RPAR           ')'
4,11-4,12:          NEWLINE        '\n'
5,0-5,0:            ENDMARKER      '@w{'}
@end example

Example of tokenizing a file programmatically, reading unicode
strings instead of bytes with @ref{12fa,,generate_tokens()}:

@example
import tokenize

with tokenize.open('hello.py') as f:
    tokens = tokenize.generate_tokens(f.readline)
    for token in tokens:
        print(token)
@end example

Or reading bytes directly with @ref{e67,,tokenize()}:

@example
import tokenize

with open('hello.py', 'rb') as f:
    tokens = tokenize.tokenize(f.readline)
    for token in tokens:
        print(token)
@end example

@node tabnanny — Detection of ambiguous indentation,pyclbr — Python module browser support,tokenize — Tokenizer for Python source,Python Language Services
@anchor{library/tabnanny doc}@anchor{3e0b}@anchor{library/tabnanny module-tabnanny}@anchor{fa}@anchor{library/tabnanny tabnanny-detection-of-ambiguous-indentation}@anchor{3e0c}
@subsection @code{tabnanny} — Detection of ambiguous indentation


@geindex module; tabnanny

@c rudimentary documentation based on module comments

@strong{Source code:} Lib/tabnanny.py@footnote{https://github.com/python/cpython/tree/main/Lib/tabnanny.py}


__________________________________________________________________


For the time being this module is intended to be called as a script. However it
is possible to import it into an IDE and use the function @ref{3e0d,,check()}
described below.

@cartouche
@quotation Note 
The API provided by this module is likely to change in future releases; such
changes may not be backward compatible.
@end quotation
@end cartouche

@geindex check() (in module tabnanny)
@anchor{library/tabnanny tabnanny check}@anchor{3e0d}
@deffn {Function} tabnanny.check (file_or_dir)

If @emph{file_or_dir} is a directory and not a symbolic link, then recursively
descend the directory tree named by @emph{file_or_dir}, checking all @code{.py}
files along the way.  If @emph{file_or_dir} is an ordinary Python source file, it
is checked for whitespace related problems.  The diagnostic messages are
written to standard output using the @ref{af6,,print()} function.
@end deffn

@geindex verbose (in module tabnanny)
@anchor{library/tabnanny tabnanny verbose}@anchor{3e0e}
@deffn {Data} tabnanny.verbose

Flag indicating whether to print verbose messages. This is incremented by the
@code{-v} option if called as a script.
@end deffn

@geindex filename_only (in module tabnanny)
@anchor{library/tabnanny tabnanny filename_only}@anchor{3e0f}
@deffn {Data} tabnanny.filename_only

Flag indicating whether to print only the filenames of files containing
whitespace related problems.  This is set to true by the @code{-q} option if called
as a script.
@end deffn

@geindex NannyNag
@anchor{library/tabnanny tabnanny NannyNag}@anchor{3e10}
@deffn {Exception} tabnanny.NannyNag

Raised by @ref{3e11,,process_tokens()} if detecting an ambiguous indent. Captured and
handled in @ref{3e0d,,check()}.
@end deffn

@geindex process_tokens() (in module tabnanny)
@anchor{library/tabnanny tabnanny process_tokens}@anchor{3e11}
@deffn {Function} tabnanny.process_tokens (tokens)

This function is used by @ref{3e0d,,check()} to process tokens generated by the
@ref{118,,tokenize} module.
@end deffn

@c XXX document errprint, format_witnesses, Whitespace, check_equal, indents,
@c reset_globals


@subsubheading See also



@table @asis

@item Module @ref{118,,tokenize}

Lexical scanner for Python source code.
@end table


@node pyclbr — Python module browser support,py_compile — Compile Python source files,tabnanny — Detection of ambiguous indentation,Python Language Services
@anchor{library/pyclbr doc}@anchor{3e12}@anchor{library/pyclbr module-pyclbr}@anchor{d3}@anchor{library/pyclbr pyclbr-python-module-browser-support}@anchor{3e13}
@subsection @code{pyclbr} — Python module browser support


@geindex module; pyclbr

@strong{Source code:} Lib/pyclbr.py@footnote{https://github.com/python/cpython/tree/main/Lib/pyclbr.py}


__________________________________________________________________


The @ref{d3,,pyclbr} module provides limited information about the
functions, classes, and methods defined in a Python-coded module.  The
information is sufficient to implement a module browser.  The
information is extracted from the Python source code rather than by
importing the module, so this module is safe to use with untrusted code.
This restriction makes it impossible to use this module with modules not
implemented in Python, including all standard and optional extension
modules.

@geindex readmodule() (in module pyclbr)
@anchor{library/pyclbr pyclbr readmodule}@anchor{142a}
@deffn {Function} pyclbr.readmodule (module, path=None)

Return a dictionary mapping module-level class names to class
descriptors.  If possible, descriptors for imported base classes are
included.  Parameter @emph{module} is a string with the name of the module
to read; it may be the name of a module within a package.  If given,
@emph{path} is a sequence of directory paths prepended to @code{sys.path},
which is used to locate the module source code.

This function is the original interface and is only kept for back
compatibility.  It returns a filtered version of the following.
@end deffn

@geindex readmodule_ex() (in module pyclbr)
@anchor{library/pyclbr pyclbr readmodule_ex}@anchor{142b}
@deffn {Function} pyclbr.readmodule_ex (module, path=None)

Return a dictionary-based tree containing a function or class
descriptors for each function and class defined in the module with a
@code{def} or @code{class} statement.  The returned dictionary maps
module-level function and class names to their descriptors.  Nested
objects are entered into the children dictionary of their parent.  As
with readmodule, @emph{module} names the module to be read and @emph{path} is
prepended to sys.path.  If the module being read is a package, the
returned dictionary has a key @code{'__path__'} whose value is a list
containing the package search path.
@end deffn


New in version 3.7: Descriptors for nested definitions.  They are accessed through the
new children attribute.  Each has a new parent attribute.


The descriptors returned by these functions are instances of
Function and Class classes.  Users are not expected to create instances
of these classes.

@menu
* Function Objects:: 
* Class Objects: Class Objects<2>. 

@end menu

@node Function Objects,Class Objects<2>,,pyclbr — Python module browser support
@anchor{library/pyclbr function-objects}@anchor{3e14}@anchor{library/pyclbr pyclbr-function-objects}@anchor{3e15}
@subsubsection Function Objects


Class @code{Function} instances describe functions defined by def
statements.  They have the following attributes:

@geindex file (pyclbr.Function attribute)
@anchor{library/pyclbr pyclbr Function file}@anchor{3e16}
@deffn {Attribute} Function.file

Name of the file in which the function is defined.
@end deffn

@geindex module (pyclbr.Function attribute)
@anchor{library/pyclbr pyclbr Function module}@anchor{3e17}
@deffn {Attribute} Function.module

The name of the module defining the function described.
@end deffn

@geindex name (pyclbr.Function attribute)
@anchor{library/pyclbr pyclbr Function name}@anchor{3e18}
@deffn {Attribute} Function.name

The name of the function.
@end deffn

@geindex lineno (pyclbr.Function attribute)
@anchor{library/pyclbr pyclbr Function lineno}@anchor{3e19}
@deffn {Attribute} Function.lineno

The line number in the file where the definition starts.
@end deffn

@geindex parent (pyclbr.Function attribute)
@anchor{library/pyclbr pyclbr Function parent}@anchor{3e1a}
@deffn {Attribute} Function.parent

For top-level functions, None.  For nested functions, the parent.


New in version 3.7.

@end deffn

@geindex children (pyclbr.Function attribute)
@anchor{library/pyclbr pyclbr Function children}@anchor{3e1b}
@deffn {Attribute} Function.children

A dictionary mapping names to descriptors for nested functions and
classes.


New in version 3.7.

@end deffn

@geindex is_async (pyclbr.Function attribute)
@anchor{library/pyclbr pyclbr Function is_async}@anchor{3e1c}
@deffn {Attribute} Function.is_async

@code{True} for functions that are defined with the @code{async} prefix, @code{False} otherwise.


New in version 3.10.

@end deffn

@node Class Objects<2>,,Function Objects,pyclbr — Python module browser support
@anchor{library/pyclbr class-objects}@anchor{3e1d}@anchor{library/pyclbr pyclbr-class-objects}@anchor{3e1e}
@subsubsection Class Objects


Class @code{Class} instances describe classes defined by class
statements.  They have the same attributes as Functions and two more.

@geindex file (pyclbr.Class attribute)
@anchor{library/pyclbr pyclbr Class file}@anchor{3e1f}
@deffn {Attribute} Class.file

Name of the file in which the class is defined.
@end deffn

@geindex module (pyclbr.Class attribute)
@anchor{library/pyclbr pyclbr Class module}@anchor{3e20}
@deffn {Attribute} Class.module

The name of the module defining the class described.
@end deffn

@geindex name (pyclbr.Class attribute)
@anchor{library/pyclbr pyclbr Class name}@anchor{3e21}
@deffn {Attribute} Class.name

The name of the class.
@end deffn

@geindex lineno (pyclbr.Class attribute)
@anchor{library/pyclbr pyclbr Class lineno}@anchor{3e22}
@deffn {Attribute} Class.lineno

The line number in the file where the definition starts.
@end deffn

@geindex parent (pyclbr.Class attribute)
@anchor{library/pyclbr pyclbr Class parent}@anchor{3e23}
@deffn {Attribute} Class.parent

For top-level classes, None.  For nested classes, the parent.


New in version 3.7.

@end deffn

@geindex children (pyclbr.Class attribute)
@anchor{library/pyclbr pyclbr Class children}@anchor{3e24}
@deffn {Attribute} Class.children

A dictionary mapping names to descriptors for nested functions and
classes.


New in version 3.7.

@end deffn

@geindex super (pyclbr.Class attribute)
@anchor{library/pyclbr pyclbr Class super}@anchor{3e25}
@deffn {Attribute} Class.super

A list of @code{Class} objects which describe the immediate base
classes of the class being described.  Classes which are named as
superclasses but which are not discoverable by @ref{142b,,readmodule_ex()}
are listed as a string with the class name instead of as
@code{Class} objects.
@end deffn

@geindex methods (pyclbr.Class attribute)
@anchor{library/pyclbr pyclbr Class methods}@anchor{3e26}
@deffn {Attribute} Class.methods

A dictionary mapping method names to line numbers.  This can be
derived from the newer children dictionary, but remains for
back-compatibility.
@end deffn

@node py_compile — Compile Python source files,compileall — Byte-compile Python libraries,pyclbr — Python module browser support,Python Language Services
@anchor{library/py_compile doc}@anchor{3e27}@anchor{library/py_compile module-py_compile}@anchor{d2}@anchor{library/py_compile py-compile-compile-python-source-files}@anchor{3e28}
@subsection @code{py_compile} — Compile Python source files


@geindex module; py_compile

@c documentation based on module docstrings

@strong{Source code:} Lib/py_compile.py@footnote{https://github.com/python/cpython/tree/main/Lib/py_compile.py}

@geindex file; byte-code


__________________________________________________________________


The @ref{d2,,py_compile} module provides a function to generate a byte-code file
from a source file, and another function used when the module source file is
invoked as a script.

Though not often needed, this function can be useful when installing modules for
shared use, especially if some of the users may not have permission to write the
byte-code cache files in the directory containing the source code.

@geindex PyCompileError
@anchor{library/py_compile py_compile PyCompileError}@anchor{3e29}
@deffn {Exception} py_compile.PyCompileError

Exception raised when an error occurs while attempting to compile the file.
@end deffn

@geindex compile() (in module py_compile)
@anchor{library/py_compile py_compile compile}@anchor{534}
@deffn {Function} py_compile.compile (file, cfile=None, dfile=None, doraise=False, optimize=- 1, invalidation_mode=PycInvalidationMode.TIMESTAMP, quiet=0)

Compile a source file to byte-code and write out the byte-code cache file.
The source code is loaded from the file named @emph{file}.  The byte-code is
written to @emph{cfile}, which defaults to the 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147}/
@geindex Python Enhancement Proposals; PEP 488
PEP 488@footnote{https://www.python.org/dev/peps/pep-0488} path, ending
in @code{.pyc}.
For example, if @emph{file} is @code{/foo/bar/baz.py} @emph{cfile} will default to
@code{/foo/bar/__pycache__/baz.cpython-32.pyc} for Python 3.2.  If @emph{dfile} is
specified, it is used as the name of the source file in error messages
instead of @emph{file}.  If @emph{doraise} is true, a @ref{3e29,,PyCompileError} is raised
when an error is encountered while compiling @emph{file}. If @emph{doraise} is false
(the default), an error string is written to @code{sys.stderr}, but no exception
is raised.  This function returns the path to byte-compiled file, i.e.
whatever @emph{cfile} value was used.

The @emph{doraise} and @emph{quiet} arguments determine how errors are handled while
compiling file. If @emph{quiet} is 0 or 1, and @emph{doraise} is false, the default
behaviour is enabled: an error string is written to @code{sys.stderr}, and the
function returns @code{None} instead of a path. If @emph{doraise} is true,
a @ref{3e29,,PyCompileError} is raised instead. However if @emph{quiet} is 2,
no message is written, and @emph{doraise} has no effect.

If the path that @emph{cfile} becomes (either explicitly specified or computed)
is a symlink or non-regular file, @ref{bb1,,FileExistsError} will be raised.
This is to act as a warning that import will turn those paths into regular
files if it is allowed to write byte-compiled files to those paths. This is
a side-effect of import using file renaming to place the final byte-compiled
file into place to prevent concurrent file writing issues.

@emph{optimize} controls the optimization level and is passed to the built-in
@ref{4e2,,compile()} function.  The default of @code{-1} selects the optimization
level of the current interpreter.

@emph{invalidation_mode} should be a member of the @ref{3e2a,,PycInvalidationMode}
enum and controls how the generated bytecode cache is invalidated at
runtime.  The default is @ref{3e2b,,PycInvalidationMode.CHECKED_HASH} if
the 
@geindex SOURCE_DATE_EPOCH
@geindex environment variable; SOURCE_DATE_EPOCH
@code{SOURCE_DATE_EPOCH} environment variable is set, otherwise
the default is @ref{3e2c,,PycInvalidationMode.TIMESTAMP}.


Changed in version 3.2: Changed default value of @emph{cfile} to be 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147}-compliant.  Previous
default was @emph{file} + @code{'c'} (@code{'o'} if optimization was enabled).
Also added the @emph{optimize} parameter.



Changed in version 3.4: Changed code to use @ref{96,,importlib} for the byte-code cache file writing.
This means file creation/writing semantics now match what @ref{96,,importlib}
does, e.g. permissions, write-and-move semantics, etc. Also added the
caveat that @ref{bb1,,FileExistsError} is raised if @emph{cfile} is a symlink or
non-regular file.



Changed in version 3.7: The @emph{invalidation_mode} parameter was added as specified in 
@geindex Python Enhancement Proposals; PEP 552
PEP 552@footnote{https://www.python.org/dev/peps/pep-0552}.
If the 
@geindex SOURCE_DATE_EPOCH
@geindex environment variable; SOURCE_DATE_EPOCH
@code{SOURCE_DATE_EPOCH} environment variable is set,
@emph{invalidation_mode} will be forced to
@ref{3e2b,,PycInvalidationMode.CHECKED_HASH}.



Changed in version 3.7.2: The 
@geindex SOURCE_DATE_EPOCH
@geindex environment variable; SOURCE_DATE_EPOCH
@code{SOURCE_DATE_EPOCH} environment variable no longer
overrides the value of the @emph{invalidation_mode} argument, and determines
its default value instead.



Changed in version 3.8: The @emph{quiet} parameter was added.

@end deffn

@geindex PycInvalidationMode (class in py_compile)
@anchor{library/py_compile py_compile PycInvalidationMode}@anchor{3e2a}
@deffn {Class} py_compile.PycInvalidationMode

A enumeration of possible methods the interpreter can use to determine
whether a bytecode file is up to date with a source file. The @code{.pyc} file
indicates the desired invalidation mode in its header. See
@ref{607,,Cached bytecode invalidation} for more information on how Python invalidates
@code{.pyc} files at runtime.


New in version 3.7.


@geindex TIMESTAMP (py_compile.PycInvalidationMode attribute)
@anchor{library/py_compile py_compile PycInvalidationMode TIMESTAMP}@anchor{3e2c}
@deffn {Attribute} TIMESTAMP

The @code{.pyc} file includes the timestamp and size of the source file,
which Python will compare against the metadata of the source file at
runtime to determine if the @code{.pyc} file needs to be regenerated.
@end deffn

@geindex CHECKED_HASH (py_compile.PycInvalidationMode attribute)
@anchor{library/py_compile py_compile PycInvalidationMode CHECKED_HASH}@anchor{3e2b}
@deffn {Attribute} CHECKED_HASH

The @code{.pyc} file includes a hash of the source file content, which Python
will compare against the source at runtime to determine if the @code{.pyc}
file needs to be regenerated.
@end deffn

@geindex UNCHECKED_HASH (py_compile.PycInvalidationMode attribute)
@anchor{library/py_compile py_compile PycInvalidationMode UNCHECKED_HASH}@anchor{3e2d}
@deffn {Attribute} UNCHECKED_HASH

Like @ref{3e2b,,CHECKED_HASH}, the @code{.pyc} file includes a hash of the source
file content. However, Python will at runtime assume the @code{.pyc} file is
up to date and not validate the @code{.pyc} against the source file at all.

This option is useful when the @code{.pycs} are kept up to date by some
system external to Python like a build system.
@end deffn
@end deffn

@menu
* Command-Line Interface: Command-Line Interface<6>. 

@end menu

@node Command-Line Interface<6>,,,py_compile — Compile Python source files
@anchor{library/py_compile command-line-interface}@anchor{3e2e}
@subsubsection Command-Line Interface


This module can be invoked as a script to compile several source
files.  The files named in @emph{filenames} are compiled and the resulting
bytecode is cached in the normal manner.  This program does not search
a directory structure to locate source files; it only compiles files
named explicitly. The exit status is nonzero if one of the files could
not be compiled.

@geindex python--m-py_compile command line option; <file> ... <fileN>
@geindex python--m-py_compile command line option; -
@anchor{library/py_compile cmdoption-python-m-py_compile-arg-file}@anchor{3e2f}@anchor{library/py_compile cmdoption-python-m-py-compile-arg-file}@anchor{3e30}
@deffn {Program Option} <file> ... <fileN>
@anchor{library/py_compile cmdoption-python-m-py_compile}@anchor{3e31}@anchor{library/py_compile cmdoption-python-m-py-compile}@anchor{3e32}
@deffnx {Program Option} @w{-}

Positional arguments are files to compile.  If @code{-} is the only
parameter, the list of files is taken from standard input.
@end deffn

@geindex python--m-py_compile command line option; -q
@geindex python--m-py_compile command line option; --quiet
@anchor{library/py_compile cmdoption-python-m-py_compile-q}@anchor{3e33}@anchor{library/py_compile cmdoption-python-m-py-compile-q}@anchor{3e34}@anchor{library/py_compile cmdoption-python-m-py_compile-quiet}@anchor{3e35}@anchor{library/py_compile cmdoption-python-m-py-compile-quiet}@anchor{3e36}
@deffn {Program Option} @w{-}q, @w{-}@w{-}quiet

Suppress errors output.
@end deffn


Changed in version 3.2: Added support for @code{-}.



Changed in version 3.10: Added support for @ref{3e33,,-q}.



@subsubheading See also



@table @asis

@item Module @ref{1f,,compileall}

Utilities to compile all Python source files in a directory tree.
@end table


@node compileall — Byte-compile Python libraries,dis — Disassembler for Python bytecode,py_compile — Compile Python source files,Python Language Services
@anchor{library/compileall doc}@anchor{3e37}@anchor{library/compileall compileall-byte-compile-python-libraries}@anchor{3e38}@anchor{library/compileall module-compileall}@anchor{1f}
@subsection @code{compileall} — Byte-compile Python libraries


@geindex module; compileall

@strong{Source code:} Lib/compileall.py@footnote{https://github.com/python/cpython/tree/main/Lib/compileall.py}


__________________________________________________________________


This module provides some utility functions to support installing Python
libraries.  These functions compile Python source files in a directory tree.
This module can be used to create the cached byte-code files at library
installation time, which makes them available for use even by users who don’t
have write permission to the library directories.

@menu
* Command-line use:: 
* Public functions:: 

@end menu

@node Command-line use,Public functions,,compileall — Byte-compile Python libraries
@anchor{library/compileall command-line-use}@anchor{3e39}
@subsubsection Command-line use


This module can work as a script (using @code{python -m compileall}) to
compile Python sources.

@geindex compileall command line option; directory ...
@geindex compileall command line option; file ...
@anchor{library/compileall cmdoption-compileall-arg-directory}@anchor{3e3a}
@deffn {Program Option} directory ...
@anchor{library/compileall cmdoption-compileall-arg-file}@anchor{3e3b}
@deffnx {Program Option} file ...

Positional arguments are files to compile or directories that contain
source files, traversed recursively.  If no argument is given, behave as if
the command line was @code{-l <directories from sys.path>}.
@end deffn

@geindex compileall command line option; -l
@anchor{library/compileall cmdoption-compileall-l}@anchor{3e3c}
@deffn {Program Option} @w{-}l

Do not recurse into subdirectories, only compile source code files directly
contained in the named or implied directories.
@end deffn

@geindex compileall command line option; -f
@anchor{library/compileall cmdoption-compileall-f}@anchor{3e3d}
@deffn {Program Option} @w{-}f

Force rebuild even if timestamps are up-to-date.
@end deffn

@geindex compileall command line option; -q
@anchor{library/compileall cmdoption-compileall-q}@anchor{3e3e}
@deffn {Program Option} @w{-}q

Do not print the list of files compiled. If passed once, error messages will
still be printed. If passed twice (@code{-qq}), all output is suppressed.
@end deffn

@geindex compileall command line option; -d destdir
@anchor{library/compileall cmdoption-compileall-d}@anchor{3e3f}
@deffn {Program Option} @w{-}d destdir

Directory prepended to the path to each file being compiled.  This will
appear in compilation time tracebacks, and is also compiled in to the
byte-code file, where it will be used in tracebacks and other messages in
cases where the source file does not exist at the time the byte-code file is
executed.
@end deffn

@geindex compileall command line option; -s strip_prefix
@anchor{library/compileall cmdoption-compileall-s}@anchor{3e40}
@deffn {Program Option} @w{-}s strip_prefix
@end deffn

@geindex compileall command line option; -p prepend_prefix
@anchor{library/compileall cmdoption-compileall-p}@anchor{3e41}
@deffn {Program Option} @w{-}p prepend_prefix

Remove (@code{-s}) or append (@code{-p}) the given prefix of paths
recorded in the @code{.pyc} files.
Cannot be combined with @code{-d}.
@end deffn

@geindex compileall command line option; -x regex
@anchor{library/compileall cmdoption-compileall-x}@anchor{3e42}
@deffn {Program Option} @w{-}x regex

regex is used to search the full path to each file considered for
compilation, and if the regex produces a match, the file is skipped.
@end deffn

@geindex compileall command line option; -i list
@anchor{library/compileall cmdoption-compileall-i}@anchor{3e43}
@deffn {Program Option} @w{-}i list

Read the file @code{list} and add each line that it contains to the list of
files and directories to compile.  If @code{list} is @code{-}, read lines from
@code{stdin}.
@end deffn

@geindex compileall command line option; -b
@anchor{library/compileall cmdoption-compileall-b}@anchor{3e44}
@deffn {Program Option} @w{-}b

Write the byte-code files to their legacy locations and names, which may
overwrite byte-code files created by another version of Python.  The default
is to write files to their 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} locations and names, which allows
byte-code files from multiple versions of Python to coexist.
@end deffn

@geindex compileall command line option; -r
@anchor{library/compileall cmdoption-compileall-r}@anchor{3e45}
@deffn {Program Option} @w{-}r

Control the maximum recursion level for subdirectories.
If this is given, then @code{-l} option will not be taken into account.
@code{python -m compileall <directory> -r 0} is equivalent to
@code{python -m compileall <directory> -l}.
@end deffn

@geindex compileall command line option; -j N
@anchor{library/compileall cmdoption-compileall-j}@anchor{3e46}
@deffn {Program Option} @w{-}j N

Use @emph{N} workers to compile the files within the given directory.
If @code{0} is used, then the result of @ref{298,,os.cpu_count()}
will be used.
@end deffn

@geindex compileall command line option; --invalidation-mode [timestamp|checked-hash|unchecked-hash]
@anchor{library/compileall cmdoption-compileall-invalidation-mode}@anchor{3e47}
@deffn {Program Option} @w{-}@w{-}invalidation@w{-}mode [timestamp|checked@w{-}hash|unchecked@w{-}hash]

Control how the generated byte-code files are invalidated at runtime.
The @code{timestamp} value, means that @code{.pyc} files with the source timestamp
and size embedded will be generated. The @code{checked-hash} and
@code{unchecked-hash} values cause hash-based pycs to be generated. Hash-based
pycs embed a hash of the source file contents rather than a timestamp. See
@ref{607,,Cached bytecode invalidation} for more information on how Python validates
bytecode cache files at runtime.
The default is @code{timestamp} if the 
@geindex SOURCE_DATE_EPOCH
@geindex environment variable; SOURCE_DATE_EPOCH
@code{SOURCE_DATE_EPOCH} environment
variable is not set, and @code{checked-hash} if the @code{SOURCE_DATE_EPOCH}
environment variable is set.
@end deffn

@geindex compileall command line option; -o level
@anchor{library/compileall cmdoption-compileall-o}@anchor{3e48}
@deffn {Program Option} @w{-}o level

Compile with the given optimization level. May be used multiple times
to compile for multiple levels at a time (for example,
@code{compileall -o 1 -o 2}).
@end deffn

@geindex compileall command line option; -e dir
@anchor{library/compileall cmdoption-compileall-e}@anchor{3e49}
@deffn {Program Option} @w{-}e dir

Ignore symlinks pointing outside the given directory.
@end deffn

@geindex compileall command line option; --hardlink-dupes
@anchor{library/compileall cmdoption-compileall-hardlink-dupes}@anchor{3e4a}
@deffn {Program Option} @w{-}@w{-}hardlink@w{-}dupes

If two @code{.pyc} files with different optimization level have
the same content, use hard links to consolidate duplicate files.
@end deffn


Changed in version 3.2: Added the @code{-i}, @code{-b} and @code{-h} options.



Changed in version 3.5: Added the  @code{-j}, @code{-r}, and @code{-qq} options.  @code{-q} option
was changed to a multilevel value.  @code{-b} will always produce a
byte-code file ending in @code{.pyc}, never @code{.pyo}.



Changed in version 3.7: Added the @code{--invalidation-mode} option.



Changed in version 3.9: Added the @code{-s}, @code{-p}, @code{-e} and @code{--hardlink-dupes} options.
Raised the default recursion limit from 10 to
@ref{106a,,sys.getrecursionlimit()}.
Added the possibility to specify the @code{-o} option multiple times.


There is no command-line option to control the optimization level used by the
@ref{4e2,,compile()} function, because the Python interpreter itself already
provides the option: @code{python -O -m compileall}.

Similarly, the @ref{4e2,,compile()} function respects the @ref{4ab,,sys.pycache_prefix}
setting. The generated bytecode cache will only be useful if @ref{4e2,,compile()} is
run with the same @ref{4ab,,sys.pycache_prefix} (if any) that will be used at
runtime.

@node Public functions,,Command-line use,compileall — Byte-compile Python libraries
@anchor{library/compileall public-functions}@anchor{3e4b}
@subsubsection Public functions


@geindex compile_dir() (in module compileall)
@anchor{library/compileall compileall compile_dir}@anchor{645}
@deffn {Function} compileall.compile_dir (dir, maxlevels=sys.getrecursionlimit(), ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=- 1, workers=1, invalidation_mode=None, *, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False)

Recursively descend the directory tree named by @emph{dir}, compiling all @code{.py}
files along the way. Return a true value if all the files compiled successfully,
and a false value otherwise.

The @emph{maxlevels} parameter is used to limit the depth of the recursion; it
defaults to @code{sys.getrecursionlimit()}.

If @emph{ddir} is given, it is prepended to the path to each file being compiled
for use in compilation time tracebacks, and is also compiled in to the
byte-code file, where it will be used in tracebacks and other messages in
cases where the source file does not exist at the time the byte-code file is
executed.

If @emph{force} is true, modules are re-compiled even if the timestamps are up to
date.

If @emph{rx} is given, its @code{search} method is called on the complete path to each
file considered for compilation, and if it returns a true value, the file
is skipped. This can be used to exclude files matching a regular expression,
given as a @ref{b07,,re.Pattern} object.

If @emph{quiet} is @code{False} or @code{0} (the default), the filenames and other
information are printed to standard out. Set to @code{1}, only errors are
printed. Set to @code{2}, all output is suppressed.

If @emph{legacy} is true, byte-code files are written to their legacy locations
and names, which may overwrite byte-code files created by another version of
Python.  The default is to write files to their 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} locations and
names, which allows byte-code files from multiple versions of Python to
coexist.

@emph{optimize} specifies the optimization level for the compiler.  It is passed to
the built-in @ref{4e2,,compile()} function. Accepts also a sequence of optimization
levels which lead to multiple compilations of one @code{.py} file in one call.

The argument @emph{workers} specifies how many workers are used to
compile files in parallel. The default is to not use multiple workers.
If the platform can’t use multiple workers and @emph{workers} argument is given,
then sequential compilation will be used as a fallback.  If @emph{workers}
is 0, the number of cores in the system is used.  If @emph{workers} is
lower than @code{0}, a @ref{17c,,ValueError} will be raised.

@emph{invalidation_mode} should be a member of the
@ref{3e2a,,py_compile.PycInvalidationMode} enum and controls how the generated
pycs are invalidated at runtime.

The @emph{stripdir}, @emph{prependdir} and @emph{limit_sl_dest} arguments correspond to
the @code{-s}, @code{-p} and @code{-e} options described above.
They may be specified as @code{str}, @code{bytes} or @ref{795,,os.PathLike}.

If @emph{hardlink_dupes} is true and two @code{.pyc} files with different optimization
level have the same content, use hard links to consolidate duplicate files.


Changed in version 3.2: Added the @emph{legacy} and @emph{optimize} parameter.



Changed in version 3.5: Added the @emph{workers} parameter.



Changed in version 3.5: @emph{quiet} parameter was changed to a multilevel value.



Changed in version 3.5: The @emph{legacy} parameter only writes out @code{.pyc} files, not @code{.pyo} files
no matter what the value of @emph{optimize} is.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.



Changed in version 3.7: The @emph{invalidation_mode} parameter was added.



Changed in version 3.7.2: The @emph{invalidation_mode} parameter’s default value is updated to None.



Changed in version 3.8: Setting @emph{workers} to 0 now chooses the optimal number of cores.



Changed in version 3.9: Added @emph{stripdir}, @emph{prependdir}, @emph{limit_sl_dest} and @emph{hardlink_dupes} arguments.
Default value of @emph{maxlevels} was changed from @code{10} to @code{sys.getrecursionlimit()}

@end deffn

@geindex compile_file() (in module compileall)
@anchor{library/compileall compileall compile_file}@anchor{940}
@deffn {Function} compileall.compile_file (fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=- 1, invalidation_mode=None, *, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False)

Compile the file with path @emph{fullname}. Return a true value if the file
compiled successfully, and a false value otherwise.

If @emph{ddir} is given, it is prepended to the path to the file being compiled
for use in compilation time tracebacks, and is also compiled in to the
byte-code file, where it will be used in tracebacks and other messages in
cases where the source file does not exist at the time the byte-code file is
executed.

If @emph{rx} is given, its @code{search} method is passed the full path name to the
file being compiled, and if it returns a true value, the file is not
compiled and @code{True} is returned. This can be used to exclude files matching
a regular expression, given as a @ref{b07,,re.Pattern} object.

If @emph{quiet} is @code{False} or @code{0} (the default), the filenames and other
information are printed to standard out. Set to @code{1}, only errors are
printed. Set to @code{2}, all output is suppressed.

If @emph{legacy} is true, byte-code files are written to their legacy locations
and names, which may overwrite byte-code files created by another version of
Python.  The default is to write files to their 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} locations and
names, which allows byte-code files from multiple versions of Python to
coexist.

@emph{optimize} specifies the optimization level for the compiler.  It is passed to
the built-in @ref{4e2,,compile()} function. Accepts also a sequence of optimization
levels which lead to multiple compilations of one @code{.py} file in one call.

@emph{invalidation_mode} should be a member of the
@ref{3e2a,,py_compile.PycInvalidationMode} enum and controls how the generated
pycs are invalidated at runtime.

The @emph{stripdir}, @emph{prependdir} and @emph{limit_sl_dest} arguments correspond to
the @code{-s}, @code{-p} and @code{-e} options described above.
They may be specified as @code{str}, @code{bytes} or @ref{795,,os.PathLike}.

If @emph{hardlink_dupes} is true and two @code{.pyc} files with different optimization
level have the same content, use hard links to consolidate duplicate files.


New in version 3.2.



Changed in version 3.5: @emph{quiet} parameter was changed to a multilevel value.



Changed in version 3.5: The @emph{legacy} parameter only writes out @code{.pyc} files, not @code{.pyo} files
no matter what the value of @emph{optimize} is.



Changed in version 3.7: The @emph{invalidation_mode} parameter was added.



Changed in version 3.7.2: The @emph{invalidation_mode} parameter’s default value is updated to None.



Changed in version 3.9: Added @emph{stripdir}, @emph{prependdir}, @emph{limit_sl_dest} and @emph{hardlink_dupes} arguments.

@end deffn

@geindex compile_path() (in module compileall)
@anchor{library/compileall compileall compile_path}@anchor{941}
@deffn {Function} compileall.compile_path (skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=- 1, invalidation_mode=None)

Byte-compile all the @code{.py} files found along @code{sys.path}. Return a
true value if all the files compiled successfully, and a false value otherwise.

If @emph{skip_curdir} is true (the default), the current directory is not included
in the search.  All other parameters are passed to the @ref{645,,compile_dir()}
function.  Note that unlike the other compile functions, @code{maxlevels}
defaults to @code{0}.


Changed in version 3.2: Added the @emph{legacy} and @emph{optimize} parameter.



Changed in version 3.5: @emph{quiet} parameter was changed to a multilevel value.



Changed in version 3.5: The @emph{legacy} parameter only writes out @code{.pyc} files, not @code{.pyo} files
no matter what the value of @emph{optimize} is.



Changed in version 3.7: The @emph{invalidation_mode} parameter was added.



Changed in version 3.7.2: The @emph{invalidation_mode} parameter’s default value is updated to None.

@end deffn

To force a recompile of all the @code{.py} files in the @code{Lib/}
subdirectory and all its subdirectories:

@example
import compileall

compileall.compile_dir('Lib/', force=True)

# Perform same compilation, excluding files in .svn directories.
import re
compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True)

# pathlib.Path objects can also be used.
import pathlib
compileall.compile_dir(pathlib.Path('Lib/'), force=True)
@end example


@subsubheading See also



@table @asis

@item Module @ref{d2,,py_compile}

Byte-compile a single source file.
@end table


@node dis — Disassembler for Python bytecode,pickletools — Tools for pickle developers,compileall — Byte-compile Python libraries,Python Language Services
@anchor{library/dis doc}@anchor{3e4c}@anchor{library/dis dis-disassembler-for-python-bytecode}@anchor{3e4d}@anchor{library/dis module-dis}@anchor{36}
@subsection @code{dis} — Disassembler for Python bytecode


@geindex module; dis

@strong{Source code:} Lib/dis.py@footnote{https://github.com/python/cpython/tree/main/Lib/dis.py}


__________________________________________________________________


The @ref{36,,dis} module supports the analysis of CPython @ref{8a8,,bytecode} by
disassembling it. The CPython bytecode which this module takes as an input is
defined in the file @code{Include/opcode.h} and used by the compiler and the
interpreter.

@strong{CPython implementation detail:} Bytecode is an implementation detail of the CPython interpreter.  No
guarantees are made that bytecode will not be added, removed, or changed
between versions of Python.  Use of this module should not be considered to
work across Python VMs or Python releases.


Changed in version 3.6: Use 2 bytes for each instruction. Previously the number of bytes varied
by instruction.



Changed in version 3.11: Some instructions are accompanied by one or more inline cache entries,
which take the form of @code{CACHE} instructions. These instructions
are hidden by default, but can be shown by passing @code{show_caches=True} to
any @ref{36,,dis} utility.


Example: Given the function @code{myfunc()}:

@example
def myfunc(alist):
    return len(alist)
@end example

the following command can be used to display the disassembly of
@code{myfunc()}:

@example
>>> dis.dis(myfunc)
  1           0 RESUME                   0

  2           2 PUSH_NULL
              4 LOAD_GLOBAL              0 (len)
              6 LOAD_FAST                0 (alist)
              8 PRECALL                  1
             10 CALL                     1
             12 RETURN_VALUE
@end example

(The “2” is a line number).

@menu
* Bytecode analysis:: 
* Analysis functions:: 
* Python Bytecode Instructions:: 
* Opcode collections:: 

@end menu

@node Bytecode analysis,Analysis functions,,dis — Disassembler for Python bytecode
@anchor{library/dis bytecode-analysis}@anchor{3e4e}
@subsubsection Bytecode analysis



New in version 3.4.


The bytecode analysis API allows pieces of Python code to be wrapped in a
@ref{aab,,Bytecode} object that provides easy access to details of the compiled
code.

@geindex Bytecode (class in dis)
@anchor{library/dis dis Bytecode}@anchor{aab}
@deffn {Class} dis.Bytecode (x, *, first_line=None, current_offset=None, show_caches=False)

Analyse the bytecode corresponding to a function, generator, asynchronous
generator, coroutine, method, string of source code, or a code object (as
returned by @ref{4e2,,compile()}).

This is a convenience wrapper around many of the functions listed below, most
notably @ref{aaa,,get_instructions()}, as iterating over a @ref{aab,,Bytecode}
instance yields the bytecode operations as @ref{aa9,,Instruction} instances.

If @emph{first_line} is not @code{None}, it indicates the line number that should be
reported for the first source line in the disassembled code.  Otherwise, the
source line information (if any) is taken directly from the disassembled code
object.

If @emph{current_offset} is not @code{None}, it refers to an instruction offset in the
disassembled code. Setting this means @ref{aac,,dis()} will display a “current
instruction” marker against the specified opcode.

@geindex from_traceback() (dis.Bytecode class method)
@anchor{library/dis dis Bytecode from_traceback}@anchor{aad}
@deffn {Method} classmethod from_traceback (tb, *, show_caches=False)

Construct a @ref{aab,,Bytecode} instance from the given traceback, setting
@emph{current_offset} to the instruction responsible for the exception.
@end deffn

@geindex Bytecode.codeobj (in module dis)
@anchor{library/dis dis Bytecode codeobj}@anchor{3e4f}
@deffn {Data} codeobj

The compiled code object.
@end deffn

@geindex Bytecode.first_line (in module dis)
@anchor{library/dis dis Bytecode first_line}@anchor{3e50}
@deffn {Data} first_line

The first source line of the code object (if available)
@end deffn

@geindex dis() (dis.Bytecode method)
@anchor{library/dis dis Bytecode dis}@anchor{aac}
@deffn {Method} dis ()

Return a formatted view of the bytecode operations (the same as printed by
@ref{655,,dis.dis()}, but returned as a multi-line string).
@end deffn

@geindex info() (dis.Bytecode method)
@anchor{library/dis dis Bytecode info}@anchor{3e51}
@deffn {Method} info ()

Return a formatted multi-line string with detailed information about the
code object, like @ref{deb,,code_info()}.
@end deffn


Changed in version 3.7: This can now handle coroutine and asynchronous generator objects.



Changed in version 3.11: Added the @code{show_caches} parameter.

@end deffn

Example:

@example
>>> bytecode = dis.Bytecode(myfunc)
>>> for instr in bytecode:
...     print(instr.opname)
...
RESUME
PUSH_NULL
LOAD_GLOBAL
LOAD_FAST
PRECALL
CALL
RETURN_VALUE
@end example

@node Analysis functions,Python Bytecode Instructions,Bytecode analysis,dis — Disassembler for Python bytecode
@anchor{library/dis analysis-functions}@anchor{3e52}
@subsubsection Analysis functions


The @ref{36,,dis} module also defines the following analysis functions that convert
the input directly to the desired output. They can be useful if only a single
operation is being performed, so the intermediate analysis object isn’t useful:

@geindex code_info() (in module dis)
@anchor{library/dis dis code_info}@anchor{deb}
@deffn {Function} dis.code_info (x)

Return a formatted multi-line string with detailed code object information
for the supplied function, generator, asynchronous generator, coroutine,
method, source code string or code object.

Note that the exact contents of code info strings are highly implementation
dependent and they may change arbitrarily across Python VMs or Python
releases.


New in version 3.2.



Changed in version 3.7: This can now handle coroutine and asynchronous generator objects.

@end deffn

@geindex show_code() (in module dis)
@anchor{library/dis dis show_code}@anchor{aa6}
@deffn {Function} dis.show_code (x, *, file=None)

Print detailed code object information for the supplied function, method,
source code string or code object to @emph{file} (or @code{sys.stdout} if @emph{file}
is not specified).

This is a convenient shorthand for @code{print(code_info(x), file=file)},
intended for interactive exploration at the interpreter prompt.


New in version 3.2.



Changed in version 3.4: Added @emph{file} parameter.

@end deffn

@geindex dis() (in module dis)
@anchor{library/dis dis dis}@anchor{655}
@deffn {Function} dis.dis (x=None, *, file=None, depth=None, show_caches=False)

Disassemble the @emph{x} object.  @emph{x} can denote either a module, a class, a
method, a function, a generator, an asynchronous generator, a coroutine,
a code object, a string of source code or a byte sequence of raw bytecode.
For a module, it disassembles all functions. For a class, it disassembles
all methods (including class and static methods). For a code object or
sequence of raw bytecode, it prints one line per bytecode instruction.
It also recursively disassembles nested code objects (the code of
comprehensions, generator expressions and nested functions, and the code
used for building nested classes).
Strings are first compiled to code objects with the @ref{4e2,,compile()}
built-in function before being disassembled.  If no object is provided, this
function disassembles the last traceback.

The disassembly is written as text to the supplied @emph{file} argument if
provided and to @code{sys.stdout} otherwise.

The maximal depth of recursion is limited by @emph{depth} unless it is @code{None}.
@code{depth=0} means no recursion.


Changed in version 3.4: Added @emph{file} parameter.



Changed in version 3.7: Implemented recursive disassembling and added @emph{depth} parameter.



Changed in version 3.7: This can now handle coroutine and asynchronous generator objects.



Changed in version 3.11: Added the @code{show_caches} parameter.

@end deffn

@geindex distb() (in module dis)
@anchor{library/dis dis distb}@anchor{aa7}
@deffn {Function} dis.distb (tb=None, *, file=None, show_caches=False)

Disassemble the top-of-stack function of a traceback, using the last
traceback if none was passed.  The instruction causing the exception is
indicated.

The disassembly is written as text to the supplied @emph{file} argument if
provided and to @code{sys.stdout} otherwise.


Changed in version 3.4: Added @emph{file} parameter.



Changed in version 3.11: Added the @code{show_caches} parameter.

@end deffn

@geindex disassemble() (in module dis)
@geindex disco() (in module dis)
@anchor{library/dis dis disassemble}@anchor{aa8}
@deffn {Function} dis.disassemble (code, lasti=- 1, *, file=None, show_caches=False)
@anchor{library/dis dis disco}@anchor{3e53}
@deffnx {Function} dis.disco (code, lasti=- 1, *, file=None, show_caches=False)

Disassemble a code object, indicating the last instruction if @emph{lasti} was
provided.  The output is divided in the following columns:


@enumerate 

@item 
the line number, for the first instruction of each line

@item 
the current instruction, indicated as @code{-->},

@item 
a labelled instruction, indicated with @code{>>},

@item 
the address of the instruction,

@item 
the operation code name,

@item 
operation parameters, and

@item 
interpretation of the parameters in parentheses.
@end enumerate

The parameter interpretation recognizes local and global variable names,
constant values, branch targets, and compare operators.

The disassembly is written as text to the supplied @emph{file} argument if
provided and to @code{sys.stdout} otherwise.


Changed in version 3.4: Added @emph{file} parameter.



Changed in version 3.11: Added the @code{show_caches} parameter.

@end deffn

@geindex get_instructions() (in module dis)
@anchor{library/dis dis get_instructions}@anchor{aaa}
@deffn {Function} dis.get_instructions (x, *, first_line=None, show_caches=False)

Return an iterator over the instructions in the supplied function, method,
source code string or code object.

The iterator generates a series of @ref{aa9,,Instruction} named tuples giving
the details of each operation in the supplied code.

If @emph{first_line} is not @code{None}, it indicates the line number that should be
reported for the first source line in the disassembled code.  Otherwise, the
source line information (if any) is taken directly from the disassembled code
object.


New in version 3.4.



Changed in version 3.11: Added the @code{show_caches} parameter.

@end deffn

@geindex findlinestarts() (in module dis)
@anchor{library/dis dis findlinestarts}@anchor{3e54}
@deffn {Function} dis.findlinestarts (code)

This generator function uses the @code{co_firstlineno} and @code{co_lnotab}
attributes of the code object @emph{code} to find the offsets which are starts of
lines in the source code.  They are generated as @code{(offset, lineno)} pairs.
See Objects/lnotab_notes.txt@footnote{https://github.com/python/cpython/tree/main/Objects/lnotab_notes.txt} for the @code{co_lnotab} format and
how to decode it.


Changed in version 3.6: Line numbers can be decreasing. Before, they were always increasing.

@end deffn

@geindex findlabels() (in module dis)
@anchor{library/dis dis findlabels}@anchor{3e55}
@deffn {Function} dis.findlabels (code)

Detect all offsets in the raw compiled bytecode string @emph{code} which are jump targets, and
return a list of these offsets.
@end deffn

@geindex stack_effect() (in module dis)
@anchor{library/dis dis stack_effect}@anchor{aae}
@deffn {Function} dis.stack_effect (opcode, oparg=None, *, jump=None)

Compute the stack effect of @emph{opcode} with argument @emph{oparg}.

If the code has a jump target and @emph{jump} is @code{True}, @ref{aae,,stack_effect()}
will return the stack effect of jumping.  If @emph{jump} is @code{False},
it will return the stack effect of not jumping. And if @emph{jump} is
@code{None} (default), it will return the maximal stack effect of both cases.


New in version 3.4.



Changed in version 3.8: Added @emph{jump} parameter.

@end deffn

@node Python Bytecode Instructions,Opcode collections,Analysis functions,dis — Disassembler for Python bytecode
@anchor{library/dis bytecodes}@anchor{3e56}@anchor{library/dis python-bytecode-instructions}@anchor{3e57}
@subsubsection Python Bytecode Instructions


The @ref{aaa,,get_instructions()} function and @ref{aab,,Bytecode} class provide
details of bytecode instructions as @ref{aa9,,Instruction} instances:

@geindex Instruction (class in dis)
@anchor{library/dis dis Instruction}@anchor{aa9}
@deffn {Class} dis.Instruction

Details for a bytecode operation

@geindex Instruction.opcode (in module dis)
@anchor{library/dis dis Instruction opcode}@anchor{3e58}
@deffn {Data} opcode

numeric code for operation, corresponding to the opcode values listed
below and the bytecode values in the @ref{3e59,,Opcode collections}.
@end deffn

@geindex Instruction.opname (in module dis)
@anchor{library/dis dis Instruction opname}@anchor{3e5a}
@deffn {Data} opname

human readable name for operation
@end deffn

@geindex Instruction.arg (in module dis)
@anchor{library/dis dis Instruction arg}@anchor{3e5b}
@deffn {Data} arg

numeric argument to operation (if any), otherwise @code{None}
@end deffn

@geindex Instruction.argval (in module dis)
@anchor{library/dis dis Instruction argval}@anchor{3e5c}
@deffn {Data} argval

resolved arg value (if any), otherwise @code{None}
@end deffn

@geindex Instruction.argrepr (in module dis)
@anchor{library/dis dis Instruction argrepr}@anchor{3e5d}
@deffn {Data} argrepr

human readable description of operation argument (if any),
otherwise an empty string.
@end deffn

@geindex Instruction.offset (in module dis)
@anchor{library/dis dis Instruction offset}@anchor{3e5e}
@deffn {Data} offset

start index of operation within bytecode sequence
@end deffn

@geindex Instruction.starts_line (in module dis)
@anchor{library/dis dis Instruction starts_line}@anchor{3e5f}
@deffn {Data} starts_line

line started by this opcode (if any), otherwise @code{None}
@end deffn

@geindex Instruction.is_jump_target (in module dis)
@anchor{library/dis dis Instruction is_jump_target}@anchor{3e60}
@deffn {Data} is_jump_target

@code{True} if other code jumps to here, otherwise @code{False}
@end deffn

@geindex Instruction.positions (in module dis)
@anchor{library/dis dis Instruction positions}@anchor{3e61}
@deffn {Data} positions

@ref{3e62,,dis.Positions} object holding the
start and end locations that are covered by this instruction.
@end deffn


New in version 3.4.



Changed in version 3.11: Field @code{positions} is added.

@end deffn

@geindex Positions (class in dis)
@anchor{library/dis dis Positions}@anchor{3e62}
@deffn {Class} dis.Positions

In case the information is not available, some fields might be @cite{None}.

@geindex Positions.lineno (in module dis)
@anchor{library/dis dis Positions lineno}@anchor{3e63}
@deffn {Data} lineno
@end deffn

@geindex Positions.end_lineno (in module dis)
@anchor{library/dis dis Positions end_lineno}@anchor{3e64}
@deffn {Data} end_lineno
@end deffn

@geindex Positions.col_offset (in module dis)
@anchor{library/dis dis Positions col_offset}@anchor{3e65}
@deffn {Data} col_offset
@end deffn

@geindex Positions.end_col_offset (in module dis)
@anchor{library/dis dis Positions end_col_offset}@anchor{3e66}
@deffn {Data} end_col_offset
@end deffn


New in version 3.11.

@end deffn

The Python compiler currently generates the following bytecode instructions.

@strong{General instructions}

@geindex NOP (opcode)
@anchor{library/dis opcode-NOP}@anchor{3e67}
@deffn {Opcode} NOP

Do nothing code.  Used as a placeholder by the bytecode optimizer, and to
generate line tracing events.
@end deffn

@geindex POP_TOP (opcode)
@anchor{library/dis opcode-POP_TOP}@anchor{3e68}
@deffn {Opcode} POP_TOP

Removes the top-of-stack (TOS) item.
@end deffn

@geindex COPY (opcode)
@anchor{library/dis opcode-COPY}@anchor{1a5}
@deffn {Opcode} COPY (i)

Push the @emph{i}-th item to the top of the stack. The item is not removed from its
original location.


New in version 3.11.

@end deffn

@geindex SWAP (opcode)
@anchor{library/dis opcode-SWAP}@anchor{1a6}
@deffn {Opcode} SWAP (i)

Swap TOS with the item at position @emph{i}.


New in version 3.11.

@end deffn

@strong{Unary operations}

Unary operations take the top of the stack, apply the operation, and push the
result back on the stack.

@geindex UNARY_POSITIVE (opcode)
@anchor{library/dis opcode-UNARY_POSITIVE}@anchor{3e69}
@deffn {Opcode} UNARY_POSITIVE

Implements @code{TOS = +TOS}.
@end deffn

@geindex UNARY_NEGATIVE (opcode)
@anchor{library/dis opcode-UNARY_NEGATIVE}@anchor{3e6a}
@deffn {Opcode} UNARY_NEGATIVE

Implements @code{TOS = -TOS}.
@end deffn

@geindex UNARY_NOT (opcode)
@anchor{library/dis opcode-UNARY_NOT}@anchor{3e6b}
@deffn {Opcode} UNARY_NOT

Implements @code{TOS = not TOS}.
@end deffn

@geindex UNARY_INVERT (opcode)
@anchor{library/dis opcode-UNARY_INVERT}@anchor{3e6c}
@deffn {Opcode} UNARY_INVERT

Implements @code{TOS = ~TOS}.
@end deffn

@geindex GET_ITER (opcode)
@anchor{library/dis opcode-GET_ITER}@anchor{3e6d}
@deffn {Opcode} GET_ITER

Implements @code{TOS = iter(TOS)}.
@end deffn

@geindex GET_YIELD_FROM_ITER (opcode)
@anchor{library/dis opcode-GET_YIELD_FROM_ITER}@anchor{3e6e}
@deffn {Opcode} GET_YIELD_FROM_ITER

If @code{TOS} is a @ref{715,,generator iterator} or @ref{3cf,,coroutine} object
it is left as is.  Otherwise, implements @code{TOS = iter(TOS)}.


New in version 3.5.

@end deffn

@strong{Binary and in-place operations}

Binary operations remove the top of the stack (TOS) and the second top-most
stack item (TOS1) from the stack.  They perform the operation, and put the
result back on the stack.

In-place operations are like binary operations, in that they remove TOS and
TOS1, and push the result back on the stack, but the operation is done in-place
when TOS1 supports it, and the resulting TOS may be (but does not have to be)
the original TOS1.

@geindex BINARY_OP (opcode)
@anchor{library/dis opcode-BINARY_OP}@anchor{19e}
@deffn {Opcode} BINARY_OP (op)

Implements the binary and in-place operators (depending on the value of
@emph{op}).


New in version 3.11.

@end deffn

@geindex BINARY_SUBSCR (opcode)
@anchor{library/dis opcode-BINARY_SUBSCR}@anchor{1080}
@deffn {Opcode} BINARY_SUBSCR

Implements @code{TOS = TOS1[TOS]}.
@end deffn

@geindex STORE_SUBSCR (opcode)
@anchor{library/dis opcode-STORE_SUBSCR}@anchor{3e6f}
@deffn {Opcode} STORE_SUBSCR

Implements @code{TOS1[TOS] = TOS2}.
@end deffn

@geindex DELETE_SUBSCR (opcode)
@anchor{library/dis opcode-DELETE_SUBSCR}@anchor{3e70}
@deffn {Opcode} DELETE_SUBSCR

Implements @code{del TOS1[TOS]}.
@end deffn

@strong{Coroutine opcodes}

@geindex GET_AWAITABLE (opcode)
@anchor{library/dis opcode-GET_AWAITABLE}@anchor{107c}
@deffn {Opcode} GET_AWAITABLE (where)

Implements @code{TOS = get_awaitable(TOS)}, where @code{get_awaitable(o)}
returns @code{o} if @code{o} is a coroutine object or a generator object with
the CO_ITERABLE_COROUTINE flag, or resolves
@code{o.__await__}.

@quotation

If the @code{where} operand is nonzero, it indicates where the instruction
occurs:


@itemize *

@item 
@code{1} After a call to @code{__aenter__}

@item 
@code{2} After a call to @code{__aexit__}
@end itemize
@end quotation


New in version 3.5.



Changed in version 3.11: Previously, this instruction did not have an oparg.

@end deffn

@geindex GET_AITER (opcode)
@anchor{library/dis opcode-GET_AITER}@anchor{3e71}
@deffn {Opcode} GET_AITER

Implements @code{TOS = TOS.__aiter__()}.


New in version 3.5.



Changed in version 3.7: Returning awaitable objects from @code{__aiter__} is no longer
supported.

@end deffn

@geindex GET_ANEXT (opcode)
@anchor{library/dis opcode-GET_ANEXT}@anchor{3e72}
@deffn {Opcode} GET_ANEXT

Implements @code{PUSH(get_awaitable(TOS.__anext__()))}.  See @code{GET_AWAITABLE}
for details about @code{get_awaitable}


New in version 3.5.

@end deffn

@geindex END_ASYNC_FOR (opcode)
@anchor{library/dis opcode-END_ASYNC_FOR}@anchor{5cc}
@deffn {Opcode} END_ASYNC_FOR

Terminates an @ref{5cd,,async for} loop.  Handles an exception raised
when awaiting a next item.  If TOS is @ref{12da,,StopAsyncIteration} pop 3
values from the stack and restore the exception state using the second
of them.  Otherwise re-raise the exception using the value
from the stack.  An exception handler block is removed from the block stack.


New in version 3.8: 


Changed in version 3.11: Exception representation on the stack now consist of one, not three, items.


@end deffn

@geindex BEFORE_ASYNC_WITH (opcode)
@anchor{library/dis opcode-BEFORE_ASYNC_WITH}@anchor{3e73}
@deffn {Opcode} BEFORE_ASYNC_WITH

Resolves @code{__aenter__} and @code{__aexit__} from the object on top of the
stack.  Pushes @code{__aexit__} and result of @code{__aenter__()} to the stack.


New in version 3.5.

@end deffn

@strong{Miscellaneous opcodes}

@geindex PRINT_EXPR (opcode)
@anchor{library/dis opcode-PRINT_EXPR}@anchor{3e74}
@deffn {Opcode} PRINT_EXPR

Implements the expression statement for the interactive mode.  TOS is removed
from the stack and printed.  In non-interactive mode, an expression statement
is terminated with @ref{3e68,,POP_TOP}.
@end deffn

@geindex SET_ADD (opcode)
@anchor{library/dis opcode-SET_ADD}@anchor{3e75}
@deffn {Opcode} SET_ADD (i)

Calls @code{set.add(TOS1[-i], TOS)}.  Used to implement set comprehensions.
@end deffn

@geindex LIST_APPEND (opcode)
@anchor{library/dis opcode-LIST_APPEND}@anchor{3e76}
@deffn {Opcode} LIST_APPEND (i)

Calls @code{list.append(TOS1[-i], TOS)}.  Used to implement list comprehensions.
@end deffn

@geindex MAP_ADD (opcode)
@anchor{library/dis opcode-MAP_ADD}@anchor{5ce}
@deffn {Opcode} MAP_ADD (i)

Calls @code{dict.__setitem__(TOS1[-i], TOS1, TOS)}.  Used to implement dict
comprehensions.


New in version 3.1.



Changed in version 3.8: Map value is TOS and map key is TOS1. Before, those were reversed.

@end deffn

For all of the @ref{3e75,,SET_ADD}, @ref{3e76,,LIST_APPEND} and @ref{5ce,,MAP_ADD}
instructions, while the added value or key/value pair is popped off, the
container object remains on the stack so that it is available for further
iterations of the loop.

@geindex RETURN_VALUE (opcode)
@anchor{library/dis opcode-RETURN_VALUE}@anchor{3e77}
@deffn {Opcode} RETURN_VALUE

Returns with TOS to the caller of the function.
@end deffn

@geindex YIELD_VALUE (opcode)
@anchor{library/dis opcode-YIELD_VALUE}@anchor{3e78}
@deffn {Opcode} YIELD_VALUE

Pops TOS and yields it from a @ref{bf7,,generator}.
@end deffn

@geindex YIELD_FROM (opcode)
@anchor{library/dis opcode-YIELD_FROM}@anchor{3e79}
@deffn {Opcode} YIELD_FROM

Pops TOS and delegates to it as a subiterator from a @ref{bf7,,generator}.


New in version 3.3.

@end deffn

@geindex SETUP_ANNOTATIONS (opcode)
@anchor{library/dis opcode-SETUP_ANNOTATIONS}@anchor{8ae}
@deffn {Opcode} SETUP_ANNOTATIONS

Checks whether @code{__annotations__} is defined in @code{locals()}, if not it is
set up to an empty @code{dict}. This opcode is only emitted if a class
or module body contains @ref{8af,,variable annotations}
statically.


New in version 3.6.

@end deffn

@geindex IMPORT_STAR (opcode)
@anchor{library/dis opcode-IMPORT_STAR}@anchor{3e7a}
@deffn {Opcode} IMPORT_STAR

Loads all symbols not starting with @code{'_'} directly from the module TOS to
the local namespace. The module is popped after loading all names. This
opcode implements @code{from module import *}.
@end deffn

@geindex POP_EXCEPT (opcode)
@anchor{library/dis opcode-POP_EXCEPT}@anchor{3e7b}
@deffn {Opcode} POP_EXCEPT

Pops a value from the stack, which is used to restore the exception state.

@quotation


Changed in version 3.11: Exception representation on the stack now consist of one, not three, items.

@end quotation
@end deffn

@geindex RERAISE (opcode)
@anchor{library/dis opcode-RERAISE}@anchor{3e7c}
@deffn {Opcode} RERAISE

Re-raises the exception currently on top of the stack. If oparg is non-zero,
pops an additional value from the stack which is used to set @code{f_lasti}
of the current frame.


New in version 3.9.



Changed in version 3.11: Exception representation on the stack now consist of one, not three, items.

@end deffn

@geindex PUSH_EXC_INFO (opcode)
@anchor{library/dis opcode-PUSH_EXC_INFO}@anchor{3e7d}
@deffn {Opcode} PUSH_EXC_INFO

Pops a value from the stack. Pushes the current exception to the top of the stack.
Pushes the value originally popped back to the stack.
Used in exception handlers.


New in version 3.11.

@end deffn

@geindex WITH_EXCEPT_START (opcode)
@anchor{library/dis opcode-WITH_EXCEPT_START}@anchor{3e7e}
@deffn {Opcode} WITH_EXCEPT_START

Calls the function in position 4 on the stack with arguments (type, val, tb)
representing the exception at the top of the stack.
Used to implement the call @code{context_manager.__exit__(*exc_info())} when an exception
has occurred in a @ref{168,,with} statement.


New in version 3.9.



Changed in version 3.11: The @code{__exit__} function is in position 4 of the stack rather than 7.
Exception representation on the stack now consist of one, not three, items.

@end deffn

@geindex LOAD_ASSERTION_ERROR (opcode)
@anchor{library/dis opcode-LOAD_ASSERTION_ERROR}@anchor{46c}
@deffn {Opcode} LOAD_ASSERTION_ERROR

Pushes @ref{46e,,AssertionError} onto the stack.  Used by the @ref{46d,,assert}
statement.


New in version 3.9.

@end deffn

@geindex LOAD_BUILD_CLASS (opcode)
@anchor{library/dis opcode-LOAD_BUILD_CLASS}@anchor{3e7f}
@deffn {Opcode} LOAD_BUILD_CLASS

Pushes @code{builtins.__build_class__()} onto the stack.  It is later called
to construct a class.
@end deffn

@geindex BEFORE_WITH (opcode)
@anchor{library/dis opcode-BEFORE_WITH}@anchor{3e80}
@deffn {Opcode} BEFORE_WITH (delta)

This opcode performs several operations before a with block starts.  First,
it loads @ref{1812,,__exit__()} from the context manager and pushes it onto
the stack for later use by @ref{3e7e,,WITH_EXCEPT_START}.  Then,
@ref{1811,,__enter__()} is called. Finally, the result of calling the
@code{__enter__()} method is pushed onto the stack.


New in version 3.11.

@end deffn

@geindex GET_LEN (opcode)
@anchor{library/dis opcode-GET_LEN}@anchor{3e81}
@deffn {Opcode} GET_LEN

Push @code{len(TOS)} onto the stack.


New in version 3.10.

@end deffn

@geindex MATCH_MAPPING (opcode)
@anchor{library/dis opcode-MATCH_MAPPING}@anchor{3e82}
@deffn {Opcode} MATCH_MAPPING

If TOS is an instance of @ref{3ab,,collections.abc.Mapping} (or, more technically: if
it has the @ref{110a,,Py_TPFLAGS_MAPPING} flag set in its
@ref{1133,,tp_flags}), push @code{True} onto the stack.  Otherwise, push
@code{False}.


New in version 3.10.

@end deffn

@geindex MATCH_SEQUENCE (opcode)
@anchor{library/dis opcode-MATCH_SEQUENCE}@anchor{3e83}
@deffn {Opcode} MATCH_SEQUENCE

If TOS is an instance of @ref{110c,,collections.abc.Sequence} and is @emph{not} an instance
of @ref{2ef,,str}/@ref{2f0,,bytes}/@ref{2f1,,bytearray} (or, more technically: if it has
the @ref{110b,,Py_TPFLAGS_SEQUENCE} flag set in its @ref{1133,,tp_flags}),
push @code{True} onto the stack.  Otherwise, push @code{False}.


New in version 3.10.

@end deffn

@geindex MATCH_KEYS (opcode)
@anchor{library/dis opcode-MATCH_KEYS}@anchor{1a4}
@deffn {Opcode} MATCH_KEYS

TOS is a tuple of mapping keys, and TOS1 is the match subject.  If TOS1
contains all of the keys in TOS, push a @ref{3e5,,tuple} containing the
corresponding values. Otherwise, push @code{None}.


New in version 3.10.



Changed in version 3.11: Previously, this instruction also pushed a boolean value indicating
success (@code{True}) or failure (@code{False}).

@end deffn

@geindex STORE_NAME (opcode)
@anchor{library/dis opcode-STORE_NAME}@anchor{3e84}
@deffn {Opcode} STORE_NAME (namei)

Implements @code{name = TOS}. @emph{namei} is the index of @emph{name} in the attribute
@code{co_names} of the code object. The compiler tries to use
@ref{3e85,,STORE_FAST} or @ref{3e86,,STORE_GLOBAL} if possible.
@end deffn

@geindex DELETE_NAME (opcode)
@anchor{library/dis opcode-DELETE_NAME}@anchor{3e87}
@deffn {Opcode} DELETE_NAME (namei)

Implements @code{del name}, where @emph{namei} is the index into @code{co_names}
attribute of the code object.
@end deffn

@geindex UNPACK_SEQUENCE (opcode)
@anchor{library/dis opcode-UNPACK_SEQUENCE}@anchor{1081}
@deffn {Opcode} UNPACK_SEQUENCE (count)

Unpacks TOS into @emph{count} individual values, which are put onto the stack
right-to-left.
@end deffn

@geindex UNPACK_EX (opcode)
@anchor{library/dis opcode-UNPACK_EX}@anchor{3e88}
@deffn {Opcode} UNPACK_EX (counts)

Implements assignment with a starred target: Unpacks an iterable in TOS into
individual values, where the total number of values can be smaller than the
number of items in the iterable: one of the new values will be a list of all
leftover items.

The low byte of @emph{counts} is the number of values before the list value, the
high byte of @emph{counts} the number of values after it.  The resulting values
are put onto the stack right-to-left.
@end deffn

@geindex STORE_ATTR (opcode)
@anchor{library/dis opcode-STORE_ATTR}@anchor{107f}
@deffn {Opcode} STORE_ATTR (namei)

Implements @code{TOS.name = TOS1}, where @emph{namei} is the index of name in
@code{co_names}.
@end deffn

@geindex DELETE_ATTR (opcode)
@anchor{library/dis opcode-DELETE_ATTR}@anchor{3e89}
@deffn {Opcode} DELETE_ATTR (namei)

Implements @code{del TOS.name}, using @emph{namei} as index into @code{co_names}.
@end deffn

@geindex STORE_GLOBAL (opcode)
@anchor{library/dis opcode-STORE_GLOBAL}@anchor{3e86}
@deffn {Opcode} STORE_GLOBAL (namei)

Works as @ref{3e84,,STORE_NAME}, but stores the name as a global.
@end deffn

@geindex DELETE_GLOBAL (opcode)
@anchor{library/dis opcode-DELETE_GLOBAL}@anchor{3e8a}
@deffn {Opcode} DELETE_GLOBAL (namei)

Works as @ref{3e87,,DELETE_NAME}, but deletes a global name.
@end deffn

@geindex LOAD_CONST (opcode)
@anchor{library/dis opcode-LOAD_CONST}@anchor{3e8b}
@deffn {Opcode} LOAD_CONST (consti)

Pushes @code{co_consts[consti]} onto the stack.
@end deffn

@geindex LOAD_NAME (opcode)
@anchor{library/dis opcode-LOAD_NAME}@anchor{3e8c}
@deffn {Opcode} LOAD_NAME (namei)

Pushes the value associated with @code{co_names[namei]} onto the stack.
@end deffn

@geindex BUILD_TUPLE (opcode)
@anchor{library/dis opcode-BUILD_TUPLE}@anchor{3e8d}
@deffn {Opcode} BUILD_TUPLE (count)

Creates a tuple consuming @emph{count} items from the stack, and pushes the
resulting tuple onto the stack.
@end deffn

@geindex BUILD_LIST (opcode)
@anchor{library/dis opcode-BUILD_LIST}@anchor{3e8e}
@deffn {Opcode} BUILD_LIST (count)

Works as @ref{3e8d,,BUILD_TUPLE}, but creates a list.
@end deffn

@geindex BUILD_SET (opcode)
@anchor{library/dis opcode-BUILD_SET}@anchor{3e8f}
@deffn {Opcode} BUILD_SET (count)

Works as @ref{3e8d,,BUILD_TUPLE}, but creates a set.
@end deffn

@geindex BUILD_MAP (opcode)
@anchor{library/dis opcode-BUILD_MAP}@anchor{3e90}
@deffn {Opcode} BUILD_MAP (count)

Pushes a new dictionary object onto the stack.  Pops @code{2 * count} items
so that the dictionary holds @emph{count} entries:
@code{@{..., TOS3: TOS2, TOS1: TOS@}}.


Changed in version 3.5: The dictionary is created from stack items instead of creating an
empty dictionary pre-sized to hold @emph{count} items.

@end deffn

@geindex BUILD_CONST_KEY_MAP (opcode)
@anchor{library/dis opcode-BUILD_CONST_KEY_MAP}@anchor{8ab}
@deffn {Opcode} BUILD_CONST_KEY_MAP (count)

The version of @ref{3e90,,BUILD_MAP} specialized for constant keys. Pops the
top element on the stack which contains a tuple of keys, then starting from
@code{TOS1}, pops @emph{count} values to form values in the built dictionary.


New in version 3.6.

@end deffn

@geindex BUILD_STRING (opcode)
@anchor{library/dis opcode-BUILD_STRING}@anchor{8aa}
@deffn {Opcode} BUILD_STRING (count)

Concatenates @emph{count} strings from the stack and pushes the resulting string
onto the stack.


New in version 3.6.

@end deffn

@geindex LIST_TO_TUPLE (opcode)
@anchor{library/dis opcode-LIST_TO_TUPLE}@anchor{3e91}
@deffn {Opcode} LIST_TO_TUPLE

Pops a list from the stack and pushes a tuple containing the same values.


New in version 3.9.

@end deffn

@geindex LIST_EXTEND (opcode)
@anchor{library/dis opcode-LIST_EXTEND}@anchor{3e92}
@deffn {Opcode} LIST_EXTEND (i)

Calls @code{list.extend(TOS1[-i], TOS)}.  Used to build lists.


New in version 3.9.

@end deffn

@geindex SET_UPDATE (opcode)
@anchor{library/dis opcode-SET_UPDATE}@anchor{3e93}
@deffn {Opcode} SET_UPDATE (i)

Calls @code{set.update(TOS1[-i], TOS)}.  Used to build sets.


New in version 3.9.

@end deffn

@geindex DICT_UPDATE (opcode)
@anchor{library/dis opcode-DICT_UPDATE}@anchor{3e94}
@deffn {Opcode} DICT_UPDATE (i)

Calls @code{dict.update(TOS1[-i], TOS)}.  Used to build dicts.


New in version 3.9.

@end deffn

@geindex DICT_MERGE (opcode)
@anchor{library/dis opcode-DICT_MERGE}@anchor{3e95}
@deffn {Opcode} DICT_MERGE (i)

Like @ref{3e94,,DICT_UPDATE} but raises an exception for duplicate keys.


New in version 3.9.

@end deffn

@geindex LOAD_ATTR (opcode)
@anchor{library/dis opcode-LOAD_ATTR}@anchor{107e}
@deffn {Opcode} LOAD_ATTR (namei)

Replaces TOS with @code{getattr(TOS, co_names[namei])}.
@end deffn

@geindex COMPARE_OP (opcode)
@anchor{library/dis opcode-COMPARE_OP}@anchor{46f}
@deffn {Opcode} COMPARE_OP (opname)

Performs a Boolean operation.  The operation name can be found in
@code{cmp_op[opname]}.
@end deffn

@geindex IS_OP (opcode)
@anchor{library/dis opcode-IS_OP}@anchor{3e96}
@deffn {Opcode} IS_OP (invert)

Performs @code{is} comparison, or @code{is not} if @code{invert} is 1.


New in version 3.9.

@end deffn

@geindex CONTAINS_OP (opcode)
@anchor{library/dis opcode-CONTAINS_OP}@anchor{3e97}
@deffn {Opcode} CONTAINS_OP (invert)

Performs @code{in} comparison, or @code{not in} if @code{invert} is 1.


New in version 3.9.

@end deffn

@geindex IMPORT_NAME (opcode)
@anchor{library/dis opcode-IMPORT_NAME}@anchor{3e98}
@deffn {Opcode} IMPORT_NAME (namei)

Imports the module @code{co_names[namei]}.  TOS and TOS1 are popped and provide
the @emph{fromlist} and @emph{level} arguments of @ref{3b7,,__import__()}.  The module
object is pushed onto the stack.  The current namespace is not affected: for
a proper import statement, a subsequent @ref{3e85,,STORE_FAST} instruction
modifies the namespace.
@end deffn

@geindex IMPORT_FROM (opcode)
@anchor{library/dis opcode-IMPORT_FROM}@anchor{3e99}
@deffn {Opcode} IMPORT_FROM (namei)

Loads the attribute @code{co_names[namei]} from the module found in TOS. The
resulting object is pushed onto the stack, to be subsequently stored by a
@ref{3e85,,STORE_FAST} instruction.
@end deffn

@geindex JUMP_FORWARD (opcode)
@anchor{library/dis opcode-JUMP_FORWARD}@anchor{3e9a}
@deffn {Opcode} JUMP_FORWARD (delta)

Increments bytecode counter by @emph{delta}.
@end deffn

@geindex POP_JUMP_IF_TRUE (opcode)
@anchor{library/dis opcode-POP_JUMP_IF_TRUE}@anchor{3e9b}
@deffn {Opcode} POP_JUMP_IF_TRUE (target)

If TOS is true, sets the bytecode counter to @emph{target}.  TOS is popped.


New in version 3.1.

@end deffn

@geindex POP_JUMP_IF_FALSE (opcode)
@anchor{library/dis opcode-POP_JUMP_IF_FALSE}@anchor{3e9c}
@deffn {Opcode} POP_JUMP_IF_FALSE (target)

If TOS is false, sets the bytecode counter to @emph{target}.  TOS is popped.


New in version 3.1.

@end deffn

@geindex JUMP_IF_NOT_EXC_MATCH (opcode)
@anchor{library/dis opcode-JUMP_IF_NOT_EXC_MATCH}@anchor{1a9}
@deffn {Opcode} JUMP_IF_NOT_EXC_MATCH (target)

Performs exception matching for @code{except}.
Tests whether the second value on the stack is an exception matching TOS,
and jumps if it is not. Pops one value from the stack.


New in version 3.9.



Changed in version 3.11: This opcode no longer pops the active exception.

@end deffn

@geindex JUMP_IF_NOT_EG_MATCH (opcode)
@anchor{library/dis opcode-JUMP_IF_NOT_EG_MATCH}@anchor{3e9d}
@deffn {Opcode} JUMP_IF_NOT_EG_MATCH (target)

Performs exception matching for @code{except*}. Applies @code{split(TOS)} on
the exception group representing TOS1. Jumps if no match is found.

Pops one item from the stack (the match type). If a match was found,
next item (the exception) and pushes the non-matching part of the
exception group followed by the matching part.


New in version 3.11.

@end deffn

@geindex POP_JUMP_IF_NOT_NONE (opcode)
@anchor{library/dis opcode-POP_JUMP_IF_NOT_NONE}@anchor{1a7}
@deffn {Opcode} POP_JUMP_IF_NOT_NONE (target)

If TOS is not none, sets the bytecode counter to @emph{target}.  TOS is popped.


New in version 3.11.

@end deffn

@geindex POP_JUMP_IF_NONE (opcode)
@anchor{library/dis opcode-POP_JUMP_IF_NONE}@anchor{1a8}
@deffn {Opcode} POP_JUMP_IF_NONE (target)

If TOS is none, sets the bytecode counter to @emph{target}.  TOS is popped.


New in version 3.11.

@end deffn

@geindex PREP_RERAISE_STAR (opcode)
@anchor{library/dis opcode-PREP_RERAISE_STAR}@anchor{10b6}
@deffn {Opcode} PREP_RERAISE_STAR

Combines the raised and reraised exceptions list from TOS, into an exception
group to propagate from a try-except* block. Uses the original exception
group from TOS1 to reconstruct the structure of reraised exceptions. Pops
two items from the stack and pushes the exception to reraise or @code{None}
if there isn’t one.


New in version 3.11.

@end deffn

@geindex JUMP_IF_TRUE_OR_POP (opcode)
@anchor{library/dis opcode-JUMP_IF_TRUE_OR_POP}@anchor{3e9e}
@deffn {Opcode} JUMP_IF_TRUE_OR_POP (target)

If TOS is true, sets the bytecode counter to @emph{target} and leaves TOS on the
stack.  Otherwise (TOS is false), TOS is popped.


New in version 3.1.

@end deffn

@geindex JUMP_IF_FALSE_OR_POP (opcode)
@anchor{library/dis opcode-JUMP_IF_FALSE_OR_POP}@anchor{3e9f}
@deffn {Opcode} JUMP_IF_FALSE_OR_POP (target)

If TOS is false, sets the bytecode counter to @emph{target} and leaves TOS on the
stack.  Otherwise (TOS is true), TOS is popped.


New in version 3.1.

@end deffn

@geindex JUMP_ABSOLUTE (opcode)
@anchor{library/dis opcode-JUMP_ABSOLUTE}@anchor{3ea0}
@deffn {Opcode} JUMP_ABSOLUTE (target)

Set bytecode counter to @emph{target}.
@end deffn

@geindex JUMP_NO_INTERRUPT (opcode)
@anchor{library/dis opcode-JUMP_NO_INTERRUPT}@anchor{3ea1}
@deffn {Opcode} JUMP_NO_INTERRUPT (target)

Set bytecode counter to @emph{target}. Do not check for interrupts.


New in version 3.11.

@end deffn

@geindex FOR_ITER (opcode)
@anchor{library/dis opcode-FOR_ITER}@anchor{3ea2}
@deffn {Opcode} FOR_ITER (delta)

TOS is an @ref{11c3,,iterator}.  Call its @ref{e6d,,__next__()} method.  If
this yields a new value, push it on the stack (leaving the iterator below
it).  If the iterator indicates it is exhausted, TOS is popped, and the byte
code counter is incremented by @emph{delta}.
@end deffn

@geindex LOAD_GLOBAL (opcode)
@anchor{library/dis opcode-LOAD_GLOBAL}@anchor{1082}
@deffn {Opcode} LOAD_GLOBAL (namei)

Loads the global named @code{co_names[namei]} onto the stack.
@end deffn

@geindex LOAD_FAST (opcode)
@anchor{library/dis opcode-LOAD_FAST}@anchor{3ea3}
@deffn {Opcode} LOAD_FAST (var_num)

Pushes a reference to the local @code{co_varnames[var_num]} onto the stack.
@end deffn

@geindex STORE_FAST (opcode)
@anchor{library/dis opcode-STORE_FAST}@anchor{3e85}
@deffn {Opcode} STORE_FAST (var_num)

Stores TOS into the local @code{co_varnames[var_num]}.
@end deffn

@geindex DELETE_FAST (opcode)
@anchor{library/dis opcode-DELETE_FAST}@anchor{3ea4}
@deffn {Opcode} DELETE_FAST (var_num)

Deletes local @code{co_varnames[var_num]}.
@end deffn

@geindex MAKE_CELL (opcode)
@anchor{library/dis opcode-MAKE_CELL}@anchor{3ea5}
@deffn {Opcode} MAKE_CELL (i)

Creates a new cell in slot @code{i}.  If that slot is empty then
that value is stored into the new cell.


New in version 3.11.

@end deffn

@geindex LOAD_CLOSURE (opcode)
@anchor{library/dis opcode-LOAD_CLOSURE}@anchor{3ea6}
@deffn {Opcode} LOAD_CLOSURE (i)

Pushes a reference to the cell contained in slot @code{i} of the “fast locals”
storage.  The name of the variable is @code{co_fastlocalnames[i]}.

Note that @code{LOAD_CLOSURE} is effectively an alias for @code{LOAD_FAST}.
It exists to keep bytecode a little more readable.


Changed in version 3.11: @code{i} is no longer offset by the length of @code{co_varnames}.

@end deffn

@geindex LOAD_DEREF (opcode)
@anchor{library/dis opcode-LOAD_DEREF}@anchor{3ea7}
@deffn {Opcode} LOAD_DEREF (i)

Loads the cell contained in slot @code{i} of the “fast locals” storage.
Pushes a reference to the object the cell contains on the stack.


Changed in version 3.11: @code{i} is no longer offset by the length of @code{co_varnames}.

@end deffn

@geindex LOAD_CLASSDEREF (opcode)
@anchor{library/dis opcode-LOAD_CLASSDEREF}@anchor{b90}
@deffn {Opcode} LOAD_CLASSDEREF (i)

Much like @ref{3ea7,,LOAD_DEREF} but first checks the locals dictionary before
consulting the cell.  This is used for loading free variables in class
bodies.


New in version 3.4.



Changed in version 3.11: @code{i} is no longer offset by the length of @code{co_varnames}.

@end deffn

@geindex STORE_DEREF (opcode)
@anchor{library/dis opcode-STORE_DEREF}@anchor{3ea8}
@deffn {Opcode} STORE_DEREF (i)

Stores TOS into the cell contained in slot @code{i} of the “fast locals”
storage.


Changed in version 3.11: @code{i} is no longer offset by the length of @code{co_varnames}.

@end deffn

@geindex DELETE_DEREF (opcode)
@anchor{library/dis opcode-DELETE_DEREF}@anchor{3ea9}
@deffn {Opcode} DELETE_DEREF (i)

Empties the cell contained in slot @code{i} of the “fast locals” storage.
Used by the @ref{152e,,del} statement.


New in version 3.2.



Changed in version 3.11: @code{i} is no longer offset by the length of @code{co_varnames}.

@end deffn

@geindex COPY_FREE_VARS (opcode)
@anchor{library/dis opcode-COPY_FREE_VARS}@anchor{10cd}
@deffn {Opcode} COPY_FREE_VARS (n)

Copies the @code{n} free variables from the closure into the frame.
Removes the need for special code on the caller’s side when calling
closures.


New in version 3.11.

@end deffn

@geindex RAISE_VARARGS (opcode)
@anchor{library/dis opcode-RAISE_VARARGS}@anchor{3eaa}
@deffn {Opcode} RAISE_VARARGS (argc)

Raises an exception using one of the 3 forms of the @code{raise} statement,
depending on the value of @emph{argc}:


@itemize *

@item 
0: @code{raise} (re-raise previous exception)

@item 
1: @code{raise TOS} (raise exception instance or type at @code{TOS})

@item 
2: @code{raise TOS1 from TOS} (raise exception instance or type at @code{TOS1}
with @code{__cause__} set to @code{TOS})
@end itemize
@end deffn

@geindex CALL (opcode)
@anchor{library/dis opcode-CALL}@anchor{1a1}
@deffn {Opcode} CALL (argc)

Calls a callable object with the number of arguments specified by @code{argc},
including the named arguments specified by the preceding
@ref{1a2,,KW_NAMES}, if any.
On the stack are (in ascending order), either:


@itemize *

@item 
NULL

@item 
The callable

@item 
The positional arguments

@item 
The named arguments
@end itemize

or:


@itemize *

@item 
The callable

@item 
@code{self}

@item 
The remaining positional arguments

@item 
The named arguments
@end itemize

@code{argc} is the total of the positional and named arguments, excluding
@code{self} when a @code{NULL} is not present.

@code{CALL} pops all arguments and the callable object off the stack,
calls the callable object with those arguments, and pushes the return value
returned by the callable object.


New in version 3.11.

@end deffn

@geindex CALL_FUNCTION_EX (opcode)
@anchor{library/dis opcode-CALL_FUNCTION_EX}@anchor{8ad}
@deffn {Opcode} CALL_FUNCTION_EX (flags)

Calls a callable object with variable set of positional and keyword
arguments.  If the lowest bit of @emph{flags} is set, the top of the stack
contains a mapping object containing additional keyword arguments.
Before the callable is called, the mapping object and iterable object
are each “unpacked” and their contents passed in as keyword and
positional arguments respectively.
@code{CALL_FUNCTION_EX} pops all arguments and the callable object off the stack,
calls the callable object with those arguments, and pushes the return value
returned by the callable object.


New in version 3.6.

@end deffn

@geindex LOAD_METHOD (opcode)
@anchor{library/dis opcode-LOAD_METHOD}@anchor{75c}
@deffn {Opcode} LOAD_METHOD (namei)

Loads a method named @code{co_names[namei]} from the TOS object. TOS is popped.
This bytecode distinguishes two cases: if TOS has a method with the correct
name, the bytecode pushes the unbound method and TOS. TOS will be used as
the first argument (@code{self}) by @ref{1a1,,CALL} when calling the
unbound method. Otherwise, @code{NULL} and the object return by the attribute
lookup are pushed.


New in version 3.7.

@end deffn

@geindex PRECALL (opcode)
@anchor{library/dis opcode-PRECALL}@anchor{1a0}
@deffn {Opcode} PRECALL (argc)

Prefixes @ref{1a1,,CALL}. Logically this is a no op.
It exists to enable effective specialization of calls.
@code{argc} is the number of arguments as described in @ref{1a1,,CALL}.


New in version 3.11.

@end deffn

@geindex PUSH_NULL (opcode)
@anchor{library/dis opcode-PUSH_NULL}@anchor{19f}
@deffn {Opcode} PUSH_NULL

@quotation

Pushes a @code{NULL} to the stack.
Used in the call sequence to match the @code{NULL} pushed by
@ref{75c,,LOAD_METHOD} for non-method calls.
@end quotation


New in version 3.11.

@end deffn

@geindex KW_NAMES (opcode)
@anchor{library/dis opcode-KW_NAMES}@anchor{1a2}
@deffn {Opcode} KW_NAMES (i)

Prefixes @ref{1a0,,PRECALL}.
Stores a reference to @code{co_consts[consti]} into an internal variable
for use by @ref{1a1,,CALL}. @code{co_consts[consti]} must be a tuple of strings.


New in version 3.11.

@end deffn

@geindex MAKE_FUNCTION (opcode)
@anchor{library/dis opcode-MAKE_FUNCTION}@anchor{8ac}
@deffn {Opcode} MAKE_FUNCTION (flags)

Pushes a new function object on the stack.  From bottom to top, the consumed
stack must consist of values if the argument carries a specified flag value


@itemize *

@item 
@code{0x01} a tuple of default values for positional-only and
positional-or-keyword parameters in positional order

@item 
@code{0x02} a dictionary of keyword-only parameters’ default values

@item 
@code{0x04} a tuple of strings containing parameters’ annotations

@item 
@code{0x08} a tuple containing cells for free variables, making a closure

@item 
the code associated with the function (at TOS1)

@item 
the @ref{11bf,,qualified name} of the function (at TOS)
@end itemize


Changed in version 3.10: Flag value @code{0x04} is a tuple of strings instead of dictionary

@end deffn

@geindex BUILD_SLICE (opcode)
@anchor{library/dis opcode-BUILD_SLICE}@anchor{3eab}
@deffn {Opcode} BUILD_SLICE (argc)

@geindex built-in function; slice

Pushes a slice object on the stack.  @emph{argc} must be 2 or 3.  If it is 2,
@code{slice(TOS1, TOS)} is pushed; if it is 3, @code{slice(TOS2, TOS1, TOS)} is
pushed. See the @ref{ffb,,slice()} built-in function for more information.
@end deffn

@geindex EXTENDED_ARG (opcode)
@anchor{library/dis opcode-EXTENDED_ARG}@anchor{107d}
@deffn {Opcode} EXTENDED_ARG (ext)

Prefixes any opcode which has an argument too big to fit into the default one
byte. @emph{ext} holds an additional byte which act as higher bits in the argument.
For each opcode, at most three prefixal @code{EXTENDED_ARG} are allowed, forming
an argument from two-byte to four-byte.
@end deffn

@geindex FORMAT_VALUE (opcode)
@anchor{library/dis opcode-FORMAT_VALUE}@anchor{8a9}
@deffn {Opcode} FORMAT_VALUE (flags)

Used for implementing formatted literal strings (f-strings).  Pops
an optional @emph{fmt_spec} from the stack, then a required @emph{value}.
@emph{flags} is interpreted as follows:


@itemize *

@item 
@code{(flags & 0x03) == 0x00}: @emph{value} is formatted as-is.

@item 
@code{(flags & 0x03) == 0x01}: call @ref{2ef,,str()} on @emph{value} before
formatting it.

@item 
@code{(flags & 0x03) == 0x02}: call @ref{27d,,repr()} on @emph{value} before
formatting it.

@item 
@code{(flags & 0x03) == 0x03}: call @ref{f47,,ascii()} on @emph{value} before
formatting it.

@item 
@code{(flags & 0x04) == 0x04}: pop @emph{fmt_spec} from the stack and use
it, else use an empty @emph{fmt_spec}.
@end itemize

Formatting is performed using @code{PyObject_Format()}.  The
result is pushed on the stack.


New in version 3.6.

@end deffn

@geindex MATCH_CLASS (opcode)
@anchor{library/dis opcode-MATCH_CLASS}@anchor{1a3}
@deffn {Opcode} MATCH_CLASS (count)

TOS is a tuple of keyword attribute names, TOS1 is the class being matched
against, and TOS2 is the match subject.  @emph{count} is the number of positional
sub-patterns.

Pop TOS, TOS1, and TOS2.  If TOS2 is an instance of TOS1 and has the
positional and keyword attributes required by @emph{count} and TOS, push a tuple
of extracted attributes.  Otherwise, push @code{None}.


New in version 3.10.



Changed in version 3.11: Previously, this instruction also pushed a boolean value indicating
success (@code{True}) or failure (@code{False}).

@end deffn

@geindex RESUME (opcode)
@anchor{library/dis opcode-RESUME}@anchor{1083}
@deffn {Opcode} RESUME (where)

@quotation

A no-op. Performs internal tracing, debugging and optimization checks.

The @code{where} operand marks where the @code{RESUME} occurs:


@itemize *

@item 
@code{0} The start of a function

@item 
@code{1} After a @code{yield} expression

@item 
@code{2} After a @code{yield from} expression

@item 
@code{3} After an @code{await} expression
@end itemize
@end quotation


New in version 3.11.

@end deffn

@geindex RETURN_GENERATOR (opcode)
@anchor{library/dis opcode-RETURN_GENERATOR}@anchor{3eac}
@deffn {Opcode} RETURN_GENERATOR

Create a generator, coroutine, or async generator from the current frame.
Clear the current frame and return the newly created generator.


New in version 3.11.

@end deffn

@geindex SEND (opcode)
@anchor{library/dis opcode-SEND}@anchor{3ead}
@deffn {Opcode} SEND

Sends @code{None} to the sub-generator of this generator.
Used in @code{yield from} and @code{await} statements.


New in version 3.11.

@end deffn

@geindex ASYNC_GEN_WRAP (opcode)
@anchor{library/dis opcode-ASYNC_GEN_WRAP}@anchor{3eae}
@deffn {Opcode} ASYNC_GEN_WRAP

Wraps the value on top of the stack in an @code{async_generator_wrapped_value}.
Used to yield in async generators.


New in version 3.11.

@end deffn

@geindex HAVE_ARGUMENT (opcode)
@anchor{library/dis opcode-HAVE_ARGUMENT}@anchor{3eaf}
@deffn {Opcode} HAVE_ARGUMENT

This is not really an opcode.  It identifies the dividing line between
opcodes which don’t use their argument and those that do
(@code{< HAVE_ARGUMENT} and @code{>= HAVE_ARGUMENT}, respectively).


Changed in version 3.6: Now every instruction has an argument, but opcodes @code{< HAVE_ARGUMENT}
ignore it. Before, only opcodes @code{>= HAVE_ARGUMENT} had an argument.

@end deffn

@node Opcode collections,,Python Bytecode Instructions,dis — Disassembler for Python bytecode
@anchor{library/dis id1}@anchor{3eb0}@anchor{library/dis opcode-collections}@anchor{3e59}
@subsubsection Opcode collections


These collections are provided for automatic introspection of bytecode
instructions:

@geindex opname (in module dis)
@anchor{library/dis dis opname}@anchor{3eb1}
@deffn {Data} dis.opname

Sequence of operation names, indexable using the bytecode.
@end deffn

@geindex opmap (in module dis)
@anchor{library/dis dis opmap}@anchor{3eb2}
@deffn {Data} dis.opmap

Dictionary mapping operation names to bytecodes.
@end deffn

@geindex cmp_op (in module dis)
@anchor{library/dis dis cmp_op}@anchor{3eb3}
@deffn {Data} dis.cmp_op

Sequence of all compare operation names.
@end deffn

@geindex hasconst (in module dis)
@anchor{library/dis dis hasconst}@anchor{3eb4}
@deffn {Data} dis.hasconst

Sequence of bytecodes that access a constant.
@end deffn

@geindex hasfree (in module dis)
@anchor{library/dis dis hasfree}@anchor{3eb5}
@deffn {Data} dis.hasfree

Sequence of bytecodes that access a free variable (note that ‘free’ in this
context refers to names in the current scope that are referenced by inner
scopes or names in outer scopes that are referenced from this scope.  It does
@emph{not} include references to global or builtin scopes).
@end deffn

@geindex hasname (in module dis)
@anchor{library/dis dis hasname}@anchor{3eb6}
@deffn {Data} dis.hasname

Sequence of bytecodes that access an attribute by name.
@end deffn

@geindex hasjrel (in module dis)
@anchor{library/dis dis hasjrel}@anchor{3eb7}
@deffn {Data} dis.hasjrel

Sequence of bytecodes that have a relative jump target.
@end deffn

@geindex hasjabs (in module dis)
@anchor{library/dis dis hasjabs}@anchor{3eb8}
@deffn {Data} dis.hasjabs

Sequence of bytecodes that have an absolute jump target.
@end deffn

@geindex haslocal (in module dis)
@anchor{library/dis dis haslocal}@anchor{3eb9}
@deffn {Data} dis.haslocal

Sequence of bytecodes that access a local variable.
@end deffn

@geindex hascompare (in module dis)
@anchor{library/dis dis hascompare}@anchor{3eba}
@deffn {Data} dis.hascompare

Sequence of bytecodes of Boolean operations.
@end deffn

@node pickletools — Tools for pickle developers,,dis — Disassembler for Python bytecode,Python Language Services
@anchor{library/pickletools doc}@anchor{3ebb}@anchor{library/pickletools module-pickletools}@anchor{c6}@anchor{library/pickletools pickletools-tools-for-pickle-developers}@anchor{3ebc}
@subsection @code{pickletools} — Tools for pickle developers


@geindex module; pickletools

@strong{Source code:} Lib/pickletools.py@footnote{https://github.com/python/cpython/tree/main/Lib/pickletools.py}


__________________________________________________________________


This module contains various constants relating to the intimate details of the
@ref{c5,,pickle} module, some lengthy comments about the implementation, and a
few useful functions for analyzing pickled data.  The contents of this module
are useful for Python core developers who are working on the @ref{c5,,pickle};
ordinary users of the @ref{c5,,pickle} module probably won’t find the
@ref{c6,,pickletools} module relevant.

@menu
* Command line usage: Command line usage<2>. 
* Programmatic Interface: Programmatic Interface<2>. 

@end menu

@node Command line usage<2>,Programmatic Interface<2>,,pickletools — Tools for pickle developers
@anchor{library/pickletools command-line-usage}@anchor{3ebd}
@subsubsection Command line usage



New in version 3.2.


When invoked from the command line, @code{python -m pickletools} will
disassemble the contents of one or more pickle files.  Note that if
you want to see the Python object stored in the pickle rather than the
details of pickle format, you may want to use @code{-m pickle} instead.
However, when the pickle file that you want to examine comes from an
untrusted source, @code{-m pickletools} is a safer option because it does
not execute pickle bytecode.

For example, with a tuple @code{(1, 2)} pickled in file @code{x.pickle}:

@example
$ python -m pickle x.pickle
(1, 2)

$ python -m pickletools x.pickle
    0: \x80 PROTO      3
    2: K    BININT1    1
    4: K    BININT1    2
    6: \x86 TUPLE2
    7: q    BINPUT     0
    9: .    STOP
highest protocol among opcodes = 2
@end example

@menu
* Command line options: Command line options<3>. 

@end menu

@node Command line options<3>,,,Command line usage<2>
@anchor{library/pickletools command-line-options}@anchor{3ebe}
@subsubsection Command line options


@geindex pickletools command line option; -a
@geindex pickletools command line option; --annotate
@anchor{library/pickletools cmdoption-pickletools-a}@anchor{3ebf}@anchor{library/pickletools cmdoption-pickletools-annotate}@anchor{3ec0}
@deffn {Program Option} @w{-}a, @w{-}@w{-}annotate

Annotate each line with a short opcode description.
@end deffn

@geindex pickletools command line option; -o
@geindex pickletools command line option; --output=<file>
@anchor{library/pickletools cmdoption-pickletools-o}@anchor{3ec1}@anchor{library/pickletools cmdoption-pickletools-output}@anchor{3ec2}
@deffn {Program Option} @w{-}o, @w{-}@w{-}output=<file>

Name of a file where the output should be written.
@end deffn

@geindex pickletools command line option; -l
@geindex pickletools command line option; --indentlevel=<num>
@anchor{library/pickletools cmdoption-pickletools-l}@anchor{3ec3}@anchor{library/pickletools cmdoption-pickletools-indentlevel}@anchor{3ec4}
@deffn {Program Option} @w{-}l, @w{-}@w{-}indentlevel=<num>

The number of blanks by which to indent a new MARK level.
@end deffn

@geindex pickletools command line option; -m
@geindex pickletools command line option; --memo
@anchor{library/pickletools cmdoption-pickletools-m}@anchor{3ec5}@anchor{library/pickletools cmdoption-pickletools-memo}@anchor{3ec6}
@deffn {Program Option} @w{-}m, @w{-}@w{-}memo

When multiple objects are disassembled, preserve memo between
disassemblies.
@end deffn

@geindex pickletools command line option; -p
@geindex pickletools command line option; --preamble=<preamble>
@anchor{library/pickletools cmdoption-pickletools-p}@anchor{3ec7}@anchor{library/pickletools cmdoption-pickletools-preamble}@anchor{3ec8}
@deffn {Program Option} @w{-}p, @w{-}@w{-}preamble=<preamble>

When more than one pickle file are specified, print given preamble
before each disassembly.
@end deffn

@node Programmatic Interface<2>,,Command line usage<2>,pickletools — Tools for pickle developers
@anchor{library/pickletools programmatic-interface}@anchor{3ec9}
@subsubsection Programmatic Interface


@geindex dis() (in module pickletools)
@anchor{library/pickletools pickletools dis}@anchor{80c}
@deffn {Function} pickletools.dis (pickle, out=None, memo=None, indentlevel=4, annotate=0)

Outputs a symbolic disassembly of the pickle to the file-like
object @emph{out}, defaulting to @code{sys.stdout}.  @emph{pickle} can be a
string or a file-like object.  @emph{memo} can be a Python dictionary
that will be used as the pickle’s memo; it can be used to perform
disassemblies across multiple pickles created by the same
pickler. Successive levels, indicated by @code{MARK} opcodes in the
stream, are indented by @emph{indentlevel} spaces.  If a nonzero value
is given to @emph{annotate}, each opcode in the output is annotated with
a short description.  The value of @emph{annotate} is used as a hint for
the column where annotation should start.


New in version 3.2: The @emph{annotate} argument.

@end deffn

@geindex genops() (in module pickletools)
@anchor{library/pickletools pickletools genops}@anchor{3eca}
@deffn {Function} pickletools.genops (pickle)

Provides an @ref{11c3,,iterator} over all of the opcodes in a pickle, returning a
sequence of @code{(opcode, arg, pos)} triples.  @emph{opcode} is an instance of an
@code{OpcodeInfo} class; @emph{arg} is the decoded value, as a Python object, of
the opcode’s argument; @emph{pos} is the position at which this opcode is located.
@emph{pickle} can be a string or a file-like object.
@end deffn

@geindex optimize() (in module pickletools)
@anchor{library/pickletools pickletools optimize}@anchor{2151}
@deffn {Function} pickletools.optimize (picklestring)

Returns a new equivalent pickle string after eliminating unused @code{PUT}
opcodes. The optimized pickle is shorter, takes less transmission time,
requires less storage space, and unpickles more efficiently.
@end deffn

@node MS Windows Specific Services,Unix Specific Services,Python Language Services,The Python Standard Library
@anchor{library/windows doc}@anchor{3ecb}@anchor{library/windows ms-windows-specific-services}@anchor{3ecc}@anchor{library/windows mswin-specific-services}@anchor{16cb}
@section MS Windows Specific Services


This chapter describes modules that are only available on MS Windows platforms.

@menu
* msilib — Read and write Microsoft Installer files:: 
* msvcrt — Useful routines from the MS VC++ runtime:: 
* winreg — Windows registry access:: 
* winsound — Sound-playing interface for Windows:: 

@end menu

@node msilib — Read and write Microsoft Installer files,msvcrt — Useful routines from the MS VC++ runtime,,MS Windows Specific Services
@anchor{library/msilib doc}@anchor{3ecd}@anchor{library/msilib module-msilib}@anchor{b1}@anchor{library/msilib msilib-read-and-write-microsoft-installer-files}@anchor{3ece}
@subsection @code{msilib} — Read and write Microsoft Installer files


@geindex module; msilib

@strong{Source code:} Lib/msilib/__init__.py@footnote{https://github.com/python/cpython/tree/main/Lib/msilib/__init__.py}

@geindex msi


__________________________________________________________________


The @ref{b1,,msilib} supports the creation of Microsoft Installer (@code{.msi}) files.
Because these files often contain an embedded “cabinet” file (@code{.cab}), it also
exposes an API to create CAB files. Support for reading @code{.cab} files is
currently not implemented; read support for the @code{.msi} database is possible.

This package aims to provide complete access to all tables in an @code{.msi} file,
therefore, it is a fairly low-level API. One primary application of this
package is the creation of Python installer package itself (although that currently
uses a different version of @code{msilib}).

The package contents can be roughly split into four parts: low-level CAB
routines, low-level MSI routines, higher-level MSI routines, and standard table
structures.

@geindex FCICreate() (in module msilib)
@anchor{library/msilib msilib FCICreate}@anchor{3ecf}
@deffn {Function} msilib.FCICreate (cabname, files)

Create a new CAB file named @emph{cabname}. @emph{files} must be a list of tuples, each
containing the name of the file on disk, and the name of the file inside the CAB
file.

The files are added to the CAB file in the order they appear in the list. All
files are added into a single CAB file, using the MSZIP compression algorithm.

Callbacks to Python for the various steps of MSI creation are currently not
exposed.
@end deffn

@geindex UuidCreate() (in module msilib)
@anchor{library/msilib msilib UuidCreate}@anchor{3ed0}
@deffn {Function} msilib.UuidCreate ()

Return the string representation of a new unique identifier. This wraps the
Windows API functions @code{UuidCreate()} and @code{UuidToString()}.
@end deffn

@geindex OpenDatabase() (in module msilib)
@anchor{library/msilib msilib OpenDatabase}@anchor{11d5}
@deffn {Function} msilib.OpenDatabase (path, persist)

Return a new database object by calling MsiOpenDatabase.   @emph{path} is the file
name of the MSI file; @emph{persist} can be one of the constants
@code{MSIDBOPEN_CREATEDIRECT}, @code{MSIDBOPEN_CREATE}, @code{MSIDBOPEN_DIRECT},
@code{MSIDBOPEN_READONLY}, or @code{MSIDBOPEN_TRANSACT}, and may include the flag
@code{MSIDBOPEN_PATCHFILE}. See the Microsoft documentation for the meaning of
these flags; depending on the flags, an existing database is opened, or a new
one created.
@end deffn

@geindex CreateRecord() (in module msilib)
@anchor{library/msilib msilib CreateRecord}@anchor{3ed1}
@deffn {Function} msilib.CreateRecord (count)

Return a new record object by calling @code{MSICreateRecord()}. @emph{count} is the
number of fields of the record.
@end deffn

@geindex init_database() (in module msilib)
@anchor{library/msilib msilib init_database}@anchor{11d6}
@deffn {Function} msilib.init_database (name, schema, ProductName, ProductCode, ProductVersion, Manufacturer)

Create and return a new database @emph{name}, initialize it with @emph{schema}, and set
the properties @emph{ProductName}, @emph{ProductCode}, @emph{ProductVersion}, and
@emph{Manufacturer}.

@emph{schema} must be a module object containing @code{tables} and
@code{_Validation_records} attributes; typically, @ref{3ed2,,msilib.schema} should be
used.

The database will contain just the schema and the validation records when this
function returns.
@end deffn

@geindex add_data() (in module msilib)
@anchor{library/msilib msilib add_data}@anchor{3ed3}
@deffn {Function} msilib.add_data (database, table, records)

Add all @emph{records} to the table named @emph{table} in @emph{database}.

The @emph{table} argument must be one of the predefined tables in the MSI schema,
e.g. @code{'Feature'}, @code{'File'}, @code{'Component'}, @code{'Dialog'}, @code{'Control'},
etc.

@emph{records} should be a list of tuples, each one containing all fields of a
record according to the schema of the table.  For optional fields,
@code{None} can be passed.

Field values can be ints, strings, or instances of the Binary class.
@end deffn

@geindex Binary (class in msilib)
@anchor{library/msilib msilib Binary}@anchor{3ed4}
@deffn {Class} msilib.Binary (filename)

Represents entries in the Binary table; inserting such an object using
@ref{3ed3,,add_data()} reads the file named @emph{filename} into the table.
@end deffn

@geindex add_tables() (in module msilib)
@anchor{library/msilib msilib add_tables}@anchor{3ed5}
@deffn {Function} msilib.add_tables (database, module)

Add all table content from @emph{module} to @emph{database}. @emph{module} must contain an
attribute @emph{tables} listing all tables for which content should be added, and one
attribute per table that has the actual content.

This is typically used to install the sequence tables.
@end deffn

@geindex add_stream() (in module msilib)
@anchor{library/msilib msilib add_stream}@anchor{3ed6}
@deffn {Function} msilib.add_stream (database, name, path)

Add the file @emph{path} into the @code{_Stream} table of @emph{database}, with the stream
name @emph{name}.
@end deffn

@geindex gen_uuid() (in module msilib)
@anchor{library/msilib msilib gen_uuid}@anchor{3ed7}
@deffn {Function} msilib.gen_uuid ()

Return a new UUID, in the format that MSI typically requires (i.e. in curly
braces, and with all hexdigits in uppercase).
@end deffn


@subsubheading See also


FCICreate@footnote{https://msdn.microsoft.com/en-us/library/bb432265.aspx}
UuidCreate@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa379205.aspx}
UuidToString@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa379352.aspx}


@menu
* Database Objects:: 
* View Objects:: 
* Summary Information Objects:: 
* Record Objects:: 
* Errors:: 
* CAB Objects:: 
* Directory Objects:: 
* Features: Features<3>. 
* GUI classes:: 
* Precomputed tables:: 

@end menu

@node Database Objects,View Objects,,msilib — Read and write Microsoft Installer files
@anchor{library/msilib database-objects}@anchor{3ed8}@anchor{library/msilib id1}@anchor{3ed9}
@subsubsection Database Objects


@geindex OpenView() (msilib.Database method)
@anchor{library/msilib msilib Database OpenView}@anchor{11d7}
@deffn {Method} Database.OpenView (sql)

Return a view object, by calling @code{MSIDatabaseOpenView()}. @emph{sql} is the SQL
statement to execute.
@end deffn

@geindex Commit() (msilib.Database method)
@anchor{library/msilib msilib Database Commit}@anchor{3eda}
@deffn {Method} Database.Commit ()

Commit the changes pending in the current transaction, by calling
@code{MSIDatabaseCommit()}.
@end deffn

@geindex GetSummaryInformation() (msilib.Database method)
@anchor{library/msilib msilib Database GetSummaryInformation}@anchor{3edb}
@deffn {Method} Database.GetSummaryInformation (count)

Return a new summary information object, by calling
@code{MsiGetSummaryInformation()}.  @emph{count} is the maximum number of updated
values.
@end deffn

@geindex Close() (msilib.Database method)
@anchor{library/msilib msilib Database Close}@anchor{67d}
@deffn {Method} Database.Close ()

Close the database object, through @code{MsiCloseHandle()}.


New in version 3.7.

@end deffn


@subsubheading See also


MSIDatabaseOpenView@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370082.aspx}
MSIDatabaseCommit@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370075.aspx}
MSIGetSummaryInformation@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370301.aspx}
MsiCloseHandle@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370067.aspx}


@node View Objects,Summary Information Objects,Database Objects,msilib — Read and write Microsoft Installer files
@anchor{library/msilib id2}@anchor{3edc}@anchor{library/msilib view-objects}@anchor{3edd}
@subsubsection View Objects


@geindex Execute() (msilib.View method)
@anchor{library/msilib msilib View Execute}@anchor{3ede}
@deffn {Method} View.Execute (params)

Execute the SQL query of the view, through @code{MSIViewExecute()}. If
@emph{params} is not @code{None}, it is a record describing actual values of the
parameter tokens in the query.
@end deffn

@geindex GetColumnInfo() (msilib.View method)
@anchor{library/msilib msilib View GetColumnInfo}@anchor{3edf}
@deffn {Method} View.GetColumnInfo (kind)

Return a record describing the columns of the view, through calling
@code{MsiViewGetColumnInfo()}. @emph{kind} can be either @code{MSICOLINFO_NAMES} or
@code{MSICOLINFO_TYPES}.
@end deffn

@geindex Fetch() (msilib.View method)
@anchor{library/msilib msilib View Fetch}@anchor{3ee0}
@deffn {Method} View.Fetch ()

Return a result record of the query, through calling @code{MsiViewFetch()}.
@end deffn

@geindex Modify() (msilib.View method)
@anchor{library/msilib msilib View Modify}@anchor{3ee1}
@deffn {Method} View.Modify (kind, data)

Modify the view, by calling @code{MsiViewModify()}. @emph{kind} can be one of
@code{MSIMODIFY_SEEK}, @code{MSIMODIFY_REFRESH}, @code{MSIMODIFY_INSERT},
@code{MSIMODIFY_UPDATE}, @code{MSIMODIFY_ASSIGN}, @code{MSIMODIFY_REPLACE},
@code{MSIMODIFY_MERGE}, @code{MSIMODIFY_DELETE}, @code{MSIMODIFY_INSERT_TEMPORARY},
@code{MSIMODIFY_VALIDATE}, @code{MSIMODIFY_VALIDATE_NEW},
@code{MSIMODIFY_VALIDATE_FIELD}, or @code{MSIMODIFY_VALIDATE_DELETE}.

@emph{data} must be a record describing the new data.
@end deffn

@geindex Close() (msilib.View method)
@anchor{library/msilib msilib View Close}@anchor{3ee2}
@deffn {Method} View.Close ()

Close the view, through @code{MsiViewClose()}.
@end deffn


@subsubheading See also


MsiViewExecute@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370513.aspx}
MSIViewGetColumnInfo@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370516.aspx}
MsiViewFetch@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370514.aspx}
MsiViewModify@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370519.aspx}
MsiViewClose@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370510.aspx}


@node Summary Information Objects,Record Objects,View Objects,msilib — Read and write Microsoft Installer files
@anchor{library/msilib summary-information-objects}@anchor{3ee3}@anchor{library/msilib summary-objects}@anchor{3ee4}
@subsubsection Summary Information Objects


@geindex GetProperty() (msilib.SummaryInformation method)
@anchor{library/msilib msilib SummaryInformation GetProperty}@anchor{1292}
@deffn {Method} SummaryInformation.GetProperty (field)

Return a property of the summary, through @code{MsiSummaryInfoGetProperty()}.
@emph{field} is the name of the property, and can be one of the constants
@code{PID_CODEPAGE}, @code{PID_TITLE}, @code{PID_SUBJECT}, @code{PID_AUTHOR},
@code{PID_KEYWORDS}, @code{PID_COMMENTS}, @code{PID_TEMPLATE}, @code{PID_LASTAUTHOR},
@code{PID_REVNUMBER}, @code{PID_LASTPRINTED}, @code{PID_CREATE_DTM},
@code{PID_LASTSAVE_DTM}, @code{PID_PAGECOUNT}, @code{PID_WORDCOUNT}, @code{PID_CHARCOUNT},
@code{PID_APPNAME}, or @code{PID_SECURITY}.
@end deffn

@geindex GetPropertyCount() (msilib.SummaryInformation method)
@anchor{library/msilib msilib SummaryInformation GetPropertyCount}@anchor{3ee5}
@deffn {Method} SummaryInformation.GetPropertyCount ()

Return the number of summary properties, through
@code{MsiSummaryInfoGetPropertyCount()}.
@end deffn

@geindex SetProperty() (msilib.SummaryInformation method)
@anchor{library/msilib msilib SummaryInformation SetProperty}@anchor{3ee6}
@deffn {Method} SummaryInformation.SetProperty (field, value)

Set a property through @code{MsiSummaryInfoSetProperty()}. @emph{field} can have the
same values as in @ref{1292,,GetProperty()}, @emph{value} is the new value of the property.
Possible value types are integer and string.
@end deffn

@geindex Persist() (msilib.SummaryInformation method)
@anchor{library/msilib msilib SummaryInformation Persist}@anchor{3ee7}
@deffn {Method} SummaryInformation.Persist ()

Write the modified properties to the summary information stream, using
@code{MsiSummaryInfoPersist()}.
@end deffn


@subsubheading See also


MsiSummaryInfoGetProperty@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370409.aspx}
MsiSummaryInfoGetPropertyCount@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370488.aspx}
MsiSummaryInfoSetProperty@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370491.aspx}
MsiSummaryInfoPersist@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370490.aspx}


@node Record Objects,Errors,Summary Information Objects,msilib — Read and write Microsoft Installer files
@anchor{library/msilib id3}@anchor{3ee8}@anchor{library/msilib record-objects}@anchor{3ee9}
@subsubsection Record Objects


@geindex GetFieldCount() (msilib.Record method)
@anchor{library/msilib msilib Record GetFieldCount}@anchor{3eea}
@deffn {Method} Record.GetFieldCount ()

Return the number of fields of the record, through
@code{MsiRecordGetFieldCount()}.
@end deffn

@geindex GetInteger() (msilib.Record method)
@anchor{library/msilib msilib Record GetInteger}@anchor{3eeb}
@deffn {Method} Record.GetInteger (field)

Return the value of @emph{field} as an integer where possible.  @emph{field} must
be an integer.
@end deffn

@geindex GetString() (msilib.Record method)
@anchor{library/msilib msilib Record GetString}@anchor{3eec}
@deffn {Method} Record.GetString (field)

Return the value of @emph{field} as a string where possible.  @emph{field} must
be an integer.
@end deffn

@geindex SetString() (msilib.Record method)
@anchor{library/msilib msilib Record SetString}@anchor{3eed}
@deffn {Method} Record.SetString (field, value)

Set @emph{field} to @emph{value} through @code{MsiRecordSetString()}. @emph{field} must be an
integer; @emph{value} a string.
@end deffn

@geindex SetStream() (msilib.Record method)
@anchor{library/msilib msilib Record SetStream}@anchor{3eee}
@deffn {Method} Record.SetStream (field, value)

Set @emph{field} to the contents of the file named @emph{value}, through
@code{MsiRecordSetStream()}. @emph{field} must be an integer; @emph{value} a string.
@end deffn

@geindex SetInteger() (msilib.Record method)
@anchor{library/msilib msilib Record SetInteger}@anchor{3eef}
@deffn {Method} Record.SetInteger (field, value)

Set @emph{field} to @emph{value} through @code{MsiRecordSetInteger()}. Both @emph{field} and
@emph{value} must be an integer.
@end deffn

@geindex ClearData() (msilib.Record method)
@anchor{library/msilib msilib Record ClearData}@anchor{3ef0}
@deffn {Method} Record.ClearData ()

Set all fields of the record to 0, through @code{MsiRecordClearData()}.
@end deffn


@subsubheading See also


MsiRecordGetFieldCount@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370366.aspx}
MsiRecordSetString@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370373.aspx}
MsiRecordSetStream@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370372.aspx}
MsiRecordSetInteger@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370371.aspx}
MsiRecordClearData@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370364.aspx}


@node Errors,CAB Objects,Record Objects,msilib — Read and write Microsoft Installer files
@anchor{library/msilib errors}@anchor{3ef1}@anchor{library/msilib msi-errors}@anchor{3ef2}
@subsubsection Errors


All wrappers around MSI functions raise @code{MSIError}; the string inside the
exception will contain more detail.

@node CAB Objects,Directory Objects,Errors,msilib — Read and write Microsoft Installer files
@anchor{library/msilib cab}@anchor{3ef3}@anchor{library/msilib cab-objects}@anchor{3ef4}
@subsubsection CAB Objects


@geindex CAB (class in msilib)
@anchor{library/msilib msilib CAB}@anchor{3ef5}
@deffn {Class} msilib.CAB (name)

The class @ref{3ef5,,CAB} represents a CAB file. During MSI construction, files
will be added simultaneously to the @code{Files} table, and to a CAB file. Then,
when all files have been added, the CAB file can be written, then added to the
MSI file.

@emph{name} is the name of the CAB file in the MSI file.

@geindex append() (msilib.CAB method)
@anchor{library/msilib msilib CAB append}@anchor{3ef6}
@deffn {Method} append (full, file, logical)

Add the file with the pathname @emph{full} to the CAB file, under the name
@emph{logical}.  If there is already a file named @emph{logical}, a new file name is
created.

Return the index of the file in the CAB file, and the new name of the file
inside the CAB file.
@end deffn

@geindex commit() (msilib.CAB method)
@anchor{library/msilib msilib CAB commit}@anchor{3ef7}
@deffn {Method} commit (database)

Generate a CAB file, add it as a stream to the MSI file, put it into the
@code{Media} table, and remove the generated file from the disk.
@end deffn
@end deffn

@node Directory Objects,Features<3>,CAB Objects,msilib — Read and write Microsoft Installer files
@anchor{library/msilib directory-objects}@anchor{3ef8}@anchor{library/msilib msi-directory}@anchor{3ef9}
@subsubsection Directory Objects


@geindex Directory (class in msilib)
@anchor{library/msilib msilib Directory}@anchor{3efa}
@deffn {Class} msilib.Directory (database, cab, basedir, physical, logical, default[, componentflags])

Create a new directory in the Directory table. There is a current component at
each point in time for the directory, which is either explicitly created through
@ref{1293,,start_component()}, or implicitly when files are added for the first time.
Files are added into the current component, and into the cab file.  To create a
directory, a base directory object needs to be specified (can be @code{None}), the
path to the physical directory, and a logical directory name.  @emph{default}
specifies the DefaultDir slot in the directory table. @emph{componentflags} specifies
the default flags that new components get.

@geindex start_component() (msilib.Directory method)
@anchor{library/msilib msilib Directory start_component}@anchor{1293}
@deffn {Method} start_component (component=None, feature=None, flags=None, keyfile=None, uuid=None)

Add an entry to the Component table, and make this component the current
component for this directory. If no component name is given, the directory
name is used. If no @emph{feature} is given, the current feature is used. If no
@emph{flags} are given, the directory’s default flags are used. If no @emph{keyfile}
is given, the KeyPath is left null in the Component table.
@end deffn

@geindex add_file() (msilib.Directory method)
@anchor{library/msilib msilib Directory add_file}@anchor{3efb}
@deffn {Method} add_file (file, src=None, version=None, language=None)

Add a file to the current component of the directory, starting a new one
if there is no current component. By default, the file name in the source
and the file table will be identical. If the @emph{src} file is specified, it
is interpreted relative to the current directory. Optionally, a @emph{version}
and a @emph{language} can be specified for the entry in the File table.
@end deffn

@geindex glob() (msilib.Directory method)
@anchor{library/msilib msilib Directory glob}@anchor{3efc}
@deffn {Method} glob (pattern, exclude=None)

Add a list of files to the current component as specified in the glob
pattern.  Individual files can be excluded in the @emph{exclude} list.
@end deffn

@geindex remove_pyc() (msilib.Directory method)
@anchor{library/msilib msilib Directory remove_pyc}@anchor{3efd}
@deffn {Method} remove_pyc ()

Remove @code{.pyc} files on uninstall.
@end deffn
@end deffn


@subsubheading See also


Directory Table@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa368295.aspx}
File Table@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa368596.aspx}
Component Table@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa368007.aspx}
FeatureComponents Table@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa368579.aspx}


@node Features<3>,GUI classes,Directory Objects,msilib — Read and write Microsoft Installer files
@anchor{library/msilib features}@anchor{3efe}@anchor{library/msilib id4}@anchor{3eff}
@subsubsection Features


@geindex Feature (class in msilib)
@anchor{library/msilib msilib Feature}@anchor{3f00}
@deffn {Class} msilib.Feature (db, id, title, desc, display, level=1, parent=None, directory=None, attributes=0)

Add a new record to the @code{Feature} table, using the values @emph{id}, @emph{parent.id},
@emph{title}, @emph{desc}, @emph{display}, @emph{level}, @emph{directory}, and @emph{attributes}. The
resulting feature object can be passed to the @code{start_component()} method of
@ref{3efa,,Directory}.

@geindex set_current() (msilib.Feature method)
@anchor{library/msilib msilib Feature set_current}@anchor{3f01}
@deffn {Method} set_current ()

Make this feature the current feature of @ref{b1,,msilib}. New components are
automatically added to the default feature, unless a feature is explicitly
specified.
@end deffn
@end deffn


@subsubheading See also


Feature Table@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa368585.aspx}


@node GUI classes,Precomputed tables,Features<3>,msilib — Read and write Microsoft Installer files
@anchor{library/msilib gui-classes}@anchor{3f02}@anchor{library/msilib msi-gui}@anchor{3f03}
@subsubsection GUI classes


@ref{b1,,msilib} provides several classes that wrap the GUI tables in an MSI
database. However, no standard user interface is provided.

@geindex Control (class in msilib)
@anchor{library/msilib msilib Control}@anchor{3f04}
@deffn {Class} msilib.Control (dlg, name)

Base class of the dialog controls. @emph{dlg} is the dialog object the control
belongs to, and @emph{name} is the control’s name.

@geindex event() (msilib.Control method)
@anchor{library/msilib msilib Control event}@anchor{3f05}
@deffn {Method} event (event, argument, condition=1, ordering=None)

Make an entry into the @code{ControlEvent} table for this control.
@end deffn

@geindex mapping() (msilib.Control method)
@anchor{library/msilib msilib Control mapping}@anchor{3f06}
@deffn {Method} mapping (event, attribute)

Make an entry into the @code{EventMapping} table for this control.
@end deffn

@geindex condition() (msilib.Control method)
@anchor{library/msilib msilib Control condition}@anchor{3f07}
@deffn {Method} condition (action, condition)

Make an entry into the @code{ControlCondition} table for this control.
@end deffn
@end deffn

@geindex RadioButtonGroup (class in msilib)
@anchor{library/msilib msilib RadioButtonGroup}@anchor{3f08}
@deffn {Class} msilib.RadioButtonGroup (dlg, name, property)

Create a radio button control named @emph{name}. @emph{property} is the installer property
that gets set when a radio button is selected.

@geindex add() (msilib.RadioButtonGroup method)
@anchor{library/msilib msilib RadioButtonGroup add}@anchor{3f09}
@deffn {Method} add (name, x, y, width, height, text, value=None)

Add a radio button named @emph{name} to the group, at the coordinates @emph{x}, @emph{y},
@emph{width}, @emph{height}, and with the label @emph{text}. If @emph{value} is @code{None}, it
defaults to @emph{name}.
@end deffn
@end deffn

@geindex Dialog (class in msilib)
@anchor{library/msilib msilib Dialog}@anchor{3f0a}
@deffn {Class} msilib.Dialog (db, name, x, y, w, h, attr, title, first, default, cancel)

Return a new @ref{3f0a,,Dialog} object. An entry in the @code{Dialog} table is made,
with the specified coordinates, dialog attributes, title, name of the first,
default, and cancel controls.

@geindex control() (msilib.Dialog method)
@anchor{library/msilib msilib Dialog control}@anchor{3f0b}
@deffn {Method} control (name, type, x, y, width, height, attributes, property, text, control_next, help)

Return a new @ref{3f04,,Control} object. An entry in the @code{Control} table is
made with the specified parameters.

This is a generic method; for specific types, specialized methods are
provided.
@end deffn

@geindex text() (msilib.Dialog method)
@anchor{library/msilib msilib Dialog text}@anchor{3f0c}
@deffn {Method} text (name, x, y, width, height, attributes, text)

Add and return a @code{Text} control.
@end deffn

@geindex bitmap() (msilib.Dialog method)
@anchor{library/msilib msilib Dialog bitmap}@anchor{3f0d}
@deffn {Method} bitmap (name, x, y, width, height, text)

Add and return a @code{Bitmap} control.
@end deffn

@geindex line() (msilib.Dialog method)
@anchor{library/msilib msilib Dialog line}@anchor{3f0e}
@deffn {Method} line (name, x, y, width, height)

Add and return a @code{Line} control.
@end deffn

@geindex pushbutton() (msilib.Dialog method)
@anchor{library/msilib msilib Dialog pushbutton}@anchor{3f0f}
@deffn {Method} pushbutton (name, x, y, width, height, attributes, text, next_control)

Add and return a @code{PushButton} control.
@end deffn

@geindex radiogroup() (msilib.Dialog method)
@anchor{library/msilib msilib Dialog radiogroup}@anchor{3f10}
@deffn {Method} radiogroup (name, x, y, width, height, attributes, property, text, next_control)

Add and return a @code{RadioButtonGroup} control.
@end deffn

@geindex checkbox() (msilib.Dialog method)
@anchor{library/msilib msilib Dialog checkbox}@anchor{3f11}
@deffn {Method} checkbox (name, x, y, width, height, attributes, property, text, next_control)

Add and return a @code{CheckBox} control.
@end deffn
@end deffn


@subsubheading See also


Dialog Table@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa368286.aspx}
Control Table@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa368044.aspx}
Control Types@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa368039.aspx}
ControlCondition Table@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa368035.aspx}
ControlEvent Table@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa368037.aspx}
EventMapping Table@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa368559.aspx}
RadioButton Table@footnote{https://msdn.microsoft.com/en-us/library/windows/desktop/aa370962.aspx}


@node Precomputed tables,,GUI classes,msilib — Read and write Microsoft Installer files
@anchor{library/msilib msi-tables}@anchor{3f12}@anchor{library/msilib precomputed-tables}@anchor{3f13}
@subsubsection Precomputed tables


@ref{b1,,msilib} provides a few subpackages that contain only schema and table
definitions. Currently, these definitions are based on MSI version 2.0.

@geindex schema (in module msilib)
@anchor{library/msilib msilib schema}@anchor{3ed2}
@deffn {Data} msilib.schema

This is the standard MSI schema for MSI 2.0, with the @emph{tables} variable
providing a list of table definitions, and @emph{_Validation_records} providing the
data for MSI validation.
@end deffn

@geindex sequence (in module msilib)
@anchor{library/msilib msilib sequence}@anchor{3f14}
@deffn {Data} msilib.sequence

This module contains table contents for the standard sequence tables:
@emph{AdminExecuteSequence}, @emph{AdminUISequence}, @emph{AdvtExecuteSequence},
@emph{InstallExecuteSequence}, and @emph{InstallUISequence}.
@end deffn

@geindex text (in module msilib)
@anchor{library/msilib msilib text}@anchor{3f15}
@deffn {Data} msilib.text

This module contains definitions for the UIText and ActionText tables, for the
standard installer actions.
@end deffn

@node msvcrt — Useful routines from the MS VC++ runtime,winreg — Windows registry access,msilib — Read and write Microsoft Installer files,MS Windows Specific Services
@anchor{library/msvcrt doc}@anchor{3f16}@anchor{library/msvcrt module-msvcrt}@anchor{b2}@anchor{library/msvcrt msvcrt-useful-routines-from-the-ms-vc-runtime}@anchor{3f17}
@subsection @code{msvcrt} — Useful routines from the MS VC++ runtime


@geindex module; msvcrt


__________________________________________________________________


These functions provide access to some useful capabilities on Windows platforms.
Some higher-level modules use these functions to build the  Windows
implementations of their services.  For example, the @ref{82,,getpass} module uses
this in the implementation of the @ref{82,,getpass()} function.

Further documentation on these functions can be found in the Platform API
documentation.

The module implements both the normal and wide char variants of the console I/O
api. The normal API deals only with ASCII characters and is of limited use
for internationalized applications. The wide char API should be used where
ever possible.


Changed in version 3.3: Operations in this module now raise @ref{2a1,,OSError} where @ref{be8,,IOError}
was raised.


@menu
* File Operations:: 
* Console I/O:: 
* Other Functions:: 

@end menu

@node File Operations,Console I/O,,msvcrt — Useful routines from the MS VC++ runtime
@anchor{library/msvcrt file-operations}@anchor{3f18}@anchor{library/msvcrt msvcrt-files}@anchor{3f19}
@subsubsection File Operations


@geindex locking() (in module msvcrt)
@anchor{library/msvcrt msvcrt locking}@anchor{39f4}
@deffn {Function} msvcrt.locking (fd, mode, nbytes)

Lock part of a file based on file descriptor @emph{fd} from the C runtime.  Raises
@ref{2a1,,OSError} on failure.  The locked region of the file extends from the
current file position for @emph{nbytes} bytes, and may continue beyond the end of the
file.  @emph{mode} must be one of the @code{LK_*} constants listed below. Multiple
regions in a file may be locked at the same time, but may not overlap.  Adjacent
regions are not merged; they must be unlocked individually.

Raises an @ref{1130,,auditing event} @code{msvcrt.locking} with arguments @code{fd}, @code{mode}, @code{nbytes}.
@end deffn

@geindex LK_LOCK (in module msvcrt)
@geindex LK_RLCK (in module msvcrt)
@anchor{library/msvcrt msvcrt LK_LOCK}@anchor{3f1a}
@deffn {Data} msvcrt.LK_LOCK
@anchor{library/msvcrt msvcrt LK_RLCK}@anchor{3f1b}
@deffnx {Data} msvcrt.LK_RLCK

Locks the specified bytes. If the bytes cannot be locked, the program
immediately tries again after 1 second.  If, after 10 attempts, the bytes cannot
be locked, @ref{2a1,,OSError} is raised.
@end deffn

@geindex LK_NBLCK (in module msvcrt)
@geindex LK_NBRLCK (in module msvcrt)
@anchor{library/msvcrt msvcrt LK_NBLCK}@anchor{3f1c}
@deffn {Data} msvcrt.LK_NBLCK
@anchor{library/msvcrt msvcrt LK_NBRLCK}@anchor{3f1d}
@deffnx {Data} msvcrt.LK_NBRLCK

Locks the specified bytes. If the bytes cannot be locked, @ref{2a1,,OSError} is
raised.
@end deffn

@geindex LK_UNLCK (in module msvcrt)
@anchor{library/msvcrt msvcrt LK_UNLCK}@anchor{3f1e}
@deffn {Data} msvcrt.LK_UNLCK

Unlocks the specified bytes, which must have been previously locked.
@end deffn

@geindex setmode() (in module msvcrt)
@anchor{library/msvcrt msvcrt setmode}@anchor{3f1f}
@deffn {Function} msvcrt.setmode (fd, flags)

Set the line-end translation mode for the file descriptor @emph{fd}. To set it to
text mode, @emph{flags} should be @ref{23be,,os.O_TEXT}; for binary, it should be
@ref{23af,,os.O_BINARY}.
@end deffn

@geindex open_osfhandle() (in module msvcrt)
@anchor{library/msvcrt msvcrt open_osfhandle}@anchor{39f5}
@deffn {Function} msvcrt.open_osfhandle (handle, flags)

Create a C runtime file descriptor from the file handle @emph{handle}.  The @emph{flags}
parameter should be a bitwise OR of @ref{23b1,,os.O_APPEND}, @ref{23ad,,os.O_RDONLY},
and @ref{23be,,os.O_TEXT}.  The returned file descriptor may be used as a parameter
to @ref{179a,,os.fdopen()} to create a file object.

Raises an @ref{1130,,auditing event} @code{msvcrt.open_osfhandle} with arguments @code{handle}, @code{flags}.
@end deffn

@geindex get_osfhandle() (in module msvcrt)
@anchor{library/msvcrt msvcrt get_osfhandle}@anchor{39f3}
@deffn {Function} msvcrt.get_osfhandle (fd)

Return the file handle for the file descriptor @emph{fd}.  Raises @ref{2a1,,OSError} if
@emph{fd} is not recognized.

Raises an @ref{1130,,auditing event} @code{msvcrt.get_osfhandle} with argument @code{fd}.
@end deffn

@node Console I/O,Other Functions,File Operations,msvcrt — Useful routines from the MS VC++ runtime
@anchor{library/msvcrt console-i-o}@anchor{3f20}@anchor{library/msvcrt msvcrt-console}@anchor{3f21}
@subsubsection Console I/O


@geindex kbhit() (in module msvcrt)
@anchor{library/msvcrt msvcrt kbhit}@anchor{3f22}
@deffn {Function} msvcrt.kbhit ()

Return @code{True} if a keypress is waiting to be read.
@end deffn

@geindex getch() (in module msvcrt)
@anchor{library/msvcrt msvcrt getch}@anchor{3f23}
@deffn {Function} msvcrt.getch ()

Read a keypress and return the resulting character as a byte string.
Nothing is echoed to the console.  This call will block if a keypress
is not already available, but will not wait for @code{Enter} to be
pressed. If the pressed key was a special function key, this will
return @code{'\000'} or @code{'\xe0'}; the next call will return the keycode.
The @code{Control-C} keypress cannot be read with this function.
@end deffn

@geindex getwch() (in module msvcrt)
@anchor{library/msvcrt msvcrt getwch}@anchor{3f24}
@deffn {Function} msvcrt.getwch ()

Wide char variant of @ref{3f23,,getch()}, returning a Unicode value.
@end deffn

@geindex getche() (in module msvcrt)
@anchor{library/msvcrt msvcrt getche}@anchor{3f25}
@deffn {Function} msvcrt.getche ()

Similar to @ref{3f23,,getch()}, but the keypress will be echoed if it  represents a
printable character.
@end deffn

@geindex getwche() (in module msvcrt)
@anchor{library/msvcrt msvcrt getwche}@anchor{3f26}
@deffn {Function} msvcrt.getwche ()

Wide char variant of @ref{3f25,,getche()}, returning a Unicode value.
@end deffn

@geindex putch() (in module msvcrt)
@anchor{library/msvcrt msvcrt putch}@anchor{3f27}
@deffn {Function} msvcrt.putch (char)

Print the byte string @emph{char} to the console without buffering.
@end deffn

@geindex putwch() (in module msvcrt)
@anchor{library/msvcrt msvcrt putwch}@anchor{3f28}
@deffn {Function} msvcrt.putwch (unicode_char)

Wide char variant of @ref{3f27,,putch()}, accepting a Unicode value.
@end deffn

@geindex ungetch() (in module msvcrt)
@anchor{library/msvcrt msvcrt ungetch}@anchor{3f29}
@deffn {Function} msvcrt.ungetch (char)

Cause the byte string @emph{char} to be “pushed back” into the console buffer;
it will be the next character read by @ref{3f23,,getch()} or @ref{3f25,,getche()}.
@end deffn

@geindex ungetwch() (in module msvcrt)
@anchor{library/msvcrt msvcrt ungetwch}@anchor{3f2a}
@deffn {Function} msvcrt.ungetwch (unicode_char)

Wide char variant of @ref{3f29,,ungetch()}, accepting a Unicode value.
@end deffn

@node Other Functions,,Console I/O,msvcrt — Useful routines from the MS VC++ runtime
@anchor{library/msvcrt msvcrt-other}@anchor{3f2b}@anchor{library/msvcrt other-functions}@anchor{3f2c}
@subsubsection Other Functions


@geindex heapmin() (in module msvcrt)
@anchor{library/msvcrt msvcrt heapmin}@anchor{3f2d}
@deffn {Function} msvcrt.heapmin ()

Force the @code{malloc()} heap to clean itself up and return unused blocks to
the operating system.  On failure, this raises @ref{2a1,,OSError}.
@end deffn

@node winreg — Windows registry access,winsound — Sound-playing interface for Windows,msvcrt — Useful routines from the MS VC++ runtime,MS Windows Specific Services
@anchor{library/winreg doc}@anchor{3f2e}@anchor{library/winreg module-winreg}@anchor{131}@anchor{library/winreg winreg-windows-registry-access}@anchor{3f2f}
@subsection @code{winreg} — Windows registry access


@geindex module; winreg


__________________________________________________________________


These functions expose the Windows registry API to Python.  Instead of using an
integer as the registry handle, a @ref{3f30,,handle object} is used
to ensure that the handles are closed correctly, even if the programmer neglects
to explicitly close them.
@anchor{library/winreg exception-changed}@anchor{3f31}

Changed in version 3.3: Several functions in this module used to raise a
@ref{bea,,WindowsError}, which is now an alias of @ref{2a1,,OSError}.


@menu
* Functions: Functions<12>. 
* Constants: Constants<10>. 
* Registry Handle Objects:: 

@end menu

@node Functions<12>,Constants<10>,,winreg — Windows registry access
@anchor{library/winreg functions}@anchor{3f32}@anchor{library/winreg id1}@anchor{3f33}
@subsubsection Functions


This module offers the following functions:

@geindex CloseKey() (in module winreg)
@anchor{library/winreg winreg CloseKey}@anchor{3f34}
@deffn {Function} winreg.CloseKey (hkey)

Closes a previously opened registry key.  The @emph{hkey} argument specifies a
previously opened key.

@cartouche
@quotation Note 
If @emph{hkey} is not closed using this method (or via @ref{3f35,,hkey.Close()}), it is closed when the @emph{hkey} object is destroyed by
Python.
@end quotation
@end cartouche
@end deffn

@geindex ConnectRegistry() (in module winreg)
@anchor{library/winreg winreg ConnectRegistry}@anchor{39fb}
@deffn {Function} winreg.ConnectRegistry (computer_name, key)

Establishes a connection to a predefined registry handle on another computer,
and returns a @ref{3f30,,handle object}.

@emph{computer_name} is the name of the remote computer, of the form
@code{r"\\computername"}.  If @code{None}, the local computer is used.

@emph{key} is the predefined handle to connect to.

The return value is the handle of the opened key. If the function fails, an
@ref{2a1,,OSError} exception is raised.

Raises an @ref{1130,,auditing event} @code{winreg.ConnectRegistry} with arguments @code{computer_name}, @code{key}.


Changed in version 3.3: See @ref{3f31,,above}.

@end deffn

@geindex CreateKey() (in module winreg)
@anchor{library/winreg winreg CreateKey}@anchor{39fc}
@deffn {Function} winreg.CreateKey (key, sub_key)

Creates or opens the specified key, returning a
@ref{3f30,,handle object}.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

@emph{sub_key} is a string that names the key this method opens or creates.

If @emph{key} is one of the predefined keys, @emph{sub_key} may be @code{None}. In that
case, the handle returned is the same key handle passed in to the function.

If the key already exists, this function opens the existing key.

The return value is the handle of the opened key. If the function fails, an
@ref{2a1,,OSError} exception is raised.

Raises an @ref{1130,,auditing event} @code{winreg.CreateKey} with arguments @code{key}, @code{sub_key}, @code{access}.

Raises an @ref{1130,,auditing event} @code{winreg.OpenKey/result} with argument @code{key}.


Changed in version 3.3: See @ref{3f31,,above}.

@end deffn

@geindex CreateKeyEx() (in module winreg)
@anchor{library/winreg winreg CreateKeyEx}@anchor{39fd}
@deffn {Function} winreg.CreateKeyEx (key, sub_key, reserved=0, access=KEY_WRITE)

Creates or opens the specified key, returning a
@ref{3f30,,handle object}.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

@emph{sub_key} is a string that names the key this method opens or creates.

@emph{reserved} is a reserved integer, and must be zero. The default is zero.

@emph{access} is an integer that specifies an access mask that describes the desired
security access for the key.  Default is @ref{3f37,,KEY_WRITE}.  See
@ref{3f38,,Access Rights} for other allowed values.

If @emph{key} is one of the predefined keys, @emph{sub_key} may be @code{None}. In that
case, the handle returned is the same key handle passed in to the function.

If the key already exists, this function opens the existing key.

The return value is the handle of the opened key. If the function fails, an
@ref{2a1,,OSError} exception is raised.

Raises an @ref{1130,,auditing event} @code{winreg.CreateKey} with arguments @code{key}, @code{sub_key}, @code{access}.

Raises an @ref{1130,,auditing event} @code{winreg.OpenKey/result} with argument @code{key}.


New in version 3.2.



Changed in version 3.3: See @ref{3f31,,above}.

@end deffn

@geindex DeleteKey() (in module winreg)
@anchor{library/winreg winreg DeleteKey}@anchor{39fe}
@deffn {Function} winreg.DeleteKey (key, sub_key)

Deletes the specified key.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

@emph{sub_key} is a string that must be a subkey of the key identified by the @emph{key}
parameter.  This value must not be @code{None}, and the key may not have subkeys.

@emph{This method can not delete keys with subkeys.}

If the method succeeds, the entire key, including all of its values, is removed.
If the method fails, an @ref{2a1,,OSError} exception is raised.

Raises an @ref{1130,,auditing event} @code{winreg.DeleteKey} with arguments @code{key}, @code{sub_key}, @code{access}.


Changed in version 3.3: See @ref{3f31,,above}.

@end deffn

@geindex DeleteKeyEx() (in module winreg)
@anchor{library/winreg winreg DeleteKeyEx}@anchor{39ff}
@deffn {Function} winreg.DeleteKeyEx (key, sub_key, access=KEY_WOW64_64KEY, reserved=0)

Deletes the specified key.

@cartouche
@quotation Note 
The @ref{39ff,,DeleteKeyEx()} function is implemented with the RegDeleteKeyEx
Windows API function, which is specific to 64-bit versions of Windows.
See the RegDeleteKeyEx documentation@footnote{https://msdn.microsoft.com/en-us/library/ms724847%28VS.85%29.aspx}.
@end quotation
@end cartouche

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

@emph{sub_key} is a string that must be a subkey of the key identified by the
@emph{key} parameter. This value must not be @code{None}, and the key may not have
subkeys.

@emph{reserved} is a reserved integer, and must be zero. The default is zero.

@emph{access} is an integer that specifies an access mask that describes the desired
security access for the key.  Default is @ref{3f39,,KEY_WOW64_64KEY}.  See
@ref{3f38,,Access Rights} for other allowed values.

@emph{This method can not delete keys with subkeys.}

If the method succeeds, the entire key, including all of its values, is
removed. If the method fails, an @ref{2a1,,OSError} exception is raised.

On unsupported Windows versions, @ref{8a3,,NotImplementedError} is raised.

Raises an @ref{1130,,auditing event} @code{winreg.DeleteKey} with arguments @code{key}, @code{sub_key}, @code{access}.


New in version 3.2.



Changed in version 3.3: See @ref{3f31,,above}.

@end deffn

@geindex DeleteValue() (in module winreg)
@anchor{library/winreg winreg DeleteValue}@anchor{3a00}
@deffn {Function} winreg.DeleteValue (key, value)

Removes a named value from a registry key.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

@emph{value} is a string that identifies the value to remove.

Raises an @ref{1130,,auditing event} @code{winreg.DeleteValue} with arguments @code{key}, @code{value}.
@end deffn

@geindex EnumKey() (in module winreg)
@anchor{library/winreg winreg EnumKey}@anchor{3a03}
@deffn {Function} winreg.EnumKey (key, index)

Enumerates subkeys of an open registry key, returning a string.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

@emph{index} is an integer that identifies the index of the key to retrieve.

The function retrieves the name of one subkey each time it is called.  It is
typically called repeatedly until an @ref{2a1,,OSError} exception is
raised, indicating, no more values are available.

Raises an @ref{1130,,auditing event} @code{winreg.EnumKey} with arguments @code{key}, @code{index}.


Changed in version 3.3: See @ref{3f31,,above}.

@end deffn

@geindex EnumValue() (in module winreg)
@anchor{library/winreg winreg EnumValue}@anchor{3a04}
@deffn {Function} winreg.EnumValue (key, index)

Enumerates values of an open registry key, returning a tuple.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

@emph{index} is an integer that identifies the index of the value to retrieve.

The function retrieves the name of one subkey each time it is called. It is
typically called repeatedly, until an @ref{2a1,,OSError} exception is
raised, indicating no more values.

The result is a tuple of 3 items:


@multitable {xxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Index

@tab

Meaning

@item

@code{0}

@tab

A string that identifies the value name

@item

@code{1}

@tab

An object that holds the value data, and
whose type depends on the underlying
registry type

@item

@code{2}

@tab

An integer that identifies the type of the
value data (see table in docs for
@ref{3a0f,,SetValueEx()})

@end multitable


Raises an @ref{1130,,auditing event} @code{winreg.EnumValue} with arguments @code{key}, @code{index}.


Changed in version 3.3: See @ref{3f31,,above}.

@end deffn

@geindex % (percent); environment variables expansion (Windows)

@geindex ExpandEnvironmentStrings() (in module winreg)
@anchor{library/winreg winreg ExpandEnvironmentStrings}@anchor{3a05}
@deffn {Function} winreg.ExpandEnvironmentStrings (str)

Expands environment variable placeholders @code{%NAME%} in strings like
@ref{3f3a,,REG_EXPAND_SZ}:

@example
>>> ExpandEnvironmentStrings('%windir%')
'C:\\Windows'
@end example

Raises an @ref{1130,,auditing event} @code{winreg.ExpandEnvironmentStrings} with argument @code{str}.
@end deffn

@geindex FlushKey() (in module winreg)
@anchor{library/winreg winreg FlushKey}@anchor{3f3b}
@deffn {Function} winreg.FlushKey (key)

Writes all the attributes of a key to the registry.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

It is not necessary to call @ref{3f3b,,FlushKey()} to change a key. Registry changes are
flushed to disk by the registry using its lazy flusher.  Registry changes are
also flushed to disk at system shutdown.  Unlike @ref{3f34,,CloseKey()}, the
@ref{3f3b,,FlushKey()} method returns only when all the data has been written to the
registry. An application should only call @ref{3f3b,,FlushKey()} if it requires
absolute certainty that registry changes are on disk.

@cartouche
@quotation Note 
If you don’t know whether a @ref{3f3b,,FlushKey()} call is required, it probably
isn’t.
@end quotation
@end cartouche
@end deffn

@geindex LoadKey() (in module winreg)
@anchor{library/winreg winreg LoadKey}@anchor{3a06}
@deffn {Function} winreg.LoadKey (key, sub_key, file_name)

Creates a subkey under the specified key and stores registration information
from a specified file into that subkey.

@emph{key} is a handle returned by @ref{39fb,,ConnectRegistry()} or one of the constants
@ref{3f3c,,HKEY_USERS} or @ref{3f3d,,HKEY_LOCAL_MACHINE}.

@emph{sub_key} is a string that identifies the subkey to load.

@emph{file_name} is the name of the file to load registry data from. This file must
have been created with the @ref{3a0d,,SaveKey()} function. Under the file allocation
table (FAT) file system, the filename may not have an extension.

A call to @ref{3a06,,LoadKey()} fails if the calling process does not have the
@code{SE_RESTORE_PRIVILEGE} privilege.  Note that privileges are different
from permissions – see the RegLoadKey documentation@footnote{https://msdn.microsoft.com/en-us/library/ms724889%28v=VS.85%29.aspx} for
more details.

If @emph{key} is a handle returned by @ref{39fb,,ConnectRegistry()}, then the path
specified in @emph{file_name} is relative to the remote computer.

Raises an @ref{1130,,auditing event} @code{winreg.LoadKey} with arguments @code{key}, @code{sub_key}, @code{file_name}.
@end deffn

@geindex OpenKey() (in module winreg)
@geindex OpenKeyEx() (in module winreg)
@anchor{library/winreg winreg OpenKey}@anchor{3a07}
@deffn {Function} winreg.OpenKey (key, sub_key, reserved=0, access=KEY_READ)
@anchor{library/winreg winreg OpenKeyEx}@anchor{3f3e}
@deffnx {Function} winreg.OpenKeyEx (key, sub_key, reserved=0, access=KEY_READ)

Opens the specified key, returning a @ref{3f30,,handle object}.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

@emph{sub_key} is a string that identifies the sub_key to open.

@emph{reserved} is a reserved integer, and must be zero.  The default is zero.

@emph{access} is an integer that specifies an access mask that describes the desired
security access for the key.  Default is @ref{3f3f,,KEY_READ}.  See @ref{3f38,,Access Rights} for other allowed values.

The result is a new handle to the specified key.

If the function fails, @ref{2a1,,OSError} is raised.

Raises an @ref{1130,,auditing event} @code{winreg.OpenKey} with arguments @code{key}, @code{sub_key}, @code{access}.

Raises an @ref{1130,,auditing event} @code{winreg.OpenKey/result} with argument @code{key}.


Changed in version 3.2: Allow the use of named arguments.



Changed in version 3.3: See @ref{3f31,,above}.

@end deffn

@geindex QueryInfoKey() (in module winreg)
@anchor{library/winreg winreg QueryInfoKey}@anchor{3a09}
@deffn {Function} winreg.QueryInfoKey (key)

Returns information about a key, as a tuple.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

The result is a tuple of 3 items:


@multitable {xxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Index

@tab

Meaning

@item

@code{0}

@tab

An integer giving the number of sub keys
this key has.

@item

@code{1}

@tab

An integer giving the number of values this
key has.

@item

@code{2}

@tab

An integer giving when the key was last
modified (if available) as 100’s of
nanoseconds since Jan 1, 1601.

@end multitable


Raises an @ref{1130,,auditing event} @code{winreg.QueryInfoKey} with argument @code{key}.
@end deffn

@geindex QueryValue() (in module winreg)
@anchor{library/winreg winreg QueryValue}@anchor{3a0b}
@deffn {Function} winreg.QueryValue (key, sub_key)

Retrieves the unnamed value for a key, as a string.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

@emph{sub_key} is a string that holds the name of the subkey with which the value is
associated.  If this parameter is @code{None} or empty, the function retrieves the
value set by the @ref{3a0e,,SetValue()} method for the key identified by @emph{key}.

Values in the registry have name, type, and data components. This method
retrieves the data for a key’s first value that has a @code{NULL} name. But the
underlying API call doesn’t return the type, so always use
@ref{3a0c,,QueryValueEx()} if possible.

Raises an @ref{1130,,auditing event} @code{winreg.QueryValue} with arguments @code{key}, @code{sub_key}, @code{value_name}.
@end deffn

@geindex QueryValueEx() (in module winreg)
@anchor{library/winreg winreg QueryValueEx}@anchor{3a0c}
@deffn {Function} winreg.QueryValueEx (key, value_name)

Retrieves the type and data for a specified value name associated with
an open registry key.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

@emph{value_name} is a string indicating the value to query.

The result is a tuple of 2 items:


@multitable {xxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Index

@tab

Meaning

@item

@code{0}

@tab

The value of the registry item.

@item

@code{1}

@tab

An integer giving the registry type for
this value (see table in docs for
@ref{3a0f,,SetValueEx()})

@end multitable


Raises an @ref{1130,,auditing event} @code{winreg.QueryValue} with arguments @code{key}, @code{sub_key}, @code{value_name}.
@end deffn

@geindex SaveKey() (in module winreg)
@anchor{library/winreg winreg SaveKey}@anchor{3a0d}
@deffn {Function} winreg.SaveKey (key, file_name)

Saves the specified key, and all its subkeys to the specified file.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

@emph{file_name} is the name of the file to save registry data to.  This file
cannot already exist. If this filename includes an extension, it cannot be
used on file allocation table (FAT) file systems by the @ref{3a06,,LoadKey()}
method.

If @emph{key} represents a key on a remote computer, the path described by
@emph{file_name} is relative to the remote computer. The caller of this method must
possess the @code{SeBackupPrivilege} security privilege.  Note that
privileges are different than permissions – see the
Conflicts Between User Rights and Permissions documentation@footnote{https://msdn.microsoft.com/en-us/library/ms724878%28v=VS.85%29.aspx}
for more details.

This function passes @code{NULL} for @emph{security_attributes} to the API.

Raises an @ref{1130,,auditing event} @code{winreg.SaveKey} with arguments @code{key}, @code{file_name}.
@end deffn

@geindex SetValue() (in module winreg)
@anchor{library/winreg winreg SetValue}@anchor{3a0e}
@deffn {Function} winreg.SetValue (key, sub_key, type, value)

Associates a value with a specified key.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

@emph{sub_key} is a string that names the subkey with which the value is associated.

@emph{type} is an integer that specifies the type of the data. Currently this must be
@ref{3f40,,REG_SZ}, meaning only strings are supported.  Use the @ref{3a0f,,SetValueEx()}
function for support for other data types.

@emph{value} is a string that specifies the new value.

If the key specified by the @emph{sub_key} parameter does not exist, the SetValue
function creates it.

Value lengths are limited by available memory. Long values (more than 2048
bytes) should be stored as files with the filenames stored in the configuration
registry.  This helps the registry perform efficiently.

The key identified by the @emph{key} parameter must have been opened with
@ref{3f41,,KEY_SET_VALUE} access.

Raises an @ref{1130,,auditing event} @code{winreg.SetValue} with arguments @code{key}, @code{sub_key}, @code{type}, @code{value}.
@end deffn

@geindex SetValueEx() (in module winreg)
@anchor{library/winreg winreg SetValueEx}@anchor{3a0f}
@deffn {Function} winreg.SetValueEx (key, value_name, reserved, type, value)

Stores data in the value field of an open registry key.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

@emph{value_name} is a string that names the subkey with which the value is
associated.

@emph{reserved} can be anything – zero is always passed to the API.

@emph{type} is an integer that specifies the type of the data. See
@ref{3f42,,Value Types} for the available types.

@emph{value} is a string that specifies the new value.

This method can also set additional value and type information for the specified
key.  The key identified by the key parameter must have been opened with
@ref{3f41,,KEY_SET_VALUE} access.

To open the key, use the @ref{39fc,,CreateKey()} or @ref{3a07,,OpenKey()} methods.

Value lengths are limited by available memory. Long values (more than 2048
bytes) should be stored as files with the filenames stored in the configuration
registry.  This helps the registry perform efficiently.

Raises an @ref{1130,,auditing event} @code{winreg.SetValue} with arguments @code{key}, @code{sub_key}, @code{type}, @code{value}.
@end deffn

@geindex DisableReflectionKey() (in module winreg)
@anchor{library/winreg winreg DisableReflectionKey}@anchor{3a01}
@deffn {Function} winreg.DisableReflectionKey (key)

Disables registry reflection for 32-bit processes running on a 64-bit
operating system.

@emph{key} is an already open key, or one of the predefined @ref{3f36,,HKEY_* constants}.

Will generally raise @ref{8a3,,NotImplementedError} if executed on a 32-bit operating
system.

If the key is not on the reflection list, the function succeeds but has no
effect.  Disabling reflection for a key does not affect reflection of any
subkeys.

Raises an @ref{1130,,auditing event} @code{winreg.DisableReflectionKey} with argument @code{key}.
@end deffn

@geindex EnableReflectionKey() (in module winreg)
@anchor{library/winreg winreg EnableReflectionKey}@anchor{3a02}
@deffn {Function} winreg.EnableReflectionKey (key)

Restores registry reflection for the specified disabled key.

@emph{key} is an already open key, or one of the predefined @ref{3f36,,HKEY_* constants}.

Will generally raise @ref{8a3,,NotImplementedError} if executed on a 32-bit operating
system.

Restoring reflection for a key does not affect reflection of any subkeys.

Raises an @ref{1130,,auditing event} @code{winreg.EnableReflectionKey} with argument @code{key}.
@end deffn

@geindex QueryReflectionKey() (in module winreg)
@anchor{library/winreg winreg QueryReflectionKey}@anchor{3a0a}
@deffn {Function} winreg.QueryReflectionKey (key)

Determines the reflection state for the specified key.

@emph{key} is an already open key, or one of the predefined
@ref{3f36,,HKEY_* constants}.

Returns @code{True} if reflection is disabled.

Will generally raise @ref{8a3,,NotImplementedError} if executed on a 32-bit
operating system.

Raises an @ref{1130,,auditing event} @code{winreg.QueryReflectionKey} with argument @code{key}.
@end deffn

@node Constants<10>,Registry Handle Objects,Functions<12>,winreg — Windows registry access
@anchor{library/winreg constants}@anchor{3f43}@anchor{library/winreg id2}@anchor{3f44}
@subsubsection Constants


The following constants are defined for use in many @code{_winreg} functions.

@menu
* HKEY_* Constants:: 
* Access Rights:: 
* Value Types:: 

@end menu

@node HKEY_* Constants,Access Rights,,Constants<10>
@anchor{library/winreg hkey-constants}@anchor{3f36}@anchor{library/winreg id3}@anchor{3f45}
@subsubsection HKEY_* Constants


@geindex HKEY_CLASSES_ROOT (in module winreg)
@anchor{library/winreg winreg HKEY_CLASSES_ROOT}@anchor{3f46}
@deffn {Data} winreg.HKEY_CLASSES_ROOT

Registry entries subordinate to this key define types (or classes) of
documents and the properties associated with those types. Shell and
COM applications use the information stored under this key.
@end deffn

@geindex HKEY_CURRENT_USER (in module winreg)
@anchor{library/winreg winreg HKEY_CURRENT_USER}@anchor{3f47}
@deffn {Data} winreg.HKEY_CURRENT_USER

Registry entries subordinate to this key define the preferences of
the current user. These preferences include the settings of
environment variables, data about program groups, colors, printers,
network connections, and application preferences.
@end deffn

@geindex HKEY_LOCAL_MACHINE (in module winreg)
@anchor{library/winreg winreg HKEY_LOCAL_MACHINE}@anchor{3f3d}
@deffn {Data} winreg.HKEY_LOCAL_MACHINE

Registry entries subordinate to this key define the physical state
of the computer, including data about the bus type, system memory,
and installed hardware and software.
@end deffn

@geindex HKEY_USERS (in module winreg)
@anchor{library/winreg winreg HKEY_USERS}@anchor{3f3c}
@deffn {Data} winreg.HKEY_USERS

Registry entries subordinate to this key define the default user
configuration for new users on the local computer and the user
configuration for the current user.
@end deffn

@geindex HKEY_PERFORMANCE_DATA (in module winreg)
@anchor{library/winreg winreg HKEY_PERFORMANCE_DATA}@anchor{3f48}
@deffn {Data} winreg.HKEY_PERFORMANCE_DATA

Registry entries subordinate to this key allow you to access
performance data. The data is not actually stored in the registry;
the registry functions cause the system to collect the data from
its source.
@end deffn

@geindex HKEY_CURRENT_CONFIG (in module winreg)
@anchor{library/winreg winreg HKEY_CURRENT_CONFIG}@anchor{3f49}
@deffn {Data} winreg.HKEY_CURRENT_CONFIG

Contains information about the current hardware profile of the
local computer system.
@end deffn

@geindex HKEY_DYN_DATA (in module winreg)
@anchor{library/winreg winreg HKEY_DYN_DATA}@anchor{3f4a}
@deffn {Data} winreg.HKEY_DYN_DATA

This key is not used in versions of Windows after 98.
@end deffn

@node Access Rights,Value Types,HKEY_* Constants,Constants<10>
@anchor{library/winreg access-rights}@anchor{3f38}@anchor{library/winreg id4}@anchor{3f4b}
@subsubsection Access Rights


For more information, see Registry Key Security and Access@footnote{https://msdn.microsoft.com/en-us/library/ms724878%28v=VS.85%29.aspx}.

@geindex KEY_ALL_ACCESS (in module winreg)
@anchor{library/winreg winreg KEY_ALL_ACCESS}@anchor{3f4c}
@deffn {Data} winreg.KEY_ALL_ACCESS

Combines the STANDARD_RIGHTS_REQUIRED, @ref{3f4d,,KEY_QUERY_VALUE},
@ref{3f41,,KEY_SET_VALUE}, @ref{3f4e,,KEY_CREATE_SUB_KEY},
@ref{3f4f,,KEY_ENUMERATE_SUB_KEYS}, @ref{3f50,,KEY_NOTIFY},
and @ref{3f51,,KEY_CREATE_LINK} access rights.
@end deffn

@geindex KEY_WRITE (in module winreg)
@anchor{library/winreg winreg KEY_WRITE}@anchor{3f37}
@deffn {Data} winreg.KEY_WRITE

Combines the STANDARD_RIGHTS_WRITE, @ref{3f41,,KEY_SET_VALUE}, and
@ref{3f4e,,KEY_CREATE_SUB_KEY} access rights.
@end deffn

@geindex KEY_READ (in module winreg)
@anchor{library/winreg winreg KEY_READ}@anchor{3f3f}
@deffn {Data} winreg.KEY_READ

Combines the STANDARD_RIGHTS_READ, @ref{3f4d,,KEY_QUERY_VALUE},
@ref{3f4f,,KEY_ENUMERATE_SUB_KEYS}, and @ref{3f50,,KEY_NOTIFY} values.
@end deffn

@geindex KEY_EXECUTE (in module winreg)
@anchor{library/winreg winreg KEY_EXECUTE}@anchor{3f52}
@deffn {Data} winreg.KEY_EXECUTE

Equivalent to @ref{3f3f,,KEY_READ}.
@end deffn

@geindex KEY_QUERY_VALUE (in module winreg)
@anchor{library/winreg winreg KEY_QUERY_VALUE}@anchor{3f4d}
@deffn {Data} winreg.KEY_QUERY_VALUE

Required to query the values of a registry key.
@end deffn

@geindex KEY_SET_VALUE (in module winreg)
@anchor{library/winreg winreg KEY_SET_VALUE}@anchor{3f41}
@deffn {Data} winreg.KEY_SET_VALUE

Required to create, delete, or set a registry value.
@end deffn

@geindex KEY_CREATE_SUB_KEY (in module winreg)
@anchor{library/winreg winreg KEY_CREATE_SUB_KEY}@anchor{3f4e}
@deffn {Data} winreg.KEY_CREATE_SUB_KEY

Required to create a subkey of a registry key.
@end deffn

@geindex KEY_ENUMERATE_SUB_KEYS (in module winreg)
@anchor{library/winreg winreg KEY_ENUMERATE_SUB_KEYS}@anchor{3f4f}
@deffn {Data} winreg.KEY_ENUMERATE_SUB_KEYS

Required to enumerate the subkeys of a registry key.
@end deffn

@geindex KEY_NOTIFY (in module winreg)
@anchor{library/winreg winreg KEY_NOTIFY}@anchor{3f50}
@deffn {Data} winreg.KEY_NOTIFY

Required to request change notifications for a registry key or for
subkeys of a registry key.
@end deffn

@geindex KEY_CREATE_LINK (in module winreg)
@anchor{library/winreg winreg KEY_CREATE_LINK}@anchor{3f51}
@deffn {Data} winreg.KEY_CREATE_LINK

Reserved for system use.
@end deffn

@menu
* 64-bit Specific:: 

@end menu

@node 64-bit Specific,,,Access Rights
@anchor{library/winreg bit-access-rights}@anchor{3f53}@anchor{library/winreg bit-specific}@anchor{3f54}
@subsubsection 64-bit Specific


For more information, see Accessing an Alternate Registry View@footnote{https://msdn.microsoft.com/en-us/library/aa384129(v=VS.85).aspx}.

@geindex KEY_WOW64_64KEY (in module winreg)
@anchor{library/winreg winreg KEY_WOW64_64KEY}@anchor{3f39}
@deffn {Data} winreg.KEY_WOW64_64KEY

Indicates that an application on 64-bit Windows should operate on
the 64-bit registry view.
@end deffn

@geindex KEY_WOW64_32KEY (in module winreg)
@anchor{library/winreg winreg KEY_WOW64_32KEY}@anchor{3f55}
@deffn {Data} winreg.KEY_WOW64_32KEY

Indicates that an application on 64-bit Windows should operate on
the 32-bit registry view.
@end deffn

@node Value Types,,Access Rights,Constants<10>
@anchor{library/winreg id5}@anchor{3f56}@anchor{library/winreg value-types}@anchor{3f42}
@subsubsection Value Types


For more information, see Registry Value Types@footnote{https://msdn.microsoft.com/en-us/library/ms724884%28v=VS.85%29.aspx}.

@geindex REG_BINARY (in module winreg)
@anchor{library/winreg winreg REG_BINARY}@anchor{3f57}
@deffn {Data} winreg.REG_BINARY

Binary data in any form.
@end deffn

@geindex REG_DWORD (in module winreg)
@anchor{library/winreg winreg REG_DWORD}@anchor{3f58}
@deffn {Data} winreg.REG_DWORD

32-bit number.
@end deffn

@geindex REG_DWORD_LITTLE_ENDIAN (in module winreg)
@anchor{library/winreg winreg REG_DWORD_LITTLE_ENDIAN}@anchor{3f59}
@deffn {Data} winreg.REG_DWORD_LITTLE_ENDIAN

A 32-bit number in little-endian format. Equivalent to @ref{3f58,,REG_DWORD}.
@end deffn

@geindex REG_DWORD_BIG_ENDIAN (in module winreg)
@anchor{library/winreg winreg REG_DWORD_BIG_ENDIAN}@anchor{3f5a}
@deffn {Data} winreg.REG_DWORD_BIG_ENDIAN

A 32-bit number in big-endian format.
@end deffn

@geindex REG_EXPAND_SZ (in module winreg)
@anchor{library/winreg winreg REG_EXPAND_SZ}@anchor{3f3a}
@deffn {Data} winreg.REG_EXPAND_SZ

Null-terminated string containing references to environment
variables (@code{%PATH%}).
@end deffn

@geindex REG_LINK (in module winreg)
@anchor{library/winreg winreg REG_LINK}@anchor{3f5b}
@deffn {Data} winreg.REG_LINK

A Unicode symbolic link.
@end deffn

@geindex REG_MULTI_SZ (in module winreg)
@anchor{library/winreg winreg REG_MULTI_SZ}@anchor{1280}
@deffn {Data} winreg.REG_MULTI_SZ

A sequence of null-terminated strings, terminated by two null characters.
(Python handles this termination automatically.)
@end deffn

@geindex REG_NONE (in module winreg)
@anchor{library/winreg winreg REG_NONE}@anchor{3f5c}
@deffn {Data} winreg.REG_NONE

No defined value type.
@end deffn

@geindex REG_QWORD (in module winreg)
@anchor{library/winreg winreg REG_QWORD}@anchor{84d}
@deffn {Data} winreg.REG_QWORD

A 64-bit number.


New in version 3.6.

@end deffn

@geindex REG_QWORD_LITTLE_ENDIAN (in module winreg)
@anchor{library/winreg winreg REG_QWORD_LITTLE_ENDIAN}@anchor{3f5d}
@deffn {Data} winreg.REG_QWORD_LITTLE_ENDIAN

A 64-bit number in little-endian format. Equivalent to @ref{84d,,REG_QWORD}.


New in version 3.6.

@end deffn

@geindex REG_RESOURCE_LIST (in module winreg)
@anchor{library/winreg winreg REG_RESOURCE_LIST}@anchor{3f5e}
@deffn {Data} winreg.REG_RESOURCE_LIST

A device-driver resource list.
@end deffn

@geindex REG_FULL_RESOURCE_DESCRIPTOR (in module winreg)
@anchor{library/winreg winreg REG_FULL_RESOURCE_DESCRIPTOR}@anchor{3f5f}
@deffn {Data} winreg.REG_FULL_RESOURCE_DESCRIPTOR

A hardware setting.
@end deffn

@geindex REG_RESOURCE_REQUIREMENTS_LIST (in module winreg)
@anchor{library/winreg winreg REG_RESOURCE_REQUIREMENTS_LIST}@anchor{3f60}
@deffn {Data} winreg.REG_RESOURCE_REQUIREMENTS_LIST

A hardware resource list.
@end deffn

@geindex REG_SZ (in module winreg)
@anchor{library/winreg winreg REG_SZ}@anchor{3f40}
@deffn {Data} winreg.REG_SZ

A null-terminated string.
@end deffn

@node Registry Handle Objects,,Constants<10>,winreg — Windows registry access
@anchor{library/winreg handle-object}@anchor{3f30}@anchor{library/winreg registry-handle-objects}@anchor{3f61}
@subsubsection Registry Handle Objects


This object wraps a Windows HKEY object, automatically closing it when the
object is destroyed.  To guarantee cleanup, you can call either the
@ref{3f35,,Close()} method on the object, or the @ref{3f34,,CloseKey()} function.

All registry functions in this module return one of these objects.

All registry functions in this module which accept a handle object also accept
an integer, however, use of the handle object is encouraged.

Handle objects provide semantics for @code{__bool__()} – thus

@example
if handle:
    print("Yes")
@end example

will print @code{Yes} if the handle is currently valid (has not been closed or
detached).

The object also support comparison semantics, so handle objects will compare
true if they both reference the same underlying Windows handle value.

Handle objects can be converted to an integer (e.g., using the built-in
@ref{1b1,,int()} function), in which case the underlying Windows handle value is
returned.  You can also use the @ref{3a08,,Detach()} method to return the
integer handle, and also disconnect the Windows handle from the handle object.

@geindex Close() (winreg.PyHKEY method)
@anchor{library/winreg winreg PyHKEY Close}@anchor{3f35}
@deffn {Method} PyHKEY.Close ()

Closes the underlying Windows handle.

If the handle is already closed, no error is raised.
@end deffn

@geindex Detach() (winreg.PyHKEY method)
@anchor{library/winreg winreg PyHKEY Detach}@anchor{3a08}
@deffn {Method} PyHKEY.Detach ()

Detaches the Windows handle from the handle object.

The result is an integer that holds the value of the handle before it is
detached.  If the handle is already detached or closed, this will return
zero.

After calling this function, the handle is effectively invalidated, but the
handle is not closed.  You would call this function when you need the
underlying Win32 handle to exist beyond the lifetime of the handle object.

Raises an @ref{1130,,auditing event} @code{winreg.PyHKEY.Detach} with argument @code{key}.
@end deffn

@geindex __enter__() (winreg.PyHKEY method)
@geindex __exit__() (winreg.PyHKEY method)
@anchor{library/winreg winreg PyHKEY __enter__}@anchor{3f62}
@deffn {Method} PyHKEY.__enter__ ()
@anchor{library/winreg winreg PyHKEY __exit__}@anchor{3f63}
@deffnx {Method} PyHKEY.__exit__ (*exc_info)

The HKEY object implements @ref{1811,,__enter__()} and
@ref{1812,,__exit__()} and thus supports the context protocol for the
@ref{168,,with} statement:

@example
with OpenKey(HKEY_LOCAL_MACHINE, "foo") as key:
    ...  # work with key
@end example

will automatically close @emph{key} when control leaves the @ref{168,,with} block.
@end deffn

@node winsound — Sound-playing interface for Windows,,winreg — Windows registry access,MS Windows Specific Services
@anchor{library/winsound doc}@anchor{3f64}@anchor{library/winsound module-winsound}@anchor{132}@anchor{library/winsound winsound-sound-playing-interface-for-windows}@anchor{3f65}
@subsection @code{winsound} — Sound-playing interface for Windows


@geindex module; winsound


__________________________________________________________________


The @ref{132,,winsound} module provides access to the basic sound-playing machinery
provided by Windows platforms.  It includes functions and several constants.

@geindex Beep() (in module winsound)
@anchor{library/winsound winsound Beep}@anchor{84f}
@deffn {Function} winsound.Beep (frequency, duration)

Beep the PC’s speaker. The @emph{frequency} parameter specifies frequency, in hertz,
of the sound, and must be in the range 37 through 32,767. The @emph{duration}
parameter specifies the number of milliseconds the sound should last.  If the
system is not able to beep the speaker, @ref{5af,,RuntimeError} is raised.
@end deffn

@geindex PlaySound() (in module winsound)
@anchor{library/winsound winsound PlaySound}@anchor{851}
@deffn {Function} winsound.PlaySound (sound, flags)

Call the underlying @code{PlaySound()} function from the Platform API.  The
@emph{sound} parameter may be a filename, a system sound alias, audio data as a
@ref{87e,,bytes-like object}, or @code{None}.  Its
interpretation depends on the value of @emph{flags}, which can be a bitwise ORed
combination of the constants described below. If the @emph{sound} parameter is
@code{None}, any currently playing waveform sound is stopped. If the system
indicates an error, @ref{5af,,RuntimeError} is raised.
@end deffn

@geindex MessageBeep() (in module winsound)
@anchor{library/winsound winsound MessageBeep}@anchor{850}
@deffn {Function} winsound.MessageBeep (type=MB_OK)

Call the underlying @code{MessageBeep()} function from the Platform API.  This
plays a sound as specified in the registry.  The @emph{type} argument specifies which
sound to play; possible values are @code{-1}, @code{MB_ICONASTERISK},
@code{MB_ICONEXCLAMATION}, @code{MB_ICONHAND}, @code{MB_ICONQUESTION}, and @code{MB_OK}, all
described below.  The value @code{-1} produces a “simple beep”; this is the final
fallback if a sound cannot be played otherwise.  If the system indicates an
error, @ref{5af,,RuntimeError} is raised.
@end deffn

@geindex SND_FILENAME (in module winsound)
@anchor{library/winsound winsound SND_FILENAME}@anchor{3f66}
@deffn {Data} winsound.SND_FILENAME

The @emph{sound} parameter is the name of a WAV file. Do not use with
@ref{3f67,,SND_ALIAS}.
@end deffn

@geindex SND_ALIAS (in module winsound)
@anchor{library/winsound winsound SND_ALIAS}@anchor{3f67}
@deffn {Data} winsound.SND_ALIAS

The @emph{sound} parameter is a sound association name from the registry.  If the
registry contains no such name, play the system default sound unless
@ref{3f68,,SND_NODEFAULT} is also specified. If no default sound is registered,
raise @ref{5af,,RuntimeError}. Do not use with @ref{3f66,,SND_FILENAME}.

All Win32 systems support at least the following; most systems support many
more:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

@ref{851,,PlaySound()} @emph{name}

@tab

Corresponding Control Panel Sound name

@item

@code{'SystemAsterisk'}

@tab

Asterisk

@item

@code{'SystemExclamation'}

@tab

Exclamation

@item

@code{'SystemExit'}

@tab

Exit Windows

@item

@code{'SystemHand'}

@tab

Critical Stop

@item

@code{'SystemQuestion'}

@tab

Question

@end multitable


For example:

@example
import winsound
# Play Windows exit sound.
winsound.PlaySound("SystemExit", winsound.SND_ALIAS)

# Probably play Windows default sound, if any is registered (because
# "*" probably isn't the registered name of any sound).
winsound.PlaySound("*", winsound.SND_ALIAS)
@end example
@end deffn

@geindex SND_LOOP (in module winsound)
@anchor{library/winsound winsound SND_LOOP}@anchor{3f69}
@deffn {Data} winsound.SND_LOOP

Play the sound repeatedly.  The @ref{3f6a,,SND_ASYNC} flag must also be used to
avoid blocking.  Cannot be used with @ref{3f6b,,SND_MEMORY}.
@end deffn

@geindex SND_MEMORY (in module winsound)
@anchor{library/winsound winsound SND_MEMORY}@anchor{3f6b}
@deffn {Data} winsound.SND_MEMORY

The @emph{sound} parameter to @ref{851,,PlaySound()} is a memory image of a WAV file, as a
@ref{87e,,bytes-like object}.

@cartouche
@quotation Note 
This module does not support playing from a memory image asynchronously, so a
combination of this flag and @ref{3f6a,,SND_ASYNC} will raise @ref{5af,,RuntimeError}.
@end quotation
@end cartouche
@end deffn

@geindex SND_PURGE (in module winsound)
@anchor{library/winsound winsound SND_PURGE}@anchor{3f6c}
@deffn {Data} winsound.SND_PURGE

Stop playing all instances of the specified sound.

@cartouche
@quotation Note 
This flag is not supported on modern Windows platforms.
@end quotation
@end cartouche
@end deffn

@geindex SND_ASYNC (in module winsound)
@anchor{library/winsound winsound SND_ASYNC}@anchor{3f6a}
@deffn {Data} winsound.SND_ASYNC

Return immediately, allowing sounds to play asynchronously.
@end deffn

@geindex SND_NODEFAULT (in module winsound)
@anchor{library/winsound winsound SND_NODEFAULT}@anchor{3f68}
@deffn {Data} winsound.SND_NODEFAULT

If the specified sound cannot be found, do not play the system default sound.
@end deffn

@geindex SND_NOSTOP (in module winsound)
@anchor{library/winsound winsound SND_NOSTOP}@anchor{3f6d}
@deffn {Data} winsound.SND_NOSTOP

Do not interrupt sounds currently playing.
@end deffn

@geindex SND_NOWAIT (in module winsound)
@anchor{library/winsound winsound SND_NOWAIT}@anchor{3f6e}
@deffn {Data} winsound.SND_NOWAIT

Return immediately if the sound driver is busy.

@cartouche
@quotation Note 
This flag is not supported on modern Windows platforms.
@end quotation
@end cartouche
@end deffn

@geindex MB_ICONASTERISK (in module winsound)
@anchor{library/winsound winsound MB_ICONASTERISK}@anchor{3f6f}
@deffn {Data} winsound.MB_ICONASTERISK

Play the @code{SystemDefault} sound.
@end deffn

@geindex MB_ICONEXCLAMATION (in module winsound)
@anchor{library/winsound winsound MB_ICONEXCLAMATION}@anchor{3f70}
@deffn {Data} winsound.MB_ICONEXCLAMATION

Play the @code{SystemExclamation} sound.
@end deffn

@geindex MB_ICONHAND (in module winsound)
@anchor{library/winsound winsound MB_ICONHAND}@anchor{3f71}
@deffn {Data} winsound.MB_ICONHAND

Play the @code{SystemHand} sound.
@end deffn

@geindex MB_ICONQUESTION (in module winsound)
@anchor{library/winsound winsound MB_ICONQUESTION}@anchor{3f72}
@deffn {Data} winsound.MB_ICONQUESTION

Play the @code{SystemQuestion} sound.
@end deffn

@geindex MB_OK (in module winsound)
@anchor{library/winsound winsound MB_OK}@anchor{3f73}
@deffn {Data} winsound.MB_OK

Play the @code{SystemDefault} sound.
@end deffn

@node Unix Specific Services,Superseded Modules,MS Windows Specific Services,The Python Standard Library
@anchor{library/unix doc}@anchor{3f74}@anchor{library/unix unix}@anchor{3f75}@anchor{library/unix unix-specific-services}@anchor{3f76}
@section Unix Specific Services


The modules described in this chapter provide interfaces to features that are
unique to the Unix operating system, or in some cases to some or many variants
of it.  Here’s an overview:

@menu
* posix — The most common POSIX system calls:: 
* pwd — The password database:: 
* spwd — The shadow password database:: 
* grp — The group database:: 
* crypt — Function to check Unix passwords:: 
* termios — POSIX style tty control:: 
* tty — Terminal control functions:: 
* pty — Pseudo-terminal utilities:: 
* fcntl — The fcntl and ioctl system calls:: 
* pipes — Interface to shell pipelines:: 
* resource — Resource usage information:: 
* nis — Interface to Sun’s NIS (Yellow Pages): nis — Interface to Sun’s NIS Yellow Pages. 
* syslog — Unix syslog library routines:: 

@end menu

@node posix — The most common POSIX system calls,pwd — The password database,,Unix Specific Services
@anchor{library/posix doc}@anchor{3f77}@anchor{library/posix module-posix}@anchor{cc}@anchor{library/posix posix-the-most-common-posix-system-calls}@anchor{3f78}
@subsection @code{posix} — The most common POSIX system calls


@geindex module; posix


__________________________________________________________________


This module provides access to operating system functionality that is
standardized by the C Standard and the POSIX standard (a thinly disguised Unix
interface).

@geindex module; os

@strong{Do not import this module directly.}  Instead, import the module @ref{c0,,os},
which provides a @emph{portable} version of this interface.  On Unix, the @ref{c0,,os}
module provides a superset of the @ref{cc,,posix} interface.  On non-Unix operating
systems the @ref{cc,,posix} module is not available, but a subset is always
available through the @ref{c0,,os} interface.  Once @ref{c0,,os} is imported, there is
@emph{no} performance penalty in using it instead of @ref{cc,,posix}.  In addition,
@ref{c0,,os} provides some additional functionality, such as automatically calling
@ref{410,,putenv()} when an entry in @code{os.environ} is changed.

Errors are reported as exceptions; the usual exceptions are given for type
errors, while errors reported by the system calls raise @ref{2a1,,OSError}.

@menu
* Large File Support:: 
* Notable Module Contents:: 

@end menu

@node Large File Support,Notable Module Contents,,posix — The most common POSIX system calls
@anchor{library/posix large-file-support}@anchor{3f79}@anchor{library/posix posix-large-files}@anchor{3f7a}
@subsubsection Large File Support


@geindex large files
@geindex file; large files

Several operating systems (including AIX, HP-UX and Solaris) provide
support for files that are larger than 2 GiB from a C programming model where
int and long are 32-bit values. This is typically accomplished
by defining the relevant size and offset types as 64-bit values. Such files are
sometimes referred to as @emph{large files}.

Large file support is enabled in Python when the size of an @code{off_t} is
larger than a long and the long  long is at least as large
as an @code{off_t}.
It may be necessary to configure and compile Python with certain compiler flags
to enable this mode. For example, with Solaris 2.6 and 2.7 you need to do
something like:

@example
CFLAGS="`getconf LFS_CFLAGS`" OPT="-g -O2 $CFLAGS" \
        ./configure
@end example

On large-file-capable Linux systems, this might work:

@example
CFLAGS='-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64' OPT="-g -O2 $CFLAGS" \
        ./configure
@end example

@node Notable Module Contents,,Large File Support,posix — The most common POSIX system calls
@anchor{library/posix notable-module-contents}@anchor{3f7b}@anchor{library/posix posix-contents}@anchor{3f7c}
@subsubsection Notable Module Contents


In addition to many functions described in the @ref{c0,,os} module documentation,
@ref{cc,,posix} defines the following data item:

@geindex environ (in module posix)
@anchor{library/posix posix environ}@anchor{3f7d}
@deffn {Data} posix.environ

A dictionary representing the string environment at the time the interpreter
was started. Keys and values are bytes on Unix and str on Windows. For
example, @code{environ[b'HOME']} (@code{environ['HOME']} on Windows) is the
pathname of your home directory, equivalent to @code{getenv("HOME")} in C.

Modifying this dictionary does not affect the string environment passed on by
@ref{2391,,execv()}, @ref{59f,,popen()} or @ref{fbb,,system()}; if you need to
change the environment, pass @code{environ} to @ref{c8c,,execve()} or add
variable assignments and export statements to the command string for
@ref{fbb,,system()} or @ref{59f,,popen()}.


Changed in version 3.2: On Unix, keys and values are bytes.


@cartouche
@quotation Note 
The @ref{c0,,os} module provides an alternate implementation of @code{environ}
which updates the environment on modification. Note also that updating
@ref{d60,,os.environ} will render this dictionary obsolete. Use of the
@ref{c0,,os} module version of this is recommended over direct access to the
@ref{cc,,posix} module.
@end quotation
@end cartouche
@end deffn

@node pwd — The password database,spwd — The shadow password database,posix — The most common POSIX system calls,Unix Specific Services
@anchor{library/pwd doc}@anchor{3f7e}@anchor{library/pwd module-pwd}@anchor{d1}@anchor{library/pwd pwd-the-password-database}@anchor{3f7f}
@subsection @code{pwd} — The password database


@geindex module; pwd


__________________________________________________________________


This module provides access to the Unix user account and password database.  It
is available on all Unix versions.

Password database entries are reported as a tuple-like object, whose attributes
correspond to the members of the @code{passwd} structure (Attribute field below,
see @code{<pwd.h>}):


@multitable {xxxxxxxxx} {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Index

@tab

Attribute

@tab

Meaning

@item

0

@tab

@code{pw_name}

@tab

Login name

@item

1

@tab

@code{pw_passwd}

@tab

Optional encrypted password

@item

2

@tab

@code{pw_uid}

@tab

Numerical user ID

@item

3

@tab

@code{pw_gid}

@tab

Numerical group ID

@item

4

@tab

@code{pw_gecos}

@tab

User name or comment field

@item

5

@tab

@code{pw_dir}

@tab

User home directory

@item

6

@tab

@code{pw_shell}

@tab

User command interpreter

@end multitable


The uid and gid items are integers, all others are strings. @ref{5bb,,KeyError} is
raised if the entry asked for cannot be found.

@cartouche
@quotation Note 
@geindex module; crypt

In traditional Unix the field @code{pw_passwd} usually contains a password
encrypted with a DES derived algorithm (see module @ref{27,,crypt}).  However most
modern unices  use a so-called @emph{shadow password} system.  On those unices the
@emph{pw_passwd} field only contains an asterisk (@code{'*'}) or the  letter @code{'x'}
where the encrypted password is stored in a file @code{/etc/shadow} which is
not world readable.  Whether the @emph{pw_passwd} field contains anything useful is
system-dependent.  If available, the @ref{ec,,spwd} module should be used where
access to the encrypted password is required.
@end quotation
@end cartouche

It defines the following items:

@geindex getpwuid() (in module pwd)
@anchor{library/pwd pwd getpwuid}@anchor{12df}
@deffn {Function} pwd.getpwuid (uid)

Return the password database entry for the given numeric user ID.
@end deffn

@geindex getpwnam() (in module pwd)
@anchor{library/pwd pwd getpwnam}@anchor{12f1}
@deffn {Function} pwd.getpwnam (name)

Return the password database entry for the given user name.
@end deffn

@geindex getpwall() (in module pwd)
@anchor{library/pwd pwd getpwall}@anchor{3f80}
@deffn {Function} pwd.getpwall ()

Return a list of all available password database entries, in arbitrary order.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{86,,grp}

An interface to the group database, similar to this.

@item Module @ref{ec,,spwd}

An interface to the shadow password database, similar to this.
@end table


@node spwd — The shadow password database,grp — The group database,pwd — The password database,Unix Specific Services
@anchor{library/spwd doc}@anchor{3f81}@anchor{library/spwd module-spwd}@anchor{ec}@anchor{library/spwd spwd-the-shadow-password-database}@anchor{3f82}
@subsection @code{spwd} — The shadow password database


@geindex module; spwd


__________________________________________________________________


This module provides access to the Unix shadow password database. It is
available on various Unix versions.

You must have enough privileges to access the shadow password database (this
usually means you have to be root).

Shadow password database entries are reported as a tuple-like object, whose
attributes correspond to the members of the @code{spwd} structure (Attribute field
below, see @code{<shadow.h>}):


@multitable {xxxxxxxxx} {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Index

@tab

Attribute

@tab

Meaning

@item

0

@tab

@code{sp_namp}

@tab

Login name

@item

1

@tab

@code{sp_pwdp}

@tab

Encrypted password

@item

2

@tab

@code{sp_lstchg}

@tab

Date of last change

@item

3

@tab

@code{sp_min}

@tab

Minimal number of days between
changes

@item

4

@tab

@code{sp_max}

@tab

Maximum number of days between
changes

@item

5

@tab

@code{sp_warn}

@tab

Number of days before password
expires to warn user about it

@item

6

@tab

@code{sp_inact}

@tab

Number of days after password
expires until account is
disabled

@item

7

@tab

@code{sp_expire}

@tab

Number of days since 1970-01-01
when account expires

@item

8

@tab

@code{sp_flag}

@tab

Reserved

@end multitable


The sp_namp and sp_pwdp items are strings, all others are integers.
@ref{5bb,,KeyError} is raised if the entry asked for cannot be found.

The following functions are defined:

@geindex getspnam() (in module spwd)
@anchor{library/spwd spwd getspnam}@anchor{894}
@deffn {Function} spwd.getspnam (name)

Return the shadow password database entry for the given user name.


Changed in version 3.6: Raises a @ref{895,,PermissionError} instead of @ref{5bb,,KeyError} if the user
doesn’t have privileges.

@end deffn

@geindex getspall() (in module spwd)
@anchor{library/spwd spwd getspall}@anchor{3f83}
@deffn {Function} spwd.getspall ()

Return a list of all available shadow password database entries, in arbitrary
order.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{86,,grp}

An interface to the group database, similar to this.

@item Module @ref{d1,,pwd}

An interface to the normal password database, similar to this.
@end table


@node grp — The group database,crypt — Function to check Unix passwords,spwd — The shadow password database,Unix Specific Services
@anchor{library/grp doc}@anchor{3f84}@anchor{library/grp grp-the-group-database}@anchor{3f85}@anchor{library/grp module-grp}@anchor{86}
@subsection @code{grp} — The group database


@geindex module; grp


__________________________________________________________________


This module provides access to the Unix group database. It is available on all
Unix versions.

Group database entries are reported as a tuple-like object, whose attributes
correspond to the members of the @code{group} structure (Attribute field below, see
@code{<pwd.h>}):


@multitable {xxxxxxxxx} {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Index

@tab

Attribute

@tab

Meaning

@item

0

@tab

gr_name

@tab

the name of the group

@item

1

@tab

gr_passwd

@tab

the (encrypted) group password;
often empty

@item

2

@tab

gr_gid

@tab

the numerical group ID

@item

3

@tab

gr_mem

@tab

all the group member’s  user
names

@end multitable


The gid is an integer, name and password are strings, and the member list is a
list of strings. (Note that most users are not explicitly listed as members of
the group they are in according to the password database.  Check both databases
to get complete membership information.  Also note that a @code{gr_name} that
starts with a @code{+} or @code{-} is likely to be a YP/NIS reference and may not be
accessible via @ref{12f2,,getgrnam()} or @ref{877,,getgrgid()}.)

It defines the following items:

@geindex getgrgid() (in module grp)
@anchor{library/grp grp getgrgid}@anchor{877}
@deffn {Function} grp.getgrgid (gid)

Return the group database entry for the given numeric group ID. @ref{5bb,,KeyError}
is raised if the entry asked for cannot be found.


Deprecated since version 3.6: Since Python 3.6 the support of non-integer arguments like floats or
strings in @ref{877,,getgrgid()} is deprecated.

@end deffn

@geindex getgrnam() (in module grp)
@anchor{library/grp grp getgrnam}@anchor{12f2}
@deffn {Function} grp.getgrnam (name)

Return the group database entry for the given group name. @ref{5bb,,KeyError} is
raised if the entry asked for cannot be found.
@end deffn

@geindex getgrall() (in module grp)
@anchor{library/grp grp getgrall}@anchor{3f86}
@deffn {Function} grp.getgrall ()

Return a list of all available group entries, in arbitrary order.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{d1,,pwd}

An interface to the user database, similar to this.

@item Module @ref{ec,,spwd}

An interface to the shadow password database, similar to this.
@end table


@node crypt — Function to check Unix passwords,termios — POSIX style tty control,grp — The group database,Unix Specific Services
@anchor{library/crypt doc}@anchor{3f87}@anchor{library/crypt crypt-function-to-check-unix-passwords}@anchor{3f88}@anchor{library/crypt module-crypt}@anchor{27}
@subsection @code{crypt} — Function to check Unix passwords


@geindex module; crypt

@strong{Source code:} Lib/crypt.py@footnote{https://github.com/python/cpython/tree/main/Lib/crypt.py}

@geindex crypt(3)
@geindex cipher; DES


__________________________________________________________________


This module implements an interface to the @code{crypt(3)} routine, which is
a one-way hash function based upon a modified DES algorithm; see the Unix man
page for further details.  Possible uses include storing hashed passwords
so you can check passwords without storing the actual password, or attempting
to crack Unix passwords with a dictionary.

@geindex crypt(3)

Notice that the behavior of this module depends on the actual implementation  of
the @code{crypt(3)} routine in the running system.  Therefore, any
extensions available on the current implementation will also  be available on
this module.

@ref{1629,,Availability}: Unix. Not available on VxWorks.

@menu
* Hashing Methods:: 
* Module Attributes:: 
* Module Functions: Module Functions<2>. 
* Examples: Examples<34>. 

@end menu

@node Hashing Methods,Module Attributes,,crypt — Function to check Unix passwords
@anchor{library/crypt hashing-methods}@anchor{3f89}
@subsubsection Hashing Methods



New in version 3.3.


The @ref{27,,crypt} module defines the list of hashing methods (not all methods
are available on all platforms):

@geindex METHOD_SHA512 (in module crypt)
@anchor{library/crypt crypt METHOD_SHA512}@anchor{3f8a}
@deffn {Data} crypt.METHOD_SHA512

A Modular Crypt Format method with 16 character salt and 86 character
hash based on the SHA-512 hash function.  This is the strongest method.
@end deffn

@geindex METHOD_SHA256 (in module crypt)
@anchor{library/crypt crypt METHOD_SHA256}@anchor{3f8b}
@deffn {Data} crypt.METHOD_SHA256

Another Modular Crypt Format method with 16 character salt and 43
character hash based on the SHA-256 hash function.
@end deffn

@geindex METHOD_BLOWFISH (in module crypt)
@anchor{library/crypt crypt METHOD_BLOWFISH}@anchor{3f8c}
@deffn {Data} crypt.METHOD_BLOWFISH

Another Modular Crypt Format method with 22 character salt and 31
character hash based on the Blowfish cipher.


New in version 3.7.

@end deffn

@geindex METHOD_MD5 (in module crypt)
@anchor{library/crypt crypt METHOD_MD5}@anchor{3f8d}
@deffn {Data} crypt.METHOD_MD5

Another Modular Crypt Format method with 8 character salt and 22
character hash based on the MD5 hash function.
@end deffn

@geindex METHOD_CRYPT (in module crypt)
@anchor{library/crypt crypt METHOD_CRYPT}@anchor{89f}
@deffn {Data} crypt.METHOD_CRYPT

The traditional method with a 2 character salt and 13 characters of
hash.  This is the weakest method.
@end deffn

@node Module Attributes,Module Functions<2>,Hashing Methods,crypt — Function to check Unix passwords
@anchor{library/crypt module-attributes}@anchor{3f8e}
@subsubsection Module Attributes



New in version 3.3.


@geindex methods (in module crypt)
@anchor{library/crypt crypt methods}@anchor{3f8f}
@deffn {Attribute} crypt.methods

A list of available password hashing algorithms, as
@code{crypt.METHOD_*} objects.  This list is sorted from strongest to
weakest.
@end deffn

@node Module Functions<2>,Examples<34>,Module Attributes,crypt — Function to check Unix passwords
@anchor{library/crypt module-functions}@anchor{3f90}
@subsubsection Module Functions


The @ref{27,,crypt} module defines the following functions:

@geindex crypt() (in module crypt)
@anchor{library/crypt crypt crypt}@anchor{3f91}
@deffn {Function} crypt.crypt (word, salt=None)

@emph{word} will usually be a user’s password as typed at a prompt or  in a graphical
interface.  The optional @emph{salt} is either a string as returned from
@ref{64e,,mksalt()}, one of the @code{crypt.METHOD_*} values (though not all
may be available on all platforms), or a full encrypted password
including salt, as returned by this function.  If @emph{salt} is not
provided, the strongest method will be used (as returned by
@ref{3f8f,,methods()}).

Checking a password is usually done by passing the plain-text password
as @emph{word} and the full results of a previous @ref{27,,crypt()} call,
which should be the same as the results of this call.

@emph{salt} (either a random 2 or 16 character string, possibly prefixed with
@code{$digit$} to indicate the method) which will be used to perturb the
encryption algorithm.  The characters in @emph{salt} must be in the set
@code{[./a-zA-Z0-9]}, with the exception of Modular Crypt Format which
prefixes a @code{$digit$}.

Returns the hashed password as a string, which will be composed of
characters from the same alphabet as the salt.

@geindex crypt(3)

Since a few @code{crypt(3)} extensions allow different values, with
different sizes in the @emph{salt}, it is recommended to use  the full crypted
password as salt when checking for a password.


Changed in version 3.3: Accept @code{crypt.METHOD_*} values in addition to strings for @emph{salt}.

@end deffn

@geindex mksalt() (in module crypt)
@anchor{library/crypt crypt mksalt}@anchor{64e}
@deffn {Function} crypt.mksalt (method=None, *, rounds=None)

Return a randomly generated salt of the specified method.  If no
@emph{method} is given, the strongest method available as returned by
@ref{3f8f,,methods()} is used.

The return value is a string suitable for passing as the @emph{salt} argument
to @ref{27,,crypt()}.

@emph{rounds} specifies the number of rounds for @code{METHOD_SHA256},
@code{METHOD_SHA512} and @code{METHOD_BLOWFISH}.
For @code{METHOD_SHA256} and @code{METHOD_SHA512} it must be an integer between
@code{1000} and @code{999_999_999}, the default is @code{5000}.  For
@code{METHOD_BLOWFISH} it must be a power of two between @code{16} (2@w{^4})
and @code{2_147_483_648} (2@w{^31}), the default is @code{4096}
(2@w{^12}).


New in version 3.3.



Changed in version 3.7: Added the @emph{rounds} parameter.

@end deffn

@node Examples<34>,,Module Functions<2>,crypt — Function to check Unix passwords
@anchor{library/crypt examples}@anchor{3f92}
@subsubsection Examples


A simple example illustrating typical use (a constant-time comparison
operation is needed to limit exposure to timing attacks.
@ref{c5b,,hmac.compare_digest()} is suitable for this purpose):

@example
import pwd
import crypt
import getpass
from hmac import compare_digest as compare_hash

def login():
    username = input('Python login: ')
    cryptedpasswd = pwd.getpwnam(username)[1]
    if cryptedpasswd:
        if cryptedpasswd == 'x' or cryptedpasswd == '*':
            raise ValueError('no support for shadow passwords')
        cleartext = getpass.getpass()
        return compare_hash(crypt.crypt(cleartext, cryptedpasswd), cryptedpasswd)
    else:
        return True
@end example

To generate a hash of a password using the strongest available method and
check it against the original:

@example
import crypt
from hmac import compare_digest as compare_hash

hashed = crypt.crypt(plaintext)
if not compare_hash(hashed, crypt.crypt(plaintext, hashed)):
    raise ValueError("hashed version doesn't validate against original")
@end example

@node termios — POSIX style tty control,tty — Terminal control functions,crypt — Function to check Unix passwords,Unix Specific Services
@anchor{library/termios doc}@anchor{3f93}@anchor{library/termios module-termios}@anchor{fe}@anchor{library/termios termios-posix-style-tty-control}@anchor{3f94}
@subsection @code{termios} — POSIX style tty control


@geindex module; termios

@geindex POSIX; I/O control
@geindex tty; I/O control


__________________________________________________________________


This module provides an interface to the POSIX calls for tty I/O control. For a
complete description of these calls, see @code{termios(3)} Unix manual
page.  It is only available for those Unix versions that support POSIX
@emph{termios} style tty I/O control configured during installation.

All functions in this module take a file descriptor @emph{fd} as their first
argument.  This can be an integer file descriptor, such as returned by
@code{sys.stdin.fileno()}, or a @ref{d68,,file object}, such as @code{sys.stdin} itself.

This module also defines all the constants needed to work with the functions
provided here; these have the same name as their counterparts in C.  Please
refer to your system documentation for more information on using these terminal
control interfaces.

The module defines the following functions:

@geindex tcgetattr() (in module termios)
@anchor{library/termios termios tcgetattr}@anchor{3f95}
@deffn {Function} termios.tcgetattr (fd)

Return a list containing the tty attributes for file descriptor @emph{fd}, as
follows: @code{[iflag, oflag, cflag, lflag, ispeed, ospeed, cc]} where @emph{cc} is a
list of the tty special characters (each a string of length 1, except the
items with indices @code{VMIN} and @code{VTIME}, which are integers when
these fields are defined).  The interpretation of the flags and the speeds as
well as the indexing in the @emph{cc} array must be done using the symbolic
constants defined in the @ref{fe,,termios} module.
@end deffn

@geindex tcsetattr() (in module termios)
@anchor{library/termios termios tcsetattr}@anchor{3f96}
@deffn {Function} termios.tcsetattr (fd, when, attributes)

Set the tty attributes for file descriptor @emph{fd} from the @emph{attributes}, which is
a list like the one returned by @ref{3f95,,tcgetattr()}.  The @emph{when} argument
determines when the attributes are changed: @code{TCSANOW} to change
immediately, @code{TCSADRAIN} to change after transmitting all queued output,
or @code{TCSAFLUSH} to change after transmitting all queued output and
discarding all queued input.
@end deffn

@geindex tcsendbreak() (in module termios)
@anchor{library/termios termios tcsendbreak}@anchor{3f97}
@deffn {Function} termios.tcsendbreak (fd, duration)

Send a break on file descriptor @emph{fd}.  A zero @emph{duration} sends a break for
0.25–0.5 seconds; a nonzero @emph{duration} has a system dependent meaning.
@end deffn

@geindex tcdrain() (in module termios)
@anchor{library/termios termios tcdrain}@anchor{3f98}
@deffn {Function} termios.tcdrain (fd)

Wait until all output written to file descriptor @emph{fd} has been transmitted.
@end deffn

@geindex tcflush() (in module termios)
@anchor{library/termios termios tcflush}@anchor{3f99}
@deffn {Function} termios.tcflush (fd, queue)

Discard queued data on file descriptor @emph{fd}.  The @emph{queue} selector specifies
which queue: @code{TCIFLUSH} for the input queue, @code{TCOFLUSH} for the
output queue, or @code{TCIOFLUSH} for both queues.
@end deffn

@geindex tcflow() (in module termios)
@anchor{library/termios termios tcflow}@anchor{3f9a}
@deffn {Function} termios.tcflow (fd, action)

Suspend or resume input or output on file descriptor @emph{fd}.  The @emph{action}
argument can be @code{TCOOFF} to suspend output, @code{TCOON} to restart
output, @code{TCIOFF} to suspend input, or @code{TCION} to restart input.
@end deffn

@geindex tcgetwinsize() (in module termios)
@anchor{library/termios termios tcgetwinsize}@anchor{3f9b}
@deffn {Function} termios.tcgetwinsize (fd)

Return a tuple @code{(ws_row, ws_col)} containing the tty window size for file
descriptor @emph{fd}. Requires @code{termios.TIOCGWINSZ} or
@code{termios.TIOCGSIZE}.


New in version 3.11.

@end deffn

@geindex tcsetwinsize() (in module termios)
@anchor{library/termios termios tcsetwinsize}@anchor{3f9c}
@deffn {Function} termios.tcsetwinsize (fd, winsize)

@quotation

Set the tty window size for file descriptor @emph{fd} from @emph{winsize}, which is
a two-item tuple @code{(ws_row, ws_col)} like the one returned by
@ref{3f9b,,tcgetwinsize()}. Requires at least one of the pairs
(@code{termios.TIOCGWINSZ}, @code{termios.TIOCSWINSZ});
(@code{termios.TIOCGSIZE}, @code{termios.TIOCSSIZE}) to be defined.
@end quotation


New in version 3.11.

@end deffn


@subsubheading See also



@table @asis

@item Module @ref{11c,,tty}

Convenience functions for common terminal control operations.
@end table


@menu
* Example: Example<15>. 

@end menu

@node Example<15>,,,termios — POSIX style tty control
@anchor{library/termios example}@anchor{3f9d}@anchor{library/termios termios-example}@anchor{3f9e}
@subsubsection Example


Here’s a function that prompts for a password with echoing turned off.  Note the
technique using a separate @ref{3f95,,tcgetattr()} call and a @ref{f6d,,try} …
@ref{4c1,,finally} statement to ensure that the old tty attributes are restored
exactly no matter what happens:

@example
def getpass(prompt="Password: "):
    import termios, sys
    fd = sys.stdin.fileno()
    old = termios.tcgetattr(fd)
    new = termios.tcgetattr(fd)
    new[3] = new[3] & ~termios.ECHO          # lflags
    try:
        termios.tcsetattr(fd, termios.TCSADRAIN, new)
        passwd = input(prompt)
    finally:
        termios.tcsetattr(fd, termios.TCSADRAIN, old)
    return passwd
@end example

@node tty — Terminal control functions,pty — Pseudo-terminal utilities,termios — POSIX style tty control,Unix Specific Services
@anchor{library/tty doc}@anchor{3f9f}@anchor{library/tty module-tty}@anchor{11c}@anchor{library/tty tty-terminal-control-functions}@anchor{3fa0}
@subsection @code{tty} — Terminal control functions


@geindex module; tty

@strong{Source code:} Lib/tty.py@footnote{https://github.com/python/cpython/tree/main/Lib/tty.py}


__________________________________________________________________


The @ref{11c,,tty} module defines functions for putting the tty into cbreak and raw
modes.

Because it requires the @ref{fe,,termios} module, it will work only on Unix.

The @ref{11c,,tty} module defines the following functions:

@geindex setraw() (in module tty)
@anchor{library/tty tty setraw}@anchor{3fa1}
@deffn {Function} tty.setraw (fd, when=termios.TCSAFLUSH)

Change the mode of the file descriptor @emph{fd} to raw. If @emph{when} is omitted, it
defaults to @code{termios.TCSAFLUSH}, and is passed to
@ref{3f96,,termios.tcsetattr()}.
@end deffn

@geindex setcbreak() (in module tty)
@anchor{library/tty tty setcbreak}@anchor{3fa2}
@deffn {Function} tty.setcbreak (fd, when=termios.TCSAFLUSH)

Change the mode of file descriptor @emph{fd} to cbreak. If @emph{when} is omitted, it
defaults to @code{termios.TCSAFLUSH}, and is passed to
@ref{3f96,,termios.tcsetattr()}.
@end deffn


@subsubheading See also



@table @asis

@item Module @ref{fe,,termios}

Low-level terminal control interface.
@end table


@node pty — Pseudo-terminal utilities,fcntl — The fcntl and ioctl system calls,tty — Terminal control functions,Unix Specific Services
@anchor{library/pty doc}@anchor{3fa3}@anchor{library/pty module-pty}@anchor{d0}@anchor{library/pty pty-pseudo-terminal-utilities}@anchor{3fa4}
@subsection @code{pty} — Pseudo-terminal utilities


@geindex module; pty

@strong{Source code:} Lib/pty.py@footnote{https://github.com/python/cpython/tree/main/Lib/pty.py}


__________________________________________________________________


The @ref{d0,,pty} module defines operations for handling the pseudo-terminal
concept: starting another process and being able to write to and read from its
controlling terminal programmatically.

Pseudo-terminal handling is highly platform dependent. This code is mainly
tested on Linux, FreeBSD, and macOS (it is supposed to work on other POSIX
platforms but it’s not been thoroughly tested).

The @ref{d0,,pty} module defines the following functions:

@geindex fork() (in module pty)
@anchor{library/pty pty fork}@anchor{3fa5}
@deffn {Function} pty.fork ()

Fork. Connect the child’s controlling terminal to a pseudo-terminal. Return
value is @code{(pid, fd)}. Note that the child  gets @emph{pid} 0, and the @emph{fd} is
@emph{invalid}. The parent’s return value is the @emph{pid} of the child, and @emph{fd} is a
file descriptor connected to the child’s controlling terminal (and also to the
child’s standard input and output).
@end deffn

@geindex openpty() (in module pty)
@anchor{library/pty pty openpty}@anchor{3fa6}
@deffn {Function} pty.openpty ()

Open a new pseudo-terminal pair, using @ref{23c4,,os.openpty()} if possible, or
emulation code for generic Unix systems. Return a pair of file descriptors
@code{(master, slave)}, for the master and the slave end, respectively.
@end deffn

@geindex spawn() (in module pty)
@anchor{library/pty pty spawn}@anchor{b03}
@deffn {Function} pty.spawn (argv[, master_read[, stdin_read]])

Spawn a process, and connect its controlling terminal with the current
process’s standard io. This is often used to baffle programs which insist on
reading from the controlling terminal. It is expected that the process
spawned behind the pty will eventually terminate, and when it does @emph{spawn}
will return.

A loop copies STDIN of the current process to the child and data received
from the child to STDOUT of the current process. It is not signaled to the
child if STDIN of the current process closes down.

The functions @emph{master_read} and @emph{stdin_read} are passed a file descriptor
which they should read from, and they should always return a byte string. In
order to force spawn to return before the child process exits an
empty byte array should be returned to signal end of file.

The default implementation for both functions will read and return up to 1024
bytes each time the function is called. The @emph{master_read} callback is passed
the pseudoterminal’s master file descriptor to read output from the child
process, and @emph{stdin_read} is passed file descriptor 0, to read from the
parent process’s standard input.

Returning an empty byte string from either callback is interpreted as an
end-of-file (EOF) condition, and that callback will not be called after
that. If @emph{stdin_read} signals EOF the controlling terminal can no longer
communicate with the parent process OR the child process. Unless the child
process will quit without any input, @emph{spawn} will then loop forever. If
@emph{master_read} signals EOF the same behavior results (on linux at least).

Return the exit status value from @ref{8f6,,os.waitpid()} on the child process.

@code{waitstatus_to_exitcode()} can be used to convert the exit status into
an exit code.

Raises an @ref{1130,,auditing event} @code{pty.spawn} with argument @code{argv}.


Changed in version 3.4: @ref{b03,,spawn()} now returns the status value from @ref{8f6,,os.waitpid()}
on the child process.

@end deffn

@menu
* Example: Example<16>. 

@end menu

@node Example<16>,,,pty — Pseudo-terminal utilities
@anchor{library/pty example}@anchor{3fa7}
@subsubsection Example


The following program acts like the Unix command @code{script(1)}, using a
pseudo-terminal to record all input and output of a terminal session in a
“typescript”.

@example
import argparse
import os
import pty
import sys
import time

parser = argparse.ArgumentParser()
parser.add_argument('-a', dest='append', action='store_true')
parser.add_argument('-p', dest='use_python', action='store_true')
parser.add_argument('filename', nargs='?', default='typescript')
options = parser.parse_args()

shell = sys.executable if options.use_python else os.environ.get('SHELL', 'sh')
filename = options.filename
mode = 'ab' if options.append else 'wb'

with open(filename, mode) as script:
    def read(fd):
        data = os.read(fd, 1024)
        script.write(data)
        return data

    print('Script started, file is', filename)
    script.write(('Script started on %s\n' % time.asctime()).encode())

    pty.spawn(shell, read)

    script.write(('Script done on %s\n' % time.asctime()).encode())
    print('Script done, file is', filename)
@end example

@node fcntl — The fcntl and ioctl system calls,pipes — Interface to shell pipelines,pty — Pseudo-terminal utilities,Unix Specific Services
@anchor{library/fcntl doc}@anchor{3fa8}@anchor{library/fcntl fcntl-the-fcntl-and-ioctl-system-calls}@anchor{3fa9}@anchor{library/fcntl module-fcntl}@anchor{79}
@subsection @code{fcntl} — The @code{fcntl} and @code{ioctl} system calls


@geindex module; fcntl

@geindex UNIX; file control
@geindex UNIX; I/O control


__________________________________________________________________


This module performs file control and I/O control on file descriptors. It is an
interface to the @code{fcntl()} and @code{ioctl()} Unix routines.  For a
complete description of these calls, see @code{fcntl(2)} and
@code{ioctl(2)} Unix manual pages.

All functions in this module take a file descriptor @emph{fd} as their first
argument.  This can be an integer file descriptor, such as returned by
@code{sys.stdin.fileno()}, or an @ref{503,,io.IOBase} object, such as @code{sys.stdin}
itself, which provides a @ref{23a2,,fileno()} that returns a genuine file
descriptor.


Changed in version 3.3: Operations in this module used to raise an @ref{be8,,IOError} where they now
raise an @ref{2a1,,OSError}.



Changed in version 3.8: The fcntl module now contains @code{F_ADD_SEALS}, @code{F_GET_SEALS}, and
@code{F_SEAL_*} constants for sealing of @ref{515,,os.memfd_create()} file
descriptors.



Changed in version 3.9: On macOS, the fcntl module exposes the @code{F_GETPATH} constant, which obtains
the path of a file from a file descriptor.
On Linux(>=3.15), the fcntl module exposes the @code{F_OFD_GETLK}, @code{F_OFD_SETLK}
and @code{F_OFD_SETLKW} constants, which working with open file description locks.



Changed in version 3.10: On Linux >= 2.6.11, the fcntl module exposes the @code{F_GETPIPE_SZ} and
@code{F_SETPIPE_SZ} constants, which allow to check and modify a pipe’s size
respectively.



Changed in version 3.11: On FreeBSD, the fcntl module exposes the @code{F_DUP2FD} and @code{F_DUP2FD_CLOEXEC}
constants, which allow to duplicate a file descriptor, the latter setting
@code{FD_CLOEXEC} flag in addition.


The module defines the following functions:

@geindex fcntl() (in module fcntl)
@anchor{library/fcntl fcntl fcntl}@anchor{2b20}
@deffn {Function} fcntl.fcntl (fd, cmd, arg=0)

Perform the operation @emph{cmd} on file descriptor @emph{fd} (file objects providing
a @ref{23a2,,fileno()} method are accepted as well).  The values used
for @emph{cmd} are operating system dependent, and are available as constants
in the @ref{79,,fcntl} module, using the same names as used in the relevant C
header files. The argument @emph{arg} can either be an integer value, or a
@ref{2f0,,bytes} object. With an integer value, the return value of this
function is the integer return value of the C @code{fcntl()} call.  When
the argument is bytes it represents a binary structure, e.g. created by
@ref{e25,,struct.pack()}. The binary data is copied to a buffer whose address is
passed to the C @code{fcntl()} call.  The return value after a successful
call is the contents of the buffer, converted to a @ref{2f0,,bytes} object.
The length of the returned object will be the same as the length of the
@emph{arg} argument. This is limited to 1024 bytes. If the information returned
in the buffer by the operating system is larger than 1024 bytes, this is
most likely to result in a segmentation violation or a more subtle data
corruption.

If the @code{fcntl()} fails, an @ref{2a1,,OSError} is raised.

Raises an @ref{1130,,auditing event} @code{fcntl.fcntl} with arguments @code{fd}, @code{cmd}, @code{arg}.
@end deffn

@geindex ioctl() (in module fcntl)
@anchor{library/fcntl fcntl ioctl}@anchor{fd7}
@deffn {Function} fcntl.ioctl (fd, request, arg=0, mutate_flag=True)

This function is identical to the @ref{2b20,,fcntl()} function, except
that the argument handling is even more complicated.

The @emph{request} parameter is limited to values that can fit in 32-bits.
Additional constants of interest for use as the @emph{request} argument can be
found in the @ref{fe,,termios} module, under the same names as used in
the relevant C header files.

The parameter @emph{arg} can be one of an integer, an object supporting the
read-only buffer interface (like @ref{2f0,,bytes}) or an object supporting
the read-write buffer interface (like @ref{2f1,,bytearray}).

In all but the last case, behaviour is as for the @ref{2b20,,fcntl()}
function.

If a mutable buffer is passed, then the behaviour is determined by the value of
the @emph{mutate_flag} parameter.

If it is false, the buffer’s mutability is ignored and behaviour is as for a
read-only buffer, except that the 1024 byte limit mentioned above is avoided –
so long as the buffer you pass is at least as long as what the operating system
wants to put there, things should work.

If @emph{mutate_flag} is true (the default), then the buffer is (in effect) passed
to the underlying @ref{fd7,,ioctl()} system call, the latter’s return code is
passed back to the calling Python, and the buffer’s new contents reflect the
action of the @ref{fd7,,ioctl()}.  This is a slight simplification, because if the
supplied buffer is less than 1024 bytes long it is first copied into a static
buffer 1024 bytes long which is then passed to @ref{fd7,,ioctl()} and copied back
into the supplied buffer.

If the @code{ioctl()} fails, an @ref{2a1,,OSError} exception is raised.

An example:

@example
>>> import array, fcntl, struct, termios, os
>>> os.getpgrp()
13341
>>> struct.unpack('h', fcntl.ioctl(0, termios.TIOCGPGRP, "  "))[0]
13341
>>> buf = array.array('h', [0])
>>> fcntl.ioctl(0, termios.TIOCGPGRP, buf, 1)
0
>>> buf
array('h', [13341])
@end example

Raises an @ref{1130,,auditing event} @code{fcntl.ioctl} with arguments @code{fd}, @code{request}, @code{arg}.
@end deffn

@geindex flock() (in module fcntl)
@anchor{library/fcntl fcntl flock}@anchor{39f2}
@deffn {Function} fcntl.flock (fd, operation)

Perform the lock operation @emph{operation} on file descriptor @emph{fd} (file objects providing
a @ref{23a2,,fileno()} method are accepted as well). See the Unix manual
@code{flock(2)} for details.  (On some systems, this function is emulated
using @code{fcntl()}.)

If the @code{flock()} fails, an @ref{2a1,,OSError} exception is raised.

Raises an @ref{1130,,auditing event} @code{fcntl.flock} with arguments @code{fd}, @code{operation}.
@end deffn

@geindex lockf() (in module fcntl)
@anchor{library/fcntl fcntl lockf}@anchor{f4d}
@deffn {Function} fcntl.lockf (fd, cmd, len=0, start=0, whence=0)

This is essentially a wrapper around the @ref{2b20,,fcntl()} locking calls.
@emph{fd} is the file descriptor (file objects providing a @ref{23a2,,fileno()}
method are accepted as well) of the file to lock or unlock, and @emph{cmd}
is one of the following values:


@itemize *

@item 
@code{LOCK_UN} – unlock

@item 
@code{LOCK_SH} – acquire a shared lock

@item 
@code{LOCK_EX} – acquire an exclusive lock
@end itemize

When @emph{cmd} is @code{LOCK_SH} or @code{LOCK_EX}, it can also be
bitwise ORed with @code{LOCK_NB} to avoid blocking on lock acquisition.
If @code{LOCK_NB} is used and the lock cannot be acquired, an
@ref{2a1,,OSError} will be raised and the exception will have an @emph{errno}
attribute set to @code{EACCES} or @code{EAGAIN} (depending on the
operating system; for portability, check for both values).  On at least some
systems, @code{LOCK_EX} can only be used if the file descriptor refers to a
file opened for writing.

@emph{len} is the number of bytes to lock, @emph{start} is the byte offset at
which the lock starts, relative to @emph{whence}, and @emph{whence} is as with
@ref{2231,,io.IOBase.seek()}, specifically:


@itemize *

@item 
@code{0} – relative to the start of the file (@ref{f8e,,os.SEEK_SET})

@item 
@code{1} – relative to the current buffer position (@ref{f8f,,os.SEEK_CUR})

@item 
@code{2} – relative to the end of the file (@ref{f90,,os.SEEK_END})
@end itemize

The default for @emph{start} is 0, which means to start at the beginning of the file.
The default for @emph{len} is 0 which means to lock to the end of the file.  The
default for @emph{whence} is also 0.

Raises an @ref{1130,,auditing event} @code{fcntl.lockf} with arguments @code{fd}, @code{cmd}, @code{len}, @code{start}, @code{whence}.
@end deffn

Examples (all on a SVR4 compliant system):

@example
import struct, fcntl, os

f = open(...)
rv = fcntl.fcntl(f, fcntl.F_SETFL, os.O_NDELAY)

lockdata = struct.pack('hhllhh', fcntl.F_WRLCK, 0, 0, 0, 0, 0)
rv = fcntl.fcntl(f, fcntl.F_SETLKW, lockdata)
@end example

Note that in the first example the return value variable @emph{rv} will hold an
integer value; in the second example it will hold a @ref{2f0,,bytes} object.  The
structure lay-out for the @emph{lockdata} variable is system dependent — therefore
using the @ref{39f2,,flock()} call may be better.


@subsubheading See also



@table @asis

@item Module @ref{c0,,os}

If the locking flags @ref{f91,,O_SHLOCK} and @ref{f92,,O_EXLOCK} are
present in the @ref{c0,,os} module (on BSD only), the @ref{8ee,,os.open()}
function provides an alternative to the @ref{f4d,,lockf()} and @ref{39f2,,flock()}
functions.
@end table


@node pipes — Interface to shell pipelines,resource — Resource usage information,fcntl — The fcntl and ioctl system calls,Unix Specific Services
@anchor{library/pipes doc}@anchor{3faa}@anchor{library/pipes module-pipes}@anchor{c7}@anchor{library/pipes pipes-interface-to-shell-pipelines}@anchor{3fab}
@subsection @code{pipes} — Interface to shell pipelines


@geindex module; pipes

@strong{Source code:} Lib/pipes.py@footnote{https://github.com/python/cpython/tree/main/Lib/pipes.py}


__________________________________________________________________


The @ref{c7,,pipes} module defines a class to abstract the concept of a @emph{pipeline}
— a sequence of converters from one file to  another.

Because the module uses @code{/bin/sh} command lines, a POSIX or compatible
shell for @ref{fbb,,os.system()} and @ref{59f,,os.popen()} is required.

@ref{1629,,Availability}: Unix. Not available on VxWorks.

The @ref{c7,,pipes} module defines the following class:

@geindex Template (class in pipes)
@anchor{library/pipes pipes Template}@anchor{3fac}
@deffn {Class} pipes.Template

An abstraction of a pipeline.
@end deffn

Example:

@example
>>> import pipes
>>> t = pipes.Template()
>>> t.append('tr a-z A-Z', '--')
>>> f = t.open('pipefile', 'w')
>>> f.write('hello world')
>>> f.close()
>>> open('pipefile').read()
'HELLO WORLD'
@end example

@menu
* Template Objects:: 

@end menu

@node Template Objects,,,pipes — Interface to shell pipelines
@anchor{library/pipes id1}@anchor{3fad}@anchor{library/pipes template-objects}@anchor{3fae}
@subsubsection Template Objects


Template objects following methods:

@geindex reset() (pipes.Template method)
@anchor{library/pipes pipes Template reset}@anchor{3faf}
@deffn {Method} Template.reset ()

Restore a pipeline template to its initial state.
@end deffn

@geindex clone() (pipes.Template method)
@anchor{library/pipes pipes Template clone}@anchor{3fb0}
@deffn {Method} Template.clone ()

Return a new, equivalent, pipeline template.
@end deffn

@geindex debug() (pipes.Template method)
@anchor{library/pipes pipes Template debug}@anchor{3fb1}
@deffn {Method} Template.debug (flag)

If @emph{flag} is true, turn debugging on. Otherwise, turn debugging off. When
debugging is on, commands to be executed are printed, and the shell is given
@code{set -x} command to be more verbose.
@end deffn

@geindex append() (pipes.Template method)
@anchor{library/pipes pipes Template append}@anchor{3fb2}
@deffn {Method} Template.append (cmd, kind)

Append a new action at the end. The @emph{cmd} variable must be a valid bourne shell
command. The @emph{kind} variable consists of two letters.

The first letter can be either of @code{'-'} (which means the command reads its
standard input), @code{'f'} (which means the commands reads a given file on the
command line) or @code{'.'} (which means the commands reads no input, and hence
must be first.)

Similarly, the second letter can be either of @code{'-'} (which means  the command
writes to standard output), @code{'f'} (which means the  command writes a file on
the command line) or @code{'.'} (which means the command does not write anything,
and hence must be last.)
@end deffn

@geindex prepend() (pipes.Template method)
@anchor{library/pipes pipes Template prepend}@anchor{3fb3}
@deffn {Method} Template.prepend (cmd, kind)

Add a new action at the beginning. See @ref{3fb2,,append()} for explanations of the
arguments.
@end deffn

@geindex open() (pipes.Template method)
@anchor{library/pipes pipes Template open}@anchor{3fb4}
@deffn {Method} Template.open (file, mode)

Return a file-like object, open to @emph{file}, but read from or written to by the
pipeline.  Note that only one of @code{'r'}, @code{'w'} may be given.
@end deffn

@geindex copy() (pipes.Template method)
@anchor{library/pipes pipes Template copy}@anchor{3fb5}
@deffn {Method} Template.copy (infile, outfile)

Copy @emph{infile} to @emph{outfile} through the pipe.
@end deffn

@node resource — Resource usage information,nis — Interface to Sun’s NIS Yellow Pages,pipes — Interface to shell pipelines,Unix Specific Services
@anchor{library/resource doc}@anchor{3fb6}@anchor{library/resource module-resource}@anchor{db}@anchor{library/resource resource-resource-usage-information}@anchor{3fb7}
@subsection @code{resource} — Resource usage information


@geindex module; resource


__________________________________________________________________


This module provides basic mechanisms for measuring and controlling system
resources utilized by a program.

Symbolic constants are used to specify particular system resources and to
request usage information about either the current process or its children.

An @ref{2a1,,OSError} is raised on syscall failure.

@geindex error
@anchor{library/resource resource error}@anchor{3fb8}
@deffn {Exception} resource.error

A deprecated alias of @ref{2a1,,OSError}.


Changed in version 3.3: Following 
@geindex Python Enhancement Proposals; PEP 3151
PEP 3151@footnote{https://www.python.org/dev/peps/pep-3151}, this class was made an alias of @ref{2a1,,OSError}.

@end deffn

@menu
* Resource Limits:: 
* Resource Usage:: 

@end menu

@node Resource Limits,Resource Usage,,resource — Resource usage information
@anchor{library/resource resource-limits}@anchor{3fb9}
@subsubsection Resource Limits


Resources usage can be limited using the @ref{3996,,setrlimit()} function described
below. Each resource is controlled by a pair of limits: a soft limit and a hard
limit. The soft limit is the current limit, and may be lowered or raised by a
process over time. The soft limit can never exceed the hard limit. The hard
limit can be lowered to any value greater than the soft limit, but not raised.
(Only processes with the effective UID of the super-user can raise a hard
limit.)

The specific resources that can be limited are system dependent. They are
described in the @code{getrlimit(2)} man page.  The resources listed below
are supported when the underlying operating system supports them; resources
which cannot be checked or controlled by the operating system are not defined in
this module for those platforms.

@geindex RLIM_INFINITY (in module resource)
@anchor{library/resource resource RLIM_INFINITY}@anchor{3fba}
@deffn {Data} resource.RLIM_INFINITY

Constant used to represent the limit for an unlimited resource.
@end deffn

@geindex getrlimit() (in module resource)
@anchor{library/resource resource getrlimit}@anchor{3fbb}
@deffn {Function} resource.getrlimit (resource)

Returns a tuple @code{(soft, hard)} with the current soft and hard limits of
@emph{resource}. Raises @ref{17c,,ValueError} if an invalid resource is specified, or
@ref{3fb8,,error} if the underlying system call fails unexpectedly.
@end deffn

@geindex setrlimit() (in module resource)
@anchor{library/resource resource setrlimit}@anchor{3996}
@deffn {Function} resource.setrlimit (resource, limits)

Sets new limits of consumption of @emph{resource}. The @emph{limits} argument must be a
tuple @code{(soft, hard)} of two integers describing the new limits. A value of
@ref{3fba,,RLIM_INFINITY} can be used to request a limit that is
unlimited.

Raises @ref{17c,,ValueError} if an invalid resource is specified, if the new soft
limit exceeds the hard limit, or if a process tries to raise its hard limit.
Specifying a limit of @ref{3fba,,RLIM_INFINITY} when the hard or
system limit for that resource is not unlimited will result in a
@ref{17c,,ValueError}.  A process with the effective UID of super-user can
request any valid limit value, including unlimited, but @ref{17c,,ValueError}
will still be raised if the requested limit exceeds the system imposed
limit.

@code{setrlimit} may also raise @ref{3fb8,,error} if the underlying system call
fails.

VxWorks only supports setting @ref{3fbc,,RLIMIT_NOFILE}.

Raises an @ref{1130,,auditing event} @code{resource.setrlimit} with arguments @code{resource}, @code{limits}.
@end deffn

@geindex prlimit() (in module resource)
@anchor{library/resource resource prlimit}@anchor{b0a}
@deffn {Function} resource.prlimit (pid, resource[, limits])

Combines @ref{3996,,setrlimit()} and @ref{3fbb,,getrlimit()} in one function and
supports to get and set the resources limits of an arbitrary process. If
@emph{pid} is 0, then the call applies to the current process. @emph{resource} and
@emph{limits} have the same meaning as in @ref{3996,,setrlimit()}, except that
@emph{limits} is optional.

When @emph{limits} is not given the function returns the @emph{resource} limit of the
process @emph{pid}. When @emph{limits} is given the @emph{resource} limit of the process is
set and the former resource limit is returned.

Raises @ref{bf1,,ProcessLookupError} when @emph{pid} can’t be found and
@ref{895,,PermissionError} when the user doesn’t have @code{CAP_SYS_RESOURCE} for
the process.

Raises an @ref{1130,,auditing event} @code{resource.prlimit} with arguments @code{pid}, @code{resource}, @code{limits}.

@ref{1629,,Availability}: Linux 2.6.36 or later with glibc 2.13 or later.


New in version 3.4.

@end deffn

These symbols define resources whose consumption can be controlled using the
@ref{3996,,setrlimit()} and @ref{3fbb,,getrlimit()} functions described below. The values of
these symbols are exactly the constants used by C programs.

The Unix man page for @code{getrlimit(2)} lists the available resources.
Note that not all systems use the same symbol or same value to denote the same
resource.  This module does not attempt to mask platform differences — symbols
not defined for a platform will not be available from this module on that
platform.

@geindex RLIMIT_CORE (in module resource)
@anchor{library/resource resource RLIMIT_CORE}@anchor{3997}
@deffn {Data} resource.RLIMIT_CORE

The maximum size (in bytes) of a core file that the current process can create.
This may result in the creation of a partial core file if a larger core would be
required to contain the entire process image.
@end deffn

@geindex RLIMIT_CPU (in module resource)
@anchor{library/resource resource RLIMIT_CPU}@anchor{3fbd}
@deffn {Data} resource.RLIMIT_CPU

The maximum amount of processor time (in seconds) that a process can use. If
this limit is exceeded, a @code{SIGXCPU} signal is sent to the process. (See
the @ref{e5,,signal} module documentation for information about how to catch this
signal and do something useful, e.g. flush open files to disk.)
@end deffn

@geindex RLIMIT_FSIZE (in module resource)
@anchor{library/resource resource RLIMIT_FSIZE}@anchor{3fbe}
@deffn {Data} resource.RLIMIT_FSIZE

The maximum size of a file which the process may create.
@end deffn

@geindex RLIMIT_DATA (in module resource)
@anchor{library/resource resource RLIMIT_DATA}@anchor{3fbf}
@deffn {Data} resource.RLIMIT_DATA

The maximum size (in bytes) of the process’s heap.
@end deffn

@geindex RLIMIT_STACK (in module resource)
@anchor{library/resource resource RLIMIT_STACK}@anchor{3fc0}
@deffn {Data} resource.RLIMIT_STACK

The maximum size (in bytes) of the call stack for the current process.  This only
affects the stack of the main thread in a multi-threaded process.
@end deffn

@geindex RLIMIT_RSS (in module resource)
@anchor{library/resource resource RLIMIT_RSS}@anchor{3fc1}
@deffn {Data} resource.RLIMIT_RSS

The maximum resident set size that should be made available to the process.
@end deffn

@geindex RLIMIT_NPROC (in module resource)
@anchor{library/resource resource RLIMIT_NPROC}@anchor{3fc2}
@deffn {Data} resource.RLIMIT_NPROC

The maximum number of processes the current process may create.
@end deffn

@geindex RLIMIT_NOFILE (in module resource)
@anchor{library/resource resource RLIMIT_NOFILE}@anchor{3fbc}
@deffn {Data} resource.RLIMIT_NOFILE

The maximum number of open file descriptors for the current process.
@end deffn

@geindex RLIMIT_OFILE (in module resource)
@anchor{library/resource resource RLIMIT_OFILE}@anchor{3fc3}
@deffn {Data} resource.RLIMIT_OFILE

The BSD name for @ref{3fbc,,RLIMIT_NOFILE}.
@end deffn

@geindex RLIMIT_MEMLOCK (in module resource)
@anchor{library/resource resource RLIMIT_MEMLOCK}@anchor{3fc4}
@deffn {Data} resource.RLIMIT_MEMLOCK

The maximum address space which may be locked in memory.
@end deffn

@geindex RLIMIT_VMEM (in module resource)
@anchor{library/resource resource RLIMIT_VMEM}@anchor{3fc5}
@deffn {Data} resource.RLIMIT_VMEM

The largest area of mapped memory which the process may occupy.
@end deffn

@geindex RLIMIT_AS (in module resource)
@anchor{library/resource resource RLIMIT_AS}@anchor{3fc6}
@deffn {Data} resource.RLIMIT_AS

The maximum area (in bytes) of address space which may be taken by the process.
@end deffn

@geindex RLIMIT_MSGQUEUE (in module resource)
@anchor{library/resource resource RLIMIT_MSGQUEUE}@anchor{b0b}
@deffn {Data} resource.RLIMIT_MSGQUEUE

The number of bytes that can be allocated for POSIX message queues.

@ref{1629,,Availability}: Linux 2.6.8 or later.


New in version 3.4.

@end deffn

@geindex RLIMIT_NICE (in module resource)
@anchor{library/resource resource RLIMIT_NICE}@anchor{b0c}
@deffn {Data} resource.RLIMIT_NICE

The ceiling for the process’s nice level (calculated as 20 - rlim_cur).

@ref{1629,,Availability}: Linux 2.6.12 or later.


New in version 3.4.

@end deffn

@geindex RLIMIT_RTPRIO (in module resource)
@anchor{library/resource resource RLIMIT_RTPRIO}@anchor{b0d}
@deffn {Data} resource.RLIMIT_RTPRIO

The ceiling of the real-time priority.

@ref{1629,,Availability}: Linux 2.6.12 or later.


New in version 3.4.

@end deffn

@geindex RLIMIT_RTTIME (in module resource)
@anchor{library/resource resource RLIMIT_RTTIME}@anchor{b0e}
@deffn {Data} resource.RLIMIT_RTTIME

The time limit (in microseconds) on CPU time that a process can spend
under real-time scheduling without making a blocking syscall.

@ref{1629,,Availability}: Linux 2.6.25 or later.


New in version 3.4.

@end deffn

@geindex RLIMIT_SIGPENDING (in module resource)
@anchor{library/resource resource RLIMIT_SIGPENDING}@anchor{b0f}
@deffn {Data} resource.RLIMIT_SIGPENDING

The number of signals which the process may queue.

@ref{1629,,Availability}: Linux 2.6.8 or later.


New in version 3.4.

@end deffn

@geindex RLIMIT_SBSIZE (in module resource)
@anchor{library/resource resource RLIMIT_SBSIZE}@anchor{b10}
@deffn {Data} resource.RLIMIT_SBSIZE

The maximum size (in bytes) of socket buffer usage for this user.
This limits the amount of network memory, and hence the amount of mbufs,
that this user may hold at any time.

@ref{1629,,Availability}: FreeBSD 9 or later.


New in version 3.4.

@end deffn

@geindex RLIMIT_SWAP (in module resource)
@anchor{library/resource resource RLIMIT_SWAP}@anchor{b11}
@deffn {Data} resource.RLIMIT_SWAP

The maximum size (in bytes) of the swap space that may be reserved or
used by all of this user id’s processes.
This limit is enforced only if bit 1 of the vm.overcommit sysctl is set.
Please see
tuning(7)@footnote{https://www.freebsd.org/cgi/man.cgi?query=tuning&sektion=7}
for a complete description of this sysctl.

@ref{1629,,Availability}: FreeBSD 9 or later.


New in version 3.4.

@end deffn

@geindex RLIMIT_NPTS (in module resource)
@anchor{library/resource resource RLIMIT_NPTS}@anchor{b12}
@deffn {Data} resource.RLIMIT_NPTS

The maximum number of pseudo-terminals created by this user id.

@ref{1629,,Availability}: FreeBSD 9 or later.


New in version 3.4.

@end deffn

@geindex RLIMIT_KQUEUES (in module resource)
@anchor{library/resource resource RLIMIT_KQUEUES}@anchor{1162}
@deffn {Data} resource.RLIMIT_KQUEUES

The maximum number of kqueues this user id is allowed to create.

@ref{1629,,Availability}: FreeBSD 11 or later.


New in version 3.10.

@end deffn

@node Resource Usage,,Resource Limits,resource — Resource usage information
@anchor{library/resource resource-usage}@anchor{3fc7}
@subsubsection Resource Usage


These functions are used to retrieve resource usage information:

@geindex getrusage() (in module resource)
@anchor{library/resource resource getrusage}@anchor{f93}
@deffn {Function} resource.getrusage (who)

This function returns an object that describes the resources consumed by either
the current process or its children, as specified by the @emph{who} parameter.  The
@emph{who} parameter should be specified using one of the @code{RUSAGE_*}
constants described below.

A simple example:

@example
from resource import *
import time

# a non CPU-bound task
time.sleep(3)
print(getrusage(RUSAGE_SELF))

# a CPU-bound task
for i in range(10 ** 8):
   _ = 1 + 1
print(getrusage(RUSAGE_SELF))
@end example

The fields of the return value each describe how a particular system resource
has been used, e.g. amount of time spent running is user mode or number of times
the process was swapped out of main memory. Some values are dependent on the
clock tick internal, e.g. the amount of memory the process is using.

For backward compatibility, the return value is also accessible as a tuple of 16
elements.

The fields @code{ru_utime} and @code{ru_stime} of the return value are
floating point values representing the amount of time spent executing in user
mode and the amount of time spent executing in system mode, respectively. The
remaining values are integers. Consult the @code{getrusage(2)} man page for
detailed information about these values. A brief summary is presented here:


@multitable {xxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Index

@tab

Field

@tab

Resource

@item

@code{0}

@tab

@code{ru_utime}

@tab

time in user mode (float seconds)

@item

@code{1}

@tab

@code{ru_stime}

@tab

time in system mode (float seconds)

@item

@code{2}

@tab

@code{ru_maxrss}

@tab

maximum resident set size

@item

@code{3}

@tab

@code{ru_ixrss}

@tab

shared memory size

@item

@code{4}

@tab

@code{ru_idrss}

@tab

unshared memory size

@item

@code{5}

@tab

@code{ru_isrss}

@tab

unshared stack size

@item

@code{6}

@tab

@code{ru_minflt}

@tab

page faults not requiring I/O

@item

@code{7}

@tab

@code{ru_majflt}

@tab

page faults requiring I/O

@item

@code{8}

@tab

@code{ru_nswap}

@tab

number of swap outs

@item

@code{9}

@tab

@code{ru_inblock}

@tab

block input operations

@item

@code{10}

@tab

@code{ru_oublock}

@tab

block output operations

@item

@code{11}

@tab

@code{ru_msgsnd}

@tab

messages sent

@item

@code{12}

@tab

@code{ru_msgrcv}

@tab

messages received

@item

@code{13}

@tab

@code{ru_nsignals}

@tab

signals received

@item

@code{14}

@tab

@code{ru_nvcsw}

@tab

voluntary context switches

@item

@code{15}

@tab

@code{ru_nivcsw}

@tab

involuntary context switches

@end multitable


This function will raise a @ref{17c,,ValueError} if an invalid @emph{who} parameter is
specified. It may also raise @ref{3fb8,,error} exception in unusual circumstances.
@end deffn

@geindex getpagesize() (in module resource)
@anchor{library/resource resource getpagesize}@anchor{3fc8}
@deffn {Function} resource.getpagesize ()

Returns the number of bytes in a system page. (This need not be the same as the
hardware page size.)
@end deffn

The following @code{RUSAGE_*} symbols are passed to the @ref{f93,,getrusage()}
function to specify which processes information should be provided for.

@geindex RUSAGE_SELF (in module resource)
@anchor{library/resource resource RUSAGE_SELF}@anchor{3fc9}
@deffn {Data} resource.RUSAGE_SELF

Pass to @ref{f93,,getrusage()} to request resources consumed by the calling
process, which is the sum of resources used by all threads in the process.
@end deffn

@geindex RUSAGE_CHILDREN (in module resource)
@anchor{library/resource resource RUSAGE_CHILDREN}@anchor{3fca}
@deffn {Data} resource.RUSAGE_CHILDREN

Pass to @ref{f93,,getrusage()} to request resources consumed by child processes
of the calling process which have been terminated and waited for.
@end deffn

@geindex RUSAGE_BOTH (in module resource)
@anchor{library/resource resource RUSAGE_BOTH}@anchor{3fcb}
@deffn {Data} resource.RUSAGE_BOTH

Pass to @ref{f93,,getrusage()} to request resources consumed by both the current
process and child processes.  May not be available on all systems.
@end deffn

@geindex RUSAGE_THREAD (in module resource)
@anchor{library/resource resource RUSAGE_THREAD}@anchor{3fcc}
@deffn {Data} resource.RUSAGE_THREAD

Pass to @ref{f93,,getrusage()} to request resources consumed by the current
thread.  May not be available on all systems.


New in version 3.2.

@end deffn

@node nis — Interface to Sun’s NIS Yellow Pages,syslog — Unix syslog library routines,resource — Resource usage information,Unix Specific Services
@anchor{library/nis doc}@anchor{3fcd}@anchor{library/nis module-nis}@anchor{bb}@anchor{library/nis nis-interface-to-sun-s-nis-yellow-pages}@anchor{3fce}
@subsection @code{nis} — Interface to Sun’s NIS (Yellow Pages)


@geindex module; nis


__________________________________________________________________


The @ref{bb,,nis} module gives a thin wrapper around the NIS library, useful for
central administration of several hosts.

Because NIS exists only on Unix systems, this module is only available for Unix.

The @ref{bb,,nis} module defines the following functions:

@geindex match() (in module nis)
@anchor{library/nis nis match}@anchor{f8c}
@deffn {Function} nis.match (key, mapname, domain=default_domain)

Return the match for @emph{key} in map @emph{mapname}, or raise an error
(@ref{3fcf,,nis.error}) if there is none. Both should be strings, @emph{key} is 8-bit
clean. Return value is an arbitrary array of bytes (may contain @code{NULL} and
other joys).

Note that @emph{mapname} is first checked if it is an alias to another name.

The @emph{domain} argument allows overriding the NIS domain used for the lookup. If
unspecified, lookup is in the default NIS domain.
@end deffn

@geindex cat() (in module nis)
@anchor{library/nis nis cat}@anchor{3fd0}
@deffn {Function} nis.cat (mapname, domain=default_domain)

Return a dictionary mapping @emph{key} to @emph{value} such that @code{match(key,
mapname)==value}. Note that both keys and values of the dictionary are
arbitrary arrays of bytes.

Note that @emph{mapname} is first checked if it is an alias to another name.

The @emph{domain} argument allows overriding the NIS domain used for the lookup. If
unspecified, lookup is in the default NIS domain.
@end deffn

@geindex maps() (in module nis)
@anchor{library/nis nis maps}@anchor{f8d}
@deffn {Function} nis.maps (domain=default_domain)

Return a list of all valid maps.

The @emph{domain} argument allows overriding the NIS domain used for the lookup. If
unspecified, lookup is in the default NIS domain.
@end deffn

@geindex get_default_domain() (in module nis)
@anchor{library/nis nis get_default_domain}@anchor{3fd1}
@deffn {Function} nis.get_default_domain ()

Return the system default NIS domain.
@end deffn

The @ref{bb,,nis} module defines the following exception:

@geindex error
@anchor{library/nis nis error}@anchor{3fcf}
@deffn {Exception} nis.error

An error raised when a NIS function returns an error code.
@end deffn

@node syslog — Unix syslog library routines,,nis — Interface to Sun’s NIS Yellow Pages,Unix Specific Services
@anchor{library/syslog doc}@anchor{3fd2}@anchor{library/syslog module-syslog}@anchor{f9}@anchor{library/syslog syslog-unix-syslog-library-routines}@anchor{3fd3}
@subsection @code{syslog} — Unix syslog library routines


@geindex module; syslog


__________________________________________________________________


This module provides an interface to the Unix @code{syslog} library routines.
Refer to the Unix manual pages for a detailed description of the @code{syslog}
facility.

This module wraps the system @code{syslog} family of routines.  A pure Python
library that can speak to a syslog server is available in the
@ref{a8,,logging.handlers} module as @code{SysLogHandler}.

The module defines the following functions:

@geindex syslog() (in module syslog)
@anchor{library/syslog syslog syslog}@anchor{39fa}
@deffn {Function} syslog.syslog (message)

@deffnx {Function} syslog.syslog (priority, message)

Send the string @emph{message} to the system logger.  A trailing newline is added
if necessary.  Each message is tagged with a priority composed of a
@emph{facility} and a @emph{level}.  The optional @emph{priority} argument, which defaults
to @code{LOG_INFO}, determines the message priority.  If the facility is
not encoded in @emph{priority} using logical-or (@code{LOG_INFO | LOG_USER}), the
value given in the @ref{1242,,openlog()} call is used.

If @ref{1242,,openlog()} has not been called prior to the call to @ref{f9,,syslog()},
@code{openlog()} will be called with no arguments.

Raises an @ref{1130,,auditing event} @code{syslog.syslog} with arguments @code{priority}, @code{message}.
@end deffn

@geindex openlog() (in module syslog)
@anchor{library/syslog syslog openlog}@anchor{1242}
@deffn {Function} syslog.openlog ([ident[, logoption[, facility]]])

Logging options of subsequent @ref{f9,,syslog()} calls can be set by calling
@ref{1242,,openlog()}.  @ref{f9,,syslog()} will call @ref{1242,,openlog()} with no arguments
if the log is not currently open.

The optional @emph{ident} keyword argument is a string which is prepended to every
message, and defaults to @code{sys.argv[0]} with leading path components
stripped.  The optional @emph{logoption} keyword argument (default is 0) is a bit
field – see below for possible values to combine.  The optional @emph{facility}
keyword argument (default is @code{LOG_USER}) sets the default facility for
messages which do not have a facility explicitly encoded.

Raises an @ref{1130,,auditing event} @code{syslog.openlog} with arguments @code{ident}, @code{logoption}, @code{facility}.


Changed in version 3.2: In previous versions, keyword arguments were not allowed, and @emph{ident} was
required.  The default for @emph{ident} was dependent on the system libraries,
and often was @code{python} instead of the name of the Python program file.

@end deffn

@geindex closelog() (in module syslog)
@anchor{library/syslog syslog closelog}@anchor{39f8}
@deffn {Function} syslog.closelog ()

Reset the syslog module values and call the system library @code{closelog()}.

This causes the module to behave as it does when initially imported.  For
example, @ref{1242,,openlog()} will be called on the first @ref{f9,,syslog()} call (if
@ref{1242,,openlog()} hasn’t already been called), and @emph{ident} and other
@ref{1242,,openlog()} parameters are reset to defaults.

Raises an @ref{1130,,auditing event} @code{syslog.closelog} with no arguments.
@end deffn

@geindex setlogmask() (in module syslog)
@anchor{library/syslog syslog setlogmask}@anchor{39f9}
@deffn {Function} syslog.setlogmask (maskpri)

Set the priority mask to @emph{maskpri} and return the previous mask value.  Calls
to @ref{f9,,syslog()} with a priority level not set in @emph{maskpri} are ignored.
The default is to log all priorities.  The function @code{LOG_MASK(pri)}
calculates the mask for the individual priority @emph{pri}.  The function
@code{LOG_UPTO(pri)} calculates the mask for all priorities up to and including
@emph{pri}.

Raises an @ref{1130,,auditing event} @code{syslog.setlogmask} with argument @code{maskpri}.
@end deffn

The module defines the following constants:


@table @asis

@item Priority levels (high to low):

@code{LOG_EMERG}, @code{LOG_ALERT}, @code{LOG_CRIT}, @code{LOG_ERR},
@code{LOG_WARNING}, @code{LOG_NOTICE}, @code{LOG_INFO},
@code{LOG_DEBUG}.

@item Facilities:

@code{LOG_KERN}, @code{LOG_USER}, @code{LOG_MAIL}, @code{LOG_DAEMON},
@code{LOG_AUTH}, @code{LOG_LPR}, @code{LOG_NEWS}, @code{LOG_UUCP},
@code{LOG_CRON}, @code{LOG_SYSLOG}, @code{LOG_LOCAL0} to
@code{LOG_LOCAL7}, and, if defined in @code{<syslog.h>},
@code{LOG_AUTHPRIV}.

@item Log options:

@code{LOG_PID}, @code{LOG_CONS}, @code{LOG_NDELAY}, and, if defined
in @code{<syslog.h>}, @code{LOG_ODELAY}, @code{LOG_NOWAIT}, and
@code{LOG_PERROR}.
@end table

@menu
* Examples: Examples<35>. 

@end menu

@node Examples<35>,,,syslog — Unix syslog library routines
@anchor{library/syslog examples}@anchor{3fd4}
@subsubsection Examples


@menu
* Simple example:: 

@end menu

@node Simple example,,,Examples<35>
@anchor{library/syslog simple-example}@anchor{3fd5}
@subsubsection Simple example


A simple set of examples:

@example
import syslog

syslog.syslog('Processing started')
if error:
    syslog.syslog(syslog.LOG_ERR, 'Processing started')
@end example

An example of setting some log options, these would include the process ID in
logged messages, and write the messages to the destination facility used for
mail logging:

@example
syslog.openlog(logoption=syslog.LOG_PID, facility=syslog.LOG_MAIL)
syslog.syslog('E-mail processing initiated...')
@end example

@node Superseded Modules,Undocumented Modules,Unix Specific Services,The Python Standard Library
@anchor{library/superseded doc}@anchor{3fd6}@anchor{library/superseded superseded}@anchor{3fd7}@anchor{library/superseded superseded-modules}@anchor{3fd8}
@section Superseded Modules


The modules described in this chapter are deprecated and only kept for
backwards compatibility. They have been superseded by other modules.

@menu
* optparse — Parser for command line options:: 
* imp — Access the import internals:: 

@end menu

@node optparse — Parser for command line options,imp — Access the import internals,,Superseded Modules
@anchor{library/optparse doc}@anchor{3fd9}@anchor{library/optparse module-optparse}@anchor{bf}@anchor{library/optparse optparse-parser-for-command-line-options}@anchor{3fda}
@subsection @code{optparse} — Parser for command line options


@geindex module; optparse

@strong{Source code:} Lib/optparse.py@footnote{https://github.com/python/cpython/tree/main/Lib/optparse.py}


Deprecated since version 3.2: The @ref{bf,,optparse} module is deprecated and will not be developed further;
development will continue with the @ref{5,,argparse} module.



__________________________________________________________________


@ref{bf,,optparse} is a more convenient, flexible, and powerful library for parsing
command-line options than the old @ref{81,,getopt} module.  @ref{bf,,optparse} uses a
more declarative style of command-line parsing: you create an instance of
@ref{3fdb,,OptionParser}, populate it with options, and parse the command
line. @ref{bf,,optparse} allows users to specify options in the conventional
GNU/POSIX syntax, and additionally generates usage and help messages for you.

Here’s an example of using @ref{bf,,optparse} in a simple script:

@example
from optparse import OptionParser
...
parser = OptionParser()
parser.add_option("-f", "--file", dest="filename",
                  help="write report to FILE", metavar="FILE")
parser.add_option("-q", "--quiet",
                  action="store_false", dest="verbose", default=True,
                  help="don't print status messages to stdout")

(options, args) = parser.parse_args()
@end example

With these few lines of code, users of your script can now do the “usual thing”
on the command-line, for example:

@example
<yourscript> --file=outfile -q
@end example

As it parses the command line, @ref{bf,,optparse} sets attributes of the
@code{options} object returned by @code{parse_args()} based on user-supplied
command-line values.  When @code{parse_args()} returns from parsing this command
line, @code{options.filename} will be @code{"outfile"} and @code{options.verbose} will be
@code{False}.  @ref{bf,,optparse} supports both long and short options, allows short
options to be merged together, and allows options to be associated with their
arguments in a variety of ways.  Thus, the following command lines are all
equivalent to the above example:

@example
<yourscript> -f outfile --quiet
<yourscript> --quiet --file outfile
<yourscript> -q -foutfile
<yourscript> -qfoutfile
@end example

Additionally, users can run one of the following

@example
<yourscript> -h
<yourscript> --help
@end example

and @ref{bf,,optparse} will print out a brief summary of your script’s options:

@example
Usage: <yourscript> [options]

Options:
  -h, --help            show this help message and exit
  -f FILE, --file=FILE  write report to FILE
  -q, --quiet           don't print status messages to stdout
@end example

where the value of @emph{yourscript} is determined at runtime (normally from
@code{sys.argv[0]}).

@menu
* Background:: 
* Tutorial: Tutorial<2>. 
* Reference Guide:: 
* Option Callbacks:: 
* Extending optparse:: 

@end menu

@node Background,Tutorial<2>,,optparse — Parser for command line options
@anchor{library/optparse background}@anchor{3fdc}@anchor{library/optparse optparse-background}@anchor{3fdd}
@subsubsection Background


@ref{bf,,optparse} was explicitly designed to encourage the creation of programs
with straightforward, conventional command-line interfaces.  To that end, it
supports only the most common command-line syntax and semantics conventionally
used under Unix.  If you are unfamiliar with these conventions, read this
section to acquaint yourself with them.

@menu
* Terminology:: 
* What are options for?:: 
* What are positional arguments for?:: 

@end menu

@node Terminology,What are options for?,,Background
@anchor{library/optparse optparse-terminology}@anchor{3fde}@anchor{library/optparse terminology}@anchor{3fdf}
@subsubsection Terminology



@table @asis

@item argument

a string entered on the command-line, and passed by the shell to @code{execl()}
or @code{execv()}.  In Python, arguments are elements of @code{sys.argv[1:]}
(@code{sys.argv[0]} is the name of the program being executed).  Unix shells
also use the term “word”.

It is occasionally desirable to substitute an argument list other than
@code{sys.argv[1:]}, so you should read “argument” as “an element of
@code{sys.argv[1:]}, or of some other list provided as a substitute for
@code{sys.argv[1:]}”.

@item option

an argument used to supply extra information to guide or customize the
execution of a program.  There are many different syntaxes for options; the
traditional Unix syntax is a hyphen (“-”) followed by a single letter,
e.g. @code{-x} or @code{-F}.  Also, traditional Unix syntax allows multiple
options to be merged into a single argument, e.g. @code{-x -F} is equivalent
to @code{-xF}.  The GNU project introduced @code{--} followed by a series of
hyphen-separated words, e.g. @code{--file} or @code{--dry-run}.  These are the
only two option syntaxes provided by @ref{bf,,optparse}.

Some other option syntaxes that the world has seen include:


@itemize *

@item 
a hyphen followed by a few letters, e.g. @code{-pf} (this is @emph{not} the same
as multiple options merged into a single argument)

@item 
a hyphen followed by a whole word, e.g. @code{-file} (this is technically
equivalent to the previous syntax, but they aren’t usually seen in the same
program)

@item 
a plus sign followed by a single letter, or a few letters, or a word, e.g.
@code{+f}, @code{+rgb}

@item 
a slash followed by a letter, or a few letters, or a word, e.g. @code{/f},
@code{/file}
@end itemize

These option syntaxes are not supported by @ref{bf,,optparse}, and they never
will be.  This is deliberate: the first three are non-standard on any
environment, and the last only makes sense if you’re exclusively targeting
VMS, MS-DOS, and/or Windows.

@item option argument

an argument that follows an option, is closely associated with that option,
and is consumed from the argument list when that option is. With
@ref{bf,,optparse}, option arguments may either be in a separate argument from
their option:

@example
-f foo
--file foo
@end example

or included in the same argument:

@example
-ffoo
--file=foo
@end example

Typically, a given option either takes an argument or it doesn’t. Lots of
people want an “optional option arguments” feature, meaning that some options
will take an argument if they see it, and won’t if they don’t.  This is
somewhat controversial, because it makes parsing ambiguous: if @code{-a} takes
an optional argument and @code{-b} is another option entirely, how do we
interpret @code{-ab}?  Because of this ambiguity, @ref{bf,,optparse} does not
support this feature.

@item positional argument

something leftover in the argument list after options have been parsed, i.e.
after options and their arguments have been parsed and removed from the
argument list.

@item required option

an option that must be supplied on the command-line; note that the phrase
“required option” is self-contradictory in English.  @ref{bf,,optparse} doesn’t
prevent you from implementing required options, but doesn’t give you much
help at it either.
@end table

For example, consider this hypothetical command-line:

@example
prog -v --report report.txt foo bar
@end example

@code{-v} and @code{--report} are both options.  Assuming that @code{--report}
takes one argument, @code{report.txt} is an option argument.  @code{foo} and
@code{bar} are positional arguments.

@node What are options for?,What are positional arguments for?,Terminology,Background
@anchor{library/optparse optparse-what-options-for}@anchor{3fe0}@anchor{library/optparse what-are-options-for}@anchor{3fe1}
@subsubsection What are options for?


Options are used to provide extra information to tune or customize the execution
of a program.  In case it wasn’t clear, options are usually @emph{optional}.  A
program should be able to run just fine with no options whatsoever.  (Pick a
random program from the Unix or GNU toolsets.  Can it run without any options at
all and still make sense?  The main exceptions are @code{find}, @code{tar}, and
@code{dd}—all of which are mutant oddballs that have been rightly criticized
for their non-standard syntax and confusing interfaces.)

Lots of people want their programs to have “required options”.  Think about it.
If it’s required, then it’s @emph{not optional}!  If there is a piece of information
that your program absolutely requires in order to run successfully, that’s what
positional arguments are for.

As an example of good command-line interface design, consider the humble @code{cp}
utility, for copying files.  It doesn’t make much sense to try to copy files
without supplying a destination and at least one source. Hence, @code{cp} fails if
you run it with no arguments.  However, it has a flexible, useful syntax that
does not require any options at all:

@example
cp SOURCE DEST
cp SOURCE ... DEST-DIR
@end example

You can get pretty far with just that.  Most @code{cp} implementations provide a
bunch of options to tweak exactly how the files are copied: you can preserve
mode and modification time, avoid following symlinks, ask before clobbering
existing files, etc.  But none of this distracts from the core mission of
@code{cp}, which is to copy either one file to another, or several files to another
directory.

@node What are positional arguments for?,,What are options for?,Background
@anchor{library/optparse optparse-what-positional-arguments-for}@anchor{3fe2}@anchor{library/optparse what-are-positional-arguments-for}@anchor{3fe3}
@subsubsection What are positional arguments for?


Positional arguments are for those pieces of information that your program
absolutely, positively requires to run.

A good user interface should have as few absolute requirements as possible.  If
your program requires 17 distinct pieces of information in order to run
successfully, it doesn’t much matter @emph{how} you get that information from the
user—most people will give up and walk away before they successfully run the
program.  This applies whether the user interface is a command-line, a
configuration file, or a GUI: if you make that many demands on your users, most
of them will simply give up.

In short, try to minimize the amount of information that users are absolutely
required to supply—use sensible defaults whenever possible.  Of course, you
also want to make your programs reasonably flexible.  That’s what options are
for.  Again, it doesn’t matter if they are entries in a config file, widgets in
the “Preferences” dialog of a GUI, or command-line options—the more options
you implement, the more flexible your program is, and the more complicated its
implementation becomes.  Too much flexibility has drawbacks as well, of course;
too many options can overwhelm users and make your code much harder to maintain.

@node Tutorial<2>,Reference Guide,Background,optparse — Parser for command line options
@anchor{library/optparse optparse-tutorial}@anchor{3fe4}@anchor{library/optparse tutorial}@anchor{3fe5}
@subsubsection Tutorial


While @ref{bf,,optparse} is quite flexible and powerful, it’s also straightforward
to use in most cases.  This section covers the code patterns that are common to
any @ref{bf,,optparse}-based program.

First, you need to import the OptionParser class; then, early in the main
program, create an OptionParser instance:

@example
from optparse import OptionParser
...
parser = OptionParser()
@end example

Then you can start defining options.  The basic syntax is:

@example
parser.add_option(opt_str, ...,
                  attr=value, ...)
@end example

Each option has one or more option strings, such as @code{-f} or @code{--file},
and several option attributes that tell @ref{bf,,optparse} what to expect and what
to do when it encounters that option on the command line.

Typically, each option will have one short option string and one long option
string, e.g.:

@example
parser.add_option("-f", "--file", ...)
@end example

You’re free to define as many short option strings and as many long option
strings as you like (including zero), as long as there is at least one option
string overall.

The option strings passed to @ref{2510,,OptionParser.add_option()} are effectively
labels for the
option defined by that call.  For brevity, we will frequently refer to
@emph{encountering an option} on the command line; in reality, @ref{bf,,optparse}
encounters @emph{option strings} and looks up options from them.

Once all of your options are defined, instruct @ref{bf,,optparse} to parse your
program’s command line:

@example
(options, args) = parser.parse_args()
@end example

(If you like, you can pass a custom argument list to @code{parse_args()}, but
that’s rarely necessary: by default it uses @code{sys.argv[1:]}.)

@code{parse_args()} returns two values:


@itemize *

@item 
@code{options}, an object containing values for all of your options—e.g. if
@code{--file} takes a single string argument, then @code{options.file} will be the
filename supplied by the user, or @code{None} if the user did not supply that
option

@item 
@code{args}, the list of positional arguments leftover after parsing options
@end itemize

This tutorial section only covers the four most important option attributes:
@ref{3fe6,,action}, @ref{3fe7,,type}, @ref{3fe8,,dest}
(destination), and @ref{3fe9,,help}. Of these, @ref{3fe6,,action} is the
most fundamental.

@menu
* Understanding option actions:: 
* The store action:: 
* Handling boolean (flag) options: Handling boolean flag options. 
* Other actions:: 
* Default values:: 
* Generating help:: 
* Printing a version string:: 
* How optparse handles errors:: 
* Putting it all together:: 

@end menu

@node Understanding option actions,The store action,,Tutorial<2>
@anchor{library/optparse optparse-understanding-option-actions}@anchor{3fea}@anchor{library/optparse understanding-option-actions}@anchor{3feb}
@subsubsection Understanding option actions


Actions tell @ref{bf,,optparse} what to do when it encounters an option on the
command line.  There is a fixed set of actions hard-coded into @ref{bf,,optparse};
adding new actions is an advanced topic covered in section
@ref{3fec,,Extending optparse}.  Most actions tell @ref{bf,,optparse} to store
a value in some variable—for example, take a string from the command line and
store it in an attribute of @code{options}.

If you don’t specify an option action, @ref{bf,,optparse} defaults to @code{store}.

@node The store action,Handling boolean flag options,Understanding option actions,Tutorial<2>
@anchor{library/optparse optparse-store-action}@anchor{3fed}@anchor{library/optparse the-store-action}@anchor{3fee}
@subsubsection The store action


The most common option action is @code{store}, which tells @ref{bf,,optparse} to take
the next argument (or the remainder of the current argument), ensure that it is
of the correct type, and store it to your chosen destination.

For example:

@example
parser.add_option("-f", "--file",
                  action="store", type="string", dest="filename")
@end example

Now let’s make up a fake command line and ask @ref{bf,,optparse} to parse it:

@example
args = ["-f", "foo.txt"]
(options, args) = parser.parse_args(args)
@end example

When @ref{bf,,optparse} sees the option string @code{-f}, it consumes the next
argument, @code{foo.txt}, and stores it in @code{options.filename}.  So, after this
call to @code{parse_args()}, @code{options.filename} is @code{"foo.txt"}.

Some other option types supported by @ref{bf,,optparse} are @code{int} and @code{float}.
Here’s an option that expects an integer argument:

@example
parser.add_option("-n", type="int", dest="num")
@end example

Note that this option has no long option string, which is perfectly acceptable.
Also, there’s no explicit action, since the default is @code{store}.

Let’s parse another fake command-line.  This time, we’ll jam the option argument
right up against the option: since @code{-n42} (one argument) is equivalent to
@code{-n 42} (two arguments), the code

@example
(options, args) = parser.parse_args(["-n42"])
print(options.num)
@end example

will print @code{42}.

If you don’t specify a type, @ref{bf,,optparse} assumes @code{string}.  Combined with
the fact that the default action is @code{store}, that means our first example can
be a lot shorter:

@example
parser.add_option("-f", "--file", dest="filename")
@end example

If you don’t supply a destination, @ref{bf,,optparse} figures out a sensible
default from the option strings: if the first long option string is
@code{--foo-bar}, then the default destination is @code{foo_bar}.  If there are no
long option strings, @ref{bf,,optparse} looks at the first short option string: the
default destination for @code{-f} is @code{f}.

@ref{bf,,optparse} also includes the built-in @code{complex} type.  Adding
types is covered in section @ref{3fec,,Extending optparse}.

@node Handling boolean flag options,Other actions,The store action,Tutorial<2>
@anchor{library/optparse handling-boolean-flag-options}@anchor{3fef}@anchor{library/optparse optparse-handling-boolean-options}@anchor{3ff0}
@subsubsection Handling boolean (flag) options


Flag options—set a variable to true or false when a particular option is
seen—are quite common.  @ref{bf,,optparse} supports them with two separate actions,
@code{store_true} and @code{store_false}.  For example, you might have a @code{verbose}
flag that is turned on with @code{-v} and off with @code{-q}:

@example
parser.add_option("-v", action="store_true", dest="verbose")
parser.add_option("-q", action="store_false", dest="verbose")
@end example

Here we have two different options with the same destination, which is perfectly
OK.  (It just means you have to be a bit careful when setting default
values—see below.)

When @ref{bf,,optparse} encounters @code{-v} on the command line, it sets
@code{options.verbose} to @code{True}; when it encounters @code{-q},
@code{options.verbose} is set to @code{False}.

@node Other actions,Default values,Handling boolean flag options,Tutorial<2>
@anchor{library/optparse optparse-other-actions}@anchor{3ff1}@anchor{library/optparse other-actions}@anchor{3ff2}
@subsubsection Other actions


Some other actions supported by @ref{bf,,optparse} are:


@table @asis

@item @code{"store_const"}

store a constant value

@item @code{"append"}

append this option’s argument to a list

@item @code{"count"}

increment a counter by one

@item @code{"callback"}

call a specified function
@end table

These are covered in section @ref{3ff3,,Reference Guide},
and section @ref{3ff4,,Option Callbacks}.

@node Default values,Generating help,Other actions,Tutorial<2>
@anchor{library/optparse default-values}@anchor{3ff5}@anchor{library/optparse optparse-default-values}@anchor{3ff6}
@subsubsection Default values


All of the above examples involve setting some variable (the “destination”) when
certain command-line options are seen.  What happens if those options are never
seen?  Since we didn’t supply any defaults, they are all set to @code{None}.  This
is usually fine, but sometimes you want more control.  @ref{bf,,optparse} lets you
supply a default value for each destination, which is assigned before the
command line is parsed.

First, consider the verbose/quiet example.  If we want @ref{bf,,optparse} to set
@code{verbose} to @code{True} unless @code{-q} is seen, then we can do this:

@example
parser.add_option("-v", action="store_true", dest="verbose", default=True)
parser.add_option("-q", action="store_false", dest="verbose")
@end example

Since default values apply to the @emph{destination} rather than to any particular
option, and these two options happen to have the same destination, this is
exactly equivalent:

@example
parser.add_option("-v", action="store_true", dest="verbose")
parser.add_option("-q", action="store_false", dest="verbose", default=True)
@end example

Consider this:

@example
parser.add_option("-v", action="store_true", dest="verbose", default=False)
parser.add_option("-q", action="store_false", dest="verbose", default=True)
@end example

Again, the default value for @code{verbose} will be @code{True}: the last default
value supplied for any particular destination is the one that counts.

A clearer way to specify default values is the @code{set_defaults()} method of
OptionParser, which you can call at any time before calling @code{parse_args()}:

@example
parser.set_defaults(verbose=True)
parser.add_option(...)
(options, args) = parser.parse_args()
@end example

As before, the last value specified for a given option destination is the one
that counts.  For clarity, try to use one method or the other of setting default
values, not both.

@node Generating help,Printing a version string,Default values,Tutorial<2>
@anchor{library/optparse generating-help}@anchor{3ff7}@anchor{library/optparse optparse-generating-help}@anchor{3ff8}
@subsubsection Generating help


@ref{bf,,optparse}’s ability to generate help and usage text automatically is
useful for creating user-friendly command-line interfaces.  All you have to do
is supply a @ref{3fe9,,help} value for each option, and optionally a short
usage message for your whole program.  Here’s an OptionParser populated with
user-friendly (documented) options:

@example
usage = "usage: %prog [options] arg1 arg2"
parser = OptionParser(usage=usage)
parser.add_option("-v", "--verbose",
                  action="store_true", dest="verbose", default=True,
                  help="make lots of noise [default]")
parser.add_option("-q", "--quiet",
                  action="store_false", dest="verbose",
                  help="be vewwy quiet (I'm hunting wabbits)")
parser.add_option("-f", "--filename",
                  metavar="FILE", help="write output to FILE")
parser.add_option("-m", "--mode",
                  default="intermediate",
                  help="interaction mode: novice, intermediate, "
                       "or expert [default: %default]")
@end example

If @ref{bf,,optparse} encounters either @code{-h} or @code{--help} on the
command-line, or if you just call @code{parser.print_help()}, it prints the
following to standard output:

@example
Usage: <yourscript> [options] arg1 arg2

Options:
  -h, --help            show this help message and exit
  -v, --verbose         make lots of noise [default]
  -q, --quiet           be vewwy quiet (I'm hunting wabbits)
  -f FILE, --filename=FILE
                        write output to FILE
  -m MODE, --mode=MODE  interaction mode: novice, intermediate, or
                        expert [default: intermediate]
@end example

(If the help output is triggered by a help option, @ref{bf,,optparse} exits after
printing the help text.)

There’s a lot going on here to help @ref{bf,,optparse} generate the best possible
help message:


@itemize *

@item 
the script defines its own usage message:

@example
usage = "usage: %prog [options] arg1 arg2"
@end example

@ref{bf,,optparse} expands @code{%prog} in the usage string to the name of the
current program, i.e. @code{os.path.basename(sys.argv[0])}.  The expanded string
is then printed before the detailed option help.

If you don’t supply a usage string, @ref{bf,,optparse} uses a bland but sensible
default: @code{"Usage: %prog [options]"}, which is fine if your script doesn’t
take any positional arguments.

@item 
every option defines a help string, and doesn’t worry about
line-wrapping—@ref{bf,,optparse} takes care of wrapping lines and making
the help output look good.

@item 
options that take a value indicate this fact in their automatically-generated
help message, e.g. for the “mode” option:

@example
-m MODE, --mode=MODE
@end example

Here, “MODE” is called the meta-variable: it stands for the argument that the
user is expected to supply to @code{-m}/@code{--mode}.  By default,
@ref{bf,,optparse} converts the destination variable name to uppercase and uses
that for the meta-variable.  Sometimes, that’s not what you want—for
example, the @code{--filename} option explicitly sets @code{metavar="FILE"},
resulting in this automatically-generated option description:

@example
-f FILE, --filename=FILE
@end example

This is important for more than just saving space, though: the manually
written help text uses the meta-variable @code{FILE} to clue the user in that
there’s a connection between the semi-formal syntax @code{-f FILE} and the informal
semantic description “write output to FILE”. This is a simple but effective
way to make your help text a lot clearer and more useful for end users.

@item 
options that have a default value can include @code{%default} in the help
string—@ref{bf,,optparse} will replace it with @ref{2ef,,str()} of the option’s
default value.  If an option has no default value (or the default value is
@code{None}), @code{%default} expands to @code{none}.
@end itemize

@menu
* Grouping Options:: 

@end menu

@node Grouping Options,,,Generating help
@anchor{library/optparse grouping-options}@anchor{3ff9}
@subsubsection Grouping Options


When dealing with many options, it is convenient to group these options for
better help output.  An @ref{3fdb,,OptionParser} can contain several option groups,
each of which can contain several options.

An option group is obtained using the class @ref{3ffa,,OptionGroup}:

@geindex OptionGroup (class in optparse)
@anchor{library/optparse optparse OptionGroup}@anchor{3ffa}
@deffn {Class} optparse.OptionGroup (parser, title, description=None)

where


@itemize *

@item 
parser is the @ref{3fdb,,OptionParser} instance the group will be inserted in
to

@item 
title is the group title

@item 
description, optional, is a long description of the group
@end itemize
@end deffn

@ref{3ffa,,OptionGroup} inherits from @code{OptionContainer} (like
@ref{3fdb,,OptionParser}) and so the @code{add_option()} method can be used to add
an option to the group.

Once all the options are declared, using the @ref{3fdb,,OptionParser} method
@code{add_option_group()} the group is added to the previously defined parser.

Continuing with the parser defined in the previous section, adding an
@ref{3ffa,,OptionGroup} to a parser is easy:

@example
group = OptionGroup(parser, "Dangerous Options",
                    "Caution: use these options at your own risk.  "
                    "It is believed that some of them bite.")
group.add_option("-g", action="store_true", help="Group option.")
parser.add_option_group(group)
@end example

This would result in the following help output:

@example
Usage: <yourscript> [options] arg1 arg2

Options:
  -h, --help            show this help message and exit
  -v, --verbose         make lots of noise [default]
  -q, --quiet           be vewwy quiet (I'm hunting wabbits)
  -f FILE, --filename=FILE
                        write output to FILE
  -m MODE, --mode=MODE  interaction mode: novice, intermediate, or
                        expert [default: intermediate]

  Dangerous Options:
    Caution: use these options at your own risk.  It is believed that some
    of them bite.

    -g                  Group option.
@end example

A bit more complete example might involve using more than one group: still
extending the previous example:

@example
group = OptionGroup(parser, "Dangerous Options",
                    "Caution: use these options at your own risk.  "
                    "It is believed that some of them bite.")
group.add_option("-g", action="store_true", help="Group option.")
parser.add_option_group(group)

group = OptionGroup(parser, "Debug Options")
group.add_option("-d", "--debug", action="store_true",
                 help="Print debug information")
group.add_option("-s", "--sql", action="store_true",
                 help="Print all SQL statements executed")
group.add_option("-e", action="store_true", help="Print every action done")
parser.add_option_group(group)
@end example

that results in the following output:

@example
Usage: <yourscript> [options] arg1 arg2

Options:
  -h, --help            show this help message and exit
  -v, --verbose         make lots of noise [default]
  -q, --quiet           be vewwy quiet (I'm hunting wabbits)
  -f FILE, --filename=FILE
                        write output to FILE
  -m MODE, --mode=MODE  interaction mode: novice, intermediate, or expert
                        [default: intermediate]

  Dangerous Options:
    Caution: use these options at your own risk.  It is believed that some
    of them bite.

    -g                  Group option.

  Debug Options:
    -d, --debug         Print debug information
    -s, --sql           Print all SQL statements executed
    -e                  Print every action done
@end example

Another interesting method, in particular when working programmatically with
option groups is:

@geindex get_option_group() (optparse.OptionParser method)
@anchor{library/optparse optparse OptionParser get_option_group}@anchor{3ffb}
@deffn {Method} OptionParser.get_option_group (opt_str)

Return the @ref{3ffa,,OptionGroup} to which the short or long option
string @emph{opt_str} (e.g. @code{'-o'} or @code{'--option'}) belongs. If
there’s no such @ref{3ffa,,OptionGroup}, return @code{None}.
@end deffn

@node Printing a version string,How optparse handles errors,Generating help,Tutorial<2>
@anchor{library/optparse optparse-printing-version-string}@anchor{3ffc}@anchor{library/optparse printing-a-version-string}@anchor{3ffd}
@subsubsection Printing a version string


Similar to the brief usage string, @ref{bf,,optparse} can also print a version
string for your program.  You have to supply the string as the @code{version}
argument to OptionParser:

@example
parser = OptionParser(usage="%prog [-f] [-q]", version="%prog 1.0")
@end example

@code{%prog} is expanded just like it is in @code{usage}.  Apart from that,
@code{version} can contain anything you like.  When you supply it, @ref{bf,,optparse}
automatically adds a @code{--version} option to your parser. If it encounters
this option on the command line, it expands your @code{version} string (by
replacing @code{%prog}), prints it to stdout, and exits.

For example, if your script is called @code{/usr/bin/foo}:

@example
$ /usr/bin/foo --version
foo 1.0
@end example

The following two methods can be used to print and get the @code{version} string:

@geindex print_version() (optparse.OptionParser method)
@anchor{library/optparse optparse OptionParser print_version}@anchor{3ffe}
@deffn {Method} OptionParser.print_version (file=None)

Print the version message for the current program (@code{self.version}) to
@emph{file} (default stdout).  As with @ref{3fff,,print_usage()}, any occurrence
of @code{%prog} in @code{self.version} is replaced with the name of the current
program.  Does nothing if @code{self.version} is empty or undefined.
@end deffn

@geindex get_version() (optparse.OptionParser method)
@anchor{library/optparse optparse OptionParser get_version}@anchor{4000}
@deffn {Method} OptionParser.get_version ()

Same as @ref{3ffe,,print_version()} but returns the version string instead of
printing it.
@end deffn

@node How optparse handles errors,Putting it all together,Printing a version string,Tutorial<2>
@anchor{library/optparse how-optparse-handles-errors}@anchor{4001}@anchor{library/optparse optparse-how-optparse-handles-errors}@anchor{4002}
@subsubsection How @code{optparse} handles errors


There are two broad classes of errors that @ref{bf,,optparse} has to worry about:
programmer errors and user errors.  Programmer errors are usually erroneous
calls to @ref{2510,,OptionParser.add_option()}, e.g. invalid option strings, unknown
option attributes, missing option attributes, etc.  These are dealt with in the
usual way: raise an exception (either @code{optparse.OptionError} or
@ref{162,,TypeError}) and let the program crash.

Handling user errors is much more important, since they are guaranteed to happen
no matter how stable your code is.  @ref{bf,,optparse} can automatically detect
some user errors, such as bad option arguments (passing @code{-n 4x} where
@code{-n} takes an integer argument), missing arguments (@code{-n} at the end
of the command line, where @code{-n} takes an argument of any type).  Also,
you can call @code{OptionParser.error()} to signal an application-defined error
condition:

@example
(options, args) = parser.parse_args()
...
if options.a and options.b:
    parser.error("options -a and -b are mutually exclusive")
@end example

In either case, @ref{bf,,optparse} handles the error the same way: it prints the
program’s usage message and an error message to standard error and exits with
error status 2.

Consider the first example above, where the user passes @code{4x} to an option
that takes an integer:

@example
$ /usr/bin/foo -n 4x
Usage: foo [options]

foo: error: option -n: invalid integer value: '4x'
@end example

Or, where the user fails to pass a value at all:

@example
$ /usr/bin/foo -n
Usage: foo [options]

foo: error: -n option requires an argument
@end example

@ref{bf,,optparse}-generated error messages take care always to mention the
option involved in the error; be sure to do the same when calling
@code{OptionParser.error()} from your application code.

If @ref{bf,,optparse}’s default error-handling behaviour does not suit your needs,
you’ll need to subclass OptionParser and override its @code{exit()}
and/or @code{error()} methods.

@node Putting it all together,,How optparse handles errors,Tutorial<2>
@anchor{library/optparse optparse-putting-it-all-together}@anchor{4003}@anchor{library/optparse putting-it-all-together}@anchor{4004}
@subsubsection Putting it all together


Here’s what @ref{bf,,optparse}-based scripts usually look like:

@example
from optparse import OptionParser
...
def main():
    usage = "usage: %prog [options] arg"
    parser = OptionParser(usage)
    parser.add_option("-f", "--file", dest="filename",
                      help="read data from FILENAME")
    parser.add_option("-v", "--verbose",
                      action="store_true", dest="verbose")
    parser.add_option("-q", "--quiet",
                      action="store_false", dest="verbose")
    ...
    (options, args) = parser.parse_args()
    if len(args) != 1:
        parser.error("incorrect number of arguments")
    if options.verbose:
        print("reading %s..." % options.filename)
    ...

if __name__ == "__main__":
    main()
@end example

@node Reference Guide,Option Callbacks,Tutorial<2>,optparse — Parser for command line options
@anchor{library/optparse optparse-reference-guide}@anchor{3ff3}@anchor{library/optparse reference-guide}@anchor{4005}
@subsubsection Reference Guide


@menu
* Creating the parser:: 
* Populating the parser:: 
* Defining options:: 
* Option attributes:: 
* Standard option actions:: 
* Standard option types:: 
* Parsing arguments: Parsing arguments<2>. 
* Querying and manipulating your option parser:: 
* Conflicts between options:: 
* Cleanup: Cleanup<2>. 
* Other methods:: 

@end menu

@node Creating the parser,Populating the parser,,Reference Guide
@anchor{library/optparse creating-the-parser}@anchor{4006}@anchor{library/optparse optparse-creating-parser}@anchor{4007}
@subsubsection Creating the parser


The first step in using @ref{bf,,optparse} is to create an OptionParser instance.

@geindex OptionParser (class in optparse)
@anchor{library/optparse optparse OptionParser}@anchor{3fdb}
@deffn {Class} optparse.OptionParser (...)

The OptionParser constructor has no required arguments, but a number of
optional keyword arguments.  You should always pass them as keyword
arguments, i.e. do not rely on the order in which the arguments are declared.


@table @asis

@item @code{usage} (default: @code{"%prog [options]"})

The usage summary to print when your program is run incorrectly or with a
help option.  When @ref{bf,,optparse} prints the usage string, it expands
@code{%prog} to @code{os.path.basename(sys.argv[0])} (or to @code{prog} if you
passed that keyword argument).  To suppress a usage message, pass the
special value @code{optparse.SUPPRESS_USAGE}.

@item @code{option_list} (default: @code{[]})

A list of Option objects to populate the parser with.  The options in
@code{option_list} are added after any options in @code{standard_option_list} (a
class attribute that may be set by OptionParser subclasses), but before
any version or help options. Deprecated; use @ref{2510,,add_option()} after
creating the parser instead.

@item @code{option_class} (default: optparse.Option)

Class to use when adding options to the parser in @ref{2510,,add_option()}.

@item @code{version} (default: @code{None})

A version string to print when the user supplies a version option. If you
supply a true value for @code{version}, @ref{bf,,optparse} automatically adds a
version option with the single option string @code{--version}.  The
substring @code{%prog} is expanded the same as for @code{usage}.

@item @code{conflict_handler} (default: @code{"error"})

Specifies what to do when options with conflicting option strings are
added to the parser; see section
@ref{4008,,Conflicts between options}.

@item @code{description} (default: @code{None})

A paragraph of text giving a brief overview of your program.
@ref{bf,,optparse} reformats this paragraph to fit the current terminal width
and prints it when the user requests help (after @code{usage}, but before the
list of options).

@item @code{formatter} (default: a new @code{IndentedHelpFormatter})

An instance of optparse.HelpFormatter that will be used for printing help
text.  @ref{bf,,optparse} provides two concrete classes for this purpose:
IndentedHelpFormatter and TitledHelpFormatter.

@item @code{add_help_option} (default: @code{True})

If true, @ref{bf,,optparse} will add a help option (with option strings @code{-h}
and @code{--help}) to the parser.

@item @code{prog}

The string to use when expanding @code{%prog} in @code{usage} and @code{version}
instead of @code{os.path.basename(sys.argv[0])}.

@item @code{epilog} (default: @code{None})

A paragraph of help text to print after the option help.
@end table
@end deffn

@node Populating the parser,Defining options,Creating the parser,Reference Guide
@anchor{library/optparse optparse-populating-parser}@anchor{4009}@anchor{library/optparse populating-the-parser}@anchor{400a}
@subsubsection Populating the parser


There are several ways to populate the parser with options.  The preferred way
is by using @ref{2510,,OptionParser.add_option()}, as shown in section
@ref{3fe4,,Tutorial}.  @code{add_option()} can be called in one of two ways:


@itemize *

@item 
pass it an Option instance (as returned by @code{make_option()})

@item 
pass it any combination of positional and keyword arguments that are
acceptable to @code{make_option()} (i.e., to the Option constructor), and it
will create the Option instance for you
@end itemize

The other alternative is to pass a list of pre-constructed Option instances to
the OptionParser constructor, as in:

@example
option_list = [
    make_option("-f", "--filename",
                action="store", type="string", dest="filename"),
    make_option("-q", "--quiet",
                action="store_false", dest="verbose"),
    ]
parser = OptionParser(option_list=option_list)
@end example

(@code{make_option()} is a factory function for creating Option instances;
currently it is an alias for the Option constructor.  A future version of
@ref{bf,,optparse} may split Option into several classes, and @code{make_option()}
will pick the right class to instantiate.  Do not instantiate Option directly.)

@node Defining options,Option attributes,Populating the parser,Reference Guide
@anchor{library/optparse defining-options}@anchor{400b}@anchor{library/optparse optparse-defining-options}@anchor{400c}
@subsubsection Defining options


Each Option instance represents a set of synonymous command-line option strings,
e.g. @code{-f} and @code{--file}.  You can specify any number of short or
long option strings, but you must specify at least one overall option string.

The canonical way to create an @code{Option} instance is with the
@code{add_option()} method of @ref{3fdb,,OptionParser}.

@geindex add_option() (optparse.OptionParser method)
@anchor{library/optparse optparse OptionParser add_option}@anchor{2510}
@deffn {Method} OptionParser.add_option (option)

@deffnx {Method} OptionParser.add_option (*opt_str, attr=value, ...)

To define an option with only a short option string:

@example
parser.add_option("-f", attr=value, ...)
@end example

And to define an option with only a long option string:

@example
parser.add_option("--foo", attr=value, ...)
@end example

The keyword arguments define attributes of the new Option object.  The most
important option attribute is @ref{3fe6,,action}, and it largely
determines which other attributes are relevant or required.  If you pass
irrelevant option attributes, or fail to pass required ones, @ref{bf,,optparse}
raises an @code{OptionError} exception explaining your mistake.

An option’s @emph{action} determines what @ref{bf,,optparse} does when it encounters
this option on the command-line.  The standard option actions hard-coded into
@ref{bf,,optparse} are:


@table @asis

@item @code{"store"}

store this option’s argument (default)

@item @code{"store_const"}

store a constant value

@item @code{"store_true"}

store @code{True}

@item @code{"store_false"}

store @code{False}

@item @code{"append"}

append this option’s argument to a list

@item @code{"append_const"}

append a constant value to a list

@item @code{"count"}

increment a counter by one

@item @code{"callback"}

call a specified function

@item @code{"help"}

print a usage message including all options and the documentation for them
@end table

(If you don’t supply an action, the default is @code{"store"}.  For this action,
you may also supply @ref{3fe7,,type} and @ref{3fe8,,dest} option
attributes; see @ref{400d,,Standard option actions}.)
@end deffn

As you can see, most actions involve storing or updating a value somewhere.
@ref{bf,,optparse} always creates a special object for this, conventionally called
@code{options} (it happens to be an instance of @code{optparse.Values}).  Option
arguments (and various other values) are stored as attributes of this object,
according to the @ref{3fe8,,dest} (destination) option attribute.

For example, when you call

@example
parser.parse_args()
@end example

one of the first things @ref{bf,,optparse} does is create the @code{options} object:

@example
options = Values()
@end example

If one of the options in this parser is defined with

@example
parser.add_option("-f", "--file", action="store", type="string", dest="filename")
@end example

and the command-line being parsed includes any of the following:

@example
-ffoo
-f foo
--file=foo
--file foo
@end example

then @ref{bf,,optparse}, on seeing this option, will do the equivalent of

@example
options.filename = "foo"
@end example

The @ref{3fe7,,type} and @ref{3fe8,,dest} option attributes are almost
as important as @ref{3fe6,,action}, but @ref{3fe6,,action} is the only
one that makes sense for @emph{all} options.

@node Option attributes,Standard option actions,Defining options,Reference Guide
@anchor{library/optparse option-attributes}@anchor{400e}@anchor{library/optparse optparse-option-attributes}@anchor{400f}
@subsubsection Option attributes


The following option attributes may be passed as keyword arguments to
@ref{2510,,OptionParser.add_option()}.  If you pass an option attribute that is not
relevant to a particular option, or fail to pass a required option attribute,
@ref{bf,,optparse} raises @code{OptionError}.

@geindex action (optparse.Option attribute)
@anchor{library/optparse optparse Option action}@anchor{3fe6}
@deffn {Attribute} Option.action

(default: @code{"store"})

Determines @ref{bf,,optparse}’s behaviour when this option is seen on the
command line; the available options are documented @ref{400d,,here}.
@end deffn

@geindex type (optparse.Option attribute)
@anchor{library/optparse optparse Option type}@anchor{3fe7}
@deffn {Attribute} Option.type

(default: @code{"string"})

The argument type expected by this option (e.g., @code{"string"} or @code{"int"});
the available option types are documented @ref{4010,,here}.
@end deffn

@geindex dest (optparse.Option attribute)
@anchor{library/optparse optparse Option dest}@anchor{3fe8}
@deffn {Attribute} Option.dest

(default: derived from option strings)

If the option’s action implies writing or modifying a value somewhere, this
tells @ref{bf,,optparse} where to write it: @ref{3fe8,,dest} names an
attribute of the @code{options} object that @ref{bf,,optparse} builds as it parses
the command line.
@end deffn

@geindex default (optparse.Option attribute)
@anchor{library/optparse optparse Option default}@anchor{4011}
@deffn {Attribute} Option.default

The value to use for this option’s destination if the option is not seen on
the command line.  See also @ref{4012,,OptionParser.set_defaults()}.
@end deffn

@geindex nargs (optparse.Option attribute)
@anchor{library/optparse optparse Option nargs}@anchor{4013}
@deffn {Attribute} Option.nargs

(default: 1)

How many arguments of type @ref{3fe7,,type} should be consumed when this
option is seen.  If > 1, @ref{bf,,optparse} will store a tuple of values to
@ref{3fe8,,dest}.
@end deffn

@geindex const (optparse.Option attribute)
@anchor{library/optparse optparse Option const}@anchor{4014}
@deffn {Attribute} Option.const

For actions that store a constant value, the constant value to store.
@end deffn

@geindex choices (optparse.Option attribute)
@anchor{library/optparse optparse Option choices}@anchor{4015}
@deffn {Attribute} Option.choices

For options of type @code{"choice"}, the list of strings the user may choose
from.
@end deffn

@geindex callback (optparse.Option attribute)
@anchor{library/optparse optparse Option callback}@anchor{4016}
@deffn {Attribute} Option.callback

For options with action @code{"callback"}, the callable to call when this option
is seen.  See section @ref{3ff4,,Option Callbacks} for detail on the
arguments passed to the callable.
@end deffn

@geindex callback_args (optparse.Option attribute)
@geindex callback_kwargs (optparse.Option attribute)
@anchor{library/optparse optparse Option callback_args}@anchor{4017}
@deffn {Attribute} Option.callback_args
@anchor{library/optparse optparse Option callback_kwargs}@anchor{4018}
@deffnx {Attribute} Option.callback_kwargs

Additional positional and keyword arguments to pass to @code{callback} after the
four standard callback arguments.
@end deffn

@geindex help (optparse.Option attribute)
@anchor{library/optparse optparse Option help}@anchor{3fe9}
@deffn {Attribute} Option.help

Help text to print for this option when listing all available options after
the user supplies a @ref{3fe9,,help} option (such as @code{--help}).  If
no help text is supplied, the option will be listed without help text.  To
hide this option, use the special value @code{optparse.SUPPRESS_HELP}.
@end deffn

@geindex metavar (optparse.Option attribute)
@anchor{library/optparse optparse Option metavar}@anchor{4019}
@deffn {Attribute} Option.metavar

(default: derived from option strings)

Stand-in for the option argument(s) to use when printing help text.  See
section @ref{3fe4,,Tutorial} for an example.
@end deffn

@node Standard option actions,Standard option types,Option attributes,Reference Guide
@anchor{library/optparse optparse-standard-option-actions}@anchor{400d}@anchor{library/optparse standard-option-actions}@anchor{401a}
@subsubsection Standard option actions


The various option actions all have slightly different requirements and effects.
Most actions have several relevant option attributes which you may specify to
guide @ref{bf,,optparse}’s behaviour; a few have required attributes, which you
must specify for any option using that action.


@itemize *

@item 
@code{"store"} [relevant: @ref{3fe7,,type}, @ref{3fe8,,dest},
@ref{4013,,nargs}, @ref{4015,,choices}]

The option must be followed by an argument, which is converted to a value
according to @ref{3fe7,,type} and stored in @ref{3fe8,,dest}.  If
@ref{4013,,nargs} > 1, multiple arguments will be consumed from the
command line; all will be converted according to @ref{3fe7,,type} and
stored to @ref{3fe8,,dest} as a tuple.  See the
@ref{4010,,Standard option types} section.

If @ref{4015,,choices} is supplied (a list or tuple of strings), the type
defaults to @code{"choice"}.

If @ref{3fe7,,type} is not supplied, it defaults to @code{"string"}.

If @ref{3fe8,,dest} is not supplied, @ref{bf,,optparse} derives a destination
from the first long option string (e.g., @code{--foo-bar} implies
@code{foo_bar}). If there are no long option strings, @ref{bf,,optparse} derives a
destination from the first short option string (e.g., @code{-f} implies @code{f}).

Example:

@example
parser.add_option("-f")
parser.add_option("-p", type="float", nargs=3, dest="point")
@end example

As it parses the command line

@example
-f foo.txt -p 1 -3.5 4 -fbar.txt
@end example

@ref{bf,,optparse} will set

@example
options.f = "foo.txt"
options.point = (1.0, -3.5, 4.0)
options.f = "bar.txt"
@end example

@item 
@code{"store_const"} [required: @ref{4014,,const}; relevant:
@ref{3fe8,,dest}]

The value @ref{4014,,const} is stored in @ref{3fe8,,dest}.

Example:

@example
parser.add_option("-q", "--quiet",
                  action="store_const", const=0, dest="verbose")
parser.add_option("-v", "--verbose",
                  action="store_const", const=1, dest="verbose")
parser.add_option("--noisy",
                  action="store_const", const=2, dest="verbose")
@end example

If @code{--noisy} is seen, @ref{bf,,optparse} will set

@example
options.verbose = 2
@end example

@item 
@code{"store_true"} [relevant: @ref{3fe8,,dest}]

A special case of @code{"store_const"} that stores @code{True} to
@ref{3fe8,,dest}.

@item 
@code{"store_false"} [relevant: @ref{3fe8,,dest}]

Like @code{"store_true"}, but stores @code{False}.

Example:

@example
parser.add_option("--clobber", action="store_true", dest="clobber")
parser.add_option("--no-clobber", action="store_false", dest="clobber")
@end example

@item 
@code{"append"} [relevant: @ref{3fe7,,type}, @ref{3fe8,,dest},
@ref{4013,,nargs}, @ref{4015,,choices}]

The option must be followed by an argument, which is appended to the list in
@ref{3fe8,,dest}.  If no default value for @ref{3fe8,,dest} is
supplied, an empty list is automatically created when @ref{bf,,optparse} first
encounters this option on the command-line.  If @ref{4013,,nargs} > 1,
multiple arguments are consumed, and a tuple of length @ref{4013,,nargs}
is appended to @ref{3fe8,,dest}.

The defaults for @ref{3fe7,,type} and @ref{3fe8,,dest} are the same as
for the @code{"store"} action.

Example:

@example
parser.add_option("-t", "--tracks", action="append", type="int")
@end example

If @code{-t3} is seen on the command-line, @ref{bf,,optparse} does the equivalent
of:

@example
options.tracks = []
options.tracks.append(int("3"))
@end example

If, a little later on, @code{--tracks=4} is seen, it does:

@example
options.tracks.append(int("4"))
@end example

The @code{append} action calls the @code{append} method on the current value of the
option.  This means that any default value specified must have an @code{append}
method.  It also means that if the default value is non-empty, the default
elements will be present in the parsed value for the option, with any values
from the command line appended after those default values:

@example
>>> parser.add_option("--files", action="append", default=['~/.mypkg/defaults'])
>>> opts, args = parser.parse_args(['--files', 'overrides.mypkg'])
>>> opts.files
['~/.mypkg/defaults', 'overrides.mypkg']
@end example

@item 
@code{"append_const"} [required: @ref{4014,,const}; relevant:
@ref{3fe8,,dest}]

Like @code{"store_const"}, but the value @ref{4014,,const} is appended to
@ref{3fe8,,dest}; as with @code{"append"}, @ref{3fe8,,dest} defaults to
@code{None}, and an empty list is automatically created the first time the option
is encountered.

@item 
@code{"count"} [relevant: @ref{3fe8,,dest}]

Increment the integer stored at @ref{3fe8,,dest}.  If no default value is
supplied, @ref{3fe8,,dest} is set to zero before being incremented the
first time.

Example:

@example
parser.add_option("-v", action="count", dest="verbosity")
@end example

The first time @code{-v} is seen on the command line, @ref{bf,,optparse} does the
equivalent of:

@example
options.verbosity = 0
options.verbosity += 1
@end example

Every subsequent occurrence of @code{-v} results in

@example
options.verbosity += 1
@end example

@item 
@code{"callback"} [required: @ref{4016,,callback}; relevant:
@ref{3fe7,,type}, @ref{4013,,nargs}, @ref{4017,,callback_args},
@ref{4018,,callback_kwargs}]

Call the function specified by @ref{4016,,callback}, which is called as

@example
func(option, opt_str, value, parser, *args, **kwargs)
@end example

See section @ref{3ff4,,Option Callbacks} for more detail.

@item 
@code{"help"}

Prints a complete help message for all the options in the current option
parser.  The help message is constructed from the @code{usage} string passed to
OptionParser’s constructor and the @ref{3fe9,,help} string passed to every
option.

If no @ref{3fe9,,help} string is supplied for an option, it will still be
listed in the help message.  To omit an option entirely, use the special value
@code{optparse.SUPPRESS_HELP}.

@ref{bf,,optparse} automatically adds a @ref{3fe9,,help} option to all
OptionParsers, so you do not normally need to create one.

Example:

@example
from optparse import OptionParser, SUPPRESS_HELP

# usually, a help option is added automatically, but that can
# be suppressed using the add_help_option argument
parser = OptionParser(add_help_option=False)

parser.add_option("-h", "--help", action="help")
parser.add_option("-v", action="store_true", dest="verbose",
                  help="Be moderately verbose")
parser.add_option("--file", dest="filename",
                  help="Input file to read data from")
parser.add_option("--secret", help=SUPPRESS_HELP)
@end example

If @ref{bf,,optparse} sees either @code{-h} or @code{--help} on the command line,
it will print something like the following help message to stdout (assuming
@code{sys.argv[0]} is @code{"foo.py"}):

@example
Usage: foo.py [options]

Options:
  -h, --help        Show this help message and exit
  -v                Be moderately verbose
  --file=FILENAME   Input file to read data from
@end example

After printing the help message, @ref{bf,,optparse} terminates your process with
@code{sys.exit(0)}.

@item 
@code{"version"}

Prints the version number supplied to the OptionParser to stdout and exits.
The version number is actually formatted and printed by the
@code{print_version()} method of OptionParser.  Generally only relevant if the
@code{version} argument is supplied to the OptionParser constructor.  As with
@ref{3fe9,,help} options, you will rarely create @code{version} options,
since @ref{bf,,optparse} automatically adds them when needed.
@end itemize

@node Standard option types,Parsing arguments<2>,Standard option actions,Reference Guide
@anchor{library/optparse optparse-standard-option-types}@anchor{4010}@anchor{library/optparse standard-option-types}@anchor{401b}
@subsubsection Standard option types


@ref{bf,,optparse} has five built-in option types: @code{"string"}, @code{"int"},
@code{"choice"}, @code{"float"} and @code{"complex"}.  If you need to add new
option types, see section @ref{3fec,,Extending optparse}.

Arguments to string options are not checked or converted in any way: the text on
the command line is stored in the destination (or passed to the callback) as-is.

Integer arguments (type @code{"int"}) are parsed as follows:


@itemize *

@item 
if the number starts with @code{0x}, it is parsed as a hexadecimal number

@item 
if the number starts with @code{0}, it is parsed as an octal number

@item 
if the number starts with @code{0b}, it is parsed as a binary number

@item 
otherwise, the number is parsed as a decimal number
@end itemize

The conversion is done by calling @ref{1b1,,int()} with the appropriate base (2, 8,
10, or 16).  If this fails, so will @ref{bf,,optparse}, although with a more useful
error message.

@code{"float"} and @code{"complex"} option arguments are converted directly with
@ref{256,,float()} and @ref{33b,,complex()}, with similar error-handling.

@code{"choice"} options are a subtype of @code{"string"} options.  The
@ref{4015,,choices} option attribute (a sequence of strings) defines the
set of allowed option arguments.  @code{optparse.check_choice()} compares
user-supplied option arguments against this master list and raises
@code{OptionValueError} if an invalid string is given.

@node Parsing arguments<2>,Querying and manipulating your option parser,Standard option types,Reference Guide
@anchor{library/optparse optparse-parsing-arguments}@anchor{401c}@anchor{library/optparse parsing-arguments}@anchor{401d}
@subsubsection Parsing arguments


The whole point of creating and populating an OptionParser is to call its
@code{parse_args()} method:

@example
(options, args) = parser.parse_args(args=None, values=None)
@end example

where the input parameters are


@table @asis

@item @code{args}

the list of arguments to process (default: @code{sys.argv[1:]})

@item @code{values}

an @code{optparse.Values} object to store option arguments in (default: a
new instance of @code{Values}) – if you give an existing object, the
option defaults will not be initialized on it
@end table

and the return values are


@table @asis

@item @code{options}

the same object that was passed in as @code{values}, or the optparse.Values
instance created by @ref{bf,,optparse}

@item @code{args}

the leftover positional arguments after all options have been processed
@end table

The most common usage is to supply neither keyword argument.  If you supply
@code{values}, it will be modified with repeated @ref{1a37,,setattr()} calls (roughly one
for every option argument stored to an option destination) and returned by
@code{parse_args()}.

If @code{parse_args()} encounters any errors in the argument list, it calls the
OptionParser’s @code{error()} method with an appropriate end-user error message.
This ultimately terminates your process with an exit status of 2 (the
traditional Unix exit status for command-line errors).

@node Querying and manipulating your option parser,Conflicts between options,Parsing arguments<2>,Reference Guide
@anchor{library/optparse optparse-querying-manipulating-option-parser}@anchor{401e}@anchor{library/optparse querying-and-manipulating-your-option-parser}@anchor{401f}
@subsubsection Querying and manipulating your option parser


The default behavior of the option parser can be customized slightly, and you
can also poke around your option parser and see what’s there.  OptionParser
provides several methods to help you out:

@geindex disable_interspersed_args() (optparse.OptionParser method)
@anchor{library/optparse optparse OptionParser disable_interspersed_args}@anchor{2511}
@deffn {Method} OptionParser.disable_interspersed_args ()

Set parsing to stop on the first non-option.  For example, if @code{-a} and
@code{-b} are both simple options that take no arguments, @ref{bf,,optparse}
normally accepts this syntax:

@example
prog -a arg1 -b arg2
@end example

and treats it as equivalent to

@example
prog -a -b arg1 arg2
@end example

To disable this feature, call @ref{2511,,disable_interspersed_args()}.  This
restores traditional Unix syntax, where option parsing stops with the first
non-option argument.

Use this if you have a command processor which runs another command which has
options of its own and you want to make sure these options don’t get
confused.  For example, each command might have a different set of options.
@end deffn

@geindex enable_interspersed_args() (optparse.OptionParser method)
@anchor{library/optparse optparse OptionParser enable_interspersed_args}@anchor{4020}
@deffn {Method} OptionParser.enable_interspersed_args ()

Set parsing to not stop on the first non-option, allowing interspersing
switches with command arguments.  This is the default behavior.
@end deffn

@geindex get_option() (optparse.OptionParser method)
@anchor{library/optparse optparse OptionParser get_option}@anchor{4021}
@deffn {Method} OptionParser.get_option (opt_str)

Returns the Option instance with the option string @emph{opt_str}, or @code{None} if
no options have that option string.
@end deffn

@geindex has_option() (optparse.OptionParser method)
@anchor{library/optparse optparse OptionParser has_option}@anchor{4022}
@deffn {Method} OptionParser.has_option (opt_str)

Return @code{True} if the OptionParser has an option with option string @emph{opt_str}
(e.g., @code{-q} or @code{--verbose}).
@end deffn

@geindex remove_option() (optparse.OptionParser method)
@anchor{library/optparse optparse OptionParser remove_option}@anchor{4023}
@deffn {Method} OptionParser.remove_option (opt_str)

If the @ref{3fdb,,OptionParser} has an option corresponding to @emph{opt_str}, that
option is removed.  If that option provided any other option strings, all of
those option strings become invalid. If @emph{opt_str} does not occur in any
option belonging to this @ref{3fdb,,OptionParser}, raises @ref{17c,,ValueError}.
@end deffn

@node Conflicts between options,Cleanup<2>,Querying and manipulating your option parser,Reference Guide
@anchor{library/optparse conflicts-between-options}@anchor{4024}@anchor{library/optparse optparse-conflicts-between-options}@anchor{4008}
@subsubsection Conflicts between options


If you’re not careful, it’s easy to define options with conflicting option
strings:

@example
parser.add_option("-n", "--dry-run", ...)
...
parser.add_option("-n", "--noisy", ...)
@end example

(This is particularly true if you’ve defined your own OptionParser subclass with
some standard options.)

Every time you add an option, @ref{bf,,optparse} checks for conflicts with existing
options.  If it finds any, it invokes the current conflict-handling mechanism.
You can set the conflict-handling mechanism either in the constructor:

@example
parser = OptionParser(..., conflict_handler=handler)
@end example

or with a separate call:

@example
parser.set_conflict_handler(handler)
@end example

The available conflict handlers are:

@quotation


@table @asis

@item @code{"error"} (default)

assume option conflicts are a programming error and raise
@code{OptionConflictError}

@item @code{"resolve"}

resolve option conflicts intelligently (see below)
@end table
@end quotation

As an example, let’s define an @ref{3fdb,,OptionParser} that resolves conflicts
intelligently and add conflicting options to it:

@example
parser = OptionParser(conflict_handler="resolve")
parser.add_option("-n", "--dry-run", ..., help="do no harm")
parser.add_option("-n", "--noisy", ..., help="be noisy")
@end example

At this point, @ref{bf,,optparse} detects that a previously-added option is already
using the @code{-n} option string.  Since @code{conflict_handler} is @code{"resolve"},
it resolves the situation by removing @code{-n} from the earlier option’s list of
option strings.  Now @code{--dry-run} is the only way for the user to activate
that option.  If the user asks for help, the help message will reflect that:

@example
Options:
  --dry-run     do no harm
  ...
  -n, --noisy   be noisy
@end example

It’s possible to whittle away the option strings for a previously-added option
until there are none left, and the user has no way of invoking that option from
the command-line.  In that case, @ref{bf,,optparse} removes that option completely,
so it doesn’t show up in help text or anywhere else. Carrying on with our
existing OptionParser:

@example
parser.add_option("--dry-run", ..., help="new dry-run option")
@end example

At this point, the original @code{-n}/@code{--dry-run} option is no longer
accessible, so @ref{bf,,optparse} removes it, leaving this help text:

@example
Options:
  ...
  -n, --noisy   be noisy
  --dry-run     new dry-run option
@end example

@node Cleanup<2>,Other methods,Conflicts between options,Reference Guide
@anchor{library/optparse cleanup}@anchor{4025}@anchor{library/optparse optparse-cleanup}@anchor{4026}
@subsubsection Cleanup


OptionParser instances have several cyclic references.  This should not be a
problem for Python’s garbage collector, but you may wish to break the cyclic
references explicitly by calling @code{destroy()} on your
OptionParser once you are done with it.  This is particularly useful in
long-running applications where large object graphs are reachable from your
OptionParser.

@node Other methods,,Cleanup<2>,Reference Guide
@anchor{library/optparse optparse-other-methods}@anchor{4027}@anchor{library/optparse other-methods}@anchor{4028}
@subsubsection Other methods


OptionParser supports several other public methods:

@geindex set_usage() (optparse.OptionParser method)
@anchor{library/optparse optparse OptionParser set_usage}@anchor{4029}
@deffn {Method} OptionParser.set_usage (usage)

Set the usage string according to the rules described above for the @code{usage}
constructor keyword argument.  Passing @code{None} sets the default usage
string; use @code{optparse.SUPPRESS_USAGE} to suppress a usage message.
@end deffn

@geindex print_usage() (optparse.OptionParser method)
@anchor{library/optparse optparse OptionParser print_usage}@anchor{3fff}
@deffn {Method} OptionParser.print_usage (file=None)

Print the usage message for the current program (@code{self.usage}) to @emph{file}
(default stdout).  Any occurrence of the string @code{%prog} in @code{self.usage}
is replaced with the name of the current program.  Does nothing if
@code{self.usage} is empty or not defined.
@end deffn

@geindex get_usage() (optparse.OptionParser method)
@anchor{library/optparse optparse OptionParser get_usage}@anchor{402a}
@deffn {Method} OptionParser.get_usage ()

Same as @ref{3fff,,print_usage()} but returns the usage string instead of
printing it.
@end deffn

@geindex set_defaults() (optparse.OptionParser method)
@anchor{library/optparse optparse OptionParser set_defaults}@anchor{4012}
@deffn {Method} OptionParser.set_defaults (dest=value, ...)

Set default values for several option destinations at once.  Using
@ref{4012,,set_defaults()} is the preferred way to set default values for options,
since multiple options can share the same destination.  For example, if
several “mode” options all set the same destination, any one of them can set
the default, and the last one wins:

@example
parser.add_option("--advanced", action="store_const",
                  dest="mode", const="advanced",
                  default="novice")    # overridden below
parser.add_option("--novice", action="store_const",
                  dest="mode", const="novice",
                  default="advanced")  # overrides above setting
@end example

To avoid this confusion, use @ref{4012,,set_defaults()}:

@example
parser.set_defaults(mode="advanced")
parser.add_option("--advanced", action="store_const",
                  dest="mode", const="advanced")
parser.add_option("--novice", action="store_const",
                  dest="mode", const="novice")
@end example
@end deffn

@node Option Callbacks,Extending optparse,Reference Guide,optparse — Parser for command line options
@anchor{library/optparse option-callbacks}@anchor{402b}@anchor{library/optparse optparse-option-callbacks}@anchor{3ff4}
@subsubsection Option Callbacks


When @ref{bf,,optparse}’s built-in actions and types aren’t quite enough for your
needs, you have two choices: extend @ref{bf,,optparse} or define a callback option.
Extending @ref{bf,,optparse} is more general, but overkill for a lot of simple
cases.  Quite often a simple callback is all you need.

There are two steps to defining a callback option:


@itemize *

@item 
define the option itself using the @code{"callback"} action

@item 
write the callback; this is a function (or method) that takes at least four
arguments, as described below
@end itemize

@menu
* Defining a callback option:: 
* How callbacks are called:: 
* Raising errors in a callback:: 
* Callback example 1; trivial callback: Callback example 1 trivial callback. 
* Callback example 2; check option order: Callback example 2 check option order. 
* Callback example 3; check option order (generalized): Callback example 3 check option order generalized. 
* Callback example 4; check arbitrary condition: Callback example 4 check arbitrary condition. 
* Callback example 5; fixed arguments: Callback example 5 fixed arguments. 
* Callback example 6; variable arguments: Callback example 6 variable arguments. 

@end menu

@node Defining a callback option,How callbacks are called,,Option Callbacks
@anchor{library/optparse defining-a-callback-option}@anchor{402c}@anchor{library/optparse optparse-defining-callback-option}@anchor{402d}
@subsubsection Defining a callback option


As always, the easiest way to define a callback option is by using the
@ref{2510,,OptionParser.add_option()} method.  Apart from @ref{3fe6,,action}, the
only option attribute you must specify is @code{callback}, the function to call:

@example
parser.add_option("-c", action="callback", callback=my_callback)
@end example

@code{callback} is a function (or other callable object), so you must have already
defined @code{my_callback()} when you create this callback option. In this simple
case, @ref{bf,,optparse} doesn’t even know if @code{-c} takes any arguments,
which usually means that the option takes no arguments—the mere presence of
@code{-c} on the command-line is all it needs to know.  In some
circumstances, though, you might want your callback to consume an arbitrary
number of command-line arguments.  This is where writing callbacks gets tricky;
it’s covered later in this section.

@ref{bf,,optparse} always passes four particular arguments to your callback, and it
will only pass additional arguments if you specify them via
@ref{4017,,callback_args} and @ref{4018,,callback_kwargs}.  Thus, the
minimal callback function signature is:

@example
def my_callback(option, opt, value, parser):
@end example

The four arguments to a callback are described below.

There are several other option attributes that you can supply when you define a
callback option:


@table @asis

@item @ref{3fe7,,type}

has its usual meaning: as with the @code{"store"} or @code{"append"} actions, it
instructs @ref{bf,,optparse} to consume one argument and convert it to
@ref{3fe7,,type}.  Rather than storing the converted value(s) anywhere,
though, @ref{bf,,optparse} passes it to your callback function.

@item @ref{4013,,nargs}

also has its usual meaning: if it is supplied and > 1, @ref{bf,,optparse} will
consume @ref{4013,,nargs} arguments, each of which must be convertible to
@ref{3fe7,,type}.  It then passes a tuple of converted values to your
callback.

@item @ref{4017,,callback_args}

a tuple of extra positional arguments to pass to the callback

@item @ref{4018,,callback_kwargs}

a dictionary of extra keyword arguments to pass to the callback
@end table

@node How callbacks are called,Raising errors in a callback,Defining a callback option,Option Callbacks
@anchor{library/optparse how-callbacks-are-called}@anchor{402e}@anchor{library/optparse optparse-how-callbacks-called}@anchor{402f}
@subsubsection How callbacks are called


All callbacks are called as follows:

@example
func(option, opt_str, value, parser, *args, **kwargs)
@end example

where


@table @asis

@item @code{option}

is the Option instance that’s calling the callback

@item @code{opt_str}

is the option string seen on the command-line that’s triggering the callback.
(If an abbreviated long option was used, @code{opt_str} will be the full,
canonical option string—e.g. if the user puts @code{--foo} on the
command-line as an abbreviation for @code{--foobar}, then @code{opt_str} will be
@code{"--foobar"}.)

@item @code{value}

is the argument to this option seen on the command-line.  @ref{bf,,optparse} will
only expect an argument if @ref{3fe7,,type} is set; the type of @code{value} will be
the type implied by the option’s type.  If @ref{3fe7,,type} for this option is
@code{None} (no argument expected), then @code{value} will be @code{None}.  If @ref{4013,,nargs}
> 1, @code{value} will be a tuple of values of the appropriate type.

@item @code{parser}

is the OptionParser instance driving the whole thing, mainly useful because
you can access some other interesting data through its instance attributes:


@table @asis

@item @code{parser.largs}

the current list of leftover arguments, ie. arguments that have been
consumed but are neither options nor option arguments. Feel free to modify
@code{parser.largs}, e.g. by adding more arguments to it.  (This list will
become @code{args}, the second return value of @code{parse_args()}.)

@item @code{parser.rargs}

the current list of remaining arguments, ie. with @code{opt_str} and
@code{value} (if applicable) removed, and only the arguments following them
still there.  Feel free to modify @code{parser.rargs}, e.g. by consuming more
arguments.

@item @code{parser.values}

the object where option values are by default stored (an instance of
optparse.OptionValues).  This lets callbacks use the same mechanism as the
rest of @ref{bf,,optparse} for storing option values; you don’t need to mess
around with globals or closures.  You can also access or modify the
value(s) of any options already encountered on the command-line.
@end table

@item @code{args}

is a tuple of arbitrary positional arguments supplied via the
@ref{4017,,callback_args} option attribute.

@item @code{kwargs}

is a dictionary of arbitrary keyword arguments supplied via
@ref{4018,,callback_kwargs}.
@end table

@node Raising errors in a callback,Callback example 1 trivial callback,How callbacks are called,Option Callbacks
@anchor{library/optparse optparse-raising-errors-in-callback}@anchor{4030}@anchor{library/optparse raising-errors-in-a-callback}@anchor{4031}
@subsubsection Raising errors in a callback


The callback function should raise @code{OptionValueError} if there are any
problems with the option or its argument(s).  @ref{bf,,optparse} catches this and
terminates the program, printing the error message you supply to stderr.  Your
message should be clear, concise, accurate, and mention the option at fault.
Otherwise, the user will have a hard time figuring out what they did wrong.

@node Callback example 1 trivial callback,Callback example 2 check option order,Raising errors in a callback,Option Callbacks
@anchor{library/optparse callback-example-1-trivial-callback}@anchor{4032}@anchor{library/optparse optparse-callback-example-1}@anchor{4033}
@subsubsection Callback example 1: trivial callback


Here’s an example of a callback option that takes no arguments, and simply
records that the option was seen:

@example
def record_foo_seen(option, opt_str, value, parser):
    parser.values.saw_foo = True

parser.add_option("--foo", action="callback", callback=record_foo_seen)
@end example

Of course, you could do that with the @code{"store_true"} action.

@node Callback example 2 check option order,Callback example 3 check option order generalized,Callback example 1 trivial callback,Option Callbacks
@anchor{library/optparse callback-example-2-check-option-order}@anchor{4034}@anchor{library/optparse optparse-callback-example-2}@anchor{4035}
@subsubsection Callback example 2: check option order


Here’s a slightly more interesting example: record the fact that @code{-a} is
seen, but blow up if it comes after @code{-b} in the command-line.

@example
def check_order(option, opt_str, value, parser):
    if parser.values.b:
        raise OptionValueError("can't use -a after -b")
    parser.values.a = 1
...
parser.add_option("-a", action="callback", callback=check_order)
parser.add_option("-b", action="store_true", dest="b")
@end example

@node Callback example 3 check option order generalized,Callback example 4 check arbitrary condition,Callback example 2 check option order,Option Callbacks
@anchor{library/optparse callback-example-3-check-option-order-generalized}@anchor{4036}@anchor{library/optparse optparse-callback-example-3}@anchor{4037}
@subsubsection Callback example 3: check option order (generalized)


If you want to re-use this callback for several similar options (set a flag, but
blow up if @code{-b} has already been seen), it needs a bit of work: the error
message and the flag that it sets must be generalized.

@example
def check_order(option, opt_str, value, parser):
    if parser.values.b:
        raise OptionValueError("can't use %s after -b" % opt_str)
    setattr(parser.values, option.dest, 1)
...
parser.add_option("-a", action="callback", callback=check_order, dest='a')
parser.add_option("-b", action="store_true", dest="b")
parser.add_option("-c", action="callback", callback=check_order, dest='c')
@end example

@node Callback example 4 check arbitrary condition,Callback example 5 fixed arguments,Callback example 3 check option order generalized,Option Callbacks
@anchor{library/optparse callback-example-4-check-arbitrary-condition}@anchor{4038}@anchor{library/optparse optparse-callback-example-4}@anchor{4039}
@subsubsection Callback example 4: check arbitrary condition


Of course, you could put any condition in there—you’re not limited to checking
the values of already-defined options.  For example, if you have options that
should not be called when the moon is full, all you have to do is this:

@example
def check_moon(option, opt_str, value, parser):
    if is_moon_full():
        raise OptionValueError("%s option invalid when moon is full"
                               % opt_str)
    setattr(parser.values, option.dest, 1)
...
parser.add_option("--foo",
                  action="callback", callback=check_moon, dest="foo")
@end example

(The definition of @code{is_moon_full()} is left as an exercise for the reader.)

@node Callback example 5 fixed arguments,Callback example 6 variable arguments,Callback example 4 check arbitrary condition,Option Callbacks
@anchor{library/optparse callback-example-5-fixed-arguments}@anchor{403a}@anchor{library/optparse optparse-callback-example-5}@anchor{403b}
@subsubsection Callback example 5: fixed arguments


Things get slightly more interesting when you define callback options that take
a fixed number of arguments.  Specifying that a callback option takes arguments
is similar to defining a @code{"store"} or @code{"append"} option: if you define
@ref{3fe7,,type}, then the option takes one argument that must be
convertible to that type; if you further define @ref{4013,,nargs}, then the
option takes @ref{4013,,nargs} arguments.

Here’s an example that just emulates the standard @code{"store"} action:

@example
def store_value(option, opt_str, value, parser):
    setattr(parser.values, option.dest, value)
...
parser.add_option("--foo",
                  action="callback", callback=store_value,
                  type="int", nargs=3, dest="foo")
@end example

Note that @ref{bf,,optparse} takes care of consuming 3 arguments and converting
them to integers for you; all you have to do is store them.  (Or whatever;
obviously you don’t need a callback for this example.)

@node Callback example 6 variable arguments,,Callback example 5 fixed arguments,Option Callbacks
@anchor{library/optparse callback-example-6-variable-arguments}@anchor{403c}@anchor{library/optparse optparse-callback-example-6}@anchor{403d}
@subsubsection Callback example 6: variable arguments


Things get hairy when you want an option to take a variable number of arguments.
For this case, you must write a callback, as @ref{bf,,optparse} doesn’t provide any
built-in capabilities for it.  And you have to deal with certain intricacies of
conventional Unix command-line parsing that @ref{bf,,optparse} normally handles for
you.  In particular, callbacks should implement the conventional rules for bare
@code{--} and @code{-} arguments:


@itemize *

@item 
either @code{--} or @code{-} can be option arguments

@item 
bare @code{--} (if not the argument to some option): halt command-line
processing and discard the @code{--}

@item 
bare @code{-} (if not the argument to some option): halt command-line
processing but keep the @code{-} (append it to @code{parser.largs})
@end itemize

If you want an option that takes a variable number of arguments, there are
several subtle, tricky issues to worry about.  The exact implementation you
choose will be based on which trade-offs you’re willing to make for your
application (which is why @ref{bf,,optparse} doesn’t support this sort of thing
directly).

Nevertheless, here’s a stab at a callback for an option with variable
arguments:

@example
def vararg_callback(option, opt_str, value, parser):
    assert value is None
    value = []

    def floatable(str):
        try:
            float(str)
            return True
        except ValueError:
            return False

    for arg in parser.rargs:
        # stop on --foo like options
        if arg[:2] == "--" and len(arg) > 2:
            break
        # stop on -a, but not on -3 or -3.0
        if arg[:1] == "-" and len(arg) > 1 and not floatable(arg):
            break
        value.append(arg)

    del parser.rargs[:len(value)]
    setattr(parser.values, option.dest, value)

...
parser.add_option("-c", "--callback", dest="vararg_attr",
                  action="callback", callback=vararg_callback)
@end example

@node Extending optparse,,Option Callbacks,optparse — Parser for command line options
@anchor{library/optparse extending-optparse}@anchor{403e}@anchor{library/optparse optparse-extending-optparse}@anchor{3fec}
@subsubsection Extending @code{optparse}


Since the two major controlling factors in how @ref{bf,,optparse} interprets
command-line options are the action and type of each option, the most likely
direction of extension is to add new actions and new types.

@menu
* Adding new types:: 
* Adding new actions:: 

@end menu

@node Adding new types,Adding new actions,,Extending optparse
@anchor{library/optparse adding-new-types}@anchor{403f}@anchor{library/optparse optparse-adding-new-types}@anchor{4040}
@subsubsection Adding new types


To add new types, you need to define your own subclass of @ref{bf,,optparse}’s
@code{Option} class.  This class has a couple of attributes that define
@ref{bf,,optparse}’s types: @ref{4041,,TYPES} and @ref{4042,,TYPE_CHECKER}.

@geindex TYPES (optparse.Option attribute)
@anchor{library/optparse optparse Option TYPES}@anchor{4041}
@deffn {Attribute} Option.TYPES

A tuple of type names; in your subclass, simply define a new tuple
@ref{4041,,TYPES} that builds on the standard one.
@end deffn

@geindex TYPE_CHECKER (optparse.Option attribute)
@anchor{library/optparse optparse Option TYPE_CHECKER}@anchor{4042}
@deffn {Attribute} Option.TYPE_CHECKER

A dictionary mapping type names to type-checking functions.  A type-checking
function has the following signature:

@example
def check_mytype(option, opt, value)
@end example

where @code{option} is an @code{Option} instance, @code{opt} is an option string
(e.g., @code{-f}), and @code{value} is the string from the command line that must
be checked and converted to your desired type.  @code{check_mytype()} should
return an object of the hypothetical type @code{mytype}.  The value returned by
a type-checking function will wind up in the OptionValues instance returned
by @code{OptionParser.parse_args()}, or be passed to a callback as the
@code{value} parameter.

Your type-checking function should raise @code{OptionValueError} if it
encounters any problems.  @code{OptionValueError} takes a single string
argument, which is passed as-is to @ref{3fdb,,OptionParser}’s @code{error()}
method, which in turn prepends the program name and the string @code{"error:"}
and prints everything to stderr before terminating the process.
@end deffn

Here’s a silly example that demonstrates adding a @code{"complex"} option type to
parse Python-style complex numbers on the command line.  (This is even sillier
than it used to be, because @ref{bf,,optparse} 1.3 added built-in support for
complex numbers, but never mind.)

First, the necessary imports:

@example
from copy import copy
from optparse import Option, OptionValueError
@end example

You need to define your type-checker first, since it’s referred to later (in the
@ref{4042,,TYPE_CHECKER} class attribute of your Option subclass):

@example
def check_complex(option, opt, value):
    try:
        return complex(value)
    except ValueError:
        raise OptionValueError(
            "option %s: invalid complex value: %r" % (opt, value))
@end example

Finally, the Option subclass:

@example
class MyOption (Option):
    TYPES = Option.TYPES + ("complex",)
    TYPE_CHECKER = copy(Option.TYPE_CHECKER)
    TYPE_CHECKER["complex"] = check_complex
@end example

(If we didn’t make a @ref{24,,copy()} of @ref{4042,,Option.TYPE_CHECKER}, we would end
up modifying the @ref{4042,,TYPE_CHECKER} attribute of @ref{bf,,optparse}’s
Option class.  This being Python, nothing stops you from doing that except good
manners and common sense.)

That’s it!  Now you can write a script that uses the new option type just like
any other @ref{bf,,optparse}-based script, except you have to instruct your
OptionParser to use MyOption instead of Option:

@example
parser = OptionParser(option_class=MyOption)
parser.add_option("-c", type="complex")
@end example

Alternately, you can build your own option list and pass it to OptionParser; if
you don’t use @code{add_option()} in the above way, you don’t need to tell
OptionParser which option class to use:

@example
option_list = [MyOption("-c", action="store", type="complex", dest="c")]
parser = OptionParser(option_list=option_list)
@end example

@node Adding new actions,,Adding new types,Extending optparse
@anchor{library/optparse adding-new-actions}@anchor{4043}@anchor{library/optparse optparse-adding-new-actions}@anchor{4044}
@subsubsection Adding new actions


Adding new actions is a bit trickier, because you have to understand that
@ref{bf,,optparse} has a couple of classifications for actions:


@table @asis

@item “store” actions

actions that result in @ref{bf,,optparse} storing a value to an attribute of the
current OptionValues instance; these options require a @ref{3fe8,,dest}
attribute to be supplied to the Option constructor.

@item “typed” actions

actions that take a value from the command line and expect it to be of a
certain type; or rather, a string that can be converted to a certain type.
These options require a @ref{3fe7,,type} attribute to the Option
constructor.
@end table

These are overlapping sets: some default “store” actions are @code{"store"},
@code{"store_const"}, @code{"append"}, and @code{"count"}, while the default “typed”
actions are @code{"store"}, @code{"append"}, and @code{"callback"}.

When you add an action, you need to categorize it by listing it in at least one
of the following class attributes of Option (all are lists of strings):

@geindex ACTIONS (optparse.Option attribute)
@anchor{library/optparse optparse Option ACTIONS}@anchor{4045}
@deffn {Attribute} Option.ACTIONS

All actions must be listed in ACTIONS.
@end deffn

@geindex STORE_ACTIONS (optparse.Option attribute)
@anchor{library/optparse optparse Option STORE_ACTIONS}@anchor{4046}
@deffn {Attribute} Option.STORE_ACTIONS

“store” actions are additionally listed here.
@end deffn

@geindex TYPED_ACTIONS (optparse.Option attribute)
@anchor{library/optparse optparse Option TYPED_ACTIONS}@anchor{4047}
@deffn {Attribute} Option.TYPED_ACTIONS

“typed” actions are additionally listed here.
@end deffn

@geindex ALWAYS_TYPED_ACTIONS (optparse.Option attribute)
@anchor{library/optparse optparse Option ALWAYS_TYPED_ACTIONS}@anchor{4048}
@deffn {Attribute} Option.ALWAYS_TYPED_ACTIONS

Actions that always take a type (i.e. whose options always take a value) are
additionally listed here.  The only effect of this is that @ref{bf,,optparse}
assigns the default type, @code{"string"}, to options with no explicit type
whose action is listed in @ref{4048,,ALWAYS_TYPED_ACTIONS}.
@end deffn

In order to actually implement your new action, you must override Option’s
@code{take_action()} method and add a case that recognizes your action.

For example, let’s add an @code{"extend"} action.  This is similar to the standard
@code{"append"} action, but instead of taking a single value from the command-line
and appending it to an existing list, @code{"extend"} will take multiple values in
a single comma-delimited string, and extend an existing list with them.  That
is, if @code{--names} is an @code{"extend"} option of type @code{"string"}, the command
line

@example
--names=foo,bar --names blah --names ding,dong
@end example

would result in a list

@example
["foo", "bar", "blah", "ding", "dong"]
@end example

Again we define a subclass of Option:

@example
class MyOption(Option):

    ACTIONS = Option.ACTIONS + ("extend",)
    STORE_ACTIONS = Option.STORE_ACTIONS + ("extend",)
    TYPED_ACTIONS = Option.TYPED_ACTIONS + ("extend",)
    ALWAYS_TYPED_ACTIONS = Option.ALWAYS_TYPED_ACTIONS + ("extend",)

    def take_action(self, action, dest, opt, value, values, parser):
        if action == "extend":
            lvalue = value.split(",")
            values.ensure_value(dest, []).extend(lvalue)
        else:
            Option.take_action(
                self, action, dest, opt, value, values, parser)
@end example

Features of note:


@itemize *

@item 
@code{"extend"} both expects a value on the command-line and stores that value
somewhere, so it goes in both @ref{4046,,STORE_ACTIONS} and
@ref{4047,,TYPED_ACTIONS}.

@item 
to ensure that @ref{bf,,optparse} assigns the default type of @code{"string"} to
@code{"extend"} actions, we put the @code{"extend"} action in
@ref{4048,,ALWAYS_TYPED_ACTIONS} as well.

@item 
@code{MyOption.take_action()} implements just this one new action, and passes
control back to @code{Option.take_action()} for the standard @ref{bf,,optparse}
actions.

@item 
@code{values} is an instance of the optparse_parser.Values class, which provides
the very useful @code{ensure_value()} method. @code{ensure_value()} is
essentially @ref{708,,getattr()} with a safety valve; it is called as

@example
values.ensure_value(attr, value)
@end example

If the @code{attr} attribute of @code{values} doesn’t exist or is @code{None}, then
ensure_value() first sets it to @code{value}, and then returns ‘value. This is
very handy for actions like @code{"extend"}, @code{"append"}, and @code{"count"}, all
of which accumulate data in a variable and expect that variable to be of a
certain type (a list for the first two, an integer for the latter).  Using
@code{ensure_value()} means that scripts using your action don’t have to worry
about setting a default value for the option destinations in question; they
can just leave the default as @code{None} and @code{ensure_value()} will take care of
getting it right when it’s needed.
@end itemize

@node imp — Access the import internals,,optparse — Parser for command line options,Superseded Modules
@anchor{library/imp doc}@anchor{4049}@anchor{library/imp imp-access-the-import-internals}@anchor{404a}@anchor{library/imp module-imp}@anchor{95}
@subsection @code{imp} — Access the import internals


@geindex module; imp

@strong{Source code:} Lib/imp.py@footnote{https://github.com/python/cpython/tree/main/Lib/imp.py}


Deprecated since version 3.4: The @ref{95,,imp} module is deprecated in favor of @ref{96,,importlib}.


@geindex statement; import


__________________________________________________________________


This module provides an interface to the mechanisms used to implement the
@ref{e35,,import} statement.  It defines the following constants and functions:

@geindex get_magic() (in module imp)
@anchor{library/imp imp get_magic}@anchor{ad5}
@deffn {Function} imp.get_magic ()

@geindex file; byte-code

Return the magic string value used to recognize byte-compiled code files
(@code{.pyc} files).  (This value may be different for each Python version.)


Deprecated since version 3.4: Use @ref{ad4,,importlib.util.MAGIC_NUMBER} instead.

@end deffn

@geindex get_suffixes() (in module imp)
@anchor{library/imp imp get_suffixes}@anchor{404b}
@deffn {Function} imp.get_suffixes ()

Return a list of 3-element tuples, each describing a particular type of
module. Each triple has the form @code{(suffix, mode, type)}, where @emph{suffix} is
a string to be appended to the module name to form the filename to search
for, @emph{mode} is the mode string to pass to the built-in @ref{1c8,,open()} function
to open the file (this can be @code{'r'} for text files or @code{'rb'} for binary
files), and @emph{type} is the file type, which has one of the values
@ref{404c,,PY_SOURCE}, @ref{404d,,PY_COMPILED}, or @ref{404e,,C_EXTENSION}, described
below.


Deprecated since version 3.3: Use the constants defined on @ref{98,,importlib.machinery} instead.

@end deffn

@geindex find_module() (in module imp)
@anchor{library/imp imp find_module}@anchor{404f}
@deffn {Function} imp.find_module (name[, path])

Try to find the module @emph{name}.  If @emph{path} is omitted or @code{None}, the list of
directory names given by @code{sys.path} is searched, but first a few special
places are searched: the function tries to find a built-in module with the
given name (@ref{4050,,C_BUILTIN}), then a frozen module (@ref{4051,,PY_FROZEN}),
and on some systems some other places are looked in as well (on Windows, it
looks in the registry which may point to a specific file).

Otherwise, @emph{path} must be a list of directory names; each directory is
searched for files with any of the suffixes returned by @ref{404b,,get_suffixes()}
above.  Invalid names in the list are silently ignored (but all list items
must be strings).

If search is successful, the return value is a 3-element tuple @code{(file,
pathname, description)}:

@emph{file} is an open @ref{d68,,file object} positioned at the beginning, @emph{pathname}
is the pathname of the file found, and @emph{description} is a 3-element tuple as
contained in the list returned by @ref{404b,,get_suffixes()} describing the kind of
module found.

If the module is built-in or frozen then @emph{file} and @emph{pathname} are both @code{None}
and the @emph{description} tuple contains empty strings for its suffix and mode;
the module type is indicated as given in parentheses above.  If the search
is unsuccessful, @ref{3b8,,ImportError} is raised.  Other exceptions indicate
problems with the arguments or environment.

If the module is a package, @emph{file} is @code{None}, @emph{pathname} is the package
path and the last item in the @emph{description} tuple is @ref{4052,,PKG_DIRECTORY}.

This function does not handle hierarchical module names (names containing
dots).  In order to find @emph{P.M}, that is, submodule @emph{M} of package @emph{P}, use
@ref{404f,,find_module()} and @ref{4053,,load_module()} to find and load package @emph{P}, and
then use @ref{404f,,find_module()} with the @emph{path} argument set to @code{P.__path__}.
When @emph{P} itself has a dotted name, apply this recipe recursively.


Deprecated since version 3.3: Use @ref{b98,,importlib.util.find_spec()} instead unless Python 3.3
compatibility is required, in which case use
@ref{310,,importlib.find_loader()}. For example usage of the former case,
see the @ref{3cd5,,Examples} section of the @ref{96,,importlib}
documentation.

@end deffn

@geindex load_module() (in module imp)
@anchor{library/imp imp load_module}@anchor{4053}
@deffn {Function} imp.load_module (name, file, pathname, description)

Load a module that was previously found by @ref{404f,,find_module()} (or by an
otherwise conducted search yielding compatible results).  This function does
more than importing the module: if the module was already imported, it will
reload the module!  The @emph{name} argument indicates the full
module name (including the package name, if this is a submodule of a
package).  The @emph{file} argument is an open file, and @emph{pathname} is the
corresponding file name; these can be @code{None} and @code{'@w{'}}, respectively, when
the module is a package or not being loaded from a file.  The @emph{description}
argument is a tuple, as would be returned by @ref{404b,,get_suffixes()}, describing
what kind of module must be loaded.

If the load is successful, the return value is the module object; otherwise,
an exception (usually @ref{3b8,,ImportError}) is raised.

@strong{Important:} the caller is responsible for closing the @emph{file} argument, if
it was not @code{None}, even when an exception is raised.  This is best done
using a @ref{f6d,,try} … @ref{4c1,,finally} statement.


Deprecated since version 3.3: If previously used in conjunction with @ref{404f,,imp.find_module()} then
consider using @ref{d3e,,importlib.import_module()}, otherwise use the loader
returned by the replacement you chose for @ref{404f,,imp.find_module()}. If you
called @ref{4053,,imp.load_module()} and related functions directly with file
path arguments then use a combination of
@ref{7fb,,importlib.util.spec_from_file_location()} and
@ref{972,,importlib.util.module_from_spec()}. See the @ref{3cd5,,Examples}
section of the @ref{96,,importlib} documentation for details of the various
approaches.

@end deffn

@geindex new_module() (in module imp)
@anchor{library/imp imp new_module}@anchor{4054}
@deffn {Function} imp.new_module (name)

Return a new empty module object called @emph{name}.  This object is @emph{not} inserted
in @code{sys.modules}.


Deprecated since version 3.4: Use @ref{972,,importlib.util.module_from_spec()} instead.

@end deffn

@geindex reload() (in module imp)
@anchor{library/imp imp reload}@anchor{e78}
@deffn {Function} imp.reload (module)

Reload a previously imported @emph{module}.  The argument must be a module object, so
it must have been successfully imported before.  This is useful if you have
edited the module source file using an external editor and want to try out the
new version without leaving the Python interpreter.  The return value is the
module object (the same as the @emph{module} argument).

When @code{reload(module)} is executed:


@itemize *

@item 
Python modules’ code is recompiled and the module-level code reexecuted,
defining a new set of objects which are bound to names in the module’s
dictionary.  The @code{init} function of extension modules is not called a second
time.

@item 
As with all other objects in Python the old objects are only reclaimed after
their reference counts drop to zero.

@item 
The names in the module namespace are updated to point to any new or changed
objects.

@item 
Other references to the old objects (such as names external to the module) are
not rebound to refer to the new objects and must be updated in each namespace
where they occur if that is desired.
@end itemize

There are a number of other caveats:

When a module is reloaded, its dictionary (containing the module’s global
variables) is retained.  Redefinitions of names will override the old
definitions, so this is generally not a problem.  If the new version of a module
does not define a name that was defined by the old version, the old definition
remains.  This feature can be used to the module’s advantage if it maintains a
global table or cache of objects — with a @ref{f6d,,try} statement it can test
for the table’s presence and skip its initialization if desired:

@example
try:
    cache
except NameError:
    cache = @{@}
@end example

It is legal though generally not very useful to reload built-in or dynamically
loaded modules, except for @ref{f7,,sys}, @ref{1,,__main__} and @ref{11,,builtins}.
In many cases, however, extension modules are not designed to be initialized
more than once, and may fail in arbitrary ways when reloaded.

If a module imports objects from another module using @ref{e51,,from} …
@ref{e35,,import} …, calling @ref{e78,,reload()} for the other module does not
redefine the objects imported from it — one way around this is to re-execute
the @code{from} statement, another is to use @code{import} and qualified
names (@emph{module}.*name*) instead.

If a module instantiates instances of a class, reloading the module that defines
the class does not affect the method definitions of the instances — they
continue to use the old class definition.  The same is true for derived classes.


Changed in version 3.3: Relies on both @code{__name__} and @code{__loader__} being defined on the module
being reloaded instead of just @code{__name__}.



Deprecated since version 3.4: Use @ref{66a,,importlib.reload()} instead.

@end deffn

The following functions are conveniences for handling 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} byte-compiled
file paths.


New in version 3.2.


@geindex cache_from_source() (in module imp)
@anchor{library/imp imp cache_from_source}@anchor{4055}
@deffn {Function} imp.cache_from_source (path, debug_override=None)

Return the 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} path to the byte-compiled file associated with the
source @emph{path}.  For example, if @emph{path} is @code{/foo/bar/baz.py} the return
value would be @code{/foo/bar/__pycache__/baz.cpython-32.pyc} for Python 3.2.
The @code{cpython-32} string comes from the current magic tag (see
@ref{4056,,get_tag()}; if @code{sys.implementation.cache_tag} is not defined then
@ref{8a3,,NotImplementedError} will be raised). By passing in @code{True} or
@code{False} for @emph{debug_override} you can override the system’s value for
@code{__debug__}, leading to optimized bytecode.

@emph{path} need not exist.


Changed in version 3.3: If @code{sys.implementation.cache_tag} is @code{None}, then
@ref{8a3,,NotImplementedError} is raised.



Deprecated since version 3.4: Use @ref{7f9,,importlib.util.cache_from_source()} instead.



Changed in version 3.5: The @emph{debug_override} parameter no longer creates a @code{.pyo} file.

@end deffn

@geindex source_from_cache() (in module imp)
@anchor{library/imp imp source_from_cache}@anchor{4057}
@deffn {Function} imp.source_from_cache (path)

Given the @emph{path} to a 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} file name, return the associated source code
file path.  For example, if @emph{path} is
@code{/foo/bar/__pycache__/baz.cpython-32.pyc} the returned path would be
@code{/foo/bar/baz.py}.  @emph{path} need not exist, however if it does not conform
to 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} format, a @ref{17c,,ValueError} is raised. If
@code{sys.implementation.cache_tag} is not defined,
@ref{8a3,,NotImplementedError} is raised.


Changed in version 3.3: Raise @ref{8a3,,NotImplementedError} when
@code{sys.implementation.cache_tag} is not defined.



Deprecated since version 3.4: Use @ref{7fa,,importlib.util.source_from_cache()} instead.

@end deffn

@geindex get_tag() (in module imp)
@anchor{library/imp imp get_tag}@anchor{4056}
@deffn {Function} imp.get_tag ()

Return the 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} magic tag string matching this version of Python’s
magic number, as returned by @ref{ad5,,get_magic()}.


Deprecated since version 3.4: Use @code{sys.implementation.cache_tag} directly starting
in Python 3.3.

@end deffn

The following functions help interact with the import system’s internal
locking mechanism.  Locking semantics of imports are an implementation
detail which may vary from release to release.  However, Python ensures
that circular imports work without any deadlocks.

@geindex lock_held() (in module imp)
@anchor{library/imp imp lock_held}@anchor{4058}
@deffn {Function} imp.lock_held ()

Return @code{True} if the global import lock is currently held, else
@code{False}. On platforms without threads, always return @code{False}.

On platforms with threads, a thread executing an import first holds a
global import lock, then sets up a per-module lock for the rest of the
import.  This blocks other threads from importing the same module until
the original import completes, preventing other threads from seeing
incomplete module objects constructed by the original thread.  An
exception is made for circular imports, which by construction have to
expose an incomplete module object at some point.


Changed in version 3.3: The locking scheme has changed to per-module locks for
the most part.  A global import lock is kept for some critical tasks,
such as initializing the per-module locks.



Deprecated since version 3.4.

@end deffn

@geindex acquire_lock() (in module imp)
@anchor{library/imp imp acquire_lock}@anchor{4059}
@deffn {Function} imp.acquire_lock ()

Acquire the interpreter’s global import lock for the current thread.
This lock should be used by import hooks to ensure thread-safety when
importing modules.

Once a thread has acquired the import lock, the same thread may acquire it
again without blocking; the thread must release it once for each time it has
acquired it.

On platforms without threads, this function does nothing.


Changed in version 3.3: The locking scheme has changed to per-module locks for
the most part.  A global import lock is kept for some critical tasks,
such as initializing the per-module locks.



Deprecated since version 3.4.

@end deffn

@geindex release_lock() (in module imp)
@anchor{library/imp imp release_lock}@anchor{405a}
@deffn {Function} imp.release_lock ()

Release the interpreter’s global import lock. On platforms without
threads, this function does nothing.


Changed in version 3.3: The locking scheme has changed to per-module locks for
the most part.  A global import lock is kept for some critical tasks,
such as initializing the per-module locks.



Deprecated since version 3.4.

@end deffn

The following constants with integer values, defined in this module, are used
to indicate the search result of @ref{404f,,find_module()}.

@geindex PY_SOURCE (in module imp)
@anchor{library/imp imp PY_SOURCE}@anchor{404c}
@deffn {Data} imp.PY_SOURCE

The module was found as a source file.


Deprecated since version 3.3.

@end deffn

@geindex PY_COMPILED (in module imp)
@anchor{library/imp imp PY_COMPILED}@anchor{404d}
@deffn {Data} imp.PY_COMPILED

The module was found as a compiled code object file.


Deprecated since version 3.3.

@end deffn

@geindex C_EXTENSION (in module imp)
@anchor{library/imp imp C_EXTENSION}@anchor{404e}
@deffn {Data} imp.C_EXTENSION

The module was found as dynamically loadable shared library.


Deprecated since version 3.3.

@end deffn

@geindex PKG_DIRECTORY (in module imp)
@anchor{library/imp imp PKG_DIRECTORY}@anchor{4052}
@deffn {Data} imp.PKG_DIRECTORY

The module was found as a package directory.


Deprecated since version 3.3.

@end deffn

@geindex C_BUILTIN (in module imp)
@anchor{library/imp imp C_BUILTIN}@anchor{4050}
@deffn {Data} imp.C_BUILTIN

The module was found as a built-in module.


Deprecated since version 3.3.

@end deffn

@geindex PY_FROZEN (in module imp)
@anchor{library/imp imp PY_FROZEN}@anchor{4051}
@deffn {Data} imp.PY_FROZEN

The module was found as a frozen module.


Deprecated since version 3.3.

@end deffn

@geindex NullImporter (class in imp)
@anchor{library/imp imp NullImporter}@anchor{c0c}
@deffn {Class} imp.NullImporter (path_string)

The @ref{c0c,,NullImporter} type is a 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} import hook that handles
non-directory path strings by failing to find any modules.  Calling this type
with an existing directory or empty string raises @ref{3b8,,ImportError}.
Otherwise, a @ref{c0c,,NullImporter} instance is returned.

Instances have only one method:

@geindex find_module() (imp.NullImporter method)
@anchor{library/imp imp NullImporter find_module}@anchor{405b}
@deffn {Method} find_module (fullname[, path])

This method always returns @code{None}, indicating that the requested module could
not be found.
@end deffn


Changed in version 3.3: @code{None} is inserted into @code{sys.path_importer_cache} instead of an
instance of @ref{c0c,,NullImporter}.



Deprecated since version 3.4: Insert @code{None} into @code{sys.path_importer_cache} instead.

@end deffn

@menu
* Examples: Examples<36>. 

@end menu

@node Examples<36>,,,imp — Access the import internals
@anchor{library/imp examples}@anchor{405c}@anchor{library/imp examples-imp}@anchor{405d}
@subsubsection Examples


The following function emulates what was the standard import statement up to
Python 1.4 (no hierarchical module names).  (This @emph{implementation} wouldn’t work
in that version, since @ref{404f,,find_module()} has been extended and
@ref{4053,,load_module()} has been added in 1.4.)

@example
import imp
import sys

def __import__(name, globals=None, locals=None, fromlist=None):
    # Fast path: see if the module has already been imported.
    try:
        return sys.modules[name]
    except KeyError:
        pass

    # If any of the following calls raises an exception,
    # there's a problem we can't handle -- let the caller handle it.

    fp, pathname, description = imp.find_module(name)

    try:
        return imp.load_module(name, fp, pathname, description)
    finally:
        # Since we may exit via an exception, close fp explicitly.
        if fp:
            fp.close()
@end example

@node Undocumented Modules,Security Considerations<3>,Superseded Modules,The Python Standard Library
@anchor{library/undoc doc}@anchor{405e}@anchor{library/undoc undoc}@anchor{405f}@anchor{library/undoc undocumented-modules}@anchor{4060}
@section Undocumented Modules


Here’s a quick listing of modules that are currently undocumented, but that
should be documented.  Feel free to contribute documentation for them!  (Send
via email to @email{docs@@python.org}.)

The idea and original contents for this chapter were taken from a posting by
Fredrik Lundh; the specific contents of this chapter have been substantially
revised.

@menu
* Platform specific modules:: 

@end menu

@node Platform specific modules,,,Undocumented Modules
@anchor{library/undoc platform-specific-modules}@anchor{4061}
@subsection Platform specific modules


These modules are used to implement the @ref{c1,,os.path} module, and are not
documented beyond this mention.  There’s little need to document these.


@table @asis

@item @code{ntpath}

— Implementation of @ref{c1,,os.path} on Win32 and Win64 platforms.

@item @code{posixpath}

— Implementation of @ref{c1,,os.path} on POSIX.
@end table
@anchor{library/security_warnings security-warnings}@anchor{4062}
@geindex security considerations

@node Security Considerations<3>,,Undocumented Modules,The Python Standard Library
@anchor{library/security_warnings doc}@anchor{4063}@anchor{library/security_warnings security-considerations}@anchor{4064}
@section Security Considerations


The following modules have specific security considerations:


@itemize *

@item 
@ref{d,,base64}: @ref{2e3f,,base64 security considerations} in
@geindex RFC; RFC 4648
RFC 4648@footnote{https://tools.ietf.org/html/rfc4648.html}

@item 
@ref{14,,cgi}: @ref{3072,,CGI security considerations}

@item 
@ref{88,,hashlib}: @ref{2340,,all constructors take a “usedforsecurity” keyword-only argument disabling known insecure and blocked algorithms}

@item 
@ref{92,,http.server} is not suitable for production use, only implementing
basic security checks

@item 
@ref{a6,,logging}: @ref{2572,,Logging configuration uses eval()}

@item 
@ref{b3,,multiprocessing}: @ref{2861,,Connection.recv() uses pickle}

@item 
@ref{c5,,pickle}: @ref{2154,,Restricting globals in pickle}

@item 
@ref{d7,,random} shouldn’t be used for security purposes, use @ref{df,,secrets}
instead

@item 
@ref{e2,,shelve}: @ref{2175,,shelve is based on pickle and thus unsuitable for dealing with untrusted sources}

@item 
@ref{ee,,ssl}: @ref{2a39,,SSL/TLS security considerations}

@item 
@ref{f4,,subprocess}: @ref{2906,,Subprocess security considerations}

@item 
@ref{fd,,tempfile}: @ref{210a,,mktemp is deprecated due to vulnerability to race conditions}

@item 
@ref{13a,,xml}: @ref{2e71,,XML vulnerabilities}

@item 
@ref{149,,zipfile}: @ref{2268,,maliciously prepared .zip files can cause disk volume exhaustion}
@end itemize

@node Extending and Embedding the Python Interpreter,Python/C API Reference Manual,The Python Standard Library,Top
@anchor{extending/index doc}@anchor{4065}@anchor{extending/index extending-and-embedding-the-python-interpreter}@anchor{4066}@anchor{extending/index extending-index}@anchor{14be}
@chapter Extending and Embedding the Python Interpreter


This document describes how to write modules in C or C++ to extend the Python
interpreter with new modules.  Those modules can not only define new functions
but also new object types and their methods.  The document also describes how
to embed the Python interpreter in another application, for use as an extension
language.  Finally, it shows how to compile and link extension modules so that
they can be loaded dynamically (at run time) into the interpreter, if the
underlying operating system supports this feature.

This document assumes basic knowledge about Python.  For an informal
introduction to the language, see @ref{14bb,,The Python Tutorial}.  @ref{14bd,,The Python Language Reference}
gives a more formal definition of the language.  @ref{14bc,,The Python Standard Library} documents
the existing object types, functions and modules (both built-in and written in
Python) that give the language its wide application range.

For a detailed description of the whole Python/C API, see the separate
@ref{14bf,,Python/C API Reference Manual}.

@menu
* Recommended third party tools:: 
* Creating extensions without third party tools:: 
* Embedding the CPython runtime in a larger application:: 

@end menu

@node Recommended third party tools,Creating extensions without third party tools,,Extending and Embedding the Python Interpreter
@anchor{extending/index recommended-third-party-tools}@anchor{4067}
@section Recommended third party tools


This guide only covers the basic tools for creating extensions provided
as part of this version of CPython. Third party tools like
Cython@footnote{http://cython.org/}, cffi@footnote{https://cffi.readthedocs.io},
SWIG@footnote{http://www.swig.org} and Numba@footnote{https://numba.pydata.org/}
offer both simpler and more sophisticated approaches to creating C and C++
extensions for Python.


@subsubheading See also



@table @asis

@item Python Packaging User Guide: Binary Extensions@footnote{https://packaging.python.org/guides/packaging-binary-extensions/}

The Python Packaging User Guide not only covers several available
tools that simplify the creation of binary extensions, but also
discusses the various reasons why creating an extension module may be
desirable in the first place.
@end table


@node Creating extensions without third party tools,Embedding the CPython runtime in a larger application,Recommended third party tools,Extending and Embedding the Python Interpreter
@anchor{extending/index creating-extensions-without-third-party-tools}@anchor{4068}
@section Creating extensions without third party tools


This section of the guide covers creating C and C++ extensions without
assistance from third party tools. It is intended primarily for creators
of those tools, rather than being a recommended way to create your own
C extensions.

@menu
* Extending Python with C or C++:: 
* Defining Extension Types; Tutorial: Defining Extension Types Tutorial. 
* Defining Extension Types; Assorted Topics: Defining Extension Types Assorted Topics. 
* Building C and C++ Extensions:: 
* Building C and C++ Extensions on Windows:: 

@end menu

@node Extending Python with C or C++,Defining Extension Types Tutorial,,Creating extensions without third party tools
@anchor{extending/extending doc}@anchor{4069}@anchor{extending/extending extending-intro}@anchor{406a}@anchor{extending/extending extending-python-with-c-or-c}@anchor{406b}
@subsection Extending Python with C or C++


It is quite easy to add new built-in modules to Python, if you know how to
program in C.  Such @emph{extension modules} can do two things that can’t be
done directly in Python: they can implement new built-in object types, and they
can call C library functions and system calls.

To support extensions, the Python API (Application Programmers Interface)
defines a set of functions, macros and variables that provide access to most
aspects of the Python run-time system.  The Python API is incorporated in a C
source file by including the header @code{"Python.h"}.

The compilation of an extension module depends on its intended use as well as on
your system setup; details are given in later chapters.

@cartouche
@quotation Note 
The C extension interface is specific to CPython, and extension modules do
not work on other Python implementations.  In many cases, it is possible to
avoid writing C extensions and preserve portability to other implementations.
For example, if your use case is calling C library functions or system calls,
you should consider using the @ref{29,,ctypes} module or the cffi@footnote{https://cffi.readthedocs.io/} library rather than writing
custom C code.
These modules let you write Python code to interface with C code and are more
portable between implementations of Python than writing and compiling a C
extension module.
@end quotation
@end cartouche

@menu
* A Simple Example:: 
* Intermezzo; Errors and Exceptions: Intermezzo Errors and Exceptions. 
* Back to the Example:: 
* The Module’s Method Table and Initialization Function:: 
* Compilation and Linkage:: 
* Calling Python Functions from C:: 
* Extracting Parameters in Extension Functions:: 
* Keyword Parameters for Extension Functions:: 
* Building Arbitrary Values:: 
* Reference Counts:: 
* Writing Extensions in C++:: 
* Providing a C API for an Extension Module:: 

@end menu

@node A Simple Example,Intermezzo Errors and Exceptions,,Extending Python with C or C++
@anchor{extending/extending a-simple-example}@anchor{406c}@anchor{extending/extending extending-simpleexample}@anchor{406d}
@subsubsection A Simple Example


Let’s create an extension module called @code{spam} (the favorite food of Monty
Python fans…) and let’s say we want to create a Python interface to the C
library function @code{system()} @footnote{@w{(1)} 
An interface for this function already exists in the standard module @ref{c0,,os}
— it was chosen as a simple and straightforward example.
}. This function takes a null-terminated
character string as argument and returns an integer.  We want this function to
be callable from Python as follows:

@example
>>> import spam
>>> status = spam.system("ls -l")
@end example

Begin by creating a file @code{spammodule.c}.  (Historically, if a module is
called @code{spam}, the C file containing its implementation is called
@code{spammodule.c}; if the module name is very long, like @code{spammify}, the
module name can be just @code{spammify.c}.)

The first two lines of our file can be:

@example
#define PY_SSIZE_T_CLEAN
#include <Python.h>
@end example

which pulls in the Python API (you can add a comment describing the purpose of
the module and a copyright notice if you like).

@cartouche
@quotation Note 
Since Python may define some pre-processor definitions which affect the standard
headers on some systems, you @emph{must} include @code{Python.h} before any standard
headers are included.

It is recommended to always define @code{PY_SSIZE_T_CLEAN} before including
@code{Python.h}.  See @ref{406e,,Extracting Parameters in Extension Functions} for a description of this macro.
@end quotation
@end cartouche

All user-visible symbols defined by @code{Python.h} have a prefix of @code{Py} or
@code{PY}, except those defined in standard header files. For convenience, and
since they are used extensively by the Python interpreter, @code{"Python.h"}
includes a few standard header files: @code{<stdio.h>}, @code{<string.h>},
@code{<errno.h>}, and @code{<stdlib.h>}.  If the latter header file does not exist on
your system, it declares the functions @code{malloc()}, @code{free()} and
@code{realloc()} directly.

The next thing we add to our module file is the C function that will be called
when the Python expression @code{spam.system(string)} is evaluated (we’ll see
shortly how it ends up being called):

@example
static PyObject *
spam_system(PyObject *self, PyObject *args)
@{
    const char *command;
    int sts;

    if (!PyArg_ParseTuple(args, "s", &command))
        return NULL;
    sts = system(command);
    return PyLong_FromLong(sts);
@}
@end example

There is a straightforward translation from the argument list in Python (for
example, the single expression @code{"ls -l"}) to the arguments passed to the C
function.  The C function always has two arguments, conventionally named @emph{self}
and @emph{args}.

The @emph{self} argument points to the module object for module-level functions;
for a method it would point to the object instance.

The @emph{args} argument will be a pointer to a Python tuple object containing the
arguments.  Each item of the tuple corresponds to an argument in the call’s
argument list.  The arguments are Python objects — in order to do anything
with them in our C function we have to convert them to C values.  The function
@ref{37f,,PyArg_ParseTuple()} in the Python API checks the argument types and
converts them to C values.  It uses a template string to determine the required
types of the arguments as well as the types of the C variables into which to
store the converted values.  More about this later.

@ref{37f,,PyArg_ParseTuple()} returns true (nonzero) if all arguments have the right
type and its components have been stored in the variables whose addresses are
passed.  It returns false (zero) if an invalid argument list was passed.  In the
latter case it also raises an appropriate exception so the calling function can
return @code{NULL} immediately (as we saw in the example).

@node Intermezzo Errors and Exceptions,Back to the Example,A Simple Example,Extending Python with C or C++
@anchor{extending/extending extending-errors}@anchor{406f}@anchor{extending/extending intermezzo-errors-and-exceptions}@anchor{4070}
@subsubsection Intermezzo: Errors and Exceptions


An important convention throughout the Python interpreter is the following: when
a function fails, it should set an exception condition and return an error value
(usually @code{-1} or a @code{NULL} pointer).  Exception information is stored in
three members of the interpreter’s thread state.  These are @code{NULL} if
there is no exception.  Otherwise they are the C equivalents of the members
of the Python tuple returned by @ref{18f,,sys.exc_info()}.  These are the
exception type, exception instance, and a traceback object.  It is important
to know about them to understand how errors are passed around.

The Python API defines a number of functions to set various types of exceptions.

The most common one is @ref{4071,,PyErr_SetString()}.  Its arguments are an exception
object and a C string.  The exception object is usually a predefined object like
@code{PyExc_ZeroDivisionError}.  The C string indicates the cause of the error
and is converted to a Python string object and stored as the “associated value”
of the exception.

Another useful function is @ref{4072,,PyErr_SetFromErrno()}, which only takes an
exception argument and constructs the associated value by inspection of the
global variable @code{errno}.  The most general function is
@ref{4073,,PyErr_SetObject()}, which takes two object arguments, the exception and
its associated value.  You don’t need to @ref{372,,Py_INCREF()} the objects passed
to any of these functions.

You can test non-destructively whether an exception has been set with
@ref{116a,,PyErr_Occurred()}.  This returns the current exception object, or @code{NULL}
if no exception has occurred.  You normally don’t need to call
@ref{116a,,PyErr_Occurred()} to see whether an error occurred in a function call,
since you should be able to tell from the return value.

When a function @emph{f} that calls another function @emph{g} detects that the latter
fails, @emph{f} should itself return an error value (usually @code{NULL} or @code{-1}).  It
should @emph{not} call one of the PyErr_* functions — one has already
been called by @emph{g}. @emph{f}’s caller is then supposed to also return an error
indication to @emph{its} caller, again @emph{without} calling PyErr_*, and so on
— the most detailed cause of the error was already reported by the function
that first detected it.  Once the error reaches the Python interpreter’s main
loop, this aborts the currently executing Python code and tries to find an
exception handler specified by the Python programmer.

(There are situations where a module can actually give a more detailed error
message by calling another PyErr_* function, and in such cases it is
fine to do so.  As a general rule, however, this is not necessary, and can cause
information about the cause of the error to be lost: most operations can fail
for a variety of reasons.)

To ignore an exception set by a function call that failed, the exception
condition must be cleared explicitly by calling @ref{bc3,,PyErr_Clear()}.  The only
time C code should call @ref{bc3,,PyErr_Clear()} is if it doesn’t want to pass the
error on to the interpreter but wants to handle it completely by itself
(possibly by trying something else, or pretending nothing went wrong).

Every failing @code{malloc()} call must be turned into an exception — the
direct caller of @code{malloc()} (or @code{realloc()}) must call
@ref{4074,,PyErr_NoMemory()} and return a failure indicator itself.  All the
object-creating functions (for example, @ref{1085,,PyLong_FromLong()}) already do
this, so this note is only relevant to those who call @code{malloc()} directly.

Also note that, with the important exception of @ref{37f,,PyArg_ParseTuple()} and
friends, functions that return an integer status usually return a positive value
or zero for success and @code{-1} for failure, like Unix system calls.

Finally, be careful to clean up garbage (by making @ref{1fd,,Py_XDECREF()} or
@ref{373,,Py_DECREF()} calls for objects you have already created) when you return
an error indicator!

The choice of which exception to raise is entirely yours.  There are predeclared
C objects corresponding to all built-in Python exceptions, such as
@code{PyExc_ZeroDivisionError}, which you can use directly. Of course, you
should choose exceptions wisely — don’t use @code{PyExc_TypeError} to mean
that a file couldn’t be opened (that should probably be @code{PyExc_IOError}).
If something’s wrong with the argument list, the @ref{37f,,PyArg_ParseTuple()}
function usually raises @code{PyExc_TypeError}.  If you have an argument whose
value must be in a particular range or must satisfy other conditions,
@code{PyExc_ValueError} is appropriate.

You can also define a new exception that is unique to your module. For this, you
usually declare a static object variable at the beginning of your file:

@example
static PyObject *SpamError;
@end example

and initialize it in your module’s initialization function (@code{PyInit_spam()})
with an exception object:

@example
PyMODINIT_FUNC
PyInit_spam(void)
@{
    PyObject *m;

    m = PyModule_Create(&spammodule);
    if (m == NULL)
        return NULL;

    SpamError = PyErr_NewException("spam.error", NULL, NULL);
    Py_XINCREF(SpamError);
    if (PyModule_AddObject(m, "error", SpamError) < 0) @{
        Py_XDECREF(SpamError);
        Py_CLEAR(SpamError);
        Py_DECREF(m);
        return NULL;
    @}

    return m;
@}
@end example

Note that the Python name for the exception object is @code{spam.error}.  The
@ref{e1a,,PyErr_NewException()} function may create a class with the base class
being @ref{4d8,,Exception} (unless another class is passed in instead of @code{NULL}),
described in @ref{10e8,,Built-in Exceptions}.

Note also that the @code{SpamError} variable retains a reference to the newly
created exception class; this is intentional!  Since the exception could be
removed from the module by external code, an owned reference to the class is
needed to ensure that it will not be discarded, causing @code{SpamError} to
become a dangling pointer. Should it become a dangling pointer, C code which
raises the exception could cause a core dump or other unintended side effects.

We discuss the use of @code{PyMODINIT_FUNC} as a function return type later in this
sample.

The @code{spam.error} exception can be raised in your extension module using a
call to @ref{4071,,PyErr_SetString()} as shown below:

@example
static PyObject *
spam_system(PyObject *self, PyObject *args)
@{
    const char *command;
    int sts;

    if (!PyArg_ParseTuple(args, "s", &command))
        return NULL;
    sts = system(command);
    if (sts < 0) @{
        PyErr_SetString(SpamError, "System command failed");
        return NULL;
    @}
    return PyLong_FromLong(sts);
@}
@end example

@node Back to the Example,The Module’s Method Table and Initialization Function,Intermezzo Errors and Exceptions,Extending Python with C or C++
@anchor{extending/extending back-to-the-example}@anchor{4075}@anchor{extending/extending backtoexample}@anchor{4076}
@subsubsection Back to the Example


Going back to our example function, you should now be able to understand this
statement:

@example
if (!PyArg_ParseTuple(args, "s", &command))
    return NULL;
@end example

It returns @code{NULL} (the error indicator for functions returning object pointers)
if an error is detected in the argument list, relying on the exception set by
@ref{37f,,PyArg_ParseTuple()}.  Otherwise the string value of the argument has been
copied to the local variable @code{command}.  This is a pointer assignment and
you are not supposed to modify the string to which it points (so in Standard C,
the variable @code{command} should properly be declared as @code{const char
*command}).

The next statement is a call to the Unix function @code{system()}, passing it
the string we just got from @ref{37f,,PyArg_ParseTuple()}:

@example
sts = system(command);
@end example

Our @code{spam.system()} function must return the value of @code{sts} as a
Python object.  This is done using the function @ref{1085,,PyLong_FromLong()}.

@example
return PyLong_FromLong(sts);
@end example

In this case, it will return an integer object.  (Yes, even integers are objects
on the heap in Python!)

If you have a C function that returns no useful argument (a function returning
void), the corresponding Python function must return @code{None}.   You
need this idiom to do so (which is implemented by the @ref{fcf,,Py_RETURN_NONE}
macro):

@example
Py_INCREF(Py_None);
return Py_None;
@end example

@ref{4077,,Py_None} is the C name for the special Python object @code{None}.  It is a
genuine Python object rather than a @code{NULL} pointer, which means “error” in most
contexts, as we have seen.

@node The Module’s Method Table and Initialization Function,Compilation and Linkage,Back to the Example,Extending Python with C or C++
@anchor{extending/extending methodtable}@anchor{4078}@anchor{extending/extending the-module-s-method-table-and-initialization-function}@anchor{4079}
@subsubsection The Module’s Method Table and Initialization Function


I promised to show how @code{spam_system()} is called from Python programs.
First, we need to list its name and address in a “method table”:

@example
static PyMethodDef SpamMethods[] = @{
    ...
    @{"system",  spam_system, METH_VARARGS,
     "Execute a shell command."@},
    ...
    @{NULL, NULL, 0, NULL@}        /* Sentinel */
@};
@end example

Note the third entry (@code{METH_VARARGS}).  This is a flag telling the interpreter
the calling convention to be used for the C function.  It should normally always
be @code{METH_VARARGS} or @code{METH_VARARGS | METH_KEYWORDS}; a value of @code{0} means
that an obsolete variant of @ref{37f,,PyArg_ParseTuple()} is used.

When using only @code{METH_VARARGS}, the function should expect the Python-level
parameters to be passed in as a tuple acceptable for parsing via
@ref{37f,,PyArg_ParseTuple()}; more information on this function is provided below.

The @code{METH_KEYWORDS} bit may be set in the third field if keyword
arguments should be passed to the function.  In this case, the C function should
accept a third @code{PyObject *} parameter which will be a dictionary of keywords.
Use @ref{863,,PyArg_ParseTupleAndKeywords()} to parse the arguments to such a
function.

The method table must be referenced in the module definition structure:

@example
static struct PyModuleDef spammodule = @{
    PyModuleDef_HEAD_INIT,
    "spam",   /* name of module */
    spam_doc, /* module documentation, may be NULL */
    -1,       /* size of per-interpreter state of the module,
                 or -1 if the module keeps state in global variables. */
    SpamMethods
@};
@end example

This structure, in turn, must be passed to the interpreter in the module’s
initialization function.  The initialization function must be named
@code{PyInit_name()}, where @emph{name} is the name of the module, and should be the
only non-@code{static} item defined in the module file:

@example
PyMODINIT_FUNC
PyInit_spam(void)
@{
    return PyModule_Create(&spammodule);
@}
@end example

Note that PyMODINIT_FUNC declares the function as @code{PyObject *} return type,
declares any special linkage declarations required by the platform, and for C++
declares the function as @code{extern "C"}.

When the Python program imports module @code{spam} for the first time,
@code{PyInit_spam()} is called. (See below for comments about embedding Python.)
It calls @ref{407a,,PyModule_Create()}, which returns a module object, and
inserts built-in function objects into the newly created module based upon the
table (an array of @ref{1014,,PyMethodDef} structures) found in the module definition.
@ref{407a,,PyModule_Create()} returns a pointer to the module object
that it creates.  It may abort with a fatal error for
certain errors, or return @code{NULL} if the module could not be initialized
satisfactorily. The init function must return the module object to its caller,
so that it then gets inserted into @code{sys.modules}.

When embedding Python, the @code{PyInit_spam()} function is not called
automatically unless there’s an entry in the @code{PyImport_Inittab} table.
To add the module to the initialization table, use @ref{112b,,PyImport_AppendInittab()},
optionally followed by an import of the module:

@example
int
main(int argc, char *argv[])
@{
    wchar_t *program = Py_DecodeLocale(argv[0], NULL);
    if (program == NULL) @{
        fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
        exit(1);
    @}

    /* Add a built-in module, before Py_Initialize */
    if (PyImport_AppendInittab("spam", PyInit_spam) == -1) @{
        fprintf(stderr, "Error: could not extend in-built modules table\n");
        exit(1);
    @}

    /* Pass argv[0] to the Python interpreter */
    Py_SetProgramName(program);

    /* Initialize the Python interpreter.  Required.
       If this step fails, it will be a fatal error. */
    Py_Initialize();

    /* Optionally import the module; alternatively,
       import can be deferred until the embedded script
       imports it. */
    PyObject *pmodule = PyImport_ImportModule("spam");
    if (!pmodule) @{
        PyErr_Print();
        fprintf(stderr, "Error: could not import module 'spam'\n");
    @}

    ...

    PyMem_RawFree(program);
    return 0;
@}
@end example

@cartouche
@quotation Note 
Removing entries from @code{sys.modules} or importing compiled modules into
multiple interpreters within a process (or following a @code{fork()} without an
intervening @code{exec()}) can create problems for some extension modules.
Extension module authors should exercise caution when initializing internal data
structures.
@end quotation
@end cartouche

A more substantial example module is included in the Python source distribution
as @code{Modules/xxmodule.c}.  This file may be used as a  template or simply
read as an example.

@cartouche
@quotation Note 
Unlike our @code{spam} example, @code{xxmodule} uses @emph{multi-phase initialization}
(new in Python 3.5), where a PyModuleDef structure is returned from
@code{PyInit_spam}, and creation of the module is left to the import machinery.
For details on multi-phase initialization, see 
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489}.
@end quotation
@end cartouche

@node Compilation and Linkage,Calling Python Functions from C,The Module’s Method Table and Initialization Function,Extending Python with C or C++
@anchor{extending/extending compilation}@anchor{407b}@anchor{extending/extending compilation-and-linkage}@anchor{407c}
@subsubsection Compilation and Linkage


There are two more things to do before you can use your new extension: compiling
and linking it with the Python system.  If you use dynamic loading, the details
may depend on the style of dynamic loading your system uses; see the chapters
about building extension modules (chapter @ref{407d,,Building C and C++ Extensions}) and additional
information that pertains only to building on Windows (chapter
@ref{16d1,,Building C and C++ Extensions on Windows}) for more information about this.

If you can’t use dynamic loading, or if you want to make your module a permanent
part of the Python interpreter, you will have to change the configuration setup
and rebuild the interpreter.  Luckily, this is very simple on Unix: just place
your file (@code{spammodule.c} for example) in the @code{Modules/} directory
of an unpacked source distribution, add a line to the file
@code{Modules/Setup.local} describing your file:

@example
spam spammodule.o
@end example

and rebuild the interpreter by running @code{make} in the toplevel
directory.  You can also run @code{make} in the @code{Modules/}
subdirectory, but then you must first rebuild @code{Makefile} there by running
‘@code{make} Makefile’.  (This is necessary each time you change the
@code{Setup} file.)

If your module requires additional libraries to link with, these can be listed
on the line in the configuration file as well, for instance:

@example
spam spammodule.o -lX11
@end example

@node Calling Python Functions from C,Extracting Parameters in Extension Functions,Compilation and Linkage,Extending Python with C or C++
@anchor{extending/extending calling-python-functions-from-c}@anchor{407e}@anchor{extending/extending callingpython}@anchor{407f}
@subsubsection Calling Python Functions from C


So far we have concentrated on making C functions callable from Python.  The
reverse is also useful: calling Python functions from C. This is especially the
case for libraries that support so-called “callback” functions.  If a C
interface makes use of callbacks, the equivalent Python often needs to provide a
callback mechanism to the Python programmer; the implementation will require
calling the Python callback functions from a C callback.  Other uses are also
imaginable.

Fortunately, the Python interpreter is easily called recursively, and there is a
standard interface to call a Python function.  (I won’t dwell on how to call the
Python parser with a particular string as input — if you’re interested, have a
look at the implementation of the @ref{1180,,-c} command line option in
@code{Modules/main.c} from the Python source code.)

Calling a Python function is easy.  First, the Python program must somehow pass
you the Python function object.  You should provide a function (or some other
interface) to do this.  When this function is called, save a pointer to the
Python function object (be careful to @ref{372,,Py_INCREF()} it!) in a global
variable — or wherever you see fit. For example, the following function might
be part of a module definition:

@example
static PyObject *my_callback = NULL;

static PyObject *
my_set_callback(PyObject *dummy, PyObject *args)
@{
    PyObject *result = NULL;
    PyObject *temp;

    if (PyArg_ParseTuple(args, "O:set_callback", &temp)) @{
        if (!PyCallable_Check(temp)) @{
            PyErr_SetString(PyExc_TypeError, "parameter must be callable");
            return NULL;
        @}
        Py_XINCREF(temp);         /* Add a reference to new callback */
        Py_XDECREF(my_callback);  /* Dispose of previous callback */
        my_callback = temp;       /* Remember new callback */
        /* Boilerplate to return "None" */
        Py_INCREF(Py_None);
        result = Py_None;
    @}
    return result;
@}
@end example

This function must be registered with the interpreter using the
@ref{103e,,METH_VARARGS} flag; this is described in section @ref{4078,,The Module’s Method Table and Initialization Function}.  The
@ref{37f,,PyArg_ParseTuple()} function and its arguments are documented in section
@ref{406e,,Extracting Parameters in Extension Functions}.

The macros @ref{57b,,Py_XINCREF()} and @ref{1fd,,Py_XDECREF()} increment/decrement the
reference count of an object and are safe in the presence of @code{NULL} pointers
(but note that @emph{temp} will not be  @code{NULL} in this context).  More info on them
in section @ref{4080,,Reference Counts}.

@geindex PyObject_CallObject()

Later, when it is time to call the function, you call the C function
@ref{4081,,PyObject_CallObject()}.  This function has two arguments, both pointers to
arbitrary Python objects: the Python function, and the argument list.  The
argument list must always be a tuple object, whose length is the number of
arguments.  To call the Python function with no arguments, pass in @code{NULL}, or
an empty tuple; to call it with one argument, pass a singleton tuple.
@ref{380,,Py_BuildValue()} returns a tuple when its format string consists of zero
or more format codes between parentheses.  For example:

@example
int arg;
PyObject *arglist;
PyObject *result;
...
arg = 123;
...
/* Time to call the callback */
arglist = Py_BuildValue("(i)", arg);
result = PyObject_CallObject(my_callback, arglist);
Py_DECREF(arglist);
@end example

@ref{4081,,PyObject_CallObject()} returns a Python object pointer: this is the return
value of the Python function.  @ref{4081,,PyObject_CallObject()} is
“reference-count-neutral” with respect to its arguments.  In the example a new
tuple was created to serve as the argument list, which is
@ref{373,,Py_DECREF()}-ed immediately after the @ref{4081,,PyObject_CallObject()}
call.

The return value of @ref{4081,,PyObject_CallObject()} is “new”: either it is a brand
new object, or it is an existing object whose reference count has been
incremented.  So, unless you want to save it in a global variable, you should
somehow @ref{373,,Py_DECREF()} the result, even (especially!) if you are not
interested in its value.

Before you do this, however, it is important to check that the return value
isn’t @code{NULL}.  If it is, the Python function terminated by raising an exception.
If the C code that called @ref{4081,,PyObject_CallObject()} is called from Python, it
should now return an error indication to its Python caller, so the interpreter
can print a stack trace, or the calling Python code can handle the exception.
If this is not possible or desirable, the exception should be cleared by calling
@ref{bc3,,PyErr_Clear()}.  For example:

@example
if (result == NULL)
    return NULL; /* Pass error back */
...use result...
Py_DECREF(result);
@end example

Depending on the desired interface to the Python callback function, you may also
have to provide an argument list to @ref{4081,,PyObject_CallObject()}.  In some cases
the argument list is also provided by the Python program, through the same
interface that specified the callback function.  It can then be saved and used
in the same manner as the function object.  In other cases, you may have to
construct a new tuple to pass as the argument list.  The simplest way to do this
is to call @ref{380,,Py_BuildValue()}.  For example, if you want to pass an integral
event code, you might use the following code:

@example
PyObject *arglist;
...
arglist = Py_BuildValue("(l)", eventcode);
result = PyObject_CallObject(my_callback, arglist);
Py_DECREF(arglist);
if (result == NULL)
    return NULL; /* Pass error back */
/* Here maybe use the result */
Py_DECREF(result);
@end example

Note the placement of @code{Py_DECREF(arglist)} immediately after the call, before
the error check!  Also note that strictly speaking this code is not complete:
@ref{380,,Py_BuildValue()} may run out of memory, and this should be checked.

You may also call a function with keyword arguments by using
@ref{46a,,PyObject_Call()}, which supports arguments and keyword arguments.  As in
the above example, we use @ref{380,,Py_BuildValue()} to construct the dictionary.

@example
PyObject *dict;
...
dict = Py_BuildValue("@{s:i@}", "name", val);
result = PyObject_Call(my_callback, NULL, dict);
Py_DECREF(dict);
if (result == NULL)
    return NULL; /* Pass error back */
/* Here maybe use the result */
Py_DECREF(result);
@end example

@node Extracting Parameters in Extension Functions,Keyword Parameters for Extension Functions,Calling Python Functions from C,Extending Python with C or C++
@anchor{extending/extending extracting-parameters-in-extension-functions}@anchor{4082}@anchor{extending/extending parsetuple}@anchor{406e}
@subsubsection Extracting Parameters in Extension Functions


@geindex PyArg_ParseTuple()

The @ref{37f,,PyArg_ParseTuple()} function is declared as follows:

@example
int PyArg_ParseTuple(PyObject *arg, const char *format, ...);
@end example

The @emph{arg} argument must be a tuple object containing an argument list passed
from Python to a C function.  The @emph{format} argument must be a format string,
whose syntax is explained in @ref{381,,Parsing arguments and building values} in the Python/C API Reference
Manual.  The remaining arguments must be addresses of variables whose type is
determined by the format string.

Note that while @ref{37f,,PyArg_ParseTuple()} checks that the Python arguments have
the required types, it cannot check the validity of the addresses of C variables
passed to the call: if you make mistakes there, your code will probably crash or
at least overwrite random bits in memory.  So be careful!

Note that any Python object references which are provided to the caller are
@emph{borrowed} references; do not decrement their reference count!

Some example calls:

@example
#define PY_SSIZE_T_CLEAN  /* Make "s#" use Py_ssize_t rather than int. */
#include <Python.h>
@end example

@example
int ok;
int i, j;
long k, l;
const char *s;
Py_ssize_t size;

ok = PyArg_ParseTuple(args, ""); /* No arguments */
    /* Python call: f() */
@end example

@example
ok = PyArg_ParseTuple(args, "s", &s); /* A string */
    /* Possible Python call: f('whoops!') */
@end example

@example
ok = PyArg_ParseTuple(args, "lls", &k, &l, &s); /* Two longs and a string */
    /* Possible Python call: f(1, 2, 'three') */
@end example

@example
ok = PyArg_ParseTuple(args, "(ii)s#", &i, &j, &s, &size);
    /* A pair of ints and a string, whose size is also returned */
    /* Possible Python call: f((1, 2), 'three') */
@end example

@example
@{
    const char *file;
    const char *mode = "r";
    int bufsize = 0;
    ok = PyArg_ParseTuple(args, "s|si", &file, &mode, &bufsize);
    /* A string, and optionally another string and an integer */
    /* Possible Python calls:
       f('spam')
       f('spam', 'w')
       f('spam', 'wb', 100000) */
@}
@end example

@example
@{
    int left, top, right, bottom, h, v;
    ok = PyArg_ParseTuple(args, "((ii)(ii))(ii)",
             &left, &top, &right, &bottom, &h, &v);
    /* A rectangle and a point */
    /* Possible Python call:
       f(((0, 0), (400, 300)), (10, 10)) */
@}
@end example

@example
@{
    Py_complex c;
    ok = PyArg_ParseTuple(args, "D:myfunction", &c);
    /* a complex, also providing a function name for errors */
    /* Possible Python call: myfunction(1+2j) */
@}
@end example

@node Keyword Parameters for Extension Functions,Building Arbitrary Values,Extracting Parameters in Extension Functions,Extending Python with C or C++
@anchor{extending/extending keyword-parameters-for-extension-functions}@anchor{4083}@anchor{extending/extending parsetupleandkeywords}@anchor{4084}
@subsubsection Keyword Parameters for Extension Functions


@geindex PyArg_ParseTupleAndKeywords()

The @ref{863,,PyArg_ParseTupleAndKeywords()} function is declared as follows:

@example
int PyArg_ParseTupleAndKeywords(PyObject *arg, PyObject *kwdict,
                                const char *format, char *kwlist[], ...);
@end example

The @emph{arg} and @emph{format} parameters are identical to those of the
@ref{37f,,PyArg_ParseTuple()} function.  The @emph{kwdict} parameter is the dictionary of
keywords received as the third parameter from the Python runtime.  The @emph{kwlist}
parameter is a @code{NULL}-terminated list of strings which identify the parameters;
the names are matched with the type information from @emph{format} from left to
right.  On success, @ref{863,,PyArg_ParseTupleAndKeywords()} returns true, otherwise
it returns false and raises an appropriate exception.

@cartouche
@quotation Note 
Nested tuples cannot be parsed when using keyword arguments!  Keyword parameters
passed in which are not present in the @emph{kwlist} will cause @ref{162,,TypeError} to
be raised.
@end quotation
@end cartouche

@geindex Philbrick@comma{} Geoff

Here is an example module which uses keywords, based on an example by Geoff
Philbrick (@email{philbrick@@hks.com}):

@example
#define PY_SSIZE_T_CLEAN  /* Make "s#" use Py_ssize_t rather than int. */
#include <Python.h>

static PyObject *
keywdarg_parrot(PyObject *self, PyObject *args, PyObject *keywds)
@{
    int voltage;
    const char *state = "a stiff";
    const char *action = "voom";
    const char *type = "Norwegian Blue";

    static char *kwlist[] = @{"voltage", "state", "action", "type", NULL@};

    if (!PyArg_ParseTupleAndKeywords(args, keywds, "i|sss", kwlist,
                                     &voltage, &state, &action, &type))
        return NULL;

    printf("-- This parrot wouldn't %s if you put %i Volts through it.\n",
           action, voltage);
    printf("-- Lovely plumage, the %s -- It's %s!\n", type, state);

    Py_RETURN_NONE;
@}

static PyMethodDef keywdarg_methods[] = @{
    /* The cast of the function is necessary since PyCFunction values
     * only take two PyObject* parameters, and keywdarg_parrot() takes
     * three.
     */
    @{"parrot", (PyCFunction)(void(*)(void))keywdarg_parrot, METH_VARARGS | METH_KEYWORDS,
     "Print a lovely skit to standard output."@},
    @{NULL, NULL, 0, NULL@}   /* sentinel */
@};

static struct PyModuleDef keywdargmodule = @{
    PyModuleDef_HEAD_INIT,
    "keywdarg",
    NULL,
    -1,
    keywdarg_methods
@};

PyMODINIT_FUNC
PyInit_keywdarg(void)
@{
    return PyModule_Create(&keywdargmodule);
@}
@end example

@node Building Arbitrary Values,Reference Counts,Keyword Parameters for Extension Functions,Extending Python with C or C++
@anchor{extending/extending building-arbitrary-values}@anchor{4085}@anchor{extending/extending buildvalue}@anchor{4086}
@subsubsection Building Arbitrary Values


This function is the counterpart to @ref{37f,,PyArg_ParseTuple()}.  It is declared
as follows:

@example
PyObject *Py_BuildValue(const char *format, ...);
@end example

It recognizes a set of format units similar to the ones recognized by
@ref{37f,,PyArg_ParseTuple()}, but the arguments (which are input to the function,
not output) must not be pointers, just values.  It returns a new Python object,
suitable for returning from a C function called from Python.

One difference with @ref{37f,,PyArg_ParseTuple()}: while the latter requires its
first argument to be a tuple (since Python argument lists are always represented
as tuples internally), @ref{380,,Py_BuildValue()} does not always build a tuple.  It
builds a tuple only if its format string contains two or more format units. If
the format string is empty, it returns @code{None}; if it contains exactly one
format unit, it returns whatever object is described by that format unit.  To
force it to return a tuple of size 0 or one, parenthesize the format string.

Examples (to the left the call, to the right the resulting Python value):

@example
Py_BuildValue("")                        None
Py_BuildValue("i", 123)                  123
Py_BuildValue("iii", 123, 456, 789)      (123, 456, 789)
Py_BuildValue("s", "hello")              'hello'
Py_BuildValue("y", "hello")              b'hello'
Py_BuildValue("ss", "hello", "world")    ('hello', 'world')
Py_BuildValue("s#", "hello", 4)          'hell'
Py_BuildValue("y#", "hello", 4)          b'hell'
Py_BuildValue("()")                      ()
Py_BuildValue("(i)", 123)                (123,)
Py_BuildValue("(ii)", 123, 456)          (123, 456)
Py_BuildValue("(i,i)", 123, 456)         (123, 456)
Py_BuildValue("[i,i]", 123, 456)         [123, 456]
Py_BuildValue("@{s:i,s:i@}",
              "abc", 123, "def", 456)    @{'abc': 123, 'def': 456@}
Py_BuildValue("((ii)(ii)) (ii)",
              1, 2, 3, 4, 5, 6)          (((1, 2), (3, 4)), (5, 6))
@end example

@node Reference Counts,Writing Extensions in C++,Building Arbitrary Values,Extending Python with C or C++
@anchor{extending/extending refcounts}@anchor{4080}@anchor{extending/extending reference-counts}@anchor{4087}
@subsubsection Reference Counts


In languages like C or C++, the programmer is responsible for dynamic allocation
and deallocation of memory on the heap.  In C, this is done using the functions
@code{malloc()} and @code{free()}.  In C++, the operators @code{new} and
@code{delete} are used with essentially the same meaning and we’ll restrict
the following discussion to the C case.

Every block of memory allocated with @code{malloc()} should eventually be
returned to the pool of available memory by exactly one call to @code{free()}.
It is important to call @code{free()} at the right time.  If a block’s address
is forgotten but @code{free()} is not called for it, the memory it occupies
cannot be reused until the program terminates.  This is called a @emph{memory
leak}.  On the other hand, if a program calls @code{free()} for a block and then
continues to use the block, it creates a conflict with re-use of the block
through another @code{malloc()} call.  This is called @emph{using freed memory}.
It has the same bad consequences as referencing uninitialized data — core
dumps, wrong results, mysterious crashes.

Common causes of memory leaks are unusual paths through the code.  For instance,
a function may allocate a block of memory, do some calculation, and then free
the block again.  Now a change in the requirements for the function may add a
test to the calculation that detects an error condition and can return
prematurely from the function.  It’s easy to forget to free the allocated memory
block when taking this premature exit, especially when it is added later to the
code.  Such leaks, once introduced, often go undetected for a long time: the
error exit is taken only in a small fraction of all calls, and most modern
machines have plenty of virtual memory, so the leak only becomes apparent in a
long-running process that uses the leaking function frequently.  Therefore, it’s
important to prevent leaks from happening by having a coding convention or
strategy that minimizes this kind of errors.

Since Python makes heavy use of @code{malloc()} and @code{free()}, it needs a
strategy to avoid memory leaks as well as the use of freed memory.  The chosen
method is called @emph{reference counting}.  The principle is simple: every
object contains a counter, which is incremented when a reference to the object
is stored somewhere, and which is decremented when a reference to it is deleted.
When the counter reaches zero, the last reference to the object has been deleted
and the object is freed.

An alternative strategy is called @emph{automatic garbage collection}.
(Sometimes, reference counting is also referred to as a garbage collection
strategy, hence my use of “automatic” to distinguish the two.)  The big
advantage of automatic garbage collection is that the user doesn’t need to call
@code{free()} explicitly.  (Another claimed advantage is an improvement in speed
or memory usage — this is no hard fact however.)  The disadvantage is that for
C, there is no truly portable automatic garbage collector, while reference
counting can be implemented portably (as long as the functions @code{malloc()}
and @code{free()} are available — which the C Standard guarantees). Maybe some
day a sufficiently portable automatic garbage collector will be available for C.
Until then, we’ll have to live with reference counts.

While Python uses the traditional reference counting implementation, it also
offers a cycle detector that works to detect reference cycles.  This allows
applications to not worry about creating direct or indirect circular references;
these are the weakness of garbage collection implemented using only reference
counting.  Reference cycles consist of objects which contain (possibly indirect)
references to themselves, so that each object in the cycle has a reference count
which is non-zero.  Typical reference counting implementations are not able to
reclaim the memory belonging to any objects in a reference cycle, or referenced
from the objects in the cycle, even though there are no further references to
the cycle itself.

The cycle detector is able to detect garbage cycles and can reclaim them.
The @ref{80,,gc} module exposes a way to run the detector (the
@ref{54b,,collect()} function), as well as configuration
interfaces and the ability to disable the detector at runtime.

@menu
* Reference Counting in Python:: 
* Ownership Rules:: 
* Thin Ice:: 
* NULL Pointers:: 

@end menu

@node Reference Counting in Python,Ownership Rules,,Reference Counts
@anchor{extending/extending refcountsinpython}@anchor{4088}@anchor{extending/extending reference-counting-in-python}@anchor{4089}
@subsubsection Reference Counting in Python


There are two macros, @code{Py_INCREF(x)} and @code{Py_DECREF(x)}, which handle the
incrementing and decrementing of the reference count. @ref{373,,Py_DECREF()} also
frees the object when the count reaches zero. For flexibility, it doesn’t call
@code{free()} directly — rather, it makes a call through a function pointer in
the object’s @emph{type object}.  For this purpose (and others), every object
also contains a pointer to its type object.

The big question now remains: when to use @code{Py_INCREF(x)} and @code{Py_DECREF(x)}?
Let’s first introduce some terms.  Nobody “owns” an object; however, you can
@emph{own a reference} to an object.  An object’s reference count is now defined
as the number of owned references to it.  The owner of a reference is
responsible for calling @ref{373,,Py_DECREF()} when the reference is no longer
needed.  Ownership of a reference can be transferred.  There are three ways to
dispose of an owned reference: pass it on, store it, or call @ref{373,,Py_DECREF()}.
Forgetting to dispose of an owned reference creates a memory leak.

It is also possible to @emph{borrow} @footnote{@w{(2)} 
The metaphor of “borrowing” a reference is not completely correct: the owner
still has a copy of the reference.
} a reference to an object.  The
borrower of a reference should not call @ref{373,,Py_DECREF()}.  The borrower must
not hold on to the object longer than the owner from which it was borrowed.
Using a borrowed reference after the owner has disposed of it risks using freed
memory and should be avoided completely @footnote{@w{(3)} 
Checking that the reference count is at least 1 @strong{does not work} — the
reference count itself could be in freed memory and may thus be reused for
another object!
}.

The advantage of borrowing over owning a reference is that you don’t need to
take care of disposing of the reference on all possible paths through the code
— in other words, with a borrowed reference you don’t run the risk of leaking
when a premature exit is taken.  The disadvantage of borrowing over owning is
that there are some subtle situations where in seemingly correct code a borrowed
reference can be used after the owner from which it was borrowed has in fact
disposed of it.

A borrowed reference can be changed into an owned reference by calling
@ref{372,,Py_INCREF()}.  This does not affect the status of the owner from which the
reference was borrowed — it creates a new owned reference, and gives full
owner responsibilities (the new owner must dispose of the reference properly, as
well as the previous owner).

@node Ownership Rules,Thin Ice,Reference Counting in Python,Reference Counts
@anchor{extending/extending ownership-rules}@anchor{408a}@anchor{extending/extending ownershiprules}@anchor{408b}
@subsubsection Ownership Rules


Whenever an object reference is passed into or out of a function, it is part of
the function’s interface specification whether ownership is transferred with the
reference or not.

Most functions that return a reference to an object pass on ownership with the
reference.  In particular, all functions whose function it is to create a new
object, such as @ref{1085,,PyLong_FromLong()} and @ref{380,,Py_BuildValue()}, pass
ownership to the receiver.  Even if the object is not actually new, you still
receive ownership of a new reference to that object.  For instance,
@ref{1085,,PyLong_FromLong()} maintains a cache of popular values and can return a
reference to a cached item.

Many functions that extract objects from other objects also transfer ownership
with the reference, for instance @ref{408c,,PyObject_GetAttrString()}.  The picture
is less clear, here, however, since a few common routines are exceptions:
@ref{408d,,PyTuple_GetItem()}, @ref{408e,,PyList_GetItem()}, @ref{384,,PyDict_GetItem()}, and
@ref{408f,,PyDict_GetItemString()} all return references that you borrow from the
tuple, list or dictionary.

The function @ref{4090,,PyImport_AddModule()} also returns a borrowed reference, even
though it may actually create the object it returns: this is possible because an
owned reference to the object is stored in @code{sys.modules}.

When you pass an object reference into another function, in general, the
function borrows the reference from you — if it needs to store it, it will use
@ref{372,,Py_INCREF()} to become an independent owner.  There are exactly two
important exceptions to this rule: @ref{4091,,PyTuple_SetItem()} and
@ref{4092,,PyList_SetItem()}.  These functions take over ownership of the item passed
to them — even if they fail!  (Note that @ref{4093,,PyDict_SetItem()} and friends
don’t take over ownership — they are “normal.”)

When a C function is called from Python, it borrows references to its arguments
from the caller.  The caller owns a reference to the object, so the borrowed
reference’s lifetime is guaranteed until the function returns.  Only when such a
borrowed reference must be stored or passed on, it must be turned into an owned
reference by calling @ref{372,,Py_INCREF()}.

The object reference returned from a C function that is called from Python must
be an owned reference — ownership is transferred from the function to its
caller.

@node Thin Ice,NULL Pointers,Ownership Rules,Reference Counts
@anchor{extending/extending thin-ice}@anchor{4094}@anchor{extending/extending thinice}@anchor{4095}
@subsubsection Thin Ice


There are a few situations where seemingly harmless use of a borrowed reference
can lead to problems.  These all have to do with implicit invocations of the
interpreter, which can cause the owner of a reference to dispose of it.

The first and most important case to know about is using @ref{373,,Py_DECREF()} on
an unrelated object while borrowing a reference to a list item.  For instance:

@example
void
bug(PyObject *list)
@{
    PyObject *item = PyList_GetItem(list, 0);

    PyList_SetItem(list, 1, PyLong_FromLong(0L));
    PyObject_Print(item, stdout, 0); /* BUG! */
@}
@end example

This function first borrows a reference to @code{list[0]}, then replaces
@code{list[1]} with the value @code{0}, and finally prints the borrowed reference.
Looks harmless, right?  But it’s not!

Let’s follow the control flow into @ref{4092,,PyList_SetItem()}.  The list owns
references to all its items, so when item 1 is replaced, it has to dispose of
the original item 1.  Now let’s suppose the original item 1 was an instance of a
user-defined class, and let’s further suppose that the class defined a
@code{__del__()} method.  If this class instance has a reference count of 1,
disposing of it will call its @code{__del__()} method.

Since it is written in Python, the @code{__del__()} method can execute arbitrary
Python code.  Could it perhaps do something to invalidate the reference to
@code{item} in @code{bug()}?  You bet!  Assuming that the list passed into
@code{bug()} is accessible to the @code{__del__()} method, it could execute a
statement to the effect of @code{del list[0]}, and assuming this was the last
reference to that object, it would free the memory associated with it, thereby
invalidating @code{item}.

The solution, once you know the source of the problem, is easy: temporarily
increment the reference count.  The correct version of the function reads:

@example
void
no_bug(PyObject *list)
@{
    PyObject *item = PyList_GetItem(list, 0);

    Py_INCREF(item);
    PyList_SetItem(list, 1, PyLong_FromLong(0L));
    PyObject_Print(item, stdout, 0);
    Py_DECREF(item);
@}
@end example

This is a true story.  An older version of Python contained variants of this bug
and someone spent a considerable amount of time in a C debugger to figure out
why his @code{__del__()} methods would fail…

The second case of problems with a borrowed reference is a variant involving
threads.  Normally, multiple threads in the Python interpreter can’t get in each
other’s way, because there is a global lock protecting Python’s entire object
space.  However, it is possible to temporarily release this lock using the macro
@ref{4096,,Py_BEGIN_ALLOW_THREADS}, and to re-acquire it using
@ref{5aa,,Py_END_ALLOW_THREADS}.  This is common around blocking I/O calls, to
let other threads use the processor while waiting for the I/O to complete.
Obviously, the following function has the same problem as the previous one:

@example
void
bug(PyObject *list)
@{
    PyObject *item = PyList_GetItem(list, 0);
    Py_BEGIN_ALLOW_THREADS
    ...some blocking I/O call...
    Py_END_ALLOW_THREADS
    PyObject_Print(item, stdout, 0); /* BUG! */
@}
@end example

@node NULL Pointers,,Thin Ice,Reference Counts
@anchor{extending/extending null-pointers}@anchor{4097}@anchor{extending/extending nullpointers}@anchor{4098}
@subsubsection NULL Pointers


In general, functions that take object references as arguments do not expect you
to pass them @code{NULL} pointers, and will dump core (or cause later core dumps) if
you do so.  Functions that return object references generally return @code{NULL} only
to indicate that an exception occurred.  The reason for not testing for @code{NULL}
arguments is that functions often pass the objects they receive on to other
function — if each function were to test for @code{NULL}, there would be a lot of
redundant tests and the code would run more slowly.

It is better to test for @code{NULL} only at the “source:” when a pointer that may be
@code{NULL} is received, for example, from @code{malloc()} or from a function that
may raise an exception.

The macros @ref{372,,Py_INCREF()} and @ref{373,,Py_DECREF()} do not check for @code{NULL}
pointers — however, their variants @ref{57b,,Py_XINCREF()} and @ref{1fd,,Py_XDECREF()}
do.

The macros for checking for a particular object type (@code{Pytype_Check()}) don’t
check for @code{NULL} pointers — again, there is much code that calls several of
these in a row to test an object against various different expected types, and
this would generate redundant tests.  There are no variants with @code{NULL}
checking.

The C function calling mechanism guarantees that the argument list passed to C
functions (@code{args} in the examples) is never @code{NULL} — in fact it guarantees
that it is always a tuple @footnote{@w{(4)} 
These guarantees don’t hold when you use the “old” style calling convention —
this is still found in much existing code.
}.

It is a severe error to ever let a @code{NULL} pointer “escape” to the Python user.

@c Frank Stajano:
@c A pedagogically buggy example, along the lines of the previous listing, would
@c be helpful here -- showing in more concrete terms what sort of actions could
@c cause the problem. I can't very well imagine it from the description.

@node Writing Extensions in C++,Providing a C API for an Extension Module,Reference Counts,Extending Python with C or C++
@anchor{extending/extending cplusplus}@anchor{4099}@anchor{extending/extending writing-extensions-in-c}@anchor{409a}
@subsubsection Writing Extensions in C++


It is possible to write extension modules in C++.  Some restrictions apply.  If
the main program (the Python interpreter) is compiled and linked by the C
compiler, global or static objects with constructors cannot be used.  This is
not a problem if the main program is linked by the C++ compiler.  Functions that
will be called by the Python interpreter (in particular, module initialization
functions) have to be declared using @code{extern "C"}. It is unnecessary to
enclose the Python header files in @code{extern "C" @{...@}} — they use this form
already if the symbol @code{__cplusplus} is defined (all recent C++ compilers
define this symbol).

@node Providing a C API for an Extension Module,,Writing Extensions in C++,Extending Python with C or C++
@anchor{extending/extending providing-a-c-api-for-an-extension-module}@anchor{409b}@anchor{extending/extending using-capsules}@anchor{ef1}
@subsubsection Providing a C API for an Extension Module


Many extension modules just provide new functions and types to be used from
Python, but sometimes the code in an extension module can be useful for other
extension modules. For example, an extension module could implement a type
“collection” which works like lists without order. Just like the standard Python
list type has a C API which permits extension modules to create and manipulate
lists, this new collection type should have a set of C functions for direct
manipulation from other extension modules.

At first sight this seems easy: just write the functions (without declaring them
@code{static}, of course), provide an appropriate header file, and document
the C API. And in fact this would work if all extension modules were always
linked statically with the Python interpreter. When modules are used as shared
libraries, however, the symbols defined in one module may not be visible to
another module. The details of visibility depend on the operating system; some
systems use one global namespace for the Python interpreter and all extension
modules (Windows, for example), whereas others require an explicit list of
imported symbols at module link time (AIX is one example), or offer a choice of
different strategies (most Unices). And even if symbols are globally visible,
the module whose functions one wishes to call might not have been loaded yet!

Portability therefore requires not to make any assumptions about symbol
visibility. This means that all symbols in extension modules should be declared
@code{static}, except for the module’s initialization function, in order to
avoid name clashes with other extension modules (as discussed in section
@ref{4078,,The Module’s Method Table and Initialization Function}). And it means that symbols that @emph{should} be accessible from
other extension modules must be exported in a different way.

Python provides a special mechanism to pass C-level information (pointers) from
one extension module to another one: Capsules. A Capsule is a Python data type
which stores a pointer (void*).  Capsules can only be created and
accessed via their C API, but they can be passed around like any other Python
object. In particular,  they can be assigned to a name in an extension module’s
namespace. Other extension modules can then import this module, retrieve the
value of this name, and then retrieve the pointer from the Capsule.

There are many ways in which Capsules can be used to export the C API of an
extension module. Each function could get its own Capsule, or all C API pointers
could be stored in an array whose address is published in a Capsule. And the
various tasks of storing and retrieving the pointers can be distributed in
different ways between the module providing the code and the client modules.

Whichever method you choose, it’s important to name your Capsules properly.
The function @ref{409c,,PyCapsule_New()} takes a name parameter
(const  char*); you’re permitted to pass in a @code{NULL} name, but
we strongly encourage you to specify a name.  Properly named Capsules provide
a degree of runtime type-safety; there is no feasible way to tell one unnamed
Capsule from another.

In particular, Capsules used to expose C APIs should be given a name following
this convention:

@example
modulename.attributename
@end example

The convenience function @ref{10c9,,PyCapsule_Import()} makes it easy to
load a C API provided via a Capsule, but only if the Capsule’s name
matches this convention.  This behavior gives C API users a high degree
of certainty that the Capsule they load contains the correct C API.

The following example demonstrates an approach that puts most of the burden on
the writer of the exporting module, which is appropriate for commonly used
library modules. It stores all C API pointers (just one in the example!) in an
array of void pointers which becomes the value of a Capsule. The header
file corresponding to the module provides a macro that takes care of importing
the module and retrieving its C API pointers; client modules only have to call
this macro before accessing the C API.

The exporting module is a modification of the @code{spam} module from section
@ref{406d,,A Simple Example}. The function @code{spam.system()} does not call
the C library function @code{system()} directly, but a function
@code{PySpam_System()}, which would of course do something more complicated in
reality (such as adding “spam” to every command). This function
@code{PySpam_System()} is also exported to other extension modules.

The function @code{PySpam_System()} is a plain C function, declared
@code{static} like everything else:

@example
static int
PySpam_System(const char *command)
@{
    return system(command);
@}
@end example

The function @code{spam_system()} is modified in a trivial way:

@example
static PyObject *
spam_system(PyObject *self, PyObject *args)
@{
    const char *command;
    int sts;

    if (!PyArg_ParseTuple(args, "s", &command))
        return NULL;
    sts = PySpam_System(command);
    return PyLong_FromLong(sts);
@}
@end example

In the beginning of the module, right after the line

@example
#include <Python.h>
@end example

two more lines must be added:

@example
#define SPAM_MODULE
#include "spammodule.h"
@end example

The @code{#define} is used to tell the header file that it is being included in the
exporting module, not a client module. Finally, the module’s initialization
function must take care of initializing the C API pointer array:

@example
PyMODINIT_FUNC
PyInit_spam(void)
@{
    PyObject *m;
    static void *PySpam_API[PySpam_API_pointers];
    PyObject *c_api_object;

    m = PyModule_Create(&spammodule);
    if (m == NULL)
        return NULL;

    /* Initialize the C API pointer array */
    PySpam_API[PySpam_System_NUM] = (void *)PySpam_System;

    /* Create a Capsule containing the API pointer array's address */
    c_api_object = PyCapsule_New((void *)PySpam_API, "spam._C_API", NULL);

    if (PyModule_AddObject(m, "_C_API", c_api_object) < 0) @{
        Py_XDECREF(c_api_object);
        Py_DECREF(m);
        return NULL;
    @}

    return m;
@}
@end example

Note that @code{PySpam_API} is declared @code{static}; otherwise the pointer
array would disappear when @code{PyInit_spam()} terminates!

The bulk of the work is in the header file @code{spammodule.h}, which looks
like this:

@example
#ifndef Py_SPAMMODULE_H
#define Py_SPAMMODULE_H
#ifdef __cplusplus
extern "C" @{
#endif

/* Header file for spammodule */

/* C API functions */
#define PySpam_System_NUM 0
#define PySpam_System_RETURN int
#define PySpam_System_PROTO (const char *command)

/* Total number of C API pointers */
#define PySpam_API_pointers 1


#ifdef SPAM_MODULE
/* This section is used when compiling spammodule.c */

static PySpam_System_RETURN PySpam_System PySpam_System_PROTO;

#else
/* This section is used in modules that use spammodule's API */

static void **PySpam_API;

#define PySpam_System \
 (*(PySpam_System_RETURN (*)PySpam_System_PROTO) PySpam_API[PySpam_System_NUM])

/* Return -1 on error, 0 on success.
 * PyCapsule_Import will set an exception if there's an error.
 */
static int
import_spam(void)
@{
    PySpam_API = (void **)PyCapsule_Import("spam._C_API", 0);
    return (PySpam_API != NULL) ? 0 : -1;
@}

#endif

#ifdef __cplusplus
@}
#endif

#endif /* !defined(Py_SPAMMODULE_H) */
@end example

All that a client module must do in order to have access to the function
@code{PySpam_System()} is to call the function (or rather macro)
@code{import_spam()} in its initialization function:

@example
PyMODINIT_FUNC
PyInit_client(void)
@{
    PyObject *m;

    m = PyModule_Create(&clientmodule);
    if (m == NULL)
        return NULL;
    if (import_spam() < 0)
        return NULL;
    /* additional initialization can happen here */
    return m;
@}
@end example

The main disadvantage of this approach is that the file @code{spammodule.h} is
rather complicated. However, the basic structure is the same for each function
that is exported, so it has to be learned only once.

Finally it should be mentioned that Capsules offer additional functionality,
which is especially useful for memory allocation and deallocation of the pointer
stored in a Capsule. The details are described in the Python/C API Reference
Manual in the section @ref{409d,,Capsules} and in the implementation of Capsules (files
@code{Include/pycapsule.h} and @code{Objects/pycapsule.c} in the Python source
code distribution).

@node Defining Extension Types Tutorial,Defining Extension Types Assorted Topics,Extending Python with C or C++,Creating extensions without third party tools
@anchor{extending/newtypes_tutorial doc}@anchor{409e}@anchor{extending/newtypes_tutorial defining-extension-types-tutorial}@anchor{409f}@anchor{extending/newtypes_tutorial defining-new-types}@anchor{40a0}
@subsection Defining Extension Types: Tutorial


Python allows the writer of a C extension module to define new types that
can be manipulated from Python code, much like the built-in @ref{2ef,,str}
and @ref{43a,,list} types.  The code for all extension types follows a
pattern, but there are some details that you need to understand before you
can get started.  This document is a gentle introduction to the topic.

@menu
* The Basics:: 
* Adding data and methods to the Basic example:: 
* Providing finer control over data attributes:: 
* Supporting cyclic garbage collection:: 
* Subclassing other types:: 

@end menu

@node The Basics,Adding data and methods to the Basic example,,Defining Extension Types Tutorial
@anchor{extending/newtypes_tutorial dnt-basics}@anchor{40a1}@anchor{extending/newtypes_tutorial the-basics}@anchor{40a2}
@subsubsection The Basics


The @ref{17a4,,CPython} runtime sees all Python objects as variables of type
@ref{375,,PyObject}*, which serves as a “base type” for all Python objects.
The @ref{375,,PyObject} structure itself only contains the object’s
@ref{40a3,,reference count} and a pointer to the object’s “type object”.
This is where the action is; the type object determines which (C) functions
get called by the interpreter when, for instance, an attribute gets looked up
on an object, a method called, or it is multiplied by another object.  These
C functions are called “type methods”.

So, if you want to define a new extension type, you need to create a new type
object.

This sort of thing can only be explained by example, so here’s a minimal, but
complete, module that defines a new type named @code{Custom} inside a C
extension module @code{custom}:

@cartouche
@quotation Note 
What we’re showing here is the traditional way of defining @emph{static}
extension types.  It should be adequate for most uses.  The C API also
allows defining heap-allocated extension types using the
@ref{469,,PyType_FromSpec()} function, which isn’t covered in this tutorial.
@end quotation
@end cartouche

@example
#define PY_SSIZE_T_CLEAN
#include <Python.h>

typedef struct @{
    PyObject_HEAD
    /* Type-specific fields go here. */
@} CustomObject;

static PyTypeObject CustomType = @{
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "custom.Custom",
    .tp_doc = "Custom objects",
    .tp_basicsize = sizeof(CustomObject),
    .tp_itemsize = 0,
    .tp_flags = Py_TPFLAGS_DEFAULT,
    .tp_new = PyType_GenericNew,
@};

static PyModuleDef custommodule = @{
    PyModuleDef_HEAD_INIT,
    .m_name = "custom",
    .m_doc = "Example module that creates an extension type.",
    .m_size = -1,
@};

PyMODINIT_FUNC
PyInit_custom(void)
@{
    PyObject *m;
    if (PyType_Ready(&CustomType) < 0)
        return NULL;

    m = PyModule_Create(&custommodule);
    if (m == NULL)
        return NULL;

    Py_INCREF(&CustomType);
    if (PyModule_AddObject(m, "Custom", (PyObject *) &CustomType) < 0) @{
        Py_DECREF(&CustomType);
        Py_DECREF(m);
        return NULL;
    @}

    return m;
@}
@end example

Now that’s quite a bit to take in at once, but hopefully bits will seem familiar
from the previous chapter.  This file defines three things:


@enumerate 

@item 
What a @code{Custom} @strong{object} contains: this is the @code{CustomObject}
struct, which is allocated once for each @code{Custom} instance.

@item 
How the @code{Custom} @strong{type} behaves: this is the @code{CustomType} struct,
which defines a set of flags and function pointers that the interpreter
inspects when specific operations are requested.

@item 
How to initialize the @code{custom} module: this is the @code{PyInit_custom}
function and the associated @code{custommodule} struct.
@end enumerate

The first bit is:

@example
typedef struct @{
    PyObject_HEAD
@} CustomObject;
@end example

This is what a Custom object will contain.  @code{PyObject_HEAD} is mandatory
at the start of each object struct and defines a field called @code{ob_base}
of type @ref{375,,PyObject}, containing a pointer to a type object and a
reference count (these can be accessed using the macros @ref{382,,Py_REFCNT}
and @ref{1f1,,Py_TYPE} respectively).  The reason for the macro is to
abstract away the layout and to enable additional fields in @ref{338,,debug builds}.

@cartouche
@quotation Note 
There is no semicolon above after the @ref{e7a,,PyObject_HEAD} macro.
Be wary of adding one by accident: some compilers will complain.
@end quotation
@end cartouche

Of course, objects generally store additional data besides the standard
@code{PyObject_HEAD} boilerplate; for example, here is the definition for
standard Python floats:

@example
typedef struct @{
    PyObject_HEAD
    double ob_fval;
@} PyFloatObject;
@end example

The second bit is the definition of the type object.

@example
static PyTypeObject CustomType = @{
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "custom.Custom",
    .tp_doc = "Custom objects",
    .tp_basicsize = sizeof(CustomObject),
    .tp_itemsize = 0,
    .tp_flags = Py_TPFLAGS_DEFAULT,
    .tp_new = PyType_GenericNew,
@};
@end example

@cartouche
@quotation Note 
We recommend using C99-style designated initializers as above, to
avoid listing all the @ref{5c6,,PyTypeObject} fields that you don’t care
about and also to avoid caring about the fields’ declaration order.
@end quotation
@end cartouche

The actual definition of @ref{5c6,,PyTypeObject} in @code{object.h} has
many more @ref{499,,fields} than the definition above.  The
remaining fields will be filled with zeros by the C compiler, and it’s
common practice to not specify them explicitly unless you need them.

We’re going to pick it apart, one field at a time:

@example
PyVarObject_HEAD_INIT(NULL, 0)
@end example

This line is mandatory boilerplate to initialize the @code{ob_base}
field mentioned above.

@example
.tp_name = "custom.Custom",
@end example

The name of our type.  This will appear in the default textual representation of
our objects and in some error messages, for example:

@example
>>> "" + custom.Custom()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can only concatenate str (not "custom.Custom") to str
@end example

Note that the name is a dotted name that includes both the module name and the
name of the type within the module. The module in this case is @code{custom} and
the type is @code{Custom}, so we set the type name to @code{custom.Custom}.
Using the real dotted import path is important to make your type compatible
with the @ref{d4,,pydoc} and @ref{c5,,pickle} modules.

@example
.tp_basicsize = sizeof(CustomObject),
.tp_itemsize = 0,
@end example

This is so that Python knows how much memory to allocate when creating
new @code{Custom} instances.  @ref{40a4,,tp_itemsize} is
only used for variable-sized objects and should otherwise be zero.

@cartouche
@quotation Note 
If you want your type to be subclassable from Python, and your type has the same
@ref{493,,tp_basicsize} as its base type, you may have problems with multiple
inheritance.  A Python subclass of your type will have to list your type first
in its @ref{ffe,,__bases__}, or else it will not be able to call your type’s
@code{__new__()} method without getting an error.  You can avoid this problem by
ensuring that your type has a larger value for @ref{493,,tp_basicsize} than its
base type does.  Most of the time, this will be true anyway, because either your
base type will be @ref{5a5,,object}, or else you will be adding data members to
your base type, and therefore increasing its size.
@end quotation
@end cartouche

We set the class flags to @ref{40a5,,Py_TPFLAGS_DEFAULT}.

@example
.tp_flags = Py_TPFLAGS_DEFAULT,
@end example

All types should include this constant in their flags.  It enables all of the
members defined until at least Python 3.3.  If you need further members,
you will need to OR the corresponding flags.

We provide a doc string for the type in @ref{40a6,,tp_doc}.

@example
.tp_doc = "Custom objects",
@end example

To enable object creation, we have to provide a @ref{40a7,,tp_new}
handler.  This is the equivalent of the Python method @code{__new__()}, but
has to be specified explicitly.  In this case, we can just use the default
implementation provided by the API function @ref{40a8,,PyType_GenericNew()}.

@example
.tp_new = PyType_GenericNew,
@end example

Everything else in the file should be familiar, except for some code in
@code{PyInit_custom()}:

@example
if (PyType_Ready(&CustomType) < 0)
    return;
@end example

This initializes the @code{Custom} type, filling in a number of members
to the appropriate default values, including @code{ob_type} that we initially
set to @code{NULL}.

@example
Py_INCREF(&CustomType);
if (PyModule_AddObject(m, "Custom", (PyObject *) &CustomType) < 0) @{
    Py_DECREF(&CustomType);
    Py_DECREF(m);
    return NULL;
@}
@end example

This adds the type to the module dictionary.  This allows us to create
@code{Custom} instances by calling the @code{Custom} class:

@example
>>> import custom
>>> mycustom = custom.Custom()
@end example

That’s it!  All that remains is to build it; put the above code in a file called
@code{custom.c} and:

@example
from distutils.core import setup, Extension
setup(name="custom", version="1.0",
      ext_modules=[Extension("custom", ["custom.c"])])
@end example

in a file called @code{setup.py}; then typing

@example
$ python setup.py build
@end example

at a shell should produce a file @code{custom.so} in a subdirectory; move to
that directory and fire up Python — you should be able to @code{import custom} and
play around with Custom objects.

That wasn’t so hard, was it?

Of course, the current Custom type is pretty uninteresting. It has no data and
doesn’t do anything. It can’t even be subclassed.

@cartouche
@quotation Note 
While this documentation showcases the standard @ref{37,,distutils} module
for building C extensions, it is recommended in real-world use cases to
use the newer and better-maintained @code{setuptools} library.  Documentation
on how to do this is out of scope for this document and can be found in
the Python Packaging User's Guide@footnote{https://packaging.python.org/tutorials/distributing-packages/}.
@end quotation
@end cartouche

@node Adding data and methods to the Basic example,Providing finer control over data attributes,The Basics,Defining Extension Types Tutorial
@anchor{extending/newtypes_tutorial adding-data-and-methods-to-the-basic-example}@anchor{40a9}
@subsubsection Adding data and methods to the Basic example


Let’s extend the basic example to add some data and methods.  Let’s also make
the type usable as a base class. We’ll create a new module, @code{custom2} that
adds these capabilities:

@example
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "structmember.h"

typedef struct @{
    PyObject_HEAD
    PyObject *first; /* first name */
    PyObject *last;  /* last name */
    int number;
@} CustomObject;

static void
Custom_dealloc(CustomObject *self)
@{
    Py_XDECREF(self->first);
    Py_XDECREF(self->last);
    Py_TYPE(self)->tp_free((PyObject *) self);
@}

static PyObject *
Custom_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
@{
    CustomObject *self;
    self = (CustomObject *) type->tp_alloc(type, 0);
    if (self != NULL) @{
        self->first = PyUnicode_FromString("");
        if (self->first == NULL) @{
            Py_DECREF(self);
            return NULL;
        @}
        self->last = PyUnicode_FromString("");
        if (self->last == NULL) @{
            Py_DECREF(self);
            return NULL;
        @}
        self->number = 0;
    @}
    return (PyObject *) self;
@}

static int
Custom_init(CustomObject *self, PyObject *args, PyObject *kwds)
@{
    static char *kwlist[] = @{"first", "last", "number", NULL@};
    PyObject *first = NULL, *last = NULL, *tmp;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", kwlist,
                                     &first, &last,
                                     &self->number))
        return -1;

    if (first) @{
        tmp = self->first;
        Py_INCREF(first);
        self->first = first;
        Py_XDECREF(tmp);
    @}
    if (last) @{
        tmp = self->last;
        Py_INCREF(last);
        self->last = last;
        Py_XDECREF(tmp);
    @}
    return 0;
@}

static PyMemberDef Custom_members[] = @{
    @{"first", T_OBJECT_EX, offsetof(CustomObject, first), 0,
     "first name"@},
    @{"last", T_OBJECT_EX, offsetof(CustomObject, last), 0,
     "last name"@},
    @{"number", T_INT, offsetof(CustomObject, number), 0,
     "custom number"@},
    @{NULL@}  /* Sentinel */
@};

static PyObject *
Custom_name(CustomObject *self, PyObject *Py_UNUSED(ignored))
@{
    if (self->first == NULL) @{
        PyErr_SetString(PyExc_AttributeError, "first");
        return NULL;
    @}
    if (self->last == NULL) @{
        PyErr_SetString(PyExc_AttributeError, "last");
        return NULL;
    @}
    return PyUnicode_FromFormat("%S %S", self->first, self->last);
@}

static PyMethodDef Custom_methods[] = @{
    @{"name", (PyCFunction) Custom_name, METH_NOARGS,
     "Return the name, combining the first and last name"
    @},
    @{NULL@}  /* Sentinel */
@};

static PyTypeObject CustomType = @{
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "custom2.Custom",
    .tp_doc = "Custom objects",
    .tp_basicsize = sizeof(CustomObject),
    .tp_itemsize = 0,
    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
    .tp_new = Custom_new,
    .tp_init = (initproc) Custom_init,
    .tp_dealloc = (destructor) Custom_dealloc,
    .tp_members = Custom_members,
    .tp_methods = Custom_methods,
@};

static PyModuleDef custommodule = @{
    PyModuleDef_HEAD_INIT,
    .m_name = "custom2",
    .m_doc = "Example module that creates an extension type.",
    .m_size = -1,
@};

PyMODINIT_FUNC
PyInit_custom2(void)
@{
    PyObject *m;
    if (PyType_Ready(&CustomType) < 0)
        return NULL;

    m = PyModule_Create(&custommodule);
    if (m == NULL)
        return NULL;

    Py_INCREF(&CustomType);
    if (PyModule_AddObject(m, "Custom", (PyObject *) &CustomType) < 0) @{
        Py_DECREF(&CustomType);
        Py_DECREF(m);
        return NULL;
    @}

    return m;
@}
@end example

This version of the module has a number of changes.

We’ve added an extra include:

@example
#include <structmember.h>
@end example

This include provides declarations that we use to handle attributes, as
described a bit later.

The  @code{Custom} type now has three data attributes in its C struct,
@emph{first}, @emph{last}, and @emph{number}.  The @emph{first} and @emph{last} variables are Python
strings containing first and last names.  The @emph{number} attribute is a C integer.

The object structure is updated accordingly:

@example
typedef struct @{
    PyObject_HEAD
    PyObject *first; /* first name */
    PyObject *last;  /* last name */
    int number;
@} CustomObject;
@end example

Because we now have data to manage, we have to be more careful about object
allocation and deallocation.  At a minimum, we need a deallocation method:

@example
static void
Custom_dealloc(CustomObject *self)
@{
    Py_XDECREF(self->first);
    Py_XDECREF(self->last);
    Py_TYPE(self)->tp_free((PyObject *) self);
@}
@end example

which is assigned to the @ref{40aa,,tp_dealloc} member:

@example
.tp_dealloc = (destructor) Custom_dealloc,
@end example

This method first clears the reference counts of the two Python attributes.
@ref{1fd,,Py_XDECREF()} correctly handles the case where its argument is
@code{NULL} (which might happen here if @code{tp_new} failed midway).  It then
calls the @ref{40ab,,tp_free} member of the object’s type
(computed by @code{Py_TYPE(self)}) to free the object’s memory.  Note that
the object’s type might not be @code{CustomType}, because the object may
be an instance of a subclass.

@cartouche
@quotation Note 
The explicit cast to @code{destructor} above is needed because we defined
@code{Custom_dealloc} to take a @code{CustomObject *} argument, but the @code{tp_dealloc}
function pointer expects to receive a @code{PyObject *} argument.  Otherwise,
the compiler will emit a warning.  This is object-oriented polymorphism,
in C!
@end quotation
@end cartouche

We want to make sure that the first and last names are initialized to empty
strings, so we provide a @code{tp_new} implementation:

@example
static PyObject *
Custom_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
@{
    CustomObject *self;
    self = (CustomObject *) type->tp_alloc(type, 0);
    if (self != NULL) @{
        self->first = PyUnicode_FromString("");
        if (self->first == NULL) @{
            Py_DECREF(self);
            return NULL;
        @}
        self->last = PyUnicode_FromString("");
        if (self->last == NULL) @{
            Py_DECREF(self);
            return NULL;
        @}
        self->number = 0;
    @}
    return (PyObject *) self;
@}
@end example

and install it in the @ref{40a7,,tp_new} member:

@example
.tp_new = Custom_new,
@end example

The @code{tp_new} handler is responsible for creating (as opposed to initializing)
objects of the type.  It is exposed in Python as the @code{__new__()} method.
It is not required to define a @code{tp_new} member, and indeed many extension
types will simply reuse @ref{40a8,,PyType_GenericNew()} as done in the first
version of the @code{Custom} type above.  In this case, we use the @code{tp_new}
handler to initialize the @code{first} and @code{last} attributes to non-@code{NULL}
default values.

@code{tp_new} is passed the type being instantiated (not necessarily @code{CustomType},
if a subclass is instantiated) and any arguments passed when the type was
called, and is expected to return the instance created.  @code{tp_new} handlers
always accept positional and keyword arguments, but they often ignore the
arguments, leaving the argument handling to initializer (a.k.a. @code{tp_init}
in C or @code{__init__} in Python) methods.

@cartouche
@quotation Note 
@code{tp_new} shouldn’t call @code{tp_init} explicitly, as the interpreter
will do it itself.
@end quotation
@end cartouche

The @code{tp_new} implementation calls the @ref{40ac,,tp_alloc}
slot to allocate memory:

@example
self = (CustomObject *) type->tp_alloc(type, 0);
@end example

Since memory allocation may fail, we must check the @ref{40ac,,tp_alloc}
result against @code{NULL} before proceeding.

@cartouche
@quotation Note 
We didn’t fill the @ref{40ac,,tp_alloc} slot ourselves. Rather
@ref{1ec,,PyType_Ready()} fills it for us by inheriting it from our base class,
which is @ref{5a5,,object} by default.  Most types use the default allocation
strategy.
@end quotation
@end cartouche

@cartouche
@quotation Note 
If you are creating a co-operative @ref{40a7,,tp_new} (one
that calls a base type’s @ref{40a7,,tp_new} or @code{__new__()}),
you must @emph{not} try to determine what method to call using method resolution
order at runtime.  Always statically determine what type you are going to
call, and call its @ref{40a7,,tp_new} directly, or via
@code{type->tp_base->tp_new}.  If you do not do this, Python subclasses of your
type that also inherit from other Python-defined classes may not work correctly.
(Specifically, you may not be able to create instances of such subclasses
without getting a @ref{162,,TypeError}.)
@end quotation
@end cartouche

We also define an initialization function which accepts arguments to provide
initial values for our instance:

@example
static int
Custom_init(CustomObject *self, PyObject *args, PyObject *kwds)
@{
    static char *kwlist[] = @{"first", "last", "number", NULL@};
    PyObject *first = NULL, *last = NULL, *tmp;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", kwlist,
                                     &first, &last,
                                     &self->number))
        return -1;

    if (first) @{
        tmp = self->first;
        Py_INCREF(first);
        self->first = first;
        Py_XDECREF(tmp);
    @}
    if (last) @{
        tmp = self->last;
        Py_INCREF(last);
        self->last = last;
        Py_XDECREF(tmp);
    @}
    return 0;
@}
@end example

by filling the @ref{40ad,,tp_init} slot.

@example
.tp_init = (initproc) Custom_init,
@end example

The @ref{40ad,,tp_init} slot is exposed in Python as the
@code{__init__()} method.  It is used to initialize an object after it’s
created.  Initializers always accept positional and keyword arguments,
and they should return either @code{0} on success or @code{-1} on error.

Unlike the @code{tp_new} handler, there is no guarantee that @code{tp_init}
is called at all (for example, the @ref{c5,,pickle} module by default
doesn’t call @code{__init__()} on unpickled instances).  It can also be
called multiple times.  Anyone can call the @code{__init__()} method on
our objects.  For this reason, we have to be extra careful when assigning
the new attribute values.  We might be tempted, for example to assign the
@code{first} member like this:

@example
if (first) @{
    Py_XDECREF(self->first);
    Py_INCREF(first);
    self->first = first;
@}
@end example

But this would be risky.  Our type doesn’t restrict the type of the
@code{first} member, so it could be any kind of object.  It could have a
destructor that causes code to be executed that tries to access the
@code{first} member; or that destructor could release the
@ref{385,,Global interpreter Lock} and let arbitrary code run in other
threads that accesses and modifies our object.

To be paranoid and protect ourselves against this possibility, we almost
always reassign members before decrementing their reference counts.  When
don’t we have to do this?


@itemize *

@item 
when we absolutely know that the reference count is greater than 1;

@item 
when we know that deallocation of the object @footnote{@w{(1)} 
This is true when we know that the object is a basic type, like a string or a
float.
} will neither release
the @ref{385,,GIL} nor cause any calls back into our type’s code;

@item 
when decrementing a reference count in a @ref{40aa,,tp_dealloc}
handler on a type which doesn’t support cyclic garbage collection @footnote{@w{(2)} 
We relied on this in the @ref{40aa,,tp_dealloc} handler
in this example, because our type doesn’t support garbage collection.
}.
@end itemize

We want to expose our instance variables as attributes. There are a
number of ways to do that. The simplest way is to define member definitions:

@example
static PyMemberDef Custom_members[] = @{
    @{"first", T_OBJECT_EX, offsetof(CustomObject, first), 0,
     "first name"@},
    @{"last", T_OBJECT_EX, offsetof(CustomObject, last), 0,
     "last name"@},
    @{"number", T_INT, offsetof(CustomObject, number), 0,
     "custom number"@},
    @{NULL@}  /* Sentinel */
@};
@end example

and put the definitions in the @ref{40ae,,tp_members} slot:

@example
.tp_members = Custom_members,
@end example

Each member definition has a member name, type, offset, access flags and
documentation string.  See the @ref{40af,,Generic Attribute Management} section
below for details.

A disadvantage of this approach is that it doesn’t provide a way to restrict the
types of objects that can be assigned to the Python attributes.  We expect the
first and last names to be strings, but any Python objects can be assigned.
Further, the attributes can be deleted, setting the C pointers to @code{NULL}.  Even
though we can make sure the members are initialized to non-@code{NULL} values, the
members can be set to @code{NULL} if the attributes are deleted.

We define a single method, @code{Custom.name()}, that outputs the objects name as the
concatenation of the first and last names.

@example
static PyObject *
Custom_name(CustomObject *self, PyObject *Py_UNUSED(ignored))
@{
    if (self->first == NULL) @{
        PyErr_SetString(PyExc_AttributeError, "first");
        return NULL;
    @}
    if (self->last == NULL) @{
        PyErr_SetString(PyExc_AttributeError, "last");
        return NULL;
    @}
    return PyUnicode_FromFormat("%S %S", self->first, self->last);
@}
@end example

The method is implemented as a C function that takes a @code{Custom} (or
@code{Custom} subclass) instance as the first argument.  Methods always take an
instance as the first argument. Methods often take positional and keyword
arguments as well, but in this case we don’t take any and don’t need to accept
a positional argument tuple or keyword argument dictionary. This method is
equivalent to the Python method:

@example
def name(self):
    return "%s %s" % (self.first, self.last)
@end example

Note that we have to check for the possibility that our @code{first} and
@code{last} members are @code{NULL}.  This is because they can be deleted, in which
case they are set to @code{NULL}.  It would be better to prevent deletion of these
attributes and to restrict the attribute values to be strings.  We’ll see how to
do that in the next section.

Now that we’ve defined the method, we need to create an array of method
definitions:

@example
static PyMethodDef Custom_methods[] = @{
    @{"name", (PyCFunction) Custom_name, METH_NOARGS,
     "Return the name, combining the first and last name"
    @},
    @{NULL@}  /* Sentinel */
@};
@end example

(note that we used the @ref{1011,,METH_NOARGS} flag to indicate that the method
is expecting no arguments other than @emph{self})

and assign it to the @ref{40b0,,tp_methods} slot:

@example
.tp_methods = Custom_methods,
@end example

Finally, we’ll make our type usable as a base class for subclassing.  We’ve
written our methods carefully so far so that they don’t make any assumptions
about the type of the object being created or used, so all we need to do is
to add the @ref{40b1,,Py_TPFLAGS_BASETYPE} to our class flag definition:

@example
.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
@end example

We rename @code{PyInit_custom()} to @code{PyInit_custom2()}, update the
module name in the @ref{484,,PyModuleDef} struct, and update the full class
name in the @ref{5c6,,PyTypeObject} struct.

Finally, we update our @code{setup.py} file to build the new module:

@example
from distutils.core import setup, Extension
setup(name="custom", version="1.0",
      ext_modules=[
         Extension("custom", ["custom.c"]),
         Extension("custom2", ["custom2.c"]),
         ])
@end example

@node Providing finer control over data attributes,Supporting cyclic garbage collection,Adding data and methods to the Basic example,Defining Extension Types Tutorial
@anchor{extending/newtypes_tutorial providing-finer-control-over-data-attributes}@anchor{40b2}
@subsubsection Providing finer control over data attributes


In this section, we’ll provide finer control over how the @code{first} and
@code{last} attributes are set in the @code{Custom} example. In the previous
version of our module, the instance variables @code{first} and @code{last}
could be set to non-string values or even deleted. We want to make sure that
these attributes always contain strings.

@example
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "structmember.h"

typedef struct @{
    PyObject_HEAD
    PyObject *first; /* first name */
    PyObject *last;  /* last name */
    int number;
@} CustomObject;

static void
Custom_dealloc(CustomObject *self)
@{
    Py_XDECREF(self->first);
    Py_XDECREF(self->last);
    Py_TYPE(self)->tp_free((PyObject *) self);
@}

static PyObject *
Custom_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
@{
    CustomObject *self;
    self = (CustomObject *) type->tp_alloc(type, 0);
    if (self != NULL) @{
        self->first = PyUnicode_FromString("");
        if (self->first == NULL) @{
            Py_DECREF(self);
            return NULL;
        @}
        self->last = PyUnicode_FromString("");
        if (self->last == NULL) @{
            Py_DECREF(self);
            return NULL;
        @}
        self->number = 0;
    @}
    return (PyObject *) self;
@}

static int
Custom_init(CustomObject *self, PyObject *args, PyObject *kwds)
@{
    static char *kwlist[] = @{"first", "last", "number", NULL@};
    PyObject *first = NULL, *last = NULL, *tmp;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|UUi", kwlist,
                                     &first, &last,
                                     &self->number))
        return -1;

    if (first) @{
        tmp = self->first;
        Py_INCREF(first);
        self->first = first;
        Py_DECREF(tmp);
    @}
    if (last) @{
        tmp = self->last;
        Py_INCREF(last);
        self->last = last;
        Py_DECREF(tmp);
    @}
    return 0;
@}

static PyMemberDef Custom_members[] = @{
    @{"number", T_INT, offsetof(CustomObject, number), 0,
     "custom number"@},
    @{NULL@}  /* Sentinel */
@};

static PyObject *
Custom_getfirst(CustomObject *self, void *closure)
@{
    Py_INCREF(self->first);
    return self->first;
@}

static int
Custom_setfirst(CustomObject *self, PyObject *value, void *closure)
@{
    PyObject *tmp;
    if (value == NULL) @{
        PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute");
        return -1;
    @}
    if (!PyUnicode_Check(value)) @{
        PyErr_SetString(PyExc_TypeError,
                        "The first attribute value must be a string");
        return -1;
    @}
    tmp = self->first;
    Py_INCREF(value);
    self->first = value;
    Py_DECREF(tmp);
    return 0;
@}

static PyObject *
Custom_getlast(CustomObject *self, void *closure)
@{
    Py_INCREF(self->last);
    return self->last;
@}

static int
Custom_setlast(CustomObject *self, PyObject *value, void *closure)
@{
    PyObject *tmp;
    if (value == NULL) @{
        PyErr_SetString(PyExc_TypeError, "Cannot delete the last attribute");
        return -1;
    @}
    if (!PyUnicode_Check(value)) @{
        PyErr_SetString(PyExc_TypeError,
                        "The last attribute value must be a string");
        return -1;
    @}
    tmp = self->last;
    Py_INCREF(value);
    self->last = value;
    Py_DECREF(tmp);
    return 0;
@}

static PyGetSetDef Custom_getsetters[] = @{
    @{"first", (getter) Custom_getfirst, (setter) Custom_setfirst,
     "first name", NULL@},
    @{"last", (getter) Custom_getlast, (setter) Custom_setlast,
     "last name", NULL@},
    @{NULL@}  /* Sentinel */
@};

static PyObject *
Custom_name(CustomObject *self, PyObject *Py_UNUSED(ignored))
@{
    return PyUnicode_FromFormat("%S %S", self->first, self->last);
@}

static PyMethodDef Custom_methods[] = @{
    @{"name", (PyCFunction) Custom_name, METH_NOARGS,
     "Return the name, combining the first and last name"
    @},
    @{NULL@}  /* Sentinel */
@};

static PyTypeObject CustomType = @{
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "custom3.Custom",
    .tp_doc = "Custom objects",
    .tp_basicsize = sizeof(CustomObject),
    .tp_itemsize = 0,
    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
    .tp_new = Custom_new,
    .tp_init = (initproc) Custom_init,
    .tp_dealloc = (destructor) Custom_dealloc,
    .tp_members = Custom_members,
    .tp_methods = Custom_methods,
    .tp_getset = Custom_getsetters,
@};

static PyModuleDef custommodule = @{
    PyModuleDef_HEAD_INIT,
    .m_name = "custom3",
    .m_doc = "Example module that creates an extension type.",
    .m_size = -1,
@};

PyMODINIT_FUNC
PyInit_custom3(void)
@{
    PyObject *m;
    if (PyType_Ready(&CustomType) < 0)
        return NULL;

    m = PyModule_Create(&custommodule);
    if (m == NULL)
        return NULL;

    Py_INCREF(&CustomType);
    if (PyModule_AddObject(m, "Custom", (PyObject *) &CustomType) < 0) @{
        Py_DECREF(&CustomType);
        Py_DECREF(m);
        return NULL;
    @}

    return m;
@}
@end example

To provide greater control, over the @code{first} and @code{last} attributes,
we’ll use custom getter and setter functions.  Here are the functions for
getting and setting the @code{first} attribute:

@example
static PyObject *
Custom_getfirst(CustomObject *self, void *closure)
@{
    Py_INCREF(self->first);
    return self->first;
@}

static int
Custom_setfirst(CustomObject *self, PyObject *value, void *closure)
@{
    PyObject *tmp;
    if (value == NULL) @{
        PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute");
        return -1;
    @}
    if (!PyUnicode_Check(value)) @{
        PyErr_SetString(PyExc_TypeError,
                        "The first attribute value must be a string");
        return -1;
    @}
    tmp = self->first;
    Py_INCREF(value);
    self->first = value;
    Py_DECREF(tmp);
    return 0;
@}
@end example

The getter function is passed a @code{Custom} object and a “closure”, which is
a void pointer.  In this case, the closure is ignored.  (The closure supports an
advanced usage in which definition data is passed to the getter and setter. This
could, for example, be used to allow a single set of getter and setter functions
that decide the attribute to get or set based on data in the closure.)

The setter function is passed the @code{Custom} object, the new value, and the
closure.  The new value may be @code{NULL}, in which case the attribute is being
deleted.  In our setter, we raise an error if the attribute is deleted or if its
new value is not a string.

We create an array of @ref{6ec,,PyGetSetDef} structures:

@example
static PyGetSetDef Custom_getsetters[] = @{
    @{"first", (getter) Custom_getfirst, (setter) Custom_setfirst,
     "first name", NULL@},
    @{"last", (getter) Custom_getlast, (setter) Custom_setlast,
     "last name", NULL@},
    @{NULL@}  /* Sentinel */
@};
@end example

and register it in the @ref{40b3,,tp_getset} slot:

@example
.tp_getset = Custom_getsetters,
@end example

The last item in a @ref{6ec,,PyGetSetDef} structure is the “closure” mentioned
above.  In this case, we aren’t using a closure, so we just pass @code{NULL}.

We also remove the member definitions for these attributes:

@example
static PyMemberDef Custom_members[] = @{
    @{"number", T_INT, offsetof(CustomObject, number), 0,
     "custom number"@},
    @{NULL@}  /* Sentinel */
@};
@end example

We also need to update the @ref{40ad,,tp_init} handler to only
allow strings @footnote{@w{(3)} 
We now know that the first and last members are strings, so perhaps we
could be less careful about decrementing their reference counts, however,
we accept instances of string subclasses.  Even though deallocating normal
strings won’t call back into our objects, we can’t guarantee that deallocating
an instance of a string subclass won’t call back into our objects.
} to be passed:

@example
static int
Custom_init(CustomObject *self, PyObject *args, PyObject *kwds)
@{
    static char *kwlist[] = @{"first", "last", "number", NULL@};
    PyObject *first = NULL, *last = NULL, *tmp;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|UUi", kwlist,
                                     &first, &last,
                                     &self->number))
        return -1;

    if (first) @{
        tmp = self->first;
        Py_INCREF(first);
        self->first = first;
        Py_DECREF(tmp);
    @}
    if (last) @{
        tmp = self->last;
        Py_INCREF(last);
        self->last = last;
        Py_DECREF(tmp);
    @}
    return 0;
@}
@end example

With these changes, we can assure that the @code{first} and @code{last} members are
never @code{NULL} so we can remove checks for @code{NULL} values in almost all cases.
This means that most of the @ref{1fd,,Py_XDECREF()} calls can be converted to
@ref{373,,Py_DECREF()} calls.  The only place we can’t change these calls is in
the @code{tp_dealloc} implementation, where there is the possibility that the
initialization of these members failed in @code{tp_new}.

We also rename the module initialization function and module name in the
initialization function, as we did before, and we add an extra definition to the
@code{setup.py} file.

@node Supporting cyclic garbage collection,Subclassing other types,Providing finer control over data attributes,Defining Extension Types Tutorial
@anchor{extending/newtypes_tutorial supporting-cyclic-garbage-collection}@anchor{40b4}
@subsubsection Supporting cyclic garbage collection


Python has a @ref{1192,,cyclic garbage collector (GC)} that
can identify unneeded objects even when their reference counts are not zero.
This can happen when objects are involved in cycles.  For example, consider:

@example
>>> l = []
>>> l.append(l)
>>> del l
@end example

In this example, we create a list that contains itself. When we delete it, it
still has a reference from itself. Its reference count doesn’t drop to zero.
Fortunately, Python’s cyclic garbage collector will eventually figure out that
the list is garbage and free it.

In the second version of the @code{Custom} example, we allowed any kind of
object to be stored in the @code{first} or @code{last} attributes @footnote{@w{(4)} 
Also, even with our attributes restricted to strings instances, the user
could pass arbitrary @ref{2ef,,str} subclasses and therefore still create
reference cycles.
}.
Besides, in the second and third versions, we allowed subclassing
@code{Custom}, and subclasses may add arbitrary attributes.  For any of
those two reasons, @code{Custom} objects can participate in cycles:

@example
>>> import custom3
>>> class Derived(custom3.Custom): pass
...
>>> n = Derived()
>>> n.some_attribute = n
@end example

To allow a @code{Custom} instance participating in a reference cycle to
be properly detected and collected by the cyclic GC, our @code{Custom} type
needs to fill two additional slots and to enable a flag that enables these slots:

@example
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "structmember.h"

typedef struct @{
    PyObject_HEAD
    PyObject *first; /* first name */
    PyObject *last;  /* last name */
    int number;
@} CustomObject;

static int
Custom_traverse(CustomObject *self, visitproc visit, void *arg)
@{
    Py_VISIT(self->first);
    Py_VISIT(self->last);
    return 0;
@}

static int
Custom_clear(CustomObject *self)
@{
    Py_CLEAR(self->first);
    Py_CLEAR(self->last);
    return 0;
@}

static void
Custom_dealloc(CustomObject *self)
@{
    PyObject_GC_UnTrack(self);
    Custom_clear(self);
    Py_TYPE(self)->tp_free((PyObject *) self);
@}

static PyObject *
Custom_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
@{
    CustomObject *self;
    self = (CustomObject *) type->tp_alloc(type, 0);
    if (self != NULL) @{
        self->first = PyUnicode_FromString("");
        if (self->first == NULL) @{
            Py_DECREF(self);
            return NULL;
        @}
        self->last = PyUnicode_FromString("");
        if (self->last == NULL) @{
            Py_DECREF(self);
            return NULL;
        @}
        self->number = 0;
    @}
    return (PyObject *) self;
@}

static int
Custom_init(CustomObject *self, PyObject *args, PyObject *kwds)
@{
    static char *kwlist[] = @{"first", "last", "number", NULL@};
    PyObject *first = NULL, *last = NULL, *tmp;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|UUi", kwlist,
                                     &first, &last,
                                     &self->number))
        return -1;

    if (first) @{
        tmp = self->first;
        Py_INCREF(first);
        self->first = first;
        Py_DECREF(tmp);
    @}
    if (last) @{
        tmp = self->last;
        Py_INCREF(last);
        self->last = last;
        Py_DECREF(tmp);
    @}
    return 0;
@}

static PyMemberDef Custom_members[] = @{
    @{"number", T_INT, offsetof(CustomObject, number), 0,
     "custom number"@},
    @{NULL@}  /* Sentinel */
@};

static PyObject *
Custom_getfirst(CustomObject *self, void *closure)
@{
    Py_INCREF(self->first);
    return self->first;
@}

static int
Custom_setfirst(CustomObject *self, PyObject *value, void *closure)
@{
    if (value == NULL) @{
        PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute");
        return -1;
    @}
    if (!PyUnicode_Check(value)) @{
        PyErr_SetString(PyExc_TypeError,
                        "The first attribute value must be a string");
        return -1;
    @}
    Py_INCREF(value);
    Py_CLEAR(self->first);
    self->first = value;
    return 0;
@}

static PyObject *
Custom_getlast(CustomObject *self, void *closure)
@{
    Py_INCREF(self->last);
    return self->last;
@}

static int
Custom_setlast(CustomObject *self, PyObject *value, void *closure)
@{
    if (value == NULL) @{
        PyErr_SetString(PyExc_TypeError, "Cannot delete the last attribute");
        return -1;
    @}
    if (!PyUnicode_Check(value)) @{
        PyErr_SetString(PyExc_TypeError,
                        "The last attribute value must be a string");
        return -1;
    @}
    Py_INCREF(value);
    Py_CLEAR(self->last);
    self->last = value;
    return 0;
@}

static PyGetSetDef Custom_getsetters[] = @{
    @{"first", (getter) Custom_getfirst, (setter) Custom_setfirst,
     "first name", NULL@},
    @{"last", (getter) Custom_getlast, (setter) Custom_setlast,
     "last name", NULL@},
    @{NULL@}  /* Sentinel */
@};

static PyObject *
Custom_name(CustomObject *self, PyObject *Py_UNUSED(ignored))
@{
    return PyUnicode_FromFormat("%S %S", self->first, self->last);
@}

static PyMethodDef Custom_methods[] = @{
    @{"name", (PyCFunction) Custom_name, METH_NOARGS,
     "Return the name, combining the first and last name"
    @},
    @{NULL@}  /* Sentinel */
@};

static PyTypeObject CustomType = @{
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "custom4.Custom",
    .tp_doc = "Custom objects",
    .tp_basicsize = sizeof(CustomObject),
    .tp_itemsize = 0,
    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC,
    .tp_new = Custom_new,
    .tp_init = (initproc) Custom_init,
    .tp_dealloc = (destructor) Custom_dealloc,
    .tp_traverse = (traverseproc) Custom_traverse,
    .tp_clear = (inquiry) Custom_clear,
    .tp_members = Custom_members,
    .tp_methods = Custom_methods,
    .tp_getset = Custom_getsetters,
@};

static PyModuleDef custommodule = @{
    PyModuleDef_HEAD_INIT,
    .m_name = "custom4",
    .m_doc = "Example module that creates an extension type.",
    .m_size = -1,
@};

PyMODINIT_FUNC
PyInit_custom4(void)
@{
    PyObject *m;
    if (PyType_Ready(&CustomType) < 0)
        return NULL;

    m = PyModule_Create(&custommodule);
    if (m == NULL)
        return NULL;

    Py_INCREF(&CustomType);
    if (PyModule_AddObject(m, "Custom", (PyObject *) &CustomType) < 0) @{
        Py_DECREF(&CustomType);
        Py_DECREF(m);
        return NULL;
    @}

    return m;
@}
@end example

First, the traversal method lets the cyclic GC know about subobjects that could
participate in cycles:

@example
static int
Custom_traverse(CustomObject *self, visitproc visit, void *arg)
@{
    int vret;
    if (self->first) @{
        vret = visit(self->first, arg);
        if (vret != 0)
            return vret;
    @}
    if (self->last) @{
        vret = visit(self->last, arg);
        if (vret != 0)
            return vret;
    @}
    return 0;
@}
@end example

For each subobject that can participate in cycles, we need to call the
@code{visit()} function, which is passed to the traversal method. The
@code{visit()} function takes as arguments the subobject and the extra argument
@emph{arg} passed to the traversal method.  It returns an integer value that must be
returned if it is non-zero.

Python provides a @ref{40b5,,Py_VISIT()} macro that automates calling visit
functions.  With @ref{40b5,,Py_VISIT()}, we can minimize the amount of boilerplate
in @code{Custom_traverse}:

@example
static int
Custom_traverse(CustomObject *self, visitproc visit, void *arg)
@{
    Py_VISIT(self->first);
    Py_VISIT(self->last);
    return 0;
@}
@end example

@cartouche
@quotation Note 
The @ref{1ee,,tp_traverse} implementation must name its
arguments exactly @emph{visit} and @emph{arg} in order to use @ref{40b5,,Py_VISIT()}.
@end quotation
@end cartouche

Second, we need to provide a method for clearing any subobjects that can
participate in cycles:

@example
static int
Custom_clear(CustomObject *self)
@{
    Py_CLEAR(self->first);
    Py_CLEAR(self->last);
    return 0;
@}
@end example

Notice the use of the @ref{fd2,,Py_CLEAR()} macro.  It is the recommended and safe
way to clear data attributes of arbitrary types while decrementing
their reference counts.  If you were to call @ref{1fd,,Py_XDECREF()} instead
on the attribute before setting it to @code{NULL}, there is a possibility
that the attribute’s destructor would call back into code that reads the
attribute again (@emph{especially} if there is a reference cycle).

@cartouche
@quotation Note 
You could emulate @ref{fd2,,Py_CLEAR()} by writing:

@example
PyObject *tmp;
tmp = self->first;
self->first = NULL;
Py_XDECREF(tmp);
@end example

Nevertheless, it is much easier and less error-prone to always
use @ref{fd2,,Py_CLEAR()} when deleting an attribute.  Don’t
try to micro-optimize at the expense of robustness!
@end quotation
@end cartouche

The deallocator @code{Custom_dealloc} may call arbitrary code when clearing
attributes.  It means the circular GC can be triggered inside the function.
Since the GC assumes reference count is not zero, we need to untrack the object
from the GC by calling @ref{103b,,PyObject_GC_UnTrack()} before clearing members.
Here is our reimplemented deallocator using @ref{103b,,PyObject_GC_UnTrack()}
and @code{Custom_clear}:

@example
static void
Custom_dealloc(CustomObject *self)
@{
    PyObject_GC_UnTrack(self);
    Custom_clear(self);
    Py_TYPE(self)->tp_free((PyObject *) self);
@}
@end example

Finally, we add the @ref{1ed,,Py_TPFLAGS_HAVE_GC} flag to the class flags:

@example
.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC,
@end example

That’s pretty much it.  If we had written custom @ref{40ac,,tp_alloc} or
@ref{40ab,,tp_free} handlers, we’d need to modify them for cyclic
garbage collection.  Most extensions will use the versions automatically provided.

@node Subclassing other types,,Supporting cyclic garbage collection,Defining Extension Types Tutorial
@anchor{extending/newtypes_tutorial subclassing-other-types}@anchor{40b6}
@subsubsection Subclassing other types


It is possible to create new extension types that are derived from existing
types. It is easiest to inherit from the built in types, since an extension can
easily use the @ref{5c6,,PyTypeObject} it needs. It can be difficult to share
these @ref{5c6,,PyTypeObject} structures between extension modules.

In this example we will create a @code{SubList} type that inherits from the
built-in @ref{43a,,list} type. The new type will be completely compatible with
regular lists, but will have an additional @code{increment()} method that
increases an internal counter:

@example
>>> import sublist
>>> s = sublist.SubList(range(3))
>>> s.extend(s)
>>> print(len(s))
6
>>> print(s.increment())
1
>>> print(s.increment())
2
@end example

@example
#define PY_SSIZE_T_CLEAN
#include <Python.h>

typedef struct @{
    PyListObject list;
    int state;
@} SubListObject;

static PyObject *
SubList_increment(SubListObject *self, PyObject *unused)
@{
    self->state++;
    return PyLong_FromLong(self->state);
@}

static PyMethodDef SubList_methods[] = @{
    @{"increment", (PyCFunction) SubList_increment, METH_NOARGS,
     PyDoc_STR("increment state counter")@},
    @{NULL@},
@};

static int
SubList_init(SubListObject *self, PyObject *args, PyObject *kwds)
@{
    if (PyList_Type.tp_init((PyObject *) self, args, kwds) < 0)
        return -1;
    self->state = 0;
    return 0;
@}

static PyTypeObject SubListType = @{
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "sublist.SubList",
    .tp_doc = "SubList objects",
    .tp_basicsize = sizeof(SubListObject),
    .tp_itemsize = 0,
    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
    .tp_init = (initproc) SubList_init,
    .tp_methods = SubList_methods,
@};

static PyModuleDef sublistmodule = @{
    PyModuleDef_HEAD_INIT,
    .m_name = "sublist",
    .m_doc = "Example module that creates an extension type.",
    .m_size = -1,
@};

PyMODINIT_FUNC
PyInit_sublist(void)
@{
    PyObject *m;
    SubListType.tp_base = &PyList_Type;
    if (PyType_Ready(&SubListType) < 0)
        return NULL;

    m = PyModule_Create(&sublistmodule);
    if (m == NULL)
        return NULL;

    Py_INCREF(&SubListType);
    if (PyModule_AddObject(m, "SubList", (PyObject *) &SubListType) < 0) @{
        Py_DECREF(&SubListType);
        Py_DECREF(m);
        return NULL;
    @}

    return m;
@}
@end example

As you can see, the source code closely resembles the @code{Custom} examples in
previous sections. We will break down the main differences between them.

@example
typedef struct @{
    PyListObject list;
    int state;
@} SubListObject;
@end example

The primary difference for derived type objects is that the base type’s
object structure must be the first value.  The base type will already include
the @ref{e7a,,PyObject_HEAD()} at the beginning of its structure.

When a Python object is a @code{SubList} instance, its @code{PyObject *} pointer
can be safely cast to both @code{PyListObject *} and @code{SubListObject *}:

@example
static int
SubList_init(SubListObject *self, PyObject *args, PyObject *kwds)
@{
    if (PyList_Type.tp_init((PyObject *) self, args, kwds) < 0)
        return -1;
    self->state = 0;
    return 0;
@}
@end example

We see above how to call through to the @code{__init__} method of the base
type.

This pattern is important when writing a type with custom
@ref{40a7,,tp_new} and @ref{40aa,,tp_dealloc}
members.  The @ref{40a7,,tp_new} handler should not actually
create the memory for the object with its @ref{40ac,,tp_alloc},
but let the base class handle it by calling its own @ref{40a7,,tp_new}.

The @ref{5c6,,PyTypeObject} struct supports a @ref{40b7,,tp_base}
specifying the type’s concrete base class.  Due to cross-platform compiler
issues, you can’t fill that field directly with a reference to
@ref{40b8,,PyList_Type}; it should be done later in the module initialization
function:

@example
PyMODINIT_FUNC
PyInit_sublist(void)
@{
    PyObject* m;
    SubListType.tp_base = &PyList_Type;
    if (PyType_Ready(&SubListType) < 0)
        return NULL;

    m = PyModule_Create(&sublistmodule);
    if (m == NULL)
        return NULL;

    Py_INCREF(&SubListType);
    if (PyModule_AddObject(m, "SubList", (PyObject *) &SubListType) < 0) @{
        Py_DECREF(&SubListType);
        Py_DECREF(m);
        return NULL;
    @}

    return m;
@}
@end example

Before calling @ref{1ec,,PyType_Ready()}, the type structure must have the
@ref{40b7,,tp_base} slot filled in.  When we are deriving an
existing type, it is not necessary to fill out the @ref{40ac,,tp_alloc}
slot with @ref{40a8,,PyType_GenericNew()} – the allocation function from the base
type will be inherited.

After that, calling @ref{1ec,,PyType_Ready()} and adding the type object to the
module is the same as with the basic @code{Custom} examples.

@node Defining Extension Types Assorted Topics,Building C and C++ Extensions,Defining Extension Types Tutorial,Creating extensions without third party tools
@anchor{extending/newtypes doc}@anchor{40b9}@anchor{extending/newtypes defining-extension-types-assorted-topics}@anchor{40ba}@anchor{extending/newtypes new-types-topics}@anchor{40bb}
@subsection Defining Extension Types: Assorted Topics

@anchor{extending/newtypes dnt-type-methods}@anchor{40bc}
This section aims to give a quick fly-by on the various type methods you can
implement and what they do.

Here is the definition of @ref{5c6,,PyTypeObject}, with some fields only used in
@ref{338,,debug builds} omitted:

@example
typedef struct _typeobject @{
    PyObject_VAR_HEAD
    const char *tp_name; /* For printing, in format "<module>.<name>" */
    Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */

    /* Methods to implement standard operations */

    destructor tp_dealloc;
    Py_ssize_t tp_vectorcall_offset;
    getattrfunc tp_getattr;
    setattrfunc tp_setattr;
    PyAsyncMethods *tp_as_async; /* formerly known as tp_compare (Python 2)
                                    or tp_reserved (Python 3) */
    reprfunc tp_repr;

    /* Method suites for standard classes */

    PyNumberMethods *tp_as_number;
    PySequenceMethods *tp_as_sequence;
    PyMappingMethods *tp_as_mapping;

    /* More standard operations (here for binary compatibility) */

    hashfunc tp_hash;
    ternaryfunc tp_call;
    reprfunc tp_str;
    getattrofunc tp_getattro;
    setattrofunc tp_setattro;

    /* Functions to access object as input/output buffer */
    PyBufferProcs *tp_as_buffer;

    /* Flags to define presence of optional/expanded features */
    unsigned long tp_flags;

    const char *tp_doc; /* Documentation string */

    /* Assigned meaning in release 2.0 */
    /* call function for all accessible objects */
    traverseproc tp_traverse;

    /* delete references to contained objects */
    inquiry tp_clear;

    /* Assigned meaning in release 2.1 */
    /* rich comparisons */
    richcmpfunc tp_richcompare;

    /* weak reference enabler */
    Py_ssize_t tp_weaklistoffset;

    /* Iterators */
    getiterfunc tp_iter;
    iternextfunc tp_iternext;

    /* Attribute descriptor and subclassing stuff */
    struct PyMethodDef *tp_methods;
    struct PyMemberDef *tp_members;
    struct PyGetSetDef *tp_getset;
    // Strong reference on a heap type, borrowed reference on a static type
    struct _typeobject *tp_base;
    PyObject *tp_dict;
    descrgetfunc tp_descr_get;
    descrsetfunc tp_descr_set;
    Py_ssize_t tp_dictoffset;
    initproc tp_init;
    allocfunc tp_alloc;
    newfunc tp_new;
    freefunc tp_free; /* Low-level free-memory routine */
    inquiry tp_is_gc; /* For PyObject_IS_GC */
    PyObject *tp_bases;
    PyObject *tp_mro; /* method resolution order */
    PyObject *tp_cache;
    PyObject *tp_subclasses;
    PyObject *tp_weaklist;
    destructor tp_del;

    /* Type attribute cache version tag. Added in version 2.6 */
    unsigned int tp_version_tag;

    destructor tp_finalize;
    vectorcallfunc tp_vectorcall;
@} PyTypeObject;
@end example

Now that’s a @emph{lot} of methods.  Don’t worry too much though – if you have
a type you want to define, the chances are very good that you will only
implement a handful of these.

As you probably expect by now, we’re going to go over this and give more
information about the various handlers.  We won’t go in the order they are
defined in the structure, because there is a lot of historical baggage that
impacts the ordering of the fields.  It’s often easiest to find an example
that includes the fields you need and then change the values to suit your new
type.

@example
const char *tp_name; /* For printing */
@end example

The name of the type – as mentioned in the previous chapter, this will appear in
various places, almost entirely for diagnostic purposes. Try to choose something
that will be helpful in such a situation!

@example
Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */
@end example

These fields tell the runtime how much memory to allocate when new objects of
this type are created.  Python has some built-in support for variable length
structures (think: strings, tuples) which is where the @ref{40a4,,tp_itemsize} field
comes in.  This will be dealt with later.

@example
const char *tp_doc;
@end example

Here you can put a string (or its address) that you want returned when the
Python script references @code{obj.__doc__} to retrieve the doc string.

Now we come to the basic type methods – the ones most extension types will
implement.

@menu
* Finalization and De-allocation:: 
* Object Presentation:: 
* Attribute Management:: 
* Object Comparison:: 
* Abstract Protocol Support:: 
* Weak Reference Support:: 
* More Suggestions:: 

@end menu

@node Finalization and De-allocation,Object Presentation,,Defining Extension Types Assorted Topics
@anchor{extending/newtypes finalization-and-de-allocation}@anchor{40bd}
@subsubsection Finalization and De-allocation


@geindex object; deallocation
@geindex deallocation@comma{} object
@geindex object; finalization
@geindex finalization@comma{} of objects

@example
destructor tp_dealloc;
@end example

This function is called when the reference count of the instance of your type is
reduced to zero and the Python interpreter wants to reclaim it.  If your type
has memory to free or other clean-up to perform, you can put it here.  The
object itself needs to be freed here as well.  Here is an example of this
function:

@example
static void
newdatatype_dealloc(newdatatypeobject *obj)
@{
    free(obj->obj_UnderlyingDatatypePtr);
    Py_TYPE(obj)->tp_free((PyObject *)obj);
@}
@end example

If your type supports garbage collection, the destructor should call
@ref{103b,,PyObject_GC_UnTrack()} before clearing any member fields:

@example
static void
newdatatype_dealloc(newdatatypeobject *obj)
@{
    PyObject_GC_UnTrack(obj);
    Py_CLEAR(obj->other_obj);
    ...
    Py_TYPE(obj)->tp_free((PyObject *)obj);
@}
@end example

@geindex PyErr_Fetch()
@geindex PyErr_Restore()

One important requirement of the deallocator function is that it leaves any
pending exceptions alone.  This is important since deallocators are frequently
called as the interpreter unwinds the Python stack; when the stack is unwound
due to an exception (rather than normal returns), nothing is done to protect the
deallocators from seeing that an exception has already been set.  Any actions
which a deallocator performs which may cause additional Python code to be
executed may detect that an exception has been set.  This can lead to misleading
errors from the interpreter.  The proper way to protect against this is to save
a pending exception before performing the unsafe action, and restoring it when
done.  This can be done using the @ref{bc2,,PyErr_Fetch()} and
@ref{40be,,PyErr_Restore()} functions:

@example
static void
my_dealloc(PyObject *obj)
@{
    MyObject *self = (MyObject *) obj;
    PyObject *cbresult;

    if (self->my_callback != NULL) @{
        PyObject *err_type, *err_value, *err_traceback;

        /* This saves the current exception state */
        PyErr_Fetch(&err_type, &err_value, &err_traceback);

        cbresult = PyObject_CallNoArgs(self->my_callback);
        if (cbresult == NULL)
            PyErr_WriteUnraisable(self->my_callback);
        else
            Py_DECREF(cbresult);

        /* This restores the saved exception state */
        PyErr_Restore(err_type, err_value, err_traceback);

        Py_DECREF(self->my_callback);
    @}
    Py_TYPE(obj)->tp_free((PyObject*)self);
@}
@end example

@cartouche
@quotation Note 
There are limitations to what you can safely do in a deallocator function.
First, if your type supports garbage collection (using @ref{1ee,,tp_traverse}
and/or @ref{40bf,,tp_clear}), some of the object’s members can have been
cleared or finalized by the time @ref{40aa,,tp_dealloc} is called.  Second, in
@ref{40aa,,tp_dealloc}, your object is in an unstable state: its reference
count is equal to zero.  Any call to a non-trivial object or API (as in the
example above) might end up calling @ref{40aa,,tp_dealloc} again, causing a
double free and a crash.

Starting with Python 3.4, it is recommended not to put any complex
finalization code in @ref{40aa,,tp_dealloc}, and instead use the new
@ref{5c7,,tp_finalize} type method.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 442
PEP 442@footnote{https://www.python.org/dev/peps/pep-0442} explains the new finalization scheme.

@end quotation
@end cartouche

@geindex string; object representation
@geindex built-in function; repr

@node Object Presentation,Attribute Management,Finalization and De-allocation,Defining Extension Types Assorted Topics
@anchor{extending/newtypes object-presentation}@anchor{40c0}
@subsubsection Object Presentation


In Python, there are two ways to generate a textual representation of an object:
the @ref{27d,,repr()} function, and the @ref{2ef,,str()} function.  (The @ref{af6,,print()}
function just calls @ref{2ef,,str()}.)  These handlers are both optional.

@example
reprfunc tp_repr;
reprfunc tp_str;
@end example

The @ref{40c1,,tp_repr} handler should return a string object containing a
representation of the instance for which it is called.  Here is a simple
example:

@example
static PyObject *
newdatatype_repr(newdatatypeobject * obj)
@{
    return PyUnicode_FromFormat("Repr-ified_newdatatype@{@{size:%d@}@}",
                                obj->obj_UnderlyingDatatypePtr->size);
@}
@end example

If no @ref{40c1,,tp_repr} handler is specified, the interpreter will supply a
representation that uses the type’s @ref{116b,,tp_name} and a uniquely-identifying
value for the object.

The @ref{40c2,,tp_str} handler is to @ref{2ef,,str()} what the @ref{40c1,,tp_repr} handler
described above is to @ref{27d,,repr()}; that is, it is called when Python code calls
@ref{2ef,,str()} on an instance of your object.  Its implementation is very similar
to the @ref{40c1,,tp_repr} function, but the resulting string is intended for human
consumption.  If @ref{40c2,,tp_str} is not specified, the @ref{40c1,,tp_repr} handler is
used instead.

Here is a simple example:

@example
static PyObject *
newdatatype_str(newdatatypeobject * obj)
@{
    return PyUnicode_FromFormat("Stringified_newdatatype@{@{size:%d@}@}",
                                obj->obj_UnderlyingDatatypePtr->size);
@}
@end example

@node Attribute Management,Object Comparison,Object Presentation,Defining Extension Types Assorted Topics
@anchor{extending/newtypes attribute-management}@anchor{40c3}
@subsubsection Attribute Management


For every object which can support attributes, the corresponding type must
provide the functions that control how the attributes are resolved.  There needs
to be a function which can retrieve attributes (if any are defined), and another
to set attributes (if setting attributes is allowed).  Removing an attribute is
a special case, for which the new value passed to the handler is @code{NULL}.

Python supports two pairs of attribute handlers; a type that supports attributes
only needs to implement the functions for one pair.  The difference is that one
pair takes the name of the attribute as a char*, while the other
accepts a @ref{375,,PyObject}*.  Each type can use whichever pair makes more
sense for the implementation’s convenience.

@example
getattrfunc  tp_getattr;        /* char * version */
setattrfunc  tp_setattr;
/* ... */
getattrofunc tp_getattro;       /* PyObject * version */
setattrofunc tp_setattro;
@end example

If accessing attributes of an object is always a simple operation (this will be
explained shortly), there are generic implementations which can be used to
provide the @ref{375,,PyObject}* version of the attribute management functions.
The actual need for type-specific attribute handlers almost completely
disappeared starting with Python 2.2, though there are many examples which have
not been updated to use some of the new generic mechanism that is available.

@menu
* Generic Attribute Management:: 
* Type-specific Attribute Management:: 

@end menu

@node Generic Attribute Management,Type-specific Attribute Management,,Attribute Management
@anchor{extending/newtypes generic-attribute-management}@anchor{40af}@anchor{extending/newtypes id1}@anchor{40c4}
@subsubsection Generic Attribute Management


Most extension types only use @emph{simple} attributes.  So, what makes the
attributes simple?  There are only a couple of conditions that must be met:


@enumerate 

@item 
The name of the attributes must be known when @ref{1ec,,PyType_Ready()} is
called.

@item 
No special processing is needed to record that an attribute was looked up or
set, nor do actions need to be taken based on the value.
@end enumerate

Note that this list does not place any restrictions on the values of the
attributes, when the values are computed, or how relevant data is stored.

When @ref{1ec,,PyType_Ready()} is called, it uses three tables referenced by the
type object to create @ref{17bc,,descriptor}s which are placed in the dictionary of the
type object.  Each descriptor controls access to one attribute of the instance
object.  Each of the tables is optional; if all three are @code{NULL}, instances of
the type will only have attributes that are inherited from their base type, and
should leave the @ref{40c5,,tp_getattro} and @ref{40c6,,tp_setattro} fields @code{NULL} as
well, allowing the base type to handle attributes.

The tables are declared as three fields of the type object:

@example
struct PyMethodDef *tp_methods;
struct PyMemberDef *tp_members;
struct PyGetSetDef *tp_getset;
@end example

If @ref{40b0,,tp_methods} is not @code{NULL}, it must refer to an array of
@ref{1014,,PyMethodDef} structures.  Each entry in the table is an instance of this
structure:

@example
typedef struct PyMethodDef @{
    const char  *ml_name;       /* method name */
    PyCFunction  ml_meth;       /* implementation function */
    int          ml_flags;      /* flags */
    const char  *ml_doc;        /* docstring */
@} PyMethodDef;
@end example

One entry should be defined for each method provided by the type; no entries are
needed for methods inherited from a base type.  One additional entry is needed
at the end; it is a sentinel that marks the end of the array.  The
@code{ml_name} field of the sentinel must be @code{NULL}.

The second table is used to define attributes which map directly to data stored
in the instance.  A variety of primitive C types are supported, and access may
be read-only or read-write.  The structures in the table are defined as:

@example
typedef struct PyMemberDef @{
    const char *name;
    int         type;
    int         offset;
    int         flags;
    const char *doc;
@} PyMemberDef;
@end example

For each entry in the table, a @ref{17bc,,descriptor} will be constructed and added to the
type which will be able to extract a value from the instance structure.  The
@ref{89e,,type} field should contain one of the type codes defined in the
@code{structmember.h} header; the value will be used to determine how to
convert Python values to and from C values.  The @code{flags} field is used to
store flags which control how the attribute can be accessed.

The following flag constants are defined in @code{structmember.h}; they may be
combined using bitwise-OR.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Meaning

@item

@code{READONLY}

@tab

Never writable.

@item

@code{PY_AUDIT_READ}

@tab

Emit an @code{object.__getattr__}
@ref{10cc,,audit events} before
reading.

@end multitable



Changed in version 3.10: @code{RESTRICTED}, @code{READ_RESTRICTED} and @code{WRITE_RESTRICTED}
are deprecated. However, @code{READ_RESTRICTED} is an alias for
@code{PY_AUDIT_READ}, so fields that specify either @code{RESTRICTED}
or @code{READ_RESTRICTED} will also raise an audit event.


@geindex READONLY
@geindex READ_RESTRICTED
@geindex WRITE_RESTRICTED
@geindex RESTRICTED
@geindex PY_AUDIT_READ

An interesting advantage of using the @ref{40ae,,tp_members} table to build
descriptors that are used at runtime is that any attribute defined this way can
have an associated doc string simply by providing the text in the table.  An
application can use the introspection API to retrieve the descriptor from the
class object, and get the doc string using its @code{__doc__} attribute.

As with the @ref{40b0,,tp_methods} table, a sentinel entry with a @code{name} value
of @code{NULL} is required.

@c XXX Descriptors need to be explained in more detail somewhere, but not here.
@c 
@c Descriptor objects have two handler functions which correspond to the
@c \member{tp_getattro} and \member{tp_setattro} handlers.  The
@c \method{__get__()} handler is a function which is passed the descriptor,
@c instance, and type objects, and returns the value of the attribute, or it
@c returns \NULL{} and sets an exception.  The \method{__set__()} handler is
@c passed the descriptor, instance, type, and new value;

@node Type-specific Attribute Management,,Generic Attribute Management,Attribute Management
@anchor{extending/newtypes type-specific-attribute-management}@anchor{40c7}
@subsubsection Type-specific Attribute Management


For simplicity, only the char* version will be demonstrated here; the
type of the name parameter is the only difference between the char*
and @ref{375,,PyObject}* flavors of the interface. This example effectively does
the same thing as the generic example above, but does not use the generic
support added in Python 2.2.  It explains how the handler functions are
called, so that if you do need to extend their functionality, you’ll understand
what needs to be done.

The @ref{40c8,,tp_getattr} handler is called when the object requires an attribute
look-up.  It is called in the same situations where the @code{__getattr__()}
method of a class would be called.

Here is an example:

@example
static PyObject *
newdatatype_getattr(newdatatypeobject *obj, char *name)
@{
    if (strcmp(name, "data") == 0)
    @{
        return PyLong_FromLong(obj->data);
    @}

    PyErr_Format(PyExc_AttributeError,
                 "'%.50s' object has no attribute '%.400s'",
                 tp->tp_name, name);
    return NULL;
@}
@end example

The @ref{40c9,,tp_setattr} handler is called when the @code{__setattr__()} or
@code{__delattr__()} method of a class instance would be called.  When an
attribute should be deleted, the third parameter will be @code{NULL}.  Here is an
example that simply raises an exception; if this were really all you wanted, the
@ref{40c9,,tp_setattr} handler should be set to @code{NULL}.

@example
static int
newdatatype_setattr(newdatatypeobject *obj, char *name, PyObject *v)
@{
    PyErr_Format(PyExc_RuntimeError, "Read-only attribute: %s", name);
    return -1;
@}
@end example

@node Object Comparison,Abstract Protocol Support,Attribute Management,Defining Extension Types Assorted Topics
@anchor{extending/newtypes object-comparison}@anchor{40ca}
@subsubsection Object Comparison


@example
richcmpfunc tp_richcompare;
@end example

The @ref{40cb,,tp_richcompare} handler is called when comparisons are needed.  It is
analogous to the @ref{17a6,,rich comparison methods}, like
@code{__lt__()}, and also called by @ref{40cc,,PyObject_RichCompare()} and
@ref{1239,,PyObject_RichCompareBool()}.

This function is called with two Python objects and the operator as arguments,
where the operator is one of @code{Py_EQ}, @code{Py_NE}, @code{Py_LE}, @code{Py_GE},
@code{Py_LT} or @code{Py_GT}.  It should compare the two objects with respect to the
specified operator and return @code{Py_True} or @code{Py_False} if the comparison is
successful, @code{Py_NotImplemented} to indicate that comparison is not
implemented and the other object’s comparison method should be tried, or @code{NULL}
if an exception was set.

Here is a sample implementation, for a datatype that is considered equal if the
size of an internal pointer is equal:

@example
static PyObject *
newdatatype_richcmp(PyObject *obj1, PyObject *obj2, int op)
@{
    PyObject *result;
    int c, size1, size2;

    /* code to make sure that both arguments are of type
       newdatatype omitted */

    size1 = obj1->obj_UnderlyingDatatypePtr->size;
    size2 = obj2->obj_UnderlyingDatatypePtr->size;

    switch (op) @{
    case Py_LT: c = size1 <  size2; break;
    case Py_LE: c = size1 <= size2; break;
    case Py_EQ: c = size1 == size2; break;
    case Py_NE: c = size1 != size2; break;
    case Py_GT: c = size1 >  size2; break;
    case Py_GE: c = size1 >= size2; break;
    @}
    result = c ? Py_True : Py_False;
    Py_INCREF(result);
    return result;
 @}
@end example

@node Abstract Protocol Support,Weak Reference Support,Object Comparison,Defining Extension Types Assorted Topics
@anchor{extending/newtypes abstract-protocol-support}@anchor{40cd}
@subsubsection Abstract Protocol Support


Python supports a variety of @emph{abstract} ‘protocols;’ the specific interfaces
provided to use these interfaces are documented in @ref{40ce,,Abstract Objects Layer}.

A number of these abstract interfaces were defined early in the development of
the Python implementation.  In particular, the number, mapping, and sequence
protocols have been part of Python since the beginning.  Other protocols have
been added over time.  For protocols which depend on several handler routines
from the type implementation, the older protocols have been defined as optional
blocks of handlers referenced by the type object.  For newer protocols there are
additional slots in the main type object, with a flag bit being set to indicate
that the slots are present and should be checked by the interpreter.  (The flag
bit does not indicate that the slot values are non-@code{NULL}. The flag may be set
to indicate the presence of a slot, but a slot may still be unfilled.)

@example
PyNumberMethods   *tp_as_number;
PySequenceMethods *tp_as_sequence;
PyMappingMethods  *tp_as_mapping;
@end example

If you wish your object to be able to act like a number, a sequence, or a
mapping object, then you place the address of a structure that implements the C
type @ref{f7b,,PyNumberMethods}, @ref{40cf,,PySequenceMethods}, or
@ref{40d0,,PyMappingMethods}, respectively. It is up to you to fill in this
structure with appropriate values. You can find examples of the use of each of
these in the @code{Objects} directory of the Python source distribution.

@example
hashfunc tp_hash;
@end example

This function, if you choose to provide it, should return a hash number for an
instance of your data type. Here is a simple example:

@example
static Py_hash_t
newdatatype_hash(newdatatypeobject *obj)
@{
    Py_hash_t result;
    result = obj->some_size + 32767 * obj->some_number;
    if (result == -1)
       result = -2;
    return result;
@}
@end example

@code{Py_hash_t} is a signed integer type with a platform-varying width.
Returning @code{-1} from @ref{40d1,,tp_hash} indicates an error,
which is why you should be careful to avoid returning it when hash computation
is successful, as seen above.

@example
ternaryfunc tp_call;
@end example

This function is called when an instance of your data type is “called”, for
example, if @code{obj1} is an instance of your data type and the Python script
contains @code{obj1('hello')}, the @ref{40d2,,tp_call} handler is invoked.

This function takes three arguments:


@enumerate 

@item 
@emph{self} is the instance of the data type which is the subject of the call.
If the call is @code{obj1('hello')}, then @emph{self} is @code{obj1}.

@item 
@emph{args} is a tuple containing the arguments to the call.  You can use
@ref{37f,,PyArg_ParseTuple()} to extract the arguments.

@item 
@emph{kwds} is a dictionary of keyword arguments that were passed. If this is
non-@code{NULL} and you support keyword arguments, use
@ref{863,,PyArg_ParseTupleAndKeywords()} to extract the arguments.  If you
do not want to support keyword arguments and this is non-@code{NULL}, raise a
@ref{162,,TypeError} with a message saying that keyword arguments are not supported.
@end enumerate

Here is a toy @code{tp_call} implementation:

@example
static PyObject *
newdatatype_call(newdatatypeobject *self, PyObject *args, PyObject *kwds)
@{
    PyObject *result;
    const char *arg1;
    const char *arg2;
    const char *arg3;

    if (!PyArg_ParseTuple(args, "sss:call", &arg1, &arg2, &arg3)) @{
        return NULL;
    @}
    result = PyUnicode_FromFormat(
        "Returning -- value: [%d] arg1: [%s] arg2: [%s] arg3: [%s]\n",
        obj->obj_UnderlyingDatatypePtr->size,
        arg1, arg2, arg3);
    return result;
@}
@end example

@example
/* Iterators */
getiterfunc tp_iter;
iternextfunc tp_iternext;
@end example

These functions provide support for the iterator protocol.  Both handlers
take exactly one parameter, the instance for which they are being called,
and return a new reference.  In the case of an error, they should set an
exception and return @code{NULL}.  @ref{1028,,tp_iter} corresponds
to the Python @code{__iter__()} method, while @ref{1029,,tp_iternext}
corresponds to the Python @ref{e6d,,__next__()} method.

Any @ref{dcc,,iterable} object must implement the @ref{1028,,tp_iter}
handler, which must return an @ref{11c3,,iterator} object.  Here the same guidelines
apply as for Python classes:


@itemize *

@item 
For collections (such as lists and tuples) which can support multiple
independent iterators, a new iterator should be created and returned by
each call to @ref{1028,,tp_iter}.

@item 
Objects which can only be iterated over once (usually due to side effects of
iteration, such as file objects) can implement @ref{1028,,tp_iter}
by returning a new reference to themselves – and should also therefore
implement the @ref{1029,,tp_iternext}  handler.
@end itemize

Any @ref{11c3,,iterator} object should implement both @ref{1028,,tp_iter}
and @ref{1029,,tp_iternext}.  An iterator’s
@ref{1028,,tp_iter} handler should return a new reference
to the iterator.  Its @ref{1029,,tp_iternext} handler should
return a new reference to the next object in the iteration, if there is one.
If the iteration has reached the end, @ref{1029,,tp_iternext}
may return @code{NULL} without setting an exception, or it may set
@ref{736,,StopIteration} @emph{in addition} to returning @code{NULL}; avoiding
the exception can yield slightly better performance.  If an actual error
occurs, @ref{1029,,tp_iternext} should always set an exception
and return @code{NULL}.

@node Weak Reference Support,More Suggestions,Abstract Protocol Support,Defining Extension Types Assorted Topics
@anchor{extending/newtypes weak-reference-support}@anchor{40d3}@anchor{extending/newtypes weakref-support}@anchor{1e1e}
@subsubsection Weak Reference Support


One of the goals of Python’s weak reference implementation is to allow any type
to participate in the weak reference mechanism without incurring the overhead on
performance-critical objects (such as numbers).


@subsubheading See also


Documentation for the @ref{12f,,weakref} module.


For an object to be weakly referencable, the extension type must do two things:


@enumerate 

@item 
Include a @ref{375,,PyObject}* field in the C object structure dedicated to
the weak reference mechanism.  The object’s constructor should leave it
@code{NULL} (which is automatic when using the default
@ref{40ac,,tp_alloc}).

@item 
Set the @ref{494,,tp_weaklistoffset} type member
to the offset of the aforementioned field in the C object structure,
so that the interpreter knows how to access and modify that field.
@end enumerate

Concretely, here is how a trivial object structure would be augmented
with the required field:

@example
typedef struct @{
    PyObject_HEAD
    PyObject *weakreflist;  /* List of weak references */
@} TrivialObject;
@end example

And the corresponding member in the statically-declared type object:

@example
static PyTypeObject TrivialType = @{
    PyVarObject_HEAD_INIT(NULL, 0)
    /* ... other members omitted for brevity ... */
    .tp_weaklistoffset = offsetof(TrivialObject, weakreflist),
@};
@end example

The only further addition is that @code{tp_dealloc} needs to clear any weak
references (by calling @code{PyObject_ClearWeakRefs()}) if the field is
non-@code{NULL}:

@example
static void
Trivial_dealloc(TrivialObject *self)
@{
    /* Clear weakrefs first before calling any destructors */
    if (self->weakreflist != NULL)
        PyObject_ClearWeakRefs((PyObject *) self);
    /* ... remainder of destruction code omitted for brevity ... */
    Py_TYPE(self)->tp_free((PyObject *) self);
@}
@end example

@node More Suggestions,,Weak Reference Support,Defining Extension Types Assorted Topics
@anchor{extending/newtypes more-suggestions}@anchor{40d4}
@subsubsection More Suggestions


In order to learn how to implement any specific method for your new data type,
get the @ref{17a4,,CPython} source code.  Go to the @code{Objects} directory,
then search the C source files for @code{tp_} plus the function you want
(for example, @code{tp_richcompare}).  You will find examples of the function
you want to implement.

When you need to verify that an object is a concrete instance of the type you
are implementing, use the @ref{116f,,PyObject_TypeCheck()} function.  A sample of
its use might be something like the following:

@example
if (!PyObject_TypeCheck(some_object, &MyType)) @{
    PyErr_SetString(PyExc_TypeError, "arg #1 not a mything");
    return NULL;
@}
@end example


@subsubheading See also



@table @asis

@item Download CPython source releases.

@indicateurl{https://www.python.org/downloads/source/}

@item The CPython project on GitHub, where the CPython source code is developed.

@indicateurl{https://github.com/python/cpython}
@end table


@node Building C and C++ Extensions,Building C and C++ Extensions on Windows,Defining Extension Types Assorted Topics,Creating extensions without third party tools
@anchor{extending/building doc}@anchor{40d5}@anchor{extending/building building}@anchor{407d}@anchor{extending/building building-c-and-c-extensions}@anchor{40d6}
@subsection Building C and C++ Extensions


A C extension for CPython is a shared library (e.g. a @code{.so} file on Linux,
@code{.pyd} on Windows), which exports an @emph{initialization function}.

To be importable, the shared library must be available on 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH},
and must be named after the module name, with an appropriate extension.
When using distutils, the correct filename is generated automatically.

The initialization function has the signature:

@geindex PyInit_modulename (C function)
@anchor{extending/building c PyInit_modulename}@anchor{40d7}
@deffn {C Function} @ref{375,,PyObject}  *PyInit_modulename (void)
@end deffn

It returns either a fully-initialized module, or a @ref{484,,PyModuleDef}
instance. See @ref{40d8,,Initializing C modules} for details.

For modules with ASCII-only names, the function must be named
@code{PyInit_<modulename>}, with @code{<modulename>} replaced by the name of the
module. When using @ref{40d9,,Multi-phase initialization}, non-ASCII module names
are allowed. In this case, the initialization function name is
@code{PyInitU_<modulename>}, with @code{<modulename>} encoded using Python’s
@emph{punycode} encoding with hyphens replaced by underscores. In Python:

@example
def initfunc_name(name):
    try:
        suffix = b'_' + name.encode('ascii')
    except UnicodeEncodeError:
        suffix = b'U_' + name.encode('punycode').replace(b'-', b'_')
    return b'PyInit' + suffix
@end example

It is possible to export multiple modules from a single shared library by
defining multiple initialization functions. However, importing them requires
using symbolic links or a custom importer, because by default only the
function corresponding to the filename is found.
See the @emph{“Multiple modules in one library”} section in 
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489} for details.

@menu
* Building C and C++ Extensions with distutils:: 
* Distributing your extension modules:: 

@end menu

@node Building C and C++ Extensions with distutils,Distributing your extension modules,,Building C and C++ Extensions
@anchor{extending/building building-c-and-c-extensions-with-distutils}@anchor{40da}
@subsubsection Building C and C++ Extensions with distutils


Extension modules can be built using distutils,  which is included in Python.
Since distutils also supports creation of binary packages, users don’t
necessarily need a compiler and distutils to install the extension.

A distutils package contains a driver script, @code{setup.py}. This is a plain
Python file, which, in the most simple case, could look like this:

@example
from distutils.core import setup, Extension

module1 = Extension('demo',
                    sources = ['demo.c'])

setup (name = 'PackageName',
       version = '1.0',
       description = 'This is a demo package',
       ext_modules = [module1])
@end example

With this @code{setup.py}, and a file @code{demo.c}, running

@example
python setup.py build
@end example

will compile @code{demo.c}, and produce an extension module named @code{demo} in
the @code{build} directory. Depending on the system, the module file will end
up in a subdirectory @code{build/lib.system}, and may have a name like
@code{demo.so} or @code{demo.pyd}.

In the @code{setup.py}, all execution is performed by calling the @code{setup}
function. This takes a variable number of keyword arguments, of which the
example above uses only a subset. Specifically, the example specifies
meta-information to build packages, and it specifies the contents of the
package.  Normally, a package will contain additional modules, like Python
source modules, documentation, subpackages, etc. Please refer to the distutils
documentation in @ref{a6c,,Distributing Python Modules (Legacy version)} to learn more about the features of
distutils; this section explains building extension modules only.

It is common to pre-compute arguments to @code{setup()}, to better structure the
driver script. In the example above, the @code{ext_modules} argument to
@ref{40db,,setup()} is a list of extension modules, each of which is
an instance of
the @code{Extension}. In the example, the instance
defines an extension named @code{demo} which is build by compiling a single source
file, @code{demo.c}.

In many cases, building an extension is more complex, since additional
preprocessor defines and libraries may be needed. This is demonstrated in the
example below.

@example
from distutils.core import setup, Extension

module1 = Extension('demo',
                    define_macros = [('MAJOR_VERSION', '1'),
                                     ('MINOR_VERSION', '0')],
                    include_dirs = ['/usr/local/include'],
                    libraries = ['tcl83'],
                    library_dirs = ['/usr/local/lib'],
                    sources = ['demo.c'])

setup (name = 'PackageName',
       version = '1.0',
       description = 'This is a demo package',
       author = 'Martin v. Loewis',
       author_email = 'martin@@v.loewis.de',
       url = 'https://docs.python.org/extending/building',
       long_description = '@w{'}'
This is really just a demo package.
'@w{'}',
       ext_modules = [module1])
@end example

In this example, @ref{40db,,setup()} is called with additional
meta-information, which
is recommended when distribution packages have to be built. For the extension
itself, it specifies preprocessor defines, include directories, library
directories, and libraries. Depending on the compiler, distutils passes this
information in different ways to the compiler. For example, on Unix, this may
result in the compilation commands

@example
gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -DMAJOR_VERSION=1 -DMINOR_VERSION=0 -I/usr/local/include -I/usr/local/include/python2.2 -c demo.c -o build/temp.linux-i686-2.2/demo.o

gcc -shared build/temp.linux-i686-2.2/demo.o -L/usr/local/lib -ltcl83 -o build/lib.linux-i686-2.2/demo.so
@end example

These lines are for demonstration purposes only; distutils users should trust
that distutils gets the invocations right.

@node Distributing your extension modules,,Building C and C++ Extensions with distutils,Building C and C++ Extensions
@anchor{extending/building distributing}@anchor{40dc}@anchor{extending/building distributing-your-extension-modules}@anchor{40dd}
@subsubsection Distributing your extension modules


When an extension has been successfully built, there are three ways to use it.

End-users will typically want to install the module, they do so by running

@example
python setup.py install
@end example

Module maintainers should produce source packages; to do so, they run

@example
python setup.py sdist
@end example

In some cases, additional files need to be included in a source distribution;
this is done through a @code{MANIFEST.in} file; see @ref{40de,,Specifying the files to distribute} for details.

If the source distribution has been built successfully, maintainers can also
create binary distributions. Depending on the platform, one of the following
commands can be used to do so.

@example
python setup.py bdist_rpm
python setup.py bdist_dumb
@end example

@node Building C and C++ Extensions on Windows,,Building C and C++ Extensions,Creating extensions without third party tools
@anchor{extending/windows doc}@anchor{40df}@anchor{extending/windows building-c-and-c-extensions-on-windows}@anchor{40e0}@anchor{extending/windows building-on-windows}@anchor{16d1}
@subsection Building C and C++ Extensions on Windows


This chapter briefly explains how to create a Windows extension module for
Python using Microsoft Visual C++, and follows with more detailed background
information on how it works.  The explanatory material is useful for both the
Windows programmer learning to build Python extensions and the Unix programmer
interested in producing software which can be successfully built on both Unix
and Windows.

Module authors are encouraged to use the distutils approach for building
extension modules, instead of the one described in this section. You will still
need the C compiler that was used to build Python; typically Microsoft Visual
C++.

@cartouche
@quotation Note 
This chapter mentions a number of filenames that include an encoded Python
version number.  These filenames are represented with the version number shown
as @code{XY}; in practice, @code{'X'} will be the major version number and @code{'Y'}
will be the minor version number of the Python release you’re working with.  For
example, if you are using Python 2.2.1, @code{XY} will actually be @code{22}.
@end quotation
@end cartouche

@menu
* A Cookbook Approach:: 
* Differences Between Unix and Windows:: 
* Using DLLs in Practice:: 

@end menu

@node A Cookbook Approach,Differences Between Unix and Windows,,Building C and C++ Extensions on Windows
@anchor{extending/windows a-cookbook-approach}@anchor{40e1}@anchor{extending/windows win-cookbook}@anchor{40e2}
@subsubsection A Cookbook Approach


There are two approaches to building extension modules on Windows, just as there
are on Unix: use the @ref{37,,distutils} package to control the build process, or
do things manually.  The distutils approach works well for most extensions;
documentation on using @ref{37,,distutils} to build and package extension modules
is available in @ref{a6c,,Distributing Python Modules (Legacy version)}.  If you find you really need to do
things manually, it may be instructive to study the project file for the
winsound@footnote{https://github.com/python/cpython/tree/main/PCbuild/winsound.vcxproj} standard library module.

@node Differences Between Unix and Windows,Using DLLs in Practice,A Cookbook Approach,Building C and C++ Extensions on Windows
@anchor{extending/windows differences-between-unix-and-windows}@anchor{40e3}@anchor{extending/windows dynamic-linking}@anchor{40e4}
@subsubsection Differences Between Unix and Windows


Unix and Windows use completely different paradigms for run-time loading of
code.  Before you try to build a module that can be dynamically loaded, be aware
of how your system works.

In Unix, a shared object (@code{.so}) file contains code to be used by the
program, and also the names of functions and data that it expects to find in the
program.  When the file is joined to the program, all references to those
functions and data in the file’s code are changed to point to the actual
locations in the program where the functions and data are placed in memory.
This is basically a link operation.

In Windows, a dynamic-link library (@code{.dll}) file has no dangling
references.  Instead, an access to functions or data goes through a lookup
table.  So the DLL code does not have to be fixed up at runtime to refer to the
program’s memory; instead, the code already uses the DLL’s lookup table, and the
lookup table is modified at runtime to point to the functions and data.

In Unix, there is only one type of library file (@code{.a}) which contains code
from several object files (@code{.o}).  During the link step to create a shared
object file (@code{.so}), the linker may find that it doesn’t know where an
identifier is defined.  The linker will look for it in the object files in the
libraries; if it finds it, it will include all the code from that object file.

In Windows, there are two types of library, a static library and an import
library (both called @code{.lib}).  A static library is like a Unix @code{.a}
file; it contains code to be included as necessary. An import library is
basically used only to reassure the linker that a certain identifier is legal,
and will be present in the program when the DLL is loaded.  So the linker uses
the information from the import library to build the lookup table for using
identifiers that are not included in the DLL.  When an application or a DLL is
linked, an import library may be generated, which will need to be used for all
future DLLs that depend on the symbols in the application or DLL.

Suppose you are building two dynamic-load modules, B and C, which should share
another block of code A.  On Unix, you would @emph{not} pass @code{A.a} to the
linker for @code{B.so} and @code{C.so}; that would cause it to be included
twice, so that B and C would each have their own copy.  In Windows, building
@code{A.dll} will also build @code{A.lib}.  You @emph{do} pass @code{A.lib} to the
linker for B and C.  @code{A.lib} does not contain code; it just contains
information which will be used at runtime to access A’s code.

In Windows, using an import library is sort of like using @code{import spam}; it
gives you access to spam’s names, but does not create a separate copy.  On Unix,
linking with a library is more like @code{from spam import *}; it does create a
separate copy.

@node Using DLLs in Practice,,Differences Between Unix and Windows,Building C and C++ Extensions on Windows
@anchor{extending/windows using-dlls-in-practice}@anchor{40e5}@anchor{extending/windows win-dlls}@anchor{40e6}
@subsubsection Using DLLs in Practice


Windows Python is built in Microsoft Visual C++; using other compilers may or
may not work (though Borland seems to).  The rest of this section is MSVC++
specific.

When creating DLLs in Windows, you must pass @code{pythonXY.lib} to the linker.
To build two DLLs, spam and ni (which uses C functions found in spam), you could
use these commands:

@example
cl /LD /I/python/include spam.c ../libs/pythonXY.lib
cl /LD /I/python/include ni.c spam.lib ../libs/pythonXY.lib
@end example

The first command created three files: @code{spam.obj}, @code{spam.dll} and
@code{spam.lib}.  @code{Spam.dll} does not contain any Python functions (such
as @ref{37f,,PyArg_ParseTuple()}), but it does know how to find the Python code
thanks to @code{pythonXY.lib}.

The second command created @code{ni.dll} (and @code{.obj} and @code{.lib}),
which knows how to find the necessary functions from spam, and also from the
Python executable.

Not every identifier is exported to the lookup table.  If you want any other
modules (including Python) to be able to see your identifiers, you have to say
@code{_declspec(dllexport)}, as in @code{void _declspec(dllexport) initspam(void)} or
@code{PyObject _declspec(dllexport) *NiGetSpamData(void)}.

Developer Studio will throw in a lot of import libraries that you do not really
need, adding about 100K to your executable.  To get rid of them, use the Project
Settings dialog, Link tab, to specify @emph{ignore default libraries}.  Add the
correct @code{msvcrtxx.lib} to the list of libraries.

@node Embedding the CPython runtime in a larger application,,Creating extensions without third party tools,Extending and Embedding the Python Interpreter
@anchor{extending/index embedding-the-cpython-runtime-in-a-larger-application}@anchor{40e7}
@section Embedding the CPython runtime in a larger application


Sometimes, rather than creating an extension that runs inside the Python
interpreter as the main application, it is desirable to instead embed
the CPython runtime inside a larger application. This section covers
some of the details involved in doing that successfully.

@menu
* Embedding Python in Another Application:: 

@end menu

@node Embedding Python in Another Application,,,Embedding the CPython runtime in a larger application
@anchor{extending/embedding doc}@anchor{40e8}@anchor{extending/embedding embedding}@anchor{40e9}@anchor{extending/embedding embedding-python-in-another-application}@anchor{40ea}
@subsection Embedding Python in Another Application


The previous chapters discussed how to extend Python, that is, how to extend the
functionality of Python by attaching a library of C functions to it.  It is also
possible to do it the other way around: enrich your C/C++ application by
embedding Python in it.  Embedding provides your application with the ability to
implement some of the functionality of your application in Python rather than C
or C++. This can be used for many purposes; one example would be to allow users
to tailor the application to their needs by writing some scripts in Python.  You
can also use it yourself if some of the functionality can be written in Python
more easily.

Embedding Python is similar to extending it, but not quite.  The difference is
that when you extend Python, the main program of the application is still the
Python interpreter, while if you embed Python, the main program may have nothing
to do with Python — instead, some parts of the application occasionally call
the Python interpreter to run some Python code.

So if you are embedding Python, you are providing your own main program.  One of
the things this main program has to do is initialize the Python interpreter.  At
the very least, you have to call the function @ref{38d,,Py_Initialize()}.  There are
optional calls to pass command line arguments to Python.  Then later you can
call the interpreter from any part of the application.

There are several different ways to call the interpreter: you can pass a string
containing Python statements to @ref{40eb,,PyRun_SimpleString()}, or you can pass a
stdio file pointer and a file name (for identification in error messages only)
to @ref{40ec,,PyRun_SimpleFile()}.  You can also call the lower-level operations
described in the previous chapters to construct and use Python objects.


@subsubheading See also



@table @asis

@item @ref{14bf,,Python/C API Reference Manual}

The details of Python’s C interface are given in this manual. A great deal of
necessary information can be found here.
@end table


@menu
* Very High Level Embedding:: 
* Beyond Very High Level Embedding; An overview: Beyond Very High Level Embedding An overview. 
* Pure Embedding:: 
* Extending Embedded Python:: 
* Embedding Python in C++:: 
* Compiling and Linking under Unix-like systems:: 

@end menu

@node Very High Level Embedding,Beyond Very High Level Embedding An overview,,Embedding Python in Another Application
@anchor{extending/embedding high-level-embedding}@anchor{40ed}@anchor{extending/embedding very-high-level-embedding}@anchor{40ee}
@subsubsection Very High Level Embedding


The simplest form of embedding Python is the use of the very high level
interface. This interface is intended to execute a Python script without needing
to interact with the application directly. This can for example be used to
perform some operation on a file.

@example
#define PY_SSIZE_T_CLEAN
#include <Python.h>

int
main(int argc, char *argv[])
@{
    wchar_t *program = Py_DecodeLocale(argv[0], NULL);
    if (program == NULL) @{
        fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
        exit(1);
    @}
    Py_SetProgramName(program);  /* optional but recommended */
    Py_Initialize();
    PyRun_SimpleString("from time import time,ctime\n"
                       "print('Today is', ctime(time()))\n");
    if (Py_FinalizeEx() < 0) @{
        exit(120);
    @}
    PyMem_RawFree(program);
    return 0;
@}
@end example

The @ref{203,,Py_SetProgramName()} function should be called before
@ref{38d,,Py_Initialize()} to inform the interpreter about paths to Python run-time
libraries.  Next, the Python interpreter is initialized with
@ref{38d,,Py_Initialize()}, followed by the execution of a hard-coded Python script
that prints the date and time.  Afterwards, the @ref{862,,Py_FinalizeEx()} call shuts
the interpreter down, followed by the end of the program.  In a real program,
you may want to get the Python script from another source, perhaps a text-editor
routine, a file, or a database.  Getting the Python code from a file can better
be done by using the @ref{40ec,,PyRun_SimpleFile()} function, which saves you the
trouble of allocating memory space and loading the file contents.

@node Beyond Very High Level Embedding An overview,Pure Embedding,Very High Level Embedding,Embedding Python in Another Application
@anchor{extending/embedding beyond-very-high-level-embedding-an-overview}@anchor{40ef}@anchor{extending/embedding lower-level-embedding}@anchor{40f0}
@subsubsection Beyond Very High Level Embedding: An overview


The high level interface gives you the ability to execute arbitrary pieces of
Python code from your application, but exchanging data values is quite
cumbersome to say the least. If you want that, you should use lower level calls.
At the cost of having to write more C code, you can achieve almost anything.

It should be noted that extending Python and embedding Python is quite the same
activity, despite the different intent. Most topics discussed in the previous
chapters are still valid. To show this, consider what the extension code from
Python to C really does:


@enumerate 

@item 
Convert data values from Python to C,

@item 
Perform a function call to a C routine using the converted values, and

@item 
Convert the data values from the call from C to Python.
@end enumerate

When embedding Python, the interface code does:


@enumerate 

@item 
Convert data values from C to Python,

@item 
Perform a function call to a Python interface routine using the converted
values, and

@item 
Convert the data values from the call from Python to C.
@end enumerate

As you can see, the data conversion steps are simply swapped to accommodate the
different direction of the cross-language transfer. The only difference is the
routine that you call between both data conversions. When extending, you call a
C routine, when embedding, you call a Python routine.

This chapter will not discuss how to convert data from Python to C and vice
versa.  Also, proper use of references and dealing with errors is assumed to be
understood.  Since these aspects do not differ from extending the interpreter,
you can refer to earlier chapters for the required information.

@node Pure Embedding,Extending Embedded Python,Beyond Very High Level Embedding An overview,Embedding Python in Another Application
@anchor{extending/embedding id1}@anchor{40f1}@anchor{extending/embedding pure-embedding}@anchor{40f2}
@subsubsection Pure Embedding


The first program aims to execute a function in a Python script. Like in the
section about the very high level interface, the Python interpreter does not
directly interact with the application (but that will change in the next
section).

The code to run a function defined in a Python script is:

@example
#define PY_SSIZE_T_CLEAN
#include <Python.h>

int
main(int argc, char *argv[])
@{
    PyObject *pName, *pModule, *pFunc;
    PyObject *pArgs, *pValue;
    int i;

    if (argc < 3) @{
        fprintf(stderr,"Usage: call pythonfile funcname [args]\n");
        return 1;
    @}

    Py_Initialize();
    pName = PyUnicode_DecodeFSDefault(argv[1]);
    /* Error checking of pName left out */

    pModule = PyImport_Import(pName);
    Py_DECREF(pName);

    if (pModule != NULL) @{
        pFunc = PyObject_GetAttrString(pModule, argv[2]);
        /* pFunc is a new reference */

        if (pFunc && PyCallable_Check(pFunc)) @{
            pArgs = PyTuple_New(argc - 3);
            for (i = 0; i < argc - 3; ++i) @{
                pValue = PyLong_FromLong(atoi(argv[i + 3]));
                if (!pValue) @{
                    Py_DECREF(pArgs);
                    Py_DECREF(pModule);
                    fprintf(stderr, "Cannot convert argument\n");
                    return 1;
                @}
                /* pValue reference stolen here: */
                PyTuple_SetItem(pArgs, i, pValue);
            @}
            pValue = PyObject_CallObject(pFunc, pArgs);
            Py_DECREF(pArgs);
            if (pValue != NULL) @{
                printf("Result of call: %ld\n", PyLong_AsLong(pValue));
                Py_DECREF(pValue);
            @}
            else @{
                Py_DECREF(pFunc);
                Py_DECREF(pModule);
                PyErr_Print();
                fprintf(stderr,"Call failed\n");
                return 1;
            @}
        @}
        else @{
            if (PyErr_Occurred())
                PyErr_Print();
            fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]);
        @}
        Py_XDECREF(pFunc);
        Py_DECREF(pModule);
    @}
    else @{
        PyErr_Print();
        fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);
        return 1;
    @}
    if (Py_FinalizeEx() < 0) @{
        return 120;
    @}
    return 0;
@}
@end example

This code loads a Python script using @code{argv[1]}, and calls the function named
in @code{argv[2]}.  Its integer arguments are the other values of the @code{argv}
array.  If you @ref{40f3,,compile and link} this program (let’s call
the finished executable @code{call}), and use it to execute a Python
script, such as:

@example
def multiply(a,b):
    print("Will compute", a, "times", b)
    c = 0
    for i in range(0, a):
        c = c + b
    return c
@end example

then the result should be:

@example
$ call multiply multiply 3 2
Will compute 3 times 2
Result of call: 6
@end example

Although the program is quite large for its functionality, most of the code is
for data conversion between Python and C, and for error reporting.  The
interesting part with respect to embedding Python starts with

@example
Py_Initialize();
pName = PyUnicode_DecodeFSDefault(argv[1]);
/* Error checking of pName left out */
pModule = PyImport_Import(pName);
@end example

After initializing the interpreter, the script is loaded using
@ref{f5a,,PyImport_Import()}.  This routine needs a Python string as its argument,
which is constructed using the @ref{40f4,,PyUnicode_FromString()} data conversion
routine.

@example
pFunc = PyObject_GetAttrString(pModule, argv[2]);
/* pFunc is a new reference */

if (pFunc && PyCallable_Check(pFunc)) @{
    ...
@}
Py_XDECREF(pFunc);
@end example

Once the script is loaded, the name we’re looking for is retrieved using
@ref{408c,,PyObject_GetAttrString()}.  If the name exists, and the object returned is
callable, you can safely assume that it is a function.  The program then
proceeds by constructing a tuple of arguments as normal.  The call to the Python
function is then made with:

@example
pValue = PyObject_CallObject(pFunc, pArgs);
@end example

Upon return of the function, @code{pValue} is either @code{NULL} or it contains a
reference to the return value of the function.  Be sure to release the reference
after examining the value.

@node Extending Embedded Python,Embedding Python in C++,Pure Embedding,Embedding Python in Another Application
@anchor{extending/embedding extending-embedded-python}@anchor{40f5}@anchor{extending/embedding extending-with-embedding}@anchor{40f6}
@subsubsection Extending Embedded Python


Until now, the embedded Python interpreter had no access to functionality from
the application itself.  The Python API allows this by extending the embedded
interpreter.  That is, the embedded interpreter gets extended with routines
provided by the application. While it sounds complex, it is not so bad.  Simply
forget for a while that the application starts the Python interpreter.  Instead,
consider the application to be a set of subroutines, and write some glue code
that gives Python access to those routines, just like you would write a normal
Python extension.  For example:

@example
static int numargs=0;

/* Return the number of arguments of the application command line */
static PyObject*
emb_numargs(PyObject *self, PyObject *args)
@{
    if(!PyArg_ParseTuple(args, ":numargs"))
        return NULL;
    return PyLong_FromLong(numargs);
@}

static PyMethodDef EmbMethods[] = @{
    @{"numargs", emb_numargs, METH_VARARGS,
     "Return the number of arguments received by the process."@},
    @{NULL, NULL, 0, NULL@}
@};

static PyModuleDef EmbModule = @{
    PyModuleDef_HEAD_INIT, "emb", NULL, -1, EmbMethods,
    NULL, NULL, NULL, NULL
@};

static PyObject*
PyInit_emb(void)
@{
    return PyModule_Create(&EmbModule);
@}
@end example

Insert the above code just above the @code{main()} function. Also, insert the
following two statements before the call to @ref{38d,,Py_Initialize()}:

@example
numargs = argc;
PyImport_AppendInittab("emb", &PyInit_emb);
@end example

These two lines initialize the @code{numargs} variable, and make the
@code{emb.numargs()} function accessible to the embedded Python interpreter.
With these extensions, the Python script can do things like

@example
import emb
print("Number of arguments", emb.numargs())
@end example

In a real application, the methods will expose an API of the application to
Python.

@c TODO: threads, code examples do not really behave well if errors happen
@c (what to watch out for)

@node Embedding Python in C++,Compiling and Linking under Unix-like systems,Extending Embedded Python,Embedding Python in Another Application
@anchor{extending/embedding embedding-python-in-c}@anchor{40f7}@anchor{extending/embedding embeddingincplusplus}@anchor{40f8}
@subsubsection Embedding Python in C++


It is also possible to embed Python in a C++ program; precisely how this is done
will depend on the details of the C++ system used; in general you will need to
write the main program in C++, and use the C++ compiler to compile and link your
program.  There is no need to recompile Python itself using C++.

@node Compiling and Linking under Unix-like systems,,Embedding Python in C++,Embedding Python in Another Application
@anchor{extending/embedding compiling}@anchor{40f3}@anchor{extending/embedding compiling-and-linking-under-unix-like-systems}@anchor{40f9}
@subsubsection Compiling and Linking under Unix-like systems


It is not necessarily trivial to find the right flags to pass to your
compiler (and linker) in order to embed the Python interpreter into your
application, particularly because Python needs to load library modules
implemented as C dynamic extensions (@code{.so} files) linked against
it.

To find out the required compiler and linker flags, you can execute the
@code{python@emph{X.Y}-config} script which is generated as part of the
installation process (a @code{python3-config} script may also be
available).  This script has several options, of which the following will
be directly useful to you:


@itemize *

@item 
@code{pythonX.Y-config --cflags} will give you the recommended flags when
compiling:

@example
$ /opt/bin/python3.4-config --cflags
-I/opt/include/python3.4m -I/opt/include/python3.4m -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
@end example

@item 
@code{pythonX.Y-config --ldflags} will give you the recommended flags when
linking:

@example
$ /opt/bin/python3.4-config --ldflags
-L/opt/lib/python3.4/config-3.4m -lpthread -ldl -lutil -lm -lpython3.4m -Xlinker -export-dynamic
@end example
@end itemize

@cartouche
@quotation Note 
To avoid confusion between several Python installations (and especially
between the system Python and your own compiled Python), it is recommended
that you use the absolute path to @code{python@emph{X.Y}-config}, as in the above
example.
@end quotation
@end cartouche

If this procedure doesn’t work for you (it is not guaranteed to work for
all Unix-like platforms; however, we welcome @ref{40fa,,bug reports})
you will have to read your system’s documentation about dynamic linking and/or
examine Python’s @code{Makefile} (use @ref{3b87,,sysconfig.get_makefile_filename()}
to find its location) and compilation
options.  In this case, the @ref{f8,,sysconfig} module is a useful tool to
programmatically extract the configuration values that you will want to
combine together.  For example:

@example
>>> import sysconfig
>>> sysconfig.get_config_var('LIBS')
'-lpthread -ldl  -lutil'
>>> sysconfig.get_config_var('LINKFORSHARED')
'-Xlinker -export-dynamic'
@end example

@c XXX similar documentation for Windows missing

@node Python/C API Reference Manual,Distributing Python Modules,Extending and Embedding the Python Interpreter,Top
@anchor{c-api/index doc}@anchor{40fb}@anchor{c-api/index c-api-index}@anchor{14bf}@anchor{c-api/index python-c-api-reference-manual}@anchor{40fc}
@chapter Python/C API Reference Manual


This manual documents the API used by C and C++ programmers who want to write
extension modules or embed Python.  It is a companion to @ref{14be,,Extending and Embedding the Python Interpreter},
which describes the general principles of extension writing but does not
document the API functions in detail.

@menu
* Introduction: Introduction<13>. 
* C API Stability:: 
* The Very High Level Layer:: 
* Reference Counting:: 
* Exception Handling:: 
* Utilities: Utilities<2>. 
* Abstract Objects Layer:: 
* Concrete Objects Layer:: 
* Initialization@comma{} Finalization@comma{} and Threads: Initialization Finalization and Threads. 
* Python Initialization Configuration:: 
* Memory Management:: 
* Object Implementation Support:: 
* API and ABI Versioning:: 

@end menu

@node Introduction<13>,C API Stability,,Python/C API Reference Manual
@anchor{c-api/intro doc}@anchor{40fd}@anchor{c-api/intro api-intro}@anchor{40fe}@anchor{c-api/intro introduction}@anchor{40ff}
@section Introduction


The Application Programmer’s Interface to Python gives C and C++ programmers
access to the Python interpreter at a variety of levels.  The API is equally
usable from C++, but for brevity it is generally referred to as the Python/C
API.  There are two fundamentally different reasons for using the Python/C API.
The first reason is to write @emph{extension modules} for specific purposes; these
are C modules that extend the Python interpreter.  This is probably the most
common use.  The second reason is to use Python as a component in a larger
application; this technique is generally referred to as @emph{embedding} Python
in an application.

Writing an extension module is a relatively well-understood process, where a
“cookbook” approach works well.  There are several tools that automate the
process to some extent.  While people have embedded Python in other
applications since its early existence, the process of embedding Python is
less straightforward than writing an extension.

Many API functions are useful independent of whether you’re embedding  or
extending Python; moreover, most applications that embed Python  will need to
provide a custom extension as well, so it’s probably a  good idea to become
familiar with writing an extension before  attempting to embed Python in a real
application.

@menu
* Coding standards:: 
* Include Files:: 
* Useful macros:: 
* Objects@comma{} Types and Reference Counts: Objects Types and Reference Counts. 
* Exceptions: Exceptions<18>. 
* Embedding Python: Embedding Python<2>. 
* Debugging Builds:: 

@end menu

@node Coding standards,Include Files,,Introduction<13>
@anchor{c-api/intro coding-standards}@anchor{4100}
@subsection Coding standards


If you’re writing C code for inclusion in CPython, you @strong{must} follow the
guidelines and standards defined in 
@geindex Python Enhancement Proposals; PEP 7
PEP 7@footnote{https://www.python.org/dev/peps/pep-0007}.  These guidelines apply
regardless of the version of Python you are contributing to.  Following these
conventions is not necessary for your own third party extension modules,
unless you eventually expect to contribute them to Python.

@node Include Files,Useful macros,Coding standards,Introduction<13>
@anchor{c-api/intro api-includes}@anchor{1f5}@anchor{c-api/intro include-files}@anchor{4101}
@subsection Include Files


All function, type and macro definitions needed to use the Python/C API are
included in your code by the following line:

@example
#define PY_SSIZE_T_CLEAN
#include <Python.h>
@end example

This implies inclusion of the following standard headers: @code{<stdio.h>},
@code{<string.h>}, @code{<errno.h>}, @code{<limits.h>}, @code{<assert.h>} and @code{<stdlib.h>}
(if available).

@cartouche
@quotation Note 
Since Python may define some pre-processor definitions which affect the standard
headers on some systems, you @emph{must} include @code{Python.h} before any standard
headers are included.

It is recommended to always define @code{PY_SSIZE_T_CLEAN} before including
@code{Python.h}.  See @ref{381,,Parsing arguments and building values} for a description of this macro.
@end quotation
@end cartouche

All user visible names defined by Python.h (except those defined by the included
standard headers) have one of the prefixes @code{Py} or @code{_Py}.  Names beginning
with @code{_Py} are for internal use by the Python implementation and should not be
used by extension writers. Structure member names do not have a reserved prefix.

@cartouche
@quotation Note 
User code should never define names that begin with @code{Py} or @code{_Py}. This
confuses the reader, and jeopardizes the portability of the user code to
future Python versions, which may define additional names beginning with one
of these prefixes.
@end quotation
@end cartouche

The header files are typically installed with Python.  On Unix, these  are
located in the directories @code{@emph{prefix}/include/pythonversion/} and
@code{@emph{exec_prefix}/include/pythonversion/}, where 
@geindex prefix
@geindex environment variable; prefix
@code{prefix} and
@geindex exec_prefix
@geindex environment variable; exec_prefix
@code{exec_prefix} are defined by the corresponding parameters to Python’s
@code{configure} script and @emph{version} is
@code{'%d.%d' % sys.version_info[:2]}.  On Windows, the headers are installed
in @code{@emph{prefix}/include}, where 
@geindex prefix
@geindex environment variable; prefix
@code{prefix} is the installation
directory specified to the installer.

To include the headers, place both directories (if different) on your compiler’s
search path for includes.  Do @emph{not} place the parent directories on the search
path and then use @code{#include <pythonX.Y/Python.h>}; this will break on
multi-platform builds since the platform independent headers under
@geindex prefix
@geindex environment variable; prefix
@code{prefix} include the platform specific headers from
@geindex exec_prefix
@geindex environment variable; exec_prefix
@code{exec_prefix}.

C++ users should note that although the API is defined entirely using C, the
header files properly declare the entry points to be @code{extern "C"}. As a result,
there is no need to do anything special to use the API from C++.

@node Useful macros,Objects Types and Reference Counts,Include Files,Introduction<13>
@anchor{c-api/intro useful-macros}@anchor{4102}
@subsection Useful macros


Several useful macros are defined in the Python header files.  Many are
defined closer to where they are useful (e.g. @ref{fcf,,Py_RETURN_NONE}).
Others of a more general utility are defined here.  This is not necessarily a
complete listing.

@geindex Py_ABS (C macro)
@anchor{c-api/intro c Py_ABS}@anchor{4103}
@deffn {C Macro} Py_ABS (x)

Return the absolute value of @code{x}.


New in version 3.3.

@end deffn

@geindex Py_ALWAYS_INLINE (C macro)
@anchor{c-api/intro c Py_ALWAYS_INLINE}@anchor{1129}
@deffn {C Macro} Py_ALWAYS_INLINE

Ask the compiler to always inline a static inline function. The compiler can
ignore it and decides to not inline the function.

It can be used to inline performance critical static inline functions when
building Python in debug mode with function inlining disabled. For example,
MSC disables function inlining when building in debug mode.

Marking blindly a static inline function with Py_ALWAYS_INLINE can result in
worse performances (due to increased code size for example). The compiler is
usually smarter than the developer for the cost/benefit analysis.

If Python is @ref{338,,built in debug mode} (if the @code{Py_DEBUG}
macro is defined), the @ref{1129,,Py_ALWAYS_INLINE} macro does nothing.

It must be specified before the function return type. Usage:

@example
static inline Py_ALWAYS_INLINE int random(void) @{ return 4; @}
@end example


New in version 3.11.

@end deffn

@geindex Py_CHARMASK (C macro)
@anchor{c-api/intro c Py_CHARMASK}@anchor{4104}
@deffn {C Macro} Py_CHARMASK (c)

Argument must be a character or an integer in the range [-128, 127] or [0,
255].  This macro returns @code{c} cast to an @code{unsigned char}.
@end deffn

@geindex Py_DEPRECATED (C macro)
@anchor{c-api/intro c Py_DEPRECATED}@anchor{5c5}
@deffn {C Macro} Py_DEPRECATED (version)

Use this for deprecated declarations.  The macro must be placed before the
symbol name.

Example:

@example
Py_DEPRECATED(3.8) PyAPI_FUNC(int) Py_OldFunction(void);
@end example


Changed in version 3.8: MSVC support was added.

@end deffn

@geindex Py_GETENV (C macro)
@anchor{c-api/intro c Py_GETENV}@anchor{4105}
@deffn {C Macro} Py_GETENV (s)

Like @code{getenv(s)}, but returns @code{NULL} if @ref{460,,-E} was passed on the
command line (i.e. if @code{Py_IgnoreEnvironmentFlag} is set).
@end deffn

@geindex Py_MAX (C macro)
@anchor{c-api/intro c Py_MAX}@anchor{4106}
@deffn {C Macro} Py_MAX (x, y)

Return the maximum value between @code{x} and @code{y}.


New in version 3.3.

@end deffn

@geindex Py_MEMBER_SIZE (C macro)
@anchor{c-api/intro c Py_MEMBER_SIZE}@anchor{4107}
@deffn {C Macro} Py_MEMBER_SIZE (type, member)

Return the size of a structure (@code{type}) @code{member} in bytes.


New in version 3.6.

@end deffn

@geindex Py_MIN (C macro)
@anchor{c-api/intro c Py_MIN}@anchor{4108}
@deffn {C Macro} Py_MIN (x, y)

Return the minimum value between @code{x} and @code{y}.


New in version 3.3.

@end deffn

@geindex Py_NO_INLINE (C macro)
@anchor{c-api/intro c Py_NO_INLINE}@anchor{112a}
@deffn {C Macro} Py_NO_INLINE

Disable inlining on a function. For example, it reduces the C stack
consumption: useful on LTO+PGO builds which heavily inline code (see
bpo-33720@footnote{https://bugs.python.org/issue33720}).

Usage:

@example
Py_NO_INLINE static int random(void) @{ return 4; @}
@end example


New in version 3.11.

@end deffn

@geindex Py_STRINGIFY (C macro)
@anchor{c-api/intro c Py_STRINGIFY}@anchor{4109}
@deffn {C Macro} Py_STRINGIFY (x)

Convert @code{x} to a C string.  E.g. @code{Py_STRINGIFY(123)} returns
@code{"123"}.


New in version 3.4.

@end deffn

@geindex Py_UNREACHABLE (C macro)
@anchor{c-api/intro c Py_UNREACHABLE}@anchor{6e8}
@deffn {C Macro} Py_UNREACHABLE ()

Use this when you have a code path that cannot be reached by design.
For example, in the @code{default:} clause in a @code{switch} statement for which
all possible values are covered in @code{case} statements.  Use this in places
where you might be tempted to put an @code{assert(0)} or @code{abort()} call.

In release mode, the macro helps the compiler to optimize the code, and
avoids a warning about unreachable code.  For example, the macro is
implemented with @code{__builtin_unreachable()} on GCC in release mode.

A use for @code{Py_UNREACHABLE()} is following a call a function that
never returns but that is not declared @code{_Py_NO_RETURN}.

If a code path is very unlikely code but can be reached under exceptional
case, this macro must not be used.  For example, under low memory condition
or if a system call returns a value out of the expected range.  In this
case, it’s better to report the error to the caller.  If the error cannot
be reported to caller, @ref{48f,,Py_FatalError()} can be used.


New in version 3.7.

@end deffn

@geindex Py_UNUSED (C macro)
@anchor{c-api/intro c Py_UNUSED}@anchor{410a}
@deffn {C Macro} Py_UNUSED (arg)

Use this for unused arguments in a function definition to silence compiler
warnings. Example: @code{int func(int a, int Py_UNUSED(b)) @{ return a; @}}.


New in version 3.4.

@end deffn

@geindex PyDoc_STRVAR (C macro)
@anchor{c-api/intro c PyDoc_STRVAR}@anchor{1200}
@deffn {C Macro} PyDoc_STRVAR (name, str)

Creates a variable with name @code{name} that can be used in docstrings.
If Python is built without docstrings, the value will be empty.

Use @ref{1200,,PyDoc_STRVAR} for docstrings to support building
Python without docstrings, as specified in 
@geindex Python Enhancement Proposals; PEP 7
PEP 7@footnote{https://www.python.org/dev/peps/pep-0007}.

Example:

@example
PyDoc_STRVAR(pop_doc, "Remove and return the rightmost element.");

static PyMethodDef deque_methods[] = @{
    // ...
    @{"pop", (PyCFunction)deque_pop, METH_NOARGS, pop_doc@},
    // ...
@}
@end example
@end deffn

@geindex PyDoc_STR (C macro)
@anchor{c-api/intro c PyDoc_STR}@anchor{410b}
@deffn {C Macro} PyDoc_STR (str)

Creates a docstring for the given input string or an empty string
if docstrings are disabled.

Use @ref{410b,,PyDoc_STR} in specifying docstrings to support
building Python without docstrings, as specified in 
@geindex Python Enhancement Proposals; PEP 7
PEP 7@footnote{https://www.python.org/dev/peps/pep-0007}.

Example:

@example
static PyMethodDef pysqlite_row_methods[] = @{
    @{"keys", (PyCFunction)pysqlite_row_keys, METH_NOARGS,
        PyDoc_STR("Returns the keys of the row.")@},
    @{NULL, NULL@}
@};
@end example
@end deffn

@node Objects Types and Reference Counts,Exceptions<18>,Useful macros,Introduction<13>
@anchor{c-api/intro api-objects}@anchor{410c}@anchor{c-api/intro objects-types-and-reference-counts}@anchor{410d}
@subsection Objects, Types and Reference Counts


@geindex object; type

Most Python/C API functions have one or more arguments as well as a return value
of type @ref{375,,PyObject}*.  This type is a pointer to an opaque data type
representing an arbitrary Python object.  Since all Python object types are
treated the same way by the Python language in most situations (e.g.,
assignments, scope rules, and argument passing), it is only fitting that they
should be represented by a single C type.  Almost all Python objects live on the
heap: you never declare an automatic or static variable of type
@ref{375,,PyObject}, only pointer variables of type @ref{375,,PyObject}* can  be
declared.  The sole exception are the type objects; since these must never be
deallocated, they are typically static @ref{5c6,,PyTypeObject} objects.

All Python objects (even Python integers) have a @emph{type} and a
@emph{reference count}.  An object’s type determines what kind of object it is
(e.g., an integer, a list, or a user-defined function; there are many more as
explained in @ref{1787,,The standard type hierarchy}).  For each of the well-known types there is a macro
to check whether an object is of that type; for instance, @code{PyList_Check(a)} is
true if (and only if) the object pointed to by @emph{a} is a Python list.

@menu
* Reference Counts: Reference Counts<2>. 
* Types:: 

@end menu

@node Reference Counts<2>,Types,,Objects Types and Reference Counts
@anchor{c-api/intro api-refcounts}@anchor{410e}@anchor{c-api/intro reference-counts}@anchor{410f}
@subsubsection Reference Counts


The reference count is important because today’s computers have a  finite (and
often severely limited) memory size; it counts how many  different places there
are that have a reference to an object.  Such a  place could be another object,
or a global (or static) C variable, or  a local variable in some C function.
When an object’s reference count  becomes zero, the object is deallocated.  If
it contains references to  other objects, their reference count is decremented.
Those other  objects may be deallocated in turn, if this decrement makes their
reference count become zero, and so on.  (There’s an obvious problem  with
objects that reference each other here; for now, the solution is  “don’t do
that.”)

@geindex Py_INCREF()
@geindex Py_DECREF()

Reference counts are always manipulated explicitly.  The normal way is  to use
the macro @ref{372,,Py_INCREF()} to increment an object’s reference count by one,
and @ref{373,,Py_DECREF()} to decrement it by   one.  The @ref{373,,Py_DECREF()} macro
is considerably more complex than the incref one, since it must check whether
the reference count becomes zero and then cause the object’s deallocator to be
called. The deallocator is a function pointer contained in the object’s type
structure.  The type-specific deallocator takes care of decrementing the
reference counts for other objects contained in the object if this is a compound
object type, such as a list, as well as performing any additional finalization
that’s needed.  There’s no chance that the reference count can overflow; at
least as many bits are used to hold the reference count as there are distinct
memory locations in virtual memory (assuming @code{sizeof(Py_ssize_t) >= sizeof(void*)}).
Thus, the reference count increment is a simple operation.

It is not necessary to increment an object’s reference count for every  local
variable that contains a pointer to an object.  In theory, the  object’s
reference count goes up by one when the variable is made to  point to it and it
goes down by one when the variable goes out of  scope.  However, these two
cancel each other out, so at the end the  reference count hasn’t changed.  The
only real reason to use the  reference count is to prevent the object from being
deallocated as  long as our variable is pointing to it.  If we know that there
is at  least one other reference to the object that lives at least as long as
our variable, there is no need to increment the reference count  temporarily.
An important situation where this arises is in objects  that are passed as
arguments to C functions in an extension module  that are called from Python;
the call mechanism guarantees to hold a  reference to every argument for the
duration of the call.

However, a common pitfall is to extract an object from a list and hold on to it
for a while without incrementing its reference count. Some other operation might
conceivably remove the object from the list, decrementing its reference count
and possibly deallocating it. The real danger is that innocent-looking
operations may invoke arbitrary Python code which could do this; there is a code
path which allows control to flow back to the user from a @ref{373,,Py_DECREF()}, so
almost any operation is potentially dangerous.

A safe approach is to always use the generic operations (functions  whose name
begins with @code{PyObject_}, @code{PyNumber_}, @code{PySequence_} or @code{PyMapping_}).
These operations always increment the reference count of the object they return.
This leaves the caller with the responsibility to call @ref{373,,Py_DECREF()} when
they are done with the result; this soon becomes second nature.

@menu
* Reference Count Details:: 

@end menu

@node Reference Count Details,,,Reference Counts<2>
@anchor{c-api/intro api-refcountdetails}@anchor{4110}@anchor{c-api/intro reference-count-details}@anchor{4111}
@subsubsection Reference Count Details


The reference count behavior of functions in the Python/C API is best  explained
in terms of @emph{ownership of references}.  Ownership pertains to references, never
to objects (objects are not owned: they are always shared).  “Owning a
reference” means being responsible for calling Py_DECREF on it when the
reference is no longer needed.  Ownership can also be transferred, meaning that
the code that receives ownership of the reference then becomes responsible for
eventually decref’ing it by calling @ref{373,,Py_DECREF()} or @ref{1fd,,Py_XDECREF()}
when it’s no longer needed—or passing on this responsibility (usually to its
caller). When a function passes ownership of a reference on to its caller, the
caller is said to receive a @emph{new} reference.  When no ownership is transferred,
the caller is said to @emph{borrow} the reference. Nothing needs to be done for a
@ref{4112,,borrowed reference}.

Conversely, when a calling function passes in a reference to an  object, there
are two possibilities: the function @emph{steals} a  reference to the object, or it
does not.  @emph{Stealing a reference} means that when you pass a reference to a
function, that function assumes that it now owns that reference, and you are not
responsible for it any longer.

@geindex PyList_SetItem()
@geindex PyTuple_SetItem()

Few functions steal references; the two notable exceptions are
@ref{4092,,PyList_SetItem()} and @ref{4091,,PyTuple_SetItem()}, which  steal a reference
to the item (but not to the tuple or list into which the item is put!).  These
functions were designed to steal a reference because of a common idiom for
populating a tuple or list with newly created objects; for example, the code to
create the tuple @code{(1, 2, "three")} could look like this (forgetting about
error handling for the moment; a better way to code this is shown below):

@example
PyObject *t;

t = PyTuple_New(3);
PyTuple_SetItem(t, 0, PyLong_FromLong(1L));
PyTuple_SetItem(t, 1, PyLong_FromLong(2L));
PyTuple_SetItem(t, 2, PyUnicode_FromString("three"));
@end example

Here, @ref{1085,,PyLong_FromLong()} returns a new reference which is immediately
stolen by @ref{4091,,PyTuple_SetItem()}.  When you want to keep using an object
although the reference to it will be stolen, use @ref{372,,Py_INCREF()} to grab
another reference before calling the reference-stealing function.

Incidentally, @ref{4091,,PyTuple_SetItem()} is the @emph{only} way to set tuple items;
@ref{1312,,PySequence_SetItem()} and @ref{4113,,PyObject_SetItem()} refuse to do this
since tuples are an immutable data type.  You should only use
@ref{4091,,PyTuple_SetItem()} for tuples that you are creating yourself.

Equivalent code for populating a list can be written using @ref{4114,,PyList_New()}
and @ref{4092,,PyList_SetItem()}.

However, in practice, you will rarely use these ways of creating and populating
a tuple or list.  There’s a generic function, @ref{380,,Py_BuildValue()}, that can
create most common objects from C values, directed by a @emph{format string}.
For example, the above two blocks of code could be replaced by the following
(which also takes care of the error checking):

@example
PyObject *tuple, *list;

tuple = Py_BuildValue("(iis)", 1, 2, "three");
list = Py_BuildValue("[iis]", 1, 2, "three");
@end example

It is much more common to use @ref{4113,,PyObject_SetItem()} and friends with items
whose references you are only borrowing, like arguments that were passed in to
the function you are writing.  In that case, their behaviour regarding reference
counts is much saner, since you don’t have to increment a reference count so you
can give a reference away (“have it be stolen”).  For example, this function
sets all items of a list (actually, any mutable sequence) to a given item:

@example
int
set_all(PyObject *target, PyObject *item)
@{
    Py_ssize_t i, n;

    n = PyObject_Length(target);
    if (n < 0)
        return -1;
    for (i = 0; i < n; i++) @{
        PyObject *index = PyLong_FromSsize_t(i);
        if (!index)
            return -1;
        if (PyObject_SetItem(target, index, item) < 0) @{
            Py_DECREF(index);
            return -1;
        @}
        Py_DECREF(index);
    @}
    return 0;
@}
@end example

@geindex set_all()

The situation is slightly different for function return values.   While passing
a reference to most functions does not change your  ownership responsibilities
for that reference, many functions that  return a reference to an object give
you ownership of the reference. The reason is simple: in many cases, the
returned object is created  on the fly, and the reference you get is the only
reference to the  object.  Therefore, the generic functions that return object
references, like @ref{4115,,PyObject_GetItem()} and  @ref{1311,,PySequence_GetItem()},
always return a new reference (the caller becomes the owner of the reference).

It is important to realize that whether you own a reference returned  by a
function depends on which function you call only — @emph{the plumage} (the type of
the object passed as an argument to the function) @emph{doesn’t enter into it!}
Thus, if you  extract an item from a list using @ref{408e,,PyList_GetItem()}, you
don’t own the reference — but if you obtain the same item from the same list
using @ref{1311,,PySequence_GetItem()} (which happens to take exactly the same
arguments), you do own a reference to the returned object.

@geindex PyList_GetItem()
@geindex PySequence_GetItem()

Here is an example of how you could write a function that computes the sum of
the items in a list of integers; once using  @ref{408e,,PyList_GetItem()}, and once
using @ref{1311,,PySequence_GetItem()}.

@example
long
sum_list(PyObject *list)
@{
    Py_ssize_t i, n;
    long total = 0, value;
    PyObject *item;

    n = PyList_Size(list);
    if (n < 0)
        return -1; /* Not a list */
    for (i = 0; i < n; i++) @{
        item = PyList_GetItem(list, i); /* Can't fail */
        if (!PyLong_Check(item)) continue; /* Skip non-integers */
        value = PyLong_AsLong(item);
        if (value == -1 && PyErr_Occurred())
            /* Integer too big to fit in a C long, bail out */
            return -1;
        total += value;
    @}
    return total;
@}
@end example

@geindex sum_list()

@example
long
sum_sequence(PyObject *sequence)
@{
    Py_ssize_t i, n;
    long total = 0, value;
    PyObject *item;
    n = PySequence_Length(sequence);
    if (n < 0)
        return -1; /* Has no length */
    for (i = 0; i < n; i++) @{
        item = PySequence_GetItem(sequence, i);
        if (item == NULL)
            return -1; /* Not a sequence, or other failure */
        if (PyLong_Check(item)) @{
            value = PyLong_AsLong(item);
            Py_DECREF(item);
            if (value == -1 && PyErr_Occurred())
                /* Integer too big to fit in a C long, bail out */
                return -1;
            total += value;
        @}
        else @{
            Py_DECREF(item); /* Discard reference ownership */
        @}
    @}
    return total;
@}
@end example

@geindex sum_sequence()

@node Types,,Reference Counts<2>,Objects Types and Reference Counts
@anchor{c-api/intro api-types}@anchor{4116}@anchor{c-api/intro types}@anchor{4117}
@subsubsection Types


There are few other data types that play a significant role in  the Python/C
API; most are simple C types such as int,  long,
double and char*.  A few structure types  are used to
describe static tables used to list the functions exported  by a module or the
data attributes of a new object type, and another is used to describe the value
of a complex number.  These will  be discussed together with the functions that
use them.

@node Exceptions<18>,Embedding Python<2>,Objects Types and Reference Counts,Introduction<13>
@anchor{c-api/intro api-exceptions}@anchor{4118}@anchor{c-api/intro exceptions}@anchor{4119}
@subsection Exceptions


The Python programmer only needs to deal with exceptions if specific  error
handling is required; unhandled exceptions are automatically  propagated to the
caller, then to the caller’s caller, and so on, until they reach the top-level
interpreter, where they are reported to the  user accompanied by a stack
traceback.

@geindex PyErr_Occurred()

For C programmers, however, error checking always has to be explicit.  All
functions in the Python/C API can raise exceptions, unless an explicit claim is
made otherwise in a function’s documentation.  In general, when a function
encounters an error, it sets an exception, discards any object references that
it owns, and returns an error indicator.  If not documented otherwise, this
indicator is either @code{NULL} or @code{-1}, depending on the function’s return type.
A few functions return a Boolean true/false result, with false indicating an
error.  Very few functions return no explicit error indicator or have an
ambiguous return value, and require explicit testing for errors with
@ref{116a,,PyErr_Occurred()}.  These exceptions are always explicitly documented.

@geindex PyErr_SetString()
@geindex PyErr_Clear()

Exception state is maintained in per-thread storage (this is  equivalent to
using global storage in an unthreaded application).  A  thread can be in one of
two states: an exception has occurred, or not. The function
@ref{116a,,PyErr_Occurred()} can be used to check for this: it returns a borrowed
reference to the exception type object when an exception has occurred, and
@code{NULL} otherwise.  There are a number of functions to set the exception state:
@ref{4071,,PyErr_SetString()} is the most common (though not the most general)
function to set the exception state, and @ref{bc3,,PyErr_Clear()} clears the
exception state.

The full exception state consists of three objects (all of which can  be
@code{NULL}): the exception type, the corresponding exception  value, and the
traceback.  These have the same meanings as the Python result of
@code{sys.exc_info()}; however, they are not the same: the Python objects represent
the last exception being handled by a Python  @ref{f6d,,try} …
@ref{16e,,except} statement, while the C level exception state only exists while
an exception is being passed on between C functions until it reaches the Python
bytecode interpreter’s  main loop, which takes care of transferring it to
@code{sys.exc_info()} and friends.

@geindex exc_info() (in module sys)

Note that starting with Python 1.5, the preferred, thread-safe way to access the
exception state from Python code is to call the function @ref{18f,,sys.exc_info()},
which returns the per-thread exception state for Python code.  Also, the
semantics of both ways to access the exception state have changed so that a
function which catches an exception will save and restore its thread’s exception
state so as to preserve the exception state of its caller.  This prevents common
bugs in exception handling code caused by an innocent-looking function
overwriting the exception being handled; it also reduces the often unwanted
lifetime extension for objects that are referenced by the stack frames in the
traceback.

As a general principle, a function that calls another function to  perform some
task should check whether the called function raised an  exception, and if so,
pass the exception state on to its caller.  It  should discard any object
references that it owns, and return an  error indicator, but it should @emph{not} set
another exception — that would overwrite the exception that was just raised,
and lose important information about the exact cause of the error.

@geindex sum_sequence()

A simple example of detecting exceptions and passing them on is shown in the
@code{sum_sequence()} example above.  It so happens that this example doesn’t
need to clean up any owned references when it detects an error.  The following
example function shows some error cleanup.  First, to remind you why you like
Python, we show the equivalent Python code:

@example
def incr_item(dict, key):
    try:
        item = dict[key]
    except KeyError:
        item = 0
    dict[key] = item + 1
@end example

@geindex incr_item()

Here is the corresponding C code, in all its glory:

@example
int
incr_item(PyObject *dict, PyObject *key)
@{
    /* Objects all initialized to NULL for Py_XDECREF */
    PyObject *item = NULL, *const_one = NULL, *incremented_item = NULL;
    int rv = -1; /* Return value initialized to -1 (failure) */

    item = PyObject_GetItem(dict, key);
    if (item == NULL) @{
        /* Handle KeyError only: */
        if (!PyErr_ExceptionMatches(PyExc_KeyError))
            goto error;

        /* Clear the error and use zero: */
        PyErr_Clear();
        item = PyLong_FromLong(0L);
        if (item == NULL)
            goto error;
    @}
    const_one = PyLong_FromLong(1L);
    if (const_one == NULL)
        goto error;

    incremented_item = PyNumber_Add(item, const_one);
    if (incremented_item == NULL)
        goto error;

    if (PyObject_SetItem(dict, key, incremented_item) < 0)
        goto error;
    rv = 0; /* Success */
    /* Continue with cleanup code */

 error:
    /* Cleanup code, shared by success and failure path */

    /* Use Py_XDECREF() to ignore NULL references */
    Py_XDECREF(item);
    Py_XDECREF(const_one);
    Py_XDECREF(incremented_item);

    return rv; /* -1 for error, 0 for success */
@}
@end example

@geindex incr_item()

@geindex PyErr_ExceptionMatches()
@geindex PyErr_Clear()
@geindex Py_XDECREF()

This example represents an endorsed use of the @code{goto} statement  in C!
It illustrates the use of @ref{411a,,PyErr_ExceptionMatches()} and
@ref{bc3,,PyErr_Clear()} to handle specific exceptions, and the use of
@ref{1fd,,Py_XDECREF()} to dispose of owned references that may be @code{NULL} (note the
@code{'X'} in the name; @ref{373,,Py_DECREF()} would crash when confronted with a
@code{NULL} reference).  It is important that the variables used to hold owned
references are initialized to @code{NULL} for this to work; likewise, the proposed
return value is initialized to @code{-1} (failure) and only set to success after
the final call made is successful.

@node Embedding Python<2>,Debugging Builds,Exceptions<18>,Introduction<13>
@anchor{c-api/intro api-embedding}@anchor{411b}@anchor{c-api/intro embedding-python}@anchor{411c}
@subsection Embedding Python


The one important task that only embedders (as opposed to extension writers) of
the Python interpreter have to worry about is the initialization, and possibly
the finalization, of the Python interpreter.  Most functionality of the
interpreter can only be used after the interpreter has been initialized.

@geindex Py_Initialize()
@geindex module; builtins
@geindex module; __main__
@geindex module; sys
@geindex module; search; path
@geindex path (in module sys)

The basic initialization function is @ref{38d,,Py_Initialize()}. This initializes
the table of loaded modules, and creates the fundamental modules
@ref{11,,builtins}, @ref{1,,__main__}, and @ref{f7,,sys}.  It also
initializes the module search path (@code{sys.path}).

@geindex PySys_SetArgvEx()

@ref{38d,,Py_Initialize()} does not set the “script argument list”  (@code{sys.argv}).
If this variable is needed by Python code that will be executed later, it must
be set explicitly with a call to  @code{PySys_SetArgvEx(argc, argv, updatepath)}
after the call to @ref{38d,,Py_Initialize()}.

On most systems (in particular, on Unix and Windows, although the details are
slightly different), @ref{38d,,Py_Initialize()} calculates the module search path
based upon its best guess for the location of the standard Python interpreter
executable, assuming that the Python library is found in a fixed location
relative to the Python interpreter executable.  In particular, it looks for a
directory named @code{lib/python@emph{X.Y}} relative to the parent directory
where the executable named @code{python} is found on the shell command search
path (the environment variable 
@geindex PATH
@geindex environment variable; PATH
@code{PATH}).

For instance, if the Python executable is found in
@code{/usr/local/bin/python}, it will assume that the libraries are in
@code{/usr/local/lib/python@emph{X.Y}}.  (In fact, this particular path is also
the “fallback” location, used when no executable file named @code{python} is
found along 
@geindex PATH
@geindex environment variable; PATH
@code{PATH}.)  The user can override this behavior by setting the
environment variable 
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME}, or insert additional directories in
front of the standard path by setting 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH}.

@geindex Py_SetProgramName()
@geindex Py_GetPath()
@geindex Py_GetPrefix()
@geindex Py_GetExecPrefix()
@geindex Py_GetProgramFullPath()

The embedding application can steer the search by calling
@code{Py_SetProgramName(file)} @emph{before} calling  @ref{38d,,Py_Initialize()}.  Note that
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME} still overrides this and 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} is still
inserted in front of the standard path.  An application that requires total
control has to provide its own implementation of @ref{387,,Py_GetPath()},
@ref{388,,Py_GetPrefix()}, @ref{389,,Py_GetExecPrefix()}, and
@ref{38a,,Py_GetProgramFullPath()} (all defined in @code{Modules/getpath.c}).

@geindex Py_IsInitialized()

Sometimes, it is desirable to “uninitialize” Python.  For instance,  the
application may want to start over (make another call to
@ref{38d,,Py_Initialize()}) or the application is simply done with its  use of
Python and wants to free memory allocated by Python.  This can be accomplished
by calling @ref{862,,Py_FinalizeEx()}.  The function @ref{411d,,Py_IsInitialized()} returns
true if Python is currently in the initialized state.  More information about
these functions is given in a later chapter. Notice that @ref{862,,Py_FinalizeEx()}
does @emph{not} free all memory allocated by the Python interpreter, e.g. memory
allocated by extension modules currently cannot be released.

@node Debugging Builds,,Embedding Python<2>,Introduction<13>
@anchor{c-api/intro api-debugging}@anchor{411e}@anchor{c-api/intro debugging-builds}@anchor{411f}
@subsection Debugging Builds


Python can be built with several macros to enable extra checks of the
interpreter and extension modules.  These checks tend to add a large amount of
overhead to the runtime so they are not enabled by default.

A full list of the various types of debugging builds is in the file
@code{Misc/SpecialBuilds.txt} in the Python source distribution. Builds are
available that support tracing of reference counts, debugging the memory
allocator, or low-level profiling of the main interpreter loop.  Only the most
frequently-used builds will be described in the remainder of this section.

Compiling the interpreter with the @code{Py_DEBUG} macro defined produces
what is generally meant by @ref{338,,a debug build of Python}.
@code{Py_DEBUG} is enabled in the Unix build by adding
@ref{165a,,--with-pydebug} to the @code{./configure} command.
It is also implied by the presence of the
not-Python-specific @code{_DEBUG} macro.  When @code{Py_DEBUG} is enabled
in the Unix build, compiler optimization is disabled.

In addition to the reference count debugging described below, extra checks are
performed, see @ref{338,,Python Debug Build}.

Defining @code{Py_TRACE_REFS} enables reference tracing
(see the @ref{376,,configure --with-trace-refs option}).
When defined, a circular doubly linked list of active objects is maintained by adding two extra
fields to every @ref{375,,PyObject}.  Total allocations are tracked as well.  Upon
exit, all existing references are printed.  (In interactive mode this happens
after every statement run by the interpreter.)

Please refer to @code{Misc/SpecialBuilds.txt} in the Python source distribution
for more detailed information.

@node C API Stability,The Very High Level Layer,Introduction<13>,Python/C API Reference Manual
@anchor{c-api/stable doc}@anchor{4120}@anchor{c-api/stable c-api-stability}@anchor{4121}@anchor{c-api/stable stable}@anchor{35d}
@section C API Stability


Python’s C API is covered by the Backwards Compatibility Policy, 
@geindex Python Enhancement Proposals; PEP 387
PEP 387@footnote{https://www.python.org/dev/peps/pep-0387}.
While the C API will change with every minor release (e.g. from 3.9 to 3.10),
most changes will be source-compatible, typically by only adding new API.
Changing existing API or removing API is only done after a deprecation period
or to fix serious issues.

CPython’s Application Binary Interface (ABI) is forward- and
backwards-compatible across a minor release (if these are compiled the same
way; see @ref{4122,,Platform Considerations} below).
So, code compiled for Python 3.10.0 will work on 3.10.8 and vice versa,
but will need to be compiled separately for 3.9.x and 3.10.x.

Names prefixed by an underscore, such as @code{_Py_InternalState},
are private API that can change without notice even in patch releases.

@menu
* Stable Application Binary Interface:: 
* Platform Considerations:: 
* Contents of Limited API:: 

@end menu

@node Stable Application Binary Interface,Platform Considerations,,C API Stability
@anchor{c-api/stable stable-application-binary-interface}@anchor{4123}
@subsection Stable Application Binary Interface


Python 3.2 introduced the @emph{Limited API}, a subset of Python’s C API.
Extensions that only use the Limited API can be
compiled once and work with multiple versions of Python.
Contents of the Limited API are @ref{210,,listed below}.

To enable this, Python provides a @emph{Stable ABI}: a set of symbols that will
remain compatible across Python 3.x versions. The Stable ABI contains symbols
exposed in the Limited API, but also other ones – for example, functions
necessary to support older versions of the Limited API.

(For simplicity, this document talks about @emph{extensions}, but the Limited API
and Stable ABI work the same way for all uses of the API – for example,
embedding Python.)

@geindex Py_LIMITED_API (C macro)
@anchor{c-api/stable c Py_LIMITED_API}@anchor{20b}
@deffn {C Macro} Py_LIMITED_API

Define this macro before including @code{Python.h} to opt in to only use
the Limited API, and to select the Limited API version.

Define @code{Py_LIMITED_API} to the value of @ref{1da,,PY_VERSION_HEX}
corresponding to the lowest Python version your extension supports.
The extension will work without recompilation with all Python 3 releases
from the specified one onward, and can use Limited API introduced up to that
version.

Rather than using the @code{PY_VERSION_HEX} macro directly, hardcode a minimum
minor version (e.g. @code{0x030A0000} for Python 3.10) for stability when
compiling with future Python versions.

You can also define @code{Py_LIMITED_API} to @code{3}. This works the same as
@code{0x03020000} (Python 3.2, the version that introduced Limited API).
@end deffn

On Windows, extensions that use the Stable ABI should be linked against
@code{python3.dll} rather than a version-specific library such as
@code{python39.dll}.

On some platforms, Python will look for and load shared library files named
with the @code{abi3} tag (e.g. @code{mymodule.abi3.so}).
It does not check if such extensions conform to a Stable ABI.
The user (or their packaging tools) need to ensure that, for example,
extensions built with the 3.10+ Limited API are not installed for lower
versions of Python.

All functions in the Stable ABI are present as functions in Python’s shared
library, not solely as macros. This makes them usable from languages that don’t
use the C preprocessor.

@menu
* Limited API Scope and Performance:: 
* Limited API Caveats:: 

@end menu

@node Limited API Scope and Performance,Limited API Caveats,,Stable Application Binary Interface
@anchor{c-api/stable limited-api-scope-and-performance}@anchor{4124}
@subsubsection Limited API Scope and Performance


The goal for the Limited API is to allow everything that is possible with the
full C API, but possibly with a performance penalty.

For example, while @ref{408e,,PyList_GetItem()} is available, its “unsafe” macro
variant @ref{4125,,PyList_GET_ITEM()} is not.
The macro can be faster because it can rely on version-specific implementation
details of the list object.

Without @code{Py_LIMITED_API} defined, some C API functions are inlined or
replaced by macros.
Defining @code{Py_LIMITED_API} disables this inlining, allowing stability as
Python’s data structures are improved, but possibly reducing performance.

By leaving out the @code{Py_LIMITED_API} definition, it is possible to compile
a Limited API extension with a version-specific ABI. This can improve
performance for that Python version, but will limit compatibility.
Compiling with @code{Py_LIMITED_API} will then yield an extension that can be
distributed where a version-specific one is not available – for example,
for prereleases of an upcoming Python version.

@node Limited API Caveats,,Limited API Scope and Performance,Stable Application Binary Interface
@anchor{c-api/stable limited-api-caveats}@anchor{4126}
@subsubsection Limited API Caveats


Note that compiling with @code{Py_LIMITED_API} is @emph{not} a complete guarantee that
code conforms to the Limited API or the Stable ABI. @code{Py_LIMITED_API} only
covers definitions, but an API also includes other issues, such as expected
semantics.

One issue that @code{Py_LIMITED_API} does not guard against is calling a function
with arguments that are invalid in a lower Python version.
For example, consider a function that starts accepting @code{NULL} for an
argument. In Python 3.9, @code{NULL} now selects a default behavior, but in
Python 3.8, the argument will be used directly, causing a @code{NULL} dereference
and crash. A similar argument works for fields of structs.

Another issue is that some struct fields are currently not hidden when
@code{Py_LIMITED_API} is defined, even though they’re part of the Limited API.

For these reasons, we recommend testing an extension with @emph{all} minor Python
versions it supports, and preferably to build with the @emph{lowest} such version.

We also recommend reviewing documentation of all used API to check
if it is explicitly part of the Limited API. Even with @code{Py_LIMITED_API}
defined, a few private declarations are exposed for technical reasons (or
even unintentionally, as bugs).

Also note that the Limited API is not necessarily stable: compiling with
@code{Py_LIMITED_API} with Python 3.8 means that the extension will
run with Python 3.12, but it will not necessarily @emph{compile} with Python 3.12.
In particular, parts of the Limited API may be deprecated and removed,
provided that the Stable ABI stays stable.

@node Platform Considerations,Contents of Limited API,Stable Application Binary Interface,C API Stability
@anchor{c-api/stable platform-considerations}@anchor{4127}@anchor{c-api/stable stable-abi-platform}@anchor{4122}
@subsection Platform Considerations


ABI stability depends not only on Python, but also on the compiler used,
lower-level libraries and compiler options. For the purposes of the Stable ABI,
these details define a “platform”. They usually depend on the OS
type and processor architecture

It is the responsibility of each particular distributor of Python
to ensure that all Python versions on a particular platform are built
in a way that does not break the Stable ABI.
This is the case with Windows and macOS releases from @code{python.org} and many
third-party distributors.

@node Contents of Limited API,,Platform Considerations,C API Stability
@anchor{c-api/stable contents-of-limited-api}@anchor{4128}@anchor{c-api/stable stable-abi-list}@anchor{210}
@subsection Contents of Limited API


Currently, the Limited API includes the following items:



@itemize *

@item 
@ref{4129,,PyAIter_Check()}

@item 
@ref{1286,,PyArg_Parse()}

@item 
@ref{37f,,PyArg_ParseTuple()}

@item 
@ref{863,,PyArg_ParseTupleAndKeywords()}

@item 
@ref{103c,,PyArg_UnpackTuple()}

@item 
@ref{412a,,PyArg_VaParse()}

@item 
@ref{fd3,,PyArg_VaParseTupleAndKeywords()}

@item 
@ref{412b,,PyArg_ValidateKeywordArguments()}

@item 
@code{PyBaseObject_Type}

@item 
@ref{412c,,PyBool_FromLong()}

@item 
@code{PyBool_Type}

@item 
@ref{1e3,,PyBuffer_FillContiguousStrides()}

@item 
@ref{1e4,,PyBuffer_FillInfo()}

@item 
@ref{1e1,,PyBuffer_FromContiguous()}

@item 
@ref{1de,,PyBuffer_GetPointer()}

@item 
@ref{1e2,,PyBuffer_IsContiguous()}

@item 
@ref{1e5,,PyBuffer_Release()}

@item 
@ref{1df,,PyBuffer_SizeFromFormat()}

@item 
@ref{1e0,,PyBuffer_ToContiguous()}

@item 
@code{PyByteArrayIter_Type}

@item 
@ref{412d,,PyByteArray_AsString()}

@item 
@ref{412e,,PyByteArray_Concat()}

@item 
@ref{f20,,PyByteArray_FromObject()}

@item 
@ref{f21,,PyByteArray_FromStringAndSize()}

@item 
@ref{412f,,PyByteArray_Resize()}

@item 
@ref{4130,,PyByteArray_Size()}

@item 
@ref{4131,,PyByteArray_Type}

@item 
@code{PyBytesIter_Type}

@item 
@ref{351,,PyBytes_AsString()}

@item 
@ref{4132,,PyBytes_AsStringAndSize()}

@item 
@ref{4133,,PyBytes_Concat()}

@item 
@ref{4134,,PyBytes_ConcatAndDel()}

@item 
@code{PyBytes_DecodeEscape()}

@item 
@ref{12d5,,PyBytes_FromFormat()}

@item 
@ref{4135,,PyBytes_FromFormatV()}

@item 
@ref{4136,,PyBytes_FromObject()}

@item 
@ref{4137,,PyBytes_FromString()}

@item 
@ref{f1f,,PyBytes_FromStringAndSize()}

@item 
@code{PyBytes_Repr()}

@item 
@ref{4138,,PyBytes_Size()}

@item 
@ref{4139,,PyBytes_Type}

@item 
@ref{fd4,,PyCFunction}

@item 
@ref{413a,,PyCFunctionWithKeywords}

@item 
@code{PyCFunction_Call()}

@item 
@code{PyCFunction_GetFlags()}

@item 
@code{PyCFunction_GetFunction()}

@item 
@code{PyCFunction_GetSelf()}

@item 
@code{PyCFunction_New()}

@item 
@code{PyCFunction_NewEx()}

@item 
@code{PyCFunction_Type}

@item 
@code{PyCMethod_New()}

@item 
@ref{413b,,PyCallIter_New()}

@item 
@ref{413c,,PyCallIter_Type}

@item 
@ref{413d,,PyCallable_Check()}

@item 
@ref{413e,,PyCapsule_Destructor}

@item 
@ref{413f,,PyCapsule_GetContext()}

@item 
@ref{4140,,PyCapsule_GetDestructor()}

@item 
@ref{4141,,PyCapsule_GetName()}

@item 
@ref{4142,,PyCapsule_GetPointer()}

@item 
@ref{10c9,,PyCapsule_Import()}

@item 
@ref{ef0,,PyCapsule_IsValid()}

@item 
@ref{409c,,PyCapsule_New()}

@item 
@ref{4143,,PyCapsule_SetContext()}

@item 
@ref{4144,,PyCapsule_SetDestructor()}

@item 
@ref{4145,,PyCapsule_SetName()}

@item 
@ref{4146,,PyCapsule_SetPointer()}

@item 
@code{PyCapsule_Type}

@item 
@code{PyClassMethodDescr_Type}

@item 
@ref{4147,,PyCodec_BackslashReplaceErrors()}

@item 
@ref{4148,,PyCodec_Decode()}

@item 
@ref{4149,,PyCodec_Decoder()}

@item 
@ref{414a,,PyCodec_Encode()}

@item 
@ref{414b,,PyCodec_Encoder()}

@item 
@ref{414c,,PyCodec_IgnoreErrors()}

@item 
@ref{414d,,PyCodec_IncrementalDecoder()}

@item 
@ref{414e,,PyCodec_IncrementalEncoder()}

@item 
@ref{414f,,PyCodec_KnownEncoding()}

@item 
@ref{4150,,PyCodec_LookupError()}

@item 
@ref{a21,,PyCodec_NameReplaceErrors()}

@item 
@ref{4151,,PyCodec_Register()}

@item 
@ref{4152,,PyCodec_RegisterError()}

@item 
@ref{4153,,PyCodec_ReplaceErrors()}

@item 
@ref{4154,,PyCodec_StreamReader()}

@item 
@ref{4155,,PyCodec_StreamWriter()}

@item 
@ref{4156,,PyCodec_StrictErrors()}

@item 
@ref{365,,PyCodec_Unregister()}

@item 
@ref{4157,,PyCodec_XMLCharRefReplaceErrors()}

@item 
@ref{4158,,PyComplex_FromDoubles()}

@item 
@ref{4159,,PyComplex_ImagAsDouble()}

@item 
@ref{415a,,PyComplex_RealAsDouble()}

@item 
@ref{415b,,PyComplex_Type}

@item 
@ref{415c,,PyDescr_NewClassMethod()}

@item 
@ref{415d,,PyDescr_NewGetSet()}

@item 
@ref{415e,,PyDescr_NewMember()}

@item 
@ref{121b,,PyDescr_NewMethod()}

@item 
@code{PyDictItems_Type}

@item 
@code{PyDictIterItem_Type}

@item 
@code{PyDictIterKey_Type}

@item 
@code{PyDictIterValue_Type}

@item 
@code{PyDictKeys_Type}

@item 
@ref{415f,,PyDictProxy_New()}

@item 
@code{PyDictProxy_Type}

@item 
@code{PyDictRevIterItem_Type}

@item 
@code{PyDictRevIterKey_Type}

@item 
@code{PyDictRevIterValue_Type}

@item 
@code{PyDictValues_Type}

@item 
@ref{4160,,PyDict_Clear()}

@item 
@ref{4161,,PyDict_Contains()}

@item 
@ref{4162,,PyDict_Copy()}

@item 
@ref{4163,,PyDict_DelItem()}

@item 
@ref{4164,,PyDict_DelItemString()}

@item 
@ref{384,,PyDict_GetItem()}

@item 
@ref{408f,,PyDict_GetItemString()}

@item 
@ref{4165,,PyDict_GetItemWithError()}

@item 
@ref{4166,,PyDict_Items()}

@item 
@ref{4167,,PyDict_Keys()}

@item 
@ref{4168,,PyDict_Merge()}

@item 
@ref{4169,,PyDict_MergeFromSeq2()}

@item 
@ref{416a,,PyDict_New()}

@item 
@ref{416b,,PyDict_Next()}

@item 
@ref{4093,,PyDict_SetItem()}

@item 
@ref{416c,,PyDict_SetItemString()}

@item 
@ref{416d,,PyDict_Size()}

@item 
@ref{416e,,PyDict_Type}

@item 
@ref{416f,,PyDict_Update()}

@item 
@ref{4170,,PyDict_Values()}

@item 
@code{PyEllipsis_Type}

@item 
@code{PyEnum_Type}

@item 
@ref{4171,,PyErr_BadArgument()}

@item 
@ref{4172,,PyErr_BadInternalCall()}

@item 
@ref{4173,,PyErr_CheckSignals()}

@item 
@ref{bc3,,PyErr_Clear()}

@item 
@code{PyErr_Display()}

@item 
@ref{411a,,PyErr_ExceptionMatches()}

@item 
@ref{bc2,,PyErr_Fetch()}

@item 
@ref{a23,,PyErr_Format()}

@item 
@ref{a22,,PyErr_FormatV()}

@item 
@ref{1d2,,PyErr_GetExcInfo()}

@item 
@ref{4174,,PyErr_GivenExceptionMatches()}

@item 
@ref{e1a,,PyErr_NewException()}

@item 
@ref{e19,,PyErr_NewExceptionWithDoc()}

@item 
@ref{4074,,PyErr_NoMemory()}

@item 
@ref{128e,,PyErr_NormalizeException()}

@item 
@ref{116a,,PyErr_Occurred()}

@item 
@ref{11c8,,PyErr_Print()}

@item 
@ref{4175,,PyErr_PrintEx()}

@item 
@code{PyErr_ProgramText()}

@item 
@ref{865,,PyErr_ResourceWarning()}

@item 
@ref{40be,,PyErr_Restore()}

@item 
@ref{4176,,PyErr_SetExcFromWindowsErr()}

@item 
@ref{4177,,PyErr_SetExcFromWindowsErrWithFilename()}

@item 
@ref{4178,,PyErr_SetExcFromWindowsErrWithFilenameObject()}

@item 
@ref{4179,,PyErr_SetExcFromWindowsErrWithFilenameObjects()}

@item 
@ref{1d1,,PyErr_SetExcInfo()}

@item 
@ref{4072,,PyErr_SetFromErrno()}

@item 
@ref{417a,,PyErr_SetFromErrnoWithFilename()}

@item 
@ref{417b,,PyErr_SetFromErrnoWithFilenameObject()}

@item 
@ref{417c,,PyErr_SetFromErrnoWithFilenameObjects()}

@item 
@ref{417d,,PyErr_SetFromWindowsErr()}

@item 
@ref{417e,,PyErr_SetFromWindowsErrWithFilename()}

@item 
@ref{88e,,PyErr_SetImportError()}

@item 
@ref{864,,PyErr_SetImportErrorSubclass()}

@item 
@ref{417f,,PyErr_SetInterrupt()}

@item 
@ref{371,,PyErr_SetInterruptEx()}

@item 
@ref{4180,,PyErr_SetNone()}

@item 
@ref{4073,,PyErr_SetObject()}

@item 
@ref{4071,,PyErr_SetString()}

@item 
@ref{4181,,PyErr_SyntaxLocation()}

@item 
@ref{4182,,PyErr_SyntaxLocationEx()}

@item 
@ref{fab,,PyErr_WarnEx()}

@item 
@ref{4183,,PyErr_WarnExplicit()}

@item 
@ref{4184,,PyErr_WarnFormat()}

@item 
@ref{128d,,PyErr_WriteUnraisable()}

@item 
@ref{5a7,,PyEval_AcquireLock()}

@item 
@ref{5a8,,PyEval_AcquireThread()}

@item 
@code{PyEval_CallFunction()}

@item 
@code{PyEval_CallMethod()}

@item 
@code{PyEval_CallObjectWithKeywords()}

@item 
@ref{34e,,PyEval_EvalCode()}

@item 
@ref{4185,,PyEval_EvalCodeEx()}

@item 
@ref{4186,,PyEval_EvalFrame()}

@item 
@ref{bbf,,PyEval_EvalFrameEx()}

@item 
@ref{4187,,PyEval_GetBuiltins()}

@item 
@ref{4188,,PyEval_GetFrame()}

@item 
@ref{4189,,PyEval_GetFuncDesc()}

@item 
@ref{418a,,PyEval_GetFuncName()}

@item 
@ref{418b,,PyEval_GetGlobals()}

@item 
@ref{418c,,PyEval_GetLocals()}

@item 
@ref{441,,PyEval_InitThreads()}

@item 
@ref{e29,,PyEval_ReleaseLock()}

@item 
@ref{418d,,PyEval_ReleaseThread()}

@item 
@ref{5a9,,PyEval_RestoreThread()}

@item 
@ref{e2a,,PyEval_SaveThread()}

@item 
@ref{442,,PyEval_ThreadsInitialized()}

@item 
@code{PyExc_ArithmeticError}

@item 
@code{PyExc_AssertionError}

@item 
@code{PyExc_AttributeError}

@item 
@code{PyExc_BaseException}

@item 
@code{PyExc_BaseExceptionGroup}

@item 
@code{PyExc_BlockingIOError}

@item 
@code{PyExc_BrokenPipeError}

@item 
@code{PyExc_BufferError}

@item 
@code{PyExc_BytesWarning}

@item 
@code{PyExc_ChildProcessError}

@item 
@code{PyExc_ConnectionAbortedError}

@item 
@code{PyExc_ConnectionError}

@item 
@code{PyExc_ConnectionRefusedError}

@item 
@code{PyExc_ConnectionResetError}

@item 
@code{PyExc_DeprecationWarning}

@item 
@code{PyExc_EOFError}

@item 
@code{PyExc_EncodingWarning}

@item 
@code{PyExc_EnvironmentError}

@item 
@code{PyExc_Exception}

@item 
@code{PyExc_FileExistsError}

@item 
@code{PyExc_FileNotFoundError}

@item 
@code{PyExc_FloatingPointError}

@item 
@code{PyExc_FutureWarning}

@item 
@code{PyExc_GeneratorExit}

@item 
@code{PyExc_IOError}

@item 
@code{PyExc_ImportError}

@item 
@code{PyExc_ImportWarning}

@item 
@code{PyExc_IndentationError}

@item 
@code{PyExc_IndexError}

@item 
@code{PyExc_InterruptedError}

@item 
@code{PyExc_IsADirectoryError}

@item 
@code{PyExc_KeyError}

@item 
@code{PyExc_KeyboardInterrupt}

@item 
@code{PyExc_LookupError}

@item 
@code{PyExc_MemoryError}

@item 
@code{PyExc_ModuleNotFoundError}

@item 
@code{PyExc_NameError}

@item 
@code{PyExc_NotADirectoryError}

@item 
@code{PyExc_NotImplementedError}

@item 
@code{PyExc_OSError}

@item 
@code{PyExc_OverflowError}

@item 
@code{PyExc_PendingDeprecationWarning}

@item 
@code{PyExc_PermissionError}

@item 
@code{PyExc_ProcessLookupError}

@item 
@code{PyExc_RecursionError}

@item 
@code{PyExc_ReferenceError}

@item 
@code{PyExc_ResourceWarning}

@item 
@code{PyExc_RuntimeError}

@item 
@code{PyExc_RuntimeWarning}

@item 
@code{PyExc_StopAsyncIteration}

@item 
@code{PyExc_StopIteration}

@item 
@code{PyExc_SyntaxError}

@item 
@code{PyExc_SyntaxWarning}

@item 
@code{PyExc_SystemError}

@item 
@code{PyExc_SystemExit}

@item 
@code{PyExc_TabError}

@item 
@code{PyExc_TimeoutError}

@item 
@code{PyExc_TypeError}

@item 
@code{PyExc_UnboundLocalError}

@item 
@code{PyExc_UnicodeDecodeError}

@item 
@code{PyExc_UnicodeEncodeError}

@item 
@code{PyExc_UnicodeError}

@item 
@code{PyExc_UnicodeTranslateError}

@item 
@code{PyExc_UnicodeWarning}

@item 
@code{PyExc_UserWarning}

@item 
@code{PyExc_ValueError}

@item 
@code{PyExc_Warning}

@item 
@code{PyExc_WindowsError}

@item 
@code{PyExc_ZeroDivisionError}

@item 
@code{PyExceptionClass_Name()}

@item 
@ref{418e,,PyException_GetCause()}

@item 
@ref{418f,,PyException_GetContext()}

@item 
@ref{4190,,PyException_GetTraceback()}

@item 
@ref{4191,,PyException_SetCause()}

@item 
@ref{4192,,PyException_SetContext()}

@item 
@ref{128f,,PyException_SetTraceback()}

@item 
@ref{4193,,PyFile_FromFd()}

@item 
@ref{4194,,PyFile_GetLine()}

@item 
@ref{4195,,PyFile_WriteObject()}

@item 
@ref{4196,,PyFile_WriteString()}

@item 
@code{PyFilter_Type}

@item 
@ref{580,,PyFloat_AsDouble()}

@item 
@ref{4197,,PyFloat_FromDouble()}

@item 
@ref{4198,,PyFloat_FromString()}

@item 
@ref{f51,,PyFloat_GetInfo()}

@item 
@ref{f4f,,PyFloat_GetMax()}

@item 
@ref{f50,,PyFloat_GetMin()}

@item 
@ref{4199,,PyFloat_Type}

@item 
@ref{1f6,,PyFrameObject}

@item 
@ref{1f8,,PyFrame_GetCode()}

@item 
@ref{1f9,,PyFrame_GetLineNumber()}

@item 
@ref{fa6,,PyFrozenSet_New()}

@item 
@ref{419a,,PyFrozenSet_Type}

@item 
@ref{49a,,PyGC_Collect()}

@item 
@ref{37c,,PyGC_Disable()}

@item 
@ref{37b,,PyGC_Enable()}

@item 
@ref{37d,,PyGC_IsEnabled()}

@item 
@ref{5ab,,PyGILState_Ensure()}

@item 
@ref{419b,,PyGILState_GetThisThreadState()}

@item 
@ref{419c,,PyGILState_Release()}

@item 
@code{PyGILState_STATE}

@item 
@ref{6ec,,PyGetSetDef}

@item 
@code{PyGetSetDescr_Type}

@item 
@ref{4090,,PyImport_AddModule()}

@item 
@ref{419d,,PyImport_AddModuleObject()}

@item 
@ref{112b,,PyImport_AppendInittab()}

@item 
@ref{419e,,PyImport_ExecCodeModule()}

@item 
@ref{419f,,PyImport_ExecCodeModuleEx()}

@item 
@ref{41a0,,PyImport_ExecCodeModuleObject()}

@item 
@ref{41a1,,PyImport_ExecCodeModuleWithPathnames()}

@item 
@ref{41a2,,PyImport_GetImporter()}

@item 
@ref{d49,,PyImport_GetMagicNumber()}

@item 
@ref{41a3,,PyImport_GetMagicTag()}

@item 
@ref{6e7,,PyImport_GetModule()}

@item 
@ref{41a4,,PyImport_GetModuleDict()}

@item 
@ref{f5a,,PyImport_Import()}

@item 
@ref{41a5,,PyImport_ImportFrozenModule()}

@item 
@ref{41a6,,PyImport_ImportFrozenModuleObject()}

@item 
@ref{34f,,PyImport_ImportModule()}

@item 
@ref{d4a,,PyImport_ImportModuleLevel()}

@item 
@ref{41a7,,PyImport_ImportModuleLevelObject()}

@item 
@ref{c11,,PyImport_ImportModuleNoBlock()}

@item 
@ref{41a8,,PyImport_ReloadModule()}

@item 
@ref{496,,PyIndex_Check()}

@item 
@ref{3a0,,PyInterpreterState}

@item 
@ref{39ed,,PyInterpreterState_Clear()}

@item 
@ref{41a9,,PyInterpreterState_Delete()}

@item 
@ref{477,,PyInterpreterState_Get()}

@item 
@ref{41aa,,PyInterpreterState_GetDict()}

@item 
@ref{6fd,,PyInterpreterState_GetID()}

@item 
@ref{39ee,,PyInterpreterState_New()}

@item 
@ref{116c,,PyIter_Check()}

@item 
@ref{41ab,,PyIter_Next()}

@item 
@ref{366,,PyIter_Send()}

@item 
@code{PyListIter_Type}

@item 
@code{PyListRevIter_Type}

@item 
@ref{41ac,,PyList_Append()}

@item 
@ref{41ad,,PyList_AsTuple()}

@item 
@ref{408e,,PyList_GetItem()}

@item 
@ref{41ae,,PyList_GetSlice()}

@item 
@ref{41af,,PyList_Insert()}

@item 
@ref{4114,,PyList_New()}

@item 
@ref{41b0,,PyList_Reverse()}

@item 
@ref{4092,,PyList_SetItem()}

@item 
@ref{41b1,,PyList_SetSlice()}

@item 
@ref{f78,,PyList_Size()}

@item 
@ref{41b2,,PyList_Sort()}

@item 
@ref{40b8,,PyList_Type}

@item 
@ref{41b3,,PyLongObject}

@item 
@code{PyLongRangeIter_Type}

@item 
@ref{41b4,,PyLong_AsDouble()}

@item 
@ref{57c,,PyLong_AsLong()}

@item 
@ref{e17,,PyLong_AsLongAndOverflow()}

@item 
@ref{41b5,,PyLong_AsLongLong()}

@item 
@ref{e16,,PyLong_AsLongLongAndOverflow()}

@item 
@ref{41b6,,PyLong_AsSize_t()}

@item 
@ref{41b7,,PyLong_AsSsize_t()}

@item 
@ref{41b8,,PyLong_AsUnsignedLong()}

@item 
@ref{e3a,,PyLong_AsUnsignedLongLong()}

@item 
@ref{1288,,PyLong_AsUnsignedLongLongMask()}

@item 
@ref{41b9,,PyLong_AsUnsignedLongMask()}

@item 
@ref{41ba,,PyLong_AsVoidPtr()}

@item 
@ref{437,,PyLong_FromDouble()}

@item 
@ref{1085,,PyLong_FromLong()}

@item 
@ref{1086,,PyLong_FromLongLong()}

@item 
@ref{125d,,PyLong_FromSize_t()}

@item 
@ref{41bb,,PyLong_FromSsize_t()}

@item 
@ref{41bc,,PyLong_FromString()}

@item 
@ref{125b,,PyLong_FromUnsignedLong()}

@item 
@ref{125c,,PyLong_FromUnsignedLongLong()}

@item 
@ref{41bd,,PyLong_FromVoidPtr()}

@item 
@code{PyLong_GetInfo()}

@item 
@ref{41be,,PyLong_Type}

@item 
@code{PyMap_Type}

@item 
@ref{41bf,,PyMapping_Check()}

@item 
@ref{41c0,,PyMapping_GetItemString()}

@item 
@ref{41c1,,PyMapping_HasKey()}

@item 
@ref{41c2,,PyMapping_HasKeyString()}

@item 
@ref{6f2,,PyMapping_Items()}

@item 
@ref{6f0,,PyMapping_Keys()}

@item 
@ref{41c3,,PyMapping_Length()}

@item 
@ref{41c4,,PyMapping_SetItemString()}

@item 
@ref{1314,,PyMapping_Size()}

@item 
@ref{6f1,,PyMapping_Values()}

@item 
@ref{a1f,,PyMem_Calloc()}

@item 
@ref{fa3,,PyMem_Free()}

@item 
@ref{7ad,,PyMem_Malloc()}

@item 
@ref{bc6,,PyMem_Realloc()}

@item 
@ref{6eb,,PyMemberDef}

@item 
@code{PyMemberDescr_Type}

@item 
@ref{1e6,,PyMemoryView_FromBuffer()}

@item 
@ref{d0d,,PyMemoryView_FromMemory()}

@item 
@ref{41c5,,PyMemoryView_FromObject()}

@item 
@ref{41c6,,PyMemoryView_GetContiguous()}

@item 
@code{PyMemoryView_Type}

@item 
@ref{1014,,PyMethodDef}

@item 
@code{PyMethodDescr_Type}

@item 
@ref{484,,PyModuleDef}

@item 
@code{PyModuleDef_Base}

@item 
@ref{41c7,,PyModuleDef_Init()}

@item 
@code{PyModuleDef_Type}

@item 
@ref{41c8,,PyModule_AddFunctions()}

@item 
@ref{41c9,,PyModule_AddIntConstant()}

@item 
@ref{369,,PyModule_AddObject()}

@item 
@ref{368,,PyModule_AddObjectRef()}

@item 
@ref{41ca,,PyModule_AddStringConstant()}

@item 
@ref{47c,,PyModule_AddType()}

@item 
@ref{41cb,,PyModule_Create2()}

@item 
@ref{a26,,PyModule_ExecDef()}

@item 
@ref{a25,,PyModule_FromDefAndSpec2()}

@item 
@ref{41cc,,PyModule_GetDef()}

@item 
@ref{41cd,,PyModule_GetDict()}

@item 
@ref{41ce,,PyModule_GetFilename()}

@item 
@ref{41cf,,PyModule_GetFilenameObject()}

@item 
@ref{41d0,,PyModule_GetName()}

@item 
@ref{41d1,,PyModule_GetNameObject()}

@item 
@ref{487,,PyModule_GetState()}

@item 
@ref{41d2,,PyModule_New()}

@item 
@ref{41d3,,PyModule_NewObject()}

@item 
@ref{41d4,,PyModule_SetDocString()}

@item 
@ref{41d5,,PyModule_Type}

@item 
@ref{41d6,,PyNumber_Absolute()}

@item 
@ref{41d7,,PyNumber_Add()}

@item 
@ref{41d8,,PyNumber_And()}

@item 
@ref{41d9,,PyNumber_AsSsize_t()}

@item 
@ref{57d,,PyNumber_Check()}

@item 
@ref{41da,,PyNumber_Divmod()}

@item 
@ref{57f,,PyNumber_Float()}

@item 
@ref{41db,,PyNumber_FloorDivide()}

@item 
@ref{41dc,,PyNumber_InPlaceAdd()}

@item 
@ref{41dd,,PyNumber_InPlaceAnd()}

@item 
@ref{41de,,PyNumber_InPlaceFloorDivide()}

@item 
@ref{41df,,PyNumber_InPlaceLshift()}

@item 
@ref{a28,,PyNumber_InPlaceMatrixMultiply()}

@item 
@ref{41e0,,PyNumber_InPlaceMultiply()}

@item 
@ref{41e1,,PyNumber_InPlaceOr()}

@item 
@ref{41e2,,PyNumber_InPlacePower()}

@item 
@ref{41e3,,PyNumber_InPlaceRemainder()}

@item 
@ref{41e4,,PyNumber_InPlaceRshift()}

@item 
@ref{41e5,,PyNumber_InPlaceSubtract()}

@item 
@ref{41e6,,PyNumber_InPlaceTrueDivide()}

@item 
@ref{41e7,,PyNumber_InPlaceXor()}

@item 
@ref{35f,,PyNumber_Index()}

@item 
@ref{41e8,,PyNumber_Invert()}

@item 
@ref{57e,,PyNumber_Long()}

@item 
@ref{41e9,,PyNumber_Lshift()}

@item 
@ref{a27,,PyNumber_MatrixMultiply()}

@item 
@ref{41ea,,PyNumber_Multiply()}

@item 
@ref{41eb,,PyNumber_Negative()}

@item 
@ref{41ec,,PyNumber_Or()}

@item 
@ref{41ed,,PyNumber_Positive()}

@item 
@ref{41ee,,PyNumber_Power()}

@item 
@ref{41ef,,PyNumber_Remainder()}

@item 
@ref{41f0,,PyNumber_Rshift()}

@item 
@ref{41f1,,PyNumber_Subtract()}

@item 
@ref{121c,,PyNumber_ToBase()}

@item 
@ref{41f2,,PyNumber_TrueDivide()}

@item 
@ref{41f3,,PyNumber_Xor()}

@item 
@ref{6f5,,PyOS_AfterFork()}

@item 
@ref{5c1,,PyOS_AfterFork_Child()}

@item 
@ref{6f7,,PyOS_AfterFork_Parent()}

@item 
@ref{6f6,,PyOS_BeforeFork()}

@item 
@ref{41f4,,PyOS_CheckStack()}

@item 
@ref{866,,PyOS_FSPath()}

@item 
@ref{41f5,,PyOS_InputHook}

@item 
@code{PyOS_InterruptOccurred()}

@item 
@ref{41f6,,PyOS_double_to_string()}

@item 
@ref{41f7,,PyOS_getsig()}

@item 
@code{PyOS_mystricmp()}

@item 
@code{PyOS_mystrnicmp()}

@item 
@ref{41f8,,PyOS_setsig()}

@item 
@code{PyOS_sighandler_t}

@item 
@ref{103f,,PyOS_snprintf()}

@item 
@ref{e3b,,PyOS_string_to_double()}

@item 
@code{PyOS_strtol()}

@item 
@code{PyOS_strtoul()}

@item 
@ref{1040,,PyOS_vsnprintf()}

@item 
@ref{375,,PyObject}

@item 
@ref{41f9,,PyObject_ASCII()}

@item 
@ref{41fa,,PyObject_AsCharBuffer()}

@item 
@ref{41fb,,PyObject_AsFileDescriptor()}

@item 
@ref{41fc,,PyObject_AsReadBuffer()}

@item 
@ref{41fd,,PyObject_AsWriteBuffer()}

@item 
@ref{41fe,,PyObject_Bytes()}

@item 
@ref{46a,,PyObject_Call()}

@item 
@ref{5c2,,PyObject_CallFunction()}

@item 
@ref{41ff,,PyObject_CallFunctionObjArgs()}

@item 
@ref{350,,PyObject_CallMethod()}

@item 
@ref{12a7,,PyObject_CallMethodObjArgs()}

@item 
@ref{479,,PyObject_CallNoArgs()}

@item 
@ref{4081,,PyObject_CallObject()}

@item 
@ref{a20,,PyObject_Calloc()}

@item 
@ref{1dc,,PyObject_CheckBuffer()}

@item 
@ref{4200,,PyObject_CheckReadBuffer()}

@item 
@code{PyObject_ClearWeakRefs()}

@item 
@ref{4201,,PyObject_CopyData()}

@item 
@ref{4202,,PyObject_DelItem()}

@item 
@code{PyObject_DelItemString()}

@item 
@ref{4203,,PyObject_Dir()}

@item 
@code{PyObject_Format()}

@item 
@ref{7ac,,PyObject_Free()}

@item 
@ref{1039,,PyObject_GC_Del()}

@item 
@ref{47e,,PyObject_GC_IsFinalized()}

@item 
@ref{47d,,PyObject_GC_IsTracked()}

@item 
@ref{103a,,PyObject_GC_Track()}

@item 
@ref{103b,,PyObject_GC_UnTrack()}

@item 
@ref{4204,,PyObject_GenericGetAttr()}

@item 
@ref{11b7,,PyObject_GenericGetDict()}

@item 
@ref{4205,,PyObject_GenericSetAttr()}

@item 
@ref{4206,,PyObject_GenericSetDict()}

@item 
@ref{4207,,PyObject_GetAIter()}

@item 
@ref{4208,,PyObject_GetAttr()}

@item 
@ref{408c,,PyObject_GetAttrString()}

@item 
@ref{1dd,,PyObject_GetBuffer()}

@item 
@ref{4115,,PyObject_GetItem()}

@item 
@ref{4209,,PyObject_GetIter()}

@item 
@ref{420a,,PyObject_HasAttr()}

@item 
@ref{420b,,PyObject_HasAttrString()}

@item 
@ref{209,,PyObject_Hash()}

@item 
@ref{f2d,,PyObject_HashNotImplemented()}

@item 
@ref{581,,PyObject_Init()}

@item 
@ref{11dd,,PyObject_InitVar()}

@item 
@ref{a17,,PyObject_IsInstance()}

@item 
@ref{a18,,PyObject_IsSubclass()}

@item 
@ref{420c,,PyObject_IsTrue()}

@item 
@ref{420d,,PyObject_Length()}

@item 
@ref{7af,,PyObject_Malloc()}

@item 
@ref{420e,,PyObject_Not()}

@item 
@ref{fa4,,PyObject_Realloc()}

@item 
@ref{bc0,,PyObject_Repr()}

@item 
@ref{40cc,,PyObject_RichCompare()}

@item 
@ref{1239,,PyObject_RichCompareBool()}

@item 
@code{PyObject_SelfIter()}

@item 
@ref{420f,,PyObject_SetAttr()}

@item 
@ref{4210,,PyObject_SetAttrString()}

@item 
@ref{4113,,PyObject_SetItem()}

@item 
@ref{4211,,PyObject_Size()}

@item 
@ref{bc1,,PyObject_Str()}

@item 
@ref{4212,,PyObject_Type()}

@item 
@ref{4213,,PyProperty_Type}

@item 
@code{PyRangeIter_Type}

@item 
@code{PyRange_Type}

@item 
@code{PyReversed_Type}

@item 
@ref{4214,,PySeqIter_New()}

@item 
@ref{4215,,PySeqIter_Type}

@item 
@ref{4216,,PySequence_Check()}

@item 
@ref{4217,,PySequence_Concat()}

@item 
@ref{4218,,PySequence_Contains()}

@item 
@ref{4219,,PySequence_Count()}

@item 
@ref{1313,,PySequence_DelItem()}

@item 
@ref{421a,,PySequence_DelSlice()}

@item 
@ref{421b,,PySequence_Fast()}

@item 
@ref{1311,,PySequence_GetItem()}

@item 
@ref{421c,,PySequence_GetSlice()}

@item 
@code{PySequence_In()}

@item 
@ref{421d,,PySequence_InPlaceConcat()}

@item 
@ref{421e,,PySequence_InPlaceRepeat()}

@item 
@ref{421f,,PySequence_Index()}

@item 
@ref{4220,,PySequence_Length()}

@item 
@ref{4221,,PySequence_List()}

@item 
@ref{4222,,PySequence_Repeat()}

@item 
@ref{1312,,PySequence_SetItem()}

@item 
@ref{4223,,PySequence_SetSlice()}

@item 
@ref{1310,,PySequence_Size()}

@item 
@ref{4224,,PySequence_Tuple()}

@item 
@code{PySetIter_Type}

@item 
@ref{fa7,,PySet_Add()}

@item 
@ref{4225,,PySet_Clear()}

@item 
@ref{fa9,,PySet_Contains()}

@item 
@ref{fa8,,PySet_Discard()}

@item 
@ref{fa5,,PySet_New()}

@item 
@ref{4226,,PySet_Pop()}

@item 
@ref{faa,,PySet_Size()}

@item 
@ref{4227,,PySet_Type}

@item 
@ref{6f4,,PySlice_AdjustIndices()}

@item 
@ref{4228,,PySlice_GetIndices()}

@item 
@ref{72b,,PySlice_GetIndicesEx()}

@item 
@ref{4229,,PySlice_New()}

@item 
@ref{422a,,PySlice_Type}

@item 
@ref{6f3,,PySlice_Unpack()}

@item 
@ref{422b,,PyState_AddModule()}

@item 
@ref{422c,,PyState_FindModule()}

@item 
@ref{422d,,PyState_RemoveModule()}

@item 
@ref{6ee,,PyStructSequence_Desc}

@item 
@ref{6ed,,PyStructSequence_Field}

@item 
@ref{422e,,PyStructSequence_GetItem()}

@item 
@ref{422f,,PyStructSequence_New()}

@item 
@ref{4230,,PyStructSequence_NewType()}

@item 
@ref{4231,,PyStructSequence_SetItem()}

@item 
@ref{48b,,PyStructSequence_UnnamedField}

@item 
@code{PySuper_Type}

@item 
@ref{200,,PySys_AddWarnOption()}

@item 
@ref{1ff,,PySys_AddWarnOptionUnicode()}

@item 
@ref{201,,PySys_AddXOption()}

@item 
@ref{4232,,PySys_FormatStderr()}

@item 
@ref{4233,,PySys_FormatStdout()}

@item 
@ref{4234,,PySys_GetObject()}

@item 
@ref{4235,,PySys_GetXOptions()}

@item 
@code{PySys_HasWarnOptions()}

@item 
@ref{4236,,PySys_ResetWarnOptions()}

@item 
@ref{eec,,PySys_SetArgv()}

@item 
@ref{e14,,PySys_SetArgvEx()}

@item 
@ref{4237,,PySys_SetObject()}

@item 
@ref{4238,,PySys_SetPath()}

@item 
@ref{4239,,PySys_WriteStderr()}

@item 
@ref{423a,,PySys_WriteStdout()}

@item 
@ref{1fa,,PyThreadState}

@item 
@ref{11f3,,PyThreadState_Clear()}

@item 
@ref{1236,,PyThreadState_Delete()}

@item 
@ref{423b,,PyThreadState_Get()}

@item 
@ref{423c,,PyThreadState_GetDict()}

@item 
@ref{1fb,,PyThreadState_GetFrame()}

@item 
@ref{478,,PyThreadState_GetID()}

@item 
@ref{476,,PyThreadState_GetInterpreter()}

@item 
@ref{423d,,PyThreadState_New()}

@item 
@ref{6fb,,PyThreadState_SetAsyncExc()}

@item 
@ref{423e,,PyThreadState_Swap()}

@item 
@code{PyThread_GetInfo()}

@item 
@ref{423f,,PyThread_ReInitTLS()}

@item 
@code{PyThread_acquire_lock()}

@item 
@code{PyThread_acquire_lock_timed()}

@item 
@code{PyThread_allocate_lock()}

@item 
@ref{4240,,PyThread_create_key()}

@item 
@ref{4241,,PyThread_delete_key()}

@item 
@ref{4242,,PyThread_delete_key_value()}

@item 
@code{PyThread_exit_thread()}

@item 
@code{PyThread_free_lock()}

@item 
@ref{4243,,PyThread_get_key_value()}

@item 
@code{PyThread_get_stacksize()}

@item 
@code{PyThread_get_thread_ident()}

@item 
@code{PyThread_get_thread_native_id()}

@item 
@code{PyThread_init_thread()}

@item 
@code{PyThread_release_lock()}

@item 
@ref{bc7,,PyThread_set_key_value()}

@item 
@code{PyThread_set_stacksize()}

@item 
@code{PyThread_start_new_thread()}

@item 
@ref{4244,,PyThread_tss_alloc()}

@item 
@ref{4245,,PyThread_tss_create()}

@item 
@ref{4246,,PyThread_tss_delete()}

@item 
@ref{4247,,PyThread_tss_free()}

@item 
@ref{4248,,PyThread_tss_get()}

@item 
@ref{4249,,PyThread_tss_is_created()}

@item 
@ref{424a,,PyThread_tss_set()}

@item 
@code{PyTraceBack_Here()}

@item 
@code{PyTraceBack_Print()}

@item 
@code{PyTraceBack_Type}

@item 
@code{PyTupleIter_Type}

@item 
@ref{408d,,PyTuple_GetItem()}

@item 
@ref{424b,,PyTuple_GetSlice()}

@item 
@ref{424c,,PyTuple_New()}

@item 
@ref{424d,,PyTuple_Pack()}

@item 
@ref{4091,,PyTuple_SetItem()}

@item 
@ref{424e,,PyTuple_Size()}

@item 
@ref{424f,,PyTuple_Type}

@item 
@ref{5c6,,PyTypeObject}

@item 
@ref{4250,,PyType_ClearCache()}

@item 
@ref{36d,,PyType_FromModuleAndSpec()}

@item 
@ref{469,,PyType_FromSpec()}

@item 
@ref{36c,,PyType_FromSpecWithBases()}

@item 
@ref{582,,PyType_GenericAlloc()}

@item 
@ref{40a8,,PyType_GenericNew()}

@item 
@ref{11dc,,PyType_GetFlags()}

@item 
@ref{474,,PyType_GetModule()}

@item 
@ref{475,,PyType_GetModuleState()}

@item 
@ref{1d5,,PyType_GetName()}

@item 
@ref{1d6,,PyType_GetQualName()}

@item 
@ref{36e,,PyType_GetSlot()}

@item 
@ref{4251,,PyType_IsSubtype()}

@item 
@ref{4252,,PyType_Modified()}

@item 
@ref{1ec,,PyType_Ready()}

@item 
@ref{4253,,PyType_Slot}

@item 
@ref{4254,,PyType_Spec}

@item 
@ref{4255,,PyType_Type}

@item 
@ref{4256,,PyUnicodeDecodeError_Create()}

@item 
@ref{4257,,PyUnicodeDecodeError_GetEncoding()}

@item 
@ref{4258,,PyUnicodeDecodeError_GetEnd()}

@item 
@ref{4259,,PyUnicodeDecodeError_GetObject()}

@item 
@ref{425a,,PyUnicodeDecodeError_GetReason()}

@item 
@ref{425b,,PyUnicodeDecodeError_GetStart()}

@item 
@ref{425c,,PyUnicodeDecodeError_SetEnd()}

@item 
@ref{425d,,PyUnicodeDecodeError_SetReason()}

@item 
@ref{425e,,PyUnicodeDecodeError_SetStart()}

@item 
@ref{425f,,PyUnicodeEncodeError_GetEncoding()}

@item 
@ref{4260,,PyUnicodeEncodeError_GetEnd()}

@item 
@ref{4261,,PyUnicodeEncodeError_GetObject()}

@item 
@ref{4262,,PyUnicodeEncodeError_GetReason()}

@item 
@ref{4263,,PyUnicodeEncodeError_GetStart()}

@item 
@ref{4264,,PyUnicodeEncodeError_SetEnd()}

@item 
@ref{4265,,PyUnicodeEncodeError_SetReason()}

@item 
@ref{4266,,PyUnicodeEncodeError_SetStart()}

@item 
@code{PyUnicodeIter_Type}

@item 
@ref{4267,,PyUnicodeTranslateError_GetEnd()}

@item 
@ref{4268,,PyUnicodeTranslateError_GetObject()}

@item 
@ref{4269,,PyUnicodeTranslateError_GetReason()}

@item 
@ref{426a,,PyUnicodeTranslateError_GetStart()}

@item 
@ref{426b,,PyUnicodeTranslateError_SetEnd()}

@item 
@ref{426c,,PyUnicodeTranslateError_SetReason()}

@item 
@ref{426d,,PyUnicodeTranslateError_SetStart()}

@item 
@code{PyUnicode_Append()}

@item 
@code{PyUnicode_AppendAndDel()}

@item 
@ref{d38,,PyUnicode_AsASCIIString()}

@item 
@ref{426e,,PyUnicode_AsCharmapString()}

@item 
@code{PyUnicode_AsDecodedObject()}

@item 
@code{PyUnicode_AsDecodedUnicode()}

@item 
@code{PyUnicode_AsEncodedObject()}

@item 
@ref{426f,,PyUnicode_AsEncodedString()}

@item 
@code{PyUnicode_AsEncodedUnicode()}

@item 
@ref{d37,,PyUnicode_AsLatin1String()}

@item 
@ref{d39,,PyUnicode_AsMBCSString()}

@item 
@ref{d36,,PyUnicode_AsRawUnicodeEscapeString()}

@item 
@ref{d17,,PyUnicode_AsUCS4()}

@item 
@ref{39e,,PyUnicode_AsUCS4Copy()}

@item 
@ref{4270,,PyUnicode_AsUTF16String()}

@item 
@ref{4271,,PyUnicode_AsUTF32String()}

@item 
@ref{367,,PyUnicode_AsUTF8AndSize()}

@item 
@ref{d34,,PyUnicode_AsUTF8String()}

@item 
@ref{d35,,PyUnicode_AsUnicodeEscapeString()}

@item 
@ref{1315,,PyUnicode_AsWideChar()}

@item 
@ref{39f,,PyUnicode_AsWideCharString()}

@item 
@code{PyUnicode_BuildEncodingMap()}

@item 
@ref{39a,,PyUnicode_Compare()}

@item 
@ref{e18,,PyUnicode_CompareWithASCIIString()}

@item 
@ref{d47,,PyUnicode_Concat()}

@item 
@ref{4272,,PyUnicode_Contains()}

@item 
@ref{4273,,PyUnicode_Count()}

@item 
@ref{1284,,PyUnicode_Decode()}

@item 
@ref{4274,,PyUnicode_DecodeASCII()}

@item 
@ref{4275,,PyUnicode_DecodeCharmap()}

@item 
@code{PyUnicode_DecodeCodePageStateful()}

@item 
@ref{4276,,PyUnicode_DecodeFSDefault()}

@item 
@ref{4277,,PyUnicode_DecodeFSDefaultAndSize()}

@item 
@ref{4278,,PyUnicode_DecodeLatin1()}

@item 
@ref{4279,,PyUnicode_DecodeLocale()}

@item 
@ref{700,,PyUnicode_DecodeLocaleAndSize()}

@item 
@ref{427a,,PyUnicode_DecodeMBCS()}

@item 
@ref{427b,,PyUnicode_DecodeMBCSStateful()}

@item 
@ref{427c,,PyUnicode_DecodeRawUnicodeEscape()}

@item 
@ref{427d,,PyUnicode_DecodeUTF16()}

@item 
@ref{427e,,PyUnicode_DecodeUTF16Stateful()}

@item 
@ref{427f,,PyUnicode_DecodeUTF32()}

@item 
@ref{4280,,PyUnicode_DecodeUTF32Stateful()}

@item 
@ref{4281,,PyUnicode_DecodeUTF7()}

@item 
@ref{4282,,PyUnicode_DecodeUTF7Stateful()}

@item 
@ref{4283,,PyUnicode_DecodeUTF8()}

@item 
@ref{4284,,PyUnicode_DecodeUTF8Stateful()}

@item 
@ref{4285,,PyUnicode_DecodeUnicodeEscape()}

@item 
@ref{d3a,,PyUnicode_EncodeCodePage()}

@item 
@ref{130e,,PyUnicode_EncodeFSDefault()}

@item 
@ref{701,,PyUnicode_EncodeLocale()}

@item 
@ref{867,,PyUnicode_FSConverter()}

@item 
@ref{868,,PyUnicode_FSDecoder()}

@item 
@ref{4286,,PyUnicode_Find()}

@item 
@ref{39c,,PyUnicode_FindChar()}

@item 
@ref{4287,,PyUnicode_Format()}

@item 
@ref{4288,,PyUnicode_FromEncodedObject()}

@item 
@ref{398,,PyUnicode_FromFormat()}

@item 
@ref{4289,,PyUnicode_FromFormatV()}

@item 
@ref{428a,,PyUnicode_FromObject()}

@item 
@code{PyUnicode_FromOrdinal()}

@item 
@ref{40f4,,PyUnicode_FromString()}

@item 
@ref{428b,,PyUnicode_FromStringAndSize()}

@item 
@ref{d2d,,PyUnicode_FromWideChar()}

@item 
@code{PyUnicode_GetDefaultEncoding()}

@item 
@ref{395,,PyUnicode_GetLength()}

@item 
@ref{d31,,PyUnicode_GetSize()}

@item 
@ref{428c,,PyUnicode_InternFromString()}

@item 
@code{PyUnicode_InternImmortal()}

@item 
@ref{393,,PyUnicode_InternInPlace()}

@item 
@ref{1233,,PyUnicode_IsIdentifier()}

@item 
@ref{d48,,PyUnicode_Join()}

@item 
@code{PyUnicode_Partition()}

@item 
@code{PyUnicode_RPartition()}

@item 
@code{PyUnicode_RSplit()}

@item 
@ref{d0e,,PyUnicode_ReadChar()}

@item 
@ref{428d,,PyUnicode_Replace()}

@item 
@code{PyUnicode_Resize()}

@item 
@ref{428e,,PyUnicode_RichCompare()}

@item 
@ref{428f,,PyUnicode_Split()}

@item 
@ref{4290,,PyUnicode_Splitlines()}

@item 
@ref{399,,PyUnicode_Substring()}

@item 
@ref{39b,,PyUnicode_Tailmatch()}

@item 
@ref{4291,,PyUnicode_Translate()}

@item 
@ref{4292,,PyUnicode_Type}

@item 
@ref{d0f,,PyUnicode_WriteChar()}

@item 
@ref{4293,,PyVarObject}

@item 
@code{PyWeakReference}

@item 
@ref{4294,,PyWeakref_GetObject()}

@item 
@ref{4295,,PyWeakref_NewProxy()}

@item 
@ref{10a1,,PyWeakref_NewRef()}

@item 
@code{PyWrapperDescr_Type}

@item 
@ref{4296,,PyWrapper_New()}

@item 
@code{PyZip_Type}

@item 
@ref{488,,Py_AddPendingCall()}

@item 
@ref{4297,,Py_AtExit()}

@item 
@ref{4096,,Py_BEGIN_ALLOW_THREADS}

@item 
@ref{4298,,Py_BLOCK_THREADS}

@item 
@ref{380,,Py_BuildValue()}

@item 
@ref{4b6,,Py_BytesMain()}

@item 
@ref{34d,,Py_CompileString()}

@item 
@code{Py_DecRef()}

@item 
@ref{6fe,,Py_DecodeLocale()}

@item 
@ref{5aa,,Py_END_ALLOW_THREADS}

@item 
@ref{6ff,,Py_EncodeLocale()}

@item 
@ref{10b5,,Py_EndInterpreter()}

@item 
@ref{47a,,Py_EnterRecursiveCall()}

@item 
@ref{8a6,,Py_Exit()}

@item 
@ref{48f,,Py_FatalError()}

@item 
@code{Py_FileSystemDefaultEncodeErrors}

@item 
@code{Py_FileSystemDefaultEncoding}

@item 
@ref{eed,,Py_Finalize()}

@item 
@ref{862,,Py_FinalizeEx()}

@item 
@ref{4299,,Py_GenericAlias()}

@item 
@ref{429a,,Py_GenericAliasType}

@item 
@ref{f94,,Py_GetBuildInfo()}

@item 
@ref{429b,,Py_GetCompiler()}

@item 
@ref{429c,,Py_GetCopyright()}

@item 
@ref{389,,Py_GetExecPrefix()}

@item 
@ref{387,,Py_GetPath()}

@item 
@ref{429d,,Py_GetPlatform()}

@item 
@ref{388,,Py_GetPrefix()}

@item 
@ref{38a,,Py_GetProgramFullPath()}

@item 
@ref{38c,,Py_GetProgramName()}

@item 
@ref{38b,,Py_GetPythonHome()}

@item 
@code{Py_GetRecursionLimit()}

@item 
@ref{429e,,Py_GetVersion()}

@item 
@code{Py_HasFileSystemDefaultEncoding}

@item 
@code{Py_IncRef()}

@item 
@ref{38d,,Py_Initialize()}

@item 
@ref{429f,,Py_InitializeEx()}

@item 
@ref{377,,Py_Is()}

@item 
@ref{37a,,Py_IsFalse()}

@item 
@ref{411d,,Py_IsInitialized()}

@item 
@ref{378,,Py_IsNone()}

@item 
@ref{379,,Py_IsTrue()}

@item 
@ref{47b,,Py_LeaveRecursiveCall()}

@item 
@ref{763,,Py_Main()}

@item 
@code{Py_MakePendingCalls()}

@item 
@ref{42a0,,Py_NewInterpreter()}

@item 
@ref{36a,,Py_NewRef()}

@item 
@ref{42a1,,Py_ReprEnter()}

@item 
@ref{42a2,,Py_ReprLeave()}

@item 
@ref{202,,Py_SetPath()}

@item 
@ref{203,,Py_SetProgramName()}

@item 
@ref{204,,Py_SetPythonHome()}

@item 
@code{Py_SetRecursionLimit()}

@item 
@ref{d12,,Py_UCS4}

@item 
@ref{42a3,,Py_UNBLOCK_THREADS}

@item 
@code{Py_UTF8Mode}

@item 
@ref{42a4,,Py_VaBuildValue()}

@item 
@ref{1d9,,Py_Version}

@item 
@ref{36b,,Py_XNewRef()}

@item 
@ref{1db,,Py_buffer}

@item 
@code{Py_intptr_t}

@item 
@code{Py_ssize_t}

@item 
@code{Py_uintptr_t}

@item 
@ref{42a5,,allocfunc}

@item 
@ref{42a6,,binaryfunc}

@item 
@ref{42a7,,descrgetfunc}

@item 
@ref{42a8,,descrsetfunc}

@item 
@ref{42a9,,destructor}

@item 
@ref{42aa,,getattrfunc}

@item 
@ref{42ab,,getattrofunc}

@item 
@ref{42ac,,getiterfunc}

@item 
@code{getter}

@item 
@ref{42ad,,hashfunc}

@item 
@ref{42ae,,initproc}

@item 
@ref{42af,,inquiry}

@item 
@ref{42b0,,iternextfunc}

@item 
@ref{42b1,,lenfunc}

@item 
@ref{42b2,,newfunc}

@item 
@ref{42b3,,objobjargproc}

@item 
@ref{42b4,,objobjproc}

@item 
@ref{42b5,,reprfunc}

@item 
@ref{42b6,,richcmpfunc}

@item 
@ref{42b7,,setattrfunc}

@item 
@ref{42b8,,setattrofunc}

@item 
@code{setter}

@item 
@ref{42b9,,ssizeargfunc}

@item 
@ref{42ba,,ssizeobjargproc}

@item 
@code{ssizessizeargfunc}

@item 
@code{ssizessizeobjargproc}

@item 
@code{symtable}

@item 
@ref{42bb,,ternaryfunc}

@item 
@ref{42bc,,traverseproc}

@item 
@ref{42bd,,unaryfunc}

@item 
@ref{42be,,visitproc}
@end itemize


@node The Very High Level Layer,Reference Counting,C API Stability,Python/C API Reference Manual
@anchor{c-api/veryhigh doc}@anchor{42bf}@anchor{c-api/veryhigh the-very-high-level-layer}@anchor{42c0}@anchor{c-api/veryhigh veryhigh}@anchor{42c1}
@section The Very High Level Layer


The functions in this chapter will let you execute Python source code given in a
file or a buffer, but they will not let you interact in a more detailed way with
the interpreter.

Several of these functions accept a start symbol from the grammar as a
parameter.  The available start symbols are @code{Py_eval_input},
@code{Py_file_input}, and @code{Py_single_input}.  These are described
following the functions which accept them as parameters.

Note also that several of these functions take FILE* parameters.  One
particular issue which needs to be handled carefully is that the @code{FILE}
structure for different C libraries can be different and incompatible.  Under
Windows (at least), it is possible for dynamically linked extensions to actually
use different libraries, so care should be taken that FILE* parameters
are only passed to these functions if it is certain that they were created by
the same library that the Python runtime is using.

@geindex Py_Main (C function)
@anchor{c-api/veryhigh c Py_Main}@anchor{763}
@deffn {C Function} int  Py_Main (int  argc, wchar_t  **argv)
@emph{ Part of the @ref{35d,,Stable ABI}.}
The main program for the standard interpreter.  This is made available for
programs which embed Python.  The @emph{argc} and @emph{argv} parameters should be
prepared exactly as those which are passed to a C program’s @code{main()}
function (converted to wchar_t according to the user’s locale).  It is
important to note that the argument list may be modified (but the contents of
the strings pointed to by the argument list are not). The return value will
be @code{0} if the interpreter exits normally (i.e., without an exception),
@code{1} if the interpreter exits due to an exception, or @code{2} if the parameter
list does not represent a valid Python command line.

Note that if an otherwise unhandled @ref{892,,SystemExit} is raised, this
function will not return @code{1}, but exit the process, as long as
@code{Py_InspectFlag} is not set.
@end deffn

@geindex Py_BytesMain (C function)
@anchor{c-api/veryhigh c Py_BytesMain}@anchor{4b6}
@deffn {C Function} int  Py_BytesMain (int  argc, char  **argv)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.8.}
Similar to @ref{763,,Py_Main()} but @emph{argv} is an array of bytes strings.


New in version 3.8.

@end deffn

@geindex PyRun_AnyFile (C function)
@anchor{c-api/veryhigh c PyRun_AnyFile}@anchor{42c2}
@deffn {C Function} int  PyRun_AnyFile (FILE  *fp, const  char  *filename)

This is a simplified interface to @ref{42c3,,PyRun_AnyFileExFlags()} below, leaving
@emph{closeit} set to @code{0} and @emph{flags} set to @code{NULL}.
@end deffn

@geindex PyRun_AnyFileFlags (C function)
@anchor{c-api/veryhigh c PyRun_AnyFileFlags}@anchor{42c4}
@deffn {C Function} int  PyRun_AnyFileFlags (FILE  *fp, const  char  *filename, PyCompilerFlags  *flags)

This is a simplified interface to @ref{42c3,,PyRun_AnyFileExFlags()} below, leaving
the @emph{closeit} argument set to @code{0}.
@end deffn

@geindex PyRun_AnyFileEx (C function)
@anchor{c-api/veryhigh c PyRun_AnyFileEx}@anchor{42c5}
@deffn {C Function} int  PyRun_AnyFileEx (FILE  *fp, const  char  *filename, int  closeit)

This is a simplified interface to @ref{42c3,,PyRun_AnyFileExFlags()} below, leaving
the @emph{flags} argument set to @code{NULL}.
@end deffn

@geindex PyRun_AnyFileExFlags (C function)
@anchor{c-api/veryhigh c PyRun_AnyFileExFlags}@anchor{42c3}
@deffn {C Function} int  PyRun_AnyFileExFlags (FILE  *fp, const  char  *filename, int  closeit, PyCompilerFlags  *flags)

If @emph{fp} refers to a file associated with an interactive device (console or
terminal input or Unix pseudo-terminal), return the value of
@ref{42c6,,PyRun_InteractiveLoop()}, otherwise return the result of
@ref{40ec,,PyRun_SimpleFile()}.  @emph{filename} is decoded from the filesystem
encoding (@ref{79e,,sys.getfilesystemencoding()}).  If @emph{filename} is @code{NULL}, this
function uses @code{"???"} as the filename.
If @emph{closeit} is true, the file is closed before
@code{PyRun_SimpleFileExFlags()} returns.
@end deffn

@geindex PyRun_SimpleString (C function)
@anchor{c-api/veryhigh c PyRun_SimpleString}@anchor{40eb}
@deffn {C Function} int  PyRun_SimpleString (const  char  *command)

This is a simplified interface to @ref{42c7,,PyRun_SimpleStringFlags()} below,
leaving the @ref{5c0,,PyCompilerFlags}* argument set to @code{NULL}.
@end deffn

@geindex PyRun_SimpleStringFlags (C function)
@anchor{c-api/veryhigh c PyRun_SimpleStringFlags}@anchor{42c7}
@deffn {C Function} int  PyRun_SimpleStringFlags (const  char  *command, PyCompilerFlags  *flags)

Executes the Python source code from @emph{command} in the @ref{1,,__main__} module
according to the @emph{flags} argument. If @ref{1,,__main__} does not already exist, it
is created.  Returns @code{0} on success or @code{-1} if an exception was raised.  If
there was an error, there is no way to get the exception information. For the
meaning of @emph{flags}, see below.

Note that if an otherwise unhandled @ref{892,,SystemExit} is raised, this
function will not return @code{-1}, but exit the process, as long as
@code{Py_InspectFlag} is not set.
@end deffn

@geindex PyRun_SimpleFile (C function)
@anchor{c-api/veryhigh c PyRun_SimpleFile}@anchor{40ec}
@deffn {C Function} int  PyRun_SimpleFile (FILE  *fp, const  char  *filename)

This is a simplified interface to @ref{1181,,PyRun_SimpleFileExFlags()} below,
leaving @emph{closeit} set to @code{0} and @emph{flags} set to @code{NULL}.
@end deffn

@geindex PyRun_SimpleFileEx (C function)
@anchor{c-api/veryhigh c PyRun_SimpleFileEx}@anchor{42c8}
@deffn {C Function} int  PyRun_SimpleFileEx (FILE  *fp, const  char  *filename, int  closeit)

This is a simplified interface to @ref{1181,,PyRun_SimpleFileExFlags()} below,
leaving @emph{flags} set to @code{NULL}.
@end deffn

@geindex PyRun_SimpleFileExFlags (C function)
@anchor{c-api/veryhigh c PyRun_SimpleFileExFlags}@anchor{1181}
@deffn {C Function} int  PyRun_SimpleFileExFlags (FILE  *fp, const  char  *filename, int  closeit, PyCompilerFlags  *flags)

Similar to @ref{42c7,,PyRun_SimpleStringFlags()}, but the Python source code is read
from @emph{fp} instead of an in-memory string. @emph{filename} should be the name of
the file, it is decoded from @ref{1628,,filesystem encoding and error handler}.
If @emph{closeit} is true, the file is closed before
@code{PyRun_SimpleFileExFlags()} returns.

@cartouche
@quotation Note 
On Windows, @emph{fp} should be opened as binary mode (e.g. @code{fopen(filename, "rb")}).
Otherwise, Python may not handle script file with LF line ending correctly.
@end quotation
@end cartouche
@end deffn

@geindex PyRun_InteractiveOne (C function)
@anchor{c-api/veryhigh c PyRun_InteractiveOne}@anchor{42c9}
@deffn {C Function} int  PyRun_InteractiveOne (FILE  *fp, const  char  *filename)

This is a simplified interface to @ref{42ca,,PyRun_InteractiveOneFlags()} below,
leaving @emph{flags} set to @code{NULL}.
@end deffn

@geindex PyRun_InteractiveOneFlags (C function)
@anchor{c-api/veryhigh c PyRun_InteractiveOneFlags}@anchor{42ca}
@deffn {C Function} int  PyRun_InteractiveOneFlags (FILE  *fp, const  char  *filename, PyCompilerFlags  *flags)

Read and execute a single statement from a file associated with an
interactive device according to the @emph{flags} argument.  The user will be
prompted using @code{sys.ps1} and @code{sys.ps2}.  @emph{filename} is decoded from the
@ref{1628,,filesystem encoding and error handler}.

Returns @code{0} when the input was
executed successfully, @code{-1} if there was an exception, or an error code
from the @code{errcode.h} include file distributed as part of Python if
there was a parse error.  (Note that @code{errcode.h} is not included by
@code{Python.h}, so must be included specifically if needed.)
@end deffn

@geindex PyRun_InteractiveLoop (C function)
@anchor{c-api/veryhigh c PyRun_InteractiveLoop}@anchor{42c6}
@deffn {C Function} int  PyRun_InteractiveLoop (FILE  *fp, const  char  *filename)

This is a simplified interface to @ref{42cb,,PyRun_InteractiveLoopFlags()} below,
leaving @emph{flags} set to @code{NULL}.
@end deffn

@geindex PyRun_InteractiveLoopFlags (C function)
@anchor{c-api/veryhigh c PyRun_InteractiveLoopFlags}@anchor{42cb}
@deffn {C Function} int  PyRun_InteractiveLoopFlags (FILE  *fp, const  char  *filename, PyCompilerFlags  *flags)

Read and execute statements from a file associated with an interactive device
until EOF is reached.  The user will be prompted using @code{sys.ps1} and
@code{sys.ps2}.  @emph{filename} is decoded from the @ref{1628,,filesystem encoding and error handler}.  Returns @code{0} at EOF or a negative number upon failure.
@end deffn

@geindex PyOS_InputHook (C var)
@anchor{c-api/veryhigh c PyOS_InputHook}@anchor{41f5}
@deffn {C Variable} int  (*PyOS_InputHook)(void)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Can be set to point to a function with the prototype
@code{int func(void)}.  The function will be called when Python’s
interpreter prompt is about to become idle and wait for user input
from the terminal.  The return value is ignored.  Overriding this
hook can be used to integrate the interpreter’s prompt with other
event loops, as done in the @code{Modules/_tkinter.c} in the
Python source code.
@end deffn

@geindex PyOS_ReadlineFunctionPointer (C var)
@anchor{c-api/veryhigh c PyOS_ReadlineFunctionPointer}@anchor{3a2}
@deffn {C Variable} char  *(*PyOS_ReadlineFunctionPointer)(FILE*,  FILE*,  const  char*)

Can be set to point to a function with the prototype
@code{char *func(FILE *stdin, FILE *stdout, char *prompt)},
overriding the default function used to read a single line of input
at the interpreter’s prompt.  The function is expected to output
the string @emph{prompt} if it’s not @code{NULL}, and then read a line of
input from the provided standard input file, returning the
resulting string.  For example, The @ref{d9,,readline} module sets
this hook to provide line-editing and tab-completion features.

The result must be a string allocated by @ref{bc4,,PyMem_RawMalloc()} or
@ref{bc5,,PyMem_RawRealloc()}, or @code{NULL} if an error occurred.


Changed in version 3.4: The result must be allocated by @ref{bc4,,PyMem_RawMalloc()} or
@ref{bc5,,PyMem_RawRealloc()}, instead of being allocated by
@ref{7ad,,PyMem_Malloc()} or @ref{bc6,,PyMem_Realloc()}.

@end deffn

@geindex PyRun_String (C function)
@anchor{c-api/veryhigh c PyRun_String}@anchor{127d}
@deffn {C Function} @ref{375,,PyObject}  *PyRun_String (const  char  *str, int  start, PyObject  *globals, PyObject  *locals)
@emph{Return value: New reference.}
This is a simplified interface to @ref{42cc,,PyRun_StringFlags()} below, leaving
@emph{flags} set to @code{NULL}.
@end deffn

@geindex PyRun_StringFlags (C function)
@anchor{c-api/veryhigh c PyRun_StringFlags}@anchor{42cc}
@deffn {C Function} @ref{375,,PyObject}  *PyRun_StringFlags (const  char  *str, int  start, PyObject  *globals, PyObject  *locals, PyCompilerFlags  *flags)
@emph{Return value: New reference.}
Execute Python source code from @emph{str} in the context specified by the
objects @emph{globals} and @emph{locals} with the compiler flags specified by
@emph{flags}.  @emph{globals} must be a dictionary; @emph{locals} can be any object
that implements the mapping protocol.  The parameter @emph{start} specifies
the start token that should be used to parse the source code.

Returns the result of executing the code as a Python object, or @code{NULL} if an
exception was raised.
@end deffn

@geindex PyRun_File (C function)
@anchor{c-api/veryhigh c PyRun_File}@anchor{42cd}
@deffn {C Function} @ref{375,,PyObject}  *PyRun_File (FILE  *fp, const  char  *filename, int  start, PyObject  *globals, PyObject  *locals)
@emph{Return value: New reference.}
This is a simplified interface to @ref{42ce,,PyRun_FileExFlags()} below, leaving
@emph{closeit} set to @code{0} and @emph{flags} set to @code{NULL}.
@end deffn

@geindex PyRun_FileEx (C function)
@anchor{c-api/veryhigh c PyRun_FileEx}@anchor{42cf}
@deffn {C Function} @ref{375,,PyObject}  *PyRun_FileEx (FILE  *fp, const  char  *filename, int  start, PyObject  *globals, PyObject  *locals, int  closeit)
@emph{Return value: New reference.}
This is a simplified interface to @ref{42ce,,PyRun_FileExFlags()} below, leaving
@emph{flags} set to @code{NULL}.
@end deffn

@geindex PyRun_FileFlags (C function)
@anchor{c-api/veryhigh c PyRun_FileFlags}@anchor{42d0}
@deffn {C Function} @ref{375,,PyObject}  *PyRun_FileFlags (FILE  *fp, const  char  *filename, int  start, PyObject  *globals, PyObject  *locals, PyCompilerFlags  *flags)
@emph{Return value: New reference.}
This is a simplified interface to @ref{42ce,,PyRun_FileExFlags()} below, leaving
@emph{closeit} set to @code{0}.
@end deffn

@geindex PyRun_FileExFlags (C function)
@anchor{c-api/veryhigh c PyRun_FileExFlags}@anchor{42ce}
@deffn {C Function} @ref{375,,PyObject}  *PyRun_FileExFlags (FILE  *fp, const  char  *filename, int  start, PyObject  *globals, PyObject  *locals, int  closeit, PyCompilerFlags  *flags)
@emph{Return value: New reference.}
Similar to @ref{42cc,,PyRun_StringFlags()}, but the Python source code is read from
@emph{fp} instead of an in-memory string. @emph{filename} should be the name of the file,
it is decoded from the @ref{1628,,filesystem encoding and error handler}.
If @emph{closeit} is true, the file is closed before @ref{42ce,,PyRun_FileExFlags()}
returns.
@end deffn

@geindex Py_CompileString (C function)
@anchor{c-api/veryhigh c Py_CompileString}@anchor{34d}
@deffn {C Function} @ref{375,,PyObject}  *Py_CompileString (const  char  *str, const  char  *filename, int  start)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
This is a simplified interface to @ref{42d1,,Py_CompileStringFlags()} below, leaving
@emph{flags} set to @code{NULL}.
@end deffn

@geindex Py_CompileStringFlags (C function)
@anchor{c-api/veryhigh c Py_CompileStringFlags}@anchor{42d1}
@deffn {C Function} @ref{375,,PyObject}  *Py_CompileStringFlags (const  char  *str, const  char  *filename, int  start, PyCompilerFlags  *flags)
@emph{Return value: New reference.}
This is a simplified interface to @ref{42d2,,Py_CompileStringExFlags()} below, with
@emph{optimize} set to @code{-1}.
@end deffn

@geindex Py_CompileStringObject (C function)
@anchor{c-api/veryhigh c Py_CompileStringObject}@anchor{42d3}
@deffn {C Function} @ref{375,,PyObject}  *Py_CompileStringObject (const  char  *str, PyObject  *filename, int  start, PyCompilerFlags  *flags, int  optimize)
@emph{Return value: New reference.}
Parse and compile the Python source code in @emph{str}, returning the resulting code
object.  The start token is given by @emph{start}; this can be used to constrain the
code which can be compiled and should be @code{Py_eval_input},
@code{Py_file_input}, or @code{Py_single_input}.  The filename specified by
@emph{filename} is used to construct the code object and may appear in tracebacks or
@ref{21a,,SyntaxError} exception messages.  This returns @code{NULL} if the code
cannot be parsed or compiled.

The integer @emph{optimize} specifies the optimization level of the compiler; a
value of @code{-1} selects the optimization level of the interpreter as given by
@ref{912,,-O} options.  Explicit levels are @code{0} (no optimization;
@code{__debug__} is true), @code{1} (asserts are removed, @code{__debug__} is false)
or @code{2} (docstrings are removed too).


New in version 3.4.

@end deffn

@geindex Py_CompileStringExFlags (C function)
@anchor{c-api/veryhigh c Py_CompileStringExFlags}@anchor{42d2}
@deffn {C Function} @ref{375,,PyObject}  *Py_CompileStringExFlags (const  char  *str, const  char  *filename, int  start, PyCompilerFlags  *flags, int  optimize)
@emph{Return value: New reference.}
Like @ref{42d3,,Py_CompileStringObject()}, but @emph{filename} is a byte string
decoded from the @ref{1628,,filesystem encoding and error handler}.


New in version 3.2.

@end deffn

@geindex PyEval_EvalCode (C function)
@anchor{c-api/veryhigh c PyEval_EvalCode}@anchor{34e}
@deffn {C Function} @ref{375,,PyObject}  *PyEval_EvalCode (PyObject  *co, PyObject  *globals, PyObject  *locals)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
This is a simplified interface to @ref{4185,,PyEval_EvalCodeEx()}, with just
the code object, and global and local variables.  The other arguments are
set to @code{NULL}.
@end deffn

@geindex PyEval_EvalCodeEx (C function)
@anchor{c-api/veryhigh c PyEval_EvalCodeEx}@anchor{4185}
@deffn {C Function} @ref{375,,PyObject}  *PyEval_EvalCodeEx (PyObject  *co, PyObject  *globals, PyObject  *locals, PyObject  *const  *args, int  argcount, PyObject  *const  *kws, int  kwcount, PyObject  *const  *defs, int  defcount, PyObject  *kwdefs, PyObject  *closure)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Evaluate a precompiled code object, given a particular environment for its
evaluation.  This environment consists of a dictionary of global variables,
a mapping object of local variables, arrays of arguments, keywords and
defaults, a dictionary of default values for @ref{5a2,,keyword-only} arguments and a closure tuple of cells.
@end deffn

@geindex PyFrameObject (C type)
@anchor{c-api/veryhigh c PyFrameObject}@anchor{1f6}
@deffn {C Type} type  PyFrameObject
@emph{ Part of the @ref{35d,,Stable ABI}.}
The C structure of the objects used to describe frame objects.

The structure is only part of the internal C API: fields should not be
access directly. Use getter functions like @ref{1f8,,PyFrame_GetCode()} and
@ref{1f7,,PyFrame_GetBack()}.

Debuggers and profilers can use the limited C API to access this structure.


Changed in version 3.11: The structure moved to the internal C API headers.

@end deffn

@geindex PyEval_EvalFrame (C function)
@anchor{c-api/veryhigh c PyEval_EvalFrame}@anchor{4186}
@deffn {C Function} @ref{375,,PyObject}  *PyEval_EvalFrame (PyFrameObject  *f)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Evaluate an execution frame.  This is a simplified interface to
@ref{bbf,,PyEval_EvalFrameEx()}, for backward compatibility.
@end deffn

@geindex PyEval_EvalFrameEx (C function)
@anchor{c-api/veryhigh c PyEval_EvalFrameEx}@anchor{bbf}
@deffn {C Function} @ref{375,,PyObject}  *PyEval_EvalFrameEx (PyFrameObject  *f, int  throwflag)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
This is the main, unvarnished function of Python interpretation.  The code
object associated with the execution frame @emph{f} is executed, interpreting
bytecode and executing calls as needed.  The additional @emph{throwflag}
parameter can mostly be ignored - if true, then it causes an exception
to immediately be thrown; this is used for the @ref{11be,,throw()}
methods of generator objects.


Changed in version 3.4: This function now includes a debug assertion to help ensure that it
does not silently discard an active exception.

@end deffn

@geindex PyEval_MergeCompilerFlags (C function)
@anchor{c-api/veryhigh c PyEval_MergeCompilerFlags}@anchor{42d4}
@deffn {C Function} int  PyEval_MergeCompilerFlags (PyCompilerFlags  *cf)

This function changes the flags of the current evaluation frame, and returns
true on success, false on failure.
@end deffn

@geindex Py_eval_input (C var)
@anchor{c-api/veryhigh c Py_eval_input}@anchor{42d5}
@deffn {C Variable} int  Py_eval_input

@geindex Py_CompileString()

The start symbol from the Python grammar for isolated expressions; for use with
@ref{34d,,Py_CompileString()}.
@end deffn

@geindex Py_file_input (C var)
@anchor{c-api/veryhigh c Py_file_input}@anchor{42d6}
@deffn {C Variable} int  Py_file_input

@geindex Py_CompileString()

The start symbol from the Python grammar for sequences of statements as read
from a file or other source; for use with @ref{34d,,Py_CompileString()}.  This is
the symbol to use when compiling arbitrarily long Python source code.
@end deffn

@geindex Py_single_input (C var)
@anchor{c-api/veryhigh c Py_single_input}@anchor{42d7}
@deffn {C Variable} int  Py_single_input

@geindex Py_CompileString()

The start symbol from the Python grammar for a single statement; for use with
@ref{34d,,Py_CompileString()}. This is the symbol used for the interactive
interpreter loop.
@end deffn

@geindex PyCompilerFlags (C struct)
@anchor{c-api/veryhigh c PyCompilerFlags}@anchor{5c0}
@deffn {C Type} struct  PyCompilerFlags

This is the structure used to hold compiler flags.  In cases where code is only
being compiled, it is passed as @code{int flags}, and in cases where code is being
executed, it is passed as @code{PyCompilerFlags *flags}.  In this case, @code{from
__future__ import} can modify @emph{flags}.

Whenever @code{PyCompilerFlags *flags} is @code{NULL}, @code{cf_flags} is treated as
equal to @code{0}, and any modification due to @code{from __future__ import} is
discarded.

@geindex PyCompilerFlags.cf_flags (C member)
@anchor{c-api/veryhigh c PyCompilerFlags cf_flags}@anchor{42d8}
@deffn {C Member} int  cf_flags

Compiler flags.
@end deffn

@geindex PyCompilerFlags.cf_feature_version (C member)
@anchor{c-api/veryhigh c PyCompilerFlags cf_feature_version}@anchor{42d9}
@deffn {C Member} int  cf_feature_version

@emph{cf_feature_version} is the minor Python version. It should be
initialized to @code{PY_MINOR_VERSION}.

The field is ignored by default, it is used if and only if
@code{PyCF_ONLY_AST} flag is set in @emph{cf_flags}.
@end deffn


Changed in version 3.8: Added @emph{cf_feature_version} field.

@end deffn

@geindex CO_FUTURE_DIVISION (C var)
@anchor{c-api/veryhigh c CO_FUTURE_DIVISION}@anchor{42da}
@deffn {C Variable} int  CO_FUTURE_DIVISION

This bit can be set in @emph{flags} to cause division operator @code{/} to be
interpreted as “true division” according to 
@geindex Python Enhancement Proposals; PEP 238
PEP 238@footnote{https://www.python.org/dev/peps/pep-0238}.
@end deffn

@node Reference Counting,Exception Handling,The Very High Level Layer,Python/C API Reference Manual
@anchor{c-api/refcounting doc}@anchor{42db}@anchor{c-api/refcounting countingrefs}@anchor{42dc}@anchor{c-api/refcounting reference-counting}@anchor{42dd}
@section Reference Counting


The macros in this section are used for managing reference counts of Python
objects.

@geindex Py_INCREF (C function)
@anchor{c-api/refcounting c Py_INCREF}@anchor{372}
@deffn {C Function} void  Py_INCREF (PyObject  *o)

Increment the reference count for object @emph{o}.

This function is usually used to convert a @ref{4112,,borrowed reference} to a
@ref{1fc,,strong reference} in-place. The @ref{36a,,Py_NewRef()} function can be
used to create a new @ref{1fc,,strong reference}.

The object must not be @code{NULL}; if you aren’t sure that it isn’t
@code{NULL}, use @ref{57b,,Py_XINCREF()}.
@end deffn

@geindex Py_XINCREF (C function)
@anchor{c-api/refcounting c Py_XINCREF}@anchor{57b}
@deffn {C Function} void  Py_XINCREF (PyObject  *o)

Increment the reference count for object @emph{o}.  The object may be @code{NULL}, in
which case the macro has no effect.

See also @ref{36b,,Py_XNewRef()}.
@end deffn

@geindex Py_NewRef (C function)
@anchor{c-api/refcounting c Py_NewRef}@anchor{36a}
@deffn {C Function} @ref{375,,PyObject}  *Py_NewRef (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Create a new @ref{1fc,,strong reference} to an object: increment the reference
count of the object @emph{o} and return the object @emph{o}.

When the @ref{1fc,,strong reference} is no longer needed, @ref{373,,Py_DECREF()}
should be called on it to decrement the object reference count.

The object @emph{o} must not be @code{NULL}; use @ref{36b,,Py_XNewRef()} if @emph{o} can be
@code{NULL}.

For example:

@example
Py_INCREF(obj);
self->attr = obj;
@end example

can be written as:

@example
self->attr = Py_NewRef(obj);
@end example

See also @ref{372,,Py_INCREF()}.


New in version 3.10.

@end deffn

@geindex Py_XNewRef (C function)
@anchor{c-api/refcounting c Py_XNewRef}@anchor{36b}
@deffn {C Function} @ref{375,,PyObject}  *Py_XNewRef (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Similar to @ref{36a,,Py_NewRef()}, but the object @emph{o} can be NULL.

If the object @emph{o} is @code{NULL}, the function just returns @code{NULL}.


New in version 3.10.

@end deffn

@geindex Py_DECREF (C function)
@anchor{c-api/refcounting c Py_DECREF}@anchor{373}
@deffn {C Function} void  Py_DECREF (PyObject  *o)

Decrement the reference count for object @emph{o}.

If the reference count reaches zero, the object’s type’s deallocation
function (which must not be @code{NULL}) is invoked.

This function is usually used to delete a @ref{1fc,,strong reference} before
exiting its scope.

The object must not be @code{NULL}; if you aren’t sure that it isn’t @code{NULL},
use @ref{1fd,,Py_XDECREF()}.

@cartouche
@quotation Warning 
The deallocation function can cause arbitrary Python code to be invoked (e.g.
when a class instance with a @code{__del__()} method is deallocated).  While
exceptions in such code are not propagated, the executed code has free access to
all Python global variables.  This means that any object that is reachable from
a global variable should be in a consistent state before @ref{373,,Py_DECREF()} is
invoked.  For example, code to delete an object from a list should copy a
reference to the deleted object in a temporary variable, update the list data
structure, and then call @ref{373,,Py_DECREF()} for the temporary variable.
@end quotation
@end cartouche
@end deffn

@geindex Py_XDECREF (C function)
@anchor{c-api/refcounting c Py_XDECREF}@anchor{1fd}
@deffn {C Function} void  Py_XDECREF (PyObject  *o)

Decrement the reference count for object @emph{o}.  The object may be @code{NULL}, in
which case the macro has no effect; otherwise the effect is the same as for
@ref{373,,Py_DECREF()}, and the same warning applies.
@end deffn

@geindex Py_CLEAR (C function)
@anchor{c-api/refcounting c Py_CLEAR}@anchor{fd2}
@deffn {C Function} void  Py_CLEAR (PyObject  *o)

Decrement the reference count for object @emph{o}.  The object may be @code{NULL}, in
which case the macro has no effect; otherwise the effect is the same as for
@ref{373,,Py_DECREF()}, except that the argument is also set to @code{NULL}.  The warning
for @ref{373,,Py_DECREF()} does not apply with respect to the object passed because
the macro carefully uses a temporary variable and sets the argument to @code{NULL}
before decrementing its reference count.

It is a good idea to use this macro whenever decrementing the reference
count of an object that might be traversed during garbage collection.
@end deffn

The following functions are for runtime dynamic embedding of Python:
@code{Py_IncRef(PyObject *o)}, @code{Py_DecRef(PyObject *o)}. They are
simply exported function versions of @ref{57b,,Py_XINCREF()} and
@ref{1fd,,Py_XDECREF()}, respectively.

The following functions or macros are only for use within the interpreter core:
@code{_Py_Dealloc()}, @code{_Py_ForgetReference()}, @code{_Py_NewReference()},
as well as the global variable @code{_Py_RefTotal}.

@node Exception Handling,Utilities<2>,Reference Counting,Python/C API Reference Manual
@anchor{c-api/exceptions doc}@anchor{42de}@anchor{c-api/exceptions exception-handling}@anchor{42df}@anchor{c-api/exceptions exceptionhandling}@anchor{3b98}
@section Exception Handling


The functions described in this chapter will let you handle and raise Python
exceptions.  It is important to understand some of the basics of Python
exception handling.  It works somewhat like the POSIX @code{errno} variable:
there is a global indicator (per thread) of the last error that occurred.  Most
C API functions don’t clear this on success, but will set it to indicate the
cause of the error on failure.  Most C API functions also return an error
indicator, usually @code{NULL} if they are supposed to return a pointer, or @code{-1}
if they return an integer (exception: the PyArg_* functions
return @code{1} for success and @code{0} for failure).

Concretely, the error indicator consists of three object pointers: the
exception’s type, the exception’s value, and the traceback object.  Any
of those pointers can be @code{NULL} if non-set (although some combinations are
forbidden, for example you can’t have a non-@code{NULL} traceback if the exception
type is @code{NULL}).

When a function must fail because some function it called failed, it generally
doesn’t set the error indicator; the function it called already set it.  It is
responsible for either handling the error and clearing the exception or
returning after cleaning up any resources it holds (such as object references or
memory allocations); it should @emph{not} continue normally if it is not prepared to
handle the error.  If returning due to an error, it is important to indicate to
the caller that an error has been set.  If the error is not handled or carefully
propagated, additional calls into the Python/C API may not behave as intended
and may fail in mysterious ways.

@cartouche
@quotation Note 
The error indicator is @strong{not} the result of @ref{18f,,sys.exc_info()}.
The former corresponds to an exception that is not yet caught (and is
therefore still propagating), while the latter returns an exception after
it is caught (and has therefore stopped propagating).
@end quotation
@end cartouche

@menu
* Printing and clearing:: 
* Raising exceptions:: 
* Issuing warnings:: 
* Querying the error indicator:: 
* Signal Handling: Signal Handling<2>. 
* Exception Classes:: 
* Exception Objects:: 
* Unicode Exception Objects:: 
* Recursion Control:: 
* Standard Exceptions:: 
* Standard Warning Categories:: 

@end menu

@node Printing and clearing,Raising exceptions,,Exception Handling
@anchor{c-api/exceptions printing-and-clearing}@anchor{42e0}
@subsection Printing and clearing


@geindex PyErr_Clear (C function)
@anchor{c-api/exceptions c PyErr_Clear}@anchor{bc3}
@deffn {C Function} void  PyErr_Clear ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Clear the error indicator.  If the error indicator is not set, there is no
effect.
@end deffn

@geindex PyErr_PrintEx (C function)
@anchor{c-api/exceptions c PyErr_PrintEx}@anchor{4175}
@deffn {C Function} void  PyErr_PrintEx (int  set_sys_last_vars)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Print a standard traceback to @code{sys.stderr} and clear the error indicator.
@strong{Unless} the error is a @code{SystemExit}, in that case no traceback is
printed and the Python process will exit with the error code specified by
the @code{SystemExit} instance.

Call this function @strong{only} when the error indicator is set.  Otherwise it
will cause a fatal error!

If @emph{set_sys_last_vars} is nonzero, the variables @ref{e66,,sys.last_type},
@ref{3b74,,sys.last_value} and @ref{3a58,,sys.last_traceback} will be set to the
type, value and traceback of the printed exception, respectively.
@end deffn

@geindex PyErr_Print (C function)
@anchor{c-api/exceptions c PyErr_Print}@anchor{11c8}
@deffn {C Function} void  PyErr_Print ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Alias for @code{PyErr_PrintEx(1)}.
@end deffn

@geindex PyErr_WriteUnraisable (C function)
@anchor{c-api/exceptions c PyErr_WriteUnraisable}@anchor{128d}
@deffn {C Function} void  PyErr_WriteUnraisable (PyObject  *obj)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Call @ref{18d,,sys.unraisablehook()} using the current exception and @emph{obj}
argument.

This utility function prints a warning message to @code{sys.stderr} when an
exception has been set but it is impossible for the interpreter to actually
raise the exception.  It is used, for example, when an exception occurs in an
@code{__del__()} method.

The function is called with a single argument @emph{obj} that identifies the context
in which the unraisable exception occurred. If possible,
the repr of @emph{obj} will be printed in the warning message.

An exception must be set when calling this function.
@end deffn

@node Raising exceptions,Issuing warnings,Printing and clearing,Exception Handling
@anchor{c-api/exceptions raising-exceptions}@anchor{42e1}
@subsection Raising exceptions


These functions help you set the current thread’s error indicator.
For convenience, some of these functions will always return a
@code{NULL} pointer for use in a @code{return} statement.

@geindex PyErr_SetString (C function)
@anchor{c-api/exceptions c PyErr_SetString}@anchor{4071}
@deffn {C Function} void  PyErr_SetString (PyObject  *type, const  char  *message)
@emph{ Part of the @ref{35d,,Stable ABI}.}
This is the most common way to set the error indicator.  The first argument
specifies the exception type; it is normally one of the standard exceptions,
e.g. @code{PyExc_RuntimeError}.  You need not increment its reference count.
The second argument is an error message; it is decoded from @code{'utf-8'}.
@end deffn

@geindex PyErr_SetObject (C function)
@anchor{c-api/exceptions c PyErr_SetObject}@anchor{4073}
@deffn {C Function} void  PyErr_SetObject (PyObject  *type, PyObject  *value)
@emph{ Part of the @ref{35d,,Stable ABI}.}
This function is similar to @ref{4071,,PyErr_SetString()} but lets you specify an
arbitrary Python object for the “value” of the exception.
@end deffn

@geindex PyErr_Format (C function)
@anchor{c-api/exceptions c PyErr_Format}@anchor{a23}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_Format (PyObject  *exception, const  char  *format, ...)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI}.}
This function sets the error indicator and returns @code{NULL}.  @emph{exception}
should be a Python exception class.  The @emph{format} and subsequent
parameters help format the error message; they have the same meaning and
values as in @ref{398,,PyUnicode_FromFormat()}. @emph{format} is an ASCII-encoded
string.
@end deffn

@geindex PyErr_FormatV (C function)
@anchor{c-api/exceptions c PyErr_FormatV}@anchor{a22}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_FormatV (PyObject  *exception, const  char  *format, va_list  vargs)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.5.}
Same as @ref{a23,,PyErr_Format()}, but taking a @code{va_list} argument rather
than a variable number of arguments.


New in version 3.5.

@end deffn

@geindex PyErr_SetNone (C function)
@anchor{c-api/exceptions c PyErr_SetNone}@anchor{4180}
@deffn {C Function} void  PyErr_SetNone (PyObject  *type)
@emph{ Part of the @ref{35d,,Stable ABI}.}
This is a shorthand for @code{PyErr_SetObject(type, Py_None)}.
@end deffn

@geindex PyErr_BadArgument (C function)
@anchor{c-api/exceptions c PyErr_BadArgument}@anchor{4171}
@deffn {C Function} int  PyErr_BadArgument ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
This is a shorthand for @code{PyErr_SetString(PyExc_TypeError, message)}, where
@emph{message} indicates that a built-in operation was invoked with an illegal
argument.  It is mostly for internal use.
@end deffn

@geindex PyErr_NoMemory (C function)
@anchor{c-api/exceptions c PyErr_NoMemory}@anchor{4074}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_NoMemory ()
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI}.}
This is a shorthand for @code{PyErr_SetNone(PyExc_MemoryError)}; it returns @code{NULL}
so an object allocation function can write @code{return PyErr_NoMemory();} when it
runs out of memory.
@end deffn

@geindex PyErr_SetFromErrno (C function)
@anchor{c-api/exceptions c PyErr_SetFromErrno}@anchor{4072}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_SetFromErrno (PyObject  *type)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex strerror()

This is a convenience function to raise an exception when a C library function
has returned an error and set the C variable @code{errno}.  It constructs a
tuple object whose first item is the integer @code{errno} value and whose
second item is the corresponding error message (gotten from @code{strerror()}),
and then calls @code{PyErr_SetObject(type, object)}.  On Unix, when the
@code{errno} value is @code{EINTR}, indicating an interrupted system call,
this calls @ref{4173,,PyErr_CheckSignals()}, and if that set the error indicator,
leaves it set to that.  The function always returns @code{NULL}, so a wrapper
function around a system call can write @code{return PyErr_SetFromErrno(type);}
when the system call returns an error.
@end deffn

@geindex PyErr_SetFromErrnoWithFilenameObject (C function)
@anchor{c-api/exceptions c PyErr_SetFromErrnoWithFilenameObject}@anchor{417b}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_SetFromErrnoWithFilenameObject (PyObject  *type, PyObject  *filenameObject)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Similar to @ref{4072,,PyErr_SetFromErrno()}, with the additional behavior that if
@emph{filenameObject} is not @code{NULL}, it is passed to the constructor of @emph{type} as
a third parameter.  In the case of @ref{2a1,,OSError} exception,
this is used to define the @code{filename} attribute of the
exception instance.
@end deffn

@geindex PyErr_SetFromErrnoWithFilenameObjects (C function)
@anchor{c-api/exceptions c PyErr_SetFromErrnoWithFilenameObjects}@anchor{417c}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_SetFromErrnoWithFilenameObjects (PyObject  *type, PyObject  *filenameObject, PyObject  *filenameObject2)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Similar to @ref{417b,,PyErr_SetFromErrnoWithFilenameObject()}, but takes a second
filename object, for raising errors when a function that takes two filenames
fails.


New in version 3.4.

@end deffn

@geindex PyErr_SetFromErrnoWithFilename (C function)
@anchor{c-api/exceptions c PyErr_SetFromErrnoWithFilename}@anchor{417a}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_SetFromErrnoWithFilename (PyObject  *type, const  char  *filename)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Similar to @ref{417b,,PyErr_SetFromErrnoWithFilenameObject()}, but the filename
is given as a C string.  @emph{filename} is decoded from the @ref{1628,,filesystem encoding and error handler}.
@end deffn

@geindex PyErr_SetFromWindowsErr (C function)
@anchor{c-api/exceptions c PyErr_SetFromWindowsErr}@anchor{417d}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_SetFromWindowsErr (int  ierr)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI} on Windows since version 3.7.}
This is a convenience function to raise @ref{bea,,WindowsError}. If called with
@emph{ierr} of 0, the error code returned by a call to @code{GetLastError()}
is used instead.  It calls the Win32 function @code{FormatMessage()} to retrieve
the Windows description of error code given by @emph{ierr} or @code{GetLastError()},
then it constructs a tuple object whose first item is the @emph{ierr} value and whose
second item is the corresponding error message (gotten from
@code{FormatMessage()}), and then calls @code{PyErr_SetObject(PyExc_WindowsError,
object)}. This function always returns @code{NULL}.

@ref{1629,,Availability}: Windows.
@end deffn

@geindex PyErr_SetExcFromWindowsErr (C function)
@anchor{c-api/exceptions c PyErr_SetExcFromWindowsErr}@anchor{4176}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_SetExcFromWindowsErr (PyObject  *type, int  ierr)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI} on Windows since version 3.7.}
Similar to @ref{417d,,PyErr_SetFromWindowsErr()}, with an additional parameter
specifying the exception type to be raised.

@ref{1629,,Availability}: Windows.
@end deffn

@geindex PyErr_SetFromWindowsErrWithFilename (C function)
@anchor{c-api/exceptions c PyErr_SetFromWindowsErrWithFilename}@anchor{417e}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_SetFromWindowsErrWithFilename (int  ierr, const  char  *filename)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI} on Windows since version 3.7.}
Similar to @code{PyErr_SetFromWindowsErrWithFilenameObject()}, but the
filename is given as a C string.  @emph{filename} is decoded from the filesystem
encoding (@ref{798,,os.fsdecode()}).

@ref{1629,,Availability}: Windows.
@end deffn

@geindex PyErr_SetExcFromWindowsErrWithFilenameObject (C function)
@anchor{c-api/exceptions c PyErr_SetExcFromWindowsErrWithFilenameObject}@anchor{4178}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_SetExcFromWindowsErrWithFilenameObject (PyObject  *type, int  ierr, PyObject  *filename)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI} on Windows since version 3.7.}
Similar to @code{PyErr_SetFromWindowsErrWithFilenameObject()}, with an
additional parameter specifying the exception type to be raised.

@ref{1629,,Availability}: Windows.
@end deffn

@geindex PyErr_SetExcFromWindowsErrWithFilenameObjects (C function)
@anchor{c-api/exceptions c PyErr_SetExcFromWindowsErrWithFilenameObjects}@anchor{4179}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_SetExcFromWindowsErrWithFilenameObjects (PyObject  *type, int  ierr, PyObject  *filename, PyObject  *filename2)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI} on Windows since version 3.7.}
Similar to @ref{4178,,PyErr_SetExcFromWindowsErrWithFilenameObject()},
but accepts a second filename object.

@ref{1629,,Availability}: Windows.


New in version 3.4.

@end deffn

@geindex PyErr_SetExcFromWindowsErrWithFilename (C function)
@anchor{c-api/exceptions c PyErr_SetExcFromWindowsErrWithFilename}@anchor{4177}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_SetExcFromWindowsErrWithFilename (PyObject  *type, int  ierr, const  char  *filename)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI} on Windows since version 3.7.}
Similar to @ref{417e,,PyErr_SetFromWindowsErrWithFilename()}, with an additional
parameter specifying the exception type to be raised.

@ref{1629,,Availability}: Windows.
@end deffn

@geindex PyErr_SetImportError (C function)
@anchor{c-api/exceptions c PyErr_SetImportError}@anchor{88e}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_SetImportError (PyObject  *msg, PyObject  *name, PyObject  *path)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
This is a convenience function to raise @ref{3b8,,ImportError}. @emph{msg} will be
set as the exception’s message string. @emph{name} and @emph{path}, both of which can
be @code{NULL}, will be set as the @ref{3b8,,ImportError}’s respective @code{name}
and @code{path} attributes.


New in version 3.3.

@end deffn

@geindex PyErr_SetImportErrorSubclass (C function)
@anchor{c-api/exceptions c PyErr_SetImportErrorSubclass}@anchor{864}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_SetImportErrorSubclass (PyObject  *exception, PyObject  *msg, PyObject  *name, PyObject  *path)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.6.}
Much like @ref{88e,,PyErr_SetImportError()} but this function allows for
specifying a subclass of @ref{3b8,,ImportError} to raise.


New in version 3.6.

@end deffn

@geindex PyErr_SyntaxLocationObject (C function)
@anchor{c-api/exceptions c PyErr_SyntaxLocationObject}@anchor{42e2}
@deffn {C Function} void  PyErr_SyntaxLocationObject (PyObject  *filename, int  lineno, int  col_offset)

Set file, line, and offset information for the current exception.  If the
current exception is not a @ref{21a,,SyntaxError}, then it sets additional
attributes, which make the exception printing subsystem think the exception
is a @ref{21a,,SyntaxError}.


New in version 3.4.

@end deffn

@geindex PyErr_SyntaxLocationEx (C function)
@anchor{c-api/exceptions c PyErr_SyntaxLocationEx}@anchor{4182}
@deffn {C Function} void  PyErr_SyntaxLocationEx (const  char  *filename, int  lineno, int  col_offset)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Like @ref{42e2,,PyErr_SyntaxLocationObject()}, but @emph{filename} is a byte string
decoded from the @ref{1628,,filesystem encoding and error handler}.


New in version 3.2.

@end deffn

@geindex PyErr_SyntaxLocation (C function)
@anchor{c-api/exceptions c PyErr_SyntaxLocation}@anchor{4181}
@deffn {C Function} void  PyErr_SyntaxLocation (const  char  *filename, int  lineno)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Like @ref{4182,,PyErr_SyntaxLocationEx()}, but the col_offset parameter is
omitted.
@end deffn

@geindex PyErr_BadInternalCall (C function)
@anchor{c-api/exceptions c PyErr_BadInternalCall}@anchor{4172}
@deffn {C Function} void  PyErr_BadInternalCall ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
This is a shorthand for @code{PyErr_SetString(PyExc_SystemError, message)},
where @emph{message} indicates that an internal operation (e.g. a Python/C API
function) was invoked with an illegal argument.  It is mostly for internal
use.
@end deffn

@node Issuing warnings,Querying the error indicator,Raising exceptions,Exception Handling
@anchor{c-api/exceptions issuing-warnings}@anchor{42e3}
@subsection Issuing warnings


Use these functions to issue warnings from C code.  They mirror similar
functions exported by the Python @ref{12d,,warnings} module.  They normally
print a warning message to @emph{sys.stderr}; however, it is
also possible that the user has specified that warnings are to be turned into
errors, and in that case they will raise an exception.  It is also possible that
the functions raise an exception because of a problem with the warning machinery.
The return value is @code{0} if no exception is raised, or @code{-1} if an exception
is raised.  (It is not possible to determine whether a warning message is
actually printed, nor what the reason is for the exception; this is
intentional.)  If an exception is raised, the caller should do its normal
exception handling (for example, @ref{373,,Py_DECREF()} owned references and return
an error value).

@geindex PyErr_WarnEx (C function)
@anchor{c-api/exceptions c PyErr_WarnEx}@anchor{fab}
@deffn {C Function} int  PyErr_WarnEx (PyObject  *category, const  char  *message, Py_ssize_t  stack_level)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Issue a warning message.  The @emph{category} argument is a warning category (see
below) or @code{NULL}; the @emph{message} argument is a UTF-8 encoded string.  @emph{stack_level} is a
positive number giving a number of stack frames; the warning will be issued from
the  currently executing line of code in that stack frame.  A @emph{stack_level} of 1
is the function calling @ref{fab,,PyErr_WarnEx()}, 2 is  the function above that,
and so forth.

Warning categories must be subclasses of @code{PyExc_Warning};
@code{PyExc_Warning} is a subclass of @code{PyExc_Exception};
the default warning category is @code{PyExc_RuntimeWarning}. The standard
Python warning categories are available as global variables whose names are
enumerated at @ref{42e4,,Standard Warning Categories}.

For information about warning control, see the documentation for the
@ref{12d,,warnings} module and the @ref{3a8,,-W} option in the command line
documentation.  There is no C API for warning control.
@end deffn

@geindex PyErr_WarnExplicitObject (C function)
@anchor{c-api/exceptions c PyErr_WarnExplicitObject}@anchor{42e5}
@deffn {C Function} int  PyErr_WarnExplicitObject (PyObject  *category, PyObject  *message, PyObject  *filename, int  lineno, PyObject  *module, PyObject  *registry)

Issue a warning message with explicit control over all warning attributes.  This
is a straightforward wrapper around the Python function
@ref{84b,,warnings.warn_explicit()}, see there for more information.  The @emph{module}
and @emph{registry} arguments may be set to @code{NULL} to get the default effect
described there.


New in version 3.4.

@end deffn

@geindex PyErr_WarnExplicit (C function)
@anchor{c-api/exceptions c PyErr_WarnExplicit}@anchor{4183}
@deffn {C Function} int  PyErr_WarnExplicit (PyObject  *category, const  char  *message, const  char  *filename, int  lineno, const  char  *module, PyObject  *registry)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Similar to @ref{42e5,,PyErr_WarnExplicitObject()} except that @emph{message} and
@emph{module} are UTF-8 encoded strings, and @emph{filename} is decoded from the
@ref{1628,,filesystem encoding and error handler}.
@end deffn

@geindex PyErr_WarnFormat (C function)
@anchor{c-api/exceptions c PyErr_WarnFormat}@anchor{4184}
@deffn {C Function} int  PyErr_WarnFormat (PyObject  *category, Py_ssize_t  stack_level, const  char  *format, ...)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Function similar to @ref{fab,,PyErr_WarnEx()}, but use
@ref{398,,PyUnicode_FromFormat()} to format the warning message.  @emph{format} is
an ASCII-encoded string.


New in version 3.2.

@end deffn

@geindex PyErr_ResourceWarning (C function)
@anchor{c-api/exceptions c PyErr_ResourceWarning}@anchor{865}
@deffn {C Function} int  PyErr_ResourceWarning (PyObject  *source, Py_ssize_t  stack_level, const  char  *format, ...)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.6.}
Function similar to @ref{4184,,PyErr_WarnFormat()}, but @emph{category} is
@ref{77b,,ResourceWarning} and it passes @emph{source} to @code{warnings.WarningMessage()}.


New in version 3.6.

@end deffn

@node Querying the error indicator,Signal Handling<2>,Issuing warnings,Exception Handling
@anchor{c-api/exceptions querying-the-error-indicator}@anchor{42e6}
@subsection Querying the error indicator


@geindex PyErr_Occurred (C function)
@anchor{c-api/exceptions c PyErr_Occurred}@anchor{116a}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_Occurred ()
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Test whether the error indicator is set.  If set, return the exception @emph{type}
(the first argument to the last call to one of the PyErr_Set*
functions or to @ref{40be,,PyErr_Restore()}).  If not set, return @code{NULL}.  You do not
own a reference to the return value, so you do not need to @ref{373,,Py_DECREF()}
it.

The caller must hold the GIL.

@cartouche
@quotation Note 
Do not compare the return value to a specific exception; use
@ref{411a,,PyErr_ExceptionMatches()} instead, shown below.  (The comparison could
easily fail since the exception may be an instance instead of a class, in the
case of a class exception, or it may be a subclass of the expected exception.)
@end quotation
@end cartouche
@end deffn

@geindex PyErr_ExceptionMatches (C function)
@anchor{c-api/exceptions c PyErr_ExceptionMatches}@anchor{411a}
@deffn {C Function} int  PyErr_ExceptionMatches (PyObject  *exc)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Equivalent to @code{PyErr_GivenExceptionMatches(PyErr_Occurred(), exc)}.  This
should only be called when an exception is actually set; a memory access
violation will occur if no exception has been raised.
@end deffn

@geindex PyErr_GivenExceptionMatches (C function)
@anchor{c-api/exceptions c PyErr_GivenExceptionMatches}@anchor{4174}
@deffn {C Function} int  PyErr_GivenExceptionMatches (PyObject  *given, PyObject  *exc)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return true if the @emph{given} exception matches the exception type in @emph{exc}.  If
@emph{exc} is a class object, this also returns true when @emph{given} is an instance
of a subclass.  If @emph{exc} is a tuple, all exception types in the tuple (and
recursively in subtuples) are searched for a match.
@end deffn

@geindex PyErr_Fetch (C function)
@anchor{c-api/exceptions c PyErr_Fetch}@anchor{bc2}
@deffn {C Function} void  PyErr_Fetch (PyObject  **ptype, PyObject  **pvalue, PyObject  **ptraceback)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Retrieve the error indicator into three variables whose addresses are passed.
If the error indicator is not set, set all three variables to @code{NULL}.  If it is
set, it will be cleared and you own a reference to each object retrieved.  The
value and traceback object may be @code{NULL} even when the type object is not.

@cartouche
@quotation Note 
This function is normally only used by code that needs to catch exceptions or
by code that needs to save and restore the error indicator temporarily, e.g.:

@example
@{
   PyObject *type, *value, *traceback;
   PyErr_Fetch(&type, &value, &traceback);

   /* ... code that might produce other errors ... */

   PyErr_Restore(type, value, traceback);
@}
@end example
@end quotation
@end cartouche
@end deffn

@geindex PyErr_Restore (C function)
@anchor{c-api/exceptions c PyErr_Restore}@anchor{40be}
@deffn {C Function} void  PyErr_Restore (PyObject  *type, PyObject  *value, PyObject  *traceback)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set  the error indicator from the three objects.  If the error indicator is
already set, it is cleared first.  If the objects are @code{NULL}, the error
indicator is cleared.  Do not pass a @code{NULL} type and non-@code{NULL} value or
traceback.  The exception type should be a class.  Do not pass an invalid
exception type or value. (Violating these rules will cause subtle problems
later.)  This call takes away a reference to each object: you must own a
reference to each object before the call and after the call you no longer own
these references.  (If you don’t understand this, don’t use this function.  I
warned you.)

@cartouche
@quotation Note 
This function is normally only used by code that needs to save and restore the
error indicator temporarily.  Use @ref{bc2,,PyErr_Fetch()} to save the current
error indicator.
@end quotation
@end cartouche
@end deffn

@geindex PyErr_NormalizeException (C function)
@anchor{c-api/exceptions c PyErr_NormalizeException}@anchor{128e}
@deffn {C Function} void  PyErr_NormalizeException (PyObject  **exc, PyObject  **val, PyObject  **tb)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Under certain circumstances, the values returned by @ref{bc2,,PyErr_Fetch()} below
can be “unnormalized”, meaning that @code{*exc} is a class object but @code{*val} is
not an instance of the  same class.  This function can be used to instantiate
the class in that case.  If the values are already normalized, nothing happens.
The delayed normalization is implemented to improve performance.

@cartouche
@quotation Note 
This function @emph{does not} implicitly set the @code{__traceback__}
attribute on the exception value. If setting the traceback
appropriately is desired, the following additional snippet is needed:

@example
if (tb != NULL) @{
  PyException_SetTraceback(val, tb);
@}
@end example
@end quotation
@end cartouche
@end deffn

@geindex PyErr_GetExcInfo (C function)
@anchor{c-api/exceptions c PyErr_GetExcInfo}@anchor{1d2}
@deffn {C Function} void  PyErr_GetExcInfo (PyObject  **ptype, PyObject  **pvalue, PyObject  **ptraceback)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Retrieve the exception info, as known from @code{sys.exc_info()}.  This refers
to an exception that was @emph{already caught}, not to an exception that was
freshly raised.  Returns new references for the three objects, any of which
may be @code{NULL}.  Does not modify the exception info state.

@cartouche
@quotation Note 
This function is not normally used by code that wants to handle exceptions.
Rather, it can be used when code needs to save and restore the exception
state temporarily.  Use @ref{1d1,,PyErr_SetExcInfo()} to restore or clear the
exception state.
@end quotation
@end cartouche


New in version 3.3.

@end deffn

@geindex PyErr_SetExcInfo (C function)
@anchor{c-api/exceptions c PyErr_SetExcInfo}@anchor{1d1}
@deffn {C Function} void  PyErr_SetExcInfo (PyObject  *type, PyObject  *value, PyObject  *traceback)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Set the exception info, as known from @code{sys.exc_info()}.  This refers
to an exception that was @emph{already caught}, not to an exception that was
freshly raised.  This function steals the references of the arguments.
To clear the exception state, pass @code{NULL} for all three arguments.

@cartouche
@quotation Note 
This function is not normally used by code that wants to handle exceptions.
Rather, it can be used when code needs to save and restore the exception
state temporarily.  Use @ref{1d2,,PyErr_GetExcInfo()} to read the exception
state.
@end quotation
@end cartouche


New in version 3.3.



Changed in version 3.11: The @code{type} and @code{traceback} arguments are no longer used and
can be NULL. The interpreter now derives them from the exception
instance (the @code{value} argument). The function still steals
references of all three arguments.

@end deffn

@node Signal Handling<2>,Exception Classes,Querying the error indicator,Exception Handling
@anchor{c-api/exceptions signal-handling}@anchor{42e7}
@subsection Signal Handling


@geindex PyErr_CheckSignals (C function)
@anchor{c-api/exceptions c PyErr_CheckSignals}@anchor{4173}
@deffn {C Function} int  PyErr_CheckSignals ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex module; signal
@geindex SIGINT
@geindex KeyboardInterrupt (built-in exception)

This function interacts with Python’s signal handling.

If the function is called from the main thread and under the main Python
interpreter, it checks whether a signal has been sent to the processes
and if so, invokes the corresponding signal handler.  If the @ref{e5,,signal}
module is supported, this can invoke a signal handler written in Python.

The function attempts to handle all pending signals, and then returns @code{0}.
However, if a Python signal handler raises an exception, the error
indicator is set and the function returns @code{-1} immediately (such that
other pending signals may not have been handled yet: they will be on the
next @ref{4173,,PyErr_CheckSignals()} invocation).

If the function is called from a non-main thread, or under a non-main
Python interpreter, it does nothing and returns @code{0}.

This function can be called by long-running C code that wants to
be interruptible by user requests (such as by pressing Ctrl-C).

@cartouche
@quotation Note 
The default Python signal handler for @code{SIGINT} raises the
@ref{4ca,,KeyboardInterrupt} exception.
@end quotation
@end cartouche
@end deffn

@geindex PyErr_SetInterrupt (C function)
@anchor{c-api/exceptions c PyErr_SetInterrupt}@anchor{417f}
@deffn {C Function} void  PyErr_SetInterrupt ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex module; signal
@geindex SIGINT
@geindex KeyboardInterrupt (built-in exception)

Simulate the effect of a @code{SIGINT} signal arriving.
This is equivalent to @code{PyErr_SetInterruptEx(SIGINT)}.

@cartouche
@quotation Note 
This function is async-signal-safe.  It can be called without
the @ref{385,,GIL} and from a C signal handler.
@end quotation
@end cartouche
@end deffn

@geindex PyErr_SetInterruptEx (C function)
@anchor{c-api/exceptions c PyErr_SetInterruptEx}@anchor{371}
@deffn {C Function} int  PyErr_SetInterruptEx (int  signum)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
@geindex module; signal
@geindex KeyboardInterrupt (built-in exception)

Simulate the effect of a signal arriving. The next time
@ref{4173,,PyErr_CheckSignals()} is called,  the Python signal handler for
the given signal number will be called.

This function can be called by C code that sets up its own signal handling
and wants Python signal handlers to be invoked as expected when an
interruption is requested (for example when the user presses Ctrl-C
to interrupt an operation).

If the given signal isn’t handled by Python (it was set to
@ref{295d,,signal.SIG_DFL} or @ref{295e,,signal.SIG_IGN}), it will be ignored.

If @emph{signum} is outside of the allowed range of signal numbers, @code{-1}
is returned.  Otherwise, @code{0} is returned.  The error indicator is
never changed by this function.

@cartouche
@quotation Note 
This function is async-signal-safe.  It can be called without
the @ref{385,,GIL} and from a C signal handler.
@end quotation
@end cartouche


New in version 3.10.

@end deffn

@geindex PySignal_SetWakeupFd (C function)
@anchor{c-api/exceptions c PySignal_SetWakeupFd}@anchor{f41}
@deffn {C Function} int  PySignal_SetWakeupFd (int  fd)

This utility function specifies a file descriptor to which the signal number
is written as a single byte whenever a signal is received. @emph{fd} must be
non-blocking. It returns the previous such file descriptor.

The value @code{-1} disables the feature; this is the initial state.
This is equivalent to @ref{69d,,signal.set_wakeup_fd()} in Python, but without any
error checking.  @emph{fd} should be a valid file descriptor.  The function should
only be called from the main thread.


Changed in version 3.5: On Windows, the function now also supports socket handles.

@end deffn

@node Exception Classes,Exception Objects,Signal Handling<2>,Exception Handling
@anchor{c-api/exceptions exception-classes}@anchor{42e8}
@subsection Exception Classes


@geindex PyErr_NewException (C function)
@anchor{c-api/exceptions c PyErr_NewException}@anchor{e1a}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_NewException (const  char  *name, PyObject  *base, PyObject  *dict)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
This utility function creates and returns a new exception class. The @emph{name}
argument must be the name of the new exception, a C string of the form
@code{module.classname}.  The @emph{base} and @emph{dict} arguments are normally @code{NULL}.
This creates a class object derived from @ref{4d8,,Exception} (accessible in C as
@code{PyExc_Exception}).

The @code{__module__} attribute of the new class is set to the first part (up
to the last dot) of the @emph{name} argument, and the class name is set to the last
part (after the last dot).  The @emph{base} argument can be used to specify alternate
base classes; it can either be only one class or a tuple of classes. The @emph{dict}
argument can be used to specify a dictionary of class variables and methods.
@end deffn

@geindex PyErr_NewExceptionWithDoc (C function)
@anchor{c-api/exceptions c PyErr_NewExceptionWithDoc}@anchor{e19}
@deffn {C Function} @ref{375,,PyObject}  *PyErr_NewExceptionWithDoc (const  char  *name, const  char  *doc, PyObject  *base, PyObject  *dict)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Same as @ref{e1a,,PyErr_NewException()}, except that the new exception class can
easily be given a docstring: If @emph{doc} is non-@code{NULL}, it will be used as the
docstring for the exception class.


New in version 3.2.

@end deffn

@node Exception Objects,Unicode Exception Objects,Exception Classes,Exception Handling
@anchor{c-api/exceptions exception-objects}@anchor{42e9}
@subsection Exception Objects


@geindex PyException_GetTraceback (C function)
@anchor{c-api/exceptions c PyException_GetTraceback}@anchor{4190}
@deffn {C Function} @ref{375,,PyObject}  *PyException_GetTraceback (PyObject  *ex)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the traceback associated with the exception as a new reference, as
accessible from Python through @code{__traceback__}.  If there is no
traceback associated, this returns @code{NULL}.
@end deffn

@geindex PyException_SetTraceback (C function)
@anchor{c-api/exceptions c PyException_SetTraceback}@anchor{128f}
@deffn {C Function} int  PyException_SetTraceback (PyObject  *ex, PyObject  *tb)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the traceback associated with the exception to @emph{tb}.  Use @code{Py_None} to
clear it.
@end deffn

@geindex PyException_GetContext (C function)
@anchor{c-api/exceptions c PyException_GetContext}@anchor{418f}
@deffn {C Function} @ref{375,,PyObject}  *PyException_GetContext (PyObject  *ex)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the context (another exception instance during whose handling @emph{ex} was
raised) associated with the exception as a new reference, as accessible from
Python through @code{__context__}.  If there is no context associated, this
returns @code{NULL}.
@end deffn

@geindex PyException_SetContext (C function)
@anchor{c-api/exceptions c PyException_SetContext}@anchor{4192}
@deffn {C Function} void  PyException_SetContext (PyObject  *ex, PyObject  *ctx)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the context associated with the exception to @emph{ctx}.  Use @code{NULL} to clear
it.  There is no type check to make sure that @emph{ctx} is an exception instance.
This steals a reference to @emph{ctx}.
@end deffn

@geindex PyException_GetCause (C function)
@anchor{c-api/exceptions c PyException_GetCause}@anchor{418e}
@deffn {C Function} @ref{375,,PyObject}  *PyException_GetCause (PyObject  *ex)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the cause (either an exception instance, or @ref{182,,None},
set by @code{raise ... from ...}) associated with the exception as a new
reference, as accessible from Python through @code{__cause__}.
@end deffn

@geindex PyException_SetCause (C function)
@anchor{c-api/exceptions c PyException_SetCause}@anchor{4191}
@deffn {C Function} void  PyException_SetCause (PyObject  *ex, PyObject  *cause)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the cause associated with the exception to @emph{cause}.  Use @code{NULL} to clear
it.  There is no type check to make sure that @emph{cause} is either an exception
instance or @ref{182,,None}.  This steals a reference to @emph{cause}.

@code{__suppress_context__} is implicitly set to @code{True} by this function.
@end deffn

@node Unicode Exception Objects,Recursion Control,Exception Objects,Exception Handling
@anchor{c-api/exceptions unicode-exception-objects}@anchor{42ea}@anchor{c-api/exceptions unicodeexceptions}@anchor{42eb}
@subsection Unicode Exception Objects


The following functions are used to create and modify Unicode exceptions from C.

@geindex PyUnicodeDecodeError_Create (C function)
@anchor{c-api/exceptions c PyUnicodeDecodeError_Create}@anchor{4256}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicodeDecodeError_Create (const  char  *encoding, const  char  *object, Py_ssize_t  length, Py_ssize_t  start, Py_ssize_t  end, const  char  *reason)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a @ref{51f,,UnicodeDecodeError} object with the attributes @emph{encoding},
@emph{object}, @emph{length}, @emph{start}, @emph{end} and @emph{reason}. @emph{encoding} and @emph{reason} are
UTF-8 encoded strings.
@end deffn

@geindex PyUnicodeDecodeError_GetEncoding (C function)
@geindex PyUnicodeEncodeError_GetEncoding (C function)
@anchor{c-api/exceptions c PyUnicodeDecodeError_GetEncoding}@anchor{4257}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicodeDecodeError_GetEncoding (PyObject  *exc)
@anchor{c-api/exceptions c PyUnicodeEncodeError_GetEncoding}@anchor{425f}
@deffnx {C Function} @ref{375,,PyObject}  *PyUnicodeEncodeError_GetEncoding (PyObject  *exc)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the @emph{encoding} attribute of the given exception object.
@end deffn

@geindex PyUnicodeDecodeError_GetObject (C function)
@geindex PyUnicodeEncodeError_GetObject (C function)
@geindex PyUnicodeTranslateError_GetObject (C function)
@anchor{c-api/exceptions c PyUnicodeDecodeError_GetObject}@anchor{4259}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicodeDecodeError_GetObject (PyObject  *exc)
@anchor{c-api/exceptions c PyUnicodeEncodeError_GetObject}@anchor{4261}
@deffnx {C Function} @ref{375,,PyObject}  *PyUnicodeEncodeError_GetObject (PyObject  *exc)
@anchor{c-api/exceptions c PyUnicodeTranslateError_GetObject}@anchor{4268}
@deffnx {C Function} @ref{375,,PyObject}  *PyUnicodeTranslateError_GetObject (PyObject  *exc)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the @emph{object} attribute of the given exception object.
@end deffn

@geindex PyUnicodeDecodeError_GetStart (C function)
@geindex PyUnicodeEncodeError_GetStart (C function)
@geindex PyUnicodeTranslateError_GetStart (C function)
@anchor{c-api/exceptions c PyUnicodeDecodeError_GetStart}@anchor{425b}
@deffn {C Function} int  PyUnicodeDecodeError_GetStart (PyObject  *exc, Py_ssize_t  *start)
@anchor{c-api/exceptions c PyUnicodeEncodeError_GetStart}@anchor{4263}
@deffnx {C Function} int  PyUnicodeEncodeError_GetStart (PyObject  *exc, Py_ssize_t  *start)
@anchor{c-api/exceptions c PyUnicodeTranslateError_GetStart}@anchor{426a}
@deffnx {C Function} int  PyUnicodeTranslateError_GetStart (PyObject  *exc, Py_ssize_t  *start)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Get the @emph{start} attribute of the given exception object and place it into
@emph{*start}.  @emph{start} must not be @code{NULL}.  Return @code{0} on success, @code{-1} on
failure.
@end deffn

@geindex PyUnicodeDecodeError_SetStart (C function)
@geindex PyUnicodeEncodeError_SetStart (C function)
@geindex PyUnicodeTranslateError_SetStart (C function)
@anchor{c-api/exceptions c PyUnicodeDecodeError_SetStart}@anchor{425e}
@deffn {C Function} int  PyUnicodeDecodeError_SetStart (PyObject  *exc, Py_ssize_t  start)
@anchor{c-api/exceptions c PyUnicodeEncodeError_SetStart}@anchor{4266}
@deffnx {C Function} int  PyUnicodeEncodeError_SetStart (PyObject  *exc, Py_ssize_t  start)
@anchor{c-api/exceptions c PyUnicodeTranslateError_SetStart}@anchor{426d}
@deffnx {C Function} int  PyUnicodeTranslateError_SetStart (PyObject  *exc, Py_ssize_t  start)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the @emph{start} attribute of the given exception object to @emph{start}.  Return
@code{0} on success, @code{-1} on failure.
@end deffn

@geindex PyUnicodeDecodeError_GetEnd (C function)
@geindex PyUnicodeEncodeError_GetEnd (C function)
@geindex PyUnicodeTranslateError_GetEnd (C function)
@anchor{c-api/exceptions c PyUnicodeDecodeError_GetEnd}@anchor{4258}
@deffn {C Function} int  PyUnicodeDecodeError_GetEnd (PyObject  *exc, Py_ssize_t  *end)
@anchor{c-api/exceptions c PyUnicodeEncodeError_GetEnd}@anchor{4260}
@deffnx {C Function} int  PyUnicodeEncodeError_GetEnd (PyObject  *exc, Py_ssize_t  *end)
@anchor{c-api/exceptions c PyUnicodeTranslateError_GetEnd}@anchor{4267}
@deffnx {C Function} int  PyUnicodeTranslateError_GetEnd (PyObject  *exc, Py_ssize_t  *end)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Get the @emph{end} attribute of the given exception object and place it into
@emph{*end}.  @emph{end} must not be @code{NULL}.  Return @code{0} on success, @code{-1} on
failure.
@end deffn

@geindex PyUnicodeDecodeError_SetEnd (C function)
@geindex PyUnicodeEncodeError_SetEnd (C function)
@geindex PyUnicodeTranslateError_SetEnd (C function)
@anchor{c-api/exceptions c PyUnicodeDecodeError_SetEnd}@anchor{425c}
@deffn {C Function} int  PyUnicodeDecodeError_SetEnd (PyObject  *exc, Py_ssize_t  end)
@anchor{c-api/exceptions c PyUnicodeEncodeError_SetEnd}@anchor{4264}
@deffnx {C Function} int  PyUnicodeEncodeError_SetEnd (PyObject  *exc, Py_ssize_t  end)
@anchor{c-api/exceptions c PyUnicodeTranslateError_SetEnd}@anchor{426b}
@deffnx {C Function} int  PyUnicodeTranslateError_SetEnd (PyObject  *exc, Py_ssize_t  end)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the @emph{end} attribute of the given exception object to @emph{end}.  Return @code{0}
on success, @code{-1} on failure.
@end deffn

@geindex PyUnicodeDecodeError_GetReason (C function)
@geindex PyUnicodeEncodeError_GetReason (C function)
@geindex PyUnicodeTranslateError_GetReason (C function)
@anchor{c-api/exceptions c PyUnicodeDecodeError_GetReason}@anchor{425a}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicodeDecodeError_GetReason (PyObject  *exc)
@anchor{c-api/exceptions c PyUnicodeEncodeError_GetReason}@anchor{4262}
@deffnx {C Function} @ref{375,,PyObject}  *PyUnicodeEncodeError_GetReason (PyObject  *exc)
@anchor{c-api/exceptions c PyUnicodeTranslateError_GetReason}@anchor{4269}
@deffnx {C Function} @ref{375,,PyObject}  *PyUnicodeTranslateError_GetReason (PyObject  *exc)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the @emph{reason} attribute of the given exception object.
@end deffn

@geindex PyUnicodeDecodeError_SetReason (C function)
@geindex PyUnicodeEncodeError_SetReason (C function)
@geindex PyUnicodeTranslateError_SetReason (C function)
@anchor{c-api/exceptions c PyUnicodeDecodeError_SetReason}@anchor{425d}
@deffn {C Function} int  PyUnicodeDecodeError_SetReason (PyObject  *exc, const  char  *reason)
@anchor{c-api/exceptions c PyUnicodeEncodeError_SetReason}@anchor{4265}
@deffnx {C Function} int  PyUnicodeEncodeError_SetReason (PyObject  *exc, const  char  *reason)
@anchor{c-api/exceptions c PyUnicodeTranslateError_SetReason}@anchor{426c}
@deffnx {C Function} int  PyUnicodeTranslateError_SetReason (PyObject  *exc, const  char  *reason)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the @emph{reason} attribute of the given exception object to @emph{reason}.  Return
@code{0} on success, @code{-1} on failure.
@end deffn

@node Recursion Control,Standard Exceptions,Unicode Exception Objects,Exception Handling
@anchor{c-api/exceptions recursion}@anchor{42ec}@anchor{c-api/exceptions recursion-control}@anchor{42ed}
@subsection Recursion Control


These two functions provide a way to perform safe recursive calls at the C
level, both in the core and in extension modules.  They are needed if the
recursive code does not necessarily invoke Python code (which tracks its
recursion depth automatically).
They are also not needed for @emph{tp_call} implementations
because the @ref{42ee,,call protocol} takes care of recursion handling.

@geindex Py_EnterRecursiveCall (C function)
@anchor{c-api/exceptions c Py_EnterRecursiveCall}@anchor{47a}
@deffn {C Function} int  Py_EnterRecursiveCall (const  char  *where)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.9.}
Marks a point where a recursive C-level call is about to be performed.

If @code{USE_STACKCHECK} is defined, this function checks if the OS
stack overflowed using @ref{41f4,,PyOS_CheckStack()}.  In this is the case, it
sets a @ref{10de,,MemoryError} and returns a nonzero value.

The function then checks if the recursion limit is reached.  If this is the
case, a @ref{8c4,,RecursionError} is set and a nonzero value is returned.
Otherwise, zero is returned.

@emph{where} should be a UTF-8 encoded string such as @code{" in instance check"} to
be concatenated to the @ref{8c4,,RecursionError} message caused by the recursion
depth limit.


Changed in version 3.9: This function is now also available in the limited API.

@end deffn

@geindex Py_LeaveRecursiveCall (C function)
@anchor{c-api/exceptions c Py_LeaveRecursiveCall}@anchor{47b}
@deffn {C Function} void  Py_LeaveRecursiveCall (void)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.9.}
Ends a @ref{47a,,Py_EnterRecursiveCall()}.  Must be called once for each
@emph{successful} invocation of @ref{47a,,Py_EnterRecursiveCall()}.


Changed in version 3.9: This function is now also available in the limited API.

@end deffn

Properly implementing @ref{40c1,,tp_repr} for container types requires
special recursion handling.  In addition to protecting the stack,
@ref{40c1,,tp_repr} also needs to track objects to prevent cycles.  The
following two functions facilitate this functionality.  Effectively,
these are the C equivalent to @ref{d95,,reprlib.recursive_repr()}.

@geindex Py_ReprEnter (C function)
@anchor{c-api/exceptions c Py_ReprEnter}@anchor{42a1}
@deffn {C Function} int  Py_ReprEnter (PyObject  *object)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Called at the beginning of the @ref{40c1,,tp_repr} implementation to
detect cycles.

If the object has already been processed, the function returns a
positive integer.  In that case the @ref{40c1,,tp_repr} implementation
should return a string object indicating a cycle.  As examples,
@ref{3b0,,dict} objects return @code{@{...@}} and @ref{43a,,list} objects
return @code{[...]}.

The function will return a negative integer if the recursion limit
is reached.  In that case the @ref{40c1,,tp_repr} implementation should
typically return @code{NULL}.

Otherwise, the function returns zero and the @ref{40c1,,tp_repr}
implementation can continue normally.
@end deffn

@geindex Py_ReprLeave (C function)
@anchor{c-api/exceptions c Py_ReprLeave}@anchor{42a2}
@deffn {C Function} void  Py_ReprLeave (PyObject  *object)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Ends a @ref{42a1,,Py_ReprEnter()}.  Must be called once for each
invocation of @ref{42a1,,Py_ReprEnter()} that returns zero.
@end deffn

@node Standard Exceptions,Standard Warning Categories,Recursion Control,Exception Handling
@anchor{c-api/exceptions standard-exceptions}@anchor{42ef}@anchor{c-api/exceptions standardexceptions}@anchor{42f0}
@subsection Standard Exceptions


All standard Python exceptions are available as global variables whose names are
@code{PyExc_} followed by the Python exception name.  These have the type
@ref{375,,PyObject}*; they are all class objects.  For completeness, here are all
the variables:

@geindex PyExc_BaseException
@geindex PyExc_Exception
@geindex PyExc_ArithmeticError
@geindex PyExc_AssertionError
@geindex PyExc_AttributeError
@geindex PyExc_BlockingIOError
@geindex PyExc_BrokenPipeError
@geindex PyExc_BufferError
@geindex PyExc_ChildProcessError
@geindex PyExc_ConnectionAbortedError
@geindex PyExc_ConnectionError
@geindex PyExc_ConnectionRefusedError
@geindex PyExc_ConnectionResetError
@geindex PyExc_EOFError
@geindex PyExc_FileExistsError
@geindex PyExc_FileNotFoundError
@geindex PyExc_FloatingPointError
@geindex PyExc_GeneratorExit
@geindex PyExc_ImportError
@geindex PyExc_IndentationError
@geindex PyExc_IndexError
@geindex PyExc_InterruptedError
@geindex PyExc_IsADirectoryError
@geindex PyExc_KeyError
@geindex PyExc_KeyboardInterrupt
@geindex PyExc_LookupError
@geindex PyExc_MemoryError
@geindex PyExc_ModuleNotFoundError
@geindex PyExc_NameError
@geindex PyExc_NotADirectoryError
@geindex PyExc_NotImplementedError
@geindex PyExc_OSError
@geindex PyExc_OverflowError
@geindex PyExc_PermissionError
@geindex PyExc_ProcessLookupError
@geindex PyExc_RecursionError
@geindex PyExc_ReferenceError
@geindex PyExc_RuntimeError
@geindex PyExc_StopAsyncIteration
@geindex PyExc_StopIteration
@geindex PyExc_SyntaxError
@geindex PyExc_SystemError
@geindex PyExc_SystemExit
@geindex PyExc_TabError
@geindex PyExc_TimeoutError
@geindex PyExc_TypeError
@geindex PyExc_UnboundLocalError
@geindex PyExc_UnicodeDecodeError
@geindex PyExc_UnicodeEncodeError
@geindex PyExc_UnicodeError
@geindex PyExc_UnicodeTranslateError
@geindex PyExc_ValueError
@geindex PyExc_ZeroDivisionError


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxx} 
@headitem

C Name

@tab

Python Name

@tab

Notes

@item

@code{PyExc_BaseException}

@tab

@ref{15f,,BaseException}

@tab

(1)

@item

@code{PyExc_Exception}

@tab

@ref{4d8,,Exception}

@tab

(1)

@item

@code{PyExc_ArithmeticError}

@tab

@ref{1b70,,ArithmeticError}

@tab

(1)

@item

@code{PyExc_AssertionError}

@tab

@ref{46e,,AssertionError}

@tab

@item

@code{PyExc_AttributeError}

@tab

@ref{163,,AttributeError}

@tab

@item

@code{PyExc_BlockingIOError}

@tab

@ref{bed,,BlockingIOError}

@tab

@item

@code{PyExc_BrokenPipeError}

@tab

@ref{bf2,,BrokenPipeError}

@tab

@item

@code{PyExc_BufferError}

@tab

@ref{1b72,,BufferError}

@tab

@item

@code{PyExc_ChildProcessError}

@tab

@ref{bee,,ChildProcessError}

@tab

@item

@code{PyExc_ConnectionAbortedError}

@tab

@ref{bf3,,ConnectionAbortedError}

@tab

@item

@code{PyExc_ConnectionError}

@tab

@ref{969,,ConnectionError}

@tab

@item

@code{PyExc_ConnectionRefusedError}

@tab

@ref{bf4,,ConnectionRefusedError}

@tab

@item

@code{PyExc_ConnectionResetError}

@tab

@ref{bf5,,ConnectionResetError}

@tab

@item

@code{PyExc_EOFError}

@tab

@ref{e74,,EOFError}

@tab

@item

@code{PyExc_FileExistsError}

@tab

@ref{bb1,,FileExistsError}

@tab

@item

@code{PyExc_FileNotFoundError}

@tab

@ref{a37,,FileNotFoundError}

@tab

@item

@code{PyExc_FloatingPointError}

@tab

@ref{1b71,,FloatingPointError}

@tab

@item

@code{PyExc_GeneratorExit}

@tab

@ref{f2e,,GeneratorExit}

@tab

@item

@code{PyExc_ImportError}

@tab

@ref{3b8,,ImportError}

@tab

@item

@code{PyExc_IndentationError}

@tab

@ref{21c,,IndentationError}

@tab

@item

@code{PyExc_IndexError}

@tab

@ref{1066,,IndexError}

@tab

@item

@code{PyExc_InterruptedError}

@tab

@ref{8e3,,InterruptedError}

@tab

@item

@code{PyExc_IsADirectoryError}

@tab

@ref{bef,,IsADirectoryError}

@tab

@item

@code{PyExc_KeyError}

@tab

@ref{5bb,,KeyError}

@tab

@item

@code{PyExc_KeyboardInterrupt}

@tab

@ref{4ca,,KeyboardInterrupt}

@tab

@item

@code{PyExc_LookupError}

@tab

@ref{1ab5,,LookupError}

@tab

(1)

@item

@code{PyExc_MemoryError}

@tab

@ref{10de,,MemoryError}

@tab

@item

@code{PyExc_ModuleNotFoundError}

@tab

@ref{66b,,ModuleNotFoundError}

@tab

@item

@code{PyExc_NameError}

@tab

@ref{21f,,NameError}

@tab

@item

@code{PyExc_NotADirectoryError}

@tab

@ref{bf0,,NotADirectoryError}

@tab

@item

@code{PyExc_NotImplementedError}

@tab

@ref{8a3,,NotImplementedError}

@tab

@item

@code{PyExc_OSError}

@tab

@ref{2a1,,OSError}

@tab

(1)

@item

@code{PyExc_OverflowError}

@tab

@ref{347,,OverflowError}

@tab

@item

@code{PyExc_PermissionError}

@tab

@ref{895,,PermissionError}

@tab

@item

@code{PyExc_ProcessLookupError}

@tab

@ref{bf1,,ProcessLookupError}

@tab

@item

@code{PyExc_RecursionError}

@tab

@ref{8c4,,RecursionError}

@tab

@item

@code{PyExc_ReferenceError}

@tab

@ref{1043,,ReferenceError}

@tab

(2)

@item

@code{PyExc_RuntimeError}

@tab

@ref{5af,,RuntimeError}

@tab

@item

@code{PyExc_StopAsyncIteration}

@tab

@ref{12da,,StopAsyncIteration}

@tab

@item

@code{PyExc_StopIteration}

@tab

@ref{736,,StopIteration}

@tab

@item

@code{PyExc_SyntaxError}

@tab

@ref{21a,,SyntaxError}

@tab

@item

@code{PyExc_SystemError}

@tab

@ref{891,,SystemError}

@tab

@item

@code{PyExc_SystemExit}

@tab

@ref{892,,SystemExit}

@tab

@item

@code{PyExc_TabError}

@tab

@ref{1068,,TabError}

@tab

@item

@code{PyExc_TimeoutError}

@tab

@ref{2b8,,TimeoutError}

@tab

@item

@code{PyExc_TypeError}

@tab

@ref{162,,TypeError}

@tab

@item

@code{PyExc_UnboundLocalError}

@tab

@ref{1067,,UnboundLocalError}

@tab

@item

@code{PyExc_UnicodeDecodeError}

@tab

@ref{51f,,UnicodeDecodeError}

@tab

@item

@code{PyExc_UnicodeEncodeError}

@tab

@ref{185,,UnicodeEncodeError}

@tab

@item

@code{PyExc_UnicodeError}

@tab

@ref{e49,,UnicodeError}

@tab

@item

@code{PyExc_UnicodeTranslateError}

@tab

@ref{1b88,,UnicodeTranslateError}

@tab

@item

@code{PyExc_ValueError}

@tab

@ref{17c,,ValueError}

@tab

@item

@code{PyExc_ZeroDivisionError}

@tab

@ref{43c,,ZeroDivisionError}

@tab

@end multitable



New in version 3.3: @code{PyExc_BlockingIOError}, @code{PyExc_BrokenPipeError},
@code{PyExc_ChildProcessError}, @code{PyExc_ConnectionError},
@code{PyExc_ConnectionAbortedError}, @code{PyExc_ConnectionRefusedError},
@code{PyExc_ConnectionResetError}, @code{PyExc_FileExistsError},
@code{PyExc_FileNotFoundError}, @code{PyExc_InterruptedError},
@code{PyExc_IsADirectoryError}, @code{PyExc_NotADirectoryError},
@code{PyExc_PermissionError}, @code{PyExc_ProcessLookupError}
and @code{PyExc_TimeoutError} were introduced following 
@geindex Python Enhancement Proposals; PEP 3151
PEP 3151@footnote{https://www.python.org/dev/peps/pep-3151}.



New in version 3.5: @code{PyExc_StopAsyncIteration} and @code{PyExc_RecursionError}.



New in version 3.6: @code{PyExc_ModuleNotFoundError}.


These are compatibility aliases to @code{PyExc_OSError}:

@geindex PyExc_EnvironmentError
@geindex PyExc_IOError
@geindex PyExc_WindowsError


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxx} 
@headitem

C Name

@tab

Notes

@item

@code{PyExc_EnvironmentError}

@tab

@item

@code{PyExc_IOError}

@tab

@item

@code{PyExc_WindowsError}

@tab

(3)

@end multitable



Changed in version 3.3: These aliases used to be separate exception types.


Notes:


@enumerate 

@item 
This is a base class for other standard exceptions.

@item 
Only defined on Windows; protect code that uses this by testing that the
preprocessor macro @code{MS_WINDOWS} is defined.
@end enumerate

@node Standard Warning Categories,,Standard Exceptions,Exception Handling
@anchor{c-api/exceptions standard-warning-categories}@anchor{42f1}@anchor{c-api/exceptions standardwarningcategories}@anchor{42e4}
@subsection Standard Warning Categories


All standard Python warning categories are available as global variables whose
names are @code{PyExc_} followed by the Python exception name. These have the type
@ref{375,,PyObject}*; they are all class objects. For completeness, here are all
the variables:

@geindex PyExc_Warning
@geindex PyExc_BytesWarning
@geindex PyExc_DeprecationWarning
@geindex PyExc_FutureWarning
@geindex PyExc_ImportWarning
@geindex PyExc_PendingDeprecationWarning
@geindex PyExc_ResourceWarning
@geindex PyExc_RuntimeWarning
@geindex PyExc_SyntaxWarning
@geindex PyExc_UnicodeWarning
@geindex PyExc_UserWarning


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxx} 
@headitem

C Name

@tab

Python Name

@tab

Notes

@item

@code{PyExc_Warning}

@tab

@ref{1b8d,,Warning}

@tab

(1)

@item

@code{PyExc_BytesWarning}

@tab

@ref{761,,BytesWarning}

@tab

@item

@code{PyExc_DeprecationWarning}

@tab

@ref{1b2,,DeprecationWarning}

@tab

@item

@code{PyExc_FutureWarning}

@tab

@ref{604,,FutureWarning}

@tab

@item

@code{PyExc_ImportWarning}

@tab

@ref{304,,ImportWarning}

@tab

@item

@code{PyExc_PendingDeprecationWarning}

@tab

@ref{3a9,,PendingDeprecationWarning}

@tab

@item

@code{PyExc_ResourceWarning}

@tab

@ref{77b,,ResourceWarning}

@tab

@item

@code{PyExc_RuntimeWarning}

@tab

@ref{5b5,,RuntimeWarning}

@tab

@item

@code{PyExc_SyntaxWarning}

@tab

@ref{4c7,,SyntaxWarning}

@tab

@item

@code{PyExc_UnicodeWarning}

@tab

@ref{f81,,UnicodeWarning}

@tab

@item

@code{PyExc_UserWarning}

@tab

@ref{1b8e,,UserWarning}

@tab

@end multitable



New in version 3.2: @code{PyExc_ResourceWarning}.


Notes:


@enumerate 

@item 
This is a base class for other standard warning categories.
@end enumerate

@node Utilities<2>,Abstract Objects Layer,Exception Handling,Python/C API Reference Manual
@anchor{c-api/utilities doc}@anchor{42f2}@anchor{c-api/utilities id1}@anchor{42f3}@anchor{c-api/utilities utilities}@anchor{42f4}
@section Utilities


The functions in this chapter perform various utility tasks, ranging from
helping C code be more portable across platforms, using Python modules from C,
and parsing function arguments and constructing Python values from C values.

@menu
* Operating System Utilities:: 
* System Functions:: 
* Process Control:: 
* Importing Modules: Importing Modules<2>. 
* Data marshalling support:: 
* Parsing arguments and building values:: 
* String conversion and formatting:: 
* Reflection:: 
* Codec registry and support functions:: 

@end menu

@node Operating System Utilities,System Functions,,Utilities<2>
@anchor{c-api/sys doc}@anchor{42f5}@anchor{c-api/sys operating-system-utilities}@anchor{42f6}@anchor{c-api/sys os}@anchor{42f7}
@subsection Operating System Utilities


@geindex PyOS_FSPath (C function)
@anchor{c-api/sys c PyOS_FSPath}@anchor{866}
@deffn {C Function} @ref{375,,PyObject}  *PyOS_FSPath (PyObject  *path)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.6.}
Return the file system representation for @emph{path}. If the object is a
@ref{2ef,,str} or @ref{2f0,,bytes} object, then its reference count is
incremented. If the object implements the @ref{795,,os.PathLike} interface,
then @ref{796,,__fspath__()} is returned as long as it is a
@ref{2ef,,str} or @ref{2f0,,bytes} object. Otherwise @ref{162,,TypeError} is raised
and @code{NULL} is returned.


New in version 3.6.

@end deffn

@geindex Py_FdIsInteractive (C function)
@anchor{c-api/sys c Py_FdIsInteractive}@anchor{42f8}
@deffn {C Function} int  Py_FdIsInteractive (FILE  *fp, const  char  *filename)

Return true (nonzero) if the standard I/O file @emph{fp} with name @emph{filename} is
deemed interactive.  This is the case for files for which @code{isatty(fileno(fp))}
is true.  If the global flag @ref{42f9,,Py_InteractiveFlag} is true, this function
also returns true if the @emph{filename} pointer is @code{NULL} or if the name is equal to
one of the strings @code{'<stdin>'} or @code{'???'}.
@end deffn

@geindex PyOS_BeforeFork (C function)
@anchor{c-api/sys c PyOS_BeforeFork}@anchor{6f6}
@deffn {C Function} void  PyOS_BeforeFork ()
@emph{ Part of the @ref{35d,,Stable ABI} on platforms with fork() since version 3.7.}
Function to prepare some internal state before a process fork.  This
should be called before calling @code{fork()} or any similar function
that clones the current process.
Only available on systems where @code{fork()} is defined.

@cartouche
@quotation Warning 
The C @code{fork()} call should only be made from the
@ref{42fa,,“main” thread} (of the
@ref{42fb,,“main” interpreter}).  The same is
true for @code{PyOS_BeforeFork()}.
@end quotation
@end cartouche


New in version 3.7.

@end deffn

@geindex PyOS_AfterFork_Parent (C function)
@anchor{c-api/sys c PyOS_AfterFork_Parent}@anchor{6f7}
@deffn {C Function} void  PyOS_AfterFork_Parent ()
@emph{ Part of the @ref{35d,,Stable ABI} on platforms with fork() since version 3.7.}
Function to update some internal state after a process fork.  This
should be called from the parent process after calling @code{fork()}
or any similar function that clones the current process, regardless
of whether process cloning was successful.
Only available on systems where @code{fork()} is defined.

@cartouche
@quotation Warning 
The C @code{fork()} call should only be made from the
@ref{42fa,,“main” thread} (of the
@ref{42fb,,“main” interpreter}).  The same is
true for @code{PyOS_AfterFork_Parent()}.
@end quotation
@end cartouche


New in version 3.7.

@end deffn

@geindex PyOS_AfterFork_Child (C function)
@anchor{c-api/sys c PyOS_AfterFork_Child}@anchor{5c1}
@deffn {C Function} void  PyOS_AfterFork_Child ()
@emph{ Part of the @ref{35d,,Stable ABI} on platforms with fork() since version 3.7.}
Function to update internal interpreter state after a process fork.
This must be called from the child process after calling @code{fork()},
or any similar function that clones the current process, if there is
any chance the process will call back into the Python interpreter.
Only available on systems where @code{fork()} is defined.

@cartouche
@quotation Warning 
The C @code{fork()} call should only be made from the
@ref{42fa,,“main” thread} (of the
@ref{42fb,,“main” interpreter}).  The same is
true for @code{PyOS_AfterFork_Child()}.
@end quotation
@end cartouche


New in version 3.7.



@subsubheading See also


@ref{685,,os.register_at_fork()} allows registering custom Python functions
to be called by @ref{6f6,,PyOS_BeforeFork()},
@ref{6f7,,PyOS_AfterFork_Parent()} and  @ref{5c1,,PyOS_AfterFork_Child()}.

@end deffn

@geindex PyOS_AfterFork (C function)
@anchor{c-api/sys c PyOS_AfterFork}@anchor{6f5}
@deffn {C Function} void  PyOS_AfterFork ()
@emph{ Part of the @ref{35d,,Stable ABI} on platforms with fork().}
Function to update some internal state after a process fork; this should be
called in the new process if the Python interpreter will continue to be used.
If a new executable is loaded into the new process, this function does not need
to be called.


Deprecated since version 3.7: This function is superseded by @ref{5c1,,PyOS_AfterFork_Child()}.

@end deffn

@geindex PyOS_CheckStack (C function)
@anchor{c-api/sys c PyOS_CheckStack}@anchor{41f4}
@deffn {C Function} int  PyOS_CheckStack ()
@emph{ Part of the @ref{35d,,Stable ABI} on platforms with USE_STACKCHECK since version 3.7.}
Return true when the interpreter runs out of stack space.  This is a reliable
check, but is only available when @code{USE_STACKCHECK} is defined (currently
on Windows using the Microsoft Visual C++ compiler).  @code{USE_STACKCHECK}
will be defined automatically; you should never change the definition in your
own code.
@end deffn

@geindex PyOS_getsig (C function)
@anchor{c-api/sys c PyOS_getsig}@anchor{41f7}
@deffn {C Function} PyOS_sighandler_t  PyOS_getsig (int  i)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the current signal handler for signal @emph{i}.  This is a thin wrapper around
either @code{sigaction()} or @code{signal()}.  Do not call those functions
directly! @code{PyOS_sighandler_t} is a typedef alias for void  (*)(int).
@end deffn

@geindex PyOS_setsig (C function)
@anchor{c-api/sys c PyOS_setsig}@anchor{41f8}
@deffn {C Function} PyOS_sighandler_t  PyOS_setsig (int  i, PyOS_sighandler_t  h)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the signal handler for signal @emph{i} to be @emph{h}; return the old signal handler.
This is a thin wrapper around either @code{sigaction()} or @code{signal()}.  Do
not call those functions directly!  @code{PyOS_sighandler_t} is a typedef
alias for void  (*)(int).
@end deffn

@geindex Py_DecodeLocale (C function)
@anchor{c-api/sys c Py_DecodeLocale}@anchor{6fe}
@deffn {C Function} wchar_t  *Py_DecodeLocale (const  char  *arg, size_t  *size)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
@cartouche
@quotation Warning 
This function should not be called directly: use the @ref{206,,PyConfig}
API with the @code{PyConfig_SetBytesString()} function which ensures
that @ref{3502,,Python is preinitialized}.

This function must not be called before @ref{3502,,Python is preinitialized} and so that the LC_CTYPE locale is properly configured: see
the @ref{4b8,,Py_PreInitialize()} function.
@end quotation
@end cartouche

Decode a byte string from the @ref{1628,,filesystem encoding and error handler}.
If the error handler is @ref{1cc9,,surrogateescape error handler}, undecodable bytes are decoded as characters in range
U+DC80..U+DCFF; and if a byte sequence can be decoded as a surrogate
character, the bytes are escaped using the surrogateescape error handler
instead of decoding them.

Return a pointer to a newly allocated wide character string, use
@ref{42fc,,PyMem_RawFree()} to free the memory. If size is not @code{NULL}, write
the number of wide characters excluding the null character into @code{*size}

Return @code{NULL} on decoding error or memory allocation error. If @emph{size} is
not @code{NULL}, @code{*size} is set to @code{(size_t)-1} on memory error or set to
@code{(size_t)-2} on decoding error.

The @ref{1628,,filesystem encoding and error handler} are selected by
@code{PyConfig_Read()}: see @ref{2378,,filesystem_encoding} and
@ref{2379,,filesystem_errors} members of @ref{206,,PyConfig}.

Decoding errors should never happen, unless there is a bug in the C
library.

Use the @ref{6ff,,Py_EncodeLocale()} function to encode the character string
back to a byte string.


@subsubheading See also


The @ref{4277,,PyUnicode_DecodeFSDefaultAndSize()} and
@ref{700,,PyUnicode_DecodeLocaleAndSize()} functions.



New in version 3.5.



Changed in version 3.7: The function now uses the UTF-8 encoding in the @ref{5f2,,Python UTF-8 Mode}.



Changed in version 3.8: The function now uses the UTF-8 encoding on Windows if
@ref{42fd,,Py_LegacyWindowsFSEncodingFlag} is zero;

@end deffn

@geindex Py_EncodeLocale (C function)
@anchor{c-api/sys c Py_EncodeLocale}@anchor{6ff}
@deffn {C Function} char  *Py_EncodeLocale (const  wchar_t  *text, size_t  *error_pos)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Encode a wide character string to the @ref{1628,,filesystem encoding and error handler}. If the error handler is @ref{1cc9,,surrogateescape error handler}, surrogate characters in the range U+DC80..U+DCFF are
converted to bytes 0x80..0xFF.

Return a pointer to a newly allocated byte string, use @ref{fa3,,PyMem_Free()}
to free the memory. Return @code{NULL} on encoding error or memory allocation
error

If error_pos is not @code{NULL}, @code{*error_pos} is set to @code{(size_t)-1} on
success,  or set to the index of the invalid character on encoding error.

The @ref{1628,,filesystem encoding and error handler} are selected by
@code{PyConfig_Read()}: see @ref{2378,,filesystem_encoding} and
@ref{2379,,filesystem_errors} members of @ref{206,,PyConfig}.

Use the @ref{6fe,,Py_DecodeLocale()} function to decode the bytes string back
to a wide character string.

@cartouche
@quotation Warning 
This function must not be called before @ref{3502,,Python is preinitialized} and so that the LC_CTYPE locale is properly configured: see
the @ref{4b8,,Py_PreInitialize()} function.
@end quotation
@end cartouche


@subsubheading See also


The @ref{130e,,PyUnicode_EncodeFSDefault()} and
@ref{701,,PyUnicode_EncodeLocale()} functions.



New in version 3.5.



Changed in version 3.7: The function now uses the UTF-8 encoding in the @ref{5f2,,Python UTF-8 Mode}.



Changed in version 3.8: The function now uses the UTF-8 encoding on Windows if
@ref{42fd,,Py_LegacyWindowsFSEncodingFlag} is zero;

@end deffn

@node System Functions,Process Control,Operating System Utilities,Utilities<2>
@anchor{c-api/sys system-functions}@anchor{42fe}@anchor{c-api/sys systemfunctions}@anchor{42ff}
@subsection System Functions


These are utility functions that make functionality from the @ref{f7,,sys} module
accessible to C code.  They all work with the current interpreter thread’s
@ref{f7,,sys} module’s dict, which is contained in the internal thread state structure.

@geindex PySys_GetObject (C function)
@anchor{c-api/sys c PySys_GetObject}@anchor{4234}
@deffn {C Function} @ref{375,,PyObject}  *PySys_GetObject (const  char  *name)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the object @emph{name} from the @ref{f7,,sys} module or @code{NULL} if it does
not exist, without setting an exception.
@end deffn

@geindex PySys_SetObject (C function)
@anchor{c-api/sys c PySys_SetObject}@anchor{4237}
@deffn {C Function} int  PySys_SetObject (const  char  *name, PyObject  *v)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set @emph{name} in the @ref{f7,,sys} module to @emph{v} unless @emph{v} is @code{NULL}, in which
case @emph{name} is deleted from the sys module. Returns @code{0} on success, @code{-1}
on error.
@end deffn

@geindex PySys_ResetWarnOptions (C function)
@anchor{c-api/sys c PySys_ResetWarnOptions}@anchor{4236}
@deffn {C Function} void  PySys_ResetWarnOptions ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Reset @ref{6dd,,sys.warnoptions} to an empty list. This function may be
called prior to @ref{38d,,Py_Initialize()}.
@end deffn

@geindex PySys_AddWarnOption (C function)
@anchor{c-api/sys c PySys_AddWarnOption}@anchor{200}
@deffn {C Function} void  PySys_AddWarnOption (const  wchar_t  *s)
@emph{ Part of the @ref{35d,,Stable ABI}.}
This API is kept for backward compatibility: setting
@ref{4300,,PyConfig.warnoptions} should be used instead, see @ref{207,,Python Initialization Configuration}.

Append @emph{s} to @ref{6dd,,sys.warnoptions}. This function must be called prior
to @ref{38d,,Py_Initialize()} in order to affect the warnings filter list.


Deprecated since version 3.11.

@end deffn

@geindex PySys_AddWarnOptionUnicode (C function)
@anchor{c-api/sys c PySys_AddWarnOptionUnicode}@anchor{1ff}
@deffn {C Function} void  PySys_AddWarnOptionUnicode (PyObject  *unicode)
@emph{ Part of the @ref{35d,,Stable ABI}.}
This API is kept for backward compatibility: setting
@ref{4300,,PyConfig.warnoptions} should be used instead, see @ref{207,,Python Initialization Configuration}.

Append @emph{unicode} to @ref{6dd,,sys.warnoptions}.

Note: this function is not currently usable from outside the CPython
implementation, as it must be called prior to the implicit import of
@ref{12d,,warnings} in @ref{38d,,Py_Initialize()} to be effective, but can’t be
called until enough of the runtime has been initialized to permit the
creation of Unicode objects.


Deprecated since version 3.11.

@end deffn

@geindex PySys_SetPath (C function)
@anchor{c-api/sys c PySys_SetPath}@anchor{4238}
@deffn {C Function} void  PySys_SetPath (const  wchar_t  *path)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set @ref{489,,sys.path} to a list object of paths found in @emph{path} which should
be a list of paths separated with the platform’s search path delimiter
(@code{:} on Unix, @code{;} on Windows).
@end deffn

@geindex PySys_WriteStdout (C function)
@anchor{c-api/sys c PySys_WriteStdout}@anchor{423a}
@deffn {C Function} void  PySys_WriteStdout (const  char  *format, ...)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Write the output string described by @emph{format} to @ref{5ef,,sys.stdout}.  No
exceptions are raised, even if truncation occurs (see below).

@emph{format} should limit the total size of the formatted output string to
1000 bytes or less – after 1000 bytes, the output string is truncated.
In particular, this means that no unrestricted “%s” formats should occur;
these should be limited using “%.<N>s” where <N> is a decimal number
calculated so that <N> plus the maximum size of other formatted text does not
exceed 1000 bytes.  Also watch out for “%f”, which can print hundreds of
digits for very large numbers.

If a problem occurs, or @ref{5ef,,sys.stdout} is unset, the formatted message
is written to the real (C level) @emph{stdout}.
@end deffn

@geindex PySys_WriteStderr (C function)
@anchor{c-api/sys c PySys_WriteStderr}@anchor{4239}
@deffn {C Function} void  PySys_WriteStderr (const  char  *format, ...)
@emph{ Part of the @ref{35d,,Stable ABI}.}
As @ref{423a,,PySys_WriteStdout()}, but write to @ref{42d,,sys.stderr} or @emph{stderr}
instead.
@end deffn

@geindex PySys_FormatStdout (C function)
@anchor{c-api/sys c PySys_FormatStdout}@anchor{4233}
@deffn {C Function} void  PySys_FormatStdout (const  char  *format, ...)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Function similar to PySys_WriteStdout() but format the message using
@ref{4289,,PyUnicode_FromFormatV()} and don’t truncate the message to an
arbitrary length.


New in version 3.2.

@end deffn

@geindex PySys_FormatStderr (C function)
@anchor{c-api/sys c PySys_FormatStderr}@anchor{4232}
@deffn {C Function} void  PySys_FormatStderr (const  char  *format, ...)
@emph{ Part of the @ref{35d,,Stable ABI}.}
As @ref{4233,,PySys_FormatStdout()}, but write to @ref{42d,,sys.stderr} or @emph{stderr}
instead.


New in version 3.2.

@end deffn

@geindex PySys_AddXOption (C function)
@anchor{c-api/sys c PySys_AddXOption}@anchor{201}
@deffn {C Function} void  PySys_AddXOption (const  wchar_t  *s)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
This API is kept for backward compatibility: setting
@ref{4301,,PyConfig.xoptions} should be used instead, see @ref{207,,Python Initialization Configuration}.

Parse @emph{s} as a set of @ref{159,,-X} options and add them to the current
options mapping as returned by @ref{4235,,PySys_GetXOptions()}. This function
may be called prior to @ref{38d,,Py_Initialize()}.


New in version 3.2.



Deprecated since version 3.11.

@end deffn

@geindex PySys_GetXOptions (C function)
@anchor{c-api/sys c PySys_GetXOptions}@anchor{4235}
@deffn {C Function} @ref{375,,PyObject}  *PySys_GetXOptions ()
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Return the current dictionary of @ref{159,,-X} options, similarly to
@ref{1618,,sys._xoptions}.  On error, @code{NULL} is returned and an exception is
set.


New in version 3.2.

@end deffn

@geindex PySys_Audit (C function)
@anchor{c-api/sys c PySys_Audit}@anchor{120d}
@deffn {C Function} int  PySys_Audit (const  char  *event, const  char  *format, ...)

Raise an auditing event with any active hooks. Return zero for success
and non-zero with an exception set on failure.

If any hooks have been added, @emph{format} and other arguments will be used
to construct a tuple to pass. Apart from @code{N}, the same format characters
as used in @ref{380,,Py_BuildValue()} are available. If the built value is not
a tuple, it will be added into a single-element tuple. (The @code{N} format
option consumes a reference, but since there is no way to know whether
arguments to this function will be consumed, using it may cause reference
leaks.)

Note that @code{#} format characters should always be treated as
@code{Py_ssize_t}, regardless of whether @code{PY_SSIZE_T_CLEAN} was defined.

@ref{11fc,,sys.audit()} performs the same function from Python code.


New in version 3.8.



Changed in version 3.8.2: Require @code{Py_ssize_t} for @code{#} format characters. Previously, an
unavoidable deprecation warning was raised.

@end deffn

@geindex PySys_AddAuditHook (C function)
@anchor{c-api/sys c PySys_AddAuditHook}@anchor{39ec}
@deffn {C Function} int  PySys_AddAuditHook (Py_AuditHookFunction  hook, void  *userData)

Append the callable @emph{hook} to the list of active auditing hooks.
Return zero for success
and non-zero on failure. If the runtime has been initialized, also set an
error on failure. Hooks added through this API are called for all
interpreters created by the runtime.

The @emph{userData} pointer is passed into the hook function. Since hook
functions may be called from different runtimes, this pointer should not
refer directly to Python state.

This function is safe to call before @ref{38d,,Py_Initialize()}. When called
after runtime initialization, existing audit hooks are notified and may
silently abort the operation by raising an error subclassed from
@ref{4d8,,Exception} (other errors will not be silenced).

The hook function is of type int  (*)(const  char  *event,  @ref{375,,PyObject}  *args,  void  *userData), where @emph{args} is guaranteed to be a
@ref{4302,,PyTupleObject}. The hook function is always called with the GIL
held by the Python interpreter that raised the event.

See 
@geindex Python Enhancement Proposals; PEP 578
PEP 578@footnote{https://www.python.org/dev/peps/pep-0578} for a detailed description of auditing.  Functions in the
runtime and standard library that raise events are listed in the
@ref{10cc,,audit events table}.
Details are in each function’s documentation.


If the interpreter is initialized, this function raises a auditing event
@code{sys.addaudithook} with no arguments. If any existing hooks raise an
exception derived from @ref{4d8,,Exception}, the new hook will not be
added and the exception is cleared. As a result, callers cannot assume
that their hook has been added unless they control all existing hooks.



New in version 3.8.

@end deffn

@node Process Control,Importing Modules<2>,System Functions,Utilities<2>
@anchor{c-api/sys process-control}@anchor{4303}@anchor{c-api/sys processcontrol}@anchor{4304}
@subsection Process Control


@geindex Py_FatalError (C function)
@anchor{c-api/sys c Py_FatalError}@anchor{48f}
@deffn {C Function} void  Py_FatalError (const  char  *message)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex abort()

Print a fatal error message and kill the process.  No cleanup is performed.
This function should only be invoked when a condition is detected that would
make it dangerous to continue using the Python interpreter; e.g., when the
object administration appears to be corrupted.  On Unix, the standard C library
function @code{abort()} is called which will attempt to produce a @code{core}
file.

The @code{Py_FatalError()} function is replaced with a macro which logs
automatically the name of the current function, unless the
@code{Py_LIMITED_API} macro is defined.


Changed in version 3.9: Log the function name automatically.

@end deffn

@geindex Py_Exit (C function)
@anchor{c-api/sys c Py_Exit}@anchor{8a6}
@deffn {C Function} void  Py_Exit (int  status)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex Py_FinalizeEx()
@geindex exit()

Exit the current process.  This calls @ref{862,,Py_FinalizeEx()} and then calls the
standard C library function @code{exit(status)}.  If @ref{862,,Py_FinalizeEx()}
indicates an error, the exit status is set to 120.


Changed in version 3.6: Errors from finalization no longer ignored.

@end deffn

@geindex Py_AtExit (C function)
@anchor{c-api/sys c Py_AtExit}@anchor{4297}
@deffn {C Function} int  Py_AtExit (void  (*func)())
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex Py_FinalizeEx()
@geindex cleanup functions

Register a cleanup function to be called by @ref{862,,Py_FinalizeEx()}.  The cleanup
function will be called with no arguments and should return no value.  At most
32 cleanup functions can be registered.  When the registration is successful,
@ref{4297,,Py_AtExit()} returns @code{0}; on failure, it returns @code{-1}.  The cleanup
function registered last is called first. Each cleanup function will be called
at most once.  Since Python’s internal finalization will have completed before
the cleanup function, no Python APIs should be called by @emph{func}.
@end deffn

@node Importing Modules<2>,Data marshalling support,Process Control,Utilities<2>
@anchor{c-api/import doc}@anchor{4305}@anchor{c-api/import importing}@anchor{4306}@anchor{c-api/import importing-modules}@anchor{4307}
@subsection Importing Modules


@geindex PyImport_ImportModule (C function)
@anchor{c-api/import c PyImport_ImportModule}@anchor{34f}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_ImportModule (const  char  *name)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex package variable; __all__
@geindex __all__ (package variable)
@geindex modules (in module sys)

This is a simplified interface to @ref{d4b,,PyImport_ImportModuleEx()} below,
leaving the @emph{globals} and @emph{locals} arguments set to @code{NULL} and @emph{level} set
to 0.  When the @emph{name}
argument contains a dot (when it specifies a submodule of a package), the
@emph{fromlist} argument is set to the list @code{['*']} so that the return value is the
named module rather than the top-level package containing it as would otherwise
be the case.  (Unfortunately, this has an additional side effect when @emph{name} in
fact specifies a subpackage instead of a submodule: the submodules specified in
the package’s @code{__all__} variable are  loaded.)  Return a new reference to the
imported module, or @code{NULL} with an exception set on failure.  A failing
import of a module doesn’t leave the module in @ref{11f7,,sys.modules}.

This function always uses absolute imports.
@end deffn

@geindex PyImport_ImportModuleNoBlock (C function)
@anchor{c-api/import c PyImport_ImportModuleNoBlock}@anchor{c11}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_ImportModuleNoBlock (const  char  *name)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
This function is a deprecated alias of @ref{34f,,PyImport_ImportModule()}.


Changed in version 3.3: This function used to fail immediately when the import lock was held
by another thread.  In Python 3.3 though, the locking scheme switched
to per-module locks for most purposes, so this function’s special
behaviour isn’t needed anymore.

@end deffn

@geindex PyImport_ImportModuleEx (C function)
@anchor{c-api/import c PyImport_ImportModuleEx}@anchor{d4b}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_ImportModuleEx (const  char  *name, PyObject  *globals, PyObject  *locals, PyObject  *fromlist)
@emph{Return value: New reference.}
@geindex built-in function; __import__

Import a module.  This is best described by referring to the built-in Python
function @ref{3b7,,__import__()}.

The return value is a new reference to the imported module or top-level
package, or @code{NULL} with an exception set on failure.  Like for
@ref{3b7,,__import__()}, the return value when a submodule of a package was
requested is normally the top-level package, unless a non-empty @emph{fromlist}
was given.

Failing imports remove incomplete module objects, like with
@ref{34f,,PyImport_ImportModule()}.
@end deffn

@geindex PyImport_ImportModuleLevelObject (C function)
@anchor{c-api/import c PyImport_ImportModuleLevelObject}@anchor{41a7}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_ImportModuleLevelObject (PyObject  *name, PyObject  *globals, PyObject  *locals, PyObject  *fromlist, int  level)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Import a module.  This is best described by referring to the built-in Python
function @ref{3b7,,__import__()}, as the standard @ref{3b7,,__import__()} function calls
this function directly.

The return value is a new reference to the imported module or top-level package,
or @code{NULL} with an exception set on failure.  Like for @ref{3b7,,__import__()},
the return value when a submodule of a package was requested is normally the
top-level package, unless a non-empty @emph{fromlist} was given.


New in version 3.3.

@end deffn

@geindex PyImport_ImportModuleLevel (C function)
@anchor{c-api/import c PyImport_ImportModuleLevel}@anchor{d4a}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_ImportModuleLevel (const  char  *name, PyObject  *globals, PyObject  *locals, PyObject  *fromlist, int  level)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Similar to @ref{41a7,,PyImport_ImportModuleLevelObject()}, but the name is a
UTF-8 encoded string instead of a Unicode object.


Changed in version 3.3: Negative values for @emph{level} are no longer accepted.

@end deffn

@geindex PyImport_Import (C function)
@anchor{c-api/import c PyImport_Import}@anchor{f5a}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_Import (PyObject  *name)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
This is a higher-level interface that calls the current “import hook
function” (with an explicit @emph{level} of 0, meaning absolute import).  It
invokes the @ref{3b7,,__import__()} function from the @code{__builtins__} of the
current globals.  This means that the import is done using whatever import
hooks are installed in the current environment.

This function always uses absolute imports.
@end deffn

@geindex PyImport_ReloadModule (C function)
@anchor{c-api/import c PyImport_ReloadModule}@anchor{41a8}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_ReloadModule (PyObject  *m)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Reload a module.  Return a new reference to the reloaded module, or @code{NULL} with
an exception set on failure (the module still exists in this case).
@end deffn

@geindex PyImport_AddModuleObject (C function)
@anchor{c-api/import c PyImport_AddModuleObject}@anchor{419d}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_AddModuleObject (PyObject  *name)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Return the module object corresponding to a module name.  The @emph{name} argument
may be of the form @code{package.module}. First check the modules dictionary if
there’s one there, and if not, create a new one and insert it in the modules
dictionary. Return @code{NULL} with an exception set on failure.

@cartouche
@quotation Note 
This function does not load or import the module; if the module wasn’t already
loaded, you will get an empty module object. Use @ref{34f,,PyImport_ImportModule()}
or one of its variants to import a module.  Package structures implied by a
dotted name for @emph{name} are not created if not already present.
@end quotation
@end cartouche


New in version 3.3.

@end deffn

@geindex PyImport_AddModule (C function)
@anchor{c-api/import c PyImport_AddModule}@anchor{4090}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_AddModule (const  char  *name)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Similar to @ref{419d,,PyImport_AddModuleObject()}, but the name is a UTF-8
encoded string instead of a Unicode object.
@end deffn

@geindex PyImport_ExecCodeModule (C function)
@anchor{c-api/import c PyImport_ExecCodeModule}@anchor{419e}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_ExecCodeModule (const  char  *name, PyObject  *co)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; compile

Given a module name (possibly of the form @code{package.module}) and a code object
read from a Python bytecode file or obtained from the built-in function
@ref{4e2,,compile()}, load the module.  Return a new reference to the module object,
or @code{NULL} with an exception set if an error occurred.  @emph{name}
is removed from @ref{11f7,,sys.modules} in error cases, even if @emph{name} was already
in @ref{11f7,,sys.modules} on entry to @ref{419e,,PyImport_ExecCodeModule()}.  Leaving
incompletely initialized modules in @ref{11f7,,sys.modules} is dangerous, as imports of
such modules have no way to know that the module object is an unknown (and
probably damaged with respect to the module author’s intents) state.

The module’s @ref{1858,,__spec__} and @ref{baf,,__loader__} will be set, if
not set already, with the appropriate values.  The spec’s loader will
be set to the module’s @code{__loader__} (if set) and to an instance of
@code{SourceFileLoader} otherwise.

The module’s @ref{1797,,__file__} attribute will be set to the code object’s
@code{co_filename}.  If applicable, @ref{d5b,,__cached__} will also
be set.

This function will reload the module if it was already imported.  See
@ref{41a8,,PyImport_ReloadModule()} for the intended way to reload a module.

If @emph{name} points to a dotted name of the form @code{package.module}, any package
structures not already created will still not be created.

See also @ref{419f,,PyImport_ExecCodeModuleEx()} and
@ref{41a1,,PyImport_ExecCodeModuleWithPathnames()}.
@end deffn

@geindex PyImport_ExecCodeModuleEx (C function)
@anchor{c-api/import c PyImport_ExecCodeModuleEx}@anchor{419f}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_ExecCodeModuleEx (const  char  *name, PyObject  *co, const  char  *pathname)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Like @ref{419e,,PyImport_ExecCodeModule()}, but the @ref{1797,,__file__} attribute of
the module object is set to @emph{pathname} if it is non-@code{NULL}.

See also @ref{41a1,,PyImport_ExecCodeModuleWithPathnames()}.
@end deffn

@geindex PyImport_ExecCodeModuleObject (C function)
@anchor{c-api/import c PyImport_ExecCodeModuleObject}@anchor{41a0}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_ExecCodeModuleObject (PyObject  *name, PyObject  *co, PyObject  *pathname, PyObject  *cpathname)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Like @ref{419f,,PyImport_ExecCodeModuleEx()}, but the @ref{d5b,,__cached__}
attribute of the module object is set to @emph{cpathname} if it is
non-@code{NULL}.  Of the three functions, this is the preferred one to use.


New in version 3.3.

@end deffn

@geindex PyImport_ExecCodeModuleWithPathnames (C function)
@anchor{c-api/import c PyImport_ExecCodeModuleWithPathnames}@anchor{41a1}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_ExecCodeModuleWithPathnames (const  char  *name, PyObject  *co, const  char  *pathname, const  char  *cpathname)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Like @ref{41a0,,PyImport_ExecCodeModuleObject()}, but @emph{name}, @emph{pathname} and
@emph{cpathname} are UTF-8 encoded strings. Attempts are also made to figure out
what the value for @emph{pathname} should be from @emph{cpathname} if the former is
set to @code{NULL}.


New in version 3.2.



Changed in version 3.3: Uses @ref{4057,,imp.source_from_cache()} in calculating the source path if
only the bytecode path is provided.

@end deffn

@geindex PyImport_GetMagicNumber (C function)
@anchor{c-api/import c PyImport_GetMagicNumber}@anchor{d49}
@deffn {C Function} long  PyImport_GetMagicNumber ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the magic number for Python bytecode files (a.k.a. @code{.pyc} file).
The magic number should be present in the first four bytes of the bytecode
file, in little-endian byte order. Returns @code{-1} on error.


Changed in version 3.3: Return value of @code{-1} upon failure.

@end deffn

@geindex PyImport_GetMagicTag (C function)
@anchor{c-api/import c PyImport_GetMagicTag}@anchor{41a3}
@deffn {C Function} const  char  *PyImport_GetMagicTag ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the magic tag string for 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} format Python bytecode file
names.  Keep in mind that the value at @code{sys.implementation.cache_tag} is
authoritative and should be used instead of this function.


New in version 3.2.

@end deffn

@geindex PyImport_GetModuleDict (C function)
@anchor{c-api/import c PyImport_GetModuleDict}@anchor{41a4}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_GetModuleDict ()
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the dictionary used for the module administration (a.k.a.
@code{sys.modules}).  Note that this is a per-interpreter variable.
@end deffn

@geindex PyImport_GetModule (C function)
@anchor{c-api/import c PyImport_GetModule}@anchor{6e7}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_GetModule (PyObject  *name)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.8.}
Return the already imported module with the given name.  If the
module has not been imported yet then returns @code{NULL} but does not set
an error.  Returns @code{NULL} and sets an error if the lookup failed.


New in version 3.7.

@end deffn

@geindex PyImport_GetImporter (C function)
@anchor{c-api/import c PyImport_GetImporter}@anchor{41a2}
@deffn {C Function} @ref{375,,PyObject}  *PyImport_GetImporter (PyObject  *path)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a finder object for a @ref{489,,sys.path}/@code{pkg.__path__} item
@emph{path}, possibly by fetching it from the @ref{74c,,sys.path_importer_cache}
dict.  If it wasn’t yet cached, traverse @ref{bb5,,sys.path_hooks} until a hook
is found that can handle the path item.  Return @code{None} if no hook could;
this tells our caller that the @ref{184d,,path based finder} could not find a
finder for this path item. Cache the result in @ref{74c,,sys.path_importer_cache}.
Return a new reference to the finder object.
@end deffn

@geindex PyImport_ImportFrozenModuleObject (C function)
@anchor{c-api/import c PyImport_ImportFrozenModuleObject}@anchor{41a6}
@deffn {C Function} int  PyImport_ImportFrozenModuleObject (PyObject  *name)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Load a frozen module named @emph{name}.  Return @code{1} for success, @code{0} if the
module is not found, and @code{-1} with an exception set if the initialization
failed.  To access the imported module on a successful load, use
@ref{34f,,PyImport_ImportModule()}.  (Note the misnomer — this function would
reload the module if it was already imported.)


New in version 3.3.



Changed in version 3.4: The @code{__file__} attribute is no longer set on the module.

@end deffn

@geindex PyImport_ImportFrozenModule (C function)
@anchor{c-api/import c PyImport_ImportFrozenModule}@anchor{41a5}
@deffn {C Function} int  PyImport_ImportFrozenModule (const  char  *name)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Similar to @ref{41a6,,PyImport_ImportFrozenModuleObject()}, but the name is a
UTF-8 encoded string instead of a Unicode object.
@end deffn

@geindex _frozen (C struct)
@anchor{c-api/import c _frozen}@anchor{1d3}
@deffn {C Type} struct  _frozen

@geindex freeze utility

This is the structure type definition for frozen module descriptors, as
generated by the @code{freeze} utility (see @code{Tools/freeze/} in the
Python source distribution).  Its definition, found in @code{Include/import.h},
is:

@example
struct _frozen @{
    const char *name;
    const unsigned char *code;
    int size;
    bool is_package;
@};
@end example


Changed in version 3.11: The new @code{is_package} field indicates whether the module is a package or not.
This replaces setting the @code{size} field to a negative value.

@end deffn

@geindex PyImport_FrozenModules (C var)
@anchor{c-api/import c PyImport_FrozenModules}@anchor{2791}
@deffn {C Variable} const  struct  @ref{1d3,,_frozen}  *PyImport_FrozenModules

This pointer is initialized to point to an array of struct  @ref{1d3,,_frozen}
records, terminated by one whose members are all @code{NULL} or zero.  When a frozen
module is imported, it is searched in this table.  Third-party code could play
tricks with this to provide a dynamically created collection of frozen modules.
@end deffn

@geindex PyImport_AppendInittab (C function)
@anchor{c-api/import c PyImport_AppendInittab}@anchor{112b}
@deffn {C Function} int  PyImport_AppendInittab (const  char  *name, PyObject  *(*initfunc)(void))
@emph{ Part of the @ref{35d,,Stable ABI}.}
Add a single module to the existing table of built-in modules.  This is a
convenience wrapper around @ref{112c,,PyImport_ExtendInittab()}, returning @code{-1} if
the table could not be extended.  The new module can be imported by the name
@emph{name}, and uses the function @emph{initfunc} as the initialization function called
on the first attempted import.  This should be called before
@ref{38d,,Py_Initialize()}.
@end deffn

@geindex _inittab (C struct)
@anchor{c-api/import c _inittab}@anchor{4308}
@deffn {C Type} struct  _inittab

Structure describing a single entry in the list of built-in modules.  Each of
these structures gives the name and initialization function for a module built
into the interpreter.  The name is an ASCII encoded string.  Programs which
embed Python may use an array of these structures in conjunction with
@ref{112c,,PyImport_ExtendInittab()} to provide additional built-in modules.
The structure is defined in @code{Include/import.h} as:

@example
struct _inittab @{
    const char *name;           /* ASCII encoded string */
    PyObject* (*initfunc)(void);
@};
@end example
@end deffn

@geindex PyImport_ExtendInittab (C function)
@anchor{c-api/import c PyImport_ExtendInittab}@anchor{112c}
@deffn {C Function} int  PyImport_ExtendInittab (struct  _inittab  *newtab)

Add a collection of modules to the table of built-in modules.  The @emph{newtab}
array must end with a sentinel entry which contains @code{NULL} for the @code{name}
field; failure to provide the sentinel value can result in a memory fault.
Returns @code{0} on success or @code{-1} if insufficient memory could be allocated to
extend the internal table.  In the event of failure, no modules are added to the
internal table.  This must be called before @ref{38d,,Py_Initialize()}.

If Python is initialized multiple times, @ref{112b,,PyImport_AppendInittab()} or
@ref{112c,,PyImport_ExtendInittab()} must be called before each Python
initialization.
@end deffn

@node Data marshalling support,Parsing arguments and building values,Importing Modules<2>,Utilities<2>
@anchor{c-api/marshal doc}@anchor{4309}@anchor{c-api/marshal data-marshalling-support}@anchor{430a}@anchor{c-api/marshal marshalling-utils}@anchor{430b}
@subsection Data marshalling support


These routines allow C code to work with serialized objects using the same
data format as the @ref{ac,,marshal} module.  There are functions to write data
into the serialization format, and additional functions that can be used to
read the data back.  Files used to store marshalled data must be opened in
binary mode.

Numeric values are stored with the least significant byte first.

The module supports two versions of the data format: version 0 is the
historical version, version 1 shares interned strings in the file, and upon
unmarshalling.  Version 2 uses a binary format for floating point numbers.
@code{Py_MARSHAL_VERSION} indicates the current file format (currently 2).

@geindex PyMarshal_WriteLongToFile (C function)
@anchor{c-api/marshal c PyMarshal_WriteLongToFile}@anchor{20c}
@deffn {C Function} void  PyMarshal_WriteLongToFile (long  value, FILE  *file, int  version)

Marshal a long integer, @emph{value}, to @emph{file}.  This will only write
the least-significant 32 bits of @emph{value}; regardless of the size of the
native long type.  @emph{version} indicates the file format.
@end deffn

@geindex PyMarshal_WriteObjectToFile (C function)
@anchor{c-api/marshal c PyMarshal_WriteObjectToFile}@anchor{20d}
@deffn {C Function} void  PyMarshal_WriteObjectToFile (PyObject  *value, FILE  *file, int  version)

Marshal a Python object, @emph{value}, to @emph{file}.
@emph{version} indicates the file format.
@end deffn

@geindex PyMarshal_WriteObjectToString (C function)
@anchor{c-api/marshal c PyMarshal_WriteObjectToString}@anchor{20f}
@deffn {C Function} @ref{375,,PyObject}  *PyMarshal_WriteObjectToString (PyObject  *value, int  version)
@emph{Return value: New reference.}
Return a bytes object containing the marshalled representation of @emph{value}.
@emph{version} indicates the file format.
@end deffn

The following functions allow marshalled values to be read back in.

@geindex PyMarshal_ReadLongFromFile (C function)
@anchor{c-api/marshal c PyMarshal_ReadLongFromFile}@anchor{430c}
@deffn {C Function} long  PyMarshal_ReadLongFromFile (FILE  *file)

Return a C long from the data stream in a FILE* opened
for reading.  Only a 32-bit value can be read in using this function,
regardless of the native size of long.

On error, sets the appropriate exception (@ref{e74,,EOFError}) and returns
@code{-1}.
@end deffn

@geindex PyMarshal_ReadShortFromFile (C function)
@anchor{c-api/marshal c PyMarshal_ReadShortFromFile}@anchor{430d}
@deffn {C Function} int  PyMarshal_ReadShortFromFile (FILE  *file)

Return a C short from the data stream in a FILE* opened
for reading.  Only a 16-bit value can be read in using this function,
regardless of the native size of short.

On error, sets the appropriate exception (@ref{e74,,EOFError}) and returns
@code{-1}.
@end deffn

@geindex PyMarshal_ReadObjectFromFile (C function)
@anchor{c-api/marshal c PyMarshal_ReadObjectFromFile}@anchor{430e}
@deffn {C Function} @ref{375,,PyObject}  *PyMarshal_ReadObjectFromFile (FILE  *file)
@emph{Return value: New reference.}
Return a Python object from the data stream in a FILE* opened for
reading.

On error, sets the appropriate exception (@ref{e74,,EOFError}, @ref{17c,,ValueError}
or @ref{162,,TypeError}) and returns @code{NULL}.
@end deffn

@geindex PyMarshal_ReadLastObjectFromFile (C function)
@anchor{c-api/marshal c PyMarshal_ReadLastObjectFromFile}@anchor{430f}
@deffn {C Function} @ref{375,,PyObject}  *PyMarshal_ReadLastObjectFromFile (FILE  *file)
@emph{Return value: New reference.}
Return a Python object from the data stream in a FILE* opened for
reading.  Unlike @ref{430e,,PyMarshal_ReadObjectFromFile()}, this function
assumes that no further objects will be read from the file, allowing it to
aggressively load file data into memory so that the de-serialization can
operate from data in memory rather than reading a byte at a time from the
file.  Only use these variant if you are certain that you won’t be reading
anything else from the file.

On error, sets the appropriate exception (@ref{e74,,EOFError}, @ref{17c,,ValueError}
or @ref{162,,TypeError}) and returns @code{NULL}.
@end deffn

@geindex PyMarshal_ReadObjectFromString (C function)
@anchor{c-api/marshal c PyMarshal_ReadObjectFromString}@anchor{20e}
@deffn {C Function} @ref{375,,PyObject}  *PyMarshal_ReadObjectFromString (const  char  *data, Py_ssize_t  len)
@emph{Return value: New reference.}
Return a Python object from the data stream in a byte buffer
containing @emph{len} bytes pointed to by @emph{data}.

On error, sets the appropriate exception (@ref{e74,,EOFError}, @ref{17c,,ValueError}
or @ref{162,,TypeError}) and returns @code{NULL}.
@end deffn

@node Parsing arguments and building values,String conversion and formatting,Data marshalling support,Utilities<2>
@anchor{c-api/arg doc}@anchor{4310}@anchor{c-api/arg arg-parsing}@anchor{381}@anchor{c-api/arg parsing-arguments-and-building-values}@anchor{4311}
@subsection Parsing arguments and building values


These functions are useful when creating your own extensions functions and
methods.  Additional information and examples are available in
@ref{14be,,Extending and Embedding the Python Interpreter}.

The first three of these functions described, @ref{37f,,PyArg_ParseTuple()},
@ref{863,,PyArg_ParseTupleAndKeywords()}, and @ref{1286,,PyArg_Parse()}, all use @emph{format
strings} which are used to tell the function about the expected arguments.  The
format strings use the same syntax for each of these functions.

@menu
* Parsing arguments: Parsing arguments<3>. 
* Building values:: 

@end menu

@node Parsing arguments<3>,Building values,,Parsing arguments and building values
@anchor{c-api/arg parsing-arguments}@anchor{4312}
@subsubsection Parsing arguments


A format string consists of zero or more “format units.”  A format unit
describes one Python object; it is usually a single character or a parenthesized
sequence of format units.  With a few exceptions, a format unit that is not a
parenthesized sequence normally corresponds to a single address argument to
these functions.  In the following description, the quoted form is the format
unit; the entry in (round) parentheses is the Python object type that matches
the format unit; and the entry in [square] brackets is the type of the C
variable(s) whose address should be passed.

@menu
* Strings and buffers:: 
* Numbers: Numbers<2>. 
* Other objects:: 
* API Functions:: 

@end menu

@node Strings and buffers,Numbers<2>,,Parsing arguments<3>
@anchor{c-api/arg strings-and-buffers}@anchor{4313}
@subsubsection Strings and buffers


These formats allow accessing an object as a contiguous chunk of memory.
You don’t have to provide raw storage for the returned unicode or bytes
area.

In general, when a format sets a pointer to a buffer, the buffer is
managed by the corresponding Python object, and the buffer shares
the lifetime of this object.  You won’t have to release any memory yourself.
The only exceptions are @code{es}, @code{es#}, @code{et} and @code{et#}.

However, when a @ref{1db,,Py_buffer} structure gets filled, the underlying
buffer is locked so that the caller can subsequently use the buffer even
inside a @ref{4096,,Py_BEGIN_ALLOW_THREADS} block without the risk of mutable data
being resized or destroyed.  As a result, @strong{you have to call}
@ref{1e5,,PyBuffer_Release()} after you have finished processing the data (or
in any early abort case).

Unless otherwise stated, buffers are not NUL-terminated.

Some formats require a read-only @ref{87e,,bytes-like object}, and set a
pointer instead of a buffer structure.  They work by checking that
the object’s @ref{1e8,,PyBufferProcs.bf_releasebuffer} field is @code{NULL},
which disallows mutable objects such as @ref{2f1,,bytearray}.

@cartouche
@quotation Note 
For all @code{#} variants of formats (@code{s#}, @code{y#}, etc.), the macro
@code{PY_SSIZE_T_CLEAN} must be defined before including
@code{Python.h}. On Python 3.9 and older, the type of the length argument
is @code{Py_ssize_t} if the @code{PY_SSIZE_T_CLEAN} macro is defined,
or int otherwise.
@end quotation
@end cartouche


@table @asis

@item @code{s} (@ref{2ef,,str}) [const char *]

Convert a Unicode object to a C pointer to a character string.
A pointer to an existing string is stored in the character pointer
variable whose address you pass.  The C string is NUL-terminated.
The Python string must not contain embedded null code points; if it does,
a @ref{17c,,ValueError} exception is raised. Unicode objects are converted
to C strings using @code{'utf-8'} encoding. If this conversion fails, a
@ref{e49,,UnicodeError} is raised.

@cartouche
@quotation Note 
This format does not accept @ref{87e,,bytes-like objects}.  If you want to accept
filesystem paths and convert them to C character strings, it is
preferable to use the @code{O&} format with @ref{867,,PyUnicode_FSConverter()}
as @emph{converter}.
@end quotation
@end cartouche


Changed in version 3.5: Previously, @ref{162,,TypeError} was raised when embedded null code points
were encountered in the Python string.


@item @code{s*} (@ref{2ef,,str} or @ref{87e,,bytes-like object}) [Py_buffer]

This format accepts Unicode objects as well as bytes-like objects.
It fills a @ref{1db,,Py_buffer} structure provided by the caller.
In this case the resulting C string may contain embedded NUL bytes.
Unicode objects are converted to C strings using @code{'utf-8'} encoding.

@item @code{s#} (@ref{2ef,,str}, read-only @ref{87e,,bytes-like object}) [const char *, @code{Py_ssize_t}]

Like @code{s*}, except that it doesn’t accept mutable objects.
The result is stored into two C variables,
the first one a pointer to a C string, the second one its length.
The string may contain embedded null bytes. Unicode objects are converted
to C strings using @code{'utf-8'} encoding.

@item @code{z} (@ref{2ef,,str} or @code{None}) [const char *]

Like @code{s}, but the Python object may also be @code{None}, in which case the C
pointer is set to @code{NULL}.

@item @code{z*} (@ref{2ef,,str}, @ref{87e,,bytes-like object} or @code{None}) [Py_buffer]

Like @code{s*}, but the Python object may also be @code{None}, in which case the
@code{buf} member of the @ref{1db,,Py_buffer} structure is set to @code{NULL}.

@item @code{z#} (@ref{2ef,,str}, read-only @ref{87e,,bytes-like object} or @code{None}) [const char *, @code{Py_ssize_t}]

Like @code{s#}, but the Python object may also be @code{None}, in which case the C
pointer is set to @code{NULL}.

@item @code{y} (read-only @ref{87e,,bytes-like object}) [const char *]

This format converts a bytes-like object to a C pointer to a character
string; it does not accept Unicode objects.  The bytes buffer must not
contain embedded null bytes; if it does, a @ref{17c,,ValueError}
exception is raised.


Changed in version 3.5: Previously, @ref{162,,TypeError} was raised when embedded null bytes were
encountered in the bytes buffer.


@item @code{y*} (@ref{87e,,bytes-like object}) [Py_buffer]

This variant on @code{s*} doesn’t accept Unicode objects, only
bytes-like objects.  @strong{This is the recommended way to accept
binary data.}

@item @code{y#} (read-only @ref{87e,,bytes-like object}) [const char *, @code{Py_ssize_t}]

This variant on @code{s#} doesn’t accept Unicode objects, only bytes-like
objects.

@item @code{S} (@ref{2f0,,bytes}) [PyBytesObject *]

Requires that the Python object is a @ref{2f0,,bytes} object, without
attempting any conversion.  Raises @ref{162,,TypeError} if the object is not
a bytes object.  The C variable may also be declared as @ref{375,,PyObject}*.

@item @code{Y} (@ref{2f1,,bytearray}) [PyByteArrayObject *]

Requires that the Python object is a @ref{2f1,,bytearray} object, without
attempting any conversion.  Raises @ref{162,,TypeError} if the object is not
a @ref{2f1,,bytearray} object. The C variable may also be declared as @ref{375,,PyObject}*.

@item @code{u} (@ref{2ef,,str}) [const Py_UNICODE *]

Convert a Python Unicode object to a C pointer to a NUL-terminated buffer of
Unicode characters.  You must pass the address of a @ref{d2c,,Py_UNICODE}
pointer variable, which will be filled with the pointer to an existing
Unicode buffer.  Please note that the width of a @ref{d2c,,Py_UNICODE}
character depends on compilation options (it is either 16 or 32 bits).
The Python string must not contain embedded null code points; if it does,
a @ref{17c,,ValueError} exception is raised.


Changed in version 3.5: Previously, @ref{162,,TypeError} was raised when embedded null code points
were encountered in the Python string.



Deprecated since version 3.3, will be removed in version 3.12: Part of the old-style @ref{d2c,,Py_UNICODE} API; please migrate to using
@ref{39f,,PyUnicode_AsWideCharString()}.


@item @code{u#} (@ref{2ef,,str}) [const Py_UNICODE *, @code{Py_ssize_t}]

This variant on @code{u} stores into two C variables, the first one a pointer to a
Unicode data buffer, the second one its length.  This variant allows
null code points.


Deprecated since version 3.3, will be removed in version 3.12: Part of the old-style @ref{d2c,,Py_UNICODE} API; please migrate to using
@ref{39f,,PyUnicode_AsWideCharString()}.


@item @code{Z} (@ref{2ef,,str} or @code{None}) [const Py_UNICODE *]

Like @code{u}, but the Python object may also be @code{None}, in which case the
@ref{d2c,,Py_UNICODE} pointer is set to @code{NULL}.


Deprecated since version 3.3, will be removed in version 3.12: Part of the old-style @ref{d2c,,Py_UNICODE} API; please migrate to using
@ref{39f,,PyUnicode_AsWideCharString()}.


@item @code{Z#} (@ref{2ef,,str} or @code{None}) [const Py_UNICODE *, @code{Py_ssize_t}]

Like @code{u#}, but the Python object may also be @code{None}, in which case the
@ref{d2c,,Py_UNICODE} pointer is set to @code{NULL}.


Deprecated since version 3.3, will be removed in version 3.12: Part of the old-style @ref{d2c,,Py_UNICODE} API; please migrate to using
@ref{39f,,PyUnicode_AsWideCharString()}.


@item @code{U} (@ref{2ef,,str}) [PyObject *]

Requires that the Python object is a Unicode object, without attempting
any conversion.  Raises @ref{162,,TypeError} if the object is not a Unicode
object.  The C variable may also be declared as @ref{375,,PyObject}*.

@item @code{w*} (read-write @ref{87e,,bytes-like object}) [Py_buffer]

This format accepts any object which implements the read-write buffer
interface. It fills a @ref{1db,,Py_buffer} structure provided by the caller.
The buffer may contain embedded null bytes. The caller have to call
@ref{1e5,,PyBuffer_Release()} when it is done with the buffer.

@item @code{es} (@ref{2ef,,str}) [const char *encoding, char **buffer]

This variant on @code{s} is used for encoding Unicode into a character buffer.
It only works for encoded data without embedded NUL bytes.

This format requires two arguments.  The first is only used as input, and
must be a const  char* which points to the name of an encoding as a
NUL-terminated string, or @code{NULL}, in which case @code{'utf-8'} encoding is used.
An exception is raised if the named encoding is not known to Python.  The
second argument must be a char**; the value of the pointer it
references will be set to a buffer with the contents of the argument text.
The text will be encoded in the encoding specified by the first argument.

@ref{37f,,PyArg_ParseTuple()} will allocate a buffer of the needed size, copy the
encoded data into this buffer and adjust @emph{*buffer} to reference the newly
allocated storage.  The caller is responsible for calling @ref{fa3,,PyMem_Free()} to
free the allocated buffer after use.

@item @code{et} (@ref{2ef,,str}, @ref{2f0,,bytes} or @ref{2f1,,bytearray}) [const char *encoding, char **buffer]

Same as @code{es} except that byte string objects are passed through without
recoding them.  Instead, the implementation assumes that the byte string object uses
the encoding passed in as parameter.

@item @code{es#} (@ref{2ef,,str}) [const char *encoding, char **buffer, @code{Py_ssize_t} *buffer_length]

This variant on @code{s#} is used for encoding Unicode into a character buffer.
Unlike the @code{es} format, this variant allows input data which contains NUL
characters.

It requires three arguments.  The first is only used as input, and must be a
const  char* which points to the name of an encoding as a
NUL-terminated string, or @code{NULL}, in which case @code{'utf-8'} encoding is used.
An exception is raised if the named encoding is not known to Python.  The
second argument must be a char**; the value of the pointer it
references will be set to a buffer with the contents of the argument text.
The text will be encoded in the encoding specified by the first argument.
The third argument must be a pointer to an integer; the referenced integer
will be set to the number of bytes in the output buffer.

There are two modes of operation:

If @emph{*buffer} points a @code{NULL} pointer, the function will allocate a buffer of
the needed size, copy the encoded data into this buffer and set @emph{*buffer} to
reference the newly allocated storage.  The caller is responsible for calling
@ref{fa3,,PyMem_Free()} to free the allocated buffer after usage.

If @emph{*buffer} points to a non-@code{NULL} pointer (an already allocated buffer),
@ref{37f,,PyArg_ParseTuple()} will use this location as the buffer and interpret the
initial value of @emph{*buffer_length} as the buffer size.  It will then copy the
encoded data into the buffer and NUL-terminate it.  If the buffer is not large
enough, a @ref{17c,,ValueError} will be set.

In both cases, @emph{*buffer_length} is set to the length of the encoded data
without the trailing NUL byte.

@item @code{et#} (@ref{2ef,,str}, @ref{2f0,,bytes} or @ref{2f1,,bytearray}) [const char *encoding, char **buffer, @code{Py_ssize_t} *buffer_length]

Same as @code{es#} except that byte string objects are passed through without recoding
them. Instead, the implementation assumes that the byte string object uses the
encoding passed in as parameter.
@end table

@node Numbers<2>,Other objects,Strings and buffers,Parsing arguments<3>
@anchor{c-api/arg numbers}@anchor{4314}
@subsubsection Numbers



@table @asis

@item @code{b} (@ref{1b1,,int}) [unsigned char]

Convert a nonnegative Python integer to an unsigned tiny int, stored in a C
unsigned  char.

@item @code{B} (@ref{1b1,,int}) [unsigned char]

Convert a Python integer to a tiny int without overflow checking, stored in a C
unsigned  char.

@item @code{h} (@ref{1b1,,int}) [short int]

Convert a Python integer to a C short  int.

@item @code{H} (@ref{1b1,,int}) [unsigned short int]

Convert a Python integer to a C unsigned  short  int, without overflow
checking.

@item @code{i} (@ref{1b1,,int}) [int]

Convert a Python integer to a plain C int.

@item @code{I} (@ref{1b1,,int}) [unsigned int]

Convert a Python integer to a C unsigned  int, without overflow
checking.

@item @code{l} (@ref{1b1,,int}) [long int]

Convert a Python integer to a C long  int.

@item @code{k} (@ref{1b1,,int}) [unsigned long]

Convert a Python integer to a C unsigned  long without
overflow checking.

@item @code{L} (@ref{1b1,,int}) [long long]

Convert a Python integer to a C long  long.

@item @code{K} (@ref{1b1,,int}) [unsigned long long]

Convert a Python integer to a C unsigned  long  long
without overflow checking.

@item @code{n} (@ref{1b1,,int}) [Py_ssize_t]

Convert a Python integer to a C @code{Py_ssize_t}.

@item @code{c} (@ref{2f0,,bytes} or @ref{2f1,,bytearray} of length 1) [char]

Convert a Python byte, represented as a @ref{2f0,,bytes} or
@ref{2f1,,bytearray} object of length 1, to a C char.


Changed in version 3.3: Allow @ref{2f1,,bytearray} objects.


@item @code{C} (@ref{2ef,,str} of length 1) [int]

Convert a Python character, represented as a @ref{2ef,,str} object of
length 1, to a C int.

@item @code{f} (@ref{256,,float}) [float]

Convert a Python floating point number to a C float.

@item @code{d} (@ref{256,,float}) [double]

Convert a Python floating point number to a C double.

@item @code{D} (@ref{33b,,complex}) [Py_complex]

Convert a Python complex number to a C @ref{4315,,Py_complex} structure.
@end table

@node Other objects,API Functions,Numbers<2>,Parsing arguments<3>
@anchor{c-api/arg other-objects}@anchor{4316}
@subsubsection Other objects



@table @asis

@item @code{O} (object) [PyObject *]

Store a Python object (without any conversion) in a C object pointer.  The C
program thus receives the actual object that was passed.  The object’s reference
count is not increased.  The pointer stored is not @code{NULL}.

@item @code{O!} (object) [@emph{typeobject}, PyObject *]

Store a Python object in a C object pointer.  This is similar to @code{O}, but
takes two C arguments: the first is the address of a Python type object, the
second is the address of the C variable (of type @ref{375,,PyObject}*) into which
the object pointer is stored.  If the Python object does not have the required
type, @ref{162,,TypeError} is raised.
@end table
@anchor{c-api/arg o-ampersand}@anchor{4317}

@table @asis

@item @code{O&} (object) [@emph{converter}, @emph{anything}]

Convert a Python object to a C variable through a @emph{converter} function.  This
takes two arguments: the first is a function, the second is the address of a C
variable (of arbitrary type), converted to void*.  The @emph{converter}
function in turn is called as follows:

@example
status = converter(object, address);
@end example

where @emph{object} is the Python object to be converted and @emph{address} is the
void* argument that was passed to the @ref{1286,,PyArg_Parse}* function.
The returned @emph{status} should be @code{1} for a successful conversion and @code{0} if
the conversion has failed.  When the conversion fails, the @emph{converter} function
should raise an exception and leave the content of @emph{address} unmodified.

If the @emph{converter} returns @code{Py_CLEANUP_SUPPORTED}, it may get called a
second time if the argument parsing eventually fails, giving the converter a
chance to release any memory that it had already allocated. In this second
call, the @emph{object} parameter will be @code{NULL}; @emph{address} will have the same value
as in the original call.


Changed in version 3.1: @code{Py_CLEANUP_SUPPORTED} was added.


@item @code{p} (@ref{2f7,,bool}) [int]

Tests the value passed in for truth (a boolean @strong{p}redicate) and converts
the result to its equivalent C true/false integer value.
Sets the int to @code{1} if the expression was true and @code{0} if it was false.
This accepts any valid Python value.  See @ref{1a3a,,Truth Value Testing} for more
information about how Python tests values for truth.


New in version 3.3.


@item @code{(items)} (@ref{3e5,,tuple}) [@emph{matching-items}]

The object must be a Python sequence whose length is the number of format units
in @emph{items}.  The C arguments must correspond to the individual format units in
@emph{items}.  Format units for sequences may be nested.
@end table

It is possible to pass “long” integers (integers whose value exceeds the
platform’s @code{LONG_MAX}) however no proper range checking is done — the
most significant bits are silently truncated when the receiving field is too
small to receive the value (actually, the semantics are inherited from downcasts
in C — your mileage may vary).

A few other characters have a meaning in a format string.  These may not occur
inside nested parentheses.  They are:


@table @asis

@item @code{|}

Indicates that the remaining arguments in the Python argument list are optional.
The C variables corresponding to optional arguments should be initialized to
their default value — when an optional argument is not specified,
@ref{37f,,PyArg_ParseTuple()} does not touch the contents of the corresponding C
variable(s).

@item @code{$}

@ref{863,,PyArg_ParseTupleAndKeywords()} only:
Indicates that the remaining arguments in the Python argument list are
keyword-only.  Currently, all keyword-only arguments must also be optional
arguments, so @code{|} must always be specified before @code{$} in the format
string.


New in version 3.3.


@item @code{:}

The list of format units ends here; the string after the colon is used as the
function name in error messages (the “associated value” of the exception that
@ref{37f,,PyArg_ParseTuple()} raises).

@item @code{;}

The list of format units ends here; the string after the semicolon is used as
the error message @emph{instead} of the default error message.  @code{:} and @code{;}
mutually exclude each other.
@end table

Note that any Python object references which are provided to the caller are
@emph{borrowed} references; do not decrement their reference count!

Additional arguments passed to these functions must be addresses of variables
whose type is determined by the format string; these are used to store values
from the input tuple.  There are a few cases, as described in the list of format
units above, where these parameters are used as input values; they should match
what is specified for the corresponding format unit in that case.

For the conversion to succeed, the @emph{arg} object must match the format
and the format must be exhausted.  On success, the
@ref{1286,,PyArg_Parse}* functions return true, otherwise they return
false and raise an appropriate exception. When the
@ref{1286,,PyArg_Parse}* functions fail due to conversion failure in one
of the format units, the variables at the addresses corresponding to that
and the following format units are left untouched.

@node API Functions,,Other objects,Parsing arguments<3>
@anchor{c-api/arg api-functions}@anchor{4318}
@subsubsection API Functions


@geindex PyArg_ParseTuple (C function)
@anchor{c-api/arg c PyArg_ParseTuple}@anchor{37f}
@deffn {C Function} int  PyArg_ParseTuple (PyObject  *args, const  char  *format, ...)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Parse the parameters of a function that takes only positional parameters into
local variables.  Returns true on success; on failure, it returns false and
raises the appropriate exception.
@end deffn

@geindex PyArg_VaParse (C function)
@anchor{c-api/arg c PyArg_VaParse}@anchor{412a}
@deffn {C Function} int  PyArg_VaParse (PyObject  *args, const  char  *format, va_list  vargs)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Identical to @ref{37f,,PyArg_ParseTuple()}, except that it accepts a va_list rather
than a variable number of arguments.
@end deffn

@geindex PyArg_ParseTupleAndKeywords (C function)
@anchor{c-api/arg c PyArg_ParseTupleAndKeywords}@anchor{863}
@deffn {C Function} int  PyArg_ParseTupleAndKeywords (PyObject  *args, PyObject  *kw, const  char  *format, char  *keywords[], ...)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Parse the parameters of a function that takes both positional and keyword
parameters into local variables.  The @emph{keywords} argument is a
@code{NULL}-terminated array of keyword parameter names.  Empty names denote
@ref{59d,,positional-only parameters}.
Returns true on success; on failure, it returns false and raises the
appropriate exception.


Changed in version 3.6: Added support for @ref{59d,,positional-only parameters}.

@end deffn

@geindex PyArg_VaParseTupleAndKeywords (C function)
@anchor{c-api/arg c PyArg_VaParseTupleAndKeywords}@anchor{fd3}
@deffn {C Function} int  PyArg_VaParseTupleAndKeywords (PyObject  *args, PyObject  *kw, const  char  *format, char  *keywords[], va_list  vargs)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Identical to @ref{863,,PyArg_ParseTupleAndKeywords()}, except that it accepts a
va_list rather than a variable number of arguments.
@end deffn

@geindex PyArg_ValidateKeywordArguments (C function)
@anchor{c-api/arg c PyArg_ValidateKeywordArguments}@anchor{412b}
@deffn {C Function} int  PyArg_ValidateKeywordArguments (PyObject*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Ensure that the keys in the keywords argument dictionary are strings.  This
is only needed if @ref{863,,PyArg_ParseTupleAndKeywords()} is not used, since the
latter already does this check.


New in version 3.2.

@end deffn

@c XXX deprecated, will be removed

@geindex PyArg_Parse (C function)
@anchor{c-api/arg c PyArg_Parse}@anchor{1286}
@deffn {C Function} int  PyArg_Parse (PyObject  *args, const  char  *format, ...)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Function used to deconstruct the argument lists of “old-style” functions —
these are functions which use the @code{METH_OLDARGS} parameter parsing
method, which has been removed in Python 3.  This is not recommended for use
in parameter parsing in new code, and most code in the standard interpreter
has been modified to no longer use this for that purpose.  It does remain a
convenient way to decompose other tuples, however, and may continue to be
used for that purpose.
@end deffn

@geindex PyArg_UnpackTuple (C function)
@anchor{c-api/arg c PyArg_UnpackTuple}@anchor{103c}
@deffn {C Function} int  PyArg_UnpackTuple (PyObject  *args, const  char  *name, Py_ssize_t  min, Py_ssize_t  max, ...)
@emph{ Part of the @ref{35d,,Stable ABI}.}
A simpler form of parameter retrieval which does not use a format string to
specify the types of the arguments.  Functions which use this method to retrieve
their parameters should be declared as @ref{103e,,METH_VARARGS} in function or
method tables.  The tuple containing the actual parameters should be passed as
@emph{args}; it must actually be a tuple.  The length of the tuple must be at least
@emph{min} and no more than @emph{max}; @emph{min} and @emph{max} may be equal.  Additional
arguments must be passed to the function, each of which should be a pointer to a
@ref{375,,PyObject}* variable; these will be filled in with the values from
@emph{args}; they will contain @ref{4112,,borrowed references}.
The variables which correspond
to optional parameters not given by @emph{args} will not be filled in; these should
be initialized by the caller. This function returns true on success and false if
@emph{args} is not a tuple or contains the wrong number of elements; an exception
will be set if there was a failure.

This is an example of the use of this function, taken from the sources for the
@code{_weakref} helper module for weak references:

@example
static PyObject *
weakref_ref(PyObject *self, PyObject *args)
@{
    PyObject *object;
    PyObject *callback = NULL;
    PyObject *result = NULL;

    if (PyArg_UnpackTuple(args, "ref", 1, 2, &object, &callback)) @{
        result = PyWeakref_NewRef(object, callback);
    @}
    return result;
@}
@end example

The call to @ref{103c,,PyArg_UnpackTuple()} in this example is entirely equivalent to
this call to @ref{37f,,PyArg_ParseTuple()}:

@example
PyArg_ParseTuple(args, "O|O:ref", &object, &callback)
@end example
@end deffn

@node Building values,,Parsing arguments<3>,Parsing arguments and building values
@anchor{c-api/arg building-values}@anchor{4319}
@subsubsection Building values


@geindex Py_BuildValue (C function)
@anchor{c-api/arg c Py_BuildValue}@anchor{380}
@deffn {C Function} @ref{375,,PyObject}  *Py_BuildValue (const  char  *format, ...)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a new value based on a format string similar to those accepted by the
@ref{1286,,PyArg_Parse}* family of functions and a sequence of values.  Returns
the value or @code{NULL} in the case of an error; an exception will be raised if
@code{NULL} is returned.

@ref{380,,Py_BuildValue()} does not always build a tuple.  It builds a tuple only if
its format string contains two or more format units.  If the format string is
empty, it returns @code{None}; if it contains exactly one format unit, it returns
whatever object is described by that format unit.  To force it to return a tuple
of size 0 or one, parenthesize the format string.

When memory buffers are passed as parameters to supply data to build objects, as
for the @code{s} and @code{s#} formats, the required data is copied.  Buffers provided
by the caller are never referenced by the objects created by
@ref{380,,Py_BuildValue()}.  In other words, if your code invokes @code{malloc()}
and passes the allocated memory to @ref{380,,Py_BuildValue()}, your code is
responsible for calling @code{free()} for that memory once
@ref{380,,Py_BuildValue()} returns.

In the following description, the quoted form is the format unit; the entry in
(round) parentheses is the Python object type that the format unit will return;
and the entry in [square] brackets is the type of the C value(s) to be passed.

The characters space, tab, colon and comma are ignored in format strings (but
not within format units such as @code{s#}).  This can be used to make long format
strings a tad more readable.


@table @asis

@item @code{s} (@ref{2ef,,str} or @code{None}) [const char *]

Convert a null-terminated C string to a Python @ref{2ef,,str} object using @code{'utf-8'}
encoding. If the C string pointer is @code{NULL}, @code{None} is used.

@item @code{s#} (@ref{2ef,,str} or @code{None}) [const char *, @code{Py_ssize_t}]

Convert a C string and its length to a Python @ref{2ef,,str} object using @code{'utf-8'}
encoding. If the C string pointer is @code{NULL}, the length is ignored and
@code{None} is returned.

@item @code{y} (@ref{2f0,,bytes}) [const char *]

This converts a C string to a Python @ref{2f0,,bytes} object.  If the C
string pointer is @code{NULL}, @code{None} is returned.

@item @code{y#} (@ref{2f0,,bytes}) [const char *, @code{Py_ssize_t}]

This converts a C string and its lengths to a Python object.  If the C
string pointer is @code{NULL}, @code{None} is returned.

@item @code{z} (@ref{2ef,,str} or @code{None}) [const char *]

Same as @code{s}.

@item @code{z#} (@ref{2ef,,str} or @code{None}) [const char *, @code{Py_ssize_t}]

Same as @code{s#}.

@item @code{u} (@ref{2ef,,str}) [const wchar_t *]

Convert a null-terminated @code{wchar_t} buffer of Unicode (UTF-16 or UCS-4)
data to a Python Unicode object.  If the Unicode buffer pointer is @code{NULL},
@code{None} is returned.

@item @code{u#} (@ref{2ef,,str}) [const wchar_t *, @code{Py_ssize_t}]

Convert a Unicode (UTF-16 or UCS-4) data buffer and its length to a Python
Unicode object.   If the Unicode buffer pointer is @code{NULL}, the length is ignored
and @code{None} is returned.

@item @code{U} (@ref{2ef,,str} or @code{None}) [const char *]

Same as @code{s}.

@item @code{U#} (@ref{2ef,,str} or @code{None}) [const char *, @code{Py_ssize_t}]

Same as @code{s#}.

@item @code{i} (@ref{1b1,,int}) [int]

Convert a plain C int to a Python integer object.

@item @code{b} (@ref{1b1,,int}) [char]

Convert a plain C char to a Python integer object.

@item @code{h} (@ref{1b1,,int}) [short int]

Convert a plain C short  int to a Python integer object.

@item @code{l} (@ref{1b1,,int}) [long int]

Convert a C long  int to a Python integer object.

@item @code{B} (@ref{1b1,,int}) [unsigned char]

Convert a C unsigned  char to a Python integer object.

@item @code{H} (@ref{1b1,,int}) [unsigned short int]

Convert a C unsigned  short  int to a Python integer object.

@item @code{I} (@ref{1b1,,int}) [unsigned int]

Convert a C unsigned  int to a Python integer object.

@item @code{k} (@ref{1b1,,int}) [unsigned long]

Convert a C unsigned  long to a Python integer object.

@item @code{L} (@ref{1b1,,int}) [long long]

Convert a C long  long to a Python integer object.

@item @code{K} (@ref{1b1,,int}) [unsigned long long]

Convert a C unsigned  long  long to a Python integer object.

@item @code{n} (@ref{1b1,,int}) [Py_ssize_t]

Convert a C @code{Py_ssize_t} to a Python integer.

@item @code{c} (@ref{2f0,,bytes} of length 1) [char]

Convert a C int representing a byte to a Python @ref{2f0,,bytes} object of
length 1.

@item @code{C} (@ref{2ef,,str} of length 1) [int]

Convert a C int representing a character to Python @ref{2ef,,str}
object of length 1.

@item @code{d} (@ref{256,,float}) [double]

Convert a C double to a Python floating point number.

@item @code{f} (@ref{256,,float}) [float]

Convert a C float to a Python floating point number.

@item @code{D} (@ref{33b,,complex}) [Py_complex *]

Convert a C @ref{4315,,Py_complex} structure to a Python complex number.

@item @code{O} (object) [PyObject *]

Pass a Python object untouched (except for its reference count, which is
incremented by one).  If the object passed in is a @code{NULL} pointer, it is assumed
that this was caused because the call producing the argument found an error and
set an exception. Therefore, @ref{380,,Py_BuildValue()} will return @code{NULL} but won’t
raise an exception.  If no exception has been raised yet, @ref{891,,SystemError} is
set.

@item @code{S} (object) [PyObject *]

Same as @code{O}.

@item @code{N} (object) [PyObject *]

Same as @code{O}, except it doesn’t increment the reference count on the object.
Useful when the object is created by a call to an object constructor in the
argument list.

@item @code{O&} (object) [@emph{converter}, @emph{anything}]

Convert @emph{anything} to a Python object through a @emph{converter} function.  The
function is called with @emph{anything} (which should be compatible with void*)
as its argument and should return a “new” Python object, or @code{NULL} if an
error occurred.

@item @code{(items)} (@ref{3e5,,tuple}) [@emph{matching-items}]

Convert a sequence of C values to a Python tuple with the same number of items.

@item @code{[items]} (@ref{43a,,list}) [@emph{matching-items}]

Convert a sequence of C values to a Python list with the same number of items.

@item @code{@{items@}} (@ref{3b0,,dict}) [@emph{matching-items}]

Convert a sequence of C values to a Python dictionary.  Each pair of consecutive
C values adds one item to the dictionary, serving as key and value,
respectively.
@end table

If there is an error in the format string, the @ref{891,,SystemError} exception is
set and @code{NULL} returned.
@end deffn

@geindex Py_VaBuildValue (C function)
@anchor{c-api/arg c Py_VaBuildValue}@anchor{42a4}
@deffn {C Function} @ref{375,,PyObject}  *Py_VaBuildValue (const  char  *format, va_list  vargs)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Identical to @ref{380,,Py_BuildValue()}, except that it accepts a va_list
rather than a variable number of arguments.
@end deffn

@node String conversion and formatting,Reflection,Parsing arguments and building values,Utilities<2>
@anchor{c-api/conversion doc}@anchor{431a}@anchor{c-api/conversion string-conversion}@anchor{431b}@anchor{c-api/conversion string-conversion-and-formatting}@anchor{431c}
@subsection String conversion and formatting


Functions for number conversion and formatted string output.

@geindex PyOS_snprintf (C function)
@anchor{c-api/conversion c PyOS_snprintf}@anchor{103f}
@deffn {C Function} int  PyOS_snprintf (char  *str, size_t  size, const  char  *format, ...)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Output not more than @emph{size} bytes to @emph{str} according to the format string
@emph{format} and the extra arguments. See the Unix man page @code{snprintf(3)}.
@end deffn

@geindex PyOS_vsnprintf (C function)
@anchor{c-api/conversion c PyOS_vsnprintf}@anchor{1040}
@deffn {C Function} int  PyOS_vsnprintf (char  *str, size_t  size, const  char  *format, va_list  va)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Output not more than @emph{size} bytes to @emph{str} according to the format string
@emph{format} and the variable argument list @emph{va}. Unix man page
@code{vsnprintf(3)}.
@end deffn

@ref{103f,,PyOS_snprintf()} and @ref{1040,,PyOS_vsnprintf()} wrap the Standard C library
functions @code{snprintf()} and @code{vsnprintf()}. Their purpose is to
guarantee consistent behavior in corner cases, which the Standard C functions do
not.

The wrappers ensure that @code{str[size-1]} is always @code{'\0'} upon return. They
never write more than @emph{size} bytes (including the trailing @code{'\0'}) into str.
Both functions require that @code{str != NULL}, @code{size > 0}, @code{format != NULL}
and @code{size < INT_MAX}.

The return value (@emph{rv}) for these functions should be interpreted as follows:


@itemize *

@item 
When @code{0 <= rv < size}, the output conversion was successful and @emph{rv}
characters were written to @emph{str} (excluding the trailing @code{'\0'} byte at
@code{str[rv]}).

@item 
When @code{rv >= size}, the output conversion was truncated and a buffer with
@code{rv + 1} bytes would have been needed to succeed. @code{str[size-1]} is @code{'\0'}
in this case.

@item 
When @code{rv < 0}, “something bad happened.” @code{str[size-1]} is @code{'\0'} in
this case too, but the rest of @emph{str} is undefined. The exact cause of the error
depends on the underlying platform.
@end itemize

The following functions provide locale-independent string to number conversions.

@geindex PyOS_string_to_double (C function)
@anchor{c-api/conversion c PyOS_string_to_double}@anchor{e3b}
@deffn {C Function} double  PyOS_string_to_double (const  char  *s, char  **endptr, PyObject  *overflow_exception)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Convert a string @code{s} to a double, raising a Python
exception on failure.  The set of accepted strings corresponds to
the set of strings accepted by Python’s @ref{256,,float()} constructor,
except that @code{s} must not have leading or trailing whitespace.
The conversion is independent of the current locale.

If @code{endptr} is @code{NULL}, convert the whole string.  Raise
@ref{17c,,ValueError} and return @code{-1.0} if the string is not a valid
representation of a floating-point number.

If endptr is not @code{NULL}, convert as much of the string as
possible and set @code{*endptr} to point to the first unconverted
character.  If no initial segment of the string is the valid
representation of a floating-point number, set @code{*endptr} to point
to the beginning of the string, raise ValueError, and return
@code{-1.0}.

If @code{s} represents a value that is too large to store in a float
(for example, @code{"1e500"} is such a string on many platforms) then
if @code{overflow_exception} is @code{NULL} return @code{Py_HUGE_VAL} (with
an appropriate sign) and don’t set any exception.  Otherwise,
@code{overflow_exception} must point to a Python exception object;
raise that exception and return @code{-1.0}.  In both cases, set
@code{*endptr} to point to the first character after the converted value.

If any other error occurs during the conversion (for example an
out-of-memory error), set the appropriate Python exception and
return @code{-1.0}.


New in version 3.1.

@end deffn

@geindex PyOS_double_to_string (C function)
@anchor{c-api/conversion c PyOS_double_to_string}@anchor{41f6}
@deffn {C Function} char  *PyOS_double_to_string (double  val, char  format_code, int  precision, int  flags, int  *ptype)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Convert a double @emph{val} to a string using supplied
@emph{format_code}, @emph{precision}, and @emph{flags}.

@emph{format_code} must be one of @code{'e'}, @code{'E'}, @code{'f'}, @code{'F'},
@code{'g'}, @code{'G'} or @code{'r'}.  For @code{'r'}, the supplied @emph{precision}
must be 0 and is ignored.  The @code{'r'} format code specifies the
standard @ref{27d,,repr()} format.

@emph{flags} can be zero or more of the values @code{Py_DTSF_SIGN},
@code{Py_DTSF_ADD_DOT_0}, or @code{Py_DTSF_ALT}, or-ed together:


@itemize *

@item 
@code{Py_DTSF_SIGN} means to always precede the returned string with a sign
character, even if @emph{val} is non-negative.

@item 
@code{Py_DTSF_ADD_DOT_0} means to ensure that the returned string will not look
like an integer.

@item 
@code{Py_DTSF_ALT} means to apply “alternate” formatting rules.  See the
documentation for the @ref{103f,,PyOS_snprintf()} @code{'#'} specifier for
details.
@end itemize

If @emph{ptype} is non-@code{NULL}, then the value it points to will be set to one of
@code{Py_DTST_FINITE}, @code{Py_DTST_INFINITE}, or @code{Py_DTST_NAN}, signifying that
@emph{val} is a finite number, an infinite number, or not a number, respectively.

The return value is a pointer to @emph{buffer} with the converted string or
@code{NULL} if the conversion failed. The caller is responsible for freeing the
returned string by calling @ref{fa3,,PyMem_Free()}.


New in version 3.1.

@end deffn

@geindex PyOS_stricmp (C function)
@anchor{c-api/conversion c PyOS_stricmp}@anchor{431d}
@deffn {C Function} int  PyOS_stricmp (const  char  *s1, const  char  *s2)

Case insensitive comparison of strings. The function works almost
identically to @code{strcmp()} except that it ignores the case.
@end deffn

@geindex PyOS_strnicmp (C function)
@anchor{c-api/conversion c PyOS_strnicmp}@anchor{431e}
@deffn {C Function} int  PyOS_strnicmp (const  char  *s1, const  char  *s2, Py_ssize_t  size)

Case insensitive comparison of strings. The function works almost
identically to @code{strncmp()} except that it ignores the case.
@end deffn

@node Reflection,Codec registry and support functions,String conversion and formatting,Utilities<2>
@anchor{c-api/reflection doc}@anchor{431f}@anchor{c-api/reflection id1}@anchor{4320}@anchor{c-api/reflection reflection}@anchor{4321}
@subsection Reflection


@geindex PyEval_GetBuiltins (C function)
@anchor{c-api/reflection c PyEval_GetBuiltins}@anchor{4187}
@deffn {C Function} @ref{375,,PyObject}  *PyEval_GetBuiltins (void)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a dictionary of the builtins in the current execution frame,
or the interpreter of the thread state if no frame is currently executing.
@end deffn

@geindex PyEval_GetLocals (C function)
@anchor{c-api/reflection c PyEval_GetLocals}@anchor{418c}
@deffn {C Function} @ref{375,,PyObject}  *PyEval_GetLocals (void)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a dictionary of the local variables in the current execution frame,
or @code{NULL} if no frame is currently executing.
@end deffn

@geindex PyEval_GetGlobals (C function)
@anchor{c-api/reflection c PyEval_GetGlobals}@anchor{418b}
@deffn {C Function} @ref{375,,PyObject}  *PyEval_GetGlobals (void)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a dictionary of the global variables in the current execution frame,
or @code{NULL} if no frame is currently executing.
@end deffn

@geindex PyEval_GetFrame (C function)
@anchor{c-api/reflection c PyEval_GetFrame}@anchor{4188}
@deffn {C Function} @ref{1f6,,PyFrameObject}  *PyEval_GetFrame (void)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the current thread state’s frame, which is @code{NULL} if no frame is
currently executing.

See also @ref{1fb,,PyThreadState_GetFrame()}.
@end deffn

@geindex PyFrame_GetBack (C function)
@anchor{c-api/reflection c PyFrame_GetBack}@anchor{1f7}
@deffn {C Function} @ref{1f6,,PyFrameObject}  *PyFrame_GetBack (PyFrameObject  *frame)

Get the @emph{frame} next outer frame.

Return a @ref{1fc,,strong reference}, or @code{NULL} if @emph{frame} has no outer frame.

@emph{frame} must not be @code{NULL}.


New in version 3.9.

@end deffn

@geindex PyFrame_GetCode (C function)
@anchor{c-api/reflection c PyFrame_GetCode}@anchor{1f8}
@deffn {C Function} @ref{4322,,PyCodeObject}  *PyFrame_GetCode (PyFrameObject  *frame)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Get the @emph{frame} code.

Return a @ref{1fc,,strong reference}.

@emph{frame} must not be @code{NULL}. The result (frame code) cannot be @code{NULL}.


New in version 3.9.

@end deffn

@geindex PyFrame_GetLineNumber (C function)
@anchor{c-api/reflection c PyFrame_GetLineNumber}@anchor{1f9}
@deffn {C Function} int  PyFrame_GetLineNumber (PyFrameObject  *frame)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Return the line number that @emph{frame} is currently executing.

@emph{frame} must not be @code{NULL}.
@end deffn

@geindex PyEval_GetFuncName (C function)
@anchor{c-api/reflection c PyEval_GetFuncName}@anchor{418a}
@deffn {C Function} const  char  *PyEval_GetFuncName (PyObject  *func)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the name of @emph{func} if it is a function, class or instance object, else the
name of @emph{func}s type.
@end deffn

@geindex PyEval_GetFuncDesc (C function)
@anchor{c-api/reflection c PyEval_GetFuncDesc}@anchor{4189}
@deffn {C Function} const  char  *PyEval_GetFuncDesc (PyObject  *func)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a description string, depending on the type of @emph{func}.
Return values include “()” for functions and methods, ” constructor”,
” instance”, and ” object”.  Concatenated with the result of
@ref{418a,,PyEval_GetFuncName()}, the result will be a description of
@emph{func}.
@end deffn

@node Codec registry and support functions,,Reflection,Utilities<2>
@anchor{c-api/codec doc}@anchor{4323}@anchor{c-api/codec codec-registry}@anchor{884}@anchor{c-api/codec codec-registry-and-support-functions}@anchor{4324}
@subsection Codec registry and support functions


@geindex PyCodec_Register (C function)
@anchor{c-api/codec c PyCodec_Register}@anchor{4151}
@deffn {C Function} int  PyCodec_Register (PyObject  *search_function)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Register a new codec search function.

As side effect, this tries to load the @code{encodings} package, if not yet
done, to make sure that it is always first in the list of search functions.
@end deffn

@geindex PyCodec_Unregister (C function)
@anchor{c-api/codec c PyCodec_Unregister}@anchor{365}
@deffn {C Function} int  PyCodec_Unregister (PyObject  *search_function)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Unregister a codec search function and clear the registry’s cache.
If the search function is not registered, do nothing.
Return 0 on success. Raise an exception and return -1 on error.


New in version 3.10.

@end deffn

@geindex PyCodec_KnownEncoding (C function)
@anchor{c-api/codec c PyCodec_KnownEncoding}@anchor{414f}
@deffn {C Function} int  PyCodec_KnownEncoding (const  char  *encoding)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return @code{1} or @code{0} depending on whether there is a registered codec for
the given @emph{encoding}.  This function always succeeds.
@end deffn

@geindex PyCodec_Encode (C function)
@anchor{c-api/codec c PyCodec_Encode}@anchor{414a}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_Encode (PyObject  *object, const  char  *encoding, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Generic codec based encoding API.

@emph{object} is passed through the encoder function found for the given
@emph{encoding} using the error handling method defined by @emph{errors}.  @emph{errors} may
be @code{NULL} to use the default method defined for the codec.  Raises a
@ref{1ab5,,LookupError} if no encoder can be found.
@end deffn

@geindex PyCodec_Decode (C function)
@anchor{c-api/codec c PyCodec_Decode}@anchor{4148}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_Decode (PyObject  *object, const  char  *encoding, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Generic codec based decoding API.

@emph{object} is passed through the decoder function found for the given
@emph{encoding} using the error handling method defined by @emph{errors}.  @emph{errors} may
be @code{NULL} to use the default method defined for the codec.  Raises a
@ref{1ab5,,LookupError} if no encoder can be found.
@end deffn

@menu
* Codec lookup API:: 
* Registry API for Unicode encoding error handlers:: 

@end menu

@node Codec lookup API,Registry API for Unicode encoding error handlers,,Codec registry and support functions
@anchor{c-api/codec codec-lookup-api}@anchor{4325}
@subsubsection Codec lookup API


In the following functions, the @emph{encoding} string is looked up converted to all
lower-case characters, which makes encodings looked up through this mechanism
effectively case-insensitive.  If no codec is found, a @ref{5bb,,KeyError} is set
and @code{NULL} returned.

@geindex PyCodec_Encoder (C function)
@anchor{c-api/codec c PyCodec_Encoder}@anchor{414b}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_Encoder (const  char  *encoding)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Get an encoder function for the given @emph{encoding}.
@end deffn

@geindex PyCodec_Decoder (C function)
@anchor{c-api/codec c PyCodec_Decoder}@anchor{4149}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_Decoder (const  char  *encoding)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Get a decoder function for the given @emph{encoding}.
@end deffn

@geindex PyCodec_IncrementalEncoder (C function)
@anchor{c-api/codec c PyCodec_IncrementalEncoder}@anchor{414e}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_IncrementalEncoder (const  char  *encoding, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Get an @ref{1cad,,IncrementalEncoder} object for the given @emph{encoding}.
@end deffn

@geindex PyCodec_IncrementalDecoder (C function)
@anchor{c-api/codec c PyCodec_IncrementalDecoder}@anchor{414d}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_IncrementalDecoder (const  char  *encoding, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Get an @ref{1cae,,IncrementalDecoder} object for the given @emph{encoding}.
@end deffn

@geindex PyCodec_StreamReader (C function)
@anchor{c-api/codec c PyCodec_StreamReader}@anchor{4154}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_StreamReader (const  char  *encoding, PyObject  *stream, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Get a @ref{1cb2,,StreamReader} factory function for the given @emph{encoding}.
@end deffn

@geindex PyCodec_StreamWriter (C function)
@anchor{c-api/codec c PyCodec_StreamWriter}@anchor{4155}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_StreamWriter (const  char  *encoding, PyObject  *stream, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Get a @ref{1cb1,,StreamWriter} factory function for the given @emph{encoding}.
@end deffn

@node Registry API for Unicode encoding error handlers,,Codec lookup API,Codec registry and support functions
@anchor{c-api/codec registry-api-for-unicode-encoding-error-handlers}@anchor{4326}
@subsubsection Registry API for Unicode encoding error handlers


@geindex PyCodec_RegisterError (C function)
@anchor{c-api/codec c PyCodec_RegisterError}@anchor{4152}
@deffn {C Function} int  PyCodec_RegisterError (const  char  *name, PyObject  *error)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Register the error handling callback function @emph{error} under the given @emph{name}.
This callback function will be called by a codec when it encounters
unencodable characters/undecodable bytes and @emph{name} is specified as the error
parameter in the call to the encode/decode function.

The callback gets a single argument, an instance of
@ref{185,,UnicodeEncodeError}, @ref{51f,,UnicodeDecodeError} or
@ref{1b88,,UnicodeTranslateError} that holds information about the problematic
sequence of characters or bytes and their offset in the original string (see
@ref{42eb,,Unicode Exception Objects} for functions to extract this information).  The
callback must either raise the given exception, or return a two-item tuple
containing the replacement for the problematic sequence, and an integer
giving the offset in the original string at which encoding/decoding should be
resumed.

Return @code{0} on success, @code{-1} on error.
@end deffn

@geindex PyCodec_LookupError (C function)
@anchor{c-api/codec c PyCodec_LookupError}@anchor{4150}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_LookupError (const  char  *name)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Lookup the error handling callback function registered under @emph{name}.  As a
special case @code{NULL} can be passed, in which case the error handling callback
for “strict” will be returned.
@end deffn

@geindex PyCodec_StrictErrors (C function)
@anchor{c-api/codec c PyCodec_StrictErrors}@anchor{4156}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_StrictErrors (PyObject  *exc)
@emph{Return value: Always NULL.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Raise @emph{exc} as an exception.
@end deffn

@geindex PyCodec_IgnoreErrors (C function)
@anchor{c-api/codec c PyCodec_IgnoreErrors}@anchor{414c}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_IgnoreErrors (PyObject  *exc)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Ignore the unicode error, skipping the faulty input.
@end deffn

@geindex PyCodec_ReplaceErrors (C function)
@anchor{c-api/codec c PyCodec_ReplaceErrors}@anchor{4153}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_ReplaceErrors (PyObject  *exc)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Replace the unicode encode error with @code{?} or @code{U+FFFD}.
@end deffn

@geindex PyCodec_XMLCharRefReplaceErrors (C function)
@anchor{c-api/codec c PyCodec_XMLCharRefReplaceErrors}@anchor{4157}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_XMLCharRefReplaceErrors (PyObject  *exc)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Replace the unicode encode error with XML character references.
@end deffn

@geindex PyCodec_BackslashReplaceErrors (C function)
@anchor{c-api/codec c PyCodec_BackslashReplaceErrors}@anchor{4147}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_BackslashReplaceErrors (PyObject  *exc)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Replace the unicode encode error with backslash escapes (@code{\x}, @code{\u} and
@code{\U}).
@end deffn

@geindex PyCodec_NameReplaceErrors (C function)
@anchor{c-api/codec c PyCodec_NameReplaceErrors}@anchor{a21}
@deffn {C Function} @ref{375,,PyObject}  *PyCodec_NameReplaceErrors (PyObject  *exc)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Replace the unicode encode error with @code{\N@{...@}} escapes.


New in version 3.5.

@end deffn

@node Abstract Objects Layer,Concrete Objects Layer,Utilities<2>,Python/C API Reference Manual
@anchor{c-api/abstract doc}@anchor{4327}@anchor{c-api/abstract abstract}@anchor{40ce}@anchor{c-api/abstract abstract-objects-layer}@anchor{4328}
@section Abstract Objects Layer


The functions in this chapter interact with Python objects regardless of their
type, or with wide classes of object types (e.g. all numerical types, or all
sequence types).  When used on object types for which they do not apply, they
will raise a Python exception.

It is not possible to use these functions on objects that are not properly
initialized, such as a list object that has been created by @ref{4114,,PyList_New()},
but whose items have not been set to some non-@code{NULL} value yet.

@menu
* Object Protocol:: 
* Call Protocol:: 
* Number Protocol:: 
* Sequence Protocol:: 
* Mapping Protocol:: 
* Iterator Protocol:: 
* Buffer Protocol:: 
* Old Buffer Protocol:: 

@end menu

@node Object Protocol,Call Protocol,,Abstract Objects Layer
@anchor{c-api/object doc}@anchor{4329}@anchor{c-api/object object}@anchor{432a}@anchor{c-api/object object-protocol}@anchor{432b}
@subsection Object Protocol


@geindex Py_NotImplemented (C var)
@anchor{c-api/object c Py_NotImplemented}@anchor{432c}
@deffn {C Variable} @ref{375,,PyObject}  *Py_NotImplemented

The @code{NotImplemented} singleton, used to signal that an operation is
not implemented for the given type combination.
@end deffn

@geindex Py_RETURN_NOTIMPLEMENTED (C macro)
@anchor{c-api/object c Py_RETURN_NOTIMPLEMENTED}@anchor{432d}
@deffn {C Macro} Py_RETURN_NOTIMPLEMENTED

Properly handle returning @ref{432c,,Py_NotImplemented} from within a C
function (that is, increment the reference count of NotImplemented and
return it).
@end deffn

@geindex PyObject_Print (C function)
@anchor{c-api/object c PyObject_Print}@anchor{130f}
@deffn {C Function} int  PyObject_Print (PyObject  *o, FILE  *fp, int  flags)

Print an object @emph{o}, on file @emph{fp}.  Returns @code{-1} on error.  The flags argument
is used to enable certain printing options.  The only option currently supported
is @code{Py_PRINT_RAW}; if given, the @ref{2ef,,str()} of the object is written
instead of the @ref{27d,,repr()}.
@end deffn

@geindex PyObject_HasAttr (C function)
@anchor{c-api/object c PyObject_HasAttr}@anchor{420a}
@deffn {C Function} int  PyObject_HasAttr (PyObject  *o, PyObject  *attr_name)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns @code{1} if @emph{o} has the attribute @emph{attr_name}, and @code{0} otherwise.  This
is equivalent to the Python expression @code{hasattr(o, attr_name)}.  This function
always succeeds.

Note that exceptions which occur while calling @code{__getattr__()} and
@code{__getattribute__()} methods will get suppressed.
To get error reporting use @ref{4208,,PyObject_GetAttr()} instead.
@end deffn

@geindex PyObject_HasAttrString (C function)
@anchor{c-api/object c PyObject_HasAttrString}@anchor{420b}
@deffn {C Function} int  PyObject_HasAttrString (PyObject  *o, const  char  *attr_name)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns @code{1} if @emph{o} has the attribute @emph{attr_name}, and @code{0} otherwise.  This
is equivalent to the Python expression @code{hasattr(o, attr_name)}.  This function
always succeeds.

Note that exceptions which occur while calling @code{__getattr__()} and
@code{__getattribute__()} methods and creating a temporary string object
will get suppressed.
To get error reporting use @ref{408c,,PyObject_GetAttrString()} instead.
@end deffn

@geindex PyObject_GetAttr (C function)
@anchor{c-api/object c PyObject_GetAttr}@anchor{4208}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_GetAttr (PyObject  *o, PyObject  *attr_name)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Retrieve an attribute named @emph{attr_name} from object @emph{o}. Returns the attribute
value on success, or @code{NULL} on failure.  This is the equivalent of the Python
expression @code{o.attr_name}.
@end deffn

@geindex PyObject_GetAttrString (C function)
@anchor{c-api/object c PyObject_GetAttrString}@anchor{408c}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_GetAttrString (PyObject  *o, const  char  *attr_name)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Retrieve an attribute named @emph{attr_name} from object @emph{o}. Returns the attribute
value on success, or @code{NULL} on failure. This is the equivalent of the Python
expression @code{o.attr_name}.
@end deffn

@geindex PyObject_GenericGetAttr (C function)
@anchor{c-api/object c PyObject_GenericGetAttr}@anchor{4204}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_GenericGetAttr (PyObject  *o, PyObject  *name)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Generic attribute getter function that is meant to be put into a type
object’s @code{tp_getattro} slot.  It looks for a descriptor in the dictionary
of classes in the object’s MRO as well as an attribute in the object’s
@ref{7a4,,__dict__} (if present).  As outlined in @ref{793,,Implementing Descriptors},
data descriptors take preference over instance attributes, while non-data
descriptors don’t.  Otherwise, an @ref{163,,AttributeError} is raised.
@end deffn

@geindex PyObject_SetAttr (C function)
@anchor{c-api/object c PyObject_SetAttr}@anchor{420f}
@deffn {C Function} int  PyObject_SetAttr (PyObject  *o, PyObject  *attr_name, PyObject  *v)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the value of the attribute named @emph{attr_name}, for object @emph{o}, to the value
@emph{v}. Raise an exception and return @code{-1} on failure;
return @code{0} on success.  This is the equivalent of the Python statement
@code{o.attr_name = v}.

If @emph{v} is @code{NULL}, the attribute is deleted. This behaviour is deprecated
in favour of using @ref{432e,,PyObject_DelAttr()}, but there are currently no
plans to remove it.
@end deffn

@geindex PyObject_SetAttrString (C function)
@anchor{c-api/object c PyObject_SetAttrString}@anchor{4210}
@deffn {C Function} int  PyObject_SetAttrString (PyObject  *o, const  char  *attr_name, PyObject  *v)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the value of the attribute named @emph{attr_name}, for object @emph{o}, to the value
@emph{v}. Raise an exception and return @code{-1} on failure;
return @code{0} on success.  This is the equivalent of the Python statement
@code{o.attr_name = v}.

If @emph{v} is @code{NULL}, the attribute is deleted, however this feature is
deprecated in favour of using @ref{432f,,PyObject_DelAttrString()}.
@end deffn

@geindex PyObject_GenericSetAttr (C function)
@anchor{c-api/object c PyObject_GenericSetAttr}@anchor{4205}
@deffn {C Function} int  PyObject_GenericSetAttr (PyObject  *o, PyObject  *name, PyObject  *value)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Generic attribute setter and deleter function that is meant
to be put into a type object’s @ref{40c6,,tp_setattro}
slot.  It looks for a data descriptor in the
dictionary of classes in the object’s MRO, and if found it takes preference
over setting or deleting the attribute in the instance dictionary. Otherwise, the
attribute is set or deleted in the object’s @ref{7a4,,__dict__} (if present).
On success, @code{0} is returned, otherwise an @ref{163,,AttributeError}
is raised and @code{-1} is returned.
@end deffn

@geindex PyObject_DelAttr (C function)
@anchor{c-api/object c PyObject_DelAttr}@anchor{432e}
@deffn {C Function} int  PyObject_DelAttr (PyObject  *o, PyObject  *attr_name)

Delete attribute named @emph{attr_name}, for object @emph{o}. Returns @code{-1} on failure.
This is the equivalent of the Python statement @code{del o.attr_name}.
@end deffn

@geindex PyObject_DelAttrString (C function)
@anchor{c-api/object c PyObject_DelAttrString}@anchor{432f}
@deffn {C Function} int  PyObject_DelAttrString (PyObject  *o, const  char  *attr_name)

Delete attribute named @emph{attr_name}, for object @emph{o}. Returns @code{-1} on failure.
This is the equivalent of the Python statement @code{del o.attr_name}.
@end deffn

@geindex PyObject_GenericGetDict (C function)
@anchor{c-api/object c PyObject_GenericGetDict}@anchor{11b7}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_GenericGetDict (PyObject  *o, void  *context)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
A generic implementation for the getter of a @code{__dict__} descriptor. It
creates the dictionary if necessary.


New in version 3.3.

@end deffn

@geindex PyObject_GenericSetDict (C function)
@anchor{c-api/object c PyObject_GenericSetDict}@anchor{4206}
@deffn {C Function} int  PyObject_GenericSetDict (PyObject  *o, PyObject  *value, void  *context)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
A generic implementation for the setter of a @code{__dict__} descriptor. This
implementation does not allow the dictionary to be deleted.


New in version 3.3.

@end deffn

@geindex PyObject_RichCompare (C function)
@anchor{c-api/object c PyObject_RichCompare}@anchor{40cc}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_RichCompare (PyObject  *o1, PyObject  *o2, int  opid)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Compare the values of @emph{o1} and @emph{o2} using the operation specified by @emph{opid},
which must be one of @code{Py_LT}, @code{Py_LE}, @code{Py_EQ},
@code{Py_NE}, @code{Py_GT}, or @code{Py_GE}, corresponding to @code{<},
@code{<=}, @code{==}, @code{!=}, @code{>}, or @code{>=} respectively. This is the equivalent of
the Python expression @code{o1 op o2}, where @code{op} is the operator corresponding
to @emph{opid}. Returns the value of the comparison on success, or @code{NULL} on failure.
@end deffn

@geindex PyObject_RichCompareBool (C function)
@anchor{c-api/object c PyObject_RichCompareBool}@anchor{1239}
@deffn {C Function} int  PyObject_RichCompareBool (PyObject  *o1, PyObject  *o2, int  opid)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Compare the values of @emph{o1} and @emph{o2} using the operation specified by @emph{opid},
which must be one of @code{Py_LT}, @code{Py_LE}, @code{Py_EQ},
@code{Py_NE}, @code{Py_GT}, or @code{Py_GE}, corresponding to @code{<},
@code{<=}, @code{==}, @code{!=}, @code{>}, or @code{>=} respectively. Returns @code{-1} on error,
@code{0} if the result is false, @code{1} otherwise. This is the equivalent of the
Python expression @code{o1 op o2}, where @code{op} is the operator corresponding to
@emph{opid}.
@end deffn

@cartouche
@quotation Note 
If @emph{o1} and @emph{o2} are the same object, @ref{1239,,PyObject_RichCompareBool()}
will always return @code{1} for @code{Py_EQ} and @code{0} for @code{Py_NE}.
@end quotation
@end cartouche

@geindex PyObject_Repr (C function)
@anchor{c-api/object c PyObject_Repr}@anchor{bc0}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_Repr (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; repr

Compute a string representation of object @emph{o}.  Returns the string
representation on success, @code{NULL} on failure.  This is the equivalent of the
Python expression @code{repr(o)}.  Called by the @ref{27d,,repr()} built-in function.


Changed in version 3.4: This function now includes a debug assertion to help ensure that it
does not silently discard an active exception.

@end deffn

@geindex PyObject_ASCII (C function)
@anchor{c-api/object c PyObject_ASCII}@anchor{41f9}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_ASCII (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; ascii

As @ref{bc0,,PyObject_Repr()}, compute a string representation of object @emph{o}, but
escape the non-ASCII characters in the string returned by
@ref{bc0,,PyObject_Repr()} with @code{\x}, @code{\u} or @code{\U} escapes.  This generates
a string similar to that returned by @ref{bc0,,PyObject_Repr()} in Python 2.
Called by the @ref{f47,,ascii()} built-in function.

@geindex string; PyObject_Str (C function)
@end deffn

@geindex PyObject_Str (C function)
@anchor{c-api/object c PyObject_Str}@anchor{bc1}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_Str (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Compute a string representation of object @emph{o}.  Returns the string
representation on success, @code{NULL} on failure.  This is the equivalent of the
Python expression @code{str(o)}.  Called by the @ref{2ef,,str()} built-in function
and, therefore, by the @ref{af6,,print()} function.


Changed in version 3.4: This function now includes a debug assertion to help ensure that it
does not silently discard an active exception.

@end deffn

@geindex PyObject_Bytes (C function)
@anchor{c-api/object c PyObject_Bytes}@anchor{41fe}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_Bytes (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; bytes

Compute a bytes representation of object @emph{o}.  @code{NULL} is returned on
failure and a bytes object on success.  This is equivalent to the Python
expression @code{bytes(o)}, when @emph{o} is not an integer.  Unlike @code{bytes(o)},
a TypeError is raised when @emph{o} is an integer instead of a zero-initialized
bytes object.
@end deffn

@geindex PyObject_IsSubclass (C function)
@anchor{c-api/object c PyObject_IsSubclass}@anchor{a18}
@deffn {C Function} int  PyObject_IsSubclass (PyObject  *derived, PyObject  *cls)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return @code{1} if the class @emph{derived} is identical to or derived from the class
@emph{cls}, otherwise return @code{0}.  In case of an error, return @code{-1}.

If @emph{cls} is a tuple, the check will be done against every entry in @emph{cls}.
The result will be @code{1} when at least one of the checks returns @code{1},
otherwise it will be @code{0}.

If @emph{cls} has a @ref{17c8,,__subclasscheck__()} method, it will be called to
determine the subclass status as described in 
@geindex Python Enhancement Proposals; PEP 3119
PEP 3119@footnote{https://www.python.org/dev/peps/pep-3119}.  Otherwise,
@emph{derived} is a subclass of @emph{cls} if it is a direct or indirect subclass,
i.e. contained in @code{cls.__mro__}.

Normally only class objects, i.e. instances of @ref{89e,,type} or a derived
class, are considered classes.  However, objects can override this by having
a @code{__bases__} attribute (which must be a tuple of base classes).
@end deffn

@geindex PyObject_IsInstance (C function)
@anchor{c-api/object c PyObject_IsInstance}@anchor{a17}
@deffn {C Function} int  PyObject_IsInstance (PyObject  *inst, PyObject  *cls)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return @code{1} if @emph{inst} is an instance of the class @emph{cls} or a subclass of
@emph{cls}, or @code{0} if not.  On error, returns @code{-1} and sets an exception.

If @emph{cls} is a tuple, the check will be done against every entry in @emph{cls}.
The result will be @code{1} when at least one of the checks returns @code{1},
otherwise it will be @code{0}.

If @emph{cls} has a @ref{17c7,,__instancecheck__()} method, it will be called to
determine the subclass status as described in 
@geindex Python Enhancement Proposals; PEP 3119
PEP 3119@footnote{https://www.python.org/dev/peps/pep-3119}.  Otherwise, @emph{inst}
is an instance of @emph{cls} if its class is a subclass of @emph{cls}.

An instance @emph{inst} can override what is considered its class by having a
@code{__class__} attribute.

An object @emph{cls} can override if it is considered a class, and what its base
classes are, by having a @code{__bases__} attribute (which must be a tuple
of base classes).
@end deffn

@geindex PyObject_Hash (C function)
@anchor{c-api/object c PyObject_Hash}@anchor{209}
@deffn {C Function} Py_hash_t  PyObject_Hash (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; hash

Compute and return the hash value of an object @emph{o}.  On failure, return @code{-1}.
This is the equivalent of the Python expression @code{hash(o)}.


Changed in version 3.2: The return type is now Py_hash_t.  This is a signed integer the same size
as Py_ssize_t.

@end deffn

@geindex PyObject_HashNotImplemented (C function)
@anchor{c-api/object c PyObject_HashNotImplemented}@anchor{f2d}
@deffn {C Function} Py_hash_t  PyObject_HashNotImplemented (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set a @ref{162,,TypeError} indicating that @code{type(o)} is not hashable and return @code{-1}.
This function receives special treatment when stored in a @code{tp_hash} slot,
allowing a type to explicitly indicate to the interpreter that it is not
hashable.
@end deffn

@geindex PyObject_IsTrue (C function)
@anchor{c-api/object c PyObject_IsTrue}@anchor{420c}
@deffn {C Function} int  PyObject_IsTrue (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns @code{1} if the object @emph{o} is considered to be true, and @code{0} otherwise.
This is equivalent to the Python expression @code{not not o}.  On failure, return
@code{-1}.
@end deffn

@geindex PyObject_Not (C function)
@anchor{c-api/object c PyObject_Not}@anchor{420e}
@deffn {C Function} int  PyObject_Not (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns @code{0} if the object @emph{o} is considered to be true, and @code{1} otherwise.
This is equivalent to the Python expression @code{not o}.  On failure, return
@code{-1}.
@end deffn

@geindex PyObject_Type (C function)
@anchor{c-api/object c PyObject_Type}@anchor{4212}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_Type (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; type

When @emph{o} is non-@code{NULL}, returns a type object corresponding to the object type
of object @emph{o}. On failure, raises @ref{891,,SystemError} and returns @code{NULL}.  This
is equivalent to the Python expression @code{type(o)}. This function increments the
reference count of the return value. There’s really no reason to use this
function instead of the common expression @code{o->ob_type}, which returns a
pointer of type @ref{5c6,,PyTypeObject}*, except when the incremented reference
count is needed.
@end deffn

@geindex PyObject_TypeCheck (C function)
@anchor{c-api/object c PyObject_TypeCheck}@anchor{116f}
@deffn {C Function} int  PyObject_TypeCheck (PyObject  *o, PyTypeObject  *type)

Return non-zero if the object @emph{o} is of type @emph{type} or a subtype of @emph{type}, and
@code{0} otherwise.  Both parameters must be non-@code{NULL}.
@end deffn

@geindex PyObject_Size (C function)
@geindex PyObject_Length (C function)
@anchor{c-api/object c PyObject_Size}@anchor{4211}
@deffn {C Function} Py_ssize_t  PyObject_Size (PyObject  *o)
@anchor{c-api/object c PyObject_Length}@anchor{420d}
@deffnx {C Function} Py_ssize_t  PyObject_Length (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; len

Return the length of object @emph{o}.  If the object @emph{o} provides either the sequence
and mapping protocols, the sequence length is returned.  On error, @code{-1} is
returned.  This is the equivalent to the Python expression @code{len(o)}.
@end deffn

@geindex PyObject_LengthHint (C function)
@anchor{c-api/object c PyObject_LengthHint}@anchor{b8a}
@deffn {C Function} Py_ssize_t  PyObject_LengthHint (PyObject  *o, Py_ssize_t  defaultvalue)

Return an estimated length for the object @emph{o}. First try to return its
actual length, then an estimate using @ref{a80,,__length_hint__()}, and
finally return the default value. On error return @code{-1}. This is the
equivalent to the Python expression @code{operator.length_hint(o, defaultvalue)}.


New in version 3.4.

@end deffn

@geindex PyObject_GetItem (C function)
@anchor{c-api/object c PyObject_GetItem}@anchor{4115}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_GetItem (PyObject  *o, PyObject  *key)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return element of @emph{o} corresponding to the object @emph{key} or @code{NULL} on failure.
This is the equivalent of the Python expression @code{o[key]}.
@end deffn

@geindex PyObject_SetItem (C function)
@anchor{c-api/object c PyObject_SetItem}@anchor{4113}
@deffn {C Function} int  PyObject_SetItem (PyObject  *o, PyObject  *key, PyObject  *v)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Map the object @emph{key} to the value @emph{v}.  Raise an exception and
return @code{-1} on failure; return @code{0} on success.  This is the
equivalent of the Python statement @code{o[key] = v}.  This function @emph{does
not} steal a reference to @emph{v}.
@end deffn

@geindex PyObject_DelItem (C function)
@anchor{c-api/object c PyObject_DelItem}@anchor{4202}
@deffn {C Function} int  PyObject_DelItem (PyObject  *o, PyObject  *key)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Remove the mapping for the object @emph{key} from the object @emph{o}.  Return @code{-1}
on failure.  This is equivalent to the Python statement @code{del o[key]}.
@end deffn

@geindex PyObject_Dir (C function)
@anchor{c-api/object c PyObject_Dir}@anchor{4203}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_Dir (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
This is equivalent to the Python expression @code{dir(o)}, returning a (possibly
empty) list of strings appropriate for the object argument, or @code{NULL} if there
was an error.  If the argument is @code{NULL}, this is like the Python @code{dir()},
returning the names of the current locals; in this case, if no execution frame
is active then @code{NULL} is returned but @ref{116a,,PyErr_Occurred()} will return false.
@end deffn

@geindex PyObject_GetIter (C function)
@anchor{c-api/object c PyObject_GetIter}@anchor{4209}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_GetIter (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
This is equivalent to the Python expression @code{iter(o)}. It returns a new
iterator for the object argument, or the object  itself if the object is already
an iterator.  Raises @ref{162,,TypeError} and returns @code{NULL} if the object cannot be
iterated.
@end deffn

@geindex PyObject_GetAIter (C function)
@anchor{c-api/object c PyObject_GetAIter}@anchor{4207}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_GetAIter (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
This is the equivalent to the Python expression @code{aiter(o)}. Takes an
@code{AsyncIterable} object and returns an @code{AsyncIterator} for it.
This is typically a new iterator but if the argument is an
@code{AsyncIterator}, this returns itself. Raises @ref{162,,TypeError} and
returns @code{NULL} if the object cannot be iterated.


New in version 3.10.

@end deffn

@node Call Protocol,Number Protocol,Object Protocol,Abstract Objects Layer
@anchor{c-api/call doc}@anchor{4330}@anchor{c-api/call call}@anchor{42ee}@anchor{c-api/call call-protocol}@anchor{4331}
@subsection Call Protocol


CPython supports two different calling protocols:
@emph{tp_call} and vectorcall.

@menu
* The tp_call Protocol:: 
* The Vectorcall Protocol:: 
* Object Calling API:: 
* Call Support API:: 

@end menu

@node The tp_call Protocol,The Vectorcall Protocol,,Call Protocol
@anchor{c-api/call the-tp-call-protocol}@anchor{4332}
@subsubsection The @emph{tp_call} Protocol


Instances of classes that set @ref{40d2,,tp_call} are callable.
The signature of the slot is:

@example
PyObject *tp_call(PyObject *callable, PyObject *args, PyObject *kwargs);
@end example

A call is made using a tuple for the positional arguments
and a dict for the keyword arguments, similarly to
@code{callable(*args, **kwargs)} in Python code.
@emph{args} must be non-NULL (use an empty tuple if there are no arguments)
but @emph{kwargs} may be @emph{NULL} if there are no keyword arguments.

This convention is not only used by @emph{tp_call}:
@ref{40a7,,tp_new} and @ref{40ad,,tp_init}
also pass arguments this way.

To call an object, use @ref{46a,,PyObject_Call()} or other
@ref{4333,,call API}.

@node The Vectorcall Protocol,Object Calling API,The tp_call Protocol,Call Protocol
@anchor{c-api/call the-vectorcall-protocol}@anchor{4334}@anchor{c-api/call vectorcall}@anchor{4bd}
@subsubsection The Vectorcall Protocol



New in version 3.9.


The vectorcall protocol was introduced in 
@geindex Python Enhancement Proposals; PEP 590
PEP 590@footnote{https://www.python.org/dev/peps/pep-0590} as an additional protocol
for making calls more efficient.

As rule of thumb, CPython will prefer the vectorcall for internal calls
if the callable supports it. However, this is not a hard rule.
Additionally, some third-party extensions use @emph{tp_call} directly
(rather than using @ref{46a,,PyObject_Call()}).
Therefore, a class supporting vectorcall must also implement
@ref{40d2,,tp_call}.
Moreover, the callable must behave the same
regardless of which protocol is used.
The recommended way to achieve this is by setting
@ref{40d2,,tp_call} to @ref{4335,,PyVectorcall_Call()}.
This bears repeating:

@cartouche
@quotation Warning 
A class supporting vectorcall @strong{must} also implement
@ref{40d2,,tp_call} with the same semantics.
@end quotation
@end cartouche

A class should not implement vectorcall if that would be slower
than @emph{tp_call}. For example, if the callee needs to convert
the arguments to an args tuple and kwargs dict anyway, then there is no point
in implementing vectorcall.

Classes can implement the vectorcall protocol by enabling the
@ref{4336,,Py_TPFLAGS_HAVE_VECTORCALL} flag and setting
@ref{4337,,tp_vectorcall_offset} to the offset inside the
object structure where a @emph{vectorcallfunc} appears.
This is a pointer to a function with the following signature:

@geindex vectorcallfunc (C type)
@anchor{c-api/call c vectorcallfunc}@anchor{4338}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*vectorcallfunc)(@ref{375,,PyObject}  *callable,  @ref{375,,PyObject}  *const  *args,  size_t  nargsf,  @ref{375,,PyObject}  *kwnames)
@end deffn


@itemize -

@item 
@emph{callable} is the object being called.

@item 

@table @asis

@item @emph{args} is a C array consisting of the positional arguments followed by the

values of the keyword arguments.
This can be @emph{NULL} if there are no arguments.
@end table

@item 

@table @asis

@item @emph{nargsf} is the number of positional arguments plus possibly the

@code{PY_VECTORCALL_ARGUMENTS_OFFSET} flag.
To get the actual number of positional arguments from @emph{nargsf},
use @ref{4339,,PyVectorcall_NARGS()}.
@end table

@item 

@table @asis

@item @emph{kwnames} is a tuple containing the names of the keyword arguments;

in other words, the keys of the kwargs dict.
These names must be strings (instances of @code{str} or a subclass)
and they must be unique.
If there are no keyword arguments, then @emph{kwnames} can instead be @emph{NULL}.
@end table
@end itemize

@geindex PY_VECTORCALL_ARGUMENTS_OFFSET (C macro)
@anchor{c-api/call c PY_VECTORCALL_ARGUMENTS_OFFSET}@anchor{433a}
@deffn {C Macro} PY_VECTORCALL_ARGUMENTS_OFFSET

If this flag is set in a vectorcall @emph{nargsf} argument, the callee is allowed
to temporarily change @code{args[-1]}. In other words, @emph{args} points to
argument 1 (not 0) in the allocated vector.
The callee must restore the value of @code{args[-1]} before returning.

For @ref{433b,,PyObject_VectorcallMethod()}, this flag means instead that
@code{args[0]} may be changed.

Whenever they can do so cheaply (without additional allocation), callers
are encouraged to use @code{PY_VECTORCALL_ARGUMENTS_OFFSET}.
Doing so will allow callables such as bound methods to make their onward
calls (which include a prepended @emph{self} argument) very efficiently.
@end deffn

To call an object that implements vectorcall, use a @ref{4333,,call API}
function as with any other callable.
@ref{1084,,PyObject_Vectorcall()} will usually be most efficient.

@cartouche
@quotation Note 
In CPython 3.8, the vectorcall API and related functions were available
provisionally under names with a leading underscore:
@code{_PyObject_Vectorcall}, @code{_Py_TPFLAGS_HAVE_VECTORCALL},
@code{_PyObject_VectorcallMethod}, @code{_PyVectorcall_Function},
@code{_PyObject_CallOneArg}, @code{_PyObject_CallMethodNoArgs},
@code{_PyObject_CallMethodOneArg}.
Additionally, @code{PyObject_VectorcallDict} was available as
@code{_PyObject_FastCallDict}.
The old names are still defined as aliases of the new, non-underscored names.
@end quotation
@end cartouche

@menu
* Recursion Control: Recursion Control<2>. 
* Vectorcall Support API:: 

@end menu

@node Recursion Control<2>,Vectorcall Support API,,The Vectorcall Protocol
@anchor{c-api/call recursion-control}@anchor{433c}
@subsubsection Recursion Control


When using @emph{tp_call}, callees do not need to worry about
@ref{42ec,,recursion}: CPython uses
@ref{47a,,Py_EnterRecursiveCall()} and @ref{47b,,Py_LeaveRecursiveCall()}
for calls made using @emph{tp_call}.

For efficiency, this is not the case for calls done using vectorcall:
the callee should use @emph{Py_EnterRecursiveCall} and @emph{Py_LeaveRecursiveCall}
if needed.

@node Vectorcall Support API,,Recursion Control<2>,The Vectorcall Protocol
@anchor{c-api/call vectorcall-support-api}@anchor{433d}
@subsubsection Vectorcall Support API


@geindex PyVectorcall_NARGS (C function)
@anchor{c-api/call c PyVectorcall_NARGS}@anchor{4339}
@deffn {C Function} Py_ssize_t  PyVectorcall_NARGS (size_t  nargsf)

Given a vectorcall @emph{nargsf} argument, return the actual number of
arguments.
Currently equivalent to:

@example
(Py_ssize_t)(nargsf & ~PY_VECTORCALL_ARGUMENTS_OFFSET)
@end example

However, the function @code{PyVectorcall_NARGS} should be used to allow
for future extensions.

This function is not part of the @ref{35d,,limited API}.


New in version 3.8.

@end deffn

@geindex PyVectorcall_Function (C function)
@anchor{c-api/call c PyVectorcall_Function}@anchor{433e}
@deffn {C Function} @ref{4338,,vectorcallfunc}  PyVectorcall_Function (PyObject  *op)

If @emph{op} does not support the vectorcall protocol (either because the type
does not or because the specific instance does not), return @emph{NULL}.
Otherwise, return the vectorcall function pointer stored in @emph{op}.
This function never raises an exception.

This is mostly useful to check whether or not @emph{op} supports vectorcall,
which can be done by checking @code{PyVectorcall_Function(op) != NULL}.

This function is not part of the @ref{35d,,limited API}.


New in version 3.8.

@end deffn

@geindex PyVectorcall_Call (C function)
@anchor{c-api/call c PyVectorcall_Call}@anchor{4335}
@deffn {C Function} @ref{375,,PyObject}  *PyVectorcall_Call (PyObject  *callable, PyObject  *tuple, PyObject  *dict)

Call @emph{callable}’s @ref{4338,,vectorcallfunc} with positional and keyword
arguments given in a tuple and dict, respectively.

This is a specialized function, intended to be put in the
@ref{40d2,,tp_call} slot or be used in an implementation of @code{tp_call}.
It does not check the @ref{4336,,Py_TPFLAGS_HAVE_VECTORCALL} flag
and it does not fall back to @code{tp_call}.

This function is not part of the @ref{35d,,limited API}.


New in version 3.8.

@end deffn

@node Object Calling API,Call Support API,The Vectorcall Protocol,Call Protocol
@anchor{c-api/call capi-call}@anchor{4333}@anchor{c-api/call object-calling-api}@anchor{433f}
@subsubsection Object Calling API


Various functions are available for calling a Python object.
Each converts its arguments to a convention supported by the called object –
either @emph{tp_call} or vectorcall.
In order to do as little conversion as possible, pick one that best fits
the format of data you have available.

The following table summarizes the available functions;
please see individual documentation for details.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxx} 
@headitem

Function

@tab

callable

@tab

args

@tab

kwargs

@item

@ref{46a,,PyObject_Call()}

@tab

@code{PyObject *}

@tab

tuple

@tab

dict/@code{NULL}

@item

@ref{479,,PyObject_CallNoArgs()}

@tab

@code{PyObject *}

@tab

—

@tab

—

@item

@ref{47f,,PyObject_CallOneArg()}

@tab

@code{PyObject *}

@tab

1 object

@tab

—

@item

@ref{4081,,PyObject_CallObject()}

@tab

@code{PyObject *}

@tab

tuple/@code{NULL}

@tab

—

@item

@ref{5c2,,PyObject_CallFunction()}

@tab

@code{PyObject *}

@tab

format

@tab

—

@item

@ref{350,,PyObject_CallMethod()}

@tab

obj + @code{char*}

@tab

format

@tab

—

@item

@ref{41ff,,PyObject_CallFunctionObjArgs()}

@tab

@code{PyObject *}

@tab

variadic

@tab

—

@item

@ref{12a7,,PyObject_CallMethodObjArgs()}

@tab

obj + name

@tab

variadic

@tab

—

@item

@ref{4340,,PyObject_CallMethodNoArgs()}

@tab

obj + name

@tab

—

@tab

—

@item

@ref{4341,,PyObject_CallMethodOneArg()}

@tab

obj + name

@tab

1 object

@tab

—

@item

@ref{1084,,PyObject_Vectorcall()}

@tab

@code{PyObject *}

@tab

vectorcall

@tab

vectorcall

@item

@ref{4342,,PyObject_VectorcallDict()}

@tab

@code{PyObject *}

@tab

vectorcall

@tab

dict/@code{NULL}

@item

@ref{433b,,PyObject_VectorcallMethod()}

@tab

arg + name

@tab

vectorcall

@tab

vectorcall

@end multitable


@geindex PyObject_Call (C function)
@anchor{c-api/call c PyObject_Call}@anchor{46a}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_Call (PyObject  *callable, PyObject  *args, PyObject  *kwargs)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Call a callable Python object @emph{callable}, with arguments given by the
tuple @emph{args}, and named arguments given by the dictionary @emph{kwargs}.

@emph{args} must not be @emph{NULL}; use an empty tuple if no arguments are needed.
If no named arguments are needed, @emph{kwargs} can be @emph{NULL}.

Return the result of the call on success, or raise an exception and return
@emph{NULL} on failure.

This is the equivalent of the Python expression:
@code{callable(*args, **kwargs)}.
@end deffn

@geindex PyObject_CallNoArgs (C function)
@anchor{c-api/call c PyObject_CallNoArgs}@anchor{479}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_CallNoArgs (PyObject  *callable)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Call a callable Python object @emph{callable} without any arguments. It is the
most efficient way to call a callable Python object without any argument.

Return the result of the call on success, or raise an exception and return
@emph{NULL} on failure.


New in version 3.9.

@end deffn

@geindex PyObject_CallOneArg (C function)
@anchor{c-api/call c PyObject_CallOneArg}@anchor{47f}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_CallOneArg (PyObject  *callable, PyObject  *arg)

Call a callable Python object @emph{callable} with exactly 1 positional argument
@emph{arg} and no keyword arguments.

Return the result of the call on success, or raise an exception and return
@emph{NULL} on failure.

This function is not part of the @ref{35d,,limited API}.


New in version 3.9.

@end deffn

@geindex PyObject_CallObject (C function)
@anchor{c-api/call c PyObject_CallObject}@anchor{4081}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_CallObject (PyObject  *callable, PyObject  *args)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Call a callable Python object @emph{callable}, with arguments given by the
tuple @emph{args}.  If no arguments are needed, then @emph{args} can be @emph{NULL}.

Return the result of the call on success, or raise an exception and return
@emph{NULL} on failure.

This is the equivalent of the Python expression: @code{callable(*args)}.
@end deffn

@geindex PyObject_CallFunction (C function)
@anchor{c-api/call c PyObject_CallFunction}@anchor{5c2}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_CallFunction (PyObject  *callable, const  char  *format, ...)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Call a callable Python object @emph{callable}, with a variable number of C arguments.
The C arguments are described using a @ref{380,,Py_BuildValue()} style format
string.  The format can be @emph{NULL}, indicating that no arguments are provided.

Return the result of the call on success, or raise an exception and return
@emph{NULL} on failure.

This is the equivalent of the Python expression: @code{callable(*args)}.

Note that if you only pass @ref{375,,PyObject}* args,
@ref{41ff,,PyObject_CallFunctionObjArgs()} is a faster alternative.


Changed in version 3.4: The type of @emph{format} was changed from @code{char *}.

@end deffn

@geindex PyObject_CallMethod (C function)
@anchor{c-api/call c PyObject_CallMethod}@anchor{350}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_CallMethod (PyObject  *obj, const  char  *name, const  char  *format, ...)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Call the method named @emph{name} of object @emph{obj} with a variable number of C
arguments.  The C arguments are described by a @ref{380,,Py_BuildValue()} format
string that should produce a tuple.

The format can be @emph{NULL}, indicating that no arguments are provided.

Return the result of the call on success, or raise an exception and return
@emph{NULL} on failure.

This is the equivalent of the Python expression:
@code{obj.name(arg1, arg2, ...)}.

Note that if you only pass @ref{375,,PyObject}* args,
@ref{12a7,,PyObject_CallMethodObjArgs()} is a faster alternative.


Changed in version 3.4: The types of @emph{name} and @emph{format} were changed from @code{char *}.

@end deffn

@geindex PyObject_CallFunctionObjArgs (C function)
@anchor{c-api/call c PyObject_CallFunctionObjArgs}@anchor{41ff}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_CallFunctionObjArgs (PyObject  *callable, ...)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Call a callable Python object @emph{callable}, with a variable number of
@ref{375,,PyObject}* arguments.  The arguments are provided as a variable number
of parameters followed by @emph{NULL}.

Return the result of the call on success, or raise an exception and return
@emph{NULL} on failure.

This is the equivalent of the Python expression:
@code{callable(arg1, arg2, ...)}.
@end deffn

@geindex PyObject_CallMethodObjArgs (C function)
@anchor{c-api/call c PyObject_CallMethodObjArgs}@anchor{12a7}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_CallMethodObjArgs (PyObject  *obj, PyObject  *name, ...)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Call a method of the Python object @emph{obj}, where the name of the method is given as a
Python string object in @emph{name}.  It is called with a variable number of
@ref{375,,PyObject}* arguments.  The arguments are provided as a variable number
of parameters followed by @emph{NULL}.

Return the result of the call on success, or raise an exception and return
@emph{NULL} on failure.
@end deffn

@geindex PyObject_CallMethodNoArgs (C function)
@anchor{c-api/call c PyObject_CallMethodNoArgs}@anchor{4340}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_CallMethodNoArgs (PyObject  *obj, PyObject  *name)

Call a method of the Python object @emph{obj} without arguments,
where the name of the method is given as a Python string object in @emph{name}.

Return the result of the call on success, or raise an exception and return
@emph{NULL} on failure.

This function is not part of the @ref{35d,,limited API}.


New in version 3.9.

@end deffn

@geindex PyObject_CallMethodOneArg (C function)
@anchor{c-api/call c PyObject_CallMethodOneArg}@anchor{4341}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_CallMethodOneArg (PyObject  *obj, PyObject  *name, PyObject  *arg)

Call a method of the Python object @emph{obj} with a single positional argument
@emph{arg}, where the name of the method is given as a Python string object in
@emph{name}.

Return the result of the call on success, or raise an exception and return
@emph{NULL} on failure.

This function is not part of the @ref{35d,,limited API}.


New in version 3.9.

@end deffn

@geindex PyObject_Vectorcall (C function)
@anchor{c-api/call c PyObject_Vectorcall}@anchor{1084}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_Vectorcall (PyObject  *callable, PyObject  *const  *args, size_t  nargsf, PyObject  *kwnames)

Call a callable Python object @emph{callable}.
The arguments are the same as for @ref{4338,,vectorcallfunc}.
If @emph{callable} supports @ref{4bd,,vectorcall}, this directly calls
the vectorcall function stored in @emph{callable}.

Return the result of the call on success, or raise an exception and return
@emph{NULL} on failure.

This function is not part of the @ref{35d,,limited API}.


New in version 3.9.

@end deffn

@geindex PyObject_VectorcallDict (C function)
@anchor{c-api/call c PyObject_VectorcallDict}@anchor{4342}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_VectorcallDict (PyObject  *callable, PyObject  *const  *args, size_t  nargsf, PyObject  *kwdict)

Call @emph{callable} with positional arguments passed exactly as in the @ref{4bd,,vectorcall} protocol,
but with keyword arguments passed as a dictionary @emph{kwdict}.
The @emph{args} array contains only the positional arguments.

Regardless of which protocol is used internally,
a conversion of arguments needs to be done.
Therefore, this function should only be used if the caller
already has a dictionary ready to use for the keyword arguments,
but not a tuple for the positional arguments.

This function is not part of the @ref{35d,,limited API}.


New in version 3.9.

@end deffn

@geindex PyObject_VectorcallMethod (C function)
@anchor{c-api/call c PyObject_VectorcallMethod}@anchor{433b}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_VectorcallMethod (PyObject  *name, PyObject  *const  *args, size_t  nargsf, PyObject  *kwnames)

Call a method using the vectorcall calling convention. The name of the method
is given as a Python string @emph{name}. The object whose method is called is
@emph{args[0]}, and the @emph{args} array starting at @emph{args[1]} represents the arguments
of the call. There must be at least one positional argument.
@emph{nargsf} is the number of positional arguments including @emph{args[0]},
plus @code{PY_VECTORCALL_ARGUMENTS_OFFSET} if the value of @code{args[0]} may
temporarily be changed. Keyword arguments can be passed just like in
@ref{1084,,PyObject_Vectorcall()}.

If the object has the @ref{4343,,Py_TPFLAGS_METHOD_DESCRIPTOR} feature,
this will call the unbound method object with the full
@emph{args} vector as arguments.

Return the result of the call on success, or raise an exception and return
@emph{NULL} on failure.

This function is not part of the @ref{35d,,limited API}.


New in version 3.9.

@end deffn

@node Call Support API,,Object Calling API,Call Protocol
@anchor{c-api/call call-support-api}@anchor{4344}
@subsubsection Call Support API


@geindex PyCallable_Check (C function)
@anchor{c-api/call c PyCallable_Check}@anchor{413d}
@deffn {C Function} int  PyCallable_Check (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Determine if the object @emph{o} is callable.  Return @code{1} if the object is callable
and @code{0} otherwise.  This function always succeeds.
@end deffn

@node Number Protocol,Sequence Protocol,Call Protocol,Abstract Objects Layer
@anchor{c-api/number doc}@anchor{4345}@anchor{c-api/number number}@anchor{4346}@anchor{c-api/number number-protocol}@anchor{4347}
@subsection Number Protocol


@geindex PyNumber_Check (C function)
@anchor{c-api/number c PyNumber_Check}@anchor{57d}
@deffn {C Function} int  PyNumber_Check (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns @code{1} if the object @emph{o} provides numeric protocols, and false otherwise.
This function always succeeds.


Changed in version 3.8: Returns @code{1} if @emph{o} is an index integer.

@end deffn

@geindex PyNumber_Add (C function)
@anchor{c-api/number c PyNumber_Add}@anchor{41d7}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Add (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the result of adding @emph{o1} and @emph{o2}, or @code{NULL} on failure.  This is the
equivalent of the Python expression @code{o1 + o2}.
@end deffn

@geindex PyNumber_Subtract (C function)
@anchor{c-api/number c PyNumber_Subtract}@anchor{41f1}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Subtract (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the result of subtracting @emph{o2} from @emph{o1}, or @code{NULL} on failure.  This is
the equivalent of the Python expression @code{o1 - o2}.
@end deffn

@geindex PyNumber_Multiply (C function)
@anchor{c-api/number c PyNumber_Multiply}@anchor{41ea}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Multiply (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the result of multiplying @emph{o1} and @emph{o2}, or @code{NULL} on failure.  This is
the equivalent of the Python expression @code{o1 * o2}.
@end deffn

@geindex PyNumber_MatrixMultiply (C function)
@anchor{c-api/number c PyNumber_MatrixMultiply}@anchor{a27}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_MatrixMultiply (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Returns the result of matrix multiplication on @emph{o1} and @emph{o2}, or @code{NULL} on
failure.  This is the equivalent of the Python expression @code{o1 @@ o2}.


New in version 3.5.

@end deffn

@geindex PyNumber_FloorDivide (C function)
@anchor{c-api/number c PyNumber_FloorDivide}@anchor{41db}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_FloorDivide (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the floor of @emph{o1} divided by @emph{o2}, or @code{NULL} on failure.  This is
equivalent to the “classic” division of integers.
@end deffn

@geindex PyNumber_TrueDivide (C function)
@anchor{c-api/number c PyNumber_TrueDivide}@anchor{41f2}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_TrueDivide (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a reasonable approximation for the mathematical value of @emph{o1} divided by
@emph{o2}, or @code{NULL} on failure.  The return value is “approximate” because binary
floating point numbers are approximate; it is not possible to represent all real
numbers in base two.  This function can return a floating point value when
passed two integers.
@end deffn

@geindex PyNumber_Remainder (C function)
@anchor{c-api/number c PyNumber_Remainder}@anchor{41ef}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Remainder (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the remainder of dividing @emph{o1} by @emph{o2}, or @code{NULL} on failure.  This is
the equivalent of the Python expression @code{o1 % o2}.
@end deffn

@geindex PyNumber_Divmod (C function)
@anchor{c-api/number c PyNumber_Divmod}@anchor{41da}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Divmod (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; divmod

See the built-in function @ref{4a8,,divmod()}. Returns @code{NULL} on failure.  This is
the equivalent of the Python expression @code{divmod(o1, o2)}.
@end deffn

@geindex PyNumber_Power (C function)
@anchor{c-api/number c PyNumber_Power}@anchor{41ee}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Power (PyObject  *o1, PyObject  *o2, PyObject  *o3)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; pow

See the built-in function @ref{4cd,,pow()}. Returns @code{NULL} on failure.  This is the
equivalent of the Python expression @code{pow(o1, o2, o3)}, where @emph{o3} is optional.
If @emph{o3} is to be ignored, pass @ref{4077,,Py_None} in its place (passing @code{NULL} for
@emph{o3} would cause an illegal memory access).
@end deffn

@geindex PyNumber_Negative (C function)
@anchor{c-api/number c PyNumber_Negative}@anchor{41eb}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Negative (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the negation of @emph{o} on success, or @code{NULL} on failure. This is the
equivalent of the Python expression @code{-o}.
@end deffn

@geindex PyNumber_Positive (C function)
@anchor{c-api/number c PyNumber_Positive}@anchor{41ed}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Positive (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns @emph{o} on success, or @code{NULL} on failure.  This is the equivalent of the
Python expression @code{+o}.
@end deffn

@geindex PyNumber_Absolute (C function)
@anchor{c-api/number c PyNumber_Absolute}@anchor{41d6}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Absolute (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; abs

Returns the absolute value of @emph{o}, or @code{NULL} on failure.  This is the equivalent
of the Python expression @code{abs(o)}.
@end deffn

@geindex PyNumber_Invert (C function)
@anchor{c-api/number c PyNumber_Invert}@anchor{41e8}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Invert (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the bitwise negation of @emph{o} on success, or @code{NULL} on failure.  This is
the equivalent of the Python expression @code{~o}.
@end deffn

@geindex PyNumber_Lshift (C function)
@anchor{c-api/number c PyNumber_Lshift}@anchor{41e9}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Lshift (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the result of left shifting @emph{o1} by @emph{o2} on success, or @code{NULL} on
failure.  This is the equivalent of the Python expression @code{o1 << o2}.
@end deffn

@geindex PyNumber_Rshift (C function)
@anchor{c-api/number c PyNumber_Rshift}@anchor{41f0}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Rshift (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the result of right shifting @emph{o1} by @emph{o2} on success, or @code{NULL} on
failure.  This is the equivalent of the Python expression @code{o1 >> o2}.
@end deffn

@geindex PyNumber_And (C function)
@anchor{c-api/number c PyNumber_And}@anchor{41d8}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_And (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the “bitwise and” of @emph{o1} and @emph{o2} on success and @code{NULL} on failure.
This is the equivalent of the Python expression @code{o1 & o2}.
@end deffn

@geindex PyNumber_Xor (C function)
@anchor{c-api/number c PyNumber_Xor}@anchor{41f3}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Xor (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the “bitwise exclusive or” of @emph{o1} by @emph{o2} on success, or @code{NULL} on
failure.  This is the equivalent of the Python expression @code{o1 ^ o2}.
@end deffn

@geindex PyNumber_Or (C function)
@anchor{c-api/number c PyNumber_Or}@anchor{41ec}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Or (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the “bitwise or” of @emph{o1} and @emph{o2} on success, or @code{NULL} on failure.
This is the equivalent of the Python expression @code{o1 | o2}.
@end deffn

@geindex PyNumber_InPlaceAdd (C function)
@anchor{c-api/number c PyNumber_InPlaceAdd}@anchor{41dc}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_InPlaceAdd (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the result of adding @emph{o1} and @emph{o2}, or @code{NULL} on failure.  The operation
is done @emph{in-place} when @emph{o1} supports it.  This is the equivalent of the Python
statement @code{o1 += o2}.
@end deffn

@geindex PyNumber_InPlaceSubtract (C function)
@anchor{c-api/number c PyNumber_InPlaceSubtract}@anchor{41e5}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_InPlaceSubtract (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the result of subtracting @emph{o2} from @emph{o1}, or @code{NULL} on failure.  The
operation is done @emph{in-place} when @emph{o1} supports it.  This is the equivalent of
the Python statement @code{o1 -= o2}.
@end deffn

@geindex PyNumber_InPlaceMultiply (C function)
@anchor{c-api/number c PyNumber_InPlaceMultiply}@anchor{41e0}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_InPlaceMultiply (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the result of multiplying @emph{o1} and @emph{o2}, or @code{NULL} on failure.  The
operation is done @emph{in-place} when @emph{o1} supports it.  This is the equivalent of
the Python statement @code{o1 *= o2}.
@end deffn

@geindex PyNumber_InPlaceMatrixMultiply (C function)
@anchor{c-api/number c PyNumber_InPlaceMatrixMultiply}@anchor{a28}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_InPlaceMatrixMultiply (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Returns the result of matrix multiplication on @emph{o1} and @emph{o2}, or @code{NULL} on
failure.  The operation is done @emph{in-place} when @emph{o1} supports it.  This is
the equivalent of the Python statement @code{o1 @@= o2}.


New in version 3.5.

@end deffn

@geindex PyNumber_InPlaceFloorDivide (C function)
@anchor{c-api/number c PyNumber_InPlaceFloorDivide}@anchor{41de}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_InPlaceFloorDivide (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the mathematical floor of dividing @emph{o1} by @emph{o2}, or @code{NULL} on failure.
The operation is done @emph{in-place} when @emph{o1} supports it.  This is the equivalent
of the Python statement @code{o1 //= o2}.
@end deffn

@geindex PyNumber_InPlaceTrueDivide (C function)
@anchor{c-api/number c PyNumber_InPlaceTrueDivide}@anchor{41e6}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_InPlaceTrueDivide (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a reasonable approximation for the mathematical value of @emph{o1} divided by
@emph{o2}, or @code{NULL} on failure.  The return value is “approximate” because binary
floating point numbers are approximate; it is not possible to represent all real
numbers in base two.  This function can return a floating point value when
passed two integers.  The operation is done @emph{in-place} when @emph{o1} supports it.
@end deffn

@geindex PyNumber_InPlaceRemainder (C function)
@anchor{c-api/number c PyNumber_InPlaceRemainder}@anchor{41e3}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_InPlaceRemainder (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the remainder of dividing @emph{o1} by @emph{o2}, or @code{NULL} on failure.  The
operation is done @emph{in-place} when @emph{o1} supports it.  This is the equivalent of
the Python statement @code{o1 %= o2}.
@end deffn

@geindex PyNumber_InPlacePower (C function)
@anchor{c-api/number c PyNumber_InPlacePower}@anchor{41e2}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_InPlacePower (PyObject  *o1, PyObject  *o2, PyObject  *o3)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; pow

See the built-in function @ref{4cd,,pow()}. Returns @code{NULL} on failure.  The operation
is done @emph{in-place} when @emph{o1} supports it.  This is the equivalent of the Python
statement @code{o1 **= o2} when o3 is @ref{4077,,Py_None}, or an in-place variant of
@code{pow(o1, o2, o3)} otherwise. If @emph{o3} is to be ignored, pass @ref{4077,,Py_None}
in its place (passing @code{NULL} for @emph{o3} would cause an illegal memory access).
@end deffn

@geindex PyNumber_InPlaceLshift (C function)
@anchor{c-api/number c PyNumber_InPlaceLshift}@anchor{41df}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_InPlaceLshift (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the result of left shifting @emph{o1} by @emph{o2} on success, or @code{NULL} on
failure.  The operation is done @emph{in-place} when @emph{o1} supports it.  This is the
equivalent of the Python statement @code{o1 <<= o2}.
@end deffn

@geindex PyNumber_InPlaceRshift (C function)
@anchor{c-api/number c PyNumber_InPlaceRshift}@anchor{41e4}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_InPlaceRshift (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the result of right shifting @emph{o1} by @emph{o2} on success, or @code{NULL} on
failure.  The operation is done @emph{in-place} when @emph{o1} supports it.  This is the
equivalent of the Python statement @code{o1 >>= o2}.
@end deffn

@geindex PyNumber_InPlaceAnd (C function)
@anchor{c-api/number c PyNumber_InPlaceAnd}@anchor{41dd}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_InPlaceAnd (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the “bitwise and” of @emph{o1} and @emph{o2} on success and @code{NULL} on failure. The
operation is done @emph{in-place} when @emph{o1} supports it.  This is the equivalent of
the Python statement @code{o1 &= o2}.
@end deffn

@geindex PyNumber_InPlaceXor (C function)
@anchor{c-api/number c PyNumber_InPlaceXor}@anchor{41e7}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_InPlaceXor (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the “bitwise exclusive or” of @emph{o1} by @emph{o2} on success, or @code{NULL} on
failure.  The operation is done @emph{in-place} when @emph{o1} supports it.  This is the
equivalent of the Python statement @code{o1 ^= o2}.
@end deffn

@geindex PyNumber_InPlaceOr (C function)
@anchor{c-api/number c PyNumber_InPlaceOr}@anchor{41e1}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_InPlaceOr (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the “bitwise or” of @emph{o1} and @emph{o2} on success, or @code{NULL} on failure.  The
operation is done @emph{in-place} when @emph{o1} supports it.  This is the equivalent of
the Python statement @code{o1 |= o2}.
@end deffn

@geindex PyNumber_Long (C function)
@anchor{c-api/number c PyNumber_Long}@anchor{57e}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Long (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; int

Returns the @emph{o} converted to an integer object on success, or @code{NULL} on
failure.  This is the equivalent of the Python expression @code{int(o)}.
@end deffn

@geindex PyNumber_Float (C function)
@anchor{c-api/number c PyNumber_Float}@anchor{57f}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Float (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; float

Returns the @emph{o} converted to a float object on success, or @code{NULL} on failure.
This is the equivalent of the Python expression @code{float(o)}.
@end deffn

@geindex PyNumber_Index (C function)
@anchor{c-api/number c PyNumber_Index}@anchor{35f}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_Index (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the @emph{o} converted to a Python int on success or @code{NULL} with a
@ref{162,,TypeError} exception raised on failure.


Changed in version 3.10: The result always has exact type @ref{1b1,,int}.  Previously, the result
could have been an instance of a subclass of @code{int}.

@end deffn

@geindex PyNumber_ToBase (C function)
@anchor{c-api/number c PyNumber_ToBase}@anchor{121c}
@deffn {C Function} @ref{375,,PyObject}  *PyNumber_ToBase (PyObject  *n, int  base)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the integer @emph{n} converted to base @emph{base} as a string.  The @emph{base}
argument must be one of 2, 8, 10, or 16.  For base 2, 8, or 16, the
returned string is prefixed with a base marker of @code{'0b'}, @code{'0o'}, or
@code{'0x'}, respectively.  If @emph{n} is not a Python int, it is converted with
@ref{35f,,PyNumber_Index()} first.
@end deffn

@geindex PyNumber_AsSsize_t (C function)
@anchor{c-api/number c PyNumber_AsSsize_t}@anchor{41d9}
@deffn {C Function} Py_ssize_t  PyNumber_AsSsize_t (PyObject  *o, PyObject  *exc)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns @emph{o} converted to a Py_ssize_t value if @emph{o} can be interpreted as an
integer.  If the call fails, an exception is raised and @code{-1} is returned.

If @emph{o} can be converted to a Python int but the attempt to
convert to a Py_ssize_t value would raise an @ref{347,,OverflowError}, then the
@emph{exc} argument is the type of exception that will be raised (usually
@ref{1066,,IndexError} or @ref{347,,OverflowError}).  If @emph{exc} is @code{NULL}, then the
exception is cleared and the value is clipped to @code{PY_SSIZE_T_MIN} for a negative
integer or @code{PY_SSIZE_T_MAX} for a positive integer.
@end deffn

@geindex PyIndex_Check (C function)
@anchor{c-api/number c PyIndex_Check}@anchor{496}
@deffn {C Function} int  PyIndex_Check (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.8.}
Returns @code{1} if @emph{o} is an index integer (has the nb_index slot of  the
tp_as_number structure filled in), and @code{0} otherwise.
This function always succeeds.
@end deffn

@node Sequence Protocol,Mapping Protocol,Number Protocol,Abstract Objects Layer
@anchor{c-api/sequence doc}@anchor{4348}@anchor{c-api/sequence sequence}@anchor{4349}@anchor{c-api/sequence sequence-protocol}@anchor{434a}
@subsection Sequence Protocol


@geindex PySequence_Check (C function)
@anchor{c-api/sequence c PySequence_Check}@anchor{4216}
@deffn {C Function} int  PySequence_Check (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return @code{1} if the object provides sequence protocol, and @code{0} otherwise.
Note that it returns @code{1} for Python classes with a @code{__getitem__()}
method unless they are @ref{3b0,,dict} subclasses since in general case it
is impossible to determine what the type of keys it supports.  This
function always succeeds.
@end deffn

@geindex PySequence_Size (C function)
@geindex PySequence_Length (C function)
@anchor{c-api/sequence c PySequence_Size}@anchor{1310}
@deffn {C Function} Py_ssize_t  PySequence_Size (PyObject  *o)
@anchor{c-api/sequence c PySequence_Length}@anchor{4220}
@deffnx {C Function} Py_ssize_t  PySequence_Length (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; len

Returns the number of objects in sequence @emph{o} on success, and @code{-1} on
failure.  This is equivalent to the Python expression @code{len(o)}.
@end deffn

@geindex PySequence_Concat (C function)
@anchor{c-api/sequence c PySequence_Concat}@anchor{4217}
@deffn {C Function} @ref{375,,PyObject}  *PySequence_Concat (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the concatenation of @emph{o1} and @emph{o2} on success, and @code{NULL} on failure.
This is the equivalent of the Python expression @code{o1 + o2}.
@end deffn

@geindex PySequence_Repeat (C function)
@anchor{c-api/sequence c PySequence_Repeat}@anchor{4222}
@deffn {C Function} @ref{375,,PyObject}  *PySequence_Repeat (PyObject  *o, Py_ssize_t  count)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the result of repeating sequence object @emph{o} @emph{count} times, or @code{NULL} on
failure.  This is the equivalent of the Python expression @code{o * count}.
@end deffn

@geindex PySequence_InPlaceConcat (C function)
@anchor{c-api/sequence c PySequence_InPlaceConcat}@anchor{421d}
@deffn {C Function} @ref{375,,PyObject}  *PySequence_InPlaceConcat (PyObject  *o1, PyObject  *o2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the concatenation of @emph{o1} and @emph{o2} on success, and @code{NULL} on failure.
The operation is done @emph{in-place} when @emph{o1} supports it.  This is the equivalent
of the Python expression @code{o1 += o2}.
@end deffn

@geindex PySequence_InPlaceRepeat (C function)
@anchor{c-api/sequence c PySequence_InPlaceRepeat}@anchor{421e}
@deffn {C Function} @ref{375,,PyObject}  *PySequence_InPlaceRepeat (PyObject  *o, Py_ssize_t  count)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the result of repeating sequence object @emph{o} @emph{count} times, or @code{NULL} on
failure.  The operation is done @emph{in-place} when @emph{o} supports it.  This is the
equivalent of the Python expression @code{o *= count}.
@end deffn

@geindex PySequence_GetItem (C function)
@anchor{c-api/sequence c PySequence_GetItem}@anchor{1311}
@deffn {C Function} @ref{375,,PyObject}  *PySequence_GetItem (PyObject  *o, Py_ssize_t  i)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the @emph{i}th element of @emph{o}, or @code{NULL} on failure. This is the equivalent of
the Python expression @code{o[i]}.
@end deffn

@geindex PySequence_GetSlice (C function)
@anchor{c-api/sequence c PySequence_GetSlice}@anchor{421c}
@deffn {C Function} @ref{375,,PyObject}  *PySequence_GetSlice (PyObject  *o, Py_ssize_t  i1, Py_ssize_t  i2)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the slice of sequence object @emph{o} between @emph{i1} and @emph{i2}, or @code{NULL} on
failure. This is the equivalent of the Python expression @code{o[i1:i2]}.
@end deffn

@geindex PySequence_SetItem (C function)
@anchor{c-api/sequence c PySequence_SetItem}@anchor{1312}
@deffn {C Function} int  PySequence_SetItem (PyObject  *o, Py_ssize_t  i, PyObject  *v)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Assign object @emph{v} to the @emph{i}th element of @emph{o}.  Raise an exception
and return @code{-1} on failure; return @code{0} on success.  This
is the equivalent of the Python statement @code{o[i] = v}.  This function @emph{does
not} steal a reference to @emph{v}.

If @emph{v} is @code{NULL}, the element is deleted, however this feature is
deprecated in favour of using @ref{1313,,PySequence_DelItem()}.
@end deffn

@geindex PySequence_DelItem (C function)
@anchor{c-api/sequence c PySequence_DelItem}@anchor{1313}
@deffn {C Function} int  PySequence_DelItem (PyObject  *o, Py_ssize_t  i)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Delete the @emph{i}th element of object @emph{o}.  Returns @code{-1} on failure.  This is the
equivalent of the Python statement @code{del o[i]}.
@end deffn

@geindex PySequence_SetSlice (C function)
@anchor{c-api/sequence c PySequence_SetSlice}@anchor{4223}
@deffn {C Function} int  PySequence_SetSlice (PyObject  *o, Py_ssize_t  i1, Py_ssize_t  i2, PyObject  *v)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Assign the sequence object @emph{v} to the slice in sequence object @emph{o} from @emph{i1} to
@emph{i2}.  This is the equivalent of the Python statement @code{o[i1:i2] = v}.
@end deffn

@geindex PySequence_DelSlice (C function)
@anchor{c-api/sequence c PySequence_DelSlice}@anchor{421a}
@deffn {C Function} int  PySequence_DelSlice (PyObject  *o, Py_ssize_t  i1, Py_ssize_t  i2)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Delete the slice in sequence object @emph{o} from @emph{i1} to @emph{i2}.  Returns @code{-1} on
failure.  This is the equivalent of the Python statement @code{del o[i1:i2]}.
@end deffn

@geindex PySequence_Count (C function)
@anchor{c-api/sequence c PySequence_Count}@anchor{4219}
@deffn {C Function} Py_ssize_t  PySequence_Count (PyObject  *o, PyObject  *value)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the number of occurrences of @emph{value} in @emph{o}, that is, return the number
of keys for which @code{o[key] == value}.  On failure, return @code{-1}.  This is
equivalent to the Python expression @code{o.count(value)}.
@end deffn

@geindex PySequence_Contains (C function)
@anchor{c-api/sequence c PySequence_Contains}@anchor{4218}
@deffn {C Function} int  PySequence_Contains (PyObject  *o, PyObject  *value)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Determine if @emph{o} contains @emph{value}.  If an item in @emph{o} is equal to @emph{value},
return @code{1}, otherwise return @code{0}. On error, return @code{-1}.  This is
equivalent to the Python expression @code{value in o}.
@end deffn

@geindex PySequence_Index (C function)
@anchor{c-api/sequence c PySequence_Index}@anchor{421f}
@deffn {C Function} Py_ssize_t  PySequence_Index (PyObject  *o, PyObject  *value)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the first index @emph{i} for which @code{o[i] == value}.  On error, return
@code{-1}.    This is equivalent to the Python expression @code{o.index(value)}.
@end deffn

@geindex PySequence_List (C function)
@anchor{c-api/sequence c PySequence_List}@anchor{4221}
@deffn {C Function} @ref{375,,PyObject}  *PySequence_List (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a list object with the same contents as the sequence or iterable @emph{o},
or @code{NULL} on failure.  The returned list is guaranteed to be new.  This is
equivalent to the Python expression @code{list(o)}.
@end deffn

@geindex PySequence_Tuple (C function)
@anchor{c-api/sequence c PySequence_Tuple}@anchor{4224}
@deffn {C Function} @ref{375,,PyObject}  *PySequence_Tuple (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; tuple

Return a tuple object with the same contents as the sequence or iterable @emph{o},
or @code{NULL} on failure.  If @emph{o} is a tuple, a new reference will be returned,
otherwise a tuple will be constructed with the appropriate contents.  This is
equivalent to the Python expression @code{tuple(o)}.
@end deffn

@geindex PySequence_Fast (C function)
@anchor{c-api/sequence c PySequence_Fast}@anchor{421b}
@deffn {C Function} @ref{375,,PyObject}  *PySequence_Fast (PyObject  *o, const  char  *m)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the sequence or iterable @emph{o} as an object usable by the other
@code{PySequence_Fast*} family of functions. If the object is not a sequence or
iterable, raises @ref{162,,TypeError} with @emph{m} as the message text. Returns
@code{NULL} on failure.

The @code{PySequence_Fast*} functions are thus named because they assume
@emph{o} is a @ref{4302,,PyTupleObject} or a @ref{434b,,PyListObject} and access
the data fields of @emph{o} directly.

As a CPython implementation detail, if @emph{o} is already a sequence or list, it
will be returned.
@end deffn

@geindex PySequence_Fast_GET_SIZE (C function)
@anchor{c-api/sequence c PySequence_Fast_GET_SIZE}@anchor{434c}
@deffn {C Function} Py_ssize_t  PySequence_Fast_GET_SIZE (PyObject  *o)

Returns the length of @emph{o}, assuming that @emph{o} was returned by
@ref{421b,,PySequence_Fast()} and that @emph{o} is not @code{NULL}.  The size can also be
gotten by calling @ref{1310,,PySequence_Size()} on @emph{o}, but
@ref{434c,,PySequence_Fast_GET_SIZE()} is faster because it can assume @emph{o} is a
list or tuple.
@end deffn

@geindex PySequence_Fast_GET_ITEM (C function)
@anchor{c-api/sequence c PySequence_Fast_GET_ITEM}@anchor{434d}
@deffn {C Function} @ref{375,,PyObject}  *PySequence_Fast_GET_ITEM (PyObject  *o, Py_ssize_t  i)
@emph{Return value: Borrowed reference.}
Return the @emph{i}th element of @emph{o}, assuming that @emph{o} was returned by
@ref{421b,,PySequence_Fast()}, @emph{o} is not @code{NULL}, and that @emph{i} is within bounds.
@end deffn

@geindex PySequence_Fast_ITEMS (C function)
@anchor{c-api/sequence c PySequence_Fast_ITEMS}@anchor{434e}
@deffn {C Function} @ref{375,,PyObject}  **PySequence_Fast_ITEMS (PyObject  *o)

Return the underlying array of PyObject pointers.  Assumes that @emph{o} was returned
by @ref{421b,,PySequence_Fast()} and @emph{o} is not @code{NULL}.

Note, if a list gets resized, the reallocation may relocate the items array.
So, only use the underlying array pointer in contexts where the sequence
cannot change.
@end deffn

@geindex PySequence_ITEM (C function)
@anchor{c-api/sequence c PySequence_ITEM}@anchor{434f}
@deffn {C Function} @ref{375,,PyObject}  *PySequence_ITEM (PyObject  *o, Py_ssize_t  i)
@emph{Return value: New reference.}
Return the @emph{i}th element of @emph{o} or @code{NULL} on failure. Faster form of
@ref{1311,,PySequence_GetItem()} but without checking that
@ref{4216,,PySequence_Check()} on @emph{o} is true and without adjustment for negative
indices.
@end deffn

@node Mapping Protocol,Iterator Protocol,Sequence Protocol,Abstract Objects Layer
@anchor{c-api/mapping doc}@anchor{4350}@anchor{c-api/mapping mapping}@anchor{4351}@anchor{c-api/mapping mapping-protocol}@anchor{4352}
@subsection Mapping Protocol


See also @ref{4115,,PyObject_GetItem()}, @ref{4113,,PyObject_SetItem()} and
@ref{4202,,PyObject_DelItem()}.

@geindex PyMapping_Check (C function)
@anchor{c-api/mapping c PyMapping_Check}@anchor{41bf}
@deffn {C Function} int  PyMapping_Check (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return @code{1} if the object provides mapping protocol or supports slicing,
and @code{0} otherwise.  Note that it returns @code{1} for Python classes with
a @code{__getitem__()} method since in general case it is impossible to
determine what type of keys it supports. This function always succeeds.
@end deffn

@geindex PyMapping_Size (C function)
@geindex PyMapping_Length (C function)
@anchor{c-api/mapping c PyMapping_Size}@anchor{1314}
@deffn {C Function} Py_ssize_t  PyMapping_Size (PyObject  *o)
@anchor{c-api/mapping c PyMapping_Length}@anchor{41c3}
@deffnx {C Function} Py_ssize_t  PyMapping_Length (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; len

Returns the number of keys in object @emph{o} on success, and @code{-1} on failure.
This is equivalent to the Python expression @code{len(o)}.
@end deffn

@geindex PyMapping_GetItemString (C function)
@anchor{c-api/mapping c PyMapping_GetItemString}@anchor{41c0}
@deffn {C Function} @ref{375,,PyObject}  *PyMapping_GetItemString (PyObject  *o, const  char  *key)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return element of @emph{o} corresponding to the string @emph{key} or @code{NULL} on failure.
This is the equivalent of the Python expression @code{o[key]}.
See also @ref{4115,,PyObject_GetItem()}.
@end deffn

@geindex PyMapping_SetItemString (C function)
@anchor{c-api/mapping c PyMapping_SetItemString}@anchor{41c4}
@deffn {C Function} int  PyMapping_SetItemString (PyObject  *o, const  char  *key, PyObject  *v)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Map the string @emph{key} to the value @emph{v} in object @emph{o}.  Returns @code{-1} on
failure.  This is the equivalent of the Python statement @code{o[key] = v}.
See also @ref{4113,,PyObject_SetItem()}.  This function @emph{does not} steal a
reference to @emph{v}.
@end deffn

@geindex PyMapping_DelItem (C function)
@anchor{c-api/mapping c PyMapping_DelItem}@anchor{4353}
@deffn {C Function} int  PyMapping_DelItem (PyObject  *o, PyObject  *key)

Remove the mapping for the object @emph{key} from the object @emph{o}.  Return @code{-1}
on failure.  This is equivalent to the Python statement @code{del o[key]}.
This is an alias of @ref{4202,,PyObject_DelItem()}.
@end deffn

@geindex PyMapping_DelItemString (C function)
@anchor{c-api/mapping c PyMapping_DelItemString}@anchor{4354}
@deffn {C Function} int  PyMapping_DelItemString (PyObject  *o, const  char  *key)

Remove the mapping for the string @emph{key} from the object @emph{o}.  Return @code{-1}
on failure.  This is equivalent to the Python statement @code{del o[key]}.
@end deffn

@geindex PyMapping_HasKey (C function)
@anchor{c-api/mapping c PyMapping_HasKey}@anchor{41c1}
@deffn {C Function} int  PyMapping_HasKey (PyObject  *o, PyObject  *key)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return @code{1} if the mapping object has the key @emph{key} and @code{0} otherwise.
This is equivalent to the Python expression @code{key in o}.
This function always succeeds.

Note that exceptions which occur while calling the @code{__getitem__()}
method will get suppressed.
To get error reporting use @ref{4115,,PyObject_GetItem()} instead.
@end deffn

@geindex PyMapping_HasKeyString (C function)
@anchor{c-api/mapping c PyMapping_HasKeyString}@anchor{41c2}
@deffn {C Function} int  PyMapping_HasKeyString (PyObject  *o, const  char  *key)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return @code{1} if the mapping object has the key @emph{key} and @code{0} otherwise.
This is equivalent to the Python expression @code{key in o}.
This function always succeeds.

Note that exceptions which occur while calling the @code{__getitem__()}
method and creating a temporary string object will get suppressed.
To get error reporting use @ref{41c0,,PyMapping_GetItemString()} instead.
@end deffn

@geindex PyMapping_Keys (C function)
@anchor{c-api/mapping c PyMapping_Keys}@anchor{6f0}
@deffn {C Function} @ref{375,,PyObject}  *PyMapping_Keys (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
On success, return a list of the keys in object @emph{o}.  On failure, return
@code{NULL}.


Changed in version 3.7: Previously, the function returned a list or a tuple.

@end deffn

@geindex PyMapping_Values (C function)
@anchor{c-api/mapping c PyMapping_Values}@anchor{6f1}
@deffn {C Function} @ref{375,,PyObject}  *PyMapping_Values (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
On success, return a list of the values in object @emph{o}.  On failure, return
@code{NULL}.


Changed in version 3.7: Previously, the function returned a list or a tuple.

@end deffn

@geindex PyMapping_Items (C function)
@anchor{c-api/mapping c PyMapping_Items}@anchor{6f2}
@deffn {C Function} @ref{375,,PyObject}  *PyMapping_Items (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
On success, return a list of the items in object @emph{o}, where each item is a
tuple containing a key-value pair.  On failure, return @code{NULL}.


Changed in version 3.7: Previously, the function returned a list or a tuple.

@end deffn

@node Iterator Protocol,Buffer Protocol,Mapping Protocol,Abstract Objects Layer
@anchor{c-api/iter doc}@anchor{4355}@anchor{c-api/iter iterator}@anchor{4356}@anchor{c-api/iter iterator-protocol}@anchor{4357}
@subsection Iterator Protocol


There are two functions specifically for working with iterators.

@geindex PyIter_Check (C function)
@anchor{c-api/iter c PyIter_Check}@anchor{116c}
@deffn {C Function} int  PyIter_Check (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.8.}
Return non-zero if the object @emph{o} can be safely passed to
@ref{41ab,,PyIter_Next()}, and @code{0} otherwise.  This function always succeeds.
@end deffn

@geindex PyAIter_Check (C function)
@anchor{c-api/iter c PyAIter_Check}@anchor{4129}
@deffn {C Function} int  PyAIter_Check (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Returns non-zero if the object ‘obj’ provides @code{AsyncIterator}
protocols, and @code{0} otherwise.  This function always succeeds.


New in version 3.10.

@end deffn

@geindex PyIter_Next (C function)
@anchor{c-api/iter c PyIter_Next}@anchor{41ab}
@deffn {C Function} @ref{375,,PyObject}  *PyIter_Next (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the next value from the iterator @emph{o}.  The object must be an iterator
according to @ref{116c,,PyIter_Check()} (it is up to the caller to check this).
If there are no remaining values, returns @code{NULL} with no exception set.
If an error occurs while retrieving the item, returns @code{NULL} and passes
along the exception.
@end deffn

To write a loop which iterates over an iterator, the C code should look
something like this:

@example
PyObject *iterator = PyObject_GetIter(obj);
PyObject *item;

if (iterator == NULL) @{
    /* propagate error */
@}

while ((item = PyIter_Next(iterator))) @{
    /* do something with item */
    ...
    /* release reference when done */
    Py_DECREF(item);
@}

Py_DECREF(iterator);

if (PyErr_Occurred()) @{
    /* propagate error */
@}
else @{
    /* continue doing useful work */
@}
@end example

@geindex PySendResult (C type)
@anchor{c-api/iter c PySendResult}@anchor{4358}
@deffn {C Type} type  PySendResult

The enum value used to represent different results of @ref{366,,PyIter_Send()}.


New in version 3.10.

@end deffn

@geindex PyIter_Send (C function)
@anchor{c-api/iter c PyIter_Send}@anchor{366}
@deffn {C Function} @ref{4358,,PySendResult}  PyIter_Send (PyObject  *iter, PyObject  *arg, PyObject  **presult)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Sends the @emph{arg} value into the iterator @emph{iter}. Returns:


@itemize -

@item 
@code{PYGEN_RETURN} if iterator returns. Return value is returned via @emph{presult}.

@item 
@code{PYGEN_NEXT} if iterator yields. Yielded value is returned via @emph{presult}.

@item 
@code{PYGEN_ERROR} if iterator has raised and exception. @emph{presult} is set to @code{NULL}.
@end itemize


New in version 3.10.

@end deffn

@geindex buffer protocol
@geindex buffer interface; (see buffer protocol)
@geindex buffer object; (see buffer protocol)

@node Buffer Protocol,Old Buffer Protocol,Iterator Protocol,Abstract Objects Layer
@anchor{c-api/buffer doc}@anchor{4359}@anchor{c-api/buffer buffer-protocol}@anchor{435a}@anchor{c-api/buffer bufferobjects}@anchor{1a3e}
@subsection Buffer Protocol


Certain objects available in Python wrap access to an underlying memory
array or @emph{buffer}.  Such objects include the built-in @ref{2f0,,bytes} and
@ref{2f1,,bytearray}, and some extension types like @ref{25e,,array.array}.
Third-party libraries may define their own types for special purposes, such
as image processing or numeric analysis.

While each of these types have their own semantics, they share the common
characteristic of being backed by a possibly large memory buffer.  It is
then desirable, in some situations, to access that buffer directly and
without intermediate copying.

Python provides such a facility at the C level in the form of the @ref{1a3e,,buffer protocol}.  This protocol has two sides:

@geindex PyBufferProcs


@itemize -

@item 
on the producer side, a type can export a “buffer interface” which allows
objects of that type to expose information about their underlying buffer.
This interface is described in the section @ref{435b,,Buffer Object Structures};

@item 
on the consumer side, several means are available to obtain a pointer to
the raw underlying data of an object (for example a method parameter).
@end itemize

Simple objects such as @ref{2f0,,bytes} and @ref{2f1,,bytearray} expose their
underlying buffer in byte-oriented form.  Other forms are possible; for example,
the elements exposed by an @ref{25e,,array.array} can be multi-byte values.

An example consumer of the buffer interface is the @ref{824,,write()}
method of file objects: any object that can export a series of bytes through
the buffer interface can be written to a file.  While @code{write()} only
needs read-only access to the internal contents of the object passed to it,
other methods such as @ref{2494,,readinto()} need write access
to the contents of their argument.  The buffer interface allows objects to
selectively allow or reject exporting of read-write and read-only buffers.

There are two ways for a consumer of the buffer interface to acquire a buffer
over a target object:


@itemize *

@item 
call @ref{1dd,,PyObject_GetBuffer()} with the right parameters;

@item 
call @ref{37f,,PyArg_ParseTuple()} (or one of its siblings) with one of the
@code{y*}, @code{w*} or @code{s*} @ref{381,,format codes}.
@end itemize

In both cases, @ref{1e5,,PyBuffer_Release()} must be called when the buffer
isn’t needed anymore.  Failure to do so could lead to various issues such as
resource leaks.

@menu
* Buffer structure:: 
* Buffer request types:: 
* Complex arrays:: 
* Buffer-related functions:: 

@end menu

@node Buffer structure,Buffer request types,,Buffer Protocol
@anchor{c-api/buffer buffer-structure}@anchor{435c}@anchor{c-api/buffer id1}@anchor{435d}
@subsubsection Buffer structure


Buffer structures (or simply “buffers”) are useful as a way to expose the
binary data from another object to the Python programmer.  They can also be
used as a zero-copy slicing mechanism.  Using their ability to reference a
block of memory, it is possible to expose any data to the Python programmer
quite easily.  The memory could be a large, constant array in a C extension,
it could be a raw block of memory for manipulation before passing to an
operating system library, or it could be used to pass around structured data
in its native, in-memory format.

Contrary to most data types exposed by the Python interpreter, buffers
are not @ref{375,,PyObject} pointers but rather simple C structures.  This
allows them to be created and copied very simply.  When a generic wrapper
around a buffer is needed, a @ref{435e,,memoryview} object
can be created.

For short instructions how to write an exporting object, see
@ref{435b,,Buffer Object Structures}. For obtaining
a buffer, see @ref{1dd,,PyObject_GetBuffer()}.

@geindex Py_buffer (C type)
@anchor{c-api/buffer c Py_buffer}@anchor{1db}
@deffn {C Type} type  Py_buffer
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
@geindex Py_buffer.buf (C member)
@anchor{c-api/buffer c Py_buffer buf}@anchor{435f}
@deffn {C Member} void  *buf

A pointer to the start of the logical structure described by the buffer
fields. This can be any location within the underlying physical memory
block of the exporter. For example, with negative @ref{4360,,strides}
the value may point to the end of the memory block.

For @ref{1b0f,,contiguous} arrays, the value points to the beginning of
the memory block.
@end deffn

@geindex Py_buffer.obj (C member)
@anchor{c-api/buffer c Py_buffer obj}@anchor{4361}
@deffn {C Member} void  *obj

A new reference to the exporting object. The reference is owned by
the consumer and automatically decremented and set to @code{NULL} by
@ref{1e5,,PyBuffer_Release()}. The field is the equivalent of the return
value of any standard C-API function.

As a special case, for @emph{temporary} buffers that are wrapped by
@ref{1e6,,PyMemoryView_FromBuffer()} or @ref{1e4,,PyBuffer_FillInfo()}
this field is @code{NULL}. In general, exporting objects MUST NOT
use this scheme.
@end deffn

@geindex Py_buffer.len (C member)
@anchor{c-api/buffer c Py_buffer len}@anchor{4362}
@deffn {C Member} Py_ssize_t  len

@code{product(shape) * itemsize}. For contiguous arrays, this is the length
of the underlying memory block. For non-contiguous arrays, it is the length
that the logical structure would have if it were copied to a contiguous
representation.

Accessing @code{((char *)buf)[0] up to ((char *)buf)[len-1]} is only valid
if the buffer has been obtained by a request that guarantees contiguity. In
most cases such a request will be @ref{4363,,PyBUF_SIMPLE} or @ref{4364,,PyBUF_WRITABLE}.
@end deffn

@geindex Py_buffer.readonly (C member)
@anchor{c-api/buffer c Py_buffer readonly}@anchor{4365}
@deffn {C Member} int  readonly

An indicator of whether the buffer is read-only. This field is controlled
by the @ref{4364,,PyBUF_WRITABLE} flag.
@end deffn

@geindex Py_buffer.itemsize (C member)
@anchor{c-api/buffer c Py_buffer itemsize}@anchor{4366}
@deffn {C Member} Py_ssize_t  itemsize

Item size in bytes of a single element. Same as the value of @ref{125e,,struct.calcsize()}
called on non-@code{NULL} @ref{4367,,format} values.

Important exception: If a consumer requests a buffer without the
@ref{4368,,PyBUF_FORMAT} flag, @ref{4367,,format} will
be set to  @code{NULL},  but @ref{4366,,itemsize} still has
the value for the original format.

If @ref{4369,,shape} is present, the equality
@code{product(shape) * itemsize == len} still holds and the consumer
can use @ref{4366,,itemsize} to navigate the buffer.

If @ref{4369,,shape} is @code{NULL} as a result of a @ref{4363,,PyBUF_SIMPLE}
or a @ref{4364,,PyBUF_WRITABLE} request, the consumer must disregard
@ref{4366,,itemsize} and assume @code{itemsize == 1}.
@end deffn

@geindex Py_buffer.format (C member)
@anchor{c-api/buffer c Py_buffer format}@anchor{4367}
@deffn {C Member} const  char  *format

A @emph{NUL} terminated string in @ref{f3,,struct} module style syntax describing
the contents of a single item. If this is @code{NULL}, @code{"B"} (unsigned bytes)
is assumed.

This field is controlled by the @ref{4368,,PyBUF_FORMAT} flag.
@end deffn

@geindex Py_buffer.ndim (C member)
@anchor{c-api/buffer c Py_buffer ndim}@anchor{436a}
@deffn {C Member} int  ndim

The number of dimensions the memory represents as an n-dimensional array.
If it is @code{0}, @ref{435f,,buf} points to a single item representing
a scalar. In this case, @ref{4369,,shape}, @ref{4360,,strides}
and @ref{436b,,suboffsets} MUST be @code{NULL}.

The macro @code{PyBUF_MAX_NDIM} limits the maximum number of dimensions
to 64. Exporters MUST respect this limit, consumers of multi-dimensional
buffers SHOULD be able to handle up to @code{PyBUF_MAX_NDIM} dimensions.
@end deffn

@geindex Py_buffer.shape (C member)
@anchor{c-api/buffer c Py_buffer shape}@anchor{4369}
@deffn {C Member} Py_ssize_t  *shape

An array of @code{Py_ssize_t} of length @ref{436a,,ndim}
indicating the shape of the memory as an n-dimensional array. Note that
@code{shape[0] * ... * shape[ndim-1] * itemsize} MUST be equal to
@ref{4362,,len}.

Shape values are restricted to @code{shape[n] >= 0}. The case
@code{shape[n] == 0} requires special attention. See @ref{436c,,complex arrays}
for further information.

The shape array is read-only for the consumer.
@end deffn

@geindex Py_buffer.strides (C member)
@anchor{c-api/buffer c Py_buffer strides}@anchor{4360}
@deffn {C Member} Py_ssize_t  *strides

An array of @code{Py_ssize_t} of length @ref{436a,,ndim}
giving the number of bytes to skip to get to a new element in each
dimension.

Stride values can be any integer. For regular arrays, strides are
usually positive, but a consumer MUST be able to handle the case
@code{strides[n] <= 0}. See @ref{436c,,complex arrays} for further information.

The strides array is read-only for the consumer.
@end deffn

@geindex Py_buffer.suboffsets (C member)
@anchor{c-api/buffer c Py_buffer suboffsets}@anchor{436b}
@deffn {C Member} Py_ssize_t  *suboffsets

An array of @code{Py_ssize_t} of length @ref{436a,,ndim}.
If @code{suboffsets[n] >= 0}, the values stored along the nth dimension are
pointers and the suboffset value dictates how many bytes to add to each
pointer after de-referencing. A suboffset value that is negative
indicates that no de-referencing should occur (striding in a contiguous
memory block).

If all suboffsets are negative (i.e. no de-referencing is needed), then
this field must be @code{NULL} (the default value).

This type of array representation is used by the Python Imaging Library
(PIL). See @ref{436c,,complex arrays} for further information how to access elements
of such an array.

The suboffsets array is read-only for the consumer.
@end deffn

@geindex Py_buffer.internal (C member)
@anchor{c-api/buffer c Py_buffer internal}@anchor{436d}
@deffn {C Member} void  *internal

This is for use internally by the exporting object. For example, this
might be re-cast as an integer by the exporter and used to store flags
about whether or not the shape, strides, and suboffsets arrays must be
freed when the buffer is released. The consumer MUST NOT alter this
value.
@end deffn
@end deffn

@node Buffer request types,Complex arrays,Buffer structure,Buffer Protocol
@anchor{c-api/buffer buffer-request-types}@anchor{436e}@anchor{c-api/buffer id2}@anchor{436f}
@subsubsection Buffer request types


Buffers are usually obtained by sending a buffer request to an exporting
object via @ref{1dd,,PyObject_GetBuffer()}. Since the complexity of the logical
structure of the memory can vary drastically, the consumer uses the @emph{flags}
argument to specify the exact buffer type it can handle.

All @ref{1db,,Py_buffer} fields are unambiguously defined by the request
type.

@menu
* request-independent fields:: 
* readonly@comma{} format: readonly format. 
* shape@comma{} strides@comma{} suboffsets: shape strides suboffsets. 
* contiguity requests:: 
* compound requests:: 

@end menu

@node request-independent fields,readonly format,,Buffer request types
@anchor{c-api/buffer request-independent-fields}@anchor{4370}
@subsubsection request-independent fields


The following fields are not influenced by @emph{flags} and must always be filled in
with the correct values: @ref{4361,,obj}, @ref{435f,,buf},
@ref{4362,,len}, @ref{4366,,itemsize}, @ref{436a,,ndim}.

@node readonly format,shape strides suboffsets,request-independent fields,Buffer request types
@anchor{c-api/buffer readonly-format}@anchor{4371}
@subsubsection readonly, format


@quotation

@geindex PyBUF_WRITABLE (C macro)
@anchor{c-api/buffer c PyBUF_WRITABLE}@anchor{4364}
@deffn {C Macro} PyBUF_WRITABLE

Controls the @ref{4365,,readonly} field. If set, the exporter
MUST provide a writable buffer or else report failure. Otherwise, the
exporter MAY provide either a read-only or writable buffer, but the choice
MUST be consistent for all consumers.
@end deffn

@geindex PyBUF_FORMAT (C macro)
@anchor{c-api/buffer c PyBUF_FORMAT}@anchor{4368}
@deffn {C Macro} PyBUF_FORMAT

Controls the @ref{4367,,format} field. If set, this field MUST
be filled in correctly. Otherwise, this field MUST be @code{NULL}.
@end deffn
@end quotation

@ref{4364,,PyBUF_WRITABLE} can be |’d to any of the flags in the next section.
Since @ref{4363,,PyBUF_SIMPLE} is defined as 0, @ref{4364,,PyBUF_WRITABLE}
can be used as a stand-alone flag to request a simple writable buffer.

@ref{4368,,PyBUF_FORMAT} can be |’d to any of the flags except @ref{4363,,PyBUF_SIMPLE}.
The latter already implies format @code{B} (unsigned bytes).

@node shape strides suboffsets,contiguity requests,readonly format,Buffer request types
@anchor{c-api/buffer shape-strides-suboffsets}@anchor{4372}
@subsubsection shape, strides, suboffsets


The flags that control the logical structure of the memory are listed
in decreasing order of complexity. Note that each flag contains all bits
of the flags below it.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} {xxxxxxxxxxx} {xxxxxxxxxxxxxx} 
@headitem

Request

@tab

shape

@tab

strides

@tab

suboffsets

@item

@geindex PyBUF_INDIRECT (C macro)
@anchor{c-api/buffer c PyBUF_INDIRECT}@anchor{4373}
@deffn {C Macro} PyBUF_INDIRECT
@end deffn

@tab

yes

@tab

yes

@tab

if needed

@item

@geindex PyBUF_STRIDES (C macro)
@anchor{c-api/buffer c PyBUF_STRIDES}@anchor{4374}
@deffn {C Macro} PyBUF_STRIDES
@end deffn

@tab

yes

@tab

yes

@tab

NULL

@item

@geindex PyBUF_ND (C macro)
@anchor{c-api/buffer c PyBUF_ND}@anchor{4375}
@deffn {C Macro} PyBUF_ND
@end deffn

@tab

yes

@tab

NULL

@tab

NULL

@item

@geindex PyBUF_SIMPLE (C macro)
@anchor{c-api/buffer c PyBUF_SIMPLE}@anchor{4363}
@deffn {C Macro} PyBUF_SIMPLE
@end deffn

@tab

NULL

@tab

NULL

@tab

NULL

@end multitable


@geindex contiguous
@geindex C-contiguous
@geindex Fortran contiguous

@node contiguity requests,compound requests,shape strides suboffsets,Buffer request types
@anchor{c-api/buffer contiguity-requests}@anchor{4376}
@subsubsection contiguity requests


C or Fortran @ref{1b0f,,contiguity} can be explicitly requested,
with and without stride information. Without stride information, the buffer
must be C-contiguous.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} {xxxxxxxxxxx} {xxxxxxxxxxxxxx} {xxxxxxxxxx} 
@headitem

Request

@tab

shape

@tab

strides

@tab

suboffsets

@tab

contig

@item

@geindex PyBUF_C_CONTIGUOUS (C macro)
@anchor{c-api/buffer c PyBUF_C_CONTIGUOUS}@anchor{4377}
@deffn {C Macro} PyBUF_C_CONTIGUOUS
@end deffn

@tab

yes

@tab

yes

@tab

NULL

@tab

C

@item

@geindex PyBUF_F_CONTIGUOUS (C macro)
@anchor{c-api/buffer c PyBUF_F_CONTIGUOUS}@anchor{4378}
@deffn {C Macro} PyBUF_F_CONTIGUOUS
@end deffn

@tab

yes

@tab

yes

@tab

NULL

@tab

F

@item

@geindex PyBUF_ANY_CONTIGUOUS (C macro)
@anchor{c-api/buffer c PyBUF_ANY_CONTIGUOUS}@anchor{4379}
@deffn {C Macro} PyBUF_ANY_CONTIGUOUS
@end deffn

@tab

yes

@tab

yes

@tab

NULL

@tab

C or F

@item

@ref{4375,,PyBUF_ND}

@tab

yes

@tab

NULL

@tab

NULL

@tab

C

@end multitable


@node compound requests,,contiguity requests,Buffer request types
@anchor{c-api/buffer compound-requests}@anchor{437a}
@subsubsection compound requests


All possible requests are fully defined by some combination of the flags in
the previous section. For convenience, the buffer protocol provides frequently
used combinations as single flags.

In the following table @emph{U} stands for undefined contiguity. The consumer would
have to call @ref{1e2,,PyBuffer_IsContiguous()} to determine contiguity.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} {xxxxxxxxxxx} {xxxxxxxxxxxxxx} {xxxxxxxxxx} {xxxxxxxxxxxx} {xxxxxxxxxx} 
@headitem

Request

@tab

shape

@tab

strides

@tab

suboffsets

@tab

contig

@tab

readonly

@tab

format

@item

@geindex PyBUF_FULL (C macro)
@anchor{c-api/buffer c PyBUF_FULL}@anchor{437b}
@deffn {C Macro} PyBUF_FULL
@end deffn

@tab

yes

@tab

yes

@tab

if needed

@tab

U

@tab

0

@tab

yes

@item

@geindex PyBUF_FULL_RO (C macro)
@anchor{c-api/buffer c PyBUF_FULL_RO}@anchor{437c}
@deffn {C Macro} PyBUF_FULL_RO
@end deffn

@tab

yes

@tab

yes

@tab

if needed

@tab

U

@tab

1 or 0

@tab

yes

@item

@geindex PyBUF_RECORDS (C macro)
@anchor{c-api/buffer c PyBUF_RECORDS}@anchor{437d}
@deffn {C Macro} PyBUF_RECORDS
@end deffn

@tab

yes

@tab

yes

@tab

NULL

@tab

U

@tab

0

@tab

yes

@item

@geindex PyBUF_RECORDS_RO (C macro)
@anchor{c-api/buffer c PyBUF_RECORDS_RO}@anchor{437e}
@deffn {C Macro} PyBUF_RECORDS_RO
@end deffn

@tab

yes

@tab

yes

@tab

NULL

@tab

U

@tab

1 or 0

@tab

yes

@item

@geindex PyBUF_STRIDED (C macro)
@anchor{c-api/buffer c PyBUF_STRIDED}@anchor{437f}
@deffn {C Macro} PyBUF_STRIDED
@end deffn

@tab

yes

@tab

yes

@tab

NULL

@tab

U

@tab

0

@tab

NULL

@item

@geindex PyBUF_STRIDED_RO (C macro)
@anchor{c-api/buffer c PyBUF_STRIDED_RO}@anchor{4380}
@deffn {C Macro} PyBUF_STRIDED_RO
@end deffn

@tab

yes

@tab

yes

@tab

NULL

@tab

U

@tab

1 or 0

@tab

NULL

@item

@geindex PyBUF_CONTIG (C macro)
@anchor{c-api/buffer c PyBUF_CONTIG}@anchor{4381}
@deffn {C Macro} PyBUF_CONTIG
@end deffn

@tab

yes

@tab

NULL

@tab

NULL

@tab

C

@tab

0

@tab

NULL

@item

@geindex PyBUF_CONTIG_RO (C macro)
@anchor{c-api/buffer c PyBUF_CONTIG_RO}@anchor{4382}
@deffn {C Macro} PyBUF_CONTIG_RO
@end deffn

@tab

yes

@tab

NULL

@tab

NULL

@tab

C

@tab

1 or 0

@tab

NULL

@end multitable


@node Complex arrays,Buffer-related functions,Buffer request types,Buffer Protocol
@anchor{c-api/buffer complex-arrays}@anchor{436c}
@subsubsection Complex arrays


@menu
* NumPy-style; shape and strides: NumPy-style shape and strides. 
* PIL-style; shape@comma{} strides and suboffsets: PIL-style shape strides and suboffsets. 

@end menu

@node NumPy-style shape and strides,PIL-style shape strides and suboffsets,,Complex arrays
@anchor{c-api/buffer numpy-style-shape-and-strides}@anchor{4383}
@subsubsection NumPy-style: shape and strides


The logical structure of NumPy-style arrays is defined by @ref{4366,,itemsize},
@ref{436a,,ndim}, @ref{4369,,shape} and @ref{4360,,strides}.

If @code{ndim == 0}, the memory location pointed to by @ref{435f,,buf} is
interpreted as a scalar of size @ref{4366,,itemsize}. In that case,
both @ref{4369,,shape} and @ref{4360,,strides} are @code{NULL}.

If @ref{4360,,strides} is @code{NULL}, the array is interpreted as
a standard n-dimensional C-array. Otherwise, the consumer must access an
n-dimensional array as follows:

@example
ptr = (char *)buf + indices[0] * strides[0] + ... + indices[n-1] * strides[n-1];
item = *((typeof(item) *)ptr);
@end example

As noted above, @ref{435f,,buf} can point to any location within
the actual memory block. An exporter can check the validity of a buffer with
this function:

@example
def verify_structure(memlen, itemsize, ndim, shape, strides, offset):
    """Verify that the parameters represent a valid array within
       the bounds of the allocated memory:
           char *mem: start of the physical memory block
           memlen: length of the physical memory block
           offset: (char *)buf - mem
    """
    if offset % itemsize:
        return False
    if offset < 0 or offset+itemsize > memlen:
        return False
    if any(v % itemsize for v in strides):
        return False

    if ndim <= 0:
        return ndim == 0 and not shape and not strides
    if 0 in shape:
        return True

    imin = sum(strides[j]*(shape[j]-1) for j in range(ndim)
               if strides[j] <= 0)
    imax = sum(strides[j]*(shape[j]-1) for j in range(ndim)
               if strides[j] > 0)

    return 0 <= offset+imin and offset+imax+itemsize <= memlen
@end example

@node PIL-style shape strides and suboffsets,,NumPy-style shape and strides,Complex arrays
@anchor{c-api/buffer pil-style-shape-strides-and-suboffsets}@anchor{4384}
@subsubsection PIL-style: shape, strides and suboffsets


In addition to the regular items, PIL-style arrays can contain pointers
that must be followed in order to get to the next element in a dimension.
For example, the regular three-dimensional C-array @code{char v[2][2][3]} can
also be viewed as an array of 2 pointers to 2 two-dimensional arrays:
@code{char (*v[2])[2][3]}. In suboffsets representation, those two pointers
can be embedded at the start of @ref{435f,,buf}, pointing
to two @code{char x[2][3]} arrays that can be located anywhere in memory.

Here is a function that returns a pointer to the element in an N-D array
pointed to by an N-dimensional index when there are both non-@code{NULL} strides
and suboffsets:

@example
void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides,
                       Py_ssize_t *suboffsets, Py_ssize_t *indices) @{
    char *pointer = (char*)buf;
    int i;
    for (i = 0; i < ndim; i++) @{
        pointer += strides[i] * indices[i];
        if (suboffsets[i] >=0 ) @{
            pointer = *((char**)pointer) + suboffsets[i];
        @}
    @}
    return (void*)pointer;
@}
@end example

@node Buffer-related functions,,Complex arrays,Buffer Protocol
@anchor{c-api/buffer buffer-related-functions}@anchor{4385}
@subsubsection Buffer-related functions


@geindex PyObject_CheckBuffer (C function)
@anchor{c-api/buffer c PyObject_CheckBuffer}@anchor{1dc}
@deffn {C Function} int  PyObject_CheckBuffer (PyObject  *obj)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Return @code{1} if @emph{obj} supports the buffer interface otherwise @code{0}.  When @code{1} is
returned, it doesn’t guarantee that @ref{1dd,,PyObject_GetBuffer()} will
succeed.  This function always succeeds.
@end deffn

@geindex PyObject_GetBuffer (C function)
@anchor{c-api/buffer c PyObject_GetBuffer}@anchor{1dd}
@deffn {C Function} int  PyObject_GetBuffer (PyObject  *exporter, Py_buffer  *view, int  flags)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Send a request to @emph{exporter} to fill in @emph{view} as specified by  @emph{flags}.
If the exporter cannot provide a buffer of the exact type, it MUST raise
@code{PyExc_BufferError}, set @code{view->obj} to @code{NULL} and
return @code{-1}.

On success, fill in @emph{view}, set @code{view->obj} to a new reference
to @emph{exporter} and return 0. In the case of chained buffer providers
that redirect requests to a single object, @code{view->obj} MAY
refer to this object instead of @emph{exporter} (See @ref{435b,,Buffer Object Structures}).

Successful calls to @ref{1dd,,PyObject_GetBuffer()} must be paired with calls
to @ref{1e5,,PyBuffer_Release()}, similar to @code{malloc()} and @code{free()}.
Thus, after the consumer is done with the buffer, @ref{1e5,,PyBuffer_Release()}
must be called exactly once.
@end deffn

@geindex PyBuffer_Release (C function)
@anchor{c-api/buffer c PyBuffer_Release}@anchor{1e5}
@deffn {C Function} void  PyBuffer_Release (Py_buffer  *view)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Release the buffer @emph{view} and decrement the reference count for
@code{view->obj}. This function MUST be called when the buffer
is no longer being used, otherwise reference leaks may occur.

It is an error to call this function on a buffer that was not obtained via
@ref{1dd,,PyObject_GetBuffer()}.
@end deffn

@geindex PyBuffer_SizeFromFormat (C function)
@anchor{c-api/buffer c PyBuffer_SizeFromFormat}@anchor{1df}
@deffn {C Function} Py_ssize_t  PyBuffer_SizeFromFormat (const  char  *format)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Return the implied @ref{4366,,itemsize} from @ref{4367,,format}.
On error, raise an exception and return -1.


New in version 3.9.

@end deffn

@geindex PyBuffer_IsContiguous (C function)
@anchor{c-api/buffer c PyBuffer_IsContiguous}@anchor{1e2}
@deffn {C Function} int  PyBuffer_IsContiguous (const  Py_buffer  *view, char  order)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Return @code{1} if the memory defined by the @emph{view} is C-style (@emph{order} is
@code{'C'}) or Fortran-style (@emph{order} is @code{'F'}) @ref{1b0f,,contiguous} or either one
(@emph{order} is @code{'A'}).  Return @code{0} otherwise.  This function always succeeds.
@end deffn

@geindex PyBuffer_GetPointer (C function)
@anchor{c-api/buffer c PyBuffer_GetPointer}@anchor{1de}
@deffn {C Function} void  *PyBuffer_GetPointer (const  Py_buffer  *view, const  Py_ssize_t  *indices)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Get the memory area pointed to by the @emph{indices} inside the given @emph{view}.
@emph{indices} must point to an array of @code{view->ndim} indices.
@end deffn

@geindex PyBuffer_FromContiguous (C function)
@anchor{c-api/buffer c PyBuffer_FromContiguous}@anchor{1e1}
@deffn {C Function} int  PyBuffer_FromContiguous (const  Py_buffer  *view, const  void  *buf, Py_ssize_t  len, char  fort)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Copy contiguous @emph{len} bytes from @emph{buf} to @emph{view}.
@emph{fort} can be @code{'C'} or @code{'F'} (for C-style or Fortran-style ordering).
@code{0} is returned on success, @code{-1} on error.
@end deffn

@geindex PyBuffer_ToContiguous (C function)
@anchor{c-api/buffer c PyBuffer_ToContiguous}@anchor{1e0}
@deffn {C Function} int  PyBuffer_ToContiguous (void  *buf, const  Py_buffer  *src, Py_ssize_t  len, char  order)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Copy @emph{len} bytes from @emph{src} to its contiguous representation in @emph{buf}.
@emph{order} can be @code{'C'} or @code{'F'} or @code{'A'} (for C-style or Fortran-style
ordering or either one). @code{0} is returned on success, @code{-1} on error.

This function fails if @emph{len} != @emph{src->len}.
@end deffn

@geindex PyObject_CopyData (C function)
@anchor{c-api/buffer c PyObject_CopyData}@anchor{4201}
@deffn {C Function} int  PyObject_CopyData (Py_buffer  *dest, Py_buffer  *src)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Copy data from @emph{src} to @emph{dest} buffer. Can convert between C-style and
or Fortran-style buffers.

@code{0} is returned on success, @code{-1} on error.
@end deffn

@geindex PyBuffer_FillContiguousStrides (C function)
@anchor{c-api/buffer c PyBuffer_FillContiguousStrides}@anchor{1e3}
@deffn {C Function} void  PyBuffer_FillContiguousStrides (int  ndims, Py_ssize_t  *shape, Py_ssize_t  *strides, int  itemsize, char  order)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Fill the @emph{strides} array with byte-strides of a @ref{1b0f,,contiguous} (C-style if
@emph{order} is @code{'C'} or Fortran-style if @emph{order} is @code{'F'}) array of the
given shape with the given number of bytes per element.
@end deffn

@geindex PyBuffer_FillInfo (C function)
@anchor{c-api/buffer c PyBuffer_FillInfo}@anchor{1e4}
@deffn {C Function} int  PyBuffer_FillInfo (Py_buffer  *view, PyObject  *exporter, void  *buf, Py_ssize_t  len, int  readonly, int  flags)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Handle buffer requests for an exporter that wants to expose @emph{buf} of size @emph{len}
with writability set according to @emph{readonly}. @emph{buf} is interpreted as a sequence
of unsigned bytes.

The @emph{flags} argument indicates the request type. This function always fills in
@emph{view} as specified by flags, unless @emph{buf} has been designated as read-only
and @ref{4364,,PyBUF_WRITABLE} is set in @emph{flags}.

On success, set @code{view->obj} to a new reference to @emph{exporter} and
return 0. Otherwise, raise @code{PyExc_BufferError}, set
@code{view->obj} to @code{NULL} and return @code{-1};

If this function is used as part of a @ref{435b,,getbufferproc},
@emph{exporter} MUST be set to the exporting object and @emph{flags} must be passed
unmodified. Otherwise, @emph{exporter} MUST be @code{NULL}.
@end deffn

@node Old Buffer Protocol,,Buffer Protocol,Abstract Objects Layer
@anchor{c-api/objbuffer doc}@anchor{4386}@anchor{c-api/objbuffer old-buffer-protocol}@anchor{4387}
@subsection Old Buffer Protocol



Deprecated since version 3.0.


These functions were part of the “old buffer protocol” API in Python 2.
In Python 3, this protocol doesn’t exist anymore but the functions are still
exposed to ease porting 2.x code.  They act as a compatibility wrapper
around the @ref{1a3e,,new buffer protocol}, but they don’t give
you control over the lifetime of the resources acquired when a buffer is
exported.

Therefore, it is recommended that you call @ref{1dd,,PyObject_GetBuffer()}
(or the @code{y*} or @code{w*} @ref{381,,format codes} with the
@ref{37f,,PyArg_ParseTuple()} family of functions) to get a buffer view over
an object, and @ref{1e5,,PyBuffer_Release()} when the buffer view can be released.

@geindex PyObject_AsCharBuffer (C function)
@anchor{c-api/objbuffer c PyObject_AsCharBuffer}@anchor{41fa}
@deffn {C Function} int  PyObject_AsCharBuffer (PyObject  *obj, const  char  **buffer, Py_ssize_t  *buffer_len)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns a pointer to a read-only memory location usable as character-based
input.  The @emph{obj} argument must support the single-segment character buffer
interface.  On success, returns @code{0}, sets @emph{buffer} to the memory location
and @emph{buffer_len} to the buffer length.  Returns @code{-1} and sets a
@ref{162,,TypeError} on error.
@end deffn

@geindex PyObject_AsReadBuffer (C function)
@anchor{c-api/objbuffer c PyObject_AsReadBuffer}@anchor{41fc}
@deffn {C Function} int  PyObject_AsReadBuffer (PyObject  *obj, const  void  **buffer, Py_ssize_t  *buffer_len)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns a pointer to a read-only memory location containing arbitrary data.
The @emph{obj} argument must support the single-segment readable buffer
interface.  On success, returns @code{0}, sets @emph{buffer} to the memory location
and @emph{buffer_len} to the buffer length.  Returns @code{-1} and sets a
@ref{162,,TypeError} on error.
@end deffn

@geindex PyObject_CheckReadBuffer (C function)
@anchor{c-api/objbuffer c PyObject_CheckReadBuffer}@anchor{4200}
@deffn {C Function} int  PyObject_CheckReadBuffer (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns @code{1} if @emph{o} supports the single-segment readable buffer interface.
Otherwise returns @code{0}.  This function always succeeds.

Note that this function tries to get and release a buffer, and exceptions
which occur while calling corresponding functions will get suppressed.
To get error reporting use @ref{1dd,,PyObject_GetBuffer()} instead.
@end deffn

@geindex PyObject_AsWriteBuffer (C function)
@anchor{c-api/objbuffer c PyObject_AsWriteBuffer}@anchor{41fd}
@deffn {C Function} int  PyObject_AsWriteBuffer (PyObject  *obj, void  **buffer, Py_ssize_t  *buffer_len)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns a pointer to a writable memory location.  The @emph{obj} argument must
support the single-segment, character buffer interface.  On success,
returns @code{0}, sets @emph{buffer} to the memory location and @emph{buffer_len} to the
buffer length.  Returns @code{-1} and sets a @ref{162,,TypeError} on error.
@end deffn

@node Concrete Objects Layer,Initialization Finalization and Threads,Abstract Objects Layer,Python/C API Reference Manual
@anchor{c-api/concrete doc}@anchor{4388}@anchor{c-api/concrete concrete}@anchor{4389}@anchor{c-api/concrete concrete-objects-layer}@anchor{438a}
@section Concrete Objects Layer


The functions in this chapter are specific to certain Python object types.
Passing them an object of the wrong type is not a good idea; if you receive an
object from a Python program and you are not sure that it has the right type,
you must perform a type check first; for example, to check that an object is a
dictionary, use @ref{438b,,PyDict_Check()}.  The chapter is structured like the
“family tree” of Python object types.

@cartouche
@quotation Warning 
While the functions described in this chapter carefully check the type of the
objects which are passed in, many of them do not check for @code{NULL} being passed
instead of a valid object.  Allowing @code{NULL} to be passed in can cause memory
access violations and immediate termination of the interpreter.
@end quotation
@end cartouche

@menu
* Fundamental Objects:: 
* Numeric Objects:: 
* Sequence Objects:: 
* Container Objects:: 
* Function Objects: Function Objects<2>. 
* Other Objects:: 

@end menu

@node Fundamental Objects,Numeric Objects,,Concrete Objects Layer
@anchor{c-api/concrete fundamental}@anchor{438c}@anchor{c-api/concrete fundamental-objects}@anchor{438d}
@subsection Fundamental Objects


This section describes Python type objects and the singleton object @code{None}.

@menu
* Type Objects: Type Objects<2>. 
* The None Object:: 

@end menu

@node Type Objects<2>,The None Object,,Fundamental Objects
@anchor{c-api/type doc}@anchor{438e}@anchor{c-api/type type-objects}@anchor{438f}@anchor{c-api/type typeobjects}@anchor{4390}
@subsubsection Type Objects


@geindex object; type

@geindex PyTypeObject (C type)
@anchor{c-api/type c PyTypeObject}@anchor{5c6}
@deffn {C Type} type  PyTypeObject
@emph{ Part of the @ref{35d,,Stable ABI}.}
The C structure of the objects used to describe built-in types.
@end deffn

@geindex PyType_Type (C var)
@anchor{c-api/type c PyType_Type}@anchor{4255}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyType_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
This is the type object for type objects; it is the same object as
@ref{89e,,type} in the Python layer.
@end deffn

@geindex PyType_Check (C function)
@anchor{c-api/type c PyType_Check}@anchor{1234}
@deffn {C Function} int  PyType_Check (PyObject  *o)

Return non-zero if the object @emph{o} is a type object, including instances of
types derived from the standard type object.  Return 0 in all other cases.
This function always succeeds.
@end deffn

@geindex PyType_CheckExact (C function)
@anchor{c-api/type c PyType_CheckExact}@anchor{1235}
@deffn {C Function} int  PyType_CheckExact (PyObject  *o)

Return non-zero if the object @emph{o} is a type object, but not a subtype of
the standard type object.  Return 0 in all other cases.  This function
always succeeds.
@end deffn

@geindex PyType_ClearCache (C function)
@anchor{c-api/type c PyType_ClearCache}@anchor{4250}
@deffn {C Function} unsigned  int  PyType_ClearCache ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Clear the internal lookup cache. Return the current version tag.
@end deffn

@geindex PyType_GetFlags (C function)
@anchor{c-api/type c PyType_GetFlags}@anchor{11dc}
@deffn {C Function} unsigned  long  PyType_GetFlags (PyTypeObject  *type)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the @ref{1133,,tp_flags} member of @emph{type}. This function is primarily
meant for use with @cite{Py_LIMITED_API}; the individual flag bits are
guaranteed to be stable across Python releases, but access to
@ref{1133,,tp_flags} itself is not part of the limited API.


New in version 3.2.



Changed in version 3.4: The return type is now @code{unsigned long} rather than @code{long}.

@end deffn

@geindex PyType_Modified (C function)
@anchor{c-api/type c PyType_Modified}@anchor{4252}
@deffn {C Function} void  PyType_Modified (PyTypeObject  *type)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Invalidate the internal lookup cache for the type and all of its
subtypes.  This function must be called after any manual
modification of the attributes or base classes of the type.
@end deffn

@geindex PyType_HasFeature (C function)
@anchor{c-api/type c PyType_HasFeature}@anchor{11db}
@deffn {C Function} int  PyType_HasFeature (PyTypeObject  *o, int  feature)

Return non-zero if the type object @emph{o} sets the feature @emph{feature}.
Type features are denoted by single bit flags.
@end deffn

@geindex PyType_IS_GC (C function)
@anchor{c-api/type c PyType_IS_GC}@anchor{4391}
@deffn {C Function} int  PyType_IS_GC (PyTypeObject  *o)

Return true if the type object includes support for the cycle detector; this
tests the type flag @ref{1ed,,Py_TPFLAGS_HAVE_GC}.
@end deffn

@geindex PyType_IsSubtype (C function)
@anchor{c-api/type c PyType_IsSubtype}@anchor{4251}
@deffn {C Function} int  PyType_IsSubtype (PyTypeObject  *a, PyTypeObject  *b)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return true if @emph{a} is a subtype of @emph{b}.

This function only checks for actual subtypes, which means that
@ref{17c8,,__subclasscheck__()} is not called on @emph{b}.  Call
@ref{a18,,PyObject_IsSubclass()} to do the same check that @ref{237,,issubclass()}
would do.
@end deffn

@geindex PyType_GenericAlloc (C function)
@anchor{c-api/type c PyType_GenericAlloc}@anchor{582}
@deffn {C Function} @ref{375,,PyObject}  *PyType_GenericAlloc (PyTypeObject  *type, Py_ssize_t  nitems)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Generic handler for the @ref{40ac,,tp_alloc} slot of a type object.  Use
Python’s default memory allocation mechanism to allocate a new instance and
initialize all its contents to @code{NULL}.
@end deffn

@geindex PyType_GenericNew (C function)
@anchor{c-api/type c PyType_GenericNew}@anchor{40a8}
@deffn {C Function} @ref{375,,PyObject}  *PyType_GenericNew (PyTypeObject  *type, PyObject  *args, PyObject  *kwds)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Generic handler for the @ref{40a7,,tp_new} slot of a type object.  Create a
new instance using the type’s @ref{40ac,,tp_alloc} slot.
@end deffn

@geindex PyType_Ready (C function)
@anchor{c-api/type c PyType_Ready}@anchor{1ec}
@deffn {C Function} int  PyType_Ready (PyTypeObject  *type)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Finalize a type object.  This should be called on all type objects to finish
their initialization.  This function is responsible for adding inherited slots
from a type’s base class.  Return @code{0} on success, or return @code{-1} and sets an
exception on error.

@cartouche
@quotation Note 
If some of the base classes implements the GC protocol and the provided
type does not include the @ref{1ed,,Py_TPFLAGS_HAVE_GC} in its flags, then
the GC protocol will be automatically implemented from its parents. On
the contrary, if the type being created does include
@ref{1ed,,Py_TPFLAGS_HAVE_GC} in its flags then it @strong{must} implement the
GC protocol itself by at least implementing the
@ref{1ee,,tp_traverse} handle.
@end quotation
@end cartouche
@end deffn

@geindex PyType_GetName (C function)
@anchor{c-api/type c PyType_GetName}@anchor{1d5}
@deffn {C Function} @ref{375,,PyObject}  *PyType_GetName (PyTypeObject  *type)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Return the type’s name. Equivalent to getting the type’s @code{__name__} attribute.


New in version 3.11.

@end deffn

@geindex PyType_GetQualName (C function)
@anchor{c-api/type c PyType_GetQualName}@anchor{1d6}
@deffn {C Function} @ref{375,,PyObject}  *PyType_GetQualName (PyTypeObject  *type)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Return the type’s qualified name. Equivalent to getting the
type’s @code{__qualname__} attribute.


New in version 3.11.

@end deffn

@geindex PyType_GetSlot (C function)
@anchor{c-api/type c PyType_GetSlot}@anchor{36e}
@deffn {C Function} void  *PyType_GetSlot (PyTypeObject  *type, int  slot)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.4.}
Return the function pointer stored in the given slot. If the
result is @code{NULL}, this indicates that either the slot is @code{NULL},
or that the function was called with invalid parameters.
Callers will typically cast the result pointer into the appropriate
function type.

See @code{PyType_Slot.slot} for possible values of the @emph{slot} argument.


New in version 3.4.



Changed in version 3.10: @ref{36e,,PyType_GetSlot()} can now accept all types.
Previously, it was limited to @ref{468,,heap types}.

@end deffn

@geindex PyType_GetModule (C function)
@anchor{c-api/type c PyType_GetModule}@anchor{474}
@deffn {C Function} @ref{375,,PyObject}  *PyType_GetModule (PyTypeObject  *type)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Return the module object associated with the given type when the type was
created using @ref{36d,,PyType_FromModuleAndSpec()}.

If no module is associated with the given type, sets @ref{162,,TypeError}
and returns @code{NULL}.

This function is usually used to get the module in which a method is defined.
Note that in such a method, @code{PyType_GetModule(Py_TYPE(self))}
may not return the intended result.
@code{Py_TYPE(self)} may be a @emph{subclass} of the intended class, and subclasses
are not necessarily defined in the same module as their superclass.
See @ref{1ea,,PyCMethod} to get the class that defines the method.
See @ref{1e9,,PyType_GetModuleByDef()} for cases when @code{PyCMethod} cannot
be used.


New in version 3.9.

@end deffn

@geindex PyType_GetModuleState (C function)
@anchor{c-api/type c PyType_GetModuleState}@anchor{475}
@deffn {C Function} void  *PyType_GetModuleState (PyTypeObject  *type)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Return the state of the module object associated with the given type.
This is a shortcut for calling @ref{487,,PyModule_GetState()} on the result
of @ref{474,,PyType_GetModule()}.

If no module is associated with the given type, sets @ref{162,,TypeError}
and returns @code{NULL}.

If the @emph{type} has an associated module but its state is @code{NULL},
returns @code{NULL} without setting an exception.


New in version 3.9.

@end deffn

@geindex PyType_GetModuleByDef (C function)
@anchor{c-api/type c PyType_GetModuleByDef}@anchor{1e9}
@deffn {C Function} @ref{375,,PyObject}  *PyType_GetModuleByDef (PyTypeObject  *type, struct  PyModuleDef  *def)

Find the first superclass whose module was created from
the given @ref{484,,PyModuleDef} @emph{def}, and return that module.

If no module is found, raises a @ref{162,,TypeError} and returns @code{NULL}.

This function is intended to be used together with
@ref{487,,PyModule_GetState()} to get module state from slot methods (such as
@ref{40ad,,tp_init} or @ref{4392,,nb_add})
and other places where a method’s defining class cannot be passed using the
@ref{1ea,,PyCMethod} calling convention.


New in version 3.11.

@end deffn

@menu
* Creating Heap-Allocated Types:: 

@end menu

@node Creating Heap-Allocated Types,,,Type Objects<2>
@anchor{c-api/type creating-heap-allocated-types}@anchor{4393}
@subsubsection Creating Heap-Allocated Types


The following functions and structs are used to create
@ref{468,,heap types}.

@geindex PyType_FromModuleAndSpec (C function)
@anchor{c-api/type c PyType_FromModuleAndSpec}@anchor{36d}
@deffn {C Function} @ref{375,,PyObject}  *PyType_FromModuleAndSpec (PyObject  *module, PyType_Spec  *spec, PyObject  *bases)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Creates and returns a @ref{468,,heap type} from the @emph{spec}
(@ref{4394,,Py_TPFLAGS_HEAPTYPE}).

The @emph{bases} argument can be used to specify base classes; it can either
be only one class or a tuple of classes.
If @emph{bases} is @code{NULL}, the @emph{Py_tp_bases} slot is used instead.
If that also is @code{NULL}, the @emph{Py_tp_base} slot is used instead.
If that also is @code{NULL}, the new type derives from @ref{5a5,,object}.

The @emph{module} argument can be used to record the module in which the new
class is defined. It must be a module object or @code{NULL}.
If not @code{NULL}, the module is associated with the new type and can later be
retrieved with @ref{474,,PyType_GetModule()}.
The associated module is not inherited by subclasses; it must be specified
for each class individually.

This function calls @ref{1ec,,PyType_Ready()} on the new type.


New in version 3.9.



Changed in version 3.10: The function now accepts a single class as the @emph{bases} argument and
@code{NULL} as the @code{tp_doc} slot.

@end deffn

@geindex PyType_FromSpecWithBases (C function)
@anchor{c-api/type c PyType_FromSpecWithBases}@anchor{36c}
@deffn {C Function} @ref{375,,PyObject}  *PyType_FromSpecWithBases (PyType_Spec  *spec, PyObject  *bases)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.3.}
Equivalent to @code{PyType_FromModuleAndSpec(NULL, spec, bases)}.


New in version 3.3.

@end deffn

@geindex PyType_FromSpec (C function)
@anchor{c-api/type c PyType_FromSpec}@anchor{469}
@deffn {C Function} @ref{375,,PyObject}  *PyType_FromSpec (PyType_Spec  *spec)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Equivalent to @code{PyType_FromSpecWithBases(spec, NULL)}.
@end deffn

@geindex PyType_Spec (C type)
@anchor{c-api/type c PyType_Spec}@anchor{4254}
@deffn {C Type} type  PyType_Spec
@emph{ Part of the @ref{35d,,Stable ABI}.}
Structure defining a type’s behavior.

@geindex PyType_Spec.PyType_Spec.name (C member)
@anchor{c-api/type c PyType_Spec PyType_Spec name}@anchor{4395}
@deffn {C Member} const  char  *PyType_Spec.name

Name of the type, used to set @ref{116b,,PyTypeObject.tp_name}.
@end deffn

@geindex PyType_Spec.PyType_Spec.basicsize (C member)
@anchor{c-api/type c PyType_Spec PyType_Spec basicsize}@anchor{4396}
@deffn {C Member} int  PyType_Spec.basicsize
@end deffn

@geindex PyType_Spec.PyType_Spec.itemsize (C member)
@anchor{c-api/type c PyType_Spec PyType_Spec itemsize}@anchor{4397}
@deffn {C Member} int  PyType_Spec.itemsize

Size of the instance in bytes, used to set
@ref{493,,PyTypeObject.tp_basicsize} and
@ref{40a4,,PyTypeObject.tp_itemsize}.
@end deffn

@geindex PyType_Spec.PyType_Spec.flags (C member)
@anchor{c-api/type c PyType_Spec PyType_Spec flags}@anchor{4398}
@deffn {C Member} int  PyType_Spec.flags

Type flags, used to set @ref{1133,,PyTypeObject.tp_flags}.

If the @code{Py_TPFLAGS_HEAPTYPE} flag is not set,
@ref{36c,,PyType_FromSpecWithBases()} sets it automatically.
@end deffn

@geindex PyType_Spec.PyType_Spec.slots (C member)
@anchor{c-api/type c PyType_Spec PyType_Spec slots}@anchor{4399}
@deffn {C Member} @ref{4253,,PyType_Slot}  *PyType_Spec.slots

Array of @ref{4253,,PyType_Slot} structures.
Terminated by the special slot value @code{@{0, NULL@}}.
@end deffn
@end deffn

@geindex PyType_Slot (C type)
@anchor{c-api/type c PyType_Slot}@anchor{4253}
@deffn {C Type} type  PyType_Slot
@emph{ Part of the @ref{35d,,Stable ABI}.}
Structure defining optional functionality of a type, containing a slot ID
and a value pointer.

@geindex PyType_Slot.PyType_Slot.slot (C member)
@anchor{c-api/type c PyType_Slot PyType_Slot slot}@anchor{439a}
@deffn {C Member} int  PyType_Slot.slot

@quotation

A slot ID.

Slot IDs are named like the field names of the structures
@ref{5c6,,PyTypeObject}, @ref{f7b,,PyNumberMethods},
@ref{40cf,,PySequenceMethods}, @ref{40d0,,PyMappingMethods} and
@ref{439b,,PyAsyncMethods} with an added @code{Py_} prefix.
For example, use:


@itemize *

@item 
@code{Py_tp_dealloc} to set @ref{40aa,,PyTypeObject.tp_dealloc}

@item 
@code{Py_nb_add} to set @ref{4392,,PyNumberMethods.nb_add}

@item 
@code{Py_sq_length} to set @ref{439c,,PySequenceMethods.sq_length}
@end itemize

The following fields cannot be set at all using @ref{4254,,PyType_Spec} and
@code{PyType_Slot}:


@itemize *

@item 
@ref{439d,,tp_dict}

@item 
@ref{439e,,tp_mro}

@item 
@ref{439f,,tp_cache}

@item 
@ref{43a0,,tp_subclasses}

@item 
@ref{43a1,,tp_weaklist}

@item 
@ref{43a2,,tp_vectorcall}

@item 
@ref{494,,tp_weaklistoffset}
(see @ref{43a3,,PyMemberDef})

@item 
@ref{43a4,,tp_dictoffset}
(see @ref{43a3,,PyMemberDef})

@item 
@ref{4337,,tp_vectorcall_offset}
(see @ref{43a3,,PyMemberDef})
@end itemize

Setting @code{Py_tp_bases} or @code{Py_tp_base} may be
problematic on some platforms.
To avoid issues, use the @emph{bases} argument of
@code{PyType_FromSpecWithBases()} instead.
@end quotation


Changed in version 3.9: Slots in @ref{43a5,,PyBufferProcs} in may be set in the unlimited API.



Changed in version 3.11: @ref{1e7,,bf_getbuffer} and
@ref{1e8,,bf_releasebuffer} are now available
under limited API.

@end deffn

@geindex PyType_Slot.PyType_Slot.pfunc (C member)
@anchor{c-api/type c PyType_Slot PyType_Slot pfunc}@anchor{43a6}
@deffn {C Member} void  *PyType_Slot.pfunc

The desired value of the slot. In most cases, this is a pointer
to a function.

Slots other than @code{Py_tp_doc} may not be @code{NULL}.
@end deffn
@end deffn

@node The None Object,,Type Objects<2>,Fundamental Objects
@anchor{c-api/none doc}@anchor{43a7}@anchor{c-api/none noneobject}@anchor{43a8}@anchor{c-api/none the-none-object}@anchor{43a9}
@subsubsection The @code{None} Object


@geindex object; None

Note that the @ref{5c6,,PyTypeObject} for @code{None} is not directly exposed in the
Python/C API.  Since @code{None} is a singleton, testing for object identity (using
@code{==} in C) is sufficient. There is no @code{PyNone_Check()} function for the
same reason.

@geindex Py_None (C var)
@anchor{c-api/none c Py_None}@anchor{4077}
@deffn {C Variable} @ref{375,,PyObject}  *Py_None

The Python @code{None} object, denoting lack of value.  This object has no methods.
It needs to be treated just like any other object with respect to reference
counts.
@end deffn

@geindex Py_RETURN_NONE (C macro)
@anchor{c-api/none c Py_RETURN_NONE}@anchor{fcf}
@deffn {C Macro} Py_RETURN_NONE

Properly handle returning @ref{4077,,Py_None} from within a C function (that is,
increment the reference count of @code{None} and return it.)
@end deffn

@node Numeric Objects,Sequence Objects,Fundamental Objects,Concrete Objects Layer
@anchor{c-api/concrete numeric-objects}@anchor{43aa}@anchor{c-api/concrete numericobjects}@anchor{43ab}
@subsection Numeric Objects


@geindex object; numeric

@menu
* Integer Objects:: 
* Boolean Objects:: 
* Floating Point Objects:: 
* Complex Number Objects:: 

@end menu

@node Integer Objects,Boolean Objects,,Numeric Objects
@anchor{c-api/long doc}@anchor{43ac}@anchor{c-api/long integer-objects}@anchor{43ad}@anchor{c-api/long longobjects}@anchor{43ae}
@subsubsection Integer Objects


@geindex object; long integer
@geindex object; integer

All integers are implemented as “long” integer objects of arbitrary size.

On error, most @code{PyLong_As*} APIs return @code{(return type)-1} which cannot be
distinguished from a number.  Use @ref{116a,,PyErr_Occurred()} to disambiguate.

@geindex PyLongObject (C type)
@anchor{c-api/long c PyLongObject}@anchor{41b3}
@deffn {C Type} type  PyLongObject
@emph{ Part of the @ref{35d,,Stable ABI}.}
This subtype of @ref{375,,PyObject} represents a Python integer object.
@end deffn

@geindex PyLong_Type (C var)
@anchor{c-api/long c PyLong_Type}@anchor{41be}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyLong_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
This instance of @ref{5c6,,PyTypeObject} represents the Python integer type.
This is the same object as @ref{1b1,,int} in the Python layer.
@end deffn

@geindex PyLong_Check (C function)
@anchor{c-api/long c PyLong_Check}@anchor{43af}
@deffn {C Function} int  PyLong_Check (PyObject  *p)

Return true if its argument is a @ref{41b3,,PyLongObject} or a subtype of
@ref{41b3,,PyLongObject}.  This function always succeeds.
@end deffn

@geindex PyLong_CheckExact (C function)
@anchor{c-api/long c PyLong_CheckExact}@anchor{43b0}
@deffn {C Function} int  PyLong_CheckExact (PyObject  *p)

Return true if its argument is a @ref{41b3,,PyLongObject}, but not a subtype of
@ref{41b3,,PyLongObject}.  This function always succeeds.
@end deffn

@geindex PyLong_FromLong (C function)
@anchor{c-api/long c PyLong_FromLong}@anchor{1085}
@deffn {C Function} @ref{375,,PyObject}  *PyLong_FromLong (long  v)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new @ref{41b3,,PyLongObject} object from @emph{v}, or @code{NULL} on failure.

The current implementation keeps an array of integer objects for all integers
between @code{-5} and @code{256}, when you create an int in that range you actually
just get back a reference to the existing object.
@end deffn

@geindex PyLong_FromUnsignedLong (C function)
@anchor{c-api/long c PyLong_FromUnsignedLong}@anchor{125b}
@deffn {C Function} @ref{375,,PyObject}  *PyLong_FromUnsignedLong (unsigned  long  v)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new @ref{41b3,,PyLongObject} object from a C unsigned  long, or
@code{NULL} on failure.
@end deffn

@geindex PyLong_FromSsize_t (C function)
@anchor{c-api/long c PyLong_FromSsize_t}@anchor{41bb}
@deffn {C Function} @ref{375,,PyObject}  *PyLong_FromSsize_t (Py_ssize_t  v)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new @ref{41b3,,PyLongObject} object from a C @code{Py_ssize_t}, or
@code{NULL} on failure.
@end deffn

@geindex PyLong_FromSize_t (C function)
@anchor{c-api/long c PyLong_FromSize_t}@anchor{125d}
@deffn {C Function} @ref{375,,PyObject}  *PyLong_FromSize_t (size_t  v)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new @ref{41b3,,PyLongObject} object from a C @code{size_t}, or
@code{NULL} on failure.
@end deffn

@geindex PyLong_FromLongLong (C function)
@anchor{c-api/long c PyLong_FromLongLong}@anchor{1086}
@deffn {C Function} @ref{375,,PyObject}  *PyLong_FromLongLong (long  long  v)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new @ref{41b3,,PyLongObject} object from a C long  long, or @code{NULL}
on failure.
@end deffn

@geindex PyLong_FromUnsignedLongLong (C function)
@anchor{c-api/long c PyLong_FromUnsignedLongLong}@anchor{125c}
@deffn {C Function} @ref{375,,PyObject}  *PyLong_FromUnsignedLongLong (unsigned  long  long  v)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new @ref{41b3,,PyLongObject} object from a C unsigned  long  long,
or @code{NULL} on failure.
@end deffn

@geindex PyLong_FromDouble (C function)
@anchor{c-api/long c PyLong_FromDouble}@anchor{437}
@deffn {C Function} @ref{375,,PyObject}  *PyLong_FromDouble (double  v)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new @ref{41b3,,PyLongObject} object from the integer part of @emph{v}, or
@code{NULL} on failure.
@end deffn

@geindex PyLong_FromString (C function)
@anchor{c-api/long c PyLong_FromString}@anchor{41bc}
@deffn {C Function} @ref{375,,PyObject}  *PyLong_FromString (const  char  *str, char  **pend, int  base)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new @ref{41b3,,PyLongObject} based on the string value in @emph{str}, which
is interpreted according to the radix in @emph{base}.  If @emph{pend} is non-@code{NULL},
@emph{*pend} will point to the first character in @emph{str} which follows the
representation of the number.  If @emph{base} is @code{0}, @emph{str} is interpreted using
the @ref{1757,,Integer literals} definition; in this case, leading zeros in a
non-zero decimal number raises a @ref{17c,,ValueError}. If @emph{base} is not @code{0},
it must be between @code{2} and @code{36}, inclusive.  Leading spaces and single
underscores after a base specifier and between digits are ignored.  If there
are no digits, @ref{17c,,ValueError} will be raised.
@end deffn

@geindex PyLong_FromUnicodeObject (C function)
@anchor{c-api/long c PyLong_FromUnicodeObject}@anchor{39d}
@deffn {C Function} @ref{375,,PyObject}  *PyLong_FromUnicodeObject (PyObject  *u, int  base)
@emph{Return value: New reference.}
Convert a sequence of Unicode digits in the string @emph{u} to a Python integer
value.


New in version 3.3.

@end deffn

@geindex PyLong_FromVoidPtr (C function)
@anchor{c-api/long c PyLong_FromVoidPtr}@anchor{41bd}
@deffn {C Function} @ref{375,,PyObject}  *PyLong_FromVoidPtr (void  *p)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a Python integer from the pointer @emph{p}. The pointer value can be
retrieved from the resulting value using @ref{41ba,,PyLong_AsVoidPtr()}.
@end deffn

@c XXX alias PyLong_AS_LONG (for now)

@geindex PyLong_AsLong (C function)
@anchor{c-api/long c PyLong_AsLong}@anchor{57c}
@deffn {C Function} long  PyLong_AsLong (PyObject  *obj)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex LONG_MAX
@geindex OverflowError (built-in exception)

Return a C long representation of @emph{obj}.  If @emph{obj} is not an
instance of @ref{41b3,,PyLongObject}, first call its @code{__index__()} method
(if present) to convert it to a @ref{41b3,,PyLongObject}.

Raise @ref{347,,OverflowError} if the value of @emph{obj} is out of range for a
long.

Returns @code{-1} on error.  Use @ref{116a,,PyErr_Occurred()} to disambiguate.


Changed in version 3.8: Use @code{__index__()} if available.



Changed in version 3.10: This function will no longer use @code{__int__()}.

@end deffn

@geindex PyLong_AsLongAndOverflow (C function)
@anchor{c-api/long c PyLong_AsLongAndOverflow}@anchor{e17}
@deffn {C Function} long  PyLong_AsLongAndOverflow (PyObject  *obj, int  *overflow)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a C long representation of @emph{obj}.  If @emph{obj} is not an
instance of @ref{41b3,,PyLongObject}, first call its @code{__index__()}
method (if present) to convert it to a @ref{41b3,,PyLongObject}.

If the value of @emph{obj} is greater than @code{LONG_MAX} or less than
@code{LONG_MIN}, set @emph{*overflow} to @code{1} or @code{-1}, respectively, and
return @code{-1}; otherwise, set @emph{*overflow} to @code{0}.  If any other exception
occurs set @emph{*overflow} to @code{0} and return @code{-1} as usual.

Returns @code{-1} on error.  Use @ref{116a,,PyErr_Occurred()} to disambiguate.


Changed in version 3.8: Use @code{__index__()} if available.



Changed in version 3.10: This function will no longer use @code{__int__()}.

@end deffn

@geindex PyLong_AsLongLong (C function)
@anchor{c-api/long c PyLong_AsLongLong}@anchor{41b5}
@deffn {C Function} long  long  PyLong_AsLongLong (PyObject  *obj)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex OverflowError (built-in exception)

Return a C long  long representation of @emph{obj}.  If @emph{obj} is not an
instance of @ref{41b3,,PyLongObject}, first call its @code{__index__()} method
(if present) to convert it to a @ref{41b3,,PyLongObject}.

Raise @ref{347,,OverflowError} if the value of @emph{obj} is out of range for a
long  long.

Returns @code{-1} on error.  Use @ref{116a,,PyErr_Occurred()} to disambiguate.


Changed in version 3.8: Use @code{__index__()} if available.



Changed in version 3.10: This function will no longer use @code{__int__()}.

@end deffn

@geindex PyLong_AsLongLongAndOverflow (C function)
@anchor{c-api/long c PyLong_AsLongLongAndOverflow}@anchor{e16}
@deffn {C Function} long  long  PyLong_AsLongLongAndOverflow (PyObject  *obj, int  *overflow)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a C long  long representation of @emph{obj}.  If @emph{obj} is not an
instance of @ref{41b3,,PyLongObject}, first call its @code{__index__()} method
(if present) to convert it to a @ref{41b3,,PyLongObject}.

If the value of @emph{obj} is greater than @code{LLONG_MAX} or less than
@code{LLONG_MIN}, set @emph{*overflow} to @code{1} or @code{-1}, respectively,
and return @code{-1}; otherwise, set @emph{*overflow} to @code{0}.  If any other
exception occurs set @emph{*overflow} to @code{0} and return @code{-1} as usual.

Returns @code{-1} on error.  Use @ref{116a,,PyErr_Occurred()} to disambiguate.


New in version 3.2.



Changed in version 3.8: Use @code{__index__()} if available.



Changed in version 3.10: This function will no longer use @code{__int__()}.

@end deffn

@geindex PyLong_AsSsize_t (C function)
@anchor{c-api/long c PyLong_AsSsize_t}@anchor{41b7}
@deffn {C Function} Py_ssize_t  PyLong_AsSsize_t (PyObject  *pylong)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex PY_SSIZE_T_MAX
@geindex OverflowError (built-in exception)

Return a C @code{Py_ssize_t} representation of @emph{pylong}.  @emph{pylong} must
be an instance of @ref{41b3,,PyLongObject}.

Raise @ref{347,,OverflowError} if the value of @emph{pylong} is out of range for a
@code{Py_ssize_t}.

Returns @code{-1} on error.  Use @ref{116a,,PyErr_Occurred()} to disambiguate.
@end deffn

@geindex PyLong_AsUnsignedLong (C function)
@anchor{c-api/long c PyLong_AsUnsignedLong}@anchor{41b8}
@deffn {C Function} unsigned  long  PyLong_AsUnsignedLong (PyObject  *pylong)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex ULONG_MAX
@geindex OverflowError (built-in exception)

Return a C unsigned  long representation of @emph{pylong}.  @emph{pylong}
must be an instance of @ref{41b3,,PyLongObject}.

Raise @ref{347,,OverflowError} if the value of @emph{pylong} is out of range for a
unsigned  long.

Returns @code{(unsigned long)-1} on error.
Use @ref{116a,,PyErr_Occurred()} to disambiguate.
@end deffn

@geindex PyLong_AsSize_t (C function)
@anchor{c-api/long c PyLong_AsSize_t}@anchor{41b6}
@deffn {C Function} size_t  PyLong_AsSize_t (PyObject  *pylong)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex SIZE_MAX
@geindex OverflowError (built-in exception)

Return a C @code{size_t} representation of @emph{pylong}.  @emph{pylong} must be
an instance of @ref{41b3,,PyLongObject}.

Raise @ref{347,,OverflowError} if the value of @emph{pylong} is out of range for a
@code{size_t}.

Returns @code{(size_t)-1} on error.
Use @ref{116a,,PyErr_Occurred()} to disambiguate.
@end deffn

@geindex PyLong_AsUnsignedLongLong (C function)
@anchor{c-api/long c PyLong_AsUnsignedLongLong}@anchor{e3a}
@deffn {C Function} unsigned  long  long  PyLong_AsUnsignedLongLong (PyObject  *pylong)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex OverflowError (built-in exception)

Return a C unsigned  long  long representation of @emph{pylong}.  @emph{pylong}
must be an instance of @ref{41b3,,PyLongObject}.

Raise @ref{347,,OverflowError} if the value of @emph{pylong} is out of range for an
unsigned  long  long.

Returns @code{(unsigned long long)-1} on error.
Use @ref{116a,,PyErr_Occurred()} to disambiguate.


Changed in version 3.1: A negative @emph{pylong} now raises @ref{347,,OverflowError}, not @ref{162,,TypeError}.

@end deffn

@geindex PyLong_AsUnsignedLongMask (C function)
@anchor{c-api/long c PyLong_AsUnsignedLongMask}@anchor{41b9}
@deffn {C Function} unsigned  long  PyLong_AsUnsignedLongMask (PyObject  *obj)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a C unsigned  long representation of @emph{obj}.  If @emph{obj} is not
an instance of @ref{41b3,,PyLongObject}, first call its @code{__index__()}
method (if present) to convert it to a @ref{41b3,,PyLongObject}.

If the value of @emph{obj} is out of range for an unsigned  long,
return the reduction of that value modulo @code{ULONG_MAX + 1}.

Returns @code{(unsigned long)-1} on error.  Use @ref{116a,,PyErr_Occurred()} to
disambiguate.


Changed in version 3.8: Use @code{__index__()} if available.



Changed in version 3.10: This function will no longer use @code{__int__()}.

@end deffn

@geindex PyLong_AsUnsignedLongLongMask (C function)
@anchor{c-api/long c PyLong_AsUnsignedLongLongMask}@anchor{1288}
@deffn {C Function} unsigned  long  long  PyLong_AsUnsignedLongLongMask (PyObject  *obj)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a C unsigned  long  long representation of @emph{obj}.  If @emph{obj}
is not an instance of @ref{41b3,,PyLongObject}, first call its
@code{__index__()} method (if present) to convert it to a
@ref{41b3,,PyLongObject}.

If the value of @emph{obj} is out of range for an unsigned  long  long,
return the reduction of that value modulo @code{ULLONG_MAX + 1}.

Returns @code{(unsigned long long)-1} on error.  Use @ref{116a,,PyErr_Occurred()}
to disambiguate.


Changed in version 3.8: Use @code{__index__()} if available.



Changed in version 3.10: This function will no longer use @code{__int__()}.

@end deffn

@geindex PyLong_AsDouble (C function)
@anchor{c-api/long c PyLong_AsDouble}@anchor{41b4}
@deffn {C Function} double  PyLong_AsDouble (PyObject  *pylong)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a C double representation of @emph{pylong}.  @emph{pylong} must be
an instance of @ref{41b3,,PyLongObject}.

Raise @ref{347,,OverflowError} if the value of @emph{pylong} is out of range for a
double.

Returns @code{-1.0} on error.  Use @ref{116a,,PyErr_Occurred()} to disambiguate.
@end deffn

@geindex PyLong_AsVoidPtr (C function)
@anchor{c-api/long c PyLong_AsVoidPtr}@anchor{41ba}
@deffn {C Function} void  *PyLong_AsVoidPtr (PyObject  *pylong)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Convert a Python integer @emph{pylong} to a C void pointer.
If @emph{pylong} cannot be converted, an @ref{347,,OverflowError} will be raised.  This
is only assured to produce a usable void pointer for values created
with @ref{41bd,,PyLong_FromVoidPtr()}.

Returns @code{NULL} on error.  Use @ref{116a,,PyErr_Occurred()} to disambiguate.
@end deffn

@node Boolean Objects,Floating Point Objects,Integer Objects,Numeric Objects
@anchor{c-api/bool doc}@anchor{43b1}@anchor{c-api/bool boolean-objects}@anchor{43b2}@anchor{c-api/bool boolobjects}@anchor{43b3}
@subsubsection Boolean Objects


Booleans in Python are implemented as a subclass of integers.  There are only
two booleans, @code{Py_False} and @code{Py_True}.  As such, the normal
creation and deletion functions don’t apply to booleans.  The following macros
are available, however.

@geindex PyBool_Check (C function)
@anchor{c-api/bool c PyBool_Check}@anchor{43b4}
@deffn {C Function} int  PyBool_Check (PyObject  *o)

Return true if @emph{o} is of type @code{PyBool_Type}.  This function always
succeeds.
@end deffn

@geindex Py_False (C var)
@anchor{c-api/bool c Py_False}@anchor{43b5}
@deffn {C Variable} @ref{375,,PyObject}  *Py_False

The Python @code{False} object.  This object has no methods.  It needs to be
treated just like any other object with respect to reference counts.
@end deffn

@geindex Py_True (C var)
@anchor{c-api/bool c Py_True}@anchor{43b6}
@deffn {C Variable} @ref{375,,PyObject}  *Py_True

The Python @code{True} object.  This object has no methods.  It needs to be treated
just like any other object with respect to reference counts.
@end deffn

@geindex Py_RETURN_FALSE (C macro)
@anchor{c-api/bool c Py_RETURN_FALSE}@anchor{fd1}
@deffn {C Macro} Py_RETURN_FALSE

Return @code{Py_False} from a function, properly incrementing its reference
count.
@end deffn

@geindex Py_RETURN_TRUE (C macro)
@anchor{c-api/bool c Py_RETURN_TRUE}@anchor{fd0}
@deffn {C Macro} Py_RETURN_TRUE

Return @code{Py_True} from a function, properly incrementing its reference
count.
@end deffn

@geindex PyBool_FromLong (C function)
@anchor{c-api/bool c PyBool_FromLong}@anchor{412c}
@deffn {C Function} @ref{375,,PyObject}  *PyBool_FromLong (long  v)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new reference to @code{Py_True} or @code{Py_False} depending on the
truth value of @emph{v}.
@end deffn

@node Floating Point Objects,Complex Number Objects,Boolean Objects,Numeric Objects
@anchor{c-api/float doc}@anchor{43b7}@anchor{c-api/float floating-point-objects}@anchor{43b8}@anchor{c-api/float floatobjects}@anchor{43b9}
@subsubsection Floating Point Objects


@geindex object; floating point

@geindex PyFloatObject (C type)
@anchor{c-api/float c PyFloatObject}@anchor{43ba}
@deffn {C Type} type  PyFloatObject

This subtype of @ref{375,,PyObject} represents a Python floating point object.
@end deffn

@geindex PyFloat_Type (C var)
@anchor{c-api/float c PyFloat_Type}@anchor{4199}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyFloat_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
This instance of @ref{5c6,,PyTypeObject} represents the Python floating point
type.  This is the same object as @ref{256,,float} in the Python layer.
@end deffn

@geindex PyFloat_Check (C function)
@anchor{c-api/float c PyFloat_Check}@anchor{43bb}
@deffn {C Function} int  PyFloat_Check (PyObject  *p)

Return true if its argument is a @ref{43ba,,PyFloatObject} or a subtype of
@ref{43ba,,PyFloatObject}.  This function always succeeds.
@end deffn

@geindex PyFloat_CheckExact (C function)
@anchor{c-api/float c PyFloat_CheckExact}@anchor{43bc}
@deffn {C Function} int  PyFloat_CheckExact (PyObject  *p)

Return true if its argument is a @ref{43ba,,PyFloatObject}, but not a subtype of
@ref{43ba,,PyFloatObject}.  This function always succeeds.
@end deffn

@geindex PyFloat_FromString (C function)
@anchor{c-api/float c PyFloat_FromString}@anchor{4198}
@deffn {C Function} @ref{375,,PyObject}  *PyFloat_FromString (PyObject  *str)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a @ref{43ba,,PyFloatObject} object based on the string value in @emph{str}, or
@code{NULL} on failure.
@end deffn

@geindex PyFloat_FromDouble (C function)
@anchor{c-api/float c PyFloat_FromDouble}@anchor{4197}
@deffn {C Function} @ref{375,,PyObject}  *PyFloat_FromDouble (double  v)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a @ref{43ba,,PyFloatObject} object from @emph{v}, or @code{NULL} on failure.
@end deffn

@geindex PyFloat_AsDouble (C function)
@anchor{c-api/float c PyFloat_AsDouble}@anchor{580}
@deffn {C Function} double  PyFloat_AsDouble (PyObject  *pyfloat)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a C double representation of the contents of @emph{pyfloat}.  If
@emph{pyfloat} is not a Python floating point object but has a @code{__float__()}
method, this method will first be called to convert @emph{pyfloat} into a float.
If @code{__float__()} is not defined then it falls back to @code{__index__()}.
This method returns @code{-1.0} upon failure, so one should call
@ref{116a,,PyErr_Occurred()} to check for errors.


Changed in version 3.8: Use @code{__index__()} if available.

@end deffn

@geindex PyFloat_AS_DOUBLE (C function)
@anchor{c-api/float c PyFloat_AS_DOUBLE}@anchor{43bd}
@deffn {C Function} double  PyFloat_AS_DOUBLE (PyObject  *pyfloat)

Return a C double representation of the contents of @emph{pyfloat}, but
without error checking.
@end deffn

@geindex PyFloat_GetInfo (C function)
@anchor{c-api/float c PyFloat_GetInfo}@anchor{f51}
@deffn {C Function} @ref{375,,PyObject}  *PyFloat_GetInfo (void)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a structseq instance which contains information about the
precision, minimum and maximum values of a float. It’s a thin wrapper
around the header file @code{float.h}.
@end deffn

@geindex PyFloat_GetMax (C function)
@anchor{c-api/float c PyFloat_GetMax}@anchor{f4f}
@deffn {C Function} double  PyFloat_GetMax ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the maximum representable finite float @emph{DBL_MAX} as C double.
@end deffn

@geindex PyFloat_GetMin (C function)
@anchor{c-api/float c PyFloat_GetMin}@anchor{f50}
@deffn {C Function} double  PyFloat_GetMin ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the minimum normalized positive float @emph{DBL_MIN} as C double.
@end deffn

@node Complex Number Objects,,Floating Point Objects,Numeric Objects
@anchor{c-api/complex doc}@anchor{43be}@anchor{c-api/complex complex-number-objects}@anchor{43bf}@anchor{c-api/complex complexobjects}@anchor{43c0}
@subsubsection Complex Number Objects


@geindex object; complex number

Python’s complex number objects are implemented as two distinct types when
viewed from the C API:  one is the Python object exposed to Python programs, and
the other is a C structure which represents the actual complex number value.
The API provides functions for working with both.

@menu
* Complex Numbers as C Structures:: 
* Complex Numbers as Python Objects:: 

@end menu

@node Complex Numbers as C Structures,Complex Numbers as Python Objects,,Complex Number Objects
@anchor{c-api/complex complex-numbers-as-c-structures}@anchor{43c1}
@subsubsection Complex Numbers as C Structures


Note that the functions which accept these structures as parameters and return
them as results do so @emph{by value} rather than dereferencing them through
pointers.  This is consistent throughout the API.

@geindex Py_complex (C type)
@anchor{c-api/complex c Py_complex}@anchor{4315}
@deffn {C Type} type  Py_complex

The C structure which corresponds to the value portion of a Python complex
number object.  Most of the functions for dealing with complex number objects
use structures of this type as input or output values, as appropriate.  It is
defined as:

@example
typedef struct @{
   double real;
   double imag;
@} Py_complex;
@end example
@end deffn

@geindex _Py_c_sum (C function)
@anchor{c-api/complex c _Py_c_sum}@anchor{43c2}
@deffn {C Function} @ref{4315,,Py_complex}  _Py_c_sum (Py_complex  left, Py_complex  right)

Return the sum of two complex numbers, using the C @ref{4315,,Py_complex}
representation.
@end deffn

@geindex _Py_c_diff (C function)
@anchor{c-api/complex c _Py_c_diff}@anchor{43c3}
@deffn {C Function} @ref{4315,,Py_complex}  _Py_c_diff (Py_complex  left, Py_complex  right)

Return the difference between two complex numbers, using the C
@ref{4315,,Py_complex} representation.
@end deffn

@geindex _Py_c_neg (C function)
@anchor{c-api/complex c _Py_c_neg}@anchor{43c4}
@deffn {C Function} @ref{4315,,Py_complex}  _Py_c_neg (Py_complex  num)

Return the negation of the complex number @emph{num}, using the C
@ref{4315,,Py_complex} representation.
@end deffn

@geindex _Py_c_prod (C function)
@anchor{c-api/complex c _Py_c_prod}@anchor{43c5}
@deffn {C Function} @ref{4315,,Py_complex}  _Py_c_prod (Py_complex  left, Py_complex  right)

Return the product of two complex numbers, using the C @ref{4315,,Py_complex}
representation.
@end deffn

@geindex _Py_c_quot (C function)
@anchor{c-api/complex c _Py_c_quot}@anchor{43c6}
@deffn {C Function} @ref{4315,,Py_complex}  _Py_c_quot (Py_complex  dividend, Py_complex  divisor)

Return the quotient of two complex numbers, using the C @ref{4315,,Py_complex}
representation.

If @emph{divisor} is null, this method returns zero and sets
@code{errno} to @code{EDOM}.
@end deffn

@geindex _Py_c_pow (C function)
@anchor{c-api/complex c _Py_c_pow}@anchor{43c7}
@deffn {C Function} @ref{4315,,Py_complex}  _Py_c_pow (Py_complex  num, Py_complex  exp)

Return the exponentiation of @emph{num} by @emph{exp}, using the C @ref{4315,,Py_complex}
representation.

If @emph{num} is null and @emph{exp} is not a positive real number,
this method returns zero and sets @code{errno} to @code{EDOM}.
@end deffn

@node Complex Numbers as Python Objects,,Complex Numbers as C Structures,Complex Number Objects
@anchor{c-api/complex complex-numbers-as-python-objects}@anchor{43c8}
@subsubsection Complex Numbers as Python Objects


@geindex PyComplexObject (C type)
@anchor{c-api/complex c PyComplexObject}@anchor{43c9}
@deffn {C Type} type  PyComplexObject

This subtype of @ref{375,,PyObject} represents a Python complex number object.
@end deffn

@geindex PyComplex_Type (C var)
@anchor{c-api/complex c PyComplex_Type}@anchor{415b}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyComplex_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
This instance of @ref{5c6,,PyTypeObject} represents the Python complex number
type. It is the same object as @ref{33b,,complex} in the Python layer.
@end deffn

@geindex PyComplex_Check (C function)
@anchor{c-api/complex c PyComplex_Check}@anchor{43ca}
@deffn {C Function} int  PyComplex_Check (PyObject  *p)

Return true if its argument is a @ref{43c9,,PyComplexObject} or a subtype of
@ref{43c9,,PyComplexObject}.  This function always succeeds.
@end deffn

@geindex PyComplex_CheckExact (C function)
@anchor{c-api/complex c PyComplex_CheckExact}@anchor{43cb}
@deffn {C Function} int  PyComplex_CheckExact (PyObject  *p)

Return true if its argument is a @ref{43c9,,PyComplexObject}, but not a subtype of
@ref{43c9,,PyComplexObject}.  This function always succeeds.
@end deffn

@geindex PyComplex_FromCComplex (C function)
@anchor{c-api/complex c PyComplex_FromCComplex}@anchor{43cc}
@deffn {C Function} @ref{375,,PyObject}  *PyComplex_FromCComplex (Py_complex  v)
@emph{Return value: New reference.}
Create a new Python complex number object from a C @ref{4315,,Py_complex} value.
@end deffn

@geindex PyComplex_FromDoubles (C function)
@anchor{c-api/complex c PyComplex_FromDoubles}@anchor{4158}
@deffn {C Function} @ref{375,,PyObject}  *PyComplex_FromDoubles (double  real, double  imag)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new @ref{43c9,,PyComplexObject} object from @emph{real} and @emph{imag}.
@end deffn

@geindex PyComplex_RealAsDouble (C function)
@anchor{c-api/complex c PyComplex_RealAsDouble}@anchor{415a}
@deffn {C Function} double  PyComplex_RealAsDouble (PyObject  *op)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the real part of @emph{op} as a C double.
@end deffn

@geindex PyComplex_ImagAsDouble (C function)
@anchor{c-api/complex c PyComplex_ImagAsDouble}@anchor{4159}
@deffn {C Function} double  PyComplex_ImagAsDouble (PyObject  *op)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the imaginary part of @emph{op} as a C double.
@end deffn

@geindex PyComplex_AsCComplex (C function)
@anchor{c-api/complex c PyComplex_AsCComplex}@anchor{f52}
@deffn {C Function} @ref{4315,,Py_complex}  PyComplex_AsCComplex (PyObject  *op)

Return the @ref{4315,,Py_complex} value of the complex number @emph{op}.

If @emph{op} is not a Python complex number object but has a @code{__complex__()}
method, this method will first be called to convert @emph{op} to a Python complex
number object.  If @code{__complex__()} is not defined then it falls back to
@code{__float__()}.  If @code{__float__()} is not defined then it falls back
to @code{__index__()}.  Upon failure, this method returns @code{-1.0} as a real
value.


Changed in version 3.8: Use @code{__index__()} if available.

@end deffn

@node Sequence Objects,Container Objects,Numeric Objects,Concrete Objects Layer
@anchor{c-api/concrete sequence-objects}@anchor{43cd}@anchor{c-api/concrete sequenceobjects}@anchor{43ce}
@subsection Sequence Objects


@geindex object; sequence

Generic operations on sequence objects were discussed in the previous chapter;
this section deals with the specific kinds of sequence objects that are
intrinsic to the Python language.

@c XXX sort out unicode, str, bytes and bytearray

@menu
* Bytes Objects: Bytes Objects<2>. 
* Byte Array Objects:: 
* Unicode Objects and Codecs:: 
* Tuple Objects:: 
* Struct Sequence Objects:: 
* List Objects:: 

@end menu

@node Bytes Objects<2>,Byte Array Objects,,Sequence Objects
@anchor{c-api/bytes doc}@anchor{43cf}@anchor{c-api/bytes bytes-objects}@anchor{43d0}@anchor{c-api/bytes bytesobjects}@anchor{43d1}
@subsubsection Bytes Objects


These functions raise @ref{162,,TypeError} when expecting a bytes parameter and are
called with a non-bytes parameter.

@geindex object; bytes

@geindex PyBytesObject (C type)
@anchor{c-api/bytes c PyBytesObject}@anchor{208}
@deffn {C Type} type  PyBytesObject

This subtype of @ref{375,,PyObject} represents a Python bytes object.
@end deffn

@geindex PyBytes_Type (C var)
@anchor{c-api/bytes c PyBytes_Type}@anchor{4139}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyBytes_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
This instance of @ref{5c6,,PyTypeObject} represents the Python bytes type; it
is the same object as @ref{2f0,,bytes} in the Python layer.
@end deffn

@geindex PyBytes_Check (C function)
@anchor{c-api/bytes c PyBytes_Check}@anchor{f1e}
@deffn {C Function} int  PyBytes_Check (PyObject  *o)

Return true if the object @emph{o} is a bytes object or an instance of a subtype
of the bytes type.  This function always succeeds.
@end deffn

@geindex PyBytes_CheckExact (C function)
@anchor{c-api/bytes c PyBytes_CheckExact}@anchor{43d2}
@deffn {C Function} int  PyBytes_CheckExact (PyObject  *o)

Return true if the object @emph{o} is a bytes object, but not an instance of a
subtype of the bytes type.  This function always succeeds.
@end deffn

@geindex PyBytes_FromString (C function)
@anchor{c-api/bytes c PyBytes_FromString}@anchor{4137}
@deffn {C Function} @ref{375,,PyObject}  *PyBytes_FromString (const  char  *v)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new bytes object with a copy of the string @emph{v} as value on success,
and @code{NULL} on failure.  The parameter @emph{v} must not be @code{NULL}; it will not be
checked.
@end deffn

@geindex PyBytes_FromStringAndSize (C function)
@anchor{c-api/bytes c PyBytes_FromStringAndSize}@anchor{f1f}
@deffn {C Function} @ref{375,,PyObject}  *PyBytes_FromStringAndSize (const  char  *v, Py_ssize_t  len)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new bytes object with a copy of the string @emph{v} as value and length
@emph{len} on success, and @code{NULL} on failure.  If @emph{v} is @code{NULL}, the contents of
the bytes object are uninitialized.
@end deffn

@geindex PyBytes_FromFormat (C function)
@anchor{c-api/bytes c PyBytes_FromFormat}@anchor{12d5}
@deffn {C Function} @ref{375,,PyObject}  *PyBytes_FromFormat (const  char  *format, ...)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Take a C @code{printf()}-style @emph{format} string and a variable number of
arguments, calculate the size of the resulting Python bytes object and return
a bytes object with the values formatted into it.  The variable arguments
must be C types and must correspond exactly to the format characters in the
@emph{format} string.  The following format characters are allowed:

@c % XXX: This should be exactly the same as the table in PyErr_Format.

@c % One should just refer to the other.

@c % XXX: The descriptions for %zd and %zu are wrong, but the truth is complicated

@c % because not all compilers support the %z width modifier -- we fake it

@c % when necessary via interpolating PY_FORMAT_SIZE_T.


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Format Characters

@tab

Type

@tab

Comment

@item

@code{%%}

@tab

@emph{n/a}

@tab

The literal % character.

@item

@code{%c}

@tab

int

@tab

A single byte,
represented as a C int.

@item

@code{%d}

@tab

int

@tab

Equivalent to
@code{printf("%d")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, lu, zd, zu, i, x): the 0-conversion
flag has effect even when a precision is given.
}

@item

@code{%u}

@tab

unsigned int

@tab

Equivalent to
@code{printf("%u")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, lu, zd, zu, i, x): the 0-conversion
flag has effect even when a precision is given.
}

@item

@code{%ld}

@tab

long

@tab

Equivalent to
@code{printf("%ld")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, lu, zd, zu, i, x): the 0-conversion
flag has effect even when a precision is given.
}

@item

@code{%lu}

@tab

unsigned long

@tab

Equivalent to
@code{printf("%lu")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, lu, zd, zu, i, x): the 0-conversion
flag has effect even when a precision is given.
}

@item

@code{%zd}

@tab

Py_ssize_t

@tab

Equivalent to
@code{printf("%zd")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, lu, zd, zu, i, x): the 0-conversion
flag has effect even when a precision is given.
}

@item

@code{%zu}

@tab

size_t

@tab

Equivalent to
@code{printf("%zu")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, lu, zd, zu, i, x): the 0-conversion
flag has effect even when a precision is given.
}

@item

@code{%i}

@tab

int

@tab

Equivalent to
@code{printf("%i")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, lu, zd, zu, i, x): the 0-conversion
flag has effect even when a precision is given.
}

@item

@code{%x}

@tab

int

@tab

Equivalent to
@code{printf("%x")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, lu, zd, zu, i, x): the 0-conversion
flag has effect even when a precision is given.
}

@item

@code{%s}

@tab

const char*

@tab

A null-terminated C character
array.

@item

@code{%p}

@tab

const void*

@tab

The hex representation of a C
pointer. Mostly equivalent to
@code{printf("%p")} except that
it is guaranteed to start with
the literal @code{0x} regardless
of what the platform’s
@code{printf} yields.

@end multitable


An unrecognized format character causes all the rest of the format string to be
copied as-is to the result object, and any extra arguments discarded.
@end deffn

@geindex PyBytes_FromFormatV (C function)
@anchor{c-api/bytes c PyBytes_FromFormatV}@anchor{4135}
@deffn {C Function} @ref{375,,PyObject}  *PyBytes_FromFormatV (const  char  *format, va_list  vargs)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Identical to @ref{12d5,,PyBytes_FromFormat()} except that it takes exactly two
arguments.
@end deffn

@geindex PyBytes_FromObject (C function)
@anchor{c-api/bytes c PyBytes_FromObject}@anchor{4136}
@deffn {C Function} @ref{375,,PyObject}  *PyBytes_FromObject (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the bytes representation of object @emph{o} that implements the buffer
protocol.
@end deffn

@geindex PyBytes_Size (C function)
@anchor{c-api/bytes c PyBytes_Size}@anchor{4138}
@deffn {C Function} Py_ssize_t  PyBytes_Size (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the length of the bytes in bytes object @emph{o}.
@end deffn

@geindex PyBytes_GET_SIZE (C function)
@anchor{c-api/bytes c PyBytes_GET_SIZE}@anchor{43d3}
@deffn {C Function} Py_ssize_t  PyBytes_GET_SIZE (PyObject  *o)

Macro form of @ref{4138,,PyBytes_Size()} but without error checking.
@end deffn

@geindex PyBytes_AsString (C function)
@anchor{c-api/bytes c PyBytes_AsString}@anchor{351}
@deffn {C Function} char  *PyBytes_AsString (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a pointer to the contents of @emph{o}.  The pointer
refers to the internal buffer of @emph{o}, which consists of @code{len(o) + 1}
bytes.  The last byte in the buffer is always null, regardless of
whether there are any other null bytes.  The data must not be
modified in any way, unless the object was just created using
@code{PyBytes_FromStringAndSize(NULL, size)}. It must not be deallocated.  If
@emph{o} is not a bytes object at all, @ref{351,,PyBytes_AsString()} returns @code{NULL}
and raises @ref{162,,TypeError}.
@end deffn

@geindex PyBytes_AS_STRING (C function)
@anchor{c-api/bytes c PyBytes_AS_STRING}@anchor{43d4}
@deffn {C Function} char  *PyBytes_AS_STRING (PyObject  *string)

Macro form of @ref{351,,PyBytes_AsString()} but without error checking.
@end deffn

@geindex PyBytes_AsStringAndSize (C function)
@anchor{c-api/bytes c PyBytes_AsStringAndSize}@anchor{4132}
@deffn {C Function} int  PyBytes_AsStringAndSize (PyObject  *obj, char  **buffer, Py_ssize_t  *length)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the null-terminated contents of the object @emph{obj}
through the output variables @emph{buffer} and @emph{length}.

If @emph{length} is @code{NULL}, the bytes object
may not contain embedded null bytes;
if it does, the function returns @code{-1} and a @ref{17c,,ValueError} is raised.

The buffer refers to an internal buffer of @emph{obj}, which includes an
additional null byte at the end (not counted in @emph{length}).  The data
must not be modified in any way, unless the object was just created using
@code{PyBytes_FromStringAndSize(NULL, size)}.  It must not be deallocated.  If
@emph{obj} is not a bytes object at all, @ref{4132,,PyBytes_AsStringAndSize()}
returns @code{-1} and raises @ref{162,,TypeError}.


Changed in version 3.5: Previously, @ref{162,,TypeError} was raised when embedded null bytes were
encountered in the bytes object.

@end deffn

@geindex PyBytes_Concat (C function)
@anchor{c-api/bytes c PyBytes_Concat}@anchor{4133}
@deffn {C Function} void  PyBytes_Concat (PyObject  **bytes, PyObject  *newpart)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a new bytes object in @emph{*bytes} containing the contents of @emph{newpart}
appended to @emph{bytes}; the caller will own the new reference.  The reference to
the old value of @emph{bytes} will be stolen.  If the new object cannot be
created, the old reference to @emph{bytes} will still be discarded and the value
of @emph{*bytes} will be set to @code{NULL}; the appropriate exception will be set.
@end deffn

@geindex PyBytes_ConcatAndDel (C function)
@anchor{c-api/bytes c PyBytes_ConcatAndDel}@anchor{4134}
@deffn {C Function} void  PyBytes_ConcatAndDel (PyObject  **bytes, PyObject  *newpart)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a new bytes object in @emph{*bytes} containing the contents of @emph{newpart}
appended to @emph{bytes}.  This version decrements the reference count of
@emph{newpart}.
@end deffn

@geindex _PyBytes_Resize (C function)
@anchor{c-api/bytes c _PyBytes_Resize}@anchor{130d}
@deffn {C Function} int  _PyBytes_Resize (PyObject  **bytes, Py_ssize_t  newsize)

A way to resize a bytes object even though it is “immutable”. Only use this
to build up a brand new bytes object; don’t use this if the bytes may already
be known in other parts of the code.  It is an error to call this function if
the refcount on the input bytes object is not one. Pass the address of an
existing bytes object as an lvalue (it may be written into), and the new size
desired.  On success, @emph{*bytes} holds the resized bytes object and @code{0} is
returned; the address in @emph{*bytes} may differ from its input value.  If the
reallocation fails, the original bytes object at @emph{*bytes} is deallocated,
@emph{*bytes} is set to @code{NULL}, @ref{10de,,MemoryError} is set, and @code{-1} is
returned.
@end deffn

@node Byte Array Objects,Unicode Objects and Codecs,Bytes Objects<2>,Sequence Objects
@anchor{c-api/bytearray doc}@anchor{43d5}@anchor{c-api/bytearray byte-array-objects}@anchor{43d6}@anchor{c-api/bytearray bytearrayobjects}@anchor{43d7}
@subsubsection Byte Array Objects


@geindex object; bytearray

@geindex PyByteArrayObject (C type)
@anchor{c-api/bytearray c PyByteArrayObject}@anchor{43d8}
@deffn {C Type} type  PyByteArrayObject

This subtype of @ref{375,,PyObject} represents a Python bytearray object.
@end deffn

@geindex PyByteArray_Type (C var)
@anchor{c-api/bytearray c PyByteArray_Type}@anchor{4131}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyByteArray_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
This instance of @ref{5c6,,PyTypeObject} represents the Python bytearray type;
it is the same object as @ref{2f1,,bytearray} in the Python layer.
@end deffn

@menu
* Type check macros:: 
* Direct API functions:: 
* Macros:: 

@end menu

@node Type check macros,Direct API functions,,Byte Array Objects
@anchor{c-api/bytearray type-check-macros}@anchor{43d9}
@subsubsection Type check macros


@geindex PyByteArray_Check (C function)
@anchor{c-api/bytearray c PyByteArray_Check}@anchor{43da}
@deffn {C Function} int  PyByteArray_Check (PyObject  *o)

Return true if the object @emph{o} is a bytearray object or an instance of a
subtype of the bytearray type.  This function always succeeds.
@end deffn

@geindex PyByteArray_CheckExact (C function)
@anchor{c-api/bytearray c PyByteArray_CheckExact}@anchor{43db}
@deffn {C Function} int  PyByteArray_CheckExact (PyObject  *o)

Return true if the object @emph{o} is a bytearray object, but not an instance of a
subtype of the bytearray type.  This function always succeeds.
@end deffn

@node Direct API functions,Macros,Type check macros,Byte Array Objects
@anchor{c-api/bytearray direct-api-functions}@anchor{43dc}
@subsubsection Direct API functions


@geindex PyByteArray_FromObject (C function)
@anchor{c-api/bytearray c PyByteArray_FromObject}@anchor{f20}
@deffn {C Function} @ref{375,,PyObject}  *PyByteArray_FromObject (PyObject  *o)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new bytearray object from any object, @emph{o}, that implements the
@ref{1a3e,,buffer protocol}.

@c XXX expand about the buffer protocol, at least somewhere
@end deffn

@geindex PyByteArray_FromStringAndSize (C function)
@anchor{c-api/bytearray c PyByteArray_FromStringAndSize}@anchor{f21}
@deffn {C Function} @ref{375,,PyObject}  *PyByteArray_FromStringAndSize (const  char  *string, Py_ssize_t  len)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a new bytearray object from @emph{string} and its length, @emph{len}.  On
failure, @code{NULL} is returned.
@end deffn

@geindex PyByteArray_Concat (C function)
@anchor{c-api/bytearray c PyByteArray_Concat}@anchor{412e}
@deffn {C Function} @ref{375,,PyObject}  *PyByteArray_Concat (PyObject  *a, PyObject  *b)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Concat bytearrays @emph{a} and @emph{b} and return a new bytearray with the result.
@end deffn

@geindex PyByteArray_Size (C function)
@anchor{c-api/bytearray c PyByteArray_Size}@anchor{4130}
@deffn {C Function} Py_ssize_t  PyByteArray_Size (PyObject  *bytearray)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the size of @emph{bytearray} after checking for a @code{NULL} pointer.
@end deffn

@geindex PyByteArray_AsString (C function)
@anchor{c-api/bytearray c PyByteArray_AsString}@anchor{412d}
@deffn {C Function} char  *PyByteArray_AsString (PyObject  *bytearray)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the contents of @emph{bytearray} as a char array after checking for a
@code{NULL} pointer.  The returned array always has an extra
null byte appended.
@end deffn

@geindex PyByteArray_Resize (C function)
@anchor{c-api/bytearray c PyByteArray_Resize}@anchor{412f}
@deffn {C Function} int  PyByteArray_Resize (PyObject  *bytearray, Py_ssize_t  len)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Resize the internal buffer of @emph{bytearray} to @emph{len}.
@end deffn

@node Macros,,Direct API functions,Byte Array Objects
@anchor{c-api/bytearray macros}@anchor{43dd}
@subsubsection Macros


These macros trade safety for speed and they don’t check pointers.

@geindex PyByteArray_AS_STRING (C function)
@anchor{c-api/bytearray c PyByteArray_AS_STRING}@anchor{43de}
@deffn {C Function} char  *PyByteArray_AS_STRING (PyObject  *bytearray)

Macro version of @ref{412d,,PyByteArray_AsString()}.
@end deffn

@geindex PyByteArray_GET_SIZE (C function)
@anchor{c-api/bytearray c PyByteArray_GET_SIZE}@anchor{43df}
@deffn {C Function} Py_ssize_t  PyByteArray_GET_SIZE (PyObject  *bytearray)

Macro version of @ref{4130,,PyByteArray_Size()}.
@end deffn

@node Unicode Objects and Codecs,Tuple Objects,Byte Array Objects,Sequence Objects
@anchor{c-api/unicode doc}@anchor{d46}@anchor{c-api/unicode unicode-objects-and-codecs}@anchor{43e0}@anchor{c-api/unicode unicodeobjects}@anchor{43e1}
@subsubsection Unicode Objects and Codecs


@menu
* Unicode Objects:: 
* Built-in Codecs:: 
* Methods and Slot Functions:: 

@end menu

@node Unicode Objects,Built-in Codecs,,Unicode Objects and Codecs
@anchor{c-api/unicode unicode-objects}@anchor{43e2}
@subsubsection Unicode Objects


Since the implementation of 
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393} in Python 3.3, Unicode objects internally
use a variety of representations, in order to allow handling the complete range
of Unicode characters while staying memory efficient.  There are special cases
for strings where all code points are below 128, 256, or 65536; otherwise, code
points must be below 1114112 (which is the full Unicode range).

@ref{d2c,,Py_UNICODE}* and UTF-8 representations are created on demand and cached
in the Unicode object.  The @ref{d2c,,Py_UNICODE}* representation is deprecated
and inefficient.

Due to the transition between the old APIs and the new APIs, Unicode objects
can internally be in two states depending on how they were created:


@itemize *

@item 
“canonical” Unicode objects are all objects created by a non-deprecated
Unicode API.  They use the most efficient representation allowed by the
implementation.

@item 
“legacy” Unicode objects have been created through one of the deprecated
APIs (typically @ref{48c,,PyUnicode_FromUnicode()}) and only bear the
@ref{d2c,,Py_UNICODE}* representation; you will have to call
@ref{d15,,PyUnicode_READY()} on them before calling any other API.
@end itemize

@cartouche
@quotation Note 
The “legacy” Unicode object will be removed in Python 3.12 with deprecated
APIs. All Unicode objects will be “canonical” since then. See 
@geindex Python Enhancement Proposals; PEP 623
PEP 623@footnote{https://www.python.org/dev/peps/pep-0623}
for more information.
@end quotation
@end cartouche

@menu
* Unicode Type:: 
* Unicode Character Properties:: 
* Creating and accessing Unicode strings:: 
* Deprecated Py_UNICODE APIs:: 
* Locale Encoding:: 
* File System Encoding:: 
* wchar_t Support:: 

@end menu

@node Unicode Type,Unicode Character Properties,,Unicode Objects
@anchor{c-api/unicode unicode-type}@anchor{43e3}
@subsubsection Unicode Type


These are the basic Unicode object types used for the Unicode implementation in
Python:

@geindex Py_UCS4 (C type)
@geindex Py_UCS2 (C type)
@geindex Py_UCS1 (C type)
@anchor{c-api/unicode c Py_UCS4}@anchor{d12}
@deffn {C Type} type  Py_UCS4
@anchor{c-api/unicode c Py_UCS2}@anchor{d11}
@deffnx {C Type} type  Py_UCS2
@anchor{c-api/unicode c Py_UCS1}@anchor{d10}
@deffnx {C Type} type  Py_UCS1
@emph{ Part of the @ref{35d,,Stable ABI}.}
These types are typedefs for unsigned integer types wide enough to contain
characters of 32 bits, 16 bits and 8 bits, respectively.  When dealing with
single Unicode characters, use @ref{d12,,Py_UCS4}.


New in version 3.3.

@end deffn

@geindex Py_UNICODE (C type)
@anchor{c-api/unicode c Py_UNICODE}@anchor{d2c}
@deffn {C Type} type  Py_UNICODE

This is a typedef of @code{wchar_t}, which is a 16-bit type or 32-bit type
depending on the platform.


Changed in version 3.3: In previous versions, this was a 16-bit type or a 32-bit type depending on
whether you selected a “narrow” or “wide” Unicode version of Python at
build time.

@end deffn

@geindex PyASCIIObject (C type)
@geindex PyCompactUnicodeObject (C type)
@geindex PyUnicodeObject (C type)
@anchor{c-api/unicode c PyASCIIObject}@anchor{d13}
@deffn {C Type} type  PyASCIIObject
@anchor{c-api/unicode c PyCompactUnicodeObject}@anchor{d14}
@deffnx {C Type} type  PyCompactUnicodeObject
@anchor{c-api/unicode c PyUnicodeObject}@anchor{43e4}
@deffnx {C Type} type  PyUnicodeObject

These subtypes of @ref{375,,PyObject} represent a Python Unicode object.  In
almost all cases, they shouldn’t be used directly, since all API functions
that deal with Unicode objects take and return @ref{375,,PyObject} pointers.


New in version 3.3.

@end deffn

@geindex PyUnicode_Type (C var)
@anchor{c-api/unicode c PyUnicode_Type}@anchor{4292}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyUnicode_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
This instance of @ref{5c6,,PyTypeObject} represents the Python Unicode type.  It
is exposed to Python code as @code{str}.
@end deffn

The following APIs are really C macros and can be used to do fast checks and to
access internal read-only data of Unicode objects:

@geindex PyUnicode_Check (C function)
@anchor{c-api/unicode c PyUnicode_Check}@anchor{43e5}
@deffn {C Function} int  PyUnicode_Check (PyObject  *o)

Return true if the object @emph{o} is a Unicode object or an instance of a Unicode
subtype.  This function always succeeds.
@end deffn

@geindex PyUnicode_CheckExact (C function)
@anchor{c-api/unicode c PyUnicode_CheckExact}@anchor{43e6}
@deffn {C Function} int  PyUnicode_CheckExact (PyObject  *o)

Return true if the object @emph{o} is a Unicode object, but not an instance of a
subtype.  This function always succeeds.
@end deffn

@geindex PyUnicode_READY (C function)
@anchor{c-api/unicode c PyUnicode_READY}@anchor{d15}
@deffn {C Function} int  PyUnicode_READY (PyObject  *o)

Ensure the string object @emph{o} is in the “canonical” representation.  This is
required before using any of the access macros described below.

@c XXX expand on when it is not required

Returns @code{0} on success and @code{-1} with an exception set on failure, which in
particular happens if memory allocation fails.


New in version 3.3.



Deprecated since version 3.10, will be removed in version 3.12: This API will be removed with @ref{48c,,PyUnicode_FromUnicode()}.

@end deffn

@geindex PyUnicode_GET_LENGTH (C function)
@anchor{c-api/unicode c PyUnicode_GET_LENGTH}@anchor{396}
@deffn {C Function} Py_ssize_t  PyUnicode_GET_LENGTH (PyObject  *o)

Return the length of the Unicode string, in code points.  @emph{o} has to be a
Unicode object in the “canonical” representation (not checked).


New in version 3.3.

@end deffn

@geindex PyUnicode_1BYTE_DATA (C function)
@geindex PyUnicode_2BYTE_DATA (C function)
@geindex PyUnicode_4BYTE_DATA (C function)
@anchor{c-api/unicode c PyUnicode_1BYTE_DATA}@anchor{d19}
@deffn {C Function} @ref{d10,,Py_UCS1}  *PyUnicode_1BYTE_DATA (PyObject  *o)
@anchor{c-api/unicode c PyUnicode_2BYTE_DATA}@anchor{d1a}
@deffnx {C Function} @ref{d11,,Py_UCS2}  *PyUnicode_2BYTE_DATA (PyObject  *o)
@anchor{c-api/unicode c PyUnicode_4BYTE_DATA}@anchor{d1b}
@deffnx {C Function} @ref{d12,,Py_UCS4}  *PyUnicode_4BYTE_DATA (PyObject  *o)

Return a pointer to the canonical representation cast to UCS1, UCS2 or UCS4
integer types for direct character access.  No checks are performed if the
canonical representation has the correct character size; use
@ref{d1c,,PyUnicode_KIND()} to select the right macro.  Make sure
@ref{d15,,PyUnicode_READY()} has been called before accessing this.


New in version 3.3.

@end deffn

@geindex PyUnicode_WCHAR_KIND (C macro)
@geindex PyUnicode_1BYTE_KIND (C macro)
@geindex PyUnicode_2BYTE_KIND (C macro)
@geindex PyUnicode_4BYTE_KIND (C macro)
@anchor{c-api/unicode c PyUnicode_WCHAR_KIND}@anchor{d1d}
@deffn {C Macro} PyUnicode_WCHAR_KIND
@anchor{c-api/unicode c PyUnicode_1BYTE_KIND}@anchor{d1e}
@deffnx {C Macro} PyUnicode_1BYTE_KIND
@anchor{c-api/unicode c PyUnicode_2BYTE_KIND}@anchor{d1f}
@deffnx {C Macro} PyUnicode_2BYTE_KIND
@anchor{c-api/unicode c PyUnicode_4BYTE_KIND}@anchor{d20}
@deffnx {C Macro} PyUnicode_4BYTE_KIND

Return values of the @ref{d1c,,PyUnicode_KIND()} macro.


New in version 3.3.



Deprecated since version 3.10, will be removed in version 3.12: @code{PyUnicode_WCHAR_KIND} is deprecated.

@end deffn

@geindex PyUnicode_KIND (C function)
@anchor{c-api/unicode c PyUnicode_KIND}@anchor{d1c}
@deffn {C Function} unsigned  int  PyUnicode_KIND (PyObject  *o)

Return one of the PyUnicode kind constants (see above) that indicate how many
bytes per character this Unicode object uses to store its data.  @emph{o} has to
be a Unicode object in the “canonical” representation (not checked).

@c XXX document "0" return value?


New in version 3.3.

@end deffn

@geindex PyUnicode_DATA (C function)
@anchor{c-api/unicode c PyUnicode_DATA}@anchor{d18}
@deffn {C Function} void  *PyUnicode_DATA (PyObject  *o)

Return a void pointer to the raw Unicode buffer.  @emph{o} has to be a Unicode
object in the “canonical” representation (not checked).


New in version 3.3.

@end deffn

@geindex PyUnicode_WRITE (C function)
@anchor{c-api/unicode c PyUnicode_WRITE}@anchor{d23}
@deffn {C Function} void  PyUnicode_WRITE (int  kind, void  *data, Py_ssize_t  index, Py_UCS4  value)

Write into a canonical representation @emph{data} (as obtained with
@ref{d18,,PyUnicode_DATA()}).  This macro does not do any sanity checks and is
intended for usage in loops.  The caller should cache the @emph{kind} value and
@emph{data} pointer as obtained from other macro calls.  @emph{index} is the index in
the string (starts at 0) and @emph{value} is the new code point value which should
be written to that location.


New in version 3.3.

@end deffn

@geindex PyUnicode_READ (C function)
@anchor{c-api/unicode c PyUnicode_READ}@anchor{d21}
@deffn {C Function} @ref{d12,,Py_UCS4}  PyUnicode_READ (int  kind, void  *data, Py_ssize_t  index)

Read a code point from a canonical representation @emph{data} (as obtained with
@ref{d18,,PyUnicode_DATA()}).  No checks or ready calls are performed.


New in version 3.3.

@end deffn

@geindex PyUnicode_READ_CHAR (C function)
@anchor{c-api/unicode c PyUnicode_READ_CHAR}@anchor{d22}
@deffn {C Function} @ref{d12,,Py_UCS4}  PyUnicode_READ_CHAR (PyObject  *o, Py_ssize_t  index)

Read a character from a Unicode object @emph{o}, which must be in the “canonical”
representation.  This is less efficient than @ref{d21,,PyUnicode_READ()} if you
do multiple consecutive reads.


New in version 3.3.

@end deffn

@geindex PyUnicode_MAX_CHAR_VALUE (C macro)
@anchor{c-api/unicode c PyUnicode_MAX_CHAR_VALUE}@anchor{d24}
@deffn {C Macro} PyUnicode_MAX_CHAR_VALUE (o)

Return the maximum code point that is suitable for creating another string
based on @emph{o}, which must be in the “canonical” representation.  This is
always an approximation but more efficient than iterating over the string.


New in version 3.3.

@end deffn

@geindex PyUnicode_GET_SIZE (C function)
@anchor{c-api/unicode c PyUnicode_GET_SIZE}@anchor{d30}
@deffn {C Function} Py_ssize_t  PyUnicode_GET_SIZE (PyObject  *o)

Return the size of the deprecated @ref{d2c,,Py_UNICODE} representation, in
code units (this includes surrogate pairs as 2 units).  @emph{o} has to be a
Unicode object (not checked).


Deprecated since version 3.3, will be removed in version 3.12: Part of the old-style Unicode API, please migrate to using
@ref{396,,PyUnicode_GET_LENGTH()}.

@end deffn

@geindex PyUnicode_GET_DATA_SIZE (C function)
@anchor{c-api/unicode c PyUnicode_GET_DATA_SIZE}@anchor{d32}
@deffn {C Function} Py_ssize_t  PyUnicode_GET_DATA_SIZE (PyObject  *o)

Return the size of the deprecated @ref{d2c,,Py_UNICODE} representation in
bytes.  @emph{o} has to be a Unicode object (not checked).


Deprecated since version 3.3, will be removed in version 3.12: Part of the old-style Unicode API, please migrate to using
@ref{396,,PyUnicode_GET_LENGTH()}.

@end deffn

@geindex PyUnicode_AS_UNICODE (C function)
@geindex PyUnicode_AS_DATA (C function)
@anchor{c-api/unicode c PyUnicode_AS_UNICODE}@anchor{d2e}
@deffn {C Function} @ref{d2c,,Py_UNICODE}  *PyUnicode_AS_UNICODE (PyObject  *o)
@anchor{c-api/unicode c PyUnicode_AS_DATA}@anchor{d2f}
@deffnx {C Function} const  char  *PyUnicode_AS_DATA (PyObject  *o)

Return a pointer to a @ref{d2c,,Py_UNICODE} representation of the object.  The
returned buffer is always terminated with an extra null code point.  It
may also contain embedded null code points, which would cause the string
to be truncated when used in most C functions.  The @code{AS_DATA} form
casts the pointer to const  char*.  The @emph{o} argument has to be
a Unicode object (not checked).


Changed in version 3.3: This macro is now inefficient – because in many cases the
@ref{d2c,,Py_UNICODE} representation does not exist and needs to be created
– and can fail (return @code{NULL} with an exception set).  Try to port the
code to use the new @code{PyUnicode_nBYTE_DATA()} macros or use
@ref{d23,,PyUnicode_WRITE()} or @ref{d21,,PyUnicode_READ()}.



Deprecated since version 3.3, will be removed in version 3.12: Part of the old-style Unicode API, please migrate to using the
@code{PyUnicode_nBYTE_DATA()} family of macros.

@end deffn

@geindex PyUnicode_IsIdentifier (C function)
@anchor{c-api/unicode c PyUnicode_IsIdentifier}@anchor{1233}
@deffn {C Function} int  PyUnicode_IsIdentifier (PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return @code{1} if the string is a valid identifier according to the language
definition, section @ref{170b,,Identifiers and keywords}. Return @code{0} otherwise.


Changed in version 3.9: The function does not call @ref{48f,,Py_FatalError()} anymore if the string
is not ready.

@end deffn

@node Unicode Character Properties,Creating and accessing Unicode strings,Unicode Type,Unicode Objects
@anchor{c-api/unicode unicode-character-properties}@anchor{43e7}
@subsubsection Unicode Character Properties


Unicode provides many different character properties. The most often needed ones
are available through these macros which are mapped to C functions depending on
the Python configuration.

@geindex Py_UNICODE_ISSPACE (C function)
@anchor{c-api/unicode c Py_UNICODE_ISSPACE}@anchor{117d}
@deffn {C Function} int  Py_UNICODE_ISSPACE (Py_UCS4  ch)

Return @code{1} or @code{0} depending on whether @emph{ch} is a whitespace character.
@end deffn

@geindex Py_UNICODE_ISLOWER (C function)
@anchor{c-api/unicode c Py_UNICODE_ISLOWER}@anchor{43e8}
@deffn {C Function} int  Py_UNICODE_ISLOWER (Py_UCS4  ch)

Return @code{1} or @code{0} depending on whether @emph{ch} is a lowercase character.
@end deffn

@geindex Py_UNICODE_ISUPPER (C function)
@anchor{c-api/unicode c Py_UNICODE_ISUPPER}@anchor{43e9}
@deffn {C Function} int  Py_UNICODE_ISUPPER (Py_UCS4  ch)

Return @code{1} or @code{0} depending on whether @emph{ch} is an uppercase character.
@end deffn

@geindex Py_UNICODE_ISTITLE (C function)
@anchor{c-api/unicode c Py_UNICODE_ISTITLE}@anchor{43ea}
@deffn {C Function} int  Py_UNICODE_ISTITLE (Py_UCS4  ch)

Return @code{1} or @code{0} depending on whether @emph{ch} is a titlecase character.
@end deffn

@geindex Py_UNICODE_ISLINEBREAK (C function)
@anchor{c-api/unicode c Py_UNICODE_ISLINEBREAK}@anchor{43eb}
@deffn {C Function} int  Py_UNICODE_ISLINEBREAK (Py_UCS4  ch)

Return @code{1} or @code{0} depending on whether @emph{ch} is a linebreak character.
@end deffn

@geindex Py_UNICODE_ISDECIMAL (C function)
@anchor{c-api/unicode c Py_UNICODE_ISDECIMAL}@anchor{43ec}
@deffn {C Function} int  Py_UNICODE_ISDECIMAL (Py_UCS4  ch)

Return @code{1} or @code{0} depending on whether @emph{ch} is a decimal character.
@end deffn

@geindex Py_UNICODE_ISDIGIT (C function)
@anchor{c-api/unicode c Py_UNICODE_ISDIGIT}@anchor{43ed}
@deffn {C Function} int  Py_UNICODE_ISDIGIT (Py_UCS4  ch)

Return @code{1} or @code{0} depending on whether @emph{ch} is a digit character.
@end deffn

@geindex Py_UNICODE_ISNUMERIC (C function)
@anchor{c-api/unicode c Py_UNICODE_ISNUMERIC}@anchor{43ee}
@deffn {C Function} int  Py_UNICODE_ISNUMERIC (Py_UCS4  ch)

Return @code{1} or @code{0} depending on whether @emph{ch} is a numeric character.
@end deffn

@geindex Py_UNICODE_ISALPHA (C function)
@anchor{c-api/unicode c Py_UNICODE_ISALPHA}@anchor{43ef}
@deffn {C Function} int  Py_UNICODE_ISALPHA (Py_UCS4  ch)

Return @code{1} or @code{0} depending on whether @emph{ch} is an alphabetic character.
@end deffn

@geindex Py_UNICODE_ISALNUM (C function)
@anchor{c-api/unicode c Py_UNICODE_ISALNUM}@anchor{43f0}
@deffn {C Function} int  Py_UNICODE_ISALNUM (Py_UCS4  ch)

Return @code{1} or @code{0} depending on whether @emph{ch} is an alphanumeric character.
@end deffn

@geindex Py_UNICODE_ISPRINTABLE (C function)
@anchor{c-api/unicode c Py_UNICODE_ISPRINTABLE}@anchor{43f1}
@deffn {C Function} int  Py_UNICODE_ISPRINTABLE (Py_UCS4  ch)

Return @code{1} or @code{0} depending on whether @emph{ch} is a printable character.
Nonprintable characters are those characters defined in the Unicode character
database as “Other” or “Separator”, excepting the ASCII space (0x20) which is
considered printable.  (Note that printable characters in this context are
those which should not be escaped when @ref{27d,,repr()} is invoked on a string.
It has no bearing on the handling of strings written to @ref{5ef,,sys.stdout} or
@ref{42d,,sys.stderr}.)
@end deffn

These APIs can be used for fast direct character conversions:

@geindex Py_UNICODE_TOLOWER (C function)
@anchor{c-api/unicode c Py_UNICODE_TOLOWER}@anchor{43f2}
@deffn {C Function} @ref{d12,,Py_UCS4}  Py_UNICODE_TOLOWER (Py_UCS4  ch)

Return the character @emph{ch} converted to lower case.


Deprecated since version 3.3: This function uses simple case mappings.

@end deffn

@geindex Py_UNICODE_TOUPPER (C function)
@anchor{c-api/unicode c Py_UNICODE_TOUPPER}@anchor{43f3}
@deffn {C Function} @ref{d12,,Py_UCS4}  Py_UNICODE_TOUPPER (Py_UCS4  ch)

Return the character @emph{ch} converted to upper case.


Deprecated since version 3.3: This function uses simple case mappings.

@end deffn

@geindex Py_UNICODE_TOTITLE (C function)
@anchor{c-api/unicode c Py_UNICODE_TOTITLE}@anchor{43f4}
@deffn {C Function} @ref{d12,,Py_UCS4}  Py_UNICODE_TOTITLE (Py_UCS4  ch)

Return the character @emph{ch} converted to title case.


Deprecated since version 3.3: This function uses simple case mappings.

@end deffn

@geindex Py_UNICODE_TODECIMAL (C function)
@anchor{c-api/unicode c Py_UNICODE_TODECIMAL}@anchor{43f5}
@deffn {C Function} int  Py_UNICODE_TODECIMAL (Py_UCS4  ch)

Return the character @emph{ch} converted to a decimal positive integer.  Return
@code{-1} if this is not possible.  This macro does not raise exceptions.
@end deffn

@geindex Py_UNICODE_TODIGIT (C function)
@anchor{c-api/unicode c Py_UNICODE_TODIGIT}@anchor{43f6}
@deffn {C Function} int  Py_UNICODE_TODIGIT (Py_UCS4  ch)

Return the character @emph{ch} converted to a single digit integer. Return @code{-1} if
this is not possible.  This macro does not raise exceptions.
@end deffn

@geindex Py_UNICODE_TONUMERIC (C function)
@anchor{c-api/unicode c Py_UNICODE_TONUMERIC}@anchor{43f7}
@deffn {C Function} double  Py_UNICODE_TONUMERIC (Py_UCS4  ch)

Return the character @emph{ch} converted to a double. Return @code{-1.0} if this is not
possible.  This macro does not raise exceptions.
@end deffn

These APIs can be used to work with surrogates:

@geindex Py_UNICODE_IS_SURROGATE (C macro)
@anchor{c-api/unicode c Py_UNICODE_IS_SURROGATE}@anchor{43f8}
@deffn {C Macro} Py_UNICODE_IS_SURROGATE (ch)

Check if @emph{ch} is a surrogate (@code{0xD800 <= ch <= 0xDFFF}).
@end deffn

@geindex Py_UNICODE_IS_HIGH_SURROGATE (C macro)
@anchor{c-api/unicode c Py_UNICODE_IS_HIGH_SURROGATE}@anchor{43f9}
@deffn {C Macro} Py_UNICODE_IS_HIGH_SURROGATE (ch)

Check if @emph{ch} is a high surrogate (@code{0xD800 <= ch <= 0xDBFF}).
@end deffn

@geindex Py_UNICODE_IS_LOW_SURROGATE (C macro)
@anchor{c-api/unicode c Py_UNICODE_IS_LOW_SURROGATE}@anchor{43fa}
@deffn {C Macro} Py_UNICODE_IS_LOW_SURROGATE (ch)

Check if @emph{ch} is a low surrogate (@code{0xDC00 <= ch <= 0xDFFF}).
@end deffn

@geindex Py_UNICODE_JOIN_SURROGATES (C macro)
@anchor{c-api/unicode c Py_UNICODE_JOIN_SURROGATES}@anchor{43fb}
@deffn {C Macro} Py_UNICODE_JOIN_SURROGATES (high, low)

Join two surrogate characters and return a single Py_UCS4 value.
@emph{high} and @emph{low} are respectively the leading and trailing surrogates in a
surrogate pair.
@end deffn

@node Creating and accessing Unicode strings,Deprecated Py_UNICODE APIs,Unicode Character Properties,Unicode Objects
@anchor{c-api/unicode creating-and-accessing-unicode-strings}@anchor{43fc}
@subsubsection Creating and accessing Unicode strings


To create Unicode objects and access their basic sequence properties, use these
APIs:

@geindex PyUnicode_New (C function)
@anchor{c-api/unicode c PyUnicode_New}@anchor{386}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_New (Py_ssize_t  size, Py_UCS4  maxchar)
@emph{Return value: New reference.}
Create a new Unicode object.  @emph{maxchar} should be the true maximum code point
to be placed in the string.  As an approximation, it can be rounded up to the
nearest value in the sequence 127, 255, 65535, 1114111.

This is the recommended way to allocate a new Unicode object.  Objects
created using this function are not resizable.


New in version 3.3.

@end deffn

@geindex PyUnicode_FromKindAndData (C function)
@anchor{c-api/unicode c PyUnicode_FromKindAndData}@anchor{d16}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_FromKindAndData (int  kind, const  void  *buffer, Py_ssize_t  size)
@emph{Return value: New reference.}
Create a new Unicode object with the given @emph{kind} (possible values are
@ref{d1e,,PyUnicode_1BYTE_KIND} etc., as returned by
@ref{d1c,,PyUnicode_KIND()}).  The @emph{buffer} must point to an array of @emph{size}
units of 1, 2 or 4 bytes per character, as given by the kind.

If necessary, the input @emph{buffer} is copied and transformed into the
canonical representation.  For example, if the @emph{buffer} is a UCS4 string
(@ref{d20,,PyUnicode_4BYTE_KIND}) and it consists only of codepoints in
the UCS1 range, it will be transformed into UCS1
(@ref{d1e,,PyUnicode_1BYTE_KIND}).


New in version 3.3.

@end deffn

@geindex PyUnicode_FromStringAndSize (C function)
@anchor{c-api/unicode c PyUnicode_FromStringAndSize}@anchor{428b}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_FromStringAndSize (const  char  *u, Py_ssize_t  size)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a Unicode object from the char buffer @emph{u}.  The bytes will be
interpreted as being UTF-8 encoded.  The buffer is copied into the new
object. If the buffer is not @code{NULL}, the return value might be a shared
object, i.e. modification of the data is not allowed.

If @emph{u} is @code{NULL}, this function behaves like @ref{48c,,PyUnicode_FromUnicode()}
with the buffer set to @code{NULL}.  This usage is deprecated in favor of
@ref{386,,PyUnicode_New()}, and will be removed in Python 3.12.
@end deffn

@geindex PyUnicode_FromString (C function)
@anchor{c-api/unicode c PyUnicode_FromString}@anchor{40f4}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_FromString (const  char  *u)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a Unicode object from a UTF-8 encoded null-terminated char buffer
@emph{u}.
@end deffn

@geindex PyUnicode_FromFormat (C function)
@anchor{c-api/unicode c PyUnicode_FromFormat}@anchor{398}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_FromFormat (const  char  *format, ...)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Take a C @code{printf()}-style @emph{format} string and a variable number of
arguments, calculate the size of the resulting Python Unicode string and return
a string with the values formatted into it.  The variable arguments must be C
types and must correspond exactly to the format characters in the @emph{format}
ASCII-encoded string. The following format characters are allowed:

@c % This should be exactly the same as the table in PyErr_Format.

@c % The descriptions for %zd and %zu are wrong, but the truth is complicated

@c % because not all compilers support the %z width modifier -- we fake it

@c % when necessary via interpolating PY_FORMAT_SIZE_T.

@c % Similar comments apply to the %ll width modifier and


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Format Characters

@tab

Type

@tab

Comment

@item

@code{%%}

@tab

@emph{n/a}

@tab

The literal % character.

@item

@code{%c}

@tab

int

@tab

A single character,
represented as a C int.

@item

@code{%d}

@tab

int

@tab

Equivalent to
@code{printf("%d")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
zu, i, x): the 0-conversion flag has effect even when a precision is given.
}

@item

@code{%u}

@tab

unsigned int

@tab

Equivalent to
@code{printf("%u")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
zu, i, x): the 0-conversion flag has effect even when a precision is given.
}

@item

@code{%ld}

@tab

long

@tab

Equivalent to
@code{printf("%ld")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
zu, i, x): the 0-conversion flag has effect even when a precision is given.
}

@item

@code{%li}

@tab

long

@tab

Equivalent to
@code{printf("%li")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
zu, i, x): the 0-conversion flag has effect even when a precision is given.
}

@item

@code{%lu}

@tab

unsigned long

@tab

Equivalent to
@code{printf("%lu")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
zu, i, x): the 0-conversion flag has effect even when a precision is given.
}

@item

@code{%lld}

@tab

long long

@tab

Equivalent to
@code{printf("%lld")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
zu, i, x): the 0-conversion flag has effect even when a precision is given.
}

@item

@code{%lli}

@tab

long long

@tab

Equivalent to
@code{printf("%lli")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
zu, i, x): the 0-conversion flag has effect even when a precision is given.
}

@item

@code{%llu}

@tab

unsigned long long

@tab

Equivalent to
@code{printf("%llu")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
zu, i, x): the 0-conversion flag has effect even when a precision is given.
}

@item

@code{%zd}

@tab

Py_ssize_t

@tab

Equivalent to
@code{printf("%zd")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
zu, i, x): the 0-conversion flag has effect even when a precision is given.
}

@item

@code{%zi}

@tab

Py_ssize_t

@tab

Equivalent to
@code{printf("%zi")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
zu, i, x): the 0-conversion flag has effect even when a precision is given.
}

@item

@code{%zu}

@tab

size_t

@tab

Equivalent to
@code{printf("%zu")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
zu, i, x): the 0-conversion flag has effect even when a precision is given.
}

@item

@code{%i}

@tab

int

@tab

Equivalent to
@code{printf("%i")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
zu, i, x): the 0-conversion flag has effect even when a precision is given.
}

@item

@code{%x}

@tab

int

@tab

Equivalent to
@code{printf("%x")}. @footnote{@w{(1)} 
For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
zu, i, x): the 0-conversion flag has effect even when a precision is given.
}

@item

@code{%s}

@tab

const char*

@tab

A null-terminated C character
array.

@item

@code{%p}

@tab

const void*

@tab

The hex representation of a C
pointer. Mostly equivalent to
@code{printf("%p")} except that
it is guaranteed to start with
the literal @code{0x} regardless
of what the platform’s
@code{printf} yields.

@item

@code{%A}

@tab

PyObject*

@tab

The result of calling
@ref{f47,,ascii()}.

@item

@code{%U}

@tab

PyObject*

@tab

A Unicode object.

@item

@code{%V}

@tab

PyObject*,
const char*

@tab

A Unicode object (which may be
@code{NULL}) and a null-terminated
C character array as a second
parameter (which will be used,
if the first parameter is
@code{NULL}).

@item

@code{%S}

@tab

PyObject*

@tab

The result of calling
@ref{bc1,,PyObject_Str()}.

@item

@code{%R}

@tab

PyObject*

@tab

The result of calling
@ref{bc0,,PyObject_Repr()}.

@end multitable


An unrecognized format character causes all the rest of the format string to be
copied as-is to the result string, and any extra arguments discarded.

@cartouche
@quotation Note 
The width formatter unit is number of characters rather than bytes.
The precision formatter unit is number of bytes for @code{"%s"} and
@code{"%V"} (if the @code{PyObject*} argument is @code{NULL}), and a number of
characters for @code{"%A"}, @code{"%U"}, @code{"%S"}, @code{"%R"} and @code{"%V"}
(if the @code{PyObject*} argument is not @code{NULL}).
@end quotation
@end cartouche


Changed in version 3.2: Support for @code{"%lld"} and @code{"%llu"} added.



Changed in version 3.3: Support for @code{"%li"}, @code{"%lli"} and @code{"%zi"} added.



Changed in version 3.4: Support width and precision formatter for @code{"%s"}, @code{"%A"}, @code{"%U"},
@code{"%V"}, @code{"%S"}, @code{"%R"} added.

@end deffn

@geindex PyUnicode_FromFormatV (C function)
@anchor{c-api/unicode c PyUnicode_FromFormatV}@anchor{4289}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_FromFormatV (const  char  *format, va_list  vargs)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Identical to @ref{398,,PyUnicode_FromFormat()} except that it takes exactly two
arguments.
@end deffn

@geindex PyUnicode_FromEncodedObject (C function)
@anchor{c-api/unicode c PyUnicode_FromEncodedObject}@anchor{4288}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_FromEncodedObject (PyObject  *obj, const  char  *encoding, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Decode an encoded object @emph{obj} to a Unicode object.

@ref{2f0,,bytes}, @ref{2f1,,bytearray} and other
@ref{87e,,bytes-like objects}
are decoded according to the given @emph{encoding} and using the error handling
defined by @emph{errors}. Both can be @code{NULL} to have the interface use the default
values (see @ref{43fd,,Built-in Codecs} for details).

All other objects, including Unicode objects, cause a @ref{162,,TypeError} to be
set.

The API returns @code{NULL} if there was an error.  The caller is responsible for
decref’ing the returned objects.
@end deffn

@geindex PyUnicode_GetLength (C function)
@anchor{c-api/unicode c PyUnicode_GetLength}@anchor{395}
@deffn {C Function} Py_ssize_t  PyUnicode_GetLength (PyObject  *unicode)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Return the length of the Unicode object, in code points.


New in version 3.3.

@end deffn

@geindex PyUnicode_CopyCharacters (C function)
@anchor{c-api/unicode c PyUnicode_CopyCharacters}@anchor{397}
@deffn {C Function} Py_ssize_t  PyUnicode_CopyCharacters (PyObject  *to, Py_ssize_t  to_start, PyObject  *from, Py_ssize_t  from_start, Py_ssize_t  how_many)

Copy characters from one Unicode object into another.  This function performs
character conversion when necessary and falls back to @code{memcpy()} if
possible.  Returns @code{-1} and sets an exception on error, otherwise returns
the number of copied characters.


New in version 3.3.

@end deffn

@geindex PyUnicode_Fill (C function)
@anchor{c-api/unicode c PyUnicode_Fill}@anchor{d33}
@deffn {C Function} Py_ssize_t  PyUnicode_Fill (PyObject  *unicode, Py_ssize_t  start, Py_ssize_t  length, Py_UCS4  fill_char)

Fill a string with a character: write @emph{fill_char} into
@code{unicode[start:start+length]}.

Fail if @emph{fill_char} is bigger than the string maximum character, or if the
string has more than 1 reference.

Return the number of written character, or return @code{-1} and raise an
exception on error.


New in version 3.3.

@end deffn

@geindex PyUnicode_WriteChar (C function)
@anchor{c-api/unicode c PyUnicode_WriteChar}@anchor{d0f}
@deffn {C Function} int  PyUnicode_WriteChar (PyObject  *unicode, Py_ssize_t  index, Py_UCS4  character)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Write a character to a string.  The string must have been created through
@ref{386,,PyUnicode_New()}.  Since Unicode strings are supposed to be immutable,
the string must not be shared, or have been hashed yet.

This function checks that @emph{unicode} is a Unicode object, that the index is
not out of bounds, and that the object can be modified safely (i.e. that it
its reference count is one).


New in version 3.3.

@end deffn

@geindex PyUnicode_ReadChar (C function)
@anchor{c-api/unicode c PyUnicode_ReadChar}@anchor{d0e}
@deffn {C Function} @ref{d12,,Py_UCS4}  PyUnicode_ReadChar (PyObject  *unicode, Py_ssize_t  index)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Read a character from a string.  This function checks that @emph{unicode} is a
Unicode object and the index is not out of bounds, in contrast to the macro
version @ref{d22,,PyUnicode_READ_CHAR()}.


New in version 3.3.

@end deffn

@geindex PyUnicode_Substring (C function)
@anchor{c-api/unicode c PyUnicode_Substring}@anchor{399}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_Substring (PyObject  *str, Py_ssize_t  start, Py_ssize_t  end)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Return a substring of @emph{str}, from character index @emph{start} (included) to
character index @emph{end} (excluded).  Negative indices are not supported.


New in version 3.3.

@end deffn

@geindex PyUnicode_AsUCS4 (C function)
@anchor{c-api/unicode c PyUnicode_AsUCS4}@anchor{d17}
@deffn {C Function} @ref{d12,,Py_UCS4}  *PyUnicode_AsUCS4 (PyObject  *u, Py_UCS4  *buffer, Py_ssize_t  buflen, int  copy_null)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Copy the string @emph{u} into a UCS4 buffer, including a null character, if
@emph{copy_null} is set.  Returns @code{NULL} and sets an exception on error (in
particular, a @ref{891,,SystemError} if @emph{buflen} is smaller than the length of
@emph{u}).  @emph{buffer} is returned on success.


New in version 3.3.

@end deffn

@geindex PyUnicode_AsUCS4Copy (C function)
@anchor{c-api/unicode c PyUnicode_AsUCS4Copy}@anchor{39e}
@deffn {C Function} @ref{d12,,Py_UCS4}  *PyUnicode_AsUCS4Copy (PyObject  *u)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Copy the string @emph{u} into a new UCS4 buffer that is allocated using
@ref{7ad,,PyMem_Malloc()}.  If this fails, @code{NULL} is returned with a
@ref{10de,,MemoryError} set.  The returned buffer always has an extra
null code point appended.


New in version 3.3.

@end deffn

@node Deprecated Py_UNICODE APIs,Locale Encoding,Creating and accessing Unicode strings,Unicode Objects
@anchor{c-api/unicode deprecated-py-unicode-apis}@anchor{43fe}
@subsubsection Deprecated Py_UNICODE APIs



Deprecated since version 3.3, will be removed in version 3.12.


These API functions are deprecated with the implementation of 
@geindex Python Enhancement Proposals; PEP 393
PEP 393@footnote{https://www.python.org/dev/peps/pep-0393}.
Extension modules can continue using them, as they will not be removed in Python
3.x, but need to be aware that their use can now cause performance and memory hits.

@geindex PyUnicode_FromUnicode (C function)
@anchor{c-api/unicode c PyUnicode_FromUnicode}@anchor{48c}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_FromUnicode (const  Py_UNICODE  *u, Py_ssize_t  size)
@emph{Return value: New reference.}
Create a Unicode object from the Py_UNICODE buffer @emph{u} of the given size. @emph{u}
may be @code{NULL} which causes the contents to be undefined. It is the user’s
responsibility to fill in the needed data.  The buffer is copied into the new
object.

If the buffer is not @code{NULL}, the return value might be a shared object.
Therefore, modification of the resulting Unicode object is only allowed when
@emph{u} is @code{NULL}.

If the buffer is @code{NULL}, @ref{d15,,PyUnicode_READY()} must be called once the
string content has been filled before using any of the access macros such as
@ref{d1c,,PyUnicode_KIND()}.


Deprecated since version 3.3, will be removed in version 3.12: Part of the old-style Unicode API, please migrate to using
@ref{d16,,PyUnicode_FromKindAndData()}, @ref{d2d,,PyUnicode_FromWideChar()}, or
@ref{386,,PyUnicode_New()}.

@end deffn

@geindex PyUnicode_AsUnicode (C function)
@anchor{c-api/unicode c PyUnicode_AsUnicode}@anchor{48d}
@deffn {C Function} @ref{d2c,,Py_UNICODE}  *PyUnicode_AsUnicode (PyObject  *unicode)

Return a read-only pointer to the Unicode object’s internal
@ref{d2c,,Py_UNICODE} buffer, or @code{NULL} on error. This will create the
@ref{d2c,,Py_UNICODE}* representation of the object if it is not yet
available. The buffer is always terminated with an extra null code point.
Note that the resulting @ref{d2c,,Py_UNICODE} string may also contain
embedded null code points, which would cause the string to be truncated when
used in most C functions.


Deprecated since version 3.3, will be removed in version 3.12: Part of the old-style Unicode API, please migrate to using
@ref{d17,,PyUnicode_AsUCS4()}, @ref{1315,,PyUnicode_AsWideChar()},
@ref{d0e,,PyUnicode_ReadChar()} or similar new APIs.

@end deffn

@geindex PyUnicode_AsUnicodeAndSize (C function)
@anchor{c-api/unicode c PyUnicode_AsUnicodeAndSize}@anchor{48e}
@deffn {C Function} @ref{d2c,,Py_UNICODE}  *PyUnicode_AsUnicodeAndSize (PyObject  *unicode, Py_ssize_t  *size)

Like @ref{48d,,PyUnicode_AsUnicode()}, but also saves the @ref{d2c,,Py_UNICODE()}
array length (excluding the extra null terminator) in @emph{size}.
Note that the resulting @ref{d2c,,Py_UNICODE}* string
may contain embedded null code points, which would cause the string to be
truncated when used in most C functions.


New in version 3.3.



Deprecated since version 3.3, will be removed in version 3.12: Part of the old-style Unicode API, please migrate to using
@ref{d17,,PyUnicode_AsUCS4()}, @ref{1315,,PyUnicode_AsWideChar()},
@ref{d0e,,PyUnicode_ReadChar()} or similar new APIs.

@end deffn

@geindex PyUnicode_GetSize (C function)
@anchor{c-api/unicode c PyUnicode_GetSize}@anchor{d31}
@deffn {C Function} Py_ssize_t  PyUnicode_GetSize (PyObject  *unicode)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the size of the deprecated @ref{d2c,,Py_UNICODE} representation, in
code units (this includes surrogate pairs as 2 units).


Deprecated since version 3.3, will be removed in version 3.12: Part of the old-style Unicode API, please migrate to using
@ref{396,,PyUnicode_GET_LENGTH()}.

@end deffn

@geindex PyUnicode_FromObject (C function)
@anchor{c-api/unicode c PyUnicode_FromObject}@anchor{428a}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_FromObject (PyObject  *obj)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Copy an instance of a Unicode subtype to a new true Unicode object if
necessary. If @emph{obj} is already a true Unicode object (not a subtype),
return the reference with incremented refcount.

Objects other than Unicode or its subtypes will cause a @ref{162,,TypeError}.
@end deffn

@node Locale Encoding,File System Encoding,Deprecated Py_UNICODE APIs,Unicode Objects
@anchor{c-api/unicode locale-encoding}@anchor{43ff}
@subsubsection Locale Encoding


The current locale encoding can be used to decode text from the operating
system.

@geindex PyUnicode_DecodeLocaleAndSize (C function)
@anchor{c-api/unicode c PyUnicode_DecodeLocaleAndSize}@anchor{700}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeLocaleAndSize (const  char  *str, Py_ssize_t  len, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Decode a string from UTF-8 on Android and VxWorks, or from the current
locale encoding on other platforms. The supported
error handlers are @code{"strict"} and @code{"surrogateescape"}
(
@geindex Python Enhancement Proposals; PEP 383
PEP 383@footnote{https://www.python.org/dev/peps/pep-0383}). The decoder uses @code{"strict"} error handler if
@emph{errors} is @code{NULL}.  @emph{str} must end with a null character but
cannot contain embedded null characters.

Use @ref{4277,,PyUnicode_DecodeFSDefaultAndSize()} to decode a string from
@code{Py_FileSystemDefaultEncoding} (the locale encoding read at
Python startup).

This function ignores the @ref{5f2,,Python UTF-8 Mode}.


@subsubheading See also


The @ref{6fe,,Py_DecodeLocale()} function.



New in version 3.3.



Changed in version 3.7: The function now also uses the current locale encoding for the
@code{surrogateescape} error handler, except on Android. Previously, @ref{6fe,,Py_DecodeLocale()}
was used for the @code{surrogateescape}, and the current locale encoding was
used for @code{strict}.

@end deffn

@geindex PyUnicode_DecodeLocale (C function)
@anchor{c-api/unicode c PyUnicode_DecodeLocale}@anchor{4279}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeLocale (const  char  *str, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Similar to @ref{700,,PyUnicode_DecodeLocaleAndSize()}, but compute the string
length using @code{strlen()}.


New in version 3.3.

@end deffn

@geindex PyUnicode_EncodeLocale (C function)
@anchor{c-api/unicode c PyUnicode_EncodeLocale}@anchor{701}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_EncodeLocale (PyObject  *unicode, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Encode a Unicode object to UTF-8 on Android and VxWorks, or to the current
locale encoding on other platforms. The
supported error handlers are @code{"strict"} and @code{"surrogateescape"}
(
@geindex Python Enhancement Proposals; PEP 383
PEP 383@footnote{https://www.python.org/dev/peps/pep-0383}). The encoder uses @code{"strict"} error handler if
@emph{errors} is @code{NULL}. Return a @ref{2f0,,bytes} object. @emph{unicode} cannot
contain embedded null characters.

Use @ref{130e,,PyUnicode_EncodeFSDefault()} to encode a string to
@code{Py_FileSystemDefaultEncoding} (the locale encoding read at
Python startup).

This function ignores the @ref{5f2,,Python UTF-8 Mode}.


@subsubheading See also


The @ref{6ff,,Py_EncodeLocale()} function.



New in version 3.3.



Changed in version 3.7: The function now also uses the current locale encoding for the
@code{surrogateescape} error handler, except on Android. Previously,
@ref{6ff,,Py_EncodeLocale()}
was used for the @code{surrogateescape}, and the current locale encoding was
used for @code{strict}.

@end deffn

@node File System Encoding,wchar_t Support,Locale Encoding,Unicode Objects
@anchor{c-api/unicode file-system-encoding}@anchor{4400}
@subsubsection File System Encoding


To encode and decode file names and other environment strings,
@code{Py_FileSystemDefaultEncoding} should be used as the encoding, and
@code{Py_FileSystemDefaultEncodeErrors} should be used as the error handler
(
@geindex Python Enhancement Proposals; PEP 383
PEP 383@footnote{https://www.python.org/dev/peps/pep-0383} and 
@geindex Python Enhancement Proposals; PEP 529
PEP 529@footnote{https://www.python.org/dev/peps/pep-0529}). To encode file names to @ref{2f0,,bytes} during
argument parsing, the @code{"O&"} converter should be used, passing
@ref{867,,PyUnicode_FSConverter()} as the conversion function:

@geindex PyUnicode_FSConverter (C function)
@anchor{c-api/unicode c PyUnicode_FSConverter}@anchor{867}
@deffn {C Function} int  PyUnicode_FSConverter (PyObject  *obj, void  *result)
@emph{ Part of the @ref{35d,,Stable ABI}.}
ParseTuple converter: encode @ref{2ef,,str} objects – obtained directly or
through the @ref{795,,os.PathLike} interface – to @ref{2f0,,bytes} using
@ref{130e,,PyUnicode_EncodeFSDefault()}; @ref{2f0,,bytes} objects are output as-is.
@emph{result} must be a @ref{208,,PyBytesObject}* which must be released when it is
no longer used.


New in version 3.1.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

To decode file names to @ref{2ef,,str} during argument parsing, the @code{"O&"}
converter should be used, passing @ref{868,,PyUnicode_FSDecoder()} as the
conversion function:

@geindex PyUnicode_FSDecoder (C function)
@anchor{c-api/unicode c PyUnicode_FSDecoder}@anchor{868}
@deffn {C Function} int  PyUnicode_FSDecoder (PyObject  *obj, void  *result)
@emph{ Part of the @ref{35d,,Stable ABI}.}
ParseTuple converter: decode @ref{2f0,,bytes} objects – obtained either
directly or indirectly through the @ref{795,,os.PathLike} interface – to
@ref{2ef,,str} using @ref{4277,,PyUnicode_DecodeFSDefaultAndSize()}; @ref{2ef,,str}
objects are output as-is. @emph{result} must be a @ref{43e4,,PyUnicodeObject}* which
must be released when it is no longer used.


New in version 3.2.



Changed in version 3.6: Accepts a @ref{63d,,path-like object}.

@end deffn

@geindex PyUnicode_DecodeFSDefaultAndSize (C function)
@anchor{c-api/unicode c PyUnicode_DecodeFSDefaultAndSize}@anchor{4277}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeFSDefaultAndSize (const  char  *s, Py_ssize_t  size)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Decode a string from the @ref{1628,,filesystem encoding and error handler}.

If @code{Py_FileSystemDefaultEncoding} is not set, fall back to the
locale encoding.

@code{Py_FileSystemDefaultEncoding} is initialized at startup from the
locale encoding and cannot be modified later. If you need to decode a string
from the current locale encoding, use
@ref{700,,PyUnicode_DecodeLocaleAndSize()}.


@subsubheading See also


The @ref{6fe,,Py_DecodeLocale()} function.



Changed in version 3.6: Use @code{Py_FileSystemDefaultEncodeErrors} error handler.

@end deffn

@geindex PyUnicode_DecodeFSDefault (C function)
@anchor{c-api/unicode c PyUnicode_DecodeFSDefault}@anchor{4276}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeFSDefault (const  char  *s)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Decode a null-terminated string from the @ref{1628,,filesystem encoding and error handler}.

If @code{Py_FileSystemDefaultEncoding} is not set, fall back to the
locale encoding.

Use @ref{4277,,PyUnicode_DecodeFSDefaultAndSize()} if you know the string length.


Changed in version 3.6: Use @code{Py_FileSystemDefaultEncodeErrors} error handler.

@end deffn

@geindex PyUnicode_EncodeFSDefault (C function)
@anchor{c-api/unicode c PyUnicode_EncodeFSDefault}@anchor{130e}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_EncodeFSDefault (PyObject  *unicode)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Encode a Unicode object to @code{Py_FileSystemDefaultEncoding} with the
@code{Py_FileSystemDefaultEncodeErrors} error handler, and return
@ref{2f0,,bytes}. Note that the resulting @ref{2f0,,bytes} object may contain
null bytes.

If @code{Py_FileSystemDefaultEncoding} is not set, fall back to the
locale encoding.

@code{Py_FileSystemDefaultEncoding} is initialized at startup from the
locale encoding and cannot be modified later. If you need to encode a string
to the current locale encoding, use @ref{701,,PyUnicode_EncodeLocale()}.


@subsubheading See also


The @ref{6ff,,Py_EncodeLocale()} function.



New in version 3.2.



Changed in version 3.6: Use @code{Py_FileSystemDefaultEncodeErrors} error handler.

@end deffn

@node wchar_t Support,,File System Encoding,Unicode Objects
@anchor{c-api/unicode wchar-t-support}@anchor{4401}
@subsubsection wchar_t Support


@code{wchar_t} support for platforms which support it:

@geindex PyUnicode_FromWideChar (C function)
@anchor{c-api/unicode c PyUnicode_FromWideChar}@anchor{d2d}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_FromWideChar (const  wchar_t  *w, Py_ssize_t  size)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a Unicode object from the @code{wchar_t} buffer @emph{w} of the given @emph{size}.
Passing @code{-1} as the @emph{size} indicates that the function must itself compute the length,
using wcslen.
Return @code{NULL} on failure.
@end deffn

@geindex PyUnicode_AsWideChar (C function)
@anchor{c-api/unicode c PyUnicode_AsWideChar}@anchor{1315}
@deffn {C Function} Py_ssize_t  PyUnicode_AsWideChar (PyObject  *unicode, wchar_t  *w, Py_ssize_t  size)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Copy the Unicode object contents into the @code{wchar_t} buffer @emph{w}.  At most
@emph{size} @code{wchar_t} characters are copied (excluding a possibly trailing
null termination character).  Return the number of @code{wchar_t} characters
copied or @code{-1} in case of an error.  Note that the resulting wchar_t*
string may or may not be null-terminated.  It is the responsibility of the caller
to make sure that the wchar_t* string is null-terminated in case this is
required by the application. Also, note that the wchar_t* string
might contain null characters, which would cause the string to be truncated
when used with most C functions.
@end deffn

@geindex PyUnicode_AsWideCharString (C function)
@anchor{c-api/unicode c PyUnicode_AsWideCharString}@anchor{39f}
@deffn {C Function} wchar_t  *PyUnicode_AsWideCharString (PyObject  *unicode, Py_ssize_t  *size)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Convert the Unicode object to a wide character string. The output string
always ends with a null character. If @emph{size} is not @code{NULL}, write the number
of wide characters (excluding the trailing null termination character) into
@emph{*size}. Note that the resulting @code{wchar_t} string might contain
null characters, which would cause the string to be truncated when used with
most C functions. If @emph{size} is @code{NULL} and the wchar_t* string
contains null characters a @ref{17c,,ValueError} is raised.

Returns a buffer allocated by @code{PyMem_Alloc()} (use
@ref{fa3,,PyMem_Free()} to free it) on success. On error, returns @code{NULL}
and @emph{*size} is undefined. Raises a @ref{10de,,MemoryError} if memory allocation
is failed.


New in version 3.2.



Changed in version 3.7: Raises a @ref{17c,,ValueError} if @emph{size} is @code{NULL} and the wchar_t*
string contains null characters.

@end deffn

@node Built-in Codecs,Methods and Slot Functions,Unicode Objects,Unicode Objects and Codecs
@anchor{c-api/unicode built-in-codecs}@anchor{4402}@anchor{c-api/unicode builtincodecs}@anchor{43fd}
@subsubsection Built-in Codecs


Python provides a set of built-in codecs which are written in C for speed. All of
these codecs are directly usable via the following functions.

Many of the following APIs take two arguments encoding and errors, and they
have the same semantics as the ones of the built-in @ref{2ef,,str()} string object
constructor.

Setting encoding to @code{NULL} causes the default encoding to be used
which is UTF-8.  The file system calls should use
@ref{867,,PyUnicode_FSConverter()} for encoding file names. This uses the
variable @code{Py_FileSystemDefaultEncoding} internally. This
variable should be treated as read-only: on some systems, it will be a
pointer to a static string, on others, it will change at run-time
(such as when the application invokes setlocale).

Error handling is set by errors which may also be set to @code{NULL} meaning to use
the default handling defined for the codec.  Default error handling for all
built-in codecs is “strict” (@ref{17c,,ValueError} is raised).

The codecs all use a similar interface.  Only deviations from the following
generic ones are documented for simplicity.

@menu
* Generic Codecs:: 
* UTF-8 Codecs:: 
* UTF-32 Codecs:: 
* UTF-16 Codecs:: 
* UTF-7 Codecs:: 
* Unicode-Escape Codecs:: 
* Raw-Unicode-Escape Codecs:: 
* Latin-1 Codecs:: 
* ASCII Codecs:: 
* Character Map Codecs:: 
* MBCS codecs for Windows:: 
* Methods & Slots:: 

@end menu

@node Generic Codecs,UTF-8 Codecs,,Built-in Codecs
@anchor{c-api/unicode generic-codecs}@anchor{4403}
@subsubsection Generic Codecs


These are the generic codec APIs:

@geindex PyUnicode_Decode (C function)
@anchor{c-api/unicode c PyUnicode_Decode}@anchor{1284}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_Decode (const  char  *s, Py_ssize_t  size, const  char  *encoding, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a Unicode object by decoding @emph{size} bytes of the encoded string @emph{s}.
@emph{encoding} and @emph{errors} have the same meaning as the parameters of the same name
in the @ref{2ef,,str()} built-in function.  The codec to be used is looked up
using the Python codec registry.  Return @code{NULL} if an exception was raised by
the codec.
@end deffn

@geindex PyUnicode_AsEncodedString (C function)
@anchor{c-api/unicode c PyUnicode_AsEncodedString}@anchor{426f}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_AsEncodedString (PyObject  *unicode, const  char  *encoding, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Encode a Unicode object and return the result as Python bytes object.
@emph{encoding} and @emph{errors} have the same meaning as the parameters of the same
name in the Unicode @ref{3ba,,encode()} method. The codec to be used is looked up
using the Python codec registry. Return @code{NULL} if an exception was raised by
the codec.
@end deffn

@node UTF-8 Codecs,UTF-32 Codecs,Generic Codecs,Built-in Codecs
@anchor{c-api/unicode utf-8-codecs}@anchor{4404}
@subsubsection UTF-8 Codecs


These are the UTF-8 codec APIs:

@geindex PyUnicode_DecodeUTF8 (C function)
@anchor{c-api/unicode c PyUnicode_DecodeUTF8}@anchor{4283}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeUTF8 (const  char  *s, Py_ssize_t  size, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a Unicode object by decoding @emph{size} bytes of the UTF-8 encoded string
@emph{s}. Return @code{NULL} if an exception was raised by the codec.
@end deffn

@geindex PyUnicode_DecodeUTF8Stateful (C function)
@anchor{c-api/unicode c PyUnicode_DecodeUTF8Stateful}@anchor{4284}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeUTF8Stateful (const  char  *s, Py_ssize_t  size, const  char  *errors, Py_ssize_t  *consumed)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
If @emph{consumed} is @code{NULL}, behave like @ref{4283,,PyUnicode_DecodeUTF8()}. If
@emph{consumed} is not @code{NULL}, trailing incomplete UTF-8 byte sequences will not be
treated as an error. Those bytes will not be decoded and the number of bytes
that have been decoded will be stored in @emph{consumed}.
@end deffn

@geindex PyUnicode_AsUTF8String (C function)
@anchor{c-api/unicode c PyUnicode_AsUTF8String}@anchor{d34}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_AsUTF8String (PyObject  *unicode)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Encode a Unicode object using UTF-8 and return the result as Python bytes
object.  Error handling is “strict”.  Return @code{NULL} if an exception was
raised by the codec.
@end deffn

@geindex PyUnicode_AsUTF8AndSize (C function)
@anchor{c-api/unicode c PyUnicode_AsUTF8AndSize}@anchor{367}
@deffn {C Function} const  char  *PyUnicode_AsUTF8AndSize (PyObject  *unicode, Py_ssize_t  *size)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Return a pointer to the UTF-8 encoding of the Unicode object, and
store the size of the encoded representation (in bytes) in @emph{size}.  The
@emph{size} argument can be @code{NULL}; in this case no size will be stored.  The
returned buffer always has an extra null byte appended (not included in
@emph{size}), regardless of whether there are any other null code points.

In the case of an error, @code{NULL} is returned with an exception set and no
@emph{size} is stored.

This caches the UTF-8 representation of the string in the Unicode object, and
subsequent calls will return a pointer to the same buffer.  The caller is not
responsible for deallocating the buffer.


New in version 3.3.



Changed in version 3.7: The return type is now @code{const char *} rather of @code{char *}.



Changed in version 3.10: This function is a part of the @ref{35d,,limited API}.

@end deffn

@geindex PyUnicode_AsUTF8 (C function)
@anchor{c-api/unicode c PyUnicode_AsUTF8}@anchor{6ef}
@deffn {C Function} const  char  *PyUnicode_AsUTF8 (PyObject  *unicode)

As @ref{367,,PyUnicode_AsUTF8AndSize()}, but does not store the size.


New in version 3.3.



Changed in version 3.7: The return type is now @code{const char *} rather of @code{char *}.

@end deffn

@node UTF-32 Codecs,UTF-16 Codecs,UTF-8 Codecs,Built-in Codecs
@anchor{c-api/unicode utf-32-codecs}@anchor{4405}
@subsubsection UTF-32 Codecs


These are the UTF-32 codec APIs:

@geindex PyUnicode_DecodeUTF32 (C function)
@anchor{c-api/unicode c PyUnicode_DecodeUTF32}@anchor{427f}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeUTF32 (const  char  *s, Py_ssize_t  size, const  char  *errors, int  *byteorder)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Decode @emph{size} bytes from a UTF-32 encoded buffer string and return the
corresponding Unicode object.  @emph{errors} (if non-@code{NULL}) defines the error
handling. It defaults to “strict”.

If @emph{byteorder} is non-@code{NULL}, the decoder starts decoding using the given byte
order:

@example
*byteorder == -1: little endian
*byteorder == 0:  native order
*byteorder == 1:  big endian
@end example

If @code{*byteorder} is zero, and the first four bytes of the input data are a
byte order mark (BOM), the decoder switches to this byte order and the BOM is
not copied into the resulting Unicode string.  If @code{*byteorder} is @code{-1} or
@code{1}, any byte order mark is copied to the output.

After completion, @emph{*byteorder} is set to the current byte order at the end
of input data.

If @emph{byteorder} is @code{NULL}, the codec starts in native order mode.

Return @code{NULL} if an exception was raised by the codec.
@end deffn

@geindex PyUnicode_DecodeUTF32Stateful (C function)
@anchor{c-api/unicode c PyUnicode_DecodeUTF32Stateful}@anchor{4280}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeUTF32Stateful (const  char  *s, Py_ssize_t  size, const  char  *errors, int  *byteorder, Py_ssize_t  *consumed)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
If @emph{consumed} is @code{NULL}, behave like @ref{427f,,PyUnicode_DecodeUTF32()}. If
@emph{consumed} is not @code{NULL}, @ref{4280,,PyUnicode_DecodeUTF32Stateful()} will not treat
trailing incomplete UTF-32 byte sequences (such as a number of bytes not divisible
by four) as an error. Those bytes will not be decoded and the number of bytes
that have been decoded will be stored in @emph{consumed}.
@end deffn

@geindex PyUnicode_AsUTF32String (C function)
@anchor{c-api/unicode c PyUnicode_AsUTF32String}@anchor{4271}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_AsUTF32String (PyObject  *unicode)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a Python byte string using the UTF-32 encoding in native byte
order. The string always starts with a BOM mark.  Error handling is “strict”.
Return @code{NULL} if an exception was raised by the codec.
@end deffn

@node UTF-16 Codecs,UTF-7 Codecs,UTF-32 Codecs,Built-in Codecs
@anchor{c-api/unicode utf-16-codecs}@anchor{4406}
@subsubsection UTF-16 Codecs


These are the UTF-16 codec APIs:

@geindex PyUnicode_DecodeUTF16 (C function)
@anchor{c-api/unicode c PyUnicode_DecodeUTF16}@anchor{427d}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeUTF16 (const  char  *s, Py_ssize_t  size, const  char  *errors, int  *byteorder)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Decode @emph{size} bytes from a UTF-16 encoded buffer string and return the
corresponding Unicode object.  @emph{errors} (if non-@code{NULL}) defines the error
handling. It defaults to “strict”.

If @emph{byteorder} is non-@code{NULL}, the decoder starts decoding using the given byte
order:

@example
*byteorder == -1: little endian
*byteorder == 0:  native order
*byteorder == 1:  big endian
@end example

If @code{*byteorder} is zero, and the first two bytes of the input data are a
byte order mark (BOM), the decoder switches to this byte order and the BOM is
not copied into the resulting Unicode string.  If @code{*byteorder} is @code{-1} or
@code{1}, any byte order mark is copied to the output (where it will result in
either a @code{\ufeff} or a @code{\ufffe} character).

After completion, @code{*byteorder} is set to the current byte order at the end
of input data.

If @emph{byteorder} is @code{NULL}, the codec starts in native order mode.

Return @code{NULL} if an exception was raised by the codec.
@end deffn

@geindex PyUnicode_DecodeUTF16Stateful (C function)
@anchor{c-api/unicode c PyUnicode_DecodeUTF16Stateful}@anchor{427e}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeUTF16Stateful (const  char  *s, Py_ssize_t  size, const  char  *errors, int  *byteorder, Py_ssize_t  *consumed)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
If @emph{consumed} is @code{NULL}, behave like @ref{427d,,PyUnicode_DecodeUTF16()}. If
@emph{consumed} is not @code{NULL}, @ref{427e,,PyUnicode_DecodeUTF16Stateful()} will not treat
trailing incomplete UTF-16 byte sequences (such as an odd number of bytes or a
split surrogate pair) as an error. Those bytes will not be decoded and the
number of bytes that have been decoded will be stored in @emph{consumed}.
@end deffn

@geindex PyUnicode_AsUTF16String (C function)
@anchor{c-api/unicode c PyUnicode_AsUTF16String}@anchor{4270}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_AsUTF16String (PyObject  *unicode)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a Python byte string using the UTF-16 encoding in native byte
order. The string always starts with a BOM mark.  Error handling is “strict”.
Return @code{NULL} if an exception was raised by the codec.
@end deffn

@node UTF-7 Codecs,Unicode-Escape Codecs,UTF-16 Codecs,Built-in Codecs
@anchor{c-api/unicode utf-7-codecs}@anchor{4407}
@subsubsection UTF-7 Codecs


These are the UTF-7 codec APIs:

@geindex PyUnicode_DecodeUTF7 (C function)
@anchor{c-api/unicode c PyUnicode_DecodeUTF7}@anchor{4281}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeUTF7 (const  char  *s, Py_ssize_t  size, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a Unicode object by decoding @emph{size} bytes of the UTF-7 encoded string
@emph{s}.  Return @code{NULL} if an exception was raised by the codec.
@end deffn

@geindex PyUnicode_DecodeUTF7Stateful (C function)
@anchor{c-api/unicode c PyUnicode_DecodeUTF7Stateful}@anchor{4282}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeUTF7Stateful (const  char  *s, Py_ssize_t  size, const  char  *errors, Py_ssize_t  *consumed)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
If @emph{consumed} is @code{NULL}, behave like @ref{4281,,PyUnicode_DecodeUTF7()}.  If
@emph{consumed} is not @code{NULL}, trailing incomplete UTF-7 base-64 sections will not
be treated as an error.  Those bytes will not be decoded and the number of
bytes that have been decoded will be stored in @emph{consumed}.
@end deffn

@node Unicode-Escape Codecs,Raw-Unicode-Escape Codecs,UTF-7 Codecs,Built-in Codecs
@anchor{c-api/unicode unicode-escape-codecs}@anchor{4408}
@subsubsection Unicode-Escape Codecs


These are the “Unicode Escape” codec APIs:

@geindex PyUnicode_DecodeUnicodeEscape (C function)
@anchor{c-api/unicode c PyUnicode_DecodeUnicodeEscape}@anchor{4285}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeUnicodeEscape (const  char  *s, Py_ssize_t  size, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a Unicode object by decoding @emph{size} bytes of the Unicode-Escape encoded
string @emph{s}.  Return @code{NULL} if an exception was raised by the codec.
@end deffn

@geindex PyUnicode_AsUnicodeEscapeString (C function)
@anchor{c-api/unicode c PyUnicode_AsUnicodeEscapeString}@anchor{d35}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_AsUnicodeEscapeString (PyObject  *unicode)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Encode a Unicode object using Unicode-Escape and return the result as a
bytes object.  Error handling is “strict”.  Return @code{NULL} if an exception was
raised by the codec.
@end deffn

@node Raw-Unicode-Escape Codecs,Latin-1 Codecs,Unicode-Escape Codecs,Built-in Codecs
@anchor{c-api/unicode raw-unicode-escape-codecs}@anchor{4409}
@subsubsection Raw-Unicode-Escape Codecs


These are the “Raw Unicode Escape” codec APIs:

@geindex PyUnicode_DecodeRawUnicodeEscape (C function)
@anchor{c-api/unicode c PyUnicode_DecodeRawUnicodeEscape}@anchor{427c}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeRawUnicodeEscape (const  char  *s, Py_ssize_t  size, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a Unicode object by decoding @emph{size} bytes of the Raw-Unicode-Escape
encoded string @emph{s}.  Return @code{NULL} if an exception was raised by the codec.
@end deffn

@geindex PyUnicode_AsRawUnicodeEscapeString (C function)
@anchor{c-api/unicode c PyUnicode_AsRawUnicodeEscapeString}@anchor{d36}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_AsRawUnicodeEscapeString (PyObject  *unicode)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Encode a Unicode object using Raw-Unicode-Escape and return the result as
a bytes object.  Error handling is “strict”.  Return @code{NULL} if an exception
was raised by the codec.
@end deffn

@node Latin-1 Codecs,ASCII Codecs,Raw-Unicode-Escape Codecs,Built-in Codecs
@anchor{c-api/unicode latin-1-codecs}@anchor{440a}
@subsubsection Latin-1 Codecs


These are the Latin-1 codec APIs: Latin-1 corresponds to the first 256 Unicode
ordinals and only these are accepted by the codecs during encoding.

@geindex PyUnicode_DecodeLatin1 (C function)
@anchor{c-api/unicode c PyUnicode_DecodeLatin1}@anchor{4278}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeLatin1 (const  char  *s, Py_ssize_t  size, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a Unicode object by decoding @emph{size} bytes of the Latin-1 encoded string
@emph{s}.  Return @code{NULL} if an exception was raised by the codec.
@end deffn

@geindex PyUnicode_AsLatin1String (C function)
@anchor{c-api/unicode c PyUnicode_AsLatin1String}@anchor{d37}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_AsLatin1String (PyObject  *unicode)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Encode a Unicode object using Latin-1 and return the result as Python bytes
object.  Error handling is “strict”.  Return @code{NULL} if an exception was
raised by the codec.
@end deffn

@node ASCII Codecs,Character Map Codecs,Latin-1 Codecs,Built-in Codecs
@anchor{c-api/unicode ascii-codecs}@anchor{440b}
@subsubsection ASCII Codecs


These are the ASCII codec APIs.  Only 7-bit ASCII data is accepted. All other
codes generate errors.

@geindex PyUnicode_DecodeASCII (C function)
@anchor{c-api/unicode c PyUnicode_DecodeASCII}@anchor{4274}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeASCII (const  char  *s, Py_ssize_t  size, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a Unicode object by decoding @emph{size} bytes of the ASCII encoded string
@emph{s}.  Return @code{NULL} if an exception was raised by the codec.
@end deffn

@geindex PyUnicode_AsASCIIString (C function)
@anchor{c-api/unicode c PyUnicode_AsASCIIString}@anchor{d38}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_AsASCIIString (PyObject  *unicode)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Encode a Unicode object using ASCII and return the result as Python bytes
object.  Error handling is “strict”.  Return @code{NULL} if an exception was
raised by the codec.
@end deffn

@node Character Map Codecs,MBCS codecs for Windows,ASCII Codecs,Built-in Codecs
@anchor{c-api/unicode character-map-codecs}@anchor{440c}
@subsubsection Character Map Codecs


This codec is special in that it can be used to implement many different codecs
(and this is in fact what was done to obtain most of the standard codecs
included in the @code{encodings} package). The codec uses mappings to encode and
decode characters.  The mapping objects provided must support the
@code{__getitem__()} mapping interface; dictionaries and sequences work well.

These are the mapping codec APIs:

@geindex PyUnicode_DecodeCharmap (C function)
@anchor{c-api/unicode c PyUnicode_DecodeCharmap}@anchor{4275}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeCharmap (const  char  *data, Py_ssize_t  size, PyObject  *mapping, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a Unicode object by decoding @emph{size} bytes of the encoded string @emph{s}
using the given @emph{mapping} object.  Return @code{NULL} if an exception was raised
by the codec.

If @emph{mapping} is @code{NULL}, Latin-1 decoding will be applied.  Else
@emph{mapping} must map bytes ordinals (integers in the range from 0 to 255)
to Unicode strings, integers (which are then interpreted as Unicode
ordinals) or @code{None}.  Unmapped data bytes – ones which cause a
@ref{1ab5,,LookupError}, as well as ones which get mapped to @code{None},
@code{0xFFFE} or @code{'\ufffe'}, are treated as undefined mappings and cause
an error.
@end deffn

@geindex PyUnicode_AsCharmapString (C function)
@anchor{c-api/unicode c PyUnicode_AsCharmapString}@anchor{426e}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_AsCharmapString (PyObject  *unicode, PyObject  *mapping)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Encode a Unicode object using the given @emph{mapping} object and return the
result as a bytes object.  Error handling is “strict”.  Return @code{NULL} if an
exception was raised by the codec.

The @emph{mapping} object must map Unicode ordinal integers to bytes objects,
integers in the range from 0 to 255 or @code{None}.  Unmapped character
ordinals (ones which cause a @ref{1ab5,,LookupError}) as well as mapped to
@code{None} are treated as “undefined mapping” and cause an error.
@end deffn

The following codec API is special in that maps Unicode to Unicode.

@geindex PyUnicode_Translate (C function)
@anchor{c-api/unicode c PyUnicode_Translate}@anchor{4291}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_Translate (PyObject  *str, PyObject  *table, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Translate a string by applying a character mapping table to it and return the
resulting Unicode object. Return @code{NULL} if an exception was raised by the
codec.

The mapping table must map Unicode ordinal integers to Unicode ordinal integers
or @code{None} (causing deletion of the character).

Mapping tables need only provide the @code{__getitem__()} interface; dictionaries
and sequences work well.  Unmapped character ordinals (ones which cause a
@ref{1ab5,,LookupError}) are left untouched and are copied as-is.

@emph{errors} has the usual meaning for codecs. It may be @code{NULL} which indicates to
use the default error handling.
@end deffn

@node MBCS codecs for Windows,Methods & Slots,Character Map Codecs,Built-in Codecs
@anchor{c-api/unicode mbcs-codecs-for-windows}@anchor{440d}
@subsubsection MBCS codecs for Windows


These are the MBCS codec APIs. They are currently only available on Windows and
use the Win32 MBCS converters to implement the conversions.  Note that MBCS (or
DBCS) is a class of encodings, not just one.  The target encoding is defined by
the user settings on the machine running the codec.

@geindex PyUnicode_DecodeMBCS (C function)
@anchor{c-api/unicode c PyUnicode_DecodeMBCS}@anchor{427a}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeMBCS (const  char  *s, Py_ssize_t  size, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} on Windows since version 3.7.}
Create a Unicode object by decoding @emph{size} bytes of the MBCS encoded string @emph{s}.
Return @code{NULL} if an exception was raised by the codec.
@end deffn

@geindex PyUnicode_DecodeMBCSStateful (C function)
@anchor{c-api/unicode c PyUnicode_DecodeMBCSStateful}@anchor{427b}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_DecodeMBCSStateful (const  char  *s, Py_ssize_t  size, const  char  *errors, Py_ssize_t  *consumed)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} on Windows since version 3.7.}
If @emph{consumed} is @code{NULL}, behave like @ref{427a,,PyUnicode_DecodeMBCS()}. If
@emph{consumed} is not @code{NULL}, @ref{427b,,PyUnicode_DecodeMBCSStateful()} will not decode
trailing lead byte and the number of bytes that have been decoded will be stored
in @emph{consumed}.
@end deffn

@geindex PyUnicode_AsMBCSString (C function)
@anchor{c-api/unicode c PyUnicode_AsMBCSString}@anchor{d39}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_AsMBCSString (PyObject  *unicode)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} on Windows since version 3.7.}
Encode a Unicode object using MBCS and return the result as Python bytes
object.  Error handling is “strict”.  Return @code{NULL} if an exception was
raised by the codec.
@end deffn

@geindex PyUnicode_EncodeCodePage (C function)
@anchor{c-api/unicode c PyUnicode_EncodeCodePage}@anchor{d3a}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_EncodeCodePage (int  code_page, PyObject  *unicode, const  char  *errors)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} on Windows since version 3.7.}
Encode the Unicode object using the specified code page and return a Python
bytes object.  Return @code{NULL} if an exception was raised by the codec. Use
@code{CP_ACP} code page to get the MBCS encoder.


New in version 3.3.

@end deffn

@node Methods & Slots,,MBCS codecs for Windows,Built-in Codecs
@anchor{c-api/unicode methods-slots}@anchor{440e}
@subsubsection Methods & Slots


@node Methods and Slot Functions,,Built-in Codecs,Unicode Objects and Codecs
@anchor{c-api/unicode methods-and-slot-functions}@anchor{440f}@anchor{c-api/unicode unicodemethodsandslots}@anchor{4410}
@subsubsection Methods and Slot Functions


The following APIs are capable of handling Unicode objects and strings on input
(we refer to them as strings in the descriptions) and return Unicode objects or
integers as appropriate.

They all return @code{NULL} or @code{-1} if an exception occurs.

@geindex PyUnicode_Concat (C function)
@anchor{c-api/unicode c PyUnicode_Concat}@anchor{d47}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_Concat (PyObject  *left, PyObject  *right)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Concat two strings giving a new Unicode string.
@end deffn

@geindex PyUnicode_Split (C function)
@anchor{c-api/unicode c PyUnicode_Split}@anchor{428f}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_Split (PyObject  *s, PyObject  *sep, Py_ssize_t  maxsplit)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Split a string giving a list of Unicode strings.  If @emph{sep} is @code{NULL}, splitting
will be done at all whitespace substrings.  Otherwise, splits occur at the given
separator.  At most @emph{maxsplit} splits will be done.  If negative, no limit is
set.  Separators are not included in the resulting list.
@end deffn

@geindex PyUnicode_Splitlines (C function)
@anchor{c-api/unicode c PyUnicode_Splitlines}@anchor{4290}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_Splitlines (PyObject  *s, int  keepend)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Split a Unicode string at line breaks, returning a list of Unicode strings.
CRLF is considered to be one line break.  If @emph{keepend} is @code{0}, the line break
characters are not included in the resulting strings.
@end deffn

@geindex PyUnicode_Join (C function)
@anchor{c-api/unicode c PyUnicode_Join}@anchor{d48}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_Join (PyObject  *separator, PyObject  *seq)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Join a sequence of strings using the given @emph{separator} and return the resulting
Unicode string.
@end deffn

@geindex PyUnicode_Tailmatch (C function)
@anchor{c-api/unicode c PyUnicode_Tailmatch}@anchor{39b}
@deffn {C Function} Py_ssize_t  PyUnicode_Tailmatch (PyObject  *str, PyObject  *substr, Py_ssize_t  start, Py_ssize_t  end, int  direction)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return @code{1} if @emph{substr} matches @code{str[start:end]} at the given tail end
(@emph{direction} == @code{-1} means to do a prefix match, @emph{direction} == @code{1} a suffix match),
@code{0} otherwise. Return @code{-1} if an error occurred.
@end deffn

@geindex PyUnicode_Find (C function)
@anchor{c-api/unicode c PyUnicode_Find}@anchor{4286}
@deffn {C Function} Py_ssize_t  PyUnicode_Find (PyObject  *str, PyObject  *substr, Py_ssize_t  start, Py_ssize_t  end, int  direction)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the first position of @emph{substr} in @code{str[start:end]} using the given
@emph{direction} (@emph{direction} == @code{1} means to do a forward search, @emph{direction} == @code{-1} a
backward search).  The return value is the index of the first match; a value of
@code{-1} indicates that no match was found, and @code{-2} indicates that an error
occurred and an exception has been set.
@end deffn

@geindex PyUnicode_FindChar (C function)
@anchor{c-api/unicode c PyUnicode_FindChar}@anchor{39c}
@deffn {C Function} Py_ssize_t  PyUnicode_FindChar (PyObject  *str, Py_UCS4  ch, Py_ssize_t  start, Py_ssize_t  end, int  direction)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Return the first position of the character @emph{ch} in @code{str[start:end]} using
the given @emph{direction} (@emph{direction} == @code{1} means to do a forward search,
@emph{direction} == @code{-1} a backward search).  The return value is the index of the
first match; a value of @code{-1} indicates that no match was found, and @code{-2}
indicates that an error occurred and an exception has been set.


New in version 3.3.



Changed in version 3.7: @emph{start} and @emph{end} are now adjusted to behave like @code{str[start:end]}.

@end deffn

@geindex PyUnicode_Count (C function)
@anchor{c-api/unicode c PyUnicode_Count}@anchor{4273}
@deffn {C Function} Py_ssize_t  PyUnicode_Count (PyObject  *str, PyObject  *substr, Py_ssize_t  start, Py_ssize_t  end)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the number of non-overlapping occurrences of @emph{substr} in
@code{str[start:end]}.  Return @code{-1} if an error occurred.
@end deffn

@geindex PyUnicode_Replace (C function)
@anchor{c-api/unicode c PyUnicode_Replace}@anchor{428d}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_Replace (PyObject  *str, PyObject  *substr, PyObject  *replstr, Py_ssize_t  maxcount)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Replace at most @emph{maxcount} occurrences of @emph{substr} in @emph{str} with @emph{replstr} and
return the resulting Unicode object. @emph{maxcount} == @code{-1} means replace all
occurrences.
@end deffn

@geindex PyUnicode_Compare (C function)
@anchor{c-api/unicode c PyUnicode_Compare}@anchor{39a}
@deffn {C Function} int  PyUnicode_Compare (PyObject  *left, PyObject  *right)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Compare two strings and return @code{-1}, @code{0}, @code{1} for less than, equal, and greater than,
respectively.

This function returns @code{-1} upon failure, so one should call
@ref{116a,,PyErr_Occurred()} to check for errors.
@end deffn

@geindex PyUnicode_CompareWithASCIIString (C function)
@anchor{c-api/unicode c PyUnicode_CompareWithASCIIString}@anchor{e18}
@deffn {C Function} int  PyUnicode_CompareWithASCIIString (PyObject  *uni, const  char  *string)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Compare a Unicode object, @emph{uni}, with @emph{string} and return @code{-1}, @code{0}, @code{1} for less
than, equal, and greater than, respectively. It is best to pass only
ASCII-encoded strings, but the function interprets the input string as
ISO-8859-1 if it contains non-ASCII characters.

This function does not raise exceptions.
@end deffn

@geindex PyUnicode_RichCompare (C function)
@anchor{c-api/unicode c PyUnicode_RichCompare}@anchor{428e}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_RichCompare (PyObject  *left, PyObject  *right, int  op)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Rich compare two Unicode strings and return one of the following:


@itemize *

@item 
@code{NULL} in case an exception was raised

@item 
@code{Py_True} or @code{Py_False} for successful comparisons

@item 
@code{Py_NotImplemented} in case the type combination is unknown
@end itemize

Possible values for @emph{op} are @code{Py_GT}, @code{Py_GE}, @code{Py_EQ},
@code{Py_NE}, @code{Py_LT}, and @code{Py_LE}.
@end deffn

@geindex PyUnicode_Format (C function)
@anchor{c-api/unicode c PyUnicode_Format}@anchor{4287}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_Format (PyObject  *format, PyObject  *args)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new string object from @emph{format} and @emph{args}; this is analogous to
@code{format % args}.
@end deffn

@geindex PyUnicode_Contains (C function)
@anchor{c-api/unicode c PyUnicode_Contains}@anchor{4272}
@deffn {C Function} int  PyUnicode_Contains (PyObject  *container, PyObject  *element)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Check whether @emph{element} is contained in @emph{container} and return true or false
accordingly.

@emph{element} has to coerce to a one element Unicode string. @code{-1} is returned
if there was an error.
@end deffn

@geindex PyUnicode_InternInPlace (C function)
@anchor{c-api/unicode c PyUnicode_InternInPlace}@anchor{393}
@deffn {C Function} void  PyUnicode_InternInPlace (PyObject  **string)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Intern the argument @emph{*string} in place.  The argument must be the address of a
pointer variable pointing to a Python Unicode string object.  If there is an
existing interned string that is the same as @emph{*string}, it sets @emph{*string} to
it (decrementing the reference count of the old string object and incrementing
the reference count of the interned string object), otherwise it leaves
@emph{*string} alone and interns it (incrementing its reference count).
(Clarification: even though there is a lot of talk about reference counts, think
of this function as reference-count-neutral; you own the object after the call
if and only if you owned it before the call.)
@end deffn

@geindex PyUnicode_InternFromString (C function)
@anchor{c-api/unicode c PyUnicode_InternFromString}@anchor{428c}
@deffn {C Function} @ref{375,,PyObject}  *PyUnicode_InternFromString (const  char  *v)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
A combination of @ref{40f4,,PyUnicode_FromString()} and
@ref{393,,PyUnicode_InternInPlace()}, returning either a new Unicode string
object that has been interned, or a new (“owned”) reference to an earlier
interned string object with the same value.
@end deffn

@node Tuple Objects,Struct Sequence Objects,Unicode Objects and Codecs,Sequence Objects
@anchor{c-api/tuple doc}@anchor{4411}@anchor{c-api/tuple tuple-objects}@anchor{4412}@anchor{c-api/tuple tupleobjects}@anchor{4413}
@subsubsection Tuple Objects


@geindex object; tuple

@geindex PyTupleObject (C type)
@anchor{c-api/tuple c PyTupleObject}@anchor{4302}
@deffn {C Type} type  PyTupleObject

This subtype of @ref{375,,PyObject} represents a Python tuple object.
@end deffn

@geindex PyTuple_Type (C var)
@anchor{c-api/tuple c PyTuple_Type}@anchor{424f}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyTuple_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
This instance of @ref{5c6,,PyTypeObject} represents the Python tuple type; it
is the same object as @ref{3e5,,tuple} in the Python layer.
@end deffn

@geindex PyTuple_Check (C function)
@anchor{c-api/tuple c PyTuple_Check}@anchor{4414}
@deffn {C Function} int  PyTuple_Check (PyObject  *p)

Return true if @emph{p} is a tuple object or an instance of a subtype of the
tuple type.  This function always succeeds.
@end deffn

@geindex PyTuple_CheckExact (C function)
@anchor{c-api/tuple c PyTuple_CheckExact}@anchor{4415}
@deffn {C Function} int  PyTuple_CheckExact (PyObject  *p)

Return true if @emph{p} is a tuple object, but not an instance of a subtype of the
tuple type.  This function always succeeds.
@end deffn

@geindex PyTuple_New (C function)
@anchor{c-api/tuple c PyTuple_New}@anchor{424c}
@deffn {C Function} @ref{375,,PyObject}  *PyTuple_New (Py_ssize_t  len)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new tuple object of size @emph{len}, or @code{NULL} on failure.
@end deffn

@geindex PyTuple_Pack (C function)
@anchor{c-api/tuple c PyTuple_Pack}@anchor{424d}
@deffn {C Function} @ref{375,,PyObject}  *PyTuple_Pack (Py_ssize_t  n, ...)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new tuple object of size @emph{n}, or @code{NULL} on failure. The tuple values
are initialized to the subsequent @emph{n} C arguments pointing to Python objects.
@code{PyTuple_Pack(2, a, b)} is equivalent to @code{Py_BuildValue("(OO)", a, b)}.
@end deffn

@geindex PyTuple_Size (C function)
@anchor{c-api/tuple c PyTuple_Size}@anchor{424e}
@deffn {C Function} Py_ssize_t  PyTuple_Size (PyObject  *p)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Take a pointer to a tuple object, and return the size of that tuple.
@end deffn

@geindex PyTuple_GET_SIZE (C function)
@anchor{c-api/tuple c PyTuple_GET_SIZE}@anchor{4416}
@deffn {C Function} Py_ssize_t  PyTuple_GET_SIZE (PyObject  *p)

Return the size of the tuple @emph{p}, which must be non-@code{NULL} and point to a tuple;
no error checking is performed.
@end deffn

@geindex PyTuple_GetItem (C function)
@anchor{c-api/tuple c PyTuple_GetItem}@anchor{408d}
@deffn {C Function} @ref{375,,PyObject}  *PyTuple_GetItem (PyObject  *p, Py_ssize_t  pos)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the object at position @emph{pos} in the tuple pointed to by @emph{p}.  If @emph{pos} is
negative or out of bounds, return @code{NULL} and set an @ref{1066,,IndexError} exception.
@end deffn

@geindex PyTuple_GET_ITEM (C function)
@anchor{c-api/tuple c PyTuple_GET_ITEM}@anchor{4417}
@deffn {C Function} @ref{375,,PyObject}  *PyTuple_GET_ITEM (PyObject  *p, Py_ssize_t  pos)
@emph{Return value: Borrowed reference.}
Like @ref{408d,,PyTuple_GetItem()}, but does no checking of its arguments.
@end deffn

@geindex PyTuple_GetSlice (C function)
@anchor{c-api/tuple c PyTuple_GetSlice}@anchor{424b}
@deffn {C Function} @ref{375,,PyObject}  *PyTuple_GetSlice (PyObject  *p, Py_ssize_t  low, Py_ssize_t  high)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the slice of the tuple pointed to by @emph{p} between @emph{low} and @emph{high},
or @code{NULL} on failure.  This is the equivalent of the Python expression
@code{p[low:high]}.  Indexing from the end of the list is not supported.
@end deffn

@geindex PyTuple_SetItem (C function)
@anchor{c-api/tuple c PyTuple_SetItem}@anchor{4091}
@deffn {C Function} int  PyTuple_SetItem (PyObject  *p, Py_ssize_t  pos, PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Insert a reference to object @emph{o} at position @emph{pos} of the tuple pointed to by
@emph{p}.  Return @code{0} on success.  If @emph{pos} is out of bounds, return @code{-1}
and set an @ref{1066,,IndexError} exception.

@cartouche
@quotation Note 
This function “steals” a reference to @emph{o} and discards a reference to
an item already in the tuple at the affected position.
@end quotation
@end cartouche
@end deffn

@geindex PyTuple_SET_ITEM (C function)
@anchor{c-api/tuple c PyTuple_SET_ITEM}@anchor{390}
@deffn {C Function} void  PyTuple_SET_ITEM (PyObject  *p, Py_ssize_t  pos, PyObject  *o)

Like @ref{4091,,PyTuple_SetItem()}, but does no error checking, and should @emph{only} be
used to fill in brand new tuples.

@cartouche
@quotation Note 
This macro “steals” a reference to @emph{o}, and, unlike
@ref{4091,,PyTuple_SetItem()}, does @emph{not} discard a reference to any item that
is being replaced; any reference in the tuple at position @emph{pos} will be
leaked.
@end quotation
@end cartouche
@end deffn

@geindex _PyTuple_Resize (C function)
@anchor{c-api/tuple c _PyTuple_Resize}@anchor{1041}
@deffn {C Function} int  _PyTuple_Resize (PyObject  **p, Py_ssize_t  newsize)

Can be used to resize a tuple.  @emph{newsize} will be the new length of the tuple.
Because tuples are @emph{supposed} to be immutable, this should only be used if there
is only one reference to the object.  Do @emph{not} use this if the tuple may already
be known to some other part of the code.  The tuple will always grow or shrink
at the end.  Think of this as destroying the old tuple and creating a new one,
only more efficiently.  Returns @code{0} on success. Client code should never
assume that the resulting value of @code{*p} will be the same as before calling
this function. If the object referenced by @code{*p} is replaced, the original
@code{*p} is destroyed.  On failure, returns @code{-1} and sets @code{*p} to @code{NULL}, and
raises @ref{10de,,MemoryError} or @ref{891,,SystemError}.
@end deffn

@node Struct Sequence Objects,List Objects,Tuple Objects,Sequence Objects
@anchor{c-api/tuple struct-sequence-objects}@anchor{4418}
@subsubsection Struct Sequence Objects


Struct sequence objects are the C equivalent of @ref{3e4,,namedtuple()}
objects, i.e. a sequence whose items can also be accessed through attributes.
To create a struct sequence, you first have to create a specific struct sequence
type.

@geindex PyStructSequence_NewType (C function)
@anchor{c-api/tuple c PyStructSequence_NewType}@anchor{4230}
@deffn {C Function} @ref{5c6,,PyTypeObject}  *PyStructSequence_NewType (PyStructSequence_Desc  *desc)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a new struct sequence type from the data in @emph{desc}, described below. Instances
of the resulting type can be created with @ref{422f,,PyStructSequence_New()}.
@end deffn

@geindex PyStructSequence_InitType (C function)
@anchor{c-api/tuple c PyStructSequence_InitType}@anchor{b89}
@deffn {C Function} void  PyStructSequence_InitType (PyTypeObject  *type, PyStructSequence_Desc  *desc)

Initializes a struct sequence type @emph{type} from @emph{desc} in place.
@end deffn

@geindex PyStructSequence_InitType2 (C function)
@anchor{c-api/tuple c PyStructSequence_InitType2}@anchor{b88}
@deffn {C Function} int  PyStructSequence_InitType2 (PyTypeObject  *type, PyStructSequence_Desc  *desc)

The same as @code{PyStructSequence_InitType}, but returns @code{0} on success and @code{-1} on
failure.


New in version 3.4.

@end deffn

@geindex PyStructSequence_Desc (C type)
@anchor{c-api/tuple c PyStructSequence_Desc}@anchor{6ee}
@deffn {C Type} type  PyStructSequence_Desc
@emph{ Part of the @ref{35d,,Stable ABI}.}
Contains the meta information of a struct sequence type to create.


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Field

@tab

C Type

@tab

Meaning

@item

@code{name}

@tab

@code{const char *}

@tab

name of the struct sequence type

@item

@code{doc}

@tab

@code{const char *}

@tab

pointer to docstring for the type
or @code{NULL} to omit

@item

@code{fields}

@tab

@code{PyStructSequence_Field *}

@tab

pointer to @code{NULL}-terminated array
with field names of the new type

@item

@code{n_in_sequence}

@tab

@code{int}

@tab

number of fields visible to the
Python side (if used as tuple)

@end multitable

@end deffn

@geindex PyStructSequence_Field (C type)
@anchor{c-api/tuple c PyStructSequence_Field}@anchor{6ed}
@deffn {C Type} type  PyStructSequence_Field
@emph{ Part of the @ref{35d,,Stable ABI}.}
Describes a field of a struct sequence. As a struct sequence is modeled as a
tuple, all fields are typed as @ref{375,,PyObject}*.  The index in the
@code{fields} array of the @ref{6ee,,PyStructSequence_Desc} determines which
field of the struct sequence is described.


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Field

@tab

C Type

@tab

Meaning

@item

@code{name}

@tab

@code{const char *}

@tab

name for the field or @code{NULL} to end
the list of named fields, set to
@ref{48b,,PyStructSequence_UnnamedField}
to leave unnamed

@item

@code{doc}

@tab

@code{const char *}

@tab

field docstring or @code{NULL} to omit

@end multitable

@end deffn

@geindex PyStructSequence_UnnamedField (C var)
@anchor{c-api/tuple c PyStructSequence_UnnamedField}@anchor{48b}
@deffn {C Variable} const  char  *const  PyStructSequence_UnnamedField
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Special value for a field name to leave it unnamed.


Changed in version 3.9: The type was changed from @code{char *}.

@end deffn

@geindex PyStructSequence_New (C function)
@anchor{c-api/tuple c PyStructSequence_New}@anchor{422f}
@deffn {C Function} @ref{375,,PyObject}  *PyStructSequence_New (PyTypeObject  *type)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Creates an instance of @emph{type}, which must have been created with
@ref{4230,,PyStructSequence_NewType()}.
@end deffn

@geindex PyStructSequence_GetItem (C function)
@anchor{c-api/tuple c PyStructSequence_GetItem}@anchor{422e}
@deffn {C Function} @ref{375,,PyObject}  *PyStructSequence_GetItem (PyObject  *p, Py_ssize_t  pos)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the object at position @emph{pos} in the struct sequence pointed to by @emph{p}.
No bounds checking is performed.
@end deffn

@geindex PyStructSequence_GET_ITEM (C function)
@anchor{c-api/tuple c PyStructSequence_GET_ITEM}@anchor{4419}
@deffn {C Function} @ref{375,,PyObject}  *PyStructSequence_GET_ITEM (PyObject  *p, Py_ssize_t  pos)
@emph{Return value: Borrowed reference.}
Macro equivalent of @ref{422e,,PyStructSequence_GetItem()}.
@end deffn

@geindex PyStructSequence_SetItem (C function)
@anchor{c-api/tuple c PyStructSequence_SetItem}@anchor{4231}
@deffn {C Function} void  PyStructSequence_SetItem (PyObject  *p, Py_ssize_t  pos, PyObject  *o)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Sets the field at index @emph{pos} of the struct sequence @emph{p} to value @emph{o}.  Like
@ref{390,,PyTuple_SET_ITEM()}, this should only be used to fill in brand new
instances.

@cartouche
@quotation Note 
This function “steals” a reference to @emph{o}.
@end quotation
@end cartouche
@end deffn

@geindex PyStructSequence_SET_ITEM (C function)
@anchor{c-api/tuple c PyStructSequence_SET_ITEM}@anchor{441a}
@deffn {C Function} void  PyStructSequence_SET_ITEM (PyObject  *p, Py_ssize_t  *pos, PyObject  *o)

Macro equivalent of @ref{4231,,PyStructSequence_SetItem()}.

@cartouche
@quotation Note 
This function “steals” a reference to @emph{o}.
@end quotation
@end cartouche
@end deffn

@node List Objects,,Struct Sequence Objects,Sequence Objects
@anchor{c-api/list doc}@anchor{441b}@anchor{c-api/list list-objects}@anchor{441c}@anchor{c-api/list listobjects}@anchor{441d}
@subsubsection List Objects


@geindex object; list

@geindex PyListObject (C type)
@anchor{c-api/list c PyListObject}@anchor{434b}
@deffn {C Type} type  PyListObject

This subtype of @ref{375,,PyObject} represents a Python list object.
@end deffn

@geindex PyList_Type (C var)
@anchor{c-api/list c PyList_Type}@anchor{40b8}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyList_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
This instance of @ref{5c6,,PyTypeObject} represents the Python list type.
This is the same object as @ref{43a,,list} in the Python layer.
@end deffn

@geindex PyList_Check (C function)
@anchor{c-api/list c PyList_Check}@anchor{441e}
@deffn {C Function} int  PyList_Check (PyObject  *p)

Return true if @emph{p} is a list object or an instance of a subtype of the list
type.  This function always succeeds.
@end deffn

@geindex PyList_CheckExact (C function)
@anchor{c-api/list c PyList_CheckExact}@anchor{441f}
@deffn {C Function} int  PyList_CheckExact (PyObject  *p)

Return true if @emph{p} is a list object, but not an instance of a subtype of
the list type.  This function always succeeds.
@end deffn

@geindex PyList_New (C function)
@anchor{c-api/list c PyList_New}@anchor{4114}
@deffn {C Function} @ref{375,,PyObject}  *PyList_New (Py_ssize_t  len)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new list of length @emph{len} on success, or @code{NULL} on failure.

@cartouche
@quotation Note 
If @emph{len} is greater than zero, the returned list object’s items are
set to @code{NULL}.  Thus you cannot use abstract API functions such as
@ref{1312,,PySequence_SetItem()}  or expose the object to Python code before
setting all items to a real object with @ref{4092,,PyList_SetItem()}.
@end quotation
@end cartouche
@end deffn

@geindex PyList_Size (C function)
@anchor{c-api/list c PyList_Size}@anchor{f78}
@deffn {C Function} Py_ssize_t  PyList_Size (PyObject  *list)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; len

Return the length of the list object in @emph{list}; this is equivalent to
@code{len(list)} on a list object.
@end deffn

@geindex PyList_GET_SIZE (C function)
@anchor{c-api/list c PyList_GET_SIZE}@anchor{4420}
@deffn {C Function} Py_ssize_t  PyList_GET_SIZE (PyObject  *list)

Macro form of @ref{f78,,PyList_Size()} without error checking.
@end deffn

@geindex PyList_GetItem (C function)
@anchor{c-api/list c PyList_GetItem}@anchor{408e}
@deffn {C Function} @ref{375,,PyObject}  *PyList_GetItem (PyObject  *list, Py_ssize_t  index)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the object at position @emph{index} in the list pointed to by @emph{list}.  The
position must be non-negative; indexing from the end of the list is not
supported.  If @emph{index} is out of bounds (<0 or >=len(list)),
return @code{NULL} and set an @ref{1066,,IndexError} exception.
@end deffn

@geindex PyList_GET_ITEM (C function)
@anchor{c-api/list c PyList_GET_ITEM}@anchor{4125}
@deffn {C Function} @ref{375,,PyObject}  *PyList_GET_ITEM (PyObject  *list, Py_ssize_t  i)
@emph{Return value: Borrowed reference.}
Macro form of @ref{408e,,PyList_GetItem()} without error checking.
@end deffn

@geindex PyList_SetItem (C function)
@anchor{c-api/list c PyList_SetItem}@anchor{4092}
@deffn {C Function} int  PyList_SetItem (PyObject  *list, Py_ssize_t  index, PyObject  *item)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the item at index @emph{index} in list to @emph{item}.  Return @code{0} on success.
If @emph{index} is out of bounds, return @code{-1} and set an @ref{1066,,IndexError}
exception.

@cartouche
@quotation Note 
This function “steals” a reference to @emph{item} and discards a reference to
an item already in the list at the affected position.
@end quotation
@end cartouche
@end deffn

@geindex PyList_SET_ITEM (C function)
@anchor{c-api/list c PyList_SET_ITEM}@anchor{38f}
@deffn {C Function} void  PyList_SET_ITEM (PyObject  *list, Py_ssize_t  i, PyObject  *o)

Macro form of @ref{4092,,PyList_SetItem()} without error checking. This is
normally only used to fill in new lists where there is no previous content.

@cartouche
@quotation Note 
This macro “steals” a reference to @emph{item}, and, unlike
@ref{4092,,PyList_SetItem()}, does @emph{not} discard a reference to any item that
is being replaced; any reference in @emph{list} at position @emph{i} will be
leaked.
@end quotation
@end cartouche
@end deffn

@geindex PyList_Insert (C function)
@anchor{c-api/list c PyList_Insert}@anchor{41af}
@deffn {C Function} int  PyList_Insert (PyObject  *list, Py_ssize_t  index, PyObject  *item)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Insert the item @emph{item} into list @emph{list} in front of index @emph{index}.  Return
@code{0} if successful; return @code{-1} and set an exception if unsuccessful.
Analogous to @code{list.insert(index, item)}.
@end deffn

@geindex PyList_Append (C function)
@anchor{c-api/list c PyList_Append}@anchor{41ac}
@deffn {C Function} int  PyList_Append (PyObject  *list, PyObject  *item)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Append the object @emph{item} at the end of list @emph{list}. Return @code{0} if
successful; return @code{-1} and set an exception if unsuccessful.  Analogous
to @code{list.append(item)}.
@end deffn

@geindex PyList_GetSlice (C function)
@anchor{c-api/list c PyList_GetSlice}@anchor{41ae}
@deffn {C Function} @ref{375,,PyObject}  *PyList_GetSlice (PyObject  *list, Py_ssize_t  low, Py_ssize_t  high)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a list of the objects in @emph{list} containing the objects @emph{between} @emph{low}
and @emph{high}.  Return @code{NULL} and set an exception if unsuccessful.  Analogous
to @code{list[low:high]}.  Indexing from the end of the list is not supported.
@end deffn

@geindex PyList_SetSlice (C function)
@anchor{c-api/list c PyList_SetSlice}@anchor{41b1}
@deffn {C Function} int  PyList_SetSlice (PyObject  *list, Py_ssize_t  low, Py_ssize_t  high, PyObject  *itemlist)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the slice of @emph{list} between @emph{low} and @emph{high} to the contents of
@emph{itemlist}.  Analogous to @code{list[low:high] = itemlist}. The @emph{itemlist} may
be @code{NULL}, indicating the assignment of an empty list (slice deletion).
Return @code{0} on success, @code{-1} on failure.  Indexing from the end of the
list is not supported.
@end deffn

@geindex PyList_Sort (C function)
@anchor{c-api/list c PyList_Sort}@anchor{41b2}
@deffn {C Function} int  PyList_Sort (PyObject  *list)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Sort the items of @emph{list} in place.  Return @code{0} on success, @code{-1} on
failure.  This is equivalent to @code{list.sort()}.
@end deffn

@geindex PyList_Reverse (C function)
@anchor{c-api/list c PyList_Reverse}@anchor{41b0}
@deffn {C Function} int  PyList_Reverse (PyObject  *list)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Reverse the items of @emph{list} in place.  Return @code{0} on success, @code{-1} on
failure.  This is the equivalent of @code{list.reverse()}.
@end deffn

@geindex PyList_AsTuple (C function)
@anchor{c-api/list c PyList_AsTuple}@anchor{41ad}
@deffn {C Function} @ref{375,,PyObject}  *PyList_AsTuple (PyObject  *list)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; tuple

Return a new tuple object containing the contents of @emph{list}; equivalent to
@code{tuple(list)}.
@end deffn

@node Container Objects,Function Objects<2>,Sequence Objects,Concrete Objects Layer
@anchor{c-api/concrete container-objects}@anchor{4421}@anchor{c-api/concrete mapobjects}@anchor{4422}
@subsection Container Objects


@geindex object; mapping

@menu
* Dictionary Objects:: 
* Set Objects:: 

@end menu

@node Dictionary Objects,Set Objects,,Container Objects
@anchor{c-api/dict doc}@anchor{4423}@anchor{c-api/dict dictionary-objects}@anchor{4424}@anchor{c-api/dict dictobjects}@anchor{4425}
@subsubsection Dictionary Objects


@geindex object; dictionary

@geindex PyDictObject (C type)
@anchor{c-api/dict c PyDictObject}@anchor{4426}
@deffn {C Type} type  PyDictObject

This subtype of @ref{375,,PyObject} represents a Python dictionary object.
@end deffn

@geindex PyDict_Type (C var)
@anchor{c-api/dict c PyDict_Type}@anchor{416e}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyDict_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
This instance of @ref{5c6,,PyTypeObject} represents the Python dictionary
type.  This is the same object as @ref{3b0,,dict} in the Python layer.
@end deffn

@geindex PyDict_Check (C function)
@anchor{c-api/dict c PyDict_Check}@anchor{438b}
@deffn {C Function} int  PyDict_Check (PyObject  *p)

Return true if @emph{p} is a dict object or an instance of a subtype of the dict
type.  This function always succeeds.
@end deffn

@geindex PyDict_CheckExact (C function)
@anchor{c-api/dict c PyDict_CheckExact}@anchor{4427}
@deffn {C Function} int  PyDict_CheckExact (PyObject  *p)

Return true if @emph{p} is a dict object, but not an instance of a subtype of
the dict type.  This function always succeeds.
@end deffn

@geindex PyDict_New (C function)
@anchor{c-api/dict c PyDict_New}@anchor{416a}
@deffn {C Function} @ref{375,,PyObject}  *PyDict_New ()
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new empty dictionary, or @code{NULL} on failure.
@end deffn

@geindex PyDictProxy_New (C function)
@anchor{c-api/dict c PyDictProxy_New}@anchor{415f}
@deffn {C Function} @ref{375,,PyObject}  *PyDictProxy_New (PyObject  *mapping)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a @ref{246,,types.MappingProxyType} object for a mapping which
enforces read-only behavior.  This is normally used to create a view to
prevent modification of the dictionary for non-dynamic class types.
@end deffn

@geindex PyDict_Clear (C function)
@anchor{c-api/dict c PyDict_Clear}@anchor{4160}
@deffn {C Function} void  PyDict_Clear (PyObject  *p)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Empty an existing dictionary of all key-value pairs.
@end deffn

@geindex PyDict_Contains (C function)
@anchor{c-api/dict c PyDict_Contains}@anchor{4161}
@deffn {C Function} int  PyDict_Contains (PyObject  *p, PyObject  *key)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Determine if dictionary @emph{p} contains @emph{key}.  If an item in @emph{p} is matches
@emph{key}, return @code{1}, otherwise return @code{0}.  On error, return @code{-1}.
This is equivalent to the Python expression @code{key in p}.
@end deffn

@geindex PyDict_Copy (C function)
@anchor{c-api/dict c PyDict_Copy}@anchor{4162}
@deffn {C Function} @ref{375,,PyObject}  *PyDict_Copy (PyObject  *p)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new dictionary that contains the same key-value pairs as @emph{p}.
@end deffn

@geindex PyDict_SetItem (C function)
@anchor{c-api/dict c PyDict_SetItem}@anchor{4093}
@deffn {C Function} int  PyDict_SetItem (PyObject  *p, PyObject  *key, PyObject  *val)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Insert @emph{val} into the dictionary @emph{p} with a key of @emph{key}.  @emph{key} must be
@ref{2dc,,hashable}; if it isn’t, @ref{162,,TypeError} will be raised. Return
@code{0} on success or @code{-1} on failure.  This function @emph{does not} steal a
reference to @emph{val}.
@end deffn

@geindex PyDict_SetItemString (C function)
@anchor{c-api/dict c PyDict_SetItemString}@anchor{416c}
@deffn {C Function} int  PyDict_SetItemString (PyObject  *p, const  char  *key, PyObject  *val)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex PyUnicode_FromString()

Insert @emph{val} into the dictionary @emph{p} using @emph{key} as a key. @emph{key} should
be a const  char*.  The key object is created using
@code{PyUnicode_FromString(key)}.  Return @code{0} on success or @code{-1} on
failure.  This function @emph{does not} steal a reference to @emph{val}.
@end deffn

@geindex PyDict_DelItem (C function)
@anchor{c-api/dict c PyDict_DelItem}@anchor{4163}
@deffn {C Function} int  PyDict_DelItem (PyObject  *p, PyObject  *key)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Remove the entry in dictionary @emph{p} with key @emph{key}. @emph{key} must be hashable;
if it isn’t, @ref{162,,TypeError} is raised.
If @emph{key} is not in the dictionary, @ref{5bb,,KeyError} is raised.
Return @code{0} on success or @code{-1} on failure.
@end deffn

@geindex PyDict_DelItemString (C function)
@anchor{c-api/dict c PyDict_DelItemString}@anchor{4164}
@deffn {C Function} int  PyDict_DelItemString (PyObject  *p, const  char  *key)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Remove the entry in dictionary @emph{p} which has a key specified by the string @emph{key}.
If @emph{key} is not in the dictionary, @ref{5bb,,KeyError} is raised.
Return @code{0} on success or @code{-1} on failure.
@end deffn

@geindex PyDict_GetItem (C function)
@anchor{c-api/dict c PyDict_GetItem}@anchor{384}
@deffn {C Function} @ref{375,,PyObject}  *PyDict_GetItem (PyObject  *p, PyObject  *key)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the object from dictionary @emph{p} which has a key @emph{key}.  Return @code{NULL}
if the key @emph{key} is not present, but @emph{without} setting an exception.

Note that exceptions which occur while calling @code{__hash__()} and
@code{__eq__()} methods will get suppressed.
To get error reporting use @ref{4165,,PyDict_GetItemWithError()} instead.


Changed in version 3.10: Calling this API without @ref{385,,GIL} held had been allowed for historical
reason. It is no longer allowed.

@end deffn

@geindex PyDict_GetItemWithError (C function)
@anchor{c-api/dict c PyDict_GetItemWithError}@anchor{4165}
@deffn {C Function} @ref{375,,PyObject}  *PyDict_GetItemWithError (PyObject  *p, PyObject  *key)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Variant of @ref{384,,PyDict_GetItem()} that does not suppress
exceptions. Return @code{NULL} @strong{with} an exception set if an exception
occurred.  Return @code{NULL} @strong{without} an exception set if the key
wasn’t present.
@end deffn

@geindex PyDict_GetItemString (C function)
@anchor{c-api/dict c PyDict_GetItemString}@anchor{408f}
@deffn {C Function} @ref{375,,PyObject}  *PyDict_GetItemString (PyObject  *p, const  char  *key)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
This is the same as @ref{384,,PyDict_GetItem()}, but @emph{key} is specified as a
const  char*, rather than a @ref{375,,PyObject}*.

Note that exceptions which occur while calling @code{__hash__()} and
@code{__eq__()} methods and creating a temporary string object
will get suppressed.
To get error reporting use @ref{4165,,PyDict_GetItemWithError()} instead.
@end deffn

@geindex PyDict_SetDefault (C function)
@anchor{c-api/dict c PyDict_SetDefault}@anchor{4428}
@deffn {C Function} @ref{375,,PyObject}  *PyDict_SetDefault (PyObject  *p, PyObject  *key, PyObject  *defaultobj)
@emph{Return value: Borrowed reference.}
This is the same as the Python-level @ref{c0f,,dict.setdefault()}.  If present, it
returns the value corresponding to @emph{key} from the dictionary @emph{p}.  If the key
is not in the dict, it is inserted with value @emph{defaultobj} and @emph{defaultobj}
is returned.  This function evaluates the hash function of @emph{key} only once,
instead of evaluating it independently for the lookup and the insertion.


New in version 3.4.

@end deffn

@geindex PyDict_Items (C function)
@anchor{c-api/dict c PyDict_Items}@anchor{4166}
@deffn {C Function} @ref{375,,PyObject}  *PyDict_Items (PyObject  *p)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a @ref{434b,,PyListObject} containing all the items from the dictionary.
@end deffn

@geindex PyDict_Keys (C function)
@anchor{c-api/dict c PyDict_Keys}@anchor{4167}
@deffn {C Function} @ref{375,,PyObject}  *PyDict_Keys (PyObject  *p)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a @ref{434b,,PyListObject} containing all the keys from the dictionary.
@end deffn

@geindex PyDict_Values (C function)
@anchor{c-api/dict c PyDict_Values}@anchor{4170}
@deffn {C Function} @ref{375,,PyObject}  *PyDict_Values (PyObject  *p)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a @ref{434b,,PyListObject} containing all the values from the dictionary
@emph{p}.
@end deffn

@geindex PyDict_Size (C function)
@anchor{c-api/dict c PyDict_Size}@anchor{416d}
@deffn {C Function} Py_ssize_t  PyDict_Size (PyObject  *p)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; len

Return the number of items in the dictionary.  This is equivalent to
@code{len(p)} on a dictionary.
@end deffn

@geindex PyDict_Next (C function)
@anchor{c-api/dict c PyDict_Next}@anchor{416b}
@deffn {C Function} int  PyDict_Next (PyObject  *p, Py_ssize_t  *ppos, PyObject  **pkey, PyObject  **pvalue)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Iterate over all key-value pairs in the dictionary @emph{p}.  The
@code{Py_ssize_t} referred to by @emph{ppos} must be initialized to @code{0}
prior to the first call to this function to start the iteration; the
function returns true for each pair in the dictionary, and false once all
pairs have been reported.  The parameters @emph{pkey} and @emph{pvalue} should either
point to @ref{375,,PyObject}* variables that will be filled in with each key
and value, respectively, or may be @code{NULL}.  Any references returned through
them are borrowed.  @emph{ppos} should not be altered during iteration. Its
value represents offsets within the internal dictionary structure, and
since the structure is sparse, the offsets are not consecutive.

For example:

@example
PyObject *key, *value;
Py_ssize_t pos = 0;

while (PyDict_Next(self->dict, &pos, &key, &value)) @{
    /* do something interesting with the values... */
    ...
@}
@end example

The dictionary @emph{p} should not be mutated during iteration.  It is safe to
modify the values of the keys as you iterate over the dictionary, but only
so long as the set of keys does not change.  For example:

@example
PyObject *key, *value;
Py_ssize_t pos = 0;

while (PyDict_Next(self->dict, &pos, &key, &value)) @{
    long i = PyLong_AsLong(value);
    if (i == -1 && PyErr_Occurred()) @{
        return -1;
    @}
    PyObject *o = PyLong_FromLong(i + 1);
    if (o == NULL)
        return -1;
    if (PyDict_SetItem(self->dict, key, o) < 0) @{
        Py_DECREF(o);
        return -1;
    @}
    Py_DECREF(o);
@}
@end example
@end deffn

@geindex PyDict_Merge (C function)
@anchor{c-api/dict c PyDict_Merge}@anchor{4168}
@deffn {C Function} int  PyDict_Merge (PyObject  *a, PyObject  *b, int  override)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Iterate over mapping object @emph{b} adding key-value pairs to dictionary @emph{a}.
@emph{b} may be a dictionary, or any object supporting @ref{6f0,,PyMapping_Keys()}
and @ref{4115,,PyObject_GetItem()}. If @emph{override} is true, existing pairs in @emph{a}
will be replaced if a matching key is found in @emph{b}, otherwise pairs will
only be added if there is not a matching key in @emph{a}. Return @code{0} on
success or @code{-1} if an exception was raised.
@end deffn

@geindex PyDict_Update (C function)
@anchor{c-api/dict c PyDict_Update}@anchor{416f}
@deffn {C Function} int  PyDict_Update (PyObject  *a, PyObject  *b)
@emph{ Part of the @ref{35d,,Stable ABI}.}
This is the same as @code{PyDict_Merge(a, b, 1)} in C, and is similar to
@code{a.update(b)} in Python except that @ref{416f,,PyDict_Update()} doesn’t fall
back to the iterating over a sequence of key value pairs if the second
argument has no “keys” attribute.  Return @code{0} on success or @code{-1} if an
exception was raised.
@end deffn

@geindex PyDict_MergeFromSeq2 (C function)
@anchor{c-api/dict c PyDict_MergeFromSeq2}@anchor{4169}
@deffn {C Function} int  PyDict_MergeFromSeq2 (PyObject  *a, PyObject  *seq2, int  override)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Update or merge into dictionary @emph{a}, from the key-value pairs in @emph{seq2}.
@emph{seq2} must be an iterable object producing iterable objects of length 2,
viewed as key-value pairs.  In case of duplicate keys, the last wins if
@emph{override} is true, else the first wins. Return @code{0} on success or @code{-1}
if an exception was raised. Equivalent Python (except for the return
value):

@example
def PyDict_MergeFromSeq2(a, seq2, override):
    for key, value in seq2:
        if override or key not in a:
            a[key] = value
@end example
@end deffn

@node Set Objects,,Dictionary Objects,Container Objects
@anchor{c-api/set doc}@anchor{4429}@anchor{c-api/set set-objects}@anchor{442a}@anchor{c-api/set setobjects}@anchor{442b}
@subsubsection Set Objects


@geindex object; set
@geindex object; frozenset

This section details the public API for @ref{370,,set} and @ref{439,,frozenset}
objects.  Any functionality not listed below is best accessed using the either
the abstract object protocol (including @ref{350,,PyObject_CallMethod()},
@ref{1239,,PyObject_RichCompareBool()}, @ref{209,,PyObject_Hash()},
@ref{bc0,,PyObject_Repr()}, @ref{420c,,PyObject_IsTrue()}, @ref{130f,,PyObject_Print()}, and
@ref{4209,,PyObject_GetIter()}) or the abstract number protocol (including
@ref{41d8,,PyNumber_And()}, @ref{41f1,,PyNumber_Subtract()}, @ref{41ec,,PyNumber_Or()},
@ref{41f3,,PyNumber_Xor()}, @ref{41dd,,PyNumber_InPlaceAnd()},
@ref{41e5,,PyNumber_InPlaceSubtract()}, @ref{41e1,,PyNumber_InPlaceOr()}, and
@ref{41e7,,PyNumber_InPlaceXor()}).

@geindex PySetObject (C type)
@anchor{c-api/set c PySetObject}@anchor{442c}
@deffn {C Type} type  PySetObject

This subtype of @ref{375,,PyObject} is used to hold the internal data for both
@ref{370,,set} and @ref{439,,frozenset} objects.  It is like a @ref{4426,,PyDictObject}
in that it is a fixed size for small sets (much like tuple storage) and will
point to a separate, variable sized block of memory for medium and large sized
sets (much like list storage). None of the fields of this structure should be
considered public and are subject to change.  All access should be done through
the documented API rather than by manipulating the values in the structure.
@end deffn

@geindex PySet_Type (C var)
@anchor{c-api/set c PySet_Type}@anchor{4227}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PySet_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
This is an instance of @ref{5c6,,PyTypeObject} representing the Python
@ref{370,,set} type.
@end deffn

@geindex PyFrozenSet_Type (C var)
@anchor{c-api/set c PyFrozenSet_Type}@anchor{419a}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyFrozenSet_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
This is an instance of @ref{5c6,,PyTypeObject} representing the Python
@ref{439,,frozenset} type.
@end deffn

The following type check macros work on pointers to any Python object. Likewise,
the constructor functions work with any iterable Python object.

@geindex PySet_Check (C function)
@anchor{c-api/set c PySet_Check}@anchor{442d}
@deffn {C Function} int  PySet_Check (PyObject  *p)

Return true if @emph{p} is a @ref{370,,set} object or an instance of a subtype.
This function always succeeds.
@end deffn

@geindex PyFrozenSet_Check (C function)
@anchor{c-api/set c PyFrozenSet_Check}@anchor{442e}
@deffn {C Function} int  PyFrozenSet_Check (PyObject  *p)

Return true if @emph{p} is a @ref{439,,frozenset} object or an instance of a
subtype.  This function always succeeds.
@end deffn

@geindex PyAnySet_Check (C function)
@anchor{c-api/set c PyAnySet_Check}@anchor{442f}
@deffn {C Function} int  PyAnySet_Check (PyObject  *p)

Return true if @emph{p} is a @ref{370,,set} object, a @ref{439,,frozenset} object, or an
instance of a subtype.  This function always succeeds.
@end deffn

@geindex PySet_CheckExact (C function)
@anchor{c-api/set c PySet_CheckExact}@anchor{36f}
@deffn {C Function} int  PySet_CheckExact (PyObject  *p)

Return true if @emph{p} is a @ref{370,,set} object but not an instance of a
subtype.  This function always succeeds.


New in version 3.10.

@end deffn

@geindex PyAnySet_CheckExact (C function)
@anchor{c-api/set c PyAnySet_CheckExact}@anchor{4430}
@deffn {C Function} int  PyAnySet_CheckExact (PyObject  *p)

Return true if @emph{p} is a @ref{370,,set} object or a @ref{439,,frozenset} object but
not an instance of a subtype.  This function always succeeds.
@end deffn

@geindex PyFrozenSet_CheckExact (C function)
@anchor{c-api/set c PyFrozenSet_CheckExact}@anchor{4431}
@deffn {C Function} int  PyFrozenSet_CheckExact (PyObject  *p)

Return true if @emph{p} is a @ref{439,,frozenset} object but not an instance of a
subtype.  This function always succeeds.
@end deffn

@geindex PySet_New (C function)
@anchor{c-api/set c PySet_New}@anchor{fa5}
@deffn {C Function} @ref{375,,PyObject}  *PySet_New (PyObject  *iterable)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new @ref{370,,set} containing objects returned by the @emph{iterable}.  The
@emph{iterable} may be @code{NULL} to create a new empty set.  Return the new set on
success or @code{NULL} on failure.  Raise @ref{162,,TypeError} if @emph{iterable} is not
actually iterable.  The constructor is also useful for copying a set
(@code{c=set(s)}).
@end deffn

@geindex PyFrozenSet_New (C function)
@anchor{c-api/set c PyFrozenSet_New}@anchor{fa6}
@deffn {C Function} @ref{375,,PyObject}  *PyFrozenSet_New (PyObject  *iterable)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new @ref{439,,frozenset} containing objects returned by the @emph{iterable}.
The @emph{iterable} may be @code{NULL} to create a new empty frozenset.  Return the new
set on success or @code{NULL} on failure.  Raise @ref{162,,TypeError} if @emph{iterable} is
not actually iterable.
@end deffn

The following functions and macros are available for instances of @ref{370,,set}
or @ref{439,,frozenset} or instances of their subtypes.

@geindex PySet_Size (C function)
@anchor{c-api/set c PySet_Size}@anchor{faa}
@deffn {C Function} Py_ssize_t  PySet_Size (PyObject  *anyset)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex built-in function; len

Return the length of a @ref{370,,set} or @ref{439,,frozenset} object. Equivalent to
@code{len(anyset)}.  Raises a @code{PyExc_SystemError} if @emph{anyset} is not a
@ref{370,,set}, @ref{439,,frozenset}, or an instance of a subtype.
@end deffn

@geindex PySet_GET_SIZE (C function)
@anchor{c-api/set c PySet_GET_SIZE}@anchor{4432}
@deffn {C Function} Py_ssize_t  PySet_GET_SIZE (PyObject  *anyset)

Macro form of @ref{faa,,PySet_Size()} without error checking.
@end deffn

@geindex PySet_Contains (C function)
@anchor{c-api/set c PySet_Contains}@anchor{fa9}
@deffn {C Function} int  PySet_Contains (PyObject  *anyset, PyObject  *key)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return @code{1} if found, @code{0} if not found, and @code{-1} if an error is encountered.  Unlike
the Python @code{__contains__()} method, this function does not automatically
convert unhashable sets into temporary frozensets.  Raise a @ref{162,,TypeError} if
the @emph{key} is unhashable. Raise @code{PyExc_SystemError} if @emph{anyset} is not a
@ref{370,,set}, @ref{439,,frozenset}, or an instance of a subtype.
@end deffn

@geindex PySet_Add (C function)
@anchor{c-api/set c PySet_Add}@anchor{fa7}
@deffn {C Function} int  PySet_Add (PyObject  *set, PyObject  *key)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Add @emph{key} to a @ref{370,,set} instance.  Also works with @ref{439,,frozenset}
instances (like @ref{4091,,PyTuple_SetItem()} it can be used to fill-in the values
of brand new frozensets before they are exposed to other code).  Return @code{0} on
success or @code{-1} on failure. Raise a @ref{162,,TypeError} if the @emph{key} is
unhashable. Raise a @ref{10de,,MemoryError} if there is no room to grow.  Raise a
@ref{891,,SystemError} if @emph{set} is not an instance of @ref{370,,set} or its
subtype.
@end deffn

The following functions are available for instances of @ref{370,,set} or its
subtypes but not for instances of @ref{439,,frozenset} or its subtypes.

@geindex PySet_Discard (C function)
@anchor{c-api/set c PySet_Discard}@anchor{fa8}
@deffn {C Function} int  PySet_Discard (PyObject  *set, PyObject  *key)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return @code{1} if found and removed, @code{0} if not found (no action taken), and @code{-1} if an
error is encountered.  Does not raise @ref{5bb,,KeyError} for missing keys.  Raise a
@ref{162,,TypeError} if the @emph{key} is unhashable.  Unlike the Python @code{discard()}
method, this function does not automatically convert unhashable sets into
temporary frozensets. Raise @code{PyExc_SystemError} if @emph{set} is not an
instance of @ref{370,,set} or its subtype.
@end deffn

@geindex PySet_Pop (C function)
@anchor{c-api/set c PySet_Pop}@anchor{4226}
@deffn {C Function} @ref{375,,PyObject}  *PySet_Pop (PyObject  *set)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new reference to an arbitrary object in the @emph{set}, and removes the
object from the @emph{set}.  Return @code{NULL} on failure.  Raise @ref{5bb,,KeyError} if the
set is empty. Raise a @ref{891,,SystemError} if @emph{set} is not an instance of
@ref{370,,set} or its subtype.
@end deffn

@geindex PySet_Clear (C function)
@anchor{c-api/set c PySet_Clear}@anchor{4225}
@deffn {C Function} int  PySet_Clear (PyObject  *set)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Empty an existing set of all elements.
@end deffn

@node Function Objects<2>,Other Objects,Container Objects,Concrete Objects Layer
@anchor{c-api/concrete function-objects}@anchor{4433}@anchor{c-api/concrete otherobjects}@anchor{4434}
@subsection Function Objects


@menu
* Function Objects: Function Objects<3>. 
* Instance Method Objects:: 
* Method Objects: Method Objects<2>. 
* Cell Objects:: 
* Code Objects: Code Objects<2>. 

@end menu

@node Function Objects<3>,Instance Method Objects,,Function Objects<2>
@anchor{c-api/function doc}@anchor{4435}@anchor{c-api/function function-objects}@anchor{4436}@anchor{c-api/function id1}@anchor{4437}
@subsubsection Function Objects


@geindex object; function

There are a few functions specific to Python functions.

@geindex PyFunctionObject (C type)
@anchor{c-api/function c PyFunctionObject}@anchor{4438}
@deffn {C Type} type  PyFunctionObject

The C structure used for functions.
@end deffn

@geindex PyFunction_Type (C var)
@anchor{c-api/function c PyFunction_Type}@anchor{4439}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyFunction_Type

@geindex MethodType (in module types)

This is an instance of @ref{5c6,,PyTypeObject} and represents the Python function
type.  It is exposed to Python programmers as @code{types.FunctionType}.
@end deffn

@geindex PyFunction_Check (C function)
@anchor{c-api/function c PyFunction_Check}@anchor{443a}
@deffn {C Function} int  PyFunction_Check (PyObject  *o)

Return true if @emph{o} is a function object (has type @ref{4439,,PyFunction_Type}).
The parameter must not be @code{NULL}.  This function always succeeds.
@end deffn

@geindex PyFunction_New (C function)
@anchor{c-api/function c PyFunction_New}@anchor{443b}
@deffn {C Function} @ref{375,,PyObject}  *PyFunction_New (PyObject  *code, PyObject  *globals)
@emph{Return value: New reference.}
Return a new function object associated with the code object @emph{code}. @emph{globals}
must be a dictionary with the global variables accessible to the function.

The function’s docstring and name are retrieved from the code object. @emph{__module__}
is retrieved from @emph{globals}. The argument defaults, annotations and closure are
set to @code{NULL}. @emph{__qualname__} is set to the same value as the code object’s
@code{co_qualname} field.
@end deffn

@geindex PyFunction_NewWithQualName (C function)
@anchor{c-api/function c PyFunction_NewWithQualName}@anchor{443c}
@deffn {C Function} @ref{375,,PyObject}  *PyFunction_NewWithQualName (PyObject  *code, PyObject  *globals, PyObject  *qualname)
@emph{Return value: New reference.}
As @ref{443b,,PyFunction_New()}, but also allows setting the function object’s
@code{__qualname__} attribute.  @emph{qualname} should be a unicode object or @code{NULL};
if @code{NULL}, the @code{__qualname__} attribute is set to the same value as the
code object’s @code{co_qualname} field.


New in version 3.3.

@end deffn

@geindex PyFunction_GetCode (C function)
@anchor{c-api/function c PyFunction_GetCode}@anchor{443d}
@deffn {C Function} @ref{375,,PyObject}  *PyFunction_GetCode (PyObject  *op)
@emph{Return value: Borrowed reference.}
Return the code object associated with the function object @emph{op}.
@end deffn

@geindex PyFunction_GetGlobals (C function)
@anchor{c-api/function c PyFunction_GetGlobals}@anchor{443e}
@deffn {C Function} @ref{375,,PyObject}  *PyFunction_GetGlobals (PyObject  *op)
@emph{Return value: Borrowed reference.}
Return the globals dictionary associated with the function object @emph{op}.
@end deffn

@geindex PyFunction_GetModule (C function)
@anchor{c-api/function c PyFunction_GetModule}@anchor{443f}
@deffn {C Function} @ref{375,,PyObject}  *PyFunction_GetModule (PyObject  *op)
@emph{Return value: Borrowed reference.}
Return a @ref{4112,,borrowed reference} to the @emph{__module__} attribute of the
function object @emph{op}. It can be @emph{NULL}.

This is normally a string containing the module name, but can be set to any
other object by Python code.
@end deffn

@geindex PyFunction_GetDefaults (C function)
@anchor{c-api/function c PyFunction_GetDefaults}@anchor{4440}
@deffn {C Function} @ref{375,,PyObject}  *PyFunction_GetDefaults (PyObject  *op)
@emph{Return value: Borrowed reference.}
Return the argument default values of the function object @emph{op}. This can be a
tuple of arguments or @code{NULL}.
@end deffn

@geindex PyFunction_SetDefaults (C function)
@anchor{c-api/function c PyFunction_SetDefaults}@anchor{4441}
@deffn {C Function} int  PyFunction_SetDefaults (PyObject  *op, PyObject  *defaults)

Set the argument default values for the function object @emph{op}. @emph{defaults} must be
@code{Py_None} or a tuple.

Raises @ref{891,,SystemError} and returns @code{-1} on failure.
@end deffn

@geindex PyFunction_GetClosure (C function)
@anchor{c-api/function c PyFunction_GetClosure}@anchor{4442}
@deffn {C Function} @ref{375,,PyObject}  *PyFunction_GetClosure (PyObject  *op)
@emph{Return value: Borrowed reference.}
Return the closure associated with the function object @emph{op}. This can be @code{NULL}
or a tuple of cell objects.
@end deffn

@geindex PyFunction_SetClosure (C function)
@anchor{c-api/function c PyFunction_SetClosure}@anchor{4443}
@deffn {C Function} int  PyFunction_SetClosure (PyObject  *op, PyObject  *closure)

Set the closure associated with the function object @emph{op}. @emph{closure} must be
@code{Py_None} or a tuple of cell objects.

Raises @ref{891,,SystemError} and returns @code{-1} on failure.
@end deffn

@geindex PyFunction_GetAnnotations (C function)
@anchor{c-api/function c PyFunction_GetAnnotations}@anchor{10c8}
@deffn {C Function} @ref{375,,PyObject}  *PyFunction_GetAnnotations (PyObject  *op)
@emph{Return value: Borrowed reference.}
Return the annotations of the function object @emph{op}. This can be a
mutable dictionary or @code{NULL}.
@end deffn

@geindex PyFunction_SetAnnotations (C function)
@anchor{c-api/function c PyFunction_SetAnnotations}@anchor{4444}
@deffn {C Function} int  PyFunction_SetAnnotations (PyObject  *op, PyObject  *annotations)

Set the annotations for the function object @emph{op}. @emph{annotations}
must be a dictionary or @code{Py_None}.

Raises @ref{891,,SystemError} and returns @code{-1} on failure.
@end deffn

@node Instance Method Objects,Method Objects<2>,Function Objects<3>,Function Objects<2>
@anchor{c-api/method doc}@anchor{4445}@anchor{c-api/method instance-method-objects}@anchor{4446}@anchor{c-api/method instancemethod-objects}@anchor{4447}
@subsubsection Instance Method Objects


@geindex object; instancemethod

An instance method is a wrapper for a @ref{fd4,,PyCFunction} and the new way
to bind a @ref{fd4,,PyCFunction} to a class object. It replaces the former call
@code{PyMethod_New(func, NULL, class)}.

@geindex PyInstanceMethod_Type (C var)
@anchor{c-api/method c PyInstanceMethod_Type}@anchor{4448}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyInstanceMethod_Type

This instance of @ref{5c6,,PyTypeObject} represents the Python instance
method type. It is not exposed to Python programs.
@end deffn

@geindex PyInstanceMethod_Check (C function)
@anchor{c-api/method c PyInstanceMethod_Check}@anchor{4449}
@deffn {C Function} int  PyInstanceMethod_Check (PyObject  *o)

Return true if @emph{o} is an instance method object (has type
@ref{4448,,PyInstanceMethod_Type}).  The parameter must not be @code{NULL}.
This function always succeeds.
@end deffn

@geindex PyInstanceMethod_New (C function)
@anchor{c-api/method c PyInstanceMethod_New}@anchor{444a}
@deffn {C Function} @ref{375,,PyObject}  *PyInstanceMethod_New (PyObject  *func)
@emph{Return value: New reference.}
Return a new instance method object, with @emph{func} being any callable object
@emph{func} is the function that will be called when the instance method is
called.
@end deffn

@geindex PyInstanceMethod_Function (C function)
@anchor{c-api/method c PyInstanceMethod_Function}@anchor{444b}
@deffn {C Function} @ref{375,,PyObject}  *PyInstanceMethod_Function (PyObject  *im)
@emph{Return value: Borrowed reference.}
Return the function object associated with the instance method @emph{im}.
@end deffn

@geindex PyInstanceMethod_GET_FUNCTION (C function)
@anchor{c-api/method c PyInstanceMethod_GET_FUNCTION}@anchor{444c}
@deffn {C Function} @ref{375,,PyObject}  *PyInstanceMethod_GET_FUNCTION (PyObject  *im)
@emph{Return value: Borrowed reference.}
Macro version of @ref{444b,,PyInstanceMethod_Function()} which avoids error checking.
@end deffn

@node Method Objects<2>,Cell Objects,Instance Method Objects,Function Objects<2>
@anchor{c-api/method id1}@anchor{444d}@anchor{c-api/method method-objects}@anchor{444e}
@subsubsection Method Objects


@geindex object; method

Methods are bound function objects. Methods are always bound to an instance of
a user-defined class. Unbound methods (methods bound to a class object) are
no longer available.

@geindex PyMethod_Type (C var)
@anchor{c-api/method c PyMethod_Type}@anchor{444f}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyMethod_Type

@geindex MethodType (in module types)

This instance of @ref{5c6,,PyTypeObject} represents the Python method type.  This
is exposed to Python programs as @code{types.MethodType}.
@end deffn

@geindex PyMethod_Check (C function)
@anchor{c-api/method c PyMethod_Check}@anchor{4450}
@deffn {C Function} int  PyMethod_Check (PyObject  *o)

Return true if @emph{o} is a method object (has type @ref{444f,,PyMethod_Type}).  The
parameter must not be @code{NULL}.  This function always succeeds.
@end deffn

@geindex PyMethod_New (C function)
@anchor{c-api/method c PyMethod_New}@anchor{4451}
@deffn {C Function} @ref{375,,PyObject}  *PyMethod_New (PyObject  *func, PyObject  *self)
@emph{Return value: New reference.}
Return a new method object, with @emph{func} being any callable object and @emph{self}
the instance the method should be bound. @emph{func} is the function that will
be called when the method is called. @emph{self} must not be @code{NULL}.
@end deffn

@geindex PyMethod_Function (C function)
@anchor{c-api/method c PyMethod_Function}@anchor{4452}
@deffn {C Function} @ref{375,,PyObject}  *PyMethod_Function (PyObject  *meth)
@emph{Return value: Borrowed reference.}
Return the function object associated with the method @emph{meth}.
@end deffn

@geindex PyMethod_GET_FUNCTION (C function)
@anchor{c-api/method c PyMethod_GET_FUNCTION}@anchor{4453}
@deffn {C Function} @ref{375,,PyObject}  *PyMethod_GET_FUNCTION (PyObject  *meth)
@emph{Return value: Borrowed reference.}
Macro version of @ref{4452,,PyMethod_Function()} which avoids error checking.
@end deffn

@geindex PyMethod_Self (C function)
@anchor{c-api/method c PyMethod_Self}@anchor{4454}
@deffn {C Function} @ref{375,,PyObject}  *PyMethod_Self (PyObject  *meth)
@emph{Return value: Borrowed reference.}
Return the instance associated with the method @emph{meth}.
@end deffn

@geindex PyMethod_GET_SELF (C function)
@anchor{c-api/method c PyMethod_GET_SELF}@anchor{4455}
@deffn {C Function} @ref{375,,PyObject}  *PyMethod_GET_SELF (PyObject  *meth)
@emph{Return value: Borrowed reference.}
Macro version of @ref{4454,,PyMethod_Self()} which avoids error checking.
@end deffn

@node Cell Objects,Code Objects<2>,Method Objects<2>,Function Objects<2>
@anchor{c-api/cell doc}@anchor{4456}@anchor{c-api/cell cell-objects}@anchor{4457}@anchor{c-api/cell id1}@anchor{4458}
@subsubsection Cell Objects


“Cell” objects are used to implement variables referenced by multiple scopes.
For each such variable, a cell object is created to store the value; the local
variables of each stack frame that references the value contains a reference to
the cells from outer scopes which also use that variable.  When the value is
accessed, the value contained in the cell is used instead of the cell object
itself.  This de-referencing of the cell object requires support from the
generated byte-code; these are not automatically de-referenced when accessed.
Cell objects are not likely to be useful elsewhere.

@geindex PyCellObject (C type)
@anchor{c-api/cell c PyCellObject}@anchor{4459}
@deffn {C Type} type  PyCellObject

The C structure used for cell objects.
@end deffn

@geindex PyCell_Type (C var)
@anchor{c-api/cell c PyCell_Type}@anchor{445a}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyCell_Type

The type object corresponding to cell objects.
@end deffn

@geindex PyCell_Check (C function)
@anchor{c-api/cell c PyCell_Check}@anchor{445b}
@deffn {C Function} int  PyCell_Check (ob)

Return true if @emph{ob} is a cell object; @emph{ob} must not be @code{NULL}.  This
function always succeeds.
@end deffn

@geindex PyCell_New (C function)
@anchor{c-api/cell c PyCell_New}@anchor{445c}
@deffn {C Function} @ref{375,,PyObject}  *PyCell_New (PyObject  *ob)
@emph{Return value: New reference.}
Create and return a new cell object containing the value @emph{ob}. The parameter may
be @code{NULL}.
@end deffn

@geindex PyCell_Get (C function)
@anchor{c-api/cell c PyCell_Get}@anchor{445d}
@deffn {C Function} @ref{375,,PyObject}  *PyCell_Get (PyObject  *cell)
@emph{Return value: New reference.}
Return the contents of the cell @emph{cell}.
@end deffn

@geindex PyCell_GET (C function)
@anchor{c-api/cell c PyCell_GET}@anchor{445e}
@deffn {C Function} @ref{375,,PyObject}  *PyCell_GET (PyObject  *cell)
@emph{Return value: Borrowed reference.}
Return the contents of the cell @emph{cell}, but without checking that @emph{cell} is
non-@code{NULL} and a cell object.
@end deffn

@geindex PyCell_Set (C function)
@anchor{c-api/cell c PyCell_Set}@anchor{445f}
@deffn {C Function} int  PyCell_Set (PyObject  *cell, PyObject  *value)

Set the contents of the cell object @emph{cell} to @emph{value}.  This releases the
reference to any current content of the cell. @emph{value} may be @code{NULL}.  @emph{cell}
must be non-@code{NULL}; if it is not a cell object, @code{-1} will be returned.  On
success, @code{0} will be returned.
@end deffn

@geindex PyCell_SET (C function)
@anchor{c-api/cell c PyCell_SET}@anchor{391}
@deffn {C Function} void  PyCell_SET (PyObject  *cell, PyObject  *value)

Sets the value of the cell object @emph{cell} to @emph{value}.  No reference counts are
adjusted, and no checks are made for safety; @emph{cell} must be non-@code{NULL} and must
be a cell object.
@end deffn
@anchor{c-api/code codeobjects}@anchor{4460}
@geindex object; code
@geindex code object

@node Code Objects<2>,,Cell Objects,Function Objects<2>
@anchor{c-api/code doc}@anchor{4461}@anchor{c-api/code code-objects}@anchor{4462}
@subsubsection Code Objects


Code objects are a low-level detail of the CPython implementation.
Each one represents a chunk of executable code that hasn’t yet been
bound into a function.

@geindex PyCodeObject (C type)
@anchor{c-api/code c PyCodeObject}@anchor{4322}
@deffn {C Type} type  PyCodeObject

The C structure of the objects used to describe code objects.  The
fields of this type are subject to change at any time.
@end deffn

@geindex PyCode_Type (C var)
@anchor{c-api/code c PyCode_Type}@anchor{4463}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyCode_Type

This is an instance of @ref{5c6,,PyTypeObject} representing the Python
@ref{19,,code} type.
@end deffn

@geindex PyCode_Check (C function)
@anchor{c-api/code c PyCode_Check}@anchor{4464}
@deffn {C Function} int  PyCode_Check (PyObject  *co)

Return true if @emph{co} is a @ref{19,,code} object.  This function always succeeds.
@end deffn

@geindex PyCode_GetNumFree (C function)
@anchor{c-api/code c PyCode_GetNumFree}@anchor{4465}
@deffn {C Function} int  PyCode_GetNumFree (PyCodeObject  *co)

Return the number of free variables in @emph{co}.
@end deffn

@geindex PyCode_New (C function)
@anchor{c-api/code c PyCode_New}@anchor{584}
@deffn {C Function} @ref{4322,,PyCodeObject}  *PyCode_New (int  argcount, int  kwonlyargcount, int  nlocals, int  stacksize, int  flags, PyObject  *code, PyObject  *consts, PyObject  *names, PyObject  *varnames, PyObject  *freevars, PyObject  *cellvars, PyObject  *filename, PyObject  *name, int  firstlineno, PyObject  *lnotab)
@emph{Return value: New reference.}
Return a new code object.  If you need a dummy code object to create a frame,
use @ref{eeb,,PyCode_NewEmpty()} instead.  Calling @ref{584,,PyCode_New()} directly
can bind you to a precise Python version since the definition of the bytecode
changes often.
@end deffn

@geindex PyCode_NewWithPosOnlyArgs (C function)
@anchor{c-api/code c PyCode_NewWithPosOnlyArgs}@anchor{583}
@deffn {C Function} @ref{4322,,PyCodeObject}  *PyCode_NewWithPosOnlyArgs (int  argcount, int  posonlyargcount, int  kwonlyargcount, int  nlocals, int  stacksize, int  flags, PyObject  *code, PyObject  *consts, PyObject  *names, PyObject  *varnames, PyObject  *freevars, PyObject  *cellvars, PyObject  *filename, PyObject  *name, int  firstlineno, PyObject  *lnotab)
@emph{Return value: New reference.}
Similar to @ref{584,,PyCode_New()}, but with an extra “posonlyargcount” for positional-only arguments.


New in version 3.8.

@end deffn

@geindex PyCode_NewEmpty (C function)
@anchor{c-api/code c PyCode_NewEmpty}@anchor{eeb}
@deffn {C Function} @ref{4322,,PyCodeObject}  *PyCode_NewEmpty (const  char  *filename, const  char  *funcname, int  firstlineno)
@emph{Return value: New reference.}
Return a new empty code object with the specified filename,
function name, and first line number.  It is illegal to
@ref{34b,,exec()} or @ref{34a,,eval()} the resulting code object.
@end deffn

@geindex PyCode_Addr2Line (C function)
@anchor{c-api/code c PyCode_Addr2Line}@anchor{352}
@deffn {C Function} int  PyCode_Addr2Line (PyCodeObject  *co, int  byte_offset)

Return the line number of the instruction that occurs on or before @code{byte_offset} and ends after it.
If you just need the line number of a frame, use @ref{1f9,,PyFrame_GetLineNumber()} instead.

For efficiently iterating over the line numbers in a code object, use the API described in PEP 626@footnote{https://www.python.org/dev/peps/pep-0626/#out-of-process-debuggers-and-profilers}.
@end deffn

@geindex PyCode_Addr2Location (C function)
@anchor{c-api/code c PyCode_Addr2Location}@anchor{15d}
@deffn {C Function} int  PyCode_Addr2Location (PyObject  *co, int  byte_offset, int  *start_line, int  *start_column, int  *end_line, int  *end_column)

Sets the passed @code{int} pointers to the source code line and column numbers
for the instruction at @code{byte_offset}. Sets the value to @code{0} when
information is not available for any particular element.

Returns @code{1} if the function succeeds and 0 otherwise.
@end deffn

@node Other Objects,,Function Objects<2>,Concrete Objects Layer
@anchor{c-api/concrete other-objects}@anchor{4466}
@subsection Other Objects


@menu
* File Objects:: 
* Module Objects:: 
* Iterator Objects:: 
* Descriptor Objects:: 
* Slice Objects:: 
* Ellipsis Object:: 
* MemoryView objects:: 
* Weak Reference Objects: Weak Reference Objects<2>. 
* Capsules: Capsules<2>. 
* Generator Objects:: 
* Coroutine Objects: Coroutine Objects<2>. 
* Context Variables Objects:: 
* DateTime Objects: DateTime Objects<2>. 
* Objects for Type Hinting:: 

@end menu

@node File Objects,Module Objects,,Other Objects
@anchor{c-api/file doc}@anchor{4467}@anchor{c-api/file file-objects}@anchor{4468}@anchor{c-api/file fileobjects}@anchor{4469}
@subsubsection File Objects


@geindex object; file

These APIs are a minimal emulation of the Python 2 C API for built-in file
objects, which used to rely on the buffered I/O (FILE*) support
from the C standard library.  In Python 3, files and streams use the new
@ref{9d,,io} module, which defines several layers over the low-level unbuffered
I/O of the operating system.  The functions described below are
convenience C wrappers over these new APIs, and meant mostly for internal
error reporting in the interpreter; third-party code is advised to access
the @ref{9d,,io} APIs instead.

@geindex PyFile_FromFd (C function)
@anchor{c-api/file c PyFile_FromFd}@anchor{4193}
@deffn {C Function} @ref{375,,PyObject}  *PyFile_FromFd (int  fd, const  char  *name, const  char  *mode, int  buffering, const  char  *encoding, const  char  *errors, const  char  *newline, int  closefd)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a Python file object from the file descriptor of an already
opened file @emph{fd}.  The arguments @emph{name}, @emph{encoding}, @emph{errors} and @emph{newline}
can be @code{NULL} to use the defaults; @emph{buffering} can be @emph{-1} to use the
default. @emph{name} is ignored and kept for backward compatibility. Return
@code{NULL} on failure. For a more comprehensive description of the arguments,
please refer to the @ref{1c9,,io.open()} function documentation.

@cartouche
@quotation Warning 
Since Python streams have their own buffering layer, mixing them with
OS-level file descriptors can produce various issues (such as unexpected
ordering of data).
@end quotation
@end cartouche


Changed in version 3.2: Ignore @emph{name} attribute.

@end deffn

@geindex PyObject_AsFileDescriptor (C function)
@anchor{c-api/file c PyObject_AsFileDescriptor}@anchor{41fb}
@deffn {C Function} int  PyObject_AsFileDescriptor (PyObject  *p)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the file descriptor associated with @emph{p} as an int.  If the
object is an integer, its value is returned.  If not, the
object’s @ref{23a2,,fileno()} method is called if it exists; the
method must return an integer, which is returned as the file descriptor
value.  Sets an exception and returns @code{-1} on failure.
@end deffn

@geindex PyFile_GetLine (C function)
@anchor{c-api/file c PyFile_GetLine}@anchor{4194}
@deffn {C Function} @ref{375,,PyObject}  *PyFile_GetLine (PyObject  *p, int  n)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex EOFError (built-in exception)

Equivalent to @code{p.readline([n])}, this function reads one line from the
object @emph{p}.  @emph{p} may be a file object or any object with a
@ref{1b74,,readline()}
method.  If @emph{n} is @code{0}, exactly one line is read, regardless of the length of
the line.  If @emph{n} is greater than @code{0}, no more than @emph{n} bytes will be read
from the file; a partial line can be returned.  In both cases, an empty string
is returned if the end of the file is reached immediately.  If @emph{n} is less than
@code{0}, however, one line is read regardless of length, but @ref{e74,,EOFError} is
raised if the end of the file is reached immediately.
@end deffn

@geindex PyFile_SetOpenCodeHook (C function)
@anchor{c-api/file c PyFile_SetOpenCodeHook}@anchor{2488}
@deffn {C Function} int  PyFile_SetOpenCodeHook (Py_OpenCodeHookFunction  handler)

Overrides the normal behavior of @ref{11f6,,io.open_code()} to pass its parameter
through the provided handler.

The handler is a function of type @ref{375,,PyObject}  *(*)(@ref{375,,PyObject}  *path,  void  *userData), where @emph{path} is guaranteed to be @ref{43e4,,PyUnicodeObject}.

The @emph{userData} pointer is passed into the hook function. Since hook
functions may be called from different runtimes, this pointer should not
refer directly to Python state.

As this hook is intentionally used during import, avoid importing new modules
during its execution unless they are known to be frozen or available in
@code{sys.modules}.

Once a hook has been set, it cannot be removed or replaced, and later calls to
@ref{2488,,PyFile_SetOpenCodeHook()} will fail. On failure, the function returns
-1 and sets an exception if the interpreter has been initialized.

This function is safe to call before @ref{38d,,Py_Initialize()}.

Raises an @ref{1130,,auditing event} @code{setopencodehook} with no arguments.


New in version 3.8.

@end deffn

@geindex PyFile_WriteObject (C function)
@anchor{c-api/file c PyFile_WriteObject}@anchor{4195}
@deffn {C Function} int  PyFile_WriteObject (PyObject  *obj, PyObject  *p, int  flags)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex Py_PRINT_RAW

Write object @emph{obj} to file object @emph{p}.  The only supported flag for @emph{flags} is
@code{Py_PRINT_RAW}; if given, the @ref{2ef,,str()} of the object is written
instead of the @ref{27d,,repr()}.  Return @code{0} on success or @code{-1} on failure; the
appropriate exception will be set.
@end deffn

@geindex PyFile_WriteString (C function)
@anchor{c-api/file c PyFile_WriteString}@anchor{4196}
@deffn {C Function} int  PyFile_WriteString (const  char  *s, PyObject  *p)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Write string @emph{s} to file object @emph{p}.  Return @code{0} on success or @code{-1} on
failure; the appropriate exception will be set.
@end deffn

@node Module Objects,Iterator Objects,File Objects,Other Objects
@anchor{c-api/module doc}@anchor{446a}@anchor{c-api/module module-objects}@anchor{446b}@anchor{c-api/module moduleobjects}@anchor{446c}
@subsubsection Module Objects


@geindex object; module

@geindex PyModule_Type (C var)
@anchor{c-api/module c PyModule_Type}@anchor{41d5}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyModule_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex ModuleType (in module types)

This instance of @ref{5c6,,PyTypeObject} represents the Python module type.  This
is exposed to Python programs as @code{types.ModuleType}.
@end deffn

@geindex PyModule_Check (C function)
@anchor{c-api/module c PyModule_Check}@anchor{446d}
@deffn {C Function} int  PyModule_Check (PyObject  *p)

Return true if @emph{p} is a module object, or a subtype of a module object.
This function always succeeds.
@end deffn

@geindex PyModule_CheckExact (C function)
@anchor{c-api/module c PyModule_CheckExact}@anchor{446e}
@deffn {C Function} int  PyModule_CheckExact (PyObject  *p)

Return true if @emph{p} is a module object, but not a subtype of
@ref{41d5,,PyModule_Type}.  This function always succeeds.
@end deffn

@geindex PyModule_NewObject (C function)
@anchor{c-api/module c PyModule_NewObject}@anchor{41d3}
@deffn {C Function} @ref{375,,PyObject}  *PyModule_NewObject (PyObject  *name)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
@geindex __name__ (module attribute)
@geindex __doc__ (module attribute)
@geindex __file__ (module attribute)
@geindex __package__ (module attribute)
@geindex __loader__ (module attribute)

Return a new module object with the @ref{f12,,__name__} attribute set to @emph{name}.
The module’s @ref{f12,,__name__}, @code{__doc__}, @ref{bae,,__package__}, and
@ref{baf,,__loader__} attributes are filled in (all but @ref{f12,,__name__} are set
to @code{None}); the caller is responsible for providing a @ref{1797,,__file__}
attribute.


New in version 3.3.



Changed in version 3.4: @ref{bae,,__package__} and @ref{baf,,__loader__} are set to @code{None}.

@end deffn

@geindex PyModule_New (C function)
@anchor{c-api/module c PyModule_New}@anchor{41d2}
@deffn {C Function} @ref{375,,PyObject}  *PyModule_New (const  char  *name)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Similar to @ref{41d3,,PyModule_NewObject()}, but the name is a UTF-8 encoded
string instead of a Unicode object.
@end deffn

@geindex PyModule_GetDict (C function)
@anchor{c-api/module c PyModule_GetDict}@anchor{41cd}
@deffn {C Function} @ref{375,,PyObject}  *PyModule_GetDict (PyObject  *module)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex __dict__ (module attribute)

Return the dictionary object that implements @emph{module}’s namespace; this object
is the same as the @ref{7a4,,__dict__} attribute of the module object.
If @emph{module} is not a module object (or a subtype of a module object),
@ref{891,,SystemError} is raised and @code{NULL} is returned.

It is recommended extensions use other PyModule_* and
PyObject_* functions rather than directly manipulate a module’s
@ref{7a4,,__dict__}.
@end deffn

@geindex PyModule_GetNameObject (C function)
@anchor{c-api/module c PyModule_GetNameObject}@anchor{41d1}
@deffn {C Function} @ref{375,,PyObject}  *PyModule_GetNameObject (PyObject  *module)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
@geindex __name__ (module attribute)
@geindex SystemError (built-in exception)

Return @emph{module}’s @ref{f12,,__name__} value.  If the module does not provide one,
or if it is not a string, @ref{891,,SystemError} is raised and @code{NULL} is returned.


New in version 3.3.

@end deffn

@geindex PyModule_GetName (C function)
@anchor{c-api/module c PyModule_GetName}@anchor{41d0}
@deffn {C Function} const  char  *PyModule_GetName (PyObject  *module)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Similar to @ref{41d1,,PyModule_GetNameObject()} but return the name encoded to
@code{'utf-8'}.
@end deffn

@geindex PyModule_GetState (C function)
@anchor{c-api/module c PyModule_GetState}@anchor{487}
@deffn {C Function} void  *PyModule_GetState (PyObject  *module)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the “state” of the module, that is, a pointer to the block of memory
allocated at module creation time, or @code{NULL}.  See
@ref{486,,PyModuleDef.m_size}.
@end deffn

@geindex PyModule_GetDef (C function)
@anchor{c-api/module c PyModule_GetDef}@anchor{41cc}
@deffn {C Function} @ref{484,,PyModuleDef}  *PyModule_GetDef (PyObject  *module)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a pointer to the @ref{484,,PyModuleDef} struct from which the module was
created, or @code{NULL} if the module wasn’t created from a definition.
@end deffn

@geindex PyModule_GetFilenameObject (C function)
@anchor{c-api/module c PyModule_GetFilenameObject}@anchor{41cf}
@deffn {C Function} @ref{375,,PyObject}  *PyModule_GetFilenameObject (PyObject  *module)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex __file__ (module attribute)
@geindex SystemError (built-in exception)

Return the name of the file from which @emph{module} was loaded using @emph{module}’s
@ref{1797,,__file__} attribute.  If this is not defined, or if it is not a
unicode string, raise @ref{891,,SystemError} and return @code{NULL}; otherwise return
a reference to a Unicode object.


New in version 3.2.

@end deffn

@geindex PyModule_GetFilename (C function)
@anchor{c-api/module c PyModule_GetFilename}@anchor{41ce}
@deffn {C Function} const  char  *PyModule_GetFilename (PyObject  *module)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Similar to @ref{41cf,,PyModule_GetFilenameObject()} but return the filename
encoded to ‘utf-8’.


Deprecated since version 3.2: @ref{41ce,,PyModule_GetFilename()} raises @code{UnicodeEncodeError} on
unencodable filenames, use @ref{41cf,,PyModule_GetFilenameObject()} instead.

@end deffn

@menu
* Initializing C modules:: 
* Module lookup:: 

@end menu

@node Initializing C modules,Module lookup,,Module Objects
@anchor{c-api/module initializing-c-modules}@anchor{446f}@anchor{c-api/module initializing-modules}@anchor{40d8}
@subsubsection Initializing C modules


Modules objects are usually created from extension modules (shared libraries
which export an initialization function), or compiled-in modules
(where the initialization function is added using @ref{112b,,PyImport_AppendInittab()}).
See @ref{407d,,Building C and C++ Extensions} or @ref{40f6,,Extending Embedded Python} for details.

The initialization function can either pass a module definition instance
to @ref{407a,,PyModule_Create()}, and return the resulting module object,
or request “multi-phase initialization” by returning the definition struct itself.

@geindex PyModuleDef (C type)
@anchor{c-api/module c PyModuleDef}@anchor{484}
@deffn {C Type} type  PyModuleDef
@emph{ Part of the @ref{35d,,Stable ABI}.}
The module definition struct, which holds all information needed to create
a module object. There is usually only one statically initialized variable
of this type for each module.

@geindex PyModuleDef.m_base (C member)
@anchor{c-api/module c PyModuleDef m_base}@anchor{4470}
@deffn {C Member} PyModuleDef_Base  m_base

Always initialize this member to @code{PyModuleDef_HEAD_INIT}.
@end deffn

@geindex PyModuleDef.m_name (C member)
@anchor{c-api/module c PyModuleDef m_name}@anchor{4471}
@deffn {C Member} const  char  *m_name

Name for the new module.
@end deffn

@geindex PyModuleDef.m_doc (C member)
@anchor{c-api/module c PyModuleDef m_doc}@anchor{4472}
@deffn {C Member} const  char  *m_doc

Docstring for the module; usually a docstring variable created with
@ref{1200,,PyDoc_STRVAR} is used.
@end deffn

@geindex PyModuleDef.m_size (C member)
@anchor{c-api/module c PyModuleDef m_size}@anchor{486}
@deffn {C Member} Py_ssize_t  m_size

Module state may be kept in a per-module memory area that can be
retrieved with @ref{487,,PyModule_GetState()}, rather than in static globals.
This makes modules safe for use in multiple sub-interpreters.

This memory area is allocated based on @emph{m_size} on module creation,
and freed when the module object is deallocated, after the
@ref{483,,m_free} function has been called, if present.

Setting @code{m_size} to @code{-1} means that the module does not support
sub-interpreters, because it has global state.

Setting it to a non-negative value means that the module can be
re-initialized and specifies the additional amount of memory it requires
for its state. Non-negative @code{m_size} is required for multi-phase
initialization.

See 
@geindex Python Enhancement Proposals; PEP 3121
PEP 3121@footnote{https://www.python.org/dev/peps/pep-3121} for more details.
@end deffn

@geindex PyModuleDef.m_methods (C member)
@anchor{c-api/module c PyModuleDef m_methods}@anchor{4473}
@deffn {C Member} @ref{1014,,PyMethodDef}  *m_methods

A pointer to a table of module-level functions, described by
@ref{1014,,PyMethodDef} values.  Can be @code{NULL} if no functions are present.
@end deffn

@geindex PyModuleDef.m_slots (C member)
@anchor{c-api/module c PyModuleDef m_slots}@anchor{4474}
@deffn {C Member} @ref{4475,,PyModuleDef_Slot}  *m_slots

An array of slot definitions for multi-phase initialization, terminated by
a @code{@{0, NULL@}} entry.
When using single-phase initialization, @emph{m_slots} must be @code{NULL}.


Changed in version 3.5: Prior to version 3.5, this member was always set to @code{NULL},
and was defined as:

@quotation

@geindex PyModuleDef.m_slots.m_reload (C member)
@anchor{c-api/module c PyModuleDef m_slots m_reload}@anchor{4476}
@deffn {C Member} @ref{42af,,inquiry}  m_reload
@end deffn
@end quotation

@end deffn

@geindex PyModuleDef.m_traverse (C member)
@anchor{c-api/module c PyModuleDef m_traverse}@anchor{481}
@deffn {C Member} @ref{42bc,,traverseproc}  m_traverse

A traversal function to call during GC traversal of the module object, or
@code{NULL} if not needed.

This function is not called if the module state was requested but is not
allocated yet. This is the case immediately after the module is created
and before the module is executed (@ref{485,,Py_mod_exec} function). More
precisely, this function is not called if @ref{486,,m_size} is greater
than 0 and the module state (as returned by @ref{487,,PyModule_GetState()})
is @code{NULL}.


Changed in version 3.9: No longer called before the module state is allocated.

@end deffn

@geindex PyModuleDef.m_clear (C member)
@anchor{c-api/module c PyModuleDef m_clear}@anchor{482}
@deffn {C Member} @ref{42af,,inquiry}  m_clear

A clear function to call during GC clearing of the module object, or
@code{NULL} if not needed.

This function is not called if the module state was requested but is not
allocated yet. This is the case immediately after the module is created
and before the module is executed (@ref{485,,Py_mod_exec} function). More
precisely, this function is not called if @ref{486,,m_size} is greater
than 0 and the module state (as returned by @ref{487,,PyModule_GetState()})
is @code{NULL}.

Like @ref{40bf,,PyTypeObject.tp_clear}, this function is not @emph{always}
called before a module is deallocated. For example, when reference
counting is enough to determine that an object is no longer used,
the cyclic garbage collector is not involved and
@ref{483,,m_free} is called directly.


Changed in version 3.9: No longer called before the module state is allocated.

@end deffn

@geindex PyModuleDef.m_free (C member)
@anchor{c-api/module c PyModuleDef m_free}@anchor{483}
@deffn {C Member} @ref{4477,,freefunc}  m_free

A function to call during deallocation of the module object, or @code{NULL}
if not needed.

This function is not called if the module state was requested but is not
allocated yet. This is the case immediately after the module is created
and before the module is executed (@ref{485,,Py_mod_exec} function). More
precisely, this function is not called if @ref{486,,m_size} is greater
than 0 and the module state (as returned by @ref{487,,PyModule_GetState()})
is @code{NULL}.


Changed in version 3.9: No longer called before the module state is allocated.

@end deffn
@end deffn

@menu
* Single-phase initialization:: 
* Multi-phase initialization:: 
* Low-level module creation functions:: 
* Support functions:: 

@end menu

@node Single-phase initialization,Multi-phase initialization,,Initializing C modules
@anchor{c-api/module single-phase-initialization}@anchor{4478}
@subsubsection Single-phase initialization


The module initialization function may create and return the module object
directly. This is referred to as “single-phase initialization”, and uses one
of the following two module creation functions:

@geindex PyModule_Create (C function)
@anchor{c-api/module c PyModule_Create}@anchor{407a}
@deffn {C Function} @ref{375,,PyObject}  *PyModule_Create (PyModuleDef  *def)
@emph{Return value: New reference.}
Create a new module object, given the definition in @emph{def}.  This behaves
like @ref{41cb,,PyModule_Create2()} with @emph{module_api_version} set to
@code{PYTHON_API_VERSION}.
@end deffn

@geindex PyModule_Create2 (C function)
@anchor{c-api/module c PyModule_Create2}@anchor{41cb}
@deffn {C Function} @ref{375,,PyObject}  *PyModule_Create2 (PyModuleDef  *def, int  module_api_version)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a new module object, given the definition in @emph{def}, assuming the
API version @emph{module_api_version}.  If that version does not match the version
of the running interpreter, a @ref{5b5,,RuntimeWarning} is emitted.

@cartouche
@quotation Note 
Most uses of this function should be using @ref{407a,,PyModule_Create()}
instead; only use this if you are sure you need it.
@end quotation
@end cartouche
@end deffn

Before it is returned from in the initialization function, the resulting module
object is typically populated using functions like @ref{368,,PyModule_AddObjectRef()}.

@node Multi-phase initialization,Low-level module creation functions,Single-phase initialization,Initializing C modules
@anchor{c-api/module id1}@anchor{4479}@anchor{c-api/module multi-phase-initialization}@anchor{40d9}
@subsubsection Multi-phase initialization


An alternate way to specify extensions is to request “multi-phase initialization”.
Extension modules created this way behave more like Python modules: the
initialization is split between the @emph{creation phase}, when the module object
is created, and the @emph{execution phase}, when it is populated.
The distinction is similar to the @code{__new__()} and @code{__init__()} methods
of classes.

Unlike modules created using single-phase initialization, these modules are not
singletons: if the @emph{sys.modules} entry is removed and the module is re-imported,
a new module object is created, and the old module is subject to normal garbage
collection – as with Python modules.
By default, multiple modules created from the same definition should be
independent: changes to one should not affect the others.
This means that all state should be specific to the module object (using e.g.
using @ref{487,,PyModule_GetState()}), or its contents (such as the module’s
@code{__dict__} or individual classes created with @ref{469,,PyType_FromSpec()}).

All modules created using multi-phase initialization are expected to support
@ref{42fb,,sub-interpreters}. Making sure multiple modules
are independent is typically enough to achieve this.

To request multi-phase initialization, the initialization function
(PyInit_modulename) returns a @ref{484,,PyModuleDef} instance with non-empty
@ref{4474,,m_slots}. Before it is returned, the @code{PyModuleDef}
instance must be initialized with the following function:

@geindex PyModuleDef_Init (C function)
@anchor{c-api/module c PyModuleDef_Init}@anchor{41c7}
@deffn {C Function} @ref{375,,PyObject}  *PyModuleDef_Init (PyModuleDef  *def)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.5.}
Ensures a module definition is a properly initialized Python object that
correctly reports its type and reference count.

Returns @emph{def} cast to @code{PyObject*}, or @code{NULL} if an error occurred.


New in version 3.5.

@end deffn

The @emph{m_slots} member of the module definition must point to an array of
@code{PyModuleDef_Slot} structures:

@geindex PyModuleDef_Slot (C type)
@anchor{c-api/module c PyModuleDef_Slot}@anchor{4475}
@deffn {C Type} type  PyModuleDef_Slot

@geindex PyModuleDef_Slot.slot (C member)
@anchor{c-api/module c PyModuleDef_Slot slot}@anchor{447a}
@deffn {C Member} int  slot

A slot ID, chosen from the available values explained below.
@end deffn

@geindex PyModuleDef_Slot.value (C member)
@anchor{c-api/module c PyModuleDef_Slot value}@anchor{447b}
@deffn {C Member} void  *value

Value of the slot, whose meaning depends on the slot ID.
@end deffn


New in version 3.5.

@end deffn

The @emph{m_slots} array must be terminated by a slot with id 0.

The available slot types are:

@geindex Py_mod_create (C macro)
@anchor{c-api/module c Py_mod_create}@anchor{447c}
@deffn {C Macro} Py_mod_create

Specifies a function that is called to create the module object itself.
The @emph{value} pointer of this slot must point to a function of the signature:

@geindex Py_mod_create.create_module (C function)
@anchor{c-api/module c Py_mod_create create_module}@anchor{447d}
@deffn {C Function} @ref{375,,PyObject}  *create_module (PyObject  *spec, PyModuleDef  *def)
@end deffn

The function receives a @ref{1853,,ModuleSpec}
instance, as defined in 
@geindex Python Enhancement Proposals; PEP 451
PEP 451@footnote{https://www.python.org/dev/peps/pep-0451}, and the module definition.
It should return a new module object, or set an error
and return @code{NULL}.

This function should be kept minimal. In particular, it should not
call arbitrary Python code, as trying to import the same module again may
result in an infinite loop.

Multiple @code{Py_mod_create} slots may not be specified in one module
definition.

If @code{Py_mod_create} is not specified, the import machinery will create
a normal module object using @ref{41d2,,PyModule_New()}. The name is taken from
@emph{spec}, not the definition, to allow extension modules to dynamically adjust
to their place in the module hierarchy and be imported under different
names through symlinks, all while sharing a single module definition.

There is no requirement for the returned object to be an instance of
@ref{41d5,,PyModule_Type}. Any type can be used, as long as it supports
setting and getting import-related attributes.
However, only @code{PyModule_Type} instances may be returned if the
@code{PyModuleDef} has non-@code{NULL} @code{m_traverse}, @code{m_clear},
@code{m_free}; non-zero @code{m_size}; or slots other than @code{Py_mod_create}.
@end deffn

@geindex Py_mod_exec (C macro)
@anchor{c-api/module c Py_mod_exec}@anchor{485}
@deffn {C Macro} Py_mod_exec

Specifies a function that is called to @emph{execute} the module.
This is equivalent to executing the code of a Python module: typically,
this function adds classes and constants to the module.
The signature of the function is:

@geindex Py_mod_exec.exec_module (C function)
@anchor{c-api/module c Py_mod_exec exec_module}@anchor{447e}
@deffn {C Function} int  exec_module (PyObject  *module)
@end deffn

If multiple @code{Py_mod_exec} slots are specified, they are processed in the
order they appear in the @emph{m_slots} array.
@end deffn

See 
@geindex Python Enhancement Proposals; PEP 489
PEP 489@footnote{https://www.python.org/dev/peps/pep-0489} for more details on multi-phase initialization.

@node Low-level module creation functions,Support functions,Multi-phase initialization,Initializing C modules
@anchor{c-api/module low-level-module-creation-functions}@anchor{447f}
@subsubsection Low-level module creation functions


The following functions are called under the hood when using multi-phase
initialization. They can be used directly, for example when creating module
objects dynamically. Note that both @code{PyModule_FromDefAndSpec} and
@code{PyModule_ExecDef} must be called to fully initialize a module.

@geindex PyModule_FromDefAndSpec (C function)
@anchor{c-api/module c PyModule_FromDefAndSpec}@anchor{a24}
@deffn {C Function} @ref{375,,PyObject}  *PyModule_FromDefAndSpec (PyModuleDef  *def, PyObject  *spec)
@emph{Return value: New reference.}
Create a new module object, given the definition in @emph{module} and the
ModuleSpec @emph{spec}.  This behaves like @ref{a25,,PyModule_FromDefAndSpec2()}
with @emph{module_api_version} set to @code{PYTHON_API_VERSION}.


New in version 3.5.

@end deffn

@geindex PyModule_FromDefAndSpec2 (C function)
@anchor{c-api/module c PyModule_FromDefAndSpec2}@anchor{a25}
@deffn {C Function} @ref{375,,PyObject}  *PyModule_FromDefAndSpec2 (PyModuleDef  *def, PyObject  *spec, int  module_api_version)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Create a new module object, given the definition in @emph{module} and the
ModuleSpec @emph{spec}, assuming the API version @emph{module_api_version}.
If that version does not match the version of the running interpreter,
a @ref{5b5,,RuntimeWarning} is emitted.

@cartouche
@quotation Note 
Most uses of this function should be using @ref{a24,,PyModule_FromDefAndSpec()}
instead; only use this if you are sure you need it.
@end quotation
@end cartouche


New in version 3.5.

@end deffn

@geindex PyModule_ExecDef (C function)
@anchor{c-api/module c PyModule_ExecDef}@anchor{a26}
@deffn {C Function} int  PyModule_ExecDef (PyObject  *module, PyModuleDef  *def)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Process any execution slots (@ref{485,,Py_mod_exec}) given in @emph{def}.


New in version 3.5.

@end deffn

@geindex PyModule_SetDocString (C function)
@anchor{c-api/module c PyModule_SetDocString}@anchor{41d4}
@deffn {C Function} int  PyModule_SetDocString (PyObject  *module, const  char  *docstring)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Set the docstring for @emph{module} to @emph{docstring}.
This function is called automatically when creating a module from
@code{PyModuleDef}, using either @code{PyModule_Create} or
@code{PyModule_FromDefAndSpec}.


New in version 3.5.

@end deffn

@geindex PyModule_AddFunctions (C function)
@anchor{c-api/module c PyModule_AddFunctions}@anchor{41c8}
@deffn {C Function} int  PyModule_AddFunctions (PyObject  *module, PyMethodDef  *functions)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Add the functions from the @code{NULL} terminated @emph{functions} array to @emph{module}.
Refer to the @ref{1014,,PyMethodDef} documentation for details on individual
entries (due to the lack of a shared module namespace, module level
“functions” implemented in C typically receive the module as their first
parameter, making them similar to instance methods on Python classes).
This function is called automatically when creating a module from
@code{PyModuleDef}, using either @code{PyModule_Create} or
@code{PyModule_FromDefAndSpec}.


New in version 3.5.

@end deffn

@node Support functions,,Low-level module creation functions,Initializing C modules
@anchor{c-api/module support-functions}@anchor{4480}
@subsubsection Support functions


The module initialization function (if using single phase initialization) or
a function called from a module execution slot (if using multi-phase
initialization), can use the following functions to help initialize the module
state:

@geindex PyModule_AddObjectRef (C function)
@anchor{c-api/module c PyModule_AddObjectRef}@anchor{368}
@deffn {C Function} int  PyModule_AddObjectRef (PyObject  *module, const  char  *name, PyObject  *value)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Add an object to @emph{module} as @emph{name}.  This is a convenience function which
can be used from the module’s initialization function.

On success, return @code{0}. On error, raise an exception and return @code{-1}.

Return @code{NULL} if @emph{value} is @code{NULL}. It must be called with an exception
raised in this case.

Example usage:

@example
static int
add_spam(PyObject *module, int value)
@{
    PyObject *obj = PyLong_FromLong(value);
    if (obj == NULL) @{
        return -1;
    @}
    int res = PyModule_AddObjectRef(module, "spam", obj);
    Py_DECREF(obj);
    return res;
 @}
@end example

The example can also be written without checking explicitly if @emph{obj} is
@code{NULL}:

@example
static int
add_spam(PyObject *module, int value)
@{
    PyObject *obj = PyLong_FromLong(value);
    int res = PyModule_AddObjectRef(module, "spam", obj);
    Py_XDECREF(obj);
    return res;
 @}
@end example

Note that @code{Py_XDECREF()} should be used instead of @code{Py_DECREF()} in
this case, since @emph{obj} can be @code{NULL}.


New in version 3.10.

@end deffn

@geindex PyModule_AddObject (C function)
@anchor{c-api/module c PyModule_AddObject}@anchor{369}
@deffn {C Function} int  PyModule_AddObject (PyObject  *module, const  char  *name, PyObject  *value)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Similar to @ref{368,,PyModule_AddObjectRef()}, but steals a reference to
@emph{value} on success (if it returns @code{0}).

The new @ref{368,,PyModule_AddObjectRef()} function is recommended, since it is
easy to introduce reference leaks by misusing the
@ref{369,,PyModule_AddObject()} function.

@cartouche
@quotation Note 
Unlike other functions that steal references, @code{PyModule_AddObject()}
only decrements the reference count of @emph{value} @strong{on success}.

This means that its return value must be checked, and calling code must
@ref{373,,Py_DECREF()} @emph{value} manually on error.
@end quotation
@end cartouche

Example usage:

@example
static int
add_spam(PyObject *module, int value)
@{
    PyObject *obj = PyLong_FromLong(value);
    if (obj == NULL) @{
        return -1;
    @}
    if (PyModule_AddObject(module, "spam", obj) < 0) @{
        Py_DECREF(obj);
        return -1;
    @}
    // PyModule_AddObject() stole a reference to obj:
    // Py_DECREF(obj) is not needed here
    return 0;
@}
@end example

The example can also be written without checking explicitly if @emph{obj} is
@code{NULL}:

@example
static int
add_spam(PyObject *module, int value)
@{
    PyObject *obj = PyLong_FromLong(value);
    if (PyModule_AddObject(module, "spam", obj) < 0) @{
        Py_XDECREF(obj);
        return -1;
    @}
    // PyModule_AddObject() stole a reference to obj:
    // Py_DECREF(obj) is not needed here
    return 0;
@}
@end example

Note that @code{Py_XDECREF()} should be used instead of @code{Py_DECREF()} in
this case, since @emph{obj} can be @code{NULL}.
@end deffn

@geindex PyModule_AddIntConstant (C function)
@anchor{c-api/module c PyModule_AddIntConstant}@anchor{41c9}
@deffn {C Function} int  PyModule_AddIntConstant (PyObject  *module, const  char  *name, long  value)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Add an integer constant to @emph{module} as @emph{name}.  This convenience function can be
used from the module’s initialization function. Return @code{-1} on error, @code{0} on
success.
@end deffn

@geindex PyModule_AddStringConstant (C function)
@anchor{c-api/module c PyModule_AddStringConstant}@anchor{41ca}
@deffn {C Function} int  PyModule_AddStringConstant (PyObject  *module, const  char  *name, const  char  *value)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Add a string constant to @emph{module} as @emph{name}.  This convenience function can be
used from the module’s initialization function.  The string @emph{value} must be
@code{NULL}-terminated.  Return @code{-1} on error, @code{0} on success.
@end deffn

@geindex PyModule_AddIntMacro (C function)
@anchor{c-api/module c PyModule_AddIntMacro}@anchor{f54}
@deffn {C Function} int  PyModule_AddIntMacro (PyObject  *module, macro)

Add an int constant to @emph{module}. The name and the value are taken from
@emph{macro}. For example @code{PyModule_AddIntMacro(module, AF_INET)} adds the int
constant @emph{AF_INET} with the value of @emph{AF_INET} to @emph{module}.
Return @code{-1} on error, @code{0} on success.
@end deffn

@geindex PyModule_AddStringMacro (C function)
@anchor{c-api/module c PyModule_AddStringMacro}@anchor{f53}
@deffn {C Function} int  PyModule_AddStringMacro (PyObject  *module, macro)

Add a string constant to @emph{module}.
@end deffn

@geindex PyModule_AddType (C function)
@anchor{c-api/module c PyModule_AddType}@anchor{47c}
@deffn {C Function} int  PyModule_AddType (PyObject  *module, PyTypeObject  *type)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Add a type object to @emph{module}.
The type object is finalized by calling internally @ref{1ec,,PyType_Ready()}.
The name of the type object is taken from the last component of
@ref{116b,,tp_name} after dot.
Return @code{-1} on error, @code{0} on success.


New in version 3.9.

@end deffn

@node Module lookup,,Initializing C modules,Module Objects
@anchor{c-api/module module-lookup}@anchor{4481}
@subsubsection Module lookup


Single-phase initialization creates singleton modules that can be looked up
in the context of the current interpreter. This allows the module object to be
retrieved later with only a reference to the module definition.

These functions will not work on modules created using multi-phase initialization,
since multiple such modules can be created from a single definition.

@geindex PyState_FindModule (C function)
@anchor{c-api/module c PyState_FindModule}@anchor{422c}
@deffn {C Function} @ref{375,,PyObject}  *PyState_FindModule (PyModuleDef  *def)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns the module object that was created from @emph{def} for the current interpreter.
This method requires that the module object has been attached to the interpreter state with
@ref{422b,,PyState_AddModule()} beforehand. In case the corresponding module object is not
found or has not been attached to the interpreter state yet, it returns @code{NULL}.
@end deffn

@geindex PyState_AddModule (C function)
@anchor{c-api/module c PyState_AddModule}@anchor{422b}
@deffn {C Function} int  PyState_AddModule (PyObject  *module, PyModuleDef  *def)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.3.}
Attaches the module object passed to the function to the interpreter state. This allows
the module object to be accessible via @ref{422c,,PyState_FindModule()}.

Only effective on modules created using single-phase initialization.

Python calls @code{PyState_AddModule} automatically after importing a module,
so it is unnecessary (but harmless) to call it from module initialization
code. An explicit call is needed only if the module’s own init code
subsequently calls @code{PyState_FindModule}.
The function is mainly intended for implementing alternative import
mechanisms (either by calling it directly, or by referring to its
implementation for details of the required state updates).

The caller must hold the GIL.

Return 0 on success or -1 on failure.


New in version 3.3.

@end deffn

@geindex PyState_RemoveModule (C function)
@anchor{c-api/module c PyState_RemoveModule}@anchor{422d}
@deffn {C Function} int  PyState_RemoveModule (PyModuleDef  *def)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.3.}
Removes the module object created from @emph{def} from the interpreter state.
Return 0 on success or -1 on failure.

The caller must hold the GIL.


New in version 3.3.

@end deffn

@node Iterator Objects,Descriptor Objects,Module Objects,Other Objects
@anchor{c-api/iterator doc}@anchor{4482}@anchor{c-api/iterator id1}@anchor{4483}@anchor{c-api/iterator iterator-objects}@anchor{4484}
@subsubsection Iterator Objects


Python provides two general-purpose iterator objects.  The first, a sequence
iterator, works with an arbitrary sequence supporting the @code{__getitem__()}
method.  The second works with a callable object and a sentinel value, calling
the callable for each item in the sequence, and ending the iteration when the
sentinel value is returned.

@geindex PySeqIter_Type (C var)
@anchor{c-api/iterator c PySeqIter_Type}@anchor{4215}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PySeqIter_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
Type object for iterator objects returned by @ref{4214,,PySeqIter_New()} and the
one-argument form of the @ref{251,,iter()} built-in function for built-in sequence
types.
@end deffn

@geindex PySeqIter_Check (C function)
@anchor{c-api/iterator c PySeqIter_Check}@anchor{4485}
@deffn {C Function} int  PySeqIter_Check (op)

Return true if the type of @emph{op} is @ref{4215,,PySeqIter_Type}.  This function
always succeeds.
@end deffn

@geindex PySeqIter_New (C function)
@anchor{c-api/iterator c PySeqIter_New}@anchor{4214}
@deffn {C Function} @ref{375,,PyObject}  *PySeqIter_New (PyObject  *seq)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return an iterator that works with a general sequence object, @emph{seq}.  The
iteration ends when the sequence raises @ref{1066,,IndexError} for the subscripting
operation.
@end deffn

@geindex PyCallIter_Type (C var)
@anchor{c-api/iterator c PyCallIter_Type}@anchor{413c}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyCallIter_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
Type object for iterator objects returned by @ref{413b,,PyCallIter_New()} and the
two-argument form of the @ref{251,,iter()} built-in function.
@end deffn

@geindex PyCallIter_Check (C function)
@anchor{c-api/iterator c PyCallIter_Check}@anchor{4486}
@deffn {C Function} int  PyCallIter_Check (op)

Return true if the type of @emph{op} is @ref{413c,,PyCallIter_Type}.  This
function always succeeds.
@end deffn

@geindex PyCallIter_New (C function)
@anchor{c-api/iterator c PyCallIter_New}@anchor{413b}
@deffn {C Function} @ref{375,,PyObject}  *PyCallIter_New (PyObject  *callable, PyObject  *sentinel)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new iterator.  The first parameter, @emph{callable}, can be any Python
callable object that can be called with no parameters; each call to it should
return the next item in the iteration.  When @emph{callable} returns a value equal to
@emph{sentinel}, the iteration will be terminated.
@end deffn

@node Descriptor Objects,Slice Objects,Iterator Objects,Other Objects
@anchor{c-api/descriptor doc}@anchor{4487}@anchor{c-api/descriptor descriptor-objects}@anchor{4488}@anchor{c-api/descriptor id1}@anchor{4489}
@subsubsection Descriptor Objects


“Descriptors” are objects that describe some attribute of an object. They are
found in the dictionary of type objects.

@c XXX document these!

@geindex PyProperty_Type (C var)
@anchor{c-api/descriptor c PyProperty_Type}@anchor{4213}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyProperty_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
The type object for the built-in descriptor types.
@end deffn

@geindex PyDescr_NewGetSet (C function)
@anchor{c-api/descriptor c PyDescr_NewGetSet}@anchor{415d}
@deffn {C Function} @ref{375,,PyObject}  *PyDescr_NewGetSet (PyTypeObject  *type, struct  PyGetSetDef  *getset)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex PyDescr_NewMember (C function)
@anchor{c-api/descriptor c PyDescr_NewMember}@anchor{415e}
@deffn {C Function} @ref{375,,PyObject}  *PyDescr_NewMember (PyTypeObject  *type, struct  PyMemberDef  *meth)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex PyDescr_NewMethod (C function)
@anchor{c-api/descriptor c PyDescr_NewMethod}@anchor{121b}
@deffn {C Function} @ref{375,,PyObject}  *PyDescr_NewMethod (PyTypeObject  *type, struct  PyMethodDef  *meth)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex PyDescr_NewWrapper (C function)
@anchor{c-api/descriptor c PyDescr_NewWrapper}@anchor{448a}
@deffn {C Function} @ref{375,,PyObject}  *PyDescr_NewWrapper (PyTypeObject  *type, struct  wrapperbase  *wrapper, void  *wrapped)
@emph{Return value: New reference.}
@end deffn

@geindex PyDescr_NewClassMethod (C function)
@anchor{c-api/descriptor c PyDescr_NewClassMethod}@anchor{415c}
@deffn {C Function} @ref{375,,PyObject}  *PyDescr_NewClassMethod (PyTypeObject  *type, PyMethodDef  *method)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex PyDescr_IsData (C function)
@anchor{c-api/descriptor c PyDescr_IsData}@anchor{116d}
@deffn {C Function} int  PyDescr_IsData (PyObject  *descr)

Return non-zero if the descriptor objects @emph{descr} describes a data attribute, or
@code{0} if it describes a method.  @emph{descr} must be a descriptor object; there is
no error checking.
@end deffn

@geindex PyWrapper_New (C function)
@anchor{c-api/descriptor c PyWrapper_New}@anchor{4296}
@deffn {C Function} @ref{375,,PyObject}  *PyWrapper_New (PyObject*, PyObject*)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@node Slice Objects,Ellipsis Object,Descriptor Objects,Other Objects
@anchor{c-api/slice doc}@anchor{448b}@anchor{c-api/slice id1}@anchor{448c}@anchor{c-api/slice slice-objects}@anchor{448d}
@subsubsection Slice Objects


@geindex PySlice_Type (C var)
@anchor{c-api/slice c PySlice_Type}@anchor{422a}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PySlice_Type
@emph{ Part of the @ref{35d,,Stable ABI}.}
The type object for slice objects.  This is the same as @ref{ffb,,slice} in the
Python layer.
@end deffn

@geindex PySlice_Check (C function)
@anchor{c-api/slice c PySlice_Check}@anchor{448e}
@deffn {C Function} int  PySlice_Check (PyObject  *ob)

Return true if @emph{ob} is a slice object; @emph{ob} must not be @code{NULL}.  This
function always succeeds.
@end deffn

@geindex PySlice_New (C function)
@anchor{c-api/slice c PySlice_New}@anchor{4229}
@deffn {C Function} @ref{375,,PyObject}  *PySlice_New (PyObject  *start, PyObject  *stop, PyObject  *step)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a new slice object with the given values.  The @emph{start}, @emph{stop}, and
@emph{step} parameters are used as the values of the slice object attributes of
the same names.  Any of the values may be @code{NULL}, in which case the
@code{None} will be used for the corresponding attribute.  Return @code{NULL} if
the new object could not be allocated.
@end deffn

@geindex PySlice_GetIndices (C function)
@anchor{c-api/slice c PySlice_GetIndices}@anchor{4228}
@deffn {C Function} int  PySlice_GetIndices (PyObject  *slice, Py_ssize_t  length, Py_ssize_t  *start, Py_ssize_t  *stop, Py_ssize_t  *step)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Retrieve the start, stop and step indices from the slice object @emph{slice},
assuming a sequence of length @emph{length}. Treats indices greater than
@emph{length} as errors.

Returns @code{0} on success and @code{-1} on error with no exception set (unless one of
the indices was not @ref{182,,None} and failed to be converted to an integer,
in which case @code{-1} is returned with an exception set).

You probably do not want to use this function.


Changed in version 3.2: The parameter type for the @emph{slice} parameter was @code{PySliceObject*}
before.

@end deffn

@geindex PySlice_GetIndicesEx (C function)
@anchor{c-api/slice c PySlice_GetIndicesEx}@anchor{72b}
@deffn {C Function} int  PySlice_GetIndicesEx (PyObject  *slice, Py_ssize_t  length, Py_ssize_t  *start, Py_ssize_t  *stop, Py_ssize_t  *step, Py_ssize_t  *slicelength)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Usable replacement for @ref{4228,,PySlice_GetIndices()}.  Retrieve the start,
stop, and step indices from the slice object @emph{slice} assuming a sequence of
length @emph{length}, and store the length of the slice in @emph{slicelength}.  Out
of bounds indices are clipped in a manner consistent with the handling of
normal slices.

Returns @code{0} on success and @code{-1} on error with exception set.

@cartouche
@quotation Note 
This function is considered not safe for resizable sequences.
Its invocation should be replaced by a combination of
@ref{6f3,,PySlice_Unpack()} and @ref{6f4,,PySlice_AdjustIndices()} where

@example
if (PySlice_GetIndicesEx(slice, length, &start, &stop, &step, &slicelength) < 0) @{
    // return error
@}
@end example

is replaced by

@example
if (PySlice_Unpack(slice, &start, &stop, &step) < 0) @{
    // return error
@}
slicelength = PySlice_AdjustIndices(length, &start, &stop, step);
@end example
@end quotation
@end cartouche


Changed in version 3.2: The parameter type for the @emph{slice} parameter was @code{PySliceObject*}
before.



Changed in version 3.6.1: If @code{Py_LIMITED_API} is not set or set to the value between @code{0x03050400}
and @code{0x03060000} (not including) or @code{0x03060100} or higher
!@ref{72b,,PySlice_GetIndicesEx} is implemented as a macro using
!@ref{6f3,,PySlice_Unpack} and !@ref{6f4,,PySlice_AdjustIndices}.
Arguments @emph{start}, @emph{stop} and @emph{step} are evaluated more than once.



Deprecated since version 3.6.1: If @code{Py_LIMITED_API} is set to the value less than @code{0x03050400} or
between @code{0x03060000} and @code{0x03060100} (not including)
!@ref{72b,,PySlice_GetIndicesEx} is a deprecated function.

@end deffn

@geindex PySlice_Unpack (C function)
@anchor{c-api/slice c PySlice_Unpack}@anchor{6f3}
@deffn {C Function} int  PySlice_Unpack (PyObject  *slice, Py_ssize_t  *start, Py_ssize_t  *stop, Py_ssize_t  *step)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Extract the start, stop and step data members from a slice object as
C integers.  Silently reduce values larger than @code{PY_SSIZE_T_MAX} to
@code{PY_SSIZE_T_MAX}, silently boost the start and stop values less than
@code{PY_SSIZE_T_MIN} to @code{PY_SSIZE_T_MIN}, and silently boost the step
values less than @code{-PY_SSIZE_T_MAX} to @code{-PY_SSIZE_T_MAX}.

Return @code{-1} on error, @code{0} on success.


New in version 3.6.1.

@end deffn

@geindex PySlice_AdjustIndices (C function)
@anchor{c-api/slice c PySlice_AdjustIndices}@anchor{6f4}
@deffn {C Function} Py_ssize_t  PySlice_AdjustIndices (Py_ssize_t  length, Py_ssize_t  *start, Py_ssize_t  *stop, Py_ssize_t  step)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Adjust start/end slice indices assuming a sequence of the specified length.
Out of bounds indices are clipped in a manner consistent with the handling
of normal slices.

Return the length of the slice.  Always successful.  Doesn’t call Python
code.


New in version 3.6.1.

@end deffn

@node Ellipsis Object,MemoryView objects,Slice Objects,Other Objects
@anchor{c-api/slice ellipsis-object}@anchor{448f}
@subsubsection Ellipsis Object


@geindex Py_Ellipsis (C var)
@anchor{c-api/slice c Py_Ellipsis}@anchor{4490}
@deffn {C Variable} @ref{375,,PyObject}  *Py_Ellipsis

The Python @code{Ellipsis} object.  This object has no methods.  It needs to be
treated just like any other object with respect to reference counts.  Like
@ref{4077,,Py_None} it is a singleton object.
@end deffn
@anchor{c-api/memoryview memoryview-objects}@anchor{435e}
@geindex object; memoryview

@node MemoryView objects,Weak Reference Objects<2>,Ellipsis Object,Other Objects
@anchor{c-api/memoryview doc}@anchor{4491}@anchor{c-api/memoryview id1}@anchor{4492}
@subsubsection MemoryView objects


A @ref{573,,memoryview} object exposes the C level @ref{1a3e,,buffer interface} as a Python object which can then be passed around like
any other object.

@geindex PyMemoryView_FromObject (C function)
@anchor{c-api/memoryview c PyMemoryView_FromObject}@anchor{41c5}
@deffn {C Function} @ref{375,,PyObject}  *PyMemoryView_FromObject (PyObject  *obj)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a memoryview object from an object that provides the buffer interface.
If @emph{obj} supports writable buffer exports, the memoryview object will be
read/write, otherwise it may be either read-only or read/write at the
discretion of the exporter.
@end deffn

@geindex PyMemoryView_FromMemory (C function)
@anchor{c-api/memoryview c PyMemoryView_FromMemory}@anchor{d0d}
@deffn {C Function} @ref{375,,PyObject}  *PyMemoryView_FromMemory (char  *mem, Py_ssize_t  size, int  flags)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Create a memoryview object using @emph{mem} as the underlying buffer.
@emph{flags} can be one of @code{PyBUF_READ} or @code{PyBUF_WRITE}.


New in version 3.3.

@end deffn

@geindex PyMemoryView_FromBuffer (C function)
@anchor{c-api/memoryview c PyMemoryView_FromBuffer}@anchor{1e6}
@deffn {C Function} @ref{375,,PyObject}  *PyMemoryView_FromBuffer (const  Py_buffer  *view)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
Create a memoryview object wrapping the given buffer structure @emph{view}.
For simple byte buffers, @ref{d0d,,PyMemoryView_FromMemory()} is the preferred
function.
@end deffn

@geindex PyMemoryView_GetContiguous (C function)
@anchor{c-api/memoryview c PyMemoryView_GetContiguous}@anchor{41c6}
@deffn {C Function} @ref{375,,PyObject}  *PyMemoryView_GetContiguous (PyObject  *obj, int  buffertype, char  order)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a memoryview object to a @ref{1b0f,,contiguous} chunk of memory (in either
‘C’ or ‘F’ortran @emph{order}) from an object that defines the buffer
interface. If memory is contiguous, the memoryview object points to the
original memory. Otherwise, a copy is made and the memoryview points to a
new bytes object.
@end deffn

@geindex PyMemoryView_Check (C function)
@anchor{c-api/memoryview c PyMemoryView_Check}@anchor{4493}
@deffn {C Function} int  PyMemoryView_Check (PyObject  *obj)

Return true if the object @emph{obj} is a memoryview object.  It is not
currently allowed to create subclasses of @ref{573,,memoryview}.  This
function always succeeds.
@end deffn

@geindex PyMemoryView_GET_BUFFER (C function)
@anchor{c-api/memoryview c PyMemoryView_GET_BUFFER}@anchor{4494}
@deffn {C Function} @ref{1db,,Py_buffer}  *PyMemoryView_GET_BUFFER (PyObject  *mview)

Return a pointer to the memoryview’s private copy of the exporter’s buffer.
@emph{mview} @strong{must} be a memoryview instance; this macro doesn’t check its type,
you must do it yourself or you will risk crashes.
@end deffn

@geindex PyMemoryView_GET_BASE (C function)
@anchor{c-api/memoryview c PyMemoryView_GET_BASE}@anchor{4495}
@deffn {C Function} @ref{1db,,Py_buffer}  *PyMemoryView_GET_BASE (PyObject  *mview)

Return either a pointer to the exporting object that the memoryview is based
on or @code{NULL} if the memoryview has been created by one of the functions
@ref{d0d,,PyMemoryView_FromMemory()} or @ref{1e6,,PyMemoryView_FromBuffer()}.
@emph{mview} @strong{must} be a memoryview instance.
@end deffn

@node Weak Reference Objects<2>,Capsules<2>,MemoryView objects,Other Objects
@anchor{c-api/weakref doc}@anchor{4496}@anchor{c-api/weakref weak-reference-objects}@anchor{4497}@anchor{c-api/weakref weakrefobjects}@anchor{4498}
@subsubsection Weak Reference Objects


Python supports @emph{weak references} as first-class objects.  There are two
specific object types which directly implement weak references.  The first is a
simple reference object, and the second acts as a proxy for the original object
as much as it can.

@geindex PyWeakref_Check (C function)
@anchor{c-api/weakref c PyWeakref_Check}@anchor{4499}
@deffn {C Function} int  PyWeakref_Check (ob)

Return true if @emph{ob} is either a reference or proxy object.  This function
always succeeds.
@end deffn

@geindex PyWeakref_CheckRef (C function)
@anchor{c-api/weakref c PyWeakref_CheckRef}@anchor{449a}
@deffn {C Function} int  PyWeakref_CheckRef (ob)

Return true if @emph{ob} is a reference object.  This function always succeeds.
@end deffn

@geindex PyWeakref_CheckProxy (C function)
@anchor{c-api/weakref c PyWeakref_CheckProxy}@anchor{449b}
@deffn {C Function} int  PyWeakref_CheckProxy (ob)

Return true if @emph{ob} is a proxy object.  This function always succeeds.
@end deffn

@geindex PyWeakref_NewRef (C function)
@anchor{c-api/weakref c PyWeakref_NewRef}@anchor{10a1}
@deffn {C Function} @ref{375,,PyObject}  *PyWeakref_NewRef (PyObject  *ob, PyObject  *callback)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a weak reference object for the object @emph{ob}.  This will always return
a new reference, but is not guaranteed to create a new object; an existing
reference object may be returned.  The second parameter, @emph{callback}, can be a
callable object that receives notification when @emph{ob} is garbage collected; it
should accept a single parameter, which will be the weak reference object
itself. @emph{callback} may also be @code{None} or @code{NULL}.  If @emph{ob} is not a
weakly-referencable object, or if @emph{callback} is not callable, @code{None}, or
@code{NULL}, this will return @code{NULL} and raise @ref{162,,TypeError}.
@end deffn

@geindex PyWeakref_NewProxy (C function)
@anchor{c-api/weakref c PyWeakref_NewProxy}@anchor{4295}
@deffn {C Function} @ref{375,,PyObject}  *PyWeakref_NewProxy (PyObject  *ob, PyObject  *callback)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a weak reference proxy object for the object @emph{ob}.  This will always
return a new reference, but is not guaranteed to create a new object; an
existing proxy object may be returned.  The second parameter, @emph{callback}, can
be a callable object that receives notification when @emph{ob} is garbage
collected; it should accept a single parameter, which will be the weak
reference object itself. @emph{callback} may also be @code{None} or @code{NULL}.  If @emph{ob}
is not a weakly-referencable object, or if @emph{callback} is not callable,
@code{None}, or @code{NULL}, this will return @code{NULL} and raise @ref{162,,TypeError}.
@end deffn

@geindex PyWeakref_GetObject (C function)
@anchor{c-api/weakref c PyWeakref_GetObject}@anchor{4294}
@deffn {C Function} @ref{375,,PyObject}  *PyWeakref_GetObject (PyObject  *ref)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the referenced object from a weak reference, @emph{ref}.  If the referent is
no longer live, returns @code{Py_None}.

@cartouche
@quotation Note 
This function returns a @ref{4112,,borrowed reference} to the referenced object.
This means that you should always call @ref{372,,Py_INCREF()} on the object
except when it cannot be destroyed before the last usage of the borrowed
reference.
@end quotation
@end cartouche
@end deffn

@geindex PyWeakref_GET_OBJECT (C function)
@anchor{c-api/weakref c PyWeakref_GET_OBJECT}@anchor{211}
@deffn {C Function} @ref{375,,PyObject}  *PyWeakref_GET_OBJECT (PyObject  *ref)
@emph{Return value: Borrowed reference.}
Similar to @ref{4294,,PyWeakref_GetObject()}, but implemented as a macro that does no
error checking.
@end deffn

@node Capsules<2>,Generator Objects,Weak Reference Objects<2>,Other Objects
@anchor{c-api/capsule doc}@anchor{449c}@anchor{c-api/capsule capsules}@anchor{409d}@anchor{c-api/capsule id1}@anchor{449d}
@subsubsection Capsules


@geindex object; Capsule

Refer to @ref{ef1,,Providing a C API for an Extension Module} for more information on using these objects.


New in version 3.1.


@geindex PyCapsule (C type)
@anchor{c-api/capsule c PyCapsule}@anchor{e21}
@deffn {C Type} type  PyCapsule

This subtype of @ref{375,,PyObject} represents an opaque value, useful for C
extension modules who need to pass an opaque value (as a void*
pointer) through Python code to other C code.  It is often used to make a C
function pointer defined in one module available to other modules, so the
regular import mechanism can be used to access C APIs defined in dynamically
loaded modules.
@end deffn

@geindex PyCapsule_Destructor (C type)
@anchor{c-api/capsule c PyCapsule_Destructor}@anchor{413e}
@deffn {C Type} type  PyCapsule_Destructor
@emph{ Part of the @ref{35d,,Stable ABI}.}
The type of a destructor callback for a capsule.  Defined as:

@example
typedef void (*PyCapsule_Destructor)(PyObject *);
@end example

See @ref{409c,,PyCapsule_New()} for the semantics of PyCapsule_Destructor
callbacks.
@end deffn

@geindex PyCapsule_CheckExact (C function)
@anchor{c-api/capsule c PyCapsule_CheckExact}@anchor{449e}
@deffn {C Function} int  PyCapsule_CheckExact (PyObject  *p)

Return true if its argument is a @ref{e21,,PyCapsule}.  This function always
succeeds.
@end deffn

@geindex PyCapsule_New (C function)
@anchor{c-api/capsule c PyCapsule_New}@anchor{409c}
@deffn {C Function} @ref{375,,PyObject}  *PyCapsule_New (void  *pointer, const  char  *name, PyCapsule_Destructor  destructor)
@emph{Return value: New reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a @ref{e21,,PyCapsule} encapsulating the @emph{pointer}.  The @emph{pointer}
argument may not be @code{NULL}.

On failure, set an exception and return @code{NULL}.

The @emph{name} string may either be @code{NULL} or a pointer to a valid C string.  If
non-@code{NULL}, this string must outlive the capsule.  (Though it is permitted to
free it inside the @emph{destructor}.)

If the @emph{destructor} argument is not @code{NULL}, it will be called with the
capsule as its argument when it is destroyed.

If this capsule will be stored as an attribute of a module, the @emph{name} should
be specified as @code{modulename.attributename}.  This will enable other modules
to import the capsule using @ref{10c9,,PyCapsule_Import()}.
@end deffn

@geindex PyCapsule_GetPointer (C function)
@anchor{c-api/capsule c PyCapsule_GetPointer}@anchor{4142}
@deffn {C Function} void  *PyCapsule_GetPointer (PyObject  *capsule, const  char  *name)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Retrieve the @emph{pointer} stored in the capsule.  On failure, set an exception
and return @code{NULL}.

The @emph{name} parameter must compare exactly to the name stored in the capsule.
If the name stored in the capsule is @code{NULL}, the @emph{name} passed in must also
be @code{NULL}.  Python uses the C function @code{strcmp()} to compare capsule
names.
@end deffn

@geindex PyCapsule_GetDestructor (C function)
@anchor{c-api/capsule c PyCapsule_GetDestructor}@anchor{4140}
@deffn {C Function} @ref{413e,,PyCapsule_Destructor}  PyCapsule_GetDestructor (PyObject  *capsule)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the current destructor stored in the capsule.  On failure, set an
exception and return @code{NULL}.

It is legal for a capsule to have a @code{NULL} destructor.  This makes a @code{NULL}
return code somewhat ambiguous; use @ref{ef0,,PyCapsule_IsValid()} or
@ref{116a,,PyErr_Occurred()} to disambiguate.
@end deffn

@geindex PyCapsule_GetContext (C function)
@anchor{c-api/capsule c PyCapsule_GetContext}@anchor{413f}
@deffn {C Function} void  *PyCapsule_GetContext (PyObject  *capsule)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the current context stored in the capsule.  On failure, set an
exception and return @code{NULL}.

It is legal for a capsule to have a @code{NULL} context.  This makes a @code{NULL}
return code somewhat ambiguous; use @ref{ef0,,PyCapsule_IsValid()} or
@ref{116a,,PyErr_Occurred()} to disambiguate.
@end deffn

@geindex PyCapsule_GetName (C function)
@anchor{c-api/capsule c PyCapsule_GetName}@anchor{4141}
@deffn {C Function} const  char  *PyCapsule_GetName (PyObject  *capsule)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the current name stored in the capsule.  On failure, set an exception
and return @code{NULL}.

It is legal for a capsule to have a @code{NULL} name.  This makes a @code{NULL} return
code somewhat ambiguous; use @ref{ef0,,PyCapsule_IsValid()} or
@ref{116a,,PyErr_Occurred()} to disambiguate.
@end deffn

@geindex PyCapsule_Import (C function)
@anchor{c-api/capsule c PyCapsule_Import}@anchor{10c9}
@deffn {C Function} void  *PyCapsule_Import (const  char  *name, int  no_block)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Import a pointer to a C object from a capsule attribute in a module.  The
@emph{name} parameter should specify the full name to the attribute, as in
@code{module.attribute}.  The @emph{name} stored in the capsule must match this
string exactly.

Return the capsule’s internal @emph{pointer} on success.  On failure, set an
exception and return @code{NULL}.


Changed in version 3.3: @emph{no_block} has no effect anymore.

@end deffn

@geindex PyCapsule_IsValid (C function)
@anchor{c-api/capsule c PyCapsule_IsValid}@anchor{ef0}
@deffn {C Function} int  PyCapsule_IsValid (PyObject  *capsule, const  char  *name)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Determines whether or not @emph{capsule} is a valid capsule.  A valid capsule is
non-@code{NULL}, passes @ref{449e,,PyCapsule_CheckExact()}, has a non-@code{NULL} pointer
stored in it, and its internal name matches the @emph{name} parameter.  (See
@ref{4142,,PyCapsule_GetPointer()} for information on how capsule names are
compared.)

In other words, if @ref{ef0,,PyCapsule_IsValid()} returns a true value, calls to
any of the accessors (any function starting with @code{PyCapsule_Get()}) are
guaranteed to succeed.

Return a nonzero value if the object is valid and matches the name passed in.
Return @code{0} otherwise.  This function will not fail.
@end deffn

@geindex PyCapsule_SetContext (C function)
@anchor{c-api/capsule c PyCapsule_SetContext}@anchor{4143}
@deffn {C Function} int  PyCapsule_SetContext (PyObject  *capsule, void  *context)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the context pointer inside @emph{capsule} to @emph{context}.

Return @code{0} on success.  Return nonzero and set an exception on failure.
@end deffn

@geindex PyCapsule_SetDestructor (C function)
@anchor{c-api/capsule c PyCapsule_SetDestructor}@anchor{4144}
@deffn {C Function} int  PyCapsule_SetDestructor (PyObject  *capsule, PyCapsule_Destructor  destructor)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the destructor inside @emph{capsule} to @emph{destructor}.

Return @code{0} on success.  Return nonzero and set an exception on failure.
@end deffn

@geindex PyCapsule_SetName (C function)
@anchor{c-api/capsule c PyCapsule_SetName}@anchor{4145}
@deffn {C Function} int  PyCapsule_SetName (PyObject  *capsule, const  char  *name)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the name inside @emph{capsule} to @emph{name}.  If non-@code{NULL}, the name must
outlive the capsule.  If the previous @emph{name} stored in the capsule was not
@code{NULL}, no attempt is made to free it.

Return @code{0} on success.  Return nonzero and set an exception on failure.
@end deffn

@geindex PyCapsule_SetPointer (C function)
@anchor{c-api/capsule c PyCapsule_SetPointer}@anchor{4146}
@deffn {C Function} int  PyCapsule_SetPointer (PyObject  *capsule, void  *pointer)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the void pointer inside @emph{capsule} to @emph{pointer}.  The pointer may not be
@code{NULL}.

Return @code{0} on success.  Return nonzero and set an exception on failure.
@end deffn

@node Generator Objects,Coroutine Objects<2>,Capsules<2>,Other Objects
@anchor{c-api/gen doc}@anchor{449f}@anchor{c-api/gen gen-objects}@anchor{44a0}@anchor{c-api/gen generator-objects}@anchor{44a1}
@subsubsection Generator Objects


Generator objects are what Python uses to implement generator iterators. They
are normally created by iterating over a function that yields values, rather
than explicitly calling @ref{44a2,,PyGen_New()} or @ref{44a3,,PyGen_NewWithQualName()}.

@geindex PyGenObject (C type)
@anchor{c-api/gen c PyGenObject}@anchor{44a4}
@deffn {C Type} type  PyGenObject

The C structure used for generator objects.
@end deffn

@geindex PyGen_Type (C var)
@anchor{c-api/gen c PyGen_Type}@anchor{44a5}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyGen_Type

The type object corresponding to generator objects.
@end deffn

@geindex PyGen_Check (C function)
@anchor{c-api/gen c PyGen_Check}@anchor{44a6}
@deffn {C Function} int  PyGen_Check (PyObject  *ob)

Return true if @emph{ob} is a generator object; @emph{ob} must not be @code{NULL}.  This
function always succeeds.
@end deffn

@geindex PyGen_CheckExact (C function)
@anchor{c-api/gen c PyGen_CheckExact}@anchor{44a7}
@deffn {C Function} int  PyGen_CheckExact (PyObject  *ob)

Return true if @emph{ob}’s type is @ref{44a5,,PyGen_Type}; @emph{ob} must not be
@code{NULL}.  This function always succeeds.
@end deffn

@geindex PyGen_New (C function)
@anchor{c-api/gen c PyGen_New}@anchor{44a2}
@deffn {C Function} @ref{375,,PyObject}  *PyGen_New (PyFrameObject  *frame)
@emph{Return value: New reference.}
Create and return a new generator object based on the @emph{frame} object.
A reference to @emph{frame} is stolen by this function. The argument must not be
@code{NULL}.
@end deffn

@geindex PyGen_NewWithQualName (C function)
@anchor{c-api/gen c PyGen_NewWithQualName}@anchor{44a3}
@deffn {C Function} @ref{375,,PyObject}  *PyGen_NewWithQualName (PyFrameObject  *frame, PyObject  *name, PyObject  *qualname)
@emph{Return value: New reference.}
Create and return a new generator object based on the @emph{frame} object,
with @code{__name__} and @code{__qualname__} set to @emph{name} and @emph{qualname}.
A reference to @emph{frame} is stolen by this function.  The @emph{frame} argument
must not be @code{NULL}.
@end deffn

@node Coroutine Objects<2>,Context Variables Objects,Generator Objects,Other Objects
@anchor{c-api/coro doc}@anchor{44a8}@anchor{c-api/coro coro-objects}@anchor{a42}@anchor{c-api/coro coroutine-objects}@anchor{44a9}
@subsubsection Coroutine Objects



New in version 3.5.


Coroutine objects are what functions declared with an @code{async} keyword
return.

@geindex PyCoroObject (C type)
@anchor{c-api/coro c PyCoroObject}@anchor{44aa}
@deffn {C Type} type  PyCoroObject

The C structure used for coroutine objects.
@end deffn

@geindex PyCoro_Type (C var)
@anchor{c-api/coro c PyCoro_Type}@anchor{44ab}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyCoro_Type

The type object corresponding to coroutine objects.
@end deffn

@geindex PyCoro_CheckExact (C function)
@anchor{c-api/coro c PyCoro_CheckExact}@anchor{44ac}
@deffn {C Function} int  PyCoro_CheckExact (PyObject  *ob)

Return true if @emph{ob}’s type is @ref{44ab,,PyCoro_Type}; @emph{ob} must not be @code{NULL}.
This function always succeeds.
@end deffn

@geindex PyCoro_New (C function)
@anchor{c-api/coro c PyCoro_New}@anchor{44ad}
@deffn {C Function} @ref{375,,PyObject}  *PyCoro_New (PyFrameObject  *frame, PyObject  *name, PyObject  *qualname)
@emph{Return value: New reference.}
Create and return a new coroutine object based on the @emph{frame} object,
with @code{__name__} and @code{__qualname__} set to @emph{name} and @emph{qualname}.
A reference to @emph{frame} is stolen by this function.  The @emph{frame} argument
must not be @code{NULL}.
@end deffn

@node Context Variables Objects,DateTime Objects<2>,Coroutine Objects<2>,Other Objects
@anchor{c-api/contextvars doc}@anchor{44ae}@anchor{c-api/contextvars context-variables-objects}@anchor{44af}@anchor{c-api/contextvars contextvarsobjects}@anchor{615}
@subsubsection Context Variables Objects

@anchor{c-api/contextvars contextvarsobjects-pointertype-change}@anchor{765}

Changed in version 3.7.1: 

@cartouche
@quotation Note 
In Python 3.7.1 the signatures of all context variables
C APIs were @strong{changed} to use @ref{375,,PyObject} pointers instead
of @ref{44b0,,PyContext}, @ref{44b1,,PyContextVar}, and
@ref{44b2,,PyContextToken}, e.g.:

@example
// in 3.7.0:
PyContext *PyContext_New(void);

// in 3.7.1+:
PyObject *PyContext_New(void);
@end example

See bpo-34762@footnote{https://bugs.python.org/issue34762} for more details.
@end quotation
@end cartouche



New in version 3.7.


This section details the public C API for the @ref{23,,contextvars} module.

@geindex PyContext (C type)
@anchor{c-api/contextvars c PyContext}@anchor{44b0}
@deffn {C Type} type  PyContext

The C structure used to represent a @ref{2943,,contextvars.Context}
object.
@end deffn

@geindex PyContextVar (C type)
@anchor{c-api/contextvars c PyContextVar}@anchor{44b1}
@deffn {C Type} type  PyContextVar

The C structure used to represent a @ref{2941,,contextvars.ContextVar}
object.
@end deffn

@geindex PyContextToken (C type)
@anchor{c-api/contextvars c PyContextToken}@anchor{44b2}
@deffn {C Type} type  PyContextToken

The C structure used to represent a @ref{2948,,contextvars.Token} object.
@end deffn

@geindex PyContext_Type (C var)
@anchor{c-api/contextvars c PyContext_Type}@anchor{44b3}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyContext_Type

The type object representing the @emph{context} type.
@end deffn

@geindex PyContextVar_Type (C var)
@anchor{c-api/contextvars c PyContextVar_Type}@anchor{44b4}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyContextVar_Type

The type object representing the @emph{context variable} type.
@end deffn

@geindex PyContextToken_Type (C var)
@anchor{c-api/contextvars c PyContextToken_Type}@anchor{44b5}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  PyContextToken_Type

The type object representing the @emph{context variable token} type.
@end deffn

Type-check macros:

@geindex PyContext_CheckExact (C function)
@anchor{c-api/contextvars c PyContext_CheckExact}@anchor{44b6}
@deffn {C Function} int  PyContext_CheckExact (PyObject  *o)

Return true if @emph{o} is of type @ref{44b3,,PyContext_Type}. @emph{o} must not be
@code{NULL}.  This function always succeeds.
@end deffn

@geindex PyContextVar_CheckExact (C function)
@anchor{c-api/contextvars c PyContextVar_CheckExact}@anchor{44b7}
@deffn {C Function} int  PyContextVar_CheckExact (PyObject  *o)

Return true if @emph{o} is of type @ref{44b4,,PyContextVar_Type}. @emph{o} must not be
@code{NULL}.  This function always succeeds.
@end deffn

@geindex PyContextToken_CheckExact (C function)
@anchor{c-api/contextvars c PyContextToken_CheckExact}@anchor{44b8}
@deffn {C Function} int  PyContextToken_CheckExact (PyObject  *o)

Return true if @emph{o} is of type @ref{44b5,,PyContextToken_Type}.
@emph{o} must not be @code{NULL}.  This function always succeeds.
@end deffn

Context object management functions:

@geindex PyContext_New (C function)
@anchor{c-api/contextvars c PyContext_New}@anchor{44b9}
@deffn {C Function} @ref{375,,PyObject}  *PyContext_New (void)
@emph{Return value: New reference.}
Create a new empty context object.  Returns @code{NULL} if an error
has occurred.
@end deffn

@geindex PyContext_Copy (C function)
@anchor{c-api/contextvars c PyContext_Copy}@anchor{44ba}
@deffn {C Function} @ref{375,,PyObject}  *PyContext_Copy (PyObject  *ctx)
@emph{Return value: New reference.}
Create a shallow copy of the passed @emph{ctx} context object.
Returns @code{NULL} if an error has occurred.
@end deffn

@geindex PyContext_CopyCurrent (C function)
@anchor{c-api/contextvars c PyContext_CopyCurrent}@anchor{44bb}
@deffn {C Function} @ref{375,,PyObject}  *PyContext_CopyCurrent (void)
@emph{Return value: New reference.}
Create a shallow copy of the current thread context.
Returns @code{NULL} if an error has occurred.
@end deffn

@geindex PyContext_Enter (C function)
@anchor{c-api/contextvars c PyContext_Enter}@anchor{44bc}
@deffn {C Function} int  PyContext_Enter (PyObject  *ctx)

Set @emph{ctx} as the current context for the current thread.
Returns @code{0} on success, and @code{-1} on error.
@end deffn

@geindex PyContext_Exit (C function)
@anchor{c-api/contextvars c PyContext_Exit}@anchor{44bd}
@deffn {C Function} int  PyContext_Exit (PyObject  *ctx)

Deactivate the @emph{ctx} context and restore the previous context as the
current context for the current thread.  Returns @code{0} on success,
and @code{-1} on error.
@end deffn

Context variable functions:

@geindex PyContextVar_New (C function)
@anchor{c-api/contextvars c PyContextVar_New}@anchor{44be}
@deffn {C Function} @ref{375,,PyObject}  *PyContextVar_New (const  char  *name, PyObject  *def)
@emph{Return value: New reference.}
Create a new @code{ContextVar} object.  The @emph{name} parameter is used
for introspection and debug purposes.  The @emph{def} parameter specifies
a default value for the context variable, or @code{NULL} for no default.
If an error has occurred, this function returns @code{NULL}.
@end deffn

@geindex PyContextVar_Get (C function)
@anchor{c-api/contextvars c PyContextVar_Get}@anchor{44bf}
@deffn {C Function} int  PyContextVar_Get (PyObject  *var, PyObject  *default_value, PyObject  **value)

Get the value of a context variable.  Returns @code{-1} if an error has
occurred during lookup, and @code{0} if no error occurred, whether or not
a value was found.

If the context variable was found, @emph{value} will be a pointer to it.
If the context variable was @emph{not} found, @emph{value} will point to:


@itemize -

@item 
@emph{default_value}, if not @code{NULL};

@item 
the default value of @emph{var}, if not @code{NULL};

@item 
@code{NULL}
@end itemize

Except for @code{NULL}, the function returns a new reference.
@end deffn

@geindex PyContextVar_Set (C function)
@anchor{c-api/contextvars c PyContextVar_Set}@anchor{44c0}
@deffn {C Function} @ref{375,,PyObject}  *PyContextVar_Set (PyObject  *var, PyObject  *value)
@emph{Return value: New reference.}
Set the value of @emph{var} to @emph{value} in the current context.  Returns
a new token object for this change, or @code{NULL} if an error has occurred.
@end deffn

@geindex PyContextVar_Reset (C function)
@anchor{c-api/contextvars c PyContextVar_Reset}@anchor{44c1}
@deffn {C Function} int  PyContextVar_Reset (PyObject  *var, PyObject  *token)

Reset the state of the @emph{var} context variable to that it was in before
@ref{44c0,,PyContextVar_Set()} that returned the @emph{token} was called.
This function returns @code{0} on success and @code{-1} on error.
@end deffn

@node DateTime Objects<2>,Objects for Type Hinting,Context Variables Objects,Other Objects
@anchor{c-api/datetime doc}@anchor{44c2}@anchor{c-api/datetime datetime-objects}@anchor{44c3}@anchor{c-api/datetime datetimeobjects}@anchor{44c4}
@subsubsection DateTime Objects


Various date and time objects are supplied by the @ref{2f,,datetime} module.
Before using any of these functions, the header file @code{datetime.h} must be
included in your source (note that this is not included by @code{Python.h}),
and the macro @code{PyDateTime_IMPORT} must be invoked, usually as part of
the module initialisation function.  The macro puts a pointer to a C structure
into a static variable, @code{PyDateTimeAPI}, that is used by the following
macros.

Macro for access to the UTC singleton:

@geindex PyDateTime_TimeZone_UTC (C var)
@anchor{c-api/datetime c PyDateTime_TimeZone_UTC}@anchor{6fa}
@deffn {C Variable} @ref{375,,PyObject}  *PyDateTime_TimeZone_UTC

Returns the time zone singleton representing UTC, the same object as
@ref{1d67,,datetime.timezone.utc}.


New in version 3.7.

@end deffn

Type-check macros:

@geindex PyDate_Check (C function)
@anchor{c-api/datetime c PyDate_Check}@anchor{44c5}
@deffn {C Function} int  PyDate_Check (PyObject  *ob)

Return true if @emph{ob} is of type @code{PyDateTime_DateType} or a subtype of
@code{PyDateTime_DateType}.  @emph{ob} must not be @code{NULL}.  This function always
succeeds.
@end deffn

@geindex PyDate_CheckExact (C function)
@anchor{c-api/datetime c PyDate_CheckExact}@anchor{44c6}
@deffn {C Function} int  PyDate_CheckExact (PyObject  *ob)

Return true if @emph{ob} is of type @code{PyDateTime_DateType}. @emph{ob} must not be
@code{NULL}.  This function always succeeds.
@end deffn

@geindex PyDateTime_Check (C function)
@anchor{c-api/datetime c PyDateTime_Check}@anchor{44c7}
@deffn {C Function} int  PyDateTime_Check (PyObject  *ob)

Return true if @emph{ob} is of type @code{PyDateTime_DateTimeType} or a subtype of
@code{PyDateTime_DateTimeType}.  @emph{ob} must not be @code{NULL}.  This function always
succeeds.
@end deffn

@geindex PyDateTime_CheckExact (C function)
@anchor{c-api/datetime c PyDateTime_CheckExact}@anchor{44c8}
@deffn {C Function} int  PyDateTime_CheckExact (PyObject  *ob)

Return true if @emph{ob} is of type @code{PyDateTime_DateTimeType}. @emph{ob} must not
be @code{NULL}.  This function always succeeds.
@end deffn

@geindex PyTime_Check (C function)
@anchor{c-api/datetime c PyTime_Check}@anchor{44c9}
@deffn {C Function} int  PyTime_Check (PyObject  *ob)

Return true if @emph{ob} is of type @code{PyDateTime_TimeType} or a subtype of
@code{PyDateTime_TimeType}.  @emph{ob} must not be @code{NULL}.  This function always
succeeds.
@end deffn

@geindex PyTime_CheckExact (C function)
@anchor{c-api/datetime c PyTime_CheckExact}@anchor{44ca}
@deffn {C Function} int  PyTime_CheckExact (PyObject  *ob)

Return true if @emph{ob} is of type @code{PyDateTime_TimeType}. @emph{ob} must not be
@code{NULL}.  This function always succeeds.
@end deffn

@geindex PyDelta_Check (C function)
@anchor{c-api/datetime c PyDelta_Check}@anchor{44cb}
@deffn {C Function} int  PyDelta_Check (PyObject  *ob)

Return true if @emph{ob} is of type @code{PyDateTime_DeltaType} or a subtype of
@code{PyDateTime_DeltaType}.  @emph{ob} must not be @code{NULL}.  This function always
succeeds.
@end deffn

@geindex PyDelta_CheckExact (C function)
@anchor{c-api/datetime c PyDelta_CheckExact}@anchor{44cc}
@deffn {C Function} int  PyDelta_CheckExact (PyObject  *ob)

Return true if @emph{ob} is of type @code{PyDateTime_DeltaType}. @emph{ob} must not be
@code{NULL}.  This function always succeeds.
@end deffn

@geindex PyTZInfo_Check (C function)
@anchor{c-api/datetime c PyTZInfo_Check}@anchor{44cd}
@deffn {C Function} int  PyTZInfo_Check (PyObject  *ob)

Return true if @emph{ob} is of type @code{PyDateTime_TZInfoType} or a subtype of
@code{PyDateTime_TZInfoType}.  @emph{ob} must not be @code{NULL}.  This function always
succeeds.
@end deffn

@geindex PyTZInfo_CheckExact (C function)
@anchor{c-api/datetime c PyTZInfo_CheckExact}@anchor{44ce}
@deffn {C Function} int  PyTZInfo_CheckExact (PyObject  *ob)

Return true if @emph{ob} is of type @code{PyDateTime_TZInfoType}. @emph{ob} must not be
@code{NULL}.  This function always succeeds.
@end deffn

Macros to create objects:

@geindex PyDate_FromDate (C function)
@anchor{c-api/datetime c PyDate_FromDate}@anchor{44cf}
@deffn {C Function} @ref{375,,PyObject}  *PyDate_FromDate (int  year, int  month, int  day)
@emph{Return value: New reference.}
Return a @ref{3e2,,datetime.date} object with the specified year, month and day.
@end deffn

@geindex PyDateTime_FromDateAndTime (C function)
@anchor{c-api/datetime c PyDateTime_FromDateAndTime}@anchor{44d0}
@deffn {C Function} @ref{375,,PyObject}  *PyDateTime_FromDateAndTime (int  year, int  month, int  day, int  hour, int  minute, int  second, int  usecond)
@emph{Return value: New reference.}
Return a @ref{363,,datetime.datetime} object with the specified year, month, day, hour,
minute, second and microsecond.
@end deffn

@geindex PyDateTime_FromDateAndTimeAndFold (C function)
@anchor{c-api/datetime c PyDateTime_FromDateAndTimeAndFold}@anchor{44d1}
@deffn {C Function} @ref{375,,PyObject}  *PyDateTime_FromDateAndTimeAndFold (int  year, int  month, int  day, int  hour, int  minute, int  second, int  usecond, int  fold)
@emph{Return value: New reference.}
Return a @ref{363,,datetime.datetime} object with the specified year, month, day, hour,
minute, second, microsecond and fold.


New in version 3.6.

@end deffn

@geindex PyTime_FromTime (C function)
@anchor{c-api/datetime c PyTime_FromTime}@anchor{44d2}
@deffn {C Function} @ref{375,,PyObject}  *PyTime_FromTime (int  hour, int  minute, int  second, int  usecond)
@emph{Return value: New reference.}
Return a @ref{364,,datetime.time} object with the specified hour, minute, second and
microsecond.
@end deffn

@geindex PyTime_FromTimeAndFold (C function)
@anchor{c-api/datetime c PyTime_FromTimeAndFold}@anchor{44d3}
@deffn {C Function} @ref{375,,PyObject}  *PyTime_FromTimeAndFold (int  hour, int  minute, int  second, int  usecond, int  fold)
@emph{Return value: New reference.}
Return a @ref{364,,datetime.time} object with the specified hour, minute, second,
microsecond and fold.


New in version 3.6.

@end deffn

@geindex PyDelta_FromDSU (C function)
@anchor{c-api/datetime c PyDelta_FromDSU}@anchor{44d4}
@deffn {C Function} @ref{375,,PyObject}  *PyDelta_FromDSU (int  days, int  seconds, int  useconds)
@emph{Return value: New reference.}
Return a @ref{4c8,,datetime.timedelta} object representing the given number
of days, seconds and microseconds.  Normalization is performed so that the
resulting number of microseconds and seconds lie in the ranges documented for
@ref{4c8,,datetime.timedelta} objects.
@end deffn

@geindex PyTimeZone_FromOffset (C function)
@anchor{c-api/datetime c PyTimeZone_FromOffset}@anchor{6f8}
@deffn {C Function} @ref{375,,PyObject}  *PyTimeZone_FromOffset (PyDateTime_DeltaType  *offset)
@emph{Return value: New reference.}
Return a @ref{c51,,datetime.timezone} object with an unnamed fixed offset
represented by the @emph{offset} argument.


New in version 3.7.

@end deffn

@geindex PyTimeZone_FromOffsetAndName (C function)
@anchor{c-api/datetime c PyTimeZone_FromOffsetAndName}@anchor{6f9}
@deffn {C Function} @ref{375,,PyObject}  *PyTimeZone_FromOffsetAndName (PyDateTime_DeltaType  *offset, PyUnicode  *name)
@emph{Return value: New reference.}
Return a @ref{c51,,datetime.timezone} object with a fixed offset represented
by the @emph{offset} argument and with tzname @emph{name}.


New in version 3.7.

@end deffn

Macros to extract fields from date objects.  The argument must be an instance of
@code{PyDateTime_Date}, including subclasses (such as
@code{PyDateTime_DateTime}).  The argument must not be @code{NULL}, and the type is
not checked:

@geindex PyDateTime_GET_YEAR (C function)
@anchor{c-api/datetime c PyDateTime_GET_YEAR}@anchor{44d5}
@deffn {C Function} int  PyDateTime_GET_YEAR (PyDateTime_Date  *o)

Return the year, as a positive int.
@end deffn

@geindex PyDateTime_GET_MONTH (C function)
@anchor{c-api/datetime c PyDateTime_GET_MONTH}@anchor{44d6}
@deffn {C Function} int  PyDateTime_GET_MONTH (PyDateTime_Date  *o)

Return the month, as an int from 1 through 12.
@end deffn

@geindex PyDateTime_GET_DAY (C function)
@anchor{c-api/datetime c PyDateTime_GET_DAY}@anchor{44d7}
@deffn {C Function} int  PyDateTime_GET_DAY (PyDateTime_Date  *o)

Return the day, as an int from 1 through 31.
@end deffn

Macros to extract fields from datetime objects.  The argument must be an
instance of @code{PyDateTime_DateTime}, including subclasses. The argument
must not be @code{NULL}, and the type is not checked:

@geindex PyDateTime_DATE_GET_HOUR (C function)
@anchor{c-api/datetime c PyDateTime_DATE_GET_HOUR}@anchor{44d8}
@deffn {C Function} int  PyDateTime_DATE_GET_HOUR (PyDateTime_DateTime  *o)

Return the hour, as an int from 0 through 23.
@end deffn

@geindex PyDateTime_DATE_GET_MINUTE (C function)
@anchor{c-api/datetime c PyDateTime_DATE_GET_MINUTE}@anchor{44d9}
@deffn {C Function} int  PyDateTime_DATE_GET_MINUTE (PyDateTime_DateTime  *o)

Return the minute, as an int from 0 through 59.
@end deffn

@geindex PyDateTime_DATE_GET_SECOND (C function)
@anchor{c-api/datetime c PyDateTime_DATE_GET_SECOND}@anchor{44da}
@deffn {C Function} int  PyDateTime_DATE_GET_SECOND (PyDateTime_DateTime  *o)

Return the second, as an int from 0 through 59.
@end deffn

@geindex PyDateTime_DATE_GET_MICROSECOND (C function)
@anchor{c-api/datetime c PyDateTime_DATE_GET_MICROSECOND}@anchor{44db}
@deffn {C Function} int  PyDateTime_DATE_GET_MICROSECOND (PyDateTime_DateTime  *o)

Return the microsecond, as an int from 0 through 999999.
@end deffn

@geindex PyDateTime_DATE_GET_TZINFO (C function)
@anchor{c-api/datetime c PyDateTime_DATE_GET_TZINFO}@anchor{361}
@deffn {C Function} @ref{375,,PyObject}  *PyDateTime_DATE_GET_TZINFO (PyDateTime_DateTime  *o)

Return the tzinfo (which may be @code{None}).


New in version 3.10.

@end deffn

Macros to extract fields from time objects.  The argument must be an instance of
@code{PyDateTime_Time}, including subclasses. The argument must not be @code{NULL},
and the type is not checked:

@geindex PyDateTime_TIME_GET_HOUR (C function)
@anchor{c-api/datetime c PyDateTime_TIME_GET_HOUR}@anchor{44dc}
@deffn {C Function} int  PyDateTime_TIME_GET_HOUR (PyDateTime_Time  *o)

Return the hour, as an int from 0 through 23.
@end deffn

@geindex PyDateTime_TIME_GET_MINUTE (C function)
@anchor{c-api/datetime c PyDateTime_TIME_GET_MINUTE}@anchor{44dd}
@deffn {C Function} int  PyDateTime_TIME_GET_MINUTE (PyDateTime_Time  *o)

Return the minute, as an int from 0 through 59.
@end deffn

@geindex PyDateTime_TIME_GET_SECOND (C function)
@anchor{c-api/datetime c PyDateTime_TIME_GET_SECOND}@anchor{44de}
@deffn {C Function} int  PyDateTime_TIME_GET_SECOND (PyDateTime_Time  *o)

Return the second, as an int from 0 through 59.
@end deffn

@geindex PyDateTime_TIME_GET_MICROSECOND (C function)
@anchor{c-api/datetime c PyDateTime_TIME_GET_MICROSECOND}@anchor{44df}
@deffn {C Function} int  PyDateTime_TIME_GET_MICROSECOND (PyDateTime_Time  *o)

Return the microsecond, as an int from 0 through 999999.
@end deffn

@geindex PyDateTime_TIME_GET_TZINFO (C function)
@anchor{c-api/datetime c PyDateTime_TIME_GET_TZINFO}@anchor{362}
@deffn {C Function} @ref{375,,PyObject}  *PyDateTime_TIME_GET_TZINFO (PyDateTime_Time  *o)

Return the tzinfo (which may be @code{None}).


New in version 3.10.

@end deffn

Macros to extract fields from time delta objects.  The argument must be an
instance of @code{PyDateTime_Delta}, including subclasses. The argument must
not be @code{NULL}, and the type is not checked:

@geindex PyDateTime_DELTA_GET_DAYS (C function)
@anchor{c-api/datetime c PyDateTime_DELTA_GET_DAYS}@anchor{44e0}
@deffn {C Function} int  PyDateTime_DELTA_GET_DAYS (PyDateTime_Delta  *o)

Return the number of days, as an int from -999999999 to 999999999.


New in version 3.3.

@end deffn

@geindex PyDateTime_DELTA_GET_SECONDS (C function)
@anchor{c-api/datetime c PyDateTime_DELTA_GET_SECONDS}@anchor{44e1}
@deffn {C Function} int  PyDateTime_DELTA_GET_SECONDS (PyDateTime_Delta  *o)

Return the number of seconds, as an int from 0 through 86399.


New in version 3.3.

@end deffn

@geindex PyDateTime_DELTA_GET_MICROSECONDS (C function)
@anchor{c-api/datetime c PyDateTime_DELTA_GET_MICROSECONDS}@anchor{44e2}
@deffn {C Function} int  PyDateTime_DELTA_GET_MICROSECONDS (PyDateTime_Delta  *o)

Return the number of microseconds, as an int from 0 through 999999.


New in version 3.3.

@end deffn

Macros for the convenience of modules implementing the DB API:

@geindex PyDateTime_FromTimestamp (C function)
@anchor{c-api/datetime c PyDateTime_FromTimestamp}@anchor{44e3}
@deffn {C Function} @ref{375,,PyObject}  *PyDateTime_FromTimestamp (PyObject  *args)
@emph{Return value: New reference.}
Create and return a new @ref{363,,datetime.datetime} object given an argument
tuple suitable for passing to @ref{134e,,datetime.datetime.fromtimestamp()}.
@end deffn

@geindex PyDate_FromTimestamp (C function)
@anchor{c-api/datetime c PyDate_FromTimestamp}@anchor{12b9}
@deffn {C Function} @ref{375,,PyObject}  *PyDate_FromTimestamp (PyObject  *args)
@emph{Return value: New reference.}
Create and return a new @ref{3e2,,datetime.date} object given an argument
tuple suitable for passing to @ref{1d28,,datetime.date.fromtimestamp()}.
@end deffn

@node Objects for Type Hinting,,DateTime Objects<2>,Other Objects
@anchor{c-api/typehints doc}@anchor{44e4}@anchor{c-api/typehints objects-for-type-hinting}@anchor{44e5}@anchor{c-api/typehints typehintobjects}@anchor{44e6}
@subsubsection Objects for Type Hinting


Various built-in types for type hinting are provided.  Currently,
two types exist – @ref{268,,GenericAlias} and
@ref{238,,Union}.  Only @code{GenericAlias} is exposed to C.

@geindex Py_GenericAlias (C function)
@anchor{c-api/typehints c Py_GenericAlias}@anchor{4299}
@deffn {C Function} @ref{375,,PyObject}  *Py_GenericAlias (PyObject  *origin, PyObject  *args)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.9.}
Create a @ref{268,,GenericAlias} object.
Equivalent to calling the Python class
@ref{26a,,types.GenericAlias}.  The @emph{origin} and @emph{args} arguments set the
@code{GenericAlias}‘s @code{__origin__} and @code{__args__} attributes respectively.
@emph{origin} should be a @ref{5c6,,PyTypeObject}*, and @emph{args} can be a
@ref{4302,,PyTupleObject}* or any @code{PyObject*}.  If @emph{args} passed is
not a tuple, a 1-tuple is automatically constructed and @code{__args__} is set
to @code{(args,)}.
Minimal checking is done for the arguments, so the function will succeed even
if @emph{origin} is not a type.
The @code{GenericAlias}‘s @code{__parameters__} attribute is constructed lazily
from @code{__args__}.  On failure, an exception is raised and @code{NULL} is
returned.

Here’s an example of how to make an extension type generic:

@example
...
static PyMethodDef my_obj_methods[] = @{
    // Other methods.
    ...
    @{"__class_getitem__", Py_GenericAlias, METH_O|METH_CLASS, "See PEP 585"@}
    ...
@}
@end example


@subsubheading See also


The data model method @code{__class_getitem__()}.



New in version 3.9.

@end deffn

@geindex Py_GenericAliasType (C var)
@anchor{c-api/typehints c Py_GenericAliasType}@anchor{429a}
@deffn {C Variable} @ref{5c6,,PyTypeObject}  Py_GenericAliasType
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.9.}
The C type of the object returned by @ref{4299,,Py_GenericAlias()}. Equivalent to
@ref{26a,,types.GenericAlias} in Python.


New in version 3.9.

@end deffn

@node Initialization Finalization and Threads,Python Initialization Configuration,Concrete Objects Layer,Python/C API Reference Manual
@anchor{c-api/init doc}@anchor{44e7}@anchor{c-api/init initialization}@anchor{44e8}@anchor{c-api/init initialization-finalization-and-threads}@anchor{44e9}
@section Initialization, Finalization, and Threads


See also @ref{207,,Python Initialization Configuration}.

@menu
* Before Python Initialization:: 
* Global configuration variables:: 
* Initializing and finalizing the interpreter:: 
* Process-wide parameters:: 
* Thread State and the Global Interpreter Lock:: 
* Sub-interpreter support:: 
* Asynchronous Notifications:: 
* Profiling and Tracing:: 
* Advanced Debugger Support:: 
* Thread Local Storage Support:: 

@end menu

@node Before Python Initialization,Global configuration variables,,Initialization Finalization and Threads
@anchor{c-api/init before-python-initialization}@anchor{44ea}@anchor{c-api/init pre-init-safe}@anchor{6fc}
@subsection Before Python Initialization


In an application embedding  Python, the @ref{38d,,Py_Initialize()} function must
be called before using any other Python/C API functions; with the exception of
a few functions and the @ref{44eb,,global configuration variables}.

The following functions can be safely called before Python is initialized:


@itemize *

@item 
Configuration functions:


@itemize *

@item 
@ref{112b,,PyImport_AppendInittab()}

@item 
@ref{112c,,PyImport_ExtendInittab()}

@item 
@code{PyInitFrozenExtensions()}

@item 
@ref{44ec,,PyMem_SetAllocator()}

@item 
@ref{7b0,,PyMem_SetupDebugHooks()}

@item 
@ref{44ed,,PyObject_SetArenaAllocator()}

@item 
@ref{202,,Py_SetPath()}

@item 
@ref{203,,Py_SetProgramName()}

@item 
@ref{204,,Py_SetPythonHome()}

@item 
@ref{205,,Py_SetStandardStreamEncoding()}

@item 
@ref{200,,PySys_AddWarnOption()}

@item 
@ref{201,,PySys_AddXOption()}

@item 
@ref{4236,,PySys_ResetWarnOptions()}
@end itemize

@item 
Informative functions:


@itemize *

@item 
@ref{411d,,Py_IsInitialized()}

@item 
@ref{44ee,,PyMem_GetAllocator()}

@item 
@ref{44ef,,PyObject_GetArenaAllocator()}

@item 
@ref{f94,,Py_GetBuildInfo()}

@item 
@ref{429b,,Py_GetCompiler()}

@item 
@ref{429c,,Py_GetCopyright()}

@item 
@ref{429d,,Py_GetPlatform()}

@item 
@ref{429e,,Py_GetVersion()}
@end itemize

@item 
Utilities:


@itemize *

@item 
@ref{6fe,,Py_DecodeLocale()}
@end itemize

@item 
Memory allocators:


@itemize *

@item 
@ref{bc4,,PyMem_RawMalloc()}

@item 
@ref{bc5,,PyMem_RawRealloc()}

@item 
@ref{a1e,,PyMem_RawCalloc()}

@item 
@ref{42fc,,PyMem_RawFree()}
@end itemize
@end itemize

@cartouche
@quotation Note 
The following functions @strong{should not be called} before
@ref{38d,,Py_Initialize()}: @ref{6ff,,Py_EncodeLocale()}, @ref{387,,Py_GetPath()},
@ref{388,,Py_GetPrefix()}, @ref{389,,Py_GetExecPrefix()},
@ref{38a,,Py_GetProgramFullPath()}, @ref{38b,,Py_GetPythonHome()},
@ref{38c,,Py_GetProgramName()} and @ref{441,,PyEval_InitThreads()}.
@end quotation
@end cartouche

@node Global configuration variables,Initializing and finalizing the interpreter,Before Python Initialization,Initialization Finalization and Threads
@anchor{c-api/init global-conf-vars}@anchor{44eb}@anchor{c-api/init global-configuration-variables}@anchor{44f0}
@subsection Global configuration variables


Python has variables for the global configuration to control different features
and options. By default, these flags are controlled by @ref{15fd,,command line options}.

When a flag is set by an option, the value of the flag is the number of times
that the option was set. For example, @code{-b} sets @ref{760,,Py_BytesWarningFlag}
to 1 and @code{-bb} sets @ref{760,,Py_BytesWarningFlag} to 2.

@geindex Py_BytesWarningFlag (C var)
@anchor{c-api/init c Py_BytesWarningFlag}@anchor{760}
@deffn {C Variable} int  Py_BytesWarningFlag

Issue a warning when comparing @ref{2f0,,bytes} or @ref{2f1,,bytearray} with
@ref{2ef,,str} or @ref{2f0,,bytes} with @ref{1b1,,int}.  Issue an error if greater
or equal to @code{2}.

Set by the @ref{6dc,,-b} option.
@end deffn

@geindex Py_DebugFlag (C var)
@anchor{c-api/init c Py_DebugFlag}@anchor{44f1}
@deffn {C Variable} int  Py_DebugFlag

Turn on parser debugging output (for expert only, depending on compilation
options).

Set by the @ref{1607,,-d} option and the 
@geindex PYTHONDEBUG
@geindex environment variable; PYTHONDEBUG
@ref{1608,,PYTHONDEBUG} environment
variable.
@end deffn

@geindex Py_DontWriteBytecodeFlag (C var)
@anchor{c-api/init c Py_DontWriteBytecodeFlag}@anchor{44f2}
@deffn {C Variable} int  Py_DontWriteBytecodeFlag

If set to non-zero, Python won’t try to write @code{.pyc} files on the
import of source modules.

Set by the @ref{f34,,-B} option and the 
@geindex PYTHONDONTWRITEBYTECODE
@geindex environment variable; PYTHONDONTWRITEBYTECODE
@ref{f35,,PYTHONDONTWRITEBYTECODE}
environment variable.
@end deffn

@geindex Py_FrozenFlag (C var)
@anchor{c-api/init c Py_FrozenFlag}@anchor{44f3}
@deffn {C Variable} int  Py_FrozenFlag

Suppress error messages when calculating the module search path in
@ref{387,,Py_GetPath()}.

Private flag used by @code{_freeze_module} and @code{frozenmain} programs.
@end deffn

@geindex Py_HashRandomizationFlag (C var)
@anchor{c-api/init c Py_HashRandomizationFlag}@anchor{44f4}
@deffn {C Variable} int  Py_HashRandomizationFlag

Set to @code{1} if the 
@geindex PYTHONHASHSEED
@geindex environment variable; PYTHONHASHSEED
@ref{c15,,PYTHONHASHSEED} environment variable is set to
a non-empty string.

If the flag is non-zero, read the 
@geindex PYTHONHASHSEED
@geindex environment variable; PYTHONHASHSEED
@ref{c15,,PYTHONHASHSEED} environment
variable to initialize the secret hash seed.
@end deffn

@geindex Py_IgnoreEnvironmentFlag (C var)
@anchor{c-api/init c Py_IgnoreEnvironmentFlag}@anchor{764}
@deffn {C Variable} int  Py_IgnoreEnvironmentFlag

Ignore all 
@geindex PYTHON*
@geindex environment variable; PYTHON*
@code{PYTHON*} environment variables, e.g.
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} and 
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME}, that might be set.

Set by the @ref{460,,-E} and @ref{461,,-I} options.
@end deffn

@geindex Py_InspectFlag (C var)
@anchor{c-api/init c Py_InspectFlag}@anchor{44f5}
@deffn {C Variable} int  Py_InspectFlag

When a script is passed as first argument or the @ref{1180,,-c} option is used,
enter interactive mode after executing the script or the command, even when
@ref{5ee,,sys.stdin} does not appear to be a terminal.

Set by the @ref{1018,,-i} option and the 
@geindex PYTHONINSPECT
@geindex environment variable; PYTHONINSPECT
@ref{1017,,PYTHONINSPECT} environment
variable.
@end deffn

@geindex Py_InteractiveFlag (C var)
@anchor{c-api/init c Py_InteractiveFlag}@anchor{42f9}
@deffn {C Variable} int  Py_InteractiveFlag

Set by the @ref{1018,,-i} option.
@end deffn

@geindex Py_IsolatedFlag (C var)
@anchor{c-api/init c Py_IsolatedFlag}@anchor{44f6}
@deffn {C Variable} int  Py_IsolatedFlag

Run Python in isolated mode. In isolated mode @ref{489,,sys.path} contains
neither the script’s directory nor the user’s site-packages directory.

Set by the @ref{461,,-I} option.


New in version 3.4.

@end deffn

@geindex Py_LegacyWindowsFSEncodingFlag (C var)
@anchor{c-api/init c Py_LegacyWindowsFSEncodingFlag}@anchor{42fd}
@deffn {C Variable} int  Py_LegacyWindowsFSEncodingFlag

If the flag is non-zero, use the @code{mbcs} encoding with @code{replace} error
handler, instead of the UTF-8 encoding with @code{surrogatepass} error handler,
for the @ref{1628,,filesystem encoding and error handler}.

Set to @code{1} if the 
@geindex PYTHONLEGACYWINDOWSFSENCODING
@geindex environment variable; PYTHONLEGACYWINDOWSFSENCODING
@ref{79f,,PYTHONLEGACYWINDOWSFSENCODING} environment
variable is set to a non-empty string.

See 
@geindex Python Enhancement Proposals; PEP 529
PEP 529@footnote{https://www.python.org/dev/peps/pep-0529} for more details.

@ref{1629,,Availability}: Windows.
@end deffn

@geindex Py_LegacyWindowsStdioFlag (C var)
@anchor{c-api/init c Py_LegacyWindowsStdioFlag}@anchor{44f7}
@deffn {C Variable} int  Py_LegacyWindowsStdioFlag

If the flag is non-zero, use @ref{ea6,,io.FileIO} instead of
@code{WindowsConsoleIO} for @ref{f7,,sys} standard streams.

Set to @code{1} if the 
@geindex PYTHONLEGACYWINDOWSSTDIO
@geindex environment variable; PYTHONLEGACYWINDOWSSTDIO
@ref{7a2,,PYTHONLEGACYWINDOWSSTDIO} environment
variable is set to a non-empty string.

See 
@geindex Python Enhancement Proposals; PEP 528
PEP 528@footnote{https://www.python.org/dev/peps/pep-0528} for more details.

@ref{1629,,Availability}: Windows.
@end deffn

@geindex Py_NoSiteFlag (C var)
@anchor{c-api/init c Py_NoSiteFlag}@anchor{44f8}
@deffn {C Variable} int  Py_NoSiteFlag

Disable the import of the module @ref{e6,,site} and the site-dependent
manipulations of @ref{489,,sys.path} that it entails.  Also disable these
manipulations if @ref{e6,,site} is explicitly imported later (call
@ref{1610,,site.main()} if you want them to be triggered).

Set by the @ref{d4e,,-S} option.
@end deffn

@geindex Py_NoUserSiteDirectory (C var)
@anchor{c-api/init c Py_NoUserSiteDirectory}@anchor{44f9}
@deffn {C Variable} int  Py_NoUserSiteDirectory

Don’t add the @ref{160f,,user site-packages directory} to
@ref{489,,sys.path}.

Set by the @ref{f15,,-s} and @ref{461,,-I} options, and the
@geindex PYTHONNOUSERSITE
@geindex environment variable; PYTHONNOUSERSITE
@ref{f16,,PYTHONNOUSERSITE} environment variable.
@end deffn

@geindex Py_OptimizeFlag (C var)
@anchor{c-api/init c Py_OptimizeFlag}@anchor{2790}
@deffn {C Variable} int  Py_OptimizeFlag

Set by the @ref{912,,-O} option and the 
@geindex PYTHONOPTIMIZE
@geindex environment variable; PYTHONOPTIMIZE
@ref{160b,,PYTHONOPTIMIZE} environment
variable.
@end deffn

@geindex Py_QuietFlag (C var)
@anchor{c-api/init c Py_QuietFlag}@anchor{44fa}
@deffn {C Variable} int  Py_QuietFlag

Don’t display the copyright and version messages even in interactive mode.

Set by the @ref{160d,,-q} option.


New in version 3.2.

@end deffn

@geindex Py_UnbufferedStdioFlag (C var)
@anchor{c-api/init c Py_UnbufferedStdioFlag}@anchor{44fb}
@deffn {C Variable} int  Py_UnbufferedStdioFlag

Force the stdout and stderr streams to be unbuffered.

Set by the @ref{123a,,-u} option and the 
@geindex PYTHONUNBUFFERED
@geindex environment variable; PYTHONUNBUFFERED
@ref{123b,,PYTHONUNBUFFERED}
environment variable.
@end deffn

@geindex Py_VerboseFlag (C var)
@anchor{c-api/init c Py_VerboseFlag}@anchor{44fc}
@deffn {C Variable} int  Py_VerboseFlag

Print a message each time a module is initialized, showing the place
(filename or built-in module) from which it is loaded.  If greater or equal
to @code{2}, print a message for each file that is checked for when
searching for a module. Also provides information on module cleanup at exit.

Set by the @ref{f82,,-v} option and the 
@geindex PYTHONVERBOSE
@geindex environment variable; PYTHONVERBOSE
@ref{1611,,PYTHONVERBOSE} environment
variable.
@end deffn

@node Initializing and finalizing the interpreter,Process-wide parameters,Global configuration variables,Initialization Finalization and Threads
@anchor{c-api/init initializing-and-finalizing-the-interpreter}@anchor{44fd}
@subsection Initializing and finalizing the interpreter


@geindex Py_Initialize (C function)
@anchor{c-api/init c Py_Initialize}@anchor{38d}
@deffn {C Function} void  Py_Initialize ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex Py_SetProgramName()
@geindex PyEval_InitThreads()
@geindex modules (in module sys)
@geindex path (in module sys)
@geindex module; builtins
@geindex module; __main__
@geindex module; sys
@geindex module; search; path
@geindex PySys_SetArgv()
@geindex PySys_SetArgvEx()
@geindex Py_FinalizeEx()

Initialize the Python interpreter.  In an application embedding  Python,
this should be called before using any other Python/C API functions; see
@ref{6fc,,Before Python Initialization} for the few exceptions.

This initializes
the table of loaded modules (@code{sys.modules}), and creates the fundamental
modules @ref{11,,builtins}, @ref{1,,__main__} and @ref{f7,,sys}.  It also initializes
the module search path (@code{sys.path}). It does not set @code{sys.argv}; use
@ref{e14,,PySys_SetArgvEx()} for that.  This is a no-op when called for a second time
(without calling @ref{862,,Py_FinalizeEx()} first).  There is no return value; it is a
fatal error if the initialization fails.

@cartouche
@quotation Note 
On Windows, changes the console mode from @code{O_TEXT} to @code{O_BINARY}, which will
also affect non-Python uses of the console using the C Runtime.
@end quotation
@end cartouche
@end deffn

@geindex Py_InitializeEx (C function)
@anchor{c-api/init c Py_InitializeEx}@anchor{429f}
@deffn {C Function} void  Py_InitializeEx (int  initsigs)
@emph{ Part of the @ref{35d,,Stable ABI}.}
This function works like @ref{38d,,Py_Initialize()} if @emph{initsigs} is @code{1}. If
@emph{initsigs} is @code{0}, it skips initialization registration of signal handlers, which
might be useful when Python is embedded.
@end deffn

@geindex Py_IsInitialized (C function)
@anchor{c-api/init c Py_IsInitialized}@anchor{411d}
@deffn {C Function} int  Py_IsInitialized ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return true (nonzero) when the Python interpreter has been initialized, false
(zero) if not.  After @ref{862,,Py_FinalizeEx()} is called, this returns false until
@ref{38d,,Py_Initialize()} is called again.
@end deffn

@geindex Py_FinalizeEx (C function)
@anchor{c-api/init c Py_FinalizeEx}@anchor{862}
@deffn {C Function} int  Py_FinalizeEx ()
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.6.}
Undo all initializations made by @ref{38d,,Py_Initialize()} and subsequent use of
Python/C API functions, and destroy all sub-interpreters (see
@ref{42a0,,Py_NewInterpreter()} below) that were created and not yet destroyed since
the last call to @ref{38d,,Py_Initialize()}.  Ideally, this frees all memory
allocated by the Python interpreter.  This is a no-op when called for a second
time (without calling @ref{38d,,Py_Initialize()} again first).  Normally the
return value is @code{0}.  If there were errors during finalization
(flushing buffered data), @code{-1} is returned.

This function is provided for a number of reasons.  An embedding application
might want to restart Python without having to restart the application itself.
An application that has loaded the Python interpreter from a dynamically
loadable library (or DLL) might want to free all memory allocated by Python
before unloading the DLL. During a hunt for memory leaks in an application a
developer might want to free all memory allocated by Python before exiting from
the application.

@strong{Bugs and caveats:} The destruction of modules and objects in modules is done
in random order; this may cause destructors (@code{__del__()} methods) to fail
when they depend on other objects (even functions) or modules.  Dynamically
loaded extension modules loaded by Python are not unloaded.  Small amounts of
memory allocated by the Python interpreter may not be freed (if you find a leak,
please report it).  Memory tied up in circular references between objects is not
freed.  Some memory allocated by extension modules may not be freed.  Some
extensions may not work properly if their initialization routine is called more
than once; this can happen if an application calls @ref{38d,,Py_Initialize()} and
@ref{862,,Py_FinalizeEx()} more than once.

Raises an @ref{1130,,auditing event} @code{cpython._PySys_ClearAuditHooks} with no arguments.


New in version 3.6.

@end deffn

@geindex Py_Finalize (C function)
@anchor{c-api/init c Py_Finalize}@anchor{eed}
@deffn {C Function} void  Py_Finalize ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
This is a backwards-compatible version of @ref{862,,Py_FinalizeEx()} that
disregards the return value.
@end deffn

@node Process-wide parameters,Thread State and the Global Interpreter Lock,Initializing and finalizing the interpreter,Initialization Finalization and Threads
@anchor{c-api/init process-wide-parameters}@anchor{44fe}
@subsection Process-wide parameters


@geindex Py_SetStandardStreamEncoding (C function)
@anchor{c-api/init c Py_SetStandardStreamEncoding}@anchor{205}
@deffn {C Function} int  Py_SetStandardStreamEncoding (const  char  *encoding, const  char  *errors)

@geindex Py_Initialize()
@geindex main()
@geindex stdin; stdout; sdterr

This API is kept for backward compatibility: setting
@ref{3b77,,PyConfig.stdio_encoding} and @ref{3b78,,PyConfig.stdio_errors}
should be used instead, see @ref{207,,Python Initialization Configuration}.

This function should be called before @ref{38d,,Py_Initialize()}, if it is
called at all. It specifies which encoding and error handling to use
with standard IO, with the same meanings as in @ref{3ba,,str.encode()}.

It overrides 
@geindex PYTHONIOENCODING
@geindex environment variable; PYTHONIOENCODING
@ref{b91,,PYTHONIOENCODING} values, and allows embedding code
to control IO encoding when the environment variable does not work.

@emph{encoding} and/or @emph{errors} may be @code{NULL} to use
@geindex PYTHONIOENCODING
@geindex environment variable; PYTHONIOENCODING
@ref{b91,,PYTHONIOENCODING} and/or default values (depending on other
settings).

Note that @ref{42d,,sys.stderr} always uses the “backslashreplace” error
handler, regardless of this (or any other) setting.

If @ref{862,,Py_FinalizeEx()} is called, this function will need to be called
again in order to affect subsequent calls to @ref{38d,,Py_Initialize()}.

Returns @code{0} if successful, a nonzero value on error (e.g. calling after the
interpreter has already been initialized).


New in version 3.4.



Deprecated since version 3.11.

@end deffn

@geindex Py_SetProgramName (C function)
@anchor{c-api/init c Py_SetProgramName}@anchor{203}
@deffn {C Function} void  Py_SetProgramName (const  wchar_t  *name)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex Py_Initialize()
@geindex main()
@geindex Py_GetPath()

This API is kept for backward compatibility: setting
@ref{44ff,,PyConfig.program_name} should be used instead, see @ref{207,,Python Initialization Configuration}.

This function should be called before @ref{38d,,Py_Initialize()} is called for
the first time, if it is called at all.  It tells the interpreter the value
of the @code{argv[0]} argument to the @code{main()} function of the program
(converted to wide characters).
This is used by @ref{387,,Py_GetPath()} and some other functions below to find
the Python run-time libraries relative to the interpreter executable.  The
default value is @code{'python'}.  The argument should point to a
zero-terminated wide character string in static storage whose contents will not
change for the duration of the program’s execution.  No code in the Python
interpreter will change the contents of this storage.

Use @ref{6fe,,Py_DecodeLocale()} to decode a bytes string to get a
wchar_* string.


Deprecated since version 3.11.

@end deffn

@geindex Py_GetProgramName (C function)
@anchor{c-api/init c Py_GetProgramName}@anchor{38c}
@deffn {C Function} wchar  *Py_GetProgramName ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex Py_SetProgramName()

Return the program name set with @ref{203,,Py_SetProgramName()}, or the default.
The returned string points into static storage; the caller should not modify its
value.

This function should not be called before @ref{38d,,Py_Initialize()}, otherwise
it returns @code{NULL}.


Changed in version 3.10: It now returns @code{NULL} if called before @ref{38d,,Py_Initialize()}.

@end deffn

@geindex Py_GetPrefix (C function)
@anchor{c-api/init c Py_GetPrefix}@anchor{388}
@deffn {C Function} wchar_t  *Py_GetPrefix ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the @emph{prefix} for installed platform-independent files. This is derived
through a number of complicated rules from the program name set with
@ref{203,,Py_SetProgramName()} and some environment variables; for example, if the
program name is @code{'/usr/local/bin/python'}, the prefix is @code{'/usr/local'}. The
returned string points into static storage; the caller should not modify its
value.  This corresponds to the @code{prefix} variable in the top-level
@code{Makefile} and the @code{--prefix} argument to the @code{configure}
script at build time.  The value is available to Python code as @code{sys.prefix}.
It is only useful on Unix.  See also the next function.

This function should not be called before @ref{38d,,Py_Initialize()}, otherwise
it returns @code{NULL}.


Changed in version 3.10: It now returns @code{NULL} if called before @ref{38d,,Py_Initialize()}.

@end deffn

@geindex Py_GetExecPrefix (C function)
@anchor{c-api/init c Py_GetExecPrefix}@anchor{389}
@deffn {C Function} wchar_t  *Py_GetExecPrefix ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the @emph{exec-prefix} for installed platform-@emph{dependent} files.  This is
derived through a number of complicated rules from the program name set with
@ref{203,,Py_SetProgramName()} and some environment variables; for example, if the
program name is @code{'/usr/local/bin/python'}, the exec-prefix is
@code{'/usr/local'}.  The returned string points into static storage; the caller
should not modify its value.  This corresponds to the @code{exec_prefix}
variable in the top-level @code{Makefile} and the @code{--exec-prefix}
argument to the @code{configure} script at build  time.  The value is
available to Python code as @code{sys.exec_prefix}.  It is only useful on Unix.

Background: The exec-prefix differs from the prefix when platform dependent
files (such as executables and shared libraries) are installed in a different
directory tree.  In a typical installation, platform dependent files may be
installed in the @code{/usr/local/plat} subtree while platform independent may
be installed in @code{/usr/local}.

Generally speaking, a platform is a combination of hardware and software
families, e.g.  Sparc machines running the Solaris 2.x operating system are
considered the same platform, but Intel machines running Solaris 2.x are another
platform, and Intel machines running Linux are yet another platform.  Different
major revisions of the same operating system generally also form different
platforms.  Non-Unix operating systems are a different story; the installation
strategies on those systems are so different that the prefix and exec-prefix are
meaningless, and set to the empty string. Note that compiled Python bytecode
files are platform independent (but not independent from the Python version by
which they were compiled!).

System administrators will know how to configure the @code{mount} or
@code{automount} programs to share @code{/usr/local} between platforms
while having @code{/usr/local/plat} be a different filesystem for each
platform.

This function should not be called before @ref{38d,,Py_Initialize()}, otherwise
it returns @code{NULL}.


Changed in version 3.10: It now returns @code{NULL} if called before @ref{38d,,Py_Initialize()}.

@end deffn

@geindex Py_GetProgramFullPath (C function)
@anchor{c-api/init c Py_GetProgramFullPath}@anchor{38a}
@deffn {C Function} wchar_t  *Py_GetProgramFullPath ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex Py_SetProgramName()
@geindex executable (in module sys)

Return the full program name of the Python executable; this is  computed as a
side-effect of deriving the default module search path  from the program name
(set by @ref{203,,Py_SetProgramName()} above). The returned string points into
static storage; the caller should not modify its value.  The value is available
to Python code as @code{sys.executable}.

This function should not be called before @ref{38d,,Py_Initialize()}, otherwise
it returns @code{NULL}.


Changed in version 3.10: It now returns @code{NULL} if called before @ref{38d,,Py_Initialize()}.

@end deffn

@geindex Py_GetPath (C function)
@anchor{c-api/init c Py_GetPath}@anchor{387}
@deffn {C Function} wchar_t  *Py_GetPath ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex module; search; path
@geindex path (in module sys)
@geindex Py_SetPath()

Return the default module search path; this is computed from the program name
(set by @ref{203,,Py_SetProgramName()} above) and some environment variables.
The returned string consists of a series of directory names separated by a
platform dependent delimiter character.  The delimiter character is @code{':'}
on Unix and macOS, @code{';'} on Windows.  The returned string points into
static storage; the caller should not modify its value.  The list
@ref{489,,sys.path} is initialized with this value on interpreter startup; it
can be (and usually is) modified later to change the search path for loading
modules.

This function should not be called before @ref{38d,,Py_Initialize()}, otherwise
it returns @code{NULL}.

@c XXX should give the exact rules


Changed in version 3.10: It now returns @code{NULL} if called before @ref{38d,,Py_Initialize()}.

@end deffn

@geindex Py_SetPath (C function)
@anchor{c-api/init c Py_SetPath}@anchor{202}
@deffn {C Function} void  Py_SetPath (const  wchar_t*)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
@geindex module; search; path
@geindex path (in module sys)
@geindex Py_GetPath()

This API is kept for backward compatibility: setting
@ref{4500,,PyConfig.module_search_paths} and
@ref{4501,,PyConfig.module_search_paths_set} should be used instead, see
@ref{207,,Python Initialization Configuration}.

Set the default module search path.  If this function is called before
@ref{38d,,Py_Initialize()}, then @ref{387,,Py_GetPath()} won’t attempt to compute a
default search path but uses the one provided instead.  This is useful if
Python is embedded by an application that has full knowledge of the location
of all modules.  The path components should be separated by the platform
dependent delimiter character, which is @code{':'} on Unix and macOS, @code{';'}
on Windows.

This also causes @ref{585,,sys.executable} to be set to the program
full path (see @ref{38a,,Py_GetProgramFullPath()}) and for @ref{3b34,,sys.prefix} and
@ref{3b35,,sys.exec_prefix} to be empty.  It is up to the caller to modify these
if required after calling @ref{38d,,Py_Initialize()}.

Use @ref{6fe,,Py_DecodeLocale()} to decode a bytes string to get a
wchar_* string.

The path argument is copied internally, so the caller may free it after the
call completes.


Changed in version 3.8: The program full path is now used for @ref{585,,sys.executable}, instead
of the program name.



Deprecated since version 3.11.

@end deffn

@geindex Py_GetVersion (C function)
@anchor{c-api/init c Py_GetVersion}@anchor{429e}
@deffn {C Function} const  char  *Py_GetVersion ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the version of this Python interpreter.  This is a string that looks
something like

@example
"3.0a5+ (py3k:63103M, May 12 2008, 00:53:55) \n[GCC 4.2.3]"
@end example

@geindex version (in module sys)

The first word (up to the first space character) is the current Python version;
the first three characters are the major and minor version separated by a
period.  The returned string points into static storage; the caller should not
modify its value.  The value is available to Python code as @ref{86c,,sys.version}.

See also the @code{Py_Version} constant.
@end deffn

@geindex Py_GetPlatform (C function)
@anchor{c-api/init c Py_GetPlatform}@anchor{429d}
@deffn {C Function} const  char  *Py_GetPlatform ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex platform (in module sys)

Return the platform identifier for the current platform.  On Unix, this is
formed from the “official” name of the operating system, converted to lower
case, followed by the major revision number; e.g., for Solaris 2.x, which is
also known as SunOS 5.x, the value is @code{'sunos5'}.  On macOS, it is
@code{'darwin'}.  On Windows, it is @code{'win'}.  The returned string points into
static storage; the caller should not modify its value.  The value is available
to Python code as @code{sys.platform}.
@end deffn

@geindex Py_GetCopyright (C function)
@anchor{c-api/init c Py_GetCopyright}@anchor{429c}
@deffn {C Function} const  char  *Py_GetCopyright ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the official copyright string for the current Python version, for example

@code{'Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam'}

@geindex copyright (in module sys)

The returned string points into static storage; the caller should not modify its
value.  The value is available to Python code as @code{sys.copyright}.
@end deffn

@geindex Py_GetCompiler (C function)
@anchor{c-api/init c Py_GetCompiler}@anchor{429b}
@deffn {C Function} const  char  *Py_GetCompiler ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return an indication of the compiler used to build the current Python version,
in square brackets, for example:

@example
"[GCC 2.7.2.2]"
@end example

@geindex version (in module sys)

The returned string points into static storage; the caller should not modify its
value.  The value is available to Python code as part of the variable
@code{sys.version}.
@end deffn

@geindex Py_GetBuildInfo (C function)
@anchor{c-api/init c Py_GetBuildInfo}@anchor{f94}
@deffn {C Function} const  char  *Py_GetBuildInfo ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return information about the sequence number and build date and time  of the
current Python interpreter instance, for example

@example
"#67, Aug  1 1997, 22:34:28"
@end example

@geindex version (in module sys)

The returned string points into static storage; the caller should not modify its
value.  The value is available to Python code as part of the variable
@code{sys.version}.
@end deffn

@geindex PySys_SetArgvEx (C function)
@anchor{c-api/init c PySys_SetArgvEx}@anchor{e14}
@deffn {C Function} void  PySys_SetArgvEx (int  argc, wchar_t  **argv, int  updatepath)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex main()
@geindex Py_FatalError()
@geindex argv (in module sys)

Set @ref{e15,,sys.argv} based on @emph{argc} and @emph{argv}.  These parameters are
similar to those passed to the program’s @code{main()} function with the
difference that the first entry should refer to the script file to be
executed rather than the executable hosting the Python interpreter.  If there
isn’t a script that will be run, the first entry in @emph{argv} can be an empty
string.  If this function fails to initialize @ref{e15,,sys.argv}, a fatal
condition is signalled using @ref{48f,,Py_FatalError()}.

If @emph{updatepath} is zero, this is all the function does.  If @emph{updatepath}
is non-zero, the function also modifies @ref{489,,sys.path} according to the
following algorithm:


@itemize -

@item 
If the name of an existing script is passed in @code{argv[0]}, the absolute
path of the directory where the script is located is prepended to
@ref{489,,sys.path}.

@item 
Otherwise (that is, if @emph{argc} is @code{0} or @code{argv[0]} doesn’t point
to an existing file name), an empty string is prepended to
@ref{489,,sys.path}, which is the same as prepending the current working
directory (@code{"."}).
@end itemize

Use @ref{6fe,,Py_DecodeLocale()} to decode a bytes string to get a
wchar_* string.

See also @ref{360,,PyConfig.orig_argv} and @ref{11a1,,PyConfig.argv}
members of the @ref{207,,Python Initialization Configuration}.

@cartouche
@quotation Note 
It is recommended that applications embedding the Python interpreter
for purposes other than executing a single script pass @code{0} as @emph{updatepath},
and update @ref{489,,sys.path} themselves if desired.
See CVE-2008-5983@footnote{https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5983}.

On versions before 3.1.3, you can achieve the same effect by manually
popping the first @ref{489,,sys.path} element after having called
@ref{eec,,PySys_SetArgv()}, for example using:

@example
PyRun_SimpleString("import sys; sys.path.pop(0)\n");
@end example
@end quotation
@end cartouche


New in version 3.1.3.


@c XXX impl. doesn't seem consistent in allowing ``0``/``NULL`` for the params;
@c check w/ Guido.
@end deffn

@geindex PySys_SetArgv (C function)
@anchor{c-api/init c PySys_SetArgv}@anchor{eec}
@deffn {C Function} void  PySys_SetArgv (int  argc, wchar_t  **argv)
@emph{ Part of the @ref{35d,,Stable ABI}.}
This function works like @ref{e14,,PySys_SetArgvEx()} with @emph{updatepath} set
to @code{1} unless the @code{python} interpreter was started with the
@ref{461,,-I}.

Use @ref{6fe,,Py_DecodeLocale()} to decode a bytes string to get a
wchar_* string.

See also @ref{360,,PyConfig.orig_argv} and @ref{11a1,,PyConfig.argv}
members of the @ref{207,,Python Initialization Configuration}.


Changed in version 3.4: The @emph{updatepath} value depends on @ref{461,,-I}.

@end deffn

@geindex Py_SetPythonHome (C function)
@anchor{c-api/init c Py_SetPythonHome}@anchor{204}
@deffn {C Function} void  Py_SetPythonHome (const  wchar_t  *home)
@emph{ Part of the @ref{35d,,Stable ABI}.}
This API is kept for backward compatibility: setting
@ref{4502,,PyConfig.home} should be used instead, see @ref{207,,Python Initialization Configuration}.

Set the default “home” directory, that is, the location of the standard
Python libraries.  See 
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME} for the meaning of the
argument string.

The argument should point to a zero-terminated character string in static
storage whose contents will not change for the duration of the program’s
execution.  No code in the Python interpreter will change the contents of
this storage.

Use @ref{6fe,,Py_DecodeLocale()} to decode a bytes string to get a
wchar_* string.


Deprecated since version 3.11.

@end deffn

@geindex Py_GetPythonHome (C function)
@anchor{c-api/init c Py_GetPythonHome}@anchor{38b}
@deffn {C Function} w_char  *Py_GetPythonHome ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the default “home”, that is, the value set by a previous call to
@ref{204,,Py_SetPythonHome()}, or the value of the 
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME}
environment variable if it is set.

This function should not be called before @ref{38d,,Py_Initialize()}, otherwise
it returns @code{NULL}.


Changed in version 3.10: It now returns @code{NULL} if called before @ref{38d,,Py_Initialize()}.

@end deffn

@node Thread State and the Global Interpreter Lock,Sub-interpreter support,Process-wide parameters,Initialization Finalization and Threads
@anchor{c-api/init thread-state-and-the-global-interpreter-lock}@anchor{4503}@anchor{c-api/init threads}@anchor{4504}
@subsection Thread State and the Global Interpreter Lock


@geindex global interpreter lock
@geindex interpreter lock
@geindex lock@comma{} interpreter

The Python interpreter is not fully thread-safe.  In order to support
multi-threaded Python programs, there’s a global lock, called the @ref{7ae,,global interpreter lock} or @ref{385,,GIL}, that must be held by the current thread before
it can safely access Python objects. Without the lock, even the simplest
operations could cause problems in a multi-threaded program: for example, when
two threads simultaneously increment the reference count of the same object, the
reference count could end up being incremented only once instead of twice.

@geindex setswitchinterval() (in module sys)

Therefore, the rule exists that only the thread that has acquired the
@ref{385,,GIL} may operate on Python objects or call Python/C API functions.
In order to emulate concurrency of execution, the interpreter regularly
tries to switch threads (see @ref{449,,sys.setswitchinterval()}).  The lock is also
released around potentially blocking I/O operations like reading or writing
a file, so that other Python threads can run in the meantime.

@geindex PyThreadState
@geindex PyThreadState

The Python interpreter keeps some thread-specific bookkeeping information
inside a data structure called @ref{1fa,,PyThreadState}.  There’s also one
global variable pointing to the current @ref{1fa,,PyThreadState}: it can
be retrieved using @ref{423b,,PyThreadState_Get()}.

@menu
* Releasing the GIL from extension code:: 
* Non-Python created threads:: 
* Cautions about fork(): Cautions about fork. 
* High-level API:: 
* Low-level API:: 

@end menu

@node Releasing the GIL from extension code,Non-Python created threads,,Thread State and the Global Interpreter Lock
@anchor{c-api/init releasing-the-gil-from-extension-code}@anchor{4505}
@subsubsection Releasing the GIL from extension code


Most extension code manipulating the @ref{385,,GIL} has the following simple
structure:

@example
Save the thread state in a local variable.
Release the global interpreter lock.
... Do some blocking I/O operation ...
Reacquire the global interpreter lock.
Restore the thread state from the local variable.
@end example

This is so common that a pair of macros exists to simplify it:

@example
Py_BEGIN_ALLOW_THREADS
... Do some blocking I/O operation ...
Py_END_ALLOW_THREADS
@end example

@geindex Py_BEGIN_ALLOW_THREADS
@geindex Py_END_ALLOW_THREADS

The @ref{4096,,Py_BEGIN_ALLOW_THREADS} macro opens a new block and declares a
hidden local variable; the @ref{5aa,,Py_END_ALLOW_THREADS} macro closes the
block.

The block above expands to the following code:

@example
PyThreadState *_save;

_save = PyEval_SaveThread();
... Do some blocking I/O operation ...
PyEval_RestoreThread(_save);
@end example

@geindex PyEval_RestoreThread()
@geindex PyEval_SaveThread()

Here is how these functions work: the global interpreter lock is used to protect the pointer to the
current thread state.  When releasing the lock and saving the thread state,
the current thread state pointer must be retrieved before the lock is released
(since another thread could immediately acquire the lock and store its own thread
state in the global variable). Conversely, when acquiring the lock and restoring
the thread state, the lock must be acquired before storing the thread state
pointer.

@cartouche
@quotation Note 
Calling system I/O functions is the most common use case for releasing
the GIL, but it can also be useful before calling long-running computations
which don’t need access to Python objects, such as compression or
cryptographic functions operating over memory buffers.  For example, the
standard @ref{14b,,zlib} and @ref{88,,hashlib} modules release the GIL when
compressing or hashing data.
@end quotation
@end cartouche

@node Non-Python created threads,Cautions about fork,Releasing the GIL from extension code,Thread State and the Global Interpreter Lock
@anchor{c-api/init gilstate}@anchor{4506}@anchor{c-api/init non-python-created-threads}@anchor{4507}
@subsubsection Non-Python created threads


When threads are created using the dedicated Python APIs (such as the
@ref{109,,threading} module), a thread state is automatically associated to them
and the code showed above is therefore correct.  However, when threads are
created from C (for example by a third-party library with its own thread
management), they don’t hold the GIL, nor is there a thread state structure
for them.

If you need to call Python code from these threads (often this will be part
of a callback API provided by the aforementioned third-party library),
you must first register these threads with the interpreter by
creating a thread state data structure, then acquiring the GIL, and finally
storing their thread state pointer, before you can start using the Python/C
API.  When you are done, you should reset the thread state pointer, release
the GIL, and finally free the thread state data structure.

The @ref{5ab,,PyGILState_Ensure()} and @ref{419c,,PyGILState_Release()} functions do
all of the above automatically.  The typical idiom for calling into Python
from a C thread is:

@example
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();

/* Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */

/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate);
@end example

Note that the PyGILState_* functions assume there is only one global
interpreter (created automatically by @ref{38d,,Py_Initialize()}).  Python
supports the creation of additional interpreters (using
@ref{42a0,,Py_NewInterpreter()}), but mixing multiple interpreters and the
PyGILState_* API is unsupported.

@node Cautions about fork,High-level API,Non-Python created threads,Thread State and the Global Interpreter Lock
@anchor{c-api/init cautions-about-fork}@anchor{4508}@anchor{c-api/init fork-and-threads}@anchor{42fa}
@subsubsection Cautions about fork()


Another important thing to note about threads is their behaviour in the face
of the C @code{fork()} call. On most systems with @code{fork()}, after a
process forks only the thread that issued the fork will exist.  This has a
concrete impact both on how locks must be handled and on all stored state
in CPython’s runtime.

The fact that only the “current” thread remains
means any locks held by other threads will never be released. Python solves
this for @ref{bb9,,os.fork()} by acquiring the locks it uses internally before
the fork, and releasing them afterwards. In addition, it resets any
@ref{27f2,,Lock Objects} in the child. When extending or embedding Python, there
is no way to inform Python of additional (non-Python) locks that need to be
acquired before or reset after a fork. OS facilities such as
@code{pthread_atfork()} would need to be used to accomplish the same thing.
Additionally, when extending or embedding Python, calling @code{fork()}
directly rather than through @ref{bb9,,os.fork()} (and returning to or calling
into Python) may result in a deadlock by one of Python’s internal locks
being held by a thread that is defunct after the fork.
@ref{5c1,,PyOS_AfterFork_Child()} tries to reset the necessary locks, but is not
always able to.

The fact that all other threads go away also means that CPython’s
runtime state there must be cleaned up properly, which @ref{bb9,,os.fork()}
does.  This means finalizing all other @ref{1fa,,PyThreadState} objects
belonging to the current interpreter and all other
@ref{3a0,,PyInterpreterState} objects.  Due to this and the special
nature of the @ref{42fb,,“main” interpreter},
@code{fork()} should only be called in that interpreter’s “main”
thread, where the CPython global runtime was originally initialized.
The only exception is if @code{exec()} will be called immediately
after.

@node High-level API,Low-level API,Cautions about fork,Thread State and the Global Interpreter Lock
@anchor{c-api/init high-level-api}@anchor{4509}
@subsubsection High-level API


These are the most commonly used types and functions when writing C extension
code, or when embedding the Python interpreter:

@geindex PyInterpreterState (C type)
@anchor{c-api/init c PyInterpreterState}@anchor{3a0}
@deffn {C Type} type  PyInterpreterState
@emph{ Part of the @ref{35d,,Stable ABI}.}
This data structure represents the state shared by a number of cooperating
threads.  Threads belonging to the same interpreter share their module
administration and a few other internal items. There are no public members in
this structure.

Threads belonging to different interpreters initially share nothing, except
process state like available memory, open file descriptors and such.  The global
interpreter lock is also shared by all threads, regardless of to which
interpreter they belong.
@end deffn

@geindex PyThreadState (C type)
@anchor{c-api/init c PyThreadState}@anchor{1fa}
@deffn {C Type} type  PyThreadState
@emph{ Part of the @ref{35d,,Stable ABI}.}
This data structure represents the state of a single thread.  The only public
data member is @code{interp} (@ref{3a0,,PyInterpreterState}*), which points to
this thread’s interpreter state.
@end deffn

@geindex PyEval_InitThreads (C function)
@anchor{c-api/init c PyEval_InitThreads}@anchor{441}
@deffn {C Function} void  PyEval_InitThreads ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex PyEval_AcquireThread()
@geindex PyEval_ReleaseThread()
@geindex PyEval_SaveThread()
@geindex PyEval_RestoreThread()

Deprecated function which does nothing.

In Python 3.6 and older, this function created the GIL if it didn’t exist.


Changed in version 3.9: The function now does nothing.



Changed in version 3.7: This function is now called by @ref{38d,,Py_Initialize()}, so you don’t
have to call it yourself anymore.



Changed in version 3.2: This function cannot be called before @ref{38d,,Py_Initialize()} anymore.



Deprecated since version 3.9, removed in version 3.11.


@geindex module; _thread
@end deffn

@geindex PyEval_ThreadsInitialized (C function)
@anchor{c-api/init c PyEval_ThreadsInitialized}@anchor{442}
@deffn {C Function} int  PyEval_ThreadsInitialized ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Returns a non-zero value if @ref{441,,PyEval_InitThreads()} has been called.  This
function can be called without holding the GIL, and therefore can be used to
avoid calls to the locking API when running single-threaded.


Changed in version 3.7: The @ref{385,,GIL} is now initialized by @ref{38d,,Py_Initialize()}.



Deprecated since version 3.9, removed in version 3.11.

@end deffn

@geindex PyEval_SaveThread (C function)
@anchor{c-api/init c PyEval_SaveThread}@anchor{e2a}
@deffn {C Function} @ref{1fa,,PyThreadState}  *PyEval_SaveThread ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Release the global interpreter lock (if it has been created) and reset the
thread state to @code{NULL}, returning the previous thread state (which is not
@code{NULL}).  If the lock has been created, the current thread must have
acquired it.
@end deffn

@geindex PyEval_RestoreThread (C function)
@anchor{c-api/init c PyEval_RestoreThread}@anchor{5a9}
@deffn {C Function} void  PyEval_RestoreThread (PyThreadState  *tstate)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Acquire the global interpreter lock (if it has been created) and set the
thread state to @emph{tstate}, which must not be @code{NULL}.  If the lock has been
created, the current thread must not have acquired it, otherwise deadlock
ensues.

@cartouche
@quotation Note 
Calling this function from a thread when the runtime is finalizing
will terminate the thread, even if the thread was not created by Python.
You can use @code{_Py_IsFinalizing()} or @ref{9e1,,sys.is_finalizing()} to
check if the interpreter is in process of being finalized before calling
this function to avoid unwanted termination.
@end quotation
@end cartouche
@end deffn

@geindex PyThreadState_Get (C function)
@anchor{c-api/init c PyThreadState_Get}@anchor{423b}
@deffn {C Function} @ref{1fa,,PyThreadState}  *PyThreadState_Get ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the current thread state.  The global interpreter lock must be held.
When the current thread state is @code{NULL}, this issues a fatal error (so that
the caller needn’t check for @code{NULL}).
@end deffn

@geindex PyThreadState_Swap (C function)
@anchor{c-api/init c PyThreadState_Swap}@anchor{423e}
@deffn {C Function} @ref{1fa,,PyThreadState}  *PyThreadState_Swap (PyThreadState  *tstate)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Swap the current thread state with the thread state given by the argument
@emph{tstate}, which may be @code{NULL}.  The global interpreter lock must be held
and is not released.
@end deffn

The following functions use thread-local storage, and are not compatible
with sub-interpreters:

@geindex PyGILState_Ensure (C function)
@anchor{c-api/init c PyGILState_Ensure}@anchor{5ab}
@deffn {C Function} PyGILState_STATE  PyGILState_Ensure ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Ensure that the current thread is ready to call the Python C API regardless
of the current state of Python, or of the global interpreter lock. This may
be called as many times as desired by a thread as long as each call is
matched with a call to @ref{419c,,PyGILState_Release()}. In general, other
thread-related APIs may be used between @ref{5ab,,PyGILState_Ensure()} and
@ref{419c,,PyGILState_Release()} calls as long as the thread state is restored to
its previous state before the Release().  For example, normal usage of the
@ref{4096,,Py_BEGIN_ALLOW_THREADS} and @ref{5aa,,Py_END_ALLOW_THREADS} macros is
acceptable.

The return value is an opaque “handle” to the thread state when
@ref{5ab,,PyGILState_Ensure()} was called, and must be passed to
@ref{419c,,PyGILState_Release()} to ensure Python is left in the same state. Even
though recursive calls are allowed, these handles @emph{cannot} be shared - each
unique call to @ref{5ab,,PyGILState_Ensure()} must save the handle for its call
to @ref{419c,,PyGILState_Release()}.

When the function returns, the current thread will hold the GIL and be able
to call arbitrary Python code.  Failure is a fatal error.

@cartouche
@quotation Note 
Calling this function from a thread when the runtime is finalizing
will terminate the thread, even if the thread was not created by Python.
You can use @code{_Py_IsFinalizing()} or @ref{9e1,,sys.is_finalizing()} to
check if the interpreter is in process of being finalized before calling
this function to avoid unwanted termination.
@end quotation
@end cartouche
@end deffn

@geindex PyGILState_Release (C function)
@anchor{c-api/init c PyGILState_Release}@anchor{419c}
@deffn {C Function} void  PyGILState_Release (PyGILState_STATE)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Release any resources previously acquired.  After this call, Python’s state will
be the same as it was prior to the corresponding @ref{5ab,,PyGILState_Ensure()} call
(but generally this state will be unknown to the caller, hence the use of the
GILState API).

Every call to @ref{5ab,,PyGILState_Ensure()} must be matched by a call to
@ref{419c,,PyGILState_Release()} on the same thread.
@end deffn

@geindex PyGILState_GetThisThreadState (C function)
@anchor{c-api/init c PyGILState_GetThisThreadState}@anchor{419b}
@deffn {C Function} @ref{1fa,,PyThreadState}  *PyGILState_GetThisThreadState ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Get the current thread state for this thread.  May return @code{NULL} if no
GILState API has been used on the current thread.  Note that the main thread
always has such a thread-state, even if no auto-thread-state call has been
made on the main thread.  This is mainly a helper/diagnostic function.
@end deffn

@geindex PyGILState_Check (C function)
@anchor{c-api/init c PyGILState_Check}@anchor{450a}
@deffn {C Function} int  PyGILState_Check ()

Return @code{1} if the current thread is holding the GIL and @code{0} otherwise.
This function can be called from any thread at any time.
Only if it has had its Python thread state initialized and currently is
holding the GIL will it return @code{1}.
This is mainly a helper/diagnostic function.  It can be useful
for example in callback contexts or memory allocation functions when
knowing that the GIL is locked can allow the caller to perform sensitive
actions or otherwise behave differently.


New in version 3.4.

@end deffn

The following macros are normally used without a trailing semicolon; look for
example usage in the Python source distribution.

@geindex Py_BEGIN_ALLOW_THREADS (C macro)
@anchor{c-api/init c Py_BEGIN_ALLOW_THREADS}@anchor{4096}
@deffn {C Macro} Py_BEGIN_ALLOW_THREADS
@emph{ Part of the @ref{35d,,Stable ABI}.}
This macro expands to @code{@{ PyThreadState *_save; _save = PyEval_SaveThread();}.
Note that it contains an opening brace; it must be matched with a following
@ref{5aa,,Py_END_ALLOW_THREADS} macro.  See above for further discussion of this
macro.
@end deffn

@geindex Py_END_ALLOW_THREADS (C macro)
@anchor{c-api/init c Py_END_ALLOW_THREADS}@anchor{5aa}
@deffn {C Macro} Py_END_ALLOW_THREADS
@emph{ Part of the @ref{35d,,Stable ABI}.}
This macro expands to @code{PyEval_RestoreThread(_save); @}}. Note that it contains
a closing brace; it must be matched with an earlier
@ref{4096,,Py_BEGIN_ALLOW_THREADS} macro.  See above for further discussion of
this macro.
@end deffn

@geindex Py_BLOCK_THREADS (C macro)
@anchor{c-api/init c Py_BLOCK_THREADS}@anchor{4298}
@deffn {C Macro} Py_BLOCK_THREADS
@emph{ Part of the @ref{35d,,Stable ABI}.}
This macro expands to @code{PyEval_RestoreThread(_save);}: it is equivalent to
@ref{5aa,,Py_END_ALLOW_THREADS} without the closing brace.
@end deffn

@geindex Py_UNBLOCK_THREADS (C macro)
@anchor{c-api/init c Py_UNBLOCK_THREADS}@anchor{42a3}
@deffn {C Macro} Py_UNBLOCK_THREADS
@emph{ Part of the @ref{35d,,Stable ABI}.}
This macro expands to @code{_save = PyEval_SaveThread();}: it is equivalent to
@ref{4096,,Py_BEGIN_ALLOW_THREADS} without the opening brace and variable
declaration.
@end deffn

@node Low-level API,,High-level API,Thread State and the Global Interpreter Lock
@anchor{c-api/init low-level-api}@anchor{450b}
@subsubsection Low-level API


All of the following functions must be called after @ref{38d,,Py_Initialize()}.


Changed in version 3.7: @ref{38d,,Py_Initialize()} now initializes the @ref{385,,GIL}.


@geindex PyInterpreterState_New (C function)
@anchor{c-api/init c PyInterpreterState_New}@anchor{39ee}
@deffn {C Function} @ref{3a0,,PyInterpreterState}  *PyInterpreterState_New ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a new interpreter state object.  The global interpreter lock need not
be held, but may be held if it is necessary to serialize calls to this
function.

Raises an @ref{1130,,auditing event} @code{cpython.PyInterpreterState_New} with no arguments.
@end deffn

@geindex PyInterpreterState_Clear (C function)
@anchor{c-api/init c PyInterpreterState_Clear}@anchor{39ed}
@deffn {C Function} void  PyInterpreterState_Clear (PyInterpreterState  *interp)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Reset all information in an interpreter state object.  The global interpreter
lock must be held.

Raises an @ref{1130,,auditing event} @code{cpython.PyInterpreterState_Clear} with no arguments.
@end deffn

@geindex PyInterpreterState_Delete (C function)
@anchor{c-api/init c PyInterpreterState_Delete}@anchor{41a9}
@deffn {C Function} void  PyInterpreterState_Delete (PyInterpreterState  *interp)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Destroy an interpreter state object.  The global interpreter lock need not be
held.  The interpreter state must have been reset with a previous call to
@ref{39ed,,PyInterpreterState_Clear()}.
@end deffn

@geindex PyThreadState_New (C function)
@anchor{c-api/init c PyThreadState_New}@anchor{423d}
@deffn {C Function} @ref{1fa,,PyThreadState}  *PyThreadState_New (PyInterpreterState  *interp)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Create a new thread state object belonging to the given interpreter object.
The global interpreter lock need not be held, but may be held if it is
necessary to serialize calls to this function.
@end deffn

@geindex PyThreadState_Clear (C function)
@anchor{c-api/init c PyThreadState_Clear}@anchor{11f3}
@deffn {C Function} void  PyThreadState_Clear (PyThreadState  *tstate)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Reset all information in a thread state object.  The global interpreter lock
must be held.


Changed in version 3.9: This function now calls the @code{PyThreadState.on_delete} callback.
Previously, that happened in @ref{1236,,PyThreadState_Delete()}.

@end deffn

@geindex PyThreadState_Delete (C function)
@anchor{c-api/init c PyThreadState_Delete}@anchor{1236}
@deffn {C Function} void  PyThreadState_Delete (PyThreadState  *tstate)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Destroy a thread state object.  The global interpreter lock need not be held.
The thread state must have been reset with a previous call to
@ref{11f3,,PyThreadState_Clear()}.
@end deffn

@geindex PyThreadState_DeleteCurrent (C function)
@anchor{c-api/init c PyThreadState_DeleteCurrent}@anchor{1275}
@deffn {C Function} void  PyThreadState_DeleteCurrent (void)

Destroy the current thread state and release the global interpreter lock.
Like @ref{1236,,PyThreadState_Delete()}, the global interpreter lock need not
be held. The thread state must have been reset with a previous call
to @ref{11f3,,PyThreadState_Clear()}.
@end deffn

@geindex PyThreadState_GetFrame (C function)
@anchor{c-api/init c PyThreadState_GetFrame}@anchor{1fb}
@deffn {C Function} @ref{1f6,,PyFrameObject}  *PyThreadState_GetFrame (PyThreadState  *tstate)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Get the current frame of the Python thread state @emph{tstate}.

Return a @ref{1fc,,strong reference}. Return @code{NULL} if no frame is currently
executing.

See also @ref{4188,,PyEval_GetFrame()}.

@emph{tstate} must not be @code{NULL}.


New in version 3.9.

@end deffn

@geindex PyThreadState_GetID (C function)
@anchor{c-api/init c PyThreadState_GetID}@anchor{478}
@deffn {C Function} uint64_t  PyThreadState_GetID (PyThreadState  *tstate)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Get the unique thread state identifier of the Python thread state @emph{tstate}.

@emph{tstate} must not be @code{NULL}.


New in version 3.9.

@end deffn

@geindex PyThreadState_GetInterpreter (C function)
@anchor{c-api/init c PyThreadState_GetInterpreter}@anchor{476}
@deffn {C Function} @ref{3a0,,PyInterpreterState}  *PyThreadState_GetInterpreter (PyThreadState  *tstate)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Get the interpreter of the Python thread state @emph{tstate}.

@emph{tstate} must not be @code{NULL}.


New in version 3.9.

@end deffn

@geindex PyThreadState_EnterTracing (C function)
@anchor{c-api/init c PyThreadState_EnterTracing}@anchor{1d7}
@deffn {C Function} void  PyThreadState_EnterTracing (PyThreadState  *tstate)

Suspend tracing and profiling in the Python thread state @emph{tstate}.

Resume them using the @ref{1d8,,PyThreadState_LeaveTracing()} function.


New in version 3.11.

@end deffn

@geindex PyThreadState_LeaveTracing (C function)
@anchor{c-api/init c PyThreadState_LeaveTracing}@anchor{1d8}
@deffn {C Function} void  PyThreadState_LeaveTracing (PyThreadState  *tstate)

Resume tracing and profiling in the Python thread state @emph{tstate} suspended
by the @ref{1d7,,PyThreadState_EnterTracing()} function.

See also @ref{1032,,PyEval_SetTrace()} and @ref{1031,,PyEval_SetProfile()}
functions.


New in version 3.11.

@end deffn

@geindex PyInterpreterState_Get (C function)
@anchor{c-api/init c PyInterpreterState_Get}@anchor{477}
@deffn {C Function} @ref{3a0,,PyInterpreterState}  *PyInterpreterState_Get (void)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.9.}
Get the current interpreter.

Issue a fatal error if there no current Python thread state or no current
interpreter. It cannot return NULL.

The caller must hold the GIL.


New in version 3.9.

@end deffn

@geindex PyInterpreterState_GetID (C function)
@anchor{c-api/init c PyInterpreterState_GetID}@anchor{6fd}
@deffn {C Function} int64_t  PyInterpreterState_GetID (PyInterpreterState  *interp)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Return the interpreter’s unique ID.  If there was any error in doing
so then @code{-1} is returned and an error is set.

The caller must hold the GIL.


New in version 3.7.

@end deffn

@geindex PyInterpreterState_GetDict (C function)
@anchor{c-api/init c PyInterpreterState_GetDict}@anchor{41aa}
@deffn {C Function} @ref{375,,PyObject}  *PyInterpreterState_GetDict (PyInterpreterState  *interp)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.8.}
Return a dictionary in which interpreter-specific data may be stored.
If this function returns @code{NULL} then no exception has been raised and
the caller should assume no interpreter-specific dict is available.

This is not a replacement for @ref{487,,PyModule_GetState()}, which
extensions should use to store interpreter-specific state information.


New in version 3.8.

@end deffn

@geindex _PyFrameEvalFunction (C type)
@anchor{c-api/init c _PyFrameEvalFunction}@anchor{121f}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*_PyFrameEvalFunction)(@ref{1fa,,PyThreadState}  *tstate,  @ref{1f6,,PyFrameObject}  *frame,  int  throwflag)

Type of a frame evaluation function.

The @emph{throwflag} parameter is used by the @code{throw()} method of generators:
if non-zero, handle the current exception.


Changed in version 3.9: The function now takes a @emph{tstate} parameter.

@end deffn

@geindex _PyInterpreterState_GetEvalFrameFunc (C function)
@anchor{c-api/init c _PyInterpreterState_GetEvalFrameFunc}@anchor{121d}
@deffn {C Function} @ref{121f,,_PyFrameEvalFunction}  _PyInterpreterState_GetEvalFrameFunc (PyInterpreterState  *interp)

Get the frame evaluation function.

See the 
@geindex Python Enhancement Proposals; PEP 523
PEP 523@footnote{https://www.python.org/dev/peps/pep-0523} “Adding a frame evaluation API to CPython”.


New in version 3.9.

@end deffn

@geindex _PyInterpreterState_SetEvalFrameFunc (C function)
@anchor{c-api/init c _PyInterpreterState_SetEvalFrameFunc}@anchor{121e}
@deffn {C Function} void  _PyInterpreterState_SetEvalFrameFunc (PyInterpreterState  *interp, _PyFrameEvalFunction  eval_frame)

Set the frame evaluation function.

See the 
@geindex Python Enhancement Proposals; PEP 523
PEP 523@footnote{https://www.python.org/dev/peps/pep-0523} “Adding a frame evaluation API to CPython”.


New in version 3.9.

@end deffn

@geindex PyThreadState_GetDict (C function)
@anchor{c-api/init c PyThreadState_GetDict}@anchor{423c}
@deffn {C Function} @ref{375,,PyObject}  *PyThreadState_GetDict ()
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Return a dictionary in which extensions can store thread-specific state
information.  Each extension should use a unique key to use to store state in
the dictionary.  It is okay to call this function when no current thread state
is available. If this function returns @code{NULL}, no exception has been raised and
the caller should assume no current thread state is available.
@end deffn

@geindex PyThreadState_SetAsyncExc (C function)
@anchor{c-api/init c PyThreadState_SetAsyncExc}@anchor{6fb}
@deffn {C Function} int  PyThreadState_SetAsyncExc (unsigned  long  id, PyObject  *exc)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Asynchronously raise an exception in a thread. The @emph{id} argument is the thread
id of the target thread; @emph{exc} is the exception object to be raised. This
function does not steal any references to @emph{exc}. To prevent naive misuse, you
must write your own C extension to call this.  Must be called with the GIL held.
Returns the number of thread states modified; this is normally one, but will be
zero if the thread id isn’t found.  If @emph{exc} is @code{NULL}, the pending
exception (if any) for the thread is cleared. This raises no exceptions.


Changed in version 3.7: The type of the @emph{id} parameter changed from long to
unsigned  long.

@end deffn

@geindex PyEval_AcquireThread (C function)
@anchor{c-api/init c PyEval_AcquireThread}@anchor{5a8}
@deffn {C Function} void  PyEval_AcquireThread (PyThreadState  *tstate)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Acquire the global interpreter lock and set the current thread state to
@emph{tstate}, which must not be @code{NULL}.  The lock must have been created earlier.
If this thread already has the lock, deadlock ensues.

@cartouche
@quotation Note 
Calling this function from a thread when the runtime is finalizing
will terminate the thread, even if the thread was not created by Python.
You can use @code{_Py_IsFinalizing()} or @ref{9e1,,sys.is_finalizing()} to
check if the interpreter is in process of being finalized before calling
this function to avoid unwanted termination.
@end quotation
@end cartouche


Changed in version 3.8: Updated to be consistent with @ref{5a9,,PyEval_RestoreThread()},
@ref{5aa,,Py_END_ALLOW_THREADS()}, and @ref{5ab,,PyGILState_Ensure()},
and terminate the current thread if called while the interpreter is finalizing.


@ref{5a9,,PyEval_RestoreThread()} is a higher-level function which is always
available (even when threads have not been initialized).
@end deffn

@geindex PyEval_ReleaseThread (C function)
@anchor{c-api/init c PyEval_ReleaseThread}@anchor{418d}
@deffn {C Function} void  PyEval_ReleaseThread (PyThreadState  *tstate)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Reset the current thread state to @code{NULL} and release the global interpreter
lock.  The lock must have been created earlier and must be held by the current
thread.  The @emph{tstate} argument, which must not be @code{NULL}, is only used to check
that it represents the current thread state — if it isn’t, a fatal error is
reported.

@ref{e2a,,PyEval_SaveThread()} is a higher-level function which is always
available (even when threads have not been initialized).
@end deffn

@geindex PyEval_AcquireLock (C function)
@anchor{c-api/init c PyEval_AcquireLock}@anchor{5a7}
@deffn {C Function} void  PyEval_AcquireLock ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Acquire the global interpreter lock.  The lock must have been created earlier.
If this thread already has the lock, a deadlock ensues.


Deprecated since version 3.2: This function does not update the current thread state.  Please use
@ref{5a9,,PyEval_RestoreThread()} or @ref{5a8,,PyEval_AcquireThread()}
instead.


@cartouche
@quotation Note 
Calling this function from a thread when the runtime is finalizing
will terminate the thread, even if the thread was not created by Python.
You can use @code{_Py_IsFinalizing()} or @ref{9e1,,sys.is_finalizing()} to
check if the interpreter is in process of being finalized before calling
this function to avoid unwanted termination.
@end quotation
@end cartouche


Changed in version 3.8: Updated to be consistent with @ref{5a9,,PyEval_RestoreThread()},
@ref{5aa,,Py_END_ALLOW_THREADS()}, and @ref{5ab,,PyGILState_Ensure()},
and terminate the current thread if called while the interpreter is finalizing.

@end deffn

@geindex PyEval_ReleaseLock (C function)
@anchor{c-api/init c PyEval_ReleaseLock}@anchor{e29}
@deffn {C Function} void  PyEval_ReleaseLock ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
Release the global interpreter lock.  The lock must have been created earlier.


Deprecated since version 3.2: This function does not update the current thread state.  Please use
@ref{e2a,,PyEval_SaveThread()} or @ref{418d,,PyEval_ReleaseThread()}
instead.

@end deffn

@node Sub-interpreter support,Asynchronous Notifications,Thread State and the Global Interpreter Lock,Initialization Finalization and Threads
@anchor{c-api/init id1}@anchor{450c}@anchor{c-api/init sub-interpreter-support}@anchor{42fb}
@subsection Sub-interpreter support


While in most uses, you will only embed a single Python interpreter, there
are cases where you need to create several independent interpreters in the
same process and perhaps even in the same thread. Sub-interpreters allow
you to do that.

The “main” interpreter is the first one created when the runtime initializes.
It is usually the only Python interpreter in a process.  Unlike sub-interpreters,
the main interpreter has unique process-global responsibilities like signal
handling.  It is also responsible for execution during runtime initialization and
is usually the active interpreter during runtime finalization.  The
@ref{450d,,PyInterpreterState_Main()} function returns a pointer to its state.

You can switch between sub-interpreters using the @ref{423e,,PyThreadState_Swap()}
function. You can create and destroy them using the following functions:

@geindex Py_NewInterpreter (C function)
@anchor{c-api/init c Py_NewInterpreter}@anchor{42a0}
@deffn {C Function} @ref{1fa,,PyThreadState}  *Py_NewInterpreter ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex module; builtins
@geindex module; __main__
@geindex module; sys
@geindex stdout (in module sys)
@geindex stderr (in module sys)
@geindex stdin (in module sys)

Create a new sub-interpreter.  This is an (almost) totally separate environment
for the execution of Python code.  In particular, the new interpreter has
separate, independent versions of all imported modules, including the
fundamental modules @ref{11,,builtins}, @ref{1,,__main__} and @ref{f7,,sys}.  The
table of loaded modules (@code{sys.modules}) and the module search path
(@code{sys.path}) are also separate.  The new environment has no @code{sys.argv}
variable.  It has new standard I/O stream file objects @code{sys.stdin},
@code{sys.stdout} and @code{sys.stderr} (however these refer to the same underlying
file descriptors).

The return value points to the first thread state created in the new
sub-interpreter.  This thread state is made in the current thread state.
Note that no actual thread is created; see the discussion of thread states
below.  If creation of the new interpreter is unsuccessful, @code{NULL} is
returned; no exception is set since the exception state is stored in the
current thread state and there may not be a current thread state.  (Like all
other Python/C API functions, the global interpreter lock must be held before
calling this function and is still held when it returns; however, unlike most
other Python/C API functions, there needn’t be a current thread state on
entry.)

@geindex Py_FinalizeEx()
@geindex Py_Initialize()

Extension modules are shared between (sub-)interpreters as follows:


@itemize *

@item 
For modules using multi-phase initialization,
e.g. @ref{a24,,PyModule_FromDefAndSpec()}, a separate module object is
created and initialized for each interpreter.
Only C-level static and global variables are shared between these
module objects.

@item 
For modules using single-phase initialization,
e.g. @ref{407a,,PyModule_Create()}, the first time a particular extension
is imported, it is initialized normally, and a (shallow) copy of its
module’s dictionary is squirreled away.
When the same extension is imported by another (sub-)interpreter, a new
module is initialized and filled with the contents of this copy; the
extension’s @code{init} function is not called.
Objects in the module’s dictionary thus end up shared across
(sub-)interpreters, which might cause unwanted behavior (see
@ref{450e,,Bugs and caveats} below).

Note that this is different from what happens when an extension is
imported after the interpreter has been completely re-initialized by
calling @ref{862,,Py_FinalizeEx()} and @ref{38d,,Py_Initialize()}; in that
case, the extension’s @code{initmodule} function @emph{is} called again.
As with multi-phase initialization, this means that only C-level static
and global variables are shared between these modules.
@end itemize

@geindex close() (in module os)
@end deffn

@geindex Py_EndInterpreter (C function)
@anchor{c-api/init c Py_EndInterpreter}@anchor{10b5}
@deffn {C Function} void  Py_EndInterpreter (PyThreadState  *tstate)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex Py_FinalizeEx()

Destroy the (sub-)interpreter represented by the given thread state. The given
thread state must be the current thread state.  See the discussion of thread
states below.  When the call returns, the current thread state is @code{NULL}.  All
thread states associated with this interpreter are destroyed.  (The global
interpreter lock must be held before calling this function and is still held
when it returns.)  @ref{862,,Py_FinalizeEx()} will destroy all sub-interpreters that
haven’t been explicitly destroyed at that point.
@end deffn

@menu
* Bugs and caveats:: 

@end menu

@node Bugs and caveats,,,Sub-interpreter support
@anchor{c-api/init bugs-and-caveats}@anchor{450e}
@subsubsection Bugs and caveats


Because sub-interpreters (and the main interpreter) are part of the same
process, the insulation between them isn’t perfect — for example, using
low-level file operations like  @ref{6a1,,os.close()} they can
(accidentally or maliciously) affect each other’s open files.  Because of the
way extensions are shared between (sub-)interpreters, some extensions may not
work properly; this is especially likely when using single-phase initialization
or (static) global variables.
It is possible to insert objects created in one sub-interpreter into
a namespace of another (sub-)interpreter; this should be avoided if possible.

Special care should be taken to avoid sharing user-defined functions,
methods, instances or classes between sub-interpreters, since import
operations executed by such objects may affect the wrong (sub-)interpreter’s
dictionary of loaded modules. It is equally important to avoid sharing
objects from which the above are reachable.

Also note that combining this functionality with PyGILState_* APIs
is delicate, because these APIs assume a bijection between Python thread states
and OS-level threads, an assumption broken by the presence of sub-interpreters.
It is highly recommended that you don’t switch sub-interpreters between a pair
of matching @ref{5ab,,PyGILState_Ensure()} and @ref{419c,,PyGILState_Release()} calls.
Furthermore, extensions (such as @ref{29,,ctypes}) using these APIs to allow calling
of Python code from non-Python created threads will probably be broken when using
sub-interpreters.

@node Asynchronous Notifications,Profiling and Tracing,Sub-interpreter support,Initialization Finalization and Threads
@anchor{c-api/init asynchronous-notifications}@anchor{450f}
@subsection Asynchronous Notifications


A mechanism is provided to make asynchronous notifications to the main
interpreter thread.  These notifications take the form of a function
pointer and a void pointer argument.

@geindex Py_AddPendingCall (C function)
@anchor{c-api/init c Py_AddPendingCall}@anchor{488}
@deffn {C Function} int  Py_AddPendingCall (int  (*func)(void*), void  *arg)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@geindex Py_AddPendingCall()

Schedule a function to be called from the main interpreter thread.  On
success, @code{0} is returned and @emph{func} is queued for being called in the
main thread.  On failure, @code{-1} is returned without setting any exception.

When successfully queued, @emph{func} will be @emph{eventually} called from the
main interpreter thread with the argument @emph{arg}.  It will be called
asynchronously with respect to normally running Python code, but with
both these conditions met:


@itemize *

@item 
on a @ref{8a8,,bytecode} boundary;

@item 
with the main thread holding the @ref{7ae,,global interpreter lock}
(@emph{func} can therefore use the full C API).
@end itemize

@emph{func} must return @code{0} on success, or @code{-1} on failure with an exception
set.  @emph{func} won’t be interrupted to perform another asynchronous
notification recursively, but it can still be interrupted to switch
threads if the global interpreter lock is released.

This function doesn’t need a current thread state to run, and it doesn’t
need the global interpreter lock.

To call this function in a subinterpreter, the caller must hold the GIL.
Otherwise, the function @emph{func} can be scheduled to be called from the wrong
interpreter.

@cartouche
@quotation Warning 
This is a low-level function, only useful for very special cases.
There is no guarantee that @emph{func} will be called as quick as
possible.  If the main thread is busy executing a system call,
@emph{func} won’t be called before the system call returns.  This
function is generally @strong{not} suitable for calling Python code from
arbitrary C threads.  Instead, use the @ref{4506,,PyGILState API}.
@end quotation
@end cartouche


Changed in version 3.9: If this function is called in a subinterpreter, the function @emph{func} is
now scheduled to be called from the subinterpreter, rather than being
called from the main interpreter. Each subinterpreter now has its own
list of scheduled calls.



New in version 3.1.

@end deffn

@node Profiling and Tracing,Advanced Debugger Support,Asynchronous Notifications,Initialization Finalization and Threads
@anchor{c-api/init profiling}@anchor{4510}@anchor{c-api/init profiling-and-tracing}@anchor{4511}
@subsection Profiling and Tracing


The Python interpreter provides some low-level support for attaching profiling
and execution tracing facilities.  These are used for profiling, debugging, and
coverage analysis tools.

This C interface allows the profiling or tracing code to avoid the overhead of
calling through Python-level callable objects, making a direct C function call
instead.  The essential attributes of the facility have not changed; the
interface allows trace functions to be installed per-thread, and the basic
events reported to the trace function are the same as had been reported to the
Python-level trace functions in previous versions.

@geindex Py_tracefunc (C type)
@anchor{c-api/init c Py_tracefunc}@anchor{4512}
@deffn {C Type} typedef  int  (*Py_tracefunc)(@ref{375,,PyObject}  *obj,  @ref{1f6,,PyFrameObject}  *frame,  int  what,  @ref{375,,PyObject}  *arg)

The type of the trace function registered using @ref{1031,,PyEval_SetProfile()} and
@ref{1032,,PyEval_SetTrace()}. The first parameter is the object passed to the
registration function as @emph{obj}, @emph{frame} is the frame object to which the event
pertains, @emph{what} is one of the constants @code{PyTrace_CALL},
@code{PyTrace_EXCEPTION}, @code{PyTrace_LINE}, @code{PyTrace_RETURN},
@code{PyTrace_C_CALL}, @code{PyTrace_C_EXCEPTION}, @code{PyTrace_C_RETURN},
or @code{PyTrace_OPCODE}, and @emph{arg} depends on the value of @emph{what}:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Value of @emph{what}

@tab

Meaning of @emph{arg}

@item

@code{PyTrace_CALL}

@tab

Always @ref{4077,,Py_None}.

@item

@code{PyTrace_EXCEPTION}

@tab

Exception information as returned by
@ref{18f,,sys.exc_info()}.

@item

@code{PyTrace_LINE}

@tab

Always @ref{4077,,Py_None}.

@item

@code{PyTrace_RETURN}

@tab

Value being returned to the caller,
or @code{NULL} if caused by an exception.

@item

@code{PyTrace_C_CALL}

@tab

Function object being called.

@item

@code{PyTrace_C_EXCEPTION}

@tab

Function object being called.

@item

@code{PyTrace_C_RETURN}

@tab

Function object being called.

@item

@code{PyTrace_OPCODE}

@tab

Always @ref{4077,,Py_None}.

@end multitable

@end deffn

@geindex PyTrace_CALL (C var)
@anchor{c-api/init c PyTrace_CALL}@anchor{4513}
@deffn {C Variable} int  PyTrace_CALL

The value of the @emph{what} parameter to a @ref{4512,,Py_tracefunc} function when a new
call to a function or method is being reported, or a new entry into a generator.
Note that the creation of the iterator for a generator function is not reported
as there is no control transfer to the Python bytecode in the corresponding
frame.
@end deffn

@geindex PyTrace_EXCEPTION (C var)
@anchor{c-api/init c PyTrace_EXCEPTION}@anchor{4514}
@deffn {C Variable} int  PyTrace_EXCEPTION

The value of the @emph{what} parameter to a @ref{4512,,Py_tracefunc} function when an
exception has been raised.  The callback function is called with this value for
@emph{what} when after any bytecode is processed after which the exception becomes
set within the frame being executed.  The effect of this is that as exception
propagation causes the Python stack to unwind, the callback is called upon
return to each frame as the exception propagates.  Only trace functions receives
these events; they are not needed by the profiler.
@end deffn

@geindex PyTrace_LINE (C var)
@anchor{c-api/init c PyTrace_LINE}@anchor{4515}
@deffn {C Variable} int  PyTrace_LINE

The value passed as the @emph{what} parameter to a @ref{4512,,Py_tracefunc} function
(but not a profiling function) when a line-number event is being reported.
It may be disabled for a frame by setting @code{f_trace_lines} to @emph{0} on that frame.
@end deffn

@geindex PyTrace_RETURN (C var)
@anchor{c-api/init c PyTrace_RETURN}@anchor{4516}
@deffn {C Variable} int  PyTrace_RETURN

The value for the @emph{what} parameter to @ref{4512,,Py_tracefunc} functions when a
call is about to return.
@end deffn

@geindex PyTrace_C_CALL (C var)
@anchor{c-api/init c PyTrace_C_CALL}@anchor{4517}
@deffn {C Variable} int  PyTrace_C_CALL

The value for the @emph{what} parameter to @ref{4512,,Py_tracefunc} functions when a C
function is about to be called.
@end deffn

@geindex PyTrace_C_EXCEPTION (C var)
@anchor{c-api/init c PyTrace_C_EXCEPTION}@anchor{4518}
@deffn {C Variable} int  PyTrace_C_EXCEPTION

The value for the @emph{what} parameter to @ref{4512,,Py_tracefunc} functions when a C
function has raised an exception.
@end deffn

@geindex PyTrace_C_RETURN (C var)
@anchor{c-api/init c PyTrace_C_RETURN}@anchor{4519}
@deffn {C Variable} int  PyTrace_C_RETURN

The value for the @emph{what} parameter to @ref{4512,,Py_tracefunc} functions when a C
function has returned.
@end deffn

@geindex PyTrace_OPCODE (C var)
@anchor{c-api/init c PyTrace_OPCODE}@anchor{451a}
@deffn {C Variable} int  PyTrace_OPCODE

The value for the @emph{what} parameter to @ref{4512,,Py_tracefunc} functions (but not
profiling functions) when a new opcode is about to be executed.  This event is
not emitted by default: it must be explicitly requested by setting
@code{f_trace_opcodes} to @emph{1} on the frame.
@end deffn

@geindex PyEval_SetProfile (C function)
@anchor{c-api/init c PyEval_SetProfile}@anchor{1031}
@deffn {C Function} void  PyEval_SetProfile (Py_tracefunc  func, PyObject  *obj)

Set the profiler function to @emph{func}.  The @emph{obj} parameter is passed to the
function as its first parameter, and may be any Python object, or @code{NULL}.  If
the profile function needs to maintain state, using a different value for @emph{obj}
for each thread provides a convenient and thread-safe place to store it.  The
profile function is called for all monitored events except @code{PyTrace_LINE}
@code{PyTrace_OPCODE} and @code{PyTrace_EXCEPTION}.

See also the @ref{1033,,sys.setprofile()} function.

The caller must hold the @ref{385,,GIL}.
@end deffn

@geindex PyEval_SetTrace (C function)
@anchor{c-api/init c PyEval_SetTrace}@anchor{1032}
@deffn {C Function} void  PyEval_SetTrace (Py_tracefunc  func, PyObject  *obj)

Set the tracing function to @emph{func}.  This is similar to
@ref{1031,,PyEval_SetProfile()}, except the tracing function does receive line-number
events and per-opcode events, but does not receive any event related to C function
objects being called.  Any trace function registered using @ref{1032,,PyEval_SetTrace()}
will not receive @code{PyTrace_C_CALL}, @code{PyTrace_C_EXCEPTION} or
@code{PyTrace_C_RETURN} as a value for the @emph{what} parameter.

See also the @ref{1034,,sys.settrace()} function.

The caller must hold the @ref{385,,GIL}.
@end deffn

@node Advanced Debugger Support,Thread Local Storage Support,Profiling and Tracing,Initialization Finalization and Threads
@anchor{c-api/init advanced-debugger-support}@anchor{451b}@anchor{c-api/init advanced-debugging}@anchor{451c}
@subsection Advanced Debugger Support


These functions are only intended to be used by advanced debugging tools.

@geindex PyInterpreterState_Head (C function)
@anchor{c-api/init c PyInterpreterState_Head}@anchor{1035}
@deffn {C Function} @ref{3a0,,PyInterpreterState}  *PyInterpreterState_Head ()

Return the interpreter state object at the head of the list of all such objects.
@end deffn

@geindex PyInterpreterState_Main (C function)
@anchor{c-api/init c PyInterpreterState_Main}@anchor{450d}
@deffn {C Function} @ref{3a0,,PyInterpreterState}  *PyInterpreterState_Main ()

Return the main interpreter state object.
@end deffn

@geindex PyInterpreterState_Next (C function)
@anchor{c-api/init c PyInterpreterState_Next}@anchor{1036}
@deffn {C Function} @ref{3a0,,PyInterpreterState}  *PyInterpreterState_Next (PyInterpreterState  *interp)

Return the next interpreter state object after @emph{interp} from the list of all
such objects.
@end deffn

@geindex PyInterpreterState_ThreadHead (C function)
@anchor{c-api/init c PyInterpreterState_ThreadHead}@anchor{1037}
@deffn {C Function} @ref{1fa,,PyThreadState}  *PyInterpreterState_ThreadHead (PyInterpreterState  *interp)

Return the pointer to the first @ref{1fa,,PyThreadState} object in the list of
threads associated with the interpreter @emph{interp}.
@end deffn

@geindex PyThreadState_Next (C function)
@anchor{c-api/init c PyThreadState_Next}@anchor{1038}
@deffn {C Function} @ref{1fa,,PyThreadState}  *PyThreadState_Next (PyThreadState  *tstate)

Return the next thread state object after @emph{tstate} from the list of all such
objects belonging to the same @ref{3a0,,PyInterpreterState} object.
@end deffn

@node Thread Local Storage Support,,Advanced Debugger Support,Initialization Finalization and Threads
@anchor{c-api/init thread-local-storage}@anchor{451d}@anchor{c-api/init thread-local-storage-support}@anchor{451e}
@subsection Thread Local Storage Support


The Python interpreter provides low-level support for thread-local storage
(TLS) which wraps the underlying native TLS implementation to support the
Python-level thread local storage API (@ref{278d,,threading.local}).  The
CPython C level APIs are similar to those offered by pthreads and Windows:
use a thread key and functions to associate a void* value per
thread.

The GIL does @emph{not} need to be held when calling these functions; they supply
their own locking.

Note that @code{Python.h} does not include the declaration of the TLS APIs,
you need to include @code{pythread.h} to use thread-local storage.

@cartouche
@quotation Note 
None of these API functions handle memory management on behalf of the
void* values.  You need to allocate and deallocate them yourself.
If the void* values happen to be @ref{375,,PyObject}*, these
functions don’t do refcount operations on them either.
@end quotation
@end cartouche

@menu
* Thread Specific Storage (TSS) API: Thread Specific Storage TSS API. 
* Thread Local Storage (TLS) API: Thread Local Storage TLS API. 

@end menu

@node Thread Specific Storage TSS API,Thread Local Storage TLS API,,Thread Local Storage Support
@anchor{c-api/init thread-specific-storage-api}@anchor{5f8}@anchor{c-api/init thread-specific-storage-tss-api}@anchor{451f}
@subsubsection Thread Specific Storage (TSS) API


TSS API is introduced to supersede the use of the existing TLS API within the
CPython interpreter.  This API uses a new type @ref{5f9,,Py_tss_t} instead of
int to represent thread keys.


New in version 3.7.



@subsubheading See also


“A New C-API for Thread-Local Storage in CPython” (
@geindex Python Enhancement Proposals; PEP 539
PEP 539@footnote{https://www.python.org/dev/peps/pep-0539})


@geindex Py_tss_t (C type)
@anchor{c-api/init c Py_tss_t}@anchor{5f9}
@deffn {C Type} type  Py_tss_t

This data structure represents the state of a thread key, the definition of
which may depend on the underlying TLS implementation, and it has an
internal field representing the key’s initialization state.  There are no
public members in this structure.

When @ref{35d,,Py_LIMITED_API} is not defined, static allocation of
this type by @ref{4520,,Py_tss_NEEDS_INIT} is allowed.
@end deffn

@geindex Py_tss_NEEDS_INIT (C macro)
@anchor{c-api/init c Py_tss_NEEDS_INIT}@anchor{4520}
@deffn {C Macro} Py_tss_NEEDS_INIT

This macro expands to the initializer for @ref{5f9,,Py_tss_t} variables.
Note that this macro won’t be defined with @ref{35d,,Py_LIMITED_API}.
@end deffn

@menu
* Dynamic Allocation:: 
* Methods: Methods<4>. 

@end menu

@node Dynamic Allocation,Methods<4>,,Thread Specific Storage TSS API
@anchor{c-api/init dynamic-allocation}@anchor{4521}
@subsubsection Dynamic Allocation


Dynamic allocation of the @ref{5f9,,Py_tss_t}, required in extension modules
built with @ref{35d,,Py_LIMITED_API}, where static allocation of this type
is not possible due to its implementation being opaque at build time.

@geindex PyThread_tss_alloc (C function)
@anchor{c-api/init c PyThread_tss_alloc}@anchor{4244}
@deffn {C Function} @ref{5f9,,Py_tss_t}  *PyThread_tss_alloc ()
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Return a value which is the same state as a value initialized with
@ref{4520,,Py_tss_NEEDS_INIT}, or @code{NULL} in the case of dynamic allocation
failure.
@end deffn

@geindex PyThread_tss_free (C function)
@anchor{c-api/init c PyThread_tss_free}@anchor{4247}
@deffn {C Function} void  PyThread_tss_free (Py_tss_t  *key)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Free the given @emph{key} allocated by @ref{4244,,PyThread_tss_alloc()}, after
first calling @ref{4246,,PyThread_tss_delete()} to ensure any associated
thread locals have been unassigned. This is a no-op if the @emph{key}
argument is @cite{NULL}.

@cartouche
@quotation Note 
A freed key becomes a dangling pointer, you should reset the key to
@cite{NULL}.
@end quotation
@end cartouche
@end deffn

@node Methods<4>,,Dynamic Allocation,Thread Specific Storage TSS API
@anchor{c-api/init methods}@anchor{4522}
@subsubsection Methods


The parameter @emph{key} of these functions must not be @code{NULL}.  Moreover, the
behaviors of @ref{424a,,PyThread_tss_set()} and @ref{4248,,PyThread_tss_get()} are
undefined if the given @ref{5f9,,Py_tss_t} has not been initialized by
@ref{4245,,PyThread_tss_create()}.

@geindex PyThread_tss_is_created (C function)
@anchor{c-api/init c PyThread_tss_is_created}@anchor{4249}
@deffn {C Function} int  PyThread_tss_is_created (Py_tss_t  *key)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Return a non-zero value if the given @ref{5f9,,Py_tss_t} has been initialized
by @ref{4245,,PyThread_tss_create()}.
@end deffn

@geindex PyThread_tss_create (C function)
@anchor{c-api/init c PyThread_tss_create}@anchor{4245}
@deffn {C Function} int  PyThread_tss_create (Py_tss_t  *key)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Return a zero value on successful initialization of a TSS key.  The behavior
is undefined if the value pointed to by the @emph{key} argument is not
initialized by @ref{4520,,Py_tss_NEEDS_INIT}.  This function can be called
repeatedly on the same key – calling it on an already initialized key is a
no-op and immediately returns success.
@end deffn

@geindex PyThread_tss_delete (C function)
@anchor{c-api/init c PyThread_tss_delete}@anchor{4246}
@deffn {C Function} void  PyThread_tss_delete (Py_tss_t  *key)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Destroy a TSS key to forget the values associated with the key across all
threads, and change the key’s initialization state to uninitialized.  A
destroyed key is able to be initialized again by
@ref{4245,,PyThread_tss_create()}. This function can be called repeatedly on
the same key – calling it on an already destroyed key is a no-op.
@end deffn

@geindex PyThread_tss_set (C function)
@anchor{c-api/init c PyThread_tss_set}@anchor{424a}
@deffn {C Function} int  PyThread_tss_set (Py_tss_t  *key, void  *value)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Return a zero value to indicate successfully associating a void*
value with a TSS key in the current thread.  Each thread has a distinct
mapping of the key to a void* value.
@end deffn

@geindex PyThread_tss_get (C function)
@anchor{c-api/init c PyThread_tss_get}@anchor{4248}
@deffn {C Function} void  *PyThread_tss_get (Py_tss_t  *key)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Return the void* value associated with a TSS key in the current
thread.  This returns @code{NULL} if no value is associated with the key in the
current thread.
@end deffn

@node Thread Local Storage TLS API,,Thread Specific Storage TSS API,Thread Local Storage Support
@anchor{c-api/init thread-local-storage-api}@anchor{5f7}@anchor{c-api/init thread-local-storage-tls-api}@anchor{4523}
@subsubsection Thread Local Storage (TLS) API



Deprecated since version 3.7: This API is superseded by
@ref{5f8,,Thread Specific Storage (TSS) API}.


@cartouche
@quotation Note 
This version of the API does not support platforms where the native TLS key
is defined in a way that cannot be safely cast to @code{int}.  On such platforms,
@ref{4240,,PyThread_create_key()} will return immediately with a failure status,
and the other TLS functions will all be no-ops on such platforms.
@end quotation
@end cartouche

Due to the compatibility problem noted above, this version of the API should not
be used in new code.

@geindex PyThread_create_key (C function)
@anchor{c-api/init c PyThread_create_key}@anchor{4240}
@deffn {C Function} int  PyThread_create_key ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex PyThread_delete_key (C function)
@anchor{c-api/init c PyThread_delete_key}@anchor{4241}
@deffn {C Function} void  PyThread_delete_key (int  key)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex PyThread_set_key_value (C function)
@anchor{c-api/init c PyThread_set_key_value}@anchor{bc7}
@deffn {C Function} int  PyThread_set_key_value (int  key, void  *value)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex PyThread_get_key_value (C function)
@anchor{c-api/init c PyThread_get_key_value}@anchor{4243}
@deffn {C Function} void  *PyThread_get_key_value (int  key)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex PyThread_delete_key_value (C function)
@anchor{c-api/init c PyThread_delete_key_value}@anchor{4242}
@deffn {C Function} void  PyThread_delete_key_value (int  key)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex PyThread_ReInitTLS (C function)
@anchor{c-api/init c PyThread_ReInitTLS}@anchor{423f}
@deffn {C Function} void  PyThread_ReInitTLS ()
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@node Python Initialization Configuration,Memory Management,Initialization Finalization and Threads,Python/C API Reference Manual
@anchor{c-api/init_config doc}@anchor{4524}@anchor{c-api/init_config init-config}@anchor{207}@anchor{c-api/init_config python-initialization-configuration}@anchor{4525}
@section Python Initialization Configuration



New in version 3.8.


Python can be initialized with @ref{4b7,,Py_InitializeFromConfig()} and the
@ref{206,,PyConfig} structure. It can be preinitialized with
@ref{4b8,,Py_PreInitialize()} and the @ref{4b3,,PyPreConfig} structure.

There are two kinds of configuration:


@itemize *

@item 
The @ref{4526,,Python Configuration} can be used to build a
customized Python which behaves as the regular Python. For example,
environments variables and command line arguments are used to configure
Python.

@item 
The @ref{4527,,Isolated Configuration} can be used to embed
Python into an application. It isolates Python from the system. For example,
environments variables are ignored, the LC_CTYPE locale is left unchanged and
no signal handler is registered.
@end itemize

The @ref{4bb,,Py_RunMain()} function can be used to write a customized Python
program.

See also @ref{44e8,,Initialization@comma{} Finalization@comma{} and Threads}.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 587
PEP 587@footnote{https://www.python.org/dev/peps/pep-0587} “Python Initialization Configuration”.


@menu
* Example: Example<17>. 
* PyWideStringList:: 
* PyStatus:: 
* PyPreConfig:: 
* Preinitialize Python with PyPreConfig:: 
* PyConfig:: 
* Initialization with PyConfig:: 
* Isolated Configuration:: 
* Python Configuration:: 
* Python Path Configuration:: 
* Py_RunMain(): Py_RunMain. 
* Py_GetArgcArgv(): Py_GetArgcArgv. 
* Multi-Phase Initialization Private Provisional API:: 

@end menu

@node Example<17>,PyWideStringList,,Python Initialization Configuration
@anchor{c-api/init_config example}@anchor{4528}
@subsection Example


Example of customized Python always running in isolated mode:

@example
int main(int argc, char **argv)
@{
    PyStatus status;

    PyConfig config;
    PyConfig_InitPythonConfig(&config);
    config.isolated = 1;

    /* Decode command line arguments.
       Implicitly preinitialize Python (in isolated mode). */
    status = PyConfig_SetBytesArgv(&config, argc, argv);
    if (PyStatus_Exception(status)) @{
        goto exception;
    @}

    status = Py_InitializeFromConfig(&config);
    if (PyStatus_Exception(status)) @{
        goto exception;
    @}
    PyConfig_Clear(&config);

    return Py_RunMain();

exception:
    PyConfig_Clear(&config);
    if (PyStatus_IsExit(status)) @{
        return status.exitcode;
    @}
    /* Display the error message and exit the process with
       non-zero exit code */
    Py_ExitStatusException(status);
@}
@end example

@node PyWideStringList,PyStatus,Example<17>,Python Initialization Configuration
@anchor{c-api/init_config pywidestringlist}@anchor{4529}
@subsection PyWideStringList


@geindex PyWideStringList (C type)
@anchor{c-api/init_config c PyWideStringList}@anchor{4b5}
@deffn {C Type} type  PyWideStringList

List of @code{wchar_t*} strings.

If @emph{length} is non-zero, @emph{items} must be non-@code{NULL} and all strings must be
non-@code{NULL}.

Methods:

@geindex PyWideStringList.PyWideStringList_Append (C function)
@anchor{c-api/init_config c PyWideStringList PyWideStringList_Append}@anchor{452a}
@deffn {C Function} @ref{4b4,,PyStatus}  PyWideStringList_Append (PyWideStringList  *list, const  wchar_t  *item)

Append @emph{item} to @emph{list}.

Python must be preinitialized to call this function.
@end deffn

@geindex PyWideStringList.PyWideStringList_Insert (C function)
@anchor{c-api/init_config c PyWideStringList PyWideStringList_Insert}@anchor{452b}
@deffn {C Function} @ref{4b4,,PyStatus}  PyWideStringList_Insert (PyWideStringList  *list, Py_ssize_t  index, const  wchar_t  *item)

Insert @emph{item} into @emph{list} at @emph{index}.

If @emph{index} is greater than or equal to @emph{list} length, append @emph{item} to
@emph{list}.

@emph{index} must be greater than or equal to 0.

Python must be preinitialized to call this function.
@end deffn

Structure fields:

@geindex PyWideStringList.length (C member)
@anchor{c-api/init_config c PyWideStringList length}@anchor{452c}
@deffn {C Member} Py_ssize_t  length

List length.
@end deffn

@geindex PyWideStringList.items (C member)
@anchor{c-api/init_config c PyWideStringList items}@anchor{452d}
@deffn {C Member} wchar_t  **items

List items.
@end deffn
@end deffn

@node PyStatus,PyPreConfig,PyWideStringList,Python Initialization Configuration
@anchor{c-api/init_config pystatus}@anchor{452e}
@subsection PyStatus


@geindex PyStatus (C type)
@anchor{c-api/init_config c PyStatus}@anchor{4b4}
@deffn {C Type} type  PyStatus

Structure to store an initialization function status: success, error
or exit.

For an error, it can store the C function name which created the error.

Structure fields:

@geindex PyStatus.exitcode (C member)
@anchor{c-api/init_config c PyStatus exitcode}@anchor{452f}
@deffn {C Member} int  exitcode

Exit code. Argument passed to @code{exit()}.
@end deffn

@geindex PyStatus.err_msg (C member)
@anchor{c-api/init_config c PyStatus err_msg}@anchor{4530}
@deffn {C Member} const  char  *err_msg

Error message.
@end deffn

@geindex PyStatus.func (C member)
@anchor{c-api/init_config c PyStatus func}@anchor{4531}
@deffn {C Member} const  char  *func

Name of the function which created an error, can be @code{NULL}.
@end deffn

Functions to create a status:

@geindex PyStatus.PyStatus_Ok (C function)
@anchor{c-api/init_config c PyStatus PyStatus_Ok}@anchor{4532}
@deffn {C Function} @ref{4b4,,PyStatus}  PyStatus_Ok (void)

Success.
@end deffn

@geindex PyStatus.PyStatus_Error (C function)
@anchor{c-api/init_config c PyStatus PyStatus_Error}@anchor{4533}
@deffn {C Function} @ref{4b4,,PyStatus}  PyStatus_Error (const  char  *err_msg)

Initialization error with a message.

@emph{err_msg} must not be @code{NULL}.
@end deffn

@geindex PyStatus.PyStatus_NoMemory (C function)
@anchor{c-api/init_config c PyStatus PyStatus_NoMemory}@anchor{4534}
@deffn {C Function} @ref{4b4,,PyStatus}  PyStatus_NoMemory (void)

Memory allocation failure (out of memory).
@end deffn

@geindex PyStatus.PyStatus_Exit (C function)
@anchor{c-api/init_config c PyStatus PyStatus_Exit}@anchor{4535}
@deffn {C Function} @ref{4b4,,PyStatus}  PyStatus_Exit (int  exitcode)

Exit Python with the specified exit code.
@end deffn

Functions to handle a status:

@geindex PyStatus.PyStatus_Exception (C function)
@anchor{c-api/init_config c PyStatus PyStatus_Exception}@anchor{4536}
@deffn {C Function} int  PyStatus_Exception (PyStatus  status)

Is the status an error or an exit? If true, the exception must be
handled; by calling @ref{4537,,Py_ExitStatusException()} for example.
@end deffn

@geindex PyStatus.PyStatus_IsError (C function)
@anchor{c-api/init_config c PyStatus PyStatus_IsError}@anchor{4538}
@deffn {C Function} int  PyStatus_IsError (PyStatus  status)

Is the result an error?
@end deffn

@geindex PyStatus.PyStatus_IsExit (C function)
@anchor{c-api/init_config c PyStatus PyStatus_IsExit}@anchor{4539}
@deffn {C Function} int  PyStatus_IsExit (PyStatus  status)

Is the result an exit?
@end deffn

@geindex PyStatus.Py_ExitStatusException (C function)
@anchor{c-api/init_config c PyStatus Py_ExitStatusException}@anchor{4537}
@deffn {C Function} void  Py_ExitStatusException (PyStatus  status)

Call @code{exit(exitcode)} if @emph{status} is an exit. Print the error
message and exit with a non-zero exit code if @emph{status} is an error.  Must
only be called if @code{PyStatus_Exception(status)} is non-zero.
@end deffn
@end deffn

@cartouche
@quotation Note 
Internally, Python uses macros which set @code{PyStatus.func},
whereas functions to create a status set @code{func} to @code{NULL}.
@end quotation
@end cartouche

Example:

@example
PyStatus alloc(void **ptr, size_t size)
@{
    *ptr = PyMem_RawMalloc(size);
    if (*ptr == NULL) @{
        return PyStatus_NoMemory();
    @}
    return PyStatus_Ok();
@}

int main(int argc, char **argv)
@{
    void *ptr;
    PyStatus status = alloc(&ptr, 16);
    if (PyStatus_Exception(status)) @{
        Py_ExitStatusException(status);
    @}
    PyMem_Free(ptr);
    return 0;
@}
@end example

@node PyPreConfig,Preinitialize Python with PyPreConfig,PyStatus,Python Initialization Configuration
@anchor{c-api/init_config pypreconfig}@anchor{453a}
@subsection PyPreConfig


@geindex PyPreConfig (C type)
@anchor{c-api/init_config c PyPreConfig}@anchor{4b3}
@deffn {C Type} type  PyPreConfig

Structure used to preinitialize Python.

Function to initialize a preconfiguration:

@geindex PyPreConfig.PyPreConfig_InitPythonConfig (C function)
@anchor{c-api/init_config c PyPreConfig PyPreConfig_InitPythonConfig}@anchor{453b}
@deffn {C Function} void  PyPreConfig_InitPythonConfig (PyPreConfig  *preconfig)

Initialize the preconfiguration with @ref{4526,,Python Configuration}.
@end deffn

@geindex PyPreConfig.PyPreConfig_InitIsolatedConfig (C function)
@anchor{c-api/init_config c PyPreConfig PyPreConfig_InitIsolatedConfig}@anchor{453c}
@deffn {C Function} void  PyPreConfig_InitIsolatedConfig (PyPreConfig  *preconfig)

Initialize the preconfiguration with @ref{4527,,Isolated Configuration}.
@end deffn

Structure fields:

@geindex PyPreConfig.allocator (C member)
@anchor{c-api/init_config c PyPreConfig allocator}@anchor{453d}
@deffn {C Member} int  allocator

Name of the Python memory allocators:


@itemize *

@item 
@code{PYMEM_ALLOCATOR_NOT_SET} (@code{0}): don’t change memory allocators
(use defaults).

@item 
@code{PYMEM_ALLOCATOR_DEFAULT} (@code{1}): @ref{1626,,default memory allocators}.

@item 
@code{PYMEM_ALLOCATOR_DEBUG} (@code{2}): @ref{1626,,default memory allocators} with @ref{1627,,debug hooks}.

@item 
@code{PYMEM_ALLOCATOR_MALLOC} (@code{3}): use @code{malloc()} of the C library.

@item 
@code{PYMEM_ALLOCATOR_MALLOC_DEBUG} (@code{4}): force usage of
@code{malloc()} with @ref{1627,,debug hooks}.

@item 
@code{PYMEM_ALLOCATOR_PYMALLOC} (@code{5}): @ref{85c,,Python pymalloc memory allocator}.

@item 
@code{PYMEM_ALLOCATOR_PYMALLOC_DEBUG} (@code{6}): @ref{85c,,Python pymalloc memory allocator} with @ref{1627,,debug hooks}.
@end itemize

@code{PYMEM_ALLOCATOR_PYMALLOC} and @code{PYMEM_ALLOCATOR_PYMALLOC_DEBUG} are
not supported if Python is @ref{1656,,configured using --without-pymalloc}.

See @ref{453e,,Memory Management}.

Default: @code{PYMEM_ALLOCATOR_NOT_SET}.
@end deffn

@geindex PyPreConfig.configure_locale (C member)
@anchor{c-api/init_config c PyPreConfig configure_locale}@anchor{453f}
@deffn {C Member} int  configure_locale

Set the LC_CTYPE locale to the user preferred locale?

If equals to 0, set @ref{4540,,coerce_c_locale} and
@ref{4541,,coerce_c_locale_warn} members to 0.

See the @ref{237a,,locale encoding}.

Default: @code{1} in Python config, @code{0} in isolated config.
@end deffn

@geindex PyPreConfig.coerce_c_locale (C member)
@anchor{c-api/init_config c PyPreConfig coerce_c_locale}@anchor{4540}
@deffn {C Member} int  coerce_c_locale

If equals to 2, coerce the C locale.

If equals to 1, read the LC_CTYPE locale to decide if it should be
coerced.

See the @ref{237a,,locale encoding}.

Default: @code{-1} in Python config, @code{0} in isolated config.
@end deffn

@geindex PyPreConfig.coerce_c_locale_warn (C member)
@anchor{c-api/init_config c PyPreConfig coerce_c_locale_warn}@anchor{4541}
@deffn {C Member} int  coerce_c_locale_warn

If non-zero, emit a warning if the C locale is coerced.

Default: @code{-1} in Python config, @code{0} in isolated config.
@end deffn

@geindex PyPreConfig.dev_mode (C member)
@anchor{c-api/init_config c PyPreConfig dev_mode}@anchor{4542}
@deffn {C Member} int  dev_mode

If non-zero, enables the @ref{3ac,,Python Development Mode}:
see @ref{4543,,PyConfig.dev_mode}.

Default: @code{-1} in Python mode, @code{0} in isolated mode.
@end deffn

@geindex PyPreConfig.isolated (C member)
@anchor{c-api/init_config c PyPreConfig isolated}@anchor{4544}
@deffn {C Member} int  isolated

Isolated mode: see @ref{4545,,PyConfig.isolated}.

Default: @code{0} in Python mode, @code{1} in isolated mode.
@end deffn

@geindex PyPreConfig.legacy_windows_fs_encoding (C member)
@anchor{c-api/init_config c PyPreConfig legacy_windows_fs_encoding}@anchor{4546}
@deffn {C Member} int  legacy_windows_fs_encoding

If non-zero:


@itemize *

@item 
Set @ref{4547,,PyPreConfig.utf8_mode} to @code{0},

@item 
Set @ref{2378,,PyConfig.filesystem_encoding} to @code{"mbcs"},

@item 
Set @ref{2379,,PyConfig.filesystem_errors} to @code{"replace"}.
@end itemize

Initialized the from 
@geindex PYTHONLEGACYWINDOWSFSENCODING
@geindex environment variable; PYTHONLEGACYWINDOWSFSENCODING
@ref{79f,,PYTHONLEGACYWINDOWSFSENCODING} environment
variable value.

Only available on Windows. @code{#ifdef MS_WINDOWS} macro can be used for
Windows specific code.

Default: @code{0}.
@end deffn

@geindex PyPreConfig.parse_argv (C member)
@anchor{c-api/init_config c PyPreConfig parse_argv}@anchor{4548}
@deffn {C Member} int  parse_argv

If non-zero, @ref{4b9,,Py_PreInitializeFromArgs()} and
@ref{4ba,,Py_PreInitializeFromBytesArgs()} parse their @code{argv} argument the
same way the regular Python parses command line arguments: see
@ref{b8d,,Command Line Arguments}.

Default: @code{1} in Python config, @code{0} in isolated config.
@end deffn

@geindex PyPreConfig.use_environment (C member)
@anchor{c-api/init_config c PyPreConfig use_environment}@anchor{4549}
@deffn {C Member} int  use_environment

Use @ref{161c,,environment variables}? See
@ref{48a,,PyConfig.use_environment}.

Default: @code{1} in Python config and @code{0} in isolated config.
@end deffn

@geindex PyPreConfig.utf8_mode (C member)
@anchor{c-api/init_config c PyPreConfig utf8_mode}@anchor{4547}
@deffn {C Member} int  utf8_mode

If non-zero, enable the @ref{5f2,,Python UTF-8 Mode}.

Set by the @ref{159,,-X utf8} command line option and the
@geindex PYTHONUTF8
@geindex environment variable; PYTHONUTF8
@ref{5f1,,PYTHONUTF8} environment variable.

Default: @code{-1} in Python config and @code{0} in isolated config.
@end deffn
@end deffn

@node Preinitialize Python with PyPreConfig,PyConfig,PyPreConfig,Python Initialization Configuration
@anchor{c-api/init_config c-preinit}@anchor{3502}@anchor{c-api/init_config preinitialize-python-with-pypreconfig}@anchor{454a}
@subsection Preinitialize Python with PyPreConfig


The preinitialization of Python:


@itemize *

@item 
Set the Python memory allocators (@ref{453d,,PyPreConfig.allocator})

@item 
Configure the LC_CTYPE locale (@ref{237a,,locale encoding})

@item 
Set the @ref{5f2,,Python UTF-8 Mode}
(@ref{4547,,PyPreConfig.utf8_mode})
@end itemize

The current preconfiguration (@code{PyPreConfig} type) is stored in
@code{_PyRuntime.preconfig}.

Functions to preinitialize Python:

@geindex Py_PreInitialize (C function)
@anchor{c-api/init_config c Py_PreInitialize}@anchor{4b8}
@deffn {C Function} @ref{4b4,,PyStatus}  Py_PreInitialize (const  PyPreConfig  *preconfig)

Preinitialize Python from @emph{preconfig} preconfiguration.

@emph{preconfig} must not be @code{NULL}.
@end deffn

@geindex Py_PreInitializeFromBytesArgs (C function)
@anchor{c-api/init_config c Py_PreInitializeFromBytesArgs}@anchor{4ba}
@deffn {C Function} @ref{4b4,,PyStatus}  Py_PreInitializeFromBytesArgs (const  PyPreConfig  *preconfig, int  argc, char  *const  *argv)

Preinitialize Python from @emph{preconfig} preconfiguration.

Parse @emph{argv} command line arguments (bytes strings) if
@ref{4548,,parse_argv} of @emph{preconfig} is non-zero.

@emph{preconfig} must not be @code{NULL}.
@end deffn

@geindex Py_PreInitializeFromArgs (C function)
@anchor{c-api/init_config c Py_PreInitializeFromArgs}@anchor{4b9}
@deffn {C Function} @ref{4b4,,PyStatus}  Py_PreInitializeFromArgs (const  PyPreConfig  *preconfig, int  argc, wchar_t  *const  *argv)

Preinitialize Python from @emph{preconfig} preconfiguration.

Parse @emph{argv} command line arguments (wide strings) if
@ref{4548,,parse_argv} of @emph{preconfig} is non-zero.

@emph{preconfig} must not be @code{NULL}.
@end deffn

The caller is responsible to handle exceptions (error or exit) using
@code{PyStatus_Exception()} and @code{Py_ExitStatusException()}.

For @ref{4526,,Python Configuration}
(@code{PyPreConfig_InitPythonConfig()}), if Python is initialized with
command line arguments, the command line arguments must also be passed to
preinitialize Python, since they have an effect on the pre-configuration
like encodings. For example, the @ref{159,,-X utf8} command line option
enables the @ref{5f2,,Python UTF-8 Mode}.

@code{PyMem_SetAllocator()} can be called after @ref{4b8,,Py_PreInitialize()} and
before @ref{4b7,,Py_InitializeFromConfig()} to install a custom memory allocator.
It can be called before @ref{4b8,,Py_PreInitialize()} if
@ref{453d,,PyPreConfig.allocator} is set to @code{PYMEM_ALLOCATOR_NOT_SET}.

Python memory allocation functions like @ref{bc4,,PyMem_RawMalloc()} must not be
used before the Python preinitialization, whereas calling directly @code{malloc()}
and @code{free()} is always safe. @ref{6fe,,Py_DecodeLocale()} must not be called
before the Python preinitialization.

Example using the preinitialization to enable
the @ref{5f2,,Python UTF-8 Mode}:

@example
PyStatus status;
PyPreConfig preconfig;
PyPreConfig_InitPythonConfig(&preconfig);

preconfig.utf8_mode = 1;

status = Py_PreInitialize(&preconfig);
if (PyStatus_Exception(status)) @{
    Py_ExitStatusException(status);
@}

/* at this point, Python speaks UTF-8 */

Py_Initialize();
/* ... use Python API here ... */
Py_Finalize();
@end example

@node PyConfig,Initialization with PyConfig,Preinitialize Python with PyPreConfig,Python Initialization Configuration
@anchor{c-api/init_config pyconfig}@anchor{454b}
@subsection PyConfig


@geindex PyConfig (C type)
@anchor{c-api/init_config c PyConfig}@anchor{206}
@deffn {C Type} type  PyConfig

Structure containing most parameters to configure Python.

When done, the @ref{454c,,PyConfig_Clear()} function must be used to release the
configuration memory.

Structure methods:

@geindex PyConfig.PyConfig_InitPythonConfig (C function)
@anchor{c-api/init_config c PyConfig PyConfig_InitPythonConfig}@anchor{454d}
@deffn {C Function} void  PyConfig_InitPythonConfig (PyConfig  *config)

Initialize configuration with the @ref{4526,,Python Configuration}.
@end deffn

@geindex PyConfig.PyConfig_InitIsolatedConfig (C function)
@anchor{c-api/init_config c PyConfig PyConfig_InitIsolatedConfig}@anchor{454e}
@deffn {C Function} void  PyConfig_InitIsolatedConfig (PyConfig  *config)

Initialize configuration with the @ref{4527,,Isolated Configuration}.
@end deffn

@geindex PyConfig.PyConfig_SetString (C function)
@anchor{c-api/init_config c PyConfig PyConfig_SetString}@anchor{454f}
@deffn {C Function} @ref{4b4,,PyStatus}  PyConfig_SetString (PyConfig  *config, wchar_t  *const  *config_str, const  wchar_t  *str)

Copy the wide character string @emph{str} into @code{*config_str}.

@ref{3502,,Preinitialize Python} if needed.
@end deffn

@geindex PyConfig.PyConfig_SetBytesString (C function)
@anchor{c-api/init_config c PyConfig PyConfig_SetBytesString}@anchor{4550}
@deffn {C Function} @ref{4b4,,PyStatus}  PyConfig_SetBytesString (PyConfig  *config, wchar_t  *const  *config_str, const  char  *str)

Decode @emph{str} using @ref{6fe,,Py_DecodeLocale()} and set the result into
@code{*config_str}.

@ref{3502,,Preinitialize Python} if needed.
@end deffn

@geindex PyConfig.PyConfig_SetArgv (C function)
@anchor{c-api/init_config c PyConfig PyConfig_SetArgv}@anchor{4551}
@deffn {C Function} @ref{4b4,,PyStatus}  PyConfig_SetArgv (PyConfig  *config, int  argc, wchar_t  *const  *argv)

Set command line arguments (@ref{11a1,,argv} member of
@emph{config}) from the @emph{argv} list of wide character strings.

@ref{3502,,Preinitialize Python} if needed.
@end deffn

@geindex PyConfig.PyConfig_SetBytesArgv (C function)
@anchor{c-api/init_config c PyConfig PyConfig_SetBytesArgv}@anchor{4552}
@deffn {C Function} @ref{4b4,,PyStatus}  PyConfig_SetBytesArgv (PyConfig  *config, int  argc, char  *const  *argv)

Set command line arguments (@ref{11a1,,argv} member of
@emph{config}) from the @emph{argv} list of bytes strings. Decode bytes using
@ref{6fe,,Py_DecodeLocale()}.

@ref{3502,,Preinitialize Python} if needed.
@end deffn

@geindex PyConfig.PyConfig_SetWideStringList (C function)
@anchor{c-api/init_config c PyConfig PyConfig_SetWideStringList}@anchor{4553}
@deffn {C Function} @ref{4b4,,PyStatus}  PyConfig_SetWideStringList (PyConfig  *config, PyWideStringList  *list, Py_ssize_t  length, wchar_t  **items)

Set the list of wide strings @emph{list} to @emph{length} and @emph{items}.

@ref{3502,,Preinitialize Python} if needed.
@end deffn

@geindex PyConfig.PyConfig_Read (C function)
@anchor{c-api/init_config c PyConfig PyConfig_Read}@anchor{4554}
@deffn {C Function} @ref{4b4,,PyStatus}  PyConfig_Read (PyConfig  *config)

Read all Python configuration.

Fields which are already initialized are left unchanged.

Fields for @ref{38e,,path configuration} are no longer
calculated or modified when calling this function, as of Python 3.11.

The @ref{4554,,PyConfig_Read()} function only parses
@ref{11a1,,PyConfig.argv} arguments once: @ref{11a2,,PyConfig.parse_argv}
is set to @code{2} after arguments are parsed. Since Python arguments are
strippped from @ref{11a1,,PyConfig.argv}, parsing arguments twice would
parse the application options as Python options.

@ref{3502,,Preinitialize Python} if needed.


Changed in version 3.10: The @ref{11a1,,PyConfig.argv} arguments are now only parsed once,
@ref{11a2,,PyConfig.parse_argv} is set to @code{2} after arguments are
parsed, and arguments are only parsed if
@ref{11a2,,PyConfig.parse_argv} equals @code{1}.



Changed in version 3.11: @ref{4554,,PyConfig_Read()} no longer calculates all paths, and so fields
listed under @ref{38e,,Python Path Configuration} may
no longer be updated until @ref{4b7,,Py_InitializeFromConfig()} is
called.

@end deffn

@geindex PyConfig.PyConfig_Clear (C function)
@anchor{c-api/init_config c PyConfig PyConfig_Clear}@anchor{454c}
@deffn {C Function} void  PyConfig_Clear (PyConfig  *config)

Release configuration memory.
@end deffn

Most @code{PyConfig} methods @ref{3502,,preinitialize Python} if needed.
In that case, the Python preinitialization configuration
(@ref{4b3,,PyPreConfig}) in based on the @ref{206,,PyConfig}. If configuration
fields which are in common with @ref{4b3,,PyPreConfig} are tuned, they must
be set before calling a @ref{206,,PyConfig} method:


@itemize *

@item 
@ref{4543,,PyConfig.dev_mode}

@item 
@ref{4545,,PyConfig.isolated}

@item 
@ref{11a2,,PyConfig.parse_argv}

@item 
@ref{48a,,PyConfig.use_environment}
@end itemize

Moreover, if @ref{4551,,PyConfig_SetArgv()} or @ref{4552,,PyConfig_SetBytesArgv()}
is used, this method must be called before other methods, since the
preinitialization configuration depends on command line arguments (if
@ref{11a2,,parse_argv} is non-zero).

The caller of these methods is responsible to handle exceptions (error or
exit) using @code{PyStatus_Exception()} and @code{Py_ExitStatusException()}.

Structure fields:

@geindex PyConfig.argv (C member)
@anchor{c-api/init_config c PyConfig argv}@anchor{11a1}
@deffn {C Member} @ref{4b5,,PyWideStringList}  argv

Command line arguments: @ref{e15,,sys.argv}.

Set @ref{11a2,,parse_argv} to @code{1} to parse
@ref{11a1,,argv} the same way the regular Python parses Python
command line arguments and then to strip Python arguments from
@ref{11a1,,argv}.

If @ref{11a1,,argv} is empty, an empty string is added to
ensure that @ref{e15,,sys.argv} always exists and is never empty.

Default: @code{NULL}.

See also the @ref{360,,orig_argv} member.
@end deffn

@geindex PyConfig.base_exec_prefix (C member)
@anchor{c-api/init_config c PyConfig base_exec_prefix}@anchor{4555}
@deffn {C Member} wchar_t  *base_exec_prefix

@ref{3b36,,sys.base_exec_prefix}.

Default: @code{NULL}.

Part of the @ref{38e,,Python Path Configuration} output.
@end deffn

@geindex PyConfig.base_executable (C member)
@anchor{c-api/init_config c PyConfig base_executable}@anchor{4556}
@deffn {C Member} wchar_t  *base_executable

Python base executable: @code{sys._base_executable}.

Set by the 
@geindex __PYVENV_LAUNCHER__
@geindex environment variable; __PYVENV_LAUNCHER__
@code{__PYVENV_LAUNCHER__} environment variable.

Set from @ref{4557,,PyConfig.executable} if @code{NULL}.

Default: @code{NULL}.

Part of the @ref{38e,,Python Path Configuration} output.
@end deffn

@geindex PyConfig.base_prefix (C member)
@anchor{c-api/init_config c PyConfig base_prefix}@anchor{4558}
@deffn {C Member} wchar_t  *base_prefix

@ref{34c7,,sys.base_prefix}.

Default: @code{NULL}.

Part of the @ref{38e,,Python Path Configuration} output.
@end deffn

@geindex PyConfig.buffered_stdio (C member)
@anchor{c-api/init_config c PyConfig buffered_stdio}@anchor{4559}
@deffn {C Member} int  buffered_stdio

If equals to 0 and @ref{455a,,configure_c_stdio} is non-zero,
disable buffering on the C streams stdout and stderr.

Set to 0 by the @ref{123a,,-u} command line option and the
@geindex PYTHONUNBUFFERED
@geindex environment variable; PYTHONUNBUFFERED
@ref{123b,,PYTHONUNBUFFERED} environment variable.

stdin is always opened in buffered mode.

Default: @code{1}.
@end deffn

@geindex PyConfig.bytes_warning (C member)
@anchor{c-api/init_config c PyConfig bytes_warning}@anchor{455b}
@deffn {C Member} int  bytes_warning

If equals to 1, issue a warning when comparing @ref{2f0,,bytes} or
@ref{2f1,,bytearray} with @ref{2ef,,str}, or comparing @ref{2f0,,bytes} with
@ref{1b1,,int}.

If equal or greater to 2, raise a @ref{761,,BytesWarning} exception in these
cases.

Incremented by the @ref{6dc,,-b} command line option.

Default: @code{0}.
@end deffn

@geindex PyConfig.warn_default_encoding (C member)
@anchor{c-api/init_config c PyConfig warn_default_encoding}@anchor{455c}
@deffn {C Member} int  warn_default_encoding

If non-zero, emit a @ref{1617,,EncodingWarning} warning when @ref{889,,io.TextIOWrapper}
uses its default encoding. See @ref{162b,,Opt-in EncodingWarning} for details.

Default: @code{0}.


New in version 3.10.

@end deffn

@geindex PyConfig.code_debug_ranges (C member)
@anchor{c-api/init_config c PyConfig code_debug_ranges}@anchor{455d}
@deffn {C Member} int  code_debug_ranges

If equals to @code{0}, disables the inclusion of the end line and column
mappings in code objects. Also disables traceback printing carets to
specific error locations.

Set to @code{0} by the 
@geindex PYTHONNODEBUGRANGES
@geindex environment variable; PYTHONNODEBUGRANGES
@ref{15a,,PYTHONNODEBUGRANGES} environment variable
and by the @ref{159,,-X no_debug_ranges} command line option.

Default: @code{1}.


New in version 3.11.

@end deffn

@geindex PyConfig.check_hash_pycs_mode (C member)
@anchor{c-api/init_config c PyConfig check_hash_pycs_mode}@anchor{455e}
@deffn {C Member} wchar_t  *check_hash_pycs_mode

Control the validation behavior of hash-based @code{.pyc} files:
value of the @ref{1606,,--check-hash-based-pycs} command line option.

Valid values:


@itemize -

@item 
@code{L"always"}: Hash the source file for invalidation regardless of
value of the ‘check_source’ flag.

@item 
@code{L"never"}: Assume that hash-based pycs always are valid.

@item 
@code{L"default"}: The ‘check_source’ flag in hash-based pycs
determines invalidation.
@end itemize

Default: @code{L"default"}.

See also 
@geindex Python Enhancement Proposals; PEP 552
PEP 552@footnote{https://www.python.org/dev/peps/pep-0552} “Deterministic pycs”.
@end deffn

@geindex PyConfig.configure_c_stdio (C member)
@anchor{c-api/init_config c PyConfig configure_c_stdio}@anchor{455a}
@deffn {C Member} int  configure_c_stdio

If non-zero, configure C standard streams:


@itemize *

@item 
On Windows, set the binary mode (@code{O_BINARY}) on stdin, stdout and
stderr.

@item 
If @ref{4559,,buffered_stdio} equals zero, disable buffering
of stdin, stdout and stderr streams.

@item 
If @ref{455f,,interactive} is non-zero, enable stream
buffering on stdin and stdout (only stdout on Windows).
@end itemize

Default: @code{1} in Python config, @code{0} in isolated config.
@end deffn

@geindex PyConfig.dev_mode (C member)
@anchor{c-api/init_config c PyConfig dev_mode}@anchor{4543}
@deffn {C Member} int  dev_mode

If non-zero, enable the @ref{3ac,,Python Development Mode}.

Default: @code{-1} in Python mode, @code{0} in isolated mode.
@end deffn

@geindex PyConfig.dump_refs (C member)
@anchor{c-api/init_config c PyConfig dump_refs}@anchor{4560}
@deffn {C Member} int  dump_refs

Dump Python references?

If non-zero, dump all objects which are still alive at exit.

Set to @code{1} by the 
@geindex PYTHONDUMPREFS
@geindex environment variable; PYTHONDUMPREFS
@ref{4ad,,PYTHONDUMPREFS} environment variable.

Need a special build of Python with the @code{Py_TRACE_REFS} macro defined:
see the @ref{376,,configure --with-trace-refs option}.

Default: @code{0}.
@end deffn

@geindex PyConfig.exec_prefix (C member)
@anchor{c-api/init_config c PyConfig exec_prefix}@anchor{4561}
@deffn {C Member} wchar_t  *exec_prefix

The site-specific directory prefix where the platform-dependent Python
files are installed: @ref{3b35,,sys.exec_prefix}.

Default: @code{NULL}.

Part of the @ref{38e,,Python Path Configuration} output.
@end deffn

@geindex PyConfig.executable (C member)
@anchor{c-api/init_config c PyConfig executable}@anchor{4557}
@deffn {C Member} wchar_t  *executable

The absolute path of the executable binary for the Python interpreter:
@ref{585,,sys.executable}.

Default: @code{NULL}.

Part of the @ref{38e,,Python Path Configuration} output.
@end deffn

@geindex PyConfig.faulthandler (C member)
@anchor{c-api/init_config c PyConfig faulthandler}@anchor{4562}
@deffn {C Member} int  faulthandler

Enable faulthandler?

If non-zero, call @ref{7ea,,faulthandler.enable()} at startup.

Set to @code{1} by @ref{159,,-X faulthandler} and the
@geindex PYTHONFAULTHANDLER
@geindex environment variable; PYTHONFAULTHANDLER
@ref{c18,,PYTHONFAULTHANDLER} environment variable.

Default: @code{-1} in Python mode, @code{0} in isolated mode.
@end deffn

@geindex PyConfig.filesystem_encoding (C member)
@anchor{c-api/init_config c PyConfig filesystem_encoding}@anchor{2378}
@deffn {C Member} wchar_t  *filesystem_encoding

@ref{1628,,Filesystem encoding}:
@ref{79e,,sys.getfilesystemencoding()}.

On macOS, Android and VxWorks: use @code{"utf-8"} by default.

On Windows: use @code{"utf-8"} by default, or @code{"mbcs"} if
@ref{4546,,legacy_windows_fs_encoding} of
@ref{4b3,,PyPreConfig} is non-zero.

Default encoding on other platforms:


@itemize *

@item 
@code{"utf-8"} if @ref{4547,,PyPreConfig.utf8_mode} is non-zero.

@item 
@code{"ascii"} if Python detects that @code{nl_langinfo(CODESET)} announces
the ASCII encoding (or Roman8 encoding on HP-UX), whereas the
@code{mbstowcs()} function decodes from a different encoding (usually
Latin1).

@item 
@code{"utf-8"} if @code{nl_langinfo(CODESET)} returns an empty string.

@item 
Otherwise, use the @ref{237a,,locale encoding}:
@code{nl_langinfo(CODESET)} result.
@end itemize

At Python startup, the encoding name is normalized to the Python codec
name. For example, @code{"ANSI_X3.4-1968"} is replaced with @code{"ascii"}.

See also the @ref{2379,,filesystem_errors} member.
@end deffn

@geindex PyConfig.filesystem_errors (C member)
@anchor{c-api/init_config c PyConfig filesystem_errors}@anchor{2379}
@deffn {C Member} wchar_t  *filesystem_errors

@ref{1628,,Filesystem error handler}:
@ref{82f,,sys.getfilesystemencodeerrors()}.

On Windows: use @code{"surrogatepass"} by default, or @code{"replace"}  if
@ref{4546,,legacy_windows_fs_encoding} of
@ref{4b3,,PyPreConfig} is non-zero.

On other platforms: use @code{"surrogateescape"} by default.

Supported error handlers:


@itemize *

@item 
@code{"strict"}

@item 
@code{"surrogateescape"}

@item 
@code{"surrogatepass"} (only supported with the UTF-8 encoding)
@end itemize

See also the @ref{2378,,filesystem_encoding} member.
@end deffn

@geindex PyConfig.hash_seed (C member)
@anchor{c-api/init_config c PyConfig hash_seed}@anchor{4563}
@deffn {C Member} unsigned  long  hash_seed
@end deffn

@geindex PyConfig.use_hash_seed (C member)
@anchor{c-api/init_config c PyConfig use_hash_seed}@anchor{4564}
@deffn {C Member} int  use_hash_seed

Randomized hash function seed.

If @ref{4564,,use_hash_seed} is zero, a seed is chosen randomly
at Python startup, and @ref{4563,,hash_seed} is ignored.

Set by the 
@geindex PYTHONHASHSEED
@geindex environment variable; PYTHONHASHSEED
@ref{c15,,PYTHONHASHSEED} environment variable.

Default @emph{use_hash_seed} value: @code{-1} in Python mode, @code{0} in isolated
mode.
@end deffn

@geindex PyConfig.home (C member)
@anchor{c-api/init_config c PyConfig home}@anchor{4502}
@deffn {C Member} wchar_t  *home

Python home directory.

If @ref{204,,Py_SetPythonHome()} has been called, use its argument if it is
not @code{NULL}.

Set by the 
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME} environment variable.

Default: @code{NULL}.

Part of the @ref{38e,,Python Path Configuration} input.
@end deffn

@geindex PyConfig.import_time (C member)
@anchor{c-api/init_config c PyConfig import_time}@anchor{4565}
@deffn {C Member} int  import_time

If non-zero, profile import time.

Set the @code{1} by the @ref{159,,-X importtime} option and the
@geindex PYTHONPROFILEIMPORTTIME
@geindex environment variable; PYTHONPROFILEIMPORTTIME
@ref{612,,PYTHONPROFILEIMPORTTIME} environment variable.

Default: @code{0}.
@end deffn

@geindex PyConfig.inspect (C member)
@anchor{c-api/init_config c PyConfig inspect}@anchor{4566}
@deffn {C Member} int  inspect

Enter interactive mode after executing a script or a command.

If greater than 0, enable inspect: when a script is passed as first
argument or the -c option is used, enter interactive mode after executing
the script or the command, even when @ref{5ee,,sys.stdin} does not appear to
be a terminal.

Incremented by the @ref{1018,,-i} command line option. Set to @code{1} if the
@geindex PYTHONINSPECT
@geindex environment variable; PYTHONINSPECT
@ref{1017,,PYTHONINSPECT} environment variable is non-empty.

Default: @code{0}.
@end deffn

@geindex PyConfig.install_signal_handlers (C member)
@anchor{c-api/init_config c PyConfig install_signal_handlers}@anchor{3a1}
@deffn {C Member} int  install_signal_handlers

Install Python signal handlers?

Default: @code{1} in Python mode, @code{0} in isolated mode.
@end deffn

@geindex PyConfig.interactive (C member)
@anchor{c-api/init_config c PyConfig interactive}@anchor{455f}
@deffn {C Member} int  interactive

If greater than 0, enable the interactive mode (REPL).

Incremented by the @ref{1018,,-i} command line option.

Default: @code{0}.
@end deffn

@geindex PyConfig.isolated (C member)
@anchor{c-api/init_config c PyConfig isolated}@anchor{4545}
@deffn {C Member} int  isolated

If greater than 0, enable isolated mode:


@itemize *

@item 
@ref{489,,sys.path} contains neither the script’s directory (computed from
@code{argv[0]} or the current directory) nor the user’s site-packages
directory.

@item 
Python REPL doesn’t import @ref{d9,,readline} nor enable default readline
configuration on interactive prompts.

@item 
Set @ref{48a,,use_environment} and
@ref{4567,,user_site_directory} to 0.
@end itemize

Default: @code{0} in Python mode, @code{1} in isolated mode.

See also @ref{4544,,PyPreConfig.isolated}.
@end deffn

@geindex PyConfig.legacy_windows_stdio (C member)
@anchor{c-api/init_config c PyConfig legacy_windows_stdio}@anchor{4568}
@deffn {C Member} int  legacy_windows_stdio

If non-zero, use @ref{ea6,,io.FileIO} instead of
@code{io.WindowsConsoleIO} for @ref{5ee,,sys.stdin}, @ref{5ef,,sys.stdout}
and @ref{42d,,sys.stderr}.

Set to @code{1} if the 
@geindex PYTHONLEGACYWINDOWSSTDIO
@geindex environment variable; PYTHONLEGACYWINDOWSSTDIO
@ref{7a2,,PYTHONLEGACYWINDOWSSTDIO} environment
variable is set to a non-empty string.

Only available on Windows. @code{#ifdef MS_WINDOWS} macro can be used for
Windows specific code.

Default: @code{0}.

See also the 
@geindex Python Enhancement Proposals; PEP 528
PEP 528@footnote{https://www.python.org/dev/peps/pep-0528} (Change Windows console encoding to UTF-8).
@end deffn

@geindex PyConfig.malloc_stats (C member)
@anchor{c-api/init_config c PyConfig malloc_stats}@anchor{4569}
@deffn {C Member} int  malloc_stats

If non-zero, dump statistics on @ref{85c,,Python pymalloc memory allocator} at exit.

Set to @code{1} by the 
@geindex PYTHONMALLOCSTATS
@geindex environment variable; PYTHONMALLOCSTATS
@ref{1423,,PYTHONMALLOCSTATS} environment variable.

The option is ignored if Python is @ref{1656,,configured using the --without-pymalloc option}.

Default: @code{0}.
@end deffn

@geindex PyConfig.platlibdir (C member)
@anchor{c-api/init_config c PyConfig platlibdir}@anchor{11bb}
@deffn {C Member} wchar_t  *platlibdir

Platform library directory name: @ref{42c,,sys.platlibdir}.

Set by the 
@geindex PYTHONPLATLIBDIR
@geindex environment variable; PYTHONPLATLIBDIR
@ref{11bd,,PYTHONPLATLIBDIR} environment variable.

Default: value of the @code{PLATLIBDIR} macro which is set by the
@ref{164d,,configure --with-platlibdir option}
(default: @code{"lib"}, or @code{"DLLs"} on Windows).

Part of the @ref{38e,,Python Path Configuration} input.


New in version 3.9.



Changed in version 3.11: This macro is now used on Windows to locate the standard
library extension modules, typically under @code{DLLs}. However,
for compatibility, note that this value is ignored for any
non-standard layouts, including in-tree builds and virtual
environments.

@end deffn

@geindex PyConfig.pythonpath_env (C member)
@anchor{c-api/init_config c PyConfig pythonpath_env}@anchor{456a}
@deffn {C Member} wchar_t  *pythonpath_env

Module search paths (@ref{489,,sys.path}) as a string separated by @code{DELIM}
(@code{os.path.pathsep}).

Set by the 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} environment variable.

Default: @code{NULL}.

Part of the @ref{38e,,Python Path Configuration} input.
@end deffn

@geindex PyConfig.module_search_paths (C member)
@anchor{c-api/init_config c PyConfig module_search_paths}@anchor{4500}
@deffn {C Member} @ref{4b5,,PyWideStringList}  module_search_paths
@end deffn

@geindex PyConfig.module_search_paths_set (C member)
@anchor{c-api/init_config c PyConfig module_search_paths_set}@anchor{4501}
@deffn {C Member} int  module_search_paths_set

Module search paths: @ref{489,,sys.path}.

If @ref{4501,,module_search_paths_set} is equal to 0,
@ref{4b7,,Py_InitializeFromConfig()} will replace
@ref{4500,,module_search_paths} and sets
@ref{4501,,module_search_paths_set} to @code{1}.

Default: empty list (@code{module_search_paths}) and @code{0}
(@code{module_search_paths_set}).

Part of the @ref{38e,,Python Path Configuration} output.
@end deffn

@geindex PyConfig.optimization_level (C member)
@anchor{c-api/init_config c PyConfig optimization_level}@anchor{456b}
@deffn {C Member} int  optimization_level

Compilation optimization level:


@itemize *

@item 
@code{0}: Peephole optimizer, set @code{__debug__} to @code{True}.

@item 
@code{1}: Level 0, remove assertions, set @code{__debug__} to @code{False}.

@item 
@code{2}: Level 1, strip docstrings.
@end itemize

Incremented by the @ref{912,,-O} command line option. Set to the
@geindex PYTHONOPTIMIZE
@geindex environment variable; PYTHONOPTIMIZE
@ref{160b,,PYTHONOPTIMIZE} environment variable value.

Default: @code{0}.
@end deffn

@geindex PyConfig.orig_argv (C member)
@anchor{c-api/init_config c PyConfig orig_argv}@anchor{360}
@deffn {C Member} @ref{4b5,,PyWideStringList}  orig_argv

The list of the original command line arguments passed to the Python
executable: @ref{2c6,,sys.orig_argv}.

If @ref{360,,orig_argv} list is empty and
@ref{11a1,,argv} is not a list only containing an empty
string, @ref{4554,,PyConfig_Read()} copies @ref{11a1,,argv} into
@ref{360,,orig_argv} before modifying
@ref{11a1,,argv} (if @ref{11a2,,parse_argv} is
non-zero).

See also the @ref{11a1,,argv} member and the
@ref{11de,,Py_GetArgcArgv()} function.

Default: empty list.


New in version 3.10.

@end deffn

@geindex PyConfig.parse_argv (C member)
@anchor{c-api/init_config c PyConfig parse_argv}@anchor{11a2}
@deffn {C Member} int  parse_argv

Parse command line arguments?

If equals to @code{1}, parse @ref{11a1,,argv} the same way the regular
Python parses @ref{b8d,,command line arguments}, and strip
Python arguments from @ref{11a1,,argv}.

The @ref{4554,,PyConfig_Read()} function only parses
@ref{11a1,,PyConfig.argv} arguments once: @ref{11a2,,PyConfig.parse_argv}
is set to @code{2} after arguments are parsed. Since Python arguments are
strippped from @ref{11a1,,PyConfig.argv}, parsing arguments twice would
parse the application options as Python options.

Default: @code{1} in Python mode, @code{0} in isolated mode.


Changed in version 3.10: The @ref{11a1,,PyConfig.argv} arguments are now only parsed if
@ref{11a2,,PyConfig.parse_argv} equals to @code{1}.

@end deffn

@geindex PyConfig.parser_debug (C member)
@anchor{c-api/init_config c PyConfig parser_debug}@anchor{456c}
@deffn {C Member} int  parser_debug

Parser debug mode. If greater than 0, turn on parser debugging output (for expert only, depending
on compilation options).

Incremented by the @ref{1607,,-d} command line option. Set to the
@geindex PYTHONDEBUG
@geindex environment variable; PYTHONDEBUG
@ref{1608,,PYTHONDEBUG} environment variable value.

Default: @code{0}.
@end deffn

@geindex PyConfig.pathconfig_warnings (C member)
@anchor{c-api/init_config c PyConfig pathconfig_warnings}@anchor{456d}
@deffn {C Member} int  pathconfig_warnings

If non-zero, calculation of path configuration is allowed to log
warnings into @code{stderr}. If equals to 0, suppress these warnings.

Default: @code{1} in Python mode, @code{0} in isolated mode.

Part of the @ref{38e,,Python Path Configuration} input.


Changed in version 3.11: Now also applies on Windows.

@end deffn

@geindex PyConfig.prefix (C member)
@anchor{c-api/init_config c PyConfig prefix}@anchor{456e}
@deffn {C Member} wchar_t  *prefix

The site-specific directory prefix where the platform independent Python
files are installed: @ref{3b34,,sys.prefix}.

Default: @code{NULL}.

Part of the @ref{38e,,Python Path Configuration} output.
@end deffn

@geindex PyConfig.program_name (C member)
@anchor{c-api/init_config c PyConfig program_name}@anchor{44ff}
@deffn {C Member} wchar_t  *program_name

Program name used to initialize @ref{4557,,executable} and in
early error messages during Python initialization.


@itemize *

@item 
If @code{Py_SetProgramName()} has been called, use its argument.

@item 
On macOS, use 
@geindex PYTHONEXECUTABLE
@geindex environment variable; PYTHONEXECUTABLE
@ref{1622,,PYTHONEXECUTABLE} environment variable if set.

@item 
If the @code{WITH_NEXT_FRAMEWORK} macro is defined, use
@geindex __PYVENV_LAUNCHER__
@geindex environment variable; __PYVENV_LAUNCHER__
@code{__PYVENV_LAUNCHER__} environment variable if set.

@item 
Use @code{argv[0]} of @ref{11a1,,argv} if available and
non-empty.

@item 
Otherwise, use @code{L"python"} on Windows, or @code{L"python3"} on other
platforms.
@end itemize

Default: @code{NULL}.

Part of the @ref{38e,,Python Path Configuration} input.
@end deffn

@geindex PyConfig.pycache_prefix (C member)
@anchor{c-api/init_config c PyConfig pycache_prefix}@anchor{456f}
@deffn {C Member} wchar_t  *pycache_prefix

Directory where cached @code{.pyc} files are written:
@ref{4ab,,sys.pycache_prefix}.

Set by the @ref{159,,-X pycache_prefix=PATH} command line option and
the 
@geindex PYTHONPYCACHEPREFIX
@geindex environment variable; PYTHONPYCACHEPREFIX
@ref{4aa,,PYTHONPYCACHEPREFIX} environment variable.

If @code{NULL}, @ref{4ab,,sys.pycache_prefix} is set to @code{None}.

Default: @code{NULL}.
@end deffn

@geindex PyConfig.quiet (C member)
@anchor{c-api/init_config c PyConfig quiet}@anchor{4570}
@deffn {C Member} int  quiet

Quiet mode. If greater than 0, don’t display the copyright and version at
Python startup in interactive mode.

Incremented by the @ref{160d,,-q} command line option.

Default: @code{0}.
@end deffn

@geindex PyConfig.run_command (C member)
@anchor{c-api/init_config c PyConfig run_command}@anchor{4571}
@deffn {C Member} wchar_t  *run_command

Value of the @ref{1180,,-c} command line option.

Used by @ref{4bb,,Py_RunMain()}.

Default: @code{NULL}.
@end deffn

@geindex PyConfig.run_filename (C member)
@anchor{c-api/init_config c PyConfig run_filename}@anchor{4572}
@deffn {C Member} wchar_t  *run_filename

Filename passed on the command line: trailing command line argument
without @ref{1180,,-c} or @ref{611,,-m}.

For example, it is set to @code{script.py} by the @code{python3 script.py arg}
command.

Used by @ref{4bb,,Py_RunMain()}.

Default: @code{NULL}.
@end deffn

@geindex PyConfig.run_module (C member)
@anchor{c-api/init_config c PyConfig run_module}@anchor{4573}
@deffn {C Member} wchar_t  *run_module

Value of the @ref{611,,-m} command line option.

Used by @ref{4bb,,Py_RunMain()}.

Default: @code{NULL}.
@end deffn

@geindex PyConfig.show_ref_count (C member)
@anchor{c-api/init_config c PyConfig show_ref_count}@anchor{4574}
@deffn {C Member} int  show_ref_count

Show total reference count at exit?

Set to 1 by @ref{159,,-X showrefcount} command line option.

Need a @ref{338,,debug build of Python} (the @code{Py_REF_DEBUG}
macro must be defined).

Default: @code{0}.
@end deffn

@geindex PyConfig.site_import (C member)
@anchor{c-api/init_config c PyConfig site_import}@anchor{4575}
@deffn {C Member} int  site_import

Import the @ref{e6,,site} module at startup?

If equal to zero, disable the import of the module site and the
site-dependent manipulations of @ref{489,,sys.path} that it entails.

Also disable these manipulations if the @ref{e6,,site} module is explicitly
imported later (call @ref{1610,,site.main()} if you want them to be triggered).

Set to @code{0} by the @ref{d4e,,-S} command line option.

@code{sys.flags.no_site} is set to the inverted value of
@ref{4575,,site_import}.

Default: @code{1}.
@end deffn

@geindex PyConfig.skip_source_first_line (C member)
@anchor{c-api/init_config c PyConfig skip_source_first_line}@anchor{4576}
@deffn {C Member} int  skip_source_first_line

If non-zero, skip the first line of the @ref{4572,,PyConfig.run_filename}
source.

It allows the usage of non-Unix forms of @code{#!cmd}. This is intended for
a DOS specific hack only.

Set to @code{1} by the @ref{1615,,-x} command line option.

Default: @code{0}.
@end deffn

@geindex PyConfig.stdio_encoding (C member)
@anchor{c-api/init_config c PyConfig stdio_encoding}@anchor{3b77}
@deffn {C Member} wchar_t  *stdio_encoding
@end deffn

@geindex PyConfig.stdio_errors (C member)
@anchor{c-api/init_config c PyConfig stdio_errors}@anchor{3b78}
@deffn {C Member} wchar_t  *stdio_errors

Encoding and encoding errors of @ref{5ee,,sys.stdin}, @ref{5ef,,sys.stdout} and
@ref{42d,,sys.stderr} (but @ref{42d,,sys.stderr} always uses
@code{"backslashreplace"} error handler).

If @ref{205,,Py_SetStandardStreamEncoding()} has been called, use its
@emph{error} and @emph{errors} arguments if they are not @code{NULL}.

Use the 
@geindex PYTHONIOENCODING
@geindex environment variable; PYTHONIOENCODING
@ref{b91,,PYTHONIOENCODING} environment variable if it is
non-empty.

Default encoding:


@itemize *

@item 
@code{"UTF-8"} if @ref{4547,,PyPreConfig.utf8_mode} is non-zero.

@item 
Otherwise, use the @ref{237a,,locale encoding}.
@end itemize

Default error handler:


@itemize *

@item 
On Windows: use @code{"surrogateescape"}.

@item 
@code{"surrogateescape"} if @ref{4547,,PyPreConfig.utf8_mode} is non-zero,
or if the LC_CTYPE locale is “C” or “POSIX”.

@item 
@code{"strict"} otherwise.
@end itemize
@end deffn

@geindex PyConfig.tracemalloc (C member)
@anchor{c-api/init_config c PyConfig tracemalloc}@anchor{4577}
@deffn {C Member} int  tracemalloc

Enable tracemalloc?

If non-zero, call @ref{1616,,tracemalloc.start()} at startup.

Set by @ref{159,,-X tracemalloc=N} command line option and by the
@geindex PYTHONTRACEMALLOC
@geindex environment variable; PYTHONTRACEMALLOC
@ref{1623,,PYTHONTRACEMALLOC} environment variable.

Default: @code{-1} in Python mode, @code{0} in isolated mode.
@end deffn

@geindex PyConfig.use_environment (C member)
@anchor{c-api/init_config c PyConfig use_environment}@anchor{48a}
@deffn {C Member} int  use_environment

Use @ref{161c,,environment variables}?

If equals to zero, ignore the @ref{161c,,environment variables}.

Default: @code{1} in Python config and @code{0} in isolated config.
@end deffn

@geindex PyConfig.user_site_directory (C member)
@anchor{c-api/init_config c PyConfig user_site_directory}@anchor{4567}
@deffn {C Member} int  user_site_directory

If non-zero, add the user site directory to @ref{489,,sys.path}.

Set to @code{0} by the @ref{f15,,-s} and @ref{461,,-I} command line options.

Set to @code{0} by the 
@geindex PYTHONNOUSERSITE
@geindex environment variable; PYTHONNOUSERSITE
@ref{f16,,PYTHONNOUSERSITE} environment variable.

Default: @code{1} in Python mode, @code{0} in isolated mode.
@end deffn

@geindex PyConfig.verbose (C member)
@anchor{c-api/init_config c PyConfig verbose}@anchor{4578}
@deffn {C Member} int  verbose

Verbose mode. If greater than 0, print a message each time a module is
imported, showing the place (filename or built-in module) from which
it is loaded.

If greater or equal to 2, print a message for each file that is checked
for when searching for a module. Also provides information on module
cleanup at exit.

Incremented by the @ref{f82,,-v} command line option.

Set to the 
@geindex PYTHONVERBOSE
@geindex environment variable; PYTHONVERBOSE
@ref{1611,,PYTHONVERBOSE} environment variable value.

Default: @code{0}.
@end deffn

@geindex PyConfig.warnoptions (C member)
@anchor{c-api/init_config c PyConfig warnoptions}@anchor{4300}
@deffn {C Member} @ref{4b5,,PyWideStringList}  warnoptions

Options of the @ref{12d,,warnings} module to build warnings filters, lowest
to highest priority: @ref{6dd,,sys.warnoptions}.

The @ref{12d,,warnings} module adds @ref{6dd,,sys.warnoptions} in the reverse
order: the last @ref{4300,,PyConfig.warnoptions} item becomes the first
item of @code{warnings.filters} which is checked first (highest
priority).

The @ref{3a8,,-W} command line options adds its value to
@ref{4300,,warnoptions}, it can be used multiple times.

The 
@geindex PYTHONWARNINGS
@geindex environment variable; PYTHONWARNINGS
@ref{6de,,PYTHONWARNINGS} environment variable can also be used to add
warning options. Multiple options can be specified, separated by commas
(@code{,}).

Default: empty list.
@end deffn

@geindex PyConfig.write_bytecode (C member)
@anchor{c-api/init_config c PyConfig write_bytecode}@anchor{4579}
@deffn {C Member} int  write_bytecode

If equal to 0, Python won’t try to write @code{.pyc} files on the import of
source modules.

Set to @code{0} by the @ref{f34,,-B} command line option and the
@geindex PYTHONDONTWRITEBYTECODE
@geindex environment variable; PYTHONDONTWRITEBYTECODE
@ref{f35,,PYTHONDONTWRITEBYTECODE} environment variable.

@ref{3b68,,sys.dont_write_bytecode} is initialized to the inverted value of
@ref{4579,,write_bytecode}.

Default: @code{1}.
@end deffn

@geindex PyConfig.xoptions (C member)
@anchor{c-api/init_config c PyConfig xoptions}@anchor{4301}
@deffn {C Member} @ref{4b5,,PyWideStringList}  xoptions

Values of the @ref{159,,-X} command line options: @ref{1618,,sys._xoptions}.

Default: empty list.
@end deffn
@end deffn

If @ref{11a2,,parse_argv} is non-zero, @ref{11a1,,argv}
arguments are parsed the same way the regular Python parses @ref{b8d,,command line arguments}, and Python arguments are stripped from
@ref{11a1,,argv}.

The @ref{4301,,xoptions} options are parsed to set other options: see
the @ref{159,,-X} command line option.


Changed in version 3.9: The @code{show_alloc_count} field has been removed.


@node Initialization with PyConfig,Isolated Configuration,PyConfig,Python Initialization Configuration
@anchor{c-api/init_config initialization-with-pyconfig}@anchor{457a}
@subsection Initialization with PyConfig


Function to initialize Python:

@geindex Py_InitializeFromConfig (C function)
@anchor{c-api/init_config c Py_InitializeFromConfig}@anchor{4b7}
@deffn {C Function} @ref{4b4,,PyStatus}  Py_InitializeFromConfig (const  PyConfig  *config)

Initialize Python from @emph{config} configuration.
@end deffn

The caller is responsible to handle exceptions (error or exit) using
@code{PyStatus_Exception()} and @code{Py_ExitStatusException()}.

If @ref{2791,,PyImport_FrozenModules()}, @ref{112b,,PyImport_AppendInittab()} or
@ref{112c,,PyImport_ExtendInittab()} are used, they must be set or called after
Python preinitialization and before the Python initialization. If Python is
initialized multiple times, @ref{112b,,PyImport_AppendInittab()} or
@ref{112c,,PyImport_ExtendInittab()} must be called before each Python
initialization.

The current configuration (@code{PyConfig} type) is stored in
@code{PyInterpreterState.config}.

Example setting the program name:

@example
void init_python(void)
@{
    PyStatus status;

    PyConfig config;
    PyConfig_InitPythonConfig(&config);

    /* Set the program name. Implicitly preinitialize Python. */
    status = PyConfig_SetString(&config, &config.program_name,
                                L"/path/to/my_program");
    if (PyStatus_Exception(status)) @{
        goto exception;
    @}

    status = Py_InitializeFromConfig(&config);
    if (PyStatus_Exception(status)) @{
        goto exception;
    @}
    PyConfig_Clear(&config);
    return;

exception:
    PyConfig_Clear(&config);
    Py_ExitStatusException(status);
@}
@end example

More complete example modifying the default configuration, read the
configuration, and then override some parameters:

@example
PyStatus init_python(const char *program_name)
@{
    PyStatus status;

    PyConfig config;
    PyConfig_InitPythonConfig(&config);

    /* Set the program name before reading the configuration
       (decode byte string from the locale encoding).

       Implicitly preinitialize Python. */
    status = PyConfig_SetBytesString(&config, &config.program_name,
                                     program_name);
    if (PyStatus_Exception(status)) @{
        goto done;
    @}

    /* Read all configuration at once */
    status = PyConfig_Read(&config);
    if (PyStatus_Exception(status)) @{
        goto done;
    @}

    /* Append our custom search path to sys.path */
    status = PyWideStringList_Append(&config.module_search_paths,
                                     L"/path/to/more/modules");
    if (PyStatus_Exception(status)) @{
        goto done;
    @}

    /* Override executable computed by PyConfig_Read() */
    status = PyConfig_SetString(&config, &config.executable,
                                L"/path/to/my_executable");
    if (PyStatus_Exception(status)) @{
        goto done;
    @}

    status = Py_InitializeFromConfig(&config);

done:
    PyConfig_Clear(&config);
    return status;
@}
@end example

@node Isolated Configuration,Python Configuration,Initialization with PyConfig,Python Initialization Configuration
@anchor{c-api/init_config init-isolated-conf}@anchor{4527}@anchor{c-api/init_config isolated-configuration}@anchor{457b}
@subsection Isolated Configuration


@code{PyPreConfig_InitIsolatedConfig()} and
@code{PyConfig_InitIsolatedConfig()} functions create a configuration to
isolate Python from the system. For example, to embed Python into an
application.

This configuration ignores global configuration variables, environments
variables, command line arguments (@ref{11a1,,PyConfig.argv} is not parsed)
and user site directory. The C standard streams (ex: @code{stdout}) and the
LC_CTYPE locale are left unchanged. Signal handlers are not installed.

Configuration files are still used with this configuration to determine
paths that are unspecified. Ensure @ref{4502,,PyConfig.home} is specified
to avoid computing the default path configuration.

@node Python Configuration,Python Path Configuration,Isolated Configuration,Python Initialization Configuration
@anchor{c-api/init_config init-python-config}@anchor{4526}@anchor{c-api/init_config python-configuration}@anchor{457c}
@subsection Python Configuration


@code{PyPreConfig_InitPythonConfig()} and @code{PyConfig_InitPythonConfig()}
functions create a configuration to build a customized Python which behaves as
the regular Python.

Environments variables and command line arguments are used to configure
Python, whereas global configuration variables are ignored.

This function enables C locale coercion (
@geindex Python Enhancement Proposals; PEP 538
PEP 538@footnote{https://www.python.org/dev/peps/pep-0538})
and @ref{5f2,,Python UTF-8 Mode}
(
@geindex Python Enhancement Proposals; PEP 540
PEP 540@footnote{https://www.python.org/dev/peps/pep-0540}) depending on the LC_CTYPE locale, 
@geindex PYTHONUTF8
@geindex environment variable; PYTHONUTF8
@ref{5f1,,PYTHONUTF8} and
@geindex PYTHONCOERCECLOCALE
@geindex environment variable; PYTHONCOERCECLOCALE
@ref{5ed,,PYTHONCOERCECLOCALE} environment variables.

@node Python Path Configuration,Py_RunMain,Python Configuration,Python Initialization Configuration
@anchor{c-api/init_config init-path-config}@anchor{38e}@anchor{c-api/init_config python-path-configuration}@anchor{457d}
@subsection Python Path Configuration


@ref{206,,PyConfig} contains multiple fields for the path configuration:


@itemize *

@item 
Path configuration inputs:


@itemize *

@item 
@ref{4502,,PyConfig.home}

@item 
@ref{11bb,,PyConfig.platlibdir}

@item 
@ref{456d,,PyConfig.pathconfig_warnings}

@item 
@ref{44ff,,PyConfig.program_name}

@item 
@ref{456a,,PyConfig.pythonpath_env}

@item 
current working directory: to get absolute paths

@item 
@code{PATH} environment variable to get the program full path
(from @ref{44ff,,PyConfig.program_name})

@item 
@code{__PYVENV_LAUNCHER__} environment variable

@item 
(Windows only) Application paths in the registry under
“SoftwarePythonPythonCoreX.YPythonPath” of HKEY_CURRENT_USER and
HKEY_LOCAL_MACHINE (where X.Y is the Python version).
@end itemize

@item 
Path configuration output fields:


@itemize *

@item 
@ref{4555,,PyConfig.base_exec_prefix}

@item 
@ref{4556,,PyConfig.base_executable}

@item 
@ref{4558,,PyConfig.base_prefix}

@item 
@ref{4561,,PyConfig.exec_prefix}

@item 
@ref{4557,,PyConfig.executable}

@item 
@ref{4501,,PyConfig.module_search_paths_set},
@ref{4500,,PyConfig.module_search_paths}

@item 
@ref{456e,,PyConfig.prefix}
@end itemize
@end itemize

If at least one “output field” is not set, Python calculates the path
configuration to fill unset fields. If
@ref{4501,,module_search_paths_set} is equal to 0,
@ref{4500,,module_search_paths} is overridden and
@ref{4501,,module_search_paths_set} is set to 1.

It is possible to completely ignore the function calculating the default
path configuration by setting explicitly all path configuration output
fields listed above. A string is considered as set even if it is non-empty.
@code{module_search_paths} is considered as set if
@code{module_search_paths_set} is set to 1. In this case, path
configuration input fields are ignored as well.

Set @ref{456d,,pathconfig_warnings} to 0 to suppress warnings when
calculating the path configuration (Unix only, Windows does not log any warning).

If @ref{4558,,base_prefix} or @ref{4555,,base_exec_prefix}
fields are not set, they inherit their value from @ref{456e,,prefix}
and @ref{4561,,exec_prefix} respectively.

@ref{4bb,,Py_RunMain()} and @ref{763,,Py_Main()} modify @ref{489,,sys.path}:


@itemize *

@item 
If @ref{4572,,run_filename} is set and is a directory which contains a
@code{__main__.py} script, prepend @ref{4572,,run_filename} to
@ref{489,,sys.path}.

@item 
If @ref{4545,,isolated} is zero:


@itemize *

@item 
If @ref{4573,,run_module} is set, prepend the current directory
to @ref{489,,sys.path}. Do nothing if the current directory cannot be read.

@item 
If @ref{4572,,run_filename} is set, prepend the directory of the
filename to @ref{489,,sys.path}.

@item 
Otherwise, prepend an empty string to @ref{489,,sys.path}.
@end itemize
@end itemize

If @ref{4575,,site_import} is non-zero, @ref{489,,sys.path} can be
modified by the @ref{e6,,site} module. If
@ref{4567,,user_site_directory} is non-zero and the user’s
site-package directory exists, the @ref{e6,,site} module appends the user’s
site-package directory to @ref{489,,sys.path}.

The following configuration files are used by the path configuration:


@itemize *

@item 
@code{pyvenv.cfg}

@item 
@code{python._pth} (Windows only)

@item 
@code{pybuilddir.txt} (Unix only)
@end itemize

The @code{__PYVENV_LAUNCHER__} environment variable is used to set
@ref{4556,,PyConfig.base_executable}

@node Py_RunMain,Py_GetArgcArgv,Python Path Configuration,Python Initialization Configuration
@anchor{c-api/init_config py-runmain}@anchor{457e}
@subsection Py_RunMain()


@geindex Py_RunMain (C function)
@anchor{c-api/init_config c Py_RunMain}@anchor{4bb}
@deffn {C Function} int  Py_RunMain (void)

Execute the command (@ref{4571,,PyConfig.run_command}), the script
(@ref{4572,,PyConfig.run_filename}) or the module
(@ref{4573,,PyConfig.run_module}) specified on the command line or in the
configuration.

By default and when if @ref{1018,,-i} option is used, run the REPL.

Finally, finalizes Python and returns an exit status that can be passed to
the @code{exit()} function.
@end deffn

See @ref{4526,,Python Configuration} for an example of
customized Python always running in isolated mode using
@ref{4bb,,Py_RunMain()}.

@node Py_GetArgcArgv,Multi-Phase Initialization Private Provisional API,Py_RunMain,Python Initialization Configuration
@anchor{c-api/init_config py-getargcargv}@anchor{457f}
@subsection Py_GetArgcArgv()


@geindex Py_GetArgcArgv (C function)
@anchor{c-api/init_config c Py_GetArgcArgv}@anchor{11de}
@deffn {C Function} void  Py_GetArgcArgv (int  *argc, wchar_t  ***argv)

Get the original command line arguments, before Python modified them.

See also @ref{360,,PyConfig.orig_argv} member.
@end deffn

@node Multi-Phase Initialization Private Provisional API,,Py_GetArgcArgv,Python Initialization Configuration
@anchor{c-api/init_config multi-phase-initialization-private-provisional-api}@anchor{4580}
@subsection Multi-Phase Initialization Private Provisional API


This section is a private provisional API introducing multi-phase
initialization, the core feature of the 
@geindex Python Enhancement Proposals; PEP 432
PEP 432@footnote{https://www.python.org/dev/peps/pep-0432}:


@itemize *

@item 
“Core” initialization phase, “bare minimum Python”:


@itemize *

@item 
Builtin types;

@item 
Builtin exceptions;

@item 
Builtin and frozen modules;

@item 
The @ref{f7,,sys} module is only partially initialized
(ex: @ref{489,,sys.path} doesn’t exist yet).
@end itemize

@item 
“Main” initialization phase, Python is fully initialized:


@itemize *

@item 
Install and configure @ref{96,,importlib};

@item 
Apply the @ref{38e,,Path Configuration};

@item 
Install signal handlers;

@item 
Finish @ref{f7,,sys} module initialization (ex: create @ref{5ef,,sys.stdout}
and @ref{489,,sys.path});

@item 
Enable optional features like @ref{78,,faulthandler} and @ref{11b,,tracemalloc};

@item 
Import the @ref{e6,,site} module;

@item 
etc.
@end itemize
@end itemize

Private provisional API:


@itemize *

@item 
@code{PyConfig._init_main}: if set to 0,
@ref{4b7,,Py_InitializeFromConfig()} stops at the “Core” initialization phase.

@item 
@code{PyConfig._isolated_interpreter}: if non-zero,
disallow threads, subprocesses and fork.
@end itemize

@geindex _Py_InitializeMain (C function)
@anchor{c-api/init_config c _Py_InitializeMain}@anchor{4581}
@deffn {C Function} @ref{4b4,,PyStatus}  _Py_InitializeMain (void)

Move to the “Main” initialization phase, finish the Python initialization.
@end deffn

No module is imported during the “Core” phase and the @code{importlib} module is
not configured: the @ref{38e,,Path Configuration} is only
applied during the “Main” phase. It may allow to customize Python in Python to
override or tune the @ref{38e,,Path Configuration}, maybe
install a custom @ref{87c,,sys.meta_path} importer or an import hook, etc.

It may become possible to calculatin the @ref{38e,,Path Configuration} in Python, after the Core phase and before the Main phase,
which is one of the 
@geindex Python Enhancement Proposals; PEP 432
PEP 432@footnote{https://www.python.org/dev/peps/pep-0432} motivation.

The “Core” phase is not properly defined: what should be and what should
not be available at this phase is not specified yet. The API is marked
as private and provisional: the API can be modified or even be removed
anytime until a proper public API is designed.

Example running Python code between “Core” and “Main” initialization
phases:

@example
void init_python(void)
@{
    PyStatus status;

    PyConfig config;
    PyConfig_InitPythonConfig(&config);
    config._init_main = 0;

    /* ... customize 'config' configuration ... */

    status = Py_InitializeFromConfig(&config);
    PyConfig_Clear(&config);
    if (PyStatus_Exception(status)) @{
        Py_ExitStatusException(status);
    @}

    /* Use sys.stderr because sys.stdout is only created
       by _Py_InitializeMain() */
    int res = PyRun_SimpleString(
        "import sys; "
        "print('Run Python code before _Py_InitializeMain', "
               "file=sys.stderr)");
    if (res < 0) @{
        exit(1);
    @}

    /* ... put more configuration code here ... */

    status = _Py_InitializeMain();
    if (PyStatus_Exception(status)) @{
        Py_ExitStatusException(status);
    @}
@}
@end example

@node Memory Management,Object Implementation Support,Python Initialization Configuration,Python/C API Reference Manual
@anchor{c-api/memory doc}@anchor{4582}@anchor{c-api/memory memory}@anchor{453e}@anchor{c-api/memory memory-management}@anchor{4583}
@section Memory Management


@menu
* Overview: Overview<4>. 
* Allocator Domains:: 
* Raw Memory Interface:: 
* Memory Interface:: 
* Object allocators:: 
* Default Memory Allocators:: 
* Customize Memory Allocators:: 
* Debug hooks on the Python memory allocators:: 
* The pymalloc allocator:: 
* tracemalloc C API:: 
* Examples: Examples<37>. 

@end menu

@node Overview<4>,Allocator Domains,,Memory Management
@anchor{c-api/memory memoryoverview}@anchor{4584}@anchor{c-api/memory overview}@anchor{4585}
@subsection Overview


Memory management in Python involves a private heap containing all Python
objects and data structures. The management of this private heap is ensured
internally by the @emph{Python memory manager}.  The Python memory manager has
different components which deal with various dynamic storage management aspects,
like sharing, segmentation, preallocation or caching.

At the lowest level, a raw memory allocator ensures that there is enough room in
the private heap for storing all Python-related data by interacting with the
memory manager of the operating system. On top of the raw memory allocator,
several object-specific allocators operate on the same heap and implement
distinct memory management policies adapted to the peculiarities of every object
type. For example, integer objects are managed differently within the heap than
strings, tuples or dictionaries because integers imply different storage
requirements and speed/space tradeoffs. The Python memory manager thus delegates
some of the work to the object-specific allocators, but ensures that the latter
operate within the bounds of the private heap.

It is important to understand that the management of the Python heap is
performed by the interpreter itself and that the user has no control over it,
even if they regularly manipulate object pointers to memory blocks inside that
heap.  The allocation of heap space for Python objects and other internal
buffers is performed on demand by the Python memory manager through the Python/C
API functions listed in this document.

@geindex malloc()
@geindex calloc()
@geindex realloc()
@geindex free()

To avoid memory corruption, extension writers should never try to operate on
Python objects with the functions exported by the C library: @code{malloc()},
@code{calloc()}, @code{realloc()} and @code{free()}.  This will result in  mixed
calls between the C allocator and the Python memory manager with fatal
consequences, because they implement different algorithms and operate on
different heaps.  However, one may safely allocate and release memory blocks
with the C library allocator for individual purposes, as shown in the following
example:

@example
PyObject *res;
char *buf = (char *) malloc(BUFSIZ); /* for I/O */

if (buf == NULL)
    return PyErr_NoMemory();
...Do some I/O operation involving buf...
res = PyBytes_FromString(buf);
free(buf); /* malloc'ed */
return res;
@end example

In this example, the memory request for the I/O buffer is handled by the C
library allocator. The Python memory manager is involved only in the allocation
of the bytes object returned as a result.

In most situations, however, it is recommended to allocate memory from the
Python heap specifically because the latter is under control of the Python
memory manager. For example, this is required when the interpreter is extended
with new object types written in C. Another reason for using the Python heap is
the desire to @emph{inform} the Python memory manager about the memory needs of the
extension module. Even when the requested memory is used exclusively for
internal, highly-specific purposes, delegating all memory requests to the Python
memory manager causes the interpreter to have a more accurate image of its
memory footprint as a whole. Consequently, under certain circumstances, the
Python memory manager may or may not trigger appropriate actions, like garbage
collection, memory compaction or other preventive procedures. Note that by using
the C library allocator as shown in the previous example, the allocated memory
for the I/O buffer escapes completely the Python memory manager.


@subsubheading See also


The 
@geindex PYTHONMALLOC
@geindex environment variable; PYTHONMALLOC
@ref{7ab,,PYTHONMALLOC} environment variable can be used to configure
the memory allocators used by Python.

The 
@geindex PYTHONMALLOCSTATS
@geindex environment variable; PYTHONMALLOCSTATS
@ref{1423,,PYTHONMALLOCSTATS} environment variable can be used to print
statistics of the @ref{85c,,pymalloc memory allocator} every time a
new pymalloc object arena is created, and on shutdown.


@node Allocator Domains,Raw Memory Interface,Overview<4>,Memory Management
@anchor{c-api/memory allocator-domains}@anchor{4586}
@subsection Allocator Domains


All allocating functions belong to one of three different “domains” (see also
@ref{4587,,PyMemAllocatorDomain}). These domains represent different allocation
strategies and are optimized for different purposes. The specific details on
how every domain allocates memory or what internal functions each domain calls
is considered an implementation detail, but for debugging purposes a simplified
table can be found at @ref{1626,,here}. There is no hard
requirement to use the memory returned by the allocation functions belonging to
a given domain for only the purposes hinted by that domain (although this is the
recommended practice). For example, one could use the memory returned by
@ref{bc4,,PyMem_RawMalloc()} for allocating Python objects or the memory returned
by @ref{7af,,PyObject_Malloc()} for allocating memory for buffers.

The three allocation domains are:


@itemize *

@item 
Raw domain: intended for allocating memory for general-purpose memory
buffers where the allocation @emph{must} go to the system allocator or where the
allocator can operate without the @ref{385,,GIL}. The memory is requested directly
to the system.

@item 
“Mem” domain: intended for allocating memory for Python buffers and
general-purpose memory buffers where the allocation must be performed with
the @ref{385,,GIL} held. The memory is taken from the Python private heap.

@item 
Object domain: intended for allocating memory belonging to Python objects. The
memory is taken from the Python private heap.
@end itemize

When freeing memory previously allocated by the allocating functions belonging to a
given domain,the matching specific deallocating functions must be used. For example,
@ref{fa3,,PyMem_Free()} must be used to free memory allocated using @ref{7ad,,PyMem_Malloc()}.

@node Raw Memory Interface,Memory Interface,Allocator Domains,Memory Management
@anchor{c-api/memory raw-memory-interface}@anchor{4588}
@subsection Raw Memory Interface


The following function sets are wrappers to the system allocator. These
functions are thread-safe, the @ref{7ae,,GIL} does not
need to be held.

The @ref{1626,,default raw memory allocator} uses
the following functions: @code{malloc()}, @code{calloc()}, @code{realloc()}
and @code{free()}; call @code{malloc(1)} (or @code{calloc(1, 1)}) when requesting
zero bytes.


New in version 3.4.


@geindex PyMem_RawMalloc (C function)
@anchor{c-api/memory c PyMem_RawMalloc}@anchor{bc4}
@deffn {C Function} void  *PyMem_RawMalloc (size_t  n)

Allocates @emph{n} bytes and returns a pointer of type void* to the
allocated memory, or @code{NULL} if the request fails.

Requesting zero bytes returns a distinct non-@code{NULL} pointer if possible, as
if @code{PyMem_RawMalloc(1)} had been called instead. The memory will not have
been initialized in any way.
@end deffn

@geindex PyMem_RawCalloc (C function)
@anchor{c-api/memory c PyMem_RawCalloc}@anchor{a1e}
@deffn {C Function} void  *PyMem_RawCalloc (size_t  nelem, size_t  elsize)

Allocates @emph{nelem} elements each whose size in bytes is @emph{elsize} and returns
a pointer of type void* to the allocated memory, or @code{NULL} if the
request fails. The memory is initialized to zeros.

Requesting zero elements or elements of size zero bytes returns a distinct
non-@code{NULL} pointer if possible, as if @code{PyMem_RawCalloc(1, 1)} had been
called instead.


New in version 3.5.

@end deffn

@geindex PyMem_RawRealloc (C function)
@anchor{c-api/memory c PyMem_RawRealloc}@anchor{bc5}
@deffn {C Function} void  *PyMem_RawRealloc (void  *p, size_t  n)

Resizes the memory block pointed to by @emph{p} to @emph{n} bytes. The contents will
be unchanged to the minimum of the old and the new sizes.

If @emph{p} is @code{NULL}, the call is equivalent to @code{PyMem_RawMalloc(n)}; else if
@emph{n} is equal to zero, the memory block is resized but is not freed, and the
returned pointer is non-@code{NULL}.

Unless @emph{p} is @code{NULL}, it must have been returned by a previous call to
@ref{bc4,,PyMem_RawMalloc()}, @ref{bc5,,PyMem_RawRealloc()} or
@ref{a1e,,PyMem_RawCalloc()}.

If the request fails, @ref{bc5,,PyMem_RawRealloc()} returns @code{NULL} and @emph{p}
remains a valid pointer to the previous memory area.
@end deffn

@geindex PyMem_RawFree (C function)
@anchor{c-api/memory c PyMem_RawFree}@anchor{42fc}
@deffn {C Function} void  PyMem_RawFree (void  *p)

Frees the memory block pointed to by @emph{p}, which must have been returned by a
previous call to @ref{bc4,,PyMem_RawMalloc()}, @ref{bc5,,PyMem_RawRealloc()} or
@ref{a1e,,PyMem_RawCalloc()}.  Otherwise, or if @code{PyMem_RawFree(p)} has been
called before, undefined behavior occurs.

If @emph{p} is @code{NULL}, no operation is performed.
@end deffn

@node Memory Interface,Object allocators,Raw Memory Interface,Memory Management
@anchor{c-api/memory memory-interface}@anchor{4589}@anchor{c-api/memory memoryinterface}@anchor{458a}
@subsection Memory Interface


The following function sets, modeled after the ANSI C standard, but specifying
behavior when requesting zero bytes, are available for allocating and releasing
memory from the Python heap.

The @ref{1626,,default memory allocator} uses the
@ref{85c,,pymalloc memory allocator}.

@cartouche
@quotation Warning 
The @ref{7ae,,GIL} must be held when using these
functions.
@end quotation
@end cartouche


Changed in version 3.6: The default allocator is now pymalloc instead of system @code{malloc()}.


@geindex PyMem_Malloc (C function)
@anchor{c-api/memory c PyMem_Malloc}@anchor{7ad}
@deffn {C Function} void  *PyMem_Malloc (size_t  n)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Allocates @emph{n} bytes and returns a pointer of type void* to the
allocated memory, or @code{NULL} if the request fails.

Requesting zero bytes returns a distinct non-@code{NULL} pointer if possible, as
if @code{PyMem_Malloc(1)} had been called instead. The memory will not have
been initialized in any way.
@end deffn

@geindex PyMem_Calloc (C function)
@anchor{c-api/memory c PyMem_Calloc}@anchor{a1f}
@deffn {C Function} void  *PyMem_Calloc (size_t  nelem, size_t  elsize)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Allocates @emph{nelem} elements each whose size in bytes is @emph{elsize} and returns
a pointer of type void* to the allocated memory, or @code{NULL} if the
request fails. The memory is initialized to zeros.

Requesting zero elements or elements of size zero bytes returns a distinct
non-@code{NULL} pointer if possible, as if @code{PyMem_Calloc(1, 1)} had been called
instead.


New in version 3.5.

@end deffn

@geindex PyMem_Realloc (C function)
@anchor{c-api/memory c PyMem_Realloc}@anchor{bc6}
@deffn {C Function} void  *PyMem_Realloc (void  *p, size_t  n)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Resizes the memory block pointed to by @emph{p} to @emph{n} bytes. The contents will be
unchanged to the minimum of the old and the new sizes.

If @emph{p} is @code{NULL}, the call is equivalent to @code{PyMem_Malloc(n)}; else if @emph{n}
is equal to zero, the memory block is resized but is not freed, and the
returned pointer is non-@code{NULL}.

Unless @emph{p} is @code{NULL}, it must have been returned by a previous call to
@ref{7ad,,PyMem_Malloc()}, @ref{bc6,,PyMem_Realloc()} or @ref{a1f,,PyMem_Calloc()}.

If the request fails, @ref{bc6,,PyMem_Realloc()} returns @code{NULL} and @emph{p} remains
a valid pointer to the previous memory area.
@end deffn

@geindex PyMem_Free (C function)
@anchor{c-api/memory c PyMem_Free}@anchor{fa3}
@deffn {C Function} void  PyMem_Free (void  *p)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Frees the memory block pointed to by @emph{p}, which must have been returned by a
previous call to @ref{7ad,,PyMem_Malloc()}, @ref{bc6,,PyMem_Realloc()} or
@ref{a1f,,PyMem_Calloc()}.  Otherwise, or if @code{PyMem_Free(p)} has been called
before, undefined behavior occurs.

If @emph{p} is @code{NULL}, no operation is performed.
@end deffn

The following type-oriented macros are provided for convenience.  Note  that
@emph{TYPE} refers to any C type.

@geindex PyMem_New (C function)
@anchor{c-api/memory c PyMem_New}@anchor{458b}
@deffn {C Function} TYPE  *PyMem_New (TYPE, size_t  n)

Same as @ref{7ad,,PyMem_Malloc()}, but allocates @code{(n * sizeof(TYPE))} bytes of
memory.  Returns a pointer cast to TYPE*.  The memory will not have
been initialized in any way.
@end deffn

@geindex PyMem_Resize (C function)
@anchor{c-api/memory c PyMem_Resize}@anchor{458c}
@deffn {C Function} TYPE  *PyMem_Resize (void  *p, TYPE, size_t  n)

Same as @ref{bc6,,PyMem_Realloc()}, but the memory block is resized to @code{(n *
sizeof(TYPE))} bytes.  Returns a pointer cast to TYPE*. On return,
@emph{p} will be a pointer to the new memory area, or @code{NULL} in the event of
failure.

This is a C preprocessor macro; @emph{p} is always reassigned.  Save the original
value of @emph{p} to avoid losing memory when handling errors.
@end deffn

@geindex PyMem_Del (C function)
@anchor{c-api/memory c PyMem_Del}@anchor{458d}
@deffn {C Function} void  PyMem_Del (void  *p)

Same as @ref{fa3,,PyMem_Free()}.
@end deffn

In addition, the following macro sets are provided for calling the Python memory
allocator directly, without involving the C API functions listed above. However,
note that their use does not preserve binary compatibility across Python
versions and is therefore deprecated in extension modules.


@itemize *

@item 
@code{PyMem_MALLOC(size)}

@item 
@code{PyMem_NEW(type, size)}

@item 
@code{PyMem_REALLOC(ptr, size)}

@item 
@code{PyMem_RESIZE(ptr, type, size)}

@item 
@code{PyMem_FREE(ptr)}

@item 
@code{PyMem_DEL(ptr)}
@end itemize

@node Object allocators,Default Memory Allocators,Memory Interface,Memory Management
@anchor{c-api/memory object-allocators}@anchor{458e}
@subsection Object allocators


The following function sets, modeled after the ANSI C standard, but specifying
behavior when requesting zero bytes, are available for allocating and releasing
memory from the Python heap.

@cartouche
@quotation Note 
There is no guarantee that the memory returned by these allocators can be
successfully casted to a Python object when intercepting the allocating
functions in this domain by the methods described in
the @ref{458f,,Customize Memory Allocators} section.
@end quotation
@end cartouche

The @ref{1626,,default object allocator} uses the
@ref{85c,,pymalloc memory allocator}.

@cartouche
@quotation Warning 
The @ref{7ae,,GIL} must be held when using these
functions.
@end quotation
@end cartouche

@geindex PyObject_Malloc (C function)
@anchor{c-api/memory c PyObject_Malloc}@anchor{7af}
@deffn {C Function} void  *PyObject_Malloc (size_t  n)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Allocates @emph{n} bytes and returns a pointer of type void* to the
allocated memory, or @code{NULL} if the request fails.

Requesting zero bytes returns a distinct non-@code{NULL} pointer if possible, as
if @code{PyObject_Malloc(1)} had been called instead. The memory will not have
been initialized in any way.
@end deffn

@geindex PyObject_Calloc (C function)
@anchor{c-api/memory c PyObject_Calloc}@anchor{a20}
@deffn {C Function} void  *PyObject_Calloc (size_t  nelem, size_t  elsize)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.7.}
Allocates @emph{nelem} elements each whose size in bytes is @emph{elsize} and returns
a pointer of type void* to the allocated memory, or @code{NULL} if the
request fails. The memory is initialized to zeros.

Requesting zero elements or elements of size zero bytes returns a distinct
non-@code{NULL} pointer if possible, as if @code{PyObject_Calloc(1, 1)} had been called
instead.


New in version 3.5.

@end deffn

@geindex PyObject_Realloc (C function)
@anchor{c-api/memory c PyObject_Realloc}@anchor{fa4}
@deffn {C Function} void  *PyObject_Realloc (void  *p, size_t  n)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Resizes the memory block pointed to by @emph{p} to @emph{n} bytes. The contents will be
unchanged to the minimum of the old and the new sizes.

If @emph{p} is @code{NULL}, the call is equivalent to @code{PyObject_Malloc(n)}; else if @emph{n}
is equal to zero, the memory block is resized but is not freed, and the
returned pointer is non-@code{NULL}.

Unless @emph{p} is @code{NULL}, it must have been returned by a previous call to
@ref{7af,,PyObject_Malloc()}, @ref{fa4,,PyObject_Realloc()} or @ref{a20,,PyObject_Calloc()}.

If the request fails, @ref{fa4,,PyObject_Realloc()} returns @code{NULL} and @emph{p} remains
a valid pointer to the previous memory area.
@end deffn

@geindex PyObject_Free (C function)
@anchor{c-api/memory c PyObject_Free}@anchor{7ac}
@deffn {C Function} void  PyObject_Free (void  *p)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Frees the memory block pointed to by @emph{p}, which must have been returned by a
previous call to @ref{7af,,PyObject_Malloc()}, @ref{fa4,,PyObject_Realloc()} or
@ref{a20,,PyObject_Calloc()}.  Otherwise, or if @code{PyObject_Free(p)} has been called
before, undefined behavior occurs.

If @emph{p} is @code{NULL}, no operation is performed.
@end deffn

@node Default Memory Allocators,Customize Memory Allocators,Object allocators,Memory Management
@anchor{c-api/memory default-memory-allocators}@anchor{1626}@anchor{c-api/memory id1}@anchor{4590}
@subsection Default Memory Allocators


Default memory allocators:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Configuration

@tab

Name

@tab

PyMem_RawMalloc

@tab

PyMem_Malloc

@tab

PyObject_Malloc

@item

Release build

@tab

@code{"pymalloc"}

@tab

@code{malloc}

@tab

@code{pymalloc}

@tab

@code{pymalloc}

@item

Debug build

@tab

@code{"pymalloc_debug"}

@tab

@code{malloc} + debug

@tab

@code{pymalloc} + debug

@tab

@code{pymalloc} + debug

@item

Release build, without pymalloc

@tab

@code{"malloc"}

@tab

@code{malloc}

@tab

@code{malloc}

@tab

@code{malloc}

@item

Debug build, without pymalloc

@tab

@code{"malloc_debug"}

@tab

@code{malloc} + debug

@tab

@code{malloc} + debug

@tab

@code{malloc} + debug

@end multitable


Legend:


@itemize *

@item 
Name: value for 
@geindex PYTHONMALLOC
@geindex environment variable; PYTHONMALLOC
@ref{7ab,,PYTHONMALLOC} environment variable.

@item 
@code{malloc}: system allocators from the standard C library, C functions:
@code{malloc()}, @code{calloc()}, @code{realloc()} and @code{free()}.

@item 
@code{pymalloc}: @ref{85c,,pymalloc memory allocator}.

@item 
“+ debug”: with @ref{1627,,debug hooks on the Python memory allocators}.

@item 
“Debug build”: @ref{338,,Python build in debug mode}.
@end itemize

@node Customize Memory Allocators,Debug hooks on the Python memory allocators,Default Memory Allocators,Memory Management
@anchor{c-api/memory customize-memory-allocators}@anchor{458f}@anchor{c-api/memory id2}@anchor{4591}
@subsection Customize Memory Allocators



New in version 3.4.


@geindex PyMemAllocatorEx (C type)
@anchor{c-api/memory c PyMemAllocatorEx}@anchor{a41}
@deffn {C Type} type  PyMemAllocatorEx

Structure used to describe a memory block allocator. The structure has
four fields:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Field

@tab

Meaning

@item

@code{void *ctx}

@tab

user context passed as first argument

@item

@code{void* malloc(void *ctx, size_t size)}

@tab

allocate a memory block

@item

@code{void* calloc(void *ctx, size_t nelem, size_t elsize)}

@tab

allocate a memory block initialized
with zeros

@item

@code{void* realloc(void *ctx, void *ptr, size_t new_size)}

@tab

allocate or resize a memory block

@item

@code{void free(void *ctx, void *ptr)}

@tab

free a memory block

@end multitable



Changed in version 3.5: The @code{PyMemAllocator} structure was renamed to
@ref{a41,,PyMemAllocatorEx} and a new @code{calloc} field was added.

@end deffn

@geindex PyMemAllocatorDomain (C type)
@anchor{c-api/memory c PyMemAllocatorDomain}@anchor{4587}
@deffn {C Type} type  PyMemAllocatorDomain

Enum used to identify an allocator domain. Domains:

@geindex PyMemAllocatorDomain.PYMEM_DOMAIN_RAW (C macro)
@anchor{c-api/memory c PyMemAllocatorDomain PYMEM_DOMAIN_RAW}@anchor{4592}
@deffn {C Macro} PYMEM_DOMAIN_RAW

Functions:


@itemize *

@item 
@ref{bc4,,PyMem_RawMalloc()}

@item 
@ref{bc5,,PyMem_RawRealloc()}

@item 
@ref{a1e,,PyMem_RawCalloc()}

@item 
@ref{42fc,,PyMem_RawFree()}
@end itemize
@end deffn

@geindex PyMemAllocatorDomain.PYMEM_DOMAIN_MEM (C macro)
@anchor{c-api/memory c PyMemAllocatorDomain PYMEM_DOMAIN_MEM}@anchor{4593}
@deffn {C Macro} PYMEM_DOMAIN_MEM

Functions:


@itemize *

@item 
@ref{7ad,,PyMem_Malloc()},

@item 
@ref{bc6,,PyMem_Realloc()}

@item 
@ref{a1f,,PyMem_Calloc()}

@item 
@ref{fa3,,PyMem_Free()}
@end itemize
@end deffn

@geindex PyMemAllocatorDomain.PYMEM_DOMAIN_OBJ (C macro)
@anchor{c-api/memory c PyMemAllocatorDomain PYMEM_DOMAIN_OBJ}@anchor{4594}
@deffn {C Macro} PYMEM_DOMAIN_OBJ

Functions:


@itemize *

@item 
@ref{7af,,PyObject_Malloc()}

@item 
@ref{fa4,,PyObject_Realloc()}

@item 
@ref{a20,,PyObject_Calloc()}

@item 
@ref{7ac,,PyObject_Free()}
@end itemize
@end deffn
@end deffn

@geindex PyMem_GetAllocator (C function)
@anchor{c-api/memory c PyMem_GetAllocator}@anchor{44ee}
@deffn {C Function} void  PyMem_GetAllocator (PyMemAllocatorDomain  domain, PyMemAllocatorEx  *allocator)

Get the memory block allocator of the specified domain.
@end deffn

@geindex PyMem_SetAllocator (C function)
@anchor{c-api/memory c PyMem_SetAllocator}@anchor{44ec}
@deffn {C Function} void  PyMem_SetAllocator (PyMemAllocatorDomain  domain, PyMemAllocatorEx  *allocator)

Set the memory block allocator of the specified domain.

The new allocator must return a distinct non-@code{NULL} pointer when requesting
zero bytes.

For the @code{PYMEM_DOMAIN_RAW} domain, the allocator must be
thread-safe: the @ref{7ae,,GIL} is not held when the
allocator is called.

If the new allocator is not a hook (does not call the previous allocator),
the @ref{7b0,,PyMem_SetupDebugHooks()} function must be called to reinstall the
debug hooks on top on the new allocator.

See also @ref{453d,,PyPreConfig.allocator} and @ref{3502,,Preinitialize Python with PyPreConfig}.
@end deffn

@geindex PyMem_SetupDebugHooks (C function)
@anchor{c-api/memory c PyMem_SetupDebugHooks}@anchor{7b0}
@deffn {C Function} void  PyMem_SetupDebugHooks (void)

Setup @ref{1627,,debug hooks in the Python memory allocators}
to detect memory errors.
@end deffn

@node Debug hooks on the Python memory allocators,The pymalloc allocator,Customize Memory Allocators,Memory Management
@anchor{c-api/memory debug-hooks-on-the-python-memory-allocators}@anchor{4595}@anchor{c-api/memory pymem-debug-hooks}@anchor{1627}
@subsection Debug hooks on the Python memory allocators


When @ref{338,,Python is built in debug mode}, the
@ref{7b0,,PyMem_SetupDebugHooks()} function is called at the @ref{3502,,Python preinitialization} to setup debug hooks on Python memory allocators
to detect memory errors.

The 
@geindex PYTHONMALLOC
@geindex environment variable; PYTHONMALLOC
@ref{7ab,,PYTHONMALLOC} environment variable can be used to install debug
hooks on a Python compiled in release mode (ex: @code{PYTHONMALLOC=debug}).

The @ref{7b0,,PyMem_SetupDebugHooks()} function can be used to set debug hooks
after calling @ref{44ec,,PyMem_SetAllocator()}.

These debug hooks fill dynamically allocated memory blocks with special,
recognizable bit patterns. Newly allocated memory is filled with the byte
@code{0xCD} (@code{PYMEM_CLEANBYTE}), freed memory is filled with the byte @code{0xDD}
(@code{PYMEM_DEADBYTE}). Memory blocks are surrounded by “forbidden bytes”
filled with the byte @code{0xFD} (@code{PYMEM_FORBIDDENBYTE}). Strings of these bytes
are unlikely to be valid addresses, floats, or ASCII strings.

Runtime checks:


@itemize -

@item 
Detect API violations. For example, detect if @ref{7ac,,PyObject_Free()} is
called on a memory block allocated by @ref{7ad,,PyMem_Malloc()}.

@item 
Detect write before the start of the buffer (buffer underflow).

@item 
Detect write after the end of the buffer (buffer overflow).

@item 
Check that the @ref{7ae,,GIL} is held when
allocator functions of @code{PYMEM_DOMAIN_OBJ} (ex:
@ref{7af,,PyObject_Malloc()}) and @code{PYMEM_DOMAIN_MEM} (ex:
@ref{7ad,,PyMem_Malloc()}) domains are called.
@end itemize

On error, the debug hooks use the @ref{11b,,tracemalloc} module to get the
traceback where a memory block was allocated. The traceback is only displayed
if @ref{11b,,tracemalloc} is tracing Python memory allocations and the memory block
was traced.

Let @emph{S} = @code{sizeof(size_t)}. @code{2*S} bytes are added at each end of each block
of @emph{N} bytes requested.  The memory layout is like so, where p represents the
address returned by a malloc-like or realloc-like function (@code{p[i:j]} means
the slice of bytes from @code{*(p+i)} inclusive up to @code{*(p+j)} exclusive; note
that the treatment of negative indices differs from a Python slice):


@table @asis

@item @code{p[-2*S:-S]}

Number of bytes originally asked for.  This is a size_t, big-endian (easier
to read in a memory dump).

@item @code{p[-S]}

API identifier (ASCII character):


@itemize *

@item 
@code{'r'} for @code{PYMEM_DOMAIN_RAW}.

@item 
@code{'m'} for @code{PYMEM_DOMAIN_MEM}.

@item 
@code{'o'} for @code{PYMEM_DOMAIN_OBJ}.
@end itemize

@item @code{p[-S+1:0]}

Copies of PYMEM_FORBIDDENBYTE.  Used to catch under- writes and reads.

@item @code{p[0:N]}

The requested memory, filled with copies of PYMEM_CLEANBYTE, used to catch
reference to uninitialized memory.  When a realloc-like function is called
requesting a larger memory block, the new excess bytes are also filled with
PYMEM_CLEANBYTE.  When a free-like function is called, these are
overwritten with PYMEM_DEADBYTE, to catch reference to freed memory.  When
a realloc- like function is called requesting a smaller memory block, the
excess old bytes are also filled with PYMEM_DEADBYTE.

@item @code{p[N:N+S]}

Copies of PYMEM_FORBIDDENBYTE.  Used to catch over- writes and reads.

@item @code{p[N+S:N+2*S]}

Only used if the @code{PYMEM_DEBUG_SERIALNO} macro is defined (not defined by
default).

A serial number, incremented by 1 on each call to a malloc-like or
realloc-like function.  Big-endian @code{size_t}.  If “bad memory” is detected
later, the serial number gives an excellent way to set a breakpoint on the
next run, to capture the instant at which this block was passed out.  The
static function bumpserialno() in obmalloc.c is the only place the serial
number is incremented, and exists so you can set such a breakpoint easily.
@end table

A realloc-like or free-like function first checks that the PYMEM_FORBIDDENBYTE
bytes at each end are intact.  If they’ve been altered, diagnostic output is
written to stderr, and the program is aborted via Py_FatalError().  The other
main failure mode is provoking a memory error when a program reads up one of
the special bit patterns and tries to use it as an address.  If you get in a
debugger then and look at the object, you’re likely to see that it’s entirely
filled with PYMEM_DEADBYTE (meaning freed memory is getting used) or
PYMEM_CLEANBYTE (meaning uninitialized memory is getting used).


Changed in version 3.6: The @ref{7b0,,PyMem_SetupDebugHooks()} function now also works on Python
compiled in release mode.  On error, the debug hooks now use
@ref{11b,,tracemalloc} to get the traceback where a memory block was allocated.
The debug hooks now also check if the GIL is held when functions of
@code{PYMEM_DOMAIN_OBJ} and @code{PYMEM_DOMAIN_MEM} domains are
called.



Changed in version 3.8: Byte patterns @code{0xCB} (@code{PYMEM_CLEANBYTE}), @code{0xDB} (@code{PYMEM_DEADBYTE})
and @code{0xFB} (@code{PYMEM_FORBIDDENBYTE}) have been replaced with @code{0xCD},
@code{0xDD} and @code{0xFD} to use the same values than Windows CRT debug
@code{malloc()} and @code{free()}.


@node The pymalloc allocator,tracemalloc C API,Debug hooks on the Python memory allocators,Memory Management
@anchor{c-api/memory pymalloc}@anchor{85c}@anchor{c-api/memory the-pymalloc-allocator}@anchor{4596}
@subsection The pymalloc allocator


Python has a @emph{pymalloc} allocator optimized for small objects (smaller or equal
to 512 bytes) with a short lifetime. It uses memory mappings called “arenas”
with a fixed size of 256 KiB. It falls back to @ref{bc4,,PyMem_RawMalloc()} and
@ref{bc5,,PyMem_RawRealloc()} for allocations larger than 512 bytes.

@emph{pymalloc} is the @ref{1626,,default allocator} of the
@code{PYMEM_DOMAIN_MEM} (ex: @ref{7ad,,PyMem_Malloc()}) and
@code{PYMEM_DOMAIN_OBJ} (ex: @ref{7af,,PyObject_Malloc()}) domains.

The arena allocator uses the following functions:


@itemize *

@item 
@code{VirtualAlloc()} and @code{VirtualFree()} on Windows,

@item 
@code{mmap()} and @code{munmap()} if available,

@item 
@code{malloc()} and @code{free()} otherwise.
@end itemize

This allocator is disabled if Python is configured with the
@ref{1656,,--without-pymalloc} option. It can also be disabled at runtime using
the 
@geindex PYTHONMALLOC
@geindex environment variable; PYTHONMALLOC
@ref{7ab,,PYTHONMALLOC} environment variable (ex: @code{PYTHONMALLOC=malloc}).

@menu
* Customize pymalloc Arena Allocator:: 

@end menu

@node Customize pymalloc Arena Allocator,,,The pymalloc allocator
@anchor{c-api/memory customize-pymalloc-arena-allocator}@anchor{4597}
@subsubsection Customize pymalloc Arena Allocator



New in version 3.4.


@geindex PyObjectArenaAllocator (C type)
@anchor{c-api/memory c PyObjectArenaAllocator}@anchor{4598}
@deffn {C Type} type  PyObjectArenaAllocator

Structure used to describe an arena allocator. The structure has
three fields:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Field

@tab

Meaning

@item

@code{void *ctx}

@tab

user context passed as first argument

@item

@code{void* alloc(void *ctx, size_t size)}

@tab

allocate an arena of size bytes

@item

@code{void free(void *ctx, void *ptr, size_t size)}

@tab

free an arena

@end multitable

@end deffn

@geindex PyObject_GetArenaAllocator (C function)
@anchor{c-api/memory c PyObject_GetArenaAllocator}@anchor{44ef}
@deffn {C Function} void  PyObject_GetArenaAllocator (PyObjectArenaAllocator  *allocator)

Get the arena allocator.
@end deffn

@geindex PyObject_SetArenaAllocator (C function)
@anchor{c-api/memory c PyObject_SetArenaAllocator}@anchor{44ed}
@deffn {C Function} void  PyObject_SetArenaAllocator (PyObjectArenaAllocator  *allocator)

Set the arena allocator.
@end deffn

@node tracemalloc C API,Examples<37>,The pymalloc allocator,Memory Management
@anchor{c-api/memory tracemalloc-c-api}@anchor{4599}
@subsection tracemalloc C API



New in version 3.7.


@geindex PyTraceMalloc_Track (C function)
@anchor{c-api/memory c PyTraceMalloc_Track}@anchor{6e9}
@deffn {C Function} int  PyTraceMalloc_Track (unsigned  int  domain, uintptr_t  ptr, size_t  size)

Track an allocated memory block in the @ref{11b,,tracemalloc} module.

Return @code{0} on success, return @code{-1} on error (failed to allocate memory to
store the trace). Return @code{-2} if tracemalloc is disabled.

If memory block is already tracked, update the existing trace.
@end deffn

@geindex PyTraceMalloc_Untrack (C function)
@anchor{c-api/memory c PyTraceMalloc_Untrack}@anchor{6ea}
@deffn {C Function} int  PyTraceMalloc_Untrack (unsigned  int  domain, uintptr_t  ptr)

Untrack an allocated memory block in the @ref{11b,,tracemalloc} module.
Do nothing if the block was not tracked.

Return @code{-2} if tracemalloc is disabled, otherwise return @code{0}.
@end deffn

@node Examples<37>,,tracemalloc C API,Memory Management
@anchor{c-api/memory examples}@anchor{459a}@anchor{c-api/memory memoryexamples}@anchor{459b}
@subsection Examples


Here is the example from section @ref{4584,,Overview}, rewritten so that the
I/O buffer is allocated from the Python heap by using the first function set:

@example
PyObject *res;
char *buf = (char *) PyMem_Malloc(BUFSIZ); /* for I/O */

if (buf == NULL)
    return PyErr_NoMemory();
/* ...Do some I/O operation involving buf... */
res = PyBytes_FromString(buf);
PyMem_Free(buf); /* allocated with PyMem_Malloc */
return res;
@end example

The same code using the type-oriented function set:

@example
PyObject *res;
char *buf = PyMem_New(char, BUFSIZ); /* for I/O */

if (buf == NULL)
    return PyErr_NoMemory();
/* ...Do some I/O operation involving buf... */
res = PyBytes_FromString(buf);
PyMem_Del(buf); /* allocated with PyMem_New */
return res;
@end example

Note that in the two examples above, the buffer is always manipulated via
functions belonging to the same set. Indeed, it is required to use the same
memory API family for a given memory block, so that the risk of mixing different
allocators is reduced to a minimum. The following code sequence contains two
errors, one of which is labeled as @emph{fatal} because it mixes two different
allocators operating on different heaps.

@example
char *buf1 = PyMem_New(char, BUFSIZ);
char *buf2 = (char *) malloc(BUFSIZ);
char *buf3 = (char *) PyMem_Malloc(BUFSIZ);
...
PyMem_Del(buf3);  /* Wrong -- should be PyMem_Free() */
free(buf2);       /* Right -- allocated via malloc() */
free(buf1);       /* Fatal -- should be PyMem_Del()  */
@end example

In addition to the functions aimed at handling raw memory blocks from the Python
heap, objects in Python are allocated and released with @ref{491,,PyObject_New()},
@ref{492,,PyObject_NewVar()} and @ref{100f,,PyObject_Del()}.

These will be explained in the next chapter on defining and implementing new
object types in C.

@node Object Implementation Support,API and ABI Versioning,Memory Management,Python/C API Reference Manual
@anchor{c-api/objimpl doc}@anchor{459c}@anchor{c-api/objimpl newtypes}@anchor{459d}@anchor{c-api/objimpl object-implementation-support}@anchor{459e}
@section Object Implementation Support


This chapter describes the functions, types, and macros used when defining new
object types.

@menu
* Allocating Objects on the Heap:: 
* Common Object Structures:: 
* Type Objects: Type Objects<3>. 
* Number Object Structures:: 
* Mapping Object Structures:: 
* Sequence Object Structures:: 
* Buffer Object Structures:: 
* Async Object Structures:: 
* Slot Type typedefs:: 
* Examples: Examples<38>. 
* Supporting Cyclic Garbage Collection:: 

@end menu

@node Allocating Objects on the Heap,Common Object Structures,,Object Implementation Support
@anchor{c-api/allocation doc}@anchor{459f}@anchor{c-api/allocation allocating-objects}@anchor{45a0}@anchor{c-api/allocation allocating-objects-on-the-heap}@anchor{45a1}
@subsection Allocating Objects on the Heap


@geindex _PyObject_New (C function)
@anchor{c-api/allocation c _PyObject_New}@anchor{45a2}
@deffn {C Function} @ref{375,,PyObject}  *_PyObject_New (PyTypeObject  *type)
@emph{Return value: New reference.}
@end deffn

@geindex _PyObject_NewVar (C function)
@anchor{c-api/allocation c _PyObject_NewVar}@anchor{45a3}
@deffn {C Function} @ref{4293,,PyVarObject}  *_PyObject_NewVar (PyTypeObject  *type, Py_ssize_t  size)
@emph{Return value: New reference.}
@end deffn

@geindex PyObject_Init (C function)
@anchor{c-api/allocation c PyObject_Init}@anchor{581}
@deffn {C Function} @ref{375,,PyObject}  *PyObject_Init (PyObject  *op, PyTypeObject  *type)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
Initialize a newly-allocated object @emph{op} with its type and initial
reference.  Returns the initialized object.  If @emph{type} indicates that the
object participates in the cyclic garbage detector, it is added to the
detector’s set of observed objects. Other fields of the object are not
affected.
@end deffn

@geindex PyObject_InitVar (C function)
@anchor{c-api/allocation c PyObject_InitVar}@anchor{11dd}
@deffn {C Function} @ref{4293,,PyVarObject}  *PyObject_InitVar (PyVarObject  *op, PyTypeObject  *type, Py_ssize_t  size)
@emph{Return value: Borrowed reference.}@emph{ Part of the @ref{35d,,Stable ABI}.}
This does everything @ref{581,,PyObject_Init()} does, and also initializes the
length information for a variable-size object.
@end deffn

@geindex PyObject_New (C function)
@anchor{c-api/allocation c PyObject_New}@anchor{491}
@deffn {C Function} TYPE  *PyObject_New (TYPE, PyTypeObject  *type)
@emph{Return value: New reference.}
Allocate a new Python object using the C structure type @emph{TYPE} and the
Python type object @emph{type}.  Fields not defined by the Python object header
are not initialized; the object’s reference count will be one.  The size of
the memory allocation is determined from the @ref{493,,tp_basicsize} field of
the type object.
@end deffn

@geindex PyObject_NewVar (C function)
@anchor{c-api/allocation c PyObject_NewVar}@anchor{492}
@deffn {C Function} TYPE  *PyObject_NewVar (TYPE, PyTypeObject  *type, Py_ssize_t  size)
@emph{Return value: New reference.}
Allocate a new Python object using the C structure type @emph{TYPE} and the
Python type object @emph{type}.  Fields not defined by the Python object header
are not initialized.  The allocated memory allows for the @emph{TYPE} structure
plus @emph{size} fields of the size given by the @ref{40a4,,tp_itemsize} field of
@emph{type}.  This is useful for implementing objects like tuples, which are
able to determine their size at construction time.  Embedding the array of
fields into the same allocation decreases the number of allocations,
improving the memory management efficiency.
@end deffn

@geindex PyObject_Del (C function)
@anchor{c-api/allocation c PyObject_Del}@anchor{100f}
@deffn {C Function} void  PyObject_Del (void  *op)

Releases memory allocated to an object using @ref{491,,PyObject_New()} or
@ref{492,,PyObject_NewVar()}.  This is normally called from the
@ref{40aa,,tp_dealloc} handler specified in the object’s type.  The fields of
the object should not be accessed after this call as the memory is no
longer a valid Python object.
@end deffn

@geindex _Py_NoneStruct (C var)
@anchor{c-api/allocation c _Py_NoneStruct}@anchor{45a4}
@deffn {C Variable} @ref{375,,PyObject}  _Py_NoneStruct

Object which is visible in Python as @code{None}.  This should only be accessed
using the @ref{4077,,Py_None} macro, which evaluates to a pointer to this
object.
@end deffn


@subsubheading See also



@table @asis

@item @ref{407a,,PyModule_Create()}

To allocate and create extension modules.
@end table


@node Common Object Structures,Type Objects<3>,Allocating Objects on the Heap,Object Implementation Support
@anchor{c-api/structures doc}@anchor{45a5}@anchor{c-api/structures common-object-structures}@anchor{45a6}@anchor{c-api/structures common-structs}@anchor{45a7}
@subsection Common Object Structures


There are a large number of structures which are used in the definition of
object types for Python.  This section describes these structures and how they
are used.

@menu
* Base object types and macros:: 
* Implementing functions and methods:: 
* Accessing attributes of extension types:: 

@end menu

@node Base object types and macros,Implementing functions and methods,,Common Object Structures
@anchor{c-api/structures base-object-types-and-macros}@anchor{45a8}
@subsubsection Base object types and macros


All Python objects ultimately share a small number of fields at the beginning
of the object’s representation in memory.  These are represented by the
@ref{375,,PyObject} and @ref{4293,,PyVarObject} types, which are defined, in turn,
by the expansions of some macros also used, whether directly or indirectly, in
the definition of all other Python objects.

@geindex PyObject (C type)
@anchor{c-api/structures c PyObject}@anchor{375}
@deffn {C Type} type  PyObject
@emph{ Part of the @ref{35d,,Stable ABI}.}
All object types are extensions of this type.  This is a type which
contains the information Python needs to treat a pointer to an object as an
object.  In a normal “release” build, it contains only the object’s
reference count and a pointer to the corresponding type object.
Nothing is actually declared to be a @ref{375,,PyObject}, but every pointer
to a Python object can be cast to a @ref{375,,PyObject}*.  Access to the
members must be done by using the macros @ref{382,,Py_REFCNT} and
@ref{1f1,,Py_TYPE}.
@end deffn

@geindex PyVarObject (C type)
@anchor{c-api/structures c PyVarObject}@anchor{4293}
@deffn {C Type} type  PyVarObject
@emph{ Part of the @ref{35d,,Stable ABI}.}
This is an extension of @ref{375,,PyObject} that adds the @code{ob_size}
field.  This is only used for objects that have some notion of @emph{length}.
This type does not often appear in the Python/C API.
Access to the members must be done by using the macros
@ref{382,,Py_REFCNT}, @ref{1f1,,Py_TYPE}, and @ref{1f3,,Py_SIZE}.
@end deffn

@geindex PyObject_HEAD (C macro)
@anchor{c-api/structures c PyObject_HEAD}@anchor{e7a}
@deffn {C Macro} PyObject_HEAD

This is a macro used when declaring new types which represent objects
without a varying length.  The PyObject_HEAD macro expands to:

@example
PyObject ob_base;
@end example

See documentation of @ref{375,,PyObject} above.
@end deffn

@geindex PyObject_VAR_HEAD (C macro)
@anchor{c-api/structures c PyObject_VAR_HEAD}@anchor{45a9}
@deffn {C Macro} PyObject_VAR_HEAD

This is a macro used when declaring new types which represent objects
with a length that varies from instance to instance.
The PyObject_VAR_HEAD macro expands to:

@example
PyVarObject ob_base;
@end example

See documentation of @ref{4293,,PyVarObject} above.
@end deffn

@geindex Py_Is (C function)
@anchor{c-api/structures c Py_Is}@anchor{377}
@deffn {C Function} int  Py_Is (const  PyObject  *x, const  PyObject  *y)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Test if the @emph{x} object is the @emph{y} object, the same as @code{x is y} in Python.


New in version 3.10.

@end deffn

@geindex Py_IsNone (C function)
@anchor{c-api/structures c Py_IsNone}@anchor{378}
@deffn {C Function} int  Py_IsNone (const  PyObject  *x)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Test if an object is the @code{None} singleton,
the same as @code{x is None} in Python.


New in version 3.10.

@end deffn

@geindex Py_IsTrue (C function)
@anchor{c-api/structures c Py_IsTrue}@anchor{379}
@deffn {C Function} int  Py_IsTrue (const  PyObject  *x)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Test if an object is the @code{True} singleton,
the same as @code{x is True} in Python.


New in version 3.10.

@end deffn

@geindex Py_IsFalse (C function)
@anchor{c-api/structures c Py_IsFalse}@anchor{37a}
@deffn {C Function} int  Py_IsFalse (const  PyObject  *x)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Test if an object is the @code{False} singleton,
the same as @code{x is False} in Python.


New in version 3.10.

@end deffn

@geindex Py_TYPE (C function)
@anchor{c-api/structures c Py_TYPE}@anchor{1f1}
@deffn {C Function} @ref{5c6,,PyTypeObject}  *Py_TYPE (const  PyObject  *o)

Get the type of the Python object @emph{o}.

Return a @ref{4112,,borrowed reference}.

Use the @ref{1f2,,Py_SET_TYPE()} function to set an object type.


Changed in version 3.11: @ref{1f1,,Py_TYPE()} is changed to an inline static function.

@end deffn

@geindex Py_IS_TYPE (C function)
@anchor{c-api/structures c Py_IS_TYPE}@anchor{112d}
@deffn {C Function} int  Py_IS_TYPE (PyObject  *o, PyTypeObject  *type)

Return non-zero if the object @emph{o} type is @emph{type}. Return zero otherwise.
Equivalent to: @code{Py_TYPE(o) == type}.


New in version 3.9.

@end deffn

@geindex Py_SET_TYPE (C function)
@anchor{c-api/structures c Py_SET_TYPE}@anchor{1f2}
@deffn {C Function} void  Py_SET_TYPE (PyObject  *o, PyTypeObject  *type)

Set the object @emph{o} type to @emph{type}.


New in version 3.9.

@end deffn

@geindex Py_REFCNT (C function)
@anchor{c-api/structures c Py_REFCNT}@anchor{382}
@deffn {C Function} Py_ssize_t  Py_REFCNT (const  PyObject  *o)

Get the reference count of the Python object @emph{o}.

Use the @ref{383,,Py_SET_REFCNT()} function to set an object reference count.


Changed in version 3.10: @ref{382,,Py_REFCNT()} is changed to the inline static function.

@end deffn

@geindex Py_SET_REFCNT (C function)
@anchor{c-api/structures c Py_SET_REFCNT}@anchor{383}
@deffn {C Function} void  Py_SET_REFCNT (PyObject  *o, Py_ssize_t  refcnt)

Set the object @emph{o} reference counter to @emph{refcnt}.


New in version 3.9.

@end deffn

@geindex Py_SIZE (C function)
@anchor{c-api/structures c Py_SIZE}@anchor{1f3}
@deffn {C Function} Py_ssize_t  Py_SIZE (const  PyVarObject  *o)

Get the size of the Python object @emph{o}.

Use the @ref{1f4,,Py_SET_SIZE()} function to set an object size.


Changed in version 3.11: @ref{1f3,,Py_SIZE()} is changed to an inline static function.

@end deffn

@geindex Py_SET_SIZE (C function)
@anchor{c-api/structures c Py_SET_SIZE}@anchor{1f4}
@deffn {C Function} void  Py_SET_SIZE (PyVarObject  *o, Py_ssize_t  size)

Set the object @emph{o} size to @emph{size}.


New in version 3.9.

@end deffn

@geindex PyObject_HEAD_INIT (C macro)
@anchor{c-api/structures c PyObject_HEAD_INIT}@anchor{45aa}
@deffn {C Macro} PyObject_HEAD_INIT (type)

This is a macro which expands to initialization values for a new
@ref{375,,PyObject} type.  This macro expands to:

@example
_PyObject_EXTRA_INIT
1, type,
@end example
@end deffn

@geindex PyVarObject_HEAD_INIT (C macro)
@anchor{c-api/structures c PyVarObject_HEAD_INIT}@anchor{45ab}
@deffn {C Macro} PyVarObject_HEAD_INIT (type, size)

This is a macro which expands to initialization values for a new
@ref{4293,,PyVarObject} type, including the @code{ob_size} field.
This macro expands to:

@example
_PyObject_EXTRA_INIT
1, type, size,
@end example
@end deffn

@node Implementing functions and methods,Accessing attributes of extension types,Base object types and macros,Common Object Structures
@anchor{c-api/structures implementing-functions-and-methods}@anchor{45ac}
@subsubsection Implementing functions and methods


@geindex PyCFunction (C type)
@anchor{c-api/structures c PyCFunction}@anchor{fd4}
@deffn {C Type} type  PyCFunction
@emph{ Part of the @ref{35d,,Stable ABI}.}
Type of the functions used to implement most Python callables in C.
Functions of this type take two @ref{375,,PyObject}* parameters and return
one such value.  If the return value is @code{NULL}, an exception shall have
been set.  If not @code{NULL}, the return value is interpreted as the return
value of the function as exposed in Python.  The function must return a new
reference.

The function signature is:

@example
PyObject *PyCFunction(PyObject *self,
                      PyObject *args);
@end example
@end deffn

@geindex PyCFunctionWithKeywords (C type)
@anchor{c-api/structures c PyCFunctionWithKeywords}@anchor{413a}
@deffn {C Type} type  PyCFunctionWithKeywords
@emph{ Part of the @ref{35d,,Stable ABI}.}
Type of the functions used to implement Python callables in C
with signature @code{METH_VARARGS | METH_KEYWORDS}.
The function signature is:

@example
PyObject *PyCFunctionWithKeywords(PyObject *self,
                                  PyObject *args,
                                  PyObject *kwargs);
@end example
@end deffn

@geindex _PyCFunctionFast (C type)
@anchor{c-api/structures c _PyCFunctionFast}@anchor{45ad}
@deffn {C Type} type  _PyCFunctionFast

Type of the functions used to implement Python callables in C
with signature @ref{1287,,METH_FASTCALL}.
The function signature is:

@example
PyObject *_PyCFunctionFast(PyObject *self,
                           PyObject *const *args,
                           Py_ssize_t nargs);
@end example
@end deffn

@geindex _PyCFunctionFastWithKeywords (C type)
@anchor{c-api/structures c _PyCFunctionFastWithKeywords}@anchor{45ae}
@deffn {C Type} type  _PyCFunctionFastWithKeywords

Type of the functions used to implement Python callables in C
with signature @code{METH_FASTCALL | METH_KEYWORDS}.
The function signature is:

@example
PyObject *_PyCFunctionFastWithKeywords(PyObject *self,
                                       PyObject *const *args,
                                       Py_ssize_t nargs,
                                       PyObject *kwnames);
@end example
@end deffn

@geindex PyCMethod (C type)
@anchor{c-api/structures c PyCMethod}@anchor{1ea}
@deffn {C Type} type  PyCMethod

Type of the functions used to implement Python callables in C
with signature @code{METH_METHOD | METH_FASTCALL | METH_KEYWORDS}.
The function signature is:

@example
PyObject *PyCMethod(PyObject *self,
                    PyTypeObject *defining_class,
                    PyObject *const *args,
                    Py_ssize_t nargs,
                    PyObject *kwnames)
@end example


New in version 3.9.

@end deffn

@geindex PyMethodDef (C type)
@anchor{c-api/structures c PyMethodDef}@anchor{1014}
@deffn {C Type} type  PyMethodDef
@emph{ Part of the @ref{35d,,Stable ABI}.}
Structure used to describe a method of an extension type.  This structure has
four fields:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Field

@tab

C Type

@tab

Meaning

@item

@code{ml_name}

@tab

const char *

@tab

name of the method

@item

@code{ml_meth}

@tab

PyCFunction

@tab

pointer to the C
implementation

@item

@code{ml_flags}

@tab

int

@tab

flag bits indicating how the
call should be constructed

@item

@code{ml_doc}

@tab

const char *

@tab

points to the contents of the
docstring

@end multitable

@end deffn

The @code{ml_meth} is a C function pointer.  The functions may be of different
types, but they always return @ref{375,,PyObject}*.  If the function is not of
the @ref{fd4,,PyCFunction}, the compiler will require a cast in the method table.
Even though @ref{fd4,,PyCFunction} defines the first parameter as
@ref{375,,PyObject}*, it is common that the method implementation uses the
specific C type of the @emph{self} object.

The @code{ml_flags} field is a bitfield which can include the following flags.
The individual flags indicate either a calling convention or a binding
convention.

There are these calling conventions:

@geindex METH_VARARGS (built-in variable)
@anchor{c-api/structures METH_VARARGS}@anchor{103e}
@deffn {Data} METH_VARARGS

This is the typical calling convention, where the methods have the type
@ref{fd4,,PyCFunction}. The function expects two @ref{375,,PyObject}* values.
The first one is the @emph{self} object for methods; for module functions, it is
the module object.  The second parameter (often called @emph{args}) is a tuple
object representing all arguments. This parameter is typically processed
using @ref{37f,,PyArg_ParseTuple()} or @ref{103c,,PyArg_UnpackTuple()}.
@end deffn


@deffn {Data} METH_VARARGS | METH_KEYWORDS

Methods with these flags must be of type @ref{413a,,PyCFunctionWithKeywords}.
The function expects three parameters: @emph{self}, @emph{args}, @emph{kwargs} where
@emph{kwargs} is a dictionary of all the keyword arguments or possibly @code{NULL}
if there are no keyword arguments.  The parameters are typically processed
using @ref{863,,PyArg_ParseTupleAndKeywords()}.
@end deffn

@geindex METH_FASTCALL (built-in variable)
@anchor{c-api/structures METH_FASTCALL}@anchor{1287}
@deffn {Data} METH_FASTCALL

Fast calling convention supporting only positional arguments.
The methods have the type @ref{45ad,,_PyCFunctionFast}.
The first parameter is @emph{self}, the second parameter is a C array
of @ref{375,,PyObject}* values indicating the arguments and the third
parameter is the number of arguments (the length of the array).


New in version 3.7.



Changed in version 3.10: @code{METH_FASTCALL} is now part of the stable ABI.

@end deffn


@deffn {Data} METH_FASTCALL | METH_KEYWORDS

Extension of @ref{1287,,METH_FASTCALL} supporting also keyword arguments,
with methods of type @ref{45ae,,_PyCFunctionFastWithKeywords}.
Keyword arguments are passed the same way as in the
@ref{4bd,,vectorcall protocol}:
there is an additional fourth @ref{375,,PyObject}* parameter
which is a tuple representing the names of the keyword arguments
(which are guaranteed to be strings)
or possibly @code{NULL} if there are no keywords.  The values of the keyword
arguments are stored in the @emph{args} array, after the positional arguments.

This is not part of the @ref{35d,,limited API}.


New in version 3.7.

@end deffn


@deffn {Data} METH_METHOD | METH_FASTCALL | METH_KEYWORDS

Extension of @code{METH_FASTCALL | METH_KEYWORDS} supporting the @emph{defining
class}, that is, the class that contains the method in question.
The defining class might be a superclass of @code{Py_TYPE(self)}.

The method needs to be of type @ref{1ea,,PyCMethod}, the same as for
@code{METH_FASTCALL | METH_KEYWORDS} with @code{defining_class} argument added after
@code{self}.


New in version 3.9.

@end deffn

@geindex METH_NOARGS (built-in variable)
@anchor{c-api/structures METH_NOARGS}@anchor{1011}
@deffn {Data} METH_NOARGS

Methods without parameters don’t need to check whether arguments are given if
they are listed with the @ref{1011,,METH_NOARGS} flag.  They need to be of type
@ref{fd4,,PyCFunction}.  The first parameter is typically named @emph{self} and will
hold a reference to the module or object instance.  In all cases the second
parameter will be @code{NULL}.
@end deffn

@geindex METH_O (built-in variable)
@anchor{c-api/structures METH_O}@anchor{103d}
@deffn {Data} METH_O

Methods with a single object argument can be listed with the @ref{103d,,METH_O}
flag, instead of invoking @ref{37f,,PyArg_ParseTuple()} with a @code{"O"} argument.
They have the type @ref{fd4,,PyCFunction}, with the @emph{self} parameter, and a
@ref{375,,PyObject}* parameter representing the single argument.
@end deffn

These two constants are not used to indicate the calling convention but the
binding when use with methods of classes.  These may not be used for functions
defined for modules.  At most one of these flags may be set for any given
method.

@geindex METH_CLASS (built-in variable)
@anchor{c-api/structures METH_CLASS}@anchor{1012}
@deffn {Data} METH_CLASS

@geindex built-in function; classmethod

The method will be passed the type object as the first parameter rather
than an instance of the type.  This is used to create @emph{class methods},
similar to what is created when using the @ref{254,,classmethod()} built-in
function.
@end deffn

@geindex METH_STATIC (built-in variable)
@anchor{c-api/structures METH_STATIC}@anchor{1013}
@deffn {Data} METH_STATIC

@geindex built-in function; staticmethod

The method will be passed @code{NULL} as the first parameter rather than an
instance of the type.  This is used to create @emph{static methods}, similar to
what is created when using the @ref{253,,staticmethod()} built-in function.
@end deffn

One other constant controls whether a method is loaded in place of another
definition with the same method name.

@geindex METH_COEXIST (built-in variable)
@anchor{c-api/structures METH_COEXIST}@anchor{45af}
@deffn {Data} METH_COEXIST

The method will be loaded in place of existing definitions.  Without
@emph{METH_COEXIST}, the default is to skip repeated definitions.  Since slot
wrappers are loaded before the method table, the existence of a
@emph{sq_contains} slot, for example, would generate a wrapped method named
@code{__contains__()} and preclude the loading of a corresponding
PyCFunction with the same name.  With the flag defined, the PyCFunction
will be loaded in place of the wrapper object and will co-exist with the
slot.  This is helpful because calls to PyCFunctions are optimized more
than wrapper object calls.
@end deffn

@node Accessing attributes of extension types,,Implementing functions and methods,Common Object Structures
@anchor{c-api/structures accessing-attributes-of-extension-types}@anchor{45b0}
@subsubsection Accessing attributes of extension types


@geindex PyMemberDef (C type)
@anchor{c-api/structures c PyMemberDef}@anchor{6eb}
@deffn {C Type} type  PyMemberDef
@emph{ Part of the @ref{35d,,Stable ABI}.}
Structure which describes an attribute of a type which corresponds to a C
struct member.  Its fields are:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Field

@tab

C Type

@tab

Meaning

@item

@code{name}

@tab

const char *

@tab

name of the member

@item

@code{type}

@tab

int

@tab

the type of the member in the
C struct

@item

@code{offset}

@tab

Py_ssize_t

@tab

the offset in bytes that the
member is located on the
type’s object struct

@item

@code{flags}

@tab

int

@tab

flag bits indicating if the
field should be read-only or
writable

@item

@code{doc}

@tab

const char *

@tab

points to the contents of the
docstring

@end multitable


@code{type} can be one of many @code{T_} macros corresponding to various C
types.  When the member is accessed in Python, it will be converted to the
equivalent Python type.


@multitable {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} 
@headitem

Macro name

@tab

C type

@item

T_SHORT

@tab

short

@item

T_INT

@tab

int

@item

T_LONG

@tab

long

@item

T_FLOAT

@tab

float

@item

T_DOUBLE

@tab

double

@item

T_STRING

@tab

const char *

@item

T_OBJECT

@tab

PyObject *

@item

T_OBJECT_EX

@tab

PyObject *

@item

T_CHAR

@tab

char

@item

T_BYTE

@tab

char

@item

T_UBYTE

@tab

unsigned char

@item

T_UINT

@tab

unsigned int

@item

T_USHORT

@tab

unsigned short

@item

T_ULONG

@tab

unsigned long

@item

T_BOOL

@tab

char

@item

T_LONGLONG

@tab

long long

@item

T_ULONGLONG

@tab

unsigned long long

@item

T_PYSSIZET

@tab

Py_ssize_t

@end multitable


@code{T_OBJECT} and @code{T_OBJECT_EX} differ in that
@code{T_OBJECT} returns @code{None} if the member is @code{NULL} and
@code{T_OBJECT_EX} raises an @ref{163,,AttributeError}.  Try to use
@code{T_OBJECT_EX} over @code{T_OBJECT} because @code{T_OBJECT_EX}
handles use of the @ref{152e,,del} statement on that attribute more correctly
than @code{T_OBJECT}.

@code{flags} can be @code{0} for write and read access or @code{READONLY} for
read-only access.  Using @code{T_STRING} for @ref{89e,,type} implies
@code{READONLY}.  @code{T_STRING} data is interpreted as UTF-8.
Only @code{T_OBJECT} and @code{T_OBJECT_EX}
members can be deleted.  (They are set to @code{NULL}).
@anchor{c-api/structures pymemberdef-offsets}@anchor{43a3}
Heap allocated types (created using @ref{469,,PyType_FromSpec()} or similar),
@code{PyMemberDef} may contain definitions for the special members
@code{__dictoffset__}, @code{__weaklistoffset__} and @code{__vectorcalloffset__},
corresponding to
@ref{43a4,,tp_dictoffset},
@ref{494,,tp_weaklistoffset} and
@ref{4337,,tp_vectorcall_offset} in type objects.
These must be defined with @code{T_PYSSIZET} and @code{READONLY}, for example:

@example
static PyMemberDef spam_type_members[] = @{
    @{"__dictoffset__", T_PYSSIZET, offsetof(Spam_object, dict), READONLY@},
    @{NULL@}  /* Sentinel */
@};
@end example
@end deffn

@geindex PyMember_GetOne (C function)
@anchor{c-api/structures c PyMember_GetOne}@anchor{45b1}
@deffn {C Function} @ref{375,,PyObject}  *PyMember_GetOne (const  char  *obj_addr, struct  PyMemberDef  *m)

Get an attribute belonging to the object at address @emph{obj_addr}.  The
attribute is described by @code{PyMemberDef} @emph{m}.  Returns @code{NULL}
on error.
@end deffn

@geindex PyMember_SetOne (C function)
@anchor{c-api/structures c PyMember_SetOne}@anchor{45b2}
@deffn {C Function} int  PyMember_SetOne (char  *obj_addr, struct  PyMemberDef  *m, PyObject  *o)

Set an attribute belonging to the object at address @emph{obj_addr} to object @emph{o}.
The attribute to set is described by @code{PyMemberDef} @emph{m}.  Returns @code{0}
if successful and a negative value on failure.
@end deffn

@geindex PyGetSetDef (C type)
@anchor{c-api/structures c PyGetSetDef}@anchor{6ec}
@deffn {C Type} type  PyGetSetDef
@emph{ Part of the @ref{35d,,Stable ABI}.}
Structure to define property-like access for a type. See also description of
the @ref{40b3,,PyTypeObject.tp_getset} slot.


@multitable {xxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Field

@tab

C Type

@tab

Meaning

@item

name

@tab

const char *

@tab

attribute name

@item

get

@tab

getter

@tab

C Function to get the attribute

@item

set

@tab

setter

@tab

optional C function to set or
delete the attribute, if omitted
the attribute is readonly

@item

doc

@tab

const char *

@tab

optional docstring

@item

closure

@tab

void *

@tab

optional function pointer,
providing additional data for
getter and setter

@end multitable


The @code{get} function takes one @ref{375,,PyObject}* parameter (the
instance) and a function pointer (the associated @code{closure}):

@example
typedef PyObject *(*getter)(PyObject *, void *);
@end example

It should return a new reference on success or @code{NULL} with a set exception
on failure.

@code{set} functions take two @ref{375,,PyObject}* parameters (the instance and
the value to be set) and a function pointer (the associated @code{closure}):

@example
typedef int (*setter)(PyObject *, PyObject *, void *);
@end example

In case the attribute should be deleted the second parameter is @code{NULL}.
Should return @code{0} on success or @code{-1} with a set exception on failure.
@end deffn

@node Type Objects<3>,Number Object Structures,Common Object Structures,Object Implementation Support
@anchor{c-api/typeobj doc}@anchor{45b3}@anchor{c-api/typeobj type-objects}@anchor{45b4}@anchor{c-api/typeobj type-structs}@anchor{499}
@subsection Type Objects


Perhaps one of the most important structures of the Python object system is the
structure that defines a new type: the @ref{5c6,,PyTypeObject} structure.  Type
objects can be handled using any of the PyObject_* or
PyType_* functions, but do not offer much that’s interesting to most
Python applications. These objects are fundamental to how objects behave, so
they are very important to the interpreter itself and to any extension module
that implements new types.

Type objects are fairly large compared to most of the standard types. The reason
for the size is that each type object stores a large number of values, mostly C
function pointers, each of which implements a small part of the type’s
functionality.  The fields of the type object are examined in detail in this
section.  The fields will be described in the order in which they occur in the
structure.

In addition to the following quick reference, the @ref{45b5,,Examples}
section provides at-a-glance insight into the meaning and use of
@ref{5c6,,PyTypeObject}.

@menu
* Quick Reference:: 
* PyTypeObject Definition:: 
* PyObject Slots:: 
* PyVarObject Slots:: 
* PyTypeObject Slots:: 
* Static Types:: 
* Heap Types:: 

@end menu

@node Quick Reference,PyTypeObject Definition,,Type Objects<3>
@anchor{c-api/typeobj quick-reference}@anchor{45b6}
@subsubsection Quick Reference


@menu
* “tp slots”:: 
* sub-slots:: 
* slot typedefs:: 

@end menu

@node “tp slots”,sub-slots,,Quick Reference
@anchor{c-api/typeobj tp-slots}@anchor{45b7}@anchor{c-api/typeobj tp-slots-table}@anchor{45b8}
@subsubsection “tp slots”



@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} {xxx} {xxx} {xxx} {xxx} 
@headitem

PyTypeObject Slot @footnote{@w{(1)} 
A slot name in parentheses indicates it is (effectively) deprecated.
Names in angle brackets should be treated as read-only.
Names in square brackets are for internal use only.
“<R>” (as a prefix) means the field is required (must be non-@code{NULL}).
}

@tab

@ref{45b9,,Type}

@tab

special
methods/attrs

@tab

Info @footnote{@w{(2)} 
Columns:

@strong{“O”}:  set on @code{PyBaseObject_Type}

@strong{“T”}:  set on @ref{4255,,PyType_Type}

@strong{“D”}:  default (if slot is set to @code{NULL})

@example
X - PyType_Ready sets this value if it is NULL
~ - PyType_Ready always sets this value (it should be NULL)
? - PyType_Ready may set this value depending on other slots

Also see the inheritance column ("I").
@end example

@strong{“I”}:  inheritance

@example
X - type slot is inherited via *PyType_Ready* if defined with a *NULL* value
% - the slots of the sub-struct are inherited individually
G - inherited, but only in combination with other slots; see the slot's description
? - it's complicated; see the slot's description
@end example

Note that some slots are effectively inherited through the normal
attribute lookup chain.
}

@tab

@tab

@tab

@item

O

@tab

T

@tab

D

@tab

I

@item

<R> @ref{116b,,tp_name}

@tab

const char *

@tab

__name__

@tab

X

@tab

X

@tab

@tab

@item

@ref{493,,tp_basicsize}

@tab

Py_ssize_t

@tab

@tab

X

@tab

X

@tab

@tab

X

@item

@ref{40a4,,tp_itemsize}

@tab

Py_ssize_t

@tab

@tab

@tab

X

@tab

@tab

X

@item

@ref{40aa,,tp_dealloc}

@tab

@ref{42a9,,destructor}

@tab

@tab

X

@tab

X

@tab

@tab

X

@item

@ref{4337,,tp_vectorcall_offset}

@tab

Py_ssize_t

@tab

@tab

@tab

X

@tab

@tab

X

@item

(@ref{40c8,,tp_getattr})

@tab

@ref{42aa,,getattrfunc}

@tab

__getattribute__,
__getattr__

@tab

@tab

@tab

@tab

G

@item

(@ref{40c9,,tp_setattr})

@tab

@ref{42b7,,setattrfunc}

@tab

__setattr__,
__delattr__

@tab

@tab

@tab

@tab

G

@item

@ref{45ba,,tp_as_async}

@tab

@ref{439b,,PyAsyncMethods} *

@tab

@ref{45bb,,sub-slots}

@tab

@tab

@tab

@tab

%

@item

@ref{40c1,,tp_repr}

@tab

@ref{42b5,,reprfunc}

@tab

__repr__

@tab

X

@tab

X

@tab

@tab

X

@item

@ref{497,,tp_as_number}

@tab

@ref{f7b,,PyNumberMethods} *

@tab

@ref{45bb,,sub-slots}

@tab

@tab

@tab

@tab

%

@item

@ref{45bc,,tp_as_sequence}

@tab

@ref{40cf,,PySequenceMethods} *

@tab

@ref{45bb,,sub-slots}

@tab

@tab

@tab

@tab

%

@item

@ref{45bd,,tp_as_mapping}

@tab

@ref{40d0,,PyMappingMethods} *

@tab

@ref{45bb,,sub-slots}

@tab

@tab

@tab

@tab

%

@item

@ref{40d1,,tp_hash}

@tab

@ref{42ad,,hashfunc}

@tab

__hash__

@tab

X

@tab

@tab

@tab

G

@item

@ref{40d2,,tp_call}

@tab

@ref{42bb,,ternaryfunc}

@tab

__call__

@tab

@tab

X

@tab

@tab

X

@item

@ref{40c2,,tp_str}

@tab

@ref{42b5,,reprfunc}

@tab

__str__

@tab

X

@tab

@tab

@tab

X

@item

@ref{40c5,,tp_getattro}

@tab

@ref{42ab,,getattrofunc}

@tab

__getattribute__,
__getattr__

@tab

X

@tab

X

@tab

@tab

G

@item

@ref{40c6,,tp_setattro}

@tab

@ref{42b8,,setattrofunc}

@tab

__setattr__,
__delattr__

@tab

X

@tab

X

@tab

@tab

G

@item

@ref{495,,tp_as_buffer}

@tab

@ref{43a5,,PyBufferProcs} *

@tab

@tab

@tab

@tab

@tab

%

@item

@ref{1133,,tp_flags}

@tab

unsigned long

@tab

@tab

X

@tab

X

@tab

@tab

?

@item

@ref{40a6,,tp_doc}

@tab

const char *

@tab

__doc__

@tab

X

@tab

X

@tab

@tab

@item

@ref{1ee,,tp_traverse}

@tab

@ref{42bc,,traverseproc}

@tab

@tab

@tab

X

@tab

@tab

G

@item

@ref{40bf,,tp_clear}

@tab

@ref{42af,,inquiry}

@tab

@tab

@tab

X

@tab

@tab

G

@item

@ref{40cb,,tp_richcompare}

@tab

@ref{42b6,,richcmpfunc}

@tab

__lt__,
__le__,
__eq__,
__ne__,
__gt__,
__ge__

@tab

X

@tab

@tab

@tab

G

@item

@ref{494,,tp_weaklistoffset}

@tab

Py_ssize_t

@tab

@tab

@tab

X

@tab

@tab

?

@item

@ref{1028,,tp_iter}

@tab

@ref{42ac,,getiterfunc}

@tab

__iter__

@tab

@tab

@tab

@tab

X

@item

@ref{1029,,tp_iternext}

@tab

@ref{42b0,,iternextfunc}

@tab

__next__

@tab

@tab

@tab

@tab

X

@item

@ref{40b0,,tp_methods}

@tab

@ref{1014,,PyMethodDef} []

@tab

@tab

X

@tab

X

@tab

@tab

@item

@ref{40ae,,tp_members}

@tab

@ref{6eb,,PyMemberDef} []

@tab

@tab

@tab

X

@tab

@tab

@item

@ref{40b3,,tp_getset}

@tab

@ref{6ec,,PyGetSetDef} []

@tab

@tab

X

@tab

X

@tab

@tab

@item

@ref{40b7,,tp_base}

@tab

@ref{5c6,,PyTypeObject} *

@tab

__base__

@tab

@tab

@tab

X

@tab

@item

@ref{439d,,tp_dict}

@tab

@ref{375,,PyObject} *

@tab

__dict__

@tab

@tab

@tab

?

@tab

@item

@ref{45be,,tp_descr_get}

@tab

@ref{42a7,,descrgetfunc}

@tab

__get__

@tab

@tab

@tab

@tab

X

@item

@ref{116e,,tp_descr_set}

@tab

@ref{42a8,,descrsetfunc}

@tab

__set__,
__delete__

@tab

@tab

@tab

@tab

X

@item

@ref{43a4,,tp_dictoffset}

@tab

Py_ssize_t

@tab

@tab

@tab

X

@tab

@tab

?

@item

@ref{40ad,,tp_init}

@tab

@ref{42ae,,initproc}

@tab

__init__

@tab

X

@tab

X

@tab

@tab

X

@item

@ref{40ac,,tp_alloc}

@tab

@ref{42a5,,allocfunc}

@tab

@tab

X

@tab

@tab

?

@tab

?

@item

@ref{40a7,,tp_new}

@tab

@ref{42b2,,newfunc}

@tab

__new__

@tab

X

@tab

X

@tab

?

@tab

?

@item

@ref{40ab,,tp_free}

@tab

@ref{4477,,freefunc}

@tab

@tab

X

@tab

X

@tab

?

@tab

?

@item

@ref{45bf,,tp_is_gc}

@tab

@ref{42af,,inquiry}

@tab

@tab

@tab

X

@tab

@tab

X

@item

<@ref{45c0,,tp_bases}>

@tab

@ref{375,,PyObject} *

@tab

__bases__

@tab

@tab

@tab

~

@tab

@item

<@ref{439e,,tp_mro}>

@tab

@ref{375,,PyObject} *

@tab

__mro__

@tab

@tab

@tab

~

@tab

@item

[@ref{439f,,tp_cache}]

@tab

@ref{375,,PyObject} *

@tab

@tab

@tab

@tab

@tab

@item

[@ref{43a0,,tp_subclasses}]

@tab

@ref{375,,PyObject} *

@tab

__subclasses__

@tab

@tab

@tab

@tab

@item

[@ref{43a1,,tp_weaklist}]

@tab

@ref{375,,PyObject} *

@tab

@tab

@tab

@tab

@tab

@item

(@ref{45c1,,tp_del})

@tab

@ref{42a9,,destructor}

@tab

@tab

@tab

@tab

@tab

@item

[@ref{45c2,,tp_version_tag}]

@tab

unsigned int

@tab

@tab

@tab

@tab

@tab

@item

@ref{5c7,,tp_finalize}

@tab

@ref{42a9,,destructor}

@tab

__del__

@tab

@tab

@tab

@tab

X

@item

@ref{43a2,,tp_vectorcall}

@tab

@ref{4338,,vectorcallfunc}

@tab

@tab

@tab

@tab

@tab

@end multitable


@node sub-slots,slot typedefs,“tp slots”,Quick Reference
@anchor{c-api/typeobj id3}@anchor{45c3}@anchor{c-api/typeobj sub-slots}@anchor{45bb}
@subsubsection sub-slots



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxx} 
@headitem

Slot

@tab

@ref{45b9,,Type}

@tab

special
methods

@item

@ref{45c4,,am_await}

@tab

@ref{42bd,,unaryfunc}

@tab

__await__

@item

@ref{45c5,,am_aiter}

@tab

@ref{42bd,,unaryfunc}

@tab

__aiter__

@item

@ref{45c6,,am_anext}

@tab

@ref{42bd,,unaryfunc}

@tab

__anext__

@item

@ref{45c7,,am_send}

@tab

@ref{45c8,,sendfunc}

@tab

@item

@tab

@tab

@item

@ref{4392,,nb_add}

@tab

@ref{42a6,,binaryfunc}

@tab

__add__
__radd__

@item

@ref{45c9,,nb_inplace_add}

@tab

@ref{42a6,,binaryfunc}

@tab

__iadd__

@item

@ref{45ca,,nb_subtract}

@tab

@ref{42a6,,binaryfunc}

@tab

__sub__
__rsub__

@item

@ref{45cb,,nb_inplace_subtract}

@tab

@ref{42a6,,binaryfunc}

@tab

__isub__

@item

@ref{45cc,,nb_multiply}

@tab

@ref{42a6,,binaryfunc}

@tab

__mul__
__rmul__

@item

@ref{45cd,,nb_inplace_multiply}

@tab

@ref{42a6,,binaryfunc}

@tab

__imul__

@item

@ref{45ce,,nb_remainder}

@tab

@ref{42a6,,binaryfunc}

@tab

__mod__
__rmod__

@item

@ref{45cf,,nb_inplace_remainder}

@tab

@ref{42a6,,binaryfunc}

@tab

__imod__

@item

@ref{45d0,,nb_divmod}

@tab

@ref{42a6,,binaryfunc}

@tab

__divmod__
__rdivmod__

@item

@ref{45d1,,nb_power}

@tab

@ref{42bb,,ternaryfunc}

@tab

__pow__
__rpow__

@item

@ref{45d2,,nb_inplace_power}

@tab

@ref{42bb,,ternaryfunc}

@tab

__ipow__

@item

@ref{45d3,,nb_negative}

@tab

@ref{42bd,,unaryfunc}

@tab

__neg__

@item

@ref{45d4,,nb_positive}

@tab

@ref{42bd,,unaryfunc}

@tab

__pos__

@item

@ref{45d5,,nb_absolute}

@tab

@ref{42bd,,unaryfunc}

@tab

__abs__

@item

@ref{45d6,,nb_bool}

@tab

@ref{42af,,inquiry}

@tab

__bool__

@item

@ref{45d7,,nb_invert}

@tab

@ref{42bd,,unaryfunc}

@tab

__invert__

@item

@ref{45d8,,nb_lshift}

@tab

@ref{42a6,,binaryfunc}

@tab

__lshift__
__rlshift__

@item

@ref{45d9,,nb_inplace_lshift}

@tab

@ref{42a6,,binaryfunc}

@tab

__ilshift__

@item

@ref{45da,,nb_rshift}

@tab

@ref{42a6,,binaryfunc}

@tab

__rshift__
__rrshift__

@item

@ref{45db,,nb_inplace_rshift}

@tab

@ref{42a6,,binaryfunc}

@tab

__irshift__

@item

@ref{45dc,,nb_and}

@tab

@ref{42a6,,binaryfunc}

@tab

__and__
__rand__

@item

@ref{45dd,,nb_inplace_and}

@tab

@ref{42a6,,binaryfunc}

@tab

__iand__

@item

@ref{45de,,nb_xor}

@tab

@ref{42a6,,binaryfunc}

@tab

__xor__
__rxor__

@item

@ref{45df,,nb_inplace_xor}

@tab

@ref{42a6,,binaryfunc}

@tab

__ixor__

@item

@ref{45e0,,nb_or}

@tab

@ref{42a6,,binaryfunc}

@tab

__or__
__ror__

@item

@ref{45e1,,nb_inplace_or}

@tab

@ref{42a6,,binaryfunc}

@tab

__ior__

@item

@ref{45e2,,nb_int}

@tab

@ref{42bd,,unaryfunc}

@tab

__int__

@item

@ref{45e3,,nb_reserved}

@tab

void *

@tab

@item

@ref{45e4,,nb_float}

@tab

@ref{42bd,,unaryfunc}

@tab

__float__

@item

@ref{45e5,,nb_floor_divide}

@tab

@ref{42a6,,binaryfunc}

@tab

__floordiv__

@item

@ref{45e6,,nb_inplace_floor_divide}

@tab

@ref{42a6,,binaryfunc}

@tab

__ifloordiv__

@item

@ref{45e7,,nb_true_divide}

@tab

@ref{42a6,,binaryfunc}

@tab

__truediv__

@item

@ref{45e8,,nb_inplace_true_divide}

@tab

@ref{42a6,,binaryfunc}

@tab

__itruediv__

@item

@ref{45e9,,nb_index}

@tab

@ref{42bd,,unaryfunc}

@tab

__index__

@item

@ref{45ea,,nb_matrix_multiply}

@tab

@ref{42a6,,binaryfunc}

@tab

__matmul__
__rmatmul__

@item

@ref{45eb,,nb_inplace_matrix_multiply}

@tab

@ref{42a6,,binaryfunc}

@tab

__imatmul__

@item

@tab

@tab

@item

@ref{45ec,,mp_length}

@tab

@ref{42b1,,lenfunc}

@tab

__len__

@item

@ref{11a7,,mp_subscript}

@tab

@ref{42a6,,binaryfunc}

@tab

__getitem__

@item

@ref{45ed,,mp_ass_subscript}

@tab

@ref{42b3,,objobjargproc}

@tab

__setitem__,
__delitem__

@item

@tab

@tab

@item

@ref{439c,,sq_length}

@tab

@ref{42b1,,lenfunc}

@tab

__len__

@item

@ref{45ee,,sq_concat}

@tab

@ref{42a6,,binaryfunc}

@tab

__add__

@item

@ref{45ef,,sq_repeat}

@tab

@ref{42b9,,ssizeargfunc}

@tab

__mul__

@item

@ref{11a6,,sq_item}

@tab

@ref{42b9,,ssizeargfunc}

@tab

__getitem__

@item

@ref{45f0,,sq_ass_item}

@tab

@ref{42ba,,ssizeobjargproc}

@tab

__setitem__
__delitem__

@item

@ref{45f1,,sq_contains}

@tab

@ref{42b4,,objobjproc}

@tab

__contains__

@item

@ref{45f2,,sq_inplace_concat}

@tab

@ref{42a6,,binaryfunc}

@tab

__iadd__

@item

@ref{45f3,,sq_inplace_repeat}

@tab

@ref{42b9,,ssizeargfunc}

@tab

__imul__

@item

@tab

@tab

@item

@ref{1e7,,bf_getbuffer}

@tab

@ref{45f4,,getbufferproc()}

@tab

@item

@ref{1e8,,bf_releasebuffer}

@tab

@ref{45f5,,releasebufferproc()}

@tab

@end multitable


@node slot typedefs,,sub-slots,Quick Reference
@anchor{c-api/typeobj slot-typedefs}@anchor{45f6}@anchor{c-api/typeobj slot-typedefs-table}@anchor{45b9}
@subsubsection slot typedefs



@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

typedef

@tab

Parameter Types

@tab

Return Type

@item

@ref{42a5,,allocfunc}

@tab


@display
@ref{5c6,,PyTypeObject} *@w{ }
Py_ssize_t@w{ }
@end display



@tab

@ref{375,,PyObject} *

@item

@ref{42a9,,destructor}

@tab

void *

@tab

void

@item

@ref{4477,,freefunc}

@tab

void *

@tab

void

@item

@ref{42bc,,traverseproc}

@tab


@display
void *@w{ }
@ref{42be,,visitproc}@w{ }
void *@w{ }
@end display



@tab

int

@item

@ref{42b2,,newfunc}

@tab


@display
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@end display



@tab

@ref{375,,PyObject} *

@item

@ref{42ae,,initproc}

@tab


@display
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@end display



@tab

int

@item

@ref{42b5,,reprfunc}

@tab

@ref{375,,PyObject} *

@tab

@ref{375,,PyObject} *

@item

@ref{42aa,,getattrfunc}

@tab


@display
@ref{375,,PyObject} *@w{ }
const char *@w{ }
@end display



@tab

@ref{375,,PyObject} *

@item

@ref{42b7,,setattrfunc}

@tab


@display
@ref{375,,PyObject} *@w{ }
const char *@w{ }
@ref{375,,PyObject} *@w{ }
@end display



@tab

int

@item

@ref{42ab,,getattrofunc}

@tab


@display
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@end display



@tab

@ref{375,,PyObject} *

@item

@ref{42b8,,setattrofunc}

@tab


@display
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@end display



@tab

int

@item

@ref{42a7,,descrgetfunc}

@tab


@display
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@end display



@tab

@ref{375,,PyObject} *

@item

@ref{42a8,,descrsetfunc}

@tab


@display
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@end display



@tab

int

@item

@ref{42ad,,hashfunc}

@tab

@ref{375,,PyObject} *

@tab

Py_hash_t

@item

@ref{42b6,,richcmpfunc}

@tab


@display
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
int@w{ }
@end display



@tab

@ref{375,,PyObject} *

@item

@ref{42ac,,getiterfunc}

@tab

@ref{375,,PyObject} *

@tab

@ref{375,,PyObject} *

@item

@ref{42b0,,iternextfunc}

@tab

@ref{375,,PyObject} *

@tab

@ref{375,,PyObject} *

@item

@ref{42b1,,lenfunc}

@tab

@ref{375,,PyObject} *

@tab

Py_ssize_t

@item

@ref{45f4,,getbufferproc}

@tab


@display
@ref{375,,PyObject} *@w{ }
@ref{1db,,Py_buffer} *@w{ }
int@w{ }
@end display



@tab

int

@item

@ref{45f5,,releasebufferproc}

@tab


@display
@ref{375,,PyObject} *@w{ }
@ref{1db,,Py_buffer} *@w{ }
@end display



@tab

void

@item

@ref{42af,,inquiry}

@tab

void *

@tab

int

@item

@ref{42bd,,unaryfunc}

@tab


@display
@ref{375,,PyObject} *@w{ }
@end display



@tab

@ref{375,,PyObject} *

@item

@ref{42a6,,binaryfunc}

@tab


@display
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@end display



@tab

@ref{375,,PyObject} *

@item

@ref{42bb,,ternaryfunc}

@tab


@display
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@end display



@tab

@ref{375,,PyObject} *

@item

@ref{42b9,,ssizeargfunc}

@tab


@display
@ref{375,,PyObject} *@w{ }
Py_ssize_t@w{ }
@end display



@tab

@ref{375,,PyObject} *

@item

@ref{42ba,,ssizeobjargproc}

@tab


@display
@ref{375,,PyObject} *@w{ }
Py_ssize_t@w{ }
@end display



@tab

int

@item

@ref{42b4,,objobjproc}

@tab


@display
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@end display



@tab

int

@item

@ref{42b3,,objobjargproc}

@tab


@display
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@ref{375,,PyObject} *@w{ }
@end display



@tab

int

@end multitable


See @ref{45f7,,Slot Type typedefs} below for more detail.

@node PyTypeObject Definition,PyObject Slots,Quick Reference,Type Objects<3>
@anchor{c-api/typeobj pytypeobject-definition}@anchor{45f8}
@subsubsection PyTypeObject Definition


The structure definition for @ref{5c6,,PyTypeObject} can be found in
@code{Include/object.h}.  For convenience of reference, this repeats the
definition found there:

@c XXX Drop this?

@example
typedef struct _typeobject @{
    PyObject_VAR_HEAD
    const char *tp_name; /* For printing, in format "<module>.<name>" */
    Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */

    /* Methods to implement standard operations */

    destructor tp_dealloc;
    Py_ssize_t tp_vectorcall_offset;
    getattrfunc tp_getattr;
    setattrfunc tp_setattr;
    PyAsyncMethods *tp_as_async; /* formerly known as tp_compare (Python 2)
                                    or tp_reserved (Python 3) */
    reprfunc tp_repr;

    /* Method suites for standard classes */

    PyNumberMethods *tp_as_number;
    PySequenceMethods *tp_as_sequence;
    PyMappingMethods *tp_as_mapping;

    /* More standard operations (here for binary compatibility) */

    hashfunc tp_hash;
    ternaryfunc tp_call;
    reprfunc tp_str;
    getattrofunc tp_getattro;
    setattrofunc tp_setattro;

    /* Functions to access object as input/output buffer */
    PyBufferProcs *tp_as_buffer;

    /* Flags to define presence of optional/expanded features */
    unsigned long tp_flags;

    const char *tp_doc; /* Documentation string */

    /* Assigned meaning in release 2.0 */
    /* call function for all accessible objects */
    traverseproc tp_traverse;

    /* delete references to contained objects */
    inquiry tp_clear;

    /* Assigned meaning in release 2.1 */
    /* rich comparisons */
    richcmpfunc tp_richcompare;

    /* weak reference enabler */
    Py_ssize_t tp_weaklistoffset;

    /* Iterators */
    getiterfunc tp_iter;
    iternextfunc tp_iternext;

    /* Attribute descriptor and subclassing stuff */
    struct PyMethodDef *tp_methods;
    struct PyMemberDef *tp_members;
    struct PyGetSetDef *tp_getset;
    // Strong reference on a heap type, borrowed reference on a static type
    struct _typeobject *tp_base;
    PyObject *tp_dict;
    descrgetfunc tp_descr_get;
    descrsetfunc tp_descr_set;
    Py_ssize_t tp_dictoffset;
    initproc tp_init;
    allocfunc tp_alloc;
    newfunc tp_new;
    freefunc tp_free; /* Low-level free-memory routine */
    inquiry tp_is_gc; /* For PyObject_IS_GC */
    PyObject *tp_bases;
    PyObject *tp_mro; /* method resolution order */
    PyObject *tp_cache;
    PyObject *tp_subclasses;
    PyObject *tp_weaklist;
    destructor tp_del;

    /* Type attribute cache version tag. Added in version 2.6 */
    unsigned int tp_version_tag;

    destructor tp_finalize;
    vectorcallfunc tp_vectorcall;
@} PyTypeObject;
@end example

@node PyObject Slots,PyVarObject Slots,PyTypeObject Definition,Type Objects<3>
@anchor{c-api/typeobj pyobject-slots}@anchor{45f9}
@subsubsection PyObject Slots


The type object structure extends the @ref{4293,,PyVarObject} structure. The
@code{ob_size} field is used for dynamic types (created by @code{type_new()},
usually called from a class statement). Note that @ref{4255,,PyType_Type} (the
metatype) initializes @ref{40a4,,tp_itemsize}, which means that its instances (i.e.
type objects) @emph{must} have the @code{ob_size} field.

@geindex PyObject.ob_refcnt (C member)
@anchor{c-api/typeobj c PyObject ob_refcnt}@anchor{374}
@deffn {C Member} Py_ssize_t  @ref{375,,PyObject}.ob_refcnt

This is the type object’s reference count, initialized to @code{1} by the
@code{PyObject_HEAD_INIT} macro.  Note that for @ref{1f0,,statically allocated type objects}, the type’s instances (objects whose @code{ob_type}
points back to the type) do @emph{not} count as references.  But for
@ref{468,,dynamically allocated type objects}, the instances @emph{do}
count as references.

@strong{Inheritance:}

This field is not inherited by subtypes.
@end deffn

@geindex PyObject.ob_type (C member)
@anchor{c-api/typeobj c PyObject ob_type}@anchor{45fa}
@deffn {C Member} @ref{5c6,,PyTypeObject}  *@ref{375,,PyObject}.ob_type

This is the type’s type, in other words its metatype.  It is initialized by the
argument to the @code{PyObject_HEAD_INIT} macro, and its value should normally be
@code{&PyType_Type}.  However, for dynamically loadable extension modules that must
be usable on Windows (at least), the compiler complains that this is not a valid
initializer.  Therefore, the convention is to pass @code{NULL} to the
@code{PyObject_HEAD_INIT} macro and to initialize this field explicitly at the
start of the module’s initialization function, before doing anything else.  This
is typically done like this:

@example
Foo_Type.ob_type = &PyType_Type;
@end example

This should be done before any instances of the type are created.
@ref{1ec,,PyType_Ready()} checks if @code{ob_type} is @code{NULL}, and if so,
initializes it to the @code{ob_type} field of the base class.
@ref{1ec,,PyType_Ready()} will not change this field if it is non-zero.

@strong{Inheritance:}

This field is inherited by subtypes.
@end deffn

@geindex PyObject._ob_next (C member)
@geindex PyObject._ob_prev (C member)
@anchor{c-api/typeobj c PyObject _ob_next}@anchor{45fb}
@deffn {C Member} @ref{375,,PyObject}  *@ref{375,,PyObject}._ob_next
@anchor{c-api/typeobj c PyObject _ob_prev}@anchor{45fc}
@deffnx {C Member} @ref{375,,PyObject}  *@ref{375,,PyObject}._ob_prev

These fields are only present when the macro @code{Py_TRACE_REFS} is defined
(see the @ref{376,,configure --with-trace-refs option}).

Their initialization to @code{NULL} is taken care of by the
@code{PyObject_HEAD_INIT} macro.  For @ref{1f0,,statically allocated objects}, these fields always remain @code{NULL}.  For @ref{468,,dynamically allocated objects}, these two fields are used to link the
object into a doubly-linked list of @emph{all} live objects on the heap.

This could be used for various debugging purposes; currently the only uses
are the @code{sys.getobjects()} function and to print the objects that are
still alive at the end of a run when the environment variable
@geindex PYTHONDUMPREFS
@geindex environment variable; PYTHONDUMPREFS
@ref{4ad,,PYTHONDUMPREFS} is set.

@strong{Inheritance:}

These fields are not inherited by subtypes.
@end deffn

@node PyVarObject Slots,PyTypeObject Slots,PyObject Slots,Type Objects<3>
@anchor{c-api/typeobj pyvarobject-slots}@anchor{45fd}
@subsubsection PyVarObject Slots


@geindex PyVarObject.ob_size (C member)
@anchor{c-api/typeobj c PyVarObject ob_size}@anchor{45fe}
@deffn {C Member} Py_ssize_t  @ref{4293,,PyVarObject}.ob_size

For @ref{1f0,,statically allocated type objects}, this should be
initialized to zero. For @ref{468,,dynamically allocated type objects}, this field has a special internal meaning.

@strong{Inheritance:}

This field is not inherited by subtypes.
@end deffn

@node PyTypeObject Slots,Static Types,PyVarObject Slots,Type Objects<3>
@anchor{c-api/typeobj pytypeobject-slots}@anchor{45ff}
@subsubsection PyTypeObject Slots


Each slot has a section describing inheritance.  If @ref{1ec,,PyType_Ready()}
may set a value when the field is set to @code{NULL} then there will also be
a “Default” section.  (Note that many fields set on @code{PyBaseObject_Type}
and @ref{4255,,PyType_Type} effectively act as defaults.)

@geindex PyTypeObject.tp_name (C member)
@anchor{c-api/typeobj c PyTypeObject tp_name}@anchor{116b}
@deffn {C Member} const  char  *@ref{5c6,,PyTypeObject}.tp_name

Pointer to a NUL-terminated string containing the name of the type. For types
that are accessible as module globals, the string should be the full module
name, followed by a dot, followed by the type name; for built-in types, it
should be just the type name.  If the module is a submodule of a package, the
full package name is part of the full module name.  For example, a type named
@code{T} defined in module @code{M} in subpackage @code{Q} in package @code{P}
should have the @ref{116b,,tp_name} initializer @code{"P.Q.M.T"}.

For @ref{468,,dynamically allocated type objects},
this should just be the type name, and
the module name explicitly stored in the type dict as the value for key
@code{'__module__'}.

For @ref{1f0,,statically allocated type objects},
the @emph{tp_name} field should contain a dot.
Everything before the last dot is made accessible as the @code{__module__}
attribute, and everything after the last dot is made accessible as the
@ref{e70,,__name__} attribute.

If no dot is present, the entire @ref{116b,,tp_name} field is made accessible as the
@ref{e70,,__name__} attribute, and the @code{__module__} attribute is undefined
(unless explicitly set in the dictionary, as explained above).  This means your
type will be impossible to pickle.  Additionally, it will not be listed in
module documentations created with pydoc.

This field must not be @code{NULL}.  It is the only required field
in @ref{5c6,,PyTypeObject()} (other than potentially
@ref{40a4,,tp_itemsize}).

@strong{Inheritance:}

This field is not inherited by subtypes.
@end deffn

@geindex PyTypeObject.tp_basicsize (C member)
@geindex PyTypeObject.tp_itemsize (C member)
@anchor{c-api/typeobj c PyTypeObject tp_basicsize}@anchor{493}
@deffn {C Member} Py_ssize_t  @ref{5c6,,PyTypeObject}.tp_basicsize
@anchor{c-api/typeobj c PyTypeObject tp_itemsize}@anchor{40a4}
@deffnx {C Member} Py_ssize_t  @ref{5c6,,PyTypeObject}.tp_itemsize

These fields allow calculating the size in bytes of instances of the type.

There are two kinds of types: types with fixed-length instances have a zero
@ref{40a4,,tp_itemsize} field, types with variable-length instances have a non-zero
@ref{40a4,,tp_itemsize} field.  For a type with fixed-length instances, all
instances have the same size, given in @ref{493,,tp_basicsize}.

For a type with variable-length instances, the instances must have an
@code{ob_size} field, and the instance size is @ref{493,,tp_basicsize} plus N
times @ref{40a4,,tp_itemsize}, where N is the “length” of the object.  The value of
N is typically stored in the instance’s @code{ob_size} field.  There are
exceptions:  for example, ints use a negative @code{ob_size} to indicate a
negative number, and N is @code{abs(ob_size)} there.  Also, the presence of an
@code{ob_size} field in the instance layout doesn’t mean that the instance
structure is variable-length (for example, the structure for the list type has
fixed-length instances, yet those instances have a meaningful @code{ob_size}
field).

The basic size includes the fields in the instance declared by the macro
@ref{e7a,,PyObject_HEAD} or @ref{45a9,,PyObject_VAR_HEAD} (whichever is used to
declare the instance struct) and this in turn includes the @code{_ob_prev} and
@code{_ob_next} fields if they are present.  This means that the only correct
way to get an initializer for the @ref{493,,tp_basicsize} is to use the
@code{sizeof} operator on the struct used to declare the instance layout.
The basic size does not include the GC header size.

A note about alignment: if the variable items require a particular alignment,
this should be taken care of by the value of @ref{493,,tp_basicsize}.  Example:
suppose a type implements an array of @code{double}. @ref{40a4,,tp_itemsize} is
@code{sizeof(double)}. It is the programmer’s responsibility that
@ref{493,,tp_basicsize} is a multiple of @code{sizeof(double)} (assuming this is the
alignment requirement for @code{double}).

For any type with variable-length instances, this field must not be @code{NULL}.

@strong{Inheritance:}

These fields are inherited separately by subtypes.  If the base type has a
non-zero @ref{40a4,,tp_itemsize}, it is generally not safe to set
@ref{40a4,,tp_itemsize} to a different non-zero value in a subtype (though this
depends on the implementation of the base type).
@end deffn

@geindex PyTypeObject.tp_dealloc (C member)
@anchor{c-api/typeobj c PyTypeObject tp_dealloc}@anchor{40aa}
@deffn {C Member} @ref{42a9,,destructor}  @ref{5c6,,PyTypeObject}.tp_dealloc

A pointer to the instance destructor function.  This function must be defined
unless the type guarantees that its instances will never be deallocated (as is
the case for the singletons @code{None} and @code{Ellipsis}).  The function signature is:

@example
void tp_dealloc(PyObject *self);
@end example

The destructor function is called by the @ref{373,,Py_DECREF()} and
@ref{1fd,,Py_XDECREF()} macros when the new reference count is zero.  At this point,
the instance is still in existence, but there are no references to it.  The
destructor function should free all references which the instance owns, free all
memory buffers owned by the instance (using the freeing function corresponding
to the allocation function used to allocate the buffer), and call the type’s
@ref{40ab,,tp_free} function.  If the type is not subtypable
(doesn’t have the @ref{40b1,,Py_TPFLAGS_BASETYPE} flag bit set), it is
permissible to call the object deallocator directly instead of via
@ref{40ab,,tp_free}.  The object deallocator should be the one used to allocate the
instance; this is normally @ref{100f,,PyObject_Del()} if the instance was allocated
using @ref{491,,PyObject_New()} or @code{PyObject_VarNew()}, or
@ref{1039,,PyObject_GC_Del()} if the instance was allocated using
@ref{5c3,,PyObject_GC_New()} or @ref{5c4,,PyObject_GC_NewVar()}.

If the type supports garbage collection (has the @ref{1ed,,Py_TPFLAGS_HAVE_GC}
flag bit set), the destructor should call @ref{103b,,PyObject_GC_UnTrack()}
before clearing any member fields.

@example
static void foo_dealloc(foo_object *self) @{
    PyObject_GC_UnTrack(self);
    Py_CLEAR(self->ref);
    Py_TYPE(self)->tp_free((PyObject *)self);
@}
@end example

Finally, if the type is heap allocated (@ref{4394,,Py_TPFLAGS_HEAPTYPE}), the
deallocator should decrement the reference count for its type object after
calling the type deallocator. In order to avoid dangling pointers, the
recommended way to achieve this is:

@example
static void foo_dealloc(foo_object *self) @{
    PyTypeObject *tp = Py_TYPE(self);
    // free references and buffers here
    tp->tp_free(self);
    Py_DECREF(tp);
@}
@end example

@strong{Inheritance:}

This field is inherited by subtypes.
@end deffn

@geindex PyTypeObject.tp_vectorcall_offset (C member)
@anchor{c-api/typeobj c PyTypeObject tp_vectorcall_offset}@anchor{4337}
@deffn {C Member} Py_ssize_t  @ref{5c6,,PyTypeObject}.tp_vectorcall_offset

An optional offset to a per-instance function that implements calling
the object using the @ref{4bd,,vectorcall protocol},
a more efficient alternative
of the simpler @ref{40d2,,tp_call}.

This field is only used if the flag @ref{4336,,Py_TPFLAGS_HAVE_VECTORCALL}
is set. If so, this must be a positive integer containing the offset in the
instance of a @ref{4338,,vectorcallfunc} pointer.

The @emph{vectorcallfunc} pointer may be @code{NULL}, in which case the instance behaves
as if @ref{4336,,Py_TPFLAGS_HAVE_VECTORCALL} was not set: calling the instance
falls back to @ref{40d2,,tp_call}.

Any class that sets @code{Py_TPFLAGS_HAVE_VECTORCALL} must also set
@ref{40d2,,tp_call} and make sure its behaviour is consistent
with the @emph{vectorcallfunc} function.
This can be done by setting @emph{tp_call} to @ref{4335,,PyVectorcall_Call()}.

@cartouche
@quotation Warning 
It is not recommended for @ref{468,,mutable heap types} to implement
the vectorcall protocol.
When a user sets @code{__call__} in Python code, only @emph{tp_call} is updated,
likely making it inconsistent with the vectorcall function.
@end quotation
@end cartouche

@cartouche
@quotation Note 
The semantics of the @code{tp_vectorcall_offset} slot are provisional and
expected to be finalized in Python 3.9.
If you use vectorcall, plan for updating your code for Python 3.9.
@end quotation
@end cartouche


Changed in version 3.8: Before version 3.8, this slot was named @code{tp_print}.
In Python 2.x, it was used for printing to a file.
In Python 3.0 to 3.7, it was unused.


@strong{Inheritance:}

This field is always inherited.
However, the @ref{4336,,Py_TPFLAGS_HAVE_VECTORCALL} flag is not
always inherited. If it’s not, then the subclass won’t use
@ref{4bd,,vectorcall}, except when
@ref{4335,,PyVectorcall_Call()} is explicitly called.
This is in particular the case for types without the
@ref{1ef,,Py_TPFLAGS_IMMUTABLETYPE} flag set (including subclasses defined in
Python).
@end deffn

@geindex PyTypeObject.tp_getattr (C member)
@anchor{c-api/typeobj c PyTypeObject tp_getattr}@anchor{40c8}
@deffn {C Member} @ref{42aa,,getattrfunc}  @ref{5c6,,PyTypeObject}.tp_getattr

An optional pointer to the get-attribute-string function.

This field is deprecated.  When it is defined, it should point to a function
that acts the same as the @ref{40c5,,tp_getattro} function, but taking a C string
instead of a Python string object to give the attribute name.

@strong{Inheritance:}

Group: @code{tp_getattr}, @code{tp_getattro}

This field is inherited by subtypes together with @ref{40c5,,tp_getattro}: a subtype
inherits both @ref{40c8,,tp_getattr} and @ref{40c5,,tp_getattro} from its base type when
the subtype’s @ref{40c8,,tp_getattr} and @ref{40c5,,tp_getattro} are both @code{NULL}.
@end deffn

@geindex PyTypeObject.tp_setattr (C member)
@anchor{c-api/typeobj c PyTypeObject tp_setattr}@anchor{40c9}
@deffn {C Member} @ref{42b7,,setattrfunc}  @ref{5c6,,PyTypeObject}.tp_setattr

An optional pointer to the function for setting and deleting attributes.

This field is deprecated.  When it is defined, it should point to a function
that acts the same as the @ref{40c6,,tp_setattro} function, but taking a C string
instead of a Python string object to give the attribute name.

@strong{Inheritance:}

Group: @code{tp_setattr}, @code{tp_setattro}

This field is inherited by subtypes together with @ref{40c6,,tp_setattro}: a subtype
inherits both @ref{40c9,,tp_setattr} and @ref{40c6,,tp_setattro} from its base type when
the subtype’s @ref{40c9,,tp_setattr} and @ref{40c6,,tp_setattro} are both @code{NULL}.
@end deffn

@geindex PyTypeObject.tp_as_async (C member)
@anchor{c-api/typeobj c PyTypeObject tp_as_async}@anchor{45ba}
@deffn {C Member} @ref{439b,,PyAsyncMethods}  *@ref{5c6,,PyTypeObject}.tp_as_async

Pointer to an additional structure that contains fields relevant only to
objects which implement @ref{7bd,,awaitable} and @ref{8d2,,asynchronous iterator}
protocols at the C-level.  See @ref{4600,,Async Object Structures} for details.


New in version 3.5: Formerly known as @code{tp_compare} and @code{tp_reserved}.


@strong{Inheritance:}

The @ref{45ba,,tp_as_async} field is not inherited,
but the contained fields are inherited individually.
@end deffn

@geindex PyTypeObject.tp_repr (C member)
@anchor{c-api/typeobj c PyTypeObject tp_repr}@anchor{40c1}
@deffn {C Member} @ref{42b5,,reprfunc}  @ref{5c6,,PyTypeObject}.tp_repr

@geindex built-in function; repr

An optional pointer to a function that implements the built-in function
@ref{27d,,repr()}.

The signature is the same as for @ref{bc0,,PyObject_Repr()}:

@example
PyObject *tp_repr(PyObject *self);
@end example

The function must return a string or a Unicode object.  Ideally,
this function should return a string that, when passed to
@ref{34a,,eval()}, given a suitable environment, returns an object with the
same value.  If this is not feasible, it should return a string starting with
@code{'<'} and ending with @code{'>'} from which both the type and the value of the
object can be deduced.

@strong{Inheritance:}

This field is inherited by subtypes.

@strong{Default:}

When this field is not set, a string of the form @code{<%s object at %p>} is
returned, where @code{%s} is replaced by the type name, and @code{%p} by the object’s
memory address.
@end deffn

@geindex PyTypeObject.tp_as_number (C member)
@anchor{c-api/typeobj c PyTypeObject tp_as_number}@anchor{497}
@deffn {C Member} @ref{f7b,,PyNumberMethods}  *@ref{5c6,,PyTypeObject}.tp_as_number

Pointer to an additional structure that contains fields relevant only to
objects which implement the number protocol.  These fields are documented in
@ref{4601,,Number Object Structures}.

@strong{Inheritance:}

The @ref{497,,tp_as_number} field is not inherited, but the contained fields are
inherited individually.
@end deffn

@geindex PyTypeObject.tp_as_sequence (C member)
@anchor{c-api/typeobj c PyTypeObject tp_as_sequence}@anchor{45bc}
@deffn {C Member} @ref{40cf,,PySequenceMethods}  *@ref{5c6,,PyTypeObject}.tp_as_sequence

Pointer to an additional structure that contains fields relevant only to
objects which implement the sequence protocol.  These fields are documented
in @ref{4602,,Sequence Object Structures}.

@strong{Inheritance:}

The @ref{45bc,,tp_as_sequence} field is not inherited, but the contained fields
are inherited individually.
@end deffn

@geindex PyTypeObject.tp_as_mapping (C member)
@anchor{c-api/typeobj c PyTypeObject tp_as_mapping}@anchor{45bd}
@deffn {C Member} @ref{40d0,,PyMappingMethods}  *@ref{5c6,,PyTypeObject}.tp_as_mapping

Pointer to an additional structure that contains fields relevant only to
objects which implement the mapping protocol.  These fields are documented in
@ref{4603,,Mapping Object Structures}.

@strong{Inheritance:}

The @ref{45bd,,tp_as_mapping} field is not inherited, but the contained fields
are inherited individually.
@end deffn

@geindex PyTypeObject.tp_hash (C member)
@anchor{c-api/typeobj c PyTypeObject tp_hash}@anchor{40d1}
@deffn {C Member} @ref{42ad,,hashfunc}  @ref{5c6,,PyTypeObject}.tp_hash

@geindex built-in function; hash

An optional pointer to a function that implements the built-in function
@ref{c14,,hash()}.

The signature is the same as for @ref{209,,PyObject_Hash()}:

@example
Py_hash_t tp_hash(PyObject *);
@end example

The value @code{-1} should not be returned as a
normal return value; when an error occurs during the computation of the hash
value, the function should set an exception and return @code{-1}.

When this field is not set (@emph{and} @code{tp_richcompare} is not set),
an attempt to take the hash of the object raises @ref{162,,TypeError}.
This is the same as setting it to @ref{f2d,,PyObject_HashNotImplemented()}.

This field can be set explicitly to @ref{f2d,,PyObject_HashNotImplemented()} to
block inheritance of the hash method from a parent type. This is interpreted
as the equivalent of @code{__hash__ = None} at the Python level, causing
@code{isinstance(o, collections.Hashable)} to correctly return @code{False}. Note
that the converse is also true - setting @code{__hash__ = None} on a class at
the Python level will result in the @code{tp_hash} slot being set to
@ref{f2d,,PyObject_HashNotImplemented()}.

@strong{Inheritance:}

Group: @code{tp_hash}, @code{tp_richcompare}

This field is inherited by subtypes together with
@ref{40cb,,tp_richcompare}: a subtype inherits both of
@ref{40cb,,tp_richcompare} and @ref{40d1,,tp_hash}, when the subtype’s
@ref{40cb,,tp_richcompare} and @ref{40d1,,tp_hash} are both @code{NULL}.
@end deffn

@geindex PyTypeObject.tp_call (C member)
@anchor{c-api/typeobj c PyTypeObject tp_call}@anchor{40d2}
@deffn {C Member} @ref{42bb,,ternaryfunc}  @ref{5c6,,PyTypeObject}.tp_call

An optional pointer to a function that implements calling the object.  This
should be @code{NULL} if the object is not callable.  The signature is the same as
for @ref{46a,,PyObject_Call()}:

@example
PyObject *tp_call(PyObject *self, PyObject *args, PyObject *kwargs);
@end example

@strong{Inheritance:}

This field is inherited by subtypes.
@end deffn

@geindex PyTypeObject.tp_str (C member)
@anchor{c-api/typeobj c PyTypeObject tp_str}@anchor{40c2}
@deffn {C Member} @ref{42b5,,reprfunc}  @ref{5c6,,PyTypeObject}.tp_str

An optional pointer to a function that implements the built-in operation
@ref{2ef,,str()}.  (Note that @ref{2ef,,str} is a type now, and @ref{2ef,,str()} calls the
constructor for that type.  This constructor calls @ref{bc1,,PyObject_Str()} to do
the actual work, and @ref{bc1,,PyObject_Str()} will call this handler.)

The signature is the same as for @ref{bc1,,PyObject_Str()}:

@example
PyObject *tp_str(PyObject *self);
@end example

The function must return a string or a Unicode object.  It should be a “friendly” string
representation of the object, as this is the representation that will be used,
among other things, by the @ref{af6,,print()} function.

@strong{Inheritance:}

This field is inherited by subtypes.

@strong{Default:}

When this field is not set, @ref{bc0,,PyObject_Repr()} is called to return a string
representation.
@end deffn

@geindex PyTypeObject.tp_getattro (C member)
@anchor{c-api/typeobj c PyTypeObject tp_getattro}@anchor{40c5}
@deffn {C Member} @ref{42ab,,getattrofunc}  @ref{5c6,,PyTypeObject}.tp_getattro

An optional pointer to the get-attribute function.

The signature is the same as for @ref{4208,,PyObject_GetAttr()}:

@example
PyObject *tp_getattro(PyObject *self, PyObject *attr);
@end example

It is usually convenient to set this field to @ref{4204,,PyObject_GenericGetAttr()},
which implements the normal way of looking for object attributes.

@strong{Inheritance:}

Group: @code{tp_getattr}, @code{tp_getattro}

This field is inherited by subtypes together with @ref{40c8,,tp_getattr}: a subtype
inherits both @ref{40c8,,tp_getattr} and @ref{40c5,,tp_getattro} from its base type when
the subtype’s @ref{40c8,,tp_getattr} and @ref{40c5,,tp_getattro} are both @code{NULL}.

@strong{Default:}

@code{PyBaseObject_Type} uses @ref{4204,,PyObject_GenericGetAttr()}.
@end deffn

@geindex PyTypeObject.tp_setattro (C member)
@anchor{c-api/typeobj c PyTypeObject tp_setattro}@anchor{40c6}
@deffn {C Member} @ref{42b8,,setattrofunc}  @ref{5c6,,PyTypeObject}.tp_setattro

An optional pointer to the function for setting and deleting attributes.

The signature is the same as for @ref{420f,,PyObject_SetAttr()}:

@example
int tp_setattro(PyObject *self, PyObject *attr, PyObject *value);
@end example

In addition, setting @emph{value} to @code{NULL} to delete an attribute must be
supported.  It is usually convenient to set this field to
@ref{4205,,PyObject_GenericSetAttr()}, which implements the normal
way of setting object attributes.

@strong{Inheritance:}

Group: @code{tp_setattr}, @code{tp_setattro}

This field is inherited by subtypes together with @ref{40c9,,tp_setattr}: a subtype
inherits both @ref{40c9,,tp_setattr} and @ref{40c6,,tp_setattro} from its base type when
the subtype’s @ref{40c9,,tp_setattr} and @ref{40c6,,tp_setattro} are both @code{NULL}.

@strong{Default:}

@code{PyBaseObject_Type} uses @ref{4205,,PyObject_GenericSetAttr()}.
@end deffn

@geindex PyTypeObject.tp_as_buffer (C member)
@anchor{c-api/typeobj c PyTypeObject tp_as_buffer}@anchor{495}
@deffn {C Member} @ref{43a5,,PyBufferProcs}  *@ref{5c6,,PyTypeObject}.tp_as_buffer

Pointer to an additional structure that contains fields relevant only to objects
which implement the buffer interface.  These fields are documented in
@ref{435b,,Buffer Object Structures}.

@strong{Inheritance:}

The @ref{495,,tp_as_buffer} field is not inherited,
but the contained fields are inherited individually.
@end deffn

@geindex PyTypeObject.tp_flags (C member)
@anchor{c-api/typeobj c PyTypeObject tp_flags}@anchor{1133}
@deffn {C Member} unsigned  long  @ref{5c6,,PyTypeObject}.tp_flags

This field is a bit mask of various flags.  Some flags indicate variant
semantics for certain situations; others are used to indicate that certain
fields in the type object (or in the extension structures referenced via
@ref{497,,tp_as_number}, @ref{45bc,,tp_as_sequence}, @ref{45bd,,tp_as_mapping}, and
@ref{495,,tp_as_buffer}) that were historically not always present are valid; if
such a flag bit is clear, the type fields it guards must not be accessed and
must be considered to have a zero or @code{NULL} value instead.

@strong{Inheritance:}

Inheritance of this field is complicated.  Most flag bits are inherited
individually, i.e. if the base type has a flag bit set, the subtype inherits
this flag bit.  The flag bits that pertain to extension structures are strictly
inherited if the extension structure is inherited, i.e. the base type’s value of
the flag bit is copied into the subtype together with a pointer to the extension
structure.  The @ref{1ed,,Py_TPFLAGS_HAVE_GC} flag bit is inherited together with
the @ref{1ee,,tp_traverse} and @ref{40bf,,tp_clear} fields, i.e. if the
@ref{1ed,,Py_TPFLAGS_HAVE_GC} flag bit is clear in the subtype and the
@ref{1ee,,tp_traverse} and @ref{40bf,,tp_clear} fields in the subtype exist and have
@code{NULL} values.

@c XXX are most flag bits *really* inherited individually?

@strong{Default:}

@code{PyBaseObject_Type} uses
@code{Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE}.

@strong{Bit Masks:}

The following bit masks are currently defined; these can be ORed together using
the @code{|} operator to form the value of the @ref{1133,,tp_flags} field.  The macro
@ref{11db,,PyType_HasFeature()} takes a type and a flags value, @emph{tp} and @emph{f}, and
checks whether @code{tp->tp_flags & f} is non-zero.

@geindex Py_TPFLAGS_HEAPTYPE (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_HEAPTYPE}@anchor{4394}
@deffn {Data} Py_TPFLAGS_HEAPTYPE

This bit is set when the type object itself is allocated on the heap, for
example, types created dynamically using @ref{469,,PyType_FromSpec()}.  In this
case, the @code{ob_type} field of its instances is considered a reference to
the type, and the type object is INCREF’ed when a new instance is created, and
DECREF’ed when an instance is destroyed (this does not apply to instances of
subtypes; only the type referenced by the instance’s ob_type gets INCREF’ed or
DECREF’ed).

@strong{Inheritance:}

???
@end deffn

@geindex Py_TPFLAGS_BASETYPE (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_BASETYPE}@anchor{40b1}
@deffn {Data} Py_TPFLAGS_BASETYPE

This bit is set when the type can be used as the base type of another type.  If
this bit is clear, the type cannot be subtyped (similar to a “final” class in
Java).

@strong{Inheritance:}

???
@end deffn

@geindex Py_TPFLAGS_READY (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_READY}@anchor{4604}
@deffn {Data} Py_TPFLAGS_READY

This bit is set when the type object has been fully initialized by
@ref{1ec,,PyType_Ready()}.

@strong{Inheritance:}

???
@end deffn

@geindex Py_TPFLAGS_READYING (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_READYING}@anchor{4605}
@deffn {Data} Py_TPFLAGS_READYING

This bit is set while @ref{1ec,,PyType_Ready()} is in the process of initializing
the type object.

@strong{Inheritance:}

???
@end deffn

@geindex Py_TPFLAGS_HAVE_GC (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_HAVE_GC}@anchor{1ed}
@deffn {Data} Py_TPFLAGS_HAVE_GC

This bit is set when the object supports garbage collection.  If this bit
is set, instances must be created using @ref{5c3,,PyObject_GC_New()} and
destroyed using @ref{1039,,PyObject_GC_Del()}.  More information in section
@ref{4606,,Supporting Cyclic Garbage Collection}.  This bit also implies that the
GC-related fields @ref{1ee,,tp_traverse} and @ref{40bf,,tp_clear} are present in
the type object.

@strong{Inheritance:}

Group: @ref{1ed,,Py_TPFLAGS_HAVE_GC}, @code{tp_traverse}, @code{tp_clear}

The @ref{1ed,,Py_TPFLAGS_HAVE_GC} flag bit is inherited
together with the @code{tp_traverse} and @code{tp_clear}
fields, i.e.  if the @ref{1ed,,Py_TPFLAGS_HAVE_GC} flag bit is
clear in the subtype and the @code{tp_traverse} and
@code{tp_clear} fields in the subtype exist and have @code{NULL}
values.
@end deffn

@geindex Py_TPFLAGS_DEFAULT (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_DEFAULT}@anchor{40a5}
@deffn {Data} Py_TPFLAGS_DEFAULT

This is a bitmask of all the bits that pertain to the existence of certain
fields in the type object and its extension structures. Currently, it includes
the following bits: @code{Py_TPFLAGS_HAVE_STACKLESS_EXTENSION}.

@strong{Inheritance:}

???
@end deffn

@geindex Py_TPFLAGS_METHOD_DESCRIPTOR (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_METHOD_DESCRIPTOR}@anchor{4343}
@deffn {Data} Py_TPFLAGS_METHOD_DESCRIPTOR

This bit indicates that objects behave like unbound methods.

If this flag is set for @code{type(meth)}, then:


@itemize -

@item 
@code{meth.__get__(obj, cls)(*args, **kwds)} (with @code{obj} not None)
must be equivalent to @code{meth(obj, *args, **kwds)}.

@item 
@code{meth.__get__(None, cls)(*args, **kwds)}
must be equivalent to @code{meth(*args, **kwds)}.
@end itemize

This flag enables an optimization for typical method calls like
@code{obj.meth()}: it avoids creating a temporary “bound method” object for
@code{obj.meth}.


New in version 3.8.


@strong{Inheritance:}

This flag is never inherited by types without the
@ref{1ef,,Py_TPFLAGS_IMMUTABLETYPE} flag set.  For extension types, it is
inherited whenever @ref{45be,,tp_descr_get} is inherited.
@end deffn

@c XXX Document more flags here?

@geindex Py_TPFLAGS_LONG_SUBCLASS (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_LONG_SUBCLASS}@anchor{4607}
@deffn {Data} Py_TPFLAGS_LONG_SUBCLASS
@end deffn

@geindex Py_TPFLAGS_LIST_SUBCLASS (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_LIST_SUBCLASS}@anchor{4608}
@deffn {Data} Py_TPFLAGS_LIST_SUBCLASS
@end deffn

@geindex Py_TPFLAGS_TUPLE_SUBCLASS (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_TUPLE_SUBCLASS}@anchor{4609}
@deffn {Data} Py_TPFLAGS_TUPLE_SUBCLASS
@end deffn

@geindex Py_TPFLAGS_BYTES_SUBCLASS (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_BYTES_SUBCLASS}@anchor{460a}
@deffn {Data} Py_TPFLAGS_BYTES_SUBCLASS
@end deffn

@geindex Py_TPFLAGS_UNICODE_SUBCLASS (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_UNICODE_SUBCLASS}@anchor{460b}
@deffn {Data} Py_TPFLAGS_UNICODE_SUBCLASS
@end deffn

@geindex Py_TPFLAGS_DICT_SUBCLASS (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_DICT_SUBCLASS}@anchor{460c}
@deffn {Data} Py_TPFLAGS_DICT_SUBCLASS
@end deffn

@geindex Py_TPFLAGS_BASE_EXC_SUBCLASS (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_BASE_EXC_SUBCLASS}@anchor{460d}
@deffn {Data} Py_TPFLAGS_BASE_EXC_SUBCLASS
@end deffn

@geindex Py_TPFLAGS_TYPE_SUBCLASS (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_TYPE_SUBCLASS}@anchor{460e}
@deffn {Data} Py_TPFLAGS_TYPE_SUBCLASS

These flags are used by functions such as
@ref{43af,,PyLong_Check()} to quickly determine if a type is a subclass
of a built-in type; such specific checks are faster than a generic
check, like @ref{a17,,PyObject_IsInstance()}. Custom types that inherit
from built-ins should have their @ref{1133,,tp_flags}
set appropriately, or the code that interacts with such types
will behave differently depending on what kind of check is used.
@end deffn

@geindex Py_TPFLAGS_HAVE_FINALIZE (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_HAVE_FINALIZE}@anchor{5c8}
@deffn {Data} Py_TPFLAGS_HAVE_FINALIZE

This bit is set when the @ref{5c7,,tp_finalize} slot is present in the
type structure.


New in version 3.4.



Deprecated since version 3.8: This flag isn’t necessary anymore, as the interpreter assumes the
@ref{5c7,,tp_finalize} slot is always present in the
type structure.

@end deffn

@geindex Py_TPFLAGS_HAVE_VECTORCALL (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_HAVE_VECTORCALL}@anchor{4336}
@deffn {Data} Py_TPFLAGS_HAVE_VECTORCALL

This bit is set when the class implements
the @ref{4bd,,vectorcall protocol}.
See @ref{4337,,tp_vectorcall_offset} for details.

@strong{Inheritance:}

This bit is inherited for types with the
@ref{1ef,,Py_TPFLAGS_IMMUTABLETYPE} flag set, if
@ref{40d2,,tp_call} is also inherited.


New in version 3.9.

@end deffn

@geindex Py_TPFLAGS_IMMUTABLETYPE (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_IMMUTABLETYPE}@anchor{1ef}
@deffn {Data} Py_TPFLAGS_IMMUTABLETYPE

This bit is set for type objects that are immutable: type attributes cannot be set nor deleted.

@ref{1ec,,PyType_Ready()} automatically applies this flag to
@ref{1f0,,static types}.

@strong{Inheritance:}

This flag is not inherited.


New in version 3.10.

@end deffn

@geindex Py_TPFLAGS_DISALLOW_INSTANTIATION (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_DISALLOW_INSTANTIATION}@anchor{460f}
@deffn {Data} Py_TPFLAGS_DISALLOW_INSTANTIATION

Disallow creating instances of the type: set
@ref{40a7,,tp_new} to NULL and don’t create the @code{__new__}
key in the type dictionary.

The flag must be set before creating the type, not after. For example, it
must be set before @ref{1ec,,PyType_Ready()} is called on the type.

The flag is set automatically on @ref{1f0,,static types} if
@ref{40b7,,tp_base} is NULL or @code{&PyBaseObject_Type} and
@ref{40a7,,tp_new} is NULL.

@strong{Inheritance:}

This flag is not inherited.


New in version 3.10.

@end deffn

@geindex Py_TPFLAGS_MAPPING (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_MAPPING}@anchor{110a}
@deffn {Data} Py_TPFLAGS_MAPPING

This bit indicates that instances of the class may match mapping patterns
when used as the subject of a @ref{290,,match} block. It is automatically
set when registering or subclassing @ref{3ab,,collections.abc.Mapping}, and
unset when registering @ref{110c,,collections.abc.Sequence}.

@cartouche
@quotation Note 
@ref{110a,,Py_TPFLAGS_MAPPING} and @ref{110b,,Py_TPFLAGS_SEQUENCE} are
mutually exclusive; it is an error to enable both flags simultaneously.
@end quotation
@end cartouche

@strong{Inheritance:}

This flag is inherited by types that do not already set
@ref{110b,,Py_TPFLAGS_SEQUENCE}.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 634
PEP 634@footnote{https://www.python.org/dev/peps/pep-0634} – Structural Pattern Matching: Specification



New in version 3.10.

@end deffn

@geindex Py_TPFLAGS_SEQUENCE (built-in variable)
@anchor{c-api/typeobj Py_TPFLAGS_SEQUENCE}@anchor{110b}
@deffn {Data} Py_TPFLAGS_SEQUENCE

This bit indicates that instances of the class may match sequence patterns
when used as the subject of a @ref{290,,match} block. It is automatically
set when registering or subclassing @ref{110c,,collections.abc.Sequence}, and
unset when registering @ref{3ab,,collections.abc.Mapping}.

@cartouche
@quotation Note 
@ref{110a,,Py_TPFLAGS_MAPPING} and @ref{110b,,Py_TPFLAGS_SEQUENCE} are
mutually exclusive; it is an error to enable both flags simultaneously.
@end quotation
@end cartouche

@strong{Inheritance:}

This flag is inherited by types that do not already set
@ref{110a,,Py_TPFLAGS_MAPPING}.


@subsubheading See also


@geindex Python Enhancement Proposals; PEP 634
PEP 634@footnote{https://www.python.org/dev/peps/pep-0634} – Structural Pattern Matching: Specification



New in version 3.10.

@end deffn
@end deffn

@geindex PyTypeObject.tp_doc (C member)
@anchor{c-api/typeobj c PyTypeObject tp_doc}@anchor{40a6}
@deffn {C Member} const  char  *@ref{5c6,,PyTypeObject}.tp_doc

An optional pointer to a NUL-terminated C string giving the docstring for this
type object.  This is exposed as the @code{__doc__} attribute on the type and
instances of the type.

@strong{Inheritance:}

This field is @emph{not} inherited by subtypes.
@end deffn

@geindex PyTypeObject.tp_traverse (C member)
@anchor{c-api/typeobj c PyTypeObject tp_traverse}@anchor{1ee}
@deffn {C Member} @ref{42bc,,traverseproc}  @ref{5c6,,PyTypeObject}.tp_traverse

An optional pointer to a traversal function for the garbage collector.  This is
only used if the @ref{1ed,,Py_TPFLAGS_HAVE_GC} flag bit is set.  The signature is:

@example
int tp_traverse(PyObject *self, visitproc visit, void *arg);
@end example

More information about Python’s garbage collection scheme can be found
in section @ref{4606,,Supporting Cyclic Garbage Collection}.

The @ref{1ee,,tp_traverse} pointer is used by the garbage collector to detect
reference cycles. A typical implementation of a @ref{1ee,,tp_traverse} function
simply calls @ref{40b5,,Py_VISIT()} on each of the instance’s members that are Python
objects that the instance owns. For example, this is function @code{local_traverse()} from the
@ref{2,,_thread} extension module:

@example
static int
local_traverse(localobject *self, visitproc visit, void *arg)
@{
    Py_VISIT(self->args);
    Py_VISIT(self->kw);
    Py_VISIT(self->dict);
    return 0;
@}
@end example

Note that @ref{40b5,,Py_VISIT()} is called only on those members that can participate
in reference cycles.  Although there is also a @code{self->key} member, it can only
be @code{NULL} or a Python string and therefore cannot be part of a reference cycle.

On the other hand, even if you know a member can never be part of a cycle, as a
debugging aid you may want to visit it anyway just so the @ref{80,,gc} module’s
@ref{286,,get_referents()} function will include it.

@cartouche
@quotation Warning 
When implementing @ref{1ee,,tp_traverse}, only the
members that the instance @emph{owns} (by having @ref{1fc,,strong references} to them) must be
visited. For instance, if an object supports weak references via the
@ref{43a1,,tp_weaklist} slot, the pointer supporting
the linked list (what @emph{tp_weaklist} points to) must @strong{not} be
visited as the instance does not directly own the weak references to itself
(the weakreference list is there to support the weak reference machinery,
but the instance has no strong reference to the elements inside it, as they
are allowed to be removed even if the instance is still alive).
@end quotation
@end cartouche

Note that @ref{40b5,,Py_VISIT()} requires the @emph{visit} and @emph{arg} parameters to
@code{local_traverse()} to have these specific names; don’t name them just
anything.

Instances of @ref{468,,heap-allocated types} hold a reference to
their type. Their traversal function must therefore either visit
@ref{1f1,,Py_TYPE(self)}, or delegate this responsibility by
calling @code{tp_traverse} of another heap-allocated type (such as a
heap-allocated superclass).
If they do not, the type object may not be garbage-collected.


Changed in version 3.9: Heap-allocated types are expected to visit @code{Py_TYPE(self)} in
@code{tp_traverse}.  In earlier versions of Python, due to
bug 40217@footnote{https://bugs.python.org/issue40217}, doing this
may lead to crashes in subclasses.


@strong{Inheritance:}

Group: @ref{1ed,,Py_TPFLAGS_HAVE_GC}, @code{tp_traverse}, @code{tp_clear}

This field is inherited by subtypes together with @ref{40bf,,tp_clear} and the
@ref{1ed,,Py_TPFLAGS_HAVE_GC} flag bit: the flag bit, @ref{1ee,,tp_traverse}, and
@ref{40bf,,tp_clear} are all inherited from the base type if they are all zero in
the subtype.
@end deffn

@geindex PyTypeObject.tp_clear (C member)
@anchor{c-api/typeobj c PyTypeObject tp_clear}@anchor{40bf}
@deffn {C Member} @ref{42af,,inquiry}  @ref{5c6,,PyTypeObject}.tp_clear

An optional pointer to a clear function for the garbage collector. This is only
used if the @ref{1ed,,Py_TPFLAGS_HAVE_GC} flag bit is set.  The signature is:

@example
int tp_clear(PyObject *);
@end example

The @ref{40bf,,tp_clear} member function is used to break reference cycles in cyclic
garbage detected by the garbage collector.  Taken together, all @ref{40bf,,tp_clear}
functions in the system must combine to break all reference cycles.  This is
subtle, and if in any doubt supply a @ref{40bf,,tp_clear} function.  For example,
the tuple type does not implement a @ref{40bf,,tp_clear} function, because it’s
possible to prove that no reference cycle can be composed entirely of tuples.
Therefore the @ref{40bf,,tp_clear} functions of other types must be sufficient to
break any cycle containing a tuple.  This isn’t immediately obvious, and there’s
rarely a good reason to avoid implementing @ref{40bf,,tp_clear}.

Implementations of @ref{40bf,,tp_clear} should drop the instance’s references to
those of its members that may be Python objects, and set its pointers to those
members to @code{NULL}, as in the following example:

@example
static int
local_clear(localobject *self)
@{
    Py_CLEAR(self->key);
    Py_CLEAR(self->args);
    Py_CLEAR(self->kw);
    Py_CLEAR(self->dict);
    return 0;
@}
@end example

The @ref{fd2,,Py_CLEAR()} macro should be used, because clearing references is
delicate:  the reference to the contained object must not be decremented until
after the pointer to the contained object is set to @code{NULL}.  This is because
decrementing the reference count may cause the contained object to become trash,
triggering a chain of reclamation activity that may include invoking arbitrary
Python code (due to finalizers, or weakref callbacks, associated with the
contained object). If it’s possible for such code to reference @emph{self} again,
it’s important that the pointer to the contained object be @code{NULL} at that time,
so that @emph{self} knows the contained object can no longer be used.  The
@ref{fd2,,Py_CLEAR()} macro performs the operations in a safe order.

Note that @ref{40bf,,tp_clear} is not @emph{always} called
before an instance is deallocated. For example, when reference counting
is enough to determine that an object is no longer used, the cyclic garbage
collector is not involved and @ref{40aa,,tp_dealloc} is
called directly.

Because the goal of @ref{40bf,,tp_clear} functions is to break reference cycles,
it’s not necessary to clear contained objects like Python strings or Python
integers, which can’t participate in reference cycles. On the other hand, it may
be convenient to clear all contained Python objects, and write the type’s
@ref{40aa,,tp_dealloc} function to invoke @ref{40bf,,tp_clear}.

More information about Python’s garbage collection scheme can be found in
section @ref{4606,,Supporting Cyclic Garbage Collection}.

@strong{Inheritance:}

Group: @ref{1ed,,Py_TPFLAGS_HAVE_GC}, @code{tp_traverse}, @code{tp_clear}

This field is inherited by subtypes together with @ref{1ee,,tp_traverse} and the
@ref{1ed,,Py_TPFLAGS_HAVE_GC} flag bit: the flag bit, @ref{1ee,,tp_traverse}, and
@ref{40bf,,tp_clear} are all inherited from the base type if they are all zero in
the subtype.
@end deffn

@geindex PyTypeObject.tp_richcompare (C member)
@anchor{c-api/typeobj c PyTypeObject tp_richcompare}@anchor{40cb}
@deffn {C Member} @ref{42b6,,richcmpfunc}  @ref{5c6,,PyTypeObject}.tp_richcompare

An optional pointer to the rich comparison function, whose signature is:

@example
PyObject *tp_richcompare(PyObject *self, PyObject *other, int op);
@end example

The first parameter is guaranteed to be an instance of the type
that is defined by @ref{5c6,,PyTypeObject}.

The function should return the result of the comparison (usually @code{Py_True}
or @code{Py_False}).  If the comparison is undefined, it must return
@code{Py_NotImplemented}, if another error occurred it must return @code{NULL} and
set an exception condition.

The following constants are defined to be used as the third argument for
@ref{40cb,,tp_richcompare} and for @ref{40cc,,PyObject_RichCompare()}:


@multitable {xxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxx} 
@headitem

Constant

@tab

Comparison

@item

@code{Py_LT}

@tab

@code{<}

@item

@code{Py_LE}

@tab

@code{<=}

@item

@code{Py_EQ}

@tab

@code{==}

@item

@code{Py_NE}

@tab

@code{!=}

@item

@code{Py_GT}

@tab

@code{>}

@item

@code{Py_GE}

@tab

@code{>=}

@end multitable


The following macro is defined to ease writing rich comparison functions:

@geindex PyTypeObject.tp_richcompare.Py_RETURN_RICHCOMPARE (C macro)
@anchor{c-api/typeobj c PyTypeObject tp_richcompare Py_RETURN_RICHCOMPARE}@anchor{4610}
@deffn {C Macro} Py_RETURN_RICHCOMPARE (VAL_A, VAL_B, op)

Return @code{Py_True} or @code{Py_False} from the function, depending on the
result of a comparison.
VAL_A and VAL_B must be orderable by C comparison operators (for example,
they may be C ints or floats). The third argument specifies the requested
operation, as for @ref{40cc,,PyObject_RichCompare()}.

The return value’s reference count is properly incremented.

On error, sets an exception and returns @code{NULL} from the function.


New in version 3.7.

@end deffn

@strong{Inheritance:}

Group: @code{tp_hash}, @code{tp_richcompare}

This field is inherited by subtypes together with @ref{40d1,,tp_hash}:
a subtype inherits @ref{40cb,,tp_richcompare} and @ref{40d1,,tp_hash} when
the subtype’s @ref{40cb,,tp_richcompare} and @ref{40d1,,tp_hash} are both
@code{NULL}.

@strong{Default:}

@code{PyBaseObject_Type} provides a @code{tp_richcompare}
implementation, which may be inherited.  However, if only
@code{tp_hash} is defined, not even the inherited function is used
and instances of the type will not be able to participate in any
comparisons.
@end deffn

@geindex PyTypeObject.tp_weaklistoffset (C member)
@anchor{c-api/typeobj c PyTypeObject tp_weaklistoffset}@anchor{494}
@deffn {C Member} Py_ssize_t  @ref{5c6,,PyTypeObject}.tp_weaklistoffset

If the instances of this type are weakly referenceable, this field is greater
than zero and contains the offset in the instance structure of the weak
reference list head (ignoring the GC header, if present); this offset is used by
@code{PyObject_ClearWeakRefs()} and the PyWeakref_* functions.  The
instance structure needs to include a field of type @ref{375,,PyObject}* which is
initialized to @code{NULL}.

Do not confuse this field with @ref{43a1,,tp_weaklist}; that is the list head for
weak references to the type object itself.

@strong{Inheritance:}

This field is inherited by subtypes, but see the rules listed below. A subtype
may override this offset; this means that the subtype uses a different weak
reference list head than the base type.  Since the list head is always found via
@ref{494,,tp_weaklistoffset}, this should not be a problem.

When a type defined by a class statement has no @ref{1025,,__slots__} declaration,
and none of its base types are weakly referenceable, the type is made weakly
referenceable by adding a weak reference list head slot to the instance layout
and setting the @ref{494,,tp_weaklistoffset} of that slot’s offset.

When a type’s @code{__slots__} declaration contains a slot named
@code{__weakref__}, that slot becomes the weak reference list head for
instances of the type, and the slot’s offset is stored in the type’s
@ref{494,,tp_weaklistoffset}.

When a type’s @code{__slots__} declaration does not contain a slot named
@code{__weakref__}, the type inherits its @ref{494,,tp_weaklistoffset} from its
base type.
@end deffn

@geindex PyTypeObject.tp_iter (C member)
@anchor{c-api/typeobj c PyTypeObject tp_iter}@anchor{1028}
@deffn {C Member} @ref{42ac,,getiterfunc}  @ref{5c6,,PyTypeObject}.tp_iter

An optional pointer to a function that returns an @ref{11c3,,iterator} for the
object.  Its presence normally signals that the instances of this type are
@ref{dcc,,iterable} (although sequences may be iterable without this function).

This function has the same signature as @ref{4209,,PyObject_GetIter()}:

@example
PyObject *tp_iter(PyObject *self);
@end example

@strong{Inheritance:}

This field is inherited by subtypes.
@end deffn

@geindex PyTypeObject.tp_iternext (C member)
@anchor{c-api/typeobj c PyTypeObject tp_iternext}@anchor{1029}
@deffn {C Member} @ref{42b0,,iternextfunc}  @ref{5c6,,PyTypeObject}.tp_iternext

An optional pointer to a function that returns the next item in an
@ref{11c3,,iterator}. The signature is:

@example
PyObject *tp_iternext(PyObject *self);
@end example

When the iterator is exhausted, it must return @code{NULL}; a @ref{736,,StopIteration}
exception may or may not be set.  When another error occurs, it must return
@code{NULL} too.  Its presence signals that the instances of this type are
iterators.

Iterator types should also define the @ref{1028,,tp_iter} function, and that
function should return the iterator instance itself (not a new iterator
instance).

This function has the same signature as @ref{41ab,,PyIter_Next()}.

@strong{Inheritance:}

This field is inherited by subtypes.
@end deffn

@geindex PyTypeObject.tp_methods (C member)
@anchor{c-api/typeobj c PyTypeObject tp_methods}@anchor{40b0}
@deffn {C Member} struct  @ref{1014,,PyMethodDef}  *@ref{5c6,,PyTypeObject}.tp_methods

An optional pointer to a static @code{NULL}-terminated array of @ref{1014,,PyMethodDef}
structures, declaring regular methods of this type.

For each entry in the array, an entry is added to the type’s dictionary (see
@ref{439d,,tp_dict} below) containing a method descriptor.

@strong{Inheritance:}

This field is not inherited by subtypes (methods are inherited through a
different mechanism).
@end deffn

@geindex PyTypeObject.tp_members (C member)
@anchor{c-api/typeobj c PyTypeObject tp_members}@anchor{40ae}
@deffn {C Member} struct  @ref{6eb,,PyMemberDef}  *@ref{5c6,,PyTypeObject}.tp_members

An optional pointer to a static @code{NULL}-terminated array of @ref{6eb,,PyMemberDef}
structures, declaring regular data members (fields or slots) of instances of
this type.

For each entry in the array, an entry is added to the type’s dictionary (see
@ref{439d,,tp_dict} below) containing a member descriptor.

@strong{Inheritance:}

This field is not inherited by subtypes (members are inherited through a
different mechanism).
@end deffn

@geindex PyTypeObject.tp_getset (C member)
@anchor{c-api/typeobj c PyTypeObject tp_getset}@anchor{40b3}
@deffn {C Member} struct  @ref{6ec,,PyGetSetDef}  *@ref{5c6,,PyTypeObject}.tp_getset

An optional pointer to a static @code{NULL}-terminated array of @ref{6ec,,PyGetSetDef}
structures, declaring computed attributes of instances of this type.

For each entry in the array, an entry is added to the type’s dictionary (see
@ref{439d,,tp_dict} below) containing a getset descriptor.

@strong{Inheritance:}

This field is not inherited by subtypes (computed attributes are inherited
through a different mechanism).
@end deffn

@geindex PyTypeObject.tp_base (C member)
@anchor{c-api/typeobj c PyTypeObject tp_base}@anchor{40b7}
@deffn {C Member} @ref{5c6,,PyTypeObject}  *@ref{5c6,,PyTypeObject}.tp_base

An optional pointer to a base type from which type properties are inherited.  At
this level, only single inheritance is supported; multiple inheritance require
dynamically creating a type object by calling the metatype.

@cartouche
@quotation Note 
@c from Modules/xxmodule.c

Slot initialization is subject to the rules of initializing globals.
C99 requires the initializers to be “address constants”.  Function
designators like @ref{40a8,,PyType_GenericNew()}, with implicit conversion
to a pointer, are valid C99 address constants.

However, the unary ‘&’ operator applied to a non-static variable
like @code{PyBaseObject_Type()} is not required to produce an address
constant.  Compilers may support this (gcc does), MSVC does not.
Both compilers are strictly standard conforming in this particular
behavior.

Consequently, @ref{40b7,,tp_base} should be set in
the extension module’s init function.
@end quotation
@end cartouche

@strong{Inheritance:}

This field is not inherited by subtypes (obviously).

@strong{Default:}

This field defaults to @code{&PyBaseObject_Type} (which to Python
programmers is known as the type @ref{5a5,,object}).
@end deffn

@geindex PyTypeObject.tp_dict (C member)
@anchor{c-api/typeobj c PyTypeObject tp_dict}@anchor{439d}
@deffn {C Member} @ref{375,,PyObject}  *@ref{5c6,,PyTypeObject}.tp_dict

The type’s dictionary is stored here by @ref{1ec,,PyType_Ready()}.

This field should normally be initialized to @code{NULL} before PyType_Ready is
called; it may also be initialized to a dictionary containing initial attributes
for the type.  Once @ref{1ec,,PyType_Ready()} has initialized the type, extra
attributes for the type may be added to this dictionary only if they don’t
correspond to overloaded operations (like @code{__add__()}).

@strong{Inheritance:}

This field is not inherited by subtypes (though the attributes defined in here
are inherited through a different mechanism).

@strong{Default:}

If this field is @code{NULL}, @ref{1ec,,PyType_Ready()} will assign a new
dictionary to it.

@cartouche
@quotation Warning 
It is not safe to use @ref{4093,,PyDict_SetItem()} on or otherwise modify
@ref{439d,,tp_dict} with the dictionary C-API.
@end quotation
@end cartouche
@end deffn

@geindex PyTypeObject.tp_descr_get (C member)
@anchor{c-api/typeobj c PyTypeObject tp_descr_get}@anchor{45be}
@deffn {C Member} @ref{42a7,,descrgetfunc}  @ref{5c6,,PyTypeObject}.tp_descr_get

An optional pointer to a “descriptor get” function.

The function signature is:

@example
PyObject * tp_descr_get(PyObject *self, PyObject *obj, PyObject *type);
@end example

@c XXX explain more?

@strong{Inheritance:}

This field is inherited by subtypes.
@end deffn

@geindex PyTypeObject.tp_descr_set (C member)
@anchor{c-api/typeobj c PyTypeObject tp_descr_set}@anchor{116e}
@deffn {C Member} @ref{42a8,,descrsetfunc}  @ref{5c6,,PyTypeObject}.tp_descr_set

An optional pointer to a function for setting and deleting
a descriptor’s value.

The function signature is:

@example
int tp_descr_set(PyObject *self, PyObject *obj, PyObject *value);
@end example

The @emph{value} argument is set to @code{NULL} to delete the value.

@c XXX explain more?

@strong{Inheritance:}

This field is inherited by subtypes.
@end deffn

@geindex PyTypeObject.tp_dictoffset (C member)
@anchor{c-api/typeobj c PyTypeObject tp_dictoffset}@anchor{43a4}
@deffn {C Member} Py_ssize_t  @ref{5c6,,PyTypeObject}.tp_dictoffset

If the instances of this type have a dictionary containing instance variables,
this field is non-zero and contains the offset in the instances of the type of
the instance variable dictionary; this offset is used by
@ref{4204,,PyObject_GenericGetAttr()}.

Do not confuse this field with @ref{439d,,tp_dict}; that is the dictionary for
attributes of the type object itself.

If the value of this field is greater than zero, it specifies the offset from
the start of the instance structure.  If the value is less than zero, it
specifies the offset from the @emph{end} of the instance structure.  A negative
offset is more expensive to use, and should only be used when the instance
structure contains a variable-length part.  This is used for example to add an
instance variable dictionary to subtypes of @ref{2ef,,str} or @ref{3e5,,tuple}. Note
that the @ref{493,,tp_basicsize} field should account for the dictionary added to
the end in that case, even though the dictionary is not included in the basic
object layout.  On a system with a pointer size of 4 bytes,
@ref{43a4,,tp_dictoffset} should be set to @code{-4} to indicate that the dictionary is
at the very end of the structure.

The real dictionary offset in an instance can be computed from a negative
@ref{43a4,,tp_dictoffset} as follows:

@example
dictoffset = tp_basicsize + abs(ob_size)*tp_itemsize + tp_dictoffset
if dictoffset is not aligned on sizeof(void*):
    round up to sizeof(void*)
@end example

where @ref{493,,tp_basicsize}, @ref{40a4,,tp_itemsize} and @ref{43a4,,tp_dictoffset} are
taken from the type object, and @code{ob_size} is taken from the instance.  The
absolute value is taken because ints use the sign of @code{ob_size} to
store the sign of the number.  (There’s never a need to do this calculation
yourself; it is done for you by @code{_PyObject_GetDictPtr()}.)

@strong{Inheritance:}

This field is inherited by subtypes, but see the rules listed below. A subtype
may override this offset; this means that the subtype instances store the
dictionary at a difference offset than the base type.  Since the dictionary is
always found via @ref{43a4,,tp_dictoffset}, this should not be a problem.

When a type defined by a class statement has no @ref{1025,,__slots__} declaration,
and none of its base types has an instance variable dictionary, a dictionary
slot is added to the instance layout and the @ref{43a4,,tp_dictoffset} is set to
that slot’s offset.

When a type defined by a class statement has a @code{__slots__} declaration,
the type inherits its @ref{43a4,,tp_dictoffset} from its base type.

(Adding a slot named @ref{7a4,,__dict__} to the @code{__slots__} declaration does
not have the expected effect, it just causes confusion.  Maybe this should be
added as a feature just like @code{__weakref__} though.)

@strong{Default:}

This slot has no default.  For @ref{1f0,,static types}, if the
field is @code{NULL} then no @code{__dict__} gets created for instances.
@end deffn

@geindex PyTypeObject.tp_init (C member)
@anchor{c-api/typeobj c PyTypeObject tp_init}@anchor{40ad}
@deffn {C Member} @ref{42ae,,initproc}  @ref{5c6,,PyTypeObject}.tp_init

An optional pointer to an instance initialization function.

This function corresponds to the @code{__init__()} method of classes.  Like
@code{__init__()}, it is possible to create an instance without calling
@code{__init__()}, and it is possible to reinitialize an instance by calling its
@code{__init__()} method again.

The function signature is:

@example
int tp_init(PyObject *self, PyObject *args, PyObject *kwds);
@end example

The self argument is the instance to be initialized; the @emph{args} and @emph{kwds}
arguments represent positional and keyword arguments of the call to
@code{__init__()}.

The @ref{40ad,,tp_init} function, if not @code{NULL}, is called when an instance is
created normally by calling its type, after the type’s @ref{40a7,,tp_new} function
has returned an instance of the type.  If the @ref{40a7,,tp_new} function returns an
instance of some other type that is not a subtype of the original type, no
@ref{40ad,,tp_init} function is called; if @ref{40a7,,tp_new} returns an instance of a
subtype of the original type, the subtype’s @ref{40ad,,tp_init} is called.

Returns @code{0} on success, @code{-1} and sets an exception on error.

@strong{Inheritance:}

This field is inherited by subtypes.

@strong{Default:}

For @ref{1f0,,static types} this field does not have a default.
@end deffn

@geindex PyTypeObject.tp_alloc (C member)
@anchor{c-api/typeobj c PyTypeObject tp_alloc}@anchor{40ac}
@deffn {C Member} @ref{42a5,,allocfunc}  @ref{5c6,,PyTypeObject}.tp_alloc

An optional pointer to an instance allocation function.

The function signature is:

@example
PyObject *tp_alloc(PyTypeObject *self, Py_ssize_t nitems);
@end example

@strong{Inheritance:}

This field is inherited by static subtypes, but not by dynamic
subtypes (subtypes created by a class statement).

@strong{Default:}

For dynamic subtypes, this field is always set to
@ref{582,,PyType_GenericAlloc()}, to force a standard heap
allocation strategy.

For static subtypes, @code{PyBaseObject_Type} uses
@ref{582,,PyType_GenericAlloc()}.  That is the recommended value
for all statically defined types.
@end deffn

@geindex PyTypeObject.tp_new (C member)
@anchor{c-api/typeobj c PyTypeObject tp_new}@anchor{40a7}
@deffn {C Member} @ref{42b2,,newfunc}  @ref{5c6,,PyTypeObject}.tp_new

An optional pointer to an instance creation function.

The function signature is:

@example
PyObject *tp_new(PyTypeObject *subtype, PyObject *args, PyObject *kwds);
@end example

The @emph{subtype} argument is the type of the object being created; the @emph{args} and
@emph{kwds} arguments represent positional and keyword arguments of the call to the
type.  Note that @emph{subtype} doesn’t have to equal the type whose @ref{40a7,,tp_new}
function is called; it may be a subtype of that type (but not an unrelated
type).

The @ref{40a7,,tp_new} function should call @code{subtype->tp_alloc(subtype, nitems)}
to allocate space for the object, and then do only as much further
initialization as is absolutely necessary.  Initialization that can safely be
ignored or repeated should be placed in the @ref{40ad,,tp_init} handler.  A good
rule of thumb is that for immutable types, all initialization should take place
in @ref{40a7,,tp_new}, while for mutable types, most initialization should be
deferred to @ref{40ad,,tp_init}.

Set the @ref{460f,,Py_TPFLAGS_DISALLOW_INSTANTIATION} flag to disallow creating
instances of the type in Python.

@strong{Inheritance:}

This field is inherited by subtypes, except it is not inherited by
@ref{1f0,,static types} whose @ref{40b7,,tp_base}
is @code{NULL} or @code{&PyBaseObject_Type}.

@strong{Default:}

For @ref{1f0,,static types} this field has no default.
This means if the slot is defined as @code{NULL}, the type cannot be called
to create new instances; presumably there is some other way to create
instances, like a factory function.
@end deffn

@geindex PyTypeObject.tp_free (C member)
@anchor{c-api/typeobj c PyTypeObject tp_free}@anchor{40ab}
@deffn {C Member} @ref{4477,,freefunc}  @ref{5c6,,PyTypeObject}.tp_free

An optional pointer to an instance deallocation function.  Its signature is:

@example
void tp_free(void *self);
@end example

An initializer that is compatible with this signature is @ref{7ac,,PyObject_Free()}.

@strong{Inheritance:}

This field is inherited by static subtypes, but not by dynamic
subtypes (subtypes created by a class statement)

@strong{Default:}

In dynamic subtypes, this field is set to a deallocator suitable to
match @ref{582,,PyType_GenericAlloc()} and the value of the
@ref{1ed,,Py_TPFLAGS_HAVE_GC} flag bit.

For static subtypes, @code{PyBaseObject_Type} uses PyObject_Del.
@end deffn

@geindex PyTypeObject.tp_is_gc (C member)
@anchor{c-api/typeobj c PyTypeObject tp_is_gc}@anchor{45bf}
@deffn {C Member} @ref{42af,,inquiry}  @ref{5c6,,PyTypeObject}.tp_is_gc

An optional pointer to a function called by the garbage collector.

The garbage collector needs to know whether a particular object is collectible
or not.  Normally, it is sufficient to look at the object’s type’s
@ref{1133,,tp_flags} field, and check the @ref{1ed,,Py_TPFLAGS_HAVE_GC} flag bit.  But
some types have a mixture of statically and dynamically allocated instances, and
the statically allocated instances are not collectible.  Such types should
define this function; it should return @code{1} for a collectible instance, and
@code{0} for a non-collectible instance. The signature is:

@example
int tp_is_gc(PyObject *self);
@end example

(The only example of this are types themselves.  The metatype,
@ref{4255,,PyType_Type}, defines this function to distinguish between statically
and @ref{468,,dynamically allocated types}.)

@strong{Inheritance:}

This field is inherited by subtypes.

@strong{Default:}

This slot has no default.  If this field is @code{NULL},
@ref{1ed,,Py_TPFLAGS_HAVE_GC} is used as the functional equivalent.
@end deffn

@geindex PyTypeObject.tp_bases (C member)
@anchor{c-api/typeobj c PyTypeObject tp_bases}@anchor{45c0}
@deffn {C Member} @ref{375,,PyObject}  *@ref{5c6,,PyTypeObject}.tp_bases

Tuple of base types.

This is set for types created by a class statement.  It should be @code{NULL} for
statically defined types.

@strong{Inheritance:}

This field is not inherited.
@end deffn

@geindex PyTypeObject.tp_mro (C member)
@anchor{c-api/typeobj c PyTypeObject tp_mro}@anchor{439e}
@deffn {C Member} @ref{375,,PyObject}  *@ref{5c6,,PyTypeObject}.tp_mro

Tuple containing the expanded set of base types, starting with the type itself
and ending with @ref{5a5,,object}, in Method Resolution Order.

@strong{Inheritance:}

This field is not inherited; it is calculated fresh by
@ref{1ec,,PyType_Ready()}.
@end deffn

@geindex PyTypeObject.tp_cache (C member)
@anchor{c-api/typeobj c PyTypeObject tp_cache}@anchor{439f}
@deffn {C Member} @ref{375,,PyObject}  *@ref{5c6,,PyTypeObject}.tp_cache

Unused.  Internal use only.

@strong{Inheritance:}

This field is not inherited.
@end deffn

@geindex PyTypeObject.tp_subclasses (C member)
@anchor{c-api/typeobj c PyTypeObject tp_subclasses}@anchor{43a0}
@deffn {C Member} @ref{375,,PyObject}  *@ref{5c6,,PyTypeObject}.tp_subclasses

List of weak references to subclasses.  Internal use only.

@strong{Inheritance:}

This field is not inherited.
@end deffn

@geindex PyTypeObject.tp_weaklist (C member)
@anchor{c-api/typeobj c PyTypeObject tp_weaklist}@anchor{43a1}
@deffn {C Member} @ref{375,,PyObject}  *@ref{5c6,,PyTypeObject}.tp_weaklist

Weak reference list head, for weak references to this type object.  Not
inherited.  Internal use only.

@strong{Inheritance:}

This field is not inherited.
@end deffn

@geindex PyTypeObject.tp_del (C member)
@anchor{c-api/typeobj c PyTypeObject tp_del}@anchor{45c1}
@deffn {C Member} @ref{42a9,,destructor}  @ref{5c6,,PyTypeObject}.tp_del

This field is deprecated.  Use @ref{5c7,,tp_finalize} instead.
@end deffn

@geindex PyTypeObject.tp_version_tag (C member)
@anchor{c-api/typeobj c PyTypeObject tp_version_tag}@anchor{45c2}
@deffn {C Member} unsigned  int  @ref{5c6,,PyTypeObject}.tp_version_tag

Used to index into the method cache.  Internal use only.

@strong{Inheritance:}

This field is not inherited.
@end deffn

@geindex PyTypeObject.tp_finalize (C member)
@anchor{c-api/typeobj c PyTypeObject tp_finalize}@anchor{5c7}
@deffn {C Member} @ref{42a9,,destructor}  @ref{5c6,,PyTypeObject}.tp_finalize

An optional pointer to an instance finalization function.  Its signature is:

@example
void tp_finalize(PyObject *self);
@end example

If @ref{5c7,,tp_finalize} is set, the interpreter calls it once when
finalizing an instance.  It is called either from the garbage
collector (if the instance is part of an isolated reference cycle) or
just before the object is deallocated.  Either way, it is guaranteed
to be called before attempting to break reference cycles, ensuring
that it finds the object in a sane state.

@ref{5c7,,tp_finalize} should not mutate the current exception status;
therefore, a recommended way to write a non-trivial finalizer is:

@example
static void
local_finalize(PyObject *self)
@{
    PyObject *error_type, *error_value, *error_traceback;

    /* Save the current exception, if any. */
    PyErr_Fetch(&error_type, &error_value, &error_traceback);

    /* ... */

    /* Restore the saved exception. */
    PyErr_Restore(error_type, error_value, error_traceback);
@}
@end example

For this field to be taken into account (even through inheritance),
you must also set the @ref{5c8,,Py_TPFLAGS_HAVE_FINALIZE} flags bit.

Also, note that, in a garbage collected Python,
@ref{40aa,,tp_dealloc} may be called from
any Python thread, not just the thread which created the object (if the object
becomes part of a refcount cycle, that cycle might be collected by a garbage
collection on any thread).  This is not a problem for Python API calls, since
the thread on which tp_dealloc is called will own the Global Interpreter Lock
(GIL). However, if the object being destroyed in turn destroys objects from some
other C or C++ library, care should be taken to ensure that destroying those
objects on the thread which called tp_dealloc will not violate any assumptions
of the library.

@strong{Inheritance:}

This field is inherited by subtypes.


New in version 3.4.



@subsubheading See also


“Safe object finalization” (
@geindex Python Enhancement Proposals; PEP 442
PEP 442@footnote{https://www.python.org/dev/peps/pep-0442})

@end deffn

@geindex PyTypeObject.tp_vectorcall (C member)
@anchor{c-api/typeobj c PyTypeObject tp_vectorcall}@anchor{43a2}
@deffn {C Member} @ref{4338,,vectorcallfunc}  @ref{5c6,,PyTypeObject}.tp_vectorcall

Vectorcall function to use for calls of this type object.
In other words, it is used to implement
@ref{4bd,,vectorcall} for @code{type.__call__}.
If @code{tp_vectorcall} is @code{NULL}, the default call implementation
using @code{__new__} and @code{__init__} is used.

@strong{Inheritance:}

This field is never inherited.


New in version 3.9: (the field exists since 3.8 but it’s only used since 3.9)

@end deffn

@node Static Types,Heap Types,PyTypeObject Slots,Type Objects<3>
@anchor{c-api/typeobj id4}@anchor{4611}@anchor{c-api/typeobj static-types}@anchor{1f0}
@subsubsection Static Types


Traditionally, types defined in C code are @emph{static}, that is,
a static @ref{5c6,,PyTypeObject} structure is defined directly in code
and initialized using @ref{1ec,,PyType_Ready()}.

This results in types that are limited relative to types defined in Python:


@itemize *

@item 
Static types are limited to one base, i.e. they cannot use multiple
inheritance.

@item 
Static type objects (but not necessarily their instances) are immutable.
It is not possible to add or modify the type object’s attributes from Python.

@item 
Static type objects are shared across
@ref{42fb,,sub-interpreters}, so they should not
include any subinterpreter-specific state.
@end itemize

Also, since @ref{5c6,,PyTypeObject} is not part of the @ref{35d,,stable ABI},
any extension modules using static types must be compiled for a specific
Python minor version.

@node Heap Types,,Static Types,Type Objects<3>
@anchor{c-api/typeobj heap-types}@anchor{468}@anchor{c-api/typeobj id5}@anchor{4612}
@subsubsection Heap Types


An alternative to @ref{1f0,,static types} is @emph{heap-allocated types},
or @emph{heap types} for short, which correspond closely to classes created by
Python’s @code{class} statement. Heap types have the @ref{4394,,Py_TPFLAGS_HEAPTYPE}
flag set.

This is done by filling a @ref{4254,,PyType_Spec} structure and calling
@ref{469,,PyType_FromSpec()}, @ref{36c,,PyType_FromSpecWithBases()},
or @ref{36d,,PyType_FromModuleAndSpec()}.

@node Number Object Structures,Mapping Object Structures,Type Objects<3>,Object Implementation Support
@anchor{c-api/typeobj number-object-structures}@anchor{4613}@anchor{c-api/typeobj number-structs}@anchor{4601}
@subsection Number Object Structures


@geindex PyNumberMethods (C type)
@anchor{c-api/typeobj c PyNumberMethods}@anchor{f7b}
@deffn {C Type} type  PyNumberMethods

This structure holds pointers to the functions which an object uses to
implement the number protocol.  Each function is used by the function of
similar name documented in the @ref{4346,,Number Protocol} section.

@c XXX Drop the definition?

Here is the structure definition:

@example
typedef struct @{
     binaryfunc nb_add;
     binaryfunc nb_subtract;
     binaryfunc nb_multiply;
     binaryfunc nb_remainder;
     binaryfunc nb_divmod;
     ternaryfunc nb_power;
     unaryfunc nb_negative;
     unaryfunc nb_positive;
     unaryfunc nb_absolute;
     inquiry nb_bool;
     unaryfunc nb_invert;
     binaryfunc nb_lshift;
     binaryfunc nb_rshift;
     binaryfunc nb_and;
     binaryfunc nb_xor;
     binaryfunc nb_or;
     unaryfunc nb_int;
     void *nb_reserved;
     unaryfunc nb_float;

     binaryfunc nb_inplace_add;
     binaryfunc nb_inplace_subtract;
     binaryfunc nb_inplace_multiply;
     binaryfunc nb_inplace_remainder;
     ternaryfunc nb_inplace_power;
     binaryfunc nb_inplace_lshift;
     binaryfunc nb_inplace_rshift;
     binaryfunc nb_inplace_and;
     binaryfunc nb_inplace_xor;
     binaryfunc nb_inplace_or;

     binaryfunc nb_floor_divide;
     binaryfunc nb_true_divide;
     binaryfunc nb_inplace_floor_divide;
     binaryfunc nb_inplace_true_divide;

     unaryfunc nb_index;

     binaryfunc nb_matrix_multiply;
     binaryfunc nb_inplace_matrix_multiply;
@} PyNumberMethods;
@end example

@cartouche
@quotation Note 
Binary and ternary functions must check the type of all their operands,
and implement the necessary conversions (at least one of the operands is
an instance of the defined type).  If the operation is not defined for the
given operands, binary and ternary functions must return
@code{Py_NotImplemented}, if another error occurred they must return @code{NULL}
and set an exception.
@end quotation
@end cartouche

@cartouche
@quotation Note 
The @ref{45e3,,nb_reserved} field should always be @code{NULL}.  It
was previously called @code{nb_long}, and was renamed in
Python 3.0.1.
@end quotation
@end cartouche
@end deffn

@geindex PyNumberMethods.nb_add (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_add}@anchor{4392}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_add
@end deffn

@geindex PyNumberMethods.nb_subtract (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_subtract}@anchor{45ca}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_subtract
@end deffn

@geindex PyNumberMethods.nb_multiply (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_multiply}@anchor{45cc}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_multiply
@end deffn

@geindex PyNumberMethods.nb_remainder (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_remainder}@anchor{45ce}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_remainder
@end deffn

@geindex PyNumberMethods.nb_divmod (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_divmod}@anchor{45d0}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_divmod
@end deffn

@geindex PyNumberMethods.nb_power (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_power}@anchor{45d1}
@deffn {C Member} @ref{42bb,,ternaryfunc}  @ref{f7b,,PyNumberMethods}.nb_power
@end deffn

@geindex PyNumberMethods.nb_negative (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_negative}@anchor{45d3}
@deffn {C Member} @ref{42bd,,unaryfunc}  @ref{f7b,,PyNumberMethods}.nb_negative
@end deffn

@geindex PyNumberMethods.nb_positive (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_positive}@anchor{45d4}
@deffn {C Member} @ref{42bd,,unaryfunc}  @ref{f7b,,PyNumberMethods}.nb_positive
@end deffn

@geindex PyNumberMethods.nb_absolute (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_absolute}@anchor{45d5}
@deffn {C Member} @ref{42bd,,unaryfunc}  @ref{f7b,,PyNumberMethods}.nb_absolute
@end deffn

@geindex PyNumberMethods.nb_bool (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_bool}@anchor{45d6}
@deffn {C Member} @ref{42af,,inquiry}  @ref{f7b,,PyNumberMethods}.nb_bool
@end deffn

@geindex PyNumberMethods.nb_invert (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_invert}@anchor{45d7}
@deffn {C Member} @ref{42bd,,unaryfunc}  @ref{f7b,,PyNumberMethods}.nb_invert
@end deffn

@geindex PyNumberMethods.nb_lshift (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_lshift}@anchor{45d8}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_lshift
@end deffn

@geindex PyNumberMethods.nb_rshift (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_rshift}@anchor{45da}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_rshift
@end deffn

@geindex PyNumberMethods.nb_and (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_and}@anchor{45dc}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_and
@end deffn

@geindex PyNumberMethods.nb_xor (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_xor}@anchor{45de}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_xor
@end deffn

@geindex PyNumberMethods.nb_or (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_or}@anchor{45e0}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_or
@end deffn

@geindex PyNumberMethods.nb_int (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_int}@anchor{45e2}
@deffn {C Member} @ref{42bd,,unaryfunc}  @ref{f7b,,PyNumberMethods}.nb_int
@end deffn

@geindex PyNumberMethods.nb_reserved (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_reserved}@anchor{45e3}
@deffn {C Member} void  *@ref{f7b,,PyNumberMethods}.nb_reserved
@end deffn

@geindex PyNumberMethods.nb_float (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_float}@anchor{45e4}
@deffn {C Member} @ref{42bd,,unaryfunc}  @ref{f7b,,PyNumberMethods}.nb_float
@end deffn

@geindex PyNumberMethods.nb_inplace_add (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_inplace_add}@anchor{45c9}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_inplace_add
@end deffn

@geindex PyNumberMethods.nb_inplace_subtract (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_inplace_subtract}@anchor{45cb}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_inplace_subtract
@end deffn

@geindex PyNumberMethods.nb_inplace_multiply (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_inplace_multiply}@anchor{45cd}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_inplace_multiply
@end deffn

@geindex PyNumberMethods.nb_inplace_remainder (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_inplace_remainder}@anchor{45cf}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_inplace_remainder
@end deffn

@geindex PyNumberMethods.nb_inplace_power (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_inplace_power}@anchor{45d2}
@deffn {C Member} @ref{42bb,,ternaryfunc}  @ref{f7b,,PyNumberMethods}.nb_inplace_power
@end deffn

@geindex PyNumberMethods.nb_inplace_lshift (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_inplace_lshift}@anchor{45d9}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_inplace_lshift
@end deffn

@geindex PyNumberMethods.nb_inplace_rshift (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_inplace_rshift}@anchor{45db}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_inplace_rshift
@end deffn

@geindex PyNumberMethods.nb_inplace_and (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_inplace_and}@anchor{45dd}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_inplace_and
@end deffn

@geindex PyNumberMethods.nb_inplace_xor (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_inplace_xor}@anchor{45df}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_inplace_xor
@end deffn

@geindex PyNumberMethods.nb_inplace_or (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_inplace_or}@anchor{45e1}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_inplace_or
@end deffn

@geindex PyNumberMethods.nb_floor_divide (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_floor_divide}@anchor{45e5}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_floor_divide
@end deffn

@geindex PyNumberMethods.nb_true_divide (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_true_divide}@anchor{45e7}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_true_divide
@end deffn

@geindex PyNumberMethods.nb_inplace_floor_divide (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_inplace_floor_divide}@anchor{45e6}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_inplace_floor_divide
@end deffn

@geindex PyNumberMethods.nb_inplace_true_divide (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_inplace_true_divide}@anchor{45e8}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_inplace_true_divide
@end deffn

@geindex PyNumberMethods.nb_index (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_index}@anchor{45e9}
@deffn {C Member} @ref{42bd,,unaryfunc}  @ref{f7b,,PyNumberMethods}.nb_index
@end deffn

@geindex PyNumberMethods.nb_matrix_multiply (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_matrix_multiply}@anchor{45ea}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_matrix_multiply
@end deffn

@geindex PyNumberMethods.nb_inplace_matrix_multiply (C member)
@anchor{c-api/typeobj c PyNumberMethods nb_inplace_matrix_multiply}@anchor{45eb}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{f7b,,PyNumberMethods}.nb_inplace_matrix_multiply
@end deffn

@node Mapping Object Structures,Sequence Object Structures,Number Object Structures,Object Implementation Support
@anchor{c-api/typeobj mapping-object-structures}@anchor{4614}@anchor{c-api/typeobj mapping-structs}@anchor{4603}
@subsection Mapping Object Structures


@geindex PyMappingMethods (C type)
@anchor{c-api/typeobj c PyMappingMethods}@anchor{40d0}
@deffn {C Type} type  PyMappingMethods

This structure holds pointers to the functions which an object uses to
implement the mapping protocol.  It has three members:
@end deffn

@geindex PyMappingMethods.mp_length (C member)
@anchor{c-api/typeobj c PyMappingMethods mp_length}@anchor{45ec}
@deffn {C Member} @ref{42b1,,lenfunc}  @ref{40d0,,PyMappingMethods}.mp_length

This function is used by @ref{1314,,PyMapping_Size()} and
@ref{4211,,PyObject_Size()}, and has the same signature.  This slot may be set to
@code{NULL} if the object has no defined length.
@end deffn

@geindex PyMappingMethods.mp_subscript (C member)
@anchor{c-api/typeobj c PyMappingMethods mp_subscript}@anchor{11a7}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{40d0,,PyMappingMethods}.mp_subscript

This function is used by @ref{4115,,PyObject_GetItem()} and
@ref{421c,,PySequence_GetSlice()}, and has the same signature as
!@ref{4115,,PyObject_GetItem}.  This slot must be filled for the
@ref{41bf,,PyMapping_Check()} function to return @code{1}, it can be @code{NULL}
otherwise.
@end deffn

@geindex PyMappingMethods.mp_ass_subscript (C member)
@anchor{c-api/typeobj c PyMappingMethods mp_ass_subscript}@anchor{45ed}
@deffn {C Member} @ref{42b3,,objobjargproc}  @ref{40d0,,PyMappingMethods}.mp_ass_subscript

This function is used by @ref{4113,,PyObject_SetItem()},
@ref{4202,,PyObject_DelItem()}, @code{PyObject_SetSlice()} and
@code{PyObject_DelSlice()}.  It has the same signature as
!@ref{4113,,PyObject_SetItem}, but @emph{v} can also be set to @code{NULL} to delete
an item.  If this slot is @code{NULL}, the object does not support item
assignment and deletion.
@end deffn

@node Sequence Object Structures,Buffer Object Structures,Mapping Object Structures,Object Implementation Support
@anchor{c-api/typeobj sequence-object-structures}@anchor{4615}@anchor{c-api/typeobj sequence-structs}@anchor{4602}
@subsection Sequence Object Structures


@geindex PySequenceMethods (C type)
@anchor{c-api/typeobj c PySequenceMethods}@anchor{40cf}
@deffn {C Type} type  PySequenceMethods

This structure holds pointers to the functions which an object uses to
implement the sequence protocol.
@end deffn

@geindex PySequenceMethods.sq_length (C member)
@anchor{c-api/typeobj c PySequenceMethods sq_length}@anchor{439c}
@deffn {C Member} @ref{42b1,,lenfunc}  @ref{40cf,,PySequenceMethods}.sq_length

This function is used by @ref{1310,,PySequence_Size()} and
@ref{4211,,PyObject_Size()}, and has the same signature.  It is also used for
handling negative indices via the @ref{11a6,,sq_item}
and the @ref{45f0,,sq_ass_item} slots.
@end deffn

@geindex PySequenceMethods.sq_concat (C member)
@anchor{c-api/typeobj c PySequenceMethods sq_concat}@anchor{45ee}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{40cf,,PySequenceMethods}.sq_concat

This function is used by @ref{4217,,PySequence_Concat()} and has the same
signature.  It is also used by the @code{+} operator, after trying the numeric
addition via the @ref{4392,,nb_add} slot.
@end deffn

@geindex PySequenceMethods.sq_repeat (C member)
@anchor{c-api/typeobj c PySequenceMethods sq_repeat}@anchor{45ef}
@deffn {C Member} @ref{42b9,,ssizeargfunc}  @ref{40cf,,PySequenceMethods}.sq_repeat

This function is used by @ref{4222,,PySequence_Repeat()} and has the same
signature.  It is also used by the @code{*} operator, after trying numeric
multiplication via the @ref{45cc,,nb_multiply} slot.
@end deffn

@geindex PySequenceMethods.sq_item (C member)
@anchor{c-api/typeobj c PySequenceMethods sq_item}@anchor{11a6}
@deffn {C Member} @ref{42b9,,ssizeargfunc}  @ref{40cf,,PySequenceMethods}.sq_item

This function is used by @ref{1311,,PySequence_GetItem()} and has the same
signature.  It is also used by @ref{4115,,PyObject_GetItem()}, after trying
the subscription via the @ref{11a7,,mp_subscript} slot.
This slot must be filled for the @ref{4216,,PySequence_Check()}
function to return @code{1}, it can be @code{NULL} otherwise.

Negative indexes are handled as follows: if the @code{sq_length} slot is
filled, it is called and the sequence length is used to compute a positive
index which is passed to @code{sq_item}.  If @code{sq_length} is @code{NULL},
the index is passed as is to the function.
@end deffn

@geindex PySequenceMethods.sq_ass_item (C member)
@anchor{c-api/typeobj c PySequenceMethods sq_ass_item}@anchor{45f0}
@deffn {C Member} @ref{42ba,,ssizeobjargproc}  @ref{40cf,,PySequenceMethods}.sq_ass_item

This function is used by @ref{1312,,PySequence_SetItem()} and has the same
signature.  It is also used by @ref{4113,,PyObject_SetItem()} and
@ref{4202,,PyObject_DelItem()}, after trying the item assignment and deletion
via the @ref{45ed,,mp_ass_subscript} slot.
This slot may be left to @code{NULL} if the object does not support
item assignment and deletion.
@end deffn

@geindex PySequenceMethods.sq_contains (C member)
@anchor{c-api/typeobj c PySequenceMethods sq_contains}@anchor{45f1}
@deffn {C Member} @ref{42b4,,objobjproc}  @ref{40cf,,PySequenceMethods}.sq_contains

This function may be used by @ref{4218,,PySequence_Contains()} and has the same
signature.  This slot may be left to @code{NULL}, in this case
!@ref{4218,,PySequence_Contains} simply traverses the sequence until it
finds a match.
@end deffn

@geindex PySequenceMethods.sq_inplace_concat (C member)
@anchor{c-api/typeobj c PySequenceMethods sq_inplace_concat}@anchor{45f2}
@deffn {C Member} @ref{42a6,,binaryfunc}  @ref{40cf,,PySequenceMethods}.sq_inplace_concat

This function is used by @ref{421d,,PySequence_InPlaceConcat()} and has the same
signature.  It should modify its first operand, and return it.  This slot
may be left to @code{NULL}, in this case !@ref{421d,,PySequence_InPlaceConcat}
will fall back to @ref{4217,,PySequence_Concat()}.  It is also used by the
augmented assignment @code{+=}, after trying numeric in-place addition
via the @ref{45c9,,nb_inplace_add} slot.
@end deffn

@geindex PySequenceMethods.sq_inplace_repeat (C member)
@anchor{c-api/typeobj c PySequenceMethods sq_inplace_repeat}@anchor{45f3}
@deffn {C Member} @ref{42b9,,ssizeargfunc}  @ref{40cf,,PySequenceMethods}.sq_inplace_repeat

This function is used by @ref{421e,,PySequence_InPlaceRepeat()} and has the same
signature.  It should modify its first operand, and return it.  This slot
may be left to @code{NULL}, in this case !@ref{421e,,PySequence_InPlaceRepeat}
will fall back to @ref{4222,,PySequence_Repeat()}.  It is also used by the
augmented assignment @code{*=}, after trying numeric in-place multiplication
via the @ref{45cd,,nb_inplace_multiply} slot.
@end deffn

@node Buffer Object Structures,Async Object Structures,Sequence Object Structures,Object Implementation Support
@anchor{c-api/typeobj buffer-object-structures}@anchor{4616}@anchor{c-api/typeobj buffer-structs}@anchor{435b}
@subsection Buffer Object Structures


@geindex PyBufferProcs (C type)
@anchor{c-api/typeobj c PyBufferProcs}@anchor{43a5}
@deffn {C Type} type  PyBufferProcs

This structure holds pointers to the functions required by the
@ref{1a3e,,Buffer protocol}. The protocol defines how
an exporter object can expose its internal data to consumer objects.
@end deffn

@geindex PyBufferProcs.bf_getbuffer (C member)
@anchor{c-api/typeobj c PyBufferProcs bf_getbuffer}@anchor{1e7}
@deffn {C Member} @ref{45f4,,getbufferproc}  @ref{43a5,,PyBufferProcs}.bf_getbuffer

The signature of this function is:

@example
int (PyObject *exporter, Py_buffer *view, int flags);
@end example

Handle a request to @emph{exporter} to fill in @emph{view} as specified by @emph{flags}.
Except for point (3), an implementation of this function MUST take these
steps:


@enumerate 

@item 
Check if the request can be met. If not, raise @code{PyExc_BufferError},
set view->obj to @code{NULL} and return @code{-1}.

@item 
Fill in the requested fields.

@item 
Increment an internal counter for the number of exports.

@item 
Set view->obj to @emph{exporter} and increment view->obj.

@item 
Return @code{0}.
@end enumerate

If @emph{exporter} is part of a chain or tree of buffer providers, two main
schemes can be used:


@itemize *

@item 
Re-export: Each member of the tree acts as the exporting object and
sets view->obj to a new reference to itself.

@item 
Redirect: The buffer request is redirected to the root object of the
tree. Here, view->obj will be a new reference to the root
object.
@end itemize

The individual fields of @emph{view} are described in section
@ref{435c,,Buffer structure}, the rules how an exporter
must react to specific requests are in section
@ref{436e,,Buffer request types}.

All memory pointed to in the @ref{1db,,Py_buffer} structure belongs to
the exporter and must remain valid until there are no consumers left.
@ref{4367,,format}, @ref{4369,,shape},
@ref{4360,,strides}, @ref{436b,,suboffsets}
and @ref{436d,,internal}
are read-only for the consumer.

@ref{1e4,,PyBuffer_FillInfo()} provides an easy way of exposing a simple
bytes buffer while dealing correctly with all request types.

@ref{1dd,,PyObject_GetBuffer()} is the interface for the consumer that
wraps this function.
@end deffn

@geindex PyBufferProcs.bf_releasebuffer (C member)
@anchor{c-api/typeobj c PyBufferProcs bf_releasebuffer}@anchor{1e8}
@deffn {C Member} @ref{45f5,,releasebufferproc}  @ref{43a5,,PyBufferProcs}.bf_releasebuffer

The signature of this function is:

@example
void (PyObject *exporter, Py_buffer *view);
@end example

Handle a request to release the resources of the buffer. If no resources
need to be released, @ref{1e8,,PyBufferProcs.bf_releasebuffer} may be
@code{NULL}. Otherwise, a standard implementation of this function will take
these optional steps:


@enumerate 

@item 
Decrement an internal counter for the number of exports.

@item 
If the counter is @code{0}, free all memory associated with @emph{view}.
@end enumerate

The exporter MUST use the @ref{436d,,internal} field to keep
track of buffer-specific resources. This field is guaranteed to remain
constant, while a consumer MAY pass a copy of the original buffer as the
@emph{view} argument.

This function MUST NOT decrement view->obj, since that is
done automatically in @ref{1e5,,PyBuffer_Release()} (this scheme is
useful for breaking reference cycles).

@ref{1e5,,PyBuffer_Release()} is the interface for the consumer that
wraps this function.
@end deffn

@node Async Object Structures,Slot Type typedefs,Buffer Object Structures,Object Implementation Support
@anchor{c-api/typeobj async-object-structures}@anchor{4617}@anchor{c-api/typeobj async-structs}@anchor{4600}
@subsection Async Object Structures



New in version 3.5.


@geindex PyAsyncMethods (C type)
@anchor{c-api/typeobj c PyAsyncMethods}@anchor{439b}
@deffn {C Type} type  PyAsyncMethods

This structure holds pointers to the functions required to implement
@ref{7bd,,awaitable} and @ref{8d2,,asynchronous iterator} objects.

Here is the structure definition:

@example
typedef struct @{
    unaryfunc am_await;
    unaryfunc am_aiter;
    unaryfunc am_anext;
    sendfunc am_send;
@} PyAsyncMethods;
@end example
@end deffn

@geindex PyAsyncMethods.am_await (C member)
@anchor{c-api/typeobj c PyAsyncMethods am_await}@anchor{45c4}
@deffn {C Member} @ref{42bd,,unaryfunc}  @ref{439b,,PyAsyncMethods}.am_await

The signature of this function is:

@example
PyObject *am_await(PyObject *self);
@end example

The returned object must be an @ref{11c3,,iterator}, i.e. @ref{116c,,PyIter_Check()}
must return @code{1} for it.

This slot may be set to @code{NULL} if an object is not an @ref{7bd,,awaitable}.
@end deffn

@geindex PyAsyncMethods.am_aiter (C member)
@anchor{c-api/typeobj c PyAsyncMethods am_aiter}@anchor{45c5}
@deffn {C Member} @ref{42bd,,unaryfunc}  @ref{439b,,PyAsyncMethods}.am_aiter

The signature of this function is:

@example
PyObject *am_aiter(PyObject *self);
@end example

Must return an @ref{8d2,,asynchronous iterator} object.
See @code{__anext__()} for details.

This slot may be set to @code{NULL} if an object does not implement
asynchronous iteration protocol.
@end deffn

@geindex PyAsyncMethods.am_anext (C member)
@anchor{c-api/typeobj c PyAsyncMethods am_anext}@anchor{45c6}
@deffn {C Member} @ref{42bd,,unaryfunc}  @ref{439b,,PyAsyncMethods}.am_anext

The signature of this function is:

@example
PyObject *am_anext(PyObject *self);
@end example

Must return an @ref{7bd,,awaitable} object.  See @code{__anext__()} for details.
This slot may be set to @code{NULL}.
@end deffn

@geindex PyAsyncMethods.am_send (C member)
@anchor{c-api/typeobj c PyAsyncMethods am_send}@anchor{45c7}
@deffn {C Member} @ref{45c8,,sendfunc}  @ref{439b,,PyAsyncMethods}.am_send

The signature of this function is:

@example
PySendResult am_send(PyObject *self, PyObject *arg, PyObject **result);
@end example

See @ref{366,,PyIter_Send()} for details.
This slot may be set to @code{NULL}.


New in version 3.10.

@end deffn

@node Slot Type typedefs,Examples<38>,Async Object Structures,Object Implementation Support
@anchor{c-api/typeobj id6}@anchor{45f7}@anchor{c-api/typeobj slot-type-typedefs}@anchor{4618}
@subsection Slot Type typedefs


@geindex allocfunc (C type)
@anchor{c-api/typeobj c allocfunc}@anchor{42a5}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*allocfunc)(@ref{5c6,,PyTypeObject}  *cls,  Py_ssize_t  nitems)
@emph{ Part of the @ref{35d,,Stable ABI}.}
The purpose of this function is to separate memory allocation from memory
initialization.  It should return a pointer to a block of memory of adequate
length for the instance, suitably aligned, and initialized to zeros, but with
@code{ob_refcnt} set to @code{1} and @code{ob_type} set to the type argument.  If
the type’s @ref{40a4,,tp_itemsize} is non-zero, the object’s @code{ob_size} field
should be initialized to @emph{nitems} and the length of the allocated memory block
should be @code{tp_basicsize + nitems*tp_itemsize}, rounded up to a multiple of
@code{sizeof(void*)}; otherwise, @emph{nitems} is not used and the length of the block
should be @ref{493,,tp_basicsize}.

This function should not do any other instance initialization, not even to
allocate additional memory; that should be done by @ref{40a7,,tp_new}.
@end deffn

@geindex destructor (C type)
@anchor{c-api/typeobj c destructor}@anchor{42a9}
@deffn {C Type} typedef  void  (*destructor)(@ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex freefunc (C type)
@anchor{c-api/typeobj c freefunc}@anchor{4477}
@deffn {C Type} typedef  void  (*freefunc)(void*)

See @ref{40ab,,tp_free}.
@end deffn

@geindex newfunc (C type)
@anchor{c-api/typeobj c newfunc}@anchor{42b2}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*newfunc)(@ref{375,,PyObject}*,  @ref{375,,PyObject}*,  @ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
See @ref{40a7,,tp_new}.
@end deffn

@geindex initproc (C type)
@anchor{c-api/typeobj c initproc}@anchor{42ae}
@deffn {C Type} typedef  int  (*initproc)(@ref{375,,PyObject}*,  @ref{375,,PyObject}*,  @ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
See @ref{40ad,,tp_init}.
@end deffn

@geindex reprfunc (C type)
@anchor{c-api/typeobj c reprfunc}@anchor{42b5}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*reprfunc)(@ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
See @ref{40c1,,tp_repr}.
@end deffn

@geindex getattrfunc (C type)
@anchor{c-api/typeobj c getattrfunc}@anchor{42aa}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*getattrfunc)(@ref{375,,PyObject}  *self,  char  *attr)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the value of the named attribute for the object.
@end deffn

@geindex setattrfunc (C type)
@anchor{c-api/typeobj c setattrfunc}@anchor{42b7}
@deffn {C Type} typedef  int  (*setattrfunc)(@ref{375,,PyObject}  *self,  char  *attr,  @ref{375,,PyObject}  *value)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the value of the named attribute for the object.
The value argument is set to @code{NULL} to delete the attribute.
@end deffn

@geindex getattrofunc (C type)
@anchor{c-api/typeobj c getattrofunc}@anchor{42ab}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*getattrofunc)(@ref{375,,PyObject}  *self,  @ref{375,,PyObject}  *attr)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Return the value of the named attribute for the object.

See @ref{40c5,,tp_getattro}.
@end deffn

@geindex setattrofunc (C type)
@anchor{c-api/typeobj c setattrofunc}@anchor{42b8}
@deffn {C Type} typedef  int  (*setattrofunc)(@ref{375,,PyObject}  *self,  @ref{375,,PyObject}  *attr,  @ref{375,,PyObject}  *value)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Set the value of the named attribute for the object.
The value argument is set to @code{NULL} to delete the attribute.

See @ref{40c6,,tp_setattro}.
@end deffn

@geindex descrgetfunc (C type)
@anchor{c-api/typeobj c descrgetfunc}@anchor{42a7}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*descrgetfunc)(@ref{375,,PyObject}*,  @ref{375,,PyObject}*,  @ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
See @code{tp_descrget}.
@end deffn

@geindex descrsetfunc (C type)
@anchor{c-api/typeobj c descrsetfunc}@anchor{42a8}
@deffn {C Type} typedef  int  (*descrsetfunc)(@ref{375,,PyObject}*,  @ref{375,,PyObject}*,  @ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
See @code{tp_descrset}.
@end deffn

@geindex hashfunc (C type)
@anchor{c-api/typeobj c hashfunc}@anchor{42ad}
@deffn {C Type} typedef  Py_hash_t  (*hashfunc)(@ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
See @ref{40d1,,tp_hash}.
@end deffn

@geindex richcmpfunc (C type)
@anchor{c-api/typeobj c richcmpfunc}@anchor{42b6}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*richcmpfunc)(@ref{375,,PyObject}*,  @ref{375,,PyObject}*,  int)
@emph{ Part of the @ref{35d,,Stable ABI}.}
See @ref{40cb,,tp_richcompare}.
@end deffn

@geindex getiterfunc (C type)
@anchor{c-api/typeobj c getiterfunc}@anchor{42ac}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*getiterfunc)(@ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
See @ref{1028,,tp_iter}.
@end deffn

@geindex iternextfunc (C type)
@anchor{c-api/typeobj c iternextfunc}@anchor{42b0}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*iternextfunc)(@ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
See @ref{1029,,tp_iternext}.
@end deffn

@geindex lenfunc (C type)
@anchor{c-api/typeobj c lenfunc}@anchor{42b1}
@deffn {C Type} typedef  Py_ssize_t  (*lenfunc)(@ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex getbufferproc (C type)
@anchor{c-api/typeobj c getbufferproc}@anchor{45f4}
@deffn {C Type} typedef  int  (*getbufferproc)(@ref{375,,PyObject}*,  @ref{1db,,Py_buffer}*,  int)
@end deffn

@geindex releasebufferproc (C type)
@anchor{c-api/typeobj c releasebufferproc}@anchor{45f5}
@deffn {C Type} typedef  void  (*releasebufferproc)(@ref{375,,PyObject}*,  @ref{1db,,Py_buffer}*)
@end deffn

@geindex unaryfunc (C type)
@anchor{c-api/typeobj c unaryfunc}@anchor{42bd}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*unaryfunc)(@ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex binaryfunc (C type)
@anchor{c-api/typeobj c binaryfunc}@anchor{42a6}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*binaryfunc)(@ref{375,,PyObject}*,  @ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex sendfunc (C type)
@anchor{c-api/typeobj c sendfunc}@anchor{45c8}
@deffn {C Type} typedef  @ref{4358,,PySendResult}  (*sendfunc)(@ref{375,,PyObject}*,  @ref{375,,PyObject}*,  @ref{375,,PyObject}**)

See @ref{45c7,,am_send}.
@end deffn

@geindex ternaryfunc (C type)
@anchor{c-api/typeobj c ternaryfunc}@anchor{42bb}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*ternaryfunc)(@ref{375,,PyObject}*,  @ref{375,,PyObject}*,  @ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex ssizeargfunc (C type)
@anchor{c-api/typeobj c ssizeargfunc}@anchor{42b9}
@deffn {C Type} typedef  @ref{375,,PyObject}  *(*ssizeargfunc)(@ref{375,,PyObject}*,  Py_ssize_t)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex ssizeobjargproc (C type)
@anchor{c-api/typeobj c ssizeobjargproc}@anchor{42ba}
@deffn {C Type} typedef  int  (*ssizeobjargproc)(@ref{375,,PyObject}*,  Py_ssize_t)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex objobjproc (C type)
@anchor{c-api/typeobj c objobjproc}@anchor{42b4}
@deffn {C Type} typedef  int  (*objobjproc)(@ref{375,,PyObject}*,  @ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@geindex objobjargproc (C type)
@anchor{c-api/typeobj c objobjargproc}@anchor{42b3}
@deffn {C Type} typedef  int  (*objobjargproc)(@ref{375,,PyObject}*,  @ref{375,,PyObject}*,  @ref{375,,PyObject}*)
@emph{ Part of the @ref{35d,,Stable ABI}.}
@end deffn

@node Examples<38>,Supporting Cyclic Garbage Collection,Slot Type typedefs,Object Implementation Support
@anchor{c-api/typeobj examples}@anchor{4619}@anchor{c-api/typeobj typedef-examples}@anchor{45b5}
@subsection Examples


The following are simple examples of Python type definitions.  They
include common usage you may encounter.  Some demonstrate tricky corner
cases.  For more examples, practical info, and a tutorial, see
@ref{40a0,,Defining Extension Types; Tutorial} and @ref{40bb,,Defining Extension Types; Assorted Topics}.

A basic @ref{1f0,,static type}:

@example
typedef struct @{
    PyObject_HEAD
    const char *data;
@} MyObject;

static PyTypeObject MyObject_Type = @{
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "mymod.MyObject",
    .tp_basicsize = sizeof(MyObject),
    .tp_doc = "My objects",
    .tp_new = myobj_new,
    .tp_dealloc = (destructor)myobj_dealloc,
    .tp_repr = (reprfunc)myobj_repr,
@};
@end example

You may also find older code (especially in the CPython code base)
with a more verbose initializer:

@example
static PyTypeObject MyObject_Type = @{
    PyVarObject_HEAD_INIT(NULL, 0)
    "mymod.MyObject",               /* tp_name */
    sizeof(MyObject),               /* tp_basicsize */
    0,                              /* tp_itemsize */
    (destructor)myobj_dealloc,      /* tp_dealloc */
    0,                              /* tp_vectorcall_offset */
    0,                              /* tp_getattr */
    0,                              /* tp_setattr */
    0,                              /* tp_as_async */
    (reprfunc)myobj_repr,           /* tp_repr */
    0,                              /* tp_as_number */
    0,                              /* tp_as_sequence */
    0,                              /* tp_as_mapping */
    0,                              /* tp_hash */
    0,                              /* tp_call */
    0,                              /* tp_str */
    0,                              /* tp_getattro */
    0,                              /* tp_setattro */
    0,                              /* tp_as_buffer */
    0,                              /* tp_flags */
    "My objects",                   /* tp_doc */
    0,                              /* tp_traverse */
    0,                              /* tp_clear */
    0,                              /* tp_richcompare */
    0,                              /* tp_weaklistoffset */
    0,                              /* tp_iter */
    0,                              /* tp_iternext */
    0,                              /* tp_methods */
    0,                              /* tp_members */
    0,                              /* tp_getset */
    0,                              /* tp_base */
    0,                              /* tp_dict */
    0,                              /* tp_descr_get */
    0,                              /* tp_descr_set */
    0,                              /* tp_dictoffset */
    0,                              /* tp_init */
    0,                              /* tp_alloc */
    myobj_new,                      /* tp_new */
@};
@end example

A type that supports weakrefs, instance dicts, and hashing:

@example
typedef struct @{
    PyObject_HEAD
    const char *data;
    PyObject *inst_dict;
    PyObject *weakreflist;
@} MyObject;

static PyTypeObject MyObject_Type = @{
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "mymod.MyObject",
    .tp_basicsize = sizeof(MyObject),
    .tp_doc = "My objects",
    .tp_weaklistoffset = offsetof(MyObject, weakreflist),
    .tp_dictoffset = offsetof(MyObject, inst_dict),
    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC,
    .tp_new = myobj_new,
    .tp_traverse = (traverseproc)myobj_traverse,
    .tp_clear = (inquiry)myobj_clear,
    .tp_alloc = PyType_GenericNew,
    .tp_dealloc = (destructor)myobj_dealloc,
    .tp_repr = (reprfunc)myobj_repr,
    .tp_hash = (hashfunc)myobj_hash,
    .tp_richcompare = PyBaseObject_Type.tp_richcompare,
@};
@end example

A str subclass that cannot be subclassed and cannot be called
to create instances (e.g. uses a separate factory func) using
@code{Py_TPFLAGS_DISALLOW_INSTANTIATION} flag:

@example
typedef struct @{
    PyUnicodeObject raw;
    char *extra;
@} MyStr;

static PyTypeObject MyStr_Type = @{
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "mymod.MyStr",
    .tp_basicsize = sizeof(MyStr),
    .tp_base = NULL,  // set to &PyUnicode_Type in module init
    .tp_doc = "my custom str",
    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_DISALLOW_INSTANTIATION,
    .tp_repr = (reprfunc)myobj_repr,
@};
@end example

The simplest @ref{1f0,,static type} with fixed-length instances:

@example
typedef struct @{
    PyObject_HEAD
@} MyObject;

static PyTypeObject MyObject_Type = @{
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "mymod.MyObject",
@};
@end example

The simplest @ref{1f0,,static type} with variable-length instances:

@example
typedef struct @{
    PyObject_VAR_HEAD
    const char *data[1];
@} MyObject;

static PyTypeObject MyObject_Type = @{
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "mymod.MyObject",
    .tp_basicsize = sizeof(MyObject) - sizeof(char *),
    .tp_itemsize = sizeof(char *),
@};
@end example

@node Supporting Cyclic Garbage Collection,,Examples<38>,Object Implementation Support
@anchor{c-api/gcsupport doc}@anchor{461a}@anchor{c-api/gcsupport supporting-cycle-detection}@anchor{4606}@anchor{c-api/gcsupport supporting-cyclic-garbage-collection}@anchor{461b}
@subsection Supporting Cyclic Garbage Collection


Python’s support for detecting and collecting garbage which involves circular
references requires support from object types which are “containers” for other
objects which may also be containers.  Types which do not store references to
other objects, or which only store references to atomic types (such as numbers
or strings), do not need to provide any explicit support for garbage
collection.

To create a container type, the @ref{1133,,tp_flags} field of the type object must
include the @ref{1ed,,Py_TPFLAGS_HAVE_GC} and provide an implementation of the
@ref{1ee,,tp_traverse} handler.  If instances of the type are mutable, a
@ref{40bf,,tp_clear} implementation must also be provided.


@deffn {Data} Py_TPFLAGS_HAVE_GC

Objects with a type with this flag set must conform with the rules
documented here.  For convenience these objects will be referred to as
container objects.
@end deffn

Constructors for container types must conform to two rules:


@enumerate 

@item 
The memory for the object must be allocated using @ref{5c3,,PyObject_GC_New()}
or @ref{5c4,,PyObject_GC_NewVar()}.

@item 
Once all the fields which may contain references to other containers are
initialized, it must call @ref{103a,,PyObject_GC_Track()}.
@end enumerate

Similarly, the deallocator for the object must conform to a similar pair of
rules:


@enumerate 

@item 
Before fields which refer to other containers are invalidated,
@ref{103b,,PyObject_GC_UnTrack()} must be called.

@item 
The object’s memory must be deallocated using @ref{1039,,PyObject_GC_Del()}.

@cartouche
@quotation Warning 
If a type adds the Py_TPFLAGS_HAVE_GC, then it @emph{must} implement at least
a @ref{1ee,,tp_traverse} handler or explicitly use one
from its subclass or subclasses.

When calling @ref{1ec,,PyType_Ready()} or some of the APIs that indirectly
call it like @ref{36c,,PyType_FromSpecWithBases()} or
@ref{469,,PyType_FromSpec()} the interpreter will automatically populate the
@ref{1133,,tp_flags}, @ref{1ee,,tp_traverse}
and @ref{40bf,,tp_clear} fields if the type inherits from a
class that implements the garbage collector protocol and the child class
does @emph{not} include the @ref{1ed,,Py_TPFLAGS_HAVE_GC} flag.
@end quotation
@end cartouche
@end enumerate

@geindex PyObject_GC_New (C function)
@anchor{c-api/gcsupport c PyObject_GC_New}@anchor{5c3}
@deffn {C Function} TYPE  *PyObject_GC_New (TYPE, PyTypeObject  *type)

Analogous to @ref{491,,PyObject_New()} but for container objects with the
@ref{1ed,,Py_TPFLAGS_HAVE_GC} flag set.
@end deffn

@geindex PyObject_GC_NewVar (C function)
@anchor{c-api/gcsupport c PyObject_GC_NewVar}@anchor{5c4}
@deffn {C Function} TYPE  *PyObject_GC_NewVar (TYPE, PyTypeObject  *type, Py_ssize_t  size)

Analogous to @ref{492,,PyObject_NewVar()} but for container objects with the
@ref{1ed,,Py_TPFLAGS_HAVE_GC} flag set.
@end deffn

@geindex PyObject_GC_Resize (C function)
@anchor{c-api/gcsupport c PyObject_GC_Resize}@anchor{461c}
@deffn {C Function} TYPE  *PyObject_GC_Resize (TYPE, PyVarObject  *op, Py_ssize_t  newsize)

Resize an object allocated by @ref{492,,PyObject_NewVar()}.  Returns the
resized object or @code{NULL} on failure.  @emph{op} must not be tracked by the collector yet.
@end deffn

@geindex PyObject_GC_Track (C function)
@anchor{c-api/gcsupport c PyObject_GC_Track}@anchor{103a}
@deffn {C Function} void  PyObject_GC_Track (PyObject  *op)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Adds the object @emph{op} to the set of container objects tracked by the
collector.  The collector can run at unexpected times so objects must be
valid while being tracked.  This should be called once all the fields
followed by the @ref{1ee,,tp_traverse} handler become valid, usually near the
end of the constructor.
@end deffn

@geindex PyObject_IS_GC (C function)
@anchor{c-api/gcsupport c PyObject_IS_GC}@anchor{490}
@deffn {C Function} int  PyObject_IS_GC (PyObject  *obj)

Returns non-zero if the object implements the garbage collector protocol,
otherwise returns 0.

The object cannot be tracked by the garbage collector if this function returns 0.
@end deffn

@geindex PyObject_GC_IsTracked (C function)
@anchor{c-api/gcsupport c PyObject_GC_IsTracked}@anchor{47d}
@deffn {C Function} int  PyObject_GC_IsTracked (PyObject  *op)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.9.}
Returns 1 if the object type of @emph{op} implements the GC protocol and @emph{op} is being
currently tracked by the garbage collector and 0 otherwise.

This is analogous to the Python function @ref{ea5,,gc.is_tracked()}.


New in version 3.9.

@end deffn

@geindex PyObject_GC_IsFinalized (C function)
@anchor{c-api/gcsupport c PyObject_GC_IsFinalized}@anchor{47e}
@deffn {C Function} int  PyObject_GC_IsFinalized (PyObject  *op)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.9.}
Returns 1 if the object type of @emph{op} implements the GC protocol and @emph{op} has been
already finalized by the garbage collector and 0 otherwise.

This is analogous to the Python function @ref{3ec,,gc.is_finalized()}.


New in version 3.9.

@end deffn

@geindex PyObject_GC_Del (C function)
@anchor{c-api/gcsupport c PyObject_GC_Del}@anchor{1039}
@deffn {C Function} void  PyObject_GC_Del (void  *op)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Releases memory allocated to an object using @ref{5c3,,PyObject_GC_New()} or
@ref{5c4,,PyObject_GC_NewVar()}.
@end deffn

@geindex PyObject_GC_UnTrack (C function)
@anchor{c-api/gcsupport c PyObject_GC_UnTrack}@anchor{103b}
@deffn {C Function} void  PyObject_GC_UnTrack (void  *op)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Remove the object @emph{op} from the set of container objects tracked by the
collector.  Note that @ref{103a,,PyObject_GC_Track()} can be called again on
this object to add it back to the set of tracked objects.  The deallocator
(@ref{40aa,,tp_dealloc} handler) should call this for the object before any of
the fields used by the @ref{1ee,,tp_traverse} handler become invalid.
@end deffn


Changed in version 3.8: The @code{_PyObject_GC_TRACK()} and @code{_PyObject_GC_UNTRACK()} macros
have been removed from the public C API.


The @ref{1ee,,tp_traverse} handler accepts a function parameter of this type:

@geindex visitproc (C type)
@anchor{c-api/gcsupport c visitproc}@anchor{42be}
@deffn {C Type} typedef  int  (*visitproc)(@ref{375,,PyObject}  *object,  void  *arg)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Type of the visitor function passed to the @ref{1ee,,tp_traverse} handler.
The function should be called with an object to traverse as @emph{object} and
the third parameter to the @ref{1ee,,tp_traverse} handler as @emph{arg}.  The
Python core uses several visitor functions to implement cyclic garbage
detection; it’s not expected that users will need to write their own
visitor functions.
@end deffn

The @ref{1ee,,tp_traverse} handler must have the following type:

@geindex traverseproc (C type)
@anchor{c-api/gcsupport c traverseproc}@anchor{42bc}
@deffn {C Type} typedef  int  (*traverseproc)(@ref{375,,PyObject}  *self,  @ref{42be,,visitproc}  visit,  void  *arg)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Traversal function for a container object.  Implementations must call the
@emph{visit} function for each object directly contained by @emph{self}, with the
parameters to @emph{visit} being the contained object and the @emph{arg} value passed
to the handler.  The @emph{visit} function must not be called with a @code{NULL}
object argument.  If @emph{visit} returns a non-zero value that value should be
returned immediately.
@end deffn

To simplify writing @ref{1ee,,tp_traverse} handlers, a @ref{40b5,,Py_VISIT()} macro is
provided.  In order to use this macro, the @ref{1ee,,tp_traverse} implementation
must name its arguments exactly @emph{visit} and @emph{arg}:

@geindex Py_VISIT (C function)
@anchor{c-api/gcsupport c Py_VISIT}@anchor{40b5}
@deffn {C Function} void  Py_VISIT (PyObject  *o)

If @emph{o} is not @code{NULL}, call the @emph{visit} callback, with arguments @emph{o}
and @emph{arg}.  If @emph{visit} returns a non-zero value, then return it.
Using this macro, @ref{1ee,,tp_traverse} handlers
look like:

@example
static int
my_traverse(Noddy *self, visitproc visit, void *arg)
@{
    Py_VISIT(self->foo);
    Py_VISIT(self->bar);
    return 0;
@}
@end example
@end deffn

The @ref{40bf,,tp_clear} handler must be of the @ref{42af,,inquiry} type, or @code{NULL}
if the object is immutable.

@geindex inquiry (C type)
@anchor{c-api/gcsupport c inquiry}@anchor{42af}
@deffn {C Type} typedef  int  (*inquiry)(@ref{375,,PyObject}  *self)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Drop references that may have created reference cycles.  Immutable objects
do not have to define this method since they can never directly create
reference cycles.  Note that the object must still be valid after calling
this method (don’t just call @ref{373,,Py_DECREF()} on a reference).  The
collector will call this method if it detects that this object is involved
in a reference cycle.
@end deffn

@menu
* Controlling the Garbage Collector State:: 

@end menu

@node Controlling the Garbage Collector State,,,Supporting Cyclic Garbage Collection
@anchor{c-api/gcsupport controlling-the-garbage-collector-state}@anchor{461d}
@subsubsection Controlling the Garbage Collector State


The C-API provides the following functions for controlling
garbage collection runs.

@geindex PyGC_Collect (C function)
@anchor{c-api/gcsupport c PyGC_Collect}@anchor{49a}
@deffn {C Function} Py_ssize_t  PyGC_Collect (void)
@emph{ Part of the @ref{35d,,Stable ABI}.}
Perform a full garbage collection, if the garbage collector is enabled.
(Note that @ref{54b,,gc.collect()} runs it unconditionally.)

Returns the number of collected + unreachable objects which cannot
be collected.
If the garbage collector is disabled or already collecting,
returns @code{0} immediately.
Errors during garbage collection are passed to @ref{18d,,sys.unraisablehook}.
This function does not raise exceptions.
@end deffn

@geindex PyGC_Enable (C function)
@anchor{c-api/gcsupport c PyGC_Enable}@anchor{37b}
@deffn {C Function} int  PyGC_Enable (void)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Enable the garbage collector: similar to @ref{3bfa,,gc.enable()}.
Returns the previous state, 0 for disabled and 1 for enabled.


New in version 3.10.

@end deffn

@geindex PyGC_Disable (C function)
@anchor{c-api/gcsupport c PyGC_Disable}@anchor{37c}
@deffn {C Function} int  PyGC_Disable (void)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Disable the garbage collector: similar to @ref{3bfb,,gc.disable()}.
Returns the previous state, 0 for disabled and 1 for enabled.


New in version 3.10.

@end deffn

@geindex PyGC_IsEnabled (C function)
@anchor{c-api/gcsupport c PyGC_IsEnabled}@anchor{37d}
@deffn {C Function} int  PyGC_IsEnabled (void)
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.10.}
Query the state of the garbage collector: similar to @ref{3bfc,,gc.isenabled()}.
Returns the current state, 0 for disabled and 1 for enabled.


New in version 3.10.

@end deffn

@node API and ABI Versioning,,Object Implementation Support,Python/C API Reference Manual
@anchor{c-api/apiabiversion doc}@anchor{461e}@anchor{c-api/apiabiversion api-and-abi-versioning}@anchor{461f}@anchor{c-api/apiabiversion apiabiversion}@anchor{3b73}
@section API and ABI Versioning


CPython exposes its version number in the following macros.
Note that these correspond to the version code is @strong{built} with,
not necessarily the version used at @strong{run time}.

See @ref{35d,,C API Stability} for a discussion of API and ABI stability across versions.

@geindex PY_MAJOR_VERSION (C macro)
@anchor{c-api/apiabiversion c PY_MAJOR_VERSION}@anchor{4620}
@deffn {C Macro} PY_MAJOR_VERSION

The @code{3} in @code{3.4.1a2}.
@end deffn

@geindex PY_MINOR_VERSION (C macro)
@anchor{c-api/apiabiversion c PY_MINOR_VERSION}@anchor{4621}
@deffn {C Macro} PY_MINOR_VERSION

The @code{4} in @code{3.4.1a2}.
@end deffn

@geindex PY_MICRO_VERSION (C macro)
@anchor{c-api/apiabiversion c PY_MICRO_VERSION}@anchor{4622}
@deffn {C Macro} PY_MICRO_VERSION

The @code{1} in @code{3.4.1a2}.
@end deffn

@geindex PY_RELEASE_LEVEL (C macro)
@anchor{c-api/apiabiversion c PY_RELEASE_LEVEL}@anchor{4623}
@deffn {C Macro} PY_RELEASE_LEVEL

The @code{a} in @code{3.4.1a2}.
This can be @code{0xA} for alpha, @code{0xB} for beta, @code{0xC} for release
candidate or @code{0xF} for final.
@end deffn

@geindex PY_RELEASE_SERIAL (C macro)
@anchor{c-api/apiabiversion c PY_RELEASE_SERIAL}@anchor{4624}
@deffn {C Macro} PY_RELEASE_SERIAL

The @code{2} in @code{3.4.1a2}. Zero for final releases.
@end deffn

@geindex PY_VERSION_HEX (C macro)
@anchor{c-api/apiabiversion c PY_VERSION_HEX}@anchor{1da}
@deffn {C Macro} PY_VERSION_HEX

The Python version number encoded in a single integer.

The underlying version information can be found by treating it as a 32 bit
number in the following manner:


@multitable {xxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Bytes

@tab

Bits (big endian order)

@tab

Meaning

@tab

Value for @code{3.4.1a2}

@item

1

@tab

1-8

@tab

@code{PY_MAJOR_VERSION}

@tab

@code{0x03}

@item

2

@tab

9-16

@tab

@code{PY_MINOR_VERSION}

@tab

@code{0x04}

@item

3

@tab

17-24

@tab

@code{PY_MICRO_VERSION}

@tab

@code{0x01}

@item

4

@tab

25-28

@tab

@code{PY_RELEASE_LEVEL}

@tab

@code{0xA}

@item

29-32

@tab

@code{PY_RELEASE_SERIAL}

@tab

@code{0x2}

@end multitable


Thus @code{3.4.1a2} is hexversion @code{0x030401a2} and @code{3.10.0} is
hexversion @code{0x030a00f0}.

This version is also available via the symbol @code{Py_Version}.
@end deffn

@geindex Py_Version (C var)
@anchor{c-api/apiabiversion c Py_Version}@anchor{1d9}
@deffn {C Variable} const  unsigned  long  Py_Version
@emph{ Part of the @ref{35d,,Stable ABI} since version 3.11.}
The Python runtime version number encoded in a single constant integer, with
the same format as the @ref{1da,,PY_VERSION_HEX} macro.
This contains the Python version used at run time.


New in version 3.11.

@end deffn

All the given macros are defined in Include/patchlevel.h@footnote{https://github.com/python/cpython/tree/main/Include/patchlevel.h}.

@node Distributing Python Modules,Installing Python Modules,Python/C API Reference Manual,Top
@anchor{distributing/index doc}@anchor{4625}@anchor{distributing/index distributing-index}@anchor{a6a}@anchor{distributing/index distributing-python-modules}@anchor{4626}
@chapter Distributing Python Modules


@*Email: 
@email{distutils-sig@@python.org}


As a popular open source development project, Python has an active
supporting community of contributors and users that also make their software
available for other Python developers to use under open source license terms.

This allows Python users to share and collaborate effectively, benefiting
from the solutions others have already created to common (and sometimes
even rare!) problems, as well as potentially contributing their own
solutions to the common pool.

This guide covers the distribution part of the process. For a guide to
installing other Python projects, refer to the
@ref{a69,,installation guide}.

@cartouche
@quotation Note 
For corporate and other institutional users, be aware that many
organisations have their own policies around using and contributing to
open source software. Please take such policies into account when making
use of the distribution and installation tools provided with Python.
@end quotation
@end cartouche

@menu
* Key terms:: 
* Open source licensing and collaboration:: 
* Installing the tools:: 
* Reading the Python Packaging User Guide:: 
* How do I…?:: 

@end menu

@node Key terms,Open source licensing and collaboration,,Distributing Python Modules
@anchor{distributing/index key-terms}@anchor{4627}
@section Key terms



@itemize *

@item 
the Python Package Index@footnote{https://pypi.org} is a public
repository of open source licensed packages made available for use by
other Python users

@item 
the Python Packaging Authority@footnote{https://www.pypa.io/} are the group of
developers and documentation authors responsible for the maintenance and
evolution of the standard packaging tools and the associated metadata and
file format standards. They maintain a variety of tools, documentation
and issue trackers on both GitHub@footnote{https://github.com/pypa} and
Bitbucket@footnote{https://bitbucket.org/pypa/}.

@item 
@ref{37,,distutils} is the original build and distribution system first added
to the Python standard library in 1998. While direct use of @ref{37,,distutils}
is being phased out, it still laid the foundation for the current packaging
and distribution infrastructure, and it not only remains part of the
standard library, but its name lives on in other ways (such as the name
of the mailing list used to coordinate Python packaging standards
development).

@item 
setuptools@footnote{https://setuptools.readthedocs.io/en/latest/} is a (largely) drop-in replacement for @ref{37,,distutils} first
published in 2004. Its most notable addition over the unmodified
@ref{37,,distutils} tools was the ability to declare dependencies on other
packages. It is currently recommended as a more regularly updated
alternative to @ref{37,,distutils} that offers consistent support for more
recent packaging standards across a wide range of Python versions.

@item 
wheel@footnote{https://wheel.readthedocs.io/} (in this context) is a project that adds the @code{bdist_wheel}
command to @ref{37,,distutils}/setuptools@footnote{https://setuptools.readthedocs.io/en/latest/}. This produces a cross platform
binary packaging format (called “wheels” or “wheel files” and defined in
@geindex Python Enhancement Proposals; PEP 427
PEP 427@footnote{https://www.python.org/dev/peps/pep-0427}) that allows Python libraries, even those including binary
extensions, to be installed on a system without needing to be built
locally.
@end itemize

@node Open source licensing and collaboration,Installing the tools,Key terms,Distributing Python Modules
@anchor{distributing/index open-source-licensing-and-collaboration}@anchor{4628}@anchor{distributing/index wheel}@anchor{4629}
@section Open source licensing and collaboration


In most parts of the world, software is automatically covered by copyright.
This means that other developers require explicit permission to copy, use,
modify and redistribute the software.

Open source licensing is a way of explicitly granting such permission in a
relatively consistent way, allowing developers to share and collaborate
efficiently by making common solutions to various problems freely available.
This leaves many developers free to spend more time focusing on the problems
that are relatively unique to their specific situation.

The distribution tools provided with Python are designed to make it
reasonably straightforward for developers to make their own contributions
back to that common pool of software if they choose to do so.

The same distribution tools can also be used to distribute software within
an organisation, regardless of whether that software is published as open
source software or not.

@node Installing the tools,Reading the Python Packaging User Guide,Open source licensing and collaboration,Distributing Python Modules
@anchor{distributing/index installing-the-tools}@anchor{462a}
@section Installing the tools


The standard library does not include build tools that support modern
Python packaging standards, as the core development team has found that it
is important to have standard tools that work consistently, even on older
versions of Python.

The currently recommended build and distribution tools can be installed
by invoking the @code{pip} module at the command line:

@example
python -m pip install setuptools wheel twine
@end example

@cartouche
@quotation Note 
For POSIX users (including macOS and Linux users), these instructions
assume the use of a @ref{15d5,,virtual environment}.

For Windows users, these instructions assume that the option to
adjust the system PATH environment variable was selected when installing
Python.
@end quotation
@end cartouche

The Python Packaging User Guide includes more details on the currently recommended tools@footnote{https://packaging.python.org/guides/tool-recommendations/#packaging-tool-recommendations}.

@geindex Python Package Index (PyPI)
@geindex PyPI; (see Python Package Index (PyPI))

@node Reading the Python Packaging User Guide,How do I…?,Installing the tools,Distributing Python Modules
@anchor{distributing/index publishing-python-packages}@anchor{462b}@anchor{distributing/index reading-the-python-packaging-user-guide}@anchor{462c}
@section Reading the Python Packaging User Guide


The Python Packaging User Guide covers the various key steps and elements
involved in creating and publishing a project:


@itemize *

@item 
Project structure@footnote{https://packaging.python.org/tutorials/packaging-projects/#packaging-python-projects}

@item 
Building and packaging the project@footnote{https://packaging.python.org/tutorials/packaging-projects/#creating-the-package-files}

@item 
Uploading the project to the Python Package Index@footnote{https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives}

@item 
The .pypirc file@footnote{https://packaging.python.org/specifications/pypirc/}
@end itemize

@node How do I…?,,Reading the Python Packaging User Guide,Distributing Python Modules
@anchor{distributing/index how-do-i}@anchor{462d}@anchor{distributing/index the-pypirc-file}@anchor{462e}
@section How do I…?


These are quick answers or links for some common tasks.

@menu
* … choose a name for my project?:: 
* … create and distribute binary extensions?:: 

@end menu

@node … choose a name for my project?,… create and distribute binary extensions?,,How do I…?
@anchor{distributing/index choose-a-name-for-my-project}@anchor{462f}
@subsection … choose a name for my project?


This isn’t an easy topic, but here are a few tips:


@itemize *

@item 
check the Python Package Index to see if the name is already in use

@item 
check popular hosting sites like GitHub, Bitbucket, etc to see if there
is already a project with that name

@item 
check what comes up in a web search for the name you’re considering

@item 
avoid particularly common words, especially ones with multiple meanings,
as they can make it difficult for users to find your software when
searching for it
@end itemize

@node … create and distribute binary extensions?,,… choose a name for my project?,How do I…?
@anchor{distributing/index create-and-distribute-binary-extensions}@anchor{4630}
@subsection … create and distribute binary extensions?


This is actually quite a complex topic, with a variety of alternatives
available depending on exactly what you’re aiming to achieve. See the
Python Packaging User Guide for more information and recommendations.


@subsubheading See also


Python Packaging User Guide: Binary Extensions@footnote{https://packaging.python.org/guides/packaging-binary-extensions/}


@c other topics:
@c 
@c Once the Development & Deployment part of PPUG is fleshed out, some of
@c those sections should be linked from new questions here (most notably,
@c we should have a question about avoiding depending on PyPI that links to
@c https://packaging.python.org/en/latest/mirrors/)

@node Installing Python Modules,Python HOWTOs,Distributing Python Modules,Top
@anchor{installing/index doc}@anchor{4631}@anchor{installing/index installing-index}@anchor{a69}@anchor{installing/index installing-python-modules}@anchor{4632}
@chapter Installing Python Modules


@*Email: 
@email{distutils-sig@@python.org}


As a popular open source development project, Python has an active
supporting community of contributors and users that also make their software
available for other Python developers to use under open source license terms.

This allows Python users to share and collaborate effectively, benefiting
from the solutions others have already created to common (and sometimes
even rare!) problems, as well as potentially contributing their own
solutions to the common pool.

This guide covers the installation part of the process. For a guide to
creating and sharing your own Python projects, refer to the
@ref{a6a,,distribution guide}.

@cartouche
@quotation Note 
For corporate and other institutional users, be aware that many
organisations have their own policies around using and contributing to
open source software. Please take such policies into account when making
use of the distribution and installation tools provided with Python.
@end quotation
@end cartouche

@menu
* Key terms: Key terms<2>. 
* Basic usage:: 
* How do I …?:: 
* Common installation issues:: 

@end menu

@node Key terms<2>,Basic usage,,Installing Python Modules
@anchor{installing/index key-terms}@anchor{4633}
@section Key terms



@itemize *

@item 
@code{pip} is the preferred installer program. Starting with Python 3.4, it
is included by default with the Python binary installers.

@item 
A @emph{virtual environment} is a semi-isolated Python environment that allows
packages to be installed for use by a particular application, rather than
being installed system wide.

@item 
@code{venv} is the standard tool for creating virtual environments, and has
been part of Python since Python 3.3. Starting with Python 3.4, it
defaults to installing @code{pip} into all created virtual environments.

@item 
@code{virtualenv} is a third party alternative (and predecessor) to
@code{venv}. It allows virtual environments to be used on versions of
Python prior to 3.4, which either don’t provide @code{venv} at all, or
aren’t able to automatically install @code{pip} into created environments.

@item 
The Python Package Index@footnote{https://pypi.org} is a public
repository of open source licensed packages made available for use by
other Python users.

@item 
the Python Packaging Authority@footnote{https://www.pypa.io/} is the group of
developers and documentation authors responsible for the maintenance and
evolution of the standard packaging tools and the associated metadata and
file format standards. They maintain a variety of tools, documentation,
and issue trackers on both GitHub@footnote{https://github.com/pypa} and
Bitbucket@footnote{https://bitbucket.org/pypa/}.

@item 
@code{distutils} is the original build and distribution system first added to
the Python standard library in 1998. While direct use of @code{distutils} is
being phased out, it still laid the foundation for the current packaging
and distribution infrastructure, and it not only remains part of the
standard library, but its name lives on in other ways (such as the name
of the mailing list used to coordinate Python packaging standards
development).
@end itemize


Changed in version 3.5: The use of @code{venv} is now recommended for creating virtual environments.



@subsubheading See also


Python Packaging User Guide: Creating and using virtual environments@footnote{https://packaging.python.org/installing/#creating-virtual-environments}


@node Basic usage,How do I …?,Key terms<2>,Installing Python Modules
@anchor{installing/index basic-usage}@anchor{4634}
@section Basic usage


The standard packaging tools are all designed to be used from the command
line.

The following command will install the latest version of a module and its
dependencies from the Python Package Index:

@example
python -m pip install SomePackage
@end example

@cartouche
@quotation Note 
For POSIX users (including macOS and Linux users), the examples in
this guide assume the use of a @ref{15d5,,virtual environment}.

For Windows users, the examples in this guide assume that the option to
adjust the system PATH environment variable was selected when installing
Python.
@end quotation
@end cartouche

It’s also possible to specify an exact or minimum version directly on the
command line. When using comparator operators such as @code{>}, @code{<} or some other
special character which get interpreted by shell, the package name and the
version should be enclosed within double quotes:

@example
python -m pip install SomePackage==1.0.4    # specific version
python -m pip install "SomePackage>=1.0.4"  # minimum version
@end example

Normally, if a suitable module is already installed, attempting to install
it again will have no effect. Upgrading existing modules must be requested
explicitly:

@example
python -m pip install --upgrade SomePackage
@end example

More information and resources regarding @code{pip} and its capabilities can be
found in the Python Packaging User Guide@footnote{https://packaging.python.org}.

Creation of virtual environments is done through the @ref{12c,,venv} module.
Installing packages into an active virtual environment uses the commands shown
above.


@subsubheading See also


Python Packaging User Guide: Installing Python Distribution Packages@footnote{https://packaging.python.org/installing/}


@node How do I …?,Common installation issues,Basic usage,Installing Python Modules
@anchor{installing/index how-do-i}@anchor{4635}
@section How do I …?


These are quick answers or links for some common tasks.

@menu
* … install pip in versions of Python prior to Python 3.4?: … install pip in versions of Python prior to Python 3 4?. 
* … install packages just for the current user?:: 
* … install scientific Python packages?:: 
* … work with multiple versions of Python installed in parallel?:: 

@end menu

@node … install pip in versions of Python prior to Python 3 4?,… install packages just for the current user?,,How do I …?
@anchor{installing/index install-pip-in-versions-of-python-prior-to-python-3-4}@anchor{4636}
@subsection … install @code{pip} in versions of Python prior to Python 3.4?


Python only started bundling @code{pip} with Python 3.4. For earlier versions,
@code{pip} needs to be “bootstrapped” as described in the Python Packaging
User Guide.


@subsubheading See also


Python Packaging User Guide: Requirements for Installing Packages@footnote{https://packaging.python.org/installing/#requirements-for-installing-packages}


@c installing-per-user-installation:

@node … install packages just for the current user?,… install scientific Python packages?,… install pip in versions of Python prior to Python 3 4?,How do I …?
@anchor{installing/index install-packages-just-for-the-current-user}@anchor{4637}
@subsection … install packages just for the current user?


Passing the @code{--user} option to @code{python -m pip install} will install a
package just for the current user, rather than for all users of the system.

@node … install scientific Python packages?,… work with multiple versions of Python installed in parallel?,… install packages just for the current user?,How do I …?
@anchor{installing/index install-scientific-python-packages}@anchor{4638}
@subsection … install scientific Python packages?


A number of scientific Python packages have complex binary dependencies, and
aren’t currently easy to install using @code{pip} directly. At this point in
time, it will often be easier for users to install these packages by
other means@footnote{https://packaging.python.org/science/}
rather than attempting to install them with @code{pip}.


@subsubheading See also


Python Packaging User Guide: Installing Scientific Packages@footnote{https://packaging.python.org/science/}


@node … work with multiple versions of Python installed in parallel?,,… install scientific Python packages?,How do I …?
@anchor{installing/index work-with-multiple-versions-of-python-installed-in-parallel}@anchor{4639}
@subsection … work with multiple versions of Python installed in parallel?


On Linux, macOS, and other POSIX systems, use the versioned Python commands
in combination with the @code{-m} switch to run the appropriate copy of
@code{pip}:

@example
python2   -m pip install SomePackage  # default Python 2
python2.7 -m pip install SomePackage  # specifically Python 2.7
python3   -m pip install SomePackage  # default Python 3
python3.4 -m pip install SomePackage  # specifically Python 3.4
@end example

Appropriately versioned @code{pip} commands may also be available.

On Windows, use the @code{py} Python launcher in combination with the @code{-m}
switch:

@example
py -2   -m pip install SomePackage  # default Python 2
py -2.7 -m pip install SomePackage  # specifically Python 2.7
py -3   -m pip install SomePackage  # default Python 3
py -3.4 -m pip install SomePackage  # specifically Python 3.4
@end example

@c other questions:
@c 
@c Once the Development & Deployment part of PPUG is fleshed out, some of
@c those sections should be linked from new questions here (most notably,
@c we should have a question about avoiding depending on PyPI that links to
@c https://packaging.python.org/en/latest/mirrors/)

@node Common installation issues,,How do I …?,Installing Python Modules
@anchor{installing/index common-installation-issues}@anchor{463a}
@section Common installation issues


@menu
* Installing into the system Python on Linux:: 
* Pip not installed:: 
* Installing binary extensions:: 

@end menu

@node Installing into the system Python on Linux,Pip not installed,,Common installation issues
@anchor{installing/index installing-into-the-system-python-on-linux}@anchor{463b}
@subsection Installing into the system Python on Linux


On Linux systems, a Python installation will typically be included as part
of the distribution. Installing into this Python installation requires
root access to the system, and may interfere with the operation of the
system package manager and other components of the system if a component
is unexpectedly upgraded using @code{pip}.

On such systems, it is often better to use a virtual environment or a
per-user installation when installing packages with @code{pip}.

@node Pip not installed,Installing binary extensions,Installing into the system Python on Linux,Common installation issues
@anchor{installing/index pip-not-installed}@anchor{463c}
@subsection Pip not installed


It is possible that @code{pip} does not get installed by default. One potential fix is:

@example
python -m ensurepip --default-pip
@end example

There are also additional resources for installing pip.@footnote{https://packaging.python.org/tutorials/installing-packages/#install-pip-setuptools-and-wheel}

@node Installing binary extensions,,Pip not installed,Common installation issues
@anchor{installing/index installing-binary-extensions}@anchor{463d}
@subsection Installing binary extensions


Python has typically relied heavily on source based distribution, with end
users being expected to compile extension modules from source as part of
the installation process.

With the introduction of support for the binary @code{wheel} format, and the
ability to publish wheels for at least Windows and macOS through the
Python Package Index, this problem is expected to diminish over time,
as users are more regularly able to install pre-built extensions rather
than needing to build them themselves.

Some of the solutions for installing scientific software@footnote{https://packaging.python.org/science/}
that are not yet available as pre-built @code{wheel} files may also help with
obtaining other binary extensions without needing to build them locally.


@subsubheading See also


Python Packaging User Guide: Binary Extensions@footnote{https://packaging.python.org/extensions/}


@node Python HOWTOs,Python Frequently Asked Questions,Installing Python Modules,Top
@anchor{howto/index doc}@anchor{463e}@anchor{howto/index python-howtos}@anchor{463f}
@chapter Python HOWTOs


Python HOWTOs are documents that cover a single, specific topic,
and attempt to cover it fairly completely. Modelled on the Linux
Documentation Project’s HOWTO collection, this collection is an
effort to foster documentation that’s more detailed than the
Python Library Reference.

Currently, the HOWTOs are:

@menu
* Porting Python 2 Code to Python 3:: 
* Porting Extension Modules to Python 3:: 
* Curses Programming with Python:: 
* Descriptor HowTo Guide:: 
* Enum HOWTO:: 
* Functional Programming HOWTO:: 
* Logging HOWTO:: 
* Logging Cookbook:: 
* Regular Expression HOWTO:: 
* Socket Programming HOWTO:: 
* Sorting HOW TO:: 
* Unicode HOWTO:: 
* HOWTO Fetch Internet Resources Using The urllib Package:: 
* Argparse Tutorial:: 
* An introduction to the ipaddress module:: 
* Argument Clinic How-To:: 
* Instrumenting CPython with DTrace and SystemTap:: 
* Annotations Best Practices:: 

@end menu

@node Porting Python 2 Code to Python 3,Porting Extension Modules to Python 3,,Python HOWTOs
@anchor{howto/pyporting doc}@anchor{4640}@anchor{howto/pyporting porting-python-2-code-to-python-3}@anchor{4641}@anchor{howto/pyporting pyporting-howto}@anchor{e84}
@section Porting Python 2 Code to Python 3


@*author: 
Brett Cannon


@subsubheading Abstract



With Python 3 being the future of Python while Python 2 is still in active
use, it is good to have your project available for both major releases of
Python. This guide is meant to help you figure out how best to support both
Python 2 & 3 simultaneously.

If you are looking to port an extension module instead of pure Python code,
please see @ref{e7e,,Porting Extension Modules to Python 3}.

If you would like to read one core Python developer’s take on why Python 3
came into existence, you can read Nick Coghlan’s Python 3 Q & A@footnote{https://ncoghlan-devs-python-notes.readthedocs.io/en/latest/python3/questions_and_answers.html} or
Brett Cannon’s Why Python 3 exists@footnote{https://snarky.ca/why-python-3-exists}.

For help with porting, you can view the archived python-porting@footnote{https://mail.python.org/pipermail/python-porting/} mailing list.

@menu
* The Short Explanation:: 
* Details:: 

@end menu

@node The Short Explanation,Details,,Porting Python 2 Code to Python 3
@anchor{howto/pyporting the-short-explanation}@anchor{4642}
@subsection The Short Explanation


To make your project be single-source Python 2/3 compatible, the basic steps
are:


@enumerate 

@item 
Only worry about supporting Python 2.7

@item 
Make sure you have good test coverage (coverage.py@footnote{https://pypi.org/project/coverage} can help;
@code{python -m pip install coverage})

@item 
Learn the differences between Python 2 & 3

@item 
Use Futurize@footnote{http://python-future.org/automatic_conversion.html} (or Modernize@footnote{https://python-modernize.readthedocs.io/}) to update your code (e.g. @code{python -m pip install future})

@item 
Use Pylint@footnote{https://pypi.org/project/pylint} to help make sure you don’t regress on your Python 3 support
(@code{python -m pip install pylint})

@item 
Use caniusepython3@footnote{https://pypi.org/project/caniusepython3} to find out which of your dependencies are blocking your
use of Python 3 (@code{python -m pip install caniusepython3})

@item 
Once your dependencies are no longer blocking you, use continuous integration
to make sure you stay compatible with Python 2 & 3 (tox@footnote{https://pypi.org/project/tox} can help test
against multiple versions of Python; @code{python -m pip install tox})

@item 
Consider using optional static type checking to make sure your type usage
works in both Python 2 & 3 (e.g. use mypy@footnote{http://mypy-lang.org/} to check your typing under both
Python 2 & Python 3; @code{python -m pip install mypy}).
@end enumerate

@cartouche
@quotation Note 
Note: Using @code{python -m pip install} guarantees that the @code{pip} you invoke
is the one installed for the Python currently in use, whether it be
a system-wide @code{pip} or one installed within a
@ref{15d2,,virtual environment}.
@end quotation
@end cartouche

@node Details,,The Short Explanation,Porting Python 2 Code to Python 3
@anchor{howto/pyporting details}@anchor{4643}
@subsection Details


A key point about supporting Python 2 & 3 simultaneously is that you can start
@strong{today}! Even if your dependencies are not supporting Python 3 yet that does
not mean you can’t modernize your code @strong{now} to support Python 3. Most changes
required to support Python 3 lead to cleaner code using newer practices even in
Python 2 code.

Another key point is that modernizing your Python 2 code to also support
Python 3 is largely automated for you. While you might have to make some API
decisions thanks to Python 3 clarifying text data versus binary data, the
lower-level work is now mostly done for you and thus can at least benefit from
the automated changes immediately.

Keep those key points in mind while you read on about the details of porting
your code to support Python 2 & 3 simultaneously.

@menu
* Drop support for Python 2.6 and older: Drop support for Python 2 6 and older. 
* Make sure you specify the proper version support in your setup.py file: Make sure you specify the proper version support in your setup py file. 
* Have good test coverage:: 
* Learn the differences between Python 2 & 3:: 
* Update your code:: 
* Prevent compatibility regressions:: 
* Check which dependencies block your transition:: 
* Update your setup.py file to denote Python 3 compatibility: Update your setup py file to denote Python 3 compatibility. 
* Use continuous integration to stay compatible:: 
* Consider using optional static type checking:: 

@end menu

@node Drop support for Python 2 6 and older,Make sure you specify the proper version support in your setup py file,,Details
@anchor{howto/pyporting drop-support-for-python-2-6-and-older}@anchor{4644}
@subsubsection Drop support for Python 2.6 and older


While you can make Python 2.5 work with Python 3, it is @strong{much} easier if you
only have to work with Python 2.7. If dropping Python 2.5 is not an
option then the six@footnote{https://pypi.org/project/six} project can help you support Python 2.5 & 3 simultaneously
(@code{python -m pip install six}). Do realize, though, that nearly all the projects listed
in this HOWTO will not be available to you.

If you are able to skip Python 2.5 and older, then the required changes
to your code should continue to look and feel like idiomatic Python code. At
worst you will have to use a function instead of a method in some instances or
have to import a function instead of using a built-in one, but otherwise the
overall transformation should not feel foreign to you.

But you should aim for only supporting Python 2.7. Python 2.6 is no longer
freely supported and thus is not receiving bugfixes. This means @strong{you} will have
to work around any issues you come across with Python 2.6. There are also some
tools mentioned in this HOWTO which do not support Python 2.6 (e.g., Pylint@footnote{https://pypi.org/project/pylint}),
and this will become more commonplace as time goes on. It will simply be easier
for you if you only support the versions of Python that you have to support.

@node Make sure you specify the proper version support in your setup py file,Have good test coverage,Drop support for Python 2 6 and older,Details
@anchor{howto/pyporting make-sure-you-specify-the-proper-version-support-in-your-setup-py-file}@anchor{4645}
@subsubsection Make sure you specify the proper version support in your @code{setup.py} file


In your @code{setup.py} file you should have the proper trove classifier@footnote{https://pypi.org/classifiers}
specifying what versions of Python you support. As your project does not support
Python 3 yet you should at least have
@code{Programming Language :: Python :: 2 :: Only} specified. Ideally you should
also specify each major/minor version of Python that you do support, e.g.
@code{Programming Language :: Python :: 2.7}.

@node Have good test coverage,Learn the differences between Python 2 & 3,Make sure you specify the proper version support in your setup py file,Details
@anchor{howto/pyporting have-good-test-coverage}@anchor{4646}
@subsubsection Have good test coverage


Once you have your code supporting the oldest version of Python 2 you want it
to, you will want to make sure your test suite has good coverage. A good rule of
thumb is that if you want to be confident enough in your test suite that any
failures that appear after having tools rewrite your code are actual bugs in the
tools and not in your code. If you want a number to aim for, try to get over 80%
coverage (and don’t feel bad if you find it hard to get better than 90%
coverage). If you don’t already have a tool to measure test coverage then
coverage.py@footnote{https://pypi.org/project/coverage} is recommended.

@node Learn the differences between Python 2 & 3,Update your code,Have good test coverage,Details
@anchor{howto/pyporting learn-the-differences-between-python-2-3}@anchor{4647}
@subsubsection Learn the differences between Python 2 & 3


Once you have your code well-tested you are ready to begin porting your code to
Python 3! But to fully understand how your code is going to change and what
you want to look out for while you code, you will want to learn what changes
Python 3 makes in terms of Python 2. Typically the two best ways of doing that
is reading the @ref{151,,“What’s New”} doc for each release of Python 3 and the
Porting to Python 3@footnote{http://python3porting.com/} book (which is free online). There is also a handy
cheat sheet@footnote{http://python-future.org/compatible_idioms.html} from the Python-Future project.

@node Update your code,Prevent compatibility regressions,Learn the differences between Python 2 & 3,Details
@anchor{howto/pyporting update-your-code}@anchor{4648}
@subsubsection Update your code


Once you feel like you know what is different in Python 3 compared to Python 2,
it’s time to update your code! You have a choice between two tools in porting
your code automatically: Futurize@footnote{http://python-future.org/automatic_conversion.html} and Modernize@footnote{https://python-modernize.readthedocs.io/}. Which tool you choose will
depend on how much like Python 3 you want your code to be. Futurize@footnote{http://python-future.org/automatic_conversion.html} does its
best to make Python 3 idioms and practices exist in Python 2, e.g. backporting
the @code{bytes} type from Python 3 so that you have semantic parity between the
major versions of Python. Modernize@footnote{https://python-modernize.readthedocs.io/},
on the other hand, is more conservative and targets a Python 2/3 subset of
Python, directly relying on six@footnote{https://pypi.org/project/six} to help provide compatibility. As Python 3 is
the future, it might be best to consider Futurize to begin adjusting to any new
practices that Python 3 introduces which you are not accustomed to yet.

Regardless of which tool you choose, they will update your code to run under
Python 3 while staying compatible with the version of Python 2 you started with.
Depending on how conservative you want to be, you may want to run the tool over
your test suite first and visually inspect the diff to make sure the
transformation is accurate. After you have transformed your test suite and
verified that all the tests still pass as expected, then you can transform your
application code knowing that any tests which fail is a translation failure.

Unfortunately the tools can’t automate everything to make your code work under
Python 3 and so there are a handful of things you will need to update manually
to get full Python 3 support (which of these steps are necessary vary between
the tools). Read the documentation for the tool you choose to use to see what it
fixes by default and what it can do optionally to know what will (not) be fixed
for you and what you may have to fix on your own (e.g. using @code{io.open()} over
the built-in @code{open()} function is off by default in Modernize). Luckily,
though, there are only a couple of things to watch out for which can be
considered large issues that may be hard to debug if not watched for.

@menu
* Division:: 
* Text versus binary data:: 
* Use feature detection instead of version detection:: 

@end menu

@node Division,Text versus binary data,,Update your code
@anchor{howto/pyporting division}@anchor{4649}
@subsubsection Division


In Python 3, @code{5 / 2 == 2.5} and not @code{2}; all division between @code{int} values
result in a @code{float}. This change has actually been planned since Python 2.2
which was released in 2002. Since then users have been encouraged to add
@code{from __future__ import division} to any and all files which use the @code{/} and
@code{//} operators or to be running the interpreter with the @code{-Q} flag. If you
have not been doing this then you will need to go through your code and do two
things:


@enumerate 

@item 
Add @code{from __future__ import division} to your files

@item 
Update any division operator as necessary to either use @code{//} to use floor
division or continue using @code{/} and expect a float
@end enumerate

The reason that @code{/} isn’t simply translated to @code{//} automatically is that if
an object defines a @code{__truediv__} method but not @code{__floordiv__} then your
code would begin to fail (e.g. a user-defined class that uses @code{/} to
signify some operation but not @code{//} for the same thing or at all).

@node Text versus binary data,Use feature detection instead of version detection,Division,Update your code
@anchor{howto/pyporting text-versus-binary-data}@anchor{464a}
@subsubsection Text versus binary data


In Python 2 you could use the @code{str} type for both text and binary data.
Unfortunately this confluence of two different concepts could lead to brittle
code which sometimes worked for either kind of data, sometimes not. It also
could lead to confusing APIs if people didn’t explicitly state that something
that accepted @code{str} accepted either text or binary data instead of one
specific type. This complicated the situation especially for anyone supporting
multiple languages as APIs wouldn’t bother explicitly supporting @code{unicode}
when they claimed text data support.

To make the distinction between text and binary data clearer and more
pronounced, Python 3 did what most languages created in the age of the internet
have done and made text and binary data distinct types that cannot blindly be
mixed together (Python predates widespread access to the internet). For any code
that deals only with text or only binary data, this separation doesn’t pose an
issue. But for code that has to deal with both, it does mean you might have to
now care about when you are using text compared to binary data, which is why
this cannot be entirely automated.

To start, you will need to decide which APIs take text and which take binary
(it is @strong{highly} recommended you don’t design APIs that can take both due to
the difficulty of keeping the code working; as stated earlier it is difficult to
do well). In Python 2 this means making sure the APIs that take text can work
with @code{unicode} and those that work with binary data work with the
@code{bytes} type from Python 3 (which is a subset of @code{str} in Python 2 and acts
as an alias for @code{bytes} type in Python 2). Usually the biggest issue is
realizing which methods exist on which types in Python 2 & 3 simultaneously
(for text that’s @code{unicode} in Python 2 and @code{str} in Python 3, for binary
that’s @code{str}/@code{bytes} in Python 2 and @code{bytes} in Python 3). The following
table lists the @strong{unique} methods of each data type across Python 2 & 3
(e.g., the @code{decode()} method is usable on the equivalent binary data type in
either Python 2 or 3, but it can’t be used by the textual data type consistently
between Python 2 and 3 because @code{str} in Python 3 doesn’t have the method). Do
note that as of Python 3.5 the @code{__mod__} method was added to the bytes type.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxx} 
@item

@strong{Text data}

@tab

@strong{Binary data}

@item



@tab

decode

@item

encode

@tab

@item

format

@tab

@item

isdecimal

@tab

@item

isnumeric

@tab

@end multitable


Making the distinction easier to handle can be accomplished by encoding and
decoding between binary data and text at the edge of your code. This means that
when you receive text in binary data, you should immediately decode it. And if
your code needs to send text as binary data then encode it as late as possible.
This allows your code to work with only text internally and thus eliminates
having to keep track of what type of data you are working with.

The next issue is making sure you know whether the string literals in your code
represent text or binary data. You should add a @code{b} prefix to any
literal that presents binary data. For text you should add a @code{u} prefix to
the text literal. (there is a @ref{0,,__future__} import to force all unspecified
literals to be Unicode, but usage has shown it isn’t as effective as adding a
@code{b} or @code{u} prefix to all literals explicitly)

As part of this dichotomy you also need to be careful about opening files.
Unless you have been working on Windows, there is a chance you have not always
bothered to add the @code{b} mode when opening a binary file (e.g., @code{rb} for
binary reading).  Under Python 3, binary files and text files are clearly
distinct and mutually incompatible; see the @ref{9d,,io} module for details.
Therefore, you @strong{must} make a decision of whether a file will be used for
binary access (allowing binary data to be read and/or written) or textual access
(allowing text data to be read and/or written). You should also use @ref{1c9,,io.open()}
for opening files instead of the built-in @ref{1c8,,open()} function as the @ref{9d,,io}
module is consistent from Python 2 to 3 while the built-in @ref{1c8,,open()} function
is not (in Python 3 it’s actually @ref{1c9,,io.open()}). Do not bother with the
outdated practice of using @ref{1ca,,codecs.open()} as that’s only necessary for
keeping compatibility with Python 2.5.

The constructors of both @code{str} and @code{bytes} have different semantics for the
same arguments between Python 2 & 3. Passing an integer to @code{bytes} in Python 2
will give you the string representation of the integer: @code{bytes(3) == '3'}.
But in Python 3, an integer argument to @code{bytes} will give you a bytes object
as long as the integer specified, filled with null bytes:
@code{bytes(3) == b'\x00\x00\x00'}. A similar worry is necessary when passing a
bytes object to @code{str}. In Python 2 you just get the bytes object back:
@code{str(b'3') == b'3'}. But in Python 3 you get the string representation of the
bytes object: @code{str(b'3') == "b'3'"}.

Finally, the indexing of binary data requires careful handling (slicing does
@strong{not} require any special handling). In Python 2,
@code{b'123'[1] == b'2'} while in Python 3 @code{b'123'[1] == 50}. Because binary data
is simply a collection of binary numbers, Python 3 returns the integer value for
the byte you index on. But in Python 2 because @code{bytes == str}, indexing
returns a one-item slice of bytes. The six@footnote{https://pypi.org/project/six} project has a function
named @code{six.indexbytes()} which will return an integer like in Python 3:
@code{six.indexbytes(b'123', 1)}.

To summarize:


@enumerate 

@item 
Decide which of your APIs take text and which take binary data

@item 
Make sure that your code that works with text also works with @code{unicode} and
code for binary data works with @code{bytes} in Python 2 (see the table above
for what methods you cannot use for each type)

@item 
Mark all binary literals with a @code{b} prefix, textual literals with a @code{u}
prefix

@item 
Decode binary data to text as soon as possible, encode text as binary data as
late as possible

@item 
Open files using @ref{1c9,,io.open()} and make sure to specify the @code{b} mode when
appropriate

@item 
Be careful when indexing into binary data
@end enumerate

@node Use feature detection instead of version detection,,Text versus binary data,Update your code
@anchor{howto/pyporting use-feature-detection-instead-of-version-detection}@anchor{464b}
@subsubsection Use feature detection instead of version detection


Inevitably you will have code that has to choose what to do based on what
version of Python is running. The best way to do this is with feature detection
of whether the version of Python you’re running under supports what you need.
If for some reason that doesn’t work then you should make the version check be
against Python 2 and not Python 3. To help explain this, let’s look at an
example.

Let’s pretend that you need access to a feature of @ref{96,,importlib} that
is available in Python’s standard library since Python 3.3 and available for
Python 2 through importlib2@footnote{https://pypi.org/project/importlib2} on PyPI. You might be tempted to write code to
access e.g. the @ref{97,,importlib.abc} module by doing the following:

@example
import sys

if sys.version_info[0] == 3:
    from importlib import abc
else:
    from importlib2 import abc
@end example

The problem with this code is what happens when Python 4 comes out? It would
be better to treat Python 2 as the exceptional case instead of Python 3 and
assume that future Python versions will be more compatible with Python 3 than
Python 2:

@example
import sys

if sys.version_info[0] > 2:
    from importlib import abc
else:
    from importlib2 import abc
@end example

The best solution, though, is to do no version detection at all and instead rely
on feature detection. That avoids any potential issues of getting the version
detection wrong and helps keep you future-compatible:

@example
try:
    from importlib import abc
except ImportError:
    from importlib2 import abc
@end example

@node Prevent compatibility regressions,Check which dependencies block your transition,Update your code,Details
@anchor{howto/pyporting prevent-compatibility-regressions}@anchor{464c}
@subsubsection Prevent compatibility regressions


Once you have fully translated your code to be compatible with Python 3, you
will want to make sure your code doesn’t regress and stop working under
Python 3. This is especially true if you have a dependency which is blocking you
from actually running under Python 3 at the moment.

To help with staying compatible, any new modules you create should have
at least the following block of code at the top of it:

@example
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
@end example

You can also run Python 2 with the @code{-3} flag to be warned about various
compatibility issues your code triggers during execution. If you turn warnings
into errors with @code{-Werror} then you can make sure that you don’t accidentally
miss a warning.

You can also use the Pylint@footnote{https://pypi.org/project/pylint} project and its @code{--py3k} flag to lint your code
to receive warnings when your code begins to deviate from Python 3
compatibility. This also prevents you from having to run Modernize@footnote{https://python-modernize.readthedocs.io/} or Futurize@footnote{http://python-future.org/automatic_conversion.html}
over your code regularly to catch compatibility regressions. This does require
you only support Python 2.7 and Python 3.4 or newer as that is Pylint’s
minimum Python version support.

@node Check which dependencies block your transition,Update your setup py file to denote Python 3 compatibility,Prevent compatibility regressions,Details
@anchor{howto/pyporting check-which-dependencies-block-your-transition}@anchor{464d}
@subsubsection Check which dependencies block your transition


@strong{After} you have made your code compatible with Python 3 you should begin to
care about whether your dependencies have also been ported. The caniusepython3@footnote{https://pypi.org/project/caniusepython3}
project was created to help you determine which projects
– directly or indirectly – are blocking you from supporting Python 3. There
is both a command-line tool as well as a web interface at
@indicateurl{https://caniusepython3.com}.

The project also provides code which you can integrate into your test suite so
that you will have a failing test when you no longer have dependencies blocking
you from using Python 3. This allows you to avoid having to manually check your
dependencies and to be notified quickly when you can start running on Python 3.

@node Update your setup py file to denote Python 3 compatibility,Use continuous integration to stay compatible,Check which dependencies block your transition,Details
@anchor{howto/pyporting update-your-setup-py-file-to-denote-python-3-compatibility}@anchor{464e}
@subsubsection Update your @code{setup.py} file to denote Python 3 compatibility


Once your code works under Python 3, you should update the classifiers in
your @code{setup.py} to contain @code{Programming Language :: Python :: 3} and to not
specify sole Python 2 support. This will tell anyone using your code that you
support Python 2 @strong{and} 3. Ideally you will also want to add classifiers for
each major/minor version of Python you now support.

@node Use continuous integration to stay compatible,Consider using optional static type checking,Update your setup py file to denote Python 3 compatibility,Details
@anchor{howto/pyporting use-continuous-integration-to-stay-compatible}@anchor{464f}
@subsubsection Use continuous integration to stay compatible


Once you are able to fully run under Python 3 you will want to make sure your
code always works under both Python 2 & 3. Probably the best tool for running
your tests under multiple Python interpreters is tox@footnote{https://pypi.org/project/tox}. You can then integrate
tox with your continuous integration system so that you never accidentally break
Python 2 or 3 support.

You may also want to use the @code{-bb} flag with the Python 3 interpreter to
trigger an exception when you are comparing bytes to strings or bytes to an int
(the latter is available starting in Python 3.5). By default type-differing
comparisons simply return @code{False}, but if you made a mistake in your
separation of text/binary data handling or indexing on bytes you wouldn’t easily
find the mistake. This flag will raise an exception when these kinds of
comparisons occur, making the mistake much easier to track down.

And that’s mostly it! At this point your code base is compatible with both
Python 2 and 3 simultaneously. Your testing will also be set up so that you
don’t accidentally break Python 2 or 3 compatibility regardless of which version
you typically run your tests under while developing.

@node Consider using optional static type checking,,Use continuous integration to stay compatible,Details
@anchor{howto/pyporting consider-using-optional-static-type-checking}@anchor{4650}
@subsubsection Consider using optional static type checking


Another way to help port your code is to use a static type checker like
mypy@footnote{http://mypy-lang.org/} or pytype@footnote{https://github.com/google/pytype} on your code. These tools can be used to analyze your code as
if it’s being run under Python 2, then you can run the tool a second time as if
your code is running under Python 3. By running a static type checker twice like
this you can discover if you’re e.g. misusing binary data type in one version
of Python compared to another. If you add optional type hints to your code you
can also explicitly state whether your APIs use textual or binary data, helping
to make sure everything functions as expected in both versions of Python.

@node Porting Extension Modules to Python 3,Curses Programming with Python,Porting Python 2 Code to Python 3,Python HOWTOs
@anchor{howto/cporting doc}@anchor{4651}@anchor{howto/cporting cporting-howto}@anchor{e7e}@anchor{howto/cporting porting-extension-modules-to-python-3}@anchor{4652}@anchor{howto/cporting why-python-3-exists}@anchor{4653}
@section Porting Extension Modules to Python 3


We recommend the following resources for porting extension modules to Python 3:


@itemize *

@item 
The Migrating C extensions@footnote{http://python3porting.com/cextensions.html} chapter from
@emph{Supporting Python 3: An in-depth guide}, a book on moving from Python 2
to Python 3 in general, guides the reader through porting an extension
module.

@item 
The Porting guide@footnote{https://py3c.readthedocs.io/en/latest/guide.html} from the @emph{py3c} project provides opinionated
suggestions with supporting code.

@item 
The Cython@footnote{http://cython.org/} and CFFI@footnote{https://cffi.readthedocs.io/en/latest/} libraries offer abstractions over
Python’s C API.
Extensions generally need to be re-written to use one of them,
but the library then handles differences between various Python
versions and implementations.
@end itemize

@node Curses Programming with Python,Descriptor HowTo Guide,Porting Extension Modules to Python 3,Python HOWTOs
@anchor{howto/curses doc}@anchor{4654}@anchor{howto/curses cffi}@anchor{4655}@anchor{howto/curses curses-howto}@anchor{25ef}@anchor{howto/curses curses-programming-with-python}@anchor{4656}
@section Curses Programming with Python


@*Author: 
A.M. Kuchling, Eric S. Raymond


@*Release: 
2.04


@subsubheading Abstract



This document describes how to use the @ref{2a,,curses} extension
module to control text-mode displays.

@menu
* What is curses?:: 
* Starting and ending a curses application:: 
* Windows and Pads:: 
* Displaying Text:: 
* User Input:: 
* For More Information:: 

@end menu

@node What is curses?,Starting and ending a curses application,,Curses Programming with Python
@anchor{howto/curses what-is-curses}@anchor{4657}
@subsection What is curses?


The curses library supplies a terminal-independent screen-painting and
keyboard-handling facility for text-based terminals; such terminals
include VT100s, the Linux console, and the simulated terminal provided
by various programs.  Display terminals support various control codes
to perform common operations such as moving the cursor, scrolling the
screen, and erasing areas.  Different terminals use widely differing
codes, and often have their own minor quirks.

In a world of graphical displays, one might ask “why bother”?  It’s
true that character-cell display terminals are an obsolete technology,
but there are niches in which being able to do fancy things with them
are still valuable.  One niche is on small-footprint or embedded
Unixes that don’t run an X server.  Another is tools such as OS
installers and kernel configurators that may have to run before any
graphical support is available.

The curses library provides fairly basic functionality, providing the
programmer with an abstraction of a display containing multiple
non-overlapping windows of text.  The contents of a window can be
changed in various ways—adding text, erasing it, changing its
appearance—and the curses library will figure out what control codes
need to be sent to the terminal to produce the right output.  curses
doesn’t provide many user-interface concepts such as buttons, checkboxes,
or dialogs; if you need such features, consider a user interface library such as
Urwid@footnote{https://pypi.org/project/urwid/}.

The curses library was originally written for BSD Unix; the later System V
versions of Unix from AT&T added many enhancements and new functions. BSD curses
is no longer maintained, having been replaced by ncurses, which is an
open-source implementation of the AT&T interface.  If you’re using an
open-source Unix such as Linux or FreeBSD, your system almost certainly uses
ncurses.  Since most current commercial Unix versions are based on System V
code, all the functions described here will probably be available.  The older
versions of curses carried by some proprietary Unixes may not support
everything, though.

The Windows version of Python doesn’t include the @ref{2a,,curses}
module.  A ported version called UniCurses@footnote{https://pypi.org/project/UniCurses} is available.

@menu
* The Python curses module:: 

@end menu

@node The Python curses module,,,What is curses?
@anchor{howto/curses the-python-curses-module}@anchor{4658}
@subsubsection The Python curses module


The Python module is a fairly simple wrapper over the C functions provided by
curses; if you’re already familiar with curses programming in C, it’s really
easy to transfer that knowledge to Python.  The biggest difference is that the
Python interface makes things simpler by merging different C functions such as
@code{addstr()}, @code{mvaddstr()}, and @code{mvwaddstr()} into a single
@ref{2602,,addstr()} method.  You’ll see this covered in more
detail later.

This HOWTO is an introduction to writing text-mode programs with curses
and Python. It doesn’t attempt to be a complete guide to the curses API; for
that, see the Python library guide’s section on ncurses, and the C manual pages
for ncurses.  It will, however, give you the basic ideas.

@node Starting and ending a curses application,Windows and Pads,What is curses?,Curses Programming with Python
@anchor{howto/curses starting-and-ending-a-curses-application}@anchor{4659}
@subsection Starting and ending a curses application


Before doing anything, curses must be initialized.  This is done by
calling the @ref{2608,,initscr()} function, which will determine the
terminal type, send any required setup codes to the terminal, and
create various internal data structures.  If successful,
@code{initscr()} returns a window object representing the entire
screen; this is usually called @code{stdscr} after the name of the
corresponding C variable.

@example
import curses
stdscr = curses.initscr()
@end example

Usually curses applications turn off automatic echoing of keys to the
screen, in order to be able to read keys and only display them under
certain circumstances.  This requires calling the
@ref{2625,,noecho()} function.

@example
curses.noecho()
@end example

Applications will also commonly need to react to keys instantly,
without requiring the Enter key to be pressed; this is called cbreak
mode, as opposed to the usual buffered input mode.

@example
curses.cbreak()
@end example

Terminals usually return special keys, such as the cursor keys or navigation
keys such as Page Up and Home, as a multibyte escape sequence.  While you could
write your application to expect such sequences and process them accordingly,
curses can do it for you, returning a special value such as
@code{curses.KEY_LEFT}.  To get curses to do the job, you’ll have to enable
keypad mode.

@example
stdscr.keypad(True)
@end example

Terminating a curses application is much easier than starting one. You’ll need
to call:

@example
curses.nocbreak()
stdscr.keypad(False)
curses.echo()
@end example

to reverse the curses-friendly terminal settings. Then call the
@ref{2605,,endwin()} function to restore the terminal to its original
operating mode.

@example
curses.endwin()
@end example

A common problem when debugging a curses application is to get your terminal
messed up when the application dies without restoring the terminal to its
previous state.  In Python this commonly happens when your code is buggy and
raises an uncaught exception.  Keys are no longer echoed to the screen when
you type them, for example, which makes using the shell difficult.

In Python you can avoid these complications and make debugging much easier by
importing the @ref{598,,curses.wrapper()} function and using it like this:

@example
from curses import wrapper

def main(stdscr):
    # Clear screen
    stdscr.clear()

    # This raises ZeroDivisionError when i == 10.
    for i in range(0, 11):
        v = i-10
        stdscr.addstr(i, 0, '10 divided by @{@} is @{@}'.format(v, 10/v))

    stdscr.refresh()
    stdscr.getkey()

wrapper(main)
@end example

The @ref{598,,wrapper()} function takes a callable object and does the
initializations described above, also initializing colors if color
support is present.  @code{wrapper()} then runs your provided callable.
Once the callable returns, @code{wrapper()} will restore the original
state of the terminal.  The callable is called inside a
@ref{f6d,,try}…@ref{16e,,except} that catches exceptions, restores
the state of the terminal, and then re-raises the exception.  Therefore
your terminal won’t be left in a funny state on exception and you’ll be
able to read the exception’s message and traceback.

@node Windows and Pads,Displaying Text,Starting and ending a curses application,Curses Programming with Python
@anchor{howto/curses windows-and-pads}@anchor{465a}
@subsection Windows and Pads


Windows are the basic abstraction in curses.  A window object represents a
rectangular area of the screen, and supports methods to display text,
erase it, allow the user to input strings, and so forth.

The @code{stdscr} object returned by the @ref{2608,,initscr()} function is a
window object that covers the entire screen.  Many programs may need
only this single window, but you might wish to divide the screen into
smaller windows, in order to redraw or clear them separately. The
@ref{2623,,newwin()} function creates a new window of a given size,
returning the new window object.

@example
begin_x = 20; begin_y = 7
height = 5; width = 40
win = curses.newwin(height, width, begin_y, begin_x)
@end example

Note that the coordinate system used in curses is unusual.
Coordinates are always passed in the order @emph{y,x}, and the top-left
corner of a window is coordinate (0,0).  This breaks the normal
convention for handling coordinates where the @emph{x} coordinate comes
first.  This is an unfortunate difference from most other computer
applications, but it’s been part of curses since it was first written,
and it’s too late to change things now.

Your application can determine the size of the screen by using the
@code{curses.LINES} and @code{curses.COLS} variables to obtain the @emph{y} and
@emph{x} sizes.  Legal coordinates will then extend from @code{(0,0)} to
@code{(curses.LINES - 1, curses.COLS - 1)}.

When you call a method to display or erase text, the effect doesn’t
immediately show up on the display.  Instead you must call the
@ref{2603,,refresh()} method of window objects to update the
screen.

This is because curses was originally written with slow 300-baud
terminal connections in mind; with these terminals, minimizing the
time required to redraw the screen was very important.  Instead curses
accumulates changes to the screen and displays them in the most
efficient manner when you call @code{refresh()}.  For example, if your
program displays some text in a window and then clears the window,
there’s no need to send the original text because they’re never
visible.

In practice, explicitly telling curses to redraw a window doesn’t
really complicate programming with curses much. Most programs go into a flurry
of activity, and then pause waiting for a keypress or some other action on the
part of the user.  All you have to do is to be sure that the screen has been
redrawn before pausing to wait for user input, by first calling
@code{stdscr.refresh()} or the @code{refresh()} method of some other relevant
window.

A pad is a special case of a window; it can be larger than the actual display
screen, and only a portion of the pad displayed at a time. Creating a pad
requires the pad’s height and width, while refreshing a pad requires giving the
coordinates of the on-screen area where a subsection of the pad will be
displayed.

@example
pad = curses.newpad(100, 100)
# These loops fill the pad with letters; addch() is
# explained in the next section
for y in range(0, 99):
    for x in range(0, 99):
        pad.addch(y,x, ord('a') + (x*x+y*y) % 26)

# Displays a section of the pad in the middle of the screen.
# (0,0) : coordinate of upper-left corner of pad area to display.
# (5,5) : coordinate of upper-left corner of window area to be filled
#         with pad content.
# (20, 75) : coordinate of lower-right corner of window area to be
#          : filled with pad content.
pad.refresh( 0,0, 5,5, 20,75)
@end example

The @code{refresh()} call displays a section of the pad in the rectangle
extending from coordinate (5,5) to coordinate (20,75) on the screen; the upper
left corner of the displayed section is coordinate (0,0) on the pad.  Beyond
that difference, pads are exactly like ordinary windows and support the same
methods.

If you have multiple windows and pads on screen there is a more
efficient way to update the screen and prevent annoying screen flicker
as each part of the screen gets updated.  @code{refresh()} actually
does two things:


@enumerate 

@item 
Calls the @ref{2601,,noutrefresh()} method of each window
to update an underlying data structure representing the desired
state of the screen.

@item 
Calls the function @ref{2600,,doupdate()} function to change the
physical screen to match the desired state recorded in the data structure.
@end enumerate

Instead you can call @code{noutrefresh()} on a number of windows to
update the data structure, and then call @code{doupdate()} to update
the screen.

@node Displaying Text,User Input,Windows and Pads,Curses Programming with Python
@anchor{howto/curses displaying-text}@anchor{465b}
@subsection Displaying Text


From a C programmer’s point of view, curses may sometimes look like a
twisty maze of functions, all subtly different.  For example,
@code{addstr()} displays a string at the current cursor location in
the @code{stdscr} window, while @code{mvaddstr()} moves to a given y,x
coordinate first before displaying the string. @code{waddstr()} is just
like @code{addstr()}, but allows specifying a window to use instead of
using @code{stdscr} by default. @code{mvwaddstr()} allows specifying both
a window and a coordinate.

Fortunately the Python interface hides all these details.  @code{stdscr}
is a window object like any other, and methods such as
@ref{2602,,addstr()} accept multiple argument forms.  Usually there
are four different forms.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Form

@tab

Description

@item

@emph{str} or @emph{ch}

@tab

Display the string @emph{str} or character @emph{ch} at
the current position

@item

@emph{str} or @emph{ch}, @emph{attr}

@tab

Display the string @emph{str} or character @emph{ch},
using attribute @emph{attr} at the current
position

@item

@emph{y}, @emph{x}, @emph{str} or @emph{ch}

@tab

Move to position @emph{y,x} within the window, and
display @emph{str} or @emph{ch}

@item

@emph{y}, @emph{x}, @emph{str} or @emph{ch}, @emph{attr}

@tab

Move to position @emph{y,x} within the window, and
display @emph{str} or @emph{ch}, using attribute @emph{attr}

@end multitable


Attributes allow displaying text in highlighted forms such as boldface,
underline, reverse code, or in color.  They’ll be explained in more detail in
the next subsection.

The @ref{2602,,addstr()} method takes a Python string or
bytestring as the value to be displayed.  The contents of bytestrings
are sent to the terminal as-is.  Strings are encoded to bytes using
the value of the window’s @code{encoding} attribute; this defaults to
the default system encoding as returned by
@ref{1b5,,locale.getpreferredencoding()}.

The @ref{263d,,addch()} methods take a character, which can be
either a string of length 1, a bytestring of length 1, or an integer.

Constants are provided for extension characters; these constants are
integers greater than 255.  For example, @code{ACS_PLMINUS} is a +/-
symbol, and @code{ACS_ULCORNER} is the upper left corner of a box
(handy for drawing borders).  You can also use the appropriate Unicode
character.

Windows remember where the cursor was left after the last operation, so if you
leave out the @emph{y,x} coordinates, the string or character will be displayed
wherever the last operation left off.  You can also move the cursor with the
@code{move(y,x)} method.  Because some terminals always display a flashing cursor,
you may want to ensure that the cursor is positioned in some location where it
won’t be distracting; it can be confusing to have the cursor blinking at some
apparently random location.

If your application doesn’t need a blinking cursor at all, you can
call @code{curs_set(False)} to make it invisible.  For compatibility
with older curses versions, there’s a @code{leaveok(bool)} function
that’s a synonym for @ref{25fa,,curs_set()}.  When @emph{bool} is true, the
curses library will attempt to suppress the flashing cursor, and you
won’t need to worry about leaving it in odd locations.

@menu
* Attributes and Color:: 

@end menu

@node Attributes and Color,,,Displaying Text
@anchor{howto/curses attributes-and-color}@anchor{465c}
@subsubsection Attributes and Color


Characters can be displayed in different ways.  Status lines in a text-based
application are commonly shown in reverse video, or a text viewer may need to
highlight certain words.  curses supports this by allowing you to specify an
attribute for each cell on the screen.

An attribute is an integer, each bit representing a different
attribute.  You can try to display text with multiple attribute bits
set, but curses doesn’t guarantee that all the possible combinations
are available, or that they’re all visually distinct.  That depends on
the ability of the terminal being used, so it’s safest to stick to the
most commonly available attributes, listed here.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Attribute

@tab

Description

@item

@code{A_BLINK}

@tab

Blinking text

@item

@code{A_BOLD}

@tab

Extra bright or bold text

@item

@code{A_DIM}

@tab

Half bright text

@item

@code{A_REVERSE}

@tab

Reverse-video text

@item

@code{A_STANDOUT}

@tab

The best highlighting mode available

@item

@code{A_UNDERLINE}

@tab

Underlined text

@end multitable


So, to display a reverse-video status line on the top line of the screen, you
could code:

@example
stdscr.addstr(0, 0, "Current mode: Typing mode",
              curses.A_REVERSE)
stdscr.refresh()
@end example

The curses library also supports color on those terminals that provide it. The
most common such terminal is probably the Linux console, followed by color
xterms.

To use color, you must call the @ref{2630,,start_color()} function soon
after calling @ref{2608,,initscr()}, to initialize the default color set
(the @ref{598,,curses.wrapper()} function does this automatically).  Once that’s
done, the @ref{2610,,has_colors()} function returns TRUE if the terminal
in use can
actually display color.  (Note: curses uses the American spelling ‘color’,
instead of the Canadian/British spelling ‘colour’.  If you’re used to the
British spelling, you’ll have to resign yourself to misspelling it for the sake
of these functions.)

The curses library maintains a finite number of color pairs, containing a
foreground (or text) color and a background color.  You can get the attribute
value corresponding to a color pair with the @ref{25f8,,color_pair()}
function; this can be bitwise-OR’ed with other attributes such as
@code{A_REVERSE}, but again, such combinations are not guaranteed to work
on all terminals.

An example, which displays a line of text using color pair 1:

@example
stdscr.addstr("Pretty text", curses.color_pair(1))
stdscr.refresh()
@end example

As I said before, a color pair consists of a foreground and background color.
The @code{init_pair(n, f, b)} function changes the definition of color pair @emph{n}, to
foreground color f and background color b.  Color pair 0 is hard-wired to white
on black, and cannot be changed.

Colors are numbered, and @code{start_color()} initializes 8 basic
colors when it activates color mode.  They are: 0:black, 1:red,
2:green, 3:yellow, 4:blue, 5:magenta, 6:cyan, and 7:white.  The @ref{2a,,curses}
module defines named constants for each of these colors:
@code{curses.COLOR_BLACK}, @code{curses.COLOR_RED}, and so forth.

Let’s put all this together. To change color 1 to red text on a white
background, you would call:

@example
curses.init_pair(1, curses.COLOR_RED, curses.COLOR_WHITE)
@end example

When you change a color pair, any text already displayed using that color pair
will change to the new colors.  You can also display new text in this color
with:

@example
stdscr.addstr(0,0, "RED ALERT!", curses.color_pair(1))
@end example

Very fancy terminals can change the definitions of the actual colors to a given
RGB value.  This lets you change color 1, which is usually red, to purple or
blue or any other color you like.  Unfortunately, the Linux console doesn’t
support this, so I’m unable to try it out, and can’t provide any examples.  You
can check if your terminal can do this by calling
@ref{25f5,,can_change_color()}, which returns @code{True} if the capability is
there.  If you’re lucky enough to have such a talented terminal, consult your
system’s man pages for more information.

@node User Input,For More Information,Displaying Text,Curses Programming with Python
@anchor{howto/curses user-input}@anchor{465d}
@subsection User Input


The C curses library offers only very simple input mechanisms. Python’s
@ref{2a,,curses} module adds a basic text-input widget.  (Other libraries
such as Urwid@footnote{https://pypi.org/project/urwid/} have more extensive
collections of widgets.)

There are two methods for getting input from a window:


@itemize *

@item 
@ref{260c,,getch()} refreshes the screen and then waits for
the user to hit a key, displaying the key if @ref{2604,,echo()} has been
called earlier.  You can optionally specify a coordinate to which
the cursor should be moved before pausing.

@item 
@ref{2655,,getkey()} does the same thing but converts the
integer to a string.  Individual characters are returned as
1-character strings, and special keys such as function keys return
longer strings containing a key name such as @code{KEY_UP} or @code{^G}.
@end itemize

It’s possible to not wait for the user using the
@ref{266b,,nodelay()} window method. After @code{nodelay(True)},
@code{getch()} and @code{getkey()} for the window become
non-blocking. To signal that no input is ready, @code{getch()} returns
@code{curses.ERR} (a value of -1) and @code{getkey()} raises an exception.
There’s also a @ref{2614,,halfdelay()} function, which can be used to (in
effect) set a timer on each @code{getch()}; if no input becomes
available within a specified delay (measured in tenths of a second),
curses raises an exception.

The @code{getch()} method returns an integer; if it’s between 0 and 255, it
represents the ASCII code of the key pressed.  Values greater than 255 are
special keys such as Page Up, Home, or the cursor keys. You can compare the
value returned to constants such as @code{curses.KEY_PPAGE},
@code{curses.KEY_HOME}, or @code{curses.KEY_LEFT}.  The main loop of
your program may look something like this:

@example
while True:
    c = stdscr.getch()
    if c == ord('p'):
        PrintDocument()
    elif c == ord('q'):
        break  # Exit the while loop
    elif c == curses.KEY_HOME:
        x = y = 0
@end example

The @ref{2b,,curses.ascii} module supplies ASCII class membership functions that
take either integer or 1-character string arguments; these may be useful in
writing more readable tests for such loops.  It also supplies
conversion functions  that take either integer or 1-character-string arguments
and return the same type.  For example, @ref{269f,,curses.ascii.ctrl()} returns the
control character corresponding to its argument.

There’s also a method to retrieve an entire string,
@ref{2658,,getstr()}.  It isn’t used very often, because its
functionality is quite limited; the only editing keys available are
the backspace key and the Enter key, which terminates the string.  It
can optionally be limited to a fixed number of characters.

@example
curses.echo()            # Enable echoing of characters

# Get a 15-character string, with the cursor on the top line
s = stdscr.getstr(0,0, 15)
@end example

The @ref{2d,,curses.textpad} module supplies a text box that supports an
Emacs-like set of keybindings.  Various methods of the
@ref{2685,,Textbox} class support editing with input
validation and gathering the edit results either with or without
trailing spaces.  Here’s an example:

@example
import curses
from curses.textpad import Textbox, rectangle

def main(stdscr):
    stdscr.addstr(0, 0, "Enter IM message: (hit Ctrl-G to send)")

    editwin = curses.newwin(5,30, 2,1)
    rectangle(stdscr, 1,0, 1+5+1, 1+30+1)
    stdscr.refresh()

    box = Textbox(editwin)

    # Let the user edit until Ctrl-G is struck.
    box.edit()

    # Get resulting contents
    message = box.gather()
@end example

See the library documentation on @ref{2d,,curses.textpad} for more details.

@node For More Information,,User Input,Curses Programming with Python
@anchor{howto/curses for-more-information}@anchor{465e}
@subsection For More Information


This HOWTO doesn’t cover some advanced topics, such as reading the
contents of the screen or capturing mouse events from an xterm
instance, but the Python library page for the @ref{2a,,curses} module is now
reasonably complete.  You should browse it next.

If you’re in doubt about the detailed behavior of the curses
functions, consult the manual pages for your curses implementation,
whether it’s ncurses or a proprietary Unix vendor’s.  The manual pages
will document any quirks, and provide complete lists of all the
functions, attributes, and @code{ACS_*} characters available to
you.

Because the curses API is so large, some functions aren’t supported in
the Python interface.  Often this isn’t because they’re difficult to
implement, but because no one has needed them yet.  Also, Python
doesn’t yet support the menu library associated with ncurses.
Patches adding support for these would be welcome; see
the Python Developer's Guide@footnote{https://devguide.python.org/} to
learn more about submitting patches to Python.


@itemize *

@item 
Writing Programs with NCURSES@footnote{http://invisible-island.net/ncurses/ncurses-intro.html}:
a lengthy tutorial for C programmers.

@item 
The ncurses man page@footnote{https://linux.die.net/man/3/ncurses}

@item 
The ncurses FAQ@footnote{http://invisible-island.net/ncurses/ncurses.faq.html}

@item 
"Use curses... don't swear"@footnote{https://www.youtube.com/watch?v=eN1eZtjLEnU}:
video of a PyCon 2013 talk on controlling terminals using curses or Urwid.

@item 
"Console Applications with Urwid"@footnote{http://www.pyvideo.org/video/1568/console-applications-with-urwid}:
video of a PyCon CA 2012 talk demonstrating some applications written using
Urwid.
@end itemize

@node Descriptor HowTo Guide,Enum HOWTO,Curses Programming with Python,Python HOWTOs
@anchor{howto/descriptor doc}@anchor{465f}@anchor{howto/descriptor descriptor-howto-guide}@anchor{4660}@anchor{howto/descriptor descriptorhowto}@anchor{4661}
@section Descriptor HowTo Guide


@*Author: 
Raymond Hettinger


@*Contact: 
<python at rcn dot com>


@ref{17bc,,Descriptors} let objects customize attribute lookup,
storage, and deletion.

This guide has four major sections:


@enumerate 

@item 
The “primer” gives a basic overview, moving gently from simple examples,
adding one feature at a time.  Start here if you’re new to descriptors.

@item 
The second section shows a complete, practical descriptor example.  If you
already know the basics, start there.

@item 
The third section provides a more technical tutorial that goes into the
detailed mechanics of how descriptors work.  Most people don’t need this
level of detail.

@item 
The last section has pure Python equivalents for built-in descriptors that
are written in C.  Read this if you’re curious about how functions turn
into bound methods or about the implementation of common tools like
@ref{254,,classmethod()}, @ref{253,,staticmethod()}, @ref{501,,property()}, and
@ref{3cd2,,__slots__}.
@end enumerate

@menu
* Primer:: 
* Complete Practical Example:: 
* Technical Tutorial:: 
* Pure Python Equivalents:: 

@end menu

@node Primer,Complete Practical Example,,Descriptor HowTo Guide
@anchor{howto/descriptor primer}@anchor{4662}
@subsection Primer


In this primer, we start with the most basic possible example and then we’ll
add new capabilities one by one.

@menu
* Simple example; A descriptor that returns a constant: Simple example A descriptor that returns a constant. 
* Dynamic lookups:: 
* Managed attributes:: 
* Customized names:: 
* Closing thoughts:: 

@end menu

@node Simple example A descriptor that returns a constant,Dynamic lookups,,Primer
@anchor{howto/descriptor simple-example-a-descriptor-that-returns-a-constant}@anchor{4663}
@subsubsection Simple example: A descriptor that returns a constant


The @code{Ten} class is a descriptor whose @code{__get__()} method always
returns the constant @code{10}:

@example
class Ten:
    def __get__(self, obj, objtype=None):
        return 10
@end example

To use the descriptor, it must be stored as a class variable in another class:

@example
class A:
    x = 5                       # Regular class attribute
    y = Ten()                   # Descriptor instance
@end example

An interactive session shows the difference between normal attribute lookup
and descriptor lookup:

@example
>>> a = A()                     # Make an instance of class A
>>> a.x                         # Normal attribute lookup
5
>>> a.y                         # Descriptor lookup
10
@end example

In the @code{a.x} attribute lookup, the dot operator finds @code{'x': 5}
in the class dictionary.  In the @code{a.y} lookup, the dot operator
finds a descriptor instance, recognized by its @code{__get__} method.
Calling that method returns @code{10}.

Note that the value @code{10} is not stored in either the class dictionary or the
instance dictionary.  Instead, the value @code{10} is computed on demand.

This example shows how a simple descriptor works, but it isn’t very useful.
For retrieving constants, normal attribute lookup would be better.

In the next section, we’ll create something more useful, a dynamic lookup.

@node Dynamic lookups,Managed attributes,Simple example A descriptor that returns a constant,Primer
@anchor{howto/descriptor dynamic-lookups}@anchor{4664}
@subsubsection Dynamic lookups


Interesting descriptors typically run computations instead of returning
constants:

@example
import os

class DirectorySize:

    def __get__(self, obj, objtype=None):
        return len(os.listdir(obj.dirname))

class Directory:

    size = DirectorySize()              # Descriptor instance

    def __init__(self, dirname):
        self.dirname = dirname          # Regular instance attribute
@end example

An interactive session shows that the lookup is dynamic — it computes
different, updated answers each time:

@example
>>> s = Directory('songs')
>>> g = Directory('games')
>>> s.size                              # The songs directory has twenty files
20
>>> g.size                              # The games directory has three files
3
>>> os.remove('games/chess')            # Delete a game
>>> g.size                              # File count is automatically updated
2
@end example

Besides showing how descriptors can run computations, this example also
reveals the purpose of the parameters to @code{__get__()}.  The @emph{self}
parameter is @emph{size}, an instance of @emph{DirectorySize}.  The @emph{obj} parameter is
either @emph{g} or @emph{s}, an instance of @emph{Directory}.  It is the @emph{obj} parameter that
lets the @code{__get__()} method learn the target directory.  The @emph{objtype}
parameter is the class @emph{Directory}.

@node Managed attributes,Customized names,Dynamic lookups,Primer
@anchor{howto/descriptor managed-attributes}@anchor{4665}
@subsubsection Managed attributes


A popular use for descriptors is managing access to instance data.  The
descriptor is assigned to a public attribute in the class dictionary while the
actual data is stored as a private attribute in the instance dictionary.  The
descriptor’s @code{__get__()} and @code{__set__()} methods are triggered when
the public attribute is accessed.

In the following example, @emph{age} is the public attribute and @emph{_age} is the
private attribute.  When the public attribute is accessed, the descriptor logs
the lookup or update:

@example
import logging

logging.basicConfig(level=logging.INFO)

class LoggedAgeAccess:

    def __get__(self, obj, objtype=None):
        value = obj._age
        logging.info('Accessing %r giving %r', 'age', value)
        return value

    def __set__(self, obj, value):
        logging.info('Updating %r to %r', 'age', value)
        obj._age = value

class Person:

    age = LoggedAgeAccess()             # Descriptor instance

    def __init__(self, name, age):
        self.name = name                # Regular instance attribute
        self.age = age                  # Calls __set__()

    def birthday(self):
        self.age += 1                   # Calls both __get__() and __set__()
@end example

An interactive session shows that all access to the managed attribute @emph{age} is
logged, but that the regular attribute @emph{name} is not logged:

@c import logging, sys
@c logging.basicConfig(level=logging.INFO, stream=sys.stdout, force=True)

@example
>>> mary = Person('Mary M', 30)         # The initial age update is logged
INFO:root:Updating 'age' to 30
>>> dave = Person('David D', 40)
INFO:root:Updating 'age' to 40

>>> vars(mary)                          # The actual data is in a private attribute
@{'name': 'Mary M', '_age': 30@}
>>> vars(dave)
@{'name': 'David D', '_age': 40@}

>>> mary.age                            # Access the data and log the lookup
INFO:root:Accessing 'age' giving 30
30
>>> mary.birthday()                     # Updates are logged as well
INFO:root:Accessing 'age' giving 30
INFO:root:Updating 'age' to 31

>>> dave.name                           # Regular attribute lookup isn't logged
'David D'
>>> dave.age                            # Only the managed attribute is logged
INFO:root:Accessing 'age' giving 40
40
@end example

One major issue with this example is that the private name @emph{_age} is hardwired in
the @emph{LoggedAgeAccess} class.  That means that each instance can only have one
logged attribute and that its name is unchangeable.  In the next example,
we’ll fix that problem.

@node Customized names,Closing thoughts,Managed attributes,Primer
@anchor{howto/descriptor customized-names}@anchor{4666}
@subsubsection Customized names


When a class uses descriptors, it can inform each descriptor about which
variable name was used.

In this example, the @code{Person} class has two descriptor instances,
@emph{name} and @emph{age}.  When the @code{Person} class is defined, it makes a
callback to @code{__set_name__()} in @emph{LoggedAccess} so that the field names can
be recorded, giving each descriptor its own @emph{public_name} and @emph{private_name}:

@example
import logging

logging.basicConfig(level=logging.INFO)

class LoggedAccess:

    def __set_name__(self, owner, name):
        self.public_name = name
        self.private_name = '_' + name

    def __get__(self, obj, objtype=None):
        value = getattr(obj, self.private_name)
        logging.info('Accessing %r giving %r', self.public_name, value)
        return value

    def __set__(self, obj, value):
        logging.info('Updating %r to %r', self.public_name, value)
        setattr(obj, self.private_name, value)

class Person:

    name = LoggedAccess()                # First descriptor instance
    age = LoggedAccess()                 # Second descriptor instance

    def __init__(self, name, age):
        self.name = name                 # Calls the first descriptor
        self.age = age                   # Calls the second descriptor

    def birthday(self):
        self.age += 1
@end example

An interactive session shows that the @code{Person} class has called
@code{__set_name__()} so that the field names would be recorded.  Here
we call @ref{12fc,,vars()} to look up the descriptor without triggering it:

@example
>>> vars(vars(Person)['name'])
@{'public_name': 'name', 'private_name': '_name'@}
>>> vars(vars(Person)['age'])
@{'public_name': 'age', 'private_name': '_age'@}
@end example

The new class now logs access to both @emph{name} and @emph{age}:

@c import logging, sys
@c logging.basicConfig(level=logging.INFO, stream=sys.stdout, force=True)

@example
>>> pete = Person('Peter P', 10)
INFO:root:Updating 'name' to 'Peter P'
INFO:root:Updating 'age' to 10
>>> kate = Person('Catherine C', 20)
INFO:root:Updating 'name' to 'Catherine C'
INFO:root:Updating 'age' to 20
@end example

The two @emph{Person} instances contain only the private names:

@example
>>> vars(pete)
@{'_name': 'Peter P', '_age': 10@}
>>> vars(kate)
@{'_name': 'Catherine C', '_age': 20@}
@end example

@node Closing thoughts,,Customized names,Primer
@anchor{howto/descriptor closing-thoughts}@anchor{4667}
@subsubsection Closing thoughts


A @ref{17bc,,descriptor} is what we call any object that defines @code{__get__()},
@code{__set__()}, or @code{__delete__()}.

Optionally, descriptors can have a @code{__set_name__()} method.  This is only
used in cases where a descriptor needs to know either the class where it was
created or the name of class variable it was assigned to.  (This method, if
present, is called even if the class is not a descriptor.)

Descriptors get invoked by the dot operator during attribute lookup.  If a
descriptor is accessed indirectly with @code{vars(some_class)[descriptor_name]},
the descriptor instance is returned without invoking it.

Descriptors only work when used as class variables.  When put in instances,
they have no effect.

The main motivation for descriptors is to provide a hook allowing objects
stored in class variables to control what happens during attribute lookup.

Traditionally, the calling class controls what happens during lookup.
Descriptors invert that relationship and allow the data being looked-up to
have a say in the matter.

Descriptors are used throughout the language.  It is how functions turn into
bound methods.  Common tools like @ref{254,,classmethod()}, @ref{253,,staticmethod()},
@ref{501,,property()}, and @ref{4f4,,functools.cached_property()} are all implemented as
descriptors.

@node Complete Practical Example,Technical Tutorial,Primer,Descriptor HowTo Guide
@anchor{howto/descriptor complete-practical-example}@anchor{4668}
@subsection Complete Practical Example


In this example, we create a practical and powerful tool for locating
notoriously hard to find data corruption bugs.

@menu
* Validator class:: 
* Custom validators:: 
* Practical application:: 

@end menu

@node Validator class,Custom validators,,Complete Practical Example
@anchor{howto/descriptor validator-class}@anchor{4669}
@subsubsection Validator class


A validator is a descriptor for managed attribute access.  Prior to storing
any data, it verifies that the new value meets various type and range
restrictions.  If those restrictions aren’t met, it raises an exception to
prevent data corruption at its source.

This @code{Validator} class is both an @ref{d5c,,abstract base class} and a
managed attribute descriptor:

@example
from abc import ABC, abstractmethod

class Validator(ABC):

    def __set_name__(self, owner, name):
        self.private_name = '_' + name

    def __get__(self, obj, objtype=None):
        return getattr(obj, self.private_name)

    def __set__(self, obj, value):
        self.validate(value)
        setattr(obj, self.private_name, value)

    @@abstractmethod
    def validate(self, value):
        pass
@end example

Custom validators need to inherit from @code{Validator} and must supply a
@code{validate()} method to test various restrictions as needed.

@node Custom validators,Practical application,Validator class,Complete Practical Example
@anchor{howto/descriptor custom-validators}@anchor{466a}
@subsubsection Custom validators


Here are three practical data validation utilities:


@enumerate 

@item 
@code{OneOf} verifies that a value is one of a restricted set of options.

@item 
@code{Number} verifies that a value is either an @ref{1b1,,int} or
@ref{256,,float}.  Optionally, it verifies that a value is between a given
minimum or maximum.

@item 
@code{String} verifies that a value is a @ref{2ef,,str}.  Optionally, it
validates a given minimum or maximum length.  It can validate a
user-defined predicate@footnote{https://en.wikipedia.org/wiki/Predicate_(mathematical_logic)} as well.
@end enumerate

@example
class OneOf(Validator):

    def __init__(self, *options):
        self.options = set(options)

    def validate(self, value):
        if value not in self.options:
            raise ValueError(f'Expected @{value!r@} to be one of @{self.options!r@}')

class Number(Validator):

    def __init__(self, minvalue=None, maxvalue=None):
        self.minvalue = minvalue
        self.maxvalue = maxvalue

    def validate(self, value):
        if not isinstance(value, (int, float)):
            raise TypeError(f'Expected @{value!r@} to be an int or float')
        if self.minvalue is not None and value < self.minvalue:
            raise ValueError(
                f'Expected @{value!r@} to be at least @{self.minvalue!r@}'
            )
        if self.maxvalue is not None and value > self.maxvalue:
            raise ValueError(
                f'Expected @{value!r@} to be no more than @{self.maxvalue!r@}'
            )

class String(Validator):

    def __init__(self, minsize=None, maxsize=None, predicate=None):
        self.minsize = minsize
        self.maxsize = maxsize
        self.predicate = predicate

    def validate(self, value):
        if not isinstance(value, str):
            raise TypeError(f'Expected @{value!r@} to be an str')
        if self.minsize is not None and len(value) < self.minsize:
            raise ValueError(
                f'Expected @{value!r@} to be no smaller than @{self.minsize!r@}'
            )
        if self.maxsize is not None and len(value) > self.maxsize:
            raise ValueError(
                f'Expected @{value!r@} to be no bigger than @{self.maxsize!r@}'
            )
        if self.predicate is not None and not self.predicate(value):
            raise ValueError(
                f'Expected @{self.predicate@} to be true for @{value!r@}'
            )
@end example

@node Practical application,,Custom validators,Complete Practical Example
@anchor{howto/descriptor practical-application}@anchor{466b}
@subsubsection Practical application


Here’s how the data validators can be used in a real class:

@example
class Component:

    name = String(minsize=3, maxsize=10, predicate=str.isupper)
    kind = OneOf('wood', 'metal', 'plastic')
    quantity = Number(minvalue=0)

    def __init__(self, name, kind, quantity):
        self.name = name
        self.kind = kind
        self.quantity = quantity
@end example

The descriptors prevent invalid instances from being created:

@example
>>> Component('Widget', 'metal', 5)      # Blocked: 'Widget' is not all uppercase
Traceback (most recent call last):
    ...
ValueError: Expected <method 'isupper' of 'str' objects> to be true for 'Widget'

>>> Component('WIDGET', 'metle', 5)      # Blocked: 'metle' is misspelled
Traceback (most recent call last):
    ...
ValueError: Expected 'metle' to be one of @{'metal', 'plastic', 'wood'@}

>>> Component('WIDGET', 'metal', -5)     # Blocked: -5 is negative
Traceback (most recent call last):
    ...
ValueError: Expected -5 to be at least 0
>>> Component('WIDGET', 'metal', 'V')    # Blocked: 'V' isn't a number
Traceback (most recent call last):
    ...
TypeError: Expected 'V' to be an int or float

>>> c = Component('WIDGET', 'metal', 5)  # Allowed:  The inputs are valid
@end example

@node Technical Tutorial,Pure Python Equivalents,Complete Practical Example,Descriptor HowTo Guide
@anchor{howto/descriptor technical-tutorial}@anchor{466c}
@subsection Technical Tutorial


What follows is a more technical tutorial for the mechanics and details of how
descriptors work.

@menu
* Abstract:: 
* Definition and introduction:: 
* Descriptor protocol:: 
* Overview of descriptor invocation:: 
* Invocation from an instance:: 
* Invocation from a class:: 
* Invocation from super:: 
* Summary of invocation logic:: 
* Automatic name notification:: 
* ORM example:: 

@end menu

@node Abstract,Definition and introduction,,Technical Tutorial
@anchor{howto/descriptor abstract}@anchor{466d}
@subsubsection Abstract


Defines descriptors, summarizes the protocol, and shows how descriptors are
called.  Provides an example showing how object relational mappings work.

Learning about descriptors not only provides access to a larger toolset, it
creates a deeper understanding of how Python works.

@node Definition and introduction,Descriptor protocol,Abstract,Technical Tutorial
@anchor{howto/descriptor definition-and-introduction}@anchor{466e}
@subsubsection Definition and introduction


In general, a descriptor is an attribute value that has one of the methods in
the descriptor protocol.  Those methods are @code{__get__()}, @code{__set__()},
and @code{__delete__()}.  If any of those methods are defined for an
attribute, it is said to be a @ref{17bc,,descriptor}.

The default behavior for attribute access is to get, set, or delete the
attribute from an object’s dictionary.  For instance, @code{a.x} has a lookup chain
starting with @code{a.__dict__['x']}, then @code{type(a).__dict__['x']}, and
continuing through the method resolution order of @code{type(a)}. If the
looked-up value is an object defining one of the descriptor methods, then Python
may override the default behavior and invoke the descriptor method instead.
Where this occurs in the precedence chain depends on which descriptor methods
were defined.

Descriptors are a powerful, general purpose protocol.  They are the mechanism
behind properties, methods, static methods, class methods, and
@ref{78c,,super()}.  They are used throughout Python itself.  Descriptors
simplify the underlying C code and offer a flexible set of new tools for
everyday Python programs.

@node Descriptor protocol,Overview of descriptor invocation,Definition and introduction,Technical Tutorial
@anchor{howto/descriptor descriptor-protocol}@anchor{466f}
@subsubsection Descriptor protocol


@code{descr.__get__(self, obj, type=None) -> value}

@code{descr.__set__(self, obj, value) -> None}

@code{descr.__delete__(self, obj) -> None}

That is all there is to it.  Define any of these methods and an object is
considered a descriptor and can override default behavior upon being looked up
as an attribute.

If an object defines @code{__set__()} or @code{__delete__()}, it is considered
a data descriptor.  Descriptors that only define @code{__get__()} are called
non-data descriptors (they are often used for methods but other uses are
possible).

Data and non-data descriptors differ in how overrides are calculated with
respect to entries in an instance’s dictionary.  If an instance’s dictionary
has an entry with the same name as a data descriptor, the data descriptor
takes precedence.  If an instance’s dictionary has an entry with the same
name as a non-data descriptor, the dictionary entry takes precedence.

To make a read-only data descriptor, define both @code{__get__()} and
@code{__set__()} with the @code{__set__()} raising an @ref{163,,AttributeError} when
called.  Defining the @code{__set__()} method with an exception raising
placeholder is enough to make it a data descriptor.

@node Overview of descriptor invocation,Invocation from an instance,Descriptor protocol,Technical Tutorial
@anchor{howto/descriptor overview-of-descriptor-invocation}@anchor{4670}
@subsubsection Overview of descriptor invocation


A descriptor can be called directly with @code{desc.__get__(obj)} or
@code{desc.__get__(None, cls)}.

But it is more common for a descriptor to be invoked automatically from
attribute access.

The expression @code{obj.x} looks up the attribute @code{x} in the chain of
namespaces for @code{obj}.  If the search finds a descriptor outside of the
instance @code{__dict__}, its @code{__get__()} method is invoked according to the
precedence rules listed below.

The details of invocation depend on whether @code{obj} is an object, class, or
instance of super.

@node Invocation from an instance,Invocation from a class,Overview of descriptor invocation,Technical Tutorial
@anchor{howto/descriptor invocation-from-an-instance}@anchor{4671}
@subsubsection Invocation from an instance


Instance lookup scans through a chain of namespaces giving data descriptors
the highest priority, followed by instance variables, then non-data
descriptors, then class variables, and lastly @code{__getattr__()} if it is
provided.

If a descriptor is found for @code{a.x}, then it is invoked with:
@code{desc.__get__(a, type(a))}.

The logic for a dotted lookup is in @ref{70a,,object.__getattribute__()}.  Here is
a pure Python equivalent:

@example
def object_getattribute(obj, name):
    "Emulate PyObject_GenericGetAttr() in Objects/object.c"
    null = object()
    objtype = type(obj)
    cls_var = getattr(objtype, name, null)
    descr_get = getattr(type(cls_var), '__get__', null)
    if descr_get is not null:
        if (hasattr(type(cls_var), '__set__')
            or hasattr(type(cls_var), '__delete__')):
            return descr_get(cls_var, obj, objtype)     # data descriptor
    if hasattr(obj, '__dict__') and name in vars(obj):
        return vars(obj)[name]                          # instance variable
    if descr_get is not null:
        return descr_get(cls_var, obj, objtype)         # non-data descriptor
    if cls_var is not null:
        return cls_var                                  # class variable
    raise AttributeError(name)
@end example

@c # Test the fidelity of object_getattribute() by comparing it with the
@c # normal object.__getattribute__().  The former will be accessed by
@c # square brackets and the latter by the dot operator.
@c 
@c class Object:
@c 
@c     def __getitem__(obj, name):
@c         try:
@c             return object_getattribute(obj, name)
@c         except AttributeError:
@c             if not hasattr(type(obj), '__getattr__'):
@c                 raise
@c         return type(obj).__getattr__(obj, name)             # __getattr__
@c 
@c class DualOperator(Object):
@c 
@c     x = 10
@c 
@c     def __init__(self, z):
@c         self.z = z
@c 
@c     @property
@c     def p2(self):
@c         return 2 * self.x
@c 
@c     @property
@c     def p3(self):
@c         return 3 * self.x
@c 
@c     def m5(self, y):
@c         return 5 * y
@c 
@c     def m7(self, y):
@c         return 7 * y
@c 
@c     def __getattr__(self, name):
@c         return ('getattr_hook', self, name)
@c 
@c class DualOperatorWithSlots:
@c 
@c     __getitem__ = Object.__getitem__
@c 
@c     __slots__ = ['z']
@c 
@c     x = 15
@c 
@c     def __init__(self, z):
@c         self.z = z
@c 
@c     @property
@c     def p2(self):
@c         return 2 * self.x
@c 
@c     def m5(self, y):
@c         return 5 * y
@c 
@c     def __getattr__(self, name):
@c         return ('getattr_hook', self, name)

@c >>> a = DualOperator(11)
@c >>> vars(a).update(p3 = '_p3', m7 = '_m7')
@c >>> a.x == a['x'] == 10
@c True
@c >>> a.z == a['z'] == 11
@c True
@c >>> a.p2 == a['p2'] == 20
@c True
@c >>> a.p3 == a['p3'] == 30
@c True
@c >>> a.m5(100) == a.m5(100) == 500
@c True
@c >>> a.m7 == a['m7'] == '_m7'
@c True
@c >>> a.g == a['g'] == ('getattr_hook', a, 'g')
@c True
@c 
@c >>> b = DualOperatorWithSlots(22)
@c >>> b.x == b['x'] == 15
@c True
@c >>> b.z == b['z'] == 22
@c True
@c >>> b.p2 == b['p2'] == 30
@c True
@c >>> b.m5(200) == b['m5'](200) == 1000
@c True
@c >>> b.g == b['g'] == ('getattr_hook', b, 'g')
@c True

Note, there is no @code{__getattr__()} hook in the @code{__getattribute__()}
code.  That is why calling @code{__getattribute__()} directly or with
@code{super().__getattribute__} will bypass @code{__getattr__()} entirely.

Instead, it is the dot operator and the @ref{708,,getattr()} function that are
responsible for invoking @code{__getattr__()} whenever @code{__getattribute__()}
raises an @ref{163,,AttributeError}.  Their logic is encapsulated in a helper
function:

@example
def getattr_hook(obj, name):
    "Emulate slot_tp_getattr_hook() in Objects/typeobject.c"
    try:
        return obj.__getattribute__(name)
    except AttributeError:
        if not hasattr(type(obj), '__getattr__'):
            raise
    return type(obj).__getattr__(obj, name)             # __getattr__
@end example

@c >>> class ClassWithGetAttr:
@c ...     x = 123
@c ...     def __getattr__(self, attr):
@c ...         return attr.upper()
@c ...
@c >>> cw = ClassWithGetAttr()
@c >>> cw.y = 456
@c >>> getattr_hook(cw, 'x')
@c 123
@c >>> getattr_hook(cw, 'y')
@c 456
@c >>> getattr_hook(cw, 'z')
@c 'Z'
@c 
@c >>> class ClassWithoutGetAttr:
@c ...     x = 123
@c ...
@c >>> cwo = ClassWithoutGetAttr()
@c >>> cwo.y = 456
@c >>> getattr_hook(cwo, 'x')
@c 123
@c >>> getattr_hook(cwo, 'y')
@c 456
@c >>> getattr_hook(cwo, 'z')
@c Traceback (most recent call last):
@c     ...
@c AttributeError: 'ClassWithoutGetAttr' object has no attribute 'z'

@node Invocation from a class,Invocation from super,Invocation from an instance,Technical Tutorial
@anchor{howto/descriptor invocation-from-a-class}@anchor{4672}
@subsubsection Invocation from a class


The logic for a dotted lookup such as @code{A.x} is in
@code{type.__getattribute__()}.  The steps are similar to those for
@ref{70a,,object.__getattribute__()} but the instance dictionary lookup is replaced
by a search through the class’s @ref{1a60,,method resolution order}.

If a descriptor is found, it is invoked with @code{desc.__get__(None, A)}.

The full C implementation can be found in @code{type_getattro()} and
@code{_PyType_Lookup()} in Objects/typeobject.c@footnote{https://github.com/python/cpython/tree/main/Objects/typeobject.c}.

@node Invocation from super,Summary of invocation logic,Invocation from a class,Technical Tutorial
@anchor{howto/descriptor invocation-from-super}@anchor{4673}
@subsubsection Invocation from super


The logic for super’s dotted lookup is in the @code{__getattribute__()} method for
object returned by @ref{78c,,super()}.

A dotted lookup such as @code{super(A, obj).m} searches @code{obj.__class__.__mro__}
for the base class @code{B} immediately following @code{A} and then returns
@code{B.__dict__['m'].__get__(obj, A)}.  If not a descriptor, @code{m} is returned
unchanged.

The full C implementation can be found in @code{super_getattro()} in
Objects/typeobject.c@footnote{https://github.com/python/cpython/tree/main/Objects/typeobject.c}.  A pure Python equivalent can be found in
Guido's Tutorial@footnote{https://www.python.org/download/releases/2.2.3/descrintro/#cooperation}.

@node Summary of invocation logic,Automatic name notification,Invocation from super,Technical Tutorial
@anchor{howto/descriptor summary-of-invocation-logic}@anchor{4674}
@subsubsection Summary of invocation logic


The mechanism for descriptors is embedded in the @code{__getattribute__()}
methods for @ref{5a5,,object}, @ref{89e,,type}, and @ref{78c,,super()}.

The important points to remember are:


@itemize *

@item 
Descriptors are invoked by the @code{__getattribute__()} method.

@item 
Classes inherit this machinery from @ref{5a5,,object}, @ref{89e,,type}, or
@ref{78c,,super()}.

@item 
Overriding @code{__getattribute__()} prevents automatic descriptor calls
because all the descriptor logic is in that method.

@item 
@ref{70a,,object.__getattribute__()} and @code{type.__getattribute__()} make
different calls to @code{__get__()}.  The first includes the instance and may
include the class.  The second puts in @code{None} for the instance and always
includes the class.

@item 
Data descriptors always override instance dictionaries.

@item 
Non-data descriptors may be overridden by instance dictionaries.
@end itemize

@node Automatic name notification,ORM example,Summary of invocation logic,Technical Tutorial
@anchor{howto/descriptor automatic-name-notification}@anchor{4675}
@subsubsection Automatic name notification


Sometimes it is desirable for a descriptor to know what class variable name it
was assigned to.  When a new class is created, the @ref{89e,,type} metaclass
scans the dictionary of the new class.  If any of the entries are descriptors
and if they define @code{__set_name__()}, that method is called with two
arguments.  The @emph{owner} is the class where the descriptor is used, and the
@emph{name} is the class variable the descriptor was assigned to.

The implementation details are in @code{type_new()} and
@code{set_names()} in Objects/typeobject.c@footnote{https://github.com/python/cpython/tree/main/Objects/typeobject.c}.

Since the update logic is in @code{type.__new__()}, notifications only take
place at the time of class creation.  If descriptors are added to the class
afterwards, @code{__set_name__()} will need to be called manually.

@node ORM example,,Automatic name notification,Technical Tutorial
@anchor{howto/descriptor orm-example}@anchor{4676}
@subsubsection ORM example


The following code is simplified skeleton showing how data descriptors could
be used to implement an object relational mapping@footnote{https://en.wikipedia.org/wiki/Object%E2%80%93relational_mapping}.

The essential idea is that the data is stored in an external database.  The
Python instances only hold keys to the database’s tables.  Descriptors take
care of lookups or updates:

@example
class Field:

    def __set_name__(self, owner, name):
        self.fetch = f'SELECT @{name@} FROM @{owner.table@} WHERE @{owner.key@}=?;'
        self.store = f'UPDATE @{owner.table@} SET @{name@}=? WHERE @{owner.key@}=?;'

    def __get__(self, obj, objtype=None):
        return conn.execute(self.fetch, [obj.key]).fetchone()[0]

    def __set__(self, obj, value):
        conn.execute(self.store, [value, obj.key])
        conn.commit()
@end example

We can use the @code{Field} class to define models@footnote{https://en.wikipedia.org/wiki/Database_model} that describe the schema for
each table in a database:

@example
class Movie:
    table = 'Movies'                    # Table name
    key = 'title'                       # Primary key
    director = Field()
    year = Field()

    def __init__(self, key):
        self.key = key

class Song:
    table = 'Music'
    key = 'title'
    artist = Field()
    year = Field()
    genre = Field()

    def __init__(self, key):
        self.key = key
@end example

To use the models, first connect to the database:

@example
>>> import sqlite3
>>> conn = sqlite3.connect('entertainment.db')
@end example

An interactive session shows how data is retrieved from the database and how
it can be updated:

@c song_data = [
@c     ('Country Roads', 'John Denver', 1972),
@c     ('Me and Bobby McGee', 'Janice Joplin', 1971),
@c     ('Coal Miners Daughter', 'Loretta Lynn', 1970),
@c ]
@c 
@c movie_data = [
@c     ('Star Wars', 'George Lucas', 1977),
@c     ('Jaws', 'Steven Spielberg', 1975),
@c     ('Aliens', 'James Cameron', 1986),
@c ]
@c 
@c import sqlite3
@c 
@c conn = sqlite3.connect(':memory:')
@c conn.execute('CREATE TABLE Music (title text, artist text, year integer);')
@c conn.execute('CREATE INDEX MusicNdx ON Music (title);')
@c conn.executemany('INSERT INTO Music VALUES (?, ?, ?);', song_data)
@c conn.execute('CREATE TABLE Movies (title text, director text, year integer);')
@c conn.execute('CREATE INDEX MovieNdx ON Music (title);')
@c conn.executemany('INSERT INTO Movies VALUES (?, ?, ?);', movie_data)
@c conn.commit()

@example
>>> Movie('Star Wars').director
'George Lucas'
>>> jaws = Movie('Jaws')
>>> f'Released in @{jaws.year@} by @{jaws.director@}'
'Released in 1975 by Steven Spielberg'

>>> Song('Country Roads').artist
'John Denver'

>>> Movie('Star Wars').director = 'J.J. Abrams'
>>> Movie('Star Wars').director
'J.J. Abrams'
@end example

@node Pure Python Equivalents,,Technical Tutorial,Descriptor HowTo Guide
@anchor{howto/descriptor pure-python-equivalents}@anchor{4677}
@subsection Pure Python Equivalents


The descriptor protocol is simple and offers exciting possibilities.  Several
use cases are so common that they have been prepackaged into built-in tools.
Properties, bound methods, static methods, class methods, and __slots__ are
all based on the descriptor protocol.

@menu
* Properties:: 
* Functions and methods:: 
* Kinds of methods:: 
* Static methods:: 
* Class methods:: 
* Member objects and __slots__:: 

@end menu

@node Properties,Functions and methods,,Pure Python Equivalents
@anchor{howto/descriptor properties}@anchor{4678}
@subsubsection Properties


Calling @ref{501,,property()} is a succinct way of building a data descriptor that
triggers a function call upon access to an attribute.  Its signature is:

@example
property(fget=None, fset=None, fdel=None, doc=None) -> property
@end example

The documentation shows a typical use to define a managed attribute @code{x}:

@example
class C:
    def getx(self): return self.__x
    def setx(self, value): self.__x = value
    def delx(self): del self.__x
    x = property(getx, setx, delx, "I'm the 'x' property.")
@end example

@c >>> C.x.__doc__
@c "I'm the 'x' property."
@c >>> c.x = 2.71828
@c >>> c.x
@c 2.71828
@c >>> del c.x
@c >>> c.x
@c Traceback (most recent call last):
@c   ...
@c AttributeError: 'C' object has no attribute '_C__x'

To see how @ref{501,,property()} is implemented in terms of the descriptor protocol,
here is a pure Python equivalent:

@example
class Property:
    "Emulate PyProperty_Type() in Objects/descrobject.c"

    def __init__(self, fget=None, fset=None, fdel=None, doc=None):
        self.fget = fget
        self.fset = fset
        self.fdel = fdel
        if doc is None and fget is not None:
            doc = fget.__doc__
        self.__doc__ = doc
        self._name = '@w{'}

    def __set_name__(self, owner, name):
        self._name = name

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        if self.fget is None:
            raise AttributeError(f"property '@{self._name@}' has no getter")
        return self.fget(obj)

    def __set__(self, obj, value):
        if self.fset is None:
            raise AttributeError(f"property '@{self._name@}' has no setter")
        self.fset(obj, value)

    def __delete__(self, obj):
        if self.fdel is None:
            raise AttributeError(f"property '@{self._name@}' has no deleter")
        self.fdel(obj)

    def getter(self, fget):
        prop = type(self)(fget, self.fset, self.fdel, self.__doc__)
        prop._name = self._name
        return prop

    def setter(self, fset):
        prop = type(self)(self.fget, fset, self.fdel, self.__doc__)
        prop._name = self._name
        return prop

    def deleter(self, fdel):
        prop = type(self)(self.fget, self.fset, fdel, self.__doc__)
        prop._name = self._name
        return prop
@end example

@c # Verify the Property() emulation
@c 
@c class CC:
@c     def getx(self):
@c         return self.__x
@c     def setx(self, value):
@c         self.__x = value
@c     def delx(self):
@c         del self.__x
@c     x = Property(getx, setx, delx, "I'm the 'x' property.")
@c 
@c # Now do it again but use the decorator style
@c 
@c class CCC:
@c     @Property
@c     def x(self):
@c         return self.__x
@c     @x.setter
@c     def x(self, value):
@c         self.__x = value
@c     @x.deleter
@c     def x(self):
@c         del self.__x

@c >>> cc = CC()
@c >>> hasattr(cc, 'x')
@c False
@c >>> cc.x = 33
@c >>> cc.x
@c 33
@c >>> del cc.x
@c >>> hasattr(cc, 'x')
@c False
@c 
@c >>> ccc = CCC()
@c >>> hasattr(ccc, 'x')
@c False
@c >>> ccc.x = 333
@c >>> ccc.x == 333
@c True
@c >>> del ccc.x
@c >>> hasattr(ccc, 'x')
@c False

The @ref{501,,property()} builtin helps whenever a user interface has granted
attribute access and then subsequent changes require the intervention of a
method.

For instance, a spreadsheet class may grant access to a cell value through
@code{Cell('b10').value}. Subsequent improvements to the program require the cell
to be recalculated on every access; however, the programmer does not want to
affect existing client code accessing the attribute directly.  The solution is
to wrap access to the value attribute in a property data descriptor:

@example
class Cell:
    ...

    @@property
    def value(self):
        "Recalculate the cell before returning value"
        self.recalc()
        return self._value
@end example

Either the built-in @ref{501,,property()} or our @code{Property()} equivalent would
work in this example.

@node Functions and methods,Kinds of methods,Properties,Pure Python Equivalents
@anchor{howto/descriptor functions-and-methods}@anchor{4679}
@subsubsection Functions and methods


Python’s object oriented features are built upon a function based environment.
Using non-data descriptors, the two are merged seamlessly.

Functions stored in class dictionaries get turned into methods when invoked.
Methods only differ from regular functions in that the object instance is
prepended to the other arguments.  By convention, the instance is called
@emph{self} but could be called @emph{this} or any other variable name.

Methods can be created manually with @ref{1e3b,,types.MethodType} which is
roughly equivalent to:

@example
class MethodType:
    "Emulate PyMethod_Type in Objects/classobject.c"

    def __init__(self, func, obj):
        self.__func__ = func
        self.__self__ = obj

    def __call__(self, *args, **kwargs):
        func = self.__func__
        obj = self.__self__
        return func(obj, *args, **kwargs)
@end example

To support automatic creation of methods, functions include the
@code{__get__()} method for binding methods during attribute access.  This
means that functions are non-data descriptors that return bound methods
during dotted lookup from an instance.  Here’s how it works:

@example
class Function:
    ...

    def __get__(self, obj, objtype=None):
        "Simulate func_descr_get() in Objects/funcobject.c"
        if obj is None:
            return self
        return MethodType(self, obj)
@end example

Running the following class in the interpreter shows how the function
descriptor works in practice:

@example
class D:
    def f(self, x):
         return x
@end example

The function has a @ref{11bf,,qualified name} attribute to support introspection:

@example
>>> D.f.__qualname__
'D.f'
@end example

Accessing the function through the class dictionary does not invoke
@code{__get__()}.  Instead, it just returns the underlying function object:

@example
>>> D.__dict__['f']
<function D.f at 0x00C45070>
@end example

Dotted access from a class calls @code{__get__()} which just returns the
underlying function unchanged:

@example
>>> D.f
<function D.f at 0x00C45070>
@end example

The interesting behavior occurs during dotted access from an instance.  The
dotted lookup calls @code{__get__()} which returns a bound method object:

@example
>>> d = D()
>>> d.f
<bound method D.f of <__main__.D object at 0x00B18C90>>
@end example

Internally, the bound method stores the underlying function and the bound
instance:

@example
>>> d.f.__func__
<function D.f at 0x00C45070>

>>> d.f.__self__
<__main__.D object at 0x1012e1f98>
@end example

If you have ever wondered where @emph{self} comes from in regular methods or where
@emph{cls} comes from in class methods, this is it!

@node Kinds of methods,Static methods,Functions and methods,Pure Python Equivalents
@anchor{howto/descriptor kinds-of-methods}@anchor{467a}
@subsubsection Kinds of methods


Non-data descriptors provide a simple mechanism for variations on the usual
patterns of binding functions into methods.

To recap, functions have a @code{__get__()} method so that they can be converted
to a method when accessed as attributes.  The non-data descriptor transforms an
@code{obj.f(*args)} call into @code{f(obj, *args)}.  Calling @code{cls.f(*args)}
becomes @code{f(*args)}.

This chart summarizes the binding and its two most useful variants:

@quotation


@multitable {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} 
@headitem

Transformation

@tab

Called from an
object

@tab

Called from a
class

@item

function

@tab

f(obj, *args)

@tab

f(*args)

@item

staticmethod

@tab

f(*args)

@tab

f(*args)

@item

classmethod

@tab

f(type(obj), *args)

@tab

f(cls, *args)

@end multitable

@end quotation

@node Static methods,Class methods,Kinds of methods,Pure Python Equivalents
@anchor{howto/descriptor static-methods}@anchor{467b}
@subsubsection Static methods


Static methods return the underlying function without changes.  Calling either
@code{c.f} or @code{C.f} is the equivalent of a direct lookup into
@code{object.__getattribute__(c, "f")} or @code{object.__getattribute__(C, "f")}. As a
result, the function becomes identically accessible from either an object or a
class.

Good candidates for static methods are methods that do not reference the
@code{self} variable.

For instance, a statistics package may include a container class for
experimental data.  The class provides normal methods for computing the average,
mean, median, and other descriptive statistics that depend on the data. However,
there may be useful functions which are conceptually related but do not depend
on the data.  For instance, @code{erf(x)} is handy conversion routine that comes up
in statistical work but does not directly depend on a particular dataset.
It can be called either from an object or the class:  @code{s.erf(1.5) --> .9332} or
@code{Sample.erf(1.5) --> .9332}.

Since static methods return the underlying function with no changes, the
example calls are unexciting:

@example
class E:
    @@staticmethod
    def f(x):
        return x * 10
@end example

@example
>>> E.f(3)
30
>>> E().f(3)
30
@end example

Using the non-data descriptor protocol, a pure Python version of
@ref{253,,staticmethod()} would look like this:

@example
class StaticMethod:
    "Emulate PyStaticMethod_Type() in Objects/funcobject.c"

    def __init__(self, f):
        self.f = f

    def __get__(self, obj, objtype=None):
        return self.f

    def __call__(self, *args, **kwds):
        return self.f(*args, **kwds)
@end example

@c class E_sim:
@c     @StaticMethod
@c     def f(x):
@c         return x * 10
@c 
@c wrapped_ord = StaticMethod(ord)

@c >>> E_sim.f(3)
@c 30
@c >>> E_sim().f(3)
@c 30
@c >>> wrapped_ord('A')
@c 65

@node Class methods,Member objects and __slots__,Static methods,Pure Python Equivalents
@anchor{howto/descriptor class-methods}@anchor{467c}
@subsubsection Class methods


Unlike static methods, class methods prepend the class reference to the
argument list before calling the function.  This format is the same
for whether the caller is an object or a class:

@example
class F:
    @@classmethod
    def f(cls, x):
        return cls.__name__, x
@end example

@example
>>> F.f(3)
('F', 3)
>>> F().f(3)
('F', 3)
@end example

This behavior is useful whenever the method only needs to have a class
reference and does not rely on data stored in a specific instance.  One use for
class methods is to create alternate class constructors.  For example, the
classmethod @ref{1b2d,,dict.fromkeys()} creates a new dictionary from a list of
keys.  The pure Python equivalent is:

@example
class Dict(dict):
    @@classmethod
    def fromkeys(cls, iterable, value=None):
        "Emulate dict_fromkeys() in Objects/dictobject.c"
        d = cls()
        for key in iterable:
            d[key] = value
        return d
@end example

Now a new dictionary of unique keys can be constructed like this:

@example
>>> d = Dict.fromkeys('abracadabra')
>>> type(d) is Dict
True
>>> d
@{'a': None, 'b': None, 'r': None, 'c': None, 'd': None@}
@end example

Using the non-data descriptor protocol, a pure Python version of
@ref{254,,classmethod()} would look like this:

@example
class ClassMethod:
    "Emulate PyClassMethod_Type() in Objects/funcobject.c"

    def __init__(self, f):
        self.f = f

    def __get__(self, obj, cls=None):
        if cls is None:
            cls = type(obj)
        if hasattr(type(self.f), '__get__'):
            return self.f.__get__(cls, cls)
        return MethodType(self.f, cls)
@end example

@c # Verify the emulation works
@c class T:
@c     @ClassMethod
@c     def cm(cls, x, y):
@c         return (cls, x, y)
@c 
@c     @ClassMethod
@c     @property
@c     def __doc__(cls):
@c         return f'A doc for {cls.__name__!r}'

@c >>> T.cm(11, 22)
@c (<class 'T'>, 11, 22)
@c 
@c # Also call it from an instance
@c >>> t = T()
@c >>> t.cm(11, 22)
@c (<class 'T'>, 11, 22)
@c 
@c # Check the alternate path for chained descriptors
@c >>> T.__doc__
@c "A doc for 'T'"

The code path for @code{hasattr(type(self.f), '__get__')} was added in
Python 3.9 and makes it possible for @ref{254,,classmethod()} to support
chained decorators.  For example, a classmethod and property could be
chained together:

@example
class G:
    @@classmethod
    @@property
    def __doc__(cls):
        return f'A doc for @{cls.__name__!r@}'
@end example

@example
>>> G.__doc__
"A doc for 'G'"
@end example

@node Member objects and __slots__,,Class methods,Pure Python Equivalents
@anchor{howto/descriptor member-objects-and-slots}@anchor{467d}
@subsubsection Member objects and __slots__


When a class defines @code{__slots__}, it replaces instance dictionaries with a
fixed-length array of slot values.  From a user point of view that has
several effects:

1. Provides immediate detection of bugs due to misspelled attribute
assignments.  Only attribute names specified in @code{__slots__} are allowed:

@example
class Vehicle:
    __slots__ = ('id_number', 'make', 'model')
@end example

@example
>>> auto = Vehicle()
>>> auto.id_nubmer = 'VYE483814LQEX'
Traceback (most recent call last):
    ...
AttributeError: 'Vehicle' object has no attribute 'id_nubmer'
@end example

2. Helps create immutable objects where descriptors manage access to private
attributes stored in @code{__slots__}:

@example
class Immutable:

    __slots__ = ('_dept', '_name')          # Replace the instance dictionary

    def __init__(self, dept, name):
        self._dept = dept                   # Store to private attribute
        self._name = name                   # Store to private attribute

    @@property                               # Read-only descriptor
    def dept(self):
        return self._dept

    @@property
    def name(self):                         # Read-only descriptor
        return self._name
@end example

@example
>>> mark = Immutable('Botany', 'Mark Watney')
>>> mark.dept
'Botany'
>>> mark.dept = 'Space Pirate'
Traceback (most recent call last):
    ...
AttributeError: property 'dept' of 'Immutable' object has no setter
>>> mark.location = 'Mars'
Traceback (most recent call last):
    ...
AttributeError: 'Immutable' object has no attribute 'location'
@end example

3. Saves memory.  On a 64-bit Linux build, an instance with two attributes
takes 48 bytes with @code{__slots__} and 152 bytes without.  This flyweight design pattern@footnote{https://en.wikipedia.org/wiki/Flyweight_pattern} likely only
matters when a large number of instances are going to be created.

4. Improves speed.  Reading instance variables is 35% faster with
@code{__slots__} (as measured with Python 3.10 on an Apple M1 processor).

5. Blocks tools like @ref{4f4,,functools.cached_property()} which require an
instance dictionary to function correctly:

@example
from functools import cached_property

class CP:
    __slots__ = ()                          # Eliminates the instance dict

    @@cached_property                        # Requires an instance dict
    def pi(self):
        return 4 * sum((-1.0)**n / (2.0*n + 1.0)
                       for n in reversed(range(100_000)))
@end example

@example
>>> CP().pi
Traceback (most recent call last):
  ...
TypeError: No '__dict__' attribute on 'CP' instance to cache 'pi' property.
@end example

It is not possible to create an exact drop-in pure Python version of
@code{__slots__} because it requires direct access to C structures and control
over object memory allocation.  However, we can build a mostly faithful
simulation where the actual C structure for slots is emulated by a private
@code{_slotvalues} list.  Reads and writes to that private structure are managed
by member descriptors:

@example
null = object()

class Member:

    def __init__(self, name, clsname, offset):
        'Emulate PyMemberDef in Include/structmember.h'
        # Also see descr_new() in Objects/descrobject.c
        self.name = name
        self.clsname = clsname
        self.offset = offset

    def __get__(self, obj, objtype=None):
        'Emulate member_get() in Objects/descrobject.c'
        # Also see PyMember_GetOne() in Python/structmember.c
        value = obj._slotvalues[self.offset]
        if value is null:
            raise AttributeError(self.name)
        return value

    def __set__(self, obj, value):
        'Emulate member_set() in Objects/descrobject.c'
        obj._slotvalues[self.offset] = value

    def __delete__(self, obj):
        'Emulate member_delete() in Objects/descrobject.c'
        value = obj._slotvalues[self.offset]
        if value is null:
            raise AttributeError(self.name)
        obj._slotvalues[self.offset] = null

    def __repr__(self):
        'Emulate member_repr() in Objects/descrobject.c'
        return f'<Member @{self.name!r@} of @{self.clsname!r@}>'
@end example

The @code{type.__new__()} method takes care of adding member objects to class
variables:

@example
class Type(type):
    'Simulate how the type metaclass adds member objects for slots'

    def __new__(mcls, clsname, bases, mapping):
        'Emulate type_new() in Objects/typeobject.c'
        # type_new() calls PyTypeReady() which calls add_methods()
        slot_names = mapping.get('slot_names', [])
        for offset, name in enumerate(slot_names):
            mapping[name] = Member(name, clsname, offset)
        return type.__new__(mcls, clsname, bases, mapping)
@end example

The @ref{1794,,object.__new__()} method takes care of creating instances that have
slots instead of an instance dictionary.  Here is a rough simulation in pure
Python:

@example
class Object:
    'Simulate how object.__new__() allocates memory for __slots__'

    def __new__(cls, *args):
        'Emulate object_new() in Objects/typeobject.c'
        inst = super().__new__(cls)
        if hasattr(cls, 'slot_names'):
            empty_slots = [null] * len(cls.slot_names)
            object.__setattr__(inst, '_slotvalues', empty_slots)
        return inst

    def __setattr__(self, name, value):
        'Emulate _PyObject_GenericSetAttrWithDict() Objects/object.c'
        cls = type(self)
        if hasattr(cls, 'slot_names') and name not in cls.slot_names:
            raise AttributeError(
                f'@{type(self).__name__!r@} object has no attribute @{name!r@}'
            )
        super().__setattr__(name, value)

    def __delattr__(self, name):
        'Emulate _PyObject_GenericSetAttrWithDict() Objects/object.c'
        cls = type(self)
        if hasattr(cls, 'slot_names') and name not in cls.slot_names:
            raise AttributeError(
                f'@{type(self).__name__!r@} object has no attribute @{name!r@}'
            )
        super().__delattr__(name)
@end example

To use the simulation in a real class, just inherit from @code{Object} and
set the @ref{17d3,,metaclass} to @code{Type}:

@example
class H(Object, metaclass=Type):
    'Instance variables stored in slots'

    slot_names = ['x', 'y']

    def __init__(self, x, y):
        self.x = x
        self.y = y
@end example

At this point, the metaclass has loaded member objects for @emph{x} and @emph{y}:

@example
>>> from pprint import pp
>>> pp(dict(vars(H)))
@{'__module__': '__main__',
 '__doc__': 'Instance variables stored in slots',
 'slot_names': ['x', 'y'],
 '__init__': <function H.__init__ at 0x7fb5d302f9d0>,
 'x': <Member 'x' of 'H'>,
 'y': <Member 'y' of 'H'>@}
@end example

@c # We test this separately because the preceding section is not
@c # doctestable due to the hex memory address for the __init__ function
@c >>> isinstance(vars(H)['x'], Member)
@c True
@c >>> isinstance(vars(H)['y'], Member)
@c True

When instances are created, they have a @code{slot_values} list where the
attributes are stored:

@example
>>> h = H(10, 20)
>>> vars(h)
@{'_slotvalues': [10, 20]@}
>>> h.x = 55
>>> vars(h)
@{'_slotvalues': [55, 20]@}
@end example

Misspelled or unassigned attributes will raise an exception:

@example
>>> h.xz
Traceback (most recent call last):
    ...
AttributeError: 'H' object has no attribute 'xz'
@end example

@c  # Examples for deleted attributes are not shown because this section
@c  # is already a bit lengthy.  We still test that code here.
@c  >>> del h.x
@c  >>> hasattr(h, 'x')
@c  False
@c 
@c  # Also test the code for uninitialized slots
@c  >>> class HU(Object, metaclass=Type):
@c  ...     slot_names = ['x', 'y']
@c  ...
@c  >>> hu = HU()
@c  >>> hasattr(hu, 'x')
@c  False
@c  >>> hasattr(hu, 'y')
@c  False

@node Enum HOWTO,Functional Programming HOWTO,Descriptor HowTo Guide,Python HOWTOs
@anchor{howto/enum doc}@anchor{467e}@anchor{howto/enum enum-howto}@anchor{467f}
@section Enum HOWTO

@anchor{howto/enum enum-basic-tutorial}@anchor{1e79}
An @ref{658,,Enum} is a set of symbolic names bound to unique values.  They are
similar to global variables, but they offer a more useful @ref{27d,,repr()},
grouping, type-safety, and a few other features.

They are most useful when you have a variable that can take one of a limited
selection of values.  For example, the days of the week:

@example
>>> from enum import Enum
>>> class Weekday(Enum):
...     MONDAY = 1
...     TUESDAY = 2
...     WEDNESDAY = 3
...     THURSDAY = 4
...     FRIDAY = 5
...     SATURDAY = 6
...     SUNDAY = 7
@end example

Or perhaps the RGB primary colors:

@example
>>> from enum import Enum
>>> class Color(Enum):
...     RED = 1
...     GREEN = 2
...     BLUE = 3
@end example

As you can see, creating an @ref{658,,Enum} is as simple as writing a class that
inherits from @ref{658,,Enum} itself.

@cartouche
@quotation Note 
Case of Enum Members

Because Enums are used to represent constants we recommend using
UPPER_CASE names for members, and will be using that style in our examples.
@end quotation
@end cartouche

Depending on the nature of the enum a member’s value may or may not be
important, but either way that value can be used to get the corresponding
member:

@example
>>> Weekday(3)
<Weekday.WEDNESDAY: 3>
@end example

As you can see, the @code{repr()} of a member shows the enum name, the member name,
and the value.  The @code{str()} of a member shows only the enum name and member
name:

@example
>>> print(Weekday.THURSDAY)
Weekday.THURSDAY
@end example

The @emph{type} of an enumeration member is the enum it belongs to:

@example
>>> type(Weekday.MONDAY)
<enum 'Weekday'>
>>> isinstance(Weekday.FRIDAY, Weekday)
True
@end example

Enum members have an attribute that contains just their @code{name}:

@example
>>> print(Weekday.TUESDAY.name)
TUESDAY
@end example

Likewise, they have an attribute for their @code{value}:

@example
>>> Weekday.WEDNESDAY.value
3
@end example

Unlike many languages that treat enumerations solely as name/value pairs,
Python Enums can have behavior added.  For example, @ref{3e2,,datetime.date}
has two methods for returning the weekday: @code{weekday()} and @code{isoweekday()}.
The difference is that one of them counts from 0-6 and the other from 1-7.
Rather than keep track of that ourselves we can add a method to the @code{Weekday}
enum to extract the day from the @code{date} instance and return the matching
enum member:

@example
@@classmethod
def from_date(cls, date):
    return cls(date.isoweekday())
@end example

The complete @code{Weekday} enum now looks like this:

@example
>>> class Weekday(Enum):
...     MONDAY = 1
...     TUESDAY = 2
...     WEDNESDAY = 3
...     THURSDAY = 4
...     FRIDAY = 5
...     SATURDAY = 6
...     SUNDAY = 7
...     #
...     @@classmethod
...     def from_date(cls, date):
...         return cls(date.isoweekday())
@end example

Now we can find out what today is!  Observe:

@example
>>> from datetime import date
>>> Weekday.from_date(date.today())     
<Weekday.TUESDAY: 2>
@end example

Of course, if you’re reading this on some other day, you’ll see that day instead.

This @code{Weekday} enum is great if our variable only needs one day, but
what if we need several?  Maybe we’re writing a function to plot chores during
a week, and don’t want to use a @ref{43a,,list} – we could use a different type
of @ref{658,,Enum}:

@example
>>> from enum import Flag
>>> class Weekday(Flag):
...     MONDAY = 1
...     TUESDAY = 2
...     WEDNESDAY = 4
...     THURSDAY = 8
...     FRIDAY = 16
...     SATURDAY = 32
...     SUNDAY = 64
@end example

We’ve changed two things: we’re inherited from @ref{7e7,,Flag}, and the values are
all powers of 2.

Just like the original @code{Weekday} enum above, we can have a single selection:

@example
>>> first_week_day = Weekday.MONDAY
>>> first_week_day
<Weekday.MONDAY: 1>
@end example

But @ref{7e7,,Flag} also allows us to combine several members into a single
variable:

@example
>>> weekend = Weekday.SATURDAY | Weekday.SUNDAY
>>> weekend
<Weekday.SATURDAY|SUNDAY: 96>
@end example

You can even iterate over a @ref{7e7,,Flag} variable:

@example
>>> for day in weekend:
...     print(day)
Weekday.SATURDAY
Weekday.SUNDAY
@end example

Okay, let’s get some chores set up:

@example
>>> chores_for_ethan = @{
...     'feed the cat': Weekday.MONDAY | Weekday.WEDNESDAY | Weekday.FRIDAY,
...     'do the dishes': Weekday.TUESDAY | Weekday.THURSDAY,
...     'answer SO questions': Weekday.SATURDAY,
...     @}
@end example

And a function to display the chores for a given day:

@example
>>> def show_chores(chores, day):
...     for chore, days in chores.items():
...         if day in days:
...             print(chore)
>>> show_chores(chores_for_ethan, Weekday.SATURDAY)
answer SO questions
@end example

In cases where the actual values of the members do not matter, you can save
yourself some work and use @ref{7e8,,auto()} for the values:

@example
>>> from enum import auto
>>> class Weekday(Flag):
...     MONDAY = auto()
...     TUESDAY = auto()
...     WEDNESDAY = auto()
...     THURSDAY = auto()
...     FRIDAY = auto()
...     SATURDAY = auto()
...     SUNDAY = auto()
@end example

@menu
* Programmatic access to enumeration members and their attributes:: 
* Duplicating enum members and values:: 
* Ensuring unique enumeration values:: 
* Using automatic values:: 
* Iteration: Iteration<2>. 
* Comparisons: Comparisons<3>. 
* Allowed members and attributes of enumerations:: 
* Restricted Enum subclassing:: 
* Pickling:: 
* Functional API: Functional API<2>. 
* Derived Enumerations:: 
* When to use __new__() vs. __init__(): When to use __new__ vs __init__. 
* How are Enums different?:: 
* Subclassing EnumType:: 

@end menu

@node Programmatic access to enumeration members and their attributes,Duplicating enum members and values,,Enum HOWTO
@anchor{howto/enum enum-advanced-tutorial}@anchor{1e7a}@anchor{howto/enum programmatic-access-to-enumeration-members-and-their-attributes}@anchor{4680}
@subsection Programmatic access to enumeration members and their attributes


Sometimes it’s useful to access members in enumerations programmatically (i.e.
situations where @code{Color.RED} won’t do because the exact color is not known
at program-writing time).  @code{Enum} allows such access:

@example
>>> Color(1)
<Color.RED: 1>
>>> Color(3)
<Color.BLUE: 3>
@end example

If you want to access enum members by @emph{name}, use item access:

@example
>>> Color['RED']
<Color.RED: 1>
>>> Color['GREEN']
<Color.GREEN: 2>
@end example

If you have an enum member and need its @code{name} or @code{value}:

@example
>>> member = Color.RED
>>> member.name
'RED'
>>> member.value
1
@end example

@node Duplicating enum members and values,Ensuring unique enumeration values,Programmatic access to enumeration members and their attributes,Enum HOWTO
@anchor{howto/enum duplicating-enum-members-and-values}@anchor{4681}
@subsection Duplicating enum members and values


Having two enum members with the same name is invalid:

@example
>>> class Shape(Enum):
...     SQUARE = 2
...     SQUARE = 3
...
Traceback (most recent call last):
...
TypeError: 'SQUARE' already defined as 2
@end example

However, an enum member can have other names associated with it.  Given two
entries @code{A} and @code{B} with the same value (and @code{A} defined first), @code{B}
is an alias for the member @code{A}.  By-value lookup of the value of @code{A} will
return the member @code{A}.  By-name lookup of @code{A} will return the member @code{A}.
By-name lookup of @code{B} will also return the member @code{A}:

@example
>>> class Shape(Enum):
...     SQUARE = 2
...     DIAMOND = 1
...     CIRCLE = 3
...     ALIAS_FOR_SQUARE = 2
...
>>> Shape.SQUARE
<Shape.SQUARE: 2>
>>> Shape.ALIAS_FOR_SQUARE
<Shape.SQUARE: 2>
>>> Shape(2)
<Shape.SQUARE: 2>
@end example

@cartouche
@quotation Note 
Attempting to create a member with the same name as an already
defined attribute (another member, a method, etc.) or attempting to create
an attribute with the same name as a member is not allowed.
@end quotation
@end cartouche

@node Ensuring unique enumeration values,Using automatic values,Duplicating enum members and values,Enum HOWTO
@anchor{howto/enum ensuring-unique-enumeration-values}@anchor{4682}
@subsection Ensuring unique enumeration values


By default, enumerations allow multiple names as aliases for the same value.
When this behavior isn’t desired, you can use the @ref{1e84,,unique()} decorator:

@example
>>> from enum import Enum, unique
>>> @@unique
... class Mistake(Enum):
...     ONE = 1
...     TWO = 2
...     THREE = 3
...     FOUR = 3
...
Traceback (most recent call last):
...
ValueError: duplicate values found in <enum 'Mistake'>: FOUR -> THREE
@end example

@node Using automatic values,Iteration<2>,Ensuring unique enumeration values,Enum HOWTO
@anchor{howto/enum using-automatic-values}@anchor{4683}
@subsection Using automatic values


If the exact value is unimportant you can use @ref{7e8,,auto}:

@example
>>> from enum import Enum, auto
>>> class Color(Enum):
...     RED = auto()
...     BLUE = auto()
...     GREEN = auto()
...
>>> [member.value for member in Color]
[1, 2, 3]
@end example

The values are chosen by @code{_generate_next_value_()}, which can be
overridden:

@example
>>> class AutoName(Enum):
...     def _generate_next_value_(name, start, count, last_values):
...         return name
...
>>> class Ordinal(AutoName):
...     NORTH = auto()
...     SOUTH = auto()
...     EAST = auto()
...     WEST = auto()
...
>>> [member.value for member in Ordinal]
['NORTH', 'SOUTH', 'EAST', 'WEST']
@end example

@cartouche
@quotation Note 
The @code{_generate_next_value_()} method must be defined before any members.
@end quotation
@end cartouche

@node Iteration<2>,Comparisons<3>,Using automatic values,Enum HOWTO
@anchor{howto/enum iteration}@anchor{4684}
@subsection Iteration


Iterating over the members of an enum does not provide the aliases:

@example
>>> list(Shape)
[<Shape.SQUARE: 2>, <Shape.DIAMOND: 1>, <Shape.CIRCLE: 3>]
@end example

The special attribute @code{__members__} is a read-only ordered mapping of names
to members.  It includes all names defined in the enumeration, including the
aliases:

@example
>>> for name, member in Shape.__members__.items():
...     name, member
...
('SQUARE', <Shape.SQUARE: 2>)
('DIAMOND', <Shape.DIAMOND: 1>)
('CIRCLE', <Shape.CIRCLE: 3>)
('ALIAS_FOR_SQUARE', <Shape.SQUARE: 2>)
@end example

The @code{__members__} attribute can be used for detailed programmatic access to
the enumeration members.  For example, finding all the aliases:

@example
>>> [name for name, member in Shape.__members__.items() if member.name != name]
['ALIAS_FOR_SQUARE']
@end example

@node Comparisons<3>,Allowed members and attributes of enumerations,Iteration<2>,Enum HOWTO
@anchor{howto/enum comparisons}@anchor{4685}
@subsection Comparisons


Enumeration members are compared by identity:

@example
>>> Color.RED is Color.RED
True
>>> Color.RED is Color.BLUE
False
>>> Color.RED is not Color.BLUE
True
@end example

Ordered comparisons between enumeration values are @emph{not} supported.  Enum
members are not integers (but see @ref{4686,,IntEnum} below):

@example
>>> Color.RED < Color.BLUE
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'Color' and 'Color'
@end example

Equality comparisons are defined though:

@example
>>> Color.BLUE == Color.RED
False
>>> Color.BLUE != Color.RED
True
>>> Color.BLUE == Color.BLUE
True
@end example

Comparisons against non-enumeration values will always compare not equal
(again, @ref{828,,IntEnum} was explicitly designed to behave differently, see
below):

@example
>>> Color.BLUE == 2
False
@end example

@node Allowed members and attributes of enumerations,Restricted Enum subclassing,Comparisons<3>,Enum HOWTO
@anchor{howto/enum allowed-members-and-attributes-of-enumerations}@anchor{4687}
@subsection Allowed members and attributes of enumerations


Most of the examples above use integers for enumeration values.  Using integers
is short and handy (and provided by default by the @ref{4688,,Functional API}), but not
strictly enforced.  In the vast majority of use-cases, one doesn’t care what
the actual value of an enumeration is.  But if the value @emph{is} important,
enumerations can have arbitrary values.

Enumerations are Python classes, and can have methods and special methods as
usual.  If we have this enumeration:

@example
>>> class Mood(Enum):
...     FUNKY = 1
...     HAPPY = 3
...
...     def describe(self):
...         # self is the member here
...         return self.name, self.value
...
...     def __str__(self):
...         return 'my custom str! @{0@}'.format(self.value)
...
...     @@classmethod
...     def favorite_mood(cls):
...         # cls here is the enumeration
...         return cls.HAPPY
...
@end example

Then:

@example
>>> Mood.favorite_mood()
<Mood.HAPPY: 3>
>>> Mood.HAPPY.describe()
('HAPPY', 3)
>>> str(Mood.FUNKY)
'my custom str! 1'
@end example

The rules for what is allowed are as follows: names that start and end with
a single underscore are reserved by enum and cannot be used; all other
attributes defined within an enumeration will become members of this
enumeration, with the exception of special methods (@code{__str__()},
@code{__add__()}, etc.), descriptors (methods are also descriptors), and
variable names listed in @code{_ignore_}.

Note:  if your enumeration defines @code{__new__()} and/or @code{__init__()} then
any value(s) given to the enum member will be passed into those methods.
See @ref{4689,,Planet} for an example.

@node Restricted Enum subclassing,Pickling,Allowed members and attributes of enumerations,Enum HOWTO
@anchor{howto/enum restricted-enum-subclassing}@anchor{468a}
@subsection Restricted Enum subclassing


A new @ref{658,,Enum} class must have one base enum class, up to one concrete
data type, and as many @ref{5a5,,object}-based mixin classes as needed.  The
order of these base classes is:

@example
class EnumName([mix-in, ...,] [data-type,] base-enum):
    pass
@end example

Also, subclassing an enumeration is allowed only if the enumeration does not define
any members.  So this is forbidden:

@example
>>> class MoreColor(Color):
...     PINK = 17
...
Traceback (most recent call last):
...
TypeError: <enum 'MoreColor'> cannot extend <enum 'Color'>
@end example

But this is allowed:

@example
>>> class Foo(Enum):
...     def some_behavior(self):
...         pass
...
>>> class Bar(Foo):
...     HAPPY = 1
...     SAD = 2
...
@end example

Allowing subclassing of enums that define members would lead to a violation of
some important invariants of types and instances.  On the other hand, it makes
sense to allow sharing some common behavior between a group of enumerations.
(See @ref{468b,,OrderedEnum} for an example.)

@node Pickling,Functional API<2>,Restricted Enum subclassing,Enum HOWTO
@anchor{howto/enum pickling}@anchor{468c}
@subsection Pickling


Enumerations can be pickled and unpickled:

@example
>>> from test.test_enum import Fruit
>>> from pickle import dumps, loads
>>> Fruit.TOMATO is loads(dumps(Fruit.TOMATO))
True
@end example

The usual restrictions for pickling apply: picklable enums must be defined in
the top level of a module, since unpickling requires them to be importable
from that module.

@cartouche
@quotation Note 
With pickle protocol version 4 it is possible to easily pickle enums
nested in other classes.
@end quotation
@end cartouche

It is possible to modify how enum members are pickled/unpickled by defining
@code{__reduce_ex__()} in the enumeration class.

@node Functional API<2>,Derived Enumerations,Pickling,Enum HOWTO
@anchor{howto/enum functional-api}@anchor{4688}
@subsection Functional API


The @ref{658,,Enum} class is callable, providing the following functional API:

@example
>>> Animal = Enum('Animal', 'ANT BEE CAT DOG')
>>> Animal
<enum 'Animal'>
>>> Animal.ANT
<Animal.ANT: 1>
>>> list(Animal)
[<Animal.ANT: 1>, <Animal.BEE: 2>, <Animal.CAT: 3>, <Animal.DOG: 4>]
@end example

The semantics of this API resemble @ref{3e4,,namedtuple}. The first
argument of the call to @ref{658,,Enum} is the name of the enumeration.

The second argument is the @emph{source} of enumeration member names.  It can be a
whitespace-separated string of names, a sequence of names, a sequence of
2-tuples with key/value pairs, or a mapping (e.g. dictionary) of names to
values.  The last two options enable assigning arbitrary values to
enumerations; the others auto-assign increasing integers starting with 1 (use
the @code{start} parameter to specify a different starting value).  A
new class derived from @ref{658,,Enum} is returned.  In other words, the above
assignment to @code{Animal} is equivalent to:

@example
>>> class Animal(Enum):
...     ANT = 1
...     BEE = 2
...     CAT = 3
...     DOG = 4
...
@end example

The reason for defaulting to @code{1} as the starting number and not @code{0} is
that @code{0} is @code{False} in a boolean sense, but by default enum members all
evaluate to @code{True}.

Pickling enums created with the functional API can be tricky as frame stack
implementation details are used to try and figure out which module the
enumeration is being created in (e.g. it will fail if you use a utility
function in a separate module, and also may not work on IronPython or Jython).
The solution is to specify the module name explicitly as follows:

@example
>>> Animal = Enum('Animal', 'ANT BEE CAT DOG', module=__name__)
@end example

@cartouche
@quotation Warning 
If @code{module} is not supplied, and Enum cannot determine what it is,
the new Enum members will not be unpicklable; to keep errors closer to
the source, pickling will be disabled.
@end quotation
@end cartouche

The new pickle protocol 4 also, in some circumstances, relies on
@ref{1790,,__qualname__} being set to the location where pickle will be able
to find the class.  For example, if the class was made available in class
SomeData in the global scope:

@example
>>> Animal = Enum('Animal', 'ANT BEE CAT DOG', qualname='SomeData.Animal')
@end example

The complete signature is:

@example
Enum(
    value='NewEnumName',
    names=<...>,
    *,
    module='...',
    qualname='...',
    type=<mixed-in class>,
    start=1,
    )
@end example

@*value: 
What the new enum class will record as its name.


@*names: 
The enum members.  This can be a whitespace- or comma-separated string
(values will start at 1 unless otherwise specified):

@example
'RED GREEN BLUE' | 'RED,GREEN,BLUE' | 'RED, GREEN, BLUE'
@end example

or an iterator of names:

@example
['RED', 'GREEN', 'BLUE']
@end example

or an iterator of (name, value) pairs:

@example
[('CYAN', 4), ('MAGENTA', 5), ('YELLOW', 6)]
@end example

or a mapping:

@example
@{'CHARTREUSE': 7, 'SEA_GREEN': 11, 'ROSEMARY': 42@}
@end example


@*module: 
name of module where new enum class can be found.


@*qualname: 
where in module new enum class can be found.


@*type: 
type to mix in to new enum class.


@*start: 
number to start counting at if only names are passed in.



Changed in version 3.5: The @emph{start} parameter was added.


@node Derived Enumerations,When to use __new__ vs __init__,Functional API<2>,Enum HOWTO
@anchor{howto/enum derived-enumerations}@anchor{468d}
@subsection Derived Enumerations


@menu
* IntEnum:: 
* StrEnum:: 
* IntFlag:: 
* Flag:: 
* Others:: 

@end menu

@node IntEnum,StrEnum,,Derived Enumerations
@anchor{howto/enum intenum}@anchor{4686}
@subsubsection IntEnum


The first variation of @ref{658,,Enum} that is provided is also a subclass of
@ref{1b1,,int}.  Members of an @ref{828,,IntEnum} can be compared to integers;
by extension, integer enumerations of different types can also be compared
to each other:

@example
>>> from enum import IntEnum
>>> class Shape(IntEnum):
...     CIRCLE = 1
...     SQUARE = 2
...
>>> class Request(IntEnum):
...     POST = 1
...     GET = 2
...
>>> Shape == 1
False
>>> Shape.CIRCLE == 1
True
>>> Shape.CIRCLE == Request.POST
True
@end example

However, they still can’t be compared to standard @ref{658,,Enum} enumerations:

@example
>>> class Shape(IntEnum):
...     CIRCLE = 1
...     SQUARE = 2
...
>>> class Color(Enum):
...     RED = 1
...     GREEN = 2
...
>>> Shape.CIRCLE == Color.RED
False
@end example

@ref{828,,IntEnum} values behave like integers in other ways you’d expect:

@example
>>> int(Shape.CIRCLE)
1
>>> ['a', 'b', 'c'][Shape.CIRCLE]
'b'
>>> [i for i in range(Shape.SQUARE)]
[0, 1]
@end example

@node StrEnum,IntFlag,IntEnum,Derived Enumerations
@anchor{howto/enum strenum}@anchor{468e}
@subsubsection StrEnum


The second variation of @ref{658,,Enum} that is provided is also a subclass of
@ref{2ef,,str}.  Members of a @ref{27e,,StrEnum} can be compared to strings;
by extension, string enumerations of different types can also be compared
to each other.


New in version 3.11.


@node IntFlag,Flag,StrEnum,Derived Enumerations
@anchor{howto/enum intflag}@anchor{468f}
@subsubsection IntFlag


The next variation of @ref{658,,Enum} provided, @ref{11c2,,IntFlag}, is also based
on @ref{1b1,,int}.  The difference being @ref{11c2,,IntFlag} members can be combined
using the bitwise operators (&, |, ^, ~) and the result is still an
@ref{11c2,,IntFlag} member, if possible.  Like @ref{828,,IntEnum}, @ref{11c2,,IntFlag}
members are also integers and can be used wherever an @ref{1b1,,int} is used.

@cartouche
@quotation Note 
Any operation on an @ref{11c2,,IntFlag} member besides the bit-wise operations will
lose the @ref{11c2,,IntFlag} membership.

Bit-wise operations that result in invalid @ref{11c2,,IntFlag} values will lose the
@ref{11c2,,IntFlag} membership.  See @ref{1e82,,FlagBoundary} for
details.
@end quotation
@end cartouche


New in version 3.6.



Changed in version 3.11.


Sample @ref{11c2,,IntFlag} class:

@example
>>> from enum import IntFlag
>>> class Perm(IntFlag):
...     R = 4
...     W = 2
...     X = 1
...
>>> Perm.R | Perm.W
<Perm.R|W: 6>
>>> Perm.R + Perm.W
6
>>> RW = Perm.R | Perm.W
>>> Perm.R in RW
True
@end example

It is also possible to name the combinations:

@example
>>> class Perm(IntFlag):
...     R = 4
...     W = 2
...     X = 1
...     RWX = 7
>>> Perm.RWX
<Perm.RWX: 7>
>>> ~Perm.RWX
<Perm: 0>
>>> Perm(7)
<Perm.RWX: 7>
@end example

@cartouche
@quotation Note 
Named combinations are considered aliases.  Aliases do not show up during
iteration, but can be returned from by-value lookups.
@end quotation
@end cartouche


Changed in version 3.11.


Another important difference between @ref{11c2,,IntFlag} and @ref{658,,Enum} is that
if no flags are set (the value is 0), its boolean evaluation is @ref{659,,False}:

@example
>>> Perm.R & Perm.X
<Perm: 0>
>>> bool(Perm.R & Perm.X)
False
@end example

Because @ref{11c2,,IntFlag} members are also subclasses of @ref{1b1,,int} they can
be combined with them (but may lose @ref{11c2,,IntFlag} membership:

@example
>>> Perm.X | 4
<Perm.R|X: 5>

>>> Perm.X | 8
9
@end example

@cartouche
@quotation Note 
The negation operator, @code{~}, always returns an @ref{11c2,,IntFlag} member with a
positive value:

@example
>>> (~Perm.X).value == (Perm.R|Perm.W).value == 6
True
@end example
@end quotation
@end cartouche

@ref{11c2,,IntFlag} members can also be iterated over:

@example
>>> list(RW)
[<Perm.R: 4>, <Perm.W: 2>]
@end example


New in version 3.11.


@node Flag,Others,IntFlag,Derived Enumerations
@anchor{howto/enum flag}@anchor{4690}
@subsubsection Flag


The last variation is @ref{7e7,,Flag}.  Like @ref{11c2,,IntFlag}, @ref{7e7,,Flag}
members can be combined using the bitwise operators (&, |, ^, ~).  Unlike
@ref{11c2,,IntFlag}, they cannot be combined with, nor compared against, any
other @ref{7e7,,Flag} enumeration, nor @ref{1b1,,int}.  While it is possible to
specify the values directly it is recommended to use @ref{7e8,,auto} as the
value and let @ref{7e7,,Flag} select an appropriate value.


New in version 3.6.


Like @ref{11c2,,IntFlag}, if a combination of @ref{7e7,,Flag} members results in no
flags being set, the boolean evaluation is @ref{659,,False}:

@example
>>> from enum import Flag, auto
>>> class Color(Flag):
...     RED = auto()
...     BLUE = auto()
...     GREEN = auto()
...
>>> Color.RED & Color.GREEN
<Color: 0>
>>> bool(Color.RED & Color.GREEN)
False
@end example

Individual flags should have values that are powers of two (1, 2, 4, 8, …),
while combinations of flags won’t:

@example
>>> class Color(Flag):
...     RED = auto()
...     BLUE = auto()
...     GREEN = auto()
...     WHITE = RED | BLUE | GREEN
...
>>> Color.WHITE
<Color.WHITE: 7>
@end example

Giving a name to the “no flags set” condition does not change its boolean
value:

@example
>>> class Color(Flag):
...     BLACK = 0
...     RED = auto()
...     BLUE = auto()
...     GREEN = auto()
...
>>> Color.BLACK
<Color.BLACK: 0>
>>> bool(Color.BLACK)
False
@end example

@ref{7e7,,Flag} members can also be iterated over:

@example
>>> purple = Color.RED | Color.BLUE
>>> list(purple)
[<Color.RED: 1>, <Color.BLUE: 2>]
@end example


New in version 3.11.


@cartouche
@quotation Note 
For the majority of new code, @ref{658,,Enum} and @ref{7e7,,Flag} are strongly
recommended, since @ref{828,,IntEnum} and @ref{11c2,,IntFlag} break some
semantic promises of an enumeration (by being comparable to integers, and
thus by transitivity to other unrelated enumerations).  @ref{828,,IntEnum}
and @ref{11c2,,IntFlag} should be used only in cases where @ref{658,,Enum} and
@ref{7e7,,Flag} will not do; for example, when integer constants are replaced
with enumerations, or for interoperability with other systems.
@end quotation
@end cartouche

@node Others,,Flag,Derived Enumerations
@anchor{howto/enum others}@anchor{4691}
@subsubsection Others


While @ref{828,,IntEnum} is part of the @ref{76,,enum} module, it would be very
simple to implement independently:

@example
class IntEnum(int, Enum):
    pass
@end example

This demonstrates how similar derived enumerations can be defined; for example
a @code{FloatEnum} that mixes in @ref{256,,float} instead of @ref{1b1,,int}.

Some rules:


@enumerate 

@item 
When subclassing @ref{658,,Enum}, mix-in types must appear before
@ref{658,,Enum} itself in the sequence of bases, as in the @ref{828,,IntEnum}
example above.

@item 
Mix-in types must be subclassable. For example, @ref{2f7,,bool} and
@ref{438,,range} are not subclassable and will throw an error during Enum
creation if used as the mix-in type.

@item 
While @ref{658,,Enum} can have members of any type, once you mix in an
additional type, all the members must have values of that type, e.g.
@ref{1b1,,int} above.  This restriction does not apply to mix-ins which only
add methods and don’t specify another type.

@item 
When another data type is mixed in, the @code{value} attribute is @emph{not the
same} as the enum member itself, although it is equivalent and will compare
equal.

@item 
%-style formatting:  @code{%s} and @code{%r} call the @ref{658,,Enum} class’s
@code{__str__()} and @code{__repr__()} respectively; other codes (such as
@code{%i} or @code{%h} for IntEnum) treat the enum member as its mixed-in type.

@item 
@ref{4b0,,Formatted string literals}, @ref{784,,str.format()},
and @ref{785,,format()} will use the enum’s @code{__str__()} method.
@end enumerate

@cartouche
@quotation Note 
Because @ref{828,,IntEnum}, @ref{11c2,,IntFlag}, and @ref{27e,,StrEnum} are
designed to be drop-in replacements for existing constants, their
@code{__str__()} method has been reset to their data types
@code{__str__()} method.
@end quotation
@end cartouche

@node When to use __new__ vs __init__,How are Enums different?,Derived Enumerations,Enum HOWTO
@anchor{howto/enum when-to-use-new-vs-init}@anchor{4692}
@subsection When to use @code{__new__()} vs. @code{__init__()}


@code{__new__()} must be used whenever you want to customize the actual value of
the @ref{658,,Enum} member.  Any other modifications may go in either
@code{__new__()} or @code{__init__()}, with @code{__init__()} being preferred.

For example, if you want to pass several items to the constructor, but only
want one of them to be the value:

@example
>>> class Coordinate(bytes, Enum):
...     """
...     Coordinate with binary codes that can be indexed by the int code.
...     """
...     def __new__(cls, value, label, unit):
...         obj = bytes.__new__(cls, [value])
...         obj._value_ = value
...         obj.label = label
...         obj.unit = unit
...         return obj
...     PX = (0, 'P.X', 'km')
...     PY = (1, 'P.Y', 'km')
...     VX = (2, 'V.X', 'km/s')
...     VY = (3, 'V.Y', 'km/s')
...

>>> print(Coordinate['PY'])
Coordinate.PY

>>> print(Coordinate(3))
Coordinate.VY
@end example

@menu
* Finer Points:: 

@end menu

@node Finer Points,,,When to use __new__ vs __init__
@anchor{howto/enum finer-points}@anchor{4693}
@subsubsection Finer Points


@menu
* Supported __dunder__ names: Supported __dunder__ names<2>. 
* Supported _sunder_ names: Supported _sunder_ names<2>. 
* _Private__names:: 
* Enum member type:: 
* Creating members that are mixed with other data types:: 
* Boolean value of Enum classes and members:: 
* Enum classes with methods:: 
* Combining members of Flag:: 
* Flag and IntFlag minutia:: 

@end menu

@node Supported __dunder__ names<2>,Supported _sunder_ names<2>,,Finer Points
@anchor{howto/enum supported-dunder-names}@anchor{4694}
@subsubsection Supported @code{__dunder__} names


@code{__members__} is a read-only ordered mapping of @code{member_name}:@code{member}
items.  It is only available on the class.

@code{__new__()}, if specified, must create and return the enum members; it is
also a very good idea to set the member’s @code{_value_} appropriately.  Once
all the members are created it is no longer used.

@node Supported _sunder_ names<2>,_Private__names,Supported __dunder__ names<2>,Finer Points
@anchor{howto/enum supported-sunder-names}@anchor{4695}
@subsubsection Supported @code{_sunder_} names



@itemize -

@item 
@code{_name_} – name of the member

@item 
@code{_value_} – value of the member; can be set / modified in @code{__new__}

@item 
@code{_missing_} – a lookup function used when a value is not found; may be
overridden

@item 
@code{_ignore_} – a list of names, either as a @ref{43a,,list} or a @ref{2ef,,str},
that will not be transformed into members, and will be removed from the final
class

@item 
@code{_order_} – used in Python 2/3 code to ensure member order is consistent
(class attribute, removed during class creation)

@item 
@code{_generate_next_value_} – used by the @ref{4688,,Functional API} and by
@ref{7e8,,auto} to get an appropriate value for an enum member; may be
overridden
@end itemize

@cartouche
@quotation Note 
For standard @ref{658,,Enum} classes the next value chosen is the last value seen
incremented by one.

For @ref{7e7,,Flag} classes the next value chosen will be the next highest
power-of-two, regardless of the last value seen.
@end quotation
@end cartouche


New in version 3.6: @code{_missing_}, @code{_order_}, @code{_generate_next_value_}



New in version 3.7: @code{_ignore_}


To help keep Python 2 / Python 3 code in sync an @code{_order_} attribute can
be provided.  It will be checked against the actual order of the enumeration
and raise an error if the two do not match:

@example
>>> class Color(Enum):
...     _order_ = 'RED GREEN BLUE'
...     RED = 1
...     BLUE = 3
...     GREEN = 2
...
Traceback (most recent call last):
...
TypeError: member order does not match _order_:
  ['RED', 'BLUE', 'GREEN']
  ['RED', 'GREEN', 'BLUE']
@end example

@cartouche
@quotation Note 
In Python 2 code the @code{_order_} attribute is necessary as definition
order is lost before it can be recorded.
@end quotation
@end cartouche

@node _Private__names,Enum member type,Supported _sunder_ names<2>,Finer Points
@anchor{howto/enum private-names}@anchor{4696}
@subsubsection _Private__names


@ref{187e,,Private names} are not converted to enum members,
but remain normal attributes.


Changed in version 3.11.


@node Enum member type,Creating members that are mixed with other data types,_Private__names,Finer Points
@anchor{howto/enum enum-member-type}@anchor{4697}
@subsubsection @code{Enum} member type


Enum members are instances of their enum class, and are normally accessed as
@code{EnumClass.member}.  In Python versions @code{3.5} to @code{3.10} you could access
members from other members – this practice was discouraged, and in @code{3.11}
@ref{658,,Enum} returns to not allowing it:

@example
>>> class FieldTypes(Enum):
...     name = 0
...     value = 1
...     size = 2
...
>>> FieldTypes.value.size
Traceback (most recent call last):
...
AttributeError: <enum 'FieldTypes'> member has no attribute 'size'
@end example


Changed in version 3.5.



Changed in version 3.11.


@node Creating members that are mixed with other data types,Boolean value of Enum classes and members,Enum member type,Finer Points
@anchor{howto/enum creating-members-that-are-mixed-with-other-data-types}@anchor{4698}
@subsubsection Creating members that are mixed with other data types


When subclassing other data types, such as @ref{1b1,,int} or @ref{2ef,,str}, with
an @ref{658,,Enum}, all values after the @code{=} are passed to that data type’s
constructor.  For example:

@example
>>> class MyEnum(IntEnum):      # help(int) -> int(x, base=10) -> integer
...     example = '11', 16      # so x='11' and base=16
...
>>> MyEnum.example.value        # and hex(11) is...
17
@end example

@node Boolean value of Enum classes and members,Enum classes with methods,Creating members that are mixed with other data types,Finer Points
@anchor{howto/enum boolean-value-of-enum-classes-and-members}@anchor{4699}
@subsubsection Boolean value of @code{Enum} classes and members


Enum classes that are mixed with non-@ref{658,,Enum} types (such as
@ref{1b1,,int}, @ref{2ef,,str}, etc.) are evaluated according to the mixed-in
type’s rules; otherwise, all members evaluate as @ref{748,,True}.  To make your
own enum’s boolean evaluation depend on the member’s value add the following to
your class:

@example
def __bool__(self):
    return bool(self.value)
@end example

Plain @ref{658,,Enum} classes always evaluate as @ref{748,,True}.

@node Enum classes with methods,Combining members of Flag,Boolean value of Enum classes and members,Finer Points
@anchor{howto/enum enum-classes-with-methods}@anchor{469a}
@subsubsection @code{Enum} classes with methods


If you give your enum subclass extra methods, like the @ref{4689,,Planet}
class below, those methods will show up in a @ref{f2f,,dir()} of the member,
but not of the class:

@example
>>> dir(Planet)                         
['EARTH', 'JUPITER', 'MARS', 'MERCURY', 'NEPTUNE', 'SATURN', 'URANUS', 'VENUS', '__class__', '__doc__', '__members__', '__module__']
>>> dir(Planet.EARTH)                   
['__class__', '__doc__', '__module__', 'mass', 'name', 'radius', 'surface_gravity', 'value']
@end example

@node Combining members of Flag,Flag and IntFlag minutia,Enum classes with methods,Finer Points
@anchor{howto/enum combining-members-of-flag}@anchor{469b}
@subsubsection Combining members of @code{Flag}


Iterating over a combination of @ref{7e7,,Flag} members will only return the members that
are comprised of a single bit:

@example
>>> class Color(Flag):
...     RED = auto()
...     GREEN = auto()
...     BLUE = auto()
...     MAGENTA = RED | BLUE
...     YELLOW = RED | GREEN
...     CYAN = GREEN | BLUE
...
>>> Color(3)  # named combination
<Color.YELLOW: 3>
>>> Color(7)      # not named combination
<Color.RED|GREEN|BLUE: 7>
@end example

@node Flag and IntFlag minutia,,Combining members of Flag,Finer Points
@anchor{howto/enum flag-and-intflag-minutia}@anchor{469c}
@subsubsection @code{Flag} and @code{IntFlag} minutia


Using the following snippet for our examples:

@example
>>> class Color(IntFlag):
...     BLACK = 0
...     RED = 1
...     GREEN = 2
...     BLUE = 4
...     PURPLE = RED | BLUE
...     WHITE = RED | GREEN | BLUE
...
@end example

the following are true:


@itemize -

@item 
single-bit flags are canonical

@item 
multi-bit and zero-bit flags are aliases

@item 
only canonical flags are returned during iteration:

@example
>>> list(Color.WHITE)
[<Color.RED: 1>, <Color.GREEN: 2>, <Color.BLUE: 4>]
@end example

@item 
negating a flag or flag set returns a new flag/flag set with the
corresponding positive integer value:

@example
>>> Color.BLUE
<Color.BLUE: 4>

>>> ~Color.BLUE
<Color.RED|GREEN: 3>
@end example

@item 
names of pseudo-flags are constructed from their members’ names:

@example
>>> (Color.RED | Color.GREEN).name
'RED|GREEN'
@end example

@item 
multi-bit flags, aka aliases, can be returned from operations:

@example
>>> Color.RED | Color.BLUE
<Color.PURPLE: 5>

>>> Color(7)  # or Color(-1)
<Color.WHITE: 7>

>>> Color(0)
<Color.BLACK: 0>
@end example

@item 
membership / containment checking: zero-valued flags are always considered
to be contained:

@example
>>> Color.BLACK in Color.WHITE
True
@end example

otherwise, only if all bits of one flag are in the other flag will True
be returned:

@example
>>> Color.PURPLE in Color.WHITE
True

>>> Color.GREEN in Color.PURPLE
False
@end example
@end itemize

There is a new boundary mechanism that controls how out-of-range / invalid
bits are handled: @code{STRICT}, @code{CONFORM}, @code{EJECT}, and @code{KEEP}:

@quotation


@itemize *

@item 
STRICT –> raises an exception when presented with invalid values

@item 
CONFORM –> discards any invalid bits

@item 
EJECT –> lose Flag status and become a normal int with the given value

@item 

@table @asis

@item KEEP –> keep the extra bits


@itemize -

@item 
keeps Flag status and extra bits

@item 
extra bits do not show up in iteration

@item 
extra bits do show up in repr() and str()
@end itemize
@end table
@end itemize
@end quotation

The default for Flag is @code{STRICT}, the default for @code{IntFlag} is @code{EJECT},
and the default for @code{_convert_} is @code{KEEP} (see @code{ssl.Options} for an
example of when @code{KEEP} is needed).

@node How are Enums different?,Subclassing EnumType,When to use __new__ vs __init__,Enum HOWTO
@anchor{howto/enum enum-class-differences}@anchor{1e7c}@anchor{howto/enum how-are-enums-different}@anchor{469d}
@subsection How are Enums different?


Enums have a custom metaclass that affects many aspects of both derived @ref{658,,Enum}
classes and their instances (members).

@menu
* Enum Classes:: 
* Enum Members (aka instances): Enum Members aka instances. 
* Omitting values:: 
* OrderedEnum:: 
* DuplicateFreeEnum:: 
* Planet:: 
* TimePeriod:: 

@end menu

@node Enum Classes,Enum Members aka instances,,How are Enums different?
@anchor{howto/enum enum-classes}@anchor{469e}
@subsubsection Enum Classes


The @ref{1e7e,,EnumType} metaclass is responsible for providing the
@code{__contains__()}, @code{__dir__()}, @code{__iter__()} and other methods that
allow one to do things with an @ref{658,,Enum} class that fail on a typical
class, such as @cite{list(Color)} or @cite{some_enum_var in Color}.  @ref{1e7e,,EnumType} is
responsible for ensuring that various other methods on the final @ref{658,,Enum}
class are correct (such as @code{__new__()}, @code{__getnewargs__()},
@code{__str__()} and @code{__repr__()}).

@node Enum Members aka instances,Omitting values,Enum Classes,How are Enums different?
@anchor{howto/enum enum-members-aka-instances}@anchor{469f}
@subsubsection Enum Members (aka instances)


The most interesting thing about enum members is that they are singletons.
@ref{1e7e,,EnumType} creates them all while it is creating the enum class itself,
and then puts a custom @code{__new__()} in place to ensure that no new ones are
ever instantiated by returning only the existing member instances.
@anchor{howto/enum enum-cookbook}@anchor{1e7b}
While @ref{658,,Enum}, @ref{828,,IntEnum}, @ref{27e,,StrEnum}, @ref{7e7,,Flag}, and
@ref{11c2,,IntFlag} are expected to cover the majority of use-cases, they cannot
cover them all.  Here are recipes for some different types of enumerations
that can be used directly, or as examples for creating one’s own.

@node Omitting values,OrderedEnum,Enum Members aka instances,How are Enums different?
@anchor{howto/enum omitting-values}@anchor{46a0}
@subsubsection Omitting values


In many use-cases, one doesn’t care what the actual value of an enumeration
is. There are several ways to define this type of simple enumeration:


@itemize -

@item 
use instances of @ref{7e8,,auto} for the value

@item 
use instances of @ref{5a5,,object} as the value

@item 
use a descriptive string as the value

@item 
use a tuple as the value and a custom @code{__new__()} to replace the
tuple with an @ref{1b1,,int} value
@end itemize

Using any of these methods signifies to the user that these values are not
important, and also enables one to add, remove, or reorder members without
having to renumber the remaining members.

@menu
* Using auto:: 
* Using object:: 
* Using a descriptive string:: 
* Using a custom __new__(): Using a custom __new__. 

@end menu

@node Using auto,Using object,,Omitting values
@anchor{howto/enum using-auto}@anchor{46a1}
@subsubsection Using @code{auto}


Using @ref{7e8,,auto} would look like:

@example
>>> class Color(Enum):
...     RED = auto()
...     BLUE = auto()
...     GREEN = auto()
...
>>> Color.GREEN
<Color.GREEN: 3>
@end example

@node Using object,Using a descriptive string,Using auto,Omitting values
@anchor{howto/enum using-object}@anchor{46a2}
@subsubsection Using @code{object}


Using @ref{5a5,,object} would look like:

@example
>>> class Color(Enum):
...     RED = object()
...     GREEN = object()
...     BLUE = object()
...
>>> Color.GREEN                         
<Color.GREEN: <object object at 0x...>>
@end example

This is also a good example of why you might want to write your own
@code{__repr__()}:

@example
>>> class Color(Enum):
...     RED = object()
...     GREEN = object()
...     BLUE = object()
...     def __repr__(self):
...         return "<%s.%s>" % (self.__class__.__name__, self._name_)
...
>>> Color.GREEN
<Color.GREEN>
@end example

@node Using a descriptive string,Using a custom __new__,Using object,Omitting values
@anchor{howto/enum using-a-descriptive-string}@anchor{46a3}
@subsubsection Using a descriptive string


Using a string as the value would look like:

@example
>>> class Color(Enum):
...     RED = 'stop'
...     GREEN = 'go'
...     BLUE = 'too fast!'
...
>>> Color.GREEN
<Color.GREEN: 'go'>
@end example

@node Using a custom __new__,,Using a descriptive string,Omitting values
@anchor{howto/enum using-a-custom-new}@anchor{46a4}
@subsubsection Using a custom @code{__new__()}


Using an auto-numbering @code{__new__()} would look like:

@example
>>> class AutoNumber(Enum):
...     def __new__(cls):
...         value = len(cls.__members__) + 1
...         obj = object.__new__(cls)
...         obj._value_ = value
...         return obj
...
>>> class Color(AutoNumber):
...     RED = ()
...     GREEN = ()
...     BLUE = ()
...
>>> Color.GREEN
<Color.GREEN: 2>
@end example

To make a more general purpose @code{AutoNumber}, add @code{*args} to the signature:

@example
>>> class AutoNumber(Enum):
...     def __new__(cls, *args):      # this is the only change from above
...         value = len(cls.__members__) + 1
...         obj = object.__new__(cls)
...         obj._value_ = value
...         return obj
...
@end example

Then when you inherit from @code{AutoNumber} you can write your own @code{__init__}
to handle any extra arguments:

@example
>>> class Swatch(AutoNumber):
...     def __init__(self, pantone='unknown'):
...         self.pantone = pantone
...     AUBURN = '3497'
...     SEA_GREEN = '1246'
...     BLEACHED_CORAL = () # New color, no Pantone code yet!
...
>>> Swatch.SEA_GREEN
<Swatch.SEA_GREEN: 2>
>>> Swatch.SEA_GREEN.pantone
'1246'
>>> Swatch.BLEACHED_CORAL.pantone
'unknown'
@end example

@cartouche
@quotation Note 
The @code{__new__()} method, if defined, is used during creation of the Enum
members; it is then replaced by Enum’s @code{__new__()} which is used after
class creation for lookup of existing members.
@end quotation
@end cartouche

@node OrderedEnum,DuplicateFreeEnum,Omitting values,How are Enums different?
@anchor{howto/enum orderedenum}@anchor{468b}
@subsubsection OrderedEnum


An ordered enumeration that is not based on @ref{828,,IntEnum} and so maintains
the normal @ref{658,,Enum} invariants (such as not being comparable to other
enumerations):

@example
>>> class OrderedEnum(Enum):
...     def __ge__(self, other):
...         if self.__class__ is other.__class__:
...             return self.value >= other.value
...         return NotImplemented
...     def __gt__(self, other):
...         if self.__class__ is other.__class__:
...             return self.value > other.value
...         return NotImplemented
...     def __le__(self, other):
...         if self.__class__ is other.__class__:
...             return self.value <= other.value
...         return NotImplemented
...     def __lt__(self, other):
...         if self.__class__ is other.__class__:
...             return self.value < other.value
...         return NotImplemented
...
>>> class Grade(OrderedEnum):
...     A = 5
...     B = 4
...     C = 3
...     D = 2
...     F = 1
...
>>> Grade.C < Grade.A
True
@end example

@node DuplicateFreeEnum,Planet,OrderedEnum,How are Enums different?
@anchor{howto/enum duplicatefreeenum}@anchor{46a5}
@subsubsection DuplicateFreeEnum


Raises an error if a duplicate member name is found instead of creating an
alias:

@example
>>> class DuplicateFreeEnum(Enum):
...     def __init__(self, *args):
...         cls = self.__class__
...         if any(self.value == e.value for e in cls):
...             a = self.name
...             e = cls(self.value).name
...             raise ValueError(
...                 "aliases not allowed in DuplicateFreeEnum:  %r --> %r"
...                 % (a, e))
...
>>> class Color(DuplicateFreeEnum):
...     RED = 1
...     GREEN = 2
...     BLUE = 3
...     GRENE = 2
...
Traceback (most recent call last):
...
ValueError: aliases not allowed in DuplicateFreeEnum:  'GRENE' --> 'GREEN'
@end example

@cartouche
@quotation Note 
This is a useful example for subclassing Enum to add or change other
behaviors as well as disallowing aliases.  If the only desired change is
disallowing aliases, the @ref{1e84,,unique()} decorator can be used instead.
@end quotation
@end cartouche

@node Planet,TimePeriod,DuplicateFreeEnum,How are Enums different?
@anchor{howto/enum planet}@anchor{4689}
@subsubsection Planet


If @code{__new__()} or @code{__init__()} is defined, the value of the enum member
will be passed to those methods:

@example
>>> class Planet(Enum):
...     MERCURY = (3.303e+23, 2.4397e6)
...     VENUS   = (4.869e+24, 6.0518e6)
...     EARTH   = (5.976e+24, 6.37814e6)
...     MARS    = (6.421e+23, 3.3972e6)
...     JUPITER = (1.9e+27,   7.1492e7)
...     SATURN  = (5.688e+26, 6.0268e7)
...     URANUS  = (8.686e+25, 2.5559e7)
...     NEPTUNE = (1.024e+26, 2.4746e7)
...     def __init__(self, mass, radius):
...         self.mass = mass       # in kilograms
...         self.radius = radius   # in meters
...     @@property
...     def surface_gravity(self):
...         # universal gravitational constant  (m3 kg-1 s-2)
...         G = 6.67300E-11
...         return G * self.mass / (self.radius * self.radius)
...
>>> Planet.EARTH.value
(5.976e+24, 6378140.0)
>>> Planet.EARTH.surface_gravity
9.802652743337129
@end example

@node TimePeriod,,Planet,How are Enums different?
@anchor{howto/enum enum-time-period}@anchor{1e91}@anchor{howto/enum timeperiod}@anchor{46a6}
@subsubsection TimePeriod


An example to show the @code{_ignore_} attribute in use:

@example
>>> from datetime import timedelta
>>> class Period(timedelta, Enum):
...     "different lengths of time"
...     _ignore_ = 'Period i'
...     Period = vars()
...     for i in range(367):
...         Period['day_%d' % i] = i
...
>>> list(Period)[:2]
[<Period.day_0: datetime.timedelta(0)>, <Period.day_1: datetime.timedelta(days=1)>]
>>> list(Period)[-2:]
[<Period.day_365: datetime.timedelta(days=365)>, <Period.day_366: datetime.timedelta(days=366)>]
@end example

@node Subclassing EnumType,,How are Enums different?,Enum HOWTO
@anchor{howto/enum enumtype-examples}@anchor{1e86}@anchor{howto/enum subclassing-enumtype}@anchor{46a7}
@subsection Subclassing EnumType


While most enum needs can be met by customizing @ref{658,,Enum} subclasses,
either with class decorators or custom functions, @ref{1e7e,,EnumType} can be
subclassed to provide a different Enum experience.

@node Functional Programming HOWTO,Logging HOWTO,Enum HOWTO,Python HOWTOs
@anchor{howto/functional doc}@anchor{46a8}@anchor{howto/functional functional-programming-howto}@anchor{46a9}
@section Functional Programming HOWTO


@*Author: 
A. M. Kuchling


@*Release: 
0.32


In this document, we’ll take a tour of Python’s features suitable for
implementing programs in a functional style.  After an introduction to the
concepts of functional programming, we’ll look at language features such as
@ref{11c3,,iterator}s and @ref{bf7,,generator}s and relevant library modules such as
@ref{9f,,itertools} and @ref{7f,,functools}.

@menu
* Introduction: Introduction<14>. 
* Iterators: Iterators<2>. 
* Generator expressions and list comprehensions:: 
* Generators: Generators<2>. 
* Built-in functions:: 
* The itertools module:: 
* The functools module:: 
* Small functions and the lambda expression:: 
* Revision History and Acknowledgements:: 
* References: References<2>. 

@end menu

@node Introduction<14>,Iterators<2>,,Functional Programming HOWTO
@anchor{howto/functional introduction}@anchor{46aa}
@subsection Introduction


This section explains the basic concept of functional programming; if
you’re just interested in learning about Python language features,
skip to the next section on @ref{46ab,,Iterators}.

Programming languages support decomposing problems in several different ways:


@itemize *

@item 
Most programming languages are @strong{procedural}: programs are lists of
instructions that tell the computer what to do with the program’s input.  C,
Pascal, and even Unix shells are procedural languages.

@item 
In @strong{declarative} languages, you write a specification that describes the
problem to be solved, and the language implementation figures out how to
perform the computation efficiently.  SQL is the declarative language you’re
most likely to be familiar with; a SQL query describes the data set you want
to retrieve, and the SQL engine decides whether to scan tables or use indexes,
which subclauses should be performed first, etc.

@item 
@strong{Object-oriented} programs manipulate collections of objects.  Objects have
internal state and support methods that query or modify this internal state in
some way. Smalltalk and Java are object-oriented languages.  C++ and Python
are languages that support object-oriented programming, but don’t force the
use of object-oriented features.

@item 
@strong{Functional} programming decomposes a problem into a set of functions.
Ideally, functions only take inputs and produce outputs, and don’t have any
internal state that affects the output produced for a given input.  Well-known
functional languages include the ML family (Standard ML, OCaml, and other
variants) and Haskell.
@end itemize

The designers of some computer languages choose to emphasize one
particular approach to programming.  This often makes it difficult to
write programs that use a different approach.  Other languages are
multi-paradigm languages that support several different approaches.
Lisp, C++, and Python are multi-paradigm; you can write programs or
libraries that are largely procedural, object-oriented, or functional
in all of these languages.  In a large program, different sections
might be written using different approaches; the GUI might be
object-oriented while the processing logic is procedural or
functional, for example.

In a functional program, input flows through a set of functions. Each function
operates on its input and produces some output.  Functional style discourages
functions with side effects that modify internal state or make other changes
that aren’t visible in the function’s return value.  Functions that have no side
effects at all are called @strong{purely functional}.  Avoiding side effects means
not using data structures that get updated as a program runs; every function’s
output must only depend on its input.

Some languages are very strict about purity and don’t even have assignment
statements such as @code{a=3} or @code{c = a + b}, but it’s difficult to avoid all
side effects, such as printing to the screen or writing to a disk file. Another
example is a call to the @ref{af6,,print()} or @ref{196,,time.sleep()} function, neither
of which returns a useful value. Both are called only for their side effects
of sending some text to the screen or pausing execution for a second.

Python programs written in functional style usually won’t go to the extreme of
avoiding all I/O or all assignments; instead, they’ll provide a
functional-appearing interface but will use non-functional features internally.
For example, the implementation of a function will still use assignments to
local variables, but won’t modify global variables or have other side effects.

Functional programming can be considered the opposite of object-oriented
programming.  Objects are little capsules containing some internal state along
with a collection of method calls that let you modify this state, and programs
consist of making the right set of state changes.  Functional programming wants
to avoid state changes as much as possible and works with data flowing between
functions.  In Python you might combine the two approaches by writing functions
that take and return instances representing objects in your application (e-mail
messages, transactions, etc.).

Functional design may seem like an odd constraint to work under.  Why should you
avoid objects and side effects?  There are theoretical and practical advantages
to the functional style:


@itemize *

@item 
Formal provability.

@item 
Modularity.

@item 
Composability.

@item 
Ease of debugging and testing.
@end itemize

@menu
* Formal provability:: 
* Modularity:: 
* Ease of debugging and testing:: 
* Composability:: 

@end menu

@node Formal provability,Modularity,,Introduction<14>
@anchor{howto/functional formal-provability}@anchor{46ac}
@subsubsection Formal provability


A theoretical benefit is that it’s easier to construct a mathematical proof that
a functional program is correct.

For a long time researchers have been interested in finding ways to
mathematically prove programs correct.  This is different from testing a program
on numerous inputs and concluding that its output is usually correct, or reading
a program’s source code and concluding that the code looks right; the goal is
instead a rigorous proof that a program produces the right result for all
possible inputs.

The technique used to prove programs correct is to write down @strong{invariants},
properties of the input data and of the program’s variables that are always
true.  For each line of code, you then show that if invariants X and Y are true
@strong{before} the line is executed, the slightly different invariants X’ and Y’ are
true @strong{after} the line is executed.  This continues until you reach the end of
the program, at which point the invariants should match the desired conditions
on the program’s output.

Functional programming’s avoidance of assignments arose because assignments are
difficult to handle with this technique; assignments can break invariants that
were true before the assignment without producing any new invariants that can be
propagated onward.

Unfortunately, proving programs correct is largely impractical and not relevant
to Python software. Even trivial programs require proofs that are several pages
long; the proof of correctness for a moderately complicated program would be
enormous, and few or none of the programs you use daily (the Python interpreter,
your XML parser, your web browser) could be proven correct.  Even if you wrote
down or generated a proof, there would then be the question of verifying the
proof; maybe there’s an error in it, and you wrongly believe you’ve proved the
program correct.

@node Modularity,Ease of debugging and testing,Formal provability,Introduction<14>
@anchor{howto/functional modularity}@anchor{46ad}
@subsubsection Modularity


A more practical benefit of functional programming is that it forces you to
break apart your problem into small pieces.  Programs are more modular as a
result.  It’s easier to specify and write a small function that does one thing
than a large function that performs a complicated transformation.  Small
functions are also easier to read and to check for errors.

@node Ease of debugging and testing,Composability,Modularity,Introduction<14>
@anchor{howto/functional ease-of-debugging-and-testing}@anchor{46ae}
@subsubsection Ease of debugging and testing


Testing and debugging a functional-style program is easier.

Debugging is simplified because functions are generally small and clearly
specified.  When a program doesn’t work, each function is an interface point
where you can check that the data are correct.  You can look at the intermediate
inputs and outputs to quickly isolate the function that’s responsible for a bug.

Testing is easier because each function is a potential subject for a unit test.
Functions don’t depend on system state that needs to be replicated before
running a test; instead you only have to synthesize the right input and then
check that the output matches expectations.

@node Composability,,Ease of debugging and testing,Introduction<14>
@anchor{howto/functional composability}@anchor{46af}
@subsubsection Composability


As you work on a functional-style program, you’ll write a number of functions
with varying inputs and outputs.  Some of these functions will be unavoidably
specialized to a particular application, but others will be useful in a wide
variety of programs.  For example, a function that takes a directory path and
returns all the XML files in the directory, or a function that takes a filename
and returns its contents, can be applied to many different situations.

Over time you’ll form a personal library of utilities.  Often you’ll assemble
new programs by arranging existing functions in a new configuration and writing
a few functions specialized for the current task.

@node Iterators<2>,Generator expressions and list comprehensions,Introduction<14>,Functional Programming HOWTO
@anchor{howto/functional functional-howto-iterators}@anchor{46ab}@anchor{howto/functional iterators}@anchor{46b0}
@subsection Iterators


I’ll start by looking at a Python language feature that’s an important
foundation for writing functional-style programs: iterators.

An iterator is an object representing a stream of data; this object returns the
data one element at a time.  A Python iterator must support a method called
@ref{e6d,,__next__()} that takes no arguments and always returns the next
element of the stream.  If there are no more elements in the stream,
@ref{e6d,,__next__()} must raise the @ref{736,,StopIteration} exception.
Iterators don’t have to be finite, though; it’s perfectly reasonable to write
an iterator that produces an infinite stream of data.

The built-in @ref{251,,iter()} function takes an arbitrary object and tries to return
an iterator that will return the object’s contents or elements, raising
@ref{162,,TypeError} if the object doesn’t support iteration.  Several of Python’s
built-in data types support iteration, the most common being lists and
dictionaries.  An object is called @ref{dcc,,iterable} if you can get an iterator
for it.

You can experiment with the iteration interface manually:

@example
>>> L = [1, 2, 3]
>>> it = iter(L)
>>> it  
<...iterator object at ...>
>>> it.__next__()  # same as next(it)
1
>>> next(it)
2
>>> next(it)
3
>>> next(it)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>>
@end example

Python expects iterable objects in several different contexts, the most
important being the @ref{161,,for} statement.  In the statement @code{for X in Y},
Y must be an iterator or some object for which @ref{251,,iter()} can create an
iterator.  These two statements are equivalent:

@example
for i in iter(obj):
    print(i)

for i in obj:
    print(i)
@end example

Iterators can be materialized as lists or tuples by using the @ref{43a,,list()} or
@ref{3e5,,tuple()} constructor functions:

@example
>>> L = [1, 2, 3]
>>> iterator = iter(L)
>>> t = tuple(iterator)
>>> t
(1, 2, 3)
@end example

Sequence unpacking also supports iterators: if you know an iterator will return
N elements, you can unpack them into an N-tuple:

@example
>>> L = [1, 2, 3]
>>> iterator = iter(L)
>>> a, b, c = iterator
>>> a, b, c
(1, 2, 3)
@end example

Built-in functions such as @ref{a7e,,max()} and @ref{a7d,,min()} can take a single
iterator argument and will return the largest or smallest element.  The @code{"in"}
and @code{"not in"} operators also support iterators: @code{X in iterator} is true if
X is found in the stream returned by the iterator.  You’ll run into obvious
problems if the iterator is infinite; @ref{a7e,,max()}, @ref{a7d,,min()}
will never return, and if the element X never appears in the stream, the
@code{"in"} and @code{"not in"} operators won’t return either.

Note that you can only go forward in an iterator; there’s no way to get the
previous element, reset the iterator, or make a copy of it.  Iterator objects
can optionally provide these additional capabilities, but the iterator protocol
only specifies the @ref{e6d,,__next__()} method.  Functions may therefore
consume all of the iterator’s output, and if you need to do something different
with the same stream, you’ll have to create a new iterator.

@menu
* Data Types That Support Iterators:: 

@end menu

@node Data Types That Support Iterators,,,Iterators<2>
@anchor{howto/functional data-types-that-support-iterators}@anchor{46b1}
@subsubsection Data Types That Support Iterators


We’ve already seen how lists and tuples support iterators.  In fact, any Python
sequence type, such as strings, will automatically support creation of an
iterator.

Calling @ref{251,,iter()} on a dictionary returns an iterator that will loop over the
dictionary’s keys:

@example
>>> m = @{'Jan': 1, 'Feb': 2, 'Mar': 3, 'Apr': 4, 'May': 5, 'Jun': 6,
...      'Jul': 7, 'Aug': 8, 'Sep': 9, 'Oct': 10, 'Nov': 11, 'Dec': 12@}
>>> for key in m:
...     print(key, m[key])
Jan 1
Feb 2
Mar 3
Apr 4
May 5
Jun 6
Jul 7
Aug 8
Sep 9
Oct 10
Nov 11
Dec 12
@end example

Note that starting with Python 3.7, dictionary iteration order is guaranteed
to be the same as the insertion order. In earlier versions, the behaviour was
unspecified and could vary between implementations.

Applying @ref{251,,iter()} to a dictionary always loops over the keys, but
dictionaries have methods that return other iterators.  If you want to iterate
over values or key/value pairs, you can explicitly call the
@ref{244,,values()} or @ref{245,,items()} methods to get an appropriate
iterator.

The @ref{3b0,,dict()} constructor can accept an iterator that returns a finite stream
of @code{(key, value)} tuples:

@example
>>> L = [('Italy', 'Rome'), ('France', 'Paris'), ('US', 'Washington DC')]
>>> dict(iter(L))
@{'Italy': 'Rome', 'France': 'Paris', 'US': 'Washington DC'@}
@end example

Files also support iteration by calling the @ref{209b,,readline()}
method until there are no more lines in the file.  This means you can read each
line of a file like this:

@example
for line in file:
    # do something for each line
    ...
@end example

Sets can take their contents from an iterable and let you iterate over the set’s
elements:

@example
S = @{2, 3, 5, 7, 11, 13@}
for i in S:
    print(i)
@end example

@node Generator expressions and list comprehensions,Generators<2>,Iterators<2>,Functional Programming HOWTO
@anchor{howto/functional generator-expressions-and-list-comprehensions}@anchor{46b2}
@subsection Generator expressions and list comprehensions


Two common operations on an iterator’s output are 1) performing some operation
for every element, 2) selecting a subset of elements that meet some condition.
For example, given a list of strings, you might want to strip off trailing
whitespace from each line or extract all the strings containing a given
substring.

List comprehensions and generator expressions (short form: “listcomps” and
“genexps”) are a concise notation for such operations, borrowed from the
functional programming language Haskell (@indicateurl{https://www.haskell.org/}).  You can strip
all the whitespace from a stream of strings with the following code:

@example
line_list = ['  line 1\n', 'line 2  \n', ...]

# Generator expression -- returns iterator
stripped_iter = (line.strip() for line in line_list)

# List comprehension -- returns list
stripped_list = [line.strip() for line in line_list]
@end example

You can select only certain elements by adding an @code{"if"} condition:

@example
stripped_list = [line.strip() for line in line_list
                 if line != ""]
@end example

With a list comprehension, you get back a Python list; @code{stripped_list} is a
list containing the resulting lines, not an iterator.  Generator expressions
return an iterator that computes the values as necessary, not needing to
materialize all the values at once.  This means that list comprehensions aren’t
useful if you’re working with iterators that return an infinite stream or a very
large amount of data.  Generator expressions are preferable in these situations.

Generator expressions are surrounded by parentheses (“()”) and list
comprehensions are surrounded by square brackets (“[]”).  Generator expressions
have the form:

@example
( expression for expr in sequence1
             if condition1
             for expr2 in sequence2
             if condition2
             for expr3 in sequence3 ...
             if condition3
             for exprN in sequenceN
             if conditionN )
@end example

Again, for a list comprehension only the outside brackets are different (square
brackets instead of parentheses).

The elements of the generated output will be the successive values of
@code{expression}.  The @code{if} clauses are all optional; if present, @code{expression}
is only evaluated and added to the result when @code{condition} is true.

Generator expressions always have to be written inside parentheses, but the
parentheses signalling a function call also count.  If you want to create an
iterator that will be immediately passed to a function you can write:

@example
obj_total = sum(obj.count for obj in list_all_objects())
@end example

The @code{for...in} clauses contain the sequences to be iterated over.  The
sequences do not have to be the same length, because they are iterated over from
left to right, @strong{not} in parallel.  For each element in @code{sequence1},
@code{sequence2} is looped over from the beginning.  @code{sequence3} is then looped
over for each resulting pair of elements from @code{sequence1} and @code{sequence2}.

To put it another way, a list comprehension or generator expression is
equivalent to the following Python code:

@example
for expr1 in sequence1:
    if not (condition1):
        continue   # Skip this element
    for expr2 in sequence2:
        if not (condition2):
            continue   # Skip this element
        ...
        for exprN in sequenceN:
            if not (conditionN):
                continue   # Skip this element

            # Output the value of
            # the expression.
@end example

This means that when there are multiple @code{for...in} clauses but no @code{if}
clauses, the length of the resulting output will be equal to the product of the
lengths of all the sequences.  If you have two lists of length 3, the output
list is 9 elements long:

@example
>>> seq1 = 'abc'
>>> seq2 = (1, 2, 3)
>>> [(x, y) for x in seq1 for y in seq2]  
[('a', 1), ('a', 2), ('a', 3),
 ('b', 1), ('b', 2), ('b', 3),
 ('c', 1), ('c', 2), ('c', 3)]
@end example

To avoid introducing an ambiguity into Python’s grammar, if @code{expression} is
creating a tuple, it must be surrounded with parentheses.  The first list
comprehension below is a syntax error, while the second one is correct:

@example
# Syntax error
[x, y for x in seq1 for y in seq2]
# Correct
[(x, y) for x in seq1 for y in seq2]
@end example

@node Generators<2>,Built-in functions,Generator expressions and list comprehensions,Functional Programming HOWTO
@anchor{howto/functional generators}@anchor{46b3}
@subsection Generators


Generators are a special class of functions that simplify the task of writing
iterators.  Regular functions compute a value and return it, but generators
return an iterator that returns a stream of values.

You’re doubtless familiar with how regular function calls work in Python or C.
When you call a function, it gets a private namespace where its local variables
are created.  When the function reaches a @code{return} statement, the local
variables are destroyed and the value is returned to the caller.  A later call
to the same function creates a new private namespace and a fresh set of local
variables. But, what if the local variables weren’t thrown away on exiting a
function?  What if you could later resume the function where it left off?  This
is what generators provide; they can be thought of as resumable functions.

Here’s the simplest example of a generator function:

@example
>>> def generate_ints(N):
...    for i in range(N):
...        yield i
@end example

Any function containing a @ref{4c5,,yield} keyword is a generator function;
this is detected by Python’s @ref{8a8,,bytecode} compiler which compiles the
function specially as a result.

When you call a generator function, it doesn’t return a single value; instead it
returns a generator object that supports the iterator protocol.  On executing
the @code{yield} expression, the generator outputs the value of @code{i}, similar to a
@code{return} statement.  The big difference between @code{yield} and a @code{return}
statement is that on reaching a @code{yield} the generator’s state of execution is
suspended and local variables are preserved.  On the next call to the
generator’s @ref{159d,,__next__()} method, the function will resume
executing.

Here’s a sample usage of the @code{generate_ints()} generator:

@example
>>> gen = generate_ints(3)
>>> gen  
<generator object generate_ints at ...>
>>> next(gen)
0
>>> next(gen)
1
>>> next(gen)
2
>>> next(gen)
Traceback (most recent call last):
  File "stdin", line 1, in <module>
  File "stdin", line 2, in generate_ints
StopIteration
@end example

You could equally write @code{for i in generate_ints(5)}, or @code{a, b, c =
generate_ints(3)}.

Inside a generator function, @code{return value} causes @code{StopIteration(value)}
to be raised from the @ref{159d,,__next__()} method.  Once this happens, or
the bottom of the function is reached, the procession of values ends and the
generator cannot yield any further values.

You could achieve the effect of generators manually by writing your own class
and storing all the local variables of the generator as instance variables.  For
example, returning a list of integers could be done by setting @code{self.count} to
0, and having the @ref{e6d,,__next__()} method increment @code{self.count} and
return it.
However, for a moderately complicated generator, writing a corresponding class
can be much messier.

The test suite included with Python’s library,
Lib/test/test_generators.py@footnote{https://github.com/python/cpython/tree/main/Lib/test/test_generators.py}, contains
a number of more interesting examples.  Here’s one generator that implements an
in-order traversal of a tree using generators recursively.

@example
# A recursive generator that generates Tree leaves in in-order.
def inorder(t):
    if t:
        for x in inorder(t.left):
            yield x

        yield t.label

        for x in inorder(t.right):
            yield x
@end example

Two other examples in @code{test_generators.py} produce solutions for the N-Queens
problem (placing N queens on an NxN chess board so that no queen threatens
another) and the Knight’s Tour (finding a route that takes a knight to every
square of an NxN chessboard without visiting any square twice).

@menu
* Passing values into a generator:: 

@end menu

@node Passing values into a generator,,,Generators<2>
@anchor{howto/functional passing-values-into-a-generator}@anchor{46b4}
@subsubsection Passing values into a generator


In Python 2.4 and earlier, generators only produced output.  Once a generator’s
code was invoked to create an iterator, there was no way to pass any new
information into the function when its execution is resumed.  You could hack
together this ability by making the generator look at a global variable or by
passing in some mutable object that callers then modify, but these approaches
are messy.

In Python 2.5 there’s a simple way to pass values into a generator.
@ref{4c5,,yield} became an expression, returning a value that can be assigned to
a variable or otherwise operated on:

@example
val = (yield i)
@end example

I recommend that you @strong{always} put parentheses around a @code{yield} expression
when you’re doing something with the returned value, as in the above example.
The parentheses aren’t always necessary, but it’s easier to always add them
instead of having to remember when they’re needed.

(
@geindex Python Enhancement Proposals; PEP 342
PEP 342@footnote{https://www.python.org/dev/peps/pep-0342} explains the exact rules, which are that a @code{yield}-expression must
always be parenthesized except when it occurs at the top-level expression on the
right-hand side of an assignment.  This means you can write @code{val = yield i}
but have to use parentheses when there’s an operation, as in @code{val = (yield i)
+ 12}.)

Values are sent into a generator by calling its @ref{181c,,send(value)} method.  This method resumes the generator’s code and the
@code{yield} expression returns the specified value.  If the regular
@ref{159d,,__next__()} method is called, the @code{yield} returns @code{None}.

Here’s a simple counter that increments by 1 and allows changing the value of
the internal counter.

@example
def counter(maximum):
    i = 0
    while i < maximum:
        val = (yield i)
        # If value provided, change counter
        if val is not None:
            i = val
        else:
            i += 1
@end example

And here’s an example of changing the counter:

@example
>>> it = counter(10)  
>>> next(it)  
0
>>> next(it)  
1
>>> it.send(8)  
8
>>> next(it)  
9
>>> next(it)  
Traceback (most recent call last):
  File "t.py", line 15, in <module>
    it.next()
StopIteration
@end example

Because @code{yield} will often be returning @code{None}, you should always check for
this case.  Don’t just use its value in expressions unless you’re sure that the
@ref{181c,,send()} method will be the only method used to resume your
generator function.

In addition to @ref{181c,,send()}, there are two other methods on
generators:


@itemize *

@item 
@ref{11be,,throw(type@comma{} value=None@comma{} traceback=None)} is used to
raise an exception inside the generator; the exception is raised by the
@code{yield} expression where the generator’s execution is paused.

@item 
@ref{181f,,close()} raises a @ref{f2e,,GeneratorExit} exception inside the
generator to terminate the iteration.  On receiving this exception, the
generator’s code must either raise @ref{f2e,,GeneratorExit} or
@ref{736,,StopIteration}; catching the exception and doing anything else is
illegal and will trigger a @ref{5af,,RuntimeError}.  @ref{181f,,close()}
will also be called by Python’s garbage collector when the generator is
garbage-collected.

If you need to run cleanup code when a @ref{f2e,,GeneratorExit} occurs, I suggest
using a @code{try: ... finally:} suite instead of catching @ref{f2e,,GeneratorExit}.
@end itemize

The cumulative effect of these changes is to turn generators from one-way
producers of information into both producers and consumers.

Generators also become @strong{coroutines}, a more generalized form of subroutines.
Subroutines are entered at one point and exited at another point (the top of the
function, and a @code{return} statement), but coroutines can be entered, exited,
and resumed at many different points (the @code{yield} statements).

@node Built-in functions,The itertools module,Generators<2>,Functional Programming HOWTO
@anchor{howto/functional built-in-functions}@anchor{46b5}
@subsection Built-in functions


Let’s look in more detail at built-in functions often used with iterators.

Two of Python’s built-in functions, @ref{2f4,,map()} and @ref{2f5,,filter()} duplicate the
features of generator expressions:


@table @asis

@item @ref{2f4,,map(f@comma{} iterA@comma{} iterB@comma{} ...)} returns an iterator over the sequence

@code{f(iterA[0], iterB[0]), f(iterA[1], iterB[1]), f(iterA[2], iterB[2]), ...}.

@example
>>> def upper(s):
...     return s.upper()
@end example

@example
>>> list(map(upper, ['sentence', 'fragment']))
['SENTENCE', 'FRAGMENT']
>>> [upper(s) for s in ['sentence', 'fragment']]
['SENTENCE', 'FRAGMENT']
@end example
@end table

You can of course achieve the same effect with a list comprehension.

@ref{2f5,,filter(predicate@comma{} iter)} returns an iterator over all the
sequence elements that meet a certain condition, and is similarly duplicated by
list comprehensions.  A @strong{predicate} is a function that returns the truth
value of some condition; for use with @ref{2f5,,filter()}, the predicate must take a
single value.

@example
>>> def is_even(x):
...     return (x % 2) == 0
@end example

@example
>>> list(filter(is_even, range(10)))
[0, 2, 4, 6, 8]
@end example

This can also be written as a list comprehension:

@example
>>> list(x for x in range(10) if is_even(x))
[0, 2, 4, 6, 8]
@end example

@ref{fdc,,enumerate(iter@comma{} start=0)} counts off the elements in the
iterable returning 2-tuples containing the count (from @emph{start}) and
each element.

@example
>>> for item in enumerate(['subject', 'verb', 'object']):
...     print(item)
(0, 'subject')
(1, 'verb')
(2, 'object')
@end example

@ref{fdc,,enumerate()} is often used when looping through a list and recording the
indexes at which certain conditions are met:

@example
f = open('data.txt', 'r')
for i, line in enumerate(f):
    if line.strip() == '@w{'}:
        print('Blank line at line #%i' % i)
@end example

@ref{704,,sorted(iterable@comma{} key=None@comma{} reverse=False)} collects all the
elements of the iterable into a list, sorts the list, and returns the sorted
result.  The @emph{key} and @emph{reverse} arguments are passed through to the
constructed list’s @ref{705,,sort()} method.

@example
>>> import random
>>> # Generate 8 random numbers between [0, 10000)
>>> rand_list = random.sample(range(10000), 8)
>>> rand_list  
[769, 7953, 9828, 6431, 8442, 9878, 6213, 2207]
>>> sorted(rand_list)  
[769, 2207, 6213, 6431, 7953, 8442, 9828, 9878]
>>> sorted(rand_list, reverse=True)  
[9878, 9828, 8442, 7953, 6431, 6213, 2207, 769]
@end example

(For a more detailed discussion of sorting, see the @ref{1a5e,,Sorting HOW TO}.)

The @ref{f7e,,any(iter)} and @ref{f7f,,all(iter)} built-ins look at the
truth values of an iterable’s contents.  @ref{f7e,,any()} returns @code{True} if any element
in the iterable is a true value, and @ref{f7f,,all()} returns @code{True} if all of the
elements are true values:

@example
>>> any([0, 1, 0])
True
>>> any([0, 0, 0])
False
>>> any([1, 1, 1])
True
>>> all([0, 1, 0])
False
>>> all([0, 0, 0])
False
>>> all([1, 1, 1])
True
@end example

@ref{247,,zip(iterA@comma{} iterB@comma{} ...)} takes one element from each iterable and
returns them in a tuple:

@example
zip(['a', 'b', 'c'], (1, 2, 3)) =>
  ('a', 1), ('b', 2), ('c', 3)
@end example

It doesn’t construct an in-memory list and exhaust all the input iterators
before returning; instead tuples are constructed and returned only if they’re
requested.  (The technical term for this behaviour is lazy evaluation@footnote{https://en.wikipedia.org/wiki/Lazy_evaluation}.)

This iterator is intended to be used with iterables that are all of the same
length.  If the iterables are of different lengths, the resulting stream will be
the same length as the shortest iterable.

@example
zip(['a', 'b'], (1, 2, 3)) =>
  ('a', 1), ('b', 2)
@end example

You should avoid doing this, though, because an element may be taken from the
longer iterators and discarded.  This means you can’t go on to use the iterators
further because you risk skipping a discarded element.

@node The itertools module,The functools module,Built-in functions,Functional Programming HOWTO
@anchor{howto/functional the-itertools-module}@anchor{46b6}
@subsection The itertools module


The @ref{9f,,itertools} module contains a number of commonly-used iterators as well
as functions for combining several iterators.  This section will introduce the
module’s contents by showing small examples.

The module’s functions fall into a few broad classes:


@itemize *

@item 
Functions that create a new iterator based on an existing iterator.

@item 
Functions for treating an iterator’s elements as function arguments.

@item 
Functions for selecting portions of an iterator’s output.

@item 
A function for grouping an iterator’s output.
@end itemize

@menu
* Creating new iterators:: 
* Calling functions on elements:: 
* Selecting elements:: 
* Combinatoric functions:: 
* Grouping elements:: 

@end menu

@node Creating new iterators,Calling functions on elements,,The itertools module
@anchor{howto/functional creating-new-iterators}@anchor{46b7}
@subsubsection Creating new iterators


@ref{e33,,itertools.count(start@comma{} step)} returns an infinite
stream of evenly spaced values.  You can optionally supply the starting number,
which defaults to 0, and the interval between numbers, which defaults to 1:

@example
itertools.count() =>
  0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ...
itertools.count(10) =>
  10, 11, 12, 13, 14, 15, 16, 17, 18, 19, ...
itertools.count(10, 5) =>
  10, 15, 20, 25, 30, 35, 40, 45, 50, 55, ...
@end example

@ref{1fde,,itertools.cycle(iter)} saves a copy of the contents of
a provided iterable and returns a new iterator that returns its elements from
first to last.  The new iterator will repeat these elements infinitely.

@example
itertools.cycle([1, 2, 3, 4, 5]) =>
  1, 2, 3, 4, 5, 1, 2, 3, 4, 5, ...
@end example

@ref{1373,,itertools.repeat(elem@comma{} [n])} returns the provided
element @emph{n} times, or returns the element endlessly if @emph{n} is not provided.

@example
itertools.repeat('abc') =>
  abc, abc, abc, abc, abc, abc, abc, abc, abc, abc, ...
itertools.repeat('abc', 5) =>
  abc, abc, abc, abc, abc
@end example

@ref{1a5f,,itertools.chain(iterA@comma{} iterB@comma{} ...)} takes an arbitrary
number of iterables as input, and returns all the elements of the first
iterator, then all the elements of the second, and so on, until all of the
iterables have been exhausted.

@example
itertools.chain(['a', 'b', 'c'], (1, 2, 3)) =>
  a, b, c, 1, 2, 3
@end example

@ref{672,,itertools.islice(iter@comma{} [start]@comma{} stop@comma{} [step])} returns
a stream that’s a slice of the iterator.  With a single @emph{stop} argument, it
will return the first @emph{stop} elements.  If you supply a starting index, you’ll
get @emph{stop-start} elements, and if you supply a value for @emph{step}, elements
will be skipped accordingly.  Unlike Python’s string and list slicing, you can’t
use negative values for @emph{start}, @emph{stop}, or @emph{step}.

@example
itertools.islice(range(10), 8) =>
  0, 1, 2, 3, 4, 5, 6, 7
itertools.islice(range(10), 2, 8) =>
  2, 3, 4, 5, 6, 7
itertools.islice(range(10), 2, 8, 2) =>
  2, 4, 6
@end example

@ref{1fe4,,itertools.tee(iter@comma{} [n])} replicates an iterator; it
returns @emph{n} independent iterators that will all return the contents of the
source iterator.
If you don’t supply a value for @emph{n}, the default is 2.  Replicating iterators
requires saving some of the contents of the source iterator, so this can consume
significant memory if the iterator is large and one of the new iterators is
consumed more than the others.

@example
itertools.tee( itertools.count() ) =>
   iterA, iterB

where iterA ->
   0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ...

and   iterB ->
   0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ...
@end example

@node Calling functions on elements,Selecting elements,Creating new iterators,The itertools module
@anchor{howto/functional calling-functions-on-elements}@anchor{46b8}
@subsubsection Calling functions on elements


The @ref{be,,operator} module contains a set of functions corresponding to Python’s
operators.  Some examples are @ref{2006,,operator.add(a@comma{} b)} (adds
two values), @ref{1ff6,,operator.ne(a@comma{} b)} (same as @code{a != b}), and
@ref{99a,,operator.attrgetter('id')}
(returns a callable that fetches the @code{.id} attribute).

@ref{c76,,itertools.starmap(func@comma{} iter)} assumes that the
iterable will return a stream of tuples, and calls @emph{func} using these tuples as
the arguments:

@example
itertools.starmap(os.path.join,
                  [('/bin', 'python'), ('/usr', 'bin', 'java'),
                   ('/usr', 'bin', 'perl'), ('/usr', 'bin', 'ruby')])
=>
  /bin/python, /usr/bin/java, /usr/bin/perl, /usr/bin/ruby
@end example

@node Selecting elements,Combinatoric functions,Calling functions on elements,The itertools module
@anchor{howto/functional selecting-elements}@anchor{46b9}
@subsubsection Selecting elements


Another group of functions chooses a subset of an iterator’s elements based on a
predicate.

@ref{1a44,,itertools.filterfalse(predicate@comma{} iter)} is the
opposite of @ref{2f5,,filter()}, returning all elements for which the predicate
returns false:

@example
itertools.filterfalse(is_even, itertools.count()) =>
  1, 3, 5, 7, 9, 11, 13, 15, ...
@end example

@ref{1fe3,,itertools.takewhile(predicate@comma{} iter)} returns
elements for as long as the predicate returns true.  Once the predicate returns
false, the iterator will signal the end of its results.

@example
def less_than_10(x):
    return x < 10

itertools.takewhile(less_than_10, itertools.count()) =>
  0, 1, 2, 3, 4, 5, 6, 7, 8, 9

itertools.takewhile(is_even, itertools.count()) =>
  0
@end example

@ref{1fe0,,itertools.dropwhile(predicate@comma{} iter)} discards
elements while the predicate returns true, and then returns the rest of the
iterable’s results.

@example
itertools.dropwhile(less_than_10, itertools.count()) =>
  10, 11, 12, 13, 14, 15, 16, 17, 18, 19, ...

itertools.dropwhile(is_even, itertools.count()) =>
  1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ...
@end example

@ref{e32,,itertools.compress(data@comma{} selectors)} takes two
iterators and returns only those elements of @emph{data} for which the corresponding
element of @emph{selectors} is true, stopping whenever either one is exhausted:

@example
itertools.compress([1, 2, 3, 4, 5], [True, True, False, False, True]) =>
   1, 2, 5
@end example

@node Combinatoric functions,Grouping elements,Selecting elements,The itertools module
@anchor{howto/functional combinatoric-functions}@anchor{46ba}
@subsubsection Combinatoric functions


The @ref{ea8,,itertools.combinations(iterable@comma{} r)}
returns an iterator giving all possible @emph{r}-tuple combinations of the
elements contained in @emph{iterable}.

@example
itertools.combinations([1, 2, 3, 4, 5], 2) =>
  (1, 2), (1, 3), (1, 4), (1, 5),
  (2, 3), (2, 4), (2, 5),
  (3, 4), (3, 5),
  (4, 5)

itertools.combinations([1, 2, 3, 4, 5], 3) =>
  (1, 2, 3), (1, 2, 4), (1, 2, 5), (1, 3, 4), (1, 3, 5), (1, 4, 5),
  (2, 3, 4), (2, 3, 5), (2, 4, 5),
  (3, 4, 5)
@end example

The elements within each tuple remain in the same order as
@emph{iterable} returned them.  For example, the number 1 is always before
2, 3, 4, or 5 in the examples above.  A similar function,
@ref{1193,,itertools.permutations(iterable@comma{} r=None)},
removes this constraint on the order, returning all possible
arrangements of length @emph{r}:

@example
itertools.permutations([1, 2, 3, 4, 5], 2) =>
  (1, 2), (1, 3), (1, 4), (1, 5),
  (2, 1), (2, 3), (2, 4), (2, 5),
  (3, 1), (3, 2), (3, 4), (3, 5),
  (4, 1), (4, 2), (4, 3), (4, 5),
  (5, 1), (5, 2), (5, 3), (5, 4)

itertools.permutations([1, 2, 3, 4, 5]) =>
  (1, 2, 3, 4, 5), (1, 2, 3, 5, 4), (1, 2, 4, 3, 5),
  ...
  (5, 4, 3, 2, 1)
@end example

If you don’t supply a value for @emph{r} the length of the iterable is used,
meaning that all the elements are permuted.

Note that these functions produce all of the possible combinations by
position and don’t require that the contents of @emph{iterable} are unique:

@example
itertools.permutations('aba', 3) =>
  ('a', 'b', 'a'), ('a', 'a', 'b'), ('b', 'a', 'a'),
  ('b', 'a', 'a'), ('a', 'a', 'b'), ('a', 'b', 'a')
@end example

The identical tuple @code{('a', 'a', 'b')} occurs twice, but the two ‘a’
strings came from different positions.

The @ref{e31,,itertools.combinations_with_replacement(iterable@comma{} r)}
function relaxes a different constraint: elements can be repeated
within a single tuple.  Conceptually an element is selected for the
first position of each tuple and then is replaced before the second
element is selected.

@example
itertools.combinations_with_replacement([1, 2, 3, 4, 5], 2) =>
  (1, 1), (1, 2), (1, 3), (1, 4), (1, 5),
  (2, 2), (2, 3), (2, 4), (2, 5),
  (3, 3), (3, 4), (3, 5),
  (4, 4), (4, 5),
  (5, 5)
@end example

@node Grouping elements,,Combinatoric functions,The itertools module
@anchor{howto/functional grouping-elements}@anchor{46bb}
@subsubsection Grouping elements


The last function I’ll discuss, @ref{1fe1,,itertools.groupby(iter@comma{} key_func=None)}, is the most complicated.  @code{key_func(elem)} is a function
that can compute a key value for each element returned by the iterable.  If you
don’t supply a key function, the key is simply each element itself.

@ref{1fe1,,groupby()} collects all the consecutive elements from the
underlying iterable that have the same key value, and returns a stream of
2-tuples containing a key value and an iterator for the elements with that key.

@example
city_list = [('Decatur', 'AL'), ('Huntsville', 'AL'), ('Selma', 'AL'),
             ('Anchorage', 'AK'), ('Nome', 'AK'),
             ('Flagstaff', 'AZ'), ('Phoenix', 'AZ'), ('Tucson', 'AZ'),
             ...
            ]

def get_state(city_state):
    return city_state[1]

itertools.groupby(city_list, get_state) =>
  ('AL', iterator-1),
  ('AK', iterator-2),
  ('AZ', iterator-3), ...

where
iterator-1 =>
  ('Decatur', 'AL'), ('Huntsville', 'AL'), ('Selma', 'AL')
iterator-2 =>
  ('Anchorage', 'AK'), ('Nome', 'AK')
iterator-3 =>
  ('Flagstaff', 'AZ'), ('Phoenix', 'AZ'), ('Tucson', 'AZ')
@end example

@ref{1fe1,,groupby()} assumes that the underlying iterable’s contents will
already be sorted based on the key.  Note that the returned iterators also use
the underlying iterable, so you have to consume the results of iterator-1 before
requesting iterator-2 and its corresponding key.

@node The functools module,Small functions and the lambda expression,The itertools module,Functional Programming HOWTO
@anchor{howto/functional the-functools-module}@anchor{46bc}
@subsection The functools module


The @ref{7f,,functools} module in Python 2.5 contains some higher-order functions.
A @strong{higher-order function} takes one or more functions as input and returns a
new function.  The most useful tool in this module is the
@ref{a3f,,functools.partial()} function.

For programs written in a functional style, you’ll sometimes want to construct
variants of existing functions that have some of the parameters filled in.
Consider a Python function @code{f(a, b, c)}; you may wish to create a new function
@code{g(b, c)} that’s equivalent to @code{f(1, b, c)}; you’re filling in a value for
one of @code{f()}’s parameters.  This is called “partial function application”.

The constructor for @ref{a3f,,partial()} takes the arguments
@code{(function, arg1, arg2, ..., kwarg1=value1, kwarg2=value2)}.  The resulting
object is callable, so you can just call it to invoke @code{function} with the
filled-in arguments.

Here’s a small but realistic example:

@example
import functools

def log(message, subsystem):
    """Write the contents of 'message' to the specified subsystem."""
    print('%s: %s' % (subsystem, message))
    ...

server_log = functools.partial(log, subsystem='server')
server_log('Unable to open socket')
@end example

@ref{e77,,functools.reduce(func@comma{} iter@comma{} [initial_value])}
cumulatively performs an operation on all the iterable’s elements and,
therefore, can’t be applied to infinite iterables. @emph{func} must be a function
that takes two elements and returns a single value.  @ref{e77,,functools.reduce()}
takes the first two elements A and B returned by the iterator and calculates
@code{func(A, B)}.  It then requests the third element, C, calculates
@code{func(func(A, B), C)}, combines this result with the fourth element returned,
and continues until the iterable is exhausted.  If the iterable returns no
values at all, a @ref{162,,TypeError} exception is raised.  If the initial value is
supplied, it’s used as a starting point and @code{func(initial_value, A)} is the
first calculation.

@example
>>> import operator, functools
>>> functools.reduce(operator.concat, ['A', 'BB', 'C'])
'ABBC'
>>> functools.reduce(operator.concat, [])
Traceback (most recent call last):
  ...
TypeError: reduce() of empty sequence with no initial value
>>> functools.reduce(operator.mul, [1, 2, 3], 1)
6
>>> functools.reduce(operator.mul, [], 1)
1
@end example

If you use @ref{2006,,operator.add()} with @ref{e77,,functools.reduce()}, you’ll add up all the
elements of the iterable.  This case is so common that there’s a special
built-in called @ref{50d,,sum()} to compute it:

@example
>>> import functools, operator
>>> functools.reduce(operator.add, [1, 2, 3, 4], 0)
10
>>> sum([1, 2, 3, 4])
10
>>> sum([])
0
@end example

For many uses of @ref{e77,,functools.reduce()}, though, it can be clearer to just
write the obvious @ref{161,,for} loop:

@example
import functools
# Instead of:
product = functools.reduce(operator.mul, [1, 2, 3], 1)

# You can write:
product = 1
for i in [1, 2, 3]:
    product *= i
@end example

A related function is @ref{505,,itertools.accumulate(iterable@comma{} func=operator.add)}.  It performs the same calculation, but instead of
returning only the final result, @code{accumulate()} returns an iterator that
also yields each partial result:

@example
itertools.accumulate([1, 2, 3, 4, 5]) =>
  1, 3, 6, 10, 15

itertools.accumulate([1, 2, 3, 4, 5], operator.mul) =>
  1, 2, 6, 24, 120
@end example

@menu
* The operator module:: 

@end menu

@node The operator module,,,The functools module
@anchor{howto/functional the-operator-module}@anchor{46bd}
@subsubsection The operator module


The @ref{be,,operator} module was mentioned earlier.  It contains a set of
functions corresponding to Python’s operators.  These functions are often useful
in functional-style code because they save you from writing trivial functions
that perform a single operation.

Some of the functions in this module are:


@itemize *

@item 
Math operations: @code{add()}, @code{sub()}, @code{mul()}, @code{floordiv()}, @code{abs()}, …

@item 
Logical operations: @code{not_()}, @code{truth()}.

@item 
Bitwise operations: @code{and_()}, @code{or_()}, @code{invert()}.

@item 
Comparisons: @code{eq()}, @code{ne()}, @code{lt()}, @code{le()}, @code{gt()}, and @code{ge()}.

@item 
Object identity: @code{is_()}, @code{is_not()}.
@end itemize

Consult the operator module’s documentation for a complete list.

@node Small functions and the lambda expression,Revision History and Acknowledgements,The functools module,Functional Programming HOWTO
@anchor{howto/functional small-functions-and-the-lambda-expression}@anchor{46be}
@subsection Small functions and the lambda expression


When writing functional-style programs, you’ll often need little functions that
act as predicates or that combine elements in some way.

If there’s a Python built-in or a module function that’s suitable, you don’t
need to define a new function at all:

@example
stripped_lines = [line.strip() for line in lines]
existing_files = filter(os.path.exists, file_list)
@end example

If the function you need doesn’t exist, you need to write it.  One way to write
small functions is to use the @ref{e42,,lambda} expression.  @code{lambda} takes a
number of parameters and an expression combining these parameters, and creates
an anonymous function that returns the value of the expression:

@example
adder = lambda x, y: x+y

print_assign = lambda name, value: name + '=' + str(value)
@end example

An alternative is to just use the @code{def} statement and define a function in the
usual way:

@example
def adder(x, y):
    return x + y

def print_assign(name, value):
    return name + '=' + str(value)
@end example

Which alternative is preferable?  That’s a style question; my usual course is to
avoid using @code{lambda}.

One reason for my preference is that @code{lambda} is quite limited in the
functions it can define.  The result has to be computable as a single
expression, which means you can’t have multiway @code{if... elif... else}
comparisons or @code{try... except} statements.  If you try to do too much in a
@code{lambda} statement, you’ll end up with an overly complicated expression that’s
hard to read.  Quick, what’s the following code doing?

@example
import functools
total = functools.reduce(lambda a, b: (0, a[1] + b[1]), items)[1]
@end example

You can figure it out, but it takes time to disentangle the expression to figure
out what’s going on.  Using a short nested @code{def} statements makes things a
little bit better:

@example
import functools
def combine(a, b):
    return 0, a[1] + b[1]

total = functools.reduce(combine, items)[1]
@end example

But it would be best of all if I had simply used a @code{for} loop:

@example
total = 0
for a, b in items:
    total += b
@end example

Or the @ref{50d,,sum()} built-in and a generator expression:

@example
total = sum(b for a, b in items)
@end example

Many uses of @ref{e77,,functools.reduce()} are clearer when written as @code{for} loops.

Fredrik Lundh once suggested the following set of rules for refactoring uses of
@code{lambda}:


@enumerate 

@item 
Write a lambda function.

@item 
Write a comment explaining what the heck that lambda does.

@item 
Study the comment for a while, and think of a name that captures the essence
of the comment.

@item 
Convert the lambda to a def statement, using that name.

@item 
Remove the comment.
@end enumerate

I really like these rules, but you’re free to disagree
about whether this lambda-free style is better.

@node Revision History and Acknowledgements,References<2>,Small functions and the lambda expression,Functional Programming HOWTO
@anchor{howto/functional revision-history-and-acknowledgements}@anchor{46bf}
@subsection Revision History and Acknowledgements


The author would like to thank the following people for offering suggestions,
corrections and assistance with various drafts of this article: Ian Bicking,
Nick Coghlan, Nick Efford, Raymond Hettinger, Jim Jewett, Mike Krell, Leandro
Lameiro, Jussi Salmela, Collin Winter, Blake Winton.

Version 0.1: posted June 30 2006.

Version 0.11: posted July 1 2006.  Typo fixes.

Version 0.2: posted July 10 2006.  Merged genexp and listcomp sections into one.
Typo fixes.

Version 0.21: Added more references suggested on the tutor mailing list.

Version 0.30: Adds a section on the @code{functional} module written by Collin
Winter; adds short section on the operator module; a few other edits.

@node References<2>,,Revision History and Acknowledgements,Functional Programming HOWTO
@anchor{howto/functional references}@anchor{46c0}
@subsection References


@menu
* General:: 
* Python-specific:: 
* Python documentation:: 

@end menu

@node General,Python-specific,,References<2>
@anchor{howto/functional general}@anchor{46c1}
@subsubsection General


@strong{Structure and Interpretation of Computer Programs}, by Harold Abelson and
Gerald Jay Sussman with Julie Sussman.  Full text at
@indicateurl{https://mitpress.mit.edu/sicp/}.  In this classic textbook of computer science,
chapters 2 and 3 discuss the use of sequences and streams to organize the data
flow inside a program.  The book uses Scheme for its examples, but many of the
design approaches described in these chapters are applicable to functional-style
Python code.

@indicateurl{http://www.defmacro.org/ramblings/fp.html}: A general introduction to functional
programming that uses Java examples and has a lengthy historical introduction.

@indicateurl{https://en.wikipedia.org/wiki/Functional_programming}: General Wikipedia entry
describing functional programming.

@indicateurl{https://en.wikipedia.org/wiki/Coroutine}: Entry for coroutines.

@indicateurl{https://en.wikipedia.org/wiki/Currying}: Entry for the concept of currying.

@node Python-specific,Python documentation,General,References<2>
@anchor{howto/functional python-specific}@anchor{46c2}
@subsubsection Python-specific


@indicateurl{http://gnosis.cx/TPiP/}: The first chapter of David Mertz’s book
@cite{Text Processing in Python} discusses functional programming
for text processing, in the section titled “Utilizing Higher-Order Functions in
Text Processing”.

Mertz also wrote a 3-part series of articles on functional programming
for IBM’s DeveloperWorks site; see
part 1@footnote{https://developer.ibm.com/articles/l-prog/},
part 2@footnote{https://developer.ibm.com/tutorials/l-prog2/}, and
part 3@footnote{https://developer.ibm.com/tutorials/l-prog3/},

@node Python documentation,,Python-specific,References<2>
@anchor{howto/functional python-documentation}@anchor{46c3}
@subsubsection Python documentation


Documentation for the @ref{9f,,itertools} module.

Documentation for the @ref{7f,,functools} module.

Documentation for the @ref{be,,operator} module.

@geindex Python Enhancement Proposals; PEP 289
PEP 289@footnote{https://www.python.org/dev/peps/pep-0289}: “Generator Expressions”

@geindex Python Enhancement Proposals; PEP 342
PEP 342@footnote{https://www.python.org/dev/peps/pep-0342}: “Coroutines via Enhanced Generators” describes the new generator
features in Python 2.5.

@c comment
@c 
@c Handy little function for printing part of an iterator -- used
@c while writing this document.
@c 
@c import itertools
@c def print_iter(it):
@c      slice = itertools.islice(it, 10)
@c      for elem in slice[:-1]:
@c          sys.stdout.write(str(elem))
@c          sys.stdout.write(', ')
@c     print(elem[-1])

@node Logging HOWTO,Logging Cookbook,Functional Programming HOWTO,Python HOWTOs
@anchor{howto/logging doc}@anchor{46c4}@anchor{howto/logging logging-howto}@anchor{46c5}
@section Logging HOWTO


@*Author: 
Vinay Sajip <vinay_sajip at red-dove dot com>


@menu
* Basic Logging Tutorial:: 
* Advanced Logging Tutorial:: 
* Logging Levels: Logging Levels<2>. 
* Useful Handlers:: 
* Exceptions raised during logging:: 
* Using arbitrary objects as messages:: 
* Optimization:: 

@end menu

@node Basic Logging Tutorial,Advanced Logging Tutorial,,Logging HOWTO
@anchor{howto/logging basic-logging-tutorial}@anchor{46c6}@anchor{howto/logging logging-basic-tutorial}@anchor{d97}
@subsection Basic Logging Tutorial


Logging is a means of tracking events that happen when some software runs. The
software’s developer adds logging calls to their code to indicate that certain
events have occurred. An event is described by a descriptive message which can
optionally contain variable data (i.e. data that is potentially different for
each occurrence of the event). Events also have an importance which the
developer ascribes to the event; the importance can also be called the @emph{level}
or @emph{severity}.

@menu
* When to use logging:: 
* A simple example:: 
* Logging to a file:: 
* Logging from multiple modules:: 
* Logging variable data:: 
* Changing the format of displayed messages:: 
* Displaying the date/time in messages:: 
* Next Steps:: 

@end menu

@node When to use logging,A simple example,,Basic Logging Tutorial
@anchor{howto/logging when-to-use-logging}@anchor{46c7}
@subsubsection When to use logging


Logging provides a set of convenience functions for simple logging usage. These
are @ref{2523,,debug()}, @ref{254d,,info()}, @ref{255f,,warning()}, @ref{2560,,error()} and
@ref{2561,,critical()}. To determine when to use logging, see the table below, which
states, for each of a set of common tasks, the best tool to use for it.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Task you want to perform

@tab

The best tool for the task

@item

Display console output for ordinary
usage of a command line script or
program

@tab

@ref{af6,,print()}

@item

Report events that occur during
normal operation of a program (e.g.
for status monitoring or fault
investigation)

@tab

@ref{254d,,logging.info()} (or
@ref{2523,,logging.debug()} for very
detailed output for diagnostic
purposes)

@item

Issue a warning regarding a
particular runtime event

@tab

@ref{104d,,warnings.warn()} in library
code if the issue is avoidable and
the client application should be
modified to eliminate the warning

@ref{255f,,logging.warning()} if there is
nothing the client application can do
about the situation, but the event
should still be noted

@item

Report an error regarding a
particular runtime event

@tab

Raise an exception

@item

Report suppression of an error
without raising an exception (e.g.
error handler in a long-running
server process)

@tab

@ref{2560,,logging.error()},
@ref{2562,,logging.exception()} or
@ref{2561,,logging.critical()} as
appropriate for the specific error
and application domain

@end multitable


The logging functions are named after the level or severity of the events
they are used to track. The standard levels and their applicability are
described below (in increasing order of severity):


@multitable {xxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Level

@tab

When it’s used

@item

@code{DEBUG}

@tab

Detailed information, typically of interest
only when diagnosing problems.

@item

@code{INFO}

@tab

Confirmation that things are working as
expected.

@item

@code{WARNING}

@tab

An indication that something unexpected
happened, or indicative of some problem in
the near future (e.g. ‘disk space low’).
The software is still working as expected.

@item

@code{ERROR}

@tab

Due to a more serious problem, the software
has not been able to perform some function.

@item

@code{CRITICAL}

@tab

A serious error, indicating that the program
itself may be unable to continue running.

@end multitable


The default level is @code{WARNING}, which means that only events of this level
and above will be tracked, unless the logging package is configured to do
otherwise.

Events that are tracked can be handled in different ways. The simplest way of
handling tracked events is to print them to the console. Another common way
is to write them to a disk file.

@node A simple example,Logging to a file,When to use logging,Basic Logging Tutorial
@anchor{howto/logging a-simple-example}@anchor{46c8}@anchor{howto/logging howto-minimal-example}@anchor{46c9}
@subsubsection A simple example


A very simple example is:

@example
import logging
logging.warning('Watch out!')  # will print a message to the console
logging.info('I told you so')  # will not print anything
@end example

If you type these lines into a script and run it, you’ll see:

@example
WARNING:root:Watch out!
@end example

printed out on the console. The @code{INFO} message doesn’t appear because the
default level is @code{WARNING}. The printed message includes the indication of
the level and the description of the event provided in the logging call, i.e.
‘Watch out!’. Don’t worry about the ‘root’ part for now: it will be explained
later. The actual output can be formatted quite flexibly if you need that;
formatting options will also be explained later.

@node Logging to a file,Logging from multiple modules,A simple example,Basic Logging Tutorial
@anchor{howto/logging logging-to-a-file}@anchor{46ca}
@subsubsection Logging to a file


A very common situation is that of recording logging events in a file, so let’s
look at that next. Be sure to try the following in a newly-started Python
interpreter, and don’t just continue from the session described above:

@example
import logging
logging.basicConfig(filename='example.log', encoding='utf-8', level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
logging.error('And non-ASCII stuff, too, like Øresund and Malmö')
@end example


Changed in version 3.9: The @emph{encoding} argument was added. In earlier Python versions, or if not
specified, the encoding used is the default value used by @ref{1c8,,open()}. While
not shown in the above example, an @emph{errors} argument can also now be passed,
which determines how encoding errors are handled. For available values and
the default, see the documentation for @ref{1c8,,open()}.


And now if we open the file and look at what we have, we should find the log
messages:

@example
DEBUG:root:This message should go to the log file
INFO:root:So should this
WARNING:root:And this, too
ERROR:root:And non-ASCII stuff, too, like Øresund and Malmö
@end example

This example also shows how you can set the logging level which acts as the
threshold for tracking. In this case, because we set the threshold to
@code{DEBUG}, all of the messages were printed.

If you want to set the logging level from a command-line option such as:

@example
--log=INFO
@end example

and you have the value of the parameter passed for @code{--log} in some variable
@emph{loglevel}, you can use:

@example
getattr(logging, loglevel.upper())
@end example

to get the value which you’ll pass to @ref{508,,basicConfig()} via the @emph{level}
argument. You may want to error check any user input value, perhaps as in the
following example:

@example
# assuming loglevel is bound to the string value obtained from the
# command line argument. Convert to upper case to allow the user to
# specify --log=DEBUG or --log=debug
numeric_level = getattr(logging, loglevel.upper(), None)
if not isinstance(numeric_level, int):
    raise ValueError('Invalid log level: %s' % loglevel)
logging.basicConfig(level=numeric_level, ...)
@end example

The call to @ref{508,,basicConfig()} should come @emph{before} any calls to @ref{2523,,debug()},
@ref{254d,,info()} etc. As it’s intended as a one-off simple configuration facility,
only the first call will actually do anything: subsequent calls are effectively
no-ops.

If you run the above script several times, the messages from successive runs
are appended to the file @emph{example.log}. If you want each run to start afresh,
not remembering the messages from earlier runs, you can specify the @emph{filemode}
argument, by changing the call in the above example to:

@example
logging.basicConfig(filename='example.log', filemode='w', level=logging.DEBUG)
@end example

The output will be the same as before, but the log file is no longer appended
to, so the messages from earlier runs are lost.

@node Logging from multiple modules,Logging variable data,Logging to a file,Basic Logging Tutorial
@anchor{howto/logging logging-from-multiple-modules}@anchor{46cb}
@subsubsection Logging from multiple modules


If your program consists of multiple modules, here’s an example of how you
could organize logging in it:

@example
# myapp.py
import logging
import mylib

def main():
    logging.basicConfig(filename='myapp.log', level=logging.INFO)
    logging.info('Started')
    mylib.do_something()
    logging.info('Finished')

if __name__ == '__main__':
    main()
@end example

@example
# mylib.py
import logging

def do_something():
    logging.info('Doing something')
@end example

If you run @emph{myapp.py}, you should see this in @emph{myapp.log}:

@example
INFO:root:Started
INFO:root:Doing something
INFO:root:Finished
@end example

which is hopefully what you were expecting to see. You can generalize this to
multiple modules, using the pattern in @emph{mylib.py}. Note that for this simple
usage pattern, you won’t know, by looking in the log file, @emph{where} in your
application your messages came from, apart from looking at the event
description. If you want to track the location of your messages, you’ll need
to refer to the documentation beyond the tutorial level – see
@ref{d98,,Advanced Logging Tutorial}.

@node Logging variable data,Changing the format of displayed messages,Logging from multiple modules,Basic Logging Tutorial
@anchor{howto/logging logging-variable-data}@anchor{46cc}
@subsubsection Logging variable data


To log variable data, use a format string for the event description message and
append the variable data as arguments. For example:

@example
import logging
logging.warning('%s before you %s', 'Look', 'leap!')
@end example

will display:

@example
WARNING:root:Look before you leap!
@end example

As you can see, merging of variable data into the event description message
uses the old, %-style of string formatting. This is for backwards
compatibility: the logging package pre-dates newer formatting options such as
@ref{784,,str.format()} and @ref{6b5,,string.Template}. These newer formatting
options @emph{are} supported, but exploring them is outside the scope of this
tutorial: see @ref{2546,,Using particular formatting styles throughout your application} for more information.

@node Changing the format of displayed messages,Displaying the date/time in messages,Logging variable data,Basic Logging Tutorial
@anchor{howto/logging changing-the-format-of-displayed-messages}@anchor{46cd}
@subsubsection Changing the format of displayed messages


To change the format which is used to display messages, you need to
specify the format you want to use:

@example
import logging
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
logging.debug('This message should appear on the console')
logging.info('So should this')
logging.warning('And this, too')
@end example

which would print:

@example
DEBUG:This message should appear on the console
INFO:So should this
WARNING:And this, too
@end example

Notice that the ‘root’ which appeared in earlier examples has disappeared. For
a full set of things that can appear in format strings, you can refer to the
documentation for @ref{2544,,LogRecord attributes}, but for simple usage, you just
need the @emph{levelname} (severity), @emph{message} (event description, including
variable data) and perhaps to display when the event occurred. This is
described in the next section.

@node Displaying the date/time in messages,Next Steps,Changing the format of displayed messages,Basic Logging Tutorial
@anchor{howto/logging displaying-the-date-time-in-messages}@anchor{46ce}
@subsubsection Displaying the date/time in messages


To display the date and time of an event, you would place ‘%(asctime)s’ in
your format string:

@example
import logging
logging.basicConfig(format='%(asctime)s %(message)s')
logging.warning('is when this event was logged.')
@end example

which should print something like this:

@example
2010-12-12 11:41:42,612 is when this event was logged.
@end example

The default format for date/time display (shown above) is like ISO8601 or
@geindex RFC; RFC 3339
RFC 3339@footnote{https://tools.ietf.org/html/rfc3339.html}. If you need more control over the formatting of the date/time, provide
a @emph{datefmt} argument to @code{basicConfig}, as in this example:

@example
import logging
logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')
logging.warning('is when this event was logged.')
@end example

which would display something like this:

@example
12/12/2010 11:46:36 AM is when this event was logged.
@end example

The format of the @emph{datefmt} argument is the same as supported by
@ref{d8b,,time.strftime()}.

@node Next Steps,,Displaying the date/time in messages,Basic Logging Tutorial
@anchor{howto/logging next-steps}@anchor{46cf}
@subsubsection Next Steps


That concludes the basic tutorial. It should be enough to get you up and
running with logging. There’s a lot more that the logging package offers, but
to get the best out of it, you’ll need to invest a little more of your time in
reading the following sections. If you’re ready for that, grab some of your
favourite beverage and carry on.

If your logging needs are simple, then use the above examples to incorporate
logging into your own scripts, and if you run into problems or don’t
understand something, please post a question on the comp.lang.python Usenet
group (available at @indicateurl{https://groups.google.com/forum/#!forum/comp.lang.python}) and you
should receive help before too long.

Still here? You can carry on reading the next few sections, which provide a
slightly more advanced/in-depth tutorial than the basic one above. After that,
you can take a look at the @ref{d99,,Logging Cookbook}.

@node Advanced Logging Tutorial,Logging Levels<2>,Basic Logging Tutorial,Logging HOWTO
@anchor{howto/logging advanced-logging-tutorial}@anchor{46d0}@anchor{howto/logging logging-advanced-tutorial}@anchor{d98}
@subsection Advanced Logging Tutorial


The logging library takes a modular approach and offers several categories
of components: loggers, handlers, filters, and formatters.


@itemize *

@item 
Loggers expose the interface that application code directly uses.

@item 
Handlers send the log records (created by loggers) to the appropriate
destination.

@item 
Filters provide a finer grained facility for determining which log records
to output.

@item 
Formatters specify the layout of log records in the final output.
@end itemize

Log event information is passed between loggers, handlers, filters and
formatters in a @ref{b6c,,LogRecord} instance.

Logging is performed by calling methods on instances of the @ref{676,,Logger}
class (hereafter called @emph{loggers}). Each instance has a name, and they are
conceptually arranged in a namespace hierarchy using dots (periods) as
separators. For example, a logger named ‘scan’ is the parent of loggers
‘scan.text’, ‘scan.html’ and ‘scan.pdf’. Logger names can be anything you want,
and indicate the area of an application in which a logged message originates.

A good convention to use when naming loggers is to use a module-level logger,
in each module which uses logging, named as follows:

@example
logger = logging.getLogger(__name__)
@end example

This means that logger names track the package/module hierarchy, and it’s
intuitively obvious where events are logged just from the logger name.

The root of the hierarchy of loggers is called the root logger. That’s the
logger used by the functions @ref{2523,,debug()}, @ref{254d,,info()}, @ref{255f,,warning()},
@ref{2560,,error()} and @ref{2561,,critical()}, which just call the same-named method of
the root logger. The functions and the methods have the same signatures. The
root logger’s name is printed as ‘root’ in the logged output.

It is, of course, possible to log messages to different destinations. Support
is included in the package for writing log messages to files, HTTP GET/POST
locations, email via SMTP, generic sockets, queues, or OS-specific logging
mechanisms such as syslog or the Windows NT event log. Destinations are served
by @emph{handler} classes. You can create your own log destination class if
you have special requirements not met by any of the built-in handler classes.

By default, no destination is set for any logging messages. You can specify
a destination (such as console or file) by using @ref{508,,basicConfig()} as in the
tutorial examples. If you call the functions  @ref{2523,,debug()}, @ref{254d,,info()},
@ref{255f,,warning()}, @ref{2560,,error()} and @ref{2561,,critical()}, they will check to see
if no destination is set; and if one is not set, they will set a destination
of the console (@code{sys.stderr}) and a default format for the displayed
message before delegating to the root logger to do the actual message output.

The default format set by @ref{508,,basicConfig()} for messages is:

@example
severity:logger name:message
@end example

You can change this by passing a format string to @ref{508,,basicConfig()} with the
@emph{format} keyword argument. For all options regarding how a format string is
constructed, see @ref{2542,,Formatter Objects}.

@menu
* Logging Flow:: 
* Loggers:: 
* Handlers:: 
* Formatters:: 
* Configuring Logging:: 
* What happens if no configuration is provided:: 
* Configuring Logging for a Library:: 

@end menu

@node Logging Flow,Loggers,,Advanced Logging Tutorial
@anchor{howto/logging logging-flow}@anchor{46d1}
@subsubsection Logging Flow


The flow of log event information in loggers and handlers is illustrated in the
following diagram.

@image{python-figures/logging_flow,,,,png}

@node Loggers,Handlers,Logging Flow,Advanced Logging Tutorial
@anchor{howto/logging loggers}@anchor{46d2}
@subsubsection Loggers


@ref{676,,Logger} objects have a threefold job.  First, they expose several
methods to application code so that applications can log messages at runtime.
Second, logger objects determine which log messages to act upon based upon
severity (the default filtering facility) or filter objects.  Third, logger
objects pass along relevant log messages to all interested log handlers.

The most widely used methods on logger objects fall into two categories:
configuration and message sending.

These are the most common configuration methods:


@itemize *

@item 
@ref{251d,,Logger.setLevel()} specifies the lowest-severity log message a logger
will handle, where debug is the lowest built-in severity level and critical
is the highest built-in severity.  For example, if the severity level is
INFO, the logger will handle only INFO, WARNING, ERROR, and CRITICAL messages
and will ignore DEBUG messages.

@item 
@ref{2529,,Logger.addHandler()} and @ref{252a,,Logger.removeHandler()} add and remove
handler objects from the logger object.  Handlers are covered in more detail
in @ref{46d3,,Handlers}.

@item 
@ref{2526,,Logger.addFilter()} and @ref{2527,,Logger.removeFilter()} add and remove filter
objects from the logger object.  Filters are covered in more detail in
@ref{254a,,Filter Objects}.
@end itemize

You don’t need to always call these methods on every logger you create. See the
last two paragraphs in this section.

With the logger object configured, the following methods create log messages:


@itemize *

@item 
@ref{991,,Logger.debug()}, @ref{2522,,Logger.info()}, @ref{2524,,Logger.warning()},
@ref{2525,,Logger.error()}, and @ref{990,,Logger.critical()} all create log records with
a message and a level that corresponds to their respective method names. The
message is actually a format string, which may contain the standard string
substitution syntax of @code{%s}, @code{%d}, @code{%f}, and so on.  The
rest of their arguments is a list of objects that correspond with the
substitution fields in the message.  With regard to @code{**kwargs}, the
logging methods care only about a keyword of @code{exc_info} and use it to
determine whether to log exception information.

@item 
@ref{98f,,Logger.exception()} creates a log message similar to
@ref{2525,,Logger.error()}.  The difference is that @ref{98f,,Logger.exception()} dumps a
stack trace along with it.  Call this method only from an exception handler.

@item 
@ref{98e,,Logger.log()} takes a log level as an explicit argument.  This is a
little more verbose for logging messages than using the log level convenience
methods listed above, but this is how to log at custom log levels.
@end itemize

@ref{463,,getLogger()} returns a reference to a logger instance with the specified
name if it is provided, or @code{root} if not.  The names are period-separated
hierarchical structures.  Multiple calls to @ref{463,,getLogger()} with the same name
will return a reference to the same logger object.  Loggers that are further
down in the hierarchical list are children of loggers higher up in the list.
For example, given a logger with a name of @code{foo}, loggers with names of
@code{foo.bar}, @code{foo.bar.baz}, and @code{foo.bam} are all descendants of @code{foo}.

Loggers have a concept of @emph{effective level}. If a level is not explicitly set
on a logger, the level of its parent is used instead as its effective level.
If the parent has no explicit level set, @emph{its} parent is examined, and so on -
all ancestors are searched until an explicitly set level is found. The root
logger always has an explicit level set (@code{WARNING} by default). When deciding
whether to process an event, the effective level of the logger is used to
determine whether the event is passed to the logger’s handlers.

Child loggers propagate messages up to the handlers associated with their
ancestor loggers. Because of this, it is unnecessary to define and configure
handlers for all the loggers an application uses. It is sufficient to
configure handlers for a top-level logger and create child loggers as needed.
(You can, however, turn off propagation by setting the @emph{propagate}
attribute of a logger to @code{False}.)

@node Handlers,Formatters,Loggers,Advanced Logging Tutorial
@anchor{howto/logging handler-basic}@anchor{46d3}@anchor{howto/logging handlers}@anchor{46d4}
@subsubsection Handlers


@ref{2521,,Handler} objects are responsible for dispatching the
appropriate log messages (based on the log messages’ severity) to the handler’s
specified destination.  @ref{676,,Logger} objects can add zero or more handler
objects to themselves with an @ref{2529,,addHandler()} method.  As an example
scenario, an application may want to send all log messages to a log file, all
log messages of error or higher to stdout, and all messages of critical to an
email address. This scenario requires three individual handlers where each
handler is responsible for sending messages of a specific severity to a specific
location.

The standard library includes quite a few handler types (see
@ref{46d5,,Useful Handlers}); the tutorials use mainly @ref{d9a,,StreamHandler} and
@ref{1104,,FileHandler} in its examples.

There are very few methods in a handler for application developers to concern
themselves with.  The only handler methods that seem relevant for application
developers who are using the built-in handler objects (that is, not creating
custom handlers) are the following configuration methods:


@itemize *

@item 
The @ref{2536,,setLevel()} method, just as in logger objects, specifies the
lowest severity that will be dispatched to the appropriate destination.  Why
are there two @code{setLevel()} methods?  The level set in the logger
determines which severity of messages it will pass to its handlers.  The level
set in each handler determines which messages that handler will send on.

@item 
@ref{2537,,setFormatter()} selects a Formatter object for this handler to
use.

@item 
@ref{2538,,addFilter()} and @ref{2539,,removeFilter()} respectively
configure and deconfigure filter objects on handlers.
@end itemize

Application code should not directly instantiate and use instances of
@ref{2521,,Handler}.  Instead, the @ref{2521,,Handler} class is a base class that
defines the interface that all handlers should have and establishes some
default behavior that child classes can use (or override).

@node Formatters,Configuring Logging,Handlers,Advanced Logging Tutorial
@anchor{howto/logging formatters}@anchor{46d6}
@subsubsection Formatters


Formatter objects configure the final order, structure, and contents of the log
message.  Unlike the base @ref{2521,,logging.Handler} class, application code may
instantiate formatter classes, although you could likely subclass the formatter
if your application needs special behavior.  The constructor takes three
optional arguments – a message format string, a date format string and a style
indicator.

@geindex __init__() (logging.logging.Formatter method)
@anchor{howto/logging logging logging Formatter __init__}@anchor{46d7}
@deffn {Method} logging.Formatter.__init__ (fmt=None, datefmt=None, style='%')
@end deffn

If there is no message format string, the default is to use the
raw message.  If there is no date format string, the default date format is:

@example
%Y-%m-%d %H:%M:%S
@end example

with the milliseconds tacked on at the end. The @code{style} is one of @cite{%}, ‘@{’
or ‘$’. If one of these is not specified, then ‘%’ will be used.

If the @code{style} is ‘%’, the message format string uses
@code{%(<dictionary key>)s} styled string substitution; the possible keys are
documented in @ref{2544,,LogRecord attributes}. If the style is ‘@{’, the message
format string is assumed to be compatible with @ref{784,,str.format()} (using
keyword arguments), while if the style is ‘$’ then the message format string
should conform to what is expected by @ref{15c2,,string.Template.substitute()}.


Changed in version 3.2: Added the @code{style} parameter.


The following message format string will log the time in a human-readable
format, the severity of the message, and the contents of the message, in that
order:

@example
'%(asctime)s - %(levelname)s - %(message)s'
@end example

Formatters use a user-configurable function to convert the creation time of a
record to a tuple. By default, @ref{1224,,time.localtime()} is used; to change this
for a particular formatter instance, set the @code{converter} attribute of the
instance to a function with the same signature as @ref{1224,,time.localtime()} or
@ref{d65,,time.gmtime()}. To change it for all formatters, for example if you want
all logging times to be shown in GMT, set the @code{converter} attribute in the
Formatter class (to @code{time.gmtime} for GMT display).

@node Configuring Logging,What happens if no configuration is provided,Formatters,Advanced Logging Tutorial
@anchor{howto/logging configuring-logging}@anchor{46d8}
@subsubsection Configuring Logging


Programmers can configure logging in three ways:


@enumerate 

@item 
Creating loggers, handlers, and formatters explicitly using Python
code that calls the configuration methods listed above.

@item 
Creating a logging config file and reading it using the @ref{678,,fileConfig()}
function.

@item 
Creating a dictionary of configuration information and passing it
to the @ref{d55,,dictConfig()} function.
@end enumerate

For the reference documentation on the last two options, see
@ref{e8f,,Configuration functions}.  The following example configures a very simple
logger, a console handler, and a simple formatter using Python code:

@example
import logging

# create logger
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)

# create console handler and set level to debug
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)

# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# add formatter to ch
ch.setFormatter(formatter)

# add ch to logger
logger.addHandler(ch)

# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warning('warn message')
logger.error('error message')
logger.critical('critical message')
@end example

Running this module from the command line produces the following output:

@example
$ python simple_logging_module.py
2005-03-19 15:10:26,618 - simple_example - DEBUG - debug message
2005-03-19 15:10:26,620 - simple_example - INFO - info message
2005-03-19 15:10:26,695 - simple_example - WARNING - warn message
2005-03-19 15:10:26,697 - simple_example - ERROR - error message
2005-03-19 15:10:26,773 - simple_example - CRITICAL - critical message
@end example

The following Python module creates a logger, handler, and formatter nearly
identical to those in the example listed above, with the only difference being
the names of the objects:

@example
import logging
import logging.config

logging.config.fileConfig('logging.conf')

# create logger
logger = logging.getLogger('simpleExample')

# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warning('warn message')
logger.error('error message')
logger.critical('critical message')
@end example

Here is the logging.conf file:

@example
[loggers]
keys=root,simpleExample

[handlers]
keys=consoleHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=DEBUG
handlers=consoleHandler

[logger_simpleExample]
level=DEBUG
handlers=consoleHandler
qualname=simpleExample
propagate=0

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
@end example

The output is nearly identical to that of the non-config-file-based example:

@example
$ python simple_logging_config.py
2005-03-19 15:38:55,977 - simpleExample - DEBUG - debug message
2005-03-19 15:38:55,979 - simpleExample - INFO - info message
2005-03-19 15:38:56,054 - simpleExample - WARNING - warn message
2005-03-19 15:38:56,055 - simpleExample - ERROR - error message
2005-03-19 15:38:56,130 - simpleExample - CRITICAL - critical message
@end example

You can see that the config file approach has a few advantages over the Python
code approach, mainly separation of configuration and code and the ability of
noncoders to easily modify the logging properties.

@cartouche
@quotation Warning 
The @ref{678,,fileConfig()} function takes a default parameter,
@code{disable_existing_loggers}, which defaults to @code{True} for reasons of
backward compatibility. This may or may not be what you want, since it
will cause any non-root loggers existing before the @ref{678,,fileConfig()}
call to be disabled unless they (or an ancestor) are explicitly named in
the configuration. Please refer to the reference documentation for more
information, and specify @code{False} for this parameter if you wish.

The dictionary passed to @ref{d55,,dictConfig()} can also specify a Boolean
value with key @code{disable_existing_loggers}, which if not specified
explicitly in the dictionary also defaults to being interpreted as
@code{True}. This leads to the logger-disabling behaviour described above,
which may not be what you want - in which case, provide the key
explicitly with a value of @code{False}.
@end quotation
@end cartouche

Note that the class names referenced in config files need to be either relative
to the logging module, or absolute values which can be resolved using normal
import mechanisms. Thus, you could use either
@ref{2599,,WatchedFileHandler} (relative to the logging module) or
@code{mypackage.mymodule.MyHandler} (for a class defined in package @code{mypackage}
and module @code{mymodule}, where @code{mypackage} is available on the Python import
path).

In Python 3.2, a new means of configuring logging has been introduced, using
dictionaries to hold configuration information. This provides a superset of the
functionality of the config-file-based approach outlined above, and is the
recommended configuration method for new applications and deployments. Because
a Python dictionary is used to hold configuration information, and since you
can populate that dictionary using different means, you have more options for
configuration. For example, you can use a configuration file in JSON format,
or, if you have access to YAML processing functionality, a file in YAML
format, to populate the configuration dictionary. Or, of course, you can
construct the dictionary in Python code, receive it in pickled form over a
socket, or use whatever approach makes sense for your application.

Here’s an example of the same configuration as above, in YAML format for
the new dictionary-based approach:

@example
version: 1
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: simple
    stream: ext://sys.stdout
loggers:
  simpleExample:
    level: DEBUG
    handlers: [console]
    propagate: no
root:
  level: DEBUG
  handlers: [console]
@end example

For more information about logging using a dictionary, see
@ref{e8f,,Configuration functions}.

@node What happens if no configuration is provided,Configuring Logging for a Library,Configuring Logging,Advanced Logging Tutorial
@anchor{howto/logging what-happens-if-no-configuration-is-provided}@anchor{46d9}
@subsubsection What happens if no configuration is provided


If no logging configuration is provided, it is possible to have a situation
where a logging event needs to be output, but no handlers can be found to
output the event. The behaviour of the logging package in these
circumstances is dependent on the Python version.

For versions of Python prior to 3.2, the behaviour is as follows:


@itemize *

@item 
If @emph{logging.raiseExceptions} is @code{False} (production mode), the event is
silently dropped.

@item 
If @emph{logging.raiseExceptions} is @code{True} (development mode), a message
‘No handlers could be found for logger X.Y.Z’ is printed once.
@end itemize

In Python 3.2 and later, the behaviour is as follows:


@itemize *

@item 
The event is output using a ‘handler of last resort’, stored in
@code{logging.lastResort}. This internal handler is not associated with any
logger, and acts like a @ref{d9a,,StreamHandler} which writes the
event description message to the current value of @code{sys.stderr} (therefore
respecting any redirections which may be in effect). No formatting is
done on the message - just the bare event description message is printed.
The handler’s level is set to @code{WARNING}, so all events at this and
greater severities will be output.
@end itemize

To obtain the pre-3.2 behaviour, @code{logging.lastResort} can be set to @code{None}.

@node Configuring Logging for a Library,,What happens if no configuration is provided,Advanced Logging Tutorial
@anchor{howto/logging configuring-logging-for-a-library}@anchor{46da}@anchor{howto/logging library-config}@anchor{2596}
@subsubsection Configuring Logging for a Library


When developing a library which uses logging, you should take care to
document how the library uses logging - for example, the names of loggers
used. Some consideration also needs to be given to its logging configuration.
If the using application does not use logging, and library code makes logging
calls, then (as described in the previous section) events of severity
@code{WARNING} and greater will be printed to @code{sys.stderr}. This is regarded as
the best default behaviour.

If for some reason you @emph{don’t} want these messages printed in the absence of
any logging configuration, you can attach a do-nothing handler to the top-level
logger for your library. This avoids the message being printed, since a handler
will always be found for the library’s events: it just doesn’t produce any
output. If the library user configures logging for application use, presumably
that configuration will add some handlers, and if levels are suitably
configured then logging calls made in library code will send output to those
handlers, as normal.

A do-nothing handler is included in the logging package:
@ref{e34,,NullHandler} (since Python 3.1). An instance of this handler
could be added to the top-level logger of the logging namespace used by the
library (@emph{if} you want to prevent your library’s logged events being output to
@code{sys.stderr} in the absence of logging configuration). If all logging by a
library @emph{foo} is done using loggers with names matching ‘foo.x’, ‘foo.x.y’,
etc. then the code:

@example
import logging
logging.getLogger('foo').addHandler(logging.NullHandler())
@end example

should have the desired effect. If an organisation produces a number of
libraries, then the logger name specified can be ‘orgname.foo’ rather than
just ‘foo’.

@cartouche
@quotation Note 
It is strongly advised that you @emph{do not log to the root logger}
in your library. Instead, use a logger with a unique and easily
identifiable name, such as the @code{__name__} for your library’s top-level package
or module. Logging to the root logger will make it difficult or impossible for
the application developer to configure the logging verbosity or handlers of
your library as they wish.
@end quotation
@end cartouche

@cartouche
@quotation Note 
It is strongly advised that you @emph{do not add any handlers other
than} @ref{e34,,NullHandler} @emph{to your library’s loggers}. This is
because the configuration of handlers is the prerogative of the application
developer who uses your library. The application developer knows their
target audience and what handlers are most appropriate for their
application: if you add handlers ‘under the hood’, you might well interfere
with their ability to carry out unit tests and deliver logs which suit their
requirements.
@end quotation
@end cartouche

@node Logging Levels<2>,Useful Handlers,Advanced Logging Tutorial,Logging HOWTO
@anchor{howto/logging logging-levels}@anchor{46db}
@subsection Logging Levels


The numeric values of logging levels are given in the following table. These are
primarily of interest if you want to define your own levels, and need them to
have specific values relative to the predefined levels. If you define a level
with the same numeric value, it overwrites the predefined value; the predefined
name is lost.


@multitable {xxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxx} 
@headitem

Level

@tab

Numeric value

@item

@code{CRITICAL}

@tab

50

@item

@code{ERROR}

@tab

40

@item

@code{WARNING}

@tab

30

@item

@code{INFO}

@tab

20

@item

@code{DEBUG}

@tab

10

@item

@code{NOTSET}

@tab

0

@end multitable


Levels can also be associated with loggers, being set either by the developer or
through loading a saved logging configuration. When a logging method is called
on a logger, the logger compares its own level with the level associated with
the method call. If the logger’s level is higher than the method call’s, no
logging message is actually generated. This is the basic mechanism controlling
the verbosity of logging output.

Logging messages are encoded as instances of the @ref{b6c,,LogRecord}
class. When a logger decides to actually log an event, a
@ref{b6c,,LogRecord} instance is created from the logging message.

Logging messages are subjected to a dispatch mechanism through the use of
@emph{handlers}, which are instances of subclasses of the @ref{2521,,Handler}
class. Handlers are responsible for ensuring that a logged message (in the form
of a @ref{b6c,,LogRecord}) ends up in a particular location (or set of locations)
which is useful for the target audience for that message (such as end users,
support desk staff, system administrators, developers). Handlers are passed
@ref{b6c,,LogRecord} instances intended for particular destinations. Each logger
can have zero, one or more handlers associated with it (via the
@ref{2529,,addHandler()} method of @ref{676,,Logger}). In addition to any
handlers directly associated with a logger, @emph{all handlers associated with all
ancestors of the logger} are called to dispatch the message (unless the
@emph{propagate} flag for a logger is set to a false value, at which point the
passing to ancestor handlers stops).

Just as for loggers, handlers can have levels associated with them. A handler’s
level acts as a filter in the same way as a logger’s level does. If a handler
decides to actually dispatch an event, the @ref{2540,,emit()} method is used
to send the message to its destination. Most user-defined subclasses of
@ref{2521,,Handler} will need to override this @ref{2540,,emit()}.

@menu
* Custom Levels:: 

@end menu

@node Custom Levels,,,Logging Levels<2>
@anchor{howto/logging custom-levels}@anchor{2566}@anchor{howto/logging id1}@anchor{46dc}
@subsubsection Custom Levels


Defining your own levels is possible, but should not be necessary, as the
existing levels have been chosen on the basis of practical experience.
However, if you are convinced that you need custom levels, great care should
be exercised when doing this, and it is possibly @emph{a very bad idea to define
custom levels if you are developing a library}. That’s because if multiple
library authors all define their own custom levels, there is a chance that
the logging output from such multiple libraries used together will be
difficult for the using developer to control and/or interpret, because a
given numeric value might mean different things for different libraries.

@node Useful Handlers,Exceptions raised during logging,Logging Levels<2>,Logging HOWTO
@anchor{howto/logging id2}@anchor{46dd}@anchor{howto/logging useful-handlers}@anchor{46d5}
@subsection Useful Handlers


In addition to the base @ref{2521,,Handler} class, many useful subclasses are
provided:


@enumerate 

@item 
@ref{d9a,,StreamHandler} instances send messages to streams (file-like
objects).

@item 
@ref{1104,,FileHandler} instances send messages to disk files.

@item 
@ref{259d,,BaseRotatingHandler} is the base class for handlers that
rotate log files at a certain point. It is not meant to be  instantiated
directly. Instead, use @ref{2579,,RotatingFileHandler} or
@ref{adf,,TimedRotatingFileHandler}.

@item 
@ref{2579,,RotatingFileHandler} instances send messages to disk
files, with support for maximum log file sizes and log file rotation.

@item 
@ref{adf,,TimedRotatingFileHandler} instances send messages to
disk files, rotating the log file at certain timed intervals.

@item 
@ref{ae0,,SocketHandler} instances send messages to TCP/IP
sockets. Since 3.4, Unix domain sockets are also supported.

@item 
@ref{ae1,,DatagramHandler} instances send messages to UDP
sockets. Since 3.4, Unix domain sockets are also supported.

@item 
@ref{25ca,,SMTPHandler} instances send messages to a designated
email address.

@item 
@ref{c6c,,SysLogHandler} instances send messages to a Unix
syslog daemon, possibly on a remote machine.

@item 
@ref{25c2,,NTEventLogHandler} instances send messages to a
Windows NT/2000/XP event log.

@item 
@ref{2582,,MemoryHandler} instances send messages to a buffer
in memory, which is flushed whenever specific criteria are met.

@item 
@ref{992,,HTTPHandler} instances send messages to an HTTP
server using either @code{GET} or @code{POST} semantics.

@item 
@ref{2599,,WatchedFileHandler} instances watch the file they are
logging to. If the file changes, it is closed and reopened using the file
name. This handler is only useful on Unix-like systems; Windows does not
support the underlying mechanism used.

@item 
@ref{25dd,,QueueHandler} instances send messages to a queue, such as
those implemented in the @ref{d5,,queue} or @ref{b3,,multiprocessing} modules.

@item 
@ref{e34,,NullHandler} instances do nothing with error messages. They are used
by library developers who want to use logging, but want to avoid the ‘No
handlers could be found for logger XXX’ message which can be displayed if
the library user has not configured logging. See @ref{2596,,Configuring Logging for a Library} for
more information.
@end enumerate


New in version 3.1: The @ref{e34,,NullHandler} class.



New in version 3.2: The @ref{25dd,,QueueHandler} class.


The @ref{e34,,NullHandler}, @ref{d9a,,StreamHandler} and @ref{1104,,FileHandler}
classes are defined in the core logging package. The other handlers are
defined in a sub-module, @ref{a8,,logging.handlers}. (There is also another
sub-module, @ref{a7,,logging.config}, for configuration functionality.)

Logged messages are formatted for presentation through instances of the
@ref{11c6,,Formatter} class. They are initialized with a format string suitable for
use with the % operator and a dictionary.

For formatting multiple messages in a batch, instances of
@code{BufferingFormatter} can be used. In addition to the format
string (which is applied to each message in the batch), there is provision for
header and trailer format strings.

When filtering based on logger level and/or handler level is not enough,
instances of @ref{d9c,,Filter} can be added to both @ref{676,,Logger} and
@ref{2521,,Handler} instances (through their @ref{2538,,addFilter()} method).
Before deciding to process a message further, both loggers and handlers consult
all their filters for permission. If any filter returns a false value, the
message is not processed further.

The basic @ref{d9c,,Filter} functionality allows filtering by specific logger
name. If this feature is used, messages sent to the named logger and its
children are allowed through the filter, and all others dropped.

@node Exceptions raised during logging,Using arbitrary objects as messages,Useful Handlers,Logging HOWTO
@anchor{howto/logging exceptions-raised-during-logging}@anchor{46de}@anchor{howto/logging logging-exceptions}@anchor{46df}
@subsection Exceptions raised during logging


The logging package is designed to swallow exceptions which occur while logging
in production. This is so that errors which occur while handling logging events
- such as logging misconfiguration, network or other similar errors - do not
cause the application using logging to terminate prematurely.

@ref{892,,SystemExit} and @ref{4ca,,KeyboardInterrupt} exceptions are never
swallowed. Other exceptions which occur during the @ref{2540,,emit()} method
of a @ref{2521,,Handler} subclass are passed to its @ref{253f,,handleError()}
method.

The default implementation of @ref{253f,,handleError()} in @ref{2521,,Handler}
checks to see if a module-level variable, @code{raiseExceptions}, is set. If
set, a traceback is printed to @ref{42d,,sys.stderr}. If not set, the exception is
swallowed.

@cartouche
@quotation Note 
The default value of @code{raiseExceptions} is @code{True}. This is
because during development, you typically want to be notified of any
exceptions that occur. It’s advised that you set @code{raiseExceptions} to
@code{False} for production usage.
@end quotation
@end cartouche

@node Using arbitrary objects as messages,Optimization,Exceptions raised during logging,Logging HOWTO
@anchor{howto/logging arbitrary-object-messages}@anchor{2556}@anchor{howto/logging using-arbitrary-objects-as-messages}@anchor{46e0}
@subsection Using arbitrary objects as messages


In the preceding sections and examples, it has been assumed that the message
passed when logging the event is a string. However, this is not the only
possibility. You can pass an arbitrary object as a message, and its
@ref{178a,,__str__()} method will be called when the logging system needs to
convert it to a string representation. In fact, if you want to, you can avoid
computing a string representation altogether - for example, the
@ref{ae0,,SocketHandler} emits an event by pickling it and sending it
over the wire.

@node Optimization,,Using arbitrary objects as messages,Logging HOWTO
@anchor{howto/logging optimization}@anchor{46e1}
@subsection Optimization


Formatting of message arguments is deferred until it cannot be avoided.
However, computing the arguments passed to the logging method can also be
expensive, and you may want to avoid doing it if the logger will just throw
away your event. To decide what to do, you can call the
@ref{2520,,isEnabledFor()} method which takes a level argument and returns
true if the event would be created by the Logger for that level of call.
You can write code like this:

@example
if logger.isEnabledFor(logging.DEBUG):
    logger.debug('Message with %s, %s', expensive_func1(),
                                        expensive_func2())
@end example

so that if the logger’s threshold is set above @code{DEBUG}, the calls to
@code{expensive_func1()} and @code{expensive_func2()} are never made.

@cartouche
@quotation Note 
In some cases, @ref{2520,,isEnabledFor()} can itself be more
expensive than you’d like (e.g. for deeply nested loggers where an explicit
level is only set high up in the logger hierarchy). In such cases (or if you
want to avoid calling a method in tight loops), you can cache the result of a
call to @ref{2520,,isEnabledFor()} in a local or instance variable, and use
that instead of calling the method each time. Such a cached value would only
need to be recomputed when the logging configuration changes dynamically
while the application is running (which is not all that common).
@end quotation
@end cartouche

There are other optimizations which can be made for specific applications which
need more precise control over what logging information is collected. Here’s a
list of things you can do to avoid processing during logging which you don’t
need:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

What you don’t want to collect

@tab

How to avoid collecting it

@item

Information about where calls were made from.

@tab

Set @code{logging._srcfile} to @code{None}.
This avoids calling @ref{1057,,sys._getframe()}, which
may help to speed up your code in environments
like PyPy (which can’t speed up code that uses
@ref{1057,,sys._getframe()}).

@item

Threading information.

@tab

Set @code{logging.logThreads} to @code{False}.

@item

Current process ID (@ref{238b,,os.getpid()})

@tab

Set @code{logging.logProcesses} to @code{False}.

@item

Current process name when using @code{multiprocessing}
to manage multiple processes.

@tab

Set @code{logging.logMultiprocessing} to @code{False}.

@end multitable


Also note that the core logging module only includes the basic handlers. If
you don’t import @ref{a8,,logging.handlers} and @ref{a7,,logging.config}, they won’t
take up any memory.


@subsubheading See also



@table @asis

@item Module @ref{a6,,logging}

API reference for the logging module.

@item Module @ref{a7,,logging.config}

Configuration API for the logging module.

@item Module @ref{a8,,logging.handlers}

Useful handlers included with the logging module.
@end table

@ref{d99,,A logging cookbook}


@node Logging Cookbook,Regular Expression HOWTO,Logging HOWTO,Python HOWTOs
@anchor{howto/logging-cookbook doc}@anchor{46e2}@anchor{howto/logging-cookbook id1}@anchor{46e3}@anchor{howto/logging-cookbook logging-cookbook}@anchor{d99}
@section Logging Cookbook


@*Author: 
Vinay Sajip <vinay_sajip at red-dove dot com>


This page contains a number of recipes related to logging, which have been found
useful in the past.

@menu
* Using logging in multiple modules:: 
* Logging from multiple threads:: 
* Multiple handlers and formatters:: 
* Logging to multiple destinations:: 
* Configuration server example:: 
* Dealing with handlers that block:: 
* Sending and receiving logging events across a network:: 
* Adding contextual information to your logging output:: 
* Logging to a single file from multiple processes:: 
* Using file rotation:: 
* Use of alternative formatting styles:: 
* Customizing LogRecord:: 
* Subclassing QueueHandler - a ZeroMQ example:: 
* Subclassing QueueListener - a ZeroMQ example:: 
* An example dictionary-based configuration:: 
* Using a rotator and namer to customize log rotation processing:: 
* A more elaborate multiprocessing example:: 
* Inserting a BOM into messages sent to a SysLogHandler:: 
* Implementing structured logging:: 
* Customizing handlers with dictConfig(): Customizing handlers with dictConfig. 
* Using particular formatting styles throughout your application:: 
* Configuring filters with dictConfig(): Configuring filters with dictConfig. 
* Customized exception formatting:: 
* Speaking logging messages:: 
* Buffering logging messages and outputting them conditionally:: 
* Formatting times using UTC (GMT) via configuration: Formatting times using UTC GMT via configuration. 
* Using a context manager for selective logging:: 
* A CLI application starter template:: 
* A Qt GUI for logging:: 
* Patterns to avoid:: 

@end menu

@node Using logging in multiple modules,Logging from multiple threads,,Logging Cookbook
@anchor{howto/logging-cookbook using-logging-in-multiple-modules}@anchor{46e4}
@subsection Using logging in multiple modules


Multiple calls to @code{logging.getLogger('someLogger')} return a reference to the
same logger object.  This is true not only within the same module, but also
across modules as long as it is in the same Python interpreter process.  It is
true for references to the same object; additionally, application code can
define and configure a parent logger in one module and create (but not
configure) a child logger in a separate module, and all logger calls to the
child will pass up to the parent.  Here is a main module:

@example
import logging
import auxiliary_module

# create logger with 'spam_application'
logger = logging.getLogger('spam_application')
logger.setLevel(logging.DEBUG)
# create file handler which logs even debug messages
fh = logging.FileHandler('spam.log')
fh.setLevel(logging.DEBUG)
# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)
# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
ch.setFormatter(formatter)
# add the handlers to the logger
logger.addHandler(fh)
logger.addHandler(ch)

logger.info('creating an instance of auxiliary_module.Auxiliary')
a = auxiliary_module.Auxiliary()
logger.info('created an instance of auxiliary_module.Auxiliary')
logger.info('calling auxiliary_module.Auxiliary.do_something')
a.do_something()
logger.info('finished auxiliary_module.Auxiliary.do_something')
logger.info('calling auxiliary_module.some_function()')
auxiliary_module.some_function()
logger.info('done with auxiliary_module.some_function()')
@end example

Here is the auxiliary module:

@example
import logging

# create logger
module_logger = logging.getLogger('spam_application.auxiliary')

class Auxiliary:
    def __init__(self):
        self.logger = logging.getLogger('spam_application.auxiliary.Auxiliary')
        self.logger.info('creating an instance of Auxiliary')

    def do_something(self):
        self.logger.info('doing something')
        a = 1 + 1
        self.logger.info('done doing something')

def some_function():
    module_logger.info('received a call to "some_function"')
@end example

The output looks like this:

@example
2005-03-23 23:47:11,663 - spam_application - INFO -
   creating an instance of auxiliary_module.Auxiliary
2005-03-23 23:47:11,665 - spam_application.auxiliary.Auxiliary - INFO -
   creating an instance of Auxiliary
2005-03-23 23:47:11,665 - spam_application - INFO -
   created an instance of auxiliary_module.Auxiliary
2005-03-23 23:47:11,668 - spam_application - INFO -
   calling auxiliary_module.Auxiliary.do_something
2005-03-23 23:47:11,668 - spam_application.auxiliary.Auxiliary - INFO -
   doing something
2005-03-23 23:47:11,669 - spam_application.auxiliary.Auxiliary - INFO -
   done doing something
2005-03-23 23:47:11,670 - spam_application - INFO -
   finished auxiliary_module.Auxiliary.do_something
2005-03-23 23:47:11,671 - spam_application - INFO -
   calling auxiliary_module.some_function()
2005-03-23 23:47:11,672 - spam_application.auxiliary - INFO -
   received a call to 'some_function'
2005-03-23 23:47:11,673 - spam_application - INFO -
   done with auxiliary_module.some_function()
@end example

@node Logging from multiple threads,Multiple handlers and formatters,Using logging in multiple modules,Logging Cookbook
@anchor{howto/logging-cookbook logging-from-multiple-threads}@anchor{46e5}
@subsection Logging from multiple threads


Logging from multiple threads requires no special effort. The following example
shows logging from the main (initial) thread and another thread:

@example
import logging
import threading
import time

def worker(arg):
    while not arg['stop']:
        logging.debug('Hi from myfunc')
        time.sleep(0.5)

def main():
    logging.basicConfig(level=logging.DEBUG, format='%(relativeCreated)6d %(threadName)s %(message)s')
    info = @{'stop': False@}
    thread = threading.Thread(target=worker, args=(info,))
    thread.start()
    while True:
        try:
            logging.debug('Hello from main')
            time.sleep(0.75)
        except KeyboardInterrupt:
            info['stop'] = True
            break
    thread.join()

if __name__ == '__main__':
    main()
@end example

When run, the script should print something like the following:

@example
   0 Thread-1 Hi from myfunc
   3 MainThread Hello from main
 505 Thread-1 Hi from myfunc
 755 MainThread Hello from main
1007 Thread-1 Hi from myfunc
1507 MainThread Hello from main
1508 Thread-1 Hi from myfunc
2010 Thread-1 Hi from myfunc
2258 MainThread Hello from main
2512 Thread-1 Hi from myfunc
3009 MainThread Hello from main
3013 Thread-1 Hi from myfunc
3515 Thread-1 Hi from myfunc
3761 MainThread Hello from main
4017 Thread-1 Hi from myfunc
4513 MainThread Hello from main
4518 Thread-1 Hi from myfunc
@end example

This shows the logging output interspersed as one might expect. This approach
works for more threads than shown here, of course.

@node Multiple handlers and formatters,Logging to multiple destinations,Logging from multiple threads,Logging Cookbook
@anchor{howto/logging-cookbook multiple-handlers-and-formatters}@anchor{46e6}
@subsection Multiple handlers and formatters


Loggers are plain Python objects.  The @ref{2529,,addHandler()} method has no
minimum or maximum quota for the number of handlers you may add.  Sometimes it
will be beneficial for an application to log all messages of all severities to a
text file while simultaneously logging errors or above to the console.  To set
this up, simply configure the appropriate handlers.  The logging calls in the
application code will remain unchanged.  Here is a slight modification to the
previous simple module-based configuration example:

@example
import logging

logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)
# create file handler which logs even debug messages
fh = logging.FileHandler('spam.log')
fh.setLevel(logging.DEBUG)
# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)
# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)
fh.setFormatter(formatter)
# add the handlers to logger
logger.addHandler(ch)
logger.addHandler(fh)

# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warning('warn message')
logger.error('error message')
logger.critical('critical message')
@end example

Notice that the ‘application’ code does not care about multiple handlers.  All
that changed was the addition and configuration of a new handler named @emph{fh}.

The ability to create new handlers with higher- or lower-severity filters can be
very helpful when writing and testing an application.  Instead of using many
@code{print} statements for debugging, use @code{logger.debug}: Unlike the print
statements, which you will have to delete or comment out later, the logger.debug
statements can remain intact in the source code and remain dormant until you
need them again.  At that time, the only change that needs to happen is to
modify the severity level of the logger and/or handler to debug.

@node Logging to multiple destinations,Configuration server example,Multiple handlers and formatters,Logging Cookbook
@anchor{howto/logging-cookbook logging-to-multiple-destinations}@anchor{46e7}@anchor{howto/logging-cookbook multiple-destinations}@anchor{46e8}
@subsection Logging to multiple destinations


Let’s say you want to log to console and file with different message formats and
in differing circumstances. Say you want to log messages with levels of DEBUG
and higher to file, and those messages at level INFO and higher to the console.
Let’s also assume that the file should contain timestamps, but the console
messages should not. Here’s how you can achieve this:

@example
import logging

# set up logging to file - see previous section for more details
logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
                    datefmt='%m-%d %H:%M',
                    filename='/temp/myapp.log',
                    filemode='w')
# define a Handler which writes INFO messages or higher to the sys.stderr
console = logging.StreamHandler()
console.setLevel(logging.INFO)
# set a format which is simpler for console use
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
# tell the handler to use this format
console.setFormatter(formatter)
# add the handler to the root logger
logging.getLogger('@w{'}).addHandler(console)

# Now, we can log to the root logger, or any other logger. First the root...
logging.info('Jackdaws love my big sphinx of quartz.')

# Now, define a couple of other loggers which might represent areas in your
# application:

logger1 = logging.getLogger('myapp.area1')
logger2 = logging.getLogger('myapp.area2')

logger1.debug('Quick zephyrs blow, vexing daft Jim.')
logger1.info('How quickly daft jumping zebras vex.')
logger2.warning('Jail zesty vixen who grabbed pay from quack.')
logger2.error('The five boxing wizards jump quickly.')
@end example

When you run this, on the console you will see

@example
root        : INFO     Jackdaws love my big sphinx of quartz.
myapp.area1 : INFO     How quickly daft jumping zebras vex.
myapp.area2 : WARNING  Jail zesty vixen who grabbed pay from quack.
myapp.area2 : ERROR    The five boxing wizards jump quickly.
@end example

and in the file you will see something like

@example
10-22 22:19 root         INFO     Jackdaws love my big sphinx of quartz.
10-22 22:19 myapp.area1  DEBUG    Quick zephyrs blow, vexing daft Jim.
10-22 22:19 myapp.area1  INFO     How quickly daft jumping zebras vex.
10-22 22:19 myapp.area2  WARNING  Jail zesty vixen who grabbed pay from quack.
10-22 22:19 myapp.area2  ERROR    The five boxing wizards jump quickly.
@end example

As you can see, the DEBUG message only shows up in the file. The other messages
are sent to both destinations.

This example uses console and file handlers, but you can use any number and
combination of handlers you choose.

@node Configuration server example,Dealing with handlers that block,Logging to multiple destinations,Logging Cookbook
@anchor{howto/logging-cookbook configuration-server-example}@anchor{46e9}
@subsection Configuration server example


Here is an example of a module using the logging configuration server:

@example
import logging
import logging.config
import time
import os

# read initial config file
logging.config.fileConfig('logging.conf')

# create and start listener on port 9999
t = logging.config.listen(9999)
t.start()

logger = logging.getLogger('simpleExample')

try:
    # loop through logging calls to see the difference
    # new configurations make, until Ctrl+C is pressed
    while True:
        logger.debug('debug message')
        logger.info('info message')
        logger.warning('warn message')
        logger.error('error message')
        logger.critical('critical message')
        time.sleep(5)
except KeyboardInterrupt:
    # cleanup
    logging.config.stopListening()
    t.join()
@end example

And here is a script that takes a filename and sends that file to the server,
properly preceded with the binary-encoded length, as the new logging
configuration:

@example
#!/usr/bin/env python
import socket, sys, struct

with open(sys.argv[1], 'rb') as f:
    data_to_send = f.read()

HOST = 'localhost'
PORT = 9999
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print('connecting...')
s.connect((HOST, PORT))
print('sending config...')
s.send(struct.pack('>L', len(data_to_send)))
s.send(data_to_send)
s.close()
print('complete')
@end example

@node Dealing with handlers that block,Sending and receiving logging events across a network,Configuration server example,Logging Cookbook
@anchor{howto/logging-cookbook dealing-with-handlers-that-block}@anchor{46ea}
@subsection Dealing with handlers that block


Sometimes you have to get your logging handlers to do their work without
blocking the thread you’re logging from. This is common in web applications,
though of course it also occurs in other scenarios.

A common culprit which demonstrates sluggish behaviour is the
@ref{25ca,,SMTPHandler}: sending emails can take a long time, for a
number of reasons outside the developer’s control (for example, a poorly
performing mail or network infrastructure). But almost any network-based
handler can block: Even a @ref{ae0,,SocketHandler} operation may do a
DNS query under the hood which is too slow (and this query can be deep in the
socket library code, below the Python layer, and outside your control).

One solution is to use a two-part approach. For the first part, attach only a
@ref{25dd,,QueueHandler} to those loggers which are accessed from
performance-critical threads. They simply write to their queue, which can be
sized to a large enough capacity or initialized with no upper bound to their
size. The write to the queue will typically be accepted quickly, though you
will probably need to catch the @ref{2842,,queue.Full} exception as a precaution
in your code. If you are a library developer who has performance-critical
threads in their code, be sure to document this (together with a suggestion to
attach only @code{QueueHandlers} to your loggers) for the benefit of other
developers who will use your code.

The second part of the solution is @ref{993,,QueueListener}, which has been
designed as the counterpart to @ref{25dd,,QueueHandler}.  A
@ref{993,,QueueListener} is very simple: it’s passed a queue and some handlers,
and it fires up an internal thread which listens to its queue for LogRecords
sent from @code{QueueHandlers} (or any other source of @code{LogRecords}, for that
matter). The @code{LogRecords} are removed from the queue and passed to the
handlers for processing.

The advantage of having a separate @ref{993,,QueueListener} class is that you
can use the same instance to service multiple @code{QueueHandlers}. This is more
resource-friendly than, say, having threaded versions of the existing handler
classes, which would eat up one thread per handler for no particular benefit.

An example of using these two classes follows (imports omitted):

@example
que = queue.Queue(-1)  # no limit on size
queue_handler = QueueHandler(que)
handler = logging.StreamHandler()
listener = QueueListener(que, handler)
root = logging.getLogger()
root.addHandler(queue_handler)
formatter = logging.Formatter('%(threadName)s: %(message)s')
handler.setFormatter(formatter)
listener.start()
# The log output will display the thread which generated
# the event (the main thread) rather than the internal
# thread which monitors the internal queue. This is what
# you want to happen.
root.warning('Look out!')
listener.stop()
@end example

which, when run, will produce:

@example
MainThread: Look out!
@end example


Changed in version 3.5: Prior to Python 3.5, the @ref{993,,QueueListener} always passed every message
received from the queue to every handler it was initialized with. (This was
because it was assumed that level filtering was all done on the other side,
where the queue is filled.) From 3.5 onwards, this behaviour can be changed
by passing a keyword argument @code{respect_handler_level=True} to the
listener’s constructor. When this is done, the listener compares the level
of each message with the handler’s level, and only passes a message to a
handler if it’s appropriate to do so.


@node Sending and receiving logging events across a network,Adding contextual information to your logging output,Dealing with handlers that block,Logging Cookbook
@anchor{howto/logging-cookbook network-logging}@anchor{46eb}@anchor{howto/logging-cookbook sending-and-receiving-logging-events-across-a-network}@anchor{46ec}
@subsection Sending and receiving logging events across a network


Let’s say you want to send logging events across a network, and handle them at
the receiving end. A simple way of doing this is attaching a
@ref{ae0,,SocketHandler} instance to the root logger at the sending end:

@example
import logging, logging.handlers

rootLogger = logging.getLogger('@w{'})
rootLogger.setLevel(logging.DEBUG)
socketHandler = logging.handlers.SocketHandler('localhost',
                    logging.handlers.DEFAULT_TCP_LOGGING_PORT)
# don't bother with a formatter, since a socket handler sends the event as
# an unformatted pickle
rootLogger.addHandler(socketHandler)

# Now, we can log to the root logger, or any other logger. First the root...
logging.info('Jackdaws love my big sphinx of quartz.')

# Now, define a couple of other loggers which might represent areas in your
# application:

logger1 = logging.getLogger('myapp.area1')
logger2 = logging.getLogger('myapp.area2')

logger1.debug('Quick zephyrs blow, vexing daft Jim.')
logger1.info('How quickly daft jumping zebras vex.')
logger2.warning('Jail zesty vixen who grabbed pay from quack.')
logger2.error('The five boxing wizards jump quickly.')
@end example

At the receiving end, you can set up a receiver using the @ref{eb,,socketserver}
module. Here is a basic working example:

@example
import pickle
import logging
import logging.handlers
import socketserver
import struct


class LogRecordStreamHandler(socketserver.StreamRequestHandler):
    """Handler for a streaming logging request.

    This basically logs the record using whatever logging policy is
    configured locally.
    """

    def handle(self):
        """
        Handle multiple requests - each expected to be a 4-byte length,
        followed by the LogRecord in pickle format. Logs the record
        according to whatever policy is configured locally.
        """
        while True:
            chunk = self.connection.recv(4)
            if len(chunk) < 4:
                break
            slen = struct.unpack('>L', chunk)[0]
            chunk = self.connection.recv(slen)
            while len(chunk) < slen:
                chunk = chunk + self.connection.recv(slen - len(chunk))
            obj = self.unPickle(chunk)
            record = logging.makeLogRecord(obj)
            self.handleLogRecord(record)

    def unPickle(self, data):
        return pickle.loads(data)

    def handleLogRecord(self, record):
        # if a name is specified, we use the named logger rather than the one
        # implied by the record.
        if self.server.logname is not None:
            name = self.server.logname
        else:
            name = record.name
        logger = logging.getLogger(name)
        # N.B. EVERY record gets logged. This is because Logger.handle
        # is normally called AFTER logger-level filtering. If you want
        # to do filtering, do it at the client end to save wasting
        # cycles and network bandwidth!
        logger.handle(record)

class LogRecordSocketReceiver(socketserver.ThreadingTCPServer):
    """
    Simple TCP socket-based logging receiver suitable for testing.
    """

    allow_reuse_address = True

    def __init__(self, host='localhost',
                 port=logging.handlers.DEFAULT_TCP_LOGGING_PORT,
                 handler=LogRecordStreamHandler):
        socketserver.ThreadingTCPServer.__init__(self, (host, port), handler)
        self.abort = 0
        self.timeout = 1
        self.logname = None

    def serve_until_stopped(self):
        import select
        abort = 0
        while not abort:
            rd, wr, ex = select.select([self.socket.fileno()],
                                       [], [],
                                       self.timeout)
            if rd:
                self.handle_request()
            abort = self.abort

def main():
    logging.basicConfig(
        format='%(relativeCreated)5d %(name)-15s %(levelname)-8s %(message)s')
    tcpserver = LogRecordSocketReceiver()
    print('About to start TCP server...')
    tcpserver.serve_until_stopped()

if __name__ == '__main__':
    main()
@end example

First run the server, and then the client. On the client side, nothing is
printed on the console; on the server side, you should see something like:

@example
About to start TCP server...
   59 root            INFO     Jackdaws love my big sphinx of quartz.
   59 myapp.area1     DEBUG    Quick zephyrs blow, vexing daft Jim.
   69 myapp.area1     INFO     How quickly daft jumping zebras vex.
   69 myapp.area2     WARNING  Jail zesty vixen who grabbed pay from quack.
   69 myapp.area2     ERROR    The five boxing wizards jump quickly.
@end example

Note that there are some security issues with pickle in some scenarios. If
these affect you, you can use an alternative serialization scheme by overriding
the @code{makePickle()} method and implementing your
alternative there, as well as adapting the above script to use your alternative
serialization.

@menu
* Running a logging socket listener in production:: 

@end menu

@node Running a logging socket listener in production,,,Sending and receiving logging events across a network
@anchor{howto/logging-cookbook running-a-logging-socket-listener-in-production}@anchor{46ed}
@subsubsection Running a logging socket listener in production


To run a logging listener in production, you may need to use a process-management tool
such as Supervisor@footnote{http://supervisord.org/}. Here@footnote{https://gist.github.com/vsajip/4b227eeec43817465ca835ca66f75e2b} is a Gist which
provides the bare-bones files to run the above functionality using Supervisor: you
will need to change the @cite{/path/to/} parts in the Gist to reflect the actual paths you
want to use.

@node Adding contextual information to your logging output,Logging to a single file from multiple processes,Sending and receiving logging events across a network,Logging Cookbook
@anchor{howto/logging-cookbook adding-contextual-information-to-your-logging-output}@anchor{46ee}@anchor{howto/logging-cookbook context-info}@anchor{2559}
@subsection Adding contextual information to your logging output


Sometimes you want logging output to contain contextual information in
addition to the parameters passed to the logging call. For example, in a
networked application, it may be desirable to log client-specific information
in the log (e.g. remote client’s username, or IP address). Although you could
use the @emph{extra} parameter to achieve this, it’s not always convenient to pass
the information in this way. While it might be tempting to create
@code{Logger} instances on a per-connection basis, this is not a good idea
because these instances are not garbage collected. While this is not a problem
in practice, when the number of @code{Logger} instances is dependent on the
level of granularity you want to use in logging an application, it could
be hard to manage if the number of @code{Logger} instances becomes
effectively unbounded.

@menu
* Using LoggerAdapters to impart contextual information:: 
* Using Filters to impart contextual information:: 

@end menu

@node Using LoggerAdapters to impart contextual information,Using Filters to impart contextual information,,Adding contextual information to your logging output
@anchor{howto/logging-cookbook using-loggeradapters-to-impart-contextual-information}@anchor{46ef}
@subsubsection Using LoggerAdapters to impart contextual information


An easy way in which you can pass contextual information to be output along
with logging event information is to use the @code{LoggerAdapter} class.
This class is designed to look like a @code{Logger}, so that you can call
@code{debug()}, @code{info()}, @code{warning()}, @code{error()},
@code{exception()}, @code{critical()} and @code{log()}. These methods have the
same signatures as their counterparts in @code{Logger}, so you can use the
two types of instances interchangeably.

When you create an instance of @code{LoggerAdapter}, you pass it a
@code{Logger} instance and a dict-like object which contains your contextual
information. When you call one of the logging methods on an instance of
@code{LoggerAdapter}, it delegates the call to the underlying instance of
@code{Logger} passed to its constructor, and arranges to pass the contextual
information in the delegated call. Here’s a snippet from the code of
@code{LoggerAdapter}:

@example
def debug(self, msg, /, *args, **kwargs):
    """
    Delegate a debug call to the underlying logger, after adding
    contextual information from this adapter instance.
    """
    msg, kwargs = self.process(msg, kwargs)
    self.logger.debug(msg, *args, **kwargs)
@end example

The @code{process()} method of @code{LoggerAdapter} is where the
contextual information is added to the logging output. It’s passed the message
and keyword arguments of the logging call, and it passes back (potentially)
modified versions of these to use in the call to the underlying logger. The
default implementation of this method leaves the message alone, but inserts
an ‘extra’ key in the keyword argument whose value is the dict-like object
passed to the constructor. Of course, if you had passed an ‘extra’ keyword
argument in the call to the adapter, it will be silently overwritten.

The advantage of using ‘extra’ is that the values in the dict-like object are
merged into the @code{LogRecord} instance’s __dict__, allowing you to use
customized strings with your @code{Formatter} instances which know about
the keys of the dict-like object. If you need a different method, e.g. if you
want to prepend or append the contextual information to the message string,
you just need to subclass @code{LoggerAdapter} and override
@code{process()} to do what you need. Here is a simple example:

@example
class CustomAdapter(logging.LoggerAdapter):
    """
    This example adapter expects the passed in dict-like object to have a
    'connid' key, whose value in brackets is prepended to the log message.
    """
    def process(self, msg, kwargs):
        return '[%s] %s' % (self.extra['connid'], msg), kwargs
@end example

which you can use like this:

@example
logger = logging.getLogger(__name__)
adapter = CustomAdapter(logger, @{'connid': some_conn_id@})
@end example

Then any events that you log to the adapter will have the value of
@code{some_conn_id} prepended to the log messages.

@menu
* Using objects other than dicts to pass contextual information:: 

@end menu

@node Using objects other than dicts to pass contextual information,,,Using LoggerAdapters to impart contextual information
@anchor{howto/logging-cookbook using-objects-other-than-dicts-to-pass-contextual-information}@anchor{46f0}
@subsubsection Using objects other than dicts to pass contextual information


You don’t need to pass an actual dict to a @code{LoggerAdapter} - you could
pass an instance of a class which implements @code{__getitem__} and @code{__iter__} so
that it looks like a dict to logging. This would be useful if you want to
generate values dynamically (whereas the values in a dict would be constant).

@node Using Filters to impart contextual information,,Using LoggerAdapters to impart contextual information,Adding contextual information to your logging output
@anchor{howto/logging-cookbook filters-contextual}@anchor{254e}@anchor{howto/logging-cookbook using-filters-to-impart-contextual-information}@anchor{46f1}
@subsubsection Using Filters to impart contextual information


You can also add contextual information to log output using a user-defined
@code{Filter}. @code{Filter} instances are allowed to modify the @code{LogRecords}
passed to them, including adding additional attributes which can then be output
using a suitable format string, or if needed a custom @code{Formatter}.

For example in a web application, the request being processed (or at least,
the interesting parts of it) can be stored in a threadlocal
(@ref{278d,,threading.local}) variable, and then accessed from a @code{Filter} to
add, say, information from the request - say, the remote IP address and remote
user’s username - to the @code{LogRecord}, using the attribute names ‘ip’ and
‘user’ as in the @code{LoggerAdapter} example above. In that case, the same format
string can be used to get similar output to that shown above. Here’s an example
script:

@example
import logging
from random import choice

class ContextFilter(logging.Filter):
    """
    This is a filter which injects contextual information into the log.

    Rather than use actual contextual information, we just use random
    data in this demo.
    """

    USERS = ['jim', 'fred', 'sheila']
    IPS = ['123.231.231.123', '127.0.0.1', '192.168.0.1']

    def filter(self, record):

        record.ip = choice(ContextFilter.IPS)
        record.user = choice(ContextFilter.USERS)
        return True

if __name__ == '__main__':
    levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL)
    logging.basicConfig(level=logging.DEBUG,
                        format='%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s')
    a1 = logging.getLogger('a.b.c')
    a2 = logging.getLogger('d.e.f')

    f = ContextFilter()
    a1.addFilter(f)
    a2.addFilter(f)
    a1.debug('A debug message')
    a1.info('An info message with %s', 'some parameters')
    for x in range(10):
        lvl = choice(levels)
        lvlname = logging.getLevelName(lvl)
        a2.log(lvl, 'A message at %s level with %d %s', lvlname, 2, 'parameters')
@end example

which, when run, produces something like:

@example
2010-09-06 22:38:15,292 a.b.c DEBUG    IP: 123.231.231.123 User: fred     A debug message
2010-09-06 22:38:15,300 a.b.c INFO     IP: 192.168.0.1     User: sheila   An info message with some parameters
2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1       User: sheila   A message at CRITICAL level with 2 parameters
2010-09-06 22:38:15,300 d.e.f ERROR    IP: 127.0.0.1       User: jim      A message at ERROR level with 2 parameters
2010-09-06 22:38:15,300 d.e.f DEBUG    IP: 127.0.0.1       User: sheila   A message at DEBUG level with 2 parameters
2010-09-06 22:38:15,300 d.e.f ERROR    IP: 123.231.231.123 User: fred     A message at ERROR level with 2 parameters
2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 192.168.0.1     User: jim      A message at CRITICAL level with 2 parameters
2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1       User: sheila   A message at CRITICAL level with 2 parameters
2010-09-06 22:38:15,300 d.e.f DEBUG    IP: 192.168.0.1     User: jim      A message at DEBUG level with 2 parameters
2010-09-06 22:38:15,301 d.e.f ERROR    IP: 127.0.0.1       User: sheila   A message at ERROR level with 2 parameters
2010-09-06 22:38:15,301 d.e.f DEBUG    IP: 123.231.231.123 User: fred     A message at DEBUG level with 2 parameters
2010-09-06 22:38:15,301 d.e.f INFO     IP: 123.231.231.123 User: fred     A message at INFO level with 2 parameters
@end example

@node Logging to a single file from multiple processes,Using file rotation,Adding contextual information to your logging output,Logging Cookbook
@anchor{howto/logging-cookbook logging-to-a-single-file-from-multiple-processes}@anchor{46f2}@anchor{howto/logging-cookbook multiple-processes}@anchor{46f3}
@subsection Logging to a single file from multiple processes


Although logging is thread-safe, and logging to a single file from multiple
threads in a single process @emph{is} supported, logging to a single file from
@emph{multiple processes} is @emph{not} supported, because there is no standard way to
serialize access to a single file across multiple processes in Python. If you
need to log to a single file from multiple processes, one way of doing this is
to have all the processes log to a @code{SocketHandler}, and have a
separate process which implements a socket server which reads from the socket
and logs to file. (If you prefer, you can dedicate one thread in one of the
existing processes to perform this function.)
@ref{46eb,,This section} documents this approach in more detail and
includes a working socket receiver which can be used as a starting point for you
to adapt in your own applications.

You could also write your own handler which uses the @ref{2825,,Lock}
class from the @ref{b3,,multiprocessing} module to serialize access to the
file from your processes. The existing @code{FileHandler} and subclasses do
not make use of @ref{b3,,multiprocessing} at present, though they may do so in the
future. Note that at present, the @ref{b3,,multiprocessing} module does not provide
working lock functionality on all platforms (see
@indicateurl{https://bugs.python.org/issue3770}).

Alternatively, you can use a @code{Queue} and a @ref{25dd,,QueueHandler} to send
all logging events to one of the processes in your multi-process application.
The following example script demonstrates how you can do this; in the example
a separate listener process listens for events sent by other processes and logs
them according to its own logging configuration. Although the example only
demonstrates one way of doing it (for example, you may want to use a listener
thread rather than a separate listener process – the implementation would be
analogous) it does allow for completely different logging configurations for
the listener and the other processes in your application, and can be used as
the basis for code meeting your own specific requirements:

@example
# You'll need these imports in your own code
import logging
import logging.handlers
import multiprocessing

# Next two import lines for this demo only
from random import choice, random
import time

#
# Because you'll want to define the logging configurations for listener and workers, the
# listener and worker process functions take a configurer parameter which is a callable
# for configuring logging for that process. These functions are also passed the queue,
# which they use for communication.
#
# In practice, you can configure the listener however you want, but note that in this
# simple example, the listener does not apply level or filter logic to received records.
# In practice, you would probably want to do this logic in the worker processes, to avoid
# sending events which would be filtered out between processes.
#
# The size of the rotated files is made small so you can see the results easily.
def listener_configurer():
    root = logging.getLogger()
    h = logging.handlers.RotatingFileHandler('mptest.log', 'a', 300, 10)
    f = logging.Formatter('%(asctime)s %(processName)-10s %(name)s %(levelname)-8s %(message)s')
    h.setFormatter(f)
    root.addHandler(h)

# This is the listener process top-level loop: wait for logging events
# (LogRecords)on the queue and handle them, quit when you get a None for a
# LogRecord.
def listener_process(queue, configurer):
    configurer()
    while True:
        try:
            record = queue.get()
            if record is None:  # We send this as a sentinel to tell the listener to quit.
                break
            logger = logging.getLogger(record.name)
            logger.handle(record)  # No level or filter logic applied - just do it!
        except Exception:
            import sys, traceback
            print('Whoops! Problem:', file=sys.stderr)
            traceback.print_exc(file=sys.stderr)

# Arrays used for random selections in this demo

LEVELS = [logging.DEBUG, logging.INFO, logging.WARNING,
          logging.ERROR, logging.CRITICAL]

LOGGERS = ['a.b.c', 'd.e.f']

MESSAGES = [
    'Random message #1',
    'Random message #2',
    'Random message #3',
]

# The worker configuration is done at the start of the worker process run.
# Note that on Windows you can't rely on fork semantics, so each process
# will run the logging configuration code when it starts.
def worker_configurer(queue):
    h = logging.handlers.QueueHandler(queue)  # Just the one handler needed
    root = logging.getLogger()
    root.addHandler(h)
    # send all messages, for demo; no other level or filter logic applied.
    root.setLevel(logging.DEBUG)

# This is the worker process top-level loop, which just logs ten events with
# random intervening delays before terminating.
# The print messages are just so you know it's doing something!
def worker_process(queue, configurer):
    configurer(queue)
    name = multiprocessing.current_process().name
    print('Worker started: %s' % name)
    for i in range(10):
        time.sleep(random())
        logger = logging.getLogger(choice(LOGGERS))
        level = choice(LEVELS)
        message = choice(MESSAGES)
        logger.log(level, message)
    print('Worker finished: %s' % name)

# Here's where the demo gets orchestrated. Create the queue, create and start
# the listener, create ten workers and start them, wait for them to finish,
# then send a None to the queue to tell the listener to finish.
def main():
    queue = multiprocessing.Queue(-1)
    listener = multiprocessing.Process(target=listener_process,
                                       args=(queue, listener_configurer))
    listener.start()
    workers = []
    for i in range(10):
        worker = multiprocessing.Process(target=worker_process,
                                         args=(queue, worker_configurer))
        workers.append(worker)
        worker.start()
    for w in workers:
        w.join()
    queue.put_nowait(None)
    listener.join()

if __name__ == '__main__':
    main()
@end example

A variant of the above script keeps the logging in the main process, in a
separate thread:

@example
import logging
import logging.config
import logging.handlers
from multiprocessing import Process, Queue
import random
import threading
import time

def logger_thread(q):
    while True:
        record = q.get()
        if record is None:
            break
        logger = logging.getLogger(record.name)
        logger.handle(record)


def worker_process(q):
    qh = logging.handlers.QueueHandler(q)
    root = logging.getLogger()
    root.setLevel(logging.DEBUG)
    root.addHandler(qh)
    levels = [logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR,
              logging.CRITICAL]
    loggers = ['foo', 'foo.bar', 'foo.bar.baz',
               'spam', 'spam.ham', 'spam.ham.eggs']
    for i in range(100):
        lvl = random.choice(levels)
        logger = logging.getLogger(random.choice(loggers))
        logger.log(lvl, 'Message no. %d', i)

if __name__ == '__main__':
    q = Queue()
    d = @{
        'version': 1,
        'formatters': @{
            'detailed': @{
                'class': 'logging.Formatter',
                'format': '%(asctime)s %(name)-15s %(levelname)-8s %(processName)-10s %(message)s'
            @}
        @},
        'handlers': @{
            'console': @{
                'class': 'logging.StreamHandler',
                'level': 'INFO',
            @},
            'file': @{
                'class': 'logging.FileHandler',
                'filename': 'mplog.log',
                'mode': 'w',
                'formatter': 'detailed',
            @},
            'foofile': @{
                'class': 'logging.FileHandler',
                'filename': 'mplog-foo.log',
                'mode': 'w',
                'formatter': 'detailed',
            @},
            'errors': @{
                'class': 'logging.FileHandler',
                'filename': 'mplog-errors.log',
                'mode': 'w',
                'level': 'ERROR',
                'formatter': 'detailed',
            @},
        @},
        'loggers': @{
            'foo': @{
                'handlers': ['foofile']
            @}
        @},
        'root': @{
            'level': 'DEBUG',
            'handlers': ['console', 'file', 'errors']
        @},
    @}
    workers = []
    for i in range(5):
        wp = Process(target=worker_process, name='worker %d' % (i + 1), args=(q,))
        workers.append(wp)
        wp.start()
    logging.config.dictConfig(d)
    lp = threading.Thread(target=logger_thread, args=(q,))
    lp.start()
    # At this point, the main process could do some useful work of its own
    # Once it's done that, it can wait for the workers to terminate...
    for wp in workers:
        wp.join()
    # And now tell the logging thread to finish up, too
    q.put(None)
    lp.join()
@end example

This variant shows how you can e.g. apply configuration for particular loggers
- e.g. the @code{foo} logger has a special handler which stores all events in the
@code{foo} subsystem in a file @code{mplog-foo.log}. This will be used by the logging
machinery in the main process (even though the logging events are generated in
the worker processes) to direct the messages to the appropriate destinations.

@menu
* Using concurrent.futures.ProcessPoolExecutor: Using concurrent futures ProcessPoolExecutor. 
* Deploying Web applications using Gunicorn and uWSGI:: 

@end menu

@node Using concurrent futures ProcessPoolExecutor,Deploying Web applications using Gunicorn and uWSGI,,Logging to a single file from multiple processes
@anchor{howto/logging-cookbook using-concurrent-futures-processpoolexecutor}@anchor{46f4}
@subsubsection Using concurrent.futures.ProcessPoolExecutor


If you want to use @ref{3da,,concurrent.futures.ProcessPoolExecutor} to start
your worker processes, you need to create the queue slightly differently.
Instead of

@example
queue = multiprocessing.Queue(-1)
@end example

you should use

@example
queue = multiprocessing.Manager().Queue(-1)  # also works with the examples above
@end example

and you can then replace the worker creation from this:

@example
workers = []
for i in range(10):
    worker = multiprocessing.Process(target=worker_process,
                                     args=(queue, worker_configurer))
    workers.append(worker)
    worker.start()
for w in workers:
    w.join()
@end example

to this (remembering to first import @ref{20,,concurrent.futures}):

@example
with concurrent.futures.ProcessPoolExecutor(max_workers=10) as executor:
    for i in range(10):
        executor.submit(worker_process, queue, worker_configurer)
@end example

@node Deploying Web applications using Gunicorn and uWSGI,,Using concurrent futures ProcessPoolExecutor,Logging to a single file from multiple processes
@anchor{howto/logging-cookbook deploying-web-applications-using-gunicorn-and-uwsgi}@anchor{46f5}
@subsubsection Deploying Web applications using Gunicorn and uWSGI


When deploying Web applications using Gunicorn@footnote{https://gunicorn.org/} or uWSGI@footnote{https://uwsgi-docs.readthedocs.io/en/latest/} (or similar), multiple worker
processes are created to handle client requests. In such environments, avoid creating
file-based handlers directly in your web application. Instead, use a
@ref{ae0,,SocketHandler} to log from the web application to a listener in a separate
process. This can be set up using a process management tool such as Supervisor - see
@ref{46ed,,Running a logging socket listener in production} for more details.

@node Using file rotation,Use of alternative formatting styles,Logging to a single file from multiple processes,Logging Cookbook
@anchor{howto/logging-cookbook using-file-rotation}@anchor{46f6}
@subsection Using file rotation


@c (see <https://pymotw.com/3/logging/>)

Sometimes you want to let a log file grow to a certain size, then open a new
file and log to that. You may want to keep a certain number of these files, and
when that many files have been created, rotate the files so that the number of
files and the size of the files both remain bounded. For this usage pattern, the
logging package provides a @code{RotatingFileHandler}:

@example
import glob
import logging
import logging.handlers

LOG_FILENAME = 'logging_rotatingfile_example.out'

# Set up a specific logger with our desired output level
my_logger = logging.getLogger('MyLogger')
my_logger.setLevel(logging.DEBUG)

# Add the log message handler to the logger
handler = logging.handlers.RotatingFileHandler(
              LOG_FILENAME, maxBytes=20, backupCount=5)

my_logger.addHandler(handler)

# Log some messages
for i in range(20):
    my_logger.debug('i = %d' % i)

# See what files are created
logfiles = glob.glob('%s*' % LOG_FILENAME)

for filename in logfiles:
    print(filename)
@end example

The result should be 6 separate files, each with part of the log history for the
application:

@example
logging_rotatingfile_example.out
logging_rotatingfile_example.out.1
logging_rotatingfile_example.out.2
logging_rotatingfile_example.out.3
logging_rotatingfile_example.out.4
logging_rotatingfile_example.out.5
@end example

The most current file is always @code{logging_rotatingfile_example.out},
and each time it reaches the size limit it is renamed with the suffix
@code{.1}. Each of the existing backup files is renamed to increment the suffix
(@code{.1} becomes @code{.2}, etc.)  and the @code{.6} file is erased.

Obviously this example sets the log length much too small as an extreme
example.  You would want to set @emph{maxBytes} to an appropriate value.

@node Use of alternative formatting styles,Customizing LogRecord,Using file rotation,Logging Cookbook
@anchor{howto/logging-cookbook format-styles}@anchor{46f7}@anchor{howto/logging-cookbook use-of-alternative-formatting-styles}@anchor{46f8}
@subsection Use of alternative formatting styles


When logging was added to the Python standard library, the only way of
formatting messages with variable content was to use the %-formatting
method. Since then, Python has gained two new formatting approaches:
@ref{6b5,,string.Template} (added in Python 2.4) and @ref{784,,str.format()}
(added in Python 2.6).

Logging (as of 3.2) provides improved support for these two additional
formatting styles. The @code{Formatter} class been enhanced to take an
additional, optional keyword parameter named @code{style}. This defaults to
@code{'%'}, but other possible values are @code{'@{'} and @code{'$'}, which correspond
to the other two formatting styles. Backwards compatibility is maintained by
default (as you would expect), but by explicitly specifying a style parameter,
you get the ability to specify format strings which work with
@ref{784,,str.format()} or @ref{6b5,,string.Template}. Here’s an example console
session to show the possibilities:

@example
>>> import logging
>>> root = logging.getLogger()
>>> root.setLevel(logging.DEBUG)
>>> handler = logging.StreamHandler()
>>> bf = logging.Formatter('@{asctime@} @{name@} @{levelname:8s@} @{message@}',
...                        style='@{')
>>> handler.setFormatter(bf)
>>> root.addHandler(handler)
>>> logger = logging.getLogger('foo.bar')
>>> logger.debug('This is a DEBUG message')
2010-10-28 15:11:55,341 foo.bar DEBUG    This is a DEBUG message
>>> logger.critical('This is a CRITICAL message')
2010-10-28 15:12:11,526 foo.bar CRITICAL This is a CRITICAL message
>>> df = logging.Formatter('$asctime $name $@{levelname@} $message',
...                        style='$')
>>> handler.setFormatter(df)
>>> logger.debug('This is a DEBUG message')
2010-10-28 15:13:06,924 foo.bar DEBUG This is a DEBUG message
>>> logger.critical('This is a CRITICAL message')
2010-10-28 15:13:11,494 foo.bar CRITICAL This is a CRITICAL message
>>>
@end example

Note that the formatting of logging messages for final output to logs is
completely independent of how an individual logging message is constructed.
That can still use %-formatting, as shown here:

@example
>>> logger.error('This is an%s %s %s', 'other,', 'ERROR,', 'message')
2010-10-28 15:19:29,833 foo.bar ERROR This is another, ERROR, message
>>>
@end example

Logging calls (@code{logger.debug()}, @code{logger.info()} etc.) only take
positional parameters for the actual logging message itself, with keyword
parameters used only for determining options for how to handle the actual
logging call (e.g. the @code{exc_info} keyword parameter to indicate that
traceback information should be logged, or the @code{extra} keyword parameter
to indicate additional contextual information to be added to the log). So
you cannot directly make logging calls using @ref{784,,str.format()} or
@ref{6b5,,string.Template} syntax, because internally the logging package
uses %-formatting to merge the format string and the variable arguments.
There would be no changing this while preserving backward compatibility, since
all logging calls which are out there in existing code will be using %-format
strings.

There is, however, a way that you can use @{@}- and $- formatting to construct
your individual log messages. Recall that for a message you can use an
arbitrary object as a message format string, and that the logging package will
call @code{str()} on that object to get the actual format string. Consider the
following two classes:

@example
class BraceMessage:
    def __init__(self, fmt, /, *args, **kwargs):
        self.fmt = fmt
        self.args = args
        self.kwargs = kwargs

    def __str__(self):
        return self.fmt.format(*self.args, **self.kwargs)

class DollarMessage:
    def __init__(self, fmt, /, **kwargs):
        self.fmt = fmt
        self.kwargs = kwargs

    def __str__(self):
        from string import Template
        return Template(self.fmt).substitute(**self.kwargs)
@end example

Either of these can be used in place of a format string, to allow @{@}- or
$-formatting to be used to build the actual “message” part which appears in the
formatted log output in place of “%(message)s” or “@{message@}” or “$message”.
It’s a little unwieldy to use the class names whenever you want to log
something, but it’s quite palatable if you use an alias such as __ (double
underscore — not to be confused with _, the single underscore used as a
synonym/alias for @ref{fc6,,gettext.gettext()} or its brethren).

The above classes are not included in Python, though they’re easy enough to
copy and paste into your own code. They can be used as follows (assuming that
they’re declared in a module called @code{wherever}):

@example
>>> from wherever import BraceMessage as __
>>> print(__('Message with @{0@} @{name@}', 2, name='placeholders'))
Message with 2 placeholders
>>> class Point: pass
...
>>> p = Point()
>>> p.x = 0.5
>>> p.y = 0.5
>>> print(__('Message with coordinates: (@{point.x:.2f@}, @{point.y:.2f@})',
...       point=p))
Message with coordinates: (0.50, 0.50)
>>> from wherever import DollarMessage as __
>>> print(__('Message with $num $what', num=2, what='placeholders'))
Message with 2 placeholders
>>>
@end example

While the above examples use @code{print()} to show how the formatting works, you
would of course use @code{logger.debug()} or similar to actually log using this
approach.

One thing to note is that you pay no significant performance penalty with this
approach: the actual formatting happens not when you make the logging call, but
when (and if) the logged message is actually about to be output to a log by a
handler. So the only slightly unusual thing which might trip you up is that the
parentheses go around the format string and the arguments, not just the format
string. That’s because the __ notation is just syntax sugar for a constructor
call to one of the XXXMessage classes.

If you prefer, you can use a @code{LoggerAdapter} to achieve a similar effect
to the above, as in the following example:

@example
import logging

class Message:
    def __init__(self, fmt, args):
        self.fmt = fmt
        self.args = args

    def __str__(self):
        return self.fmt.format(*self.args)

class StyleAdapter(logging.LoggerAdapter):
    def __init__(self, logger, extra=None):
        super().__init__(logger, extra or @{@})

    def log(self, level, msg, /, *args, **kwargs):
        if self.isEnabledFor(level):
            msg, kwargs = self.process(msg, kwargs)
            self.logger._log(level, Message(msg, args), (), **kwargs)

logger = StyleAdapter(logging.getLogger(__name__))

def main():
    logger.debug('Hello, @{@}', 'world!')

if __name__ == '__main__':
    logging.basicConfig(level=logging.DEBUG)
    main()
@end example

The above script should log the message @code{Hello, world!} when run with
Python 3.2 or later.

@node Customizing LogRecord,Subclassing QueueHandler - a ZeroMQ example,Use of alternative formatting styles,Logging Cookbook
@anchor{howto/logging-cookbook custom-logrecord}@anchor{46f9}@anchor{howto/logging-cookbook customizing-logrecord}@anchor{46fa}
@subsection Customizing @code{LogRecord}


Every logging event is represented by a @ref{b6c,,LogRecord} instance.
When an event is logged and not filtered out by a logger’s level, a
@ref{b6c,,LogRecord} is created, populated with information about the event and
then passed to the handlers for that logger (and its ancestors, up to and
including the logger where further propagation up the hierarchy is disabled).
Before Python 3.2, there were only two places where this creation was done:


@itemize *

@item 
@ref{252d,,Logger.makeRecord()}, which is called in the normal process of
logging an event. This invoked @ref{b6c,,LogRecord} directly to create an
instance.

@item 
@ref{2551,,makeLogRecord()}, which is called with a dictionary containing
attributes to be added to the LogRecord. This is typically invoked when a
suitable dictionary has been received over the network (e.g. in pickle form
via a @ref{ae0,,SocketHandler}, or in JSON form via an
@ref{992,,HTTPHandler}).
@end itemize

This has usually meant that if you need to do anything special with a
@ref{b6c,,LogRecord}, you’ve had to do one of the following.


@itemize *

@item 
Create your own @ref{676,,Logger} subclass, which overrides
@ref{252d,,Logger.makeRecord()}, and set it using @ref{255e,,setLoggerClass()}
before any loggers that you care about are instantiated.

@item 
Add a @ref{d9c,,Filter} to a logger or handler, which does the
necessary special manipulation you need when its
@ref{254c,,filter()} method is called.
@end itemize

The first approach would be a little unwieldy in the scenario where (say)
several different libraries wanted to do different things. Each would attempt
to set its own @ref{676,,Logger} subclass, and the one which did this last would
win.

The second approach works reasonably well for many cases, but does not allow
you to e.g. use a specialized subclass of @ref{b6c,,LogRecord}. Library
developers can set a suitable filter on their loggers, but they would have to
remember to do this every time they introduced a new logger (which they would
do simply by adding new packages or modules and doing

@example
logger = logging.getLogger(__name__)
@end example

at module level). It’s probably one too many things to think about. Developers
could also add the filter to a @ref{e34,,NullHandler} attached to their
top-level logger, but this would not be invoked if an application developer
attached a handler to a lower-level library logger — so output from that
handler would not reflect the intentions of the library developer.

In Python 3.2 and later, @ref{b6c,,LogRecord} creation is done through a
factory, which you can specify. The factory is just a callable you can set with
@ref{2554,,setLogRecordFactory()}, and interrogate with
@ref{2553,,getLogRecordFactory()}. The factory is invoked with the same
signature as the @ref{b6c,,LogRecord} constructor, as @ref{b6c,,LogRecord}
is the default setting for the factory.

This approach allows a custom factory to control all aspects of LogRecord
creation. For example, you could return a subclass, or just add some additional
attributes to the record once created, using a pattern similar to this:

@example
old_factory = logging.getLogRecordFactory()

def record_factory(*args, **kwargs):
    record = old_factory(*args, **kwargs)
    record.custom_attribute = 0xdecafbad
    return record

logging.setLogRecordFactory(record_factory)
@end example

This pattern allows different libraries to chain factories together, and as
long as they don’t overwrite each other’s attributes or unintentionally
overwrite the attributes provided as standard, there should be no surprises.
However, it should be borne in mind that each link in the chain adds run-time
overhead to all logging operations, and the technique should only be used when
the use of a @ref{d9c,,Filter} does not provide the desired result.

@node Subclassing QueueHandler - a ZeroMQ example,Subclassing QueueListener - a ZeroMQ example,Customizing LogRecord,Logging Cookbook
@anchor{howto/logging-cookbook subclassing-queuehandler-a-zeromq-example}@anchor{46fb}@anchor{howto/logging-cookbook zeromq-handlers}@anchor{46fc}
@subsection Subclassing QueueHandler - a ZeroMQ example


You can use a @code{QueueHandler} subclass to send messages to other kinds
of queues, for example a ZeroMQ ‘publish’ socket. In the example below,the
socket is created separately and passed to the handler (as its ‘queue’):

@example
import zmq   # using pyzmq, the Python binding for ZeroMQ
import json  # for serializing records portably

ctx = zmq.Context()
sock = zmq.Socket(ctx, zmq.PUB)  # or zmq.PUSH, or other suitable value
sock.bind('tcp://*:5556')        # or wherever

class ZeroMQSocketHandler(QueueHandler):
    def enqueue(self, record):
        self.queue.send_json(record.__dict__)


handler = ZeroMQSocketHandler(sock)
@end example

Of course there are other ways of organizing this, for example passing in the
data needed by the handler to create the socket:

@example
class ZeroMQSocketHandler(QueueHandler):
    def __init__(self, uri, socktype=zmq.PUB, ctx=None):
        self.ctx = ctx or zmq.Context()
        socket = zmq.Socket(self.ctx, socktype)
        socket.bind(uri)
        super().__init__(socket)

    def enqueue(self, record):
        self.queue.send_json(record.__dict__)

    def close(self):
        self.queue.close()
@end example

@node Subclassing QueueListener - a ZeroMQ example,An example dictionary-based configuration,Subclassing QueueHandler - a ZeroMQ example,Logging Cookbook
@anchor{howto/logging-cookbook subclassing-queuelistener-a-zeromq-example}@anchor{46fd}
@subsection Subclassing QueueListener - a ZeroMQ example


You can also subclass @code{QueueListener} to get messages from other kinds
of queues, for example a ZeroMQ ‘subscribe’ socket. Here’s an example:

@example
class ZeroMQSocketListener(QueueListener):
    def __init__(self, uri, /, *handlers, **kwargs):
        self.ctx = kwargs.get('ctx') or zmq.Context()
        socket = zmq.Socket(self.ctx, zmq.SUB)
        socket.setsockopt_string(zmq.SUBSCRIBE, '@w{'})  # subscribe to everything
        socket.connect(uri)
        super().__init__(socket, *handlers, **kwargs)

    def dequeue(self):
        msg = self.queue.recv_json()
        return logging.makeLogRecord(msg)
@end example


@subsubheading See also



@table @asis

@item Module @ref{a6,,logging}

API reference for the logging module.

@item Module @ref{a7,,logging.config}

Configuration API for the logging module.

@item Module @ref{a8,,logging.handlers}

Useful handlers included with the logging module.
@end table

@ref{d97,,A basic logging tutorial}

@ref{d98,,A more advanced logging tutorial}


@node An example dictionary-based configuration,Using a rotator and namer to customize log rotation processing,Subclassing QueueListener - a ZeroMQ example,Logging Cookbook
@anchor{howto/logging-cookbook an-example-dictionary-based-configuration}@anchor{46fe}
@subsection An example dictionary-based configuration


Below is an example of a logging configuration dictionary - it’s taken from
the documentation on the Django project@footnote{https://docs.djangoproject.com/en/stable/topics/logging/#configuring-logging}.
This dictionary is passed to @ref{d55,,dictConfig()} to put the configuration into effect:

@example
LOGGING = @{
    'version': 1,
    'disable_existing_loggers': True,
    'formatters': @{
        'verbose': @{
            'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
        @},
        'simple': @{
            'format': '%(levelname)s %(message)s'
        @},
    @},
    'filters': @{
        'special': @{
            '()': 'project.logging.SpecialFilter',
            'foo': 'bar',
        @}
    @},
    'handlers': @{
        'null': @{
            'level':'DEBUG',
            'class':'django.utils.log.NullHandler',
        @},
        'console':@{
            'level':'DEBUG',
            'class':'logging.StreamHandler',
            'formatter': 'simple'
        @},
        'mail_admins': @{
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler',
            'filters': ['special']
        @}
    @},
    'loggers': @{
        'django': @{
            'handlers':['null'],
            'propagate': True,
            'level':'INFO',
        @},
        'django.request': @{
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': False,
        @},
        'myproject.custom': @{
            'handlers': ['console', 'mail_admins'],
            'level': 'INFO',
            'filters': ['special']
        @}
    @}
@}
@end example

For more information about this configuration, you can see the relevant section@footnote{https://docs.djangoproject.com/en/stable/topics/logging/#configuring-logging}
of the Django documentation.

@node Using a rotator and namer to customize log rotation processing,A more elaborate multiprocessing example,An example dictionary-based configuration,Logging Cookbook
@anchor{howto/logging-cookbook cookbook-rotator-namer}@anchor{25a3}@anchor{howto/logging-cookbook using-a-rotator-and-namer-to-customize-log-rotation-processing}@anchor{46ff}
@subsection Using a rotator and namer to customize log rotation processing


An example of how you can define a namer and rotator is given in the following
snippet, which shows zlib-based compression of the log file:

@example
def namer(name):
    return name + ".gz"

def rotator(source, dest):
    with open(source, "rb") as sf:
        data = sf.read()
        compressed = zlib.compress(data, 9)
        with open(dest, "wb") as df:
            df.write(compressed)
    os.remove(source)

rh = logging.handlers.RotatingFileHandler(...)
rh.rotator = rotator
rh.namer = namer
@end example

These are not “true” .gz files, as they are bare compressed data, with no
“container” such as you’d find in an actual gzip file. This snippet is just
for illustration purposes.

@node A more elaborate multiprocessing example,Inserting a BOM into messages sent to a SysLogHandler,Using a rotator and namer to customize log rotation processing,Logging Cookbook
@anchor{howto/logging-cookbook a-more-elaborate-multiprocessing-example}@anchor{4700}
@subsection A more elaborate multiprocessing example


The following working example shows how logging can be used with multiprocessing
using configuration files. The configurations are fairly simple, but serve to
illustrate how more complex ones could be implemented in a real multiprocessing
scenario.

In the example, the main process spawns a listener process and some worker
processes. Each of the main process, the listener and the workers have three
separate configurations (the workers all share the same configuration). We can
see logging in the main process, how the workers log to a QueueHandler and how
the listener implements a QueueListener and a more complex logging
configuration, and arranges to dispatch events received via the queue to the
handlers specified in the configuration. Note that these configurations are
purely illustrative, but you should be able to adapt this example to your own
scenario.

Here’s the script - the docstrings and the comments hopefully explain how it
works:

@example
import logging
import logging.config
import logging.handlers
from multiprocessing import Process, Queue, Event, current_process
import os
import random
import time

class MyHandler:
    """
    A simple handler for logging events. It runs in the listener process and
    dispatches events to loggers based on the name in the received record,
    which then get dispatched, by the logging system, to the handlers
    configured for those loggers.
    """

    def handle(self, record):
        if record.name == "root":
            logger = logging.getLogger()
        else:
            logger = logging.getLogger(record.name)

        if logger.isEnabledFor(record.levelno):
            # The process name is transformed just to show that it's the listener
            # doing the logging to files and console
            record.processName = '%s (for %s)' % (current_process().name, record.processName)
            logger.handle(record)

def listener_process(q, stop_event, config):
    """
    This could be done in the main process, but is just done in a separate
    process for illustrative purposes.

    This initialises logging according to the specified configuration,
    starts the listener and waits for the main process to signal completion
    via the event. The listener is then stopped, and the process exits.
    """
    logging.config.dictConfig(config)
    listener = logging.handlers.QueueListener(q, MyHandler())
    listener.start()
    if os.name == 'posix':
        # On POSIX, the setup logger will have been configured in the
        # parent process, but should have been disabled following the
        # dictConfig call.
        # On Windows, since fork isn't used, the setup logger won't
        # exist in the child, so it would be created and the message
        # would appear - hence the "if posix" clause.
        logger = logging.getLogger('setup')
        logger.critical('Should not appear, because of disabled logger ...')
    stop_event.wait()
    listener.stop()

def worker_process(config):
    """
    A number of these are spawned for the purpose of illustration. In
    practice, they could be a heterogeneous bunch of processes rather than
    ones which are identical to each other.

    This initialises logging according to the specified configuration,
    and logs a hundred messages with random levels to randomly selected
    loggers.

    A small sleep is added to allow other processes a chance to run. This
    is not strictly needed, but it mixes the output from the different
    processes a bit more than if it's left out.
    """
    logging.config.dictConfig(config)
    levels = [logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR,
              logging.CRITICAL]
    loggers = ['foo', 'foo.bar', 'foo.bar.baz',
               'spam', 'spam.ham', 'spam.ham.eggs']
    if os.name == 'posix':
        # On POSIX, the setup logger will have been configured in the
        # parent process, but should have been disabled following the
        # dictConfig call.
        # On Windows, since fork isn't used, the setup logger won't
        # exist in the child, so it would be created and the message
        # would appear - hence the "if posix" clause.
        logger = logging.getLogger('setup')
        logger.critical('Should not appear, because of disabled logger ...')
    for i in range(100):
        lvl = random.choice(levels)
        logger = logging.getLogger(random.choice(loggers))
        logger.log(lvl, 'Message no. %d', i)
        time.sleep(0.01)

def main():
    q = Queue()
    # The main process gets a simple configuration which prints to the console.
    config_initial = @{
        'version': 1,
        'handlers': @{
            'console': @{
                'class': 'logging.StreamHandler',
                'level': 'INFO'
            @}
        @},
        'root': @{
            'handlers': ['console'],
            'level': 'DEBUG'
        @}
    @}
    # The worker process configuration is just a QueueHandler attached to the
    # root logger, which allows all messages to be sent to the queue.
    # We disable existing loggers to disable the "setup" logger used in the
    # parent process. This is needed on POSIX because the logger will
    # be there in the child following a fork().
    config_worker = @{
        'version': 1,
        'disable_existing_loggers': True,
        'handlers': @{
            'queue': @{
                'class': 'logging.handlers.QueueHandler',
                'queue': q
            @}
        @},
        'root': @{
            'handlers': ['queue'],
            'level': 'DEBUG'
        @}
    @}
    # The listener process configuration shows that the full flexibility of
    # logging configuration is available to dispatch events to handlers however
    # you want.
    # We disable existing loggers to disable the "setup" logger used in the
    # parent process. This is needed on POSIX because the logger will
    # be there in the child following a fork().
    config_listener = @{
        'version': 1,
        'disable_existing_loggers': True,
        'formatters': @{
            'detailed': @{
                'class': 'logging.Formatter',
                'format': '%(asctime)s %(name)-15s %(levelname)-8s %(processName)-10s %(message)s'
            @},
            'simple': @{
                'class': 'logging.Formatter',
                'format': '%(name)-15s %(levelname)-8s %(processName)-10s %(message)s'
            @}
        @},
        'handlers': @{
            'console': @{
                'class': 'logging.StreamHandler',
                'formatter': 'simple',
                'level': 'INFO'
            @},
            'file': @{
                'class': 'logging.FileHandler',
                'filename': 'mplog.log',
                'mode': 'w',
                'formatter': 'detailed'
            @},
            'foofile': @{
                'class': 'logging.FileHandler',
                'filename': 'mplog-foo.log',
                'mode': 'w',
                'formatter': 'detailed'
            @},
            'errors': @{
                'class': 'logging.FileHandler',
                'filename': 'mplog-errors.log',
                'mode': 'w',
                'formatter': 'detailed',
                'level': 'ERROR'
            @}
        @},
        'loggers': @{
            'foo': @{
                'handlers': ['foofile']
            @}
        @},
        'root': @{
            'handlers': ['console', 'file', 'errors'],
            'level': 'DEBUG'
        @}
    @}
    # Log some initial events, just to show that logging in the parent works
    # normally.
    logging.config.dictConfig(config_initial)
    logger = logging.getLogger('setup')
    logger.info('About to create workers ...')
    workers = []
    for i in range(5):
        wp = Process(target=worker_process, name='worker %d' % (i + 1),
                     args=(config_worker,))
        workers.append(wp)
        wp.start()
        logger.info('Started worker: %s', wp.name)
    logger.info('About to create listener ...')
    stop_event = Event()
    lp = Process(target=listener_process, name='listener',
                 args=(q, stop_event, config_listener))
    lp.start()
    logger.info('Started listener')
    # We now hang around for the workers to finish their work.
    for wp in workers:
        wp.join()
    # Workers all done, listening can now stop.
    # Logging in the parent still works normally.
    logger.info('Telling listener to stop ...')
    stop_event.set()
    lp.join()
    logger.info('All done.')

if __name__ == '__main__':
    main()
@end example

@node Inserting a BOM into messages sent to a SysLogHandler,Implementing structured logging,A more elaborate multiprocessing example,Logging Cookbook
@anchor{howto/logging-cookbook inserting-a-bom-into-messages-sent-to-a-sysloghandler}@anchor{4701}
@subsection Inserting a BOM into messages sent to a SysLogHandler


@geindex RFC; RFC 5424
RFC 5424@footnote{https://tools.ietf.org/html/rfc5424.html} requires that a
Unicode message be sent to a syslog daemon as a set of bytes which have the
following structure: an optional pure-ASCII component, followed by a UTF-8 Byte
Order Mark (BOM), followed by Unicode encoded using UTF-8. (See the
@geindex RFC; RFC 5424#section-6
relevant section of the specification@footnote{https://tools.ietf.org/html/rfc5424.html#section-6}.)

In Python 3.1, code was added to
@ref{c6c,,SysLogHandler} to insert a BOM into the message, but
unfortunately, it was implemented incorrectly, with the BOM appearing at the
beginning of the message and hence not allowing any pure-ASCII component to
appear before it.

As this behaviour is broken, the incorrect BOM insertion code is being removed
from Python 3.2.4 and later. However, it is not being replaced, and if you
want to produce 
@geindex RFC; RFC 5424
RFC 5424@footnote{https://tools.ietf.org/html/rfc5424.html}-compliant messages which include a BOM, an optional
pure-ASCII sequence before it and arbitrary Unicode after it, encoded using
UTF-8, then you need to do the following:


@enumerate 

@item 
Attach a @ref{11c6,,Formatter} instance to your
@ref{c6c,,SysLogHandler} instance, with a format string
such as:

@example
'ASCII section\ufeffUnicode section'
@end example

The Unicode code point U+FEFF, when encoded using UTF-8, will be
encoded as a UTF-8 BOM – the byte-string @code{b'\xef\xbb\xbf'}.

@item 
Replace the ASCII section with whatever placeholders you like, but make sure
that the data that appears in there after substitution is always ASCII (that
way, it will remain unchanged after UTF-8 encoding).

@item 
Replace the Unicode section with whatever placeholders you like; if the data
which appears there after substitution contains characters outside the ASCII
range, that’s fine – it will be encoded using UTF-8.
@end enumerate

The formatted message @emph{will} be encoded using UTF-8 encoding by
@code{SysLogHandler}. If you follow the above rules, you should be able to produce
@geindex RFC; RFC 5424
RFC 5424@footnote{https://tools.ietf.org/html/rfc5424.html}-compliant messages. If you don’t, logging may not complain, but your
messages will not be RFC 5424-compliant, and your syslog daemon may complain.

@node Implementing structured logging,Customizing handlers with dictConfig,Inserting a BOM into messages sent to a SysLogHandler,Logging Cookbook
@anchor{howto/logging-cookbook implementing-structured-logging}@anchor{4702}
@subsection Implementing structured logging


Although most logging messages are intended for reading by humans, and thus not
readily machine-parseable, there might be circumstances where you want to output
messages in a structured format which @emph{is} capable of being parsed by a program
(without needing complex regular expressions to parse the log message). This is
straightforward to achieve using the logging package. There are a number of
ways in which this could be achieved, but the following is a simple approach
which uses JSON to serialise the event in a machine-parseable manner:

@example
import json
import logging

class StructuredMessage:
    def __init__(self, message, /, **kwargs):
        self.message = message
        self.kwargs = kwargs

    def __str__(self):
        return '%s >>> %s' % (self.message, json.dumps(self.kwargs))

_ = StructuredMessage   # optional, to improve readability

logging.basicConfig(level=logging.INFO, format='%(message)s')
logging.info(_('message 1', foo='bar', bar='baz', num=123, fnum=123.456))
@end example

If the above script is run, it prints:

@example
message 1 >>> @{"fnum": 123.456, "num": 123, "bar": "baz", "foo": "bar"@}
@end example

Note that the order of items might be different according to the version of
Python used.

If you need more specialised processing, you can use a custom JSON encoder,
as in the following complete example:

@example
from __future__ import unicode_literals

import json
import logging

# This next bit is to ensure the script runs unchanged on 2.x and 3.x
try:
    unicode
except NameError:
    unicode = str

class Encoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, set):
            return tuple(o)
        elif isinstance(o, unicode):
            return o.encode('unicode_escape').decode('ascii')
        return super().default(o)

class StructuredMessage:
    def __init__(self, message, /, **kwargs):
        self.message = message
        self.kwargs = kwargs

    def __str__(self):
        s = Encoder().encode(self.kwargs)
        return '%s >>> %s' % (self.message, s)

_ = StructuredMessage   # optional, to improve readability

def main():
    logging.basicConfig(level=logging.INFO, format='%(message)s')
    logging.info(_('message 1', set_value=@{1, 2, 3@}, snowman='\u2603'))

if __name__ == '__main__':
    main()
@end example

When the above script is run, it prints:

@example
message 1 >>> @{"snowman": "\u2603", "set_value": [1, 2, 3]@}
@end example

Note that the order of items might be different according to the version of
Python used.

@node Customizing handlers with dictConfig,Using particular formatting styles throughout your application,Implementing structured logging,Logging Cookbook
@anchor{howto/logging-cookbook custom-handlers}@anchor{4703}@anchor{howto/logging-cookbook customizing-handlers-with-dictconfig}@anchor{4704}
@subsection Customizing handlers with @code{dictConfig()}


There are times when you want to customize logging handlers in particular ways,
and if you use @ref{d55,,dictConfig()} you may be able to do this without
subclassing. As an example, consider that you may want to set the ownership of a
log file. On POSIX, this is easily done using @ref{cc2,,shutil.chown()}, but the file
handlers in the stdlib don’t offer built-in support. You can customize handler
creation using a plain function such as:

@example
def owned_file_handler(filename, mode='a', encoding=None, owner=None):
    if owner:
        if not os.path.exists(filename):
            open(filename, 'a').close()
        shutil.chown(filename, *owner)
    return logging.FileHandler(filename, mode, encoding)
@end example

You can then specify, in a logging configuration passed to @ref{d55,,dictConfig()},
that a logging handler be created by calling this function:

@example
LOGGING = @{
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': @{
        'default': @{
            'format': '%(asctime)s %(levelname)s %(name)s %(message)s'
        @},
    @},
    'handlers': @{
        'file':@{
            # The values below are popped from this dictionary and
            # used to create the handler, set the handler's level and
            # its formatter.
            '()': owned_file_handler,
            'level':'DEBUG',
            'formatter': 'default',
            # The values below are passed to the handler creator callable
            # as keyword arguments.
            'owner': ['pulse', 'pulse'],
            'filename': 'chowntest.log',
            'mode': 'w',
            'encoding': 'utf-8',
        @},
    @},
    'root': @{
        'handlers': ['file'],
        'level': 'DEBUG',
    @},
@}
@end example

In this example I am setting the ownership using the @code{pulse} user and group,
just for the purposes of illustration. Putting it together into a working
script, @code{chowntest.py}:

@example
import logging, logging.config, os, shutil

def owned_file_handler(filename, mode='a', encoding=None, owner=None):
    if owner:
        if not os.path.exists(filename):
            open(filename, 'a').close()
        shutil.chown(filename, *owner)
    return logging.FileHandler(filename, mode, encoding)

LOGGING = @{
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': @{
        'default': @{
            'format': '%(asctime)s %(levelname)s %(name)s %(message)s'
        @},
    @},
    'handlers': @{
        'file':@{
            # The values below are popped from this dictionary and
            # used to create the handler, set the handler's level and
            # its formatter.
            '()': owned_file_handler,
            'level':'DEBUG',
            'formatter': 'default',
            # The values below are passed to the handler creator callable
            # as keyword arguments.
            'owner': ['pulse', 'pulse'],
            'filename': 'chowntest.log',
            'mode': 'w',
            'encoding': 'utf-8',
        @},
    @},
    'root': @{
        'handlers': ['file'],
        'level': 'DEBUG',
    @},
@}

logging.config.dictConfig(LOGGING)
logger = logging.getLogger('mylogger')
logger.debug('A debug message')
@end example

To run this, you will probably need to run as @code{root}:

@example
$ sudo python3.3 chowntest.py
$ cat chowntest.log
2013-11-05 09:34:51,128 DEBUG mylogger A debug message
$ ls -l chowntest.log
-rw-r--r-- 1 pulse pulse 55 2013-11-05 09:34 chowntest.log
@end example

Note that this example uses Python 3.3 because that’s where @ref{cc2,,shutil.chown()}
makes an appearance. This approach should work with any Python version that
supports @ref{d55,,dictConfig()} - namely, Python 2.7, 3.2 or later. With pre-3.3
versions, you would need to implement the actual ownership change using e.g.
@ref{c81,,os.chown()}.

In practice, the handler-creating function may be in a utility module somewhere
in your project. Instead of the line in the configuration:

@example
'()': owned_file_handler,
@end example

you could use e.g.:

@example
'()': 'ext://project.util.owned_file_handler',
@end example

where @code{project.util} can be replaced with the actual name of the package
where the function resides. In the above working script, using
@code{'ext://__main__.owned_file_handler'} should work. Here, the actual callable
is resolved by @ref{d55,,dictConfig()} from the @code{ext://} specification.

This example hopefully also points the way to how you could implement other
types of file change - e.g. setting specific POSIX permission bits - in the
same way, using @ref{c80,,os.chmod()}.

Of course, the approach could also be extended to types of handler other than a
@ref{1104,,FileHandler} - for example, one of the rotating file handlers,
or a different type of handler altogether.

@node Using particular formatting styles throughout your application,Configuring filters with dictConfig,Customizing handlers with dictConfig,Logging Cookbook
@anchor{howto/logging-cookbook formatting-styles}@anchor{2546}@anchor{howto/logging-cookbook using-particular-formatting-styles-throughout-your-application}@anchor{4705}
@subsection Using particular formatting styles throughout your application


In Python 3.2, the @ref{11c6,,Formatter} gained a @code{style} keyword
parameter which, while defaulting to @code{%} for backward compatibility, allowed
the specification of @code{@{} or @code{$} to support the formatting approaches
supported by @ref{784,,str.format()} and @ref{6b5,,string.Template}. Note that this
governs the formatting of logging messages for final output to logs, and is
completely orthogonal to how an individual logging message is constructed.

Logging calls (@ref{991,,debug()}, @ref{2522,,info()} etc.) only take
positional parameters for the actual logging message itself, with keyword
parameters used only for determining options for how to handle the logging call
(e.g. the @code{exc_info} keyword parameter to indicate that traceback information
should be logged, or the @code{extra} keyword parameter to indicate additional
contextual information to be added to the log). So you cannot directly make
logging calls using @ref{784,,str.format()} or @ref{6b5,,string.Template} syntax,
because internally the logging package uses %-formatting to merge the format
string and the variable arguments. There would no changing this while preserving
backward compatibility, since all logging calls which are out there in existing
code will be using %-format strings.

There have been suggestions to associate format styles with specific loggers,
but that approach also runs into backward compatibility problems because any
existing code could be using a given logger name and using %-formatting.

For logging to work interoperably between any third-party libraries and your
code, decisions about formatting need to be made at the level of the
individual logging call. This opens up a couple of ways in which alternative
formatting styles can be accommodated.

@menu
* Using LogRecord factories:: 
* Using custom message objects:: 

@end menu

@node Using LogRecord factories,Using custom message objects,,Using particular formatting styles throughout your application
@anchor{howto/logging-cookbook using-logrecord-factories}@anchor{4706}
@subsubsection Using LogRecord factories


In Python 3.2, along with the @ref{11c6,,Formatter} changes mentioned
above, the logging package gained the ability to allow users to set their own
@ref{b6c,,LogRecord} subclasses, using the @ref{2554,,setLogRecordFactory()} function.
You can use this to set your own subclass of @ref{b6c,,LogRecord}, which does the
Right Thing by overriding the @ref{2552,,getMessage()} method. The base
class implementation of this method is where the @code{msg % args} formatting
happens, and where you can substitute your alternate formatting; however, you
should be careful to support all formatting styles and allow %-formatting as
the default, to ensure interoperability with other code. Care should also be
taken to call @code{str(self.msg)}, just as the base implementation does.

Refer to the reference documentation on @ref{2554,,setLogRecordFactory()} and
@ref{b6c,,LogRecord} for more information.

@node Using custom message objects,,Using LogRecord factories,Using particular formatting styles throughout your application
@anchor{howto/logging-cookbook using-custom-message-objects}@anchor{4707}
@subsubsection Using custom message objects


There is another, perhaps simpler way that you can use @{@}- and $- formatting to
construct your individual log messages. You may recall (from
@ref{2556,,Using arbitrary objects as messages}) that when logging you can use an arbitrary
object as a message format string, and that the logging package will call
@ref{2ef,,str()} on that object to get the actual format string. Consider the
following two classes:

@example
class BraceMessage:
    def __init__(self, fmt, /, *args, **kwargs):
        self.fmt = fmt
        self.args = args
        self.kwargs = kwargs

    def __str__(self):
        return self.fmt.format(*self.args, **self.kwargs)

class DollarMessage:
    def __init__(self, fmt, /, **kwargs):
        self.fmt = fmt
        self.kwargs = kwargs

    def __str__(self):
        from string import Template
        return Template(self.fmt).substitute(**self.kwargs)
@end example

Either of these can be used in place of a format string, to allow @{@}- or
$-formatting to be used to build the actual “message” part which appears in the
formatted log output in place of “%(message)s” or “@{message@}” or “$message”.
If you find it a little unwieldy to use the class names whenever you want to log
something, you can make it more palatable if you use an alias such as @code{M} or
@code{_} for the message (or perhaps @code{__}, if you are using @code{_} for
localization).

Examples of this approach are given below. Firstly, formatting with
@ref{784,,str.format()}:

@example
>>> __ = BraceMessage
>>> print(__('Message with @{0@} @{1@}', 2, 'placeholders'))
Message with 2 placeholders
>>> class Point: pass
...
>>> p = Point()
>>> p.x = 0.5
>>> p.y = 0.5
>>> print(__('Message with coordinates: (@{point.x:.2f@}, @{point.y:.2f@})', point=p))
Message with coordinates: (0.50, 0.50)
@end example

Secondly, formatting with @ref{6b5,,string.Template}:

@example
>>> __ = DollarMessage
>>> print(__('Message with $num $what', num=2, what='placeholders'))
Message with 2 placeholders
>>>
@end example

One thing to note is that you pay no significant performance penalty with this
approach: the actual formatting happens not when you make the logging call, but
when (and if) the logged message is actually about to be output to a log by a
handler. So the only slightly unusual thing which might trip you up is that the
parentheses go around the format string and the arguments, not just the format
string. That’s because the __ notation is just syntax sugar for a constructor
call to one of the @code{XXXMessage} classes shown above.

@node Configuring filters with dictConfig,Customized exception formatting,Using particular formatting styles throughout your application,Logging Cookbook
@anchor{howto/logging-cookbook configuring-filters-with-dictconfig}@anchor{4708}@anchor{howto/logging-cookbook filters-dictconfig}@anchor{4709}
@subsection Configuring filters with @code{dictConfig()}


You @emph{can} configure filters using @ref{d55,,dictConfig()}, though it
might not be obvious at first glance how to do it (hence this recipe). Since
@ref{d9c,,Filter} is the only filter class included in the standard
library, and it is unlikely to cater to many requirements (it’s only there as a
base class), you will typically need to define your own @ref{d9c,,Filter}
subclass with an overridden @ref{254c,,filter()} method. To do this,
specify the @code{()} key in the configuration dictionary for the filter,
specifying a callable which will be used to create the filter (a class is the
most obvious, but you can provide any callable which returns a
@ref{d9c,,Filter} instance). Here is a complete example:

@example
import logging
import logging.config
import sys

class MyFilter(logging.Filter):
    def __init__(self, param=None):
        self.param = param

    def filter(self, record):
        if self.param is None:
            allow = True
        else:
            allow = self.param not in record.msg
        if allow:
            record.msg = 'changed: ' + record.msg
        return allow

LOGGING = @{
    'version': 1,
    'filters': @{
        'myfilter': @{
            '()': MyFilter,
            'param': 'noshow',
        @}
    @},
    'handlers': @{
        'console': @{
            'class': 'logging.StreamHandler',
            'filters': ['myfilter']
        @}
    @},
    'root': @{
        'level': 'DEBUG',
        'handlers': ['console']
    @},
@}

if __name__ == '__main__':
    logging.config.dictConfig(LOGGING)
    logging.debug('hello')
    logging.debug('hello - noshow')
@end example

This example shows how you can pass configuration data to the callable which
constructs the instance, in the form of keyword parameters. When run, the above
script will print:

@example
changed: hello
@end example

which shows that the filter is working as configured.

A couple of extra points to note:


@itemize *

@item 
If you can’t refer to the callable directly in the configuration (e.g. if it
lives in a different module, and you can’t import it directly where the
configuration dictionary is), you can use the form @code{ext://...} as described
in @ref{257f,,Access to external objects}. For example, you could have used
the text @code{'ext://__main__.MyFilter'} instead of @code{MyFilter} in the above
example.

@item 
As well as for filters, this technique can also be used to configure custom
handlers and formatters. See @ref{2574,,User-defined objects} for more
information on how logging supports using user-defined objects in its
configuration, and see the other cookbook recipe @ref{4703,,Customizing handlers with dictConfig()} above.
@end itemize

@node Customized exception formatting,Speaking logging messages,Configuring filters with dictConfig,Logging Cookbook
@anchor{howto/logging-cookbook custom-format-exception}@anchor{470a}@anchor{howto/logging-cookbook customized-exception-formatting}@anchor{470b}
@subsection Customized exception formatting


There might be times when you want to do customized exception formatting - for
argument’s sake, let’s say you want exactly one line per logged event, even
when exception information is present. You can do this with a custom formatter
class, as shown in the following example:

@example
import logging

class OneLineExceptionFormatter(logging.Formatter):
    def formatException(self, exc_info):
        """
        Format an exception so that it prints on a single line.
        """
        result = super().formatException(exc_info)
        return repr(result)  # or format into one line however you want to

    def format(self, record):
        s = super().format(record)
        if record.exc_text:
            s = s.replace('\n', '@w{'}) + '|'
        return s

def configure_logging():
    fh = logging.FileHandler('output.txt', 'w')
    f = OneLineExceptionFormatter('%(asctime)s|%(levelname)s|%(message)s|',
                                  '%d/%m/%Y %H:%M:%S')
    fh.setFormatter(f)
    root = logging.getLogger()
    root.setLevel(logging.DEBUG)
    root.addHandler(fh)

def main():
    configure_logging()
    logging.info('Sample message')
    try:
        x = 1 / 0
    except ZeroDivisionError as e:
        logging.exception('ZeroDivisionError: %s', e)

if __name__ == '__main__':
    main()
@end example

When run, this produces a file with exactly two lines:

@example
28/01/2015 07:21:23|INFO|Sample message|
28/01/2015 07:21:23|ERROR|ZeroDivisionError: integer division or modulo by zero|'Traceback (most recent call last):\n  File "logtest7.py", line 30, in main\n    x = 1 / 0\nZeroDivisionError: integer division or modulo by zero'|
@end example

While the above treatment is simplistic, it points the way to how exception
information can be formatted to your liking. The @ref{11a,,traceback} module may be
helpful for more specialized needs.

@node Speaking logging messages,Buffering logging messages and outputting them conditionally,Customized exception formatting,Logging Cookbook
@anchor{howto/logging-cookbook speaking-logging-messages}@anchor{470c}@anchor{howto/logging-cookbook spoken-messages}@anchor{470d}
@subsection Speaking logging messages


There might be situations when it is desirable to have logging messages rendered
in an audible rather than a visible format. This is easy to do if you have
text-to-speech (TTS) functionality available in your system, even if it doesn’t have
a Python binding. Most TTS systems have a command line program you can run, and
this can be invoked from a handler using @ref{f4,,subprocess}. It’s assumed here
that TTS command line programs won’t expect to interact with users or take a
long time to complete, and that the frequency of logged messages will be not so
high as to swamp the user with messages, and that it’s acceptable to have the
messages spoken one at a time rather than concurrently, The example implementation
below waits for one message to be spoken before the next is processed, and this
might cause other handlers to be kept waiting. Here is a short example showing
the approach, which assumes that the @code{espeak} TTS package is available:

@example
import logging
import subprocess
import sys

class TTSHandler(logging.Handler):
    def emit(self, record):
        msg = self.format(record)
        # Speak slowly in a female English voice
        cmd = ['espeak', '-s150', '-ven+f3', msg]
        p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
                             stderr=subprocess.STDOUT)
        # wait for the program to finish
        p.communicate()

def configure_logging():
    h = TTSHandler()
    root = logging.getLogger()
    root.addHandler(h)
    # the default formatter just returns the message
    root.setLevel(logging.DEBUG)

def main():
    logging.info('Hello')
    logging.debug('Goodbye')

if __name__ == '__main__':
    configure_logging()
    sys.exit(main())
@end example

When run, this script should say “Hello” and then “Goodbye” in a female voice.

The above approach can, of course, be adapted to other TTS systems and even
other systems altogether which can process messages via external programs run
from a command line.

@node Buffering logging messages and outputting them conditionally,Formatting times using UTC GMT via configuration,Speaking logging messages,Logging Cookbook
@anchor{howto/logging-cookbook buffered-logging}@anchor{470e}@anchor{howto/logging-cookbook buffering-logging-messages-and-outputting-them-conditionally}@anchor{470f}
@subsection Buffering logging messages and outputting them conditionally


There might be situations where you want to log messages in a temporary area
and only output them if a certain condition occurs. For example, you may want to
start logging debug events in a function, and if the function completes without
errors, you don’t want to clutter the log with the collected debug information,
but if there is an error, you want all the debug information to be output as well
as the error.

Here is an example which shows how you could do this using a decorator for your
functions where you want logging to behave this way. It makes use of the
@ref{2582,,logging.handlers.MemoryHandler}, which allows buffering of logged events
until some condition occurs, at which point the buffered events are @code{flushed}
- passed to another handler (the @code{target} handler) for processing. By default,
the @code{MemoryHandler} flushed when its buffer gets filled up or an event whose
level is greater than or equal to a specified threshold is seen. You can use this
recipe with a more specialised subclass of @code{MemoryHandler} if you want custom
flushing behavior.

The example script has a simple function, @code{foo}, which just cycles through
all the logging levels, writing to @code{sys.stderr} to say what level it’s about
to log at, and then actually logging a message at that level. You can pass a
parameter to @code{foo} which, if true, will log at ERROR and CRITICAL levels -
otherwise, it only logs at DEBUG, INFO and WARNING levels.

The script just arranges to decorate @code{foo} with a decorator which will do the
conditional logging that’s required. The decorator takes a logger as a parameter
and attaches a memory handler for the duration of the call to the decorated
function. The decorator can be additionally parameterised using a target handler,
a level at which flushing should occur, and a capacity for the buffer (number of
records buffered). These default to a @ref{d9a,,StreamHandler} which
writes to @code{sys.stderr}, @code{logging.ERROR} and @code{100} respectively.

Here’s the script:

@example
import logging
from logging.handlers import MemoryHandler
import sys

logger = logging.getLogger(__name__)
logger.addHandler(logging.NullHandler())

def log_if_errors(logger, target_handler=None, flush_level=None, capacity=None):
    if target_handler is None:
        target_handler = logging.StreamHandler()
    if flush_level is None:
        flush_level = logging.ERROR
    if capacity is None:
        capacity = 100
    handler = MemoryHandler(capacity, flushLevel=flush_level, target=target_handler)

    def decorator(fn):
        def wrapper(*args, **kwargs):
            logger.addHandler(handler)
            try:
                return fn(*args, **kwargs)
            except Exception:
                logger.exception('call failed')
                raise
            finally:
                super(MemoryHandler, handler).flush()
                logger.removeHandler(handler)
        return wrapper

    return decorator

def write_line(s):
    sys.stderr.write('%s\n' % s)

def foo(fail=False):
    write_line('about to log at DEBUG ...')
    logger.debug('Actually logged at DEBUG')
    write_line('about to log at INFO ...')
    logger.info('Actually logged at INFO')
    write_line('about to log at WARNING ...')
    logger.warning('Actually logged at WARNING')
    if fail:
        write_line('about to log at ERROR ...')
        logger.error('Actually logged at ERROR')
        write_line('about to log at CRITICAL ...')
        logger.critical('Actually logged at CRITICAL')
    return fail

decorated_foo = log_if_errors(logger)(foo)

if __name__ == '__main__':
    logger.setLevel(logging.DEBUG)
    write_line('Calling undecorated foo with False')
    assert not foo(False)
    write_line('Calling undecorated foo with True')
    assert foo(True)
    write_line('Calling decorated foo with False')
    assert not decorated_foo(False)
    write_line('Calling decorated foo with True')
    assert decorated_foo(True)
@end example

When this script is run, the following output should be observed:

@example
Calling undecorated foo with False
about to log at DEBUG ...
about to log at INFO ...
about to log at WARNING ...
Calling undecorated foo with True
about to log at DEBUG ...
about to log at INFO ...
about to log at WARNING ...
about to log at ERROR ...
about to log at CRITICAL ...
Calling decorated foo with False
about to log at DEBUG ...
about to log at INFO ...
about to log at WARNING ...
Calling decorated foo with True
about to log at DEBUG ...
about to log at INFO ...
about to log at WARNING ...
about to log at ERROR ...
Actually logged at DEBUG
Actually logged at INFO
Actually logged at WARNING
Actually logged at ERROR
about to log at CRITICAL ...
Actually logged at CRITICAL
@end example

As you can see, actual logging output only occurs when an event is logged whose
severity is ERROR or greater, but in that case, any previous events at lower
severities are also logged.

You can of course use the conventional means of decoration:

@example
@@log_if_errors(logger)
def foo(fail=False):
    ...
@end example

@node Formatting times using UTC GMT via configuration,Using a context manager for selective logging,Buffering logging messages and outputting them conditionally,Logging Cookbook
@anchor{howto/logging-cookbook formatting-times-using-utc-gmt-via-configuration}@anchor{4710}@anchor{howto/logging-cookbook utc-formatting}@anchor{4711}
@subsection Formatting times using UTC (GMT) via configuration


Sometimes you want to format times using UTC, which can be done using a class
such as @cite{UTCFormatter}, shown below:

@example
import logging
import time

class UTCFormatter(logging.Formatter):
    converter = time.gmtime
@end example

and you can then use the @code{UTCFormatter} in your code instead of
@ref{11c6,,Formatter}. If you want to do that via configuration, you can
use the @ref{d55,,dictConfig()} API with an approach illustrated by
the following complete example:

@example
import logging
import logging.config
import time

class UTCFormatter(logging.Formatter):
    converter = time.gmtime

LOGGING = @{
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': @{
        'utc': @{
            '()': UTCFormatter,
            'format': '%(asctime)s %(message)s',
        @},
        'local': @{
            'format': '%(asctime)s %(message)s',
        @}
    @},
    'handlers': @{
        'console1': @{
            'class': 'logging.StreamHandler',
            'formatter': 'utc',
        @},
        'console2': @{
            'class': 'logging.StreamHandler',
            'formatter': 'local',
        @},
    @},
    'root': @{
        'handlers': ['console1', 'console2'],
   @}
@}

if __name__ == '__main__':
    logging.config.dictConfig(LOGGING)
    logging.warning('The local time is %s', time.asctime())
@end example

When this script is run, it should print something like:

@example
2015-10-17 12:53:29,501 The local time is Sat Oct 17 13:53:29 2015
2015-10-17 13:53:29,501 The local time is Sat Oct 17 13:53:29 2015
@end example

showing how the time is formatted both as local time and UTC, one for each
handler.

@node Using a context manager for selective logging,A CLI application starter template,Formatting times using UTC GMT via configuration,Logging Cookbook
@anchor{howto/logging-cookbook context-manager}@anchor{4712}@anchor{howto/logging-cookbook using-a-context-manager-for-selective-logging}@anchor{4713}
@subsection Using a context manager for selective logging


There are times when it would be useful to temporarily change the logging
configuration and revert it back after doing something. For this, a context
manager is the most obvious way of saving and restoring the logging context.
Here is a simple example of such a context manager, which allows you to
optionally change the logging level and add a logging handler purely in the
scope of the context manager:

@example
import logging
import sys

class LoggingContext:
    def __init__(self, logger, level=None, handler=None, close=True):
        self.logger = logger
        self.level = level
        self.handler = handler
        self.close = close

    def __enter__(self):
        if self.level is not None:
            self.old_level = self.logger.level
            self.logger.setLevel(self.level)
        if self.handler:
            self.logger.addHandler(self.handler)

    def __exit__(self, et, ev, tb):
        if self.level is not None:
            self.logger.setLevel(self.old_level)
        if self.handler:
            self.logger.removeHandler(self.handler)
        if self.handler and self.close:
            self.handler.close()
        # implicit return of None => don't swallow exceptions
@end example

If you specify a level value, the logger’s level is set to that value in the
scope of the with block covered by the context manager. If you specify a
handler, it is added to the logger on entry to the block and removed on exit
from the block. You can also ask the manager to close the handler for you on
block exit - you could do this if you don’t need the handler any more.

To illustrate how it works, we can add the following block of code to the
above:

@example
if __name__ == '__main__':
    logger = logging.getLogger('foo')
    logger.addHandler(logging.StreamHandler())
    logger.setLevel(logging.INFO)
    logger.info('1. This should appear just once on stderr.')
    logger.debug('2. This should not appear.')
    with LoggingContext(logger, level=logging.DEBUG):
        logger.debug('3. This should appear once on stderr.')
    logger.debug('4. This should not appear.')
    h = logging.StreamHandler(sys.stdout)
    with LoggingContext(logger, level=logging.DEBUG, handler=h, close=True):
        logger.debug('5. This should appear twice - once on stderr and once on stdout.')
    logger.info('6. This should appear just once on stderr.')
    logger.debug('7. This should not appear.')
@end example

We initially set the logger’s level to @code{INFO}, so message #1 appears and
message #2 doesn’t. We then change the level to @code{DEBUG} temporarily in the
following @code{with} block, and so message #3 appears. After the block exits, the
logger’s level is restored to @code{INFO} and so message #4 doesn’t appear. In the
next @code{with} block, we set the level to @code{DEBUG} again but also add a handler
writing to @code{sys.stdout}. Thus, message #5 appears twice on the console (once
via @code{stderr} and once via @code{stdout}). After the @code{with} statement’s
completion, the status is as it was before so message #6 appears (like message
#1) whereas message #7 doesn’t (just like message #2).

If we run the resulting script, the result is as follows:

@example
$ python logctx.py
1. This should appear just once on stderr.
3. This should appear once on stderr.
5. This should appear twice - once on stderr and once on stdout.
5. This should appear twice - once on stderr and once on stdout.
6. This should appear just once on stderr.
@end example

If we run it again, but pipe @code{stderr} to @code{/dev/null}, we see the following,
which is the only message written to @code{stdout}:

@example
$ python logctx.py 2>/dev/null
5. This should appear twice - once on stderr and once on stdout.
@end example

Once again, but piping @code{stdout} to @code{/dev/null}, we get:

@example
$ python logctx.py >/dev/null
1. This should appear just once on stderr.
3. This should appear once on stderr.
5. This should appear twice - once on stderr and once on stdout.
6. This should appear just once on stderr.
@end example

In this case, the message #5 printed to @code{stdout} doesn’t appear, as expected.

Of course, the approach described here can be generalised, for example to attach
logging filters temporarily. Note that the above code works in Python 2 as well
as Python 3.

@node A CLI application starter template,A Qt GUI for logging,Using a context manager for selective logging,Logging Cookbook
@anchor{howto/logging-cookbook a-cli-application-starter-template}@anchor{4714}@anchor{howto/logging-cookbook starter-template}@anchor{4715}
@subsection A CLI application starter template


Here’s an example which shows how you can:


@itemize *

@item 
Use a logging level based on command-line arguments

@item 
Dispatch to multiple subcommands in separate files, all logging at the same
level in a consistent way

@item 
Make use of simple, minimal configuration
@end itemize

Suppose we have a command-line application whose job is to stop, start or
restart some services. This could be organised for the purposes of illustration
as a file @code{app.py} that is the main script for the application, with individual
commands implemented in @code{start.py}, @code{stop.py} and @code{restart.py}. Suppose
further that we want to control the verbosity of the application via a
command-line argument, defaulting to @code{logging.INFO}. Here’s one way that
@code{app.py} could be written:

@example
import argparse
import importlib
import logging
import os
import sys

def main(args=None):
    scriptname = os.path.basename(__file__)
    parser = argparse.ArgumentParser(scriptname)
    levels = ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL')
    parser.add_argument('--log-level', default='INFO', choices=levels)
    subparsers = parser.add_subparsers(dest='command',
                                       help='Available commands:')
    start_cmd = subparsers.add_parser('start', help='Start a service')
    start_cmd.add_argument('name', metavar='NAME',
                           help='Name of service to start')
    stop_cmd = subparsers.add_parser('stop',
                                     help='Stop one or more services')
    stop_cmd.add_argument('names', metavar='NAME', nargs='+',
                          help='Name of service to stop')
    restart_cmd = subparsers.add_parser('restart',
                                        help='Restart one or more services')
    restart_cmd.add_argument('names', metavar='NAME', nargs='+',
                             help='Name of service to restart')
    options = parser.parse_args()
    # the code to dispatch commands could all be in this file. For the purposes
    # of illustration only, we implement each command in a separate module.
    try:
        mod = importlib.import_module(options.command)
        cmd = getattr(mod, 'command')
    except (ImportError, AttributeError):
        print('Unable to find the code for command \'%s\'@w{'} % options.command)
        return 1
    # Could get fancy here and load configuration from file or dictionary
    logging.basicConfig(level=options.log_level,
                        format='%(levelname)s %(name)s %(message)s')
    cmd(options)

if __name__ == '__main__':
    sys.exit(main())
@end example

And the @code{start}, @code{stop} and @code{restart} commands can be implemented in
separate modules, like so for starting:

@example
# start.py
import logging

logger = logging.getLogger(__name__)

def command(options):
    logger.debug('About to start %s', options.name)
    # actually do the command processing here ...
    logger.info('Started the \'%s\' service.', options.name)
@end example

and thus for stopping:

@example
# stop.py
import logging

logger = logging.getLogger(__name__)

def command(options):
    n = len(options.names)
    if n == 1:
        plural = '@w{'}
        services = '\'%s\'@w{'} % options.names[0]
    else:
        plural = 's'
        services = ', '.join('\'%s\'@w{'} % name for name in options.names)
        i = services.rfind(', ')
        services = services[:i] + ' and ' + services[i + 2:]
    logger.debug('About to stop %s', services)
    # actually do the command processing here ...
    logger.info('Stopped the %s service%s.', services, plural)
@end example

and similarly for restarting:

@example
# restart.py
import logging

logger = logging.getLogger(__name__)

def command(options):
    n = len(options.names)
    if n == 1:
        plural = '@w{'}
        services = '\'%s\'@w{'} % options.names[0]
    else:
        plural = 's'
        services = ', '.join('\'%s\'@w{'} % name for name in options.names)
        i = services.rfind(', ')
        services = services[:i] + ' and ' + services[i + 2:]
    logger.debug('About to restart %s', services)
    # actually do the command processing here ...
    logger.info('Restarted the %s service%s.', services, plural)
@end example

If we run this application with the default log level, we get output like this:

@example
$ python app.py start foo
INFO start Started the 'foo' service.

$ python app.py stop foo bar
INFO stop Stopped the 'foo' and 'bar' services.

$ python app.py restart foo bar baz
INFO restart Restarted the 'foo', 'bar' and 'baz' services.
@end example

The first word is the logging level, and the second word is the module or
package name of the place where the event was logged.

If we change the logging level, then we can change the information sent to the
log. For example, if we want more information:

@example
$ python app.py --log-level DEBUG start foo
DEBUG start About to start foo
INFO start Started the 'foo' service.

$ python app.py --log-level DEBUG stop foo bar
DEBUG stop About to stop 'foo' and 'bar'
INFO stop Stopped the 'foo' and 'bar' services.

$ python app.py --log-level DEBUG restart foo bar baz
DEBUG restart About to restart 'foo', 'bar' and 'baz'
INFO restart Restarted the 'foo', 'bar' and 'baz' services.
@end example

And if we want less:

@example
$ python app.py --log-level WARNING start foo
$ python app.py --log-level WARNING stop foo bar
$ python app.py --log-level WARNING restart foo bar baz
@end example

In this case, the commands don’t print anything to the console, since nothing
at @code{WARNING} level or above is logged by them.

@node A Qt GUI for logging,Patterns to avoid,A CLI application starter template,Logging Cookbook
@anchor{howto/logging-cookbook a-qt-gui-for-logging}@anchor{4716}@anchor{howto/logging-cookbook qt-gui}@anchor{4717}
@subsection A Qt GUI for logging


A question that comes up from time to time is about how to log to a GUI
application. The Qt@footnote{https://www.qt.io/} framework is a popular
cross-platform UI framework with Python bindings using PySide2@footnote{https://pypi.org/project/PySide2/} or PyQt5@footnote{https://pypi.org/project/PyQt5/} libraries.

The following example shows how to log to a Qt GUI. This introduces a simple
@code{QtHandler} class which takes a callable, which should be a slot in the main
thread that does GUI updates. A worker thread is also created to show how you
can log to the GUI from both the UI itself (via a button for manual logging)
as well as a worker thread doing work in the background (here, just logging
messages at random levels with random short delays in between).

The worker thread is implemented using Qt’s @code{QThread} class rather than the
@ref{109,,threading} module, as there are circumstances where one has to use
@code{QThread}, which offers better integration with other @code{Qt} components.

The code should work with recent releases of either @code{PySide2} or @code{PyQt5}.
You should be able to adapt the approach to earlier versions of Qt. Please
refer to the comments in the code snippet for more detailed information.

@example
import datetime
import logging
import random
import sys
import time

# Deal with minor differences between PySide2 and PyQt5
try:
    from PySide2 import QtCore, QtGui, QtWidgets
    Signal = QtCore.Signal
    Slot = QtCore.Slot
except ImportError:
    from PyQt5 import QtCore, QtGui, QtWidgets
    Signal = QtCore.pyqtSignal
    Slot = QtCore.pyqtSlot


logger = logging.getLogger(__name__)


#
# Signals need to be contained in a QObject or subclass in order to be correctly
# initialized.
#
class Signaller(QtCore.QObject):
    signal = Signal(str, logging.LogRecord)

#
# Output to a Qt GUI is only supposed to happen on the main thread. So, this
# handler is designed to take a slot function which is set up to run in the main
# thread. In this example, the function takes a string argument which is a
# formatted log message, and the log record which generated it. The formatted
# string is just a convenience - you could format a string for output any way
# you like in the slot function itself.
#
# You specify the slot function to do whatever GUI updates you want. The handler
# doesn't know or care about specific UI elements.
#
class QtHandler(logging.Handler):
    def __init__(self, slotfunc, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.signaller = Signaller()
        self.signaller.signal.connect(slotfunc)

    def emit(self, record):
        s = self.format(record)
        self.signaller.signal.emit(s, record)

#
# This example uses QThreads, which means that the threads at the Python level
# are named something like "Dummy-1". The function below gets the Qt name of the
# current thread.
#
def ctname():
    return QtCore.QThread.currentThread().objectName()


#
# Used to generate random levels for logging.
#
LEVELS = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR,
          logging.CRITICAL)

#
# This worker class represents work that is done in a thread separate to the
# main thread. The way the thread is kicked off to do work is via a button press
# that connects to a slot in the worker.
#
# Because the default threadName value in the LogRecord isn't much use, we add
# a qThreadName which contains the QThread name as computed above, and pass that
# value in an "extra" dictionary which is used to update the LogRecord with the
# QThread name.
#
# This example worker just outputs messages sequentially, interspersed with
# random delays of the order of a few seconds.
#
class Worker(QtCore.QObject):
    @@Slot()
    def start(self):
        extra = @{'qThreadName': ctname() @}
        logger.debug('Started work', extra=extra)
        i = 1
        # Let the thread run until interrupted. This allows reasonably clean
        # thread termination.
        while not QtCore.QThread.currentThread().isInterruptionRequested():
            delay = 0.5 + random.random() * 2
            time.sleep(delay)
            level = random.choice(LEVELS)
            logger.log(level, 'Message after delay of %3.1f: %d', delay, i, extra=extra)
            i += 1

#
# Implement a simple UI for this cookbook example. This contains:
#
# * A read-only text edit window which holds formatted log messages
# * A button to start work and log stuff in a separate thread
# * A button to log something from the main thread
# * A button to clear the log window
#
class Window(QtWidgets.QWidget):

    COLORS = @{
        logging.DEBUG: 'black',
        logging.INFO: 'blue',
        logging.WARNING: 'orange',
        logging.ERROR: 'red',
        logging.CRITICAL: 'purple',
    @}

    def __init__(self, app):
        super().__init__()
        self.app = app
        self.textedit = te = QtWidgets.QPlainTextEdit(self)
        # Set whatever the default monospace font is for the platform
        f = QtGui.QFont('nosuchfont')
        f.setStyleHint(f.Monospace)
        te.setFont(f)
        te.setReadOnly(True)
        PB = QtWidgets.QPushButton
        self.work_button = PB('Start background work', self)
        self.log_button = PB('Log a message at a random level', self)
        self.clear_button = PB('Clear log window', self)
        self.handler = h = QtHandler(self.update_status)
        # Remember to use qThreadName rather than threadName in the format string.
        fs = '%(asctime)s %(qThreadName)-12s %(levelname)-8s %(message)s'
        formatter = logging.Formatter(fs)
        h.setFormatter(formatter)
        logger.addHandler(h)
        # Set up to terminate the QThread when we exit
        app.aboutToQuit.connect(self.force_quit)

        # Lay out all the widgets
        layout = QtWidgets.QVBoxLayout(self)
        layout.addWidget(te)
        layout.addWidget(self.work_button)
        layout.addWidget(self.log_button)
        layout.addWidget(self.clear_button)
        self.setFixedSize(900, 400)

        # Connect the non-worker slots and signals
        self.log_button.clicked.connect(self.manual_update)
        self.clear_button.clicked.connect(self.clear_display)

        # Start a new worker thread and connect the slots for the worker
        self.start_thread()
        self.work_button.clicked.connect(self.worker.start)
        # Once started, the button should be disabled
        self.work_button.clicked.connect(lambda : self.work_button.setEnabled(False))

    def start_thread(self):
        self.worker = Worker()
        self.worker_thread = QtCore.QThread()
        self.worker.setObjectName('Worker')
        self.worker_thread.setObjectName('WorkerThread')  # for qThreadName
        self.worker.moveToThread(self.worker_thread)
        # This will start an event loop in the worker thread
        self.worker_thread.start()

    def kill_thread(self):
        # Just tell the worker to stop, then tell it to quit and wait for that
        # to happen
        self.worker_thread.requestInterruption()
        if self.worker_thread.isRunning():
            self.worker_thread.quit()
            self.worker_thread.wait()
        else:
            print('worker has already exited.')

    def force_quit(self):
        # For use when the window is closed
        if self.worker_thread.isRunning():
            self.kill_thread()

    # The functions below update the UI and run in the main thread because
    # that's where the slots are set up

    @@Slot(str, logging.LogRecord)
    def update_status(self, status, record):
        color = self.COLORS.get(record.levelno, 'black')
        s = '<pre><font color="%s">%s</font></pre>' % (color, status)
        self.textedit.appendHtml(s)

    @@Slot()
    def manual_update(self):
        # This function uses the formatted message passed in, but also uses
        # information from the record to format the message in an appropriate
        # color according to its severity (level).
        level = random.choice(LEVELS)
        extra = @{'qThreadName': ctname() @}
        logger.log(level, 'Manually logged!', extra=extra)

    @@Slot()
    def clear_display(self):
        self.textedit.clear()


def main():
    QtCore.QThread.currentThread().setObjectName('MainThread')
    logging.getLogger().setLevel(logging.DEBUG)
    app = QtWidgets.QApplication(sys.argv)
    example = Window(app)
    example.show()
    sys.exit(app.exec_())

if __name__=='__main__':
    main()
@end example

@c patterns-to-avoid:

@node Patterns to avoid,,A Qt GUI for logging,Logging Cookbook
@anchor{howto/logging-cookbook patterns-to-avoid}@anchor{4718}
@subsection Patterns to avoid


Although the preceding sections have described ways of doing things you might
need to do or deal with, it is worth mentioning some usage patterns which are
@emph{unhelpful}, and which should therefore be avoided in most cases. The following
sections are in no particular order.

@menu
* Opening the same log file multiple times:: 
* Using loggers as attributes in a class or passing them as parameters:: 
* Adding handlers other than NullHandler to a logger in a library:: 
* Creating a lot of loggers:: 

@end menu

@node Opening the same log file multiple times,Using loggers as attributes in a class or passing them as parameters,,Patterns to avoid
@anchor{howto/logging-cookbook opening-the-same-log-file-multiple-times}@anchor{4719}
@subsubsection Opening the same log file multiple times


On Windows, you will generally not be able to open the same file multiple times
as this will lead to a “file is in use by another process” error. However, on
POSIX platforms you’ll not get any errors if you open the same file multiple
times. This could be done accidentally, for example by:


@itemize *

@item 
Adding a file handler more than once which references the same file (e.g. by
a copy/paste/forget-to-change error).

@item 
Opening two files that look different, as they have different names, but are
the same because one is a symbolic link to the other.

@item 
Forking a process, following which both parent and child have a reference to
the same file. This might be through use of the @ref{b3,,multiprocessing} module,
for example.
@end itemize

Opening a file multiple times might @emph{appear} to work most of the time, but can
lead to a number of problems in practice:


@itemize *

@item 
Logging output can be garbled because multiple threads or processes try to
write to the same file. Although logging guards against concurrent use of the
same handler instance by multiple threads, there is no such protection if
concurrent writes are attempted by two different threads using two different
handler instances which happen to point to the same file.

@item 
An attempt to delete a file (e.g. during file rotation) silently fails,
because there is another reference pointing to it. This can lead to confusion
and wasted debugging time - log entries end up in unexpected places, or are
lost altogether. Or a file that was supposed to be moved remains in place,
and grows in size unexpectedly despite size-based rotation being supposedly
in place.
@end itemize

Use the techniques outlined in @ref{46f3,,Logging to a single file from multiple processes} to circumvent such
issues.

@node Using loggers as attributes in a class or passing them as parameters,Adding handlers other than NullHandler to a logger in a library,Opening the same log file multiple times,Patterns to avoid
@anchor{howto/logging-cookbook using-loggers-as-attributes-in-a-class-or-passing-them-as-parameters}@anchor{471a}
@subsubsection Using loggers as attributes in a class or passing them as parameters


While there might be unusual cases where you’ll need to do this, in general
there is no point because loggers are singletons. Code can always access a
given logger instance by name using @code{logging.getLogger(name)}, so passing
instances around and holding them as instance attributes is pointless. Note
that in other languages such as Java and C#, loggers are often static class
attributes. However, this pattern doesn’t make sense in Python, where the
module (and not the class) is the unit of software decomposition.

@node Adding handlers other than NullHandler to a logger in a library,Creating a lot of loggers,Using loggers as attributes in a class or passing them as parameters,Patterns to avoid
@anchor{howto/logging-cookbook adding-handlers-other-than-nullhandler-to-a-logger-in-a-library}@anchor{471b}
@subsubsection Adding handlers other than @code{NullHandler} to a logger in a library


Configuring logging by adding handlers, formatters and filters is the
responsibility of the application developer, not the library developer. If you
are maintaining a library, ensure that you don’t add handlers to any of your
loggers other than a @ref{e34,,NullHandler} instance.

@node Creating a lot of loggers,,Adding handlers other than NullHandler to a logger in a library,Patterns to avoid
@anchor{howto/logging-cookbook creating-a-lot-of-loggers}@anchor{471c}
@subsubsection Creating a lot of loggers


Loggers are singletons that are never freed during a script execution, and so
creating lots of loggers will use up memory which can’t then be freed. Rather
than create a logger per e.g. file processed or network connection made, use
the @ref{2559,,existing mechanisms} for passing contextual
information into your logs and restrict the loggers created to those describing
areas within your application (generally modules, but occasionally slightly
more fine-grained than that).

@node Regular Expression HOWTO,Socket Programming HOWTO,Logging Cookbook,Python HOWTOs
@anchor{howto/regex doc}@anchor{471d}@anchor{howto/regex regex-howto}@anchor{1bd8}@anchor{howto/regex regular-expression-howto}@anchor{471e}
@section Regular Expression HOWTO


@*Author: 
A.M. Kuchling <@email{amk@@amk.ca}>


@c TODO:
@c Document lookbehind assertions
@c Better way of displaying a RE, a string, and what it matches
@c Mention optional argument to match.groups()
@c Unicode (at least a reference)

@subsubheading Abstract



This document is an introductory tutorial to using regular expressions in Python
with the @ref{d8,,re} module.  It provides a gentler introduction than the
corresponding section in the Library Reference.

@menu
* Introduction: Introduction<15>. 
* Simple Patterns:: 
* Using Regular Expressions:: 
* More Pattern Power:: 
* Modifying Strings:: 
* Common Problems:: 
* Feedback:: 

@end menu

@node Introduction<15>,Simple Patterns,,Regular Expression HOWTO
@anchor{howto/regex introduction}@anchor{471f}
@subsection Introduction


Regular expressions (called REs, or regexes, or regex patterns) are essentially
a tiny, highly specialized programming language embedded inside Python and made
available through the @ref{d8,,re} module. Using this little language, you specify
the rules for the set of possible strings that you want to match; this set might
contain English sentences, or e-mail addresses, or TeX commands, or anything you
like.  You can then ask questions such as “Does this string match the pattern?”,
or “Is there a match for the pattern anywhere in this string?”.  You can also
use REs to modify a string or to split it apart in various ways.

Regular expression patterns are compiled into a series of bytecodes which are
then executed by a matching engine written in C.  For advanced use, it may be
necessary to pay careful attention to how the engine will execute a given RE,
and write the RE in a certain way in order to produce bytecode that runs faster.
Optimization isn’t covered in this document, because it requires that you have a
good understanding of the matching engine’s internals.

The regular expression language is relatively small and restricted, so not all
possible string processing tasks can be done using regular expressions.  There
are also tasks that @emph{can} be done with regular expressions, but the expressions
turn out to be very complicated.  In these cases, you may be better off writing
Python code to do the processing; while Python code will be slower than an
elaborate regular expression, it will also probably be more understandable.

@node Simple Patterns,Using Regular Expressions,Introduction<15>,Regular Expression HOWTO
@anchor{howto/regex simple-patterns}@anchor{4720}
@subsection Simple Patterns


We’ll start by learning about the simplest possible regular expressions.  Since
regular expressions are used to operate on strings, we’ll begin with the most
common task: matching characters.

For a detailed explanation of the computer science underlying regular
expressions (deterministic and non-deterministic finite automata), you can refer
to almost any textbook on writing compilers.

@menu
* Matching Characters:: 
* Repeating Things:: 

@end menu

@node Matching Characters,Repeating Things,,Simple Patterns
@anchor{howto/regex matching-characters}@anchor{4721}
@subsubsection Matching Characters


Most letters and characters will simply match themselves.  For example, the
regular expression @code{test} will match the string @code{test} exactly.  (You can
enable a case-insensitive mode that would let this RE match @code{Test} or @code{TEST}
as well; more about this later.)

There are exceptions to this rule; some characters are special
@emph{metacharacters}, and don’t match themselves.  Instead, they signal that
some out-of-the-ordinary thing should be matched, or they affect other portions
of the RE by repeating them or changing their meaning.  Much of this document is
devoted to discussing various metacharacters and what they do.

Here’s a complete list of the metacharacters; their meanings will be discussed
in the rest of this HOWTO.

@example
. ^ $ * + ? @{ @} [ ] \ | ( )
@end example

The first metacharacters we’ll look at are @code{[} and @code{]}. They’re used for
specifying a character class, which is a set of characters that you wish to
match.  Characters can be listed individually, or a range of characters can be
indicated by giving two characters and separating them by a @code{'-'}.  For
example, @code{[abc]} will match any of the characters @code{a}, @code{b}, or @code{c}; this
is the same as @code{[a-c]}, which uses a range to express the same set of
characters.  If you wanted to match only lowercase letters, your RE would be
@code{[a-z]}.

Metacharacters are not active inside classes.  For example, @code{[akm$]} will
match any of the characters @code{'a'}, @code{'k'}, @code{'m'}, or @code{'$'}; @code{'$'} is
usually a metacharacter, but inside a character class it’s stripped of its
special nature.

You can match the characters not listed within the class by @emph{complementing}
the set.  This is indicated by including a @code{'^'} as the first character of the
class. For example, @code{[^5]} will match any character except @code{'5'}.  If the
caret appears elsewhere in a character class, it does not have special meaning.
For example: @code{[5^]} will match either a @code{'5'} or a @code{'^'}.

Perhaps the most important metacharacter is the backslash, @code{\}.   As in Python
string literals, the backslash can be followed by various characters to signal
various special sequences.  It’s also used to escape all the metacharacters so
you can still match them in patterns; for example, if you need to match a @code{[}
or  @code{\}, you can precede them with a backslash to remove their special
meaning: @code{\[} or @code{\\}.

Some of the special sequences beginning with @code{'\'} represent
predefined sets of characters that are often useful, such as the set
of digits, the set of letters, or the set of anything that isn’t
whitespace.

Let’s take an example: @code{\w} matches any alphanumeric character.  If
the regex pattern is expressed in bytes, this is equivalent to the
class @code{[a-zA-Z0-9_]}.  If the regex pattern is a string, @code{\w} will
match all the characters marked as letters in the Unicode database
provided by the @ref{121,,unicodedata} module.  You can use the more
restricted definition of @code{\w} in a string pattern by supplying the
@ref{697,,re.ASCII} flag when compiling the regular expression.

The following list of special sequences isn’t complete. For a complete
list of sequences and expanded class definitions for Unicode string
patterns, see the last part of @ref{1bd5,,Regular Expression Syntax} in the Standard Library reference.  In general, the
Unicode versions match any character that’s in the appropriate
category in the Unicode database.


@table @asis

@item @code{\d}

Matches any decimal digit; this is equivalent to the class @code{[0-9]}.

@item @code{\D}

Matches any non-digit character; this is equivalent to the class @code{[^0-9]}.

@item @code{\s}

Matches any whitespace character; this is equivalent to the class @code{[
\t\n\r\f\v]}.

@item @code{\S}

Matches any non-whitespace character; this is equivalent to the class @code{[^
\t\n\r\f\v]}.

@item @code{\w}

Matches any alphanumeric character; this is equivalent to the class
@code{[a-zA-Z0-9_]}.

@item @code{\W}

Matches any non-alphanumeric character; this is equivalent to the class
@code{[^a-zA-Z0-9_]}.
@end table

These sequences can be included inside a character class.  For example,
@code{[\s,.]} is a character class that will match any whitespace character, or
@code{','} or @code{'.'}.

The final metacharacter in this section is @code{.}.  It matches anything except a
newline character, and there’s an alternate mode (@ref{1bd9,,re.DOTALL}) where it will
match even a newline.  @code{.} is often used where you want to match “any
character”.

@node Repeating Things,,Matching Characters,Simple Patterns
@anchor{howto/regex repeating-things}@anchor{4722}
@subsubsection Repeating Things


Being able to match varying sets of characters is the first thing regular
expressions can do that isn’t already possible with the methods available on
strings.  However, if that was the only additional capability of regexes, they
wouldn’t be much of an advance. Another capability is that you can specify that
portions of the RE must be repeated a certain number of times.

The first metacharacter for repeating things that we’ll look at is @code{*}.  @code{*}
doesn’t match the literal character @code{'*'}; instead, it specifies that the
previous character can be matched zero or more times, instead of exactly once.

For example, @code{ca*t} will match @code{'ct'} (0 @code{'a'} characters), @code{'cat'} (1 @code{'a'}),
@code{'caaat'} (3 @code{'a'} characters), and so forth.

Repetitions such as @code{*} are @emph{greedy}; when repeating a RE, the matching
engine will try to repeat it as many times as possible. If later portions of the
pattern don’t match, the matching engine will then back up and try again with
fewer repetitions.

A step-by-step example will make this more obvious.  Let’s consider the
expression @code{a[bcd]*b}.  This matches the letter @code{'a'}, zero or more letters
from the class @code{[bcd]}, and finally ends with a @code{'b'}.  Now imagine matching
this RE against the string @code{'abcbd'}.


@multitable {xxxxxxxx} {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Step

@tab

Matched

@tab

Explanation

@item

1

@tab

@code{a}

@tab

The @code{a} in the RE matches.

@item

2

@tab

@code{abcbd}

@tab

The engine matches @code{[bcd]*},
going as far as it can, which
is to the end of the string.

@item

3

@tab

@emph{Failure}

@tab

The engine tries to match
@code{b}, but the current position
is at the end of the string, so
it fails.

@item

4

@tab

@code{abcb}

@tab

Back up, so that  @code{[bcd]*}
matches one less character.

@item

5

@tab

@emph{Failure}

@tab

Try @code{b} again, but the
current position is at the last
character, which is a @code{'d'}.

@item

6

@tab

@code{abc}

@tab

Back up again, so that
@code{[bcd]*} is only matching
@code{bc}.

@item

6

@tab

@code{abcb}

@tab

Try @code{b} again.  This time
the character at the
current position is @code{'b'}, so
it succeeds.

@end multitable


The end of the RE has now been reached, and it has matched @code{'abcb'}.  This
demonstrates how the matching engine goes as far as it can at first, and if no
match is found it will then progressively back up and retry the rest of the RE
again and again.  It will back up until it has tried zero matches for
@code{[bcd]*}, and if that subsequently fails, the engine will conclude that the
string doesn’t match the RE at all.

Another repeating metacharacter is @code{+}, which matches one or more times.  Pay
careful attention to the difference between @code{*} and @code{+}; @code{*} matches
@emph{zero} or more times, so whatever’s being repeated may not be present at all,
while @code{+} requires at least @emph{one} occurrence.  To use a similar example,
@code{ca+t} will match @code{'cat'} (1 @code{'a'}), @code{'caaat'} (3 @code{'a'}s), but won’t
match @code{'ct'}.

There are two more repeating qualifiers.  The question mark character, @code{?},
matches either once or zero times; you can think of it as marking something as
being optional.  For example, @code{home-?brew} matches either @code{'homebrew'} or
@code{'home-brew'}.

The most complicated repeated qualifier is @code{@{m,n@}}, where @emph{m} and @emph{n} are
decimal integers.  This qualifier means there must be at least @emph{m} repetitions,
and at most @emph{n}.  For example, @code{a/@{1,3@}b} will match @code{'a/b'}, @code{'a//b'}, and
@code{'a///b'}.  It won’t match @code{'ab'}, which has no slashes, or @code{'a////b'}, which
has four.

You can omit either @emph{m} or @emph{n}; in that case, a reasonable value is assumed for
the missing value.  Omitting @emph{m} is interpreted as a lower limit of 0, while
omitting @emph{n} results in an upper bound of infinity.

Readers of a reductionist bent may notice that the three other qualifiers can
all be expressed using this notation.  @code{@{0,@}} is the same as @code{*}, @code{@{1,@}}
is equivalent to @code{+}, and @code{@{0,1@}} is the same as @code{?}.  It’s better to use
@code{*}, @code{+}, or @code{?} when you can, simply because they’re shorter and easier
to read.

@node Using Regular Expressions,More Pattern Power,Simple Patterns,Regular Expression HOWTO
@anchor{howto/regex using-regular-expressions}@anchor{4723}
@subsection Using Regular Expressions


Now that we’ve looked at some simple regular expressions, how do we actually use
them in Python?  The @ref{d8,,re} module provides an interface to the regular
expression engine, allowing you to compile REs into objects and then perform
matches with them.

@menu
* Compiling Regular Expressions:: 
* The Backslash Plague:: 
* Performing Matches:: 
* Module-Level Functions: Module-Level Functions<2>. 
* Compilation Flags:: 

@end menu

@node Compiling Regular Expressions,The Backslash Plague,,Using Regular Expressions
@anchor{howto/regex compiling-regular-expressions}@anchor{4724}
@subsubsection Compiling Regular Expressions


Regular expressions are compiled into pattern objects, which have
methods for various operations such as searching for pattern matches or
performing string substitutions.

@example
>>> import re
>>> p = re.compile('ab*')
>>> p
re.compile('ab*')
@end example

@ref{70b,,re.compile()} also accepts an optional @emph{flags} argument, used to enable
various special features and syntax variations.  We’ll go over the available
settings later, but for now a single example will do:

@example
>>> p = re.compile('ab*', re.IGNORECASE)
@end example

The RE is passed to @ref{70b,,re.compile()} as a string.  REs are handled as strings
because regular expressions aren’t part of the core Python language, and no
special syntax was created for expressing them.  (There are applications that
don’t need REs at all, so there’s no need to bloat the language specification by
including them.) Instead, the @ref{d8,,re} module is simply a C extension module
included with Python, just like the @ref{ea,,socket} or @ref{14b,,zlib} modules.

Putting REs in strings keeps the Python language simpler, but has one
disadvantage which is the topic of the next section.

@node The Backslash Plague,Performing Matches,Compiling Regular Expressions,Using Regular Expressions
@anchor{howto/regex id1}@anchor{4725}@anchor{howto/regex the-backslash-plague}@anchor{4726}
@subsubsection The Backslash Plague


As stated earlier, regular expressions use the backslash character (@code{'\'}) to
indicate special forms or to allow special characters to be used without
invoking their special meaning. This conflicts with Python’s usage of the same
character for the same purpose in string literals.

Let’s say you want to write a RE that matches the string @code{\section}, which
might be found in a LaTeX file.  To figure out what to write in the program
code, start with the desired string to be matched.  Next, you must escape any
backslashes and other metacharacters by preceding them with a backslash,
resulting in the string @code{\\section}.  The resulting string that must be passed
to @ref{70b,,re.compile()} must be @code{\\section}.  However, to express this as a
Python string literal, both backslashes must be escaped @emph{again}.


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Characters

@tab

Stage

@item

@code{\section}

@tab

Text string to be matched

@item

@code{\\section}

@tab

Escaped backslash for @ref{70b,,re.compile()}

@item

@code{"\\\\section"}

@tab

Escaped backslashes for a string literal

@end multitable


In short, to match a literal backslash, one has to write @code{'\\\\'} as the RE
string, because the regular expression must be @code{\\}, and each backslash must
be expressed as @code{\\} inside a regular Python string literal.  In REs that
feature backslashes repeatedly, this leads to lots of repeated backslashes and
makes the resulting strings difficult to understand.

The solution is to use Python’s raw string notation for regular expressions;
backslashes are not handled in any special way in a string literal prefixed with
@code{'r'}, so @code{r"\n"} is a two-character string containing @code{'\'} and @code{'n'},
while @code{"\n"} is a one-character string containing a newline. Regular
expressions will often be written in Python code using this raw string notation.

In addition, special escape sequences that are valid in regular expressions,
but not valid as Python string literals, now result in a
@ref{1b2,,DeprecationWarning} and will eventually become a @ref{21a,,SyntaxError},
which means the sequences will be invalid if raw string notation or escaping
the backslashes isn’t used.


@multitable {xxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxx} 
@headitem

Regular String

@tab

Raw string

@item

@code{"ab*"}

@tab

@code{r"ab*"}

@item

@code{"\\\\section"}

@tab

@code{r"\\section"}

@item

@code{"\\w+\\s+\\1"}

@tab

@code{r"\w+\s+\1"}

@end multitable


@node Performing Matches,Module-Level Functions<2>,The Backslash Plague,Using Regular Expressions
@anchor{howto/regex performing-matches}@anchor{4727}
@subsubsection Performing Matches


Once you have an object representing a compiled regular expression, what do you
do with it?  Pattern objects have several methods and attributes.
Only the most significant ones will be covered here; consult the @ref{d8,,re} docs
for a complete listing.


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Method/Attribute

@tab

Purpose

@item

@code{match()}

@tab

Determine if the RE matches at the beginning
of the string.

@item

@code{search()}

@tab

Scan through a string, looking for any
location where this RE matches.

@item

@code{findall()}

@tab

Find all substrings where the RE matches, and
returns them as a list.

@item

@code{finditer()}

@tab

Find all substrings where the RE matches, and
returns them as an @ref{11c3,,iterator}.

@end multitable


@ref{1be3,,match()} and @ref{1be4,,search()} return @code{None} if no match can be found.  If
they’re successful, a @ref{b08,,match object} instance is returned,
containing information about the match: where it starts and ends, the substring
it matched, and more.

You can learn about this by interactively experimenting with the @ref{d8,,re}
module.  If you have @ref{10c,,tkinter} available, you may also want to look at
Tools/demo/redemo.py@footnote{https://github.com/python/cpython/tree/main/Tools/demo/redemo.py}, a demonstration program included with the
Python distribution.  It allows you to enter REs and strings, and displays
whether the RE matches or fails. @code{redemo.py} can be quite useful when
trying to debug a complicated RE.

This HOWTO uses the standard Python interpreter for its examples. First, run the
Python interpreter, import the @ref{d8,,re} module, and compile a RE:

@example
>>> import re
>>> p = re.compile('[a-z]+')
>>> p
re.compile('[a-z]+')
@end example

Now, you can try matching various strings against the RE @code{[a-z]+}.  An empty
string shouldn’t match at all, since @code{+} means ‘one or more repetitions’.
@ref{1be3,,match()} should return @code{None} in this case, which will cause the
interpreter to print no output.  You can explicitly print the result of
@code{match()} to make this clear.

@example
>>> p.match("")
>>> print(p.match(""))
None
@end example

Now, let’s try it on a string that it should match, such as @code{tempo}.  In this
case, @ref{1be3,,match()} will return a @ref{b08,,match object}, so you
should store the result in a variable for later use.

@example
>>> m = p.match('tempo')
>>> m
<re.Match object; span=(0, 5), match='tempo'>
@end example

Now you can query the @ref{b08,,match object} for information
about the matching string.  Match object instances
also have several methods and attributes; the most important ones are:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Method/Attribute

@tab

Purpose

@item

@code{group()}

@tab

Return the string matched by the RE

@item

@code{start()}

@tab

Return the starting position of the match

@item

@code{end()}

@tab

Return the ending position of the match

@item

@code{span()}

@tab

Return a tuple containing the (start, end)
positions  of the match

@end multitable


Trying these methods will soon clarify their meaning:

@example
>>> m.group()
'tempo'
>>> m.start(), m.end()
(0, 5)
>>> m.span()
(0, 5)
@end example

@ref{1bf8,,group()} returns the substring that was matched by the RE.  @ref{1bfc,,start()}
and @ref{1bfd,,end()} return the starting and ending index of the match. @ref{1bfe,,span()}
returns both start and end indexes in a single tuple.  Since the @ref{1be3,,match()}
method only checks if the RE matches at the start of a string, @code{start()}
will always be zero.  However, the @ref{1be4,,search()} method of patterns
scans through the string, so  the match may not start at zero in that
case.

@example
>>> print(p.match('::: message'))
None
>>> m = p.search('::: message'); print(m)
<re.Match object; span=(4, 11), match='message'>
>>> m.group()
'message'
>>> m.span()
(4, 11)
@end example

In actual programs, the most common style is to store the
@ref{b08,,match object} in a variable, and then check if it was
@code{None}.  This usually looks like:

@example
p = re.compile( ... )
m = p.match( 'string goes here' )
if m:
    print('Match found: ', m.group())
else:
    print('No match')
@end example

Two pattern methods return all of the matches for a pattern.
@ref{1bee,,findall()} returns a list of matching strings:

@example
>>> p = re.compile(r'\d+')
>>> p.findall('12 drummers drumming, 11 pipers piping, 10 lords a-leaping')
['12', '11', '10']
@end example

The @code{r} prefix, making the literal a raw string literal, is needed in this
example because escape sequences in a normal “cooked” string literal that are
not recognized by Python, as opposed to regular expressions, now result in a
@ref{1b2,,DeprecationWarning} and will eventually become a @ref{21a,,SyntaxError}.  See
@ref{4726,,The Backslash Plague}.

@ref{1bee,,findall()} has to create the entire list before it can be returned as the
result.  The @ref{1bef,,finditer()} method returns a sequence of
@ref{b08,,match object} instances as an @ref{11c3,,iterator}:

@example
>>> iterator = p.finditer('12 drummers drumming, 11 ... 10 ...')
>>> iterator  
<callable_iterator object at 0x...>
>>> for match in iterator:
...     print(match.span())
...
(0, 2)
(22, 24)
(29, 31)
@end example

@node Module-Level Functions<2>,Compilation Flags,Performing Matches,Using Regular Expressions
@anchor{howto/regex module-level-functions}@anchor{4728}
@subsubsection Module-Level Functions


You don’t have to create a pattern object and call its methods; the
@ref{d8,,re} module also provides top-level functions called @ref{dd3,,match()},
@ref{dd2,,search()}, @ref{bbb,,findall()}, @ref{72e,,sub()}, and so forth.  These functions
take the same arguments as the corresponding pattern method with
the RE string added as the first argument, and still return either @code{None} or a
@ref{b08,,match object} instance.

@example
>>> print(re.match(r'From\s+', 'Fromage amk'))
None
>>> re.match(r'From\s+', 'From amk Thu May 14 19:12:10 1998')  
<re.Match object; span=(0, 5), match='From '>
@end example

Under the hood, these functions simply create a pattern object for you
and call the appropriate method on it.  They also store the compiled
object in a cache, so future calls using the same RE won’t need to
parse the pattern again and again.

Should you use these module-level functions, or should you get the
pattern and call its methods yourself?  If you’re accessing a regex
within a loop, pre-compiling it will save a few function calls.
Outside of loops, there’s not much difference thanks to the internal
cache.

@node Compilation Flags,,Module-Level Functions<2>,Using Regular Expressions
@anchor{howto/regex compilation-flags}@anchor{4729}
@subsubsection Compilation Flags


Compilation flags let you modify some aspects of how regular expressions work.
Flags are available in the @ref{d8,,re} module under two names, a long name such as
@code{IGNORECASE} and a short, one-letter form such as @code{I}.  (If you’re
familiar with Perl’s pattern modifiers, the one-letter forms use the same
letters; the short form of @ref{1be7,,re.VERBOSE} is @ref{1be0,,re.X}, for example.)
Multiple flags can be specified by bitwise OR-ing them; @code{re.I | re.M} sets
both the @code{I} and @code{M} flags, for example.

Here’s a table of the available flags, followed by a more detailed explanation
of each one.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Flag

@tab

Meaning

@item

@code{ASCII}, @code{A}

@tab

Makes several escapes like @code{\w}, @code{\b},
@code{\s} and @code{\d} match only on ASCII
characters with the respective property.

@item

@code{DOTALL}, @code{S}

@tab

Make @code{.} match any character, including
newlines.

@item

@code{IGNORECASE}, @code{I}

@tab

Do case-insensitive matches.

@item

@code{LOCALE}, @code{L}

@tab

Do a locale-aware match.

@item

@code{MULTILINE}, @code{M}

@tab

Multi-line matching, affecting @code{^} and
@code{$}.

@item

@code{VERBOSE}, @code{X}
(for ‘extended’)

@tab

Enable verbose REs, which can be organized
more cleanly and understandably.

@end multitable



@deffn {Data} I

@deffnx {Data} IGNORECASE

Perform case-insensitive matching; character class and literal strings will
match letters by ignoring case.  For example, @code{[A-Z]} will match lowercase
letters, too. Full Unicode matching also works unless the @code{ASCII}
flag is used to disable non-ASCII matches.  When the Unicode patterns
@code{[a-z]} or @code{[A-Z]} are used in combination with the @code{IGNORECASE}
flag, they will match the 52 ASCII letters and 4 additional non-ASCII
letters: ‘İ’ (U+0130, Latin capital letter I with dot above), ‘ı’ (U+0131,
Latin small letter dotless i), ‘ſ’ (U+017F, Latin small letter long s) and
‘K’ (U+212A, Kelvin sign).  @code{Spam} will match @code{'Spam'}, @code{'spam'},
@code{'spAM'}, or @code{'ſpam'} (the latter is matched only in Unicode mode).
This lowercasing doesn’t take the current locale into account;
it will if you also set the @code{LOCALE} flag.
@end deffn


@deffn {Data} L

@deffnx {Data} LOCALE

Make @code{\w}, @code{\W}, @code{\b}, @code{\B} and case-insensitive matching dependent
on the current locale instead of the Unicode database.

Locales are a feature of the C library intended to help in writing programs
that take account of language differences.  For example, if you’re
processing encoded French text, you’d want to be able to write @code{\w+} to
match words, but @code{\w} only matches the character class @code{[A-Za-z]} in
bytes patterns; it won’t match bytes corresponding to @code{é} or @code{ç}.
If your system is configured properly and a French locale is selected,
certain C functions will tell the program that the byte corresponding to
@code{é} should also be considered a letter.
Setting the @code{LOCALE} flag when compiling a regular expression will cause
the resulting compiled object to use these C functions for @code{\w}; this is
slower, but also enables @code{\w+} to match French words as you’d expect.
The use of this flag is discouraged in Python 3 as the locale mechanism
is very unreliable, it only handles one “culture” at a time, and it only
works with 8-bit locales.  Unicode matching is already enabled by default
in Python 3 for Unicode (str) patterns, and it is able to handle different
locales/languages.
@end deffn


@deffn {Data} M

@deffnx {Data} MULTILINE

(@code{^} and @code{$} haven’t been explained yet;  they’ll be introduced in section
@ref{472a,,More Metacharacters}.)

Usually @code{^} matches only at the beginning of the string, and @code{$} matches
only at the end of the string and immediately before the newline (if any) at the
end of the string. When this flag is specified, @code{^} matches at the beginning
of the string and at the beginning of each line within the string, immediately
following each newline.  Similarly, the @code{$} metacharacter matches either at
the end of the string and at the end of each line (immediately preceding each
newline).
@end deffn


@deffn {Data} S

@deffnx {Data} DOTALL

Makes the @code{'.'} special character match any character at all, including a
newline; without this flag, @code{'.'} will match anything @emph{except} a newline.
@end deffn


@deffn {Data} A

@deffnx {Data} ASCII

Make @code{\w}, @code{\W}, @code{\b}, @code{\B}, @code{\s} and @code{\S} perform ASCII-only
matching instead of full Unicode matching. This is only meaningful for
Unicode patterns, and is ignored for byte patterns.
@end deffn


@deffn {Data} X

@deffnx {Data} VERBOSE

This flag allows you to write regular expressions that are more readable by
granting you more flexibility in how you can format them.  When this flag has
been specified, whitespace within the RE string is ignored, except when the
whitespace is in a character class or preceded by an unescaped backslash; this
lets you organize and indent the RE more clearly.  This flag also lets you put
comments within a RE that will be ignored by the engine; comments are marked by
a @code{'#'} that’s neither in a character class or preceded by an unescaped
backslash.

For example, here’s a RE that uses @ref{1be7,,re.VERBOSE}; see how much easier it
is to read?

@example
charref = re.compile(r"""
 &[#]                # Start of a numeric entity reference
 (
     0[0-7]+         # Octal form
   | [0-9]+          # Decimal form
   | x[0-9a-fA-F]+   # Hexadecimal form
 )
 ;                   # Trailing semicolon
""", re.VERBOSE)
@end example

Without the verbose setting, the RE would look like this:

@example
charref = re.compile("&#(0[0-7]+"
                     "|[0-9]+"
                     "|x[0-9a-fA-F]+);")
@end example

In the above example, Python’s automatic concatenation of string literals has
been used to break up the RE into smaller pieces, but it’s still more difficult
to understand than the version using @ref{1be7,,re.VERBOSE}.
@end deffn

@node More Pattern Power,Modifying Strings,Using Regular Expressions,Regular Expression HOWTO
@anchor{howto/regex more-pattern-power}@anchor{472b}
@subsection More Pattern Power


So far we’ve only covered a part of the features of regular expressions.  In
this section, we’ll cover some new metacharacters, and how to use groups to
retrieve portions of the text that was matched.

@menu
* More Metacharacters:: 
* Grouping:: 
* Non-capturing and Named Groups:: 
* Lookahead Assertions:: 

@end menu

@node More Metacharacters,Grouping,,More Pattern Power
@anchor{howto/regex id2}@anchor{472c}@anchor{howto/regex more-metacharacters}@anchor{472a}
@subsubsection More Metacharacters


There are some metacharacters that we haven’t covered yet.  Most of them will be
covered in this section.

Some of the remaining metacharacters to be discussed are @emph{zero-width
assertions}.  They don’t cause the engine to advance through the string;
instead, they consume no characters at all, and simply succeed or fail.  For
example, @code{\b} is an assertion that the current position is located at a word
boundary; the position isn’t changed by the @code{\b} at all.  This means that
zero-width assertions should never be repeated, because if they match once at a
given location, they can obviously be matched an infinite number of times.


@table @asis

@item @code{|}

Alternation, or the “or” operator.   If @emph{A} and @emph{B} are regular expressions,
@code{A|B} will match any string that matches either @emph{A} or @emph{B}. @code{|} has very
low precedence in order to make it work reasonably when you’re alternating
multi-character strings. @code{Crow|Servo} will match either @code{'Crow'} or @code{'Servo'},
not @code{'Cro'}, a @code{'w'} or an @code{'S'}, and @code{'ervo'}.

To match a literal @code{'|'}, use @code{\|}, or enclose it inside a character class,
as in @code{[|]}.

@item @code{^}

Matches at the beginning of lines.  Unless the @code{MULTILINE} flag has been
set, this will only match at the beginning of the string.  In @code{MULTILINE}
mode, this also matches immediately after each newline within the string.

For example, if you wish to match the word @code{From} only at the beginning of a
line, the RE to use is @code{^From}.

@example
>>> print(re.search('^From', 'From Here to Eternity'))  
<re.Match object; span=(0, 4), match='From'>
>>> print(re.search('^From', 'Reciting From Memory'))
None
@end example

To match a literal @code{'^'}, use @code{\^}.

@item @code{$}

Matches at the end of a line, which is defined as either the end of the string,
or any location followed by a newline character.

@example
>>> print(re.search('@}$', '@{block@}'))  
<re.Match object; span=(6, 7), match='@}'>
>>> print(re.search('@}$', '@{block@} '))
None
>>> print(re.search('@}$', '@{block@}\n'))  
<re.Match object; span=(6, 7), match='@}'>
@end example

To match a literal @code{'$'}, use @code{\$} or enclose it inside a character class,
as in  @code{[$]}.

@item @code{\A}

Matches only at the start of the string.  When not in @code{MULTILINE} mode,
@code{\A} and @code{^} are effectively the same.  In @code{MULTILINE} mode, they’re
different: @code{\A} still matches only at the beginning of the string, but @code{^}
may match at any location inside the string that follows a newline character.

@item @code{\Z}

Matches only at the end of the string.

@item @code{\b}

Word boundary.  This is a zero-width assertion that matches only at the
beginning or end of a word.  A word is defined as a sequence of alphanumeric
characters, so the end of a word is indicated by whitespace or a
non-alphanumeric character.

The following example matches @code{class} only when it’s a complete word; it won’t
match when it’s contained inside another word.

@example
>>> p = re.compile(r'\bclass\b')
>>> print(p.search('no class at all'))
<re.Match object; span=(3, 8), match='class'>
>>> print(p.search('the declassified algorithm'))
None
>>> print(p.search('one subclass is'))
None
@end example

There are two subtleties you should remember when using this special sequence.
First, this is the worst collision between Python’s string literals and regular
expression sequences.  In Python’s string literals, @code{\b} is the backspace
character, ASCII value 8.  If you’re not using raw strings, then Python will
convert the @code{\b} to a backspace, and your RE won’t match as you expect it to.
The following example looks the same as our previous RE, but omits the @code{'r'}
in front of the RE string.

@example
>>> p = re.compile('\bclass\b')
>>> print(p.search('no class at all'))
None
>>> print(p.search('\b' + 'class' + '\b'))
<re.Match object; span=(0, 7), match='\x08class\x08'>
@end example

Second, inside a character class, where there’s no use for this assertion,
@code{\b} represents the backspace character, for compatibility with Python’s
string literals.

@item @code{\B}

Another zero-width assertion, this is the opposite of @code{\b}, only matching when
the current position is not at a word boundary.
@end table

@node Grouping,Non-capturing and Named Groups,More Metacharacters,More Pattern Power
@anchor{howto/regex grouping}@anchor{472d}
@subsubsection Grouping


Frequently you need to obtain more information than just whether the RE matched
or not.  Regular expressions are often used to dissect strings by writing a RE
divided into several subgroups which match different components of interest.
For example, an RFC-822 header line is divided into a header name and a value,
separated by a @code{':'}, like this:

@example
From: author@@example.com
User-Agent: Thunderbird 1.5.0.9 (X11/20061227)
MIME-Version: 1.0
To: editor@@example.com
@end example

This can be handled by writing a regular expression which matches an entire
header line, and has one group which matches the header name, and another group
which matches the header’s value.

Groups are marked by the @code{'('}, @code{')'} metacharacters. @code{'('} and @code{')'}
have much the same meaning as they do in mathematical expressions; they group
together the expressions contained inside them, and you can repeat the contents
of a group with a repeating qualifier, such as @code{*}, @code{+}, @code{?}, or
@code{@{m,n@}}.  For example, @code{(ab)*} will match zero or more repetitions of
@code{ab}.

@example
>>> p = re.compile('(ab)*')
>>> print(p.match('ababababab').span())
(0, 10)
@end example

Groups indicated with @code{'('}, @code{')'} also capture the starting and ending
index of the text that they match; this can be retrieved by passing an argument
to @ref{1bf8,,group()}, @ref{1bfc,,start()}, @ref{1bfd,,end()}, and
@ref{1bfe,,span()}.  Groups are
numbered starting with 0.  Group 0 is always present; it’s the whole RE, so
@ref{b08,,match object} methods all have group 0 as their default
argument.  Later we’ll see how to express groups that don’t capture the span
of text that they match.

@example
>>> p = re.compile('(a)b')
>>> m = p.match('ab')
>>> m.group()
'ab'
>>> m.group(0)
'ab'
@end example

Subgroups are numbered from left to right, from 1 upward.  Groups can be nested;
to determine the number, just count the opening parenthesis characters, going
from left to right.

@example
>>> p = re.compile('(a(b)c)d')
>>> m = p.match('abcd')
>>> m.group(0)
'abcd'
>>> m.group(1)
'abc'
>>> m.group(2)
'b'
@end example

@ref{1bf8,,group()} can be passed multiple group numbers at a time, in which case it
will return a tuple containing the corresponding values for those groups.

@example
>>> m.group(2,1,2)
('b', 'abc', 'b')
@end example

The @ref{1bfa,,groups()} method returns a tuple containing the strings for all the
subgroups, from 1 up to however many there are.

@example
>>> m.groups()
('abc', 'b')
@end example

Backreferences in a pattern allow you to specify that the contents of an earlier
capturing group must also be found at the current location in the string.  For
example, @code{\1} will succeed if the exact contents of group 1 can be found at
the current position, and fails otherwise.  Remember that Python’s string
literals also use a backslash followed by numbers to allow including arbitrary
characters in a string, so be sure to use a raw string when incorporating
backreferences in a RE.

For example, the following RE detects doubled words in a string.

@example
>>> p = re.compile(r'\b(\w+)\s+\1\b')
>>> p.search('Paris in the the spring').group()
'the the'
@end example

Backreferences like this aren’t often useful for just searching through a string
— there are few text formats which repeat data in this way — but you’ll soon
find out that they’re @emph{very} useful when performing string substitutions.

@node Non-capturing and Named Groups,Lookahead Assertions,Grouping,More Pattern Power
@anchor{howto/regex non-capturing-and-named-groups}@anchor{472e}
@subsubsection Non-capturing and Named Groups


Elaborate REs may use many groups, both to capture substrings of interest, and
to group and structure the RE itself.  In complex REs, it becomes difficult to
keep track of the group numbers.  There are two features which help with this
problem.  Both of them use a common syntax for regular expression extensions, so
we’ll look at that first.

Perl 5 is well known for its powerful additions to standard regular expressions.
For these new features the Perl developers couldn’t choose new single-keystroke metacharacters
or new special sequences beginning with @code{\} without making Perl’s regular
expressions confusingly different from standard REs.  If they chose @code{&} as a
new metacharacter, for example, old expressions would be assuming that @code{&} was
a regular character and wouldn’t have escaped it by writing @code{\&} or @code{[&]}.

The solution chosen by the Perl developers was to use @code{(?...)} as the
extension syntax.  @code{?} immediately after a parenthesis was a syntax error
because the @code{?} would have nothing to repeat, so this didn’t introduce any
compatibility problems.  The characters immediately after the @code{?}  indicate
what extension is being used, so @code{(?=foo)} is one thing (a positive lookahead
assertion) and @code{(?:foo)} is something else (a non-capturing group containing
the subexpression @code{foo}).

Python supports several of Perl’s extensions and adds an extension
syntax to Perl’s extension syntax.  If the first character after the
question mark is a @code{P}, you know that it’s an extension that’s
specific to Python.

Now that we’ve looked at the general extension syntax, we can return
to the features that simplify working with groups in complex REs.

Sometimes you’ll want to use a group to denote a part of a regular expression,
but aren’t interested in retrieving the group’s contents. You can make this fact
explicit by using a non-capturing group: @code{(?:...)}, where you can replace the
@code{...} with any other regular expression.

@example
>>> m = re.match("([abc])+", "abc")
>>> m.groups()
('c',)
>>> m = re.match("(?:[abc])+", "abc")
>>> m.groups()
()
@end example

Except for the fact that you can’t retrieve the contents of what the group
matched, a non-capturing group behaves exactly the same as a capturing group;
you can put anything inside it, repeat it with a repetition metacharacter such
as @code{*}, and nest it within other groups (capturing or non-capturing).
@code{(?:...)} is particularly useful when modifying an existing pattern, since you
can add new groups without changing how all the other groups are numbered.  It
should be mentioned that there’s no performance difference in searching between
capturing and non-capturing groups; neither form is any faster than the other.

A more significant feature is named groups: instead of referring to them by
numbers, groups can be referenced by a name.

The syntax for a named group is one of the Python-specific extensions:
@code{(?P<name>...)}.  @emph{name} is, obviously, the name of the group.  Named groups
behave exactly like capturing groups, and additionally associate a name
with a group.  The @ref{b08,,match object} methods that deal with
capturing groups all accept either integers that refer to the group by number
or strings that contain the desired group’s name.  Named groups are still
given numbers, so you can retrieve information about a group in two ways:

@example
>>> p = re.compile(r'(?P<word>\b\w+\b)')
>>> m = p.search( '(((( Lots of punctuation )))' )
>>> m.group('word')
'Lots'
>>> m.group(1)
'Lots'
@end example

Additionally, you can retrieve named groups as a dictionary with
@ref{1bfb,,groupdict()}:

@example
>>> m = re.match(r'(?P<first>\w+) (?P<last>\w+)', 'Jane Doe')
>>> m.groupdict()
@{'first': 'Jane', 'last': 'Doe'@}
@end example

Named groups are handy because they let you use easily-remembered names, instead
of having to remember numbers.  Here’s an example RE from the @ref{93,,imaplib}
module:

@example
InternalDate = re.compile(r'INTERNALDATE "'
        r'(?P<day>[ 123][0-9])-(?P<mon>[A-Z][a-z][a-z])-'
        r'(?P<year>[0-9][0-9][0-9][0-9])'
        r' (?P<hour>[0-9][0-9]):(?P<min>[0-9][0-9]):(?P<sec>[0-9][0-9])'
        r' (?P<zonen>[-+])(?P<zoneh>[0-9][0-9])(?P<zonem>[0-9][0-9])'
        r'"')
@end example

It’s obviously much easier to retrieve @code{m.group('zonem')}, instead of having
to remember to retrieve group 9.

The syntax for backreferences in an expression such as @code{(...)\1} refers to the
number of the group.  There’s naturally a variant that uses the group name
instead of the number. This is another Python extension: @code{(?P=name)} indicates
that the contents of the group called @emph{name} should again be matched at the
current point.  The regular expression for finding doubled words,
@code{\b(\w+)\s+\1\b} can also be written as @code{\b(?P<word>\w+)\s+(?P=word)\b}:

@example
>>> p = re.compile(r'\b(?P<word>\w+)\s+(?P=word)\b')
>>> p.search('Paris in the the spring').group()
'the the'
@end example

@node Lookahead Assertions,,Non-capturing and Named Groups,More Pattern Power
@anchor{howto/regex lookahead-assertions}@anchor{472f}
@subsubsection Lookahead Assertions


Another zero-width assertion is the lookahead assertion.  Lookahead assertions
are available in both positive and negative form, and  look like this:


@table @asis

@item @code{(?=...)}

Positive lookahead assertion.  This succeeds if the contained regular
expression, represented here by @code{...}, successfully matches at the current
location, and fails otherwise. But, once the contained expression has been
tried, the matching engine doesn’t advance at all; the rest of the pattern is
tried right where the assertion started.

@item @code{(?!...)}

Negative lookahead assertion.  This is the opposite of the positive assertion;
it succeeds if the contained expression @emph{doesn’t} match at the current position
in the string.
@end table

To make this concrete, let’s look at a case where a lookahead is useful.
Consider a simple pattern to match a filename and split it apart into a base
name and an extension, separated by a @code{.}.  For example, in @code{news.rc},
@code{news} is the base name, and @code{rc} is the filename’s extension.

The pattern to match this is quite simple:

@code{.*[.].*$}

Notice that the @code{.} needs to be treated specially because it’s a
metacharacter, so it’s inside a character class to only match that
specific character.  Also notice the trailing @code{$}; this is added to
ensure that all the rest of the string must be included in the
extension.  This regular expression matches @code{foo.bar} and
@code{autoexec.bat} and @code{sendmail.cf} and @code{printers.conf}.

Now, consider complicating the problem a bit; what if you want to match
filenames where the extension is not @code{bat}? Some incorrect attempts:

@code{.*[.][^b].*$}  The first attempt above tries to exclude @code{bat} by requiring
that the first character of the extension is not a @code{b}.  This is wrong,
because the pattern also doesn’t match @code{foo.bar}.

@code{.*[.]([^b]..|.[^a].|..[^t])$}

The expression gets messier when you try to patch up the first solution by
requiring one of the following cases to match: the first character of the
extension isn’t @code{b}; the second character isn’t @code{a}; or the third character
isn’t @code{t}.  This accepts @code{foo.bar} and rejects @code{autoexec.bat}, but it
requires a three-letter extension and won’t accept a filename with a two-letter
extension such as @code{sendmail.cf}.  We’ll complicate the pattern again in an
effort to fix it.

@code{.*[.]([^b].?.?|.[^a]?.?|..?[^t]?)$}

In the third attempt, the second and third letters are all made optional in
order to allow matching extensions shorter than three characters, such as
@code{sendmail.cf}.

The pattern’s getting really complicated now, which makes it hard to read and
understand.  Worse, if the problem changes and you want to exclude both @code{bat}
and @code{exe} as extensions, the pattern would get even more complicated and
confusing.

A negative lookahead cuts through all this confusion:

@code{.*[.](?!bat$)[^.]*$}  The negative lookahead means: if the expression @code{bat}
doesn’t match at this point, try the rest of the pattern; if @code{bat$} does
match, the whole pattern will fail.  The trailing @code{$} is required to ensure
that something like @code{sample.batch}, where the extension only starts with
@code{bat}, will be allowed.  The @code{[^.]*} makes sure that the pattern works
when there are multiple dots in the filename.

Excluding another filename extension is now easy; simply add it as an
alternative inside the assertion.  The following pattern excludes filenames that
end in either @code{bat} or @code{exe}:

@code{.*[.](?!bat$|exe$)[^.]*$}

@node Modifying Strings,Common Problems,More Pattern Power,Regular Expression HOWTO
@anchor{howto/regex modifying-strings}@anchor{4730}
@subsection Modifying Strings


Up to this point, we’ve simply performed searches against a static string.
Regular expressions are also commonly used to modify strings in various ways,
using the following pattern methods:


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Method/Attribute

@tab

Purpose

@item

@code{split()}

@tab

Split the string into a list, splitting it
wherever the RE matches

@item

@code{sub()}

@tab

Find all substrings where the RE matches, and
replace them with a different string

@item

@code{subn()}

@tab

Does the same thing as @code{sub()},  but
returns the new string and the number of
replacements

@end multitable


@menu
* Splitting Strings:: 
* Search and Replace:: 

@end menu

@node Splitting Strings,Search and Replace,,Modifying Strings
@anchor{howto/regex splitting-strings}@anchor{4731}
@subsubsection Splitting Strings


The @ref{1bed,,split()} method of a pattern splits a string apart
wherever the RE matches, returning a list of the pieces. It’s similar to the
@ref{a1b,,split()} method of strings but provides much more generality in the
delimiters that you can split by; string @code{split()} only supports splitting by
whitespace or by a fixed string.  As you’d expect, there’s a module-level
@ref{699,,re.split()} function, too.


@deffn {Method} .split (string[, maxsplit=0])

Split @emph{string} by the matches of the regular expression.  If capturing
parentheses are used in the RE, then their contents will also be returned as
part of the resulting list.  If @emph{maxsplit} is nonzero, at most @emph{maxsplit} splits
are performed.
@end deffn

You can limit the number of splits made, by passing a value for @emph{maxsplit}.
When @emph{maxsplit} is nonzero, at most @emph{maxsplit} splits will be made, and the
remainder of the string is returned as the final element of the list.  In the
following example, the delimiter is any sequence of non-alphanumeric characters.

@example
>>> p = re.compile(r'\W+')
>>> p.split('This is a test, short and sweet, of split().')
['This', 'is', 'a', 'test', 'short', 'and', 'sweet', 'of', 'split', '@w{'}]
>>> p.split('This is a test, short and sweet, of split().', 3)
['This', 'is', 'a', 'test, short and sweet, of split().']
@end example

Sometimes you’re not only interested in what the text between delimiters is, but
also need to know what the delimiter was.  If capturing parentheses are used in
the RE, then their values are also returned as part of the list.  Compare the
following calls:

@example
>>> p = re.compile(r'\W+')
>>> p2 = re.compile(r'(\W+)')
>>> p.split('This... is a test.')
['This', 'is', 'a', 'test', '@w{'}]
>>> p2.split('This... is a test.')
['This', '... ', 'is', ' ', 'a', ' ', 'test', '.', '@w{'}]
@end example

The module-level function @ref{699,,re.split()} adds the RE to be used as the first
argument, but is otherwise the same.

@example
>>> re.split(r'[\W]+', 'Words, words, words.')
['Words', 'words', 'words', '@w{'}]
>>> re.split(r'([\W]+)', 'Words, words, words.')
['Words', ', ', 'words', ', ', 'words', '.', '@w{'}]
>>> re.split(r'[\W]+', 'Words, words, words.', 1)
['Words', 'words, words.']
@end example

@node Search and Replace,,Splitting Strings,Modifying Strings
@anchor{howto/regex search-and-replace}@anchor{4732}
@subsubsection Search and Replace


Another common task is to find all the matches for a pattern, and replace them
with a different string.  The @ref{1bf0,,sub()} method takes a replacement value,
which can be either a string or a function, and the string to be processed.


@deffn {Method} .sub (replacement, string[, count=0])

Returns the string obtained by replacing the leftmost non-overlapping
occurrences of the RE in @emph{string} by the replacement @emph{replacement}.  If the
pattern isn’t found, @emph{string} is returned unchanged.

The optional argument @emph{count} is the maximum number of pattern occurrences to be
replaced; @emph{count} must be a non-negative integer.  The default value of 0 means
to replace all occurrences.
@end deffn

Here’s a simple example of using the @ref{1bf0,,sub()} method.  It replaces colour
names with the word @code{colour}:

@example
>>> p = re.compile('(blue|white|red)')
>>> p.sub('colour', 'blue socks and red shoes')
'colour socks and colour shoes'
>>> p.sub('colour', 'blue socks and red shoes', count=1)
'colour socks and red shoes'
@end example

The @ref{1bf1,,subn()} method does the same work, but returns a 2-tuple containing the
new string value and the number of replacements  that were performed:

@example
>>> p = re.compile('(blue|white|red)')
>>> p.subn('colour', 'blue socks and red shoes')
('colour socks and colour shoes', 2)
>>> p.subn('colour', 'no colours at all')
('no colours at all', 0)
@end example

Empty matches are replaced only when they’re not adjacent to a previous empty match.

@example
>>> p = re.compile('x*')
>>> p.sub('-', 'abxd')
'-a-b--d-'
@end example

If @emph{replacement} is a string, any backslash escapes in it are processed.  That
is, @code{\n} is converted to a single newline character, @code{\r} is converted to a
carriage return, and so forth. Unknown escapes such as @code{\&} are left alone.
Backreferences, such as @code{\6}, are replaced with the substring matched by the
corresponding group in the RE.  This lets you incorporate portions of the
original text in the resulting replacement string.

This example matches the word @code{section} followed by a string enclosed in
@code{@{}, @code{@}}, and changes @code{section} to @code{subsection}:

@example
>>> p = re.compile('section@{ ( [^@}]* ) @}', re.VERBOSE)
>>> p.sub(r'subsection@{\1@}','section@{First@} section@{second@}')
'subsection@{First@} subsection@{second@}'
@end example

There’s also a syntax for referring to named groups as defined by the
@code{(?P<name>...)} syntax.  @code{\g<name>} will use the substring matched by the
group named @code{name}, and  @code{\g<number>}  uses the corresponding group number.
@code{\g<2>} is therefore equivalent to @code{\2},  but isn’t ambiguous in a
replacement string such as @code{\g<2>0}.  (@code{\20} would be interpreted as a
reference to group 20, not a reference to group 2 followed by the literal
character @code{'0'}.)  The following substitutions are all equivalent, but use all
three variations of the replacement string.

@example
>>> p = re.compile('section@{ (?P<name> [^@}]* ) @}', re.VERBOSE)
>>> p.sub(r'subsection@{\1@}','section@{First@}')
'subsection@{First@}'
>>> p.sub(r'subsection@{\g<1>@}','section@{First@}')
'subsection@{First@}'
>>> p.sub(r'subsection@{\g<name>@}','section@{First@}')
'subsection@{First@}'
@end example

@emph{replacement} can also be a function, which gives you even more control.  If
@emph{replacement} is a function, the function is called for every non-overlapping
occurrence of @emph{pattern}.  On each call, the function is passed a
@ref{b08,,match object} argument for the match and can use this
information to compute the desired replacement string and return it.

In the following example, the replacement function translates decimals into
hexadecimal:

@example
>>> def hexrepl(match):
...     "Return the hex string for a decimal number"
...     value = int(match.group())
...     return hex(value)
...
>>> p = re.compile(r'\d+')
>>> p.sub(hexrepl, 'Call 65490 for printing, 49152 for user code.')
'Call 0xffd2 for printing, 0xc000 for user code.'
@end example

When using the module-level @ref{72e,,re.sub()} function, the pattern is passed as
the first argument.  The pattern may be provided as an object or as a string; if
you need to specify regular expression flags, you must either use a
pattern object as the first parameter, or use embedded modifiers in the
pattern string, e.g. @code{sub("(?i)b+", "x", "bbbb BBBB")} returns @code{'x x'}.

@node Common Problems,Feedback,Modifying Strings,Regular Expression HOWTO
@anchor{howto/regex common-problems}@anchor{4733}
@subsection Common Problems


Regular expressions are a powerful tool for some applications, but in some ways
their behaviour isn’t intuitive and at times they don’t behave the way you may
expect them to.  This section will point out some of the most common pitfalls.

@menu
* Use String Methods:: 
* match() versus search(): match versus search. 
* Greedy versus Non-Greedy:: 
* Using re.VERBOSE: Using re VERBOSE. 

@end menu

@node Use String Methods,match versus search,,Common Problems
@anchor{howto/regex use-string-methods}@anchor{4734}
@subsubsection Use String Methods


Sometimes using the @ref{d8,,re} module is a mistake.  If you’re matching a fixed
string, or a single character class, and you’re not using any @ref{d8,,re} features
such as the @ref{1151,,IGNORECASE} flag, then the full power of regular expressions
may not be required. Strings have several methods for performing operations with
fixed strings and they’re usually much faster, because the implementation is a
single small C loop that’s been optimized for the purpose, instead of the large,
more generalized regular expression engine.

One example might be replacing a single fixed string with another one; for
example, you might replace @code{word} with @code{deed}.  @ref{72e,,re.sub()} seems like the
function to use for this, but consider the @ref{1aac,,replace()} method.  Note that
@code{replace()} will also replace @code{word} inside words, turning @code{swordfish}
into @code{sdeedfish}, but the  naive RE @code{word} would have done that, too.  (To
avoid performing the substitution on parts of words, the pattern would have to
be @code{\bword\b}, in order to require that @code{word} have a word boundary on
either side.  This takes the job beyond  @code{replace()}’s abilities.)

Another common task is deleting every occurrence of a single character from a
string or replacing it with another single character.  You might do this with
something like @code{re.sub('\n', ' ', S)}, but @ref{1aab,,translate()} is capable of
doing both tasks and will be faster than any regular expression operation can
be.

In short, before turning to the @ref{d8,,re} module, consider whether your problem
can be solved with a faster and simpler string method.

@node match versus search,Greedy versus Non-Greedy,Use String Methods,Common Problems
@anchor{howto/regex match-versus-search}@anchor{4735}
@subsubsection match() versus search()


The @ref{dd3,,match()} function only checks if the RE matches at the beginning of the
string while @ref{dd2,,search()} will scan forward through the string for a match.
It’s important to keep this distinction in mind.  Remember,  @code{match()} will
only report a successful match which will start at 0; if the match wouldn’t
start at zero,  @code{match()} will @emph{not} report it.

@example
>>> print(re.match('super', 'superstition').span())
(0, 5)
>>> print(re.match('super', 'insuperable'))
None
@end example

On the other hand, @ref{dd2,,search()} will scan forward through the string,
reporting the first match it finds.

@example
>>> print(re.search('super', 'superstition').span())
(0, 5)
>>> print(re.search('super', 'insuperable').span())
(2, 7)
@end example

Sometimes you’ll be tempted to keep using @ref{dd3,,re.match()}, and just add @code{.*}
to the front of your RE.  Resist this temptation and use @ref{dd2,,re.search()}
instead.  The regular expression compiler does some analysis of REs in order to
speed up the process of looking for a match.  One such analysis figures out what
the first character of a match must be; for example, a pattern starting with
@code{Crow} must match starting with a @code{'C'}.  The analysis lets the engine
quickly scan through the string looking for the starting character, only trying
the full match if a @code{'C'} is found.

Adding @code{.*} defeats this optimization, requiring scanning to the end of the
string and then backtracking to find a match for the rest of the RE.  Use
@ref{dd2,,re.search()} instead.

@node Greedy versus Non-Greedy,Using re VERBOSE,match versus search,Common Problems
@anchor{howto/regex greedy-versus-non-greedy}@anchor{4736}
@subsubsection Greedy versus Non-Greedy


When repeating a regular expression, as in @code{a*}, the resulting action is to
consume as much of the pattern as possible.  This fact often bites you when
you’re trying to match a pair of balanced delimiters, such as the angle brackets
surrounding an HTML tag.  The naive pattern for matching a single HTML tag
doesn’t work because of the greedy nature of @code{.*}.

@example
>>> s = '<html><head><title>Title</title>'
>>> len(s)
32
>>> print(re.match('<.*>', s).span())
(0, 32)
>>> print(re.match('<.*>', s).group())
<html><head><title>Title</title>
@end example

The RE matches the @code{'<'} in @code{'<html>'}, and the @code{.*} consumes the rest of
the string.  There’s still more left in the RE, though, and the @code{>} can’t
match at the end of the string, so the regular expression engine has to
backtrack character by character until it finds a match for the @code{>}.   The
final match extends from the @code{'<'} in @code{'<html>'} to the @code{'>'} in
@code{'</title>'}, which isn’t what you want.

In this case, the solution is to use the non-greedy qualifiers @code{*?}, @code{+?},
@code{??}, or @code{@{m,n@}?}, which match as @emph{little} text as possible.  In the above
example, the @code{'>'} is tried immediately after the first @code{'<'} matches, and
when it fails, the engine advances a character at a time, retrying the @code{'>'}
at every step.  This produces just the right result:

@example
>>> print(re.match('<.*?>', s).group())
<html>
@end example

(Note that parsing HTML or XML with regular expressions is painful.
Quick-and-dirty patterns will handle common cases, but HTML and XML have special
cases that will break the obvious regular expression; by the time you’ve written
a regular expression that handles all of the possible cases, the patterns will
be @emph{very} complicated.  Use an HTML or XML parser module for such tasks.)

@node Using re VERBOSE,,Greedy versus Non-Greedy,Common Problems
@anchor{howto/regex using-re-verbose}@anchor{4737}
@subsubsection Using re.VERBOSE


By now you’ve probably noticed that regular expressions are a very compact
notation, but they’re not terribly readable.  REs of moderate complexity can
become lengthy collections of backslashes, parentheses, and metacharacters,
making them difficult to read and understand.

For such REs, specifying the @ref{1be7,,re.VERBOSE} flag when compiling the regular
expression can be helpful, because it allows you to format the regular
expression more clearly.

The @code{re.VERBOSE} flag has several effects.  Whitespace in the regular
expression that @emph{isn’t} inside a character class is ignored.  This means that an
expression such as @code{dog | cat} is equivalent to the less readable @code{dog|cat},
but @code{[a b]} will still match the characters @code{'a'}, @code{'b'}, or a space.  In
addition, you can also put comments inside a RE; comments extend from a @code{#}
character to the next newline.  When used with triple-quoted strings, this
enables REs to be formatted more neatly:

@example
pat = re.compile(r"""
 \s*                 # Skip leading whitespace
 (?P<header>[^:]+)   # Header name
 \s* :               # Whitespace, and a colon
 (?P<value>.*?)      # The header's value -- *? used to
                     # lose the following trailing whitespace
 \s*$                # Trailing whitespace to end-of-line
""", re.VERBOSE)
@end example

This is far more readable than:

@example
pat = re.compile(r"\s*(?P<header>[^:]+)\s*:(?P<value>.*?)\s*$")
@end example

@node Feedback,,Common Problems,Regular Expression HOWTO
@anchor{howto/regex feedback}@anchor{4738}
@subsection Feedback


Regular expressions are a complicated topic.  Did this document help you
understand them?  Were there parts that were unclear, or Problems you
encountered that weren’t covered here?  If so, please send suggestions for
improvements to the author.

The most complete book on regular expressions is almost certainly Jeffrey
Friedl’s Mastering Regular Expressions, published by O’Reilly.  Unfortunately,
it exclusively concentrates on Perl and Java’s flavours of regular expressions,
and doesn’t contain any Python material at all, so it won’t be useful as a
reference for programming in Python.  (The first edition covered Python’s
now-removed @code{regex} module, which won’t help you much.)  Consider checking
it out from your library.

@node Socket Programming HOWTO,Sorting HOW TO,Regular Expression HOWTO,Python HOWTOs
@anchor{howto/sockets doc}@anchor{4739}@anchor{howto/sockets socket-howto}@anchor{2b21}@anchor{howto/sockets socket-programming-howto}@anchor{473a}
@section Socket Programming HOWTO


@*Author: 
Gordon McMillan


@subsubheading Abstract



Sockets are used nearly everywhere, but are one of the most severely
misunderstood technologies around. This is a 10,000 foot overview of sockets.
It’s not really a tutorial - you’ll still have work to do in getting things
operational. It doesn’t cover the fine points (and there are a lot of them), but
I hope it will give you enough background to begin using them decently.

@menu
* Sockets:: 
* Creating a Socket:: 
* Using a Socket:: 
* Disconnecting:: 
* Non-blocking Sockets:: 

@end menu

@node Sockets,Creating a Socket,,Socket Programming HOWTO
@anchor{howto/sockets sockets}@anchor{473b}
@subsection Sockets


I’m only going to talk about INET (i.e. IPv4) sockets, but they account for at least 99% of
the sockets in use. And I’ll only talk about STREAM (i.e. TCP) sockets - unless you really
know what you’re doing (in which case this HOWTO isn’t for you!), you’ll get
better behavior and performance from a STREAM socket than anything else. I will
try to clear up the mystery of what a socket is, as well as some hints on how to
work with blocking and non-blocking sockets. But I’ll start by talking about
blocking sockets. You’ll need to know how they work before dealing with
non-blocking sockets.

Part of the trouble with understanding these things is that “socket” can mean a
number of subtly different things, depending on context. So first, let’s make a
distinction between a “client” socket - an endpoint of a conversation, and a
“server” socket, which is more like a switchboard operator. The client
application (your browser, for example) uses “client” sockets exclusively; the
web server it’s talking to uses both “server” sockets and “client” sockets.

@menu
* History:: 

@end menu

@node History,,,Sockets
@anchor{howto/sockets history}@anchor{473c}
@subsubsection History


Of the various forms of @abbr{IPC,Inter Process Communication},
sockets are by far the most popular.  On any given platform, there are
likely to be other forms of IPC that are faster, but for
cross-platform communication, sockets are about the only game in town.

They were invented in Berkeley as part of the BSD flavor of Unix. They spread
like wildfire with the internet. With good reason — the combination of sockets
with INET makes talking to arbitrary machines around the world unbelievably easy
(at least compared to other schemes).

@node Creating a Socket,Using a Socket,Sockets,Socket Programming HOWTO
@anchor{howto/sockets creating-a-socket}@anchor{473d}
@subsection Creating a Socket


Roughly speaking, when you clicked on the link that brought you to this page,
your browser did something like the following:

@example
# create an INET, STREAMing socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# now connect to the web server on port 80 - the normal http port
s.connect(("www.python.org", 80))
@end example

When the @code{connect} completes, the socket @code{s} can be used to send
in a request for the text of the page. The same socket will read the
reply, and then be destroyed. That’s right, destroyed. Client sockets
are normally only used for one exchange (or a small set of sequential
exchanges).

What happens in the web server is a bit more complex. First, the web server
creates a “server socket”:

@example
# create an INET, STREAMing socket
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# bind the socket to a public host, and a well-known port
serversocket.bind((socket.gethostname(), 80))
# become a server socket
serversocket.listen(5)
@end example

A couple things to notice: we used @code{socket.gethostname()} so that the socket
would be visible to the outside world.  If we had used @code{s.bind(('localhost',
80))} or @code{s.bind(('127.0.0.1', 80))} we would still have a “server” socket,
but one that was only visible within the same machine.  @code{s.bind(('@w{'}, 80))}
specifies that the socket is reachable by any address the machine happens to
have.

A second thing to note: low number ports are usually reserved for “well known”
services (HTTP, SNMP etc). If you’re playing around, use a nice high number (4
digits).

Finally, the argument to @code{listen} tells the socket library that we want it to
queue up as many as 5 connect requests (the normal max) before refusing outside
connections. If the rest of the code is written properly, that should be plenty.

Now that we have a “server” socket, listening on port 80, we can enter the
mainloop of the web server:

@example
while True:
    # accept connections from outside
    (clientsocket, address) = serversocket.accept()
    # now do something with the clientsocket
    # in this case, we'll pretend this is a threaded server
    ct = client_thread(clientsocket)
    ct.run()
@end example

There’s actually 3 general ways in which this loop could work - dispatching a
thread to handle @code{clientsocket}, create a new process to handle
@code{clientsocket}, or restructure this app to use non-blocking sockets, and
multiplex between our “server” socket and any active @code{clientsocket}s using
@code{select}. More about that later. The important thing to understand now is
this: this is @emph{all} a “server” socket does. It doesn’t send any data. It doesn’t
receive any data. It just produces “client” sockets. Each @code{clientsocket} is
created in response to some @emph{other} “client” socket doing a @code{connect()} to the
host and port we’re bound to. As soon as we’ve created that @code{clientsocket}, we
go back to listening for more connections. The two “clients” are free to chat it
up - they are using some dynamically allocated port which will be recycled when
the conversation ends.

@menu
* IPC:: 

@end menu

@node IPC,,,Creating a Socket
@anchor{howto/sockets ipc}@anchor{473e}
@subsubsection IPC


If you need fast IPC between two processes on one machine, you should look into
pipes or shared memory.  If you do decide to use AF_INET sockets, bind the
“server” socket to @code{'localhost'}. On most platforms, this will take a
shortcut around a couple of layers of network code and be quite a bit faster.


@subsubheading See also


The @ref{b3,,multiprocessing} integrates cross-platform IPC into a higher-level
API.


@node Using a Socket,Disconnecting,Creating a Socket,Socket Programming HOWTO
@anchor{howto/sockets using-a-socket}@anchor{473f}
@subsection Using a Socket


The first thing to note, is that the web browser’s “client” socket and the web
server’s “client” socket are identical beasts. That is, this is a “peer to peer”
conversation. Or to put it another way, @emph{as the designer, you will have to
decide what the rules of etiquette are for a conversation}. Normally, the
@code{connect}ing socket starts the conversation, by sending in a request, or
perhaps a signon. But that’s a design decision - it’s not a rule of sockets.

Now there are two sets of verbs to use for communication. You can use @code{send}
and @code{recv}, or you can transform your client socket into a file-like beast and
use @code{read} and @code{write}. The latter is the way Java presents its sockets.
I’m not going to talk about it here, except to warn you that you need to use
@code{flush} on sockets. These are buffered “files”, and a common mistake is to
@code{write} something, and then @code{read} for a reply. Without a @code{flush} in
there, you may wait forever for the reply, because the request may still be in
your output buffer.

Now we come to the major stumbling block of sockets - @code{send} and @code{recv} operate
on the network buffers. They do not necessarily handle all the bytes you hand
them (or expect from them), because their major focus is handling the network
buffers. In general, they return when the associated network buffers have been
filled (@code{send}) or emptied (@code{recv}). They then tell you how many bytes they
handled. It is @emph{your} responsibility to call them again until your message has
been completely dealt with.

When a @code{recv} returns 0 bytes, it means the other side has closed (or is in
the process of closing) the connection.  You will not receive any more data on
this connection. Ever.  You may be able to send data successfully; I’ll talk
more about this later.

A protocol like HTTP uses a socket for only one transfer. The client sends a
request, then reads a reply.  That’s it. The socket is discarded. This means that
a client can detect the end of the reply by receiving 0 bytes.

But if you plan to reuse your socket for further transfers, you need to realize
that @emph{there is no} @abbr{EOT,End of Transfer} @emph{on a socket.} I repeat: if a socket
@code{send} or @code{recv} returns after handling 0 bytes, the connection has been
broken.  If the connection has @emph{not} been broken, you may wait on a @code{recv}
forever, because the socket will @emph{not} tell you that there’s nothing more to
read (for now).  Now if you think about that a bit, you’ll come to realize a
fundamental truth of sockets: @emph{messages must either be fixed length} (yuck), @emph{or
be delimited} (shrug), @emph{or indicate how long they are} (much better), @emph{or end by
shutting down the connection}. The choice is entirely yours, (but some ways are
righter than others).

Assuming you don’t want to end the connection, the simplest solution is a fixed
length message:

@example
class MySocket:
    """demonstration class only
      - coded for clarity, not efficiency
    """

    def __init__(self, sock=None):
        if sock is None:
            self.sock = socket.socket(
                            socket.AF_INET, socket.SOCK_STREAM)
        else:
            self.sock = sock

    def connect(self, host, port):
        self.sock.connect((host, port))

    def mysend(self, msg):
        totalsent = 0
        while totalsent < MSGLEN:
            sent = self.sock.send(msg[totalsent:])
            if sent == 0:
                raise RuntimeError("socket connection broken")
            totalsent = totalsent + sent

    def myreceive(self):
        chunks = []
        bytes_recd = 0
        while bytes_recd < MSGLEN:
            chunk = self.sock.recv(min(MSGLEN - bytes_recd, 2048))
            if chunk == b'@w{'}:
                raise RuntimeError("socket connection broken")
            chunks.append(chunk)
            bytes_recd = bytes_recd + len(chunk)
        return b'@w{'}.join(chunks)
@end example

The sending code here is usable for almost any messaging scheme - in Python you
send strings, and you can use @code{len()} to determine its length (even if it has
embedded @code{\0} characters). It’s mostly the receiving code that gets more
complex. (And in C, it’s not much worse, except you can’t use @code{strlen} if the
message has embedded @code{\0}s.)

The easiest enhancement is to make the first character of the message an
indicator of message type, and have the type determine the length. Now you have
two @code{recv}s - the first to get (at least) that first character so you can
look up the length, and the second in a loop to get the rest. If you decide to
go the delimited route, you’ll be receiving in some arbitrary chunk size, (4096
or 8192 is frequently a good match for network buffer sizes), and scanning what
you’ve received for a delimiter.

One complication to be aware of: if your conversational protocol allows multiple
messages to be sent back to back (without some kind of reply), and you pass
@code{recv} an arbitrary chunk size, you may end up reading the start of a
following message. You’ll need to put that aside and hold onto it, until it’s
needed.

Prefixing the message with its length (say, as 5 numeric characters) gets more
complex, because (believe it or not), you may not get all 5 characters in one
@code{recv}. In playing around, you’ll get away with it; but in high network loads,
your code will very quickly break unless you use two @code{recv} loops - the first
to determine the length, the second to get the data part of the message. Nasty.
This is also when you’ll discover that @code{send} does not always manage to get
rid of everything in one pass. And despite having read this, you will eventually
get bit by it!

In the interests of space, building your character, (and preserving my
competitive position), these enhancements are left as an exercise for the
reader. Lets move on to cleaning up.

@menu
* Binary Data:: 

@end menu

@node Binary Data,,,Using a Socket
@anchor{howto/sockets binary-data}@anchor{4740}
@subsubsection Binary Data


It is perfectly possible to send binary data over a socket. The major problem is
that not all machines use the same formats for binary data. For example, a
Motorola chip will represent a 16 bit integer with the value 1 as the two hex
bytes 00 01. Intel and DEC, however, are byte-reversed - that same 1 is 01 00.
Socket libraries have calls for converting 16 and 32 bit integers - @code{ntohl,
htonl, ntohs, htons} where “n” means @emph{network} and “h” means @emph{host}, “s” means
@emph{short} and “l” means @emph{long}. Where network order is host order, these do
nothing, but where the machine is byte-reversed, these swap the bytes around
appropriately.

In these days of 32 bit machines, the ascii representation of binary data is
frequently smaller than the binary representation. That’s because a surprising
amount of the time, all those longs have the value 0, or maybe 1. The string “0”
would be two bytes, while binary is four. Of course, this doesn’t fit well with
fixed-length messages. Decisions, decisions.

@node Disconnecting,Non-blocking Sockets,Using a Socket,Socket Programming HOWTO
@anchor{howto/sockets disconnecting}@anchor{4741}
@subsection Disconnecting


Strictly speaking, you’re supposed to use @code{shutdown} on a socket before you
@code{close} it.  The @code{shutdown} is an advisory to the socket at the other end.
Depending on the argument you pass it, it can mean “I’m not going to send
anymore, but I’ll still listen”, or “I’m not listening, good riddance!”.  Most
socket libraries, however, are so used to programmers neglecting to use this
piece of etiquette that normally a @code{close} is the same as @code{shutdown();
close()}.  So in most situations, an explicit @code{shutdown} is not needed.

One way to use @code{shutdown} effectively is in an HTTP-like exchange. The client
sends a request and then does a @code{shutdown(1)}. This tells the server “This
client is done sending, but can still receive.”  The server can detect “EOF” by
a receive of 0 bytes. It can assume it has the complete request.  The server
sends a reply. If the @code{send} completes successfully then, indeed, the client
was still receiving.

Python takes the automatic shutdown a step further, and says that when a socket
is garbage collected, it will automatically do a @code{close} if it’s needed. But
relying on this is a very bad habit. If your socket just disappears without
doing a @code{close}, the socket at the other end may hang indefinitely, thinking
you’re just being slow. @emph{Please} @code{close} your sockets when you’re done.

@menu
* When Sockets Die:: 

@end menu

@node When Sockets Die,,,Disconnecting
@anchor{howto/sockets when-sockets-die}@anchor{4742}
@subsubsection When Sockets Die


Probably the worst thing about using blocking sockets is what happens when the
other side comes down hard (without doing a @code{close}). Your socket is likely to
hang. TCP is a reliable protocol, and it will wait a long, long time
before giving up on a connection. If you’re using threads, the entire thread is
essentially dead. There’s not much you can do about it. As long as you aren’t
doing something dumb, like holding a lock while doing a blocking read, the
thread isn’t really consuming much in the way of resources. Do @emph{not} try to kill
the thread - part of the reason that threads are more efficient than processes
is that they avoid the overhead associated with the automatic recycling of
resources. In other words, if you do manage to kill the thread, your whole
process is likely to be screwed up.

@node Non-blocking Sockets,,Disconnecting,Socket Programming HOWTO
@anchor{howto/sockets non-blocking-sockets}@anchor{4743}
@subsection Non-blocking Sockets


If you’ve understood the preceding, you already know most of what you need to
know about the mechanics of using sockets. You’ll still use the same calls, in
much the same ways. It’s just that, if you do it right, your app will be almost
inside-out.

In Python, you use @code{socket.setblocking(False)} to make it non-blocking. In C, it’s
more complex, (for one thing, you’ll need to choose between the BSD flavor
@code{O_NONBLOCK} and the almost indistinguishable POSIX flavor @code{O_NDELAY}, which
is completely different from @code{TCP_NODELAY}), but it’s the exact same idea. You
do this after creating the socket, but before using it. (Actually, if you’re
nuts, you can switch back and forth.)

The major mechanical difference is that @code{send}, @code{recv}, @code{connect} and
@code{accept} can return without having done anything. You have (of course) a
number of choices. You can check return code and error codes and generally drive
yourself crazy. If you don’t believe me, try it sometime. Your app will grow
large, buggy and suck CPU. So let’s skip the brain-dead solutions and do it
right.

Use @code{select}.

In C, coding @code{select} is fairly complex. In Python, it’s a piece of cake, but
it’s close enough to the C version that if you understand @code{select} in Python,
you’ll have little trouble with it in C:

@example
ready_to_read, ready_to_write, in_error = \
               select.select(
                  potential_readers,
                  potential_writers,
                  potential_errs,
                  timeout)
@end example

You pass @code{select} three lists: the first contains all sockets that you might
want to try reading; the second all the sockets you might want to try writing
to, and the last (normally left empty) those that you want to check for errors.
You should note that a socket can go into more than one list. The @code{select}
call is blocking, but you can give it a timeout. This is generally a sensible
thing to do - give it a nice long timeout (say a minute) unless you have good
reason to do otherwise.

In return, you will get three lists. They contain the sockets that are actually
readable, writable and in error. Each of these lists is a subset (possibly
empty) of the corresponding list you passed in.

If a socket is in the output readable list, you can be
as-close-to-certain-as-we-ever-get-in-this-business that a @code{recv} on that
socket will return @emph{something}. Same idea for the writable list. You’ll be able
to send @emph{something}. Maybe not all you want to, but @emph{something} is better than
nothing.  (Actually, any reasonably healthy socket will return as writable - it
just means outbound network buffer space is available.)

If you have a “server” socket, put it in the potential_readers list. If it comes
out in the readable list, your @code{accept} will (almost certainly) work. If you
have created a new socket to @code{connect} to someone else, put it in the
potential_writers list. If it shows up in the writable list, you have a decent
chance that it has connected.

Actually, @code{select} can be handy even with blocking sockets. It’s one way of
determining whether you will block - the socket returns as readable when there’s
something in the buffers.  However, this still doesn’t help with the problem of
determining whether the other end is done, or just busy with something else.

@strong{Portability alert}: On Unix, @code{select} works both with the sockets and
files. Don’t try this on Windows. On Windows, @code{select} works with sockets
only. Also note that in C, many of the more advanced socket options are done
differently on Windows. In fact, on Windows I usually use threads (which work
very, very well) with my sockets.

@node Sorting HOW TO,Unicode HOWTO,Socket Programming HOWTO,Python HOWTOs
@anchor{howto/sorting doc}@anchor{4744}@anchor{howto/sorting sorting-how-to}@anchor{4745}@anchor{howto/sorting sortinghowto}@anchor{1a5e}
@section Sorting HOW TO


@*Author: 
Andrew Dalke and Raymond Hettinger


@*Release: 
0.1


Python lists have a built-in @ref{705,,list.sort()} method that modifies the list
in-place.  There is also a @ref{704,,sorted()} built-in function that builds a new
sorted list from an iterable.

In this document, we explore the various techniques for sorting data using Python.

@menu
* Sorting Basics:: 
* Key Functions:: 
* Operator Module Functions:: 
* Ascending and Descending:: 
* Sort Stability and Complex Sorts:: 
* The Old Way Using Decorate-Sort-Undecorate:: 
* The Old Way Using the cmp Parameter:: 
* Odd and Ends:: 

@end menu

@node Sorting Basics,Key Functions,,Sorting HOW TO
@anchor{howto/sorting sorting-basics}@anchor{4746}
@subsection Sorting Basics


A simple ascending sort is very easy: just call the @ref{704,,sorted()} function. It
returns a new sorted list:

@example
>>> sorted([5, 2, 3, 1, 4])
[1, 2, 3, 4, 5]
@end example

You can also use the @ref{705,,list.sort()} method. It modifies the list
in-place (and returns @code{None} to avoid confusion). Usually it’s less convenient
than @ref{704,,sorted()} - but if you don’t need the original list, it’s slightly
more efficient.

@example
>>> a = [5, 2, 3, 1, 4]
>>> a.sort()
>>> a
[1, 2, 3, 4, 5]
@end example

Another difference is that the @ref{705,,list.sort()} method is only defined for
lists. In contrast, the @ref{704,,sorted()} function accepts any iterable.

@example
>>> sorted(@{1: 'D', 2: 'B', 3: 'B', 4: 'E', 5: 'A'@})
[1, 2, 3, 4, 5]
@end example

@node Key Functions,Operator Module Functions,Sorting Basics,Sorting HOW TO
@anchor{howto/sorting key-functions}@anchor{4747}
@subsection Key Functions


Both @ref{705,,list.sort()} and @ref{704,,sorted()} have a @emph{key} parameter to specify a
function (or other callable) to be called on each list element prior to making
comparisons.

For example, here’s a case-insensitive string comparison:

@example
>>> sorted("This is a test string from Andrew".split(), key=str.lower)
['a', 'Andrew', 'from', 'is', 'string', 'test', 'This']
@end example

The value of the @emph{key} parameter should be a function (or other callable) that
takes a single argument and returns a key to use for sorting purposes. This
technique is fast because the key function is called exactly once for each
input record.

A common pattern is to sort complex objects using some of the object’s indices
as keys. For example:

@example
>>> student_tuples = [
...     ('john', 'A', 15),
...     ('jane', 'B', 12),
...     ('dave', 'B', 10),
... ]
>>> sorted(student_tuples, key=lambda student: student[2])   # sort by age
[('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]
@end example

The same technique works for objects with named attributes. For example:

@example
>>> class Student:
...     def __init__(self, name, grade, age):
...         self.name = name
...         self.grade = grade
...         self.age = age
...     def __repr__(self):
...         return repr((self.name, self.grade, self.age))

>>> student_objects = [
...     Student('john', 'A', 15),
...     Student('jane', 'B', 12),
...     Student('dave', 'B', 10),
... ]
>>> sorted(student_objects, key=lambda student: student.age)   # sort by age
[('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]
@end example

@node Operator Module Functions,Ascending and Descending,Key Functions,Sorting HOW TO
@anchor{howto/sorting operator-module-functions}@anchor{4748}
@subsection Operator Module Functions


The key-function patterns shown above are very common, so Python provides
convenience functions to make accessor functions easier and faster. The
@ref{be,,operator} module has @ref{578,,itemgetter()},
@ref{99a,,attrgetter()}, and a @ref{99b,,methodcaller()} function.

Using those functions, the above examples become simpler and faster:

@example
>>> from operator import itemgetter, attrgetter

>>> sorted(student_tuples, key=itemgetter(2))
[('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]

>>> sorted(student_objects, key=attrgetter('age'))
[('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]
@end example

The operator module functions allow multiple levels of sorting. For example, to
sort by @emph{grade} then by @emph{age}:

@example
>>> sorted(student_tuples, key=itemgetter(1,2))
[('john', 'A', 15), ('dave', 'B', 10), ('jane', 'B', 12)]

>>> sorted(student_objects, key=attrgetter('grade', 'age'))
[('john', 'A', 15), ('dave', 'B', 10), ('jane', 'B', 12)]
@end example

@node Ascending and Descending,Sort Stability and Complex Sorts,Operator Module Functions,Sorting HOW TO
@anchor{howto/sorting ascending-and-descending}@anchor{4749}
@subsection Ascending and Descending


Both @ref{705,,list.sort()} and @ref{704,,sorted()} accept a @emph{reverse} parameter with a
boolean value. This is used to flag descending sorts. For example, to get the
student data in reverse @emph{age} order:

@example
>>> sorted(student_tuples, key=itemgetter(2), reverse=True)
[('john', 'A', 15), ('jane', 'B', 12), ('dave', 'B', 10)]

>>> sorted(student_objects, key=attrgetter('age'), reverse=True)
[('john', 'A', 15), ('jane', 'B', 12), ('dave', 'B', 10)]
@end example

@node Sort Stability and Complex Sorts,The Old Way Using Decorate-Sort-Undecorate,Ascending and Descending,Sorting HOW TO
@anchor{howto/sorting sort-stability-and-complex-sorts}@anchor{474a}
@subsection Sort Stability and Complex Sorts


Sorts are guaranteed to be stable@footnote{https://en.wikipedia.org/wiki/Sorting_algorithm#Stability}. That means that
when multiple records have the same key, their original order is preserved.

@example
>>> data = [('red', 1), ('blue', 1), ('red', 2), ('blue', 2)]
>>> sorted(data, key=itemgetter(0))
[('blue', 1), ('blue', 2), ('red', 1), ('red', 2)]
@end example

Notice how the two records for @emph{blue} retain their original order so that
@code{('blue', 1)} is guaranteed to precede @code{('blue', 2)}.

This wonderful property lets you build complex sorts in a series of sorting
steps. For example, to sort the student data by descending @emph{grade} and then
ascending @emph{age}, do the @emph{age} sort first and then sort again using @emph{grade}:

@example
>>> s = sorted(student_objects, key=attrgetter('age'))     # sort on secondary key
>>> sorted(s, key=attrgetter('grade'), reverse=True)       # now sort on primary key, descending
[('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]
@end example

This can be abstracted out into a wrapper function that can take a list and
tuples of field and order to sort them on multiple passes.

@example
>>> def multisort(xs, specs):
...     for key, reverse in reversed(specs):
...         xs.sort(key=attrgetter(key), reverse=reverse)
...     return xs

>>> multisort(list(student_objects), (('grade', True), ('age', False)))
[('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]
@end example

The Timsort@footnote{https://en.wikipedia.org/wiki/Timsort} algorithm used in Python
does multiple sorts efficiently because it can take advantage of any ordering
already present in a dataset.

@node The Old Way Using Decorate-Sort-Undecorate,The Old Way Using the cmp Parameter,Sort Stability and Complex Sorts,Sorting HOW TO
@anchor{howto/sorting the-old-way-using-decorate-sort-undecorate}@anchor{474b}
@subsection The Old Way Using Decorate-Sort-Undecorate


This idiom is called Decorate-Sort-Undecorate after its three steps:


@itemize *

@item 
First, the initial list is decorated with new values that control the sort order.

@item 
Second, the decorated list is sorted.

@item 
Finally, the decorations are removed, creating a list that contains only the
initial values in the new order.
@end itemize

For example, to sort the student data by @emph{grade} using the DSU approach:

@example
>>> decorated = [(student.grade, i, student) for i, student in enumerate(student_objects)]
>>> decorated.sort()
>>> [student for grade, i, student in decorated]               # undecorate
[('john', 'A', 15), ('jane', 'B', 12), ('dave', 'B', 10)]
@end example

This idiom works because tuples are compared lexicographically; the first items
are compared; if they are the same then the second items are compared, and so
on.

It is not strictly necessary in all cases to include the index @emph{i} in the
decorated list, but including it gives two benefits:


@itemize *

@item 
The sort is stable – if two items have the same key, their order will be
preserved in the sorted list.

@item 
The original items do not have to be comparable because the ordering of the
decorated tuples will be determined by at most the first two items. So for
example the original list could contain complex numbers which cannot be sorted
directly.
@end itemize

Another name for this idiom is
Schwartzian transform@footnote{https://en.wikipedia.org/wiki/Schwartzian_transform},
after Randal L. Schwartz, who popularized it among Perl programmers.

Now that Python sorting provides key-functions, this technique is not often needed.

@node The Old Way Using the cmp Parameter,Odd and Ends,The Old Way Using Decorate-Sort-Undecorate,Sorting HOW TO
@anchor{howto/sorting the-old-way-using-the-cmp-parameter}@anchor{474c}
@subsection The Old Way Using the @emph{cmp} Parameter


Many constructs given in this HOWTO assume Python 2.4 or later. Before that,
there was no @ref{704,,sorted()} builtin and @ref{705,,list.sort()} took no keyword
arguments. Instead, all of the Py2.x versions supported a @emph{cmp} parameter to
handle user specified comparison functions.

In Py3.0, the @emph{cmp} parameter was removed entirely (as part of a larger effort to
simplify and unify the language, eliminating the conflict between rich
comparisons and the @code{__cmp__()} magic method).

In Py2.x, sort allowed an optional function which can be called for doing the
comparisons. That function should take two arguments to be compared and then
return a negative value for less-than, return zero if they are equal, or return
a positive value for greater-than. For example, we can do:

@example
>>> def numeric_compare(x, y):
...     return x - y
>>> sorted([5, 2, 4, 1, 3], cmp=numeric_compare) 
[1, 2, 3, 4, 5]
@end example

Or you can reverse the order of comparison with:

@example
>>> def reverse_numeric(x, y):
...     return y - x
>>> sorted([5, 2, 4, 1, 3], cmp=reverse_numeric) 
[5, 4, 3, 2, 1]
@end example

When porting code from Python 2.x to 3.x, the situation can arise when you have
the user supplying a comparison function and you need to convert that to a key
function. The following wrapper makes that easy to do:

@example
def cmp_to_key(mycmp):
    'Convert a cmp= function into a key= function'
    class K:
        def __init__(self, obj, *args):
            self.obj = obj
        def __lt__(self, other):
            return mycmp(self.obj, other.obj) < 0
        def __gt__(self, other):
            return mycmp(self.obj, other.obj) > 0
        def __eq__(self, other):
            return mycmp(self.obj, other.obj) == 0
        def __le__(self, other):
            return mycmp(self.obj, other.obj) <= 0
        def __ge__(self, other):
            return mycmp(self.obj, other.obj) >= 0
        def __ne__(self, other):
            return mycmp(self.obj, other.obj) != 0
    return K
@end example

@c >>> sorted([5, 2, 4, 1, 3], key=cmp_to_key(reverse_numeric))
@c [5, 4, 3, 2, 1]

To convert to a key function, just wrap the old comparison function:

@c from functools import cmp_to_key

@example
>>> sorted([5, 2, 4, 1, 3], key=cmp_to_key(reverse_numeric))
[5, 4, 3, 2, 1]
@end example

In Python 3.2, the @ref{d7c,,functools.cmp_to_key()} function was added to the
@ref{7f,,functools} module in the standard library.

@node Odd and Ends,,The Old Way Using the cmp Parameter,Sorting HOW TO
@anchor{howto/sorting odd-and-ends}@anchor{474d}
@subsection Odd and Ends



@itemize *

@item 
For locale aware sorting, use @ref{3507,,locale.strxfrm()} for a key function or
@ref{3505,,locale.strcoll()} for a comparison function.

@item 
The @emph{reverse} parameter still maintains sort stability (so that records with
equal keys retain the original order). Interestingly, that effect can be
simulated without the parameter by using the builtin @ref{2f6,,reversed()} function
twice:

@example
>>> data = [('red', 1), ('blue', 1), ('red', 2), ('blue', 2)]
>>> standard_way = sorted(data, key=itemgetter(0), reverse=True)
>>> double_reversed = list(reversed(sorted(reversed(data), key=itemgetter(0))))
>>> assert standard_way == double_reversed
>>> standard_way
[('red', 1), ('red', 2), ('blue', 1), ('blue', 2)]
@end example

@item 
The sort routines are guaranteed to use @code{__lt__()} when making comparisons
between two objects. So, it is easy to add a standard sort order to a class by
defining an @code{__lt__()} method:

@example
>>> Student.__lt__ = lambda self, other: self.age < other.age
>>> sorted(student_objects)
[('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)]
@end example

@item 
Key functions need not depend directly on the objects being sorted. A key
function can also access external resources. For instance, if the student grades
are stored in a dictionary, they can be used to sort a separate list of student
names:

@example
>>> students = ['dave', 'john', 'jane']
>>> newgrades = @{'john': 'F', 'jane':'A', 'dave': 'C'@}
>>> sorted(students, key=newgrades.__getitem__)
['jane', 'dave', 'john']
@end example
@end itemize

@node Unicode HOWTO,HOWTO Fetch Internet Resources Using The urllib Package,Sorting HOW TO,Python HOWTOs
@anchor{howto/unicode doc}@anchor{474e}@anchor{howto/unicode id1}@anchor{474f}@anchor{howto/unicode unicode-howto}@anchor{e4a}
@section Unicode HOWTO


@*Release: 
1.12


This HOWTO discusses Python’s support for the Unicode specification
for representing textual data, and explains various problems that
people commonly encounter when trying to work with Unicode.

@menu
* Introduction to Unicode:: 
* Python’s Unicode Support:: 
* Reading and Writing Unicode Data:: 
* Acknowledgements: Acknowledgements<10>. 

@end menu

@node Introduction to Unicode,Python’s Unicode Support,,Unicode HOWTO
@anchor{howto/unicode introduction-to-unicode}@anchor{4750}
@subsection Introduction to Unicode


@menu
* Definitions:: 
* Encodings:: 
* References: References<3>. 

@end menu

@node Definitions,Encodings,,Introduction to Unicode
@anchor{howto/unicode definitions}@anchor{4751}
@subsubsection Definitions


Today’s programs need to be able to handle a wide variety of
characters.  Applications are often internationalized to display
messages and output in a variety of user-selectable languages; the
same program might need to output an error message in English, French,
Japanese, Hebrew, or Russian.  Web content can be written in any of
these languages and can also include a variety of emoji symbols.
Python’s string type uses the Unicode Standard for representing
characters, which lets Python programs work with all these different
possible characters.

Unicode (@indicateurl{https://www.unicode.org/}) is a specification that aims to
list every character used by human languages and give each character
its own unique code.  The Unicode specifications are continually
revised and updated to add new languages and symbols.

A @strong{character} is the smallest possible component of a text.  ‘A’, ‘B’, ‘C’,
etc., are all different characters.  So are ‘È’ and ‘Í’.  Characters vary
depending on the language or context you’re talking
about.  For example, there’s a character for “Roman Numeral One”, ‘Ⅰ’, that’s
separate from the uppercase letter ‘I’.  They’ll usually look the same,
but these are two different characters that have different meanings.

The Unicode standard describes how characters are represented by
@strong{code points}.  A code point value is an integer in the range 0 to
0x10FFFF (about 1.1 million values, the
actual number assigned@footnote{https://www.unicode.org/versions/latest/#Summary}
is less than that). In the standard and in this document, a code point is written
using the notation @code{U+265E} to mean the character with value
@code{0x265e} (9,822 in decimal).

The Unicode standard contains a lot of tables listing characters and
their corresponding code points:

@example
0061    'a'; LATIN SMALL LETTER A
0062    'b'; LATIN SMALL LETTER B
0063    'c'; LATIN SMALL LETTER C
...
007B    '@{'; LEFT CURLY BRACKET
...
2167    'Ⅷ'; ROMAN NUMERAL EIGHT
2168    'Ⅸ'; ROMAN NUMERAL NINE
...
265E    '♞'; BLACK CHESS KNIGHT
265F    '♟'; BLACK CHESS PAWN
...
1F600   '😀'; GRINNING FACE
1F609   '😉'; WINKING FACE
...
@end example

Strictly, these definitions imply that it’s meaningless to say ‘this is
character @code{U+265E}’.  @code{U+265E} is a code point, which represents some particular
character; in this case, it represents the character ‘BLACK CHESS KNIGHT’,
‘♞’.  In
informal contexts, this distinction between code points and characters will
sometimes be forgotten.

A character is represented on a screen or on paper by a set of graphical
elements that’s called a @strong{glyph}.  The glyph for an uppercase A, for example,
is two diagonal strokes and a horizontal stroke, though the exact details will
depend on the font being used.  Most Python code doesn’t need to worry about
glyphs; figuring out the correct glyph to display is generally the job of a GUI
toolkit or a terminal’s font renderer.

@node Encodings,References<3>,Definitions,Introduction to Unicode
@anchor{howto/unicode encodings}@anchor{4752}
@subsubsection Encodings


To summarize the previous section: a Unicode string is a sequence of
code points, which are numbers from 0 through @code{0x10FFFF} (1,114,111
decimal).  This sequence of code points needs to be represented in
memory as a set of @strong{code units}, and @strong{code units} are then mapped
to 8-bit bytes.  The rules for translating a Unicode string into a
sequence of bytes are called a @strong{character encoding}, or just
an @strong{encoding}.

The first encoding you might think of is using 32-bit integers as the
code unit, and then using the CPU’s representation of 32-bit integers.
In this representation, the string “Python” might look like this:

@example
   P           y           t           h           o           n
0x50 00 00 00 79 00 00 00 74 00 00 00 68 00 00 00 6f 00 00 00 6e 00 00 00
   0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
@end example

This representation is straightforward but using it presents a number of
problems.


@enumerate 

@item 
It’s not portable; different processors order the bytes differently.

@item 
It’s very wasteful of space.  In most texts, the majority of the code points
are less than 127, or less than 255, so a lot of space is occupied by @code{0x00}
bytes.  The above string takes 24 bytes compared to the 6 bytes needed for an
ASCII representation.  Increased RAM usage doesn’t matter too much (desktop
computers have gigabytes of RAM, and strings aren’t usually that large), but
expanding our usage of disk and network bandwidth by a factor of 4 is
intolerable.

@item 
It’s not compatible with existing C functions such as @code{strlen()}, so a new
family of wide string functions would need to be used.
@end enumerate

Therefore this encoding isn’t used very much, and people instead choose other
encodings that are more efficient and convenient, such as UTF-8.

UTF-8 is one of the most commonly used encodings, and Python often
defaults to using it.  UTF stands for “Unicode Transformation Format”,
and the ‘8’ means that 8-bit values are used in the encoding.  (There
are also UTF-16 and UTF-32 encodings, but they are less frequently
used than UTF-8.)  UTF-8 uses the following rules:


@enumerate 

@item 
If the code point is < 128, it’s represented by the corresponding byte value.

@item 
If the code point is >= 128, it’s turned into a sequence of two, three, or
four bytes, where each byte of the sequence is between 128 and 255.
@end enumerate

UTF-8 has several convenient properties:


@enumerate 

@item 
It can handle any Unicode code point.

@item 
A Unicode string is turned into a sequence of bytes that contains embedded
zero bytes only where they represent the null character (U+0000). This means
that UTF-8 strings can be processed by C functions such as @code{strcpy()} and sent
through protocols that can’t handle zero bytes for anything other than
end-of-string markers.

@item 
A string of ASCII text is also valid UTF-8 text.

@item 
UTF-8 is fairly compact; the majority of commonly used characters can be
represented with one or two bytes.

@item 
If bytes are corrupted or lost, it’s possible to determine the start of the
next UTF-8-encoded code point and resynchronize.  It’s also unlikely that
random 8-bit data will look like valid UTF-8.

@item 
UTF-8 is a byte oriented encoding. The encoding specifies that each
character is represented by a specific sequence of one or more bytes. This
avoids the byte-ordering issues that can occur with integer and word oriented
encodings, like UTF-16 and UTF-32, where the sequence of bytes varies depending
on the hardware on which the string was encoded.
@end enumerate

@node References<3>,,Encodings,Introduction to Unicode
@anchor{howto/unicode references}@anchor{4753}
@subsubsection References


The Unicode Consortium site@footnote{https://www.unicode.org} has character charts, a
glossary, and PDF versions of the Unicode specification.  Be prepared for some
difficult reading.  A chronology@footnote{https://www.unicode.org/history/} of the
origin and development of Unicode is also available on the site.

On the Computerphile Youtube channel, Tom Scott briefly
discusses the history of Unicode and UTF-8@footnote{https://www.youtube.com/watch?v=MijmeoH9LT4}
(9 minutes 36 seconds).

To help understand the standard, Jukka Korpela has written an introductory guide@footnote{http://jkorpela.fi/unicode/guide.html} to reading the
Unicode character tables.

Another good introductory article@footnote{https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/}
was written by Joel Spolsky.
If this introduction didn’t make things clear to you, you should try
reading this alternate article before continuing.

Wikipedia entries are often helpful; see the entries for “character encoding@footnote{https://en.wikipedia.org/wiki/Character_encoding}” and UTF-8@footnote{https://en.wikipedia.org/wiki/UTF-8}, for example.

@node Python’s Unicode Support,Reading and Writing Unicode Data,Introduction to Unicode,Unicode HOWTO
@anchor{howto/unicode python-s-unicode-support}@anchor{4754}
@subsection Python’s Unicode Support


Now that you’ve learned the rudiments of Unicode, we can look at Python’s
Unicode features.

@menu
* The String Type:: 
* Converting to Bytes:: 
* Unicode Literals in Python Source Code:: 
* Unicode Properties:: 
* Comparing Strings:: 
* Unicode Regular Expressions:: 
* References: References<4>. 

@end menu

@node The String Type,Converting to Bytes,,Python’s Unicode Support
@anchor{howto/unicode the-string-type}@anchor{4755}
@subsubsection The String Type


Since Python 3.0, the language’s @ref{2ef,,str} type contains Unicode
characters, meaning any string created using @code{"unicode rocks!"}, @code{'unicode
rocks!'}, or the triple-quoted string syntax is stored as Unicode.

The default encoding for Python source code is UTF-8, so you can simply
include a Unicode character in a string literal:

@example
try:
    with open('/tmp/input.txt', 'r') as f:
        ...
except OSError:
    # 'File not found' error message.
    print("Fichier non trouvé")
@end example

Side note: Python 3 also supports using Unicode characters in identifiers:

@example
répertoire = "/tmp/records.log"
with open(répertoire, "w") as f:
    f.write("test\n")
@end example

If you can’t enter a particular character in your editor or want to
keep the source code ASCII-only for some reason, you can also use
escape sequences in string literals. (Depending on your system,
you may see the actual capital-delta glyph instead of a u escape.)

@example
>>> "\N@{GREEK CAPITAL LETTER DELTA@}"  # Using the character name
'\u0394'
>>> "\u0394"                          # Using a 16-bit hex value
'\u0394'
>>> "\U00000394"                      # Using a 32-bit hex value
'\u0394'
@end example

In addition, one can create a string using the @ref{3bb,,decode()} method of
@ref{2f0,,bytes}.  This method takes an @emph{encoding} argument, such as @code{UTF-8},
and optionally an @emph{errors} argument.

The @emph{errors} argument specifies the response when the input string can’t be
converted according to the encoding’s rules.  Legal values for this argument are
@code{'strict'} (raise a @ref{51f,,UnicodeDecodeError} exception), @code{'replace'} (use
@code{U+FFFD}, @code{REPLACEMENT CHARACTER}), @code{'ignore'} (just leave the
character out of the Unicode result), or @code{'backslashreplace'} (inserts a
@code{\xNN} escape sequence).
The following examples show the differences:

@example
>>> b'\x80abc'.decode("utf-8", "strict")  
Traceback (most recent call last):
    ...
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0:
  invalid start byte
>>> b'\x80abc'.decode("utf-8", "replace")
'\ufffdabc'
>>> b'\x80abc'.decode("utf-8", "backslashreplace")
'\\x80abc'
>>> b'\x80abc'.decode("utf-8", "ignore")
'abc'
@end example

Encodings are specified as strings containing the encoding’s name.  Python
comes with roughly 100 different encodings; see the Python Library Reference at
@ref{916,,Standard Encodings} for a list.  Some encodings have multiple names; for
example, @code{'latin-1'}, @code{'iso_8859_1'} and @code{'8859}’ are all synonyms for
the same encoding.

One-character Unicode strings can also be created with the @ref{178f,,chr()}
built-in function, which takes integers and returns a Unicode string of length 1
that contains the corresponding code point.  The reverse operation is the
built-in @ref{178e,,ord()} function that takes a one-character Unicode string and
returns the code point value:

@example
>>> chr(57344)
'\ue000'
>>> ord('\ue000')
57344
@end example

@node Converting to Bytes,Unicode Literals in Python Source Code,The String Type,Python’s Unicode Support
@anchor{howto/unicode converting-to-bytes}@anchor{4756}
@subsubsection Converting to Bytes


The opposite method of @ref{3bb,,bytes.decode()} is @ref{3ba,,str.encode()},
which returns a @ref{2f0,,bytes} representation of the Unicode string, encoded in the
requested @emph{encoding}.

The @emph{errors} parameter is the same as the parameter of the
@ref{3bb,,decode()} method but supports a few more possible handlers. As well as
@code{'strict'}, @code{'ignore'}, and @code{'replace'} (which in this case
inserts a question mark instead of the unencodable character), there is
also @code{'xmlcharrefreplace'} (inserts an XML character reference),
@code{backslashreplace} (inserts a @code{\uNNNN} escape sequence) and
@code{namereplace} (inserts a @code{\N@{...@}} escape sequence).

The following example shows the different results:

@example
>>> u = chr(40960) + 'abcd' + chr(1972)
>>> u.encode('utf-8')
b'\xea\x80\x80abcd\xde\xb4'
>>> u.encode('ascii')  
Traceback (most recent call last):
    ...
UnicodeEncodeError: 'ascii' codec can't encode character '\ua000' in
  position 0: ordinal not in range(128)
>>> u.encode('ascii', 'ignore')
b'abcd'
>>> u.encode('ascii', 'replace')
b'?abcd?'
>>> u.encode('ascii', 'xmlcharrefreplace')
b'&#40960;abcd&#1972;'
>>> u.encode('ascii', 'backslashreplace')
b'\\ua000abcd\\u07b4'
>>> u.encode('ascii', 'namereplace')
b'\\N@{YI SYLLABLE IT@}abcd\\u07b4'
@end example

The low-level routines for registering and accessing the available
encodings are found in the @ref{1a,,codecs} module.  Implementing new
encodings also requires understanding the @ref{1a,,codecs} module.
However, the encoding and decoding functions returned by this module
are usually more low-level than is comfortable, and writing new encodings
is a specialized task, so the module won’t be covered in this HOWTO.

@node Unicode Literals in Python Source Code,Unicode Properties,Converting to Bytes,Python’s Unicode Support
@anchor{howto/unicode unicode-literals-in-python-source-code}@anchor{4757}
@subsubsection Unicode Literals in Python Source Code


In Python source code, specific Unicode code points can be written using the
@code{\u} escape sequence, which is followed by four hex digits giving the code
point.  The @code{\U} escape sequence is similar, but expects eight hex digits,
not four:

@example
>>> s = "a\xac\u1234\u20ac\U00008000"
... #     ^^^^ two-digit hex escape
... #         ^^^^^^ four-digit Unicode escape
... #                     ^^^^^^^^^^ eight-digit Unicode escape
>>> [ord(c) for c in s]
[97, 172, 4660, 8364, 32768]
@end example

Using escape sequences for code points greater than 127 is fine in small doses,
but becomes an annoyance if you’re using many accented characters, as you would
in a program with messages in French or some other accent-using language.  You
can also assemble strings using the @ref{178f,,chr()} built-in function, but this is
even more tedious.

Ideally, you’d want to be able to write literals in your language’s natural
encoding.  You could then edit Python source code with your favorite editor
which would display the accented characters naturally, and have the right
characters used at runtime.

Python supports writing source code in UTF-8 by default, but you can use almost
any encoding if you declare the encoding being used.  This is done by including
a special comment as either the first or second line of the source file:

@example
#!/usr/bin/env python
# -*- coding: latin-1 -*-

u = 'abcdé'
print(ord(u[-1]))
@end example

The syntax is inspired by Emacs’s notation for specifying variables local to a
file.  Emacs supports many different variables, but Python only supports
‘coding’.  The @code{-*-} symbols indicate to Emacs that the comment is special;
they have no significance to Python but are a convention.  Python looks for
@code{coding: name} or @code{coding=name} in the comment.

If you don’t include such a comment, the default encoding used will be UTF-8 as
already mentioned.  See also 
@geindex Python Enhancement Proposals; PEP 263
PEP 263@footnote{https://www.python.org/dev/peps/pep-0263} for more information.

@node Unicode Properties,Comparing Strings,Unicode Literals in Python Source Code,Python’s Unicode Support
@anchor{howto/unicode unicode-properties}@anchor{4758}
@subsubsection Unicode Properties


The Unicode specification includes a database of information about
code points.  For each defined code point, the information includes
the character’s name, its category, the numeric value if applicable
(for characters representing numeric concepts such as the Roman
numerals, fractions such as one-third and four-fifths, etc.).  There
are also display-related properties, such as how to use the code point
in bidirectional text.

The following program displays some information about several characters, and
prints the numeric value of one particular character:

@example
import unicodedata

u = chr(233) + chr(0x0bf2) + chr(3972) + chr(6000) + chr(13231)

for i, c in enumerate(u):
    print(i, '%04x' % ord(c), unicodedata.category(c), end=" ")
    print(unicodedata.name(c))

# Get numeric value of second character
print(unicodedata.numeric(u[1]))
@end example

When run, this prints:

@example
0 00e9 Ll LATIN SMALL LETTER E WITH ACUTE
1 0bf2 No TAMIL NUMBER ONE THOUSAND
2 0f84 Mn TIBETAN MARK HALANTA
3 1770 Lo TAGBANWA LETTER SA
4 33af So SQUARE RAD OVER S SQUARED
1000.0
@end example

The category codes are abbreviations describing the nature of the character.
These are grouped into categories such as “Letter”, “Number”, “Punctuation”, or
“Symbol”, which in turn are broken up into subcategories.  To take the codes
from the above output, @code{'Ll'} means ‘Letter, lowercase’, @code{'No'} means
“Number, other”, @code{'Mn'} is “Mark, nonspacing”, and @code{'So'} is “Symbol,
other”.  See
the General Category Values section of the Unicode Character Database documentation@footnote{https://www.unicode.org/reports/tr44/#General_Category_Values} for a
list of category codes.

@node Comparing Strings,Unicode Regular Expressions,Unicode Properties,Python’s Unicode Support
@anchor{howto/unicode comparing-strings}@anchor{4759}
@subsubsection Comparing Strings


Unicode adds some complication to comparing strings, because the same
set of characters can be represented by different sequences of code
points.  For example, a letter like ‘ê’ can be represented as a single
code point U+00EA, or as U+0065 U+0302, which is the code point for
‘e’ followed by a code point for ‘COMBINING CIRCUMFLEX ACCENT’.  These
will produce the same output when printed, but one is a string of
length 1 and the other is of length 2.

One tool for a case-insensitive comparison is the
@ref{935,,casefold()} string method that converts a string to a
case-insensitive form following an algorithm described by the Unicode
Standard.  This algorithm has special handling for characters such as
the German letter ‘ß’ (code point U+00DF), which becomes the pair of
lowercase letters ‘ss’.

@example
>>> street = 'Gürzenichstraße'
>>> street.casefold()
'gürzenichstrasse'
@end example

A second tool is the @ref{121,,unicodedata} module’s
@ref{19a,,normalize()} function that converts strings to one
of several normal forms, where letters followed by a combining
character are replaced with single characters.  @code{normalize()} can
be used to perform string comparisons that won’t falsely report
inequality if two strings use combining characters differently:

@example
import unicodedata

def compare_strs(s1, s2):
    def NFD(s):
        return unicodedata.normalize('NFD', s)

    return NFD(s1) == NFD(s2)

single_char = 'ê'
multiple_chars = '\N@{LATIN SMALL LETTER E@}\N@{COMBINING CIRCUMFLEX ACCENT@}'
print('length of first string=', len(single_char))
print('length of second string=', len(multiple_chars))
print(compare_strs(single_char, multiple_chars))
@end example

When run, this outputs:

@example
$ python3 compare-strs.py
length of first string= 1
length of second string= 2
True
@end example

The first argument to the @ref{19a,,normalize()} function is a
string giving the desired normalization form, which can be one of
‘NFC’, ‘NFKC’, ‘NFD’, and ‘NFKD’.

The Unicode Standard also specifies how to do caseless comparisons:

@example
import unicodedata

def compare_caseless(s1, s2):
    def NFD(s):
        return unicodedata.normalize('NFD', s)

    return NFD(NFD(s1).casefold()) == NFD(NFD(s2).casefold())

# Example usage
single_char = 'ê'
multiple_chars = '\N@{LATIN CAPITAL LETTER E@}\N@{COMBINING CIRCUMFLEX ACCENT@}'

print(compare_caseless(single_char, multiple_chars))
@end example

This will print @code{True}.  (Why is @code{NFD()} invoked twice?  Because
there are a few characters that make @code{casefold()} return a
non-normalized string, so the result needs to be normalized again. See
section 3.13 of the Unicode Standard for a discussion and an example.)

@node Unicode Regular Expressions,References<4>,Comparing Strings,Python’s Unicode Support
@anchor{howto/unicode unicode-regular-expressions}@anchor{475a}
@subsubsection Unicode Regular Expressions


The regular expressions supported by the @ref{d8,,re} module can be provided
either as bytes or strings.  Some of the special character sequences such as
@code{\d} and @code{\w} have different meanings depending on whether
the pattern is supplied as bytes or a string.  For example,
@code{\d} will match the characters @code{[0-9]} in bytes but
in strings will match any character that’s in the @code{'Nd'} category.

The string in this example has the number 57 written in both Thai and
Arabic numerals:

@example
import re
p = re.compile(r'\d+')

s = "Over \u0e55\u0e57 57 flavours"
m = p.search(s)
print(repr(m.group()))
@end example

When executed, @code{\d+} will match the Thai numerals and print them
out.  If you supply the @ref{697,,re.ASCII} flag to
@ref{70b,,compile()}, @code{\d+} will match the substring “57” instead.

Similarly, @code{\w} matches a wide variety of Unicode characters but
only @code{[a-zA-Z0-9_]} in bytes or if @ref{697,,re.ASCII} is supplied,
and @code{\s} will match either Unicode whitespace characters or
@code{[ \t\n\r\f\v]}.

@node References<4>,,Unicode Regular Expressions,Python’s Unicode Support
@anchor{howto/unicode id2}@anchor{475b}
@subsubsection References


@c comment should these be mentioned earlier, e.g. at the start of the "introduction to Unicode" first section?

Some good alternative discussions of Python’s Unicode support are:


@itemize *

@item 
Processing Text Files in Python 3@footnote{http://python-notes.curiousefficiency.org/en/latest/python3/text_file_processing.html}, by Nick Coghlan.

@item 
Pragmatic Unicode@footnote{https://nedbatchelder.com/text/unipain.html}, a PyCon 2012 presentation by Ned Batchelder.
@end itemize

The @ref{2ef,,str} type is described in the Python library reference at
@ref{14e2,,Text Sequence Type — str}.

The documentation for the @ref{121,,unicodedata} module.

The documentation for the @ref{1a,,codecs} module.

Marc-André Lemburg gave a presentation titled "Python and Unicode" (PDF slides)@footnote{https://downloads.egenix.com/python/Unicode-EPC2002-Talk.pdf} at
EuroPython 2002.  The slides are an excellent overview of the design of Python
2’s Unicode features (where the Unicode string type is called @code{unicode} and
literals start with @code{u}).

@node Reading and Writing Unicode Data,Acknowledgements<10>,Python’s Unicode Support,Unicode HOWTO
@anchor{howto/unicode reading-and-writing-unicode-data}@anchor{475c}
@subsection Reading and Writing Unicode Data


Once you’ve written some code that works with Unicode data, the next problem is
input/output.  How do you get Unicode strings into your program, and how do you
convert Unicode into a form suitable for storage or transmission?

It’s possible that you may not need to do anything depending on your input
sources and output destinations; you should check whether the libraries used in
your application support Unicode natively.  XML parsers often return Unicode
data, for example.  Many relational databases also support Unicode-valued
columns and can return Unicode values from an SQL query.

Unicode data is usually converted to a particular encoding before it gets
written to disk or sent over a socket.  It’s possible to do all the work
yourself: open a file, read an 8-bit bytes object from it, and convert the bytes
with @code{bytes.decode(encoding)}.  However, the manual approach is not recommended.

One problem is the multi-byte nature of encodings; one Unicode character can be
represented by several bytes.  If you want to read the file in arbitrary-sized
chunks (say, 1024 or 4096 bytes), you need to write error-handling code to catch the case
where only part of the bytes encoding a single Unicode character are read at the
end of a chunk.  One solution would be to read the entire file into memory and
then perform the decoding, but that prevents you from working with files that
are extremely large; if you need to read a 2 GiB file, you need 2 GiB of RAM.
(More, really, since for at least a moment you’d need to have both the encoded
string and its Unicode version in memory.)

The solution would be to use the low-level decoding interface to catch the case
of partial coding sequences.  The work of implementing this has already been
done for you: the built-in @ref{1c8,,open()} function can return a file-like object
that assumes the file’s contents are in a specified encoding and accepts Unicode
parameters for methods such as @ref{24a6,,read()} and
@ref{24a9,,write()}.  This works through @ref{1c8,,open()}'s @emph{encoding} and
@emph{errors} parameters which are interpreted just like those in @ref{3ba,,str.encode()}
and @ref{3bb,,bytes.decode()}.

Reading Unicode from a file is therefore simple:

@example
with open('unicode.txt', encoding='utf-8') as f:
    for line in f:
        print(repr(line))
@end example

It’s also possible to open files in update mode, allowing both reading and
writing:

@example
with open('test', encoding='utf-8', mode='w+') as f:
    f.write('\u4500 blah blah blah\n')
    f.seek(0)
    print(repr(f.readline()[:1]))
@end example

The Unicode character @code{U+FEFF} is used as a byte-order mark (BOM), and is often
written as the first character of a file in order to assist with autodetection
of the file’s byte ordering.  Some encodings, such as UTF-16, expect a BOM to be
present at the start of a file; when such an encoding is used, the BOM will be
automatically written as the first character and will be silently dropped when
the file is read.  There are variants of these encodings, such as ‘utf-16-le’
and ‘utf-16-be’ for little-endian and big-endian encodings, that specify one
particular byte ordering and don’t skip the BOM.

In some areas, it is also convention to use a “BOM” at the start of UTF-8
encoded files; the name is misleading since UTF-8 is not byte-order dependent.
The mark simply announces that the file is encoded in UTF-8.  For reading such
files, use the ‘utf-8-sig’ codec to automatically skip the mark if present.

@menu
* Unicode filenames:: 
* Tips for Writing Unicode-aware Programs:: 
* References: References<5>. 

@end menu

@node Unicode filenames,Tips for Writing Unicode-aware Programs,,Reading and Writing Unicode Data
@anchor{howto/unicode unicode-filenames}@anchor{475d}
@subsubsection Unicode filenames


Most of the operating systems in common use today support filenames
that contain arbitrary Unicode characters.  Usually this is
implemented by converting the Unicode string into some encoding that
varies depending on the system.  Today Python is converging on using
UTF-8: Python on MacOS has used UTF-8 for several versions, and Python
3.6 switched to using UTF-8 on Windows as well.  On Unix systems,
there will only be a @ref{1628,,filesystem encoding}. if you’ve set the @code{LANG} or @code{LC_CTYPE} environment variables; if
you haven’t, the default encoding is again UTF-8.

The @ref{79e,,sys.getfilesystemencoding()} function returns the encoding to use on
your current system, in case you want to do the encoding manually, but there’s
not much reason to bother.  When opening a file for reading or writing, you can
usually just provide the Unicode string as the filename, and it will be
automatically converted to the right encoding for you:

@example
filename = 'filename\u4500abc'
with open(filename, 'w') as f:
    f.write('blah\n')
@end example

Functions in the @ref{c0,,os} module such as @ref{516,,os.stat()} will also accept Unicode
filenames.

The @ref{c8d,,os.listdir()} function returns filenames, which raises an issue: should it return
the Unicode version of filenames, or should it return bytes containing
the encoded versions?  @ref{c8d,,os.listdir()} can do both, depending on whether you
provided the directory path as bytes or a Unicode string.  If you pass a
Unicode string as the path, filenames will be decoded using the filesystem’s
encoding and a list of Unicode strings will be returned, while passing a byte
path will return the filenames as bytes.  For example,
assuming the default @ref{1628,,filesystem encoding} is UTF-8, running the following program:

@example
fn = 'filename\u4500abc'
f = open(fn, 'w')
f.close()

import os
print(os.listdir(b'.'))
print(os.listdir('.'))
@end example

will produce the following output:

@example
$ python listdir-test.py
[b'filename\xe4\x94\x80abc', ...]
['filename\u4500abc', ...]
@end example

The first list contains UTF-8-encoded filenames, and the second list contains
the Unicode versions.

Note that on most occasions, you should can just stick with using
Unicode with these APIs.  The bytes APIs should only be used on
systems where undecodable file names can be present; that’s
pretty much only Unix systems now.

@node Tips for Writing Unicode-aware Programs,References<5>,Unicode filenames,Reading and Writing Unicode Data
@anchor{howto/unicode tips-for-writing-unicode-aware-programs}@anchor{475e}
@subsubsection Tips for Writing Unicode-aware Programs


This section provides some suggestions on writing software that deals with
Unicode.

The most important tip is:

@quotation

Software should only work with Unicode strings internally, decoding the input
data as soon as possible and encoding the output only at the end.
@end quotation

If you attempt to write processing functions that accept both Unicode and byte
strings, you will find your program vulnerable to bugs wherever you combine the
two different kinds of strings.  There is no automatic encoding or decoding: if
you do e.g. @code{str + bytes}, a @ref{162,,TypeError} will be raised.

When using data coming from a web browser or some other untrusted source, a
common technique is to check for illegal characters in a string before using the
string in a generated command line or storing it in a database.  If you’re doing
this, be careful to check the decoded string, not the encoded bytes data;
some encodings may have interesting properties, such as not being bijective
or not being fully ASCII-compatible.  This is especially true if the input
data also specifies the encoding, since the attacker can then choose a
clever way to hide malicious text in the encoded bytestream.

@menu
* Converting Between File Encodings:: 
* Files in an Unknown Encoding:: 

@end menu

@node Converting Between File Encodings,Files in an Unknown Encoding,,Tips for Writing Unicode-aware Programs
@anchor{howto/unicode converting-between-file-encodings}@anchor{475f}
@subsubsection Converting Between File Encodings


The @ref{129b,,StreamRecoder} class can transparently convert between
encodings, taking a stream that returns data in encoding #1
and behaving like a stream returning data in encoding #2.

For example, if you have an input file @emph{f} that’s in Latin-1, you
can wrap it with a @ref{129b,,StreamRecoder} to return bytes encoded in
UTF-8:

@example
new_f = codecs.StreamRecoder(f,
    # en/decoder: used by read() to encode its results and
    # by write() to decode its input.
    codecs.getencoder('utf-8'), codecs.getdecoder('utf-8'),

    # reader/writer: used to read and write to the stream.
    codecs.getreader('latin-1'), codecs.getwriter('latin-1') )
@end example

@node Files in an Unknown Encoding,,Converting Between File Encodings,Tips for Writing Unicode-aware Programs
@anchor{howto/unicode files-in-an-unknown-encoding}@anchor{4760}
@subsubsection Files in an Unknown Encoding


What can you do if you need to make a change to a file, but don’t know
the file’s encoding?  If you know the encoding is ASCII-compatible and
only want to examine or modify the ASCII parts, you can open the file
with the @code{surrogateescape} error handler:

@example
with open(fname, 'r', encoding="ascii", errors="surrogateescape") as f:
    data = f.read()

# make changes to the string 'data'

with open(fname + '.new', 'w',
          encoding="ascii", errors="surrogateescape") as f:
    f.write(data)
@end example

The @code{surrogateescape} error handler will decode any non-ASCII bytes
as code points in a special range running from U+DC80 to
U+DCFF.  These code points will then turn back into the
same bytes when the @code{surrogateescape} error handler is used to
encode the data and write it back out.

@node References<5>,,Tips for Writing Unicode-aware Programs,Reading and Writing Unicode Data
@anchor{howto/unicode id3}@anchor{4761}
@subsubsection References


One section of Mastering Python 3 Input/Output@footnote{http://pyvideo.org/video/289/pycon-2010--mastering-python-3-i-o},
a PyCon 2010 talk by David Beazley, discusses text processing and binary data handling.

The PDF slides for Marc-André Lemburg's presentation "Writing Unicode-aware Applications in Python"@footnote{https://downloads.egenix.com/python/LSM2005-Developing-Unicode-aware-applications-in-Python.pdf}
discuss questions of character encodings as well as how to internationalize
and localize an application.  These slides cover Python 2.x only.

The Guts of Unicode in Python@footnote{http://pyvideo.org/video/1768/the-guts-of-unicode-in-python}
is a PyCon 2013 talk by Benjamin Peterson that discusses the internal Unicode
representation in Python 3.3.

@node Acknowledgements<10>,,Reading and Writing Unicode Data,Unicode HOWTO
@anchor{howto/unicode acknowledgements}@anchor{4762}
@subsection Acknowledgements


The initial draft of this document was written by Andrew Kuchling.
It has since been revised further by Alexander Belopolsky, Georg Brandl,
Andrew Kuchling, and Ezio Melotti.

Thanks to the following people who have noted errors or offered
suggestions on this article: Éric Araujo, Nicholas Bastin, Nick
Coghlan, Marius Gedminas, Kent Johnson, Ken Krugler, Marc-André
Lemburg, Martin von Löwis, Terry J. Reedy, Serhiy Storchaka,
Eryk Sun, Chad Whitacre, Graham Wideman.

@node HOWTO Fetch Internet Resources Using The urllib Package,Argparse Tutorial,Unicode HOWTO,Python HOWTOs
@anchor{howto/urllib2 doc}@anchor{4763}@anchor{howto/urllib2 howto-fetch-internet-resources-using-the-urllib-package}@anchor{4764}@anchor{howto/urllib2 urllib-howto}@anchor{3134}
@section HOWTO Fetch Internet Resources Using The urllib Package


@*Author: 
Michael Foord@footnote{http://www.voidspace.org.uk/python/index.shtml}


@cartouche
@quotation Note 
There is a French translation of an earlier revision of this
HOWTO, available at urllib2 - Le Manuel manquant@footnote{http://www.voidspace.org.uk/python/articles/urllib2_francais.shtml}.
@end quotation
@end cartouche

@menu
* Introduction: Introduction<16>. 
* Fetching URLs:: 
* Handling Exceptions: Handling Exceptions<2>. 
* info and geturl:: 
* Openers and Handlers:: 
* Basic Authentication:: 
* Proxies:: 
* Sockets and Layers:: 
* Footnotes:: 

@end menu

@node Introduction<16>,Fetching URLs,,HOWTO Fetch Internet Resources Using The urllib Package
@anchor{howto/urllib2 introduction}@anchor{4765}
@subsection Introduction


@subsubheading Related Articles



You may also find useful the following article on fetching web resources
with Python:


@itemize *

@item 
Basic Authentication@footnote{http://www.voidspace.org.uk/python/articles/authentication.shtml}

@quotation

A tutorial on @emph{Basic Authentication}, with examples in Python.
@end quotation
@end itemize

@strong{urllib.request} is a Python module for fetching URLs
(Uniform Resource Locators). It offers a very simple interface, in the form of
the @emph{urlopen} function. This is capable of fetching URLs using a variety of
different protocols. It also offers a slightly more complex interface for
handling common situations - like basic authentication, cookies, proxies and so
on. These are provided by objects called handlers and openers.

urllib.request supports fetching URLs for many “URL schemes” (identified by the string
before the @code{":"} in URL - for example @code{"ftp"} is the URL scheme of
@code{"ftp://python.org/"}) using their associated network protocols (e.g. FTP, HTTP).
This tutorial focuses on the most common case, HTTP.

For straightforward situations @emph{urlopen} is very easy to use. But as soon as you
encounter errors or non-trivial cases when opening HTTP URLs, you will need some
understanding of the HyperText Transfer Protocol. The most comprehensive and
authoritative reference to HTTP is 
@geindex RFC; RFC 2616
RFC 2616@footnote{https://tools.ietf.org/html/rfc2616.html}. This is a technical document and
not intended to be easy to read. This HOWTO aims to illustrate using @emph{urllib},
with enough detail about HTTP to help you through. It is not intended to replace
the @ref{127,,urllib.request} docs, but is supplementary to them.

@node Fetching URLs,Handling Exceptions<2>,Introduction<16>,HOWTO Fetch Internet Resources Using The urllib Package
@anchor{howto/urllib2 fetching-urls}@anchor{4766}
@subsection Fetching URLs


The simplest way to use urllib.request is as follows:

@example
import urllib.request
with urllib.request.urlopen('http://python.org/') as response:
   html = response.read()
@end example

If you wish to retrieve a resource via URL and store it in a temporary
location, you can do so via the @ref{574,,shutil.copyfileobj()} and
@ref{f44,,tempfile.NamedTemporaryFile()} functions:

@example
import shutil
import tempfile
import urllib.request

with urllib.request.urlopen('http://python.org/') as response:
    with tempfile.NamedTemporaryFile(delete=False) as tmp_file:
        shutil.copyfileobj(response, tmp_file)

with open(tmp_file.name) as html:
    pass
@end example

Many uses of urllib will be that simple (note that instead of an ‘http:’ URL we
could have used a URL starting with ‘ftp:’, ‘file:’, etc.).  However, it’s the
purpose of this tutorial to explain the more complicated cases, concentrating on
HTTP.

HTTP is based on requests and responses - the client makes requests and servers
send responses. urllib.request mirrors this with a @code{Request} object which represents
the HTTP request you are making. In its simplest form you create a Request
object that specifies the URL you want to fetch. Calling @code{urlopen} with this
Request object returns a response object for the URL requested. This response is
a file-like object, which means you can for example call @code{.read()} on the
response:

@example
import urllib.request

req = urllib.request.Request('http://www.voidspace.org.uk')
with urllib.request.urlopen(req) as response:
   the_page = response.read()
@end example

Note that urllib.request makes use of the same Request interface to handle all URL
schemes.  For example, you can make an FTP request like so:

@example
req = urllib.request.Request('ftp://example.com/')
@end example

In the case of HTTP, there are two extra things that Request objects allow you
to do: First, you can pass data to be sent to the server.  Second, you can pass
extra information (“metadata”) @emph{about} the data or about the request itself, to
the server - this information is sent as HTTP “headers”.  Let’s look at each of
these in turn.

@menu
* Data:: 
* Headers:: 

@end menu

@node Data,Headers,,Fetching URLs
@anchor{howto/urllib2 data}@anchor{4767}
@subsubsection Data


Sometimes you want to send data to a URL (often the URL will refer to a CGI
(Common Gateway Interface) script or other web application). With HTTP,
this is often done using what’s known as a @strong{POST} request. This is often what
your browser does when you submit a HTML form that you filled in on the web. Not
all POSTs have to come from forms: you can use a POST to transmit arbitrary data
to your own application. In the common case of HTML forms, the data needs to be
encoded in a standard way, and then passed to the Request object as the @code{data}
argument. The encoding is done using a function from the @ref{126,,urllib.parse}
library.

@example
import urllib.parse
import urllib.request

url = 'http://www.someserver.com/cgi-bin/register.cgi'
values = @{'name' : 'Michael Foord',
          'location' : 'Northampton',
          'language' : 'Python' @}

data = urllib.parse.urlencode(values)
data = data.encode('ascii') # data should be bytes
req = urllib.request.Request(url, data)
with urllib.request.urlopen(req) as response:
   the_page = response.read()
@end example

Note that other encodings are sometimes required (e.g. for file upload from HTML
forms - see HTML Specification@comma{} Form Submission@footnote{https://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13} for more
details).

If you do not pass the @code{data} argument, urllib uses a @strong{GET} request. One
way in which GET and POST requests differ is that POST requests often have
“side-effects”: they change the state of the system in some way (for example by
placing an order with the website for a hundredweight of tinned spam to be
delivered to your door).  Though the HTTP standard makes it clear that POSTs are
intended to @emph{always} cause side-effects, and GET requests @emph{never} to cause
side-effects, nothing prevents a GET request from having side-effects, nor a
POST requests from having no side-effects. Data can also be passed in an HTTP
GET request by encoding it in the URL itself.

This is done as follows:

@example
>>> import urllib.request
>>> import urllib.parse
>>> data = @{@}
>>> data['name'] = 'Somebody Here'
>>> data['location'] = 'Northampton'
>>> data['language'] = 'Python'
>>> url_values = urllib.parse.urlencode(data)
>>> print(url_values)  # The order may differ from below.  
name=Somebody+Here&language=Python&location=Northampton
>>> url = 'http://www.example.com/example.cgi'
>>> full_url = url + '?' + url_values
>>> data = urllib.request.urlopen(full_url)
@end example

Notice that the full URL is created by adding a @code{?} to the URL, followed by
the encoded values.

@node Headers,,Data,Fetching URLs
@anchor{howto/urllib2 headers}@anchor{4768}
@subsubsection Headers


We’ll discuss here one particular HTTP header, to illustrate how to add headers
to your HTTP request.

Some websites @footnote{@w{(1)} 
Google for example.
} dislike being browsed by programs, or send different versions
to different browsers @footnote{@w{(2)} 
Browser sniffing is a very bad practice for website design - building
sites using web standards is much more sensible. Unfortunately a lot of
sites still send different versions to different browsers.
}. By default urllib identifies itself as
@code{Python-urllib/x.y} (where @code{x} and @code{y} are the major and minor version
numbers of the Python release,
e.g. @code{Python-urllib/2.5}), which may confuse the site, or just plain
not work. The way a browser identifies itself is through the
@code{User-Agent} header @footnote{@w{(3)} 
The user agent for MSIE 6 is
@emph{‘Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)’}
}. When you create a Request object you can
pass a dictionary of headers in. The following example makes the same
request as above, but identifies itself as a version of Internet
Explorer @footnote{@w{(4)} 
For details of more HTTP request headers, see
@uref{http://jkorpela.fi/http.html,Quick Reference to HTTP Headers}.
}.

@example
import urllib.parse
import urllib.request

url = 'http://www.someserver.com/cgi-bin/register.cgi'
user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)'
values = @{'name': 'Michael Foord',
          'location': 'Northampton',
          'language': 'Python' @}
headers = @{'User-Agent': user_agent@}

data = urllib.parse.urlencode(values)
data = data.encode('ascii')
req = urllib.request.Request(url, data, headers)
with urllib.request.urlopen(req) as response:
   the_page = response.read()
@end example

The response also has two useful methods. See the section on @ref{4769,,info and geturl}
which comes after we have a look at what happens when things go wrong.

@node Handling Exceptions<2>,info and geturl,Fetching URLs,HOWTO Fetch Internet Resources Using The urllib Package
@anchor{howto/urllib2 handling-exceptions}@anchor{476a}
@subsection Handling Exceptions


@emph{urlopen} raises @code{URLError} when it cannot handle a response (though as
usual with Python APIs, built-in exceptions such as @ref{17c,,ValueError},
@ref{162,,TypeError} etc. may also be raised).

@code{HTTPError} is the subclass of @code{URLError} raised in the specific case of
HTTP URLs.

The exception classes are exported from the @ref{125,,urllib.error} module.

@menu
* URLError:: 
* HTTPError:: 
* Wrapping it Up:: 

@end menu

@node URLError,HTTPError,,Handling Exceptions<2>
@anchor{howto/urllib2 urlerror}@anchor{476b}
@subsubsection URLError


Often, URLError is raised because there is no network connection (no route to
the specified server), or the specified server doesn’t exist.  In this case, the
exception raised will have a ‘reason’ attribute, which is a tuple containing an
error code and a text error message.

e.g.

@example
>>> req = urllib.request.Request('http://www.pretend_server.org')
>>> try: urllib.request.urlopen(req)
... except urllib.error.URLError as e:
...     print(e.reason)      
...
(4, 'getaddrinfo failed')
@end example

@node HTTPError,Wrapping it Up,URLError,Handling Exceptions<2>
@anchor{howto/urllib2 httperror}@anchor{476c}
@subsubsection HTTPError


Every HTTP response from the server contains a numeric “status code”. Sometimes
the status code indicates that the server is unable to fulfil the request. The
default handlers will handle some of these responses for you (for example, if
the response is a “redirection” that requests the client fetch the document from
a different URL, urllib will handle that for you). For those it can’t handle,
urlopen will raise an @code{HTTPError}. Typical errors include ‘404’ (page not
found), ‘403’ (request forbidden), and ‘401’ (authentication required).

See section 10 of 
@geindex RFC; RFC 2616
RFC 2616@footnote{https://tools.ietf.org/html/rfc2616.html} for a reference on all the HTTP error codes.

The @code{HTTPError} instance raised will have an integer ‘code’ attribute, which
corresponds to the error sent by the server.

@menu
* Error Codes:: 

@end menu

@node Error Codes,,,HTTPError
@anchor{howto/urllib2 error-codes}@anchor{476d}
@subsubsection Error Codes


Because the default handlers handle redirects (codes in the 300 range), and
codes in the 100–299 range indicate success, you will usually only see error
codes in the 400–599 range.

@ref{315e,,http.server.BaseHTTPRequestHandler.responses} is a useful dictionary of
response codes in that shows all the response codes used by 
@geindex RFC; RFC 2616
RFC 2616@footnote{https://tools.ietf.org/html/rfc2616.html}. The
dictionary is reproduced here for convenience

@example
# Table mapping response codes to messages; entries have the
# form @{code: (shortmessage, longmessage)@}.
responses = @{
    100: ('Continue', 'Request received, please continue'),
    101: ('Switching Protocols',
          'Switching to new protocol; obey Upgrade header'),

    200: ('OK', 'Request fulfilled, document follows'),
    201: ('Created', 'Document created, URL follows'),
    202: ('Accepted',
          'Request accepted, processing continues off-line'),
    203: ('Non-Authoritative Information', 'Request fulfilled from cache'),
    204: ('No Content', 'Request fulfilled, nothing follows'),
    205: ('Reset Content', 'Clear input form for further input.'),
    206: ('Partial Content', 'Partial content follows.'),

    300: ('Multiple Choices',
          'Object has several resources -- see URI list'),
    301: ('Moved Permanently', 'Object moved permanently -- see URI list'),
    302: ('Found', 'Object moved temporarily -- see URI list'),
    303: ('See Other', 'Object moved -- see Method and URL list'),
    304: ('Not Modified',
          'Document has not changed since given time'),
    305: ('Use Proxy',
          'You must use proxy specified in Location to access this '
          'resource.'),
    307: ('Temporary Redirect',
          'Object moved temporarily -- see URI list'),

    400: ('Bad Request',
          'Bad request syntax or unsupported method'),
    401: ('Unauthorized',
          'No permission -- see authorization schemes'),
    402: ('Payment Required',
          'No payment -- see charging schemes'),
    403: ('Forbidden',
          'Request forbidden -- authorization will not help'),
    404: ('Not Found', 'Nothing matches the given URI'),
    405: ('Method Not Allowed',
          'Specified method is invalid for this server.'),
    406: ('Not Acceptable', 'URI not available in preferred format.'),
    407: ('Proxy Authentication Required', 'You must authenticate with '
          'this proxy before proceeding.'),
    408: ('Request Timeout', 'Request timed out; try again later.'),
    409: ('Conflict', 'Request conflict.'),
    410: ('Gone',
          'URI no longer exists and has been permanently removed.'),
    411: ('Length Required', 'Client must specify Content-Length.'),
    412: ('Precondition Failed', 'Precondition in headers is false.'),
    413: ('Request Entity Too Large', 'Entity is too large.'),
    414: ('Request-URI Too Long', 'URI is too long.'),
    415: ('Unsupported Media Type', 'Entity body in unsupported format.'),
    416: ('Requested Range Not Satisfiable',
          'Cannot satisfy request range.'),
    417: ('Expectation Failed',
          'Expect condition could not be satisfied.'),

    500: ('Internal Server Error', 'Server got itself in trouble'),
    501: ('Not Implemented',
          'Server does not support this operation'),
    502: ('Bad Gateway', 'Invalid responses from another server/proxy.'),
    503: ('Service Unavailable',
          'The server cannot process the request due to a high load'),
    504: ('Gateway Timeout',
          'The gateway server did not receive a timely response'),
    505: ('HTTP Version Not Supported', 'Cannot fulfill request.'),
    @}
@end example

When an error is raised the server responds by returning an HTTP error code
@emph{and} an error page. You can use the @code{HTTPError} instance as a response on the
page returned. This means that as well as the code attribute, it also has read,
geturl, and info, methods as returned by the @code{urllib.response} module:

@example
>>> req = urllib.request.Request('http://www.python.org/fish.html')
>>> try:
...     urllib.request.urlopen(req)
... except urllib.error.HTTPError as e:
...     print(e.code)
...     print(e.read())  
...
404
b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n\n\n<html
  ...
  <title>Page Not Found</title>\n
  ...
@end example

@node Wrapping it Up,,HTTPError,Handling Exceptions<2>
@anchor{howto/urllib2 wrapping-it-up}@anchor{476e}
@subsubsection Wrapping it Up


So if you want to be prepared for @code{HTTPError} @emph{or} @code{URLError} there are two
basic approaches. I prefer the second approach.

@menu
* Number 1:: 
* Number 2:: 

@end menu

@node Number 1,Number 2,,Wrapping it Up
@anchor{howto/urllib2 number-1}@anchor{476f}
@subsubsection Number 1


@example
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError
req = Request(someurl)
try:
    response = urlopen(req)
except HTTPError as e:
    print('The server couldn\'t fulfill the request.')
    print('Error code: ', e.code)
except URLError as e:
    print('We failed to reach a server.')
    print('Reason: ', e.reason)
else:
    # everything is fine
@end example

@cartouche
@quotation Note 
The @code{except HTTPError} @emph{must} come first, otherwise @code{except URLError}
will @emph{also} catch an @code{HTTPError}.
@end quotation
@end cartouche

@node Number 2,,Number 1,Wrapping it Up
@anchor{howto/urllib2 number-2}@anchor{4770}
@subsubsection Number 2


@example
from urllib.request import Request, urlopen
from urllib.error import URLError
req = Request(someurl)
try:
    response = urlopen(req)
except URLError as e:
    if hasattr(e, 'reason'):
        print('We failed to reach a server.')
        print('Reason: ', e.reason)
    elif hasattr(e, 'code'):
        print('The server couldn\'t fulfill the request.')
        print('Error code: ', e.code)
else:
    # everything is fine
@end example

@node info and geturl,Openers and Handlers,Handling Exceptions<2>,HOWTO Fetch Internet Resources Using The urllib Package
@anchor{howto/urllib2 info-and-geturl}@anchor{4769}
@subsection info and geturl


The response returned by urlopen (or the @code{HTTPError} instance) has two
useful methods @code{info()} and @code{geturl()} and is defined in the module
@ref{128,,urllib.response}..

@strong{geturl} - this returns the real URL of the page fetched. This is useful
because @code{urlopen} (or the opener object used) may have followed a
redirect. The URL of the page fetched may not be the same as the URL requested.

@strong{info} - this returns a dictionary-like object that describes the page
fetched, particularly the headers sent by the server. It is currently an
@code{http.client.HTTPMessage} instance.

Typical headers include ‘Content-length’, ‘Content-type’, and so on. See the
Quick Reference to HTTP Headers@footnote{http://jkorpela.fi/http.html}
for a useful listing of HTTP headers with brief explanations of their meaning
and use.

@node Openers and Handlers,Basic Authentication,info and geturl,HOWTO Fetch Internet Resources Using The urllib Package
@anchor{howto/urllib2 openers-and-handlers}@anchor{4771}
@subsection Openers and Handlers


When you fetch a URL you use an opener (an instance of the perhaps
confusingly-named @ref{30bc,,urllib.request.OpenerDirector}). Normally we have been using
the default opener - via @code{urlopen} - but you can create custom
openers. Openers use handlers. All the “heavy lifting” is done by the
handlers. Each handler knows how to open URLs for a particular URL scheme (http,
ftp, etc.), or how to handle an aspect of URL opening, for example HTTP
redirections or HTTP cookies.

You will want to create openers if you want to fetch URLs with specific handlers
installed, for example to get an opener that handles cookies, or to get an
opener that does not handle redirections.

To create an opener, instantiate an @code{OpenerDirector}, and then call
@code{.add_handler(some_handler_instance)} repeatedly.

Alternatively, you can use @code{build_opener}, which is a convenience function for
creating opener objects with a single function call.  @code{build_opener} adds
several handlers by default, but provides a quick way to add more and/or
override the default handlers.

Other sorts of handlers you might want to can handle proxies, authentication,
and other common but slightly specialised situations.

@code{install_opener} can be used to make an @code{opener} object the (global) default
opener. This means that calls to @code{urlopen} will use the opener you have
installed.

Opener objects have an @code{open} method, which can be called directly to fetch
urls in the same way as the @code{urlopen} function: there’s no need to call
@code{install_opener}, except as a convenience.

@node Basic Authentication,Proxies,Openers and Handlers,HOWTO Fetch Internet Resources Using The urllib Package
@anchor{howto/urllib2 id5}@anchor{4772}
@subsection Basic Authentication


To illustrate creating and installing a handler we will use the
@code{HTTPBasicAuthHandler}. For a more detailed discussion of this subject –
including an explanation of how Basic Authentication works - see the Basic Authentication Tutorial@footnote{http://www.voidspace.org.uk/python/articles/authentication.shtml}.

When authentication is required, the server sends a header (as well as the 401
error code) requesting authentication.  This specifies the authentication scheme
and a ‘realm’. The header looks like: @code{WWW-Authenticate: SCHEME
realm="REALM"}.

e.g.

@example
WWW-Authenticate: Basic realm="cPanel Users"
@end example

The client should then retry the request with the appropriate name and password
for the realm included as a header in the request. This is ‘basic
authentication’. In order to simplify this process we can create an instance of
@code{HTTPBasicAuthHandler} and an opener to use this handler.

The @code{HTTPBasicAuthHandler} uses an object called a password manager to handle
the mapping of URLs and realms to passwords and usernames. If you know what the
realm is (from the authentication header sent by the server), then you can use a
@code{HTTPPasswordMgr}. Frequently one doesn’t care what the realm is. In that
case, it is convenient to use @code{HTTPPasswordMgrWithDefaultRealm}. This allows
you to specify a default username and password for a URL. This will be supplied
in the absence of you providing an alternative combination for a specific
realm. We indicate this by providing @code{None} as the realm argument to the
@code{add_password} method.

The top-level URL is the first URL that requires authentication. URLs “deeper”
than the URL you pass to .add_password() will also match.

@example
# create a password manager
password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()

# Add the username and password.
# If we knew the realm, we could use it instead of None.
top_level_url = "http://example.com/foo/"
password_mgr.add_password(None, top_level_url, username, password)

handler = urllib.request.HTTPBasicAuthHandler(password_mgr)

# create "opener" (OpenerDirector instance)
opener = urllib.request.build_opener(handler)

# use the opener to fetch a URL
opener.open(a_url)

# Install the opener.
# Now all calls to urllib.request.urlopen use our opener.
urllib.request.install_opener(opener)
@end example

@cartouche
@quotation Note 
In the above example we only supplied our @code{HTTPBasicAuthHandler} to
@code{build_opener}. By default openers have the handlers for normal situations
– @code{ProxyHandler} (if a proxy setting such as an 
@geindex http_proxy
@geindex environment variable; http_proxy
@code{http_proxy}
environment variable is set), @code{UnknownHandler}, @code{HTTPHandler},
@code{HTTPDefaultErrorHandler}, @code{HTTPRedirectHandler}, @code{FTPHandler},
@code{FileHandler}, @code{DataHandler}, @code{HTTPErrorProcessor}.
@end quotation
@end cartouche

@code{top_level_url} is in fact @emph{either} a full URL (including the ‘http:’ scheme
component and the hostname and optionally the port number)
e.g. @code{"http://example.com/"} @emph{or} an “authority” (i.e. the hostname,
optionally including the port number) e.g. @code{"example.com"} or @code{"example.com:8080"}
(the latter example includes a port number).  The authority, if present, must
NOT contain the “userinfo” component - for example @code{"joe:password@@example.com"} is
not correct.

@node Proxies,Sockets and Layers,Basic Authentication,HOWTO Fetch Internet Resources Using The urllib Package
@anchor{howto/urllib2 proxies}@anchor{4773}
@subsection Proxies


@strong{urllib} will auto-detect your proxy settings and use those. This is through
the @code{ProxyHandler}, which is part of the normal handler chain when a proxy
setting is detected.  Normally that’s a good thing, but there are occasions
when it may not be helpful @footnote{@w{(5)} 
In my case I have to use a proxy to access the internet at work. If you
attempt to fetch @emph{localhost} URLs through this proxy it blocks them. IE
is set to use the proxy, which urllib picks up on. In order to test
scripts with a localhost server, I have to prevent urllib from using
the proxy.
}. One way to do this is to setup our own
@code{ProxyHandler}, with no proxies defined. This is done using similar steps to
setting up a Basic Authentication@footnote{http://www.voidspace.org.uk/python/articles/authentication.shtml} handler:

@example
>>> proxy_support = urllib.request.ProxyHandler(@{@})
>>> opener = urllib.request.build_opener(proxy_support)
>>> urllib.request.install_opener(opener)
@end example

@cartouche
@quotation Note 
Currently @code{urllib.request} @emph{does not} support fetching of @code{https} locations
through a proxy.  However, this can be enabled by extending urllib.request as
shown in the recipe @footnote{@w{(6)} 
urllib opener for SSL proxy (CONNECT method): @uref{https://code.activestate.com/recipes/456195/,ASPN Cookbook Recipe}.
}.
@end quotation
@end cartouche

@cartouche
@quotation Note 
@code{HTTP_PROXY} will be ignored if a variable @code{REQUEST_METHOD} is set; see
the documentation on @ref{30c9,,getproxies()}.
@end quotation
@end cartouche

@node Sockets and Layers,Footnotes,Proxies,HOWTO Fetch Internet Resources Using The urllib Package
@anchor{howto/urllib2 sockets-and-layers}@anchor{4774}
@subsection Sockets and Layers


The Python support for fetching resources from the web is layered.  urllib uses
the @ref{8f,,http.client} library, which in turn uses the socket library.

As of Python 2.3 you can specify how long a socket should wait for a response
before timing out. This can be useful in applications which have to fetch web
pages. By default the socket module has @emph{no timeout} and can hang. Currently,
the socket timeout is not exposed at the http.client or urllib.request levels.
However, you can set the default timeout globally for all sockets using

@example
import socket
import urllib.request

# timeout in seconds
timeout = 10
socket.setdefaulttimeout(timeout)

# this call to urllib.request.urlopen now uses the default timeout
# we have set in the socket module
req = urllib.request.Request('http://www.voidspace.org.uk')
response = urllib.request.urlopen(req)
@end example


__________________________________________________________________


@node Footnotes,,Sockets and Layers,HOWTO Fetch Internet Resources Using The urllib Package
@anchor{howto/urllib2 footnotes}@anchor{4775}
@subsection Footnotes


This document was reviewed and revised by John Lee.

@node Argparse Tutorial,An introduction to the ipaddress module,HOWTO Fetch Internet Resources Using The urllib Package,Python HOWTOs
@anchor{howto/argparse doc}@anchor{4776}@anchor{howto/argparse argparse-tutorial}@anchor{4777}
@section Argparse Tutorial


@*author: 
Tshepang Lekhonkhobe

@anchor{howto/argparse id1}@anchor{24c8}
This tutorial is intended to be a gentle introduction to @ref{5,,argparse}, the
recommended command-line parsing module in the Python standard library.

@cartouche
@quotation Note 
There are two other modules that fulfill the same task, namely
@ref{81,,getopt} (an equivalent for @code{getopt()} from the C
language) and the deprecated @ref{bf,,optparse}.
Note also that @ref{5,,argparse} is based on @ref{bf,,optparse},
and therefore very similar in terms of usage.
@end quotation
@end cartouche

@menu
* Concepts:: 
* The basics:: 
* Introducing Positional arguments:: 
* Introducing Optional arguments:: 
* Combining Positional and Optional arguments:: 
* Getting a little more advanced:: 
* Conclusion:: 

@end menu

@node Concepts,The basics,,Argparse Tutorial
@anchor{howto/argparse concepts}@anchor{4778}
@subsection Concepts


Let’s show the sort of functionality that we are going to explore in this
introductory tutorial by making use of the @code{ls} command:

@example
$ ls
cpython  devguide  prog.py  pypy  rm-unused-function.patch
$ ls pypy
ctypes_configure  demo  dotviewer  include  lib_pypy  lib-python ...
$ ls -l
total 20
drwxr-xr-x 19 wena wena 4096 Feb 18 18:51 cpython
drwxr-xr-x  4 wena wena 4096 Feb  8 12:04 devguide
-rwxr-xr-x  1 wena wena  535 Feb 19 00:05 prog.py
drwxr-xr-x 14 wena wena 4096 Feb  7 00:59 pypy
-rw-r--r--  1 wena wena  741 Feb 18 01:01 rm-unused-function.patch
$ ls --help
Usage: ls [OPTION]... [FILE]...
List information about the FILEs (the current directory by default).
Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.
...
@end example

A few concepts we can learn from the four commands:


@itemize *

@item 
The @code{ls} command is useful when run without any options at all. It defaults
to displaying the contents of the current directory.

@item 
If we want beyond what it provides by default, we tell it a bit more. In
this case, we want it to display a different directory, @code{pypy}.
What we did is specify what is known as a positional argument. It’s named so
because the program should know what to do with the value, solely based on
where it appears on the command line. This concept is more relevant
to a command like @code{cp}, whose most basic usage is @code{cp SRC DEST}.
The first position is @emph{what you want copied,} and the second
position is @emph{where you want it copied to}.

@item 
Now, say we want to change behaviour of the program. In our example,
we display more info for each file instead of just showing the file names.
The @code{-l} in that case is known as an optional argument.

@item 
That’s a snippet of the help text. It’s very useful in that you can
come across a program you have never used before, and can figure out
how it works simply by reading its help text.
@end itemize

@node The basics,Introducing Positional arguments,Concepts,Argparse Tutorial
@anchor{howto/argparse the-basics}@anchor{4779}
@subsection The basics


Let us start with a very simple example which does (almost) nothing:

@example
import argparse
parser = argparse.ArgumentParser()
parser.parse_args()
@end example

Following is a result of running the code:

@example
$ python3 prog.py
$ python3 prog.py --help
usage: prog.py [-h]

options:
  -h, --help  show this help message and exit
$ python3 prog.py --verbose
usage: prog.py [-h]
prog.py: error: unrecognized arguments: --verbose
$ python3 prog.py foo
usage: prog.py [-h]
prog.py: error: unrecognized arguments: foo
@end example

Here is what is happening:


@itemize *

@item 
Running the script without any options results in nothing displayed to
stdout. Not so useful.

@item 
The second one starts to display the usefulness of the @ref{5,,argparse}
module. We have done almost nothing, but already we get a nice help message.

@item 
The @code{--help} option, which can also be shortened to @code{-h}, is the only
option we get for free (i.e. no need to specify it). Specifying anything
else results in an error. But even then, we do get a useful usage message,
also for free.
@end itemize

@node Introducing Positional arguments,Introducing Optional arguments,The basics,Argparse Tutorial
@anchor{howto/argparse introducing-positional-arguments}@anchor{477a}
@subsection Introducing Positional arguments


An example:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("echo")
args = parser.parse_args()
print(args.echo)
@end example

And running the code:

@example
$ python3 prog.py
usage: prog.py [-h] echo
prog.py: error: the following arguments are required: echo
$ python3 prog.py --help
usage: prog.py [-h] echo

positional arguments:
  echo

options:
  -h, --help  show this help message and exit
$ python3 prog.py foo
foo
@end example

Here is what’s happening:


@itemize *

@item 
We’ve added the @code{add_argument()} method, which is what we use to specify
which command-line options the program is willing to accept. In this case,
I’ve named it @code{echo} so that it’s in line with its function.

@item 
Calling our program now requires us to specify an option.

@item 
The @code{parse_args()} method actually returns some data from the
options specified, in this case, @code{echo}.

@item 
The variable is some form of ‘magic’ that @ref{5,,argparse} performs for free
(i.e. no need to specify which variable that value is stored in).
You will also notice that its name matches the string argument given
to the method, @code{echo}.
@end itemize

Note however that, although the help display looks nice and all, it currently
is not as helpful as it can be. For example we see that we got @code{echo} as a
positional argument, but we don’t know what it does, other than by guessing or
by reading the source code. So, let’s make it a bit more useful:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("echo", help="echo the string you use here")
args = parser.parse_args()
print(args.echo)
@end example

And we get:

@example
$ python3 prog.py -h
usage: prog.py [-h] echo

positional arguments:
  echo        echo the string you use here

options:
  -h, --help  show this help message and exit
@end example

Now, how about doing something even more useful:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", help="display a square of a given number")
args = parser.parse_args()
print(args.square**2)
@end example

Following is a result of running the code:

@example
$ python3 prog.py 4
Traceback (most recent call last):
  File "prog.py", line 5, in <module>
    print(args.square**2)
TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int'
@end example

That didn’t go so well. That’s because @ref{5,,argparse} treats the options we
give it as strings, unless we tell it otherwise. So, let’s tell
@ref{5,,argparse} to treat that input as an integer:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", help="display a square of a given number",
                    type=int)
args = parser.parse_args()
print(args.square**2)
@end example

Following is a result of running the code:

@example
$ python3 prog.py 4
16
$ python3 prog.py four
usage: prog.py [-h] square
prog.py: error: argument square: invalid int value: 'four'
@end example

That went well. The program now even helpfully quits on bad illegal input
before proceeding.

@node Introducing Optional arguments,Combining Positional and Optional arguments,Introducing Positional arguments,Argparse Tutorial
@anchor{howto/argparse introducing-optional-arguments}@anchor{477b}
@subsection Introducing Optional arguments


So far we have been playing with positional arguments. Let us
have a look on how to add optional ones:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--verbosity", help="increase output verbosity")
args = parser.parse_args()
if args.verbosity:
    print("verbosity turned on")
@end example

And the output:

@example
$ python3 prog.py --verbosity 1
verbosity turned on
$ python3 prog.py
$ python3 prog.py --help
usage: prog.py [-h] [--verbosity VERBOSITY]

options:
  -h, --help            show this help message and exit
  --verbosity VERBOSITY
                        increase output verbosity
$ python3 prog.py --verbosity
usage: prog.py [-h] [--verbosity VERBOSITY]
prog.py: error: argument --verbosity: expected one argument
@end example

Here is what is happening:


@itemize *

@item 
The program is written so as to display something when @code{--verbosity} is
specified and display nothing when not.

@item 
To show that the option is actually optional, there is no error when running
the program without it. Note that by default, if an optional argument isn’t
used, the relevant variable, in this case @code{args.verbosity}, is
given @code{None} as a value, which is the reason it fails the truth
test of the @ref{2fb,,if} statement.

@item 
The help message is a bit different.

@item 
When using the @code{--verbosity} option, one must also specify some value,
any value.
@end itemize

The above example accepts arbitrary integer values for @code{--verbosity}, but for
our simple program, only two values are actually useful, @code{True} or @code{False}.
Let’s modify the code accordingly:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--verbose", help="increase output verbosity",
                    action="store_true")
args = parser.parse_args()
if args.verbose:
    print("verbosity turned on")
@end example

And the output:

@example
$ python3 prog.py --verbose
verbosity turned on
$ python3 prog.py --verbose 1
usage: prog.py [-h] [--verbose]
prog.py: error: unrecognized arguments: 1
$ python3 prog.py --help
usage: prog.py [-h] [--verbose]

options:
  -h, --help  show this help message and exit
  --verbose   increase output verbosity
@end example

Here is what is happening:


@itemize *

@item 
The option is now more of a flag than something that requires a value.
We even changed the name of the option to match that idea.
Note that we now specify a new keyword, @code{action}, and give it the value
@code{"store_true"}. This means that, if the option is specified,
assign the value @code{True} to @code{args.verbose}.
Not specifying it implies @code{False}.

@item 
It complains when you specify a value, in true spirit of what flags
actually are.

@item 
Notice the different help text.
@end itemize

@menu
* Short options:: 

@end menu

@node Short options,,,Introducing Optional arguments
@anchor{howto/argparse short-options}@anchor{477c}
@subsubsection Short options


If you are familiar with command line usage,
you will notice that I haven’t yet touched on the topic of short
versions of the options. It’s quite simple:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("-v", "--verbose", help="increase output verbosity",
                    action="store_true")
args = parser.parse_args()
if args.verbose:
    print("verbosity turned on")
@end example

And here goes:

@example
$ python3 prog.py -v
verbosity turned on
$ python3 prog.py --help
usage: prog.py [-h] [-v]

options:
  -h, --help     show this help message and exit
  -v, --verbose  increase output verbosity
@end example

Note that the new ability is also reflected in the help text.

@node Combining Positional and Optional arguments,Getting a little more advanced,Introducing Optional arguments,Argparse Tutorial
@anchor{howto/argparse combining-positional-and-optional-arguments}@anchor{477d}
@subsection Combining Positional and Optional arguments


Our program keeps growing in complexity:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
                    help="display a square of a given number")
parser.add_argument("-v", "--verbose", action="store_true",
                    help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbose:
    print(f"the square of @{args.square@} equals @{answer@}")
else:
    print(answer)
@end example

And now the output:

@example
$ python3 prog.py
usage: prog.py [-h] [-v] square
prog.py: error: the following arguments are required: square
$ python3 prog.py 4
16
$ python3 prog.py 4 --verbose
the square of 4 equals 16
$ python3 prog.py --verbose 4
the square of 4 equals 16
@end example


@itemize *

@item 
We’ve brought back a positional argument, hence the complaint.

@item 
Note that the order does not matter.
@end itemize

How about we give this program of ours back the ability to have
multiple verbosity values, and actually get to use them:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
                    help="display a square of a given number")
parser.add_argument("-v", "--verbosity", type=int,
                    help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbosity == 2:
    print(f"the square of @{args.square@} equals @{answer@}")
elif args.verbosity == 1:
    print(f"@{args.square@}^2 == @{answer@}")
else:
    print(answer)
@end example

And the output:

@example
$ python3 prog.py 4
16
$ python3 prog.py 4 -v
usage: prog.py [-h] [-v VERBOSITY] square
prog.py: error: argument -v/--verbosity: expected one argument
$ python3 prog.py 4 -v 1
4^2 == 16
$ python3 prog.py 4 -v 2
the square of 4 equals 16
$ python3 prog.py 4 -v 3
16
@end example

These all look good except the last one, which exposes a bug in our program.
Let’s fix it by restricting the values the @code{--verbosity} option can accept:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
                    help="display a square of a given number")
parser.add_argument("-v", "--verbosity", type=int, choices=[0, 1, 2],
                    help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbosity == 2:
    print(f"the square of @{args.square@} equals @{answer@}")
elif args.verbosity == 1:
    print(f"@{args.square@}^2 == @{answer@}")
else:
    print(answer)
@end example

And the output:

@example
$ python3 prog.py 4 -v 3
usage: prog.py [-h] [-v @{0,1,2@}] square
prog.py: error: argument -v/--verbosity: invalid choice: 3 (choose from 0, 1, 2)
$ python3 prog.py 4 -h
usage: prog.py [-h] [-v @{0,1,2@}] square

positional arguments:
  square                display a square of a given number

options:
  -h, --help            show this help message and exit
  -v @{0,1,2@}, --verbosity @{0,1,2@}
                        increase output verbosity
@end example

Note that the change also reflects both in the error message as well as the
help string.

Now, let’s use a different approach of playing with verbosity, which is pretty
common. It also matches the way the CPython executable handles its own
verbosity argument (check the output of @code{python --help}):

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
                    help="display the square of a given number")
parser.add_argument("-v", "--verbosity", action="count",
                    help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbosity == 2:
    print(f"the square of @{args.square@} equals @{answer@}")
elif args.verbosity == 1:
    print(f"@{args.square@}^2 == @{answer@}")
else:
    print(answer)
@end example

We have introduced another action, “count”,
to count the number of occurrences of specific options.

@example
$ python3 prog.py 4
16
$ python3 prog.py 4 -v
4^2 == 16
$ python3 prog.py 4 -vv
the square of 4 equals 16
$ python3 prog.py 4 --verbosity --verbosity
the square of 4 equals 16
$ python3 prog.py 4 -v 1
usage: prog.py [-h] [-v] square
prog.py: error: unrecognized arguments: 1
$ python3 prog.py 4 -h
usage: prog.py [-h] [-v] square

positional arguments:
  square           display a square of a given number

options:
  -h, --help       show this help message and exit
  -v, --verbosity  increase output verbosity
$ python3 prog.py 4 -vvv
16
@end example


@itemize *

@item 
Yes, it’s now more of a flag (similar to @code{action="store_true"}) in the
previous version of our script. That should explain the complaint.

@item 
It also behaves similar to “store_true” action.

@item 
Now here’s a demonstration of what the “count” action gives. You’ve probably
seen this sort of usage before.

@item 
And if you don’t specify the @code{-v} flag, that flag is considered to have
@code{None} value.

@item 
As should be expected, specifying the long form of the flag, we should get
the same output.

@item 
Sadly, our help output isn’t very informative on the new ability our script
has acquired, but that can always be fixed by improving the documentation for
our script (e.g. via the @code{help} keyword argument).

@item 
That last output exposes a bug in our program.
@end itemize

Let’s fix:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
                    help="display a square of a given number")
parser.add_argument("-v", "--verbosity", action="count",
                    help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2

# bugfix: replace == with >=
if args.verbosity >= 2:
    print(f"the square of @{args.square@} equals @{answer@}")
elif args.verbosity >= 1:
    print(f"@{args.square@}^2 == @{answer@}")
else:
    print(answer)
@end example

And this is what it gives:

@example
$ python3 prog.py 4 -vvv
the square of 4 equals 16
$ python3 prog.py 4 -vvvv
the square of 4 equals 16
$ python3 prog.py 4
Traceback (most recent call last):
  File "prog.py", line 11, in <module>
    if args.verbosity >= 2:
TypeError: '>=' not supported between instances of 'NoneType' and 'int'
@end example


@itemize *

@item 
First output went well, and fixes the bug we had before.
That is, we want any value >= 2 to be as verbose as possible.

@item 
Third output not so good.
@end itemize

Let’s fix that bug:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
                    help="display a square of a given number")
parser.add_argument("-v", "--verbosity", action="count", default=0,
                    help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbosity >= 2:
    print(f"the square of @{args.square@} equals @{answer@}")
elif args.verbosity >= 1:
    print(f"@{args.square@}^2 == @{answer@}")
else:
    print(answer)
@end example

We’ve just introduced yet another keyword, @code{default}.
We’ve set it to @code{0} in order to make it comparable to the other int values.
Remember that by default,
if an optional argument isn’t specified,
it gets the @code{None} value, and that cannot be compared to an int value
(hence the @ref{162,,TypeError} exception).

And:

@example
$ python3 prog.py 4
16
@end example

You can go quite far just with what we’ve learned so far,
and we have only scratched the surface.
The @ref{5,,argparse} module is very powerful,
and we’ll explore a bit more of it before we end this tutorial.

@node Getting a little more advanced,Conclusion,Combining Positional and Optional arguments,Argparse Tutorial
@anchor{howto/argparse getting-a-little-more-advanced}@anchor{477e}
@subsection Getting a little more advanced


What if we wanted to expand our tiny program to perform other powers,
not just squares:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("x", type=int, help="the base")
parser.add_argument("y", type=int, help="the exponent")
parser.add_argument("-v", "--verbosity", action="count", default=0)
args = parser.parse_args()
answer = args.x**args.y
if args.verbosity >= 2:
    print(f"@{args.x@} to the power @{args.y@} equals @{answer@}")
elif args.verbosity >= 1:
    print(f"@{args.x@}^@{args.y@} == @{answer@}")
else:
    print(answer)
@end example

Output:

@example
$ python3 prog.py
usage: prog.py [-h] [-v] x y
prog.py: error: the following arguments are required: x, y
$ python3 prog.py -h
usage: prog.py [-h] [-v] x y

positional arguments:
  x                the base
  y                the exponent

options:
  -h, --help       show this help message and exit
  -v, --verbosity
$ python3 prog.py 4 2 -v
4^2 == 16
@end example

Notice that so far we’ve been using verbosity level to @emph{change} the text
that gets displayed. The following example instead uses verbosity level
to display @emph{more} text instead:

@example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("x", type=int, help="the base")
parser.add_argument("y", type=int, help="the exponent")
parser.add_argument("-v", "--verbosity", action="count", default=0)
args = parser.parse_args()
answer = args.x**args.y
if args.verbosity >= 2:
    print(f"Running '@{__file__@}'")
if args.verbosity >= 1:
    print(f"@{args.x@}^@{args.y@} == ", end="")
print(answer)
@end example

Output:

@example
$ python3 prog.py 4 2
16
$ python3 prog.py 4 2 -v
4^2 == 16
$ python3 prog.py 4 2 -vv
Running 'prog.py'
4^2 == 16
@end example

@menu
* Conflicting options:: 

@end menu

@node Conflicting options,,,Getting a little more advanced
@anchor{howto/argparse conflicting-options}@anchor{477f}
@subsubsection Conflicting options


So far, we have been working with two methods of an
@ref{91c,,argparse.ArgumentParser} instance. Let’s introduce a third one,
@code{add_mutually_exclusive_group()}. It allows for us to specify options that
conflict with each other. Let’s also change the rest of the program so that
the new functionality makes more sense:
we’ll introduce the @code{--quiet} option,
which will be the opposite of the @code{--verbose} one:

@example
import argparse

parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("-v", "--verbose", action="store_true")
group.add_argument("-q", "--quiet", action="store_true")
parser.add_argument("x", type=int, help="the base")
parser.add_argument("y", type=int, help="the exponent")
args = parser.parse_args()
answer = args.x**args.y

if args.quiet:
    print(answer)
elif args.verbose:
    print(f"@{args.x@} to the power @{args.y@} equals @{answer@}")
else:
    print(f"@{args.x@}^@{args.y@} == @{answer@}")
@end example

Our program is now simpler, and we’ve lost some functionality for the sake of
demonstration. Anyways, here’s the output:

@example
$ python3 prog.py 4 2
4^2 == 16
$ python3 prog.py 4 2 -q
16
$ python3 prog.py 4 2 -v
4 to the power 2 equals 16
$ python3 prog.py 4 2 -vq
usage: prog.py [-h] [-v | -q] x y
prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose
$ python3 prog.py 4 2 -v --quiet
usage: prog.py [-h] [-v | -q] x y
prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose
@end example

That should be easy to follow. I’ve added that last output so you can see the
sort of flexibility you get, i.e. mixing long form options with short form
ones.

Before we conclude, you probably want to tell your users the main purpose of
your program, just in case they don’t know:

@example
import argparse

parser = argparse.ArgumentParser(description="calculate X to the power of Y")
group = parser.add_mutually_exclusive_group()
group.add_argument("-v", "--verbose", action="store_true")
group.add_argument("-q", "--quiet", action="store_true")
parser.add_argument("x", type=int, help="the base")
parser.add_argument("y", type=int, help="the exponent")
args = parser.parse_args()
answer = args.x**args.y

if args.quiet:
    print(answer)
elif args.verbose:
    print("@{@} to the power @{@} equals @{@}".format(args.x, args.y, answer))
else:
    print("@{@}^@{@} == @{@}".format(args.x, args.y, answer))
@end example

Note that slight difference in the usage text. Note the @code{[-v | -q]},
which tells us that we can either use @code{-v} or @code{-q},
but not both at the same time:

@example
$ python3 prog.py --help
usage: prog.py [-h] [-v | -q] x y

calculate X to the power of Y

positional arguments:
  x              the base
  y              the exponent

options:
  -h, --help     show this help message and exit
  -v, --verbose
  -q, --quiet
@end example

@node Conclusion,,Getting a little more advanced,Argparse Tutorial
@anchor{howto/argparse conclusion}@anchor{4780}
@subsection Conclusion


The @ref{5,,argparse} module offers a lot more than shown here.
Its docs are quite detailed and thorough, and full of examples.
Having gone through this tutorial, you should easily digest them
without feeling overwhelmed.

@c import ipaddress

@node An introduction to the ipaddress module,Argument Clinic How-To,Argparse Tutorial,Python HOWTOs
@anchor{howto/ipaddress doc}@anchor{4781}@anchor{howto/ipaddress an-introduction-to-the-ipaddress-module}@anchor{4782}@anchor{howto/ipaddress ipaddress-howto}@anchor{338b}
@section An introduction to the ipaddress module


@*author: 
Peter Moody


@*author: 
Nick Coghlan


@subsubheading Overview



This document aims to provide a gentle introduction to the
@ref{9e,,ipaddress} module. It is aimed primarily at users that aren’t
already familiar with IP networking terminology, but may also be useful
to network engineers wanting an overview of how @ref{9e,,ipaddress}
represents IP network addressing concepts.

@menu
* Creating Address/Network/Interface objects:: 
* Inspecting Address/Network/Interface Objects:: 
* Networks as lists of Addresses:: 
* Comparisons: Comparisons<4>. 
* Using IP Addresses with other modules:: 
* Getting more detail when instance creation fails:: 

@end menu

@node Creating Address/Network/Interface objects,Inspecting Address/Network/Interface Objects,,An introduction to the ipaddress module
@anchor{howto/ipaddress creating-address-network-interface-objects}@anchor{4783}
@subsection Creating Address/Network/Interface objects


Since @ref{9e,,ipaddress} is a module for inspecting and manipulating IP addresses,
the first thing you’ll want to do is create some objects.  You can use
@ref{9e,,ipaddress} to create objects from strings and integers.

@menu
* A Note on IP Versions:: 
* IP Host Addresses:: 
* Defining Networks:: 
* Host Interfaces:: 

@end menu

@node A Note on IP Versions,IP Host Addresses,,Creating Address/Network/Interface objects
@anchor{howto/ipaddress a-note-on-ip-versions}@anchor{4784}
@subsubsection A Note on IP Versions


For readers that aren’t particularly familiar with IP addressing, it’s
important to know that the Internet Protocol (IP) is currently in the process
of moving from version 4 of the protocol to version 6. This transition is
occurring largely because version 4 of the protocol doesn’t provide enough
addresses to handle the needs of the whole world, especially given the
increasing number of devices with direct connections to the internet.

Explaining the details of the differences between the two versions of the
protocol is beyond the scope of this introduction, but readers need to at
least be aware that these two versions exist, and it will sometimes be
necessary to force the use of one version or the other.

@node IP Host Addresses,Defining Networks,A Note on IP Versions,Creating Address/Network/Interface objects
@anchor{howto/ipaddress ip-host-addresses}@anchor{4785}
@subsubsection IP Host Addresses


Addresses, often referred to as “host addresses” are the most basic unit
when working with IP addressing. The simplest way to create addresses is
to use the @ref{338d,,ipaddress.ip_address()} factory function, which automatically
determines whether to create an IPv4 or IPv6 address based on the passed in
value:

@example
>>> ipaddress.ip_address('192.0.2.1')
IPv4Address('192.0.2.1')
>>> ipaddress.ip_address('2001:DB8::1')
IPv6Address('2001:db8::1')
@end example

Addresses can also be created directly from integers. Values that will
fit within 32 bits are assumed to be IPv4 addresses:

@example
>>> ipaddress.ip_address(3221225985)
IPv4Address('192.0.2.1')
>>> ipaddress.ip_address(42540766411282592856903984951653826561)
IPv6Address('2001:db8::1')
@end example

To force the use of IPv4 or IPv6 addresses, the relevant classes can be
invoked directly. This is particularly useful to force creation of IPv6
addresses for small integers:

@example
>>> ipaddress.ip_address(1)
IPv4Address('0.0.0.1')
>>> ipaddress.IPv4Address(1)
IPv4Address('0.0.0.1')
>>> ipaddress.IPv6Address(1)
IPv6Address('::1')
@end example

@node Defining Networks,Host Interfaces,IP Host Addresses,Creating Address/Network/Interface objects
@anchor{howto/ipaddress defining-networks}@anchor{4786}
@subsubsection Defining Networks


Host addresses are usually grouped together into IP networks, so
@ref{9e,,ipaddress} provides a way to create, inspect and manipulate network
definitions. IP network objects are constructed from strings that define the
range of host addresses that are part of that network. The simplest form
for that information is a “network address/network prefix” pair, where the
prefix defines the number of leading bits that are compared to determine
whether or not an address is part of the network and the network address
defines the expected value of those bits.

As for addresses, a factory function is provided that determines the correct
IP version automatically:

@example
>>> ipaddress.ip_network('192.0.2.0/24')
IPv4Network('192.0.2.0/24')
>>> ipaddress.ip_network('2001:db8::0/96')
IPv6Network('2001:db8::/96')
@end example

Network objects cannot have any host bits set.  The practical effect of this
is that @code{192.0.2.1/24} does not describe a network.  Such definitions are
referred to as interface objects since the ip-on-a-network notation is
commonly used to describe network interfaces of a computer on a given network
and are described further in the next section.

By default, attempting to create a network object with host bits set will
result in @ref{17c,,ValueError} being raised. To request that the
additional bits instead be coerced to zero, the flag @code{strict=False} can
be passed to the constructor:

@example
>>> ipaddress.ip_network('192.0.2.1/24')
Traceback (most recent call last):
   ...
ValueError: 192.0.2.1/24 has host bits set
>>> ipaddress.ip_network('192.0.2.1/24', strict=False)
IPv4Network('192.0.2.0/24')
@end example

While the string form offers significantly more flexibility, networks can
also be defined with integers, just like host addresses. In this case, the
network is considered to contain only the single address identified by the
integer, so the network prefix includes the entire network address:

@example
>>> ipaddress.ip_network(3221225984)
IPv4Network('192.0.2.0/32')
>>> ipaddress.ip_network(42540766411282592856903984951653826560)
IPv6Network('2001:db8::/128')
@end example

As with addresses, creation of a particular kind of network can be forced
by calling the class constructor directly instead of using the factory
function.

@node Host Interfaces,,Defining Networks,Creating Address/Network/Interface objects
@anchor{howto/ipaddress host-interfaces}@anchor{4787}
@subsubsection Host Interfaces


As mentioned just above, if you need to describe an address on a particular
network, neither the address nor the network classes are sufficient.
Notation like @code{192.0.2.1/24} is commonly used by network engineers and the
people who write tools for firewalls and routers as shorthand for “the host
@code{192.0.2.1} on the network @code{192.0.2.0/24}”, Accordingly, @ref{9e,,ipaddress}
provides a set of hybrid classes that associate an address with a particular
network. The interface for creation is identical to that for defining network
objects, except that the address portion isn’t constrained to being a network
address.

@example
>>> ipaddress.ip_interface('192.0.2.1/24')
IPv4Interface('192.0.2.1/24')
>>> ipaddress.ip_interface('2001:db8::1/96')
IPv6Interface('2001:db8::1/96')
@end example

Integer inputs are accepted (as with networks), and use of a particular IP
version can be forced by calling the relevant constructor directly.

@node Inspecting Address/Network/Interface Objects,Networks as lists of Addresses,Creating Address/Network/Interface objects,An introduction to the ipaddress module
@anchor{howto/ipaddress inspecting-address-network-interface-objects}@anchor{4788}
@subsection Inspecting Address/Network/Interface Objects


You’ve gone to the trouble of creating an IPv(4|6)(Address|Network|Interface)
object, so you probably want to get information about it.  @ref{9e,,ipaddress}
tries to make doing this easy and intuitive.

Extracting the IP version:

@example
>>> addr4 = ipaddress.ip_address('192.0.2.1')
>>> addr6 = ipaddress.ip_address('2001:db8::1')
>>> addr6.version
6
>>> addr4.version
4
@end example

Obtaining the network from an interface:

@example
>>> host4 = ipaddress.ip_interface('192.0.2.1/24')
>>> host4.network
IPv4Network('192.0.2.0/24')
>>> host6 = ipaddress.ip_interface('2001:db8::1/96')
>>> host6.network
IPv6Network('2001:db8::/96')
@end example

Finding out how many individual addresses are in a network:

@example
>>> net4 = ipaddress.ip_network('192.0.2.0/24')
>>> net4.num_addresses
256
>>> net6 = ipaddress.ip_network('2001:db8::0/96')
>>> net6.num_addresses
4294967296
@end example

Iterating through the “usable” addresses on a network:

@example
>>> net4 = ipaddress.ip_network('192.0.2.0/24')
>>> for x in net4.hosts():
...     print(x)  
192.0.2.1
192.0.2.2
192.0.2.3
192.0.2.4
...
192.0.2.252
192.0.2.253
192.0.2.254
@end example

Obtaining the netmask (i.e. set bits corresponding to the network prefix) or
the hostmask (any bits that are not part of the netmask):

@example
>>> net4 = ipaddress.ip_network('192.0.2.0/24')
>>> net4.netmask
IPv4Address('255.255.255.0')
>>> net4.hostmask
IPv4Address('0.0.0.255')
>>> net6 = ipaddress.ip_network('2001:db8::0/96')
>>> net6.netmask
IPv6Address('ffff:ffff:ffff:ffff:ffff:ffff::')
>>> net6.hostmask
IPv6Address('::ffff:ffff')
@end example

Exploding or compressing the address:

@example
>>> addr6.exploded
'2001:0db8:0000:0000:0000:0000:0000:0001'
>>> addr6.compressed
'2001:db8::1'
>>> net6.exploded
'2001:0db8:0000:0000:0000:0000:0000:0000/96'
>>> net6.compressed
'2001:db8::/96'
@end example

While IPv4 doesn’t support explosion or compression, the associated objects
still provide the relevant properties so that version neutral code can
easily ensure the most concise or most verbose form is used for IPv6
addresses while still correctly handling IPv4 addresses.

@node Networks as lists of Addresses,Comparisons<4>,Inspecting Address/Network/Interface Objects,An introduction to the ipaddress module
@anchor{howto/ipaddress networks-as-lists-of-addresses}@anchor{4789}
@subsection Networks as lists of Addresses


It’s sometimes useful to treat networks as lists.  This means it is possible
to index them like this:

@example
>>> net4[1]
IPv4Address('192.0.2.1')
>>> net4[-1]
IPv4Address('192.0.2.255')
>>> net6[1]
IPv6Address('2001:db8::1')
>>> net6[-1]
IPv6Address('2001:db8::ffff:ffff')
@end example

It also means that network objects lend themselves to using the list
membership test syntax like this:

@example
if address in network:
    # do something
@end example

Containment testing is done efficiently based on the network prefix:

@example
>>> addr4 = ipaddress.ip_address('192.0.2.1')
>>> addr4 in ipaddress.ip_network('192.0.2.0/24')
True
>>> addr4 in ipaddress.ip_network('192.0.3.0/24')
False
@end example

@node Comparisons<4>,Using IP Addresses with other modules,Networks as lists of Addresses,An introduction to the ipaddress module
@anchor{howto/ipaddress comparisons}@anchor{478a}
@subsection Comparisons


@ref{9e,,ipaddress} provides some simple, hopefully intuitive ways to compare
objects, where it makes sense:

@example
>>> ipaddress.ip_address('192.0.2.1') < ipaddress.ip_address('192.0.2.2')
True
@end example

A @ref{162,,TypeError} exception is raised if you try to compare objects of
different versions or different types.

@node Using IP Addresses with other modules,Getting more detail when instance creation fails,Comparisons<4>,An introduction to the ipaddress module
@anchor{howto/ipaddress using-ip-addresses-with-other-modules}@anchor{478b}
@subsection Using IP Addresses with other modules


Other modules that use IP addresses (such as @ref{ea,,socket}) usually won’t
accept objects from this module directly. Instead, they must be coerced to
an integer or string that the other module will accept:

@example
>>> addr4 = ipaddress.ip_address('192.0.2.1')
>>> str(addr4)
'192.0.2.1'
>>> int(addr4)
3221225985
@end example

@node Getting more detail when instance creation fails,,Using IP Addresses with other modules,An introduction to the ipaddress module
@anchor{howto/ipaddress getting-more-detail-when-instance-creation-fails}@anchor{478c}
@subsection Getting more detail when instance creation fails


When creating address/network/interface objects using the version-agnostic
factory functions, any errors will be reported as @ref{17c,,ValueError} with
a generic error message that simply says the passed in value was not
recognized as an object of that type. The lack of a specific error is
because it’s necessary to know whether the value is @emph{supposed} to be IPv4
or IPv6 in order to provide more detail on why it has been rejected.

To support use cases where it is useful to have access to this additional
detail, the individual class constructors actually raise the
@ref{17c,,ValueError} subclasses @ref{3395,,ipaddress.AddressValueError} and
@ref{33bb,,ipaddress.NetmaskValueError} to indicate exactly which part of
the definition failed to parse correctly.

The error messages are significantly more detailed when using the
class constructors directly. For example:

@example
>>> ipaddress.ip_address("192.168.0.256")
Traceback (most recent call last):
  ...
ValueError: '192.168.0.256' does not appear to be an IPv4 or IPv6 address
>>> ipaddress.IPv4Address("192.168.0.256")
Traceback (most recent call last):
  ...
ipaddress.AddressValueError: Octet 256 (> 255) not permitted in '192.168.0.256'

>>> ipaddress.ip_network("192.168.0.1/64")
Traceback (most recent call last):
  ...
ValueError: '192.168.0.1/64' does not appear to be an IPv4 or IPv6 network
>>> ipaddress.IPv4Network("192.168.0.1/64")
Traceback (most recent call last):
  ...
ipaddress.NetmaskValueError: '64' is not a valid netmask
@end example

However, both of the module specific exceptions have @ref{17c,,ValueError} as their
parent class, so if you’re not concerned with the particular type of error,
you can still write code like the following:

@example
try:
    network = ipaddress.IPv4Network(address)
except ValueError:
    print('address/netmask is invalid for IPv4:', address)
@end example

@node Argument Clinic How-To,Instrumenting CPython with DTrace and SystemTap,An introduction to the ipaddress module,Python HOWTOs
@anchor{howto/clinic doc}@anchor{478d}@anchor{howto/clinic argument-clinic-how-to}@anchor{478e}
@section Argument Clinic How-To


@*author: 
Larry Hastings


@subsubheading Abstract



Argument Clinic is a preprocessor for CPython C files.
Its purpose is to automate all the boilerplate involved
with writing argument parsing code for “builtins”.
This document shows you how to convert your first C
function to work with Argument Clinic, and then introduces
some advanced topics on Argument Clinic usage.

Currently Argument Clinic is considered internal-only
for CPython.  Its use is not supported for files outside
CPython, and no guarantees are made regarding backwards
compatibility for future versions.  In other words: if you
maintain an external C extension for CPython, you’re welcome
to experiment with Argument Clinic in your own code.  But the
version of Argument Clinic that ships with the next version
of CPython @emph{could} be totally incompatible and break all your code.

@menu
* The Goals Of Argument Clinic:: 
* Basic Concepts And Usage:: 
* Converting Your First Function:: 
* Advanced Topics:: 

@end menu

@node The Goals Of Argument Clinic,Basic Concepts And Usage,,Argument Clinic How-To
@anchor{howto/clinic the-goals-of-argument-clinic}@anchor{478f}
@subsection The Goals Of Argument Clinic


Argument Clinic’s primary goal
is to take over responsibility for all argument parsing code
inside CPython.  This means that, when you convert a function
to work with Argument Clinic, that function should no longer
do any of its own argument parsing—the code generated by
Argument Clinic should be a “black box” to you, where CPython
calls in at the top, and your code gets called at the bottom,
with @code{PyObject *args} (and maybe @code{PyObject *kwargs})
magically converted into the C variables and types you need.

In order for Argument Clinic to accomplish its primary goal,
it must be easy to use.  Currently, working with CPython’s
argument parsing library is a chore, requiring maintaining
redundant information in a surprising number of places.
When you use Argument Clinic, you don’t have to repeat yourself.

Obviously, no one would want to use Argument Clinic unless
it’s solving their problem—and without creating new problems of
its own.
So it’s paramount that Argument Clinic generate correct code.
It’d be nice if the code was faster, too, but at the very least
it should not introduce a major speed regression.  (Eventually Argument
Clinic @emph{should} make a major speedup possible—we could
rewrite its code generator to produce tailor-made argument
parsing code, rather than calling the general-purpose CPython
argument parsing library.  That would make for the fastest
argument parsing possible!)

Additionally, Argument Clinic must be flexible enough to
work with any approach to argument parsing.  Python has
some functions with some very strange parsing behaviors;
Argument Clinic’s goal is to support all of them.

Finally, the original motivation for Argument Clinic was
to provide introspection “signatures” for CPython builtins.
It used to be, the introspection query functions would throw
an exception if you passed in a builtin.  With Argument
Clinic, that’s a thing of the past!

One idea you should keep in mind, as you work with
Argument Clinic: the more information you give it, the
better job it’ll be able to do.
Argument Clinic is admittedly relatively simple right
now.  But as it evolves it will get more sophisticated,
and it should be able to do many interesting and smart
things with all the information you give it.

@node Basic Concepts And Usage,Converting Your First Function,The Goals Of Argument Clinic,Argument Clinic How-To
@anchor{howto/clinic basic-concepts-and-usage}@anchor{4790}
@subsection Basic Concepts And Usage


Argument Clinic ships with CPython; you’ll find it in @code{Tools/clinic/clinic.py}.
If you run that script, specifying a C file as an argument:

@example
$ python3 Tools/clinic/clinic.py foo.c
@end example

Argument Clinic will scan over the file looking for lines that
look exactly like this:

@example
/*[clinic input]
@end example

When it finds one, it reads everything up to a line that looks
exactly like this:

@example
[clinic start generated code]*/
@end example

Everything in between these two lines is input for Argument Clinic.
All of these lines, including the beginning and ending comment
lines, are collectively called an Argument Clinic “block”.

When Argument Clinic parses one of these blocks, it
generates output.  This output is rewritten into the C file
immediately after the block, followed by a comment containing a checksum.
The Argument Clinic block now looks like this:

@example
/*[clinic input]
... clinic input goes here ...
[clinic start generated code]*/
... clinic output goes here ...
/*[clinic end generated code: checksum=...]*/
@end example

If you run Argument Clinic on the same file a second time, Argument Clinic
will discard the old output and write out the new output with a fresh checksum
line.  However, if the input hasn’t changed, the output won’t change either.

You should never modify the output portion of an Argument Clinic block.  Instead,
change the input until it produces the output you want.  (That’s the purpose of the
checksum—to detect if someone changed the output, as these edits would be lost
the next time Argument Clinic writes out fresh output.)

For the sake of clarity, here’s the terminology we’ll use with Argument Clinic:


@itemize *

@item 
The first line of the comment (@code{/*[clinic input]}) is the @emph{start line}.

@item 
The last line of the initial comment (@code{[clinic start generated code]*/}) is the @emph{end line}.

@item 
The last line (@code{/*[clinic end generated code: checksum=...]*/}) is the @emph{checksum line}.

@item 
In between the start line and the end line is the @emph{input}.

@item 
In between the end line and the checksum line is the @emph{output}.

@item 
All the text collectively, from the start line to the checksum line inclusively,
is the @emph{block}.  (A block that hasn’t been successfully processed by Argument
Clinic yet doesn’t have output or a checksum line, but it’s still considered
a block.)
@end itemize

@node Converting Your First Function,Advanced Topics,Basic Concepts And Usage,Argument Clinic How-To
@anchor{howto/clinic converting-your-first-function}@anchor{4791}
@subsection Converting Your First Function


The best way to get a sense of how Argument Clinic works is to
convert a function to work with it.  Here, then, are the bare
minimum steps you’d need to follow to convert a function to
work with Argument Clinic.  Note that for code you plan to
check in to CPython, you really should take the conversion farther,
using some of the advanced concepts you’ll see later on in
the document (like “return converters” and “self converters”).
But we’ll keep it simple for this walkthrough so you can learn.

Let’s dive in!


@enumerate 0

@item 
Make sure you’re working with a freshly updated checkout
of the CPython trunk.

@item 
Find a Python builtin that calls either @ref{37f,,PyArg_ParseTuple()}
or @ref{863,,PyArg_ParseTupleAndKeywords()}, and hasn’t been converted
to work with Argument Clinic yet.
For my example I’m using @code{_pickle.Pickler.dump()}.

@item 
If the call to the @code{PyArg_Parse} function uses any of the
following format units:

@example
O&
O!
es
es#
et
et#
@end example

or if it has multiple calls to @ref{37f,,PyArg_ParseTuple()},
you should choose a different function.  Argument Clinic @emph{does}
support all of these scenarios.  But these are advanced
topics—let’s do something simpler for your first function.

Also, if the function has multiple calls to @ref{37f,,PyArg_ParseTuple()}
or @ref{863,,PyArg_ParseTupleAndKeywords()} where it supports different
types for the same argument, or if the function uses something besides
PyArg_Parse functions to parse its arguments, it probably
isn’t suitable for conversion to Argument Clinic.  Argument Clinic
doesn’t support generic functions or polymorphic parameters.

@item 
Add the following boilerplate above the function, creating our block:

@example
/*[clinic input]
[clinic start generated code]*/
@end example

@item 
Cut the docstring and paste it in between the @code{[clinic]} lines,
removing all the junk that makes it a properly quoted C string.
When you’re done you should have just the text, based at the left
margin, with no line wider than 80 characters.
(Argument Clinic will preserve indents inside the docstring.)

If the old docstring had a first line that looked like a function
signature, throw that line away.  (The docstring doesn’t need it
anymore—when you use @code{help()} on your builtin in the future,
the first line will be built automatically based on the function’s
signature.)

Sample:

@example
/*[clinic input]
Write a pickled representation of obj to the open file.
[clinic start generated code]*/
@end example

@item 
If your docstring doesn’t have a “summary” line, Argument Clinic will
complain.  So let’s make sure it has one.  The “summary” line should
be a paragraph consisting of a single 80-column line
at the beginning of the docstring.

(Our example docstring consists solely of a summary line, so the sample
code doesn’t have to change for this step.)

@item 
Above the docstring, enter the name of the function, followed
by a blank line.  This should be the Python name of the function,
and should be the full dotted path
to the function—it should start with the name of the module,
include any sub-modules, and if the function is a method on
a class it should include the class name too.

Sample:

@example
/*[clinic input]
_pickle.Pickler.dump

Write a pickled representation of obj to the open file.
[clinic start generated code]*/
@end example

@item 
If this is the first time that module or class has been used with Argument
Clinic in this C file,
you must declare the module and/or class.  Proper Argument Clinic hygiene
prefers declaring these in a separate block somewhere near the
top of the C file, in the same way that include files and statics go at
the top.  (In our sample code we’ll just show the two blocks next to
each other.)

The name of the class and module should be the same as the one
seen by Python.  Check the name defined in the @ref{484,,PyModuleDef}
or @ref{5c6,,PyTypeObject} as appropriate.

When you declare a class, you must also specify two aspects of its type
in C: the type declaration you’d use for a pointer to an instance of
this class, and a pointer to the @ref{5c6,,PyTypeObject} for this class.

Sample:

@example
/*[clinic input]
module _pickle
class _pickle.Pickler "PicklerObject *" "&Pickler_Type"
[clinic start generated code]*/

/*[clinic input]
_pickle.Pickler.dump

Write a pickled representation of obj to the open file.
[clinic start generated code]*/
@end example

@item 
Declare each of the parameters to the function.  Each parameter
should get its own line.  All the parameter lines should be
indented from the function name and the docstring.

The general form of these parameter lines is as follows:

@example
name_of_parameter: converter
@end example

If the parameter has a default value, add that after the
converter:

@example
name_of_parameter: converter = default_value
@end example

Argument Clinic’s support for “default values” is quite sophisticated;
please see @ref{4792,,the section below on default values}
for more information.

Add a blank line below the parameters.

What’s a “converter”?  It establishes both the type
of the variable used in C, and the method to convert the Python
value into a C value at runtime.
For now you’re going to use what’s called a “legacy converter”—a
convenience syntax intended to make porting old code into Argument
Clinic easier.

For each parameter, copy the “format unit” for that
parameter from the @code{PyArg_Parse()} format argument and
specify @emph{that} as its converter, as a quoted
string.  (“format unit” is the formal name for the one-to-three
character substring of the @code{format} parameter that tells
the argument parsing function what the type of the variable
is and how to convert it.  For more on format units please
see @ref{381,,Parsing arguments and building values}.)

For multicharacter format units like @code{z#}, use the
entire two-or-three character string.

Sample:

@example
 /*[clinic input]
 module _pickle
 class _pickle.Pickler "PicklerObject *" "&Pickler_Type"
 [clinic start generated code]*/

 /*[clinic input]
 _pickle.Pickler.dump

    obj: 'O'

Write a pickled representation of obj to the open file.
[clinic start generated code]*/
@end example

@item 
If your function has @code{|} in the format string, meaning some
parameters have default values, you can ignore it.  Argument
Clinic infers which parameters are optional based on whether
or not they have default values.

If your function has @code{$} in the format string, meaning it
takes keyword-only arguments, specify @code{*} on a line by
itself before the first keyword-only argument, indented the
same as the parameter lines.

(@code{_pickle.Pickler.dump} has neither, so our sample is unchanged.)

@item 
If the existing C function calls @ref{37f,,PyArg_ParseTuple()}
(as opposed to @ref{863,,PyArg_ParseTupleAndKeywords()}), then all its
arguments are positional-only.

To mark all parameters as positional-only in Argument Clinic,
add a @code{/} on a line by itself after the last parameter,
indented the same as the parameter lines.

Currently this is all-or-nothing; either all parameters are
positional-only, or none of them are.  (In the future Argument
Clinic may relax this restriction.)

Sample:

@example
/*[clinic input]
module _pickle
class _pickle.Pickler "PicklerObject *" "&Pickler_Type"
[clinic start generated code]*/

/*[clinic input]
_pickle.Pickler.dump

    obj: 'O'
    /

Write a pickled representation of obj to the open file.
[clinic start generated code]*/
@end example

@item 
It’s helpful to write a per-parameter docstring for each parameter.
But per-parameter docstrings are optional; you can skip this step
if you prefer.

Here’s how to add a per-parameter docstring.  The first line
of the per-parameter docstring must be indented further than the
parameter definition.  The left margin of this first line establishes
the left margin for the whole per-parameter docstring; all the text
you write will be outdented by this amount.  You can write as much
text as you like, across multiple lines if you wish.

Sample:

@example
/*[clinic input]
module _pickle
class _pickle.Pickler "PicklerObject *" "&Pickler_Type"
[clinic start generated code]*/

/*[clinic input]
_pickle.Pickler.dump

    obj: 'O'
        The object to be pickled.
    /

Write a pickled representation of obj to the open file.
[clinic start generated code]*/
@end example

@item 
Save and close the file, then run @code{Tools/clinic/clinic.py} on
it.  With luck everything worked—your block now has output, and
a @code{.c.h} file has been generated! Reopen the file in your
text editor to see:

@example
/*[clinic input]
_pickle.Pickler.dump

    obj: 'O'
        The object to be pickled.
    /

Write a pickled representation of obj to the open file.
[clinic start generated code]*/

static PyObject *
_pickle_Pickler_dump(PicklerObject *self, PyObject *obj)
/*[clinic end generated code: output=87ecad1261e02ac7 input=552eb1c0f52260d9]*/
@end example

Obviously, if Argument Clinic didn’t produce any output, it’s because
it found an error in your input.  Keep fixing your errors and retrying
until Argument Clinic processes your file without complaint.

For readability, most of the glue code has been generated to a @code{.c.h}
file.  You’ll need to include that in your original @code{.c} file,
typically right after the clinic module block:

@example
#include "clinic/_pickle.c.h"
@end example

@item 
Double-check that the argument-parsing code Argument Clinic generated
looks basically the same as the existing code.

First, ensure both places use the same argument-parsing function.
The existing code must call either
@ref{37f,,PyArg_ParseTuple()} or @ref{863,,PyArg_ParseTupleAndKeywords()};
ensure that the code generated by Argument Clinic calls the
@emph{exact} same function.

Second, the format string passed in to @ref{37f,,PyArg_ParseTuple()} or
@ref{863,,PyArg_ParseTupleAndKeywords()} should be @emph{exactly} the same
as the hand-written one in the existing function, up to the colon
or semi-colon.

(Argument Clinic always generates its format strings
with a @code{:} followed by the name of the function.  If the
existing code’s format string ends with @code{;}, to provide
usage help, this change is harmless—don’t worry about it.)

Third, for parameters whose format units require two arguments
(like a length variable, or an encoding string, or a pointer
to a conversion function), ensure that the second argument is
@emph{exactly} the same between the two invocations.

Fourth, inside the output portion of the block you’ll find a preprocessor
macro defining the appropriate static @ref{1014,,PyMethodDef} structure for
this builtin:

@example
#define __PICKLE_PICKLER_DUMP_METHODDEF    \
@{"dump", (PyCFunction)__pickle_Pickler_dump, METH_O, __pickle_Pickler_dump__doc__@},
@end example

This static structure should be @emph{exactly} the same as the existing static
@ref{1014,,PyMethodDef} structure for this builtin.

If any of these items differ in @emph{any way},
adjust your Argument Clinic function specification and rerun
@code{Tools/clinic/clinic.py} until they @emph{are} the same.

@item 
Notice that the last line of its output is the declaration
of your “impl” function.  This is where the builtin’s implementation goes.
Delete the existing prototype of the function you’re modifying, but leave
the opening curly brace.  Now delete its argument parsing code and the
declarations of all the variables it dumps the arguments into.
Notice how the Python arguments are now arguments to this impl function;
if the implementation used different names for these variables, fix it.

Let’s reiterate, just because it’s kind of weird.  Your code should now
look like this:

@example
static return_type
your_function_impl(...)
/*[clinic end generated code: checksum=...]*/
@{
...
@end example

Argument Clinic generated the checksum line and the function prototype just
above it.  You should write the opening (and closing) curly braces for the
function, and the implementation inside.

Sample:

@example
/*[clinic input]
module _pickle
class _pickle.Pickler "PicklerObject *" "&Pickler_Type"
[clinic start generated code]*/
/*[clinic end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/

/*[clinic input]
_pickle.Pickler.dump

    obj: 'O'
        The object to be pickled.
    /

Write a pickled representation of obj to the open file.
[clinic start generated code]*/

PyDoc_STRVAR(__pickle_Pickler_dump__doc__,
"Write a pickled representation of obj to the open file.\n"
"\n"
...
static PyObject *
_pickle_Pickler_dump_impl(PicklerObject *self, PyObject *obj)
/*[clinic end generated code: checksum=3bd30745bf206a48f8b576a1da3d90f55a0a4187]*/
@{
    /* Check whether the Pickler was initialized correctly (issue3664).
       Developers often forget to call __init__() in their subclasses, which
       would trigger a segfault without this check. */
    if (self->write == NULL) @{
        PyErr_Format(PicklingError,
                     "Pickler.__init__() was not called by %s.__init__()",
                     Py_TYPE(self)->tp_name);
        return NULL;
    @}

    if (_Pickler_ClearBuffer(self) < 0)
        return NULL;

    ...
@end example

@item 
Remember the macro with the @ref{1014,,PyMethodDef} structure for this
function?  Find the existing @ref{1014,,PyMethodDef} structure for this
function and replace it with a reference to the macro.  (If the builtin
is at module scope, this will probably be very near the end of the file;
if the builtin is a class method, this will probably be below but relatively
near to the implementation.)

Note that the body of the macro contains a trailing comma.  So when you
replace the existing static @ref{1014,,PyMethodDef} structure with the macro,
@emph{don’t} add a comma to the end.

Sample:

@example
static struct PyMethodDef Pickler_methods[] = @{
    __PICKLE_PICKLER_DUMP_METHODDEF
    __PICKLE_PICKLER_CLEAR_MEMO_METHODDEF
    @{NULL, NULL@}                /* sentinel */
@};
@end example

@item 
Compile, then run the relevant portions of the regression-test suite.
This change should not introduce any new compile-time warnings or errors,
and there should be no externally-visible change to Python’s behavior.

Well, except for one difference: @code{inspect.signature()} run on your function
should now provide a valid signature!

Congratulations, you’ve ported your first function to work with Argument Clinic!
@end enumerate

@node Advanced Topics,,Converting Your First Function,Argument Clinic How-To
@anchor{howto/clinic advanced-topics}@anchor{4793}
@subsection Advanced Topics


Now that you’ve had some experience working with Argument Clinic, it’s time
for some advanced topics.

@menu
* Symbolic default values:: 
* Renaming the C functions and variables generated by Argument Clinic:: 
* Converting functions using PyArg_UnpackTuple:: 
* Optional Groups:: 
* Using real Argument Clinic converters@comma{} instead of “legacy converters”: Using real Argument Clinic converters instead of “legacy converters”. 
* Py_buffer:: 
* Advanced converters:: 
* Parameter default values:: 
* The NULL default value:: 
* Expressions specified as default values:: 
* Using a return converter:: 
* Cloning existing functions:: 
* Calling Python code:: 
* Using a “self converter”:: 
* Using a “defining class” converter:: 
* Writing a custom converter:: 
* Writing a custom return converter:: 
* METH_O and METH_NOARGS:: 
* tp_new and tp_init functions:: 
* Changing and redirecting Clinic’s output:: 
* The #ifdef trick:: 
* Using Argument Clinic in Python files:: 

@end menu

@node Symbolic default values,Renaming the C functions and variables generated by Argument Clinic,,Advanced Topics
@anchor{howto/clinic symbolic-default-values}@anchor{4794}
@subsubsection Symbolic default values


The default value you provide for a parameter can’t be any arbitrary
expression.  Currently the following are explicitly supported:


@itemize *

@item 
Numeric constants (integer and float)

@item 
String constants

@item 
@code{True}, @code{False}, and @code{None}

@item 
Simple symbolic constants like @code{sys.maxsize}, which must
start with the name of the module
@end itemize

(In the future, this may need to get even more elaborate,
to allow full expressions like @code{CONSTANT - 1}.)

@node Renaming the C functions and variables generated by Argument Clinic,Converting functions using PyArg_UnpackTuple,Symbolic default values,Advanced Topics
@anchor{howto/clinic renaming-the-c-functions-and-variables-generated-by-argument-clinic}@anchor{4795}
@subsubsection Renaming the C functions and variables generated by Argument Clinic


Argument Clinic automatically names the functions it generates for you.
Occasionally this may cause a problem, if the generated name collides with
the name of an existing C function.  There’s an easy solution: override the names
used for the C functions.  Just add the keyword @code{"as"}
to your function declaration line, followed by the function name you wish to use.
Argument Clinic will use that function name for the base (generated) function,
then add @code{"_impl"} to the end and use that for the name of the impl function.

For example, if we wanted to rename the C function names generated for
@code{pickle.Pickler.dump}, it’d look like this:

@example
/*[clinic input]
pickle.Pickler.dump as pickler_dumper

...
@end example

The base function would now be named @code{pickler_dumper()},
and the impl function would now be named @code{pickler_dumper_impl()}.

Similarly, you may have a problem where you want to give a parameter
a specific Python name, but that name may be inconvenient in C.  Argument
Clinic allows you to give a parameter different names in Python and in C,
using the same @code{"as"} syntax:

@example
/*[clinic input]
pickle.Pickler.dump

    obj: object
    file as file_obj: object
    protocol: object = NULL
    *
    fix_imports: bool = True
@end example

Here, the name used in Python (in the signature and the @code{keywords}
array) would be @code{file}, but the C variable would be named @code{file_obj}.

You can use this to rename the @code{self} parameter too!

@node Converting functions using PyArg_UnpackTuple,Optional Groups,Renaming the C functions and variables generated by Argument Clinic,Advanced Topics
@anchor{howto/clinic converting-functions-using-pyarg-unpacktuple}@anchor{4796}
@subsubsection Converting functions using PyArg_UnpackTuple


To convert a function parsing its arguments with @ref{103c,,PyArg_UnpackTuple()},
simply write out all the arguments, specifying each as an @code{object}.  You
may specify the @code{type} argument to cast the type as appropriate.  All
arguments should be marked positional-only (add a @code{/} on a line by itself
after the last argument).

Currently the generated code will use @ref{37f,,PyArg_ParseTuple()}, but this
will change soon.

@node Optional Groups,Using real Argument Clinic converters instead of “legacy converters”,Converting functions using PyArg_UnpackTuple,Advanced Topics
@anchor{howto/clinic optional-groups}@anchor{4797}
@subsubsection Optional Groups


Some legacy functions have a tricky approach to parsing their arguments:
they count the number of positional arguments, then use a @code{switch} statement
to call one of several different @ref{37f,,PyArg_ParseTuple()} calls depending on
how many positional arguments there are.  (These functions cannot accept
keyword-only arguments.)  This approach was used to simulate optional
arguments back before @ref{863,,PyArg_ParseTupleAndKeywords()} was created.

While functions using this approach can often be converted to
use @ref{863,,PyArg_ParseTupleAndKeywords()}, optional arguments, and default values,
it’s not always possible.  Some of these legacy functions have
behaviors @ref{863,,PyArg_ParseTupleAndKeywords()} doesn’t directly support.
The most obvious example is the builtin function @code{range()}, which has
an optional argument on the @emph{left} side of its required argument!
Another example is @code{curses.window.addch()}, which has a group of two
arguments that must always be specified together.  (The arguments are
called @code{x} and @code{y}; if you call the function passing in @code{x},
you must also pass in @code{y}—and if you don’t pass in @code{x} you may not
pass in @code{y} either.)

In any case, the goal of Argument Clinic is to support argument parsing
for all existing CPython builtins without changing their semantics.
Therefore Argument Clinic supports
this alternate approach to parsing, using what are called @emph{optional groups}.
Optional groups are groups of arguments that must all be passed in together.
They can be to the left or the right of the required arguments.  They
can @emph{only} be used with positional-only parameters.

@cartouche
@quotation Note 
Optional groups are @emph{only} intended for use when converting
functions that make multiple calls to @ref{37f,,PyArg_ParseTuple()}!
Functions that use @emph{any} other approach for parsing arguments
should @emph{almost never} be converted to Argument Clinic using
optional groups.  Functions using optional groups currently
cannot have accurate signatures in Python, because Python just
doesn’t understand the concept.  Please avoid using optional
groups wherever possible.
@end quotation
@end cartouche

To specify an optional group, add a @code{[} on a line by itself before
the parameters you wish to group together, and a @code{]} on a line by itself
after these parameters.  As an example, here’s how @code{curses.window.addch}
uses optional groups to make the first two parameters and the last
parameter optional:

@example
/*[clinic input]

curses.window.addch

    [
    x: int
      X-coordinate.
    y: int
      Y-coordinate.
    ]

    ch: object
      Character to add.

    [
    attr: long
      Attributes for the character.
    ]
    /

...
@end example

Notes:


@itemize *

@item 
For every optional group, one additional parameter will be passed into the
impl function representing the group.  The parameter will be an int named
@code{group_@{direction@}_@{number@}},
where @code{@{direction@}} is either @code{right} or @code{left} depending on whether the group
is before or after the required parameters, and @code{@{number@}} is a monotonically
increasing number (starting at 1) indicating how far away the group is from
the required parameters.  When the impl is called, this parameter will be set
to zero if this group was unused, and set to non-zero if this group was used.
(By used or unused, I mean whether or not the parameters received arguments
in this invocation.)

@item 
If there are no required arguments, the optional groups will behave
as if they’re to the right of the required arguments.

@item 
In the case of ambiguity, the argument parsing code
favors parameters on the left (before the required parameters).

@item 
Optional groups can only contain positional-only parameters.

@item 
Optional groups are @emph{only} intended for legacy code.  Please do not
use optional groups for new code.
@end itemize

@node Using real Argument Clinic converters instead of “legacy converters”,Py_buffer,Optional Groups,Advanced Topics
@anchor{howto/clinic using-real-argument-clinic-converters-instead-of-legacy-converters}@anchor{4798}
@subsubsection Using real Argument Clinic converters, instead of “legacy converters”


To save time, and to minimize how much you need to learn
to achieve your first port to Argument Clinic, the walkthrough above tells
you to use “legacy converters”.  “Legacy converters” are a convenience,
designed explicitly to make porting existing code to Argument Clinic
easier.  And to be clear, their use is acceptable when porting code for
Python 3.4.

However, in the long term we probably want all our blocks to
use Argument Clinic’s real syntax for converters.  Why?  A couple
reasons:


@itemize *

@item 
The proper converters are far easier to read and clearer in their intent.

@item 
There are some format units that are unsupported as “legacy converters”,
because they require arguments, and the legacy converter syntax doesn’t
support specifying arguments.

@item 
In the future we may have a new argument parsing library that isn’t
restricted to what @ref{37f,,PyArg_ParseTuple()} supports; this flexibility
won’t be available to parameters using legacy converters.
@end itemize

Therefore, if you don’t mind a little extra effort, please use the normal
converters instead of legacy converters.

In a nutshell, the syntax for Argument Clinic (non-legacy) converters
looks like a Python function call.  However, if there are no explicit
arguments to the function (all functions take their default values),
you may omit the parentheses.  Thus @code{bool} and @code{bool()} are exactly
the same converters.

All arguments to Argument Clinic converters are keyword-only.
All Argument Clinic converters accept the following arguments:

@quotation


@table @asis

@item @code{c_default}

The default value for this parameter when defined in C.
Specifically, this will be the initializer for the variable declared
in the “parse function”.  See @ref{4792,,the section on default values}
for how to use this.
Specified as a string.

@item @code{annotation}

The annotation value for this parameter.  Not currently supported,
because 
@geindex Python Enhancement Proposals; PEP 8
PEP 8@footnote{https://www.python.org/dev/peps/pep-0008} mandates that the Python library may not use
annotations.
@end table
@end quotation

In addition, some converters accept additional arguments.  Here is a list
of these arguments, along with their meanings:

@quotation


@table @asis

@item @code{accept}

A set of Python types (and possibly pseudo-types);
this restricts the allowable Python argument to values of these types.
(This is not a general-purpose facility; as a rule it only supports
specific lists of types as shown in the legacy converter table.)

To accept @code{None}, add @code{NoneType} to this set.

@item @code{bitwise}

Only supported for unsigned integers.  The native integer value of this
Python argument will be written to the parameter without any range checking,
even for negative values.

@item @code{converter}

Only supported by the @code{object} converter.  Specifies the name of a
@ref{4317,,C “converter function”}
to use to convert this object to a native type.

@item @code{encoding}

Only supported for strings.  Specifies the encoding to use when converting
this string from a Python str (Unicode) value into a C @code{char *} value.

@item @code{subclass_of}

Only supported for the @code{object} converter.  Requires that the Python
value be a subclass of a Python type, as expressed in C.

@item @code{type}

Only supported for the @code{object} and @code{self} converters.  Specifies
the C type that will be used to declare the variable.  Default value is
@code{"PyObject *"}.

@item @code{zeroes}

Only supported for strings.  If true, embedded NUL bytes (@code{'\\0'}) are
permitted inside the value.  The length of the string will be passed in
to the impl function, just after the string parameter, as a parameter named
@code{<parameter_name>_length}.
@end table
@end quotation

Please note, not every possible combination of arguments will work.
Usually these arguments are implemented by specific @code{PyArg_ParseTuple}
@emph{format units}, with specific behavior.  For example, currently you cannot
call @code{unsigned_short} without also specifying @code{bitwise=True}.
Although it’s perfectly reasonable to think this would work, these semantics don’t
map to any existing format unit.  So Argument Clinic doesn’t support it.  (Or, at
least, not yet.)

Below is a table showing the mapping of legacy converters into real
Argument Clinic converters.  On the left is the legacy converter,
on the right is the text you’d replace it with.


@multitable {xxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

@code{'B'}

@tab

@code{unsigned_char(bitwise=True)}

@item

@code{'b'}

@tab

@code{unsigned_char}

@item

@code{'c'}

@tab

@code{char}

@item

@code{'C'}

@tab

@code{int(accept=@{str@})}

@item

@code{'d'}

@tab

@code{double}

@item

@code{'D'}

@tab

@code{Py_complex}

@item

@code{'es'}

@tab

@code{str(encoding='name_of_encoding')}

@item

@code{'es#'}

@tab

@code{str(encoding='name_of_encoding', zeroes=True)}

@item

@code{'et'}

@tab

@code{str(encoding='name_of_encoding', accept=@{bytes, bytearray, str@})}

@item

@code{'et#'}

@tab

@code{str(encoding='name_of_encoding', accept=@{bytes, bytearray, str@}, zeroes=True)}

@item

@code{'f'}

@tab

@code{float}

@item

@code{'h'}

@tab

@code{short}

@item

@code{'H'}

@tab

@code{unsigned_short(bitwise=True)}

@item

@code{'i'}

@tab

@code{int}

@item

@code{'I'}

@tab

@code{unsigned_int(bitwise=True)}

@item

@code{'k'}

@tab

@code{unsigned_long(bitwise=True)}

@item

@code{'K'}

@tab

@code{unsigned_long_long(bitwise=True)}

@item

@code{'l'}

@tab

@code{long}

@item

@code{'L'}

@tab

@code{long long}

@item

@code{'n'}

@tab

@code{Py_ssize_t}

@item

@code{'O'}

@tab

@code{object}

@item

@code{'O!'}

@tab

@code{object(subclass_of='&PySomething_Type')}

@item

@code{'O&'}

@tab

@code{object(converter='name_of_c_function')}

@item

@code{'p'}

@tab

@code{bool}

@item

@code{'S'}

@tab

@code{PyBytesObject}

@item

@code{'s'}

@tab

@code{str}

@item

@code{'s#'}

@tab

@code{str(zeroes=True)}

@item

@code{'s*'}

@tab

@code{Py_buffer(accept=@{buffer, str@})}

@item

@code{'U'}

@tab

@code{unicode}

@item

@code{'u'}

@tab

@code{Py_UNICODE}

@item

@code{'u#'}

@tab

@code{Py_UNICODE(zeroes=True)}

@item

@code{'w*'}

@tab

@code{Py_buffer(accept=@{rwbuffer@})}

@item

@code{'Y'}

@tab

@code{PyByteArrayObject}

@item

@code{'y'}

@tab

@code{str(accept=@{bytes@})}

@item

@code{'y#'}

@tab

@code{str(accept=@{robuffer@}, zeroes=True)}

@item

@code{'y*'}

@tab

@code{Py_buffer}

@item

@code{'Z'}

@tab

@code{Py_UNICODE(accept=@{str, NoneType@})}

@item

@code{'Z#'}

@tab

@code{Py_UNICODE(accept=@{str, NoneType@}, zeroes=True)}

@item

@code{'z'}

@tab

@code{str(accept=@{str, NoneType@})}

@item

@code{'z#'}

@tab

@code{str(accept=@{str, NoneType@}, zeroes=True)}

@item

@code{'z*'}

@tab

@code{Py_buffer(accept=@{buffer, str, NoneType@})}

@end multitable


As an example, here’s our sample @code{pickle.Pickler.dump} using the proper
converter:

@example
/*[clinic input]
pickle.Pickler.dump

    obj: object
        The object to be pickled.
    /

Write a pickled representation of obj to the open file.
[clinic start generated code]*/
@end example

One advantage of real converters is that they’re more flexible than legacy
converters.  For example, the @code{unsigned_int} converter (and all the
@code{unsigned_} converters) can be specified without @code{bitwise=True}.  Their
default behavior performs range checking on the value, and they won’t accept
negative numbers.  You just can’t do that with a legacy converter!

Argument Clinic will show you all the converters it has
available.  For each converter it’ll show you all the parameters
it accepts, along with the default value for each parameter.
Just run @code{Tools/clinic/clinic.py --converters} to see the full list.

@node Py_buffer,Advanced converters,Using real Argument Clinic converters instead of “legacy converters”,Advanced Topics
@anchor{howto/clinic py-buffer}@anchor{4799}
@subsubsection Py_buffer


When using the @code{Py_buffer} converter
(or the @code{'s*'}, @code{'w*'}, @code{'*y'}, or @code{'z*'} legacy converters),
you @emph{must} not call @ref{1e5,,PyBuffer_Release()} on the provided buffer.
Argument Clinic generates code that does it for you (in the parsing function).

@node Advanced converters,Parameter default values,Py_buffer,Advanced Topics
@anchor{howto/clinic advanced-converters}@anchor{479a}
@subsubsection Advanced converters


Remember those format units you skipped for your first
time because they were advanced?  Here’s how to handle those too.

The trick is, all those format units take arguments—either
conversion functions, or types, or strings specifying an encoding.
(But “legacy converters” don’t support arguments.  That’s why we
skipped them for your first function.)  The argument you specified
to the format unit is now an argument to the converter; this
argument is either @code{converter} (for @code{O&}), @code{subclass_of} (for @code{O!}),
or @code{encoding} (for all the format units that start with @code{e}).

When using @code{subclass_of}, you may also want to use the other
custom argument for @code{object()}: @code{type}, which lets you set the type
actually used for the parameter.  For example, if you want to ensure
that the object is a subclass of @code{PyUnicode_Type}, you probably want
to use the converter @code{object(type='PyUnicodeObject *', subclass_of='&PyUnicode_Type')}.

One possible problem with using Argument Clinic: it takes away some possible
flexibility for the format units starting with @code{e}.  When writing a
@code{PyArg_Parse} call by hand, you could theoretically decide at runtime what
encoding string to pass in to @ref{37f,,PyArg_ParseTuple()}.   But now this string must
be hard-coded at Argument-Clinic-preprocessing-time.  This limitation is deliberate;
it made supporting this format unit much easier, and may allow for future optimizations.
This restriction doesn’t seem unreasonable; CPython itself always passes in static
hard-coded encoding strings for parameters whose format units start with @code{e}.

@node Parameter default values,The NULL default value,Advanced converters,Advanced Topics
@anchor{howto/clinic default-values}@anchor{4792}@anchor{howto/clinic parameter-default-values}@anchor{479b}
@subsubsection Parameter default values


Default values for parameters can be any of a number of values.
At their simplest, they can be string, int, or float literals:

@example
foo: str = "abc"
bar: int = 123
bat: float = 45.6
@end example

They can also use any of Python’s built-in constants:

@example
yep:  bool = True
nope: bool = False
nada: object = None
@end example

There’s also special support for a default value of @code{NULL}, and
for simple expressions, documented in the following sections.

@node The NULL default value,Expressions specified as default values,Parameter default values,Advanced Topics
@anchor{howto/clinic the-null-default-value}@anchor{479c}
@subsubsection The @code{NULL} default value


For string and object parameters, you can set them to @code{None} to indicate
that there’s no default.  However, that means the C variable will be
initialized to @code{Py_None}.  For convenience’s sakes, there’s a special
value called @code{NULL} for just this reason: from Python’s perspective it
behaves like a default value of @code{None}, but the C variable is initialized
with @code{NULL}.

@node Expressions specified as default values,Using a return converter,The NULL default value,Advanced Topics
@anchor{howto/clinic expressions-specified-as-default-values}@anchor{479d}
@subsubsection Expressions specified as default values


The default value for a parameter can be more than just a literal value.
It can be an entire expression, using math operators and looking up attributes
on objects.  However, this support isn’t exactly simple, because of some
non-obvious semantics.

Consider the following example:

@example
foo: Py_ssize_t = sys.maxsize - 1
@end example

@code{sys.maxsize} can have different values on different platforms.  Therefore
Argument Clinic can’t simply evaluate that expression locally and hard-code it
in C.  So it stores the default in such a way that it will get evaluated at
runtime, when the user asks for the function’s signature.

What namespace is available when the expression is evaluated?  It’s evaluated
in the context of the module the builtin came from.  So, if your module has an
attribute called “@code{max_widgets}”, you may simply use it:

@example
foo: Py_ssize_t = max_widgets
@end example

If the symbol isn’t found in the current module, it fails over to looking in
@code{sys.modules}.  That’s how it can find @code{sys.maxsize} for example.  (Since you
don’t know in advance what modules the user will load into their interpreter,
it’s best to restrict yourself to modules that are preloaded by Python itself.)

Evaluating default values only at runtime means Argument Clinic can’t compute
the correct equivalent C default value.  So you need to tell it explicitly.
When you use an expression, you must also specify the equivalent expression
in C, using the @code{c_default} parameter to the converter:

@example
foo: Py_ssize_t(c_default="PY_SSIZE_T_MAX - 1") = sys.maxsize - 1
@end example

Another complication: Argument Clinic can’t know in advance whether or not the
expression you supply is valid.  It parses it to make sure it looks legal, but
it can’t @emph{actually} know.  You must be very careful when using expressions to
specify values that are guaranteed to be valid at runtime!

Finally, because expressions must be representable as static C values, there
are many restrictions on legal expressions.  Here’s a list of Python features
you’re not permitted to use:


@itemize *

@item 
Function calls.

@item 
Inline if statements (@code{3 if foo else 5}).

@item 
Automatic sequence unpacking (@code{*[1, 2, 3]}).

@item 
List/set/dict comprehensions and generator expressions.

@item 
Tuple/list/set/dict literals.
@end itemize

@node Using a return converter,Cloning existing functions,Expressions specified as default values,Advanced Topics
@anchor{howto/clinic using-a-return-converter}@anchor{479e}
@subsubsection Using a return converter


By default the impl function Argument Clinic generates for you returns @code{PyObject *}.
But your C function often computes some C type, then converts it into the @code{PyObject *}
at the last moment.  Argument Clinic handles converting your inputs from Python types
into native C types—why not have it convert your return value from a native C type
into a Python type too?

That’s what a “return converter” does.  It changes your impl function to return
some C type, then adds code to the generated (non-impl) function to handle converting
that value into the appropriate @code{PyObject *}.

The syntax for return converters is similar to that of parameter converters.
You specify the return converter like it was a return annotation on the
function itself.  Return converters behave much the same as parameter converters;
they take arguments, the arguments are all keyword-only, and if you’re not changing
any of the default arguments you can omit the parentheses.

(If you use both @code{"as"} @emph{and} a return converter for your function,
the @code{"as"} should come before the return converter.)

There’s one additional complication when using return converters: how do you
indicate an error has occurred?  Normally, a function returns a valid (non-@code{NULL})
pointer for success, and @code{NULL} for failure.  But if you use an integer return converter,
all integers are valid.  How can Argument Clinic detect an error?  Its solution: each return
converter implicitly looks for a special value that indicates an error.  If you return
that value, and an error has been set (@code{PyErr_Occurred()} returns a true
value), then the generated code will propagate the error.  Otherwise it will
encode the value you return like normal.

Currently Argument Clinic supports only a few return converters:

@example
bool
int
unsigned int
long
unsigned int
size_t
Py_ssize_t
float
double
DecodeFSDefault
@end example

None of these take parameters.  For the first three, return -1 to indicate
error.  For @code{DecodeFSDefault}, the return type is @code{const char *}; return a @code{NULL}
pointer to indicate an error.

(There’s also an experimental @code{NoneType} converter, which lets you
return @code{Py_None} on success or @code{NULL} on failure, without having
to increment the reference count on @code{Py_None}.  I’m not sure it adds
enough clarity to be worth using.)

To see all the return converters Argument Clinic supports, along with
their parameters (if any),
just run @code{Tools/clinic/clinic.py --converters} for the full list.

@node Cloning existing functions,Calling Python code,Using a return converter,Advanced Topics
@anchor{howto/clinic cloning-existing-functions}@anchor{479f}
@subsubsection Cloning existing functions


If you have a number of functions that look similar, you may be able to
use Clinic’s “clone” feature.  When you clone an existing function,
you reuse:


@itemize *

@item 
its parameters, including


@itemize *

@item 
their names,

@item 
their converters, with all parameters,

@item 
their default values,

@item 
their per-parameter docstrings,

@item 
their @emph{kind} (whether they’re positional only,
positional or keyword, or keyword only), and
@end itemize

@item 
its return converter.
@end itemize

The only thing not copied from the original function is its docstring;
the syntax allows you to specify a new docstring.

Here’s the syntax for cloning a function:

@example
/*[clinic input]
module.class.new_function [as c_basename] = module.class.existing_function

Docstring for new_function goes here.
[clinic start generated code]*/
@end example

(The functions can be in different modules or classes.  I wrote
@code{module.class} in the sample just to illustrate that you must
use the full path to @emph{both} functions.)

Sorry, there’s no syntax for partially-cloning a function, or cloning a function
then modifying it.  Cloning is an all-or nothing proposition.

Also, the function you are cloning from must have been previously defined
in the current file.

@node Calling Python code,Using a “self converter”,Cloning existing functions,Advanced Topics
@anchor{howto/clinic calling-python-code}@anchor{47a0}
@subsubsection Calling Python code


The rest of the advanced topics require you to write Python code
which lives inside your C file and modifies Argument Clinic’s
runtime state.  This is simple: you simply define a Python block.

A Python block uses different delimiter lines than an Argument
Clinic function block.  It looks like this:

@example
/*[python input]
# python code goes here
[python start generated code]*/
@end example

All the code inside the Python block is executed at the
time it’s parsed.  All text written to stdout inside the block
is redirected into the “output” after the block.

As an example, here’s a Python block that adds a static integer
variable to the C code:

@example
/*[python input]
print('static int __ignored_unused_variable__ = 0;')
[python start generated code]*/
static int __ignored_unused_variable__ = 0;
/*[python checksum:...]*/
@end example

@node Using a “self converter”,Using a “defining class” converter,Calling Python code,Advanced Topics
@anchor{howto/clinic using-a-self-converter}@anchor{47a1}
@subsubsection Using a “self converter”


Argument Clinic automatically adds a “self” parameter for you
using a default converter.  It automatically sets the @code{type}
of this parameter to the “pointer to an instance” you specified
when you declared the type.  However, you can override
Argument Clinic’s converter and specify one yourself.
Just add your own @code{self} parameter as the first parameter in a
block, and ensure that its converter is an instance of
@code{self_converter} or a subclass thereof.

What’s the point?  This lets you override the type of @code{self},
or give it a different default name.

How do you specify the custom type you want to cast @code{self} to?
If you only have one or two functions with the same type for @code{self},
you can directly use Argument Clinic’s existing @code{self} converter,
passing in the type you want to use as the @code{type} parameter:

@example
/*[clinic input]

_pickle.Pickler.dump

  self: self(type="PicklerObject *")
  obj: object
  /

Write a pickled representation of the given object to the open file.
[clinic start generated code]*/
@end example

On the other hand, if you have a lot of functions that will use the same
type for @code{self}, it’s best to create your own converter, subclassing
@code{self_converter} but overwriting the @code{type} member:

@example
/*[python input]
class PicklerObject_converter(self_converter):
    type = "PicklerObject *"
[python start generated code]*/

/*[clinic input]

_pickle.Pickler.dump

  self: PicklerObject
  obj: object
  /

Write a pickled representation of the given object to the open file.
[clinic start generated code]*/
@end example

@node Using a “defining class” converter,Writing a custom converter,Using a “self converter”,Advanced Topics
@anchor{howto/clinic using-a-defining-class-converter}@anchor{47a2}
@subsubsection Using a “defining class” converter


Argument Clinic facilitates gaining access to the defining class of a method.
This is useful for @ref{468,,heap type} methods that need to fetch
module level state.  Use @ref{36d,,PyType_FromModuleAndSpec()} to associate a new
heap type with a module.  You can now use @ref{475,,PyType_GetModuleState()} on
the defining class to fetch the module state, for example from a module method.

Example from @code{Modules/zlibmodule.c}.  First, @code{defining_class} is added to
the clinic input:

@example
/*[clinic input]
zlib.Compress.compress

  cls: defining_class
  data: Py_buffer
    Binary data to be compressed.
  /
@end example

After running the Argument Clinic tool, the following function signature is
generated:

@example
/*[clinic start generated code]*/
static PyObject *
zlib_Compress_compress_impl(compobject *self, PyTypeObject *cls,
                            Py_buffer *data)
/*[clinic end generated code: output=6731b3f0ff357ca6 input=04d00f65ab01d260]*/
@end example

The following code can now use @code{PyType_GetModuleState(cls)} to fetch the
module state:

@example
zlibstate *state = PyType_GetModuleState(cls);
@end example

Each method may only have one argument using this converter, and it must appear
after @code{self}, or, if @code{self} is not used, as the first argument.  The argument
will be of type @code{PyTypeObject *}.  The argument will not appear in the
@code{__text_signature__}.

The @code{defining_class} converter is not compatible with @code{__init__} and @code{__new__}
methods, which cannot use the @code{METH_METHOD} convention.

It is not possible to use @code{defining_class} with slot methods.  In order to
fetch the module state from such methods, use @ref{1e9,,PyType_GetModuleByDef()}
to look up the module and then @ref{487,,PyModule_GetState()} to fetch the module
state.  Example from the @code{setattro} slot method in
@code{Modules/_threadmodule.c}:

@example
static int
local_setattro(localobject *self, PyObject *name, PyObject *v)
@{
    PyObject *module = PyType_GetModuleByDef(Py_TYPE(self), &thread_module);
    thread_module_state *state = get_thread_state(module);
    ...
@}
@end example

See also 
@geindex Python Enhancement Proposals; PEP 573
PEP 573@footnote{https://www.python.org/dev/peps/pep-0573}.

@node Writing a custom converter,Writing a custom return converter,Using a “defining class” converter,Advanced Topics
@anchor{howto/clinic writing-a-custom-converter}@anchor{47a3}
@subsubsection Writing a custom converter


As we hinted at in the previous section… you can write your own converters!
A converter is simply a Python class that inherits from @code{CConverter}.
The main purpose of a custom converter is if you have a parameter using
the @code{O&} format unit—parsing this parameter means calling
a @ref{37f,,PyArg_ParseTuple()} “converter function”.

Your converter class should be named @code{*something*_converter}.
If the name follows this convention, then your converter class
will be automatically registered with Argument Clinic; its name
will be the name of your class with the @code{_converter} suffix
stripped off.  (This is accomplished with a metaclass.)

You shouldn’t subclass @code{CConverter.__init__}.  Instead, you should
write a @code{converter_init()} function.  @code{converter_init()}
always accepts a @code{self} parameter; after that, all additional
parameters @emph{must} be keyword-only.  Any arguments passed in to
the converter in Argument Clinic will be passed along to your
@code{converter_init()}.

There are some additional members of @code{CConverter} you may wish
to specify in your subclass.  Here’s the current list:


@table @asis

@item @code{type}

The C type to use for this variable.
@code{type} should be a Python string specifying the type, e.g. @code{int}.
If this is a pointer type, the type string should end with @code{' *'}.

@item @code{default}

The Python default value for this parameter, as a Python value.
Or the magic value @code{unspecified} if there is no default.

@item @code{py_default}

@code{default} as it should appear in Python code,
as a string.
Or @code{None} if there is no default.

@item @code{c_default}

@code{default} as it should appear in C code,
as a string.
Or @code{None} if there is no default.

@item @code{c_ignored_default}

The default value used to initialize the C variable when
there is no default, but not specifying a default may
result in an “uninitialized variable” warning.  This can
easily happen when using option groups—although
properly-written code will never actually use this value,
the variable does get passed in to the impl, and the
C compiler will complain about the “use” of the
uninitialized value.  This value should always be a
non-empty string.

@item @code{converter}

The name of the C converter function, as a string.

@item @code{impl_by_reference}

A boolean value.  If true,
Argument Clinic will add a @code{&} in front of the name of
the variable when passing it into the impl function.

@item @code{parse_by_reference}

A boolean value.  If true,
Argument Clinic will add a @code{&} in front of the name of
the variable when passing it into @ref{37f,,PyArg_ParseTuple()}.
@end table

Here’s the simplest example of a custom converter, from @code{Modules/zlibmodule.c}:

@example
/*[python input]

class ssize_t_converter(CConverter):
    type = 'Py_ssize_t'
    converter = 'ssize_t_converter'

[python start generated code]*/
/*[python end generated code: output=da39a3ee5e6b4b0d input=35521e4e733823c7]*/
@end example

This block adds a converter to Argument Clinic named @code{ssize_t}.  Parameters
declared as @code{ssize_t} will be declared as type @code{Py_ssize_t}, and will
be parsed by the @code{'O&'} format unit, which will call the
@code{ssize_t_converter} converter function.  @code{ssize_t} variables
automatically support default values.

More sophisticated custom converters can insert custom C code to
handle initialization and cleanup.
You can see more examples of custom converters in the CPython
source tree; grep the C files for the string @code{CConverter}.

@node Writing a custom return converter,METH_O and METH_NOARGS,Writing a custom converter,Advanced Topics
@anchor{howto/clinic writing-a-custom-return-converter}@anchor{47a4}
@subsubsection Writing a custom return converter


Writing a custom return converter is much like writing
a custom converter.  Except it’s somewhat simpler, because return
converters are themselves much simpler.

Return converters must subclass @code{CReturnConverter}.
There are no examples yet of custom return converters,
because they are not widely used yet.  If you wish to
write your own return converter, please read @code{Tools/clinic/clinic.py},
specifically the implementation of @code{CReturnConverter} and
all its subclasses.

@node METH_O and METH_NOARGS,tp_new and tp_init functions,Writing a custom return converter,Advanced Topics
@anchor{howto/clinic meth-o-and-meth-noargs}@anchor{47a5}
@subsubsection METH_O and METH_NOARGS


To convert a function using @code{METH_O}, make sure the function’s
single argument is using the @code{object} converter, and mark the
arguments as positional-only:

@example
/*[clinic input]
meth_o_sample

     argument: object
     /
[clinic start generated code]*/
@end example

To convert a function using @code{METH_NOARGS}, just don’t specify
any arguments.

You can still use a self converter, a return converter, and specify
a @code{type} argument to the object converter for @code{METH_O}.

@node tp_new and tp_init functions,Changing and redirecting Clinic’s output,METH_O and METH_NOARGS,Advanced Topics
@anchor{howto/clinic tp-new-and-tp-init-functions}@anchor{47a6}
@subsubsection tp_new and tp_init functions


You can convert @code{tp_new} and @code{tp_init} functions.  Just name
them @code{__new__} or @code{__init__} as appropriate.  Notes:


@itemize *

@item 
The function name generated for @code{__new__} doesn’t end in @code{__new__}
like it would by default.  It’s just the name of the class, converted
into a valid C identifier.

@item 
No @code{PyMethodDef} @code{#define} is generated for these functions.

@item 
@code{__init__} functions return @code{int}, not @code{PyObject *}.

@item 
Use the docstring as the class docstring.

@item 
Although @code{__new__} and @code{__init__} functions must always
accept both the @code{args} and @code{kwargs} objects, when converting
you may specify any signature for these functions that you like.
(If your function doesn’t support keywords, the parsing function
generated will throw an exception if it receives any.)
@end itemize

@node Changing and redirecting Clinic’s output,The #ifdef trick,tp_new and tp_init functions,Advanced Topics
@anchor{howto/clinic changing-and-redirecting-clinic-s-output}@anchor{47a7}
@subsubsection Changing and redirecting Clinic’s output


It can be inconvenient to have Clinic’s output interspersed with
your conventional hand-edited C code.  Luckily, Clinic is configurable:
you can buffer up its output for printing later (or earlier!), or write
its output to a separate file.  You can also add a prefix or suffix to
every line of Clinic’s generated output.

While changing Clinic’s output in this manner can be a boon to readability,
it may result in Clinic code using types before they are defined, or
your code attempting to use Clinic-generated code before it is defined.
These problems can be easily solved by rearranging the declarations in your file,
or moving where Clinic’s generated code goes.  (This is why the default behavior
of Clinic is to output everything into the current block; while many people
consider this hampers readability, it will never require rearranging your
code to fix definition-before-use problems.)

Let’s start with defining some terminology:


@table @asis

@item @emph{field}

A field, in this context, is a subsection of Clinic’s output.
For example, the @code{#define} for the @code{PyMethodDef} structure
is a field, called @code{methoddef_define}.  Clinic has seven
different fields it can output per function definition:

@example
docstring_prototype
docstring_definition
methoddef_define
impl_prototype
parser_prototype
parser_definition
impl_definition
@end example

All the names are of the form @code{"<a>_<b>"},
where @code{"<a>"} is the semantic object represented (the parsing function,
the impl function, the docstring, or the methoddef structure) and @code{"<b>"}
represents what kind of statement the field is.  Field names that end in
@code{"_prototype"}
represent forward declarations of that thing, without the actual body/data
of the thing; field names that end in @code{"_definition"} represent the actual
definition of the thing, with the body/data of the thing.  (@code{"methoddef"}
is special, it’s the only one that ends with @code{"_define"}, representing that
it’s a preprocessor #define.)

@item @emph{destination}

A destination is a place Clinic can write output to.  There are
five built-in destinations:


@table @asis

@item @code{block}

The default destination: printed in the output section of
the current Clinic block.

@item @code{buffer}

A text buffer where you can save text for later.  Text sent
here is appended to the end of any existing text.  It’s an
error to have any text left in the buffer when Clinic finishes
processing a file.

@item @code{file}

A separate “clinic file” that will be created automatically by Clinic.
The filename chosen for the file is @code{@{basename@}.clinic@{extension@}},
where @code{basename} and @code{extension} were assigned the output
from @code{os.path.splitext()} run on the current file.  (Example:
the @code{file} destination for @code{_pickle.c} would be written to
@code{_pickle.clinic.c}.)

@strong{Important: When using a} @code{file} @strong{destination, you}
@emph{must check in} @strong{the generated file!}

@item @code{two-pass}

A buffer like @code{buffer}.  However, a two-pass buffer can only
be dumped once, and it prints out all text sent to it during
all processing, even from Clinic blocks @emph{after} the dumping point.

@item @code{suppress}

The text is suppressed—thrown away.
@end table
@end table

Clinic defines five new directives that let you reconfigure its output.

The first new directive is @code{dump}:

@example
dump <destination>
@end example

This dumps the current contents of the named destination into the output of
the current block, and empties it.  This only works with @code{buffer} and
@code{two-pass} destinations.

The second new directive is @code{output}.  The most basic form of @code{output}
is like this:

@example
output <field> <destination>
@end example

This tells Clinic to output @emph{field} to @emph{destination}.  @code{output} also
supports a special meta-destination, called @code{everything}, which tells
Clinic to output @emph{all} fields to that @emph{destination}.

@code{output} has a number of other functions:

@example
output push
output pop
output preset <preset>
@end example

@code{output push} and @code{output pop} allow you to push and pop
configurations on an internal configuration stack, so that you
can temporarily modify the output configuration, then easily restore
the previous configuration.  Simply push before your change to save
the current configuration, then pop when you wish to restore the
previous configuration.

@code{output preset} sets Clinic’s output to one of several built-in
preset configurations, as follows:

@quotation


@table @asis

@item @code{block}

Clinic’s original starting configuration.  Writes everything
immediately after the input block.

Suppress the @code{parser_prototype}
and @code{docstring_prototype}, write everything else to @code{block}.

@item @code{file}

Designed to write everything to the “clinic file” that it can.
You then @code{#include} this file near the top of your file.
You may need to rearrange your file to make this work, though
usually this just means creating forward declarations for various
@code{typedef} and @code{PyTypeObject} definitions.

Suppress the @code{parser_prototype}
and @code{docstring_prototype}, write the @code{impl_definition} to
@code{block}, and write everything else to @code{file}.

The default filename is @code{"@{dirname@}/clinic/@{basename@}.h"}.

@item @code{buffer}

Save up most of the output from Clinic, to be written into
your file near the end.  For Python files implementing modules
or builtin types, it’s recommended that you dump the buffer
just above the static structures for your module or
builtin type; these are normally very near the end.  Using
@code{buffer} may require even more editing than @code{file}, if
your file has static @code{PyMethodDef} arrays defined in the
middle of the file.

Suppress the @code{parser_prototype}, @code{impl_prototype},
and @code{docstring_prototype}, write the @code{impl_definition} to
@code{block}, and write everything else to @code{file}.

@item @code{two-pass}

Similar to the @code{buffer} preset, but writes forward declarations to
the @code{two-pass} buffer, and definitions to the @code{buffer}.
This is similar to the @code{buffer} preset, but may require
less editing than @code{buffer}.  Dump the @code{two-pass} buffer
near the top of your file, and dump the @code{buffer} near
the end just like you would when using the @code{buffer} preset.

Suppresses the @code{impl_prototype}, write the @code{impl_definition}
to @code{block}, write @code{docstring_prototype}, @code{methoddef_define},
and @code{parser_prototype} to @code{two-pass}, write everything else
to @code{buffer}.

@item @code{partial-buffer}

Similar to the @code{buffer} preset, but writes more things to @code{block},
only writing the really big chunks of generated code to @code{buffer}.
This avoids the definition-before-use problem of @code{buffer} completely,
at the small cost of having slightly more stuff in the block’s output.
Dump the @code{buffer} near the end, just like you would when using
the @code{buffer} preset.

Suppresses the @code{impl_prototype}, write the @code{docstring_definition}
and @code{parser_definition} to @code{buffer}, write everything else to @code{block}.
@end table
@end quotation

The third new directive is @code{destination}:

@example
destination <name> <command> [...]
@end example

This performs an operation on the destination named @code{name}.

There are two defined subcommands: @code{new} and @code{clear}.

The @code{new} subcommand works like this:

@example
destination <name> new <type>
@end example

This creates a new destination with name @code{<name>} and type @code{<type>}.

There are five destination types:

@quotation


@table @asis

@item @code{suppress}

Throws the text away.

@item @code{block}

Writes the text to the current block.  This is what Clinic
originally did.

@item @code{buffer}

A simple text buffer, like the “buffer” builtin destination above.

@item @code{file}

A text file.  The file destination takes an extra argument,
a template to use for building the filename, like so:

@quotation

destination <name> new <type> <file_template>
@end quotation

The template can use three strings internally that will be replaced
by bits of the filename:

@quotation


@table @asis

@item @{path@}

The full path to the file, including directory and full filename.

@item @{dirname@}

The name of the directory the file is in.

@item @{basename@}

Just the name of the file, not including the directory.

@item @{basename_root@}

Basename with the extension clipped off
(everything up to but not including the last ‘.’).

@item @{basename_extension@}

The last ‘.’ and everything after it.  If the basename
does not contain a period, this will be the empty string.
@end table
@end quotation

If there are no periods in the filename, @{basename@} and @{filename@}
are the same, and @{extension@} is empty.  “@{basename@}@{extension@}”
is always exactly the same as “@{filename@}”.”

@item @code{two-pass}

A two-pass buffer, like the “two-pass” builtin destination above.
@end table
@end quotation

The @code{clear} subcommand works like this:

@example
destination <name> clear
@end example

It removes all the accumulated text up to this point in the destination.
(I don’t know what you’d need this for, but I thought maybe it’d be
useful while someone’s experimenting.)

The fourth new directive is @code{set}:

@example
set line_prefix "string"
set line_suffix "string"
@end example

@code{set} lets you set two internal variables in Clinic.
@code{line_prefix} is a string that will be prepended to every line of Clinic’s output;
@code{line_suffix} is a string that will be appended to every line of Clinic’s output.

Both of these support two format strings:

@quotation


@table @asis

@item @code{@{block comment start@}}

Turns into the string @code{/*}, the start-comment text sequence for C files.

@item @code{@{block comment end@}}

Turns into the string @code{*/}, the end-comment text sequence for C files.
@end table
@end quotation

The final new directive is one you shouldn’t need to use directly,
called @code{preserve}:

@example
preserve
@end example

This tells Clinic that the current contents of the output should be kept, unmodified.
This is used internally by Clinic when dumping output into @code{file} files; wrapping
it in a Clinic block lets Clinic use its existing checksum functionality to ensure
the file was not modified by hand before it gets overwritten.

@node The #ifdef trick,Using Argument Clinic in Python files,Changing and redirecting Clinic’s output,Advanced Topics
@anchor{howto/clinic the-ifdef-trick}@anchor{47a8}
@subsubsection The #ifdef trick


If you’re converting a function that isn’t available on all platforms,
there’s a trick you can use to make life a little easier.  The existing
code probably looks like this:

@example
#ifdef HAVE_FUNCTIONNAME
static module_functionname(...)
@{
...
@}
#endif /* HAVE_FUNCTIONNAME */
@end example

And then in the @code{PyMethodDef} structure at the bottom the existing code
will have:

@example
#ifdef HAVE_FUNCTIONNAME
@{'functionname', ... @},
#endif /* HAVE_FUNCTIONNAME */
@end example

In this scenario, you should enclose the body of your impl function inside the @code{#ifdef},
like so:

@example
#ifdef HAVE_FUNCTIONNAME
/*[clinic input]
module.functionname
...
[clinic start generated code]*/
static module_functionname(...)
@{
...
@}
#endif /* HAVE_FUNCTIONNAME */
@end example

Then, remove those three lines from the @code{PyMethodDef} structure,
replacing them with the macro Argument Clinic generated:

@example
MODULE_FUNCTIONNAME_METHODDEF
@end example

(You can find the real name for this macro inside the generated code.
Or you can calculate it yourself: it’s the name of your function as defined
on the first line of your block, but with periods changed to underscores,
uppercased, and @code{"_METHODDEF"} added to the end.)

Perhaps you’re wondering: what if @code{HAVE_FUNCTIONNAME} isn’t defined?
The @code{MODULE_FUNCTIONNAME_METHODDEF} macro won’t be defined either!

Here’s where Argument Clinic gets very clever.  It actually detects that the
Argument Clinic block might be deactivated by the @code{#ifdef}.  When that
happens, it generates a little extra code that looks like this:

@example
#ifndef MODULE_FUNCTIONNAME_METHODDEF
    #define MODULE_FUNCTIONNAME_METHODDEF
#endif /* !defined(MODULE_FUNCTIONNAME_METHODDEF) */
@end example

That means the macro always works.  If the function is defined, this turns
into the correct structure, including the trailing comma.  If the function is
undefined, this turns into nothing.

However, this causes one ticklish problem: where should Argument Clinic put this
extra code when using the “block” output preset?  It can’t go in the output block,
because that could be deactivated by the @code{#ifdef}.  (That’s the whole point!)

In this situation, Argument Clinic writes the extra code to the “buffer” destination.
This may mean that you get a complaint from Argument Clinic:

@example
Warning in file "Modules/posixmodule.c" on line 12357:
Destination buffer 'buffer' not empty at end of file, emptying.
@end example

When this happens, just open your file, find the @code{dump buffer} block that
Argument Clinic added to your file (it’ll be at the very bottom), then
move it above the @code{PyMethodDef} structure where that macro is used.

@node Using Argument Clinic in Python files,,The #ifdef trick,Advanced Topics
@anchor{howto/clinic using-argument-clinic-in-python-files}@anchor{47a9}
@subsubsection Using Argument Clinic in Python files


It’s actually possible to use Argument Clinic to preprocess Python files.
There’s no point to using Argument Clinic blocks, of course, as the output
wouldn’t make any sense to the Python interpreter.  But using Argument Clinic
to run Python blocks lets you use Python as a Python preprocessor!

Since Python comments are different from C comments, Argument Clinic
blocks embedded in Python files look slightly different.  They look like this:

@example
#/*[python input]
#print("def foo(): pass")
#[python start generated code]*/
def foo(): pass
#/*[python checksum:...]*/
@end example

@node Instrumenting CPython with DTrace and SystemTap,Annotations Best Practices,Argument Clinic How-To,Python HOWTOs
@anchor{howto/instrumentation doc}@anchor{47aa}@anchor{howto/instrumentation instrumentation}@anchor{7b2}@anchor{howto/instrumentation instrumenting-cpython-with-dtrace-and-systemtap}@anchor{47ab}
@section Instrumenting CPython with DTrace and SystemTap


@*author: 
David Malcolm


@*author: 
Łukasz Langa


DTrace and SystemTap are monitoring tools, each providing a way to inspect
what the processes on a computer system are doing.  They both use
domain-specific languages allowing a user to write scripts which:

@quotation


@itemize -

@item 
filter which processes are to be observed

@item 
gather data from the processes of interest

@item 
generate reports on the data
@end itemize
@end quotation

As of Python 3.6, CPython can be built with embedded “markers”, also
known as “probes”, that can be observed by a DTrace or SystemTap script,
making it easier to monitor what the CPython processes on a system are
doing.

@strong{CPython implementation detail:} DTrace markers are implementation details of the CPython interpreter.
No guarantees are made about probe compatibility between versions of
CPython. DTrace scripts can stop working or work incorrectly without
warning when changing CPython versions.

@menu
* Enabling the static markers:: 
* Static DTrace probes:: 
* Static SystemTap markers:: 
* Available static markers:: 
* SystemTap Tapsets:: 
* Examples: Examples<39>. 

@end menu

@node Enabling the static markers,Static DTrace probes,,Instrumenting CPython with DTrace and SystemTap
@anchor{howto/instrumentation enabling-the-static-markers}@anchor{47ac}
@subsection Enabling the static markers


macOS comes with built-in support for DTrace.  On Linux, in order to
build CPython with the embedded markers for SystemTap, the SystemTap
development tools must be installed.

On a Linux machine, this can be done via:

@example
$ yum install systemtap-sdt-devel
@end example

or:

@example
$ sudo apt-get install systemtap-sdt-dev
@end example

CPython must then be @ref{165f,,configured with the --with-dtrace option}:

@example
checking for --with-dtrace... yes
@end example

On macOS, you can list available DTrace probes by running a Python
process in the background and listing all probes made available by the
Python provider:

@example
$ python3.6 -q &
$ sudo dtrace -l -P python$!  # or: dtrace -l -m python3.6

   ID   PROVIDER            MODULE                          FUNCTION NAME
29564 python18035        python3.6          _PyEval_EvalFrameDefault function-entry
29565 python18035        python3.6             dtrace_function_entry function-entry
29566 python18035        python3.6          _PyEval_EvalFrameDefault function-return
29567 python18035        python3.6            dtrace_function_return function-return
29568 python18035        python3.6                           collect gc-done
29569 python18035        python3.6                           collect gc-start
29570 python18035        python3.6          _PyEval_EvalFrameDefault line
29571 python18035        python3.6                 maybe_dtrace_line line
@end example

On Linux, you can verify if the SystemTap static markers are present in
the built binary by seeing if it contains a “.note.stapsdt” section.

@example
$ readelf -S ./python | grep .note.stapsdt
[30] .note.stapsdt        NOTE         0000000000000000 00308d78
@end example

If you’ve built Python as a shared library
(with the @ref{2f3,,--enable-shared} configure option), you
need to look instead within the shared library.  For example:

@example
$ readelf -S libpython3.3dm.so.1.0 | grep .note.stapsdt
[29] .note.stapsdt        NOTE         0000000000000000 00365b68
@end example

Sufficiently modern readelf can print the metadata:

@example
$ readelf -n ./python

Displaying notes found at file offset 0x00000254 with length 0x00000020:
    Owner                 Data size          Description
    GNU                  0x00000010          NT_GNU_ABI_TAG (ABI version tag)
        OS: Linux, ABI: 2.6.32

Displaying notes found at file offset 0x00000274 with length 0x00000024:
    Owner                 Data size          Description
    GNU                  0x00000014          NT_GNU_BUILD_ID (unique build ID bitstring)
        Build ID: df924a2b08a7e89f6e11251d4602022977af2670

Displaying notes found at file offset 0x002d6c30 with length 0x00000144:
    Owner                 Data size          Description
    stapsdt              0x00000031          NT_STAPSDT (SystemTap probe descriptors)
        Provider: python
        Name: gc__start
        Location: 0x00000000004371c3, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bf6
        Arguments: -4@@%ebx
    stapsdt              0x00000030          NT_STAPSDT (SystemTap probe descriptors)
        Provider: python
        Name: gc__done
        Location: 0x00000000004374e1, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bf8
        Arguments: -8@@%rax
    stapsdt              0x00000045          NT_STAPSDT (SystemTap probe descriptors)
        Provider: python
        Name: function__entry
        Location: 0x000000000053db6c, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6be8
        Arguments: 8@@%rbp 8@@%r12 -4@@%eax
    stapsdt              0x00000046          NT_STAPSDT (SystemTap probe descriptors)
        Provider: python
        Name: function__return
        Location: 0x000000000053dba8, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bea
        Arguments: 8@@%rbp 8@@%r12 -4@@%eax
@end example

The above metadata contains information for SystemTap describing how it
can patch strategically-placed machine code instructions to enable the
tracing hooks used by a SystemTap script.

@node Static DTrace probes,Static SystemTap markers,Enabling the static markers,Instrumenting CPython with DTrace and SystemTap
@anchor{howto/instrumentation static-dtrace-probes}@anchor{47ad}
@subsection Static DTrace probes


The following example DTrace script can be used to show the call/return
hierarchy of a Python script, only tracing within the invocation of
a function called “start”. In other words, import-time function
invocations are not going to be listed:

@example
self int indent;

python$target:::function-entry
/copyinstr(arg1) == "start"/
@{
        self->trace = 1;
@}

python$target:::function-entry
/self->trace/
@{
        printf("%d\t%*s:", timestamp, 15, probename);
        printf("%*s", self->indent, "");
        printf("%s:%s:%d\n", basename(copyinstr(arg0)), copyinstr(arg1), arg2);
        self->indent++;
@}

python$target:::function-return
/self->trace/
@{
        self->indent--;
        printf("%d\t%*s:", timestamp, 15, probename);
        printf("%*s", self->indent, "");
        printf("%s:%s:%d\n", basename(copyinstr(arg0)), copyinstr(arg1), arg2);
@}

python$target:::function-return
/copyinstr(arg1) == "start"/
@{
        self->trace = 0;
@}
@end example

It can be invoked like this:

@example
$ sudo dtrace -q -s call_stack.d -c "python3.6 script.py"
@end example

The output looks like this:

@example
156641360502280  function-entry:call_stack.py:start:23
156641360518804  function-entry: call_stack.py:function_1:1
156641360532797  function-entry:  call_stack.py:function_3:9
156641360546807 function-return:  call_stack.py:function_3:10
156641360563367 function-return: call_stack.py:function_1:2
156641360578365  function-entry: call_stack.py:function_2:5
156641360591757  function-entry:  call_stack.py:function_1:1
156641360605556  function-entry:   call_stack.py:function_3:9
156641360617482 function-return:   call_stack.py:function_3:10
156641360629814 function-return:  call_stack.py:function_1:2
156641360642285 function-return: call_stack.py:function_2:6
156641360656770  function-entry: call_stack.py:function_3:9
156641360669707 function-return: call_stack.py:function_3:10
156641360687853  function-entry: call_stack.py:function_4:13
156641360700719 function-return: call_stack.py:function_4:14
156641360719640  function-entry: call_stack.py:function_5:18
156641360732567 function-return: call_stack.py:function_5:21
156641360747370 function-return:call_stack.py:start:28
@end example

@node Static SystemTap markers,Available static markers,Static DTrace probes,Instrumenting CPython with DTrace and SystemTap
@anchor{howto/instrumentation static-systemtap-markers}@anchor{47ae}
@subsection Static SystemTap markers


The low-level way to use the SystemTap integration is to use the static
markers directly.  This requires you to explicitly state the binary file
containing them.

For example, this SystemTap script can be used to show the call/return
hierarchy of a Python script:

@example
probe process("python").mark("function__entry") @{
     filename = user_string($arg1);
     funcname = user_string($arg2);
     lineno = $arg3;

     printf("%s => %s in %s:%d\\n",
            thread_indent(1), funcname, filename, lineno);
@}

probe process("python").mark("function__return") @{
    filename = user_string($arg1);
    funcname = user_string($arg2);
    lineno = $arg3;

    printf("%s <= %s in %s:%d\\n",
           thread_indent(-1), funcname, filename, lineno);
@}
@end example

It can be invoked like this:

@example
$ stap \
  show-call-hierarchy.stp \
  -c "./python test.py"
@end example

The output looks like this:

@example
11408 python(8274):        => __contains__ in Lib/_abcoll.py:362
11414 python(8274):         => __getitem__ in Lib/os.py:425
11418 python(8274):          => encode in Lib/os.py:490
11424 python(8274):          <= encode in Lib/os.py:493
11428 python(8274):         <= __getitem__ in Lib/os.py:426
11433 python(8274):        <= __contains__ in Lib/_abcoll.py:366
@end example

where the columns are:

@quotation


@itemize -

@item 
time in microseconds since start of script

@item 
name of executable

@item 
PID of process
@end itemize
@end quotation

and the remainder indicates the call/return hierarchy as the script executes.

For a @ref{2f3,,--enable-shared} build of CPython, the markers are contained within the
libpython shared library, and the probe’s dotted path needs to reflect this. For
example, this line from the above example:

@example
probe process("python").mark("function__entry") @{
@end example

should instead read:

@example
probe process("python").library("libpython3.6dm.so.1.0").mark("function__entry") @{
@end example

(assuming a @ref{338,,debug build} of CPython 3.6)

@node Available static markers,SystemTap Tapsets,Static SystemTap markers,Instrumenting CPython with DTrace and SystemTap
@anchor{howto/instrumentation available-static-markers}@anchor{47af}
@subsection Available static markers



@deffn {Object} function__entry(str filename, str funcname, int lineno)

This marker indicates that execution of a Python function has begun.
It is only triggered for pure-Python (bytecode) functions.

The filename, function name, and line number are provided back to the
tracing script as positional arguments, which must be accessed using
@code{$arg1}, @code{$arg2}, @code{$arg3}:

@quotation


@itemize *

@item 
@code{$arg1} : @code{(const char *)} filename, accessible using @code{user_string($arg1)}

@item 
@code{$arg2} : @code{(const char *)} function name, accessible using
@code{user_string($arg2)}

@item 
@code{$arg3} : @code{int} line number
@end itemize
@end quotation
@end deffn


@deffn {Object} function__return(str filename, str funcname, int lineno)

This marker is the converse of @code{function__entry()}, and indicates that
execution of a Python function has ended (either via @code{return}, or via an
exception).  It is only triggered for pure-Python (bytecode) functions.

The arguments are the same as for @code{function__entry()}
@end deffn


@deffn {Object} line(str filename, str funcname, int lineno)

This marker indicates a Python line is about to be executed.  It is
the equivalent of line-by-line tracing with a Python profiler.  It is
not triggered within C functions.

The arguments are the same as for @code{function__entry()}.
@end deffn


@deffn {Object} gc__start(int generation)

Fires when the Python interpreter starts a garbage collection cycle.
@code{arg0} is the generation to scan, like @ref{54b,,gc.collect()}.
@end deffn


@deffn {Object} gc__done(long collected)

Fires when the Python interpreter finishes a garbage collection
cycle. @code{arg0} is the number of collected objects.
@end deffn


@deffn {Object} import__find__load__start(str modulename)

Fires before @ref{96,,importlib} attempts to find and load the module.
@code{arg0} is the module name.


New in version 3.7.

@end deffn


@deffn {Object} import__find__load__done(str modulename, int found)

Fires after @ref{96,,importlib}’s find_and_load function is called.
@code{arg0} is the module name, @code{arg1} indicates if module was
successfully loaded.


New in version 3.7.

@end deffn


@deffn {Object} audit(str event, void *tuple)

Fires when @ref{11fc,,sys.audit()} or @ref{120d,,PySys_Audit()} is called.
@code{arg0} is the event name as C string, @code{arg1} is a @ref{375,,PyObject}
pointer to a tuple object.


New in version 3.8.

@end deffn

@node SystemTap Tapsets,Examples<39>,Available static markers,Instrumenting CPython with DTrace and SystemTap
@anchor{howto/instrumentation systemtap-tapsets}@anchor{47b0}
@subsection SystemTap Tapsets


The higher-level way to use the SystemTap integration is to use a “tapset”:
SystemTap’s equivalent of a library, which hides some of the lower-level
details of the static markers.

Here is a tapset file, based on a non-shared build of CPython:

@example
/*
   Provide a higher-level wrapping around the function__entry and
   function__return markers:
 \*/
probe python.function.entry = process("python").mark("function__entry")
@{
    filename = user_string($arg1);
    funcname = user_string($arg2);
    lineno = $arg3;
    frameptr = $arg4
@}
probe python.function.return = process("python").mark("function__return")
@{
    filename = user_string($arg1);
    funcname = user_string($arg2);
    lineno = $arg3;
    frameptr = $arg4
@}
@end example

If this file is installed in SystemTap’s tapset directory (e.g.
@code{/usr/share/systemtap/tapset}), then these additional probepoints become
available:


@deffn {Object} python.function.entry(str filename, str funcname, int lineno, frameptr)

This probe point indicates that execution of a Python function has begun.
It is only triggered for pure-Python (bytecode) functions.
@end deffn


@deffn {Object} python.function.return(str filename, str funcname, int lineno, frameptr)

This probe point is the converse of @code{python.function.return}, and
indicates that execution of a Python function has ended (either via
@code{return}, or via an exception).  It is only triggered for pure-Python
(bytecode) functions.
@end deffn

@node Examples<39>,,SystemTap Tapsets,Instrumenting CPython with DTrace and SystemTap
@anchor{howto/instrumentation examples}@anchor{47b1}
@subsection Examples


This SystemTap script uses the tapset above to more cleanly implement the
example given above of tracing the Python function-call hierarchy, without
needing to directly name the static markers:

@example
probe python.function.entry
@{
  printf("%s => %s in %s:%d\n",
         thread_indent(1), funcname, filename, lineno);
@}

probe python.function.return
@{
  printf("%s <= %s in %s:%d\n",
         thread_indent(-1), funcname, filename, lineno);
@}
@end example

The following script uses the tapset above to provide a top-like view of all
running CPython code, showing the top 20 most frequently-entered bytecode
frames, each second, across the whole system:

@example
global fn_calls;

probe python.function.entry
@{
    fn_calls[pid(), filename, funcname, lineno] += 1;
@}

probe timer.ms(1000) @{
    printf("\033[2J\033[1;1H") /* clear screen \*/
    printf("%6s %80s %6s %30s %6s\n",
           "PID", "FILENAME", "LINE", "FUNCTION", "CALLS")
    foreach ([pid, filename, funcname, lineno] in fn_calls- limit 20) @{
        printf("%6d %80s %6d %30s %6d\n",
            pid, filename, lineno, funcname,
            fn_calls[pid, filename, funcname, lineno]);
    @}
    delete fn_calls;
@}
@end example

@node Annotations Best Practices,,Instrumenting CPython with DTrace and SystemTap,Python HOWTOs
@anchor{howto/annotations doc}@anchor{47b2}@anchor{howto/annotations annotations-best-practices}@anchor{47b3}@anchor{howto/annotations annotations-howto}@anchor{255}
@section Annotations Best Practices


@*author: 
Larry Hastings


@subsubheading Abstract



This document is designed to encapsulate the best practices
for working with annotations dicts.  If you write Python code
that examines @code{__annotations__} on Python objects, we
encourage you to follow the guidelines described below.

The document is organized into four sections:
best practices for accessing the annotations of an object
in Python versions 3.10 and newer,
best practices for accessing the annotations of an object
in Python versions 3.9 and older,
other best practices
for @code{__annotations__} that apply to any Python version,
and
quirks of @code{__annotations__}.

Note that this document is specifically about working with
@code{__annotations__}, not uses @emph{for} annotations.
If you’re looking for information on how to use “type hints”
in your code, please see the @ref{120,,typing} module.

@menu
* Accessing The Annotations Dict Of An Object In Python 3.10 And Newer: Accessing The Annotations Dict Of An Object In Python 3 10 And Newer. 
* Accessing The Annotations Dict Of An Object In Python 3.9 And Older: Accessing The Annotations Dict Of An Object In Python 3 9 And Older. 
* Manually Un-Stringizing Stringized Annotations:: 
* Best Practices For __annotations__ In Any Python Version:: 
* __annotations__ Quirks:: 

@end menu

@node Accessing The Annotations Dict Of An Object In Python 3 10 And Newer,Accessing The Annotations Dict Of An Object In Python 3 9 And Older,,Annotations Best Practices
@anchor{howto/annotations accessing-the-annotations-dict-of-an-object-in-python-3-10-and-newer}@anchor{47b4}
@subsection Accessing The Annotations Dict Of An Object In Python 3.10 And Newer


@quotation

Python 3.10 adds a new function to the standard library:
@ref{295,,inspect.get_annotations()}.  In Python versions 3.10
and newer, calling this function is the best practice for
accessing the annotations dict of any object that supports
annotations.  This function can also “un-stringize”
stringized annotations for you.

If for some reason @ref{295,,inspect.get_annotations()} isn’t
viable for your use case, you may access the
@code{__annotations__} data member manually.  Best practice
for this changed in Python 3.10 as well: as of Python 3.10,
@code{o.__annotations__} is guaranteed to @emph{always} work
on Python functions, classes, and modules.  If you’re
certain the object you’re examining is one of these three
@emph{specific} objects, you may simply use @code{o.__annotations__}
to get at the object’s annotations dict.

However, other types of callables–for example,
callables created by @ref{a3f,,functools.partial()}–may
not have an @code{__annotations__} attribute defined.  When
accessing the @code{__annotations__} of a possibly unknown
object,  best practice in Python versions 3.10 and
newer is to call @ref{708,,getattr()} with three arguments,
for example @code{getattr(o, '__annotations__', None)}.
@end quotation

@node Accessing The Annotations Dict Of An Object In Python 3 9 And Older,Manually Un-Stringizing Stringized Annotations,Accessing The Annotations Dict Of An Object In Python 3 10 And Newer,Annotations Best Practices
@anchor{howto/annotations accessing-the-annotations-dict-of-an-object-in-python-3-9-and-older}@anchor{47b5}
@subsection Accessing The Annotations Dict Of An Object In Python 3.9 And Older


@quotation

In Python 3.9 and older, accessing the annotations dict
of an object is much more complicated than in newer versions.
The problem is a design flaw in these older versions of Python,
specifically to do with class annotations.

Best practice for accessing the annotations dict of other
objects–functions, other callables, and modules–is the same
as best practice for 3.10, assuming you aren’t calling
@ref{295,,inspect.get_annotations()}: you should use three-argument
@ref{708,,getattr()} to access the object’s @code{__annotations__}
attribute.

Unfortunately, this isn’t best practice for classes.  The problem
is that, since @code{__annotations__} is optional on classes, and
because classes can inherit attributes from their base classes,
accessing the @code{__annotations__} attribute of a class may
inadvertently return the annotations dict of a @emph{base class.}
As an example:

@example
class Base:
    a: int = 3
    b: str = 'abc'

class Derived(Base):
    pass

print(Derived.__annotations__)
@end example

This will print the annotations dict from @code{Base}, not
@code{Derived}.

Your code will have to have a separate code path if the object
you’re examining is a class (@code{isinstance(o, type)}).
In that case, best practice relies on an implementation detail
of Python 3.9 and before: if a class has annotations defined,
they are stored in the class’s @code{__dict__} dictionary.  Since
the class may or may not have annotations defined, best practice
is to call the @code{get} method on the class dict.

To put it all together, here is some sample code that safely
accesses the @code{__annotations__} attribute on an arbitrary
object in Python 3.9 and before:

@example
if isinstance(o, type):
    ann = o.__dict__.get('__annotations__', None)
else:
    ann = getattr(o, '__annotations__', None)
@end example

After running this code, @code{ann} should be either a
dictionary or @code{None}.  You’re encouraged to double-check
the type of @code{ann} using @ref{236,,isinstance()} before further
examination.

Note that some exotic or malformed type objects may not have
a @code{__dict__} attribute, so for extra safety you may also wish
to use @ref{708,,getattr()} to access @code{__dict__}.
@end quotation

@node Manually Un-Stringizing Stringized Annotations,Best Practices For __annotations__ In Any Python Version,Accessing The Annotations Dict Of An Object In Python 3 9 And Older,Annotations Best Practices
@anchor{howto/annotations manually-un-stringizing-stringized-annotations}@anchor{47b6}
@subsection Manually Un-Stringizing Stringized Annotations


@quotation

In situations where some annotations may be “stringized”,
and you wish to evaluate those strings to produce the
Python values they represent, it really is best to
call @ref{295,,inspect.get_annotations()} to do this work
for you.

If you’re using Python 3.9 or older, or if for some reason
you can’t use @ref{295,,inspect.get_annotations()}, you’ll need
to duplicate its logic.  You’re encouraged to examine the
implementation of @ref{295,,inspect.get_annotations()} in the
current Python version and follow a similar approach.

In a nutshell, if you wish to evaluate a stringized annotation
on an arbitrary object @code{o}:


@itemize *

@item 
If @code{o} is a module, use @code{o.__dict__} as the
@code{globals} when calling @ref{34a,,eval()}.

@item 
If @code{o} is a class, use @code{sys.modules[o.__module__].__dict__}
as the @code{globals}, and @code{dict(vars(o))} as the @code{locals},
when calling @ref{34a,,eval()}.

@item 
If @code{o} is a wrapped callable using @ref{bb3,,functools.update_wrapper()},
@ref{adb,,functools.wraps()}, or @ref{a3f,,functools.partial()}, iteratively
unwrap it by accessing either @code{o.__wrapped__} or @code{o.func} as
appropriate, until you have found the root unwrapped function.

@item 
If @code{o} is a callable (but not a class), use
@code{o.__globals__} as the globals when calling @ref{34a,,eval()}.
@end itemize

However, not all string values used as annotations can
be successfully turned into Python values by @ref{34a,,eval()}.
String values could theoretically contain any valid string,
and in practice there are valid use cases for type hints that
require annotating with string values that specifically
@emph{can’t} be evaluated.  For example:


@itemize *

@item 
@geindex Python Enhancement Proposals; PEP 604
PEP 604@footnote{https://www.python.org/dev/peps/pep-0604} union types using @code{|}, before support for this
was added to Python 3.10.

@item 
Definitions that aren’t needed at runtime, only imported
when @ref{83f,,typing.TYPE_CHECKING} is true.
@end itemize

If @ref{34a,,eval()} attempts to evaluate such values, it will
fail and raise an exception.  So, when designing a library
API that works with annotations, it’s recommended to only
attempt to evaluate string values when explicitly requested
to by the caller.
@end quotation

@node Best Practices For __annotations__ In Any Python Version,__annotations__ Quirks,Manually Un-Stringizing Stringized Annotations,Annotations Best Practices
@anchor{howto/annotations best-practices-for-annotations-in-any-python-version}@anchor{47b7}
@subsection Best Practices For @code{__annotations__} In Any Python Version


@quotation


@itemize *

@item 
You should avoid assigning to the @code{__annotations__} member
of objects directly.  Let Python manage setting @code{__annotations__}.

@item 
If you do assign directly to the @code{__annotations__} member
of an object, you should always set it to a @code{dict} object.

@item 
If you directly access the @code{__annotations__} member
of an object, you should ensure that it’s a
dictionary before attempting to examine its contents.

@item 
You should avoid modifying @code{__annotations__} dicts.

@item 
You should avoid deleting the @code{__annotations__} attribute
of an object.
@end itemize
@end quotation

@node __annotations__ Quirks,,Best Practices For __annotations__ In Any Python Version,Annotations Best Practices
@anchor{howto/annotations annotations-quirks}@anchor{47b8}
@subsection @code{__annotations__} Quirks


@quotation

In all versions of Python 3, function
objects lazy-create an annotations dict if no annotations
are defined on that object.  You can delete the @code{__annotations__}
attribute using @code{del fn.__annotations__}, but if you then
access @code{fn.__annotations__} the object will create a new empty dict
that it will store and return as its annotations.  Deleting the
annotations on a function before it has lazily created its annotations
dict will throw an @code{AttributeError}; using @code{del fn.__annotations__}
twice in a row is guaranteed to always throw an @code{AttributeError}.

Everything in the above paragraph also applies to class and module
objects in Python 3.10 and newer.

In all versions of Python 3, you can set @code{__annotations__}
on a function object to @code{None}.  However, subsequently
accessing the annotations on that object using @code{fn.__annotations__}
will lazy-create an empty dictionary as per the first paragraph of
this section.  This is @emph{not} true of modules and classes, in any Python
version; those objects permit setting @code{__annotations__} to any
Python value, and will retain whatever value is set.

If Python stringizes your annotations for you
(using @code{from __future__ import annotations}), and you
specify a string as an annotation, the string will
itself be quoted.  In effect the annotation is quoted
@emph{twice.}  For example:

@example
from __future__ import annotations
def foo(a: "str"): pass

print(foo.__annotations__)
@end example

This prints @code{@{'a': "'str'"@}}.  This shouldn’t really be considered
a “quirk”; it’s mentioned here simply because it might be surprising.
@end quotation

@node Python Frequently Asked Questions,Glossary,Python HOWTOs,Top
@anchor{faq/index doc}@anchor{47b9}@anchor{faq/index faq-index}@anchor{15db}@anchor{faq/index python-frequently-asked-questions}@anchor{47ba}
@chapter Python Frequently Asked Questions


@menu
* General Python FAQ:: 
* Programming FAQ:: 
* Design and History FAQ:: 
* Library and Extension FAQ:: 
* Extending/Embedding FAQ:: 
* Python on Windows FAQ:: 
* Graphic User Interface FAQ:: 
* “Why is Python Installed on my Computer?” FAQ:: 

@end menu

@node General Python FAQ,Programming FAQ,,Python Frequently Asked Questions
@anchor{faq/general doc}@anchor{47bb}@anchor{faq/general general-python-faq}@anchor{47bc}
@section General Python FAQ



@menu
* General Information:: 
* Python in the real world:: 

@end menu

@node General Information,Python in the real world,,General Python FAQ
@anchor{faq/general general-information}@anchor{47bd}
@subsection General Information


@menu
* What is Python?:: 
* What is the Python Software Foundation?:: 
* Are there copyright restrictions on the use of Python?:: 
* Why was Python created in the first place?:: 
* What is Python good for?:: 
* How does the Python version numbering scheme work?:: 
* How do I obtain a copy of the Python source?:: 
* How do I get documentation on Python?:: 
* I’ve never programmed before. Is there a Python tutorial?: I’ve never programmed before Is there a Python tutorial?. 
* Is there a newsgroup or mailing list devoted to Python?:: 
* How do I get a beta test version of Python?:: 
* How do I submit bug reports and patches for Python?:: 
* Are there any published articles about Python that I can reference?:: 
* Are there any books on Python?:: 
* Where in the world is www.python.org located?: Where in the world is www python org located?. 
* Why is it called Python?:: 
* Do I have to like “Monty Python’s Flying Circus”?:: 

@end menu

@node What is Python?,What is the Python Software Foundation?,,General Information
@anchor{faq/general what-is-python}@anchor{47be}
@subsubsection What is Python?


Python is an interpreted, interactive, object-oriented programming language.  It
incorporates modules, exceptions, dynamic typing, very high level dynamic data
types, and classes.  It supports multiple programming paradigms beyond
object-oriented programming, such as procedural and functional programming.
Python combines remarkable power with very clear syntax. It has interfaces to
many system calls and libraries, as well as to various window systems, and is
extensible in C or C++.  It is also usable as an extension language for
applications that need a programmable interface. Finally, Python is portable:
it runs on many Unix variants including Linux and macOS, and on Windows.

To find out more, start with @ref{14bb,,The Python Tutorial}.  The Beginner's Guide to Python@footnote{https://wiki.python.org/moin/BeginnersGuide} links to other
introductory tutorials and resources for learning Python.

@node What is the Python Software Foundation?,Are there copyright restrictions on the use of Python?,What is Python?,General Information
@anchor{faq/general what-is-the-python-software-foundation}@anchor{47bf}
@subsubsection What is the Python Software Foundation?


The Python Software Foundation is an independent non-profit organization that
holds the copyright on Python versions 2.1 and newer.  The PSF’s mission is to
advance open source technology related to the Python programming language and to
publicize the use of Python.  The PSF’s home page is at
@indicateurl{https://www.python.org/psf/}.

Donations to the PSF are tax-exempt in the US.  If you use Python and find it
helpful, please contribute via the PSF donation page@footnote{https://www.python.org/psf/donations/}.

@node Are there copyright restrictions on the use of Python?,Why was Python created in the first place?,What is the Python Software Foundation?,General Information
@anchor{faq/general are-there-copyright-restrictions-on-the-use-of-python}@anchor{47c0}
@subsubsection Are there copyright restrictions on the use of Python?


You can do anything you want with the source, as long as you leave the
copyrights in and display those copyrights in any documentation about Python
that you produce.  If you honor the copyright rules, it’s OK to use Python for
commercial use, to sell copies of Python in source or binary form (modified or
unmodified), or to sell products that incorporate Python in some form.  We would
still like to know about all commercial use of Python, of course.

See the PSF license page@footnote{https://www.python.org/psf/license/} to find further
explanations and a link to the full text of the license.

The Python logo is trademarked, and in certain cases permission is required to
use it.  Consult the Trademark Usage Policy@footnote{https://www.python.org/psf/trademarks/} for more information.

@node Why was Python created in the first place?,What is Python good for?,Are there copyright restrictions on the use of Python?,General Information
@anchor{faq/general why-was-python-created-in-the-first-place}@anchor{47c1}
@subsubsection Why was Python created in the first place?


Here’s a @emph{very} brief summary of what started it all, written by Guido van
Rossum:

@quotation

I had extensive experience with implementing an interpreted language in the
ABC group at CWI, and from working with this group I had learned a lot about
language design.  This is the origin of many Python features, including the
use of indentation for statement grouping and the inclusion of
very-high-level data types (although the details are all different in
Python).

I had a number of gripes about the ABC language, but also liked many of its
features.  It was impossible to extend the ABC language (or its
implementation) to remedy my complaints – in fact its lack of extensibility
was one of its biggest problems.  I had some experience with using Modula-2+
and talked with the designers of Modula-3 and read the Modula-3 report.
Modula-3 is the origin of the syntax and semantics used for exceptions, and
some other Python features.

I was working in the Amoeba distributed operating system group at CWI.  We
needed a better way to do system administration than by writing either C
programs or Bourne shell scripts, since Amoeba had its own system call
interface which wasn’t easily accessible from the Bourne shell.  My
experience with error handling in Amoeba made me acutely aware of the
importance of exceptions as a programming language feature.

It occurred to me that a scripting language with a syntax like ABC but with
access to the Amoeba system calls would fill the need.  I realized that it
would be foolish to write an Amoeba-specific language, so I decided that I
needed a language that was generally extensible.

During the 1989 Christmas holidays, I had a lot of time on my hand, so I
decided to give it a try.  During the next year, while still mostly working
on it in my own time, Python was used in the Amoeba project with increasing
success, and the feedback from colleagues made me add many early
improvements.

In February 1991, after just over a year of development, I decided to post to
USENET.  The rest is in the @code{Misc/HISTORY} file.
@end quotation

@node What is Python good for?,How does the Python version numbering scheme work?,Why was Python created in the first place?,General Information
@anchor{faq/general what-is-python-good-for}@anchor{47c2}
@subsubsection What is Python good for?


Python is a high-level general-purpose programming language that can be applied
to many different classes of problems.

The language comes with a large standard library that covers areas such as
string processing (regular expressions, Unicode, calculating differences between
files), internet protocols (HTTP, FTP, SMTP, XML-RPC, POP, IMAP, CGI
programming), software engineering (unit testing, logging, profiling, parsing
Python code), and operating system interfaces (system calls, filesystems, TCP/IP
sockets).  Look at the table of contents for @ref{14bc,,The Python Standard Library} to get an idea
of what’s available.  A wide variety of third-party extensions are also
available.  Consult the Python Package Index@footnote{https://pypi.org} to
find packages of interest to you.

@node How does the Python version numbering scheme work?,How do I obtain a copy of the Python source?,What is Python good for?,General Information
@anchor{faq/general how-does-the-python-version-numbering-scheme-work}@anchor{47c3}
@subsubsection How does the Python version numbering scheme work?


Python versions are numbered A.B.C or A.B.  A is the major version number – it
is only incremented for really major changes in the language.  B is the minor
version number, incremented for less earth-shattering changes.  C is the
micro-level – it is incremented for each bugfix release.  See 
@geindex Python Enhancement Proposals; PEP 6
PEP 6@footnote{https://www.python.org/dev/peps/pep-0006} for more
information about bugfix releases.

Not all releases are bugfix releases.  In the run-up to a new major release, a
series of development releases are made, denoted as alpha, beta, or release
candidate.  Alphas are early releases in which interfaces aren’t yet finalized;
it’s not unexpected to see an interface change between two alpha releases.
Betas are more stable, preserving existing interfaces but possibly adding new
modules, and release candidates are frozen, making no changes except as needed
to fix critical bugs.

Alpha, beta and release candidate versions have an additional suffix.  The
suffix for an alpha version is “aN” for some small number N, the suffix for a
beta version is “bN” for some small number N, and the suffix for a release
candidate version is “rcN” for some small number N.  In other words, all versions
labeled 2.0aN precede the versions labeled 2.0bN, which precede versions labeled
2.0rcN, and @emph{those} precede 2.0.

You may also find version numbers with a “+” suffix, e.g. “2.2+”.  These are
unreleased versions, built directly from the CPython development repository.  In
practice, after a final minor release is made, the version is incremented to the
next minor version, which becomes the “a0” version, e.g. “2.4a0”.

See also the documentation for @ref{86c,,sys.version}, @ref{3b72,,sys.hexversion}, and
@ref{d45,,sys.version_info}.

@node How do I obtain a copy of the Python source?,How do I get documentation on Python?,How does the Python version numbering scheme work?,General Information
@anchor{faq/general how-do-i-obtain-a-copy-of-the-python-source}@anchor{47c4}
@subsubsection How do I obtain a copy of the Python source?


The latest Python source distribution is always available from python.org, at
@indicateurl{https://www.python.org/downloads/}.  The latest development sources can be obtained
at @indicateurl{https://github.com/python/cpython/}.

The source distribution is a gzipped tar file containing the complete C source,
Sphinx-formatted documentation, Python library modules, example programs, and
several useful pieces of freely distributable software.  The source will compile
and run out of the box on most UNIX platforms.

Consult the Getting Started section of the Python Developer's Guide@footnote{https://devguide.python.org/setup/} for more
information on getting the source code and compiling it.

@node How do I get documentation on Python?,I’ve never programmed before Is there a Python tutorial?,How do I obtain a copy of the Python source?,General Information
@anchor{faq/general how-do-i-get-documentation-on-python}@anchor{47c5}
@subsubsection How do I get documentation on Python?


@c XXX mention py3k

The standard documentation for the current stable version of Python is available
at @indicateurl{https://docs.python.org/3/}.  PDF, plain text, and downloadable HTML versions are
also available at @indicateurl{https://docs.python.org/3/download.html}.

The documentation is written in reStructuredText and processed by the Sphinx documentation tool@footnote{http://sphinx-doc.org/}.  The reStructuredText source for
the documentation is part of the Python source distribution.

@node I’ve never programmed before Is there a Python tutorial?,Is there a newsgroup or mailing list devoted to Python?,How do I get documentation on Python?,General Information
@anchor{faq/general i-ve-never-programmed-before-is-there-a-python-tutorial}@anchor{47c6}
@subsubsection I’ve never programmed before. Is there a Python tutorial?


There are numerous tutorials and books available.  The standard documentation
includes @ref{14bb,,The Python Tutorial}.

Consult the Beginner's Guide@footnote{https://wiki.python.org/moin/BeginnersGuide} to
find information for beginning Python programmers, including lists of tutorials.

@node Is there a newsgroup or mailing list devoted to Python?,How do I get a beta test version of Python?,I’ve never programmed before Is there a Python tutorial?,General Information
@anchor{faq/general is-there-a-newsgroup-or-mailing-list-devoted-to-python}@anchor{47c7}
@subsubsection Is there a newsgroup or mailing list devoted to Python?


There is a newsgroup, @code{comp.lang.python}, and a mailing list,
python-list@footnote{https://mail.python.org/mailman/listinfo/python-list}.  The
newsgroup and mailing list are gatewayed into each other – if you can read news
it’s unnecessary to subscribe to the mailing list.
@code{comp.lang.python} is high-traffic, receiving hundreds of postings
every day, and Usenet readers are often more able to cope with this volume.

Announcements of new software releases and events can be found in
comp.lang.python.announce, a low-traffic moderated list that receives about five
postings per day.  It’s available as the python-announce mailing list@footnote{https://mail.python.org/mailman/listinfo/python-announce-list}.

More info about other mailing lists and newsgroups
can be found at @indicateurl{https://www.python.org/community/lists/}.

@node How do I get a beta test version of Python?,How do I submit bug reports and patches for Python?,Is there a newsgroup or mailing list devoted to Python?,General Information
@anchor{faq/general how-do-i-get-a-beta-test-version-of-python}@anchor{47c8}
@subsubsection How do I get a beta test version of Python?


Alpha and beta releases are available from @indicateurl{https://www.python.org/downloads/}.  All
releases are announced on the comp.lang.python and comp.lang.python.announce
newsgroups and on the Python home page at @indicateurl{https://www.python.org/}; an RSS feed of
news is available.

You can also access the development version of Python through Git.  See
The Python Developer's Guide@footnote{https://devguide.python.org/} for details.

@node How do I submit bug reports and patches for Python?,Are there any published articles about Python that I can reference?,How do I get a beta test version of Python?,General Information
@anchor{faq/general how-do-i-submit-bug-reports-and-patches-for-python}@anchor{47c9}
@subsubsection How do I submit bug reports and patches for Python?


To report a bug or submit a patch, please use the Roundup installation at
@indicateurl{https://bugs.python.org/}.

You must have a Roundup account to report bugs; this makes it possible for us to
contact you if we have follow-up questions.  It will also enable Roundup to send
you updates as we act on your bug. If you had previously used SourceForge to
report bugs to Python, you can obtain your Roundup password through Roundup’s
password reset procedure@footnote{https://bugs.python.org/user?@@template=forgotten}.

For more information on how Python is developed, consult the Python Developer's Guide@footnote{https://devguide.python.org/}.

@node Are there any published articles about Python that I can reference?,Are there any books on Python?,How do I submit bug reports and patches for Python?,General Information
@anchor{faq/general are-there-any-published-articles-about-python-that-i-can-reference}@anchor{47ca}
@subsubsection Are there any published articles about Python that I can reference?


It’s probably best to cite your favorite book about Python.

The very first article about Python was written in 1991 and is now quite
outdated.

@quotation

Guido van Rossum and Jelke de Boer, “Interactively Testing Remote Servers
Using the Python Programming Language”, CWI Quarterly, Volume 4, Issue 4
(December 1991), Amsterdam, pp 283–303.
@end quotation

@node Are there any books on Python?,Where in the world is www python org located?,Are there any published articles about Python that I can reference?,General Information
@anchor{faq/general are-there-any-books-on-python}@anchor{47cb}
@subsubsection Are there any books on Python?


Yes, there are many, and more are being published.  See the python.org wiki at
@indicateurl{https://wiki.python.org/moin/PythonBooks} for a list.

You can also search online bookstores for “Python” and filter out the Monty
Python references; or perhaps search for “Python” and “language”.

@node Where in the world is www python org located?,Why is it called Python?,Are there any books on Python?,General Information
@anchor{faq/general where-in-the-world-is-www-python-org-located}@anchor{47cc}
@subsubsection Where in the world is www.python.org located?


The Python project’s infrastructure is located all over the world and is managed
by the Python Infrastructure Team. Details here@footnote{http://infra.psf.io}.

@node Why is it called Python?,Do I have to like “Monty Python’s Flying Circus”?,Where in the world is www python org located?,General Information
@anchor{faq/general why-is-it-called-python}@anchor{47cd}
@subsubsection Why is it called Python?


When he began implementing Python, Guido van Rossum was also reading the
published scripts from "Monty Python's Flying Circus"@footnote{https://en.wikipedia.org/wiki/Monty_Python}, a BBC comedy series from the 1970s.  Van Rossum
thought he needed a name that was short, unique, and slightly mysterious, so he
decided to call the language Python.

@node Do I have to like “Monty Python’s Flying Circus”?,,Why is it called Python?,General Information
@anchor{faq/general do-i-have-to-like-monty-python-s-flying-circus}@anchor{47ce}
@subsubsection Do I have to like “Monty Python’s Flying Circus”?


No, but it helps.  :)

@node Python in the real world,,General Information,General Python FAQ
@anchor{faq/general python-in-the-real-world}@anchor{47cf}
@subsection Python in the real world


@menu
* How stable is Python?:: 
* How many people are using Python?:: 
* Have any significant projects been done in Python?:: 
* What new developments are expected for Python in the future?:: 
* Is it reasonable to propose incompatible changes to Python?:: 
* Is Python a good language for beginning programmers?:: 

@end menu

@node How stable is Python?,How many people are using Python?,,Python in the real world
@anchor{faq/general how-stable-is-python}@anchor{47d0}
@subsubsection How stable is Python?


Very stable.  New, stable releases have been coming out roughly every 6 to 18
months since 1991, and this seems likely to continue.  As of version 3.9,
Python will have a major new release every 12 months (
@geindex Python Enhancement Proposals; PEP 602
PEP 602@footnote{https://www.python.org/dev/peps/pep-0602}).

The developers issue “bugfix” releases of older versions, so the stability of
existing releases gradually improves.  Bugfix releases, indicated by a third
component of the version number (e.g. 3.5.3, 3.6.2), are managed for stability;
only fixes for known problems are included in a bugfix release, and it’s
guaranteed that interfaces will remain the same throughout a series of bugfix
releases.

The latest stable releases can always be found on the Python download page@footnote{https://www.python.org/downloads/}.  There are two production-ready versions
of Python: 2.x and 3.x. The recommended version is 3.x, which is supported by
most widely used libraries.  Although 2.x is still widely used, it is not maintained anymore@footnote{https://www.python.org/dev/peps/pep-0373/}.

@node How many people are using Python?,Have any significant projects been done in Python?,How stable is Python?,Python in the real world
@anchor{faq/general how-many-people-are-using-python}@anchor{47d1}
@subsubsection How many people are using Python?


There are probably millions of users, though it’s difficult to obtain an exact
count.

Python is available for free download, so there are no sales figures, and it’s
available from many different sites and packaged with many Linux distributions,
so download statistics don’t tell the whole story either.

The comp.lang.python newsgroup is very active, but not all Python users post to
the group or even read it.

@node Have any significant projects been done in Python?,What new developments are expected for Python in the future?,How many people are using Python?,Python in the real world
@anchor{faq/general have-any-significant-projects-been-done-in-python}@anchor{47d2}
@subsubsection Have any significant projects been done in Python?


See @indicateurl{https://www.python.org/about/success} for a list of projects that use Python.
Consulting the proceedings for past Python conferences@footnote{https://www.python.org/community/workshops/} will reveal contributions from many
different companies and organizations.

High-profile Python projects include the Mailman mailing list manager@footnote{http://www.list.org} and the Zope application server@footnote{http://www.zope.org}.  Several Linux distributions, most notably Red Hat@footnote{https://www.redhat.com}, have written part or all of their installer and
system administration software in Python.  Companies that use Python internally
include Google, Yahoo, and Lucasfilm Ltd.

@node What new developments are expected for Python in the future?,Is it reasonable to propose incompatible changes to Python?,Have any significant projects been done in Python?,Python in the real world
@anchor{faq/general what-new-developments-are-expected-for-python-in-the-future}@anchor{47d3}
@subsubsection What new developments are expected for Python in the future?


See @indicateurl{https://www.python.org/dev/peps/} for the Python Enhancement Proposals
(PEPs). PEPs are design documents describing a suggested new feature for Python,
providing a concise technical specification and a rationale.  Look for a PEP
titled “Python X.Y Release Schedule”, where X.Y is a version that hasn’t been
publicly released yet.

New development is discussed on the python-dev mailing list@footnote{https://mail.python.org/mailman/listinfo/python-dev/}.

@node Is it reasonable to propose incompatible changes to Python?,Is Python a good language for beginning programmers?,What new developments are expected for Python in the future?,Python in the real world
@anchor{faq/general is-it-reasonable-to-propose-incompatible-changes-to-python}@anchor{47d4}
@subsubsection Is it reasonable to propose incompatible changes to Python?


In general, no.  There are already millions of lines of Python code around the
world, so any change in the language that invalidates more than a very small
fraction of existing programs has to be frowned upon.  Even if you can provide a
conversion program, there’s still the problem of updating all documentation;
many books have been written about Python, and we don’t want to invalidate them
all at a single stroke.

Providing a gradual upgrade path is necessary if a feature has to be changed.
@geindex Python Enhancement Proposals; PEP 5
PEP 5@footnote{https://www.python.org/dev/peps/pep-0005} describes the procedure followed for introducing backward-incompatible
changes while minimizing disruption for users.

@node Is Python a good language for beginning programmers?,,Is it reasonable to propose incompatible changes to Python?,Python in the real world
@anchor{faq/general is-python-a-good-language-for-beginning-programmers}@anchor{47d5}
@subsubsection Is Python a good language for beginning programmers?


Yes.

It is still common to start students with a procedural and statically typed
language such as Pascal, C, or a subset of C++ or Java.  Students may be better
served by learning Python as their first language.  Python has a very simple and
consistent syntax and a large standard library and, most importantly, using
Python in a beginning programming course lets students concentrate on important
programming skills such as problem decomposition and data type design.  With
Python, students can be quickly introduced to basic concepts such as loops and
procedures.  They can probably even work with user-defined objects in their very
first course.

For a student who has never programmed before, using a statically typed language
seems unnatural.  It presents additional complexity that the student must master
and slows the pace of the course.  The students are trying to learn to think
like a computer, decompose problems, design consistent interfaces, and
encapsulate data.  While learning to use a statically typed language is
important in the long term, it is not necessarily the best topic to address in
the students’ first programming course.

Many other aspects of Python make it a good first language.  Like Java, Python
has a large standard library so that students can be assigned programming
projects very early in the course that @emph{do} something.  Assignments aren’t
restricted to the standard four-function calculator and check balancing
programs.  By using the standard library, students can gain the satisfaction of
working on realistic applications as they learn the fundamentals of programming.
Using the standard library also teaches students about code reuse.  Third-party
modules such as PyGame are also helpful in extending the students’ reach.

Python’s interactive interpreter enables students to test language features
while they’re programming.  They can keep a window with the interpreter running
while they enter their program’s source in another window.  If they can’t
remember the methods for a list, they can do something like this:

@example
>>> L = []
>>> dir(L) 
['__add__', '__class__', '__contains__', '__delattr__', '__delitem__',
'__dir__', '__doc__', '__eq__', '__format__', '__ge__',
'__getattribute__', '__getitem__', '__gt__', '__hash__', '__iadd__',
'__imul__', '__init__', '__iter__', '__le__', '__len__', '__lt__',
'__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__',
'__sizeof__', '__str__', '__subclasshook__', 'append', 'clear',
'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove',
'reverse', 'sort']
>>> [d for d in dir(L) if '__' not in d]
['append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']

>>> help(L.append)
Help on built-in function append:

append(...)
    L.append(object) -> None -- append object to end

>>> L.append(1)
>>> L
[1]
@end example

With the interpreter, documentation is never far from the student as they are
programming.

There are also good IDEs for Python.  IDLE is a cross-platform IDE for Python
that is written in Python using Tkinter.  PythonWin is a Windows-specific IDE.
Emacs users will be happy to know that there is a very good Python mode for
Emacs.  All of these programming environments provide syntax highlighting,
auto-indenting, and access to the interactive interpreter while coding.  Consult
the Python wiki@footnote{https://wiki.python.org/moin/PythonEditors} for a full list
of Python editing environments.

If you want to discuss Python’s use in education, you may be interested in
joining the edu-sig mailing list@footnote{https://www.python.org/community/sigs/current/edu-sig}.

@node Programming FAQ,Design and History FAQ,General Python FAQ,Python Frequently Asked Questions
@anchor{faq/programming doc}@anchor{47d6}@anchor{faq/programming programming-faq}@anchor{47d7}
@section Programming FAQ



@menu
* General Questions:: 
* Core Language:: 
* Numbers and strings:: 
* Performance: Performance<4>. 
* Sequences (Tuples/Lists): Sequences Tuples/Lists. 
* Objects:: 
* Modules: Modules<3>. 

@end menu

@node General Questions,Core Language,,Programming FAQ
@anchor{faq/programming general-questions}@anchor{47d8}
@subsection General Questions


@menu
* Is there a source code level debugger with breakpoints@comma{} single-stepping@comma{} etc.?: Is there a source code level debugger with breakpoints single-stepping etc ?. 
* Are there tools to help find bugs or perform static analysis?:: 
* How can I create a stand-alone binary from a Python script?:: 
* Are there coding standards or a style guide for Python programs?:: 

@end menu

@node Is there a source code level debugger with breakpoints single-stepping etc ?,Are there tools to help find bugs or perform static analysis?,,General Questions
@anchor{faq/programming is-there-a-source-code-level-debugger-with-breakpoints-single-stepping-etc}@anchor{47d9}
@subsubsection Is there a source code level debugger with breakpoints, single-stepping, etc.?


Yes.

Several debuggers for Python are described below, and the built-in function
@ref{5db,,breakpoint()} allows you to drop into any of them.

The pdb module is a simple but adequate console-mode debugger for Python. It is
part of the standard Python library, and is @ref{c4,,documented in the Library Reference Manual}. You can also write your own debugger by using the code
for pdb as an example.

The IDLE interactive development environment, which is part of the standard
Python distribution (normally available as Tools/scripts/idle), includes a
graphical debugger.

PythonWin is a Python IDE that includes a GUI debugger based on pdb.  The
PythonWin debugger colors breakpoints and has quite a few cool features such as
debugging non-PythonWin programs.  PythonWin is available as part of
pywin32@footnote{https://github.com/mhammond/pywin32} project and
as a part of the
ActivePython@footnote{https://www.activestate.com/products/python/} distribution.

Eric@footnote{http://eric-ide.python-projects.org/} is an IDE built on PyQt
and the Scintilla editing component.

trepan3k@footnote{https://github.com/rocky/python3-trepan/} is a gdb-like debugger.

Visual Studio Code@footnote{https://code.visualstudio.com/} is an IDE with debugging
tools that integrates with version-control software.

There are a number of commercial Python IDEs that include graphical debuggers.
They include:


@itemize *

@item 
Wing IDE@footnote{https://wingware.com/}

@item 
Komodo IDE@footnote{https://www.activestate.com/products/komodo-ide/}

@item 
PyCharm@footnote{https://www.jetbrains.com/pycharm/}
@end itemize

@node Are there tools to help find bugs or perform static analysis?,How can I create a stand-alone binary from a Python script?,Is there a source code level debugger with breakpoints single-stepping etc ?,General Questions
@anchor{faq/programming are-there-tools-to-help-find-bugs-or-perform-static-analysis}@anchor{47da}
@subsubsection Are there tools to help find bugs or perform static analysis?


Yes.

Pylint@footnote{https://www.pylint.org/} and
Pyflakes@footnote{https://github.com/PyCQA/pyflakes} do basic checking that will
help you catch bugs sooner.

Static type checkers such as Mypy@footnote{http://mypy-lang.org/},
Pyre@footnote{https://pyre-check.org/}, and
Pytype@footnote{https://github.com/google/pytype} can check type hints in Python
source code.

@node How can I create a stand-alone binary from a Python script?,Are there coding standards or a style guide for Python programs?,Are there tools to help find bugs or perform static analysis?,General Questions
@anchor{faq/programming faq-create-standalone-binary}@anchor{47db}@anchor{faq/programming how-can-i-create-a-stand-alone-binary-from-a-python-script}@anchor{47dc}
@subsubsection How can I create a stand-alone binary from a Python script?


You don’t need the ability to compile Python to C code if all you want is a
stand-alone program that users can download and run without having to install
the Python distribution first.  There are a number of tools that determine the
set of modules required by a program and bind these modules together with a
Python binary to produce a single executable.

One is to use the freeze tool, which is included in the Python source tree as
@code{Tools/freeze}. It converts Python byte code to C arrays; a C compiler you can
embed all your modules into a new program, which is then linked with the
standard Python modules.

It works by scanning your source recursively for import statements (in both
forms) and looking for the modules in the standard Python path as well as in the
source directory (for built-in modules).  It then turns the bytecode for modules
written in Python into C code (array initializers that can be turned into code
objects using the marshal module) and creates a custom-made config file that
only contains those built-in modules which are actually used in the program.  It
then compiles the generated C code and links it with the rest of the Python
interpreter to form a self-contained binary which acts exactly like your script.

The following packages can help with the creation of console and GUI
executables:


@itemize *

@item 
Nuitka@footnote{https://nuitka.net/} (Cross-platform)

@item 
PyInstaller@footnote{http://www.pyinstaller.org/} (Cross-platform)

@item 
PyOxidizer@footnote{https://pyoxidizer.readthedocs.io/en/stable/} (Cross-platform)

@item 
cx_Freeze@footnote{https://marcelotduarte.github.io/cx_Freeze/} (Cross-platform)

@item 
py2app@footnote{https://github.com/ronaldoussoren/py2app} (macOS only)

@item 
py2exe@footnote{http://www.py2exe.org/} (Windows only)
@end itemize

@node Are there coding standards or a style guide for Python programs?,,How can I create a stand-alone binary from a Python script?,General Questions
@anchor{faq/programming are-there-coding-standards-or-a-style-guide-for-python-programs}@anchor{47dd}
@subsubsection Are there coding standards or a style guide for Python programs?


Yes.  The coding style required for standard library modules is documented as
@geindex Python Enhancement Proposals; PEP 8
PEP 8@footnote{https://www.python.org/dev/peps/pep-0008}.

@node Core Language,Numbers and strings,General Questions,Programming FAQ
@anchor{faq/programming core-language}@anchor{47de}
@subsection Core Language


@menu
* Why am I getting an UnboundLocalError when the variable has a value?:: 
* What are the rules for local and global variables in Python?:: 
* Why do lambdas defined in a loop with different values all return the same result?:: 
* How do I share global variables across modules?:: 
* What are the “best practices” for using import in a module?:: 
* Why are default values shared between objects?:: 
* How can I pass optional or keyword parameters from one function to another?:: 
* What is the difference between arguments and parameters?:: 
* Why did changing list ‘y’ also change list ‘x’?:: 
* How do I write a function with output parameters (call by reference)?: How do I write a function with output parameters call by reference ?. 
* How do you make a higher order function in Python?:: 
* How do I copy an object in Python?:: 
* How can I find the methods or attributes of an object?:: 
* How can my code discover the name of an object?:: 
* What’s up with the comma operator’s precedence?:: 
* Is there an equivalent of C’s “?;” ternary operator?: Is there an equivalent of C’s “? ” ternary operator?. 
* Is it possible to write obfuscated one-liners in Python?:: 
* What does the slash(/) in the parameter list of a function mean?: What does the slash / in the parameter list of a function mean?. 

@end menu

@node Why am I getting an UnboundLocalError when the variable has a value?,What are the rules for local and global variables in Python?,,Core Language
@anchor{faq/programming why-am-i-getting-an-unboundlocalerror-when-the-variable-has-a-value}@anchor{47df}
@subsubsection Why am I getting an UnboundLocalError when the variable has a value?


It can be a surprise to get the UnboundLocalError in previously working
code when it is modified by adding an assignment statement somewhere in
the body of a function.

This code:

@example
>>> x = 10
>>> def bar():
...     print(x)
>>> bar()
10
@end example

works, but this code:

@example
>>> x = 10
>>> def foo():
...     print(x)
...     x += 1
@end example

results in an UnboundLocalError:

@example
>>> foo()
Traceback (most recent call last):
  ...
UnboundLocalError: local variable 'x' referenced before assignment
@end example

This is because when you make an assignment to a variable in a scope, that
variable becomes local to that scope and shadows any similarly named variable
in the outer scope.  Since the last statement in foo assigns a new value to
@code{x}, the compiler recognizes it as a local variable.  Consequently when the
earlier @code{print(x)} attempts to print the uninitialized local variable and
an error results.

In the example above you can access the outer scope variable by declaring it
global:

@example
>>> x = 10
>>> def foobar():
...     global x
...     print(x)
...     x += 1
>>> foobar()
10
@end example

This explicit declaration is required in order to remind you that (unlike the
superficially analogous situation with class and instance variables) you are
actually modifying the value of the variable in the outer scope:

@example
>>> print(x)
11
@end example

You can do a similar thing in a nested scope using the @ref{e4d,,nonlocal}
keyword:

@example
>>> def foo():
...    x = 10
...    def bar():
...        nonlocal x
...        print(x)
...        x += 1
...    bar()
...    print(x)
>>> foo()
10
11
@end example

@node What are the rules for local and global variables in Python?,Why do lambdas defined in a loop with different values all return the same result?,Why am I getting an UnboundLocalError when the variable has a value?,Core Language
@anchor{faq/programming what-are-the-rules-for-local-and-global-variables-in-python}@anchor{47e0}
@subsubsection What are the rules for local and global variables in Python?


In Python, variables that are only referenced inside a function are implicitly
global.  If a variable is assigned a value anywhere within the function’s body,
it’s assumed to be a local unless explicitly declared as global.

Though a bit surprising at first, a moment’s consideration explains this.  On
one hand, requiring @ref{1504,,global} for assigned variables provides a bar
against unintended side-effects.  On the other hand, if @code{global} was required
for all global references, you’d be using @code{global} all the time.  You’d have
to declare as global every reference to a built-in function or to a component of
an imported module.  This clutter would defeat the usefulness of the @code{global}
declaration for identifying side-effects.

@node Why do lambdas defined in a loop with different values all return the same result?,How do I share global variables across modules?,What are the rules for local and global variables in Python?,Core Language
@anchor{faq/programming why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result}@anchor{47e1}
@subsubsection Why do lambdas defined in a loop with different values all return the same result?


Assume you use a for loop to define a few different lambdas (or even plain
functions), e.g.:

@example
>>> squares = []
>>> for x in range(5):
...     squares.append(lambda: x**2)
@end example

This gives you a list that contains 5 lambdas that calculate @code{x**2}.  You
might expect that, when called, they would return, respectively, @code{0}, @code{1},
@code{4}, @code{9}, and @code{16}.  However, when you actually try you will see that
they all return @code{16}:

@example
>>> squares[2]()
16
>>> squares[4]()
16
@end example

This happens because @code{x} is not local to the lambdas, but is defined in
the outer scope, and it is accessed when the lambda is called — not when it
is defined.  At the end of the loop, the value of @code{x} is @code{4}, so all the
functions now return @code{4**2}, i.e. @code{16}.  You can also verify this by
changing the value of @code{x} and see how the results of the lambdas change:

@example
>>> x = 8
>>> squares[2]()
64
@end example

In order to avoid this, you need to save the values in variables local to the
lambdas, so that they don’t rely on the value of the global @code{x}:

@example
>>> squares = []
>>> for x in range(5):
...     squares.append(lambda n=x: n**2)
@end example

Here, @code{n=x} creates a new variable @code{n} local to the lambda and computed
when the lambda is defined so that it has the same value that @code{x} had at
that point in the loop.  This means that the value of @code{n} will be @code{0}
in the first lambda, @code{1} in the second, @code{2} in the third, and so on.
Therefore each lambda will now return the correct result:

@example
>>> squares[2]()
4
>>> squares[4]()
16
@end example

Note that this behaviour is not peculiar to lambdas, but applies to regular
functions too.

@node How do I share global variables across modules?,What are the “best practices” for using import in a module?,Why do lambdas defined in a loop with different values all return the same result?,Core Language
@anchor{faq/programming how-do-i-share-global-variables-across-modules}@anchor{47e2}
@subsubsection How do I share global variables across modules?


The canonical way to share information across modules within a single program is
to create a special module (often called config or cfg).  Just import the config
module in all modules of your application; the module then becomes available as
a global name.  Because there is only one instance of each module, any changes
made to the module object get reflected everywhere.  For example:

config.py:

@example
x = 0   # Default value of the 'x' configuration setting
@end example

mod.py:

@example
import config
config.x = 1
@end example

main.py:

@example
import config
import mod
print(config.x)
@end example

Note that using a module is also the basis for implementing the Singleton design
pattern, for the same reason.

@node What are the “best practices” for using import in a module?,Why are default values shared between objects?,How do I share global variables across modules?,Core Language
@anchor{faq/programming what-are-the-best-practices-for-using-import-in-a-module}@anchor{47e3}
@subsubsection What are the “best practices” for using import in a module?


In general, don’t use @code{from modulename import *}.  Doing so clutters the
importer’s namespace, and makes it much harder for linters to detect undefined
names.

Import modules at the top of a file.  Doing so makes it clear what other modules
your code requires and avoids questions of whether the module name is in scope.
Using one import per line makes it easy to add and delete module imports, but
using multiple imports per line uses less screen space.

It’s good practice if you import modules in the following order:


@enumerate 

@item 
standard library modules – e.g. @code{sys}, @code{os}, @code{getopt}, @code{re}

@item 
third-party library modules (anything installed in Python’s site-packages
directory) – e.g. mx.DateTime, ZODB, PIL.Image, etc.

@item 
locally-developed modules
@end enumerate

It is sometimes necessary to move imports to a function or class to avoid
problems with circular imports.  Gordon McMillan says:

@quotation

Circular imports are fine where both modules use the “import <module>” form
of import.  They fail when the 2nd module wants to grab a name out of the
first (“from module import name”) and the import is at the top level.  That’s
because names in the 1st are not yet available, because the first module is
busy importing the 2nd.
@end quotation

In this case, if the second module is only used in one function, then the import
can easily be moved into that function.  By the time the import is called, the
first module will have finished initializing, and the second module can do its
import.

It may also be necessary to move imports out of the top level of code if some of
the modules are platform-specific.  In that case, it may not even be possible to
import all of the modules at the top of the file.  In this case, importing the
correct modules in the corresponding platform-specific code is a good option.

Only move imports into a local scope, such as inside a function definition, if
it’s necessary to solve a problem such as avoiding a circular import or are
trying to reduce the initialization time of a module.  This technique is
especially helpful if many of the imports are unnecessary depending on how the
program executes.  You may also want to move imports into a function if the
modules are only ever used in that function.  Note that loading a module the
first time may be expensive because of the one time initialization of the
module, but loading a module multiple times is virtually free, costing only a
couple of dictionary lookups.  Even if the module name has gone out of scope,
the module is probably available in @ref{11f7,,sys.modules}.

@node Why are default values shared between objects?,How can I pass optional or keyword parameters from one function to another?,What are the “best practices” for using import in a module?,Core Language
@anchor{faq/programming why-are-default-values-shared-between-objects}@anchor{47e4}
@subsubsection Why are default values shared between objects?


This type of bug commonly bites neophyte programmers.  Consider this function:

@example
def foo(mydict=@{@}):  # Danger: shared reference to one dict for all calls
    ... compute something ...
    mydict[key] = value
    return mydict
@end example

The first time you call this function, @code{mydict} contains a single item.  The
second time, @code{mydict} contains two items because when @code{foo()} begins
executing, @code{mydict} starts out with an item already in it.

It is often expected that a function call creates new objects for default
values. This is not what happens. Default values are created exactly once, when
the function is defined.  If that object is changed, like the dictionary in this
example, subsequent calls to the function will refer to this changed object.

By definition, immutable objects such as numbers, strings, tuples, and @code{None},
are safe from change. Changes to mutable objects such as dictionaries, lists,
and class instances can lead to confusion.

Because of this feature, it is good programming practice to not use mutable
objects as default values.  Instead, use @code{None} as the default value and
inside the function, check if the parameter is @code{None} and create a new
list/dictionary/whatever if it is.  For example, don’t write:

@example
def foo(mydict=@{@}):
    ...
@end example

but:

@example
def foo(mydict=None):
    if mydict is None:
        mydict = @{@}  # create a new dict for local namespace
@end example

This feature can be useful.  When you have a function that’s time-consuming to
compute, a common technique is to cache the parameters and the resulting value
of each call to the function, and return the cached value if the same value is
requested again.  This is called “memoizing”, and can be implemented like this:

@example
# Callers can only provide two parameters and optionally pass _cache by keyword
def expensive(arg1, arg2, *, _cache=@{@}):
    if (arg1, arg2) in _cache:
        return _cache[(arg1, arg2)]

    # Calculate the value
    result = ... expensive computation ...
    _cache[(arg1, arg2)] = result           # Store result in the cache
    return result
@end example

You could use a global variable containing a dictionary instead of the default
value; it’s a matter of taste.

@node How can I pass optional or keyword parameters from one function to another?,What is the difference between arguments and parameters?,Why are default values shared between objects?,Core Language
@anchor{faq/programming how-can-i-pass-optional-or-keyword-parameters-from-one-function-to-another}@anchor{47e5}
@subsubsection How can I pass optional or keyword parameters from one function to another?


Collect the arguments using the @code{*} and @code{**} specifiers in the function’s
parameter list; this gives you the positional arguments as a tuple and the
keyword arguments as a dictionary.  You can then pass these arguments when
calling another function by using @code{*} and @code{**}:

@example
def f(x, *args, **kwargs):
    ...
    kwargs['width'] = '14.3c'
    ...
    g(x, *args, **kwargs)
@end example

@geindex argument; difference from parameter
@geindex parameter; difference from argument

@node What is the difference between arguments and parameters?,Why did changing list ‘y’ also change list ‘x’?,How can I pass optional or keyword parameters from one function to another?,Core Language
@anchor{faq/programming faq-argument-vs-parameter}@anchor{47e6}@anchor{faq/programming what-is-the-difference-between-arguments-and-parameters}@anchor{47e7}
@subsubsection What is the difference between arguments and parameters?


@ref{18e1,,Parameters} are defined by the names that appear in a
function definition, whereas @ref{18d2,,arguments} are the values
actually passed to a function when calling it.  Parameters define what types of
arguments a function can accept.  For example, given the function definition:

@example
def func(foo, bar=None, **kwargs):
    pass
@end example

@emph{foo}, @emph{bar} and @emph{kwargs} are parameters of @code{func}.  However, when calling
@code{func}, for example:

@example
func(42, bar=314, extra=somevar)
@end example

the values @code{42}, @code{314}, and @code{somevar} are arguments.

@node Why did changing list ‘y’ also change list ‘x’?,How do I write a function with output parameters call by reference ?,What is the difference between arguments and parameters?,Core Language
@anchor{faq/programming why-did-changing-list-y-also-change-list-x}@anchor{47e8}
@subsubsection Why did changing list ‘y’ also change list ‘x’?


If you wrote code like:

@example
>>> x = []
>>> y = x
>>> y.append(10)
>>> y
[10]
>>> x
[10]
@end example

you might be wondering why appending an element to @code{y} changed @code{x} too.

There are two factors that produce this result:


@enumerate 

@item 
Variables are simply names that refer to objects.  Doing @code{y = x} doesn’t
create a copy of the list – it creates a new variable @code{y} that refers to
the same object @code{x} refers to.  This means that there is only one object
(the list), and both @code{x} and @code{y} refer to it.

@item 
Lists are @ref{14e9,,mutable}, which means that you can change their content.
@end enumerate

After the call to @code{append()}, the content of the mutable object has
changed from @code{[]} to @code{[10]}.  Since both the variables refer to the same
object, using either name accesses the modified value @code{[10]}.

If we instead assign an immutable object to @code{x}:

@example
>>> x = 5  # ints are immutable
>>> y = x
>>> x = x + 1  # 5 can't be mutated, we are creating a new object here
>>> x
6
>>> y
5
@end example

we can see that in this case @code{x} and @code{y} are not equal anymore.  This is
because integers are @ref{14e1,,immutable}, and when we do @code{x = x + 1} we are not
mutating the int @code{5} by incrementing its value; instead, we are creating a
new object (the int @code{6}) and assigning it to @code{x} (that is, changing which
object @code{x} refers to).  After this assignment we have two objects (the ints
@code{6} and @code{5}) and two variables that refer to them (@code{x} now refers to
@code{6} but @code{y} still refers to @code{5}).

Some operations (for example @code{y.append(10)} and @code{y.sort()}) mutate the
object, whereas superficially similar operations (for example @code{y = y + [10]}
and @code{sorted(y)}) create a new object.  In general in Python (and in all cases
in the standard library) a method that mutates an object will return @code{None}
to help avoid getting the two types of operations confused.  So if you
mistakenly write @code{y.sort()} thinking it will give you a sorted copy of @code{y},
you’ll instead end up with @code{None}, which will likely cause your program to
generate an easily diagnosed error.

However, there is one class of operations where the same operation sometimes
has different behaviors with different types:  the augmented assignment
operators.  For example, @code{+=} mutates lists but not tuples or ints (@code{a_list
+= [1, 2, 3]} is equivalent to @code{a_list.extend([1, 2, 3])} and mutates
@code{a_list}, whereas @code{some_tuple += (1, 2, 3)} and @code{some_int += 1} create
new objects).

In other words:


@itemize *

@item 
If we have a mutable object (@ref{43a,,list}, @ref{3b0,,dict}, @ref{370,,set},
etc.), we can use some specific operations to mutate it and all the variables
that refer to it will see the change.

@item 
If we have an immutable object (@ref{2ef,,str}, @ref{1b1,,int}, @ref{3e5,,tuple},
etc.), all the variables that refer to it will always see the same value,
but operations that transform that value into a new value always return a new
object.
@end itemize

If you want to know if two variables refer to the same object or not, you can
use the @ref{2fd,,is} operator, or the built-in function @ref{f80,,id()}.

@node How do I write a function with output parameters call by reference ?,How do you make a higher order function in Python?,Why did changing list ‘y’ also change list ‘x’?,Core Language
@anchor{faq/programming how-do-i-write-a-function-with-output-parameters-call-by-reference}@anchor{47e9}
@subsubsection How do I write a function with output parameters (call by reference)?


Remember that arguments are passed by assignment in Python.  Since assignment
just creates references to objects, there’s no alias between an argument name in
the caller and callee, and so no call-by-reference per se.  You can achieve the
desired effect in a number of ways.


@enumerate 

@item 
By returning a tuple of the results:

@example
>>> def func1(a, b):
...     a = 'new-value'        # a and b are local names
...     b = b + 1              # assigned to new objects
...     return a, b            # return new values
...
>>> x, y = 'old-value', 99
>>> func1(x, y)
('new-value', 100)
@end example

This is almost always the clearest solution.

@item 
By using global variables.  This isn’t thread-safe, and is not recommended.

@item 
By passing a mutable (changeable in-place) object:

@example
>>> def func2(a):
...     a[0] = 'new-value'     # 'a' references a mutable list
...     a[1] = a[1] + 1        # changes a shared object
...
>>> args = ['old-value', 99]
>>> func2(args)
>>> args
['new-value', 100]
@end example

@item 
By passing in a dictionary that gets mutated:

@example
>>> def func3(args):
...     args['a'] = 'new-value'     # args is a mutable dictionary
...     args['b'] = args['b'] + 1   # change it in-place
...
>>> args = @{'a': 'old-value', 'b': 99@}
>>> func3(args)
>>> args
@{'a': 'new-value', 'b': 100@}
@end example

@item 
Or bundle up values in a class instance:

@example
>>> class Namespace:
...     def __init__(self, /, **args):
...         for key, value in args.items():
...             setattr(self, key, value)
...
>>> def func4(args):
...     args.a = 'new-value'        # args is a mutable Namespace
...     args.b = args.b + 1         # change object in-place
...
>>> args = Namespace(a='old-value', b=99)
>>> func4(args)
>>> vars(args)
@{'a': 'new-value', 'b': 100@}
@end example

There’s almost never a good reason to get this complicated.
@end enumerate

Your best choice is to return a tuple containing the multiple results.

@node How do you make a higher order function in Python?,How do I copy an object in Python?,How do I write a function with output parameters call by reference ?,Core Language
@anchor{faq/programming how-do-you-make-a-higher-order-function-in-python}@anchor{47ea}
@subsubsection How do you make a higher order function in Python?


You have two choices: you can use nested scopes or you can use callable objects.
For example, suppose you wanted to define @code{linear(a,b)} which returns a
function @code{f(x)} that computes the value @code{a*x+b}.  Using nested scopes:

@example
def linear(a, b):
    def result(x):
        return a * x + b
    return result
@end example

Or using a callable object:

@example
class linear:

    def __init__(self, a, b):
        self.a, self.b = a, b

    def __call__(self, x):
        return self.a * x + self.b
@end example

In both cases,

@example
taxes = linear(0.3, 2)
@end example

gives a callable object where @code{taxes(10e6) == 0.3 * 10e6 + 2}.

The callable object approach has the disadvantage that it is a bit slower and
results in slightly longer code.  However, note that a collection of callables
can share their signature via inheritance:

@example
class exponential(linear):
    # __init__ inherited
    def __call__(self, x):
        return self.a * (x ** self.b)
@end example

Object can encapsulate state for several methods:

@example
class counter:

    value = 0

    def set(self, x):
        self.value = x

    def up(self):
        self.value = self.value + 1

    def down(self):
        self.value = self.value - 1

count = counter()
inc, dec, reset = count.up, count.down, count.set
@end example

Here @code{inc()}, @code{dec()} and @code{reset()} act like functions which share the
same counting variable.

@node How do I copy an object in Python?,How can I find the methods or attributes of an object?,How do you make a higher order function in Python?,Core Language
@anchor{faq/programming how-do-i-copy-an-object-in-python}@anchor{47eb}
@subsubsection How do I copy an object in Python?


In general, try @ref{69a,,copy.copy()} or @ref{69b,,copy.deepcopy()} for the general case.
Not all objects can be copied, but most can.

Some objects can be copied more easily.  Dictionaries have a @ref{706,,copy()}
method:

@example
newdict = olddict.copy()
@end example

Sequences can be copied by slicing:

@example
new_l = l[:]
@end example

@node How can I find the methods or attributes of an object?,How can my code discover the name of an object?,How do I copy an object in Python?,Core Language
@anchor{faq/programming how-can-i-find-the-methods-or-attributes-of-an-object}@anchor{47ec}
@subsubsection How can I find the methods or attributes of an object?


For an instance x of a user-defined class, @code{dir(x)} returns an alphabetized
list of the names containing the instance attributes and methods and attributes
defined by its class.

@node How can my code discover the name of an object?,What’s up with the comma operator’s precedence?,How can I find the methods or attributes of an object?,Core Language
@anchor{faq/programming how-can-my-code-discover-the-name-of-an-object}@anchor{47ed}
@subsubsection How can my code discover the name of an object?


Generally speaking, it can’t, because objects don’t really have names.
Essentially, assignment always binds a name to a value; the same is true of
@code{def} and @code{class} statements, but in that case the value is a
callable. Consider the following code:

@example
>>> class A:
...     pass
...
>>> B = A
>>> a = B()
>>> b = a
>>> print(b)
<__main__.A object at 0x16D07CC>
>>> print(a)
<__main__.A object at 0x16D07CC>
@end example

Arguably the class has a name: even though it is bound to two names and invoked
through the name B the created instance is still reported as an instance of
class A.  However, it is impossible to say whether the instance’s name is a or
b, since both names are bound to the same value.

Generally speaking it should not be necessary for your code to “know the names”
of particular values. Unless you are deliberately writing introspective
programs, this is usually an indication that a change of approach might be
beneficial.

In comp.lang.python, Fredrik Lundh once gave an excellent analogy in answer to
this question:

@quotation

The same way as you get the name of that cat you found on your porch: the cat
(object) itself cannot tell you its name, and it doesn’t really care – so
the only way to find out what it’s called is to ask all your neighbours
(namespaces) if it’s their cat (object)…

….and don’t be surprised if you’ll find that it’s known by many names, or
no name at all!
@end quotation

@node What’s up with the comma operator’s precedence?,Is there an equivalent of C’s “? ” ternary operator?,How can my code discover the name of an object?,Core Language
@anchor{faq/programming what-s-up-with-the-comma-operator-s-precedence}@anchor{47ee}
@subsubsection What’s up with the comma operator’s precedence?


Comma is not an operator in Python.  Consider this session:

@example
>>> "a" in "b", "a"
(False, 'a')
@end example

Since the comma is not an operator, but a separator between expressions the
above is evaluated as if you had entered:

@example
("a" in "b"), "a"
@end example

not:

@example
"a" in ("b", "a")
@end example

The same is true of the various assignment operators (@code{=}, @code{+=} etc).  They
are not truly operators but syntactic delimiters in assignment statements.

@node Is there an equivalent of C’s “? ” ternary operator?,Is it possible to write obfuscated one-liners in Python?,What’s up with the comma operator’s precedence?,Core Language
@anchor{faq/programming is-there-an-equivalent-of-c-s-ternary-operator}@anchor{47ef}
@subsubsection Is there an equivalent of C’s “?:” ternary operator?


Yes, there is. The syntax is as follows:

@example
[on_true] if [expression] else [on_false]

x, y = 50, 25
small = x if x < y else y
@end example

Before this syntax was introduced in Python 2.5, a common idiom was to use
logical operators:

@example
[expression] and [on_true] or [on_false]
@end example

However, this idiom is unsafe, as it can give wrong results when @emph{on_true}
has a false boolean value.  Therefore, it is always better to use
the @code{... if ... else ...} form.

@node Is it possible to write obfuscated one-liners in Python?,What does the slash / in the parameter list of a function mean?,Is there an equivalent of C’s “? ” ternary operator?,Core Language
@anchor{faq/programming is-it-possible-to-write-obfuscated-one-liners-in-python}@anchor{47f0}
@subsubsection Is it possible to write obfuscated one-liners in Python?


Yes.  Usually this is done by nesting @ref{e42,,lambda} within
@code{lambda}.  See the following three examples, due to Ulf Bartelt:

@example
from functools import reduce

# Primes < 1000
print(list(filter(None,map(lambda y:y*reduce(lambda x,y:x*y!=0,
map(lambda x,y=y:y%x,range(2,int(pow(y,0.5)+1))),1),range(2,1000)))))

# First 10 Fibonacci numbers
print(list(map(lambda x,f=lambda x,f:(f(x-1,f)+f(x-2,f)) if x>1 else 1:
f(x,f), range(10))))

# Mandelbrot set
print((lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y,
Iu=Iu,Io=Io,Ru=Ru,Ro=Ro,Sy=Sy,L=lambda yc,Iu=Iu,Io=Io,Ru=Ru,Ro=Ro,i=IM,
Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro,
i=i,Sx=Sx,F=lambda xc,yc,x,y,k,f=lambda xc,yc,x,y,k,f:(k<=0)or (x*x+y*y
>=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr(
64+F(Ru+x*(Ro-Ru)/Sx,yc,0,0,i)),range(Sx))):L(Iu+y*(Io-Iu)/Sy),range(Sy
))))(-2.1, 0.7, -1.2, 1.2, 30, 80, 24))
#    \___ ___/  \___ ___/  |   |   |__ lines on screen
#        V          V      |   |______ columns on screen
#        |          |      |__________ maximum of "iterations"
#        |          |_________________ range on y axis
#        |____________________________ range on x axis
@end example

Don’t try this at home, kids!

@node What does the slash / in the parameter list of a function mean?,,Is it possible to write obfuscated one-liners in Python?,Core Language
@anchor{faq/programming faq-positional-only-arguments}@anchor{1a50}@anchor{faq/programming what-does-the-slash-in-the-parameter-list-of-a-function-mean}@anchor{47f1}
@subsubsection What does the slash(/) in the parameter list of a function mean?


A slash in the argument list of a function denotes that the parameters prior to
it are positional-only.  Positional-only parameters are the ones without an
externally-usable name.  Upon calling a function that accepts positional-only
parameters, arguments are mapped to parameters based solely on their position.
For example, @ref{4a8,,divmod()} is a function that accepts positional-only
parameters. Its documentation looks like this:

@example
>>> help(divmod)
Help on built-in function divmod in module builtins:

divmod(x, y, /)
    Return the tuple (x//y, x%y).  Invariant: div*y + mod == x.
@end example

The slash at the end of the parameter list means that both parameters are
positional-only. Thus, calling @ref{4a8,,divmod()} with keyword arguments would lead
to an error:

@example
>>> divmod(x=3, y=4)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: divmod() takes no keyword arguments
@end example

@node Numbers and strings,Performance<4>,Core Language,Programming FAQ
@anchor{faq/programming numbers-and-strings}@anchor{47f2}
@subsection Numbers and strings


@menu
* How do I specify hexadecimal and octal integers?:: 
* Why does -22 // 10 return -3?:: 
* How do I get int literal attribute instead of SyntaxError?:: 
* How do I convert a string to a number?:: 
* How do I convert a number to a string?:: 
* How do I modify a string in place?:: 
* How do I use strings to call functions/methods?:: 
* Is there an equivalent to Perl’s chomp() for removing trailing newlines from strings?: Is there an equivalent to Perl’s chomp for removing trailing newlines from strings?. 
* Is there a scanf() or sscanf() equivalent?: Is there a scanf or sscanf equivalent?. 
* What does ‘UnicodeDecodeError’ or ‘UnicodeEncodeError’ error mean?:: 

@end menu

@node How do I specify hexadecimal and octal integers?,Why does -22 // 10 return -3?,,Numbers and strings
@anchor{faq/programming how-do-i-specify-hexadecimal-and-octal-integers}@anchor{47f3}
@subsubsection How do I specify hexadecimal and octal integers?


To specify an octal digit, precede the octal value with a zero, and then a lower
or uppercase “o”.  For example, to set the variable “a” to the octal value “10”
(8 in decimal), type:

@example
>>> a = 0o10
>>> a
8
@end example

Hexadecimal is just as easy.  Simply precede the hexadecimal number with a zero,
and then a lower or uppercase “x”.  Hexadecimal digits can be specified in lower
or uppercase.  For example, in the Python interpreter:

@example
>>> a = 0xa5
>>> a
165
>>> b = 0XB2
>>> b
178
@end example

@node Why does -22 // 10 return -3?,How do I get int literal attribute instead of SyntaxError?,How do I specify hexadecimal and octal integers?,Numbers and strings
@anchor{faq/programming why-does-22-10-return-3}@anchor{47f4}
@subsubsection Why does -22 // 10 return -3?


It’s primarily driven by the desire that @code{i % j} have the same sign as @code{j}.
If you want that, and also want:

@example
i == (i // j) * j + (i % j)
@end example

then integer division has to return the floor.  C also requires that identity to
hold, and then compilers that truncate @code{i // j} need to make @code{i % j} have
the same sign as @code{i}.

There are few real use cases for @code{i % j} when @code{j} is negative.  When @code{j}
is positive, there are many, and in virtually all of them it’s more useful for
@code{i % j} to be @code{>= 0}.  If the clock says 10 now, what did it say 200 hours
ago?  @code{-190 % 12 == 2} is useful; @code{-190 % 12 == -10} is a bug waiting to
bite.

@node How do I get int literal attribute instead of SyntaxError?,How do I convert a string to a number?,Why does -22 // 10 return -3?,Numbers and strings
@anchor{faq/programming how-do-i-get-int-literal-attribute-instead-of-syntaxerror}@anchor{47f5}
@subsubsection How do I get int literal attribute instead of SyntaxError?


Trying to lookup an @code{int} literal attribute in the normal manner gives
a syntax error because the period is seen as a decimal point:

@example
>>> 1.__class__
  File "<stdin>", line 1
  1.__class__
   ^
SyntaxError: invalid decimal literal
@end example

The solution is to separate the literal from the period
with either a space or parentheses.

@example
>>> 1 .__class__
<class 'int'>
>>> (1).__class__
<class 'int'>
@end example

@node How do I convert a string to a number?,How do I convert a number to a string?,How do I get int literal attribute instead of SyntaxError?,Numbers and strings
@anchor{faq/programming how-do-i-convert-a-string-to-a-number}@anchor{47f6}
@subsubsection How do I convert a string to a number?


For integers, use the built-in @ref{1b1,,int()} type constructor, e.g. @code{int('144')
== 144}.  Similarly, @ref{256,,float()} converts to floating-point,
e.g. @code{float('144') == 144.0}.

By default, these interpret the number as decimal, so that @code{int('0144') ==
144} holds true, and @code{int('0x144')} raises @ref{17c,,ValueError}. @code{int(string,
base)} takes the base to convert from as a second optional argument, so @code{int(
'0x144', 16) == 324}.  If the base is specified as 0, the number is interpreted
using Python’s rules: a leading ‘0o’ indicates octal, and ‘0x’ indicates a hex
number.

Do not use the built-in function @ref{34a,,eval()} if all you need is to convert
strings to numbers.  @ref{34a,,eval()} will be significantly slower and it presents a
security risk: someone could pass you a Python expression that might have
unwanted side effects.  For example, someone could pass
@code{__import__('os').system("rm -rf $HOME")} which would erase your home
directory.

@ref{34a,,eval()} also has the effect of interpreting numbers as Python expressions,
so that e.g. @code{eval('09')} gives a syntax error because Python does not allow
leading ‘0’ in a decimal number (except ‘0’).

@node How do I convert a number to a string?,How do I modify a string in place?,How do I convert a string to a number?,Numbers and strings
@anchor{faq/programming how-do-i-convert-a-number-to-a-string}@anchor{47f7}
@subsubsection How do I convert a number to a string?


To convert, e.g., the number 144 to the string ‘144’, use the built-in type
constructor @ref{2ef,,str()}.  If you want a hexadecimal or octal representation, use
the built-in functions @ref{e6f,,hex()} or @ref{e6e,,oct()}.  For fancy formatting, see
the @ref{4b0,,Formatted string literals} and @ref{f1a,,Format String Syntax} sections,
e.g. @code{"@{:04d@}".format(144)} yields
@code{'0144'} and @code{"@{:.3f@}".format(1.0/3.0)} yields @code{'0.333'}.

@node How do I modify a string in place?,How do I use strings to call functions/methods?,How do I convert a number to a string?,Numbers and strings
@anchor{faq/programming how-do-i-modify-a-string-in-place}@anchor{47f8}
@subsubsection How do I modify a string in place?


You can’t, because strings are immutable.  In most situations, you should
simply construct a new string from the various parts you want to assemble
it from.  However, if you need an object with the ability to modify in-place
unicode data, try using an @ref{aa1,,io.StringIO} object or the @ref{6,,array}
module:

@example
>>> import io
>>> s = "Hello, world"
>>> sio = io.StringIO(s)
>>> sio.getvalue()
'Hello, world'
>>> sio.seek(7)
7
>>> sio.write("there!")
6
>>> sio.getvalue()
'Hello, there!'

>>> import array
>>> a = array.array('u', s)
>>> print(a)
array('u', 'Hello, world')
>>> a[0] = 'y'
>>> print(a)
array('u', 'yello, world')
>>> a.tounicode()
'yello, world'
@end example

@node How do I use strings to call functions/methods?,Is there an equivalent to Perl’s chomp for removing trailing newlines from strings?,How do I modify a string in place?,Numbers and strings
@anchor{faq/programming how-do-i-use-strings-to-call-functions-methods}@anchor{47f9}
@subsubsection How do I use strings to call functions/methods?


There are various techniques.


@itemize *

@item 
The best is to use a dictionary that maps strings to functions.  The primary
advantage of this technique is that the strings do not need to match the names
of the functions.  This is also the primary technique used to emulate a case
construct:

@example
def a():
    pass

def b():
    pass

dispatch = @{'go': a, 'stop': b@}  # Note lack of parens for funcs

dispatch[get_input()]()  # Note trailing parens to call function
@end example

@item 
Use the built-in function @ref{708,,getattr()}:

@example
import foo
getattr(foo, 'bar')()
@end example

Note that @ref{708,,getattr()} works on any object, including classes, class
instances, modules, and so on.

This is used in several places in the standard library, like this:

@example
class Foo:
    def do_foo(self):
        ...

    def do_bar(self):
        ...

f = getattr(foo_instance, 'do_' + opname)
f()
@end example

@item 
Use @ref{713,,locals()} to resolve the function name:

@example
def myFunc():
    print("hello")

fname = "myFunc"

f = locals()[fname]
f()
@end example
@end itemize

@node Is there an equivalent to Perl’s chomp for removing trailing newlines from strings?,Is there a scanf or sscanf equivalent?,How do I use strings to call functions/methods?,Numbers and strings
@anchor{faq/programming is-there-an-equivalent-to-perl-s-chomp-for-removing-trailing-newlines-from-strings}@anchor{47fa}
@subsubsection Is there an equivalent to Perl’s chomp() for removing trailing newlines from strings?


You can use @code{S.rstrip("\r\n")} to remove all occurrences of any line
terminator from the end of the string @code{S} without removing other trailing
whitespace.  If the string @code{S} represents more than one line, with several
empty lines at the end, the line terminators for all the blank lines will
be removed:

@example
>>> lines = ("line 1 \r\n"
...          "\r\n"
...          "\r\n")
>>> lines.rstrip("\n\r")
'line 1 '
@end example

Since this is typically only desired when reading text one line at a time, using
@code{S.rstrip()} this way works well.

@node Is there a scanf or sscanf equivalent?,What does ‘UnicodeDecodeError’ or ‘UnicodeEncodeError’ error mean?,Is there an equivalent to Perl’s chomp for removing trailing newlines from strings?,Numbers and strings
@anchor{faq/programming is-there-a-scanf-or-sscanf-equivalent}@anchor{47fb}
@subsubsection Is there a scanf() or sscanf() equivalent?


Not as such.

For simple input parsing, the easiest approach is usually to split the line into
whitespace-delimited words using the @ref{a1b,,split()} method of string objects
and then convert decimal strings to numeric values using @ref{1b1,,int()} or
@ref{256,,float()}.  @code{split()} supports an optional “sep” parameter which is useful
if the line uses something other than whitespace as a separator.

For more complicated input parsing, regular expressions are more powerful
than C’s @code{sscanf()} and better suited for the task.

@node What does ‘UnicodeDecodeError’ or ‘UnicodeEncodeError’ error mean?,,Is there a scanf or sscanf equivalent?,Numbers and strings
@anchor{faq/programming what-does-unicodedecodeerror-or-unicodeencodeerror-error-mean}@anchor{47fc}
@subsubsection What does ‘UnicodeDecodeError’ or ‘UnicodeEncodeError’ error  mean?


See the @ref{e4a,,Unicode HOWTO}.

@node Performance<4>,Sequences Tuples/Lists,Numbers and strings,Programming FAQ
@anchor{faq/programming performance}@anchor{47fd}
@subsection Performance


@menu
* My program is too slow. How do I speed it up?: My program is too slow How do I speed it up?. 
* What is the most efficient way to concatenate many strings together?:: 

@end menu

@node My program is too slow How do I speed it up?,What is the most efficient way to concatenate many strings together?,,Performance<4>
@anchor{faq/programming my-program-is-too-slow-how-do-i-speed-it-up}@anchor{47fe}
@subsubsection My program is too slow. How do I speed it up?


That’s a tough one, in general.  First, here are a list of things to
remember before diving further:


@itemize *

@item 
Performance characteristics vary across Python implementations.  This FAQ
focuses on @ref{17a4,,CPython}.

@item 
Behaviour can vary across operating systems, especially when talking about
I/O or multi-threading.

@item 
You should always find the hot spots in your program @emph{before} attempting to
optimize any code (see the @ref{ce,,profile} module).

@item 
Writing benchmark scripts will allow you to iterate quickly when searching
for improvements (see the @ref{10b,,timeit} module).

@item 
It is highly recommended to have good code coverage (through unit testing
or any other technique) before potentially introducing regressions hidden
in sophisticated optimizations.
@end itemize

That being said, there are many tricks to speed up Python code.  Here are
some general principles which go a long way towards reaching acceptable
performance levels:


@itemize *

@item 
Making your algorithms faster (or changing to faster ones) can yield
much larger benefits than trying to sprinkle micro-optimization tricks
all over your code.

@item 
Use the right data structures.  Study documentation for the @ref{1a70,,Built-in Types}
and the @ref{1c,,collections} module.

@item 
When the standard library provides a primitive for doing something, it is
likely (although not guaranteed) to be faster than any alternative you
may come up with.  This is doubly true for primitives written in C, such
as builtins and some extension types.  For example, be sure to use
either the @ref{705,,list.sort()} built-in method or the related @ref{704,,sorted()}
function to do sorting (and see the @ref{1a5e,,Sorting HOW TO} for examples
of moderately advanced usage).

@item 
Abstractions tend to create indirections and force the interpreter to work
more.  If the levels of indirection outweigh the amount of useful work
done, your program will be slower.  You should avoid excessive abstraction,
especially under the form of tiny functions or methods (which are also often
detrimental to readability).
@end itemize

If you have reached the limit of what pure Python can allow, there are tools
to take you further away.  For example, Cython@footnote{http://cython.org} can
compile a slightly modified version of Python code into a C extension, and
can be used on many different platforms.  Cython can take advantage of
compilation (and optional type annotations) to make your code significantly
faster than when interpreted.  If you are confident in your C programming
skills, you can also @ref{14be,,write a C extension module}
yourself.


@subsubheading See also


The wiki page devoted to performance tips@footnote{https://wiki.python.org/moin/PythonSpeed/PerformanceTips}.


@node What is the most efficient way to concatenate many strings together?,,My program is too slow How do I speed it up?,Performance<4>
@anchor{faq/programming efficient-string-concatenation}@anchor{47ff}@anchor{faq/programming what-is-the-most-efficient-way-to-concatenate-many-strings-together}@anchor{4800}
@subsubsection What is the most efficient way to concatenate many strings together?


@ref{2ef,,str} and @ref{2f0,,bytes} objects are immutable, therefore concatenating
many strings together is inefficient as each concatenation creates a new
object.  In the general case, the total runtime cost is quadratic in the
total string length.

To accumulate many @ref{2ef,,str} objects, the recommended idiom is to place
them into a list and call @ref{1a8b,,str.join()} at the end:

@example
chunks = []
for s in my_strings:
    chunks.append(s)
result = '@w{'}.join(chunks)
@end example

(another reasonably efficient idiom is to use @ref{aa1,,io.StringIO})

To accumulate many @ref{2f0,,bytes} objects, the recommended idiom is to extend
a @ref{2f1,,bytearray} object using in-place concatenation (the @code{+=} operator):

@example
result = bytearray()
for b in my_bytes_objects:
    result += b
@end example

@node Sequences Tuples/Lists,Objects,Performance<4>,Programming FAQ
@anchor{faq/programming sequences-tuples-lists}@anchor{4801}
@subsection Sequences (Tuples/Lists)


@menu
* How do I convert between tuples and lists?:: 
* What’s a negative index?:: 
* How do I iterate over a sequence in reverse order?:: 
* How do you remove duplicates from a list?:: 
* How do you remove multiple items from a list:: 
* How do you make an array in Python?:: 
* How do I create a multidimensional list?:: 
* How do I apply a method to a sequence of objects?:: 
* Why does a_tuple[i] += [‘item’] raise an exception when the addition works?:: 
* I want to do a complicated sort; can you do a Schwartzian Transform in Python?: I want to do a complicated sort can you do a Schwartzian Transform in Python?. 
* How can I sort one list by values from another list?:: 

@end menu

@node How do I convert between tuples and lists?,What’s a negative index?,,Sequences Tuples/Lists
@anchor{faq/programming how-do-i-convert-between-tuples-and-lists}@anchor{4802}
@subsubsection How do I convert between tuples and lists?


The type constructor @code{tuple(seq)} converts any sequence (actually, any
iterable) into a tuple with the same items in the same order.

For example, @code{tuple([1, 2, 3])} yields @code{(1, 2, 3)} and @code{tuple('abc')}
yields @code{('a', 'b', 'c')}.  If the argument is a tuple, it does not make a copy
but returns the same object, so it is cheap to call @ref{3e5,,tuple()} when you
aren’t sure that an object is already a tuple.

The type constructor @code{list(seq)} converts any sequence or iterable into a list
with the same items in the same order.  For example, @code{list((1, 2, 3))} yields
@code{[1, 2, 3]} and @code{list('abc')} yields @code{['a', 'b', 'c']}.  If the argument
is a list, it makes a copy just like @code{seq[:]} would.

@node What’s a negative index?,How do I iterate over a sequence in reverse order?,How do I convert between tuples and lists?,Sequences Tuples/Lists
@anchor{faq/programming what-s-a-negative-index}@anchor{4803}
@subsubsection What’s a negative index?


Python sequences are indexed with positive numbers and negative numbers.  For
positive numbers 0 is the first index 1 is the second index and so forth.  For
negative indices -1 is the last index and -2 is the penultimate (next to last)
index and so forth.  Think of @code{seq[-n]} as the same as @code{seq[len(seq)-n]}.

Using negative indices can be very convenient.  For example @code{S[:-1]} is all of
the string except for its last character, which is useful for removing the
trailing newline from a string.

@node How do I iterate over a sequence in reverse order?,How do you remove duplicates from a list?,What’s a negative index?,Sequences Tuples/Lists
@anchor{faq/programming how-do-i-iterate-over-a-sequence-in-reverse-order}@anchor{4804}
@subsubsection How do I iterate over a sequence in reverse order?


Use the @ref{2f6,,reversed()} built-in function:

@example
for x in reversed(sequence):
    ...  # do something with x ...
@end example

This won’t touch your original sequence, but build a new copy with reversed
order to iterate over.

@node How do you remove duplicates from a list?,How do you remove multiple items from a list,How do I iterate over a sequence in reverse order?,Sequences Tuples/Lists
@anchor{faq/programming how-do-you-remove-duplicates-from-a-list}@anchor{4805}
@subsubsection How do you remove duplicates from a list?


See the Python Cookbook for a long discussion of many ways to do this:

@quotation

@indicateurl{https://code.activestate.com/recipes/52560/}
@end quotation

If you don’t mind reordering the list, sort it and then scan from the end of the
list, deleting duplicates as you go:

@example
if mylist:
    mylist.sort()
    last = mylist[-1]
    for i in range(len(mylist)-2, -1, -1):
        if last == mylist[i]:
            del mylist[i]
        else:
            last = mylist[i]
@end example

If all elements of the list may be used as set keys (i.e. they are all
@ref{2dc,,hashable}) this is often faster

@example
mylist = list(set(mylist))
@end example

This converts the list into a set, thereby removing duplicates, and then back
into a list.

@node How do you remove multiple items from a list,How do you make an array in Python?,How do you remove duplicates from a list?,Sequences Tuples/Lists
@anchor{faq/programming how-do-you-remove-multiple-items-from-a-list}@anchor{4806}
@subsubsection How do you remove multiple items from a list


As with removing duplicates, explicitly iterating in reverse with a
delete condition is one possibility.  However, it is easier and faster
to use slice replacement with an implicit or explicit forward iteration.
Here are three variations.:

@example
mylist[:] = filter(keep_function, mylist)
mylist[:] = (x for x in mylist if keep_condition)
mylist[:] = [x for x in mylist if keep_condition]
@end example

The list comprehension may be fastest.

@node How do you make an array in Python?,How do I create a multidimensional list?,How do you remove multiple items from a list,Sequences Tuples/Lists
@anchor{faq/programming how-do-you-make-an-array-in-python}@anchor{4807}
@subsubsection How do you make an array in Python?


Use a list:

@example
["this", 1, "is", "an", "array"]
@end example

Lists are equivalent to C or Pascal arrays in their time complexity; the primary
difference is that a Python list can contain objects of many different types.

The @code{array} module also provides methods for creating arrays of fixed types
with compact representations, but they are slower to index than lists.  Also
note that NumPy and other third party packages define array-like structures with
various characteristics as well.

To get Lisp-style linked lists, you can emulate cons cells using tuples:

@example
lisp_list = ("like",  ("this",  ("example", None) ) )
@end example

If mutability is desired, you could use lists instead of tuples.  Here the
analogue of lisp car is @code{lisp_list[0]} and the analogue of cdr is
@code{lisp_list[1]}.  Only do this if you’re sure you really need to, because it’s
usually a lot slower than using Python lists.

@node How do I create a multidimensional list?,How do I apply a method to a sequence of objects?,How do you make an array in Python?,Sequences Tuples/Lists
@anchor{faq/programming faq-multidimensional-list}@anchor{1a8a}@anchor{faq/programming how-do-i-create-a-multidimensional-list}@anchor{4808}
@subsubsection How do I create a multidimensional list?


You probably tried to make a multidimensional array like this:

@example
>>> A = [[None] * 2] * 3
@end example

This looks correct if you print it:

@c A = [[None] * 2] * 3

@example
>>> A
[[None, None], [None, None], [None, None]]
@end example

But when you assign a value, it shows up in multiple places:

@c A = [[None] * 2] * 3

@example
>>> A[0][0] = 5
>>> A
[[5, None], [5, None], [5, None]]
@end example

The reason is that replicating a list with @code{*} doesn’t create copies, it only
creates references to the existing objects.  The @code{*3} creates a list
containing 3 references to the same list of length two.  Changes to one row will
show in all rows, which is almost certainly not what you want.

The suggested approach is to create a list of the desired length first and then
fill in each element with a newly created list:

@example
A = [None] * 3
for i in range(3):
    A[i] = [None] * 2
@end example

This generates a list containing 3 different lists of length two.  You can also
use a list comprehension:

@example
w, h = 2, 3
A = [[None] * w for i in range(h)]
@end example

Or, you can use an extension that provides a matrix datatype; NumPy@footnote{http://www.numpy.org/} is the best known.

@node How do I apply a method to a sequence of objects?,Why does a_tuple[i] += [‘item’] raise an exception when the addition works?,How do I create a multidimensional list?,Sequences Tuples/Lists
@anchor{faq/programming how-do-i-apply-a-method-to-a-sequence-of-objects}@anchor{4809}
@subsubsection How do I apply a method to a sequence of objects?


Use a list comprehension:

@example
result = [obj.method() for obj in mylist]
@end example

@node Why does a_tuple[i] += [‘item’] raise an exception when the addition works?,I want to do a complicated sort can you do a Schwartzian Transform in Python?,How do I apply a method to a sequence of objects?,Sequences Tuples/Lists
@anchor{faq/programming faq-augmented-assignment-tuple-error}@anchor{1805}@anchor{faq/programming why-does-a-tuple-i-item-raise-an-exception-when-the-addition-works}@anchor{480a}
@subsubsection Why does a_tuple[i] += [‘item’] raise an exception when the addition works?


This is because of a combination of the fact that augmented assignment
operators are @emph{assignment} operators, and the difference between mutable and
immutable objects in Python.

This discussion applies in general when augmented assignment operators are
applied to elements of a tuple that point to mutable objects, but we’ll use
a @code{list} and @code{+=} as our exemplar.

If you wrote:

@example
>>> a_tuple = (1, 2)
>>> a_tuple[0] += 1
Traceback (most recent call last):
   ...
TypeError: 'tuple' object does not support item assignment
@end example

The reason for the exception should be immediately clear: @code{1} is added to the
object @code{a_tuple[0]} points to (@code{1}), producing the result object, @code{2},
but when we attempt to assign the result of the computation, @code{2}, to element
@code{0} of the tuple, we get an error because we can’t change what an element of
a tuple points to.

Under the covers, what this augmented assignment statement is doing is
approximately this:

@example
>>> result = a_tuple[0] + 1
>>> a_tuple[0] = result
Traceback (most recent call last):
  ...
TypeError: 'tuple' object does not support item assignment
@end example

It is the assignment part of the operation that produces the error, since a
tuple is immutable.

When you write something like:

@example
>>> a_tuple = (['foo'], 'bar')
>>> a_tuple[0] += ['item']
Traceback (most recent call last):
  ...
TypeError: 'tuple' object does not support item assignment
@end example

The exception is a bit more surprising, and even more surprising is the fact
that even though there was an error, the append worked:

@example
>>> a_tuple[0]
['foo', 'item']
@end example

To see why this happens, you need to know that (a) if an object implements an
@code{__iadd__} magic method, it gets called when the @code{+=} augmented assignment
is executed, and its return value is what gets used in the assignment statement;
and (b) for lists, @code{__iadd__} is equivalent to calling @code{extend} on the list
and returning the list.  That’s why we say that for lists, @code{+=} is a
“shorthand” for @code{list.extend}:

@example
>>> a_list = []
>>> a_list += [1]
>>> a_list
[1]
@end example

This is equivalent to:

@example
>>> result = a_list.__iadd__([1])
>>> a_list = result
@end example

The object pointed to by a_list has been mutated, and the pointer to the
mutated object is assigned back to @code{a_list}.  The end result of the
assignment is a no-op, since it is a pointer to the same object that @code{a_list}
was previously pointing to, but the assignment still happens.

Thus, in our tuple example what is happening is equivalent to:

@example
>>> result = a_tuple[0].__iadd__(['item'])
>>> a_tuple[0] = result
Traceback (most recent call last):
  ...
TypeError: 'tuple' object does not support item assignment
@end example

The @code{__iadd__} succeeds, and thus the list is extended, but even though
@code{result} points to the same object that @code{a_tuple[0]} already points to,
that final assignment still results in an error, because tuples are immutable.

@node I want to do a complicated sort can you do a Schwartzian Transform in Python?,How can I sort one list by values from another list?,Why does a_tuple[i] += [‘item’] raise an exception when the addition works?,Sequences Tuples/Lists
@anchor{faq/programming i-want-to-do-a-complicated-sort-can-you-do-a-schwartzian-transform-in-python}@anchor{480b}
@subsubsection I want to do a complicated sort: can you do a Schwartzian Transform in Python?


The technique, attributed to Randal Schwartz of the Perl community, sorts the
elements of a list by a metric which maps each element to its “sort value”. In
Python, use the @code{key} argument for the @ref{705,,list.sort()} method:

@example
Isorted = L[:]
Isorted.sort(key=lambda s: int(s[10:15]))
@end example

@node How can I sort one list by values from another list?,,I want to do a complicated sort can you do a Schwartzian Transform in Python?,Sequences Tuples/Lists
@anchor{faq/programming how-can-i-sort-one-list-by-values-from-another-list}@anchor{480c}
@subsubsection How can I sort one list by values from another list?


Merge them into an iterator of tuples, sort the resulting list, and then pick
out the element you want.

@example
>>> list1 = ["what", "I'm", "sorting", "by"]
>>> list2 = ["something", "else", "to", "sort"]
>>> pairs = zip(list1, list2)
>>> pairs = sorted(pairs)
>>> pairs
[("I'm", 'else'), ('by', 'sort'), ('sorting', 'to'), ('what', 'something')]
>>> result = [x[1] for x in pairs]
>>> result
['else', 'sort', 'to', 'something']
@end example

@node Objects,Modules<3>,Sequences Tuples/Lists,Programming FAQ
@anchor{faq/programming objects}@anchor{480d}
@subsection Objects


@menu
* What is a class?:: 
* What is a method?:: 
* What is self?:: 
* How do I check if an object is an instance of a given class or of a subclass of it?:: 
* What is delegation?:: 
* How do I call a method defined in a base class from a derived class that extends it?:: 
* How can I organize my code to make it easier to change the base class?:: 
* How do I create static class data and static class methods?:: 
* How can I overload constructors (or methods) in Python?: How can I overload constructors or methods in Python?. 
* I try to use __spam and I get an error about _SomeClassName__spam.: I try to use __spam and I get an error about _SomeClassName__spam. 
* My class defines __del__ but it is not called when I delete the object.: My class defines __del__ but it is not called when I delete the object. 
* How do I get a list of all instances of a given class?:: 
* Why does the result of id() appear to be not unique?: Why does the result of id appear to be not unique?. 
* When can I rely on identity tests with the is operator?:: 
* How can a subclass control what data is stored in an immutable instance?:: 
* How do I cache method calls?:: 

@end menu

@node What is a class?,What is a method?,,Objects
@anchor{faq/programming what-is-a-class}@anchor{480e}
@subsubsection What is a class?


A class is the particular object type created by executing a class statement.
Class objects are used as templates to create instance objects, which embody
both the data (attributes) and code (methods) specific to a datatype.

A class can be based on one or more other classes, called its base class(es). It
then inherits the attributes and methods of its base classes. This allows an
object model to be successively refined by inheritance.  You might have a
generic @code{Mailbox} class that provides basic accessor methods for a mailbox,
and subclasses such as @code{MboxMailbox}, @code{MaildirMailbox}, @code{OutlookMailbox}
that handle various specific mailbox formats.

@node What is a method?,What is self?,What is a class?,Objects
@anchor{faq/programming what-is-a-method}@anchor{480f}
@subsubsection What is a method?


A method is a function on some object @code{x} that you normally call as
@code{x.name(arguments...)}.  Methods are defined as functions inside the class
definition:

@example
class C:
    def meth(self, arg):
        return arg * 2 + self.attribute
@end example

@node What is self?,How do I check if an object is an instance of a given class or of a subclass of it?,What is a method?,Objects
@anchor{faq/programming what-is-self}@anchor{4810}
@subsubsection What is self?


Self is merely a conventional name for the first argument of a method.  A method
defined as @code{meth(self, a, b, c)} should be called as @code{x.meth(a, b, c)} for
some instance @code{x} of the class in which the definition occurs; the called
method will think it is called as @code{meth(x, a, b, c)}.

See also @ref{4811,,Why must ‘self’ be used explicitly in method definitions and calls?}.

@node How do I check if an object is an instance of a given class or of a subclass of it?,What is delegation?,What is self?,Objects
@anchor{faq/programming how-do-i-check-if-an-object-is-an-instance-of-a-given-class-or-of-a-subclass-of-it}@anchor{4812}
@subsubsection How do I check if an object is an instance of a given class or of a subclass of it?


Use the built-in function @code{isinstance(obj, cls)}.  You can check if an object
is an instance of any of a number of classes by providing a tuple instead of a
single class, e.g. @code{isinstance(obj, (class1, class2, ...))}, and can also
check whether an object is one of Python’s built-in types, e.g.
@code{isinstance(obj, str)} or @code{isinstance(obj, (int, float, complex))}.

Note that @ref{236,,isinstance()} also checks for virtual inheritance from an
@ref{d5c,,abstract base class}.  So, the test will return @code{True} for a
registered class even if hasn’t directly or indirectly inherited from it.  To
test for “true inheritance”, scan the @ref{2cfa,,MRO} of the class:

@example
from collections.abc import Mapping

class P:
     pass

class C(P):
    pass

Mapping.register(P)
@end example

@example
>>> c = C()
>>> isinstance(c, C)        # direct
True
>>> isinstance(c, P)        # indirect
True
>>> isinstance(c, Mapping)  # virtual
True

# Actual inheritance chain
>>> type(c).__mro__
(<class 'C'>, <class 'P'>, <class 'object'>)

# Test for "true inheritance"
>>> Mapping in type(c).__mro__
False
@end example

Note that most programs do not use @ref{236,,isinstance()} on user-defined classes
very often.  If you are developing the classes yourself, a more proper
object-oriented style is to define methods on the classes that encapsulate a
particular behaviour, instead of checking the object’s class and doing a
different thing based on what class it is.  For example, if you have a function
that does something:

@example
def search(obj):
    if isinstance(obj, Mailbox):
        ...  # code to search a mailbox
    elif isinstance(obj, Document):
        ...  # code to search a document
    elif ...
@end example

A better approach is to define a @code{search()} method on all the classes and just
call it:

@example
class Mailbox:
    def search(self):
        ...  # code to search a mailbox

class Document:
    def search(self):
        ...  # code to search a document

obj.search()
@end example

@node What is delegation?,How do I call a method defined in a base class from a derived class that extends it?,How do I check if an object is an instance of a given class or of a subclass of it?,Objects
@anchor{faq/programming what-is-delegation}@anchor{4813}
@subsubsection What is delegation?


Delegation is an object oriented technique (also called a design pattern).
Let’s say you have an object @code{x} and want to change the behaviour of just one
of its methods.  You can create a new class that provides a new implementation
of the method you’re interested in changing and delegates all other methods to
the corresponding method of @code{x}.

Python programmers can easily implement delegation.  For example, the following
class implements a class that behaves like a file but converts all written data
to uppercase:

@example
class UpperOut:

    def __init__(self, outfile):
        self._outfile = outfile

    def write(self, s):
        self._outfile.write(s.upper())

    def __getattr__(self, name):
        return getattr(self._outfile, name)
@end example

Here the @code{UpperOut} class redefines the @code{write()} method to convert the
argument string to uppercase before calling the underlying
@code{self._outfile.write()} method.  All other methods are delegated to the
underlying @code{self._outfile} object.  The delegation is accomplished via the
@code{__getattr__} method; consult @ref{17ae,,the language reference}
for more information about controlling attribute access.

Note that for more general cases delegation can get trickier. When attributes
must be set as well as retrieved, the class must define a @code{__setattr__()}
method too, and it must do so carefully.  The basic implementation of
@code{__setattr__()} is roughly equivalent to the following:

@example
class X:
    ...
    def __setattr__(self, name, value):
        self.__dict__[name] = value
    ...
@end example

Most @code{__setattr__()} implementations must modify @code{self.__dict__} to store
local state for self without causing an infinite recursion.

@node How do I call a method defined in a base class from a derived class that extends it?,How can I organize my code to make it easier to change the base class?,What is delegation?,Objects
@anchor{faq/programming how-do-i-call-a-method-defined-in-a-base-class-from-a-derived-class-that-extends-it}@anchor{4814}
@subsubsection How do I call a method defined in a base class from a derived class that extends it?


Use the built-in @ref{78c,,super()} function:

@example
class Derived(Base):
    def meth(self):
        super().meth()  # calls Base.meth
@end example

In the example, @ref{78c,,super()} will automatically determine the instance from
which it was called (the @code{self} value), look up the @ref{1a60,,method resolution order} (MRO) with @code{type(self).__mro__}, and return the next in line after
@code{Derived} in the MRO: @code{Base}.

@node How can I organize my code to make it easier to change the base class?,How do I create static class data and static class methods?,How do I call a method defined in a base class from a derived class that extends it?,Objects
@anchor{faq/programming how-can-i-organize-my-code-to-make-it-easier-to-change-the-base-class}@anchor{4815}
@subsubsection How can I organize my code to make it easier to change the base class?


You could assign the base class to an alias and derive from the alias.  Then all
you have to change is the value assigned to the alias.  Incidentally, this trick
is also handy if you want to decide dynamically (e.g. depending on availability
of resources) which base class to use.  Example:

@example
class Base:
    ...

BaseAlias = Base

class Derived(BaseAlias):
    ...
@end example

@node How do I create static class data and static class methods?,How can I overload constructors or methods in Python?,How can I organize my code to make it easier to change the base class?,Objects
@anchor{faq/programming how-do-i-create-static-class-data-and-static-class-methods}@anchor{4816}
@subsubsection How do I create static class data and static class methods?


Both static data and static methods (in the sense of C++ or Java) are supported
in Python.

For static data, simply define a class attribute.  To assign a new value to the
attribute, you have to explicitly use the class name in the assignment:

@example
class C:
    count = 0   # number of times C.__init__ called

    def __init__(self):
        C.count = C.count + 1

    def getcount(self):
        return C.count  # or return self.count
@end example

@code{c.count} also refers to @code{C.count} for any @code{c} such that @code{isinstance(c,
C)} holds, unless overridden by @code{c} itself or by some class on the base-class
search path from @code{c.__class__} back to @code{C}.

Caution: within a method of C, an assignment like @code{self.count = 42} creates a
new and unrelated instance named “count” in @code{self}’s own dict.  Rebinding of a
class-static data name must always specify the class whether inside a method or
not:

@example
C.count = 314
@end example

Static methods are possible:

@example
class C:
    @@staticmethod
    def static(arg1, arg2, arg3):
        # No 'self' parameter!
        ...
@end example

However, a far more straightforward way to get the effect of a static method is
via a simple module-level function:

@example
def getcount():
    return C.count
@end example

If your code is structured so as to define one class (or tightly related class
hierarchy) per module, this supplies the desired encapsulation.

@node How can I overload constructors or methods in Python?,I try to use __spam and I get an error about _SomeClassName__spam,How do I create static class data and static class methods?,Objects
@anchor{faq/programming how-can-i-overload-constructors-or-methods-in-python}@anchor{4817}
@subsubsection How can I overload constructors (or methods) in Python?


This answer actually applies to all methods, but the question usually comes up
first in the context of constructors.

In C++ you’d write

@example
class C @{
    C() @{ cout << "No arguments\n"; @}
    C(int i) @{ cout << "Argument is " << i << "\n"; @}
@}
@end example

In Python you have to write a single constructor that catches all cases using
default arguments.  For example:

@example
class C:
    def __init__(self, i=None):
        if i is None:
            print("No arguments")
        else:
            print("Argument is", i)
@end example

This is not entirely equivalent, but close enough in practice.

You could also try a variable-length argument list, e.g.

@example
def __init__(self, *args):
    ...
@end example

The same approach works for all method definitions.

@node I try to use __spam and I get an error about _SomeClassName__spam,My class defines __del__ but it is not called when I delete the object,How can I overload constructors or methods in Python?,Objects
@anchor{faq/programming i-try-to-use-spam-and-i-get-an-error-about-someclassname-spam}@anchor{4818}
@subsubsection I try to use __spam and I get an error about _SomeClassName__spam.


Variable names with double leading underscores are “mangled” to provide a simple
but effective way to define class private variables.  Any identifier of the form
@code{__spam} (at least two leading underscores, at most one trailing underscore)
is textually replaced with @code{_classname__spam}, where @code{classname} is the
current class name with any leading underscores stripped.

This doesn’t guarantee privacy: an outside user can still deliberately access
the “_classname__spam” attribute, and private values are visible in the object’s
@code{__dict__}.  Many Python programmers never bother to use private variable
names at all.

@node My class defines __del__ but it is not called when I delete the object,How do I get a list of all instances of a given class?,I try to use __spam and I get an error about _SomeClassName__spam,Objects
@anchor{faq/programming my-class-defines-del-but-it-is-not-called-when-i-delete-the-object}@anchor{4819}
@subsubsection My class defines __del__ but it is not called when I delete the object.


There are several possible reasons for this.

The del statement does not necessarily call @code{__del__()} – it simply
decrements the object’s reference count, and if this reaches zero
@code{__del__()} is called.

If your data structures contain circular links (e.g. a tree where each child has
a parent reference and each parent has a list of children) the reference counts
will never go back to zero.  Once in a while Python runs an algorithm to detect
such cycles, but the garbage collector might run some time after the last
reference to your data structure vanishes, so your @code{__del__()} method may be
called at an inconvenient and random time. This is inconvenient if you’re trying
to reproduce a problem. Worse, the order in which object’s @code{__del__()}
methods are executed is arbitrary.  You can run @ref{54b,,gc.collect()} to force a
collection, but there @emph{are} pathological cases where objects will never be
collected.

Despite the cycle collector, it’s still a good idea to define an explicit
@code{close()} method on objects to be called whenever you’re done with them.  The
@code{close()} method can then remove attributes that refer to subobjects.  Don’t
call @code{__del__()} directly – @code{__del__()} should call @code{close()} and
@code{close()} should make sure that it can be called more than once for the same
object.

Another way to avoid cyclical references is to use the @ref{12f,,weakref} module,
which allows you to point to objects without incrementing their reference count.
Tree data structures, for instance, should use weak references for their parent
and sibling references (if they need them!).

@c XXX relevant for Python 3?
@c 
@c If the object has ever been a local variable in a function that caught an
@c expression in an except clause, chances are that a reference to the object
@c still exists in that function's stack frame as contained in the stack trace.
@c Normally, calling :func:`sys.exc_clear` will take care of this by clearing
@c the last recorded exception.

Finally, if your @code{__del__()} method raises an exception, a warning message
is printed to @ref{42d,,sys.stderr}.

@node How do I get a list of all instances of a given class?,Why does the result of id appear to be not unique?,My class defines __del__ but it is not called when I delete the object,Objects
@anchor{faq/programming how-do-i-get-a-list-of-all-instances-of-a-given-class}@anchor{481a}
@subsubsection How do I get a list of all instances of a given class?


Python does not keep track of all instances of a class (or of a built-in type).
You can program the class’s constructor to keep track of all instances by
keeping a list of weak references to each instance.

@node Why does the result of id appear to be not unique?,When can I rely on identity tests with the is operator?,How do I get a list of all instances of a given class?,Objects
@anchor{faq/programming why-does-the-result-of-id-appear-to-be-not-unique}@anchor{481b}
@subsubsection Why does the result of @code{id()} appear to be not unique?


The @ref{f80,,id()} builtin returns an integer that is guaranteed to be unique during
the lifetime of the object.  Since in CPython, this is the object’s memory
address, it happens frequently that after an object is deleted from memory, the
next freshly created object is allocated at the same position in memory.  This
is illustrated by this example:

@example
>>> id(1000) 
13901272
>>> id(2000) 
13901272
@end example

The two ids belong to different integer objects that are created before, and
deleted immediately after execution of the @code{id()} call.  To be sure that
objects whose id you want to examine are still alive, create another reference
to the object:

@example
>>> a = 1000; b = 2000
>>> id(a) 
13901272
>>> id(b) 
13891296
@end example

@node When can I rely on identity tests with the is operator?,How can a subclass control what data is stored in an immutable instance?,Why does the result of id appear to be not unique?,Objects
@anchor{faq/programming when-can-i-rely-on-identity-tests-with-the-is-operator}@anchor{481c}
@subsubsection When can I rely on identity tests with the @emph{is} operator?


The @code{is} operator tests for object identity.  The test @code{a is b} is
equivalent to @code{id(a) == id(b)}.

The most important property of an identity test is that an object is always
identical to itself, @code{a is a} always returns @code{True}.  Identity tests are
usually faster than equality tests.  And unlike equality tests, identity tests
are guaranteed to return a boolean @code{True} or @code{False}.

However, identity tests can @emph{only} be substituted for equality tests when
object identity is assured.  Generally, there are three circumstances where
identity is guaranteed:

1) Assignments create new names but do not change object identity.  After the
assignment @code{new = old}, it is guaranteed that @code{new is old}.

2) Putting an object in a container that stores object references does not
change object identity.  After the list assignment @code{s[0] = x}, it is
guaranteed that @code{s[0] is x}.

3) If an object is a singleton, it means that only one instance of that object
can exist.  After the assignments @code{a = None} and @code{b = None}, it is
guaranteed that @code{a is b} because @code{None} is a singleton.

In most other circumstances, identity tests are inadvisable and equality tests
are preferred.  In particular, identity tests should not be used to check
constants such as @ref{1b1,,int} and @ref{2ef,,str} which aren’t guaranteed to be
singletons:

@example
>>> a = 1000
>>> b = 500
>>> c = b + 500
>>> a is c
False

>>> a = 'Python'
>>> b = 'Py'
>>> c = b + 'thon'
>>> a is c
False
@end example

Likewise, new instances of mutable containers are never identical:

@example
>>> a = []
>>> b = []
>>> a is b
False
@end example

In the standard library code, you will see several common patterns for
correctly using identity tests:

1) As recommended by 
@geindex Python Enhancement Proposals; PEP 8
PEP 8@footnote{https://www.python.org/dev/peps/pep-0008}, an identity test is the preferred way to check
for @code{None}.  This reads like plain English in code and avoids confusion with
other objects that may have boolean values that evaluate to false.

2) Detecting optional arguments can be tricky when @code{None} is a valid input
value.  In those situations, you can create a singleton sentinel object
guaranteed to be distinct from other objects.  For example, here is how
to implement a method that behaves like @ref{1b30,,dict.pop()}:

@example
_sentinel = object()

def pop(self, key, default=_sentinel):
    if key in self:
        value = self[key]
        del self[key]
        return value
    if default is _sentinel:
        raise KeyError(key)
    return default
@end example

3) Container implementations sometimes need to augment equality tests with
identity tests.  This prevents the code from being confused by objects such as
@code{float('NaN')} that are not equal to themselves.

For example, here is the implementation of
@code{collections.abc.Sequence.__contains__()}:

@example
def __contains__(self, value):
    for v in self:
        if v is value or v == value:
            return True
    return False
@end example

@node How can a subclass control what data is stored in an immutable instance?,How do I cache method calls?,When can I rely on identity tests with the is operator?,Objects
@anchor{faq/programming how-can-a-subclass-control-what-data-is-stored-in-an-immutable-instance}@anchor{481d}
@subsubsection How can a subclass control what data is stored in an immutable instance?


When subclassing an immutable type, override the @code{__new__()} method
instead of the @code{__init__()} method.  The latter only runs @emph{after} an
instance is created, which is too late to alter data in an immutable
instance.

All of these immutable classes have a different signature than their
parent class:

@example
from datetime import date

class FirstOfMonthDate(date):
    "Always choose the first day of the month"
    def __new__(cls, year, month, day):
        return super().__new__(cls, year, month, 1)

class NamedInt(int):
    "Allow text names for some numbers"
    xlat = @{'zero': 0, 'one': 1, 'ten': 10@}
    def __new__(cls, value):
        value = cls.xlat.get(value, value)
        return super().__new__(cls, value)

class TitleStr(str):
    "Convert str to name suitable for a URL path"
    def __new__(cls, s):
        s = s.lower().replace(' ', '-')
        s = '@w{'}.join([c for c in s if c.isalnum() or c == '-'])
        return super().__new__(cls, s)
@end example

The classes can be used like this:

@example
>>> FirstOfMonthDate(2012, 2, 14)
FirstOfMonthDate(2012, 2, 1)
>>> NamedInt('ten')
10
>>> NamedInt(20)
20
>>> TitleStr('Blog: Why Python Rocks')
'blog-why-python-rocks'
@end example

@node How do I cache method calls?,,How can a subclass control what data is stored in an immutable instance?,Objects
@anchor{faq/programming how-do-i-cache-method-calls}@anchor{481e}
@subsubsection How do I cache method calls?


The two principal tools for caching methods are
@ref{4f4,,functools.cached_property()} and @ref{4f3,,functools.lru_cache()}.  The
former stores results at the instance level and the latter at the class
level.

The @emph{cached_property} approach only works with methods that do not take
any arguments.  It does not create a reference to the instance.  The
cached method result will be kept only as long as the instance is alive.

The advantage is that when an instance is no longer used, the cached
method result will be released right away.  The disadvantage is that if
instances accumulate, so too will the accumulated method results.  They
can grow without bound.

The @emph{lru_cache} approach works with methods that have hashable
arguments.  It creates a reference to the instance unless special
efforts are made to pass in weak references.

The advantage of the least recently used algorithm is that the cache is
bounded by the specified @emph{maxsize}.  The disadvantage is that instances
are kept alive until they age out of the cache or until the cache is
cleared.

This example shows the various techniques:

@example
class Weather:
    "Lookup weather information on a government website"

    def __init__(self, station_id):
        self._station_id = station_id
        # The _station_id is private and immutable

    def current_temperature(self):
        "Latest hourly observation"
        # Do not cache this because old results
        # can be out of date.

    @@cached_property
    def location(self):
        "Return the longitude/latitude coordinates of the station"
        # Result only depends on the station_id

    @@lru_cache(maxsize=20)
    def historic_rainfall(self, date, units='mm'):
        "Rainfall on a given date"
        # Depends on the station_id, date, and units.
@end example

The above example assumes that the @emph{station_id} never changes.  If the
relevant instance attributes are mutable, the @emph{cached_property} approach
can’t be made to work because it cannot detect changes to the
attributes.

The @emph{lru_cache} approach can be made to work, but the class needs to define the
@emph{__eq__} and @emph{__hash__} methods so the cache can detect relevant attribute
updates:

@example
class Weather:
    "Example with a mutable station identifier"

    def __init__(self, station_id):
        self.station_id = station_id

    def change_station(self, station_id):
        self.station_id = station_id

    def __eq__(self, other):
        return self.station_id == other.station_id

    def __hash__(self):
        return hash(self.station_id)

    @@lru_cache(maxsize=20)
    def historic_rainfall(self, date, units='cm'):
        'Rainfall on a given date'
        # Depends on the station_id, date, and units.
@end example

@node Modules<3>,,Objects,Programming FAQ
@anchor{faq/programming modules}@anchor{481f}
@subsection Modules


@menu
* How do I create a .pyc file?: How do I create a pyc file?. 
* How do I find the current module name?:: 
* How can I have modules that mutually import each other?:: 
* __import__(‘x.y.z’) returns <module ‘x’>; how do I get z?: __import__ ‘x y z’ returns <module ‘x’>; how do I get z?. 
* When I edit an imported module and reimport it@comma{} the changes don’t show up. Why does this happen?: When I edit an imported module and reimport it the changes don’t show up Why does this happen?. 

@end menu

@node How do I create a pyc file?,How do I find the current module name?,,Modules<3>
@anchor{faq/programming how-do-i-create-a-pyc-file}@anchor{4820}
@subsubsection How do I create a .pyc file?


When a module is imported for the first time (or when the source file has
changed since the current compiled file was created) a @code{.pyc} file containing
the compiled code should be created in a @code{__pycache__} subdirectory of the
directory containing the @code{.py} file.  The @code{.pyc} file will have a
filename that starts with the same name as the @code{.py} file, and ends with
@code{.pyc}, with a middle component that depends on the particular @code{python}
binary that created it.  (See 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} for details.)

One reason that a @code{.pyc} file may not be created is a permissions problem
with the directory containing the source file, meaning that the @code{__pycache__}
subdirectory cannot be created. This can happen, for example, if you develop as
one user but run as another, such as if you are testing with a web server.

Unless the 
@geindex PYTHONDONTWRITEBYTECODE
@geindex environment variable; PYTHONDONTWRITEBYTECODE
@ref{f35,,PYTHONDONTWRITEBYTECODE} environment variable is set,
creation of a .pyc file is automatic if you’re importing a module and Python
has the ability (permissions, free space, etc…) to create a @code{__pycache__}
subdirectory and write the compiled module to that subdirectory.

Running Python on a top level script is not considered an import and no
@code{.pyc} will be created.  For example, if you have a top-level module
@code{foo.py} that imports another module @code{xyz.py}, when you run @code{foo} (by
typing @code{python foo.py} as a shell command), a @code{.pyc} will be created for
@code{xyz} because @code{xyz} is imported, but no @code{.pyc} file will be created for
@code{foo} since @code{foo.py} isn’t being imported.

If you need to create a @code{.pyc} file for @code{foo} – that is, to create a
@code{.pyc} file for a module that is not imported – you can, using the
@ref{d2,,py_compile} and @ref{1f,,compileall} modules.

The @ref{d2,,py_compile} module can manually compile any module.  One way is to use
the @code{compile()} function in that module interactively:

@example
>>> import py_compile
>>> py_compile.compile('foo.py')                 
@end example

This will write the @code{.pyc} to a @code{__pycache__} subdirectory in the same
location as @code{foo.py} (or you can override that with the optional parameter
@code{cfile}).

You can also automatically compile all files in a directory or directories using
the @ref{1f,,compileall} module.  You can do it from the shell prompt by running
@code{compileall.py} and providing the path of a directory containing Python files
to compile:

@example
python -m compileall .
@end example

@node How do I find the current module name?,How can I have modules that mutually import each other?,How do I create a pyc file?,Modules<3>
@anchor{faq/programming how-do-i-find-the-current-module-name}@anchor{4821}
@subsubsection How do I find the current module name?


A module can find out its own module name by looking at the predefined global
variable @code{__name__}.  If this has the value @code{'__main__'}, the program is
running as a script.  Many modules that are usually used by importing them also
provide a command-line interface or a self-test, and only execute this code
after checking @code{__name__}:

@example
def main():
    print('Running test...')
    ...

if __name__ == '__main__':
    main()
@end example

@node How can I have modules that mutually import each other?,__import__ ‘x y z’ returns <module ‘x’>; how do I get z?,How do I find the current module name?,Modules<3>
@anchor{faq/programming how-can-i-have-modules-that-mutually-import-each-other}@anchor{4822}
@subsubsection How can I have modules that mutually import each other?


Suppose you have the following modules:

@code{foo.py}:

@example
from bar import bar_var
foo_var = 1
@end example

@code{bar.py}:

@example
from foo import foo_var
bar_var = 2
@end example

The problem is that the interpreter will perform the following steps:


@itemize *

@item 
main imports @code{foo}

@item 
Empty globals for @code{foo} are created

@item 
@code{foo} is compiled and starts executing

@item 
@code{foo} imports @code{bar}

@item 
Empty globals for @code{bar} are created

@item 
@code{bar} is compiled and starts executing

@item 
@code{bar} imports @code{foo} (which is a no-op since there already is a module named @code{foo})

@item 
The import mechanism tries to read @code{foo_var} from @code{foo} globals, to set @code{bar.foo_var = foo.foo_var}
@end itemize

The last step fails, because Python isn’t done with interpreting @code{foo} yet and
the global symbol dictionary for @code{foo} is still empty.

The same thing happens when you use @code{import foo}, and then try to access
@code{foo.foo_var} in global code.

There are (at least) three possible workarounds for this problem.

Guido van Rossum recommends avoiding all uses of @code{from <module> import ...},
and placing all code inside functions.  Initializations of global variables and
class variables should use constants or built-in functions only.  This means
everything from an imported module is referenced as @code{<module>.<name>}.

Jim Roskind suggests performing steps in the following order in each module:


@itemize *

@item 
exports (globals, functions, and classes that don’t need imported base
classes)

@item 
@code{import} statements

@item 
active code (including globals that are initialized from imported values).
@end itemize

Van Rossum doesn’t like this approach much because the imports appear in a
strange place, but it does work.

Matthias Urlichs recommends restructuring your code so that the recursive import
is not necessary in the first place.

These solutions are not mutually exclusive.

@node __import__ ‘x y z’ returns <module ‘x’>; how do I get z?,When I edit an imported module and reimport it the changes don’t show up Why does this happen?,How can I have modules that mutually import each other?,Modules<3>
@anchor{faq/programming import-x-y-z-returns-module-x-how-do-i-get-z}@anchor{4823}
@subsubsection __import__(‘x.y.z’) returns <module ‘x’>; how do I get z?


Consider using the convenience function @ref{d3e,,import_module()} from
@ref{96,,importlib} instead:

@example
z = importlib.import_module('x.y.z')
@end example

@node When I edit an imported module and reimport it the changes don’t show up Why does this happen?,,__import__ ‘x y z’ returns <module ‘x’>; how do I get z?,Modules<3>
@anchor{faq/programming when-i-edit-an-imported-module-and-reimport-it-the-changes-don-t-show-up-why-does-this-happen}@anchor{4824}
@subsubsection When I edit an imported module and reimport it, the changes don’t show up.  Why does this happen?


For reasons of efficiency as well as consistency, Python only reads the module
file on the first time a module is imported.  If it didn’t, in a program
consisting of many modules where each one imports the same basic module, the
basic module would be parsed and re-parsed many times.  To force re-reading of a
changed module, do this:

@example
import importlib
import modname
importlib.reload(modname)
@end example

Warning: this technique is not 100% fool-proof.  In particular, modules
containing statements like

@example
from modname import some_objects
@end example

will continue to work with the old version of the imported objects.  If the
module contains class definitions, existing class instances will @emph{not} be
updated to use the new class definition.  This can result in the following
paradoxical behaviour:

@example
>>> import importlib
>>> import cls
>>> c = cls.C()                # Create an instance of C
>>> importlib.reload(cls)
<module 'cls' from 'cls.py'>
>>> isinstance(c, cls.C)       # isinstance is false?!?
False
@end example

The nature of the problem is made clear if you print out the “identity” of the
class objects:

@example
>>> hex(id(c.__class__))
'0x7352a0'
>>> hex(id(cls.C))
'0x4198d0'
@end example

@node Design and History FAQ,Library and Extension FAQ,Programming FAQ,Python Frequently Asked Questions
@anchor{faq/design doc}@anchor{4825}@anchor{faq/design design-and-history-faq}@anchor{4826}
@section Design and History FAQ



@menu
* Why does Python use indentation for grouping of statements?:: 
* Why am I getting strange results with simple arithmetic operations?:: 
* Why are floating-point calculations so inaccurate?:: 
* Why are Python strings immutable?:: 
* Why must ‘self’ be used explicitly in method definitions and calls?:: 
* Why can’t I use an assignment in an expression?:: 
* Why does Python use methods for some functionality (e.g. list.index()) but functions for other (e.g. len(list))?: Why does Python use methods for some functionality e g list index but functions for other e g len list ?. 
* Why is join() a string method instead of a list or tuple method?: Why is join a string method instead of a list or tuple method?. 
* How fast are exceptions?:: 
* Why isn’t there a switch or case statement in Python?:: 
* Can’t you emulate threads in the interpreter instead of relying on an OS-specific thread implementation?:: 
* Why can’t lambda expressions contain statements?:: 
* Can Python be compiled to machine code@comma{} C or some other language?: Can Python be compiled to machine code C or some other language?. 
* How does Python manage memory?:: 
* Why doesn’t CPython use a more traditional garbage collection scheme?:: 
* Why isn’t all memory freed when CPython exits?:: 
* Why are there separate tuple and list data types?:: 
* How are lists implemented in CPython?:: 
* How are dictionaries implemented in CPython?:: 
* Why must dictionary keys be immutable?:: 
* Why doesn’t list.sort() return the sorted list?: Why doesn’t list sort return the sorted list?. 
* How do you specify and enforce an interface spec in Python?:: 
* Why is there no goto?:: 
* Why can’t raw strings (r-strings) end with a backslash?: Why can’t raw strings r-strings end with a backslash?. 
* Why doesn’t Python have a “with” statement for attribute assignments?:: 
* Why don’t generators support the with statement?:: 
* Why are colons required for the if/while/def/class statements?:: 
* Why does Python allow commas at the end of lists and tuples?:: 

@end menu

@node Why does Python use indentation for grouping of statements?,Why am I getting strange results with simple arithmetic operations?,,Design and History FAQ
@anchor{faq/design why-does-python-use-indentation-for-grouping-of-statements}@anchor{4827}
@subsection Why does Python use indentation for grouping of statements?


Guido van Rossum believes that using indentation for grouping is extremely
elegant and contributes a lot to the clarity of the average Python program.
Most people learn to love this feature after a while.

Since there are no begin/end brackets there cannot be a disagreement between
grouping perceived by the parser and the human reader.  Occasionally C
programmers will encounter a fragment of code like this:

@example
if (x <= y)
        x++;
        y--;
z++;
@end example

Only the @code{x++} statement is executed if the condition is true, but the
indentation leads many to believe otherwise.  Even experienced C programmers will
sometimes stare at it a long time wondering as to why @code{y} is being decremented even
for @code{x > y}.

Because there are no begin/end brackets, Python is much less prone to
coding-style conflicts.  In C there are many different ways to place the braces.
After becoming used to reading and writing code using a particular style,
it is normal to feel somewhat uneasy when reading (or being required to write)
in a different one.

Many coding styles place begin/end brackets on a line by themselves.  This makes
programs considerably longer and wastes valuable screen space, making it harder
to get a good overview of a program.  Ideally, a function should fit on one
screen (say, 20–30 lines).  20 lines of Python can do a lot more work than 20
lines of C.  This is not solely due to the lack of begin/end brackets – the
lack of declarations and the high-level data types are also responsible – but
the indentation-based syntax certainly helps.

@node Why am I getting strange results with simple arithmetic operations?,Why are floating-point calculations so inaccurate?,Why does Python use indentation for grouping of statements?,Design and History FAQ
@anchor{faq/design why-am-i-getting-strange-results-with-simple-arithmetic-operations}@anchor{4828}
@subsection Why am I getting strange results with simple arithmetic operations?


See the next question.

@node Why are floating-point calculations so inaccurate?,Why are Python strings immutable?,Why am I getting strange results with simple arithmetic operations?,Design and History FAQ
@anchor{faq/design why-are-floating-point-calculations-so-inaccurate}@anchor{4829}
@subsection Why are floating-point calculations so inaccurate?


Users are often surprised by results like this:

@example
>>> 1.2 - 1.0
0.19999999999999996
@end example

and think it is a bug in Python.  It’s not.  This has little to do with Python,
and much more to do with how the underlying platform handles floating-point
numbers.

The @ref{256,,float} type in CPython uses a C @code{double} for storage.  A
@ref{256,,float} object’s value is stored in binary floating-point with a fixed
precision (typically 53 bits) and Python uses C operations, which in turn rely
on the hardware implementation in the processor, to perform floating-point
operations. This means that as far as floating-point operations are concerned,
Python behaves like many popular languages including C and Java.

Many numbers that can be written easily in decimal notation cannot be expressed
exactly in binary floating-point.  For example, after:

@example
>>> x = 1.2
@end example

the value stored for @code{x} is a (very good) approximation to the decimal value
@code{1.2}, but is not exactly equal to it.  On a typical machine, the actual
stored value is:

@example
1.0011001100110011001100110011001100110011001100110011 (binary)
@end example

which is exactly:

@example
1.1999999999999999555910790149937383830547332763671875 (decimal)
@end example

The typical precision of 53 bits provides Python floats with 15–16
decimal digits of accuracy.

For a fuller explanation, please see the @ref{15e5,,floating point arithmetic} chapter in the Python tutorial.

@node Why are Python strings immutable?,Why must ‘self’ be used explicitly in method definitions and calls?,Why are floating-point calculations so inaccurate?,Design and History FAQ
@anchor{faq/design why-are-python-strings-immutable}@anchor{482a}
@subsection Why are Python strings immutable?


There are several advantages.

One is performance: knowing that a string is immutable means we can allocate
space for it at creation time, and the storage requirements are fixed and
unchanging.  This is also one of the reasons for the distinction between tuples
and lists.

Another advantage is that strings in Python are considered as “elemental” as
numbers.  No amount of activity will change the value 8 to anything else, and in
Python, no amount of activity will change the string “eight” to anything else.

@node Why must ‘self’ be used explicitly in method definitions and calls?,Why can’t I use an assignment in an expression?,Why are Python strings immutable?,Design and History FAQ
@anchor{faq/design why-must-self-be-used-explicitly-in-method-definitions-and-calls}@anchor{482b}@anchor{faq/design why-self}@anchor{4811}
@subsection Why must ‘self’ be used explicitly in method definitions and calls?


The idea was borrowed from Modula-3.  It turns out to be very useful, for a
variety of reasons.

First, it’s more obvious that you are using a method or instance attribute
instead of a local variable.  Reading @code{self.x} or @code{self.meth()} makes it
absolutely clear that an instance variable or method is used even if you don’t
know the class definition by heart.  In C++, you can sort of tell by the lack of
a local variable declaration (assuming globals are rare or easily recognizable)
– but in Python, there are no local variable declarations, so you’d have to
look up the class definition to be sure.  Some C++ and Java coding standards
call for instance attributes to have an @code{m_} prefix, so this explicitness is
still useful in those languages, too.

Second, it means that no special syntax is necessary if you want to explicitly
reference or call the method from a particular class.  In C++, if you want to
use a method from a base class which is overridden in a derived class, you have
to use the @code{::} operator – in Python you can write
@code{baseclass.methodname(self, <argument list>)}.  This is particularly useful
for @code{__init__()} methods, and in general in cases where a derived class
method wants to extend the base class method of the same name and thus has to
call the base class method somehow.

Finally, for instance variables it solves a syntactic problem with assignment:
since local variables in Python are (by definition!) those variables to which a
value is assigned in a function body (and that aren’t explicitly declared
global), there has to be some way to tell the interpreter that an assignment was
meant to assign to an instance variable instead of to a local variable, and it
should preferably be syntactic (for efficiency reasons).  C++ does this through
declarations, but Python doesn’t have declarations and it would be a pity having
to introduce them just for this purpose.  Using the explicit @code{self.var} solves
this nicely.  Similarly, for using instance variables, having to write
@code{self.var} means that references to unqualified names inside a method don’t
have to search the instance’s directories.  To put it another way, local
variables and instance variables live in two different namespaces, and you need
to tell Python which namespace to use.

@node Why can’t I use an assignment in an expression?,Why does Python use methods for some functionality e g list index but functions for other e g len list ?,Why must ‘self’ be used explicitly in method definitions and calls?,Design and History FAQ
@anchor{faq/design id1}@anchor{482c}@anchor{faq/design why-can-t-i-use-an-assignment-in-an-expression}@anchor{1538}
@subsection Why can’t I use an assignment in an expression?


Starting in Python 3.8, you can!

Assignment expressions using the walrus operator @cite{:=} assign a variable in an
expression:

@example
while chunk := fp.read(200):
   print(chunk)
@end example

See 
@geindex Python Enhancement Proposals; PEP 572
PEP 572@footnote{https://www.python.org/dev/peps/pep-0572} for more information.

@node Why does Python use methods for some functionality e g list index but functions for other e g len list ?,Why is join a string method instead of a list or tuple method?,Why can’t I use an assignment in an expression?,Design and History FAQ
@anchor{faq/design why-does-python-use-methods-for-some-functionality-e-g-list-index-but-functions-for-other-e-g-len-list}@anchor{482d}
@subsection Why does Python use methods for some functionality (e.g. list.index()) but functions for other (e.g. len(list))?


As Guido said:

@quotation

(a) For some operations, prefix notation just reads better than
postfix – prefix (and infix!) operations have a long tradition in
mathematics which likes notations where the visuals help the
mathematician thinking about a problem. Compare the easy with which we
rewrite a formula like x*(a+b) into x*a + x*b to the clumsiness of
doing the same thing using a raw OO notation.

(b) When I read code that says len(x) I @emph{know} that it is asking for
the length of something. This tells me two things: the result is an
integer, and the argument is some kind of container. To the contrary,
when I read x.len(), I have to already know that x is some kind of
container implementing an interface or inheriting from a class that
has a standard len(). Witness the confusion we occasionally have when
a class that is not implementing a mapping has a get() or keys()
method, or something that isn’t a file has a write() method.


@center --- @indicateurl{https://mail.python.org/pipermail/python-3000/2006-November/004643.html}

@end quotation

@node Why is join a string method instead of a list or tuple method?,How fast are exceptions?,Why does Python use methods for some functionality e g list index but functions for other e g len list ?,Design and History FAQ
@anchor{faq/design why-is-join-a-string-method-instead-of-a-list-or-tuple-method}@anchor{482e}
@subsection Why is join() a string method instead of a list or tuple method?


Strings became much more like other standard types starting in Python 1.6, when
methods were added which give the same functionality that has always been
available using the functions of the string module.  Most of these new methods
have been widely accepted, but the one which appears to make some programmers
feel uncomfortable is:

@example
", ".join(['1', '2', '4', '8', '16'])
@end example

which gives the result:

@example
"1, 2, 4, 8, 16"
@end example

There are two common arguments against this usage.

The first runs along the lines of: “It looks really ugly using a method of a
string literal (string constant)”, to which the answer is that it might, but a
string literal is just a fixed value. If the methods are to be allowed on names
bound to strings there is no logical reason to make them unavailable on
literals.

The second objection is typically cast as: “I am really telling a sequence to
join its members together with a string constant”.  Sadly, you aren’t.  For some
reason there seems to be much less difficulty with having @ref{a1b,,split()} as
a string method, since in that case it is easy to see that

@example
"1, 2, 4, 8, 16".split(", ")
@end example

is an instruction to a string literal to return the substrings delimited by the
given separator (or, by default, arbitrary runs of white space).

@ref{1a8b,,join()} is a string method because in using it you are telling the
separator string to iterate over a sequence of strings and insert itself between
adjacent elements.  This method can be used with any argument which obeys the
rules for sequence objects, including any new classes you might define yourself.
Similar methods exist for bytes and bytearray objects.

@node How fast are exceptions?,Why isn’t there a switch or case statement in Python?,Why is join a string method instead of a list or tuple method?,Design and History FAQ
@anchor{faq/design how-fast-are-exceptions}@anchor{482f}
@subsection How fast are exceptions?


A try/except block is extremely efficient if no exceptions are raised.  Actually
catching an exception is expensive.  In versions of Python prior to 2.0 it was
common to use this idiom:

@example
try:
    value = mydict[key]
except KeyError:
    mydict[key] = getvalue(key)
    value = mydict[key]
@end example

This only made sense when you expected the dict to have the key almost all the
time.  If that wasn’t the case, you coded it like this:

@example
if key in mydict:
    value = mydict[key]
else:
    value = mydict[key] = getvalue(key)
@end example

For this specific case, you could also use @code{value = dict.setdefault(key,
getvalue(key))}, but only if the @code{getvalue()} call is cheap enough because it
is evaluated in all cases.

@node Why isn’t there a switch or case statement in Python?,Can’t you emulate threads in the interpreter instead of relying on an OS-specific thread implementation?,How fast are exceptions?,Design and History FAQ
@anchor{faq/design why-isn-t-there-a-switch-or-case-statement-in-python}@anchor{4830}
@subsection Why isn’t there a switch or case statement in Python?


You can do this easily enough with a sequence of @code{if... elif... elif... else}.
For literal values, or constants within a namespace, you can also use a
@code{match ... case} statement.

For cases where you need to choose from a very large number of possibilities,
you can create a dictionary mapping case values to functions to call.  For
example:

@example
functions = @{'a': function_1,
             'b': function_2,
             'c': self.method_1@}

func = functions[value]
func()
@end example

For calling methods on objects, you can simplify yet further by using the
@ref{708,,getattr()} built-in to retrieve methods with a particular name:

@example
class MyVisitor:
    def visit_a(self):
        ...

    def dispatch(self, value):
        method_name = 'visit_' + str(value)
        method = getattr(self, method_name)
        method()
@end example

It’s suggested that you use a prefix for the method names, such as @code{visit_} in
this example.  Without such a prefix, if values are coming from an untrusted
source, an attacker would be able to call any method on your object.

@node Can’t you emulate threads in the interpreter instead of relying on an OS-specific thread implementation?,Why can’t lambda expressions contain statements?,Why isn’t there a switch or case statement in Python?,Design and History FAQ
@anchor{faq/design can-t-you-emulate-threads-in-the-interpreter-instead-of-relying-on-an-os-specific-thread-implementation}@anchor{4831}
@subsection Can’t you emulate threads in the interpreter instead of relying on an OS-specific thread implementation?


Answer 1: Unfortunately, the interpreter pushes at least one C stack frame for
each Python stack frame.  Also, extensions can call back into Python at almost
random moments.  Therefore, a complete threads implementation requires thread
support for C.

Answer 2: Fortunately, there is Stackless Python@footnote{https://github.com/stackless-dev/stackless/wiki},
which has a completely redesigned interpreter loop that avoids the C stack.

@node Why can’t lambda expressions contain statements?,Can Python be compiled to machine code C or some other language?,Can’t you emulate threads in the interpreter instead of relying on an OS-specific thread implementation?,Design and History FAQ
@anchor{faq/design why-can-t-lambda-expressions-contain-statements}@anchor{4832}
@subsection Why can’t lambda expressions contain statements?


Python lambda expressions cannot contain statements because Python’s syntactic
framework can’t handle statements nested inside expressions.  However, in
Python, this is not a serious problem.  Unlike lambda forms in other languages,
where they add functionality, Python lambdas are only a shorthand notation if
you’re too lazy to define a function.

Functions are already first class objects in Python, and can be declared in a
local scope.  Therefore the only advantage of using a lambda instead of a
locally-defined function is that you don’t need to invent a name for the
function – but that’s just a local variable to which the function object (which
is exactly the same type of object that a lambda expression yields) is assigned!

@node Can Python be compiled to machine code C or some other language?,How does Python manage memory?,Why can’t lambda expressions contain statements?,Design and History FAQ
@anchor{faq/design can-python-be-compiled-to-machine-code-c-or-some-other-language}@anchor{4833}
@subsection Can Python be compiled to machine code, C or some other language?


Cython@footnote{http://cython.org/} compiles a modified version of Python with
optional annotations into C extensions.  Nuitka@footnote{http://www.nuitka.net/} is
an up-and-coming compiler of Python into C++ code, aiming to support the full
Python language. For compiling to Java you can consider
VOC@footnote{https://voc.readthedocs.io}.

@node How does Python manage memory?,Why doesn’t CPython use a more traditional garbage collection scheme?,Can Python be compiled to machine code C or some other language?,Design and History FAQ
@anchor{faq/design how-does-python-manage-memory}@anchor{4834}
@subsection How does Python manage memory?


The details of Python memory management depend on the implementation.  The
standard implementation of Python, @ref{17a4,,CPython}, uses reference counting to
detect inaccessible objects, and another mechanism to collect reference cycles,
periodically executing a cycle detection algorithm which looks for inaccessible
cycles and deletes the objects involved. The @ref{80,,gc} module provides functions
to perform a garbage collection, obtain debugging statistics, and tune the
collector’s parameters.

Other implementations (such as Jython@footnote{http://www.jython.org} or
PyPy@footnote{http://www.pypy.org}), however, can rely on a different mechanism
such as a full-blown garbage collector.  This difference can cause some
subtle porting problems if your Python code depends on the behavior of the
reference counting implementation.

In some Python implementations, the following code (which is fine in CPython)
will probably run out of file descriptors:

@example
for file in very_long_list_of_files:
    f = open(file)
    c = f.read(1)
@end example

Indeed, using CPython’s reference counting and destructor scheme, each new
assignment to @emph{f} closes the previous file.  With a traditional GC, however,
those file objects will only get collected (and closed) at varying and possibly
long intervals.

If you want to write code that will work with any Python implementation,
you should explicitly close the file or use the @ref{168,,with} statement;
this will work regardless of memory management scheme:

@example
for file in very_long_list_of_files:
    with open(file) as f:
        c = f.read(1)
@end example

@node Why doesn’t CPython use a more traditional garbage collection scheme?,Why isn’t all memory freed when CPython exits?,How does Python manage memory?,Design and History FAQ
@anchor{faq/design why-doesn-t-cpython-use-a-more-traditional-garbage-collection-scheme}@anchor{4835}
@subsection Why doesn’t CPython use a more traditional garbage collection scheme?


For one thing, this is not a C standard feature and hence it’s not portable.
(Yes, we know about the Boehm GC library.  It has bits of assembler code for
@emph{most} common platforms, not for all of them, and although it is mostly
transparent, it isn’t completely transparent; patches are required to get
Python to work with it.)

Traditional GC also becomes a problem when Python is embedded into other
applications.  While in a standalone Python it’s fine to replace the standard
malloc() and free() with versions provided by the GC library, an application
embedding Python may want to have its @emph{own} substitute for malloc() and free(),
and may not want Python’s.  Right now, CPython works with anything that
implements malloc() and free() properly.

@node Why isn’t all memory freed when CPython exits?,Why are there separate tuple and list data types?,Why doesn’t CPython use a more traditional garbage collection scheme?,Design and History FAQ
@anchor{faq/design why-isn-t-all-memory-freed-when-cpython-exits}@anchor{4836}
@subsection Why isn’t all memory freed when CPython exits?


Objects referenced from the global namespaces of Python modules are not always
deallocated when Python exits.  This may happen if there are circular
references.  There are also certain bits of memory that are allocated by the C
library that are impossible to free (e.g. a tool like Purify will complain about
these).  Python is, however, aggressive about cleaning up memory on exit and
does try to destroy every single object.

If you want to force Python to delete certain things on deallocation use the
@ref{b,,atexit} module to run a function that will force those deletions.

@node Why are there separate tuple and list data types?,How are lists implemented in CPython?,Why isn’t all memory freed when CPython exits?,Design and History FAQ
@anchor{faq/design why-are-there-separate-tuple-and-list-data-types}@anchor{4837}
@subsection Why are there separate tuple and list data types?


Lists and tuples, while similar in many respects, are generally used in
fundamentally different ways.  Tuples can be thought of as being similar to
Pascal records or C structs; they’re small collections of related data which may
be of different types which are operated on as a group.  For example, a
Cartesian coordinate is appropriately represented as a tuple of two or three
numbers.

Lists, on the other hand, are more like arrays in other languages.  They tend to
hold a varying number of objects all of which have the same type and which are
operated on one-by-one.  For example, @code{os.listdir('.')} returns a list of
strings representing the files in the current directory.  Functions which
operate on this output would generally not break if you added another file or
two to the directory.

Tuples are immutable, meaning that once a tuple has been created, you can’t
replace any of its elements with a new value.  Lists are mutable, meaning that
you can always change a list’s elements.  Only immutable elements can be used as
dictionary keys, and hence only tuples and not lists can be used as keys.

@node How are lists implemented in CPython?,How are dictionaries implemented in CPython?,Why are there separate tuple and list data types?,Design and History FAQ
@anchor{faq/design how-are-lists-implemented-in-cpython}@anchor{4838}
@subsection How are lists implemented in CPython?


CPython’s lists are really variable-length arrays, not Lisp-style linked lists.
The implementation uses a contiguous array of references to other objects, and
keeps a pointer to this array and the array’s length in a list head structure.

This makes indexing a list @code{a[i]} an operation whose cost is independent of
the size of the list or the value of the index.

When items are appended or inserted, the array of references is resized.  Some
cleverness is applied to improve the performance of appending items repeatedly;
when the array must be grown, some extra space is allocated so the next few
times don’t require an actual resize.

@node How are dictionaries implemented in CPython?,Why must dictionary keys be immutable?,How are lists implemented in CPython?,Design and History FAQ
@anchor{faq/design how-are-dictionaries-implemented-in-cpython}@anchor{4839}
@subsection How are dictionaries implemented in CPython?


CPython’s dictionaries are implemented as resizable hash tables.  Compared to
B-trees, this gives better performance for lookup (the most common operation by
far) under most circumstances, and the implementation is simpler.

Dictionaries work by computing a hash code for each key stored in the dictionary
using the @ref{c14,,hash()} built-in function.  The hash code varies widely depending
on the key and a per-process seed; for example, “Python” could hash to
-539294296 while “python”, a string that differs by a single bit, could hash
to 1142331976.  The hash code is then used to calculate a location in an
internal array where the value will be stored.  Assuming that you’re storing
keys that all have different hash values, this means that dictionaries take
constant time – O(1), in Big-O notation – to retrieve a key.

@node Why must dictionary keys be immutable?,Why doesn’t list sort return the sorted list?,How are dictionaries implemented in CPython?,Design and History FAQ
@anchor{faq/design why-must-dictionary-keys-be-immutable}@anchor{483a}
@subsection Why must dictionary keys be immutable?


The hash table implementation of dictionaries uses a hash value calculated from
the key value to find the key.  If the key were a mutable object, its value
could change, and thus its hash could also change.  But since whoever changes
the key object can’t tell that it was being used as a dictionary key, it can’t
move the entry around in the dictionary.  Then, when you try to look up the same
object in the dictionary it won’t be found because its hash value is different.
If you tried to look up the old value it wouldn’t be found either, because the
value of the object found in that hash bin would be different.

If you want a dictionary indexed with a list, simply convert the list to a tuple
first; the function @code{tuple(L)} creates a tuple with the same entries as the
list @code{L}.  Tuples are immutable and can therefore be used as dictionary keys.

Some unacceptable solutions that have been proposed:


@itemize -

@item 
Hash lists by their address (object ID).  This doesn’t work because if you
construct a new list with the same value it won’t be found; e.g.:

@example
mydict = @{[1, 2]: '12'@}
print(mydict[[1, 2]])
@end example

would raise a @ref{5bb,,KeyError} exception because the id of the @code{[1, 2]} used in the
second line differs from that in the first line.  In other words, dictionary
keys should be compared using @code{==}, not using @ref{2fd,,is}.

@item 
Make a copy when using a list as a key.  This doesn’t work because the list,
being a mutable object, could contain a reference to itself, and then the
copying code would run into an infinite loop.

@item 
Allow lists as keys but tell the user not to modify them.  This would allow a
class of hard-to-track bugs in programs when you forgot or modified a list by
accident. It also invalidates an important invariant of dictionaries: every
value in @code{d.keys()} is usable as a key of the dictionary.

@item 
Mark lists as read-only once they are used as a dictionary key.  The problem
is that it’s not just the top-level object that could change its value; you
could use a tuple containing a list as a key.  Entering anything as a key into
a dictionary would require marking all objects reachable from there as
read-only – and again, self-referential objects could cause an infinite loop.
@end itemize

There is a trick to get around this if you need to, but use it at your own risk:
You can wrap a mutable structure inside a class instance which has both a
@code{__eq__()} and a @code{__hash__()} method.  You must then make sure that the
hash value for all such wrapper objects that reside in a dictionary (or other
hash based structure), remain fixed while the object is in the dictionary (or
other structure).

@example
class ListWrapper:
    def __init__(self, the_list):
        self.the_list = the_list

    def __eq__(self, other):
        return self.the_list == other.the_list

    def __hash__(self):
        l = self.the_list
        result = 98767 - len(l)*555
        for i, el in enumerate(l):
            try:
                result = result + (hash(el) % 9999999) * 1001 + i
            except Exception:
                result = (result % 7777777) + i * 333
        return result
@end example

Note that the hash computation is complicated by the possibility that some
members of the list may be unhashable and also by the possibility of arithmetic
overflow.

Furthermore it must always be the case that if @code{o1 == o2} (ie @code{o1.__eq__(o2)
is True}) then @code{hash(o1) == hash(o2)} (ie, @code{o1.__hash__() == o2.__hash__()}),
regardless of whether the object is in a dictionary or not.  If you fail to meet
these restrictions dictionaries and other hash based structures will misbehave.

In the case of ListWrapper, whenever the wrapper object is in a dictionary the
wrapped list must not change to avoid anomalies.  Don’t do this unless you are
prepared to think hard about the requirements and the consequences of not
meeting them correctly.  Consider yourself warned.

@node Why doesn’t list sort return the sorted list?,How do you specify and enforce an interface spec in Python?,Why must dictionary keys be immutable?,Design and History FAQ
@anchor{faq/design why-doesn-t-list-sort-return-the-sorted-list}@anchor{483b}
@subsection Why doesn’t list.sort() return the sorted list?


In situations where performance matters, making a copy of the list just to sort
it would be wasteful. Therefore, @ref{705,,list.sort()} sorts the list in place. In
order to remind you of that fact, it does not return the sorted list.  This way,
you won’t be fooled into accidentally overwriting a list when you need a sorted
copy but also need to keep the unsorted version around.

If you want to return a new list, use the built-in @ref{704,,sorted()} function
instead.  This function creates a new list from a provided iterable, sorts
it and returns it.  For example, here’s how to iterate over the keys of a
dictionary in sorted order:

@example
for key in sorted(mydict):
    ...  # do whatever with mydict[key]...
@end example

@node How do you specify and enforce an interface spec in Python?,Why is there no goto?,Why doesn’t list sort return the sorted list?,Design and History FAQ
@anchor{faq/design how-do-you-specify-and-enforce-an-interface-spec-in-python}@anchor{483c}
@subsection How do you specify and enforce an interface spec in Python?


An interface specification for a module as provided by languages such as C++ and
Java describes the prototypes for the methods and functions of the module.  Many
feel that compile-time enforcement of interface specifications helps in the
construction of large programs.

Python 2.6 adds an @ref{3,,abc} module that lets you define Abstract Base Classes
(ABCs).  You can then use @ref{236,,isinstance()} and @ref{237,,issubclass()} to check
whether an instance or a class implements a particular ABC.  The
@ref{1d,,collections.abc} module defines a set of useful ABCs such as
@ref{1b3c,,Iterable}, @ref{1b3e,,Container}, and
@ref{c3f,,MutableMapping}.

For Python, many of the advantages of interface specifications can be obtained
by an appropriate test discipline for components.

A good test suite for a module can both provide a regression test and serve as a
module interface specification and a set of examples.  Many Python modules can
be run as a script to provide a simple “self test.”  Even modules which use
complex external interfaces can often be tested in isolation using trivial
“stub” emulations of the external interface.  The @ref{63,,doctest} and
@ref{122,,unittest} modules or third-party test frameworks can be used to construct
exhaustive test suites that exercise every line of code in a module.

An appropriate testing discipline can help build large complex applications in
Python as well as having interface specifications would.  In fact, it can be
better because an interface specification cannot test certain properties of a
program.  For example, the @code{append()} method is expected to add new elements
to the end of some internal list; an interface specification cannot test that
your @code{append()} implementation will actually do this correctly, but it’s
trivial to check this property in a test suite.

Writing test suites is very helpful, and you might want to design your code to
make it easily tested. One increasingly popular technique, test-driven
development, calls for writing parts of the test suite first, before you write
any of the actual code.  Of course Python allows you to be sloppy and not write
test cases at all.

@node Why is there no goto?,Why can’t raw strings r-strings end with a backslash?,How do you specify and enforce an interface spec in Python?,Design and History FAQ
@anchor{faq/design why-is-there-no-goto}@anchor{483d}
@subsection Why is there no goto?


In the 1970s people realized that unrestricted goto could lead
to messy “spaghetti” code that was hard to understand and revise.
In a high-level language, it is also unneeded as long as there
are ways to branch (in Python, with @code{if} statements and @code{or},
@code{and}, and @code{if-else} expressions) and loop (with @code{while}
and @code{for} statements, possibly containing @code{continue} and @code{break}).

One can also use exceptions to provide a “structured goto”
that works even across
function calls.  Many feel that exceptions can conveniently emulate all
reasonable uses of the “go” or “goto” constructs of C, Fortran, and other
languages.  For example:

@example
class label(Exception): pass  # declare a label

try:
    ...
    if condition: raise label()  # goto label
    ...
except label:  # where to goto
    pass
...
@end example

This doesn’t allow you to jump into the middle of a loop, but that’s usually
considered an abuse of goto anyway.  Use sparingly.

@node Why can’t raw strings r-strings end with a backslash?,Why doesn’t Python have a “with” statement for attribute assignments?,Why is there no goto?,Design and History FAQ
@anchor{faq/design why-can-t-raw-strings-r-strings-end-with-a-backslash}@anchor{483e}
@subsection Why can’t raw strings (r-strings) end with a backslash?


More precisely, they can’t end with an odd number of backslashes: the unpaired
backslash at the end escapes the closing quote character, leaving an
unterminated string.

Raw strings were designed to ease creating input for processors (chiefly regular
expression engines) that want to do their own backslash escape processing. Such
processors consider an unmatched trailing backslash to be an error anyway, so
raw strings disallow that.  In return, they allow you to pass on the string
quote character by escaping it with a backslash.  These rules work well when
r-strings are used for their intended purpose.

If you’re trying to build Windows pathnames, note that all Windows system calls
accept forward slashes too:

@example
f = open("/mydir/file.txt")  # works fine!
@end example

If you’re trying to build a pathname for a DOS command, try e.g. one of

@example
dir = r"\this\is\my\dos\dir" "\\"
dir = r"\this\is\my\dos\dir\ "[:-1]
dir = "\\this\\is\\my\\dos\\dir\\"
@end example

@node Why doesn’t Python have a “with” statement for attribute assignments?,Why don’t generators support the with statement?,Why can’t raw strings r-strings end with a backslash?,Design and History FAQ
@anchor{faq/design why-doesn-t-python-have-a-with-statement-for-attribute-assignments}@anchor{483f}
@subsection Why doesn’t Python have a “with” statement for attribute assignments?


Python has a ‘with’ statement that wraps the execution of a block, calling code
on the entrance and exit from the block.  Some languages have a construct that
looks like this:

@example
with obj:
    a = 1               # equivalent to obj.a = 1
    total = total + 1   # obj.total = obj.total + 1
@end example

In Python, such a construct would be ambiguous.

Other languages, such as Object Pascal, Delphi, and C++, use static types, so
it’s possible to know, in an unambiguous way, what member is being assigned
to. This is the main point of static typing – the compiler @emph{always} knows the
scope of every variable at compile time.

Python uses dynamic types. It is impossible to know in advance which attribute
will be referenced at runtime. Member attributes may be added or removed from
objects on the fly. This makes it impossible to know, from a simple reading,
what attribute is being referenced: a local one, a global one, or a member
attribute?

For instance, take the following incomplete snippet:

@example
def foo(a):
    with a:
        print(x)
@end example

The snippet assumes that “a” must have a member attribute called “x”.  However,
there is nothing in Python that tells the interpreter this. What should happen
if “a” is, let us say, an integer?  If there is a global variable named “x”,
will it be used inside the with block?  As you see, the dynamic nature of Python
makes such choices much harder.

The primary benefit of “with” and similar language features (reduction of code
volume) can, however, easily be achieved in Python by assignment.  Instead of:

@example
function(args).mydict[index][index].a = 21
function(args).mydict[index][index].b = 42
function(args).mydict[index][index].c = 63
@end example

write this:

@example
ref = function(args).mydict[index][index]
ref.a = 21
ref.b = 42
ref.c = 63
@end example

This also has the side-effect of increasing execution speed because name
bindings are resolved at run-time in Python, and the second version only needs
to perform the resolution once.

@node Why don’t generators support the with statement?,Why are colons required for the if/while/def/class statements?,Why doesn’t Python have a “with” statement for attribute assignments?,Design and History FAQ
@anchor{faq/design why-don-t-generators-support-the-with-statement}@anchor{4840}
@subsection Why don’t generators support the with statement?


For technical reasons, a generator used directly as a context manager
would not work correctly.  When, as is most common, a generator is used as
an iterator run to completion, no closing is needed.  When it is, wrap
it as “contextlib.closing(generator)” in the ‘with’ statement.

@node Why are colons required for the if/while/def/class statements?,Why does Python allow commas at the end of lists and tuples?,Why don’t generators support the with statement?,Design and History FAQ
@anchor{faq/design why-are-colons-required-for-the-if-while-def-class-statements}@anchor{4841}
@subsection Why are colons required for the if/while/def/class statements?


The colon is required primarily to enhance readability (one of the results of
the experimental ABC language).  Consider this:

@example
if a == b
    print(a)
@end example

versus

@example
if a == b:
    print(a)
@end example

Notice how the second one is slightly easier to read.  Notice further how a
colon sets off the example in this FAQ answer; it’s a standard usage in English.

Another minor reason is that the colon makes it easier for editors with syntax
highlighting; they can look for colons to decide when indentation needs to be
increased instead of having to do a more elaborate parsing of the program text.

@node Why does Python allow commas at the end of lists and tuples?,,Why are colons required for the if/while/def/class statements?,Design and History FAQ
@anchor{faq/design why-does-python-allow-commas-at-the-end-of-lists-and-tuples}@anchor{4842}
@subsection Why does Python allow commas at the end of lists and tuples?


Python lets you add a trailing comma at the end of lists, tuples, and
dictionaries:

@example
[1, 2, 3,]
('a', 'b', 'c',)
d = @{
    "A": [1, 5],
    "B": [6, 7],  # last trailing comma is optional but good style
@}
@end example

There are several reasons to allow this.

When you have a literal value for a list, tuple, or dictionary spread across
multiple lines, it’s easier to add more elements because you don’t have to
remember to add a comma to the previous line.  The lines can also be reordered
without creating a syntax error.

Accidentally omitting the comma can lead to errors that are hard to diagnose.
For example:

@example
x = [
  "fee",
  "fie"
  "foo",
  "fum"
]
@end example

This list looks like it has four elements, but it actually contains three:
“fee”, “fiefoo” and “fum”.  Always adding the comma avoids this source of error.

Allowing the trailing comma may also make programmatic code generation easier.

@node Library and Extension FAQ,Extending/Embedding FAQ,Design and History FAQ,Python Frequently Asked Questions
@anchor{faq/library doc}@anchor{4843}@anchor{faq/library library-and-extension-faq}@anchor{4844}
@section Library and Extension FAQ



@menu
* General Library Questions:: 
* Common tasks:: 
* Threads:: 
* Input and Output: Input and Output<2>. 
* Network/Internet Programming:: 
* Databases:: 
* Mathematics and Numerics:: 

@end menu

@node General Library Questions,Common tasks,,Library and Extension FAQ
@anchor{faq/library general-library-questions}@anchor{4845}
@subsection General Library Questions


@menu
* How do I find a module or application to perform task X?:: 
* Where is the math.py (socket.py@comma{} regex.py@comma{} etc.) source file?: Where is the math py socket py regex py etc source file?. 
* How do I make a Python script executable on Unix?:: 
* Is there a curses/termcap package for Python?:: 
* Is there an equivalent to C’s onexit() in Python?: Is there an equivalent to C’s onexit in Python?. 
* Why don’t my signal handlers work?:: 

@end menu

@node How do I find a module or application to perform task X?,Where is the math py socket py regex py etc source file?,,General Library Questions
@anchor{faq/library how-do-i-find-a-module-or-application-to-perform-task-x}@anchor{4846}
@subsubsection How do I find a module or application to perform task X?


Check @ref{14bc,,the Library Reference} to see if there’s a relevant
standard library module.  (Eventually you’ll learn what’s in the standard
library and will be able to skip this step.)

For third-party packages, search the Python Package Index@footnote{https://pypi.org} or try Google@footnote{https://www.google.com} or
another web search engine.  Searching for “Python” plus a keyword or two for
your topic of interest will usually find something helpful.

@node Where is the math py socket py regex py etc source file?,How do I make a Python script executable on Unix?,How do I find a module or application to perform task X?,General Library Questions
@anchor{faq/library where-is-the-math-py-socket-py-regex-py-etc-source-file}@anchor{4847}
@subsubsection Where is the math.py (socket.py, regex.py, etc.) source file?


If you can’t find a source file for a module it may be a built-in or
dynamically loaded module implemented in C, C++ or other compiled language.
In this case you may not have the source file or it may be something like
@code{mathmodule.c}, somewhere in a C source directory (not on the Python Path).

There are (at least) three kinds of modules in Python:


@enumerate 

@item 
modules written in Python (.py);

@item 
modules written in C and dynamically loaded (.dll, .pyd, .so, .sl, etc);

@item 
modules written in C and linked with the interpreter; to get a list of these,
type:

@example
import sys
print(sys.builtin_module_names)
@end example
@end enumerate

@node How do I make a Python script executable on Unix?,Is there a curses/termcap package for Python?,Where is the math py socket py regex py etc source file?,General Library Questions
@anchor{faq/library how-do-i-make-a-python-script-executable-on-unix}@anchor{4848}
@subsubsection How do I make a Python script executable on Unix?


You need to do two things: the script file’s mode must be executable and the
first line must begin with @code{#!} followed by the path of the Python
interpreter.

The first is done by executing @code{chmod +x scriptfile} or perhaps @code{chmod 755
scriptfile}.

The second can be done in a number of ways.  The most straightforward way is to
write

@example
#!/usr/local/bin/python
@end example

as the very first line of your file, using the pathname for where the Python
interpreter is installed on your platform.

If you would like the script to be independent of where the Python interpreter
lives, you can use the @code{env} program.  Almost all Unix variants support
the following, assuming the Python interpreter is in a directory on the user’s
@geindex PATH
@geindex environment variable; PATH
@code{PATH}:

@example
#!/usr/bin/env python
@end example

@emph{Don’t} do this for CGI scripts.  The 
@geindex PATH
@geindex environment variable; PATH
@code{PATH} variable for CGI scripts is
often very minimal, so you need to use the actual absolute pathname of the
interpreter.

Occasionally, a user’s environment is so full that the @code{/usr/bin/env}
program fails; or there’s no env program at all.  In that case, you can try the
following hack (due to Alex Rezinsky):

@example
#! /bin/sh
""":"
exec python $0 $@{1+"$@@"@}
"""
@end example

The minor disadvantage is that this defines the script’s __doc__ string.
However, you can fix that by adding

@example
__doc__ = """...Whatever..."""
@end example

@node Is there a curses/termcap package for Python?,Is there an equivalent to C’s onexit in Python?,How do I make a Python script executable on Unix?,General Library Questions
@anchor{faq/library is-there-a-curses-termcap-package-for-python}@anchor{4849}
@subsubsection Is there a curses/termcap package for Python?


@c XXX curses *is* built by default, isn't it?

For Unix variants: The standard Python source distribution comes with a curses
module in the Modules@footnote{https://github.com/python/cpython/tree/main/Modules} subdirectory, though it’s not compiled by default.
(Note that this is not available in the Windows distribution – there is no
curses module for Windows.)

The @ref{2a,,curses} module supports basic curses features as well as many additional
functions from ncurses and SYSV curses such as colour, alternative character set
support, pads, and mouse support. This means the module isn’t compatible with
operating systems that only have BSD curses, but there don’t seem to be any
currently maintained OSes that fall into this category.

@node Is there an equivalent to C’s onexit in Python?,Why don’t my signal handlers work?,Is there a curses/termcap package for Python?,General Library Questions
@anchor{faq/library is-there-an-equivalent-to-c-s-onexit-in-python}@anchor{484a}
@subsubsection Is there an equivalent to C’s onexit() in Python?


The @ref{b,,atexit} module provides a register function that is similar to C’s
@code{onexit()}.

@node Why don’t my signal handlers work?,,Is there an equivalent to C’s onexit in Python?,General Library Questions
@anchor{faq/library why-don-t-my-signal-handlers-work}@anchor{484b}
@subsubsection Why don’t my signal handlers work?


The most common problem is that the signal handler is declared with the wrong
argument list.  It is called as

@example
handler(signum, frame)
@end example

so it should be declared with two parameters:

@example
def handler(signum, frame):
    ...
@end example

@node Common tasks,Threads,General Library Questions,Library and Extension FAQ
@anchor{faq/library common-tasks}@anchor{484c}
@subsection Common tasks


@menu
* How do I test a Python program or component?:: 
* How do I create documentation from doc strings?:: 
* How do I get a single keypress at a time?:: 

@end menu

@node How do I test a Python program or component?,How do I create documentation from doc strings?,,Common tasks
@anchor{faq/library how-do-i-test-a-python-program-or-component}@anchor{484d}
@subsubsection How do I test a Python program or component?


Python comes with two testing frameworks.  The @ref{63,,doctest} module finds
examples in the docstrings for a module and runs them, comparing the output with
the expected output given in the docstring.

The @ref{122,,unittest} module is a fancier testing framework modelled on Java and
Smalltalk testing frameworks.

To make testing easier, you should use good modular design in your program.
Your program should have almost all functionality
encapsulated in either functions or class methods – and this sometimes has the
surprising and delightful effect of making the program run faster (because local
variable accesses are faster than global accesses).  Furthermore the program
should avoid depending on mutating global variables, since this makes testing
much more difficult to do.

The “global main logic” of your program may be as simple as

@example
if __name__ == "__main__":
    main_logic()
@end example

at the bottom of the main module of your program.

Once your program is organized as a tractable collection of function and class
behaviours, you should write test functions that exercise the behaviours.  A
test suite that automates a sequence of tests can be associated with each module.
This sounds like a lot of work, but since Python is so terse and flexible it’s
surprisingly easy.  You can make coding much more pleasant and fun by writing
your test functions in parallel with the “production code”, since this makes it
easy to find bugs and even design flaws earlier.

“Support modules” that are not intended to be the main module of a program may
include a self-test of the module.

@example
if __name__ == "__main__":
    self_test()
@end example

Even programs that interact with complex external interfaces may be tested when
the external interfaces are unavailable by using “fake” interfaces implemented
in Python.

@node How do I create documentation from doc strings?,How do I get a single keypress at a time?,How do I test a Python program or component?,Common tasks
@anchor{faq/library how-do-i-create-documentation-from-doc-strings}@anchor{484e}
@subsubsection How do I create documentation from doc strings?


The @ref{d4,,pydoc} module can create HTML from the doc strings in your Python
source code.  An alternative for creating API documentation purely from
docstrings is epydoc@footnote{http://epydoc.sourceforge.net/}.  Sphinx@footnote{http://sphinx-doc.org} can also include docstring content.

@node How do I get a single keypress at a time?,,How do I create documentation from doc strings?,Common tasks
@anchor{faq/library how-do-i-get-a-single-keypress-at-a-time}@anchor{484f}
@subsubsection How do I get a single keypress at a time?


For Unix variants there are several solutions.  It’s straightforward to do this
using curses, but curses is a fairly large module to learn.

@c XXX this doesn't work out of the box, some IO expert needs to check why
@c 
@c Here's a solution without curses::
@c 
@c import termios, fcntl, sys, os
@c fd = sys.stdin.fileno()
@c 
@c oldterm = termios.tcgetattr(fd)
@c newattr = termios.tcgetattr(fd)
@c newattr[3] = newattr[3] & ~termios.ICANON & ~termios.ECHO
@c termios.tcsetattr(fd, termios.TCSANOW, newattr)
@c 
@c oldflags = fcntl.fcntl(fd, fcntl.F_GETFL)
@c fcntl.fcntl(fd, fcntl.F_SETFL, oldflags | os.O_NONBLOCK)
@c 
@c try:
@c     while True:
@c         try:
@c             c = sys.stdin.read(1)
@c             print("Got character", repr(c))
@c         except OSError:
@c             pass
@c finally:
@c     termios.tcsetattr(fd, termios.TCSAFLUSH, oldterm)
@c     fcntl.fcntl(fd, fcntl.F_SETFL, oldflags)
@c 
@c You need the :mod:`termios` and the :mod:`fcntl` module for any of this to
@c work, and I've only tried it on Linux, though it should work elsewhere.  In
@c this code, characters are read and printed one at a time.
@c 
@c :func:`termios.tcsetattr` turns off stdin's echoing and disables canonical
@c mode.  :func:`fcntl.fnctl` is used to obtain stdin's file descriptor flags
@c and modify them for non-blocking mode.  Since reading stdin when it is empty
@c results in an :exc:`OSError`, this error is caught and ignored.
@c 
@c .. versionchanged:: 3.3
@c    *sys.stdin.read* used to raise :exc:`IOError`. Starting from Python 3.3
@c    :exc:`IOError` is alias for :exc:`OSError`.

@node Threads,Input and Output<2>,Common tasks,Library and Extension FAQ
@anchor{faq/library threads}@anchor{4850}
@subsection Threads


@menu
* How do I program using threads?:: 
* None of my threads seem to run; why?: None of my threads seem to run why?. 
* How do I parcel out work among a bunch of worker threads?:: 
* What kinds of global value mutation are thread-safe?:: 
* Can’t we get rid of the Global Interpreter Lock?:: 

@end menu

@node How do I program using threads?,None of my threads seem to run why?,,Threads
@anchor{faq/library how-do-i-program-using-threads}@anchor{4851}
@subsubsection How do I program using threads?


Be sure to use the @ref{109,,threading} module and not the @ref{2,,_thread} module.
The @ref{109,,threading} module builds convenient abstractions on top of the
low-level primitives provided by the @ref{2,,_thread} module.

@node None of my threads seem to run why?,How do I parcel out work among a bunch of worker threads?,How do I program using threads?,Threads
@anchor{faq/library none-of-my-threads-seem-to-run-why}@anchor{4852}
@subsubsection None of my threads seem to run: why?


As soon as the main thread exits, all threads are killed.  Your main thread is
running too quickly, giving the threads no time to do any work.

A simple fix is to add a sleep to the end of the program that’s long enough for
all the threads to finish:

@example
import threading, time

def thread_task(name, n):
    for i in range(n):
        print(name, i)

for i in range(10):
    T = threading.Thread(target=thread_task, args=(str(i), i))
    T.start()

time.sleep(10)  # <---------------------------!
@end example

But now (on many platforms) the threads don’t run in parallel, but appear to run
sequentially, one at a time!  The reason is that the OS thread scheduler doesn’t
start a new thread until the previous thread is blocked.

A simple fix is to add a tiny sleep to the start of the run function:

@example
def thread_task(name, n):
    time.sleep(0.001)  # <--------------------!
    for i in range(n):
        print(name, i)

for i in range(10):
    T = threading.Thread(target=thread_task, args=(str(i), i))
    T.start()

time.sleep(10)
@end example

Instead of trying to guess a good delay value for @ref{196,,time.sleep()},
it’s better to use some kind of semaphore mechanism.  One idea is to use the
@ref{d5,,queue} module to create a queue object, let each thread append a token to
the queue when it finishes, and let the main thread read as many tokens from the
queue as there are threads.

@node How do I parcel out work among a bunch of worker threads?,What kinds of global value mutation are thread-safe?,None of my threads seem to run why?,Threads
@anchor{faq/library how-do-i-parcel-out-work-among-a-bunch-of-worker-threads}@anchor{4853}
@subsubsection How do I parcel out work among a bunch of worker threads?


The easiest way is to use the @ref{20,,concurrent.futures} module,
especially the @ref{1c7,,ThreadPoolExecutor} class.

Or, if you want fine control over the dispatching algorithm, you can write
your own logic manually.  Use the @ref{d5,,queue} module to create a queue
containing a list of jobs.  The @ref{f18,,Queue} class maintains a
list of objects and has a @code{.put(obj)} method that adds items to the queue and
a @code{.get()} method to return them.  The class will take care of the locking
necessary to ensure that each job is handed out exactly once.

Here’s a trivial example:

@example
import threading, queue, time

# The worker thread gets jobs off the queue.  When the queue is empty, it
# assumes there will be no more work and exits.
# (Realistically workers will run until terminated.)
def worker():
    print('Running worker')
    time.sleep(0.1)
    while True:
        try:
            arg = q.get(block=False)
        except queue.Empty:
            print('Worker', threading.current_thread(), end=' ')
            print('queue empty')
            break
        else:
            print('Worker', threading.current_thread(), end=' ')
            print('running with argument', arg)
            time.sleep(0.5)

# Create queue
q = queue.Queue()

# Start a pool of 5 workers
for i in range(5):
    t = threading.Thread(target=worker, name='worker %i' % (i+1))
    t.start()

# Begin adding work to the queue
for i in range(50):
    q.put(i)

# Give threads time to run
print('Main thread sleeping')
time.sleep(5)
@end example

When run, this will produce the following output:

@example
Running worker
Running worker
Running worker
Running worker
Running worker
Main thread sleeping
Worker <Thread(worker 1, started 130283832797456)> running with argument 0
Worker <Thread(worker 2, started 130283824404752)> running with argument 1
Worker <Thread(worker 3, started 130283816012048)> running with argument 2
Worker <Thread(worker 4, started 130283807619344)> running with argument 3
Worker <Thread(worker 5, started 130283799226640)> running with argument 4
Worker <Thread(worker 1, started 130283832797456)> running with argument 5
...
@end example

Consult the module’s documentation for more details; the @ref{f18,,Queue}
class provides a featureful interface.

@node What kinds of global value mutation are thread-safe?,Can’t we get rid of the Global Interpreter Lock?,How do I parcel out work among a bunch of worker threads?,Threads
@anchor{faq/library what-kinds-of-global-value-mutation-are-thread-safe}@anchor{4854}
@subsubsection What kinds of global value mutation are thread-safe?


A @ref{7ae,,global interpreter lock} (GIL) is used internally to ensure that only one
thread runs in the Python VM at a time.  In general, Python offers to switch
among threads only between bytecode instructions; how frequently it switches can
be set via @ref{449,,sys.setswitchinterval()}.  Each bytecode instruction and
therefore all the C implementation code reached from each instruction is
therefore atomic from the point of view of a Python program.

In theory, this means an exact accounting requires an exact understanding of the
PVM bytecode implementation.  In practice, it means that operations on shared
variables of built-in data types (ints, lists, dicts, etc) that “look atomic”
really are.

For example, the following operations are all atomic (L, L1, L2 are lists, D,
D1, D2 are dicts, x, y are objects, i, j are ints):

@example
L.append(x)
L1.extend(L2)
x = L[i]
x = L.pop()
L1[i:j] = L2
L.sort()
x = y
x.field = y
D[x] = y
D1.update(D2)
D.keys()
@end example

These aren’t:

@example
i = i+1
L.append(L[-1])
L[i] = L[j]
D[x] = D[x] + 1
@end example

Operations that replace other objects may invoke those other objects’
@code{__del__()} method when their reference count reaches zero, and that can
affect things.  This is especially true for the mass updates to dictionaries and
lists.  When in doubt, use a mutex!

@node Can’t we get rid of the Global Interpreter Lock?,,What kinds of global value mutation are thread-safe?,Threads
@anchor{faq/library can-t-we-get-rid-of-the-global-interpreter-lock}@anchor{4855}
@subsubsection Can’t we get rid of the Global Interpreter Lock?


@c XXX link to dbeazley's talk about GIL?

The @ref{7ae,,global interpreter lock} (GIL) is often seen as a hindrance to Python’s
deployment on high-end multiprocessor server machines, because a multi-threaded
Python program effectively only uses one CPU, due to the insistence that
(almost) all Python code can only run while the GIL is held.

Back in the days of Python 1.5, Greg Stein actually implemented a comprehensive
patch set (the “free threading” patches) that removed the GIL and replaced it
with fine-grained locking.  Adam Olsen recently did a similar experiment
in his python-safethread@footnote{https://code.google.com/archive/p/python-safethread}
project.  Unfortunately, both experiments exhibited a sharp drop in single-thread
performance (at least 30% slower), due to the amount of fine-grained locking
necessary to compensate for the removal of the GIL.

This doesn’t mean that you can’t make good use of Python on multi-CPU machines!
You just have to be creative with dividing the work up between multiple
@emph{processes} rather than multiple @emph{threads}.  The
@ref{3da,,ProcessPoolExecutor} class in the new
@ref{20,,concurrent.futures} module provides an easy way of doing so; the
@ref{b3,,multiprocessing} module provides a lower-level API in case you want
more control over dispatching of tasks.

Judicious use of C extensions will also help; if you use a C extension to
perform a time-consuming task, the extension can release the GIL while the
thread of execution is in the C code and allow other threads to get some work
done.  Some standard library modules such as @ref{14b,,zlib} and @ref{88,,hashlib}
already do this.

It has been suggested that the GIL should be a per-interpreter-state lock rather
than truly global; interpreters then wouldn’t be able to share objects.
Unfortunately, this isn’t likely to happen either.  It would be a tremendous
amount of work, because many object implementations currently have global state.
For example, small integers and short strings are cached; these caches would
have to be moved to the interpreter state.  Other object types have their own
free list; these free lists would have to be moved to the interpreter state.
And so on.

And I doubt that it can even be done in finite time, because the same problem
exists for 3rd party extensions.  It is likely that 3rd party extensions are
being written at a faster rate than you can convert them to store all their
global state in the interpreter state.

And finally, once you have multiple interpreters not sharing any state, what
have you gained over running each interpreter in a separate process?

@node Input and Output<2>,Network/Internet Programming,Threads,Library and Extension FAQ
@anchor{faq/library input-and-output}@anchor{4856}
@subsection Input and Output


@menu
* How do I delete a file? (And other file questions…): How do I delete a file? And other file questions…. 
* How do I copy a file?:: 
* How do I read (or write) binary data?: How do I read or write binary data?. 
* I can’t seem to use os.read() on a pipe created with os.popen(); why?: I can’t seem to use os read on a pipe created with os popen ; why?. 
* How do I access the serial (RS232) port?: How do I access the serial RS232 port?. 
* Why doesn’t closing sys.stdout (stdin@comma{} stderr) really close it?: Why doesn’t closing sys stdout stdin stderr really close it?. 

@end menu

@node How do I delete a file? And other file questions…,How do I copy a file?,,Input and Output<2>
@anchor{faq/library how-do-i-delete-a-file-and-other-file-questions}@anchor{4857}
@subsubsection How do I delete a file? (And other file questions…)


Use @code{os.remove(filename)} or @code{os.unlink(filename)}; for documentation, see
the @ref{c0,,os} module.  The two functions are identical; @ref{c89,,unlink()} is simply
the name of the Unix system call for this function.

To remove a directory, use @ref{c87,,os.rmdir()}; use @ref{c83,,os.mkdir()} to create one.
@code{os.makedirs(path)} will create any intermediate directories in @code{path} that
don’t exist. @code{os.removedirs(path)} will remove intermediate directories as
long as they’re empty; if you want to delete an entire directory tree and its
contents, use @ref{53a,,shutil.rmtree()}.

To rename a file, use @code{os.rename(old_path, new_path)}.

To truncate a file, open it using @code{f = open(filename, "rb+")}, and use
@code{f.truncate(offset)}; offset defaults to the current seek position.  There’s
also @code{os.ftruncate(fd, offset)} for files opened with @ref{8ee,,os.open()}, where
@emph{fd} is the file descriptor (a small integer).

The @ref{e4,,shutil} module also contains a number of functions to work on files
including @ref{56f,,copyfile()}, @ref{538,,copytree()}, and
@ref{53a,,rmtree()}.

@node How do I copy a file?,How do I read or write binary data?,How do I delete a file? And other file questions…,Input and Output<2>
@anchor{faq/library how-do-i-copy-a-file}@anchor{4858}
@subsubsection How do I copy a file?


The @ref{e4,,shutil} module contains a @ref{56f,,copyfile()} function.  Note
that on MacOS 9 it doesn’t copy the resource fork and Finder info.

@node How do I read or write binary data?,I can’t seem to use os read on a pipe created with os popen ; why?,How do I copy a file?,Input and Output<2>
@anchor{faq/library how-do-i-read-or-write-binary-data}@anchor{4859}
@subsubsection How do I read (or write) binary data?


To read or write complex binary data formats, it’s best to use the @ref{f3,,struct}
module.  It allows you to take a string containing binary data (usually numbers)
and convert it to Python objects; and vice versa.

For example, the following code reads two 2-byte integers and one 4-byte integer
in big-endian format from a file:

@example
import struct

with open(filename, "rb") as f:
    s = f.read(8)
    x, y, z = struct.unpack(">hhl", s)
@end example

The ‘>’ in the format string forces big-endian data; the letter ‘h’ reads one
“short integer” (2 bytes), and ‘l’ reads one “long integer” (4 bytes) from the
string.

For data that is more regular (e.g. a homogeneous list of ints or floats),
you can also use the @ref{6,,array} module.

@cartouche
@quotation Note 
To read and write binary data, it is mandatory to open the file in
binary mode (here, passing @code{"rb"} to @ref{1c8,,open()}).  If you use
@code{"r"} instead (the default), the file will be open in text mode
and @code{f.read()} will return @ref{2ef,,str} objects rather than
@ref{2f0,,bytes} objects.
@end quotation
@end cartouche

@node I can’t seem to use os read on a pipe created with os popen ; why?,How do I access the serial RS232 port?,How do I read or write binary data?,Input and Output<2>
@anchor{faq/library i-can-t-seem-to-use-os-read-on-a-pipe-created-with-os-popen-why}@anchor{485a}
@subsubsection I can’t seem to use os.read() on a pipe created with os.popen(); why?


@ref{8f1,,os.read()} is a low-level function which takes a file descriptor, a small
integer representing the opened file.  @ref{59f,,os.popen()} creates a high-level
file object, the same type returned by the built-in @ref{1c8,,open()} function.
Thus, to read @emph{n} bytes from a pipe @emph{p} created with @ref{59f,,os.popen()}, you need to
use @code{p.read(n)}.

@c XXX update to use subprocess. See the :ref:`subprocess-replacements` section.
@c 
@c How do I run a subprocess with pipes connected to both input and output?
@c ------------------------------------------------------------------------
@c 
@c Use the :mod:`popen2` module.  For example::
@c 
@c    import popen2
@c    fromchild, tochild = popen2.popen2("command")
@c    tochild.write("input\n")
@c    tochild.flush()
@c    output = fromchild.readline()
@c 
@c Warning: in general it is unwise to do this because you can easily cause a
@c deadlock where your process is blocked waiting for output from the child
@c while the child is blocked waiting for input from you.  This can be caused
@c by the parent expecting the child to output more text than it does or
@c by data being stuck in stdio buffers due to lack of flushing.
@c The Python parent can of course explicitly flush the data it sends to the
@c child before it reads any output, but if the child is a naive C program it
@c may have been written to never explicitly flush its output, even if it is
@c interactive, since flushing is normally automatic.
@c 
@c Note that a deadlock is also possible if you use :func:`popen3` to read
@c stdout and stderr. If one of the two is too large for the internal buffer
@c (increasing the buffer size does not help) and you ``read()`` the other one
@c first, there is a deadlock, too.
@c 
@c Note on a bug in popen2: unless your program calls ``wait()`` or
@c ``waitpid()``, finished child processes are never removed, and eventually
@c calls to popen2 will fail because of a limit on the number of child
@c processes.  Calling :func:`os.waitpid` with the :data:`os.WNOHANG` option can
@c prevent this; a good place to insert such a call would be before calling
@c ``popen2`` again.
@c 
@c In many cases, all you really need is to run some data through a command and
@c get the result back.  Unless the amount of data is very large, the easiest
@c way to do this is to write it to a temporary file and run the command with
@c that temporary file as input.  The standard module :mod:`tempfile` exports a
@c :func:`~tempfile.mktemp` function to generate unique temporary file names. ::
@c 
@c    import tempfile
@c    import os
@c 
@c    class Popen3:
@c        """
@c        This is a deadlock-safe version of popen that returns
@c        an object with errorlevel, out (a string) and err (a string).
@c        (capturestderr may not work under windows.)
@c        Example: print(Popen3('grep spam','\n\nhere spam\n\n').out)
@c        """
@c        def __init__(self,command,input=None,capturestderr=None):
@c            outfile=tempfile.mktemp()
@c            command="( %s ) > %s" % (command,outfile)
@c            if input:
@c                infile=tempfile.mktemp()
@c                open(infile,"w").write(input)
@c                command=command+" <"+infile
@c            if capturestderr:
@c                errfile=tempfile.mktemp()
@c                command=command+" 2>"+errfile
@c            self.errorlevel=os.system(command) >> 8
@c            self.out=open(outfile,"r").read()
@c            os.remove(outfile)
@c            if input:
@c                os.remove(infile)
@c            if capturestderr:
@c                self.err=open(errfile,"r").read()
@c                os.remove(errfile)
@c 
@c Note that many interactive programs (e.g. vi) don't work well with pipes
@c substituted for standard input and output.  You will have to use pseudo ttys
@c ("ptys") instead of pipes. Or you can use a Python interface to Don Libes'
@c "expect" library.  A Python extension that interfaces to expect is called
@c "expy" and available from http://expectpy.sourceforge.net.  A pure Python
@c solution that works like expect is `pexpect
@c <https://pypi.org/project/pexpect/>`_.

@node How do I access the serial RS232 port?,Why doesn’t closing sys stdout stdin stderr really close it?,I can’t seem to use os read on a pipe created with os popen ; why?,Input and Output<2>
@anchor{faq/library how-do-i-access-the-serial-rs232-port}@anchor{485b}
@subsubsection How do I access the serial (RS232) port?


For Win32, OSX, Linux, BSD, Jython, IronPython:

@quotation

@indicateurl{https://pypi.org/project/pyserial/}
@end quotation

For Unix, see a Usenet post by Mitch Chapman:

@quotation

@indicateurl{https://groups.google.com/groups?selm=34A04430.CF9@@ohioee.com}
@end quotation

@node Why doesn’t closing sys stdout stdin stderr really close it?,,How do I access the serial RS232 port?,Input and Output<2>
@anchor{faq/library why-doesn-t-closing-sys-stdout-stdin-stderr-really-close-it}@anchor{485c}
@subsubsection Why doesn’t closing sys.stdout (stdin, stderr) really close it?


Python @ref{d68,,file objects} are a high-level layer of
abstraction on low-level C file descriptors.

For most file objects you create in Python via the built-in @ref{1c8,,open()}
function, @code{f.close()} marks the Python file object as being closed from
Python’s point of view, and also arranges to close the underlying C file
descriptor.  This also happens automatically in @code{f}’s destructor, when
@code{f} becomes garbage.

But stdin, stdout and stderr are treated specially by Python, because of the
special status also given to them by C.  Running @code{sys.stdout.close()} marks
the Python-level file object as being closed, but does @emph{not} close the
associated C file descriptor.

To close the underlying C file descriptor for one of these three, you should
first be sure that’s what you really want to do (e.g., you may confuse
extension modules trying to do I/O).  If it is, use @ref{6a1,,os.close()}:

@example
os.close(stdin.fileno())
os.close(stdout.fileno())
os.close(stderr.fileno())
@end example

Or you can use the numeric constants 0, 1 and 2, respectively.

@node Network/Internet Programming,Databases,Input and Output<2>,Library and Extension FAQ
@anchor{faq/library network-internet-programming}@anchor{485d}
@subsection Network/Internet Programming


@menu
* What WWW tools are there for Python?:: 
* How can I mimic CGI form submission (METHOD=POST)?: How can I mimic CGI form submission METHOD=POST ?. 
* What module should I use to help with generating HTML?:: 
* How do I send mail from a Python script?:: 
* How do I avoid blocking in the connect() method of a socket?: How do I avoid blocking in the connect method of a socket?. 

@end menu

@node What WWW tools are there for Python?,How can I mimic CGI form submission METHOD=POST ?,,Network/Internet Programming
@anchor{faq/library what-www-tools-are-there-for-python}@anchor{485e}
@subsubsection What WWW tools are there for Python?


See the chapters titled @ref{3050,,Internet Protocols and Support} and @ref{2c4e,,Internet Data Handling} in the Library
Reference Manual.  Python has many modules that will help you build server-side
and client-side web systems.

@c XXX check if wiki page is still up to date

A summary of available frameworks is maintained by Paul Boddie at
@indicateurl{https://wiki.python.org/moin/WebProgramming}.

Cameron Laird maintains a useful set of pages about Python web technologies at
@indicateurl{http://phaseit.net/claird/comp.lang.python/web_python}.

@node How can I mimic CGI form submission METHOD=POST ?,What module should I use to help with generating HTML?,What WWW tools are there for Python?,Network/Internet Programming
@anchor{faq/library how-can-i-mimic-cgi-form-submission-method-post}@anchor{485f}
@subsubsection How can I mimic CGI form submission (METHOD=POST)?


I would like to retrieve web pages that are the result of POSTing a form. Is
there existing code that would let me do this easily?

Yes. Here’s a simple example that uses @ref{127,,urllib.request}:

@example
#!/usr/local/bin/python

import urllib.request

# build the query string
qs = "First=Josephine&MI=Q&Last=Public"

# connect and send the server a path
req = urllib.request.urlopen('http://www.some-server.out-there'
                             '/cgi-bin/some-cgi-script', data=qs)
with req:
    msg, hdrs = req.read(), req.info()
@end example

Note that in general for percent-encoded POST operations, query strings must be
quoted using @ref{a05,,urllib.parse.urlencode()}.  For example, to send
@code{name=Guy Steele, Jr.}:

@example
>>> import urllib.parse
>>> urllib.parse.urlencode(@{'name': 'Guy Steele, Jr.'@})
'name=Guy+Steele%2C+Jr.'
@end example


@subsubheading See also


@ref{3134,,HOWTO Fetch Internet Resources Using The urllib Package} for extensive examples.


@node What module should I use to help with generating HTML?,How do I send mail from a Python script?,How can I mimic CGI form submission METHOD=POST ?,Network/Internet Programming
@anchor{faq/library what-module-should-i-use-to-help-with-generating-html}@anchor{4860}
@subsubsection What module should I use to help with generating HTML?


@c XXX add modern template languages

You can find a collection of useful links on the Web Programming wiki page@footnote{https://wiki.python.org/moin/WebProgramming}.

@node How do I send mail from a Python script?,How do I avoid blocking in the connect method of a socket?,What module should I use to help with generating HTML?,Network/Internet Programming
@anchor{faq/library how-do-i-send-mail-from-a-python-script}@anchor{4861}
@subsubsection How do I send mail from a Python script?


Use the standard library module @ref{e8,,smtplib}.

Here’s a very simple interactive mail sender that uses it.  This method will
work on any host that supports an SMTP listener.

@example
import sys, smtplib

fromaddr = input("From: ")
toaddrs  = input("To: ").split(',')
print("Enter message, end with ^D:")
msg = '@w{'}
while True:
    line = sys.stdin.readline()
    if not line:
        break
    msg += line

# The actual mail send
server = smtplib.SMTP('localhost')
server.sendmail(fromaddr, toaddrs, msg)
server.quit()
@end example

A Unix-only alternative uses sendmail.  The location of the sendmail program
varies between systems; sometimes it is @code{/usr/lib/sendmail}, sometimes
@code{/usr/sbin/sendmail}.  The sendmail manual page will help you out.  Here’s
some sample code:

@example
import os

SENDMAIL = "/usr/sbin/sendmail"  # sendmail location
p = os.popen("%s -t -i" % SENDMAIL, "w")
p.write("To: receiver@@example.com\n")
p.write("Subject: test\n")
p.write("\n")  # blank line separating headers from body
p.write("Some text\n")
p.write("some more text\n")
sts = p.close()
if sts != 0:
    print("Sendmail exit status", sts)
@end example

@node How do I avoid blocking in the connect method of a socket?,,How do I send mail from a Python script?,Network/Internet Programming
@anchor{faq/library how-do-i-avoid-blocking-in-the-connect-method-of-a-socket}@anchor{4862}
@subsubsection How do I avoid blocking in the connect() method of a socket?


The @ref{e0,,select} module is commonly used to help with asynchronous I/O on
sockets.

To prevent the TCP connect from blocking, you can set the socket to non-blocking
mode.  Then when you do the @code{socket.connect()}, you will either connect immediately
(unlikely) or get an exception that contains the error number as @code{.errno}.
@code{errno.EINPROGRESS} indicates that the connection is in progress, but hasn’t
finished yet.  Different OSes will return different values, so you’re going to
have to check what’s returned on your system.

You can use the @code{socket.connect_ex()} method to avoid creating an exception.  It will
just return the errno value.  To poll, you can call @code{socket.connect_ex()} again later
– @code{0} or @code{errno.EISCONN} indicate that you’re connected – or you can pass this
socket to @ref{8fc,,select.select()} to check if it’s writable.

@cartouche
@quotation Note 
The @ref{9,,asyncio} module provides a general purpose single-threaded and
concurrent asynchronous library, which can be used for writing non-blocking
network code.
The third-party Twisted@footnote{https://twistedmatrix.com/trac/} library is
a popular and feature-rich alternative.
@end quotation
@end cartouche

@node Databases,Mathematics and Numerics,Network/Internet Programming,Library and Extension FAQ
@anchor{faq/library databases}@anchor{4863}
@subsection Databases


@menu
* Are there any interfaces to database packages in Python?:: 
* How do you implement persistent objects in Python?:: 

@end menu

@node Are there any interfaces to database packages in Python?,How do you implement persistent objects in Python?,,Databases
@anchor{faq/library are-there-any-interfaces-to-database-packages-in-python}@anchor{4864}
@subsubsection Are there any interfaces to database packages in Python?


Yes.

Interfaces to disk-based hashes such as @ref{33,,DBM} and @ref{32,,GDBM} are also included with standard Python.  There is also the
@ref{ed,,sqlite3} module, which provides a lightweight disk-based relational
database.

Support for most relational databases is available.  See the
DatabaseProgramming wiki page@footnote{https://wiki.python.org/moin/DatabaseProgramming} for details.

@node How do you implement persistent objects in Python?,,Are there any interfaces to database packages in Python?,Databases
@anchor{faq/library how-do-you-implement-persistent-objects-in-python}@anchor{4865}
@subsubsection How do you implement persistent objects in Python?


The @ref{c5,,pickle} library module solves this in a very general way (though you
still can’t store things like open files, sockets or windows), and the
@ref{e2,,shelve} library module uses pickle and (g)dbm to create persistent
mappings containing arbitrary Python objects.

@node Mathematics and Numerics,,Databases,Library and Extension FAQ
@anchor{faq/library mathematics-and-numerics}@anchor{4866}
@subsection Mathematics and Numerics


@menu
* How do I generate random numbers in Python?:: 

@end menu

@node How do I generate random numbers in Python?,,,Mathematics and Numerics
@anchor{faq/library how-do-i-generate-random-numbers-in-python}@anchor{4867}
@subsubsection How do I generate random numbers in Python?


The standard module @ref{d7,,random} implements a random number generator.  Usage
is simple:

@example
import random
random.random()
@end example

This returns a random floating point number in the range [0, 1).

There are also many other specialized generators in this module, such as:


@itemize *

@item 
@code{randrange(a, b)} chooses an integer in the range [a, b).

@item 
@code{uniform(a, b)} chooses a floating point number in the range [a, b).

@item 
@code{normalvariate(mean, sdev)} samples the normal (Gaussian) distribution.
@end itemize

Some higher-level functions operate on sequences directly, such as:


@itemize *

@item 
@code{choice(S)} chooses a random element from a given sequence.

@item 
@code{shuffle(L)} shuffles a list in-place, i.e. permutes it randomly.
@end itemize

There’s also a @code{Random} class you can instantiate to create independent
multiple random number generators.

@node Extending/Embedding FAQ,Python on Windows FAQ,Library and Extension FAQ,Python Frequently Asked Questions
@anchor{faq/extending doc}@anchor{4868}@anchor{faq/extending extending-embedding-faq}@anchor{4869}
@section Extending/Embedding FAQ



@c XXX need review for Python 3.

@menu
* Can I create my own functions in C?:: 
* Can I create my own functions in C++?:: 
* Writing C is hard; are there any alternatives?:: 
* How can I execute arbitrary Python statements from C?:: 
* How can I evaluate an arbitrary Python expression from C?:: 
* How do I extract C values from a Python object?:: 
* How do I use Py_BuildValue() to create a tuple of arbitrary length?: How do I use Py_BuildValue to create a tuple of arbitrary length?. 
* How do I call an object’s method from C?:: 
* How do I catch the output from PyErr_Print() (or anything that prints to stdout/stderr)?: How do I catch the output from PyErr_Print or anything that prints to stdout/stderr ?. 
* How do I access a module written in Python from C?:: 
* How do I interface to C++ objects from Python?:: 
* I added a module using the Setup file and the make fails; why?:: 
* How do I debug an extension?:: 
* I want to compile a Python module on my Linux system@comma{} but some files are missing. Why?: I want to compile a Python module on my Linux system but some files are missing Why?. 
* How do I tell “incomplete input” from “invalid input”?:: 
* How do I find undefined g++ symbols __builtin_new or __pure_virtual?:: 
* Can I create an object class with some methods implemented in C and others in Python (e.g. through inheritance)?: Can I create an object class with some methods implemented in C and others in Python e g through inheritance ?. 

@end menu

@node Can I create my own functions in C?,Can I create my own functions in C++?,,Extending/Embedding FAQ
@anchor{faq/extending can-i-create-my-own-functions-in-c}@anchor{486a}
@subsection Can I create my own functions in C?


Yes, you can create built-in modules containing functions, variables, exceptions
and even new types in C.  This is explained in the document
@ref{14be,,Extending and Embedding the Python Interpreter}.

Most intermediate or advanced Python books will also cover this topic.

@node Can I create my own functions in C++?,Writing C is hard; are there any alternatives?,Can I create my own functions in C?,Extending/Embedding FAQ
@anchor{faq/extending id1}@anchor{486b}
@subsection Can I create my own functions in C++?


Yes, using the C compatibility features found in C++.  Place @code{extern "C" @{
... @}} around the Python include files and put @code{extern "C"} before each
function that is going to be called by the Python interpreter.  Global or static
C++ objects with constructors are probably not a good idea.

@node Writing C is hard; are there any alternatives?,How can I execute arbitrary Python statements from C?,Can I create my own functions in C++?,Extending/Embedding FAQ
@anchor{faq/extending c-wrapper-software}@anchor{486c}@anchor{faq/extending writing-c-is-hard-are-there-any-alternatives}@anchor{486d}
@subsection Writing C is hard; are there any alternatives?


There are a number of alternatives to writing your own C extensions, depending
on what you’re trying to do.

@c XXX make sure these all work

Cython@footnote{http://cython.org} and its relative Pyrex@footnote{https://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/} are compilers
that accept a slightly modified form of Python and generate the corresponding
C code.  Cython and Pyrex make it possible to write an extension without having
to learn Python’s C API.

If you need to interface to some C or C++ library for which no Python extension
currently exists, you can try wrapping the library’s data types and functions
with a tool such as SWIG@footnote{http://www.swig.org}.  SIP@footnote{https://riverbankcomputing.com/software/sip/intro}, CXX@footnote{http://cxx.sourceforge.net/} Boost@footnote{http://www.boost.org/libs/python/doc/index.html}, or Weave@footnote{https://github.com/scipy/weave} are also
alternatives for wrapping C++ libraries.

@node How can I execute arbitrary Python statements from C?,How can I evaluate an arbitrary Python expression from C?,Writing C is hard; are there any alternatives?,Extending/Embedding FAQ
@anchor{faq/extending how-can-i-execute-arbitrary-python-statements-from-c}@anchor{486e}
@subsection How can I execute arbitrary Python statements from C?


The highest-level function to do this is @ref{40eb,,PyRun_SimpleString()} which takes
a single string argument to be executed in the context of the module
@code{__main__} and returns @code{0} for success and @code{-1} when an exception occurred
(including @ref{21a,,SyntaxError}).  If you want more control, use
@ref{127d,,PyRun_String()}; see the source for @ref{40eb,,PyRun_SimpleString()} in
@code{Python/pythonrun.c}.

@node How can I evaluate an arbitrary Python expression from C?,How do I extract C values from a Python object?,How can I execute arbitrary Python statements from C?,Extending/Embedding FAQ
@anchor{faq/extending how-can-i-evaluate-an-arbitrary-python-expression-from-c}@anchor{486f}
@subsection How can I evaluate an arbitrary Python expression from C?


Call the function @ref{127d,,PyRun_String()} from the previous question with the
start symbol @ref{42d5,,Py_eval_input}; it parses an expression, evaluates it and
returns its value.

@node How do I extract C values from a Python object?,How do I use Py_BuildValue to create a tuple of arbitrary length?,How can I evaluate an arbitrary Python expression from C?,Extending/Embedding FAQ
@anchor{faq/extending how-do-i-extract-c-values-from-a-python-object}@anchor{4870}
@subsection How do I extract C values from a Python object?


That depends on the object’s type.  If it’s a tuple, @ref{424e,,PyTuple_Size()}
returns its length and @ref{408d,,PyTuple_GetItem()} returns the item at a specified
index.  Lists have similar functions, @code{PyListSize()} and
@ref{408e,,PyList_GetItem()}.

For bytes, @ref{4138,,PyBytes_Size()} returns its length and
@ref{4132,,PyBytes_AsStringAndSize()} provides a pointer to its value and its
length.  Note that Python bytes objects may contain null bytes so C’s
@code{strlen()} should not be used.

To test the type of an object, first make sure it isn’t @code{NULL}, and then use
@ref{f1e,,PyBytes_Check()}, @ref{4414,,PyTuple_Check()}, @ref{441e,,PyList_Check()}, etc.

There is also a high-level API to Python objects which is provided by the
so-called ‘abstract’ interface – read @code{Include/abstract.h} for further
details.  It allows interfacing with any kind of Python sequence using calls
like @ref{4220,,PySequence_Length()}, @ref{1311,,PySequence_GetItem()}, etc. as well
as many other useful protocols such as numbers (@ref{35f,,PyNumber_Index()} et
al.) and mappings in the PyMapping APIs.

@node How do I use Py_BuildValue to create a tuple of arbitrary length?,How do I call an object’s method from C?,How do I extract C values from a Python object?,Extending/Embedding FAQ
@anchor{faq/extending how-do-i-use-py-buildvalue-to-create-a-tuple-of-arbitrary-length}@anchor{4871}
@subsection How do I use Py_BuildValue() to create a tuple of arbitrary length?


You can’t.  Use @ref{424d,,PyTuple_Pack()} instead.

@node How do I call an object’s method from C?,How do I catch the output from PyErr_Print or anything that prints to stdout/stderr ?,How do I use Py_BuildValue to create a tuple of arbitrary length?,Extending/Embedding FAQ
@anchor{faq/extending how-do-i-call-an-object-s-method-from-c}@anchor{4872}
@subsection How do I call an object’s method from C?


The @ref{350,,PyObject_CallMethod()} function can be used to call an arbitrary
method of an object.  The parameters are the object, the name of the method to
call, a format string like that used with @ref{380,,Py_BuildValue()}, and the
argument values:

@example
PyObject *
PyObject_CallMethod(PyObject *object, const char *method_name,
                    const char *arg_format, ...);
@end example

This works for any object that has methods – whether built-in or user-defined.
You are responsible for eventually @ref{373,,Py_DECREF()}‘ing the return value.

To call, e.g., a file object’s “seek” method with arguments 10, 0 (assuming the
file object pointer is “f”):

@example
res = PyObject_CallMethod(f, "seek", "(ii)", 10, 0);
if (res == NULL) @{
        ... an exception occurred ...
@}
else @{
        Py_DECREF(res);
@}
@end example

Note that since @ref{4081,,PyObject_CallObject()} @emph{always} wants a tuple for the
argument list, to call a function without arguments, pass “()” for the format,
and to call a function with one argument, surround the argument in parentheses,
e.g. “(i)”.

@node How do I catch the output from PyErr_Print or anything that prints to stdout/stderr ?,How do I access a module written in Python from C?,How do I call an object’s method from C?,Extending/Embedding FAQ
@anchor{faq/extending how-do-i-catch-the-output-from-pyerr-print-or-anything-that-prints-to-stdout-stderr}@anchor{4873}
@subsection How do I catch the output from PyErr_Print() (or anything that prints to stdout/stderr)?


In Python code, define an object that supports the @code{write()} method.  Assign
this object to @ref{5ef,,sys.stdout} and @ref{42d,,sys.stderr}.  Call print_error, or
just allow the standard traceback mechanism to work. Then, the output will go
wherever your @code{write()} method sends it.

The easiest way to do this is to use the @ref{aa1,,io.StringIO} class:

@example
>>> import io, sys
>>> sys.stdout = io.StringIO()
>>> print('foo')
>>> print('hello world!')
>>> sys.stderr.write(sys.stdout.getvalue())
foo
hello world!
@end example

A custom object to do the same would look like this:

@example
>>> import io, sys
>>> class StdoutCatcher(io.TextIOBase):
...     def __init__(self):
...         self.data = []
...     def write(self, stuff):
...         self.data.append(stuff)
...
>>> import sys
>>> sys.stdout = StdoutCatcher()
>>> print('foo')
>>> print('hello world!')
>>> sys.stderr.write('@w{'}.join(sys.stdout.data))
foo
hello world!
@end example

@node How do I access a module written in Python from C?,How do I interface to C++ objects from Python?,How do I catch the output from PyErr_Print or anything that prints to stdout/stderr ?,Extending/Embedding FAQ
@anchor{faq/extending how-do-i-access-a-module-written-in-python-from-c}@anchor{4874}
@subsection How do I access a module written in Python from C?


You can get a pointer to the module object as follows:

@example
module = PyImport_ImportModule("<modulename>");
@end example

If the module hasn’t been imported yet (i.e. it is not yet present in
@ref{11f7,,sys.modules}), this initializes the module; otherwise it simply returns
the value of @code{sys.modules["<modulename>"]}.  Note that it doesn’t enter the
module into any namespace – it only ensures it has been initialized and is
stored in @ref{11f7,,sys.modules}.

You can then access the module’s attributes (i.e. any name defined in the
module) as follows:

@example
attr = PyObject_GetAttrString(module, "<attrname>");
@end example

Calling @ref{4210,,PyObject_SetAttrString()} to assign to variables in the module
also works.

@node How do I interface to C++ objects from Python?,I added a module using the Setup file and the make fails; why?,How do I access a module written in Python from C?,Extending/Embedding FAQ
@anchor{faq/extending how-do-i-interface-to-c-objects-from-python}@anchor{4875}
@subsection How do I interface to C++ objects from Python?


Depending on your requirements, there are many approaches.  To do this manually,
begin by reading @ref{14be,,the “Extending and Embedding” document}.  Realize that for the Python run-time system, there isn’t a
whole lot of difference between C and C++ – so the strategy of building a new
Python type around a C structure (pointer) type will also work for C++ objects.

For C++ libraries, see @ref{486c,,Writing C is hard; are there any alternatives?}.

@node I added a module using the Setup file and the make fails; why?,How do I debug an extension?,How do I interface to C++ objects from Python?,Extending/Embedding FAQ
@anchor{faq/extending i-added-a-module-using-the-setup-file-and-the-make-fails-why}@anchor{4876}
@subsection I added a module using the Setup file and the make fails; why?


Setup must end in a newline, if there is no newline there, the build process
fails.  (Fixing this requires some ugly shell script hackery, and this bug is so
minor that it doesn’t seem worth the effort.)

@node How do I debug an extension?,I want to compile a Python module on my Linux system but some files are missing Why?,I added a module using the Setup file and the make fails; why?,Extending/Embedding FAQ
@anchor{faq/extending how-do-i-debug-an-extension}@anchor{4877}
@subsection How do I debug an extension?


When using GDB with dynamically loaded extensions, you can’t set a breakpoint in
your extension until your extension is loaded.

In your @code{.gdbinit} file (or interactively), add the command:

@example
br _PyImport_LoadDynamicModule
@end example

Then, when you run GDB:

@example
$ gdb /local/bin/python
gdb) run myscript.py
gdb) continue # repeat until your extension is loaded
gdb) finish   # so that your extension is loaded
gdb) br myfunction.c:50
gdb) continue
@end example

@node I want to compile a Python module on my Linux system but some files are missing Why?,How do I tell “incomplete input” from “invalid input”?,How do I debug an extension?,Extending/Embedding FAQ
@anchor{faq/extending i-want-to-compile-a-python-module-on-my-linux-system-but-some-files-are-missing-why}@anchor{4878}
@subsection I want to compile a Python module on my Linux system, but some files are missing. Why?


Most packaged versions of Python don’t include the
@code{/usr/lib/python2.@emph{x}/config/} directory, which contains various files
required for compiling Python extensions.

For Red Hat, install the python-devel RPM to get the necessary files.

For Debian, run @code{apt-get install python-dev}.

@node How do I tell “incomplete input” from “invalid input”?,How do I find undefined g++ symbols __builtin_new or __pure_virtual?,I want to compile a Python module on my Linux system but some files are missing Why?,Extending/Embedding FAQ
@anchor{faq/extending how-do-i-tell-incomplete-input-from-invalid-input}@anchor{4879}
@subsection How do I tell “incomplete input” from “invalid input”?


Sometimes you want to emulate the Python interactive interpreter’s behavior,
where it gives you a continuation prompt when the input is incomplete (e.g. you
typed the start of an “if” statement or you didn’t close your parentheses or
triple string quotes), but it gives you a syntax error message immediately when
the input is invalid.

In Python you can use the @ref{1b,,codeop} module, which approximates the parser’s
behavior sufficiently.  IDLE uses this, for example.

The easiest way to do it in C is to call @ref{42c6,,PyRun_InteractiveLoop()} (perhaps
in a separate thread) and let the Python interpreter handle the input for
you. You can also set the @ref{3a2,,PyOS_ReadlineFunctionPointer()} to point at your
custom input function. See @code{Modules/readline.c} and @code{Parser/myreadline.c}
for more hints.

@node How do I find undefined g++ symbols __builtin_new or __pure_virtual?,Can I create an object class with some methods implemented in C and others in Python e g through inheritance ?,How do I tell “incomplete input” from “invalid input”?,Extending/Embedding FAQ
@anchor{faq/extending how-do-i-find-undefined-g-symbols-builtin-new-or-pure-virtual}@anchor{487a}
@subsection How do I find undefined g++ symbols __builtin_new or __pure_virtual?


To dynamically load g++ extension modules, you must recompile Python, relink it
using g++ (change LINKCC in the Python Modules Makefile), and link your
extension module using g++ (e.g., @code{g++ -shared -o mymodule.so mymodule.o}).

@node Can I create an object class with some methods implemented in C and others in Python e g through inheritance ?,,How do I find undefined g++ symbols __builtin_new or __pure_virtual?,Extending/Embedding FAQ
@anchor{faq/extending can-i-create-an-object-class-with-some-methods-implemented-in-c-and-others-in-python-e-g-through-inheritance}@anchor{487b}
@subsection Can I create an object class with some methods implemented in C and others in Python (e.g. through inheritance)?


Yes, you can inherit from built-in classes such as @ref{1b1,,int}, @ref{43a,,list},
@ref{3b0,,dict}, etc.

The Boost Python Library (BPL, @indicateurl{http://www.boost.org/libs/python/doc/index.html})
provides a way of doing this from C++ (i.e. you can inherit from an extension
class written in C++ using the BPL).

@node Python on Windows FAQ,Graphic User Interface FAQ,Extending/Embedding FAQ,Python Frequently Asked Questions
@anchor{faq/windows doc}@anchor{487c}@anchor{faq/windows python-on-windows-faq}@anchor{487d}@anchor{faq/windows windows-faq}@anchor{487e}
@section Python on Windows FAQ



@c XXX need review for Python 3.
@c XXX need review for Windows Vista/Seven?

@menu
* How do I run a Python program under Windows?:: 
* How do I make Python scripts executable?:: 
* Why does Python sometimes take so long to start?:: 
* How do I make an executable from a Python script?:: 
* Is a *.pyd file the same as a DLL?: Is a * pyd file the same as a DLL?. 
* How can I embed Python into a Windows application?:: 
* How do I keep editors from inserting tabs into my Python source?:: 
* How do I check for a keypress without blocking?:: 

@end menu

@node How do I run a Python program under Windows?,How do I make Python scripts executable?,,Python on Windows FAQ
@anchor{faq/windows faq-run-program-under-windows}@anchor{487f}@anchor{faq/windows how-do-i-run-a-python-program-under-windows}@anchor{4880}
@subsection How do I run a Python program under Windows?


This is not necessarily a straightforward question. If you are already familiar
with running programs from the Windows command line then everything will seem
obvious; otherwise, you might need a little more guidance.

Unless you use some sort of integrated development environment, you will end up
@emph{typing} Windows commands into what is variously referred to as a “DOS window”
or “Command prompt window”.  Usually you can create such a window from your
search bar by searching for @code{cmd}.  You should be able to recognize
when you have started such a window because you will see a Windows “command
prompt”, which usually looks like this:

@example
C:\>
@end example

The letter may be different, and there might be other things after it, so you
might just as easily see something like:

@example
D:\YourName\Projects\Python>
@end example

depending on how your computer has been set up and what else you have recently
done with it.  Once you have started such a window, you are well on the way to
running Python programs.

You need to realize that your Python scripts have to be processed by another
program called the Python @emph{interpreter}.  The interpreter reads your script,
compiles it into bytecodes, and then executes the bytecodes to run your
program. So, how do you arrange for the interpreter to handle your Python?

First, you need to make sure that your command window recognises the word
“py” as an instruction to start the interpreter.  If you have opened a
command window, you should try entering the command @code{py} and hitting
return:

@example
C:\Users\YourName> py
@end example

You should then see something like:

@example
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
@end example

You have started the interpreter in “interactive mode”. That means you can enter
Python statements or expressions interactively and have them executed or
evaluated while you wait.  This is one of Python’s strongest features.  Check it
by entering a few expressions of your choice and seeing the results:

@example
>>> print("Hello")
Hello
>>> "Hello" * 3
'HelloHelloHello'
@end example

Many people use the interactive mode as a convenient yet highly programmable
calculator.  When you want to end your interactive Python session,
call the @ref{1a6b,,exit()} function or hold the @code{Ctrl} key down
while you enter a @code{Z}, then hit the “@code{Enter}” key to get
back to your Windows command prompt.

You may also find that you have a Start-menu entry such as Start
‣ Programs ‣ Python 3.x ‣ Python (command line) that results in you
seeing the @code{>>>} prompt in a new window.  If so, the window will disappear
after you call the @ref{1a6b,,exit()} function or enter the @code{Ctrl-Z}
character; Windows is running a single “python”
command in the window, and closes it when you terminate the interpreter.

Now that we know the @code{py} command is recognized, you can give your
Python script to it. You’ll have to give either an absolute or a
relative path to the Python script. Let’s say your Python script is
located in your desktop and is named @code{hello.py}, and your command
prompt is nicely opened in your home directory so you’re seeing something
similar to:

@example
C:\Users\YourName>
@end example

So now you’ll ask the @code{py} command to give your script to Python by
typing @code{py} followed by your script path:

@example
C:\Users\YourName> py Desktop\hello.py
hello
@end example

@node How do I make Python scripts executable?,Why does Python sometimes take so long to start?,How do I run a Python program under Windows?,Python on Windows FAQ
@anchor{faq/windows how-do-i-make-python-scripts-executable}@anchor{4881}
@subsection How do I make Python scripts executable?


On Windows, the standard Python installer already associates the .py
extension with a file type (Python.File) and gives that file type an open
command that runs the interpreter (@code{D:\Program Files\Python\python.exe "%1"
%*}).  This is enough to make scripts executable from the command prompt as
‘foo.py’.  If you’d rather be able to execute the script by simple typing ‘foo’
with no extension you need to add .py to the PATHEXT environment variable.

@node Why does Python sometimes take so long to start?,How do I make an executable from a Python script?,How do I make Python scripts executable?,Python on Windows FAQ
@anchor{faq/windows why-does-python-sometimes-take-so-long-to-start}@anchor{4882}
@subsection Why does Python sometimes take so long to start?


Usually Python starts very quickly on Windows, but occasionally there are bug
reports that Python suddenly begins to take a long time to start up.  This is
made even more puzzling because Python will work fine on other Windows systems
which appear to be configured identically.

The problem may be caused by a misconfiguration of virus checking software on
the problem machine.  Some virus scanners have been known to introduce startup
overhead of two orders of magnitude when the scanner is configured to monitor
all reads from the filesystem.  Try checking the configuration of virus scanning
software on your systems to ensure that they are indeed configured identically.
McAfee, when configured to scan all file system read activity, is a particular
offender.

@node How do I make an executable from a Python script?,Is a * pyd file the same as a DLL?,Why does Python sometimes take so long to start?,Python on Windows FAQ
@anchor{faq/windows how-do-i-make-an-executable-from-a-python-script}@anchor{4883}
@subsection How do I make an executable from a Python script?


See @ref{47db,,How can I create a stand-alone binary from a Python script?} for a list of tools that can be used to
make executables.

@node Is a * pyd file the same as a DLL?,How can I embed Python into a Windows application?,How do I make an executable from a Python script?,Python on Windows FAQ
@anchor{faq/windows is-a-pyd-file-the-same-as-a-dll}@anchor{4884}
@subsection Is a @code{*.pyd} file the same as a DLL?


Yes, .pyd files are dll’s, but there are a few differences.  If you have a DLL
named @code{foo.pyd}, then it must have a function @code{PyInit_foo()}.  You can then
write Python “import foo”, and Python will search for foo.pyd (as well as
foo.py, foo.pyc) and if it finds it, will attempt to call @code{PyInit_foo()} to
initialize it.  You do not link your .exe with foo.lib, as that would cause
Windows to require the DLL to be present.

Note that the search path for foo.pyd is PYTHONPATH, not the same as the path
that Windows uses to search for foo.dll.  Also, foo.pyd need not be present to
run your program, whereas if you linked your program with a dll, the dll is
required.  Of course, foo.pyd is required if you want to say @code{import foo}.  In
a DLL, linkage is declared in the source code with @code{__declspec(dllexport)}.
In a .pyd, linkage is defined in a list of available functions.

@node How can I embed Python into a Windows application?,How do I keep editors from inserting tabs into my Python source?,Is a * pyd file the same as a DLL?,Python on Windows FAQ
@anchor{faq/windows how-can-i-embed-python-into-a-windows-application}@anchor{4885}
@subsection How can I embed Python into a Windows application?


Embedding the Python interpreter in a Windows app can be summarized as follows:


@enumerate 

@item 
Do _not_ build Python into your .exe file directly.  On Windows, Python must
be a DLL to handle importing modules that are themselves DLL’s.  (This is the
first key undocumented fact.)  Instead, link to @code{python@emph{NN}.dll}; it is
typically installed in @code{C:\Windows\System}.  @emph{NN} is the Python version, a
number such as “33” for Python 3.3.

You can link to Python in two different ways.  Load-time linking means
linking against @code{python@emph{NN}.lib}, while run-time linking means linking
against @code{python@emph{NN}.dll}.  (General note: @code{python@emph{NN}.lib} is the
so-called “import lib” corresponding to @code{python@emph{NN}.dll}.  It merely
defines symbols for the linker.)

Run-time linking greatly simplifies link options; everything happens at run
time.  Your code must load @code{python@emph{NN}.dll} using the Windows
@code{LoadLibraryEx()} routine.  The code must also use access routines and data
in @code{python@emph{NN}.dll} (that is, Python’s C API’s) using pointers obtained
by the Windows @code{GetProcAddress()} routine.  Macros can make using these
pointers transparent to any C code that calls routines in Python’s C API.

Borland note: convert @code{python@emph{NN}.lib} to OMF format using Coff2Omf.exe
first.

@c XXX what about static linking?

@item 
If you use SWIG, it is easy to create a Python “extension module” that will
make the app’s data and methods available to Python.  SWIG will handle just
about all the grungy details for you.  The result is C code that you link
@emph{into} your .exe file (!)  You do _not_ have to create a DLL file, and this
also simplifies linking.

@item 
SWIG will create an init function (a C function) whose name depends on the
name of the extension module.  For example, if the name of the module is leo,
the init function will be called initleo().  If you use SWIG shadow classes,
as you should, the init function will be called initleoc().  This initializes
a mostly hidden helper class used by the shadow class.

The reason you can link the C code in step 2 into your .exe file is that
calling the initialization function is equivalent to importing the module
into Python! (This is the second key undocumented fact.)

@item 
In short, you can use the following code to initialize the Python interpreter
with your extension module.

@example
#include <Python.h>
...
Py_Initialize();  // Initialize Python.
initmyAppc();  // Initialize (import) the helper class.
PyRun_SimpleString("import myApp");  // Import the shadow class.
@end example

@item 
There are two problems with Python’s C API which will become apparent if you
use a compiler other than MSVC, the compiler used to build pythonNN.dll.

Problem 1: The so-called “Very High Level” functions that take FILE *
arguments will not work in a multi-compiler environment because each
compiler’s notion of a struct FILE will be different.  From an implementation
standpoint these are very _low_ level functions.

Problem 2: SWIG generates the following code when generating wrappers to void
functions:

@example
Py_INCREF(Py_None);
_resultobj = Py_None;
return _resultobj;
@end example

Alas, Py_None is a macro that expands to a reference to a complex data
structure called _Py_NoneStruct inside pythonNN.dll.  Again, this code will
fail in a mult-compiler environment.  Replace such code by:

@example
return Py_BuildValue("");
@end example

It may be possible to use SWIG’s @code{%typemap} command to make the change
automatically, though I have not been able to get this to work (I’m a
complete SWIG newbie).

@item 
Using a Python shell script to put up a Python interpreter window from inside
your Windows app is not a good idea; the resulting window will be independent
of your app’s windowing system.  Rather, you (or the wxPythonWindow class)
should create a “native” interpreter window.  It is easy to connect that
window to the Python interpreter.  You can redirect Python’s i/o to _any_
object that supports read and write, so all you need is a Python object
(defined in your extension module) that contains read() and write() methods.
@end enumerate

@node How do I keep editors from inserting tabs into my Python source?,How do I check for a keypress without blocking?,How can I embed Python into a Windows application?,Python on Windows FAQ
@anchor{faq/windows how-do-i-keep-editors-from-inserting-tabs-into-my-python-source}@anchor{4886}
@subsection How do I keep editors from inserting tabs into my Python source?


The FAQ does not recommend using tabs, and the Python style guide, 
@geindex Python Enhancement Proposals; PEP 8
PEP 8@footnote{https://www.python.org/dev/peps/pep-0008},
recommends 4 spaces for distributed Python code; this is also the Emacs
python-mode default.

Under any editor, mixing tabs and spaces is a bad idea.  MSVC is no different in
this respect, and is easily configured to use spaces: Take Tools
‣ Options ‣ Tabs, and for file type “Default” set “Tab size” and “Indent
size” to 4, and select the “Insert spaces” radio button.

Python raises @ref{21c,,IndentationError} or @ref{1068,,TabError} if mixed tabs
and spaces are causing problems in leading whitespace.
You may also run the @ref{fa,,tabnanny} module to check a directory tree
in batch mode.

@node How do I check for a keypress without blocking?,,How do I keep editors from inserting tabs into my Python source?,Python on Windows FAQ
@anchor{faq/windows how-do-i-check-for-a-keypress-without-blocking}@anchor{4887}
@subsection How do I check for a keypress without blocking?


Use the @ref{b2,,msvcrt} module.  This is a standard Windows-specific extension module.
It defines a function @code{kbhit()} which checks whether a keyboard hit is
present, and @code{getch()} which gets one character without echoing it.

@node Graphic User Interface FAQ,“Why is Python Installed on my Computer?” FAQ,Python on Windows FAQ,Python Frequently Asked Questions
@anchor{faq/gui doc}@anchor{4888}@anchor{faq/gui graphic-user-interface-faq}@anchor{4889}
@section Graphic User Interface FAQ



@c XXX need review for Python 3.

@menu
* General GUI Questions:: 
* What GUI toolkits exist for Python?:: 
* Tkinter questions:: 

@end menu

@node General GUI Questions,What GUI toolkits exist for Python?,,Graphic User Interface FAQ
@anchor{faq/gui general-gui-questions}@anchor{488a}
@subsection General GUI Questions


@node What GUI toolkits exist for Python?,Tkinter questions,General GUI Questions,Graphic User Interface FAQ
@anchor{faq/gui what-gui-toolkits-exist-for-python}@anchor{488b}
@subsection What GUI toolkits exist for Python?


Standard builds of Python include an object-oriented interface to the Tcl/Tk
widget set, called @ref{35f1,,tkinter}.  This is probably the easiest to
install (since it comes included with most
binary distributions@footnote{https://www.python.org/downloads/} of Python) and use.
For more info about Tk, including pointers to the source, see the
Tcl/Tk home page@footnote{https://www.tcl.tk}.  Tcl/Tk is fully portable to the
macOS, Windows, and Unix platforms.

Depending on what platform(s) you are aiming at, there are also several
alternatives. A list of cross-platform@footnote{https://wiki.python.org/moin/GuiProgramming#Cross-Platform_Frameworks} and
platform-specific@footnote{https://wiki.python.org/moin/GuiProgramming#Platform-specific_Frameworks} GUI
frameworks can be found on the python wiki.

@node Tkinter questions,,What GUI toolkits exist for Python?,Graphic User Interface FAQ
@anchor{faq/gui tkinter-questions}@anchor{488c}
@subsection Tkinter questions


@menu
* How do I freeze Tkinter applications?:: 
* Can I have Tk events handled while waiting for I/O?:: 
* I can’t get key bindings to work in Tkinter; why?: I can’t get key bindings to work in Tkinter why?. 

@end menu

@node How do I freeze Tkinter applications?,Can I have Tk events handled while waiting for I/O?,,Tkinter questions
@anchor{faq/gui how-do-i-freeze-tkinter-applications}@anchor{488d}
@subsubsection How do I freeze Tkinter applications?


Freeze is a tool to create stand-alone applications.  When freezing Tkinter
applications, the applications will not be truly stand-alone, as the application
will still need the Tcl and Tk libraries.

One solution is to ship the application with the Tcl and Tk libraries, and point
to them at run-time using the 
@geindex TCL_LIBRARY
@geindex environment variable; TCL_LIBRARY
@code{TCL_LIBRARY} and 
@geindex TK_LIBRARY
@geindex environment variable; TK_LIBRARY
@code{TK_LIBRARY}
environment variables.

To get truly stand-alone applications, the Tcl scripts that form the library
have to be integrated into the application as well. One tool supporting that is
SAM (stand-alone modules), which is part of the Tix distribution
(@indicateurl{http://tix.sourceforge.net/}).

Build Tix with SAM enabled, perform the appropriate call to
@code{Tclsam_init()}, etc. inside Python’s
@code{Modules/tkappinit.c}, and link with libtclsam and libtksam (you
might include the Tix libraries as well).

@node Can I have Tk events handled while waiting for I/O?,I can’t get key bindings to work in Tkinter why?,How do I freeze Tkinter applications?,Tkinter questions
@anchor{faq/gui can-i-have-tk-events-handled-while-waiting-for-i-o}@anchor{488e}
@subsubsection Can I have Tk events handled while waiting for I/O?


On platforms other than Windows, yes, and you don’t even
need threads!  But you’ll have to restructure your I/O
code a bit.  Tk has the equivalent of Xt’s @code{XtAddInput()} call, which allows you
to register a callback function which will be called from the Tk mainloop when
I/O is possible on a file descriptor.  See @ref{360c,,File Handlers}.

@node I can’t get key bindings to work in Tkinter why?,,Can I have Tk events handled while waiting for I/O?,Tkinter questions
@anchor{faq/gui i-can-t-get-key-bindings-to-work-in-tkinter-why}@anchor{488f}
@subsubsection I can’t get key bindings to work in Tkinter: why?


An often-heard complaint is that event handlers bound to events with the
@code{bind()} method don’t get handled even when the appropriate key is pressed.

The most common cause is that the widget to which the binding applies doesn’t
have “keyboard focus”.  Check out the Tk documentation for the focus command.
Usually a widget is given the keyboard focus by clicking in it (but not for
labels; see the takefocus option).

@node “Why is Python Installed on my Computer?” FAQ,,Graphic User Interface FAQ,Python Frequently Asked Questions
@anchor{faq/installed doc}@anchor{4890}@anchor{faq/installed why-is-python-installed-on-my-computer-faq}@anchor{4891}
@section “Why is Python Installed on my Computer?” FAQ


@menu
* What is Python?: What is Python?<2>. 
* Why is Python installed on my machine?:: 
* Can I delete Python?:: 

@end menu

@node What is Python?<2>,Why is Python installed on my machine?,,“Why is Python Installed on my Computer?” FAQ
@anchor{faq/installed what-is-python}@anchor{4892}
@subsection What is Python?


Python is a programming language.  It’s used for many different applications.
It’s used in some high schools and colleges as an introductory programming
language because Python is easy to learn, but it’s also used by professional
software developers at places such as Google, NASA, and Lucasfilm Ltd.

If you wish to learn more about Python, start with the Beginner's Guide to Python@footnote{https://wiki.python.org/moin/BeginnersGuide}.

@node Why is Python installed on my machine?,Can I delete Python?,What is Python?<2>,“Why is Python Installed on my Computer?” FAQ
@anchor{faq/installed why-is-python-installed-on-my-machine}@anchor{4893}
@subsection Why is Python installed on my machine?


If you find Python installed on your system but don’t remember installing it,
there are several possible ways it could have gotten there.


@itemize *

@item 
Perhaps another user on the computer wanted to learn programming and installed
it; you’ll have to figure out who’s been using the machine and might have
installed it.

@item 
A third-party application installed on the machine might have been written in
Python and included a Python installation.  There are many such applications,
from GUI programs to network servers and administrative scripts.

@item 
Some Windows machines also have Python installed.  At this writing we’re aware
of computers from Hewlett-Packard and Compaq that include Python.  Apparently
some of HP/Compaq’s administrative tools are written in Python.

@item 
Many Unix-compatible operating systems, such as macOS and some Linux
distributions, have Python installed by default; it’s included in the base
installation.
@end itemize

@node Can I delete Python?,,Why is Python installed on my machine?,“Why is Python Installed on my Computer?” FAQ
@anchor{faq/installed can-i-delete-python}@anchor{4894}
@subsection Can I delete Python?


That depends on where Python came from.

If someone installed it deliberately, you can remove it without hurting
anything.  On Windows, use the Add/Remove Programs icon in the Control Panel.

If Python was installed by a third-party application, you can also remove it,
but that application will no longer work.  You should use that application’s
uninstaller rather than removing Python directly.

If Python came with your operating system, removing it is not recommended.  If
you remove it, whatever tools were written in Python will no longer run, and
some of them might be important to you.  Reinstalling the whole system would
then be required to fix things again.

@node Glossary,About these documents,Python Frequently Asked Questions,Top
@anchor{glossary doc}@anchor{4895}@anchor{glossary glossary}@anchor{14c0}@anchor{glossary id1}@anchor{4896}
@chapter Glossary


@c if you add new entries, keep the alphabetical sorting!


@table @asis
@anchor{glossary term-0}@anchor{14d8}
@geindex >>>

@item @code{>>>}

The default Python prompt of the interactive shell.  Often seen for code
examples which can be executed interactively in the interpreter.
@anchor{glossary term-}@anchor{14d9}
@geindex ...

@item @code{...}

Can refer to:


@itemize *

@item 
The default Python prompt of the interactive shell when entering the
code for an indented code block, when within a pair of matching left and
right delimiters (parentheses, square brackets, curly braces or triple
quotes), or after specifying a decorator.

@item 
The @ref{1a67,,Ellipsis} built-in constant.
@end itemize
@anchor{glossary term-2to3}@anchor{12ea}
@geindex 2to3

@item 2to3

A tool that tries to convert Python 2.x code to Python 3.x code by
handling most of the incompatibilities which can be detected by parsing the
source and traversing the parse tree.

2to3 is available in the standard library as @ref{a3,,lib2to3}; a standalone
entry point is provided as @code{Tools/scripts/2to3}.  See
@ref{e7d,,2to3 - Automated Python 2 to 3 code translation}.
@anchor{glossary term-abstract-base-class}@anchor{d5c}
@geindex abstract base class

@item abstract base class

Abstract base classes complement @ref{4897,,duck-typing} by
providing a way to define interfaces when other techniques like
@ref{707,,hasattr()} would be clumsy or subtly wrong (for example with
@ref{17b1,,magic methods}).  ABCs introduce virtual
subclasses, which are classes that don’t inherit from a class but are
still recognized by @ref{236,,isinstance()} and @ref{237,,issubclass()}; see the
@ref{3,,abc} module documentation.  Python comes with many built-in ABCs for
data structures (in the @ref{1d,,collections.abc} module), numbers (in the
@ref{bd,,numbers} module), streams (in the @ref{9d,,io} module), import finders
and loaders (in the @ref{97,,importlib.abc} module).  You can create your own
ABCs with the @ref{3,,abc} module.
@anchor{glossary term-annotation}@anchor{17ca}
@geindex annotation

@item annotation

A label associated with a variable, a class
attribute or a function parameter or return value,
used by convention as a @ref{17ce,,type hint}.

Annotations of local variables cannot be accessed at runtime, but
annotations of global variables, class attributes, and functions
are stored in the @code{__annotations__}
special attribute of modules, classes, and functions,
respectively.

See @ref{8af,,variable annotation}, @ref{151d,,function annotation}, 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484}
and 
@geindex Python Enhancement Proposals; PEP 526
PEP 526@footnote{https://www.python.org/dev/peps/pep-0526}, which describe this functionality.
Also see @ref{255,,Annotations Best Practices}
for best practices on working with annotations.
@anchor{glossary term-argument}@anchor{18d2}
@geindex argument

@item argument

A value passed to a @ref{18d1,,function} (or @ref{4898,,method}) when calling the
function.  There are two kinds of argument:


@itemize *

@item 
@emph{keyword argument}: an argument preceded by an identifier (e.g.
@code{name=}) in a function call or passed as a value in a dictionary
preceded by @code{**}.  For example, @code{3} and @code{5} are both keyword
arguments in the following calls to @ref{33b,,complex()}:

@example
complex(real=3, imag=5)
complex(**@{'real': 3, 'imag': 5@})
@end example

@item 
@emph{positional argument}: an argument that is not a keyword argument.
Positional arguments can appear at the beginning of an argument list
and/or be passed as elements of an @ref{dcc,,iterable} preceded by @code{*}.
For example, @code{3} and @code{5} are both positional arguments in the
following calls:

@example
complex(3, 5)
complex(*(3, 5))
@end example
@end itemize

Arguments are assigned to the named local variables in a function body.
See the @ref{8d6,,Calls} section for the rules governing this assignment.
Syntactically, any expression can be used to represent an argument; the
evaluated value is assigned to the local variable.

See also the @ref{18e1,,parameter} glossary entry, the FAQ question on
@ref{47e6,,the difference between arguments and parameters}, and 
@geindex Python Enhancement Proposals; PEP 362
PEP 362@footnote{https://www.python.org/dev/peps/pep-0362}.
@anchor{glossary term-asynchronous-context-manager}@anchor{167}
@geindex asynchronous context manager

@item asynchronous context manager

An object which controls the environment seen in an
@ref{169,,async with} statement by defining @code{__aenter__()} and
@code{__aexit__()} methods.  Introduced by 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492}.
@anchor{glossary term-asynchronous-generator}@anchor{18ac}
@geindex asynchronous generator

@item asynchronous generator

A function which returns an @ref{3b71,,asynchronous generator iterator}.  It
looks like a coroutine function defined with @ref{1c0,,async def} except
that it contains @ref{4c5,,yield} expressions for producing a series of
values usable in an @ref{5cd,,async for} loop.

Usually refers to an asynchronous generator function, but may refer to an
@emph{asynchronous generator iterator} in some contexts.  In cases where the
intended meaning isn’t clear, using the full terms avoids ambiguity.

An asynchronous generator function may contain @ref{4d5,,await}
expressions as well as @ref{5cd,,async for}, and @ref{169,,async with}
statements.
@anchor{glossary term-asynchronous-generator-iterator}@anchor{3b71}
@geindex asynchronous generator iterator

@item asynchronous generator iterator

An object created by a @ref{18ac,,asynchronous generator} function.

This is an @ref{8d2,,asynchronous iterator} which when called using the
@code{__anext__()} method returns an awaitable object which will execute
the body of the asynchronous generator function until the next
@ref{4c5,,yield} expression.

Each @ref{4c5,,yield} temporarily suspends processing, remembering the
location execution state (including local variables and pending
try-statements).  When the @emph{asynchronous generator iterator} effectively
resumes with another awaitable returned by @code{__anext__()}, it
picks up where it left off.  See 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492} and 
@geindex Python Enhancement Proposals; PEP 525
PEP 525@footnote{https://www.python.org/dev/peps/pep-0525}.
@anchor{glossary term-asynchronous-iterable}@anchor{8d0}
@geindex asynchronous iterable

@item asynchronous iterable

An object, that can be used in an @ref{5cd,,async for} statement.
Must return an @ref{8d2,,asynchronous iterator} from its
@code{__aiter__()} method.  Introduced by 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492}.
@anchor{glossary term-asynchronous-iterator}@anchor{8d2}
@geindex asynchronous iterator

@item asynchronous iterator

An object that implements the @code{__aiter__()} and @code{__anext__()}
methods.  @code{__anext__} must return an @ref{7bd,,awaitable} object.
@ref{5cd,,async for} resolves the awaitables returned by an asynchronous
iterator’s @code{__anext__()} method until it raises a
@ref{12da,,StopAsyncIteration} exception.  Introduced by 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492}.
@anchor{glossary term-attribute}@anchor{1d86}
@geindex attribute

@item attribute

A value associated with an object which is referenced by name using
dotted expressions.  For example, if an object @emph{o} has an attribute
@emph{a} it would be referenced as @emph{o.a}.
@anchor{glossary term-awaitable}@anchor{7bd}
@geindex awaitable

@item awaitable

An object that can be used in an @ref{4d5,,await} expression.  Can be
a @ref{3cf,,coroutine} or an object with an @code{__await__()} method.
See also 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492}.
@anchor{glossary term-BDFL}@anchor{4899}
@geindex BDFL

@item BDFL

Benevolent Dictator For Life, a.k.a. Guido van Rossum@footnote{https://gvanrossum.github.io/}, Python’s creator.
@anchor{glossary term-binary-file}@anchor{2138}
@geindex binary file

@item binary file

A @ref{d68,,file object} able to read and write
@ref{87e,,bytes-like objects}.
Examples of binary files are files opened in binary mode (@code{'rb'},
@code{'wb'} or @code{'rb+'}), @code{sys.stdin.buffer},
@code{sys.stdout.buffer}, and instances of @ref{a15,,io.BytesIO} and
@ref{43f,,gzip.GzipFile}.

See also @ref{1566,,text file} for a file object able to read and write
@ref{2ef,,str} objects.
@anchor{glossary term-borrowed-reference}@anchor{4112}
@geindex borrowed reference

@item borrowed reference

In Python’s C API, a borrowed reference is a reference to an object.
It does not modify the object reference count. It becomes a dangling
pointer if the object is destroyed. For example, a garbage collection can
remove the last @ref{1fc,,strong reference} to the object and so destroy it.

Calling @ref{372,,Py_INCREF()} on the @ref{4112,,borrowed reference} is
recommended to convert it to a @ref{1fc,,strong reference} in-place, except
when the object cannot be destroyed before the last usage of the borrowed
reference. The @ref{36a,,Py_NewRef()} function can be used to create a new
@ref{1fc,,strong reference}.
@anchor{glossary term-bytes-like-object}@anchor{87e}
@geindex bytes-like object

@item bytes-like object

An object that supports the @ref{1a3e,,Buffer Protocol} and can
export a C-@ref{1b0f,,contiguous} buffer. This includes all @ref{2f0,,bytes},
@ref{2f1,,bytearray}, and @ref{25e,,array.array} objects, as well as many
common @ref{573,,memoryview} objects.  Bytes-like objects can
be used for various operations that work with binary data; these include
compression, saving to a binary file, and sending over a socket.

Some operations need the binary data to be mutable.  The documentation
often refers to these as “read-write bytes-like objects”.  Example
mutable buffer objects include @ref{2f1,,bytearray} and a
@ref{573,,memoryview} of a @ref{2f1,,bytearray}.
Other operations require the binary data to be stored in
immutable objects (“read-only bytes-like objects”); examples
of these include @ref{2f0,,bytes} and a @ref{573,,memoryview}
of a @ref{2f0,,bytes} object.
@anchor{glossary term-bytecode}@anchor{8a8}
@geindex bytecode

@item bytecode

Python source code is compiled into bytecode, the internal representation
of a Python program in the CPython interpreter.  The bytecode is also
cached in @code{.pyc} files so that executing the same file is
faster the second time (recompilation from source to bytecode can be
avoided).  This “intermediate language” is said to run on a
@ref{2bf5,,virtual machine} that executes the machine code corresponding to
each bytecode. Do note that bytecodes are not expected to work between
different Python virtual machines, nor to be stable between Python
releases.

A list of bytecode instructions can be found in the documentation for
@ref{3e56,,the dis module}.
@anchor{glossary term-callback}@anchor{2a2d}
@geindex callback

@item callback

A subroutine function which is passed as an argument to be executed at
some point in the future.
@anchor{glossary term-class}@anchor{11ae}
@geindex class

@item class

A template for creating user-defined objects. Class definitions
normally contain method definitions which operate on instances of the
class.
@anchor{glossary term-class-variable}@anchor{489a}
@geindex class variable

@item class variable

A variable defined in a class and intended to be modified only at
class level (i.e., not in an instance of the class).
@anchor{glossary term-complex-number}@anchor{489b}
@geindex complex number

@item complex number

An extension of the familiar real number system in which all numbers are
expressed as a sum of a real part and an imaginary part.  Imaginary
numbers are real multiples of the imaginary unit (the square root of
@code{-1}), often written @code{i} in mathematics or @code{j} in
engineering.  Python has built-in support for complex numbers, which are
written with this latter notation; the imaginary part is written with a
@code{j} suffix, e.g., @code{3+1j}.  To get access to complex equivalents of the
@ref{ad,,math} module, use @ref{17,,cmath}.  Use of complex numbers is a fairly
advanced mathematical feature.  If you’re not aware of a need for them,
it’s almost certain you can safely ignore them.
@anchor{glossary term-context-manager}@anchor{166}
@geindex context manager

@item context manager

An object which controls the environment seen in a @ref{168,,with}
statement by defining @code{__enter__()} and @code{__exit__()} methods.
See 
@geindex Python Enhancement Proposals; PEP 343
PEP 343@footnote{https://www.python.org/dev/peps/pep-0343}.
@anchor{glossary term-context-variable}@anchor{489c}
@geindex context variable

@item context variable

A variable which can have different values depending on its context.
This is similar to Thread-Local Storage in which each execution
thread may have a different value for a variable. However, with context
variables, there may be several contexts in one execution thread and the
main usage for context variables is to keep track of variables in
concurrent asynchronous tasks.
See @ref{23,,contextvars}.
@anchor{glossary term-contiguous}@anchor{1b0f}
@geindex contiguous

@item contiguous

@geindex C-contiguous
@geindex Fortran contiguous

A buffer is considered contiguous exactly if it is either
@emph{C-contiguous} or @emph{Fortran contiguous}.  Zero-dimensional buffers are
C and Fortran contiguous.  In one-dimensional arrays, the items
must be laid out in memory next to each other, in order of
increasing indexes starting from zero.  In multidimensional
C-contiguous arrays, the last index varies the fastest when
visiting items in order of memory address.  However, in
Fortran contiguous arrays, the first index varies the fastest.
@anchor{glossary term-coroutine}@anchor{3cf}
@geindex coroutine

@item coroutine

Coroutines are a more generalized form of subroutines. Subroutines are
entered at one point and exited at another point.  Coroutines can be
entered, exited, and resumed at many different points.  They can be
implemented with the @ref{1c0,,async def} statement.  See also
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492}.
@anchor{glossary term-coroutine-function}@anchor{8cf}
@geindex coroutine function

@item coroutine function

A function which returns a @ref{3cf,,coroutine} object.  A coroutine
function may be defined with the @ref{1c0,,async def} statement,
and may contain @ref{4d5,,await}, @ref{5cd,,async for}, and
@ref{169,,async with} keywords.  These were introduced
by 
@geindex Python Enhancement Proposals; PEP 492
PEP 492@footnote{https://www.python.org/dev/peps/pep-0492}.
@anchor{glossary term-CPython}@anchor{17a4}
@geindex CPython

@item CPython

The canonical implementation of the Python programming language, as
distributed on python.org@footnote{https://www.python.org}.  The term “CPython”
is used when necessary to distinguish this implementation from others
such as Jython or IronPython.
@anchor{glossary term-decorator}@anchor{1bf}
@geindex decorator

@item decorator

A function returning another function, usually applied as a function
transformation using the @code{@@wrapper} syntax.  Common examples for
decorators are @ref{254,,classmethod()} and @ref{253,,staticmethod()}.

The decorator syntax is merely syntactic sugar, the following two
function definitions are semantically equivalent:

@example
def f(arg):
    ...
f = staticmethod(f)

@@staticmethod
def f(arg):
    ...
@end example

The same concept exists for classes, but is less commonly used there.  See
the documentation for @ref{151c,,function definitions} and
@ref{e72,,class definitions} for more about decorators.
@anchor{glossary term-descriptor}@anchor{17bc}
@geindex descriptor

@item descriptor

Any object which defines the methods @code{__get__()}, @code{__set__()}, or
@code{__delete__()}.  When a class attribute is a descriptor, its special
binding behavior is triggered upon attribute lookup.  Normally, using
@emph{a.b} to get, set or delete an attribute looks up the object named @emph{b} in
the class dictionary for @emph{a}, but if @emph{b} is a descriptor, the respective
descriptor method gets called.  Understanding descriptors is a key to a
deep understanding of Python because they are the basis for many features
including functions, methods, properties, class methods, static methods,
and reference to super classes.

For more information about descriptors’ methods, see @ref{793,,Implementing Descriptors}
or the @ref{4661,,Descriptor How To Guide}.
@anchor{glossary term-dictionary}@anchor{489d}
@geindex dictionary

@item dictionary

An associative array, where arbitrary keys are mapped to values.  The
keys can be any object with @code{__hash__()} and @code{__eq__()} methods.
Called a hash in Perl.
@anchor{glossary term-dictionary-comprehension}@anchor{489e}
@geindex dictionary comprehension

@item dictionary comprehension

A compact way to process all or part of the elements in an iterable and
return a dictionary with the results. @code{results = @{n: n ** 2 for n in
range(10)@}} generates a dictionary containing key @code{n} mapped to
value @code{n ** 2}. See @ref{1887,,Displays for lists@comma{} sets and dictionaries}.
@anchor{glossary term-dictionary-view}@anchor{1dde}
@geindex dictionary view

@item dictionary view

The objects returned from @ref{243,,dict.keys()}, @ref{244,,dict.values()}, and
@ref{245,,dict.items()} are called dictionary views. They provide a dynamic
view on the dictionary’s entries, which means that when the dictionary
changes, the view reflects these changes. To force the
dictionary view to become a full list use @code{list(dictview)}.  See
@ref{1b2f,,Dictionary view objects}.
@anchor{glossary term-docstring}@anchor{1a05}
@geindex docstring

@item docstring

A string literal which appears as the first expression in a class,
function or module.  While ignored when the suite is executed, it is
recognized by the compiler and put into the @code{__doc__} attribute
of the enclosing class, function or module.  Since it is available via
introspection, it is the canonical place for documentation of the
object.
@anchor{glossary term-duck-typing}@anchor{4897}
@geindex duck-typing

@item duck-typing

A programming style which does not look at an object’s type to determine
if it has the right interface; instead, the method or attribute is simply
called or used (“If it looks like a duck and quacks like a duck, it
must be a duck.”)  By emphasizing interfaces rather than specific types,
well-designed code improves its flexibility by allowing polymorphic
substitution.  Duck-typing avoids tests using @ref{89e,,type()} or
@ref{236,,isinstance()}.  (Note, however, that duck-typing can be complemented
with @ref{d5c,,abstract base classes}.)  Instead, it
typically employs @ref{707,,hasattr()} tests or @ref{23e6,,EAFP} programming.
@anchor{glossary term-EAFP}@anchor{23e6}
@geindex EAFP

@item EAFP

Easier to ask for forgiveness than permission.  This common Python coding
style assumes the existence of valid keys or attributes and catches
exceptions if the assumption proves false.  This clean and fast style is
characterized by the presence of many @ref{f6d,,try} and @ref{16e,,except}
statements.  The technique contrasts with the @ref{489f,,LBYL} style
common to many other languages such as C.
@anchor{glossary term-expression}@anchor{17d2}
@geindex expression

@item expression

A piece of syntax which can be evaluated to some value.  In other words,
an expression is an accumulation of expression elements like literals,
names, attribute access, operators or function calls which all return a
value.  In contrast to many other languages, not all language constructs
are expressions.  There are also @ref{2034,,statement}s which cannot be used
as expressions, such as @ref{14ec,,while}.  Assignments are also statements,
not expressions.
@anchor{glossary term-extension-module}@anchor{48a0}
@geindex extension module

@item extension module

A module written in C or C++, using Python’s C API to interact with the
core and with user code.
@anchor{glossary term-f-string}@anchor{4af}
@geindex f-string

@item f-string

String literals prefixed with @code{'f'} or @code{'F'} are commonly called
“f-strings” which is short for
@ref{4b0,,formatted string literals}.  See also 
@geindex Python Enhancement Proposals; PEP 498
PEP 498@footnote{https://www.python.org/dev/peps/pep-0498}.
@anchor{glossary term-file-object}@anchor{d68}
@geindex file object

@item file object

An object exposing a file-oriented API (with methods such as
@code{read()} or @code{write()}) to an underlying resource.  Depending
on the way it was created, a file object can mediate access to a real
on-disk file or to another type of storage or communication device
(for example standard input/output, in-memory buffers, sockets, pipes,
etc.).  File objects are also called @emph{file-like objects} or
@emph{streams}.

There are actually three categories of file objects: raw
@ref{2138,,binary files}, buffered
@ref{2138,,binary files} and @ref{1566,,text files}.
Their interfaces are defined in the @ref{9d,,io} module.  The canonical
way to create a file object is by using the @ref{1c8,,open()} function.
@anchor{glossary term-file-like-object}@anchor{1e54}
@geindex file-like object

@item file-like object

A synonym for @ref{d68,,file object}.
@anchor{glossary term-filesystem-encoding-and-error-handler}@anchor{1628}
@geindex filesystem encoding and error handler

@item filesystem encoding and error handler

Encoding and error handler used by Python to decode bytes from the
operating system and encode Unicode to the operating system.

The filesystem encoding must guarantee to successfully decode all bytes
below 128. If the file system encoding fails to provide this guarantee,
API functions can raise @ref{e49,,UnicodeError}.

The @ref{79e,,sys.getfilesystemencoding()} and
@ref{82f,,sys.getfilesystemencodeerrors()} functions can be used to get the
filesystem encoding and error handler.

The @ref{1628,,filesystem encoding and error handler} are configured at
Python startup by the @code{PyConfig_Read()} function: see
@ref{2378,,filesystem_encoding} and
@ref{2379,,filesystem_errors} members of @ref{206,,PyConfig}.

See also the @ref{237a,,locale encoding}.
@anchor{glossary term-finder}@anchor{1847}
@geindex finder

@item finder

An object that tries to find the @ref{1848,,loader} for a module that is
being imported.

Since Python 3.3, there are two types of finder: @ref{c05,,meta path finders} for use with @ref{87c,,sys.meta_path}, and @ref{c06,,path entry finders} for use with @ref{bb5,,sys.path_hooks}.

See 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302}, 
@geindex Python Enhancement Proposals; PEP 420
PEP 420@footnote{https://www.python.org/dev/peps/pep-0420} and 
@geindex Python Enhancement Proposals; PEP 451
PEP 451@footnote{https://www.python.org/dev/peps/pep-0451} for much more detail.
@anchor{glossary term-floor-division}@anchor{43b}
@geindex floor division

@item floor division

Mathematical division that rounds down to nearest integer.  The floor
division operator is @code{//}.  For example, the expression @code{11 // 4}
evaluates to @code{2} in contrast to the @code{2.75} returned by float true
division.  Note that @code{(-11) // 4} is @code{-3} because that is @code{-2.75}
rounded @emph{downward}. See 
@geindex Python Enhancement Proposals; PEP 238
PEP 238@footnote{https://www.python.org/dev/peps/pep-0238}.
@anchor{glossary term-function}@anchor{18d1}
@geindex function

@item function

A series of statements which returns some value to a caller. It can also
be passed zero or more @ref{18d2,,arguments} which may be used in
the execution of the body. See also @ref{18e1,,parameter}, @ref{4898,,method},
and the @ref{151c,,Function definitions} section.
@anchor{glossary term-function-annotation}@anchor{151d}
@geindex function annotation

@item function annotation

An @ref{17ca,,annotation} of a function parameter or return value.

Function annotations are usually used for
@ref{17ce,,type hints}: for example, this function is expected to take two
@ref{1b1,,int} arguments and is also expected to have an @ref{1b1,,int}
return value:

@example
def sum_two_numbers(a: int, b: int) -> int:
   return a + b
@end example

Function annotation syntax is explained in section @ref{151c,,Function definitions}.

See @ref{8af,,variable annotation} and 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484},
which describe this functionality.
Also see @ref{255,,Annotations Best Practices}
for best practices on working with annotations.
@anchor{glossary term-__future__}@anchor{90a}
@geindex __future__

@item __future__

A @ref{1969,,future statement}, @code{from __future__ import <feature>},
directs the compiler to compile the current module using syntax or
semantics that will become standard in a future release of Python.
The @ref{0,,__future__} module documents the possible values of
@emph{feature}.  By importing this module and evaluating its variables,
you can see when a new feature was first added to the language and
when it will (or did) become the default:

@example
>>> import __future__
>>> __future__.division
_Feature((2, 2, 0, 'alpha', 2), (3, 0, 0, 'alpha', 0), 8192)
@end example
@anchor{glossary term-garbage-collection}@anchor{1192}
@geindex garbage collection

@item garbage collection

The process of freeing memory when it is not used anymore.  Python
performs garbage collection via reference counting and a cyclic garbage
collector that is able to detect and break reference cycles.  The
garbage collector can be controlled using the @ref{80,,gc} module.

@geindex generator
@anchor{glossary term-generator}@anchor{bf7}
@geindex generator

@item generator

A function which returns a @ref{715,,generator iterator}.  It looks like a
normal function except that it contains @ref{4c5,,yield} expressions
for producing a series of values usable in a for-loop or that can be
retrieved one at a time with the @ref{252,,next()} function.

Usually refers to a generator function, but may refer to a
@emph{generator iterator} in some contexts.  In cases where the intended
meaning isn’t clear, using the full terms avoids ambiguity.
@anchor{glossary term-generator-iterator}@anchor{715}
@geindex generator iterator

@item generator iterator

An object created by a @ref{bf7,,generator} function.

Each @ref{4c5,,yield} temporarily suspends processing, remembering the
location execution state (including local variables and pending
try-statements).  When the @emph{generator iterator} resumes, it picks up where
it left off (in contrast to functions which start fresh on every
invocation).

@geindex generator expression
@anchor{glossary term-generator-expression}@anchor{48a1}
@geindex generator expression

@item generator expression

An expression that returns an iterator.  It looks like a normal expression
followed by a @code{for} clause defining a loop variable, range,
and an optional @code{if} clause.  The combined expression
generates values for an enclosing function:

@example
>>> sum(i*i for i in range(10))         # sum of squares 0, 1, 4, ... 81
285
@end example
@anchor{glossary term-generic-function}@anchor{4f6}
@geindex generic function

@item generic function

A function composed of multiple functions implementing the same operation
for different types. Which implementation should be used during a call is
determined by the dispatch algorithm.

See also the @ref{4f7,,single dispatch} glossary entry, the
@ref{65b,,functools.singledispatch()} decorator, and 
@geindex Python Enhancement Proposals; PEP 443
PEP 443@footnote{https://www.python.org/dev/peps/pep-0443}.
@anchor{glossary term-generic-type}@anchor{11b0}
@geindex generic type

@item generic type

A @ref{11af,,type} that can be parameterized; typically a
@ref{17d7,,container class} such as @ref{43a,,list} or
@ref{3b0,,dict}. Used for @ref{17ce,,type hints} and
@ref{17ca,,annotations}.

For more details, see @ref{268,,generic alias types},
@geindex Python Enhancement Proposals; PEP 483
PEP 483@footnote{https://www.python.org/dev/peps/pep-0483}, 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484}, 
@geindex Python Enhancement Proposals; PEP 585
PEP 585@footnote{https://www.python.org/dev/peps/pep-0585}, and the @ref{120,,typing} module.
@anchor{glossary term-GIL}@anchor{385}
@geindex GIL

@item GIL

See @ref{7ae,,global interpreter lock}.
@anchor{glossary term-global-interpreter-lock}@anchor{7ae}
@geindex global interpreter lock

@item global interpreter lock

The mechanism used by the @ref{17a4,,CPython} interpreter to assure that
only one thread executes Python @ref{8a8,,bytecode} at a time.
This simplifies the CPython implementation by making the object model
(including critical built-in types such as @ref{3b0,,dict}) implicitly
safe against concurrent access.  Locking the entire interpreter
makes it easier for the interpreter to be multi-threaded, at the
expense of much of the parallelism afforded by multi-processor
machines.

However, some extension modules, either standard or third-party,
are designed so as to release the GIL when doing computationally-intensive
tasks such as compression or hashing.  Also, the GIL is always released
when doing I/O.

Past efforts to create a “free-threaded” interpreter (one which locks
shared data at a much finer granularity) have not been successful
because performance suffered in the common single-processor case. It
is believed that overcoming this performance issue would make the
implementation much more complicated and therefore costlier to maintain.
@anchor{glossary term-hash-based-pyc}@anchor{48a2}
@geindex hash-based pyc

@item hash-based pyc

A bytecode cache file that uses the hash rather than the last-modified
time of the corresponding source file to determine its validity. See
@ref{607,,Cached bytecode invalidation}.
@anchor{glossary term-hashable}@anchor{2dc}
@geindex hashable

@item hashable

An object is @emph{hashable} if it has a hash value which never changes during
its lifetime (it needs a @code{__hash__()} method), and can be compared to
other objects (it needs an @code{__eq__()} method).  Hashable objects which
compare equal must have the same hash value.

Hashability makes an object usable as a dictionary key and a set member,
because these data structures use the hash value internally.

Most of Python’s immutable built-in objects are hashable; mutable
containers (such as lists or dictionaries) are not; immutable
containers (such as tuples and frozensets) are only hashable if
their elements are hashable.  Objects which are
instances of user-defined classes are hashable by default.  They all
compare unequal (except with themselves), and their hash value is derived
from their @ref{f80,,id()}.
@anchor{glossary term-IDLE}@anchor{48a3}
@geindex IDLE

@item IDLE

An Integrated Development Environment for Python.  IDLE is a basic editor
and interpreter environment which ships with the standard distribution of
Python.
@anchor{glossary term-immutable}@anchor{14e1}
@geindex immutable

@item immutable

An object with a fixed value.  Immutable objects include numbers, strings and
tuples.  Such an object cannot be altered.  A new object has to
be created if a different value has to be stored.  They play an important
role in places where a constant hash value is needed, for example as a key
in a dictionary.
@anchor{glossary term-import-path}@anchor{184a}
@geindex import path

@item import path

A list of locations (or @ref{1864,,path entries}) that are
searched by the @ref{184d,,path based finder} for modules to import. During
import, this list of locations usually comes from @ref{489,,sys.path}, but
for subpackages it may also come from the parent package’s @code{__path__}
attribute.
@anchor{glossary term-importing}@anchor{183a}
@geindex importing

@item importing

The process by which Python code in one module is made available to
Python code in another module.
@anchor{glossary term-importer}@anchor{1849}
@geindex importer

@item importer

An object that both finds and loads a module; both a
@ref{1847,,finder} and @ref{1848,,loader} object.
@anchor{glossary term-interactive}@anchor{48a4}
@geindex interactive

@item interactive

Python has an interactive interpreter which means you can enter
statements and expressions at the interpreter prompt, immediately
execute them and see their results.  Just launch @code{python} with no
arguments (possibly by selecting it from your computer’s main
menu). It is a very powerful way to test out new ideas or inspect
modules and packages (remember @code{help(x)}).
@anchor{glossary term-interpreted}@anchor{48a5}
@geindex interpreted

@item interpreted

Python is an interpreted language, as opposed to a compiled one,
though the distinction can be blurry because of the presence of the
bytecode compiler.  This means that source files can be run directly
without explicitly creating an executable which is then run.
Interpreted languages typically have a shorter development/debug cycle
than compiled ones, though their programs generally also run more
slowly.  See also @ref{48a4,,interactive}.
@anchor{glossary term-interpreter-shutdown}@anchor{9e2}
@geindex interpreter shutdown

@item interpreter shutdown

When asked to shut down, the Python interpreter enters a special phase
where it gradually releases all allocated resources, such as modules
and various critical internal structures.  It also makes several calls
to the @ref{1192,,garbage collector}. This can trigger
the execution of code in user-defined destructors or weakref callbacks.
Code executed during the shutdown phase can encounter various
exceptions as the resources it relies on may not function anymore
(common examples are library modules or the warnings machinery).

The main reason for interpreter shutdown is that the @code{__main__} module
or the script being run has finished executing.
@anchor{glossary term-iterable}@anchor{dcc}
@geindex iterable

@item iterable

An object capable of returning its members one at a time. Examples of
iterables include all sequence types (such as @ref{43a,,list}, @ref{2ef,,str},
and @ref{3e5,,tuple}) and some non-sequence types like @ref{3b0,,dict},
@ref{d68,,file objects}, and objects of any classes you define
with an @code{__iter__()} method or with a @code{__getitem__()} method
that implements @ref{14e7,,Sequence} semantics.

Iterables can be
used in a @ref{161,,for} loop and in many other places where a sequence is
needed (@ref{247,,zip()}, @ref{2f4,,map()}, …).  When an iterable object is passed
as an argument to the built-in function @ref{251,,iter()}, it returns an
iterator for the object.  This iterator is good for one pass over the set
of values.  When using iterables, it is usually not necessary to call
@ref{251,,iter()} or deal with iterator objects yourself.  The @code{for}
statement does that automatically for you, creating a temporary unnamed
variable to hold the iterator for the duration of the loop.  See also
@ref{11c3,,iterator}, @ref{14e7,,sequence}, and @ref{bf7,,generator}.
@anchor{glossary term-iterator}@anchor{11c3}
@geindex iterator

@item iterator

An object representing a stream of data.  Repeated calls to the iterator’s
@ref{e6d,,__next__()} method (or passing it to the built-in function
@ref{252,,next()}) return successive items in the stream.  When no more data
are available a @ref{736,,StopIteration} exception is raised instead.  At this
point, the iterator object is exhausted and any further calls to its
@code{__next__()} method just raise @ref{736,,StopIteration} again.  Iterators
are required to have an @code{__iter__()} method that returns the iterator
object itself so every iterator is also iterable and may be used in most
places where other iterables are accepted.  One notable exception is code
which attempts multiple iteration passes.  A container object (such as a
@ref{43a,,list}) produces a fresh new iterator each time you pass it to the
@ref{251,,iter()} function or use it in a @ref{161,,for} loop.  Attempting this
with an iterator will just return the same exhausted iterator object used
in the previous iteration pass, making it appear like an empty container.

More information can be found in @ref{1a51,,Iterator Types}.

@strong{CPython implementation detail:} CPython does not consistently apply the requirement that an iterator
define @code{__iter__()}.
@anchor{glossary term-key-function}@anchor{964}
@geindex key function

@item key function

A key function or collation function is a callable that returns a value
used for sorting or ordering.  For example, @ref{3507,,locale.strxfrm()} is
used to produce a sort key that is aware of locale specific sort
conventions.

A number of tools in Python accept key functions to control how elements
are ordered or grouped.  They include @ref{a7d,,min()}, @ref{a7e,,max()},
@ref{704,,sorted()}, @ref{705,,list.sort()}, @ref{963,,heapq.merge()},
@ref{1df5,,heapq.nsmallest()}, @ref{1df4,,heapq.nlargest()}, and
@ref{1fe1,,itertools.groupby()}.

There are several ways to create a key function.  For example. the
@ref{1a9c,,str.lower()} method can serve as a key function for case insensitive
sorts.  Alternatively, a key function can be built from a
@ref{e42,,lambda} expression such as @code{lambda r: (r[0], r[2])}.  Also,
the @ref{be,,operator} module provides three key function constructors:
@ref{99a,,attrgetter()}, @ref{578,,itemgetter()}, and
@ref{99b,,methodcaller()}.  See the @ref{1a5e,,Sorting HOW TO} for examples of how to create and use key functions.
@anchor{glossary term-keyword-argument}@anchor{85f}
@geindex keyword argument

@item keyword argument

See @ref{18d2,,argument}.
@anchor{glossary term-lambda}@anchor{1fec}
@geindex lambda

@item lambda

An anonymous inline function consisting of a single @ref{17d2,,expression}
which is evaluated when the function is called.  The syntax to create
a lambda function is @code{lambda [parameters]: expression}
@anchor{glossary term-LBYL}@anchor{489f}
@geindex LBYL

@item LBYL

Look before you leap.  This coding style explicitly tests for
pre-conditions before making calls or lookups.  This style contrasts with
the @ref{23e6,,EAFP} approach and is characterized by the presence of many
@ref{2fb,,if} statements.

In a multi-threaded environment, the LBYL approach can risk introducing a
race condition between “the looking” and “the leaping”.  For example, the
code, @code{if key in mapping: return mapping[key]} can fail if another
thread removes @emph{key} from @emph{mapping} after the test, but before the lookup.
This issue can be solved with locks or by using the EAFP approach.
@anchor{glossary term-locale-encoding}@anchor{237a}
@geindex locale encoding

@item locale encoding

On Unix, it is the encoding of the LC_CTYPE locale. It can be set with
@code{locale.setlocale(locale.LC_CTYPE, new_locale)}.

On Windows, it is the ANSI code page (ex: @code{cp1252}).

@code{locale.getpreferredencoding(False)} can be used to get the locale
encoding.

Python uses the @ref{1628,,filesystem encoding and error handler} to convert
between Unicode filenames and bytes filenames.
@anchor{glossary term-list}@anchor{48a6}
@geindex list

@item list

A built-in Python @ref{14e7,,sequence}.  Despite its name it is more akin
to an array in other languages than to a linked list since access to
elements is O(1).
@anchor{glossary term-list-comprehension}@anchor{48a7}
@geindex list comprehension

@item list comprehension

A compact way to process all or part of the elements in a sequence and
return a list with the results.  @code{result = ['@{:#04x@}'.format(x) for x in
range(256) if x % 2 == 0]} generates a list of strings containing
even hex numbers (0x..) in the range from 0 to 255. The @ref{2fb,,if}
clause is optional.  If omitted, all elements in @code{range(256)} are
processed.
@anchor{glossary term-loader}@anchor{1848}
@geindex loader

@item loader

An object that loads a module. It must define a method named
@code{load_module()}. A loader is typically returned by a
@ref{1847,,finder}. See 
@geindex Python Enhancement Proposals; PEP 302
PEP 302@footnote{https://www.python.org/dev/peps/pep-0302} for details and
@ref{a39,,importlib.abc.Loader} for an @ref{d5c,,abstract base class}.
@anchor{glossary term-magic-method}@anchor{48a8}
@geindex magic method

@item magic method

@geindex magic; method

An informal synonym for @ref{111a,,special method}.
@anchor{glossary term-mapping}@anchor{d62}
@geindex mapping

@item mapping

A container object that supports arbitrary key lookups and implements the
methods specified in the @ref{3ab,,Mapping} or
@ref{c3f,,MutableMapping}
@ref{33d,,abstract base classes}.  Examples
include @ref{3b0,,dict}, @ref{d63,,collections.defaultdict},
@ref{4e5,,collections.OrderedDict} and @ref{c2a,,collections.Counter}.
@anchor{glossary term-meta-path-finder}@anchor{c05}
@geindex meta path finder

@item meta path finder

A @ref{1847,,finder} returned by a search of @ref{87c,,sys.meta_path}.  Meta path
finders are related to, but different from @ref{c06,,path entry finders}.

See @ref{30e,,importlib.abc.MetaPathFinder} for the methods that meta path
finders implement.
@anchor{glossary term-metaclass}@anchor{17d3}
@geindex metaclass

@item metaclass

The class of a class.  Class definitions create a class name, a class
dictionary, and a list of base classes.  The metaclass is responsible for
taking those three arguments and creating the class.  Most object oriented
programming languages provide a default implementation.  What makes Python
special is that it is possible to create custom metaclasses.  Most users
never need this tool, but when the need arises, metaclasses can provide
powerful, elegant solutions.  They have been used for logging attribute
access, adding thread-safety, tracking object creation, implementing
singletons, and many other tasks.

More information can be found in @ref{17bf,,Metaclasses}.
@anchor{glossary term-method}@anchor{4898}
@geindex method

@item method

A function which is defined inside a class body.  If called as an attribute
of an instance of that class, the method will get the instance object as
its first @ref{18d2,,argument} (which is usually called @code{self}).
See @ref{18d1,,function} and @ref{1a43,,nested scope}.
@anchor{glossary term-method-resolution-order}@anchor{1a60}
@geindex method resolution order

@item method resolution order

Method Resolution Order is the order in which base classes are searched
for a member during lookup. See The Python 2.3 Method Resolution Order@footnote{https://www.python.org/download/releases/2.3/mro/} for details of the
algorithm used by the Python interpreter since the 2.3 release.
@anchor{glossary term-module}@anchor{1839}
@geindex module

@item module

An object that serves as an organizational unit of Python code.  Modules
have a namespace containing arbitrary Python objects.  Modules are loaded
into Python by the process of @ref{183a,,importing}.

See also @ref{183d,,package}.
@anchor{glossary term-module-spec}@anchor{3b75}
@geindex module spec

@item module spec

A namespace containing the import-related information used to load a
module. An instance of @ref{1853,,importlib.machinery.ModuleSpec}.
@anchor{glossary term-MRO}@anchor{2cfa}
@geindex MRO

@item MRO

See @ref{1a60,,method resolution order}.
@anchor{glossary term-mutable}@anchor{14e9}
@geindex mutable

@item mutable

Mutable objects can change their value but keep their @ref{f80,,id()}.  See
also @ref{14e1,,immutable}.
@anchor{glossary term-named-tuple}@anchor{ce9}
@geindex named tuple

@item named tuple

The term “named tuple” applies to any type or class that inherits from
tuple and whose indexable elements are also accessible using named
attributes.  The type or class may have other features as well.

Several built-in types are named tuples, including the values returned
by @ref{1224,,time.localtime()} and @ref{516,,os.stat()}.  Another example is
@ref{1290,,sys.float_info}:

@example
>>> sys.float_info[1]                   # indexed access
1024
>>> sys.float_info.max_exp              # named field access
1024
>>> isinstance(sys.float_info, tuple)   # kind of tuple
True
@end example

Some named tuples are built-in types (such as the above examples).
Alternatively, a named tuple can be created from a regular class
definition that inherits from @ref{3e5,,tuple} and that defines named
fields.  Such a class can be written by hand or it can be created with
the factory function @ref{3e4,,collections.namedtuple()}.  The latter
technique also adds some extra methods that may not be found in
hand-written or built-in named tuples.
@anchor{glossary term-namespace}@anchor{3b95}
@geindex namespace

@item namespace

The place where a variable is stored.  Namespaces are implemented as
dictionaries.  There are the local, global and built-in namespaces as well
as nested namespaces in objects (in methods).  Namespaces support
modularity by preventing naming conflicts.  For instance, the functions
@ref{1c8,,builtins.open} and @ref{8ee,,os.open()} are distinguished by
their namespaces.  Namespaces also aid readability and maintainability by
making it clear which module implements a function.  For instance, writing
@ref{e22,,random.seed()} or @ref{672,,itertools.islice()} makes it clear that those
functions are implemented by the @ref{d7,,random} and @ref{9f,,itertools}
modules, respectively.
@anchor{glossary term-namespace-package}@anchor{1840}
@geindex namespace package

@item namespace package

A 
@geindex Python Enhancement Proposals; PEP 420
PEP 420@footnote{https://www.python.org/dev/peps/pep-0420} @ref{183d,,package} which serves only as a container for
subpackages.  Namespace packages may have no physical representation,
and specifically are not like a @ref{183f,,regular package} because they
have no @code{__init__.py} file.

See also @ref{1839,,module}.
@anchor{glossary term-nested-scope}@anchor{1a43}
@geindex nested scope

@item nested scope

The ability to refer to a variable in an enclosing definition.  For
instance, a function defined inside another function can refer to
variables in the outer function.  Note that nested scopes by default work
only for reference and not for assignment.  Local variables both read and
write in the innermost scope.  Likewise, global variables read and write
to the global namespace.  The @ref{e4d,,nonlocal} allows writing to outer
scopes.
@anchor{glossary term-new-style-class}@anchor{2140}
@geindex new-style class

@item new-style class

Old name for the flavor of classes now used for all class objects.  In
earlier Python versions, only new-style classes could use Python’s newer,
versatile features like @ref{1025,,__slots__}, descriptors,
properties, @code{__getattribute__()}, class methods, and static methods.
@anchor{glossary term-object}@anchor{48a9}
@geindex object

@item object

Any data with state (attributes or value) and defined behavior
(methods).  Also the ultimate base class of any @ref{2140,,new-style class}.
@anchor{glossary term-package}@anchor{183d}
@geindex package

@item package

A Python @ref{1839,,module} which can contain submodules or recursively,
subpackages.  Technically, a package is a Python module with an
@code{__path__} attribute.

See also @ref{183f,,regular package} and @ref{1840,,namespace package}.
@anchor{glossary term-parameter}@anchor{18e1}
@geindex parameter

@item parameter

A named entity in a @ref{18d1,,function} (or method) definition that
specifies an @ref{18d2,,argument} (or in some cases, arguments) that the
function can accept.  There are five kinds of parameter:


@itemize *

@item 
@emph{positional-or-keyword}: specifies an argument that can be passed
either @ref{18d2,,positionally} or as a @ref{18d2,,keyword argument}.  This is the default kind of parameter, for example @emph{foo}
and @emph{bar} in the following:

@example
def func(foo, bar=None): ...
@end example
@end itemize
@anchor{glossary positional-only-parameter}@anchor{59d}

@itemize *

@item 
@emph{positional-only}: specifies an argument that can be supplied only
by position. Positional-only parameters can be defined by including a
@code{/} character in the parameter list of the function definition after
them, for example @emph{posonly1} and @emph{posonly2} in the following:

@example
def func(posonly1, posonly2, /, positional_or_keyword): ...
@end example
@end itemize
@anchor{glossary keyword-only-parameter}@anchor{5a2}

@itemize *

@item 
@emph{keyword-only}: specifies an argument that can be supplied only
by keyword.  Keyword-only parameters can be defined by including a
single var-positional parameter or bare @code{*} in the parameter list
of the function definition before them, for example @emph{kw_only1} and
@emph{kw_only2} in the following:

@example
def func(arg, *, kw_only1, kw_only2): ...
@end example

@item 
@emph{var-positional}: specifies that an arbitrary sequence of
positional arguments can be provided (in addition to any positional
arguments already accepted by other parameters).  Such a parameter can
be defined by prepending the parameter name with @code{*}, for example
@emph{args} in the following:

@example
def func(*args, **kwargs): ...
@end example

@item 
@emph{var-keyword}: specifies that arbitrarily many keyword arguments
can be provided (in addition to any keyword arguments already accepted
by other parameters).  Such a parameter can be defined by prepending
the parameter name with @code{**}, for example @emph{kwargs} in the example
above.
@end itemize

Parameters can specify both optional and required arguments, as well as
default values for some optional arguments.

See also the @ref{18d2,,argument} glossary entry, the FAQ question on
@ref{47e6,,the difference between arguments and parameters}, the @ref{976,,inspect.Parameter} class, the
@ref{151c,,Function definitions} section, and 
@geindex Python Enhancement Proposals; PEP 362
PEP 362@footnote{https://www.python.org/dev/peps/pep-0362}.
@anchor{glossary term-path-entry}@anchor{1864}
@geindex path entry

@item path entry

A single location on the @ref{184a,,import path} which the @ref{184d,,path based finder} consults to find modules for importing.
@anchor{glossary term-path-entry-finder}@anchor{c06}
@geindex path entry finder

@item path entry finder

A @ref{1847,,finder} returned by a callable on @ref{bb5,,sys.path_hooks}
(i.e. a @ref{1866,,path entry hook}) which knows how to locate modules given
a @ref{1864,,path entry}.

See @ref{30f,,importlib.abc.PathEntryFinder} for the methods that path entry
finders implement.
@anchor{glossary term-path-entry-hook}@anchor{1866}
@geindex path entry hook

@item path entry hook

A callable on the @code{sys.path_hook} list which returns a @ref{c06,,path entry finder} if it knows how to find modules on a specific @ref{1864,,path entry}.
@anchor{glossary term-path-based-finder}@anchor{184d}
@geindex path based finder

@item path based finder

One of the default @ref{c05,,meta path finders} which
searches an @ref{184a,,import path} for modules.
@anchor{glossary term-path-like-object}@anchor{63d}
@geindex path-like object

@item path-like object

An object representing a file system path. A path-like object is either
a @ref{2ef,,str} or @ref{2f0,,bytes} object representing a path, or an object
implementing the @ref{795,,os.PathLike} protocol. An object that supports
the @ref{795,,os.PathLike} protocol can be converted to a @ref{2ef,,str} or
@ref{2f0,,bytes} file system path by calling the @ref{797,,os.fspath()} function;
@ref{798,,os.fsdecode()} and @ref{799,,os.fsencode()} can be used to guarantee a
@ref{2ef,,str} or @ref{2f0,,bytes} result instead, respectively. Introduced
by 
@geindex Python Enhancement Proposals; PEP 519
PEP 519@footnote{https://www.python.org/dev/peps/pep-0519}.
@anchor{glossary term-PEP}@anchor{48aa}
@geindex PEP

@item PEP

Python Enhancement Proposal. A PEP is a design document
providing information to the Python community, or describing a new
feature for Python or its processes or environment. PEPs should
provide a concise technical specification and a rationale for proposed
features.

PEPs are intended to be the primary mechanisms for proposing major new
features, for collecting community input on an issue, and for documenting
the design decisions that have gone into Python. The PEP author is
responsible for building consensus within the community and documenting
dissenting opinions.

See 
@geindex Python Enhancement Proposals; PEP 1
PEP 1@footnote{https://www.python.org/dev/peps/pep-0001}.
@anchor{glossary term-portion}@anchor{1842}
@geindex portion

@item portion

A set of files in a single directory (possibly stored in a zip file)
that contribute to a namespace package, as defined in 
@geindex Python Enhancement Proposals; PEP 420
PEP 420@footnote{https://www.python.org/dev/peps/pep-0420}.
@anchor{glossary term-positional-argument}@anchor{860}
@geindex positional argument

@item positional argument

See @ref{18d2,,argument}.
@anchor{glossary term-provisional-API}@anchor{621}
@geindex provisional API

@item provisional API

A provisional API is one which has been deliberately excluded from
the standard library’s backwards compatibility guarantees.  While major
changes to such interfaces are not expected, as long as they are marked
provisional, backwards incompatible changes (up to and including removal
of the interface) may occur if deemed necessary by core developers.  Such
changes will not be made gratuitously – they will occur only if serious
fundamental flaws are uncovered that were missed prior to the inclusion
of the API.

Even for provisional APIs, backwards incompatible changes are seen as
a “solution of last resort” - every attempt will still be made to find
a backwards compatible resolution to any identified problems.

This process allows the standard library to continue to evolve over
time, without locking in problematic design errors for extended periods
of time.  See 
@geindex Python Enhancement Proposals; PEP 411
PEP 411@footnote{https://www.python.org/dev/peps/pep-0411} for more details.
@anchor{glossary term-provisional-package}@anchor{bd6}
@geindex provisional package

@item provisional package

See @ref{621,,provisional API}.
@anchor{glossary term-Python-3000}@anchor{48ab}
@geindex Python 3000

@item Python 3000

Nickname for the Python 3.x release line (coined long ago when the
release of version 3 was something in the distant future.)  This is also
abbreviated “Py3k”.
@anchor{glossary term-Pythonic}@anchor{48ac}
@geindex Pythonic

@item Pythonic

An idea or piece of code which closely follows the most common idioms
of the Python language, rather than implementing code using concepts
common to other languages.  For example, a common idiom in Python is
to loop over all elements of an iterable using a @ref{161,,for}
statement.  Many other languages don’t have this type of construct, so
people unfamiliar with Python sometimes use a numerical counter instead:

@example
for i in range(len(food)):
    print(food[i])
@end example

As opposed to the cleaner, Pythonic method:

@example
for piece in food:
    print(piece)
@end example
@anchor{glossary term-qualified-name}@anchor{11bf}
@geindex qualified name

@item qualified name

A dotted name showing the “path” from a module’s global scope to a
class, function or method defined in that module, as defined in
@geindex Python Enhancement Proposals; PEP 3155
PEP 3155@footnote{https://www.python.org/dev/peps/pep-3155}.  For top-level functions and classes, the qualified name
is the same as the object’s name:

@example
>>> class C:
...     class D:
...         def meth(self):
...             pass
...
>>> C.__qualname__
'C'
>>> C.D.__qualname__
'C.D'
>>> C.D.meth.__qualname__
'C.D.meth'
@end example

When used to refer to modules, the @emph{fully qualified name} means the
entire dotted path to the module, including any parent packages,
e.g. @code{email.mime.text}:

@example
>>> import email.mime.text
>>> email.mime.text.__name__
'email.mime.text'
@end example
@anchor{glossary term-reference-count}@anchor{40a3}
@geindex reference count

@item reference count

The number of references to an object.  When the reference count of an
object drops to zero, it is deallocated.  Reference counting is
generally not visible to Python code, but it is a key element of the
@ref{17a4,,CPython} implementation.  The @ref{f7,,sys} module defines a
@ref{3b6f,,getrefcount()} function that programmers can call to return the
reference count for a particular object.
@anchor{glossary term-regular-package}@anchor{183f}
@geindex regular package

@item regular package

A traditional @ref{183d,,package}, such as a directory containing an
@code{__init__.py} file.

See also @ref{1840,,namespace package}.
@anchor{glossary term-__slots__}@anchor{3cd2}
@geindex __slots__

@item __slots__

A declaration inside a class that saves memory by pre-declaring space for
instance attributes and eliminating instance dictionaries.  Though
popular, the technique is somewhat tricky to get right and is best
reserved for rare cases where there are large numbers of instances in a
memory-critical application.
@anchor{glossary term-sequence}@anchor{14e7}
@geindex sequence

@item sequence

An @ref{dcc,,iterable} which supports efficient element access using integer
indices via the @code{__getitem__()} special method and defines a
@code{__len__()} method that returns the length of the sequence.
Some built-in sequence types are @ref{43a,,list}, @ref{2ef,,str},
@ref{3e5,,tuple}, and @ref{2f0,,bytes}. Note that @ref{3b0,,dict} also
supports @code{__getitem__()} and @code{__len__()}, but is considered a
mapping rather than a sequence because the lookups use arbitrary
@ref{14e1,,immutable} keys rather than integers.

The @ref{110c,,collections.abc.Sequence} abstract base class
defines a much richer interface that goes beyond just
@code{__getitem__()} and @code{__len__()}, adding @code{count()},
@code{index()}, @code{__contains__()}, and
@code{__reversed__()}. Types that implement this expanded
interface can be registered explicitly using
@ref{c21,,register()}.
@anchor{glossary term-set-comprehension}@anchor{48ad}
@geindex set comprehension

@item set comprehension

A compact way to process all or part of the elements in an iterable and
return a set with the results. @code{results = @{c for c in 'abracadabra' if
c not in 'abc'@}} generates the set of strings @code{@{'r', 'd'@}}.  See
@ref{1887,,Displays for lists@comma{} sets and dictionaries}.
@anchor{glossary term-single-dispatch}@anchor{4f7}
@geindex single dispatch

@item single dispatch

A form of @ref{4f6,,generic function} dispatch where the implementation is
chosen based on the type of a single argument.
@anchor{glossary term-slice}@anchor{1a5d}
@geindex slice

@item slice

An object usually containing a portion of a @ref{14e7,,sequence}.  A slice is
created using the subscript notation, @code{[]} with colons between numbers
when several are given, such as in @code{variable_name[1:3:5]}.  The bracket
(subscript) notation uses @ref{ffb,,slice} objects internally.
@anchor{glossary term-special-method}@anchor{111a}
@geindex special method

@item special method

@geindex special; method

A method that is called implicitly by Python to execute a certain
operation on a type, such as addition.  Such methods have names starting
and ending with double underscores.  Special methods are documented in
@ref{7b4,,Special method names}.
@anchor{glossary term-statement}@anchor{2034}
@geindex statement

@item statement

A statement is part of a suite (a “block” of code).  A statement is either
an @ref{17d2,,expression} or one of several constructs with a keyword, such
as @ref{2fb,,if}, @ref{14ec,,while} or @ref{161,,for}.
@anchor{glossary term-strong-reference}@anchor{1fc}
@geindex strong reference

@item strong reference

In Python’s C API, a strong reference is a reference to an object
which increments the object’s reference count when it is created and
decrements the object’s reference count when it is deleted.

The @ref{36a,,Py_NewRef()} function can be used to create a strong reference
to an object. Usually, the @ref{373,,Py_DECREF()} function must be called on
the strong reference before exiting the scope of the strong reference, to
avoid leaking one reference.

See also @ref{4112,,borrowed reference}.
@anchor{glossary term-text-encoding}@anchor{1a58}
@geindex text encoding

@item text encoding

A codec which encodes Unicode strings to bytes.
@anchor{glossary term-text-file}@anchor{1566}
@geindex text file

@item text file

A @ref{d68,,file object} able to read and write @ref{2ef,,str} objects.
Often, a text file actually accesses a byte-oriented datastream
and handles the @ref{1a58,,text encoding} automatically.
Examples of text files are files opened in text mode (@code{'r'} or @code{'w'}),
@ref{5ee,,sys.stdin}, @ref{5ef,,sys.stdout}, and instances of
@ref{aa1,,io.StringIO}.

See also @ref{2138,,binary file} for a file object able to read and write
@ref{87e,,bytes-like objects}.
@anchor{glossary term-triple-quoted-string}@anchor{48ae}
@geindex triple-quoted string

@item triple-quoted string

A string which is bound by three instances of either a quotation mark
(”) or an apostrophe (‘).  While they don’t provide any functionality
not available with single-quoted strings, they are useful for a number
of reasons.  They allow you to include unescaped single and double
quotes within a string and they can span multiple lines without the
use of the continuation character, making them especially useful when
writing docstrings.
@anchor{glossary term-type}@anchor{11af}
@geindex type

@item type

The type of a Python object determines what kind of object it is; every
object has a type.  An object’s type is accessible as its
@ref{fff,,__class__} attribute or can be retrieved with
@code{type(obj)}.
@anchor{glossary term-type-alias}@anchor{48af}
@geindex type alias

@item type alias

A synonym for a type, created by assigning the type to an identifier.

Type aliases are useful for simplifying @ref{17ce,,type hints}.
For example:

@example
def remove_gray_shades(
        colors: list[tuple[int, int, int]]) -> list[tuple[int, int, int]]:
    pass
@end example

could be made more readable like this:

@example
Color = tuple[int, int, int]

def remove_gray_shades(colors: list[Color]) -> list[Color]:
    pass
@end example

See @ref{120,,typing} and 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484}, which describe this functionality.
@anchor{glossary term-type-hint}@anchor{17ce}
@geindex type hint

@item type hint

An @ref{17ca,,annotation} that specifies the expected type for a variable, a class
attribute, or a function parameter or return value.

Type hints are optional and are not enforced by Python but
they are useful to static type analysis tools, and aid IDEs with code
completion and refactoring.

Type hints of global variables, class attributes, and functions,
but not local variables, can be accessed using
@ref{432,,typing.get_type_hints()}.

See @ref{120,,typing} and 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484}, which describe this functionality.
@anchor{glossary term-universal-newlines}@anchor{88a}
@geindex universal newlines

@item universal newlines

A manner of interpreting text streams in which all of the following are
recognized as ending a line: the Unix end-of-line convention @code{'\n'},
the Windows convention @code{'\r\n'}, and the old Macintosh convention
@code{'\r'}.  See 
@geindex Python Enhancement Proposals; PEP 278
PEP 278@footnote{https://www.python.org/dev/peps/pep-0278} and 
@geindex Python Enhancement Proposals; PEP 3116
PEP 3116@footnote{https://www.python.org/dev/peps/pep-3116}, as well as
@ref{1afd,,bytes.splitlines()} for an additional use.
@anchor{glossary term-variable-annotation}@anchor{8af}
@geindex variable annotation

@item variable annotation

An @ref{17ca,,annotation} of a variable or a class attribute.

When annotating a variable or a class attribute, assignment is optional:

@example
class C:
    field: 'annotation'
@end example

Variable annotations are usually used for
@ref{17ce,,type hints}: for example this variable is expected to take
@ref{1b1,,int} values:

@example
count: int = 0
@end example

Variable annotation syntax is explained in section @ref{1945,,Annotated assignment statements}.

See @ref{151d,,function annotation}, 
@geindex Python Enhancement Proposals; PEP 484
PEP 484@footnote{https://www.python.org/dev/peps/pep-0484}
and 
@geindex Python Enhancement Proposals; PEP 526
PEP 526@footnote{https://www.python.org/dev/peps/pep-0526}, which describe this functionality.
Also see @ref{255,,Annotations Best Practices}
for best practices on working with annotations.
@anchor{glossary term-virtual-environment}@anchor{15d5}
@geindex virtual environment

@item virtual environment

A cooperatively isolated runtime environment that allows Python users
and applications to install and upgrade Python distribution packages
without interfering with the behaviour of other Python applications
running on the same system.

See also @ref{12c,,venv}.
@anchor{glossary term-virtual-machine}@anchor{2bf5}
@geindex virtual machine

@item virtual machine

A computer defined entirely in software.  Python’s virtual machine
executes the @ref{8a8,,bytecode} emitted by the bytecode compiler.
@anchor{glossary term-Zen-of-Python}@anchor{48b0}
@geindex Zen of Python

@item Zen of Python

Listing of Python design principles and philosophies that are helpful in
understanding and using the language.  The listing can be found by typing
“@code{import this}” at the interactive prompt.
@end table

@node About these documents,Dealing with Bugs,Glossary,Top
@anchor{about doc}@anchor{48b1}@anchor{about about-these-documents}@anchor{48b2}
@chapter About these documents


These documents are generated from reStructuredText@footnote{http://docutils.sourceforge.net/rst.html} sources by Sphinx@footnote{http://sphinx-doc.org/}, a
document processor specifically written for the Python documentation.

@c In the online version of these documents, you can submit comments and suggest
@c changes directly on the documentation pages.

Development of the documentation and its toolchain is an entirely volunteer
effort, just like Python itself.  If you want to contribute, please take a
look at the @ref{40fa,,Dealing with Bugs} page for information on how to do so.  New
volunteers are always welcome!

Many thanks go to:


@itemize *

@item 
Fred L. Drake, Jr., the creator of the original Python documentation toolset
and writer of much of the content;

@item 
the Docutils@footnote{http://docutils.sourceforge.net/} project for creating
reStructuredText and the Docutils suite;

@item 
Fredrik Lundh for his Alternative Python Reference project from which Sphinx
got many good ideas.
@end itemize

@menu
* Contributors to the Python Documentation:: 

@end menu

@node Contributors to the Python Documentation,,,About these documents
@anchor{about contributors-to-the-python-documentation}@anchor{48b3}
@section Contributors to the Python Documentation


Many people have contributed to the Python language, the Python standard
library, and the Python documentation.  See Misc/ACKS@footnote{https://github.com/python/cpython/tree/main/Misc/ACKS} in the Python
source distribution for a partial list of contributors.

It is only with the input and contributions of the Python community
that Python has such wonderful documentation – Thank You!

@node Dealing with Bugs,Copyright,About these documents,Top
@anchor{bugs doc}@anchor{48b4}@anchor{bugs dealing-with-bugs}@anchor{48b5}@anchor{bugs reporting-bugs}@anchor{40fa}
@chapter Dealing with Bugs


Python is a mature programming language which has established a reputation for
stability.  In order to maintain this reputation, the developers would like to
know of any deficiencies you find in Python.

It can be sometimes faster to fix bugs yourself and contribute patches to
Python as it streamlines the process and involves less people. Learn how to
@ref{48b6,,contribute}.

@menu
* Documentation bugs:: 
* Using the Python issue tracker:: 
* Getting started contributing to Python yourself:: 

@end menu

@node Documentation bugs,Using the Python issue tracker,,Dealing with Bugs
@anchor{bugs documentation-bugs}@anchor{48b7}
@section Documentation bugs


If you find a bug in this documentation or would like to propose an improvement,
please submit a bug report on the @ref{48b8,,tracker}.  If you
have a suggestion on how to fix it, include that as well.

If you’re short on time, you can also email documentation bug reports to
@email{docs@@python.org} (behavioral bugs can be sent to @email{python-list@@python.org}).
‘docs@@’ is a mailing list run by volunteers; your request will be noticed,
though it may take a while to be processed.


@subsubheading See also



@table @asis

@item Documentation bugs@footnote{https://bugs.python.org/issue?@@filter=status&@@filter=components&components=4&status=1&@@columns=id@comma{}activity@comma{}title@comma{}status&@@sort=-activity}

A list of documentation bugs that have been submitted to the Python issue tracker.

@item Issue Tracking@footnote{https://devguide.python.org/tracker/}

Overview of the process involved in reporting an improvement on the tracker.

@item Helping with Documentation@footnote{https://devguide.python.org/docquality/#helping-with-documentation}

Comprehensive guide for individuals that are interested in contributing to Python documentation.
@end table


@node Using the Python issue tracker,Getting started contributing to Python yourself,Documentation bugs,Dealing with Bugs
@anchor{bugs using-the-python-issue-tracker}@anchor{48b9}@anchor{bugs using-the-tracker}@anchor{48b8}
@section Using the Python issue tracker


Bug reports for Python itself should be submitted via the Python Bug Tracker
(@indicateurl{https://bugs.python.org/}).  The bug tracker offers a web form which allows
pertinent information to be entered and submitted to the developers.

The first step in filing a report is to determine whether the problem has
already been reported.  The advantage in doing so, aside from saving the
developers time, is that you learn what has been done to fix it; it may be that
the problem has already been fixed for the next release, or additional
information is needed (in which case you are welcome to provide it if you can!).
To do this, search the bug database using the search box on the top of the page.

If the problem you’re reporting is not already in the bug tracker, go back to
the Python Bug Tracker and log in.  If you don’t already have a tracker account,
select the “Register” link or, if you use OpenID, one of the OpenID provider
logos in the sidebar.  It is not possible to submit a bug report anonymously.

Being now logged in, you can submit a bug.  Select the “Create New” link in the
sidebar to open the bug reporting form.

The submission form has a number of fields.  For the “Title” field, enter a
@emph{very} short description of the problem; less than ten words is good.  In the
“Type” field, select the type of your problem; also select the “Component” and
“Versions” to which the bug relates.

In the “Comment” field, describe the problem in detail, including what you
expected to happen and what did happen.  Be sure to include whether any
extension modules were involved, and what hardware and software platform you
were using (including version information as appropriate).

Each bug report will be assigned to a developer who will determine what needs to
be done to correct the problem.  You will receive an update each time action is
taken on the bug.


@subsubheading See also



@table @asis

@item How to Report Bugs Effectively@footnote{https://www.chiark.greenend.org.uk/~sgtatham/bugs.html}

Article which goes into some detail about how to create a useful bug report.
This describes what kind of information is useful and why it is useful.

@item Bug Writing Guidelines@footnote{https://bugzilla.mozilla.org/page.cgi?id=bug-writing.html}

Information about writing a good bug report.  Some of this is specific to the
Mozilla project, but describes general good practices.
@end table


@node Getting started contributing to Python yourself,,Using the Python issue tracker,Dealing with Bugs
@anchor{bugs contributing-to-python}@anchor{48b6}@anchor{bugs getting-started-contributing-to-python-yourself}@anchor{48ba}
@section Getting started contributing to Python yourself


Beyond just reporting bugs that you find, you are also welcome to submit
patches to fix them.  You can find more information on how to get started
patching Python in the Python Developer's Guide@footnote{https://devguide.python.org/}.  If you have questions,
the core-mentorship mailing list@footnote{https://mail.python.org/mailman3/lists/core-mentorship.python.org/} is a friendly place to get answers to
any and all questions pertaining to the process of fixing issues in Python.

@node Copyright,History and License,Dealing with Bugs,Top
@anchor{copyright doc}@anchor{48bb}@anchor{copyright copyright}@anchor{48bc}@anchor{copyright core-mentorship-mailing-list}@anchor{48bd}
@chapter Copyright


Python and this documentation is:

Copyright © 2001-2022 Python Software Foundation. All rights reserved.

Copyright © 2000 BeOpen.com. All rights reserved.

Copyright © 1995-2000 Corporation for National Research Initiatives. All rights
reserved.

Copyright © 1991-1995 Stichting Mathematisch Centrum. All rights reserved.


__________________________________________________________________


See @ref{48be,,History and License} for complete license and permissions information.

@node History and License,Distributing Python Modules Legacy version,Copyright,Top
@anchor{license doc}@anchor{48bf}@anchor{license history-and-license}@anchor{48be}@anchor{license id1}@anchor{48c0}
@chapter History and License


@menu
* History of the software:: 
* Terms and conditions for accessing or otherwise using Python:: 
* Licenses and Acknowledgements for Incorporated Software:: 

@end menu

@node History of the software,Terms and conditions for accessing or otherwise using Python,,History and License
@anchor{license history-of-the-software}@anchor{48c1}
@section History of the software


Python was created in the early 1990s by Guido van Rossum at Stichting
Mathematisch Centrum (CWI, see @indicateurl{https://www.cwi.nl/}) in the Netherlands as a
successor of a language called ABC.  Guido remains Python’s principal author,
although it includes many contributions from others.

In 1995, Guido continued his work on Python at the Corporation for National
Research Initiatives (CNRI, see @indicateurl{https://www.cnri.reston.va.us/}) in Reston,
Virginia where he released several versions of the software.

In May 2000, Guido and the Python core development team moved to BeOpen.com to
form the BeOpen PythonLabs team.  In October of the same year, the PythonLabs
team moved to Digital Creations (now Zope Corporation; see
@indicateurl{https://www.zope.org/}).  In 2001, the Python Software Foundation (PSF, see
@indicateurl{https://www.python.org/psf/}) was formed, a non-profit organization created
specifically to own Python-related Intellectual Property.  Zope Corporation is a
sponsoring member of the PSF.

All Python releases are Open Source (see @indicateurl{https://opensource.org/} for the Open
Source Definition). Historically, most, but not all, Python releases have also
been GPL-compatible; the table below summarizes the various releases.


@multitable {xxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxx} {xxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxx} 
@headitem

Release

@tab

Derived from

@tab

Year

@tab

Owner

@tab

GPL compatible?

@item

0.9.0 thru 1.2

@tab

n/a

@tab

1991-1995

@tab

CWI

@tab

yes

@item

1.3 thru 1.5.2

@tab

1.2

@tab

1995-1999

@tab

CNRI

@tab

yes

@item

1.6

@tab

1.5.2

@tab

2000

@tab

CNRI

@tab

no

@item

2.0

@tab

1.6

@tab

2000

@tab

BeOpen.com

@tab

no

@item

1.6.1

@tab

1.6

@tab

2001

@tab

CNRI

@tab

no

@item

2.1

@tab

2.0+1.6.1

@tab

2001

@tab

PSF

@tab

no

@item

2.0.1

@tab

2.0+1.6.1

@tab

2001

@tab

PSF

@tab

yes

@item

2.1.1

@tab

2.1+2.0.1

@tab

2001

@tab

PSF

@tab

yes

@item

2.1.2

@tab

2.1.1

@tab

2002

@tab

PSF

@tab

yes

@item

2.1.3

@tab

2.1.2

@tab

2002

@tab

PSF

@tab

yes

@item

2.2 and above

@tab

2.1.1

@tab

2001-now

@tab

PSF

@tab

yes

@end multitable


@cartouche
@quotation Note 
GPL-compatible doesn’t mean that we’re distributing Python under the GPL.  All
Python licenses, unlike the GPL, let you distribute a modified version without
making your changes open source. The GPL-compatible licenses make it possible to
combine Python with other software that is released under the GPL; the others
don’t.
@end quotation
@end cartouche

Thanks to the many outside volunteers who have worked under Guido’s direction to
make these releases possible.

@node Terms and conditions for accessing or otherwise using Python,Licenses and Acknowledgements for Incorporated Software,History of the software,History and License
@anchor{license terms-and-conditions-for-accessing-or-otherwise-using-python}@anchor{48c2}
@section Terms and conditions for accessing or otherwise using Python


Python software and documentation are licensed under the
@ref{48c3,,PSF License Agreement}.

Starting with Python 3.8.6, examples, recipes, and other code in
the documentation are dual licensed under the PSF License Agreement
and the @ref{48c4,,Zero-Clause BSD license}.

Some software incorporated into Python is under different licenses.
The licenses are listed with code falling under that license.
See @ref{48c5,,Licenses and Acknowledgements for Incorporated Software} for an incomplete list of these licenses.

@menu
* PSF LICENSE AGREEMENT FOR PYTHON 3.11.0a5: PSF LICENSE AGREEMENT FOR PYTHON 3 11 0a5. 
* BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0: BEOPEN COM LICENSE AGREEMENT FOR PYTHON 2 0. 
* CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1: CNRI LICENSE AGREEMENT FOR PYTHON 1 6 1. 
* CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2: CWI LICENSE AGREEMENT FOR PYTHON 0 9 0 THROUGH 1 2. 
* ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON 3.11.0a5 DOCUMENTATION: ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON 3 11 0a5 DOCUMENTATION. 

@end menu

@node PSF LICENSE AGREEMENT FOR PYTHON 3 11 0a5,BEOPEN COM LICENSE AGREEMENT FOR PYTHON 2 0,,Terms and conditions for accessing or otherwise using Python
@anchor{license psf-license}@anchor{48c3}@anchor{license psf-license-agreement-for-python-release}@anchor{48c6}
@subsection PSF LICENSE AGREEMENT FOR PYTHON 3.11.0a5


@example
1. This LICENSE AGREEMENT is between the Python Software Foundation ("PSF"), and
   the Individual or Organization ("Licensee") accessing and otherwise using Python
   3.11.0a5 software in source or binary form and its associated documentation.

2. Subject to the terms and conditions of this License Agreement, PSF hereby
   grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce,
   analyze, test, perform and/or display publicly, prepare derivative works,
   distribute, and otherwise use Python 3.11.0a5 alone or in any derivative
   version, provided, however, that PSF's License Agreement and PSF's notice of
   copyright, i.e., "Copyright © 2001-2022 Python Software Foundation; All Rights
   Reserved" are retained in Python 3.11.0a5 alone or in any derivative version
   prepared by Licensee.

3. In the event Licensee prepares a derivative work that is based on or
   incorporates Python 3.11.0a5 or any part thereof, and wants to make the
   derivative work available to others as provided herein, then Licensee hereby
   agrees to include in any such work a brief summary of the changes made to Python
   3.11.0a5.

4. PSF is making Python 3.11.0a5 available to Licensee on an "AS IS" basis.
   PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED.  BY WAY OF
   EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND DISCLAIMS ANY REPRESENTATION OR
   WARRANTY OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE
   USE OF PYTHON 3.11.0a5 WILL NOT INFRINGE ANY THIRD PARTY RIGHTS.

5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON 3.11.0a5
   FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS A RESULT OF
   MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 3.11.0a5, OR ANY DERIVATIVE
   THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.

6. This License Agreement will automatically terminate upon a material breach of
   its terms and conditions.

7. Nothing in this License Agreement shall be deemed to create any relationship
   of agency, partnership, or joint venture between PSF and Licensee.  This License
   Agreement does not grant permission to use PSF trademarks or trade name in a
   trademark sense to endorse or promote products or services of Licensee, or any
   third party.

8. By copying, installing or otherwise using Python 3.11.0a5, Licensee agrees
   to be bound by the terms and conditions of this License Agreement.
@end example

@node BEOPEN COM LICENSE AGREEMENT FOR PYTHON 2 0,CNRI LICENSE AGREEMENT FOR PYTHON 1 6 1,PSF LICENSE AGREEMENT FOR PYTHON 3 11 0a5,Terms and conditions for accessing or otherwise using Python
@anchor{license beopen-com-license-agreement-for-python-2-0}@anchor{48c7}
@subsection BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0


BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1

@example
1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an office at
   160 Saratoga Avenue, Santa Clara, CA 95051, and the Individual or Organization
   ("Licensee") accessing and otherwise using this software in source or binary
   form and its associated documentation ("the Software").

2. Subject to the terms and conditions of this BeOpen Python License Agreement,
   BeOpen hereby grants Licensee a non-exclusive, royalty-free, world-wide license
   to reproduce, analyze, test, perform and/or display publicly, prepare derivative
   works, distribute, and otherwise use the Software alone or in any derivative
   version, provided, however, that the BeOpen Python License is retained in the
   Software, alone or in any derivative version prepared by Licensee.

3. BeOpen is making the Software available to Licensee on an "AS IS" basis.
   BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED.  BY WAY OF
   EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND DISCLAIMS ANY REPRESENTATION OR
   WARRANTY OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE
   USE OF THE SOFTWARE WILL NOT INFRINGE ANY THIRD PARTY RIGHTS.

4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE SOFTWARE FOR
   ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS A RESULT OF USING,
   MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY DERIVATIVE THEREOF, EVEN IF
   ADVISED OF THE POSSIBILITY THEREOF.

5. This License Agreement will automatically terminate upon a material breach of
   its terms and conditions.

6. This License Agreement shall be governed by and interpreted in all respects
   by the law of the State of California, excluding conflict of law provisions.
   Nothing in this License Agreement shall be deemed to create any relationship of
   agency, partnership, or joint venture between BeOpen and Licensee.  This License
   Agreement does not grant permission to use BeOpen trademarks or trade names in a
   trademark sense to endorse or promote products or services of Licensee, or any
   third party.  As an exception, the "BeOpen Python" logos available at
   @indicateurl{http://www.pythonlabs.com/logos.html} may be used according to the permissions
   granted on that web page.

7. By copying, installing or otherwise using the software, Licensee agrees to be
   bound by the terms and conditions of this License Agreement.
@end example

@node CNRI LICENSE AGREEMENT FOR PYTHON 1 6 1,CWI LICENSE AGREEMENT FOR PYTHON 0 9 0 THROUGH 1 2,BEOPEN COM LICENSE AGREEMENT FOR PYTHON 2 0,Terms and conditions for accessing or otherwise using Python
@anchor{license cnri-license-agreement-for-python-1-6-1}@anchor{48c8}
@subsection CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1


@example
1. This LICENSE AGREEMENT is between the Corporation for National Research
   Initiatives, having an office at 1895 Preston White Drive, Reston, VA 20191
   ("CNRI"), and the Individual or Organization ("Licensee") accessing and
   otherwise using Python 1.6.1 software in source or binary form and its
   associated documentation.

2. Subject to the terms and conditions of this License Agreement, CNRI hereby
   grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce,
   analyze, test, perform and/or display publicly, prepare derivative works,
   distribute, and otherwise use Python 1.6.1 alone or in any derivative version,
   provided, however, that CNRI's License Agreement and CNRI's notice of copyright,
   i.e., "Copyright © 1995-2001 Corporation for National Research Initiatives; All
   Rights Reserved" are retained in Python 1.6.1 alone or in any derivative version
   prepared by Licensee.  Alternately, in lieu of CNRI's License Agreement,
   Licensee may substitute the following text (omitting the quotes): "Python 1.6.1
   is made available subject to the terms and conditions in CNRI's License
   Agreement.  This Agreement together with Python 1.6.1 may be located on the
   internet using the following unique, persistent identifier (known as a handle):
   1895.22/1013.  This Agreement may also be obtained from a proxy server on the
   internet using the following URL: @indicateurl{http://hdl.handle.net/1895.22/1013}."

3. In the event Licensee prepares a derivative work that is based on or
   incorporates Python 1.6.1 or any part thereof, and wants to make the derivative
   work available to others as provided herein, then Licensee hereby agrees to
   include in any such work a brief summary of the changes made to Python 1.6.1.

4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" basis.  CNRI
   MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED.  BY WAY OF EXAMPLE,
   BUT NOT LIMITATION, CNRI MAKES NO AND DISCLAIMS ANY REPRESENTATION OR WARRANTY
   OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF
   PYTHON 1.6.1 WILL NOT INFRINGE ANY THIRD PARTY RIGHTS.

5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON 1.6.1 FOR
   ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS A RESULT OF
   MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, OR ANY DERIVATIVE
   THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.

6. This License Agreement will automatically terminate upon a material breach of
   its terms and conditions.

7. This License Agreement shall be governed by the federal intellectual property
   law of the United States, including without limitation the federal copyright
   law, and, to the extent such U.S. federal law does not apply, by the law of the
   Commonwealth of Virginia, excluding Virginia's conflict of law provisions.
   Notwithstanding the foregoing, with regard to derivative works based on Python
   1.6.1 that incorporate non-separable material that was previously distributed
   under the GNU General Public License (GPL), the law of the Commonwealth of
   Virginia shall govern this License Agreement only as to issues arising under or
   with respect to Paragraphs 4, 5, and 7 of this License Agreement.  Nothing in
   this License Agreement shall be deemed to create any relationship of agency,
   partnership, or joint venture between CNRI and Licensee.  This License Agreement
   does not grant permission to use CNRI trademarks or trade name in a trademark
   sense to endorse or promote products or services of Licensee, or any third
   party.

8. By clicking on the "ACCEPT" button where indicated, or by copying, installing
   or otherwise using Python 1.6.1, Licensee agrees to be bound by the terms and
   conditions of this License Agreement.
@end example

@node CWI LICENSE AGREEMENT FOR PYTHON 0 9 0 THROUGH 1 2,ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON 3 11 0a5 DOCUMENTATION,CNRI LICENSE AGREEMENT FOR PYTHON 1 6 1,Terms and conditions for accessing or otherwise using Python
@anchor{license cwi-license-agreement-for-python-0-9-0-through-1-2}@anchor{48c9}
@subsection CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2


@example
Copyright © 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, The
Netherlands.  All rights reserved.

Permission to use, copy, modify, and distribute this software and its
documentation for any purpose and without fee is hereby granted, provided that
the above copyright notice appear in all copies and that both that copyright
notice and this permission notice appear in supporting documentation, and that
the name of Stichting Mathematisch Centrum or CWI not be used in advertising or
publicity pertaining to distribution of the software without specific, written
prior permission.

STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS
SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO
EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE FOR ANY SPECIAL, INDIRECT
OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE,
DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS
SOFTWARE.
@end example

@node ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON 3 11 0a5 DOCUMENTATION,,CWI LICENSE AGREEMENT FOR PYTHON 0 9 0 THROUGH 1 2,Terms and conditions for accessing or otherwise using Python
@anchor{license bsd0}@anchor{48c4}@anchor{license zero-clause-bsd-license-for-code-in-the-python-release-documentation}@anchor{48ca}
@subsection ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON 3.11.0a5 DOCUMENTATION


@example
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
@end example

@node Licenses and Acknowledgements for Incorporated Software,,Terms and conditions for accessing or otherwise using Python,History and License
@anchor{license licenses-and-acknowledgements-for-incorporated-software}@anchor{48cb}@anchor{license otherlicenses}@anchor{48c5}
@section Licenses and Acknowledgements for Incorporated Software


This section is an incomplete, but growing list of licenses and acknowledgements
for third-party software incorporated in the Python distribution.

@menu
* Mersenne Twister:: 
* Sockets: Sockets<2>. 
* Asynchronous socket services:: 
* Cookie management:: 
* Execution tracing:: 
* UUencode and UUdecode functions:: 
* XML Remote Procedure Calls:: 
* test_epoll:: 
* Select kqueue:: 
* SipHash24:: 
* strtod and dtoa:: 
* OpenSSL:: 
* expat:: 
* libffi:: 
* zlib: zlib<3>. 
* cfuhash:: 
* libmpdec:: 
* W3C C14N test suite:: 

@end menu

@node Mersenne Twister,Sockets<2>,,Licenses and Acknowledgements for Incorporated Software
@anchor{license mersenne-twister}@anchor{48cc}
@subsection Mersenne Twister


The @code{_random} module includes code based on a download from
@indicateurl{http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html}. The following are
the verbatim comments from the original code:

@example
A C-program for MT19937, with initialization improved 2002/1/26.
Coded by Takuji Nishimura and Makoto Matsumoto.

Before using, initialize the state by using init_genrand(seed)
or init_by_array(init_key, key_length).

Copyright (C) 1997 - 2002, Makoto Matsumoto and Takuji Nishimura,
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:

 1. Redistributions of source code must retain the above copyright
    notice, this list of conditions and the following disclaimer.

 2. Redistributions in binary form must reproduce the above copyright
    notice, this list of conditions and the following disclaimer in the
    documentation and/or other materials provided with the distribution.

 3. The names of its contributors may not be used to endorse or promote
    products derived from this software without specific prior written
    permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


Any feedback is very welcome.
http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html
email: m-mat @@ math.sci.hiroshima-u.ac.jp (remove space)
@end example

@node Sockets<2>,Asynchronous socket services,Mersenne Twister,Licenses and Acknowledgements for Incorporated Software
@anchor{license sockets}@anchor{48cd}
@subsection Sockets


The @ref{ea,,socket} module uses the functions, @code{getaddrinfo()}, and
@code{getnameinfo()}, which are coded in separate source files from the WIDE
Project, @indicateurl{http://www.wide.ad.jp/}.

@example
Copyright (C) 1995, 1996, 1997, and 1998 WIDE Project.
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.
3. Neither the name of the project nor the names of its contributors
   may be used to endorse or promote products derived from this software
   without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS `@w{`}AS IS'@w{'} AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED.  IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
@end example

@node Asynchronous socket services,Cookie management,Sockets<2>,Licenses and Acknowledgements for Incorporated Software
@anchor{license asynchronous-socket-services}@anchor{48ce}
@subsection Asynchronous socket services


The @ref{8,,asynchat} and @ref{a,,asyncore} modules contain the following notice:

@example
Copyright 1996 by Sam Rushing

                        All Rights Reserved

Permission to use, copy, modify, and distribute this software and
its documentation for any purpose and without fee is hereby
granted, provided that the above copyright notice appear in all
copies and that both that copyright notice and this permission
notice appear in supporting documentation, and that the name of Sam
Rushing not be used in advertising or publicity pertaining to
distribution of the software without specific, written prior
permission.

SAM RUSHING DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE,
INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN
NO EVENT SHALL SAM RUSHING BE LIABLE FOR ANY SPECIAL, INDIRECT OR
CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
@end example

@node Cookie management,Execution tracing,Asynchronous socket services,Licenses and Acknowledgements for Incorporated Software
@anchor{license cookie-management}@anchor{48cf}
@subsection Cookie management


The @ref{91,,http.cookies} module contains the following notice:

@example
Copyright 2000 by Timothy O'Malley <timo@@alum.mit.edu>

               All Rights Reserved

Permission to use, copy, modify, and distribute this software
and its documentation for any purpose and without fee is hereby
granted, provided that the above copyright notice appear in all
copies and that both that copyright notice and this permission
notice appear in supporting documentation, and that the name of
Timothy O'Malley  not be used in advertising or publicity
pertaining to distribution of the software without specific, written
prior permission.

Timothy O'Malley DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS
SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS, IN NO EVENT SHALL Timothy O'Malley BE LIABLE FOR
ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
@end example

@node Execution tracing,UUencode and UUdecode functions,Cookie management,Licenses and Acknowledgements for Incorporated Software
@anchor{license execution-tracing}@anchor{48d0}
@subsection Execution tracing


The @ref{119,,trace} module contains the following notice:

@example
portions copyright 2001, Autonomous Zones Industries, Inc., all rights...
err...  reserved and offered to the public under the terms of the
Python 2.2 license.
Author: Zooko O'Whielacronx
http://zooko.com/
mailto:zooko@@zooko.com

Copyright 2000, Mojam Media, Inc., all rights reserved.
Author: Skip Montanaro

Copyright 1999, Bioreason, Inc., all rights reserved.
Author: Andrew Dalke

Copyright 1995-1997, Automatrix, Inc., all rights reserved.
Author: Skip Montanaro

Copyright 1991-1995, Stichting Mathematisch Centrum, all rights reserved.


Permission to use, copy, modify, and distribute this Python software and
its associated documentation for any purpose without fee is hereby
granted, provided that the above copyright notice appears in all copies,
and that both that copyright notice and this permission notice appear in
supporting documentation, and that the name of neither Automatrix,
Bioreason or Mojam Media be used in advertising or publicity pertaining to
distribution of the software without specific, written prior permission.
@end example

@node UUencode and UUdecode functions,XML Remote Procedure Calls,Execution tracing,Licenses and Acknowledgements for Incorporated Software
@anchor{license uuencode-and-uudecode-functions}@anchor{48d1}
@subsection UUencode and UUdecode functions


The @ref{12a,,uu} module contains the following notice:

@example
Copyright 1994 by Lance Ellinghouse
Cathedral City, California Republic, United States of America.
                       All Rights Reserved
Permission to use, copy, modify, and distribute this software and its
documentation for any purpose and without fee is hereby granted,
provided that the above copyright notice appear in all copies and that
both that copyright notice and this permission notice appear in
supporting documentation, and that the name of Lance Ellinghouse
not be used in advertising or publicity pertaining to distribution
of the software without specific, written prior permission.
LANCE ELLINGHOUSE DISCLAIMS ALL WARRANTIES WITH REGARD TO
THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS, IN NO EVENT SHALL LANCE ELLINGHOUSE CENTRUM BE LIABLE
FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT
OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

Modified by Jack Jansen, CWI, July 1995:
- Use binascii module to do the actual line-by-line conversion
  between ascii and binary. This results in a 1000-fold speedup. The C
  version is still 5 times faster, though.
- Arguments more compliant with Python standard
@end example

@node XML Remote Procedure Calls,test_epoll,UUencode and UUdecode functions,Licenses and Acknowledgements for Incorporated Software
@anchor{license xml-remote-procedure-calls}@anchor{48d2}
@subsection XML Remote Procedure Calls


The @ref{146,,xmlrpc.client} module contains the following notice:

@example
    The XML-RPC client interface is

Copyright (c) 1999-2002 by Secret Labs AB
Copyright (c) 1999-2002 by Fredrik Lundh

By obtaining, using, and/or copying this software and/or its
associated documentation, you agree that you have read, understood,
and will comply with the following terms and conditions:

Permission to use, copy, modify, and distribute this software and
its associated documentation for any purpose and without fee is
hereby granted, provided that the above copyright notice appears in
all copies, and that both that copyright notice and this permission
notice appear in supporting documentation, and that the name of
Secret Labs AB or the author not be used in advertising or publicity
pertaining to distribution of the software without specific, written
prior permission.

SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD
TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANT-
ABILITY AND FITNESS.  IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR
BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY
DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
OF THIS SOFTWARE.
@end example

@node test_epoll,Select kqueue,XML Remote Procedure Calls,Licenses and Acknowledgements for Incorporated Software
@anchor{license test-epoll}@anchor{48d3}
@subsection test_epoll


The @code{test_epoll} module contains the following notice:

@example
Copyright (c) 2001-2006 Twisted Matrix Laboratories.

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@end example

@node Select kqueue,SipHash24,test_epoll,Licenses and Acknowledgements for Incorporated Software
@anchor{license select-kqueue}@anchor{48d4}
@subsection Select kqueue


The @ref{e0,,select} module contains the following notice for the kqueue
interface:

@example
Copyright (c) 2000 Doug White, 2006 James Knight, 2007 Christian Heimes
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS `@w{`}AS IS'@w{'} AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
@end example

@node SipHash24,strtod and dtoa,Select kqueue,Licenses and Acknowledgements for Incorporated Software
@anchor{license siphash24}@anchor{48d5}
@subsection SipHash24


The file @code{Python/pyhash.c} contains Marek Majkowski’ implementation of
Dan Bernstein’s SipHash24 algorithm. It contains the following note:

@example
<MIT License>
Copyright (c) 2013  Marek Majkowski <marek@@popcount.org>

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
</MIT License>

Original location:
   https://github.com/majek/csiphash/

Solution inspired by code from:
   Samuel Neves (supercop/crypto_auth/siphash24/little)
   djb (supercop/crypto_auth/siphash24/little2)
   Jean-Philippe Aumasson (https://131002.net/siphash/siphash24.c)
@end example

@node strtod and dtoa,OpenSSL,SipHash24,Licenses and Acknowledgements for Incorporated Software
@anchor{license strtod-and-dtoa}@anchor{48d6}
@subsection strtod and dtoa


The file @code{Python/dtoa.c}, which supplies C functions dtoa and
strtod for conversion of C doubles to and from strings, is derived
from the file of the same name by David M. Gay, currently available
from @indicateurl{http://www.netlib.org/fp/}.  The original file, as retrieved on
March 16, 2009, contains the following copyright and licensing
notice:

@example
/****************************************************************
 *
 * The author of this software is David M. Gay.
 *
 * Copyright (c) 1991, 2000, 2001 by Lucent Technologies.
 *
 * Permission to use, copy, modify, and distribute this software for any
 * purpose without fee is hereby granted, provided that this entire notice
 * is included in all copies of any software which is or includes a copy
 * or modification of this software and in all copies of the supporting
 * documentation for such software.
 *
 * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED
 * WARRANTY.  IN PARTICULAR, NEITHER THE AUTHOR NOR LUCENT MAKES ANY
 * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY
 * OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE.
 *
 ***************************************************************/
@end example

@node OpenSSL,expat,strtod and dtoa,Licenses and Acknowledgements for Incorporated Software
@anchor{license openssl}@anchor{48d7}
@subsection OpenSSL


The modules @ref{88,,hashlib}, @ref{cc,,posix}, @ref{ee,,ssl}, @ref{27,,crypt} use
the OpenSSL library for added performance if made available by the
operating system. Additionally, the Windows and macOS installers for
Python may include a copy of the OpenSSL libraries, so we include a copy
of the OpenSSL license here:

@example
 LICENSE ISSUES
 ==============

 The OpenSSL toolkit stays under a dual license, i.e. both the conditions of
 the OpenSSL License and the original SSLeay license apply to the toolkit.
 See below for the actual license texts. Actually both licenses are BSD-style
 Open Source licenses. In case of any license issues related to OpenSSL
 please contact openssl-core@@openssl.org.

 OpenSSL License
 ---------------

   /* ====================================================================
    * Copyright (c) 1998-2008 The OpenSSL Project.  All rights reserved.
    *
    * Redistribution and use in source and binary forms, with or without
    * modification, are permitted provided that the following conditions
    * are met:
    *
    * 1. Redistributions of source code must retain the above copyright
    *    notice, this list of conditions and the following disclaimer.
    *
    * 2. Redistributions in binary form must reproduce the above copyright
    *    notice, this list of conditions and the following disclaimer in
    *    the documentation and/or other materials provided with the
    *    distribution.
    *
    * 3. All advertising materials mentioning features or use of this
    *    software must display the following acknowledgment:
    *    "This product includes software developed by the OpenSSL Project
    *    for use in the OpenSSL Toolkit. (http://www.openssl.org/)"
    *
    * 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to
    *    endorse or promote products derived from this software without
    *    prior written permission. For written permission, please contact
    *    openssl-core@@openssl.org.
    *
    * 5. Products derived from this software may not be called "OpenSSL"
    *    nor may "OpenSSL" appear in their names without prior written
    *    permission of the OpenSSL Project.
    *
    * 6. Redistributions of any form whatsoever must retain the following
    *    acknowledgment:
    *    "This product includes software developed by the OpenSSL Project
    *    for use in the OpenSSL Toolkit (http://www.openssl.org/)"
    *
    * THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT `@w{`}AS IS'@w{'} AND ANY
    * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
    * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
    * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE OpenSSL PROJECT OR
    * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
    * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
    * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
    * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
    * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
    * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
    * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
    * OF THE POSSIBILITY OF SUCH DAMAGE.
    * ====================================================================
    *
    * This product includes cryptographic software written by Eric Young
    * (eay@@cryptsoft.com).  This product includes software written by Tim
    * Hudson (tjh@@cryptsoft.com).
    *
    */

Original SSLeay License
-----------------------

   /* Copyright (C) 1995-1998 Eric Young (eay@@cryptsoft.com)
    * All rights reserved.
    *
    * This package is an SSL implementation written
    * by Eric Young (eay@@cryptsoft.com).
    * The implementation was written so as to conform with Netscapes SSL.
    *
    * This library is free for commercial and non-commercial use as long as
    * the following conditions are aheared to.  The following conditions
    * apply to all code found in this distribution, be it the RC4, RSA,
    * lhash, DES, etc., code; not just the SSL code.  The SSL documentation
    * included with this distribution is covered by the same copyright terms
    * except that the holder is Tim Hudson (tjh@@cryptsoft.com).
    *
    * Copyright remains Eric Young's, and as such any Copyright notices in
    * the code are not to be removed.
    * If this package is used in a product, Eric Young should be given attribution
    * as the author of the parts of the library used.
    * This can be in the form of a textual message at program startup or
    * in documentation (online or textual) provided with the package.
    *
    * Redistribution and use in source and binary forms, with or without
    * modification, are permitted provided that the following conditions
    * are met:
    * 1. Redistributions of source code must retain the copyright
    *    notice, this list of conditions and the following disclaimer.
    * 2. Redistributions in binary form must reproduce the above copyright
    *    notice, this list of conditions and the following disclaimer in the
    *    documentation and/or other materials provided with the distribution.
    * 3. All advertising materials mentioning features or use of this software
    *    must display the following acknowledgement:
    *    "This product includes cryptographic software written by
    *     Eric Young (eay@@cryptsoft.com)"
    *    The word 'cryptographic' can be left out if the rouines from the library
    *    being used are not cryptographic related :-).
    * 4. If you include any Windows specific code (or a derivative thereof) from
    *    the apps directory (application code) you must include an acknowledgement:
    *    "This product includes software written by Tim Hudson (tjh@@cryptsoft.com)"
    *
    * THIS SOFTWARE IS PROVIDED BY ERIC YOUNG `@w{`}AS IS'@w{'} AND
    * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
    * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
    * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
    * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
    * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
    * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
    * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
    * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
    * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
    * SUCH DAMAGE.
    *
    * The licence and distribution terms for any publically available version or
    * derivative of this code cannot be changed.  i.e. this code cannot simply be
    * copied and put under another distribution licence
    * [including the GNU Public Licence.]
    */
@end example

@node expat,libffi,OpenSSL,Licenses and Acknowledgements for Incorporated Software
@anchor{license expat}@anchor{48d8}
@subsection expat


The @code{pyexpat} extension is built using an included copy of the expat
sources unless the build is configured @code{--with-system-expat}:

@example
Copyright (c) 1998, 1999, 2000 Thai Open Source Software Center Ltd
                               and Clark Cooper

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@end example

@node libffi,zlib<3>,expat,Licenses and Acknowledgements for Incorporated Software
@anchor{license libffi}@anchor{48d9}
@subsection libffi


The @code{_ctypes} extension is built using an included copy of the libffi
sources unless the build is configured @code{--with-system-libffi}:

@example
Copyright (c) 1996-2008  Red Hat, Inc and others.

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
`@w{`}Software'@w{'}), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED `@w{`}AS IS'@w{'}, WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT.  IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
@end example

@node zlib<3>,cfuhash,libffi,Licenses and Acknowledgements for Incorporated Software
@anchor{license zlib}@anchor{48da}
@subsection zlib


The @ref{14b,,zlib} extension is built using an included copy of the zlib
sources if the zlib version found on the system is too old to be
used for the build:

@example
Copyright (C) 1995-2011 Jean-loup Gailly and Mark Adler

This software is provided 'as-is', without any express or implied
warranty.  In no event will the authors be held liable for any damages
arising from the use of this software.

Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:

1. The origin of this software must not be misrepresented; you must not
   claim that you wrote the original software. If you use this software
   in a product, an acknowledgment in the product documentation would be
   appreciated but is not required.

2. Altered source versions must be plainly marked as such, and must not be
   misrepresented as being the original software.

3. This notice may not be removed or altered from any source distribution.

Jean-loup Gailly        Mark Adler
jloup@@gzip.org          madler@@alumni.caltech.edu
@end example

@node cfuhash,libmpdec,zlib<3>,Licenses and Acknowledgements for Incorporated Software
@anchor{license cfuhash}@anchor{48db}
@subsection cfuhash


The implementation of the hash table used by the @ref{11b,,tracemalloc} is based
on the cfuhash project:

@example
Copyright (c) 2005 Don Owens
All rights reserved.

This code is released under the BSD license:

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:

  * Redistributions of source code must retain the above copyright
    notice, this list of conditions and the following disclaimer.

  * Redistributions in binary form must reproduce the above
    copyright notice, this list of conditions and the following
    disclaimer in the documentation and/or other materials provided
    with the distribution.

  * Neither the name of the author nor the names of its
    contributors may be used to endorse or promote products derived
    from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
OF THE POSSIBILITY OF SUCH DAMAGE.
@end example

@node libmpdec,W3C C14N test suite,cfuhash,Licenses and Acknowledgements for Incorporated Software
@anchor{license libmpdec}@anchor{48dc}
@subsection libmpdec


The @code{_decimal} module is built using an included copy of the libmpdec
library unless the build is configured @code{--with-system-libmpdec}:

@example
Copyright (c) 2008-2020 Stefan Krah. All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:

1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
@end example

@node W3C C14N test suite,,libmpdec,Licenses and Acknowledgements for Incorporated Software
@anchor{license w3c-c14n-test-suite}@anchor{48dd}
@subsection W3C C14N test suite


The C14N 2.0 test suite in the @ref{ff,,test} package
(@code{Lib/test/xmltestdata/c14n-20/}) was retrieved from the W3C website at
@indicateurl{https://www.w3.org/TR/xml-c14n2-testcases/} and is distributed under the
3-clause BSD license:

@example
Copyright (c) 2013 W3C(R) (MIT, ERCIM, Keio, Beihang),
All Rights Reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:

* Redistributions of works must retain the original copyright notice,
  this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the original copyright
  notice, this list of conditions and the following disclaimer in the
  documentation and/or other materials provided with the distribution.
* Neither the name of the W3C nor the names of its contributors may be
  used to endorse or promote products derived from this work without
  specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
@end example

@c to include legacy packaging docs in build

@node Distributing Python Modules Legacy version,Installing Python Modules Legacy version,History and License,Top
@anchor{distutils/index doc}@anchor{48de}@anchor{distutils/index distributing-python-modules-legacy-version}@anchor{48df}@anchor{distutils/index distutils-index}@anchor{a6c}
@chapter Distributing Python Modules (Legacy version)


@*Authors: 
Greg Ward, Anthony Baxter


@*Email: 
@email{distutils-sig@@python.org}



@subsubheading See also



@table @asis

@item @ref{a6a,,Distributing Python Modules}

The up to date module distribution documentations
@end table


@cartouche
@quotation Note 
The entire @code{distutils} package has been deprecated and will be
removed in Python 3.12. This documentation is retained as a
reference only, and will be removed with the package. See the
@ref{279,,What’s New} entry for more information.
@end quotation
@end cartouche

@cartouche
@quotation Note 
This document is being retained solely until the @code{setuptools} documentation
at @indicateurl{https://setuptools.readthedocs.io/en/latest/setuptools.html}
independently covers all of the relevant information currently included here.
@end quotation
@end cartouche

@cartouche
@quotation Note 
This guide only covers the basic tools for building and distributing
extensions that are provided as part of this version of Python. Third party
tools offer easier to use and more secure alternatives. Refer to the quick recommendations section@footnote{https://packaging.python.org/guides/tool-recommendations/}
in the Python Packaging User Guide for more information.
@end quotation
@end cartouche

This document describes the Python Distribution Utilities (“Distutils”) from
the module developer’s point of view, describing the underlying capabilities
that @code{setuptools} builds on to allow Python developers to make Python modules
and extensions readily available to a wider audience.

@menu
* An Introduction to Distutils:: 
* Writing the Setup Script:: 
* Writing the Setup Configuration File:: 
* Creating a Source Distribution:: 
* Creating Built Distributions:: 
* Distutils Examples:: 
* Extending Distutils:: 
* Command Reference:: 
* API Reference:: 

@end menu

@node An Introduction to Distutils,Writing the Setup Script,,Distributing Python Modules Legacy version
@anchor{distutils/introduction doc}@anchor{48e0}@anchor{distutils/introduction an-introduction-to-distutils}@anchor{48e1}@anchor{distutils/introduction distutils-intro}@anchor{48e2}
@section An Introduction to Distutils


@cartouche
@quotation Note 
This document is being retained solely until the @code{setuptools} documentation
at @indicateurl{https://setuptools.readthedocs.io/en/latest/setuptools.html}
independently covers all of the relevant information currently included here.
@end quotation
@end cartouche

This document covers using the Distutils to distribute your Python modules,
concentrating on the role of developer/distributor: if you’re looking for
information on installing Python modules, you should refer to the
@ref{a6b,,Installing Python Modules (Legacy version)} chapter.

@menu
* Concepts & Terminology:: 
* A Simple Example: A Simple Example<2>. 
* General Python terminology:: 
* Distutils-specific terminology:: 

@end menu

@node Concepts & Terminology,A Simple Example<2>,,An Introduction to Distutils
@anchor{distutils/introduction concepts-terminology}@anchor{48e3}@anchor{distutils/introduction distutils-concepts}@anchor{48e4}
@subsection Concepts & Terminology


Using the Distutils is quite simple, both for module developers and for
users/administrators installing third-party modules.  As a developer, your
responsibilities (apart from writing solid, well-documented and well-tested
code, of course!) are:


@itemize *

@item 
write a setup script (@code{setup.py} by convention)

@item 
(optional) write a setup configuration file

@item 
create a source distribution

@item 
(optional) create one or more built (binary) distributions
@end itemize

Each of these tasks is covered in this document.

Not all module developers have access to a multitude of platforms, so it’s not
always feasible to expect them to create a multitude of built distributions.  It
is hoped that a class of intermediaries, called @emph{packagers}, will arise to
address this need.  Packagers will take source distributions released by module
developers, build them on one or more platforms, and release the resulting built
distributions.  Thus, users on the most popular platforms will be able to
install most popular Python module distributions in the most natural way for
their platform, without having to run a single setup script or compile a line of
code.

@node A Simple Example<2>,General Python terminology,Concepts & Terminology,An Introduction to Distutils
@anchor{distutils/introduction a-simple-example}@anchor{48e5}@anchor{distutils/introduction distutils-simple-example}@anchor{48e6}
@subsection A Simple Example


The setup script is usually quite simple, although since it’s written in Python,
there are no arbitrary limits to what you can do with it, though you should be
careful about putting arbitrarily expensive operations in your setup script.
Unlike, say, Autoconf-style configure scripts, the setup script may be run
multiple times in the course of building and installing your module
distribution.

If all you want to do is distribute a module called @code{foo}, contained in a
file @code{foo.py}, then your setup script can be as simple as this:

@example
from distutils.core import setup
setup(name='foo',
      version='1.0',
      py_modules=['foo'],
      )
@end example

Some observations:


@itemize *

@item 
most information that you supply to the Distutils is supplied as keyword
arguments to the @code{setup()} function

@item 
those keyword arguments fall into two categories: package metadata (name,
version number) and information about what’s in the package (a list of pure
Python modules, in this case)

@item 
modules are specified by module name, not filename (the same will hold true
for packages and extensions)

@item 
it’s recommended that you supply a little more metadata, in particular your
name, email address and a URL for the project (see section @ref{48e7,,Writing the Setup Script}
for an example)
@end itemize

To create a source distribution for this module, you would create a setup
script, @code{setup.py}, containing the above code, and run this command from a
terminal:

@example
python setup.py sdist
@end example

For Windows, open a command prompt window (Start ‣
Accessories) and change the command to:

@example
setup.py sdist
@end example

@code{sdist} will create an archive file (e.g., tarball on Unix, ZIP file on Windows)
containing your setup script @code{setup.py}, and your module @code{foo.py}.
The archive file will be named @code{foo-1.0.tar.gz} (or @code{.zip}), and
will unpack into a directory @code{foo-1.0}.

If an end-user wishes to install your @code{foo} module, all they have to do is
download @code{foo-1.0.tar.gz} (or @code{.zip}), unpack it, and—from the
@code{foo-1.0} directory—run

@example
python setup.py install
@end example

which will ultimately copy @code{foo.py} to the appropriate directory for
third-party modules in their Python installation.

This simple example demonstrates some fundamental concepts of the Distutils.
First, both developers and installers have the same basic user interface, i.e.
the setup script.  The difference is which Distutils @emph{commands} they use: the
@code{sdist} command is almost exclusively for module developers, while
@code{install} is more often for installers (although most developers will
want to install their own code occasionally).

Other useful built distribution formats are RPM, implemented by the
@code{bdist_rpm} command, Solaris @code{pkgtool}
(@code{bdist_pkgtool}), and HP-UX @code{swinstall}
(@code{bdist_sdux}).  For example, the following command will create an RPM
file called @code{foo-1.0.noarch.rpm}:

@example
python setup.py bdist_rpm
@end example

(The @code{bdist_rpm} command uses the @code{rpm} executable, therefore
this has to be run on an RPM-based system such as Red Hat Linux, SuSE Linux, or
Mandrake Linux.)

You can find out what distribution formats are available at any time by running

@example
python setup.py bdist --help-formats
@end example

@node General Python terminology,Distutils-specific terminology,A Simple Example<2>,An Introduction to Distutils
@anchor{distutils/introduction general-python-terminology}@anchor{48e8}@anchor{distutils/introduction python-terms}@anchor{48e9}
@subsection General Python terminology


If you’re reading this document, you probably have a good idea of what modules,
extensions, and so forth are.  Nevertheless, just to be sure that everyone is
operating from a common starting point, we offer the following glossary of
common Python terms:


@table @asis

@item module

the basic unit of code reusability in Python: a block of code imported by some
other code.  Three types of modules concern us here: pure Python modules,
extension modules, and packages.

@item pure Python module

a module written in Python and contained in a single @code{.py} file (and
possibly associated @code{.pyc} files).  Sometimes referred to as a
“pure module.”

@item extension module

a module written in the low-level language of the Python implementation: C/C++
for Python, Java for Jython. Typically contained in a single dynamically
loadable pre-compiled file, e.g. a shared object (@code{.so}) file for Python
extensions on Unix, a DLL (given the @code{.pyd} extension) for Python
extensions on Windows, or a Java class file for Jython extensions.  (Note that
currently, the Distutils only handles C/C++ extensions for Python.)

@item package

a module that contains other modules; typically contained in a directory in the
filesystem and distinguished from other directories by the presence of a file
@code{__init__.py}.

@item root package

the root of the hierarchy of packages.  (This isn’t really a package, since it
doesn’t have an @code{__init__.py} file.  But we have to call it something.)
The vast majority of the standard library is in the root package, as are many
small, standalone third-party modules that don’t belong to a larger module
collection. Unlike regular packages, modules in the root package can be found in
many directories: in fact, every directory listed in @code{sys.path} contributes
modules to the root package.
@end table

@node Distutils-specific terminology,,General Python terminology,An Introduction to Distutils
@anchor{distutils/introduction distutils-specific-terminology}@anchor{48ea}@anchor{distutils/introduction distutils-term}@anchor{48eb}
@subsection Distutils-specific terminology


The following terms apply more specifically to the domain of distributing Python
modules using the Distutils:


@table @asis

@item module distribution

a collection of Python modules distributed together as a single downloadable
resource and meant to be installed @emph{en masse}.  Examples of some well-known
module distributions are NumPy, SciPy, Pillow,
or mxBase.  (This would be called a @emph{package}, except that term is
already taken in the Python context: a single module distribution may contain
zero, one, or many Python packages.)

@item pure module distribution

a module distribution that contains only pure Python modules and packages.
Sometimes referred to as a “pure distribution.”

@item non-pure module distribution

a module distribution that contains at least one extension module.  Sometimes
referred to as a “non-pure distribution.”

@item distribution root

the top-level directory of your source tree (or  source distribution); the
directory where @code{setup.py} exists.  Generally  @code{setup.py} will be
run from this directory.
@end table

@node Writing the Setup Script,Writing the Setup Configuration File,An Introduction to Distutils,Distributing Python Modules Legacy version
@anchor{distutils/setupscript doc}@anchor{48ec}@anchor{distutils/setupscript setup-script}@anchor{48e7}@anchor{distutils/setupscript writing-the-setup-script}@anchor{48ed}
@section Writing the Setup Script


@cartouche
@quotation Note 
This document is being retained solely until the @code{setuptools} documentation
at @indicateurl{https://setuptools.readthedocs.io/en/latest/setuptools.html}
independently covers all of the relevant information currently included here.
@end quotation
@end cartouche

The setup script is the centre of all activity in building, distributing, and
installing modules using the Distutils.  The main purpose of the setup script is
to describe your module distribution to the Distutils, so that the various
commands that operate on your modules do the right thing.  As we saw in section
@ref{48e6,,A Simple Example} above, the setup script consists mainly of a call to
@code{setup()}, and most information supplied to the Distutils by the module
developer is supplied as keyword arguments to @code{setup()}.

Here’s a slightly more involved example, which we’ll follow for the next couple
of sections: the Distutils’ own setup script.  (Keep in mind that although the
Distutils are included with Python 1.6 and later, they also have an independent
existence so that Python 1.5.2 users can use them to install other module
distributions.  The Distutils’ own setup script, shown here, is used to install
the package into Python 1.5.2.)

@example
#!/usr/bin/env python

from distutils.core import setup

setup(name='Distutils',
      version='1.0',
      description='Python Distribution Utilities',
      author='Greg Ward',
      author_email='gward@@python.net',
      url='https://www.python.org/sigs/distutils-sig/',
      packages=['distutils', 'distutils.command'],
     )
@end example

There are only two differences between this and the trivial one-file
distribution presented in section @ref{48e6,,A Simple Example}: more metadata, and the
specification of pure Python modules by package, rather than by module.  This is
important since the Distutils consist of a couple of dozen modules split into
(so far) two packages; an explicit list of every module would be tedious to
generate and difficult to maintain.  For more information on the additional
meta-data, see section @ref{48ee,,Additional meta-data}.

Note that any pathnames (files or directories) supplied in the setup script
should be written using the Unix convention, i.e. slash-separated.  The
Distutils will take care of converting this platform-neutral representation into
whatever is appropriate on your current platform before actually using the
pathname.  This makes your setup script portable across operating systems, which
of course is one of the major goals of the Distutils.  In this spirit, all
pathnames in this document are slash-separated.

This, of course, only applies to pathnames given to Distutils functions.  If
you, for example, use standard Python functions such as @ref{288,,glob.glob()} or
@ref{c8d,,os.listdir()} to specify files, you should be careful to write portable
code instead of hardcoding path separators:

@example
glob.glob(os.path.join('mydir', 'subdir', '*.html'))
os.listdir(os.path.join('mydir', 'subdir'))
@end example

@menu
* Listing whole packages:: 
* Listing individual modules:: 
* Describing extension modules:: 
* Relationships between Distributions and Packages:: 
* Installing Scripts:: 
* Installing Package Data:: 
* Installing Additional Files:: 
* Additional meta-data:: 
* Debugging the setup script:: 

@end menu

@node Listing whole packages,Listing individual modules,,Writing the Setup Script
@anchor{distutils/setupscript listing-packages}@anchor{48ef}@anchor{distutils/setupscript listing-whole-packages}@anchor{48f0}
@subsection Listing whole packages


The @code{packages} option tells the Distutils to process (build, distribute,
install, etc.) all pure Python modules found in each package mentioned in the
@code{packages} list.  In order to do this, of course, there has to be a
correspondence between package names and directories in the filesystem.  The
default correspondence is the most obvious one, i.e. package @ref{37,,distutils} is
found in the directory @code{distutils} relative to the distribution root.
Thus, when you say @code{packages = ['foo']} in your setup script, you are
promising that the Distutils will find a file @code{foo/__init__.py} (which
might be spelled differently on your system, but you get the idea) relative to
the directory where your setup script lives.  If you break this promise, the
Distutils will issue a warning but still process the broken package anyway.

If you use a different convention to lay out your source directory, that’s no
problem: you just have to supply the @code{package_dir} option to tell the
Distutils about your convention.  For example, say you keep all Python source
under @code{lib}, so that modules in the “root package” (i.e., not in any
package at all) are in @code{lib}, modules in the @code{foo} package are in
@code{lib/foo}, and so forth.  Then you would put

@example
package_dir = @{'@w{'}: 'lib'@}
@end example

in your setup script.  The keys to this dictionary are package names, and an
empty package name stands for the root package.  The values are directory names
relative to your distribution root.  In this case, when you say @code{packages =
['foo']}, you are promising that the file @code{lib/foo/__init__.py} exists.

Another possible convention is to put the @code{foo} package right in
@code{lib}, the @code{foo.bar} package in @code{lib/bar}, etc.  This would be
written in the setup script as

@example
package_dir = @{'foo': 'lib'@}
@end example

A @code{package: dir} entry in the @code{package_dir} dictionary implicitly
applies to all packages below @emph{package}, so the @code{foo.bar} case is
automatically handled here.  In this example, having @code{packages = ['foo',
'foo.bar']} tells the Distutils to look for @code{lib/__init__.py} and
@code{lib/bar/__init__.py}.  (Keep in mind that although @code{package_dir}
applies recursively, you must explicitly list all packages in
@code{packages}: the Distutils will @emph{not} recursively scan your source tree
looking for any directory with an @code{__init__.py} file.)

@node Listing individual modules,Describing extension modules,Listing whole packages,Writing the Setup Script
@anchor{distutils/setupscript listing-individual-modules}@anchor{48f1}@anchor{distutils/setupscript listing-modules}@anchor{48f2}
@subsection Listing individual modules


For a small module distribution, you might prefer to list all modules rather
than listing packages—especially the case of a single module that goes in the
“root package” (i.e., no package at all).  This simplest case was shown in
section @ref{48e6,,A Simple Example}; here is a slightly more involved example:

@example
py_modules = ['mod1', 'pkg.mod2']
@end example

This describes two modules, one of them in the “root” package, the other in the
@code{pkg} package.  Again, the default package/directory layout implies that
these two modules can be found in @code{mod1.py} and @code{pkg/mod2.py}, and
that @code{pkg/__init__.py} exists as well. And again, you can override the
package/directory correspondence using the @code{package_dir} option.

@node Describing extension modules,Relationships between Distributions and Packages,Listing individual modules,Writing the Setup Script
@anchor{distutils/setupscript describing-extension-modules}@anchor{48f3}@anchor{distutils/setupscript describing-extensions}@anchor{48f4}
@subsection Describing extension modules


Just as writing Python extension modules is a bit more complicated than writing
pure Python modules, describing them to the Distutils is a bit more complicated.
Unlike pure modules, it’s not enough just to list modules or packages and expect
the Distutils to go out and find the right files; you have to specify the
extension name, source file(s), and any compile/link requirements (include
directories, libraries to link with, etc.).

@c XXX read over this section

All of this is done through another keyword argument to @code{setup()}, the
@code{ext_modules} option.  @code{ext_modules} is just a list of
@ref{48f5,,Extension} instances, each of which describes a
single extension module.
Suppose your distribution includes a single extension, called @code{foo} and
implemented by @code{foo.c}.  If no additional instructions to the
compiler/linker are needed, describing this extension is quite simple:

@example
Extension('foo', ['foo.c'])
@end example

The @code{Extension} class can be imported from @ref{50,,distutils.core} along
with @code{setup()}.  Thus, the setup script for a module distribution that
contains only this one extension and nothing else might be:

@example
from distutils.core import setup, Extension
setup(name='foo',
      version='1.0',
      ext_modules=[Extension('foo', ['foo.c'])],
      )
@end example

The @code{Extension} class (actually, the underlying extension-building
machinery implemented by the @code{build_ext} command) supports a great deal
of flexibility in describing Python extensions, which is explained in the
following sections.

@menu
* Extension names and packages:: 
* Extension source files:: 
* Preprocessor options:: 
* Library options:: 
* Other options:: 

@end menu

@node Extension names and packages,Extension source files,,Describing extension modules
@anchor{distutils/setupscript extension-names-and-packages}@anchor{48f6}
@subsubsection Extension names and packages


The first argument to the @ref{48f5,,Extension} constructor is
always the name of the extension, including any package names.  For example,

@example
Extension('foo', ['src/foo1.c', 'src/foo2.c'])
@end example

describes an extension that lives in the root package, while

@example
Extension('pkg.foo', ['src/foo1.c', 'src/foo2.c'])
@end example

describes the same extension in the @code{pkg} package.  The source files and
resulting object code are identical in both cases; the only difference is where
in the filesystem (and therefore where in Python’s namespace hierarchy) the
resulting extension lives.

If you have a number of extensions all in the same package (or all under the
same base package), use the @code{ext_package} keyword argument to
@code{setup()}.  For example,

@example
setup(...,
      ext_package='pkg',
      ext_modules=[Extension('foo', ['foo.c']),
                   Extension('subpkg.bar', ['bar.c'])],
     )
@end example

will compile @code{foo.c} to the extension @code{pkg.foo}, and @code{bar.c} to
@code{pkg.subpkg.bar}.

@node Extension source files,Preprocessor options,Extension names and packages,Describing extension modules
@anchor{distutils/setupscript extension-source-files}@anchor{48f7}
@subsubsection Extension source files


The second argument to the @ref{48f5,,Extension} constructor is
a list of source
files.  Since the Distutils currently only support C, C++, and Objective-C
extensions, these are normally C/C++/Objective-C source files.  (Be sure to use
appropriate extensions to distinguish C++ source files: @code{.cc} and
@code{.cpp} seem to be recognized by both Unix and Windows compilers.)

However, you can also include SWIG interface (@code{.i}) files in the list; the
@code{build_ext} command knows how to deal with SWIG extensions: it will run
SWIG on the interface file and compile the resulting C/C++ file into your
extension.

@c XXX SWIG support is rough around the edges and largely untested!

This warning notwithstanding, options to SWIG can be currently passed like
this:

@example
setup(...,
      ext_modules=[Extension('_foo', ['foo.i'],
                             swig_opts=['-modern', '-I../include'])],
      py_modules=['foo'],
     )
@end example

Or on the commandline like this:

@example
> python setup.py build_ext --swig-opts="-modern -I../include"
@end example

On some platforms, you can include non-source files that are processed by the
compiler and included in your extension.  Currently, this just means Windows
message text (@code{.mc}) files and resource definition (@code{.rc}) files for
Visual C++. These will be compiled to binary resource (@code{.res}) files and
linked into the executable.

@node Preprocessor options,Library options,Extension source files,Describing extension modules
@anchor{distutils/setupscript preprocessor-options}@anchor{48f8}
@subsubsection Preprocessor options


Three optional arguments to @ref{48f5,,Extension} will help if
you need to specify include directories to search or preprocessor macros to
define/undefine: @code{include_dirs}, @code{define_macros}, and @code{undef_macros}.

For example, if your extension requires header files in the @code{include}
directory under your distribution root, use the @code{include_dirs} option:

@example
Extension('foo', ['foo.c'], include_dirs=['include'])
@end example

You can specify absolute directories there; if you know that your extension will
only be built on Unix systems with X11R6 installed to @code{/usr}, you can get
away with

@example
Extension('foo', ['foo.c'], include_dirs=['/usr/include/X11'])
@end example

You should avoid this sort of non-portable usage if you plan to distribute your
code: it’s probably better to write C code like

@example
#include <X11/Xlib.h>
@end example

If you need to include header files from some other Python extension, you can
take advantage of the fact that header files are installed in a consistent way
by the Distutils @code{install_headers} command.  For example, the Numerical
Python header files are installed (on a standard Unix installation) to
@code{/usr/local/include/python1.5/Numerical}. (The exact location will differ
according to your platform and Python installation.)  Since the Python include
directory—@code{/usr/local/include/python1.5} in this case—is always
included in the search path when building Python extensions, the best approach
is to write C code like

@example
#include <Numerical/arrayobject.h>
@end example

If you must put the @code{Numerical} include directory right into your header
search path, though, you can find that directory using the Distutils
@ref{5e,,distutils.sysconfig} module:

@example
from distutils.sysconfig import get_python_inc
incdir = os.path.join(get_python_inc(plat_specific=1), 'Numerical')
setup(...,
      Extension(..., include_dirs=[incdir]),
      )
@end example

Even though this is quite portable—it will work on any Python installation,
regardless of platform—it’s probably easier to just write your C code in the
sensible way.

You can define and undefine pre-processor macros with the @code{define_macros} and
@code{undef_macros} options. @code{define_macros} takes a list of @code{(name, value)}
tuples, where @code{name} is the name of the macro to define (a string) and
@code{value} is its value: either a string or @code{None}.  (Defining a macro @code{FOO}
to @code{None} is the equivalent of a bare @code{#define FOO} in your C source: with
most compilers, this sets @code{FOO} to the string @code{1}.)  @code{undef_macros} is
just a list of macros to undefine.

For example:

@example
Extension(...,
          define_macros=[('NDEBUG', '1'),
                         ('HAVE_STRFTIME', None)],
          undef_macros=['HAVE_FOO', 'HAVE_BAR'])
@end example

is the equivalent of having this at the top of every C source file:

@example
#define NDEBUG 1
#define HAVE_STRFTIME
#undef HAVE_FOO
#undef HAVE_BAR
@end example

@node Library options,Other options,Preprocessor options,Describing extension modules
@anchor{distutils/setupscript library-options}@anchor{48f9}
@subsubsection Library options


You can also specify the libraries to link against when building your extension,
and the directories to search for those libraries.  The @code{libraries} option is
a list of libraries to link against, @code{library_dirs} is a list of directories
to search for libraries at  link-time, and @code{runtime_library_dirs} is a list of
directories to  search for shared (dynamically loaded) libraries at run-time.

For example, if you need to link against libraries known to be in the standard
library search path on target systems

@example
Extension(...,
          libraries=['gdbm', 'readline'])
@end example

If you need to link with libraries in a non-standard location, you’ll have to
include the location in @code{library_dirs}:

@example
Extension(...,
          library_dirs=['/usr/X11R6/lib'],
          libraries=['X11', 'Xt'])
@end example

(Again, this sort of non-portable construct should be avoided if you intend to
distribute your code.)

@c XXX Should mention clib libraries here or somewhere else!

@node Other options,,Library options,Describing extension modules
@anchor{distutils/setupscript other-options}@anchor{48fa}
@subsubsection Other options


There are still some other options which can be used to handle special cases.

The @code{optional} option is a boolean; if it is true,
a build failure in the extension will not abort the build process, but
instead simply not install the failing extension.

The @code{extra_objects} option is a list of object files to be passed to the
linker. These files must not have extensions, as the default extension for the
compiler is used.

@code{extra_compile_args} and @code{extra_link_args} can be used to
specify additional command line options for the respective compiler and linker
command lines.

@code{export_symbols} is only useful on Windows.  It can contain a list of
symbols (functions or variables) to be exported. This option is not needed when
building compiled extensions: Distutils  will automatically add @code{initmodule}
to the list of exported symbols.

The @code{depends} option is a list of files that the extension depends on
(for example header files). The build command will call the compiler on the
sources to rebuild extension if any on this files has been modified since the
previous build.

@node Relationships between Distributions and Packages,Installing Scripts,Describing extension modules,Writing the Setup Script
@anchor{distutils/setupscript relationships-between-distributions-and-packages}@anchor{48fb}
@subsection Relationships between Distributions and Packages


A distribution may relate to packages in three specific ways:


@enumerate 

@item 
It can require packages or modules.

@item 
It can provide packages or modules.

@item 
It can obsolete packages or modules.
@end enumerate

These relationships can be specified using keyword arguments to the
@ref{40db,,distutils.core.setup()} function.

Dependencies on other Python modules and packages can be specified by supplying
the @emph{requires} keyword argument to @code{setup()}. The value must be a list of
strings.  Each string specifies a package that is required, and optionally what
versions are sufficient.

To specify that any version of a module or package is required, the string
should consist entirely of the module or package name. Examples include
@code{'mymodule'} and @code{'xml.parsers.expat'}.

If specific versions are required, a sequence of qualifiers can be supplied in
parentheses.  Each qualifier may consist of a comparison operator and a version
number.  The accepted comparison operators are:

@example
<    >    ==
<=   >=   !=
@end example

These can be combined by using multiple qualifiers separated by commas (and
optional whitespace).  In this case, all of the qualifiers must be matched; a
logical AND is used to combine the evaluations.

Let’s look at a bunch of examples:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Requires Expression

@tab

Explanation

@item

@code{==1.0}

@tab

Only version @code{1.0} is compatible

@item

@code{>1.0, !=1.5.1, <2.0}

@tab

Any version after @code{1.0} and before @code{2.0}
is compatible, except @code{1.5.1}

@end multitable


Now that we can specify dependencies, we also need to be able to specify what we
provide that other distributions can require.  This is done using the @emph{provides}
keyword argument to @code{setup()}. The value for this keyword is a list of
strings, each of which names a Python module or package, and optionally
identifies the version.  If the version is not specified, it is assumed to match
that of the distribution.

Some examples:


@multitable {xxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Provides Expression

@tab

Explanation

@item

@code{mypkg}

@tab

Provide @code{mypkg}, using the distribution
version

@item

@code{mypkg (1.1)}

@tab

Provide @code{mypkg} version 1.1, regardless of
the distribution version

@end multitable


A package can declare that it obsoletes other packages using the @emph{obsoletes}
keyword argument.  The value for this is similar to that of the @emph{requires}
keyword: a list of strings giving module or package specifiers.  Each specifier
consists of a module or package name optionally followed by one or more version
qualifiers.  Version qualifiers are given in parentheses after the module or
package name.

The versions identified by the qualifiers are those that are obsoleted by the
distribution being described.  If no qualifiers are given, all versions of the
named module or package are understood to be obsoleted.

@node Installing Scripts,Installing Package Data,Relationships between Distributions and Packages,Writing the Setup Script
@anchor{distutils/setupscript distutils-installing-scripts}@anchor{48fc}@anchor{distutils/setupscript installing-scripts}@anchor{48fd}
@subsection Installing Scripts


So far we have been dealing with pure and non-pure Python modules, which are
usually not run by themselves but imported by scripts.

Scripts are files containing Python source code, intended to be started from the
command line.  Scripts don’t require Distutils to do anything very complicated.
The only clever feature is that if the first line of the script starts with
@code{#!} and contains the word “python”, the Distutils will adjust the first line
to refer to the current interpreter location. By default, it is replaced with
the current interpreter location.  The @code{--executable} (or @code{-e})
option will allow the interpreter path to be explicitly overridden.

The @code{scripts} option simply is a list of files to be handled in this
way.  From the PyXML setup script:

@example
setup(...,
      scripts=['scripts/xmlproc_parse', 'scripts/xmlproc_val']
      )
@end example


Changed in version 3.1: All the scripts will also be added to the @code{MANIFEST} file if no template is
provided.  See @ref{40de,,Specifying the files to distribute}.


@node Installing Package Data,Installing Additional Files,Installing Scripts,Writing the Setup Script
@anchor{distutils/setupscript distutils-installing-package-data}@anchor{48fe}@anchor{distutils/setupscript installing-package-data}@anchor{48ff}
@subsection Installing Package Data


Often, additional files need to be installed into a package.  These files are
often data that’s closely related to the package’s implementation, or text files
containing documentation that might be of interest to programmers using the
package.  These files are called @emph{package data}.

Package data can be added to packages using the @code{package_data} keyword
argument to the @code{setup()} function.  The value must be a mapping from
package name to a list of relative path names that should be copied into the
package.  The paths are interpreted as relative to the directory containing the
package (information from the @code{package_dir} mapping is used if appropriate);
that is, the files are expected to be part of the package in the source
directories. They may contain glob patterns as well.

The path names may contain directory portions; any necessary directories will be
created in the installation.

For example, if a package should contain a subdirectory with several data files,
the files can be arranged like this in the source tree:

@example
setup.py
src/
    mypkg/
        __init__.py
        module.py
        data/
            tables.dat
            spoons.dat
            forks.dat
@end example

The corresponding call to @code{setup()} might be:

@example
setup(...,
      packages=['mypkg'],
      package_dir=@{'mypkg': 'src/mypkg'@},
      package_data=@{'mypkg': ['data/*.dat']@},
      )
@end example


Changed in version 3.1: All the files that match @code{package_data} will be added to the @code{MANIFEST}
file if no template is provided.  See @ref{40de,,Specifying the files to distribute}.


@node Installing Additional Files,Additional meta-data,Installing Package Data,Writing the Setup Script
@anchor{distutils/setupscript distutils-additional-files}@anchor{4900}@anchor{distutils/setupscript installing-additional-files}@anchor{4901}
@subsection Installing Additional Files


The @code{data_files} option can be used to specify additional files needed
by the module distribution: configuration files, message catalogs, data files,
anything which doesn’t fit in the previous categories.

@code{data_files} specifies a sequence of (@emph{directory}, @emph{files}) pairs in the
following way:

@example
setup(...,
      data_files=[('bitmaps', ['bm/b1.gif', 'bm/b2.gif']),
                  ('config', ['cfg/data.cfg'])],
     )
@end example

Each (@emph{directory}, @emph{files}) pair in the sequence specifies the installation
directory and the files to install there.

Each file name in @emph{files} is interpreted relative to the @code{setup.py}
script at the top of the package source distribution. Note that you can
specify the directory where the data files will be installed, but you cannot
rename the data files themselves.

The @emph{directory} should be a relative path. It is interpreted relative to the
installation prefix (Python’s @code{sys.prefix} for system installations;
@code{site.USER_BASE} for user installations). Distutils allows @emph{directory} to be
an absolute installation path, but this is discouraged since it is
incompatible with the wheel packaging format. No directory information from
@emph{files} is used to determine the final location of the installed file; only
the name of the file is used.

You can specify the @code{data_files} options as a simple sequence of files
without specifying a target directory, but this is not recommended, and the
@code{install} command will print a warning in this case. To install data
files directly in the target directory, an empty string should be given as the
directory.


Changed in version 3.1: All the files that match @code{data_files} will be added to the @code{MANIFEST}
file if no template is provided.  See @ref{40de,,Specifying the files to distribute}.


@node Additional meta-data,Debugging the setup script,Installing Additional Files,Writing the Setup Script
@anchor{distutils/setupscript additional-meta-data}@anchor{4902}@anchor{distutils/setupscript meta-data}@anchor{48ee}
@subsection Additional meta-data


The setup script may include additional meta-data beyond the name and version.
This information includes:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxx} 
@headitem

Meta-Data

@tab

Description

@tab

Value

@tab

Notes

@item

@code{name}

@tab

name of the package

@tab

short string

@tab

(1)

@item

@code{version}

@tab

version of this release

@tab

short string

@tab

(1)(2)

@item

@code{author}

@tab

package author’s name

@tab

short string

@tab

(3)

@item

@code{author_email}

@tab

email address of the
package author

@tab

email address

@tab

(3)

@item

@code{maintainer}

@tab

package maintainer’s name

@tab

short string

@tab

(3)

@item

@code{maintainer_email}

@tab

email address of the
package maintainer

@tab

email address

@tab

(3)

@item

@code{url}

@tab

home page for the package

@tab

URL

@tab

(1)

@item

@code{description}

@tab

short, summary
description of the
package

@tab

short string

@tab

@item

@code{long_description}

@tab

longer description of the
package

@tab

long string

@tab

(4)

@item

@code{download_url}

@tab

location where the
package may be downloaded

@tab

URL

@tab

@item

@code{classifiers}

@tab

a list of classifiers

@tab

list of strings

@tab

(6)(7)

@item

@code{platforms}

@tab

a list of platforms

@tab

list of strings

@tab

(6)(8)

@item

@code{keywords}

@tab

a list of keywords

@tab

list of strings

@tab

(6)(8)

@item

@code{license}

@tab

license for the package

@tab

short string

@tab

(5)

@end multitable


Notes:


@enumerate 

@item 
These fields are required.

@item 
It is recommended that versions take the form @emph{major.minor[.patch[.sub]]}.

@item 
Either the author or the maintainer must be identified. If maintainer is
provided, distutils lists it as the author in @code{PKG-INFO}.

@item 
The @code{long_description} field is used by PyPI when you publish a package,
to build its project page.

@item 
The @code{license} field is a text indicating the license covering the
package where the license is not a selection from the “License” Trove
classifiers. See the @code{Classifier} field. Notice that
there’s a @code{licence} distribution option which is deprecated but still
acts as an alias for @code{license}.

@item 
This field must be a list.

@item 
The valid classifiers are listed on
PyPI@footnote{https://pypi.org/classifiers}.

@item 
To preserve backward compatibility, this field also accepts a string. If
you pass a comma-separated string @code{'foo, bar'}, it will be converted to
@code{['foo', 'bar']}, Otherwise, it will be converted to a list of one
string.
@end enumerate


@table @asis

@item ‘short string’

A single line of text, not more than 200 characters.

@item ‘long string’

Multiple lines of plain text in reStructuredText format (see
@indicateurl{http://docutils.sourceforge.net/}).

@item ‘list of strings’

See below.
@end table

Encoding the version information is an art in itself. Python packages generally
adhere to the version format @emph{major.minor[.patch][sub]}. The major number is 0
for initial, experimental releases of software. It is incremented for releases
that represent major milestones in a package. The minor number is incremented
when important new features are added to the package. The patch number
increments when bug-fix releases are made. Additional trailing version
information is sometimes used to indicate sub-releases.  These are
“a1,a2,…,aN” (for alpha releases, where functionality and API may change),
“b1,b2,…,bN” (for beta releases, which only fix bugs) and “pr1,pr2,…,prN”
(for final pre-release release testing). Some examples:


@table @asis

@item 0.1.0

the first, experimental release of a package

@item 1.0.1a2

the second alpha release of the first patch version of 1.0
@end table

@code{classifiers} must be specified in a list:

@example
setup(...,
      classifiers=[
          'Development Status :: 4 - Beta',
          'Environment :: Console',
          'Environment :: Web Environment',
          'Intended Audience :: End Users/Desktop',
          'Intended Audience :: Developers',
          'Intended Audience :: System Administrators',
          'License :: OSI Approved :: Python Software Foundation License',
          'Operating System :: MacOS :: MacOS X',
          'Operating System :: Microsoft :: Windows',
          'Operating System :: POSIX',
          'Programming Language :: Python',
          'Topic :: Communications :: Email',
          'Topic :: Office/Business',
          'Topic :: Software Development :: Bug Tracking',
          ],
      )
@end example


Changed in version 3.7: @ref{40db,,setup} now warns when @code{classifiers}, @code{keywords}
or @code{platforms} fields are not specified as a list or a string.


@node Debugging the setup script,,Additional meta-data,Writing the Setup Script
@anchor{distutils/setupscript debug-setup-script}@anchor{4903}@anchor{distutils/setupscript debugging-the-setup-script}@anchor{4904}
@subsection Debugging the setup script


Sometimes things go wrong, and the setup script doesn’t do what the developer
wants.

Distutils catches any exceptions when running the setup script, and print a
simple error message before the script is terminated.  The motivation for this
behaviour is to not confuse administrators who don’t know much about Python and
are trying to install a package.  If they get a big long traceback from deep
inside the guts of Distutils, they may think the package or the Python
installation is broken because they don’t read all the way down to the bottom
and see that it’s a permission problem.

On the other hand, this doesn’t help the developer to find the cause of the
failure. For this purpose, the 
@geindex DISTUTILS_DEBUG
@geindex environment variable; DISTUTILS_DEBUG
@code{DISTUTILS_DEBUG} environment variable can be set
to anything except an empty string, and distutils will now print detailed
information about what it is doing, dump the full traceback when an exception
occurs, and print the whole command line when an external program (like a C
compiler) fails.

@node Writing the Setup Configuration File,Creating a Source Distribution,Writing the Setup Script,Distributing Python Modules Legacy version
@anchor{distutils/configfile doc}@anchor{4905}@anchor{distutils/configfile setup-config}@anchor{4906}@anchor{distutils/configfile writing-the-setup-configuration-file}@anchor{4907}
@section Writing the Setup Configuration File


@cartouche
@quotation Note 
This document is being retained solely until the @code{setuptools} documentation
at @indicateurl{https://setuptools.readthedocs.io/en/latest/setuptools.html}
independently covers all of the relevant information currently included here.
@end quotation
@end cartouche

Often, it’s not possible to write down everything needed to build a distribution
@emph{a priori}: you may need to get some information from the user, or from the
user’s system, in order to proceed.  As long as that information is fairly
simple—a list of directories to search for C header files or libraries, for
example—then providing a configuration file, @code{setup.cfg}, for users to
edit is a cheap and easy way to solicit it.  Configuration files also let you
provide default values for any command option, which the installer can then
override either on the command-line or by editing the config file.

The setup configuration file is a useful middle-ground between the setup
script—which, ideally, would be opaque to installers @footnote{@w{(1)} 
This ideal probably won’t be achieved until auto-configuration is fully
supported by the Distutils.
}—and the command-line to
the setup script, which is outside of your control and entirely up to the
installer.  In fact, @code{setup.cfg} (and any other Distutils configuration
files present on the target system) are processed after the contents of the
setup script, but before the command-line.  This has  several useful
consequences:

@c % (If you have more advanced needs, such as determining which extensions

@c % to build based on what capabilities are present on the target system,

@c % then you need the Distutils ``auto-configuration'' facility.  This

@c % started to appear in Distutils 0.9 but, as of this writing, isn't mature

@c % or stable enough yet for real-world use.)


@itemize *

@item 
installers can override some of what you put in @code{setup.py} by editing
@code{setup.cfg}

@item 
you can provide non-standard defaults for options that are not easily set in
@code{setup.py}

@item 
installers can override anything in @code{setup.cfg} using the command-line
options to @code{setup.py}
@end itemize

The basic syntax of the configuration file is simple:

@example
[command]
option=value
...
@end example

where @emph{command} is one of the Distutils commands (e.g. @code{build_py},
@code{install}), and @emph{option} is one of the options that command supports.
Any number of options can be supplied for each command, and any number of
command sections can be included in the file.  Blank lines are ignored, as are
comments, which run from a @code{'#'} character until the end of the line.  Long
option values can be split across multiple lines simply by indenting the
continuation lines.

You can find out the list of options supported by a particular command with the
universal @code{--help} option, e.g.

@example
$ python setup.py --help build_ext
[...]
Options for 'build_ext' command:
  --build-lib (-b)     directory for compiled extension modules
  --build-temp (-t)    directory for temporary files (build by-products)
  --inplace (-i)       ignore build-lib and put compiled extensions into the
                       source directory alongside your pure Python modules
  --include-dirs (-I)  list of directories to search for header files
  --define (-D)        C preprocessor macros to define
  --undef (-U)         C preprocessor macros to undefine
  --swig-opts          list of SWIG command line options
[...]
@end example

Note that an option spelled @code{--foo-bar} on the command-line  is spelled
@code{foo_bar} in configuration files.
@anchor{distutils/configfile distutils-build-ext-inplace}@anchor{4908}
For example, say you want your extensions to be built “in-place”—that is, you
have an extension @code{pkg.ext}, and you want the compiled extension file
(@code{ext.so} on Unix, say) to be put in the same source directory as your
pure Python modules @code{pkg.mod1} and @code{pkg.mod2}.  You can always use the
@code{--inplace} option on the command-line to ensure this:

@example
python setup.py build_ext --inplace
@end example

But this requires that you always specify the @code{build_ext} command
explicitly, and remember to provide @code{--inplace}. An easier way is to
“set and forget” this option, by encoding it in @code{setup.cfg}, the
configuration file for this distribution:

@example
[build_ext]
inplace=1
@end example

This will affect all builds of this module distribution, whether or not you
explicitly specify @code{build_ext}.  If you include @code{setup.cfg} in
your source distribution, it will also affect end-user builds—which is
probably a bad idea for this option, since always building extensions in-place
would break installation of the module distribution.  In certain peculiar cases,
though, modules are built right in their installation directory, so this is
conceivably a useful ability.  (Distributing extensions that expect to be built
in their installation directory is almost always a bad idea, though.)

Another example: certain commands take a lot of options that don’t change from
run to run; for example, @code{bdist_rpm} needs to know everything required
to generate a “spec” file for creating an RPM distribution.  Some of this
information comes from the setup script, and some is automatically generated by
the Distutils (such as the list of files installed).  But some of it has to be
supplied as options to @code{bdist_rpm}, which would be very tedious to do
on the command-line for every run.  Hence, here is a snippet from the Distutils’
own @code{setup.cfg}:

@example
[bdist_rpm]
release = 1
packager = Greg Ward <gward@@python.net>
doc_files = CHANGES.txt
            README.txt
            USAGE.txt
            doc/
            examples/
@end example

Note that the @code{doc_files} option is simply a whitespace-separated string
split across multiple lines for readability.


@subsubheading See also



@table @asis

@item @ref{4909,,Syntax of config files} in “Installing Python Modules”

More information on the configuration files is available in the manual for
system administrators.
@end table


@node Creating a Source Distribution,Creating Built Distributions,Writing the Setup Configuration File,Distributing Python Modules Legacy version
@anchor{distutils/sourcedist doc}@anchor{490a}@anchor{distutils/sourcedist creating-a-source-distribution}@anchor{490b}@anchor{distutils/sourcedist source-dist}@anchor{490c}
@section Creating a Source Distribution


@cartouche
@quotation Note 
This document is being retained solely until the @code{setuptools} documentation
at @indicateurl{https://setuptools.readthedocs.io/en/latest/setuptools.html}
independently covers all of the relevant information currently included here.
@end quotation
@end cartouche

As shown in section @ref{48e6,,A Simple Example}, you use the @code{sdist} command
to create a source distribution.  In the simplest case,

@example
python setup.py sdist
@end example

(assuming you haven’t specified any @code{sdist} options in the setup script
or config file), @code{sdist} creates the archive of the default format for
the current platform.  The default format is a gzip’ed tar file
(@code{.tar.gz}) on Unix, and ZIP file on Windows.

You can specify as many formats as you like using the @code{--formats}
option, for example:

@example
python setup.py sdist --formats=gztar,zip
@end example

to create a gzipped tarball and a zip file.  The available formats are:


@multitable {xxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxx} 
@headitem

Format

@tab

Description

@tab

Notes

@item

@code{zip}

@tab

zip file (@code{.zip})

@tab

(1),(3)

@item

@code{gztar}

@tab

gzip’ed tar file
(@code{.tar.gz})

@tab

(2)

@item

@code{bztar}

@tab

bzip2’ed tar file
(@code{.tar.bz2})

@tab

(5)

@item

@code{xztar}

@tab

xz’ed tar file
(@code{.tar.xz})

@tab

(5)

@item

@code{ztar}

@tab

compressed tar file
(@code{.tar.Z})

@tab

(4),(5)

@item

@code{tar}

@tab

tar file (@code{.tar})

@tab

(5)

@end multitable



Changed in version 3.5: Added support for the @code{xztar} format.


Notes:


@enumerate 

@item 
default on Windows

@item 
default on Unix

@item 
requires either external @code{zip} utility or @ref{149,,zipfile} module (part
of the standard Python library since Python 1.6)

@item 
requires the @code{compress} program. Notice that this format is now
pending for deprecation and will be removed in the future versions of Python.

@item 
deprecated by PEP 527@footnote{https://www.python.org/dev/peps/pep-0527/};
PyPI@footnote{https://pypi.org} only accepts @code{.zip} and @code{.tar.gz} files.
@end enumerate

When using any @code{tar} format (@code{gztar}, @code{bztar}, @code{xztar}, @code{ztar} or
@code{tar}), under Unix you can specify the @code{owner} and @code{group} names
that will be set for each member of the archive.

For example, if you want all files of the archive to be owned by root:

@example
python setup.py sdist --owner=root --group=root
@end example

@menu
* Specifying the files to distribute:: 
* Manifest-related options:: 

@end menu

@node Specifying the files to distribute,Manifest-related options,,Creating a Source Distribution
@anchor{distutils/sourcedist manifest}@anchor{40de}@anchor{distutils/sourcedist specifying-the-files-to-distribute}@anchor{490d}
@subsection Specifying the files to distribute


If you don’t supply an explicit list of files (or instructions on how to
generate one), the @code{sdist} command puts a minimal default set into the
source distribution:


@itemize *

@item 
all Python source files implied by the @code{py_modules} and
@code{packages} options

@item 
all C source files mentioned in the @code{ext_modules} or
@code{libraries} options

@c XXX getting C library sources currently broken---no
@c :meth:`get_source_files` method in :file:`build_clib.py`!

@item 
scripts identified by the @code{scripts} option
See @ref{48fc,,Installing Scripts}.

@item 
anything that looks like a test script: @code{test/test*.py} (currently, the
Distutils don’t do anything with test scripts except include them in source
distributions, but in the future there will be a standard for testing Python
module distributions)

@item 
Any of the standard README files (@code{README}, @code{README.txt},
or @code{README.rst}), @code{setup.py} (or whatever you called your setup
script), and @code{setup.cfg}.

@item 
all files that matches the @code{package_data} metadata.
See @ref{48fe,,Installing Package Data}.

@item 
all files that matches the @code{data_files} metadata.
See @ref{4900,,Installing Additional Files}.
@end itemize

Sometimes this is enough, but usually you will want to specify additional files
to distribute.  The typical way to do this is to write a @emph{manifest template},
called @code{MANIFEST.in} by default.  The manifest template is just a list of
instructions for how to generate your manifest file, @code{MANIFEST}, which is
the exact list of files to include in your source distribution.  The
@code{sdist} command processes this template and generates a manifest based
on its instructions and what it finds in the filesystem.

If you prefer to roll your own manifest file, the format is simple: one filename
per line, regular files (or symlinks to them) only.  If you do supply your own
@code{MANIFEST}, you must specify everything: the default set of files
described above does not apply in this case.


Changed in version 3.1: An existing generated @code{MANIFEST} will be regenerated without
@code{sdist} comparing its modification time to the one of
@code{MANIFEST.in} or @code{setup.py}.



Changed in version 3.1.3: @code{MANIFEST} files start with a comment indicating they are generated.
Files without this comment are not overwritten or removed.



Changed in version 3.2.2: @code{sdist} will read a @code{MANIFEST} file if no @code{MANIFEST.in}
exists, like it used to do.



Changed in version 3.7: @code{README.rst} is now included in the list of distutils standard READMEs.


The manifest template has one command per line, where each command specifies a
set of files to include or exclude from the source distribution.  For an
example, again we turn to the Distutils’ own manifest template:

@example
include *.txt
recursive-include examples *.txt *.py
prune examples/sample?/build
@end example

The meanings should be fairly clear: include all files in the distribution root
matching @code{*.txt}, all files anywhere under the @code{examples} directory
matching @code{*.txt} or @code{*.py}, and exclude all directories matching
@code{examples/sample?/build}.  All of this is done @emph{after} the standard
include set, so you can exclude files from the standard set with explicit
instructions in the manifest template.  (Or, you can use the
@code{--no-defaults} option to disable the standard set entirely.)  There are
several other commands available in the manifest template mini-language; see
section @ref{490e,,Creating a source distribution; the sdist command}.

The order of commands in the manifest template matters: initially, we have the
list of default files as described above, and each command in the template adds
to or removes from that list of files.  Once we have fully processed the
manifest template, we remove files that should not be included in the source
distribution:


@itemize *

@item 
all files in the Distutils “build” tree (default @code{build/})

@item 
all files in directories named @code{RCS}, @code{CVS}, @code{.svn},
@code{.hg}, @code{.git}, @code{.bzr} or @code{_darcs}
@end itemize

Now we have our complete list of files, which is written to the manifest for
future reference, and then used to build the source distribution archive(s).

You can disable the default set of included files with the
@code{--no-defaults} option, and you can disable the standard exclude set
with @code{--no-prune}.

Following the Distutils’ own manifest template, let’s trace how the
@code{sdist} command builds the list of files to include in the Distutils
source distribution:


@enumerate 

@item 
include all Python source files in the @code{distutils} and
@code{distutils/command} subdirectories (because packages corresponding to
those two directories were mentioned in the @code{packages} option in the
setup script—see section @ref{48e7,,Writing the Setup Script})

@item 
include @code{README.txt}, @code{setup.py}, and @code{setup.cfg} (standard
files)

@item 
include @code{test/test*.py} (standard files)

@item 
include @code{*.txt} in the distribution root (this will find
@code{README.txt} a second time, but such redundancies are weeded out later)

@item 
include anything matching @code{*.txt} or @code{*.py} in the sub-tree
under @code{examples},

@item 
exclude all files in the sub-trees starting at directories matching
@code{examples/sample?/build}—this may exclude files included by the
previous two steps, so it’s important that the @code{prune} command in the manifest
template comes after the @code{recursive-include} command

@item 
exclude the entire @code{build} tree, and any @code{RCS}, @code{CVS},
@code{.svn}, @code{.hg}, @code{.git}, @code{.bzr} and @code{_darcs}
directories
@end enumerate

Just like in the setup script, file and directory names in the manifest template
should always be slash-separated; the Distutils will take care of converting
them to the standard representation on your platform. That way, the manifest
template is portable across operating systems.

@node Manifest-related options,,Specifying the files to distribute,Creating a Source Distribution
@anchor{distutils/sourcedist manifest-options}@anchor{490f}@anchor{distutils/sourcedist manifest-related-options}@anchor{4910}
@subsection Manifest-related options


The normal course of operations for the @code{sdist} command is as follows:


@itemize *

@item 
if the manifest file (@code{MANIFEST} by default) exists and the first line
does not have a comment indicating it is generated from @code{MANIFEST.in},
then it is used as is, unaltered

@item 
if the manifest file doesn’t exist or has been previously automatically
generated, read @code{MANIFEST.in} and create the manifest

@item 
if neither @code{MANIFEST} nor @code{MANIFEST.in} exist, create a manifest
with just the default file set

@item 
use the list of files now in @code{MANIFEST} (either just generated or read
in) to create the source distribution archive(s)
@end itemize

There are a couple of options that modify this behaviour.  First, use the
@code{--no-defaults} and @code{--no-prune} to disable the standard
“include” and “exclude” sets.

Second, you might just want to (re)generate the manifest, but not create a source
distribution:

@example
python setup.py sdist --manifest-only
@end example

@code{-o} is a shortcut for @code{--manifest-only}.

@node Creating Built Distributions,Distutils Examples,Creating a Source Distribution,Distributing Python Modules Legacy version
@anchor{distutils/builtdist doc}@anchor{4911}@anchor{distutils/builtdist built-dist}@anchor{4912}@anchor{distutils/builtdist creating-built-distributions}@anchor{4913}
@section Creating Built Distributions


@cartouche
@quotation Note 
This document is being retained solely until the @code{setuptools} documentation
at @indicateurl{https://setuptools.readthedocs.io/en/latest/setuptools.html}
independently covers all of the relevant information currently included here.
@end quotation
@end cartouche

A “built distribution” is what you’re probably used to thinking of either as a
“binary package” or an “installer” (depending on your background).  It’s not
necessarily binary, though, because it might contain only Python source code
and/or byte-code; and we don’t call it a package, because that word is already
spoken for in Python.  (And “installer” is a term specific to the world of
mainstream desktop systems.)

A built distribution is how you make life as easy as possible for installers of
your module distribution: for users of RPM-based Linux systems, it’s a binary
RPM; for Windows users, it’s an executable installer; for Debian-based Linux
users, it’s a Debian package; and so forth.  Obviously, no one person will be
able to create built distributions for every platform under the sun, so the
Distutils are designed to enable module developers to concentrate on their
specialty—writing code and creating source distributions—while an
intermediary species called @emph{packagers} springs up to turn source distributions
into built distributions for as many platforms as there are packagers.

Of course, the module developer could be their own packager; or the packager could
be a volunteer “out there” somewhere who has access to a platform which the
original developer does not; or it could be software periodically grabbing new
source distributions and turning them into built distributions for as many
platforms as the software has access to.  Regardless of who they are, a packager
uses the setup script and the @code{bdist} command family to generate built
distributions.

As a simple example, if I run the following command in the Distutils source
tree:

@example
python setup.py bdist
@end example

then the Distutils builds my module distribution (the Distutils itself in this
case), does a “fake” installation (also in the @code{build} directory), and
creates the default type of built distribution for my platform.  The default
format for built distributions is a “dumb” tar file on Unix, and a simple
executable installer on Windows.  (That tar file is considered “dumb” because it
has to be unpacked in a specific location to work.)

Thus, the above command on a Unix system creates
@code{Distutils-1.0.@emph{plat}.tar.gz}; unpacking this tarball from the right place
installs the Distutils just as though you had downloaded the source distribution
and run @code{python setup.py install}.  (The “right place” is either the root of
the filesystem or  Python’s @code{@emph{prefix}} directory, depending on the options
given to the @code{bdist_dumb} command; the default is to make dumb
distributions relative to @code{@emph{prefix}}.)

Obviously, for pure Python distributions, this isn’t any simpler than just
running @code{python setup.py install}—but for non-pure distributions, which
include extensions that would need to be compiled, it can mean the difference
between someone being able to use your extensions or not.  And creating “smart”
built distributions, such as an RPM package or an executable installer for
Windows, is far more convenient for users even if your distribution doesn’t
include any extensions.

The @code{bdist} command has a @code{--formats} option, similar to the
@code{sdist} command, which you can use to select the types of built
distribution to generate: for example,

@example
python setup.py bdist --format=zip
@end example

would, when run on a Unix system, create
@code{Distutils-1.0.@emph{plat}.zip}—again, this archive would be unpacked
from the root directory to install the Distutils.

The available formats for built distributions are:


@multitable {xxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxx} 
@headitem

Format

@tab

Description

@tab

Notes

@item

@code{gztar}

@tab

gzipped tar file
(@code{.tar.gz})

@tab

(1)

@item

@code{bztar}

@tab

bzipped tar file
(@code{.tar.bz2})

@tab

@item

@code{xztar}

@tab

xzipped tar file
(@code{.tar.xz})

@tab

@item

@code{ztar}

@tab

compressed tar file
(@code{.tar.Z})

@tab

(3)

@item

@code{tar}

@tab

tar file (@code{.tar})

@tab

@item

@code{zip}

@tab

zip file (@code{.zip})

@tab

(2),(4)

@item

@code{rpm}

@tab

RPM

@tab

(5)

@item

@code{pkgtool}

@tab

Solaris @code{pkgtool}

@tab

@item

@code{sdux}

@tab

HP-UX @code{swinstall}

@tab

@item

@code{msi}

@tab

Microsoft Installer.

@tab

@end multitable



Changed in version 3.5: Added support for the @code{xztar} format.


Notes:


@enumerate 

@item 
default on Unix

@item 
default on Windows

@item 
requires external @code{compress} utility.

@item 
requires either external @code{zip} utility or @ref{149,,zipfile} module (part
of the standard Python library since Python 1.6)

@item 
requires external @code{rpm} utility, version 3.0.4 or better (use @code{rpm
--version} to find out which version you have)
@end enumerate

You don’t have to use the @code{bdist} command with the @code{--formats}
option; you can also use the command that directly implements the format you’re
interested in.  Some of these @code{bdist} “sub-commands” actually generate
several similar formats; for instance, the @code{bdist_dumb} command
generates all the “dumb” archive formats (@code{tar}, @code{gztar}, @code{bztar},
@code{xztar}, @code{ztar}, and @code{zip}), and @code{bdist_rpm} generates both
binary and source RPMs.  The @code{bdist} sub-commands, and the formats
generated by each, are:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Command

@tab

Formats

@item

@code{bdist_dumb}

@tab

tar, gztar, bztar, xztar, ztar, zip

@item

@code{bdist_rpm}

@tab

rpm, srpm

@end multitable


The following sections give details on the individual @code{bdist_*}
commands.

@c .. _creating-dumb:

@c Creating dumb built distributions

@c =================================

@c XXX Need to document absolute vs. prefix-relative packages here, but first
@c I have to implement it!

@menu
* Creating RPM packages:: 
* Cross-compiling on Windows:: 

@end menu

@node Creating RPM packages,Cross-compiling on Windows,,Creating Built Distributions
@anchor{distutils/builtdist creating-rpm-packages}@anchor{4914}@anchor{distutils/builtdist creating-rpms}@anchor{4915}
@subsection Creating RPM packages


The RPM format is used by many popular Linux distributions, including Red Hat,
SuSE, and Mandrake.  If one of these (or any of the other RPM-based Linux
distributions) is your usual environment, creating RPM packages for other users
of that same distribution is trivial. Depending on the complexity of your module
distribution and differences between Linux distributions, you may also be able
to create RPMs that work on different RPM-based distributions.

The usual way to create an RPM of your module distribution is to run the
@code{bdist_rpm} command:

@example
python setup.py bdist_rpm
@end example

or the @code{bdist} command with the @code{--format} option:

@example
python setup.py bdist --formats=rpm
@end example

The former allows you to specify RPM-specific options; the latter allows  you to
easily specify multiple formats in one run.  If you need to do both, you can
explicitly specify multiple @code{bdist_*} commands and their options:

@example
python setup.py bdist_rpm --packager="John Doe <jdoe@@example.org>"
@end example

Creating RPM packages is driven by a @code{.spec} file, much as using the
Distutils is driven by the setup script.  To make your life easier, the
@code{bdist_rpm} command normally creates a @code{.spec} file based on the
information you supply in the setup script, on the command line, and in any
Distutils configuration files.  Various options and sections in the
@code{.spec} file are derived from options in the setup script as follows:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

RPM @code{.spec} file option or section

@tab

Distutils setup script option

@item

Name

@tab

@code{name}

@item

Summary (in preamble)

@tab

@code{description}

@item

Version

@tab

@code{version}

@item

Vendor

@tab

@code{author} and @code{author_email},
or  — & @code{maintainer} and
@code{maintainer_email}

@item

Copyright

@tab

@code{license}

@item

Url

@tab

@code{url}

@item

%description (section)

@tab

@code{long_description}

@end multitable


Additionally, there are many options in @code{.spec} files that don’t have
corresponding options in the setup script.  Most of these are handled through
options to the @code{bdist_rpm} command as follows:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

RPM @code{.spec} file option
or section

@tab

@code{bdist_rpm} option

@tab

default value

@item

Release

@tab

@code{release}

@tab

“1”

@item

Group

@tab

@code{group}

@tab

“Development/Libraries”

@item

Vendor

@tab

@code{vendor}

@tab

(see above)

@item

Packager

@tab

@code{packager}

@tab

(none)

@item

Provides

@tab

@code{provides}

@tab

(none)

@item

Requires

@tab

@code{requires}

@tab

(none)

@item

Conflicts

@tab

@code{conflicts}

@tab

(none)

@item

Obsoletes

@tab

@code{obsoletes}

@tab

(none)

@item

Distribution

@tab

@code{distribution_name}

@tab

(none)

@item

BuildRequires

@tab

@code{build_requires}

@tab

(none)

@item

Icon

@tab

@code{icon}

@tab

(none)

@end multitable


Obviously, supplying even a few of these options on the command-line would be
tedious and error-prone, so it’s usually best to put them in the setup
configuration file, @code{setup.cfg}—see section @ref{4906,,Writing the Setup Configuration File}.  If
you distribute or package many Python module distributions, you might want to
put options that apply to all of them in your personal Distutils configuration
file (@code{~/.pydistutils.cfg}).  If you want to temporarily disable
this file, you can pass the @code{--no-user-cfg} option to @code{setup.py}.

There are three steps to building a binary RPM package, all of which are
handled automatically by the Distutils:


@enumerate 

@item 
create a @code{.spec} file, which describes the package (analogous  to the
Distutils setup script; in fact, much of the information in the  setup script
winds up in the @code{.spec} file)

@item 
create the source RPM

@item 
create the “binary” RPM (which may or may not contain binary code, depending
on whether your module distribution contains Python extensions)
@end enumerate

Normally, RPM bundles the last two steps together; when you use the Distutils,
all three steps are typically bundled together.

If you wish, you can separate these three steps.  You can use the
@code{--spec-only} option to make @code{bdist_rpm} just create the
@code{.spec} file and exit; in this case, the @code{.spec} file will be
written to the “distribution directory”—normally @code{dist/}, but
customizable with the @code{--dist-dir} option.  (Normally, the @code{.spec}
file winds up deep in the “build tree,” in a temporary directory created by
@code{bdist_rpm}.)

@c % \XXX{this isn't implemented yet---is it needed?!}

@c % You can also specify a custom \file{.spec} file with the

@c % \longprogramopt{spec-file} option; used in conjunction with

@c % \longprogramopt{spec-only}, this gives you an opportunity to customize

@c % the \file{.spec} file manually:

@c %

@c % \ begin{verbatim}

@c % > python setup.py bdist_rpm --spec-only

@c % # ...edit dist/FooBar-1.0.spec

@c % > python setup.py bdist_rpm --spec-file=dist/FooBar-1.0.spec

@c % \ end{verbatim}

@c %

@c % (Although a better way to do this is probably to override the standard

@c % \command{bdist\_rpm} command with one that writes whatever else you want

@c % to the \file{.spec} file.)

@node Cross-compiling on Windows,,Creating RPM packages,Creating Built Distributions
@anchor{distutils/builtdist cross-compile-windows}@anchor{4916}@anchor{distutils/builtdist cross-compiling-on-windows}@anchor{4917}
@subsection Cross-compiling on Windows


Starting with Python 2.6, distutils is capable of cross-compiling between
Windows platforms.  In practice, this means that with the correct tools
installed, you can use a 32bit version of Windows to create 64bit extensions
and vice-versa.

To build for an alternate platform, specify the @code{--plat-name} option
to the build command.  Valid values are currently ‘win32’, and  ‘win-amd64’.
For example, on a 32bit version of Windows, you could execute:

@example
python setup.py build --plat-name=win-amd64
@end example

to build a 64bit version of your extension.

would create a 64bit installation executable on your 32bit version of Windows.

To cross-compile, you must download the Python source code and cross-compile
Python itself for the platform you are targeting - it is not possible from a
binary installation of Python (as the .lib etc file for other platforms are
not included.)  In practice, this means the user of a 32 bit operating
system will need to use Visual Studio 2008 to open the
@code{PCbuild/PCbuild.sln} solution in the Python source tree and build the
“x64” configuration of the ‘pythoncore’ project before cross-compiling
extensions is possible.

Note that by default, Visual Studio 2008 does not install 64bit compilers or
tools.  You may need to reexecute the Visual Studio setup process and select
these tools (using Control Panel->[Add/Remove] Programs is a convenient way to
check or modify your existing install.)

@menu
* The Postinstallation script:: 

@end menu

@node The Postinstallation script,,,Cross-compiling on Windows
@anchor{distutils/builtdist postinstallation-script}@anchor{4918}@anchor{distutils/builtdist the-postinstallation-script}@anchor{4919}
@subsubsection The Postinstallation script


Starting with Python 2.3, a postinstallation script can be specified with the
@code{--install-script} option.  The basename of the script must be
specified, and the script filename must also be listed in the scripts argument
to the setup function.

This script will be run at installation time on the target system after all the
files have been copied, with @code{argv[1]} set to @code{-install}, and again at
uninstallation time before the files are removed with @code{argv[1]} set to
@code{-remove}.

The installation script runs embedded in the windows installer, every output
(@code{sys.stdout}, @code{sys.stderr}) is redirected into a buffer and will be
displayed in the GUI after the script has finished.

Some functions especially useful in this context are available as additional
built-in functions in the installation script.

@geindex built-in function; directory_created()
@geindex built-in function; file_created()
@anchor{distutils/builtdist directory_created}@anchor{491a}
@deffn {Function} directory_created (path)
@anchor{distutils/builtdist file_created}@anchor{491b}
@deffnx {Function} file_created (path)

These functions should be called when a directory or file is created by the
postinstall script at installation time.  It will register @emph{path} with the
uninstaller, so that it will be removed when the distribution is uninstalled.
To be safe, directories are only removed if they are empty.
@end deffn

@geindex built-in function; get_special_folder_path()
@anchor{distutils/builtdist get_special_folder_path}@anchor{491c}
@deffn {Function} get_special_folder_path (csidl_string)

This function can be used to retrieve special folder locations on Windows like
the Start Menu or the Desktop.  It returns the full path to the folder.
@emph{csidl_string} must be one of the following strings:

@example
"CSIDL_APPDATA"

"CSIDL_COMMON_STARTMENU"
"CSIDL_STARTMENU"

"CSIDL_COMMON_DESKTOPDIRECTORY"
"CSIDL_DESKTOPDIRECTORY"

"CSIDL_COMMON_STARTUP"
"CSIDL_STARTUP"

"CSIDL_COMMON_PROGRAMS"
"CSIDL_PROGRAMS"

"CSIDL_FONTS"
@end example

If the folder cannot be retrieved, @ref{2a1,,OSError} is raised.

Which folders are available depends on the exact Windows version, and probably
also the configuration.  For details refer to Microsoft’s documentation of the
@code{SHGetSpecialFolderPath()} function.
@end deffn

@geindex built-in function; create_shortcut()
@anchor{distutils/builtdist create_shortcut}@anchor{491d}
@deffn {Function} create_shortcut (target, description, filename[, arguments[, workdir[, iconpath[, iconindex]]]])

This function creates a shortcut. @emph{target} is the path to the program to be
started by the shortcut. @emph{description} is the description of the shortcut.
@emph{filename} is the title of the shortcut that the user will see. @emph{arguments}
specifies the command line arguments, if any. @emph{workdir} is the working directory
for the program. @emph{iconpath} is the file containing the icon for the shortcut,
and @emph{iconindex} is the index of the icon in the file @emph{iconpath}.  Again, for
details consult the Microsoft documentation for the @code{IShellLink}
interface.
@end deffn

@node Distutils Examples,Extending Distutils,Creating Built Distributions,Distributing Python Modules Legacy version
@anchor{distutils/examples doc}@anchor{491e}@anchor{distutils/examples distutils-examples}@anchor{491f}@anchor{distutils/examples id1}@anchor{4920}
@section Distutils Examples


@cartouche
@quotation Note 
This document is being retained solely until the @code{setuptools} documentation
at @indicateurl{https://setuptools.readthedocs.io/en/latest/setuptools.html}
independently covers all of the relevant information currently included here.
@end quotation
@end cartouche

This chapter provides a number of basic examples to help get started with
distutils.  Additional information about using distutils can be found in the
Distutils Cookbook.


@subsubheading See also



@table @asis

@item Distutils Cookbook@footnote{https://wiki.python.org/moin/Distutils/Cookbook}

Collection of recipes showing how to achieve more control over distutils.
@end table


@menu
* Pure Python distribution (by module): Pure Python distribution by module. 
* Pure Python distribution (by package): Pure Python distribution by package. 
* Single extension module:: 
* Checking a package:: 
* Reading the metadata:: 

@end menu

@node Pure Python distribution by module,Pure Python distribution by package,,Distutils Examples
@anchor{distutils/examples pure-mod}@anchor{4921}@anchor{distutils/examples pure-python-distribution-by-module}@anchor{4922}
@subsection Pure Python distribution (by module)


If you’re just distributing a couple of modules, especially if they don’t live
in a particular package, you can specify them individually using the
@code{py_modules} option in the setup script.

In the simplest case, you’ll have two files to worry about: a setup script and
the single module you’re distributing, @code{foo.py} in this example:

@example
<root>/
        setup.py
        foo.py
@end example

(In all diagrams in this section, @emph{<root>} will refer to the distribution root
directory.)  A minimal setup script to describe this situation would be:

@example
from distutils.core import setup
setup(name='foo',
      version='1.0',
      py_modules=['foo'],
      )
@end example

Note that the name of the distribution is specified independently with the
@code{name} option, and there’s no rule that says it has to be the same as
the name of the sole module in the distribution (although that’s probably a good
convention to follow).  However, the distribution name is used to generate
filenames, so you should stick to letters, digits, underscores, and hyphens.

Since @code{py_modules} is a list, you can of course specify multiple
modules, eg. if you’re distributing modules @code{foo} and @code{bar}, your
setup might look like this:

@example
<root>/
        setup.py
        foo.py
        bar.py
@end example

and the setup script might be

@example
from distutils.core import setup
setup(name='foobar',
      version='1.0',
      py_modules=['foo', 'bar'],
      )
@end example

You can put module source files into another directory, but if you have enough
modules to do that, it’s probably easier to specify modules by package rather
than listing them individually.

@node Pure Python distribution by package,Single extension module,Pure Python distribution by module,Distutils Examples
@anchor{distutils/examples pure-pkg}@anchor{4923}@anchor{distutils/examples pure-python-distribution-by-package}@anchor{4924}
@subsection Pure Python distribution (by package)


If you have more than a couple of modules to distribute, especially if they are
in multiple packages, it’s probably easier to specify whole packages rather than
individual modules.  This works even if your modules are not in a package; you
can just tell the Distutils to process modules from the root package, and that
works the same as any other package (except that you don’t have to have an
@code{__init__.py} file).

The setup script from the last example could also be written as

@example
from distutils.core import setup
setup(name='foobar',
      version='1.0',
      packages=['@w{'}],
      )
@end example

(The empty string stands for the root package.)

If those two files are moved into a subdirectory, but remain in the root
package, e.g.:

@example
<root>/
        setup.py
        src/      foo.py
                  bar.py
@end example

then you would still specify the root package, but you have to tell the
Distutils where source files in the root package live:

@example
from distutils.core import setup
setup(name='foobar',
      version='1.0',
      package_dir=@{'@w{'}: 'src'@},
      packages=['@w{'}],
      )
@end example

More typically, though, you will want to distribute multiple modules in the same
package (or in sub-packages).  For example, if the @code{foo}  and @code{bar}
modules belong in package @code{foobar}, one way to layout your source tree is

@example
<root>/
        setup.py
        foobar/
                 __init__.py
                 foo.py
                 bar.py
@end example

This is in fact the default layout expected by the Distutils, and the one that
requires the least work to describe in your setup script:

@example
from distutils.core import setup
setup(name='foobar',
      version='1.0',
      packages=['foobar'],
      )
@end example

If you want to put modules in directories not named for their package, then you
need to use the @code{package_dir} option again.  For example, if the
@code{src} directory holds modules in the @code{foobar} package:

@example
<root>/
        setup.py
        src/
                 __init__.py
                 foo.py
                 bar.py
@end example

an appropriate setup script would be

@example
from distutils.core import setup
setup(name='foobar',
      version='1.0',
      package_dir=@{'foobar': 'src'@},
      packages=['foobar'],
      )
@end example

Or, you might put modules from your main package right in the distribution
root:

@example
<root>/
        setup.py
        __init__.py
        foo.py
        bar.py
@end example

in which case your setup script would be

@example
from distutils.core import setup
setup(name='foobar',
      version='1.0',
      package_dir=@{'foobar': '@w{'}@},
      packages=['foobar'],
      )
@end example

(The empty string also stands for the current directory.)

If you have sub-packages, they must be explicitly listed in @code{packages},
but any entries in @code{package_dir} automatically extend to sub-packages.
(In other words, the Distutils does @emph{not} scan your source tree, trying to
figure out which directories correspond to Python packages by looking for
@code{__init__.py} files.)  Thus, if the default layout grows a sub-package:

@example
<root>/
        setup.py
        foobar/
                 __init__.py
                 foo.py
                 bar.py
                 subfoo/
                           __init__.py
                           blah.py
@end example

then the corresponding setup script would be

@example
from distutils.core import setup
setup(name='foobar',
      version='1.0',
      packages=['foobar', 'foobar.subfoo'],
      )
@end example

@node Single extension module,Checking a package,Pure Python distribution by package,Distutils Examples
@anchor{distutils/examples single-ext}@anchor{4925}@anchor{distutils/examples single-extension-module}@anchor{4926}
@subsection Single extension module


Extension modules are specified using the @code{ext_modules} option.
@code{package_dir} has no effect on where extension source files are found;
it only affects the source for pure Python modules.  The simplest  case, a
single extension module in a single C source file, is:

@example
<root>/
        setup.py
        foo.c
@end example

If the @code{foo} extension belongs in the root package, the setup script for
this could be

@example
from distutils.core import setup
from distutils.extension import Extension
setup(name='foobar',
      version='1.0',
      ext_modules=[Extension('foo', ['foo.c'])],
      )
@end example

If the extension actually belongs in a package, say @code{foopkg}, then

With exactly the same source tree layout, this extension can be put in the
@code{foopkg} package simply by changing the name of the extension:

@example
from distutils.core import setup
from distutils.extension import Extension
setup(name='foobar',
      version='1.0',
      ext_modules=[Extension('foopkg.foo', ['foo.c'])],
      )
@end example

@node Checking a package,Reading the metadata,Single extension module,Distutils Examples
@anchor{distutils/examples checking-a-package}@anchor{4927}
@subsection Checking a package


The @code{check} command allows you to verify if your package meta-data
meet the minimum requirements to build a distribution.

To run it, just call it using your @code{setup.py} script. If something is
missing, @code{check} will display a warning.

Let’s take an example with a simple script:

@example
from distutils.core import setup

setup(name='foobar')
@end example

Running the @code{check} command will display some warnings:

@example
$ python setup.py check
running check
warning: check: missing required meta-data: version, url
warning: check: missing meta-data: either (author and author_email) or
         (maintainer and maintainer_email) should be supplied
@end example

If you use the reStructuredText syntax in the @code{long_description} field and
docutils@footnote{http://docutils.sourceforge.net}  is installed you can check if the syntax is fine with the
@code{check} command, using the @code{restructuredtext} option.

For example, if the @code{setup.py} script is changed like this:

@example
from distutils.core import setup

desc = """\
My description
==============

This is the description of the `@w{`}foobar`@w{`} package.
"""

setup(name='foobar', version='1', author='tarek',
    author_email='tarek@@ziade.org',
    url='http://example.com', long_description=desc)
@end example

Where the long description is broken, @code{check} will be able to detect it
by using the @code{docutils} parser:

@example
$ python setup.py check --restructuredtext
running check
warning: check: Title underline too short. (line 2)
warning: check: Could not finish the parsing.
@end example

@node Reading the metadata,,Checking a package,Distutils Examples
@anchor{distutils/examples reading-the-metadata}@anchor{4928}
@subsection Reading the metadata


The @ref{40db,,distutils.core.setup()} function provides a command-line interface
that allows you to query the metadata fields of a project through the
@code{setup.py} script of a given project:

@example
$ python setup.py --name
distribute
@end example

This call reads the @code{name} metadata by running the
@ref{40db,,distutils.core.setup()}  function. Although, when a source or binary
distribution is created with Distutils, the metadata fields are written
in a static file called @code{PKG-INFO}. When a Distutils-based project is
installed in Python, the @code{PKG-INFO} file is copied alongside the modules
and packages of the distribution under @code{NAME-VERSION-pyX.X.egg-info},
where @code{NAME} is the name of the project, @code{VERSION} its version as defined
in the Metadata, and @code{pyX.X} the major and minor version of Python like
@code{2.7} or @code{3.2}.

You can read back this static file, by using the
@code{distutils.dist.DistributionMetadata} class and its
@code{read_pkg_file()} method:

@example
>>> from distutils.dist import DistributionMetadata
>>> metadata = DistributionMetadata()
>>> metadata.read_pkg_file(open('distribute-0.6.8-py2.7.egg-info'))
>>> metadata.name
'distribute'
>>> metadata.version
'0.6.8'
>>> metadata.description
'Easily download, build, install, upgrade, and uninstall Python packages'
@end example

Notice that the class can also be instantiated with a metadata file path to
loads its values:

@example
>>> pkg_info_path = 'distribute-0.6.8-py2.7.egg-info'
>>> DistributionMetadata(pkg_info_path).name
'distribute'
@end example

@c % \section{Multiple extension modules}

@c % \label{multiple-ext}

@c % \section{Putting it all together}

@node Extending Distutils,Command Reference,Distutils Examples,Distributing Python Modules Legacy version
@anchor{distutils/extending doc}@anchor{4929}@anchor{distutils/extending docutils}@anchor{492a}@anchor{distutils/extending extending-distutils}@anchor{16ce}@anchor{distutils/extending id1}@anchor{492b}
@section Extending Distutils


@cartouche
@quotation Note 
This document is being retained solely until the @code{setuptools} documentation
at @indicateurl{https://setuptools.readthedocs.io/en/latest/setuptools.html}
independently covers all of the relevant information currently included here.
@end quotation
@end cartouche

Distutils can be extended in various ways.  Most extensions take the form of new
commands or replacements for existing commands.  New commands may be written to
support new types of platform-specific packaging, for example, while
replacements for existing commands may be made to modify details of how the
command operates on a package.

Most extensions of the distutils are made within @code{setup.py} scripts that
want to modify existing commands; many simply add a few file extensions that
should be copied into packages in addition to @code{.py} files as a
convenience.

Most distutils command implementations are subclasses of the
@ref{492c,,distutils.cmd.Command} class.  New commands may directly inherit from
@code{Command}, while replacements often derive from @code{Command}
indirectly, directly subclassing the command they are replacing.  Commands are
required to derive from @code{Command}.

@c % \section{Extending existing commands}

@c % \label{extend-existing}

@c % \section{Writing new commands}

@c % \label{new-commands}

@c % \XXX{Would an uninstall command be a good example here?}

@menu
* Integrating new commands:: 
* Adding new distribution types:: 

@end menu

@node Integrating new commands,Adding new distribution types,,Extending Distutils
@anchor{distutils/extending integrating-new-commands}@anchor{492d}
@subsection Integrating new commands


There are different ways to integrate new command implementations into
distutils.  The most difficult is to lobby for the inclusion of the new features
in distutils itself, and wait for (and require) a version of Python that
provides that support.  This is really hard for many reasons.

The most common, and possibly the most reasonable for most needs, is to include
the new implementations with your @code{setup.py} script, and cause the
@ref{40db,,distutils.core.setup()} function use them:

@example
from distutils.command.build_py import build_py as _build_py
from distutils.core import setup

class build_py(_build_py):
    """Specialized Python source builder."""

    # implement whatever needs to be different...

setup(cmdclass=@{'build_py': build_py@},
      ...)
@end example

This approach is most valuable if the new implementations must be used to use a
particular package, as everyone interested in the package will need to have the
new command implementation.

Beginning with Python 2.4, a third option is available, intended to allow new
commands to be added which can support existing @code{setup.py} scripts without
requiring modifications to the Python installation.  This is expected to allow
third-party extensions to provide support for additional packaging systems, but
the commands can be used for anything distutils commands can be used for.  A new
configuration option, @code{command_packages} (command-line option
@code{--command-packages}), can be used to specify additional packages to be
searched for modules implementing commands.  Like all distutils options, this
can be specified on the command line or in a configuration file.  This option
can only be set in the @code{[global]} section of a configuration file, or before
any commands on the command line.  If set in a configuration file, it can be
overridden from the command line; setting it to an empty string on the command
line causes the default to be used.  This should never be set in a configuration
file provided with a package.

This new option can be used to add any number of packages to the list of
packages searched for command implementations; multiple package names should be
separated by commas.  When not specified, the search is only performed in the
@ref{3c,,distutils.command} package.  When @code{setup.py} is run with the option
@code{--command-packages distcmds,buildcmds}, however, the packages
@ref{3c,,distutils.command}, @code{distcmds}, and @code{buildcmds} will be searched
in that order.  New commands are expected to be implemented in modules of the
same name as the command by classes sharing the same name.  Given the example
command line option above, the command @code{bdist_openpkg} could be
implemented by the class @code{distcmds.bdist_openpkg.bdist_openpkg} or
@code{buildcmds.bdist_openpkg.bdist_openpkg}.

@node Adding new distribution types,,Integrating new commands,Extending Distutils
@anchor{distutils/extending adding-new-distribution-types}@anchor{492e}
@subsection Adding new distribution types


Commands that create distributions (files in the @code{dist/} directory) need
to add @code{(command, filename)} pairs to @code{self.distribution.dist_files} so that
@code{upload} can upload it to PyPI.  The @emph{filename} in the pair contains no
path information, only the name of the file itself.  In dry-run mode, pairs
should still be added to represent what would have been created.

@node Command Reference,API Reference,Extending Distutils,Distributing Python Modules Legacy version
@anchor{distutils/commandref doc}@anchor{492f}@anchor{distutils/commandref command-reference}@anchor{4930}@anchor{distutils/commandref reference}@anchor{4931}
@section Command Reference


@cartouche
@quotation Note 
This document is being retained solely until the @code{setuptools} documentation
at @indicateurl{https://setuptools.readthedocs.io/en/latest/setuptools.html}
independently covers all of the relevant information currently included here.
@end quotation
@end cartouche

@c % \section{Building modules: the \protect\command{build} command family}

@c % \label{build-cmds}

@c % \subsubsection{\protect\command{build}}

@c % \label{build-cmd}

@c % \subsubsection{\protect\command{build\_py}}

@c % \label{build-py-cmd}

@c % \subsubsection{\protect\command{build\_ext}}

@c % \label{build-ext-cmd}

@c % \subsubsection{\protect\command{build\_clib}}

@c % \label{build-clib-cmd}

@menu
* Installing modules; the install command family: Installing modules the install command family. 
* Creating a source distribution; the sdist command: Creating a source distribution the sdist command. 

@end menu

@node Installing modules the install command family,Creating a source distribution the sdist command,,Command Reference
@anchor{distutils/commandref install-cmd}@anchor{4932}@anchor{distutils/commandref installing-modules-the-install-command-family}@anchor{4933}
@subsection Installing modules: the @code{install} command family


The install command ensures that the build commands have been run and then runs
the subcommands @code{install_lib}, @code{install_data} and
@code{install_scripts}.

@c % \subsubsection{\protect\command{install\_lib}}

@c % \label{install-lib-cmd}

@menu
* install_data:: 
* install_scripts:: 

@end menu

@node install_data,install_scripts,,Installing modules the install command family
@anchor{distutils/commandref install-data}@anchor{4934}@anchor{distutils/commandref install-data-cmd}@anchor{4935}
@subsubsection @code{install_data}


This command installs all data files provided with the distribution.

@node install_scripts,,install_data,Installing modules the install command family
@anchor{distutils/commandref install-scripts}@anchor{4936}@anchor{distutils/commandref install-scripts-cmd}@anchor{4937}
@subsubsection @code{install_scripts}


This command installs all (Python) scripts in the distribution.

@c % \subsection{Cleaning up: the \protect\command{clean} command}

@c % \label{clean-cmd}

@node Creating a source distribution the sdist command,,Installing modules the install command family,Command Reference
@anchor{distutils/commandref creating-a-source-distribution-the-sdist-command}@anchor{4938}@anchor{distutils/commandref sdist-cmd}@anchor{490e}
@subsection Creating a source distribution: the @code{sdist} command


@c XXX fragment moved down from above: needs context!

The manifest template commands are:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Command

@tab

Description

@item

@code{include pat1 pat2 ...}

@tab

include all files matching any of the listed
patterns

@item

@code{exclude pat1 pat2 ...}

@tab

exclude all files matching any of the listed
patterns

@item

@code{recursive-include dir pat1 pat2
...}

@tab

include all files under @emph{dir} matching any of
the listed patterns

@item

@code{recursive-exclude dir pat1 pat2
...}

@tab

exclude all files under @emph{dir} matching any of
the listed patterns

@item

@code{global-include pat1 pat2 ...}

@tab

include all files anywhere in the source tree
matching — & any of the listed patterns

@item

@code{global-exclude pat1 pat2 ...}

@tab

exclude all files anywhere in the source tree
matching — & any of the listed patterns

@item

@code{prune dir}

@tab

exclude all files under @emph{dir}

@item

@code{graft dir}

@tab

include all files under @emph{dir}

@end multitable


The patterns here are Unix-style “glob” patterns: @code{*} matches any sequence of
regular filename characters, @code{?} matches any single regular filename
character, and @code{[range]} matches any of the characters in @emph{range} (e.g.,
@code{a-z}, @code{a-zA-Z}, @code{a-f0-9_.}).  The definition of “regular filename
character” is platform-specific: on Unix it is anything except slash; on Windows
anything except backslash or colon.

@c XXX Windows support not there yet

@c % \section{Creating a built distribution: the

@c % \protect\command{bdist} command family}

@c % \label{bdist-cmds}

@c % \subsection{\protect\command{bdist}}

@c % \subsection{\protect\command{bdist\_dumb}}

@c % \subsection{\protect\command{bdist\_rpm}}

@node API Reference,,Command Reference,Distributing Python Modules Legacy version
@anchor{distutils/apiref doc}@anchor{4939}@anchor{distutils/apiref api-reference}@anchor{493a}@anchor{distutils/apiref id1}@anchor{493b}
@section API Reference



@subsubheading See also



@table @asis

@item New and changed setup.py arguments in setuptools@footnote{https://setuptools.readthedocs.io/en/latest/setuptools.html#new-and-changed-setup-keywords}

The @code{setuptools} project adds new capabilities to the @code{setup} function
and other APIs, makes the API consistent across different Python versions,
and is hence recommended over using @code{distutils} directly.
@end table


@cartouche
@quotation Note 
This document is being retained solely until the @code{setuptools} documentation
at @indicateurl{https://setuptools.readthedocs.io/en/latest/setuptools.html}
independently covers all of the relevant information currently included here.
@end quotation
@end cartouche

@menu
* distutils.core — Core Distutils functionality: distutils core — Core Distutils functionality. 
* distutils.ccompiler — CCompiler base class: distutils ccompiler — CCompiler base class. 
* distutils.unixccompiler — Unix C Compiler: distutils unixccompiler — Unix C Compiler. 
* distutils.msvccompiler — Microsoft Compiler: distutils msvccompiler — Microsoft Compiler. 
* distutils.bcppcompiler — Borland Compiler: distutils bcppcompiler — Borland Compiler. 
* distutils.cygwincompiler — Cygwin Compiler: distutils cygwincompiler — Cygwin Compiler. 
* distutils.archive_util — Archiving utilities: distutils archive_util — Archiving utilities. 
* distutils.dep_util — Dependency checking: distutils dep_util — Dependency checking. 
* distutils.dir_util — Directory tree operations: distutils dir_util — Directory tree operations. 
* distutils.file_util — Single file operations: distutils file_util — Single file operations. 
* distutils.util — Miscellaneous other utility functions: distutils util — Miscellaneous other utility functions. 
* distutils.dist — The Distribution class: distutils dist — The Distribution class. 
* distutils.extension — The Extension class: distutils extension — The Extension class. 
* distutils.debug — Distutils debug mode: distutils debug — Distutils debug mode. 
* distutils.errors — Distutils exceptions: distutils errors — Distutils exceptions. 
* distutils.fancy_getopt — Wrapper around the standard getopt module: distutils fancy_getopt — Wrapper around the standard getopt module. 
* distutils.filelist — The FileList class: distutils filelist — The FileList class. 
* distutils.log — Simple PEP 282-style logging: distutils log — Simple PEP 282-style logging. 
* distutils.spawn — Spawn a sub-process: distutils spawn — Spawn a sub-process. 
* distutils.sysconfig — System configuration information: distutils sysconfig — System configuration information. 
* distutils.text_file — The TextFile class: distutils text_file — The TextFile class. 
* distutils.version — Version number classes: distutils version — Version number classes. 
* distutils.cmd — Abstract base class for Distutils commands: distutils cmd — Abstract base class for Distutils commands. 
* Creating a new Distutils command:: 
* distutils.command — Individual Distutils commands: distutils command — Individual Distutils commands. 
* distutils.command.bdist — Build a binary installer: distutils command bdist — Build a binary installer. 
* distutils.command.bdist_packager — Abstract base class for packagers: distutils command bdist_packager — Abstract base class for packagers. 
* distutils.command.bdist_dumb — Build a “dumb” installer: distutils command bdist_dumb — Build a “dumb” installer. 
* distutils.command.bdist_rpm — Build a binary distribution as a Redhat RPM and SRPM: distutils command bdist_rpm — Build a binary distribution as a Redhat RPM and SRPM. 
* distutils.command.sdist — Build a source distribution: distutils command sdist — Build a source distribution. 
* distutils.command.build — Build all files of a package: distutils command build — Build all files of a package. 
* distutils.command.build_clib — Build any C libraries in a package: distutils command build_clib — Build any C libraries in a package. 
* distutils.command.build_ext — Build any extensions in a package: distutils command build_ext — Build any extensions in a package. 
* distutils.command.build_py — Build the .py/.pyc files of a package: distutils command build_py — Build the py/ pyc files of a package. 
* distutils.command.build_scripts — Build the scripts of a package: distutils command build_scripts — Build the scripts of a package. 
* distutils.command.clean — Clean a package build area: distutils command clean — Clean a package build area. 
* distutils.command.config — Perform package configuration: distutils command config — Perform package configuration. 
* distutils.command.install — Install a package: distutils command install — Install a package. 
* distutils.command.install_data — Install data files from a package: distutils command install_data — Install data files from a package. 
* distutils.command.install_headers — Install C/C++ header files from a package: distutils command install_headers — Install C/C++ header files from a package. 
* distutils.command.install_lib — Install library files from a package: distutils command install_lib — Install library files from a package. 
* distutils.command.install_scripts — Install script files from a package: distutils command install_scripts — Install script files from a package. 
* distutils.command.register — Register a module with the Python Package Index: distutils command register — Register a module with the Python Package Index. 
* distutils.command.check — Check the meta-data of a package: distutils command check — Check the meta-data of a package. 

@end menu

@node distutils core — Core Distutils functionality,distutils ccompiler — CCompiler base class,,API Reference
@anchor{distutils/apiref distutils-core-core-distutils-functionality}@anchor{493c}@anchor{distutils/apiref module-distutils core}@anchor{50}
@subsection @code{distutils.core} — Core Distutils functionality


@geindex module; distutils.core

The @ref{50,,distutils.core} module is the only module that needs to be installed
to use the Distutils. It provides the @ref{40db,,setup()} (which is called from the
setup script). Indirectly provides the  @code{distutils.dist.Distribution} and
@ref{492c,,distutils.cmd.Command} class.

@geindex setup() (in module distutils.core)
@anchor{distutils/apiref distutils core setup}@anchor{40db}
@deffn {Function} distutils.core.setup (arguments)

The basic do-everything function that does most everything you could ever ask
for from a Distutils method.

The setup function takes a large number of arguments. These are laid out in the
following table.


@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

argument name

@tab

value

@tab

type

@item

@emph{name}

@tab

The name of the package

@tab

a string

@item

@emph{version}

@tab

The version number of the
package; see
@ref{62,,distutils.version}

@tab

a string

@item

@emph{description}

@tab

A single line describing the
package

@tab

a string

@item

@emph{long_description}

@tab

Longer description of the
package

@tab

a string

@item

@emph{author}

@tab

The name of the package author

@tab

a string

@item

@emph{author_email}

@tab

The email address of the
package author

@tab

a string

@item

@emph{maintainer}

@tab

The name of the current
maintainer, if different from
the author. Note that if
the maintainer is provided,
distutils will use it as the
author in @code{PKG-INFO}

@tab

a string

@item

@emph{maintainer_email}

@tab

The email address of the
current maintainer, if
different from the author

@tab

a string

@item

@emph{url}

@tab

A URL for the package
(homepage)

@tab

a string

@item

@emph{download_url}

@tab

A URL to download the package

@tab

a string

@item

@emph{packages}

@tab

A list of Python packages that
distutils will manipulate

@tab

a list of strings

@item

@emph{py_modules}

@tab

A list of Python modules that
distutils will manipulate

@tab

a list of strings

@item

@emph{scripts}

@tab

A list of standalone script
files to be built and
installed

@tab

a list of strings

@item

@emph{ext_modules}

@tab

A list of Python extensions to
be built

@tab

a list of instances of
@ref{48f5,,distutils.core.Extension}

@item

@emph{classifiers}

@tab

A list of categories for the
package

@tab

a list of strings; valid classifiers are listed on PyPI@footnote{https://pypi.org/classifiers}.

@item

@emph{distclass}

@tab

the @ref{493d,,Distribution}
class to use

@tab

a subclass of
@ref{493d,,distutils.core.Distribution}

@item

@emph{script_name}

@tab

The name of the setup.py
script - defaults to
@code{sys.argv[0]}

@tab

a string

@item

@emph{script_args}

@tab

Arguments to supply to the
setup script

@tab

a list of strings

@item

@emph{options}

@tab

default options for the setup
script

@tab

a dictionary

@item

@emph{license}

@tab

The license for the package

@tab

a string

@item

@emph{keywords}

@tab

Descriptive meta-data, see
@geindex Python Enhancement Proposals; PEP 314
PEP 314@footnote{https://www.python.org/dev/peps/pep-0314}

@tab

a list of strings or a comma-separated string

@item

@emph{platforms}

@tab

@tab

a list of strings or a comma-separated string

@item

@emph{cmdclass}

@tab

A mapping of command names to
@ref{493e,,Command} subclasses

@tab

a dictionary

@item

@emph{data_files}

@tab

A list of data files to
install

@tab

a list

@item

@emph{package_dir}

@tab

A mapping of package to
directory names

@tab

a dictionary

@end multitable

@end deffn

@geindex run_setup() (in module distutils.core)
@anchor{distutils/apiref distutils core run_setup}@anchor{493f}
@deffn {Function} distutils.core.run_setup (script_name[, script_args=None, stop_after='run'])

Run a setup script in a somewhat controlled environment, and return  the
@code{distutils.dist.Distribution} instance that drives things.   This is
useful if you need to find out the distribution meta-data  (passed as keyword
args from @emph{script} to @ref{40db,,setup()}), or  the contents of the config files or
command-line.

@emph{script_name} is a file that will be read and run with @ref{34b,,exec()}.  @code{sys.argv[0]}
will be replaced with @emph{script} for the duration of the call.  @emph{script_args} is a
list of strings; if supplied, @code{sys.argv[1:]} will be replaced by @emph{script_args}
for the duration  of the call.

@emph{stop_after} tells @ref{40db,,setup()} when to stop processing; possible  values:


@multitable {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

value

@tab

description

@item

@emph{init}

@tab

Stop after the @ref{493d,,Distribution}
instance has been created  and populated
with the keyword arguments to @ref{40db,,setup()}

@item

@emph{config}

@tab

Stop after config files have been parsed
(and their data stored in the
@ref{493d,,Distribution} instance)

@item

@emph{commandline}

@tab

Stop after the command-line
(@code{sys.argv[1:]} or  @emph{script_args}) have
been parsed (and the data stored in the
@ref{493d,,Distribution} instance.)

@item

@emph{run}

@tab

Stop after all commands have been run (the
same as  if @ref{40db,,setup()} had been called
in the usual way). This is the default
value.

@end multitable

@end deffn

In addition, the @ref{50,,distutils.core} module exposed a number of  classes that
live elsewhere.


@itemize *

@item 
@code{Extension} from @ref{57,,distutils.extension}

@item 
@ref{492c,,Command} from @ref{3b,,distutils.cmd}

@item 
@code{Distribution} from @ref{55,,distutils.dist}
@end itemize

A short description of each of these follows, but see the relevant module for
the full reference.

@geindex Extension (class in distutils.core)
@anchor{distutils/apiref distutils core Extension}@anchor{48f5}
@deffn {Class} distutils.core.Extension

The Extension class describes a single C or C++ extension module in a setup
script. It accepts the following keyword arguments in its constructor:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

argument name

@tab

value

@tab

type

@item

@emph{name}

@tab

the full name of the
extension, including any
packages — ie. @emph{not} a
filename or pathname, but
Python dotted name

@tab

a string

@item

@emph{sources}

@tab

list of source filenames,
relative to the distribution
root (where the setup script
lives), in Unix form
(slash-separated) for
portability.
Source files may be C, C++,
SWIG (.i), platform-specific
resource files, or whatever
else is recognized by the
@code{build_ext} command
as source for a Python
extension.

@tab

a list of strings

@item

@emph{include_dirs}

@tab

list of directories to search
for C/C++ header files (in
Unix form for portability)

@tab

a list of strings

@item

@emph{define_macros}

@tab

list of macros to define; each
macro is defined using a
2-tuple @code{(name, value)},
where @emph{value} is
either the string to define it
to or @code{None} to define it
without a particular value
(equivalent of @code{#define FOO}
in source or @code{-DFOO}
on Unix C compiler command
line)

@tab

a list of tuples

@item

@emph{undef_macros}

@tab

list of macros to undefine
explicitly

@tab

a list of strings

@item

@emph{library_dirs}

@tab

list of directories to search
for C/C++ libraries at link
time

@tab

a list of strings

@item

@emph{libraries}

@tab

list of library names (not
filenames or paths) to link
against

@tab

a list of strings

@item

@emph{runtime_library_dirs}

@tab

list of directories to search
for C/C++ libraries at run
time (for shared extensions,
this is when the extension is
loaded)

@tab

a list of strings

@item

@emph{extra_objects}

@tab

list of extra files to link
with (eg. object files not
implied by ‘sources’, static
library that must be
explicitly specified, binary
resource files, etc.)

@tab

a list of strings

@item

@emph{extra_compile_args}

@tab

any extra platform- and
compiler-specific information
to use when compiling the
source files in ‘sources’. For
platforms and compilers where
a command line makes sense,
this is typically a list of
command-line arguments, but
for other platforms it could
be anything.

@tab

a list of strings

@item

@emph{extra_link_args}

@tab

any extra platform- and
compiler-specific information
to use when linking object
files together to create the
extension (or to create a new
static Python interpreter).
Similar interpretation as for
‘extra_compile_args’.

@tab

a list of strings

@item

@emph{export_symbols}

@tab

list of symbols to be exported
from a shared extension. Not
used on all platforms, and not
generally necessary for Python
extensions, which typically
export exactly one symbol:
@code{init} + extension_name.

@tab

a list of strings

@item

@emph{depends}

@tab

list of files that the
extension depends on

@tab

a list of strings

@item

@emph{language}

@tab

extension language (i.e.
@code{'c'}, @code{'c++'},
@code{'objc'}). Will be detected
from the source extensions if
not provided.

@tab

a string

@item

@emph{optional}

@tab

specifies that a build failure
in the extension should not
abort the build process, but
simply skip the extension.

@tab

a boolean

@end multitable



Changed in version 3.8: On Unix, C extensions are no longer linked to libpython except on
Android and Cygwin.

@end deffn

@geindex Distribution (class in distutils.core)
@anchor{distutils/apiref distutils core Distribution}@anchor{493d}
@deffn {Class} distutils.core.Distribution

A @ref{493d,,Distribution} describes how to build, install and package up a Python
software package.

See the @ref{40db,,setup()} function for a list of keyword arguments accepted  by the
Distribution constructor. @ref{40db,,setup()} creates a Distribution instance.


Changed in version 3.7: @ref{493d,,Distribution} now warns if @code{classifiers},
@code{keywords} and @code{platforms} fields are not specified as a list or
a string.

@end deffn

@geindex Command (class in distutils.core)
@anchor{distutils/apiref distutils core Command}@anchor{493e}
@deffn {Class} distutils.core.Command

A @ref{493e,,Command} class (or rather, an instance of one of its subclasses)
implement a single distutils command.
@end deffn

@node distutils ccompiler — CCompiler base class,distutils unixccompiler — Unix C Compiler,distutils core — Core Distutils functionality,API Reference
@anchor{distutils/apiref distutils-ccompiler-ccompiler-base-class}@anchor{4940}@anchor{distutils/apiref module-distutils ccompiler}@anchor{3a}
@subsection @code{distutils.ccompiler} — CCompiler base class


@geindex module; distutils.ccompiler

This module provides the abstract base class for the @ref{4941,,CCompiler}
classes.  A @ref{4941,,CCompiler} instance can be used for all the compile  and
link steps needed to build a single project. Methods are provided to  set
options for the compiler — macro definitions, include directories,  link path,
libraries and the like.

This module provides the following functions.

@geindex gen_lib_options() (in module distutils.ccompiler)
@anchor{distutils/apiref distutils ccompiler gen_lib_options}@anchor{4942}
@deffn {Function} distutils.ccompiler.gen_lib_options (compiler, library_dirs, runtime_library_dirs, libraries)

Generate linker options for searching library directories and linking with
specific libraries.  @emph{libraries} and @emph{library_dirs} are, respectively, lists of
library names (not filenames!) and search directories.  Returns a list of
command-line options suitable for use with some compiler (depending on the two
format strings passed in).
@end deffn

@geindex gen_preprocess_options() (in module distutils.ccompiler)
@anchor{distutils/apiref distutils ccompiler gen_preprocess_options}@anchor{4943}
@deffn {Function} distutils.ccompiler.gen_preprocess_options (macros, include_dirs)

Generate C pre-processor options (@code{-D}, @code{-U}, @code{-I}) as
used by at least two types of compilers: the typical Unix compiler and Visual
C++. @emph{macros} is the usual thing, a list of 1- or 2-tuples, where @code{(name,)}
means undefine (@code{-U}) macro @emph{name}, and @code{(name, value)} means define
(@code{-D}) macro @emph{name} to @emph{value}.  @emph{include_dirs} is just a list of
directory names to be added to the header file search path (@code{-I}).
Returns a list of command-line options suitable for either Unix compilers or
Visual C++.
@end deffn

@geindex get_default_compiler() (in module distutils.ccompiler)
@anchor{distutils/apiref distutils ccompiler get_default_compiler}@anchor{4944}
@deffn {Function} distutils.ccompiler.get_default_compiler (osname, platform)

Determine the default compiler to use for the given platform.

@emph{osname} should be one of the standard Python OS names (i.e. the ones returned
by @code{os.name}) and @emph{platform} the common value returned by @code{sys.platform} for
the platform in question.

The default values are @code{os.name} and @code{sys.platform} in case the parameters
are not given.
@end deffn

@geindex new_compiler() (in module distutils.ccompiler)
@anchor{distutils/apiref distutils ccompiler new_compiler}@anchor{4945}
@deffn {Function} distutils.ccompiler.new_compiler (plat=None, compiler=None, verbose=0, dry_run=0, force=0)

Factory function to generate an instance of some CCompiler subclass for the
supplied platform/compiler combination. @emph{plat} defaults to @code{os.name} (eg.
@code{'posix'}, @code{'nt'}), and @emph{compiler}  defaults to the default compiler for
that platform. Currently only @code{'posix'} and @code{'nt'} are supported, and the
default compilers are “traditional Unix interface” (@code{UnixCCompiler}
class) and Visual C++ (@code{MSVCCompiler} class).  Note that it’s perfectly
possible to ask for a Unix compiler object under Windows, and a Microsoft
compiler object under Unix—if you supply a value for @emph{compiler}, @emph{plat} is
ignored.

@c % Is the posix/nt only thing still true? macOS seems to work, and

@c % returns a UnixCCompiler instance. How to document this... hmm.
@end deffn

@geindex show_compilers() (in module distutils.ccompiler)
@anchor{distutils/apiref distutils ccompiler show_compilers}@anchor{4946}
@deffn {Function} distutils.ccompiler.show_compilers ()

Print list of available compilers (used by the @code{--help-compiler} options
to @code{build}, @code{build_ext}, @code{build_clib}).
@end deffn

@geindex CCompiler (class in distutils.ccompiler)
@anchor{distutils/apiref distutils ccompiler CCompiler}@anchor{4941}
@deffn {Class} distutils.ccompiler.CCompiler ([verbose=0, dry_run=0, force=0])

The abstract base class @ref{4941,,CCompiler} defines the interface that  must be
implemented by real compiler classes.  The class also has  some utility methods
used by several compiler classes.

The basic idea behind a compiler abstraction class is that each instance can be
used for all the compile/link steps in building a single project.  Thus,
attributes common to all of those compile and link steps — include
directories, macros to define, libraries to link against, etc. — are
attributes of the compiler instance.  To allow for variability in how individual
files are treated, most of those attributes may be varied on a per-compilation
or per-link basis.

The constructor for each subclass creates an instance of the Compiler object.
Flags are @emph{verbose} (show verbose output), @emph{dry_run} (don’t actually execute the
steps) and @emph{force} (rebuild everything, regardless of dependencies). All of
these flags default to @code{0} (off). Note that you probably don’t want to
instantiate @ref{4941,,CCompiler} or one of its subclasses directly - use the
@code{distutils.CCompiler.new_compiler()} factory function instead.

The following methods allow you to manually alter compiler options for  the
instance of the Compiler class.

@geindex add_include_dir() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler add_include_dir}@anchor{4947}
@deffn {Method} add_include_dir (dir)

Add @emph{dir} to the list of directories that will be searched for header files.
The compiler is instructed to search directories in the order in which they are
supplied by successive calls to @ref{4947,,add_include_dir()}.
@end deffn

@geindex set_include_dirs() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler set_include_dirs}@anchor{4948}
@deffn {Method} set_include_dirs (dirs)

Set the list of directories that will be searched to @emph{dirs} (a list of strings).
Overrides any preceding calls to @ref{4947,,add_include_dir()}; subsequent calls to
@ref{4947,,add_include_dir()} add to the list passed to @ref{4948,,set_include_dirs()}.
This does not affect any list of standard include directories that the compiler
may search by default.
@end deffn

@geindex add_library() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler add_library}@anchor{4949}
@deffn {Method} add_library (libname)

Add @emph{libname} to the list of libraries that will be included in all links driven
by this compiler object.  Note that @emph{libname} should *not* be the name of a
file containing a library, but the name of the library itself: the actual
filename will be inferred by the linker, the compiler, or the compiler class
(depending on the platform).

The linker will be instructed to link against libraries in the order they were
supplied to @ref{4949,,add_library()} and/or @ref{494a,,set_libraries()}.  It is perfectly
valid to duplicate library names; the linker will be instructed to link against
libraries as many times as they are mentioned.
@end deffn

@geindex set_libraries() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler set_libraries}@anchor{494a}
@deffn {Method} set_libraries (libnames)

Set the list of libraries to be included in all links driven by this compiler
object to @emph{libnames} (a list of strings).  This does not affect any standard
system libraries that the linker may include by default.
@end deffn

@geindex add_library_dir() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler add_library_dir}@anchor{494b}
@deffn {Method} add_library_dir (dir)

Add @emph{dir} to the list of directories that will be searched for libraries
specified to @ref{4949,,add_library()} and @ref{494a,,set_libraries()}.  The linker will be
instructed to search for libraries in the order they are supplied to
@ref{494b,,add_library_dir()} and/or @ref{494c,,set_library_dirs()}.
@end deffn

@geindex set_library_dirs() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler set_library_dirs}@anchor{494c}
@deffn {Method} set_library_dirs (dirs)

Set the list of library search directories to @emph{dirs} (a list of strings).  This
does not affect any standard library search path that the linker may search by
default.
@end deffn

@geindex add_runtime_library_dir() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler add_runtime_library_dir}@anchor{494d}
@deffn {Method} add_runtime_library_dir (dir)

Add @emph{dir} to the list of directories that will be searched for shared libraries
at runtime.
@end deffn

@geindex set_runtime_library_dirs() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler set_runtime_library_dirs}@anchor{494e}
@deffn {Method} set_runtime_library_dirs (dirs)

Set the list of directories to search for shared libraries at runtime to @emph{dirs}
(a list of strings).  This does not affect any standard search path that the
runtime linker may search by default.
@end deffn

@geindex define_macro() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler define_macro}@anchor{494f}
@deffn {Method} define_macro (name[, value=None])

Define a preprocessor macro for all compilations driven by this compiler object.
The optional parameter @emph{value} should be a string; if it is not supplied, then
the macro will be defined without an explicit value and the exact outcome
depends on the compiler used.

@c XXX true? does ANSI say anything about this?
@end deffn

@geindex undefine_macro() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler undefine_macro}@anchor{4950}
@deffn {Method} undefine_macro (name)

Undefine a preprocessor macro for all compilations driven by this compiler
object.  If the same macro is defined by @ref{494f,,define_macro()} and
undefined by @ref{4950,,undefine_macro()} the last call takes precedence
(including multiple redefinitions or undefinitions).  If the macro is
redefined/undefined on a per-compilation basis (ie. in the call to
@ref{4e2,,compile()}), then that takes precedence.
@end deffn

@geindex add_link_object() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler add_link_object}@anchor{4951}
@deffn {Method} add_link_object (object)

Add @emph{object} to the list of object files (or analogues, such as explicitly named
library files or the output of “resource compilers”) to be included in every
link driven by this compiler object.
@end deffn

@geindex set_link_objects() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler set_link_objects}@anchor{4952}
@deffn {Method} set_link_objects (objects)

Set the list of object files (or analogues) to be included in every link to
@emph{objects}.  This does not affect any standard object files that the linker may
include by default (such as system libraries).
@end deffn

The following methods implement methods for autodetection of compiler  options,
providing some functionality similar to GNU @code{autoconf}.

@geindex detect_language() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler detect_language}@anchor{4953}
@deffn {Method} detect_language (sources)

Detect the language of a given file, or list of files. Uses the  instance
attributes @code{language_map} (a dictionary), and  @code{language_order} (a
list) to do the job.
@end deffn

@geindex find_library_file() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler find_library_file}@anchor{4954}
@deffn {Method} find_library_file (dirs, lib[, debug=0])

Search the specified list of directories for a static or shared library file
@emph{lib} and return the full path to that file.  If @emph{debug} is true, look for a
debugging version (if that makes sense on the current platform).  Return
@code{None} if @emph{lib} wasn’t found in any of the specified directories.
@end deffn

@geindex has_function() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler has_function}@anchor{4955}
@deffn {Method} has_function (funcname[, includes=None, include_dirs=None, libraries=None, library_dirs=None])

Return a boolean indicating whether @emph{funcname} is supported on the current
platform.  The optional arguments can be used to augment the compilation
environment by providing additional include files and paths and libraries and
paths.
@end deffn

@geindex library_dir_option() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler library_dir_option}@anchor{4956}
@deffn {Method} library_dir_option (dir)

Return the compiler option to add @emph{dir} to the list of directories searched for
libraries.
@end deffn

@geindex library_option() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler library_option}@anchor{4957}
@deffn {Method} library_option (lib)

Return the compiler option to add @emph{lib} to the list of libraries linked into the
shared library or executable.
@end deffn

@geindex runtime_library_dir_option() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler runtime_library_dir_option}@anchor{4958}
@deffn {Method} runtime_library_dir_option (dir)

Return the compiler option to add @emph{dir} to the list of directories searched for
runtime libraries.
@end deffn

@geindex set_executables() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler set_executables}@anchor{4959}
@deffn {Method} set_executables (**args)

Define the executables (and options for them) that will be run to perform the
various stages of compilation.  The exact set of executables that may be
specified here depends on the compiler class (via the ‘executables’ class
attribute), but most will have:


@multitable {xxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

attribute

@tab

description

@item

@emph{compiler}

@tab

the C/C++ compiler

@item

@emph{linker_so}

@tab

linker used to create shared objects and
libraries

@item

@emph{linker_exe}

@tab

linker used to create binary executables

@item

@emph{archiver}

@tab

static library creator

@end multitable


On platforms with a command-line (Unix, DOS/Windows), each of these is a string
that will be split into executable name and (optional) list of arguments.
(Splitting the string is done similarly to how Unix shells operate: words are
delimited by spaces, but quotes and backslashes can override this.  See
@ref{495a,,distutils.util.split_quoted()}.)
@end deffn

The following methods invoke stages in the build process.

@geindex compile() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler compile}@anchor{495b}
@deffn {Method} compile (sources[, output_dir=None, macros=None, include_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, depends=None])

Compile one or more source files. Generates object files (e.g.  transforms a
@code{.c} file to a @code{.o} file.)

@emph{sources} must be a list of filenames, most likely C/C++ files, but in reality
anything that can be handled by a particular compiler and compiler class (eg.
@code{MSVCCompiler} can handle resource files in @emph{sources}).  Return a list of
object filenames, one per source filename in @emph{sources}.  Depending on the
implementation, not all source files will necessarily be compiled, but all
corresponding object filenames will be returned.

If @emph{output_dir} is given, object files will be put under it, while retaining
their original path component.  That is, @code{foo/bar.c} normally compiles to
@code{foo/bar.o} (for a Unix implementation); if @emph{output_dir} is @emph{build}, then
it would compile to @code{build/foo/bar.o}.

@emph{macros}, if given, must be a list of macro definitions.  A macro definition is
either a @code{(name, value)} 2-tuple or a @code{(name,)} 1-tuple. The former defines
a macro; if the value is @code{None}, the macro is defined without an explicit
value.  The 1-tuple case undefines a macro.  Later
definitions/redefinitions/undefinitions take precedence.

@emph{include_dirs}, if given, must be a list of strings, the directories to add to
the default include file search path for this compilation only.

@emph{debug} is a boolean; if true, the compiler will be instructed to output debug
symbols in (or alongside) the object file(s).

@emph{extra_preargs} and @emph{extra_postargs} are implementation-dependent. On platforms
that have the notion of a command-line (e.g. Unix, DOS/Windows), they are most
likely lists of strings: extra command-line arguments to prepend/append to the
compiler command line.  On other platforms, consult the implementation class
documentation.  In any event, they are intended as an escape hatch for those
occasions when the abstract compiler framework doesn’t cut the mustard.

@emph{depends}, if given, is a list of filenames that all targets depend on.  If a
source file is older than any file in depends, then the source file will be
recompiled.  This supports dependency tracking, but only at a coarse
granularity.

Raises @code{CompileError} on failure.
@end deffn

@geindex create_static_lib() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler create_static_lib}@anchor{495c}
@deffn {Method} create_static_lib (objects, output_libname[, output_dir=None, debug=0, target_lang=None])

Link a bunch of stuff together to create a static library file. The “bunch of
stuff” consists of the list of object files supplied as @emph{objects}, the extra
object files supplied to @ref{4951,,add_link_object()} and/or
@ref{4952,,set_link_objects()}, the libraries supplied to @ref{4949,,add_library()} and/or
@ref{494a,,set_libraries()}, and the libraries supplied as @emph{libraries} (if any).

@emph{output_libname} should be a library name, not a filename; the filename will be
inferred from the library name.  @emph{output_dir} is the directory where the library
file will be put.

@c XXX defaults to what?

@emph{debug} is a boolean; if true, debugging information will be included in the
library (note that on most platforms, it is the compile step where this matters:
the @emph{debug} flag is included here just for consistency).

@emph{target_lang} is the target language for which the given objects are being
compiled. This allows specific linkage time treatment of certain languages.

Raises @code{LibError} on failure.
@end deffn

@geindex link() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler link}@anchor{495d}
@deffn {Method} link (target_desc, objects, output_filename[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None])

Link a bunch of stuff together to create an executable or shared library file.

The “bunch of stuff” consists of the list of object files supplied as @emph{objects}.
@emph{output_filename} should be a filename.  If @emph{output_dir} is supplied,
@emph{output_filename} is relative to it (i.e. @emph{output_filename} can provide
directory components if needed).

@emph{libraries} is a list of libraries to link against.  These are library names,
not filenames, since they’re translated into filenames in a platform-specific
way (eg. @emph{foo} becomes @code{libfoo.a} on Unix and @code{foo.lib} on
DOS/Windows).  However, they can include a directory component, which means the
linker will look in that specific directory rather than searching all the normal
locations.

@emph{library_dirs}, if supplied, should be a list of directories to search for
libraries that were specified as bare library names (ie. no directory
component).  These are on top of the system default and those supplied to
@ref{494b,,add_library_dir()} and/or @ref{494c,,set_library_dirs()}.  @emph{runtime_library_dirs}
is a list of directories that will be embedded into the shared library and used
to search for other shared libraries that *it* depends on at run-time.  (This
may only be relevant on Unix.)

@emph{export_symbols} is a list of symbols that the shared library will export.
(This appears to be relevant only on Windows.)

@emph{debug} is as for @ref{4e2,,compile()} and @ref{495c,,create_static_lib()},  with the
slight distinction that it actually matters on most platforms (as opposed to
@ref{495c,,create_static_lib()}, which includes a @emph{debug} flag mostly for form’s
sake).

@emph{extra_preargs} and @emph{extra_postargs} are as for @ref{4e2,,compile()}  (except of
course that they supply command-line arguments for the particular linker being
used).

@emph{target_lang} is the target language for which the given objects are being
compiled. This allows specific linkage time treatment of certain languages.

Raises @code{LinkError} on failure.
@end deffn

@geindex link_executable() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler link_executable}@anchor{495e}
@deffn {Method} link_executable (objects, output_progname[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, target_lang=None])

Link an executable.  @emph{output_progname} is the name of the file executable, while
@emph{objects} are a list of object filenames to link in. Other arguments  are as for
the @ref{495d,,link()} method.
@end deffn

@geindex link_shared_lib() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler link_shared_lib}@anchor{495f}
@deffn {Method} link_shared_lib (objects, output_libname[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None])

Link a shared library. @emph{output_libname} is the name of the output  library,
while @emph{objects} is a list of object filenames to link in.  Other arguments are
as for the @ref{495d,,link()} method.
@end deffn

@geindex link_shared_object() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler link_shared_object}@anchor{4960}
@deffn {Method} link_shared_object (objects, output_filename[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None])

Link a shared object. @emph{output_filename} is the name of the shared object that
will be created, while @emph{objects} is a list of object filenames  to link in.
Other arguments are as for the @ref{495d,,link()} method.
@end deffn

@geindex preprocess() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler preprocess}@anchor{4961}
@deffn {Method} preprocess (source[, output_file=None, macros=None, include_dirs=None, extra_preargs=None, extra_postargs=None])

Preprocess a single C/C++ source file, named in @emph{source}. Output will be written
to file named @emph{output_file}, or @emph{stdout} if @emph{output_file} not supplied.
@emph{macros} is a list of macro definitions as for @ref{4e2,,compile()}, which will
augment the macros set with @ref{494f,,define_macro()} and @ref{4950,,undefine_macro()}.
@emph{include_dirs} is a list of directory names that will be added to the  default
list, in the same way as @ref{4947,,add_include_dir()}.

Raises @code{PreprocessError} on failure.
@end deffn

The following utility methods are defined by the @ref{4941,,CCompiler} class, for
use by the various concrete subclasses.

@geindex executable_filename() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler executable_filename}@anchor{4962}
@deffn {Method} executable_filename (basename[, strip_dir=0, output_dir='@w{'}])

Returns the filename of the executable for the given @emph{basename}.  Typically for
non-Windows platforms this is the same as the basename,  while Windows will get
a @code{.exe} added.
@end deffn

@geindex library_filename() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler library_filename}@anchor{4963}
@deffn {Method} library_filename (libname[, lib_type='static', strip_dir=0, output_dir='@w{'}])

Returns the filename for the given library name on the current platform. On Unix
a library with @emph{lib_type} of @code{'static'} will typically  be of the form
@code{liblibname.a}, while a @emph{lib_type} of @code{'dynamic'}  will be of the form
@code{liblibname.so}.
@end deffn

@geindex object_filenames() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler object_filenames}@anchor{4964}
@deffn {Method} object_filenames (source_filenames[, strip_dir=0, output_dir='@w{'}])

Returns the name of the object files for the given source files.
@emph{source_filenames} should be a list of filenames.
@end deffn

@geindex shared_object_filename() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler shared_object_filename}@anchor{4965}
@deffn {Method} shared_object_filename (basename[, strip_dir=0, output_dir='@w{'}])

Returns the name of a shared object file for the given file name @emph{basename}.
@end deffn

@geindex execute() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler execute}@anchor{4966}
@deffn {Method} execute (func, args[, msg=None, level=1])

Invokes @ref{4967,,distutils.util.execute()}. This method invokes a  Python function
@emph{func} with the given arguments @emph{args}, after  logging and taking into account
the @emph{dry_run} flag.
@end deffn

@geindex spawn() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler spawn}@anchor{4968}
@deffn {Method} spawn (cmd)

Invokes @code{distutils.util.spawn()}. This invokes an external  process to run
the given command.
@end deffn

@geindex mkpath() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler mkpath}@anchor{4969}
@deffn {Method} mkpath (name[, mode=511])

Invokes @ref{496a,,distutils.dir_util.mkpath()}. This creates a directory  and any
missing ancestor directories.
@end deffn

@geindex move_file() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler move_file}@anchor{496b}
@deffn {Method} move_file (src, dst)

Invokes @ref{496c,,distutils.file_util.move_file()}. Renames @emph{src} to  @emph{dst}.
@end deffn

@geindex announce() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler announce}@anchor{496d}
@deffn {Method} announce (msg[, level=1])

Write a message using @code{distutils.log.debug()}.
@end deffn

@geindex warn() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler warn}@anchor{496e}
@deffn {Method} warn (msg)

Write a warning message @emph{msg} to standard error.
@end deffn

@geindex debug_print() (distutils.ccompiler.CCompiler method)
@anchor{distutils/apiref distutils ccompiler CCompiler debug_print}@anchor{496f}
@deffn {Method} debug_print (msg)

If the @emph{debug} flag is set on this @ref{4941,,CCompiler} instance, print  @emph{msg} to
standard output, otherwise do nothing.
@end deffn
@end deffn

@c % \subsection{Compiler-specific modules}

@c %

@c % The following modules implement concrete subclasses of the abstract

@c % \class{CCompiler} class. They should not be instantiated directly, but should

@c % be created using \function{distutils.ccompiler.new_compiler()} factory

@c % function.

@node distutils unixccompiler — Unix C Compiler,distutils msvccompiler — Microsoft Compiler,distutils ccompiler — CCompiler base class,API Reference
@anchor{distutils/apiref distutils-unixccompiler-unix-c-compiler}@anchor{4970}@anchor{distutils/apiref module-distutils unixccompiler}@anchor{60}
@subsection @code{distutils.unixccompiler} — Unix C Compiler


@geindex module; distutils.unixccompiler

This module provides the @code{UnixCCompiler} class, a subclass of
@code{CCompiler} that handles the typical Unix-style command-line  C compiler:


@itemize *

@item 
macros defined with @code{-Dname[=value]}

@item 
macros undefined with @code{-Uname}

@item 
include search directories specified with @code{-Idir}

@item 
libraries specified with @code{-llib}

@item 
library search directories specified with @code{-Ldir}

@item 
compile handled by @code{cc} (or similar) executable with @code{-c}
option: compiles @code{.c} to @code{.o}

@item 
link static library handled by @code{ar} command (possibly with
@code{ranlib})

@item 
link shared library handled by @code{cc} @code{-shared}
@end itemize

@node distutils msvccompiler — Microsoft Compiler,distutils bcppcompiler — Borland Compiler,distutils unixccompiler — Unix C Compiler,API Reference
@anchor{distutils/apiref distutils-msvccompiler-microsoft-compiler}@anchor{4971}@anchor{distutils/apiref module-distutils msvccompiler}@anchor{5c}
@subsection @code{distutils.msvccompiler} — Microsoft Compiler


@geindex module; distutils.msvccompiler

@c XXX: This is *waaaaay* out of date!

This module provides @code{MSVCCompiler}, an implementation of the abstract
@code{CCompiler} class for Microsoft Visual Studio. Typically, extension
modules need to be compiled with the same compiler that was used to compile
Python. For Python 2.3 and earlier, the compiler was Visual Studio 6. For Python
2.4 and 2.5, the compiler is Visual Studio .NET 2003.

@code{MSVCCompiler} will normally choose the right compiler, linker etc. on
its own. To override this choice, the environment variables @emph{DISTUTILS_USE_SDK}
and @emph{MSSdk} must be both set. @emph{MSSdk} indicates that the current environment has
been setup by the SDK’s @code{SetEnv.Cmd} script, or that the environment variables
had been registered when the SDK was installed; @emph{DISTUTILS_USE_SDK} indicates
that the distutils user has made an explicit choice to override the compiler
selection by @code{MSVCCompiler}.

@node distutils bcppcompiler — Borland Compiler,distutils cygwincompiler — Cygwin Compiler,distutils msvccompiler — Microsoft Compiler,API Reference
@anchor{distutils/apiref distutils-bcppcompiler-borland-compiler}@anchor{4972}@anchor{distutils/apiref module-distutils bcppcompiler}@anchor{39}
@subsection @code{distutils.bcppcompiler} — Borland Compiler


@geindex module; distutils.bcppcompiler

This module provides @code{BorlandCCompiler}, a subclass of the abstract
@code{CCompiler} class for the Borland C++ compiler.

@node distutils cygwincompiler — Cygwin Compiler,distutils archive_util — Archiving utilities,distutils bcppcompiler — Borland Compiler,API Reference
@anchor{distutils/apiref distutils-cygwincompiler-cygwin-compiler}@anchor{4973}@anchor{distutils/apiref module-distutils cygwinccompiler}@anchor{51}
@subsection @code{distutils.cygwincompiler} — Cygwin Compiler


@geindex module; distutils.cygwinccompiler

This module provides the @code{CygwinCCompiler} class, a subclass of
@code{UnixCCompiler} that handles the Cygwin port of the GNU C compiler to
Windows.  It also contains the Mingw32CCompiler class which handles the mingw32
port of GCC (same as cygwin in no-cygwin mode).

@node distutils archive_util — Archiving utilities,distutils dep_util — Dependency checking,distutils cygwincompiler — Cygwin Compiler,API Reference
@anchor{distutils/apiref distutils-archive-util-archiving-utilities}@anchor{4974}@anchor{distutils/apiref module-distutils archive_util}@anchor{38}
@subsection @code{distutils.archive_util} —  Archiving utilities


@geindex module; distutils.archive_util

This module provides a few functions for creating archive files, such as
tarballs or zipfiles.

@geindex make_archive() (in module distutils.archive_util)
@anchor{distutils/apiref distutils archive_util make_archive}@anchor{4975}
@deffn {Function} distutils.archive_util.make_archive (base_name, format[, root_dir=None, base_dir=None, verbose=0, dry_run=0])

Create an archive file (eg. @code{zip} or @code{tar}).  @emph{base_name}  is the name of
the file to create, minus any format-specific extension;  @emph{format} is the
archive format: one of @code{zip}, @code{tar}, @code{gztar}, @code{bztar}, @code{xztar}, or
@code{ztar}. @emph{root_dir} is a directory that will be the root directory of the
archive; ie. we typically @code{chdir} into @emph{root_dir} before  creating the
archive.  @emph{base_dir} is the directory where we start  archiving from; ie.
@emph{base_dir} will be the common prefix of all files and directories in the
archive.  @emph{root_dir} and @emph{base_dir} both default to the current directory.
Returns the name of the archive file.


Changed in version 3.5: Added support for the @code{xztar} format.

@end deffn

@geindex make_tarball() (in module distutils.archive_util)
@anchor{distutils/apiref distutils archive_util make_tarball}@anchor{4976}
@deffn {Function} distutils.archive_util.make_tarball (base_name, base_dir[, compress='gzip', verbose=0, dry_run=0])

‘Create an (optional compressed) archive as a tar file from all files in and
under @emph{base_dir}. @emph{compress} must be @code{'gzip'} (the default),
@code{'bzip2'}, @code{'xz'}, @code{'compress'}, or @code{None}.  For the @code{'compress'}
method the compression utility named by @code{compress} must be on the
default program search path, so this is probably Unix-specific.  The output
tar file will be named @code{base_dir.tar}, possibly plus the appropriate
compression extension (@code{.gz}, @code{.bz2}, @code{.xz} or @code{.Z}).  Return the
output filename.


Changed in version 3.5: Added support for the @code{xz} compression.

@end deffn

@geindex make_zipfile() (in module distutils.archive_util)
@anchor{distutils/apiref distutils archive_util make_zipfile}@anchor{4977}
@deffn {Function} distutils.archive_util.make_zipfile (base_name, base_dir[, verbose=0, dry_run=0])

Create a zip file from all files in and under @emph{base_dir}.  The output zip file
will be named @emph{base_name} + @code{.zip}.  Uses either the  @ref{149,,zipfile} Python
module (if available) or the InfoZIP @code{zip}  utility (if installed and
found on the default search path).  If neither  tool is available, raises
@code{DistutilsExecError}.   Returns the name of the output zip file.
@end deffn

@node distutils dep_util — Dependency checking,distutils dir_util — Directory tree operations,distutils archive_util — Archiving utilities,API Reference
@anchor{distutils/apiref distutils-dep-util-dependency-checking}@anchor{4978}@anchor{distutils/apiref module-distutils dep_util}@anchor{53}
@subsection @code{distutils.dep_util} — Dependency checking


@geindex module; distutils.dep_util

This module provides functions for performing simple, timestamp-based
dependency of files and groups of files; also, functions based entirely  on such
timestamp dependency analysis.

@geindex newer() (in module distutils.dep_util)
@anchor{distutils/apiref distutils dep_util newer}@anchor{4979}
@deffn {Function} distutils.dep_util.newer (source, target)

Return true if @emph{source} exists and is more recently modified than @emph{target}, or
if @emph{source} exists and @emph{target} doesn’t. Return false if both exist and @emph{target}
is the same age or newer  than @emph{source}. Raise @code{DistutilsFileError} if
@emph{source} does not exist.
@end deffn

@geindex newer_pairwise() (in module distutils.dep_util)
@anchor{distutils/apiref distutils dep_util newer_pairwise}@anchor{497a}
@deffn {Function} distutils.dep_util.newer_pairwise (sources, targets)

Walk two filename lists in parallel, testing if each source is newer than its
corresponding target.  Return a pair of lists (@emph{sources}, @emph{targets}) where
source is newer than target, according to the semantics of @ref{4979,,newer()}.

@c % % equivalent to a listcomp...
@end deffn

@geindex newer_group() (in module distutils.dep_util)
@anchor{distutils/apiref distutils dep_util newer_group}@anchor{497b}
@deffn {Function} distutils.dep_util.newer_group (sources, target[, missing='error'])

Return true if @emph{target} is out-of-date with respect to any file listed in
@emph{sources}.  In other words, if @emph{target} exists and is newer than every file in
@emph{sources}, return false; otherwise return true. @emph{missing} controls what we do
when a source file is missing; the default (@code{'error'}) is to blow up with an
@ref{2a1,,OSError} from  inside @ref{516,,os.stat()}; if it is @code{'ignore'}, we silently
drop any missing source files; if it is @code{'newer'}, any missing source files
make us assume that @emph{target} is out-of-date (this is handy in “dry-run” mode:
it’ll make you pretend to carry out commands that wouldn’t work because inputs
are missing, but that doesn’t matter because you’re not actually going to run
the commands).
@end deffn

@node distutils dir_util — Directory tree operations,distutils file_util — Single file operations,distutils dep_util — Dependency checking,API Reference
@anchor{distutils/apiref distutils-dir-util-directory-tree-operations}@anchor{497c}@anchor{distutils/apiref module-distutils dir_util}@anchor{54}
@subsection @code{distutils.dir_util} — Directory tree operations


@geindex module; distutils.dir_util

This module provides functions for operating on directories and trees of
directories.

@geindex mkpath() (in module distutils.dir_util)
@anchor{distutils/apiref distutils dir_util mkpath}@anchor{496a}
@deffn {Function} distutils.dir_util.mkpath (name[, mode=0o777, verbose=0, dry_run=0])

Create a directory and any missing ancestor directories.  If the directory
already exists (or if @emph{name} is the empty string, which means the current
directory, which of course exists), then do nothing.  Raise
@code{DistutilsFileError} if unable to create some directory along the way (eg.
some sub-path exists, but is a file rather than a directory).  If @emph{verbose} is
true, print a one-line summary of each mkdir to stdout.  Return the list of
directories actually created.
@end deffn

@geindex create_tree() (in module distutils.dir_util)
@anchor{distutils/apiref distutils dir_util create_tree}@anchor{497d}
@deffn {Function} distutils.dir_util.create_tree (base_dir, files[, mode=0o777, verbose=0, dry_run=0])

Create all the empty directories under @emph{base_dir} needed to put @emph{files} there.
@emph{base_dir} is just the name of a directory which doesn’t necessarily exist
yet; @emph{files} is a list of filenames to be interpreted relative to @emph{base_dir}.
@emph{base_dir} + the directory portion of every file in @emph{files} will be created if
it doesn’t already exist.  @emph{mode}, @emph{verbose} and @emph{dry_run} flags  are as for
@ref{496a,,mkpath()}.
@end deffn

@geindex copy_tree() (in module distutils.dir_util)
@anchor{distutils/apiref distutils dir_util copy_tree}@anchor{497e}
@deffn {Function} distutils.dir_util.copy_tree (src, dst[, preserve_mode=1, preserve_times=1, preserve_symlinks=0, update=0, verbose=0, dry_run=0])

Copy an entire directory tree @emph{src} to a new location @emph{dst}.  Both @emph{src} and
@emph{dst} must be directory names.  If @emph{src} is not a directory, raise
@code{DistutilsFileError}.  If @emph{dst} does  not exist, it is created with
@ref{496a,,mkpath()}.  The end result of the  copy is that every file in @emph{src} is
copied to @emph{dst}, and  directories under @emph{src} are recursively copied to @emph{dst}.
Return the list of files that were copied or might have been copied, using their
output name. The return value is unaffected by @emph{update} or @emph{dry_run}: it is
simply the list of all files under @emph{src}, with the names changed to be under
@emph{dst}.

@emph{preserve_mode} and @emph{preserve_times} are the same as for
@ref{497f,,distutils.file_util.copy_file()}; note that they only apply to
regular files, not to
directories.  If @emph{preserve_symlinks} is true, symlinks will be copied as
symlinks (on platforms that support them!); otherwise (the default), the
destination of the symlink will be copied.  @emph{update} and @emph{verbose} are the same
as for @code{copy_file()}.

Files in @emph{src} that begin with @code{.nfs} are skipped (more information on
these files is available in answer D2 of the NFS FAQ page@footnote{http://nfs.sourceforge.net/#section_d}).


Changed in version 3.3.1: NFS files are ignored.

@end deffn

@geindex remove_tree() (in module distutils.dir_util)
@anchor{distutils/apiref distutils dir_util remove_tree}@anchor{4980}
@deffn {Function} distutils.dir_util.remove_tree (directory[, verbose=0, dry_run=0])

Recursively remove @emph{directory} and all files and directories underneath it. Any
errors are ignored (apart from being reported to @code{sys.stdout} if @emph{verbose} is
true).
@end deffn

@node distutils file_util — Single file operations,distutils util — Miscellaneous other utility functions,distutils dir_util — Directory tree operations,API Reference
@anchor{distutils/apiref distutils-file-util-single-file-operations}@anchor{4981}@anchor{distutils/apiref module-distutils file_util}@anchor{59}
@subsection @code{distutils.file_util} — Single file operations


@geindex module; distutils.file_util

This module contains some utility functions for operating on individual files.

@geindex copy_file() (in module distutils.file_util)
@anchor{distutils/apiref distutils file_util copy_file}@anchor{497f}
@deffn {Function} distutils.file_util.copy_file (src, dst[, preserve_mode=1, preserve_times=1, update=0, link=None, verbose=0, dry_run=0])

Copy file @emph{src} to @emph{dst}. If @emph{dst} is a directory, then @emph{src} is copied there
with the same name; otherwise, it must be a filename. (If the file exists, it
will be ruthlessly clobbered.) If @emph{preserve_mode} is true (the default), the
file’s mode (type and permission bits, or whatever is analogous on the
current platform) is copied. If @emph{preserve_times} is true (the default), the
last-modified and last-access times are copied as well. If @emph{update} is true,
@emph{src} will only be copied if @emph{dst} does not exist, or if @emph{dst} does exist but
is older than @emph{src}.

@emph{link} allows you to make hard links (using @ref{c82,,os.link()}) or symbolic links
(using @ref{c88,,os.symlink()}) instead of copying: set it to @code{'hard'} or
@code{'sym'}; if it is @code{None} (the default), files are copied. Don’t set @emph{link}
on systems that don’t support it: @ref{497f,,copy_file()} doesn’t check if hard or
symbolic linking is available.  It uses @code{_copy_file_contents()} to copy file
contents.

Return a tuple @code{(dest_name, copied)}: @emph{dest_name} is the actual  name of the
output file, and @emph{copied} is true if the file was copied  (or would have been
copied, if @emph{dry_run} true).

@c % XXX if the destination file already exists, we clobber it if

@c % copying, but blow up if linking.  Hmmm.  And I don't know what

@c % macostools.copyfile() does.  Should definitely be consistent, and

@c % should probably blow up if destination exists and we would be

@c % changing it (ie. it's not already a hard/soft link to src OR

@c % (not update) and (src newer than dst)).
@end deffn

@geindex move_file() (in module distutils.file_util)
@anchor{distutils/apiref distutils file_util move_file}@anchor{496c}
@deffn {Function} distutils.file_util.move_file (src, dst[, verbose, dry_run])

Move file @emph{src} to @emph{dst}. If @emph{dst} is a directory, the file will be moved into
it with the same name; otherwise, @emph{src} is just renamed to @emph{dst}.  Returns the
new full name of the file.

@cartouche
@quotation Warning 
Handles cross-device moves on Unix using @ref{497f,,copy_file()}.  What about
other systems?
@end quotation
@end cartouche
@end deffn

@geindex write_file() (in module distutils.file_util)
@anchor{distutils/apiref distutils file_util write_file}@anchor{4982}
@deffn {Function} distutils.file_util.write_file (filename, contents)

Create a file called @emph{filename} and write @emph{contents} (a sequence of strings
without line terminators) to it.
@end deffn

@node distutils util — Miscellaneous other utility functions,distutils dist — The Distribution class,distutils file_util — Single file operations,API Reference
@anchor{distutils/apiref distutils-util-miscellaneous-other-utility-functions}@anchor{4983}@anchor{distutils/apiref module-distutils util}@anchor{61}
@subsection @code{distutils.util} — Miscellaneous other utility functions


@geindex module; distutils.util

This module contains other assorted bits and pieces that don’t fit into  any
other utility module.

@geindex get_platform() (in module distutils.util)
@anchor{distutils/apiref distutils util get_platform}@anchor{4984}
@deffn {Function} distutils.util.get_platform ()

Return a string that identifies the current platform.  This is used mainly to
distinguish platform-specific build directories and platform-specific built
distributions.  Typically includes the OS name and version and the
architecture (as supplied by ‘os.uname()’), although the exact information
included depends on the OS; e.g., on Linux, the kernel version isn’t
particularly important.

Examples of returned values:


@itemize *

@item 
@code{linux-i586}

@item 
@code{linux-alpha}

@item 
@code{solaris-2.6-sun4u}
@end itemize

For non-POSIX platforms, currently just returns @code{sys.platform}.

For macOS systems the OS version reflects the minimal version on which
binaries will run (that is, the value of @code{MACOSX_DEPLOYMENT_TARGET}
during the build of Python), not the OS version of the current system.

For universal binary builds on macOS the architecture value reflects
the universal binary status instead of the architecture of the current
processor. For 32-bit universal binaries the architecture is @code{fat},
for 64-bit universal binaries the architecture is @code{fat64}, and
for 4-way universal binaries the architecture is @code{universal}. Starting
from Python 2.7 and Python 3.2 the architecture @code{fat3} is used for
a 3-way universal build (ppc, i386, x86_64) and @code{intel} is used for
a universal build with the i386 and x86_64 architectures

Examples of returned values on macOS:


@itemize *

@item 
@code{macosx-10.3-ppc}

@item 
@code{macosx-10.3-fat}

@item 
@code{macosx-10.5-universal}

@item 
@code{macosx-10.6-intel}
@end itemize

For AIX, Python 3.9 and later return a string starting with “aix”, followed
by additional fields (separated by @code{'-'}) that represent the combined
values of AIX Version, Release and Technology Level (first field), Build Date
(second field), and bit-size (third field). Python 3.8 and earlier returned
only a single additional field with the AIX Version and Release.

Examples of returned values on AIX:


@itemize *

@item 
@code{aix-5307-0747-32} # 32-bit build on AIX @code{oslevel -s}: 5300-07-00-0000

@item 
@code{aix-7105-1731-64} # 64-bit build on AIX @code{oslevel -s}: 7100-05-01-1731

@item 
@code{aix-7.2}          # Legacy form reported in Python 3.8 and earlier
@end itemize


Changed in version 3.9: The AIX platform string format now also includes the technology level,
build date, and ABI bit-size.

@end deffn

@geindex convert_path() (in module distutils.util)
@anchor{distutils/apiref distutils util convert_path}@anchor{4985}
@deffn {Function} distutils.util.convert_path (pathname)

Return ‘pathname’ as a name that will work on the native filesystem, i.e. split
it on ‘/’ and put it back together again using the current directory separator.
Needed because filenames in the setup script are always supplied in Unix style,
and have to be converted to the local convention before we can actually use them
in the filesystem.  Raises @ref{17c,,ValueError} on non-Unix-ish systems if
@emph{pathname} either  starts or ends with a slash.
@end deffn

@geindex change_root() (in module distutils.util)
@anchor{distutils/apiref distutils util change_root}@anchor{4986}
@deffn {Function} distutils.util.change_root (new_root, pathname)

Return @emph{pathname} with @emph{new_root} prepended.  If @emph{pathname} is relative, this is
equivalent to @code{os.path.join(new_root,pathname)} Otherwise, it requires making
@emph{pathname} relative and then joining the two, which is tricky on DOS/Windows.
@end deffn

@geindex check_environ() (in module distutils.util)
@anchor{distutils/apiref distutils util check_environ}@anchor{4987}
@deffn {Function} distutils.util.check_environ ()

Ensure that ‘os.environ’ has all the environment variables we guarantee that
users can use in config files, command-line options, etc.  Currently this
includes:


@itemize *

@item 
@geindex HOME
@geindex environment variable; HOME
@code{HOME} - user’s home directory (Unix only)

@item 
@geindex PLAT
@geindex environment variable; PLAT
@code{PLAT} - description of the current platform, including hardware and
OS (see @ref{4984,,get_platform()})
@end itemize
@end deffn

@geindex subst_vars() (in module distutils.util)
@anchor{distutils/apiref distutils util subst_vars}@anchor{4988}
@deffn {Function} distutils.util.subst_vars (s, local_vars)

Perform shell/Perl-style variable substitution on @emph{s}.  Every occurrence of
@code{$} followed by a name is considered a variable, and variable is substituted
by the value found in the @emph{local_vars} dictionary, or in @code{os.environ} if it’s
not in @emph{local_vars}. @emph{os.environ} is first checked/augmented to guarantee that
it contains certain values: see @ref{4987,,check_environ()}.  Raise @ref{17c,,ValueError}
for any variables not found in either @emph{local_vars} or @code{os.environ}.

Note that this is not a fully-fledged string interpolation function. A valid
@code{$variable} can consist only of upper and lower case letters, numbers and an
underscore. No @{ @} or ( ) style quoting is available.
@end deffn

@geindex split_quoted() (in module distutils.util)
@anchor{distutils/apiref distutils util split_quoted}@anchor{495a}
@deffn {Function} distutils.util.split_quoted (s)

Split a string up according to Unix shell-like rules for quotes and backslashes.
In short: words are delimited by spaces, as long as those spaces are not escaped
by a backslash, or inside a quoted string. Single and double quotes are
equivalent, and the quote characters can be backslash-escaped.  The backslash is
stripped from any two-character escape sequence, leaving only the escaped
character.  The quote characters are stripped from any quoted string.  Returns a
list of words.

@c % Should probably be moved into the standard library.
@end deffn

@geindex execute() (in module distutils.util)
@anchor{distutils/apiref distutils util execute}@anchor{4967}
@deffn {Function} distutils.util.execute (func, args[, msg=None, verbose=0, dry_run=0])

Perform some action that affects the outside world (for instance, writing to the
filesystem).  Such actions are special because they are disabled by the
@emph{dry_run} flag.  This method takes  care of all that bureaucracy for you; all
you have to do is supply the function to call and an argument tuple for it (to
embody the “external action” being performed), and an optional message to print.
@end deffn

@geindex strtobool() (in module distutils.util)
@anchor{distutils/apiref distutils util strtobool}@anchor{4989}
@deffn {Function} distutils.util.strtobool (val)

Convert a string representation of truth to true (1) or false (0).

True values are @code{y}, @code{yes}, @code{t}, @code{true}, @code{on}  and @code{1}; false values
are @code{n}, @code{no}, @code{f}, @code{false},  @code{off} and @code{0}.  Raises
@ref{17c,,ValueError} if @emph{val}  is anything else.
@end deffn

@geindex byte_compile() (in module distutils.util)
@anchor{distutils/apiref distutils util byte_compile}@anchor{498a}
@deffn {Function} distutils.util.byte_compile (py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None])

Byte-compile a collection of Python source files to @code{.pyc} files in a
@code{__pycache__} subdirectory (see 
@geindex Python Enhancement Proposals; PEP 3147
PEP 3147@footnote{https://www.python.org/dev/peps/pep-3147} and 
@geindex Python Enhancement Proposals; PEP 488
PEP 488@footnote{https://www.python.org/dev/peps/pep-0488}).
@emph{py_files} is a list of files to compile; any files that don’t end in
@code{.py} are silently skipped.  @emph{optimize} must be one of the following:


@itemize *

@item 
@code{0} - don’t optimize

@item 
@code{1} - normal optimization (like @code{python -O})

@item 
@code{2} - extra optimization (like @code{python -OO})
@end itemize

If @emph{force} is true, all files are recompiled regardless of timestamps.

The source filename encoded in each @ref{8a8,,bytecode} file defaults to the filenames
listed in @emph{py_files}; you can modify these with @emph{prefix} and @emph{basedir}.
@emph{prefix} is a string that will be stripped off of each source filename, and
@emph{base_dir} is a directory name that will be prepended (after @emph{prefix} is
stripped).  You can supply either or both (or neither) of @emph{prefix} and
@emph{base_dir}, as you wish.

If @emph{dry_run} is true, doesn’t actually do anything that would affect the
filesystem.

Byte-compilation is either done directly in this interpreter process with the
standard @ref{d2,,py_compile} module, or indirectly by writing a temporary script
and executing it.  Normally, you should let @ref{498a,,byte_compile()} figure out to
use direct compilation or not (see the source for details).  The @emph{direct} flag
is used by the script generated in indirect mode; unless you know what you’re
doing, leave it set to @code{None}.


Changed in version 3.2.3: Create @code{.pyc} files with an @ref{4056,,import magic tag} in their name, in a @code{__pycache__} subdirectory
instead of files without tag in the current directory.



Changed in version 3.5: Create @code{.pyc} files according to 
@geindex Python Enhancement Proposals; PEP 488
PEP 488@footnote{https://www.python.org/dev/peps/pep-0488}.

@end deffn

@geindex rfc822_escape() (in module distutils.util)
@anchor{distutils/apiref distutils util rfc822_escape}@anchor{498b}
@deffn {Function} distutils.util.rfc822_escape (header)

Return a version of @emph{header} escaped for inclusion in an 
@geindex RFC; RFC 822
RFC 822@footnote{https://tools.ietf.org/html/rfc822.html} header, by
ensuring there are 8 spaces space after each newline. Note that it does no other
modification of the string.

@c % this _can_ be replaced
@end deffn

@c % \subsection{Distutils objects}

@node distutils dist — The Distribution class,distutils extension — The Extension class,distutils util — Miscellaneous other utility functions,API Reference
@anchor{distutils/apiref distutils-dist-the-distribution-class}@anchor{498c}@anchor{distutils/apiref module-distutils dist}@anchor{55}
@subsection @code{distutils.dist} — The Distribution class


@geindex module; distutils.dist

This module provides the @ref{493d,,Distribution} class, which
represents the module distribution being built/installed/distributed.

@node distutils extension — The Extension class,distutils debug — Distutils debug mode,distutils dist — The Distribution class,API Reference
@anchor{distutils/apiref distutils-extension-the-extension-class}@anchor{498d}@anchor{distutils/apiref module-distutils extension}@anchor{57}
@subsection @code{distutils.extension} — The Extension class


@geindex module; distutils.extension

This module provides the @code{Extension} class, used to describe C/C++
extension modules in setup scripts.

@c % \subsection{Ungrouped modules}

@c % The following haven't been moved into a more appropriate section yet.

@node distutils debug — Distutils debug mode,distutils errors — Distutils exceptions,distutils extension — The Extension class,API Reference
@anchor{distutils/apiref distutils-debug-distutils-debug-mode}@anchor{498e}@anchor{distutils/apiref module-distutils debug}@anchor{52}
@subsection @code{distutils.debug} — Distutils debug mode


@geindex module; distutils.debug

This module provides the DEBUG flag.

@node distutils errors — Distutils exceptions,distutils fancy_getopt — Wrapper around the standard getopt module,distutils debug — Distutils debug mode,API Reference
@anchor{distutils/apiref distutils-errors-distutils-exceptions}@anchor{498f}@anchor{distutils/apiref module-distutils errors}@anchor{56}
@subsection @code{distutils.errors} — Distutils exceptions


@geindex module; distutils.errors

Provides exceptions used by the Distutils modules.  Note that Distutils modules
may raise standard exceptions; in particular, SystemExit is usually raised for
errors that are obviously the end-user’s fault (eg. bad command-line arguments).

This module is safe to use in @code{from ... import *} mode; it only exports
symbols whose names start with @code{Distutils} and end with @code{Error}.

@node distutils fancy_getopt — Wrapper around the standard getopt module,distutils filelist — The FileList class,distutils errors — Distutils exceptions,API Reference
@anchor{distutils/apiref distutils-fancy-getopt-wrapper-around-the-standard-getopt-module}@anchor{4990}@anchor{distutils/apiref module-distutils fancy_getopt}@anchor{58}
@subsection @code{distutils.fancy_getopt} — Wrapper around the standard getopt module


@geindex module; distutils.fancy_getopt

This module provides a wrapper around the standard @ref{81,,getopt}  module that
provides the following additional features:


@itemize *

@item 
short and long options are tied together

@item 
options have help strings, so @ref{4991,,fancy_getopt()} could potentially  create a
complete usage summary

@item 
options set attributes of a passed-in object

@item 
boolean options can have “negative aliases” — eg. if @code{--quiet} is
the “negative alias” of @code{--verbose}, then @code{--quiet} on the
command line sets @emph{verbose} to false.
@end itemize

@geindex fancy_getopt() (in module distutils.fancy_getopt)
@anchor{distutils/apiref distutils fancy_getopt fancy_getopt}@anchor{4991}
@deffn {Function} distutils.fancy_getopt.fancy_getopt (options, negative_opt, object, args)

Wrapper function. @emph{options} is a list of @code{(long_option, short_option,
help_string)} 3-tuples as described in the constructor for
@ref{4992,,FancyGetopt}. @emph{negative_opt} should be a dictionary mapping option names
to option names, both the key and value should be in the @emph{options} list.
@emph{object} is an object which will be used to store values (see the @ref{81,,getopt()}
method of the @ref{4992,,FancyGetopt} class). @emph{args} is the argument list. Will use
@code{sys.argv[1:]} if you  pass @code{None} as @emph{args}.
@end deffn

@geindex wrap_text() (in module distutils.fancy_getopt)
@anchor{distutils/apiref distutils fancy_getopt wrap_text}@anchor{4993}
@deffn {Function} distutils.fancy_getopt.wrap_text (text, width)

Wraps @emph{text} to less than @emph{width} wide.
@end deffn

@geindex FancyGetopt (class in distutils.fancy_getopt)
@anchor{distutils/apiref distutils fancy_getopt FancyGetopt}@anchor{4992}
@deffn {Class} distutils.fancy_getopt.FancyGetopt ([option_table=None])

The option_table is a list of 3-tuples: @code{(long_option, short_option,
help_string)}

If an option takes an argument, its @emph{long_option} should have @code{'='} appended;
@emph{short_option} should just be a single character, no @code{':'} in any case.
@emph{short_option} should be @code{None} if a @emph{long_option}  doesn’t have a
corresponding @emph{short_option}. All option tuples must have long options.
@end deffn

The @ref{4992,,FancyGetopt} class provides the following methods:

@geindex getopt() (distutils.fancy_getopt.FancyGetopt method)
@anchor{distutils/apiref distutils fancy_getopt FancyGetopt getopt}@anchor{4994}
@deffn {Method} FancyGetopt.getopt ([args=None, object=None])

Parse command-line options in args. Store as attributes on @emph{object}.

If @emph{args} is @code{None} or not supplied, uses @code{sys.argv[1:]}.  If @emph{object} is
@code{None} or not supplied, creates a new @code{OptionDummy} instance, stores
option values there, and returns a tuple @code{(args, object)}.  If @emph{object} is
supplied, it is modified in place and @ref{81,,getopt()} just returns @emph{args}; in
both cases, the returned @emph{args} is a modified copy of the passed-in @emph{args} list,
which is left untouched.

@c % and args returned are?
@end deffn

@geindex get_option_order() (distutils.fancy_getopt.FancyGetopt method)
@anchor{distutils/apiref distutils fancy_getopt FancyGetopt get_option_order}@anchor{4995}
@deffn {Method} FancyGetopt.get_option_order ()

Returns the list of @code{(option, value)} tuples processed by the previous run of
@ref{81,,getopt()}  Raises @ref{5af,,RuntimeError} if @ref{81,,getopt()} hasn’t been called
yet.
@end deffn

@geindex generate_help() (distutils.fancy_getopt.FancyGetopt method)
@anchor{distutils/apiref distutils fancy_getopt FancyGetopt generate_help}@anchor{4996}
@deffn {Method} FancyGetopt.generate_help ([header=None])

Generate help text (a list of strings, one per suggested line of output) from
the option table for this @ref{4992,,FancyGetopt} object.

If supplied, prints the supplied @emph{header} at the top of the help.
@end deffn

@node distutils filelist — The FileList class,distutils log — Simple PEP 282-style logging,distutils fancy_getopt — Wrapper around the standard getopt module,API Reference
@anchor{distutils/apiref distutils-filelist-the-filelist-class}@anchor{4997}@anchor{distutils/apiref module-distutils filelist}@anchor{5a}
@subsection @code{distutils.filelist} — The FileList class


@geindex module; distutils.filelist

This module provides the @code{FileList} class, used for poking about the
filesystem and building lists of files.

@node distutils log — Simple PEP 282-style logging,distutils spawn — Spawn a sub-process,distutils filelist — The FileList class,API Reference
@anchor{distutils/apiref distutils-log-simple-pep-282-style-logging}@anchor{4998}@anchor{distutils/apiref module-distutils log}@anchor{5b}
@subsection @code{distutils.log} — Simple 
@geindex Python Enhancement Proposals; PEP 282
@strong{PEP 282}-style logging


@geindex module; distutils.log

@node distutils spawn — Spawn a sub-process,distutils sysconfig — System configuration information,distutils log — Simple PEP 282-style logging,API Reference
@anchor{distutils/apiref distutils-spawn-spawn-a-sub-process}@anchor{4999}@anchor{distutils/apiref module-distutils spawn}@anchor{5d}
@subsection @code{distutils.spawn} — Spawn a sub-process


@geindex module; distutils.spawn

This module provides the @code{spawn()} function, a front-end to  various
platform-specific functions for launching another program in a  sub-process.
Also provides @code{find_executable()} to search the path for a given executable
name.

@node distutils sysconfig — System configuration information,distutils text_file — The TextFile class,distutils spawn — Spawn a sub-process,API Reference
@anchor{distutils/apiref distutils-sysconfig-system-configuration-information}@anchor{499a}@anchor{distutils/apiref module-distutils sysconfig}@anchor{5e}
@subsection @code{distutils.sysconfig} — System configuration information


@geindex module; distutils.sysconfig


Deprecated since version 3.10: @ref{5e,,distutils.sysconfig} has been merged into @ref{f8,,sysconfig}.


The @ref{5e,,distutils.sysconfig} module provides access to Python’s low-level
configuration information.  The specific configuration variables available
depend heavily on the platform and configuration. The specific variables depend
on the build process for the specific version of Python being run; the variables
are those found in the @code{Makefile} and configuration header that are
installed with Python on Unix systems.  The configuration header is called
@code{pyconfig.h} for Python versions starting with 2.2, and @code{config.h}
for earlier versions of Python.

Some additional functions are provided which perform some useful manipulations
for other parts of the @ref{37,,distutils} package.

@geindex PREFIX (in module distutils.sysconfig)
@anchor{distutils/apiref distutils sysconfig PREFIX}@anchor{499b}
@deffn {Data} distutils.sysconfig.PREFIX

The result of @code{os.path.normpath(sys.prefix)}.
@end deffn

@geindex EXEC_PREFIX (in module distutils.sysconfig)
@anchor{distutils/apiref distutils sysconfig EXEC_PREFIX}@anchor{499c}
@deffn {Data} distutils.sysconfig.EXEC_PREFIX

The result of @code{os.path.normpath(sys.exec_prefix)}.
@end deffn

@geindex get_config_var() (in module distutils.sysconfig)
@anchor{distutils/apiref distutils sysconfig get_config_var}@anchor{499d}
@deffn {Function} distutils.sysconfig.get_config_var (name)

Return the value of a single variable.  This is equivalent to
@code{get_config_vars().get(name)}.
@end deffn

@geindex get_config_vars() (in module distutils.sysconfig)
@anchor{distutils/apiref distutils sysconfig get_config_vars}@anchor{499e}
@deffn {Function} distutils.sysconfig.get_config_vars (...)

Return a set of variable definitions.  If there are no arguments, this returns a
dictionary mapping names of configuration variables to values.  If arguments are
provided, they should be strings, and the return value will be a sequence giving
the associated values. If a given name does not have a corresponding value,
@code{None} will be included for that variable.
@end deffn

@geindex get_config_h_filename() (in module distutils.sysconfig)
@anchor{distutils/apiref distutils sysconfig get_config_h_filename}@anchor{499f}
@deffn {Function} distutils.sysconfig.get_config_h_filename ()

Return the full path name of the configuration header.  For Unix, this will be
the header generated by the @code{configure} script; for other platforms the
header will have been supplied directly by the Python source distribution.  The
file is a platform-specific text file.
@end deffn

@geindex get_makefile_filename() (in module distutils.sysconfig)
@anchor{distutils/apiref distutils sysconfig get_makefile_filename}@anchor{49a0}
@deffn {Function} distutils.sysconfig.get_makefile_filename ()

Return the full path name of the @code{Makefile} used to build Python.  For
Unix, this will be a file generated by the @code{configure} script; the
meaning for other platforms will vary.  The file is a platform-specific text
file, if it exists. This function is only useful on POSIX platforms.
@end deffn

The following functions are deprecated together with this module and they
have no direct replacement.

@geindex get_python_inc() (in module distutils.sysconfig)
@anchor{distutils/apiref distutils sysconfig get_python_inc}@anchor{49a1}
@deffn {Function} distutils.sysconfig.get_python_inc ([plat_specific[, prefix]])

Return the directory for either the general or platform-dependent C include
files.  If @emph{plat_specific} is true, the platform-dependent include directory is
returned; if false or omitted, the platform-independent directory is returned.
If @emph{prefix} is given, it is used as either the prefix instead of
@ref{499b,,PREFIX}, or as the exec-prefix instead of @ref{499c,,EXEC_PREFIX} if
@emph{plat_specific} is true.
@end deffn

@geindex get_python_lib() (in module distutils.sysconfig)
@anchor{distutils/apiref distutils sysconfig get_python_lib}@anchor{49a2}
@deffn {Function} distutils.sysconfig.get_python_lib ([plat_specific[, standard_lib[, prefix]]])

Return the directory for either the general or platform-dependent library
installation.  If @emph{plat_specific} is true, the platform-dependent include
directory is returned; if false or omitted, the platform-independent directory
is returned.  If @emph{prefix} is given, it is used as either the prefix instead of
@ref{499b,,PREFIX}, or as the exec-prefix instead of @ref{499c,,EXEC_PREFIX} if
@emph{plat_specific} is true.  If @emph{standard_lib} is true, the directory for the
standard library is returned rather than the directory for the installation of
third-party extensions.
@end deffn

The following function is only intended for use within the @ref{37,,distutils}
package.

@geindex customize_compiler() (in module distutils.sysconfig)
@anchor{distutils/apiref distutils sysconfig customize_compiler}@anchor{49a3}
@deffn {Function} distutils.sysconfig.customize_compiler (compiler)

Do any platform-specific customization of a
@ref{4941,,distutils.ccompiler.CCompiler} instance.

This function is only needed on Unix at this time, but should be called
consistently to support forward-compatibility.  It inserts the information that
varies across Unix flavors and is stored in Python’s @code{Makefile}.  This
information includes the selected compiler, compiler and linker options, and the
extension used by the linker for shared objects.
@end deffn

This function is even more special-purpose, and should only be used from
Python’s own build procedures.

@geindex set_python_build() (in module distutils.sysconfig)
@anchor{distutils/apiref distutils sysconfig set_python_build}@anchor{49a4}
@deffn {Function} distutils.sysconfig.set_python_build ()

Inform the @ref{5e,,distutils.sysconfig} module that it is being used as part of
the build process for Python.  This changes a lot of relative locations for
files, allowing them to be located in the build area rather than in an installed
Python.
@end deffn

@node distutils text_file — The TextFile class,distutils version — Version number classes,distutils sysconfig — System configuration information,API Reference
@anchor{distutils/apiref distutils-text-file-the-textfile-class}@anchor{49a5}@anchor{distutils/apiref module-distutils text_file}@anchor{5f}
@subsection @code{distutils.text_file} — The TextFile class


@geindex module; distutils.text_file

This module provides the @ref{49a6,,TextFile} class, which gives an interface  to
text files that (optionally) takes care of stripping comments, ignoring  blank
lines, and joining lines with backslashes.

@geindex TextFile (class in distutils.text_file)
@anchor{distutils/apiref distutils text_file TextFile}@anchor{49a6}
@deffn {Class} distutils.text_file.TextFile ([filename=None, file=None, **options])

This class provides a file-like object that takes care of all  the things you
commonly want to do when processing a text file  that has some line-by-line
syntax: strip comments (as long as @code{#}  is your comment character), skip blank
lines, join adjacent lines by escaping the newline (ie. backslash at end of
line), strip leading and/or trailing whitespace.  All of these are optional and
independently controllable.

The class provides a @ref{49a7,,warn()} method so you can generate  warning messages
that report physical line number, even if the  logical line in question spans
multiple physical lines.  Also  provides @ref{49a8,,unreadline()} for implementing
line-at-a-time lookahead.

@ref{49a6,,TextFile} instances are create with either @emph{filename}, @emph{file}, or both.
@ref{5af,,RuntimeError} is raised if both are @code{None}. @emph{filename} should be a
string, and @emph{file} a file object (or something that provides @ref{d9,,readline()}
and @ref{49a9,,close()}  methods).  It is recommended that you supply at least
@emph{filename},  so that @ref{49a6,,TextFile} can include it in warning messages.  If
@emph{file} is not supplied, @ref{49a6,,TextFile} creates its own using the
@ref{1c8,,open()} built-in function.

The options are all boolean, and affect the values returned by @ref{d9,,readline()}


@multitable {xxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxx} 
@headitem

option name

@tab

description

@tab

default

@item

@emph{strip_comments}

@tab

strip from @code{'#'} to
end-of-line, as well as any
whitespace leading up to the
@code{'#'}—unless it is
escaped by a backslash

@tab

true

@item

@emph{lstrip_ws}

@tab

strip leading whitespace from
each line before returning it

@tab

false

@item

@emph{rstrip_ws}

@tab

strip trailing whitespace
(including line terminator!)
from each line before
returning it.

@tab

true

@item

@emph{skip_blanks}

@tab

skip lines that are empty
*after* stripping comments
and whitespace.  (If both
lstrip_ws and rstrip_ws are
false, then some lines may
consist of solely whitespace:
these will *not* be skipped,
even if @emph{skip_blanks} is
true.)

@tab

true

@item

@emph{join_lines}

@tab

if a backslash is the last
non-newline character on a
line after stripping comments
and whitespace, join the
following line to it to form
one logical line; if N
consecutive lines end with a
backslash, then N+1 physical
lines will be joined to form
one logical line.

@tab

false

@item

@emph{collapse_join}

@tab

strip leading whitespace from
lines that are joined to their
predecessor; only matters if
@code{(join_lines and not
lstrip_ws)}

@tab

false

@end multitable


Note that since @emph{rstrip_ws} can strip the trailing newline, the semantics of
@ref{d9,,readline()} must differ from those of the built-in file object’s
@ref{d9,,readline()} method!  In particular, @ref{d9,,readline()}  returns @code{None} for
end-of-file: an empty string might just be a  blank line (or an all-whitespace
line), if @emph{rstrip_ws} is true  but @emph{skip_blanks} is not.

@geindex open() (distutils.text_file.TextFile method)
@anchor{distutils/apiref distutils text_file TextFile open}@anchor{49aa}
@deffn {Method} open (filename)

Open a new file @emph{filename}.  This overrides any @emph{file} or @emph{filename}
constructor arguments.
@end deffn

@geindex close() (distutils.text_file.TextFile method)
@anchor{distutils/apiref distutils text_file TextFile close}@anchor{49a9}
@deffn {Method} close ()

Close the current file and forget everything we know about it (including the
filename and the current line number).
@end deffn

@geindex warn() (distutils.text_file.TextFile method)
@anchor{distutils/apiref distutils text_file TextFile warn}@anchor{49a7}
@deffn {Method} warn (msg[, line=None])

Print (to stderr) a warning message tied to the current logical line in the
current file.  If the current logical line in the file spans multiple physical
lines, the warning refers to the whole range, such as @code{"lines 3-5"}.  If
@emph{line} is supplied,  it overrides the current line number; it may be a list or
tuple  to indicate a range of physical lines, or an integer for a  single
physical line.
@end deffn

@geindex readline() (distutils.text_file.TextFile method)
@anchor{distutils/apiref distutils text_file TextFile readline}@anchor{49ab}
@deffn {Method} readline ()

Read and return a single logical line from the current file (or from an internal
buffer if lines have previously been “unread” with @ref{49a8,,unreadline()}).  If the
@emph{join_lines} option  is true, this may involve reading multiple physical lines
concatenated into a single string.  Updates the current line number,  so calling
@ref{49a7,,warn()} after @ref{d9,,readline()} emits a warning  about the physical line(s)
just read.  Returns @code{None} on end-of-file,  since the empty string can occur
if @emph{rstrip_ws} is true but  @emph{strip_blanks} is not.
@end deffn

@geindex readlines() (distutils.text_file.TextFile method)
@anchor{distutils/apiref distutils text_file TextFile readlines}@anchor{49ac}
@deffn {Method} readlines ()

Read and return the list of all logical lines remaining in the current file.
This updates the current line number to the last line of the file.
@end deffn

@geindex unreadline() (distutils.text_file.TextFile method)
@anchor{distutils/apiref distutils text_file TextFile unreadline}@anchor{49a8}
@deffn {Method} unreadline (line)

Push @emph{line} (a string) onto an internal buffer that will be checked by future
@ref{d9,,readline()} calls.  Handy for implementing a parser with line-at-a-time
lookahead. Note that lines that are “unread” with @ref{49a8,,unreadline()} are not
subsequently re-cleansed (whitespace  stripped, or whatever) when read with
@ref{d9,,readline()}. If multiple calls are made to @ref{49a8,,unreadline()} before a call
to @ref{d9,,readline()}, the lines will be returned most in most recent first order.
@end deffn
@end deffn

@node distutils version — Version number classes,distutils cmd — Abstract base class for Distutils commands,distutils text_file — The TextFile class,API Reference
@anchor{distutils/apiref distutils-version-version-number-classes}@anchor{49ad}@anchor{distutils/apiref module-distutils version}@anchor{62}
@subsection @code{distutils.version} — Version number classes


@geindex module; distutils.version

@c % todo

@c % \section{Distutils Commands}

@c %

@c % This part of Distutils implements the various Distutils commands, such

@c % as \code{build}, \code{install} \&c. Each command is implemented as a

@c % separate module, with the command name as the name of the module.

@node distutils cmd — Abstract base class for Distutils commands,Creating a new Distutils command,distutils version — Version number classes,API Reference
@anchor{distutils/apiref distutils-cmd-abstract-base-class-for-distutils-commands}@anchor{49ae}@anchor{distutils/apiref module-distutils cmd}@anchor{3b}
@subsection @code{distutils.cmd} — Abstract base class for Distutils commands


@geindex module; distutils.cmd

This module supplies the abstract base class @ref{492c,,Command}.

@geindex Command (class in distutils.cmd)
@anchor{distutils/apiref distutils cmd Command}@anchor{492c}
@deffn {Class} distutils.cmd.Command (dist)

Abstract base class for defining command classes, the “worker bees” of the
Distutils.  A useful analogy for command classes is to think of them as
subroutines with local variables called @emph{options}.  The options are declared
in @ref{49af,,initialize_options()} and defined (given their final values) in
@ref{49b0,,finalize_options()}, both of which must be defined by every command
class.  The distinction between the two is necessary because option values
might come from the outside world (command line, config file, …), and any
options dependent on other options must be computed after these outside
influences have been processed — hence @ref{49b0,,finalize_options()}.  The body
of the subroutine, where it does all its work based on the values of its
options, is the @ref{49b1,,run()} method, which must also be implemented by every
command class.

The class constructor takes a single argument @emph{dist}, a
@ref{493d,,Distribution} instance.
@end deffn

@node Creating a new Distutils command,distutils command — Individual Distutils commands,distutils cmd — Abstract base class for Distutils commands,API Reference
@anchor{distutils/apiref creating-a-new-distutils-command}@anchor{49b2}
@subsection Creating a new Distutils command


This section outlines the steps to create a new Distutils command.

A new command lives in a module in the @ref{3c,,distutils.command} package. There
is a sample template in that directory called @code{command_template}.  Copy
this file to a new module with the same name as the new command you’re
implementing.  This module should implement a class with the same name as the
module (and the command).  So, for instance, to create the command
@code{peel_banana} (so that users can run @code{setup.py peel_banana}), you’d copy
@code{command_template} to @code{distutils/command/peel_banana.py}, then edit
it so that it’s implementing the class @code{peel_banana}, a subclass of
@ref{492c,,distutils.cmd.Command}.

Subclasses of @ref{492c,,Command} must define the following methods.

@geindex initialize_options() (distutils.cmd.Command method)
@anchor{distutils/apiref distutils cmd Command initialize_options}@anchor{49af}
@deffn {Method} Command.initialize_options ()

Set default values for all the options that this command supports.  Note that
these defaults may be overridden by other commands, by the setup script, by
config files, or by the command-line.  Thus, this is not the place to code
dependencies between options; generally, @ref{49af,,initialize_options()}
implementations are just a bunch of @code{self.foo = None} assignments.
@end deffn

@geindex finalize_options() (distutils.cmd.Command method)
@anchor{distutils/apiref distutils cmd Command finalize_options}@anchor{49b0}
@deffn {Method} Command.finalize_options ()

Set final values for all the options that this command supports. This is
always called as late as possible, ie.  after any option assignments from the
command-line or from other commands have been done.  Thus, this is the place
to code option dependencies: if @emph{foo} depends on @emph{bar}, then it is safe to
set @emph{foo} from @emph{bar} as long as @emph{foo} still has the same value it was
assigned in @ref{49af,,initialize_options()}.
@end deffn

@geindex run() (distutils.cmd.Command method)
@anchor{distutils/apiref distutils cmd Command run}@anchor{49b1}
@deffn {Method} Command.run ()

A command’s raison d’etre: carry out the action it exists to perform, controlled
by the options initialized in @ref{49af,,initialize_options()}, customized by other
commands, the setup script, the command-line, and config files, and finalized in
@ref{49b0,,finalize_options()}.  All terminal output and filesystem interaction should
be done by @ref{49b1,,run()}.
@end deffn

@geindex sub_commands (distutils.cmd.Command attribute)
@anchor{distutils/apiref distutils cmd Command sub_commands}@anchor{49b3}
@deffn {Attribute} Command.sub_commands

@emph{sub_commands} formalizes the notion of a “family” of commands,
e.g. @code{install} as the parent with sub-commands @code{install_lib},
@code{install_headers}, etc.  The parent of a family of commands defines
@emph{sub_commands} as a class attribute; it’s a list of 2-tuples @code{(command_name,
predicate)}, with @emph{command_name} a string and @emph{predicate} a function, a
string or @code{None}.  @emph{predicate} is a method of the parent command that
determines whether the corresponding command is applicable in the current
situation.  (E.g. @code{install_headers} is only applicable if we have any C
header files to install.)  If @emph{predicate} is @code{None}, that command is always
applicable.

@emph{sub_commands} is usually defined at the @emph{end} of a class, because
predicates can be methods of the class, so they must already have been
defined.  The canonical example is the @code{install} command.
@end deffn

@node distutils command — Individual Distutils commands,distutils command bdist — Build a binary installer,Creating a new Distutils command,API Reference
@anchor{distutils/apiref distutils-command-individual-distutils-commands}@anchor{49b4}@anchor{distutils/apiref module-distutils command}@anchor{3c}
@subsection @code{distutils.command} — Individual Distutils commands


@geindex module; distutils.command

@c % \subsubsection{Individual Distutils commands}

@c % todo

@node distutils command bdist — Build a binary installer,distutils command bdist_packager — Abstract base class for packagers,distutils command — Individual Distutils commands,API Reference
@anchor{distutils/apiref distutils-command-bdist-build-a-binary-installer}@anchor{49b5}@anchor{distutils/apiref module-distutils command bdist}@anchor{3d}
@subsection @code{distutils.command.bdist} — Build a binary installer


@geindex module; distutils.command.bdist

@c % todo

@node distutils command bdist_packager — Abstract base class for packagers,distutils command bdist_dumb — Build a “dumb” installer,distutils command bdist — Build a binary installer,API Reference
@anchor{distutils/apiref distutils-command-bdist-packager-abstract-base-class-for-packagers}@anchor{49b6}@anchor{distutils/apiref module-distutils command bdist_packager}@anchor{3f}
@subsection @code{distutils.command.bdist_packager} — Abstract base class for packagers


@geindex module; distutils.command.bdist_packager

@c % todo

@node distutils command bdist_dumb — Build a “dumb” installer,distutils command bdist_rpm — Build a binary distribution as a Redhat RPM and SRPM,distutils command bdist_packager — Abstract base class for packagers,API Reference
@anchor{distutils/apiref distutils-command-bdist-dumb-build-a-dumb-installer}@anchor{49b7}@anchor{distutils/apiref module-distutils command bdist_dumb}@anchor{3e}
@subsection @code{distutils.command.bdist_dumb} — Build a “dumb” installer


@geindex module; distutils.command.bdist_dumb

@c % todo

@node distutils command bdist_rpm — Build a binary distribution as a Redhat RPM and SRPM,distutils command sdist — Build a source distribution,distutils command bdist_dumb — Build a “dumb” installer,API Reference
@anchor{distutils/apiref distutils-command-bdist-rpm-build-a-binary-distribution-as-a-redhat-rpm-and-srpm}@anchor{49b8}@anchor{distutils/apiref module-distutils command bdist_rpm}@anchor{40}
@subsection @code{distutils.command.bdist_rpm} — Build a binary distribution as a Redhat RPM and SRPM


@geindex module; distutils.command.bdist_rpm

@c % todo

@node distutils command sdist — Build a source distribution,distutils command build — Build all files of a package,distutils command bdist_rpm — Build a binary distribution as a Redhat RPM and SRPM,API Reference
@anchor{distutils/apiref distutils-command-sdist-build-a-source-distribution}@anchor{49b9}@anchor{distutils/apiref module-distutils command sdist}@anchor{4f}
@subsection @code{distutils.command.sdist} — Build a source distribution


@geindex module; distutils.command.sdist

@c % todo

@node distutils command build — Build all files of a package,distutils command build_clib — Build any C libraries in a package,distutils command sdist — Build a source distribution,API Reference
@anchor{distutils/apiref distutils-command-build-build-all-files-of-a-package}@anchor{49ba}@anchor{distutils/apiref module-distutils command build}@anchor{41}
@subsection @code{distutils.command.build} — Build all files of a package


@geindex module; distutils.command.build

@c % todo

@node distutils command build_clib — Build any C libraries in a package,distutils command build_ext — Build any extensions in a package,distutils command build — Build all files of a package,API Reference
@anchor{distutils/apiref distutils-command-build-clib-build-any-c-libraries-in-a-package}@anchor{49bb}@anchor{distutils/apiref module-distutils command build_clib}@anchor{42}
@subsection @code{distutils.command.build_clib} — Build any C libraries in a package


@geindex module; distutils.command.build_clib

@c % todo

@node distutils command build_ext — Build any extensions in a package,distutils command build_py — Build the py/ pyc files of a package,distutils command build_clib — Build any C libraries in a package,API Reference
@anchor{distutils/apiref distutils-command-build-ext-build-any-extensions-in-a-package}@anchor{49bc}@anchor{distutils/apiref module-distutils command build_ext}@anchor{43}
@subsection @code{distutils.command.build_ext} — Build any extensions in a package


@geindex module; distutils.command.build_ext

@c % todo

@node distutils command build_py — Build the py/ pyc files of a package,distutils command build_scripts — Build the scripts of a package,distutils command build_ext — Build any extensions in a package,API Reference
@anchor{distutils/apiref distutils-command-build-py-build-the-py-pyc-files-of-a-package}@anchor{49bd}@anchor{distutils/apiref module-distutils command build_py}@anchor{44}
@subsection @code{distutils.command.build_py} — Build the .py/.pyc files of a package


@geindex module; distutils.command.build_py

@geindex build_py (class in distutils.command.build_py)
@anchor{distutils/apiref distutils command build_py build_py}@anchor{49be}
@deffn {Class} distutils.command.build_py.build_py
@end deffn

@geindex build_py_2to3 (class in distutils.command.build_py)
@anchor{distutils/apiref distutils command build_py build_py_2to3}@anchor{49bf}
@deffn {Class} distutils.command.build_py.build_py_2to3

Alternative implementation of build_py which also runs the
2to3 conversion library on each .py file that is going to be
installed. To use this in a setup.py file for a distribution
that is designed to run with both Python 2.x and 3.x, add:

@example
try:
    from distutils.command.build_py import build_py_2to3 as build_py
except ImportError:
    from distutils.command.build_py import build_py
@end example

to your setup.py, and later:

@example
cmdclass = @{'build_py': build_py@}
@end example

to the invocation of setup().
@end deffn

@node distutils command build_scripts — Build the scripts of a package,distutils command clean — Clean a package build area,distutils command build_py — Build the py/ pyc files of a package,API Reference
@anchor{distutils/apiref distutils-command-build-scripts-build-the-scripts-of-a-package}@anchor{49c0}@anchor{distutils/apiref module-distutils command build_scripts}@anchor{45}
@subsection @code{distutils.command.build_scripts} — Build the scripts of a package


@geindex module; distutils.command.build_scripts

@c % todo

@node distutils command clean — Clean a package build area,distutils command config — Perform package configuration,distutils command build_scripts — Build the scripts of a package,API Reference
@anchor{distutils/apiref distutils-command-clean-clean-a-package-build-area}@anchor{49c1}@anchor{distutils/apiref module-distutils command clean}@anchor{47}
@subsection @code{distutils.command.clean} — Clean a package build area


@geindex module; distutils.command.clean

This command removes the temporary files created by @code{build}
and its subcommands, like intermediary compiled object files.  With
the @code{--all} option, the complete build directory will be removed.

Extension modules built @ref{4908,,in place}
will not be cleaned, as they are not in the build directory.

@node distutils command config — Perform package configuration,distutils command install — Install a package,distutils command clean — Clean a package build area,API Reference
@anchor{distutils/apiref distutils-command-config-perform-package-configuration}@anchor{49c2}@anchor{distutils/apiref module-distutils command config}@anchor{48}
@subsection @code{distutils.command.config} — Perform package configuration


@geindex module; distutils.command.config

@c % todo

@node distutils command install — Install a package,distutils command install_data — Install data files from a package,distutils command config — Perform package configuration,API Reference
@anchor{distutils/apiref distutils-command-install-install-a-package}@anchor{49c3}@anchor{distutils/apiref module-distutils command install}@anchor{49}
@subsection @code{distutils.command.install} — Install a package


@geindex module; distutils.command.install

@c % todo

@node distutils command install_data — Install data files from a package,distutils command install_headers — Install C/C++ header files from a package,distutils command install — Install a package,API Reference
@anchor{distutils/apiref distutils-command-install-data-install-data-files-from-a-package}@anchor{49c4}@anchor{distutils/apiref module-distutils command install_data}@anchor{4a}
@subsection @code{distutils.command.install_data} — Install data files from a package


@geindex module; distutils.command.install_data

@c % todo

@node distutils command install_headers — Install C/C++ header files from a package,distutils command install_lib — Install library files from a package,distutils command install_data — Install data files from a package,API Reference
@anchor{distutils/apiref distutils-command-install-headers-install-c-c-header-files-from-a-package}@anchor{49c5}@anchor{distutils/apiref module-distutils command install_headers}@anchor{4b}
@subsection @code{distutils.command.install_headers} — Install C/C++ header files from a package


@geindex module; distutils.command.install_headers

@c % todo

@node distutils command install_lib — Install library files from a package,distutils command install_scripts — Install script files from a package,distutils command install_headers — Install C/C++ header files from a package,API Reference
@anchor{distutils/apiref distutils-command-install-lib-install-library-files-from-a-package}@anchor{49c6}@anchor{distutils/apiref module-distutils command install_lib}@anchor{4c}
@subsection @code{distutils.command.install_lib} — Install library files from a package


@geindex module; distutils.command.install_lib

@c % todo

@node distutils command install_scripts — Install script files from a package,distutils command register — Register a module with the Python Package Index,distutils command install_lib — Install library files from a package,API Reference
@anchor{distutils/apiref distutils-command-install-scripts-install-script-files-from-a-package}@anchor{49c7}@anchor{distutils/apiref module-distutils command install_scripts}@anchor{4d}
@subsection @code{distutils.command.install_scripts} — Install script files from a package


@geindex module; distutils.command.install_scripts

@c % todo

@node distutils command register — Register a module with the Python Package Index,distutils command check — Check the meta-data of a package,distutils command install_scripts — Install script files from a package,API Reference
@anchor{distutils/apiref distutils-command-register-register-a-module-with-the-python-package-index}@anchor{49c8}@anchor{distutils/apiref module-distutils command register}@anchor{4e}
@subsection @code{distutils.command.register} — Register a module with the Python Package Index


@geindex module; distutils.command.register

The @code{register} command registers the package with the Python Package  Index.
This is described in more detail in 
@geindex Python Enhancement Proposals; PEP 301
PEP 301@footnote{https://www.python.org/dev/peps/pep-0301}.

@c % todo

@node distutils command check — Check the meta-data of a package,,distutils command register — Register a module with the Python Package Index,API Reference
@anchor{distutils/apiref distutils-command-check-check-the-meta-data-of-a-package}@anchor{49c9}@anchor{distutils/apiref module-distutils command check}@anchor{46}
@subsection @code{distutils.command.check} — Check the meta-data of a package


@geindex module; distutils.command.check

The @code{check} command performs some tests on the meta-data of a package.
For example, it verifies that all required meta-data are provided as
the arguments passed to the @code{setup()} function.

@c % todo

@node Installing Python Modules Legacy version,Python Module Index,Distributing Python Modules Legacy version,Top
@anchor{install/index doc}@anchor{49ca}@anchor{install/index install-index}@anchor{a6b}@anchor{install/index installing-python-modules-legacy-version}@anchor{49cb}
@chapter Installing Python Modules (Legacy version)


@*Author: 
Greg Ward


@c TODO: Fill in XXX comments

@cartouche
@quotation Note 
The entire @code{distutils} package has been deprecated and will be
removed in Python 3.12. This documentation is retained as a
reference only, and will be removed with the package. See the
@ref{279,,What’s New} entry for more information.
@end quotation
@end cartouche


@subsubheading See also



@table @asis

@item @ref{a69,,Installing Python Modules}

The up to date module installation documentation. For regular Python
usage, you almost certainly want that document rather than this one.
@end table


@cartouche
@quotation Note 
This document is being retained solely until the @code{setuptools} documentation
at @indicateurl{https://setuptools.readthedocs.io/en/latest/setuptools.html}
independently covers all of the relevant information currently included here.
@end quotation
@end cartouche

@cartouche
@quotation Note 
This guide only covers the basic tools for building and distributing
extensions that are provided as part of this version of Python. Third party
tools offer easier to use and more secure alternatives. Refer to the quick recommendations section@footnote{https://packaging.python.org/guides/tool-recommendations/}
in the Python Packaging User Guide for more information.
@end quotation
@end cartouche

@menu
* Introduction: Introduction<17>. 
* Standard Build and Install:: 
* Alternate Installation:: 
* Custom Installation:: 
* Distutils Configuration Files:: 
* Building Extensions; Tips and Tricks: Building Extensions Tips and Tricks. 

@end menu

@node Introduction<17>,Standard Build and Install,,Installing Python Modules Legacy version
@anchor{install/index inst-intro}@anchor{49cc}@anchor{install/index introduction}@anchor{49cd}
@section Introduction


In Python 2.0, the @code{distutils} API was first added to the standard library.
This provided Linux distro maintainers with a standard way of converting
Python projects into Linux distro packages, and system administrators with a
standard way of installing them directly onto target systems.

In the many years since Python 2.0 was released, tightly coupling the build
system and package installer to the language runtime release cycle has turned
out to be problematic, and it is now recommended that projects use the
@code{pip} package installer and the @code{setuptools} build system, rather than
using @code{distutils} directly.

See @ref{a69,,Installing Python Modules} and @ref{a6a,,Distributing Python Modules} for more details.

This legacy documentation is being retained only until we’re confident that the
@code{setuptools} documentation covers everything needed.

@menu
* Distutils based source distributions:: 

@end menu

@node Distutils based source distributions,,,Introduction<17>
@anchor{install/index distutils-based-source-distributions}@anchor{49ce}@anchor{install/index inst-new-standard}@anchor{49cf}
@subsection Distutils based source distributions


If you download a module source distribution, you can tell pretty quickly if it
was packaged and distributed in the standard way, i.e. using the Distutils.
First, the distribution’s name and version number will be featured prominently
in the name of the downloaded archive, e.g. @code{foo-1.0.tar.gz} or
@code{widget-0.9.7.zip}.  Next, the archive will unpack into a similarly-named
directory: @code{foo-1.0} or @code{widget-0.9.7}.  Additionally, the
distribution will contain a setup script @code{setup.py}, and a file named
@code{README.txt} or possibly just @code{README}, which should explain that
building and installing the module distribution is a simple matter of running
one command from a terminal:

@example
python setup.py install
@end example

For Windows, this command should be run from a command prompt window
(Start ‣ Accessories):

@example
setup.py install
@end example

If all these things are true, then you already know how to build and install the
modules you’ve just downloaded:  Run the command above. Unless you need to
install things in a non-standard way or customize the build process, you don’t
really need this manual.  Or rather, the above command is everything you need to
get out of this manual.

@node Standard Build and Install,Alternate Installation,Introduction<17>,Installing Python Modules Legacy version
@anchor{install/index inst-standard-install}@anchor{49d0}@anchor{install/index standard-build-and-install}@anchor{49d1}
@section Standard Build and Install


As described in section @ref{49cf,,Distutils based source distributions}, building and installing a module
distribution using the Distutils is usually one simple command to run from a
terminal:

@example
python setup.py install
@end example

@menu
* Platform variations:: 
* Splitting the job up:: 
* How building works:: 
* How installation works:: 

@end menu

@node Platform variations,Splitting the job up,,Standard Build and Install
@anchor{install/index inst-platform-variations}@anchor{49d2}@anchor{install/index platform-variations}@anchor{49d3}
@subsection Platform variations


You should always run the setup command from the distribution root directory,
i.e. the top-level subdirectory that the module source distribution unpacks
into.  For example, if you’ve just downloaded a module source distribution
@code{foo-1.0.tar.gz} onto a Unix system, the normal thing to do is:

@example
gunzip -c foo-1.0.tar.gz | tar xf -    # unpacks into directory foo-1.0
cd foo-1.0
python setup.py install
@end example

On Windows, you’d probably download @code{foo-1.0.zip}.  If you downloaded the
archive file to @code{C:\Temp}, then it would unpack into
@code{C:\Temp\foo-1.0}; you can use either an archive manipulator with a
graphical user interface (such as WinZip) or a command-line tool (such as
@code{unzip} or @code{pkunzip}) to unpack the archive.  Then, open a
command prompt window and run:

@example
cd c:\Temp\foo-1.0
python setup.py install
@end example

@node Splitting the job up,How building works,Platform variations,Standard Build and Install
@anchor{install/index inst-splitting-up}@anchor{49d4}@anchor{install/index splitting-the-job-up}@anchor{49d5}
@subsection Splitting the job up


Running @code{setup.py install} builds and installs all modules in one run.  If you
prefer to work incrementally—especially useful if you want to customize the
build process, or if things are going wrong—you can use the setup script to do
one thing at a time.  This is particularly helpful when the build and install
will be done by different users—for example, you might want to build a module
distribution and hand it off to a system administrator for installation (or do
it yourself, with super-user privileges).

For example, you can build everything in one step, and then install everything
in a second step, by invoking the setup script twice:

@example
python setup.py build
python setup.py install
@end example

If you do this, you will notice that running the @code{install} command
first runs the @code{build} command, which—in this case—quickly notices
that it has nothing to do, since everything in the @code{build} directory is
up-to-date.

You may not need this ability to break things down often if all you do is
install modules downloaded off the ‘net, but it’s very handy for more advanced
tasks.  If you get into distributing your own Python modules and extensions,
you’ll run lots of individual Distutils commands on their own.

@node How building works,How installation works,Splitting the job up,Standard Build and Install
@anchor{install/index how-building-works}@anchor{49d6}@anchor{install/index inst-how-build-works}@anchor{49d7}
@subsection How building works


As implied above, the @code{build} command is responsible for putting the
files to install into a @emph{build directory}.  By default, this is @code{build}
under the distribution root; if you’re excessively concerned with speed, or want
to keep the source tree pristine, you can change the build directory with the
@code{--build-base} option. For example:

@example
python setup.py build --build-base=/path/to/pybuild/foo-1.0
@end example

(Or you could do this permanently with a directive in your system or personal
Distutils configuration file; see section @ref{49d8,,Distutils Configuration Files}.)  Normally, this
isn’t necessary.

The default layout for the build tree is as follows:

@example
--- build/ --- lib/
or
--- build/ --- lib.<plat>/
               temp.<plat>/
@end example

where @code{<plat>} expands to a brief description of the current OS/hardware
platform and Python version.  The first form, with just a @code{lib} directory,
is used for “pure module distributions”—that is, module distributions that
include only pure Python modules.  If a module distribution contains any
extensions (modules written in C/C++), then the second form, with two @code{<plat>}
directories, is used.  In that case, the @code{temp.@emph{plat}} directory holds
temporary files generated by the compile/link process that don’t actually get
installed.  In either case, the @code{lib} (or @code{lib.@emph{plat}}) directory
contains all Python modules (pure Python and extensions) that will be installed.

In the future, more directories will be added to handle Python scripts,
documentation, binary executables, and whatever else is needed to handle the job
of installing Python modules and applications.

@node How installation works,,How building works,Standard Build and Install
@anchor{install/index how-installation-works}@anchor{49d9}@anchor{install/index inst-how-install-works}@anchor{49da}
@subsection How installation works


After the @code{build} command runs (whether you run it explicitly, or the
@code{install} command does it for you), the work of the @code{install}
command is relatively simple: all it has to do is copy everything under
@code{build/lib} (or @code{build/lib.@emph{plat}}) to your chosen installation
directory.

If you don’t choose an installation directory—i.e., if you just run @code{setup.py
install}—then the @code{install} command installs to the standard
location for third-party Python modules.  This location varies by platform and
by how you built/installed Python itself.  On Unix (and macOS, which is also
Unix-based), it also depends on whether the module distribution being installed
is pure Python or contains extensions (“non-pure”):


@multitable {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} 
@headitem

Platform

@tab

Standard installation location

@tab

Default value

@tab

Notes

@item

Unix (pure)

@tab

@code{@emph{prefix}/lib/python@emph{X.Y}/site-packages}

@tab

@code{/usr/local/lib/python@emph{X.Y}/site-packages}

@tab

(1)

@item

Unix (non-pure)

@tab

@code{@emph{exec-prefix}/lib/python@emph{X.Y}/site-packages}

@tab

@code{/usr/local/lib/python@emph{X.Y}/site-packages}

@tab

(1)

@item

Windows

@tab

@code{@emph{prefix}\Lib\site-packages}

@tab

@code{C:\Python@emph{XY}\Lib\site-packages}

@tab

(2)

@end multitable


Notes:


@enumerate 

@item 
Most Linux distributions include Python as a standard part of the system, so
@code{@emph{prefix}} and @code{@emph{exec-prefix}} are usually both @code{/usr} on
Linux.  If you build Python yourself on Linux (or any Unix-like system), the
default @code{@emph{prefix}} and @code{@emph{exec-prefix}} are @code{/usr/local}.

@item 
The default installation directory on Windows was @code{C:\Program
Files\Python} under Python 1.6a1, 1.5.2, and earlier.
@end enumerate

@code{@emph{prefix}} and @code{@emph{exec-prefix}} stand for the directories that Python
is installed to, and where it finds its libraries at run-time.  They are always
the same under Windows, and very often the same under Unix and macOS.  You
can find out what your Python installation uses for @code{@emph{prefix}} and
@code{@emph{exec-prefix}} by running Python in interactive mode and typing a few
simple commands. Under Unix, just type @code{python} at the shell prompt.  Under
Windows, choose Start ‣ Programs ‣ Python X.Y ‣
Python (command line).   Once the interpreter is started, you type Python code
at the prompt.  For example, on my Linux system, I type the three Python
statements shown below, and get the output as shown, to find out my
@code{@emph{prefix}} and @code{@emph{exec-prefix}}:

@example
Python 2.4 (#26, Aug  7 2004, 17:19:02)
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/usr'
>>> sys.exec_prefix
'/usr'
@end example

A few other placeholders are used in this document: @code{@emph{X.Y}} stands for the
version of Python, for example @code{3.2}; @code{@emph{abiflags}} will be replaced by
the value of @ref{57a,,sys.abiflags} or the empty string for platforms which don’t
define ABI flags; @code{@emph{distname}} will be replaced by the name of the module
distribution being installed.  Dots and capitalization are important in the
paths; for example, a value that uses @code{python3.2} on UNIX will typically use
@code{Python32} on Windows.

If you don’t want to install modules to the standard location, or if you don’t
have permission to write there, then you need to read about alternate
installations in section @ref{49db,,Alternate Installation}.  If you want to customize your
installation directories more heavily, see section @ref{49dc,,Custom Installation} on
custom installations.

@node Alternate Installation,Custom Installation,Standard Build and Install,Installing Python Modules Legacy version
@anchor{install/index alternate-installation}@anchor{49dd}@anchor{install/index inst-alt-install}@anchor{49db}
@section Alternate Installation


Often, it is necessary or desirable to install modules to a location other than
the standard location for third-party Python modules.  For example, on a Unix
system you might not have permission to write to the standard third-party module
directory.  Or you might wish to try out a module before making it a standard
part of your local Python installation.  This is especially true when upgrading
a distribution already present: you want to make sure your existing base of
scripts still works with the new version before actually upgrading.

The Distutils @code{install} command is designed to make installing module
distributions to an alternate location simple and painless.  The basic idea is
that you supply a base directory for the installation, and the
@code{install} command picks a set of directories (called an @emph{installation
scheme}) under this base directory in which to install files.  The details
differ across platforms, so read whichever of the following sections applies to
you.

Note that the various alternate installation schemes are mutually exclusive: you
can pass @code{--user}, or @code{--home}, or @code{--prefix} and @code{--exec-prefix}, or
@code{--install-base} and @code{--install-platbase}, but you can’t mix from these
groups.

@menu
* Alternate installation; the user scheme: Alternate installation the user scheme. 
* Alternate installation; the home scheme: Alternate installation the home scheme. 
* Alternate installation; Unix (the prefix scheme): Alternate installation Unix the prefix scheme. 
* Alternate installation; Windows (the prefix scheme): Alternate installation Windows the prefix scheme. 

@end menu

@node Alternate installation the user scheme,Alternate installation the home scheme,,Alternate Installation
@anchor{install/index alternate-installation-the-user-scheme}@anchor{49de}@anchor{install/index inst-alt-install-user}@anchor{1621}
@subsection Alternate installation: the user scheme


This scheme is designed to be the most convenient solution for users that don’t
have write permission to the global site-packages directory or don’t want to
install into it.  It is enabled with a simple option:

@example
python setup.py install --user
@end example

Files will be installed into subdirectories of @ref{1620,,site.USER_BASE} (written
as @code{@emph{userbase}} hereafter).  This scheme installs pure Python modules and
extension modules in the same location (also known as @ref{160f,,site.USER_SITE}).
Here are the values for UNIX, including macOS:


@multitable {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Type of file

@tab

Installation directory

@item

modules

@tab

@code{@emph{userbase}/lib/python@emph{X.Y}/site-packages}

@item

scripts

@tab

@code{@emph{userbase}/bin}

@item

data

@tab

@code{@emph{userbase}}

@item

C headers

@tab

@code{@emph{userbase}/include/python@emph{X.Y}@emph{abiflags}/@emph{distname}}

@end multitable


And here are the values used on Windows:


@multitable {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Type of file

@tab

Installation directory

@item

modules

@tab

@code{@emph{userbase}\Python@emph{XY}\site-packages}

@item

scripts

@tab

@code{@emph{userbase}\Python@emph{XY}\Scripts}

@item

data

@tab

@code{@emph{userbase}}

@item

C headers

@tab

@code{@emph{userbase}\Python@emph{XY}\Include@{distname@}}

@end multitable


The advantage of using this scheme compared to the other ones described below is
that the user site-packages directory is under normal conditions always included
in @ref{489,,sys.path} (see @ref{e6,,site} for more information), which means that
there is no additional step to perform after running the @code{setup.py} script
to finalize the installation.

The @code{build_ext} command also has a @code{--user} option to add
@code{@emph{userbase}/include} to the compiler search path for header files and
@code{@emph{userbase}/lib} to the compiler search path for libraries as well as to
the runtime search path for shared C libraries (rpath).

@node Alternate installation the home scheme,Alternate installation Unix the prefix scheme,Alternate installation the user scheme,Alternate Installation
@anchor{install/index alternate-installation-the-home-scheme}@anchor{49df}@anchor{install/index inst-alt-install-home}@anchor{49e0}
@subsection Alternate installation: the home scheme


The idea behind the “home scheme” is that you build and maintain a personal
stash of Python modules.  This scheme’s name is derived from the idea of a
“home” directory on Unix, since it’s not unusual for a Unix user to make their
home directory have a layout similar to @code{/usr/} or @code{/usr/local/}.
This scheme can be used by anyone, regardless of the operating system they
are installing for.

Installing a new module distribution is as simple as

@example
python setup.py install --home=<dir>
@end example

where you can supply any directory you like for the @code{--home} option.  On
Unix, lazy typists can just type a tilde (@code{~}); the @code{install} command
will expand this to your home directory:

@example
python setup.py install --home=~
@end example

To make Python find the distributions installed with this scheme, you may have
to @ref{49e1,,modify Python’s search path} or edit
@code{sitecustomize} (see @ref{e6,,site}) to call @ref{3c4d,,site.addsitedir()} or edit
@ref{489,,sys.path}.

The @code{--home} option defines the installation base directory.  Files are
installed to the following directories under the installation base as follows:


@multitable {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Type of file

@tab

Installation directory

@item

modules

@tab

@code{@emph{home}/lib/python}

@item

scripts

@tab

@code{@emph{home}/bin}

@item

data

@tab

@code{@emph{home}}

@item

C headers

@tab

@code{@emph{home}/include/python/@emph{distname}}

@end multitable


(Mentally replace slashes with backslashes if you’re on Windows.)

@node Alternate installation Unix the prefix scheme,Alternate installation Windows the prefix scheme,Alternate installation the home scheme,Alternate Installation
@anchor{install/index alternate-installation-unix-the-prefix-scheme}@anchor{49e2}@anchor{install/index inst-alt-install-prefix-unix}@anchor{49e3}
@subsection Alternate installation: Unix (the prefix scheme)


The “prefix scheme” is useful when you wish to use one Python installation to
perform the build/install (i.e., to run the setup script), but install modules
into the third-party module directory of a different Python installation (or
something that looks like a different Python installation).  If this sounds a
trifle unusual, it is—that’s why the user and home schemes come before.  However,
there are at least two known cases where the prefix scheme will be useful.

First, consider that many Linux distributions put Python in @code{/usr}, rather
than the more traditional @code{/usr/local}.  This is entirely appropriate,
since in those cases Python is part of “the system” rather than a local add-on.
However, if you are installing Python modules from source, you probably want
them to go in @code{/usr/local/lib/python2.@emph{X}} rather than
@code{/usr/lib/python2.@emph{X}}.  This can be done with

@example
/usr/bin/python setup.py install --prefix=/usr/local
@end example

Another possibility is a network filesystem where the name used to write to a
remote directory is different from the name used to read it: for example, the
Python interpreter accessed as @code{/usr/local/bin/python} might search for
modules in @code{/usr/local/lib/python2.@emph{X}}, but those modules would have to
be installed to, say, @code{/mnt/@emph{@@server}/export/lib/python2.@emph{X}}.  This could
be done with

@example
/usr/local/bin/python setup.py install --prefix=/mnt/@@server/export
@end example

In either case, the @code{--prefix} option defines the installation base, and
the @code{--exec-prefix} option defines the platform-specific installation
base, which is used for platform-specific files.  (Currently, this just means
non-pure module distributions, but could be expanded to C libraries, binary
executables, etc.)  If @code{--exec-prefix} is not supplied, it defaults to
@code{--prefix}.  Files are installed as follows:


@multitable {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Type of file

@tab

Installation directory

@item

Python modules

@tab

@code{@emph{prefix}/lib/python@emph{X.Y}/site-packages}

@item

extension modules

@tab

@code{@emph{exec-prefix}/lib/python@emph{X.Y}/site-packages}

@item

scripts

@tab

@code{@emph{prefix}/bin}

@item

data

@tab

@code{@emph{prefix}}

@item

C headers

@tab

@code{@emph{prefix}/include/python@emph{X.Y}@emph{abiflags}/@emph{distname}}

@end multitable


There is no requirement that @code{--prefix} or @code{--exec-prefix}
actually point to an alternate Python installation; if the directories listed
above do not already exist, they are created at installation time.

Incidentally, the real reason the prefix scheme is important is simply that a
standard Unix installation uses the prefix scheme, but with @code{--prefix}
and @code{--exec-prefix} supplied by Python itself as @code{sys.prefix} and
@code{sys.exec_prefix}.  Thus, you might think you’ll never use the prefix scheme,
but every time you run @code{python setup.py install} without any other options,
you’re using it.

Note that installing extensions to an alternate Python installation has no
effect on how those extensions are built: in particular, the Python header files
(@code{Python.h} and friends) installed with the Python interpreter used to run
the setup script will be used in compiling extensions.  It is your
responsibility to ensure that the interpreter used to run extensions installed
in this way is compatible with the interpreter used to build them.  The best way
to do this is to ensure that the two interpreters are the same version of Python
(possibly different builds, or possibly copies of the same build).  (Of course,
if your @code{--prefix} and @code{--exec-prefix} don’t even point to an
alternate Python installation, this is immaterial.)

@node Alternate installation Windows the prefix scheme,,Alternate installation Unix the prefix scheme,Alternate Installation
@anchor{install/index alternate-installation-windows-the-prefix-scheme}@anchor{49e4}@anchor{install/index inst-alt-install-prefix-windows}@anchor{49e5}
@subsection Alternate installation: Windows (the prefix scheme)


Windows has no concept of a user’s home directory, and since the standard Python
installation under Windows is simpler than under Unix, the @code{--prefix}
option has traditionally been used to install additional packages in separate
locations on Windows.

@example
python setup.py install --prefix="\Temp\Python"
@end example

to install modules to the @code{\Temp\Python} directory on the current drive.

The installation base is defined by the @code{--prefix} option; the
@code{--exec-prefix} option is not supported under Windows, which means that
pure Python modules and extension modules are installed into the same location.
Files are installed as follows:


@multitable {xxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Type of file

@tab

Installation directory

@item

modules

@tab

@code{@emph{prefix}\Lib\site-packages}

@item

scripts

@tab

@code{@emph{prefix}\Scripts}

@item

data

@tab

@code{@emph{prefix}}

@item

C headers

@tab

@code{@emph{prefix}\Include@{distname@}}

@end multitable


@node Custom Installation,Distutils Configuration Files,Alternate Installation,Installing Python Modules Legacy version
@anchor{install/index custom-installation}@anchor{49e6}@anchor{install/index inst-custom-install}@anchor{49dc}
@section Custom Installation


Sometimes, the alternate installation schemes described in section
@ref{49db,,Alternate Installation} just don’t do what you want.  You might want to tweak just
one or two directories while keeping everything under the same base directory,
or you might want to completely redefine the installation scheme.  In either
case, you’re creating a @emph{custom installation scheme}.

To create a custom installation scheme, you start with one of the alternate
schemes and override some of the installation directories used for the various
types of files, using these options:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Type of file

@tab

Override option

@item

Python modules

@tab

@code{--install-purelib}

@item

extension modules

@tab

@code{--install-platlib}

@item

all modules

@tab

@code{--install-lib}

@item

scripts

@tab

@code{--install-scripts}

@item

data

@tab

@code{--install-data}

@item

C headers

@tab

@code{--install-headers}

@end multitable


These override options can be relative, absolute,
or explicitly defined in terms of one of the installation base directories.
(There are two installation base directories, and they are normally the
same—they only differ when you use the Unix “prefix scheme” and supply
different @code{--prefix} and @code{--exec-prefix} options; using @code{--install-lib}
will override values computed or given for @code{--install-purelib} and
@code{--install-platlib}, and is recommended for schemes that don’t make a
difference between Python and extension modules.)

For example, say you’re installing a module distribution to your home directory
under Unix—but you want scripts to go in @code{~/scripts} rather than
@code{~/bin}. As you might expect, you can override this directory with the
@code{--install-scripts} option; in this case, it makes most sense to supply
a relative path, which will be interpreted relative to the installation base
directory (your home directory, in this case):

@example
python setup.py install --home=~ --install-scripts=scripts
@end example

Another Unix example: suppose your Python installation was built and installed
with a prefix of @code{/usr/local/python}, so under a standard  installation
scripts will wind up in @code{/usr/local/python/bin}.  If you want them in
@code{/usr/local/bin} instead, you would supply this absolute directory for the
@code{--install-scripts} option:

@example
python setup.py install --install-scripts=/usr/local/bin
@end example

(This performs an installation using the “prefix scheme”, where the prefix is
whatever your Python interpreter was installed with— @code{/usr/local/python}
in this case.)

If you maintain Python on Windows, you might want third-party modules to live in
a subdirectory of @code{@emph{prefix}}, rather than right in @code{@emph{prefix}}
itself.  This is almost as easy as customizing the script installation
directory—you just have to remember that there are two types of modules
to worry about, Python and extension modules, which can conveniently be both
controlled by one option:

@example
python setup.py install --install-lib=Site
@end example

The specified installation directory is relative to @code{@emph{prefix}}.  Of
course, you also have to ensure that this directory is in Python’s module
search path, such as by putting a @code{.pth} file in a site directory (see
@ref{e6,,site}).  See section @ref{49e1,,Modifying Python’s Search Path} to find out how to modify
Python’s search path.

If you want to define an entire installation scheme, you just have to supply all
of the installation directory options.  The recommended way to do this is to
supply relative paths; for example, if you want to maintain all Python
module-related files under @code{python} in your home directory, and you want a
separate directory for each platform that you use your home directory from, you
might define the following installation scheme:

@example
python setup.py install --home=~ \
                        --install-purelib=python/lib \
                        --install-platlib=python/lib.$PLAT \
                        --install-scripts=python/scripts
                        --install-data=python/data
@end example

or, equivalently,

@example
python setup.py install --home=~/python \
                        --install-purelib=lib \
                        --install-platlib='lib.$PLAT' \
                        --install-scripts=scripts
                        --install-data=data
@end example

@code{$PLAT} is not (necessarily) an environment variable—it will be expanded by
the Distutils as it parses your command line options, just as it does when
parsing your configuration file(s).

Obviously, specifying the entire installation scheme every time you install a
new module distribution would be very tedious.  Thus, you can put these options
into your Distutils config file (see section @ref{49d8,,Distutils Configuration Files}):

@example
[install]
install-base=$HOME
install-purelib=python/lib
install-platlib=python/lib.$PLAT
install-scripts=python/scripts
install-data=python/data
@end example

or, equivalently,

@example
[install]
install-base=$HOME/python
install-purelib=lib
install-platlib=lib.$PLAT
install-scripts=scripts
install-data=data
@end example

Note that these two are @emph{not} equivalent if you supply a different installation
base directory when you run the setup script.  For example,

@example
python setup.py install --install-base=/tmp
@end example

would install pure modules to @code{/tmp/python/lib} in the first case, and
to @code{/tmp/lib} in the second case.  (For the second case, you probably
want to supply an installation base of @code{/tmp/python}.)

You probably noticed the use of @code{$HOME} and @code{$PLAT} in the sample
configuration file input.  These are Distutils configuration variables, which
bear a strong resemblance to environment variables. In fact, you can use
environment variables in config files on platforms that have such a notion but
the Distutils additionally define a few extra variables that may not be in your
environment, such as @code{$PLAT}.  (And of course, on systems that don’t have
environment variables, such as Mac OS 9, the configuration variables supplied by
the Distutils are the only ones you can use.) See section @ref{49d8,,Distutils Configuration Files}
for details.

@cartouche
@quotation Note 
When a @ref{3b33,,virtual environment} is activated, any options
that change the installation path will be ignored from all distutils configuration
files to prevent inadvertently installing projects outside of the virtual
environment.
@end quotation
@end cartouche

@c XXX need some Windows examples---when would custom installation schemes be
@c needed on those platforms?

@c XXX Move this to Doc/using

@menu
* Modifying Python’s Search Path:: 

@end menu

@node Modifying Python’s Search Path,,,Custom Installation
@anchor{install/index inst-search-path}@anchor{49e1}@anchor{install/index modifying-python-s-search-path}@anchor{49e7}
@subsection Modifying Python’s Search Path


When the Python interpreter executes an @ref{e35,,import} statement, it searches
for both Python code and extension modules along a search path.  A default value
for the path is configured into the Python binary when the interpreter is built.
You can determine the path by importing the @ref{f7,,sys} module and printing the
value of @code{sys.path}.

@example
$ python
Python 2.2 (#11, Oct  3 2002, 13:31:27)
[GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-112)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['@w{'}, '/usr/local/lib/python2.3', '/usr/local/lib/python2.3/plat-linux2',
 '/usr/local/lib/python2.3/lib-tk', '/usr/local/lib/python2.3/lib-dynload',
 '/usr/local/lib/python2.3/site-packages']
>>>
@end example

The null string in @code{sys.path} represents the current working directory.

The expected convention for locally installed packages is to put them in the
@code{@emph{...}/site-packages/} directory, but you may want to install Python
modules into some arbitrary directory.  For example, your site may have a
convention of keeping all software related to the web server under @code{/www}.
Add-on Python modules might then belong in @code{/www/python}, and in order to
import them, this directory must be added to @code{sys.path}.  There are several
different ways to add the directory.

The most convenient way is to add a path configuration file to a directory
that’s already on Python’s path, usually to the @code{.../site-packages/}
directory.  Path configuration files have an extension of @code{.pth}, and each
line must contain a single path that will be appended to @code{sys.path}.  (Because
the new paths are appended to @code{sys.path}, modules in the added directories
will not override standard modules.  This means you can’t use this mechanism for
installing fixed versions of standard modules.)

Paths can be absolute or relative, in which case they’re relative to the
directory containing the @code{.pth} file.  See the documentation of
the @ref{e6,,site} module for more information.

A slightly less convenient way is to edit the @code{site.py} file in Python’s
standard library, and modify @code{sys.path}.  @code{site.py} is automatically
imported when the Python interpreter is executed, unless the @ref{d4e,,-S} switch
is supplied to suppress this behaviour.  So you could simply edit
@code{site.py} and add two lines to it:

@example
import sys
sys.path.append('/www/python/')
@end example

However, if you reinstall the same major version of Python (perhaps when
upgrading from 2.2 to 2.2.2, for example) @code{site.py} will be overwritten by
the stock version.  You’d have to remember that it was modified and save a copy
before doing the installation.

There are two environment variables that can modify @code{sys.path}.
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME} sets an alternate value for the prefix of the Python
installation.  For example, if 
@geindex PYTHONHOME
@geindex environment variable; PYTHONHOME
@ref{780,,PYTHONHOME} is set to @code{/www/python},
the search path will be set to @code{['@w{'}, '/www/python/lib/pythonX.Y/',
'/www/python/lib/pythonX.Y/plat-linux2', ...]}.

The 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} variable can be set to a list of paths that will be
added to the beginning of @code{sys.path}.  For example, if 
@geindex PYTHONPATH
@geindex environment variable; PYTHONPATH
@ref{bac,,PYTHONPATH} is
set to @code{/www/python:/opt/py}, the search path will begin with
@code{['/www/python', '/opt/py']}.  (Note that directories must exist in order to
be added to @code{sys.path}; the @ref{e6,,site} module removes paths that don’t
exist.)

Finally, @code{sys.path} is just a regular Python list, so any Python application
can modify it by adding or removing entries.

@node Distutils Configuration Files,Building Extensions Tips and Tricks,Custom Installation,Installing Python Modules Legacy version
@anchor{install/index distutils-configuration-files}@anchor{49e8}@anchor{install/index inst-config-files}@anchor{49d8}
@section Distutils Configuration Files


As mentioned above, you can use Distutils configuration files to record personal
or site preferences for any Distutils options.  That is, any option to any
command can be stored in one of two or three (depending on your platform)
configuration files, which will be consulted before the command-line is parsed.
This means that configuration files will override default values, and the
command-line will in turn override configuration files.  Furthermore, if
multiple configuration files apply, values from “earlier” files are overridden
by “later” files.

@menu
* Location and names of config files:: 
* Syntax of config files:: 

@end menu

@node Location and names of config files,Syntax of config files,,Distutils Configuration Files
@anchor{install/index inst-config-filenames}@anchor{49e9}@anchor{install/index location-and-names-of-config-files}@anchor{49ea}
@subsection Location and names of config files


The names and locations of the configuration files vary slightly across
platforms.  On Unix and macOS, the three configuration files (in the order
they are processed) are:


@multitable {xxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} 
@headitem

Type of file

@tab

Location and filename

@tab

Notes

@item

system

@tab

@code{@emph{prefix}/lib/python@emph{ver}/distutils/distutils.cfg}

@tab

(1)

@item

personal

@tab

@code{$HOME/.pydistutils.cfg}

@tab

(2)

@item

local

@tab

@code{setup.cfg}

@tab

(3)

@end multitable


And on Windows, the configuration files are:


@multitable {xxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxx} 
@headitem

Type of file

@tab

Location and filename

@tab

Notes

@item

system

@tab

@code{@emph{prefix}\Lib\distutils\distutils.cfg}

@tab

(4)

@item

personal

@tab

@code{%HOME%\pydistutils.cfg}

@tab

(5)

@item

local

@tab

@code{setup.cfg}

@tab

(3)

@end multitable


On all platforms, the “personal” file can be temporarily disabled by
passing the @cite{–no-user-cfg} option.

Notes:


@enumerate 

@item 
Strictly speaking, the system-wide configuration file lives in the directory
where the Distutils are installed; under Python 1.6 and later on Unix, this is
as shown. For Python 1.5.2, the Distutils will normally be installed to
@code{@emph{prefix}/lib/python1.5/site-packages/distutils}, so the system
configuration file should be put there under Python 1.5.2.

@item 
On Unix, if the 
@geindex HOME
@geindex environment variable; HOME
@code{HOME} environment variable is not defined, the user’s
home directory will be determined with the @code{getpwuid()} function from the
standard @ref{d1,,pwd} module. This is done by the @ref{520,,os.path.expanduser()}
function used by Distutils.

@item 
I.e., in the current directory (usually the location of the setup script).

@item 
(See also note (1).)  Under Python 1.6 and later, Python’s default “installation
prefix” is @code{C:\Python}, so the system configuration file is normally
@code{C:\Python\Lib\distutils\distutils.cfg}. Under Python 1.5.2, the
default prefix was @code{C:\Program Files\Python}, and the Distutils were not
part of the standard library—so the system configuration file would be
@code{C:\Program Files\Python\distutils\distutils.cfg} in a standard Python
1.5.2 installation under Windows.

@item 
On Windows, if the 
@geindex HOME
@geindex environment variable; HOME
@code{HOME} environment variable is not defined,
@geindex USERPROFILE
@geindex environment variable; USERPROFILE
@code{USERPROFILE} then 
@geindex HOMEDRIVE
@geindex environment variable; HOMEDRIVE
@code{HOMEDRIVE} and 
@geindex HOMEPATH
@geindex environment variable; HOMEPATH
@code{HOMEPATH} will
be tried. This is done by the @ref{520,,os.path.expanduser()} function used
by Distutils.
@end enumerate

@node Syntax of config files,,Location and names of config files,Distutils Configuration Files
@anchor{install/index inst-config-syntax}@anchor{4909}@anchor{install/index syntax-of-config-files}@anchor{49eb}
@subsection Syntax of config files


The Distutils configuration files all have the same syntax.  The config files
are grouped into sections.  There is one section for each Distutils command,
plus a @code{global} section for global options that affect every command.  Each
section consists of one option per line, specified as @code{option=value}.

For example, the following is a complete config file that just forces all
commands to run quietly by default:

@example
[global]
verbose=0
@end example

If this is installed as the system config file, it will affect all processing of
any Python module distribution by any user on the current system.  If it is
installed as your personal config file (on systems that support them), it will
affect only module distributions processed by you.  And if it is used as the
@code{setup.cfg} for a particular module distribution, it affects only that
distribution.

You could override the default “build base” directory and make the
@code{build*} commands always forcibly rebuild all files with the
following:

@example
[build]
build-base=blib
force=1
@end example

which corresponds to the command-line arguments

@example
python setup.py build --build-base=blib --force
@end example

except that including the @code{build} command on the command-line means
that command will be run.  Including a particular command in config files has no
such implication; it only means that if the command is run, the options in the
config file will apply.  (Or if other commands that derive values from it are
run, they will use the values in the config file.)

You can find out the complete list of options for any command using the
@code{--help} option, e.g.:

@example
python setup.py build --help
@end example

and you can find out the complete list of global options by using
@code{--help} without a command:

@example
python setup.py --help
@end example

See also the “Reference” section of the “Distributing Python Modules” manual.

@node Building Extensions Tips and Tricks,,Distutils Configuration Files,Installing Python Modules Legacy version
@anchor{install/index building-extensions-tips-and-tricks}@anchor{49ec}@anchor{install/index inst-building-ext}@anchor{49ed}
@section Building Extensions: Tips and Tricks


Whenever possible, the Distutils try to use the configuration information made
available by the Python interpreter used to run the @code{setup.py} script.
For example, the same compiler and linker flags used to compile Python will also
be used for compiling extensions.  Usually this will work well, but in
complicated situations this might be inappropriate.  This section discusses how
to override the usual Distutils behaviour.

@menu
* Tweaking compiler/linker flags:: 
* Using non-Microsoft compilers on Windows:: 

@end menu

@node Tweaking compiler/linker flags,Using non-Microsoft compilers on Windows,,Building Extensions Tips and Tricks
@anchor{install/index inst-tweak-flags}@anchor{49ee}@anchor{install/index tweaking-compiler-linker-flags}@anchor{49ef}
@subsection Tweaking compiler/linker flags


Compiling a Python extension written in C or C++ will sometimes require
specifying custom flags for the compiler and linker in order to use a particular
library or produce a special kind of object code. This is especially true if the
extension hasn’t been tested on your platform, or if you’re trying to
cross-compile Python.

In the most general case, the extension author might have foreseen that
compiling the extensions would be complicated, and provided a @code{Setup} file
for you to edit.  This will likely only be done if the module distribution
contains many separate extension modules, or if they often require elaborate
sets of compiler flags in order to work.

A @code{Setup} file, if present, is parsed in order to get a list of extensions
to build.  Each line in a @code{Setup} describes a single module.  Lines have
the following structure:

@example
module ... [sourcefile ...] [cpparg ...] [library ...]
@end example

Let’s examine each of the fields in turn.


@itemize *

@item 
@emph{module} is the name of the extension module to be built, and should be a
valid Python identifier.  You can’t just change this in order to rename a module
(edits to the source code would also be needed), so this should be left alone.

@item 
@emph{sourcefile} is anything that’s likely to be a source code file, at least
judging by the filename.  Filenames ending in @code{.c} are assumed to be
written in C, filenames ending in @code{.C}, @code{.cc}, and @code{.c++} are
assumed to be C++, and filenames ending in @code{.m} or @code{.mm} are assumed
to be in Objective C.

@item 
@emph{cpparg} is an argument for the C preprocessor,  and is anything starting with
@code{-I}, @code{-D}, @code{-U} or @code{-C}.

@item 
@emph{library} is anything ending in @code{.a} or beginning with @code{-l} or
@code{-L}.
@end itemize

If a particular platform requires a special library on your platform, you can
add it by editing the @code{Setup} file and running @code{python setup.py build}.
For example, if the module defined by the line

@example
foo foomodule.c
@end example

must be linked with the math library @code{libm.a} on your platform, simply add
@code{-lm} to the line:

@example
foo foomodule.c -lm
@end example

Arbitrary switches intended for the compiler or the linker can be supplied with
the @code{-Xcompiler} @emph{arg} and @code{-Xlinker} @emph{arg} options:

@example
foo foomodule.c -Xcompiler -o32 -Xlinker -shared -lm
@end example

The next option after @code{-Xcompiler} and @code{-Xlinker} will be
appended to the proper command line, so in the above example the compiler will
be passed the @code{-o32} option, and the linker will be passed
@code{-shared}.  If a compiler option requires an argument, you’ll have to
supply multiple @code{-Xcompiler} options; for example, to pass @code{-x c++}
the @code{Setup} file would have to contain @code{-Xcompiler -x -Xcompiler c++}.

Compiler flags can also be supplied through setting the 
@geindex CFLAGS
@geindex environment variable; CFLAGS
@ref{1005,,CFLAGS}
environment variable.  If set, the contents of 
@geindex CFLAGS
@geindex environment variable; CFLAGS
@ref{1005,,CFLAGS} will be added to
the compiler flags specified in the  @code{Setup} file.

@node Using non-Microsoft compilers on Windows,,Tweaking compiler/linker flags,Building Extensions Tips and Tricks
@anchor{install/index inst-non-ms-compilers}@anchor{49f0}@anchor{install/index using-non-microsoft-compilers-on-windows}@anchor{49f1}
@subsection Using non-Microsoft compilers on Windows


@menu
* Borland/CodeGear C++:: 
* GNU C / Cygwin / MinGW:: 

@end menu

@node Borland/CodeGear C++,GNU C / Cygwin / MinGW,,Using non-Microsoft compilers on Windows
@anchor{install/index borland-codegear-c}@anchor{49f2}
@subsubsection Borland/CodeGear C++


This subsection describes the necessary steps to use Distutils with the Borland
C++ compiler version 5.5.  First you have to know that Borland’s object file
format (OMF) is different from the format used by the Python version you can
download from the Python or ActiveState web site.  (Python is built with
Microsoft Visual C++, which uses COFF as the object file format.) For this
reason you have to convert Python’s library @code{python25.lib} into the
Borland format.  You can do this as follows:

@c Should we mention that users have to create cfg-files for the compiler?

@c see also http://community.borland.com/article/0,1410,21205,00.html

@example
coff2omf python25.lib python25_bcpp.lib
@end example

The @code{coff2omf} program comes with the Borland compiler.  The file
@code{python25.lib} is in the @code{Libs} directory of your Python
installation.  If your extension uses other libraries (zlib, …) you have to
convert them too.

The converted files have to reside in the same directories as the normal
libraries.

How does Distutils manage to use these libraries with their changed names?  If
the extension needs a library (eg. @code{foo}) Distutils checks first if it
finds a library with suffix @code{_bcpp} (eg. @code{foo_bcpp.lib}) and then
uses this library.  In the case it doesn’t find such a special library it uses
the default name (@code{foo.lib}.) @footnote{@w{(1)} 
This also means you could replace all existing COFF-libraries with OMF-libraries
of the same name.
}

To let Distutils compile your extension with Borland C++ you now have to type:

@example
python setup.py build --compiler=bcpp
@end example

If you want to use the Borland C++ compiler as the default, you could specify
this in your personal or system-wide configuration file for Distutils (see
section @ref{49d8,,Distutils Configuration Files}.)


@subsubheading See also



@table @asis

@item C++Builder Compiler@footnote{https://www.embarcadero.com/products}

Information about the free C++ compiler from Borland, including links to the
download pages.

@item Creating Python Extensions Using Borland's Free Compiler@footnote{http://www.cyberus.ca/~g_will/pyExtenDL.shtml}

Document describing how to use Borland’s free command-line C++ compiler to build
Python.
@end table


@node GNU C / Cygwin / MinGW,,Borland/CodeGear C++,Using non-Microsoft compilers on Windows
@anchor{install/index gnu-c-cygwin-mingw}@anchor{49f3}
@subsubsection GNU C / Cygwin / MinGW


This section describes the necessary steps to use Distutils with the GNU C/C++
compilers in their Cygwin and MinGW distributions. @footnote{@w{(2)} 
Check @indicateurl{https://www.sourceware.org/cygwin/} for more information
} For a Python interpreter
that was built with Cygwin, everything should work without any of these
following steps.

Not all extensions can be built with MinGW or Cygwin, but many can.  Extensions
most likely to not work are those that use C++ or depend on Microsoft Visual C
extensions.

To let Distutils compile your extension with Cygwin you have to type:

@example
python setup.py build --compiler=cygwin
@end example

and for Cygwin in no-cygwin mode @footnote{@w{(3)} 
Then you have no POSIX emulation available, but you also don’t need
@code{cygwin1.dll}.
} or for MinGW type:

@example
python setup.py build --compiler=mingw32
@end example

If you want to use any of these options/compilers as default, you should
consider writing it in your personal or system-wide configuration file for
Distutils (see section @ref{49d8,,Distutils Configuration Files}.)

@menu
* Older Versions of Python and MinGW:: 

@end menu

@node Older Versions of Python and MinGW,,,GNU C / Cygwin / MinGW
@anchor{install/index older-versions-of-python-and-mingw}@anchor{49f4}
@subsubsection Older Versions of Python and MinGW


The following instructions only apply if you’re using a version of Python
inferior to 2.4.1 with a MinGW inferior to 3.0.0 (with
binutils-2.13.90-20030111-1).

These compilers require some special libraries.  This task is more complex than
for Borland’s C++, because there is no program to convert the library.  First
you have to create a list of symbols which the Python DLL exports. (You can find
a good program for this task at
@indicateurl{https://sourceforge.net/projects/mingw/files/MinGW/Extension/pexports/}).

@c I don't understand what the next line means. --amk

@c (inclusive the references on data structures.)

@example
pexports python25.dll >python25.def
@end example

The location of an installed @code{python25.dll} will depend on the
installation options and the version and language of Windows.  In a “just for
me” installation, it will appear in the root of the installation directory.  In
a shared installation, it will be located in the system directory.

Then you can create from these information an import library for gcc.

@example
/cygwin/bin/dlltool --dllname python25.dll --def python25.def --output-lib libpython25.a
@end example

The resulting library has to be placed in the same directory as
@code{python25.lib}. (Should be the @code{libs} directory under your Python
installation directory.)

If your extension uses other libraries (zlib,…) you might  have to convert
them too. The converted files have to reside in the same directories as the
normal libraries do.


@subsubheading See also



@table @asis

@item Building Python modules on MS Windows platform with MinGW@footnote{http://old.zope.org/Members/als/tips/win32_mingw_modules}

Information about building the required libraries for the MinGW environment.
@end table


@node Python Module Index,Index,Installing Python Modules Legacy version,Top
@unnumbered Python Module Index


@menu
* __future__: 0. Future statement definitions
* __main__: 1. The environment where top-level code is run. Covers
                        command-line interfaces@comma{} import-time
                        behavior@comma{} and `@w{`}__name__ ==
                        '__main__'`@w{`}.
* _thread: 2. Low-level threading API.
* abc: 3. Abstract base classes according to :pep:`3119`.
* aifc: 4. Read and write audio files in AIFF or AIFC format.
* argparse: 5. Command-line option and argument parsing library.
* array: 6. Space efficient arrays of uniformly typed numeric
                        values.
* ast: 7. Abstract Syntax Tree classes and manipulation.
* asynchat: 8. Support for asynchronous command/response protocols.
* asyncio: 9. Asynchronous I/O.
* asyncore: a. A base class for developing asynchronous socket
                        handling services.
* atexit: b. Register and execute cleanup functions.
* audioop: c. Manipulate raw audio data.
* base64: d. RFC 4648: Base16@comma{} Base32@comma{} Base64 Data
                        Encodings; Base85 and Ascii85
* bdb: e. Debugger framework.
* binascii: f. Tools for converting between binary and various ASCII-
                        encoded binary representations.
* bisect: 10. Array bisection algorithms for binary searching.
* builtins: 11. The module that provides the built-in namespace.
* bz2: 12. Interfaces for bzip2 compression and decompression.
* calendar: 13. Functions for working with calendars@comma{} including
                        some emulation of the Unix cal program.
* cgi: 14. Helpers for running Python scripts via the Common
                        Gateway Interface.
* cgitb: 15. Configurable traceback handler for CGI scripts.
* chunk: 16. Module to read IFF chunks.
* cmath: 17. Mathematical functions for complex numbers.
* cmd: 18. Build line-oriented command interpreters.
* code: 19. Facilities to implement read-eval-print loops.
* codecs: 1a. Encode and decode data and streams.
* codeop: 1b. Compile (possibly incomplete) Python code.
* collections: 1c. Container datatypes
* collections.abc: 1d. Abstract base classes for containers
* colorsys: 1e. Conversion functions between RGB and other color
                        systems.
* compileall: 1f. Tools for byte-compiling all Python source files in a
                        directory tree.
* concurrent.futures: 20. Execute computations concurrently using threads or
                        processes.
* configparser: 21. Configuration file parser.
* contextlib: 22. Utilities for with-statement contexts.
* contextvars: 23. Context Variables
* copy: 24. Shallow and deep copy operations.
* copyreg: 25. Register pickle support functions.
* cProfile: 26. 
* crypt: 27. The crypt() function used to check Unix passwords.
* csv: 28. Write and read tabular data to and from delimited
                        files.
* ctypes: 29. A foreign function library for Python.
* curses: 2a. An interface to the curses library@comma{} providing
                        portable terminal handling.
* curses.ascii: 2b. Constants and set-membership functions for ASCII
                        characters.
* curses.panel: 2c. A panel stack extension that adds depth to curses
                        windows.
* curses.textpad: 2d. Emacs-like input editing in a curses window.
* dataclasses: 2e. Generate special methods on user-defined classes.
* datetime: 2f. Basic date and time types.
* dbm: 30. Interfaces to various Unix "database" formats.
* dbm.dumb: 31. Portable implementation of the simple DBM interface.
* dbm.gnu: 32. GNU's reinterpretation of dbm.
* dbm.ndbm: 33. The standard "database" interface@comma{} based on
                        ndbm.
* decimal: 34. Implementation of the General Decimal Arithmetic
                        Specification.
* difflib: 35. Helpers for computing differences between objects.
* dis: 36. Disassembler for Python bytecode.
* distutils: 37. Support for building and installing Python modules
                        into an existing Python installation.
* distutils.archive_util: 38. Utility functions for creating archive files
                          (tarballs@comma{} zip files@comma{} ...)
* distutils.bcppcompiler: 39. 
* distutils.ccompiler: 3a. Abstract CCompiler class
* distutils.cmd: 3b. Provides the abstract base class
                        :class:`~distutils.cmd.Command`. This class is
                        subclassed by the modules in the distutils.command
                        subpackage.
* distutils.command: 3c. Contains one module for each standard Distutils
                        command.
* distutils.command.bdist: 3d. Build a binary installer for a package
* distutils.command.bdist_dumb: 3e. Build a "dumb" installer - a simple archive of
                                files
* distutils.command.bdist_packager: 3f. Abstract base class for packagers
* distutils.command.bdist_rpm: 40. Build a binary distribution as a Redhat RPM and
                               SRPM
* distutils.command.build: 41. Build all files of a package
* distutils.command.build_clib: 42. Build any C libraries in a package
* distutils.command.build_ext: 43. Build any extensions in a package
* distutils.command.build_py: 44. Build the .py/.pyc files of a package
* distutils.command.build_scripts: 45. Build the scripts of a package
* distutils.command.check: 46. Check the meta-data of a package
* distutils.command.clean: 47. Clean a package build area
* distutils.command.config: 48. Perform package configuration
* distutils.command.install: 49. Install a package
* distutils.command.install_data: 4a. Install data files from a package
* distutils.command.install_headers: 4b. Install C/C++ header files from a package
* distutils.command.install_lib: 4c. Install library files from a package
* distutils.command.install_scripts: 4d. Install script files from a package
* distutils.command.register: 4e. Register a module with the Python Package Index
* distutils.command.sdist: 4f. Build a source distribution
* distutils.core: 50. The core Distutils functionality
* distutils.cygwinccompiler: 51. 
* distutils.debug: 52. Provides the debug flag for distutils
* distutils.dep_util: 53. Utility functions for simple dependency checking
* distutils.dir_util: 54. Utility functions for operating on directories and
                        directory trees
* distutils.dist: 55. Provides the Distribution class@comma{} which
                        represents the module distribution being
                        built/installed/distributed
* distutils.errors: 56. Provides standard distutils exceptions
* distutils.extension: 57. Provides the Extension class@comma{} used to describe
                        C/C++ extension modules in setup scripts
* distutils.fancy_getopt: 58. Additional getopt functionality
* distutils.file_util: 59. Utility functions for operating on single files
* distutils.filelist: 5a. The FileList class@comma{} used for poking about the
                        file system and building lists of files.
* distutils.log: 5b. A simple logging mechanism@comma{} :pep:`282`-style
* distutils.msvccompiler: 5c. Microsoft Compiler
* distutils.spawn: 5d. Provides the spawn() function
* distutils.sysconfig: 5e. Low-level access to configuration information of the
                        Python interpreter.
* distutils.text_file: 5f. Provides the TextFile class@comma{} a simple interface
                        to text files
* distutils.unixccompiler: 60. UNIX C Compiler
* distutils.util: 61. Miscellaneous other utility functions
* distutils.version: 62. Implements classes that represent module version
                        numbers.
* doctest: 63. Test pieces of code within docstrings.
* email: 64. Package supporting the parsing@comma{}
                        manipulating@comma{} and generating email messages.
* email.charset: 65. Character Sets
* email.contentmanager: 66. Storing and Retrieving Content from MIME Parts
* email.encoders: 67. Encoders for email message payloads.
* email.errors: 68. The exception classes used by the email package.
* email.generator: 69. Generate flat text email messages from a message
                        structure.
* email.header: 6a. Representing non-ASCII headers
* email.headerregistry: 6b. Automatic Parsing of headers based on the field name
* email.iterators: 6c. Iterate over a message object tree.
* email.message: 6d. The base class representing email messages.
* email.mime: 6e. Build MIME messages.
* email.parser: 6f. Parse flat text email messages to produce a message
                        object structure.
* email.policy: 70. Controlling the parsing and generating of messages
* email.utils: 71. Miscellaneous email package utilities.
* encodings.idna: 72. Internationalized Domain Names implementation
* encodings.mbcs: 73. Windows ANSI codepage
* encodings.utf_8_sig: 74. UTF-8 codec with BOM signature
* ensurepip: 75. Bootstrapping the "pip" installer into an existing
                        Python installation or virtual environment.
* enum: 76. Implementation of an enumeration class.
* errno: 77. Standard errno system symbols.
* faulthandler: 78. Dump the Python traceback.
* fcntl: 79. The fcntl() and ioctl() system calls.
* filecmp: 7a. Compare files efficiently.
* fileinput: 7b. Loop over standard input or a list of files.
* fnmatch: 7c. Unix shell style filename pattern matching.
* fractions: 7d. Rational numbers.
* ftplib: 7e. FTP protocol client (requires sockets).
* functools: 7f. Higher-order functions and operations on callable
                        objects.
* gc: 80. Interface to the cycle-detecting garbage collector.
* getopt: 81. Portable parser for command line options; support both
                        short and long option names.
* getpass: 82. Portable reading of passwords and retrieval of the
                        userid.
* gettext: 83. Multilingual internationalization services.
* glob: 84. Unix shell style pathname pattern expansion.
* graphlib: 85. Functionality to operate with graph-like structures
* grp: 86. The group database (getgrnam() and friends).
* gzip: 87. Interfaces for gzip compression and decompression
                        using file objects.
* hashlib: 88. Secure hash and message digest algorithms.
* heapq: 89. Heap queue algorithm (a.k.a. priority queue).
* hmac: 8a. Keyed-Hashing for Message Authentication (HMAC)
                        implementation
* html: 8b. Helpers for manipulating HTML.
* html.entities: 8c. Definitions of HTML general entities.
* html.parser: 8d. A simple parser that can handle HTML and XHTML.
* http: 8e. HTTP status codes and messages
* http.client: 8f. HTTP and HTTPS protocol client (requires sockets).
* http.cookiejar: 90. Classes for automatic handling of HTTP cookies.
* http.cookies: 91. Support for HTTP state management (cookies).
* http.server: 92. HTTP server and request handlers.
* imaplib: 93. IMAP4 protocol client (requires sockets).
* imghdr: 94. Determine the type of image contained in a file or
                        byte stream.
* imp: 95. Access the implementation of the import statement.
* importlib: 96. The implementation of the import machinery.
* importlib.abc: 97. Abstract base classes related to import
* importlib.machinery: 98. Importers and path hooks
* importlib.metadata: 99. The implementation of the importlib metadata.
* importlib.resources: 9a. Package resource reading@comma{} opening@comma{} and
                        access
* importlib.util: 9b. Utility code for importers
* inspect: 9c. Extract information and source code from live objects.
* io: 9d. Core tools for working with streams.
* ipaddress: 9e. IPv4/IPv6 manipulation library.
* itertools: 9f. Functions creating iterators for efficient looping.
* json: a0. Encode and decode the JSON format.
* json.tool: a1. A command line to validate and pretty-print JSON.
* keyword: a2. Test whether a string is a keyword in Python.
* lib2to3: a3. The 2to3 library
* linecache: a4. Provides random access to individual lines from text
                        files.
* locale: a5. Internationalization services.
* logging: a6. Flexible event logging system for applications.
* logging.config: a7. Configuration of the logging module.
* logging.handlers: a8. Handlers for the logging module.
* lzma: a9. A Python wrapper for the liblzma compression library.
* mailbox: aa. Manipulate mailboxes in various formats
* mailcap: ab. Mailcap file handling.
* marshal: ac. Convert Python objects to streams of bytes and back
                        (with different constraints).
* math: ad. Mathematical functions (sin() etc.).
* mimetypes: ae. Mapping of filename extensions to MIME types.
* mmap: af. Interface to memory-mapped files for Unix and Windows.
* modulefinder: b0. Find modules used by a script.
* msilib: b1. Creation of Microsoft Installer files@comma{} and CAB
                        files.
* msvcrt: b2. Miscellaneous useful routines from the MS VC++
                        runtime.
* multiprocessing: b3. Process-based parallelism.
* multiprocessing.connection: b4. API for dealing with sockets.
* multiprocessing.dummy: b5. Dumb wrapper around threading.
* multiprocessing.managers: b6. Share data between process with shared objects.
* multiprocessing.pool: b7. Create pools of processes.
* multiprocessing.shared_memory: b8. Provides shared memory for direct access
                                 across processes.
* multiprocessing.sharedctypes: b9. Allocate ctypes objects from shared memory.
* netrc: ba. Loading of .netrc files.
* nis: bb. Interface to Sun's NIS (Yellow Pages) library.
* nntplib: bc. NNTP protocol client (requires sockets).
* numbers: bd. Numeric abstract base classes (Complex@comma{}
                        Real@comma{} Integral@comma{} etc.).
* operator: be. Functions corresponding to the standard operators.
* optparse: bf. Command-line option parsing library.
* os: c0. Miscellaneous operating system interfaces.
* os.path: c1. Operations on pathnames.
* ossaudiodev: c2. Access to OSS-compatible audio devices.
* pathlib: c3. Object-oriented filesystem paths
* pdb: c4. The Python debugger for interactive interpreters.
* pickle: c5. Convert Python objects to streams of bytes and back.
* pickletools: c6. Contains extensive comments about the pickle protocols
                        and pickle-machine opcodes@comma{} as well as some
                        useful functions.
* pipes: c7. A Python interface to Unix shell pipelines.
* pkgutil: c8. Utilities for the import system.
* platform: c9. Retrieves as much platform identifying data as
                        possible.
* plistlib: ca. Generate and parse Apple plist files.
* poplib: cb. POP3 protocol client (requires sockets).
* posix: cc. The most common POSIX system calls (normally used via
                        module os).
* pprint: cd. Data pretty printer.
* profile: ce. Python source profiler.
* pstats: cf. Statistics object for use with the profiler.
* pty: d0. Pseudo-Terminal Handling for Unix.
* pwd: d1. The password database (getpwnam() and friends).
* py_compile: d2. Generate byte-code files from Python source files.
* pyclbr: d3. Supports information extraction for a Python module
                        browser.
* pydoc: d4. Documentation generator and online help system.
* queue: d5. A synchronized queue class.
* quopri: d6. Encode and decode files using the MIME quoted-
                        printable encoding.
* random: d7. Generate pseudo-random numbers with various common
                        distributions.
* re: d8. Regular expression operations.
* readline: d9. GNU readline support for Python.
* reprlib: da. Alternate repr() implementation with size limits.
* resource: db. An interface to provide resource usage information on
                        the current process.
* rlcompleter: dc. Python identifier completion@comma{} suitable for the
                        GNU readline library.
* runpy: dd. Locate and run Python modules without importing them
                        first.
* sched: de. General purpose event scheduler.
* secrets: df. Generate secure random numbers for managing secrets.
* select: e0. Wait for I/O completion on multiple streams.
* selectors: e1. High-level I/O multiplexing.
* shelve: e2. Python object persistence.
* shlex: e3. Simple lexical analysis for Unix shell-like languages.
* shutil: e4. High-level file operations@comma{} including copying.
* signal: e5. Set handlers for asynchronous events.
* site: e6. Module responsible for site-specific configuration.
* smtpd: e7. A SMTP server implementation in Python.
* smtplib: e8. SMTP protocol client (requires sockets).
* sndhdr: e9. Determine type of a sound file.
* socket: ea. Low-level networking interface.
* socketserver: eb. A framework for network servers.
* spwd: ec. The shadow password database (getspnam() and friends).
* sqlite3: ed. A DB-API 2.0 implementation using SQLite 3.x.
* ssl: ee. TLS/SSL wrapper for socket objects
* stat: ef. Utilities for interpreting the results of
                        os.stat()@comma{} os.lstat() and os.fstat().
* statistics: f0. Mathematical statistics functions
* string: f1. Common string operations.
* stringprep: f2. String preparation@comma{} as per RFC 3453
* struct: f3. Interpret bytes as packed binary data.
* subprocess: f4. Subprocess management.
* sunau: f5. Provide an interface to the Sun AU sound format.
* symtable: f6. Interface to the compiler's internal symbol tables.
* sys: f7. Access system-specific parameters and functions.
* sysconfig: f8. Python's configuration information
* syslog: f9. An interface to the Unix syslog library routines.
* tabnanny: fa. Tool for detecting white space related problems in
                        Python source files in a directory tree.
* tarfile: fb. Read and write tar-format archive files.
* telnetlib: fc. Telnet client class.
* tempfile: fd. Generate temporary files and directories.
* termios: fe. POSIX style tty control.
* test: ff. Regression tests package containing the testing suite
                        for Python.
* test.support: 100. Support for Python's regression test suite.
* test.support.bytecode_helper: 101. Support tools for testing correct bytecode
                                generation.
* test.support.import_helper: 102. Support for import tests.
* test.support.os_helper: 103. Support for os tests.
* test.support.script_helper: 104. Support for Python's script execution tests.
* test.support.socket_helper: 105. Support for socket tests.
* test.support.threading_helper: 106. Support for threading tests.
* test.support.warnings_helper: 107. Support for warnings tests.
* textwrap: 108. Text wrapping and filling
* threading: 109. Thread-based parallelism.
* time: 10a. Time access and conversions.
* timeit: 10b. Measure the execution time of small code snippets.
* tkinter: 10c. Interface to Tcl/Tk for graphical user interfaces
* tkinter.colorchooser: 10d. Color choosing dialog
* tkinter.commondialog: 10e. Tkinter base class for dialogs
* tkinter.dnd: 10f. Tkinter drag-and-drop interface
* tkinter.filedialog: 110. Dialog classes for file selection
* tkinter.font: 111. Tkinter font-wrapping class
* tkinter.messagebox: 112. Various types of alert dialogs
* tkinter.scrolledtext: 113. Text widget with a vertical scroll bar.
* tkinter.simpledialog: 114. Simple dialog windows
* tkinter.tix: 115. Tk Extension Widgets for Tkinter
* tkinter.ttk: 116. Tk themed widget set
* token: 117. Constants representing terminal nodes of the parse
                        tree.
* tokenize: 118. Lexical scanner for Python source code.
* trace: 119. Trace or track Python statement execution.
* traceback: 11a. Print or retrieve a stack traceback.
* tracemalloc: 11b. Trace memory allocations.
* tty: 11c. Utility functions that perform common terminal control
                        operations.
* turtle: 11d. An educational framework for simple graphics
                        applications
* turtledemo: 11e. A viewer for example turtle scripts
* types: 11f. Names for built-in types.
* typing: 120. Support for type hints (see :pep:`484`).
* unicodedata: 121. Access the Unicode Database.
* unittest: 122. Unit testing framework for Python.
* unittest.mock: 123. Mock object library.
* urllib: 124. 
* urllib.error: 125. Exception classes raised by urllib.request.
* urllib.parse: 126. Parse URLs into or assemble them from components.
* urllib.request: 127. Extensible library for opening URLs.
* urllib.response: 128. Response classes used by urllib.
* urllib.robotparser: 129. Load a robots.txt file and answer questions about
                        fetchability of other URLs.
* uu: 12a. Encode and decode files in uuencode format.
* uuid: 12b. UUID objects (universally unique identifiers)
                        according to RFC 4122
* venv: 12c. Creation of virtual environments.
* warnings: 12d. Issue warning messages and control their disposition.
* wave: 12e. Provide an interface to the WAV sound format.
* weakref: 12f. Support for weak references and weak dictionaries.
* webbrowser: 130. Easy-to-use controller for web browsers.
* winreg: 131. Routines and objects for manipulating the Windows
                        registry.
* winsound: 132. Access to the sound-playing machinery for Windows.
* wsgiref: 133. WSGI Utilities and Reference Implementation.
* wsgiref.handlers: 134. WSGI server/gateway base classes.
* wsgiref.headers: 135. WSGI response header tools.
* wsgiref.simple_server: 136. A simple WSGI HTTP server.
* wsgiref.util: 137. WSGI environment utilities.
* wsgiref.validate: 138. WSGI conformance checker.
* xdrlib: 139. Encoders and decoders for the External Data
                        Representation (XDR).
* xml: 13a. Package containing XML processing modules
* xml.dom: 13b. Document Object Model API for Python.
* xml.dom.minidom: 13c. Minimal Document Object Model (DOM) implementation.
* xml.dom.pulldom: 13d. Support for building partial DOM trees from SAX
                        events.
* xml.etree.ElementTree: 13e. Implementation of the ElementTree API.
* xml.parsers.expat: 13f. An interface to the Expat non-validating XML parser.
* xml.parsers.expat.errors: 140. 
* xml.parsers.expat.model: 141. 
* xml.sax: 142. Package containing SAX2 base classes and convenience
                        functions.
* xml.sax.handler: 143. Base classes for SAX event handlers.
* xml.sax.saxutils: 144. Convenience functions and classes for use with SAX.
* xml.sax.xmlreader: 145. Interface which SAX-compliant XML parsers must
                        implement.
* xmlrpc.client: 146. XML-RPC client access.
* xmlrpc.server: 147. Basic XML-RPC server implementations.
* zipapp: 148. Manage executable Python zip archives
* zipfile: 149. Read and write ZIP-format archive files.
* zipimport: 14a. Support for importing Python modules from ZIP
                        archives.
* zlib: 14b. Low-level interface to compression and decompression
                        routines compatible with gzip.
* zoneinfo: 14c. IANA time zone support
@end menu


@node Index,,Python Module Index,Top
@unnumbered Index


@printindex ge

@anchor{using/cmdline audit_event_cpython_run_file_0}@w{                              }
@anchor{library/sqlite3 sqlite3 load_extension}@w{                              }
@anchor{using/cmdline audit_event_cpython_run_stdin_0}@w{                              }
@anchor{39ef}@w{                              }
@anchor{library/sqlite3 sqlite3 enable_load_extension}@w{                              }
@anchor{39f7}@w{                              }
@anchor{39f6}@w{                              }
@anchor{39f0}@w{                              }

@c %**end of body
@bye
